2024-04-01T01:11:38.623 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-01T01:11:38.623 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228 --name teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228/orig.config.yaml 2024-04-01T01:11:38.679 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T01:11:38.777 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7633228' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 0 mountpoint: /tmp/cephfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 kclient: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 1 mountpoint: /tmp/cephfs selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8238 sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 sleep_before_teardown: 0 subset: 361/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi070.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM9jQWWj66GDM7ZdisuynmE338mNKCrjMq5PgThFsESeE9oGE/S6q4iF3nCSqxngmE3/FNS+yWwdmDuPNiS4O2w= smithi081.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN/QIvzLiOCB0dWOLoXDnjWVcn0qfrxSviZXXlAEWi13md98CtWX7jhxaNLhKeKPH+uESkprkUephqY1d6i7R8c= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHAAbmeIHy5hq2+LSvDcE19DMskEyf/tRbSjbdTKPbKse28S85pcjFV+2ouCmZBFnd4SrA+geFP41uA2JmPyXpg= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - exec: client.0: - sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/ - sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres - sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data - sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?" - sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local - sudo restorecon -R -v /tmp/cephfs/postgres - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf - sudo -u postgres -- postgresql-setup --initdb - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo systemctl start postgresql - sudo -u postgres -- pgbench -s 32 -i - sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900 - sudo systemctl stop postgresql - sudo ls -lZaR /tmp/cephfs/postgres/ teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-31_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-01T01:11:38.777 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa; will attempt to use it 2024-04-01T01:11:38.778 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks 2024-04-01T01:11:38.778 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-01T01:11:38.780 INFO:teuthology.task.internal:Checking packages... 2024-04-01T01:11:38.806 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '384f31f3d3aee7f8ba3f60846b8844c049165583' 2024-04-01T01:11:38.806 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-01T01:11:38.806 INFO:teuthology.packaging:ref: None 2024-04-01T01:11:38.806 INFO:teuthology.packaging:tag: None 2024-04-01T01:11:38.806 INFO:teuthology.packaging:branch: squid 2024-04-01T01:11:38.806 INFO:teuthology.packaging:sha1: 384f31f3d3aee7f8ba3f60846b8844c049165583 2024-04-01T01:11:38.806 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-01T01:11:39.154 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1977.g384f31f3 2024-04-01T01:11:39.155 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-01T01:11:39.161 INFO:teuthology.task.internal:no buildpackages task found 2024-04-01T01:11:39.161 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-01T01:11:39.166 INFO:teuthology.task.internal:Saving configuration 2024-04-01T01:11:39.187 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-01T01:11:39.193 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-01T01:11:39.238 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-01 01:04:59.171091', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM9jQWWj66GDM7ZdisuynmE338mNKCrjMq5PgThFsESeE9oGE/S6q4iF3nCSqxngmE3/FNS+yWwdmDuPNiS4O2w='} 2024-04-01T01:11:39.271 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-01 01:04:59.168871', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN/QIvzLiOCB0dWOLoXDnjWVcn0qfrxSviZXXlAEWi13md98CtWX7jhxaNLhKeKPH+uESkprkUephqY1d6i7R8c='} 2024-04-01T01:11:39.305 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-01 01:04:59.170063', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHAAbmeIHy5hq2+LSvDcE19DMskEyf/tRbSjbdTKPbKse28S85pcjFV+2ouCmZBFnd4SrA+geFP41uA2JmPyXpg='} 2024-04-01T01:11:39.306 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-01T01:11:39.312 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-01T01:11:39.312 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-01T01:11:39.312 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-01T01:11:39.312 INFO:teuthology.run_tasks:Running task console_log... 2024-04-01T01:11:39.496 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5bd3d4c160>, signals=[15]) 2024-04-01T01:11:39.496 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-01T01:11:39.502 INFO:teuthology.task.internal:Opening connections... 2024-04-01T01:11:39.502 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2024-04-01T01:11:39.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T01:11:39.626 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2024-04-01T01:11:39.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T01:11:39.742 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-04-01T01:11:39.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T01:11:39.861 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-01T01:11:39.867 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2024-04-01T01:11:39.884 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2024-04-01T01:11:39.884 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2024-04-01T01:11:39.941 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Stream" 2024-04-01T01:11:39.941 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="9" 2024-04-01T01:11:39.941 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2024-04-01T01:11:39.941 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2024-04-01T01:11:39.941 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="9" 2024-04-01T01:11:39.941 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el9" 2024-04-01T01:11:39.941 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T01:11:39.941 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2024-04-01T01:11:39.941 INFO:teuthology.orchestra.run.smithi070.stdout:LOGO="fedora-logo-icon" 2024-04-01T01:11:39.941 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T01:11:39.941 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://centos.org/" 2024-04-01T01:11:39.941 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T01:11:39.941 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T01:11:39.942 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T01:11:39.942 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2024-04-01T01:11:39.970 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2024-04-01T01:11:39.989 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2024-04-01T01:11:39.989 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2024-04-01T01:11:40.046 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="CentOS Stream" 2024-04-01T01:11:40.046 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="9" 2024-04-01T01:11:40.046 INFO:teuthology.orchestra.run.smithi081.stdout:ID="centos" 2024-04-01T01:11:40.046 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="rhel fedora" 2024-04-01T01:11:40.046 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="9" 2024-04-01T01:11:40.046 INFO:teuthology.orchestra.run.smithi081.stdout:PLATFORM_ID="platform:el9" 2024-04-01T01:11:40.047 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T01:11:40.047 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2024-04-01T01:11:40.047 INFO:teuthology.orchestra.run.smithi081.stdout:LOGO="fedora-logo-icon" 2024-04-01T01:11:40.047 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T01:11:40.047 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://centos.org/" 2024-04-01T01:11:40.047 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T01:11:40.047 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T01:11:40.047 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T01:11:40.047 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2024-04-01T01:11:40.079 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-04-01T01:11:40.097 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-04-01T01:11:40.097 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-04-01T01:11:40.154 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="CentOS Stream" 2024-04-01T01:11:40.154 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="9" 2024-04-01T01:11:40.154 INFO:teuthology.orchestra.run.smithi189.stdout:ID="centos" 2024-04-01T01:11:40.154 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="rhel fedora" 2024-04-01T01:11:40.155 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="9" 2024-04-01T01:11:40.155 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el9" 2024-04-01T01:11:40.155 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-01T01:11:40.155 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31" 2024-04-01T01:11:40.155 INFO:teuthology.orchestra.run.smithi189.stdout:LOGO="fedora-logo-icon" 2024-04-01T01:11:40.155 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-01T01:11:40.155 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://centos.org/" 2024-04-01T01:11:40.155 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-01T01:11:40.155 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-01T01:11:40.155 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-01T01:11:40.155 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-04-01T01:11:40.187 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-01T01:11:40.197 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-01T01:11:40.202 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-01T01:11:40.202 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2024-04-01T01:11:40.205 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2024-04-01T01:11:40.207 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-04-01T01:11:40.223 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-01T01:11:40.228 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-01T01:11:40.229 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2024-04-01T01:11:40.262 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2024-04-01T01:11:40.264 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-04-01T01:11:40.300 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-01T01:11:40.403 INFO:teuthology.run_tasks:Running task kernel... 2024-04-01T01:11:40.419 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-01T01:11:40.419 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-01T01:11:40.419 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-01T01:11:40.420 DEBUG:teuthology.orchestra.run.smithi081:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-01T01:11:40.420 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-01T01:11:40.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T01:11:40.436 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-04-01T01:11:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T01:11:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T01:11:40.438 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-04-01T01:11:40.439 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-04-01T01:11:40.492 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-361.el9.x86_64 2024-04-01T01:11:40.493 INFO:teuthology.task.kernel:Running kernel on smithi081: 5.14.0-361.el9.x86_64 2024-04-01T01:11:40.493 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum install -y kernel 2024-04-01T01:11:40.495 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-361.el9.x86_64 2024-04-01T01:11:40.495 INFO:teuthology.task.kernel:Running kernel on smithi070: 5.14.0-361.el9.x86_64 2024-04-01T01:11:40.496 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum install -y kernel 2024-04-01T01:11:40.497 INFO:teuthology.orchestra.run.smithi189.stdout:5.14.0-361.el9.x86_64 2024-04-01T01:11:40.497 INFO:teuthology.task.kernel:Running kernel on smithi189: 5.14.0-361.el9.x86_64 2024-04-01T01:11:40.497 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum install -y kernel 2024-04-01T01:11:43.056 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - AppStream 7.3 MB/s | 6.7 MB 00:00 2024-04-01T01:11:43.114 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - AppStream 7.2 MB/s | 6.7 MB 00:00 2024-04-01T01:11:43.132 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - AppStream 7.0 MB/s | 6.7 MB 00:00 2024-04-01T01:11:44.431 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-04-01T01:11:44.522 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-04-01T01:11:44.750 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-04-01T01:11:45.599 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 9.5 MB/s | 8.0 MB 00:00 2024-04-01T01:11:46.298 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 8.0 MB 00:01 2024-04-01T01:11:47.432 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-01T01:11:47.983 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-04-01T01:11:51.643 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 6.1 MB 00:00 2024-04-01T01:11:51.991 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-04-01T01:11:54.092 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-04-01T01:11:54.458 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-04-01T01:11:54.845 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 825 kB/s | 8.0 MB 00:09 2024-04-01T01:11:55.879 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-01T01:11:56.704 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-04-01T01:11:58.304 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 5.1 MB/s | 21 MB 00:04 2024-04-01T01:12:00.826 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-01T01:12:00.888 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 8.3 MB/s | 6.1 MB 00:00 2024-04-01T01:12:02.339 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 16 kB 00:00 2024-04-01T01:12:03.082 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-01T01:12:03.083 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-01T01:12:03.171 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-01T01:12:03.176 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-01T01:12:03.176 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-04-01T01:12:03.176 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-01T01:12:03.176 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-01T01:12:03.176 INFO:teuthology.orchestra.run.smithi070.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-04-01T01:12:03.177 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-04-01T01:12:03.177 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-04-01T01:12:03.177 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-04-01T01:12:03.177 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-04-01T01:12:03.177 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:12:03.177 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-01T01:12:03.177 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-01T01:12:03.177 INFO:teuthology.orchestra.run.smithi070.stdout:Install 4 Packages 2024-04-01T01:12:03.177 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:12:03.178 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 76 M 2024-04-01T01:12:03.178 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 123 M 2024-04-01T01:12:03.178 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-01T01:12:03.318 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-01T01:12:03.485 INFO:teuthology.orchestra.run.smithi070.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 624 kB/s | 192 kB 00:00 2024-04-01T01:12:04.402 INFO:teuthology.orchestra.run.smithi070.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 12 MB/s | 15 MB 00:01 2024-04-01T01:12:05.435 INFO:teuthology.orchestra.run.smithi070.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 14 MB/s | 28 MB 00:01 2024-04-01T01:12:05.558 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-01T01:12:05.559 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-01T01:12:05.673 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-04-01T01:12:05.681 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-04-01T01:12:05.681 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-04-01T01:12:05.681 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-04-01T01:12:05.681 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-04-01T01:12:05.681 INFO:teuthology.orchestra.run.smithi081.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-04-01T01:12:05.681 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-04-01T01:12:05.681 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-04-01T01:12:05.681 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-04-01T01:12:05.681 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-04-01T01:12:05.681 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:12:05.682 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-04-01T01:12:05.682 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-04-01T01:12:05.682 INFO:teuthology.orchestra.run.smithi081.stdout:Install 4 Packages 2024-04-01T01:12:05.682 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:12:05.682 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 76 M 2024-04-01T01:12:05.682 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 123 M 2024-04-01T01:12:05.682 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-04-01T01:12:05.878 INFO:teuthology.orchestra.run.smithi081.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 981 kB/s | 192 kB 00:00 2024-04-01T01:12:05.952 INFO:teuthology.orchestra.run.smithi070.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 12 MB/s | 34 MB 00:02 2024-04-01T01:12:05.956 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-01T01:12:05.956 INFO:teuthology.orchestra.run.smithi070.stdout:Total 28 MB/s | 76 MB 00:02 2024-04-01T01:12:05.956 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-01T01:12:06.350 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-01T01:12:06.351 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-01T01:12:06.837 INFO:teuthology.orchestra.run.smithi081.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 13 MB/s | 15 MB 00:01 2024-04-01T01:12:06.851 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 4.7 MB/s | 21 MB 00:04 2024-04-01T01:12:07.193 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-01T01:12:07.193 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-01T01:12:07.762 INFO:teuthology.orchestra.run.smithi081.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 16 MB/s | 34 MB 00:02 2024-04-01T01:12:09.303 INFO:teuthology.orchestra.run.smithi081.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 8.1 MB/s | 28 MB 00:03 2024-04-01T01:12:09.307 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-04-01T01:12:09.307 INFO:teuthology.orchestra.run.smithi081.stdout:Total 21 MB/s | 76 MB 00:03 2024-04-01T01:12:09.307 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-04-01T01:12:09.504 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-01T01:12:09.675 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-04-01T01:12:09.675 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-04-01T01:12:10.102 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-04-01T01:12:10.186 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T01:12:10.533 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-04-01T01:12:10.533 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-04-01T01:12:11.756 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-01T01:12:12.831 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-04-01T01:12:12.945 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T01:12:13.087 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T01:12:13.418 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-04-01T01:12:13.505 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T01:12:14.038 INFO:teuthology.orchestra.run.smithi189.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-01T01:12:14.039 INFO:teuthology.orchestra.run.smithi189.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-01T01:12:14.133 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-01T01:12:14.142 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-01T01:12:14.142 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-01T01:12:14.142 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-01T01:12:14.142 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-01T01:12:14.142 INFO:teuthology.orchestra.run.smithi189.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-04-01T01:12:14.142 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-01T01:12:14.142 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-04-01T01:12:14.142 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-04-01T01:12:14.142 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-04-01T01:12:14.143 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:12:14.143 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-01T01:12:14.143 INFO:teuthology.orchestra.run.smithi189.stdout:================================================================================ 2024-04-01T01:12:14.143 INFO:teuthology.orchestra.run.smithi189.stdout:Install 4 Packages 2024-04-01T01:12:14.143 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:12:14.143 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 76 M 2024-04-01T01:12:14.144 INFO:teuthology.orchestra.run.smithi189.stdout:Installed size: 123 M 2024-04-01T01:12:14.144 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-01T01:12:14.416 INFO:teuthology.orchestra.run.smithi189.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 704 kB/s | 192 kB 00:00 2024-04-01T01:12:15.341 INFO:teuthology.orchestra.run.smithi189.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 13 MB/s | 15 MB 00:01 2024-04-01T01:12:16.000 INFO:teuthology.orchestra.run.smithi189.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-04-01T01:12:16.254 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T01:12:16.409 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T01:12:16.516 INFO:teuthology.orchestra.run.smithi189.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 14 MB/s | 34 MB 00:02 2024-04-01T01:12:16.519 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-01T01:12:16.520 INFO:teuthology.orchestra.run.smithi189.stdout:Total 32 MB/s | 76 MB 00:02 2024-04-01T01:12:16.520 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-01T01:12:16.905 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-01T01:12:16.905 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-01T01:12:17.729 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-01T01:12:17.730 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-01T01:12:17.793 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T01:12:17.829 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:12:20.036 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-01T01:12:20.849 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-04-01T01:12:20.942 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T01:12:21.099 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T01:12:21.200 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:12:22.549 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:12:23.690 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T01:12:23.819 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T01:12:25.926 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:12:28.571 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T01:12:28.677 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:12:33.441 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:12:51.749 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:12:51.749 INFO:teuthology.orchestra.run.smithi070.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-01T01:12:51.749 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:12:51.792 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:12:52.160 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:12:52.160 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-04-01T01:12:52.161 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T01:12:52.161 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T01:12:52.595 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:12:52.595 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:12:52.595 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-01T01:12:52.595 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-5.14.0-434.el9.x86_64 2024-04-01T01:12:52.595 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-04-01T01:12:52.595 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-04-01T01:12:52.595 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-04-01T01:12:52.595 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:12:52.595 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-01T01:12:52.905 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-04-01T01:12:53.044 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-01T01:12:53.045 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-01T01:12:53.045 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-04-01T01:12:53.045 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-01T01:12:53.045 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-01T01:12:53.045 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-01T01:12:53.045 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi070.front.sepia.ceph.com, path=None, version=distro) 2024-04-01T01:12:53.045 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-04-01T01:12:53.125 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-01T01:12:53.126 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-01T01:12:53.126 DEBUG:teuthology.orchestra.run.smithi070:> sudo rpm -qi grub2-tools 2024-04-01T01:12:53.178 INFO:teuthology.orchestra.run.smithi070.stdout:Name : grub2-tools 2024-04-01T01:12:53.178 INFO:teuthology.orchestra.run.smithi070.stdout:Epoch : 1 2024-04-01T01:12:53.178 INFO:teuthology.orchestra.run.smithi070.stdout:Version : 2.06 2024-04-01T01:12:53.178 INFO:teuthology.orchestra.run.smithi070.stdout:Release : 68.el9 2024-04-01T01:12:53.178 INFO:teuthology.orchestra.run.smithi070.stdout:Architecture: x86_64 2024-04-01T01:12:53.178 INFO:teuthology.orchestra.run.smithi070.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-01T01:12:53.178 INFO:teuthology.orchestra.run.smithi070.stdout:Group : Unspecified 2024-04-01T01:12:53.178 INFO:teuthology.orchestra.run.smithi070.stdout:Size : 8264254 2024-04-01T01:12:53.179 INFO:teuthology.orchestra.run.smithi070.stdout:License : GPLv3+ 2024-04-01T01:12:53.179 INFO:teuthology.orchestra.run.smithi070.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-01T01:12:53.179 INFO:teuthology.orchestra.run.smithi070.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-01T01:12:53.179 INFO:teuthology.orchestra.run.smithi070.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-01T01:12:53.179 INFO:teuthology.orchestra.run.smithi070.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-01T01:12:53.179 INFO:teuthology.orchestra.run.smithi070.stdout:Packager : builder@centos.org 2024-04-01T01:12:53.179 INFO:teuthology.orchestra.run.smithi070.stdout:Vendor : CentOS 2024-04-01T01:12:53.179 INFO:teuthology.orchestra.run.smithi070.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-01T01:12:53.179 INFO:teuthology.orchestra.run.smithi070.stdout:Summary : Support tools for GRUB. 2024-04-01T01:12:53.179 INFO:teuthology.orchestra.run.smithi070.stdout:Description : 2024-04-01T01:12:53.179 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:12:53.179 INFO:teuthology.orchestra.run.smithi070.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-01T01:12:53.179 INFO:teuthology.orchestra.run.smithi070.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-01T01:12:53.179 INFO:teuthology.orchestra.run.smithi070.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-01T01:12:53.179 INFO:teuthology.orchestra.run.smithi070.stdout:hardware devices. 2024-04-01T01:12:53.180 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:12:53.180 INFO:teuthology.orchestra.run.smithi070.stdout:This subpackage provides tools for support of all platforms. 2024-04-01T01:12:53.180 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-01T01:12:53.181 INFO:teuthology.task.kernel:Updating grub on smithi070 to boot 5.14.0-434.el9.x86_64 2024-04-01T01:12:53.181 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-01T01:12:53.764 INFO:teuthology.orchestra.run.smithi070.stderr:Generating grub configuration file ... 2024-04-01T01:12:55.241 INFO:teuthology.orchestra.run.smithi070.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-01T01:12:55.271 INFO:teuthology.orchestra.run.smithi070.stderr:done 2024-04-01T01:12:55.273 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-04-01T01:12:55.288 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.ONAqFEKLKI 2024-04-01T01:12:55.289 DEBUG:teuthology.orchestra.run.smithi070:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ONAqFEKLKI 2024-04-01T01:12:55.357 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.ONAqFEKLKI 2024-04-01T01:12:55.407 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:12:55.407 INFO:teuthology.orchestra.run.smithi081.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-01T01:12:55.407 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:12:55.450 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:12:55.495 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.ONAqFEKLKI is 6KB 2024-04-01T01:12:55.544 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.ONAqFEKLKI 2024-04-01T01:12:55.558 DEBUG:teuthology.orchestra.run.smithi070:> sudo /bin/ls /boot/loader/entries || true 2024-04-01T01:12:55.624 INFO:teuthology.orchestra.run.smithi070.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-01T01:12:55.624 INFO:teuthology.orchestra.run.smithi070.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-01T01:12:55.624 INFO:teuthology.orchestra.run.smithi070.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-01T01:12:55.624 INFO:teuthology.orchestra.run.smithi070.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-04-01T01:12:55.626 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-04-01T01:12:55.758 DEBUG:teuthology.orchestra.run.smithi070:> sudo shutdown -r now 2024-04-01T01:12:55.846 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:12:55.846 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-04-01T01:12:55.846 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T01:12:55.846 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T01:12:56.279 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:12:56.279 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:12:56.279 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-04-01T01:12:56.279 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-5.14.0-434.el9.x86_64 2024-04-01T01:12:56.279 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-04-01T01:12:56.279 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-04-01T01:12:56.279 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-04-01T01:12:56.279 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:12:56.279 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-04-01T01:12:56.602 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2024-04-01T01:12:56.752 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-01T01:12:56.752 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-01T01:12:56.752 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-04-01T01:12:56.752 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-01T01:12:56.752 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-01T01:12:56.752 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-01T01:12:56.752 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=None, version=distro) 2024-04-01T01:12:56.753 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2024-04-01T01:12:56.833 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-01T01:12:56.833 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-01T01:12:56.833 DEBUG:teuthology.orchestra.run.smithi081:> sudo rpm -qi grub2-tools 2024-04-01T01:12:56.866 INFO:teuthology.orchestra.run.smithi081.stdout:Name : grub2-tools 2024-04-01T01:12:56.867 INFO:teuthology.orchestra.run.smithi081.stdout:Epoch : 1 2024-04-01T01:12:56.867 INFO:teuthology.orchestra.run.smithi081.stdout:Version : 2.06 2024-04-01T01:12:56.867 INFO:teuthology.orchestra.run.smithi081.stdout:Release : 68.el9 2024-04-01T01:12:56.867 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: x86_64 2024-04-01T01:12:56.867 INFO:teuthology.orchestra.run.smithi081.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-01T01:12:56.867 INFO:teuthology.orchestra.run.smithi081.stdout:Group : Unspecified 2024-04-01T01:12:56.867 INFO:teuthology.orchestra.run.smithi081.stdout:Size : 8264254 2024-04-01T01:12:56.867 INFO:teuthology.orchestra.run.smithi081.stdout:License : GPLv3+ 2024-04-01T01:12:56.867 INFO:teuthology.orchestra.run.smithi081.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-01T01:12:56.867 INFO:teuthology.orchestra.run.smithi081.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-01T01:12:56.867 INFO:teuthology.orchestra.run.smithi081.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-01T01:12:56.867 INFO:teuthology.orchestra.run.smithi081.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-01T01:12:56.868 INFO:teuthology.orchestra.run.smithi081.stdout:Packager : builder@centos.org 2024-04-01T01:12:56.868 INFO:teuthology.orchestra.run.smithi081.stdout:Vendor : CentOS 2024-04-01T01:12:56.868 INFO:teuthology.orchestra.run.smithi081.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-01T01:12:56.868 INFO:teuthology.orchestra.run.smithi081.stdout:Summary : Support tools for GRUB. 2024-04-01T01:12:56.868 INFO:teuthology.orchestra.run.smithi081.stdout:Description : 2024-04-01T01:12:56.868 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:12:56.868 INFO:teuthology.orchestra.run.smithi081.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-01T01:12:56.868 INFO:teuthology.orchestra.run.smithi081.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-01T01:12:56.868 INFO:teuthology.orchestra.run.smithi081.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-01T01:12:56.868 INFO:teuthology.orchestra.run.smithi081.stdout:hardware devices. 2024-04-01T01:12:56.868 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:12:56.868 INFO:teuthology.orchestra.run.smithi081.stdout:This subpackage provides tools for support of all platforms. 2024-04-01T01:12:56.869 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-01T01:12:56.869 INFO:teuthology.task.kernel:Updating grub on smithi081 to boot 5.14.0-434.el9.x86_64 2024-04-01T01:12:56.869 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-01T01:12:57.471 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2024-04-01T01:12:58.932 INFO:teuthology.orchestra.run.smithi081.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-01T01:12:58.964 INFO:teuthology.orchestra.run.smithi081.stderr:done 2024-04-01T01:12:58.967 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-04-01T01:12:58.982 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.yvg5cuQ8rt 2024-04-01T01:12:58.982 DEBUG:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yvg5cuQ8rt 2024-04-01T01:12:59.051 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.yvg5cuQ8rt 2024-04-01T01:12:59.203 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.yvg5cuQ8rt is 6KB 2024-04-01T01:12:59.251 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.yvg5cuQ8rt 2024-04-01T01:12:59.265 DEBUG:teuthology.orchestra.run.smithi081:> sudo /bin/ls /boot/loader/entries || true 2024-04-01T01:12:59.329 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-01T01:12:59.329 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-01T01:12:59.329 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-01T01:12:59.329 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-04-01T01:12:59.331 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-04-01T01:12:59.476 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2024-04-01T01:13:03.095 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:13:03.095 INFO:teuthology.orchestra.run.smithi189.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-01T01:13:03.095 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:13:03.138 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:13:03.481 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:13:03.481 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-04-01T01:13:03.481 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-04-01T01:13:03.481 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-04-01T01:13:03.861 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-04-01T01:13:03.862 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:13:03.862 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-01T01:13:03.862 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-5.14.0-434.el9.x86_64 2024-04-01T01:13:03.862 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-04-01T01:13:03.862 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-04-01T01:13:03.862 INFO:teuthology.orchestra.run.smithi189.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-04-01T01:13:03.862 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:13:03.862 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-01T01:13:04.192 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q kernel | sort -rV | head -n 1 2024-04-01T01:13:04.306 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-01T01:13:04.306 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-01T01:13:04.307 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-04-01T01:13:04.307 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-01T01:13:04.307 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-01T01:13:04.307 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-01T01:13:04.307 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi189.front.sepia.ceph.com, path=None, version=distro) 2024-04-01T01:13:04.307 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q kernel | sort -rV | head -n 1 2024-04-01T01:13:04.388 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-5.14.0-434.el9.x86_64 2024-04-01T01:13:04.388 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-04-01T01:13:04.388 DEBUG:teuthology.orchestra.run.smithi189:> sudo rpm -qi grub2-tools 2024-04-01T01:13:04.422 INFO:teuthology.orchestra.run.smithi189.stdout:Name : grub2-tools 2024-04-01T01:13:04.423 INFO:teuthology.orchestra.run.smithi189.stdout:Epoch : 1 2024-04-01T01:13:04.423 INFO:teuthology.orchestra.run.smithi189.stdout:Version : 2.06 2024-04-01T01:13:04.423 INFO:teuthology.orchestra.run.smithi189.stdout:Release : 68.el9 2024-04-01T01:13:04.423 INFO:teuthology.orchestra.run.smithi189.stdout:Architecture: x86_64 2024-04-01T01:13:04.423 INFO:teuthology.orchestra.run.smithi189.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-01T01:13:04.423 INFO:teuthology.orchestra.run.smithi189.stdout:Group : Unspecified 2024-04-01T01:13:04.423 INFO:teuthology.orchestra.run.smithi189.stdout:Size : 8264254 2024-04-01T01:13:04.423 INFO:teuthology.orchestra.run.smithi189.stdout:License : GPLv3+ 2024-04-01T01:13:04.423 INFO:teuthology.orchestra.run.smithi189.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-01T01:13:04.423 INFO:teuthology.orchestra.run.smithi189.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-01T01:13:04.423 INFO:teuthology.orchestra.run.smithi189.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-01T01:13:04.423 INFO:teuthology.orchestra.run.smithi189.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-01T01:13:04.423 INFO:teuthology.orchestra.run.smithi189.stdout:Packager : builder@centos.org 2024-04-01T01:13:04.423 INFO:teuthology.orchestra.run.smithi189.stdout:Vendor : CentOS 2024-04-01T01:13:04.424 INFO:teuthology.orchestra.run.smithi189.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-01T01:13:04.424 INFO:teuthology.orchestra.run.smithi189.stdout:Summary : Support tools for GRUB. 2024-04-01T01:13:04.424 INFO:teuthology.orchestra.run.smithi189.stdout:Description : 2024-04-01T01:13:04.424 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:13:04.424 INFO:teuthology.orchestra.run.smithi189.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-01T01:13:04.424 INFO:teuthology.orchestra.run.smithi189.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-01T01:13:04.424 INFO:teuthology.orchestra.run.smithi189.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-01T01:13:04.424 INFO:teuthology.orchestra.run.smithi189.stdout:hardware devices. 2024-04-01T01:13:04.424 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:13:04.424 INFO:teuthology.orchestra.run.smithi189.stdout:This subpackage provides tools for support of all platforms. 2024-04-01T01:13:04.425 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-01T01:13:04.425 INFO:teuthology.task.kernel:Updating grub on smithi189 to boot 5.14.0-434.el9.x86_64 2024-04-01T01:13:04.425 DEBUG:teuthology.orchestra.run.smithi189:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-01T01:13:05.079 INFO:teuthology.orchestra.run.smithi189.stderr:Generating grub configuration file ... 2024-04-01T01:13:06.585 INFO:teuthology.orchestra.run.smithi189.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-01T01:13:06.616 INFO:teuthology.orchestra.run.smithi189.stderr:done 2024-04-01T01:13:06.618 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-04-01T01:13:06.633 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.IPJW2UTPrI 2024-04-01T01:13:06.633 DEBUG:teuthology.orchestra.run.smithi189:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.IPJW2UTPrI 2024-04-01T01:13:06.701 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.IPJW2UTPrI 2024-04-01T01:13:06.835 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.IPJW2UTPrI is 6KB 2024-04-01T01:13:06.885 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.IPJW2UTPrI 2024-04-01T01:13:06.899 DEBUG:teuthology.orchestra.run.smithi189:> sudo /bin/ls /boot/loader/entries || true 2024-04-01T01:13:06.964 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-01T01:13:06.964 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-01T01:13:06.964 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-01T01:13:06.965 INFO:teuthology.orchestra.run.smithi189.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-04-01T01:13:06.966 DEBUG:teuthology.orchestra.run.smithi189:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-04-01T01:13:07.118 DEBUG:teuthology.orchestra.run.smithi189:> sudo shutdown -r now 2024-04-01T01:13:25.768 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-01T01:13:25.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-01T01:13:25.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T01:13:29.482 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-01T01:13:29.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-01T01:13:29.484 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T01:13:37.129 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-01T01:13:37.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-01T01:13:37.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T01:13:44.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2024-04-01T01:13:47.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-04-01T01:13:53.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-01T01:13:53.230 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T01:13:55.490 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-04-01T01:13:56.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2024-04-01T01:13:56.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-01T01:13:56.809 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T01:13:59.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-04-01T01:14:04.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-01T01:14:04.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T01:14:07.550 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-04-01T01:14:08.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-01T01:14:08.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T01:14:11.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-01T01:14:11.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T01:14:19.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-01T01:14:19.559 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T01:14:37.983 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-04-01T01:14:40.131 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-04-01T01:14:40.587 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-01T01:14:40.588 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-04-01T01:14:40.588 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-04-01T01:14:40.602 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-434.el9.x86_64 2024-04-01T01:14:40.602 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-04-01T01:14:40.602 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-01T01:14:40.602 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-01T01:14:41.604 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-01T01:14:41.604 DEBUG:teuthology.orchestra.run.smithi070:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-01T01:14:41.664 INFO:teuthology.orchestra.run.smithi070.stdout:ttyS1 2024-04-01T01:14:41.711 DEBUG:teuthology.parallel:result is None 2024-04-01T01:14:43.796 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-04-01T01:14:44.369 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-01T01:14:44.369 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-04-01T01:14:44.370 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-04-01T01:14:44.384 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-434.el9.x86_64 2024-04-01T01:14:44.384 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-04-01T01:14:44.384 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-01T01:14:44.384 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-01T01:14:45.385 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-01T01:14:45.385 DEBUG:teuthology.orchestra.run.smithi081:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-01T01:14:45.473 INFO:teuthology.orchestra.run.smithi081.stdout:ttyS1 2024-04-01T01:14:45.497 DEBUG:teuthology.parallel:result is None 2024-04-01T01:14:52.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-01T01:14:52.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T01:14:53.352 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-04-01T01:14:53.794 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-01T01:14:53.794 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-434.el9.x86_64"... 2024-04-01T01:14:53.794 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-04-01T01:14:53.808 INFO:teuthology.orchestra.run.smithi189.stdout:5.14.0-434.el9.x86_64 2024-04-01T01:14:53.809 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-04-01T01:14:53.809 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-01T01:14:53.809 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-01T01:14:54.810 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-01T01:14:54.810 DEBUG:teuthology.orchestra.run.smithi189:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-01T01:14:54.878 INFO:teuthology.orchestra.run.smithi189.stdout:ttyS1 2024-04-01T01:14:54.927 DEBUG:teuthology.parallel:result is None 2024-04-01T01:14:54.927 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-01T01:14:54.935 INFO:teuthology.task.internal:Creating test directory... 2024-04-01T01:14:54.935 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-01T01:14:54.938 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-01T01:14:54.941 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-01T01:14:54.956 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-01T01:14:54.994 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-01T01:14:55.069 INFO:teuthology.task.internal:Creating archive directory... 2024-04-01T01:14:55.070 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-01T01:14:55.072 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-01T01:14:55.074 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-01T01:14:55.319 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-01T01:14:55.326 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-01T01:14:55.326 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-01T01:14:55.329 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-01T01:14:55.332 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-01T01:14:55.363 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T01:14:55.366 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T01:14:55.371 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T01:14:55.377 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T01:14:55.379 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T01:14:55.385 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-01T01:14:55.387 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-01T01:14:55.394 INFO:teuthology.task.internal:Configuring sudo... 2024-04-01T01:14:55.394 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-01T01:14:55.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-01T01:14:55.430 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-01T01:14:55.458 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-01T01:14:55.469 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-01T01:14:55.469 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-01T01:14:55.491 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-01T01:14:55.501 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-01T01:14:55.517 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-01T01:14:55.590 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-01T01:14:55.657 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-01T01:14:55.698 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-01T01:14:55.741 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:14:55.741 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-01T01:14:55.809 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-01T01:14:55.851 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-01T01:14:55.901 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-01T01:14:55.988 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-01T01:14:56.028 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-01T01:14:56.028 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-01T01:14:56.096 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-01T01:14:56.139 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-01T01:14:56.190 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-01T01:14:56.272 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-01T01:14:56.314 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-01T01:14:56.314 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-01T01:14:56.388 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2024-04-01T01:14:56.390 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2024-04-01T01:14:56.392 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-04-01T01:14:56.439 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-01T01:14:56.444 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-01T01:14:56.473 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-01T01:14:56.869 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-01T01:14:56.877 INFO:teuthology.task.internal:Starting timer... 2024-04-01T01:14:56.877 INFO:teuthology.run_tasks:Running task pcp... 2024-04-01T01:14:56.942 INFO:teuthology.run_tasks:Running task selinux... 2024-04-01T01:14:56.950 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-01T01:14:56.951 DEBUG:teuthology.orchestra.run.smithi070:> sudo service auditd rotate 2024-04-01T01:14:57.009 INFO:teuthology.orchestra.run.smithi070.stdout:Rotating logs: 2024-04-01T01:14:57.011 DEBUG:teuthology.orchestra.run.smithi081:> sudo service auditd rotate 2024-04-01T01:14:57.057 INFO:teuthology.orchestra.run.smithi081.stdout:Rotating logs: 2024-04-01T01:14:57.059 DEBUG:teuthology.orchestra.run.smithi189:> sudo service auditd rotate 2024-04-01T01:14:57.135 INFO:teuthology.orchestra.run.smithi189.stdout:Rotating logs: 2024-04-01T01:14:57.140 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-01T01:14:57.141 DEBUG:teuthology.orchestra.run.smithi070:> /usr/sbin/getenforce 2024-04-01T01:14:57.165 INFO:teuthology.orchestra.run.smithi070.stdout:Permissive 2024-04-01T01:14:57.165 DEBUG:teuthology.orchestra.run.smithi081:> /usr/sbin/getenforce 2024-04-01T01:14:57.209 INFO:teuthology.orchestra.run.smithi081.stdout:Permissive 2024-04-01T01:14:57.210 DEBUG:teuthology.orchestra.run.smithi189:> /usr/sbin/getenforce 2024-04-01T01:14:57.241 INFO:teuthology.orchestra.run.smithi189.stdout:Permissive 2024-04-01T01:14:57.242 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi070.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi081.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi189.front.sepia.ceph.com': 'permissive'} 2024-04-01T01:14:57.242 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-01T01:14:57.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T01:14:57.270 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-01T01:14:57.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T01:14:57.300 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-01T01:14:57.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T01:14:57.350 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-01T01:14:57.350 DEBUG:teuthology.orchestra.run.smithi070:> sudo /usr/sbin/setenforce permissive 2024-04-01T01:14:57.384 DEBUG:teuthology.orchestra.run.smithi081:> sudo /usr/sbin/setenforce permissive 2024-04-01T01:14:57.428 DEBUG:teuthology.orchestra.run.smithi189:> sudo /usr/sbin/setenforce permissive 2024-04-01T01:14:57.585 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-01T01:14:57.595 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-01T01:14:57.613 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-01T01:14:57.615 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi070.front.sepia.ceph.com,smithi081.front.sepia.ceph.com,smithi189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-01T01:19:17.374 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi070.front.sepia.ceph.com'), Remote(name='ubuntu@smithi081.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com')] 2024-04-01T01:19:17.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-01T01:19:17.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T01:19:17.465 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-04-01T01:19:17.544 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-01T01:19:17.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-01T01:19:17.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T01:19:17.639 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-04-01T01:19:17.718 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-01T01:19:17.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-01T01:19:17.719 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-01T01:19:17.812 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-04-01T01:19:17.897 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-01T01:19:17.898 INFO:teuthology.run_tasks:Running task clock... 2024-04-01T01:19:17.908 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-01T01:19:17.909 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-01T01:19:17.909 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-01T01:19:17.912 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-01T01:19:17.912 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-01T01:19:17.914 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-01T01:19:17.914 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-01T01:19:17.941 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-01T01:19:17.944 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-01T01:19:17.958 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-01T01:19:17.961 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-01T01:19:17.969 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-01T01:19:17.986 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-01T01:19:17.988 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: ntpd: command not found 2024-04-01T01:19:17.991 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: ntpd: command not found 2024-04-01T01:19:18.002 INFO:teuthology.orchestra.run.smithi070.stdout:506 Cannot talk to daemon 2024-04-01T01:19:18.005 INFO:teuthology.orchestra.run.smithi081.stdout:506 Cannot talk to daemon 2024-04-01T01:19:18.015 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: ntpd: command not found 2024-04-01T01:19:18.019 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-01T01:19:18.021 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-01T01:19:18.029 INFO:teuthology.orchestra.run.smithi189.stdout:506 Cannot talk to daemon 2024-04-01T01:19:18.035 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-01T01:19:18.037 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-01T01:19:18.045 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-01T01:19:18.062 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-01T01:19:18.083 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-04-01T01:19:18.087 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-01T01:19:18.087 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-04-01T01:19:18.087 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T01:19:18.087 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T01:19:18.087 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T01:19:18.087 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T01:19:18.089 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-04-01T01:19:18.092 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-01T01:19:18.092 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-04-01T01:19:18.092 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T01:19:18.092 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T01:19:18.093 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T01:19:18.093 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T01:19:18.116 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-04-01T01:19:18.119 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-01T01:19:18.119 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-04-01T01:19:18.119 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T01:19:18.119 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T01:19:18.120 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T01:19:18.120 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T01:19:18.120 INFO:teuthology.run_tasks:Running task install... 2024-04-01T01:19:18.129 DEBUG:teuthology.task.install:project ceph 2024-04-01T01:19:18.129 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583'}} 2024-04-01T01:19:18.129 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583'} 2024-04-01T01:19:18.129 INFO:teuthology.task.install:Using flavor: default 2024-04-01T01:19:18.135 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-01T01:19:18.135 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-01T01:19:18.136 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583', 'tag': None, 'wait_for_package': False} 2024-04-01T01:19:18.136 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-04-01T01:19:18.139 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583', 'tag': None, 'wait_for_package': False} 2024-04-01T01:19:18.139 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-04-01T01:19:18.140 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583', 'tag': None, 'wait_for_package': False} 2024-04-01T01:19:18.140 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-04-01T01:19:18.364 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/ 2024-04-01T01:19:18.365 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1977.g384f31f3 2024-04-01T01:19:18.368 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/ 2024-04-01T01:19:18.369 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1977.g384f31f3 2024-04-01T01:19:18.371 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/ 2024-04-01T01:19:18.371 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1977.g384f31f3 2024-04-01T01:19:18.485 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-01T01:19:18.485 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-01T01:19:18.485 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-01T01:19:18.489 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-01T01:19:18.489 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:19:18.489 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-01T01:19:18.517 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-01T01:19:18.518 DEBUG:teuthology.orchestra.run.smithi189:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/384f31f3d3aee7f8ba3f60846b8844c049165583/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-01T01:19:18.519 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-01T01:19:18.520 DEBUG:teuthology.orchestra.run.smithi070:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/384f31f3d3aee7f8ba3f60846b8844c049165583/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-01T01:19:18.541 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-01T01:19:18.541 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-01T01:19:18.541 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-01T01:19:18.569 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-01T01:19:18.570 DEBUG:teuthology.orchestra.run.smithi081:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/384f31f3d3aee7f8ba3f60846b8844c049165583/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-01T01:19:18.585 DEBUG:teuthology.orchestra.run.smithi189:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-01T01:19:18.588 DEBUG:teuthology.orchestra.run.smithi070:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-01T01:19:18.638 DEBUG:teuthology.orchestra.run.smithi081:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-01T01:19:18.665 DEBUG:teuthology.orchestra.run.smithi189:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-01T01:19:18.667 DEBUG:teuthology.orchestra.run.smithi070:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-01T01:19:18.718 DEBUG:teuthology.orchestra.run.smithi081:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-01T01:19:18.732 INFO:teuthology.orchestra.run.smithi189.stdout:check_obsoletes = 1 2024-04-01T01:19:18.733 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum clean all 2024-04-01T01:19:18.734 INFO:teuthology.orchestra.run.smithi070.stdout:check_obsoletes = 1 2024-04-01T01:19:18.735 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum clean all 2024-04-01T01:19:18.784 INFO:teuthology.orchestra.run.smithi081.stdout:check_obsoletes = 1 2024-04-01T01:19:18.786 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum clean all 2024-04-01T01:19:18.977 INFO:teuthology.orchestra.run.smithi189.stdout:74 files removed 2024-04-01T01:19:18.982 INFO:teuthology.orchestra.run.smithi070.stdout:74 files removed 2024-04-01T01:19:19.004 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-01T01:19:19.009 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-01T01:19:19.024 INFO:teuthology.orchestra.run.smithi081.stdout:74 files removed 2024-04-01T01:19:19.050 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-01T01:19:19.697 INFO:teuthology.orchestra.run.smithi189.stdout:ceph packages for x86_64 214 kB/s | 83 kB 00:00 2024-04-01T01:19:19.781 INFO:teuthology.orchestra.run.smithi081.stdout:ceph packages for x86_64 193 kB/s | 83 kB 00:00 2024-04-01T01:19:19.781 INFO:teuthology.orchestra.run.smithi070.stdout:ceph packages for x86_64 176 kB/s | 83 kB 00:00 2024-04-01T01:19:20.116 INFO:teuthology.orchestra.run.smithi189.stdout:ceph noarch packages 31 kB/s | 11 kB 00:00 2024-04-01T01:19:20.148 INFO:teuthology.orchestra.run.smithi081.stdout:ceph noarch packages 36 kB/s | 11 kB 00:00 2024-04-01T01:19:20.182 INFO:teuthology.orchestra.run.smithi070.stdout:ceph noarch packages 33 kB/s | 11 kB 00:00 2024-04-01T01:19:20.532 INFO:teuthology.orchestra.run.smithi081.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-04-01T01:19:20.558 INFO:teuthology.orchestra.run.smithi189.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-04-01T01:19:20.599 INFO:teuthology.orchestra.run.smithi070.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-04-01T01:19:21.019 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-04-01T01:19:21.259 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-04-01T01:19:21.335 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - AppStream 8.7 MB/s | 6.7 MB 00:00 2024-04-01T01:19:22.827 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-04-01T01:19:22.910 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-04-01T01:19:22.911 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - BaseOS 2.6 MB/s | 2.0 MB 00:00 2024-04-01T01:19:24.712 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.0 MB 00:01 2024-04-01T01:19:24.875 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.0 MB 00:01 2024-04-01T01:19:27.610 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.0 MB 00:04 2024-04-01T01:19:29.036 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 6.2 MB/s | 19 MB 00:03 2024-04-01T01:19:34.195 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.1 MB 00:01 2024-04-01T01:19:35.562 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 16 kB 00:00 2024-04-01T01:19:37.563 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-01T01:19:37.993 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 2.0 MB/s | 19 MB 00:09 2024-04-01T01:19:42.513 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-04-01T01:19:42.618 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 5.3 MB/s | 6.1 MB 00:01 2024-04-01T01:19:44.345 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-04-01T01:19:45.065 INFO:teuthology.orchestra.run.smithi070.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-01T01:19:45.066 INFO:teuthology.orchestra.run.smithi070.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-01T01:19:45.080 INFO:teuthology.orchestra.run.smithi070.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-01T01:19:45.080 INFO:teuthology.orchestra.run.smithi070.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-01T01:19:45.082 INFO:teuthology.orchestra.run.smithi070.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-01T01:19:45.083 INFO:teuthology.orchestra.run.smithi070.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-01T01:19:45.085 INFO:teuthology.orchestra.run.smithi070.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-01T01:19:45.088 INFO:teuthology.orchestra.run.smithi070.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-01T01:19:45.088 INFO:teuthology.orchestra.run.smithi070.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-01T01:19:45.168 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-01T01:19:45.181 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================ 2024-04-01T01:19:45.181 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-04-01T01:19:45.181 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================ 2024-04-01T01:19:45.181 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-01T01:19:45.181 INFO:teuthology.orchestra.run.smithi070.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-01T01:19:45.181 INFO:teuthology.orchestra.run.smithi070.stdout: ceph x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 6.6 k 2024-04-01T01:19:45.181 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 5.3 M 2024-04-01T01:19:45.181 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 889 k 2024-04-01T01:19:45.181 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 145 k 2024-04-01T01:19:45.181 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 1.5 M 2024-04-01T01:19:45.181 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 139 k 2024-04-01T01:19:45.182 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 3.5 M 2024-04-01T01:19:45.182 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 7.4 M 2024-04-01T01:19:45.182 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 49 k 2024-04-01T01:19:45.182 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 9.8 M 2024-04-01T01:19:45.182 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 47 M 2024-04-01T01:19:45.182 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 264 k 2024-04-01T01:19:45.182 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 748 k 2024-04-01T01:19:45.182 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-mirror x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 221 k 2024-04-01T01:19:45.182 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-shell x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 27 k 2024-04-01T01:19:45.182 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-top noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 18 k 2024-04-01T01:19:45.182 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-01T01:19:45.183 INFO:teuthology.orchestra.run.smithi070.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-01T01:19:45.183 INFO:teuthology.orchestra.run.smithi070.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-01T01:19:45.183 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 32 k 2024-04-01T01:19:45.183 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 746 k 2024-04-01T01:19:45.183 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 127 k 2024-04-01T01:19:45.183 INFO:teuthology.orchestra.run.smithi070.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-01T01:19:45.183 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-01T01:19:45.183 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-01T01:19:45.183 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-01T01:19:45.183 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-01T01:19:45.184 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-01T01:19:45.184 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 161 k 2024-04-01T01:19:45.184 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 320 k 2024-04-01T01:19:45.184 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 298 k 2024-04-01T01:19:45.184 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 99 k 2024-04-01T01:19:45.184 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 85 k 2024-04-01T01:19:45.184 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.1 M 2024-04-01T01:19:45.184 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 171 k 2024-04-01T01:19:45.184 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-01T01:19:45.184 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrading: 2024-04-01T01:19:45.184 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-04-01T01:19:45.184 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-04-01T01:19:45.184 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-04-01T01:19:45.184 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-04-01T01:19:45.185 INFO:teuthology.orchestra.run.smithi070.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-01T01:19:45.185 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-01T01:19:45.185 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-01T01:19:45.185 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-01T01:19:45.185 INFO:teuthology.orchestra.run.smithi070.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-01T01:19:45.185 INFO:teuthology.orchestra.run.smithi070.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-01T01:19:45.185 INFO:teuthology.orchestra.run.smithi070.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-01T01:19:45.185 INFO:teuthology.orchestra.run.smithi070.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-01T01:19:45.185 INFO:teuthology.orchestra.run.smithi070.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-01T01:19:45.185 INFO:teuthology.orchestra.run.smithi070.stdout: librados2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.4 M 2024-04-01T01:19:45.185 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.1 M 2024-04-01T01:19:45.185 INFO:teuthology.orchestra.run.smithi070.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-01T01:19:45.185 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-01T01:19:45.186 INFO:teuthology.orchestra.run.smithi070.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-01T01:19:45.186 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-01T01:19:45.186 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-01T01:19:45.186 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-01T01:19:45.186 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-01T01:19:45.186 INFO:teuthology.orchestra.run.smithi070.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-01T01:19:45.186 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-01T01:19:45.186 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-04-01T01:19:45.187 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-01T01:19:45.187 INFO:teuthology.orchestra.run.smithi070.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-01T01:19:45.187 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-01T01:19:45.187 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 21 M 2024-04-01T01:19:45.187 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 31 k 2024-04-01T01:19:45.187 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 2.4 M 2024-04-01T01:19:45.187 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 248 k 2024-04-01T01:19:45.187 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 4.7 M 2024-04-01T01:19:45.187 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 17 M 2024-04-01T01:19:45.187 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 15 k 2024-04-01T01:19:45.187 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 25 k 2024-04-01T01:19:45.188 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-01T01:19:45.188 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-01T01:19:45.188 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-01T01:19:45.188 INFO:teuthology.orchestra.run.smithi070.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-01T01:19:45.188 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-01T01:19:45.188 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-01T01:19:45.188 INFO:teuthology.orchestra.run.smithi070.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-01T01:19:45.188 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-01T01:19:45.188 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-01T01:19:45.188 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 163 k 2024-04-01T01:19:45.188 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-01T01:19:45.188 INFO:teuthology.orchestra.run.smithi070.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-01T01:19:45.189 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-01T01:19:45.189 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-01T01:19:45.189 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-01T01:19:45.189 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 501 k 2024-04-01T01:19:45.189 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-01T01:19:45.189 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 5.1 M 2024-04-01T01:19:45.189 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-01T01:19:45.189 INFO:teuthology.orchestra.run.smithi070.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-01T01:19:45.189 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-01T01:19:45.189 INFO:teuthology.orchestra.run.smithi070.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-01T01:19:45.189 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-01T01:19:45.189 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-01T01:19:45.190 INFO:teuthology.orchestra.run.smithi070.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-01T01:19:45.190 INFO:teuthology.orchestra.run.smithi070.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-01T01:19:45.190 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-01T01:19:45.190 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-01T01:19:45.190 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-01T01:19:45.190 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-01T01:19:45.190 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-01T01:19:45.190 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-01T01:19:45.190 INFO:teuthology.orchestra.run.smithi070.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-01T01:19:45.190 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-01T01:19:45.190 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-01T01:19:45.190 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-01T01:19:45.190 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 45 k 2024-04-01T01:19:45.191 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 131 k 2024-04-01T01:19:45.191 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-01T01:19:45.191 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-01T01:19:45.191 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-01T01:19:45.191 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-01T01:19:45.191 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-01T01:19:45.191 INFO:teuthology.orchestra.run.smithi070.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-01T01:19:45.191 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-01T01:19:45.191 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-01T01:19:45.191 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-01T01:19:45.191 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-01T01:19:45.191 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-01T01:19:45.191 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-01T01:19:45.191 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-01T01:19:45.192 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-01T01:19:45.192 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-01T01:19:45.192 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-01T01:19:45.192 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-01T01:19:45.192 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-01T01:19:45.192 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-01T01:19:45.192 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-01T01:19:45.192 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-01T01:19:45.192 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-01T01:19:45.192 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-01T01:19:45.192 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-01T01:19:45.192 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-01T01:19:45.192 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-01T01:19:45.192 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-01T01:19:45.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-01T01:19:45.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-01T01:19:45.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-01T01:19:45.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-01T01:19:45.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-01T01:19:45.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-01T01:19:45.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-01T01:19:45.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-01T01:19:45.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-01T01:19:45.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-01T01:19:45.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-01T01:19:45.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-01T01:19:45.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-01T01:19:45.194 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-01T01:19:45.194 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-01T01:19:45.194 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-01T01:19:45.194 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-01T01:19:45.194 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-01T01:19:45.194 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-01T01:19:45.194 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-01T01:19:45.194 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-01T01:19:45.194 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-01T01:19:45.194 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-01T01:19:45.194 INFO:teuthology.orchestra.run.smithi070.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-01T01:19:45.194 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-01T01:19:45.194 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-01T01:19:45.195 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-01T01:19:45.195 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-01T01:19:45.195 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-01T01:19:45.195 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-01T01:19:45.195 INFO:teuthology.orchestra.run.smithi070.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-01T01:19:45.195 INFO:teuthology.orchestra.run.smithi070.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-01T01:19:45.195 INFO:teuthology.orchestra.run.smithi070.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-01T01:19:45.195 INFO:teuthology.orchestra.run.smithi070.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-01T01:19:45.195 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-01T01:19:45.195 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-01T01:19:45.195 INFO:teuthology.orchestra.run.smithi070.stdout:Installing weak dependencies: 2024-04-01T01:19:45.195 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-01T01:19:45.195 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:19:45.196 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-01T01:19:45.196 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================ 2024-04-01T01:19:45.196 INFO:teuthology.orchestra.run.smithi070.stdout:Install 150 Packages 2024-04-01T01:19:45.196 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrade 24 Packages 2024-04-01T01:19:45.196 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:19:45.197 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-04-01T01:19:45.198 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 226 M 2024-04-01T01:19:45.198 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-01T01:19:46.346 INFO:teuthology.orchestra.run.smithi070.stdout:(1/174): ceph-19.0.0-1977.g384f31f3.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-04-01T01:19:47.046 INFO:teuthology.orchestra.run.smithi070.stdout:(2/174): ceph-fuse-19.0.0-1977.g384f31f3.el9.x8 1.2 MB/s | 889 kB 00:00 2024-04-01T01:19:47.147 INFO:teuthology.orchestra.run.smithi070.stdout:(3/174): ceph-immutable-object-cache-19.0.0-197 1.4 MB/s | 145 kB 00:00 2024-04-01T01:19:48.189 INFO:teuthology.orchestra.run.smithi070.stdout:(4/174): ceph-mds-19.0.0-1977.g384f31f3.el9.x86 2.3 MB/s | 2.4 MB 00:01 2024-04-01T01:19:48.980 INFO:teuthology.orchestra.run.smithi070.stdout:(5/174): ceph-mgr-19.0.0-1977.g384f31f3.el9.x86 1.9 MB/s | 1.5 MB 00:00 2024-04-01T01:19:50.129 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-01T01:19:50.716 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 776 kB/s | 19 MB 00:24 2024-04-01T01:19:50.939 INFO:teuthology.orchestra.run.smithi070.stdout:(6/174): ceph-mon-19.0.0-1977.g384f31f3.el9.x86 2.4 MB/s | 4.7 MB 00:01 2024-04-01T01:19:52.147 INFO:teuthology.orchestra.run.smithi070.stdout:(7/174): ceph-base-19.0.0-1977.g384f31f3.el9.x8 876 kB/s | 5.3 MB 00:06 2024-04-01T01:19:52.663 INFO:teuthology.orchestra.run.smithi189.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-01T01:19:52.663 INFO:teuthology.orchestra.run.smithi189.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-01T01:19:52.677 INFO:teuthology.orchestra.run.smithi189.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-01T01:19:52.678 INFO:teuthology.orchestra.run.smithi189.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-01T01:19:52.679 INFO:teuthology.orchestra.run.smithi189.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-01T01:19:52.680 INFO:teuthology.orchestra.run.smithi189.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-01T01:19:52.682 INFO:teuthology.orchestra.run.smithi189.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-01T01:19:52.685 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-01T01:19:52.686 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-01T01:19:52.766 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-01T01:19:52.778 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================================ 2024-04-01T01:19:52.778 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-01T01:19:52.778 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================================ 2024-04-01T01:19:52.778 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-01T01:19:52.778 INFO:teuthology.orchestra.run.smithi189.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-01T01:19:52.778 INFO:teuthology.orchestra.run.smithi189.stdout: ceph x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 6.6 k 2024-04-01T01:19:52.779 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 5.3 M 2024-04-01T01:19:52.779 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 889 k 2024-04-01T01:19:52.779 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 145 k 2024-04-01T01:19:52.779 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 1.5 M 2024-04-01T01:19:52.779 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 139 k 2024-04-01T01:19:52.779 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 3.5 M 2024-04-01T01:19:52.779 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 7.4 M 2024-04-01T01:19:52.779 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 49 k 2024-04-01T01:19:52.779 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 9.8 M 2024-04-01T01:19:52.779 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 47 M 2024-04-01T01:19:52.779 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 264 k 2024-04-01T01:19:52.779 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 748 k 2024-04-01T01:19:52.780 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 221 k 2024-04-01T01:19:52.780 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-shell x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 27 k 2024-04-01T01:19:52.780 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 18 k 2024-04-01T01:19:52.780 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-01T01:19:52.780 INFO:teuthology.orchestra.run.smithi189.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-01T01:19:52.780 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-01T01:19:52.780 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 32 k 2024-04-01T01:19:52.780 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 746 k 2024-04-01T01:19:52.780 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 127 k 2024-04-01T01:19:52.780 INFO:teuthology.orchestra.run.smithi189.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-01T01:19:52.780 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-01T01:19:52.780 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-01T01:19:52.780 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-01T01:19:52.780 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-01T01:19:52.781 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-01T01:19:52.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 161 k 2024-04-01T01:19:52.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 320 k 2024-04-01T01:19:52.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 298 k 2024-04-01T01:19:52.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 99 k 2024-04-01T01:19:52.781 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 85 k 2024-04-01T01:19:52.781 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.1 M 2024-04-01T01:19:52.781 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 171 k 2024-04-01T01:19:52.781 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-01T01:19:52.781 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrading: 2024-04-01T01:19:52.781 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-04-01T01:19:52.781 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-04-01T01:19:52.781 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-04-01T01:19:52.781 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-04-01T01:19:52.781 INFO:teuthology.orchestra.run.smithi189.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-01T01:19:52.782 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-01T01:19:52.782 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-01T01:19:52.782 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-01T01:19:52.782 INFO:teuthology.orchestra.run.smithi189.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-01T01:19:52.783 INFO:teuthology.orchestra.run.smithi189.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-01T01:19:52.783 INFO:teuthology.orchestra.run.smithi189.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-01T01:19:52.783 INFO:teuthology.orchestra.run.smithi189.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-01T01:19:52.783 INFO:teuthology.orchestra.run.smithi189.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-01T01:19:52.783 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.4 M 2024-04-01T01:19:52.783 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.1 M 2024-04-01T01:19:52.783 INFO:teuthology.orchestra.run.smithi189.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-01T01:19:52.783 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-01T01:19:52.783 INFO:teuthology.orchestra.run.smithi189.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-01T01:19:52.783 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-01T01:19:52.783 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-01T01:19:52.783 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-01T01:19:52.784 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-01T01:19:52.784 INFO:teuthology.orchestra.run.smithi189.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-01T01:19:52.784 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-01T01:19:52.784 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-01T01:19:52.784 INFO:teuthology.orchestra.run.smithi189.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-01T01:19:52.784 INFO:teuthology.orchestra.run.smithi189.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-01T01:19:52.784 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-01T01:19:52.784 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 21 M 2024-04-01T01:19:52.784 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 31 k 2024-04-01T01:19:52.784 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 2.4 M 2024-04-01T01:19:52.784 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 248 k 2024-04-01T01:19:52.784 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 4.7 M 2024-04-01T01:19:52.784 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 17 M 2024-04-01T01:19:52.785 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 15 k 2024-04-01T01:19:52.785 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 25 k 2024-04-01T01:19:52.785 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-01T01:19:52.785 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-01T01:19:52.785 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-01T01:19:52.785 INFO:teuthology.orchestra.run.smithi189.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-01T01:19:52.785 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-01T01:19:52.785 INFO:teuthology.orchestra.run.smithi189.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-01T01:19:52.785 INFO:teuthology.orchestra.run.smithi189.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-01T01:19:52.785 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-01T01:19:52.785 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-01T01:19:52.785 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 163 k 2024-04-01T01:19:52.785 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-01T01:19:52.785 INFO:teuthology.orchestra.run.smithi189.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-01T01:19:52.786 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-01T01:19:52.786 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-01T01:19:52.786 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-01T01:19:52.786 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 501 k 2024-04-01T01:19:52.786 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-01T01:19:52.786 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 5.1 M 2024-04-01T01:19:52.786 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-01T01:19:52.786 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-01T01:19:52.786 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-01T01:19:52.786 INFO:teuthology.orchestra.run.smithi189.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-01T01:19:52.786 INFO:teuthology.orchestra.run.smithi189.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-01T01:19:52.786 INFO:teuthology.orchestra.run.smithi189.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-01T01:19:52.786 INFO:teuthology.orchestra.run.smithi189.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-01T01:19:52.787 INFO:teuthology.orchestra.run.smithi189.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-01T01:19:52.787 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-01T01:19:52.787 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-01T01:19:52.787 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-01T01:19:52.787 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-01T01:19:52.787 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-01T01:19:52.787 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-01T01:19:52.787 INFO:teuthology.orchestra.run.smithi189.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-01T01:19:52.787 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-01T01:19:52.787 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-01T01:19:52.787 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-01T01:19:52.787 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 45 k 2024-04-01T01:19:52.787 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 131 k 2024-04-01T01:19:52.787 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-01T01:19:52.787 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-01T01:19:52.788 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-01T01:19:52.788 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-01T01:19:52.788 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-01T01:19:52.788 INFO:teuthology.orchestra.run.smithi189.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-01T01:19:52.788 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-01T01:19:52.788 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-01T01:19:52.788 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-01T01:19:52.788 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-01T01:19:52.788 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-01T01:19:52.788 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-01T01:19:52.788 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-01T01:19:52.788 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-01T01:19:52.788 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-01T01:19:52.789 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-01T01:19:52.789 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-01T01:19:52.789 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-01T01:19:52.789 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-01T01:19:52.789 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-01T01:19:52.789 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-01T01:19:52.789 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-01T01:19:52.789 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-01T01:19:52.789 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-01T01:19:52.789 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-01T01:19:52.789 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-01T01:19:52.789 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-01T01:19:52.789 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-01T01:19:52.789 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-01T01:19:52.789 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-01T01:19:52.790 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-01T01:19:52.790 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-01T01:19:52.790 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-01T01:19:52.790 INFO:teuthology.orchestra.run.smithi189.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-01T01:19:52.790 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-01T01:19:52.790 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-01T01:19:52.790 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-01T01:19:52.791 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-01T01:19:52.791 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-01T01:19:52.791 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-01T01:19:52.791 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-01T01:19:52.791 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-01T01:19:52.791 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-01T01:19:52.791 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-01T01:19:52.791 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-01T01:19:52.791 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-01T01:19:52.791 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-01T01:19:52.791 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-01T01:19:52.791 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-01T01:19:52.791 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-01T01:19:52.791 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-01T01:19:52.792 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-01T01:19:52.792 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-01T01:19:52.792 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-01T01:19:52.792 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-01T01:19:52.792 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-01T01:19:52.792 INFO:teuthology.orchestra.run.smithi189.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-01T01:19:52.792 INFO:teuthology.orchestra.run.smithi189.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-01T01:19:52.792 INFO:teuthology.orchestra.run.smithi189.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-01T01:19:52.792 INFO:teuthology.orchestra.run.smithi189.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-01T01:19:52.792 INFO:teuthology.orchestra.run.smithi189.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-01T01:19:52.792 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-01T01:19:52.792 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-01T01:19:52.792 INFO:teuthology.orchestra.run.smithi189.stdout:Installing weak dependencies: 2024-04-01T01:19:52.792 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-01T01:19:52.793 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:19:52.793 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-01T01:19:52.793 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================================ 2024-04-01T01:19:52.793 INFO:teuthology.orchestra.run.smithi189.stdout:Install 150 Packages 2024-04-01T01:19:52.793 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrade 24 Packages 2024-04-01T01:19:52.793 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:19:52.793 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 226 M 2024-04-01T01:19:52.794 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-01T01:19:53.405 INFO:teuthology.orchestra.run.smithi070.stdout:(8/174): ceph-common-19.0.0-1977.g384f31f3.el9. 2.7 MB/s | 21 MB 00:07 2024-04-01T01:19:53.464 INFO:teuthology.orchestra.run.smithi070.stdout:(9/174): ceph-selinux-19.0.0-1977.g384f31f3.el9 433 kB/s | 25 kB 00:00 2024-04-01T01:19:54.130 INFO:teuthology.orchestra.run.smithi189.stdout:(1/174): ceph-19.0.0-1977.g384f31f3.el9.x86_64. 10 kB/s | 6.6 kB 00:00 2024-04-01T01:19:54.855 INFO:teuthology.orchestra.run.smithi189.stdout:(2/174): ceph-fuse-19.0.0-1977.g384f31f3.el9.x8 1.2 MB/s | 889 kB 00:00 2024-04-01T01:19:54.972 INFO:teuthology.orchestra.run.smithi189.stdout:(3/174): ceph-immutable-object-cache-19.0.0-197 1.2 MB/s | 145 kB 00:00 2024-04-01T01:19:55.423 INFO:teuthology.orchestra.run.smithi189.stdout:(4/174): ceph-base-19.0.0-1977.g384f31f3.el9.x8 2.8 MB/s | 5.3 MB 00:01 2024-04-01T01:19:55.698 INFO:teuthology.orchestra.run.smithi189.stdout:(5/174): ceph-mgr-19.0.0-1977.g384f31f3.el9.x86 5.4 MB/s | 1.5 MB 00:00 2024-04-01T01:19:56.156 INFO:teuthology.orchestra.run.smithi189.stdout:(6/174): ceph-mds-19.0.0-1977.g384f31f3.el9.x86 2.0 MB/s | 2.4 MB 00:01 2024-04-01T01:19:56.290 INFO:teuthology.orchestra.run.smithi189.stdout:(7/174): ceph-mon-19.0.0-1977.g384f31f3.el9.x86 7.9 MB/s | 4.7 MB 00:00 2024-04-01T01:19:57.340 INFO:teuthology.orchestra.run.smithi189.stdout:(8/174): ceph-radosgw-19.0.0-1977.g384f31f3.el9 9.3 MB/s | 9.8 MB 00:01 2024-04-01T01:19:57.407 INFO:teuthology.orchestra.run.smithi189.stdout:(9/174): ceph-selinux-19.0.0-1977.g384f31f3.el9 378 kB/s | 25 kB 00:00 2024-04-01T01:19:58.013 INFO:teuthology.orchestra.run.smithi070.stdout:(10/174): ceph-radosgw-19.0.0-1977.g384f31f3.el 1.7 MB/s | 9.8 MB 00:05 2024-04-01T01:19:58.032 INFO:teuthology.orchestra.run.smithi189.stdout:(10/174): ceph-common-19.0.0-1977.g384f31f3.el9 4.5 MB/s | 21 MB 00:04 2024-04-01T01:19:58.124 INFO:teuthology.orchestra.run.smithi189.stdout:(11/174): cephfs-mirror-19.0.0-1977.g384f31f3.e 2.4 MB/s | 221 kB 00:00 2024-04-01T01:19:58.147 INFO:teuthology.orchestra.run.smithi070.stdout:(11/174): cephfs-mirror-19.0.0-1977.g384f31f3.e 1.6 MB/s | 221 kB 00:00 2024-04-01T01:19:58.199 INFO:teuthology.orchestra.run.smithi189.stdout:(12/174): cephfs-shell-19.0.0-1977.g384f31f3.el 361 kB/s | 27 kB 00:00 2024-04-01T01:19:58.222 INFO:teuthology.orchestra.run.smithi070.stdout:(12/174): cephfs-shell-19.0.0-1977.g384f31f3.el 361 kB/s | 27 kB 00:00 2024-04-01T01:19:58.275 INFO:teuthology.orchestra.run.smithi189.stdout:(13/174): libcephfs-devel-19.0.0-1977.g384f31f3 423 kB/s | 32 kB 00:00 2024-04-01T01:19:58.306 INFO:teuthology.orchestra.run.smithi070.stdout:(13/174): libcephfs-devel-19.0.0-1977.g384f31f3 381 kB/s | 32 kB 00:00 2024-04-01T01:19:58.383 INFO:teuthology.orchestra.run.smithi189.stdout:(14/174): libcephfs2-19.0.0-1977.g384f31f3.el9. 6.7 MB/s | 746 kB 00:00 2024-04-01T01:19:58.467 INFO:teuthology.orchestra.run.smithi189.stdout:(15/174): libcephsqlite-19.0.0-1977.g384f31f3.e 1.9 MB/s | 163 kB 00:00 2024-04-01T01:19:58.559 INFO:teuthology.orchestra.run.smithi189.stdout:(16/174): librados-devel-19.0.0-1977.g384f31f3. 1.4 MB/s | 127 kB 00:00 2024-04-01T01:19:58.597 INFO:teuthology.orchestra.run.smithi070.stdout:(14/174): libcephfs2-19.0.0-1977.g384f31f3.el9. 2.5 MB/s | 746 kB 00:00 2024-04-01T01:19:58.668 INFO:teuthology.orchestra.run.smithi189.stdout:(17/174): libradosstriper1-19.0.0-1977.g384f31f 4.5 MB/s | 501 kB 00:00 2024-04-01T01:19:58.689 INFO:teuthology.orchestra.run.smithi070.stdout:(15/174): libcephsqlite-19.0.0-1977.g384f31f3.e 1.7 MB/s | 163 kB 00:00 2024-04-01T01:19:59.093 INFO:teuthology.orchestra.run.smithi189.stdout:(18/174): librgw2-19.0.0-1977.g384f31f3.el9.x86 12 MB/s | 5.1 MB 00:00 2024-04-01T01:19:59.152 INFO:teuthology.orchestra.run.smithi189.stdout:(19/174): python3-ceph-argparse-19.0.0-1977.g38 779 kB/s | 45 kB 00:00 2024-04-01T01:19:59.227 INFO:teuthology.orchestra.run.smithi189.stdout:(20/174): python3-ceph-common-19.0.0-1977.g384f 1.7 MB/s | 131 kB 00:00 2024-04-01T01:19:59.310 INFO:teuthology.orchestra.run.smithi189.stdout:(21/174): python3-cephfs-19.0.0-1977.g384f31f3. 1.9 MB/s | 161 kB 00:00 2024-04-01T01:19:59.314 INFO:teuthology.orchestra.run.smithi070.stdout:(16/174): librados-devel-19.0.0-1977.g384f31f3. 203 kB/s | 127 kB 00:00 2024-04-01T01:19:59.394 INFO:teuthology.orchestra.run.smithi189.stdout:(22/174): python3-rados-19.0.0-1977.g384f31f3.e 3.8 MB/s | 320 kB 00:00 2024-04-01T01:19:59.486 INFO:teuthology.orchestra.run.smithi189.stdout:(23/174): python3-rbd-19.0.0-1977.g384f31f3.el9 3.2 MB/s | 298 kB 00:00 2024-04-01T01:19:59.540 INFO:teuthology.orchestra.run.smithi070.stdout:(17/174): libradosstriper1-19.0.0-1977.g384f31f 2.2 MB/s | 501 kB 00:00 2024-04-01T01:19:59.561 INFO:teuthology.orchestra.run.smithi189.stdout:(24/174): python3-rgw-19.0.0-1977.g384f31f3.el9 1.3 MB/s | 99 kB 00:00 2024-04-01T01:19:59.645 INFO:teuthology.orchestra.run.smithi189.stdout:(25/174): rbd-fuse-19.0.0-1977.g384f31f3.el9.x8 1.0 MB/s | 85 kB 00:00 2024-04-01T01:19:59.937 INFO:teuthology.orchestra.run.smithi189.stdout:(26/174): rbd-mirror-19.0.0-1977.g384f31f3.el9. 11 MB/s | 3.1 MB 00:00 2024-04-01T01:20:00.004 INFO:teuthology.orchestra.run.smithi189.stdout:(27/174): rbd-nbd-19.0.0-1977.g384f31f3.el9.x86 2.5 MB/s | 171 kB 00:00 2024-04-01T01:20:00.063 INFO:teuthology.orchestra.run.smithi189.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1977.g 539 kB/s | 31 kB 00:00 2024-04-01T01:20:00.138 INFO:teuthology.orchestra.run.smithi189.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1977.g384f31f 1.8 MB/s | 139 kB 00:00 2024-04-01T01:20:00.365 INFO:teuthology.orchestra.run.smithi070.stdout:(18/174): ceph-osd-19.0.0-1977.g384f31f3.el9.x8 1.8 MB/s | 17 MB 00:09 2024-04-01T01:20:00.448 INFO:teuthology.orchestra.run.smithi070.stdout:(19/174): python3-ceph-argparse-19.0.0-1977.g38 544 kB/s | 45 kB 00:00 2024-04-01T01:20:00.497 INFO:teuthology.orchestra.run.smithi189.stdout:(30/174): ceph-mgr-dashboard-19.0.0-1977.g384f3 9.8 MB/s | 3.5 MB 00:00 2024-04-01T01:20:00.549 INFO:teuthology.orchestra.run.smithi070.stdout:(20/174): python3-ceph-common-19.0.0-1977.g384f 1.3 MB/s | 131 kB 00:00 2024-04-01T01:20:00.707 INFO:teuthology.orchestra.run.smithi070.stdout:(21/174): python3-cephfs-19.0.0-1977.g384f31f3. 1.0 MB/s | 161 kB 00:00 2024-04-01T01:20:01.049 INFO:teuthology.orchestra.run.smithi070.stdout:(22/174): python3-rados-19.0.0-1977.g384f31f3.e 938 kB/s | 320 kB 00:00 2024-04-01T01:20:01.197 INFO:teuthology.orchestra.run.smithi189.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-04-01T01:20:01.249 INFO:teuthology.orchestra.run.smithi070.stdout:(23/174): python3-rbd-19.0.0-1977.g384f31f3.el9 1.5 MB/s | 298 kB 00:00 2024-04-01T01:20:01.282 INFO:teuthology.orchestra.run.smithi189.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1977.g38 2.9 MB/s | 248 kB 00:00 2024-04-01T01:20:01.350 INFO:teuthology.orchestra.run.smithi070.stdout:(24/174): python3-rgw-19.0.0-1977.g384f31f3.el9 995 kB/s | 99 kB 00:00 2024-04-01T01:20:01.355 INFO:teuthology.orchestra.run.smithi189.stdout:(33/174): ceph-mgr-rook-19.0.0-1977.g384f31f3.e 657 kB/s | 49 kB 00:00 2024-04-01T01:20:01.423 INFO:teuthology.orchestra.run.smithi189.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1977.g3 230 kB/s | 15 kB 00:00 2024-04-01T01:20:01.450 INFO:teuthology.orchestra.run.smithi070.stdout:(25/174): librgw2-19.0.0-1977.g384f31f3.el9.x86 2.7 MB/s | 5.1 MB 00:01 2024-04-01T01:20:01.475 INFO:teuthology.orchestra.run.smithi070.stdout:(26/174): rbd-fuse-19.0.0-1977.g384f31f3.el9.x8 681 kB/s | 85 kB 00:00 2024-04-01T01:20:01.498 INFO:teuthology.orchestra.run.smithi189.stdout:(35/174): ceph-volume-19.0.0-1977.g384f31f3.el9 3.4 MB/s | 264 kB 00:00 2024-04-01T01:20:01.615 INFO:teuthology.orchestra.run.smithi189.stdout:(36/174): cephadm-19.0.0-1977.g384f31f3.el9.noa 6.3 MB/s | 748 kB 00:00 2024-04-01T01:20:01.651 INFO:teuthology.orchestra.run.smithi070.stdout:(27/174): rbd-nbd-19.0.0-1977.g384f31f3.el9.x86 979 kB/s | 171 kB 00:00 2024-04-01T01:20:01.707 INFO:teuthology.orchestra.run.smithi189.stdout:(37/174): cephfs-top-19.0.0-1977.g384f31f3.el9. 198 kB/s | 18 kB 00:00 2024-04-01T01:20:01.726 INFO:teuthology.orchestra.run.smithi070.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1977.g 420 kB/s | 31 kB 00:00 2024-04-01T01:20:01.893 INFO:teuthology.orchestra.run.smithi070.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1977.g384f31f 837 kB/s | 139 kB 00:00 2024-04-01T01:20:02.090 INFO:teuthology.orchestra.run.smithi189.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 1.7 MB/s | 662 kB 00:00 2024-04-01T01:20:02.224 INFO:teuthology.orchestra.run.smithi189.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 6.7 MB/s | 920 kB 00:00 2024-04-01T01:20:02.266 INFO:teuthology.orchestra.run.smithi189.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-04-01T01:20:02.325 INFO:teuthology.orchestra.run.smithi189.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-04-01T01:20:02.367 INFO:teuthology.orchestra.run.smithi189.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 755 kB/s | 32 kB 00:00 2024-04-01T01:20:02.526 INFO:teuthology.orchestra.run.smithi189.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 19 MB/s | 3.0 MB 00:00 2024-04-01T01:20:02.559 INFO:teuthology.orchestra.run.smithi189.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 514 kB/s | 17 kB 00:00 2024-04-01T01:20:02.593 INFO:teuthology.orchestra.run.smithi189.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 302 kB/s | 10 kB 00:00 2024-04-01T01:20:02.635 INFO:teuthology.orchestra.run.smithi189.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-04-01T01:20:02.669 INFO:teuthology.orchestra.run.smithi189.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-04-01T01:20:02.727 INFO:teuthology.orchestra.run.smithi189.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-04-01T01:20:02.794 INFO:teuthology.orchestra.run.smithi189.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-04-01T01:20:02.836 INFO:teuthology.orchestra.run.smithi189.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-04-01T01:20:02.876 INFO:teuthology.orchestra.run.smithi070.stdout:(30/174): ceph-test-19.0.0-1977.g384f31f3.el9.x 5.0 MB/s | 47 MB 00:09 2024-04-01T01:20:02.935 INFO:teuthology.orchestra.run.smithi070.stdout:(31/174): rbd-mirror-19.0.0-1977.g384f31f3.el9. 2.1 MB/s | 3.1 MB 00:01 2024-04-01T01:20:03.092 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 707 kB/s | 6.1 MB 00:08 2024-04-01T01:20:03.101 INFO:teuthology.orchestra.run.smithi070.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1977.g38 1.5 MB/s | 248 kB 00:00 2024-04-01T01:20:03.202 INFO:teuthology.orchestra.run.smithi070.stdout:(33/174): ceph-mgr-rook-19.0.0-1977.g384f31f3.e 494 kB/s | 49 kB 00:00 2024-04-01T01:20:03.245 INFO:teuthology.orchestra.run.smithi189.stdout:(51/174): ceph-test-19.0.0-1977.g384f31f3.el9.x 8.1 MB/s | 47 MB 00:05 2024-04-01T01:20:03.270 INFO:teuthology.orchestra.run.smithi189.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 651 kB/s | 282 kB 00:00 2024-04-01T01:20:03.302 INFO:teuthology.orchestra.run.smithi070.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1977.g3 153 kB/s | 15 kB 00:00 2024-04-01T01:20:03.320 INFO:teuthology.orchestra.run.smithi189.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 187 kB 00:00 2024-04-01T01:20:03.345 INFO:teuthology.orchestra.run.smithi189.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-04-01T01:20:03.387 INFO:teuthology.orchestra.run.smithi189.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 514 kB/s | 34 kB 00:00 2024-04-01T01:20:03.677 INFO:teuthology.orchestra.run.smithi070.stdout:(35/174): ceph-mgr-diskprediction-local-19.0.0- 9.2 MB/s | 7.4 MB 00:00 2024-04-01T01:20:03.702 INFO:teuthology.orchestra.run.smithi070.stdout:(36/174): ceph-volume-19.0.0-1977.g384f31f3.el9 659 kB/s | 264 kB 00:00 2024-04-01T01:20:03.786 INFO:teuthology.orchestra.run.smithi070.stdout:(37/174): cephadm-19.0.0-1977.g384f31f3.el9.noa 6.7 MB/s | 748 kB 00:00 2024-04-01T01:20:03.811 INFO:teuthology.orchestra.run.smithi070.stdout:(38/174): cephfs-top-19.0.0-1977.g384f31f3.el9. 168 kB/s | 18 kB 00:00 2024-04-01T01:20:03.863 INFO:teuthology.orchestra.run.smithi189.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 9.2 MB/s | 4.7 MB 00:00 2024-04-01T01:20:03.896 INFO:teuthology.orchestra.run.smithi189.stdout:(57/174): postgresql-13.14-1.el9.x86_64.rpm 3.0 MB/s | 1.5 MB 00:00 2024-04-01T01:20:03.972 INFO:teuthology.orchestra.run.smithi189.stdout:(58/174): postgresql-contrib-13.14-1.el9.x86_64 7.4 MB/s | 817 kB 00:00 2024-04-01T01:20:03.997 INFO:teuthology.orchestra.run.smithi189.stdout:(59/174): postgresql-private-libs-13.14-1.el9.x 1.3 MB/s | 132 kB 00:00 2024-04-01T01:20:04.162 INFO:teuthology.orchestra.run.smithi070.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 1.7 MB/s | 662 kB 00:00 2024-04-01T01:20:04.187 INFO:teuthology.orchestra.run.smithi070.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 2.4 MB/s | 920 kB 00:00 2024-04-01T01:20:04.212 INFO:teuthology.orchestra.run.smithi070.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 106 kB 00:00 2024-04-01T01:20:04.238 INFO:teuthology.orchestra.run.smithi070.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-04-01T01:20:04.263 INFO:teuthology.orchestra.run.smithi070.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 638 kB/s | 32 kB 00:00 2024-04-01T01:20:04.272 INFO:teuthology.orchestra.run.smithi189.stdout:(60/174): postgresql-server-13.14-1.el9.x86_64. 19 MB/s | 5.7 MB 00:00 2024-04-01T01:20:04.306 INFO:teuthology.orchestra.run.smithi189.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 3.3 MB/s | 1.0 MB 00:00 2024-04-01T01:20:04.331 INFO:teuthology.orchestra.run.smithi189.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-04-01T01:20:04.346 INFO:teuthology.orchestra.run.smithi070.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-04-01T01:20:04.371 INFO:teuthology.orchestra.run.smithi070.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 159 kB/s | 17 kB 00:00 2024-04-01T01:20:04.397 INFO:teuthology.orchestra.run.smithi070.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-04-01T01:20:04.422 INFO:teuthology.orchestra.run.smithi070.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-04-01T01:20:04.447 INFO:teuthology.orchestra.run.smithi070.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 889 kB/s | 44 kB 00:00 2024-04-01T01:20:04.473 INFO:teuthology.orchestra.run.smithi070.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-04-01T01:20:04.515 INFO:teuthology.orchestra.run.smithi070.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.6 MB/s | 578 kB 00:00 2024-04-01T01:20:04.540 INFO:teuthology.orchestra.run.smithi070.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 695 kB/s | 46 kB 00:00 2024-04-01T01:20:04.565 INFO:teuthology.orchestra.run.smithi070.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-04-01T01:20:04.599 INFO:teuthology.orchestra.run.smithi070.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 187 kB 00:00 2024-04-01T01:20:04.606 INFO:teuthology.orchestra.run.smithi189.stdout:(63/174): ceph-osd-19.0.0-1977.g384f31f3.el9.x8 2.0 MB/s | 17 MB 00:08 2024-04-01T01:20:04.673 INFO:teuthology.orchestra.run.smithi189.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-04-01T01:20:04.699 INFO:teuthology.orchestra.run.smithi189.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 66 kB/s | 24 kB 00:00 2024-04-01T01:20:04.724 INFO:teuthology.orchestra.run.smithi189.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 156 kB/s | 18 kB 00:00 2024-04-01T01:20:04.749 INFO:teuthology.orchestra.run.smithi189.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-04-01T01:20:04.766 INFO:teuthology.orchestra.run.smithi070.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-04-01T01:20:04.774 INFO:teuthology.orchestra.run.smithi189.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 420 kB/s | 32 kB 00:00 2024-04-01T01:20:04.791 INFO:teuthology.orchestra.run.smithi070.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 178 kB/s | 34 kB 00:00 2024-04-01T01:20:04.792 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-04-01T01:20:04.925 INFO:teuthology.orchestra.run.smithi070.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-04-01T01:20:04.933 INFO:teuthology.orchestra.run.smithi189.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2024-04-01T01:20:04.958 INFO:teuthology.orchestra.run.smithi189.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-04-01T01:20:04.967 INFO:teuthology.orchestra.run.smithi070.stdout:(57/174): postgresql-13.14-1.el9.x86_64.rpm 8.8 MB/s | 1.5 MB 00:00 2024-04-01T01:20:04.984 INFO:teuthology.orchestra.run.smithi189.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 844 kB/s | 174 kB 00:00 2024-04-01T01:20:05.000 INFO:teuthology.orchestra.run.smithi070.stdout:(58/174): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 817 kB 00:00 2024-04-01T01:20:05.009 INFO:teuthology.orchestra.run.smithi189.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 908 kB/s | 69 kB 00:00 2024-04-01T01:20:05.026 INFO:teuthology.orchestra.run.smithi070.stdout:(59/174): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 132 kB 00:00 2024-04-01T01:20:05.034 INFO:teuthology.orchestra.run.smithi189.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 820 kB/s | 41 kB 00:00 2024-04-01T01:20:05.060 INFO:teuthology.orchestra.run.smithi189.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-04-01T01:20:05.085 INFO:teuthology.orchestra.run.smithi189.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-01T01:20:05.118 INFO:teuthology.orchestra.run.smithi189.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 814 kB/s | 47 kB 00:00 2024-04-01T01:20:05.144 INFO:teuthology.orchestra.run.smithi189.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2024-04-01T01:20:05.159 INFO:teuthology.orchestra.run.smithi070.stdout:(60/174): postgresql-server-13.14-1.el9.x86_64. 36 MB/s | 5.7 MB 00:00 2024-04-01T01:20:05.168 INFO:teuthology.orchestra.run.smithi189.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 510 kB/s | 43 kB 00:00 2024-04-01T01:20:05.193 INFO:teuthology.orchestra.run.smithi070.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.1 MB/s | 1.0 MB 00:00 2024-04-01T01:20:05.218 INFO:teuthology.orchestra.run.smithi070.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-04-01T01:20:05.377 INFO:teuthology.orchestra.run.smithi070.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 32 MB/s | 5.8 MB 00:00 2024-04-01T01:20:05.393 INFO:teuthology.orchestra.run.smithi070.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 140 kB/s | 24 kB 00:00 2024-04-01T01:20:05.410 INFO:teuthology.orchestra.run.smithi070.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 544 kB/s | 18 kB 00:00 2024-04-01T01:20:05.411 INFO:teuthology.orchestra.run.smithi189.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 161 kB/s | 39 kB 00:00 2024-04-01T01:20:05.436 INFO:teuthology.orchestra.run.smithi189.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 155 kB/s | 41 kB 00:00 2024-04-01T01:20:05.452 INFO:teuthology.orchestra.run.smithi070.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-04-01T01:20:05.486 INFO:teuthology.orchestra.run.smithi070.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 418 kB/s | 32 kB 00:00 2024-04-01T01:20:05.553 INFO:teuthology.orchestra.run.smithi070.stdout:(68/174): ceph-mgr-dashboard-19.0.0-1977.g384f3 982 kB/s | 3.5 MB 00:03 2024-04-01T01:20:05.603 INFO:teuthology.orchestra.run.smithi189.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-04-01T01:20:05.628 INFO:teuthology.orchestra.run.smithi189.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.0 MB/s | 220 kB 00:00 2024-04-01T01:20:05.644 INFO:teuthology.orchestra.run.smithi070.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-04-01T01:20:05.653 INFO:teuthology.orchestra.run.smithi189.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 162 kB/s | 35 kB 00:00 2024-04-01T01:20:05.670 INFO:teuthology.orchestra.run.smithi070.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-04-01T01:20:05.678 INFO:teuthology.orchestra.run.smithi189.stdout:(84/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 619 kB/s | 46 kB 00:00 2024-04-01T01:20:05.696 INFO:teuthology.orchestra.run.smithi070.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-04-01T01:20:05.704 INFO:teuthology.orchestra.run.smithi189.stdout:(85/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-04-01T01:20:05.721 INFO:teuthology.orchestra.run.smithi070.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.4 MB/s | 69 kB 00:00 2024-04-01T01:20:05.729 INFO:teuthology.orchestra.run.smithi189.stdout:(86/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-04-01T01:20:05.746 INFO:teuthology.orchestra.run.smithi070.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 551 kB/s | 41 kB 00:00 2024-04-01T01:20:05.762 INFO:teuthology.orchestra.run.smithi189.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 749 kB/s | 63 kB 00:00 2024-04-01T01:20:05.771 INFO:teuthology.orchestra.run.smithi070.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-04-01T01:20:05.788 INFO:teuthology.orchestra.run.smithi189.stdout:(88/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 529 kB/s | 44 kB 00:00 2024-04-01T01:20:05.796 INFO:teuthology.orchestra.run.smithi070.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-04-01T01:20:05.813 INFO:teuthology.orchestra.run.smithi189.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-04-01T01:20:05.821 INFO:teuthology.orchestra.run.smithi070.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.8 MB/s | 211 kB 00:00 2024-04-01T01:20:05.838 INFO:teuthology.orchestra.run.smithi189.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-04-01T01:20:05.847 INFO:teuthology.orchestra.run.smithi070.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 633 kB/s | 47 kB 00:00 2024-04-01T01:20:05.872 INFO:teuthology.orchestra.run.smithi070.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 858 kB/s | 43 kB 00:00 2024-04-01T01:20:05.872 INFO:teuthology.orchestra.run.smithi189.stdout:(91/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-04-01T01:20:05.897 INFO:teuthology.orchestra.run.smithi189.stdout:(92/174): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2024-04-01T01:20:05.923 INFO:teuthology.orchestra.run.smithi189.stdout:(93/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-04-01T01:20:05.948 INFO:teuthology.orchestra.run.smithi189.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-04-01T01:20:05.948 INFO:teuthology.orchestra.run.smithi070.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 386 kB/s | 39 kB 00:00 2024-04-01T01:20:06.073 INFO:teuthology.orchestra.run.smithi189.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-04-01T01:20:06.098 INFO:teuthology.orchestra.run.smithi189.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-04-01T01:20:06.148 INFO:teuthology.orchestra.run.smithi070.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 150 kB/s | 41 kB 00:00 2024-04-01T01:20:06.148 INFO:teuthology.orchestra.run.smithi189.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-04-01T01:20:06.182 INFO:teuthology.orchestra.run.smithi189.stdout:(98/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.4 MB/s | 803 kB 00:00 2024-04-01T01:20:06.282 INFO:teuthology.orchestra.run.smithi189.stdout:(99/174): elfutils-libelf-devel-0.190-2.el9.x86 117 kB/s | 21 kB 00:00 2024-04-01T01:20:06.298 INFO:teuthology.orchestra.run.smithi070.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-04-01T01:20:06.299 INFO:teuthology.orchestra.run.smithi189.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 169 kB/s | 25 kB 00:00 2024-04-01T01:20:06.323 INFO:teuthology.orchestra.run.smithi070.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 589 kB/s | 220 kB 00:00 2024-04-01T01:20:06.348 INFO:teuthology.orchestra.run.smithi070.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 175 kB/s | 35 kB 00:00 2024-04-01T01:20:06.373 INFO:teuthology.orchestra.run.smithi070.stdout:(84/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 925 kB/s | 46 kB 00:00 2024-04-01T01:20:06.399 INFO:teuthology.orchestra.run.smithi070.stdout:(85/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-04-01T01:20:06.424 INFO:teuthology.orchestra.run.smithi070.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 746 kB/s | 56 kB 00:00 2024-04-01T01:20:06.449 INFO:teuthology.orchestra.run.smithi070.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 836 kB/s | 63 kB 00:00 2024-04-01T01:20:06.474 INFO:teuthology.orchestra.run.smithi070.stdout:(88/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 587 kB/s | 44 kB 00:00 2024-04-01T01:20:06.475 INFO:teuthology.orchestra.run.smithi189.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 1.2 MB/s | 245 kB 00:00 2024-04-01T01:20:06.499 INFO:teuthology.orchestra.run.smithi070.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-04-01T01:20:06.500 INFO:teuthology.orchestra.run.smithi189.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.2 MB/s | 248 kB 00:00 2024-04-01T01:20:06.525 INFO:teuthology.orchestra.run.smithi070.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-04-01T01:20:06.567 INFO:teuthology.orchestra.run.smithi070.stdout:(91/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-04-01T01:20:06.567 INFO:teuthology.orchestra.run.smithi189.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 332 kB/s | 22 kB 00:00 2024-04-01T01:20:06.592 INFO:teuthology.orchestra.run.smithi070.stdout:(92/174): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-04-01T01:20:06.592 INFO:teuthology.orchestra.run.smithi189.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 798 kB/s | 94 kB 00:00 2024-04-01T01:20:06.605 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-01T01:20:06.617 INFO:teuthology.orchestra.run.smithi070.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-04-01T01:20:06.642 INFO:teuthology.orchestra.run.smithi070.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-04-01T01:20:06.776 INFO:teuthology.orchestra.run.smithi070.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-04-01T01:20:06.801 INFO:teuthology.orchestra.run.smithi070.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-01T01:20:06.827 INFO:teuthology.orchestra.run.smithi070.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-01T01:20:06.852 INFO:teuthology.orchestra.run.smithi070.stdout:(98/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.4 MB/s | 803 kB 00:00 2024-04-01T01:20:06.919 INFO:teuthology.orchestra.run.smithi070.stdout:(99/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 277 kB/s | 25 kB 00:00 2024-04-01T01:20:06.944 INFO:teuthology.orchestra.run.smithi070.stdout:(100/174): elfutils-libelf-devel-0.190-2.el9.x8 151 kB/s | 21 kB 00:00 2024-04-01T01:20:06.951 INFO:teuthology.orchestra.run.smithi189.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.2 MB/s | 863 kB 00:00 2024-04-01T01:20:06.976 INFO:teuthology.orchestra.run.smithi189.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.4 MB/s | 548 kB 00:00 2024-04-01T01:20:07.028 INFO:teuthology.orchestra.run.smithi070.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 2.2 MB/s | 245 kB 00:00 2024-04-01T01:20:07.035 INFO:teuthology.orchestra.run.smithi189.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 3.6 MB/s | 309 kB 00:00 2024-04-01T01:20:07.060 INFO:teuthology.orchestra.run.smithi189.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 232 kB/s | 19 kB 00:00 2024-04-01T01:20:07.078 INFO:teuthology.orchestra.run.smithi070.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-04-01T01:20:07.103 INFO:teuthology.orchestra.run.smithi070.stdout:(103/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.5 MB/s | 248 kB 00:00 2024-04-01T01:20:07.170 INFO:teuthology.orchestra.run.smithi070.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 13 MB/s | 4.1 MB 00:00 2024-04-01T01:20:07.211 INFO:teuthology.orchestra.run.smithi189.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 173 kB/s | 26 kB 00:00 2024-04-01T01:20:07.262 INFO:teuthology.orchestra.run.smithi070.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 121 kB/s | 22 kB 00:00 2024-04-01T01:20:07.294 INFO:teuthology.orchestra.run.smithi189.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 596 kB/s | 49 kB 00:00 2024-04-01T01:20:07.377 INFO:teuthology.orchestra.run.smithi189.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-01T01:20:07.403 INFO:teuthology.orchestra.run.smithi189.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 629 kB/s | 67 kB 00:00 2024-04-01T01:20:07.429 INFO:teuthology.orchestra.run.smithi070.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.1 MB/s | 548 kB 00:00 2024-04-01T01:20:07.461 INFO:teuthology.orchestra.run.smithi189.stdout:(113/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.2 MB/s | 4.1 MB 00:01 2024-04-01T01:20:07.479 INFO:teuthology.orchestra.run.smithi070.stdout:(107/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.2 MB/s | 863 kB 00:00 2024-04-01T01:20:07.487 INFO:teuthology.orchestra.run.smithi189.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-04-01T01:20:07.496 INFO:teuthology.orchestra.run.smithi070.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-04-01T01:20:07.512 INFO:teuthology.orchestra.run.smithi189.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 838 kB 00:00 2024-04-01T01:20:07.521 INFO:teuthology.orchestra.run.smithi070.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-04-01T01:20:07.537 INFO:teuthology.orchestra.run.smithi189.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 7.2 MB/s | 548 kB 00:00 2024-04-01T01:20:07.538 INFO:teuthology.orchestra.run.smithi070.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 625 kB/s | 26 kB 00:00 2024-04-01T01:20:07.563 INFO:teuthology.orchestra.run.smithi189.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 578 kB/s | 43 kB 00:00 2024-04-01T01:20:07.572 INFO:teuthology.orchestra.run.smithi070.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 973 kB/s | 49 kB 00:00 2024-04-01T01:20:07.588 INFO:teuthology.orchestra.run.smithi189.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 432 kB/s | 32 kB 00:00 2024-04-01T01:20:07.613 INFO:teuthology.orchestra.run.smithi189.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-04-01T01:20:07.622 INFO:teuthology.orchestra.run.smithi070.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 807 kB/s | 67 kB 00:00 2024-04-01T01:20:07.638 INFO:teuthology.orchestra.run.smithi189.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-01T01:20:07.647 INFO:teuthology.orchestra.run.smithi070.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-01T01:20:07.663 INFO:teuthology.orchestra.run.smithi189.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-01T01:20:07.689 INFO:teuthology.orchestra.run.smithi189.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-04-01T01:20:07.714 INFO:teuthology.orchestra.run.smithi189.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 471 kB/s | 35 kB 00:00 2024-04-01T01:20:07.715 INFO:teuthology.orchestra.run.smithi070.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2024-04-01T01:20:07.773 INFO:teuthology.orchestra.run.smithi070.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-04-01T01:20:07.798 INFO:teuthology.orchestra.run.smithi070.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 521 kB/s | 43 kB 00:00 2024-04-01T01:20:07.823 INFO:teuthology.orchestra.run.smithi189.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.2 MB/s | 203 kB 00:00 2024-04-01T01:20:07.832 INFO:teuthology.orchestra.run.smithi070.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 551 kB/s | 32 kB 00:00 2024-04-01T01:20:07.857 INFO:teuthology.orchestra.run.smithi070.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 256 kB/s | 15 kB 00:00 2024-04-01T01:20:07.890 INFO:teuthology.orchestra.run.smithi070.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-01T01:20:07.957 INFO:teuthology.orchestra.run.smithi070.stdout:(120/174): libarrow-9.0.0-10.el9.x86_64.rpm 9.2 MB/s | 4.4 MB 00:00 2024-04-01T01:20:07.965 INFO:teuthology.orchestra.run.smithi189.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 577 kB/s | 144 kB 00:00 2024-04-01T01:20:07.983 INFO:teuthology.orchestra.run.smithi070.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-04-01T01:20:08.008 INFO:teuthology.orchestra.run.smithi070.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-04-01T01:20:08.015 INFO:teuthology.orchestra.run.smithi189.stdout:(126/174): python3-grpcio-1.46.7-10.el9.x86_64. 6.3 MB/s | 2.0 MB 00:00 2024-04-01T01:20:08.032 INFO:teuthology.orchestra.run.smithi189.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 51 kB/s | 11 kB 00:00 2024-04-01T01:20:08.033 INFO:teuthology.orchestra.run.smithi070.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 469 kB/s | 35 kB 00:00 2024-04-01T01:20:08.049 INFO:teuthology.orchestra.run.smithi189.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 212 kB/s | 18 kB 00:00 2024-04-01T01:20:08.058 INFO:teuthology.orchestra.run.smithi070.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-04-01T01:20:08.091 INFO:teuthology.orchestra.run.smithi189.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 398 kB/s | 23 kB 00:00 2024-04-01T01:20:08.107 INFO:teuthology.orchestra.run.smithi189.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 259 kB/s | 19 kB 00:00 2024-04-01T01:20:08.117 INFO:teuthology.orchestra.run.smithi070.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 19 MB/s | 2.0 MB 00:00 2024-04-01T01:20:08.125 INFO:teuthology.orchestra.run.smithi189.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 260 kB/s | 20 kB 00:00 2024-04-01T01:20:08.142 INFO:teuthology.orchestra.run.smithi070.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-04-01T01:20:08.159 INFO:teuthology.orchestra.run.smithi070.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-01T01:20:08.176 INFO:teuthology.orchestra.run.smithi070.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 301 kB/s | 18 kB 00:00 2024-04-01T01:20:08.193 INFO:teuthology.orchestra.run.smithi070.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-04-01T01:20:08.210 INFO:teuthology.orchestra.run.smithi070.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 385 kB/s | 19 kB 00:00 2024-04-01T01:20:08.227 INFO:teuthology.orchestra.run.smithi070.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 390 kB/s | 20 kB 00:00 2024-04-01T01:20:08.233 INFO:teuthology.orchestra.run.smithi189.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-04-01T01:20:08.259 INFO:teuthology.orchestra.run.smithi189.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 307 kB/s | 46 kB 00:00 2024-04-01T01:20:08.268 INFO:teuthology.orchestra.run.smithi070.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 14 MB/s | 1.0 MB 00:00 2024-04-01T01:20:08.283 INFO:teuthology.orchestra.run.smithi189.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 496 kB/s | 79 kB 00:00 2024-04-01T01:20:08.295 INFO:teuthology.orchestra.run.smithi070.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 550 kB/s | 46 kB 00:00 2024-04-01T01:20:08.300 INFO:teuthology.orchestra.run.smithi189.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 862 kB/s | 58 kB 00:00 2024-04-01T01:20:08.311 INFO:teuthology.orchestra.run.smithi070.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 936 kB/s | 79 kB 00:00 2024-04-01T01:20:08.336 INFO:teuthology.orchestra.run.smithi070.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 855 kB/s | 58 kB 00:00 2024-04-01T01:20:08.359 INFO:teuthology.orchestra.run.smithi189.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.6 MB/s | 272 kB 00:00 2024-04-01T01:20:08.361 INFO:teuthology.orchestra.run.smithi070.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-04-01T01:20:08.376 INFO:teuthology.orchestra.run.smithi189.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 179 kB/s | 16 kB 00:00 2024-04-01T01:20:08.387 INFO:teuthology.orchestra.run.smithi070.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-04-01T01:20:08.410 INFO:teuthology.orchestra.run.smithi189.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 826 kB/s | 90 kB 00:00 2024-04-01T01:20:08.412 INFO:teuthology.orchestra.run.smithi070.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-01T01:20:08.435 INFO:teuthology.orchestra.run.smithi189.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-04-01T01:20:08.437 INFO:teuthology.orchestra.run.smithi070.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-04-01T01:20:08.460 INFO:teuthology.orchestra.run.smithi189.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 367 kB/s | 31 kB 00:00 2024-04-01T01:20:08.462 INFO:teuthology.orchestra.run.smithi070.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 409 kB/s | 31 kB 00:00 2024-04-01T01:20:08.488 INFO:teuthology.orchestra.run.smithi070.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-01T01:20:08.494 INFO:teuthology.orchestra.run.smithi189.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-04-01T01:20:08.513 INFO:teuthology.orchestra.run.smithi070.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 785 kB/s | 59 kB 00:00 2024-04-01T01:20:08.519 INFO:teuthology.orchestra.run.smithi189.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 705 kB/s | 59 kB 00:00 2024-04-01T01:20:08.538 INFO:teuthology.orchestra.run.smithi070.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 473 kB/s | 36 kB 00:00 2024-04-01T01:20:08.544 INFO:teuthology.orchestra.run.smithi189.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 428 kB/s | 36 kB 00:00 2024-04-01T01:20:08.564 INFO:teuthology.orchestra.run.smithi070.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-04-01T01:20:08.569 INFO:teuthology.orchestra.run.smithi189.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-04-01T01:20:08.589 INFO:teuthology.orchestra.run.smithi070.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-01T01:20:08.603 INFO:teuthology.orchestra.run.smithi189.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-01T01:20:08.614 INFO:teuthology.orchestra.run.smithi070.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-01T01:20:08.628 INFO:teuthology.orchestra.run.smithi189.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-04-01T01:20:08.639 INFO:teuthology.orchestra.run.smithi070.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-04-01T01:20:08.662 INFO:teuthology.orchestra.run.smithi189.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.6 MB/s | 427 kB 00:00 2024-04-01T01:20:08.664 INFO:teuthology.orchestra.run.smithi070.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 264 kB/s | 20 kB 00:00 2024-04-01T01:20:08.687 INFO:teuthology.orchestra.run.smithi189.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 241 kB/s | 20 kB 00:00 2024-04-01T01:20:08.690 INFO:teuthology.orchestra.run.smithi070.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-01T01:20:08.732 INFO:teuthology.orchestra.run.smithi070.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-04-01T01:20:08.771 INFO:teuthology.orchestra.run.smithi189.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-04-01T01:20:08.888 INFO:teuthology.orchestra.run.smithi189.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-04-01T01:20:09.013 INFO:teuthology.orchestra.run.smithi189.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 18 MB/s | 2.3 MB 00:00 2024-04-01T01:20:09.138 INFO:teuthology.orchestra.run.smithi189.stdout:(152/174): NetworkManager-libnm-1.47.2-1.el9.x8 15 MB/s | 1.8 MB 00:00 2024-04-01T01:20:09.197 INFO:teuthology.orchestra.run.smithi189.stdout:(153/174): NetworkManager-team-1.47.2-1.el9.x86 704 kB/s | 41 kB 00:00 2024-04-01T01:20:09.264 INFO:teuthology.orchestra.run.smithi189.stdout:(154/174): NetworkManager-tui-1.47.2-1.el9.x86_ 3.6 MB/s | 247 kB 00:00 2024-04-01T01:20:09.290 INFO:teuthology.orchestra.run.smithi070.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 4.1 MB/s | 2.3 MB 00:00 2024-04-01T01:20:09.339 INFO:teuthology.orchestra.run.smithi189.stdout:(155/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.2 MB/s | 321 kB 00:00 2024-04-01T01:20:09.439 INFO:teuthology.orchestra.run.smithi189.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 713 kB/s | 71 kB 00:00 2024-04-01T01:20:09.482 INFO:teuthology.orchestra.run.smithi189.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 874 kB/s | 36 kB 00:00 2024-04-01T01:20:09.548 INFO:teuthology.orchestra.run.smithi189.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-04-01T01:20:09.574 INFO:teuthology.orchestra.run.smithi070.stdout:(152/174): NetworkManager-libnm-1.47.2-1.el9.x8 6.4 MB/s | 1.8 MB 00:00 2024-04-01T01:20:09.599 INFO:teuthology.orchestra.run.smithi189.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 256 kB 00:00 2024-04-01T01:20:09.607 INFO:teuthology.orchestra.run.smithi070.stdout:(153/174): NetworkManager-team-1.47.2-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-04-01T01:20:09.641 INFO:teuthology.orchestra.run.smithi189.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 554 kB/s | 23 kB 00:00 2024-04-01T01:20:09.666 INFO:teuthology.orchestra.run.smithi070.stdout:(154/174): NetworkManager-tui-1.47.2-1.el9.x86_ 4.1 MB/s | 247 kB 00:00 2024-04-01T01:20:09.691 INFO:teuthology.orchestra.run.smithi189.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-04-01T01:20:09.733 INFO:teuthology.orchestra.run.smithi189.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 155 kB 00:00 2024-04-01T01:20:09.758 INFO:teuthology.orchestra.run.smithi070.stdout:(155/174): xfsdump-3.1.12-4.el9.x86_64.rpm 3.4 MB/s | 321 kB 00:00 2024-04-01T01:20:09.775 INFO:teuthology.orchestra.run.smithi189.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 136 kB 00:00 2024-04-01T01:20:09.834 INFO:teuthology.orchestra.run.smithi189.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-04-01T01:20:09.850 INFO:teuthology.orchestra.run.smithi070.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 778 kB/s | 71 kB 00:00 2024-04-01T01:20:09.884 INFO:teuthology.orchestra.run.smithi189.stdout:(165/174): librbd1-19.0.0-1977.g384f31f3.el9.x8 2.8 MB/s | 3.1 MB 00:01 2024-04-01T01:20:09.892 INFO:teuthology.orchestra.run.smithi070.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 878 kB/s | 36 kB 00:00 2024-04-01T01:20:09.901 INFO:teuthology.orchestra.run.smithi189.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 428 kB/s | 29 kB 00:00 2024-04-01T01:20:09.959 INFO:teuthology.orchestra.run.smithi070.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-04-01T01:20:10.009 INFO:teuthology.orchestra.run.smithi070.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 256 kB 00:00 2024-04-01T01:20:10.009 INFO:teuthology.orchestra.run.smithi189.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 20 MB/s | 2.2 MB 00:00 2024-04-01T01:20:10.051 INFO:teuthology.orchestra.run.smithi070.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 557 kB/s | 23 kB 00:00 2024-04-01T01:20:10.060 INFO:teuthology.orchestra.run.smithi189.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 6.9 MB/s | 1.2 MB 00:00 2024-04-01T01:20:10.093 INFO:teuthology.orchestra.run.smithi070.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 108 kB 00:00 2024-04-01T01:20:10.110 INFO:teuthology.orchestra.run.smithi189.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 9.1 MB/s | 465 kB 00:00 2024-04-01T01:20:10.135 INFO:teuthology.orchestra.run.smithi070.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 155 kB 00:00 2024-04-01T01:20:10.177 INFO:teuthology.orchestra.run.smithi070.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 136 kB 00:00 2024-04-01T01:20:10.177 INFO:teuthology.orchestra.run.smithi189.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 14 MB/s | 2.3 MB 00:00 2024-04-01T01:20:10.202 INFO:teuthology.orchestra.run.smithi189.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 339 kB/s | 31 kB 00:00 2024-04-01T01:20:10.218 INFO:teuthology.orchestra.run.smithi070.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 64 kB 00:00 2024-04-01T01:20:10.277 INFO:teuthology.orchestra.run.smithi070.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 492 kB/s | 29 kB 00:00 2024-04-01T01:20:10.311 INFO:teuthology.orchestra.run.smithi189.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.0 MB/s | 1.1 MB 00:00 2024-04-01T01:20:10.336 INFO:teuthology.orchestra.run.smithi189.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 557 kB/s | 74 kB 00:00 2024-04-01T01:20:10.353 INFO:teuthology.orchestra.run.smithi070.stdout:(166/174): openssl-3.0.7-27.el9.x86_64.rpm 16 MB/s | 1.2 MB 00:00 2024-04-01T01:20:10.436 INFO:teuthology.orchestra.run.smithi070.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 26 MB/s | 2.2 MB 00:00 2024-04-01T01:20:10.503 INFO:teuthology.orchestra.run.smithi070.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 34 MB/s | 2.3 MB 00:00 2024-04-01T01:20:10.546 INFO:teuthology.orchestra.run.smithi070.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 11 MB/s | 465 kB 00:00 2024-04-01T01:20:10.587 INFO:teuthology.orchestra.run.smithi070.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 751 kB/s | 31 kB 00:00 2024-04-01T01:20:10.645 INFO:teuthology.orchestra.run.smithi070.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-04-01T01:20:10.762 INFO:teuthology.orchestra.run.smithi070.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 637 kB/s | 74 kB 00:00 2024-04-01T01:20:10.846 INFO:teuthology.orchestra.run.smithi070.stdout:(173/174): librados2-19.0.0-1977.g384f31f3.el9. 1.6 MB/s | 3.4 MB 00:02 2024-04-01T01:20:11.486 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-01T01:20:11.663 INFO:teuthology.orchestra.run.smithi070.stdout:(174/174): librbd1-19.0.0-1977.g384f31f3.el9.x8 1.1 MB/s | 3.1 MB 00:02 2024-04-01T01:20:11.666 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-01T01:20:11.666 INFO:teuthology.orchestra.run.smithi070.stdout:Total 8.5 MB/s | 226 MB 00:26 2024-04-01T01:20:12.032 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-01T01:20:12.310 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-01T01:20:12.311 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-01T01:20:14.053 INFO:teuthology.orchestra.run.smithi081.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-01T01:20:14.054 INFO:teuthology.orchestra.run.smithi081.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-01T01:20:14.066 INFO:teuthology.orchestra.run.smithi081.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-01T01:20:14.066 INFO:teuthology.orchestra.run.smithi081.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-01T01:20:14.068 INFO:teuthology.orchestra.run.smithi081.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-01T01:20:14.068 INFO:teuthology.orchestra.run.smithi081.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-01T01:20:14.070 INFO:teuthology.orchestra.run.smithi081.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-01T01:20:14.073 INFO:teuthology.orchestra.run.smithi081.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-01T01:20:14.073 INFO:teuthology.orchestra.run.smithi081.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-01T01:20:14.154 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-04-01T01:20:14.165 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================================ 2024-04-01T01:20:14.166 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-04-01T01:20:14.166 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================================ 2024-04-01T01:20:14.166 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-04-01T01:20:14.166 INFO:teuthology.orchestra.run.smithi081.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-01T01:20:14.166 INFO:teuthology.orchestra.run.smithi081.stdout: ceph x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 6.6 k 2024-04-01T01:20:14.166 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 5.3 M 2024-04-01T01:20:14.166 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 889 k 2024-04-01T01:20:14.166 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 145 k 2024-04-01T01:20:14.166 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 1.5 M 2024-04-01T01:20:14.166 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 139 k 2024-04-01T01:20:14.167 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 3.5 M 2024-04-01T01:20:14.167 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 7.4 M 2024-04-01T01:20:14.167 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 49 k 2024-04-01T01:20:14.167 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 9.8 M 2024-04-01T01:20:14.167 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 47 M 2024-04-01T01:20:14.167 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 264 k 2024-04-01T01:20:14.167 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 748 k 2024-04-01T01:20:14.167 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-mirror x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 221 k 2024-04-01T01:20:14.167 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-shell x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 27 k 2024-04-01T01:20:14.167 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-top noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 18 k 2024-04-01T01:20:14.167 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-01T01:20:14.167 INFO:teuthology.orchestra.run.smithi081.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-01T01:20:14.167 INFO:teuthology.orchestra.run.smithi081.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-01T01:20:14.168 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 32 k 2024-04-01T01:20:14.168 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 746 k 2024-04-01T01:20:14.168 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 127 k 2024-04-01T01:20:14.168 INFO:teuthology.orchestra.run.smithi081.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-01T01:20:14.168 INFO:teuthology.orchestra.run.smithi081.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-01T01:20:14.168 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-01T01:20:14.168 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-01T01:20:14.168 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-01T01:20:14.168 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-01T01:20:14.168 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 161 k 2024-04-01T01:20:14.168 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 320 k 2024-04-01T01:20:14.168 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 298 k 2024-04-01T01:20:14.168 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 99 k 2024-04-01T01:20:14.169 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 85 k 2024-04-01T01:20:14.169 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.1 M 2024-04-01T01:20:14.169 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 171 k 2024-04-01T01:20:14.169 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-01T01:20:14.169 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrading: 2024-04-01T01:20:14.169 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager x86_64 1:1.47.2-1.el9 CentOS-BaseOS 2.3 M 2024-04-01T01:20:14.169 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 CentOS-BaseOS 1.8 M 2024-04-01T01:20:14.169 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 CentOS-BaseOS 41 k 2024-04-01T01:20:14.169 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 CentOS-BaseOS 247 k 2024-04-01T01:20:14.169 INFO:teuthology.orchestra.run.smithi081.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-01T01:20:14.169 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-01T01:20:14.169 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-01T01:20:14.169 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-01T01:20:14.169 INFO:teuthology.orchestra.run.smithi081.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-01T01:20:14.170 INFO:teuthology.orchestra.run.smithi081.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-01T01:20:14.170 INFO:teuthology.orchestra.run.smithi081.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-01T01:20:14.170 INFO:teuthology.orchestra.run.smithi081.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-01T01:20:14.170 INFO:teuthology.orchestra.run.smithi081.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-01T01:20:14.170 INFO:teuthology.orchestra.run.smithi081.stdout: librados2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.4 M 2024-04-01T01:20:14.170 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 3.1 M 2024-04-01T01:20:14.170 INFO:teuthology.orchestra.run.smithi081.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-01T01:20:14.170 INFO:teuthology.orchestra.run.smithi081.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-01T01:20:14.170 INFO:teuthology.orchestra.run.smithi081.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-01T01:20:14.170 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-01T01:20:14.170 INFO:teuthology.orchestra.run.smithi081.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-01T01:20:14.170 INFO:teuthology.orchestra.run.smithi081.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-01T01:20:14.170 INFO:teuthology.orchestra.run.smithi081.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-01T01:20:14.171 INFO:teuthology.orchestra.run.smithi081.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-01T01:20:14.171 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-01T01:20:14.171 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-04-01T01:20:14.171 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-01T01:20:14.171 INFO:teuthology.orchestra.run.smithi081.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-01T01:20:14.171 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-01T01:20:14.171 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 21 M 2024-04-01T01:20:14.171 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 31 k 2024-04-01T01:20:14.171 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 2.4 M 2024-04-01T01:20:14.171 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 248 k 2024-04-01T01:20:14.171 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 4.7 M 2024-04-01T01:20:14.171 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 17 M 2024-04-01T01:20:14.171 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1977.g384f31f3.el9 ceph-noarch 15 k 2024-04-01T01:20:14.172 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 25 k 2024-04-01T01:20:14.172 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-01T01:20:14.172 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-01T01:20:14.172 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-01T01:20:14.172 INFO:teuthology.orchestra.run.smithi081.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-01T01:20:14.172 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-01T01:20:14.172 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-01T01:20:14.172 INFO:teuthology.orchestra.run.smithi081.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-01T01:20:14.172 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-01T01:20:14.172 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-01T01:20:14.172 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 163 k 2024-04-01T01:20:14.172 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-01T01:20:14.172 INFO:teuthology.orchestra.run.smithi081.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-01T01:20:14.173 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-01T01:20:14.173 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-01T01:20:14.173 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-01T01:20:14.173 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 501 k 2024-04-01T01:20:14.173 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-01T01:20:14.173 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2 x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 5.1 M 2024-04-01T01:20:14.173 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-01T01:20:14.173 INFO:teuthology.orchestra.run.smithi081.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-01T01:20:14.173 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-01T01:20:14.173 INFO:teuthology.orchestra.run.smithi081.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-01T01:20:14.173 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-01T01:20:14.173 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-01T01:20:14.173 INFO:teuthology.orchestra.run.smithi081.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-01T01:20:14.174 INFO:teuthology.orchestra.run.smithi081.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-01T01:20:14.174 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-01T01:20:14.174 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-01T01:20:14.174 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-01T01:20:14.174 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-01T01:20:14.174 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-01T01:20:14.174 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-01T01:20:14.174 INFO:teuthology.orchestra.run.smithi081.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-01T01:20:14.174 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-01T01:20:14.174 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-01T01:20:14.174 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-01T01:20:14.174 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 45 k 2024-04-01T01:20:14.174 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common x86_64 2:19.0.0-1977.g384f31f3.el9 ceph 131 k 2024-04-01T01:20:14.174 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-01T01:20:14.175 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-01T01:20:14.175 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-01T01:20:14.175 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-01T01:20:14.175 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-01T01:20:14.175 INFO:teuthology.orchestra.run.smithi081.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-01T01:20:14.175 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-01T01:20:14.175 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-01T01:20:14.175 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-01T01:20:14.175 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-01T01:20:14.175 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-01T01:20:14.175 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-01T01:20:14.175 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-01T01:20:14.175 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-01T01:20:14.176 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-01T01:20:14.176 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-01T01:20:14.176 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-01T01:20:14.176 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-01T01:20:14.176 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-01T01:20:14.176 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-01T01:20:14.176 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-01T01:20:14.176 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-01T01:20:14.176 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-01T01:20:14.176 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-01T01:20:14.176 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-01T01:20:14.176 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-01T01:20:14.176 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-01T01:20:14.176 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-01T01:20:14.177 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-01T01:20:14.177 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-01T01:20:14.177 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-01T01:20:14.177 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-01T01:20:14.177 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-01T01:20:14.177 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-01T01:20:14.177 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-01T01:20:14.177 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-01T01:20:14.177 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-01T01:20:14.177 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-01T01:20:14.177 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-01T01:20:14.177 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-01T01:20:14.177 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-01T01:20:14.177 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-01T01:20:14.177 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-01T01:20:14.178 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-01T01:20:14.178 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-01T01:20:14.178 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-01T01:20:14.179 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-01T01:20:14.179 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-01T01:20:14.179 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-01T01:20:14.179 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-01T01:20:14.179 INFO:teuthology.orchestra.run.smithi081.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-01T01:20:14.179 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-01T01:20:14.179 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-01T01:20:14.180 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-01T01:20:14.180 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-01T01:20:14.180 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-01T01:20:14.180 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-01T01:20:14.180 INFO:teuthology.orchestra.run.smithi081.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-01T01:20:14.181 INFO:teuthology.orchestra.run.smithi081.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-01T01:20:14.181 INFO:teuthology.orchestra.run.smithi081.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-01T01:20:14.181 INFO:teuthology.orchestra.run.smithi081.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-01T01:20:14.181 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-01T01:20:14.181 INFO:teuthology.orchestra.run.smithi081.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-01T01:20:14.181 INFO:teuthology.orchestra.run.smithi081.stdout:Installing weak dependencies: 2024-04-01T01:20:14.181 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-01T01:20:14.181 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:20:14.181 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-04-01T01:20:14.181 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================================ 2024-04-01T01:20:14.181 INFO:teuthology.orchestra.run.smithi081.stdout:Install 150 Packages 2024-04-01T01:20:14.181 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrade 24 Packages 2024-04-01T01:20:14.181 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:20:14.182 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 226 M 2024-04-01T01:20:14.182 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-04-01T01:20:14.211 INFO:teuthology.orchestra.run.smithi189.stdout:(174/174): librados2-19.0.0-1977.g384f31f3.el9. 627 kB/s | 3.4 MB 00:05 2024-04-01T01:20:14.216 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-01T01:20:14.216 INFO:teuthology.orchestra.run.smithi189.stdout:Total 11 MB/s | 226 MB 00:21 2024-04-01T01:20:14.311 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-01T01:20:14.311 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-01T01:20:14.580 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-01T01:20:14.762 INFO:teuthology.orchestra.run.smithi081.stdout:(1/174): ceph-19.0.0-1977.g384f31f3.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-04-01T01:20:14.856 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-01T01:20:14.856 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-01T01:20:15.687 INFO:teuthology.orchestra.run.smithi081.stdout:(2/174): ceph-fuse-19.0.0-1977.g384f31f3.el9.x8 962 kB/s | 889 kB 00:00 2024-04-01T01:20:15.888 INFO:teuthology.orchestra.run.smithi081.stdout:(3/174): ceph-immutable-object-cache-19.0.0-197 724 kB/s | 145 kB 00:00 2024-04-01T01:20:16.997 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-01T01:20:16.998 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-01T01:20:17.363 INFO:teuthology.orchestra.run.smithi081.stdout:(4/174): ceph-base-19.0.0-1977.g384f31f3.el9.x8 1.9 MB/s | 5.3 MB 00:02 2024-04-01T01:20:17.738 INFO:teuthology.orchestra.run.smithi081.stdout:(5/174): ceph-mgr-19.0.0-1977.g384f31f3.el9.x86 4.0 MB/s | 1.5 MB 00:00 2024-04-01T01:20:18.088 INFO:teuthology.orchestra.run.smithi081.stdout:(6/174): ceph-mds-19.0.0-1977.g384f31f3.el9.x86 1.1 MB/s | 2.4 MB 00:02 2024-04-01T01:20:18.597 INFO:teuthology.orchestra.run.smithi081.stdout:(7/174): ceph-mon-19.0.0-1977.g384f31f3.el9.x86 5.5 MB/s | 4.7 MB 00:00 2024-04-01T01:20:20.281 INFO:teuthology.orchestra.run.smithi081.stdout:(8/174): ceph-radosgw-19.0.0-1977.g384f31f3.el9 5.8 MB/s | 9.8 MB 00:01 2024-04-01T01:20:20.339 INFO:teuthology.orchestra.run.smithi081.stdout:(9/174): ceph-selinux-19.0.0-1977.g384f31f3.el9 434 kB/s | 25 kB 00:00 2024-04-01T01:20:21.399 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-01T01:20:21.500 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-01T01:20:21.534 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-01T01:20:21.581 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-01T01:20:21.590 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-01T01:20:21.631 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-01T01:20:21.691 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-01T01:20:21.725 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-01T01:20:21.751 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-01T01:20:21.801 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-01T01:20:22.123 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-01T01:20:22.177 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-04-01T01:20:22.314 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-04-01T01:20:22.345 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-04-01T01:20:22.436 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-04-01T01:20:22.466 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 12/198 2024-04-01T01:20:22.502 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-01T01:20:22.696 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-01T01:20:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-01T01:20:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-04-01T01:20:22.886 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-04-01T01:20:22.917 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 17/198 2024-04-01T01:20:22.991 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 17/198 2024-04-01T01:20:23.037 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-01T01:20:23.060 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-01T01:20:23.075 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-01T01:20:23.269 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-01T01:20:23.300 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-01T01:20:23.339 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-01T01:20:23.524 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-01T01:20:23.633 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-04-01T01:20:23.701 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-04-01T01:20:23.800 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-01T01:20:23.852 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-01T01:20:24.069 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-01T01:20:24.198 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-01T01:20:24.294 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-01T01:20:24.332 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-01T01:20:24.359 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-01T01:20:24.402 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-01T01:20:24.476 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-01T01:20:24.530 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-01T01:20:24.564 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-01T01:20:24.568 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-01T01:20:24.633 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-01T01:20:24.668 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-01T01:20:24.674 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-01T01:20:24.711 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-01T01:20:24.726 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-01T01:20:24.744 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-01T01:20:24.771 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-01T01:20:24.772 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-01T01:20:24.781 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-01T01:20:24.821 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-01T01:20:24.848 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-01T01:20:24.876 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-01T01:20:24.885 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-common-2:19.0.0-1977.g384f31f3.el9. 39/198 2024-04-01T01:20:24.917 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-01T01:20:24.919 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el 40/198 2024-04-01T01:20:24.942 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-01T01:20:24.948 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_6 41/198 2024-04-01T01:20:24.990 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-01T01:20:25.018 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-01T01:20:25.063 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-01T01:20:25.106 INFO:teuthology.orchestra.run.smithi081.stdout:(10/174): ceph-test-19.0.0-1977.g384f31f3.el9.x 10 MB/s | 47 MB 00:04 2024-04-01T01:20:25.136 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-01T01:20:25.206 INFO:teuthology.orchestra.run.smithi081.stdout:(11/174): cephfs-mirror-19.0.0-1977.g384f31f3.e 2.2 MB/s | 221 kB 00:00 2024-04-01T01:20:25.248 INFO:teuthology.orchestra.run.smithi081.stdout:(12/174): cephfs-shell-19.0.0-1977.g384f31f3.el 651 kB/s | 27 kB 00:00 2024-04-01T01:20:25.268 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-01T01:20:25.307 INFO:teuthology.orchestra.run.smithi081.stdout:(13/174): libcephfs-devel-19.0.0-1977.g384f31f3 541 kB/s | 32 kB 00:00 2024-04-01T01:20:25.317 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-01T01:20:25.322 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-01T01:20:25.393 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-04-01T01:20:25.507 INFO:teuthology.orchestra.run.smithi081.stdout:(14/174): libcephfs2-19.0.0-1977.g384f31f3.el9. 3.6 MB/s | 746 kB 00:00 2024-04-01T01:20:25.522 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-04-01T01:20:25.553 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-04-01T01:20:25.557 INFO:teuthology.orchestra.run.smithi081.stdout:(15/174): libcephsqlite-19.0.0-1977.g384f31f3.e 3.2 MB/s | 163 kB 00:00 2024-04-01T01:20:25.616 INFO:teuthology.orchestra.run.smithi081.stdout:(16/174): librados-devel-19.0.0-1977.g384f31f3. 2.1 MB/s | 127 kB 00:00 2024-04-01T01:20:25.644 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-04-01T01:20:25.684 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 12/198 2024-04-01T01:20:25.699 INFO:teuthology.orchestra.run.smithi081.stdout:(17/174): libradosstriper1-19.0.0-1977.g384f31f 5.9 MB/s | 501 kB 00:00 2024-04-01T01:20:25.733 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-01T01:20:25.826 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-01T01:20:25.900 INFO:teuthology.orchestra.run.smithi081.stdout:(18/174): ceph-osd-19.0.0-1977.g384f31f3.el9.x8 2.2 MB/s | 17 MB 00:07 2024-04-01T01:20:25.903 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-01T01:20:25.909 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-01T01:20:25.963 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-01T01:20:25.975 INFO:teuthology.orchestra.run.smithi081.stdout:(19/174): python3-ceph-argparse-19.0.0-1977.g38 605 kB/s | 45 kB 00:00 2024-04-01T01:20:25.979 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-01T01:20:25.983 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-04-01T01:20:26.023 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-01T01:20:26.032 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-01T01:20:26.050 INFO:teuthology.orchestra.run.smithi081.stdout:(20/174): python3-ceph-common-19.0.0-1977.g384f 1.7 MB/s | 131 kB 00:00 2024-04-01T01:20:26.065 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-01T01:20:26.086 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-04-01T01:20:26.103 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-01T01:20:26.109 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 17/198 2024-04-01T01:20:26.117 INFO:teuthology.orchestra.run.smithi081.stdout:(21/174): python3-cephfs-19.0.0-1977.g384f31f3. 2.4 MB/s | 161 kB 00:00 2024-04-01T01:20:26.141 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_6 53/198 2024-04-01T01:20:26.166 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 17/198 2024-04-01T01:20:26.171 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-01T01:20:26.205 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-01T01:20:26.228 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-01T01:20:26.243 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-01T01:20:26.243 INFO:teuthology.orchestra.run.smithi081.stdout:(22/174): python3-rados-19.0.0-1977.g384f31f3.e 2.5 MB/s | 320 kB 00:00 2024-04-01T01:20:26.318 INFO:teuthology.orchestra.run.smithi081.stdout:(23/174): librgw2-19.0.0-1977.g384f31f3.el9.x86 8.3 MB/s | 5.1 MB 00:00 2024-04-01T01:20:26.360 INFO:teuthology.orchestra.run.smithi081.stdout:(24/174): python3-rbd-19.0.0-1977.g384f31f3.el9 2.5 MB/s | 298 kB 00:00 2024-04-01T01:20:26.385 INFO:teuthology.orchestra.run.smithi081.stdout:(25/174): python3-rgw-19.0.0-1977.g384f31f3.el9 1.5 MB/s | 99 kB 00:00 2024-04-01T01:20:26.411 INFO:teuthology.orchestra.run.smithi081.stdout:(26/174): rbd-fuse-19.0.0-1977.g384f31f3.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-01T01:20:26.428 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-01T01:20:26.445 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-01T01:20:26.459 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-01T01:20:26.498 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-01T01:20:26.528 INFO:teuthology.orchestra.run.smithi081.stdout:(27/174): rbd-nbd-19.0.0-1977.g384f31f3.el9.x86 1.4 MB/s | 171 kB 00:00 2024-04-01T01:20:26.594 INFO:teuthology.orchestra.run.smithi081.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1977.g 473 kB/s | 31 kB 00:00 2024-04-01T01:20:26.664 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-01T01:20:26.670 INFO:teuthology.orchestra.run.smithi081.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1977.g384f31f 1.8 MB/s | 139 kB 00:00 2024-04-01T01:20:26.689 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-01T01:20:26.729 INFO:teuthology.orchestra.run.smithi081.stdout:(30/174): rbd-mirror-19.0.0-1977.g384f31f3.el9. 9.0 MB/s | 3.1 MB 00:00 2024-04-01T01:20:26.739 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-01T01:20:26.783 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-01T01:20:26.811 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-04-01T01:20:26.872 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-04-01T01:20:26.916 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-01T01:20:26.962 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-01T01:20:26.969 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-01T01:20:27.023 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-01T01:20:27.024 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-01T01:20:27.096 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-01T01:20:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-01T01:20:27.167 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-01T01:20:27.214 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-01T01:20:27.239 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-01T01:20:27.258 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-01T01:20:27.301 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-01T01:20:27.343 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-01T01:20:27.365 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-01T01:20:27.379 INFO:teuthology.orchestra.run.smithi081.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-04-01T01:20:27.429 INFO:teuthology.orchestra.run.smithi081.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1977.g38 4.9 MB/s | 248 kB 00:00 2024-04-01T01:20:27.464 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-01T01:20:27.476 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-01T01:20:27.503 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-01T01:20:27.504 INFO:teuthology.orchestra.run.smithi081.stdout:(33/174): ceph-mgr-dashboard-19.0.0-1977.g384f3 4.2 MB/s | 3.5 MB 00:00 2024-04-01T01:20:27.529 INFO:teuthology.orchestra.run.smithi081.stdout:(34/174): ceph-mgr-rook-19.0.0-1977.g384f31f3.e 491 kB/s | 49 kB 00:00 2024-04-01T01:20:27.530 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-01T01:20:27.532 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-01T01:20:27.554 INFO:teuthology.orchestra.run.smithi081.stdout:(35/174): ceph-prometheus-alerts-19.0.0-1977.g3 305 kB/s | 15 kB 00:00 2024-04-01T01:20:27.563 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-01T01:20:27.565 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-01T01:20:27.604 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-01T01:20:27.605 INFO:teuthology.orchestra.run.smithi081.stdout:(36/174): ceph-volume-19.0.0-1977.g384f31f3.el9 3.4 MB/s | 264 kB 00:00 2024-04-01T01:20:27.639 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-01T01:20:27.647 INFO:teuthology.orchestra.run.smithi081.stdout:(37/174): cephfs-top-19.0.0-1977.g384f31f3.el9. 439 kB/s | 18 kB 00:00 2024-04-01T01:20:27.672 INFO:teuthology.orchestra.run.smithi081.stdout:(38/174): cephadm-19.0.0-1977.g384f31f3.el9.noa 6.2 MB/s | 748 kB 00:00 2024-04-01T01:20:27.692 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-01T01:20:27.697 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-01T01:20:27.731 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-01T01:20:27.737 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-01T01:20:27.777 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-01T01:20:27.797 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-01T01:20:27.803 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-01T01:20:27.828 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-01T01:20:27.831 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-01T01:20:27.875 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-01T01:20:27.907 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-01T01:20:27.911 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-01T01:20:27.935 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-01T01:20:28.004 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-01T01:20:28.035 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-01T01:20:28.040 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-common-2:19.0.0-1977.g384f31f3.el9. 39/198 2024-04-01T01:20:28.075 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el 40/198 2024-04-01T01:20:28.089 INFO:teuthology.orchestra.run.smithi081.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 1.5 MB/s | 662 kB 00:00 2024-04-01T01:20:28.104 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_6 41/198 2024-04-01T01:20:28.143 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-01T01:20:28.174 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-01T01:20:28.195 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-01T01:20:28.198 INFO:teuthology.orchestra.run.smithi081.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 977 kB/s | 106 kB 00:00 2024-04-01T01:20:28.219 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-01T01:20:28.242 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-01T01:20:28.292 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-01T01:20:28.298 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-01T01:20:28.332 INFO:teuthology.orchestra.run.smithi081.stdout:(41/174): bison-3.7.4-5.el9.x86_64.rpm 1.4 MB/s | 920 kB 00:00 2024-04-01T01:20:28.399 INFO:teuthology.orchestra.run.smithi081.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 1.5 MB/s | 307 kB 00:00 2024-04-01T01:20:28.416 INFO:teuthology.orchestra.run.smithi081.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 378 kB/s | 32 kB 00:00 2024-04-01T01:20:28.423 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-01T01:20:28.471 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-01T01:20:28.480 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-01T01:20:28.532 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-01T01:20:28.616 INFO:teuthology.orchestra.run.smithi081.stdout:(44/174): ceph-common-19.0.0-1977.g384f31f3.el9 1.5 MB/s | 21 MB 00:14 2024-04-01T01:20:28.660 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-01T01:20:28.722 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-01T01:20:28.754 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-01T01:20:28.785 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-01T01:20:28.813 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-01T01:20:28.848 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-01T01:20:28.895 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-01T01:20:28.927 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-01T01:20:28.957 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-01T01:20:28.957 INFO:teuthology.orchestra.run.smithi081.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 32 kB/s | 17 kB 00:00 2024-04-01T01:20:28.990 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-01T01:20:29.032 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-01T01:20:29.033 INFO:teuthology.orchestra.run.smithi081.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 24 kB/s | 10 kB 00:00 2024-04-01T01:20:29.066 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-01T01:20:29.067 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-01T01:20:29.101 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-01T01:20:29.142 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-01T01:20:29.159 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-01T01:20:29.177 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-01T01:20:29.194 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-01T01:20:29.204 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-01T01:20:29.253 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-01T01:20:29.264 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-01T01:20:29.266 INFO:teuthology.orchestra.run.smithi081.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 516 kB/s | 159 kB 00:00 2024-04-01T01:20:29.302 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-01T01:20:29.348 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_6 53/198 2024-04-01T01:20:29.358 INFO:teuthology.orchestra.run.smithi081.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 136 kB/s | 44 kB 00:00 2024-04-01T01:20:29.371 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-01T01:20:29.379 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-01T01:20:29.416 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-01T01:20:29.448 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-01T01:20:29.476 INFO:teuthology.orchestra.run.smithi081.stdout:(49/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.8 MB/s | 3.0 MB 00:01 2024-04-01T01:20:29.494 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-01T01:20:29.591 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-01T01:20:29.601 INFO:teuthology.orchestra.run.smithi081.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 1.9 MB/s | 661 kB 00:00 2024-04-01T01:20:29.640 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-01T01:20:29.648 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-01T01:20:29.676 INFO:teuthology.orchestra.run.smithi081.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 1.8 MB/s | 578 kB 00:00 2024-04-01T01:20:29.701 INFO:teuthology.orchestra.run.smithi081.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 206 kB/s | 46 kB 00:00 2024-04-01T01:20:29.726 INFO:teuthology.orchestra.run.smithi081.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.2 MB/s | 282 kB 00:00 2024-04-01T01:20:29.752 INFO:teuthology.orchestra.run.smithi081.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-04-01T01:20:29.785 INFO:teuthology.orchestra.run.smithi081.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.5 MB/s | 217 kB 00:00 2024-04-01T01:20:29.811 INFO:teuthology.orchestra.run.smithi081.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 408 kB/s | 34 kB 00:00 2024-04-01T01:20:29.846 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-01T01:20:29.939 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-01T01:20:29.983 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-01T01:20:30.116 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-01T01:20:30.161 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-01T01:20:30.215 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-01T01:20:30.289 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-01T01:20:30.328 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-01T01:20:30.356 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-01T01:20:30.364 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-01T01:20:30.369 INFO:teuthology.orchestra.run.smithi081.stdout:(57/174): postgresql-contrib-13.14-1.el9.x86_64 1.4 MB/s | 817 kB 00:00 2024-04-01T01:20:30.405 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-01T01:20:30.411 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-01T01:20:30.439 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-01T01:20:30.447 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-01T01:20:30.486 INFO:teuthology.orchestra.run.smithi081.stdout:(58/174): postgresql-13.14-1.el9.x86_64.rpm 2.2 MB/s | 1.5 MB 00:00 2024-04-01T01:20:30.487 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-01T01:20:30.490 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-01T01:20:30.532 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-01T01:20:30.567 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-01T01:20:30.620 INFO:teuthology.orchestra.run.smithi081.stdout:(59/174): openblas-openmp-0.3.21-2.el9.x86_64.r 5.5 MB/s | 4.7 MB 00:00 2024-04-01T01:20:30.623 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-01T01:20:30.645 INFO:teuthology.orchestra.run.smithi081.stdout:(60/174): postgresql-private-libs-13.14-1.el9.x 481 kB/s | 132 kB 00:00 2024-04-01T01:20:30.665 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-01T01:20:30.713 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-01T01:20:30.743 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-01T01:20:30.784 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-01T01:20:30.800 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-01T01:20:30.833 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-01T01:20:30.878 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-01T01:20:30.911 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-01T01:20:30.917 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-01T01:20:30.954 INFO:teuthology.orchestra.run.smithi081.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 3.0 MB/s | 1.0 MB 00:00 2024-04-01T01:20:30.957 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-01T01:20:30.979 INFO:teuthology.orchestra.run.smithi081.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 233 kB/s | 78 kB 00:00 2024-04-01T01:20:30.983 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-01T01:20:31.008 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-01T01:20:31.054 INFO:teuthology.orchestra.run.smithi081.stdout:(63/174): postgresql-server-13.14-1.el9.x86_64. 10 MB/s | 5.7 MB 00:00 2024-04-01T01:20:31.097 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-01T01:20:31.196 INFO:teuthology.orchestra.run.smithi081.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 113 kB/s | 24 kB 00:00 2024-04-01T01:20:31.217 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-01T01:20:31.305 INFO:teuthology.orchestra.run.smithi081.stdout:(65/174): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-04-01T01:20:31.324 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-01T01:20:31.330 INFO:teuthology.orchestra.run.smithi081.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 66 kB/s | 18 kB 00:00 2024-04-01T01:20:31.355 INFO:teuthology.orchestra.run.smithi081.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 914 kB/s | 144 kB 00:00 2024-04-01T01:20:31.374 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-01T01:20:31.380 INFO:teuthology.orchestra.run.smithi081.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 421 kB/s | 32 kB 00:00 2024-04-01T01:20:31.406 INFO:teuthology.orchestra.run.smithi081.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.3 MB/s | 217 kB 00:00 2024-04-01T01:20:31.423 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-01T01:20:31.478 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-01T01:20:31.541 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-01T01:20:31.606 INFO:teuthology.orchestra.run.smithi081.stdout:(70/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-04-01T01:20:31.632 INFO:teuthology.orchestra.run.smithi081.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 697 kB/s | 174 kB 00:00 2024-04-01T01:20:31.657 INFO:teuthology.orchestra.run.smithi081.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 275 kB/s | 69 kB 00:00 2024-04-01T01:20:31.662 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-01T01:20:31.683 INFO:teuthology.orchestra.run.smithi081.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 551 kB/s | 41 kB 00:00 2024-04-01T01:20:31.695 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-01T01:20:31.707 INFO:teuthology.orchestra.run.smithi081.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-04-01T01:20:31.711 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-01T01:20:31.733 INFO:teuthology.orchestra.run.smithi081.stdout:(75/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-04-01T01:20:31.758 INFO:teuthology.orchestra.run.smithi081.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-04-01T01:20:31.775 INFO:teuthology.orchestra.run.smithi081.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 715 kB/s | 47 kB 00:00 2024-04-01T01:20:31.792 INFO:teuthology.orchestra.run.smithi081.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-04-01T01:20:31.842 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-01T01:20:31.918 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-01T01:20:31.950 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-01T01:20:31.975 INFO:teuthology.orchestra.run.smithi081.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 193 kB/s | 39 kB 00:00 2024-04-01T01:20:31.981 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-01T01:20:31.992 INFO:teuthology.orchestra.run.smithi081.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 206 kB/s | 41 kB 00:00 2024-04-01T01:20:32.009 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-01T01:20:32.043 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-01T01:20:32.082 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-01T01:20:32.114 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-01T01:20:32.152 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-01T01:20:32.193 INFO:teuthology.orchestra.run.smithi081.stdout:(81/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.0 MB/s | 220 kB 00:00 2024-04-01T01:20:32.209 INFO:teuthology.orchestra.run.smithi081.stdout:(82/174): qatlib-service-24.02.0-1.el9.x86_64.r 162 kB/s | 35 kB 00:00 2024-04-01T01:20:32.229 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-01T01:20:32.263 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-01T01:20:32.297 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-01T01:20:32.337 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-01T01:20:32.385 INFO:teuthology.orchestra.run.smithi081.stdout:(83/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-04-01T01:20:32.402 INFO:teuthology.orchestra.run.smithi081.stdout:(84/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 222 kB/s | 46 kB 00:00 2024-04-01T01:20:32.425 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-01T01:20:32.427 INFO:teuthology.orchestra.run.smithi081.stdout:(85/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-04-01T01:20:32.452 INFO:teuthology.orchestra.run.smithi081.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 854 kB/s | 56 kB 00:00 2024-04-01T01:20:32.477 INFO:teuthology.orchestra.run.smithi081.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-04-01T01:20:32.494 INFO:teuthology.orchestra.run.smithi081.stdout:(88/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 660 kB/s | 44 kB 00:00 2024-04-01T01:20:32.519 INFO:teuthology.orchestra.run.smithi081.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 1.2 MB/s | 79 kB 00:00 2024-04-01T01:20:32.570 INFO:teuthology.orchestra.run.smithi081.stdout:(90/174): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.1 MB 00:00 2024-04-01T01:20:32.595 INFO:teuthology.orchestra.run.smithi081.stdout:(91/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 241 kB 00:00 2024-04-01T01:20:32.611 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-01T01:20:32.620 INFO:teuthology.orchestra.run.smithi081.stdout:(92/174): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-04-01T01:20:32.645 INFO:teuthology.orchestra.run.smithi081.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 124 kB 00:00 2024-04-01T01:20:32.672 INFO:teuthology.orchestra.run.smithi081.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-04-01T01:20:32.897 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-01T01:20:32.907 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-01T01:20:32.953 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-01T01:20:32.984 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-01T01:20:32.992 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-01T01:20:33.031 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-01T01:20:33.032 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-01T01:20:33.063 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-01T01:20:33.090 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-01T01:20:33.129 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-01T01:20:33.186 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-01T01:20:33.402 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-01T01:20:33.445 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 124/198 2024-04-01T01:20:33.513 INFO:teuthology.orchestra.run.smithi081.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 210 kB/s | 187 kB 00:00 2024-04-01T01:20:33.535 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 124/198 2024-04-01T01:20:33.567 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 125/198 2024-04-01T01:20:33.596 INFO:teuthology.orchestra.run.smithi081.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 211 kB/s | 195 kB 00:00 2024-04-01T01:20:33.603 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-01T01:20:33.755 INFO:teuthology.orchestra.run.smithi081.stdout:(97/174): elfutils-libelf-devel-0.190-2.el9.x86 136 kB/s | 21 kB 00:00 2024-04-01T01:20:33.838 INFO:teuthology.orchestra.run.smithi081.stdout:(98/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 306 kB/s | 25 kB 00:00 2024-04-01T01:20:33.914 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-01T01:20:33.958 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-01T01:20:33.992 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-01T01:20:34.040 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-01T01:20:34.058 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-01T01:20:34.090 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 128/198 2024-04-01T01:20:34.134 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-01T01:20:34.161 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 128/198 2024-04-01T01:20:34.174 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-01T01:20:34.183 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 129/198 2024-04-01T01:20:34.330 INFO:teuthology.orchestra.run.smithi081.stdout:(99/174): libgfortran-11.4.1-3.el9.x86_64.rpm 477 kB/s | 803 kB 00:01 2024-04-01T01:20:34.345 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-01T01:20:34.372 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-01T01:20:34.430 INFO:teuthology.orchestra.run.smithi081.stdout:(100/174): python3-requests-2.25.1-8.el9.noarch 136 kB/s | 125 kB 00:00 2024-04-01T01:20:34.458 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-01T01:20:34.531 INFO:teuthology.orchestra.run.smithi081.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 1.2 MB/s | 245 kB 00:00 2024-04-01T01:20:34.589 INFO:teuthology.orchestra.run.smithi081.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-04-01T01:20:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:(103/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.2 MB/s | 248 kB 00:00 2024-04-01T01:20:34.790 INFO:teuthology.orchestra.run.smithi081.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 4.5 MB/s | 4.1 MB 00:00 2024-04-01T01:20:35.015 INFO:teuthology.orchestra.run.smithi081.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.4 MB/s | 548 kB 00:00 2024-04-01T01:20:35.082 INFO:teuthology.orchestra.run.smithi081.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 4.6 MB/s | 309 kB 00:00 2024-04-01T01:20:35.111 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-01T01:20:35.124 INFO:teuthology.orchestra.run.smithi081.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 466 kB/s | 19 kB 00:00 2024-04-01T01:20:35.225 INFO:teuthology.orchestra.run.smithi081.stdout:(108/174): lua-devel-5.4.4-4.el9.x86_64.rpm 35 kB/s | 22 kB 00:00 2024-04-01T01:20:35.229 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-01T01:20:35.342 INFO:teuthology.orchestra.run.smithi081.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-04-01T01:20:35.450 INFO:teuthology.orchestra.run.smithi081.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 449 kB/s | 49 kB 00:00 2024-04-01T01:20:35.559 INFO:teuthology.orchestra.run.smithi081.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2024-04-01T01:20:35.726 INFO:teuthology.orchestra.run.smithi081.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 933 kB/s | 151 kB 00:00 2024-04-01T01:20:35.801 INFO:teuthology.orchestra.run.smithi081.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 6.5 MB/s | 4.4 MB 00:00 2024-04-01T01:20:35.901 INFO:teuthology.orchestra.run.smithi081.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 5.4 MB/s | 548 kB 00:00 2024-04-01T01:20:35.960 INFO:teuthology.orchestra.run.smithi081.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 744 kB/s | 43 kB 00:00 2024-04-01T01:20:36.019 INFO:teuthology.orchestra.run.smithi081.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 553 kB/s | 32 kB 00:00 2024-04-01T01:20:36.052 INFO:teuthology.orchestra.run.smithi081.stdout:(117/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 838 kB 00:00 2024-04-01T01:20:36.078 INFO:teuthology.orchestra.run.smithi081.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 256 kB/s | 15 kB 00:00 2024-04-01T01:20:36.103 INFO:teuthology.orchestra.run.smithi081.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-04-01T01:20:36.170 INFO:teuthology.orchestra.run.smithi081.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.5 MB/s | 303 kB 00:00 2024-04-01T01:20:36.197 INFO:teuthology.orchestra.run.smithi081.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-04-01T01:20:36.220 INFO:teuthology.orchestra.run.smithi081.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 701 kB/s | 35 kB 00:00 2024-04-01T01:20:36.271 INFO:teuthology.orchestra.run.smithi081.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-04-01T01:20:36.321 INFO:teuthology.orchestra.run.smithi081.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-04-01T01:20:36.371 INFO:teuthology.orchestra.run.smithi081.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 215 kB/s | 11 kB 00:00 2024-04-01T01:20:36.376 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-04-01T01:20:36.422 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-04-01T01:20:36.422 INFO:teuthology.orchestra.run.smithi081.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-04-01T01:20:36.472 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-01T01:20:36.497 INFO:teuthology.orchestra.run.smithi081.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 7.4 MB/s | 2.0 MB 00:00 2024-04-01T01:20:36.522 INFO:teuthology.orchestra.run.smithi081.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-04-01T01:20:36.555 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-01T01:20:36.564 INFO:teuthology.orchestra.run.smithi081.stdout:(129/174): protobuf-compiler-3.14.0-13.el9.x86_ 447 kB/s | 863 kB 00:01 2024-04-01T01:20:36.586 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-01T01:20:36.589 INFO:teuthology.orchestra.run.smithi081.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-04-01T01:20:36.623 INFO:teuthology.orchestra.run.smithi081.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 195 kB/s | 20 kB 00:00 2024-04-01T01:20:36.625 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-01T01:20:36.653 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-01T01:20:36.673 INFO:teuthology.orchestra.run.smithi081.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-04-01T01:20:36.716 INFO:teuthology.orchestra.run.smithi081.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 369 kB/s | 46 kB 00:00 2024-04-01T01:20:36.740 INFO:teuthology.orchestra.run.smithi081.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 863 kB/s | 58 kB 00:00 2024-04-01T01:20:36.774 INFO:teuthology.orchestra.run.smithi081.stdout:(135/174): python3-kubernetes-26.1.0-2.el9.noar 5.5 MB/s | 1.0 MB 00:00 2024-04-01T01:20:36.799 INFO:teuthology.orchestra.run.smithi081.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-04-01T01:20:36.825 INFO:teuthology.orchestra.run.smithi081.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-04-01T01:20:36.842 INFO:teuthology.orchestra.run.smithi081.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 674 kB/s | 28 kB 00:00 2024-04-01T01:20:36.884 INFO:teuthology.orchestra.run.smithi081.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 527 kB/s | 31 kB 00:00 2024-04-01T01:20:36.909 INFO:teuthology.orchestra.run.smithi081.stdout:(140/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-04-01T01:20:36.926 INFO:teuthology.orchestra.run.smithi081.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-04-01T01:20:36.951 INFO:teuthology.orchestra.run.smithi081.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-04-01T01:20:36.956 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-04-01T01:20:36.968 INFO:teuthology.orchestra.run.smithi081.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 609 kB/s | 36 kB 00:00 2024-04-01T01:20:36.968 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-01T01:20:36.985 INFO:teuthology.orchestra.run.smithi081.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 866 kB/s | 51 kB 00:00 2024-04-01T01:20:36.992 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 131/198 2024-04-01T01:20:36.999 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 124/198 2024-04-01T01:20:37.019 INFO:teuthology.orchestra.run.smithi081.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-04-01T01:20:37.052 INFO:teuthology.orchestra.run.smithi081.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-01T01:20:37.077 INFO:teuthology.orchestra.run.smithi081.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 341 kB/s | 20 kB 00:00 2024-04-01T01:20:37.082 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 131/198 2024-04-01T01:20:37.082 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-01T01:20:37.083 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-01T01:20:37.083 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:20:37.090 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 124/198 2024-04-01T01:20:37.121 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 125/198 2024-04-01T01:20:37.127 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 132/198 2024-04-01T01:20:37.127 INFO:teuthology.orchestra.run.smithi081.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-01T01:20:37.158 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-01T01:20:37.211 INFO:teuthology.orchestra.run.smithi081.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 1.8 MB/s | 427 kB 00:00 2024-04-01T01:20:37.295 INFO:teuthology.orchestra.run.smithi081.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-04-01T01:20:37.395 INFO:teuthology.orchestra.run.smithi081.stdout:(151/174): NetworkManager-1.47.2-1.el9.x86_64.r 23 MB/s | 2.3 MB 00:00 2024-04-01T01:20:37.487 INFO:teuthology.orchestra.run.smithi081.stdout:(152/174): NetworkManager-libnm-1.47.2-1.el9.x8 20 MB/s | 1.8 MB 00:00 2024-04-01T01:20:37.529 INFO:teuthology.orchestra.run.smithi081.stdout:(153/174): NetworkManager-team-1.47.2-1.el9.x86 986 kB/s | 41 kB 00:00 2024-04-01T01:20:37.573 INFO:teuthology.orchestra.run.smithi081.stdout:(154/174): NetworkManager-tui-1.47.2-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-04-01T01:20:37.613 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-01T01:20:37.613 INFO:teuthology.orchestra.run.smithi081.stdout:(155/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-04-01T01:20:37.653 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 128/198 2024-04-01T01:20:37.671 INFO:teuthology.orchestra.run.smithi081.stdout:(156/174): librbd1-19.0.0-1977.g384f31f3.el9.x8 6.8 MB/s | 3.1 MB 00:00 2024-04-01T01:20:37.723 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 128/198 2024-04-01T01:20:37.746 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 129/198 2024-04-01T01:20:37.889 INFO:teuthology.orchestra.run.smithi081.stdout:(157/174): librados2-19.0.0-1977.g384f31f3.el9. 4.5 MB/s | 3.4 MB 00:00 2024-04-01T01:20:38.305 INFO:teuthology.orchestra.run.smithi081.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 57 kB/s | 36 kB 00:00 2024-04-01T01:20:38.347 INFO:teuthology.orchestra.run.smithi081.stdout:(159/174): acl-2.3.1-4.el9.x86_64.rpm 97 kB/s | 71 kB 00:00 2024-04-01T01:20:38.798 INFO:teuthology.orchestra.run.smithi081.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 224 kB/s | 192 kB 00:00 2024-04-01T01:20:38.923 INFO:teuthology.orchestra.run.smithi081.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 40 kB/s | 23 kB 00:00 2024-04-01T01:20:39.232 INFO:teuthology.orchestra.run.smithi081.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 276 kB/s | 256 kB 00:00 2024-04-01T01:20:39.540 INFO:teuthology.orchestra.run.smithi081.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 146 kB/s | 108 kB 00:00 2024-04-01T01:20:39.716 INFO:teuthology.orchestra.run.smithi081.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 196 kB/s | 155 kB 00:00 2024-04-01T01:20:40.007 INFO:teuthology.orchestra.run.smithi081.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 175 kB/s | 136 kB 00:00 2024-04-01T01:20:40.058 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-04-01T01:20:40.094 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-04-01T01:20:40.233 INFO:teuthology.orchestra.run.smithi081.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 92 kB/s | 64 kB 00:00 2024-04-01T01:20:40.308 INFO:teuthology.orchestra.run.smithi081.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 48 kB/s | 29 kB 00:00 2024-04-01T01:20:40.641 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-04-01T01:20:40.680 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 131/198 2024-04-01T01:20:40.778 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 131/198 2024-04-01T01:20:40.778 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-01T01:20:40.778 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-01T01:20:40.778 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:20:40.806 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 132/198 2024-04-01T01:20:42.258 INFO:teuthology.orchestra.run.smithi081.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 551 kB/s | 1.2 MB 00:02 2024-04-01T01:20:43.633 INFO:teuthology.orchestra.run.smithi081.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 338 kB/s | 465 kB 00:01 2024-04-01T01:20:43.675 INFO:teuthology.orchestra.run.smithi081.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 646 kB/s | 2.2 MB 00:03 2024-04-01T01:20:43.959 INFO:teuthology.orchestra.run.smithi081.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 640 kB/s | 2.3 MB 00:03 2024-04-01T01:20:44.092 INFO:teuthology.orchestra.run.smithi081.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 694 kB/s | 74 kB 00:00 2024-04-01T01:20:44.301 INFO:teuthology.orchestra.run.smithi081.stdout:(173/174): util-linux-user-2.37.4-18.el9.x86_64 47 kB/s | 31 kB 00:00 2024-04-01T01:20:45.792 INFO:teuthology.orchestra.run.smithi081.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 519 kB/s | 1.1 MB 00:02 2024-04-01T01:20:45.797 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-04-01T01:20:45.798 INFO:teuthology.orchestra.run.smithi081.stdout:Total 7.1 MB/s | 226 MB 00:31 2024-04-01T01:20:46.158 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-04-01T01:20:46.434 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-04-01T01:20:46.435 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-04-01T01:20:48.445 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-04-01T01:20:48.446 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-04-01T01:20:49.094 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 132/198 2024-04-01T01:20:49.095 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /sys 2024-04-01T01:20:49.095 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /proc 2024-04-01T01:20:49.095 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /mnt 2024-04-01T01:20:49.095 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /var/tmp 2024-04-01T01:20:49.095 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /home 2024-04-01T01:20:49.095 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /root 2024-04-01T01:20:49.095 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /tmp 2024-04-01T01:20:49.095 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:20:49.257 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 133/198 2024-04-01T01:20:49.272 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 133/198 2024-04-01T01:20:49.273 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:49.273 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-01T01:20:49.273 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:20:50.744 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 134/198 2024-04-01T01:20:50.769 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 134/198 2024-04-01T01:20:50.770 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:50.770 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-01T01:20:50.771 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-01T01:20:50.771 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-01T01:20:50.771 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:20:50.994 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 135/198 2024-04-01T01:20:51.016 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 135/198 2024-04-01T01:20:51.016 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:51.016 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-01T01:20:51.016 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-01T01:20:51.017 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-01T01:20:51.017 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:20:51.453 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 136/198 2024-04-01T01:20:51.475 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 136/198 2024-04-01T01:20:51.476 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:51.476 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-01T01:20:51.476 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-01T01:20:51.476 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-01T01:20:51.476 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:20:51.563 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-01T01:20:52.022 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 138/198 2024-04-01T01:20:52.053 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 138/198 2024-04-01T01:20:52.085 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.e 139/198 2024-04-01T01:20:52.145 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3. 140/198 2024-04-01T01:20:52.177 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 141/198 2024-04-01T01:20:52.677 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 132/198 2024-04-01T01:20:52.677 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /sys 2024-04-01T01:20:52.677 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /proc 2024-04-01T01:20:52.677 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /mnt 2024-04-01T01:20:52.677 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /var/tmp 2024-04-01T01:20:52.677 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /home 2024-04-01T01:20:52.677 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /root 2024-04-01T01:20:52.677 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /tmp 2024-04-01T01:20:52.677 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:20:52.833 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 133/198 2024-04-01T01:20:52.848 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 133/198 2024-04-01T01:20:52.849 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:52.849 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-01T01:20:52.849 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:20:53.205 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 141/198 2024-04-01T01:20:53.245 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 142/198 2024-04-01T01:20:54.209 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 142/198 2024-04-01T01:20:54.245 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 143/198 2024-04-01T01:20:54.346 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 143/198 2024-04-01T01:20:54.351 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 134/198 2024-04-01T01:20:54.378 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 134/198 2024-04-01T01:20:54.378 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:54.378 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-01T01:20:54.378 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-01T01:20:54.378 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-01T01:20:54.378 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:20:54.520 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el 144/198 2024-04-01T01:20:54.547 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 145/198 2024-04-01T01:20:54.570 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 145/198 2024-04-01T01:20:54.570 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:54.570 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-01T01:20:54.570 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-01T01:20:54.570 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-01T01:20:54.570 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:20:54.606 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 146/198 2024-04-01T01:20:54.617 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 146/198 2024-04-01T01:20:54.620 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 135/198 2024-04-01T01:20:54.642 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 135/198 2024-04-01T01:20:54.642 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:54.643 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-01T01:20:54.643 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-01T01:20:54.643 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-01T01:20:54.643 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:20:54.665 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 147/198 2024-04-01T01:20:55.088 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 136/198 2024-04-01T01:20:55.110 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 136/198 2024-04-01T01:20:55.110 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:55.110 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-01T01:20:55.111 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-01T01:20:55.111 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-01T01:20:55.111 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:20:55.206 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-01T01:20:55.395 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-04-01T01:20:55.495 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-01T01:20:55.523 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-01T01:20:55.525 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-01T01:20:55.553 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-01T01:20:55.559 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 149/198 2024-04-01T01:20:55.562 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-01T01:20:55.581 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 149/198 2024-04-01T01:20:55.581 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:55.581 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-01T01:20:55.581 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-01T01:20:55.581 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-01T01:20:55.581 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:20:55.621 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-01T01:20:55.626 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1977.g384f3 150/198 2024-04-01T01:20:55.640 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 138/198 2024-04-01T01:20:55.648 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1977.g384f3 150/198 2024-04-01T01:20:55.648 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:55.648 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-01T01:20:55.648 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:20:55.663 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 138/198 2024-04-01T01:20:55.684 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-01T01:20:55.693 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-04-01T01:20:55.694 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.e 139/198 2024-04-01T01:20:55.716 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-04-01T01:20:55.716 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:55.716 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-01T01:20:55.716 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-01T01:20:55.716 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-01T01:20:55.716 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:20:55.732 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-01T01:20:55.755 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3. 140/198 2024-04-01T01:20:55.766 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-01T01:20:55.778 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 141/198 2024-04-01T01:20:55.830 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-01T01:20:56.027 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 152/198 2024-04-01T01:20:56.050 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 152/198 2024-04-01T01:20:56.050 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:56.050 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-01T01:20:56.050 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-01T01:20:56.050 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-01T01:20:56.051 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:20:56.161 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-01T01:20:56.218 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-04-01T01:20:56.339 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-04-01T01:20:56.369 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-04-01T01:20:56.460 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-04-01T01:20:56.489 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 12/198 2024-04-01T01:20:56.534 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-01T01:20:56.720 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-01T01:20:56.779 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-01T01:20:56.804 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 141/198 2024-04-01T01:20:56.807 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-04-01T01:20:56.837 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 142/198 2024-04-01T01:20:56.893 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-04-01T01:20:56.917 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 17/198 2024-04-01T01:20:56.982 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 17/198 2024-04-01T01:20:57.012 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-01T01:20:57.035 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-01T01:20:57.050 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-01T01:20:57.235 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-01T01:20:57.267 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-01T01:20:57.310 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-01T01:20:57.473 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-01T01:20:57.628 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-04-01T01:20:57.688 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-04-01T01:20:57.780 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-01T01:20:57.802 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 142/198 2024-04-01T01:20:57.829 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 143/198 2024-04-01T01:20:57.831 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-01T01:20:57.929 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 143/198 2024-04-01T01:20:58.068 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-04-01T01:20:58.094 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el 144/198 2024-04-01T01:20:58.123 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 145/198 2024-04-01T01:20:58.146 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 145/198 2024-04-01T01:20:58.146 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:58.146 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-01T01:20:58.146 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-01T01:20:58.146 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-01T01:20:58.146 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:20:58.182 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-01T01:20:58.183 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 146/198 2024-04-01T01:20:58.193 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 146/198 2024-04-01T01:20:58.242 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 147/198 2024-04-01T01:20:58.274 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-01T01:20:58.312 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-01T01:20:58.344 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-01T01:20:58.394 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-01T01:20:58.467 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-01T01:20:58.521 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-01T01:20:58.559 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-01T01:20:58.623 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-01T01:20:58.658 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-01T01:20:58.702 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-01T01:20:58.748 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-01T01:20:58.804 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-01T01:20:58.863 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-01T01:20:58.900 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-common-2:19.0.0-1977.g384f31f3.el9. 39/198 2024-04-01T01:20:58.943 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el 40/198 2024-04-01T01:20:58.972 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_6 41/198 2024-04-01T01:20:59.042 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-01T01:20:59.087 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-01T01:20:59.092 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-01T01:20:59.126 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 149/198 2024-04-01T01:20:59.148 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 149/198 2024-04-01T01:20:59.148 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:59.148 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-01T01:20:59.149 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-01T01:20:59.149 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-01T01:20:59.149 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:20:59.160 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-01T01:20:59.193 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1977.g384f3 150/198 2024-04-01T01:20:59.214 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1977.g384f3 150/198 2024-04-01T01:20:59.214 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:59.215 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-01T01:20:59.215 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:20:59.260 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-04-01T01:20:59.282 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-04-01T01:20:59.282 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:59.283 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-01T01:20:59.283 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-01T01:20:59.283 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-01T01:20:59.283 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:20:59.293 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-01T01:20:59.365 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-01T01:20:59.594 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 152/198 2024-04-01T01:20:59.617 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 152/198 2024-04-01T01:20:59.617 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:20:59.617 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-01T01:20:59.617 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-01T01:20:59.617 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-01T01:20:59.618 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:20:59.879 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-01T01:20:59.959 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-01T01:21:00.040 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-01T01:21:00.089 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-01T01:21:00.098 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-01T01:21:00.137 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-01T01:21:00.186 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-01T01:21:00.224 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_6 53/198 2024-04-01T01:21:00.254 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-01T01:21:00.490 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 153/198 2024-04-01T01:21:00.507 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-01T01:21:00.531 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 154/198 2024-04-01T01:21:00.570 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 155/198 2024-04-01T01:21:00.597 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-01T01:21:00.708 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-01T01:21:00.776 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 157/198 2024-04-01T01:21:00.781 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-01T01:21:00.825 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-01T01:21:00.888 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-01T01:21:00.928 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 159/198 2024-04-01T01:21:00.954 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-01T01:21:00.982 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-01T01:21:01.003 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-01T01:21:01.037 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-01T01:21:01.057 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-01T01:21:01.080 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-01T01:21:01.113 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_ 163/198 2024-04-01T01:21:01.122 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-01T01:21:01.145 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-01T01:21:01.153 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-01T01:21:01.175 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-04-01T01:21:01.190 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-01T01:21:01.237 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-01T01:21:01.247 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-04-01T01:21:01.273 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-01T01:21:01.297 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-01T01:21:01.308 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-01T01:21:01.332 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-01T01:21:01.350 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-01T01:21:01.482 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-01T01:21:01.522 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-01T01:21:01.552 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-01T01:21:01.594 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-01T01:21:01.687 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-01T01:21:01.719 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-01T01:21:01.750 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-01T01:21:01.776 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-01T01:21:01.801 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-01T01:21:01.882 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-01T01:21:01.955 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-01T01:21:01.997 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-01T01:21:02.008 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-01T01:21:02.016 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-01T01:21:02.121 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-01T01:21:02.160 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-01T01:21:02.207 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-01T01:21:02.263 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-01T01:21:02.445 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-01T01:21:02.488 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-01T01:21:02.618 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-01T01:21:02.670 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-01T01:21:02.702 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-01T01:21:02.741 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-01T01:21:02.769 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-01T01:21:02.803 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-01T01:21:02.824 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 170/198 2024-04-01T01:21:02.842 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-01T01:21:02.874 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-01T01:21:02.904 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-01T01:21:02.910 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-01T01:21:02.950 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-01T01:21:02.980 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-01T01:21:03.015 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-01T01:21:03.039 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-01T01:21:03.040 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-01T01:21:03.071 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-01T01:21:03.081 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-01T01:21:03.096 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-01T01:21:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-01T01:21:03.121 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-01T01:21:03.148 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-01T01:21:03.148 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-01T01:21:03.205 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-01T01:21:03.229 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-01T01:21:03.257 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-01T01:21:03.257 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-01T01:21:03.293 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-01T01:21:03.338 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-01T01:21:03.342 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-01T01:21:03.368 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-01T01:21:03.369 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-01T01:21:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-01T01:21:03.408 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-01T01:21:03.427 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-01T01:21:03.452 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-01T01:21:03.487 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-01T01:21:03.505 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-01T01:21:03.524 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-01T01:21:03.554 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-01T01:21:03.556 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-01T01:21:03.582 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-01T01:21:03.607 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-01T01:21:03.632 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-01T01:21:03.668 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-01T01:21:03.674 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-01T01:21:03.674 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-01T01:21:03.846 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-01T01:21:03.846 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-01T01:21:03.937 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-01T01:21:03.969 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-01T01:21:03.995 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-01T01:21:04.022 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-01T01:21:04.040 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 153/198 2024-04-01T01:21:04.046 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-01T01:21:04.081 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 154/198 2024-04-01T01:21:04.112 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-01T01:21:04.112 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 155/198 2024-04-01T01:21:04.164 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-01T01:21:04.272 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-01T01:21:04.326 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-01T01:21:04.335 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 157/198 2024-04-01T01:21:04.362 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-01T01:21:04.409 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-01T01:21:04.454 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-01T01:21:04.489 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-01T01:21:04.494 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 159/198 2024-04-01T01:21:04.529 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-01T01:21:04.551 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-01T01:21:04.611 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-01T01:21:04.655 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-01T01:21:04.679 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_ 163/198 2024-04-01T01:21:04.705 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-01T01:21:04.725 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-01T01:21:04.744 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-01T01:21:04.758 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-04-01T01:21:04.816 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-04-01T01:21:04.831 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-01T01:21:04.867 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-01T01:21:04.897 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-01T01:21:05.446 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-01T01:21:05.466 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-01T01:21:05.487 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-01T01:21:05.507 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-01T01:21:05.573 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-01T01:21:06.334 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 170/198 2024-04-01T01:21:06.423 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-01T01:21:06.464 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-01T01:21:06.551 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-01T01:21:06.594 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-01T01:21:06.619 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-01T01:21:06.644 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-01T01:21:06.671 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-01T01:21:06.671 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-01T01:21:06.741 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-01T01:21:06.771 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-01T01:21:06.771 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-01T01:21:06.796 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-01T01:21:06.855 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-01T01:21:06.882 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-01T01:21:06.886 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-01T01:21:06.916 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-01T01:21:06.917 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-01T01:21:06.941 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-01T01:21:06.948 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-01T01:21:06.965 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-01T01:21:06.976 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-01T01:21:07.000 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-01T01:21:07.057 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-01T01:21:07.088 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-01T01:21:07.117 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-01T01:21:07.139 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-01T01:21:07.164 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-01T01:21:07.189 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-01T01:21:07.196 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-01T01:21:07.196 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-01T01:21:07.287 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-01T01:21:07.314 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 124/198 2024-04-01T01:21:07.375 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-01T01:21:07.376 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-01T01:21:07.413 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 124/198 2024-04-01T01:21:07.444 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 125/198 2024-04-01T01:21:07.458 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-01T01:21:07.481 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-01T01:21:07.490 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-01T01:21:07.517 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-01T01:21:07.543 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-01T01:21:07.568 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-01T01:21:07.633 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-01T01:21:07.935 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-01T01:21:07.959 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 128/198 2024-04-01T01:21:08.030 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 128/198 2024-04-01T01:21:08.044 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 129/198 2024-04-01T01:21:09.423 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-01T01:21:09.424 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 1/198 2024-04-01T01:21:09.424 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 2/198 2024-04-01T01:21:09.424 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 3/198 2024-04-01T01:21:09.424 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 4/198 2024-04-01T01:21:09.424 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1977.g384f3 5/198 2024-04-01T01:21:09.424 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 6/198 2024-04-01T01:21:09.424 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 7/198 2024-04-01T01:21:09.424 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 8/198 2024-04-01T01:21:09.424 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 9/198 2024-04-01T01:21:09.424 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-04-01T01:21:09.424 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-04-01T01:21:09.425 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 12/198 2024-04-01T01:21:09.425 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 13/198 2024-04-01T01:21:09.425 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 14/198 2024-04-01T01:21:09.425 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_ 15/198 2024-04-01T01:21:09.425 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-04-01T01:21:09.425 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 17/198 2024-04-01T01:21:09.425 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_6 18/198 2024-04-01T01:21:09.425 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 19/198 2024-04-01T01:21:09.425 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 20/198 2024-04-01T01:21:09.425 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el 21/198 2024-04-01T01:21:09.425 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-common-2:19.0.0-1977.g384f31f3.el9. 22/198 2024-04-01T01:21:09.425 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_6 23/198 2024-04-01T01:21:09.425 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 24/198 2024-04-01T01:21:09.426 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 25/198 2024-04-01T01:21:09.426 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 26/198 2024-04-01T01:21:09.426 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 27/198 2024-04-01T01:21:09.441 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 28/198 2024-04-01T01:21:09.442 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 29/198 2024-04-01T01:21:09.442 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3. 30/198 2024-04-01T01:21:09.442 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 31/198 2024-04-01T01:21:09.442 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 32/198 2024-04-01T01:21:09.442 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 33/198 2024-04-01T01:21:09.442 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el 34/198 2024-04-01T01:21:09.442 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 35/198 2024-04-01T01:21:09.442 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.e 36/198 2024-04-01T01:21:09.442 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 37/198 2024-04-01T01:21:09.442 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 38/198 2024-04-01T01:21:09.442 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 39/198 2024-04-01T01:21:09.442 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-01T01:21:09.443 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-01T01:21:09.443 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-01T01:21:09.443 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-01T01:21:09.443 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-01T01:21:09.443 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-01T01:21:09.443 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-01T01:21:09.443 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-01T01:21:09.443 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-01T01:21:09.443 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-01T01:21:09.443 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-01T01:21:09.443 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-01T01:21:09.443 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-01T01:21:09.443 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-01T01:21:09.444 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-01T01:21:09.444 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-01T01:21:09.444 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-01T01:21:09.444 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-01T01:21:09.444 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-01T01:21:09.444 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-01T01:21:09.444 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-01T01:21:09.444 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-01T01:21:09.444 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-01T01:21:09.444 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-01T01:21:09.444 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-01T01:21:09.444 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-01T01:21:09.444 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-01T01:21:09.445 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-01T01:21:09.445 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-01T01:21:09.445 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-01T01:21:09.445 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-01T01:21:09.445 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-01T01:21:09.445 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-01T01:21:09.445 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-01T01:21:09.445 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-01T01:21:09.445 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-01T01:21:09.445 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-01T01:21:09.445 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-01T01:21:09.445 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-01T01:21:09.445 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-01T01:21:09.445 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-01T01:21:09.446 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-01T01:21:09.446 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-01T01:21:09.446 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-01T01:21:09.446 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-01T01:21:09.446 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-01T01:21:09.446 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-01T01:21:09.446 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-01T01:21:09.446 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-01T01:21:09.446 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-01T01:21:09.446 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-01T01:21:09.446 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-01T01:21:09.446 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-01T01:21:09.446 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-01T01:21:09.446 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-01T01:21:09.447 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-01T01:21:09.447 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-01T01:21:09.447 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-01T01:21:09.447 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-01T01:21:09.447 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-01T01:21:09.447 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-01T01:21:09.447 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-01T01:21:09.447 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-01T01:21:09.447 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-01T01:21:09.447 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-01T01:21:09.447 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-01T01:21:09.447 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-01T01:21:09.447 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-01T01:21:09.447 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-01T01:21:09.448 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-01T01:21:09.448 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-01T01:21:09.448 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-01T01:21:09.448 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-01T01:21:09.448 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-01T01:21:09.448 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-01T01:21:09.448 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-01T01:21:09.448 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-01T01:21:09.448 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-01T01:21:09.448 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-01T01:21:09.448 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-01T01:21:09.448 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-01T01:21:09.448 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-01T01:21:09.448 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-01T01:21:09.448 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-01T01:21:09.449 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-01T01:21:09.449 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-01T01:21:09.449 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-01T01:21:09.449 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-01T01:21:09.449 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-01T01:21:09.449 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-01T01:21:09.449 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-01T01:21:09.449 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-01T01:21:09.449 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-01T01:21:09.449 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-01T01:21:09.449 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-01T01:21:09.449 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-01T01:21:09.449 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-01T01:21:09.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-01T01:21:09.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-01T01:21:09.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-01T01:21:09.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-01T01:21:09.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-01T01:21:09.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-01T01:21:09.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-01T01:21:09.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-01T01:21:09.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-01T01:21:09.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-01T01:21:09.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-01T01:21:09.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-01T01:21:09.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-01T01:21:09.450 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-01T01:21:09.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-04-01T01:21:09.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-01T01:21:09.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 153/198 2024-04-01T01:21:09.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-01T01:21:09.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-04-01T01:21:09.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-01T01:21:09.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-04-01T01:21:09.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-01T01:21:09.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-04-01T01:21:09.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-01T01:21:09.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-04-01T01:21:09.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-01T01:21:09.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-01T01:21:09.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-01T01:21:09.451 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-01T01:21:09.452 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-01T01:21:09.452 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-01T01:21:09.452 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-01T01:21:09.452 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-01T01:21:09.452 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-01T01:21:09.452 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-01T01:21:09.452 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-01T01:21:09.452 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-01T01:21:09.452 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-01T01:21:09.452 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-01T01:21:09.452 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-01T01:21:09.452 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-01T01:21:09.452 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-01T01:21:09.452 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-01T01:21:09.453 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-01T01:21:09.453 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-01T01:21:09.453 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-01T01:21:09.453 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-01T01:21:09.453 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-01T01:21:09.453 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-01T01:21:09.453 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-01T01:21:09.453 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-01T01:21:09.453 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-01T01:21:09.453 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-01T01:21:09.453 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-01T01:21:09.453 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-01T01:21:09.453 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-01T01:21:09.453 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-01T01:21:09.454 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-01T01:21:09.454 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-01T01:21:09.454 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-01T01:21:09.454 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-01T01:21:10.303 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-04-01T01:21:10.332 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-04-01T01:21:10.873 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 130/198 2024-04-01T01:21:10.911 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 131/198 2024-04-01T01:21:11.002 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 131/198 2024-04-01T01:21:11.002 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-01T01:21:11.002 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-01T01:21:11.002 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:21:11.037 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 132/198 2024-04-01T01:21:12.435 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-01T01:21:12.436 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:21:12.436 INFO:teuthology.orchestra.run.smithi070.stdout:Upgraded: 2024-04-01T01:21:12.436 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-04-01T01:21:12.436 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-04-01T01:21:12.436 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-04-01T01:21:12.436 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-04-01T01:21:12.436 INFO:teuthology.orchestra.run.smithi070.stdout: acl-2.3.1-4.el9.x86_64 2024-04-01T01:21:12.436 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-01T01:21:12.436 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-01T01:21:12.436 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-01T01:21:12.436 INFO:teuthology.orchestra.run.smithi070.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-01T01:21:12.436 INFO:teuthology.orchestra.run.smithi070.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-01T01:21:12.437 INFO:teuthology.orchestra.run.smithi070.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-01T01:21:12.437 INFO:teuthology.orchestra.run.smithi070.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-01T01:21:12.437 INFO:teuthology.orchestra.run.smithi070.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-01T01:21:12.437 INFO:teuthology.orchestra.run.smithi070.stdout: librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.437 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.437 INFO:teuthology.orchestra.run.smithi070.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-01T01:21:12.437 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-01T01:21:12.437 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-01T01:21:12.437 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-01T01:21:12.437 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-01T01:21:12.437 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-01T01:21:12.437 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-01T01:21:12.438 INFO:teuthology.orchestra.run.smithi070.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-01T01:21:12.438 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-01T01:21:12.438 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-01T01:21:12.438 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-01T01:21:12.438 INFO:teuthology.orchestra.run.smithi070.stdout: automake-1.16.2-8.el9.noarch 2024-04-01T01:21:12.438 INFO:teuthology.orchestra.run.smithi070.stdout: bison-3.7.4-5.el9.x86_64 2024-04-01T01:21:12.438 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-01T01:21:12.438 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.438 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.438 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.438 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.438 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:12.439 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.439 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.439 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.439 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:12.439 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:12.439 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:12.439 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:12.439 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:12.439 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.439 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.439 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:12.439 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.440 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.440 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.440 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:12.440 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:12.440 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.440 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.440 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:12.440 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-01T01:21:12.440 INFO:teuthology.orchestra.run.smithi070.stdout: flex-2.6.4-9.el9.x86_64 2024-04-01T01:21:12.440 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-01T01:21:12.440 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-01T01:21:12.440 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-01T01:21:12.440 INFO:teuthology.orchestra.run.smithi070.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-01T01:21:12.441 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-01T01:21:12.441 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-01T01:21:12.441 INFO:teuthology.orchestra.run.smithi070.stdout: jq-1.6-16.el9.x86_64 2024-04-01T01:21:12.441 INFO:teuthology.orchestra.run.smithi070.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-01T01:21:12.441 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-01T01:21:12.441 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-01T01:21:12.441 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.441 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.441 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.441 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-01T01:21:12.441 INFO:teuthology.orchestra.run.smithi070.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-01T01:21:12.441 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-01T01:21:12.441 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-01T01:21:12.441 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-01T01:21:12.442 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.442 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.442 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-01T01:21:12.442 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.442 INFO:teuthology.orchestra.run.smithi070.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-01T01:21:12.442 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-01T01:21:12.442 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-01T01:21:12.442 INFO:teuthology.orchestra.run.smithi070.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-01T01:21:12.442 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-01T01:21:12.442 INFO:teuthology.orchestra.run.smithi070.stdout: lua-5.4.4-4.el9.x86_64 2024-04-01T01:21:12.442 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-01T01:21:12.442 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-01T01:21:12.442 INFO:teuthology.orchestra.run.smithi070.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-01T01:21:12.443 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-01T01:21:12.443 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-01T01:21:12.443 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-01T01:21:12.443 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-01T01:21:12.443 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-13.14-1.el9.x86_64 2024-04-01T01:21:12.443 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-01T01:21:12.443 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-01T01:21:12.443 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-01T01:21:12.443 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-01T01:21:12.443 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-01T01:21:12.443 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-01T01:21:12.443 INFO:teuthology.orchestra.run.smithi070.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-01T01:21:12.443 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-01T01:21:12.443 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-01T01:21:12.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-01T01:21:12.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-01T01:21:12.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-01T01:21:12.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-01T01:21:12.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-01T01:21:12.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-01T01:21:12.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-01T01:21:12.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-01T01:21:12.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-01T01:21:12.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-01T01:21:12.444 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-01T01:21:12.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-01T01:21:12.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-01T01:21:12.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-01T01:21:12.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-01T01:21:12.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-01T01:21:12.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-01T01:21:12.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-01T01:21:12.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-01T01:21:12.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-01T01:21:12.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-01T01:21:12.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-01T01:21:12.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-01T01:21:12.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-01T01:21:12.445 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-01T01:21:12.446 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-01T01:21:12.446 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-01T01:21:12.446 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-01T01:21:12.446 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-01T01:21:12.446 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-01T01:21:12.446 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-01T01:21:12.447 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply-3.11-14.el9.noarch 2024-04-01T01:21:12.447 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-01T01:21:12.447 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-01T01:21:12.447 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-01T01:21:12.447 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-01T01:21:12.447 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-01T01:21:12.447 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-01T01:21:12.447 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-01T01:21:12.448 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-01T01:21:12.448 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-01T01:21:12.448 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.448 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.448 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-01T01:21:12.448 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-01T01:21:12.448 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-01T01:21:12.448 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.448 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-01T01:21:12.448 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-01T01:21:12.448 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-01T01:21:12.448 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-01T01:21:12.448 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-01T01:21:12.449 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-01T01:21:12.449 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-01T01:21:12.449 INFO:teuthology.orchestra.run.smithi070.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-01T01:21:12.449 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-01T01:21:12.449 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-01T01:21:12.449 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-01T01:21:12.449 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-01T01:21:12.449 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-01T01:21:12.449 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-01T01:21:12.449 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-01T01:21:12.449 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.449 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.449 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:12.449 INFO:teuthology.orchestra.run.smithi070.stdout: re2-1:20211101-3.el9.x86_64 2024-04-01T01:21:12.450 INFO:teuthology.orchestra.run.smithi070.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-01T01:21:12.450 INFO:teuthology.orchestra.run.smithi070.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-01T01:21:12.450 INFO:teuthology.orchestra.run.smithi070.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-01T01:21:12.450 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-01T01:21:12.450 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-01T01:21:12.450 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-01T01:21:12.450 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:21:12.450 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-01T01:21:12.894 DEBUG:teuthology.parallel:result is None 2024-04-01T01:21:13.122 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-01T01:21:13.122 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 1/198 2024-04-01T01:21:13.123 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 2/198 2024-04-01T01:21:13.123 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 3/198 2024-04-01T01:21:13.123 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 4/198 2024-04-01T01:21:13.123 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1977.g384f3 5/198 2024-04-01T01:21:13.123 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 6/198 2024-04-01T01:21:13.123 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 7/198 2024-04-01T01:21:13.123 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 8/198 2024-04-01T01:21:13.123 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 9/198 2024-04-01T01:21:13.123 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-04-01T01:21:13.123 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-04-01T01:21:13.123 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 12/198 2024-04-01T01:21:13.123 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 13/198 2024-04-01T01:21:13.123 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 14/198 2024-04-01T01:21:13.124 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_ 15/198 2024-04-01T01:21:13.124 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-04-01T01:21:13.124 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 17/198 2024-04-01T01:21:13.124 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_6 18/198 2024-04-01T01:21:13.124 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 19/198 2024-04-01T01:21:13.124 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 20/198 2024-04-01T01:21:13.124 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el 21/198 2024-04-01T01:21:13.124 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-common-2:19.0.0-1977.g384f31f3.el9. 22/198 2024-04-01T01:21:13.124 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_6 23/198 2024-04-01T01:21:13.124 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 24/198 2024-04-01T01:21:13.124 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 25/198 2024-04-01T01:21:13.124 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 26/198 2024-04-01T01:21:13.124 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 27/198 2024-04-01T01:21:13.124 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 28/198 2024-04-01T01:21:13.124 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 29/198 2024-04-01T01:21:13.141 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3. 30/198 2024-04-01T01:21:13.141 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 31/198 2024-04-01T01:21:13.141 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 32/198 2024-04-01T01:21:13.141 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 33/198 2024-04-01T01:21:13.141 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el 34/198 2024-04-01T01:21:13.141 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 35/198 2024-04-01T01:21:13.141 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.e 36/198 2024-04-01T01:21:13.141 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 37/198 2024-04-01T01:21:13.142 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 38/198 2024-04-01T01:21:13.142 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 39/198 2024-04-01T01:21:13.142 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-01T01:21:13.142 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-01T01:21:13.142 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-01T01:21:13.142 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-01T01:21:13.142 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-01T01:21:13.142 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-01T01:21:13.142 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-01T01:21:13.142 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-01T01:21:13.142 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-01T01:21:13.142 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-01T01:21:13.143 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-01T01:21:13.143 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-01T01:21:13.143 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-01T01:21:13.143 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-01T01:21:13.143 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-01T01:21:13.143 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-01T01:21:13.143 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-01T01:21:13.143 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-01T01:21:13.143 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-01T01:21:13.143 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-01T01:21:13.143 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-01T01:21:13.143 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-01T01:21:13.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-01T01:21:13.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-01T01:21:13.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-01T01:21:13.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-01T01:21:13.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-01T01:21:13.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-01T01:21:13.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-01T01:21:13.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-01T01:21:13.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-01T01:21:13.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-01T01:21:13.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-01T01:21:13.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-01T01:21:13.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-01T01:21:13.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-01T01:21:13.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-01T01:21:13.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-01T01:21:13.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-01T01:21:13.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-01T01:21:13.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-01T01:21:13.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-01T01:21:13.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-01T01:21:13.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-01T01:21:13.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-01T01:21:13.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-01T01:21:13.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-01T01:21:13.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-01T01:21:13.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-01T01:21:13.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-01T01:21:13.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-01T01:21:13.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-01T01:21:13.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-01T01:21:13.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-01T01:21:13.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-01T01:21:13.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-01T01:21:13.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-01T01:21:13.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-01T01:21:13.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-01T01:21:13.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-01T01:21:13.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-01T01:21:13.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-01T01:21:13.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-01T01:21:13.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-01T01:21:13.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-01T01:21:13.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-01T01:21:13.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-01T01:21:13.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-01T01:21:13.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-01T01:21:13.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-01T01:21:13.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-01T01:21:13.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-01T01:21:13.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-01T01:21:13.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-01T01:21:13.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-01T01:21:13.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-01T01:21:13.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-01T01:21:13.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-01T01:21:13.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-01T01:21:13.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-01T01:21:13.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-01T01:21:13.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-01T01:21:13.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-01T01:21:13.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-01T01:21:13.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-01T01:21:13.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-01T01:21:13.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-01T01:21:13.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-01T01:21:13.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-01T01:21:13.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-01T01:21:13.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-01T01:21:13.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-01T01:21:13.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-01T01:21:13.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-01T01:21:13.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-01T01:21:13.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-01T01:21:13.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-01T01:21:13.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-01T01:21:13.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-01T01:21:13.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-01T01:21:13.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-01T01:21:13.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-01T01:21:13.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-01T01:21:13.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-01T01:21:13.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-01T01:21:13.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-01T01:21:13.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-01T01:21:13.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-01T01:21:13.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-01T01:21:13.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-01T01:21:13.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-01T01:21:13.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-04-01T01:21:13.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-01T01:21:13.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 153/198 2024-04-01T01:21:13.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-01T01:21:13.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-04-01T01:21:13.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-01T01:21:13.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-04-01T01:21:13.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-01T01:21:13.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-04-01T01:21:13.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-01T01:21:13.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-04-01T01:21:13.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-01T01:21:13.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-01T01:21:13.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-01T01:21:13.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-01T01:21:13.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-01T01:21:13.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-01T01:21:13.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-01T01:21:13.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-01T01:21:13.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-01T01:21:13.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-01T01:21:13.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-01T01:21:13.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-01T01:21:13.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-01T01:21:13.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-01T01:21:13.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-01T01:21:13.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-01T01:21:13.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-01T01:21:13.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-01T01:21:13.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-01T01:21:13.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-01T01:21:13.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-01T01:21:13.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-01T01:21:13.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-01T01:21:13.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-01T01:21:13.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-01T01:21:13.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-01T01:21:13.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-01T01:21:13.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-01T01:21:13.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-01T01:21:13.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-01T01:21:13.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-01T01:21:13.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-01T01:21:13.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-01T01:21:13.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-01T01:21:13.154 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-01T01:21:13.154 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-01T01:21:15.970 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-01T01:21:15.970 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:21:15.971 INFO:teuthology.orchestra.run.smithi189.stdout:Upgraded: 2024-04-01T01:21:15.971 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-04-01T01:21:15.971 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-04-01T01:21:15.971 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-04-01T01:21:15.971 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-04-01T01:21:15.971 INFO:teuthology.orchestra.run.smithi189.stdout: acl-2.3.1-4.el9.x86_64 2024-04-01T01:21:15.971 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-01T01:21:15.971 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-01T01:21:15.971 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-01T01:21:15.971 INFO:teuthology.orchestra.run.smithi189.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-01T01:21:15.971 INFO:teuthology.orchestra.run.smithi189.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-01T01:21:15.971 INFO:teuthology.orchestra.run.smithi189.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-01T01:21:15.971 INFO:teuthology.orchestra.run.smithi189.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-01T01:21:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-01T01:21:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-01T01:21:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-01T01:21:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-01T01:21:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-01T01:21:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-01T01:21:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-01T01:21:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-01T01:21:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-01T01:21:15.972 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-01T01:21:15.972 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-01T01:21:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-01T01:21:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: automake-1.16.2-8.el9.noarch 2024-04-01T01:21:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: bison-3.7.4-5.el9.x86_64 2024-04-01T01:21:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-01T01:21:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:15.973 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.974 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-01T01:21:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: flex-2.6.4-9.el9.x86_64 2024-04-01T01:21:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-01T01:21:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-01T01:21:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-01T01:21:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-01T01:21:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-01T01:21:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-01T01:21:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: jq-1.6-16.el9.x86_64 2024-04-01T01:21:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-01T01:21:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-01T01:21:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-01T01:21:15.975 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-01T01:21:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-01T01:21:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-01T01:21:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-01T01:21:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-01T01:21:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-01T01:21:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-01T01:21:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-01T01:21:15.976 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-01T01:21:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-01T01:21:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-01T01:21:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: lua-5.4.4-4.el9.x86_64 2024-04-01T01:21:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-01T01:21:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-01T01:21:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-01T01:21:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-01T01:21:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-01T01:21:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-01T01:21:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-01T01:21:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-13.14-1.el9.x86_64 2024-04-01T01:21:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-01T01:21:15.977 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-01T01:21:15.978 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-01T01:21:15.978 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-01T01:21:15.978 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-01T01:21:15.978 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-01T01:21:15.978 INFO:teuthology.orchestra.run.smithi189.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-01T01:21:15.978 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-01T01:21:15.978 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-01T01:21:15.978 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-01T01:21:15.978 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.979 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.979 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.979 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-01T01:21:15.979 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-01T01:21:15.979 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-01T01:21:15.979 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-01T01:21:15.979 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-01T01:21:15.979 INFO:teuthology.orchestra.run.smithi189.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-01T01:21:15.979 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-01T01:21:15.979 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-01T01:21:15.979 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-01T01:21:15.980 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-01T01:21:15.980 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-01T01:21:15.980 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-01T01:21:15.980 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-01T01:21:15.980 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-01T01:21:15.980 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-01T01:21:15.980 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-01T01:21:15.980 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-01T01:21:15.980 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-01T01:21:15.980 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-01T01:21:15.980 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-01T01:21:15.980 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-01T01:21:15.980 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-01T01:21:15.980 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-01T01:21:15.980 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-01T01:21:15.981 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-01T01:21:15.981 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-01T01:21:15.981 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-01T01:21:15.981 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-01T01:21:15.981 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-01T01:21:15.981 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-01T01:21:15.981 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply-3.11-14.el9.noarch 2024-04-01T01:21:15.981 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-01T01:21:15.981 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-01T01:21:15.981 INFO:teuthology.orchestra.run.smithi189.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-01T01:21:15.981 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-01T01:21:15.981 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-01T01:21:15.981 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-01T01:21:15.981 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-01T01:21:15.982 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-01T01:21:15.982 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-01T01:21:15.982 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.982 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.982 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-01T01:21:15.982 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-01T01:21:15.982 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-01T01:21:15.982 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.982 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-01T01:21:15.982 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-01T01:21:15.982 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-01T01:21:15.982 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-01T01:21:15.982 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-01T01:21:15.983 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-01T01:21:15.983 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-01T01:21:15.983 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-01T01:21:15.983 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-01T01:21:15.983 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-01T01:21:15.983 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-01T01:21:15.983 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-01T01:21:15.983 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-01T01:21:15.983 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-01T01:21:15.983 INFO:teuthology.orchestra.run.smithi189.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-01T01:21:15.983 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.983 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.983 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:15.983 INFO:teuthology.orchestra.run.smithi189.stdout: re2-1:20211101-3.el9.x86_64 2024-04-01T01:21:15.984 INFO:teuthology.orchestra.run.smithi189.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-01T01:21:15.984 INFO:teuthology.orchestra.run.smithi189.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-01T01:21:15.984 INFO:teuthology.orchestra.run.smithi189.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-01T01:21:15.984 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-01T01:21:15.984 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-01T01:21:15.984 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-01T01:21:15.985 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:21:15.985 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-01T01:21:16.421 DEBUG:teuthology.parallel:result is None 2024-04-01T01:21:22.996 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 132/198 2024-04-01T01:21:22.996 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /sys 2024-04-01T01:21:22.996 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /proc 2024-04-01T01:21:22.996 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /mnt 2024-04-01T01:21:22.996 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /var/tmp 2024-04-01T01:21:22.996 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /home 2024-04-01T01:21:22.996 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /root 2024-04-01T01:21:22.996 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /tmp 2024-04-01T01:21:22.996 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:21:23.159 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 133/198 2024-04-01T01:21:23.175 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 133/198 2024-04-01T01:21:23.175 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:21:23.175 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-01T01:21:23.175 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:21:24.663 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 134/198 2024-04-01T01:21:24.688 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 134/198 2024-04-01T01:21:24.688 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:21:24.688 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-01T01:21:24.688 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-01T01:21:24.688 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-01T01:21:24.688 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:21:24.913 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 135/198 2024-04-01T01:21:24.938 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 135/198 2024-04-01T01:21:24.938 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:21:24.938 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-01T01:21:24.938 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-01T01:21:24.938 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-01T01:21:24.938 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:21:25.372 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 136/198 2024-04-01T01:21:25.394 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 136/198 2024-04-01T01:21:25.394 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:21:25.394 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-01T01:21:25.394 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-01T01:21:25.394 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-01T01:21:25.394 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:21:25.490 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-01T01:21:25.941 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 138/198 2024-04-01T01:21:25.972 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 138/198 2024-04-01T01:21:26.003 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.e 139/198 2024-04-01T01:21:26.065 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3. 140/198 2024-04-01T01:21:26.095 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 141/198 2024-04-01T01:21:27.131 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 141/198 2024-04-01T01:21:27.172 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 142/198 2024-04-01T01:21:28.132 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 142/198 2024-04-01T01:21:28.164 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 143/198 2024-04-01T01:21:28.263 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 143/198 2024-04-01T01:21:28.435 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el 144/198 2024-04-01T01:21:28.466 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 145/198 2024-04-01T01:21:28.488 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 145/198 2024-04-01T01:21:28.488 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:21:28.489 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-01T01:21:28.489 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-01T01:21:28.489 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-01T01:21:28.489 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:21:28.533 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 146/198 2024-04-01T01:21:28.544 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 146/198 2024-04-01T01:21:28.599 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 147/198 2024-04-01T01:21:29.452 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-01T01:21:29.477 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 149/198 2024-04-01T01:21:29.499 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 149/198 2024-04-01T01:21:29.499 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:21:29.499 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-01T01:21:29.499 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-01T01:21:29.500 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-01T01:21:29.500 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:21:29.553 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1977.g384f3 150/198 2024-04-01T01:21:29.574 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1977.g384f3 150/198 2024-04-01T01:21:29.574 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:21:29.575 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-01T01:21:29.575 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:21:29.620 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-04-01T01:21:29.642 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-04-01T01:21:29.642 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:21:29.642 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-01T01:21:29.642 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-01T01:21:29.643 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-01T01:21:29.643 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:21:29.954 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 152/198 2024-04-01T01:21:29.976 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 152/198 2024-04-01T01:21:29.977 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-01T01:21:29.977 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-01T01:21:29.977 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-01T01:21:29.977 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-01T01:21:29.977 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:21:34.417 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 153/198 2024-04-01T01:21:34.475 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 154/198 2024-04-01T01:21:34.505 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 155/198 2024-04-01T01:21:34.532 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-01T01:21:34.715 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 157/198 2024-04-01T01:21:34.823 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-01T01:21:34.864 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 159/198 2024-04-01T01:21:34.919 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-01T01:21:34.972 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-01T01:21:34.999 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-01T01:21:35.023 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_ 163/198 2024-04-01T01:21:35.072 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-01T01:21:35.101 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-04-01T01:21:35.166 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-04-01T01:21:35.241 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-01T01:21:35.277 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-01T01:21:35.824 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-01T01:21:35.867 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-01T01:21:35.886 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-01T01:21:36.703 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 170/198 2024-04-01T01:21:36.787 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-01T01:21:36.828 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-01T01:21:36.918 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-01T01:21:36.957 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-01T01:21:36.983 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-01T01:21:37.008 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-01T01:21:37.034 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-01T01:21:37.034 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-01T01:21:37.122 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-01T01:21:37.152 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-01T01:21:37.152 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-01T01:21:37.237 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-01T01:21:37.263 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-01T01:21:37.296 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-01T01:21:37.321 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-01T01:21:37.346 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-01T01:21:37.386 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-01T01:21:37.427 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-01T01:21:37.467 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-01T01:21:37.493 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-01T01:21:37.518 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-01T01:21:37.543 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-01T01:21:37.569 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-01T01:21:37.575 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-01T01:21:37.575 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-01T01:21:37.755 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-01T01:21:37.755 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-01T01:21:37.846 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-01T01:21:37.877 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-01T01:21:37.905 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-01T01:21:37.931 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-01T01:21:37.955 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-01T01:21:38.021 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-01T01:21:43.457 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-01T01:21:43.457 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 1/198 2024-04-01T01:21:43.457 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 2/198 2024-04-01T01:21:43.457 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 3/198 2024-04-01T01:21:43.457 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 4/198 2024-04-01T01:21:43.457 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1977.g384f3 5/198 2024-04-01T01:21:43.457 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 6/198 2024-04-01T01:21:43.458 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 7/198 2024-04-01T01:21:43.458 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 8/198 2024-04-01T01:21:43.458 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 9/198 2024-04-01T01:21:43.458 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 10/198 2024-04-01T01:21:43.458 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 11/198 2024-04-01T01:21:43.458 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 12/198 2024-04-01T01:21:43.458 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 13/198 2024-04-01T01:21:43.458 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 14/198 2024-04-01T01:21:43.458 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_ 15/198 2024-04-01T01:21:43.458 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 16/198 2024-04-01T01:21:43.458 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 17/198 2024-04-01T01:21:43.458 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_6 18/198 2024-04-01T01:21:43.458 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86 19/198 2024-04-01T01:21:43.459 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 20/198 2024-04-01T01:21:43.459 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el 21/198 2024-04-01T01:21:43.459 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-common-2:19.0.0-1977.g384f31f3.el9. 22/198 2024-04-01T01:21:43.459 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_6 23/198 2024-04-01T01:21:43.459 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 24/198 2024-04-01T01:21:43.459 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 25/198 2024-04-01T01:21:43.475 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 26/198 2024-04-01T01:21:43.475 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 27/198 2024-04-01T01:21:43.475 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 28/198 2024-04-01T01:21:43.475 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 29/198 2024-04-01T01:21:43.475 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3. 30/198 2024-04-01T01:21:43.475 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noa 31/198 2024-04-01T01:21:43.475 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.n 32/198 2024-04-01T01:21:43.476 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1977.g384 33/198 2024-04-01T01:21:43.476 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el 34/198 2024-04-01T01:21:43.476 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 35/198 2024-04-01T01:21:43.476 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.e 36/198 2024-04-01T01:21:43.476 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 37/198 2024-04-01T01:21:43.476 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 38/198 2024-04-01T01:21:43.476 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 39/198 2024-04-01T01:21:43.476 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-01T01:21:43.476 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-01T01:21:43.476 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-01T01:21:43.476 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-01T01:21:43.476 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-01T01:21:43.476 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-01T01:21:43.477 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-01T01:21:43.477 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-01T01:21:43.477 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-01T01:21:43.477 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-01T01:21:43.477 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-01T01:21:43.477 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-01T01:21:43.477 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-01T01:21:43.477 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-01T01:21:43.477 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-01T01:21:43.477 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-01T01:21:43.477 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-01T01:21:43.477 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-01T01:21:43.478 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-01T01:21:43.478 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-01T01:21:43.478 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-01T01:21:43.478 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-01T01:21:43.478 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-01T01:21:43.478 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-01T01:21:43.478 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-01T01:21:43.478 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-01T01:21:43.478 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-01T01:21:43.478 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-01T01:21:43.478 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-01T01:21:43.478 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-01T01:21:43.478 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-01T01:21:43.478 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-01T01:21:43.479 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-01T01:21:43.479 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-01T01:21:43.479 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-01T01:21:43.479 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-01T01:21:43.479 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-01T01:21:43.479 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-01T01:21:43.479 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-01T01:21:43.479 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-01T01:21:43.479 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-01T01:21:43.479 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-01T01:21:43.479 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-01T01:21:43.479 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-01T01:21:43.479 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-01T01:21:43.480 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-01T01:21:43.480 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-01T01:21:43.480 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-01T01:21:43.480 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-01T01:21:43.480 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-01T01:21:43.480 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-01T01:21:43.480 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-01T01:21:43.480 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-01T01:21:43.480 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-01T01:21:43.480 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-01T01:21:43.480 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-01T01:21:43.480 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-01T01:21:43.480 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-01T01:21:43.480 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-01T01:21:43.481 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-01T01:21:43.481 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-01T01:21:43.481 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-01T01:21:43.481 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-01T01:21:43.481 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-01T01:21:43.481 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-01T01:21:43.481 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-01T01:21:43.481 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-01T01:21:43.481 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-01T01:21:43.481 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-01T01:21:43.481 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-01T01:21:43.481 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-01T01:21:43.481 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-01T01:21:43.481 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-01T01:21:43.482 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-01T01:21:43.482 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-01T01:21:43.482 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-01T01:21:43.482 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-01T01:21:43.482 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-01T01:21:43.482 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-01T01:21:43.482 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-01T01:21:43.482 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-01T01:21:43.482 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-01T01:21:43.482 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-01T01:21:43.482 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-01T01:21:43.482 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-01T01:21:43.482 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-01T01:21:43.482 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-01T01:21:43.483 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-01T01:21:43.483 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-01T01:21:43.483 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-01T01:21:43.483 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-01T01:21:43.483 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-01T01:21:43.483 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-01T01:21:43.483 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-01T01:21:43.483 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-01T01:21:43.483 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-01T01:21:43.483 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-01T01:21:43.483 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-01T01:21:43.483 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-01T01:21:43.484 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-01T01:21:43.484 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-01T01:21:43.484 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-01T01:21:43.484 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-01T01:21:43.484 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-01T01:21:43.484 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-01T01:21:43.484 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-01T01:21:43.484 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-01T01:21:43.484 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-01T01:21:43.484 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-01T01:21:43.484 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-01T01:21:43.484 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-01T01:21:43.484 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 151/198 2024-04-01T01:21:43.484 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-01T01:21:43.484 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 153/198 2024-04-01T01:21:43.485 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-01T01:21:43.485 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 155/198 2024-04-01T01:21:43.485 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-01T01:21:43.485 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 157/198 2024-04-01T01:21:43.485 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-01T01:21:43.485 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 159/198 2024-04-01T01:21:43.485 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-01T01:21:43.485 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 161/198 2024-04-01T01:21:43.485 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-01T01:21:43.485 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-01T01:21:43.485 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-01T01:21:43.485 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-01T01:21:43.485 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-01T01:21:43.486 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-01T01:21:43.486 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-01T01:21:43.486 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-01T01:21:43.486 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-01T01:21:43.486 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-01T01:21:43.486 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-01T01:21:43.486 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-01T01:21:43.486 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-01T01:21:43.486 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-01T01:21:43.486 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-01T01:21:43.486 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-01T01:21:43.486 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-01T01:21:43.486 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-01T01:21:43.487 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-01T01:21:43.487 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-01T01:21:43.487 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-01T01:21:43.487 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-01T01:21:43.487 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-01T01:21:43.487 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-01T01:21:43.487 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-01T01:21:43.487 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-01T01:21:43.487 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-01T01:21:43.487 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-01T01:21:43.487 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-01T01:21:43.487 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-01T01:21:43.488 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-01T01:21:43.488 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-01T01:21:43.488 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-01T01:21:43.488 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-01T01:21:43.488 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-01T01:21:43.488 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-01T01:21:46.670 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-01T01:21:46.670 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:21:46.670 INFO:teuthology.orchestra.run.smithi081.stdout:Upgraded: 2024-04-01T01:21:46.670 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-04-01T01:21:46.670 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-04-01T01:21:46.670 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-04-01T01:21:46.670 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-04-01T01:21:46.670 INFO:teuthology.orchestra.run.smithi081.stdout: acl-2.3.1-4.el9.x86_64 2024-04-01T01:21:46.670 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-01T01:21:46.670 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-01T01:21:46.671 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-01T01:21:46.671 INFO:teuthology.orchestra.run.smithi081.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-01T01:21:46.671 INFO:teuthology.orchestra.run.smithi081.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-01T01:21:46.671 INFO:teuthology.orchestra.run.smithi081.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-01T01:21:46.671 INFO:teuthology.orchestra.run.smithi081.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-01T01:21:46.671 INFO:teuthology.orchestra.run.smithi081.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-01T01:21:46.671 INFO:teuthology.orchestra.run.smithi081.stdout: librados2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.671 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.671 INFO:teuthology.orchestra.run.smithi081.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-01T01:21:46.671 INFO:teuthology.orchestra.run.smithi081.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-01T01:21:46.671 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-01T01:21:46.671 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-01T01:21:46.672 INFO:teuthology.orchestra.run.smithi081.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-01T01:21:46.672 INFO:teuthology.orchestra.run.smithi081.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-01T01:21:46.672 INFO:teuthology.orchestra.run.smithi081.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-01T01:21:46.672 INFO:teuthology.orchestra.run.smithi081.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-01T01:21:46.672 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-01T01:21:46.672 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-04-01T01:21:46.672 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-01T01:21:46.672 INFO:teuthology.orchestra.run.smithi081.stdout: automake-1.16.2-8.el9.noarch 2024-04-01T01:21:46.672 INFO:teuthology.orchestra.run.smithi081.stdout: bison-3.7.4-5.el9.x86_64 2024-04-01T01:21:46.672 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-01T01:21:46.672 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.672 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.672 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.672 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.673 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:46.673 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.673 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.673 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.673 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:46.673 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:46.673 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:46.673 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:46.673 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:46.673 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.673 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.673 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:46.673 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.674 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.674 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.674 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:46.674 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:46.674 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.674 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-shell-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.674 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-top-2:19.0.0-1977.g384f31f3.el9.noarch 2024-04-01T01:21:46.674 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-01T01:21:46.674 INFO:teuthology.orchestra.run.smithi081.stdout: flex-2.6.4-9.el9.x86_64 2024-04-01T01:21:46.674 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-01T01:21:46.674 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-01T01:21:46.674 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-01T01:21:46.674 INFO:teuthology.orchestra.run.smithi081.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-01T01:21:46.675 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-01T01:21:46.675 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-01T01:21:46.675 INFO:teuthology.orchestra.run.smithi081.stdout: jq-1.6-16.el9.x86_64 2024-04-01T01:21:46.675 INFO:teuthology.orchestra.run.smithi081.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-01T01:21:46.675 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-01T01:21:46.675 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-01T01:21:46.675 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.675 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.675 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.675 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-01T01:21:46.675 INFO:teuthology.orchestra.run.smithi081.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-01T01:21:46.675 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-01T01:21:46.676 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-01T01:21:46.676 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-01T01:21:46.676 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.676 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.676 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-01T01:21:46.676 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.676 INFO:teuthology.orchestra.run.smithi081.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-01T01:21:46.676 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-01T01:21:46.676 INFO:teuthology.orchestra.run.smithi081.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-01T01:21:46.676 INFO:teuthology.orchestra.run.smithi081.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-01T01:21:46.676 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-01T01:21:46.676 INFO:teuthology.orchestra.run.smithi081.stdout: lua-5.4.4-4.el9.x86_64 2024-04-01T01:21:46.676 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-01T01:21:46.677 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-01T01:21:46.677 INFO:teuthology.orchestra.run.smithi081.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-01T01:21:46.677 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-01T01:21:46.677 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-01T01:21:46.677 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-01T01:21:46.677 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-01T01:21:46.677 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-13.14-1.el9.x86_64 2024-04-01T01:21:46.677 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-01T01:21:46.677 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-01T01:21:46.677 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-01T01:21:46.677 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-01T01:21:46.677 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-01T01:21:46.678 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-01T01:21:46.678 INFO:teuthology.orchestra.run.smithi081.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-01T01:21:46.678 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-01T01:21:46.678 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-01T01:21:46.678 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-01T01:21:46.678 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.678 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.678 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.678 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-01T01:21:46.678 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-01T01:21:46.678 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-01T01:21:46.678 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-01T01:21:46.679 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-01T01:21:46.679 INFO:teuthology.orchestra.run.smithi081.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-01T01:21:46.679 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-01T01:21:46.679 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-01T01:21:46.679 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-01T01:21:46.679 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-01T01:21:46.679 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-01T01:21:46.679 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-01T01:21:46.679 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-01T01:21:46.679 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-01T01:21:46.679 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-01T01:21:46.679 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-01T01:21:46.679 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-01T01:21:46.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-01T01:21:46.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-01T01:21:46.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-01T01:21:46.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-01T01:21:46.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-01T01:21:46.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-01T01:21:46.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-01T01:21:46.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-01T01:21:46.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-01T01:21:46.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-01T01:21:46.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-01T01:21:46.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-01T01:21:46.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-01T01:21:46.680 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply-3.11-14.el9.noarch 2024-04-01T01:21:46.681 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-01T01:21:46.681 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-01T01:21:46.681 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-01T01:21:46.681 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-01T01:21:46.681 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-01T01:21:46.681 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-01T01:21:46.681 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-01T01:21:46.681 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-01T01:21:46.681 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-01T01:21:46.681 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.681 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.681 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-01T01:21:46.681 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-01T01:21:46.682 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-01T01:21:46.682 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.682 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-01T01:21:46.682 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-01T01:21:46.682 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-01T01:21:46.682 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-01T01:21:46.682 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-01T01:21:46.682 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-01T01:21:46.682 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-01T01:21:46.682 INFO:teuthology.orchestra.run.smithi081.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-01T01:21:46.682 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-01T01:21:46.682 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-01T01:21:46.683 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-01T01:21:46.683 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-01T01:21:46.683 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-01T01:21:46.683 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-01T01:21:46.683 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-01T01:21:46.683 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.683 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.683 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd-2:19.0.0-1977.g384f31f3.el9.x86_64 2024-04-01T01:21:46.683 INFO:teuthology.orchestra.run.smithi081.stdout: re2-1:20211101-3.el9.x86_64 2024-04-01T01:21:46.683 INFO:teuthology.orchestra.run.smithi081.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-01T01:21:46.683 INFO:teuthology.orchestra.run.smithi081.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-01T01:21:46.683 INFO:teuthology.orchestra.run.smithi081.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-01T01:21:46.683 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-01T01:21:46.683 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-01T01:21:46.684 INFO:teuthology.orchestra.run.smithi081.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-01T01:21:46.684 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-01T01:21:46.684 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-04-01T01:21:47.115 DEBUG:teuthology.parallel:result is None 2024-04-01T01:21:47.115 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-04-01T01:21:47.335 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-01T01:21:47.362 INFO:teuthology.orchestra.run.smithi070.stdout:19.0.0-1977.g384f31f3.el9 2024-04-01T01:21:47.362 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1977.g384f31f3.el9 2024-04-01T01:21:47.363 INFO:teuthology.task.install:The correct ceph version 19.0.0-1977.g384f31f3 is installed. 2024-04-01T01:21:47.364 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-04-01T01:21:47.504 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-01T01:21:47.527 INFO:teuthology.orchestra.run.smithi081.stdout:19.0.0-1977.g384f31f3.el9 2024-04-01T01:21:47.527 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1977.g384f31f3.el9 2024-04-01T01:21:47.527 INFO:teuthology.task.install:The correct ceph version 19.0.0-1977.g384f31f3 is installed. 2024-04-01T01:21:47.529 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-04-01T01:21:47.821 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-01T01:21:47.847 INFO:teuthology.orchestra.run.smithi189.stdout:19.0.0-1977.g384f31f3.el9 2024-04-01T01:21:47.847 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1977.g384f31f3.el9 2024-04-01T01:21:47.847 INFO:teuthology.task.install:The correct ceph version 19.0.0-1977.g384f31f3 is installed. 2024-04-01T01:21:47.849 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-01T01:21:47.849 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:21:47.849 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-01T01:21:47.884 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-01T01:21:47.885 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-01T01:21:47.917 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-01T01:21:47.918 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-01T01:21:47.952 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-01T01:21:47.952 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:21:47.952 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/daemon-helper 2024-04-01T01:21:47.983 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-01T01:21:48.051 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-01T01:21:48.052 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/daemon-helper 2024-04-01T01:21:48.081 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-01T01:21:48.149 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-01T01:21:48.149 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2024-04-01T01:21:48.181 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-01T01:21:48.251 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-01T01:21:48.252 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:21:48.252 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-01T01:21:48.281 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-01T01:21:48.350 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-01T01:21:48.350 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-01T01:21:48.378 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-01T01:21:48.446 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-01T01:21:48.446 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-01T01:21:48.474 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-01T01:21:48.541 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-01T01:21:48.542 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:21:48.542 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/stdin-killer 2024-04-01T01:21:48.569 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-01T01:21:48.637 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-01T01:21:48.637 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/stdin-killer 2024-04-01T01:21:48.664 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-01T01:21:48.733 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-01T01:21:48.733 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/stdin-killer 2024-04-01T01:21:48.760 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-01T01:21:48.828 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-01T01:21:48.932 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-01T01:21:48.932 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 2024-04-01T01:21:48.932 INFO:tasks.cephadm:Cluster fsid is 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:21:48.933 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-01T01:21:48.933 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.70', 'mon.b': '172.21.15.81', 'mon.c': '172.21.15.189'} 2024-04-01T01:21:48.933 INFO:tasks.cephadm:First mon is mon.a on smithi070 2024-04-01T01:21:48.933 INFO:tasks.cephadm:First mgr is x 2024-04-01T01:21:48.933 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-01T01:21:48.933 DEBUG:teuthology.orchestra.run.smithi070:> sudo hostname $(hostname -s) 2024-04-01T01:21:48.962 DEBUG:teuthology.orchestra.run.smithi081:> sudo hostname $(hostname -s) 2024-04-01T01:21:48.991 DEBUG:teuthology.orchestra.run.smithi189:> sudo hostname $(hostname -s) 2024-04-01T01:21:49.019 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-01T01:21:49.019 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-04-01T01:21:49.159 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '384f31f3d3aee7f8ba3f60846b8844c049165583', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54774/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1977-g384f31f3', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1977.g384f31f3'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-31 17:40:06.063697', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-04-01T01:21:49.382 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 384f31f3d3aee7f8ba3f60846b8844c049165583 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=384f31f3d3aee7f8ba3f60846b8844c049165583 2024-04-01T01:21:49.384 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/x86_64/flavors/default/cephadm 2024-04-01T01:21:49.384 DEBUG:teuthology.orchestra.run.smithi070:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-01T01:21:49.837 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-r--r--. 1 ubuntu ubuntu 767490 Apr 1 01:21 /home/ubuntu/cephtest/cephadm 2024-04-01T01:21:49.837 DEBUG:teuthology.orchestra.run.smithi081:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-01T01:21:50.932 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r--. 1 ubuntu ubuntu 767490 Apr 1 01:21 /home/ubuntu/cephtest/cephadm 2024-04-01T01:21:50.932 DEBUG:teuthology.orchestra.run.smithi189:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/384f31f3d3aee7f8ba3f60846b8844c049165583/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-01T01:21:52.157 INFO:teuthology.orchestra.run.smithi189.stdout:-rw-r--r--. 1 ubuntu ubuntu 767490 Apr 1 01:21 /home/ubuntu/cephtest/cephadm 2024-04-01T01:21:52.158 DEBUG:teuthology.orchestra.run.smithi070:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-01T01:21:52.177 DEBUG:teuthology.orchestra.run.smithi081:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-01T01:21:52.196 DEBUG:teuthology.orchestra.run.smithi189:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-01T01:21:52.231 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 on all hosts... 2024-04-01T01:21:52.231 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 pull 2024-04-01T01:21:52.233 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 pull 2024-04-01T01:21:52.238 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 pull 2024-04-01T01:21:53.024 INFO:teuthology.orchestra.run.smithi081.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583... 2024-04-01T01:21:53.053 INFO:teuthology.orchestra.run.smithi070.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583... 2024-04-01T01:21:53.280 INFO:teuthology.orchestra.run.smithi189.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583... 2024-04-01T01:22:46.918 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-01T01:22:46.918 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_version": "ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev)", 2024-04-01T01:22:46.918 INFO:teuthology.orchestra.run.smithi189.stdout: "image_id": "ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791", 2024-04-01T01:22:46.918 INFO:teuthology.orchestra.run.smithi189.stdout: "repo_digests": [ 2024-04-01T01:22:46.919 INFO:teuthology.orchestra.run.smithi189.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175" 2024-04-01T01:22:46.919 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-01T01:22:46.919 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-01T01:23:11.445 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-01T01:23:11.445 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_version": "ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev)", 2024-04-01T01:23:11.446 INFO:teuthology.orchestra.run.smithi081.stdout: "image_id": "ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791", 2024-04-01T01:23:11.446 INFO:teuthology.orchestra.run.smithi081.stdout: "repo_digests": [ 2024-04-01T01:23:11.446 INFO:teuthology.orchestra.run.smithi081.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175" 2024-04-01T01:23:11.446 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-01T01:23:11.446 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-01T01:23:11.679 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:23:11.679 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev)", 2024-04-01T01:23:11.679 INFO:teuthology.orchestra.run.smithi070.stdout: "image_id": "ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791", 2024-04-01T01:23:11.679 INFO:teuthology.orchestra.run.smithi070.stdout: "repo_digests": [ 2024-04-01T01:23:11.679 INFO:teuthology.orchestra.run.smithi070.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175" 2024-04-01T01:23:11.679 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-01T01:23:11.679 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:23:11.704 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /etc/ceph 2024-04-01T01:23:11.764 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph 2024-04-01T01:23:11.795 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph 2024-04-01T01:23:11.830 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 777 /etc/ceph 2024-04-01T01:23:11.857 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 777 /etc/ceph 2024-04-01T01:23:11.885 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /etc/ceph 2024-04-01T01:23:11.915 INFO:tasks.cephadm:Writing seed config... 2024-04-01T01:23:11.916 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-01T01:23:11.916 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-01T01:23:11.916 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-01T01:23:11.917 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-01T01:23:11.917 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-01T01:23:11.917 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-01T01:23:11.917 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-01T01:23:11.917 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-01T01:23:11.917 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-01T01:23:11.917 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-01T01:23:11.917 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-01T01:23:11.917 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-01T01:23:11.917 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-01T01:23:11.917 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-01T01:23:11.917 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-01T01:23:11.917 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-01T01:23:11.917 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-01T01:23:11.918 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-01T01:23:11.918 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-01T01:23:11.918 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-01T01:23:11.918 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-01T01:23:11.918 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-01T01:23:11.918 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-01T01:23:11.918 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-01T01:23:11.918 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-01T01:23:11.918 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-01T01:23:11.918 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-01T01:23:11.918 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-01T01:23:11.918 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-01T01:23:11.919 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-01T01:23:11.919 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-01T01:23:11.919 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-01T01:23:11.919 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-01T01:23:11.919 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-01T01:23:11.919 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-01T01:23:11.919 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-01T01:23:11.919 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-01T01:23:11.919 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-01T01:23:11.919 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-01T01:23:11.919 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-01T01:23:11.919 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-01T01:23:11.919 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-01T01:23:11.919 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-01T01:23:11.919 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-04-01T01:23:11.920 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:23:11.920 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-01T01:23:11.935 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 36577394-efc6-11ee-b647-cb9ed24678a4 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-01T01:23:11.935 DEBUG:teuthology.orchestra.run.smithi070:mon.a> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mon.a.service 2024-04-01T01:23:11.977 DEBUG:teuthology.orchestra.run.smithi070:mgr.x> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mgr.x.service 2024-04-01T01:23:12.021 INFO:tasks.cephadm:Bootstrapping... 2024-04-01T01:23:12.021 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 -v bootstrap --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.70 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-01T01:23:12.212 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-01T01:23:12.212 INFO:teuthology.orchestra.run.smithi070.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583', '-v', 'bootstrap', '--fsid', '36577394-efc6-11ee-b647-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.70', '--skip-admin-label'] 2024-04-01T01:23:12.213 INFO:teuthology.orchestra.run.smithi070.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-01T01:23:12.213 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying podman|docker is present... 2024-04-01T01:23:12.239 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-01T01:23:12.239 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying lvm2 is present... 2024-04-01T01:23:12.240 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying time synchronization is in place... 2024-04-01T01:23:12.247 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-01T01:23:12.247 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-01T01:23:12.253 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-01T01:23:12.253 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-04-01T01:23:12.259 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout enabled 2024-04-01T01:23:12.265 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout active 2024-04-01T01:23:12.265 INFO:teuthology.orchestra.run.smithi070.stdout:Unit chronyd.service is enabled and running 2024-04-01T01:23:12.265 INFO:teuthology.orchestra.run.smithi070.stdout:Repeating the final host check... 2024-04-01T01:23:12.288 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-01T01:23:12.288 INFO:teuthology.orchestra.run.smithi070.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-01T01:23:12.288 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl is present 2024-04-01T01:23:12.288 INFO:teuthology.orchestra.run.smithi070.stdout:lvcreate is present 2024-04-01T01:23:12.294 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-01T01:23:12.294 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-01T01:23:12.300 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-01T01:23:12.300 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-04-01T01:23:12.306 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout enabled 2024-04-01T01:23:12.312 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout active 2024-04-01T01:23:12.312 INFO:teuthology.orchestra.run.smithi070.stdout:Unit chronyd.service is enabled and running 2024-04-01T01:23:12.312 INFO:teuthology.orchestra.run.smithi070.stdout:Host looks OK 2024-04-01T01:23:12.312 INFO:teuthology.orchestra.run.smithi070.stdout:Cluster fsid: 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:23:12.312 INFO:teuthology.orchestra.run.smithi070.stdout:Acquiring lock 140524225563712 on /run/cephadm/36577394-efc6-11ee-b647-cb9ed24678a4.lock 2024-04-01T01:23:12.312 INFO:teuthology.orchestra.run.smithi070.stdout:Lock 140524225563712 acquired on /run/cephadm/36577394-efc6-11ee-b647-cb9ed24678a4.lock 2024-04-01T01:23:12.313 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying IP 172.21.15.70 port 3300 ... 2024-04-01T01:23:12.313 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying IP 172.21.15.70 port 6789 ... 2024-04-01T01:23:12.313 INFO:teuthology.orchestra.run.smithi070.stdout:Base mon IP(s) is [172.21.15.70:3300, 172.21.15.70:6789], mon addrv is [v2:172.21.15.70:3300,v1:172.21.15.70:6789] 2024-04-01T01:23:12.316 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.70 metric 100 2024-04-01T01:23:12.316 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.70 metric 100 2024-04-01T01:23:12.319 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-01T01:23:12.319 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-01T01:23:12.319 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1761sec hoplimit 64 pref medium 2024-04-01T01:23:12.321 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-01T01:23:12.321 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-01T01:23:12.321 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-01T01:23:12.321 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-01T01:23:12.321 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-01T01:23:12.321 INFO:teuthology.orchestra.run.smithi070.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-01T01:23:12.322 INFO:teuthology.orchestra.run.smithi070.stdout:Mon IP `172.21.15.70` is in CIDR network `172.21.0.0/20` 2024-04-01T01:23:12.322 INFO:teuthology.orchestra.run.smithi070.stdout:Mon IP `172.21.15.70` is in CIDR network `172.21.0.0/20` 2024-04-01T01:23:12.322 INFO:teuthology.orchestra.run.smithi070.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-01T01:23:12.322 INFO:teuthology.orchestra.run.smithi070.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-01T01:23:12.324 INFO:teuthology.orchestra.run.smithi070.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583... 2024-04-01T01:23:12.798 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stdout ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 2024-04-01T01:23:12.798 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583... 2024-04-01T01:23:12.799 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Getting image source signatures 2024-04-01T01:23:12.799 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Copying blob sha256:bbb14f026003c3be33a985e165b4ecff104c8c0e04a9fce5f3c6b6743271902a 2024-04-01T01:23:12.799 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Copying blob sha256:11da15500eca1f826c328eb3265caa30b0a3730510e2a32023084d6fe9cee20c 2024-04-01T01:23:12.799 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Copying config sha256:ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 2024-04-01T01:23:12.799 INFO:teuthology.orchestra.run.smithi070.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-01T01:23:13.719 INFO:teuthology.orchestra.run.smithi070.stdout:ceph: stdout ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev) 2024-04-01T01:23:13.720 INFO:teuthology.orchestra.run.smithi070.stdout:Ceph version: ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev) 2024-04-01T01:23:13.720 INFO:teuthology.orchestra.run.smithi070.stdout:Extracting ceph user uid/gid from container image... 2024-04-01T01:23:14.609 INFO:teuthology.orchestra.run.smithi070.stdout:stat: stdout 167 167 2024-04-01T01:23:14.609 INFO:teuthology.orchestra.run.smithi070.stdout:Creating initial keys... 2024-04-01T01:23:15.523 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph-authtool: stdout AQCCDApmDvBSNxAAVJm3YG6+91MCJouJnfUOdQ== 2024-04-01T01:23:16.570 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph-authtool: stdout AQCDDApmkZktNBAA2kmgbGQe45EFPx40E3pF6w== 2024-04-01T01:23:17.426 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph-authtool: stdout AQCEDApmn6QMNhAAFmWJ/JZVyLLNkSBmmd66BA== 2024-04-01T01:23:17.427 INFO:teuthology.orchestra.run.smithi070.stdout:Creating initial monmap... 2024-04-01T01:23:18.341 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-01T01:23:18.341 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-01T01:23:18.341 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:23:18.341 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-01T01:23:18.341 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool for a [v2:172.21.15.70:3300,v1:172.21.15.70:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-01T01:23:18.341 INFO:teuthology.orchestra.run.smithi070.stdout:setting min_mon_release = quincy 2024-04-01T01:23:18.341 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: set fsid to 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:23:18.342 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-01T01:23:18.342 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:23:18.342 INFO:teuthology.orchestra.run.smithi070.stdout:Creating mon... 2024-04-01T01:23:19.255 INFO:teuthology.orchestra.run.smithi070.stdout:create mon.a on 2024-04-01T01:23:19.465 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-01T01:23:19.615 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-01T01:23:19.793 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-36577394-efc6-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-36577394-efc6-11ee-b647-cb9ed24678a4.target. 2024-04-01T01:23:19.793 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-36577394-efc6-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-36577394-efc6-11ee-b647-cb9ed24678a4.target. 2024-04-01T01:23:20.031 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mon.a 2024-04-01T01:23:20.031 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Failed to reset failed state of unit ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mon.a.service: Unit ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mon.a.service not loaded. 2024-04-01T01:23:20.193 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-36577394-efc6-11ee-b647-cb9ed24678a4.target.wants/ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mon.a.service → /etc/systemd/system/ceph-36577394-efc6-11ee-b647-cb9ed24678a4@.service. 2024-04-01T01:23:20.732 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-01T01:23:20.732 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout disabled 2024-04-01T01:23:20.738 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-01T01:23:20.738 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-04-01T01:23:20.739 INFO:teuthology.orchestra.run.smithi070.stdout:firewalld.service is not enabled 2024-04-01T01:23:20.739 INFO:teuthology.orchestra.run.smithi070.stdout:Not possible to enable service . firewalld.service is not available 2024-04-01T01:23:20.739 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mon to start... 2024-04-01T01:23:20.739 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mon... 2024-04-01T01:23:21.807 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout cluster: 2024-04-01T01:23:21.808 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout id: 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:23:21.808 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-01T01:23:21.808 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-04-01T01:23:21.808 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout services: 2024-04-01T01:23:21.808 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.540744s) 2024-04-01T01:23:21.808 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-01T01:23:21.808 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-01T01:23:21.808 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-04-01T01:23:21.808 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout data: 2024-04-01T01:23:21.808 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-01T01:23:21.808 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-01T01:23:21.808 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-01T01:23:21.808 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout pgs: 2024-04-01T01:23:21.809 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-04-01T01:23:21.809 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.158+0000 7f10f705c700 1 Processor -- start 2024-04-01T01:23:21.809 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.158+0000 7f10f705c700 1 -- start start 2024-04-01T01:23:21.809 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.158+0000 7f10f705c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10f0107ab0 0x7f10f0107ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:21.809 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.158+0000 7f10f705c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f10f01084a0 con 0x7f10f0107ab0 2024-04-01T01:23:21.809 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.159+0000 7f10f4df8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10f0107ab0 0x7f10f0107ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:21.809 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.159+0000 7f10f4df8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10f0107ab0 0x7f10f0107ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46434/0 (socket says 172.21.15.70:46434) 2024-04-01T01:23:21.809 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.159+0000 7f10f4df8700 1 -- 172.21.15.70:0/3430657026 learned_addr learned my addr 172.21.15.70:0/3430657026 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:21.809 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.159+0000 7f10f4df8700 1 -- 172.21.15.70:0/3430657026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f0108cb0 con 0x7f10f0107ab0 2024-04-01T01:23:21.809 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.159+0000 7f10f4df8700 1 --2- 172.21.15.70:0/3430657026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10f0107ab0 0x7f10f0107ed0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f10ec0098a0 tx=0x7f10ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54776f4688789b1e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:21.809 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.159+0000 7f10e77fe700 1 -- 172.21.15.70:0/3430657026 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10ec004160 con 0x7f10f0107ab0 2024-04-01T01:23:21.809 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.159+0000 7f10e77fe700 1 -- 172.21.15.70:0/3430657026 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10ec004300 con 0x7f10f0107ab0 2024-04-01T01:23:21.809 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.159+0000 7f10e77fe700 1 -- 172.21.15.70:0/3430657026 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10ec0036a0 con 0x7f10f0107ab0 2024-04-01T01:23:21.810 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.160+0000 7f10f705c700 1 -- 172.21.15.70:0/3430657026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10f0107ab0 msgr2=0x7f10f0107ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:21.810 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.160+0000 7f10f705c700 1 --2- 172.21.15.70:0/3430657026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10f0107ab0 0x7f10f0107ed0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f10ec0098a0 tx=0x7f10ec009bb0 comp rx=0 tx=0).stop 2024-04-01T01:23:21.810 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.160+0000 7f10f705c700 1 -- 172.21.15.70:0/3430657026 shutdown_connections 2024-04-01T01:23:21.810 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.160+0000 7f10f705c700 1 --2- 172.21.15.70:0/3430657026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10f0107ab0 0x7f10f0107ed0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:21.810 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.160+0000 7f10f705c700 1 -- 172.21.15.70:0/3430657026 >> 172.21.15.70:0/3430657026 conn(0x7f10f01030d0 msgr2=0x7f10f01054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:21.810 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.160+0000 7f10f705c700 1 -- 172.21.15.70:0/3430657026 shutdown_connections 2024-04-01T01:23:21.810 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.160+0000 7f10f705c700 1 -- 172.21.15.70:0/3430657026 wait complete. 2024-04-01T01:23:21.810 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.161+0000 7f10f705c700 1 Processor -- start 2024-04-01T01:23:21.810 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.161+0000 7f10f705c700 1 -- start start 2024-04-01T01:23:21.810 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.161+0000 7f10f705c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10f0107ab0 0x7f10f019aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:21.810 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.161+0000 7f10f705c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f10f01091e0 con 0x7f10f0107ab0 2024-04-01T01:23:21.810 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.161+0000 7f10f4df8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10f0107ab0 0x7f10f019aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:21.810 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.161+0000 7f10f4df8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10f0107ab0 0x7f10f019aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46444/0 (socket says 172.21.15.70:46444) 2024-04-01T01:23:21.811 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.161+0000 7f10f4df8700 1 -- 172.21.15.70:0/2490492500 learned_addr learned my addr 172.21.15.70:0/2490492500 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:21.811 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.161+0000 7f10f4df8700 1 -- 172.21.15.70:0/2490492500 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f019b530 con 0x7f10f0107ab0 2024-04-01T01:23:21.811 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.162+0000 7f10f4df8700 1 --2- 172.21.15.70:0/2490492500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10f0107ab0 0x7f10f019aff0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f10ec000c00 tx=0x7f10ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:21.811 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.162+0000 7f10e5ffb700 1 -- 172.21.15.70:0/2490492500 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10ec003db0 con 0x7f10f0107ab0 2024-04-01T01:23:21.811 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.162+0000 7f10e5ffb700 1 -- 172.21.15.70:0/2490492500 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10ec011630 con 0x7f10f0107ab0 2024-04-01T01:23:21.811 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.162+0000 7f10f705c700 1 -- 172.21.15.70:0/2490492500 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f019b7c0 con 0x7f10f0107ab0 2024-04-01T01:23:21.811 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.162+0000 7f10e5ffb700 1 -- 172.21.15.70:0/2490492500 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10ec011950 con 0x7f10f0107ab0 2024-04-01T01:23:21.811 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.162+0000 7f10f705c700 1 -- 172.21.15.70:0/2490492500 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f019e4b0 con 0x7f10f0107ab0 2024-04-01T01:23:21.811 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.162+0000 7f10e5ffb700 1 -- 172.21.15.70:0/2490492500 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f10ec011af0 con 0x7f10f0107ab0 2024-04-01T01:23:21.811 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.162+0000 7f10e5ffb700 1 -- 172.21.15.70:0/2490492500 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f10ec01c450 con 0x7f10f0107ab0 2024-04-01T01:23:21.811 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.163+0000 7f10f705c700 1 -- 172.21.15.70:0/2490492500 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10f0063870 con 0x7f10f0107ab0 2024-04-01T01:23:21.811 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.165+0000 7f10e5ffb700 1 -- 172.21.15.70:0/2490492500 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f10ec032020 con 0x7f10f0107ab0 2024-04-01T01:23:21.811 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.235+0000 7f10f705c700 1 -- 172.21.15.70:0/2490492500 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f10f002dcb0 con 0x7f10f0107ab0 2024-04-01T01:23:21.811 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.235+0000 7f10e5ffb700 1 -- 172.21.15.70:0/2490492500 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f10ec004260 con 0x7f10f0107ab0 2024-04-01T01:23:21.812 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.236+0000 7f10f705c700 1 -- 172.21.15.70:0/2490492500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10f0107ab0 msgr2=0x7f10f019aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:21.812 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.236+0000 7f10f705c700 1 --2- 172.21.15.70:0/2490492500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10f0107ab0 0x7f10f019aff0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f10ec000c00 tx=0x7f10ec004750 comp rx=0 tx=0).stop 2024-04-01T01:23:21.812 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.236+0000 7f10f705c700 1 -- 172.21.15.70:0/2490492500 shutdown_connections 2024-04-01T01:23:21.812 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.236+0000 7f10f705c700 1 --2- 172.21.15.70:0/2490492500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10f0107ab0 0x7f10f019aff0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:21.812 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.236+0000 7f10f705c700 1 -- 172.21.15.70:0/2490492500 >> 172.21.15.70:0/2490492500 conn(0x7f10f01030d0 msgr2=0x7f10f0191cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:21.812 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.236+0000 7f10f705c700 1 -- 172.21.15.70:0/2490492500 shutdown_connections 2024-04-01T01:23:21.812 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:21.236+0000 7f10f705c700 1 -- 172.21.15.70:0/2490492500 wait complete. 2024-04-01T01:23:21.812 INFO:teuthology.orchestra.run.smithi070.stdout:mon is available 2024-04-01T01:23:21.812 INFO:teuthology.orchestra.run.smithi070.stdout:Assimilating anything we can from ceph.conf... 2024-04-01T01:23:22.846 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-04-01T01:23:22.847 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [global] 2024-04-01T01:23:22.847 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout fsid = 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:23:22.847 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.70:3300,v1:172.21.15.70:6789] 2024-04-01T01:23:22.847 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-01T01:23:22.847 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-01T01:23:22.847 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-01T01:23:22.847 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-01T01:23:22.847 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-04-01T01:23:22.847 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-01T01:23:22.847 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-01T01:23:22.847 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-04-01T01:23:22.847 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [osd] 2024-04-01T01:23:22.848 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-01T01:23:22.848 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-01T01:23:22.848 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-01T01:23:22.848 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-01T01:23:22.848 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-01T01:23:22.848 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.216+0000 7efef6815700 1 Processor -- start 2024-04-01T01:23:22.848 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.216+0000 7efef6815700 1 -- start start 2024-04-01T01:23:22.848 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.216+0000 7efef6815700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efef0106920 0x7efef0106d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:22.848 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.216+0000 7efef6815700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efef0107310 con 0x7efef0106920 2024-04-01T01:23:22.848 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.216+0000 7efeeffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efef0106920 0x7efef0106d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:22.848 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.216+0000 7efeeffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efef0106920 0x7efef0106d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46460/0 (socket says 172.21.15.70:46460) 2024-04-01T01:23:22.848 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.216+0000 7efeeffff700 1 -- 172.21.15.70:0/481631821 learned_addr learned my addr 172.21.15.70:0/481631821 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:22.848 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.216+0000 7efeeffff700 1 -- 172.21.15.70:0/481631821 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef0107b20 con 0x7efef0106920 2024-04-01T01:23:22.848 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.217+0000 7efeeffff700 1 --2- 172.21.15.70:0/481631821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efef0106920 0x7efef0106d40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7efee00098a0 tx=0x7efee0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d84ea740c67d04c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:22.849 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.217+0000 7efeeeffd700 1 -- 172.21.15.70:0/481631821 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efee0004160 con 0x7efef0106920 2024-04-01T01:23:22.849 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.217+0000 7efeeeffd700 1 -- 172.21.15.70:0/481631821 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee0004300 con 0x7efef0106920 2024-04-01T01:23:22.849 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.217+0000 7efeeeffd700 1 -- 172.21.15.70:0/481631821 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efee00036a0 con 0x7efef0106920 2024-04-01T01:23:22.849 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.217+0000 7efef6815700 1 -- 172.21.15.70:0/481631821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efef0106920 msgr2=0x7efef0106d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:22.849 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.217+0000 7efef6815700 1 --2- 172.21.15.70:0/481631821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efef0106920 0x7efef0106d40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7efee00098a0 tx=0x7efee0009bb0 comp rx=0 tx=0).stop 2024-04-01T01:23:22.849 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.217+0000 7efef6815700 1 -- 172.21.15.70:0/481631821 shutdown_connections 2024-04-01T01:23:22.849 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.217+0000 7efef6815700 1 --2- 172.21.15.70:0/481631821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efef0106920 0x7efef0106d40 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:22.849 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.217+0000 7efef6815700 1 -- 172.21.15.70:0/481631821 >> 172.21.15.70:0/481631821 conn(0x7efef0101ea0 msgr2=0x7efef0104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:22.849 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.217+0000 7efef6815700 1 -- 172.21.15.70:0/481631821 shutdown_connections 2024-04-01T01:23:22.849 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.217+0000 7efef6815700 1 -- 172.21.15.70:0/481631821 wait complete. 2024-04-01T01:23:22.849 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.218+0000 7efef6815700 1 Processor -- start 2024-04-01T01:23:22.849 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.218+0000 7efef6815700 1 -- start start 2024-04-01T01:23:22.849 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.218+0000 7efef6815700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efef0106920 0x7efef007ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:22.850 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.218+0000 7efef6815700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efef0108050 con 0x7efef0106920 2024-04-01T01:23:22.850 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.218+0000 7efeeffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efef0106920 0x7efef007ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:22.850 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.218+0000 7efeeffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efef0106920 0x7efef007ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46464/0 (socket says 172.21.15.70:46464) 2024-04-01T01:23:22.850 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.219+0000 7efeeffff700 1 -- 172.21.15.70:0/1519583189 learned_addr learned my addr 172.21.15.70:0/1519583189 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:22.850 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.219+0000 7efeeffff700 1 -- 172.21.15.70:0/1519583189 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef019dde0 con 0x7efef0106920 2024-04-01T01:23:22.850 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.219+0000 7efeeffff700 1 --2- 172.21.15.70:0/1519583189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efef0106920 0x7efef007ba40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7efee0000c00 tx=0x7efee0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:22.850 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.219+0000 7efeed7fa700 1 -- 172.21.15.70:0/1519583189 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efee0003db0 con 0x7efef0106920 2024-04-01T01:23:22.850 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.219+0000 7efeed7fa700 1 -- 172.21.15.70:0/1519583189 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee0011630 con 0x7efef0106920 2024-04-01T01:23:22.850 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.219+0000 7efef6815700 1 -- 172.21.15.70:0/1519583189 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef019dfb0 con 0x7efef0106920 2024-04-01T01:23:22.850 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.219+0000 7efeed7fa700 1 -- 172.21.15.70:0/1519583189 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efee0011950 con 0x7efef0106920 2024-04-01T01:23:22.850 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.219+0000 7efef6815700 1 -- 172.21.15.70:0/1519583189 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef019e350 con 0x7efef0106920 2024-04-01T01:23:22.850 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.219+0000 7efeed7fa700 1 -- 172.21.15.70:0/1519583189 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efee001a460 con 0x7efef0106920 2024-04-01T01:23:22.850 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.219+0000 7efeed7fa700 1 -- 172.21.15.70:0/1519583189 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efee001b780 con 0x7efef0106920 2024-04-01T01:23:22.851 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.220+0000 7efef6815700 1 -- 172.21.15.70:0/1519583189 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efef0063870 con 0x7efef0106920 2024-04-01T01:23:22.851 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.222+0000 7efeed7fa700 1 -- 172.21.15.70:0/1519583189 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7efee0031020 con 0x7efef0106920 2024-04-01T01:23:22.851 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.289+0000 7efef6815700 1 -- 172.21.15.70:0/1519583189 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7efef002d180 con 0x7efef0106920 2024-04-01T01:23:22.851 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.292+0000 7efeed7fa700 1 -- 172.21.15.70:0/1519583189 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7efee001b9d0 con 0x7efef0106920 2024-04-01T01:23:22.851 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.293+0000 7efeed7fa700 1 -- 172.21.15.70:0/1519583189 <== mon.0 v2:172.21.15.70:3300/0 8 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7efee001ad70 con 0x7efef0106920 2024-04-01T01:23:22.851 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.294+0000 7efef6815700 1 -- 172.21.15.70:0/1519583189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efef0106920 msgr2=0x7efef007ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:22.851 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.294+0000 7efef6815700 1 --2- 172.21.15.70:0/1519583189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efef0106920 0x7efef007ba40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7efee0000c00 tx=0x7efee0004750 comp rx=0 tx=0).stop 2024-04-01T01:23:22.851 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.294+0000 7efef6815700 1 -- 172.21.15.70:0/1519583189 shutdown_connections 2024-04-01T01:23:22.851 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.294+0000 7efef6815700 1 --2- 172.21.15.70:0/1519583189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efef0106920 0x7efef007ba40 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:22.851 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.294+0000 7efef6815700 1 -- 172.21.15.70:0/1519583189 >> 172.21.15.70:0/1519583189 conn(0x7efef0101ea0 msgr2=0x7efef0190c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:22.851 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.294+0000 7efef6815700 1 -- 172.21.15.70:0/1519583189 shutdown_connections 2024-04-01T01:23:22.851 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:22.294+0000 7efef6815700 1 -- 172.21.15.70:0/1519583189 wait complete. 2024-04-01T01:23:22.851 INFO:teuthology.orchestra.run.smithi070.stdout:Generating new minimal ceph.conf... 2024-04-01T01:23:23.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.261+0000 7f6d72bdb700 1 Processor -- start 2024-04-01T01:23:23.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.261+0000 7f6d72bdb700 1 -- start start 2024-04-01T01:23:23.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.262+0000 7f6d72bdb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d6c107a50 0x7f6d6c107e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:23.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.262+0000 7f6d72bdb700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c108440 con 0x7f6d6c107a50 2024-04-01T01:23:23.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.262+0000 7f6d70977700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d6c107a50 0x7f6d6c107e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:23.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.262+0000 7f6d70977700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d6c107a50 0x7f6d6c107e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46476/0 (socket says 172.21.15.70:46476) 2024-04-01T01:23:23.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.262+0000 7f6d70977700 1 -- 172.21.15.70:0/1660986773 learned_addr learned my addr 172.21.15.70:0/1660986773 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:23.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.262+0000 7f6d70977700 1 -- 172.21.15.70:0/1660986773 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d6c108c50 con 0x7f6d6c107a50 2024-04-01T01:23:23.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.262+0000 7f6d70977700 1 --2- 172.21.15.70:0/1660986773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d6c107a50 0x7f6d6c107e70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6d600098a0 tx=0x7f6d60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f871fd7825b80e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:23.967 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.262+0000 7f6d6b7fe700 1 -- 172.21.15.70:0/1660986773 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d60004160 con 0x7f6d6c107a50 2024-04-01T01:23:23.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.263+0000 7f6d6b7fe700 1 -- 172.21.15.70:0/1660986773 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6d60004300 con 0x7f6d6c107a50 2024-04-01T01:23:23.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.263+0000 7f6d6b7fe700 1 -- 172.21.15.70:0/1660986773 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d60011a80 con 0x7f6d6c107a50 2024-04-01T01:23:23.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.263+0000 7f6d72bdb700 1 -- 172.21.15.70:0/1660986773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d6c107a50 msgr2=0x7f6d6c107e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:23.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.263+0000 7f6d72bdb700 1 --2- 172.21.15.70:0/1660986773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d6c107a50 0x7f6d6c107e70 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6d600098a0 tx=0x7f6d60009bb0 comp rx=0 tx=0).stop 2024-04-01T01:23:23.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.263+0000 7f6d72bdb700 1 -- 172.21.15.70:0/1660986773 shutdown_connections 2024-04-01T01:23:23.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.263+0000 7f6d72bdb700 1 --2- 172.21.15.70:0/1660986773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d6c107a50 0x7f6d6c107e70 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:23.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.263+0000 7f6d72bdb700 1 -- 172.21.15.70:0/1660986773 >> 172.21.15.70:0/1660986773 conn(0x7f6d6c102ff0 msgr2=0x7f6d6c105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:23.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.263+0000 7f6d72bdb700 1 -- 172.21.15.70:0/1660986773 shutdown_connections 2024-04-01T01:23:23.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.263+0000 7f6d72bdb700 1 -- 172.21.15.70:0/1660986773 wait complete. 2024-04-01T01:23:23.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.264+0000 7f6d72bdb700 1 Processor -- start 2024-04-01T01:23:23.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.264+0000 7f6d72bdb700 1 -- start start 2024-04-01T01:23:23.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.264+0000 7f6d72bdb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d6c107a50 0x7f6d6c07db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:23.968 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.264+0000 7f6d72bdb700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c10c000 con 0x7f6d6c107a50 2024-04-01T01:23:23.969 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.264+0000 7f6d70977700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d6c107a50 0x7f6d6c07db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:23.969 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.264+0000 7f6d70977700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d6c107a50 0x7f6d6c07db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46478/0 (socket says 172.21.15.70:46478) 2024-04-01T01:23:23.969 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.264+0000 7f6d70977700 1 -- 172.21.15.70:0/2568130616 learned_addr learned my addr 172.21.15.70:0/2568130616 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:23.969 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.264+0000 7f6d70977700 1 -- 172.21.15.70:0/2568130616 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d6c07e0a0 con 0x7f6d6c107a50 2024-04-01T01:23:23.969 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.264+0000 7f6d70977700 1 --2- 172.21.15.70:0/2568130616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d6c107a50 0x7f6d6c07db60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6d60000c00 tx=0x7f6d60011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:23.969 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.265+0000 7f6d69ffb700 1 -- 172.21.15.70:0/2568130616 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d600039f0 con 0x7f6d6c107a50 2024-04-01T01:23:23.969 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.265+0000 7f6d69ffb700 1 -- 172.21.15.70:0/2568130616 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6d60003b90 con 0x7f6d6c107a50 2024-04-01T01:23:23.969 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.265+0000 7f6d72bdb700 1 -- 172.21.15.70:0/2568130616 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d6c07c290 con 0x7f6d6c107a50 2024-04-01T01:23:23.969 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.265+0000 7f6d69ffb700 1 -- 172.21.15.70:0/2568130616 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d6001a740 con 0x7f6d6c107a50 2024-04-01T01:23:23.969 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.265+0000 7f6d72bdb700 1 -- 172.21.15.70:0/2568130616 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d6c07c7d0 con 0x7f6d6c107a50 2024-04-01T01:23:23.969 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.265+0000 7f6d69ffb700 1 -- 172.21.15.70:0/2568130616 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6d6002d3f0 con 0x7f6d6c107a50 2024-04-01T01:23:23.969 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.265+0000 7f6d69ffb700 1 -- 172.21.15.70:0/2568130616 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6d60019700 con 0x7f6d6c107a50 2024-04-01T01:23:23.969 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.266+0000 7f6d72bdb700 1 -- 172.21.15.70:0/2568130616 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d6c063870 con 0x7f6d6c107a50 2024-04-01T01:23:23.969 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.268+0000 7f6d69ffb700 1 -- 172.21.15.70:0/2568130616 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6d60024af0 con 0x7f6d6c107a50 2024-04-01T01:23:23.970 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.335+0000 7f6d72bdb700 1 -- 172.21.15.70:0/2568130616 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f6d6c07ca80 con 0x7f6d6c107a50 2024-04-01T01:23:23.970 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.335+0000 7f6d69ffb700 1 -- 172.21.15.70:0/2568130616 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f6d600199a0 con 0x7f6d6c107a50 2024-04-01T01:23:23.970 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.336+0000 7f6d72bdb700 1 -- 172.21.15.70:0/2568130616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d6c107a50 msgr2=0x7f6d6c07db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:23.970 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.336+0000 7f6d72bdb700 1 --2- 172.21.15.70:0/2568130616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d6c107a50 0x7f6d6c07db60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6d60000c00 tx=0x7f6d60011b00 comp rx=0 tx=0).stop 2024-04-01T01:23:23.970 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.336+0000 7f6d72bdb700 1 -- 172.21.15.70:0/2568130616 shutdown_connections 2024-04-01T01:23:23.970 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.336+0000 7f6d72bdb700 1 --2- 172.21.15.70:0/2568130616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d6c107a50 0x7f6d6c07db60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:23.970 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.336+0000 7f6d72bdb700 1 -- 172.21.15.70:0/2568130616 >> 172.21.15.70:0/2568130616 conn(0x7f6d6c102ff0 msgr2=0x7f6d6c1039f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:23.970 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.336+0000 7f6d72bdb700 1 -- 172.21.15.70:0/2568130616 shutdown_connections 2024-04-01T01:23:23.970 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:23.336+0000 7f6d72bdb700 1 -- 172.21.15.70:0/2568130616 wait complete. 2024-04-01T01:23:23.970 INFO:teuthology.orchestra.run.smithi070.stdout:Restarting the monitor... 2024-04-01T01:23:24.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:24 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-a[31803]: 2024-04-01T01:23:24.205+0000 7fc5bcfe4700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-01T01:23:24.620 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:24 smithi070 podman[32053]: 2024-04-01 01:23:24.307621633 +0000 UTC m=+0.183552750 container died 7c579193c51538fae1d701657fb29b4752d4621da3c4642ac516c1efa8c527ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-01T01:23:24.620 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:24 smithi070 podman[32053]: 2024-04-01 01:23:24.62153869 +0000 UTC m=+0.497469819 container cleanup 7c579193c51538fae1d701657fb29b4752d4621da3c4642ac516c1efa8c527ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-01T01:23:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:24 smithi070 bash[32053]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-a 2024-04-01T01:23:24.877 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:24 smithi070 podman[32067]: 2024-04-01 01:23:24.789233994 +0000 UTC m=+0.485466623 container remove 7c579193c51538fae1d701657fb29b4752d4621da3c4642ac516c1efa8c527ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-a, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-01T01:23:25.197 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:24 smithi070 systemd[1]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mon.a.service: Deactivated successfully. 2024-04-01T01:23:25.197 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:24 smithi070 systemd[1]: Stopped Ceph mon.a for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:23:25.197 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:24 smithi070 systemd[1]: Starting Ceph mon.a for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:23:25.483 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 podman[32160]: 2024-04-01 01:23:25.198224355 +0000 UTC m=+0.105501010 container create 3284b805c9455080371237fc50d172cf40066feb8605cb7bb0fe88e233bfc5d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-01T01:23:25.483 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 podman[32160]: 2024-04-01 01:23:25.109897703 +0000 UTC m=+0.017174359 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 2024-04-01T01:23:25.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 podman[32160]: 2024-04-01 01:23:25.408729006 +0000 UTC m=+0.316005671 container init 3284b805c9455080371237fc50d172cf40066feb8605cb7bb0fe88e233bfc5d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.build-date=20240326, ceph=True, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-01T01:23:25.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 podman[32160]: 2024-04-01 01:23:25.411922532 +0000 UTC m=+0.319199189 container start 3284b805c9455080371237fc50d172cf40066feb8605cb7bb0fe88e233bfc5d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-a, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-01T01:23:25.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: set uid:gid to 167:167 (ceph:ceph) 2024-04-01T01:23:25.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev), process ceph-mon, pid 2 2024-04-01T01:23:25.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: pidfile_write: ignore empty --pid-file 2024-04-01T01:23:25.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: load: jerasure load: lrc 2024-04-01T01:23:25.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: RocksDB version: 7.9.2 2024-04-01T01:23:25.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Git sha 0 2024-04-01T01:23:25.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Compile date 2024-03-31 16:17:26 2024-04-01T01:23:25.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: DB SUMMARY 2024-04-01T01:23:25.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: DB Session ID: WKB4M4UG4APEGCOIGSMO 2024-04-01T01:23:25.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: CURRENT file: CURRENT 2024-04-01T01:23:25.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: IDENTITY file: IDENTITY 2024-04-01T01:23:25.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-01T01:23:25.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-01T01:23:25.484 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 103764 ; 2024-04-01T01:23:25.485 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.error_if_exists: 0 2024-04-01T01:23:25.485 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.create_if_missing: 0 2024-04-01T01:23:25.485 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.paranoid_checks: 1 2024-04-01T01:23:25.485 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-01T01:23:25.485 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-01T01:23:25.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-01T01:23:25.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.env: 0x562e27de4c60 2024-04-01T01:23:25.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.fs: PosixFileSystem 2024-04-01T01:23:25.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.info_log: 0x562e2a213ae0 2024-04-01T01:23:25.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_file_opening_threads: 16 2024-04-01T01:23:25.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.statistics: (nil) 2024-04-01T01:23:25.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.use_fsync: 0 2024-04-01T01:23:25.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_log_file_size: 0 2024-04-01T01:23:25.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-01T01:23:25.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-01T01:23:25.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.keep_log_file_num: 1000 2024-04-01T01:23:25.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.recycle_log_file_num: 0 2024-04-01T01:23:25.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.allow_fallocate: 1 2024-04-01T01:23:25.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.allow_mmap_reads: 0 2024-04-01T01:23:25.486 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.allow_mmap_writes: 0 2024-04-01T01:23:25.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.use_direct_reads: 0 2024-04-01T01:23:25.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-01T01:23:25.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.create_missing_column_families: 0 2024-04-01T01:23:25.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.db_log_dir: 2024-04-01T01:23:25.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.wal_dir: 2024-04-01T01:23:25.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-01T01:23:25.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-01T01:23:25.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-01T01:23:25.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-01T01:23:25.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-01T01:23:25.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-01T01:23:25.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.advise_random_on_open: 1 2024-04-01T01:23:25.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.db_write_buffer_size: 0 2024-04-01T01:23:25.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.write_buffer_manager: 0x562e2a1e61e0 2024-04-01T01:23:25.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-01T01:23:25.487 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-01T01:23:25.488 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-01T01:23:25.488 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.rate_limiter: (nil) 2024-04-01T01:23:25.488 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-01T01:23:25.488 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.wal_recovery_mode: 2 2024-04-01T01:23:25.488 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.enable_thread_tracking: 0 2024-04-01T01:23:25.488 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.enable_pipelined_write: 0 2024-04-01T01:23:25.488 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.unordered_write: 0 2024-04-01T01:23:25.488 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-01T01:23:25.488 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-01T01:23:25.488 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-01T01:23:25.488 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-01T01:23:25.488 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.row_cache: None 2024-04-01T01:23:25.488 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.wal_filter: None 2024-04-01T01:23:25.488 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-01T01:23:25.488 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.allow_ingest_behind: 0 2024-04-01T01:23:25.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.two_write_queues: 0 2024-04-01T01:23:25.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.manual_wal_flush: 0 2024-04-01T01:23:25.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.wal_compression: 0 2024-04-01T01:23:25.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.atomic_flush: 0 2024-04-01T01:23:25.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-01T01:23:25.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-01T01:23:25.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-01T01:23:25.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.log_readahead_size: 0 2024-04-01T01:23:25.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-01T01:23:25.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.best_efforts_recovery: 0 2024-04-01T01:23:25.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-01T01:23:25.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-01T01:23:25.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.allow_data_in_errors: 0 2024-04-01T01:23:25.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.db_host_id: __hostname__ 2024-04-01T01:23:25.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-01T01:23:25.489 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_background_jobs: 2 2024-04-01T01:23:25.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_background_compactions: -1 2024-04-01T01:23:25.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_subcompactions: 1 2024-04-01T01:23:25.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-01T01:23:25.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-01T01:23:25.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-01T01:23:25.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_total_wal_size: 0 2024-04-01T01:23:25.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-01T01:23:25.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-01T01:23:25.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-01T01:23:25.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-01T01:23:25.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_open_files: -1 2024-04-01T01:23:25.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.bytes_per_sync: 0 2024-04-01T01:23:25.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-01T01:23:25.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-01T01:23:25.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compaction_readahead_size: 0 2024-04-01T01:23:25.490 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_background_flushes: -1 2024-04-01T01:23:25.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Compression algorithms supported: 2024-04-01T01:23:25.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-01T01:23:25.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: kZSTD supported: 0 2024-04-01T01:23:25.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: kXpressCompression supported: 0 2024-04-01T01:23:25.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: kLZ4HCCompression supported: 1 2024-04-01T01:23:25.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: kZlibCompression supported: 1 2024-04-01T01:23:25.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: kSnappyCompression supported: 1 2024-04-01T01:23:25.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: kLZ4Compression supported: 1 2024-04-01T01:23:25.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: kBZip2Compression supported: 0 2024-04-01T01:23:25.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-01T01:23:25.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-01T01:23:25.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-01T01:23:25.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-01T01:23:25.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-01T01:23:25.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.merge_operator: 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compaction_filter: None 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compaction_filter_factory: None 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.sst_partitioner_factory: None 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562e2a213be0) 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout: cache_index_and_filter_blocks: 1 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout: pin_top_level_index_and_filter: 1 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_type: 0 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout: data_block_index_type: 0 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_shortening: 1 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout: checksum: 4 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout: no_block_cache: 0 2024-04-01T01:23:25.492 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache: 0x562e2a217090 2024-04-01T01:23:25.493 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_name: BinnedLRUCache 2024-04-01T01:23:25.493 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_options: 2024-04-01T01:23:25.493 INFO:journalctl@ceph.mon.a.smithi070.stdout: capacity : 536870912 2024-04-01T01:23:25.493 INFO:journalctl@ceph.mon.a.smithi070.stdout: num_shard_bits : 4 2024-04-01T01:23:25.493 INFO:journalctl@ceph.mon.a.smithi070.stdout: strict_capacity_limit : 0 2024-04-01T01:23:25.493 INFO:journalctl@ceph.mon.a.smithi070.stdout: high_pri_pool_ratio: 0.000 2024-04-01T01:23:25.493 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_cache_compressed: (nil) 2024-04-01T01:23:25.493 INFO:journalctl@ceph.mon.a.smithi070.stdout: persistent_cache: (nil) 2024-04-01T01:23:25.493 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_size: 4096 2024-04-01T01:23:25.493 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_size_deviation: 10 2024-04-01T01:23:25.493 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_restart_interval: 16 2024-04-01T01:23:25.493 INFO:journalctl@ceph.mon.a.smithi070.stdout: index_block_restart_interval: 1 2024-04-01T01:23:25.494 INFO:journalctl@ceph.mon.a.smithi070.stdout: metadata_block_size: 4096 2024-04-01T01:23:25.494 INFO:journalctl@ceph.mon.a.smithi070.stdout: partition_filters: 0 2024-04-01T01:23:25.494 INFO:journalctl@ceph.mon.a.smithi070.stdout: use_delta_encoding: 1 2024-04-01T01:23:25.494 INFO:journalctl@ceph.mon.a.smithi070.stdout: filter_policy: bloomfilter 2024-04-01T01:23:25.494 INFO:journalctl@ceph.mon.a.smithi070.stdout: whole_key_filtering: 1 2024-04-01T01:23:25.494 INFO:journalctl@ceph.mon.a.smithi070.stdout: verify_compression: 0 2024-04-01T01:23:25.494 INFO:journalctl@ceph.mon.a.smithi070.stdout: read_amp_bytes_per_bit: 0 2024-04-01T01:23:25.494 INFO:journalctl@ceph.mon.a.smithi070.stdout: format_version: 5 2024-04-01T01:23:25.494 INFO:journalctl@ceph.mon.a.smithi070.stdout: enable_index_compression: 1 2024-04-01T01:23:25.494 INFO:journalctl@ceph.mon.a.smithi070.stdout: block_align: 0 2024-04-01T01:23:25.494 INFO:journalctl@ceph.mon.a.smithi070.stdout: max_auto_readahead_size: 262144 2024-04-01T01:23:25.494 INFO:journalctl@ceph.mon.a.smithi070.stdout: prepopulate_block_cache: 0 2024-04-01T01:23:25.494 INFO:journalctl@ceph.mon.a.smithi070.stdout: initial_auto_readahead_size: 8192 2024-04-01T01:23:25.495 INFO:journalctl@ceph.mon.a.smithi070.stdout: num_file_reads_for_auto_readahead: 2 2024-04-01T01:23:25.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.write_buffer_size: 33554432 2024-04-01T01:23:25.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_write_buffer_number: 2 2024-04-01T01:23:25.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compression: NoCompression 2024-04-01T01:23:25.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.bottommost_compression: Disabled 2024-04-01T01:23:25.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.prefix_extractor: nullptr 2024-04-01T01:23:25.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-01T01:23:25.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.num_levels: 7 2024-04-01T01:23:25.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-01T01:23:25.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-01T01:23:25.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-01T01:23:25.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-01T01:23:25.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-01T01:23:25.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-01T01:23:25.495 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-01T01:23:25.496 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-01T01:23:25.496 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-01T01:23:25.496 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-01T01:23:25.496 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-01T01:23:25.496 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-01T01:23:25.496 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-01T01:23:25.496 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compression_opts.level: 32767 2024-04-01T01:23:25.496 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compression_opts.strategy: 0 2024-04-01T01:23:25.496 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-01T01:23:25.496 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-01T01:23:25.496 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-01T01:23:25.496 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-01T01:23:25.496 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compression_opts.enabled: false 2024-04-01T01:23:25.496 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-01T01:23:25.496 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-01T01:23:25.496 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.target_file_size_base: 67108864 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-01T01:23:25.497 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.arena_block_size: 1048576 2024-04-01T01:23:25.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-01T01:23:25.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-01T01:23:25.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.disable_auto_compactions: 0 2024-04-01T01:23:25.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-01T01:23:25.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-01T01:23:25.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-01T01:23:25.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-01T01:23:25.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-01T01:23:25.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-01T01:23:25.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-01T01:23:25.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-01T01:23:25.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-01T01:23:25.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-01T01:23:25.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.table_properties_collectors: 2024-04-01T01:23:25.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.inplace_update_support: 0 2024-04-01T01:23:25.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-01T01:23:25.499 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-01T01:23:25.499 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-01T01:23:25.499 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-01T01:23:25.499 INFO:teuthology.orchestra.run.smithi070.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-01T01:23:25.499 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.bloom_locality: 0 2024-04-01T01:23:25.499 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.max_successive_merges: 0 2024-04-01T01:23:25.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-01T01:23:25.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.paranoid_file_checks: 0 2024-04-01T01:23:25.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.force_consistency_checks: 1 2024-04-01T01:23:25.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.report_bg_io_stats: 0 2024-04-01T01:23:25.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.ttl: 2592000 2024-04-01T01:23:25.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-01T01:23:25.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-01T01:23:25.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-01T01:23:25.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.enable_blob_files: false 2024-04-01T01:23:25.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.min_blob_size: 0 2024-04-01T01:23:25.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.blob_file_size: 268435456 2024-04-01T01:23:25.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-01T01:23:25.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-01T01:23:25.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-01T01:23:25.500 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-01T01:23:25.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-01T01:23:25.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.blob_file_starting_level: 0 2024-04-01T01:23:25.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-01T01:23:25.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-01T01:23:25.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-01T01:23:25.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ff67780b-2d74-4469-9aca-da75fc46ef70 2024-04-01T01:23:25.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711934605441067, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-01T01:23:25.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-01T01:23:25.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711934605442482, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 98928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 322, "table_properties": {"data_size": 96912, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13929, "raw_average_key_size": 49, "raw_value_size": 89320, "raw_average_value_size": 317, "num_data_blocks": 12, "num_entries": 281, "num_filter_entries": 281, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711934605, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ff67780b-2d74-4469-9aca-da75fc46ef70", "db_session_id": "WKB4M4UG4APEGCOIGSMO", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-01T01:23:25.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711934605442570, "job": 1, "event": "recovery_finished"} 2024-04-01T01:23:25.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-01T01:23:25.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-01T01:23:25.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562e2a308000 2024-04-01T01:23:25.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: DB pointer 0x562e2a2f2000 2024-04-01T01:23:25.501 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-01T01:23:25.502 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-01T01:23:25.502 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** DB Stats ** 2024-04-01T01:23:25.502 INFO:journalctl@ceph.mon.a.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-01T01:23:25.502 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-01T01:23:25.502 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-01T01:23:25.502 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-01T01:23:25.502 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-01T01:23:25.502 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-01T01:23:25.502 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-01T01:23:25.502 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-01T01:23:25.502 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** Compaction Stats [default] ** 2024-04-01T01:23:25.502 INFO:journalctl@ceph.mon.a.smithi070.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-01T01:23:25.502 INFO:journalctl@ceph.mon.a.smithi070.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-01T01:23:25.502 INFO:journalctl@ceph.mon.a.smithi070.stdout: L0 2/0 98.44 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 118.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-01T01:23:25.502 INFO:journalctl@ceph.mon.a.smithi070.stdout: Sum 2/0 98.44 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 118.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-01T01:23:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 118.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-01T01:23:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-01T01:23:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** Compaction Stats [default] ** 2024-04-01T01:23:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-01T01:23:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-01T01:23:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 118.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-01T01:23:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-01T01:23:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-01T01:23:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-01T01:23:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-01T01:23:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-01T01:23:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-01T01:23:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-01T01:23:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-01T01:23:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-01T01:23:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout: Cumulative compaction: 0.00 GB write, 25.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-01T01:23:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout: Interval compaction: 0.00 GB write, 25.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-01T01:23:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-01T01:23:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout: Block cache BinnedLRUCache@0x562e2a217090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-01T01:23:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-01T01:23:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-01T01:23:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-01T01:23:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: starting mon.a rank 0 at public addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] at bind addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:23:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: mon.a@-1(???) e1 preinit fsid 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:23:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: mon.a@-1(???).mds e1 new map 2024-04-01T01:23:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: mon.a@-1(???).mds e1 print_map 2024-04-01T01:23:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout: e1 2024-04-01T01:23:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-01T01:23:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-01T01:23:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout: legacy client fscid: -1 2024-04-01T01:23:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout: 2024-04-01T01:23:25.505 INFO:journalctl@ceph.mon.a.smithi070.stdout: No filesystems configured 2024-04-01T01:23:25.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-01T01:23:25.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-01T01:23:25.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-01T01:23:25.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-01T01:23:25.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-01T01:23:25.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 bash[32160]: 3284b805c9455080371237fc50d172cf40066feb8605cb7bb0fe88e233bfc5d2 2024-04-01T01:23:25.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 systemd[1]: Started Ceph mon.a for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:23:25.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-01T01:23:25.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: monmap epoch 1 2024-04-01T01:23:25.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: fsid 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:23:25.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: last_changed 2024-04-01T01:23:17.752525+0000 2024-04-01T01:23:25.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: created 2024-04-01T01:23:17.752525+0000 2024-04-01T01:23:25.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: min_mon_release 19 (squid) 2024-04-01T01:23:25.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: election_strategy: 1 2024-04-01T01:23:25.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-01T01:23:25.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: fsmap 2024-04-01T01:23:25.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: osdmap e1: 0 total, 0 up, 0 in 2024-04-01T01:23:25.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:25 smithi070 ceph-mon[32179]: mgrmap e1: no daemons active 2024-04-01T01:23:26.521 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.904+0000 7f6199e6d700 1 Processor -- start 2024-04-01T01:23:26.522 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.904+0000 7f6199e6d700 1 -- start start 2024-04-01T01:23:26.522 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.904+0000 7f6199e6d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61941062c0 0x7f61941066e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:26.522 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.904+0000 7f6199e6d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f619407be60 con 0x7f61941062c0 2024-04-01T01:23:26.522 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.904+0000 7f61937fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61941062c0 0x7f61941066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:26.522 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.904+0000 7f61937fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61941062c0 0x7f61941066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46486/0 (socket says 172.21.15.70:46486) 2024-04-01T01:23:26.522 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.904+0000 7f61937fe700 1 -- 172.21.15.70:0/1146642442 learned_addr learned my addr 172.21.15.70:0/1146642442 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:26.522 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.905+0000 7f61937fe700 1 -- 172.21.15.70:0/1146642442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6194106c20 con 0x7f61941062c0 2024-04-01T01:23:26.522 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.905+0000 7f61937fe700 1 --2- 172.21.15.70:0/1146642442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61941062c0 0x7f61941066e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f61880098a0 tx=0x7f6188009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfbe2694fab8dc10 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:26.522 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.905+0000 7f61927fc700 1 -- 172.21.15.70:0/1146642442 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6188004160 con 0x7f61941062c0 2024-04-01T01:23:26.522 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.905+0000 7f61927fc700 1 -- 172.21.15.70:0/1146642442 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6188004300 con 0x7f61941062c0 2024-04-01T01:23:26.522 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.905+0000 7f61927fc700 1 -- 172.21.15.70:0/1146642442 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6188011a80 con 0x7f61941062c0 2024-04-01T01:23:26.522 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.905+0000 7f6199e6d700 1 -- 172.21.15.70:0/1146642442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61941062c0 msgr2=0x7f61941066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:26.523 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.905+0000 7f6199e6d700 1 --2- 172.21.15.70:0/1146642442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61941062c0 0x7f61941066e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f61880098a0 tx=0x7f6188009bb0 comp rx=0 tx=0).stop 2024-04-01T01:23:26.523 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.905+0000 7f6199e6d700 1 -- 172.21.15.70:0/1146642442 shutdown_connections 2024-04-01T01:23:26.523 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.905+0000 7f6199e6d700 1 --2- 172.21.15.70:0/1146642442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61941062c0 0x7f61941066e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:26.523 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.906+0000 7f6199e6d700 1 -- 172.21.15.70:0/1146642442 >> 172.21.15.70:0/1146642442 conn(0x7f6194101ea0 msgr2=0x7f6194104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:26.523 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.906+0000 7f6199e6d700 1 -- 172.21.15.70:0/1146642442 shutdown_connections 2024-04-01T01:23:26.523 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.906+0000 7f6199e6d700 1 -- 172.21.15.70:0/1146642442 wait complete. 2024-04-01T01:23:26.523 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.906+0000 7f6199e6d700 1 Processor -- start 2024-04-01T01:23:26.523 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.906+0000 7f6199e6d700 1 -- start start 2024-04-01T01:23:26.523 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.906+0000 7f6199e6d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61941062c0 0x7f619419da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:26.523 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.906+0000 7f6199e6d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6194107c50 con 0x7f61941062c0 2024-04-01T01:23:26.523 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.907+0000 7f61937fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61941062c0 0x7f619419da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:26.523 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.907+0000 7f61937fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61941062c0 0x7f619419da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46494/0 (socket says 172.21.15.70:46494) 2024-04-01T01:23:26.523 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.907+0000 7f61937fe700 1 -- 172.21.15.70:0/2831884668 learned_addr learned my addr 172.21.15.70:0/2831884668 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:26.523 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.907+0000 7f61937fe700 1 -- 172.21.15.70:0/2831884668 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f619419dfd0 con 0x7f61941062c0 2024-04-01T01:23:26.523 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.907+0000 7f61937fe700 1 --2- 172.21.15.70:0/2831884668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61941062c0 0x7f619419da90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6188006af0 tx=0x7f6188011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:26.524 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.907+0000 7f6190ff9700 1 -- 172.21.15.70:0/2831884668 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6188003930 con 0x7f61941062c0 2024-04-01T01:23:26.524 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.907+0000 7f6199e6d700 1 -- 172.21.15.70:0/2831884668 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f619419e260 con 0x7f61941062c0 2024-04-01T01:23:26.524 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.907+0000 7f6190ff9700 1 -- 172.21.15.70:0/2831884668 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6188003ad0 con 0x7f61941062c0 2024-04-01T01:23:26.524 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.907+0000 7f6190ff9700 1 -- 172.21.15.70:0/2831884668 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61880195e0 con 0x7f61941062c0 2024-04-01T01:23:26.524 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.907+0000 7f6199e6d700 1 -- 172.21.15.70:0/2831884668 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61941a0f50 con 0x7f61941062c0 2024-04-01T01:23:26.524 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.907+0000 7f6190ff9700 1 -- 172.21.15.70:0/2831884668 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6188019c90 con 0x7f61941062c0 2024-04-01T01:23:26.524 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.907+0000 7f6190ff9700 1 -- 172.21.15.70:0/2831884668 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6188024730 con 0x7f61941062c0 2024-04-01T01:23:26.524 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.908+0000 7f6199e6d700 1 -- 172.21.15.70:0/2831884668 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6194063870 con 0x7f61941062c0 2024-04-01T01:23:26.524 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.910+0000 7f6190ff9700 1 -- 172.21.15.70:0/2831884668 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6188043dd0 con 0x7f61941062c0 2024-04-01T01:23:26.524 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.976+0000 7f6199e6d700 1 -- 172.21.15.70:0/2831884668 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f619402de10 con 0x7f61941062c0 2024-04-01T01:23:26.524 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.978+0000 7f6190ff9700 1 -- 172.21.15.70:0/2831884668 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f6188032020 con 0x7f61941062c0 2024-04-01T01:23:26.524 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.979+0000 7f6199e6d700 1 -- 172.21.15.70:0/2831884668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61941062c0 msgr2=0x7f619419da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:26.524 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.979+0000 7f6199e6d700 1 --2- 172.21.15.70:0/2831884668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61941062c0 0x7f619419da90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6188006af0 tx=0x7f6188011b70 comp rx=0 tx=0).stop 2024-04-01T01:23:26.524 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.979+0000 7f6199e6d700 1 -- 172.21.15.70:0/2831884668 shutdown_connections 2024-04-01T01:23:26.524 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.979+0000 7f6199e6d700 1 --2- 172.21.15.70:0/2831884668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61941062c0 0x7f619419da90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:26.525 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.979+0000 7f6199e6d700 1 -- 172.21.15.70:0/2831884668 >> 172.21.15.70:0/2831884668 conn(0x7f6194101ea0 msgr2=0x7f6194194730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:26.525 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.979+0000 7f6199e6d700 1 -- 172.21.15.70:0/2831884668 shutdown_connections 2024-04-01T01:23:26.525 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:25.979+0000 7f6199e6d700 1 -- 172.21.15.70:0/2831884668 wait complete. 2024-04-01T01:23:26.525 INFO:teuthology.orchestra.run.smithi070.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-01T01:23:26.538 INFO:teuthology.orchestra.run.smithi070.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-01T01:23:26.539 INFO:teuthology.orchestra.run.smithi070.stdout:Creating mgr... 2024-04-01T01:23:26.539 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-01T01:23:26.539 INFO:teuthology.orchestra.run.smithi070.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-01T01:23:26.784 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mgr.x 2024-04-01T01:23:26.784 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Failed to reset failed state of unit ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mgr.x.service: Unit ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mgr.x.service not loaded. 2024-04-01T01:23:26.933 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-36577394-efc6-11ee-b647-cb9ed24678a4.target.wants/ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mgr.x.service → /etc/systemd/system/ceph-36577394-efc6-11ee-b647-cb9ed24678a4@.service. 2024-04-01T01:23:27.167 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:26 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2831884668' entity='client.admin' 2024-04-01T01:23:27.451 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:27 smithi070 podman[32397]: 2024-04-01 01:23:27.388149304 +0000 UTC m=+0.334581513 container start aed29d5f424085705770a436e499e640e113ad1e429d818dcfc91f5444981f25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, io.buildah.version=1.33.5) 2024-04-01T01:23:27.451 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:27 smithi070 bash[32397]: aed29d5f424085705770a436e499e640e113ad1e429d818dcfc91f5444981f25 2024-04-01T01:23:27.467 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-01T01:23:27.467 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout disabled 2024-04-01T01:23:27.475 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-01T01:23:27.475 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-04-01T01:23:27.475 INFO:teuthology.orchestra.run.smithi070.stdout:firewalld.service is not enabled 2024-04-01T01:23:27.475 INFO:teuthology.orchestra.run.smithi070.stdout:Not possible to enable service . firewalld.service is not available 2024-04-01T01:23:27.485 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-01T01:23:27.485 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout disabled 2024-04-01T01:23:27.493 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-01T01:23:27.494 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-04-01T01:23:27.494 INFO:teuthology.orchestra.run.smithi070.stdout:firewalld.service is not enabled 2024-04-01T01:23:27.494 INFO:teuthology.orchestra.run.smithi070.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-01T01:23:27.494 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mgr to start... 2024-04-01T01:23:27.494 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mgr... 2024-04-01T01:23:27.703 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:27 smithi070 systemd[1]: Started Ceph mgr.x for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:23:28.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:27 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1748071948' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-01T01:23:28.232 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:27 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:27.964+0000 7fbe881a7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-01T01:23:28.233 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:28 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:28.066+0000 7fbe881a7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-01T01:23:28.484 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-04-01T01:23:28.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-04-01T01:23:28.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsid": "36577394-efc6-11ee-b647-cb9ed24678a4", 2024-04-01T01:23:28.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "health": { 2024-04-01T01:23:28.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-01T01:23:28.487 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-01T01:23:28.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-01T01:23:28.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:28.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-01T01:23:28.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-01T01:23:28.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 0 2024-04-01T01:23:28.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-04-01T01:23:28.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-01T01:23:28.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "a" 2024-04-01T01:23:28.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-04-01T01:23:28.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-01T01:23:28.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-01T01:23:28.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-01T01:23:28.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-01T01:23:28.488 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-01T01:23:28.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:28.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-01T01:23:28.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-01T01:23:28.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-01T01:23:28.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-01T01:23:28.489 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-01T01:23:28.497 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-01T01:23:28.497 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-01T01:23:28.497 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-01T01:23:28.497 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:28.497 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-01T01:23:28.497 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-01T01:23:28.497 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-01T01:23:28.498 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-01T01:23:28.498 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-01T01:23:28.498 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-01T01:23:28.498 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-01T01:23:28.498 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-01T01:23:28.498 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-01T01:23:28.498 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:28.498 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-01T01:23:28.498 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-01T01:23:28.498 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-01T01:23:28.498 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-01T01:23:28.498 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:28.498 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-01T01:23:28.498 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-01T01:23:28.499 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-01T01:23:28.499 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-01T01:23:28.499 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-01T01:23:28.499 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-01T01:23:28.499 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "restful" 2024-04-01T01:23:28.499 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-04-01T01:23:28.499 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-01T01:23:28.499 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:28.499 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-01T01:23:28.499 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-01T01:23:28.499 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modified": "2024-04-01T01:23:20.697216+0000", 2024-04-01T01:23:28.499 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-01T01:23:28.499 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:28.499 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-01T01:23:28.499 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-04-01T01:23:28.500 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.880+0000 7f044aee3700 1 Processor -- start 2024-04-01T01:23:28.500 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.880+0000 7f044aee3700 1 -- start start 2024-04-01T01:23:28.500 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.880+0000 7f044aee3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04441084c0 0x7f044410a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:28.500 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.880+0000 7f044aee3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f044407be80 con 0x7f04441084c0 2024-04-01T01:23:28.500 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.880+0000 7f0448c7f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04441084c0 0x7f044410a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:28.500 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.881+0000 7f0448c7f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04441084c0 0x7f044410a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39210/0 (socket says 172.21.15.70:39210) 2024-04-01T01:23:28.500 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.881+0000 7f0448c7f700 1 -- 172.21.15.70:0/2022429614 learned_addr learned my addr 172.21.15.70:0/2022429614 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:28.500 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.881+0000 7f0448c7f700 1 -- 172.21.15.70:0/2022429614 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044410adf0 con 0x7f04441084c0 2024-04-01T01:23:28.500 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.881+0000 7f0448c7f700 1 --2- 172.21.15.70:0/2022429614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04441084c0 0x7f044410a8b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f04380098a0 tx=0x7f0438009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c03d17bed4d4fd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:28.500 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.881+0000 7f04437fe700 1 -- 172.21.15.70:0/2022429614 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0438004160 con 0x7f04441084c0 2024-04-01T01:23:28.500 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.881+0000 7f04437fe700 1 -- 172.21.15.70:0/2022429614 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0438004300 con 0x7f04441084c0 2024-04-01T01:23:28.500 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.881+0000 7f04437fe700 1 -- 172.21.15.70:0/2022429614 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0438011a80 con 0x7f04441084c0 2024-04-01T01:23:28.500 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.882+0000 7f044aee3700 1 -- 172.21.15.70:0/2022429614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04441084c0 msgr2=0x7f044410a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:28.501 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.882+0000 7f044aee3700 1 --2- 172.21.15.70:0/2022429614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04441084c0 0x7f044410a8b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f04380098a0 tx=0x7f0438009bb0 comp rx=0 tx=0).stop 2024-04-01T01:23:28.501 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.882+0000 7f044aee3700 1 -- 172.21.15.70:0/2022429614 shutdown_connections 2024-04-01T01:23:28.501 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.882+0000 7f044aee3700 1 --2- 172.21.15.70:0/2022429614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04441084c0 0x7f044410a8b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:28.501 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.882+0000 7f044aee3700 1 -- 172.21.15.70:0/2022429614 >> 172.21.15.70:0/2022429614 conn(0x7f0444101ed0 msgr2=0x7f0444104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:28.501 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.882+0000 7f044aee3700 1 -- 172.21.15.70:0/2022429614 shutdown_connections 2024-04-01T01:23:28.501 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.882+0000 7f044aee3700 1 -- 172.21.15.70:0/2022429614 wait complete. 2024-04-01T01:23:28.501 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.883+0000 7f044aee3700 1 Processor -- start 2024-04-01T01:23:28.501 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.883+0000 7f044aee3700 1 -- start start 2024-04-01T01:23:28.501 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.883+0000 7f044aee3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04441084c0 0x7f044419b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:28.501 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.883+0000 7f044aee3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f044410bd50 con 0x7f04441084c0 2024-04-01T01:23:28.501 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.883+0000 7f0448c7f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04441084c0 0x7f044419b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:28.501 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.883+0000 7f0448c7f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04441084c0 0x7f044419b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39220/0 (socket says 172.21.15.70:39220) 2024-04-01T01:23:28.501 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.883+0000 7f0448c7f700 1 -- 172.21.15.70:0/1748071948 learned_addr learned my addr 172.21.15.70:0/1748071948 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:28.501 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.883+0000 7f0448c7f700 1 -- 172.21.15.70:0/1748071948 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044419bda0 con 0x7f04441084c0 2024-04-01T01:23:28.501 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.883+0000 7f0448c7f700 1 --2- 172.21.15.70:0/1748071948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04441084c0 0x7f044419b860 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0438006af0 tx=0x7f0438011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:28.502 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.884+0000 7f0441ffb700 1 -- 172.21.15.70:0/1748071948 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0438003930 con 0x7f04441084c0 2024-04-01T01:23:28.502 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.884+0000 7f0441ffb700 1 -- 172.21.15.70:0/1748071948 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0438003ad0 con 0x7f04441084c0 2024-04-01T01:23:28.502 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.884+0000 7f0441ffb700 1 -- 172.21.15.70:0/1748071948 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0438019590 con 0x7f04441084c0 2024-04-01T01:23:28.502 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.884+0000 7f044aee3700 1 -- 172.21.15.70:0/1748071948 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f044419c090 con 0x7f04441084c0 2024-04-01T01:23:28.502 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.884+0000 7f044aee3700 1 -- 172.21.15.70:0/1748071948 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f044419ed20 con 0x7f04441084c0 2024-04-01T01:23:28.502 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.884+0000 7f0441ffb700 1 -- 172.21.15.70:0/1748071948 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0438019c20 con 0x7f04441084c0 2024-04-01T01:23:28.502 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.884+0000 7f0441ffb700 1 -- 172.21.15.70:0/1748071948 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0438024730 con 0x7f04441084c0 2024-04-01T01:23:28.502 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.885+0000 7f044aee3700 1 -- 172.21.15.70:0/1748071948 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0444063870 con 0x7f04441084c0 2024-04-01T01:23:28.502 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.887+0000 7f0441ffb700 1 -- 172.21.15.70:0/1748071948 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f04380249d0 con 0x7f04441084c0 2024-04-01T01:23:28.502 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.959+0000 7f044aee3700 1 -- 172.21.15.70:0/1748071948 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f044419eef0 con 0x7f04441084c0 2024-04-01T01:23:28.502 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.959+0000 7f0441ffb700 1 -- 172.21.15.70:0/1748071948 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f0438024bb0 con 0x7f04441084c0 2024-04-01T01:23:28.502 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.960+0000 7f044aee3700 1 -- 172.21.15.70:0/1748071948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04441084c0 msgr2=0x7f044419b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:28.502 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.960+0000 7f044aee3700 1 --2- 172.21.15.70:0/1748071948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04441084c0 0x7f044419b860 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0438006af0 tx=0x7f0438011b70 comp rx=0 tx=0).stop 2024-04-01T01:23:28.503 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.960+0000 7f044aee3700 1 -- 172.21.15.70:0/1748071948 shutdown_connections 2024-04-01T01:23:28.503 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.960+0000 7f044aee3700 1 --2- 172.21.15.70:0/1748071948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04441084c0 0x7f044419b860 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:28.503 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.960+0000 7f044aee3700 1 -- 172.21.15.70:0/1748071948 >> 172.21.15.70:0/1748071948 conn(0x7f0444101ed0 msgr2=0x7f0444102900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:28.503 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.961+0000 7f044aee3700 1 -- 172.21.15.70:0/1748071948 shutdown_connections 2024-04-01T01:23:28.503 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:27.961+0000 7f044aee3700 1 -- 172.21.15.70:0/1748071948 wait complete. 2024-04-01T01:23:28.503 INFO:teuthology.orchestra.run.smithi070.stdout:mgr not available, waiting (1/15)... 2024-04-01T01:23:28.753 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:28 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:28.710+0000 7fbe881a7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-01T01:23:29.117 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:28 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:28.802+0000 7fbe881a7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-01T01:23:29.117 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:28 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:28.942+0000 7fbe881a7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-01T01:23:29.117 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:29 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:29.036+0000 7fbe881a7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-01T01:23:29.117 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:29 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:29.117+0000 7fbe881a7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-01T01:23:29.664 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:29 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:29.377+0000 7fbe881a7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-01T01:23:29.664 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:29 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:29.486+0000 7fbe881a7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-01T01:23:29.664 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:29 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:29.572+0000 7fbe881a7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-01T01:23:29.664 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:29 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:29.664+0000 7fbe881a7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-01T01:23:30.003 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:29 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:29.839+0000 7fbe881a7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-01T01:23:30.949 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:30 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:30.818+0000 7fbe881a7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-01T01:23:30.949 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:30 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:30.950+0000 7fbe881a7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-01T01:23:31.573 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:31 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:31.288+0000 7fbe881a7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-01T01:23:31.573 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:31 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:31.379+0000 7fbe881a7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-01T01:23:31.573 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:31 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:31.574+0000 7fbe881a7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-01T01:23:32.003 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:31 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:31.759+0000 7fbe881a7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-01T01:23:32.299 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:32 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:32.042+0000 7fbe881a7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-01T01:23:32.300 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:32 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:32.300+0000 7fbe881a7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-01T01:23:32.753 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:32 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:32.539+0000 7fbe881a7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-01T01:23:33.622 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:33 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:33.347+0000 7fbe881a7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-01T01:23:33.933 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:33 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2323632027' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-01T01:23:34.246 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:33 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:33.932+0000 7fbe881a7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-01T01:23:34.246 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:34 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:34.030+0000 7fbe881a7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-01T01:23:34.246 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:34 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:34.246+0000 7fbe881a7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-01T01:23:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:34 smithi070 ceph-mon[32179]: Activating manager daemon x 2024-04-01T01:23:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:34 smithi070 ceph-mon[32179]: mgrmap e2: x(active, starting, since 0.00311319s) 2024-04-01T01:23:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:34 smithi070 ceph-mon[32179]: from='mgr.14100 172.21.15.70:0/993926535' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-01T01:23:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:34 smithi070 ceph-mon[32179]: from='mgr.14100 172.21.15.70:0/993926535' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-01T01:23:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:34 smithi070 ceph-mon[32179]: from='mgr.14100 172.21.15.70:0/993926535' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-01T01:23:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:34 smithi070 ceph-mon[32179]: from='mgr.14100 172.21.15.70:0/993926535' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:23:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:34 smithi070 ceph-mon[32179]: from='mgr.14100 172.21.15.70:0/993926535' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-01T01:23:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:34 smithi070 ceph-mon[32179]: Manager daemon x is now available 2024-04-01T01:23:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:34 smithi070 ceph-mon[32179]: from='mgr.14100 172.21.15.70:0/993926535' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-01T01:23:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:34 smithi070 ceph-mon[32179]: from='mgr.14100 172.21.15.70:0/993926535' entity='mgr.x' 2024-04-01T01:23:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:34 smithi070 ceph-mon[32179]: from='mgr.14100 172.21.15.70:0/993926535' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-01T01:23:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:34 smithi070 ceph-mon[32179]: from='mgr.14100 172.21.15.70:0/993926535' entity='mgr.x' 2024-04-01T01:23:35.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:34 smithi070 ceph-mon[32179]: from='mgr.14100 172.21.15.70:0/993926535' entity='mgr.x' 2024-04-01T01:23:36.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:36 smithi070 ceph-mon[32179]: mgrmap e3: x(active, since 1.00691s) 2024-04-01T01:23:37.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:37 smithi070 ceph-mon[32179]: mgrmap e4: x(active, since 2s) 2024-04-01T01:23:40.043 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-04-01T01:23:40.043 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-04-01T01:23:40.043 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsid": "36577394-efc6-11ee-b647-cb9ed24678a4", 2024-04-01T01:23:40.043 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "health": { 2024-04-01T01:23:40.043 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-01T01:23:40.043 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-01T01:23:40.043 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-01T01:23:40.044 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:40.044 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-01T01:23:40.044 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-01T01:23:40.044 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 0 2024-04-01T01:23:40.044 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-04-01T01:23:40.044 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-01T01:23:40.044 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "a" 2024-04-01T01:23:40.044 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-04-01T01:23:40.044 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-01T01:23:40.044 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-01T01:23:40.044 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-01T01:23:40.044 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-01T01:23:40.044 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-01T01:23:40.044 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:40.044 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-01T01:23:40.044 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-01T01:23:40.045 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-01T01:23:40.045 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-01T01:23:40.045 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-01T01:23:40.052 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-01T01:23:40.053 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-01T01:23:40.053 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-01T01:23:40.053 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:40.053 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-01T01:23:40.053 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-01T01:23:40.053 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-01T01:23:40.053 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-01T01:23:40.053 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-01T01:23:40.053 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-01T01:23:40.053 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-01T01:23:40.053 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-01T01:23:40.053 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-01T01:23:40.053 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:40.054 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-01T01:23:40.054 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-01T01:23:40.054 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-01T01:23:40.054 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-01T01:23:40.054 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:40.054 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-01T01:23:40.054 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-01T01:23:40.054 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-01T01:23:40.054 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-01T01:23:40.054 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-01T01:23:40.054 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-01T01:23:40.054 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "restful" 2024-04-01T01:23:40.054 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-04-01T01:23:40.054 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-01T01:23:40.055 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:40.055 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-01T01:23:40.055 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-01T01:23:40.055 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modified": "2024-04-01T01:23:20.697216+0000", 2024-04-01T01:23:40.055 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-01T01:23:40.055 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:40.055 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-01T01:23:40.055 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-04-01T01:23:40.055 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.720+0000 7f873f4b7700 1 Processor -- start 2024-04-01T01:23:40.055 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.721+0000 7f873f4b7700 1 -- start start 2024-04-01T01:23:40.055 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.721+0000 7f873f4b7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8738106930 0x7f8738106d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:40.055 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.721+0000 7f873f4b7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8738107320 con 0x7f8738106930 2024-04-01T01:23:40.055 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.721+0000 7f873d253700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8738106930 0x7f8738106d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:40.055 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.721+0000 7f873d253700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8738106930 0x7f8738106d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39222/0 (socket says 172.21.15.70:39222) 2024-04-01T01:23:40.055 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.721+0000 7f873d253700 1 -- 172.21.15.70:0/3107636225 learned_addr learned my addr 172.21.15.70:0/3107636225 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:40.056 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.721+0000 7f873d253700 1 -- 172.21.15.70:0/3107636225 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8738107b30 con 0x7f8738106930 2024-04-01T01:23:40.056 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.721+0000 7f873d253700 1 --2- 172.21.15.70:0/3107636225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8738106930 0x7f8738106d50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f87340098a0 tx=0x7f8734009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82bf1bf4968e83d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:40.056 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.721+0000 7f872ffff700 1 -- 172.21.15.70:0/3107636225 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8734004160 con 0x7f8738106930 2024-04-01T01:23:40.056 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.721+0000 7f872ffff700 1 -- 172.21.15.70:0/3107636225 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8734004300 con 0x7f8738106930 2024-04-01T01:23:40.056 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.721+0000 7f872ffff700 1 -- 172.21.15.70:0/3107636225 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8734011a80 con 0x7f8738106930 2024-04-01T01:23:40.056 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.722+0000 7f873f4b7700 1 -- 172.21.15.70:0/3107636225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8738106930 msgr2=0x7f8738106d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:40.056 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.722+0000 7f873f4b7700 1 --2- 172.21.15.70:0/3107636225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8738106930 0x7f8738106d50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f87340098a0 tx=0x7f8734009bb0 comp rx=0 tx=0).stop 2024-04-01T01:23:40.056 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.722+0000 7f873f4b7700 1 -- 172.21.15.70:0/3107636225 shutdown_connections 2024-04-01T01:23:40.056 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.722+0000 7f873f4b7700 1 --2- 172.21.15.70:0/3107636225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8738106930 0x7f8738106d50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:40.056 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.722+0000 7f873f4b7700 1 -- 172.21.15.70:0/3107636225 >> 172.21.15.70:0/3107636225 conn(0x7f8738101ed0 msgr2=0x7f8738104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:40.056 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.722+0000 7f873f4b7700 1 -- 172.21.15.70:0/3107636225 shutdown_connections 2024-04-01T01:23:40.056 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.722+0000 7f873f4b7700 1 -- 172.21.15.70:0/3107636225 wait complete. 2024-04-01T01:23:40.056 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.723+0000 7f873f4b7700 1 Processor -- start 2024-04-01T01:23:40.056 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.723+0000 7f873f4b7700 1 -- start start 2024-04-01T01:23:40.056 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.723+0000 7f873f4b7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8738106930 0x7f87381042a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:40.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.723+0000 7f873f4b7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f873810aee0 con 0x7f8738106930 2024-04-01T01:23:40.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.723+0000 7f873d253700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8738106930 0x7f87381042a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:40.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.723+0000 7f873d253700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8738106930 0x7f87381042a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39238/0 (socket says 172.21.15.70:39238) 2024-04-01T01:23:40.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.723+0000 7f873d253700 1 -- 172.21.15.70:0/2323632027 learned_addr learned my addr 172.21.15.70:0/2323632027 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:40.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.723+0000 7f873d253700 1 -- 172.21.15.70:0/2323632027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87381047e0 con 0x7f8738106930 2024-04-01T01:23:40.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.723+0000 7f873d253700 1 --2- 172.21.15.70:0/2323632027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8738106930 0x7f87381042a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8734006af0 tx=0x7f8734011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:40.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.723+0000 7f872e7fc700 1 -- 172.21.15.70:0/2323632027 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8734003930 con 0x7f8738106930 2024-04-01T01:23:40.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.723+0000 7f872e7fc700 1 -- 172.21.15.70:0/2323632027 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8734003ad0 con 0x7f8738106930 2024-04-01T01:23:40.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.723+0000 7f873f4b7700 1 -- 172.21.15.70:0/2323632027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87381029d0 con 0x7f8738106930 2024-04-01T01:23:40.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.723+0000 7f872e7fc700 1 -- 172.21.15.70:0/2323632027 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87340195e0 con 0x7f8738106930 2024-04-01T01:23:40.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.724+0000 7f873f4b7700 1 -- 172.21.15.70:0/2323632027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8738102f10 con 0x7f8738106930 2024-04-01T01:23:40.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.724+0000 7f872e7fc700 1 -- 172.21.15.70:0/2323632027 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8734019c20 con 0x7f8738106930 2024-04-01T01:23:40.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.724+0000 7f872e7fc700 1 -- 172.21.15.70:0/2323632027 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8734024730 con 0x7f8738106930 2024-04-01T01:23:40.057 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.725+0000 7f873f4b7700 1 -- 172.21.15.70:0/2323632027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8738063870 con 0x7f8738106930 2024-04-01T01:23:40.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.727+0000 7f872e7fc700 1 -- 172.21.15.70:0/2323632027 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8734043dd0 con 0x7f8738106930 2024-04-01T01:23:40.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.799+0000 7f873f4b7700 1 -- 172.21.15.70:0/2323632027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f8738194da0 con 0x7f8738106930 2024-04-01T01:23:40.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.799+0000 7f872e7fc700 1 -- 172.21.15.70:0/2323632027 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f8734032030 con 0x7f8738106930 2024-04-01T01:23:40.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.800+0000 7f873f4b7700 1 -- 172.21.15.70:0/2323632027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8738106930 msgr2=0x7f87381042a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:40.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.800+0000 7f873f4b7700 1 --2- 172.21.15.70:0/2323632027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8738106930 0x7f87381042a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8734006af0 tx=0x7f8734011b70 comp rx=0 tx=0).stop 2024-04-01T01:23:40.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.800+0000 7f873f4b7700 1 -- 172.21.15.70:0/2323632027 shutdown_connections 2024-04-01T01:23:40.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.800+0000 7f873f4b7700 1 --2- 172.21.15.70:0/2323632027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8738106930 0x7f87381042a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:40.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.800+0000 7f873f4b7700 1 -- 172.21.15.70:0/2323632027 >> 172.21.15.70:0/2323632027 conn(0x7f8738101ed0 msgr2=0x7f8738192500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:40.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.800+0000 7f873f4b7700 1 -- 172.21.15.70:0/2323632027 shutdown_connections 2024-04-01T01:23:40.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:33.800+0000 7f873f4b7700 1 -- 172.21.15.70:0/2323632027 wait complete. 2024-04-01T01:23:40.058 INFO:teuthology.orchestra.run.smithi070.stdout:mgr not available, waiting (2/15)... 2024-04-01T01:23:43.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:42 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/4111772951' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-01T01:23:43.311 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-04-01T01:23:43.311 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-04-01T01:23:43.311 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsid": "36577394-efc6-11ee-b647-cb9ed24678a4", 2024-04-01T01:23:43.311 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "health": { 2024-04-01T01:23:43.311 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-01T01:23:43.311 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-01T01:23:43.312 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-01T01:23:43.312 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:43.312 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-01T01:23:43.312 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-01T01:23:43.312 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 0 2024-04-01T01:23:43.312 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-04-01T01:23:43.312 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-01T01:23:43.312 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "a" 2024-04-01T01:23:43.312 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-04-01T01:23:43.312 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "quorum_age": 17, 2024-04-01T01:23:43.312 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-01T01:23:43.312 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-01T01:23:43.312 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-01T01:23:43.312 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-01T01:23:43.312 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:43.312 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-01T01:23:43.313 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-01T01:23:43.313 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-01T01:23:43.313 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-01T01:23:43.313 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-01T01:23:43.321 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-01T01:23:43.321 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-01T01:23:43.321 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-01T01:23:43.321 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:43.321 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-01T01:23:43.321 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-01T01:23:43.321 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-01T01:23:43.321 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-01T01:23:43.322 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-01T01:23:43.322 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-01T01:23:43.322 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-01T01:23:43.322 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-01T01:23:43.322 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-01T01:23:43.322 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:43.322 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-01T01:23:43.322 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-01T01:23:43.322 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-01T01:23:43.322 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-01T01:23:43.322 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:43.322 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-01T01:23:43.322 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-01T01:23:43.322 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-01T01:23:43.322 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-01T01:23:43.323 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-01T01:23:43.323 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-01T01:23:43.323 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "restful" 2024-04-01T01:23:43.323 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ], 2024-04-01T01:23:43.323 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-01T01:23:43.323 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:43.323 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-01T01:23:43.323 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-01T01:23:43.323 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "modified": "2024-04-01T01:23:20.697216+0000", 2024-04-01T01:23:43.323 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-01T01:23:43.323 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout }, 2024-04-01T01:23:43.323 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-01T01:23:43.323 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-04-01T01:23:43.323 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.458+0000 7f0c2942d700 1 Processor -- start 2024-04-01T01:23:43.323 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.458+0000 7f0c2942d700 1 -- start start 2024-04-01T01:23:43.323 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.459+0000 7f0c2942d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c24106930 0x7f0c24106d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:43.324 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.459+0000 7f0c2942d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0c24107320 con 0x7f0c24106930 2024-04-01T01:23:43.324 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.459+0000 7f0c22ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c24106930 0x7f0c24106d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:43.324 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.459+0000 7f0c22ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c24106930 0x7f0c24106d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53468/0 (socket says 172.21.15.70:53468) 2024-04-01T01:23:43.324 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.459+0000 7f0c22ffd700 1 -- 172.21.15.70:0/3337743106 learned_addr learned my addr 172.21.15.70:0/3337743106 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:43.324 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.459+0000 7f0c22ffd700 1 -- 172.21.15.70:0/3337743106 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c24107b30 con 0x7f0c24106930 2024-04-01T01:23:43.324 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.459+0000 7f0c22ffd700 1 --2- 172.21.15.70:0/3337743106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c24106930 0x7f0c24106d50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c0098a0 tx=0x7f0c0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c06c3a8b04a52e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:43.324 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.459+0000 7f0c21ffb700 1 -- 172.21.15.70:0/3337743106 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c0c004160 con 0x7f0c24106930 2024-04-01T01:23:43.324 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.460+0000 7f0c21ffb700 1 -- 172.21.15.70:0/3337743106 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0c0c004300 con 0x7f0c24106930 2024-04-01T01:23:43.324 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.460+0000 7f0c21ffb700 1 -- 172.21.15.70:0/3337743106 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c0c011a80 con 0x7f0c24106930 2024-04-01T01:23:43.324 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.460+0000 7f0c2942d700 1 -- 172.21.15.70:0/3337743106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c24106930 msgr2=0x7f0c24106d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:43.324 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.460+0000 7f0c2942d700 1 --2- 172.21.15.70:0/3337743106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c24106930 0x7f0c24106d50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c0098a0 tx=0x7f0c0c009bb0 comp rx=0 tx=0).stop 2024-04-01T01:23:43.324 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.460+0000 7f0c2942d700 1 -- 172.21.15.70:0/3337743106 shutdown_connections 2024-04-01T01:23:43.324 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.460+0000 7f0c2942d700 1 --2- 172.21.15.70:0/3337743106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c24106930 0x7f0c24106d50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:43.324 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.460+0000 7f0c2942d700 1 -- 172.21.15.70:0/3337743106 >> 172.21.15.70:0/3337743106 conn(0x7f0c24101ed0 msgr2=0x7f0c24104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:43.325 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.460+0000 7f0c2942d700 1 -- 172.21.15.70:0/3337743106 shutdown_connections 2024-04-01T01:23:43.325 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.460+0000 7f0c2942d700 1 -- 172.21.15.70:0/3337743106 wait complete. 2024-04-01T01:23:43.325 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.461+0000 7f0c2942d700 1 Processor -- start 2024-04-01T01:23:43.325 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.461+0000 7f0c2942d700 1 -- start start 2024-04-01T01:23:43.325 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.461+0000 7f0c2942d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c24106930 0x7f0c2407ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:43.325 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.461+0000 7f0c2942d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2410aee0 con 0x7f0c24106930 2024-04-01T01:23:43.325 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.461+0000 7f0c22ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c24106930 0x7f0c2407ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:43.325 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.461+0000 7f0c22ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c24106930 0x7f0c2407ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53472/0 (socket says 172.21.15.70:53472) 2024-04-01T01:23:43.325 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.461+0000 7f0c22ffd700 1 -- 172.21.15.70:0/4111772951 learned_addr learned my addr 172.21.15.70:0/4111772951 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:43.325 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.461+0000 7f0c22ffd700 1 -- 172.21.15.70:0/4111772951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c2419e620 con 0x7f0c24106930 2024-04-01T01:23:43.325 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.462+0000 7f0c22ffd700 1 --2- 172.21.15.70:0/4111772951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c24106930 0x7f0c2407ba60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c000c00 tx=0x7f0c0c004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:43.325 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.462+0000 7f0c1bfff700 1 -- 172.21.15.70:0/4111772951 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c0c0039c0 con 0x7f0c24106930 2024-04-01T01:23:43.325 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.462+0000 7f0c1bfff700 1 -- 172.21.15.70:0/4111772951 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0c0c003b60 con 0x7f0c24106930 2024-04-01T01:23:43.325 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.462+0000 7f0c1bfff700 1 -- 172.21.15.70:0/4111772951 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c0c01a6f0 con 0x7f0c24106930 2024-04-01T01:23:43.325 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.462+0000 7f0c2942d700 1 -- 172.21.15.70:0/4111772951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c2419e7f0 con 0x7f0c24106930 2024-04-01T01:23:43.326 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.462+0000 7f0c2942d700 1 -- 172.21.15.70:0/4111772951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c2419ec10 con 0x7f0c24106930 2024-04-01T01:23:43.326 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.463+0000 7f0c2942d700 1 -- 172.21.15.70:0/4111772951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c24063870 con 0x7f0c24106930 2024-04-01T01:23:43.326 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.463+0000 7f0c1bfff700 1 -- 172.21.15.70:0/4111772951 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 4) ==== 46105+0+0 (secure 0 0 0) 0x7f0c0c01a890 con 0x7f0c24106930 2024-04-01T01:23:43.326 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.463+0000 7f0c1bfff700 1 --2- 172.21.15.70:0/4111772951 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f0c10038ec0 0x7f0c1003b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:43.326 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.463+0000 7f0c1bfff700 1 -- 172.21.15.70:0/4111772951 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0c0c04cff0 con 0x7f0c24106930 2024-04-01T01:23:43.326 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.463+0000 7f0c227fc700 1 --2- 172.21.15.70:0/4111772951 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f0c10038ec0 0x7f0c1003b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:43.326 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.463+0000 7f0c227fc700 1 --2- 172.21.15.70:0/4111772951 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f0c10038ec0 0x7f0c1003b380 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0c14006fd0 tx=0x7f0c14006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:43.326 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.468+0000 7f0c1bfff700 1 -- 172.21.15.70:0/4111772951 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0c0c04ca10 con 0x7f0c24106930 2024-04-01T01:23:43.326 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.735+0000 7f0c2942d700 1 -- 172.21.15.70:0/4111772951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f0c2419f430 con 0x7f0c24106930 2024-04-01T01:23:43.326 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.736+0000 7f0c1bfff700 1 -- 172.21.15.70:0/4111772951 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f0c0c016dc0 con 0x7f0c24106930 2024-04-01T01:23:43.326 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.738+0000 7f0c2942d700 1 -- 172.21.15.70:0/4111772951 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f0c10038ec0 msgr2=0x7f0c1003b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:43.326 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.738+0000 7f0c2942d700 1 --2- 172.21.15.70:0/4111772951 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f0c10038ec0 0x7f0c1003b380 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0c14006fd0 tx=0x7f0c14006e40 comp rx=0 tx=0).stop 2024-04-01T01:23:43.326 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.738+0000 7f0c2942d700 1 -- 172.21.15.70:0/4111772951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c24106930 msgr2=0x7f0c2407ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:43.327 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.738+0000 7f0c2942d700 1 --2- 172.21.15.70:0/4111772951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c24106930 0x7f0c2407ba60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c000c00 tx=0x7f0c0c004290 comp rx=0 tx=0).stop 2024-04-01T01:23:43.327 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.738+0000 7f0c2942d700 1 -- 172.21.15.70:0/4111772951 shutdown_connections 2024-04-01T01:23:43.327 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.738+0000 7f0c2942d700 1 --2- 172.21.15.70:0/4111772951 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f0c10038ec0 0x7f0c1003b380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:43.327 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.738+0000 7f0c2942d700 1 --2- 172.21.15.70:0/4111772951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c24106930 0x7f0c2407ba60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:43.327 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.738+0000 7f0c2942d700 1 -- 172.21.15.70:0/4111772951 >> 172.21.15.70:0/4111772951 conn(0x7f0c24101ed0 msgr2=0x7f0c24102940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:43.327 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.738+0000 7f0c2942d700 1 -- 172.21.15.70:0/4111772951 shutdown_connections 2024-04-01T01:23:43.327 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:42.738+0000 7f0c2942d700 1 -- 172.21.15.70:0/4111772951 wait complete. 2024-04-01T01:23:43.327 INFO:teuthology.orchestra.run.smithi070.stdout:mgr is available 2024-04-01T01:23:44.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:43 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2150270945' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-01T01:23:44.539 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-04-01T01:23:44.539 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [global] 2024-04-01T01:23:44.539 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout fsid = 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:23:44.539 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-01T01:23:44.539 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-01T01:23:44.540 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-01T01:23:44.540 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-01T01:23:44.540 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-04-01T01:23:44.540 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-01T01:23:44.540 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-01T01:23:44.540 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 2024-04-01T01:23:44.540 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout [osd] 2024-04-01T01:23:44.540 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-01T01:23:44.540 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-01T01:23:44.540 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-01T01:23:44.540 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-01T01:23:44.540 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-01T01:23:44.540 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.705+0000 7f6cd3f02700 1 Processor -- start 2024-04-01T01:23:44.540 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.705+0000 7f6cd3f02700 1 -- start start 2024-04-01T01:23:44.540 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.705+0000 7f6cd3f02700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc107f20 0x7f6ccc108340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:44.541 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.705+0000 7f6cd3f02700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ccc108910 con 0x7f6ccc107f20 2024-04-01T01:23:44.541 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.705+0000 7f6cd1c9e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc107f20 0x7f6ccc108340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:44.541 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.705+0000 7f6cd1c9e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc107f20 0x7f6ccc108340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53478/0 (socket says 172.21.15.70:53478) 2024-04-01T01:23:44.541 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.706+0000 7f6cd1c9e700 1 -- 172.21.15.70:0/4261296409 learned_addr learned my addr 172.21.15.70:0/4261296409 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:44.541 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.706+0000 7f6cd1c9e700 1 -- 172.21.15.70:0/4261296409 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ccc106b30 con 0x7f6ccc107f20 2024-04-01T01:23:44.541 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.706+0000 7f6cd1c9e700 1 --2- 172.21.15.70:0/4261296409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc107f20 0x7f6ccc108340 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6cc80098a0 tx=0x7f6cc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64dedc5f3332002c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:44.541 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.706+0000 7f6cd0c9c700 1 -- 172.21.15.70:0/4261296409 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cc8004160 con 0x7f6ccc107f20 2024-04-01T01:23:44.541 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.706+0000 7f6cd0c9c700 1 -- 172.21.15.70:0/4261296409 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6cc8004300 con 0x7f6ccc107f20 2024-04-01T01:23:44.541 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.706+0000 7f6cd0c9c700 1 -- 172.21.15.70:0/4261296409 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cc8011a80 con 0x7f6ccc107f20 2024-04-01T01:23:44.541 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.707+0000 7f6cd3f02700 1 -- 172.21.15.70:0/4261296409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc107f20 msgr2=0x7f6ccc108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:44.541 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.707+0000 7f6cd3f02700 1 --2- 172.21.15.70:0/4261296409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc107f20 0x7f6ccc108340 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6cc80098a0 tx=0x7f6cc8009bb0 comp rx=0 tx=0).stop 2024-04-01T01:23:44.541 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.707+0000 7f6cd3f02700 1 -- 172.21.15.70:0/4261296409 shutdown_connections 2024-04-01T01:23:44.541 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.707+0000 7f6cd3f02700 1 --2- 172.21.15.70:0/4261296409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc107f20 0x7f6ccc108340 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:44.541 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.707+0000 7f6cd3f02700 1 -- 172.21.15.70:0/4261296409 >> 172.21.15.70:0/4261296409 conn(0x7f6ccc101620 msgr2=0x7f6ccc103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:44.541 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.707+0000 7f6cd3f02700 1 -- 172.21.15.70:0/4261296409 shutdown_connections 2024-04-01T01:23:44.542 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.707+0000 7f6cd3f02700 1 -- 172.21.15.70:0/4261296409 wait complete. 2024-04-01T01:23:44.542 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.707+0000 7f6cd3f02700 1 Processor -- start 2024-04-01T01:23:44.542 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.707+0000 7f6cd3f02700 1 -- start start 2024-04-01T01:23:44.542 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.708+0000 7f6cd3f02700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc107f20 0x7f6ccc199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:44.542 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.708+0000 7f6cd3f02700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ccc105a20 con 0x7f6ccc107f20 2024-04-01T01:23:44.542 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.708+0000 7f6cd1c9e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc107f20 0x7f6ccc199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:44.542 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.708+0000 7f6cd1c9e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc107f20 0x7f6ccc199700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53492/0 (socket says 172.21.15.70:53492) 2024-04-01T01:23:44.542 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.708+0000 7f6cd1c9e700 1 -- 172.21.15.70:0/2150270945 learned_addr learned my addr 172.21.15.70:0/2150270945 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:44.542 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.708+0000 7f6cd1c9e700 1 -- 172.21.15.70:0/2150270945 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ccc199c40 con 0x7f6ccc107f20 2024-04-01T01:23:44.542 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.708+0000 7f6cd1c9e700 1 --2- 172.21.15.70:0/2150270945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc107f20 0x7f6ccc199700 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8006af0 tx=0x7f6cc8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:44.542 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.708+0000 7f6cc2ffd700 1 -- 172.21.15.70:0/2150270945 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cc8003930 con 0x7f6ccc107f20 2024-04-01T01:23:44.542 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.708+0000 7f6cc2ffd700 1 -- 172.21.15.70:0/2150270945 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6cc8003ad0 con 0x7f6ccc107f20 2024-04-01T01:23:44.543 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.708+0000 7f6cd3f02700 1 -- 172.21.15.70:0/2150270945 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ccc199ed0 con 0x7f6ccc107f20 2024-04-01T01:23:44.543 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.708+0000 7f6cc2ffd700 1 -- 172.21.15.70:0/2150270945 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6cc80195e0 con 0x7f6ccc107f20 2024-04-01T01:23:44.543 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.708+0000 7f6cd3f02700 1 -- 172.21.15.70:0/2150270945 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ccc19cbc0 con 0x7f6ccc107f20 2024-04-01T01:23:44.543 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.709+0000 7f6cc2ffd700 1 -- 172.21.15.70:0/2150270945 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 4) ==== 46105+0+0 (secure 0 0 0) 0x7f6cc80198d0 con 0x7f6ccc107f20 2024-04-01T01:23:44.543 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.709+0000 7f6cc2ffd700 1 --2- 172.21.15.70:0/2150270945 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f6cb8038e70 0x7f6cb803b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:44.543 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.709+0000 7f6cc2ffd700 1 -- 172.21.15.70:0/2150270945 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6cc804d220 con 0x7f6ccc107f20 2024-04-01T01:23:44.543 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.709+0000 7f6cd3f02700 1 -- 172.21.15.70:0/2150270945 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ccc063870 con 0x7f6ccc107f20 2024-04-01T01:23:44.543 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.709+0000 7f6cd149d700 1 --2- 172.21.15.70:0/2150270945 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f6cb8038e70 0x7f6cb803b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:44.543 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.710+0000 7f6cd149d700 1 --2- 172.21.15.70:0/2150270945 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f6cb8038e70 0x7f6cb803b330 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc006fd0 tx=0x7f6cbc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:44.543 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.714+0000 7f6cc2ffd700 1 -- 172.21.15.70:0/2150270945 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6cc802d960 con 0x7f6ccc107f20 2024-04-01T01:23:44.543 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.906+0000 7f6cd3f02700 1 -- 172.21.15.70:0/2150270945 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f6ccc1a26c0 con 0x7f6ccc107f20 2024-04-01T01:23:44.543 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.907+0000 7f6cc2ffd700 1 -- 172.21.15.70:0/2150270945 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f6cc8032020 con 0x7f6ccc107f20 2024-04-01T01:23:44.543 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.909+0000 7f6cd3f02700 1 -- 172.21.15.70:0/2150270945 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f6cb8038e70 msgr2=0x7f6cb803b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:44.543 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.909+0000 7f6cd3f02700 1 --2- 172.21.15.70:0/2150270945 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f6cb8038e70 0x7f6cb803b330 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc006fd0 tx=0x7f6cbc006e40 comp rx=0 tx=0).stop 2024-04-01T01:23:44.543 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.909+0000 7f6cd3f02700 1 -- 172.21.15.70:0/2150270945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc107f20 msgr2=0x7f6ccc199700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:44.544 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.909+0000 7f6cd3f02700 1 --2- 172.21.15.70:0/2150270945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc107f20 0x7f6ccc199700 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8006af0 tx=0x7f6cc8011b70 comp rx=0 tx=0).stop 2024-04-01T01:23:44.544 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.909+0000 7f6cd3f02700 1 -- 172.21.15.70:0/2150270945 shutdown_connections 2024-04-01T01:23:44.544 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.909+0000 7f6cd3f02700 1 --2- 172.21.15.70:0/2150270945 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f6cb8038e70 0x7f6cb803b330 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:44.544 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.909+0000 7f6cd3f02700 1 --2- 172.21.15.70:0/2150270945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ccc107f20 0x7f6ccc199700 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:44.544 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.909+0000 7f6cd3f02700 1 -- 172.21.15.70:0/2150270945 >> 172.21.15.70:0/2150270945 conn(0x7f6ccc101620 msgr2=0x7f6ccc18fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:44.544 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.909+0000 7f6cd3f02700 1 -- 172.21.15.70:0/2150270945 shutdown_connections 2024-04-01T01:23:44.544 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:43.909+0000 7f6cd3f02700 1 -- 172.21.15.70:0/2150270945 wait complete. 2024-04-01T01:23:44.544 INFO:teuthology.orchestra.run.smithi070.stdout:Enabling cephadm module... 2024-04-01T01:23:45.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:45 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1944104631' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-01T01:23:45.504 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:45 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: ignoring --setuser ceph since I am not root 2024-04-01T01:23:45.504 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:45 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: ignoring --setgroup ceph since I am not root 2024-04-01T01:23:46.243 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:45 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:45.796+0000 7f0728587200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-01T01:23:46.244 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:45 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:45.890+0000 7f0728587200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-01T01:23:46.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:46 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1944104631' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-01T01:23:46.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:46 smithi070 ceph-mon[32179]: mgrmap e5: x(active, since 10s) 2024-04-01T01:23:46.859 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:46 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:46.534+0000 7f0728587200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-01T01:23:46.859 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:46 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:46.630+0000 7f0728587200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-01T01:23:46.859 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:46 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:46.773+0000 7f0728587200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-01T01:23:47.134 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:46 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:46.869+0000 7f0728587200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-01T01:23:47.134 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:46 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:46.952+0000 7f0728587200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-01T01:23:47.141 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.933+0000 7fdf64011700 1 Processor -- start 2024-04-01T01:23:47.141 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.933+0000 7fdf64011700 1 -- start start 2024-04-01T01:23:47.141 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.933+0000 7fdf64011700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf5c106930 0x7fdf5c106d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:47.141 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.933+0000 7fdf64011700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c107320 con 0x7fdf5c106930 2024-04-01T01:23:47.141 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.933+0000 7fdf61dad700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf5c106930 0x7fdf5c106d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:47.141 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.933+0000 7fdf61dad700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf5c106930 0x7fdf5c106d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53504/0 (socket says 172.21.15.70:53504) 2024-04-01T01:23:47.141 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.933+0000 7fdf61dad700 1 -- 172.21.15.70:0/1020692118 learned_addr learned my addr 172.21.15.70:0/1020692118 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:47.141 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.933+0000 7fdf61dad700 1 -- 172.21.15.70:0/1020692118 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5c107b30 con 0x7fdf5c106930 2024-04-01T01:23:47.142 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.934+0000 7fdf61dad700 1 --2- 172.21.15.70:0/1020692118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf5c106930 0x7fdf5c106d50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdf5800bb40 tx=0x7fdf5800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4110cd876808c75c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:47.142 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.934+0000 7fdf60dab700 1 -- 172.21.15.70:0/1020692118 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf5800dbe0 con 0x7fdf5c106930 2024-04-01T01:23:47.142 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.934+0000 7fdf60dab700 1 -- 172.21.15.70:0/1020692118 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fdf580044c0 con 0x7fdf5c106930 2024-04-01T01:23:47.142 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.934+0000 7fdf60dab700 1 -- 172.21.15.70:0/1020692118 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf58010470 con 0x7fdf5c106930 2024-04-01T01:23:47.142 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.934+0000 7fdf64011700 1 -- 172.21.15.70:0/1020692118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf5c106930 msgr2=0x7fdf5c106d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:47.142 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.934+0000 7fdf64011700 1 --2- 172.21.15.70:0/1020692118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf5c106930 0x7fdf5c106d50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdf5800bb40 tx=0x7fdf5800be50 comp rx=0 tx=0).stop 2024-04-01T01:23:47.142 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.934+0000 7fdf64011700 1 -- 172.21.15.70:0/1020692118 shutdown_connections 2024-04-01T01:23:47.142 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.934+0000 7fdf64011700 1 --2- 172.21.15.70:0/1020692118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf5c106930 0x7fdf5c106d50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:47.142 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.934+0000 7fdf64011700 1 -- 172.21.15.70:0/1020692118 >> 172.21.15.70:0/1020692118 conn(0x7fdf5c101ed0 msgr2=0x7fdf5c104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:47.142 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.934+0000 7fdf64011700 1 -- 172.21.15.70:0/1020692118 shutdown_connections 2024-04-01T01:23:47.142 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.935+0000 7fdf64011700 1 -- 172.21.15.70:0/1020692118 wait complete. 2024-04-01T01:23:47.142 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.935+0000 7fdf64011700 1 Processor -- start 2024-04-01T01:23:47.142 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.935+0000 7fdf64011700 1 -- start start 2024-04-01T01:23:47.142 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.935+0000 7fdf64011700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf5c106930 0x7fdf5c1996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:47.143 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.935+0000 7fdf64011700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c07afc0 con 0x7fdf5c106930 2024-04-01T01:23:47.143 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.935+0000 7fdf61dad700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf5c106930 0x7fdf5c1996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:47.143 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.935+0000 7fdf61dad700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf5c106930 0x7fdf5c1996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53512/0 (socket says 172.21.15.70:53512) 2024-04-01T01:23:47.143 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.935+0000 7fdf61dad700 1 -- 172.21.15.70:0/1944104631 learned_addr learned my addr 172.21.15.70:0/1944104631 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:47.143 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.936+0000 7fdf61dad700 1 -- 172.21.15.70:0/1944104631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5c199c10 con 0x7fdf5c106930 2024-04-01T01:23:47.143 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.936+0000 7fdf61dad700 1 --2- 172.21.15.70:0/1944104631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf5c106930 0x7fdf5c1996d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdf5800b340 tx=0x7fdf58003fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:47.143 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.936+0000 7fdf52ffd700 1 -- 172.21.15.70:0/1944104631 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf58027070 con 0x7fdf5c106930 2024-04-01T01:23:47.143 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.936+0000 7fdf64011700 1 -- 172.21.15.70:0/1944104631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf5c199ea0 con 0x7fdf5c106930 2024-04-01T01:23:47.143 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.936+0000 7fdf52ffd700 1 -- 172.21.15.70:0/1944104631 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fdf580104e0 con 0x7fdf5c106930 2024-04-01T01:23:47.143 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.936+0000 7fdf52ffd700 1 -- 172.21.15.70:0/1944104631 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf580229e0 con 0x7fdf5c106930 2024-04-01T01:23:47.143 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.936+0000 7fdf64011700 1 -- 172.21.15.70:0/1944104631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf5c19cb90 con 0x7fdf5c106930 2024-04-01T01:23:47.143 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.937+0000 7fdf52ffd700 1 -- 172.21.15.70:0/1944104631 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 4) ==== 46105+0+0 (secure 0 0 0) 0x7fdf5802a070 con 0x7fdf5c106930 2024-04-01T01:23:47.143 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.937+0000 7fdf64011700 1 -- 172.21.15.70:0/1944104631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf5c063870 con 0x7fdf5c106930 2024-04-01T01:23:47.144 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.937+0000 7fdf52ffd700 1 --2- 172.21.15.70:0/1944104631 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fdf48038e70 0x7fdf4803b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:47.144 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.937+0000 7fdf52ffd700 1 -- 172.21.15.70:0/1944104631 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdf5801f070 con 0x7fdf5c106930 2024-04-01T01:23:47.144 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.937+0000 7fdf615ac700 1 --2- 172.21.15.70:0/1944104631 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fdf48038e70 0x7fdf4803b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:47.144 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.937+0000 7fdf615ac700 1 --2- 172.21.15.70:0/1944104631 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fdf48038e70 0x7fdf4803b330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c006fd0 tx=0x7fdf4c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:47.144 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:44.942+0000 7fdf52ffd700 1 -- 172.21.15.70:0/1944104631 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdf58015070 con 0x7fdf5c106930 2024-04-01T01:23:47.144 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:45.188+0000 7fdf64011700 1 -- 172.21.15.70:0/1944104631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fdf5c19cd60 con 0x7fdf5c106930 2024-04-01T01:23:47.144 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:45.239+0000 7fdf52ffd700 1 -- 172.21.15.70:0/1944104631 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fdf5c19cd60 con 0x7fdf5c106930 2024-04-01T01:23:47.144 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:45.241+0000 7fdf52ffd700 1 -- 172.21.15.70:0/1944104631 <== mon.0 v2:172.21.15.70:3300/0 8 ==== mgrmap(e 5) ==== 46116+0+0 (secure 0 0 0) 0x7fdf58010810 con 0x7fdf5c106930 2024-04-01T01:23:47.144 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:45.242+0000 7fdf64011700 1 -- 172.21.15.70:0/1944104631 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fdf48038e70 msgr2=0x7fdf4803b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:47.144 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:45.242+0000 7fdf64011700 1 --2- 172.21.15.70:0/1944104631 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fdf48038e70 0x7fdf4803b330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c006fd0 tx=0x7fdf4c006e40 comp rx=0 tx=0).stop 2024-04-01T01:23:47.144 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:45.242+0000 7fdf64011700 1 -- 172.21.15.70:0/1944104631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf5c106930 msgr2=0x7fdf5c1996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:47.144 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:45.242+0000 7fdf64011700 1 --2- 172.21.15.70:0/1944104631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf5c106930 0x7fdf5c1996d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdf5800b340 tx=0x7fdf58003fe0 comp rx=0 tx=0).stop 2024-04-01T01:23:47.144 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:45.242+0000 7fdf64011700 1 -- 172.21.15.70:0/1944104631 shutdown_connections 2024-04-01T01:23:47.145 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:45.242+0000 7fdf64011700 1 --2- 172.21.15.70:0/1944104631 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fdf48038e70 0x7fdf4803b330 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:47.145 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:45.242+0000 7fdf64011700 1 --2- 172.21.15.70:0/1944104631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf5c106930 0x7fdf5c1996d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:47.145 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:45.243+0000 7fdf64011700 1 -- 172.21.15.70:0/1944104631 >> 172.21.15.70:0/1944104631 conn(0x7fdf5c101ed0 msgr2=0x7fdf5c18fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:47.145 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:45.243+0000 7fdf64011700 1 -- 172.21.15.70:0/1944104631 shutdown_connections 2024-04-01T01:23:47.145 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:45.243+0000 7fdf64011700 1 -- 172.21.15.70:0/1944104631 wait complete. 2024-04-01T01:23:47.388 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:47 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:47.222+0000 7f0728587200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-01T01:23:47.388 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:47 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:47.333+0000 7f0728587200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-01T01:23:47.706 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:47 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:47.425+0000 7f0728587200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-01T01:23:47.706 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:47 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:47.523+0000 7f0728587200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-01T01:23:48.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:47 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3353054494' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-01T01:23:48.004 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:47 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:47.706+0000 7f0728587200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-01T01:23:48.399 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-04-01T01:23:48.399 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-01T01:23:48.399 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-01T01:23:48.399 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-01T01:23:48.399 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-01T01:23:48.399 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-04-01T01:23:48.399 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.569+0000 7f297a686700 1 Processor -- start 2024-04-01T01:23:48.399 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.569+0000 7f297a686700 1 -- start start 2024-04-01T01:23:48.399 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.569+0000 7f297a686700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2974106930 0x7f2974106d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:48.399 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.569+0000 7f297a686700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2974107320 con 0x7f2974106930 2024-04-01T01:23:48.399 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.569+0000 7f2973fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2974106930 0x7f2974106d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:48.399 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.569+0000 7f2973fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2974106930 0x7f2974106d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57108/0 (socket says 172.21.15.70:57108) 2024-04-01T01:23:48.400 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.569+0000 7f2973fff700 1 -- 172.21.15.70:0/2339553084 learned_addr learned my addr 172.21.15.70:0/2339553084 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:48.400 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.570+0000 7f2973fff700 1 -- 172.21.15.70:0/2339553084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2974107b30 con 0x7f2974106930 2024-04-01T01:23:48.400 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.570+0000 7f2973fff700 1 --2- 172.21.15.70:0/2339553084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2974106930 0x7f2974106d50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f29680098a0 tx=0x7f2968009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89cd6556f03d290b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:48.400 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.570+0000 7f2972ffd700 1 -- 172.21.15.70:0/2339553084 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2968004160 con 0x7f2974106930 2024-04-01T01:23:48.400 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.570+0000 7f2972ffd700 1 -- 172.21.15.70:0/2339553084 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2968004300 con 0x7f2974106930 2024-04-01T01:23:48.400 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.570+0000 7f2972ffd700 1 -- 172.21.15.70:0/2339553084 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2968011a80 con 0x7f2974106930 2024-04-01T01:23:48.400 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.571+0000 7f297a686700 1 -- 172.21.15.70:0/2339553084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2974106930 msgr2=0x7f2974106d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:48.400 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.571+0000 7f297a686700 1 --2- 172.21.15.70:0/2339553084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2974106930 0x7f2974106d50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f29680098a0 tx=0x7f2968009bb0 comp rx=0 tx=0).stop 2024-04-01T01:23:48.400 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.571+0000 7f297a686700 1 -- 172.21.15.70:0/2339553084 shutdown_connections 2024-04-01T01:23:48.400 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.571+0000 7f297a686700 1 --2- 172.21.15.70:0/2339553084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2974106930 0x7f2974106d50 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:48.400 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.571+0000 7f297a686700 1 -- 172.21.15.70:0/2339553084 >> 172.21.15.70:0/2339553084 conn(0x7f2974101ed0 msgr2=0x7f2974104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:48.400 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.571+0000 7f297a686700 1 -- 172.21.15.70:0/2339553084 shutdown_connections 2024-04-01T01:23:48.400 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.571+0000 7f297a686700 1 -- 172.21.15.70:0/2339553084 wait complete. 2024-04-01T01:23:48.400 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.572+0000 7f297a686700 1 Processor -- start 2024-04-01T01:23:48.400 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.572+0000 7f297a686700 1 -- start start 2024-04-01T01:23:48.401 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.572+0000 7f297a686700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2974106930 0x7f297407ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:48.401 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.572+0000 7f297a686700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f297410aee0 con 0x7f2974106930 2024-04-01T01:23:48.401 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.572+0000 7f2973fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2974106930 0x7f297407ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:48.401 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.572+0000 7f2973fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2974106930 0x7f297407ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57114/0 (socket says 172.21.15.70:57114) 2024-04-01T01:23:48.401 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.572+0000 7f2973fff700 1 -- 172.21.15.70:0/3353054494 learned_addr learned my addr 172.21.15.70:0/3353054494 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:48.401 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.573+0000 7f2973fff700 1 -- 172.21.15.70:0/3353054494 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f297419e620 con 0x7f2974106930 2024-04-01T01:23:48.401 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.573+0000 7f2973fff700 1 --2- 172.21.15.70:0/3353054494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2974106930 0x7f297407ba60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2968006af0 tx=0x7f2968011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:48.401 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.573+0000 7f29717fa700 1 -- 172.21.15.70:0/3353054494 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2968003930 con 0x7f2974106930 2024-04-01T01:23:48.401 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.573+0000 7f29717fa700 1 -- 172.21.15.70:0/3353054494 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2968003ad0 con 0x7f2974106930 2024-04-01T01:23:48.401 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.573+0000 7f297a686700 1 -- 172.21.15.70:0/3353054494 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f297419e7f0 con 0x7f2974106930 2024-04-01T01:23:48.401 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.573+0000 7f29717fa700 1 -- 172.21.15.70:0/3353054494 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29680195e0 con 0x7f2974106930 2024-04-01T01:23:48.401 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.573+0000 7f297a686700 1 -- 172.21.15.70:0/3353054494 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f297419ec10 con 0x7f2974106930 2024-04-01T01:23:48.401 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.574+0000 7f29717fa700 1 -- 172.21.15.70:0/3353054494 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 46116+0+0 (secure 0 0 0) 0x7f2968019890 con 0x7f2974106930 2024-04-01T01:23:48.401 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.574+0000 7f297a686700 1 -- 172.21.15.70:0/3353054494 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2974063870 con 0x7f2974106930 2024-04-01T01:23:48.402 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.574+0000 7f29717fa700 1 --2- 172.21.15.70:0/3353054494 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f295c038ec0 0x7f295c03b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:48.402 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.574+0000 7f29717fa700 1 -- 172.21.15.70:0/3353054494 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f296804d250 con 0x7f2974106930 2024-04-01T01:23:48.402 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.574+0000 7f29737fe700 1 -- 172.21.15.70:0/3353054494 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f295c038ec0 msgr2=0x7f295c03b380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2121290979 2024-04-01T01:23:48.402 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.574+0000 7f29737fe700 1 --2- 172.21.15.70:0/3353054494 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f295c038ec0 0x7f295c03b380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-01T01:23:48.402 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.583+0000 7f29717fa700 1 -- 172.21.15.70:0/3353054494 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2968020240 con 0x7f2974106930 2024-04-01T01:23:48.402 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.775+0000 7f29737fe700 1 -- 172.21.15.70:0/3353054494 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f295c038ec0 msgr2=0x7f295c03b380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2121290979 2024-04-01T01:23:48.402 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.775+0000 7f29737fe700 1 --2- 172.21.15.70:0/3353054494 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f295c038ec0 0x7f295c03b380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-01T01:23:48.402 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.841+0000 7f297a686700 1 -- 172.21.15.70:0/3353054494 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f297419f430 con 0x7f2974106930 2024-04-01T01:23:48.402 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.841+0000 7f29717fa700 1 -- 172.21.15.70:0/3353054494 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f296800bc40 con 0x7f2974106930 2024-04-01T01:23:48.402 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.843+0000 7f297a686700 1 -- 172.21.15.70:0/3353054494 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f295c038ec0 msgr2=0x7f295c03b380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:23:48.402 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.843+0000 7f297a686700 1 --2- 172.21.15.70:0/3353054494 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f295c038ec0 0x7f295c03b380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:48.402 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.843+0000 7f297a686700 1 -- 172.21.15.70:0/3353054494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2974106930 msgr2=0x7f297407ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:48.402 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.843+0000 7f297a686700 1 --2- 172.21.15.70:0/3353054494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2974106930 0x7f297407ba60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2968006af0 tx=0x7f2968011b70 comp rx=0 tx=0).stop 2024-04-01T01:23:48.403 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.844+0000 7f297a686700 1 -- 172.21.15.70:0/3353054494 shutdown_connections 2024-04-01T01:23:48.403 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.844+0000 7f297a686700 1 --2- 172.21.15.70:0/3353054494 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7f295c038ec0 0x7f295c03b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:48.403 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.844+0000 7f297a686700 1 --2- 172.21.15.70:0/3353054494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2974106930 0x7f297407ba60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:48.403 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.844+0000 7f297a686700 1 -- 172.21.15.70:0/3353054494 >> 172.21.15.70:0/3353054494 conn(0x7f2974101ed0 msgr2=0x7f2974102940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:48.403 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.844+0000 7f297a686700 1 -- 172.21.15.70:0/3353054494 shutdown_connections 2024-04-01T01:23:48.403 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:47.844+0000 7f297a686700 1 -- 172.21.15.70:0/3353054494 wait complete. 2024-04-01T01:23:48.403 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for the mgr to restart... 2024-04-01T01:23:48.403 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mgr epoch 5... 2024-04-01T01:23:48.998 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:48 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:48.681+0000 7f0728587200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-01T01:23:48.998 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:48 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:48.819+0000 7f0728587200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-01T01:23:49.253 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:49 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:49.151+0000 7f0728587200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-01T01:23:49.253 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:49 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:49.239+0000 7f0728587200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-01T01:23:49.753 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:49 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:49.433+0000 7f0728587200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-01T01:23:49.754 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:49 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:49.616+0000 7f0728587200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-01T01:23:50.158 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:49 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:49.901+0000 7f0728587200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-01T01:23:50.503 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:50 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:50.157+0000 7f0728587200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-01T01:23:50.504 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:50 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:50.394+0000 7f0728587200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-01T01:23:51.503 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:51 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:51.197+0000 7f0728587200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-01T01:23:52.003 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:51 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:51.771+0000 7f0728587200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-01T01:23:52.004 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:51 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:51.869+0000 7f0728587200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-01T01:23:52.503 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:23:52.088+0000 7f0728587200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-01T01:23:52.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-mon[32179]: Active manager daemon x restarted 2024-04-01T01:23:52.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-mon[32179]: Activating manager daemon x 2024-04-01T01:23:52.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-mon[32179]: osdmap e2: 0 total, 0 up, 0 in 2024-04-01T01:23:52.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-mon[32179]: mgrmap e6: x(active, starting, since 0.00317709s) 2024-04-01T01:23:52.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:23:52.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-01T01:23:52.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-01T01:23:52.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-01T01:23:52.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-01T01:23:52.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-mon[32179]: Manager daemon x is now available 2024-04-01T01:23:52.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' 2024-04-01T01:23:52.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' 2024-04-01T01:23:52.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:23:52.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:23:52.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-01T01:23:52.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:52 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-01T01:23:53.396 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:53 smithi070 ceph-mon[32179]: Found migration_current of "None". Setting to last migration. 2024-04-01T01:23:53.396 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:53 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' 2024-04-01T01:23:53.396 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:53 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' 2024-04-01T01:23:53.396 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:53 smithi070 ceph-mon[32179]: mgrmap e7: x(active, since 1.00589s) 2024-04-01T01:23:53.396 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:53 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:23:53.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-04-01T01:23:53.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-01T01:23:53.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-01T01:23:53.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-04-01T01:23:53.672 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.792+0000 7fbcfe893700 1 Processor -- start 2024-04-01T01:23:53.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.792+0000 7fbcfe893700 1 -- start start 2024-04-01T01:23:53.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.793+0000 7fbcfe893700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf81060c0 0x7fbcf81064e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:53.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.793+0000 7fbcfe893700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf8106ab0 con 0x7fbcf81060c0 2024-04-01T01:23:53.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.793+0000 7fbcf7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf81060c0 0x7fbcf81064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:53.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.793+0000 7fbcf7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf81060c0 0x7fbcf81064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57118/0 (socket says 172.21.15.70:57118) 2024-04-01T01:23:53.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.793+0000 7fbcf7fff700 1 -- 172.21.15.70:0/2129169656 learned_addr learned my addr 172.21.15.70:0/2129169656 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:53.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.793+0000 7fbcf7fff700 1 -- 172.21.15.70:0/2129169656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf81072c0 con 0x7fbcf81060c0 2024-04-01T01:23:53.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.793+0000 7fbcf7fff700 1 --2- 172.21.15.70:0/2129169656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf81060c0 0x7fbcf81064e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbce800aa40 tx=0x7fbce800ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c06ccd15502a86 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:53.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.793+0000 7fbcf6ffd700 1 -- 172.21.15.70:0/2129169656 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbce800b040 con 0x7fbcf81060c0 2024-04-01T01:23:53.673 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.794+0000 7fbcf6ffd700 1 -- 172.21.15.70:0/2129169656 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fbce800c8f0 con 0x7fbcf81060c0 2024-04-01T01:23:53.674 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.794+0000 7fbcf6ffd700 1 -- 172.21.15.70:0/2129169656 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbce8003b70 con 0x7fbcf81060c0 2024-04-01T01:23:53.674 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.794+0000 7fbcfe893700 1 -- 172.21.15.70:0/2129169656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf81060c0 msgr2=0x7fbcf81064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:53.674 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.794+0000 7fbcfe893700 1 --2- 172.21.15.70:0/2129169656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf81060c0 0x7fbcf81064e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbce800aa40 tx=0x7fbce800ad50 comp rx=0 tx=0).stop 2024-04-01T01:23:53.674 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.794+0000 7fbcfe893700 1 -- 172.21.15.70:0/2129169656 shutdown_connections 2024-04-01T01:23:53.674 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.794+0000 7fbcfe893700 1 --2- 172.21.15.70:0/2129169656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf81060c0 0x7fbcf81064e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:53.674 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.794+0000 7fbcfe893700 1 -- 172.21.15.70:0/2129169656 >> 172.21.15.70:0/2129169656 conn(0x7fbcf8101620 msgr2=0x7fbcf8103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:53.674 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.794+0000 7fbcfe893700 1 -- 172.21.15.70:0/2129169656 shutdown_connections 2024-04-01T01:23:53.674 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.794+0000 7fbcfe893700 1 -- 172.21.15.70:0/2129169656 wait complete. 2024-04-01T01:23:53.674 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.795+0000 7fbcfe893700 1 Processor -- start 2024-04-01T01:23:53.674 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.795+0000 7fbcfe893700 1 -- start start 2024-04-01T01:23:53.674 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.795+0000 7fbcfe893700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf81060c0 0x7fbcf818ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:53.674 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.795+0000 7fbcfe893700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf81079e0 con 0x7fbcf81060c0 2024-04-01T01:23:53.675 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.795+0000 7fbcf7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf81060c0 0x7fbcf818ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:53.675 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.795+0000 7fbcf7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf81060c0 0x7fbcf818ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57120/0 (socket says 172.21.15.70:57120) 2024-04-01T01:23:53.675 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.795+0000 7fbcf7fff700 1 -- 172.21.15.70:0/1279230738 learned_addr learned my addr 172.21.15.70:0/1279230738 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:53.675 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.796+0000 7fbcf7fff700 1 -- 172.21.15.70:0/1279230738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf818f1c0 con 0x7fbcf81060c0 2024-04-01T01:23:53.675 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.796+0000 7fbcf7fff700 1 --2- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf81060c0 0x7fbcf818ec80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbce800c550 tx=0x7fbce8004110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:53.675 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.796+0000 7fbcf57fa700 1 -- 172.21.15.70:0/1279230738 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbce8025070 con 0x7fbcf81060c0 2024-04-01T01:23:53.675 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.796+0000 7fbcf57fa700 1 -- 172.21.15.70:0/1279230738 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fbce800cdf0 con 0x7fbcf81060c0 2024-04-01T01:23:53.675 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.796+0000 7fbcfe893700 1 -- 172.21.15.70:0/1279230738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcf818d3b0 con 0x7fbcf81060c0 2024-04-01T01:23:53.675 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.796+0000 7fbcf57fa700 1 -- 172.21.15.70:0/1279230738 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbce8010740 con 0x7fbcf81060c0 2024-04-01T01:23:53.675 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.796+0000 7fbcfe893700 1 -- 172.21.15.70:0/1279230738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcf818d8f0 con 0x7fbcf81060c0 2024-04-01T01:23:53.675 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.797+0000 7fbcf57fa700 1 -- 172.21.15.70:0/1279230738 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 46116+0+0 (secure 0 0 0) 0x7fbce8028070 con 0x7fbcf81060c0 2024-04-01T01:23:53.675 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.797+0000 7fbcf57fa700 1 --2- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fbce0038e70 0x7fbce003b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:53.675 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.797+0000 7fbcf57fa700 1 -- 172.21.15.70:0/1279230738 --> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbce003ba40 con 0x7fbce0038e70 2024-04-01T01:23:53.675 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.797+0000 7fbcf57fa700 1 -- 172.21.15.70:0/1279230738 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbce804cbd0 con 0x7fbcf81060c0 2024-04-01T01:23:53.676 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.797+0000 7fbcf77fe700 1 -- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fbce0038e70 msgr2=0x7fbce003b330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2121290979 2024-04-01T01:23:53.676 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.797+0000 7fbcf77fe700 1 --2- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fbce0038e70 0x7fbce003b330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-01T01:23:53.676 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.998+0000 7fbcf77fe700 1 -- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fbce0038e70 msgr2=0x7fbce003b330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2121290979 2024-04-01T01:23:53.676 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:48.998+0000 7fbcf77fe700 1 --2- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fbce0038e70 0x7fbce003b330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-01T01:23:53.676 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:49.399+0000 7fbcf77fe700 1 -- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fbce0038e70 msgr2=0x7fbce003b330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2121290979 2024-04-01T01:23:53.676 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:49.399+0000 7fbcf77fe700 1 --2- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fbce0038e70 0x7fbce003b330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-01T01:23:53.676 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:50.200+0000 7fbcf77fe700 1 -- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fbce0038e70 msgr2=0x7fbce003b330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2121290979 2024-04-01T01:23:53.676 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:50.200+0000 7fbcf77fe700 1 --2- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fbce0038e70 0x7fbce003b330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-01T01:23:53.676 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:51.801+0000 7fbcf77fe700 1 -- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fbce0038e70 msgr2=0x7fbce003b330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2121290979 2024-04-01T01:23:53.676 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:51.801+0000 7fbcf77fe700 1 --2- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fbce0038e70 0x7fbce003b330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-01T01:23:53.676 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:52.091+0000 7fbcf57fa700 1 -- 172.21.15.70:0/1279230738 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mgrmap(e 6) ==== 45883+0+0 (secure 0 0 0) 0x7fbce8010980 con 0x7fbcf81060c0 2024-04-01T01:23:53.676 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:52.092+0000 7fbcf57fa700 1 -- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fbce0038e70 msgr2=0x7fbce003b330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:23:53.676 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:52.092+0000 7fbcf57fa700 1 --2- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fbce0038e70 0x7fbce003b330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:53.677 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.093+0000 7fbcf57fa700 1 -- 172.21.15.70:0/1279230738 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 7) ==== 46010+0+0 (secure 0 0 0) 0x7fbce804e990 con 0x7fbcf81060c0 2024-04-01T01:23:53.677 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.093+0000 7fbcf57fa700 1 --2- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fbce003c8c0 0x7fbce003ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:53.677 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.094+0000 7fbcf57fa700 1 -- 172.21.15.70:0/1279230738 --> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbce8023e70 con 0x7fbce003c8c0 2024-04-01T01:23:53.677 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.094+0000 7fbcf77fe700 1 --2- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fbce003c8c0 0x7fbce003ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:53.677 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.094+0000 7fbcf77fe700 1 --2- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fbce003c8c0 0x7fbce003ecb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fbcec003a10 tx=0x7fbcec009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:53.677 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.095+0000 7fbcf57fa700 1 -- 172.21.15.70:0/1279230738 <== mgr.14118 v2:172.21.15.70:6800/2836712828 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fbce8023e70 con 0x7fbce003c8c0 2024-04-01T01:23:53.677 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.101+0000 7fbcfe893700 1 -- 172.21.15.70:0/1279230738 --> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fbcf8063870 con 0x7fbce003c8c0 2024-04-01T01:23:53.677 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.101+0000 7fbcf57fa700 1 -- 172.21.15.70:0/1279230738 <== mgr.14118 v2:172.21.15.70:6800/2836712828 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fbcf8063870 con 0x7fbce003c8c0 2024-04-01T01:23:53.677 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.102+0000 7fbcfe893700 1 -- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fbce003c8c0 msgr2=0x7fbce003ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:53.677 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.102+0000 7fbcfe893700 1 --2- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fbce003c8c0 0x7fbce003ecb0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fbcec003a10 tx=0x7fbcec009290 comp rx=0 tx=0).stop 2024-04-01T01:23:53.677 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.102+0000 7fbcfe893700 1 -- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf81060c0 msgr2=0x7fbcf818ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:53.677 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.102+0000 7fbcfe893700 1 --2- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf81060c0 0x7fbcf818ec80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbce800c550 tx=0x7fbce8004110 comp rx=0 tx=0).stop 2024-04-01T01:23:53.677 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.102+0000 7fbcfe893700 1 -- 172.21.15.70:0/1279230738 shutdown_connections 2024-04-01T01:23:53.677 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.102+0000 7fbcfe893700 1 --2- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fbce003c8c0 0x7fbce003ecb0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:53.678 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.102+0000 7fbcfe893700 1 --2- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf81060c0 0x7fbcf818ec80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:53.678 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.102+0000 7fbcfe893700 1 --2- 172.21.15.70:0/1279230738 >> [v2:172.21.15.70:6800/2121290979,v1:172.21.15.70:6801/2121290979] conn(0x7fbce0038e70 0x7fbce003b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:53.678 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.102+0000 7fbcfe893700 1 -- 172.21.15.70:0/1279230738 >> 172.21.15.70:0/1279230738 conn(0x7fbcf8101620 msgr2=0x7fbcf8192520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:53.678 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.102+0000 7fbcfe893700 1 -- 172.21.15.70:0/1279230738 shutdown_connections 2024-04-01T01:23:53.678 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:53.102+0000 7fbcfe893700 1 -- 172.21.15.70:0/1279230738 wait complete. 2024-04-01T01:23:53.678 INFO:teuthology.orchestra.run.smithi070.stdout:mgr epoch 5 is available 2024-04-01T01:23:53.678 INFO:teuthology.orchestra.run.smithi070.stdout:Setting orchestrator backend to cephadm... 2024-04-01T01:23:54.601 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:54 smithi070 ceph-mon[32179]: [01/Apr/2024:01:23:52] ENGINE Bus STARTING 2024-04-01T01:23:54.601 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:54 smithi070 ceph-mon[32179]: [01/Apr/2024:01:23:53] ENGINE Serving on http://172.21.15.70:8765 2024-04-01T01:23:54.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:54 smithi070 ceph-mon[32179]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-01T01:23:54.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:54 smithi070 ceph-mon[32179]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-01T01:23:54.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:54 smithi070 ceph-mon[32179]: [01/Apr/2024:01:23:53] ENGINE Serving on https://172.21.15.70:7150 2024-04-01T01:23:54.602 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:54 smithi070 ceph-mon[32179]: [01/Apr/2024:01:23:53] ENGINE Bus STARTED 2024-04-01T01:23:54.868 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.083+0000 7fb6b1f46700 1 Processor -- start 2024-04-01T01:23:54.868 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.083+0000 7fb6b1f46700 1 -- start start 2024-04-01T01:23:54.868 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.084+0000 7fb6b1f46700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6ac107a50 0x7fb6ac107e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:54.868 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.084+0000 7fb6b1f46700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac108440 con 0x7fb6ac107a50 2024-04-01T01:23:54.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.084+0000 7fb6ab7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6ac107a50 0x7fb6ac107e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:54.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.084+0000 7fb6ab7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6ac107a50 0x7fb6ac107e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57208/0 (socket says 172.21.15.70:57208) 2024-04-01T01:23:54.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.084+0000 7fb6ab7fe700 1 -- 172.21.15.70:0/4122227074 learned_addr learned my addr 172.21.15.70:0/4122227074 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:54.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.084+0000 7fb6ab7fe700 1 -- 172.21.15.70:0/4122227074 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ac108c50 con 0x7fb6ac107a50 2024-04-01T01:23:54.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.084+0000 7fb6ab7fe700 1 --2- 172.21.15.70:0/4122227074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6ac107a50 0x7fb6ac107e70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb69c0098a0 tx=0x7fb69c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2697689a224bb566 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:54.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.084+0000 7fb6aa7fc700 1 -- 172.21.15.70:0/4122227074 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb69c004160 con 0x7fb6ac107a50 2024-04-01T01:23:54.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.084+0000 7fb6aa7fc700 1 -- 172.21.15.70:0/4122227074 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fb69c004300 con 0x7fb6ac107a50 2024-04-01T01:23:54.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.084+0000 7fb6aa7fc700 1 -- 172.21.15.70:0/4122227074 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb69c011a80 con 0x7fb6ac107a50 2024-04-01T01:23:54.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.085+0000 7fb6b1f46700 1 -- 172.21.15.70:0/4122227074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6ac107a50 msgr2=0x7fb6ac107e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:54.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.085+0000 7fb6b1f46700 1 --2- 172.21.15.70:0/4122227074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6ac107a50 0x7fb6ac107e70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb69c0098a0 tx=0x7fb69c009bb0 comp rx=0 tx=0).stop 2024-04-01T01:23:54.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.085+0000 7fb6b1f46700 1 -- 172.21.15.70:0/4122227074 shutdown_connections 2024-04-01T01:23:54.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.085+0000 7fb6b1f46700 1 --2- 172.21.15.70:0/4122227074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6ac107a50 0x7fb6ac107e70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:54.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.085+0000 7fb6b1f46700 1 -- 172.21.15.70:0/4122227074 >> 172.21.15.70:0/4122227074 conn(0x7fb6ac102ff0 msgr2=0x7fb6ac105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:54.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.085+0000 7fb6b1f46700 1 -- 172.21.15.70:0/4122227074 shutdown_connections 2024-04-01T01:23:54.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.085+0000 7fb6b1f46700 1 -- 172.21.15.70:0/4122227074 wait complete. 2024-04-01T01:23:54.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.086+0000 7fb6b1f46700 1 Processor -- start 2024-04-01T01:23:54.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.086+0000 7fb6b1f46700 1 -- start start 2024-04-01T01:23:54.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.086+0000 7fb6b1f46700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6ac107a50 0x7fb6ac19b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:54.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.086+0000 7fb6b1f46700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac10c000 con 0x7fb6ac107a50 2024-04-01T01:23:54.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.086+0000 7fb6ab7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6ac107a50 0x7fb6ac19b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:54.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.086+0000 7fb6ab7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6ac107a50 0x7fb6ac19b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57224/0 (socket says 172.21.15.70:57224) 2024-04-01T01:23:54.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.086+0000 7fb6ab7fe700 1 -- 172.21.15.70:0/2885348759 learned_addr learned my addr 172.21.15.70:0/2885348759 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:54.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.086+0000 7fb6ab7fe700 1 -- 172.21.15.70:0/2885348759 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ac19bdd0 con 0x7fb6ac107a50 2024-04-01T01:23:54.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.086+0000 7fb6ab7fe700 1 --2- 172.21.15.70:0/2885348759 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6ac107a50 0x7fb6ac19b890 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb69c000c00 tx=0x7fb69c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:54.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.087+0000 7fb6a8ff9700 1 -- 172.21.15.70:0/2885348759 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb69c0039f0 con 0x7fb6ac107a50 2024-04-01T01:23:54.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.087+0000 7fb6a8ff9700 1 -- 172.21.15.70:0/2885348759 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fb69c003b90 con 0x7fb6ac107a50 2024-04-01T01:23:54.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.087+0000 7fb6b1f46700 1 -- 172.21.15.70:0/2885348759 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6ac19c060 con 0x7fb6ac107a50 2024-04-01T01:23:54.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.087+0000 7fb6a8ff9700 1 -- 172.21.15.70:0/2885348759 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb69c01a740 con 0x7fb6ac107a50 2024-04-01T01:23:54.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.087+0000 7fb6b1f46700 1 -- 172.21.15.70:0/2885348759 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6ac19ed50 con 0x7fb6ac107a50 2024-04-01T01:23:54.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.087+0000 7fb6a8ff9700 1 -- 172.21.15.70:0/2885348759 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 7) ==== 46010+0+0 (secure 0 0 0) 0x7fb69c01a9f0 con 0x7fb6ac107a50 2024-04-01T01:23:54.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.088+0000 7fb6b1f46700 1 -- 172.21.15.70:0/2885348759 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6ac063870 con 0x7fb6ac107a50 2024-04-01T01:23:54.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.088+0000 7fb6a8ff9700 1 --2- 172.21.15.70:0/2885348759 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fb694038df0 0x7fb69403b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:54.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.088+0000 7fb6a8ff9700 1 -- 172.21.15.70:0/2885348759 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb69c04d010 con 0x7fb6ac107a50 2024-04-01T01:23:54.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.088+0000 7fb6aaffd700 1 --2- 172.21.15.70:0/2885348759 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fb694038df0 0x7fb69403b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:54.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.088+0000 7fb6aaffd700 1 --2- 172.21.15.70:0/2885348759 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fb694038df0 0x7fb69403b2b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0006fd0 tx=0x7fb6a0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:54.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.093+0000 7fb6a8ff9700 1 -- 172.21.15.70:0/2885348759 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb69c01ad90 con 0x7fb6ac107a50 2024-04-01T01:23:54.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.282+0000 7fb6a8ff9700 1 -- 172.21.15.70:0/2885348759 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7fb69c02d910 con 0x7fb6ac107a50 2024-04-01T01:23:54.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.306+0000 7fb6b1f46700 1 -- 172.21.15.70:0/2885348759 --> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fb6ac104130 con 0x7fb694038df0 2024-04-01T01:23:54.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.312+0000 7fb6a8ff9700 1 -- 172.21.15.70:0/2885348759 <== mgr.14118 v2:172.21.15.70:6800/2836712828 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb6ac104130 con 0x7fb694038df0 2024-04-01T01:23:54.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.314+0000 7fb6b1f46700 1 -- 172.21.15.70:0/2885348759 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fb694038df0 msgr2=0x7fb69403b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:54.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.314+0000 7fb6b1f46700 1 --2- 172.21.15.70:0/2885348759 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fb694038df0 0x7fb69403b2b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0006fd0 tx=0x7fb6a0006e40 comp rx=0 tx=0).stop 2024-04-01T01:23:54.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.314+0000 7fb6b1f46700 1 -- 172.21.15.70:0/2885348759 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6ac107a50 msgr2=0x7fb6ac19b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:54.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.314+0000 7fb6b1f46700 1 --2- 172.21.15.70:0/2885348759 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6ac107a50 0x7fb6ac19b890 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb69c000c00 tx=0x7fb69c011b00 comp rx=0 tx=0).stop 2024-04-01T01:23:54.872 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.314+0000 7fb6b1f46700 1 -- 172.21.15.70:0/2885348759 shutdown_connections 2024-04-01T01:23:54.872 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.314+0000 7fb6b1f46700 1 --2- 172.21.15.70:0/2885348759 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fb694038df0 0x7fb69403b2b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:54.872 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.314+0000 7fb6b1f46700 1 --2- 172.21.15.70:0/2885348759 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb6ac107a50 0x7fb6ac19b890 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:54.872 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.314+0000 7fb6b1f46700 1 -- 172.21.15.70:0/2885348759 >> 172.21.15.70:0/2885348759 conn(0x7fb6ac102ff0 msgr2=0x7fb6ac103a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:54.872 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.314+0000 7fb6b1f46700 1 -- 172.21.15.70:0/2885348759 shutdown_connections 2024-04-01T01:23:54.872 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:54.314+0000 7fb6b1f46700 1 -- 172.21.15.70:0/2885348759 wait complete. 2024-04-01T01:23:55.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:55 smithi070 ceph-mon[32179]: mgrmap e8: x(active, since 2s) 2024-04-01T01:23:55.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:55 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' 2024-04-01T01:23:55.493 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:55 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:23:56.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-01T01:23:56.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.286+0000 7fd4a812f700 1 Processor -- start 2024-04-01T01:23:56.067 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.286+0000 7fd4a812f700 1 -- start start 2024-04-01T01:23:56.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.286+0000 7fd4a812f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4a0107a00 0x7fd4a0107e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:56.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.286+0000 7fd4a812f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a01083f0 con 0x7fd4a0107a00 2024-04-01T01:23:56.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.287+0000 7fd4a5ecb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4a0107a00 0x7fd4a0107e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:56.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.287+0000 7fd4a5ecb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4a0107a00 0x7fd4a0107e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57232/0 (socket says 172.21.15.70:57232) 2024-04-01T01:23:56.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.287+0000 7fd4a5ecb700 1 -- 172.21.15.70:0/1278792032 learned_addr learned my addr 172.21.15.70:0/1278792032 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:56.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.287+0000 7fd4a5ecb700 1 -- 172.21.15.70:0/1278792032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a0108c00 con 0x7fd4a0107a00 2024-04-01T01:23:56.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.287+0000 7fd4a5ecb700 1 --2- 172.21.15.70:0/1278792032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4a0107a00 0x7fd4a0107e20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd498009e80 tx=0x7fd49800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98dcc5cda90bffe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:56.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.287+0000 7fd4a4ec9700 1 -- 172.21.15.70:0/1278792032 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd498017070 con 0x7fd4a0107a00 2024-04-01T01:23:56.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.287+0000 7fd4a4ec9700 1 -- 172.21.15.70:0/1278792032 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fd49800b8c0 con 0x7fd4a0107a00 2024-04-01T01:23:56.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.287+0000 7fd4a4ec9700 1 -- 172.21.15.70:0/1278792032 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd498004520 con 0x7fd4a0107a00 2024-04-01T01:23:56.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.288+0000 7fd4a812f700 1 -- 172.21.15.70:0/1278792032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4a0107a00 msgr2=0x7fd4a0107e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:56.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.288+0000 7fd4a812f700 1 --2- 172.21.15.70:0/1278792032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4a0107a00 0x7fd4a0107e20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd498009e80 tx=0x7fd49800b330 comp rx=0 tx=0).stop 2024-04-01T01:23:56.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.288+0000 7fd4a812f700 1 -- 172.21.15.70:0/1278792032 shutdown_connections 2024-04-01T01:23:56.068 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.288+0000 7fd4a812f700 1 --2- 172.21.15.70:0/1278792032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4a0107a00 0x7fd4a0107e20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:56.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.288+0000 7fd4a812f700 1 -- 172.21.15.70:0/1278792032 >> 172.21.15.70:0/1278792032 conn(0x7fd4a0102fa0 msgr2=0x7fd4a01053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:56.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.288+0000 7fd4a812f700 1 -- 172.21.15.70:0/1278792032 shutdown_connections 2024-04-01T01:23:56.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.288+0000 7fd4a812f700 1 -- 172.21.15.70:0/1278792032 wait complete. 2024-04-01T01:23:56.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.288+0000 7fd4a812f700 1 Processor -- start 2024-04-01T01:23:56.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.289+0000 7fd4a812f700 1 -- start start 2024-04-01T01:23:56.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.289+0000 7fd4a812f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4a0107a00 0x7fd4a019b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:56.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.289+0000 7fd4a812f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd4a010bfb0 con 0x7fd4a0107a00 2024-04-01T01:23:56.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.289+0000 7fd4a5ecb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4a0107a00 0x7fd4a019b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:56.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.289+0000 7fd4a5ecb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4a0107a00 0x7fd4a019b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57236/0 (socket says 172.21.15.70:57236) 2024-04-01T01:23:56.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.289+0000 7fd4a5ecb700 1 -- 172.21.15.70:0/3112773507 learned_addr learned my addr 172.21.15.70:0/3112773507 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:56.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.289+0000 7fd4a5ecb700 1 -- 172.21.15.70:0/3112773507 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4a019be30 con 0x7fd4a0107a00 2024-04-01T01:23:56.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.289+0000 7fd4a5ecb700 1 --2- 172.21.15.70:0/3112773507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4a0107a00 0x7fd4a019b8f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd498009e50 tx=0x7fd498004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:56.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.289+0000 7fd49effd700 1 -- 172.21.15.70:0/3112773507 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd498020020 con 0x7fd4a0107a00 2024-04-01T01:23:56.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.289+0000 7fd4a812f700 1 -- 172.21.15.70:0/3112773507 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4a019c0c0 con 0x7fd4a0107a00 2024-04-01T01:23:56.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.289+0000 7fd49effd700 1 -- 172.21.15.70:0/3112773507 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fd49800bba0 con 0x7fd4a0107a00 2024-04-01T01:23:56.069 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.289+0000 7fd49effd700 1 -- 172.21.15.70:0/3112773507 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd498017040 con 0x7fd4a0107a00 2024-04-01T01:23:56.070 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.289+0000 7fd4a812f700 1 -- 172.21.15.70:0/3112773507 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4a019edb0 con 0x7fd4a0107a00 2024-04-01T01:23:56.070 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.290+0000 7fd49effd700 1 -- 172.21.15.70:0/3112773507 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7fd498023020 con 0x7fd4a0107a00 2024-04-01T01:23:56.070 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.290+0000 7fd4a812f700 1 -- 172.21.15.70:0/3112773507 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4a0063870 con 0x7fd4a0107a00 2024-04-01T01:23:56.070 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.290+0000 7fd49effd700 1 --2- 172.21.15.70:0/3112773507 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fd488038d60 0x7fd48803b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:56.070 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.290+0000 7fd49effd700 1 -- 172.21.15.70:0/3112773507 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd49800f050 con 0x7fd4a0107a00 2024-04-01T01:23:56.070 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.291+0000 7fd4a56ca700 1 --2- 172.21.15.70:0/3112773507 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fd488038d60 0x7fd48803b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:56.070 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.291+0000 7fd4a56ca700 1 --2- 172.21.15.70:0/3112773507 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fd488038d60 0x7fd48803b220 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd494006fd0 tx=0x7fd494006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:56.070 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.296+0000 7fd49effd700 1 -- 172.21.15.70:0/3112773507 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd49802a020 con 0x7fd4a0107a00 2024-04-01T01:23:56.070 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.494+0000 7fd4a812f700 1 -- 172.21.15.70:0/3112773507 --> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fd4a0104190 con 0x7fd488038d60 2024-04-01T01:23:56.070 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.494+0000 7fd49effd700 1 -- 172.21.15.70:0/3112773507 <== mgr.14118 v2:172.21.15.70:6800/2836712828 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fd4a0104190 con 0x7fd488038d60 2024-04-01T01:23:56.070 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.497+0000 7fd4a812f700 1 -- 172.21.15.70:0/3112773507 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fd488038d60 msgr2=0x7fd48803b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:56.070 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.497+0000 7fd4a812f700 1 --2- 172.21.15.70:0/3112773507 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fd488038d60 0x7fd48803b220 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd494006fd0 tx=0x7fd494006e40 comp rx=0 tx=0).stop 2024-04-01T01:23:56.071 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.497+0000 7fd4a812f700 1 -- 172.21.15.70:0/3112773507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4a0107a00 msgr2=0x7fd4a019b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:56.071 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.497+0000 7fd4a812f700 1 --2- 172.21.15.70:0/3112773507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4a0107a00 0x7fd4a019b8f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd498009e50 tx=0x7fd498004670 comp rx=0 tx=0).stop 2024-04-01T01:23:56.071 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.497+0000 7fd4a812f700 1 -- 172.21.15.70:0/3112773507 shutdown_connections 2024-04-01T01:23:56.071 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.497+0000 7fd4a812f700 1 --2- 172.21.15.70:0/3112773507 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fd488038d60 0x7fd48803b220 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:56.071 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.497+0000 7fd4a812f700 1 --2- 172.21.15.70:0/3112773507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4a0107a00 0x7fd4a019b8f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:56.071 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.497+0000 7fd4a812f700 1 -- 172.21.15.70:0/3112773507 >> 172.21.15.70:0/3112773507 conn(0x7fd4a0102fa0 msgr2=0x7fd4a0103a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:56.071 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.497+0000 7fd4a812f700 1 -- 172.21.15.70:0/3112773507 shutdown_connections 2024-04-01T01:23:56.071 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:55.497+0000 7fd4a812f700 1 -- 172.21.15.70:0/3112773507 wait complete. 2024-04-01T01:23:56.071 INFO:teuthology.orchestra.run.smithi070.stdout:Generating ssh key... 2024-04-01T01:23:56.324 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-mon[32179]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:23:57.004 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: Generating public/private rsa key pair. 2024-04-01T01:23:57.004 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: Your identification has been saved in /tmp/tmpv6qqoqpp/key. 2024-04-01T01:23:57.004 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: Your public key has been saved in /tmp/tmpv6qqoqpp/key.pub. 2024-04-01T01:23:57.004 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: The key fingerprint is: 2024-04-01T01:23:57.004 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: SHA256:uO7+YXVBhQJhiTaq4h5SShoSk/RWXdpXKTuZlDvm4JI ceph-36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:23:57.005 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: The key's randomart image is: 2024-04-01T01:23:57.005 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: +---[RSA 3072]----+ 2024-04-01T01:23:57.005 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: | . ...== o+o | 2024-04-01T01:23:57.005 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: |... . +o+ .=o. | 2024-04-01T01:23:57.005 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: |+ o o o ..oB | 2024-04-01T01:23:57.005 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: | o. . . ..O . | 2024-04-01T01:23:57.005 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: |o... . S = + | 2024-04-01T01:23:57.005 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: |=+. E o o | 2024-04-01T01:23:57.005 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: |=o . + | 2024-04-01T01:23:57.005 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: |... . . . | 2024-04-01T01:23:57.005 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: |.. o+.. | 2024-04-01T01:23:57.005 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:23:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: +----[SHA256]-----+ 2024-04-01T01:23:57.404 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.498+0000 7fc67cf22700 1 Processor -- start 2024-04-01T01:23:57.405 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.498+0000 7fc67cf22700 1 -- start start 2024-04-01T01:23:57.405 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.499+0000 7fc67cf22700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc678108500 0x7fc67810a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:57.405 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.499+0000 7fc67cf22700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc67807be40 con 0x7fc678108500 2024-04-01T01:23:57.405 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.499+0000 7fc67659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc678108500 0x7fc67810a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:57.405 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.499+0000 7fc67659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc678108500 0x7fc67810a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57242/0 (socket says 172.21.15.70:57242) 2024-04-01T01:23:57.405 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.499+0000 7fc67659c700 1 -- 172.21.15.70:0/1912372398 learned_addr learned my addr 172.21.15.70:0/1912372398 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:57.405 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.499+0000 7fc67659c700 1 -- 172.21.15.70:0/1912372398 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc67810ae30 con 0x7fc678108500 2024-04-01T01:23:57.405 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.499+0000 7fc67659c700 1 --2- 172.21.15.70:0/1912372398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc678108500 0x7fc67810a8f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc670009e80 tx=0x7fc67000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e3b2f220664b68 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:57.405 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.499+0000 7fc67559a700 1 -- 172.21.15.70:0/1912372398 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc670017070 con 0x7fc678108500 2024-04-01T01:23:57.405 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.499+0000 7fc67559a700 1 -- 172.21.15.70:0/1912372398 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fc67000b8c0 con 0x7fc678108500 2024-04-01T01:23:57.405 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.499+0000 7fc67559a700 1 -- 172.21.15.70:0/1912372398 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc670004520 con 0x7fc678108500 2024-04-01T01:23:57.405 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.500+0000 7fc67cf22700 1 -- 172.21.15.70:0/1912372398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc678108500 msgr2=0x7fc67810a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:57.406 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.500+0000 7fc67cf22700 1 --2- 172.21.15.70:0/1912372398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc678108500 0x7fc67810a8f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc670009e80 tx=0x7fc67000b330 comp rx=0 tx=0).stop 2024-04-01T01:23:57.406 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.500+0000 7fc67cf22700 1 -- 172.21.15.70:0/1912372398 shutdown_connections 2024-04-01T01:23:57.406 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.500+0000 7fc67cf22700 1 --2- 172.21.15.70:0/1912372398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc678108500 0x7fc67810a8f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:57.406 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.500+0000 7fc67cf22700 1 -- 172.21.15.70:0/1912372398 >> 172.21.15.70:0/1912372398 conn(0x7fc678101ed0 msgr2=0x7fc678104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:57.406 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.500+0000 7fc67cf22700 1 -- 172.21.15.70:0/1912372398 shutdown_connections 2024-04-01T01:23:57.406 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.500+0000 7fc67cf22700 1 -- 172.21.15.70:0/1912372398 wait complete. 2024-04-01T01:23:57.406 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.501+0000 7fc67cf22700 1 Processor -- start 2024-04-01T01:23:57.406 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.501+0000 7fc67cf22700 1 -- start start 2024-04-01T01:23:57.406 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.501+0000 7fc67cf22700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc678108500 0x7fc678107d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:57.406 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.501+0000 7fc67cf22700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc67810bda0 con 0x7fc678108500 2024-04-01T01:23:57.406 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.501+0000 7fc67659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc678108500 0x7fc678107d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:57.406 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.501+0000 7fc67659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc678108500 0x7fc678107d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57250/0 (socket says 172.21.15.70:57250) 2024-04-01T01:23:57.406 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.501+0000 7fc67659c700 1 -- 172.21.15.70:0/1831756586 learned_addr learned my addr 172.21.15.70:0/1831756586 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:57.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.501+0000 7fc67659c700 1 -- 172.21.15.70:0/1831756586 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6781082c0 con 0x7fc678108500 2024-04-01T01:23:57.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.502+0000 7fc67659c700 1 --2- 172.21.15.70:0/1831756586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc678108500 0x7fc678107d80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc67000b7b0 tx=0x7fc670004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:57.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.502+0000 7fc6677fe700 1 -- 172.21.15.70:0/1831756586 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc670020020 con 0x7fc678108500 2024-04-01T01:23:57.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.502+0000 7fc6677fe700 1 -- 172.21.15.70:0/1831756586 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fc67000bba0 con 0x7fc678108500 2024-04-01T01:23:57.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.502+0000 7fc67cf22700 1 -- 172.21.15.70:0/1831756586 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6781064b0 con 0x7fc678108500 2024-04-01T01:23:57.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.502+0000 7fc6677fe700 1 -- 172.21.15.70:0/1831756586 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc670017040 con 0x7fc678108500 2024-04-01T01:23:57.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.502+0000 7fc67cf22700 1 -- 172.21.15.70:0/1831756586 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6781069f0 con 0x7fc678108500 2024-04-01T01:23:57.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.503+0000 7fc6677fe700 1 -- 172.21.15.70:0/1831756586 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7fc670023020 con 0x7fc678108500 2024-04-01T01:23:57.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.503+0000 7fc67cf22700 1 -- 172.21.15.70:0/1831756586 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc678063870 con 0x7fc678108500 2024-04-01T01:23:57.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.503+0000 7fc6677fe700 1 --2- 172.21.15.70:0/1831756586 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fc660038ec0 0x7fc66003b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:57.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.503+0000 7fc6677fe700 1 -- 172.21.15.70:0/1831756586 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc67000f050 con 0x7fc678108500 2024-04-01T01:23:57.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.503+0000 7fc675d9b700 1 --2- 172.21.15.70:0/1831756586 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fc660038ec0 0x7fc66003b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:57.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.503+0000 7fc675d9b700 1 --2- 172.21.15.70:0/1831756586 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fc660038ec0 0x7fc66003b380 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc66c006fd0 tx=0x7fc66c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:57.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.508+0000 7fc6677fe700 1 -- 172.21.15.70:0/1831756586 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc67002a020 con 0x7fc678108500 2024-04-01T01:23:57.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.699+0000 7fc67cf22700 1 -- 172.21.15.70:0/1831756586 --> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fc678107250 con 0x7fc660038ec0 2024-04-01T01:23:57.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.812+0000 7fc6677fe700 1 -- 172.21.15.70:0/1831756586 <== mgr.14118 v2:172.21.15.70:6800/2836712828 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc678107250 con 0x7fc660038ec0 2024-04-01T01:23:57.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.814+0000 7fc67cf22700 1 -- 172.21.15.70:0/1831756586 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fc660038ec0 msgr2=0x7fc66003b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:57.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.814+0000 7fc67cf22700 1 --2- 172.21.15.70:0/1831756586 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fc660038ec0 0x7fc66003b380 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc66c006fd0 tx=0x7fc66c006e40 comp rx=0 tx=0).stop 2024-04-01T01:23:57.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.814+0000 7fc67cf22700 1 -- 172.21.15.70:0/1831756586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc678108500 msgr2=0x7fc678107d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:57.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.814+0000 7fc67cf22700 1 --2- 172.21.15.70:0/1831756586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc678108500 0x7fc678107d80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc67000b7b0 tx=0x7fc670004670 comp rx=0 tx=0).stop 2024-04-01T01:23:57.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.814+0000 7fc67cf22700 1 -- 172.21.15.70:0/1831756586 shutdown_connections 2024-04-01T01:23:57.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.814+0000 7fc67cf22700 1 --2- 172.21.15.70:0/1831756586 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fc660038ec0 0x7fc66003b380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:57.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.814+0000 7fc67cf22700 1 --2- 172.21.15.70:0/1831756586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc678108500 0x7fc678107d80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:57.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.814+0000 7fc67cf22700 1 -- 172.21.15.70:0/1831756586 >> 172.21.15.70:0/1831756586 conn(0x7fc678101ed0 msgr2=0x7fc6781028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:57.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.814+0000 7fc67cf22700 1 -- 172.21.15.70:0/1831756586 shutdown_connections 2024-04-01T01:23:57.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:56.815+0000 7fc67cf22700 1 -- 172.21.15.70:0/1831756586 wait complete. 2024-04-01T01:23:57.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:57 smithi070 ceph-mon[32179]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:23:57.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:57 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' 2024-04-01T01:23:57.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:57 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' 2024-04-01T01:23:58.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:58 smithi070 ceph-mon[32179]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:23:58.542 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:58 smithi070 ceph-mon[32179]: Generating ssh key... 2024-04-01T01:23:58.570 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKsoBv+8wsrN+7luwiiFWSrs2097Luy4eDj+sEwq3Z1IysB8JfZNGA9hXzi/SzScL0AMbOO0/AoFt4p0A+IOgYQGIauqC/sY1TS8Ta2Px1lzhSdsMSKcxxJ6CI4fPUaoQUpjyYlnEkdp8BgSEI7i+hVaB6PP4aoZAzAYS/gog39S9m3rBLA5MRPPVzhbAXce9QvWI4kKSmZdAluINGzNKR4dV+uaDBlg7D5Slu32ns8i09tVGtPLRXeOcTpLQfdSQFfxSHWpn34de0uVYKzIUX4BtHGL/oU4m70O8JcuMN6L6afF6/uw8y13hK1Z7Xg6WVrpr4SiNKPfbuqeyDar0HHmkWA54o4O9fb8URVvqUpjp8xNuehlE7SsWYD94uaBxyKiEetLmaO9GC9qiDIRuNrH/fBbYHAsiExzjyKhVi+CAe8EgoRj077jMm027MnwfDvT3ZLkLDQI2VJ6Qm3U/3AhYXyi52j4Z6HCjYIdbT+CxqJRyGK8Nutp2nj/3271s= ceph-36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:23:58.570 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.806+0000 7f6d50157700 1 Processor -- start 2024-04-01T01:23:58.570 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.806+0000 7f6d50157700 1 -- start start 2024-04-01T01:23:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.806+0000 7f6d50157700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d481062f0 0x7f6d48106710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.806+0000 7f6d50157700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4807be40 con 0x7f6d481062f0 2024-04-01T01:23:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.807+0000 7f6d4def3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d481062f0 0x7f6d48106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.807+0000 7f6d4def3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d481062f0 0x7f6d48106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47232/0 (socket says 172.21.15.70:47232) 2024-04-01T01:23:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.807+0000 7f6d4def3700 1 -- 172.21.15.70:0/3892365303 learned_addr learned my addr 172.21.15.70:0/3892365303 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.807+0000 7f6d4def3700 1 -- 172.21.15.70:0/3892365303 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d48106c50 con 0x7f6d481062f0 2024-04-01T01:23:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.807+0000 7f6d4def3700 1 --2- 172.21.15.70:0/3892365303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d481062f0 0x7f6d48106710 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6d40009e80 tx=0x7f6d4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94efe66bd07ae2b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.807+0000 7f6d4cef1700 1 -- 172.21.15.70:0/3892365303 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d40017070 con 0x7f6d481062f0 2024-04-01T01:23:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.807+0000 7f6d4cef1700 1 -- 172.21.15.70:0/3892365303 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6d4000b8c0 con 0x7f6d481062f0 2024-04-01T01:23:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.807+0000 7f6d4cef1700 1 -- 172.21.15.70:0/3892365303 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d40004520 con 0x7f6d481062f0 2024-04-01T01:23:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.808+0000 7f6d50157700 1 -- 172.21.15.70:0/3892365303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d481062f0 msgr2=0x7f6d48106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.808+0000 7f6d50157700 1 --2- 172.21.15.70:0/3892365303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d481062f0 0x7f6d48106710 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6d40009e80 tx=0x7f6d4000b330 comp rx=0 tx=0).stop 2024-04-01T01:23:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.808+0000 7f6d50157700 1 -- 172.21.15.70:0/3892365303 shutdown_connections 2024-04-01T01:23:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.808+0000 7f6d50157700 1 --2- 172.21.15.70:0/3892365303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d481062f0 0x7f6d48106710 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.808+0000 7f6d50157700 1 -- 172.21.15.70:0/3892365303 >> 172.21.15.70:0/3892365303 conn(0x7f6d48101ed0 msgr2=0x7f6d48104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.808+0000 7f6d50157700 1 -- 172.21.15.70:0/3892365303 shutdown_connections 2024-04-01T01:23:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.808+0000 7f6d50157700 1 -- 172.21.15.70:0/3892365303 wait complete. 2024-04-01T01:23:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.808+0000 7f6d50157700 1 Processor -- start 2024-04-01T01:23:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.808+0000 7f6d50157700 1 -- start start 2024-04-01T01:23:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.809+0000 7f6d50157700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d481062f0 0x7f6d4819da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.809+0000 7f6d50157700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6d48107c80 con 0x7f6d481062f0 2024-04-01T01:23:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.809+0000 7f6d4def3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d481062f0 0x7f6d4819da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.809+0000 7f6d4def3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d481062f0 0x7f6d4819da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47242/0 (socket says 172.21.15.70:47242) 2024-04-01T01:23:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.809+0000 7f6d4def3700 1 -- 172.21.15.70:0/1304096323 learned_addr learned my addr 172.21.15.70:0/1304096323 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:23:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.809+0000 7f6d4def3700 1 -- 172.21.15.70:0/1304096323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4819dfb0 con 0x7f6d481062f0 2024-04-01T01:23:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.809+0000 7f6d4def3700 1 --2- 172.21.15.70:0/1304096323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d481062f0 0x7f6d4819da70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6d400042b0 tx=0x7f6d40004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.809+0000 7f6d46ffd700 1 -- 172.21.15.70:0/1304096323 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d40020020 con 0x7f6d481062f0 2024-04-01T01:23:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.809+0000 7f6d46ffd700 1 -- 172.21.15.70:0/1304096323 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f6d4000bba0 con 0x7f6d481062f0 2024-04-01T01:23:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.809+0000 7f6d50157700 1 -- 172.21.15.70:0/1304096323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d4819e240 con 0x7f6d481062f0 2024-04-01T01:23:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.809+0000 7f6d46ffd700 1 -- 172.21.15.70:0/1304096323 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d40017040 con 0x7f6d481062f0 2024-04-01T01:23:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.809+0000 7f6d50157700 1 -- 172.21.15.70:0/1304096323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d481a0f30 con 0x7f6d481062f0 2024-04-01T01:23:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.810+0000 7f6d46ffd700 1 -- 172.21.15.70:0/1304096323 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7f6d40023020 con 0x7f6d481062f0 2024-04-01T01:23:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.810+0000 7f6d50157700 1 -- 172.21.15.70:0/1304096323 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d48063870 con 0x7f6d481062f0 2024-04-01T01:23:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.810+0000 7f6d46ffd700 1 --2- 172.21.15.70:0/1304096323 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f6d30038f10 0x7f6d3003b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:23:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.810+0000 7f6d46ffd700 1 -- 172.21.15.70:0/1304096323 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6d4000f050 con 0x7f6d481062f0 2024-04-01T01:23:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.810+0000 7f6d4d6f2700 1 --2- 172.21.15.70:0/1304096323 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f6d30038f10 0x7f6d3003b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:23:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.811+0000 7f6d4d6f2700 1 --2- 172.21.15.70:0/1304096323 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f6d30038f10 0x7f6d3003b3d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c006fd0 tx=0x7f6d3c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:23:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:57.815+0000 7f6d46ffd700 1 -- 172.21.15.70:0/1304096323 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6d40053080 con 0x7f6d481062f0 2024-04-01T01:23:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.008+0000 7f6d50157700 1 -- 172.21.15.70:0/1304096323 --> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f6d4802dee0 con 0x7f6d30038f10 2024-04-01T01:23:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.009+0000 7f6d46ffd700 1 -- 172.21.15.70:0/1304096323 <== mgr.14118 v2:172.21.15.70:6800/2836712828 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f6d4802dee0 con 0x7f6d30038f10 2024-04-01T01:23:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.011+0000 7f6d50157700 1 -- 172.21.15.70:0/1304096323 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f6d30038f10 msgr2=0x7f6d3003b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.011+0000 7f6d50157700 1 --2- 172.21.15.70:0/1304096323 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f6d30038f10 0x7f6d3003b3d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c006fd0 tx=0x7f6d3c006e40 comp rx=0 tx=0).stop 2024-04-01T01:23:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.011+0000 7f6d50157700 1 -- 172.21.15.70:0/1304096323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d481062f0 msgr2=0x7f6d4819da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:23:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.011+0000 7f6d50157700 1 --2- 172.21.15.70:0/1304096323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d481062f0 0x7f6d4819da70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6d400042b0 tx=0x7f6d40004670 comp rx=0 tx=0).stop 2024-04-01T01:23:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.011+0000 7f6d50157700 1 -- 172.21.15.70:0/1304096323 shutdown_connections 2024-04-01T01:23:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.011+0000 7f6d50157700 1 --2- 172.21.15.70:0/1304096323 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f6d30038f10 0x7f6d3003b3d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.011+0000 7f6d50157700 1 --2- 172.21.15.70:0/1304096323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d481062f0 0x7f6d4819da70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:23:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.011+0000 7f6d50157700 1 -- 172.21.15.70:0/1304096323 >> 172.21.15.70:0/1304096323 conn(0x7f6d48101ed0 msgr2=0x7f6d48194710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:23:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.011+0000 7f6d50157700 1 -- 172.21.15.70:0/1304096323 shutdown_connections 2024-04-01T01:23:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.011+0000 7f6d50157700 1 -- 172.21.15.70:0/1304096323 wait complete. 2024-04-01T01:23:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-01T01:23:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:Adding key to root@localhost authorized_keys... 2024-04-01T01:23:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:Adding host smithi070... 2024-04-01T01:23:59.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:23:59 smithi070 ceph-mon[32179]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:24:00.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:00 smithi070 ceph-mon[32179]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi070", "addr": "172.21.15.70", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:24:01.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:01 smithi070 ceph-mon[32179]: Deploying cephadm binary to smithi070 2024-04-01T01:24:03.868 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout Added host 'smithi070' with addr '172.21.15.70' 2024-04-01T01:24:03.868 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.973+0000 7efd8833a700 1 Processor -- start 2024-04-01T01:24:03.868 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.973+0000 7efd8833a700 1 -- start start 2024-04-01T01:24:03.868 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.973+0000 7efd8833a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd80108480 0x7efd8010a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:03.868 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.973+0000 7efd8833a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efd8007bea0 con 0x7efd80108480 2024-04-01T01:24:03.868 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.973+0000 7efd860d6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd80108480 0x7efd8010a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:03.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.973+0000 7efd860d6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd80108480 0x7efd8010a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47246/0 (socket says 172.21.15.70:47246) 2024-04-01T01:24:03.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.973+0000 7efd860d6700 1 -- 172.21.15.70:0/4053313401 learned_addr learned my addr 172.21.15.70:0/4053313401 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:03.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.974+0000 7efd860d6700 1 -- 172.21.15.70:0/4053313401 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8010adb0 con 0x7efd80108480 2024-04-01T01:24:03.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.974+0000 7efd860d6700 1 --2- 172.21.15.70:0/4053313401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd80108480 0x7efd8010a870 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0098a0 tx=0x7efd7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e77eccdacd9450c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:03.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.974+0000 7efd850d4700 1 -- 172.21.15.70:0/4053313401 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd7c004160 con 0x7efd80108480 2024-04-01T01:24:03.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.974+0000 7efd850d4700 1 -- 172.21.15.70:0/4053313401 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7efd7c004300 con 0x7efd80108480 2024-04-01T01:24:03.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.974+0000 7efd850d4700 1 -- 172.21.15.70:0/4053313401 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd7c011a80 con 0x7efd80108480 2024-04-01T01:24:03.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.975+0000 7efd8833a700 1 -- 172.21.15.70:0/4053313401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd80108480 msgr2=0x7efd8010a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:03.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.975+0000 7efd8833a700 1 --2- 172.21.15.70:0/4053313401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd80108480 0x7efd8010a870 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0098a0 tx=0x7efd7c009bb0 comp rx=0 tx=0).stop 2024-04-01T01:24:03.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.975+0000 7efd8833a700 1 -- 172.21.15.70:0/4053313401 shutdown_connections 2024-04-01T01:24:03.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.975+0000 7efd8833a700 1 --2- 172.21.15.70:0/4053313401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd80108480 0x7efd8010a870 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:03.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.975+0000 7efd8833a700 1 -- 172.21.15.70:0/4053313401 >> 172.21.15.70:0/4053313401 conn(0x7efd80101e90 msgr2=0x7efd801042f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:03.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.975+0000 7efd8833a700 1 -- 172.21.15.70:0/4053313401 shutdown_connections 2024-04-01T01:24:03.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.975+0000 7efd8833a700 1 -- 172.21.15.70:0/4053313401 wait complete. 2024-04-01T01:24:03.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.975+0000 7efd8833a700 1 Processor -- start 2024-04-01T01:24:03.869 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.975+0000 7efd8833a700 1 -- start start 2024-04-01T01:24:03.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.975+0000 7efd8833a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd80108480 0x7efd80107d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:03.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.975+0000 7efd8833a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efd8010bd80 con 0x7efd80108480 2024-04-01T01:24:03.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.976+0000 7efd860d6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd80108480 0x7efd80107d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:03.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.976+0000 7efd860d6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd80108480 0x7efd80107d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47254/0 (socket says 172.21.15.70:47254) 2024-04-01T01:24:03.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.976+0000 7efd860d6700 1 -- 172.21.15.70:0/2117356783 learned_addr learned my addr 172.21.15.70:0/2117356783 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:03.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.976+0000 7efd860d6700 1 -- 172.21.15.70:0/2117356783 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd80108240 con 0x7efd80108480 2024-04-01T01:24:03.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.976+0000 7efd860d6700 1 --2- 172.21.15.70:0/2117356783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd80108480 0x7efd80107d00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efd7c006af0 tx=0x7efd7c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:03.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.976+0000 7efd777fe700 1 -- 172.21.15.70:0/2117356783 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd7c003930 con 0x7efd80108480 2024-04-01T01:24:03.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.976+0000 7efd777fe700 1 -- 172.21.15.70:0/2117356783 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7efd7c003ad0 con 0x7efd80108480 2024-04-01T01:24:03.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.976+0000 7efd8833a700 1 -- 172.21.15.70:0/2117356783 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd80106430 con 0x7efd80108480 2024-04-01T01:24:03.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.976+0000 7efd777fe700 1 -- 172.21.15.70:0/2117356783 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd7c0195e0 con 0x7efd80108480 2024-04-01T01:24:03.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.976+0000 7efd8833a700 1 -- 172.21.15.70:0/2117356783 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd80106970 con 0x7efd80108480 2024-04-01T01:24:03.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.977+0000 7efd777fe700 1 -- 172.21.15.70:0/2117356783 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7efd7c01a430 con 0x7efd80108480 2024-04-01T01:24:03.870 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.977+0000 7efd777fe700 1 --2- 172.21.15.70:0/2117356783 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7efd6c038e70 0x7efd6c03b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:03.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.977+0000 7efd777fe700 1 -- 172.21.15.70:0/2117356783 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7efd7c04d260 con 0x7efd80108480 2024-04-01T01:24:03.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.977+0000 7efd858d5700 1 --2- 172.21.15.70:0/2117356783 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7efd6c038e70 0x7efd6c03b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:03.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.977+0000 7efd8833a700 1 -- 172.21.15.70:0/2117356783 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd80063870 con 0x7efd80108480 2024-04-01T01:24:03.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.978+0000 7efd858d5700 1 --2- 172.21.15.70:0/2117356783 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7efd6c038e70 0x7efd6c03b330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efd70006fd0 tx=0x7efd70006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:03.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:58.982+0000 7efd777fe700 1 -- 172.21.15.70:0/2117356783 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efd7c0172b0 con 0x7efd80108480 2024-04-01T01:24:03.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:23:59.178+0000 7efd8833a700 1 -- 172.21.15.70:0/2117356783 --> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi070", "addr": "172.21.15.70", "target": ["mon-mgr", ""]}) -- 0x7efd80107090 con 0x7efd6c038e70 2024-04-01T01:24:03.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:03.342+0000 7efd777fe700 1 -- 172.21.15.70:0/2117356783 <== mgr.14118 v2:172.21.15.70:6800/2836712828 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7efd80107090 con 0x7efd6c038e70 2024-04-01T01:24:03.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:03.344+0000 7efd8833a700 1 -- 172.21.15.70:0/2117356783 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7efd6c038e70 msgr2=0x7efd6c03b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:03.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:03.344+0000 7efd8833a700 1 --2- 172.21.15.70:0/2117356783 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7efd6c038e70 0x7efd6c03b330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efd70006fd0 tx=0x7efd70006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:03.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:03.344+0000 7efd8833a700 1 -- 172.21.15.70:0/2117356783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd80108480 msgr2=0x7efd80107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:03.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:03.344+0000 7efd8833a700 1 --2- 172.21.15.70:0/2117356783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd80108480 0x7efd80107d00 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efd7c006af0 tx=0x7efd7c011b70 comp rx=0 tx=0).stop 2024-04-01T01:24:03.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:03.344+0000 7efd8833a700 1 -- 172.21.15.70:0/2117356783 shutdown_connections 2024-04-01T01:24:03.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:03.344+0000 7efd8833a700 1 --2- 172.21.15.70:0/2117356783 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7efd6c038e70 0x7efd6c03b330 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:03.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:03.345+0000 7efd8833a700 1 --2- 172.21.15.70:0/2117356783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd80108480 0x7efd80107d00 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:03.871 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:03.345+0000 7efd8833a700 1 -- 172.21.15.70:0/2117356783 >> 172.21.15.70:0/2117356783 conn(0x7efd80101e90 msgr2=0x7efd801028e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:03.872 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:03.345+0000 7efd8833a700 1 -- 172.21.15.70:0/2117356783 shutdown_connections 2024-04-01T01:24:03.872 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:03.345+0000 7efd8833a700 1 -- 172.21.15.70:0/2117356783 wait complete. 2024-04-01T01:24:03.872 INFO:teuthology.orchestra.run.smithi070.stdout:Deploying unmanaged mon service... 2024-04-01T01:24:04.586 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:04 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' 2024-04-01T01:24:04.586 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:04 smithi070 ceph-mon[32179]: Added host smithi070 2024-04-01T01:24:04.586 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:04 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:24:05.590 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-01T01:24:05.590 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.359+0000 7f1c57080700 1 Processor -- start 2024-04-01T01:24:05.590 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.359+0000 7f1c57080700 1 -- start start 2024-04-01T01:24:05.590 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.359+0000 7f1c57080700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c501084a0 0x7f1c5010a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:05.590 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.359+0000 7f1c57080700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5007bea0 con 0x7f1c501084a0 2024-04-01T01:24:05.590 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.360+0000 7f1c54e1c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c501084a0 0x7f1c5010a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:05.590 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.360+0000 7f1c54e1c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c501084a0 0x7f1c5010a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47258/0 (socket says 172.21.15.70:47258) 2024-04-01T01:24:05.590 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.360+0000 7f1c54e1c700 1 -- 172.21.15.70:0/2437407307 learned_addr learned my addr 172.21.15.70:0/2437407307 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:05.590 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.360+0000 7f1c54e1c700 1 -- 172.21.15.70:0/2437407307 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c5010add0 con 0x7f1c501084a0 2024-04-01T01:24:05.590 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.360+0000 7f1c54e1c700 1 --2- 172.21.15.70:0/2437407307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c501084a0 0x7f1c5010a890 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1c440098a0 tx=0x7f1c44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14315f649a1656ea server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:05.590 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.360+0000 7f1c4f7fe700 1 -- 172.21.15.70:0/2437407307 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c44004160 con 0x7f1c501084a0 2024-04-01T01:24:05.590 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.360+0000 7f1c4f7fe700 1 -- 172.21.15.70:0/2437407307 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1c44004300 con 0x7f1c501084a0 2024-04-01T01:24:05.591 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.360+0000 7f1c4f7fe700 1 -- 172.21.15.70:0/2437407307 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c44011a80 con 0x7f1c501084a0 2024-04-01T01:24:05.591 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.361+0000 7f1c57080700 1 -- 172.21.15.70:0/2437407307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c501084a0 msgr2=0x7f1c5010a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:05.591 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.361+0000 7f1c57080700 1 --2- 172.21.15.70:0/2437407307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c501084a0 0x7f1c5010a890 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1c440098a0 tx=0x7f1c44009bb0 comp rx=0 tx=0).stop 2024-04-01T01:24:05.591 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.361+0000 7f1c57080700 1 -- 172.21.15.70:0/2437407307 shutdown_connections 2024-04-01T01:24:05.591 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.361+0000 7f1c57080700 1 --2- 172.21.15.70:0/2437407307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c501084a0 0x7f1c5010a890 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:05.591 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.361+0000 7f1c57080700 1 -- 172.21.15.70:0/2437407307 >> 172.21.15.70:0/2437407307 conn(0x7f1c50101ed0 msgr2=0x7f1c50104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:05.591 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.361+0000 7f1c57080700 1 -- 172.21.15.70:0/2437407307 shutdown_connections 2024-04-01T01:24:05.591 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.361+0000 7f1c57080700 1 -- 172.21.15.70:0/2437407307 wait complete. 2024-04-01T01:24:05.591 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.362+0000 7f1c57080700 1 Processor -- start 2024-04-01T01:24:05.591 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.362+0000 7f1c57080700 1 -- start start 2024-04-01T01:24:05.591 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.362+0000 7f1c57080700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c501084a0 0x7f1c5019b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:05.591 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.362+0000 7f1c57080700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5010bda0 con 0x7f1c501084a0 2024-04-01T01:24:05.591 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.362+0000 7f1c54e1c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c501084a0 0x7f1c5019b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:05.591 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.362+0000 7f1c54e1c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c501084a0 0x7f1c5019b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47268/0 (socket says 172.21.15.70:47268) 2024-04-01T01:24:05.592 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.362+0000 7f1c54e1c700 1 -- 172.21.15.70:0/4230993865 learned_addr learned my addr 172.21.15.70:0/4230993865 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:05.592 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.362+0000 7f1c54e1c700 1 -- 172.21.15.70:0/4230993865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c5019bdf0 con 0x7f1c501084a0 2024-04-01T01:24:05.592 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.362+0000 7f1c54e1c700 1 --2- 172.21.15.70:0/4230993865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c501084a0 0x7f1c5019b8b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1c44011a50 tx=0x7f1c44004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:05.592 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.362+0000 7f1c4dffb700 1 -- 172.21.15.70:0/4230993865 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c440039f0 con 0x7f1c501084a0 2024-04-01T01:24:05.592 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.362+0000 7f1c4dffb700 1 -- 172.21.15.70:0/4230993865 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f1c44003b90 con 0x7f1c501084a0 2024-04-01T01:24:05.592 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.362+0000 7f1c57080700 1 -- 172.21.15.70:0/4230993865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c5019c080 con 0x7f1c501084a0 2024-04-01T01:24:05.592 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.362+0000 7f1c4dffb700 1 -- 172.21.15.70:0/4230993865 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c4401a600 con 0x7f1c501084a0 2024-04-01T01:24:05.592 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.363+0000 7f1c57080700 1 -- 172.21.15.70:0/4230993865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c5019ed70 con 0x7f1c501084a0 2024-04-01T01:24:05.592 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.363+0000 7f1c4dffb700 1 -- 172.21.15.70:0/4230993865 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7f1c4401a8b0 con 0x7f1c501084a0 2024-04-01T01:24:05.592 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.363+0000 7f1c57080700 1 -- 172.21.15.70:0/4230993865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c50063870 con 0x7f1c501084a0 2024-04-01T01:24:05.592 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.364+0000 7f1c4dffb700 1 --2- 172.21.15.70:0/4230993865 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f1c40038f10 0x7f1c4003b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:05.592 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.364+0000 7f1c4dffb700 1 -- 172.21.15.70:0/4230993865 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1c4404d310 con 0x7f1c501084a0 2024-04-01T01:24:05.592 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.364+0000 7f1c4ffff700 1 --2- 172.21.15.70:0/4230993865 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f1c40038f10 0x7f1c4003b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:05.592 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.364+0000 7f1c4ffff700 1 --2- 172.21.15.70:0/4230993865 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f1c40038f10 0x7f1c4003b3d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c006fd0 tx=0x7f1c3c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:05.592 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.369+0000 7f1c4dffb700 1 -- 172.21.15.70:0/4230993865 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1c4401ac50 con 0x7f1c501084a0 2024-04-01T01:24:05.593 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.586+0000 7f1c57080700 1 -- 172.21.15.70:0/4230993865 --> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f1c50103010 con 0x7f1c40038f10 2024-04-01T01:24:05.593 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.591+0000 7f1c4dffb700 1 -- 172.21.15.70:0/4230993865 <== mgr.14118 v2:172.21.15.70:6800/2836712828 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1c50103010 con 0x7f1c40038f10 2024-04-01T01:24:05.593 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.593+0000 7f1c57080700 1 -- 172.21.15.70:0/4230993865 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f1c40038f10 msgr2=0x7f1c4003b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:05.593 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.593+0000 7f1c57080700 1 --2- 172.21.15.70:0/4230993865 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f1c40038f10 0x7f1c4003b3d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c006fd0 tx=0x7f1c3c006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:05.593 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.593+0000 7f1c57080700 1 -- 172.21.15.70:0/4230993865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c501084a0 msgr2=0x7f1c5019b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:05.593 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.593+0000 7f1c57080700 1 --2- 172.21.15.70:0/4230993865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c501084a0 0x7f1c5019b8b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1c44011a50 tx=0x7f1c44004290 comp rx=0 tx=0).stop 2024-04-01T01:24:05.593 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.593+0000 7f1c57080700 1 -- 172.21.15.70:0/4230993865 shutdown_connections 2024-04-01T01:24:05.593 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.593+0000 7f1c57080700 1 --2- 172.21.15.70:0/4230993865 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f1c40038f10 0x7f1c4003b3d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:05.593 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.593+0000 7f1c57080700 1 --2- 172.21.15.70:0/4230993865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c501084a0 0x7f1c5019b8b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:05.593 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.593+0000 7f1c57080700 1 -- 172.21.15.70:0/4230993865 >> 172.21.15.70:0/4230993865 conn(0x7f1c50101ed0 msgr2=0x7f1c50102900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:05.593 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.593+0000 7f1c57080700 1 -- 172.21.15.70:0/4230993865 shutdown_connections 2024-04-01T01:24:05.593 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:04.593+0000 7f1c57080700 1 -- 172.21.15.70:0/4230993865 wait complete. 2024-04-01T01:24:05.593 INFO:teuthology.orchestra.run.smithi070.stdout:Deploying unmanaged mgr service... 2024-04-01T01:24:06.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:05 smithi070 ceph-mon[32179]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:24:06.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:05 smithi070 ceph-mon[32179]: Saving service mon spec with placement count:5 2024-04-01T01:24:06.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:05 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' 2024-04-01T01:24:07.373 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:07 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' 2024-04-01T01:24:07.373 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:07 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' 2024-04-01T01:24:07.373 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:07 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' 2024-04-01T01:24:07.381 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-01T01:24:07.381 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.554+0000 7fa953c75700 1 Processor -- start 2024-04-01T01:24:07.381 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.554+0000 7fa953c75700 1 -- start start 2024-04-01T01:24:07.381 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.555+0000 7fa953c75700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa94c107a60 0x7fa94c107e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:07.381 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.555+0000 7fa953c75700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c108450 con 0x7fa94c107a60 2024-04-01T01:24:07.381 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.555+0000 7fa951a11700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa94c107a60 0x7fa94c107e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:07.381 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.555+0000 7fa951a11700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa94c107a60 0x7fa94c107e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47274/0 (socket says 172.21.15.70:47274) 2024-04-01T01:24:07.381 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.555+0000 7fa951a11700 1 -- 172.21.15.70:0/3528476808 learned_addr learned my addr 172.21.15.70:0/3528476808 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:07.381 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.555+0000 7fa951a11700 1 -- 172.21.15.70:0/3528476808 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c108c60 con 0x7fa94c107a60 2024-04-01T01:24:07.381 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.555+0000 7fa951a11700 1 --2- 172.21.15.70:0/3528476808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa94c107a60 0x7fa94c107e80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa9480098a0 tx=0x7fa948009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92a4c172329fa2ea server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:07.382 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.555+0000 7fa950a0f700 1 -- 172.21.15.70:0/3528476808 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa948004160 con 0x7fa94c107a60 2024-04-01T01:24:07.382 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.555+0000 7fa950a0f700 1 -- 172.21.15.70:0/3528476808 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa948004300 con 0x7fa94c107a60 2024-04-01T01:24:07.382 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.555+0000 7fa950a0f700 1 -- 172.21.15.70:0/3528476808 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa948011a50 con 0x7fa94c107a60 2024-04-01T01:24:07.382 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.556+0000 7fa953c75700 1 -- 172.21.15.70:0/3528476808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa94c107a60 msgr2=0x7fa94c107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:07.382 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.556+0000 7fa953c75700 1 --2- 172.21.15.70:0/3528476808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa94c107a60 0x7fa94c107e80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa9480098a0 tx=0x7fa948009bb0 comp rx=0 tx=0).stop 2024-04-01T01:24:07.382 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.556+0000 7fa953c75700 1 -- 172.21.15.70:0/3528476808 shutdown_connections 2024-04-01T01:24:07.382 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.556+0000 7fa953c75700 1 --2- 172.21.15.70:0/3528476808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa94c107a60 0x7fa94c107e80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:07.382 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.556+0000 7fa953c75700 1 -- 172.21.15.70:0/3528476808 >> 172.21.15.70:0/3528476808 conn(0x7fa94c102fe0 msgr2=0x7fa94c105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:07.382 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.556+0000 7fa953c75700 1 -- 172.21.15.70:0/3528476808 shutdown_connections 2024-04-01T01:24:07.382 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.556+0000 7fa953c75700 1 -- 172.21.15.70:0/3528476808 wait complete. 2024-04-01T01:24:07.382 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.557+0000 7fa953c75700 1 Processor -- start 2024-04-01T01:24:07.382 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.557+0000 7fa953c75700 1 -- start start 2024-04-01T01:24:07.382 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.557+0000 7fa953c75700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa94c107a60 0x7fa94c07db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:07.382 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.557+0000 7fa953c75700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c10c010 con 0x7fa94c107a60 2024-04-01T01:24:07.383 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.557+0000 7fa951a11700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa94c107a60 0x7fa94c07db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:07.383 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.557+0000 7fa951a11700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa94c107a60 0x7fa94c07db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47278/0 (socket says 172.21.15.70:47278) 2024-04-01T01:24:07.383 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.557+0000 7fa951a11700 1 -- 172.21.15.70:0/69712831 learned_addr learned my addr 172.21.15.70:0/69712831 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:07.383 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.557+0000 7fa951a11700 1 -- 172.21.15.70:0/69712831 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c07e0a0 con 0x7fa94c107a60 2024-04-01T01:24:07.383 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.558+0000 7fa951a11700 1 --2- 172.21.15.70:0/69712831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa94c107a60 0x7fa94c07db60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa948006ba0 tx=0x7fa948011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:07.383 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.558+0000 7fa942ffd700 1 -- 172.21.15.70:0/69712831 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa948003af0 con 0x7fa94c107a60 2024-04-01T01:24:07.383 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.558+0000 7fa942ffd700 1 -- 172.21.15.70:0/69712831 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa94801a8e0 con 0x7fa94c107a60 2024-04-01T01:24:07.383 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.558+0000 7fa953c75700 1 -- 172.21.15.70:0/69712831 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94c07c290 con 0x7fa94c107a60 2024-04-01T01:24:07.383 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.558+0000 7fa942ffd700 1 -- 172.21.15.70:0/69712831 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa948024900 con 0x7fa94c107a60 2024-04-01T01:24:07.383 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.558+0000 7fa953c75700 1 -- 172.21.15.70:0/69712831 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa94c07c7d0 con 0x7fa94c107a60 2024-04-01T01:24:07.383 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.559+0000 7fa942ffd700 1 -- 172.21.15.70:0/69712831 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7fa948022020 con 0x7fa94c107a60 2024-04-01T01:24:07.383 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.559+0000 7fa953c75700 1 -- 172.21.15.70:0/69712831 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa94c063870 con 0x7fa94c107a60 2024-04-01T01:24:07.383 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.559+0000 7fa942ffd700 1 --2- 172.21.15.70:0/69712831 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fa938038ec0 0x7fa93803b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:07.383 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.559+0000 7fa942ffd700 1 -- 172.21.15.70:0/69712831 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa94804d0e0 con 0x7fa94c107a60 2024-04-01T01:24:07.383 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.559+0000 7fa951210700 1 --2- 172.21.15.70:0/69712831 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fa938038ec0 0x7fa93803b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:07.384 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.560+0000 7fa951210700 1 --2- 172.21.15.70:0/69712831 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fa938038ec0 0x7fa93803b380 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa93c006fd0 tx=0x7fa93c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:07.384 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.564+0000 7fa942ffd700 1 -- 172.21.15.70:0/69712831 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa94802c3a0 con 0x7fa94c107a60 2024-04-01T01:24:07.384 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.788+0000 7fa953c75700 1 -- 172.21.15.70:0/69712831 --> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fa94c07cfb0 con 0x7fa938038ec0 2024-04-01T01:24:07.384 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.793+0000 7fa942ffd700 1 -- 172.21.15.70:0/69712831 <== mgr.14118 v2:172.21.15.70:6800/2836712828 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fa94c07cfb0 con 0x7fa938038ec0 2024-04-01T01:24:07.384 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.795+0000 7fa953c75700 1 -- 172.21.15.70:0/69712831 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fa938038ec0 msgr2=0x7fa93803b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:07.384 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.795+0000 7fa953c75700 1 --2- 172.21.15.70:0/69712831 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fa938038ec0 0x7fa93803b380 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa93c006fd0 tx=0x7fa93c006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:07.384 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.795+0000 7fa953c75700 1 -- 172.21.15.70:0/69712831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa94c107a60 msgr2=0x7fa94c07db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:07.384 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.795+0000 7fa953c75700 1 --2- 172.21.15.70:0/69712831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa94c107a60 0x7fa94c07db60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa948006ba0 tx=0x7fa948011ad0 comp rx=0 tx=0).stop 2024-04-01T01:24:07.384 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.795+0000 7fa953c75700 1 -- 172.21.15.70:0/69712831 shutdown_connections 2024-04-01T01:24:07.384 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.795+0000 7fa953c75700 1 --2- 172.21.15.70:0/69712831 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fa938038ec0 0x7fa93803b380 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:07.384 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.795+0000 7fa953c75700 1 --2- 172.21.15.70:0/69712831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa94c107a60 0x7fa94c07db60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:07.384 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.795+0000 7fa953c75700 1 -- 172.21.15.70:0/69712831 >> 172.21.15.70:0/69712831 conn(0x7fa94c102fe0 msgr2=0x7fa94c103930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:07.384 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.795+0000 7fa953c75700 1 -- 172.21.15.70:0/69712831 shutdown_connections 2024-04-01T01:24:07.384 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:06.795+0000 7fa953c75700 1 -- 172.21.15.70:0/69712831 wait complete. 2024-04-01T01:24:08.491 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:08 smithi070 ceph-mon[32179]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:24:08.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:08 smithi070 ceph-mon[32179]: Saving service mgr spec with placement count:2 2024-04-01T01:24:08.492 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:08 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3541961773' entity='client.admin' 2024-04-01T01:24:09.199 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.795+0000 7f316400c700 1 Processor -- start 2024-04-01T01:24:09.199 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.795+0000 7f316400c700 1 -- start start 2024-04-01T01:24:09.199 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.795+0000 7f316400c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c07deb0 0x7f315c07c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:09.199 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.795+0000 7f316400c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f315c07c8e0 con 0x7f315c07deb0 2024-04-01T01:24:09.199 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.795+0000 7f3161da8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c07deb0 0x7f315c07c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:09.199 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.795+0000 7f3161da8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c07deb0 0x7f315c07c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44936/0 (socket says 172.21.15.70:44936) 2024-04-01T01:24:09.199 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.795+0000 7f3161da8700 1 -- 172.21.15.70:0/3872793028 learned_addr learned my addr 172.21.15.70:0/3872793028 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:09.200 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.795+0000 7f3161da8700 1 -- 172.21.15.70:0/3872793028 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315c07ca60 con 0x7f315c07deb0 2024-04-01T01:24:09.200 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.796+0000 7f3161da8700 1 --2- 172.21.15.70:0/3872793028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c07deb0 0x7f315c07c310 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f31580098a0 tx=0x7f3158009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603c845f493c8b6b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:09.200 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.796+0000 7f3160da6700 1 -- 172.21.15.70:0/3872793028 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3158004160 con 0x7f315c07deb0 2024-04-01T01:24:09.200 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.796+0000 7f3160da6700 1 -- 172.21.15.70:0/3872793028 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3158004300 con 0x7f315c07deb0 2024-04-01T01:24:09.200 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.796+0000 7f3160da6700 1 -- 172.21.15.70:0/3872793028 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3158011a50 con 0x7f315c07deb0 2024-04-01T01:24:09.200 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.796+0000 7f316400c700 1 -- 172.21.15.70:0/3872793028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c07deb0 msgr2=0x7f315c07c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:09.200 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.796+0000 7f316400c700 1 --2- 172.21.15.70:0/3872793028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c07deb0 0x7f315c07c310 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f31580098a0 tx=0x7f3158009bb0 comp rx=0 tx=0).stop 2024-04-01T01:24:09.200 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.797+0000 7f316400c700 1 -- 172.21.15.70:0/3872793028 shutdown_connections 2024-04-01T01:24:09.200 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.797+0000 7f316400c700 1 --2- 172.21.15.70:0/3872793028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c07deb0 0x7f315c07c310 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:09.200 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.797+0000 7f316400c700 1 -- 172.21.15.70:0/3872793028 >> 172.21.15.70:0/3872793028 conn(0x7f315c102ff0 msgr2=0x7f315c105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:09.200 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.797+0000 7f316400c700 1 -- 172.21.15.70:0/3872793028 shutdown_connections 2024-04-01T01:24:09.200 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.797+0000 7f316400c700 1 -- 172.21.15.70:0/3872793028 wait complete. 2024-04-01T01:24:09.200 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.797+0000 7f316400c700 1 Processor -- start 2024-04-01T01:24:09.200 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.797+0000 7f316400c700 1 -- start start 2024-04-01T01:24:09.201 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.797+0000 7f316400c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c07deb0 0x7f315c19da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:09.201 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.797+0000 7f316400c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f315c10e030 con 0x7f315c07deb0 2024-04-01T01:24:09.201 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.797+0000 7f3161da8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c07deb0 0x7f315c19da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:09.201 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.797+0000 7f3161da8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c07deb0 0x7f315c19da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44950/0 (socket says 172.21.15.70:44950) 2024-04-01T01:24:09.201 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.798+0000 7f3161da8700 1 -- 172.21.15.70:0/3541961773 learned_addr learned my addr 172.21.15.70:0/3541961773 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:09.201 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.798+0000 7f3161da8700 1 -- 172.21.15.70:0/3541961773 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315c19df80 con 0x7f315c07deb0 2024-04-01T01:24:09.201 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.798+0000 7f3161da8700 1 --2- 172.21.15.70:0/3541961773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c07deb0 0x7f315c19da40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3158006ba0 tx=0x7f3158011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:09.201 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.798+0000 7f3152ffd700 1 -- 172.21.15.70:0/3541961773 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3158003af0 con 0x7f315c07deb0 2024-04-01T01:24:09.201 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.798+0000 7f3152ffd700 1 -- 172.21.15.70:0/3541961773 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f315801a8e0 con 0x7f315c07deb0 2024-04-01T01:24:09.201 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.798+0000 7f316400c700 1 -- 172.21.15.70:0/3541961773 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f315c19e210 con 0x7f315c07deb0 2024-04-01T01:24:09.201 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.798+0000 7f3152ffd700 1 -- 172.21.15.70:0/3541961773 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3158024900 con 0x7f315c07deb0 2024-04-01T01:24:09.201 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.798+0000 7f316400c700 1 -- 172.21.15.70:0/3541961773 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f315c1a0f00 con 0x7f315c07deb0 2024-04-01T01:24:09.201 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.799+0000 7f3152ffd700 1 -- 172.21.15.70:0/3541961773 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7f3158022020 con 0x7f315c07deb0 2024-04-01T01:24:09.202 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.799+0000 7f316400c700 1 -- 172.21.15.70:0/3541961773 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f315c063870 con 0x7f315c07deb0 2024-04-01T01:24:09.202 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.799+0000 7f3152ffd700 1 --2- 172.21.15.70:0/3541961773 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f3148038ec0 0x7f314803b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:09.202 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.799+0000 7f3152ffd700 1 -- 172.21.15.70:0/3541961773 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f315804d340 con 0x7f315c07deb0 2024-04-01T01:24:09.202 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.799+0000 7f31615a7700 1 --2- 172.21.15.70:0/3541961773 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f3148038ec0 0x7f314803b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:09.202 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.799+0000 7f31615a7700 1 --2- 172.21.15.70:0/3541961773 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f3148038ec0 0x7f314803b380 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f314c006fd0 tx=0x7f314c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:09.202 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:07.804+0000 7f3152ffd700 1 -- 172.21.15.70:0/3541961773 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f315800b3e0 con 0x7f315c07deb0 2024-04-01T01:24:09.202 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:08.022+0000 7f316400c700 1 -- 172.21.15.70:0/3541961773 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f315c1a14b0 con 0x7f315c07deb0 2024-04-01T01:24:09.202 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:08.025+0000 7f3152ffd700 1 -- 172.21.15.70:0/3541961773 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f315800fdd0 con 0x7f315c07deb0 2024-04-01T01:24:09.202 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:08.027+0000 7f316400c700 1 -- 172.21.15.70:0/3541961773 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f3148038ec0 msgr2=0x7f314803b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:09.202 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:08.027+0000 7f316400c700 1 --2- 172.21.15.70:0/3541961773 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f3148038ec0 0x7f314803b380 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f314c006fd0 tx=0x7f314c006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:09.202 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:08.027+0000 7f316400c700 1 -- 172.21.15.70:0/3541961773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c07deb0 msgr2=0x7f315c19da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:09.202 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:08.027+0000 7f316400c700 1 --2- 172.21.15.70:0/3541961773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c07deb0 0x7f315c19da40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3158006ba0 tx=0x7f3158011ad0 comp rx=0 tx=0).stop 2024-04-01T01:24:09.202 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:08.027+0000 7f316400c700 1 -- 172.21.15.70:0/3541961773 shutdown_connections 2024-04-01T01:24:09.203 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:08.027+0000 7f316400c700 1 --2- 172.21.15.70:0/3541961773 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f3148038ec0 0x7f314803b380 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:09.203 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:08.027+0000 7f316400c700 1 --2- 172.21.15.70:0/3541961773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c07deb0 0x7f315c19da40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:09.203 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:08.027+0000 7f316400c700 1 -- 172.21.15.70:0/3541961773 >> 172.21.15.70:0/3541961773 conn(0x7f315c102ff0 msgr2=0x7f315c103980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:09.203 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:08.027+0000 7f316400c700 1 -- 172.21.15.70:0/3541961773 shutdown_connections 2024-04-01T01:24:09.203 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:08.027+0000 7f316400c700 1 -- 172.21.15.70:0/3541961773 wait complete. 2024-04-01T01:24:10.189 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:09 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' 2024-04-01T01:24:10.189 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:09 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3027502508' entity='client.admin' 2024-04-01T01:24:10.513 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.652+0000 7f888b306700 1 Processor -- start 2024-04-01T01:24:10.513 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.652+0000 7f888b306700 1 -- start start 2024-04-01T01:24:10.513 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.652+0000 7f888b306700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88841084e0 0x7f888410a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:10.513 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.652+0000 7f888b306700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f888407be60 con 0x7f88841084e0 2024-04-01T01:24:10.513 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.652+0000 7f88890a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88841084e0 0x7f888410a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:10.513 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.652+0000 7f88890a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88841084e0 0x7f888410a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44966/0 (socket says 172.21.15.70:44966) 2024-04-01T01:24:10.513 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.652+0000 7f88890a2700 1 -- 172.21.15.70:0/1852941166 learned_addr learned my addr 172.21.15.70:0/1852941166 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:10.513 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.652+0000 7f88890a2700 1 -- 172.21.15.70:0/1852941166 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f888410ae10 con 0x7f88841084e0 2024-04-01T01:24:10.513 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.653+0000 7f88890a2700 1 --2- 172.21.15.70:0/1852941166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88841084e0 0x7f888410a8d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f887c0098a0 tx=0x7f887c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c6776ca47e77a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:10.513 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.653+0000 7f8877fff700 1 -- 172.21.15.70:0/1852941166 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f887c004160 con 0x7f88841084e0 2024-04-01T01:24:10.514 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.653+0000 7f8877fff700 1 -- 172.21.15.70:0/1852941166 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f887c004300 con 0x7f88841084e0 2024-04-01T01:24:10.514 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.653+0000 7f8877fff700 1 -- 172.21.15.70:0/1852941166 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f887c011a50 con 0x7f88841084e0 2024-04-01T01:24:10.514 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.653+0000 7f888b306700 1 -- 172.21.15.70:0/1852941166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88841084e0 msgr2=0x7f888410a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:10.514 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.653+0000 7f888b306700 1 --2- 172.21.15.70:0/1852941166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88841084e0 0x7f888410a8d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f887c0098a0 tx=0x7f887c009bb0 comp rx=0 tx=0).stop 2024-04-01T01:24:10.514 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.653+0000 7f888b306700 1 -- 172.21.15.70:0/1852941166 shutdown_connections 2024-04-01T01:24:10.514 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.654+0000 7f888b306700 1 --2- 172.21.15.70:0/1852941166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88841084e0 0x7f888410a8d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:10.514 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.654+0000 7f888b306700 1 -- 172.21.15.70:0/1852941166 >> 172.21.15.70:0/1852941166 conn(0x7f8884101ed0 msgr2=0x7f8884104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:10.514 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.654+0000 7f888b306700 1 -- 172.21.15.70:0/1852941166 shutdown_connections 2024-04-01T01:24:10.514 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.654+0000 7f888b306700 1 -- 172.21.15.70:0/1852941166 wait complete. 2024-04-01T01:24:10.514 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.654+0000 7f888b306700 1 Processor -- start 2024-04-01T01:24:10.514 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.654+0000 7f888b306700 1 -- start start 2024-04-01T01:24:10.514 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.654+0000 7f888b306700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88841084e0 0x7f888419b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:10.514 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.654+0000 7f888b306700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f888410be10 con 0x7f88841084e0 2024-04-01T01:24:10.514 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.654+0000 7f88890a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88841084e0 0x7f888419b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:10.514 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.654+0000 7f88890a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88841084e0 0x7f888419b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44978/0 (socket says 172.21.15.70:44978) 2024-04-01T01:24:10.515 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.654+0000 7f88890a2700 1 -- 172.21.15.70:0/3027502508 learned_addr learned my addr 172.21.15.70:0/3027502508 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:10.515 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.655+0000 7f88890a2700 1 -- 172.21.15.70:0/3027502508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f888419bd90 con 0x7f88841084e0 2024-04-01T01:24:10.515 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.655+0000 7f88890a2700 1 --2- 172.21.15.70:0/3027502508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88841084e0 0x7f888419b850 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f887c011a20 tx=0x7f887c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:10.515 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.655+0000 7f88767fc700 1 -- 172.21.15.70:0/3027502508 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f887c0039c0 con 0x7f88841084e0 2024-04-01T01:24:10.515 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.655+0000 7f888b306700 1 -- 172.21.15.70:0/3027502508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f888419c020 con 0x7f88841084e0 2024-04-01T01:24:10.515 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.655+0000 7f88767fc700 1 -- 172.21.15.70:0/3027502508 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f887c003b60 con 0x7f88841084e0 2024-04-01T01:24:10.515 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.655+0000 7f88767fc700 1 -- 172.21.15.70:0/3027502508 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f887c01a6b0 con 0x7f88841084e0 2024-04-01T01:24:10.515 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.655+0000 7f888b306700 1 -- 172.21.15.70:0/3027502508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f888419ed10 con 0x7f88841084e0 2024-04-01T01:24:10.515 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.656+0000 7f88767fc700 1 -- 172.21.15.70:0/3027502508 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7f887c01a9d0 con 0x7f88841084e0 2024-04-01T01:24:10.515 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.656+0000 7f888b306700 1 -- 172.21.15.70:0/3027502508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8884063870 con 0x7f88841084e0 2024-04-01T01:24:10.515 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.656+0000 7f88767fc700 1 --2- 172.21.15.70:0/3027502508 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f8870038e70 0x7f887003b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:10.515 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.656+0000 7f88767fc700 1 -- 172.21.15.70:0/3027502508 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f887c04d370 con 0x7f88841084e0 2024-04-01T01:24:10.515 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.656+0000 7f88888a1700 1 --2- 172.21.15.70:0/3027502508 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f8870038e70 0x7f887003b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:10.515 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.656+0000 7f88888a1700 1 --2- 172.21.15.70:0/3027502508 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f8870038e70 0x7f887003b330 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f888000ad30 tx=0x7f88800093f0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:10.515 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.661+0000 7f88767fc700 1 -- 172.21.15.70:0/3027502508 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f887c019c70 con 0x7f88841084e0 2024-04-01T01:24:10.515 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.872+0000 7f888b306700 1 -- 172.21.15.70:0/3027502508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f888419eee0 con 0x7f88841084e0 2024-04-01T01:24:10.516 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.875+0000 7f88767fc700 1 -- 172.21.15.70:0/3027502508 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f887c01ad70 con 0x7f88841084e0 2024-04-01T01:24:10.516 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.877+0000 7f888b306700 1 -- 172.21.15.70:0/3027502508 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f8870038e70 msgr2=0x7f887003b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:10.516 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.877+0000 7f888b306700 1 --2- 172.21.15.70:0/3027502508 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f8870038e70 0x7f887003b330 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f888000ad30 tx=0x7f88800093f0 comp rx=0 tx=0).stop 2024-04-01T01:24:10.516 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.877+0000 7f888b306700 1 -- 172.21.15.70:0/3027502508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88841084e0 msgr2=0x7f888419b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:10.516 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.877+0000 7f888b306700 1 --2- 172.21.15.70:0/3027502508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88841084e0 0x7f888419b850 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f887c011a20 tx=0x7f887c011b00 comp rx=0 tx=0).stop 2024-04-01T01:24:10.516 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.878+0000 7f888b306700 1 -- 172.21.15.70:0/3027502508 shutdown_connections 2024-04-01T01:24:10.516 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.878+0000 7f888b306700 1 --2- 172.21.15.70:0/3027502508 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f8870038e70 0x7f887003b330 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:10.516 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.878+0000 7f888b306700 1 --2- 172.21.15.70:0/3027502508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88841084e0 0x7f888419b850 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:10.516 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.878+0000 7f888b306700 1 -- 172.21.15.70:0/3027502508 >> 172.21.15.70:0/3027502508 conn(0x7f8884101ed0 msgr2=0x7f88841028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:10.516 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.878+0000 7f888b306700 1 -- 172.21.15.70:0/3027502508 shutdown_connections 2024-04-01T01:24:10.516 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:09.878+0000 7f888b306700 1 -- 172.21.15.70:0/3027502508 wait complete. 2024-04-01T01:24:10.516 INFO:teuthology.orchestra.run.smithi070.stdout:Enabling the dashboard module... 2024-04-01T01:24:11.751 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:11 smithi070 ceph-mon[32179]: from='mgr.14118 172.21.15.70:0/2700915255' entity='mgr.x' 2024-04-01T01:24:11.751 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:11 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3641055646' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-01T01:24:11.752 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:11 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: ignoring --setuser ceph since I am not root 2024-04-01T01:24:11.752 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:11 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: ignoring --setgroup ceph since I am not root 2024-04-01T01:24:12.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.929+0000 7fb96ccd4700 1 Processor -- start 2024-04-01T01:24:12.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.929+0000 7fb96ccd4700 1 -- start start 2024-04-01T01:24:12.316 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.929+0000 7fb96ccd4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9681084f0 0x7fb96810a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:12.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.929+0000 7fb96ccd4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb96807be80 con 0x7fb9681084f0 2024-04-01T01:24:12.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.929+0000 7fb96659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9681084f0 0x7fb96810a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:12.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.929+0000 7fb96659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9681084f0 0x7fb96810a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44986/0 (socket says 172.21.15.70:44986) 2024-04-01T01:24:12.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.929+0000 7fb96659c700 1 -- 172.21.15.70:0/2169935946 learned_addr learned my addr 172.21.15.70:0/2169935946 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:12.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.930+0000 7fb96659c700 1 -- 172.21.15.70:0/2169935946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb96810ae20 con 0x7fb9681084f0 2024-04-01T01:24:12.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.930+0000 7fb96659c700 1 --2- 172.21.15.70:0/2169935946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9681084f0 0x7fb96810a8e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb9580098a0 tx=0x7fb958009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed9b23a5122dc4d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:12.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.930+0000 7fb96559a700 1 -- 172.21.15.70:0/2169935946 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb958004160 con 0x7fb9681084f0 2024-04-01T01:24:12.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.930+0000 7fb96559a700 1 -- 172.21.15.70:0/2169935946 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb958004300 con 0x7fb9681084f0 2024-04-01T01:24:12.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.930+0000 7fb96559a700 1 -- 172.21.15.70:0/2169935946 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb958011a50 con 0x7fb9681084f0 2024-04-01T01:24:12.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.931+0000 7fb96ccd4700 1 -- 172.21.15.70:0/2169935946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9681084f0 msgr2=0x7fb96810a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:12.317 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.931+0000 7fb96ccd4700 1 --2- 172.21.15.70:0/2169935946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9681084f0 0x7fb96810a8e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb9580098a0 tx=0x7fb958009bb0 comp rx=0 tx=0).stop 2024-04-01T01:24:12.318 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.931+0000 7fb96ccd4700 1 -- 172.21.15.70:0/2169935946 shutdown_connections 2024-04-01T01:24:12.318 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.931+0000 7fb96ccd4700 1 --2- 172.21.15.70:0/2169935946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9681084f0 0x7fb96810a8e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:12.318 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.931+0000 7fb96ccd4700 1 -- 172.21.15.70:0/2169935946 >> 172.21.15.70:0/2169935946 conn(0x7fb968101f20 msgr2=0x7fb968104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:12.318 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.931+0000 7fb96ccd4700 1 -- 172.21.15.70:0/2169935946 shutdown_connections 2024-04-01T01:24:12.318 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.931+0000 7fb96ccd4700 1 -- 172.21.15.70:0/2169935946 wait complete. 2024-04-01T01:24:12.318 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.932+0000 7fb96ccd4700 1 Processor -- start 2024-04-01T01:24:12.318 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.932+0000 7fb96ccd4700 1 -- start start 2024-04-01T01:24:12.318 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.932+0000 7fb96ccd4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9681084f0 0x7fb968107d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:12.318 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.932+0000 7fb96ccd4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb96810bdc0 con 0x7fb9681084f0 2024-04-01T01:24:12.318 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.932+0000 7fb96659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9681084f0 0x7fb968107d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:12.318 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.932+0000 7fb96659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9681084f0 0x7fb968107d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44994/0 (socket says 172.21.15.70:44994) 2024-04-01T01:24:12.318 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.932+0000 7fb96659c700 1 -- 172.21.15.70:0/3641055646 learned_addr learned my addr 172.21.15.70:0/3641055646 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:12.318 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.932+0000 7fb96659c700 1 -- 172.21.15.70:0/3641055646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9681082b0 con 0x7fb9681084f0 2024-04-01T01:24:12.318 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.933+0000 7fb96659c700 1 --2- 172.21.15.70:0/3641055646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9681084f0 0x7fb968107d70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb958011a20 tx=0x7fb958011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:12.319 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.933+0000 7fb9577fe700 1 -- 172.21.15.70:0/3641055646 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9580039c0 con 0x7fb9681084f0 2024-04-01T01:24:12.319 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.933+0000 7fb9577fe700 1 -- 172.21.15.70:0/3641055646 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb958003b60 con 0x7fb9681084f0 2024-04-01T01:24:12.319 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.933+0000 7fb96ccd4700 1 -- 172.21.15.70:0/3641055646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9681064a0 con 0x7fb9681084f0 2024-04-01T01:24:12.319 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.933+0000 7fb9577fe700 1 -- 172.21.15.70:0/3641055646 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb95801a6b0 con 0x7fb9681084f0 2024-04-01T01:24:12.319 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.933+0000 7fb96ccd4700 1 -- 172.21.15.70:0/3641055646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9681069e0 con 0x7fb9681084f0 2024-04-01T01:24:12.319 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.934+0000 7fb9577fe700 1 -- 172.21.15.70:0/3641055646 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 8) ==== 46116+0+0 (secure 0 0 0) 0x7fb95801aa30 con 0x7fb9681084f0 2024-04-01T01:24:12.319 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.934+0000 7fb9577fe700 1 --2- 172.21.15.70:0/3641055646 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fb950038db0 0x7fb95003b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:12.319 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.934+0000 7fb9577fe700 1 -- 172.21.15.70:0/3641055646 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb95804e1a0 con 0x7fb9681084f0 2024-04-01T01:24:12.319 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.934+0000 7fb965d9b700 1 --2- 172.21.15.70:0/3641055646 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fb950038db0 0x7fb95003b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:12.319 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.934+0000 7fb96ccd4700 1 -- 172.21.15.70:0/3641055646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb968063870 con 0x7fb9681084f0 2024-04-01T01:24:12.319 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.935+0000 7fb965d9b700 1 --2- 172.21.15.70:0/3641055646 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fb950038db0 0x7fb95003b270 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb95c006fd0 tx=0x7fb95c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:12.319 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:10.939+0000 7fb9577fe700 1 -- 172.21.15.70:0/3641055646 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb95800bc40 con 0x7fb9681084f0 2024-04-01T01:24:12.319 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:11.207+0000 7fb96ccd4700 1 -- 172.21.15.70:0/3641055646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fb968107060 con 0x7fb9681084f0 2024-04-01T01:24:12.320 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:11.578+0000 7fb9577fe700 1 -- 172.21.15.70:0/3641055646 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fb958024a70 con 0x7fb9681084f0 2024-04-01T01:24:12.320 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:11.580+0000 7fb9577fe700 1 -- 172.21.15.70:0/3641055646 <== mon.0 v2:172.21.15.70:3300/0 8 ==== mgrmap(e 9) ==== 46129+0+0 (secure 0 0 0) 0x7fb95803cd10 con 0x7fb9681084f0 2024-04-01T01:24:12.320 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:11.581+0000 7fb96ccd4700 1 -- 172.21.15.70:0/3641055646 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fb950038db0 msgr2=0x7fb95003b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:12.320 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:11.581+0000 7fb96ccd4700 1 --2- 172.21.15.70:0/3641055646 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fb950038db0 0x7fb95003b270 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb95c006fd0 tx=0x7fb95c006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:12.320 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:11.581+0000 7fb96ccd4700 1 -- 172.21.15.70:0/3641055646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9681084f0 msgr2=0x7fb968107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:12.320 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:11.581+0000 7fb96ccd4700 1 --2- 172.21.15.70:0/3641055646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9681084f0 0x7fb968107d70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb958011a20 tx=0x7fb958011b00 comp rx=0 tx=0).stop 2024-04-01T01:24:12.320 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:11.581+0000 7fb96ccd4700 1 -- 172.21.15.70:0/3641055646 shutdown_connections 2024-04-01T01:24:12.320 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:11.581+0000 7fb96ccd4700 1 --2- 172.21.15.70:0/3641055646 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7fb950038db0 0x7fb95003b270 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:12.320 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:11.581+0000 7fb96ccd4700 1 --2- 172.21.15.70:0/3641055646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9681084f0 0x7fb968107d70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:12.320 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:11.581+0000 7fb96ccd4700 1 -- 172.21.15.70:0/3641055646 >> 172.21.15.70:0/3641055646 conn(0x7fb968101f20 msgr2=0x7fb968102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:12.320 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:11.581+0000 7fb96ccd4700 1 -- 172.21.15.70:0/3641055646 shutdown_connections 2024-04-01T01:24:12.320 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:11.582+0000 7fb96ccd4700 1 -- 172.21.15.70:0/3641055646 wait complete. 2024-04-01T01:24:12.441 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:12 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:12.166+0000 7f384afd4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-01T01:24:12.441 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:12 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:12.263+0000 7f384afd4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-01T01:24:12.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:12 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3641055646' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-01T01:24:12.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:12 smithi070 ceph-mon[32179]: mgrmap e9: x(active, since 19s) 2024-04-01T01:24:13.031 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:12 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:12.936+0000 7f384afd4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-01T01:24:13.031 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:13 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:13.030+0000 7f384afd4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-01T01:24:13.361 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:13 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:13.180+0000 7f384afd4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-01T01:24:13.361 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:13 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:13.276+0000 7f384afd4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-01T01:24:13.361 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:13 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:13.360+0000 7f384afd4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-01T01:24:13.628 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:13 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/403849232' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-01T01:24:13.750 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-04-01T01:24:13.750 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-01T01:24:13.750 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-01T01:24:13.750 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-01T01:24:13.750 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-01T01:24:13.750 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-04-01T01:24:13.750 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.781+0000 7f9c1fb90700 1 Processor -- start 2024-04-01T01:24:13.750 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.781+0000 7f9c1fb90700 1 -- start start 2024-04-01T01:24:13.750 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.781+0000 7f9c1fb90700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c18105d30 0x7f9c18106150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:13.751 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.781+0000 7f9c1fb90700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9c18106720 con 0x7f9c18105d30 2024-04-01T01:24:13.751 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.781+0000 7f9c1d92c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c18105d30 0x7f9c18106150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:13.751 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.781+0000 7f9c1d92c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c18105d30 0x7f9c18106150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45032/0 (socket says 172.21.15.70:45032) 2024-04-01T01:24:13.751 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.781+0000 7f9c1d92c700 1 -- 172.21.15.70:0/3677125241 learned_addr learned my addr 172.21.15.70:0/3677125241 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:13.751 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.782+0000 7f9c1d92c700 1 -- 172.21.15.70:0/3677125241 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c18108d60 con 0x7f9c18105d30 2024-04-01T01:24:13.751 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.782+0000 7f9c1d92c700 1 --2- 172.21.15.70:0/3677125241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c18105d30 0x7f9c18106150 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9c080098a0 tx=0x7f9c08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9db28c09f37fabf server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:13.751 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.782+0000 7f9c1c92a700 1 -- 172.21.15.70:0/3677125241 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c08004160 con 0x7f9c18105d30 2024-04-01T01:24:13.751 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.782+0000 7f9c1c92a700 1 -- 172.21.15.70:0/3677125241 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c08004300 con 0x7f9c18105d30 2024-04-01T01:24:13.751 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.782+0000 7f9c1c92a700 1 -- 172.21.15.70:0/3677125241 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c08011a50 con 0x7f9c18105d30 2024-04-01T01:24:13.751 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.783+0000 7f9c1fb90700 1 -- 172.21.15.70:0/3677125241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c18105d30 msgr2=0x7f9c18106150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:13.751 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.783+0000 7f9c1fb90700 1 --2- 172.21.15.70:0/3677125241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c18105d30 0x7f9c18106150 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9c080098a0 tx=0x7f9c08009bb0 comp rx=0 tx=0).stop 2024-04-01T01:24:13.751 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.783+0000 7f9c1fb90700 1 -- 172.21.15.70:0/3677125241 shutdown_connections 2024-04-01T01:24:13.751 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.783+0000 7f9c1fb90700 1 --2- 172.21.15.70:0/3677125241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c18105d30 0x7f9c18106150 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:13.751 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.783+0000 7f9c1fb90700 1 -- 172.21.15.70:0/3677125241 >> 172.21.15.70:0/3677125241 conn(0x7f9c18101620 msgr2=0x7f9c18103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:13.751 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.783+0000 7f9c1fb90700 1 -- 172.21.15.70:0/3677125241 shutdown_connections 2024-04-01T01:24:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.783+0000 7f9c1fb90700 1 -- 172.21.15.70:0/3677125241 wait complete. 2024-04-01T01:24:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.783+0000 7f9c1fb90700 1 Processor -- start 2024-04-01T01:24:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.783+0000 7f9c1fb90700 1 -- start start 2024-04-01T01:24:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.784+0000 7f9c1fb90700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c18105d30 0x7f9c1819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.784+0000 7f9c1fb90700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1810bf90 con 0x7f9c18105d30 2024-04-01T01:24:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.784+0000 7f9c1d92c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c18105d30 0x7f9c1819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.784+0000 7f9c1d92c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c18105d30 0x7f9c1819b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45034/0 (socket says 172.21.15.70:45034) 2024-04-01T01:24:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.784+0000 7f9c1d92c700 1 -- 172.21.15.70:0/403849232 learned_addr learned my addr 172.21.15.70:0/403849232 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.784+0000 7f9c1d92c700 1 -- 172.21.15.70:0/403849232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c1819bde0 con 0x7f9c18105d30 2024-04-01T01:24:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.784+0000 7f9c1d92c700 1 --2- 172.21.15.70:0/403849232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c18105d30 0x7f9c1819b8a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c08011a20 tx=0x7f9c08011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.784+0000 7f9c0effd700 1 -- 172.21.15.70:0/403849232 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c08003af0 con 0x7f9c18105d30 2024-04-01T01:24:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.784+0000 7f9c0effd700 1 -- 172.21.15.70:0/403849232 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c0801a8e0 con 0x7f9c18105d30 2024-04-01T01:24:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.784+0000 7f9c1fb90700 1 -- 172.21.15.70:0/403849232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c1819c070 con 0x7f9c18105d30 2024-04-01T01:24:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.784+0000 7f9c0effd700 1 -- 172.21.15.70:0/403849232 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c08024860 con 0x7f9c18105d30 2024-04-01T01:24:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.784+0000 7f9c1fb90700 1 -- 172.21.15.70:0/403849232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c1819ed60 con 0x7f9c18105d30 2024-04-01T01:24:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.785+0000 7f9c0effd700 1 -- 172.21.15.70:0/403849232 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 9) ==== 46129+0+0 (secure 0 0 0) 0x7f9c08022020 con 0x7f9c18105d30 2024-04-01T01:24:13.753 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.785+0000 7f9c1fb90700 1 -- 172.21.15.70:0/403849232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c18063870 con 0x7f9c18105d30 2024-04-01T01:24:13.753 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.786+0000 7f9c0effd700 1 --2- 172.21.15.70:0/403849232 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f9c04038f10 0x7f9c0403b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:13.753 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.786+0000 7f9c0effd700 1 -- 172.21.15.70:0/403849232 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9c0804d060 con 0x7f9c18105d30 2024-04-01T01:24:13.753 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.786+0000 7f9c1d12b700 1 -- 172.21.15.70:0/403849232 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f9c04038f10 msgr2=0x7f9c0403b3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2836712828 2024-04-01T01:24:13.753 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.786+0000 7f9c1d12b700 1 --2- 172.21.15.70:0/403849232 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f9c04038f10 0x7f9c0403b3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-01T01:24:13.753 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.790+0000 7f9c0effd700 1 -- 172.21.15.70:0/403849232 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9c0802c3a0 con 0x7f9c18105d30 2024-04-01T01:24:13.753 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.987+0000 7f9c1d12b700 1 -- 172.21.15.70:0/403849232 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f9c04038f10 msgr2=0x7f9c0403b3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2836712828 2024-04-01T01:24:13.753 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:12.987+0000 7f9c1d12b700 1 --2- 172.21.15.70:0/403849232 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f9c04038f10 0x7f9c0403b3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-01T01:24:13.753 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:13.047+0000 7f9c1fb90700 1 -- 172.21.15.70:0/403849232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f9c1819ef30 con 0x7f9c18105d30 2024-04-01T01:24:13.753 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:13.047+0000 7f9c0effd700 1 -- 172.21.15.70:0/403849232 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f9c08016260 con 0x7f9c18105d30 2024-04-01T01:24:13.753 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:13.049+0000 7f9c1fb90700 1 -- 172.21.15.70:0/403849232 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f9c04038f10 msgr2=0x7f9c0403b3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:24:13.753 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:13.049+0000 7f9c1fb90700 1 --2- 172.21.15.70:0/403849232 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f9c04038f10 0x7f9c0403b3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:13.753 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:13.049+0000 7f9c1fb90700 1 -- 172.21.15.70:0/403849232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c18105d30 msgr2=0x7f9c1819b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:13.754 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:13.049+0000 7f9c1fb90700 1 --2- 172.21.15.70:0/403849232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c18105d30 0x7f9c1819b8a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c08011a20 tx=0x7f9c08011b00 comp rx=0 tx=0).stop 2024-04-01T01:24:13.754 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:13.050+0000 7f9c1fb90700 1 -- 172.21.15.70:0/403849232 shutdown_connections 2024-04-01T01:24:13.754 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:13.050+0000 7f9c1fb90700 1 --2- 172.21.15.70:0/403849232 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f9c04038f10 0x7f9c0403b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:13.754 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:13.050+0000 7f9c1fb90700 1 --2- 172.21.15.70:0/403849232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c18105d30 0x7f9c1819b8a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:13.754 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:13.050+0000 7f9c1fb90700 1 -- 172.21.15.70:0/403849232 >> 172.21.15.70:0/403849232 conn(0x7f9c18101620 msgr2=0x7f9c18102080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:13.754 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:13.050+0000 7f9c1fb90700 1 -- 172.21.15.70:0/403849232 shutdown_connections 2024-04-01T01:24:13.754 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:13.050+0000 7f9c1fb90700 1 -- 172.21.15.70:0/403849232 wait complete. 2024-04-01T01:24:13.754 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for the mgr to restart... 2024-04-01T01:24:13.754 INFO:teuthology.orchestra.run.smithi070.stdout:Waiting for mgr epoch 9... 2024-04-01T01:24:13.929 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:13 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:13.628+0000 7f384afd4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-01T01:24:13.929 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:13 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:13.740+0000 7f384afd4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-01T01:24:13.929 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:13 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:13.831+0000 7f384afd4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-01T01:24:13.929 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:13 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:13.928+0000 7f384afd4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-01T01:24:14.253 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:14 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:14.112+0000 7f384afd4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-01T01:24:15.503 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:15 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:15.073+0000 7f384afd4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-01T01:24:15.504 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:15 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:15.203+0000 7f384afd4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-01T01:24:15.821 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:15 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:15.535+0000 7f384afd4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-01T01:24:15.822 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:15 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:15.625+0000 7f384afd4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-01T01:24:15.822 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:15 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:15.821+0000 7f384afd4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-01T01:24:16.253 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:16 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:16.005+0000 7f384afd4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-01T01:24:16.548 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:16 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:16.290+0000 7f384afd4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-01T01:24:16.549 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:16 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:16.548+0000 7f384afd4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-01T01:24:17.003 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:16 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:16.785+0000 7f384afd4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-01T01:24:18.003 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:17 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:17.590+0000 7f384afd4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-01T01:24:18.475 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:18 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:18.158+0000 7f384afd4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-01T01:24:18.475 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:18 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:18.257+0000 7f384afd4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-01T01:24:18.475 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:24:18 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:24:18.475+0000 7f384afd4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-01T01:24:18.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:18 smithi070 ceph-mon[32179]: Active manager daemon x restarted 2024-04-01T01:24:18.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:18 smithi070 ceph-mon[32179]: Activating manager daemon x 2024-04-01T01:24:18.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:18 smithi070 ceph-mon[32179]: osdmap e3: 0 total, 0 up, 0 in 2024-04-01T01:24:18.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:18 smithi070 ceph-mon[32179]: mgrmap e10: x(active, starting, since 0.0033145s) 2024-04-01T01:24:18.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:24:18.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-01T01:24:18.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-01T01:24:18.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-01T01:24:18.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-01T01:24:18.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:18 smithi070 ceph-mon[32179]: Manager daemon x is now available 2024-04-01T01:24:18.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:24:18.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-01T01:24:19.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-01T01:24:19.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:19 smithi070 ceph-mon[32179]: mgrmap e11: x(active, since 1.00656s) 2024-04-01T01:24:20.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout { 2024-04-01T01:24:20.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-01T01:24:20.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-01T01:24:20.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout } 2024-04-01T01:24:20.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.162+0000 7f2b3d563700 1 Processor -- start 2024-04-01T01:24:20.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.162+0000 7f2b3d563700 1 -- start start 2024-04-01T01:24:20.058 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.162+0000 7f2b3d563700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b38108540 0x7f2b3810a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:20.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.162+0000 7f2b3d563700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3807be00 con 0x7f2b38108540 2024-04-01T01:24:20.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.162+0000 7f2b36ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b38108540 0x7f2b3810a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:20.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.162+0000 7f2b36ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b38108540 0x7f2b3810a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45048/0 (socket says 172.21.15.70:45048) 2024-04-01T01:24:20.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.163+0000 7f2b36ffd700 1 -- 172.21.15.70:0/910172127 learned_addr learned my addr 172.21.15.70:0/910172127 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:20.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.163+0000 7f2b36ffd700 1 -- 172.21.15.70:0/910172127 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b3810ae70 con 0x7f2b38108540 2024-04-01T01:24:20.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.163+0000 7f2b36ffd700 1 --2- 172.21.15.70:0/910172127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b38108540 0x7f2b3810a930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2b280098a0 tx=0x7f2b28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4531105e1d068a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:20.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.163+0000 7f2b35ffb700 1 -- 172.21.15.70:0/910172127 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b28004160 con 0x7f2b38108540 2024-04-01T01:24:20.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.163+0000 7f2b35ffb700 1 -- 172.21.15.70:0/910172127 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b28004300 con 0x7f2b38108540 2024-04-01T01:24:20.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.163+0000 7f2b35ffb700 1 -- 172.21.15.70:0/910172127 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b28011a50 con 0x7f2b38108540 2024-04-01T01:24:20.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.164+0000 7f2b3d563700 1 -- 172.21.15.70:0/910172127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b38108540 msgr2=0x7f2b3810a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:20.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.164+0000 7f2b3d563700 1 --2- 172.21.15.70:0/910172127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b38108540 0x7f2b3810a930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2b280098a0 tx=0x7f2b28009bb0 comp rx=0 tx=0).stop 2024-04-01T01:24:20.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.164+0000 7f2b3d563700 1 -- 172.21.15.70:0/910172127 shutdown_connections 2024-04-01T01:24:20.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.164+0000 7f2b3d563700 1 --2- 172.21.15.70:0/910172127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b38108540 0x7f2b3810a930 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:20.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.164+0000 7f2b3d563700 1 -- 172.21.15.70:0/910172127 >> 172.21.15.70:0/910172127 conn(0x7f2b38101ed0 msgr2=0x7f2b38104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:20.059 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.164+0000 7f2b3d563700 1 -- 172.21.15.70:0/910172127 shutdown_connections 2024-04-01T01:24:20.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.164+0000 7f2b3d563700 1 -- 172.21.15.70:0/910172127 wait complete. 2024-04-01T01:24:20.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.165+0000 7f2b3d563700 1 Processor -- start 2024-04-01T01:24:20.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.165+0000 7f2b3d563700 1 -- start start 2024-04-01T01:24:20.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.165+0000 7f2b3d563700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b38108540 0x7f2b3819b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:20.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.165+0000 7f2b3d563700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3810be10 con 0x7f2b38108540 2024-04-01T01:24:20.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.165+0000 7f2b36ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b38108540 0x7f2b3819b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:20.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.165+0000 7f2b36ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b38108540 0x7f2b3819b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45060/0 (socket says 172.21.15.70:45060) 2024-04-01T01:24:20.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.165+0000 7f2b36ffd700 1 -- 172.21.15.70:0/3690924138 learned_addr learned my addr 172.21.15.70:0/3690924138 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:20.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.165+0000 7f2b36ffd700 1 -- 172.21.15.70:0/3690924138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b3819bd70 con 0x7f2b38108540 2024-04-01T01:24:20.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.166+0000 7f2b36ffd700 1 --2- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b38108540 0x7f2b3819b830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2b28011a20 tx=0x7f2b28011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:20.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.166+0000 7f2b1ffff700 1 -- 172.21.15.70:0/3690924138 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b280039c0 con 0x7f2b38108540 2024-04-01T01:24:20.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.166+0000 7f2b1ffff700 1 -- 172.21.15.70:0/3690924138 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2b28003b60 con 0x7f2b38108540 2024-04-01T01:24:20.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.166+0000 7f2b3d563700 1 -- 172.21.15.70:0/3690924138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b3819c000 con 0x7f2b38108540 2024-04-01T01:24:20.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.166+0000 7f2b1ffff700 1 -- 172.21.15.70:0/3690924138 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2b2801a6b0 con 0x7f2b38108540 2024-04-01T01:24:20.060 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.166+0000 7f2b3d563700 1 -- 172.21.15.70:0/3690924138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b3819ecf0 con 0x7f2b38108540 2024-04-01T01:24:20.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.167+0000 7f2b1ffff700 1 -- 172.21.15.70:0/3690924138 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 9) ==== 46129+0+0 (secure 0 0 0) 0x7f2b2801aa10 con 0x7f2b38108540 2024-04-01T01:24:20.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.167+0000 7f2b1ffff700 1 --2- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f2b20038ec0 0x7f2b2003b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:20.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.167+0000 7f2b3d563700 1 -- 172.21.15.70:0/3690924138 --> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2b38063870 con 0x7f2b20038ec0 2024-04-01T01:24:20.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.167+0000 7f2b1ffff700 1 -- 172.21.15.70:0/3690924138 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2b2804e270 con 0x7f2b38108540 2024-04-01T01:24:20.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.167+0000 7f2b367fc700 1 -- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f2b20038ec0 msgr2=0x7f2b2003b380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2836712828 2024-04-01T01:24:20.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.167+0000 7f2b367fc700 1 --2- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f2b20038ec0 0x7f2b2003b380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-01T01:24:20.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.368+0000 7f2b367fc700 1 -- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f2b20038ec0 msgr2=0x7f2b2003b380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2836712828 2024-04-01T01:24:20.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.368+0000 7f2b367fc700 1 --2- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f2b20038ec0 0x7f2b2003b380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-01T01:24:20.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.769+0000 7f2b367fc700 1 -- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f2b20038ec0 msgr2=0x7f2b2003b380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2836712828 2024-04-01T01:24:20.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:14.769+0000 7f2b367fc700 1 --2- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f2b20038ec0 0x7f2b2003b380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-01T01:24:20.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:15.570+0000 7f2b367fc700 1 -- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f2b20038ec0 msgr2=0x7f2b2003b380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2836712828 2024-04-01T01:24:20.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:15.570+0000 7f2b367fc700 1 --2- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f2b20038ec0 0x7f2b2003b380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-01T01:24:20.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:17.171+0000 7f2b367fc700 1 -- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f2b20038ec0 msgr2=0x7f2b2003b380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.70:6800/2836712828 2024-04-01T01:24:20.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:17.171+0000 7f2b367fc700 1 --2- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f2b20038ec0 0x7f2b2003b380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-01T01:24:20.061 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:18.478+0000 7f2b1ffff700 1 -- 172.21.15.70:0/3690924138 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mgrmap(e 10) ==== 45896+0+0 (secure 0 0 0) 0x7f2b2802dc90 con 0x7f2b38108540 2024-04-01T01:24:20.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:18.478+0000 7f2b1ffff700 1 -- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f2b20038ec0 msgr2=0x7f2b2003b380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:24:20.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:18.478+0000 7f2b1ffff700 1 --2- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f2b20038ec0 0x7f2b2003b380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:20.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.480+0000 7f2b1ffff700 1 -- 172.21.15.70:0/3690924138 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 11) ==== 46023+0+0 (secure 0 0 0) 0x7f2b2804f890 con 0x7f2b38108540 2024-04-01T01:24:20.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.480+0000 7f2b1ffff700 1 --2- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2b2003c790 0x7f2b2003eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:20.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.480+0000 7f2b1ffff700 1 -- 172.21.15.70:0/3690924138 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2b28001d30 con 0x7f2b2003c790 2024-04-01T01:24:20.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.481+0000 7f2b367fc700 1 --2- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2b2003c790 0x7f2b2003eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:20.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.481+0000 7f2b367fc700 1 --2- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2b2003c790 0x7f2b2003eb80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c003a10 tx=0x7f2b2c0092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:20.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.482+0000 7f2b1ffff700 1 -- 172.21.15.70:0/3690924138 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f2b28001d30 con 0x7f2b2003c790 2024-04-01T01:24:20.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.489+0000 7f2b3d563700 1 -- 172.21.15.70:0/3690924138 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f2b38103030 con 0x7f2b2003c790 2024-04-01T01:24:20.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.489+0000 7f2b1ffff700 1 -- 172.21.15.70:0/3690924138 <== mgr.14150 v2:172.21.15.70:6800/2070166741 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f2b38103030 con 0x7f2b2003c790 2024-04-01T01:24:20.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.489+0000 7f2b3d563700 1 -- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2b2003c790 msgr2=0x7f2b2003eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:20.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.489+0000 7f2b3d563700 1 --2- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2b2003c790 0x7f2b2003eb80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c003a10 tx=0x7f2b2c0092b0 comp rx=0 tx=0).stop 2024-04-01T01:24:20.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.489+0000 7f2b3d563700 1 -- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b38108540 msgr2=0x7f2b3819b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:20.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.489+0000 7f2b3d563700 1 --2- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b38108540 0x7f2b3819b830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2b28011a20 tx=0x7f2b28011b00 comp rx=0 tx=0).stop 2024-04-01T01:24:20.062 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.489+0000 7f2b3d563700 1 -- 172.21.15.70:0/3690924138 shutdown_connections 2024-04-01T01:24:20.063 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.489+0000 7f2b3d563700 1 --2- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2b2003c790 0x7f2b2003eb80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:20.063 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.489+0000 7f2b3d563700 1 --2- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b38108540 0x7f2b3819b830 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:20.063 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.489+0000 7f2b3d563700 1 --2- 172.21.15.70:0/3690924138 >> [v2:172.21.15.70:6800/2836712828,v1:172.21.15.70:6801/2836712828] conn(0x7f2b20038ec0 0x7f2b2003b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:20.063 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.489+0000 7f2b3d563700 1 -- 172.21.15.70:0/3690924138 >> 172.21.15.70:0/3690924138 conn(0x7f2b38101ed0 msgr2=0x7f2b38102920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:20.063 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.489+0000 7f2b3d563700 1 -- 172.21.15.70:0/3690924138 shutdown_connections 2024-04-01T01:24:20.063 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:19.489+0000 7f2b3d563700 1 -- 172.21.15.70:0/3690924138 wait complete. 2024-04-01T01:24:20.063 INFO:teuthology.orchestra.run.smithi070.stdout:mgr epoch 9 is available 2024-04-01T01:24:20.063 INFO:teuthology.orchestra.run.smithi070.stdout:Generating a dashboard self-signed certificate... 2024-04-01T01:24:20.804 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:20 smithi070 ceph-mon[32179]: [01/Apr/2024:01:24:18] ENGINE Bus STARTING 2024-04-01T01:24:20.804 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:20 smithi070 ceph-mon[32179]: [01/Apr/2024:01:24:19] ENGINE Serving on https://172.21.15.70:7150 2024-04-01T01:24:20.804 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:20 smithi070 ceph-mon[32179]: [01/Apr/2024:01:24:19] ENGINE Serving on http://172.21.15.70:8765 2024-04-01T01:24:20.804 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:20 smithi070 ceph-mon[32179]: [01/Apr/2024:01:24:19] ENGINE Bus STARTED 2024-04-01T01:24:20.804 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:20 smithi070 ceph-mon[32179]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-01T01:24:20.804 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:20 smithi070 ceph-mon[32179]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-01T01:24:20.805 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:20 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:21.831 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-01T01:24:21.831 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.819+0000 7f2734c44700 1 Processor -- start 2024-04-01T01:24:21.831 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.819+0000 7f2734c44700 1 -- start start 2024-04-01T01:24:21.831 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.820+0000 7f2734c44700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2730108520 0x7f273010a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:21.831 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.820+0000 7f2734c44700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f273007be20 con 0x7f2730108520 2024-04-01T01:24:21.831 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.820+0000 7f272e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2730108520 0x7f273010a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:21.831 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.820+0000 7f272e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2730108520 0x7f273010a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56302/0 (socket says 172.21.15.70:56302) 2024-04-01T01:24:21.831 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.820+0000 7f272e59c700 1 -- 172.21.15.70:0/2721209199 learned_addr learned my addr 172.21.15.70:0/2721209199 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:21.831 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.820+0000 7f272e59c700 1 -- 172.21.15.70:0/2721209199 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f273010ae50 con 0x7f2730108520 2024-04-01T01:24:21.831 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.820+0000 7f272e59c700 1 --2- 172.21.15.70:0/2721209199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2730108520 0x7f273010a910 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f27200098a0 tx=0x7f2720009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b689430080f96812 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:21.832 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.820+0000 7f272d59a700 1 -- 172.21.15.70:0/2721209199 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2720004160 con 0x7f2730108520 2024-04-01T01:24:21.832 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.820+0000 7f272d59a700 1 -- 172.21.15.70:0/2721209199 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2720004300 con 0x7f2730108520 2024-04-01T01:24:21.832 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.820+0000 7f272d59a700 1 -- 172.21.15.70:0/2721209199 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2720011a50 con 0x7f2730108520 2024-04-01T01:24:21.832 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.821+0000 7f2734c44700 1 -- 172.21.15.70:0/2721209199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2730108520 msgr2=0x7f273010a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:21.832 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.821+0000 7f2734c44700 1 --2- 172.21.15.70:0/2721209199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2730108520 0x7f273010a910 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f27200098a0 tx=0x7f2720009bb0 comp rx=0 tx=0).stop 2024-04-01T01:24:21.832 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.821+0000 7f2734c44700 1 -- 172.21.15.70:0/2721209199 shutdown_connections 2024-04-01T01:24:21.832 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.821+0000 7f2734c44700 1 --2- 172.21.15.70:0/2721209199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2730108520 0x7f273010a910 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:21.832 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.821+0000 7f2734c44700 1 -- 172.21.15.70:0/2721209199 >> 172.21.15.70:0/2721209199 conn(0x7f2730101ed0 msgr2=0x7f2730104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:21.832 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.821+0000 7f2734c44700 1 -- 172.21.15.70:0/2721209199 shutdown_connections 2024-04-01T01:24:21.832 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.821+0000 7f2734c44700 1 -- 172.21.15.70:0/2721209199 wait complete. 2024-04-01T01:24:21.832 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.822+0000 7f2734c44700 1 Processor -- start 2024-04-01T01:24:21.832 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.822+0000 7f2734c44700 1 -- start start 2024-04-01T01:24:21.832 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.823+0000 7f2734c44700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2730108520 0x7f273019b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:21.832 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.823+0000 7f2734c44700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f273010bdf0 con 0x7f2730108520 2024-04-01T01:24:21.832 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.823+0000 7f272e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2730108520 0x7f273019b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:21.833 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.823+0000 7f272e59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2730108520 0x7f273019b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56318/0 (socket says 172.21.15.70:56318) 2024-04-01T01:24:21.833 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.823+0000 7f272e59c700 1 -- 172.21.15.70:0/555821919 learned_addr learned my addr 172.21.15.70:0/555821919 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:21.833 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.823+0000 7f272e59c700 1 -- 172.21.15.70:0/555821919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f273019bda0 con 0x7f2730108520 2024-04-01T01:24:21.833 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.823+0000 7f272e59c700 1 --2- 172.21.15.70:0/555821919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2730108520 0x7f273019b860 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2720011a20 tx=0x7f2720011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:21.833 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.823+0000 7f271f7fe700 1 -- 172.21.15.70:0/555821919 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27200039c0 con 0x7f2730108520 2024-04-01T01:24:21.833 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.823+0000 7f271f7fe700 1 -- 172.21.15.70:0/555821919 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2720003b60 con 0x7f2730108520 2024-04-01T01:24:21.833 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.823+0000 7f2734c44700 1 -- 172.21.15.70:0/555821919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f273019c030 con 0x7f2730108520 2024-04-01T01:24:21.833 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.823+0000 7f271f7fe700 1 -- 172.21.15.70:0/555821919 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f272001a6b0 con 0x7f2730108520 2024-04-01T01:24:21.833 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.823+0000 7f2734c44700 1 -- 172.21.15.70:0/555821919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f273019ed20 con 0x7f2730108520 2024-04-01T01:24:21.833 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.824+0000 7f271f7fe700 1 -- 172.21.15.70:0/555821919 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 11) ==== 46023+0+0 (secure 0 0 0) 0x7f272001a980 con 0x7f2730108520 2024-04-01T01:24:21.833 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.824+0000 7f271f7fe700 1 --2- 172.21.15.70:0/555821919 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2718038e00 0x7f271803b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:21.833 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.824+0000 7f271f7fe700 1 -- 172.21.15.70:0/555821919 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f272004e530 con 0x7f2730108520 2024-04-01T01:24:21.833 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.824+0000 7f2734c44700 1 -- 172.21.15.70:0/555821919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2730063870 con 0x7f2730108520 2024-04-01T01:24:21.834 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.825+0000 7f272dd9b700 1 --2- 172.21.15.70:0/555821919 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2718038e00 0x7f271803b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:21.834 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.825+0000 7f272dd9b700 1 --2- 172.21.15.70:0/555821919 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2718038e00 0x7f271803b2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2724006fd0 tx=0x7f2724006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:21.834 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:20.830+0000 7f271f7fe700 1 -- 172.21.15.70:0/555821919 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f272002c510 con 0x7f2730108520 2024-04-01T01:24:21.834 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:21.052+0000 7f2734c44700 1 -- 172.21.15.70:0/555821919 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f2730102fc0 con 0x7f2718038e00 2024-04-01T01:24:21.834 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:21.233+0000 7f271f7fe700 1 -- 172.21.15.70:0/555821919 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f2730102fc0 con 0x7f2718038e00 2024-04-01T01:24:21.834 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:21.235+0000 7f2734c44700 1 -- 172.21.15.70:0/555821919 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2718038e00 msgr2=0x7f271803b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:21.834 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:21.235+0000 7f2734c44700 1 --2- 172.21.15.70:0/555821919 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2718038e00 0x7f271803b2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2724006fd0 tx=0x7f2724006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:21.834 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:21.235+0000 7f2734c44700 1 -- 172.21.15.70:0/555821919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2730108520 msgr2=0x7f273019b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:21.834 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:21.235+0000 7f2734c44700 1 --2- 172.21.15.70:0/555821919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2730108520 0x7f273019b860 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2720011a20 tx=0x7f2720011b00 comp rx=0 tx=0).stop 2024-04-01T01:24:21.834 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:21.235+0000 7f2734c44700 1 -- 172.21.15.70:0/555821919 shutdown_connections 2024-04-01T01:24:21.834 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:21.235+0000 7f2734c44700 1 --2- 172.21.15.70:0/555821919 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2718038e00 0x7f271803b2c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:21.834 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:21.235+0000 7f2734c44700 1 --2- 172.21.15.70:0/555821919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2730108520 0x7f273019b860 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:21.834 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:21.235+0000 7f2734c44700 1 -- 172.21.15.70:0/555821919 >> 172.21.15.70:0/555821919 conn(0x7f2730101ed0 msgr2=0x7f27301028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:21.835 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:21.236+0000 7f2734c44700 1 -- 172.21.15.70:0/555821919 shutdown_connections 2024-04-01T01:24:21.835 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:21.236+0000 7f2734c44700 1 -- 172.21.15.70:0/555821919 wait complete. 2024-04-01T01:24:21.835 INFO:teuthology.orchestra.run.smithi070.stdout:Creating initial admin user... 2024-04-01T01:24:22.451 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:22 smithi070 ceph-mon[32179]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:24:22.451 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:22.451 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:22.451 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:22 smithi070 ceph-mon[32179]: mgrmap e12: x(active, since 2s) 2024-04-01T01:24:22.451 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:23.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:23 smithi070 ceph-mon[32179]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:24:23.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:23 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:24.159 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$H0IyzH.wme6PXqaOrVSFoe2s6W7rm4szaZp07MPm6nZE82pccA.ne", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711934662, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-01T01:24:24.159 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.226+0000 7f66f3de7700 1 Processor -- start 2024-04-01T01:24:24.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.226+0000 7f66f3de7700 1 -- start start 2024-04-01T01:24:24.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.227+0000 7f66f3de7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66ec105d30 0x7f66ec106150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:24.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.227+0000 7f66f3de7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f66ec106720 con 0x7f66ec105d30 2024-04-01T01:24:24.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.227+0000 7f66f1b83700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66ec105d30 0x7f66ec106150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:24.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.227+0000 7f66f1b83700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66ec105d30 0x7f66ec106150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56326/0 (socket says 172.21.15.70:56326) 2024-04-01T01:24:24.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.227+0000 7f66f1b83700 1 -- 172.21.15.70:0/2505490446 learned_addr learned my addr 172.21.15.70:0/2505490446 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:24.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.227+0000 7f66f1b83700 1 -- 172.21.15.70:0/2505490446 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ec1068a0 con 0x7f66ec105d30 2024-04-01T01:24:24.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.227+0000 7f66f1b83700 1 --2- 172.21.15.70:0/2505490446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66ec105d30 0x7f66ec106150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f66e4009e80 tx=0x7f66e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef5417fb36a32e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:24.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.227+0000 7f66f0b81700 1 -- 172.21.15.70:0/2505490446 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66e4017070 con 0x7f66ec105d30 2024-04-01T01:24:24.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.227+0000 7f66f0b81700 1 -- 172.21.15.70:0/2505490446 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66e400b8c0 con 0x7f66ec105d30 2024-04-01T01:24:24.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.227+0000 7f66f0b81700 1 -- 172.21.15.70:0/2505490446 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66e4004490 con 0x7f66ec105d30 2024-04-01T01:24:24.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.228+0000 7f66f3de7700 1 -- 172.21.15.70:0/2505490446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66ec105d30 msgr2=0x7f66ec106150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:24.160 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.228+0000 7f66f3de7700 1 --2- 172.21.15.70:0/2505490446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66ec105d30 0x7f66ec106150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f66e4009e80 tx=0x7f66e400b330 comp rx=0 tx=0).stop 2024-04-01T01:24:24.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.228+0000 7f66f3de7700 1 -- 172.21.15.70:0/2505490446 shutdown_connections 2024-04-01T01:24:24.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.228+0000 7f66f3de7700 1 --2- 172.21.15.70:0/2505490446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66ec105d30 0x7f66ec106150 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:24.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.228+0000 7f66f3de7700 1 -- 172.21.15.70:0/2505490446 >> 172.21.15.70:0/2505490446 conn(0x7f66ec101650 msgr2=0x7f66ec103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:24.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.228+0000 7f66f3de7700 1 -- 172.21.15.70:0/2505490446 shutdown_connections 2024-04-01T01:24:24.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.228+0000 7f66f3de7700 1 -- 172.21.15.70:0/2505490446 wait complete. 2024-04-01T01:24:24.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.229+0000 7f66f3de7700 1 Processor -- start 2024-04-01T01:24:24.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.229+0000 7f66f3de7700 1 -- start start 2024-04-01T01:24:24.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.229+0000 7f66f3de7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66ec105d30 0x7f66ec108630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:24.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.229+0000 7f66f3de7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f66ec109c40 con 0x7f66ec105d30 2024-04-01T01:24:24.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.229+0000 7f66f1b83700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66ec105d30 0x7f66ec108630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:24.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.229+0000 7f66f1b83700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66ec105d30 0x7f66ec108630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56336/0 (socket says 172.21.15.70:56336) 2024-04-01T01:24:24.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.229+0000 7f66f1b83700 1 -- 172.21.15.70:0/3091499290 learned_addr learned my addr 172.21.15.70:0/3091499290 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:24.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.230+0000 7f66f1b83700 1 -- 172.21.15.70:0/3091499290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ec108b70 con 0x7f66ec105d30 2024-04-01T01:24:24.161 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.230+0000 7f66f1b83700 1 --2- 172.21.15.70:0/3091499290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66ec105d30 0x7f66ec108630 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f66e4004220 tx=0x7f66e4003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:24.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.232+0000 7f66e2ffd700 1 -- 172.21.15.70:0/3091499290 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66e4020020 con 0x7f66ec105d30 2024-04-01T01:24:24.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.232+0000 7f66e2ffd700 1 -- 172.21.15.70:0/3091499290 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66e400bcb0 con 0x7f66ec105d30 2024-04-01T01:24:24.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.232+0000 7f66e2ffd700 1 -- 172.21.15.70:0/3091499290 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f66e4017040 con 0x7f66ec105d30 2024-04-01T01:24:24.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.232+0000 7f66f3de7700 1 -- 172.21.15.70:0/3091499290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66ec106d60 con 0x7f66ec105d30 2024-04-01T01:24:24.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.232+0000 7f66f3de7700 1 -- 172.21.15.70:0/3091499290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66ec1072a0 con 0x7f66ec105d30 2024-04-01T01:24:24.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.233+0000 7f66e2ffd700 1 -- 172.21.15.70:0/3091499290 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 12) ==== 46129+0+0 (secure 0 0 0) 0x7f66e4023020 con 0x7f66ec105d30 2024-04-01T01:24:24.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.233+0000 7f66f3de7700 1 -- 172.21.15.70:0/3091499290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66ec063870 con 0x7f66ec105d30 2024-04-01T01:24:24.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.233+0000 7f66e2ffd700 1 --2- 172.21.15.70:0/3091499290 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66d8038e60 0x7f66d803b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:24.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.233+0000 7f66e2ffd700 1 -- 172.21.15.70:0/3091499290 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f66e400f050 con 0x7f66ec105d30 2024-04-01T01:24:24.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.234+0000 7f66f1382700 1 --2- 172.21.15.70:0/3091499290 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66d8038e60 0x7f66d803b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:24.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.234+0000 7f66f1382700 1 --2- 172.21.15.70:0/3091499290 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66d8038e60 0x7f66d803b320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f66dc006fd0 tx=0x7f66dc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:24.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.238+0000 7f66e2ffd700 1 -- 172.21.15.70:0/3091499290 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f66e402a020 con 0x7f66ec105d30 2024-04-01T01:24:24.162 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.450+0000 7f66f3de7700 1 -- 172.21.15.70:0/3091499290 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f66ec107520 con 0x7f66d8038e60 2024-04-01T01:24:24.163 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.678+0000 7f66e2ffd700 1 -- 172.21.15.70:0/3091499290 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f66ec107520 con 0x7f66d8038e60 2024-04-01T01:24:24.163 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.680+0000 7f66f3de7700 1 -- 172.21.15.70:0/3091499290 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66d8038e60 msgr2=0x7f66d803b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:24.163 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.680+0000 7f66f3de7700 1 --2- 172.21.15.70:0/3091499290 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66d8038e60 0x7f66d803b320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f66dc006fd0 tx=0x7f66dc006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:24.163 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.680+0000 7f66f3de7700 1 -- 172.21.15.70:0/3091499290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66ec105d30 msgr2=0x7f66ec108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:24.163 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.680+0000 7f66f3de7700 1 --2- 172.21.15.70:0/3091499290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66ec105d30 0x7f66ec108630 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f66e4004220 tx=0x7f66e4003d00 comp rx=0 tx=0).stop 2024-04-01T01:24:24.163 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.680+0000 7f66f3de7700 1 -- 172.21.15.70:0/3091499290 shutdown_connections 2024-04-01T01:24:24.163 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.680+0000 7f66f3de7700 1 --2- 172.21.15.70:0/3091499290 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66d8038e60 0x7f66d803b320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:24.163 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.680+0000 7f66f3de7700 1 --2- 172.21.15.70:0/3091499290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66ec105d30 0x7f66ec108630 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:24.163 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.680+0000 7f66f3de7700 1 -- 172.21.15.70:0/3091499290 >> 172.21.15.70:0/3091499290 conn(0x7f66ec101650 msgr2=0x7f66ec1021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:24.163 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.680+0000 7f66f3de7700 1 -- 172.21.15.70:0/3091499290 shutdown_connections 2024-04-01T01:24:24.163 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:22.680+0000 7f66f3de7700 1 -- 172.21.15.70:0/3091499290 wait complete. 2024-04-01T01:24:24.163 INFO:teuthology.orchestra.run.smithi070.stdout:Fetching dashboard port number... 2024-04-01T01:24:25.406 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stdout 8443 2024-04-01T01:24:25.406 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.591+0000 7f73869a9700 1 Processor -- start 2024-04-01T01:24:25.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.591+0000 7f73869a9700 1 -- start start 2024-04-01T01:24:25.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.591+0000 7f73869a9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7380105d20 0x7f7380106140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:25.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.591+0000 7f73869a9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7380106710 con 0x7f7380105d20 2024-04-01T01:24:25.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.592+0000 7f737ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7380105d20 0x7f7380106140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:25.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.592+0000 7f737ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7380105d20 0x7f7380106140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56340/0 (socket says 172.21.15.70:56340) 2024-04-01T01:24:25.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.592+0000 7f737ffff700 1 -- 172.21.15.70:0/1300992241 learned_addr learned my addr 172.21.15.70:0/1300992241 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:25.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.592+0000 7f737ffff700 1 -- 172.21.15.70:0/1300992241 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7380106890 con 0x7f7380105d20 2024-04-01T01:24:25.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.592+0000 7f737ffff700 1 --2- 172.21.15.70:0/1300992241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7380105d20 0x7f7380106140 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f73700098a0 tx=0x7f7370009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc3bc8df814f31f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:25.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.592+0000 7f737effd700 1 -- 172.21.15.70:0/1300992241 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7370004160 con 0x7f7380105d20 2024-04-01T01:24:25.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.592+0000 7f737effd700 1 -- 172.21.15.70:0/1300992241 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7370004300 con 0x7f7380105d20 2024-04-01T01:24:25.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.592+0000 7f737effd700 1 -- 172.21.15.70:0/1300992241 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7370011a50 con 0x7f7380105d20 2024-04-01T01:24:25.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.593+0000 7f73869a9700 1 -- 172.21.15.70:0/1300992241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7380105d20 msgr2=0x7f7380106140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:25.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.593+0000 7f73869a9700 1 --2- 172.21.15.70:0/1300992241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7380105d20 0x7f7380106140 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f73700098a0 tx=0x7f7370009bb0 comp rx=0 tx=0).stop 2024-04-01T01:24:25.407 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.593+0000 7f73869a9700 1 -- 172.21.15.70:0/1300992241 shutdown_connections 2024-04-01T01:24:25.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.593+0000 7f73869a9700 1 --2- 172.21.15.70:0/1300992241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7380105d20 0x7f7380106140 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:25.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.593+0000 7f73869a9700 1 -- 172.21.15.70:0/1300992241 >> 172.21.15.70:0/1300992241 conn(0x7f7380101680 msgr2=0x7f7380103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:25.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.593+0000 7f73869a9700 1 -- 172.21.15.70:0/1300992241 shutdown_connections 2024-04-01T01:24:25.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.593+0000 7f73869a9700 1 -- 172.21.15.70:0/1300992241 wait complete. 2024-04-01T01:24:25.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.593+0000 7f73869a9700 1 Processor -- start 2024-04-01T01:24:25.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.593+0000 7f73869a9700 1 -- start start 2024-04-01T01:24:25.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.594+0000 7f73869a9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7380105d20 0x7f738019b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:25.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.594+0000 7f73869a9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f738010bfd0 con 0x7f7380105d20 2024-04-01T01:24:25.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.594+0000 7f737ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7380105d20 0x7f738019b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:25.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.594+0000 7f737ffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7380105d20 0x7f738019b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56350/0 (socket says 172.21.15.70:56350) 2024-04-01T01:24:25.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.594+0000 7f737ffff700 1 -- 172.21.15.70:0/262819305 learned_addr learned my addr 172.21.15.70:0/262819305 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:25.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.594+0000 7f737ffff700 1 -- 172.21.15.70:0/262819305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f738019bde0 con 0x7f7380105d20 2024-04-01T01:24:25.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.594+0000 7f737ffff700 1 --2- 172.21.15.70:0/262819305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7380105d20 0x7f738019b8a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7370011a20 tx=0x7f7370011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:25.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.594+0000 7f737d7fa700 1 -- 172.21.15.70:0/262819305 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7370003af0 con 0x7f7380105d20 2024-04-01T01:24:25.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.594+0000 7f737d7fa700 1 -- 172.21.15.70:0/262819305 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f737001a8e0 con 0x7f7380105d20 2024-04-01T01:24:25.408 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.594+0000 7f73869a9700 1 -- 172.21.15.70:0/262819305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f738019c070 con 0x7f7380105d20 2024-04-01T01:24:25.409 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.594+0000 7f737d7fa700 1 -- 172.21.15.70:0/262819305 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7370024860 con 0x7f7380105d20 2024-04-01T01:24:25.409 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.595+0000 7f73869a9700 1 -- 172.21.15.70:0/262819305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f738019ed60 con 0x7f7380105d20 2024-04-01T01:24:25.409 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.595+0000 7f737d7fa700 1 -- 172.21.15.70:0/262819305 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f7370022020 con 0x7f7380105d20 2024-04-01T01:24:25.409 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.595+0000 7f73869a9700 1 -- 172.21.15.70:0/262819305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7380063870 con 0x7f7380105d20 2024-04-01T01:24:25.409 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.595+0000 7f737d7fa700 1 --2- 172.21.15.70:0/262819305 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7368038fb0 0x7f736803b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:25.409 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.596+0000 7f737d7fa700 1 -- 172.21.15.70:0/262819305 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f737004d240 con 0x7f7380105d20 2024-04-01T01:24:25.409 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.596+0000 7f737f7fe700 1 --2- 172.21.15.70:0/262819305 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7368038fb0 0x7f736803b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:25.409 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.596+0000 7f737f7fe700 1 --2- 172.21.15.70:0/262819305 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7368038fb0 0x7f736803b470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7374006fd0 tx=0x7f7374006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:25.409 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.600+0000 7f737d7fa700 1 -- 172.21.15.70:0/262819305 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f737000b3e0 con 0x7f7380105d20 2024-04-01T01:24:25.409 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.792+0000 7f73869a9700 1 -- 172.21.15.70:0/262819305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f738019f310 con 0x7f7380105d20 2024-04-01T01:24:25.409 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.792+0000 7f737d7fa700 1 -- 172.21.15.70:0/262819305 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f7370016df0 con 0x7f7380105d20 2024-04-01T01:24:25.409 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.794+0000 7f73869a9700 1 -- 172.21.15.70:0/262819305 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7368038fb0 msgr2=0x7f736803b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:25.409 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.794+0000 7f73869a9700 1 --2- 172.21.15.70:0/262819305 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7368038fb0 0x7f736803b470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7374006fd0 tx=0x7f7374006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:25.409 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.794+0000 7f73869a9700 1 -- 172.21.15.70:0/262819305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7380105d20 msgr2=0x7f738019b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:25.409 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.794+0000 7f73869a9700 1 --2- 172.21.15.70:0/262819305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7380105d20 0x7f738019b8a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7370011a20 tx=0x7f7370011b00 comp rx=0 tx=0).stop 2024-04-01T01:24:25.410 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.795+0000 7f73869a9700 1 -- 172.21.15.70:0/262819305 shutdown_connections 2024-04-01T01:24:25.410 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.795+0000 7f73869a9700 1 --2- 172.21.15.70:0/262819305 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7368038fb0 0x7f736803b470 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:25.410 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.795+0000 7f73869a9700 1 --2- 172.21.15.70:0/262819305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7380105d20 0x7f738019b8a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:25.410 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.795+0000 7f73869a9700 1 -- 172.21.15.70:0/262819305 >> 172.21.15.70:0/262819305 conn(0x7f7380101680 msgr2=0x7f7380102080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:25.410 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.795+0000 7f73869a9700 1 -- 172.21.15.70:0/262819305 shutdown_connections 2024-04-01T01:24:25.410 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:24.795+0000 7f73869a9700 1 -- 172.21.15.70:0/262819305 wait complete. 2024-04-01T01:24:25.414 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-01T01:24:25.414 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout disabled 2024-04-01T01:24:25.420 INFO:teuthology.orchestra.run.smithi070.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-01T01:24:25.421 INFO:teuthology.orchestra.run.smithi070.stdout:systemctl: stdout inactive 2024-04-01T01:24:25.421 INFO:teuthology.orchestra.run.smithi070.stdout:firewalld.service is not enabled 2024-04-01T01:24:25.421 INFO:teuthology.orchestra.run.smithi070.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-01T01:24:25.422 INFO:teuthology.orchestra.run.smithi070.stdout:Ceph Dashboard is now available at: 2024-04-01T01:24:25.422 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:24:25.422 INFO:teuthology.orchestra.run.smithi070.stdout: URL: https://smithi070.front.sepia.ceph.com:8443/ 2024-04-01T01:24:25.422 INFO:teuthology.orchestra.run.smithi070.stdout: User: admin 2024-04-01T01:24:25.422 INFO:teuthology.orchestra.run.smithi070.stdout: Password: xk1yq6novn 2024-04-01T01:24:25.422 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:24:25.423 INFO:teuthology.orchestra.run.smithi070.stdout:Saving cluster configuration to /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config directory 2024-04-01T01:24:25.423 INFO:teuthology.orchestra.run.smithi070.stdout:Enabling autotune for osd_memory_target 2024-04-01T01:24:25.718 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:25 smithi070 ceph-mon[32179]: mgrmap e13: x(active, since 6s) 2024-04-01T01:24:25.719 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:25 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/262819305' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-01T01:24:26.552 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.819+0000 7f248722f700 1 Processor -- start 2024-04-01T01:24:26.552 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.819+0000 7f248722f700 1 -- start start 2024-04-01T01:24:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.819+0000 7f248722f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480107a00 0x7f2480107e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.819+0000 7f248722f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f24801083f0 con 0x7f2480107a00 2024-04-01T01:24:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.820+0000 7f2484fcb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480107a00 0x7f2480107e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.820+0000 7f2484fcb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480107a00 0x7f2480107e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56354/0 (socket says 172.21.15.70:56354) 2024-04-01T01:24:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.820+0000 7f2484fcb700 1 -- 172.21.15.70:0/555323213 learned_addr learned my addr 172.21.15.70:0/555323213 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.820+0000 7f2484fcb700 1 -- 172.21.15.70:0/555323213 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2480108c00 con 0x7f2480107a00 2024-04-01T01:24:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.820+0000 7f2484fcb700 1 --2- 172.21.15.70:0/555323213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480107a00 0x7f2480107e20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f247c009e80 tx=0x7f247c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=322387cf2cc4e590 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.820+0000 7f24777fe700 1 -- 172.21.15.70:0/555323213 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f247c017070 con 0x7f2480107a00 2024-04-01T01:24:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.820+0000 7f24777fe700 1 -- 172.21.15.70:0/555323213 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f247c00b8c0 con 0x7f2480107a00 2024-04-01T01:24:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.820+0000 7f24777fe700 1 -- 172.21.15.70:0/555323213 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f247c004490 con 0x7f2480107a00 2024-04-01T01:24:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.821+0000 7f248722f700 1 -- 172.21.15.70:0/555323213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480107a00 msgr2=0x7f2480107e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:26.554 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.821+0000 7f248722f700 1 --2- 172.21.15.70:0/555323213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480107a00 0x7f2480107e20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f247c009e80 tx=0x7f247c00b330 comp rx=0 tx=0).stop 2024-04-01T01:24:26.554 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.821+0000 7f248722f700 1 -- 172.21.15.70:0/555323213 shutdown_connections 2024-04-01T01:24:26.554 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.821+0000 7f248722f700 1 --2- 172.21.15.70:0/555323213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480107a00 0x7f2480107e20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:26.554 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.821+0000 7f248722f700 1 -- 172.21.15.70:0/555323213 >> 172.21.15.70:0/555323213 conn(0x7f2480102fa0 msgr2=0x7f24801053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:26.554 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.821+0000 7f248722f700 1 -- 172.21.15.70:0/555323213 shutdown_connections 2024-04-01T01:24:26.554 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.821+0000 7f248722f700 1 -- 172.21.15.70:0/555323213 wait complete. 2024-04-01T01:24:26.554 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.822+0000 7f248722f700 1 Processor -- start 2024-04-01T01:24:26.554 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.822+0000 7f248722f700 1 -- start start 2024-04-01T01:24:26.554 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.822+0000 7f248722f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480107a00 0x7f248019b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:26.554 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.822+0000 7f248722f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f248010bfb0 con 0x7f2480107a00 2024-04-01T01:24:26.554 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.822+0000 7f2484fcb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480107a00 0x7f248019b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:26.554 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.822+0000 7f2484fcb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480107a00 0x7f248019b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56360/0 (socket says 172.21.15.70:56360) 2024-04-01T01:24:26.554 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.822+0000 7f2484fcb700 1 -- 172.21.15.70:0/1032506202 learned_addr learned my addr 172.21.15.70:0/1032506202 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:26.555 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.822+0000 7f2484fcb700 1 -- 172.21.15.70:0/1032506202 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f248019be60 con 0x7f2480107a00 2024-04-01T01:24:26.555 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.822+0000 7f2484fcb700 1 --2- 172.21.15.70:0/1032506202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480107a00 0x7f248019b920 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f247c003c50 tx=0x7f247c003d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:26.555 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.822+0000 7f2475ffb700 1 -- 172.21.15.70:0/1032506202 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f247c020020 con 0x7f2480107a00 2024-04-01T01:24:26.555 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.822+0000 7f2475ffb700 1 -- 172.21.15.70:0/1032506202 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f247c00bc60 con 0x7f2480107a00 2024-04-01T01:24:26.555 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.822+0000 7f248722f700 1 -- 172.21.15.70:0/1032506202 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f248019c0f0 con 0x7f2480107a00 2024-04-01T01:24:26.555 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.822+0000 7f2475ffb700 1 -- 172.21.15.70:0/1032506202 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f247c017040 con 0x7f2480107a00 2024-04-01T01:24:26.555 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.823+0000 7f248722f700 1 -- 172.21.15.70:0/1032506202 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f248019ede0 con 0x7f2480107a00 2024-04-01T01:24:26.555 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.823+0000 7f2475ffb700 1 -- 172.21.15.70:0/1032506202 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f247c023020 con 0x7f2480107a00 2024-04-01T01:24:26.555 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.824+0000 7f248722f700 1 -- 172.21.15.70:0/1032506202 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2480063870 con 0x7f2480107a00 2024-04-01T01:24:26.556 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.824+0000 7f2475ffb700 1 --2- 172.21.15.70:0/1032506202 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2468038f50 0x7f246803b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:26.556 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.824+0000 7f2477fff700 1 --2- 172.21.15.70:0/1032506202 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2468038f50 0x7f246803b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:26.556 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.824+0000 7f2475ffb700 1 -- 172.21.15.70:0/1032506202 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f247c00f050 con 0x7f2480107a00 2024-04-01T01:24:26.556 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.824+0000 7f2477fff700 1 --2- 172.21.15.70:0/1032506202 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2468038f50 0x7f246803b410 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2470006fd0 tx=0x7f2470006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:26.556 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:25.829+0000 7f2475ffb700 1 -- 172.21.15.70:0/1032506202 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f247c02a020 con 0x7f2480107a00 2024-04-01T01:24:26.556 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.023+0000 7f248722f700 1 -- 172.21.15.70:0/1032506202 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f248019efb0 con 0x7f2480107a00 2024-04-01T01:24:26.556 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.023+0000 7f2475ffb700 1 -- 172.21.15.70:0/1032506202 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f248019efb0 con 0x7f2480107a00 2024-04-01T01:24:26.556 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.025+0000 7f248722f700 1 -- 172.21.15.70:0/1032506202 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2468038f50 msgr2=0x7f246803b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:26.556 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.025+0000 7f248722f700 1 --2- 172.21.15.70:0/1032506202 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2468038f50 0x7f246803b410 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2470006fd0 tx=0x7f2470006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:26.556 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.025+0000 7f248722f700 1 -- 172.21.15.70:0/1032506202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480107a00 msgr2=0x7f248019b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:26.556 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.025+0000 7f248722f700 1 --2- 172.21.15.70:0/1032506202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480107a00 0x7f248019b920 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f247c003c50 tx=0x7f247c003d30 comp rx=0 tx=0).stop 2024-04-01T01:24:26.556 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.026+0000 7f248722f700 1 -- 172.21.15.70:0/1032506202 shutdown_connections 2024-04-01T01:24:26.557 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.026+0000 7f248722f700 1 --2- 172.21.15.70:0/1032506202 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2468038f50 0x7f246803b410 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:26.557 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.026+0000 7f248722f700 1 --2- 172.21.15.70:0/1032506202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2480107a00 0x7f248019b920 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:26.557 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.026+0000 7f248722f700 1 -- 172.21.15.70:0/1032506202 >> 172.21.15.70:0/1032506202 conn(0x7f2480102fa0 msgr2=0x7f2480103a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:26.557 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.026+0000 7f248722f700 1 -- 172.21.15.70:0/1032506202 shutdown_connections 2024-04-01T01:24:26.557 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.026+0000 7f248722f700 1 -- 172.21.15.70:0/1032506202 wait complete. 2024-04-01T01:24:27.821 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.927+0000 7f08c9499700 1 Processor -- start 2024-04-01T01:24:27.821 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.927+0000 7f08c9499700 1 -- start start 2024-04-01T01:24:27.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.927+0000 7f08c9499700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08c4107ef0 0x7f08c4108310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:27.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.927+0000 7f08c9499700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08c41088e0 con 0x7f08c4107ef0 2024-04-01T01:24:27.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.927+0000 7f08c2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08c4107ef0 0x7f08c4108310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:27.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.927+0000 7f08c2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08c4107ef0 0x7f08c4108310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57400/0 (socket says 172.21.15.70:57400) 2024-04-01T01:24:27.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.927+0000 7f08c2ffd700 1 -- 172.21.15.70:0/1749962368 learned_addr learned my addr 172.21.15.70:0/1749962368 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:27.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.928+0000 7f08c2ffd700 1 -- 172.21.15.70:0/1749962368 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c4106b50 con 0x7f08c4107ef0 2024-04-01T01:24:27.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.928+0000 7f08c2ffd700 1 --2- 172.21.15.70:0/1749962368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08c4107ef0 0x7f08c4108310 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f08bc0098a0 tx=0x7f08bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b69aaf5ff0e142 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:27.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.928+0000 7f08c1ffb700 1 -- 172.21.15.70:0/1749962368 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08bc004160 con 0x7f08c4107ef0 2024-04-01T01:24:27.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.928+0000 7f08c1ffb700 1 -- 172.21.15.70:0/1749962368 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08bc004300 con 0x7f08c4107ef0 2024-04-01T01:24:27.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.928+0000 7f08c1ffb700 1 -- 172.21.15.70:0/1749962368 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08bc011a50 con 0x7f08c4107ef0 2024-04-01T01:24:27.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.928+0000 7f08c9499700 1 -- 172.21.15.70:0/1749962368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08c4107ef0 msgr2=0x7f08c4108310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:27.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.928+0000 7f08c9499700 1 --2- 172.21.15.70:0/1749962368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08c4107ef0 0x7f08c4108310 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f08bc0098a0 tx=0x7f08bc009bb0 comp rx=0 tx=0).stop 2024-04-01T01:24:27.822 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.928+0000 7f08c9499700 1 -- 172.21.15.70:0/1749962368 shutdown_connections 2024-04-01T01:24:27.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.928+0000 7f08c9499700 1 --2- 172.21.15.70:0/1749962368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08c4107ef0 0x7f08c4108310 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:27.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.928+0000 7f08c9499700 1 -- 172.21.15.70:0/1749962368 >> 172.21.15.70:0/1749962368 conn(0x7f08c4101620 msgr2=0x7f08c4103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:27.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.929+0000 7f08c9499700 1 -- 172.21.15.70:0/1749962368 shutdown_connections 2024-04-01T01:24:27.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.929+0000 7f08c9499700 1 -- 172.21.15.70:0/1749962368 wait complete. 2024-04-01T01:24:27.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.929+0000 7f08c9499700 1 Processor -- start 2024-04-01T01:24:27.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.929+0000 7f08c9499700 1 -- start start 2024-04-01T01:24:27.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.929+0000 7f08c9499700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08c4107ef0 0x7f08c419b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:27.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.929+0000 7f08c9499700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08c410af10 con 0x7f08c4107ef0 2024-04-01T01:24:27.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.930+0000 7f08c2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08c4107ef0 0x7f08c419b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:27.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.930+0000 7f08c2ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08c4107ef0 0x7f08c419b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57414/0 (socket says 172.21.15.70:57414) 2024-04-01T01:24:27.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.930+0000 7f08c2ffd700 1 -- 172.21.15.70:0/1315518536 learned_addr learned my addr 172.21.15.70:0/1315518536 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:27.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.930+0000 7f08c2ffd700 1 -- 172.21.15.70:0/1315518536 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c419bdf0 con 0x7f08c4107ef0 2024-04-01T01:24:27.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.930+0000 7f08c2ffd700 1 --2- 172.21.15.70:0/1315518536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08c4107ef0 0x7f08c419b8b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f08bc011a00 tx=0x7f08bc011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:27.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.930+0000 7f08bbfff700 1 -- 172.21.15.70:0/1315518536 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08bc003af0 con 0x7f08c4107ef0 2024-04-01T01:24:27.823 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.930+0000 7f08bbfff700 1 -- 172.21.15.70:0/1315518536 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08bc01a8e0 con 0x7f08c4107ef0 2024-04-01T01:24:27.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.930+0000 7f08c9499700 1 -- 172.21.15.70:0/1315518536 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08c419c080 con 0x7f08c4107ef0 2024-04-01T01:24:27.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.930+0000 7f08bbfff700 1 -- 172.21.15.70:0/1315518536 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08bc024840 con 0x7f08c4107ef0 2024-04-01T01:24:27.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.930+0000 7f08c9499700 1 -- 172.21.15.70:0/1315518536 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08c419ed20 con 0x7f08c4107ef0 2024-04-01T01:24:27.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.931+0000 7f08bbfff700 1 -- 172.21.15.70:0/1315518536 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f08bc022020 con 0x7f08c4107ef0 2024-04-01T01:24:27.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.931+0000 7f08c9499700 1 -- 172.21.15.70:0/1315518536 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08c4063870 con 0x7f08c4107ef0 2024-04-01T01:24:27.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.931+0000 7f08bbfff700 1 --2- 172.21.15.70:0/1315518536 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08b0038fb0 0x7f08b003b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:27.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.931+0000 7f08bbfff700 1 -- 172.21.15.70:0/1315518536 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f08bc04d290 con 0x7f08c4107ef0 2024-04-01T01:24:27.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.931+0000 7f08c27fc700 1 --2- 172.21.15.70:0/1315518536 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08b0038fb0 0x7f08b003b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:27.824 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.932+0000 7f08c27fc700 1 --2- 172.21.15.70:0/1315518536 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08b0038fb0 0x7f08b003b470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f08ac006fd0 tx=0x7f08ac006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:27.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:26.936+0000 7f08bbfff700 1 -- 172.21.15.70:0/1315518536 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f08bc021e00 con 0x7f08c4107ef0 2024-04-01T01:24:27.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:27.239+0000 7f08c9499700 1 -- 172.21.15.70:0/1315518536 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f08c402dd80 con 0x7f08c4107ef0 2024-04-01T01:24:27.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:27.242+0000 7f08bbfff700 1 -- 172.21.15.70:0/1315518536 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f08bc016df0 con 0x7f08c4107ef0 2024-04-01T01:24:27.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:27.244+0000 7f08c9499700 1 -- 172.21.15.70:0/1315518536 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08b0038fb0 msgr2=0x7f08b003b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:27.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:27.244+0000 7f08c9499700 1 --2- 172.21.15.70:0/1315518536 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08b0038fb0 0x7f08b003b470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f08ac006fd0 tx=0x7f08ac006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:27.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:27.244+0000 7f08c9499700 1 -- 172.21.15.70:0/1315518536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08c4107ef0 msgr2=0x7f08c419b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:27.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:27.244+0000 7f08c9499700 1 --2- 172.21.15.70:0/1315518536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08c4107ef0 0x7f08c419b8b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f08bc011a00 tx=0x7f08bc011ae0 comp rx=0 tx=0).stop 2024-04-01T01:24:27.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:27.244+0000 7f08c9499700 1 -- 172.21.15.70:0/1315518536 shutdown_connections 2024-04-01T01:24:27.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:27.244+0000 7f08c9499700 1 --2- 172.21.15.70:0/1315518536 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08b0038fb0 0x7f08b003b470 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:27.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:27.244+0000 7f08c9499700 1 --2- 172.21.15.70:0/1315518536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08c4107ef0 0x7f08c419b8b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:27.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:27.244+0000 7f08c9499700 1 -- 172.21.15.70:0/1315518536 >> 172.21.15.70:0/1315518536 conn(0x7f08c4101620 msgr2=0x7f08c4192470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:27.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:27.244+0000 7f08c9499700 1 -- 172.21.15.70:0/1315518536 shutdown_connections 2024-04-01T01:24:27.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr 2024-04-01T01:24:27.244+0000 7f08c9499700 1 -- 172.21.15.70:0/1315518536 wait complete. 2024-04-01T01:24:27.825 INFO:teuthology.orchestra.run.smithi070.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-01T01:24:27.825 INFO:teuthology.orchestra.run.smithi070.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-01T01:24:27.826 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:24:27.826 INFO:teuthology.orchestra.run.smithi070.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-01T01:24:27.826 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:24:27.826 INFO:teuthology.orchestra.run.smithi070.stdout:Or, if you are only running a single cluster on this host: 2024-04-01T01:24:27.826 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:24:27.826 INFO:teuthology.orchestra.run.smithi070.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-01T01:24:27.826 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:24:27.826 INFO:teuthology.orchestra.run.smithi070.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-01T01:24:27.826 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:24:27.826 INFO:teuthology.orchestra.run.smithi070.stdout: ceph telemetry on 2024-04-01T01:24:27.826 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:24:27.826 INFO:teuthology.orchestra.run.smithi070.stdout:For more information see: 2024-04-01T01:24:27.826 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:24:27.826 INFO:teuthology.orchestra.run.smithi070.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-01T01:24:27.826 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:24:27.826 INFO:teuthology.orchestra.run.smithi070.stdout:Bootstrap complete. 2024-04-01T01:24:27.862 INFO:tasks.cephadm:Fetching config... 2024-04-01T01:24:27.863 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:24:27.863 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-01T01:24:27.878 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-01T01:24:27.878 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:24:27.878 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-01T01:24:27.933 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-01T01:24:27.933 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:24:27.933 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/keyring of=/dev/stdout 2024-04-01T01:24:28.000 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-01T01:24:28.000 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:24:28.000 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-01T01:24:28.055 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-01T01:24:28.056 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKsoBv+8wsrN+7luwiiFWSrs2097Luy4eDj+sEwq3Z1IysB8JfZNGA9hXzi/SzScL0AMbOO0/AoFt4p0A+IOgYQGIauqC/sY1TS8Ta2Px1lzhSdsMSKcxxJ6CI4fPUaoQUpjyYlnEkdp8BgSEI7i+hVaB6PP4aoZAzAYS/gog39S9m3rBLA5MRPPVzhbAXce9QvWI4kKSmZdAluINGzNKR4dV+uaDBlg7D5Slu32ns8i09tVGtPLRXeOcTpLQfdSQFfxSHWpn34de0uVYKzIUX4BtHGL/oU4m70O8JcuMN6L6afF6/uw8y13hK1Z7Xg6WVrpr4SiNKPfbuqeyDar0HHmkWA54o4O9fb8URVvqUpjp8xNuehlE7SsWYD94uaBxyKiEetLmaO9GC9qiDIRuNrH/fBbYHAsiExzjyKhVi+CAe8EgoRj077jMm027MnwfDvT3ZLkLDQI2VJ6Qm3U/3AhYXyi52j4Z6HCjYIdbT+CxqJRyGK8Nutp2nj/3271s= ceph-36577394-efc6-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-01T01:24:28.133 INFO:teuthology.orchestra.run.smithi070.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKsoBv+8wsrN+7luwiiFWSrs2097Luy4eDj+sEwq3Z1IysB8JfZNGA9hXzi/SzScL0AMbOO0/AoFt4p0A+IOgYQGIauqC/sY1TS8Ta2Px1lzhSdsMSKcxxJ6CI4fPUaoQUpjyYlnEkdp8BgSEI7i+hVaB6PP4aoZAzAYS/gog39S9m3rBLA5MRPPVzhbAXce9QvWI4kKSmZdAluINGzNKR4dV+uaDBlg7D5Slu32ns8i09tVGtPLRXeOcTpLQfdSQFfxSHWpn34de0uVYKzIUX4BtHGL/oU4m70O8JcuMN6L6afF6/uw8y13hK1Z7Xg6WVrpr4SiNKPfbuqeyDar0HHmkWA54o4O9fb8URVvqUpjp8xNuehlE7SsWYD94uaBxyKiEetLmaO9GC9qiDIRuNrH/fBbYHAsiExzjyKhVi+CAe8EgoRj077jMm027MnwfDvT3ZLkLDQI2VJ6Qm3U/3AhYXyi52j4Z6HCjYIdbT+CxqJRyGK8Nutp2nj/3271s= ceph-36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:24:28.158 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKsoBv+8wsrN+7luwiiFWSrs2097Luy4eDj+sEwq3Z1IysB8JfZNGA9hXzi/SzScL0AMbOO0/AoFt4p0A+IOgYQGIauqC/sY1TS8Ta2Px1lzhSdsMSKcxxJ6CI4fPUaoQUpjyYlnEkdp8BgSEI7i+hVaB6PP4aoZAzAYS/gog39S9m3rBLA5MRPPVzhbAXce9QvWI4kKSmZdAluINGzNKR4dV+uaDBlg7D5Slu32ns8i09tVGtPLRXeOcTpLQfdSQFfxSHWpn34de0uVYKzIUX4BtHGL/oU4m70O8JcuMN6L6afF6/uw8y13hK1Z7Xg6WVrpr4SiNKPfbuqeyDar0HHmkWA54o4O9fb8URVvqUpjp8xNuehlE7SsWYD94uaBxyKiEetLmaO9GC9qiDIRuNrH/fBbYHAsiExzjyKhVi+CAe8EgoRj077jMm027MnwfDvT3ZLkLDQI2VJ6Qm3U/3AhYXyi52j4Z6HCjYIdbT+CxqJRyGK8Nutp2nj/3271s= ceph-36577394-efc6-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-01T01:24:28.204 INFO:teuthology.orchestra.run.smithi081.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKsoBv+8wsrN+7luwiiFWSrs2097Luy4eDj+sEwq3Z1IysB8JfZNGA9hXzi/SzScL0AMbOO0/AoFt4p0A+IOgYQGIauqC/sY1TS8Ta2Px1lzhSdsMSKcxxJ6CI4fPUaoQUpjyYlnEkdp8BgSEI7i+hVaB6PP4aoZAzAYS/gog39S9m3rBLA5MRPPVzhbAXce9QvWI4kKSmZdAluINGzNKR4dV+uaDBlg7D5Slu32ns8i09tVGtPLRXeOcTpLQfdSQFfxSHWpn34de0uVYKzIUX4BtHGL/oU4m70O8JcuMN6L6afF6/uw8y13hK1Z7Xg6WVrpr4SiNKPfbuqeyDar0HHmkWA54o4O9fb8URVvqUpjp8xNuehlE7SsWYD94uaBxyKiEetLmaO9GC9qiDIRuNrH/fBbYHAsiExzjyKhVi+CAe8EgoRj077jMm027MnwfDvT3ZLkLDQI2VJ6Qm3U/3AhYXyi52j4Z6HCjYIdbT+CxqJRyGK8Nutp2nj/3271s= ceph-36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:24:28.219 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKsoBv+8wsrN+7luwiiFWSrs2097Luy4eDj+sEwq3Z1IysB8JfZNGA9hXzi/SzScL0AMbOO0/AoFt4p0A+IOgYQGIauqC/sY1TS8Ta2Px1lzhSdsMSKcxxJ6CI4fPUaoQUpjyYlnEkdp8BgSEI7i+hVaB6PP4aoZAzAYS/gog39S9m3rBLA5MRPPVzhbAXce9QvWI4kKSmZdAluINGzNKR4dV+uaDBlg7D5Slu32ns8i09tVGtPLRXeOcTpLQfdSQFfxSHWpn34de0uVYKzIUX4BtHGL/oU4m70O8JcuMN6L6afF6/uw8y13hK1Z7Xg6WVrpr4SiNKPfbuqeyDar0HHmkWA54o4O9fb8URVvqUpjp8xNuehlE7SsWYD94uaBxyKiEetLmaO9GC9qiDIRuNrH/fBbYHAsiExzjyKhVi+CAe8EgoRj077jMm027MnwfDvT3ZLkLDQI2VJ6Qm3U/3AhYXyi52j4Z6HCjYIdbT+CxqJRyGK8Nutp2nj/3271s= ceph-36577394-efc6-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-01T01:24:28.265 INFO:teuthology.orchestra.run.smithi189.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKsoBv+8wsrN+7luwiiFWSrs2097Luy4eDj+sEwq3Z1IysB8JfZNGA9hXzi/SzScL0AMbOO0/AoFt4p0A+IOgYQGIauqC/sY1TS8Ta2Px1lzhSdsMSKcxxJ6CI4fPUaoQUpjyYlnEkdp8BgSEI7i+hVaB6PP4aoZAzAYS/gog39S9m3rBLA5MRPPVzhbAXce9QvWI4kKSmZdAluINGzNKR4dV+uaDBlg7D5Slu32ns8i09tVGtPLRXeOcTpLQfdSQFfxSHWpn34de0uVYKzIUX4BtHGL/oU4m70O8JcuMN6L6afF6/uw8y13hK1Z7Xg6WVrpr4SiNKPfbuqeyDar0HHmkWA54o4O9fb8URVvqUpjp8xNuehlE7SsWYD94uaBxyKiEetLmaO9GC9qiDIRuNrH/fBbYHAsiExzjyKhVi+CAe8EgoRj077jMm027MnwfDvT3ZLkLDQI2VJ6Qm3U/3AhYXyi52j4Z6HCjYIdbT+CxqJRyGK8Nutp2nj/3271s= ceph-36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:24:28.280 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-01T01:24:28.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:28 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1315518536' entity='client.admin' 2024-04-01T01:24:28.522 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:24:31.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.134+0000 7f52e5be7700 1 -- 172.21.15.70:0/2501224754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e0100b70 msgr2=0x7f52e0100f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:31.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.134+0000 7f52e5be7700 1 --2- 172.21.15.70:0/2501224754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e0100b70 0x7f52e0100f90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f52d0009910 tx=0x7f52d0009c20 comp rx=0 tx=0).stop 2024-04-01T01:24:31.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.134+0000 7f52e5be7700 1 -- 172.21.15.70:0/2501224754 shutdown_connections 2024-04-01T01:24:31.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.134+0000 7f52e5be7700 1 --2- 172.21.15.70:0/2501224754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e0100b70 0x7f52e0100f90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:31.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.134+0000 7f52e5be7700 1 -- 172.21.15.70:0/2501224754 >> 172.21.15.70:0/2501224754 conn(0x7f52e00fc110 msgr2=0x7f52e00fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:31.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.134+0000 7f52e5be7700 1 -- 172.21.15.70:0/2501224754 shutdown_connections 2024-04-01T01:24:31.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.134+0000 7f52e5be7700 1 -- 172.21.15.70:0/2501224754 wait complete. 2024-04-01T01:24:31.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.135+0000 7f52e5be7700 1 Processor -- start 2024-04-01T01:24:31.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.136+0000 7f52e5be7700 1 -- start start 2024-04-01T01:24:31.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.136+0000 7f52e5be7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e0100b70 0x7f52e0197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:31.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.136+0000 7f52e5be7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f52e0106ab0 con 0x7f52e0100b70 2024-04-01T01:24:31.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.136+0000 7f52df7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e0100b70 0x7f52e0197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:31.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.136+0000 7f52df7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e0100b70 0x7f52e0197310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57444/0 (socket says 172.21.15.70:57444) 2024-04-01T01:24:31.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.136+0000 7f52df7fe700 1 -- 172.21.15.70:0/1229025298 learned_addr learned my addr 172.21.15.70:0/1229025298 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:31.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.136+0000 7f52df7fe700 1 -- 172.21.15.70:0/1229025298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e0197850 con 0x7f52e0100b70 2024-04-01T01:24:31.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.136+0000 7f52df7fe700 1 --2- 172.21.15.70:0/1229025298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e0100b70 0x7f52e0197310 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f52d0005950 tx=0x7f52d0004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:31.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.137+0000 7f52dcff9700 1 -- 172.21.15.70:0/1229025298 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52d0026070 con 0x7f52e0100b70 2024-04-01T01:24:31.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.137+0000 7f52dcff9700 1 -- 172.21.15.70:0/1229025298 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f52d000bc00 con 0x7f52e0100b70 2024-04-01T01:24:31.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.137+0000 7f52e5be7700 1 -- 172.21.15.70:0/1229025298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52e0197ae0 con 0x7f52e0100b70 2024-04-01T01:24:31.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.137+0000 7f52dcff9700 1 -- 172.21.15.70:0/1229025298 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52d00039a0 con 0x7f52e0100b70 2024-04-01T01:24:31.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.137+0000 7f52e5be7700 1 -- 172.21.15.70:0/1229025298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52e019a780 con 0x7f52e0100b70 2024-04-01T01:24:31.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.138+0000 7f52dcff9700 1 -- 172.21.15.70:0/1229025298 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f52d0029070 con 0x7f52e0100b70 2024-04-01T01:24:31.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.138+0000 7f52e5be7700 1 -- 172.21.15.70:0/1229025298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52e0107230 con 0x7f52e0100b70 2024-04-01T01:24:31.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.138+0000 7f52dcff9700 1 --2- 172.21.15.70:0/1229025298 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f52c8038ea0 0x7f52c803b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:31.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.138+0000 7f52dcff9700 1 -- 172.21.15.70:0/1229025298 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f52d001e070 con 0x7f52e0100b70 2024-04-01T01:24:31.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.138+0000 7f52deffd700 1 --2- 172.21.15.70:0/1229025298 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f52c8038ea0 0x7f52c803b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:31.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.138+0000 7f52deffd700 1 --2- 172.21.15.70:0/1229025298 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f52c8038ea0 0x7f52c803b360 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f52d4006fd0 tx=0x7f52d4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:31.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.143+0000 7f52dcff9700 1 -- 172.21.15.70:0/1229025298 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f52d0013070 con 0x7f52e0100b70 2024-04-01T01:24:31.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.345+0000 7f52e5be7700 1 -- 172.21.15.70:0/1229025298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f52e019aba0 con 0x7f52e0100b70 2024-04-01T01:24:31.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.348+0000 7f52dcff9700 1 -- 172.21.15.70:0/1229025298 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f52d0030070 con 0x7f52e0100b70 2024-04-01T01:24:31.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.350+0000 7f52e5be7700 1 -- 172.21.15.70:0/1229025298 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f52c8038ea0 msgr2=0x7f52c803b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:31.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.350+0000 7f52e5be7700 1 --2- 172.21.15.70:0/1229025298 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f52c8038ea0 0x7f52c803b360 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f52d4006fd0 tx=0x7f52d4006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:31.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.350+0000 7f52e5be7700 1 -- 172.21.15.70:0/1229025298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e0100b70 msgr2=0x7f52e0197310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:31.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.350+0000 7f52e5be7700 1 --2- 172.21.15.70:0/1229025298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e0100b70 0x7f52e0197310 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f52d0005950 tx=0x7f52d0004fa0 comp rx=0 tx=0).stop 2024-04-01T01:24:31.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.350+0000 7f52e5be7700 1 -- 172.21.15.70:0/1229025298 shutdown_connections 2024-04-01T01:24:31.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.350+0000 7f52e5be7700 1 --2- 172.21.15.70:0/1229025298 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f52c8038ea0 0x7f52c803b360 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:31.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.350+0000 7f52e5be7700 1 --2- 172.21.15.70:0/1229025298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e0100b70 0x7f52e0197310 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:31.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.350+0000 7f52e5be7700 1 -- 172.21.15.70:0/1229025298 >> 172.21.15.70:0/1229025298 conn(0x7f52e00fc110 msgr2=0x7f52e0069630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:31.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.350+0000 7f52e5be7700 1 -- 172.21.15.70:0/1229025298 shutdown_connections 2024-04-01T01:24:31.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:31.350+0000 7f52e5be7700 1 -- 172.21.15.70:0/1229025298 wait complete. 2024-04-01T01:24:32.459 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-01T01:24:32.459 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-01T01:24:32.695 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:24:32.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:32 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1229025298' entity='client.admin' 2024-04-01T01:24:32.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:32.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:32.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi070", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:24:32.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:32.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:24:32.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:34.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.298+0000 7f7791c0f700 1 -- 172.21.15.70:0/696004158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f778c100b70 msgr2=0x7f778c100f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:34.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.298+0000 7f7791c0f700 1 --2- 172.21.15.70:0/696004158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f778c100b70 0x7f778c100f90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f777c009910 tx=0x7f777c009c20 comp rx=0 tx=0).stop 2024-04-01T01:24:34.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.298+0000 7f7791c0f700 1 -- 172.21.15.70:0/696004158 shutdown_connections 2024-04-01T01:24:34.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.298+0000 7f7791c0f700 1 --2- 172.21.15.70:0/696004158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f778c100b70 0x7f778c100f90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:34.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.298+0000 7f7791c0f700 1 -- 172.21.15.70:0/696004158 >> 172.21.15.70:0/696004158 conn(0x7f778c0fc110 msgr2=0x7f778c0fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:34.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.298+0000 7f7791c0f700 1 -- 172.21.15.70:0/696004158 shutdown_connections 2024-04-01T01:24:34.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.298+0000 7f7791c0f700 1 -- 172.21.15.70:0/696004158 wait complete. 2024-04-01T01:24:34.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.299+0000 7f7791c0f700 1 Processor -- start 2024-04-01T01:24:34.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.299+0000 7f7791c0f700 1 -- start start 2024-04-01T01:24:34.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.299+0000 7f7791c0f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f778c100b70 0x7f778c0fdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:34.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.299+0000 7f7791c0f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f778c106ab0 con 0x7f778c100b70 2024-04-01T01:24:34.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.299+0000 7f778b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f778c100b70 0x7f778c0fdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:34.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.299+0000 7f778b7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f778c100b70 0x7f778c0fdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57450/0 (socket says 172.21.15.70:57450) 2024-04-01T01:24:34.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.299+0000 7f778b7fe700 1 -- 172.21.15.70:0/3839393356 learned_addr learned my addr 172.21.15.70:0/3839393356 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.300+0000 7f778b7fe700 1 -- 172.21.15.70:0/3839393356 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f778c0fe510 con 0x7f778c100b70 2024-04-01T01:24:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.300+0000 7f778b7fe700 1 --2- 172.21.15.70:0/3839393356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f778c100b70 0x7f778c0fdfd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f777c005800 tx=0x7f777c009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.300+0000 7f7788ff9700 1 -- 172.21.15.70:0/3839393356 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f777c027070 con 0x7f778c100b70 2024-04-01T01:24:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.300+0000 7f7788ff9700 1 -- 172.21.15.70:0/3839393356 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f777c00be10 con 0x7f778c100b70 2024-04-01T01:24:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.300+0000 7f7791c0f700 1 -- 172.21.15.70:0/3839393356 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f778c0fc700 con 0x7f778c100b70 2024-04-01T01:24:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.300+0000 7f7788ff9700 1 -- 172.21.15.70:0/3839393356 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f777c00f460 con 0x7f778c100b70 2024-04-01T01:24:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.300+0000 7f7791c0f700 1 -- 172.21.15.70:0/3839393356 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f778c0fcc40 con 0x7f778c100b70 2024-04-01T01:24:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.301+0000 7f7788ff9700 1 -- 172.21.15.70:0/3839393356 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f777c02a070 con 0x7f778c100b70 2024-04-01T01:24:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.301+0000 7f7791c0f700 1 -- 172.21.15.70:0/3839393356 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f778c107230 con 0x7f778c100b70 2024-04-01T01:24:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.301+0000 7f7788ff9700 1 --2- 172.21.15.70:0/3839393356 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7774038ea0 0x7f777403b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.301+0000 7f7788ff9700 1 -- 172.21.15.70:0/3839393356 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f777c01e030 con 0x7f778c100b70 2024-04-01T01:24:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.301+0000 7f778affd700 1 --2- 172.21.15.70:0/3839393356 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7774038ea0 0x7f777403b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:34.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.301+0000 7f778affd700 1 --2- 172.21.15.70:0/3839393356 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7774038ea0 0x7f777403b360 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7780006fd0 tx=0x7f7780006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:34.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.306+0000 7f7788ff9700 1 -- 172.21.15.70:0/3839393356 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f777c013070 con 0x7f778c100b70 2024-04-01T01:24:34.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.505+0000 7f7791c0f700 1 -- 172.21.15.70:0/3839393356 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f778c1907e0 con 0x7f7774038ea0 2024-04-01T01:24:34.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.509+0000 7f7788ff9700 1 -- 172.21.15.70:0/3839393356 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f778c1907e0 con 0x7f7774038ea0 2024-04-01T01:24:34.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.511+0000 7f7791c0f700 1 -- 172.21.15.70:0/3839393356 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7774038ea0 msgr2=0x7f777403b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:34.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.511+0000 7f7791c0f700 1 --2- 172.21.15.70:0/3839393356 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7774038ea0 0x7f777403b360 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7780006fd0 tx=0x7f7780006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:34.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.511+0000 7f7791c0f700 1 -- 172.21.15.70:0/3839393356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f778c100b70 msgr2=0x7f778c0fdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:34.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.511+0000 7f7791c0f700 1 --2- 172.21.15.70:0/3839393356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f778c100b70 0x7f778c0fdfd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f777c005800 tx=0x7f777c009f60 comp rx=0 tx=0).stop 2024-04-01T01:24:34.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.511+0000 7f7791c0f700 1 -- 172.21.15.70:0/3839393356 shutdown_connections 2024-04-01T01:24:34.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.511+0000 7f7791c0f700 1 --2- 172.21.15.70:0/3839393356 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7774038ea0 0x7f777403b360 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:34.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.511+0000 7f7791c0f700 1 --2- 172.21.15.70:0/3839393356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f778c100b70 0x7f778c0fdfd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:34.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.511+0000 7f7791c0f700 1 -- 172.21.15.70:0/3839393356 >> 172.21.15.70:0/3839393356 conn(0x7f778c0fc110 msgr2=0x7f778c069b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:34.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.511+0000 7f7791c0f700 1 -- 172.21.15.70:0/3839393356 shutdown_connections 2024-04-01T01:24:34.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:34.511+0000 7f7791c0f700 1 -- 172.21.15.70:0/3839393356 wait complete. 2024-04-01T01:24:35.286 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi081 2024-04-01T01:24:35.286 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-01T01:24:35.286 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/etc/ceph/ceph.conf 2024-04-01T01:24:35.304 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-01T01:24:35.305 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-01T01:24:35.362 INFO:tasks.cephadm:Adding host smithi081 to orchestrator... 2024-04-01T01:24:35.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi081 2024-04-01T01:24:35.597 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:24:35.818 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:35 smithi070 ceph-mon[32179]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:24:35.819 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:35.819 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:24:35.819 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:24:35.819 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:24:35.819 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:35 smithi070 ceph-mon[32179]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-01T01:24:35.819 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:35.819 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:35.819 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:36.775 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:36 smithi070 ceph-mon[32179]: Updating smithi070:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-01T01:24:36.775 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:36 smithi070 ceph-mon[32179]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-01T01:24:36.775 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:36 smithi070 ceph-mon[32179]: Updating smithi070:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-01T01:24:37.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.001+0000 7f8c87143700 1 -- 172.21.15.70:0/925110857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c80101e70 msgr2=0x7f8c80102250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:37.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.001+0000 7f8c87143700 1 --2- 172.21.15.70:0/925110857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c80101e70 0x7f8c80102250 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8c74009910 tx=0x7f8c74009c20 comp rx=0 tx=0).stop 2024-04-01T01:24:37.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.002+0000 7f8c87143700 1 -- 172.21.15.70:0/925110857 shutdown_connections 2024-04-01T01:24:37.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.002+0000 7f8c87143700 1 --2- 172.21.15.70:0/925110857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c80101e70 0x7f8c80102250 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:37.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.002+0000 7f8c87143700 1 -- 172.21.15.70:0/925110857 >> 172.21.15.70:0/925110857 conn(0x7f8c800fd720 msgr2=0x7f8c800ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:37.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.002+0000 7f8c87143700 1 -- 172.21.15.70:0/925110857 shutdown_connections 2024-04-01T01:24:37.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.002+0000 7f8c87143700 1 -- 172.21.15.70:0/925110857 wait complete. 2024-04-01T01:24:37.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.002+0000 7f8c87143700 1 Processor -- start 2024-04-01T01:24:37.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.002+0000 7f8c87143700 1 -- start start 2024-04-01T01:24:37.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.002+0000 7f8c87143700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c80101e70 0x7f8c80197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:37.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.002+0000 7f8c87143700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8010c0f0 con 0x7f8c80101e70 2024-04-01T01:24:37.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.002+0000 7f8c84edf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c80101e70 0x7f8c80197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:37.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.002+0000 7f8c84edf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c80101e70 0x7f8c80197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58990/0 (socket says 172.21.15.70:58990) 2024-04-01T01:24:37.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.002+0000 7f8c84edf700 1 -- 172.21.15.70:0/3561381874 learned_addr learned my addr 172.21.15.70:0/3561381874 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:37.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.003+0000 7f8c84edf700 1 -- 172.21.15.70:0/3561381874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c80197b90 con 0x7f8c80101e70 2024-04-01T01:24:37.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.003+0000 7f8c84edf700 1 --2- 172.21.15.70:0/3561381874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c80101e70 0x7f8c80197650 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8c740056d0 tx=0x7f8c740057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:37.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.003+0000 7f8c7dffb700 1 -- 172.21.15.70:0/3561381874 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c74026070 con 0x7f8c80101e70 2024-04-01T01:24:37.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.003+0000 7f8c7dffb700 1 -- 172.21.15.70:0/3561381874 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c7400bba0 con 0x7f8c80101e70 2024-04-01T01:24:37.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.003+0000 7f8c7dffb700 1 -- 172.21.15.70:0/3561381874 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c740037f0 con 0x7f8c80101e70 2024-04-01T01:24:37.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.003+0000 7f8c87143700 1 -- 172.21.15.70:0/3561381874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c80197e80 con 0x7f8c80101e70 2024-04-01T01:24:37.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.003+0000 7f8c87143700 1 -- 172.21.15.70:0/3561381874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c8019bf20 con 0x7f8c80101e70 2024-04-01T01:24:37.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.004+0000 7f8c7dffb700 1 -- 172.21.15.70:0/3561381874 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f8c74029070 con 0x7f8c80101e70 2024-04-01T01:24:37.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.004+0000 7f8c87143700 1 -- 172.21.15.70:0/3561381874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c80103350 con 0x7f8c80101e70 2024-04-01T01:24:37.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.004+0000 7f8c7dffb700 1 --2- 172.21.15.70:0/3561381874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c70038f00 0x7f8c7003b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:37.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.004+0000 7f8c7dffb700 1 -- 172.21.15.70:0/3561381874 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8c7401e070 con 0x7f8c80101e70 2024-04-01T01:24:37.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.005+0000 7f8c7ffff700 1 --2- 172.21.15.70:0/3561381874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c70038f00 0x7f8c7003b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:37.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.005+0000 7f8c7ffff700 1 --2- 172.21.15.70:0/3561381874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c70038f00 0x7f8c7003b3c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c006fd0 tx=0x7f8c6c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:37.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.009+0000 7f8c7dffb700 1 -- 172.21.15.70:0/3561381874 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8c74013070 con 0x7f8c80101e70 2024-04-01T01:24:37.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:37.206+0000 7f8c87143700 1 -- 172.21.15.70:0/3561381874 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi081", "target": ["mon-mgr", ""]}) -- 0x7f8c801989b0 con 0x7f8c70038f00 2024-04-01T01:24:38.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:37 smithi070 ceph-mon[32179]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi081", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:24:39.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:39 smithi070 ceph-mon[32179]: Deploying cephadm binary to smithi081 2024-04-01T01:24:39.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:39 smithi070 ceph-mon[32179]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:24:41.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:41.571+0000 7f8c7dffb700 1 -- 172.21.15.70:0/3561381874 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f8c801989b0 con 0x7f8c70038f00 2024-04-01T01:24:41.572 INFO:teuthology.orchestra.run.smithi070.stdout:Added host 'smithi081' with addr '172.21.15.81' 2024-04-01T01:24:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:41.574+0000 7f8c87143700 1 -- 172.21.15.70:0/3561381874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c70038f00 msgr2=0x7f8c7003b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:41.574+0000 7f8c87143700 1 --2- 172.21.15.70:0/3561381874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c70038f00 0x7f8c7003b3c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c006fd0 tx=0x7f8c6c006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:41.574+0000 7f8c87143700 1 -- 172.21.15.70:0/3561381874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c80101e70 msgr2=0x7f8c80197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:41.574+0000 7f8c87143700 1 --2- 172.21.15.70:0/3561381874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c80101e70 0x7f8c80197650 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8c740056d0 tx=0x7f8c740057b0 comp rx=0 tx=0).stop 2024-04-01T01:24:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:41.574+0000 7f8c87143700 1 -- 172.21.15.70:0/3561381874 shutdown_connections 2024-04-01T01:24:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:41.574+0000 7f8c87143700 1 --2- 172.21.15.70:0/3561381874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c70038f00 0x7f8c7003b3c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:41.574+0000 7f8c87143700 1 --2- 172.21.15.70:0/3561381874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c80101e70 0x7f8c80197650 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:41.574+0000 7f8c87143700 1 -- 172.21.15.70:0/3561381874 >> 172.21.15.70:0/3561381874 conn(0x7f8c800fd720 msgr2=0x7f8c80103840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:41.575+0000 7f8c87143700 1 -- 172.21.15.70:0/3561381874 shutdown_connections 2024-04-01T01:24:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:41.575+0000 7f8c87143700 1 -- 172.21.15.70:0/3561381874 wait complete. 2024-04-01T01:24:41.893 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:41 smithi070 ceph-mon[32179]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:24:42.234 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-01T01:24:42.471 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:24:42.854 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:42 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:42.855 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:42 smithi070 ceph-mon[32179]: Added host smithi081 2024-04-01T01:24:42.855 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:42 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:24:42.855 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:42 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:43.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:43 smithi070 ceph-mon[32179]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:24:43.608 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:44.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.095+0000 7f2f8dc60700 1 -- 172.21.15.70:0/698590581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f88101e70 msgr2=0x7f2f88102250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.095+0000 7f2f8dc60700 1 --2- 172.21.15.70:0/698590581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f88101e70 0x7f2f88102250 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2f78009910 tx=0x7f2f78009c20 comp rx=0 tx=0).stop 2024-04-01T01:24:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.095+0000 7f2f8dc60700 1 -- 172.21.15.70:0/698590581 shutdown_connections 2024-04-01T01:24:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.095+0000 7f2f8dc60700 1 --2- 172.21.15.70:0/698590581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f88101e70 0x7f2f88102250 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.095+0000 7f2f8dc60700 1 -- 172.21.15.70:0/698590581 >> 172.21.15.70:0/698590581 conn(0x7f2f880fd720 msgr2=0x7f2f880ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.096+0000 7f2f8dc60700 1 -- 172.21.15.70:0/698590581 shutdown_connections 2024-04-01T01:24:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.096+0000 7f2f8dc60700 1 -- 172.21.15.70:0/698590581 wait complete. 2024-04-01T01:24:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.096+0000 7f2f8dc60700 1 Processor -- start 2024-04-01T01:24:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.096+0000 7f2f8dc60700 1 -- start start 2024-04-01T01:24:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.096+0000 7f2f8dc60700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f88101e70 0x7f2f88197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.096+0000 7f2f8dc60700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8810c0f0 con 0x7f2f88101e70 2024-04-01T01:24:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.097+0000 7f2f877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f88101e70 0x7f2f88197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:44.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.097+0000 7f2f877fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f88101e70 0x7f2f88197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59018/0 (socket says 172.21.15.70:59018) 2024-04-01T01:24:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.097+0000 7f2f877fe700 1 -- 172.21.15.70:0/2707748923 learned_addr learned my addr 172.21.15.70:0/2707748923 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.097+0000 7f2f877fe700 1 -- 172.21.15.70:0/2707748923 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f88197bb0 con 0x7f2f88101e70 2024-04-01T01:24:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.097+0000 7f2f877fe700 1 --2- 172.21.15.70:0/2707748923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f88101e70 0x7f2f88197670 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2f780056d0 tx=0x7f2f780057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.097+0000 7f2f84ff9700 1 -- 172.21.15.70:0/2707748923 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f78026070 con 0x7f2f88101e70 2024-04-01T01:24:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.097+0000 7f2f84ff9700 1 -- 172.21.15.70:0/2707748923 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2f7800bba0 con 0x7f2f88101e70 2024-04-01T01:24:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.097+0000 7f2f8dc60700 1 -- 172.21.15.70:0/2707748923 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f88197ea0 con 0x7f2f88101e70 2024-04-01T01:24:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.097+0000 7f2f84ff9700 1 -- 172.21.15.70:0/2707748923 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f780037f0 con 0x7f2f88101e70 2024-04-01T01:24:44.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.097+0000 7f2f8dc60700 1 -- 172.21.15.70:0/2707748923 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f8819bf40 con 0x7f2f88101e70 2024-04-01T01:24:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.098+0000 7f2f84ff9700 1 -- 172.21.15.70:0/2707748923 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f2f78029070 con 0x7f2f88101e70 2024-04-01T01:24:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.098+0000 7f2f8dc60700 1 -- 172.21.15.70:0/2707748923 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f88103350 con 0x7f2f88101e70 2024-04-01T01:24:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.098+0000 7f2f84ff9700 1 --2- 172.21.15.70:0/2707748923 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2f70038ea0 0x7f2f7003b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.098+0000 7f2f84ff9700 1 -- 172.21.15.70:0/2707748923 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2f7801e070 con 0x7f2f88101e70 2024-04-01T01:24:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.098+0000 7f2f86ffd700 1 --2- 172.21.15.70:0/2707748923 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2f70038ea0 0x7f2f7003b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:44.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.099+0000 7f2f86ffd700 1 --2- 172.21.15.70:0/2707748923 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2f70038ea0 0x7f2f7003b360 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c006fd0 tx=0x7f2f7c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:44.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.103+0000 7f2f84ff9700 1 -- 172.21.15.70:0/2707748923 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2f78013070 con 0x7f2f88101e70 2024-04-01T01:24:44.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.305+0000 7f2f8dc60700 1 -- 172.21.15.70:0/2707748923 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2f881989d0 con 0x7f2f70038ea0 2024-04-01T01:24:44.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.306+0000 7f2f84ff9700 1 -- 172.21.15.70:0/2707748923 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f2f881989d0 con 0x7f2f70038ea0 2024-04-01T01:24:44.306 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:24:44.306 INFO:teuthology.orchestra.run.smithi070.stdout:[{"addr": "172.21.15.70", "hostname": "smithi070", "labels": [], "status": ""}, {"addr": "172.21.15.81", "hostname": "smithi081", "labels": [], "status": ""}] 2024-04-01T01:24:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.308+0000 7f2f8dc60700 1 -- 172.21.15.70:0/2707748923 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2f70038ea0 msgr2=0x7f2f7003b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.308+0000 7f2f8dc60700 1 --2- 172.21.15.70:0/2707748923 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2f70038ea0 0x7f2f7003b360 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c006fd0 tx=0x7f2f7c006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.308+0000 7f2f8dc60700 1 -- 172.21.15.70:0/2707748923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f88101e70 msgr2=0x7f2f88197670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.308+0000 7f2f8dc60700 1 --2- 172.21.15.70:0/2707748923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f88101e70 0x7f2f88197670 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2f780056d0 tx=0x7f2f780057b0 comp rx=0 tx=0).stop 2024-04-01T01:24:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.308+0000 7f2f8dc60700 1 -- 172.21.15.70:0/2707748923 shutdown_connections 2024-04-01T01:24:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.308+0000 7f2f8dc60700 1 --2- 172.21.15.70:0/2707748923 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2f70038ea0 0x7f2f7003b360 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:44.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.308+0000 7f2f8dc60700 1 --2- 172.21.15.70:0/2707748923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f88101e70 0x7f2f88197670 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:44.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.308+0000 7f2f8dc60700 1 -- 172.21.15.70:0/2707748923 >> 172.21.15.70:0/2707748923 conn(0x7f2f880fd720 msgr2=0x7f2f88103840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:44.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.308+0000 7f2f8dc60700 1 -- 172.21.15.70:0/2707748923 shutdown_connections 2024-04-01T01:24:44.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:44.308+0000 7f2f8dc60700 1 -- 172.21.15.70:0/2707748923 wait complete. 2024-04-01T01:24:44.889 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:44 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:44.917 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi189 2024-04-01T01:24:44.917 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-01T01:24:44.917 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.conf 2024-04-01T01:24:44.937 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-01T01:24:44.937 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-01T01:24:44.994 INFO:tasks.cephadm:Adding host smithi189 to orchestrator... 2024-04-01T01:24:44.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi189 2024-04-01T01:24:45.227 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:24:46.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:45 smithi070 ceph-mon[32179]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:24:46.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:45 smithi070 ceph-mon[32179]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:24:46.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.666+0000 7f7cbb80c700 1 -- 172.21.15.70:0/1754390004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cb4106c50 msgr2=0x7f7cb4107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:46.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.666+0000 7f7cbb80c700 1 --2- 172.21.15.70:0/1754390004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cb4106c50 0x7f7cb4107030 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009910 tx=0x7f7cb0009c20 comp rx=0 tx=0).stop 2024-04-01T01:24:46.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.666+0000 7f7cbb80c700 1 -- 172.21.15.70:0/1754390004 shutdown_connections 2024-04-01T01:24:46.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.666+0000 7f7cbb80c700 1 --2- 172.21.15.70:0/1754390004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cb4106c50 0x7f7cb4107030 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:46.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.666+0000 7f7cbb80c700 1 -- 172.21.15.70:0/1754390004 >> 172.21.15.70:0/1754390004 conn(0x7f7cb40fbd90 msgr2=0x7f7cb40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:46.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.666+0000 7f7cbb80c700 1 -- 172.21.15.70:0/1754390004 shutdown_connections 2024-04-01T01:24:46.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.667+0000 7f7cbb80c700 1 -- 172.21.15.70:0/1754390004 wait complete. 2024-04-01T01:24:46.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.667+0000 7f7cbb80c700 1 Processor -- start 2024-04-01T01:24:46.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.667+0000 7f7cbb80c700 1 -- start start 2024-04-01T01:24:46.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.668+0000 7f7cbb80c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cb4106c50 0x7f7cb4197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:46.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.668+0000 7f7cbb80c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb410c260 con 0x7f7cb4106c50 2024-04-01T01:24:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.668+0000 7f7cb95a8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cb4106c50 0x7f7cb4197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.668+0000 7f7cb95a8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cb4106c50 0x7f7cb4197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46104/0 (socket says 172.21.15.70:46104) 2024-04-01T01:24:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.668+0000 7f7cb95a8700 1 -- 172.21.15.70:0/3542684709 learned_addr learned my addr 172.21.15.70:0/3542684709 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.668+0000 7f7cb95a8700 1 -- 172.21.15.70:0/3542684709 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb4197bd0 con 0x7f7cb4106c50 2024-04-01T01:24:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.668+0000 7f7cb95a8700 1 --2- 172.21.15.70:0/3542684709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cb4106c50 0x7f7cb4197690 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7cb00056d0 tx=0x7f7cb00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.668+0000 7f7caa7fc700 1 -- 172.21.15.70:0/3542684709 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cb0026070 con 0x7f7cb4106c50 2024-04-01T01:24:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.668+0000 7f7cbb80c700 1 -- 172.21.15.70:0/3542684709 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb4197e60 con 0x7f7cb4106c50 2024-04-01T01:24:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.668+0000 7f7caa7fc700 1 -- 172.21.15.70:0/3542684709 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cb000bc00 con 0x7f7cb4106c50 2024-04-01T01:24:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.668+0000 7f7caa7fc700 1 -- 172.21.15.70:0/3542684709 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cb00039b0 con 0x7f7cb4106c50 2024-04-01T01:24:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.668+0000 7f7cbb80c700 1 -- 172.21.15.70:0/3542684709 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb41982c0 con 0x7f7cb4106c50 2024-04-01T01:24:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.669+0000 7f7cbb80c700 1 -- 172.21.15.70:0/3542684709 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cb4103390 con 0x7f7cb4106c50 2024-04-01T01:24:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.669+0000 7f7caa7fc700 1 -- 172.21.15.70:0/3542684709 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f7cb0029070 con 0x7f7cb4106c50 2024-04-01T01:24:46.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.669+0000 7f7caa7fc700 1 --2- 172.21.15.70:0/3542684709 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7ca0038ea0 0x7f7ca003b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:46.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.669+0000 7f7caa7fc700 1 -- 172.21.15.70:0/3542684709 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7cb001e070 con 0x7f7cb4106c50 2024-04-01T01:24:46.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.669+0000 7f7cb8da7700 1 --2- 172.21.15.70:0/3542684709 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7ca0038ea0 0x7f7ca003b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:46.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.670+0000 7f7cb8da7700 1 --2- 172.21.15.70:0/3542684709 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7ca0038ea0 0x7f7ca003b360 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4006fd0 tx=0x7f7ca4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:46.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.674+0000 7f7caa7fc700 1 -- 172.21.15.70:0/3542684709 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7cb0013070 con 0x7f7cb4106c50 2024-04-01T01:24:46.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:46.876+0000 7f7cbb80c700 1 -- 172.21.15.70:0/3542684709 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}) -- 0x7f7cb41989f0 con 0x7f7ca0038ea0 2024-04-01T01:24:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:47 smithi070 ceph-mon[32179]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:24:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi081", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:24:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:24:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:24:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:48.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:49.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:48 smithi070 ceph-mon[32179]: Updating smithi081:/etc/ceph/ceph.conf 2024-04-01T01:24:49.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:48 smithi070 ceph-mon[32179]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:24:49.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:48 smithi070 ceph-mon[32179]: Updating smithi081:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-01T01:24:49.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:48 smithi070 ceph-mon[32179]: Updating smithi081:/etc/ceph/ceph.client.admin.keyring 2024-04-01T01:24:49.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:48 smithi070 ceph-mon[32179]: Updating smithi081:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-01T01:24:49.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:48 smithi070 ceph-mon[32179]: Deploying cephadm binary to smithi189 2024-04-01T01:24:50.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:49 smithi070 ceph-mon[32179]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:24:51.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:51.225+0000 7f7caa7fc700 1 -- 172.21.15.70:0/3542684709 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f7cb41989f0 con 0x7f7ca0038ea0 2024-04-01T01:24:51.225 INFO:teuthology.orchestra.run.smithi070.stdout:Added host 'smithi189' with addr '172.21.15.189' 2024-04-01T01:24:51.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:51.227+0000 7f7cbb80c700 1 -- 172.21.15.70:0/3542684709 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7ca0038ea0 msgr2=0x7f7ca003b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:51.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:51.227+0000 7f7cbb80c700 1 --2- 172.21.15.70:0/3542684709 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7ca0038ea0 0x7f7ca003b360 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4006fd0 tx=0x7f7ca4006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:51.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:51.227+0000 7f7cbb80c700 1 -- 172.21.15.70:0/3542684709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cb4106c50 msgr2=0x7f7cb4197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:51.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:51.227+0000 7f7cbb80c700 1 --2- 172.21.15.70:0/3542684709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cb4106c50 0x7f7cb4197690 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7cb00056d0 tx=0x7f7cb00057b0 comp rx=0 tx=0).stop 2024-04-01T01:24:51.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:51.227+0000 7f7cbb80c700 1 -- 172.21.15.70:0/3542684709 shutdown_connections 2024-04-01T01:24:51.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:51.227+0000 7f7cbb80c700 1 --2- 172.21.15.70:0/3542684709 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7ca0038ea0 0x7f7ca003b360 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:51.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:51.227+0000 7f7cbb80c700 1 --2- 172.21.15.70:0/3542684709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7cb4106c50 0x7f7cb4197690 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:51.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:51.227+0000 7f7cbb80c700 1 -- 172.21.15.70:0/3542684709 >> 172.21.15.70:0/3542684709 conn(0x7f7cb40fbd90 msgr2=0x7f7cb41097c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:51.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:51.227+0000 7f7cbb80c700 1 -- 172.21.15.70:0/3542684709 shutdown_connections 2024-04-01T01:24:51.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:51.227+0000 7f7cbb80c700 1 -- 172.21.15.70:0/3542684709 wait complete. 2024-04-01T01:24:51.827 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:51 smithi070 ceph-mon[32179]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:24:51.827 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:51 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:51.827 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:51 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:24:51.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-01T01:24:52.087 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:24:52.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:52 smithi070 ceph-mon[32179]: Added host smithi189 2024-04-01T01:24:52.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:52 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:52.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:52 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:53.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.545+0000 7f8f78dd0700 1 -- 172.21.15.70:0/4187819326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f74102dd0 msgr2=0x7f8f741031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:53.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.545+0000 7f8f78dd0700 1 --2- 172.21.15.70:0/4187819326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f74102dd0 0x7f8f741031b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8f6400b0a0 tx=0x7f8f6400b3b0 comp rx=0 tx=0).stop 2024-04-01T01:24:53.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.545+0000 7f8f78dd0700 1 -- 172.21.15.70:0/4187819326 shutdown_connections 2024-04-01T01:24:53.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.545+0000 7f8f78dd0700 1 --2- 172.21.15.70:0/4187819326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f74102dd0 0x7f8f741031b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.545+0000 7f8f78dd0700 1 -- 172.21.15.70:0/4187819326 >> 172.21.15.70:0/4187819326 conn(0x7f8f74077780 msgr2=0x7f8f740fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.545+0000 7f8f78dd0700 1 -- 172.21.15.70:0/4187819326 shutdown_connections 2024-04-01T01:24:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.545+0000 7f8f78dd0700 1 -- 172.21.15.70:0/4187819326 wait complete. 2024-04-01T01:24:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.546+0000 7f8f78dd0700 1 Processor -- start 2024-04-01T01:24:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.546+0000 7f8f78dd0700 1 -- start start 2024-04-01T01:24:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.546+0000 7f8f78dd0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f74102dd0 0x7f8f741976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.546+0000 7f8f78dd0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7410c170 con 0x7f8f74102dd0 2024-04-01T01:24:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.546+0000 7f8f7259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f74102dd0 0x7f8f741976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.546+0000 7f8f7259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f74102dd0 0x7f8f741976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46118/0 (socket says 172.21.15.70:46118) 2024-04-01T01:24:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.546+0000 7f8f7259c700 1 -- 172.21.15.70:0/1777213739 learned_addr learned my addr 172.21.15.70:0/1777213739 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.546+0000 7f8f7259c700 1 -- 172.21.15.70:0/1777213739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f74197be0 con 0x7f8f74102dd0 2024-04-01T01:24:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.546+0000 7f8f7259c700 1 --2- 172.21.15.70:0/1777213739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f74102dd0 0x7f8f741976a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8f640076a0 tx=0x7f8f64009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.546+0000 7f8f637fe700 1 -- 172.21.15.70:0/1777213739 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f64017040 con 0x7f8f74102dd0 2024-04-01T01:24:53.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.547+0000 7f8f637fe700 1 -- 172.21.15.70:0/1777213739 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f64004040 con 0x7f8f74102dd0 2024-04-01T01:24:53.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.547+0000 7f8f78dd0700 1 -- 172.21.15.70:0/1777213739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f74197e70 con 0x7f8f74102dd0 2024-04-01T01:24:53.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.547+0000 7f8f637fe700 1 -- 172.21.15.70:0/1777213739 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f6401d450 con 0x7f8f74102dd0 2024-04-01T01:24:53.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.547+0000 7f8f78dd0700 1 -- 172.21.15.70:0/1777213739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f741982d0 con 0x7f8f74102dd0 2024-04-01T01:24:53.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.547+0000 7f8f637fe700 1 -- 172.21.15.70:0/1777213739 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f8f64024020 con 0x7f8f74102dd0 2024-04-01T01:24:53.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.547+0000 7f8f78dd0700 1 -- 172.21.15.70:0/1777213739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f74069f10 con 0x7f8f74102dd0 2024-04-01T01:24:53.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.548+0000 7f8f637fe700 1 --2- 172.21.15.70:0/1777213739 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f5c038ea0 0x7f8f5c03b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:53.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.548+0000 7f8f637fe700 1 -- 172.21.15.70:0/1777213739 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8f64022020 con 0x7f8f74102dd0 2024-04-01T01:24:53.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.548+0000 7f8f71d9b700 1 --2- 172.21.15.70:0/1777213739 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f5c038ea0 0x7f8f5c03b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:53.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.548+0000 7f8f71d9b700 1 --2- 172.21.15.70:0/1777213739 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f5c038ea0 0x7f8f5c03b360 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8f68006fd0 tx=0x7f8f68006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:53.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.553+0000 7f8f637fe700 1 -- 172.21.15.70:0/1777213739 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8f6402b020 con 0x7f8f74102dd0 2024-04-01T01:24:53.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.756+0000 7f8f78dd0700 1 -- 172.21.15.70:0/1777213739 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8f74198a70 con 0x7f8f5c038ea0 2024-04-01T01:24:53.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.757+0000 7f8f637fe700 1 -- 172.21.15.70:0/1777213739 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f8f74198a70 con 0x7f8f5c038ea0 2024-04-01T01:24:53.757 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:24:53.757 INFO:teuthology.orchestra.run.smithi070.stdout:[{"addr": "172.21.15.70", "hostname": "smithi070", "labels": [], "status": ""}, {"addr": "172.21.15.81", "hostname": "smithi081", "labels": [], "status": ""}, {"addr": "172.21.15.189", "hostname": "smithi189", "labels": [], "status": ""}] 2024-04-01T01:24:53.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.759+0000 7f8f78dd0700 1 -- 172.21.15.70:0/1777213739 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f5c038ea0 msgr2=0x7f8f5c03b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:53.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.759+0000 7f8f78dd0700 1 --2- 172.21.15.70:0/1777213739 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f5c038ea0 0x7f8f5c03b360 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8f68006fd0 tx=0x7f8f68006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:53.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.759+0000 7f8f78dd0700 1 -- 172.21.15.70:0/1777213739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f74102dd0 msgr2=0x7f8f741976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:53.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.759+0000 7f8f78dd0700 1 --2- 172.21.15.70:0/1777213739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f74102dd0 0x7f8f741976a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8f640076a0 tx=0x7f8f64009800 comp rx=0 tx=0).stop 2024-04-01T01:24:53.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.759+0000 7f8f78dd0700 1 -- 172.21.15.70:0/1777213739 shutdown_connections 2024-04-01T01:24:53.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.759+0000 7f8f78dd0700 1 --2- 172.21.15.70:0/1777213739 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f5c038ea0 0x7f8f5c03b360 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:53.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.759+0000 7f8f78dd0700 1 --2- 172.21.15.70:0/1777213739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f74102dd0 0x7f8f741976a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:53.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.759+0000 7f8f78dd0700 1 -- 172.21.15.70:0/1777213739 >> 172.21.15.70:0/1777213739 conn(0x7f8f74077780 msgr2=0x7f8f74103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:53.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.759+0000 7f8f78dd0700 1 -- 172.21.15.70:0/1777213739 shutdown_connections 2024-04-01T01:24:53.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:53.759+0000 7f8f78dd0700 1 -- 172.21.15.70:0/1777213739 wait complete. 2024-04-01T01:24:53.770 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:53 smithi070 ceph-mon[32179]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:24:53.770 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:53 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:54.400 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-01T01:24:54.401 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd crush tunables default 2024-04-01T01:24:54.634 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:24:55.824 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:55 smithi070 ceph-mon[32179]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:24:55.824 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:55 smithi070 ceph-mon[32179]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:24:56.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.031+0000 7f05617af700 1 -- 172.21.15.70:0/2823433196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f055c102e40 msgr2=0x7f055c103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:56.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.031+0000 7f05617af700 1 --2- 172.21.15.70:0/2823433196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f055c102e40 0x7f055c103220 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f055400b0a0 tx=0x7f055400b3b0 comp rx=0 tx=0).stop 2024-04-01T01:24:56.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.032+0000 7f05617af700 1 -- 172.21.15.70:0/2823433196 shutdown_connections 2024-04-01T01:24:56.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.032+0000 7f05617af700 1 --2- 172.21.15.70:0/2823433196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f055c102e40 0x7f055c103220 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:56.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.032+0000 7f05617af700 1 -- 172.21.15.70:0/2823433196 >> 172.21.15.70:0/2823433196 conn(0x7f055c0fc630 msgr2=0x7f055c0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:56.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.032+0000 7f05617af700 1 -- 172.21.15.70:0/2823433196 shutdown_connections 2024-04-01T01:24:56.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.032+0000 7f05617af700 1 -- 172.21.15.70:0/2823433196 wait complete. 2024-04-01T01:24:56.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.032+0000 7f05617af700 1 Processor -- start 2024-04-01T01:24:56.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.032+0000 7f05617af700 1 -- start start 2024-04-01T01:24:56.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.033+0000 7f05617af700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f055c102e40 0x7f055c197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:56.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.033+0000 7f05617af700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f055c10c170 con 0x7f055c102e40 2024-04-01T01:24:56.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.033+0000 7f055affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f055c102e40 0x7f055c197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:56.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.033+0000 7f055affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f055c102e40 0x7f055c197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46136/0 (socket says 172.21.15.70:46136) 2024-04-01T01:24:56.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.033+0000 7f055affd700 1 -- 172.21.15.70:0/3109714928 learned_addr learned my addr 172.21.15.70:0/3109714928 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:24:56.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.033+0000 7f055affd700 1 -- 172.21.15.70:0/3109714928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f055c197bb0 con 0x7f055c102e40 2024-04-01T01:24:56.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.033+0000 7f055affd700 1 --2- 172.21.15.70:0/3109714928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f055c102e40 0x7f055c197670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f05540076a0 tx=0x7f0554009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:56.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.033+0000 7f0543fff700 1 -- 172.21.15.70:0/3109714928 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0554017040 con 0x7f055c102e40 2024-04-01T01:24:56.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.033+0000 7f05617af700 1 -- 172.21.15.70:0/3109714928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f055c197e40 con 0x7f055c102e40 2024-04-01T01:24:56.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.033+0000 7f0543fff700 1 -- 172.21.15.70:0/3109714928 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0554004040 con 0x7f055c102e40 2024-04-01T01:24:56.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.033+0000 7f0543fff700 1 -- 172.21.15.70:0/3109714928 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f055401d450 con 0x7f055c102e40 2024-04-01T01:24:56.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.033+0000 7f05617af700 1 -- 172.21.15.70:0/3109714928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f055c1982a0 con 0x7f055c102e40 2024-04-01T01:24:56.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.034+0000 7f0543fff700 1 -- 172.21.15.70:0/3109714928 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f0554024020 con 0x7f055c102e40 2024-04-01T01:24:56.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.034+0000 7f05617af700 1 -- 172.21.15.70:0/3109714928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f055c06a9a0 con 0x7f055c102e40 2024-04-01T01:24:56.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.034+0000 7f0543fff700 1 --2- 172.21.15.70:0/3109714928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0544038ea0 0x7f054403b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:56.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.034+0000 7f0543fff700 1 -- 172.21.15.70:0/3109714928 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0554022020 con 0x7f055c102e40 2024-04-01T01:24:56.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.034+0000 7f055a7fc700 1 --2- 172.21.15.70:0/3109714928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0544038ea0 0x7f054403b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:56.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.035+0000 7f055a7fc700 1 --2- 172.21.15.70:0/3109714928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0544038ea0 0x7f054403b360 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0550006fd0 tx=0x7f0550006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:56.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.039+0000 7f0543fff700 1 -- 172.21.15.70:0/3109714928 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f055402b020 con 0x7f055c102e40 2024-04-01T01:24:56.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.236+0000 7f05617af700 1 -- 172.21.15.70:0/3109714928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f055c1989d0 con 0x7f055c102e40 2024-04-01T01:24:56.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.628+0000 7f0543fff700 1 -- 172.21.15.70:0/3109714928 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f0554004560 con 0x7f055c102e40 2024-04-01T01:24:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.630+0000 7f05617af700 1 -- 172.21.15.70:0/3109714928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0544038ea0 msgr2=0x7f054403b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.630+0000 7f05617af700 1 --2- 172.21.15.70:0/3109714928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0544038ea0 0x7f054403b360 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0550006fd0 tx=0x7f0550006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.630+0000 7f05617af700 1 -- 172.21.15.70:0/3109714928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f055c102e40 msgr2=0x7f055c197670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.630+0000 7f05617af700 1 --2- 172.21.15.70:0/3109714928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f055c102e40 0x7f055c197670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f05540076a0 tx=0x7f0554009800 comp rx=0 tx=0).stop 2024-04-01T01:24:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.630+0000 7f05617af700 1 -- 172.21.15.70:0/3109714928 shutdown_connections 2024-04-01T01:24:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.630+0000 7f05617af700 1 --2- 172.21.15.70:0/3109714928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0544038ea0 0x7f054403b360 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.630+0000 7f05617af700 1 --2- 172.21.15.70:0/3109714928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f055c102e40 0x7f055c197670 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.630+0000 7f05617af700 1 -- 172.21.15.70:0/3109714928 >> 172.21.15.70:0/3109714928 conn(0x7f055c0fc630 msgr2=0x7f055c103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:56.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.630+0000 7f05617af700 1 -- 172.21.15.70:0/3109714928 shutdown_connections 2024-04-01T01:24:56.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:24:56.630+0000 7f05617af700 1 -- 172.21.15.70:0/3109714928 wait complete. 2024-04-01T01:24:56.632 INFO:teuthology.orchestra.run.smithi070.stderr:adjusted tunables profile to default 2024-04-01T01:24:56.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:56 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3109714928' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-01T01:24:56.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:56.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:56.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:56.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:56.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:24:56.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:24:56.919 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:24:57.244 INFO:tasks.cephadm:Adding mon.a on smithi070 2024-04-01T01:24:57.244 INFO:tasks.cephadm:Adding mon.b on smithi081 2024-04-01T01:24:57.245 INFO:tasks.cephadm:Adding mon.c on smithi189 2024-04-01T01:24:57.245 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch apply mon '3;smithi070:172.21.15.70=a;smithi081:172.21.15.81=b;smithi189:172.21.15.189=c' 2024-04-01T01:24:57.471 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-01T01:24:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:57 smithi070 ceph-mon[32179]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-01T01:24:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:57 smithi070 ceph-mon[32179]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:24:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:57 smithi070 ceph-mon[32179]: Updating smithi189:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-01T01:24:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:57 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3109714928' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-01T01:24:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:57 smithi070 ceph-mon[32179]: osdmap e4: 0 total, 0 up, 0 in 2024-04-01T01:24:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:57 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:57 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:57 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:24:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.800+0000 7f54e117c700 1 -- 172.21.15.189:0/4081171074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54dc106bf0 msgr2=0x7f54dc106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.800+0000 7f54e117c700 1 --2- 172.21.15.189:0/4081171074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54dc106bf0 0x7f54dc106fd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f54d0009910 tx=0x7f54d0009c20 comp rx=0 tx=0).stop 2024-04-01T01:24:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.800+0000 7f54e117c700 1 -- 172.21.15.189:0/4081171074 shutdown_connections 2024-04-01T01:24:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.800+0000 7f54e117c700 1 --2- 172.21.15.189:0/4081171074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54dc106bf0 0x7f54dc106fd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.800+0000 7f54e117c700 1 -- 172.21.15.189:0/4081171074 >> 172.21.15.189:0/4081171074 conn(0x7f54dc0fc610 msgr2=0x7f54dc0fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.800+0000 7f54e117c700 1 -- 172.21.15.189:0/4081171074 shutdown_connections 2024-04-01T01:24:58.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.800+0000 7f54e117c700 1 -- 172.21.15.189:0/4081171074 wait complete. 2024-04-01T01:24:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.801+0000 7f54e117c700 1 Processor -- start 2024-04-01T01:24:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.801+0000 7f54e117c700 1 -- start start 2024-04-01T01:24:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.801+0000 7f54e117c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54dc106bf0 0x7f54dc0fcac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.801+0000 7f54e117c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f54dc10b060 con 0x7f54dc106bf0 2024-04-01T01:24:58.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.801+0000 7f54dad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54dc106bf0 0x7f54dc0fcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.801+0000 7f54dad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54dc106bf0 0x7f54dc0fcac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.189:43256/0 (socket says 172.21.15.189:43256) 2024-04-01T01:24:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.801+0000 7f54dad9d700 1 -- 172.21.15.189:0/1376528501 learned_addr learned my addr 172.21.15.189:0/1376528501 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-01T01:24:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.802+0000 7f54dad9d700 1 -- 172.21.15.189:0/1376528501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54dc0fe9b0 con 0x7f54dc106bf0 2024-04-01T01:24:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.802+0000 7f54dad9d700 1 --2- 172.21.15.189:0/1376528501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54dc106bf0 0x7f54dc0fcac0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f54d0006010 tx=0x7f54d0005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.802+0000 7f54c3fff700 1 -- 172.21.15.189:0/1376528501 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54d0026070 con 0x7f54dc106bf0 2024-04-01T01:24:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.802+0000 7f54c3fff700 1 -- 172.21.15.189:0/1376528501 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54d000bc00 con 0x7f54dc106bf0 2024-04-01T01:24:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.802+0000 7f54c3fff700 1 -- 172.21.15.189:0/1376528501 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54d0003930 con 0x7f54dc106bf0 2024-04-01T01:24:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.803+0000 7f54e117c700 1 -- 172.21.15.189:0/1376528501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54dc0fd000 con 0x7f54dc106bf0 2024-04-01T01:24:58.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.803+0000 7f54e117c700 1 -- 172.21.15.189:0/1376528501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54dc0fd4c0 con 0x7f54dc106bf0 2024-04-01T01:24:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.803+0000 7f54c3fff700 1 -- 172.21.15.189:0/1376528501 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f54d0029070 con 0x7f54dc106bf0 2024-04-01T01:24:58.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.804+0000 7f54e117c700 1 -- 172.21.15.189:0/1376528501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54dc1021f0 con 0x7f54dc106bf0 2024-04-01T01:24:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.804+0000 7f54c3fff700 1 --2- 172.21.15.189:0/1376528501 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54c4038ea0 0x7f54c403b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:24:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.804+0000 7f54c3fff700 1 -- 172.21.15.189:0/1376528501 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f54d001e070 con 0x7f54dc106bf0 2024-04-01T01:24:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.804+0000 7f54da59c700 1 --2- 172.21.15.189:0/1376528501 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54c4038ea0 0x7f54c403b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:24:58.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.805+0000 7f54da59c700 1 --2- 172.21.15.189:0/1376528501 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54c4038ea0 0x7f54c403b360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f54cc006fd0 tx=0x7f54cc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:24:58.809 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:58.809+0000 7f54c3fff700 1 -- 172.21.15.189:0/1376528501 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f54d0013070 con 0x7f54dc106bf0 2024-04-01T01:24:59.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:58 smithi070 ceph-mon[32179]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-04-01T01:24:59.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:58 smithi070 ceph-mon[32179]: Updating smithi189:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-01T01:24:59.013 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:59.013+0000 7f54e117c700 1 -- 172.21.15.189:0/1376528501 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi070:172.21.15.70=a;smithi081:172.21.15.81=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}) -- 0x7f54dc0fd6f0 con 0x7f54c4038ea0 2024-04-01T01:24:59.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:59.020+0000 7f54c3fff700 1 -- 172.21.15.189:0/1376528501 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f54dc0fd6f0 con 0x7f54c4038ea0 2024-04-01T01:24:59.020 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mon update... 2024-04-01T01:24:59.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:59.022+0000 7f54e117c700 1 -- 172.21.15.189:0/1376528501 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54c4038ea0 msgr2=0x7f54c403b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:59.022 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:59.022+0000 7f54e117c700 1 --2- 172.21.15.189:0/1376528501 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54c4038ea0 0x7f54c403b360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f54cc006fd0 tx=0x7f54cc006e40 comp rx=0 tx=0).stop 2024-04-01T01:24:59.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:59.022+0000 7f54e117c700 1 -- 172.21.15.189:0/1376528501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54dc106bf0 msgr2=0x7f54dc0fcac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:24:59.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:59.022+0000 7f54e117c700 1 --2- 172.21.15.189:0/1376528501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54dc106bf0 0x7f54dc0fcac0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f54d0006010 tx=0x7f54d0005780 comp rx=0 tx=0).stop 2024-04-01T01:24:59.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:59.022+0000 7f54e117c700 1 -- 172.21.15.189:0/1376528501 shutdown_connections 2024-04-01T01:24:59.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:59.022+0000 7f54e117c700 1 --2- 172.21.15.189:0/1376528501 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54c4038ea0 0x7f54c403b360 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:59.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:59.022+0000 7f54e117c700 1 --2- 172.21.15.189:0/1376528501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54dc106bf0 0x7f54dc0fcac0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:24:59.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:59.022+0000 7f54e117c700 1 -- 172.21.15.189:0/1376528501 >> 172.21.15.189:0/1376528501 conn(0x7f54dc0fc610 msgr2=0x7f54dc06a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:24:59.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:59.022+0000 7f54e117c700 1 -- 172.21.15.189:0/1376528501 shutdown_connections 2024-04-01T01:24:59.023 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:24:59.022+0000 7f54e117c700 1 -- 172.21.15.189:0/1376528501 wait complete. 2024-04-01T01:24:59.670 DEBUG:teuthology.orchestra.run.smithi081:mon.b> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mon.b.service 2024-04-01T01:24:59.673 DEBUG:teuthology.orchestra.run.smithi189:mon.c> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mon.c.service 2024-04-01T01:24:59.675 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-01T01:24:59.675 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-01T01:25:00.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:59 smithi070 ceph-mon[32179]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:00.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:59 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:00.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:59 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:00.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:59 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:00.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:59 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:00.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:59 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:00.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:59 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-01T01:25:00.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:24:59 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:00.399 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-01T01:25:01.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:00 smithi070 ceph-mon[32179]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi070:172.21.15.70=a;smithi081:172.21.15.81=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:25:01.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:00 smithi070 ceph-mon[32179]: Saving service mon spec with placement smithi070:172.21.15.70=a;smithi081:172.21.15.81=b;smithi189:172.21.15.189=c;count:3 2024-04-01T01:25:01.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:00 smithi070 ceph-mon[32179]: Deploying daemon mon.c on smithi189 2024-04-01T01:25:02.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:01 smithi070 ceph-mon[32179]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:02.211 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.210+0000 7fac89f1f700 1 -- 172.21.15.189:0/3431948966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac84102e70 msgr2=0x7fac84103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:02.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.210+0000 7fac89f1f700 1 --2- 172.21.15.189:0/3431948966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac84102e70 0x7fac84103250 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fac74009910 tx=0x7fac74009c20 comp rx=0 tx=0).stop 2024-04-01T01:25:02.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.210+0000 7fac89f1f700 1 -- 172.21.15.189:0/3431948966 shutdown_connections 2024-04-01T01:25:02.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.210+0000 7fac89f1f700 1 --2- 172.21.15.189:0/3431948966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac84102e70 0x7fac84103250 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:02.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.210+0000 7fac89f1f700 1 -- 172.21.15.189:0/3431948966 >> 172.21.15.189:0/3431948966 conn(0x7fac84077780 msgr2=0x7fac840fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:25:02.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.210+0000 7fac89f1f700 1 -- 172.21.15.189:0/3431948966 shutdown_connections 2024-04-01T01:25:02.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.210+0000 7fac89f1f700 1 -- 172.21.15.189:0/3431948966 wait complete. 2024-04-01T01:25:02.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.211+0000 7fac89f1f700 1 Processor -- start 2024-04-01T01:25:02.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.211+0000 7fac89f1f700 1 -- start start 2024-04-01T01:25:02.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.211+0000 7fac89f1f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac84102e70 0x7fac841976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:02.212 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.211+0000 7fac89f1f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fac8410c170 con 0x7fac84102e70 2024-04-01T01:25:02.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.211+0000 7fac837fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac84102e70 0x7fac841976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:02.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.211+0000 7fac837fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac84102e70 0x7fac841976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.189:43264/0 (socket says 172.21.15.189:43264) 2024-04-01T01:25:02.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.211+0000 7fac837fe700 1 -- 172.21.15.189:0/1407220077 learned_addr learned my addr 172.21.15.189:0/1407220077 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-01T01:25:02.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.212+0000 7fac837fe700 1 -- 172.21.15.189:0/1407220077 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac84197c10 con 0x7fac84102e70 2024-04-01T01:25:02.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.212+0000 7fac837fe700 1 --2- 172.21.15.189:0/1407220077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac84102e70 0x7fac841976d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fac740056d0 tx=0x7fac740057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:25:02.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.212+0000 7fac80ff9700 1 -- 172.21.15.189:0/1407220077 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac74026070 con 0x7fac84102e70 2024-04-01T01:25:02.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.212+0000 7fac89f1f700 1 -- 172.21.15.189:0/1407220077 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac84197ea0 con 0x7fac84102e70 2024-04-01T01:25:02.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.212+0000 7fac80ff9700 1 -- 172.21.15.189:0/1407220077 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac74004370 con 0x7fac84102e70 2024-04-01T01:25:02.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.212+0000 7fac80ff9700 1 -- 172.21.15.189:0/1407220077 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fac740039b0 con 0x7fac84102e70 2024-04-01T01:25:02.213 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.212+0000 7fac89f1f700 1 -- 172.21.15.189:0/1407220077 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac84198300 con 0x7fac84102e70 2024-04-01T01:25:02.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.213+0000 7fac89f1f700 1 -- 172.21.15.189:0/1407220077 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac84069e70 con 0x7fac84102e70 2024-04-01T01:25:02.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.213+0000 7fac80ff9700 1 -- 172.21.15.189:0/1407220077 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7fac74029070 con 0x7fac84102e70 2024-04-01T01:25:02.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.213+0000 7fac80ff9700 1 --2- 172.21.15.189:0/1407220077 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac6c038ea0 0x7fac6c03b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:02.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.213+0000 7fac80ff9700 1 -- 172.21.15.189:0/1407220077 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fac7401e070 con 0x7fac84102e70 2024-04-01T01:25:02.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.213+0000 7fac82ffd700 1 --2- 172.21.15.189:0/1407220077 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac6c038ea0 0x7fac6c03b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:02.214 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.214+0000 7fac82ffd700 1 --2- 172.21.15.189:0/1407220077 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac6c038ea0 0x7fac6c03b360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fac78006fd0 tx=0x7fac78006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:25:02.218 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.219+0000 7fac80ff9700 1 -- 172.21.15.189:0/1407220077 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fac74013070 con 0x7fac84102e70 2024-04-01T01:25:02.511 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.511+0000 7fac89f1f700 1 -- 172.21.15.189:0/1407220077 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fac84198a30 con 0x7fac84102e70 2024-04-01T01:25:02.514 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.512+0000 7fac80ff9700 1 -- 172.21.15.189:0/1407220077 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fac74030020 con 0x7fac84102e70 2024-04-01T01:25:02.514 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:25:02.514 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","modified":"2024-04-01T01:23:17.752525Z","created":"2024-04-01T01:23:17.752525Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-01T01:25:02.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.514+0000 7fac89f1f700 1 -- 172.21.15.189:0/1407220077 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac6c038ea0 msgr2=0x7fac6c03b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:02.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.514+0000 7fac89f1f700 1 --2- 172.21.15.189:0/1407220077 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac6c038ea0 0x7fac6c03b360 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fac78006fd0 tx=0x7fac78006e40 comp rx=0 tx=0).stop 2024-04-01T01:25:02.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.514+0000 7fac89f1f700 1 -- 172.21.15.189:0/1407220077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac84102e70 msgr2=0x7fac841976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:02.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.514+0000 7fac89f1f700 1 --2- 172.21.15.189:0/1407220077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac84102e70 0x7fac841976d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fac740056d0 tx=0x7fac740057b0 comp rx=0 tx=0).stop 2024-04-01T01:25:02.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.514+0000 7fac89f1f700 1 -- 172.21.15.189:0/1407220077 shutdown_connections 2024-04-01T01:25:02.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.514+0000 7fac89f1f700 1 --2- 172.21.15.189:0/1407220077 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac6c038ea0 0x7fac6c03b360 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:02.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.514+0000 7fac89f1f700 1 --2- 172.21.15.189:0/1407220077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac84102e70 0x7fac841976d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:02.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.514+0000 7fac89f1f700 1 -- 172.21.15.189:0/1407220077 >> 172.21.15.189:0/1407220077 conn(0x7fac84077780 msgr2=0x7fac84103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:25:02.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.514+0000 7fac89f1f700 1 -- 172.21.15.189:0/1407220077 shutdown_connections 2024-04-01T01:25:02.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:02.514+0000 7fac89f1f700 1 -- 172.21.15.189:0/1407220077 wait complete. 2024-04-01T01:25:02.516 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-01T01:25:03.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:02 smithi070 ceph-mon[32179]: from='client.? 172.21.15.189:0/1407220077' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-01T01:25:04.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:03 smithi070 ceph-mon[32179]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:04.557 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:04 smithi189 ceph-mon[35753]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-01T01:25:04.557 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:04 smithi189 bash[35734]: 72e7bf301c270dc40b9e417b706fd5d50b20635662da27ad0b1df30e44ec847e 2024-04-01T01:25:04.665 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-01T01:25:04.666 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-01T01:25:04.901 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-01T01:25:04.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:04 smithi189 systemd[1]: Started Ceph mon.c for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:25:09.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.867+0000 7fbc07122700 1 -- 172.21.15.189:0/4011747527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbf00035c0 msgr2=0x7fbbf0005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:09.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.867+0000 7fbc07122700 1 --2- 172.21.15.189:0/4011747527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbf00035c0 0x7fbbf0005a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbc00079540 tx=0x7fbbf4014ec0 comp rx=0 tx=0).stop 2024-04-01T01:25:09.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.867+0000 7fbc07122700 1 -- 172.21.15.189:0/4011747527 shutdown_connections 2024-04-01T01:25:09.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.867+0000 7fbc07122700 1 --2- 172.21.15.189:0/4011747527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbf00035c0 0x7fbbf0005a50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:09.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.867+0000 7fbc07122700 1 -- 172.21.15.189:0/4011747527 >> 172.21.15.189:0/4011747527 conn(0x7fbc00077330 msgr2=0x7fbc00077740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:25:09.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.867+0000 7fbc07122700 1 -- 172.21.15.189:0/4011747527 shutdown_connections 2024-04-01T01:25:09.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.867+0000 7fbc07122700 1 -- 172.21.15.189:0/4011747527 wait complete. 2024-04-01T01:25:09.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.868+0000 7fbc07122700 1 Processor -- start 2024-04-01T01:25:09.869 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.868+0000 7fbc07122700 1 -- start start 2024-04-01T01:25:09.869 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:09.869 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:25:09.869 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: mon.a calling monitor election 2024-04-01T01:25:09.869 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:09.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:09.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:09.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:09.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: mon.c calling monitor election 2024-04-01T01:25:09.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:09.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:09.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:09.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:09.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-01T01:25:09.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: monmap epoch 2 2024-04-01T01:25:09.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: fsid 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:25:09.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: last_changed 2024-04-01T01:25:04.545194+0000 2024-04-01T01:25:09.870 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: created 2024-04-01T01:23:17.752525+0000 2024-04-01T01:25:09.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: min_mon_release 19 (squid) 2024-04-01T01:25:09.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: election_strategy: 1 2024-04-01T01:25:09.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-01T01:25:09.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-01T01:25:09.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: fsmap 2024-04-01T01:25:09.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: osdmap e4: 0 total, 0 up, 0 in 2024-04-01T01:25:09.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: mgrmap e13: x(active, since 51s) 2024-04-01T01:25:09.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: overall HEALTH_OK 2024-04-01T01:25:09.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:09.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:09.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:09.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-01T01:25:09.871 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:09.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.868+0000 7fbc07122700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbf00035c0 0x7fbc00102990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:09.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.868+0000 7fbc07122700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc00102ed0 0x7fbc00103350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:09.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.868+0000 7fbc07122700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbc00108c10 con 0x7fbbf00035c0 2024-04-01T01:25:09.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.868+0000 7fbc07122700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbc00108a90 con 0x7fbc00102ed0 2024-04-01T01:25:09.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.869+0000 7fbbfffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc00102ed0 0x7fbc00103350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:09.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.869+0000 7fbbfffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc00102ed0 0x7fbc00103350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48278/0 (socket says 172.21.15.189:48278) 2024-04-01T01:25:09.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.869+0000 7fbbfffff700 1 -- 172.21.15.189:0/938765598 learned_addr learned my addr 172.21.15.189:0/938765598 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-01T01:25:09.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.869+0000 7fbc04ebe700 1 --2- 172.21.15.189:0/938765598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbf00035c0 0x7fbc00102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:09.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.869+0000 7fbbfffff700 1 -- 172.21.15.189:0/938765598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc00102ed0 msgr2=0x7fbc00103350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-01T01:25:09.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.869+0000 7fbbfffff700 1 -- 172.21.15.189:0/938765598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc00102ed0 msgr2=0x7fbc00103350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-01T01:25:09.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.869+0000 7fbbfffff700 1 --2- 172.21.15.189:0/938765598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc00102ed0 0x7fbc00103350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-01T01:25:09.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.869+0000 7fbbfffff700 1 --2- 172.21.15.189:0/938765598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc00102ed0 0x7fbc00103350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-01T01:25:09.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.869+0000 7fbc04ebe700 1 -- 172.21.15.189:0/938765598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc00102ed0 msgr2=0x7fbc00103350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:25:09.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.869+0000 7fbc04ebe700 1 --2- 172.21.15.189:0/938765598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc00102ed0 0x7fbc00103350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:09.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.869+0000 7fbc04ebe700 1 -- 172.21.15.189:0/938765598 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc00104860 con 0x7fbbf00035c0 2024-04-01T01:25:09.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.869+0000 7fbc04ebe700 1 --2- 172.21.15.189:0/938765598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbf00035c0 0x7fbc00102990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbbf401d000 tx=0x7fbbf400fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:25:09.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.869+0000 7fbbfdffb700 1 -- 172.21.15.189:0/938765598 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fbbf4004aa0 con 0x7fbbf00035c0 2024-04-01T01:25:09.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.870+0000 7fbbfdffb700 1 -- 172.21.15.189:0/938765598 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbf4004c40 con 0x7fbbf00035c0 2024-04-01T01:25:09.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.870+0000 7fbbfdffb700 1 -- 172.21.15.189:0/938765598 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fbbf4012650 con 0x7fbbf00035c0 2024-04-01T01:25:09.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.870+0000 7fbc07122700 1 -- 172.21.15.189:0/938765598 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc001a3a80 con 0x7fbbf00035c0 2024-04-01T01:25:09.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.870+0000 7fbc07122700 1 -- 172.21.15.189:0/938765598 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc001a3fe0 con 0x7fbbf00035c0 2024-04-01T01:25:09.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.871+0000 7fbbfdffb700 1 -- 172.21.15.189:0/938765598 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7fbbf40127f0 con 0x7fbbf00035c0 2024-04-01T01:25:09.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.871+0000 7fbbfdffb700 1 --2- 172.21.15.189:0/938765598 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbe4038cb0 0x7fbbe403b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:09.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.871+0000 7fbbfdffb700 1 -- 172.21.15.189:0/938765598 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fbbf4025030 con 0x7fbbf00035c0 2024-04-01T01:25:09.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.871+0000 7fbc07122700 1 -- 172.21.15.189:0/938765598 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc00102530 con 0x7fbbf00035c0 2024-04-01T01:25:09.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.871+0000 7fbbfffff700 1 --2- 172.21.15.189:0/938765598 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbe4038cb0 0x7fbbe403b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:09.875 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.872+0000 7fbbfffff700 1 --2- 172.21.15.189:0/938765598 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbe4038cb0 0x7fbbe403b170 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbbec009180 tx=0x7fbbec008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:25:09.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:09.877+0000 7fbbfdffb700 1 -- 172.21.15.189:0/938765598 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbbf401c370 con 0x7fbbf00035c0 2024-04-01T01:25:10.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:10.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:25:10.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: mon.a calling monitor election 2024-04-01T01:25:10.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:10.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:10.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:10.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:10.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: mon.c calling monitor election 2024-04-01T01:25:10.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:10.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:10.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:10.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:10.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-01T01:25:10.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: monmap epoch 2 2024-04-01T01:25:10.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: fsid 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:25:10.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: last_changed 2024-04-01T01:25:04.545194+0000 2024-04-01T01:25:10.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: created 2024-04-01T01:23:17.752525+0000 2024-04-01T01:25:10.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: min_mon_release 19 (squid) 2024-04-01T01:25:10.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: election_strategy: 1 2024-04-01T01:25:10.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-01T01:25:10.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-01T01:25:10.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: fsmap 2024-04-01T01:25:10.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: osdmap e4: 0 total, 0 up, 0 in 2024-04-01T01:25:10.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: mgrmap e13: x(active, since 51s) 2024-04-01T01:25:10.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: overall HEALTH_OK 2024-04-01T01:25:10.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:10.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:10.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:10.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-01T01:25:10.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:10.148 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:10.149+0000 7fbc07122700 1 -- 172.21.15.189:0/938765598 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fbc001a42d0 con 0x7fbbf00035c0 2024-04-01T01:25:10.149 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:10.149+0000 7fbbfdffb700 1 -- 172.21.15.189:0/938765598 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fbbf4038020 con 0x7fbbf00035c0 2024-04-01T01:25:10.149 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:25:10.149 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":2,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","modified":"2024-04-01T01:25:04.545194Z","created":"2024-04-01T01:23:17.752525Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-01T01:25:10.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:10.151+0000 7fbc07122700 1 -- 172.21.15.189:0/938765598 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbe4038cb0 msgr2=0x7fbbe403b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:10.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:10.151+0000 7fbc07122700 1 --2- 172.21.15.189:0/938765598 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbe4038cb0 0x7fbbe403b170 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbbec009180 tx=0x7fbbec008040 comp rx=0 tx=0).stop 2024-04-01T01:25:10.151 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:10.151+0000 7fbc07122700 1 -- 172.21.15.189:0/938765598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbf00035c0 msgr2=0x7fbc00102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:10.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:10.151+0000 7fbc07122700 1 --2- 172.21.15.189:0/938765598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbf00035c0 0x7fbc00102990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbbf401d000 tx=0x7fbbf400fc60 comp rx=0 tx=0).stop 2024-04-01T01:25:10.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:10.151+0000 7fbc07122700 1 -- 172.21.15.189:0/938765598 shutdown_connections 2024-04-01T01:25:10.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:10.151+0000 7fbc07122700 1 --2- 172.21.15.189:0/938765598 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbe4038cb0 0x7fbbe403b170 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:10.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:10.151+0000 7fbc07122700 1 --2- 172.21.15.189:0/938765598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbf00035c0 0x7fbc00102990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:10.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:10.151+0000 7fbc07122700 1 --2- 172.21.15.189:0/938765598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc00102ed0 0x7fbc00103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:10.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:10.151+0000 7fbc07122700 1 -- 172.21.15.189:0/938765598 >> 172.21.15.189:0/938765598 conn(0x7fbc00077330 msgr2=0x7fbc00105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:25:10.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:10.152+0000 7fbc07122700 1 -- 172.21.15.189:0/938765598 shutdown_connections 2024-04-01T01:25:10.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:10.152+0000 7fbc07122700 1 -- 172.21.15.189:0/938765598 wait complete. 2024-04-01T01:25:10.152 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 2 2024-04-01T01:25:10.742 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:10 smithi189 ceph-mon[35753]: Deploying daemon mon.b on smithi081 2024-04-01T01:25:10.742 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:10 smithi189 ceph-mon[35753]: from='client.? 172.21.15.189:0/938765598' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-01T01:25:10.742 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:10 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:11.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:10 smithi070 ceph-mon[32179]: Deploying daemon mon.b on smithi081 2024-04-01T01:25:11.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:10 smithi070 ceph-mon[32179]: from='client.? 172.21.15.189:0/938765598' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-01T01:25:11.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:10 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:11.768 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-01T01:25:11.768 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-04-01T01:25:11.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:11 smithi189 ceph-mon[35753]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:12.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:11 smithi070 ceph-mon[32179]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:12.014 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-01T01:25:13.049 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 systemd[1]: Starting Ceph mon.b for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:25:13.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.310+0000 7fe07c64e700 1 -- 172.21.15.189:0/1479614559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe074101f80 msgr2=0x7fe074102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:13.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.310+0000 7fe07c64e700 1 --2- 172.21.15.189:0/1479614559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe074101f80 0x7fe074102360 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe070009910 tx=0x7fe070009c20 comp rx=0 tx=0).stop 2024-04-01T01:25:13.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.310+0000 7fe07c64e700 1 -- 172.21.15.189:0/1479614559 shutdown_connections 2024-04-01T01:25:13.312 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.310+0000 7fe07c64e700 1 --2- 172.21.15.189:0/1479614559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe074101f80 0x7fe074102360 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:13.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.310+0000 7fe07c64e700 1 -- 172.21.15.189:0/1479614559 >> 172.21.15.189:0/1479614559 conn(0x7fe0740fd810 msgr2=0x7fe0740ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:25:13.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.310+0000 7fe07c64e700 1 -- 172.21.15.189:0/1479614559 shutdown_connections 2024-04-01T01:25:13.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.310+0000 7fe07c64e700 1 -- 172.21.15.189:0/1479614559 wait complete. 2024-04-01T01:25:13.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.311+0000 7fe07c64e700 1 Processor -- start 2024-04-01T01:25:13.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.311+0000 7fe07c64e700 1 -- start start 2024-04-01T01:25:13.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.311+0000 7fe07c64e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe074101f80 0x7fe07419d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:13.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.311+0000 7fe07c64e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe07419dbb0 0x7fe074197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:13.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.311+0000 7fe07c64e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe07410c200 con 0x7fe074101f80 2024-04-01T01:25:13.313 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.311+0000 7fe07c64e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe07410c080 con 0x7fe07419dbb0 2024-04-01T01:25:13.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.311+0000 7fe079be9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe07419dbb0 0x7fe074197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:13.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.311+0000 7fe079be9700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe07419dbb0 0x7fe074197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48290/0 (socket says 172.21.15.189:48290) 2024-04-01T01:25:13.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.311+0000 7fe079be9700 1 -- 172.21.15.189:0/619376696 learned_addr learned my addr 172.21.15.189:0/619376696 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-01T01:25:13.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.311+0000 7fe07a3ea700 1 --2- 172.21.15.189:0/619376696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe074101f80 0x7fe07419d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:13.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.311+0000 7fe079be9700 1 -- 172.21.15.189:0/619376696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe07419dbb0 msgr2=0x7fe074197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-01T01:25:13.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.311+0000 7fe079be9700 1 -- 172.21.15.189:0/619376696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe07419dbb0 msgr2=0x7fe074197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-01T01:25:13.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.311+0000 7fe079be9700 1 --2- 172.21.15.189:0/619376696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe07419dbb0 0x7fe074197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-01T01:25:13.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.312+0000 7fe079be9700 1 --2- 172.21.15.189:0/619376696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe07419dbb0 0x7fe074197740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-01T01:25:13.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.312+0000 7fe07a3ea700 1 -- 172.21.15.189:0/619376696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe07419dbb0 msgr2=0x7fe074197740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:25:13.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.312+0000 7fe07a3ea700 1 --2- 172.21.15.189:0/619376696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe07419dbb0 0x7fe074197740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:13.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.312+0000 7fe07a3ea700 1 -- 172.21.15.189:0/619376696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe07419e130 con 0x7fe074101f80 2024-04-01T01:25:13.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.312+0000 7fe07a3ea700 1 --2- 172.21.15.189:0/619376696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe074101f80 0x7fe07419d670 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe070014000 tx=0x7fe070005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:25:13.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.312+0000 7fe06b7fe700 1 -- 172.21.15.189:0/619376696 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fe070005dc0 con 0x7fe074101f80 2024-04-01T01:25:13.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.312+0000 7fe07c64e700 1 -- 172.21.15.189:0/619376696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe074197ce0 con 0x7fe074101f80 2024-04-01T01:25:13.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.312+0000 7fe06b7fe700 1 -- 172.21.15.189:0/619376696 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe07000b750 con 0x7fe074101f80 2024-04-01T01:25:13.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.312+0000 7fe06b7fe700 1 -- 172.21.15.189:0/619376696 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fe0700039a0 con 0x7fe074101f80 2024-04-01T01:25:13.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.312+0000 7fe07c64e700 1 -- 172.21.15.189:0/619376696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe074198270 con 0x7fe074101f80 2024-04-01T01:25:13.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.313+0000 7fe06b7fe700 1 -- 172.21.15.189:0/619376696 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7fe070029070 con 0x7fe074101f80 2024-04-01T01:25:13.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.313+0000 7fe07c64e700 1 -- 172.21.15.189:0/619376696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe074103460 con 0x7fe074101f80 2024-04-01T01:25:13.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.313+0000 7fe06b7fe700 1 --2- 172.21.15.189:0/619376696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe060038c40 0x7fe06003b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:13.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.313+0000 7fe06b7fe700 1 -- 172.21.15.189:0/619376696 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe07001e070 con 0x7fe074101f80 2024-04-01T01:25:13.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.313+0000 7fe079be9700 1 --2- 172.21.15.189:0/619376696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe060038c40 0x7fe06003b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:13.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.314+0000 7fe079be9700 1 --2- 172.21.15.189:0/619376696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe060038c40 0x7fe06003b100 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe064009180 tx=0x7fe064008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:25:13.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.319+0000 7fe06b7fe700 1 -- 172.21.15.189:0/619376696 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe070013070 con 0x7fe074101f80 2024-04-01T01:25:13.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 podman[35124]: 2024-04-01 01:25:13.248185869 +0000 UTC m=+0.117934972 container create be19c90532f555e6f26e81b53c206f5ac644a169cb29f88b084bc56b20d90531 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-01T01:25:13.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 podman[35124]: 2024-04-01 01:25:13.151715291 +0000 UTC m=+0.021464411 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:25:13.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.508+0000 7fe06b7fe700 1 -- 172.21.15.189:0/619376696 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07002ae20 con 0x7fe074101f80 2024-04-01T01:25:13.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:13.594+0000 7fe07c64e700 1 -- 172.21.15.189:0/619376696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fe0741985e0 con 0x7fe074101f80 2024-04-01T01:25:13.623 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 podman[35124]: 2024-04-01 01:25:13.44237045 +0000 UTC m=+0.312119563 container init be19c90532f555e6f26e81b53c206f5ac644a169cb29f88b084bc56b20d90531 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-b, io.buildah.version=1.33.5, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, ceph=True) 2024-04-01T01:25:13.623 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 podman[35124]: 2024-04-01 01:25:13.445547622 +0000 UTC m=+0.315296725 container start be19c90532f555e6f26e81b53c206f5ac644a169cb29f88b084bc56b20d90531 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-01T01:25:13.624 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: set uid:gid to 167:167 (ceph:ceph) 2024-04-01T01:25:13.624 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev), process ceph-mon, pid 2 2024-04-01T01:25:13.624 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: pidfile_write: ignore empty --pid-file 2024-04-01T01:25:13.624 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: load: jerasure load: lrc 2024-04-01T01:25:13.624 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: RocksDB version: 7.9.2 2024-04-01T01:25:13.624 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Git sha 0 2024-04-01T01:25:13.624 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Compile date 2024-03-31 16:17:26 2024-04-01T01:25:13.624 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: DB SUMMARY 2024-04-01T01:25:13.624 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: DB Session ID: STK2P1U0SCWA2DVDLH10 2024-04-01T01:25:13.624 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: CURRENT file: CURRENT 2024-04-01T01:25:13.624 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: IDENTITY file: IDENTITY 2024-04-01T01:25:13.624 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-01T01:25:13.625 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-01T01:25:13.625 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-01T01:25:13.625 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.error_if_exists: 0 2024-04-01T01:25:13.625 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.create_if_missing: 0 2024-04-01T01:25:13.625 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.paranoid_checks: 1 2024-04-01T01:25:13.625 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-01T01:25:13.625 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-01T01:25:13.625 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-01T01:25:13.625 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.env: 0x5617e8495c60 2024-04-01T01:25:13.625 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.fs: PosixFileSystem 2024-04-01T01:25:13.625 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.info_log: 0x5617eac21ae0 2024-04-01T01:25:13.625 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_file_opening_threads: 16 2024-04-01T01:25:13.626 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.statistics: (nil) 2024-04-01T01:25:13.626 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.use_fsync: 0 2024-04-01T01:25:13.626 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_log_file_size: 0 2024-04-01T01:25:13.627 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-01T01:25:13.627 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-01T01:25:13.627 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.keep_log_file_num: 1000 2024-04-01T01:25:13.627 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.recycle_log_file_num: 0 2024-04-01T01:25:13.627 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.allow_fallocate: 1 2024-04-01T01:25:13.627 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.allow_mmap_reads: 0 2024-04-01T01:25:13.627 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.allow_mmap_writes: 0 2024-04-01T01:25:13.627 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.use_direct_reads: 0 2024-04-01T01:25:13.627 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-01T01:25:13.627 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.create_missing_column_families: 0 2024-04-01T01:25:13.627 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.db_log_dir: 2024-04-01T01:25:13.627 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.wal_dir: 2024-04-01T01:25:13.628 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-01T01:25:13.628 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-01T01:25:13.628 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-01T01:25:13.628 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-01T01:25:13.628 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-01T01:25:13.628 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-01T01:25:13.628 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.advise_random_on_open: 1 2024-04-01T01:25:13.628 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.db_write_buffer_size: 0 2024-04-01T01:25:13.628 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.write_buffer_manager: 0x5617eabf41e0 2024-04-01T01:25:13.628 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-01T01:25:13.628 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-01T01:25:13.628 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-01T01:25:13.628 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.rate_limiter: (nil) 2024-04-01T01:25:13.628 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-01T01:25:13.629 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.wal_recovery_mode: 2 2024-04-01T01:25:13.629 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.enable_thread_tracking: 0 2024-04-01T01:25:13.629 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.enable_pipelined_write: 0 2024-04-01T01:25:13.629 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.unordered_write: 0 2024-04-01T01:25:13.629 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-01T01:25:13.629 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-01T01:25:13.629 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-01T01:25:13.629 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-01T01:25:13.629 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.row_cache: None 2024-04-01T01:25:13.629 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.wal_filter: None 2024-04-01T01:25:13.629 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-01T01:25:13.629 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.allow_ingest_behind: 0 2024-04-01T01:25:13.629 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.two_write_queues: 0 2024-04-01T01:25:13.630 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.manual_wal_flush: 0 2024-04-01T01:25:13.630 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.wal_compression: 0 2024-04-01T01:25:13.630 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.atomic_flush: 0 2024-04-01T01:25:13.630 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-01T01:25:13.630 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-01T01:25:13.630 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-01T01:25:13.630 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.log_readahead_size: 0 2024-04-01T01:25:13.630 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-01T01:25:13.630 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.best_efforts_recovery: 0 2024-04-01T01:25:13.630 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-01T01:25:13.630 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-01T01:25:13.630 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.allow_data_in_errors: 0 2024-04-01T01:25:13.631 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.db_host_id: __hostname__ 2024-04-01T01:25:13.631 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-01T01:25:13.631 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_background_jobs: 2 2024-04-01T01:25:13.631 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_background_compactions: -1 2024-04-01T01:25:13.631 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_subcompactions: 1 2024-04-01T01:25:13.631 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-01T01:25:13.631 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-01T01:25:13.631 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-01T01:25:13.631 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_total_wal_size: 0 2024-04-01T01:25:13.631 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-01T01:25:13.631 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-01T01:25:13.631 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-01T01:25:13.631 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-01T01:25:13.631 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_open_files: -1 2024-04-01T01:25:13.631 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.bytes_per_sync: 0 2024-04-01T01:25:13.632 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-01T01:25:13.632 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-01T01:25:13.632 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compaction_readahead_size: 0 2024-04-01T01:25:13.632 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_background_flushes: -1 2024-04-01T01:25:13.632 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Compression algorithms supported: 2024-04-01T01:25:13.632 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-01T01:25:13.632 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: kZSTD supported: 0 2024-04-01T01:25:13.632 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: kXpressCompression supported: 0 2024-04-01T01:25:13.632 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: kLZ4HCCompression supported: 1 2024-04-01T01:25:13.632 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: kZlibCompression supported: 1 2024-04-01T01:25:13.632 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: kSnappyCompression supported: 1 2024-04-01T01:25:13.632 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: kLZ4Compression supported: 1 2024-04-01T01:25:13.632 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: kBZip2Compression supported: 0 2024-04-01T01:25:13.632 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-01T01:25:13.632 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-01T01:25:13.633 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-01T01:25:13.633 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-01T01:25:13.633 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-01T01:25:13.633 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.merge_operator: 2024-04-01T01:25:13.633 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compaction_filter: None 2024-04-01T01:25:13.633 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compaction_filter_factory: None 2024-04-01T01:25:13.633 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.sst_partitioner_factory: None 2024-04-01T01:25:13.633 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-01T01:25:13.633 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-01T01:25:13.633 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5617eac21be0) 2024-04-01T01:25:13.634 INFO:journalctl@ceph.mon.b.smithi081.stdout: cache_index_and_filter_blocks: 1 2024-04-01T01:25:13.634 INFO:journalctl@ceph.mon.b.smithi081.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-01T01:25:13.634 INFO:journalctl@ceph.mon.b.smithi081.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-01T01:25:13.634 INFO:journalctl@ceph.mon.b.smithi081.stdout: pin_top_level_index_and_filter: 1 2024-04-01T01:25:13.634 INFO:journalctl@ceph.mon.b.smithi081.stdout: index_type: 0 2024-04-01T01:25:13.634 INFO:journalctl@ceph.mon.b.smithi081.stdout: data_block_index_type: 0 2024-04-01T01:25:13.634 INFO:journalctl@ceph.mon.b.smithi081.stdout: index_shortening: 1 2024-04-01T01:25:13.634 INFO:journalctl@ceph.mon.b.smithi081.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-01T01:25:13.634 INFO:journalctl@ceph.mon.b.smithi081.stdout: checksum: 4 2024-04-01T01:25:13.634 INFO:journalctl@ceph.mon.b.smithi081.stdout: no_block_cache: 0 2024-04-01T01:25:13.634 INFO:journalctl@ceph.mon.b.smithi081.stdout: block_cache: 0x5617eac25090 2024-04-01T01:25:13.635 INFO:journalctl@ceph.mon.b.smithi081.stdout: block_cache_name: BinnedLRUCache 2024-04-01T01:25:13.635 INFO:journalctl@ceph.mon.b.smithi081.stdout: block_cache_options: 2024-04-01T01:25:13.635 INFO:journalctl@ceph.mon.b.smithi081.stdout: capacity : 536870912 2024-04-01T01:25:13.635 INFO:journalctl@ceph.mon.b.smithi081.stdout: num_shard_bits : 4 2024-04-01T01:25:13.635 INFO:journalctl@ceph.mon.b.smithi081.stdout: strict_capacity_limit : 0 2024-04-01T01:25:13.635 INFO:journalctl@ceph.mon.b.smithi081.stdout: high_pri_pool_ratio: 0.000 2024-04-01T01:25:13.635 INFO:journalctl@ceph.mon.b.smithi081.stdout: block_cache_compressed: (nil) 2024-04-01T01:25:13.635 INFO:journalctl@ceph.mon.b.smithi081.stdout: persistent_cache: (nil) 2024-04-01T01:25:13.635 INFO:journalctl@ceph.mon.b.smithi081.stdout: block_size: 4096 2024-04-01T01:25:13.635 INFO:journalctl@ceph.mon.b.smithi081.stdout: block_size_deviation: 10 2024-04-01T01:25:13.635 INFO:journalctl@ceph.mon.b.smithi081.stdout: block_restart_interval: 16 2024-04-01T01:25:13.635 INFO:journalctl@ceph.mon.b.smithi081.stdout: index_block_restart_interval: 1 2024-04-01T01:25:13.635 INFO:journalctl@ceph.mon.b.smithi081.stdout: metadata_block_size: 4096 2024-04-01T01:25:13.635 INFO:journalctl@ceph.mon.b.smithi081.stdout: partition_filters: 0 2024-04-01T01:25:13.636 INFO:journalctl@ceph.mon.b.smithi081.stdout: use_delta_encoding: 1 2024-04-01T01:25:13.636 INFO:journalctl@ceph.mon.b.smithi081.stdout: filter_policy: bloomfilter 2024-04-01T01:25:13.636 INFO:journalctl@ceph.mon.b.smithi081.stdout: whole_key_filtering: 1 2024-04-01T01:25:13.636 INFO:journalctl@ceph.mon.b.smithi081.stdout: verify_compression: 0 2024-04-01T01:25:13.636 INFO:journalctl@ceph.mon.b.smithi081.stdout: read_amp_bytes_per_bit: 0 2024-04-01T01:25:13.636 INFO:journalctl@ceph.mon.b.smithi081.stdout: format_version: 5 2024-04-01T01:25:13.636 INFO:journalctl@ceph.mon.b.smithi081.stdout: enable_index_compression: 1 2024-04-01T01:25:13.636 INFO:journalctl@ceph.mon.b.smithi081.stdout: block_align: 0 2024-04-01T01:25:13.636 INFO:journalctl@ceph.mon.b.smithi081.stdout: max_auto_readahead_size: 262144 2024-04-01T01:25:13.636 INFO:journalctl@ceph.mon.b.smithi081.stdout: prepopulate_block_cache: 0 2024-04-01T01:25:13.636 INFO:journalctl@ceph.mon.b.smithi081.stdout: initial_auto_readahead_size: 8192 2024-04-01T01:25:13.636 INFO:journalctl@ceph.mon.b.smithi081.stdout: num_file_reads_for_auto_readahead: 2 2024-04-01T01:25:13.636 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.write_buffer_size: 33554432 2024-04-01T01:25:13.636 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_write_buffer_number: 2 2024-04-01T01:25:13.636 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compression: NoCompression 2024-04-01T01:25:13.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.bottommost_compression: Disabled 2024-04-01T01:25:13.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.prefix_extractor: nullptr 2024-04-01T01:25:13.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-01T01:25:13.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.num_levels: 7 2024-04-01T01:25:13.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-01T01:25:13.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-01T01:25:13.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-01T01:25:13.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-01T01:25:13.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-01T01:25:13.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-01T01:25:13.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-01T01:25:13.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-01T01:25:13.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-01T01:25:13.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-01T01:25:13.638 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-01T01:25:13.638 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-01T01:25:13.638 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-01T01:25:13.638 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compression_opts.level: 32767 2024-04-01T01:25:13.638 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compression_opts.strategy: 0 2024-04-01T01:25:13.638 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-01T01:25:13.638 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-01T01:25:13.638 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-01T01:25:13.638 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-01T01:25:13.638 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compression_opts.enabled: false 2024-04-01T01:25:13.638 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-01T01:25:13.638 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-01T01:25:13.638 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-01T01:25:13.638 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-01T01:25:13.638 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.target_file_size_base: 67108864 2024-04-01T01:25:13.639 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-01T01:25:13.639 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-01T01:25:13.639 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-01T01:25:13.639 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-01T01:25:13.639 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-01T01:25:13.639 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-01T01:25:13.640 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-01T01:25:13.640 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-01T01:25:13.640 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-01T01:25:13.640 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-01T01:25:13.640 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-01T01:25:13.641 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-01T01:25:13.641 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-01T01:25:13.641 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-01T01:25:13.641 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.arena_block_size: 1048576 2024-04-01T01:25:13.641 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-01T01:25:13.641 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-01T01:25:13.641 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.disable_auto_compactions: 0 2024-04-01T01:25:13.641 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-01T01:25:13.641 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-01T01:25:13.641 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-01T01:25:13.641 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-01T01:25:13.641 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-01T01:25:13.642 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-01T01:25:13.642 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-01T01:25:13.642 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-01T01:25:13.642 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-01T01:25:13.642 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-01T01:25:13.642 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.table_properties_collectors: 2024-04-01T01:25:13.642 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.inplace_update_support: 0 2024-04-01T01:25:13.642 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-01T01:25:13.642 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-01T01:25:13.642 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-01T01:25:13.642 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-01T01:25:13.642 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.bloom_locality: 0 2024-04-01T01:25:13.642 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.max_successive_merges: 0 2024-04-01T01:25:13.643 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-01T01:25:13.643 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.paranoid_file_checks: 0 2024-04-01T01:25:13.643 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.force_consistency_checks: 1 2024-04-01T01:25:13.643 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.report_bg_io_stats: 0 2024-04-01T01:25:13.643 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.ttl: 2592000 2024-04-01T01:25:13.643 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-01T01:25:13.643 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-01T01:25:13.643 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-01T01:25:13.643 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.enable_blob_files: false 2024-04-01T01:25:13.643 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.min_blob_size: 0 2024-04-01T01:25:13.643 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.blob_file_size: 268435456 2024-04-01T01:25:13.643 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-01T01:25:13.643 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-01T01:25:13.644 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-01T01:25:13.644 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-01T01:25:13.644 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-01T01:25:13.644 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.blob_file_starting_level: 0 2024-04-01T01:25:13.644 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-01T01:25:13.644 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-01T01:25:13.644 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-01T01:25:13.644 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: de7dfe19-5956-4f3d-9169-1e6e871efd69 2024-04-01T01:25:13.644 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711934713486302, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-01T01:25:13.644 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-01T01:25:13.644 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711934713486740, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711934713, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "de7dfe19-5956-4f3d-9169-1e6e871efd69", "db_session_id": "STK2P1U0SCWA2DVDLH10", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-01T01:25:13.644 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711934713486819, "job": 1, "event": "recovery_finished"} 2024-04-01T01:25:13.644 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-01T01:25:13.644 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-01T01:25:13.645 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5617ead16000 2024-04-01T01:25:13.645 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: DB pointer 0x5617ead00000 2024-04-01T01:25:13.645 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-01T01:25:13.645 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-01T01:25:13.645 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-01T01:25:13.645 INFO:journalctl@ceph.mon.b.smithi081.stdout: ** DB Stats ** 2024-04-01T01:25:13.645 INFO:journalctl@ceph.mon.b.smithi081.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-01T01:25:13.645 INFO:journalctl@ceph.mon.b.smithi081.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-01T01:25:13.645 INFO:journalctl@ceph.mon.b.smithi081.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-01T01:25:13.645 INFO:journalctl@ceph.mon.b.smithi081.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-01T01:25:13.645 INFO:journalctl@ceph.mon.b.smithi081.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-01T01:25:13.645 INFO:journalctl@ceph.mon.b.smithi081.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-01T01:25:13.645 INFO:journalctl@ceph.mon.b.smithi081.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-01T01:25:13.645 INFO:journalctl@ceph.mon.b.smithi081.stdout: 2024-04-01T01:25:13.645 INFO:journalctl@ceph.mon.b.smithi081.stdout: ** Compaction Stats [default] ** 2024-04-01T01:25:13.646 INFO:journalctl@ceph.mon.b.smithi081.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-01T01:25:13.646 INFO:journalctl@ceph.mon.b.smithi081.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-01T01:25:13.646 INFO:journalctl@ceph.mon.b.smithi081.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-01T01:25:13.646 INFO:journalctl@ceph.mon.b.smithi081.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-01T01:25:13.646 INFO:journalctl@ceph.mon.b.smithi081.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-01T01:25:13.646 INFO:journalctl@ceph.mon.b.smithi081.stdout: 2024-04-01T01:25:13.646 INFO:journalctl@ceph.mon.b.smithi081.stdout: ** Compaction Stats [default] ** 2024-04-01T01:25:13.646 INFO:journalctl@ceph.mon.b.smithi081.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-01T01:25:13.646 INFO:journalctl@ceph.mon.b.smithi081.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-01T01:25:13.646 INFO:journalctl@ceph.mon.b.smithi081.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-01T01:25:13.646 INFO:journalctl@ceph.mon.b.smithi081.stdout: 2024-04-01T01:25:13.646 INFO:journalctl@ceph.mon.b.smithi081.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-01T01:25:13.647 INFO:journalctl@ceph.mon.b.smithi081.stdout: 2024-04-01T01:25:13.647 INFO:journalctl@ceph.mon.b.smithi081.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-01T01:25:13.647 INFO:journalctl@ceph.mon.b.smithi081.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-01T01:25:13.647 INFO:journalctl@ceph.mon.b.smithi081.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-01T01:25:13.647 INFO:journalctl@ceph.mon.b.smithi081.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-01T01:25:13.647 INFO:journalctl@ceph.mon.b.smithi081.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-01T01:25:13.647 INFO:journalctl@ceph.mon.b.smithi081.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-01T01:25:13.647 INFO:journalctl@ceph.mon.b.smithi081.stdout: Cumulative compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-01T01:25:13.647 INFO:journalctl@ceph.mon.b.smithi081.stdout: Interval compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-01T01:25:13.647 INFO:journalctl@ceph.mon.b.smithi081.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-01T01:25:13.647 INFO:journalctl@ceph.mon.b.smithi081.stdout: Block cache BinnedLRUCache@0x5617eac25090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-01T01:25:13.647 INFO:journalctl@ceph.mon.b.smithi081.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-01T01:25:13.647 INFO:journalctl@ceph.mon.b.smithi081.stdout: 2024-04-01T01:25:13.648 INFO:journalctl@ceph.mon.b.smithi081.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-01T01:25:13.648 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: using public_addr v2:172.21.15.81:0/0 -> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] 2024-04-01T01:25:13.648 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: starting mon.b rank -1 at public addrs [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] at bind addrs [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:25:13.648 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.b@-1(???) e0 preinit fsid 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:25:13.648 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.b@-1(synchronizing).mds e1 new map 2024-04-01T01:25:13.648 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-01T01:25:13.648 INFO:journalctl@ceph.mon.b.smithi081.stdout: e1 2024-04-01T01:25:13.648 INFO:journalctl@ceph.mon.b.smithi081.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-01T01:25:13.648 INFO:journalctl@ceph.mon.b.smithi081.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-01T01:25:13.648 INFO:journalctl@ceph.mon.b.smithi081.stdout: legacy client fscid: -1 2024-04-01T01:25:13.649 INFO:journalctl@ceph.mon.b.smithi081.stdout: 2024-04-01T01:25:13.649 INFO:journalctl@ceph.mon.b.smithi081.stdout: No filesystems configured 2024-04-01T01:25:13.649 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-01T01:25:13.649 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-01T01:25:13.649 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-01T01:25:13.649 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-01T01:25:13.649 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-01T01:25:13.649 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-01T01:25:13.649 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-01T01:25:13.649 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-01T01:25:13.649 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-01T01:25:13.649 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-01T01:25:13.649 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:25:13.649 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:13.649 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:13.650 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.650 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.650 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.650 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.650 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi081", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:25:13.650 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:13.650 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:13.650 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.650 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.650 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.650 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: Updating smithi081:/etc/ceph/ceph.conf 2024-04-01T01:25:13.650 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:25:13.650 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: Updating smithi081:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-01T01:25:13.650 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: Updating smithi081:/etc/ceph/ceph.client.admin.keyring 2024-04-01T01:25:13.651 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: Updating smithi081:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-01T01:25:13.651 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: Deploying cephadm binary to smithi189 2024-04-01T01:25:13.651 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:13.651 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:13.651 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.651 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:13.651 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: Added host smithi189 2024-04-01T01:25:13.651 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.651 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.651 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:13.651 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.651 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:25:13.651 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:13.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3109714928' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-01T01:25:13.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:25:13.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:13.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:13.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-01T01:25:13.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:13.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: Updating smithi189:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-01T01:25:13.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3109714928' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-01T01:25:13.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: osdmap e4: 0 total, 0 up, 0 in 2024-04-01T01:25:13.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.653 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.653 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-04-01T01:25:13.653 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: Updating smithi189:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-04-01T01:25:13.653 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:13.653 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.653 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:13.653 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:13.653 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:13.653 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.653 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-01T01:25:13.653 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:13.653 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi070:172.21.15.70=a;smithi081:172.21.15.81=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:25:13.653 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: Saving service mon spec with placement smithi070:172.21.15.70=a;smithi081:172.21.15.81=b;smithi189:172.21.15.189=c;count:3 2024-04-01T01:25:13.653 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: Deploying daemon mon.c on smithi189 2024-04-01T01:25:13.653 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:13.653 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='client.? 172.21.15.189:0/1407220077' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-01T01:25:13.654 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:13.654 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:13.654 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:25:13.654 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.a calling monitor election 2024-04-01T01:25:13.654 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:13.654 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:13.654 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:13.654 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:13.654 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.c calling monitor election 2024-04-01T01:25:13.654 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:13.654 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:13.654 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:13.654 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:13.655 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-01T01:25:13.655 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: monmap epoch 2 2024-04-01T01:25:13.655 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: fsid 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:25:13.655 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: last_changed 2024-04-01T01:25:04.545194+0000 2024-04-01T01:25:13.655 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: created 2024-04-01T01:23:17.752525+0000 2024-04-01T01:25:13.655 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: min_mon_release 19 (squid) 2024-04-01T01:25:13.655 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: election_strategy: 1 2024-04-01T01:25:13.655 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-01T01:25:13.655 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-01T01:25:13.655 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: fsmap 2024-04-01T01:25:13.655 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: osdmap e4: 0 total, 0 up, 0 in 2024-04-01T01:25:13.655 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mgrmap e13: x(active, since 51s) 2024-04-01T01:25:13.655 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: overall HEALTH_OK 2024-04-01T01:25:13.655 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.655 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.656 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:13.656 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-01T01:25:13.656 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:13.656 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: Deploying daemon mon.b on smithi081 2024-04-01T01:25:13.656 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='client.? 172.21.15.189:0/938765598' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-01T01:25:13.656 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:13.656 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:13.656 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 ceph-mon[35142]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-01T01:25:13.656 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 bash[35124]: be19c90532f555e6f26e81b53c206f5ac644a169cb29f88b084bc56b20d90531 2024-04-01T01:25:13.656 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:13 smithi081 systemd[1]: Started Ceph mon.b for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:25:18.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:18.517+0000 7fe06b7fe700 1 -- 172.21.15.189:0/619376696 <== mon.0 v2:172.21.15.70:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7fe070030090 con 0x7fe074101f80 2024-04-01T01:25:18.518 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-01T01:25:18.518 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":3,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","modified":"2024-04-01T01:25:13.508440Z","created":"2024-04-01T01:23:17.752525Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-01T01:25:18.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:18.520+0000 7fe07c64e700 1 -- 172.21.15.189:0/619376696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe060038c40 msgr2=0x7fe06003b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:18.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:18.520+0000 7fe07c64e700 1 --2- 172.21.15.189:0/619376696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe060038c40 0x7fe06003b100 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe064009180 tx=0x7fe064008040 comp rx=0 tx=0).stop 2024-04-01T01:25:18.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:18.520+0000 7fe07c64e700 1 -- 172.21.15.189:0/619376696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe074101f80 msgr2=0x7fe07419d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:18.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:18.520+0000 7fe07c64e700 1 --2- 172.21.15.189:0/619376696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe074101f80 0x7fe07419d670 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe070014000 tx=0x7fe070005030 comp rx=0 tx=0).stop 2024-04-01T01:25:18.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:18.520+0000 7fe07c64e700 1 -- 172.21.15.189:0/619376696 shutdown_connections 2024-04-01T01:25:18.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:18.520+0000 7fe07c64e700 1 --2- 172.21.15.189:0/619376696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe060038c40 0x7fe06003b100 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:18.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:18.520+0000 7fe07c64e700 1 --2- 172.21.15.189:0/619376696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe074101f80 0x7fe07419d670 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:18.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:18.520+0000 7fe07c64e700 1 --2- 172.21.15.189:0/619376696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe07419dbb0 0x7fe074197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:18.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:18.520+0000 7fe07c64e700 1 -- 172.21.15.189:0/619376696 >> 172.21.15.189:0/619376696 conn(0x7fe0740fd810 msgr2=0x7fe074103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:25:18.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:18.520+0000 7fe07c64e700 1 -- 172.21.15.189:0/619376696 shutdown_connections 2024-04-01T01:25:18.521 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:18.520+0000 7fe07c64e700 1 -- 172.21.15.189:0/619376696 wait complete. 2024-04-01T01:25:18.521 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 3 2024-04-01T01:25:18.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:25:18.837 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: mon.c calling monitor election 2024-04-01T01:25:18.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: mon.a calling monitor election 2024-04-01T01:25:18.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:18.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:18.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: from='client.? 172.21.15.189:0/619376696' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-01T01:25:18.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:18.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:18.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:18.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:18.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:18.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:18.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:18.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-01T01:25:18.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: monmap epoch 3 2024-04-01T01:25:18.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: fsid 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:25:18.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: last_changed 2024-04-01T01:25:13.508440+0000 2024-04-01T01:25:18.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: created 2024-04-01T01:23:17.752525+0000 2024-04-01T01:25:18.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: min_mon_release 19 (squid) 2024-04-01T01:25:18.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: election_strategy: 1 2024-04-01T01:25:18.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-01T01:25:18.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-01T01:25:18.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: 2: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.b 2024-04-01T01:25:18.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: fsmap 2024-04-01T01:25:18.840 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: osdmap e4: 0 total, 0 up, 0 in 2024-04-01T01:25:18.840 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: mgrmap e13: x(active, since 60s) 2024-04-01T01:25:18.840 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: overall HEALTH_OK 2024-04-01T01:25:18.840 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:18.840 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:18.840 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:18.840 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:19.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:25:19.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: mon.c calling monitor election 2024-04-01T01:25:19.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: mon.a calling monitor election 2024-04-01T01:25:19.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:19.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:19.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: from='client.? 172.21.15.189:0/619376696' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-01T01:25:19.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:19.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:19.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:19.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:19.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:19.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:19.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:19.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-01T01:25:19.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: monmap epoch 3 2024-04-01T01:25:19.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: fsid 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:25:19.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: last_changed 2024-04-01T01:25:13.508440+0000 2024-04-01T01:25:19.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: created 2024-04-01T01:23:17.752525+0000 2024-04-01T01:25:19.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: min_mon_release 19 (squid) 2024-04-01T01:25:19.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: election_strategy: 1 2024-04-01T01:25:19.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-01T01:25:19.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-01T01:25:19.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: 2: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.b 2024-04-01T01:25:19.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: fsmap 2024-04-01T01:25:19.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: osdmap e4: 0 total, 0 up, 0 in 2024-04-01T01:25:19.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: mgrmap e13: x(active, since 60s) 2024-04-01T01:25:19.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: overall HEALTH_OK 2024-04-01T01:25:19.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:19.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:19.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:19.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:19.175 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-01T01:25:19.175 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-04-01T01:25:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:25:19.844 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:19 smithi070 ceph-mon[32179]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:19.844 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:19.844 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:19 smithi189 ceph-mon[35753]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:20.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:25:20.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: mon.c calling monitor election 2024-04-01T01:25:20.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: mon.a calling monitor election 2024-04-01T01:25:20.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:20.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:25:20.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: from='client.? 172.21.15.189:0/619376696' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-01T01:25:20.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:20.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:20.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:20.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:20.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:20.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:20.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:20.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-01T01:25:20.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: monmap epoch 3 2024-04-01T01:25:20.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: fsid 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:25:20.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: last_changed 2024-04-01T01:25:13.508440+0000 2024-04-01T01:25:20.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: created 2024-04-01T01:23:17.752525+0000 2024-04-01T01:25:20.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: min_mon_release 19 (squid) 2024-04-01T01:25:20.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: election_strategy: 1 2024-04-01T01:25:20.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-01T01:25:20.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-01T01:25:20.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: 2: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.b 2024-04-01T01:25:20.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: fsmap 2024-04-01T01:25:20.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: osdmap e4: 0 total, 0 up, 0 in 2024-04-01T01:25:20.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: mgrmap e13: x(active, since 60s) 2024-04-01T01:25:20.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: overall HEALTH_OK 2024-04-01T01:25:20.831 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:20.831 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:20.831 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:20.831 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:20.831 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:20.831 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:20.831 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:21.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.063+0000 7f65e62b8700 1 -- 172.21.15.70:0/2611086106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e0102dc0 msgr2=0x7f65e01031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:21.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.063+0000 7f65e62b8700 1 --2- 172.21.15.70:0/2611086106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e0102dc0 0x7f65e01031a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f65d0009910 tx=0x7f65d0009c20 comp rx=0 tx=0).stop 2024-04-01T01:25:21.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.063+0000 7f65e62b8700 1 -- 172.21.15.70:0/2611086106 shutdown_connections 2024-04-01T01:25:21.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.063+0000 7f65e62b8700 1 --2- 172.21.15.70:0/2611086106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e0102dc0 0x7f65e01031a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:21.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.063+0000 7f65e62b8700 1 -- 172.21.15.70:0/2611086106 >> 172.21.15.70:0/2611086106 conn(0x7f65e00fc5f0 msgr2=0x7f65e00fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:25:21.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.063+0000 7f65e62b8700 1 -- 172.21.15.70:0/2611086106 shutdown_connections 2024-04-01T01:25:21.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.063+0000 7f65e62b8700 1 -- 172.21.15.70:0/2611086106 wait complete. 2024-04-01T01:25:21.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.064+0000 7f65e62b8700 1 Processor -- start 2024-04-01T01:25:21.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.064+0000 7f65e62b8700 1 -- start start 2024-04-01T01:25:21.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65e62b8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e0102dc0 0x7f65e0100a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:21.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65e62b8700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65e0100f60 0x7f65e01013e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:21.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65e62b8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65e01a3b00 0x7f65e01a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:21.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65e62b8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65e010c160 con 0x7f65e0102dc0 2024-04-01T01:25:21.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65e62b8700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f65e010bfe0 con 0x7f65e0100f60 2024-04-01T01:25:21.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65e62b8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f65e010c2e0 con 0x7f65e01a3b00 2024-04-01T01:25:21.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65dffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e0102dc0 0x7f65e0100a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:21.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65dffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e0102dc0 0x7f65e0100a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54224/0 (socket says 172.21.15.70:54224) 2024-04-01T01:25:21.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65dffff700 1 -- 172.21.15.70:0/4062811514 learned_addr learned my addr 172.21.15.70:0/4062811514 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:25:21.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65df7fe700 1 --2- 172.21.15.70:0/4062811514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65e0100f60 0x7f65e01013e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:21.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65dffff700 1 -- 172.21.15.70:0/4062811514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65e01a3b00 msgr2=0x7f65e01a5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:21.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65dffff700 1 --2- 172.21.15.70:0/4062811514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65e01a3b00 0x7f65e01a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:21.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65dffff700 1 -- 172.21.15.70:0/4062811514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65e0100f60 msgr2=0x7f65e01013e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:21.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65dffff700 1 --2- 172.21.15.70:0/4062811514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65e0100f60 0x7f65e01013e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:21.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65dffff700 1 -- 172.21.15.70:0/4062811514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e01a6620 con 0x7f65e0102dc0 2024-04-01T01:25:21.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65df7fe700 1 --2- 172.21.15.70:0/4062811514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65e0100f60 0x7f65e01013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:25:21.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65dffff700 1 --2- 172.21.15.70:0/4062811514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e0102dc0 0x7f65e0100a20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f65d0005b40 tx=0x7f65d0004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:25:21.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65dd7fa700 1 -- 172.21.15.70:0/4062811514 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d0027070 con 0x7f65e0102dc0 2024-04-01T01:25:21.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65dd7fa700 1 -- 172.21.15.70:0/4062811514 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f65d000be10 con 0x7f65e0102dc0 2024-04-01T01:25:21.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65e62b8700 1 -- 172.21.15.70:0/4062811514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65e01a68b0 con 0x7f65e0102dc0 2024-04-01T01:25:21.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.065+0000 7f65dd7fa700 1 -- 172.21.15.70:0/4062811514 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d0022e60 con 0x7f65e0102dc0 2024-04-01T01:25:21.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.066+0000 7f65e62b8700 1 -- 172.21.15.70:0/4062811514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65e01a6de0 con 0x7f65e0102dc0 2024-04-01T01:25:21.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.066+0000 7f65e62b8700 1 -- 172.21.15.70:0/4062811514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65e0063870 con 0x7f65e0102dc0 2024-04-01T01:25:21.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.067+0000 7f65dd7fa700 1 -- 172.21.15.70:0/4062811514 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7f65d002a070 con 0x7f65e0102dc0 2024-04-01T01:25:21.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.067+0000 7f65dd7fa700 1 --2- 172.21.15.70:0/4062811514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65c8039330 0x7f65c803b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:21.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.067+0000 7f65dd7fa700 1 -- 172.21.15.70:0/4062811514 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f65d004cf70 con 0x7f65e0102dc0 2024-04-01T01:25:21.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.067+0000 7f65df7fe700 1 --2- 172.21.15.70:0/4062811514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65c8039330 0x7f65c803b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:21.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.067+0000 7f65df7fe700 1 --2- 172.21.15.70:0/4062811514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65c8039330 0x7f65c803b7f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f65d40079b0 tx=0x7f65d4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:25:21.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.072+0000 7f65dd7fa700 1 -- 172.21.15.70:0/4062811514 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f65d0013070 con 0x7f65e0102dc0 2024-04-01T01:25:21.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.296+0000 7f65e62b8700 1 -- 172.21.15.70:0/4062811514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f65e01a70e0 con 0x7f65e0102dc0 2024-04-01T01:25:21.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.297+0000 7f65dd7fa700 1 -- 172.21.15.70:0/4062811514 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f65d000e3e0 con 0x7f65e0102dc0 2024-04-01T01:25:21.297 INFO:teuthology.orchestra.run.smithi070.stdout:# minimal ceph.conf for 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:25:21.298 INFO:teuthology.orchestra.run.smithi070.stdout:[global] 2024-04-01T01:25:21.298 INFO:teuthology.orchestra.run.smithi070.stdout: fsid = 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:25:21.298 INFO:teuthology.orchestra.run.smithi070.stdout: mon_host = [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-01T01:25:21.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.300+0000 7f65e62b8700 1 -- 172.21.15.70:0/4062811514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65c8039330 msgr2=0x7f65c803b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:21.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.300+0000 7f65e62b8700 1 --2- 172.21.15.70:0/4062811514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65c8039330 0x7f65c803b7f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f65d40079b0 tx=0x7f65d4008040 comp rx=0 tx=0).stop 2024-04-01T01:25:21.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.300+0000 7f65e62b8700 1 -- 172.21.15.70:0/4062811514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e0102dc0 msgr2=0x7f65e0100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:21.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.300+0000 7f65e62b8700 1 --2- 172.21.15.70:0/4062811514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e0102dc0 0x7f65e0100a20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f65d0005b40 tx=0x7f65d0004f80 comp rx=0 tx=0).stop 2024-04-01T01:25:21.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.300+0000 7f65e62b8700 1 -- 172.21.15.70:0/4062811514 shutdown_connections 2024-04-01T01:25:21.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.300+0000 7f65e62b8700 1 --2- 172.21.15.70:0/4062811514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65c8039330 0x7f65c803b7f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:21.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.300+0000 7f65e62b8700 1 --2- 172.21.15.70:0/4062811514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65e01a3b00 0x7f65e01a5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:21.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.300+0000 7f65e62b8700 1 --2- 172.21.15.70:0/4062811514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e0102dc0 0x7f65e0100a20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:21.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.300+0000 7f65e62b8700 1 --2- 172.21.15.70:0/4062811514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65e0100f60 0x7f65e01013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:21.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.300+0000 7f65e62b8700 1 -- 172.21.15.70:0/4062811514 >> 172.21.15.70:0/4062811514 conn(0x7f65e00fc5f0 msgr2=0x7f65e0105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:25:21.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.300+0000 7f65e62b8700 1 -- 172.21.15.70:0/4062811514 shutdown_connections 2024-04-01T01:25:21.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:21.300+0000 7f65e62b8700 1 -- 172.21.15.70:0/4062811514 wait complete. 2024-04-01T01:25:21.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: mon.b calling monitor election 2024-04-01T01:25:21.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-01T01:25:21.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: Updating smithi081:/etc/ceph/ceph.conf 2024-04-01T01:25:21.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-01T01:25:21.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: Updating smithi070:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-01T01:25:21.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: Updating smithi081:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-01T01:25:21.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: Updating smithi189:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-01T01:25:21.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:21.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: mon.b calling monitor election 2024-04-01T01:25:21.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: mon.c calling monitor election 2024-04-01T01:25:21.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: mon.a calling monitor election 2024-04-01T01:25:21.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-01T01:25:21.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: monmap epoch 3 2024-04-01T01:25:21.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: fsid 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:25:21.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: last_changed 2024-04-01T01:25:13.508440+0000 2024-04-01T01:25:21.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: created 2024-04-01T01:23:17.752525+0000 2024-04-01T01:25:21.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: min_mon_release 19 (squid) 2024-04-01T01:25:21.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: election_strategy: 1 2024-04-01T01:25:21.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-01T01:25:21.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-01T01:25:21.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: 2: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.b 2024-04-01T01:25:21.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: fsmap 2024-04-01T01:25:21.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: osdmap e4: 0 total, 0 up, 0 in 2024-04-01T01:25:21.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: mgrmap e13: x(active, since 62s) 2024-04-01T01:25:21.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: overall HEALTH_OK 2024-04-01T01:25:21.749 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:21.750 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:21.750 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:21.750 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:21.750 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: Reconfiguring mon.a (unknown last config time)... 2024-04-01T01:25:21.750 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-01T01:25:21.750 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-01T01:25:21.750 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:21.750 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: Reconfiguring daemon mon.a on smithi070 2024-04-01T01:25:21.750 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:21 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/4062811514' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:21.750 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: mon.b calling monitor election 2024-04-01T01:25:21.750 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: Updating smithi081:/etc/ceph/ceph.conf 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: Updating smithi070:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: Updating smithi081:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: Updating smithi189:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: mon.b calling monitor election 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: mon.c calling monitor election 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: mon.a calling monitor election 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: monmap epoch 3 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: fsid 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: last_changed 2024-04-01T01:25:13.508440+0000 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: created 2024-04-01T01:23:17.752525+0000 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: min_mon_release 19 (squid) 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: election_strategy: 1 2024-04-01T01:25:21.751 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-01T01:25:21.752 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-01T01:25:21.752 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: 2: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.b 2024-04-01T01:25:21.752 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: fsmap 2024-04-01T01:25:21.752 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: osdmap e4: 0 total, 0 up, 0 in 2024-04-01T01:25:21.752 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: mgrmap e13: x(active, since 62s) 2024-04-01T01:25:21.752 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: overall HEALTH_OK 2024-04-01T01:25:21.752 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:21.752 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:21.752 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:21.752 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:21.752 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: Reconfiguring mon.a (unknown last config time)... 2024-04-01T01:25:21.752 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-01T01:25:21.752 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-01T01:25:21.752 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:21.752 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: Reconfiguring daemon mon.a on smithi070 2024-04-01T01:25:21.752 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:21 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/4062811514' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:21.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: mon.b calling monitor election 2024-04-01T01:25:21.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-01T01:25:21.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: Updating smithi081:/etc/ceph/ceph.conf 2024-04-01T01:25:21.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: Updating smithi070:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: Updating smithi081:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: Updating smithi189:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: mon.b calling monitor election 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: mon.c calling monitor election 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: mon.a calling monitor election 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: monmap epoch 3 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: fsid 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: last_changed 2024-04-01T01:25:13.508440+0000 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: created 2024-04-01T01:23:17.752525+0000 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: min_mon_release 19 (squid) 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: election_strategy: 1 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: 0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-01T01:25:21.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: 2: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.b 2024-04-01T01:25:21.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: fsmap 2024-04-01T01:25:21.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: osdmap e4: 0 total, 0 up, 0 in 2024-04-01T01:25:21.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: mgrmap e13: x(active, since 62s) 2024-04-01T01:25:21.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: overall HEALTH_OK 2024-04-01T01:25:21.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:21.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:21.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:21.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:21.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: Reconfiguring mon.a (unknown last config time)... 2024-04-01T01:25:21.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-01T01:25:21.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-01T01:25:21.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:21.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: Reconfiguring daemon mon.a on smithi070 2024-04-01T01:25:21.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:21 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/4062811514' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:22.552 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:25:22 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:25:22.507+0000 7f380117e700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-01T01:25:22.553 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:22.553 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:22.553 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:22.553 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-01T01:25:22.553 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-01T01:25:22.553 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:22.663 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-01T01:25:22.663 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:25:22.663 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.conf 2024-04-01T01:25:22.692 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:25:22.692 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-01T01:25:22.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:22.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:22.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:22.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-01T01:25:22.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-01T01:25:22.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:22.759 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-01T01:25:22.759 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.conf 2024-04-01T01:25:22.783 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:25:22.784 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:22.784 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:22.784 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-01T01:25:22.784 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-01T01:25:22.784 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:22.788 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-01T01:25:22.788 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-01T01:25:22.855 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-01T01:25:22.855 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.conf 2024-04-01T01:25:22.889 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-01T01:25:22.890 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-01T01:25:22.959 INFO:tasks.cephadm:Adding mgr.x on smithi070 2024-04-01T01:25:22.959 INFO:tasks.cephadm:Adding mgr.y on smithi081 2024-04-01T01:25:22.959 INFO:tasks.cephadm:Adding mgr.z on smithi189 2024-04-01T01:25:22.959 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch apply mgr '3;smithi070=x;smithi081=y;smithi189=z' 2024-04-01T01:25:23.233 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-01T01:25:23.619 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:23 smithi189 ceph-mon[35753]: Reconfiguring mon.b (monmap changed)... 2024-04-01T01:25:23.619 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:23 smithi189 ceph-mon[35753]: Reconfiguring daemon mon.b on smithi081 2024-04-01T01:25:23.619 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:23 smithi189 ceph-mon[35753]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:23.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:23 smithi070 ceph-mon[32179]: Reconfiguring mon.b (monmap changed)... 2024-04-01T01:25:23.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:23 smithi070 ceph-mon[32179]: Reconfiguring daemon mon.b on smithi081 2024-04-01T01:25:23.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:23 smithi070 ceph-mon[32179]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:23.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:23 smithi081 ceph-mon[35142]: Reconfiguring mon.b (monmap changed)... 2024-04-01T01:25:23.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:23 smithi081 ceph-mon[35142]: Reconfiguring daemon mon.b on smithi081 2024-04-01T01:25:23.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:23 smithi081 ceph-mon[35142]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:24.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:24 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:24.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:24 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:24.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:24 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:24.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:24 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-01T01:25:24.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:24 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-01T01:25:24.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:24 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:24.918 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:24 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:24.918 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:24 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:24.918 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:24 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:24.919 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:24 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-01T01:25:24.919 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:24 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-01T01:25:24.919 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:24 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:25.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:24 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:25.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:24 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:25.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:24 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:25.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:24 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-01T01:25:25.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:24 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-01T01:25:25.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:24 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:25.251 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.250+0000 7fe70a85d700 1 -- 172.21.15.189:0/2671958223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe704101db0 msgr2=0x7fe7041021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.250+0000 7fe70a85d700 1 --2- 172.21.15.189:0/2671958223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe704101db0 0x7fe7041021d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4009910 tx=0x7fe6f4009c20 comp rx=0 tx=0).stop 2024-04-01T01:25:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.250+0000 7fe70a85d700 1 -- 172.21.15.189:0/2671958223 shutdown_connections 2024-04-01T01:25:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.250+0000 7fe70a85d700 1 --2- 172.21.15.189:0/2671958223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe704101db0 0x7fe7041021d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.250+0000 7fe70a85d700 1 -- 172.21.15.189:0/2671958223 >> 172.21.15.189:0/2671958223 conn(0x7fe7040fd3b0 msgr2=0x7fe7040ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:25:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.250+0000 7fe70a85d700 1 -- 172.21.15.189:0/2671958223 shutdown_connections 2024-04-01T01:25:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.250+0000 7fe70a85d700 1 -- 172.21.15.189:0/2671958223 wait complete. 2024-04-01T01:25:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.250+0000 7fe70a85d700 1 Processor -- start 2024-04-01T01:25:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.250+0000 7fe70a85d700 1 -- start start 2024-04-01T01:25:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe70a85d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe704101db0 0x7fe704075fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe70a85d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe704074630 0x7fe704074ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe70a85d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe704074ff0 0x7fe7041a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe70a85d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe704107cf0 con 0x7fe704074630 2024-04-01T01:25:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe70a85d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe704107b70 con 0x7fe704074ff0 2024-04-01T01:25:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe70a85d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe704107e70 con 0x7fe704101db0 2024-04-01T01:25:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe703fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe704101db0 0x7fe704075fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe7037fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe704074630 0x7fe704074ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe703fff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe704101db0 0x7fe704075fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40516/0 (socket says 172.21.15.189:40516) 2024-04-01T01:25:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe703fff700 1 -- 172.21.15.189:0/3682435114 learned_addr learned my addr 172.21.15.189:0/3682435114 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-01T01:25:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe708dfa700 1 --2- 172.21.15.189:0/3682435114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe704074ff0 0x7fe7041a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe703fff700 1 -- 172.21.15.189:0/3682435114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe704074ff0 msgr2=0x7fe7041a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe703fff700 1 --2- 172.21.15.189:0/3682435114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe704074ff0 0x7fe7041a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe703fff700 1 -- 172.21.15.189:0/3682435114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe704074630 msgr2=0x7fe704074ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe703fff700 1 --2- 172.21.15.189:0/3682435114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe704074630 0x7fe704074ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe703fff700 1 -- 172.21.15.189:0/3682435114 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7041a1b00 con 0x7fe704101db0 2024-04-01T01:25:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe708dfa700 1 --2- 172.21.15.189:0/3682435114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe704074ff0 0x7fe7041a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:25:25.253 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe7037fe700 1 --2- 172.21.15.189:0/3682435114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe704074630 0x7fe704074ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:25:25.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe703fff700 1 --2- 172.21.15.189:0/3682435114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe704101db0 0x7fe704075fe0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4005e00 tx=0x7fe6f4005020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:25:25.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.251+0000 7fe7017fa700 1 -- 172.21.15.189:0/3682435114 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4027070 con 0x7fe704101db0 2024-04-01T01:25:25.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.252+0000 7fe7017fa700 1 -- 172.21.15.189:0/3682435114 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6f400be10 con 0x7fe704101db0 2024-04-01T01:25:25.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.252+0000 7fe70a85d700 1 -- 172.21.15.189:0/3682435114 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7041a1d90 con 0x7fe704101db0 2024-04-01T01:25:25.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.252+0000 7fe7017fa700 1 -- 172.21.15.189:0/3682435114 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4022da0 con 0x7fe704101db0 2024-04-01T01:25:25.254 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.252+0000 7fe70a85d700 1 -- 172.21.15.189:0/3682435114 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7041a2280 con 0x7fe704101db0 2024-04-01T01:25:25.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.252+0000 7fe7017fa700 1 -- 172.21.15.189:0/3682435114 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7fe6f402a070 con 0x7fe704101db0 2024-04-01T01:25:25.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.253+0000 7fe7017fa700 1 --2- 172.21.15.189:0/3682435114 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6ec039330 0x7fe6ec03b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:25.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.253+0000 7fe70a85d700 1 -- 172.21.15.189:0/3682435114 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe704063870 con 0x7fe704101db0 2024-04-01T01:25:25.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.253+0000 7fe7017fa700 1 -- 172.21.15.189:0/3682435114 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe6f404d230 con 0x7fe704101db0 2024-04-01T01:25:25.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.253+0000 7fe7037fe700 1 --2- 172.21.15.189:0/3682435114 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6ec039330 0x7fe6ec03b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:25.255 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.253+0000 7fe7037fe700 1 --2- 172.21.15.189:0/3682435114 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6ec039330 0x7fe6ec03b7f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8009ec0 tx=0x7fe6f8009210 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:25:25.257 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.258+0000 7fe7017fa700 1 -- 172.21.15.189:0/3682435114 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe6f4031030 con 0x7fe704101db0 2024-04-01T01:25:25.468 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.468+0000 7fe70a85d700 1 -- 172.21.15.189:0/3682435114 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi070=x;smithi081=y;smithi189=z", "target": ["mon-mgr", ""]}) -- 0x7fe704062710 con 0x7fe6ec039330 2024-04-01T01:25:25.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.474+0000 7fe7017fa700 1 -- 172.21.15.189:0/3682435114 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe704062710 con 0x7fe6ec039330 2024-04-01T01:25:25.474 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mgr update... 2024-04-01T01:25:25.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.476+0000 7fe70a85d700 1 -- 172.21.15.189:0/3682435114 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6ec039330 msgr2=0x7fe6ec03b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:25.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.476+0000 7fe70a85d700 1 --2- 172.21.15.189:0/3682435114 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6ec039330 0x7fe6ec03b7f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8009ec0 tx=0x7fe6f8009210 comp rx=0 tx=0).stop 2024-04-01T01:25:25.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.476+0000 7fe70a85d700 1 -- 172.21.15.189:0/3682435114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe704101db0 msgr2=0x7fe704075fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:25.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.476+0000 7fe70a85d700 1 --2- 172.21.15.189:0/3682435114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe704101db0 0x7fe704075fe0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4005e00 tx=0x7fe6f4005020 comp rx=0 tx=0).stop 2024-04-01T01:25:25.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.476+0000 7fe70a85d700 1 -- 172.21.15.189:0/3682435114 shutdown_connections 2024-04-01T01:25:25.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.476+0000 7fe70a85d700 1 --2- 172.21.15.189:0/3682435114 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6ec039330 0x7fe6ec03b7f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:25.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.476+0000 7fe70a85d700 1 --2- 172.21.15.189:0/3682435114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe704074ff0 0x7fe7041a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:25.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.476+0000 7fe70a85d700 1 --2- 172.21.15.189:0/3682435114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe704101db0 0x7fe704075fe0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:25.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.476+0000 7fe70a85d700 1 --2- 172.21.15.189:0/3682435114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe704074630 0x7fe704074ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:25.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.476+0000 7fe70a85d700 1 -- 172.21.15.189:0/3682435114 >> 172.21.15.189:0/3682435114 conn(0x7fe7040fd3b0 msgr2=0x7fe7040fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:25:25.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.477+0000 7fe70a85d700 1 -- 172.21.15.189:0/3682435114 shutdown_connections 2024-04-01T01:25:25.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:25:25.477+0000 7fe70a85d700 1 -- 172.21.15.189:0/3682435114 wait complete. 2024-04-01T01:25:25.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:25 smithi189 ceph-mon[35753]: Reconfiguring mon.c (monmap changed)... 2024-04-01T01:25:25.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:25 smithi189 ceph-mon[35753]: Reconfiguring daemon mon.c on smithi189 2024-04-01T01:25:25.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:25 smithi189 ceph-mon[35753]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:25.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:25 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:25.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:25 smithi081 ceph-mon[35142]: Reconfiguring mon.c (monmap changed)... 2024-04-01T01:25:25.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:25 smithi081 ceph-mon[35142]: Reconfiguring daemon mon.c on smithi189 2024-04-01T01:25:25.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:25 smithi081 ceph-mon[35142]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:25.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:25 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:26.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:25 smithi070 ceph-mon[32179]: Reconfiguring mon.c (monmap changed)... 2024-04-01T01:25:26.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:25 smithi070 ceph-mon[32179]: Reconfiguring daemon mon.c on smithi189 2024-04-01T01:25:26.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:25 smithi070 ceph-mon[32179]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:26.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:25 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:26.296 DEBUG:teuthology.orchestra.run.smithi081:mgr.y> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mgr.y.service 2024-04-01T01:25:26.299 DEBUG:teuthology.orchestra.run.smithi189:mgr.z> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mgr.z.service 2024-04-01T01:25:26.302 INFO:tasks.cephadm:Deploying OSDs... 2024-04-01T01:25:26.302 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:25:26.302 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/scratch_devs of=/dev/stdout 2024-04-01T01:25:26.317 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-01T01:25:26.318 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_1 2024-04-01T01:25:26.373 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-01T01:25:26.373 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T01:25:26.373 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-01T01:25:26.373 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T01:25:26.373 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T01:25:26.373 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-01 01:24:31.035229025 +0000 2024-04-01T01:25:26.373 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-01 01:18:53.463198449 +0000 2024-04-01T01:25:26.373 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-01 01:18:53.463198449 +0000 2024-04-01T01:25:26.373 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-01 01:18:53.463198449 +0000 2024-04-01T01:25:26.373 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-01T01:25:26.441 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-01T01:25:26.442 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-01T01:25:26.442 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000109977 s, 4.7 MB/s 2024-04-01T01:25:26.443 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-01T01:25:26.498 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_2 2024-04-01T01:25:26.553 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-01T01:25:26.553 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T01:25:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-01T01:25:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T01:25:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T01:25:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-01 01:24:31.066228085 +0000 2024-04-01T01:25:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-01 01:18:53.788188599 +0000 2024-04-01T01:25:26.553 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-01 01:18:53.788188599 +0000 2024-04-01T01:25:26.553 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-01 01:18:53.788188599 +0000 2024-04-01T01:25:26.553 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-01T01:25:26.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:26 smithi081 ceph-mon[35142]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi070=x;smithi081=y;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:25:26.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:26 smithi081 ceph-mon[35142]: Saving service mgr spec with placement smithi070=x;smithi081=y;smithi189=z;count:3 2024-04-01T01:25:26.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:26 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:26.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:26 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:26.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:26 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:26.620 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-01T01:25:26.620 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-01T01:25:26.620 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000116789 s, 4.4 MB/s 2024-04-01T01:25:26.621 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-01T01:25:26.677 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_3 2024-04-01T01:25:26.732 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-01T01:25:26.732 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T01:25:26.732 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-01T01:25:26.732 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T01:25:26.732 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T01:25:26.733 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-01 01:24:31.095227205 +0000 2024-04-01T01:25:26.733 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-01 01:18:54.134178113 +0000 2024-04-01T01:25:26.733 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-01 01:18:54.134178113 +0000 2024-04-01T01:25:26.733 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-01 01:18:54.134178113 +0000 2024-04-01T01:25:26.733 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-01T01:25:26.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:26 smithi070 ceph-mon[32179]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi070=x;smithi081=y;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:25:26.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:26 smithi070 ceph-mon[32179]: Saving service mgr spec with placement smithi070=x;smithi081=y;smithi189=z;count:3 2024-04-01T01:25:26.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:26 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:26.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:26 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:26.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:26 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:26.778 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-01T01:25:26.778 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-01T01:25:26.778 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000108709 s, 4.7 MB/s 2024-04-01T01:25:26.779 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-01T01:25:26.834 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_4 2024-04-01T01:25:26.888 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-01T01:25:26.889 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T01:25:26.889 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-01T01:25:26.889 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T01:25:26.889 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T01:25:26.889 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-01 01:24:31.126226265 +0000 2024-04-01T01:25:26.889 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-01 01:18:54.467168021 +0000 2024-04-01T01:25:26.889 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-01 01:18:54.467168021 +0000 2024-04-01T01:25:26.889 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-01 01:18:54.467168021 +0000 2024-04-01T01:25:26.889 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-01T01:25:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-01T01:25:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-01T01:25:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 8.7068e-05 s, 5.9 MB/s 2024-04-01T01:25:26.955 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-01T01:25:26.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:26 smithi189 ceph-mon[35753]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi070=x;smithi081=y;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:25:26.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:26 smithi189 ceph-mon[35753]: Saving service mgr spec with placement smithi070=x;smithi081=y;smithi189=z;count:3 2024-04-01T01:25:26.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:26 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:26.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:26 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:26.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:26 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:27.010 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-01T01:25:27.010 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2024-04-01T01:25:27.029 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-01T01:25:27.029 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_1 2024-04-01T01:25:27.086 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-01T01:25:27.087 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T01:25:27.087 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-01T01:25:27.087 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T01:25:27.087 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T01:25:27.087 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-01 01:25:12.932686323 +0000 2024-04-01T01:25:27.087 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-01 01:18:53.734749126 +0000 2024-04-01T01:25:27.087 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-01 01:18:53.734749126 +0000 2024-04-01T01:25:27.087 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-04-01 01:18:53.734749126 +0000 2024-04-01T01:25:27.087 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-01T01:25:27.157 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-04-01T01:25:27.157 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-04-01T01:25:27.157 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000353114 s, 1.4 MB/s 2024-04-01T01:25:27.159 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-01T01:25:27.218 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_2 2024-04-01T01:25:27.275 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-01T01:25:27.275 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T01:25:27.275 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-04-01T01:25:27.276 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T01:25:27.276 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T01:25:27.276 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-01 01:25:12.933686304 +0000 2024-04-01T01:25:27.276 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-01 01:18:54.100741872 +0000 2024-04-01T01:25:27.276 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-01 01:18:54.100741872 +0000 2024-04-01T01:25:27.276 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-04-01 01:18:54.100741872 +0000 2024-04-01T01:25:27.276 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-01T01:25:27.345 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-04-01T01:25:27.345 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-04-01T01:25:27.346 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.00022366 s, 2.3 MB/s 2024-04-01T01:25:27.347 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-01T01:25:27.405 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_3 2024-04-01T01:25:27.462 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-01T01:25:27.462 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T01:25:27.463 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-01T01:25:27.463 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T01:25:27.463 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T01:25:27.463 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-01 01:25:12.933686304 +0000 2024-04-01T01:25:27.463 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-01 01:18:54.435735232 +0000 2024-04-01T01:25:27.463 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-01 01:18:54.435735232 +0000 2024-04-01T01:25:27.463 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-04-01 01:18:54.435735232 +0000 2024-04-01T01:25:27.463 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-01T01:25:27.531 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-04-01T01:25:27.531 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-04-01T01:25:27.531 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000249927 s, 2.0 MB/s 2024-04-01T01:25:27.532 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-01T01:25:27.591 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_4 2024-04-01T01:25:27.647 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-01T01:25:27.647 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T01:25:27.647 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-01T01:25:27.647 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T01:25:27.647 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T01:25:27.648 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-01 01:25:12.933686304 +0000 2024-04-01T01:25:27.648 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-01 01:18:54.782728354 +0000 2024-04-01T01:25:27.648 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-01 01:18:54.782728354 +0000 2024-04-01T01:25:27.648 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-04-01 01:18:54.782728354 +0000 2024-04-01T01:25:27.648 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-01T01:25:27.714 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-04-01T01:25:27.714 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-04-01T01:25:27.715 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000214319 s, 2.4 MB/s 2024-04-01T01:25:27.716 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-01T01:25:27.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:27 smithi189 ceph-mon[35753]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:27.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:27.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:27.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:27.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-01T01:25:27.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-01T01:25:27.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-01T01:25:27.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:27.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:27 smithi189 ceph-mon[35753]: Deploying daemon mgr.z on smithi189 2024-04-01T01:25:27.772 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-01T01:25:27.772 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-04-01T01:25:27.789 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-01T01:25:27.789 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-04-01T01:25:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:27 smithi081 ceph-mon[35142]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-01T01:25:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-01T01:25:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-01T01:25:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:27 smithi081 ceph-mon[35142]: Deploying daemon mgr.z on smithi189 2024-04-01T01:25:27.844 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-01T01:25:27.845 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T01:25:27.845 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-04-01T01:25:27.845 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T01:25:27.845 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T01:25:27.845 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-01 01:25:04.010654987 +0000 2024-04-01T01:25:27.845 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-01 01:18:53.097498502 +0000 2024-04-01T01:25:27.845 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-01 01:18:53.097498502 +0000 2024-04-01T01:25:27.845 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-01 01:18:53.097498502 +0000 2024-04-01T01:25:27.845 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-01T01:25:27.911 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-01T01:25:27.911 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-01T01:25:27.911 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000191376 s, 2.7 MB/s 2024-04-01T01:25:27.913 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-01T01:25:27.968 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-04-01T01:25:28.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:27 smithi070 ceph-mon[32179]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:28.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:28.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:28.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:28.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-01T01:25:28.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-01T01:25:28.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-01T01:25:28.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:28.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:27 smithi070 ceph-mon[32179]: Deploying daemon mgr.z on smithi189 2024-04-01T01:25:28.023 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-01T01:25:28.023 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T01:25:28.023 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-01T01:25:28.023 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T01:25:28.023 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T01:25:28.023 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-01 01:25:04.010654987 +0000 2024-04-01T01:25:28.023 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-01 01:18:53.441491922 +0000 2024-04-01T01:25:28.023 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-01 01:18:53.441491922 +0000 2024-04-01T01:25:28.024 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-01 01:18:53.441491922 +0000 2024-04-01T01:25:28.024 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-01T01:25:28.090 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-01T01:25:28.090 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-01T01:25:28.090 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000209913 s, 2.4 MB/s 2024-04-01T01:25:28.091 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-01T01:25:28.147 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-04-01T01:25:28.201 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-01T01:25:28.202 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T01:25:28.202 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-01T01:25:28.202 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T01:25:28.202 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T01:25:28.202 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-01 01:25:04.011654968 +0000 2024-04-01T01:25:28.202 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-01 01:18:53.795485151 +0000 2024-04-01T01:25:28.202 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-01 01:18:53.795485151 +0000 2024-04-01T01:25:28.202 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-01 01:18:53.795485151 +0000 2024-04-01T01:25:28.202 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-01T01:25:28.298 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-01T01:25:28.299 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-01T01:25:28.299 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000200246 s, 2.6 MB/s 2024-04-01T01:25:28.300 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-01T01:25:28.314 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-04-01T01:25:28.375 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-01T01:25:28.375 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-01T01:25:28.376 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-01T01:25:28.376 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-01T01:25:28.376 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-01T01:25:28.376 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-01 01:25:28.254190674 +0000 2024-04-01T01:25:28.376 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-01 01:18:54.143478494 +0000 2024-04-01T01:25:28.376 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-01 01:18:54.143478494 +0000 2024-04-01T01:25:28.376 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-01 01:18:54.143478494 +0000 2024-04-01T01:25:28.376 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-01T01:25:28.475 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-01T01:25:28.475 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-01T01:25:28.475 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000266236 s, 1.9 MB/s 2024-04-01T01:25:28.477 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-01T01:25:28.491 INFO:tasks.cephadm:Deploying osd.0 on smithi070 with /dev/vg_nvme/lv_4... 2024-04-01T01:25:28.492 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-01T01:25:28.710 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:28 smithi189 systemd[1]: Starting Ceph mgr.z for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:25:28.732 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:25:28.996 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:28 smithi189 podman[37588]: 2024-04-01 01:25:28.61388398 +0000 UTC m=+0.017218721 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:25:28.997 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:28 smithi189 podman[37588]: 2024-04-01 01:25:28.711420443 +0000 UTC m=+0.114755180 container create d852bc6452591807f09150d4be370d925fdb3f45fc44459e916f8469ced5874e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326) 2024-04-01T01:25:28.997 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:28 smithi189 podman[37588]: 2024-04-01 01:25:28.913779452 +0000 UTC m=+0.317114179 container init d852bc6452591807f09150d4be370d925fdb3f45fc44459e916f8469ced5874e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-01T01:25:28.997 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:28 smithi189 podman[37588]: 2024-04-01 01:25:28.916215579 +0000 UTC m=+0.319550307 container start d852bc6452591807f09150d4be370d925fdb3f45fc44459e916f8469ced5874e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326) 2024-04-01T01:25:28.997 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:28 smithi189 ceph-mgr[37607]: -- 172.21.15.189:0/3146269881 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d0b770e5a0 con 0x55d0b76e3000 2024-04-01T01:25:28.997 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:28 smithi189 bash[37588]: d852bc6452591807f09150d4be370d925fdb3f45fc44459e916f8469ced5874e 2024-04-01T01:25:29.248 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:29 smithi189 systemd[1]: Started Ceph mgr.z for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:25:29.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:29 smithi189 ceph-mon[35753]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:29.587 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:29 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:29.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:29 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:29.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:29 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:29.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:29 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-01T01:25:29.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:29 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-01T01:25:29.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:29 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-01T01:25:29.588 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:29 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:29.588 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:29 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:29.489+0000 7fbf95d9f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-01T01:25:29.588 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:29 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:29.587+0000 7fbf95d9f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-01T01:25:29.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:29 smithi081 ceph-mon[35142]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:29.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:29 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:29.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:29 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:29.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:29 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:29.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:29 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-01T01:25:29.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:29 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-01T01:25:29.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:29 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-01T01:25:29.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:29 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:29.849 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:29 smithi070 ceph-mon[32179]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:29.849 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:29 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:29.849 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:29 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:29.849 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:29 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:29.849 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:29 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-01T01:25:29.849 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:29 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-01T01:25:29.849 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:29 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-01T01:25:29.849 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:29 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:30.542 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:30 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:30.256+0000 7fbf95d9f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-01T01:25:30.542 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:30 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:30.349+0000 7fbf95d9f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-01T01:25:30.542 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:30 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:30.491+0000 7fbf95d9f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-01T01:25:30.557 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-01T01:25:30.557 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-01T01:25:30.557 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-04-01T01:25:30.557 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-04-01T01:25:30.557 INFO:teuthology.orchestra.run.smithi070.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00841447 s, 1.2 GB/s 2024-04-01T01:25:30.557 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-04-01T01:25:30.656 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:30 smithi081 ceph-mon[35142]: Deploying daemon mgr.y on smithi081 2024-04-01T01:25:30.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:30 smithi070 ceph-mon[32179]: Deploying daemon mgr.y on smithi081 2024-04-01T01:25:30.945 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:30 smithi189 ceph-mon[35753]: Deploying daemon mgr.y on smithi081 2024-04-01T01:25:30.945 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:30 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:30.587+0000 7fbf95d9f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-01T01:25:30.945 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:30 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:30.671+0000 7fbf95d9f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-01T01:25:30.945 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:30 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:30.945+0000 7fbf95d9f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-01T01:25:31.201 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi070:vg_nvme/lv_4 2024-04-01T01:25:31.236 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:31 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:31.056+0000 7fbf95d9f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-01T01:25:31.236 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:31 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:31.142+0000 7fbf95d9f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-01T01:25:31.236 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:31 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:31.236+0000 7fbf95d9f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-01T01:25:31.328 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:31 smithi081 podman[36510]: 2024-04-01 01:25:31.235694501 +0000 UTC m=+0.139619569 container create 2e186bfb704ac6f29a42abb78454e9bf0ad41fd3f93f157f228c0a4789746f89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326) 2024-04-01T01:25:31.435 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:25:31.498 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:31 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:31.413+0000 7fbf95d9f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-01T01:25:31.616 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:31 smithi081 ceph-mon[35142]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:31.616 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:31 smithi081 podman[36510]: 2024-04-01 01:25:31.446321621 +0000 UTC m=+0.350246667 container init 2e186bfb704ac6f29a42abb78454e9bf0ad41fd3f93f157f228c0a4789746f89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-01T01:25:31.616 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:31 smithi081 podman[36510]: 2024-04-01 01:25:31.449458637 +0000 UTC m=+0.353383664 container start 2e186bfb704ac6f29a42abb78454e9bf0ad41fd3f93f157f228c0a4789746f89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-01T01:25:31.617 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:31 smithi081 ceph-mgr[36530]: -- 172.21.15.81:0/3000120120 <== mon.2 v2:172.21.15.81:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557ba26de5a0 con 0x557ba26b2c00 2024-04-01T01:25:31.617 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:31 smithi081 bash[36510]: 2e186bfb704ac6f29a42abb78454e9bf0ad41fd3f93f157f228c0a4789746f89 2024-04-01T01:25:31.617 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:31 smithi081 systemd[1]: Started Ceph mgr.y for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:25:31.869 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:31 smithi070 ceph-mon[32179]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:31.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:31 smithi189 ceph-mon[35753]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:32.328 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:32 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:32.035+0000 7fefdc8a7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-01T01:25:32.328 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:32 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:32.133+0000 7fefdc8a7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-01T01:25:32.748 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:32 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:32.454+0000 7fbf95d9f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-01T01:25:32.749 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:32 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:32.585+0000 7fbf95d9f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-01T01:25:32.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:32.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:32.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:32.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:32.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:32.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:32.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:32.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:32.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:32.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:32.839 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:32 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:32.827+0000 7fefdc8a7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-01T01:25:32.840 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:32.840 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:32.840 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:32.840 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:32.840 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:32.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.961+0000 7fe27103f700 1 -- 172.21.15.70:0/1667623898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe26c102390 msgr2=0x7fe26c1027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:32.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.961+0000 7fe27103f700 1 --2- 172.21.15.70:0/1667623898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe26c102390 0x7fe26c1027f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe25c0099c0 tx=0x7fe25c009cd0 comp rx=0 tx=0).stop 2024-04-01T01:25:32.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.961+0000 7fe27103f700 1 -- 172.21.15.70:0/1667623898 shutdown_connections 2024-04-01T01:25:32.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.961+0000 7fe27103f700 1 --2- 172.21.15.70:0/1667623898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe26c102d30 0x7fe26c1095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:32.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.961+0000 7fe27103f700 1 --2- 172.21.15.70:0/1667623898 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe26c101190 0x7fe26c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:32.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.961+0000 7fe27103f700 1 --2- 172.21.15.70:0/1667623898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe26c102390 0x7fe26c1027f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:32.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.961+0000 7fe27103f700 1 -- 172.21.15.70:0/1667623898 >> 172.21.15.70:0/1667623898 conn(0x7fe26c0fc730 msgr2=0x7fe26c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:25:32.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.961+0000 7fe27103f700 1 -- 172.21.15.70:0/1667623898 shutdown_connections 2024-04-01T01:25:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.961+0000 7fe27103f700 1 -- 172.21.15.70:0/1667623898 wait complete. 2024-04-01T01:25:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe27103f700 1 Processor -- start 2024-04-01T01:25:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe27103f700 1 -- start start 2024-04-01T01:25:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe27103f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe26c101190 0x7fe26c1983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe27103f700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe26c102390 0x7fe26c198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe27103f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe26c102d30 0x7fe26c19f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe27103f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe26c10cc30 con 0x7fe26c101190 2024-04-01T01:25:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe27103f700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe26c10cab0 con 0x7fe26c102390 2024-04-01T01:25:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe27103f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe26c10cdb0 con 0x7fe26c102d30 2024-04-01T01:25:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe26ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe26c101190 0x7fe26c1983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe26ad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe26c101190 0x7fe26c1983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57576/0 (socket says 172.21.15.70:57576) 2024-04-01T01:25:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe26ad9d700 1 -- 172.21.15.70:0/3696317028 learned_addr learned my addr 172.21.15.70:0/3696317028 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:25:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe26b59e700 1 --2- 172.21.15.70:0/3696317028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe26c102d30 0x7fe26c19f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe26a59c700 1 --2- 172.21.15.70:0/3696317028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe26c102390 0x7fe26c198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe26ad9d700 1 -- 172.21.15.70:0/3696317028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe26c102d30 msgr2=0x7fe26c19f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe26ad9d700 1 --2- 172.21.15.70:0/3696317028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe26c102d30 0x7fe26c19f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe26ad9d700 1 -- 172.21.15.70:0/3696317028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe26c102390 msgr2=0x7fe26c198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe26ad9d700 1 --2- 172.21.15.70:0/3696317028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe26c102390 0x7fe26c198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.962+0000 7fe26ad9d700 1 -- 172.21.15.70:0/3696317028 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26c1a0020 con 0x7fe26c101190 2024-04-01T01:25:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.963+0000 7fe26ad9d700 1 --2- 172.21.15.70:0/3696317028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe26c101190 0x7fe26c1983c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe26000e9a0 tx=0x7fe26000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:25:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.963+0000 7fe253fff700 1 -- 172.21.15.70:0/3696317028 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe260004220 con 0x7fe26c101190 2024-04-01T01:25:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.963+0000 7fe27103f700 1 -- 172.21.15.70:0/3696317028 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe26c1a0310 con 0x7fe26c101190 2024-04-01T01:25:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.963+0000 7fe253fff700 1 -- 172.21.15.70:0/3696317028 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2600043c0 con 0x7fe26c101190 2024-04-01T01:25:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.963+0000 7fe253fff700 1 -- 172.21.15.70:0/3696317028 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe260003ce0 con 0x7fe26c101190 2024-04-01T01:25:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.963+0000 7fe27103f700 1 -- 172.21.15.70:0/3696317028 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe26c1a0850 con 0x7fe26c101190 2024-04-01T01:25:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.964+0000 7fe253fff700 1 -- 172.21.15.70:0/3696317028 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46172+0+0 (secure 0 0 0) 0x7fe260018430 con 0x7fe26c101190 2024-04-01T01:25:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.964+0000 7fe27103f700 1 -- 172.21.15.70:0/3696317028 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe26c068f00 con 0x7fe26c101190 2024-04-01T01:25:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.964+0000 7fe253fff700 1 --2- 172.21.15.70:0/3696317028 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2540390e0 0x7fe25403b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.964+0000 7fe253fff700 1 -- 172.21.15.70:0/3696317028 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe26004c200 con 0x7fe26c101190 2024-04-01T01:25:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.964+0000 7fe26a59c700 1 --2- 172.21.15.70:0/3696317028 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2540390e0 0x7fe25403b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:32.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.964+0000 7fe26a59c700 1 --2- 172.21.15.70:0/3696317028 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2540390e0 0x7fe25403b5a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe25c000c00 tx=0x7fe25c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:25:32.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:32.969+0000 7fe253fff700 1 -- 172.21.15.70:0/3696317028 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe2600071c0 con 0x7fe26c101190 2024-04-01T01:25:33.167 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:32 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:32.920+0000 7fefdc8a7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-01T01:25:33.168 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:33.061+0000 7fefdc8a7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-01T01:25:33.168 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:33.167+0000 7fefdc8a7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-01T01:25:33.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:33.180+0000 7fe27103f700 1 -- 172.21.15.70:0/3696317028 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fe26c0fe8f0 con 0x7fe2540390e0 2024-04-01T01:25:33.243 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:32 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:32.935+0000 7fbf95d9f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-01T01:25:33.243 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:33.026+0000 7fbf95d9f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-01T01:25:33.243 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:33.243+0000 7fbf95d9f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-01T01:25:33.498 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:33.432+0000 7fbf95d9f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-01T01:25:33.509 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:33.251+0000 7fefdc8a7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-01T01:25:33.795 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:33.509+0000 7fefdc8a7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-01T01:25:33.795 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:33.618+0000 7fefdc8a7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-01T01:25:33.795 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:33.703+0000 7fefdc8a7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-01T01:25:33.795 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:33.795+0000 7fefdc8a7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-01T01:25:33.795 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-mon[35142]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:33.795 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.795 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.795 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.795 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.796 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:33.796 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:33.796 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.796 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:25:33.796 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:25:33.796 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:33.796 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-01T01:25:33.796 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-01T01:25:33.796 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:33.796 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.914 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:33 smithi070 ceph-mon[32179]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:33.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:33.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:33.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:25:33.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:25:33.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:33.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-01T01:25:33.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-01T01:25:33.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:33.915 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.977 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:33.715+0000 7fbf95d9f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-01T01:25:33.977 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:33.977+0000 7fbf95d9f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-01T01:25:33.977 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-mon[35753]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:33.977 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.977 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.977 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.977 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.977 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:33.978 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:33.978 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:33.978 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:25:33.978 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:25:33.978 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:33.978 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-01T01:25:33.978 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-01T01:25:33.978 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:33.978 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:34.078 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:33 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:33.968+0000 7fefdc8a7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-01T01:25:34.248 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:34 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:34.212+0000 7fbf95d9f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-01T01:25:34.867 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:34 smithi070 ceph-mon[32179]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:25:34.867 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:34 smithi070 ceph-mon[32179]: Reconfiguring mgr.x (unknown last config time)... 2024-04-01T01:25:34.867 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:34 smithi070 ceph-mon[32179]: Reconfiguring daemon mgr.x on smithi070 2024-04-01T01:25:34.922 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:34 smithi081 ceph-mon[35142]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:25:34.922 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:34 smithi081 ceph-mon[35142]: Reconfiguring mgr.x (unknown last config time)... 2024-04-01T01:25:34.923 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:34 smithi081 ceph-mon[35142]: Reconfiguring daemon mgr.x on smithi070 2024-04-01T01:25:34.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:34 smithi189 ceph-mon[35753]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:25:34.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:34 smithi189 ceph-mon[35753]: Reconfiguring mgr.x (unknown last config time)... 2024-04-01T01:25:34.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:34 smithi189 ceph-mon[35753]: Reconfiguring daemon mgr.x on smithi070 2024-04-01T01:25:35.328 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:34 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:34.923+0000 7fefdc8a7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-01T01:25:35.328 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:35 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:35.050+0000 7fefdc8a7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-01T01:25:35.498 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:35 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:35.020+0000 7fbf95d9f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-01T01:25:35.651 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:35 smithi081 ceph-mon[35142]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:35.652 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:35 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:35.375+0000 7fefdc8a7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-01T01:25:35.652 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:35 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:35.461+0000 7fefdc8a7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-01T01:25:35.652 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:35 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:35.652+0000 7fefdc8a7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-01T01:25:35.909 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:35 smithi189 ceph-mon[35753]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:35.909 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:35 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:35.593+0000 7fbf95d9f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-01T01:25:35.909 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:35 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:35.692+0000 7fbf95d9f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-01T01:25:35.909 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:25:35 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z[37603]: 2024-04-01T01:25:35.909+0000 7fbf95d9f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-01T01:25:35.918 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:35 smithi070 ceph-mon[32179]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:36.078 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:35 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:35.837+0000 7fefdc8a7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-01T01:25:36.363 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:36 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:36.111+0000 7fefdc8a7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-01T01:25:36.364 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:36 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:36.364+0000 7fefdc8a7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-01T01:25:36.653 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:36 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:36.596+0000 7fefdc8a7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-01T01:25:36.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:36.707+0000 7fe253fff700 1 -- 172.21.15.70:0/3696317028 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 14) ==== 91718+0+0 (secure 0 0 0) 0x7fe260013d60 con 0x7fe26c101190 2024-04-01T01:25:36.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:36 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:36.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:36 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:36.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:36 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:36.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:36 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:36.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:36 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:36.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:36 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:36.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:36 smithi189 ceph-mon[35753]: Standby manager daemon z started 2024-04-01T01:25:36.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:36 smithi189 ceph-mon[35753]: from='mgr.? 172.21.15.189:0/3413383668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-01T01:25:36.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:36 smithi189 ceph-mon[35753]: from='mgr.? 172.21.15.189:0/3413383668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-01T01:25:36.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:36 smithi189 ceph-mon[35753]: from='mgr.? 172.21.15.189:0/3413383668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-01T01:25:36.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:36 smithi189 ceph-mon[35753]: from='mgr.? 172.21.15.189:0/3413383668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-01T01:25:36.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:36 smithi189 ceph-mon[35753]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:37.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:36 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:37.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:36 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:37.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:36 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:37.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:36 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:37.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:36 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:37.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:36 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:37.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:36 smithi070 ceph-mon[32179]: Standby manager daemon z started 2024-04-01T01:25:37.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:36 smithi070 ceph-mon[32179]: from='mgr.? 172.21.15.189:0/3413383668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-01T01:25:37.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:36 smithi070 ceph-mon[32179]: from='mgr.? 172.21.15.189:0/3413383668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-01T01:25:37.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:36 smithi070 ceph-mon[32179]: from='mgr.? 172.21.15.189:0/3413383668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-01T01:25:37.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:36 smithi070 ceph-mon[32179]: from='mgr.? 172.21.15.189:0/3413383668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-01T01:25:37.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:36 smithi070 ceph-mon[32179]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:36 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:36 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:36 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:36 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:37.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:36 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:37.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:36 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:37.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:36 smithi081 ceph-mon[35142]: Standby manager daemon z started 2024-04-01T01:25:37.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:36 smithi081 ceph-mon[35142]: from='mgr.? 172.21.15.189:0/3413383668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-01T01:25:37.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:36 smithi081 ceph-mon[35142]: from='mgr.? 172.21.15.189:0/3413383668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-01T01:25:37.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:36 smithi081 ceph-mon[35142]: from='mgr.? 172.21.15.189:0/3413383668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-01T01:25:37.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:36 smithi081 ceph-mon[35142]: from='mgr.? 172.21.15.189:0/3413383668' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-01T01:25:37.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:36 smithi081 ceph-mon[35142]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:37.700 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:37 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:37.383+0000 7fefdc8a7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-01T01:25:37.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:37 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/721272016' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4e9041f-f0ad-4b4b-80f6-2726874d9231"}]: dispatch 2024-04-01T01:25:37.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:37 smithi189 ceph-mon[35753]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-01T01:25:37.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:37 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-01T01:25:37.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:37 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/721272016' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4e9041f-f0ad-4b4b-80f6-2726874d9231"}]': finished 2024-04-01T01:25:37.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:37 smithi189 ceph-mon[35753]: osdmap e5: 1 total, 0 up, 1 in 2024-04-01T01:25:37.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:37 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:37.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:37 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/93136010' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:25:38.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:37 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/721272016' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4e9041f-f0ad-4b4b-80f6-2726874d9231"}]: dispatch 2024-04-01T01:25:38.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:37 smithi070 ceph-mon[32179]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-01T01:25:38.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:37 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-01T01:25:38.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:37 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/721272016' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4e9041f-f0ad-4b4b-80f6-2726874d9231"}]': finished 2024-04-01T01:25:38.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:37 smithi070 ceph-mon[32179]: osdmap e5: 1 total, 0 up, 1 in 2024-04-01T01:25:38.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:37 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:38.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:37 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/93136010' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:25:38.044 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:37 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:37.947+0000 7fefdc8a7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-01T01:25:38.044 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:38 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:38.045+0000 7fefdc8a7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-01T01:25:38.044 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:37 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/721272016' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a4e9041f-f0ad-4b4b-80f6-2726874d9231"}]: dispatch 2024-04-01T01:25:38.044 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:37 smithi081 ceph-mon[35142]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-01T01:25:38.044 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:37 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-01T01:25:38.045 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:37 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/721272016' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a4e9041f-f0ad-4b4b-80f6-2726874d9231"}]': finished 2024-04-01T01:25:38.045 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:37 smithi081 ceph-mon[35142]: osdmap e5: 1 total, 0 up, 1 in 2024-04-01T01:25:38.045 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:37 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:38.045 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:37 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/93136010' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:25:38.328 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:25:38 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y[36525]: 2024-04-01T01:25:38.258+0000 7fefdc8a7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-01T01:25:38.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:38.711+0000 7fe253fff700 1 -- 172.21.15.70:0/3696317028 <== mon.0 v2:172.21.15.70:3300/0 8 ==== mgrmap(e 15) ==== 137264+0+0 (secure 0 0 0) 0x7fe260017cd0 con 0x7fe26c101190 2024-04-01T01:25:38.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:38 smithi189 ceph-mon[35753]: Standby manager daemon y started 2024-04-01T01:25:38.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:38 smithi189 ceph-mon[35753]: from='mgr.? 172.21.15.81:0/2926525896' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-01T01:25:38.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:38 smithi189 ceph-mon[35753]: from='mgr.? 172.21.15.81:0/2926525896' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-01T01:25:38.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:38 smithi189 ceph-mon[35753]: from='mgr.? 172.21.15.81:0/2926525896' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-01T01:25:38.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:38 smithi189 ceph-mon[35753]: from='mgr.? 172.21.15.81:0/2926525896' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-01T01:25:38.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:38 smithi189 ceph-mon[35753]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:39.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:38 smithi070 ceph-mon[32179]: Standby manager daemon y started 2024-04-01T01:25:39.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:38 smithi070 ceph-mon[32179]: from='mgr.? 172.21.15.81:0/2926525896' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-01T01:25:39.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:38 smithi070 ceph-mon[32179]: from='mgr.? 172.21.15.81:0/2926525896' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-01T01:25:39.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:38 smithi070 ceph-mon[32179]: from='mgr.? 172.21.15.81:0/2926525896' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-01T01:25:39.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:38 smithi070 ceph-mon[32179]: from='mgr.? 172.21.15.81:0/2926525896' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-01T01:25:39.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:38 smithi070 ceph-mon[32179]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:38 smithi081 ceph-mon[35142]: Standby manager daemon y started 2024-04-01T01:25:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:38 smithi081 ceph-mon[35142]: from='mgr.? 172.21.15.81:0/2926525896' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-01T01:25:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:38 smithi081 ceph-mon[35142]: from='mgr.? 172.21.15.81:0/2926525896' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-01T01:25:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:38 smithi081 ceph-mon[35142]: from='mgr.? 172.21.15.81:0/2926525896' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-01T01:25:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:38 smithi081 ceph-mon[35142]: from='mgr.? 172.21.15.81:0/2926525896' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-01T01:25:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:38 smithi081 ceph-mon[35142]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:39.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:39 smithi189 ceph-mon[35753]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-01T01:25:39.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:39 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-01T01:25:40.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:39 smithi070 ceph-mon[32179]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-01T01:25:40.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:39 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-01T01:25:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:39 smithi081 ceph-mon[35142]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-01T01:25:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:39 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-01T01:25:40.912 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:40 smithi070 ceph-mon[32179]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:40.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:40 smithi189 ceph-mon[35753]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:41.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:40 smithi081 ceph-mon[35142]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:43.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:43 smithi081 ceph-mon[35142]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:43.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-01T01:25:43.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:43.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:43 smithi189 ceph-mon[35753]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:43.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-01T01:25:43.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:44.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:43 smithi070 ceph-mon[32179]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:44.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-01T01:25:44.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:44.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:44 smithi070 ceph-mon[32179]: Deploying daemon osd.0 on smithi070 2024-04-01T01:25:44.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:44 smithi081 ceph-mon[35142]: Deploying daemon osd.0 on smithi070 2024-04-01T01:25:44.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:44 smithi189 ceph-mon[35753]: Deploying daemon osd.0 on smithi070 2024-04-01T01:25:45.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:45 smithi081 ceph-mon[35142]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:45.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:45 smithi189 ceph-mon[35753]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:46.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:45 smithi070 ceph-mon[32179]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:47.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:47 smithi081 ceph-mon[35142]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:47.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:47 smithi189 ceph-mon[35753]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:48.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:47 smithi070 ceph-mon[32179]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:49.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:49 smithi070 ceph-mon[32179]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:49.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:49 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:49.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:49 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:49.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:49 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:49.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:49 smithi081 ceph-mon[35142]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:49.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:49 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:49.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:49 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:49.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:49 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:49.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:49 smithi189 ceph-mon[35753]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:49.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:49 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:49.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:49 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:49.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:49 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:51.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:51 smithi081 ceph-mon[35142]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:51.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:51 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:51.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:51 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:51.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:51 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:51.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:51 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:51.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:51 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:51.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:51 smithi189 ceph-mon[35753]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:51.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:51 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:51.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:51 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:51.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:51 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:51.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:51 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:51.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:51 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:52.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:51 smithi070 ceph-mon[32179]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:52.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:51 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:52.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:51 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:52.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:51 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:52.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:51 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:52.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:51 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:52.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:52.390+0000 7fe253fff700 1 -- 172.21.15.70:0/3696317028 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe26c0fe8f0 con 0x7fe2540390e0 2024-04-01T01:25:52.390 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 0 on host 'smithi070' 2024-04-01T01:25:52.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:52.392+0000 7fe27103f700 1 -- 172.21.15.70:0/3696317028 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2540390e0 msgr2=0x7fe25403b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:52.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:52.392+0000 7fe27103f700 1 --2- 172.21.15.70:0/3696317028 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2540390e0 0x7fe25403b5a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe25c000c00 tx=0x7fe25c005fb0 comp rx=0 tx=0).stop 2024-04-01T01:25:52.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:52.392+0000 7fe27103f700 1 -- 172.21.15.70:0/3696317028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe26c101190 msgr2=0x7fe26c1983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:52.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:52.392+0000 7fe27103f700 1 --2- 172.21.15.70:0/3696317028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe26c101190 0x7fe26c1983c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe26000e9a0 tx=0x7fe26000ecb0 comp rx=0 tx=0).stop 2024-04-01T01:25:52.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:52.392+0000 7fe27103f700 1 -- 172.21.15.70:0/3696317028 shutdown_connections 2024-04-01T01:25:52.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:52.392+0000 7fe27103f700 1 --2- 172.21.15.70:0/3696317028 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2540390e0 0x7fe25403b5a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:52.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:52.392+0000 7fe27103f700 1 --2- 172.21.15.70:0/3696317028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe26c102d30 0x7fe26c19f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:52.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:52.392+0000 7fe27103f700 1 --2- 172.21.15.70:0/3696317028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe26c101190 0x7fe26c1983c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:52.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:52.392+0000 7fe27103f700 1 --2- 172.21.15.70:0/3696317028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe26c102390 0x7fe26c198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:52.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:52.392+0000 7fe27103f700 1 -- 172.21.15.70:0/3696317028 >> 172.21.15.70:0/3696317028 conn(0x7fe26c0fc730 msgr2=0x7fe26c0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:25:52.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:52.392+0000 7fe27103f700 1 -- 172.21.15.70:0/3696317028 shutdown_connections 2024-04-01T01:25:52.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:25:52.392+0000 7fe27103f700 1 -- 172.21.15.70:0/3696317028 wait complete. 2024-04-01T01:25:52.986 DEBUG:teuthology.orchestra.run.smithi070:osd.0> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.0.service 2024-04-01T01:25:52.988 INFO:tasks.cephadm:Deploying osd.1 on smithi081 with /dev/vg_nvme/lv_4... 2024-04-01T01:25:52.988 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-01T01:25:53.231 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-01T01:25:53.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:53 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:53.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:53 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:53.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:53 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:53.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:53 smithi070 ceph-mon[32179]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:53.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:53 smithi070 ceph-mon[32179]: from='osd.0 [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-01T01:25:53.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:53 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:53.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:53 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:53.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:53 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:53.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:53 smithi081 ceph-mon[35142]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:53.652 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:53 smithi081 ceph-mon[35142]: from='osd.0 [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-01T01:25:53.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:53 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:53.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:53 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:53.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:53 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:25:53.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:53 smithi189 ceph-mon[35753]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:53.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:53 smithi189 ceph-mon[35753]: from='osd.0 [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-01T01:25:54.412 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:54 smithi070 ceph-mon[32179]: from='osd.0 [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-01T01:25:54.413 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:54 smithi070 ceph-mon[32179]: osdmap e6: 1 total, 0 up, 1 in 2024-04-01T01:25:54.413 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:54 smithi070 ceph-mon[32179]: from='osd.0 [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-01T01:25:54.413 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:54 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:54.413 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:25:54 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-0[47150]: 2024-04-01T01:25:54.354+0000 7ffad0279700 -1 osd.0 0 waiting for initial osdmap 2024-04-01T01:25:54.413 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:25:54 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-0[47150]: 2024-04-01T01:25:54.360+0000 7ffacc06c700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-01T01:25:54.573 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:54 smithi081 ceph-mon[35142]: from='osd.0 [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-01T01:25:54.573 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:54 smithi081 ceph-mon[35142]: osdmap e6: 1 total, 0 up, 1 in 2024-04-01T01:25:54.573 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:54 smithi081 ceph-mon[35142]: from='osd.0 [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-01T01:25:54.573 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:54 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:54.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:54 smithi189 ceph-mon[35753]: from='osd.0 [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-01T01:25:54.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:54 smithi189 ceph-mon[35753]: osdmap e6: 1 total, 0 up, 1 in 2024-04-01T01:25:54.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:54 smithi189 ceph-mon[35753]: from='osd.0 [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-01T01:25:54.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:54 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:55.138 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-01T01:25:55.138 INFO:teuthology.orchestra.run.smithi081.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-01T01:25:55.138 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10+0 records in 2024-04-01T01:25:55.138 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2024-04-01T01:25:55.138 INFO:teuthology.orchestra.run.smithi081.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114077 s, 919 MB/s 2024-04-01T01:25:55.138 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 2024-04-01T01:25:55.138 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping successful for: 2024-04-01T01:25:55.442 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:55 smithi081 ceph-mon[35142]: from='osd.0 [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-01T01:25:55.442 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:55 smithi081 ceph-mon[35142]: osdmap e7: 1 total, 0 up, 1 in 2024-04-01T01:25:55.442 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:55.442 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:55.442 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:55 smithi081 ceph-mon[35142]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:55.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:55 smithi070 ceph-mon[32179]: from='osd.0 [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-01T01:25:55.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:55 smithi070 ceph-mon[32179]: osdmap e7: 1 total, 0 up, 1 in 2024-04-01T01:25:55.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:55.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:55.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:55 smithi070 ceph-mon[32179]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:55.739 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi081:vg_nvme/lv_4 2024-04-01T01:25:55.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:55 smithi189 ceph-mon[35753]: from='osd.0 [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-01T01:25:55.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:55 smithi189 ceph-mon[35753]: osdmap e7: 1 total, 0 up, 1 in 2024-04-01T01:25:55.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:55.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:55.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:55 smithi189 ceph-mon[35753]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-01T01:25:55.988 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-01T01:25:56.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:56 smithi189 ceph-mon[35753]: purged_snaps scrub starts 2024-04-01T01:25:56.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:56 smithi189 ceph-mon[35753]: purged_snaps scrub ok 2024-04-01T01:25:56.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:56.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:56 smithi189 ceph-mon[35753]: osd.0 [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043] boot 2024-04-01T01:25:56.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:56 smithi189 ceph-mon[35753]: osdmap e8: 1 total, 1 up, 1 in 2024-04-01T01:25:56.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:56.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:56.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:56.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:25:56.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:56.750 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:56.750 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:56.750 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:56.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:56 smithi070 ceph-mon[32179]: purged_snaps scrub starts 2024-04-01T01:25:56.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:56 smithi070 ceph-mon[32179]: purged_snaps scrub ok 2024-04-01T01:25:56.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:56.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:56 smithi070 ceph-mon[32179]: osd.0 [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043] boot 2024-04-01T01:25:56.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:56 smithi070 ceph-mon[32179]: osdmap e8: 1 total, 1 up, 1 in 2024-04-01T01:25:56.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:56.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:56.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:56.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:25:56.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:56.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:56.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:56.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:56.810 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:56 smithi081 ceph-mon[35142]: purged_snaps scrub starts 2024-04-01T01:25:56.810 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:56 smithi081 ceph-mon[35142]: purged_snaps scrub ok 2024-04-01T01:25:56.810 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:56.810 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:56 smithi081 ceph-mon[35142]: osd.0 [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043] boot 2024-04-01T01:25:56.810 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:56 smithi081 ceph-mon[35142]: osdmap e8: 1 total, 1 up, 1 in 2024-04-01T01:25:56.810 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-01T01:25:56.810 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:56.810 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:56.810 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:25:56.810 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:56.810 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:56.811 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:25:56.811 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:25:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.625+0000 7eff9c465700 1 -- 172.21.15.81:0/3036598596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff94105700 msgr2=0x7eff94107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.625+0000 7eff9c465700 1 --2- 172.21.15.81:0/3036598596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff94105700 0x7eff94107af0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7eff84009990 tx=0x7eff84009ca0 comp rx=0 tx=0).stop 2024-04-01T01:25:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.625+0000 7eff9c465700 1 -- 172.21.15.81:0/3036598596 shutdown_connections 2024-04-01T01:25:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.625+0000 7eff9c465700 1 --2- 172.21.15.81:0/3036598596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff94108030 0x7eff9410a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.625+0000 7eff9c465700 1 --2- 172.21.15.81:0/3036598596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff94102dd0 0x7eff941051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.625+0000 7eff9c465700 1 --2- 172.21.15.81:0/3036598596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff94105700 0x7eff94107af0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:57.626 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.625+0000 7eff9c465700 1 -- 172.21.15.81:0/3036598596 >> 172.21.15.81:0/3036598596 conn(0x7eff940fc760 msgr2=0x7eff940feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:25:57.627 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:57 smithi081 ceph-mon[35142]: Detected new or changed devices on smithi070 2024-04-01T01:25:57.627 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:57 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi070 to 17088M 2024-04-01T01:25:57.627 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:57 smithi081 ceph-mon[35142]: osdmap e9: 1 total, 1 up, 1 in 2024-04-01T01:25:57.627 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:57 smithi081 ceph-mon[35142]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:25:57.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.625+0000 7eff9c465700 1 -- 172.21.15.81:0/3036598596 shutdown_connections 2024-04-01T01:25:57.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.626+0000 7eff9c465700 1 -- 172.21.15.81:0/3036598596 wait complete. 2024-04-01T01:25:57.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.626+0000 7eff9c465700 1 Processor -- start 2024-04-01T01:25:57.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.626+0000 7eff9c465700 1 -- start start 2024-04-01T01:25:57.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.626+0000 7eff9c465700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff94102dd0 0x7eff94198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:57.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.626+0000 7eff9c465700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff94105700 0x7eff94198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:57.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.626+0000 7eff9c465700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff94108030 0x7eff9419f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:57.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.626+0000 7eff9c465700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7eff9410dd60 con 0x7eff94102dd0 2024-04-01T01:25:57.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.626+0000 7eff9c465700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7eff9410dbe0 con 0x7eff94105700 2024-04-01T01:25:57.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.626+0000 7eff9c465700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7eff9410dee0 con 0x7eff94108030 2024-04-01T01:25:57.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.626+0000 7eff99a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff94105700 0x7eff94198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:57.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.627+0000 7eff99a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff94105700 0x7eff94198850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34764/0 (socket says 172.21.15.81:34764) 2024-04-01T01:25:57.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.627+0000 7eff99a00700 1 -- 172.21.15.81:0/2664489420 learned_addr learned my addr 172.21.15.81:0/2664489420 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-01T01:25:57.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.627+0000 7eff9a201700 1 --2- 172.21.15.81:0/2664489420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff94102dd0 0x7eff94198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:57.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.627+0000 7eff9aa02700 1 --2- 172.21.15.81:0/2664489420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff94108030 0x7eff9419f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:57.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.627+0000 7eff99a00700 1 -- 172.21.15.81:0/2664489420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff94108030 msgr2=0x7eff9419f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:57.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.627+0000 7eff99a00700 1 --2- 172.21.15.81:0/2664489420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff94108030 0x7eff9419f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:57.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.627+0000 7eff99a00700 1 -- 172.21.15.81:0/2664489420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff94102dd0 msgr2=0x7eff94198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:25:57.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.627+0000 7eff99a00700 1 --2- 172.21.15.81:0/2664489420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff94102dd0 0x7eff94198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:25:57.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.627+0000 7eff99a00700 1 -- 172.21.15.81:0/2664489420 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9419ff70 con 0x7eff94105700 2024-04-01T01:25:57.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.627+0000 7eff99a00700 1 --2- 172.21.15.81:0/2664489420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff94105700 0x7eff94198850 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7eff84000c00 tx=0x7eff840047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:25:57.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.627+0000 7eff8b7fe700 1 -- 172.21.15.81:0/2664489420 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff84027070 con 0x7eff94105700 2024-04-01T01:25:57.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.627+0000 7eff8b7fe700 1 -- 172.21.15.81:0/2664489420 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff840041d0 con 0x7eff94105700 2024-04-01T01:25:57.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.627+0000 7eff8b7fe700 1 -- 172.21.15.81:0/2664489420 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8400f460 con 0x7eff94105700 2024-04-01T01:25:57.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.627+0000 7eff9c465700 1 -- 172.21.15.81:0/2664489420 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff941a0200 con 0x7eff94105700 2024-04-01T01:25:57.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.627+0000 7eff9c465700 1 -- 172.21.15.81:0/2664489420 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff941a0600 con 0x7eff94105700 2024-04-01T01:25:57.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.628+0000 7eff9c465700 1 -- 172.21.15.81:0/2664489420 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff94068f40 con 0x7eff94105700 2024-04-01T01:25:57.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.629+0000 7eff8b7fe700 1 -- 172.21.15.81:0/2664489420 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 15) ==== 137264+0+0 (secure 0 0 0) 0x7eff8402a070 con 0x7eff94105700 2024-04-01T01:25:57.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.630+0000 7eff8b7fe700 1 --2- 172.21.15.81:0/2664489420 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7eff800a2520 0x7eff800a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:25:57.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.630+0000 7eff8b7fe700 1 -- 172.21.15.81:0/2664489420 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7eff840cdbd0 con 0x7eff94105700 2024-04-01T01:25:57.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.630+0000 7eff9a201700 1 --2- 172.21.15.81:0/2664489420 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7eff800a2520 0x7eff800a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:25:57.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.631+0000 7eff9a201700 1 --2- 172.21.15.81:0/2664489420 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7eff800a2520 0x7eff800a49e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7eff90005950 tx=0x7eff9000a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:25:57.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.635+0000 7eff8b7fe700 1 -- 172.21.15.81:0/2664489420 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7eff8405e280 con 0x7eff94105700 2024-04-01T01:25:57.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:57 smithi189 ceph-mon[35753]: Detected new or changed devices on smithi070 2024-04-01T01:25:57.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:57 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi070 to 17088M 2024-04-01T01:25:57.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:57 smithi189 ceph-mon[35753]: osdmap e9: 1 total, 1 up, 1 in 2024-04-01T01:25:57.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:57 smithi189 ceph-mon[35753]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:25:57.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:57 smithi070 ceph-mon[32179]: Detected new or changed devices on smithi070 2024-04-01T01:25:57.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:57 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi070 to 17088M 2024-04-01T01:25:57.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:57 smithi070 ceph-mon[32179]: osdmap e9: 1 total, 1 up, 1 in 2024-04-01T01:25:57.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:57 smithi070 ceph-mon[32179]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:25:57.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:25:57.836+0000 7eff9c465700 1 -- 172.21.15.81:0/2664489420 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7eff94062680 con 0x7eff800a2520 2024-04-01T01:25:58.650 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:58 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:25:58.650 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:58 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:25:58.650 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:58 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:58.660 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:58 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:25:58.660 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:58 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:25:58.660 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:58 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:58.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:58 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:25:58.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:58 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:25:58.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:58 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:25:59.470 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:59 smithi081 ceph-mon[35142]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:25:59.470 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:25:59 smithi081 ceph-mon[35142]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:25:59.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:59 smithi189 ceph-mon[35753]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:25:59.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:25:59 smithi189 ceph-mon[35753]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:25:59.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:59 smithi070 ceph-mon[32179]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:25:59.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:25:59 smithi070 ceph-mon[32179]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:00.672 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:00 smithi081 ceph-mon[35142]: from='client.? 172.21.15.81:0/34809111' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9df24cdb-3f8d-427c-afd8-dd579612d782"}]: dispatch 2024-04-01T01:26:00.672 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:00 smithi081 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9df24cdb-3f8d-427c-afd8-dd579612d782"}]: dispatch 2024-04-01T01:26:00.672 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:00 smithi081 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9df24cdb-3f8d-427c-afd8-dd579612d782"}]': finished 2024-04-01T01:26:00.673 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:00 smithi081 ceph-mon[35142]: osdmap e10: 2 total, 1 up, 2 in 2024-04-01T01:26:00.673 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-01T01:26:00.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:00 smithi189 ceph-mon[35753]: from='client.? 172.21.15.81:0/34809111' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9df24cdb-3f8d-427c-afd8-dd579612d782"}]: dispatch 2024-04-01T01:26:00.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:00 smithi189 ceph-mon[35753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9df24cdb-3f8d-427c-afd8-dd579612d782"}]: dispatch 2024-04-01T01:26:00.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:00 smithi189 ceph-mon[35753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9df24cdb-3f8d-427c-afd8-dd579612d782"}]': finished 2024-04-01T01:26:00.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:00 smithi189 ceph-mon[35753]: osdmap e10: 2 total, 1 up, 2 in 2024-04-01T01:26:00.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-01T01:26:00.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:00 smithi070 ceph-mon[32179]: from='client.? 172.21.15.81:0/34809111' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9df24cdb-3f8d-427c-afd8-dd579612d782"}]: dispatch 2024-04-01T01:26:00.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:00 smithi070 ceph-mon[32179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9df24cdb-3f8d-427c-afd8-dd579612d782"}]: dispatch 2024-04-01T01:26:00.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:00 smithi070 ceph-mon[32179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9df24cdb-3f8d-427c-afd8-dd579612d782"}]': finished 2024-04-01T01:26:00.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:00 smithi070 ceph-mon[32179]: osdmap e10: 2 total, 1 up, 2 in 2024-04-01T01:26:00.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-01T01:26:01.682 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:01 smithi081 ceph-mon[35142]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:01.682 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:01 smithi081 ceph-mon[35142]: from='client.? 172.21.15.81:0/2855193102' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:26:01.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:01 smithi189 ceph-mon[35753]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:01.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:01 smithi189 ceph-mon[35753]: from='client.? 172.21.15.81:0/2855193102' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:26:01.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:01 smithi070 ceph-mon[32179]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:01.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:01 smithi070 ceph-mon[32179]: from='client.? 172.21.15.81:0/2855193102' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:26:03.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:03 smithi081 ceph-mon[35142]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:03.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:03 smithi189 ceph-mon[35753]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:04.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:03 smithi070 ceph-mon[32179]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:05.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:05 smithi081 ceph-mon[35142]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:05.964 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:05 smithi070 ceph-mon[32179]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:05.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:05 smithi189 ceph-mon[35753]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:07.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:07 smithi081 ceph-mon[35142]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:07.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-01T01:26:07.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:07.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:07 smithi189 ceph-mon[35753]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:07.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-01T01:26:07.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:08.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:07 smithi070 ceph-mon[32179]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:08.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-01T01:26:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:08.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:08 smithi081 ceph-mon[35142]: Deploying daemon osd.1 on smithi081 2024-04-01T01:26:08.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:08 smithi189 ceph-mon[35753]: Deploying daemon osd.1 on smithi081 2024-04-01T01:26:09.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:08 smithi070 ceph-mon[32179]: Deploying daemon osd.1 on smithi081 2024-04-01T01:26:09.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:09 smithi189 ceph-mon[35753]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:09.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:09 smithi081 ceph-mon[35142]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:10.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:09 smithi070 ceph-mon[32179]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:11.713 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:11 smithi081 ceph-mon[35142]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:11.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:11 smithi189 ceph-mon[35753]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:12.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:11 smithi070 ceph-mon[32179]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:13.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:13 smithi081 ceph-mon[35142]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:13.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:13.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:13.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:26:13.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:13 smithi189 ceph-mon[35753]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:13.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:13 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:13.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:13 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:13.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:13 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:26:14.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:13 smithi070 ceph-mon[32179]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:14.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:13 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:14.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:13 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:14.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:13 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:26:15.691 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:15 smithi081 ceph-mon[35142]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:15.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:15 smithi189 ceph-mon[35753]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:16.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:26:15.999+0000 7eff8b7fe700 1 -- 172.21.15.81:0/2664489420 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7eff94062680 con 0x7eff800a2520 2024-04-01T01:26:16.000 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 1 on host 'smithi081' 2024-04-01T01:26:16.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:26:16.002+0000 7eff9c465700 1 -- 172.21.15.81:0/2664489420 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7eff800a2520 msgr2=0x7eff800a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:26:16.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:26:16.002+0000 7eff9c465700 1 --2- 172.21.15.81:0/2664489420 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7eff800a2520 0x7eff800a49e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7eff90005950 tx=0x7eff9000a300 comp rx=0 tx=0).stop 2024-04-01T01:26:16.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:26:16.002+0000 7eff9c465700 1 -- 172.21.15.81:0/2664489420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff94105700 msgr2=0x7eff94198850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:26:16.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:26:16.002+0000 7eff9c465700 1 --2- 172.21.15.81:0/2664489420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff94105700 0x7eff94198850 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7eff84000c00 tx=0x7eff840047b0 comp rx=0 tx=0).stop 2024-04-01T01:26:16.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:26:16.002+0000 7eff9c465700 1 -- 172.21.15.81:0/2664489420 shutdown_connections 2024-04-01T01:26:16.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:26:16.002+0000 7eff9c465700 1 --2- 172.21.15.81:0/2664489420 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7eff800a2520 0x7eff800a49e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:16.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:26:16.002+0000 7eff9c465700 1 --2- 172.21.15.81:0/2664489420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff94108030 0x7eff9419f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:16.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:26:16.002+0000 7eff9c465700 1 --2- 172.21.15.81:0/2664489420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff94102dd0 0x7eff94198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:16.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:26:16.002+0000 7eff9c465700 1 --2- 172.21.15.81:0/2664489420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff94105700 0x7eff94198850 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:16.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:26:16.002+0000 7eff9c465700 1 -- 172.21.15.81:0/2664489420 >> 172.21.15.81:0/2664489420 conn(0x7eff940fc760 msgr2=0x7eff94103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:26:16.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:26:16.002+0000 7eff9c465700 1 -- 172.21.15.81:0/2664489420 shutdown_connections 2024-04-01T01:26:16.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:26:16.002+0000 7eff9c465700 1 -- 172.21.15.81:0/2664489420 wait complete. 2024-04-01T01:26:16.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:15 smithi070 ceph-mon[32179]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:16.546 DEBUG:teuthology.orchestra.run.smithi081:osd.1> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.1.service 2024-04-01T01:26:16.549 INFO:tasks.cephadm:Deploying osd.2 on smithi189 with /dev/vg_nvme/lv_4... 2024-04-01T01:26:16.549 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-01T01:26:16.729 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:16.729 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:16.729 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:16.729 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:26:16.729 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:16.729 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:16.729 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:16.729 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:26:16.729 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:16 smithi081 ceph-mon[35142]: from='osd.1 [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-01T01:26:16.729 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:16 smithi081 ceph-mon[35142]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-01T01:26:16.793 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-01T01:26:16.897 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:16.897 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:16.897 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:16.897 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:26:16.897 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:16.897 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:16.897 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:16.897 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:26:16.897 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:16 smithi189 ceph-mon[35753]: from='osd.1 [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-01T01:26:16.897 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:16 smithi189 ceph-mon[35753]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-01T01:26:17.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:17.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:17.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:17.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:26:17.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:17.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:17.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:17.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:26:17.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:16 smithi070 ceph-mon[32179]: from='osd.1 [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-01T01:26:17.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:16 smithi070 ceph-mon[32179]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-01T01:26:17.754 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:17 smithi081 ceph-mon[35142]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:17.754 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:17 smithi081 ceph-mon[35142]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-01T01:26:17.754 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:17 smithi081 ceph-mon[35142]: from='osd.1 [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:26:17.754 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:17 smithi081 ceph-mon[35142]: osdmap e11: 2 total, 1 up, 2 in 2024-04-01T01:26:17.754 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:17 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-01T01:26:17.754 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:17 smithi081 ceph-mon[35142]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:26:17.900 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:17 smithi189 ceph-mon[35753]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:17.900 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:17 smithi189 ceph-mon[35753]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-01T01:26:17.901 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:17 smithi189 ceph-mon[35753]: from='osd.1 [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:26:17.901 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:17 smithi189 ceph-mon[35753]: osdmap e11: 2 total, 1 up, 2 in 2024-04-01T01:26:17.901 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:17 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-01T01:26:17.901 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:17 smithi189 ceph-mon[35753]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:26:18.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:17 smithi070 ceph-mon[32179]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:18.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:17 smithi070 ceph-mon[32179]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-01T01:26:18.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:17 smithi070 ceph-mon[32179]: from='osd.1 [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:26:18.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:17 smithi070 ceph-mon[32179]: osdmap e11: 2 total, 1 up, 2 in 2024-04-01T01:26:18.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:17 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-01T01:26:18.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:17 smithi070 ceph-mon[32179]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:26:18.328 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:26:18 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-1[39718]: 2024-04-01T01:26:18.004+0000 7f66ecd87700 -1 osd.1 0 waiting for initial osdmap 2024-04-01T01:26:18.328 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:26:18 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-1[39718]: 2024-04-01T01:26:18.007+0000 7f66e8b7a700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-01T01:26:18.604 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-01T01:26:18.605 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-01T01:26:18.605 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-04-01T01:26:18.605 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-04-01T01:26:18.605 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117201 s, 895 MB/s 2024-04-01T01:26:18.605 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-04-01T01:26:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:19 smithi081 ceph-mon[35142]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-01T01:26:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:19 smithi081 ceph-mon[35142]: osdmap e12: 2 total, 1 up, 2 in 2024-04-01T01:26:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-01T01:26:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-01T01:26:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:19 smithi081 ceph-mon[35142]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:19.192 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:19 smithi189 ceph-mon[35753]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-01T01:26:19.193 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:19 smithi189 ceph-mon[35753]: osdmap e12: 2 total, 1 up, 2 in 2024-04-01T01:26:19.193 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-01T01:26:19.193 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-01T01:26:19.193 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:19 smithi189 ceph-mon[35753]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:19.194 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi189:vg_nvme/lv_4 2024-04-01T01:26:19.438 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-01T01:26:19.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:19 smithi070 ceph-mon[32179]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-01T01:26:19.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:19 smithi070 ceph-mon[32179]: osdmap e12: 2 total, 1 up, 2 in 2024-04-01T01:26:19.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-01T01:26:19.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-01T01:26:19.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:19 smithi070 ceph-mon[32179]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-01T01:26:20.197 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:20 smithi189 ceph-mon[35753]: purged_snaps scrub starts 2024-04-01T01:26:20.198 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:20 smithi189 ceph-mon[35753]: purged_snaps scrub ok 2024-04-01T01:26:20.198 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:20 smithi189 ceph-mon[35753]: osd.1 [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424] boot 2024-04-01T01:26:20.198 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:20 smithi189 ceph-mon[35753]: osdmap e13: 2 total, 2 up, 2 in 2024-04-01T01:26:20.198 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:20 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-01T01:26:20.198 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:20 smithi189 ceph-mon[35753]: Detected new or changed devices on smithi081 2024-04-01T01:26:20.198 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:20 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:20.198 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:20 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:20.198 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:20 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:26:20.198 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:20 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi081 to 17088M 2024-04-01T01:26:20.198 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:20 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:20.198 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:20 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:20.198 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:20 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:26:20.198 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:20 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:20.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:20 smithi081 ceph-mon[35142]: purged_snaps scrub starts 2024-04-01T01:26:20.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:20 smithi081 ceph-mon[35142]: purged_snaps scrub ok 2024-04-01T01:26:20.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:20 smithi081 ceph-mon[35142]: osd.1 [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424] boot 2024-04-01T01:26:20.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:20 smithi081 ceph-mon[35142]: osdmap e13: 2 total, 2 up, 2 in 2024-04-01T01:26:20.330 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-01T01:26:20.330 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:20 smithi081 ceph-mon[35142]: Detected new or changed devices on smithi081 2024-04-01T01:26:20.330 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:20.330 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:20.330 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:26:20.330 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:20 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi081 to 17088M 2024-04-01T01:26:20.330 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:20.330 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:20.330 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:26:20.330 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:20.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:20 smithi070 ceph-mon[32179]: purged_snaps scrub starts 2024-04-01T01:26:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:20 smithi070 ceph-mon[32179]: purged_snaps scrub ok 2024-04-01T01:26:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:20 smithi070 ceph-mon[32179]: osd.1 [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424] boot 2024-04-01T01:26:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:20 smithi070 ceph-mon[32179]: osdmap e13: 2 total, 2 up, 2 in 2024-04-01T01:26:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:20 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-01T01:26:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:20 smithi070 ceph-mon[32179]: Detected new or changed devices on smithi081 2024-04-01T01:26:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:20 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:20 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:20 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:26:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:20 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi081 to 17088M 2024-04-01T01:26:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:20 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:20 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:20 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:26:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:20 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:20.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.905+0000 7f9440706700 1 -- 172.21.15.189:0/1443590322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438102260 msgr2=0x7f9438102680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:26:20.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.905+0000 7f9440706700 1 --2- 172.21.15.189:0/1443590322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438102260 0x7f9438102680 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9434009990 tx=0x7f9434009ca0 comp rx=0 tx=0).stop 2024-04-01T01:26:20.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.905+0000 7f9440706700 1 -- 172.21.15.189:0/1443590322 shutdown_connections 2024-04-01T01:26:20.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.905+0000 7f9440706700 1 --2- 172.21.15.189:0/1443590322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9438103e00 0x7f943810a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:20.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.905+0000 7f9440706700 1 --2- 172.21.15.189:0/1443590322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438102260 0x7f9438102680 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:20.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.905+0000 7f9440706700 1 --2- 172.21.15.189:0/1443590322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9438103460 0x7f94381038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:20.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.905+0000 7f9440706700 1 -- 172.21.15.189:0/1443590322 >> 172.21.15.189:0/1443590322 conn(0x7f94380fd820 msgr2=0x7f94380ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:26:20.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.905+0000 7f9440706700 1 -- 172.21.15.189:0/1443590322 shutdown_connections 2024-04-01T01:26:20.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.905+0000 7f9440706700 1 -- 172.21.15.189:0/1443590322 wait complete. 2024-04-01T01:26:20.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.905+0000 7f9440706700 1 Processor -- start 2024-04-01T01:26:20.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.905+0000 7f9440706700 1 -- start start 2024-04-01T01:26:20.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f9440706700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438102260 0x7f9438198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:26:20.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f9440706700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9438103460 0x7f9438198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:26:20.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f9440706700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9438103e00 0x7f943819fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:26:20.907 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f9440706700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f943810dd40 con 0x7f9438103e00 2024-04-01T01:26:20.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f9440706700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f943810dbc0 con 0x7f9438103460 2024-04-01T01:26:20.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f9440706700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f943810dec0 con 0x7f9438102260 2024-04-01T01:26:20.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f943e4a2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438102260 0x7f9438198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:26:20.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f943e4a2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438102260 0x7f9438198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49694/0 (socket says 172.21.15.189:49694) 2024-04-01T01:26:20.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f943e4a2700 1 -- 172.21.15.189:0/178680326 learned_addr learned my addr 172.21.15.189:0/178680326 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-01T01:26:20.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f943eca3700 1 --2- 172.21.15.189:0/178680326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9438103e00 0x7f943819fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:26:20.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f943dca1700 1 --2- 172.21.15.189:0/178680326 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9438103460 0x7f9438198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:26:20.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f943e4a2700 1 -- 172.21.15.189:0/178680326 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9438103460 msgr2=0x7f9438198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:26:20.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f943e4a2700 1 --2- 172.21.15.189:0/178680326 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9438103460 0x7f9438198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:20.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f943e4a2700 1 -- 172.21.15.189:0/178680326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9438103e00 msgr2=0x7f943819fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:26:20.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f943e4a2700 1 --2- 172.21.15.189:0/178680326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9438103e00 0x7f943819fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:20.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f943e4a2700 1 -- 172.21.15.189:0/178680326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94381a00a0 con 0x7f9438102260 2024-04-01T01:26:20.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f943eca3700 1 --2- 172.21.15.189:0/178680326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9438103e00 0x7f943819fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:26:20.908 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f943e4a2700 1 --2- 172.21.15.189:0/178680326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438102260 0x7f9438198440 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9434000c00 tx=0x7f9434003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:26:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f942f7fe700 1 -- 172.21.15.189:0/178680326 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9434027070 con 0x7f9438102260 2024-04-01T01:26:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f942f7fe700 1 -- 172.21.15.189:0/178680326 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9434004080 con 0x7f9438102260 2024-04-01T01:26:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f942f7fe700 1 -- 172.21.15.189:0/178680326 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9434022da0 con 0x7f9438102260 2024-04-01T01:26:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.906+0000 7f9440706700 1 -- 172.21.15.189:0/178680326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94381a0330 con 0x7f9438102260 2024-04-01T01:26:20.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.907+0000 7f9440706700 1 -- 172.21.15.189:0/178680326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94381a0730 con 0x7f9438102260 2024-04-01T01:26:20.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.907+0000 7f9440706700 1 -- 172.21.15.189:0/178680326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9438068f40 con 0x7f9438102260 2024-04-01T01:26:20.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.909+0000 7f942f7fe700 1 -- 172.21.15.189:0/178680326 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) ==== 137264+0+0 (secure 0 0 0) 0x7f943402a070 con 0x7f9438102260 2024-04-01T01:26:20.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.909+0000 7f942f7fe700 1 --2- 172.21.15.189:0/178680326 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94240a2520 0x7f94240a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:26:20.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.909+0000 7f942f7fe700 1 -- 172.21.15.189:0/178680326 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f94340ce520 con 0x7f9438102260 2024-04-01T01:26:20.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.910+0000 7f943dca1700 1 --2- 172.21.15.189:0/178680326 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94240a2520 0x7f94240a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:26:20.910 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.910+0000 7f943dca1700 1 --2- 172.21.15.189:0/178680326 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94240a2520 0x7f94240a49e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9428005950 tx=0x7f9428005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:26:20.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:20.913+0000 7f942f7fe700 1 -- 172.21.15.189:0/178680326 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9434031020 con 0x7f9438102260 2024-04-01T01:26:21.118 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:21.118+0000 7f9440706700 1 -- 172.21.15.189:0/178680326 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9438062680 con 0x7f94240a2520 2024-04-01T01:26:21.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:21 smithi081 ceph-mon[35142]: osdmap e14: 2 total, 2 up, 2 in 2024-04-01T01:26:21.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:21 smithi081 ceph-mon[35142]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:21.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:21 smithi189 ceph-mon[35753]: osdmap e14: 2 total, 2 up, 2 in 2024-04-01T01:26:21.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:21 smithi189 ceph-mon[35753]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:21.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:21 smithi070 ceph-mon[32179]: osdmap e14: 2 total, 2 up, 2 in 2024-04-01T01:26:21.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:21 smithi070 ceph-mon[32179]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:22.216 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:22 smithi189 ceph-mon[35753]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:26:22.216 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:26:22.216 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:26:22.216 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:22.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:22 smithi081 ceph-mon[35142]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:26:22.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:26:22.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:26:22.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:22.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:22 smithi070 ceph-mon[32179]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:26:22.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:26:22.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:26:22.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:23.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:23 smithi081 ceph-mon[35142]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:23.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:23 smithi189 ceph-mon[35753]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:23.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:23 smithi070 ceph-mon[32179]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:24.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:24 smithi189 ceph-mon[35753]: from='client.? 172.21.15.189:0/2116066636' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800721f9-d63a-4bea-8bbb-ae09c8c161be"}]: dispatch 2024-04-01T01:26:24.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:24 smithi189 ceph-mon[35753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800721f9-d63a-4bea-8bbb-ae09c8c161be"}]: dispatch 2024-04-01T01:26:24.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:24 smithi189 ceph-mon[35753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "800721f9-d63a-4bea-8bbb-ae09c8c161be"}]': finished 2024-04-01T01:26:24.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:24 smithi189 ceph-mon[35753]: osdmap e15: 3 total, 2 up, 3 in 2024-04-01T01:26:24.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:24 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-01T01:26:24.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:24 smithi081 ceph-mon[35142]: from='client.? 172.21.15.189:0/2116066636' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800721f9-d63a-4bea-8bbb-ae09c8c161be"}]: dispatch 2024-04-01T01:26:24.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:24 smithi081 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800721f9-d63a-4bea-8bbb-ae09c8c161be"}]: dispatch 2024-04-01T01:26:24.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:24 smithi081 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "800721f9-d63a-4bea-8bbb-ae09c8c161be"}]': finished 2024-04-01T01:26:24.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:24 smithi081 ceph-mon[35142]: osdmap e15: 3 total, 2 up, 3 in 2024-04-01T01:26:24.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:24 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-01T01:26:24.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:24 smithi070 ceph-mon[32179]: from='client.? 172.21.15.189:0/2116066636' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800721f9-d63a-4bea-8bbb-ae09c8c161be"}]: dispatch 2024-04-01T01:26:24.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:24 smithi070 ceph-mon[32179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "800721f9-d63a-4bea-8bbb-ae09c8c161be"}]: dispatch 2024-04-01T01:26:24.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:24 smithi070 ceph-mon[32179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "800721f9-d63a-4bea-8bbb-ae09c8c161be"}]': finished 2024-04-01T01:26:24.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:24 smithi070 ceph-mon[32179]: osdmap e15: 3 total, 2 up, 3 in 2024-04-01T01:26:24.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:24 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-01T01:26:25.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:25 smithi081 ceph-mon[35142]: from='client.? 172.21.15.189:0/932811823' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:26:25.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:25 smithi081 ceph-mon[35142]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:25.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:25 smithi189 ceph-mon[35753]: from='client.? 172.21.15.189:0/932811823' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:26:25.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:25 smithi189 ceph-mon[35753]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:25 smithi070 ceph-mon[32179]: from='client.? 172.21.15.189:0/932811823' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:26:25.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:25 smithi070 ceph-mon[32179]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:27 smithi081 ceph-mon[35142]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:27.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:27 smithi189 ceph-mon[35753]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:28.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:27 smithi070 ceph-mon[32179]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:29.701 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:29 smithi189 ceph-mon[35753]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:29.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:29 smithi081 ceph-mon[35142]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:30.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:29 smithi070 ceph-mon[32179]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:30.768 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:30 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-01T01:26:30.768 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:30 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:30.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:30 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-01T01:26:30.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:30 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:31.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:30 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-01T01:26:31.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:30 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:31.735 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:31 smithi189 ceph-mon[35753]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:31.735 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:31 smithi189 ceph-mon[35753]: Deploying daemon osd.2 on smithi189 2024-04-01T01:26:31.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:31 smithi081 ceph-mon[35142]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:31.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:31 smithi081 ceph-mon[35142]: Deploying daemon osd.2 on smithi189 2024-04-01T01:26:32.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:31 smithi070 ceph-mon[32179]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:32.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:31 smithi070 ceph-mon[32179]: Deploying daemon osd.2 on smithi189 2024-04-01T01:26:33.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:33 smithi189 ceph-mon[35753]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:33.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:33 smithi081 ceph-mon[35142]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:34.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:33 smithi070 ceph-mon[32179]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:35.661 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:35 smithi189 ceph-mon[35753]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:35.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:35 smithi081 ceph-mon[35142]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:36.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:35 smithi070 ceph-mon[32179]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:36.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:36 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:26:36.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:36 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:36.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:36 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:36.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:36 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:26:36.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:36 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:36.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:36 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:37.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:36 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:26:37.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:36 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:37.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:36 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:37.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:37 smithi081 ceph-mon[35142]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:37.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:37 smithi189 ceph-mon[35753]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:38.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:37 smithi070 ceph-mon[32179]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:39.131 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:39.131+0000 7f942f7fe700 1 -- 172.21.15.189:0/178680326 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9438062680 con 0x7f94240a2520 2024-04-01T01:26:39.131 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 2 on host 'smithi189' 2024-04-01T01:26:39.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:39.133+0000 7f9440706700 1 -- 172.21.15.189:0/178680326 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94240a2520 msgr2=0x7f94240a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:26:39.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:39.133+0000 7f9440706700 1 --2- 172.21.15.189:0/178680326 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94240a2520 0x7f94240a49e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9428005950 tx=0x7f9428005eb0 comp rx=0 tx=0).stop 2024-04-01T01:26:39.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:39.133+0000 7f9440706700 1 -- 172.21.15.189:0/178680326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438102260 msgr2=0x7f9438198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:26:39.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:39.133+0000 7f9440706700 1 --2- 172.21.15.189:0/178680326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438102260 0x7f9438198440 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9434000c00 tx=0x7f9434003680 comp rx=0 tx=0).stop 2024-04-01T01:26:39.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:39.133+0000 7f9440706700 1 -- 172.21.15.189:0/178680326 shutdown_connections 2024-04-01T01:26:39.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:39.133+0000 7f9440706700 1 --2- 172.21.15.189:0/178680326 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94240a2520 0x7f94240a49e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:39.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:39.133+0000 7f9440706700 1 --2- 172.21.15.189:0/178680326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9438103e00 0x7f943819fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:39.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:39.133+0000 7f9440706700 1 --2- 172.21.15.189:0/178680326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9438102260 0x7f9438198440 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:39.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:39.133+0000 7f9440706700 1 --2- 172.21.15.189:0/178680326 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9438103460 0x7f9438198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:39.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:39.133+0000 7f9440706700 1 -- 172.21.15.189:0/178680326 >> 172.21.15.189:0/178680326 conn(0x7f94380fd820 msgr2=0x7f9438106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:26:39.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:39.134+0000 7f9440706700 1 -- 172.21.15.189:0/178680326 shutdown_connections 2024-04-01T01:26:39.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:26:39.134+0000 7f9440706700 1 -- 172.21.15.189:0/178680326 wait complete. 2024-04-01T01:26:39.439 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:39 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:39.440 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:39 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:39.440 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:39 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:39.440 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:39 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:26:39.440 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:39 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:39.440 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:39 smithi189 ceph-mon[35753]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:39.440 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:39 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:39.440 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:39 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:39.440 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:39 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:26:39.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:39 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:39.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:39 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:39.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:39 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:39.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:39 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:26:39.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:39 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:39.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:39 smithi070 ceph-mon[32179]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:39.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:39 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:39.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:39 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:39.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:39 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:26:39.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:39 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:39.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:39 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:39.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:39 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:39.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:39 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:26:39.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:39 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:39.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:39 smithi081 ceph-mon[35142]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:39.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:39 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:39.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:39 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:39.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:39 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:26:39.741 DEBUG:teuthology.orchestra.run.smithi189:osd.2> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.2.service 2024-04-01T01:26:39.743 INFO:tasks.cephadm:Deploying osd.3 on smithi070 with /dev/vg_nvme/lv_3... 2024-04-01T01:26:39.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-01T01:26:39.989 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:26:40.453 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:40 smithi070 ceph-mon[32179]: from='osd.2 [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-01T01:26:40.453 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:40 smithi070 ceph-mon[32179]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-01T01:26:40.480 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:40 smithi189 ceph-mon[35753]: from='osd.2 [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-01T01:26:40.480 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:40 smithi189 ceph-mon[35753]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-01T01:26:40.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:40 smithi081 ceph-mon[35142]: from='osd.2 [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-01T01:26:40.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:40 smithi081 ceph-mon[35142]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-01T01:26:41.266 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:41 smithi189 ceph-mon[35753]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-01T01:26:41.266 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:41 smithi189 ceph-mon[35753]: osdmap e16: 3 total, 2 up, 3 in 2024-04-01T01:26:41.266 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:41 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-01T01:26:41.266 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:41 smithi189 ceph-mon[35753]: from='osd.2 [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:26:41.266 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:41 smithi189 ceph-mon[35753]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:26:41.267 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:41 smithi189 ceph-mon[35753]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:41.341 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:41 smithi070 ceph-mon[32179]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-01T01:26:41.342 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:41 smithi070 ceph-mon[32179]: osdmap e16: 3 total, 2 up, 3 in 2024-04-01T01:26:41.342 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:41 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-01T01:26:41.342 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:41 smithi070 ceph-mon[32179]: from='osd.2 [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:26:41.342 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:41 smithi070 ceph-mon[32179]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:26:41.342 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:41 smithi070 ceph-mon[32179]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:41.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:41 smithi081 ceph-mon[35142]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-01T01:26:41.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:41 smithi081 ceph-mon[35142]: osdmap e16: 3 total, 2 up, 3 in 2024-04-01T01:26:41.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:41 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-01T01:26:41.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:41 smithi081 ceph-mon[35142]: from='osd.2 [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:26:41.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:41 smithi081 ceph-mon[35142]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:26:41.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:41 smithi081 ceph-mon[35142]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:41.748 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:26:41 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-2[40897]: 2024-04-01T01:26:41.265+0000 7fc32b232700 -1 osd.2 0 waiting for initial osdmap 2024-04-01T01:26:41.749 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:26:41 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-2[40897]: 2024-04-01T01:26:41.269+0000 7fc327025700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-01T01:26:41.910 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-01T01:26:41.910 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-01T01:26:41.910 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-04-01T01:26:41.910 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-04-01T01:26:41.910 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00864067 s, 1.2 GB/s 2024-04-01T01:26:41.910 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-04-01T01:26:42.388 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:42 smithi189 ceph-mon[35753]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-01T01:26:42.388 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:42 smithi189 ceph-mon[35753]: osdmap e17: 3 total, 2 up, 3 in 2024-04-01T01:26:42.388 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:42 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-01T01:26:42.388 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:42 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-01T01:26:42.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:42 smithi070 ceph-mon[32179]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-01T01:26:42.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:42 smithi070 ceph-mon[32179]: osdmap e17: 3 total, 2 up, 3 in 2024-04-01T01:26:42.468 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:42 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-01T01:26:42.468 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:42 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-01T01:26:42.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:42 smithi081 ceph-mon[35142]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-01T01:26:42.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:42 smithi081 ceph-mon[35142]: osdmap e17: 3 total, 2 up, 3 in 2024-04-01T01:26:42.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:42 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-01T01:26:42.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:42 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-01T01:26:42.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi070:vg_nvme/lv_3 2024-04-01T01:26:42.823 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:26:43.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:43 smithi070 ceph-mon[32179]: purged_snaps scrub starts 2024-04-01T01:26:43.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:43 smithi070 ceph-mon[32179]: purged_snaps scrub ok 2024-04-01T01:26:43.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:43 smithi070 ceph-mon[32179]: osd.2 [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460] boot 2024-04-01T01:26:43.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:43 smithi070 ceph-mon[32179]: osdmap e18: 3 total, 3 up, 3 in 2024-04-01T01:26:43.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-01T01:26:43.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:43 smithi070 ceph-mon[32179]: Detected new or changed devices on smithi189 2024-04-01T01:26:43.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:43.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:43.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:26:43.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:43 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi189 to 17088M 2024-04-01T01:26:43.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:43.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:43.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:26:43.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:43.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:43 smithi070 ceph-mon[32179]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:43.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-01T01:26:43.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:43 smithi081 ceph-mon[35142]: purged_snaps scrub starts 2024-04-01T01:26:43.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:43 smithi081 ceph-mon[35142]: purged_snaps scrub ok 2024-04-01T01:26:43.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:43 smithi081 ceph-mon[35142]: osd.2 [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460] boot 2024-04-01T01:26:43.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:43 smithi081 ceph-mon[35142]: osdmap e18: 3 total, 3 up, 3 in 2024-04-01T01:26:43.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-01T01:26:43.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:43 smithi081 ceph-mon[35142]: Detected new or changed devices on smithi189 2024-04-01T01:26:43.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:43.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:43.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:26:43.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:43 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi189 to 17088M 2024-04-01T01:26:43.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:43.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:43.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:26:43.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:43.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:43 smithi081 ceph-mon[35142]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:43.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-01T01:26:43.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:43 smithi189 ceph-mon[35753]: purged_snaps scrub starts 2024-04-01T01:26:43.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:43 smithi189 ceph-mon[35753]: purged_snaps scrub ok 2024-04-01T01:26:43.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:43 smithi189 ceph-mon[35753]: osd.2 [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460] boot 2024-04-01T01:26:43.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:43 smithi189 ceph-mon[35753]: osdmap e18: 3 total, 3 up, 3 in 2024-04-01T01:26:43.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-01T01:26:43.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:43 smithi189 ceph-mon[35753]: Detected new or changed devices on smithi189 2024-04-01T01:26:43.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:43.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:43.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:26:43.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:43 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi189 to 17088M 2024-04-01T01:26:43.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:43.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:43.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:26:43.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:43.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:43 smithi189 ceph-mon[35753]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-01T01:26:43.750 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-01T01:26:44.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.239+0000 7f8cb6465700 1 -- 172.21.15.70:0/821678373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb01023b0 msgr2=0x7f8cb0102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:26:44.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.239+0000 7f8cb6465700 1 --2- 172.21.15.70:0/821678373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb01023b0 0x7f8cb0102810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8ca40099c0 tx=0x7f8ca4009cd0 comp rx=0 tx=0).stop 2024-04-01T01:26:44.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.239+0000 7f8cb6465700 1 -- 172.21.15.70:0/821678373 shutdown_connections 2024-04-01T01:26:44.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.239+0000 7f8cb6465700 1 --2- 172.21.15.70:0/821678373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cb0102d50 0x7f8cb0109600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:44.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.239+0000 7f8cb6465700 1 --2- 172.21.15.70:0/821678373 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8cb01011b0 0x7f8cb01015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:44.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.239+0000 7f8cb6465700 1 --2- 172.21.15.70:0/821678373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb01023b0 0x7f8cb0102810 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:44.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.239+0000 7f8cb6465700 1 -- 172.21.15.70:0/821678373 >> 172.21.15.70:0/821678373 conn(0x7f8cb00fc730 msgr2=0x7f8cb00feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:26:44.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.239+0000 7f8cb6465700 1 -- 172.21.15.70:0/821678373 shutdown_connections 2024-04-01T01:26:44.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.239+0000 7f8cb6465700 1 -- 172.21.15.70:0/821678373 wait complete. 2024-04-01T01:26:44.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.240+0000 7f8cb6465700 1 Processor -- start 2024-04-01T01:26:44.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.240+0000 7f8cb6465700 1 -- start start 2024-04-01T01:26:44.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.240+0000 7f8cb6465700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8cb01011b0 0x7f8cb01972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:26:44.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.240+0000 7f8cb6465700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb01023b0 0x7f8cb0197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:26:44.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.240+0000 7f8cb6465700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cb0102d50 0x7f8cb019e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:26:44.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.240+0000 7f8cb6465700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb010ccb0 con 0x7f8cb01023b0 2024-04-01T01:26:44.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.240+0000 7f8cb6465700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb010cb30 con 0x7f8cb01011b0 2024-04-01T01:26:44.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.240+0000 7f8cb6465700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb010ce30 con 0x7f8cb0102d50 2024-04-01T01:26:44.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.241+0000 7f8caffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8cb01011b0 0x7f8cb01972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:26:44.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.241+0000 7f8caf7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb01023b0 0x7f8cb0197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:26:44.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.241+0000 7f8caffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8cb01011b0 0x7f8cb01972f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33398/0 (socket says 172.21.15.70:33398) 2024-04-01T01:26:44.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.241+0000 7f8caffff700 1 -- 172.21.15.70:0/3182825110 learned_addr learned my addr 172.21.15.70:0/3182825110 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:26:44.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.241+0000 7f8cb4a02700 1 --2- 172.21.15.70:0/3182825110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cb0102d50 0x7f8cb019e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:26:44.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.241+0000 7f8caf7fe700 1 -- 172.21.15.70:0/3182825110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cb0102d50 msgr2=0x7f8cb019e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:26:44.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.241+0000 7f8caf7fe700 1 --2- 172.21.15.70:0/3182825110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cb0102d50 0x7f8cb019e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:44.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.241+0000 7f8caf7fe700 1 -- 172.21.15.70:0/3182825110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8cb01011b0 msgr2=0x7f8cb01972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:26:44.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.241+0000 7f8caf7fe700 1 --2- 172.21.15.70:0/3182825110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8cb01011b0 0x7f8cb01972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:26:44.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.241+0000 7f8caf7fe700 1 -- 172.21.15.70:0/3182825110 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb019ef50 con 0x7f8cb01023b0 2024-04-01T01:26:44.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.241+0000 7f8caf7fe700 1 --2- 172.21.15.70:0/3182825110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb01023b0 0x7f8cb0197830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8ca40099c0 tx=0x7f8ca4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:26:44.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.241+0000 7f8cad7fa700 1 -- 172.21.15.70:0/3182825110 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca4027070 con 0x7f8cb01023b0 2024-04-01T01:26:44.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.241+0000 7f8cb6465700 1 -- 172.21.15.70:0/3182825110 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cb019f1e0 con 0x7f8cb01023b0 2024-04-01T01:26:44.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.241+0000 7f8cad7fa700 1 -- 172.21.15.70:0/3182825110 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8ca4004050 con 0x7f8cb01023b0 2024-04-01T01:26:44.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.241+0000 7f8cad7fa700 1 -- 172.21.15.70:0/3182825110 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca4022bb0 con 0x7f8cb01023b0 2024-04-01T01:26:44.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.241+0000 7f8cb6465700 1 -- 172.21.15.70:0/3182825110 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cb019f740 con 0x7f8cb01023b0 2024-04-01T01:26:44.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.242+0000 7f8cb6465700 1 -- 172.21.15.70:0/3182825110 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cb0068f00 con 0x7f8cb01023b0 2024-04-01T01:26:44.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.243+0000 7f8cad7fa700 1 -- 172.21.15.70:0/3182825110 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 15) ==== 137264+0+0 (secure 0 0 0) 0x7f8ca402a070 con 0x7f8cb01023b0 2024-04-01T01:26:44.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.244+0000 7f8cad7fa700 1 --2- 172.21.15.70:0/3182825110 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c980a24a0 0x7f8c980a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:26:44.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.244+0000 7f8caffff700 1 --2- 172.21.15.70:0/3182825110 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c980a24a0 0x7f8c980a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:26:44.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.244+0000 7f8cad7fa700 1 -- 172.21.15.70:0/3182825110 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7f8ca40ce0e0 con 0x7f8cb01023b0 2024-04-01T01:26:44.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.244+0000 7f8caffff700 1 --2- 172.21.15.70:0/3182825110 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c980a24a0 0x7f8c980a4960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8ca000bd70 tx=0x7f8ca000b4a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:26:44.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.248+0000 7f8cad7fa700 1 -- 172.21.15.70:0/3182825110 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8ca4013070 con 0x7f8cb01023b0 2024-04-01T01:26:44.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:44.453+0000 7f8cb6465700 1 -- 172.21.15.70:0/3182825110 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f8cb0062740 con 0x7f8c980a24a0 2024-04-01T01:26:44.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:44 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-01T01:26:44.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:44 smithi189 ceph-mon[35753]: osdmap e19: 3 total, 3 up, 3 in 2024-04-01T01:26:44.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:44 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-01T01:26:44.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:44 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-01T01:26:44.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:44 smithi070 ceph-mon[32179]: osdmap e19: 3 total, 3 up, 3 in 2024-04-01T01:26:44.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:44 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-01T01:26:44.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:44 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-01T01:26:44.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:44 smithi081 ceph-mon[35142]: osdmap e19: 3 total, 3 up, 3 in 2024-04-01T01:26:44.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:44 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-01T01:26:45.065 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:26:44 smithi070 sudo[51742]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-01T01:26:45.065 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:26:44 smithi070 sudo[51742]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-01T01:26:45.065 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:26:44 smithi070 sudo[51742]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-01T01:26:45.065 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:26:44 smithi070 sudo[51742]: pam_unix(sudo:session): session closed for user root 2024-04-01T01:26:45.343 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:26:45 smithi070 sudo[51802]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-01T01:26:45.343 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:26:45 smithi070 sudo[51802]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-01T01:26:45.343 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:26:45 smithi070 sudo[51802]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-01T01:26:45.344 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:26:45 smithi070 sudo[51802]: pam_unix(sudo:session): session closed for user root 2024-04-01T01:26:45.583 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:26:45 smithi081 sudo[43395]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-01T01:26:45.584 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:26:45 smithi081 sudo[43395]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-01T01:26:45.584 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:26:45 smithi081 sudo[43395]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-01T01:26:45.584 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:26:45 smithi081 sudo[43395]: pam_unix(sudo:session): session closed for user root 2024-04-01T01:26:45.584 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-01T01:26:45.584 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:45 smithi081 ceph-mon[35142]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:26:45.584 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:45 smithi081 ceph-mon[35142]: osdmap e20: 3 total, 3 up, 3 in 2024-04-01T01:26:45.584 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:26:45.584 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:26:45.584 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:45.584 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:45 smithi081 ceph-mon[35142]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:45.584 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:45.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-01T01:26:45.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:45 smithi189 ceph-mon[35753]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:26:45.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:45 smithi189 ceph-mon[35753]: osdmap e20: 3 total, 3 up, 3 in 2024-04-01T01:26:45.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:26:45.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:26:45.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:45.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:45 smithi189 ceph-mon[35753]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:45.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:45.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-01T01:26:45.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:45 smithi070 ceph-mon[32179]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:26:45.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:45 smithi070 ceph-mon[32179]: osdmap e20: 3 total, 3 up, 3 in 2024-04-01T01:26:45.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:26:45.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:26:45.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:45.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:45 smithi070 ceph-mon[32179]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:45.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:46.078 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:26:45 smithi081 sudo[43398]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-01T01:26:46.078 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:26:45 smithi081 sudo[43398]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-01T01:26:46.078 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:26:45 smithi081 sudo[43398]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-01T01:26:46.078 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:26:45 smithi081 sudo[43398]: pam_unix(sudo:session): session closed for user root 2024-04-01T01:26:46.109 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:26:45 smithi189 sudo[44669]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-01T01:26:46.109 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:26:45 smithi189 sudo[44669]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-01T01:26:46.109 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:26:45 smithi189 sudo[44669]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-01T01:26:46.109 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:26:45 smithi189 sudo[44669]: pam_unix(sudo:session): session closed for user root 2024-04-01T01:26:46.498 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:26:46 smithi189 sudo[44672]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-01T01:26:46.498 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:26:46 smithi189 sudo[44672]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-01T01:26:46.498 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:26:46 smithi189 sudo[44672]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-01T01:26:46.499 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:26:46 smithi189 sudo[44672]: pam_unix(sudo:session): session closed for user root 2024-04-01T01:26:46.648 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:46 smithi070 sudo[51852]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-01T01:26:46.648 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:46 smithi070 sudo[51852]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-01T01:26:46.648 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:46 smithi070 sudo[51852]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-01T01:26:46.648 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:46 smithi070 sudo[51852]: pam_unix(sudo:session): session closed for user root 2024-04-01T01:26:46.911 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:46 smithi189 ceph-mon[35753]: osdmap e21: 3 total, 3 up, 3 in 2024-04-01T01:26:46.911 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:46 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:46.911 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:46 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:46.911 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:46 smithi189 ceph-mon[35753]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-01T01:26:46.911 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:46 smithi189 sudo[44675]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-01T01:26:46.911 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:46 smithi189 sudo[44675]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-01T01:26:46.911 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:46 smithi189 sudo[44675]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-01T01:26:46.911 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:46 smithi189 sudo[44675]: pam_unix(sudo:session): session closed for user root 2024-04-01T01:26:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:46 smithi070 sudo[51861]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-01T01:26:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:46 smithi070 sudo[51861]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-01T01:26:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:46 smithi070 sudo[51861]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-01T01:26:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:46 smithi070 sudo[51861]: pam_unix(sudo:session): session closed for user root 2024-04-01T01:26:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:46 smithi070 ceph-mon[32179]: osdmap e21: 3 total, 3 up, 3 in 2024-04-01T01:26:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:46 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:46 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:46 smithi070 ceph-mon[32179]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-01T01:26:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:46 smithi081 ceph-mon[35142]: osdmap e21: 3 total, 3 up, 3 in 2024-04-01T01:26:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:46 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:46 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:26:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:46 smithi081 ceph-mon[35142]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-01T01:26:47.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 sudo[44678]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-01T01:26:47.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 sudo[44678]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-01T01:26:47.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 sudo[44678]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-01T01:26:47.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 sudo[44678]: pam_unix(sudo:session): session closed for user root 2024-04-01T01:26:47.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 sudo[43401]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-01T01:26:47.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 sudo[43401]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-01T01:26:47.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 sudo[43401]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-01T01:26:47.637 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 sudo[43401]: pam_unix(sudo:session): session closed for user root 2024-04-01T01:26:47.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:26:47.650+0000 7f8cad7fa700 1 -- 172.21.15.70:0/3182825110 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f8ca4098880 con 0x7f8cb01023b0 2024-04-01T01:26:47.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:47.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-01T01:26:47.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:26:47.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:26:47.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:26:47.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:26:47.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-01T01:26:47.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:26:47.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:26:47.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3998857420' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "835bbe5b-ecc9-4ebb-b084-9707dce28d70"}]: dispatch 2024-04-01T01:26:48.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-01T01:26:48.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3998857420' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "835bbe5b-ecc9-4ebb-b084-9707dce28d70"}]': finished 2024-04-01T01:26:48.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-01T01:26:48.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: osdmap e22: 4 total, 3 up, 4 in 2024-04-01T01:26:48.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-01T01:26:48.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:26:48.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:26:48.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:26:48.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-01T01:26:48.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:47 smithi189 ceph-mon[35753]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-01T01:26:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-01T01:26:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:26:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:26:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:26:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:26:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-01T01:26:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:26:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:26:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3998857420' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "835bbe5b-ecc9-4ebb-b084-9707dce28d70"}]: dispatch 2024-04-01T01:26:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-01T01:26:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3998857420' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "835bbe5b-ecc9-4ebb-b084-9707dce28d70"}]': finished 2024-04-01T01:26:48.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-01T01:26:48.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: osdmap e22: 4 total, 3 up, 4 in 2024-04-01T01:26:48.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-01T01:26:48.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:26:48.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:26:48.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:26:48.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-01T01:26:48.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:47 smithi070 ceph-mon[32179]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-01T01:26:48.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 sudo[43404]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-01T01:26:48.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 sudo[43404]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-01T01:26:48.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 sudo[43404]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-01T01:26:48.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 sudo[43404]: pam_unix(sudo:session): session closed for user root 2024-04-01T01:26:48.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:48.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-01T01:26:48.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:26:48.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:26:48.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:26:48.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:26:48.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-01T01:26:48.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:26:48.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:26:48.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3998857420' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "835bbe5b-ecc9-4ebb-b084-9707dce28d70"}]: dispatch 2024-04-01T01:26:48.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-01T01:26:48.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3998857420' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "835bbe5b-ecc9-4ebb-b084-9707dce28d70"}]': finished 2024-04-01T01:26:48.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-01T01:26:48.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: osdmap e22: 4 total, 3 up, 4 in 2024-04-01T01:26:48.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-01T01:26:48.080 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-01T01:26:48.080 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-01T01:26:48.080 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-01T01:26:48.080 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-01T01:26:48.080 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:47 smithi081 ceph-mon[35142]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-01T01:26:48.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:48 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3464554366' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:26:49.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:48 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3464554366' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:26:49.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:48 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3464554366' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:26:49.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:49 smithi189 ceph-mon[35753]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:50.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:49 smithi070 ceph-mon[32179]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:50.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:49 smithi081 ceph-mon[35142]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:50.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:50 smithi189 ceph-mon[35753]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:51.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:50 smithi070 ceph-mon[32179]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:51.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:50 smithi081 ceph-mon[35142]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:53.726 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:53 smithi070 ceph-mon[32179]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:53.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:53 smithi081 ceph-mon[35142]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:53.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:53 smithi189 ceph-mon[35753]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:54.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:54 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-01T01:26:54.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:54 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:54.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:54 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-01T01:26:54.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:54 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:55.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:54 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-01T01:26:55.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:54 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:26:55.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:55 smithi081 ceph-mon[35142]: Deploying daemon osd.3 on smithi070 2024-04-01T01:26:55.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:55 smithi081 ceph-mon[35142]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:55.844 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:55 smithi070 ceph-mon[32179]: Deploying daemon osd.3 on smithi070 2024-04-01T01:26:55.844 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:55 smithi070 ceph-mon[32179]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:55.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:55 smithi189 ceph-mon[35753]: Deploying daemon osd.3 on smithi070 2024-04-01T01:26:55.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:55 smithi189 ceph-mon[35753]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:57.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:57 smithi081 ceph-mon[35142]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:57.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:57 smithi070 ceph-mon[32179]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:57.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:57 smithi189 ceph-mon[35753]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:59.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:26:59 smithi081 ceph-mon[35142]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:26:59.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:26:59 smithi189 ceph-mon[35753]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:27:00.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:26:59 smithi070 ceph-mon[32179]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:27:00.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:00.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:00.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:00.850 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:00 smithi070 ceph-mon[32179]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:27:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:00 smithi081 ceph-mon[35142]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:27:01.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:01.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:00 smithi189 ceph-mon[35753]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:27:03.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:03.256+0000 7f8cad7fa700 1 -- 172.21.15.70:0/3182825110 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8cb0062740 con 0x7f8c980a24a0 2024-04-01T01:27:03.257 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 3 on host 'smithi070' 2024-04-01T01:27:03.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:03.259+0000 7f8cb6465700 1 -- 172.21.15.70:0/3182825110 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c980a24a0 msgr2=0x7f8c980a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:27:03.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:03.259+0000 7f8cb6465700 1 --2- 172.21.15.70:0/3182825110 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c980a24a0 0x7f8c980a4960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8ca000bd70 tx=0x7f8ca000b4a0 comp rx=0 tx=0).stop 2024-04-01T01:27:03.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:03.259+0000 7f8cb6465700 1 -- 172.21.15.70:0/3182825110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb01023b0 msgr2=0x7f8cb0197830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:27:03.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:03.259+0000 7f8cb6465700 1 --2- 172.21.15.70:0/3182825110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb01023b0 0x7f8cb0197830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8ca40099c0 tx=0x7f8ca4009580 comp rx=0 tx=0).stop 2024-04-01T01:27:03.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:03.259+0000 7f8cb6465700 1 -- 172.21.15.70:0/3182825110 shutdown_connections 2024-04-01T01:27:03.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:03.259+0000 7f8cb6465700 1 --2- 172.21.15.70:0/3182825110 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c980a24a0 0x7f8c980a4960 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:03.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:03.259+0000 7f8cb6465700 1 --2- 172.21.15.70:0/3182825110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cb0102d50 0x7f8cb019e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:03.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:03.259+0000 7f8cb6465700 1 --2- 172.21.15.70:0/3182825110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8cb01011b0 0x7f8cb01972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:03.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:03.259+0000 7f8cb6465700 1 --2- 172.21.15.70:0/3182825110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8cb01023b0 0x7f8cb0197830 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:03.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:03.259+0000 7f8cb6465700 1 -- 172.21.15.70:0/3182825110 >> 172.21.15.70:0/3182825110 conn(0x7f8cb00fc730 msgr2=0x7f8cb0105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:27:03.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:03.259+0000 7f8cb6465700 1 -- 172.21.15.70:0/3182825110 shutdown_connections 2024-04-01T01:27:03.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:03.260+0000 7f8cb6465700 1 -- 172.21.15.70:0/3182825110 wait complete. 2024-04-01T01:27:03.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:03 smithi081 ceph-mon[35142]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:27:03.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:03.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:03.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:03.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:03.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:03 smithi070 ceph-mon[32179]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:27:03.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:03.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:03.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:03.884 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:03.914 DEBUG:teuthology.orchestra.run.smithi070:osd.3> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.3.service 2024-04-01T01:27:03.916 INFO:tasks.cephadm:Deploying osd.4 on smithi081 with /dev/vg_nvme/lv_3... 2024-04-01T01:27:03.916 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-01T01:27:03.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:03 smithi189 ceph-mon[35753]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:27:03.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:03.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:03.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:03.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:04.162 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-01T01:27:04.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:04 smithi070 ceph-mon[32179]: from='osd.3 [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-01T01:27:04.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:04 smithi081 ceph-mon[35142]: from='osd.3 [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-01T01:27:04.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:04 smithi189 ceph-mon[35753]: from='osd.3 [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-01T01:27:05.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:05 smithi081 ceph-mon[35142]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:27:05.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:05 smithi081 ceph-mon[35142]: from='osd.3 [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-01T01:27:05.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:05 smithi081 ceph-mon[35142]: osdmap e23: 4 total, 3 up, 4 in 2024-04-01T01:27:05.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:05 smithi081 ceph-mon[35142]: from='osd.3 [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-01T01:27:05.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-01T01:27:05.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:05 smithi189 ceph-mon[35753]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:27:05.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:05 smithi189 ceph-mon[35753]: from='osd.3 [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-01T01:27:05.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:05 smithi189 ceph-mon[35753]: osdmap e23: 4 total, 3 up, 4 in 2024-04-01T01:27:05.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:05 smithi189 ceph-mon[35753]: from='osd.3 [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-01T01:27:05.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-01T01:27:06.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:05 smithi070 ceph-mon[32179]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:27:06.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:05 smithi070 ceph-mon[32179]: from='osd.3 [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-01T01:27:06.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:05 smithi070 ceph-mon[32179]: osdmap e23: 4 total, 3 up, 4 in 2024-04-01T01:27:06.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:05 smithi070 ceph-mon[32179]: from='osd.3 [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-01T01:27:06.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-01T01:27:06.004 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:27:05 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3[53932]: 2024-04-01T01:27:05.572+0000 7f3f756ab700 -1 osd.3 0 waiting for initial osdmap 2024-04-01T01:27:06.004 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:27:05 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3[53932]: 2024-04-01T01:27:05.576+0000 7f3f7149e700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-01T01:27:06.118 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-01T01:27:06.118 INFO:teuthology.orchestra.run.smithi081.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-01T01:27:06.118 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10+0 records in 2024-04-01T01:27:06.118 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2024-04-01T01:27:06.118 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116023 s, 904 MB/s 2024-04-01T01:27:06.118 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping successful for: 2024-04-01T01:27:06.682 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:06 smithi070 ceph-mon[32179]: from='osd.3 [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-01T01:27:06.682 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:06 smithi070 ceph-mon[32179]: osdmap e24: 4 total, 3 up, 4 in 2024-04-01T01:27:06.682 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:06 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-01T01:27:06.682 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:06 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-01T01:27:06.682 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:06 smithi070 ceph-mon[32179]: osd.3 [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617] boot 2024-04-01T01:27:06.682 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:06 smithi070 ceph-mon[32179]: osdmap e25: 4 total, 4 up, 4 in 2024-04-01T01:27:06.682 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:06 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-01T01:27:06.715 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:06 smithi081 ceph-mon[35142]: from='osd.3 [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-01T01:27:06.716 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:06 smithi081 ceph-mon[35142]: osdmap e24: 4 total, 3 up, 4 in 2024-04-01T01:27:06.716 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:06 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-01T01:27:06.716 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:06 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-01T01:27:06.716 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:06 smithi081 ceph-mon[35142]: osd.3 [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617] boot 2024-04-01T01:27:06.716 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:06 smithi081 ceph-mon[35142]: osdmap e25: 4 total, 4 up, 4 in 2024-04-01T01:27:06.716 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:06 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-01T01:27:06.743 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi081:vg_nvme/lv_3 2024-04-01T01:27:06.984 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-01T01:27:06.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:06 smithi189 ceph-mon[35753]: from='osd.3 [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-01T01:27:06.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:06 smithi189 ceph-mon[35753]: osdmap e24: 4 total, 3 up, 4 in 2024-04-01T01:27:06.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:06 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-01T01:27:06.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:06 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-01T01:27:06.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:06 smithi189 ceph-mon[35753]: osd.3 [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617] boot 2024-04-01T01:27:06.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:06 smithi189 ceph-mon[35753]: osdmap e25: 4 total, 4 up, 4 in 2024-04-01T01:27:06.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:06 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-01T01:27:07.781 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:07 smithi081 ceph-mon[35142]: purged_snaps scrub starts 2024-04-01T01:27:07.781 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:07 smithi081 ceph-mon[35142]: purged_snaps scrub ok 2024-04-01T01:27:07.781 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:07 smithi081 ceph-mon[35142]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:27:07.781 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:07.782 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:07.782 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:07.782 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:07.782 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:07.782 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:07.782 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:07.782 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:07.782 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:07.782 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:07.782 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:07.782 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:07.782 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:07 smithi081 ceph-mon[35142]: osdmap e26: 4 total, 4 up, 4 in 2024-04-01T01:27:07.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:07 smithi189 ceph-mon[35753]: purged_snaps scrub starts 2024-04-01T01:27:07.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:07 smithi189 ceph-mon[35753]: purged_snaps scrub ok 2024-04-01T01:27:07.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:07 smithi189 ceph-mon[35753]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:27:07.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:07.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:07.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:07.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:07.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:07.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:08.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:08.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:08.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:08.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:08.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:08.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:08.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:07 smithi189 ceph-mon[35753]: osdmap e26: 4 total, 4 up, 4 in 2024-04-01T01:27:08.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:07 smithi070 ceph-mon[32179]: purged_snaps scrub starts 2024-04-01T01:27:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:07 smithi070 ceph-mon[32179]: purged_snaps scrub ok 2024-04-01T01:27:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:07 smithi070 ceph-mon[32179]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-04-01T01:27:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:08.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:08.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:07 smithi070 ceph-mon[32179]: osdmap e26: 4 total, 4 up, 4 in 2024-04-01T01:27:08.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.614+0000 7fb410fe4700 1 -- 172.21.15.81:0/4243389688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb40c100550 msgr2=0x7fb40c100970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:27:08.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.614+0000 7fb410fe4700 1 --2- 172.21.15.81:0/4243389688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb40c100550 0x7fb40c100970 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc009990 tx=0x7fb3fc009ca0 comp rx=0 tx=0).stop 2024-04-01T01:27:08.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.615+0000 7fb410fe4700 1 -- 172.21.15.81:0/4243389688 shutdown_connections 2024-04-01T01:27:08.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.615+0000 7fb410fe4700 1 --2- 172.21.15.81:0/4243389688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb40c103ed0 0x7fb40c10a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:08.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.615+0000 7fb410fe4700 1 --2- 172.21.15.81:0/4243389688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb40c100550 0x7fb40c100970 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:08.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.615+0000 7fb410fe4700 1 --2- 172.21.15.81:0/4243389688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb40c103530 0x7fb40c103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:08.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.615+0000 7fb410fe4700 1 -- 172.21.15.81:0/4243389688 >> 172.21.15.81:0/4243389688 conn(0x7fb40c0fbed0 msgr2=0x7fb40c0fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:27:08.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.615+0000 7fb410fe4700 1 -- 172.21.15.81:0/4243389688 shutdown_connections 2024-04-01T01:27:08.616 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.615+0000 7fb410fe4700 1 -- 172.21.15.81:0/4243389688 wait complete. 2024-04-01T01:27:08.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.615+0000 7fb410fe4700 1 Processor -- start 2024-04-01T01:27:08.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.615+0000 7fb410fe4700 1 -- start start 2024-04-01T01:27:08.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.615+0000 7fb410fe4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb40c100550 0x7fb40c198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:27:08.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.615+0000 7fb410fe4700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb40c103530 0x7fb40c1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:27:08.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb410fe4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb40c103ed0 0x7fb40c19f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:27:08.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb410fe4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb40c10dd60 con 0x7fb40c103ed0 2024-04-01T01:27:08.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb410fe4700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb40c10dbe0 con 0x7fb40c103530 2024-04-01T01:27:08.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb410fe4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb40c10dee0 con 0x7fb40c100550 2024-04-01T01:27:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb409d9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb40c103530 0x7fb40c1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:27:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb409d9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb40c103530 0x7fb40c1989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:53736/0 (socket says 172.21.15.81:53736) 2024-04-01T01:27:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb409d9b700 1 -- 172.21.15.81:0/142252943 learned_addr learned my addr 172.21.15.81:0/142252943 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-01T01:27:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb40ad9d700 1 --2- 172.21.15.81:0/142252943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb40c103ed0 0x7fb40c19f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:27:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb40a59c700 1 --2- 172.21.15.81:0/142252943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb40c100550 0x7fb40c198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:27:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb409d9b700 1 -- 172.21.15.81:0/142252943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb40c100550 msgr2=0x7fb40c198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:27:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb409d9b700 1 --2- 172.21.15.81:0/142252943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb40c100550 0x7fb40c198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb409d9b700 1 -- 172.21.15.81:0/142252943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb40c103ed0 msgr2=0x7fb40c19f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:27:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb409d9b700 1 --2- 172.21.15.81:0/142252943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb40c103ed0 0x7fb40c19f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:08.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb409d9b700 1 -- 172.21.15.81:0/142252943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40c1a0070 con 0x7fb40c103530 2024-04-01T01:27:08.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb40ad9d700 1 --2- 172.21.15.81:0/142252943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb40c103ed0 0x7fb40c19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:27:08.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb40a59c700 1 --2- 172.21.15.81:0/142252943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb40c100550 0x7fb40c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:27:08.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb409d9b700 1 --2- 172.21.15.81:0/142252943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb40c103530 0x7fb40c1989a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb40000c920 tx=0x7fb40000cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:27:08.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb3fb7fe700 1 -- 172.21.15.81:0/142252943 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb400007b00 con 0x7fb40c103530 2024-04-01T01:27:08.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb3fb7fe700 1 -- 172.21.15.81:0/142252943 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb400004ce0 con 0x7fb40c103530 2024-04-01T01:27:08.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb410fe4700 1 -- 172.21.15.81:0/142252943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb40c1a0360 con 0x7fb40c103530 2024-04-01T01:27:08.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb3fb7fe700 1 -- 172.21.15.81:0/142252943 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb400005600 con 0x7fb40c103530 2024-04-01T01:27:08.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.616+0000 7fb410fe4700 1 -- 172.21.15.81:0/142252943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb40c1a08a0 con 0x7fb40c103530 2024-04-01T01:27:08.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.617+0000 7fb410fe4700 1 -- 172.21.15.81:0/142252943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb40c068f00 con 0x7fb40c103530 2024-04-01T01:27:08.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.619+0000 7fb3fb7fe700 1 -- 172.21.15.81:0/142252943 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fb400004750 con 0x7fb40c103530 2024-04-01T01:27:08.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.620+0000 7fb3fb7fe700 1 --2- 172.21.15.81:0/142252943 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3f40a2530 0x7fb3f40a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:27:08.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.620+0000 7fb3fb7fe700 1 -- 172.21.15.81:0/142252943 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7fb4000cd570 con 0x7fb40c103530 2024-04-01T01:27:08.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.620+0000 7fb40a59c700 1 --2- 172.21.15.81:0/142252943 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3f40a2530 0x7fb3f40a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:27:08.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.621+0000 7fb40a59c700 1 --2- 172.21.15.81:0/142252943 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3f40a2530 0x7fb3f40a49f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc006010 tx=0x7fb3fc005e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:27:08.623 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.623+0000 7fb3fb7fe700 1 -- 172.21.15.81:0/142252943 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb400014030 con 0x7fb40c103530 2024-04-01T01:27:08.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:08.827+0000 7fb410fe4700 1 -- 172.21.15.81:0/142252943 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb40c062680 con 0x7fb3f40a2530 2024-04-01T01:27:08.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:08 smithi081 ceph-mon[35142]: Detected new or changed devices on smithi070 2024-04-01T01:27:08.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:08 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi070 to 8544M 2024-04-01T01:27:08.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:08 smithi081 ceph-mon[35142]: osdmap e27: 4 total, 4 up, 4 in 2024-04-01T01:27:08.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:08 smithi189 ceph-mon[35753]: Detected new or changed devices on smithi070 2024-04-01T01:27:08.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:08 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi070 to 8544M 2024-04-01T01:27:08.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:08 smithi189 ceph-mon[35753]: osdmap e27: 4 total, 4 up, 4 in 2024-04-01T01:27:09.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:08 smithi070 ceph-mon[32179]: Detected new or changed devices on smithi070 2024-04-01T01:27:09.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:08 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi070 to 8544M 2024-04-01T01:27:09.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:08 smithi070 ceph-mon[32179]: osdmap e27: 4 total, 4 up, 4 in 2024-04-01T01:27:09.730 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:09 smithi081 ceph-mon[35142]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:09.730 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:27:09.730 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:27:09.730 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:09.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:09 smithi189 ceph-mon[35753]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:09.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:27:09.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:27:09.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:10.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:09 smithi070 ceph-mon[32179]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:10.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:27:10.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:27:10.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:10.657 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:10 smithi081 ceph-mon[35142]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:27:10.658 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:10 smithi081 ceph-mon[35142]: osdmap e28: 4 total, 4 up, 4 in 2024-04-01T01:27:10.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:10 smithi189 ceph-mon[35753]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:27:10.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:10 smithi189 ceph-mon[35753]: osdmap e28: 4 total, 4 up, 4 in 2024-04-01T01:27:11.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:10 smithi070 ceph-mon[32179]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:27:11.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:10 smithi070 ceph-mon[32179]: osdmap e28: 4 total, 4 up, 4 in 2024-04-01T01:27:11.905 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:11 smithi070 ceph-mon[32179]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-01T01:27:11.905 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:11 smithi070 ceph-mon[32179]: osdmap e29: 4 total, 4 up, 4 in 2024-04-01T01:27:11.905 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:11 smithi070 ceph-mon[32179]: from='client.? 172.21.15.81:0/1712195781' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ddf50f36-6795-440a-9890-2f20c63820fe"}]: dispatch 2024-04-01T01:27:11.905 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:11 smithi070 ceph-mon[32179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ddf50f36-6795-440a-9890-2f20c63820fe"}]: dispatch 2024-04-01T01:27:11.905 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:11 smithi070 ceph-mon[32179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ddf50f36-6795-440a-9890-2f20c63820fe"}]': finished 2024-04-01T01:27:11.905 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:11 smithi070 ceph-mon[32179]: osdmap e30: 5 total, 4 up, 5 in 2024-04-01T01:27:11.905 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:11 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-01T01:27:11.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:11 smithi189 ceph-mon[35753]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-01T01:27:11.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:11 smithi189 ceph-mon[35753]: osdmap e29: 4 total, 4 up, 4 in 2024-04-01T01:27:11.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:11 smithi189 ceph-mon[35753]: from='client.? 172.21.15.81:0/1712195781' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ddf50f36-6795-440a-9890-2f20c63820fe"}]: dispatch 2024-04-01T01:27:11.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:11 smithi189 ceph-mon[35753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ddf50f36-6795-440a-9890-2f20c63820fe"}]: dispatch 2024-04-01T01:27:11.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:11 smithi189 ceph-mon[35753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ddf50f36-6795-440a-9890-2f20c63820fe"}]': finished 2024-04-01T01:27:11.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:11 smithi189 ceph-mon[35753]: osdmap e30: 5 total, 4 up, 5 in 2024-04-01T01:27:11.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:11 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-01T01:27:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:11 smithi081 ceph-mon[35142]: pgmap v104: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-01T01:27:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:11 smithi081 ceph-mon[35142]: osdmap e29: 4 total, 4 up, 4 in 2024-04-01T01:27:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:11 smithi081 ceph-mon[35142]: from='client.? 172.21.15.81:0/1712195781' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ddf50f36-6795-440a-9890-2f20c63820fe"}]: dispatch 2024-04-01T01:27:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:11 smithi081 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ddf50f36-6795-440a-9890-2f20c63820fe"}]: dispatch 2024-04-01T01:27:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:11 smithi081 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ddf50f36-6795-440a-9890-2f20c63820fe"}]': finished 2024-04-01T01:27:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:11 smithi081 ceph-mon[35142]: osdmap e30: 5 total, 4 up, 5 in 2024-04-01T01:27:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:11 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-01T01:27:12.921 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:12 smithi070 ceph-mon[32179]: from='client.? 172.21.15.81:0/535276478' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:27:12.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:12 smithi189 ceph-mon[35753]: from='client.? 172.21.15.81:0/535276478' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:27:13.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:12 smithi081 ceph-mon[35142]: from='client.? 172.21.15.81:0/535276478' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:27:13.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:13 smithi189 ceph-mon[35753]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-01T01:27:14.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:13 smithi070 ceph-mon[32179]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-01T01:27:14.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:13 smithi081 ceph-mon[35142]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-01T01:27:15.768 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:15 smithi081 ceph-mon[35142]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-01T01:27:15.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:15 smithi189 ceph-mon[35753]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-01T01:27:16.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:15 smithi070 ceph-mon[32179]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-01T01:27:17.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:17 smithi189 ceph-mon[35753]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:18.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:17 smithi070 ceph-mon[32179]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:18.010 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:17 smithi081 ceph-mon[35142]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:19.989 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:19 smithi081 ceph-mon[35142]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:19.989 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-01T01:27:19.989 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:19 smithi189 ceph-mon[35753]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-01T01:27:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:20.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:19 smithi070 ceph-mon[32179]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:20.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-01T01:27:20.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:20.993 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:20 smithi081 ceph-mon[35142]: Deploying daemon osd.4 on smithi081 2024-04-01T01:27:20.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:20 smithi189 ceph-mon[35753]: Deploying daemon osd.4 on smithi081 2024-04-01T01:27:21.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:20 smithi070 ceph-mon[32179]: Deploying daemon osd.4 on smithi081 2024-04-01T01:27:21.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:21 smithi081 ceph-mon[35142]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:21.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:21 smithi189 ceph-mon[35753]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:22.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:21 smithi070 ceph-mon[32179]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:23.981 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:23 smithi081 ceph-mon[35142]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:23.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:23 smithi189 ceph-mon[35753]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:24.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:23 smithi070 ceph-mon[32179]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:25 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:25 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:25 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:25.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:25 smithi070 ceph-mon[32179]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:25.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:25 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:25.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:25 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:25.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:25 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:25.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:25 smithi081 ceph-mon[35142]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:25.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:25 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:25.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:25 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:25.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:25 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:25.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:25 smithi189 ceph-mon[35753]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:27.713 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:27 smithi081 ceph-mon[35142]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:27.791 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 4 on host 'smithi081' 2024-04-01T01:27:27.792 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:27.792+0000 7fb3fb7fe700 1 -- 172.21.15.81:0/142252943 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb40c062680 con 0x7fb3f40a2530 2024-04-01T01:27:27.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:27.795+0000 7fb410fe4700 1 -- 172.21.15.81:0/142252943 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3f40a2530 msgr2=0x7fb3f40a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:27:27.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:27.795+0000 7fb410fe4700 1 --2- 172.21.15.81:0/142252943 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3f40a2530 0x7fb3f40a49f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc006010 tx=0x7fb3fc005e80 comp rx=0 tx=0).stop 2024-04-01T01:27:27.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:27.795+0000 7fb410fe4700 1 -- 172.21.15.81:0/142252943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb40c103530 msgr2=0x7fb40c1989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:27:27.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:27.795+0000 7fb410fe4700 1 --2- 172.21.15.81:0/142252943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb40c103530 0x7fb40c1989a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb40000c920 tx=0x7fb40000cce0 comp rx=0 tx=0).stop 2024-04-01T01:27:27.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:27.795+0000 7fb410fe4700 1 -- 172.21.15.81:0/142252943 shutdown_connections 2024-04-01T01:27:27.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:27.795+0000 7fb410fe4700 1 --2- 172.21.15.81:0/142252943 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3f40a2530 0x7fb3f40a49f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:27.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:27.795+0000 7fb410fe4700 1 --2- 172.21.15.81:0/142252943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb40c103ed0 0x7fb40c19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:27.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:27.795+0000 7fb410fe4700 1 --2- 172.21.15.81:0/142252943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb40c100550 0x7fb40c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:27.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:27.795+0000 7fb410fe4700 1 --2- 172.21.15.81:0/142252943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb40c103530 0x7fb40c1989a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:27.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:27.795+0000 7fb410fe4700 1 -- 172.21.15.81:0/142252943 >> 172.21.15.81:0/142252943 conn(0x7fb40c0fbed0 msgr2=0x7fb40c106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:27:27.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:27.795+0000 7fb410fe4700 1 -- 172.21.15.81:0/142252943 shutdown_connections 2024-04-01T01:27:27.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:27:27.795+0000 7fb410fe4700 1 -- 172.21.15.81:0/142252943 wait complete. 2024-04-01T01:27:27.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:27 smithi189 ceph-mon[35753]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:28.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:27 smithi070 ceph-mon[32179]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:28.461 DEBUG:teuthology.orchestra.run.smithi081:osd.4> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.4.service 2024-04-01T01:27:28.463 INFO:tasks.cephadm:Deploying osd.5 on smithi189 with /dev/vg_nvme/lv_3... 2024-04-01T01:27:28.463 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-01T01:27:28.709 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-01T01:27:28.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:28 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:28.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:28 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:28.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:28 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:28.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:28 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:28.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:28 smithi189 ceph-mon[35753]: from='osd.4 [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-01T01:27:28.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:28 smithi189 ceph-mon[35753]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-01T01:27:28.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:28 smithi189 ceph-mon[35753]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:29.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:28 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:29.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:28 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:29.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:28 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:29.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:28 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:29.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:28 smithi070 ceph-mon[32179]: from='osd.4 [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-01T01:27:29.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:28 smithi070 ceph-mon[32179]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-01T01:27:29.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:28 smithi070 ceph-mon[32179]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:28 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:28 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:28 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:28 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:28 smithi081 ceph-mon[35142]: from='osd.4 [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-01T01:27:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:28 smithi081 ceph-mon[35142]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-01T01:27:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:28 smithi081 ceph-mon[35142]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:30.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:30 smithi189 ceph-mon[35753]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-01T01:27:30.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:30 smithi189 ceph-mon[35753]: osdmap e31: 5 total, 4 up, 5 in 2024-04-01T01:27:30.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:30 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-01T01:27:30.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:30 smithi189 ceph-mon[35753]: from='osd.4 [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:27:30.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:30 smithi189 ceph-mon[35753]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:27:30.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:30 smithi081 ceph-mon[35142]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-01T01:27:30.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:30 smithi081 ceph-mon[35142]: osdmap e31: 5 total, 4 up, 5 in 2024-04-01T01:27:30.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:30 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-01T01:27:30.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:30 smithi081 ceph-mon[35142]: from='osd.4 [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:27:30.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:30 smithi081 ceph-mon[35142]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:27:30.329 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:27:30 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4[46199]: 2024-04-01T01:27:30.070+0000 7f378a341700 -1 osd.4 0 waiting for initial osdmap 2024-04-01T01:27:30.329 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:27:30 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4[46199]: 2024-04-01T01:27:30.075+0000 7f3786134700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-01T01:27:30.414 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-01T01:27:30.414 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-01T01:27:30.414 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-04-01T01:27:30.414 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-04-01T01:27:30.414 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115272 s, 910 MB/s 2024-04-01T01:27:30.414 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-04-01T01:27:30.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:30 smithi070 ceph-mon[32179]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-01T01:27:30.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:30 smithi070 ceph-mon[32179]: osdmap e31: 5 total, 4 up, 5 in 2024-04-01T01:27:30.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:30 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-01T01:27:30.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:30 smithi070 ceph-mon[32179]: from='osd.4 [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:27:30.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:30 smithi070 ceph-mon[32179]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:27:31.041 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi189:vg_nvme/lv_3 2024-04-01T01:27:31.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:31 smithi189 ceph-mon[35753]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-01T01:27:31.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:31 smithi189 ceph-mon[35753]: osdmap e32: 5 total, 4 up, 5 in 2024-04-01T01:27:31.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:31 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-01T01:27:31.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:31 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-01T01:27:31.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:31 smithi189 ceph-mon[35753]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:31.258 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:31 smithi081 ceph-mon[35142]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-01T01:27:31.258 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:31 smithi081 ceph-mon[35142]: osdmap e32: 5 total, 4 up, 5 in 2024-04-01T01:27:31.258 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:31 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-01T01:27:31.258 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:31 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-01T01:27:31.258 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:31 smithi081 ceph-mon[35142]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:31.276 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-01T01:27:31.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:31 smithi070 ceph-mon[32179]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-01T01:27:31.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:31 smithi070 ceph-mon[32179]: osdmap e32: 5 total, 4 up, 5 in 2024-04-01T01:27:31.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:31 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-01T01:27:31.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:31 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-01T01:27:31.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:31 smithi070 ceph-mon[32179]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-04-01T01:27:32.364 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: purged_snaps scrub starts 2024-04-01T01:27:32.364 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: purged_snaps scrub ok 2024-04-01T01:27:32.364 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: osd.4 [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261] boot 2024-04-01T01:27:32.364 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: osdmap e33: 5 total, 5 up, 5 in 2024-04-01T01:27:32.364 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-01T01:27:32.364 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: Detected new or changed devices on smithi081 2024-04-01T01:27:32.364 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:32.364 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:32.364 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:32.364 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:32.364 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi081 to 8544M 2024-04-01T01:27:32.364 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:32.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:32.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:32.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:32.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:32.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:32.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:32.365 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:32.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: purged_snaps scrub starts 2024-04-01T01:27:32.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: purged_snaps scrub ok 2024-04-01T01:27:32.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: osd.4 [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261] boot 2024-04-01T01:27:32.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: osdmap e33: 5 total, 5 up, 5 in 2024-04-01T01:27:32.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-01T01:27:32.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: Detected new or changed devices on smithi081 2024-04-01T01:27:32.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:32.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:32.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:32.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:32.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi081 to 8544M 2024-04-01T01:27:32.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:32.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:32.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:32.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:32.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:32.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:32.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:32.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:32.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: purged_snaps scrub starts 2024-04-01T01:27:32.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: purged_snaps scrub ok 2024-04-01T01:27:32.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: osd.4 [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261] boot 2024-04-01T01:27:32.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: osdmap e33: 5 total, 5 up, 5 in 2024-04-01T01:27:32.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-01T01:27:32.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: Detected new or changed devices on smithi081 2024-04-01T01:27:32.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:32.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:32.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:32.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:32.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi081 to 8544M 2024-04-01T01:27:32.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:32.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:32.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:32.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:32.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:32.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:32.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:32.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:32.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.829+0000 7fea76751700 1 -- 172.21.15.189:0/648359771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea70102d30 msgr2=0x7fea701095e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:27:32.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.829+0000 7fea76751700 1 --2- 172.21.15.189:0/648359771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea70102d30 0x7fea701095e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fea6800b0a0 tx=0x7fea6800b3b0 comp rx=0 tx=0).stop 2024-04-01T01:27:32.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.829+0000 7fea76751700 1 -- 172.21.15.189:0/648359771 shutdown_connections 2024-04-01T01:27:32.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.829+0000 7fea76751700 1 --2- 172.21.15.189:0/648359771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea70102d30 0x7fea701095e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:32.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.829+0000 7fea76751700 1 --2- 172.21.15.189:0/648359771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea70101190 0x7fea701015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:32.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.829+0000 7fea76751700 1 --2- 172.21.15.189:0/648359771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea70102390 0x7fea701027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:32.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.829+0000 7fea76751700 1 -- 172.21.15.189:0/648359771 >> 172.21.15.189:0/648359771 conn(0x7fea700fc730 msgr2=0x7fea700feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:27:32.830 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.830+0000 7fea76751700 1 -- 172.21.15.189:0/648359771 shutdown_connections 2024-04-01T01:27:32.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.830+0000 7fea76751700 1 -- 172.21.15.189:0/648359771 wait complete. 2024-04-01T01:27:32.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.830+0000 7fea76751700 1 Processor -- start 2024-04-01T01:27:32.831 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.830+0000 7fea76751700 1 -- start start 2024-04-01T01:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea76751700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea70101190 0x7fea70197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea76751700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea70102390 0x7fea70197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea76751700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea70102d30 0x7fea7019e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea76751700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fea7010cca0 con 0x7fea70102390 2024-04-01T01:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea76751700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fea7010cb20 con 0x7fea70102d30 2024-04-01T01:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea76751700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fea7010ce20 con 0x7fea70101190 2024-04-01T01:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea6ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea70101190 0x7fea70197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea6ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea70101190 0x7fea70197200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:41432/0 (socket says 172.21.15.189:41432) 2024-04-01T01:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea6ffff700 1 -- 172.21.15.189:0/2965718915 learned_addr learned my addr 172.21.15.189:0/2965718915 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-01T01:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea74cee700 1 --2- 172.21.15.189:0/2965718915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea70102d30 0x7fea7019e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea6f7fe700 1 --2- 172.21.15.189:0/2965718915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea70102390 0x7fea70197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:27:32.833 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea6ffff700 1 -- 172.21.15.189:0/2965718915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea70102d30 msgr2=0x7fea7019e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea6ffff700 1 --2- 172.21.15.189:0/2965718915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea70102d30 0x7fea7019e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea6ffff700 1 -- 172.21.15.189:0/2965718915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea70102390 msgr2=0x7fea70197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea6ffff700 1 --2- 172.21.15.189:0/2965718915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea70102390 0x7fea70197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea6ffff700 1 -- 172.21.15.189:0/2965718915 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea7006aa60 con 0x7fea70101190 2024-04-01T01:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea6ffff700 1 --2- 172.21.15.189:0/2965718915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea70101190 0x7fea70197200 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fea6000b500 tx=0x7fea6000b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea6d7fa700 1 -- 172.21.15.189:0/2965718915 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea60015020 con 0x7fea70101190 2024-04-01T01:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea76751700 1 -- 172.21.15.189:0/2965718915 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea7006ad50 con 0x7fea70101190 2024-04-01T01:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea6d7fa700 1 -- 172.21.15.189:0/2965718915 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea60011ce0 con 0x7fea70101190 2024-04-01T01:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.831+0000 7fea6d7fa700 1 -- 172.21.15.189:0/2965718915 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea60010d60 con 0x7fea70101190 2024-04-01T01:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.832+0000 7fea76751700 1 -- 172.21.15.189:0/2965718915 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea7006b310 con 0x7fea70101190 2024-04-01T01:27:32.834 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.833+0000 7fea76751700 1 -- 172.21.15.189:0/2965718915 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea70068f00 con 0x7fea70101190 2024-04-01T01:27:32.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.834+0000 7fea6d7fa700 1 -- 172.21.15.189:0/2965718915 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fea60018020 con 0x7fea70101190 2024-04-01T01:27:32.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.834+0000 7fea6d7fa700 1 --2- 172.21.15.189:0/2965718915 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fea580a2530 0x7fea580a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:27:32.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.834+0000 7fea6d7fa700 1 -- 172.21.15.189:0/2965718915 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fea600cce60 con 0x7fea70101190 2024-04-01T01:27:32.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.834+0000 7fea6f7fe700 1 --2- 172.21.15.189:0/2965718915 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fea580a2530 0x7fea580a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:27:32.836 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.835+0000 7fea6f7fe700 1 --2- 172.21.15.189:0/2965718915 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fea580a2530 0x7fea580a49f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fea64005ea0 tx=0x7fea64005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:27:32.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:32.839+0000 7fea6d7fa700 1 -- 172.21.15.189:0/2965718915 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fea6001f020 con 0x7fea70101190 2024-04-01T01:27:33.043 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:33.044+0000 7fea76751700 1 -- 172.21.15.189:0/2965718915 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fea7002dca0 con 0x7fea580a2530 2024-04-01T01:27:33.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:33 smithi081 ceph-mon[35142]: osdmap e34: 5 total, 5 up, 5 in 2024-04-01T01:27:33.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:33 smithi081 ceph-mon[35142]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:33.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:27:33.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:27:33.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:33.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:33 smithi189 ceph-mon[35753]: osdmap e34: 5 total, 5 up, 5 in 2024-04-01T01:27:33.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:33 smithi189 ceph-mon[35753]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:33.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:27:33.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:27:33.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:33.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:33 smithi070 ceph-mon[32179]: osdmap e34: 5 total, 5 up, 5 in 2024-04-01T01:27:33.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:33 smithi070 ceph-mon[32179]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:33.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:27:33.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:27:33.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:34.214 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:34 smithi189 ceph-mon[35753]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:27:34.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:34 smithi070 ceph-mon[32179]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:27:34.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:34 smithi081 ceph-mon[35142]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:27:35.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:35 smithi189 ceph-mon[35753]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:35.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:35 smithi070 ceph-mon[32179]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:35.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:35 smithi081 ceph-mon[35142]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:36.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:36 smithi189 ceph-mon[35753]: from='client.? 172.21.15.189:0/3159926590' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cae9565-6f21-4686-a5e9-a8c08d5fe0bb"}]: dispatch 2024-04-01T01:27:36.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:36 smithi189 ceph-mon[35753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cae9565-6f21-4686-a5e9-a8c08d5fe0bb"}]: dispatch 2024-04-01T01:27:36.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:36 smithi189 ceph-mon[35753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cae9565-6f21-4686-a5e9-a8c08d5fe0bb"}]': finished 2024-04-01T01:27:36.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:36 smithi189 ceph-mon[35753]: osdmap e35: 6 total, 5 up, 6 in 2024-04-01T01:27:36.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:36 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:36.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:36 smithi070 ceph-mon[32179]: from='client.? 172.21.15.189:0/3159926590' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cae9565-6f21-4686-a5e9-a8c08d5fe0bb"}]: dispatch 2024-04-01T01:27:36.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:36 smithi070 ceph-mon[32179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cae9565-6f21-4686-a5e9-a8c08d5fe0bb"}]: dispatch 2024-04-01T01:27:36.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:36 smithi070 ceph-mon[32179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cae9565-6f21-4686-a5e9-a8c08d5fe0bb"}]': finished 2024-04-01T01:27:36.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:36 smithi070 ceph-mon[32179]: osdmap e35: 6 total, 5 up, 6 in 2024-04-01T01:27:36.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:36 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:36.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:36 smithi081 ceph-mon[35142]: from='client.? 172.21.15.189:0/3159926590' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cae9565-6f21-4686-a5e9-a8c08d5fe0bb"}]: dispatch 2024-04-01T01:27:36.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:36 smithi081 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9cae9565-6f21-4686-a5e9-a8c08d5fe0bb"}]: dispatch 2024-04-01T01:27:36.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:36 smithi081 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9cae9565-6f21-4686-a5e9-a8c08d5fe0bb"}]': finished 2024-04-01T01:27:36.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:36 smithi081 ceph-mon[35142]: osdmap e35: 6 total, 5 up, 6 in 2024-04-01T01:27:36.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:36 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:37.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:37 smithi189 ceph-mon[35753]: from='client.? 172.21.15.189:0/1800748243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:27:37.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:37 smithi189 ceph-mon[35753]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:37.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:37 smithi070 ceph-mon[32179]: from='client.? 172.21.15.189:0/1800748243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:27:37.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:37 smithi070 ceph-mon[32179]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:37.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:37 smithi081 ceph-mon[35142]: from='client.? 172.21.15.189:0/1800748243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:27:37.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:37 smithi081 ceph-mon[35142]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:39.698 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:39 smithi189 ceph-mon[35753]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:39.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:39 smithi081 ceph-mon[35142]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:40.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:39 smithi070 ceph-mon[32179]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:41.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:41 smithi189 ceph-mon[35753]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:41.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:41 smithi081 ceph-mon[35142]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:42.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:41 smithi070 ceph-mon[32179]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:43.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:43 smithi081 ceph-mon[35142]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:43.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-01T01:27:43.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:43.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:43 smithi189 ceph-mon[35753]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:43.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-01T01:27:43.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:44.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:43 smithi070 ceph-mon[32179]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:44.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-01T01:27:44.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:44.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:44 smithi081 ceph-mon[35142]: Deploying daemon osd.5 on smithi189 2024-04-01T01:27:44.857 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:44 smithi189 ceph-mon[35753]: Deploying daemon osd.5 on smithi189 2024-04-01T01:27:45.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:44 smithi070 ceph-mon[32179]: Deploying daemon osd.5 on smithi189 2024-04-01T01:27:45.678 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:45 smithi189 ceph-mon[35753]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:46.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:45 smithi070 ceph-mon[32179]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:46.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:45 smithi081 ceph-mon[35142]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:47.737 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:47 smithi189 ceph-mon[35753]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:48.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:47 smithi070 ceph-mon[32179]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:48.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:47 smithi081 ceph-mon[35142]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:49.732 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:49 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:49.732 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:49 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:49.732 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:49 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:49.732 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:49 smithi189 ceph-mon[35753]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:49.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:49 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:49.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:49 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:49.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:49 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:49.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:49 smithi070 ceph-mon[32179]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:49.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:49 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:49.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:49 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:49.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:49 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:49.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:49 smithi081 ceph-mon[35142]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:51.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:51 smithi081 ceph-mon[35142]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:51.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:51 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:51.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:51 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:51.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:51 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:51.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:51 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:51.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:51 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:51.857 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:51 smithi189 ceph-mon[35753]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:51.857 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:51 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:51.857 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:51 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:51.857 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:51 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:51.857 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:51 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:51.857 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:51 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:51.909 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:51.909+0000 7fea6d7fa700 1 -- 172.21.15.189:0/2965718915 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fea7002dca0 con 0x7fea580a2530 2024-04-01T01:27:51.909 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 5 on host 'smithi189' 2024-04-01T01:27:51.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:51.911+0000 7fea76751700 1 -- 172.21.15.189:0/2965718915 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fea580a2530 msgr2=0x7fea580a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:27:51.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:51.911+0000 7fea76751700 1 --2- 172.21.15.189:0/2965718915 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fea580a2530 0x7fea580a49f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fea64005ea0 tx=0x7fea64005e30 comp rx=0 tx=0).stop 2024-04-01T01:27:51.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:51.911+0000 7fea76751700 1 -- 172.21.15.189:0/2965718915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea70101190 msgr2=0x7fea70197200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:27:51.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:51.911+0000 7fea76751700 1 --2- 172.21.15.189:0/2965718915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea70101190 0x7fea70197200 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fea6000b500 tx=0x7fea6000b810 comp rx=0 tx=0).stop 2024-04-01T01:27:51.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:51.911+0000 7fea76751700 1 -- 172.21.15.189:0/2965718915 shutdown_connections 2024-04-01T01:27:51.911 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:51.911+0000 7fea76751700 1 --2- 172.21.15.189:0/2965718915 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fea580a2530 0x7fea580a49f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:51.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:51.911+0000 7fea76751700 1 --2- 172.21.15.189:0/2965718915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea70102d30 0x7fea7019e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:51.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:51.911+0000 7fea76751700 1 --2- 172.21.15.189:0/2965718915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea70101190 0x7fea70197200 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:51.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:51.911+0000 7fea76751700 1 --2- 172.21.15.189:0/2965718915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea70102390 0x7fea70197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:51.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:51.911+0000 7fea76751700 1 -- 172.21.15.189:0/2965718915 >> 172.21.15.189:0/2965718915 conn(0x7fea700fc730 msgr2=0x7fea70105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:27:51.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:51.911+0000 7fea76751700 1 -- 172.21.15.189:0/2965718915 shutdown_connections 2024-04-01T01:27:51.912 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:27:51.911+0000 7fea76751700 1 -- 172.21.15.189:0/2965718915 wait complete. 2024-04-01T01:27:52.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:51 smithi070 ceph-mon[32179]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:52.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:51 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:52.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:51 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:52.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:51 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:52.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:51 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:52.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:51 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:52.496 DEBUG:teuthology.orchestra.run.smithi189:osd.5> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.5.service 2024-04-01T01:27:52.499 INFO:tasks.cephadm:Deploying osd.6 on smithi070 with /dev/vg_nvme/lv_2... 2024-04-01T01:27:52.499 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-01T01:27:52.746 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:27:53.118 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:52 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:53.118 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:52 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:53.118 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:52 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:53.118 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:52 smithi070 ceph-mon[32179]: from='osd.5 [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-01T01:27:53.118 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:52 smithi070 ceph-mon[32179]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-01T01:27:53.118 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:52 smithi070 ceph-mon[32179]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:53.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:52 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:53.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:52 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:53.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:52 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:53.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:52 smithi189 ceph-mon[35753]: from='osd.5 [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-01T01:27:53.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:52 smithi189 ceph-mon[35753]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-01T01:27:53.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:52 smithi189 ceph-mon[35753]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:53.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:52 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:53.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:52 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:53.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:52 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:27:53.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:52 smithi081 ceph-mon[35142]: from='osd.5 [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-01T01:27:53.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:52 smithi081 ceph-mon[35142]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-01T01:27:53.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:52 smithi081 ceph-mon[35142]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:54.076 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:53 smithi070 ceph-mon[32179]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-01T01:27:54.076 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:53 smithi070 ceph-mon[32179]: osdmap e36: 6 total, 5 up, 6 in 2024-04-01T01:27:54.076 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:53 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:54.076 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:53 smithi070 ceph-mon[32179]: from='osd.5 [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:27:54.076 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:53 smithi070 ceph-mon[32179]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:27:54.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:53 smithi189 ceph-mon[35753]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-01T01:27:54.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:53 smithi189 ceph-mon[35753]: osdmap e36: 6 total, 5 up, 6 in 2024-04-01T01:27:54.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:53 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:54.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:53 smithi189 ceph-mon[35753]: from='osd.5 [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:27:54.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:53 smithi189 ceph-mon[35753]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:27:54.249 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:27:53 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:27:53.912+0000 7fe72504a700 -1 osd.5 0 waiting for initial osdmap 2024-04-01T01:27:54.249 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:27:53 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:27:53.918+0000 7fe720e3d700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-01T01:27:54.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:53 smithi081 ceph-mon[35142]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-01T01:27:54.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:53 smithi081 ceph-mon[35142]: osdmap e36: 6 total, 5 up, 6 in 2024-04-01T01:27:54.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:53 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:54.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:53 smithi081 ceph-mon[35142]: from='osd.5 [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:27:54.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:53 smithi081 ceph-mon[35142]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:27:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-01T01:27:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-01T01:27:54.687 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-04-01T01:27:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-04-01T01:27:54.687 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00849307 s, 1.2 GB/s 2024-04-01T01:27:54.687 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-04-01T01:27:55.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:54 smithi070 ceph-mon[32179]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-01T01:27:55.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:54 smithi070 ceph-mon[32179]: osdmap e37: 6 total, 5 up, 6 in 2024-04-01T01:27:55.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:54 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:55.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:54 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:55.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:54 smithi070 ceph-mon[32179]: pgmap v135: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:55.243 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:54 smithi189 ceph-mon[35753]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-01T01:27:55.243 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:54 smithi189 ceph-mon[35753]: osdmap e37: 6 total, 5 up, 6 in 2024-04-01T01:27:55.243 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:54 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:55.243 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:54 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:55.243 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:54 smithi189 ceph-mon[35753]: pgmap v135: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:55.285 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi070:vg_nvme/lv_2 2024-04-01T01:27:55.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:54 smithi081 ceph-mon[35142]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-01T01:27:55.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:54 smithi081 ceph-mon[35142]: osdmap e37: 6 total, 5 up, 6 in 2024-04-01T01:27:55.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:54 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:55.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:54 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:55.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:54 smithi081 ceph-mon[35142]: pgmap v135: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-04-01T01:27:55.544 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:27:56.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: purged_snaps scrub starts 2024-04-01T01:27:56.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: purged_snaps scrub ok 2024-04-01T01:27:56.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:56.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: osd.5 [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377] boot 2024-04-01T01:27:56.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: osdmap e38: 6 total, 6 up, 6 in 2024-04-01T01:27:56.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:56.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: Detected new or changed devices on smithi189 2024-04-01T01:27:56.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:56.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:56.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:56.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:56.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi189 to 8544M 2024-04-01T01:27:56.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:56.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:56.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:56.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:56.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:55 smithi189 ceph-mon[35753]: osdmap e39: 6 total, 6 up, 6 in 2024-04-01T01:27:56.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: purged_snaps scrub starts 2024-04-01T01:27:56.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: purged_snaps scrub ok 2024-04-01T01:27:56.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:56.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: osd.5 [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377] boot 2024-04-01T01:27:56.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: osdmap e38: 6 total, 6 up, 6 in 2024-04-01T01:27:56.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:56.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: Detected new or changed devices on smithi189 2024-04-01T01:27:56.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:56.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:56.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:56.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:56.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi189 to 8544M 2024-04-01T01:27:56.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:56.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:56.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:56.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:56.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:55 smithi070 ceph-mon[32179]: osdmap e39: 6 total, 6 up, 6 in 2024-04-01T01:27:56.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: purged_snaps scrub starts 2024-04-01T01:27:56.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: purged_snaps scrub ok 2024-04-01T01:27:56.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:56.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: osd.5 [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377] boot 2024-04-01T01:27:56.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: osdmap e38: 6 total, 6 up, 6 in 2024-04-01T01:27:56.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-01T01:27:56.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: Detected new or changed devices on smithi189 2024-04-01T01:27:56.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:56.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:56.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:56.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:27:56.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi189 to 8544M 2024-04-01T01:27:56.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:56.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:56.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:27:56.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:27:56.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:55 smithi081 ceph-mon[35142]: osdmap e39: 6 total, 6 up, 6 in 2024-04-01T01:27:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.113+0000 7f86c0bb2700 1 -- 172.21.15.70:0/1104930867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86bc103490 msgr2=0x7f86bc1038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:27:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.113+0000 7f86c0bb2700 1 --2- 172.21.15.70:0/1104930867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86bc103490 0x7f86bc1038f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f86b00099c0 tx=0x7f86b0009cd0 comp rx=0 tx=0).stop 2024-04-01T01:27:57.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.113+0000 7f86c0bb2700 1 -- 172.21.15.70:0/1104930867 shutdown_connections 2024-04-01T01:27:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.113+0000 7f86c0bb2700 1 --2- 172.21.15.70:0/1104930867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86bc103e30 0x7f86bc10a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.113+0000 7f86c0bb2700 1 --2- 172.21.15.70:0/1104930867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86bc102290 0x7f86bc1026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.113+0000 7f86c0bb2700 1 --2- 172.21.15.70:0/1104930867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86bc103490 0x7f86bc1038f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.113+0000 7f86c0bb2700 1 -- 172.21.15.70:0/1104930867 >> 172.21.15.70:0/1104930867 conn(0x7f86bc0fd850 msgr2=0x7f86bc0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:27:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.113+0000 7f86c0bb2700 1 -- 172.21.15.70:0/1104930867 shutdown_connections 2024-04-01T01:27:57.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.113+0000 7f86c0bb2700 1 -- 172.21.15.70:0/1104930867 wait complete. 2024-04-01T01:27:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.113+0000 7f86c0bb2700 1 Processor -- start 2024-04-01T01:27:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.113+0000 7f86c0bb2700 1 -- start start 2024-04-01T01:27:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86c0bb2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86bc102290 0x7f86bc075fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:27:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86c0bb2700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86bc103490 0x7f86bc074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:27:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86c0bb2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86bc103e30 0x7f86bc074b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:27:57.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86c0bb2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f86bc10dd00 con 0x7f86bc103e30 2024-04-01T01:27:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86c0bb2700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f86bc10db80 con 0x7f86bc103490 2024-04-01T01:27:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86c0bb2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f86bc10de80 con 0x7f86bc102290 2024-04-01T01:27:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86bad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86bc103e30 0x7f86bc074b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:27:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86bad9d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86bc103e30 0x7f86bc074b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60422/0 (socket says 172.21.15.70:60422) 2024-04-01T01:27:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86bad9d700 1 -- 172.21.15.70:0/2687708425 learned_addr learned my addr 172.21.15.70:0/2687708425 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:27:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86b9d9b700 1 --2- 172.21.15.70:0/2687708425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86bc103490 0x7f86bc074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:27:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86ba59c700 1 --2- 172.21.15.70:0/2687708425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86bc102290 0x7f86bc075fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:27:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86bad9d700 1 -- 172.21.15.70:0/2687708425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86bc102290 msgr2=0x7f86bc075fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:27:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86bad9d700 1 --2- 172.21.15.70:0/2687708425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86bc102290 0x7f86bc075fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86bad9d700 1 -- 172.21.15.70:0/2687708425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86bc103490 msgr2=0x7f86bc074630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:27:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86bad9d700 1 --2- 172.21.15.70:0/2687708425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86bc103490 0x7f86bc074630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:27:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86bad9d700 1 -- 172.21.15.70:0/2687708425 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86bc075420 con 0x7f86bc103e30 2024-04-01T01:27:57.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86b9d9b700 1 --2- 172.21.15.70:0/2687708425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86bc103490 0x7f86bc074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:27:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86ba59c700 1 --2- 172.21.15.70:0/2687708425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86bc102290 0x7f86bc075fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:27:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86bad9d700 1 --2- 172.21.15.70:0/2687708425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86bc103e30 0x7f86bc074b70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f86b4007ee0 tx=0x7f86b400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:27:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86ab7fe700 1 -- 172.21.15.70:0/2687708425 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b400dbd0 con 0x7f86bc103e30 2024-04-01T01:27:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86c0bb2700 1 -- 172.21.15.70:0/2687708425 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86bc1a24a0 con 0x7f86bc103e30 2024-04-01T01:27:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86ab7fe700 1 -- 172.21.15.70:0/2687708425 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86b400dd70 con 0x7f86bc103e30 2024-04-01T01:27:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.114+0000 7f86ab7fe700 1 -- 172.21.15.70:0/2687708425 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86b4005a20 con 0x7f86bc103e30 2024-04-01T01:27:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.115+0000 7f86c0bb2700 1 -- 172.21.15.70:0/2687708425 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86bc1a29b0 con 0x7f86bc103e30 2024-04-01T01:27:57.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.115+0000 7f86c0bb2700 1 -- 172.21.15.70:0/2687708425 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86bc068f00 con 0x7f86bc103e30 2024-04-01T01:27:57.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.116+0000 7f86ab7fe700 1 -- 172.21.15.70:0/2687708425 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f86b401a020 con 0x7f86bc103e30 2024-04-01T01:27:57.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.117+0000 7f86ab7fe700 1 --2- 172.21.15.70:0/2687708425 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f86a40a2530 0x7f86a40a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:27:57.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.117+0000 7f86ba59c700 1 --2- 172.21.15.70:0/2687708425 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f86a40a2530 0x7f86a40a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:27:57.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.117+0000 7f86ab7fe700 1 -- 172.21.15.70:0/2687708425 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f86b40cd080 con 0x7f86bc103e30 2024-04-01T01:27:57.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.117+0000 7f86ba59c700 1 --2- 172.21.15.70:0/2687708425 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f86a40a2530 0x7f86a40a49f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f86ac005fd0 tx=0x7f86ac005e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:27:57.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.122+0000 7f86ab7fe700 1 -- 172.21.15.70:0/2687708425 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f86b4010070 con 0x7f86bc103e30 2024-04-01T01:27:57.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:27:57.326+0000 7f86c0bb2700 1 -- 172.21.15.70:0/2687708425 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f86bc062680 con 0x7f86a40a2530 2024-04-01T01:27:57.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:57 smithi189 ceph-mon[35753]: osdmap e40: 6 total, 6 up, 6 in 2024-04-01T01:27:57.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:57 smithi189 ceph-mon[35753]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:27:57.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:57 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:27:57.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:57 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:27:57.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:57 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:57.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:57 smithi070 ceph-mon[32179]: osdmap e40: 6 total, 6 up, 6 in 2024-04-01T01:27:57.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:57 smithi070 ceph-mon[32179]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:27:57.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:57 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:27:57.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:57 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:27:57.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:57 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:57.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:57 smithi081 ceph-mon[35142]: osdmap e40: 6 total, 6 up, 6 in 2024-04-01T01:27:57.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:57 smithi081 ceph-mon[35142]: pgmap v139: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:27:57.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:57 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:27:57.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:57 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:27:57.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:57 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:27:58.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:58 smithi070 ceph-mon[32179]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:27:58.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:58 smithi070 ceph-mon[32179]: osdmap e41: 6 total, 6 up, 6 in 2024-04-01T01:27:58.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:58 smithi081 ceph-mon[35142]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:27:58.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:58 smithi081 ceph-mon[35142]: osdmap e41: 6 total, 6 up, 6 in 2024-04-01T01:27:58.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:58 smithi189 ceph-mon[35753]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:27:58.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:58 smithi189 ceph-mon[35753]: osdmap e41: 6 total, 6 up, 6 in 2024-04-01T01:27:59.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:59 smithi081 ceph-mon[35142]: osdmap e42: 6 total, 6 up, 6 in 2024-04-01T01:27:59.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:27:59 smithi081 ceph-mon[35142]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:27:59.899 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:59 smithi189 ceph-mon[35753]: osdmap e42: 6 total, 6 up, 6 in 2024-04-01T01:27:59.899 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:27:59 smithi189 ceph-mon[35753]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:00.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:59 smithi070 ceph-mon[32179]: osdmap e42: 6 total, 6 up, 6 in 2024-04-01T01:28:00.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:27:59 smithi070 ceph-mon[32179]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:00.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:00 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/492684545' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0446322b-061f-4134-aa65-2f4639238ce5"}]: dispatch 2024-04-01T01:28:00.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:00 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/492684545' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0446322b-061f-4134-aa65-2f4639238ce5"}]': finished 2024-04-01T01:28:00.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:00 smithi070 ceph-mon[32179]: osdmap e43: 7 total, 6 up, 7 in 2024-04-01T01:28:00.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:00.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:00 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/492684545' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0446322b-061f-4134-aa65-2f4639238ce5"}]: dispatch 2024-04-01T01:28:00.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:00 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/492684545' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0446322b-061f-4134-aa65-2f4639238ce5"}]': finished 2024-04-01T01:28:00.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:00 smithi081 ceph-mon[35142]: osdmap e43: 7 total, 6 up, 7 in 2024-04-01T01:28:00.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:00.917 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:00 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/492684545' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0446322b-061f-4134-aa65-2f4639238ce5"}]: dispatch 2024-04-01T01:28:00.917 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:00 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/492684545' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0446322b-061f-4134-aa65-2f4639238ce5"}]': finished 2024-04-01T01:28:00.917 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:00 smithi189 ceph-mon[35753]: osdmap e43: 7 total, 6 up, 7 in 2024-04-01T01:28:00.917 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:01.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:01 smithi081 ceph-mon[35142]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-04-01T01:28:01.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:01 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/106279947' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:28:01.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:01 smithi189 ceph-mon[35753]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-04-01T01:28:01.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:01 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/106279947' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:28:02.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:01 smithi070 ceph-mon[32179]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-04-01T01:28:02.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:01 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/106279947' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:28:03.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:03 smithi081 ceph-mon[35142]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-01T01:28:03.955 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:03 smithi070 ceph-mon[32179]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-01T01:28:03.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:03 smithi189 ceph-mon[35753]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-01T01:28:05.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:05 smithi081 ceph-mon[35142]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-01T01:28:05.830 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:05 smithi070 ceph-mon[32179]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-01T01:28:05.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:05 smithi189 ceph-mon[35753]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-01T01:28:07.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:07 smithi081 ceph-mon[35142]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-01T01:28:07.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-01T01:28:07.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:07.888 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:07 smithi070 ceph-mon[32179]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-01T01:28:07.888 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-01T01:28:07.889 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:07.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:07 smithi189 ceph-mon[35753]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-01T01:28:07.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-01T01:28:07.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:08.652 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:08 smithi070 ceph-mon[32179]: Deploying daemon osd.6 on smithi070 2024-04-01T01:28:08.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:08 smithi189 ceph-mon[35753]: Deploying daemon osd.6 on smithi070 2024-04-01T01:28:09.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:08 smithi081 ceph-mon[35142]: Deploying daemon osd.6 on smithi070 2024-04-01T01:28:09.944 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:09 smithi070 ceph-mon[32179]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-01T01:28:09.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:09 smithi189 ceph-mon[35753]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-01T01:28:10.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:09 smithi081 ceph-mon[35142]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-01T01:28:11.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:11 smithi189 ceph-mon[35753]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:12.001 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:11 smithi070 ceph-mon[32179]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:11 smithi081 ceph-mon[35142]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:13.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:13 smithi081 ceph-mon[35142]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:13.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:13.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:13.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:28:13.867 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:13 smithi070 ceph-mon[32179]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:13.867 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:13 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:13.867 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:13 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:13.867 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:13 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:28:13.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:13 smithi189 ceph-mon[35753]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:13.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:13 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:13.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:13 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:13.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:13 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:28:15.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:15 smithi070 ceph-mon[32179]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:15.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:15 smithi189 ceph-mon[35753]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:16.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:15 smithi081 ceph-mon[35142]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:16.283 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 6 on host 'smithi070' 2024-04-01T01:28:16.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:28:16.282+0000 7f86ab7fe700 1 -- 172.21.15.70:0/2687708425 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f86bc062680 con 0x7f86a40a2530 2024-04-01T01:28:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:28:16.285+0000 7f86c0bb2700 1 -- 172.21.15.70:0/2687708425 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f86a40a2530 msgr2=0x7f86a40a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:28:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:28:16.285+0000 7f86c0bb2700 1 --2- 172.21.15.70:0/2687708425 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f86a40a2530 0x7f86a40a49f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f86ac005fd0 tx=0x7f86ac005e40 comp rx=0 tx=0).stop 2024-04-01T01:28:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:28:16.285+0000 7f86c0bb2700 1 -- 172.21.15.70:0/2687708425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86bc103e30 msgr2=0x7f86bc074b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:28:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:28:16.285+0000 7f86c0bb2700 1 --2- 172.21.15.70:0/2687708425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86bc103e30 0x7f86bc074b70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f86b4007ee0 tx=0x7f86b400d3b0 comp rx=0 tx=0).stop 2024-04-01T01:28:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:28:16.285+0000 7f86c0bb2700 1 -- 172.21.15.70:0/2687708425 shutdown_connections 2024-04-01T01:28:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:28:16.285+0000 7f86c0bb2700 1 --2- 172.21.15.70:0/2687708425 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f86a40a2530 0x7f86a40a49f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:16.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:28:16.285+0000 7f86c0bb2700 1 --2- 172.21.15.70:0/2687708425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f86bc103e30 0x7f86bc074b70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:16.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:28:16.285+0000 7f86c0bb2700 1 --2- 172.21.15.70:0/2687708425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86bc102290 0x7f86bc075fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:16.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:28:16.285+0000 7f86c0bb2700 1 --2- 172.21.15.70:0/2687708425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86bc103490 0x7f86bc074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:16.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:28:16.285+0000 7f86c0bb2700 1 -- 172.21.15.70:0/2687708425 >> 172.21.15.70:0/2687708425 conn(0x7f86bc0fd850 msgr2=0x7f86bc106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:28:16.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:28:16.285+0000 7f86c0bb2700 1 -- 172.21.15.70:0/2687708425 shutdown_connections 2024-04-01T01:28:16.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:28:16.285+0000 7f86c0bb2700 1 -- 172.21.15.70:0/2687708425 wait complete. 2024-04-01T01:28:17.004 DEBUG:teuthology.orchestra.run.smithi070:osd.6> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.6.service 2024-04-01T01:28:17.006 INFO:tasks.cephadm:Deploying osd.7 on smithi081 with /dev/vg_nvme/lv_2... 2024-04-01T01:28:17.006 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-01T01:28:17.254 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-01T01:28:17.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:17 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:17.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:17 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:17.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:17 smithi081 ceph-mon[35142]: from='osd.6 [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-01T01:28:17.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:17 smithi081 ceph-mon[35142]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:17.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:17 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:17.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:17 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:17.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:17 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:17.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:17 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:17.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:17 smithi189 ceph-mon[35753]: from='osd.6 [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-01T01:28:17.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:17 smithi189 ceph-mon[35753]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:17.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:17 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:17.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:17 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:17.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:17 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:17.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:17 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:17.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:17 smithi070 ceph-mon[32179]: from='osd.6 [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-01T01:28:17.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:17 smithi070 ceph-mon[32179]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:17.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:17 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:17.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:17 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:18.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:18 smithi070 ceph-mon[32179]: from='osd.6 [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-01T01:28:18.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:18 smithi070 ceph-mon[32179]: osdmap e44: 7 total, 6 up, 7 in 2024-04-01T01:28:18.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:18 smithi070 ceph-mon[32179]: from='osd.6 [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-01T01:28:18.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:18.505 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:28:18 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:28:18.290+0000 7f0a85821700 -1 osd.6 0 waiting for initial osdmap 2024-04-01T01:28:18.505 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:28:18 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:28:18.300+0000 7f0a81614700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-01T01:28:18.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:18 smithi081 ceph-mon[35142]: from='osd.6 [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-01T01:28:18.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:18 smithi081 ceph-mon[35142]: osdmap e44: 7 total, 6 up, 7 in 2024-04-01T01:28:18.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:18 smithi081 ceph-mon[35142]: from='osd.6 [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-01T01:28:18.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:18.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:18 smithi189 ceph-mon[35753]: from='osd.6 [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-01T01:28:18.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:18 smithi189 ceph-mon[35753]: osdmap e44: 7 total, 6 up, 7 in 2024-04-01T01:28:18.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:18 smithi189 ceph-mon[35753]: from='osd.6 [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-01T01:28:18.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:19.292 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-01T01:28:19.292 INFO:teuthology.orchestra.run.smithi081.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-01T01:28:19.292 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10+0 records in 2024-04-01T01:28:19.292 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2024-04-01T01:28:19.292 INFO:teuthology.orchestra.run.smithi081.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0119195 s, 880 MB/s 2024-04-01T01:28:19.292 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping successful for: 2024-04-01T01:28:19.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:19 smithi070 ceph-mon[32179]: from='osd.6 [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-01T01:28:19.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:19 smithi070 ceph-mon[32179]: osdmap e45: 7 total, 6 up, 7 in 2024-04-01T01:28:19.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:19.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:19.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:19 smithi070 ceph-mon[32179]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:19.390 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:19.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:19 smithi081 ceph-mon[35142]: from='osd.6 [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-01T01:28:19.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:19 smithi081 ceph-mon[35142]: osdmap e45: 7 total, 6 up, 7 in 2024-04-01T01:28:19.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:19.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:19.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:19 smithi081 ceph-mon[35142]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:19.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:19.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:19 smithi189 ceph-mon[35753]: from='osd.6 [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-01T01:28:19.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:19 smithi189 ceph-mon[35753]: osdmap e45: 7 total, 6 up, 7 in 2024-04-01T01:28:19.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:19.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:19.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:19 smithi189 ceph-mon[35753]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-04-01T01:28:19.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:19.933 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi081:vg_nvme/lv_2 2024-04-01T01:28:20.182 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-01T01:28:20.393 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:20 smithi070 ceph-mon[32179]: purged_snaps scrub starts 2024-04-01T01:28:20.393 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:20 smithi070 ceph-mon[32179]: purged_snaps scrub ok 2024-04-01T01:28:20.393 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:20 smithi070 ceph-mon[32179]: osd.6 [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673] boot 2024-04-01T01:28:20.393 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:20 smithi070 ceph-mon[32179]: osdmap e46: 7 total, 7 up, 7 in 2024-04-01T01:28:20.393 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:20 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:20.576 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:20 smithi081 ceph-mon[35142]: purged_snaps scrub starts 2024-04-01T01:28:20.576 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:20 smithi081 ceph-mon[35142]: purged_snaps scrub ok 2024-04-01T01:28:20.576 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:20 smithi081 ceph-mon[35142]: osd.6 [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673] boot 2024-04-01T01:28:20.576 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:20 smithi081 ceph-mon[35142]: osdmap e46: 7 total, 7 up, 7 in 2024-04-01T01:28:20.576 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:20.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:20 smithi189 ceph-mon[35753]: purged_snaps scrub starts 2024-04-01T01:28:20.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:20 smithi189 ceph-mon[35753]: purged_snaps scrub ok 2024-04-01T01:28:20.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:20 smithi189 ceph-mon[35753]: osd.6 [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673] boot 2024-04-01T01:28:20.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:20 smithi189 ceph-mon[35753]: osdmap e46: 7 total, 7 up, 7 in 2024-04-01T01:28:20.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:20 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-01T01:28:21.380 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:21 smithi081 ceph-mon[35142]: osdmap e47: 7 total, 7 up, 7 in 2024-04-01T01:28:21.380 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:21 smithi081 ceph-mon[35142]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:21.380 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:21.380 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:21.380 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:21.380 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:21.380 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:21.381 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:21.381 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:21.381 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:28:21.381 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:21.381 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:28:21.381 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:21.381 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:28:21.381 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:21.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:21 smithi189 ceph-mon[35753]: osdmap e47: 7 total, 7 up, 7 in 2024-04-01T01:28:21.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:21 smithi189 ceph-mon[35753]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:21.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:21.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:21.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:21.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:21.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:21.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:21.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:21.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:28:21.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:21.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:28:21.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:21.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:28:21.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:21.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:21 smithi070 ceph-mon[32179]: osdmap e47: 7 total, 7 up, 7 in 2024-04-01T01:28:21.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:21 smithi070 ceph-mon[32179]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:21.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:21.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:21.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:21.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:21.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:21.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:21.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:21.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:28:21.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:21.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:28:21.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:21.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:28:21.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:21.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.911+0000 7f8e07a44700 1 -- 172.21.15.81:0/1416917078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e00102370 msgr2=0x7f8e001027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:28:21.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.911+0000 7f8e07a44700 1 --2- 172.21.15.81:0/1416917078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e00102370 0x7f8e001027d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8df00099c0 tx=0x7f8df0009cd0 comp rx=0 tx=0).stop 2024-04-01T01:28:21.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.911+0000 7f8e07a44700 1 -- 172.21.15.81:0/1416917078 shutdown_connections 2024-04-01T01:28:21.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.911+0000 7f8e07a44700 1 --2- 172.21.15.81:0/1416917078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e00102d10 0x7f8e001095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:21.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.911+0000 7f8e07a44700 1 --2- 172.21.15.81:0/1416917078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e00101170 0x7f8e00101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:21.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.911+0000 7f8e07a44700 1 --2- 172.21.15.81:0/1416917078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e00102370 0x7f8e001027d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:21.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.911+0000 7f8e07a44700 1 -- 172.21.15.81:0/1416917078 >> 172.21.15.81:0/1416917078 conn(0x7f8e000fc710 msgr2=0x7f8e000feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:28:21.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.911+0000 7f8e07a44700 1 -- 172.21.15.81:0/1416917078 shutdown_connections 2024-04-01T01:28:21.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.911+0000 7f8e07a44700 1 -- 172.21.15.81:0/1416917078 wait complete. 2024-04-01T01:28:21.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.911+0000 7f8e07a44700 1 Processor -- start 2024-04-01T01:28:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.911+0000 7f8e07a44700 1 -- start start 2024-04-01T01:28:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e07a44700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e00101170 0x7f8e001983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:28:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e07a44700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e00102370 0x7f8e00198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:28:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e07a44700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e00102d10 0x7f8e0019f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:28:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e07a44700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0010cc10 con 0x7f8e00102370 2024-04-01T01:28:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e07a44700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0010ca90 con 0x7f8e00102d10 2024-04-01T01:28:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e07a44700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0010cd90 con 0x7f8e00101170 2024-04-01T01:28:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e05fe1700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e00102d10 0x7f8e0019f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:28:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e05fe1700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e00102d10 0x7f8e0019f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:49708/0 (socket says 172.21.15.81:49708) 2024-04-01T01:28:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e05fe1700 1 -- 172.21.15.81:0/2006128721 learned_addr learned my addr 172.21.15.81:0/2006128721 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-01T01:28:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e04fdf700 1 --2- 172.21.15.81:0/2006128721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e00102370 0x7f8e00198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:28:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e057e0700 1 --2- 172.21.15.81:0/2006128721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e00101170 0x7f8e001983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:28:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e05fe1700 1 -- 172.21.15.81:0/2006128721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e00101170 msgr2=0x7f8e001983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:28:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e05fe1700 1 --2- 172.21.15.81:0/2006128721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e00101170 0x7f8e001983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e05fe1700 1 -- 172.21.15.81:0/2006128721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e00102370 msgr2=0x7f8e00198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:28:21.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e05fe1700 1 --2- 172.21.15.81:0/2006128721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e00102370 0x7f8e00198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:21.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e05fe1700 1 -- 172.21.15.81:0/2006128721 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e001a0030 con 0x7f8e00102d10 2024-04-01T01:28:21.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e05fe1700 1 --2- 172.21.15.81:0/2006128721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e00102d10 0x7f8e0019f990 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8df8007d70 tx=0x7f8df800a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:28:21.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8df67fc700 1 -- 172.21.15.81:0/2006128721 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df8016020 con 0x7f8e00102d10 2024-04-01T01:28:21.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e07a44700 1 -- 172.21.15.81:0/2006128721 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e001a0320 con 0x7f8e00102d10 2024-04-01T01:28:21.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8df67fc700 1 -- 172.21.15.81:0/2006128721 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8df800ac90 con 0x7f8e00102d10 2024-04-01T01:28:21.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8df67fc700 1 -- 172.21.15.81:0/2006128721 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df80115e0 con 0x7f8e00102d10 2024-04-01T01:28:21.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.912+0000 7f8e07a44700 1 -- 172.21.15.81:0/2006128721 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e001a0860 con 0x7f8e00102d10 2024-04-01T01:28:21.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.913+0000 7f8e07a44700 1 -- 172.21.15.81:0/2006128721 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e00068f00 con 0x7f8e00102d10 2024-04-01T01:28:21.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.914+0000 7f8df67fc700 1 -- 172.21.15.81:0/2006128721 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f8df8019020 con 0x7f8e00102d10 2024-04-01T01:28:21.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.915+0000 7f8df67fc700 1 --2- 172.21.15.81:0/2006128721 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8dec0a2530 0x7f8dec0a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:28:21.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.915+0000 7f8df67fc700 1 -- 172.21.15.81:0/2006128721 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f8df80cdd90 con 0x7f8e00102d10 2024-04-01T01:28:21.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.915+0000 7f8e057e0700 1 --2- 172.21.15.81:0/2006128721 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8dec0a2530 0x7f8dec0a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:28:21.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.916+0000 7f8e057e0700 1 --2- 172.21.15.81:0/2006128721 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8dec0a2530 0x7f8dec0a49f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8e001021d0 tx=0x7f8dfc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:28:21.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:21.918+0000 7f8df67fc700 1 -- 172.21.15.81:0/2006128721 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8df8020020 con 0x7f8e00102d10 2024-04-01T01:28:22.122 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:22.122+0000 7f8e07a44700 1 -- 172.21.15.81:0/2006128721 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f8e000fe910 con 0x7f8dec0a2530 2024-04-01T01:28:22.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:22 smithi081 ceph-mon[35142]: Detected new or changed devices on smithi070 2024-04-01T01:28:22.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:22 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi070 to 5696M 2024-04-01T01:28:22.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:28:22.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:28:22.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:22.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:22 smithi189 ceph-mon[35753]: Detected new or changed devices on smithi070 2024-04-01T01:28:22.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:22 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi070 to 5696M 2024-04-01T01:28:22.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:28:22.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:28:22.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:22.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:22 smithi070 ceph-mon[32179]: Detected new or changed devices on smithi070 2024-04-01T01:28:22.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:22 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi070 to 5696M 2024-04-01T01:28:22.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:28:22.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:28:22.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:23.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:23 smithi081 ceph-mon[35142]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:28:23.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:23 smithi081 ceph-mon[35142]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:23.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:23 smithi189 ceph-mon[35753]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:28:23.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:23 smithi189 ceph-mon[35753]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:23.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:23 smithi070 ceph-mon[32179]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:28:23.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:23 smithi070 ceph-mon[32179]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:25.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:25 smithi081 ceph-mon[35142]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:25.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:25 smithi081 ceph-mon[35142]: from='client.? 172.21.15.81:0/2296679528' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d654db8e-5fdc-47ff-99d9-db10263fa237"}]: dispatch 2024-04-01T01:28:25.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:25 smithi081 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d654db8e-5fdc-47ff-99d9-db10263fa237"}]: dispatch 2024-04-01T01:28:25.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:25 smithi081 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d654db8e-5fdc-47ff-99d9-db10263fa237"}]': finished 2024-04-01T01:28:25.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:25 smithi081 ceph-mon[35142]: osdmap e48: 8 total, 7 up, 8 in 2024-04-01T01:28:25.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:25 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-01T01:28:25.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:25 smithi081 ceph-mon[35142]: from='client.? 172.21.15.81:0/753096914' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:28:25.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:25 smithi189 ceph-mon[35753]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:25.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:25 smithi189 ceph-mon[35753]: from='client.? 172.21.15.81:0/2296679528' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d654db8e-5fdc-47ff-99d9-db10263fa237"}]: dispatch 2024-04-01T01:28:25.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:25 smithi189 ceph-mon[35753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d654db8e-5fdc-47ff-99d9-db10263fa237"}]: dispatch 2024-04-01T01:28:25.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:25 smithi189 ceph-mon[35753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d654db8e-5fdc-47ff-99d9-db10263fa237"}]': finished 2024-04-01T01:28:25.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:25 smithi189 ceph-mon[35753]: osdmap e48: 8 total, 7 up, 8 in 2024-04-01T01:28:25.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:25 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-01T01:28:25.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:25 smithi189 ceph-mon[35753]: from='client.? 172.21.15.81:0/753096914' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:28:26.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:25 smithi070 ceph-mon[32179]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:26.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:25 smithi070 ceph-mon[32179]: from='client.? 172.21.15.81:0/2296679528' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d654db8e-5fdc-47ff-99d9-db10263fa237"}]: dispatch 2024-04-01T01:28:26.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:25 smithi070 ceph-mon[32179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d654db8e-5fdc-47ff-99d9-db10263fa237"}]: dispatch 2024-04-01T01:28:26.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:25 smithi070 ceph-mon[32179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d654db8e-5fdc-47ff-99d9-db10263fa237"}]': finished 2024-04-01T01:28:26.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:25 smithi070 ceph-mon[32179]: osdmap e48: 8 total, 7 up, 8 in 2024-04-01T01:28:26.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:25 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-01T01:28:26.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:25 smithi070 ceph-mon[32179]: from='client.? 172.21.15.81:0/753096914' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:28:27.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:27 smithi189 ceph-mon[35753]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:28.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:27 smithi070 ceph-mon[32179]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:28.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:27 smithi081 ceph-mon[35142]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:29.835 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:29 smithi081 ceph-mon[35142]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:29.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:29 smithi189 ceph-mon[35753]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:30.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:29 smithi070 ceph-mon[32179]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:31.899 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:31 smithi081 ceph-mon[35142]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:31.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:31 smithi189 ceph-mon[35753]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:32.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:31 smithi070 ceph-mon[32179]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:32.786 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-01T01:28:32.786 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:32.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-01T01:28:32.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:33.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-01T01:28:33.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:33.800 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:33 smithi081 ceph-mon[35142]: Deploying daemon osd.7 on smithi081 2024-04-01T01:28:33.800 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:33 smithi081 ceph-mon[35142]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:33.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:33 smithi189 ceph-mon[35753]: Deploying daemon osd.7 on smithi081 2024-04-01T01:28:33.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:33 smithi189 ceph-mon[35753]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:34.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:33 smithi070 ceph-mon[32179]: Deploying daemon osd.7 on smithi081 2024-04-01T01:28:34.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:33 smithi070 ceph-mon[32179]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:35.744 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:35 smithi081 ceph-mon[35142]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:35.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:35 smithi189 ceph-mon[35753]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:36.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:35 smithi070 ceph-mon[32179]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:37.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:37 smithi189 ceph-mon[35753]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:37.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:37 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:37.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:37 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:37.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:37 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:28:38.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:37 smithi070 ceph-mon[32179]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:38.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:37 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:38.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:37 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:38.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:37 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:28:38.044 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:37 smithi081 ceph-mon[35142]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:38.044 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:37 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:38.044 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:37 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:38.044 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:37 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:28:39.967 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:39 smithi081 ceph-mon[35142]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:39.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:39 smithi189 ceph-mon[35753]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:40.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:39 smithi070 ceph-mon[32179]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:40.839 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 7 on host 'smithi081' 2024-04-01T01:28:40.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:40.838+0000 7f8df67fc700 1 -- 172.21.15.81:0/2006128721 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8e000fe910 con 0x7f8dec0a2530 2024-04-01T01:28:40.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:40.841+0000 7f8e07a44700 1 -- 172.21.15.81:0/2006128721 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8dec0a2530 msgr2=0x7f8dec0a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:28:40.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:40.841+0000 7f8e07a44700 1 --2- 172.21.15.81:0/2006128721 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8dec0a2530 0x7f8dec0a49f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8e001021d0 tx=0x7f8dfc00b410 comp rx=0 tx=0).stop 2024-04-01T01:28:40.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:40.841+0000 7f8e07a44700 1 -- 172.21.15.81:0/2006128721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e00102d10 msgr2=0x7f8e0019f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:28:40.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:40.841+0000 7f8e07a44700 1 --2- 172.21.15.81:0/2006128721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e00102d10 0x7f8e0019f990 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8df8007d70 tx=0x7f8df800a340 comp rx=0 tx=0).stop 2024-04-01T01:28:40.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:40.841+0000 7f8e07a44700 1 -- 172.21.15.81:0/2006128721 shutdown_connections 2024-04-01T01:28:40.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:40.841+0000 7f8e07a44700 1 --2- 172.21.15.81:0/2006128721 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8dec0a2530 0x7f8dec0a49f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:40.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:40.841+0000 7f8e07a44700 1 --2- 172.21.15.81:0/2006128721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e00102d10 0x7f8e0019f990 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:40.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:40.841+0000 7f8e07a44700 1 --2- 172.21.15.81:0/2006128721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e00101170 0x7f8e001983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:40.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:40.841+0000 7f8e07a44700 1 --2- 172.21.15.81:0/2006128721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e00102370 0x7f8e00198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:40.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:40.841+0000 7f8e07a44700 1 -- 172.21.15.81:0/2006128721 >> 172.21.15.81:0/2006128721 conn(0x7f8e000fc710 msgr2=0x7f8e000fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:28:40.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:40.841+0000 7f8e07a44700 1 -- 172.21.15.81:0/2006128721 shutdown_connections 2024-04-01T01:28:40.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:28:40.841+0000 7f8e07a44700 1 -- 172.21.15.81:0/2006128721 wait complete. 2024-04-01T01:28:41.578 DEBUG:teuthology.orchestra.run.smithi081:osd.7> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.7.service 2024-04-01T01:28:41.580 INFO:tasks.cephadm:Deploying osd.8 on smithi189 with /dev/vg_nvme/lv_2... 2024-04-01T01:28:41.581 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-01T01:28:41.811 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:41 smithi081 ceph-mon[35142]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:41.811 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:41 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:41.811 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:41 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:41.811 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:41 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:41.811 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:41 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:41.811 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:41 smithi081 ceph-mon[35142]: from='osd.7 [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-01T01:28:41.811 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:41 smithi081 ceph-mon[35142]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-01T01:28:41.824 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-01T01:28:41.940 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:41 smithi189 ceph-mon[35753]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:41.940 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:41 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:41.940 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:41 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:41.940 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:41 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:41.940 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:41 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:41.940 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:41 smithi189 ceph-mon[35753]: from='osd.7 [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-01T01:28:41.940 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:41 smithi189 ceph-mon[35753]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-01T01:28:42.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:41 smithi070 ceph-mon[32179]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:42.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:41 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:42.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:41 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:42.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:41 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:42.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:41 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:42.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:41 smithi070 ceph-mon[32179]: from='osd.7 [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-01T01:28:42.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:41 smithi070 ceph-mon[32179]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-01T01:28:43.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:43 smithi081 ceph-mon[35142]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-01T01:28:43.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:43 smithi081 ceph-mon[35142]: osdmap e49: 8 total, 7 up, 8 in 2024-04-01T01:28:43.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-01T01:28:43.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:43 smithi081 ceph-mon[35142]: from='osd.7 [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:28:43.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:43 smithi081 ceph-mon[35142]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:28:43.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:43 smithi081 ceph-mon[35142]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:43.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:43 smithi189 ceph-mon[35753]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-01T01:28:43.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:43 smithi189 ceph-mon[35753]: osdmap e49: 8 total, 7 up, 8 in 2024-04-01T01:28:43.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-01T01:28:43.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:43 smithi189 ceph-mon[35753]: from='osd.7 [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:28:43.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:43 smithi189 ceph-mon[35753]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:28:43.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:43 smithi189 ceph-mon[35753]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:43.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:43 smithi070 ceph-mon[32179]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-01T01:28:43.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:43 smithi070 ceph-mon[32179]: osdmap e49: 8 total, 7 up, 8 in 2024-04-01T01:28:43.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-01T01:28:43.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:43 smithi070 ceph-mon[32179]: from='osd.7 [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:28:43.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:43 smithi070 ceph-mon[32179]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:28:43.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:43 smithi070 ceph-mon[32179]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:43.744 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-01T01:28:43.744 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-01T01:28:43.744 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-04-01T01:28:43.744 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-04-01T01:28:43.744 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130895 s, 801 MB/s 2024-04-01T01:28:43.744 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-04-01T01:28:44.181 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:28:44 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:28:44.014+0000 7f2e5a2a6700 -1 osd.7 0 waiting for initial osdmap 2024-04-01T01:28:44.181 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:28:44 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:28:44.022+0000 7f2e52091700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-01T01:28:44.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:44 smithi189 ceph-mon[35753]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-01T01:28:44.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:44 smithi189 ceph-mon[35753]: osdmap e50: 8 total, 7 up, 8 in 2024-04-01T01:28:44.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:44 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-01T01:28:44.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:44 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-01T01:28:44.312 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:44 smithi189 ceph-mon[35753]: from='osd.7 ' entity='osd.7' 2024-04-01T01:28:44.341 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi189:vg_nvme/lv_2 2024-04-01T01:28:44.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:44 smithi070 ceph-mon[32179]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-01T01:28:44.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:44 smithi070 ceph-mon[32179]: osdmap e50: 8 total, 7 up, 8 in 2024-04-01T01:28:44.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:44 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-01T01:28:44.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:44 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-01T01:28:44.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:44 smithi070 ceph-mon[32179]: from='osd.7 ' entity='osd.7' 2024-04-01T01:28:44.543 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:44 smithi081 ceph-mon[35142]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-01T01:28:44.543 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:44 smithi081 ceph-mon[35142]: osdmap e50: 8 total, 7 up, 8 in 2024-04-01T01:28:44.543 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:44 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-01T01:28:44.543 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:44 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-01T01:28:44.543 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:44 smithi081 ceph-mon[35142]: from='osd.7 ' entity='osd.7' 2024-04-01T01:28:44.598 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-01T01:28:45.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: purged_snaps scrub starts 2024-04-01T01:28:45.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: purged_snaps scrub ok 2024-04-01T01:28:45.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: osd.7 [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716] boot 2024-04-01T01:28:45.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: osdmap e51: 8 total, 8 up, 8 in 2024-04-01T01:28:45.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-01T01:28:45.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:45.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: Detected new or changed devices on smithi081 2024-04-01T01:28:45.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:45.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:45.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:45.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:45.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:45.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi081 to 5696M 2024-04-01T01:28:45.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:45.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:45.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:28:45.500 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:45.500 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:28:45.500 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:45.500 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:28:45.500 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:45.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: purged_snaps scrub starts 2024-04-01T01:28:45.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: purged_snaps scrub ok 2024-04-01T01:28:45.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: osd.7 [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716] boot 2024-04-01T01:28:45.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: osdmap e51: 8 total, 8 up, 8 in 2024-04-01T01:28:45.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-01T01:28:45.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:45.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: Detected new or changed devices on smithi081 2024-04-01T01:28:45.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:45.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:45.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:45.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:45.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:45.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi081 to 5696M 2024-04-01T01:28:45.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:45.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:45.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:28:45.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:45.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:28:45.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:45.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:28:45.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:45.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: purged_snaps scrub starts 2024-04-01T01:28:45.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: purged_snaps scrub ok 2024-04-01T01:28:45.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: osd.7 [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716] boot 2024-04-01T01:28:45.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: osdmap e51: 8 total, 8 up, 8 in 2024-04-01T01:28:45.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-01T01:28:45.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-04-01T01:28:45.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: Detected new or changed devices on smithi081 2024-04-01T01:28:45.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:45.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:45.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:45.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:45.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:28:45.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi081 to 5696M 2024-04-01T01:28:45.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:45.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:45.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:28:45.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:45.580 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:28:45.580 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:45.580 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:28:45.580 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:28:46.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.084+0000 7ff4e75c7700 1 -- 172.21.15.189:0/724081975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4e0103490 msgr2=0x7ff4e01038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:28:46.086 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.084+0000 7ff4e75c7700 1 --2- 172.21.15.189:0/724081975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4e0103490 0x7ff4e01038f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0009990 tx=0x7ff4d0009ca0 comp rx=0 tx=0).stop 2024-04-01T01:28:46.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.084+0000 7ff4e75c7700 1 -- 172.21.15.189:0/724081975 shutdown_connections 2024-04-01T01:28:46.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.084+0000 7ff4e75c7700 1 --2- 172.21.15.189:0/724081975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4e0103e30 0x7ff4e010a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:46.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.084+0000 7ff4e75c7700 1 --2- 172.21.15.189:0/724081975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4e0102290 0x7ff4e01026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:46.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.084+0000 7ff4e75c7700 1 --2- 172.21.15.189:0/724081975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4e0103490 0x7ff4e01038f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:46.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.084+0000 7ff4e75c7700 1 -- 172.21.15.189:0/724081975 >> 172.21.15.189:0/724081975 conn(0x7ff4e00fd850 msgr2=0x7ff4e00ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:28:46.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.084+0000 7ff4e75c7700 1 -- 172.21.15.189:0/724081975 shutdown_connections 2024-04-01T01:28:46.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.084+0000 7ff4e75c7700 1 -- 172.21.15.189:0/724081975 wait complete. 2024-04-01T01:28:46.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.085+0000 7ff4e75c7700 1 Processor -- start 2024-04-01T01:28:46.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.085+0000 7ff4e75c7700 1 -- start start 2024-04-01T01:28:46.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.085+0000 7ff4e75c7700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4e0102290 0x7ff4e0075fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:28:46.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.085+0000 7ff4e75c7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4e0103490 0x7ff4e0074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:28:46.087 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.085+0000 7ff4e75c7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4e0103e30 0x7ff4e0074b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:28:46.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.085+0000 7ff4e75c7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e010dda0 con 0x7ff4e0103490 2024-04-01T01:28:46.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.085+0000 7ff4e75c7700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e010dc20 con 0x7ff4e0102290 2024-04-01T01:28:46.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.085+0000 7ff4e75c7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e010df20 con 0x7ff4e0103e30 2024-04-01T01:28:46.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.085+0000 7ff4e5b64700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4e0103e30 0x7ff4e0074b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:28:46.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.085+0000 7ff4e5b64700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4e0103e30 0x7ff4e0074b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:46646/0 (socket says 172.21.15.189:46646) 2024-04-01T01:28:46.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.085+0000 7ff4e5b64700 1 -- 172.21.15.189:0/324819744 learned_addr learned my addr 172.21.15.189:0/324819744 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-01T01:28:46.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.086+0000 7ff4e5363700 1 --2- 172.21.15.189:0/324819744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4e0102290 0x7ff4e0075fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:28:46.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.086+0000 7ff4e4b62700 1 --2- 172.21.15.189:0/324819744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4e0103490 0x7ff4e0074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:28:46.088 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.086+0000 7ff4e5b64700 1 -- 172.21.15.189:0/324819744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4e0102290 msgr2=0x7ff4e0075fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:28:46.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.086+0000 7ff4e5b64700 1 --2- 172.21.15.189:0/324819744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4e0102290 0x7ff4e0075fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:46.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.086+0000 7ff4e5b64700 1 -- 172.21.15.189:0/324819744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4e0103490 msgr2=0x7ff4e0074630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:28:46.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.086+0000 7ff4e5b64700 1 --2- 172.21.15.189:0/324819744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4e0103490 0x7ff4e0074630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:28:46.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.086+0000 7ff4e5b64700 1 -- 172.21.15.189:0/324819744 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4e0075420 con 0x7ff4e0103e30 2024-04-01T01:28:46.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.086+0000 7ff4e5b64700 1 --2- 172.21.15.189:0/324819744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4e0103e30 0x7ff4e0074b70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8007d70 tx=0x7ff4d800a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:28:46.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.086+0000 7ff4d67fc700 1 -- 172.21.15.189:0/324819744 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8016020 con 0x7ff4e0103e30 2024-04-01T01:28:46.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.086+0000 7ff4d67fc700 1 -- 172.21.15.189:0/324819744 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4d800ad40 con 0x7ff4e0103e30 2024-04-01T01:28:46.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.086+0000 7ff4d67fc700 1 -- 172.21.15.189:0/324819744 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8011620 con 0x7ff4e0103e30 2024-04-01T01:28:46.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.086+0000 7ff4e75c7700 1 -- 172.21.15.189:0/324819744 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4e01a2540 con 0x7ff4e0103e30 2024-04-01T01:28:46.089 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.086+0000 7ff4e75c7700 1 -- 172.21.15.189:0/324819744 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4e01a2a80 con 0x7ff4e0103e30 2024-04-01T01:28:46.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.087+0000 7ff4e75c7700 1 -- 172.21.15.189:0/324819744 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4e0068f00 con 0x7ff4e0103e30 2024-04-01T01:28:46.090 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.088+0000 7ff4d67fc700 1 -- 172.21.15.189:0/324819744 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7ff4d8019020 con 0x7ff4e0103e30 2024-04-01T01:28:46.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.089+0000 7ff4d67fc700 1 --2- 172.21.15.189:0/324819744 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff4cc0a2530 0x7ff4cc0a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:28:46.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.089+0000 7ff4d67fc700 1 -- 172.21.15.189:0/324819744 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5426+0+0 (secure 0 0 0) 0x7ff4d80ce030 con 0x7ff4e0103e30 2024-04-01T01:28:46.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.089+0000 7ff4e5363700 1 --2- 172.21.15.189:0/324819744 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff4cc0a2530 0x7ff4cc0a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:28:46.091 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.090+0000 7ff4e5363700 1 --2- 172.21.15.189:0/324819744 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff4cc0a2530 0x7ff4cc0a49f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc006fd0 tx=0x7ff4dc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:28:46.092 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.093+0000 7ff4d67fc700 1 -- 172.21.15.189:0/324819744 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff4d8020020 con 0x7ff4e0103e30 2024-04-01T01:28:46.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:28:46.301+0000 7ff4e75c7700 1 -- 172.21.15.189:0/324819744 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff4e0062680 con 0x7ff4cc0a2530 2024-04-01T01:28:46.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:46 smithi189 ceph-mon[35753]: osdmap e52: 8 total, 8 up, 8 in 2024-04-01T01:28:46.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:46 smithi070 ceph-mon[32179]: osdmap e52: 8 total, 8 up, 8 in 2024-04-01T01:28:46.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:46 smithi081 ceph-mon[35142]: osdmap e52: 8 total, 8 up, 8 in 2024-04-01T01:28:47.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:47 smithi189 ceph-mon[35753]: osdmap e53: 8 total, 8 up, 8 in 2024-04-01T01:28:47.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:47 smithi189 ceph-mon[35753]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:28:47.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:47 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:28:47.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:47 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:28:47.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:47 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:47.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:47 smithi189 ceph-mon[35753]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:47.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:47 smithi070 ceph-mon[32179]: osdmap e53: 8 total, 8 up, 8 in 2024-04-01T01:28:47.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:47 smithi070 ceph-mon[32179]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:28:47.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:28:47.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:28:47.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:47.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:47 smithi070 ceph-mon[32179]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:47.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:47 smithi081 ceph-mon[35142]: osdmap e53: 8 total, 8 up, 8 in 2024-04-01T01:28:47.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:47 smithi081 ceph-mon[35142]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:28:47.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:47 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:28:47.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:47 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:28:47.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:47 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:47.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:47 smithi081 ceph-mon[35142]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:48.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:48 smithi189 ceph-mon[35753]: osdmap e54: 8 total, 8 up, 8 in 2024-04-01T01:28:48.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:48 smithi070 ceph-mon[32179]: osdmap e54: 8 total, 8 up, 8 in 2024-04-01T01:28:48.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:48 smithi081 ceph-mon[35142]: osdmap e54: 8 total, 8 up, 8 in 2024-04-01T01:28:49.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:49 smithi189 ceph-mon[35753]: osdmap e55: 8 total, 8 up, 8 in 2024-04-01T01:28:49.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:49 smithi189 ceph-mon[35753]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:49.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:49 smithi189 ceph-mon[35753]: from='client.? 172.21.15.189:0/190113117' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "073456b5-3f5d-491a-86a4-317161989697"}]: dispatch 2024-04-01T01:28:49.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:49 smithi189 ceph-mon[35753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "073456b5-3f5d-491a-86a4-317161989697"}]: dispatch 2024-04-01T01:28:49.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:49 smithi189 ceph-mon[35753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "073456b5-3f5d-491a-86a4-317161989697"}]': finished 2024-04-01T01:28:49.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:49 smithi189 ceph-mon[35753]: osdmap e56: 9 total, 8 up, 9 in 2024-04-01T01:28:49.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:49 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-01T01:28:49.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:49 smithi070 ceph-mon[32179]: osdmap e55: 8 total, 8 up, 8 in 2024-04-01T01:28:49.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:49 smithi070 ceph-mon[32179]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:49.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:49 smithi070 ceph-mon[32179]: from='client.? 172.21.15.189:0/190113117' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "073456b5-3f5d-491a-86a4-317161989697"}]: dispatch 2024-04-01T01:28:49.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:49 smithi070 ceph-mon[32179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "073456b5-3f5d-491a-86a4-317161989697"}]: dispatch 2024-04-01T01:28:49.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:49 smithi070 ceph-mon[32179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "073456b5-3f5d-491a-86a4-317161989697"}]': finished 2024-04-01T01:28:49.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:49 smithi070 ceph-mon[32179]: osdmap e56: 9 total, 8 up, 9 in 2024-04-01T01:28:49.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:49 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-01T01:28:49.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:49 smithi081 ceph-mon[35142]: osdmap e55: 8 total, 8 up, 8 in 2024-04-01T01:28:49.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:49 smithi081 ceph-mon[35142]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:49.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:49 smithi081 ceph-mon[35142]: from='client.? 172.21.15.189:0/190113117' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "073456b5-3f5d-491a-86a4-317161989697"}]: dispatch 2024-04-01T01:28:49.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:49 smithi081 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "073456b5-3f5d-491a-86a4-317161989697"}]: dispatch 2024-04-01T01:28:49.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:49 smithi081 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "073456b5-3f5d-491a-86a4-317161989697"}]': finished 2024-04-01T01:28:49.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:49 smithi081 ceph-mon[35142]: osdmap e56: 9 total, 8 up, 9 in 2024-04-01T01:28:49.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:49 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-01T01:28:50.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:50 smithi189 ceph-mon[35753]: from='client.? 172.21.15.189:0/3846808723' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:28:50.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:50 smithi070 ceph-mon[32179]: from='client.? 172.21.15.189:0/3846808723' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:28:50.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:50 smithi081 ceph-mon[35142]: from='client.? 172.21.15.189:0/3846808723' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:28:51.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:51 smithi189 ceph-mon[35753]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:51.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:51 smithi070 ceph-mon[32179]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:51.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:51 smithi081 ceph-mon[35142]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:53.961 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:53 smithi189 ceph-mon[35753]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:54.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:53 smithi070 ceph-mon[32179]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:54.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:53 smithi081 ceph-mon[35142]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:55.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:55 smithi189 ceph-mon[35753]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:56.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:55 smithi070 ceph-mon[32179]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:56.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:55 smithi081 ceph-mon[35142]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:56.856 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-01T01:28:56.856 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:57.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-01T01:28:57.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:57.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-01T01:28:57.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:28:57.844 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:57 smithi189 ceph-mon[35753]: Deploying daemon osd.8 on smithi189 2024-04-01T01:28:57.844 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:57 smithi189 ceph-mon[35753]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:58.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:57 smithi070 ceph-mon[32179]: Deploying daemon osd.8 on smithi189 2024-04-01T01:28:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:57 smithi070 ceph-mon[32179]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:58.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:57 smithi081 ceph-mon[35142]: Deploying daemon osd.8 on smithi189 2024-04-01T01:28:58.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:57 smithi081 ceph-mon[35142]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:28:59.847 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:28:59 smithi189 ceph-mon[35753]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:29:00.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:28:59 smithi070 ceph-mon[32179]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:29:00.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:28:59 smithi081 ceph-mon[35142]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:29:01.879 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:01 smithi081 ceph-mon[35142]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:29:01.880 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:01 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:01.880 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:01 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:01.880 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:01 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:01.935 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:01 smithi189 ceph-mon[35753]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:29:01.936 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:01 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:01.936 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:01 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:01.936 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:01 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:02.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:01 smithi070 ceph-mon[32179]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:29:02.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:01 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:02.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:01 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:02.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:01 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:03.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:03 smithi189 ceph-mon[35753]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:29:04.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:03 smithi070 ceph-mon[32179]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:29:04.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:03 smithi081 ceph-mon[35142]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:29:05.164 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:29:05.163+0000 7ff4d67fc700 1 -- 172.21.15.189:0/324819744 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff4e0062680 con 0x7ff4cc0a2530 2024-04-01T01:29:05.165 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 8 on host 'smithi189' 2024-04-01T01:29:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:29:05.165+0000 7ff4e75c7700 1 -- 172.21.15.189:0/324819744 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff4cc0a2530 msgr2=0x7ff4cc0a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:29:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:29:05.165+0000 7ff4e75c7700 1 --2- 172.21.15.189:0/324819744 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff4cc0a2530 0x7ff4cc0a49f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc006fd0 tx=0x7ff4dc008040 comp rx=0 tx=0).stop 2024-04-01T01:29:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:29:05.166+0000 7ff4e75c7700 1 -- 172.21.15.189:0/324819744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4e0103e30 msgr2=0x7ff4e0074b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:29:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:29:05.166+0000 7ff4e75c7700 1 --2- 172.21.15.189:0/324819744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4e0103e30 0x7ff4e0074b70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8007d70 tx=0x7ff4d800a340 comp rx=0 tx=0).stop 2024-04-01T01:29:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:29:05.166+0000 7ff4e75c7700 1 -- 172.21.15.189:0/324819744 shutdown_connections 2024-04-01T01:29:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:29:05.166+0000 7ff4e75c7700 1 --2- 172.21.15.189:0/324819744 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff4cc0a2530 0x7ff4cc0a49f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:29:05.166+0000 7ff4e75c7700 1 --2- 172.21.15.189:0/324819744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4e0103e30 0x7ff4e0074b70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:29:05.166+0000 7ff4e75c7700 1 --2- 172.21.15.189:0/324819744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4e0102290 0x7ff4e0075fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:29:05.166+0000 7ff4e75c7700 1 --2- 172.21.15.189:0/324819744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4e0103490 0x7ff4e0074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:29:05.166+0000 7ff4e75c7700 1 -- 172.21.15.189:0/324819744 >> 172.21.15.189:0/324819744 conn(0x7ff4e00fd850 msgr2=0x7ff4e0106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:29:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:29:05.166+0000 7ff4e75c7700 1 -- 172.21.15.189:0/324819744 shutdown_connections 2024-04-01T01:29:05.167 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:29:05.166+0000 7ff4e75c7700 1 -- 172.21.15.189:0/324819744 wait complete. 2024-04-01T01:29:05.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:05.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:05.422 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:05.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:29:05.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:05.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:05 smithi189 ceph-mon[35753]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:29:05.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:05.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:05.423 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:05.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:05.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:05.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:05.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:29:05.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:05.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:05 smithi081 ceph-mon[35142]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:29:05.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:05.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:05.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:05.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:05.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:05.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:05.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:29:05.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:05.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:05 smithi070 ceph-mon[32179]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:29:05.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:05.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:05.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:05.837 DEBUG:teuthology.orchestra.run.smithi189:osd.8> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.8.service 2024-04-01T01:29:05.839 INFO:tasks.cephadm:Deploying osd.9 on smithi070 with /dev/vg_nvme/lv_1... 2024-04-01T01:29:05.839 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-01T01:29:06.087 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:29:06.348 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:06 smithi189 ceph-mon[35753]: from='osd.8 [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-01T01:29:06.348 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:06 smithi189 ceph-mon[35753]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-01T01:29:06.473 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:06 smithi070 ceph-mon[32179]: from='osd.8 [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-01T01:29:06.473 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:06 smithi070 ceph-mon[32179]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-01T01:29:06.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:06 smithi081 ceph-mon[35142]: from='osd.8 [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-01T01:29:06.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:06 smithi081 ceph-mon[35142]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-01T01:29:07.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:07 smithi189 ceph-mon[35753]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-01T01:29:07.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:07 smithi189 ceph-mon[35753]: osdmap e57: 9 total, 8 up, 9 in 2024-04-01T01:29:07.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:07 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-01T01:29:07.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:07 smithi189 ceph-mon[35753]: from='osd.8 [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:29:07.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:07 smithi189 ceph-mon[35753]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:29:07.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:07 smithi189 ceph-mon[35753]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:29:07.499 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:29:07 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:29:07.329+0000 7fd20ac9e700 -1 osd.8 0 waiting for initial osdmap 2024-04-01T01:29:07.499 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:29:07 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:29:07.339+0000 7fd206a91700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-01T01:29:07.565 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:07 smithi070 ceph-mon[32179]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-01T01:29:07.565 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:07 smithi070 ceph-mon[32179]: osdmap e57: 9 total, 8 up, 9 in 2024-04-01T01:29:07.565 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:07 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-01T01:29:07.565 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:07 smithi070 ceph-mon[32179]: from='osd.8 [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:29:07.566 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:07 smithi070 ceph-mon[32179]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:29:07.566 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:07 smithi070 ceph-mon[32179]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:29:07.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:07 smithi081 ceph-mon[35142]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-01T01:29:07.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:07 smithi081 ceph-mon[35142]: osdmap e57: 9 total, 8 up, 9 in 2024-04-01T01:29:07.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:07 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-01T01:29:07.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:07 smithi081 ceph-mon[35142]: from='osd.8 [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:29:07.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:07 smithi081 ceph-mon[35142]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:29:07.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:07 smithi081 ceph-mon[35142]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-01T01:29:08.144 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-01T01:29:08.144 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-01T01:29:08.144 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-04-01T01:29:08.144 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-04-01T01:29:08.144 INFO:teuthology.orchestra.run.smithi070.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00859795 s, 1.2 GB/s 2024-04-01T01:29:08.144 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-04-01T01:29:08.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:08 smithi189 ceph-mon[35753]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-01T01:29:08.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:08 smithi189 ceph-mon[35753]: osdmap e58: 9 total, 8 up, 9 in 2024-04-01T01:29:08.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:08 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-01T01:29:08.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:08 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-01T01:29:08.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:08 smithi070 ceph-mon[32179]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-01T01:29:08.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:08 smithi070 ceph-mon[32179]: osdmap e58: 9 total, 8 up, 9 in 2024-04-01T01:29:08.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:08 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-01T01:29:08.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:08 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-01T01:29:08.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:08 smithi081 ceph-mon[35142]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-01T01:29:08.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:08 smithi081 ceph-mon[35142]: osdmap e58: 9 total, 8 up, 9 in 2024-04-01T01:29:08.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:08 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-01T01:29:08.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:08 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-01T01:29:08.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi070:vg_nvme/lv_1 2024-04-01T01:29:09.181 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:29:09.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:09 smithi070 ceph-mon[32179]: purged_snaps scrub starts 2024-04-01T01:29:09.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:09 smithi070 ceph-mon[32179]: purged_snaps scrub ok 2024-04-01T01:29:09.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:09 smithi070 ceph-mon[32179]: osd.8 [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021] boot 2024-04-01T01:29:09.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:09 smithi070 ceph-mon[32179]: osdmap e59: 9 total, 9 up, 9 in 2024-04-01T01:29:09.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-01T01:29:09.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:09 smithi070 ceph-mon[32179]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:09.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:09.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:09.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:09.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:09.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:09.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:09.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:09.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:29:09.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:09.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:09 smithi189 ceph-mon[35753]: purged_snaps scrub starts 2024-04-01T01:29:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:09 smithi189 ceph-mon[35753]: purged_snaps scrub ok 2024-04-01T01:29:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:09 smithi189 ceph-mon[35753]: osd.8 [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021] boot 2024-04-01T01:29:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:09 smithi189 ceph-mon[35753]: osdmap e59: 9 total, 9 up, 9 in 2024-04-01T01:29:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-01T01:29:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:09 smithi189 ceph-mon[35753]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:29:09.750 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:09.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:09 smithi081 ceph-mon[35142]: purged_snaps scrub starts 2024-04-01T01:29:09.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:09 smithi081 ceph-mon[35142]: purged_snaps scrub ok 2024-04-01T01:29:09.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:09 smithi081 ceph-mon[35142]: osd.8 [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021] boot 2024-04-01T01:29:09.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:09 smithi081 ceph-mon[35142]: osdmap e59: 9 total, 9 up, 9 in 2024-04-01T01:29:09.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-01T01:29:09.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:09 smithi081 ceph-mon[35142]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:09.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:09.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:09.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:09.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:09.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:09.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:09.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:09.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:29:09.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:10.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:10 smithi070 ceph-mon[32179]: Detected new or changed devices on smithi189 2024-04-01T01:29:10.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:10 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi189 to 5696M 2024-04-01T01:29:10.596 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:10 smithi070 ceph-mon[32179]: osdmap e60: 9 total, 9 up, 9 in 2024-04-01T01:29:10.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:10 smithi189 ceph-mon[35753]: Detected new or changed devices on smithi189 2024-04-01T01:29:10.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:10 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi189 to 5696M 2024-04-01T01:29:10.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:10 smithi189 ceph-mon[35753]: osdmap e60: 9 total, 9 up, 9 in 2024-04-01T01:29:10.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:10 smithi081 ceph-mon[35142]: Detected new or changed devices on smithi189 2024-04-01T01:29:10.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:10 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi189 to 5696M 2024-04-01T01:29:10.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:10 smithi081 ceph-mon[35142]: osdmap e60: 9 total, 9 up, 9 in 2024-04-01T01:29:11.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.003+0000 7f61609bf700 1 -- 172.21.15.70:0/1593879047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c102d40 msgr2=0x7f615c1095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:29:11.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.003+0000 7f61609bf700 1 --2- 172.21.15.70:0/1593879047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c102d40 0x7f615c1095f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f615400b0d0 tx=0x7f615400b3e0 comp rx=0 tx=0).stop 2024-04-01T01:29:11.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.003+0000 7f61609bf700 1 -- 172.21.15.70:0/1593879047 shutdown_connections 2024-04-01T01:29:11.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.003+0000 7f61609bf700 1 --2- 172.21.15.70:0/1593879047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c102d40 0x7f615c1095f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:11.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.003+0000 7f61609bf700 1 --2- 172.21.15.70:0/1593879047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f615c1011a0 0x7f615c1015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:11.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.003+0000 7f61609bf700 1 --2- 172.21.15.70:0/1593879047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f615c1023a0 0x7f615c102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:11.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.003+0000 7f61609bf700 1 -- 172.21.15.70:0/1593879047 >> 172.21.15.70:0/1593879047 conn(0x7f615c0fc760 msgr2=0x7f615c0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:29:11.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.003+0000 7f61609bf700 1 -- 172.21.15.70:0/1593879047 shutdown_connections 2024-04-01T01:29:11.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.003+0000 7f61609bf700 1 -- 172.21.15.70:0/1593879047 wait complete. 2024-04-01T01:29:11.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.004+0000 7f61609bf700 1 Processor -- start 2024-04-01T01:29:11.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.004+0000 7f61609bf700 1 -- start start 2024-04-01T01:29:11.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.004+0000 7f61609bf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c1011a0 0x7f615c06b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:29:11.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f61609bf700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f615c1023a0 0x7f615c19d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:29:11.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f61609bf700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f615c102d40 0x7f615c19f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:29:11.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f61609bf700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f615c10cc20 con 0x7f615c1011a0 2024-04-01T01:29:11.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f61609bf700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f615c10caa0 con 0x7f615c102d40 2024-04-01T01:29:11.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f61609bf700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f615c10cda0 con 0x7f615c1023a0 2024-04-01T01:29:11.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f615a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c1011a0 0x7f615c06b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:29:11.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f615a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c1011a0 0x7f615c06b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38042/0 (socket says 172.21.15.70:38042) 2024-04-01T01:29:11.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f615a59c700 1 -- 172.21.15.70:0/627697213 learned_addr learned my addr 172.21.15.70:0/627697213 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:29:11.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f6159d9b700 1 --2- 172.21.15.70:0/627697213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f615c1023a0 0x7f615c19d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:29:11.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f615ad9d700 1 --2- 172.21.15.70:0/627697213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f615c102d40 0x7f615c19f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:29:11.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f615a59c700 1 -- 172.21.15.70:0/627697213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f615c1023a0 msgr2=0x7f615c19d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:29:11.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f615a59c700 1 --2- 172.21.15.70:0/627697213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f615c1023a0 0x7f615c19d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:11.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f615a59c700 1 -- 172.21.15.70:0/627697213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f615c102d40 msgr2=0x7f615c19f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:29:11.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f615a59c700 1 --2- 172.21.15.70:0/627697213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f615c102d40 0x7f615c19f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:11.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f615a59c700 1 -- 172.21.15.70:0/627697213 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615c19ff30 con 0x7f615c1011a0 2024-04-01T01:29:11.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f615a59c700 1 --2- 172.21.15.70:0/627697213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c1011a0 0x7f615c06b500 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f615000c970 tx=0x7f615000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:29:11.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f614b7fe700 1 -- 172.21.15.70:0/627697213 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150004220 con 0x7f615c1011a0 2024-04-01T01:29:11.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f614b7fe700 1 -- 172.21.15.70:0/627697213 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61500043c0 con 0x7f615c1011a0 2024-04-01T01:29:11.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f61609bf700 1 -- 172.21.15.70:0/627697213 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f615c1a0220 con 0x7f615c1011a0 2024-04-01T01:29:11.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f614b7fe700 1 -- 172.21.15.70:0/627697213 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150003c30 con 0x7f615c1011a0 2024-04-01T01:29:11.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.005+0000 7f61609bf700 1 -- 172.21.15.70:0/627697213 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f615c1a07b0 con 0x7f615c1011a0 2024-04-01T01:29:11.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.006+0000 7f61609bf700 1 -- 172.21.15.70:0/627697213 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f615c068f00 con 0x7f615c1011a0 2024-04-01T01:29:11.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.007+0000 7f614b7fe700 1 -- 172.21.15.70:0/627697213 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f61500076d0 con 0x7f615c1011a0 2024-04-01T01:29:11.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.008+0000 7f614b7fe700 1 --2- 172.21.15.70:0/627697213 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61440a25b0 0x7f61440a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:29:11.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.008+0000 7f6159d9b700 1 --2- 172.21.15.70:0/627697213 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61440a25b0 0x7f61440a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:29:11.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.008+0000 7f614b7fe700 1 -- 172.21.15.70:0/627697213 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5829+0+0 (secure 0 0 0) 0x7f61500cd840 con 0x7f615c1011a0 2024-04-01T01:29:11.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.008+0000 7f6159d9b700 1 --2- 172.21.15.70:0/627697213 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61440a25b0 0x7f61440a4a70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f615c06ba60 tx=0x7f614c00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:29:11.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.013+0000 7f614b7fe700 1 -- 172.21.15.70:0/627697213 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6150014030 con 0x7f615c1011a0 2024-04-01T01:29:11.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:11.217+0000 7f61609bf700 1 -- 172.21.15.70:0/627697213 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f615c062680 con 0x7f61440a25b0 2024-04-01T01:29:11.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:11 smithi189 ceph-mon[35753]: osdmap e61: 9 total, 9 up, 9 in 2024-04-01T01:29:11.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:11 smithi189 ceph-mon[35753]: osdmap e62: 9 total, 9 up, 9 in 2024-04-01T01:29:11.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:11 smithi189 ceph-mon[35753]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:11.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:11 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:29:11.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:11 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:29:11.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:11 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:11.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:11 smithi070 ceph-mon[32179]: osdmap e61: 9 total, 9 up, 9 in 2024-04-01T01:29:11.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:11 smithi070 ceph-mon[32179]: osdmap e62: 9 total, 9 up, 9 in 2024-04-01T01:29:11.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:11 smithi070 ceph-mon[32179]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:11.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:11 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:29:11.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:11 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:29:11.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:11 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:11.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:11 smithi081 ceph-mon[35142]: osdmap e61: 9 total, 9 up, 9 in 2024-04-01T01:29:11.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:11 smithi081 ceph-mon[35142]: osdmap e62: 9 total, 9 up, 9 in 2024-04-01T01:29:11.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:11 smithi081 ceph-mon[35142]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 252 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:11.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:11 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:29:11.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:11 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:29:11.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:11 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:12.827 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:12 smithi070 ceph-mon[32179]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:29:12.827 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:12 smithi070 ceph-mon[32179]: osdmap e63: 9 total, 9 up, 9 in 2024-04-01T01:29:12.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:12 smithi081 ceph-mon[35142]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:29:12.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:12 smithi081 ceph-mon[35142]: osdmap e63: 9 total, 9 up, 9 in 2024-04-01T01:29:12.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:12 smithi189 ceph-mon[35753]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:29:12.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:12 smithi189 ceph-mon[35753]: osdmap e63: 9 total, 9 up, 9 in 2024-04-01T01:29:13.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:13 smithi081 ceph-mon[35142]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:13.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:13 smithi189 ceph-mon[35753]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:14.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:13 smithi070 ceph-mon[32179]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:14.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:14 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/894202157' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d"}]: dispatch 2024-04-01T01:29:14.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:14 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/894202157' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d"}]': finished 2024-04-01T01:29:14.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:14 smithi081 ceph-mon[35142]: osdmap e64: 10 total, 9 up, 10 in 2024-04-01T01:29:14.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:14 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:14.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:14 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/894202157' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d"}]: dispatch 2024-04-01T01:29:14.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:14 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/894202157' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d"}]': finished 2024-04-01T01:29:14.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:14 smithi189 ceph-mon[35753]: osdmap e64: 10 total, 9 up, 10 in 2024-04-01T01:29:14.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:14 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:15.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:14 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/894202157' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d"}]: dispatch 2024-04-01T01:29:15.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:14 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/894202157' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d"}]': finished 2024-04-01T01:29:15.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:14 smithi070 ceph-mon[32179]: osdmap e64: 10 total, 9 up, 10 in 2024-04-01T01:29:15.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:14 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:15.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:15 smithi081 ceph-mon[35142]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:15.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:15 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1768284738' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:29:15.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:15 smithi189 ceph-mon[35753]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:15.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:15 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1768284738' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:29:16.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:15 smithi070 ceph-mon[32179]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:16.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:15 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1768284738' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:29:17.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:17 smithi189 ceph-mon[35753]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-01T01:29:18.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:17 smithi070 ceph-mon[32179]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-01T01:29:18.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:17 smithi081 ceph-mon[35142]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-01T01:29:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:19 smithi189 ceph-mon[35753]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-01T01:29:20.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:19 smithi070 ceph-mon[32179]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-01T01:29:20.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:19 smithi081 ceph-mon[35142]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-01T01:29:21.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:21 smithi189 ceph-mon[35753]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-01T01:29:22.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:21 smithi070 ceph-mon[32179]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-01T01:29:22.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:21 smithi081 ceph-mon[35142]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-01T01:29:22.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-01T01:29:22.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:22.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:22 smithi189 ceph-mon[35753]: Deploying daemon osd.9 on smithi070 2024-04-01T01:29:23.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-01T01:29:23.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:23.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:22 smithi070 ceph-mon[32179]: Deploying daemon osd.9 on smithi070 2024-04-01T01:29:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-01T01:29:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:22 smithi081 ceph-mon[35142]: Deploying daemon osd.9 on smithi070 2024-04-01T01:29:23.781 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:23 smithi070 ceph-mon[32179]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-01T01:29:23.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:23 smithi189 ceph-mon[35753]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-01T01:29:24.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:23 smithi081 ceph-mon[35142]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-01T01:29:25.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:25 smithi070 ceph-mon[32179]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-01T01:29:25.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:25 smithi189 ceph-mon[35753]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-01T01:29:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:25 smithi081 ceph-mon[35142]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-04-01T01:29:27.967 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:27 smithi070 ceph-mon[32179]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-01T01:29:27.967 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:27.967 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:27.967 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:27.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:27 smithi189 ceph-mon[35753]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-01T01:29:27.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:27.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:27.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:28.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:27 smithi081 ceph-mon[35142]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-01T01:29:28.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:28.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:28.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:29.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:29 smithi070 ceph-mon[32179]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:29.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:29 smithi189 ceph-mon[35753]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:30.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:29 smithi081 ceph-mon[35142]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:30.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:30.824+0000 7f614b7fe700 1 -- 172.21.15.70:0/627697213 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f615c062680 con 0x7f61440a25b0 2024-04-01T01:29:30.823 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 9 on host 'smithi070' 2024-04-01T01:29:30.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:30.826+0000 7f61609bf700 1 -- 172.21.15.70:0/627697213 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61440a25b0 msgr2=0x7f61440a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:29:30.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:30.826+0000 7f61609bf700 1 --2- 172.21.15.70:0/627697213 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61440a25b0 0x7f61440a4a70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f615c06ba60 tx=0x7f614c00a380 comp rx=0 tx=0).stop 2024-04-01T01:29:30.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:30.826+0000 7f61609bf700 1 -- 172.21.15.70:0/627697213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c1011a0 msgr2=0x7f615c06b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:29:30.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:30.826+0000 7f61609bf700 1 --2- 172.21.15.70:0/627697213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c1011a0 0x7f615c06b500 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f615000c970 tx=0x7f615000cc80 comp rx=0 tx=0).stop 2024-04-01T01:29:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:30.826+0000 7f61609bf700 1 -- 172.21.15.70:0/627697213 shutdown_connections 2024-04-01T01:29:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:30.826+0000 7f61609bf700 1 --2- 172.21.15.70:0/627697213 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61440a25b0 0x7f61440a4a70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:30.826+0000 7f61609bf700 1 --2- 172.21.15.70:0/627697213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f615c102d40 0x7f615c19f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:30.826+0000 7f61609bf700 1 --2- 172.21.15.70:0/627697213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c1011a0 0x7f615c06b500 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:30.826+0000 7f61609bf700 1 --2- 172.21.15.70:0/627697213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f615c1023a0 0x7f615c19d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:30.826+0000 7f61609bf700 1 -- 172.21.15.70:0/627697213 >> 172.21.15.70:0/627697213 conn(0x7f615c0fc760 msgr2=0x7f615c105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:29:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:30.826+0000 7f61609bf700 1 -- 172.21.15.70:0/627697213 shutdown_connections 2024-04-01T01:29:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:29:30.826+0000 7f61609bf700 1 -- 172.21.15.70:0/627697213 wait complete. 2024-04-01T01:29:31.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:31 smithi070 ceph-mon[32179]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:31.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:31 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:31.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:31 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:31.696 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:31 smithi070 ceph-mon[32179]: from='osd.9 [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-01T01:29:31.723 DEBUG:teuthology.orchestra.run.smithi070:osd.9> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.9.service 2024-04-01T01:29:31.725 INFO:tasks.cephadm:Deploying osd.10 on smithi081 with /dev/vg_nvme/lv_1... 2024-04-01T01:29:31.725 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-01T01:29:31.880 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:31 smithi081 ceph-mon[35142]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:31.880 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:31 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:31.881 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:31 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:31.881 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:31 smithi081 ceph-mon[35142]: from='osd.9 [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-01T01:29:31.972 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-01T01:29:31.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:31 smithi189 ceph-mon[35753]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:31.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:31 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:31.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:31 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:31.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:31 smithi189 ceph-mon[35753]: from='osd.9 [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-01T01:29:33.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:32 smithi081 ceph-mon[35142]: from='osd.9 [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-01T01:29:33.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:32 smithi081 ceph-mon[35142]: osdmap e65: 10 total, 9 up, 10 in 2024-04-01T01:29:33.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:32 smithi081 ceph-mon[35142]: from='osd.9 [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-01T01:29:33.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:33.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:33.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:32 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:33.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:32 smithi081 ceph-mon[35142]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:33.194 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:32 smithi070 ceph-mon[32179]: from='osd.9 [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-01T01:29:33.194 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:32 smithi070 ceph-mon[32179]: osdmap e65: 10 total, 9 up, 10 in 2024-04-01T01:29:33.194 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:32 smithi070 ceph-mon[32179]: from='osd.9 [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-01T01:29:33.194 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:33.194 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:33.195 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:32 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:33.195 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:32 smithi070 ceph-mon[32179]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:33.195 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:29:32 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:29:32.801+0000 7fcf334f9700 -1 osd.9 0 waiting for initial osdmap 2024-04-01T01:29:33.195 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:29:32 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:29:32.814+0000 7fcf2f2ec700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-01T01:29:33.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:32 smithi189 ceph-mon[35753]: from='osd.9 [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-01T01:29:33.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:32 smithi189 ceph-mon[35753]: osdmap e65: 10 total, 9 up, 10 in 2024-04-01T01:29:33.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:32 smithi189 ceph-mon[35753]: from='osd.9 [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-01T01:29:33.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:33.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:33.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:32 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:33.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:32 smithi189 ceph-mon[35753]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:33.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:33 smithi070 ceph-mon[32179]: from='osd.9 [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-01T01:29:33.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:33 smithi070 ceph-mon[32179]: osdmap e66: 10 total, 9 up, 10 in 2024-04-01T01:29:33.881 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:33.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:33 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:33 smithi081 ceph-mon[35142]: from='osd.9 [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-01T01:29:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:33 smithi081 ceph-mon[35142]: osdmap e66: 10 total, 9 up, 10 in 2024-04-01T01:29:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:33 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:34.091 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-01T01:29:34.091 INFO:teuthology.orchestra.run.smithi081.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-01T01:29:34.091 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10+0 records in 2024-04-01T01:29:34.092 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2024-04-01T01:29:34.092 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117775 s, 890 MB/s 2024-04-01T01:29:34.092 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping successful for: 2024-04-01T01:29:34.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:33 smithi189 ceph-mon[35753]: from='osd.9 [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-01T01:29:34.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:33 smithi189 ceph-mon[35753]: osdmap e66: 10 total, 9 up, 10 in 2024-04-01T01:29:34.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:34.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:33 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:34.720 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi081:vg_nvme/lv_1 2024-04-01T01:29:34.957 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-01T01:29:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:34 smithi070 ceph-mon[32179]: purged_snaps scrub starts 2024-04-01T01:29:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:34 smithi070 ceph-mon[32179]: purged_snaps scrub ok 2024-04-01T01:29:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:34 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:34 smithi070 ceph-mon[32179]: osd.9 [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091] boot 2024-04-01T01:29:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:34 smithi070 ceph-mon[32179]: osdmap e67: 10 total, 10 up, 10 in 2024-04-01T01:29:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:34 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:34 smithi070 ceph-mon[32179]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:35.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:34 smithi081 ceph-mon[35142]: purged_snaps scrub starts 2024-04-01T01:29:35.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:34 smithi081 ceph-mon[35142]: purged_snaps scrub ok 2024-04-01T01:29:35.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:34 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:35.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:34 smithi081 ceph-mon[35142]: osd.9 [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091] boot 2024-04-01T01:29:35.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:34 smithi081 ceph-mon[35142]: osdmap e67: 10 total, 10 up, 10 in 2024-04-01T01:29:35.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:34 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:35.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:34 smithi081 ceph-mon[35142]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:35.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:34 smithi189 ceph-mon[35753]: purged_snaps scrub starts 2024-04-01T01:29:35.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:34 smithi189 ceph-mon[35753]: purged_snaps scrub ok 2024-04-01T01:29:35.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:34 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:35.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:34 smithi189 ceph-mon[35753]: osd.9 [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091] boot 2024-04-01T01:29:35.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:34 smithi189 ceph-mon[35753]: osdmap e67: 10 total, 10 up, 10 in 2024-04-01T01:29:35.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:34 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-01T01:29:35.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:34 smithi189 ceph-mon[35753]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 253 MiB used, 804 GiB / 805 GiB avail 2024-04-01T01:29:36.075 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: osdmap e68: 10 total, 10 up, 10 in 2024-04-01T01:29:36.075 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: Detected new or changed devices on smithi070 2024-04-01T01:29:36.075 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:36.075 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:36.076 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:36.076 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:36.076 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:36.076 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:36.076 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi070 to 4272M 2024-04-01T01:29:36.076 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:36.076 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:36.076 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:29:36.076 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:36.076 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:36.076 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:36.076 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:29:36.076 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:35 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:36.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: osdmap e68: 10 total, 10 up, 10 in 2024-04-01T01:29:36.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: Detected new or changed devices on smithi070 2024-04-01T01:29:36.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:36.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:36.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:36.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:36.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:36.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:36.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi070 to 4272M 2024-04-01T01:29:36.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:36.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:36.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:29:36.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:36.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:36.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:36.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:29:36.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:35 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:36.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: osdmap e68: 10 total, 10 up, 10 in 2024-04-01T01:29:36.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: Detected new or changed devices on smithi070 2024-04-01T01:29:36.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:36.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:36.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:36.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:36.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:36.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:29:36.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi070 to 4272M 2024-04-01T01:29:36.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:36.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:36.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:29:36.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:36.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:36.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:36.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:29:36.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:35 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:36.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.629+0000 7f8faaf9b700 1 -- 172.21.15.81:0/1260162035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fa4102270 msgr2=0x7f8fa4102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:29:36.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.629+0000 7f8faaf9b700 1 --2- 172.21.15.81:0/1260162035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fa4102270 0x7f8fa4102690 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8f980099c0 tx=0x7f8f98009cd0 comp rx=0 tx=0).stop 2024-04-01T01:29:36.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.629+0000 7f8faaf9b700 1 -- 172.21.15.81:0/1260162035 shutdown_connections 2024-04-01T01:29:36.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.629+0000 7f8faaf9b700 1 --2- 172.21.15.81:0/1260162035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fa4103e10 0x7f8fa410a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:36.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.629+0000 7f8faaf9b700 1 --2- 172.21.15.81:0/1260162035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fa4102270 0x7f8fa4102690 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:36.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.629+0000 7f8faaf9b700 1 --2- 172.21.15.81:0/1260162035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fa4103470 0x7f8fa41038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:36.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.629+0000 7f8faaf9b700 1 -- 172.21.15.81:0/1260162035 >> 172.21.15.81:0/1260162035 conn(0x7f8fa40fd850 msgr2=0x7f8fa40ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:29:36.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.629+0000 7f8faaf9b700 1 -- 172.21.15.81:0/1260162035 shutdown_connections 2024-04-01T01:29:36.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.629+0000 7f8faaf9b700 1 -- 172.21.15.81:0/1260162035 wait complete. 2024-04-01T01:29:36.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8faaf9b700 1 Processor -- start 2024-04-01T01:29:36.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8faaf9b700 1 -- start start 2024-04-01T01:29:36.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8faaf9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fa4102270 0x7f8fa4198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:29:36.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8faaf9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fa4103470 0x7f8fa41988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:29:36.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8faaf9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fa4103e10 0x7f8fa419f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:29:36.632 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8faaf9b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa410dd30 con 0x7f8fa4103470 2024-04-01T01:29:36.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8faaf9b700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa410dbb0 con 0x7f8fa4103e10 2024-04-01T01:29:36.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8faaf9b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa410deb0 con 0x7f8fa4102270 2024-04-01T01:29:36.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8fa9538700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fa4103e10 0x7f8fa419f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:29:36.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8fa9538700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fa4103e10 0x7f8fa419f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43252/0 (socket says 172.21.15.81:43252) 2024-04-01T01:29:36.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8fa9538700 1 -- 172.21.15.81:0/4119025079 learned_addr learned my addr 172.21.15.81:0/4119025079 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-01T01:29:36.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8fa3fff700 1 --2- 172.21.15.81:0/4119025079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fa4103470 0x7f8fa41988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:29:36.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8fa8d37700 1 --2- 172.21.15.81:0/4119025079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fa4102270 0x7f8fa4198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:29:36.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8fa9538700 1 -- 172.21.15.81:0/4119025079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fa4102270 msgr2=0x7f8fa4198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:29:36.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8fa9538700 1 --2- 172.21.15.81:0/4119025079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fa4102270 0x7f8fa4198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:36.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8fa9538700 1 -- 172.21.15.81:0/4119025079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fa4103470 msgr2=0x7f8fa41988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:29:36.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.630+0000 7f8fa9538700 1 --2- 172.21.15.81:0/4119025079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fa4103470 0x7f8fa41988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:36.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.631+0000 7f8fa9538700 1 -- 172.21.15.81:0/4119025079 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa419ffd0 con 0x7f8fa4103e10 2024-04-01T01:29:36.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.631+0000 7f8fa9538700 1 --2- 172.21.15.81:0/4119025079 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fa4103e10 0x7f8fa419f930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c007d70 tx=0x7f8f9c00a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:29:36.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.631+0000 7f8fa1ffb700 1 -- 172.21.15.81:0/4119025079 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c016020 con 0x7f8fa4103e10 2024-04-01T01:29:36.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.631+0000 7f8fa1ffb700 1 -- 172.21.15.81:0/4119025079 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f9c00ac90 con 0x7f8fa4103e10 2024-04-01T01:29:36.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.631+0000 7f8faaf9b700 1 -- 172.21.15.81:0/4119025079 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fa41a02c0 con 0x7f8fa4103e10 2024-04-01T01:29:36.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.631+0000 7f8fa1ffb700 1 -- 172.21.15.81:0/4119025079 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c0115e0 con 0x7f8fa4103e10 2024-04-01T01:29:36.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.631+0000 7f8faaf9b700 1 -- 172.21.15.81:0/4119025079 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fa41a07b0 con 0x7f8fa4103e10 2024-04-01T01:29:36.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.632+0000 7f8faaf9b700 1 -- 172.21.15.81:0/4119025079 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fa4068f00 con 0x7f8fa4103e10 2024-04-01T01:29:36.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.633+0000 7f8fa1ffb700 1 -- 172.21.15.81:0/4119025079 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f8f9c019020 con 0x7f8fa4103e10 2024-04-01T01:29:36.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.634+0000 7f8fa1ffb700 1 --2- 172.21.15.81:0/4119025079 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f940a2530 0x7f8f940a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:29:36.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.634+0000 7f8fa1ffb700 1 -- 172.21.15.81:0/4119025079 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f8f9c0cde70 con 0x7f8fa4103e10 2024-04-01T01:29:36.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.634+0000 7f8fa8d37700 1 --2- 172.21.15.81:0/4119025079 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f940a2530 0x7f8f940a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:29:36.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.634+0000 7f8fa8d37700 1 --2- 172.21.15.81:0/4119025079 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f940a2530 0x7f8f940a49f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8f98000c00 tx=0x7f8f98005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:29:36.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.638+0000 7f8fa1ffb700 1 -- 172.21.15.81:0/4119025079 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8f9c020020 con 0x7f8fa4103e10 2024-04-01T01:29:36.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:36.843+0000 7f8faaf9b700 1 -- 172.21.15.81:0/4119025079 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f8fa4062680 con 0x7f8f940a2530 2024-04-01T01:29:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:36 smithi081 ceph-mon[35142]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:37.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:36 smithi189 ceph-mon[35753]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:37.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:36 smithi070 ceph-mon[32179]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:37 smithi081 ceph-mon[35142]: from='client.24311 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:29:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:37 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:29:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:37 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:29:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:37 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:38.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:37 smithi189 ceph-mon[35753]: from='client.24311 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:29:38.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:37 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:29:38.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:37 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:29:38.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:37 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:38.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:37 smithi070 ceph-mon[32179]: from='client.24311 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:29:38.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:37 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:29:38.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:37 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:29:38.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:37 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:38 smithi081 ceph-mon[35142]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:39.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:38 smithi189 ceph-mon[35753]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:39.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:38 smithi070 ceph-mon[32179]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:39 smithi081 ceph-mon[35142]: from='client.? 172.21.15.81:0/2773879630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f58e117c-8f4b-4ca4-bddc-9ae37024d996"}]: dispatch 2024-04-01T01:29:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:39 smithi081 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f58e117c-8f4b-4ca4-bddc-9ae37024d996"}]: dispatch 2024-04-01T01:29:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:39 smithi081 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f58e117c-8f4b-4ca4-bddc-9ae37024d996"}]': finished 2024-04-01T01:29:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:39 smithi081 ceph-mon[35142]: osdmap e69: 11 total, 10 up, 11 in 2024-04-01T01:29:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:39 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:29:40.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:39 smithi189 ceph-mon[35753]: from='client.? 172.21.15.81:0/2773879630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f58e117c-8f4b-4ca4-bddc-9ae37024d996"}]: dispatch 2024-04-01T01:29:40.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:39 smithi189 ceph-mon[35753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f58e117c-8f4b-4ca4-bddc-9ae37024d996"}]: dispatch 2024-04-01T01:29:40.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:39 smithi189 ceph-mon[35753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f58e117c-8f4b-4ca4-bddc-9ae37024d996"}]': finished 2024-04-01T01:29:40.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:39 smithi189 ceph-mon[35753]: osdmap e69: 11 total, 10 up, 11 in 2024-04-01T01:29:40.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:39 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:29:40.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:39 smithi070 ceph-mon[32179]: from='client.? 172.21.15.81:0/2773879630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f58e117c-8f4b-4ca4-bddc-9ae37024d996"}]: dispatch 2024-04-01T01:29:40.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:39 smithi070 ceph-mon[32179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f58e117c-8f4b-4ca4-bddc-9ae37024d996"}]: dispatch 2024-04-01T01:29:40.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:39 smithi070 ceph-mon[32179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f58e117c-8f4b-4ca4-bddc-9ae37024d996"}]': finished 2024-04-01T01:29:40.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:39 smithi070 ceph-mon[32179]: osdmap e69: 11 total, 10 up, 11 in 2024-04-01T01:29:40.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:39 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:29:41.155 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:40 smithi070 ceph-mon[32179]: from='client.? 172.21.15.81:0/262765672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:29:41.155 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:40 smithi070 ceph-mon[32179]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:41.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:40 smithi189 ceph-mon[35753]: from='client.? 172.21.15.81:0/262765672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:29:41.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:40 smithi189 ceph-mon[35753]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:41.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:40 smithi081 ceph-mon[35142]: from='client.? 172.21.15.81:0/262765672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:29:41.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:40 smithi081 ceph-mon[35142]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:43.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:43 smithi081 ceph-mon[35142]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:43.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:43 smithi189 ceph-mon[35753]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:44.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:43 smithi070 ceph-mon[32179]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:45.815 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:45 smithi081 ceph-mon[35142]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:45.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:45 smithi189 ceph-mon[35753]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:46.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:45 smithi070 ceph-mon[32179]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:47.847 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:47 smithi081 ceph-mon[35142]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:47.847 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:47 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-01T01:29:47.847 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:47 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:47.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:47 smithi189 ceph-mon[35753]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:47.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:47 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-01T01:29:47.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:47 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:48.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:47 smithi070 ceph-mon[32179]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:48.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-01T01:29:48.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:47 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:48.957 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:48 smithi081 ceph-mon[35142]: Deploying daemon osd.10 on smithi081 2024-04-01T01:29:48.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:48 smithi189 ceph-mon[35753]: Deploying daemon osd.10 on smithi081 2024-04-01T01:29:49.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:48 smithi070 ceph-mon[32179]: Deploying daemon osd.10 on smithi081 2024-04-01T01:29:49.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:49 smithi189 ceph-mon[35753]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:50.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:49 smithi070 ceph-mon[32179]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:50.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:49 smithi081 ceph-mon[35142]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:51.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:51 smithi081 ceph-mon[35142]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:51.862 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:51 smithi070 ceph-mon[32179]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:51.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:51 smithi189 ceph-mon[35753]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:53.752 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:53 smithi081 ceph-mon[35142]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:53.752 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:53 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:53.752 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:53 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:53.752 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:53 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:53.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:53 smithi189 ceph-mon[35753]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:53.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:53 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:53.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:53 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:53.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:53 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:54.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:53 smithi070 ceph-mon[32179]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:54.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:53 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:54.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:53 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:54.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:53 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:55.906 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:55 smithi081 ceph-mon[35142]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:55.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:55 smithi189 ceph-mon[35753]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:56.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:55 smithi070 ceph-mon[32179]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:56.663 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 10 on host 'smithi081' 2024-04-01T01:29:56.663 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:56.663+0000 7f8fa1ffb700 1 -- 172.21.15.81:0/4119025079 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f8fa4062680 con 0x7f8f940a2530 2024-04-01T01:29:56.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:56.665+0000 7f8faaf9b700 1 -- 172.21.15.81:0/4119025079 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f940a2530 msgr2=0x7f8f940a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:29:56.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:56.666+0000 7f8faaf9b700 1 --2- 172.21.15.81:0/4119025079 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f940a2530 0x7f8f940a49f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8f98000c00 tx=0x7f8f98005fb0 comp rx=0 tx=0).stop 2024-04-01T01:29:56.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:56.666+0000 7f8faaf9b700 1 -- 172.21.15.81:0/4119025079 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fa4103e10 msgr2=0x7f8fa419f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:29:56.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:56.666+0000 7f8faaf9b700 1 --2- 172.21.15.81:0/4119025079 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fa4103e10 0x7f8fa419f930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c007d70 tx=0x7f8f9c00a340 comp rx=0 tx=0).stop 2024-04-01T01:29:56.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:56.666+0000 7f8faaf9b700 1 -- 172.21.15.81:0/4119025079 shutdown_connections 2024-04-01T01:29:56.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:56.666+0000 7f8faaf9b700 1 --2- 172.21.15.81:0/4119025079 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f940a2530 0x7f8f940a49f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:56.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:56.666+0000 7f8faaf9b700 1 --2- 172.21.15.81:0/4119025079 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fa4103e10 0x7f8fa419f930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:56.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:56.666+0000 7f8faaf9b700 1 --2- 172.21.15.81:0/4119025079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fa4102270 0x7f8fa4198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:56.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:56.666+0000 7f8faaf9b700 1 --2- 172.21.15.81:0/4119025079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fa4103470 0x7f8fa41988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:29:56.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:56.666+0000 7f8faaf9b700 1 -- 172.21.15.81:0/4119025079 >> 172.21.15.81:0/4119025079 conn(0x7f8fa40fd850 msgr2=0x7f8fa4106860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:29:56.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:56.666+0000 7f8faaf9b700 1 -- 172.21.15.81:0/4119025079 shutdown_connections 2024-04-01T01:29:56.666 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-01T01:29:56.666+0000 7f8faaf9b700 1 -- 172.21.15.81:0/4119025079 wait complete. 2024-04-01T01:29:56.990 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:56.990 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:56.990 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:56.990 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:29:56.990 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:56.990 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:56 smithi081 ceph-mon[35142]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:56.991 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:56.991 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:56.991 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:56.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:56.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:56.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:56.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:29:56.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:56.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:56 smithi189 ceph-mon[35753]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:56.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:56.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:56.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:57.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:57.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:57.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:29:57.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:29:57.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:57.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:56 smithi070 ceph-mon[32179]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:57.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:57.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:29:57.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:29:57.343 DEBUG:teuthology.orchestra.run.smithi081:osd.10> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.10.service 2024-04-01T01:29:57.346 INFO:tasks.cephadm:Deploying osd.11 on smithi189 with /dev/vg_nvme/lv_1... 2024-04-01T01:29:57.346 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-01T01:29:57.596 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-01T01:29:57.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:57 smithi081 ceph-mon[35142]: from='osd.10 [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-01T01:29:57.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:57 smithi081 ceph-mon[35142]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-01T01:29:57.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:57 smithi189 ceph-mon[35753]: from='osd.10 [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-01T01:29:57.969 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:57 smithi189 ceph-mon[35753]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-01T01:29:58.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:57 smithi070 ceph-mon[32179]: from='osd.10 [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-01T01:29:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:57 smithi070 ceph-mon[32179]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-01T01:29:58.928 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:58 smithi081 ceph-mon[35142]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-01T01:29:58.928 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:58 smithi081 ceph-mon[35142]: osdmap e70: 11 total, 10 up, 11 in 2024-04-01T01:29:58.929 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:58 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:29:58.929 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:58 smithi081 ceph-mon[35142]: from='osd.10 [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:29:58.929 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:58 smithi081 ceph-mon[35142]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:29:58.929 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:58 smithi081 ceph-mon[35142]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:58.982 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:58 smithi189 ceph-mon[35753]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-01T01:29:58.983 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:58 smithi189 ceph-mon[35753]: osdmap e70: 11 total, 10 up, 11 in 2024-04-01T01:29:58.983 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:58 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:29:58.983 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:58 smithi189 ceph-mon[35753]: from='osd.10 [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:29:58.983 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:58 smithi189 ceph-mon[35753]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:29:58.983 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:58 smithi189 ceph-mon[35753]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:59.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:58 smithi070 ceph-mon[32179]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-01T01:29:59.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:58 smithi070 ceph-mon[32179]: osdmap e70: 11 total, 10 up, 11 in 2024-04-01T01:29:59.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:58 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:29:59.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:58 smithi070 ceph-mon[32179]: from='osd.10 [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:29:59.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:58 smithi070 ceph-mon[32179]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-01T01:29:59.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:58 smithi070 ceph-mon[32179]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:29:59.697 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-01T01:29:59.697 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-01T01:29:59.697 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-04-01T01:29:59.697 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-04-01T01:29:59.698 INFO:teuthology.orchestra.run.smithi189.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115595 s, 907 MB/s 2024-04-01T01:29:59.698 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-04-01T01:29:59.747 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:29:59 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:29:59.443+0000 7fa9e06b7700 -1 osd.10 0 waiting for initial osdmap 2024-04-01T01:29:59.747 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:29:59 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:29:59.455+0000 7fa9d84a2700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-01T01:29:59.765 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:59 smithi189 ceph-mon[35753]: purged_snaps scrub starts 2024-04-01T01:29:59.765 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:59 smithi189 ceph-mon[35753]: purged_snaps scrub ok 2024-04-01T01:29:59.765 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:59 smithi189 ceph-mon[35753]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-01T01:29:59.765 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:59 smithi189 ceph-mon[35753]: osdmap e71: 11 total, 10 up, 11 in 2024-04-01T01:29:59.765 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:59 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:29:59.766 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:59 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:29:59.766 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:29:59 smithi189 ceph-mon[35753]: from='osd.10 ' entity='osd.10' 2024-04-01T01:30:00.075 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:59 smithi081 ceph-mon[35142]: purged_snaps scrub starts 2024-04-01T01:30:00.075 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:59 smithi081 ceph-mon[35142]: purged_snaps scrub ok 2024-04-01T01:30:00.075 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:59 smithi081 ceph-mon[35142]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-01T01:30:00.075 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:59 smithi081 ceph-mon[35142]: osdmap e71: 11 total, 10 up, 11 in 2024-04-01T01:30:00.075 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:59 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:30:00.075 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:59 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:30:00.075 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:29:59 smithi081 ceph-mon[35142]: from='osd.10 ' entity='osd.10' 2024-04-01T01:30:00.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:59 smithi070 ceph-mon[32179]: purged_snaps scrub starts 2024-04-01T01:30:00.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:59 smithi070 ceph-mon[32179]: purged_snaps scrub ok 2024-04-01T01:30:00.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:59 smithi070 ceph-mon[32179]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-01T01:30:00.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:59 smithi070 ceph-mon[32179]: osdmap e71: 11 total, 10 up, 11 in 2024-04-01T01:30:00.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:59 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:30:00.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:59 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:30:00.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:29:59 smithi070 ceph-mon[32179]: from='osd.10 ' entity='osd.10' 2024-04-01T01:30:00.444 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi189:vg_nvme/lv_1 2024-04-01T01:30:00.688 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-01T01:30:01.036 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:30:01.036 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: osd.10 [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251] boot 2024-04-01T01:30:01.036 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: osdmap e72: 11 total, 11 up, 11 in 2024-04-01T01:30:01.036 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:30:01.036 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: overall HEALTH_OK 2024-04-01T01:30:01.037 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: Detected new or changed devices on smithi081 2024-04-01T01:30:01.037 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:01.037 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:01.037 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:01.037 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:01.037 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:01.037 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:01.037 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi081 to 4272M 2024-04-01T01:30:01.037 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:01.037 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:01.037 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:30:01.037 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:01.037 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: osdmap e73: 11 total, 11 up, 11 in 2024-04-01T01:30:01.038 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:00 smithi189 ceph-mon[35753]: pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:30:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:30:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: osd.10 [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251] boot 2024-04-01T01:30:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: osdmap e72: 11 total, 11 up, 11 in 2024-04-01T01:30:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:30:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: overall HEALTH_OK 2024-04-01T01:30:01.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: Detected new or changed devices on smithi081 2024-04-01T01:30:01.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:01.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:01.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:01.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:01.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:01.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:01.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi081 to 4272M 2024-04-01T01:30:01.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:01.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:01.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:30:01.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:01.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: osdmap e73: 11 total, 11 up, 11 in 2024-04-01T01:30:01.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:00 smithi081 ceph-mon[35142]: pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:30:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:30:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: osd.10 [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251] boot 2024-04-01T01:30:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: osdmap e72: 11 total, 11 up, 11 in 2024-04-01T01:30:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-01T01:30:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: overall HEALTH_OK 2024-04-01T01:30:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: Detected new or changed devices on smithi081 2024-04-01T01:30:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:01.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:01.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:01.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:01.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:01.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:01.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi081 to 4272M 2024-04-01T01:30:01.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:01.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:01.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:30:01.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:01.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: osdmap e73: 11 total, 11 up, 11 in 2024-04-01T01:30:01.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:00 smithi070 ceph-mon[32179]: pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 281 MiB used, 894 GiB / 894 GiB avail 2024-04-01T01:30:02.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.297+0000 7f1cdc26b700 1 -- 172.21.15.189:0/4236552978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cd4103e40 msgr2=0x7f1cd410a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:02.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.297+0000 7f1cdc26b700 1 --2- 172.21.15.189:0/4236552978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cd4103e40 0x7f1cd410a6f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc00b0a0 tx=0x7f1ccc00b3b0 comp rx=0 tx=0).stop 2024-04-01T01:30:02.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.297+0000 7f1cdc26b700 1 -- 172.21.15.189:0/4236552978 shutdown_connections 2024-04-01T01:30:02.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.297+0000 7f1cdc26b700 1 --2- 172.21.15.189:0/4236552978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cd4103e40 0x7f1cd410a6f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:02.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.297+0000 7f1cdc26b700 1 --2- 172.21.15.189:0/4236552978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cd41022a0 0x7f1cd41026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:02.297 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.297+0000 7f1cdc26b700 1 --2- 172.21.15.189:0/4236552978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd41034a0 0x7f1cd4103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:02.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.297+0000 7f1cdc26b700 1 -- 172.21.15.189:0/4236552978 >> 172.21.15.189:0/4236552978 conn(0x7f1cd40fd820 msgr2=0x7f1cd40ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:02.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.297+0000 7f1cdc26b700 1 -- 172.21.15.189:0/4236552978 shutdown_connections 2024-04-01T01:30:02.298 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.297+0000 7f1cdc26b700 1 -- 172.21.15.189:0/4236552978 wait complete. 2024-04-01T01:30:02.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cdc26b700 1 Processor -- start 2024-04-01T01:30:02.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cdc26b700 1 -- start start 2024-04-01T01:30:02.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cdc26b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cd41022a0 0x7f1cd4198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:02.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cdc26b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd41034a0 0x7f1cd41988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:02.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cdc26b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cd4103e40 0x7f1cd419f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:02.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cdc26b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd410dd90 con 0x7f1cd41022a0 2024-04-01T01:30:02.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cdc26b700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd410dc10 con 0x7f1cd41034a0 2024-04-01T01:30:02.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cdc26b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd410df10 con 0x7f1cd4103e40 2024-04-01T01:30:02.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cda808700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cd4103e40 0x7f1cd419f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:02.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cda808700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cd4103e40 0x7f1cd419f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58778/0 (socket says 172.21.15.189:58778) 2024-04-01T01:30:02.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cda808700 1 -- 172.21.15.189:0/3969263533 learned_addr learned my addr 172.21.15.189:0/3969263533 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-01T01:30:02.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cda007700 1 --2- 172.21.15.189:0/3969263533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cd41022a0 0x7f1cd4198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:02.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cd9806700 1 --2- 172.21.15.189:0/3969263533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd41034a0 0x7f1cd41988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:02.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cda808700 1 -- 172.21.15.189:0/3969263533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd41034a0 msgr2=0x7f1cd41988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:02.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cda808700 1 --2- 172.21.15.189:0/3969263533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd41034a0 0x7f1cd41988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:02.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cda808700 1 -- 172.21.15.189:0/3969263533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cd41022a0 msgr2=0x7f1cd4198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:02.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cda808700 1 --2- 172.21.15.189:0/3969263533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cd41022a0 0x7f1cd4198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:02.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cda808700 1 -- 172.21.15.189:0/3969263533 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd419ffd0 con 0x7f1cd4103e40 2024-04-01T01:30:02.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.298+0000 7f1cd9806700 1 --2- 172.21.15.189:0/3969263533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd41034a0 0x7f1cd41988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:30:02.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.299+0000 7f1cda808700 1 --2- 172.21.15.189:0/3969263533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cd4103e40 0x7f1cd419f930 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc004e40 tx=0x7f1ccc004e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:02.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.299+0000 7f1ccb7fe700 1 -- 172.21.15.189:0/3969263533 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc021070 con 0x7f1cd4103e40 2024-04-01T01:30:02.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.299+0000 7f1cdc26b700 1 -- 172.21.15.189:0/3969263533 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cd41a0260 con 0x7f1cd4103e40 2024-04-01T01:30:02.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.299+0000 7f1ccb7fe700 1 -- 172.21.15.189:0/3969263533 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ccc007c80 con 0x7f1cd4103e40 2024-04-01T01:30:02.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.299+0000 7f1ccb7fe700 1 -- 172.21.15.189:0/3969263533 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc004820 con 0x7f1cd4103e40 2024-04-01T01:30:02.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.299+0000 7f1cdc26b700 1 -- 172.21.15.189:0/3969263533 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cd41a0790 con 0x7f1cd4103e40 2024-04-01T01:30:02.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.300+0000 7f1cdc26b700 1 -- 172.21.15.189:0/3969263533 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cd4068f00 con 0x7f1cd4103e40 2024-04-01T01:30:02.302 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.301+0000 7f1ccb7fe700 1 -- 172.21.15.189:0/3969263533 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f1ccc004000 con 0x7f1cd4103e40 2024-04-01T01:30:02.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.301+0000 7f1ccb7fe700 1 --2- 172.21.15.189:0/3969263533 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1cc00a2530 0x7f1cc00a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:02.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.302+0000 7f1ccb7fe700 1 -- 172.21.15.189:0/3969263533 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7f1ccc00f020 con 0x7f1cd4103e40 2024-04-01T01:30:02.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.302+0000 7f1cda007700 1 --2- 172.21.15.189:0/3969263533 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1cc00a2530 0x7f1cc00a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:02.303 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.302+0000 7f1cda007700 1 --2- 172.21.15.189:0/3969263533 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1cc00a2530 0x7f1cc00a49f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4103300 tx=0x7f1cd000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:02.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.305+0000 7f1ccb7fe700 1 -- 172.21.15.189:0/3969263533 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1ccc02a020 con 0x7f1cd4103e40 2024-04-01T01:30:02.508 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:02.508+0000 7f1cdc26b700 1 -- 172.21.15.189:0/3969263533 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f1cd4062680 con 0x7f1cc00a2530 2024-04-01T01:30:02.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:02 smithi081 ceph-mon[35142]: osdmap e74: 11 total, 11 up, 11 in 2024-04-01T01:30:02.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:02 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:30:02.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:02 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:30:02.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:02 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:02.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:02 smithi189 ceph-mon[35753]: osdmap e74: 11 total, 11 up, 11 in 2024-04-01T01:30:02.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:02 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:30:02.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:02 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:30:02.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:02 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:03.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:02 smithi070 ceph-mon[32179]: osdmap e74: 11 total, 11 up, 11 in 2024-04-01T01:30:03.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:02 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-01T01:30:03.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:02 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-01T01:30:03.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:02 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:03.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:03 smithi189 ceph-mon[35753]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:30:03.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:03 smithi189 ceph-mon[35753]: osdmap e75: 11 total, 11 up, 11 in 2024-04-01T01:30:03.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:03 smithi189 ceph-mon[35753]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:03.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:03 smithi081 ceph-mon[35142]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:30:03.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:03 smithi081 ceph-mon[35142]: osdmap e75: 11 total, 11 up, 11 in 2024-04-01T01:30:03.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:03 smithi081 ceph-mon[35142]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:04.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:03 smithi070 ceph-mon[32179]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:30:04.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:03 smithi070 ceph-mon[32179]: osdmap e75: 11 total, 11 up, 11 in 2024-04-01T01:30:04.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:03 smithi070 ceph-mon[32179]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:04.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:04 smithi189 ceph-mon[35753]: osdmap e76: 11 total, 11 up, 11 in 2024-04-01T01:30:04.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:04 smithi081 ceph-mon[35142]: osdmap e76: 11 total, 11 up, 11 in 2024-04-01T01:30:05.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:04 smithi070 ceph-mon[32179]: osdmap e76: 11 total, 11 up, 11 in 2024-04-01T01:30:05.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:05 smithi081 ceph-mon[35142]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:05.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:05 smithi081 ceph-mon[35142]: from='client.? 172.21.15.189:0/2510378283' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "365b3275-80d3-4888-8a4e-e00b571599ca"}]: dispatch 2024-04-01T01:30:05.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:05 smithi081 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "365b3275-80d3-4888-8a4e-e00b571599ca"}]: dispatch 2024-04-01T01:30:05.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:05 smithi081 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "365b3275-80d3-4888-8a4e-e00b571599ca"}]': finished 2024-04-01T01:30:05.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:05 smithi081 ceph-mon[35142]: osdmap e77: 12 total, 11 up, 12 in 2024-04-01T01:30:05.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:05.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:05 smithi189 ceph-mon[35753]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:05.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:05 smithi189 ceph-mon[35753]: from='client.? 172.21.15.189:0/2510378283' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "365b3275-80d3-4888-8a4e-e00b571599ca"}]: dispatch 2024-04-01T01:30:05.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:05 smithi189 ceph-mon[35753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "365b3275-80d3-4888-8a4e-e00b571599ca"}]: dispatch 2024-04-01T01:30:05.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:05 smithi189 ceph-mon[35753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "365b3275-80d3-4888-8a4e-e00b571599ca"}]': finished 2024-04-01T01:30:05.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:05 smithi189 ceph-mon[35753]: osdmap e77: 12 total, 11 up, 12 in 2024-04-01T01:30:05.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:06.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:05 smithi070 ceph-mon[32179]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:06.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:05 smithi070 ceph-mon[32179]: from='client.? 172.21.15.189:0/2510378283' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "365b3275-80d3-4888-8a4e-e00b571599ca"}]: dispatch 2024-04-01T01:30:06.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:05 smithi070 ceph-mon[32179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "365b3275-80d3-4888-8a4e-e00b571599ca"}]: dispatch 2024-04-01T01:30:06.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:05 smithi070 ceph-mon[32179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "365b3275-80d3-4888-8a4e-e00b571599ca"}]': finished 2024-04-01T01:30:06.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:05 smithi070 ceph-mon[32179]: osdmap e77: 12 total, 11 up, 12 in 2024-04-01T01:30:06.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:06.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:06 smithi081 ceph-mon[35142]: from='client.? 172.21.15.189:0/2795292862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:30:06.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:06 smithi189 ceph-mon[35753]: from='client.? 172.21.15.189:0/2795292862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:30:07.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:06 smithi070 ceph-mon[32179]: from='client.? 172.21.15.189:0/2795292862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-01T01:30:07.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:07 smithi081 ceph-mon[35142]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-01T01:30:07.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:07 smithi189 ceph-mon[35753]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-01T01:30:08.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:07 smithi070 ceph-mon[32179]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-01T01:30:09.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:09 smithi081 ceph-mon[35142]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-01T01:30:09.977 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:09 smithi189 ceph-mon[35753]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-01T01:30:10.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:09 smithi070 ceph-mon[32179]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-01T01:30:11.715 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:11 smithi189 ceph-mon[35753]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-01T01:30:11.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:11 smithi081 ceph-mon[35142]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-01T01:30:11.905 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:11 smithi070 ceph-mon[32179]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-01T01:30:13.641 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:13 smithi189 ceph-mon[35753]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-01T01:30:13.641 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:13 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-01T01:30:13.641 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:13 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:13.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:13 smithi081 ceph-mon[35142]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-01T01:30:13.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-01T01:30:13.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:14.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:13 smithi070 ceph-mon[32179]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-01T01:30:14.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:13 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-01T01:30:14.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:13 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:14.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:14 smithi189 ceph-mon[35753]: Deploying daemon osd.11 on smithi189 2024-04-01T01:30:14.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:14 smithi081 ceph-mon[35142]: Deploying daemon osd.11 on smithi189 2024-04-01T01:30:15.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:14 smithi070 ceph-mon[32179]: Deploying daemon osd.11 on smithi189 2024-04-01T01:30:15.630 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:15 smithi189 ceph-mon[35753]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-01T01:30:15.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:15 smithi081 ceph-mon[35142]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-01T01:30:16.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:15 smithi070 ceph-mon[32179]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-01T01:30:17.732 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:17 smithi189 ceph-mon[35753]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:17.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:17 smithi081 ceph-mon[35142]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:18.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:17 smithi070 ceph-mon[32179]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:19.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:19.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:19.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:30:19.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:19 smithi189 ceph-mon[35753]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:19.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:19.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:19.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:30:19.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:19 smithi081 ceph-mon[35142]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:19.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:19.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:19.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:30:19.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:19 smithi070 ceph-mon[32179]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:21.715 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:21 smithi189 ceph-mon[35753]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:21.773 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 11 on host 'smithi189' 2024-04-01T01:30:21.773 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:21.773+0000 7f1ccb7fe700 1 -- 172.21.15.189:0/3969263533 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f1cd4062680 con 0x7f1cc00a2530 2024-04-01T01:30:21.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:21.775+0000 7f1cdc26b700 1 -- 172.21.15.189:0/3969263533 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1cc00a2530 msgr2=0x7f1cc00a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:21.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:21.775+0000 7f1cdc26b700 1 --2- 172.21.15.189:0/3969263533 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1cc00a2530 0x7f1cc00a49f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4103300 tx=0x7f1cd000b410 comp rx=0 tx=0).stop 2024-04-01T01:30:21.775 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:21.775+0000 7f1cdc26b700 1 -- 172.21.15.189:0/3969263533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cd4103e40 msgr2=0x7f1cd419f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:21.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:21.775+0000 7f1cdc26b700 1 --2- 172.21.15.189:0/3969263533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cd4103e40 0x7f1cd419f930 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc004e40 tx=0x7f1ccc004e70 comp rx=0 tx=0).stop 2024-04-01T01:30:21.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:21.775+0000 7f1cdc26b700 1 -- 172.21.15.189:0/3969263533 shutdown_connections 2024-04-01T01:30:21.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:21.775+0000 7f1cdc26b700 1 --2- 172.21.15.189:0/3969263533 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1cc00a2530 0x7f1cc00a49f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:21.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:21.775+0000 7f1cdc26b700 1 --2- 172.21.15.189:0/3969263533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cd4103e40 0x7f1cd419f930 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:21.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:21.775+0000 7f1cdc26b700 1 --2- 172.21.15.189:0/3969263533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1cd41022a0 0x7f1cd4198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:21.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:21.775+0000 7f1cdc26b700 1 --2- 172.21.15.189:0/3969263533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd41034a0 0x7f1cd41988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:21.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:21.775+0000 7f1cdc26b700 1 -- 172.21.15.189:0/3969263533 >> 172.21.15.189:0/3969263533 conn(0x7f1cd40fd820 msgr2=0x7f1cd41067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:21.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:21.775+0000 7f1cdc26b700 1 -- 172.21.15.189:0/3969263533 shutdown_connections 2024-04-01T01:30:21.776 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:21.775+0000 7f1cdc26b700 1 -- 172.21.15.189:0/3969263533 wait complete. 2024-04-01T01:30:22.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:21 smithi070 ceph-mon[32179]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:22.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:21 smithi081 ceph-mon[35142]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:22.379 DEBUG:teuthology.orchestra.run.smithi189:osd.11> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.11.service 2024-04-01T01:30:22.382 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-01T01:30:22.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-04-01T01:30:22.635 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:30:22.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:22.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:22.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:22 smithi189 ceph-mon[35753]: from='osd.11 [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-01T01:30:22.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:22 smithi189 ceph-mon[35753]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-01T01:30:22.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:22 smithi189 ceph-mon[35753]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:22.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:22.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:23.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:23.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:23.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:22 smithi070 ceph-mon[32179]: from='osd.11 [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-01T01:30:23.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:22 smithi070 ceph-mon[32179]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-01T01:30:23.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:22 smithi070 ceph-mon[32179]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:23.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:23.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:22 smithi081 ceph-mon[35142]: from='osd.11 [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-01T01:30:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:22 smithi081 ceph-mon[35142]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-01T01:30:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:22 smithi081 ceph-mon[35142]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:23.907 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:23 smithi070 ceph-mon[32179]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-01T01:30:23.907 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:23 smithi070 ceph-mon[32179]: osdmap e78: 12 total, 11 up, 12 in 2024-04-01T01:30:23.907 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:23 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:23.907 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:23 smithi070 ceph-mon[32179]: from='osd.11 [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:30:23.907 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:23 smithi070 ceph-mon[32179]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:30:23.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:23 smithi189 ceph-mon[35753]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-01T01:30:23.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:23 smithi189 ceph-mon[35753]: osdmap e78: 12 total, 11 up, 12 in 2024-04-01T01:30:23.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:23 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:23.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:23 smithi189 ceph-mon[35753]: from='osd.11 [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:30:23.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:23 smithi189 ceph-mon[35753]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:30:24.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:23 smithi081 ceph-mon[35142]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-01T01:30:24.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:23 smithi081 ceph-mon[35142]: osdmap e78: 12 total, 11 up, 12 in 2024-04-01T01:30:24.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:23 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:24.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:23 smithi081 ceph-mon[35142]: from='osd.11 [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:30:24.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:23 smithi081 ceph-mon[35142]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-01T01:30:24.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.402+0000 7fb37e216700 1 -- 172.21.15.70:0/1317566926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb378101170 msgr2=0x7fb378101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:24.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.402+0000 7fb37e216700 1 --2- 172.21.15.70:0/1317566926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb378101170 0x7fb378101590 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb3680099c0 tx=0x7fb368009cd0 comp rx=0 tx=0).stop 2024-04-01T01:30:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.402+0000 7fb37e216700 1 -- 172.21.15.70:0/1317566926 shutdown_connections 2024-04-01T01:30:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.402+0000 7fb37e216700 1 --2- 172.21.15.70:0/1317566926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb378102d10 0x7fb3781095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.402+0000 7fb37e216700 1 --2- 172.21.15.70:0/1317566926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb378101170 0x7fb378101590 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.402+0000 7fb37e216700 1 --2- 172.21.15.70:0/1317566926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb378102370 0x7fb3781027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.402+0000 7fb37e216700 1 -- 172.21.15.70:0/1317566926 >> 172.21.15.70:0/1317566926 conn(0x7fb3780fc710 msgr2=0x7fb3780feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.402+0000 7fb37e216700 1 -- 172.21.15.70:0/1317566926 shutdown_connections 2024-04-01T01:30:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.402+0000 7fb37e216700 1 -- 172.21.15.70:0/1317566926 wait complete. 2024-04-01T01:30:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.403+0000 7fb37e216700 1 Processor -- start 2024-04-01T01:30:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.403+0000 7fb37e216700 1 -- start start 2024-04-01T01:30:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.403+0000 7fb37e216700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb378101170 0x7fb378197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.403+0000 7fb37e216700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb378102370 0x7fb3781978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.403+0000 7fb37e216700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb378102d10 0x7fb37819e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.403+0000 7fb37e216700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb37810cc80 con 0x7fb378101170 2024-04-01T01:30:24.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.403+0000 7fb37e216700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb37810cb00 con 0x7fb378102370 2024-04-01T01:30:24.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.403+0000 7fb37e216700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb37810ce00 con 0x7fb378102d10 2024-04-01T01:30:24.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.403+0000 7fb3777fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb378101170 0x7fb378197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:24.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.403+0000 7fb3777fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb378101170 0x7fb378197380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56674/0 (socket says 172.21.15.70:56674) 2024-04-01T01:30:24.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.403+0000 7fb3777fe700 1 -- 172.21.15.70:0/263150203 learned_addr learned my addr 172.21.15.70:0/263150203 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:30:24.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.403+0000 7fb377fff700 1 --2- 172.21.15.70:0/263150203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb378102d10 0x7fb37819e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:24.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.403+0000 7fb36ffff700 1 --2- 172.21.15.70:0/263150203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb378102370 0x7fb3781978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:24.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.404+0000 7fb3777fe700 1 -- 172.21.15.70:0/263150203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb378102d10 msgr2=0x7fb37819e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:24.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.404+0000 7fb3777fe700 1 --2- 172.21.15.70:0/263150203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb378102d10 0x7fb37819e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:24.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.404+0000 7fb3777fe700 1 -- 172.21.15.70:0/263150203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb378102370 msgr2=0x7fb3781978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:24.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.404+0000 7fb3777fe700 1 --2- 172.21.15.70:0/263150203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb378102370 0x7fb3781978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:24.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.404+0000 7fb3777fe700 1 -- 172.21.15.70:0/263150203 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb37819efe0 con 0x7fb378101170 2024-04-01T01:30:24.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.404+0000 7fb36ffff700 1 --2- 172.21.15.70:0/263150203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb378102370 0x7fb3781978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:30:24.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.404+0000 7fb3777fe700 1 --2- 172.21.15.70:0/263150203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb378101170 0x7fb378197380 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb368000c00 tx=0x7fb3680057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:24.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.404+0000 7fb3757fa700 1 -- 172.21.15.70:0/263150203 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb368029070 con 0x7fb378101170 2024-04-01T01:30:24.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.404+0000 7fb3757fa700 1 -- 172.21.15.70:0/263150203 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb368019cc0 con 0x7fb378101170 2024-04-01T01:30:24.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.404+0000 7fb37e216700 1 -- 172.21.15.70:0/263150203 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb378069ef0 con 0x7fb378101170 2024-04-01T01:30:24.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.404+0000 7fb3757fa700 1 -- 172.21.15.70:0/263150203 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb368024470 con 0x7fb378101170 2024-04-01T01:30:24.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.404+0000 7fb37e216700 1 -- 172.21.15.70:0/263150203 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb37806a450 con 0x7fb378101170 2024-04-01T01:30:24.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.405+0000 7fb37e216700 1 -- 172.21.15.70:0/263150203 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb378068f00 con 0x7fb378101170 2024-04-01T01:30:24.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.407+0000 7fb3757fa700 1 -- 172.21.15.70:0/263150203 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fb36802c070 con 0x7fb378101170 2024-04-01T01:30:24.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.407+0000 7fb3757fa700 1 --2- 172.21.15.70:0/263150203 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3640a2530 0x7fb3640a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:24.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.407+0000 7fb36ffff700 1 --2- 172.21.15.70:0/263150203 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3640a2530 0x7fb3640a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:24.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.407+0000 7fb3757fa700 1 -- 172.21.15.70:0/263150203 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7fb3680d1620 con 0x7fb378101170 2024-04-01T01:30:24.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.408+0000 7fb36ffff700 1 --2- 172.21.15.70:0/263150203 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3640a2530 0x7fb3640a49f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb360005fd0 tx=0x7fb360005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:24.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.411+0000 7fb3757fa700 1 -- 172.21.15.70:0/263150203 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb368033080 con 0x7fb378101170 2024-04-01T01:30:24.566 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:30:24 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:30:24.485+0000 7f8d4ab7d700 -1 osd.11 0 waiting for initial osdmap 2024-04-01T01:30:24.566 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:30:24 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:30:24.499+0000 7f8d42968700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-01T01:30:24.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.611+0000 7fb37e216700 1 -- 172.21.15.70:0/263150203 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fb378108080 con 0x7fb378101170 2024-04-01T01:30:24.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.612+0000 7fb3757fa700 1 -- 172.21.15.70:0/263150203 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) ==== 74+0+133 (secure 0 0 0) 0x7fb378108080 con 0x7fb378101170 2024-04-01T01:30:24.612 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:30:24.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.614+0000 7fb37e216700 1 -- 172.21.15.70:0/263150203 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3640a2530 msgr2=0x7fb3640a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:24.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.614+0000 7fb37e216700 1 --2- 172.21.15.70:0/263150203 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3640a2530 0x7fb3640a49f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb360005fd0 tx=0x7fb360005ee0 comp rx=0 tx=0).stop 2024-04-01T01:30:24.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.614+0000 7fb37e216700 1 -- 172.21.15.70:0/263150203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb378101170 msgr2=0x7fb378197380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:24.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.614+0000 7fb37e216700 1 --2- 172.21.15.70:0/263150203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb378101170 0x7fb378197380 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb368000c00 tx=0x7fb3680057c0 comp rx=0 tx=0).stop 2024-04-01T01:30:24.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.614+0000 7fb37e216700 1 -- 172.21.15.70:0/263150203 shutdown_connections 2024-04-01T01:30:24.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.614+0000 7fb37e216700 1 --2- 172.21.15.70:0/263150203 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3640a2530 0x7fb3640a49f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:24.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.614+0000 7fb37e216700 1 --2- 172.21.15.70:0/263150203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb378102d10 0x7fb37819e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:24.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.614+0000 7fb37e216700 1 --2- 172.21.15.70:0/263150203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb378101170 0x7fb378197380 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:24.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.614+0000 7fb37e216700 1 --2- 172.21.15.70:0/263150203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb378102370 0x7fb3781978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:24.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.614+0000 7fb37e216700 1 -- 172.21.15.70:0/263150203 >> 172.21.15.70:0/263150203 conn(0x7fb3780fc710 msgr2=0x7fb3781057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:24.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.614+0000 7fb37e216700 1 -- 172.21.15.70:0/263150203 shutdown_connections 2024-04-01T01:30:24.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:24.614+0000 7fb37e216700 1 -- 172.21.15.70:0/263150203 wait complete. 2024-04-01T01:30:24.849 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:24 smithi189 ceph-mon[35753]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-01T01:30:24.849 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:24 smithi189 ceph-mon[35753]: osdmap e79: 12 total, 11 up, 12 in 2024-04-01T01:30:24.849 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:24 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:24.849 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:24 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:24.849 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:24 smithi189 ceph-mon[35753]: from='osd.11 ' entity='osd.11' 2024-04-01T01:30:24.849 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:24 smithi189 ceph-mon[35753]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:24.849 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:24 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/263150203' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-01T01:30:25.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:24 smithi081 ceph-mon[35142]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-01T01:30:25.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:24 smithi081 ceph-mon[35142]: osdmap e79: 12 total, 11 up, 12 in 2024-04-01T01:30:25.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:24 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:25.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:24 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:25.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:24 smithi081 ceph-mon[35142]: from='osd.11 ' entity='osd.11' 2024-04-01T01:30:25.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:24 smithi081 ceph-mon[35142]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:25.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:24 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/263150203' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-01T01:30:25.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:24 smithi070 ceph-mon[32179]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-01T01:30:25.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:24 smithi070 ceph-mon[32179]: osdmap e79: 12 total, 11 up, 12 in 2024-04-01T01:30:25.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:24 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:25.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:24 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:25.231 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:24 smithi070 ceph-mon[32179]: from='osd.11 ' entity='osd.11' 2024-04-01T01:30:25.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:24 smithi070 ceph-mon[32179]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-04-01T01:30:25.232 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:24 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/263150203' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-01T01:30:25.257 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1711934999,"num_in_osds":12,"osd_in_since":1711935005,"num_remapped_pgs":0} 2024-04-01T01:30:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:25 smithi081 ceph-mon[35142]: purged_snaps scrub starts 2024-04-01T01:30:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:25 smithi081 ceph-mon[35142]: purged_snaps scrub ok 2024-04-01T01:30:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:25 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:25 smithi081 ceph-mon[35142]: osd.11 [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510] boot 2024-04-01T01:30:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:25 smithi081 ceph-mon[35142]: osdmap e80: 12 total, 12 up, 12 in 2024-04-01T01:30:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:25 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:26.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:25 smithi189 ceph-mon[35753]: purged_snaps scrub starts 2024-04-01T01:30:26.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:25 smithi189 ceph-mon[35753]: purged_snaps scrub ok 2024-04-01T01:30:26.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:25 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:26.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:25 smithi189 ceph-mon[35753]: osd.11 [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510] boot 2024-04-01T01:30:26.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:25 smithi189 ceph-mon[35753]: osdmap e80: 12 total, 12 up, 12 in 2024-04-01T01:30:26.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:25 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:26.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:25 smithi070 ceph-mon[32179]: purged_snaps scrub starts 2024-04-01T01:30:26.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:25 smithi070 ceph-mon[32179]: purged_snaps scrub ok 2024-04-01T01:30:26.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:25 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:26.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:25 smithi070 ceph-mon[32179]: osd.11 [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510] boot 2024-04-01T01:30:26.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:25 smithi070 ceph-mon[32179]: osdmap e80: 12 total, 12 up, 12 in 2024-04-01T01:30:26.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:25 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-01T01:30:26.258 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-04-01T01:30:26.506 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:30:27.648 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: Detected new or changed devices on smithi189 2024-04-01T01:30:27.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:27.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:27.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:27.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:27.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:27.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:27.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi189 to 4272M 2024-04-01T01:30:27.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:27.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:27.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:30:27.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:27.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:30:27.649 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:27.650 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:30:27.650 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:27.650 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:27.650 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:27 smithi070 ceph-mon[32179]: osdmap e81: 12 total, 12 up, 12 in 2024-04-01T01:30:27.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: Detected new or changed devices on smithi189 2024-04-01T01:30:27.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:27.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:27.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:27.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:27.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:27.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:27.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi189 to 4272M 2024-04-01T01:30:27.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:27.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:27.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:30:27.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:27.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:30:27.750 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:27.750 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:30:27.750 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:27.750 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:27.750 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:27 smithi189 ceph-mon[35753]: osdmap e81: 12 total, 12 up, 12 in 2024-04-01T01:30:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: Detected new or changed devices on smithi189 2024-04-01T01:30:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:30:27.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi189 to 4272M 2024-04-01T01:30:27.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:27.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:27.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:30:27.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:27.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:30:27.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:27.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:30:27.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:27.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:27.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:27 smithi081 ceph-mon[35142]: osdmap e81: 12 total, 12 up, 12 in 2024-04-01T01:30:28.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.159+0000 7f64437d4700 1 -- 172.21.15.70:0/1331030333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f643c107f60 msgr2=0x7f643c10a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:28.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.159+0000 7f64437d4700 1 --2- 172.21.15.70:0/1331030333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f643c107f60 0x7f643c10a3d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6438009990 tx=0x7f6438009ca0 comp rx=0 tx=0).stop 2024-04-01T01:30:28.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.160+0000 7f64437d4700 1 -- 172.21.15.70:0/1331030333 shutdown_connections 2024-04-01T01:30:28.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.160+0000 7f64437d4700 1 --2- 172.21.15.70:0/1331030333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f643c107f60 0x7f643c10a3d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:28.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.160+0000 7f64437d4700 1 --2- 172.21.15.70:0/1331030333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f643c102d00 0x7f643c1050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:28.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.160+0000 7f64437d4700 1 --2- 172.21.15.70:0/1331030333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f643c105630 0x7f643c107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:28.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.160+0000 7f64437d4700 1 -- 172.21.15.70:0/1331030333 >> 172.21.15.70:0/1331030333 conn(0x7f643c0fc730 msgr2=0x7f643c0feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:28.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.160+0000 7f64437d4700 1 -- 172.21.15.70:0/1331030333 shutdown_connections 2024-04-01T01:30:28.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.160+0000 7f64437d4700 1 -- 172.21.15.70:0/1331030333 wait complete. 2024-04-01T01:30:28.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.160+0000 7f64437d4700 1 Processor -- start 2024-04-01T01:30:28.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.160+0000 7f64437d4700 1 -- start start 2024-04-01T01:30:28.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.160+0000 7f64437d4700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f643c102d00 0x7f643c1983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:28.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.160+0000 7f64437d4700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f643c105630 0x7f643c1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:28.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.160+0000 7f64437d4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f643c107f60 0x7f643c19f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:28.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.160+0000 7f64437d4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f643c10dd00 con 0x7f643c107f60 2024-04-01T01:30:28.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.160+0000 7f64437d4700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f643c10db80 con 0x7f643c105630 2024-04-01T01:30:28.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.160+0000 7f64437d4700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f643c10de80 con 0x7f643c102d00 2024-04-01T01:30:28.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f6441d71700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f643c107f60 0x7f643c19f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:28.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f6441d71700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f643c107f60 0x7f643c19f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60512/0 (socket says 172.21.15.70:60512) 2024-04-01T01:30:28.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f6441d71700 1 -- 172.21.15.70:0/132658977 learned_addr learned my addr 172.21.15.70:0/132658977 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:30:28.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f6440d6f700 1 --2- 172.21.15.70:0/132658977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f643c105630 0x7f643c1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:28.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f6441570700 1 --2- 172.21.15.70:0/132658977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f643c102d00 0x7f643c1983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:28.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f6441d71700 1 -- 172.21.15.70:0/132658977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f643c102d00 msgr2=0x7f643c1983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:28.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f6441d71700 1 --2- 172.21.15.70:0/132658977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f643c102d00 0x7f643c1983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:28.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f6441d71700 1 -- 172.21.15.70:0/132658977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f643c105630 msgr2=0x7f643c1988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:28.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f6441d71700 1 --2- 172.21.15.70:0/132658977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f643c105630 0x7f643c1988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:28.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f6441d71700 1 -- 172.21.15.70:0/132658977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f643c1a0000 con 0x7f643c107f60 2024-04-01T01:30:28.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f6441570700 1 --2- 172.21.15.70:0/132658977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f643c102d00 0x7f643c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:30:28.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f6441d71700 1 --2- 172.21.15.70:0/132658977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f643c107f60 0x7f643c19f960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6438006010 tx=0x7f6438009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:28.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f64327fc700 1 -- 172.21.15.70:0/132658977 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6438027070 con 0x7f643c107f60 2024-04-01T01:30:28.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f64437d4700 1 -- 172.21.15.70:0/132658977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f643c1a0290 con 0x7f643c107f60 2024-04-01T01:30:28.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f64327fc700 1 -- 172.21.15.70:0/132658977 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64380041d0 con 0x7f643c107f60 2024-04-01T01:30:28.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f64327fc700 1 -- 172.21.15.70:0/132658977 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643800f460 con 0x7f643c107f60 2024-04-01T01:30:28.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.161+0000 7f64437d4700 1 -- 172.21.15.70:0/132658977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f643c1a0690 con 0x7f643c107f60 2024-04-01T01:30:28.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.162+0000 7f64437d4700 1 -- 172.21.15.70:0/132658977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f643c068f00 con 0x7f643c107f60 2024-04-01T01:30:28.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.163+0000 7f64327fc700 1 -- 172.21.15.70:0/132658977 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f643802a070 con 0x7f643c107f60 2024-04-01T01:30:28.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.164+0000 7f64327fc700 1 --2- 172.21.15.70:0/132658977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64280a2530 0x7f64280a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:28.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.164+0000 7f6441570700 1 --2- 172.21.15.70:0/132658977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64280a2530 0x7f64280a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:28.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.164+0000 7f64327fc700 1 -- 172.21.15.70:0/132658977 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f64380cf470 con 0x7f643c107f60 2024-04-01T01:30:28.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.164+0000 7f6441570700 1 --2- 172.21.15.70:0/132658977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64280a2530 0x7f64280a49f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6434009550 tx=0x7f6434009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:28.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.167+0000 7f64327fc700 1 -- 172.21.15.70:0/132658977 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6438013070 con 0x7f643c107f60 2024-04-01T01:30:28.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.361+0000 7f64437d4700 1 -- 172.21.15.70:0/132658977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f643c062e50 con 0x7f643c107f60 2024-04-01T01:30:28.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.361+0000 7f64327fc700 1 -- 172.21.15.70:0/132658977 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f643809cb60 con 0x7f643c107f60 2024-04-01T01:30:28.362 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:30:28.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.363+0000 7f64437d4700 1 -- 172.21.15.70:0/132658977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64280a2530 msgr2=0x7f64280a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:28.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.363+0000 7f64437d4700 1 --2- 172.21.15.70:0/132658977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64280a2530 0x7f64280a49f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6434009550 tx=0x7f6434009040 comp rx=0 tx=0).stop 2024-04-01T01:30:28.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.363+0000 7f64437d4700 1 -- 172.21.15.70:0/132658977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f643c107f60 msgr2=0x7f643c19f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:28.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.363+0000 7f64437d4700 1 --2- 172.21.15.70:0/132658977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f643c107f60 0x7f643c19f960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6438006010 tx=0x7f6438009580 comp rx=0 tx=0).stop 2024-04-01T01:30:28.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.363+0000 7f64437d4700 1 -- 172.21.15.70:0/132658977 shutdown_connections 2024-04-01T01:30:28.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.363+0000 7f64437d4700 1 --2- 172.21.15.70:0/132658977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64280a2530 0x7f64280a49f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:28.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.363+0000 7f64437d4700 1 --2- 172.21.15.70:0/132658977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f643c107f60 0x7f643c19f960 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:28.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.363+0000 7f64437d4700 1 --2- 172.21.15.70:0/132658977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f643c102d00 0x7f643c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:28.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.363+0000 7f64437d4700 1 --2- 172.21.15.70:0/132658977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f643c105630 0x7f643c1988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:28.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.363+0000 7f64437d4700 1 -- 172.21.15.70:0/132658977 >> 172.21.15.70:0/132658977 conn(0x7f643c0fc730 msgr2=0x7f643c103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:28.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.364+0000 7f64437d4700 1 -- 172.21.15.70:0/132658977 shutdown_connections 2024-04-01T01:30:28.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:28.364+0000 7f64437d4700 1 -- 172.21.15.70:0/132658977 wait complete. 2024-04-01T01:30:28.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:28 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/132658977' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-01T01:30:28.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:28 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/132658977' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-01T01:30:28.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:28 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/132658977' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-01T01:30:29.008 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1711935025,"num_in_osds":12,"osd_in_since":1711935005,"num_remapped_pgs":0} 2024-04-01T01:30:29.009 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-01T01:30:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:30:29.668 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:29 smithi070 ceph-mon[32179]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:29.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:29 smithi081 ceph-mon[35142]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:29.900 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:29 smithi189 ceph-mon[35753]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:30.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.925+0000 7f9248038700 1 -- 172.21.15.70:0/3741327068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92401075f0 msgr2=0x7f924010f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:30.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.925+0000 7f9248038700 1 --2- 172.21.15.70:0/3741327068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92401075f0 0x7f924010f170 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f923800b0a0 tx=0x7f923800b3b0 comp rx=0 tx=0).stop 2024-04-01T01:30:30.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.925+0000 7f9248038700 1 -- 172.21.15.70:0/3741327068 shutdown_connections 2024-04-01T01:30:30.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.925+0000 7f9248038700 1 --2- 172.21.15.70:0/3741327068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92401075f0 0x7f924010f170 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:30.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.925+0000 7f9248038700 1 --2- 172.21.15.70:0/3741327068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9240100dd0 0x7f92401011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:30.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.925+0000 7f9248038700 1 --2- 172.21.15.70:0/3741327068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9240101780 0x7f9240106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:30.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.925+0000 7f9248038700 1 -- 172.21.15.70:0/3741327068 >> 172.21.15.70:0/3741327068 conn(0x7f9240076470 msgr2=0x7f9240076880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:30.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.925+0000 7f9248038700 1 -- 172.21.15.70:0/3741327068 shutdown_connections 2024-04-01T01:30:30.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.925+0000 7f9248038700 1 -- 172.21.15.70:0/3741327068 wait complete. 2024-04-01T01:30:30.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.925+0000 7f9248038700 1 Processor -- start 2024-04-01T01:30:30.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.925+0000 7f9248038700 1 -- start start 2024-04-01T01:30:30.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.925+0000 7f9248038700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9240100dd0 0x7f924019e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:30.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.925+0000 7f9248038700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9240101780 0x7f924019eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:30.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.925+0000 7f9248038700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92401075f0 0x7f92401986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:30.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9248038700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9240112470 con 0x7f9240100dd0 2024-04-01T01:30:30.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9248038700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f92401122f0 con 0x7f92401075f0 2024-04-01T01:30:30.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9248038700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f92401125f0 con 0x7f9240101780 2024-04-01T01:30:30.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9245dd4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9240100dd0 0x7f924019e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:30.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9245dd4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9240100dd0 0x7f924019e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60536/0 (socket says 172.21.15.70:60536) 2024-04-01T01:30:30.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9245dd4700 1 -- 172.21.15.70:0/829622381 learned_addr learned my addr 172.21.15.70:0/829622381 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:30:30.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f92465d5700 1 --2- 172.21.15.70:0/829622381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92401075f0 0x7f92401986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:30.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f92455d3700 1 --2- 172.21.15.70:0/829622381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9240101780 0x7f924019eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:30.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9245dd4700 1 -- 172.21.15.70:0/829622381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9240101780 msgr2=0x7f924019eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:30.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9245dd4700 1 --2- 172.21.15.70:0/829622381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9240101780 0x7f924019eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:30.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9245dd4700 1 -- 172.21.15.70:0/829622381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92401075f0 msgr2=0x7f92401986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:30.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9245dd4700 1 --2- 172.21.15.70:0/829622381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92401075f0 0x7f92401986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:30.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9245dd4700 1 -- 172.21.15.70:0/829622381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9240198fd0 con 0x7f9240100dd0 2024-04-01T01:30:30.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9245dd4700 1 --2- 172.21.15.70:0/829622381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9240100dd0 0x7f924019e5d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f923c00c9b0 tx=0x7f923c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:30.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9236ffd700 1 -- 172.21.15.70:0/829622381 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c007b00 con 0x7f9240100dd0 2024-04-01T01:30:30.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9236ffd700 1 -- 172.21.15.70:0/829622381 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f923c007ca0 con 0x7f9240100dd0 2024-04-01T01:30:30.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9248038700 1 -- 172.21.15.70:0/829622381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92401992c0 con 0x7f9240100dd0 2024-04-01T01:30:30.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9236ffd700 1 -- 172.21.15.70:0/829622381 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c018660 con 0x7f9240100dd0 2024-04-01T01:30:30.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.926+0000 7f9248038700 1 -- 172.21.15.70:0/829622381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92401a5550 con 0x7f9240100dd0 2024-04-01T01:30:30.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.927+0000 7f9248038700 1 -- 172.21.15.70:0/829622381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f924006a470 con 0x7f9240100dd0 2024-04-01T01:30:30.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.928+0000 7f9236ffd700 1 -- 172.21.15.70:0/829622381 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f923c00f450 con 0x7f9240100dd0 2024-04-01T01:30:30.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.929+0000 7f9236ffd700 1 --2- 172.21.15.70:0/829622381 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f922c0a25b0 0x7f922c0a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:30.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.929+0000 7f92455d3700 1 --2- 172.21.15.70:0/829622381 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f922c0a25b0 0x7f922c0a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:30.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.929+0000 7f9236ffd700 1 -- 172.21.15.70:0/829622381 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f923c0ce1f0 con 0x7f9240100dd0 2024-04-01T01:30:30.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.929+0000 7f92455d3700 1 --2- 172.21.15.70:0/829622381 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f922c0a25b0 0x7f922c0a4a70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9230006fd0 tx=0x7f9230008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:30.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:30.933+0000 7f9236ffd700 1 -- 172.21.15.70:0/829622381 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f923c014030 con 0x7f9240100dd0 2024-04-01T01:30:31.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:31.130+0000 7f9248038700 1 -- 172.21.15.70:0/829622381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f924002db30 con 0x7f9240100dd0 2024-04-01T01:30:31.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:31.131+0000 7f9236ffd700 1 -- 172.21.15.70:0/829622381 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18396 (secure 0 0 0) 0x7f923c09b850 con 0x7f9240100dd0 2024-04-01T01:30:31.130 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:30:31.130 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":81,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:30:26.779800+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:30:31.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:31.133+0000 7f9248038700 1 -- 172.21.15.70:0/829622381 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f922c0a25b0 msgr2=0x7f922c0a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:31.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:31.133+0000 7f9248038700 1 --2- 172.21.15.70:0/829622381 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f922c0a25b0 0x7f922c0a4a70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9230006fd0 tx=0x7f9230008040 comp rx=0 tx=0).stop 2024-04-01T01:30:31.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:31.133+0000 7f9248038700 1 -- 172.21.15.70:0/829622381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9240100dd0 msgr2=0x7f924019e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:31.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:31.133+0000 7f9248038700 1 --2- 172.21.15.70:0/829622381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9240100dd0 0x7f924019e5d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f923c00c9b0 tx=0x7f923c00ccc0 comp rx=0 tx=0).stop 2024-04-01T01:30:31.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:31.133+0000 7f9248038700 1 -- 172.21.15.70:0/829622381 shutdown_connections 2024-04-01T01:30:31.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:31.133+0000 7f9248038700 1 --2- 172.21.15.70:0/829622381 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f922c0a25b0 0x7f922c0a4a70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:31.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:31.133+0000 7f9248038700 1 --2- 172.21.15.70:0/829622381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92401075f0 0x7f92401986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:31.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:31.133+0000 7f9248038700 1 --2- 172.21.15.70:0/829622381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9240100dd0 0x7f924019e5d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:31.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:31.133+0000 7f9248038700 1 --2- 172.21.15.70:0/829622381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9240101780 0x7f924019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:31.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:31.133+0000 7f9248038700 1 -- 172.21.15.70:0/829622381 >> 172.21.15.70:0/829622381 conn(0x7f9240076470 msgr2=0x7f92400febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:31.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:31.133+0000 7f9248038700 1 -- 172.21.15.70:0/829622381 shutdown_connections 2024-04-01T01:30:31.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:31.133+0000 7f9248038700 1 -- 172.21.15.70:0/829622381 wait complete. 2024-04-01T01:30:31.980 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:31 smithi070 ceph-mon[32179]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:31.980 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:31 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/829622381' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:30:31.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:31 smithi189 ceph-mon[35753]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:31.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:31 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/829622381' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:30:32.013 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-01T01:26:42.809779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-01T01:30:32.014 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-04-01T01:30:32.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:31 smithi081 ceph-mon[35142]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:32.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:31 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/829622381' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:30:32.255 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:30:33.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.779+0000 7f3f3fe7a700 1 -- 172.21.15.70:0/609202763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f38101790 msgr2=0x7f3f38106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:33.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.779+0000 7f3f3fe7a700 1 --2- 172.21.15.70:0/609202763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f38101790 0x7f3f38106ed0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009990 tx=0x7f3f28009ca0 comp rx=0 tx=0).stop 2024-04-01T01:30:33.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.779+0000 7f3f3fe7a700 1 -- 172.21.15.70:0/609202763 shutdown_connections 2024-04-01T01:30:33.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.779+0000 7f3f3fe7a700 1 --2- 172.21.15.70:0/609202763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f38107600 0x7f3f3810f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:33.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.779+0000 7f3f3fe7a700 1 --2- 172.21.15.70:0/609202763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f38100de0 0x7f3f381011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:33.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.779+0000 7f3f3fe7a700 1 --2- 172.21.15.70:0/609202763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f38101790 0x7f3f38106ed0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:33.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.779+0000 7f3f3fe7a700 1 -- 172.21.15.70:0/609202763 >> 172.21.15.70:0/609202763 conn(0x7f3f38076480 msgr2=0x7f3f38076890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:33.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.779+0000 7f3f3fe7a700 1 -- 172.21.15.70:0/609202763 shutdown_connections 2024-04-01T01:30:33.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.779+0000 7f3f3fe7a700 1 -- 172.21.15.70:0/609202763 wait complete. 2024-04-01T01:30:33.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.779+0000 7f3f3fe7a700 1 Processor -- start 2024-04-01T01:30:33.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.779+0000 7f3f3fe7a700 1 -- start start 2024-04-01T01:30:33.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3fe7a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f38100de0 0x7f3f3819e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:33.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3fe7a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f38101790 0x7f3f3819eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:33.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3fe7a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f38107600 0x7f3f38198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:33.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3fe7a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3f381123e0 con 0x7f3f38100de0 2024-04-01T01:30:33.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3fe7a700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3f38112260 con 0x7f3f38101790 2024-04-01T01:30:33.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3fe7a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f38112560 con 0x7f3f38107600 2024-04-01T01:30:33.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3dc16700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f38100de0 0x7f3f3819e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:33.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3dc16700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f38100de0 0x7f3f3819e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60566/0 (socket says 172.21.15.70:60566) 2024-04-01T01:30:33.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3dc16700 1 -- 172.21.15.70:0/3189201195 learned_addr learned my addr 172.21.15.70:0/3189201195 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:30:33.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3d415700 1 --2- 172.21.15.70:0/3189201195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f38101790 0x7f3f3819eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:33.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3e417700 1 --2- 172.21.15.70:0/3189201195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f38107600 0x7f3f38198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:33.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3dc16700 1 -- 172.21.15.70:0/3189201195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f38107600 msgr2=0x7f3f38198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:33.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3dc16700 1 --2- 172.21.15.70:0/3189201195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f38107600 0x7f3f38198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:33.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3dc16700 1 -- 172.21.15.70:0/3189201195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f38101790 msgr2=0x7f3f3819eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:33.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3dc16700 1 --2- 172.21.15.70:0/3189201195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f38101790 0x7f3f3819eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:33.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3dc16700 1 -- 172.21.15.70:0/3189201195 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f38198f60 con 0x7f3f38100de0 2024-04-01T01:30:33.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3e417700 1 --2- 172.21.15.70:0/3189201195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f38107600 0x7f3f38198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:30:33.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3dc16700 1 --2- 172.21.15.70:0/3189201195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f38100de0 0x7f3f3819e560 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400c9b0 tx=0x7f3f3400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:33.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f2effd700 1 -- 172.21.15.70:0/3189201195 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34007b00 con 0x7f3f38100de0 2024-04-01T01:30:33.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f2effd700 1 -- 172.21.15.70:0/3189201195 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f34007ca0 con 0x7f3f38100de0 2024-04-01T01:30:33.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3fe7a700 1 -- 172.21.15.70:0/3189201195 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f38199250 con 0x7f3f38100de0 2024-04-01T01:30:33.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f2effd700 1 -- 172.21.15.70:0/3189201195 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34018660 con 0x7f3f38100de0 2024-04-01T01:30:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.780+0000 7f3f3fe7a700 1 -- 172.21.15.70:0/3189201195 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f381a54e0 con 0x7f3f38100de0 2024-04-01T01:30:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.781+0000 7f3f3fe7a700 1 -- 172.21.15.70:0/3189201195 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f3806a450 con 0x7f3f38100de0 2024-04-01T01:30:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.782+0000 7f3f2effd700 1 -- 172.21.15.70:0/3189201195 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f3f3400f450 con 0x7f3f38100de0 2024-04-01T01:30:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.783+0000 7f3f2effd700 1 --2- 172.21.15.70:0/3189201195 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f240a2530 0x7f3f240a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.783+0000 7f3f3d415700 1 --2- 172.21.15.70:0/3189201195 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f240a2530 0x7f3f240a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.783+0000 7f3f2effd700 1 -- 172.21.15.70:0/3189201195 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3f340ce4f0 con 0x7f3f38100de0 2024-04-01T01:30:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.783+0000 7f3f3d415700 1 --2- 172.21.15.70:0/3189201195 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f240a2530 0x7f3f240a49f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3f28006010 tx=0x7f3f28005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.787+0000 7f3f2effd700 1 -- 172.21.15.70:0/3189201195 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3f34014030 con 0x7f3f38100de0 2024-04-01T01:30:33.880 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:33 smithi081 ceph-mon[35142]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:33.977 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:33 smithi070 ceph-mon[32179]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:33.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.978+0000 7f3f3fe7a700 1 -- 172.21.15.70:0/3189201195 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3f38199b60 con 0x7f3f38100de0 2024-04-01T01:30:33.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.978+0000 7f3f2effd700 1 -- 172.21.15.70:0/3189201195 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f3f3409bbe0 con 0x7f3f38100de0 2024-04-01T01:30:33.978 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-01T01:30:33.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.980+0000 7f3f3fe7a700 1 -- 172.21.15.70:0/3189201195 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f240a2530 msgr2=0x7f3f240a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:33.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.980+0000 7f3f3fe7a700 1 --2- 172.21.15.70:0/3189201195 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f240a2530 0x7f3f240a49f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3f28006010 tx=0x7f3f28005960 comp rx=0 tx=0).stop 2024-04-01T01:30:33.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.980+0000 7f3f3fe7a700 1 -- 172.21.15.70:0/3189201195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f38100de0 msgr2=0x7f3f3819e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:33.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.980+0000 7f3f3fe7a700 1 --2- 172.21.15.70:0/3189201195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f38100de0 0x7f3f3819e560 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400c9b0 tx=0x7f3f3400ccc0 comp rx=0 tx=0).stop 2024-04-01T01:30:33.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.980+0000 7f3f3fe7a700 1 -- 172.21.15.70:0/3189201195 shutdown_connections 2024-04-01T01:30:33.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.980+0000 7f3f3fe7a700 1 --2- 172.21.15.70:0/3189201195 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f240a2530 0x7f3f240a49f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:33.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.980+0000 7f3f3fe7a700 1 --2- 172.21.15.70:0/3189201195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f38107600 0x7f3f38198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:33.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.980+0000 7f3f3fe7a700 1 --2- 172.21.15.70:0/3189201195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f38100de0 0x7f3f3819e560 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:33.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.980+0000 7f3f3fe7a700 1 --2- 172.21.15.70:0/3189201195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f38101790 0x7f3f3819eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:33.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.980+0000 7f3f3fe7a700 1 -- 172.21.15.70:0/3189201195 >> 172.21.15.70:0/3189201195 conn(0x7f3f38076480 msgr2=0x7f3f380febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:33.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.981+0000 7f3f3fe7a700 1 -- 172.21.15.70:0/3189201195 shutdown_connections 2024-04-01T01:30:33.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:33.981+0000 7f3f3fe7a700 1 -- 172.21.15.70:0/3189201195 wait complete. 2024-04-01T01:30:33.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:33 smithi189 ceph-mon[35753]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:34.711 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:34 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3189201195' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:30:34.744 INFO:tasks.cephadm:Adding mds.a on smithi070 2024-04-01T01:30:34.744 INFO:tasks.cephadm:Adding mds.d on smithi070 2024-04-01T01:30:34.744 INFO:tasks.cephadm:Adding mds.g on smithi070 2024-04-01T01:30:34.744 INFO:tasks.cephadm:Adding mds.j on smithi070 2024-04-01T01:30:34.744 INFO:tasks.cephadm:Adding mds.b on smithi081 2024-04-01T01:30:34.744 INFO:tasks.cephadm:Adding mds.e on smithi081 2024-04-01T01:30:34.744 INFO:tasks.cephadm:Adding mds.h on smithi081 2024-04-01T01:30:34.744 INFO:tasks.cephadm:Adding mds.k on smithi081 2024-04-01T01:30:34.745 INFO:tasks.cephadm:Adding mds.c on smithi189 2024-04-01T01:30:34.745 INFO:tasks.cephadm:Adding mds.f on smithi189 2024-04-01T01:30:34.745 INFO:tasks.cephadm:Adding mds.i on smithi189 2024-04-01T01:30:34.745 INFO:tasks.cephadm:Adding mds.l on smithi189 2024-04-01T01:30:34.745 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph orch apply mds all '12;smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi081=b;smithi081=e;smithi081=h;smithi081=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l' 2024-04-01T01:30:34.996 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-01T01:30:34.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:34 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3189201195' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:30:35.066 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:34 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3189201195' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:30:35.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:35 smithi189 ceph-mon[35753]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:35.967 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:35 smithi070 ceph-mon[32179]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:36.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:35 smithi081 ceph-mon[35142]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 -- 172.21.15.189:0/294569538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f364c103e30 msgr2=0x7f364c10a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 --2- 172.21.15.189:0/294569538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f364c103e30 0x7f364c10a6e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f364400b0a0 tx=0x7f364400b3b0 comp rx=0 tx=0).stop 2024-04-01T01:30:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 -- 172.21.15.189:0/294569538 shutdown_connections 2024-04-01T01:30:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 --2- 172.21.15.189:0/294569538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f364c103e30 0x7f364c10a6e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 --2- 172.21.15.189:0/294569538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f364c102290 0x7f364c1026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:36.742 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 --2- 172.21.15.189:0/294569538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f364c103490 0x7f364c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 -- 172.21.15.189:0/294569538 >> 172.21.15.189:0/294569538 conn(0x7f364c0fd7f0 msgr2=0x7f364c0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 -- 172.21.15.189:0/294569538 shutdown_connections 2024-04-01T01:30:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 -- 172.21.15.189:0/294569538 wait complete. 2024-04-01T01:30:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 Processor -- start 2024-04-01T01:30:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 -- start start 2024-04-01T01:30:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f364c102290 0x7f364c1983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f364c103490 0x7f364c1988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f364c103e30 0x7f364c19f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f364c10dda0 con 0x7f364c103490 2024-04-01T01:30:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f364c10dc20 con 0x7f364c102290 2024-04-01T01:30:36.743 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f3654210700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f364c10df20 con 0x7f364c103e30 2024-04-01T01:30:36.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.643+0000 7f36527ad700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f364c103e30 0x7f364c19f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:36.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.644+0000 7f36527ad700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f364c103e30 0x7f364c19f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:48338/0 (socket says 172.21.15.189:48338) 2024-04-01T01:30:36.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.644+0000 7f36527ad700 1 -- 172.21.15.189:0/4109696933 learned_addr learned my addr 172.21.15.189:0/4109696933 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-01T01:30:36.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.644+0000 7f36517ab700 1 --2- 172.21.15.189:0/4109696933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f364c103490 0x7f364c1988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:36.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.644+0000 7f36527ad700 1 -- 172.21.15.189:0/4109696933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f364c102290 msgr2=0x7f364c1983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:36.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.644+0000 7f3651fac700 1 --2- 172.21.15.189:0/4109696933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f364c102290 0x7f364c1983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:36.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.644+0000 7f36527ad700 1 --2- 172.21.15.189:0/4109696933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f364c102290 0x7f364c1983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:36.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.644+0000 7f36527ad700 1 -- 172.21.15.189:0/4109696933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f364c103490 msgr2=0x7f364c1988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:36.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.644+0000 7f36527ad700 1 --2- 172.21.15.189:0/4109696933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f364c103490 0x7f364c1988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:36.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.644+0000 7f36527ad700 1 -- 172.21.15.189:0/4109696933 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f364c1a0010 con 0x7f364c103e30 2024-04-01T01:30:36.744 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.644+0000 7f36517ab700 1 --2- 172.21.15.189:0/4109696933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f364c103490 0x7f364c1988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:30:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.644+0000 7f36527ad700 1 --2- 172.21.15.189:0/4109696933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f364c103e30 0x7f364c19f970 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3644004e50 tx=0x7f3644004f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.644+0000 7f3642ffd700 1 -- 172.21.15.189:0/4109696933 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3644021070 con 0x7f364c103e30 2024-04-01T01:30:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.644+0000 7f3654210700 1 -- 172.21.15.189:0/4109696933 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f364c1a02a0 con 0x7f364c103e30 2024-04-01T01:30:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.644+0000 7f3642ffd700 1 -- 172.21.15.189:0/4109696933 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f364401d900 con 0x7f364c103e30 2024-04-01T01:30:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.644+0000 7f3642ffd700 1 -- 172.21.15.189:0/4109696933 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3644005550 con 0x7f364c103e30 2024-04-01T01:30:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.644+0000 7f3654210700 1 -- 172.21.15.189:0/4109696933 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f364c1a0820 con 0x7f364c103e30 2024-04-01T01:30:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.645+0000 7f3654210700 1 -- 172.21.15.189:0/4109696933 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f364c068f00 con 0x7f364c103e30 2024-04-01T01:30:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.646+0000 7f3642ffd700 1 -- 172.21.15.189:0/4109696933 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f3644024070 con 0x7f364c103e30 2024-04-01T01:30:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.647+0000 7f3642ffd700 1 --2- 172.21.15.189:0/4109696933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36380a2530 0x7f36380a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.647+0000 7f3642ffd700 1 -- 172.21.15.189:0/4109696933 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f364400f020 con 0x7f364c103e30 2024-04-01T01:30:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.647+0000 7f3651fac700 1 --2- 172.21.15.189:0/4109696933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36380a2530 0x7f36380a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:36.745 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.647+0000 7f3651fac700 1 --2- 172.21.15.189:0/4109696933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36380a2530 0x7f36380a49f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3648006fd0 tx=0x7f3648008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:36.746 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.651+0000 7f3642ffd700 1 -- 172.21.15.189:0/4109696933 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f364402b020 con 0x7f364c103e30 2024-04-01T01:30:36.853 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.853+0000 7f3654210700 1 -- 172.21.15.189:0/4109696933 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi081=b;smithi081=e;smithi081=h;smithi081=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}) -- 0x7f364c062780 con 0x7f36380a2530 2024-04-01T01:30:36.860 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.861+0000 7f3642ffd700 1 -- 172.21.15.189:0/4109696933 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f364c062780 con 0x7f36380a2530 2024-04-01T01:30:36.860 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mds.all update... 2024-04-01T01:30:36.862 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.863+0000 7f3654210700 1 -- 172.21.15.189:0/4109696933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36380a2530 msgr2=0x7f36380a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:36.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.863+0000 7f3654210700 1 --2- 172.21.15.189:0/4109696933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36380a2530 0x7f36380a49f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3648006fd0 tx=0x7f3648008040 comp rx=0 tx=0).stop 2024-04-01T01:30:36.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.863+0000 7f3654210700 1 -- 172.21.15.189:0/4109696933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f364c103e30 msgr2=0x7f364c19f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:36.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.863+0000 7f3654210700 1 --2- 172.21.15.189:0/4109696933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f364c103e30 0x7f364c19f970 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3644004e50 tx=0x7f3644004f30 comp rx=0 tx=0).stop 2024-04-01T01:30:36.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.863+0000 7f3654210700 1 -- 172.21.15.189:0/4109696933 shutdown_connections 2024-04-01T01:30:36.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.863+0000 7f3654210700 1 --2- 172.21.15.189:0/4109696933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36380a2530 0x7f36380a49f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:36.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.863+0000 7f3654210700 1 --2- 172.21.15.189:0/4109696933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f364c103e30 0x7f364c19f970 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:36.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.863+0000 7f3654210700 1 --2- 172.21.15.189:0/4109696933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f364c102290 0x7f364c1983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:36.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.863+0000 7f3654210700 1 --2- 172.21.15.189:0/4109696933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f364c103490 0x7f364c1988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:36.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.863+0000 7f3654210700 1 -- 172.21.15.189:0/4109696933 >> 172.21.15.189:0/4109696933 conn(0x7f364c0fd7f0 msgr2=0x7f364c106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:36.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.863+0000 7f3654210700 1 -- 172.21.15.189:0/4109696933 shutdown_connections 2024-04-01T01:30:36.863 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-01T01:30:36.863+0000 7f3654210700 1 -- 172.21.15.189:0/4109696933 wait complete. 2024-04-01T01:30:37.510 DEBUG:teuthology.orchestra.run.smithi070:mds.a> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.a.service 2024-04-01T01:30:37.513 DEBUG:teuthology.orchestra.run.smithi070:mds.d> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.d.service 2024-04-01T01:30:37.556 DEBUG:teuthology.orchestra.run.smithi070:mds.g> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.g.service 2024-04-01T01:30:37.600 DEBUG:teuthology.orchestra.run.smithi070:mds.j> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.j.service 2024-04-01T01:30:37.643 DEBUG:teuthology.orchestra.run.smithi081:mds.b> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.b.service 2024-04-01T01:30:37.645 DEBUG:teuthology.orchestra.run.smithi081:mds.e> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.e.service 2024-04-01T01:30:37.688 DEBUG:teuthology.orchestra.run.smithi081:mds.h> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.h.service 2024-04-01T01:30:37.731 DEBUG:teuthology.orchestra.run.smithi081:mds.k> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.k.service 2024-04-01T01:30:37.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:37 smithi189 ceph-mon[35753]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:37.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:37 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:37.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:37 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:30:37.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:37 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:37.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:37 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:30:37.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:37 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:37.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:37 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:37.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:37 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:37.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:37 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:37.774 DEBUG:teuthology.orchestra.run.smithi189:mds.c> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.c.service 2024-04-01T01:30:37.776 DEBUG:teuthology.orchestra.run.smithi189:mds.f> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.f.service 2024-04-01T01:30:37.819 DEBUG:teuthology.orchestra.run.smithi189:mds.i> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.i.service 2024-04-01T01:30:37.862 DEBUG:teuthology.orchestra.run.smithi189:mds.l> sudo journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.l.service 2024-04-01T01:30:37.906 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-01T01:30:37.907 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:30:37.932 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:37 smithi070 ceph-mon[32179]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:37.932 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:37 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:37.932 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:37 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:30:37.932 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:37 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:37.932 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:37 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:30:37.932 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:37 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:37.932 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:37 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:37.932 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:37 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:37.932 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:37 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:38.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.022+0000 7f4eb4e47640 1 -- 172.21.15.70:0/1207450859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4eb0102210 msgr2=0x7f4eb0108aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:38.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.022+0000 7f4eb4e47640 1 --2- 172.21.15.70:0/1207450859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4eb0102210 0x7f4eb0108aa0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8009ca0 tx=0x7f4ea802c0f0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.022+0000 7f4eb4e47640 1 -- 172.21.15.70:0/1207450859 shutdown_connections 2024-04-01T01:30:38.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.022+0000 7f4eb4e47640 1 --2- 172.21.15.70:0/1207450859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4eb0102210 0x7f4eb0108aa0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.022+0000 7f4eb4e47640 1 --2- 172.21.15.70:0/1207450859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb0101850 0x7f4eb0101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.022+0000 7f4eb4e47640 1 --2- 172.21.15.70:0/1207450859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eb0100660 0x7f4eb0100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.022+0000 7f4eb4e47640 1 -- 172.21.15.70:0/1207450859 >> 172.21.15.70:0/1207450859 conn(0x7f4eb00fbbd0 msgr2=0x7f4eb00fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:38.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.022+0000 7f4eb4e47640 1 -- 172.21.15.70:0/1207450859 shutdown_connections 2024-04-01T01:30:38.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.022+0000 7f4eb4e47640 1 -- 172.21.15.70:0/1207450859 wait complete. 2024-04-01T01:30:38.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.023+0000 7f4eb4e47640 1 Processor -- start 2024-04-01T01:30:38.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.023+0000 7f4eb4e47640 1 -- start start 2024-04-01T01:30:38.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.023+0000 7f4eb4e47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eb0100660 0x7f4eb0197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:38.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.023+0000 7f4eb4e47640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4eb0101850 0x7f4eb0197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:38.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.023+0000 7f4eb4e47640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb0102210 0x7f4eb019ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:38.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.023+0000 7f4eb4e47640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb010d2b0 con 0x7f4eb0101850 2024-04-01T01:30:38.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.023+0000 7f4eb4e47640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb010d130 con 0x7f4eb0100660 2024-04-01T01:30:38.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.023+0000 7f4eb4e47640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb010d430 con 0x7f4eb0102210 2024-04-01T01:30:38.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.023+0000 7f4eadd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4eb0101850 0x7f4eb0197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:38.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.023+0000 7f4eadd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4eb0101850 0x7f4eb0197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52366/0 (socket says 172.21.15.70:52366) 2024-04-01T01:30:38.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.023+0000 7f4eadd9b640 1 -- 172.21.15.70:0/2180307517 learned_addr learned my addr 172.21.15.70:0/2180307517 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:30:38.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.023+0000 7f4eae59c640 1 --2- 172.21.15.70:0/2180307517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eb0100660 0x7f4eb0197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:38.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.023+0000 7f4eaed9d640 1 --2- 172.21.15.70:0/2180307517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb0102210 0x7f4eb019ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:38.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.024+0000 7f4eadd9b640 1 -- 172.21.15.70:0/2180307517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb0102210 msgr2=0x7f4eb019ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:38.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.024+0000 7f4eadd9b640 1 --2- 172.21.15.70:0/2180307517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb0102210 0x7f4eb019ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.024+0000 7f4eadd9b640 1 -- 172.21.15.70:0/2180307517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eb0100660 msgr2=0x7f4eb0197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:38.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.024+0000 7f4eadd9b640 1 --2- 172.21.15.70:0/2180307517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eb0100660 0x7f4eb0197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.024+0000 7f4eadd9b640 1 -- 172.21.15.70:0/2180307517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eb019f570 con 0x7f4eb0101850 2024-04-01T01:30:38.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.024+0000 7f4eadd9b640 1 --2- 172.21.15.70:0/2180307517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4eb0101850 0x7f4eb0197e90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000c970 tx=0x7f4ea000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:38.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.024+0000 7f4e9f7fe640 1 -- 172.21.15.70:0/2180307517 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0007bb0 con 0x7f4eb0101850 2024-04-01T01:30:38.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.024+0000 7f4eb4e47640 1 -- 172.21.15.70:0/2180307517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eb019f860 con 0x7f4eb0101850 2024-04-01T01:30:38.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.024+0000 7f4e9f7fe640 1 -- 172.21.15.70:0/2180307517 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ea0007d50 con 0x7f4eb0101850 2024-04-01T01:30:38.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.024+0000 7f4e9f7fe640 1 -- 172.21.15.70:0/2180307517 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea000f450 con 0x7f4eb0101850 2024-04-01T01:30:38.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.024+0000 7f4eb4e47640 1 -- 172.21.15.70:0/2180307517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eb019fdb0 con 0x7f4eb0101850 2024-04-01T01:30:38.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.025+0000 7f4eb4e47640 1 -- 172.21.15.70:0/2180307517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eb0107a50 con 0x7f4eb0101850 2024-04-01T01:30:38.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.026+0000 7f4e9f7fe640 1 -- 172.21.15.70:0/2180307517 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f4ea0004f00 con 0x7f4eb0101850 2024-04-01T01:30:38.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.027+0000 7f4e9f7fe640 1 --2- 172.21.15.70:0/2180307517 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e980a24a0 0x7f4e980a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:38.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.027+0000 7f4eae59c640 1 --2- 172.21.15.70:0/2180307517 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e980a24a0 0x7f4e980a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:38.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.027+0000 7f4e9f7fe640 1 -- 172.21.15.70:0/2180307517 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4ea00ce3d0 con 0x7f4eb0101850 2024-04-01T01:30:38.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.027+0000 7f4eae59c640 1 --2- 172.21.15.70:0/2180307517 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e980a24a0 0x7f4e980a4960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4eb01016b0 tx=0x7f4ea400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:38.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.030+0000 7f4e9f7fe640 1 -- 172.21.15.70:0/2180307517 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4ea0014030 con 0x7f4eb0101850 2024-04-01T01:30:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:37 smithi081 ceph-mon[35142]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:37 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:37 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:30:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:37 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:37 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:30:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:37 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:37 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:37 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:38.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:37 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:38.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.206+0000 7f4eb4e47640 1 -- 172.21.15.70:0/2180307517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4eb002d330 con 0x7f4eb0101850 2024-04-01T01:30:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.207+0000 7f4e9f7fe640 1 -- 172.21.15.70:0/2180307517 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18396 (secure 0 0 0) 0x7f4ea009ba30 con 0x7f4eb0101850 2024-04-01T01:30:38.206 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:30:38.206 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":81,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:30:26.779800+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:30:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.209+0000 7f4eb4e47640 1 -- 172.21.15.70:0/2180307517 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e980a24a0 msgr2=0x7f4e980a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.209+0000 7f4eb4e47640 1 --2- 172.21.15.70:0/2180307517 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e980a24a0 0x7f4e980a4960 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4eb01016b0 tx=0x7f4ea400b390 comp rx=0 tx=0).stop 2024-04-01T01:30:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.209+0000 7f4eb4e47640 1 -- 172.21.15.70:0/2180307517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4eb0101850 msgr2=0x7f4eb0197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.209+0000 7f4eb4e47640 1 --2- 172.21.15.70:0/2180307517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4eb0101850 0x7f4eb0197e90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000c970 tx=0x7f4ea000ce30 comp rx=0 tx=0).stop 2024-04-01T01:30:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.209+0000 7f4eb4e47640 1 -- 172.21.15.70:0/2180307517 shutdown_connections 2024-04-01T01:30:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.209+0000 7f4eb4e47640 1 --2- 172.21.15.70:0/2180307517 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e980a24a0 0x7f4e980a4960 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.209+0000 7f4eb4e47640 1 --2- 172.21.15.70:0/2180307517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb0102210 0x7f4eb019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.209+0000 7f4eb4e47640 1 --2- 172.21.15.70:0/2180307517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4eb0101850 0x7f4eb0197e90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.209+0000 7f4eb4e47640 1 --2- 172.21.15.70:0/2180307517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4eb0100660 0x7f4eb0197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.209+0000 7f4eb4e47640 1 -- 172.21.15.70:0/2180307517 >> 172.21.15.70:0/2180307517 conn(0x7f4eb00fbbd0 msgr2=0x7f4eb0104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.209+0000 7f4eb4e47640 1 -- 172.21.15.70:0/2180307517 shutdown_connections 2024-04-01T01:30:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.209+0000 7f4eb4e47640 1 -- 172.21.15.70:0/2180307517 wait complete. 2024-04-01T01:30:38.220 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-01T01:26:42.809779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-01T01:30:38.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-01T01:30:38.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.366+0000 7f8d5eac2640 1 -- 172.21.15.70:0/1666156784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d58107430 msgr2=0x7f8d58109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:38.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.366+0000 7f8d5eac2640 1 --2- 172.21.15.70:0/1666156784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d58107430 0x7f8d58109880 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009ca0 tx=0x7f8d5002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.366+0000 7f8d5eac2640 1 -- 172.21.15.70:0/1666156784 shutdown_connections 2024-04-01T01:30:38.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.366+0000 7f8d5eac2640 1 --2- 172.21.15.70:0/1666156784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d58107430 0x7f8d58109880 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.366+0000 7f8d5eac2640 1 --2- 172.21.15.70:0/1666156784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d58104b00 0x7f8d58106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.366+0000 7f8d5eac2640 1 --2- 172.21.15.70:0/1666156784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d581021d0 0x7f8d581045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.366+0000 7f8d5eac2640 1 -- 172.21.15.70:0/1666156784 >> 172.21.15.70:0/1666156784 conn(0x7f8d580fbbc0 msgr2=0x7f8d580fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:38.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.366+0000 7f8d5eac2640 1 -- 172.21.15.70:0/1666156784 shutdown_connections 2024-04-01T01:30:38.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.366+0000 7f8d5eac2640 1 -- 172.21.15.70:0/1666156784 wait complete. 2024-04-01T01:30:38.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5eac2640 1 Processor -- start 2024-04-01T01:30:38.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5eac2640 1 -- start start 2024-04-01T01:30:38.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5eac2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d581021d0 0x7f8d581979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:38.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5eac2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d58104b00 0x7f8d58197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:38.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5eac2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d58107430 0x7f8d5819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:38.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5eac2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5810d2f0 con 0x7f8d581021d0 2024-04-01T01:30:38.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5eac2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5810d170 con 0x7f8d58104b00 2024-04-01T01:30:38.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5eac2640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5810d470 con 0x7f8d58107430 2024-04-01T01:30:38.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5c85e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d581021d0 0x7f8d581979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:38.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5c85e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d581021d0 0x7f8d581979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52380/0 (socket says 172.21.15.70:52380) 2024-04-01T01:30:38.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5c85e640 1 -- 172.21.15.70:0/2444962787 learned_addr learned my addr 172.21.15.70:0/2444962787 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:30:38.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5d05f640 1 --2- 172.21.15.70:0/2444962787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d58107430 0x7f8d5819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d57fff640 1 --2- 172.21.15.70:0/2444962787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d58104b00 0x7f8d58197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5c85e640 1 -- 172.21.15.70:0/2444962787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d58107430 msgr2=0x7f8d5819ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5c85e640 1 --2- 172.21.15.70:0/2444962787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d58107430 0x7f8d5819ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5c85e640 1 -- 172.21.15.70:0/2444962787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d58104b00 msgr2=0x7f8d58197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5c85e640 1 --2- 172.21.15.70:0/2444962787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d58104b00 0x7f8d58197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d5c85e640 1 -- 172.21.15.70:0/2444962787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d5819f5e0 con 0x7f8d581021d0 2024-04-01T01:30:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.367+0000 7f8d57fff640 1 --2- 172.21.15.70:0/2444962787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d58104b00 0x7f8d58197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:30:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.368+0000 7f8d5c85e640 1 --2- 172.21.15.70:0/2444962787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d581021d0 0x7f8d581979c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800b4a0 tx=0x7f8d4800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.368+0000 7f8d55ffb640 1 -- 172.21.15.70:0/2444962787 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d48014070 con 0x7f8d581021d0 2024-04-01T01:30:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.368+0000 7f8d55ffb640 1 -- 172.21.15.70:0/2444962787 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d48010ac0 con 0x7f8d581021d0 2024-04-01T01:30:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.368+0000 7f8d5eac2640 1 -- 172.21.15.70:0/2444962787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d5819f8d0 con 0x7f8d581021d0 2024-04-01T01:30:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.368+0000 7f8d55ffb640 1 -- 172.21.15.70:0/2444962787 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d48017c30 con 0x7f8d581021d0 2024-04-01T01:30:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.368+0000 7f8d5eac2640 1 -- 172.21.15.70:0/2444962787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d5819fea0 con 0x7f8d581021d0 2024-04-01T01:30:38.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.369+0000 7f8d5eac2640 1 -- 172.21.15.70:0/2444962787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d580fd6b0 con 0x7f8d581021d0 2024-04-01T01:30:38.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.370+0000 7f8d55ffb640 1 -- 172.21.15.70:0/2444962787 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f8d48017070 con 0x7f8d581021d0 2024-04-01T01:30:38.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.370+0000 7f8d55ffb640 1 --2- 172.21.15.70:0/2444962787 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8d400a2520 0x7f8d400a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:38.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.370+0000 7f8d57fff640 1 --2- 172.21.15.70:0/2444962787 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8d400a2520 0x7f8d400a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:38.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.370+0000 7f8d55ffb640 1 -- 172.21.15.70:0/2444962787 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8d480cd5b0 con 0x7f8d581021d0 2024-04-01T01:30:38.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.371+0000 7f8d57fff640 1 --2- 172.21.15.70:0/2444962787 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8d400a2520 0x7f8d400a49e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8d58198ed0 tx=0x7f8d4c005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:38.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.375+0000 7f8d55ffb640 1 -- 172.21.15.70:0/2444962787 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8d4809ac10 con 0x7f8d581021d0 2024-04-01T01:30:38.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.552+0000 7f8d5eac2640 1 -- 172.21.15.70:0/2444962787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8d58062350 con 0x7f8d581021d0 2024-04-01T01:30:38.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.553+0000 7f8d55ffb640 1 -- 172.21.15.70:0/2444962787 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f8d4809a510 con 0x7f8d581021d0 2024-04-01T01:30:38.552 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-01T01:30:38.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.554+0000 7f8d5eac2640 1 -- 172.21.15.70:0/2444962787 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8d400a2520 msgr2=0x7f8d400a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:38.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.554+0000 7f8d5eac2640 1 --2- 172.21.15.70:0/2444962787 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8d400a2520 0x7f8d400a49e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8d58198ed0 tx=0x7f8d4c005d30 comp rx=0 tx=0).stop 2024-04-01T01:30:38.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.554+0000 7f8d5eac2640 1 -- 172.21.15.70:0/2444962787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d581021d0 msgr2=0x7f8d581979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:38.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.554+0000 7f8d5eac2640 1 --2- 172.21.15.70:0/2444962787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d581021d0 0x7f8d581979c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8d4800b4a0 tx=0x7f8d4800b960 comp rx=0 tx=0).stop 2024-04-01T01:30:38.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.555+0000 7f8d5eac2640 1 -- 172.21.15.70:0/2444962787 shutdown_connections 2024-04-01T01:30:38.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.555+0000 7f8d5eac2640 1 --2- 172.21.15.70:0/2444962787 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8d400a2520 0x7f8d400a49e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.555+0000 7f8d5eac2640 1 --2- 172.21.15.70:0/2444962787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d58107430 0x7f8d5819ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.555+0000 7f8d5eac2640 1 --2- 172.21.15.70:0/2444962787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d58104b00 0x7f8d58197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.555+0000 7f8d5eac2640 1 --2- 172.21.15.70:0/2444962787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d581021d0 0x7f8d581979c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.555+0000 7f8d5eac2640 1 -- 172.21.15.70:0/2444962787 >> 172.21.15.70:0/2444962787 conn(0x7f8d580fbbc0 msgr2=0x7f8d58102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:38.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.555+0000 7f8d5eac2640 1 -- 172.21.15.70:0/2444962787 shutdown_connections 2024-04-01T01:30:38.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.555+0000 7f8d5eac2640 1 -- 172.21.15.70:0/2444962787 wait complete. 2024-04-01T01:30:38.565 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:30:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.669+0000 7f21b3525640 1 -- 172.21.15.70:0/1995360312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac100670 msgr2=0x7f21ac100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.669+0000 7f21b3525640 1 --2- 172.21.15.70:0/1995360312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac100670 0x7f21ac100a90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009900 tx=0x7f21a402bc30 comp rx=0 tx=0).stop 2024-04-01T01:30:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.669+0000 7f21b3525640 1 -- 172.21.15.70:0/1995360312 shutdown_connections 2024-04-01T01:30:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.669+0000 7f21b3525640 1 --2- 172.21.15.70:0/1995360312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21ac102220 0x7f21ac108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.669+0000 7f21b3525640 1 --2- 172.21.15.70:0/1995360312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21ac101860 0x7f21ac101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.669+0000 7f21b3525640 1 --2- 172.21.15.70:0/1995360312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac100670 0x7f21ac100a90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.669+0000 7f21b3525640 1 -- 172.21.15.70:0/1995360312 >> 172.21.15.70:0/1995360312 conn(0x7f21ac0fbbe0 msgr2=0x7f21ac0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.670+0000 7f21b3525640 1 -- 172.21.15.70:0/1995360312 shutdown_connections 2024-04-01T01:30:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.670+0000 7f21b3525640 1 -- 172.21.15.70:0/1995360312 wait complete. 2024-04-01T01:30:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.670+0000 7f21b3525640 1 Processor -- start 2024-04-01T01:30:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.670+0000 7f21b3525640 1 -- start start 2024-04-01T01:30:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.670+0000 7f21b3525640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21ac100670 0x7f21ac1967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.670+0000 7f21b3525640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac101860 0x7f21ac196d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.670+0000 7f21b3525640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21ac102220 0x7f21ac19dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.670+0000 7f21b3525640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac10c190 con 0x7f21ac101860 2024-04-01T01:30:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.670+0000 7f21b3525640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac10c010 con 0x7f21ac100670 2024-04-01T01:30:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.670+0000 7f21b3525640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac10c310 con 0x7f21ac102220 2024-04-01T01:30:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.670+0000 7f21b0ac0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac101860 0x7f21ac196d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.670+0000 7f21b0ac0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac101860 0x7f21ac196d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52394/0 (socket says 172.21.15.70:52394) 2024-04-01T01:30:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.670+0000 7f21b0ac0640 1 -- 172.21.15.70:0/1906726696 learned_addr learned my addr 172.21.15.70:0/1906726696 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:30:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.670+0000 7f21b1ac2640 1 --2- 172.21.15.70:0/1906726696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21ac102220 0x7f21ac19dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.671+0000 7f21b0ac0640 1 -- 172.21.15.70:0/1906726696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21ac102220 msgr2=0x7f21ac19dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:38.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.671+0000 7f21b0ac0640 1 --2- 172.21.15.70:0/1906726696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21ac102220 0x7f21ac19dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.671+0000 7f21b12c1640 1 --2- 172.21.15.70:0/1906726696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21ac100670 0x7f21ac1967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:38.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.671+0000 7f21b0ac0640 1 -- 172.21.15.70:0/1906726696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21ac100670 msgr2=0x7f21ac1967e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:38.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.671+0000 7f21b0ac0640 1 --2- 172.21.15.70:0/1906726696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21ac100670 0x7f21ac1967e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.671+0000 7f21b0ac0640 1 -- 172.21.15.70:0/1906726696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21ac19e400 con 0x7f21ac101860 2024-04-01T01:30:38.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.671+0000 7f21b1ac2640 1 --2- 172.21.15.70:0/1906726696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21ac102220 0x7f21ac19dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:30:38.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.671+0000 7f21b0ac0640 1 --2- 172.21.15.70:0/1906726696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac101860 0x7f21ac196d20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f21a000c920 tx=0x7f21a000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:38.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.671+0000 7f21aa7fc640 1 -- 172.21.15.70:0/1906726696 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0007c70 con 0x7f21ac101860 2024-04-01T01:30:38.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.671+0000 7f21b3525640 1 -- 172.21.15.70:0/1906726696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21ac19e6f0 con 0x7f21ac101860 2024-04-01T01:30:38.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.671+0000 7f21aa7fc640 1 -- 172.21.15.70:0/1906726696 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21a0007e10 con 0x7f21ac101860 2024-04-01T01:30:38.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.671+0000 7f21aa7fc640 1 -- 172.21.15.70:0/1906726696 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0018820 con 0x7f21ac101860 2024-04-01T01:30:38.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.671+0000 7f21b3525640 1 -- 172.21.15.70:0/1906726696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21ac19ec40 con 0x7f21ac101860 2024-04-01T01:30:38.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.672+0000 7f21b3525640 1 -- 172.21.15.70:0/1906726696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21ac107990 con 0x7f21ac101860 2024-04-01T01:30:38.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.673+0000 7f21aa7fc640 1 -- 172.21.15.70:0/1906726696 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f21a00189c0 con 0x7f21ac101860 2024-04-01T01:30:38.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.673+0000 7f21aa7fc640 1 --2- 172.21.15.70:0/1906726696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21940a24a0 0x7f21940a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:38.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.673+0000 7f21b12c1640 1 --2- 172.21.15.70:0/1906726696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21940a24a0 0x7f21940a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:38.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.674+0000 7f21aa7fc640 1 -- 172.21.15.70:0/1906726696 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f21a00ce4a0 con 0x7f21ac101860 2024-04-01T01:30:38.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.674+0000 7f21b12c1640 1 --2- 172.21.15.70:0/1906726696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21940a24a0 0x7f21940a4960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f21a4036040 tx=0x7f21a4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:38.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.678+0000 7f21aa7fc640 1 -- 172.21.15.70:0/1906726696 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f21a0014030 con 0x7f21ac101860 2024-04-01T01:30:38.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:38 smithi189 ceph-mon[35753]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi081=b;smithi081=e;smithi081=h;smithi081=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:30:38.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:38 smithi189 ceph-mon[35753]: Saving service mds.all spec with placement smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi081=b;smithi081=e;smithi081=h;smithi081=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-04-01T01:30:38.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:38 smithi189 ceph-mon[35753]: Deploying daemon mds.l on smithi189 2024-04-01T01:30:38.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:38 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2180307517' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:30:38.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:38 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2444962787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:30:38.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.889+0000 7f21b3525640 1 -- 172.21.15.70:0/1906726696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21ac02d170 con 0x7f21ac101860 2024-04-01T01:30:38.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.890+0000 7f21aa7fc640 1 -- 172.21.15.70:0/1906726696 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f21a009bb00 con 0x7f21ac101860 2024-04-01T01:30:38.889 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:30:38.889 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-01T01:30:38.890 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 1 2024-04-01T01:30:38.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.892+0000 7f21b3525640 1 -- 172.21.15.70:0/1906726696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21940a24a0 msgr2=0x7f21940a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:38.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.892+0000 7f21b3525640 1 --2- 172.21.15.70:0/1906726696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21940a24a0 0x7f21940a4960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f21a4036040 tx=0x7f21a4000bc0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.892+0000 7f21b3525640 1 -- 172.21.15.70:0/1906726696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac101860 msgr2=0x7f21ac196d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:38.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.892+0000 7f21b3525640 1 --2- 172.21.15.70:0/1906726696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac101860 0x7f21ac196d20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f21a000c920 tx=0x7f21a000cde0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.892+0000 7f21b3525640 1 -- 172.21.15.70:0/1906726696 shutdown_connections 2024-04-01T01:30:38.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.892+0000 7f21b3525640 1 --2- 172.21.15.70:0/1906726696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21940a24a0 0x7f21940a4960 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.892+0000 7f21b3525640 1 --2- 172.21.15.70:0/1906726696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21ac102220 0x7f21ac19dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.892+0000 7f21b3525640 1 --2- 172.21.15.70:0/1906726696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21ac101860 0x7f21ac196d20 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.892+0000 7f21b3525640 1 --2- 172.21.15.70:0/1906726696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21ac100670 0x7f21ac1967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:38.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.892+0000 7f21b3525640 1 -- 172.21.15.70:0/1906726696 >> 172.21.15.70:0/1906726696 conn(0x7f21ac0fbbe0 msgr2=0x7f21ac104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:38.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.892+0000 7f21b3525640 1 -- 172.21.15.70:0/1906726696 shutdown_connections 2024-04-01T01:30:38.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:38.892+0000 7f21b3525640 1 -- 172.21.15.70:0/1906726696 wait complete. 2024-04-01T01:30:38.901 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:38 smithi070 ceph-mon[32179]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi081=b;smithi081=e;smithi081=h;smithi081=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:30:38.901 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:38 smithi070 ceph-mon[32179]: Saving service mds.all spec with placement smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi081=b;smithi081=e;smithi081=h;smithi081=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-04-01T01:30:38.901 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:38 smithi070 ceph-mon[32179]: Deploying daemon mds.l on smithi189 2024-04-01T01:30:38.901 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:38 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2180307517' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:30:38.901 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:38 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2444962787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:30:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:38 smithi081 ceph-mon[35142]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi081=b;smithi081=e;smithi081=h;smithi081=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:30:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:38 smithi081 ceph-mon[35142]: Saving service mds.all spec with placement smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi081=b;smithi081=e;smithi081=h;smithi081=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-04-01T01:30:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:38 smithi081 ceph-mon[35142]: Deploying daemon mds.l on smithi189 2024-04-01T01:30:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:38 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2180307517' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:30:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:38 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2444962787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:30:39.498 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 01 01:30:39 smithi189 systemd[1]: Starting Ceph mds.l for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:30:39.909 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:39 smithi189 ceph-mon[35753]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:39.910 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:39 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1906726696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:39.910 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 01 01:30:39 smithi189 podman[65715]: 2024-04-01 01:30:39.616295137 +0000 UTC m=+0.104837670 container create d300843bb0a8bd7ad27cd27ac0e7f32a5263a2ff3cf673dee48cb34e89dfa4f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-l, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-01T01:30:39.910 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 01 01:30:39 smithi189 podman[65715]: 2024-04-01 01:30:39.529401189 +0000 UTC m=+0.017943704 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:30:39.910 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 01 01:30:39 smithi189 podman[65715]: 2024-04-01 01:30:39.835683979 +0000 UTC m=+0.324226481 container init d300843bb0a8bd7ad27cd27ac0e7f32a5263a2ff3cf673dee48cb34e89dfa4f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS) 2024-04-01T01:30:39.910 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 01 01:30:39 smithi189 podman[65715]: 2024-04-01 01:30:39.838295252 +0000 UTC m=+0.326837751 container start d300843bb0a8bd7ad27cd27ac0e7f32a5263a2ff3cf673dee48cb34e89dfa4f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-l, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-01T01:30:39.910 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 01 01:30:39 smithi189 ceph-mds[65734]: -- 172.21.15.189:0/68139938 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ff9f9ada40 con 0x55ff9f980800 2024-04-01T01:30:39.910 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 01 01:30:39 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-l[65730]: starting mds.l at 2024-04-01T01:30:39.911 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 01 01:30:39 smithi189 bash[65715]: d300843bb0a8bd7ad27cd27ac0e7f32a5263a2ff3cf673dee48cb34e89dfa4f7 2024-04-01T01:30:40.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:39 smithi070 ceph-mon[32179]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:40.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:39 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1906726696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:39 smithi081 ceph-mon[35142]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:39 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1906726696' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:40.248 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 01 01:30:39 smithi189 systemd[1]: Started Ceph mds.l for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:30:40.903 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:30:41.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.016+0000 7f97defe9640 1 -- 172.21.15.70:0/1337202903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97d80ff810 msgr2=0x7f97d80ffc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:41.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.016+0000 7f97defe9640 1 --2- 172.21.15.70:0/1337202903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97d80ff810 0x7f97d80ffc90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009900 tx=0x7f97c402bc30 comp rx=0 tx=0).stop 2024-04-01T01:30:41.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.016+0000 7f97defe9640 1 -- 172.21.15.70:0/1337202903 shutdown_connections 2024-04-01T01:30:41.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.016+0000 7f97defe9640 1 --2- 172.21.15.70:0/1337202903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d81001d0 0x7f97d8109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:41.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.016+0000 7f97defe9640 1 --2- 172.21.15.70:0/1337202903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97d80ff810 0x7f97d80ffc90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:41.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.016+0000 7f97defe9640 1 --2- 172.21.15.70:0/1337202903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97d8102da0 0x7f97d81031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:41.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.016+0000 7f97defe9640 1 -- 172.21.15.70:0/1337202903 >> 172.21.15.70:0/1337202903 conn(0x7f97d80fb370 msgr2=0x7f97d80fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:41.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.016+0000 7f97defe9640 1 -- 172.21.15.70:0/1337202903 shutdown_connections 2024-04-01T01:30:41.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.016+0000 7f97defe9640 1 -- 172.21.15.70:0/1337202903 wait complete. 2024-04-01T01:30:41.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.017+0000 7f97defe9640 1 Processor -- start 2024-04-01T01:30:41.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.017+0000 7f97defe9640 1 -- start start 2024-04-01T01:30:41.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.017+0000 7f97defe9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97d80ff810 0x7f97d81979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:41.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.017+0000 7f97defe9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d81001d0 0x7f97d8197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:41.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.017+0000 7f97defe9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97d8102da0 0x7f97d819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:41.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.017+0000 7f97defe9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97d810d350 con 0x7f97d8102da0 2024-04-01T01:30:41.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.017+0000 7f97defe9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f97d810d1d0 con 0x7f97d80ff810 2024-04-01T01:30:41.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.017+0000 7f97defe9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f97d810d4d0 con 0x7f97d81001d0 2024-04-01T01:30:41.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.017+0000 7f97dd586640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97d8102da0 0x7f97d819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:41.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.017+0000 7f97dd586640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97d8102da0 0x7f97d819ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52418/0 (socket says 172.21.15.70:52418) 2024-04-01T01:30:41.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.017+0000 7f97dd586640 1 -- 172.21.15.70:0/155073047 learned_addr learned my addr 172.21.15.70:0/155073047 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:30:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.017+0000 7f97dcd85640 1 --2- 172.21.15.70:0/155073047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97d80ff810 0x7f97d81979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.017+0000 7f97d7fff640 1 --2- 172.21.15.70:0/155073047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d81001d0 0x7f97d8197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.017+0000 7f97dd586640 1 -- 172.21.15.70:0/155073047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d81001d0 msgr2=0x7f97d8197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.018+0000 7f97dd586640 1 --2- 172.21.15.70:0/155073047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d81001d0 0x7f97d8197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.018+0000 7f97dd586640 1 -- 172.21.15.70:0/155073047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97d80ff810 msgr2=0x7f97d81979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:41.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.018+0000 7f97dd586640 1 --2- 172.21.15.70:0/155073047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97d80ff810 0x7f97d81979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.018+0000 7f97dd586640 1 -- 172.21.15.70:0/155073047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d819f5e0 con 0x7f97d8102da0 2024-04-01T01:30:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.018+0000 7f97d7fff640 1 --2- 172.21.15.70:0/155073047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d81001d0 0x7f97d8197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:30:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.018+0000 7f97dd586640 1 --2- 172.21.15.70:0/155073047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97d8102da0 0x7f97d819ef70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f97d000b970 tx=0x7f97d000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.018+0000 7f97d5ffb640 1 -- 172.21.15.70:0/155073047 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0019070 con 0x7f97d8102da0 2024-04-01T01:30:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.018+0000 7f97d5ffb640 1 -- 172.21.15.70:0/155073047 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f97d0014440 con 0x7f97d8102da0 2024-04-01T01:30:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.018+0000 7f97defe9640 1 -- 172.21.15.70:0/155073047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d819f8d0 con 0x7f97d8102da0 2024-04-01T01:30:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.018+0000 7f97d5ffb640 1 -- 172.21.15.70:0/155073047 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d00075d0 con 0x7f97d8102da0 2024-04-01T01:30:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.018+0000 7f97defe9640 1 -- 172.21.15.70:0/155073047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d819fec0 con 0x7f97d8102da0 2024-04-01T01:30:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.019+0000 7f97defe9640 1 -- 172.21.15.70:0/155073047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97d8068990 con 0x7f97d8102da0 2024-04-01T01:30:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.020+0000 7f97d5ffb640 1 -- 172.21.15.70:0/155073047 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f97d001c070 con 0x7f97d8102da0 2024-04-01T01:30:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.021+0000 7f97d5ffb640 1 --2- 172.21.15.70:0/155073047 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f97c80a2520 0x7f97c80a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.021+0000 7f97dcd85640 1 --2- 172.21.15.70:0/155073047 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f97c80a2520 0x7f97c80a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:41.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.021+0000 7f97d5ffb640 1 -- 172.21.15.70:0/155073047 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f97d00cd740 con 0x7f97d8102da0 2024-04-01T01:30:41.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.021+0000 7f97dcd85640 1 --2- 172.21.15.70:0/155073047 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f97c80a2520 0x7f97c80a49e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f97d8069ae0 tx=0x7f97cc00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:41.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.025+0000 7f97d5ffb640 1 -- 172.21.15.70:0/155073047 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f97d009a750 con 0x7f97d8102da0 2024-04-01T01:30:41.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.235+0000 7f97defe9640 1 -- 172.21.15.70:0/155073047 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97d802d170 con 0x7f97d8102da0 2024-04-01T01:30:41.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.236+0000 7f97d5ffb640 1 -- 172.21.15.70:0/155073047 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f97d009e7f0 con 0x7f97d8102da0 2024-04-01T01:30:41.236 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:30:41.236 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-01T01:30:41.237 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 2 2024-04-01T01:30:41.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.238+0000 7f97defe9640 1 -- 172.21.15.70:0/155073047 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f97c80a2520 msgr2=0x7f97c80a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:41.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.238+0000 7f97defe9640 1 --2- 172.21.15.70:0/155073047 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f97c80a2520 0x7f97c80a49e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f97d8069ae0 tx=0x7f97cc00b390 comp rx=0 tx=0).stop 2024-04-01T01:30:41.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.238+0000 7f97defe9640 1 -- 172.21.15.70:0/155073047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97d8102da0 msgr2=0x7f97d819ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:41.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.238+0000 7f97defe9640 1 --2- 172.21.15.70:0/155073047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97d8102da0 0x7f97d819ef70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f97d000b970 tx=0x7f97d000be30 comp rx=0 tx=0).stop 2024-04-01T01:30:41.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.238+0000 7f97defe9640 1 -- 172.21.15.70:0/155073047 shutdown_connections 2024-04-01T01:30:41.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.238+0000 7f97defe9640 1 --2- 172.21.15.70:0/155073047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97d8102da0 0x7f97d819ef70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:41.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.238+0000 7f97defe9640 1 --2- 172.21.15.70:0/155073047 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f97c80a2520 0x7f97c80a49e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:41.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.238+0000 7f97defe9640 1 --2- 172.21.15.70:0/155073047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97d81001d0 0x7f97d8197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:41.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.238+0000 7f97defe9640 1 --2- 172.21.15.70:0/155073047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97d80ff810 0x7f97d81979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:41.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.238+0000 7f97defe9640 1 -- 172.21.15.70:0/155073047 >> 172.21.15.70:0/155073047 conn(0x7f97d80fb370 msgr2=0x7f97d8105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:41.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.238+0000 7f97defe9640 1 -- 172.21.15.70:0/155073047 shutdown_connections 2024-04-01T01:30:41.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:41.238+0000 7f97defe9640 1 -- 172.21.15.70:0/155073047 wait complete. 2024-04-01T01:30:41.248 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:40 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] up:boot 2024-04-01T01:30:41.248 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:40 smithi070 ceph-mon[32179]: fsmap 1 up:standby 2024-04-01T01:30:41.248 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:40 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-01T01:30:41.248 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:40 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:41.248 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:40 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:41.249 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:40 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:41.249 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:40 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:41.249 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:40 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:41.249 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:40 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:41.249 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:40 smithi070 ceph-mon[32179]: Deploying daemon mds.k on smithi081 2024-04-01T01:30:41.249 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:40 smithi070 ceph-mon[32179]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:41.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:40 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] up:boot 2024-04-01T01:30:41.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:40 smithi189 ceph-mon[35753]: fsmap 1 up:standby 2024-04-01T01:30:41.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:40 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-01T01:30:41.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:40 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:41.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:40 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:41.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:40 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:41.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:40 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:41.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:40 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:41.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:40 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:41.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:40 smithi189 ceph-mon[35753]: Deploying daemon mds.k on smithi081 2024-04-01T01:30:41.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:40 smithi189 ceph-mon[35753]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:41.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:40 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] up:boot 2024-04-01T01:30:41.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:40 smithi081 ceph-mon[35142]: fsmap 1 up:standby 2024-04-01T01:30:41.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:40 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-01T01:30:41.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:40 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:41.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:40 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:41.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:40 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:41.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:40 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:41.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:40 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:41.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:40 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:41.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:40 smithi081 ceph-mon[35142]: Deploying daemon mds.k on smithi081 2024-04-01T01:30:41.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:40 smithi081 ceph-mon[35142]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:42.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:41 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/155073047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:42.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:41 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/155073047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:42.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:41 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/155073047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:42.901 INFO:journalctl@ceph.mds.k.smithi081.stdout:Apr 01 01:30:42 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-k[64059]: starting mds.k at 2024-04-01T01:30:42.901 INFO:journalctl@ceph.mds.k.smithi081.stdout:Apr 01 01:30:42 smithi081 bash[64044]: 8da18107b3c1b71c55a02c33717f84324bbac0234c0d8154db317af41786202b 2024-04-01T01:30:43.252 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:30:43.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:43 smithi070 ceph-mon[32179]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:43.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:43.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:43.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:43.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:43.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:43.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:43 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:43.328 INFO:journalctl@ceph.mds.k.smithi081.stdout:Apr 01 01:30:42 smithi081 systemd[1]: Started Ceph mds.k for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:30:43.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:43 smithi081 ceph-mon[35142]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:43.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:43.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:43.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:43.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:43.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:43.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:43 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:43.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.357+0000 7fba55b48640 1 -- 172.21.15.70:0/2587201009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba50101860 msgr2=0x7fba50101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:43.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.357+0000 7fba55b48640 1 --2- 172.21.15.70:0/2587201009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba50101860 0x7fba50101ce0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fba40009900 tx=0x7fba4002bc30 comp rx=0 tx=0).stop 2024-04-01T01:30:43.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.357+0000 7fba55b48640 1 -- 172.21.15.70:0/2587201009 shutdown_connections 2024-04-01T01:30:43.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.357+0000 7fba55b48640 1 --2- 172.21.15.70:0/2587201009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba50102220 0x7fba50108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:43.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.357+0000 7fba55b48640 1 --2- 172.21.15.70:0/2587201009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba50101860 0x7fba50101ce0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:43.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.357+0000 7fba55b48640 1 --2- 172.21.15.70:0/2587201009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba50100670 0x7fba50100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:43.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.357+0000 7fba55b48640 1 -- 172.21.15.70:0/2587201009 >> 172.21.15.70:0/2587201009 conn(0x7fba500fbbe0 msgr2=0x7fba500fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:43.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.357+0000 7fba55b48640 1 -- 172.21.15.70:0/2587201009 shutdown_connections 2024-04-01T01:30:43.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.357+0000 7fba55b48640 1 -- 172.21.15.70:0/2587201009 wait complete. 2024-04-01T01:30:43.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.357+0000 7fba55b48640 1 Processor -- start 2024-04-01T01:30:43.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba55b48640 1 -- start start 2024-04-01T01:30:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba55b48640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba50100670 0x7fba50069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba55b48640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba50101860 0x7fba5019c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba55b48640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba50102220 0x7fba5019ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba55b48640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fba5010c1b0 con 0x7fba50102220 2024-04-01T01:30:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba55b48640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fba5010c030 con 0x7fba50100670 2024-04-01T01:30:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba55b48640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fba5010c330 con 0x7fba50101860 2024-04-01T01:30:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba4ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba50102220 0x7fba5019ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:43.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba4ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba50102220 0x7fba5019ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52434/0 (socket says 172.21.15.70:52434) 2024-04-01T01:30:43.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba4ffff640 1 -- 172.21.15.70:0/929230865 learned_addr learned my addr 172.21.15.70:0/929230865 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:30:43.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba4effd640 1 --2- 172.21.15.70:0/929230865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba50101860 0x7fba5019c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:43.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba4f7fe640 1 --2- 172.21.15.70:0/929230865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba50100670 0x7fba50069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:43.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba4ffff640 1 -- 172.21.15.70:0/929230865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba50101860 msgr2=0x7fba5019c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:43.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba4ffff640 1 --2- 172.21.15.70:0/929230865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba50101860 0x7fba5019c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:43.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba4ffff640 1 -- 172.21.15.70:0/929230865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba50100670 msgr2=0x7fba50069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:43.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba4ffff640 1 --2- 172.21.15.70:0/929230865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba50100670 0x7fba50069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:43.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba4ffff640 1 -- 172.21.15.70:0/929230865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba5019f5c0 con 0x7fba50102220 2024-04-01T01:30:43.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba4f7fe640 1 --2- 172.21.15.70:0/929230865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba50100670 0x7fba50069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:30:43.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.358+0000 7fba4effd640 1 --2- 172.21.15.70:0/929230865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba50101860 0x7fba5019c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:30:43.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.359+0000 7fba4ffff640 1 --2- 172.21.15.70:0/929230865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba50102220 0x7fba5019ed00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fba4800eaa0 tx=0x7fba4800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:43.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.359+0000 7fba4cff9640 1 -- 172.21.15.70:0/929230865 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4800cdb0 con 0x7fba50102220 2024-04-01T01:30:43.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.359+0000 7fba55b48640 1 -- 172.21.15.70:0/929230865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba5019f8b0 con 0x7fba50102220 2024-04-01T01:30:43.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.359+0000 7fba4cff9640 1 -- 172.21.15.70:0/929230865 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba48005020 con 0x7fba50102220 2024-04-01T01:30:43.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.359+0000 7fba4cff9640 1 -- 172.21.15.70:0/929230865 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba480116d0 con 0x7fba50102220 2024-04-01T01:30:43.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.359+0000 7fba55b48640 1 -- 172.21.15.70:0/929230865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba5019fe00 con 0x7fba50102220 2024-04-01T01:30:43.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.360+0000 7fba55b48640 1 -- 172.21.15.70:0/929230865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba50107a10 con 0x7fba50102220 2024-04-01T01:30:43.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.361+0000 7fba4cff9640 1 -- 172.21.15.70:0/929230865 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fba480051d0 con 0x7fba50102220 2024-04-01T01:30:43.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.362+0000 7fba4cff9640 1 --2- 172.21.15.70:0/929230865 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba380a24a0 0x7fba380a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:43.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.362+0000 7fba4f7fe640 1 --2- 172.21.15.70:0/929230865 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba380a24a0 0x7fba380a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:43.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.362+0000 7fba4cff9640 1 -- 172.21.15.70:0/929230865 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fba480ce510 con 0x7fba50102220 2024-04-01T01:30:43.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.362+0000 7fba4f7fe640 1 --2- 172.21.15.70:0/929230865 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba380a24a0 0x7fba380a4960 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fba44005d00 tx=0x7fba44005ef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:43.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.366+0000 7fba4cff9640 1 -- 172.21.15.70:0/929230865 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fba48016030 con 0x7fba50102220 2024-04-01T01:30:43.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:43 smithi189 ceph-mon[35753]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:43.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:43.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:43.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:43.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:43.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:43.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:43 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:43.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.610+0000 7fba55b48640 1 -- 172.21.15.70:0/929230865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba50062350 con 0x7fba50102220 2024-04-01T01:30:43.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.610+0000 7fba4cff9640 1 -- 172.21.15.70:0/929230865 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fba4809bad0 con 0x7fba50102220 2024-04-01T01:30:43.611 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:30:43.611 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-01T01:30:43.611 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 3 2024-04-01T01:30:43.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.613+0000 7fba55b48640 1 -- 172.21.15.70:0/929230865 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba380a24a0 msgr2=0x7fba380a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:43.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.613+0000 7fba55b48640 1 --2- 172.21.15.70:0/929230865 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba380a24a0 0x7fba380a4960 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fba44005d00 tx=0x7fba44005ef0 comp rx=0 tx=0).stop 2024-04-01T01:30:43.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.613+0000 7fba55b48640 1 -- 172.21.15.70:0/929230865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba50102220 msgr2=0x7fba5019ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:43.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.613+0000 7fba55b48640 1 --2- 172.21.15.70:0/929230865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba50102220 0x7fba5019ed00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fba4800eaa0 tx=0x7fba4800ef60 comp rx=0 tx=0).stop 2024-04-01T01:30:43.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.613+0000 7fba55b48640 1 -- 172.21.15.70:0/929230865 shutdown_connections 2024-04-01T01:30:43.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.613+0000 7fba55b48640 1 --2- 172.21.15.70:0/929230865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba50102220 0x7fba5019ed00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:43.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.613+0000 7fba55b48640 1 --2- 172.21.15.70:0/929230865 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba380a24a0 0x7fba380a4960 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:43.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.613+0000 7fba55b48640 1 --2- 172.21.15.70:0/929230865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba50101860 0x7fba5019c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:43.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.613+0000 7fba55b48640 1 --2- 172.21.15.70:0/929230865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba50100670 0x7fba50069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:43.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.613+0000 7fba55b48640 1 -- 172.21.15.70:0/929230865 >> 172.21.15.70:0/929230865 conn(0x7fba500fbbe0 msgr2=0x7fba50104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:43.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.613+0000 7fba55b48640 1 -- 172.21.15.70:0/929230865 shutdown_connections 2024-04-01T01:30:43.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:43.613+0000 7fba55b48640 1 -- 172.21.15.70:0/929230865 wait complete. 2024-04-01T01:30:44.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:44 smithi070 ceph-mon[32179]: Deploying daemon mds.j on smithi070 2024-04-01T01:30:44.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:44 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.81:6832/1246539452,v1:172.21.15.81:6833/1246539452] up:boot 2024-04-01T01:30:44.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:44 smithi070 ceph-mon[32179]: fsmap 2 up:standby 2024-04-01T01:30:44.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:44 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-01T01:30:44.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:44 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/929230865' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:44.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:44 smithi081 ceph-mon[35142]: Deploying daemon mds.j on smithi070 2024-04-01T01:30:44.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:44 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.81:6832/1246539452,v1:172.21.15.81:6833/1246539452] up:boot 2024-04-01T01:30:44.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:44 smithi081 ceph-mon[35142]: fsmap 2 up:standby 2024-04-01T01:30:44.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:44 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-01T01:30:44.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:44 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/929230865' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:44.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:44 smithi189 ceph-mon[35753]: Deploying daemon mds.j on smithi070 2024-04-01T01:30:44.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:44 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.81:6832/1246539452,v1:172.21.15.81:6833/1246539452] up:boot 2024-04-01T01:30:44.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:44 smithi189 ceph-mon[35753]: fsmap 2 up:standby 2024-04-01T01:30:44.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:44 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-01T01:30:44.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:44 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/929230865' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:45.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:45 smithi081 ceph-mon[35142]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:45.405 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:45 smithi070 ceph-mon[32179]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:45.405 INFO:journalctl@ceph.mds.j.smithi070.stdout:Apr 01 01:30:45 smithi070 systemd[1]: Starting Ceph mds.j for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:30:45.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:45 smithi189 ceph-mon[35753]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:45.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:30:45.666 INFO:journalctl@ceph.mds.j.smithi070.stdout:Apr 01 01:30:45 smithi070 podman[72749]: 2024-04-01 01:30:45.308402348 +0000 UTC m=+0.017323499 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:30:45.667 INFO:journalctl@ceph.mds.j.smithi070.stdout:Apr 01 01:30:45 smithi070 podman[72749]: 2024-04-01 01:30:45.421513623 +0000 UTC m=+0.130434747 container create bfab699b9b086680c5ce676edf959ab637ceb44a6c7f9255852f59a3aa3ae160 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-j, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-04-01T01:30:45.667 INFO:journalctl@ceph.mds.j.smithi070.stdout:Apr 01 01:30:45 smithi070 podman[72749]: 2024-04-01 01:30:45.640525974 +0000 UTC m=+0.349447111 container init bfab699b9b086680c5ce676edf959ab637ceb44a6c7f9255852f59a3aa3ae160 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-j, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, RELEASE=squid-384f31f) 2024-04-01T01:30:45.667 INFO:journalctl@ceph.mds.j.smithi070.stdout:Apr 01 01:30:45 smithi070 podman[72749]: 2024-04-01 01:30:45.643494053 +0000 UTC m=+0.352415183 container start bfab699b9b086680c5ce676edf959ab637ceb44a6c7f9255852f59a3aa3ae160 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-j, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-01T01:30:45.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.733+0000 7fceb49e2640 1 -- 172.21.15.70:0/292316659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fceb0101310 msgr2=0x7fceb0108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:45.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.733+0000 7fceb49e2640 1 --2- 172.21.15.70:0/292316659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fceb0101310 0x7fceb0108bb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcea0009900 tx=0x7fcea002bc30 comp rx=0 tx=0).stop 2024-04-01T01:30:45.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.733+0000 7fceb49e2640 1 -- 172.21.15.70:0/292316659 shutdown_connections 2024-04-01T01:30:45.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.733+0000 7fceb49e2640 1 --2- 172.21.15.70:0/292316659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fceb0101310 0x7fceb0108bb0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:45.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.733+0000 7fceb49e2640 1 --2- 172.21.15.70:0/292316659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fceb0100950 0x7fceb0100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:45.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.733+0000 7fceb49e2640 1 --2- 172.21.15.70:0/292316659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fceb0101c20 0x7fceb0102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:45.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.733+0000 7fceb49e2640 1 -- 172.21.15.70:0/292316659 >> 172.21.15.70:0/292316659 conn(0x7fceb00fb380 msgr2=0x7fceb00fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:45.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.733+0000 7fceb49e2640 1 -- 172.21.15.70:0/292316659 shutdown_connections 2024-04-01T01:30:45.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.733+0000 7fceb49e2640 1 -- 172.21.15.70:0/292316659 wait complete. 2024-04-01T01:30:45.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.734+0000 7fceb49e2640 1 Processor -- start 2024-04-01T01:30:45.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.734+0000 7fceb49e2640 1 -- start start 2024-04-01T01:30:45.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.734+0000 7fceb49e2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fceb0100950 0x7fceb01967d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:45.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.734+0000 7fceb49e2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fceb0101310 0x7fceb0196d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:45.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.734+0000 7fceb49e2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fceb0101c20 0x7fceb019dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:45.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.734+0000 7fceb49e2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fceb010c1b0 con 0x7fceb0100950 2024-04-01T01:30:45.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.734+0000 7fceb49e2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fceb010c030 con 0x7fceb0101310 2024-04-01T01:30:45.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.734+0000 7fceb49e2640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fceb010c330 con 0x7fceb0101c20 2024-04-01T01:30:45.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.734+0000 7fceae59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fceb0100950 0x7fceb01967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:45.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.734+0000 7fceae59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fceb0100950 0x7fceb01967d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52472/0 (socket says 172.21.15.70:52472) 2024-04-01T01:30:45.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.734+0000 7fceae59c640 1 -- 172.21.15.70:0/1708237380 learned_addr learned my addr 172.21.15.70:0/1708237380 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:30:45.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.734+0000 7fceadd9b640 1 --2- 172.21.15.70:0/1708237380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fceb0101310 0x7fceb0196d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:45.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.735+0000 7fceaed9d640 1 --2- 172.21.15.70:0/1708237380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fceb0101c20 0x7fceb019dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:45.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.735+0000 7fceae59c640 1 -- 172.21.15.70:0/1708237380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fceb0101c20 msgr2=0x7fceb019dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:45.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.735+0000 7fceae59c640 1 --2- 172.21.15.70:0/1708237380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fceb0101c20 0x7fceb019dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:45.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.735+0000 7fceae59c640 1 -- 172.21.15.70:0/1708237380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fceb0101310 msgr2=0x7fceb0196d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:45.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.735+0000 7fceae59c640 1 --2- 172.21.15.70:0/1708237380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fceb0101310 0x7fceb0196d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:45.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.735+0000 7fceae59c640 1 -- 172.21.15.70:0/1708237380 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb019e3f0 con 0x7fceb0100950 2024-04-01T01:30:45.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.735+0000 7fceadd9b640 1 --2- 172.21.15.70:0/1708237380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fceb0101310 0x7fceb0196d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:30:45.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.735+0000 7fceaed9d640 1 --2- 172.21.15.70:0/1708237380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fceb0101c20 0x7fceb019dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:30:45.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.735+0000 7fceae59c640 1 --2- 172.21.15.70:0/1708237380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fceb0100950 0x7fceb01967d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcea800ebc0 tx=0x7fcea800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:45.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.735+0000 7fcea77fe640 1 -- 172.21.15.70:0/1708237380 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea8003b70 con 0x7fceb0100950 2024-04-01T01:30:45.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.735+0000 7fcea77fe640 1 -- 172.21.15.70:0/1708237380 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcea8003d10 con 0x7fceb0100950 2024-04-01T01:30:45.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.735+0000 7fceb49e2640 1 -- 172.21.15.70:0/1708237380 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceb019e6e0 con 0x7fceb0100950 2024-04-01T01:30:45.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.735+0000 7fcea77fe640 1 -- 172.21.15.70:0/1708237380 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea8019840 con 0x7fceb0100950 2024-04-01T01:30:45.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.735+0000 7fceb49e2640 1 -- 172.21.15.70:0/1708237380 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceb019ec30 con 0x7fceb0100950 2024-04-01T01:30:45.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.736+0000 7fceb49e2640 1 -- 172.21.15.70:0/1708237380 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fceb0107a90 con 0x7fceb0100950 2024-04-01T01:30:45.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.737+0000 7fcea77fe640 1 -- 172.21.15.70:0/1708237380 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fcea80199e0 con 0x7fceb0100950 2024-04-01T01:30:45.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.737+0000 7fcea77fe640 1 --2- 172.21.15.70:0/1708237380 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fce9c0a24a0 0x7fce9c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:45.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.737+0000 7fceadd9b640 1 --2- 172.21.15.70:0/1708237380 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fce9c0a24a0 0x7fce9c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:45.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.738+0000 7fcea77fe640 1 -- 172.21.15.70:0/1708237380 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcea80cec80 con 0x7fceb0100950 2024-04-01T01:30:45.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.738+0000 7fceadd9b640 1 --2- 172.21.15.70:0/1708237380 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fce9c0a24a0 0x7fce9c0a4960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fceb0197ce0 tx=0x7fce9800a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:45.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.742+0000 7fcea77fe640 1 -- 172.21.15.70:0/1708237380 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcea809c2e0 con 0x7fceb0100950 2024-04-01T01:30:45.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.963+0000 7fceb49e2640 1 -- 172.21.15.70:0/1708237380 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fceb002d170 con 0x7fceb0100950 2024-04-01T01:30:45.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.963+0000 7fcea77fe640 1 -- 172.21.15.70:0/1708237380 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fcea809bbe0 con 0x7fceb0100950 2024-04-01T01:30:45.964 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:30:45.964 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-01T01:30:45.964 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 3 2024-04-01T01:30:45.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.965+0000 7fceb49e2640 1 -- 172.21.15.70:0/1708237380 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fce9c0a24a0 msgr2=0x7fce9c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:45.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.965+0000 7fceb49e2640 1 --2- 172.21.15.70:0/1708237380 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fce9c0a24a0 0x7fce9c0a4960 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fceb0197ce0 tx=0x7fce9800a5c0 comp rx=0 tx=0).stop 2024-04-01T01:30:45.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.965+0000 7fceb49e2640 1 -- 172.21.15.70:0/1708237380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fceb0100950 msgr2=0x7fceb01967d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:45.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.965+0000 7fceb49e2640 1 --2- 172.21.15.70:0/1708237380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fceb0100950 0x7fceb01967d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcea800ebc0 tx=0x7fcea800c6a0 comp rx=0 tx=0).stop 2024-04-01T01:30:45.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.965+0000 7fceb49e2640 1 -- 172.21.15.70:0/1708237380 shutdown_connections 2024-04-01T01:30:45.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.965+0000 7fceb49e2640 1 --2- 172.21.15.70:0/1708237380 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fce9c0a24a0 0x7fce9c0a4960 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:45.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.965+0000 7fceb49e2640 1 --2- 172.21.15.70:0/1708237380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fceb0101c20 0x7fceb019dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:45.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.965+0000 7fceb49e2640 1 --2- 172.21.15.70:0/1708237380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fceb0101310 0x7fceb0196d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:45.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.965+0000 7fceb49e2640 1 --2- 172.21.15.70:0/1708237380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fceb0100950 0x7fceb01967d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:45.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.965+0000 7fceb49e2640 1 -- 172.21.15.70:0/1708237380 >> 172.21.15.70:0/1708237380 conn(0x7fceb00fb380 msgr2=0x7fceb0104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:45.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.965+0000 7fceb49e2640 1 -- 172.21.15.70:0/1708237380 shutdown_connections 2024-04-01T01:30:45.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:45.965+0000 7fceb49e2640 1 -- 172.21.15.70:0/1708237380 wait complete. 2024-04-01T01:30:45.976 INFO:journalctl@ceph.mds.j.smithi070.stdout:Apr 01 01:30:45 smithi070 ceph-mds[72784]: -- 172.21.15.70:0/3633053440 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f2cebc1a40 con 0x55f2ceb94800 2024-04-01T01:30:45.976 INFO:journalctl@ceph.mds.j.smithi070.stdout:Apr 01 01:30:45 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-j[72764]: starting mds.j at 2024-04-01T01:30:45.976 INFO:journalctl@ceph.mds.j.smithi070.stdout:Apr 01 01:30:45 smithi070 bash[72749]: bfab699b9b086680c5ce676edf959ab637ceb44a6c7f9255852f59a3aa3ae160 2024-04-01T01:30:45.976 INFO:journalctl@ceph.mds.j.smithi070.stdout:Apr 01 01:30:45 smithi070 systemd[1]: Started Ceph mds.j for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:30:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:46 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:46 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:46 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:46 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:46 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:46 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:46 smithi070 ceph-mon[32179]: Deploying daemon mds.d on smithi070 2024-04-01T01:30:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:46 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1708237380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:47.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:46 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.70:6834/33736871,v1:172.21.15.70:6835/33736871] up:boot 2024-04-01T01:30:47.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:46 smithi070 ceph-mon[32179]: fsmap 3 up:standby 2024-04-01T01:30:47.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:46 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-01T01:30:47.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:46 smithi070 ceph-mon[32179]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:46 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:46 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:46 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:47.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:46 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:47.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:46 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:47.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:46 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:47.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:46 smithi081 ceph-mon[35142]: Deploying daemon mds.d on smithi070 2024-04-01T01:30:47.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:46 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1708237380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:47.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:46 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.70:6834/33736871,v1:172.21.15.70:6835/33736871] up:boot 2024-04-01T01:30:47.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:46 smithi081 ceph-mon[35142]: fsmap 3 up:standby 2024-04-01T01:30:47.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:46 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-01T01:30:47.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:46 smithi081 ceph-mon[35142]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:47.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:46 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:47.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:46 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:47.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:46 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:47.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:46 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:47.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:46 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:47.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:46 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:47.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:46 smithi189 ceph-mon[35753]: Deploying daemon mds.d on smithi070 2024-04-01T01:30:47.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:46 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1708237380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:47.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:46 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.70:6834/33736871,v1:172.21.15.70:6835/33736871] up:boot 2024-04-01T01:30:47.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:46 smithi189 ceph-mon[35753]: fsmap 3 up:standby 2024-04-01T01:30:47.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:46 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-01T01:30:47.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:46 smithi189 ceph-mon[35753]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:47.979 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:30:48.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.170+0000 7fb402b8a640 1 -- 172.21.15.70:0/1894689339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3fc1001e0 msgr2=0x7fb3fc109d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:48.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.170+0000 7fb402b8a640 1 --2- 172.21.15.70:0/1894689339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3fc1001e0 0x7fb3fc109d90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009ca0 tx=0x7fb3f402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:30:48.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.170+0000 7fb402b8a640 1 -- 172.21.15.70:0/1894689339 shutdown_connections 2024-04-01T01:30:48.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.170+0000 7fb402b8a640 1 --2- 172.21.15.70:0/1894689339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3fc1001e0 0x7fb3fc109d90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:48.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.170+0000 7fb402b8a640 1 --2- 172.21.15.70:0/1894689339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3fc0ff820 0x7fb3fc0ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:48.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.170+0000 7fb402b8a640 1 --2- 172.21.15.70:0/1894689339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3fc102db0 0x7fb3fc1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:48.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.170+0000 7fb402b8a640 1 -- 172.21.15.70:0/1894689339 >> 172.21.15.70:0/1894689339 conn(0x7fb3fc0fb380 msgr2=0x7fb3fc0fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:48.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.170+0000 7fb402b8a640 1 -- 172.21.15.70:0/1894689339 shutdown_connections 2024-04-01T01:30:48.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.170+0000 7fb402b8a640 1 -- 172.21.15.70:0/1894689339 wait complete. 2024-04-01T01:30:48.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb402b8a640 1 Processor -- start 2024-04-01T01:30:48.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb402b8a640 1 -- start start 2024-04-01T01:30:48.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb402b8a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3fc0ff820 0x7fb3fc197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:48.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb402b8a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3fc1001e0 0x7fb3fc197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:48.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb402b8a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3fc102db0 0x7fb3fc19eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:48.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb402b8a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb3fc10d2b0 con 0x7fb3fc102db0 2024-04-01T01:30:48.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb402b8a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb3fc10d130 con 0x7fb3fc0ff820 2024-04-01T01:30:48.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb402b8a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb3fc10d430 con 0x7fb3fc1001e0 2024-04-01T01:30:48.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb401127640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3fc102db0 0x7fb3fc19eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:48.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb401127640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3fc102db0 0x7fb3fc19eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36526/0 (socket says 172.21.15.70:36526) 2024-04-01T01:30:48.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb401127640 1 -- 172.21.15.70:0/2454425614 learned_addr learned my addr 172.21.15.70:0/2454425614 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:30:48.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb400926640 1 --2- 172.21.15.70:0/2454425614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3fc0ff820 0x7fb3fc197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:48.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb3fbfff640 1 --2- 172.21.15.70:0/2454425614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3fc1001e0 0x7fb3fc197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:48.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb401127640 1 -- 172.21.15.70:0/2454425614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3fc1001e0 msgr2=0x7fb3fc197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:48.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb401127640 1 --2- 172.21.15.70:0/2454425614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3fc1001e0 0x7fb3fc197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:48.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb401127640 1 -- 172.21.15.70:0/2454425614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3fc0ff820 msgr2=0x7fb3fc197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:48.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb401127640 1 --2- 172.21.15.70:0/2454425614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3fc0ff820 0x7fb3fc197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:48.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb401127640 1 -- 172.21.15.70:0/2454425614 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3fc19f540 con 0x7fb3fc102db0 2024-04-01T01:30:48.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.171+0000 7fb3fbfff640 1 --2- 172.21.15.70:0/2454425614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3fc1001e0 0x7fb3fc197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:30:48.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.172+0000 7fb401127640 1 --2- 172.21.15.70:0/2454425614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3fc102db0 0x7fb3fc19eed0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb3f402c700 tx=0x7fb3f40328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:48.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.172+0000 7fb3f9ffb640 1 -- 172.21.15.70:0/2454425614 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4041020 con 0x7fb3fc102db0 2024-04-01T01:30:48.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.172+0000 7fb402b8a640 1 -- 172.21.15.70:0/2454425614 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3fc19f7d0 con 0x7fb3fc102db0 2024-04-01T01:30:48.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.172+0000 7fb3f9ffb640 1 -- 172.21.15.70:0/2454425614 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb3f4003e20 con 0x7fb3fc102db0 2024-04-01T01:30:48.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.172+0000 7fb3f9ffb640 1 -- 172.21.15.70:0/2454425614 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4039040 con 0x7fb3fc102db0 2024-04-01T01:30:48.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.172+0000 7fb402b8a640 1 -- 172.21.15.70:0/2454425614 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3fc19fd10 con 0x7fb3fc102db0 2024-04-01T01:30:48.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.173+0000 7fb402b8a640 1 -- 172.21.15.70:0/2454425614 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3fc108c70 con 0x7fb3fc102db0 2024-04-01T01:30:48.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.175+0000 7fb3f9ffb640 1 -- 172.21.15.70:0/2454425614 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fb3f4044020 con 0x7fb3fc102db0 2024-04-01T01:30:48.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.175+0000 7fb3f9ffb640 1 --2- 172.21.15.70:0/2454425614 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3ec0a24a0 0x7fb3ec0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:48.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.175+0000 7fb400926640 1 --2- 172.21.15.70:0/2454425614 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3ec0a24a0 0x7fb3ec0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:48.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.175+0000 7fb3f9ffb640 1 -- 172.21.15.70:0/2454425614 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb3f40f0b30 con 0x7fb3fc102db0 2024-04-01T01:30:48.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.176+0000 7fb400926640 1 --2- 172.21.15.70:0/2454425614 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3ec0a24a0 0x7fb3ec0a4960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00058c0 tx=0x7fb3f000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:48.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.179+0000 7fb3f9ffb640 1 -- 172.21.15.70:0/2454425614 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb3f404b020 con 0x7fb3fc102db0 2024-04-01T01:30:48.253 INFO:journalctl@ceph.mds.d.smithi070.stdout:Apr 01 01:30:48 smithi070 systemd[1]: Starting Ceph mds.d for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:30:48.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.401+0000 7fb402b8a640 1 -- 172.21.15.70:0/2454425614 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3fc062350 con 0x7fb3fc102db0 2024-04-01T01:30:48.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.401+0000 7fb3f9ffb640 1 -- 172.21.15.70:0/2454425614 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3040 (secure 0 0 0) 0x7fb3f40be190 con 0x7fb3fc102db0 2024-04-01T01:30:48.402 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:30:48.402 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-01T01:30:48.402 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 4 2024-04-01T01:30:48.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.403+0000 7fb402b8a640 1 -- 172.21.15.70:0/2454425614 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3ec0a24a0 msgr2=0x7fb3ec0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:48.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.403+0000 7fb402b8a640 1 --2- 172.21.15.70:0/2454425614 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3ec0a24a0 0x7fb3ec0a4960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00058c0 tx=0x7fb3f000a2d0 comp rx=0 tx=0).stop 2024-04-01T01:30:48.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.403+0000 7fb402b8a640 1 -- 172.21.15.70:0/2454425614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3fc102db0 msgr2=0x7fb3fc19eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:48.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.403+0000 7fb402b8a640 1 --2- 172.21.15.70:0/2454425614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3fc102db0 0x7fb3fc19eed0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb3f402c700 tx=0x7fb3f40328a0 comp rx=0 tx=0).stop 2024-04-01T01:30:48.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.403+0000 7fb402b8a640 1 -- 172.21.15.70:0/2454425614 shutdown_connections 2024-04-01T01:30:48.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.403+0000 7fb402b8a640 1 --2- 172.21.15.70:0/2454425614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3fc102db0 0x7fb3fc19eed0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:48.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.403+0000 7fb402b8a640 1 --2- 172.21.15.70:0/2454425614 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3ec0a24a0 0x7fb3ec0a4960 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:48.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.403+0000 7fb402b8a640 1 --2- 172.21.15.70:0/2454425614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3fc1001e0 0x7fb3fc197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:48.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.403+0000 7fb402b8a640 1 --2- 172.21.15.70:0/2454425614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3fc0ff820 0x7fb3fc197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:48.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.403+0000 7fb402b8a640 1 -- 172.21.15.70:0/2454425614 >> 172.21.15.70:0/2454425614 conn(0x7fb3fc0fb380 msgr2=0x7fb3fc105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:48.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.403+0000 7fb402b8a640 1 -- 172.21.15.70:0/2454425614 shutdown_connections 2024-04-01T01:30:48.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:48.404+0000 7fb402b8a640 1 -- 172.21.15.70:0/2454425614 wait complete. 2024-04-01T01:30:48.581 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:48 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2454425614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:48.581 INFO:journalctl@ceph.mds.d.smithi070.stdout:Apr 01 01:30:48 smithi070 podman[73133]: 2024-04-01 01:30:48.2334949 +0000 UTC m=+0.018081557 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:30:48.581 INFO:journalctl@ceph.mds.d.smithi070.stdout:Apr 01 01:30:48 smithi070 podman[73133]: 2024-04-01 01:30:48.350966015 +0000 UTC m=+0.135552644 container create c36c2d6d7d075dd0cbdd5db2a57321bd9bd3832e426d6f136f6f4ed18a562c5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-01T01:30:48.581 INFO:journalctl@ceph.mds.d.smithi070.stdout:Apr 01 01:30:48 smithi070 podman[73133]: 2024-04-01 01:30:48.57863524 +0000 UTC m=+0.363221883 container init c36c2d6d7d075dd0cbdd5db2a57321bd9bd3832e426d6f136f6f4ed18a562c5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-384f31f) 2024-04-01T01:30:48.581 INFO:journalctl@ceph.mds.d.smithi070.stdout:Apr 01 01:30:48 smithi070 podman[73133]: 2024-04-01 01:30:48.581948814 +0000 UTC m=+0.366535454 container start c36c2d6d7d075dd0cbdd5db2a57321bd9bd3832e426d6f136f6f4ed18a562c5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-d, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-01T01:30:48.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:48 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2454425614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:48.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:48 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2454425614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:49.004 INFO:journalctl@ceph.mds.d.smithi070.stdout:Apr 01 01:30:48 smithi070 ceph-mds[73154]: -- 172.21.15.70:0/1056522061 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f61068da40 con 0x55f610660800 2024-04-01T01:30:49.004 INFO:journalctl@ceph.mds.d.smithi070.stdout:Apr 01 01:30:48 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-d[73150]: starting mds.d at 2024-04-01T01:30:49.004 INFO:journalctl@ceph.mds.d.smithi070.stdout:Apr 01 01:30:48 smithi070 bash[73133]: c36c2d6d7d075dd0cbdd5db2a57321bd9bd3832e426d6f136f6f4ed18a562c5b 2024-04-01T01:30:49.004 INFO:journalctl@ceph.mds.d.smithi070.stdout:Apr 01 01:30:48 smithi070 systemd[1]: Started Ceph mds.d for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:30:49.645 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:49 smithi081 ceph-mon[35142]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:49.645 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:49 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:49.645 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:49 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:49.645 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:49 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:49.645 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:49 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:49.645 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:49 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:49.645 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:49 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:49.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:49 smithi189 ceph-mon[35753]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:49.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:49 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:49.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:49 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:49.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:49 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:49.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:49 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:49.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:49 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:49.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:49 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:49.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:49 smithi070 ceph-mon[32179]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:49.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:49 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:49.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:49 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:49.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:49 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:49.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:49 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:49.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:49 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:49.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:49 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:50.419 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:30:50.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.535+0000 7fd8e4e9a640 1 -- 172.21.15.70:0/239497438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8e01022a0 msgr2=0x7fd8e0108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:50.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.535+0000 7fd8e4e9a640 1 --2- 172.21.15.70:0/239497438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8e01022a0 0x7fd8e0108b30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8009ca0 tx=0x7fd8d802c0f0 comp rx=0 tx=0).stop 2024-04-01T01:30:50.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.535+0000 7fd8e4e9a640 1 -- 172.21.15.70:0/239497438 shutdown_connections 2024-04-01T01:30:50.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.535+0000 7fd8e4e9a640 1 --2- 172.21.15.70:0/239497438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8e01022a0 0x7fd8e0108b30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:50.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.535+0000 7fd8e4e9a640 1 --2- 172.21.15.70:0/239497438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8e01018e0 0x7fd8e0101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:50.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.535+0000 7fd8e4e9a640 1 --2- 172.21.15.70:0/239497438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e01006f0 0x7fd8e0100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:50.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.535+0000 7fd8e4e9a640 1 -- 172.21.15.70:0/239497438 >> 172.21.15.70:0/239497438 conn(0x7fd8e00fbc80 msgr2=0x7fd8e00fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:50.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.535+0000 7fd8e4e9a640 1 -- 172.21.15.70:0/239497438 shutdown_connections 2024-04-01T01:30:50.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.535+0000 7fd8e4e9a640 1 -- 172.21.15.70:0/239497438 wait complete. 2024-04-01T01:30:50.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.535+0000 7fd8e4e9a640 1 Processor -- start 2024-04-01T01:30:50.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8e4e9a640 1 -- start start 2024-04-01T01:30:50.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8e4e9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8e01006f0 0x7fd8e006a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8e4e9a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8e01018e0 0x7fd8e019c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8e4e9a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e01022a0 0x7fd8e019ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8e4e9a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e010c1b0 con 0x7fd8e01018e0 2024-04-01T01:30:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8e4e9a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e010c030 con 0x7fd8e01006f0 2024-04-01T01:30:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8e4e9a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e010c330 con 0x7fd8e01022a0 2024-04-01T01:30:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8d7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8e01018e0 0x7fd8e019c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8d7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8e01018e0 0x7fd8e019c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36558/0 (socket says 172.21.15.70:36558) 2024-04-01T01:30:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8d7fff640 1 -- 172.21.15.70:0/2633674989 learned_addr learned my addr 172.21.15.70:0/2633674989 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:30:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8de59c640 1 --2- 172.21.15.70:0/2633674989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8e01006f0 0x7fd8e006a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8ded9d640 1 --2- 172.21.15.70:0/2633674989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e01022a0 0x7fd8e019ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8d7fff640 1 -- 172.21.15.70:0/2633674989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e01022a0 msgr2=0x7fd8e019ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8d7fff640 1 --2- 172.21.15.70:0/2633674989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e01022a0 0x7fd8e019ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:50.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8d7fff640 1 -- 172.21.15.70:0/2633674989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8e01006f0 msgr2=0x7fd8e006a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8d7fff640 1 --2- 172.21.15.70:0/2633674989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8e01006f0 0x7fd8e006a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8d7fff640 1 -- 172.21.15.70:0/2633674989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e019f5d0 con 0x7fd8e01018e0 2024-04-01T01:30:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8ded9d640 1 --2- 172.21.15.70:0/2633674989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e01022a0 0x7fd8e019ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:30:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8d7fff640 1 --2- 172.21.15.70:0/2633674989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8e01018e0 0x7fd8e019c6a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800d950 tx=0x7fd8c800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8d77fe640 1 -- 172.21.15.70:0/2633674989 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8017030 con 0x7fd8e01018e0 2024-04-01T01:30:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8d77fe640 1 -- 172.21.15.70:0/2633674989 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd8c8004020 con 0x7fd8e01018e0 2024-04-01T01:30:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8e4e9a640 1 -- 172.21.15.70:0/2633674989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8e019f860 con 0x7fd8e01018e0 2024-04-01T01:30:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.536+0000 7fd8d77fe640 1 -- 172.21.15.70:0/2633674989 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c80047a0 con 0x7fd8e01018e0 2024-04-01T01:30:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.537+0000 7fd8e4e9a640 1 -- 172.21.15.70:0/2633674989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8e019fe00 con 0x7fd8e01018e0 2024-04-01T01:30:50.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.538+0000 7fd8e4e9a640 1 -- 172.21.15.70:0/2633674989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8e0107a60 con 0x7fd8e01018e0 2024-04-01T01:30:50.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.539+0000 7fd8d77fe640 1 -- 172.21.15.70:0/2633674989 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fd8c80041d0 con 0x7fd8e01018e0 2024-04-01T01:30:50.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.539+0000 7fd8d77fe640 1 --2- 172.21.15.70:0/2633674989 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd8cc0a2520 0x7fd8cc0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:50.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.539+0000 7fd8de59c640 1 --2- 172.21.15.70:0/2633674989 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd8cc0a2520 0x7fd8cc0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:50.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.539+0000 7fd8d77fe640 1 -- 172.21.15.70:0/2633674989 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd8c80cdc50 con 0x7fd8e01018e0 2024-04-01T01:30:50.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.540+0000 7fd8de59c640 1 --2- 172.21.15.70:0/2633674989 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd8cc0a2520 0x7fd8cc0a49e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0005fd0 tx=0x7fd8d000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:50.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.544+0000 7fd8d77fe640 1 -- 172.21.15.70:0/2633674989 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd8c80d3020 con 0x7fd8e01018e0 2024-04-01T01:30:50.669 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:50 smithi081 ceph-mon[35142]: Deploying daemon mds.e on smithi081 2024-04-01T01:30:50.669 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:50 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.70:6836/2674132555,v1:172.21.15.70:6837/2674132555] up:boot 2024-04-01T01:30:50.670 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:50 smithi081 ceph-mon[35142]: fsmap 4 up:standby 2024-04-01T01:30:50.670 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:50 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-01T01:30:50.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:50 smithi189 ceph-mon[35753]: Deploying daemon mds.e on smithi081 2024-04-01T01:30:50.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:50 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.70:6836/2674132555,v1:172.21.15.70:6837/2674132555] up:boot 2024-04-01T01:30:50.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:50 smithi189 ceph-mon[35753]: fsmap 4 up:standby 2024-04-01T01:30:50.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:50 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-01T01:30:50.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:50 smithi070 ceph-mon[32179]: Deploying daemon mds.e on smithi081 2024-04-01T01:30:50.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:50 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.70:6836/2674132555,v1:172.21.15.70:6837/2674132555] up:boot 2024-04-01T01:30:50.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:50 smithi070 ceph-mon[32179]: fsmap 4 up:standby 2024-04-01T01:30:50.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:50 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-01T01:30:50.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.757+0000 7fd8e4e9a640 1 -- 172.21.15.70:0/2633674989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8e002d170 con 0x7fd8e01018e0 2024-04-01T01:30:50.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.758+0000 7fd8d77fe640 1 -- 172.21.15.70:0/2633674989 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3865 (secure 0 0 0) 0x7fd8c809b2b0 con 0x7fd8e01018e0 2024-04-01T01:30:50.758 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:30:50.758 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-01T01:30:50.758 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 5 2024-04-01T01:30:50.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.760+0000 7fd8e4e9a640 1 -- 172.21.15.70:0/2633674989 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd8cc0a2520 msgr2=0x7fd8cc0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:50.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.760+0000 7fd8e4e9a640 1 --2- 172.21.15.70:0/2633674989 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd8cc0a2520 0x7fd8cc0a49e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0005fd0 tx=0x7fd8d000b390 comp rx=0 tx=0).stop 2024-04-01T01:30:50.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.760+0000 7fd8e4e9a640 1 -- 172.21.15.70:0/2633674989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8e01018e0 msgr2=0x7fd8e019c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:50.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.760+0000 7fd8e4e9a640 1 --2- 172.21.15.70:0/2633674989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8e01018e0 0x7fd8e019c6a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800d950 tx=0x7fd8c800de10 comp rx=0 tx=0).stop 2024-04-01T01:30:50.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.760+0000 7fd8e4e9a640 1 -- 172.21.15.70:0/2633674989 shutdown_connections 2024-04-01T01:30:50.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.760+0000 7fd8e4e9a640 1 --2- 172.21.15.70:0/2633674989 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd8cc0a2520 0x7fd8cc0a49e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:50.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.760+0000 7fd8e4e9a640 1 --2- 172.21.15.70:0/2633674989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8e01022a0 0x7fd8e019ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:50.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.760+0000 7fd8e4e9a640 1 --2- 172.21.15.70:0/2633674989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd8e01018e0 0x7fd8e019c6a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:50.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.760+0000 7fd8e4e9a640 1 --2- 172.21.15.70:0/2633674989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8e01006f0 0x7fd8e006a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:50.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.760+0000 7fd8e4e9a640 1 -- 172.21.15.70:0/2633674989 >> 172.21.15.70:0/2633674989 conn(0x7fd8e00fbc80 msgr2=0x7fd8e0104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:50.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.760+0000 7fd8e4e9a640 1 -- 172.21.15.70:0/2633674989 shutdown_connections 2024-04-01T01:30:50.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:50.760+0000 7fd8e4e9a640 1 -- 172.21.15.70:0/2633674989 wait complete. 2024-04-01T01:30:51.059 INFO:journalctl@ceph.mds.e.smithi081.stdout:Apr 01 01:30:50 smithi081 systemd[1]: Starting Ceph mds.e for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:30:51.322 INFO:journalctl@ceph.mds.e.smithi081.stdout:Apr 01 01:30:51 smithi081 podman[64356]: 2024-04-01 01:30:50.962434352 +0000 UTC m=+0.017085653 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:30:51.322 INFO:journalctl@ceph.mds.e.smithi081.stdout:Apr 01 01:30:51 smithi081 podman[64356]: 2024-04-01 01:30:51.086814589 +0000 UTC m=+0.141465898 container create bb9382b3a3c19b332ff7366629790af05061359e0ba49381280c41fe3206c33c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-04-01T01:30:51.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:51 smithi081 ceph-mon[35142]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:51.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:51 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2633674989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:51.581 INFO:journalctl@ceph.mds.e.smithi081.stdout:Apr 01 01:30:51 smithi081 podman[64356]: 2024-04-01 01:30:51.322980877 +0000 UTC m=+0.377632178 container init bb9382b3a3c19b332ff7366629790af05061359e0ba49381280c41fe3206c33c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-e, RELEASE=squid-384f31f, GIT_CLEAN=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-01T01:30:51.582 INFO:journalctl@ceph.mds.e.smithi081.stdout:Apr 01 01:30:51 smithi081 podman[64356]: 2024-04-01 01:30:51.325731009 +0000 UTC m=+0.380382312 container start bb9382b3a3c19b332ff7366629790af05061359e0ba49381280c41fe3206c33c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-01T01:30:51.582 INFO:journalctl@ceph.mds.e.smithi081.stdout:Apr 01 01:30:51 smithi081 ceph-mds[64376]: -- 172.21.15.81:0/2364902989 <== mon.2 v2:172.21.15.81:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561260d37a40 con 0x561260d0b000 2024-04-01T01:30:51.582 INFO:journalctl@ceph.mds.e.smithi081.stdout:Apr 01 01:30:51 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-e[64372]: starting mds.e at 2024-04-01T01:30:51.582 INFO:journalctl@ceph.mds.e.smithi081.stdout:Apr 01 01:30:51 smithi081 bash[64356]: bb9382b3a3c19b332ff7366629790af05061359e0ba49381280c41fe3206c33c 2024-04-01T01:30:51.582 INFO:journalctl@ceph.mds.e.smithi081.stdout:Apr 01 01:30:51 smithi081 systemd[1]: Started Ceph mds.e for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:30:51.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:51 smithi189 ceph-mon[35753]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:51.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:51 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2633674989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:51.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:51 smithi070 ceph-mon[32179]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:51.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:51 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2633674989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:52.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:52 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:52.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:52 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] up:boot 2024-04-01T01:30:52.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:52 smithi189 ceph-mon[35753]: fsmap 5 up:standby 2024-04-01T01:30:52.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:52 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-01T01:30:52.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:52 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:52.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:52 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:52.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:52 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:52.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:52 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:52.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:52 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:52.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:52 smithi189 ceph-mon[35753]: Deploying daemon mds.f on smithi189 2024-04-01T01:30:52.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:52 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:52.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:52 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] up:boot 2024-04-01T01:30:52.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:52 smithi070 ceph-mon[32179]: fsmap 5 up:standby 2024-04-01T01:30:52.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:52 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-01T01:30:52.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:52 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:52.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:52 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:52.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:52 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:52.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:52 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:52.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:52 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:52.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:52 smithi070 ceph-mon[32179]: Deploying daemon mds.f on smithi189 2024-04-01T01:30:52.772 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:30:52.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:52 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:52.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:52 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] up:boot 2024-04-01T01:30:52.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:52 smithi081 ceph-mon[35142]: fsmap 5 up:standby 2024-04-01T01:30:52.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:52 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-01T01:30:52.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:52 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:52.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:52 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:52.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:52 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:52.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:52 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:52.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:52 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:52.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:52 smithi081 ceph-mon[35142]: Deploying daemon mds.f on smithi189 2024-04-01T01:30:52.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.885+0000 7f32789c2640 1 -- 172.21.15.70:0/1312484814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3274102240 msgr2=0x7f3274108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.885+0000 7f32789c2640 1 --2- 172.21.15.70:0/1312484814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3274102240 0x7f3274108ad0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f326c009ca0 tx=0x7f326c02c0f0 comp rx=0 tx=0).stop 2024-04-01T01:30:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.885+0000 7f32789c2640 1 -- 172.21.15.70:0/1312484814 shutdown_connections 2024-04-01T01:30:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.885+0000 7f32789c2640 1 --2- 172.21.15.70:0/1312484814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3274102240 0x7f3274108ad0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.885+0000 7f32789c2640 1 --2- 172.21.15.70:0/1312484814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3274101880 0x7f3274101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.885+0000 7f32789c2640 1 --2- 172.21.15.70:0/1312484814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3274100690 0x7f3274100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.885+0000 7f32789c2640 1 -- 172.21.15.70:0/1312484814 >> 172.21.15.70:0/1312484814 conn(0x7f32740fbc00 msgr2=0x7f32740fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.885+0000 7f32789c2640 1 -- 172.21.15.70:0/1312484814 shutdown_connections 2024-04-01T01:30:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.885+0000 7f32789c2640 1 -- 172.21.15.70:0/1312484814 wait complete. 2024-04-01T01:30:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f32789c2640 1 Processor -- start 2024-04-01T01:30:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f32789c2640 1 -- start start 2024-04-01T01:30:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f32789c2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3274100690 0x7f3274199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f32789c2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3274101880 0x7f327419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f32789c2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3274102240 0x7f32741a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f32789c2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f327410c1c0 con 0x7f3274102240 2024-04-01T01:30:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f32789c2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f327410c040 con 0x7f3274101880 2024-04-01T01:30:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f32789c2640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f327410c340 con 0x7f3274100690 2024-04-01T01:30:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f3272d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3274102240 0x7f32741a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f3272d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3274102240 0x7f32741a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36584/0 (socket says 172.21.15.70:36584) 2024-04-01T01:30:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f3272d9d640 1 -- 172.21.15.70:0/2678658607 learned_addr learned my addr 172.21.15.70:0/2678658607 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:30:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f327259c640 1 --2- 172.21.15.70:0/2678658607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3274100690 0x7f3274199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f3272d9d640 1 -- 172.21.15.70:0/2678658607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3274100690 msgr2=0x7f3274199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f3272d9d640 1 --2- 172.21.15.70:0/2678658607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3274100690 0x7f3274199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f326bfff640 1 --2- 172.21.15.70:0/2678658607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3274101880 0x7f327419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f3272d9d640 1 -- 172.21.15.70:0/2678658607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3274101880 msgr2=0x7f327419a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f3272d9d640 1 --2- 172.21.15.70:0/2678658607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3274101880 0x7f327419a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f3272d9d640 1 -- 172.21.15.70:0/2678658607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32741a17e0 con 0x7f3274102240 2024-04-01T01:30:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.886+0000 7f327259c640 1 --2- 172.21.15.70:0/2678658607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3274100690 0x7f3274199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:30:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.887+0000 7f3272d9d640 1 --2- 172.21.15.70:0/2678658607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3274102240 0x7f32741a1170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f326c02c700 tx=0x7f326c0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.887+0000 7f326b7fe640 1 -- 172.21.15.70:0/2678658607 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c041020 con 0x7f3274102240 2024-04-01T01:30:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.887+0000 7f326b7fe640 1 -- 172.21.15.70:0/2678658607 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f326c003e20 con 0x7f3274102240 2024-04-01T01:30:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.887+0000 7f32789c2640 1 -- 172.21.15.70:0/2678658607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32741a1a70 con 0x7f3274102240 2024-04-01T01:30:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.887+0000 7f326b7fe640 1 -- 172.21.15.70:0/2678658607 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c039040 con 0x7f3274102240 2024-04-01T01:30:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.887+0000 7f32789c2640 1 -- 172.21.15.70:0/2678658607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32741a1fb0 con 0x7f3274102240 2024-04-01T01:30:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.888+0000 7f32789c2640 1 -- 172.21.15.70:0/2678658607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32741a21e0 con 0x7f3274102240 2024-04-01T01:30:52.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.889+0000 7f326b7fe640 1 -- 172.21.15.70:0/2678658607 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f326c044020 con 0x7f3274102240 2024-04-01T01:30:52.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.889+0000 7f326b7fe640 1 --2- 172.21.15.70:0/2678658607 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32600a2520 0x7f32600a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:52.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.890+0000 7f327259c640 1 --2- 172.21.15.70:0/2678658607 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32600a2520 0x7f32600a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:52.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.890+0000 7f326b7fe640 1 -- 172.21.15.70:0/2678658607 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f326c0f05f0 con 0x7f3274102240 2024-04-01T01:30:52.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.890+0000 7f327259c640 1 --2- 172.21.15.70:0/2678658607 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32600a2520 0x7f32600a49e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3264005d00 tx=0x7f3264005c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:52.894+0000 7f326b7fe640 1 -- 172.21.15.70:0/2678658607 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f326c04b020 con 0x7f3274102240 2024-04-01T01:30:53.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:53.106+0000 7f32789c2640 1 -- 172.21.15.70:0/2678658607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32741a23b0 con 0x7f3274102240 2024-04-01T01:30:53.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:53.106+0000 7f326b7fe640 1 -- 172.21.15.70:0/2678658607 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4687 (secure 0 0 0) 0x7f326c03b2f0 con 0x7f3274102240 2024-04-01T01:30:53.107 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:30:53.107 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-01T01:30:53.107 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 6 2024-04-01T01:30:53.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:53.108+0000 7f32789c2640 1 -- 172.21.15.70:0/2678658607 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32600a2520 msgr2=0x7f32600a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:53.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:53.108+0000 7f32789c2640 1 --2- 172.21.15.70:0/2678658607 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32600a2520 0x7f32600a49e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3264005d00 tx=0x7f3264005c90 comp rx=0 tx=0).stop 2024-04-01T01:30:53.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:53.108+0000 7f32789c2640 1 -- 172.21.15.70:0/2678658607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3274102240 msgr2=0x7f32741a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:53.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:53.108+0000 7f32789c2640 1 --2- 172.21.15.70:0/2678658607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3274102240 0x7f32741a1170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f326c02c700 tx=0x7f326c0328a0 comp rx=0 tx=0).stop 2024-04-01T01:30:53.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:53.109+0000 7f32789c2640 1 -- 172.21.15.70:0/2678658607 shutdown_connections 2024-04-01T01:30:53.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:53.109+0000 7f32789c2640 1 --2- 172.21.15.70:0/2678658607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3274102240 0x7f32741a1170 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:53.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:53.109+0000 7f32789c2640 1 --2- 172.21.15.70:0/2678658607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3274101880 0x7f327419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:53.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:53.109+0000 7f32789c2640 1 --2- 172.21.15.70:0/2678658607 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32600a2520 0x7f32600a49e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:53.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:53.109+0000 7f32789c2640 1 --2- 172.21.15.70:0/2678658607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3274100690 0x7f3274199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:53.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:53.109+0000 7f32789c2640 1 -- 172.21.15.70:0/2678658607 >> 172.21.15.70:0/2678658607 conn(0x7f32740fbc00 msgr2=0x7f32740fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:53.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:53.109+0000 7f32789c2640 1 -- 172.21.15.70:0/2678658607 shutdown_connections 2024-04-01T01:30:53.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:53.109+0000 7f32789c2640 1 -- 172.21.15.70:0/2678658607 wait complete. 2024-04-01T01:30:53.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:53 smithi189 ceph-mon[35753]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:53.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:53 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2678658607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:53.749 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 01 01:30:53 smithi189 systemd[1]: Starting Ceph mds.f for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:30:53.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:53 smithi070 ceph-mon[32179]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:53.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:53 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2678658607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:53.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:53 smithi081 ceph-mon[35142]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:53.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:53 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2678658607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:54.073 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 01 01:30:53 smithi189 podman[66024]: 2024-04-01 01:30:53.674173078 +0000 UTC m=+0.018264540 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:30:54.073 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 01 01:30:53 smithi189 podman[66024]: 2024-04-01 01:30:53.787817379 +0000 UTC m=+0.131908874 container create bc506a7755a36b0cebf137aa48255bd119326e394e5511efe6891ec7080db8d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-f, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-01T01:30:54.073 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 01 01:30:54 smithi189 podman[66024]: 2024-04-01 01:30:54.073941278 +0000 UTC m=+0.418032741 container init bc506a7755a36b0cebf137aa48255bd119326e394e5511efe6891ec7080db8d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-f, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-01T01:30:54.499 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 01 01:30:54 smithi189 podman[66024]: 2024-04-01 01:30:54.076646928 +0000 UTC m=+0.420738388 container start bc506a7755a36b0cebf137aa48255bd119326e394e5511efe6891ec7080db8d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-01T01:30:54.499 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 01 01:30:54 smithi189 ceph-mds[66043]: -- 172.21.15.189:0/2582787716 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b7b5d45a40 con 0x55b7b5d18c00 2024-04-01T01:30:54.499 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 01 01:30:54 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-f[66039]: starting mds.f at 2024-04-01T01:30:54.499 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 01 01:30:54 smithi189 bash[66024]: bc506a7755a36b0cebf137aa48255bd119326e394e5511efe6891ec7080db8d8 2024-04-01T01:30:54.499 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 01 01:30:54 smithi189 systemd[1]: Started Ceph mds.f for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:30:55.121 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:30:55.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.235+0000 7f8815b13640 1 -- 172.21.15.70:0/3982392978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810104b20 msgr2=0x7f8810106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:55.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.235+0000 7f8815b13640 1 --2- 172.21.15.70:0/3982392978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810104b20 0x7f8810106f10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8800009900 tx=0x7f880002bc30 comp rx=0 tx=0).stop 2024-04-01T01:30:55.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.235+0000 7f8815b13640 1 -- 172.21.15.70:0/3982392978 shutdown_connections 2024-04-01T01:30:55.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.235+0000 7f8815b13640 1 --2- 172.21.15.70:0/3982392978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8810107450 0x7f88101098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:55.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.235+0000 7f8815b13640 1 --2- 172.21.15.70:0/3982392978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810104b20 0x7f8810106f10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:55.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.235+0000 7f8815b13640 1 --2- 172.21.15.70:0/3982392978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88101021f0 0x7f88101045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:55.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.235+0000 7f8815b13640 1 -- 172.21.15.70:0/3982392978 >> 172.21.15.70:0/3982392978 conn(0x7f88100fbbe0 msgr2=0x7f88100fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:55.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.235+0000 7f8815b13640 1 -- 172.21.15.70:0/3982392978 shutdown_connections 2024-04-01T01:30:55.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.235+0000 7f8815b13640 1 -- 172.21.15.70:0/3982392978 wait complete. 2024-04-01T01:30:55.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.235+0000 7f8815b13640 1 Processor -- start 2024-04-01T01:30:55.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f8815b13640 1 -- start start 2024-04-01T01:30:55.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f8815b13640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88101021f0 0x7f8810197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:55.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f8815b13640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810104b20 0x7f8810197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:55.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f8815b13640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8810107450 0x7f881019ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:55.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f8815b13640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f881010d2e0 con 0x7f8810104b20 2024-04-01T01:30:55.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f8815b13640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f881010d160 con 0x7f8810107450 2024-04-01T01:30:55.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f8815b13640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f881010d460 con 0x7f88101021f0 2024-04-01T01:30:55.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f880effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810104b20 0x7f8810197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:55.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f880effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810104b20 0x7f8810197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36602/0 (socket says 172.21.15.70:36602) 2024-04-01T01:30:55.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f880effd640 1 -- 172.21.15.70:0/341971137 learned_addr learned my addr 172.21.15.70:0/341971137 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:30:55.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f880f7fe640 1 --2- 172.21.15.70:0/341971137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88101021f0 0x7f8810197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:55.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f880ffff640 1 --2- 172.21.15.70:0/341971137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8810107450 0x7f881019ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:55.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f880effd640 1 -- 172.21.15.70:0/341971137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88101021f0 msgr2=0x7f8810197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:55.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f880effd640 1 --2- 172.21.15.70:0/341971137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88101021f0 0x7f8810197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:55.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f880effd640 1 -- 172.21.15.70:0/341971137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8810107450 msgr2=0x7f881019ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:55.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f880effd640 1 --2- 172.21.15.70:0/341971137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8810107450 0x7f881019ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:55.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f880effd640 1 -- 172.21.15.70:0/341971137 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881019f580 con 0x7f8810104b20 2024-04-01T01:30:55.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f880effd640 1 --2- 172.21.15.70:0/341971137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810104b20 0x7f8810197ea0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8800005ab0 tx=0x7f88000042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:55.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f880cff9640 1 -- 172.21.15.70:0/341971137 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8800043020 con 0x7f8810104b20 2024-04-01T01:30:55.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f880cff9640 1 -- 172.21.15.70:0/341971137 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f88000054c0 con 0x7f8810104b20 2024-04-01T01:30:55.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f8815b13640 1 -- 172.21.15.70:0/341971137 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f881019f810 con 0x7f8810104b20 2024-04-01T01:30:55.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.236+0000 7f880cff9640 1 -- 172.21.15.70:0/341971137 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8800003d10 con 0x7f8810104b20 2024-04-01T01:30:55.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.237+0000 7f8815b13640 1 -- 172.21.15.70:0/341971137 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f881019fd30 con 0x7f8810104b20 2024-04-01T01:30:55.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.238+0000 7f8815b13640 1 -- 172.21.15.70:0/341971137 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88100fd6d0 con 0x7f8810104b20 2024-04-01T01:30:55.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.238+0000 7f880cff9640 1 -- 172.21.15.70:0/341971137 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f8800046020 con 0x7f8810104b20 2024-04-01T01:30:55.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.239+0000 7f880cff9640 1 --2- 172.21.15.70:0/341971137 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f87f80a24a0 0x7f87f80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:55.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.239+0000 7f880f7fe640 1 --2- 172.21.15.70:0/341971137 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f87f80a24a0 0x7f87f80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:55.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.239+0000 7f880cff9640 1 -- 172.21.15.70:0/341971137 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f88000f0920 con 0x7f8810104b20 2024-04-01T01:30:55.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.240+0000 7f880f7fe640 1 --2- 172.21.15.70:0/341971137 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f87f80a24a0 0x7f87f80a4960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f880400abe0 tx=0x7f880400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:55.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.243+0000 7f880cff9640 1 -- 172.21.15.70:0/341971137 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f880004d020 con 0x7f8810104b20 2024-04-01T01:30:55.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.453+0000 7f8815b13640 1 -- 172.21.15.70:0/341971137 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f881002d320 con 0x7f8810104b20 2024-04-01T01:30:55.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.454+0000 7f880cff9640 1 -- 172.21.15.70:0/341971137 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5515 (secure 0 0 0) 0x7f88000be0a0 con 0x7f8810104b20 2024-04-01T01:30:55.454 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:30:55.455 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24418,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-01T01:30:55.455 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 7 2024-04-01T01:30:55.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.456+0000 7f8815b13640 1 -- 172.21.15.70:0/341971137 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f87f80a24a0 msgr2=0x7f87f80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:55.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.456+0000 7f8815b13640 1 --2- 172.21.15.70:0/341971137 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f87f80a24a0 0x7f87f80a4960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f880400abe0 tx=0x7f880400a2d0 comp rx=0 tx=0).stop 2024-04-01T01:30:55.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.456+0000 7f8815b13640 1 -- 172.21.15.70:0/341971137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810104b20 msgr2=0x7f8810197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:55.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.456+0000 7f8815b13640 1 --2- 172.21.15.70:0/341971137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810104b20 0x7f8810197ea0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8800005ab0 tx=0x7f88000042f0 comp rx=0 tx=0).stop 2024-04-01T01:30:55.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.456+0000 7f8815b13640 1 -- 172.21.15.70:0/341971137 shutdown_connections 2024-04-01T01:30:55.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.456+0000 7f8815b13640 1 --2- 172.21.15.70:0/341971137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8810107450 0x7f881019ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:55.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.456+0000 7f8815b13640 1 --2- 172.21.15.70:0/341971137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8810104b20 0x7f8810197ea0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:55.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.456+0000 7f8815b13640 1 --2- 172.21.15.70:0/341971137 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f87f80a24a0 0x7f87f80a4960 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:55.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.456+0000 7f8815b13640 1 --2- 172.21.15.70:0/341971137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88101021f0 0x7f8810197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:55.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.456+0000 7f8815b13640 1 -- 172.21.15.70:0/341971137 >> 172.21.15.70:0/341971137 conn(0x7f88100fbbe0 msgr2=0x7f8810102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:55.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.456+0000 7f8815b13640 1 -- 172.21.15.70:0/341971137 shutdown_connections 2024-04-01T01:30:55.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:55.456+0000 7f8815b13640 1 -- 172.21.15.70:0/341971137 wait complete. 2024-04-01T01:30:55.466 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:55.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:55.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:55.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:55.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:55.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:55.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:55 smithi070 ceph-mon[32179]: Deploying daemon mds.c on smithi189 2024-04-01T01:30:55.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:55 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] up:boot 2024-04-01T01:30:55.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:55 smithi070 ceph-mon[32179]: fsmap 6 up:standby 2024-04-01T01:30:55.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-01T01:30:55.467 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:55 smithi070 ceph-mon[32179]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:55.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:55.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:55.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:55.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:55.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:55.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:55.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:55 smithi189 ceph-mon[35753]: Deploying daemon mds.c on smithi189 2024-04-01T01:30:55.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:55 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] up:boot 2024-04-01T01:30:55.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:55 smithi189 ceph-mon[35753]: fsmap 6 up:standby 2024-04-01T01:30:55.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-01T01:30:55.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:55 smithi189 ceph-mon[35753]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:55.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:55.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:55.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:55.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:55.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:55.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:55.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:55 smithi081 ceph-mon[35142]: Deploying daemon mds.c on smithi189 2024-04-01T01:30:55.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:55 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] up:boot 2024-04-01T01:30:55.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:55 smithi081 ceph-mon[35142]: fsmap 6 up:standby 2024-04-01T01:30:55.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-01T01:30:55.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:55 smithi081 ceph-mon[35142]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:56.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:56 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/341971137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:56.499 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:30:56 smithi189 systemd[1]: Starting Ceph mds.c for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:30:56.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:56 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/341971137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:56.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:56 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/341971137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:56.828 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:30:56 smithi189 podman[66326]: 2024-04-01 01:30:56.465462154 +0000 UTC m=+0.018124836 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:30:56.828 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:30:56 smithi189 podman[66326]: 2024-04-01 01:30:56.593284565 +0000 UTC m=+0.145947235 container create a3967370277d3cb31d2b0c6101c9ea16119523a9a1d3a1b93dc802fc808fc0b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240326) 2024-04-01T01:30:56.828 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:30:56 smithi189 podman[66326]: 2024-04-01 01:30:56.82926815 +0000 UTC m=+0.381930833 container init a3967370277d3cb31d2b0c6101c9ea16119523a9a1d3a1b93dc802fc808fc0b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-04-01T01:30:57.222 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:30:56 smithi189 podman[66326]: 2024-04-01 01:30:56.831856152 +0000 UTC m=+0.384518831 container start a3967370277d3cb31d2b0c6101c9ea16119523a9a1d3a1b93dc802fc808fc0b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-c, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-01T01:30:57.222 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:30:56 smithi189 ceph-mds[66346]: -- 172.21.15.189:0/1146190913 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5566c9d29a40 con 0x5566c9cfc800 2024-04-01T01:30:57.222 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:30:56 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-c[66342]: starting mds.c at 2024-04-01T01:30:57.222 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:30:56 smithi189 bash[66326]: a3967370277d3cb31d2b0c6101c9ea16119523a9a1d3a1b93dc802fc808fc0b2 2024-04-01T01:30:57.222 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:30:56 smithi189 systemd[1]: Started Ceph mds.c for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:30:57.470 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:30:57.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:57 smithi070 ceph-mon[32179]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:57.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:57 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:57.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:57 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:57.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:57 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:57.498 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:57 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:57.499 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:57 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:57.499 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:57 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:57.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:57 smithi189 ceph-mon[35753]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:57.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:57 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:57.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:57 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:57.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:57 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:57.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:57 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:57.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:57 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:57.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:57 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:57.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:57 smithi081 ceph-mon[35142]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:57.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:57 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:57.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:57 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:57.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:57 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:30:57.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:57 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:30:57.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:57 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:30:57.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:57 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:30:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.584+0000 7f94334d4640 1 -- 172.21.15.70:0/2788998227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f942c100660 msgr2=0x7f942c100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.584+0000 7f94334d4640 1 --2- 172.21.15.70:0/2788998227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f942c100660 0x7f942c100a80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9428009900 tx=0x7f942802bc30 comp rx=0 tx=0).stop 2024-04-01T01:30:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.584+0000 7f94334d4640 1 -- 172.21.15.70:0/2788998227 shutdown_connections 2024-04-01T01:30:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.584+0000 7f94334d4640 1 --2- 172.21.15.70:0/2788998227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f942c102210 0x7f942c108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.584+0000 7f94334d4640 1 --2- 172.21.15.70:0/2788998227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f942c101850 0x7f942c101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.584+0000 7f94334d4640 1 --2- 172.21.15.70:0/2788998227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f942c100660 0x7f942c100a80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.584+0000 7f94334d4640 1 -- 172.21.15.70:0/2788998227 >> 172.21.15.70:0/2788998227 conn(0x7f942c0fbbd0 msgr2=0x7f942c0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.584+0000 7f94334d4640 1 -- 172.21.15.70:0/2788998227 shutdown_connections 2024-04-01T01:30:57.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.584+0000 7f94334d4640 1 -- 172.21.15.70:0/2788998227 wait complete. 2024-04-01T01:30:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f94334d4640 1 Processor -- start 2024-04-01T01:30:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f94334d4640 1 -- start start 2024-04-01T01:30:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f94334d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f942c100660 0x7f942c1979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f94334d4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f942c101850 0x7f942c197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f94334d4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f942c102210 0x7f942c19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f94334d4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f942c10c190 con 0x7f942c101850 2024-04-01T01:30:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f94334d4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f942c10c010 con 0x7f942c100660 2024-04-01T01:30:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f94334d4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f942c10c310 con 0x7f942c102210 2024-04-01T01:30:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f9430a6f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f942c101850 0x7f942c197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f9430a6f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f942c101850 0x7f942c197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36816/0 (socket says 172.21.15.70:36816) 2024-04-01T01:30:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f9430a6f640 1 -- 172.21.15.70:0/2867399432 learned_addr learned my addr 172.21.15.70:0/2867399432 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:30:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f9431270640 1 --2- 172.21.15.70:0/2867399432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f942c100660 0x7f942c1979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:57.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f9431a71640 1 --2- 172.21.15.70:0/2867399432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f942c102210 0x7f942c19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f9430a6f640 1 -- 172.21.15.70:0/2867399432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f942c102210 msgr2=0x7f942c19ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f9430a6f640 1 --2- 172.21.15.70:0/2867399432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f942c102210 0x7f942c19ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f9430a6f640 1 -- 172.21.15.70:0/2867399432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f942c100660 msgr2=0x7f942c1979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f9430a6f640 1 --2- 172.21.15.70:0/2867399432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f942c100660 0x7f942c1979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f9430a6f640 1 -- 172.21.15.70:0/2867399432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942c19f5e0 con 0x7f942c101850 2024-04-01T01:30:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.585+0000 7f9430a6f640 1 --2- 172.21.15.70:0/2867399432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f942c101850 0x7f942c197f00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f941c00e9f0 tx=0x7f941c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.586+0000 7f94227fc640 1 -- 172.21.15.70:0/2867399432 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941c00cd80 con 0x7f942c101850 2024-04-01T01:30:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.586+0000 7f94227fc640 1 -- 172.21.15.70:0/2867399432 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f941c0108e0 con 0x7f942c101850 2024-04-01T01:30:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.586+0000 7f94227fc640 1 -- 172.21.15.70:0/2867399432 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941c018920 con 0x7f942c101850 2024-04-01T01:30:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.586+0000 7f94334d4640 1 -- 172.21.15.70:0/2867399432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f942c19f8d0 con 0x7f942c101850 2024-04-01T01:30:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.586+0000 7f94334d4640 1 -- 172.21.15.70:0/2867399432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f942c19fe70 con 0x7f942c101850 2024-04-01T01:30:57.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.587+0000 7f94334d4640 1 -- 172.21.15.70:0/2867399432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f942c1a00a0 con 0x7f942c101850 2024-04-01T01:30:57.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.588+0000 7f94227fc640 1 -- 172.21.15.70:0/2867399432 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f941c010a90 con 0x7f942c101850 2024-04-01T01:30:57.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.588+0000 7f94227fc640 1 --2- 172.21.15.70:0/2867399432 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94180a2520 0x7f94180a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:57.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.589+0000 7f9431270640 1 --2- 172.21.15.70:0/2867399432 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94180a2520 0x7f94180a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:57.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.589+0000 7f94227fc640 1 -- 172.21.15.70:0/2867399432 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f941c0ceaf0 con 0x7f942c101850 2024-04-01T01:30:57.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.589+0000 7f9431270640 1 --2- 172.21.15.70:0/2867399432 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94180a2520 0x7f94180a49e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9428006010 tx=0x7f9428005be0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:57.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.592+0000 7f94227fc640 1 -- 172.21.15.70:0/2867399432 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f941c09c130 con 0x7f942c101850 2024-04-01T01:30:57.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.803+0000 7f94334d4640 1 -- 172.21.15.70:0/2867399432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f942c1a0270 con 0x7f942c101850 2024-04-01T01:30:57.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.803+0000 7f94227fc640 1 -- 172.21.15.70:0/2867399432 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6343 (secure 0 0 0) 0x7f941c09ba30 con 0x7f942c101850 2024-04-01T01:30:57.804 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:30:57.804 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24418,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-01T01:30:57.804 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 8 2024-04-01T01:30:57.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.806+0000 7f94334d4640 1 -- 172.21.15.70:0/2867399432 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94180a2520 msgr2=0x7f94180a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:57.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.806+0000 7f94334d4640 1 --2- 172.21.15.70:0/2867399432 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94180a2520 0x7f94180a49e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9428006010 tx=0x7f9428005be0 comp rx=0 tx=0).stop 2024-04-01T01:30:57.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.806+0000 7f94334d4640 1 -- 172.21.15.70:0/2867399432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f942c101850 msgr2=0x7f942c197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:57.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.806+0000 7f94334d4640 1 --2- 172.21.15.70:0/2867399432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f942c101850 0x7f942c197f00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f941c00e9f0 tx=0x7f941c00eeb0 comp rx=0 tx=0).stop 2024-04-01T01:30:57.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.806+0000 7f94334d4640 1 -- 172.21.15.70:0/2867399432 shutdown_connections 2024-04-01T01:30:57.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.806+0000 7f94334d4640 1 --2- 172.21.15.70:0/2867399432 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94180a2520 0x7f94180a49e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:57.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.806+0000 7f94334d4640 1 --2- 172.21.15.70:0/2867399432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f942c102210 0x7f942c19ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:57.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.806+0000 7f94334d4640 1 --2- 172.21.15.70:0/2867399432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f942c101850 0x7f942c197f00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:57.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.806+0000 7f94334d4640 1 --2- 172.21.15.70:0/2867399432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f942c100660 0x7f942c1979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:57.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.806+0000 7f94334d4640 1 -- 172.21.15.70:0/2867399432 >> 172.21.15.70:0/2867399432 conn(0x7f942c0fbbd0 msgr2=0x7f942c069d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:57.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.806+0000 7f94334d4640 1 -- 172.21.15.70:0/2867399432 shutdown_connections 2024-04-01T01:30:57.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:57.806+0000 7f94334d4640 1 -- 172.21.15.70:0/2867399432 wait complete. 2024-04-01T01:30:58.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:58 smithi081 ceph-mon[35142]: Deploying daemon mds.h on smithi081 2024-04-01T01:30:58.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:58 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.189:6836/2296387110,v1:172.21.15.189:6837/2296387110] up:boot 2024-04-01T01:30:58.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:58 smithi081 ceph-mon[35142]: fsmap 7 up:standby 2024-04-01T01:30:58.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:58 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-01T01:30:58.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:58 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2867399432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:58.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:58 smithi189 ceph-mon[35753]: Deploying daemon mds.h on smithi081 2024-04-01T01:30:58.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:58 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.189:6836/2296387110,v1:172.21.15.189:6837/2296387110] up:boot 2024-04-01T01:30:58.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:58 smithi189 ceph-mon[35753]: fsmap 7 up:standby 2024-04-01T01:30:58.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:58 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-01T01:30:58.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:58 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2867399432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:58.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:58 smithi070 ceph-mon[32179]: Deploying daemon mds.h on smithi081 2024-04-01T01:30:58.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:58 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.189:6836/2296387110,v1:172.21.15.189:6837/2296387110] up:boot 2024-04-01T01:30:58.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:58 smithi070 ceph-mon[32179]: fsmap 7 up:standby 2024-04-01T01:30:58.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:58 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-01T01:30:58.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:58 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2867399432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:30:59.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:30:59 smithi081 ceph-mon[35142]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:59.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:30:59 smithi189 ceph-mon[35753]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:59.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:30:59 smithi070 ceph-mon[32179]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:30:59.818 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:30:59.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.934+0000 7facecf3e640 1 -- 172.21.15.70:0/2377262870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7face8100670 msgr2=0x7face8100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:59.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.934+0000 7facecf3e640 1 --2- 172.21.15.70:0/2377262870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7face8100670 0x7face8100a90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7facd8009900 tx=0x7facd802bc30 comp rx=0 tx=0).stop 2024-04-01T01:30:59.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.935+0000 7facecf3e640 1 -- 172.21.15.70:0/2377262870 shutdown_connections 2024-04-01T01:30:59.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.935+0000 7facecf3e640 1 --2- 172.21.15.70:0/2377262870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7face8102220 0x7face8108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:59.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.935+0000 7facecf3e640 1 --2- 172.21.15.70:0/2377262870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face8101860 0x7face8101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:59.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.935+0000 7facecf3e640 1 --2- 172.21.15.70:0/2377262870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7face8100670 0x7face8100a90 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:59.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.935+0000 7facecf3e640 1 -- 172.21.15.70:0/2377262870 >> 172.21.15.70:0/2377262870 conn(0x7face80fbbe0 msgr2=0x7face80fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:30:59.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.935+0000 7facecf3e640 1 -- 172.21.15.70:0/2377262870 shutdown_connections 2024-04-01T01:30:59.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.935+0000 7facecf3e640 1 -- 172.21.15.70:0/2377262870 wait complete. 2024-04-01T01:30:59.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.935+0000 7facecf3e640 1 Processor -- start 2024-04-01T01:30:59.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.935+0000 7facecf3e640 1 -- start start 2024-04-01T01:30:59.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7facecf3e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7face8100670 0x7face8197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:59.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7facecf3e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7face8101860 0x7face8197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:59.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7facecf3e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face8102220 0x7face819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:59.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7facecf3e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7face810c190 con 0x7face8101860 2024-04-01T01:30:59.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7facecf3e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7face810c010 con 0x7face8100670 2024-04-01T01:30:59.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7facecf3e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7face810c310 con 0x7face8102220 2024-04-01T01:30:59.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7face5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7face8101860 0x7face8197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:59.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7face5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7face8101860 0x7face8197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36832/0 (socket says 172.21.15.70:36832) 2024-04-01T01:30:59.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7face5d9b640 1 -- 172.21.15.70:0/2732857992 learned_addr learned my addr 172.21.15.70:0/2732857992 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:30:59.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7face6d9d640 1 --2- 172.21.15.70:0/2732857992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face8102220 0x7face819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:59.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7face659c640 1 --2- 172.21.15.70:0/2732857992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7face8100670 0x7face8197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:59.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7face5d9b640 1 -- 172.21.15.70:0/2732857992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face8102220 msgr2=0x7face819efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:59.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7face5d9b640 1 --2- 172.21.15.70:0/2732857992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face8102220 0x7face819efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:59.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7face5d9b640 1 -- 172.21.15.70:0/2732857992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7face8100670 msgr2=0x7face8197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:30:59.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7face5d9b640 1 --2- 172.21.15.70:0/2732857992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7face8100670 0x7face8197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:30:59.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7face5d9b640 1 -- 172.21.15.70:0/2732857992 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7face819f640 con 0x7face8101860 2024-04-01T01:30:59.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7face5d9b640 1 --2- 172.21.15.70:0/2732857992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7face8101860 0x7face8197f60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7facdc00e9f0 tx=0x7facdc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:59.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7facd77fe640 1 -- 172.21.15.70:0/2732857992 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facdc00cd80 con 0x7face8101860 2024-04-01T01:30:59.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7facd77fe640 1 -- 172.21.15.70:0/2732857992 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7facdc0108e0 con 0x7face8101860 2024-04-01T01:30:59.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7facecf3e640 1 -- 172.21.15.70:0/2732857992 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7face819f930 con 0x7face8101860 2024-04-01T01:30:59.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7facd77fe640 1 -- 172.21.15.70:0/2732857992 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facdc018920 con 0x7face8101860 2024-04-01T01:30:59.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.936+0000 7facecf3e640 1 -- 172.21.15.70:0/2732857992 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7face819fed0 con 0x7face8101860 2024-04-01T01:30:59.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.938+0000 7facecf3e640 1 -- 172.21.15.70:0/2732857992 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7face81a0160 con 0x7face8101860 2024-04-01T01:30:59.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.939+0000 7facd77fe640 1 -- 172.21.15.70:0/2732857992 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7facdc010a90 con 0x7face8101860 2024-04-01T01:30:59.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.939+0000 7facd77fe640 1 --2- 172.21.15.70:0/2732857992 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7facd00a2520 0x7facd00a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:30:59.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.939+0000 7face659c640 1 --2- 172.21.15.70:0/2732857992 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7facd00a2520 0x7facd00a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:30:59.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.939+0000 7facd77fe640 1 -- 172.21.15.70:0/2732857992 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7facdc0ce9a0 con 0x7face8101860 2024-04-01T01:30:59.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.940+0000 7face659c640 1 --2- 172.21.15.70:0/2732857992 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7facd00a2520 0x7facd00a49e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7facd8005ab0 tx=0x7facd8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:30:59.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:30:59.943+0000 7facd77fe640 1 -- 172.21.15.70:0/2732857992 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7facdc09b9b0 con 0x7face8101860 2024-04-01T01:31:00.078 INFO:journalctl@ceph.mds.h.smithi081.stdout:Apr 01 01:30:59 smithi081 systemd[1]: Started Ceph mds.h for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:31:00.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:00.155+0000 7facecf3e640 1 -- 172.21.15.70:0/2732857992 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7face81a0330 con 0x7face8101860 2024-04-01T01:31:00.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:00.155+0000 7facd77fe640 1 -- 172.21.15.70:0/2732857992 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6343 (secure 0 0 0) 0x7facdc09fa50 con 0x7face8101860 2024-04-01T01:31:00.156 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:00.156 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24418,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-01T01:31:00.156 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 8 2024-04-01T01:31:00.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:00.157+0000 7facecf3e640 1 -- 172.21.15.70:0/2732857992 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7facd00a2520 msgr2=0x7facd00a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:00.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:00.157+0000 7facecf3e640 1 --2- 172.21.15.70:0/2732857992 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7facd00a2520 0x7facd00a49e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7facd8005ab0 tx=0x7facd8005fd0 comp rx=0 tx=0).stop 2024-04-01T01:31:00.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:00.157+0000 7facecf3e640 1 -- 172.21.15.70:0/2732857992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7face8101860 msgr2=0x7face8197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:00.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:00.157+0000 7facecf3e640 1 --2- 172.21.15.70:0/2732857992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7face8101860 0x7face8197f60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7facdc00e9f0 tx=0x7facdc00eeb0 comp rx=0 tx=0).stop 2024-04-01T01:31:00.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:00.157+0000 7facecf3e640 1 -- 172.21.15.70:0/2732857992 shutdown_connections 2024-04-01T01:31:00.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:00.157+0000 7facecf3e640 1 --2- 172.21.15.70:0/2732857992 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7facd00a2520 0x7facd00a49e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:00.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:00.158+0000 7facecf3e640 1 --2- 172.21.15.70:0/2732857992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7face8102220 0x7face819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:00.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:00.158+0000 7facecf3e640 1 --2- 172.21.15.70:0/2732857992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7face8101860 0x7face8197f60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:00.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:00.158+0000 7facecf3e640 1 --2- 172.21.15.70:0/2732857992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7face8100670 0x7face8197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:00.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:00.158+0000 7facecf3e640 1 -- 172.21.15.70:0/2732857992 >> 172.21.15.70:0/2732857992 conn(0x7face80fbbe0 msgr2=0x7face8068920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:00.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:00.158+0000 7facecf3e640 1 -- 172.21.15.70:0/2732857992 shutdown_connections 2024-04-01T01:31:00.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:00.158+0000 7facecf3e640 1 -- 172.21.15.70:0/2732857992 wait complete. 2024-04-01T01:31:01.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:31:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:31:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:00 smithi189 ceph-mon[35753]: Deploying daemon mds.i on smithi189 2024-04-01T01:31:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:00 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2732857992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:00 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.81:6836/3527001697,v1:172.21.15.81:6837/3527001697] up:boot 2024-04-01T01:31:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:00 smithi189 ceph-mon[35753]: fsmap 8 up:standby 2024-04-01T01:31:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-01T01:31:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:00 smithi189 ceph-mon[35753]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:31:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:31:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:00 smithi070 ceph-mon[32179]: Deploying daemon mds.i on smithi189 2024-04-01T01:31:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:00 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2732857992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:00 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.81:6836/3527001697,v1:172.21.15.81:6837/3527001697] up:boot 2024-04-01T01:31:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:00 smithi070 ceph-mon[32179]: fsmap 8 up:standby 2024-04-01T01:31:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-01T01:31:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:00 smithi070 ceph-mon[32179]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:01.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:01.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:01.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:01.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:31:01.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:31:01.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:01.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:00 smithi081 ceph-mon[35142]: Deploying daemon mds.i on smithi189 2024-04-01T01:31:01.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:00 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2732857992' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:01.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:00 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.81:6836/3527001697,v1:172.21.15.81:6837/3527001697] up:boot 2024-04-01T01:31:01.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:00 smithi081 ceph-mon[35142]: fsmap 8 up:standby 2024-04-01T01:31:01.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-01T01:31:01.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:00 smithi081 ceph-mon[35142]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:02.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:31:02.249 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 01 01:31:02 smithi189 systemd[1]: Starting Ceph mds.i for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:31:02.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.283+0000 7f564bfe2640 1 -- 172.21.15.70:0/2478293809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5644105320 msgr2=0x7f56441057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:02.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.283+0000 7f564bfe2640 1 --2- 172.21.15.70:0/2478293809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5644105320 0x7f56441057a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f563c009ca0 tx=0x7f563c02c0f0 comp rx=0 tx=0).stop 2024-04-01T01:31:02.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.283+0000 7f564bfe2640 1 -- 172.21.15.70:0/2478293809 shutdown_connections 2024-04-01T01:31:02.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.283+0000 7f564bfe2640 1 --2- 172.21.15.70:0/2478293809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5644105320 0x7f56441057a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:02.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.283+0000 7f564bfe2640 1 --2- 172.21.15.70:0/2478293809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5644102970 0x7f5644104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:02.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.283+0000 7f564bfe2640 1 --2- 172.21.15.70:0/2478293809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5644100000 0x7f5644102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:02.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.283+0000 7f564bfe2640 1 -- 172.21.15.70:0/2478293809 >> 172.21.15.70:0/2478293809 conn(0x7f56440fbc00 msgr2=0x7f56440fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:02.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.283+0000 7f564bfe2640 1 -- 172.21.15.70:0/2478293809 shutdown_connections 2024-04-01T01:31:02.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.283+0000 7f564bfe2640 1 -- 172.21.15.70:0/2478293809 wait complete. 2024-04-01T01:31:02.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f564bfe2640 1 Processor -- start 2024-04-01T01:31:02.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f564bfe2640 1 -- start start 2024-04-01T01:31:02.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f564bfe2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5644100000 0x7f564419dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:02.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f564bfe2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5644102970 0x7f564419e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:02.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f564bfe2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5644105320 0x7f56441a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:02.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f564bfe2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f56441138b0 con 0x7f5644100000 2024-04-01T01:31:02.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f564bfe2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5644113730 con 0x7f5644105320 2024-04-01T01:31:02.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f564bfe2640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5644113a30 con 0x7f5644102970 2024-04-01T01:31:02.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f5649d7e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5644100000 0x7f564419dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:02.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f5649d7e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5644100000 0x7f564419dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36850/0 (socket says 172.21.15.70:36850) 2024-04-01T01:31:02.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f5649d7e640 1 -- 172.21.15.70:0/1113063226 learned_addr learned my addr 172.21.15.70:0/1113063226 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:02.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f564957d640 1 --2- 172.21.15.70:0/1113063226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5644102970 0x7f564419e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:02.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f564a57f640 1 --2- 172.21.15.70:0/1113063226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5644105320 0x7f56441a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:02.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f5649d7e640 1 -- 172.21.15.70:0/1113063226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5644102970 msgr2=0x7f564419e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:02.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f5649d7e640 1 --2- 172.21.15.70:0/1113063226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5644102970 0x7f564419e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:02.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f5649d7e640 1 -- 172.21.15.70:0/1113063226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5644105320 msgr2=0x7f56441a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:02.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f5649d7e640 1 --2- 172.21.15.70:0/1113063226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5644105320 0x7f56441a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:02.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f5649d7e640 1 -- 172.21.15.70:0/1113063226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56441a5bc0 con 0x7f5644100000 2024-04-01T01:31:02.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f564957d640 1 --2- 172.21.15.70:0/1113063226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5644102970 0x7f564419e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:31:02.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.284+0000 7f564a57f640 1 --2- 172.21.15.70:0/1113063226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5644105320 0x7f56441a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:31:02.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.285+0000 7f5649d7e640 1 --2- 172.21.15.70:0/1113063226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5644100000 0x7f564419dfa0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f564000c890 tx=0x7f564000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:02.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.285+0000 7f563affd640 1 -- 172.21.15.70:0/1113063226 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5640004220 con 0x7f5644100000 2024-04-01T01:31:02.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.285+0000 7f563affd640 1 -- 172.21.15.70:0/1113063226 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56400043c0 con 0x7f5644100000 2024-04-01T01:31:02.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.285+0000 7f564bfe2640 1 -- 172.21.15.70:0/1113063226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56441a5eb0 con 0x7f5644100000 2024-04-01T01:31:02.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.285+0000 7f563affd640 1 -- 172.21.15.70:0/1113063226 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5640004a00 con 0x7f5644100000 2024-04-01T01:31:02.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.285+0000 7f564bfe2640 1 -- 172.21.15.70:0/1113063226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56441a6320 con 0x7f5644100000 2024-04-01T01:31:02.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.286+0000 7f564bfe2640 1 -- 172.21.15.70:0/1113063226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56440fd6f0 con 0x7f5644100000 2024-04-01T01:31:02.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.287+0000 7f563affd640 1 -- 172.21.15.70:0/1113063226 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f5640004ba0 con 0x7f5644100000 2024-04-01T01:31:02.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.287+0000 7f563affd640 1 --2- 172.21.15.70:0/1113063226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56300a2520 0x7f56300a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:02.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.288+0000 7f564957d640 1 --2- 172.21.15.70:0/1113063226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56300a2520 0x7f56300a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:02.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.288+0000 7f563affd640 1 -- 172.21.15.70:0/1113063226 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f56400cdf60 con 0x7f5644100000 2024-04-01T01:31:02.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.288+0000 7f564957d640 1 --2- 172.21.15.70:0/1113063226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56300a2520 0x7f56300a49e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f564419f4b0 tx=0x7f5634005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:02.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.293+0000 7f563affd640 1 -- 172.21.15.70:0/1113063226 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5640014030 con 0x7f5644100000 2024-04-01T01:31:02.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.502+0000 7f564bfe2640 1 -- 172.21.15.70:0/1113063226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f564402d170 con 0x7f5644100000 2024-04-01T01:31:02.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.503+0000 7f563affd640 1 -- 172.21.15.70:0/1113063226 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7168 (secure 0 0 0) 0x7f564402d170 con 0x7f5644100000 2024-04-01T01:31:02.504 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:02.504 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24418,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24425,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-01T01:31:02.504 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 9 2024-04-01T01:31:02.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.506+0000 7f564bfe2640 1 -- 172.21.15.70:0/1113063226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56300a2520 msgr2=0x7f56300a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:02.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.506+0000 7f564bfe2640 1 --2- 172.21.15.70:0/1113063226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56300a2520 0x7f56300a49e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f564419f4b0 tx=0x7f5634005f90 comp rx=0 tx=0).stop 2024-04-01T01:31:02.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.506+0000 7f564bfe2640 1 -- 172.21.15.70:0/1113063226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5644100000 msgr2=0x7f564419dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:02.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.506+0000 7f564bfe2640 1 --2- 172.21.15.70:0/1113063226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5644100000 0x7f564419dfa0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f564000c890 tx=0x7f564000cd50 comp rx=0 tx=0).stop 2024-04-01T01:31:02.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.506+0000 7f564bfe2640 1 -- 172.21.15.70:0/1113063226 shutdown_connections 2024-04-01T01:31:02.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.506+0000 7f564bfe2640 1 --2- 172.21.15.70:0/1113063226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5644105320 0x7f56441a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:02.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.506+0000 7f564bfe2640 1 --2- 172.21.15.70:0/1113063226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56300a2520 0x7f56300a49e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:02.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.506+0000 7f564bfe2640 1 --2- 172.21.15.70:0/1113063226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5644102970 0x7f564419e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:02.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.506+0000 7f564bfe2640 1 --2- 172.21.15.70:0/1113063226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5644100000 0x7f564419dfa0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:02.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.506+0000 7f564bfe2640 1 -- 172.21.15.70:0/1113063226 >> 172.21.15.70:0/1113063226 conn(0x7f56440fbc00 msgr2=0x7f56441009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:02.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.506+0000 7f564bfe2640 1 -- 172.21.15.70:0/1113063226 shutdown_connections 2024-04-01T01:31:02.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:02.506+0000 7f564bfe2640 1 -- 172.21.15.70:0/1113063226 wait complete. 2024-04-01T01:31:02.556 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 01 01:31:02 smithi189 podman[66645]: 2024-04-01 01:31:02.290406037 +0000 UTC m=+0.112733350 container create 50ee14cddb4ccc2d1a892a41dccb4c27206093e031ac2b20d79052ae256b663e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-i, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f) 2024-04-01T01:31:02.556 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 01 01:31:02 smithi189 podman[66645]: 2024-04-01 01:31:02.195173591 +0000 UTC m=+0.017500886 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:31:02.556 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 01 01:31:02 smithi189 podman[66645]: 2024-04-01 01:31:02.493510828 +0000 UTC m=+0.315838100 container init 50ee14cddb4ccc2d1a892a41dccb4c27206093e031ac2b20d79052ae256b663e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-i, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True) 2024-04-01T01:31:02.556 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 01 01:31:02 smithi189 podman[66645]: 2024-04-01 01:31:02.496098299 +0000 UTC m=+0.318425570 container start 50ee14cddb4ccc2d1a892a41dccb4c27206093e031ac2b20d79052ae256b663e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-i, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-384f31f, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-01T01:31:02.556 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 01 01:31:02 smithi189 ceph-mds[66663]: -- 172.21.15.189:0/1300315231 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c450b93a40 con 0x55c450b66800 2024-04-01T01:31:02.556 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 01 01:31:02 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-i[66659]: starting mds.i at 2024-04-01T01:31:02.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:02 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1113063226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:02.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:02 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1113063226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:02.999 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 01 01:31:02 smithi189 bash[66645]: 50ee14cddb4ccc2d1a892a41dccb4c27206093e031ac2b20d79052ae256b663e 2024-04-01T01:31:02.999 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 01 01:31:02 smithi189 systemd[1]: Started Ceph mds.i for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:31:03.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:02 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1113063226' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:03.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:03 smithi081 ceph-mon[35142]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:03.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:03 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] up:boot 2024-04-01T01:31:03.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:03 smithi081 ceph-mon[35142]: fsmap 9 up:standby 2024-04-01T01:31:03.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-01T01:31:03.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:03.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:03.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:03.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:31:03.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:31:03.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:03.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:03 smithi081 ceph-mon[35142]: Deploying daemon mds.g on smithi070 2024-04-01T01:31:03.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:03 smithi189 ceph-mon[35753]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:03.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:03 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] up:boot 2024-04-01T01:31:03.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:03 smithi189 ceph-mon[35753]: fsmap 9 up:standby 2024-04-01T01:31:03.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-01T01:31:03.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:03.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:03.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:03.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:31:03.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:31:03.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:03.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:03 smithi189 ceph-mon[35753]: Deploying daemon mds.g on smithi070 2024-04-01T01:31:04.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:03 smithi070 ceph-mon[32179]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:04.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:03 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] up:boot 2024-04-01T01:31:04.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:03 smithi070 ceph-mon[32179]: fsmap 9 up:standby 2024-04-01T01:31:04.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-01T01:31:04.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:04.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:04.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:04.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:31:04.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:31:04.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:04.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:03 smithi070 ceph-mon[32179]: Deploying daemon mds.g on smithi070 2024-04-01T01:31:04.517 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:31:04.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.754+0000 7f46e11bc640 1 -- 172.21.15.70:0/2075116860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46dc100690 msgr2=0x7f46dc100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:04.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.754+0000 7f46e11bc640 1 --2- 172.21.15.70:0/2075116860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46dc100690 0x7f46dc100ab0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009900 tx=0x7f46cc02bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.754+0000 7f46e11bc640 1 -- 172.21.15.70:0/2075116860 shutdown_connections 2024-04-01T01:31:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.754+0000 7f46e11bc640 1 --2- 172.21.15.70:0/2075116860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46dc102240 0x7f46dc108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.754+0000 7f46e11bc640 1 --2- 172.21.15.70:0/2075116860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46dc101880 0x7f46dc101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.754+0000 7f46e11bc640 1 --2- 172.21.15.70:0/2075116860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46dc100690 0x7f46dc100ab0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.754+0000 7f46e11bc640 1 -- 172.21.15.70:0/2075116860 >> 172.21.15.70:0/2075116860 conn(0x7f46dc0fbc00 msgr2=0x7f46dc0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.754+0000 7f46e11bc640 1 -- 172.21.15.70:0/2075116860 shutdown_connections 2024-04-01T01:31:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.754+0000 7f46e11bc640 1 -- 172.21.15.70:0/2075116860 wait complete. 2024-04-01T01:31:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.754+0000 7f46e11bc640 1 Processor -- start 2024-04-01T01:31:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.754+0000 7f46e11bc640 1 -- start start 2024-04-01T01:31:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46e11bc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46dc100690 0x7f46dc06a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46e11bc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46dc101880 0x7f46dc19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46e11bc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46dc102240 0x7f46dc19ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:04.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46e11bc640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc10c1c0 con 0x7f46dc100690 2024-04-01T01:31:04.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46e11bc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc10c040 con 0x7f46dc102240 2024-04-01T01:31:04.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46e11bc640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f46dc10c340 con 0x7f46dc101880 2024-04-01T01:31:04.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46dad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46dc100690 0x7f46dc06a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:04.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46dad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46dc100690 0x7f46dc06a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36862/0 (socket says 172.21.15.70:36862) 2024-04-01T01:31:04.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46dad9d640 1 -- 172.21.15.70:0/78581663 learned_addr learned my addr 172.21.15.70:0/78581663 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:04.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46db59e640 1 --2- 172.21.15.70:0/78581663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46dc102240 0x7f46dc19ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:04.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46dad9d640 1 -- 172.21.15.70:0/78581663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46dc101880 msgr2=0x7f46dc19c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:04.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46dad9d640 1 --2- 172.21.15.70:0/78581663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46dc101880 0x7f46dc19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:04.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46dad9d640 1 -- 172.21.15.70:0/78581663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46dc102240 msgr2=0x7f46dc19ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:04.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46dad9d640 1 --2- 172.21.15.70:0/78581663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46dc102240 0x7f46dc19ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:04.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46dad9d640 1 -- 172.21.15.70:0/78581663 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46dc19f620 con 0x7f46dc100690 2024-04-01T01:31:04.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46dad9d640 1 --2- 172.21.15.70:0/78581663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46dc100690 0x7f46dc06a910 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f46cc000c00 tx=0x7f46cc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:04.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46d8d99640 1 -- 172.21.15.70:0/78581663 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46cc043020 con 0x7f46dc100690 2024-04-01T01:31:04.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46d8d99640 1 -- 172.21.15.70:0/78581663 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46cc004440 con 0x7f46dc100690 2024-04-01T01:31:04.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46e11bc640 1 -- 172.21.15.70:0/78581663 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46dc19f8b0 con 0x7f46dc100690 2024-04-01T01:31:04.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46d8d99640 1 -- 172.21.15.70:0/78581663 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46cc003c90 con 0x7f46dc100690 2024-04-01T01:31:04.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.755+0000 7f46e11bc640 1 -- 172.21.15.70:0/78581663 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46dc19fe00 con 0x7f46dc100690 2024-04-01T01:31:04.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.756+0000 7f46e11bc640 1 -- 172.21.15.70:0/78581663 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46dc107a30 con 0x7f46dc100690 2024-04-01T01:31:04.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.758+0000 7f46d8d99640 1 -- 172.21.15.70:0/78581663 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f46cc046020 con 0x7f46dc100690 2024-04-01T01:31:04.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.758+0000 7f46d8d99640 1 --2- 172.21.15.70:0/78581663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46c80a24a0 0x7f46c80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:04.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.758+0000 7f46d3fff640 1 --2- 172.21.15.70:0/78581663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46c80a24a0 0x7f46c80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:04.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.758+0000 7f46d8d99640 1 -- 172.21.15.70:0/78581663 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f46cc0f0890 con 0x7f46dc100690 2024-04-01T01:31:04.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.759+0000 7f46d3fff640 1 --2- 172.21.15.70:0/78581663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46c80a24a0 0x7f46c80a4960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f46c4005d00 tx=0x7f46c4005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:04.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.762+0000 7f46d8d99640 1 -- 172.21.15.70:0/78581663 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f46cc04d020 con 0x7f46dc100690 2024-04-01T01:31:04.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.993+0000 7f46e11bc640 1 -- 172.21.15.70:0/78581663 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46dc02d3e0 con 0x7f46dc100690 2024-04-01T01:31:04.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.994+0000 7f46d8d99640 1 -- 172.21.15.70:0/78581663 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7998 (secure 0 0 0) 0x7f46cc0bdef0 con 0x7f46dc100690 2024-04-01T01:31:04.993 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:04.993 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24418,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24425,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24451,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-01T01:31:04.994 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 10 2024-04-01T01:31:04.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.996+0000 7f46e11bc640 1 -- 172.21.15.70:0/78581663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46c80a24a0 msgr2=0x7f46c80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:04.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.996+0000 7f46e11bc640 1 --2- 172.21.15.70:0/78581663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46c80a24a0 0x7f46c80a4960 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f46c4005d00 tx=0x7f46c4005c50 comp rx=0 tx=0).stop 2024-04-01T01:31:04.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.996+0000 7f46e11bc640 1 -- 172.21.15.70:0/78581663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46dc100690 msgr2=0x7f46dc06a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:04.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.996+0000 7f46e11bc640 1 --2- 172.21.15.70:0/78581663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46dc100690 0x7f46dc06a910 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f46cc000c00 tx=0x7f46cc004040 comp rx=0 tx=0).stop 2024-04-01T01:31:04.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.996+0000 7f46e11bc640 1 -- 172.21.15.70:0/78581663 shutdown_connections 2024-04-01T01:31:04.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.996+0000 7f46e11bc640 1 --2- 172.21.15.70:0/78581663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46dc102240 0x7f46dc19ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:04.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.996+0000 7f46e11bc640 1 --2- 172.21.15.70:0/78581663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46c80a24a0 0x7f46c80a4960 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:04.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.996+0000 7f46e11bc640 1 --2- 172.21.15.70:0/78581663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46dc101880 0x7f46dc19c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:04.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.996+0000 7f46e11bc640 1 --2- 172.21.15.70:0/78581663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46dc100690 0x7f46dc06a910 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:04.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.996+0000 7f46e11bc640 1 -- 172.21.15.70:0/78581663 >> 172.21.15.70:0/78581663 conn(0x7f46dc0fbc00 msgr2=0x7f46dc104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:04.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.996+0000 7f46e11bc640 1 -- 172.21.15.70:0/78581663 shutdown_connections 2024-04-01T01:31:04.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:04.996+0000 7f46e11bc640 1 -- 172.21.15.70:0/78581663 wait complete. 2024-04-01T01:31:05.003 INFO:journalctl@ceph.mds.g.smithi070.stdout:Apr 01 01:31:04 smithi070 systemd[1]: Starting Ceph mds.g for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:31:05.267 INFO:journalctl@ceph.mds.g.smithi070.stdout:Apr 01 01:31:05 smithi070 podman[73731]: 2024-04-01 01:31:05.011767631 +0000 UTC m=+0.019222170 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:31:05.267 INFO:journalctl@ceph.mds.g.smithi070.stdout:Apr 01 01:31:05 smithi070 podman[73731]: 2024-04-01 01:31:05.144822789 +0000 UTC m=+0.152277306 container create c02fe3619d88e2ce29c20d7cd89b4685cd1a495355442b1595c8a28dbdffa99c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-01T01:31:05.584 INFO:journalctl@ceph.mds.g.smithi070.stdout:Apr 01 01:31:05 smithi070 podman[73731]: 2024-04-01 01:31:05.406409049 +0000 UTC m=+0.413863581 container init c02fe3619d88e2ce29c20d7cd89b4685cd1a495355442b1595c8a28dbdffa99c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-g, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-01T01:31:05.584 INFO:journalctl@ceph.mds.g.smithi070.stdout:Apr 01 01:31:05 smithi070 podman[73731]: 2024-04-01 01:31:05.409236439 +0000 UTC m=+0.416690960 container start c02fe3619d88e2ce29c20d7cd89b4685cd1a495355442b1595c8a28dbdffa99c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-g, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD) 2024-04-01T01:31:05.585 INFO:journalctl@ceph.mds.g.smithi070.stdout:Apr 01 01:31:05 smithi070 ceph-mds[73751]: -- 172.21.15.70:0/898259523 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564b69bffa40 con 0x564b69bd2c00 2024-04-01T01:31:05.585 INFO:journalctl@ceph.mds.g.smithi070.stdout:Apr 01 01:31:05 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-g[73747]: starting mds.g at 2024-04-01T01:31:05.585 INFO:journalctl@ceph.mds.g.smithi070.stdout:Apr 01 01:31:05 smithi070 bash[73731]: c02fe3619d88e2ce29c20d7cd89b4685cd1a495355442b1595c8a28dbdffa99c 2024-04-01T01:31:05.585 INFO:journalctl@ceph.mds.g.smithi070.stdout:Apr 01 01:31:05 smithi070 systemd[1]: Started Ceph mds.g for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:31:05.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:05 smithi189 ceph-mon[35753]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:05.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:05 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/78581663' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:05.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:05.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:05.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:05.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:31:05.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:31:05.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:06.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:05 smithi070 ceph-mon[32179]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:06.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:05 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/78581663' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:06.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:06.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:06.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:06.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:31:06.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:31:06.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:06.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:05 smithi081 ceph-mon[35142]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:06.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:05 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/78581663' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:06.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:06.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:06.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:06.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:31:06.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:31:06.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:06.737 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:06 smithi070 ceph-mon[32179]: Deploying daemon mds.a on smithi070 2024-04-01T01:31:06.738 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:06 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.70:6838/1626804615,v1:172.21.15.70:6839/1626804615] up:boot 2024-04-01T01:31:06.738 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:06 smithi070 ceph-mon[32179]: fsmap 10 up:standby 2024-04-01T01:31:06.738 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:06 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-01T01:31:06.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:06 smithi189 ceph-mon[35753]: Deploying daemon mds.a on smithi070 2024-04-01T01:31:06.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:06 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.70:6838/1626804615,v1:172.21.15.70:6839/1626804615] up:boot 2024-04-01T01:31:06.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:06 smithi189 ceph-mon[35753]: fsmap 10 up:standby 2024-04-01T01:31:06.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:06 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-01T01:31:07.008 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:31:07.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:06 smithi081 ceph-mon[35142]: Deploying daemon mds.a on smithi070 2024-04-01T01:31:07.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:06 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.70:6838/1626804615,v1:172.21.15.70:6839/1626804615] up:boot 2024-04-01T01:31:07.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:06 smithi081 ceph-mon[35142]: fsmap 10 up:standby 2024-04-01T01:31:07.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:06 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-01T01:31:07.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.116+0000 7f446702a640 1 -- 172.21.15.70:0/54555751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4460100670 msgr2=0x7f4460100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:07.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.116+0000 7f446702a640 1 --2- 172.21.15.70:0/54555751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4460100670 0x7f4460100a90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4454009900 tx=0x7f445402bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:07.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.116+0000 7f446702a640 1 -- 172.21.15.70:0/54555751 shutdown_connections 2024-04-01T01:31:07.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.116+0000 7f446702a640 1 --2- 172.21.15.70:0/54555751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4460102220 0x7f4460108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:07.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.116+0000 7f446702a640 1 --2- 172.21.15.70:0/54555751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4460101860 0x7f4460101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:07.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.116+0000 7f446702a640 1 --2- 172.21.15.70:0/54555751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4460100670 0x7f4460100a90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:07.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.116+0000 7f446702a640 1 -- 172.21.15.70:0/54555751 >> 172.21.15.70:0/54555751 conn(0x7f44600fbbe0 msgr2=0x7f44600fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:07.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.116+0000 7f446702a640 1 -- 172.21.15.70:0/54555751 shutdown_connections 2024-04-01T01:31:07.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.116+0000 7f446702a640 1 -- 172.21.15.70:0/54555751 wait complete. 2024-04-01T01:31:07.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.116+0000 7f446702a640 1 Processor -- start 2024-04-01T01:31:07.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f446702a640 1 -- start start 2024-04-01T01:31:07.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f446702a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4460100670 0x7f4460197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:07.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f446702a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4460101860 0x7f44601980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:07.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f446702a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4460102220 0x7f446019f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:07.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f446702a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f446010c270 con 0x7f4460101860 2024-04-01T01:31:07.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f446702a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f446010c0f0 con 0x7f4460100670 2024-04-01T01:31:07.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f446702a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f446010c3f0 con 0x7f4460102220 2024-04-01T01:31:07.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f445ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4460101860 0x7f44601980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:07.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f445ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4460101860 0x7f44601980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50862/0 (socket says 172.21.15.70:50862) 2024-04-01T01:31:07.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f445ffff640 1 -- 172.21.15.70:0/2745160725 learned_addr learned my addr 172.21.15.70:0/2745160725 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:07.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f4464dc6640 1 --2- 172.21.15.70:0/2745160725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4460100670 0x7f4460197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:07.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f445ffff640 1 -- 172.21.15.70:0/2745160725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4460102220 msgr2=0x7f446019f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:07.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f445ffff640 1 --2- 172.21.15.70:0/2745160725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4460102220 0x7f446019f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:07.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f445ffff640 1 -- 172.21.15.70:0/2745160725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4460100670 msgr2=0x7f4460197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:07.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f445ffff640 1 --2- 172.21.15.70:0/2745160725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4460100670 0x7f4460197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:07.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f445ffff640 1 -- 172.21.15.70:0/2745160725 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f446019f7b0 con 0x7f4460101860 2024-04-01T01:31:07.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f445ffff640 1 --2- 172.21.15.70:0/2745160725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4460101860 0x7f44601980d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f444c00b4a0 tx=0x7f444c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:07.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f445dffb640 1 -- 172.21.15.70:0/2745160725 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c014070 con 0x7f4460101860 2024-04-01T01:31:07.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f445dffb640 1 -- 172.21.15.70:0/2745160725 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f444c010ac0 con 0x7f4460101860 2024-04-01T01:31:07.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f446702a640 1 -- 172.21.15.70:0/2745160725 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f446019faa0 con 0x7f4460101860 2024-04-01T01:31:07.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.117+0000 7f445dffb640 1 -- 172.21.15.70:0/2745160725 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c017c30 con 0x7f4460101860 2024-04-01T01:31:07.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.118+0000 7f446702a640 1 -- 172.21.15.70:0/2745160725 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44601a0070 con 0x7f4460101860 2024-04-01T01:31:07.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.119+0000 7f446702a640 1 -- 172.21.15.70:0/2745160725 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44601a0300 con 0x7f4460101860 2024-04-01T01:31:07.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.120+0000 7f445dffb640 1 -- 172.21.15.70:0/2745160725 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f444c017070 con 0x7f4460101860 2024-04-01T01:31:07.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.120+0000 7f445dffb640 1 --2- 172.21.15.70:0/2745160725 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f44500a2520 0x7f44500a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:07.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.120+0000 7f4464dc6640 1 --2- 172.21.15.70:0/2745160725 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f44500a2520 0x7f44500a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:07.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.120+0000 7f445dffb640 1 -- 172.21.15.70:0/2745160725 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f444c0cd5b0 con 0x7f4460101860 2024-04-01T01:31:07.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.121+0000 7f4464dc6640 1 --2- 172.21.15.70:0/2745160725 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f44500a2520 0x7f44500a49e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4454009230 tx=0x7f4454005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:07.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.125+0000 7f445dffb640 1 -- 172.21.15.70:0/2745160725 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f444c09ac10 con 0x7f4460101860 2024-04-01T01:31:07.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.353+0000 7f446702a640 1 -- 172.21.15.70:0/2745160725 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44601a04d0 con 0x7f4460101860 2024-04-01T01:31:07.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.354+0000 7f445dffb640 1 -- 172.21.15.70:0/2745160725 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8824 (secure 0 0 0) 0x7f444c09a510 con 0x7f4460101860 2024-04-01T01:31:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:07.353 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24418,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24425,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24451,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-01T01:31:07.353 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 11 2024-04-01T01:31:07.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.356+0000 7f446702a640 1 -- 172.21.15.70:0/2745160725 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f44500a2520 msgr2=0x7f44500a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:07.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.356+0000 7f446702a640 1 --2- 172.21.15.70:0/2745160725 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f44500a2520 0x7f44500a49e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4454009230 tx=0x7f4454005fd0 comp rx=0 tx=0).stop 2024-04-01T01:31:07.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.356+0000 7f446702a640 1 -- 172.21.15.70:0/2745160725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4460101860 msgr2=0x7f44601980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:07.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.356+0000 7f446702a640 1 --2- 172.21.15.70:0/2745160725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4460101860 0x7f44601980d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f444c00b4a0 tx=0x7f444c00b960 comp rx=0 tx=0).stop 2024-04-01T01:31:07.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.356+0000 7f446702a640 1 -- 172.21.15.70:0/2745160725 shutdown_connections 2024-04-01T01:31:07.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.356+0000 7f446702a640 1 --2- 172.21.15.70:0/2745160725 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f44500a2520 0x7f44500a49e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:07.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.356+0000 7f446702a640 1 --2- 172.21.15.70:0/2745160725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4460102220 0x7f446019f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:07.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.356+0000 7f446702a640 1 --2- 172.21.15.70:0/2745160725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4460101860 0x7f44601980d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:07.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.356+0000 7f446702a640 1 --2- 172.21.15.70:0/2745160725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4460100670 0x7f4460197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:07.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.356+0000 7f446702a640 1 -- 172.21.15.70:0/2745160725 >> 172.21.15.70:0/2745160725 conn(0x7f44600fbbe0 msgr2=0x7f4460068960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:07.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.356+0000 7f446702a640 1 -- 172.21.15.70:0/2745160725 shutdown_connections 2024-04-01T01:31:07.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:07.356+0000 7f446702a640 1 -- 172.21.15.70:0/2745160725 wait complete. 2024-04-01T01:31:07.907 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:07 smithi070 ceph-mon[32179]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:07.907 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:07 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2745160725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:07.907 INFO:journalctl@ceph.mds.a.smithi070.stdout:Apr 01 01:31:07 smithi070 systemd[1]: Starting Ceph mds.a for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:31:07.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:07 smithi189 ceph-mon[35753]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:07.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:07 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2745160725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:08.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:07 smithi081 ceph-mon[35142]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:08.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:07 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2745160725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:08.176 INFO:journalctl@ceph.mds.a.smithi070.stdout:Apr 01 01:31:07 smithi070 podman[74094]: 2024-04-01 01:31:07.907532424 +0000 UTC m=+0.113708798 container create 08971e5b39e2304c914cf05398993d50f806a54db26a9a9db9ef812d348fd053 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-a, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-01T01:31:08.176 INFO:journalctl@ceph.mds.a.smithi070.stdout:Apr 01 01:31:07 smithi070 podman[74094]: 2024-04-01 01:31:07.811386714 +0000 UTC m=+0.017563097 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:31:08.176 INFO:journalctl@ceph.mds.a.smithi070.stdout:Apr 01 01:31:08 smithi070 podman[74094]: 2024-04-01 01:31:08.151812115 +0000 UTC m=+0.357988489 container init 08971e5b39e2304c914cf05398993d50f806a54db26a9a9db9ef812d348fd053 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-01T01:31:08.176 INFO:journalctl@ceph.mds.a.smithi070.stdout:Apr 01 01:31:08 smithi070 podman[74094]: 2024-04-01 01:31:08.154376472 +0000 UTC m=+0.360552847 container start 08971e5b39e2304c914cf05398993d50f806a54db26a9a9db9ef812d348fd053 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-a, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326) 2024-04-01T01:31:08.503 INFO:journalctl@ceph.mds.a.smithi070.stdout:Apr 01 01:31:08 smithi070 ceph-mds[74112]: -- 172.21.15.70:0/693763227 <== mon.0 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a609d59a40 con 0x55a609d2cc00 2024-04-01T01:31:08.504 INFO:journalctl@ceph.mds.a.smithi070.stdout:Apr 01 01:31:08 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-a[74108]: starting mds.a at 2024-04-01T01:31:08.504 INFO:journalctl@ceph.mds.a.smithi070.stdout:Apr 01 01:31:08 smithi070 bash[74094]: 08971e5b39e2304c914cf05398993d50f806a54db26a9a9db9ef812d348fd053 2024-04-01T01:31:08.504 INFO:journalctl@ceph.mds.a.smithi070.stdout:Apr 01 01:31:08 smithi070 systemd[1]: Started Ceph mds.a for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:31:09.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:31:09.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.485+0000 7fa398eae640 1 -- 172.21.15.70:0/4281734825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3941018e0 msgr2=0x7fa394101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:09.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.485+0000 7fa398eae640 1 --2- 172.21.15.70:0/4281734825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3941018e0 0x7fa394101d60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa388009900 tx=0x7fa38802bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:09.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.485+0000 7fa398eae640 1 -- 172.21.15.70:0/4281734825 shutdown_connections 2024-04-01T01:31:09.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.485+0000 7fa398eae640 1 --2- 172.21.15.70:0/4281734825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3941022a0 0x7fa394108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:09.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.485+0000 7fa398eae640 1 --2- 172.21.15.70:0/4281734825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3941018e0 0x7fa394101d60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:09.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.485+0000 7fa398eae640 1 --2- 172.21.15.70:0/4281734825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3941006f0 0x7fa394100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:09.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.485+0000 7fa398eae640 1 -- 172.21.15.70:0/4281734825 >> 172.21.15.70:0/4281734825 conn(0x7fa3940fbc80 msgr2=0x7fa3940fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:09.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.485+0000 7fa398eae640 1 -- 172.21.15.70:0/4281734825 shutdown_connections 2024-04-01T01:31:09.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.485+0000 7fa398eae640 1 -- 172.21.15.70:0/4281734825 wait complete. 2024-04-01T01:31:09.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.485+0000 7fa398eae640 1 Processor -- start 2024-04-01T01:31:09.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.485+0000 7fa398eae640 1 -- start start 2024-04-01T01:31:09.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa398eae640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3941006f0 0x7fa39406a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:09.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa398eae640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3941018e0 0x7fa39419c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:09.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa398eae640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3941022a0 0x7fa39419eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:09.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa398eae640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa39410c190 con 0x7fa3941018e0 2024-04-01T01:31:09.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa398eae640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa39410c010 con 0x7fa3941006f0 2024-04-01T01:31:09.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa398eae640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa39410c310 con 0x7fa3941022a0 2024-04-01T01:31:09.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa391d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3941018e0 0x7fa39419c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:09.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa391d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3941018e0 0x7fa39419c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50896/0 (socket says 172.21.15.70:50896) 2024-04-01T01:31:09.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa391d9b640 1 -- 172.21.15.70:0/681519344 learned_addr learned my addr 172.21.15.70:0/681519344 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:09.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa39259c640 1 --2- 172.21.15.70:0/681519344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3941006f0 0x7fa39406a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:09.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa391d9b640 1 -- 172.21.15.70:0/681519344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3941022a0 msgr2=0x7fa39419eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:09.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa392d9d640 1 --2- 172.21.15.70:0/681519344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3941022a0 0x7fa39419eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:09.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa391d9b640 1 --2- 172.21.15.70:0/681519344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3941022a0 0x7fa39419eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:09.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa391d9b640 1 -- 172.21.15.70:0/681519344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3941006f0 msgr2=0x7fa39406a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:09.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa391d9b640 1 --2- 172.21.15.70:0/681519344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3941006f0 0x7fa39406a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:09.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa391d9b640 1 -- 172.21.15.70:0/681519344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa39419f5b0 con 0x7fa3941018e0 2024-04-01T01:31:09.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa391d9b640 1 --2- 172.21.15.70:0/681519344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3941018e0 0x7fa39419c680 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa388006010 tx=0x7fa3880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:09.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa3837fe640 1 -- 172.21.15.70:0/681519344 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa388043020 con 0x7fa3941018e0 2024-04-01T01:31:09.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa3837fe640 1 -- 172.21.15.70:0/681519344 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa388004aa0 con 0x7fa3941018e0 2024-04-01T01:31:09.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa398eae640 1 -- 172.21.15.70:0/681519344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa39419f7e0 con 0x7fa3941018e0 2024-04-01T01:31:09.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa3837fe640 1 -- 172.21.15.70:0/681519344 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa388003c60 con 0x7fa3941018e0 2024-04-01T01:31:09.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.486+0000 7fa398eae640 1 -- 172.21.15.70:0/681519344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa39419fd30 con 0x7fa3941018e0 2024-04-01T01:31:09.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.487+0000 7fa398eae640 1 -- 172.21.15.70:0/681519344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa394107a90 con 0x7fa3941018e0 2024-04-01T01:31:09.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.488+0000 7fa3837fe640 1 -- 172.21.15.70:0/681519344 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fa388046020 con 0x7fa3941018e0 2024-04-01T01:31:09.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.489+0000 7fa3837fe640 1 --2- 172.21.15.70:0/681519344 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa37c0a24a0 0x7fa37c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:09.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.489+0000 7fa39259c640 1 --2- 172.21.15.70:0/681519344 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa37c0a24a0 0x7fa37c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:09.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.489+0000 7fa3837fe640 1 -- 172.21.15.70:0/681519344 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa3880f0d90 con 0x7fa3941018e0 2024-04-01T01:31:09.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.489+0000 7fa39259c640 1 --2- 172.21.15.70:0/681519344 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa37c0a24a0 0x7fa37c0a4960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa384005fd0 tx=0x7fa384005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:09.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.492+0000 7fa3837fe640 1 -- 172.21.15.70:0/681519344 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa38804d020 con 0x7fa3941018e0 2024-04-01T01:31:09.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:09.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:09.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:09.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:31:09.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:31:09.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:09.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:09 smithi081 ceph-mon[35142]: Deploying daemon mds.b on smithi081 2024-04-01T01:31:09.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:09 smithi081 ceph-mon[35142]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:09.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:09 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.70:6840/2380158677,v1:172.21.15.70:6841/2380158677] up:boot 2024-04-01T01:31:09.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:09 smithi081 ceph-mon[35142]: fsmap 11 up:standby 2024-04-01T01:31:09.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:09 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-01T01:31:09.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.703+0000 7fa398eae640 1 -- 172.21.15.70:0/681519344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa39402d330 con 0x7fa3941018e0 2024-04-01T01:31:09.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.704+0000 7fa3837fe640 1 -- 172.21.15.70:0/681519344 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9650 (secure 0 0 0) 0x7fa3880be3f0 con 0x7fa3941018e0 2024-04-01T01:31:09.703 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:09.703 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24418,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24425,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24451,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-01T01:31:09.703 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 12 2024-04-01T01:31:09.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.706+0000 7fa398eae640 1 -- 172.21.15.70:0/681519344 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa37c0a24a0 msgr2=0x7fa37c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:09.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.706+0000 7fa398eae640 1 --2- 172.21.15.70:0/681519344 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa37c0a24a0 0x7fa37c0a4960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa384005fd0 tx=0x7fa384005850 comp rx=0 tx=0).stop 2024-04-01T01:31:09.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.706+0000 7fa398eae640 1 -- 172.21.15.70:0/681519344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3941018e0 msgr2=0x7fa39419c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:09.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.706+0000 7fa398eae640 1 --2- 172.21.15.70:0/681519344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3941018e0 0x7fa39419c680 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa388006010 tx=0x7fa3880042f0 comp rx=0 tx=0).stop 2024-04-01T01:31:09.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.706+0000 7fa398eae640 1 -- 172.21.15.70:0/681519344 shutdown_connections 2024-04-01T01:31:09.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.706+0000 7fa398eae640 1 --2- 172.21.15.70:0/681519344 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa37c0a24a0 0x7fa37c0a4960 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:09.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.706+0000 7fa398eae640 1 --2- 172.21.15.70:0/681519344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3941022a0 0x7fa39419eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:09.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.706+0000 7fa398eae640 1 --2- 172.21.15.70:0/681519344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3941018e0 0x7fa39419c680 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:09.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.706+0000 7fa398eae640 1 --2- 172.21.15.70:0/681519344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3941006f0 0x7fa39406a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:09.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.706+0000 7fa398eae640 1 -- 172.21.15.70:0/681519344 >> 172.21.15.70:0/681519344 conn(0x7fa3940fbc80 msgr2=0x7fa394104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:09.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.706+0000 7fa398eae640 1 -- 172.21.15.70:0/681519344 shutdown_connections 2024-04-01T01:31:09.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:09.706+0000 7fa398eae640 1 -- 172.21.15.70:0/681519344 wait complete. 2024-04-01T01:31:09.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:09.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:09.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:09.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:31:09.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:31:09.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:09.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:09 smithi070 ceph-mon[32179]: Deploying daemon mds.b on smithi081 2024-04-01T01:31:09.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:09 smithi070 ceph-mon[32179]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:09.716 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:09 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.70:6840/2380158677,v1:172.21.15.70:6841/2380158677] up:boot 2024-04-01T01:31:09.716 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:09 smithi070 ceph-mon[32179]: fsmap 11 up:standby 2024-04-01T01:31:09.716 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:09 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-01T01:31:09.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:09.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:09.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-01T01:31:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-01T01:31:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:09 smithi189 ceph-mon[35753]: Deploying daemon mds.b on smithi081 2024-04-01T01:31:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:09 smithi189 ceph-mon[35753]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:09 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.70:6840/2380158677,v1:172.21.15.70:6841/2380158677] up:boot 2024-04-01T01:31:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:09 smithi189 ceph-mon[35753]: fsmap 11 up:standby 2024-04-01T01:31:09.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:09 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-01T01:31:10.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:10 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/681519344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:10.750 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:10 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/681519344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:10.751 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:31:10 smithi081 systemd[1]: Starting Ceph mds.b for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:31:10.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:10 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/681519344' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:11.002 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:31:10 smithi081 podman[64989]: 2024-04-01 01:31:10.654424865 +0000 UTC m=+0.017980038 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:31:11.002 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:31:10 smithi081 podman[64989]: 2024-04-01 01:31:10.775499823 +0000 UTC m=+0.139055014 container create de2b5306512919bec671c8d5bfeb447118a2e58fd6a48c7f3a7b7f29dd8a28d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-01T01:31:11.002 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:31:11 smithi081 podman[64989]: 2024-04-01 01:31:11.003260891 +0000 UTC m=+0.366816039 container init de2b5306512919bec671c8d5bfeb447118a2e58fd6a48c7f3a7b7f29dd8a28d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-01T01:31:11.328 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:31:11 smithi081 podman[64989]: 2024-04-01 01:31:11.006127585 +0000 UTC m=+0.369682742 container start de2b5306512919bec671c8d5bfeb447118a2e58fd6a48c7f3a7b7f29dd8a28d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, ceph=True) 2024-04-01T01:31:11.328 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:31:11 smithi081 ceph-mds[65008]: -- 172.21.15.81:0/4209816010 <== mon.2 v2:172.21.15.81:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eca3cc1a40 con 0x55eca3c95000 2024-04-01T01:31:11.328 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:31:11 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-b[65004]: starting mds.b at 2024-04-01T01:31:11.328 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:31:11 smithi081 bash[64989]: de2b5306512919bec671c8d5bfeb447118a2e58fd6a48c7f3a7b7f29dd8a28d3 2024-04-01T01:31:11.328 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:31:11 smithi081 systemd[1]: Started Ceph mds.b for 36577394-efc6-11ee-b647-cb9ed24678a4. 2024-04-01T01:31:11.655 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:11 smithi070 ceph-mon[32179]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:11.655 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:11 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:11.655 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:11 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:11.655 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:11 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:11.656 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:11 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:11.656 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:11 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:11.656 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:11 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:31:11.717 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:31:11.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:11 smithi189 ceph-mon[35753]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:11.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:11 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:11.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:11 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:11.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:11 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:11.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:11 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:11.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:11 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:11.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:11 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:31:11.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:11 smithi081 ceph-mon[35142]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:11.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:11 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:11.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:11 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:11.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:11 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:11.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:11 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:11.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:11 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:11.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:11 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:31:11.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.831+0000 7f04803d0640 1 -- 172.21.15.70:0/550521225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0478104b20 msgr2=0x7f0478106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:11.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.831+0000 7f04803d0640 1 --2- 172.21.15.70:0/550521225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0478104b20 0x7f0478106f10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0468009900 tx=0x7f046802bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:11.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.831+0000 7f04803d0640 1 -- 172.21.15.70:0/550521225 shutdown_connections 2024-04-01T01:31:11.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.831+0000 7f04803d0640 1 --2- 172.21.15.70:0/550521225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0478107450 0x7f04781098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:11.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.831+0000 7f04803d0640 1 --2- 172.21.15.70:0/550521225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0478104b20 0x7f0478106f10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:11.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.831+0000 7f04803d0640 1 --2- 172.21.15.70:0/550521225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04781021f0 0x7f04781045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:11.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.831+0000 7f04803d0640 1 -- 172.21.15.70:0/550521225 >> 172.21.15.70:0/550521225 conn(0x7f04780fbbe0 msgr2=0x7f04780fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:11.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.831+0000 7f04803d0640 1 -- 172.21.15.70:0/550521225 shutdown_connections 2024-04-01T01:31:11.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.831+0000 7f04803d0640 1 -- 172.21.15.70:0/550521225 wait complete. 2024-04-01T01:31:11.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f04803d0640 1 Processor -- start 2024-04-01T01:31:11.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f04803d0640 1 -- start start 2024-04-01T01:31:11.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f04803d0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04781021f0 0x7f0478197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:11.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f04803d0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0478104b20 0x7f0478197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:11.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f04803d0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0478107450 0x7f047819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:11.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f04803d0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f047810d2b0 con 0x7f04781021f0 2024-04-01T01:31:11.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f04803d0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f047810d130 con 0x7f0478104b20 2024-04-01T01:31:11.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f04803d0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f047810d430 con 0x7f0478107450 2024-04-01T01:31:11.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f047e16c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04781021f0 0x7f0478197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:11.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f047e16c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04781021f0 0x7f0478197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50916/0 (socket says 172.21.15.70:50916) 2024-04-01T01:31:11.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f047e16c640 1 -- 172.21.15.70:0/648737914 learned_addr learned my addr 172.21.15.70:0/648737914 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:11.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f047e96d640 1 --2- 172.21.15.70:0/648737914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0478107450 0x7f047819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:11.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f047d96b640 1 --2- 172.21.15.70:0/648737914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0478104b20 0x7f0478197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:11.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f047e16c640 1 -- 172.21.15.70:0/648737914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0478107450 msgr2=0x7f047819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:11.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f047e16c640 1 --2- 172.21.15.70:0/648737914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0478107450 0x7f047819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:11.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f047e16c640 1 -- 172.21.15.70:0/648737914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0478104b20 msgr2=0x7f0478197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:11.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f047e16c640 1 --2- 172.21.15.70:0/648737914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0478104b20 0x7f0478197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:11.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f047e16c640 1 -- 172.21.15.70:0/648737914 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f047819f620 con 0x7f04781021f0 2024-04-01T01:31:11.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f047d96b640 1 --2- 172.21.15.70:0/648737914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0478104b20 0x7f0478197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:31:11.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f047e16c640 1 --2- 172.21.15.70:0/648737914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04781021f0 0x7f0478197a50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f047400e9f0 tx=0x7f047400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:11.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.832+0000 7f046f7fe640 1 -- 172.21.15.70:0/648737914 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047400cd80 con 0x7f04781021f0 2024-04-01T01:31:11.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.833+0000 7f04803d0640 1 -- 172.21.15.70:0/648737914 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f047819f910 con 0x7f04781021f0 2024-04-01T01:31:11.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.833+0000 7f046f7fe640 1 -- 172.21.15.70:0/648737914 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f04740054d0 con 0x7f04781021f0 2024-04-01T01:31:11.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.833+0000 7f046f7fe640 1 -- 172.21.15.70:0/648737914 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0474010490 con 0x7f04781021f0 2024-04-01T01:31:11.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.833+0000 7f04803d0640 1 -- 172.21.15.70:0/648737914 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f047819fde0 con 0x7f04781021f0 2024-04-01T01:31:11.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.834+0000 7f04803d0640 1 -- 172.21.15.70:0/648737914 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0478068540 con 0x7f04781021f0 2024-04-01T01:31:11.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.835+0000 7f046f7fe640 1 -- 172.21.15.70:0/648737914 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f0474005020 con 0x7f04781021f0 2024-04-01T01:31:11.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.835+0000 7f046f7fe640 1 --2- 172.21.15.70:0/648737914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f04640a24a0 0x7f04640a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:11.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.836+0000 7f047d96b640 1 --2- 172.21.15.70:0/648737914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f04640a24a0 0x7f04640a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:11.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.836+0000 7f046f7fe640 1 -- 172.21.15.70:0/648737914 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f04740ce6b0 con 0x7f04781021f0 2024-04-01T01:31:11.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.836+0000 7f047d96b640 1 --2- 172.21.15.70:0/648737914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f04640a24a0 0x7f04640a4960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0468006010 tx=0x7f046802c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:11.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:11.840+0000 7f046f7fe640 1 -- 172.21.15.70:0/648737914 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f047409bd10 con 0x7f04781021f0 2024-04-01T01:31:12.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.066+0000 7f04803d0640 1 -- 172.21.15.70:0/648737914 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04780fd6d0 con 0x7f04781021f0 2024-04-01T01:31:12.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.067+0000 7f046f7fe640 1 -- 172.21.15.70:0/648737914 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10476 (secure 0 0 0) 0x7f047409b610 con 0x7f04781021f0 2024-04-01T01:31:12.066 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:12.067 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24418,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24425,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24451,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24464,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-01T01:31:12.067 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 13 2024-04-01T01:31:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.069+0000 7f04803d0640 1 -- 172.21.15.70:0/648737914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f04640a24a0 msgr2=0x7f04640a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.069+0000 7f04803d0640 1 --2- 172.21.15.70:0/648737914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f04640a24a0 0x7f04640a4960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0468006010 tx=0x7f046802c7e0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.069+0000 7f04803d0640 1 -- 172.21.15.70:0/648737914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04781021f0 msgr2=0x7f0478197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:12.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.069+0000 7f04803d0640 1 --2- 172.21.15.70:0/648737914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04781021f0 0x7f0478197a50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f047400e9f0 tx=0x7f047400edf0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.069+0000 7f04803d0640 1 -- 172.21.15.70:0/648737914 shutdown_connections 2024-04-01T01:31:12.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.069+0000 7f04803d0640 1 --2- 172.21.15.70:0/648737914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f04640a24a0 0x7f04640a4960 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.069+0000 7f04803d0640 1 --2- 172.21.15.70:0/648737914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0478107450 0x7f047819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.069+0000 7f04803d0640 1 --2- 172.21.15.70:0/648737914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0478104b20 0x7f0478197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.069+0000 7f04803d0640 1 --2- 172.21.15.70:0/648737914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04781021f0 0x7f0478197a50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.069+0000 7f04803d0640 1 -- 172.21.15.70:0/648737914 >> 172.21.15.70:0/648737914 conn(0x7f04780fbbe0 msgr2=0x7f0478102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:12.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.069+0000 7f04803d0640 1 -- 172.21.15.70:0/648737914 shutdown_connections 2024-04-01T01:31:12.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.069+0000 7f04803d0640 1 -- 172.21.15.70:0/648737914 wait complete. 2024-04-01T01:31:12.080 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:31:12.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.187+0000 7fb0ecde9640 1 -- 172.21.15.70:0/3712383572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e8101860 msgr2=0x7fb0e8101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:12.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.187+0000 7fb0ecde9640 1 --2- 172.21.15.70:0/3712383572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e8101860 0x7fb0e8101ce0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009900 tx=0x7fb0dc02bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:12.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.188+0000 7fb0ecde9640 1 -- 172.21.15.70:0/3712383572 shutdown_connections 2024-04-01T01:31:12.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.188+0000 7fb0ecde9640 1 --2- 172.21.15.70:0/3712383572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0e8102220 0x7fb0e8108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.188+0000 7fb0ecde9640 1 --2- 172.21.15.70:0/3712383572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e8101860 0x7fb0e8101ce0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.188+0000 7fb0ecde9640 1 --2- 172.21.15.70:0/3712383572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0e8100670 0x7fb0e8100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.188+0000 7fb0ecde9640 1 -- 172.21.15.70:0/3712383572 >> 172.21.15.70:0/3712383572 conn(0x7fb0e80fbbe0 msgr2=0x7fb0e80fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:12.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.188+0000 7fb0ecde9640 1 -- 172.21.15.70:0/3712383572 shutdown_connections 2024-04-01T01:31:12.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.188+0000 7fb0ecde9640 1 -- 172.21.15.70:0/3712383572 wait complete. 2024-04-01T01:31:12.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.188+0000 7fb0ecde9640 1 Processor -- start 2024-04-01T01:31:12.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.188+0000 7fb0ecde9640 1 -- start start 2024-04-01T01:31:12.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.188+0000 7fb0ecde9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e8100670 0x7fb0e8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:12.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.188+0000 7fb0ecde9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0e8101860 0x7fb0e819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:12.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.188+0000 7fb0ecde9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0e8102220 0x7fb0e819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:12.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.188+0000 7fb0ecde9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e810c190 con 0x7fb0e8100670 2024-04-01T01:31:12.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0ecde9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e810c010 con 0x7fb0e8101860 2024-04-01T01:31:12.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0ecde9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e810c310 con 0x7fb0e8102220 2024-04-01T01:31:12.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0e659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e8100670 0x7fb0e8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:12.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0e659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e8100670 0x7fb0e8069420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50932/0 (socket says 172.21.15.70:50932) 2024-04-01T01:31:12.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0e659c640 1 -- 172.21.15.70:0/2165353281 learned_addr learned my addr 172.21.15.70:0/2165353281 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:12.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0e5d9b640 1 --2- 172.21.15.70:0/2165353281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0e8101860 0x7fb0e819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:12.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0e6d9d640 1 --2- 172.21.15.70:0/2165353281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0e8102220 0x7fb0e819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:12.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0e659c640 1 -- 172.21.15.70:0/2165353281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0e8102220 msgr2=0x7fb0e819ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:12.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0e659c640 1 --2- 172.21.15.70:0/2165353281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0e8102220 0x7fb0e819ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0e659c640 1 -- 172.21.15.70:0/2165353281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0e8101860 msgr2=0x7fb0e819c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:12.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0e659c640 1 --2- 172.21.15.70:0/2165353281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0e8101860 0x7fb0e819c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0e659c640 1 -- 172.21.15.70:0/2165353281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e819f670 con 0x7fb0e8100670 2024-04-01T01:31:12.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0e659c640 1 --2- 172.21.15.70:0/2165353281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e8100670 0x7fb0e8069420 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb0d800e9c0 tx=0x7fb0d800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:12.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0d77fe640 1 -- 172.21.15.70:0/2165353281 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d800cd80 con 0x7fb0e8100670 2024-04-01T01:31:12.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0ecde9640 1 -- 172.21.15.70:0/2165353281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e819f900 con 0x7fb0e8100670 2024-04-01T01:31:12.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0d77fe640 1 -- 172.21.15.70:0/2165353281 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0d80035f0 con 0x7fb0e8100670 2024-04-01T01:31:12.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0d77fe640 1 -- 172.21.15.70:0/2165353281 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8010690 con 0x7fb0e8100670 2024-04-01T01:31:12.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.189+0000 7fb0ecde9640 1 -- 172.21.15.70:0/2165353281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e819fe70 con 0x7fb0e8100670 2024-04-01T01:31:12.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.191+0000 7fb0ecde9640 1 -- 172.21.15.70:0/2165353281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0e8068540 con 0x7fb0e8100670 2024-04-01T01:31:12.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.192+0000 7fb0d77fe640 1 -- 172.21.15.70:0/2165353281 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fb0d80037a0 con 0x7fb0e8100670 2024-04-01T01:31:12.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.193+0000 7fb0d77fe640 1 --2- 172.21.15.70:0/2165353281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb0d00a24a0 0x7fb0d00a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:12.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.193+0000 7fb0e5d9b640 1 --2- 172.21.15.70:0/2165353281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb0d00a24a0 0x7fb0d00a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:12.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.193+0000 7fb0d77fe640 1 -- 172.21.15.70:0/2165353281 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb0d80ce610 con 0x7fb0e8100670 2024-04-01T01:31:12.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.193+0000 7fb0e5d9b640 1 --2- 172.21.15.70:0/2165353281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb0d00a24a0 0x7fb0d00a4960 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009230 tx=0x7fb0dc005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:12.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.197+0000 7fb0d77fe640 1 -- 172.21.15.70:0/2165353281 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb0d809bc70 con 0x7fb0e8100670 2024-04-01T01:31:12.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.380+0000 7fb0ecde9640 1 -- 172.21.15.70:0/2165353281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb0e8062350 con 0x7fb0e8100670 2024-04-01T01:31:12.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.381+0000 7fb0d77fe640 1 -- 172.21.15.70:0/2165353281 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18396 (secure 0 0 0) 0x7fb0d809b570 con 0x7fb0e8100670 2024-04-01T01:31:12.381 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:12.381 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":81,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:30:26.779800+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:31:12.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.383+0000 7fb0ecde9640 1 -- 172.21.15.70:0/2165353281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb0d00a24a0 msgr2=0x7fb0d00a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:12.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.383+0000 7fb0ecde9640 1 --2- 172.21.15.70:0/2165353281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb0d00a24a0 0x7fb0d00a4960 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009230 tx=0x7fb0dc005fd0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.383+0000 7fb0ecde9640 1 -- 172.21.15.70:0/2165353281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e8100670 msgr2=0x7fb0e8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:12.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.383+0000 7fb0ecde9640 1 --2- 172.21.15.70:0/2165353281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e8100670 0x7fb0e8069420 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb0d800e9c0 tx=0x7fb0d800ee80 comp rx=0 tx=0).stop 2024-04-01T01:31:12.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.383+0000 7fb0ecde9640 1 -- 172.21.15.70:0/2165353281 shutdown_connections 2024-04-01T01:31:12.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.383+0000 7fb0ecde9640 1 --2- 172.21.15.70:0/2165353281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb0d00a24a0 0x7fb0d00a4960 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.383+0000 7fb0ecde9640 1 --2- 172.21.15.70:0/2165353281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0e8102220 0x7fb0e819ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.383+0000 7fb0ecde9640 1 --2- 172.21.15.70:0/2165353281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0e8101860 0x7fb0e819c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.383+0000 7fb0ecde9640 1 --2- 172.21.15.70:0/2165353281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e8100670 0x7fb0e8069420 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.383+0000 7fb0ecde9640 1 -- 172.21.15.70:0/2165353281 >> 172.21.15.70:0/2165353281 conn(0x7fb0e80fbbe0 msgr2=0x7fb0e8104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:12.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.383+0000 7fb0ecde9640 1 -- 172.21.15.70:0/2165353281 shutdown_connections 2024-04-01T01:31:12.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.383+0000 7fb0ecde9640 1 -- 172.21.15.70:0/2165353281 wait complete. 2024-04-01T01:31:12.394 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-01T01:26:42.809779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-01T01:31:12.394 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-01T01:31:12.397 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:12 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] up:boot 2024-04-01T01:31:12.397 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:12 smithi189 ceph-mon[35753]: fsmap 12 up:standby 2024-04-01T01:31:12.398 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:12 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-01T01:31:12.398 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:12 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/648737914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:12.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.500+0000 7fe95d6fb640 1 -- 172.21.15.70:0/2640195400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe958103300 msgr2=0x7fe958109b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:12.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.500+0000 7fe95d6fb640 1 --2- 172.21.15.70:0/2640195400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe958103300 0x7fe958109b90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe950009ca0 tx=0x7fe95002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.500+0000 7fe95d6fb640 1 -- 172.21.15.70:0/2640195400 shutdown_connections 2024-04-01T01:31:12.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.500+0000 7fe95d6fb640 1 --2- 172.21.15.70:0/2640195400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe958103300 0x7fe958109b90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.500+0000 7fe95d6fb640 1 --2- 172.21.15.70:0/2640195400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe958102940 0x7fe958102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.500+0000 7fe95d6fb640 1 --2- 172.21.15.70:0/2640195400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe958101750 0x7fe958101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.501+0000 7fe95d6fb640 1 -- 172.21.15.70:0/2640195400 >> 172.21.15.70:0/2640195400 conn(0x7fe9580fcce0 msgr2=0x7fe9580ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.501+0000 7fe95d6fb640 1 -- 172.21.15.70:0/2640195400 shutdown_connections 2024-04-01T01:31:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.501+0000 7fe95d6fb640 1 -- 172.21.15.70:0/2640195400 wait complete. 2024-04-01T01:31:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.501+0000 7fe95d6fb640 1 Processor -- start 2024-04-01T01:31:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.501+0000 7fe95d6fb640 1 -- start start 2024-04-01T01:31:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.501+0000 7fe95d6fb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe958101750 0x7fe9581978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.501+0000 7fe95d6fb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe958102940 0x7fe958197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.501+0000 7fe95d6fb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe958103300 0x7fe95819ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.501+0000 7fe95d6fb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe95810d280 con 0x7fe958102940 2024-04-01T01:31:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.501+0000 7fe95d6fb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe95810d100 con 0x7fe958103300 2024-04-01T01:31:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.501+0000 7fe95d6fb640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe95810d400 con 0x7fe958101750 2024-04-01T01:31:12.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.501+0000 7fe9567fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe958102940 0x7fe958197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.502+0000 7fe9567fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe958102940 0x7fe958197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50964/0 (socket says 172.21.15.70:50964) 2024-04-01T01:31:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.502+0000 7fe9567fc640 1 -- 172.21.15.70:0/3453215272 learned_addr learned my addr 172.21.15.70:0/3453215272 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.502+0000 7fe9577fe640 1 --2- 172.21.15.70:0/3453215272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe958103300 0x7fe95819ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.502+0000 7fe956ffd640 1 --2- 172.21.15.70:0/3453215272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe958101750 0x7fe9581978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.502+0000 7fe9567fc640 1 -- 172.21.15.70:0/3453215272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe958101750 msgr2=0x7fe9581978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.502+0000 7fe9567fc640 1 --2- 172.21.15.70:0/3453215272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe958101750 0x7fe9581978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.502+0000 7fe9567fc640 1 -- 172.21.15.70:0/3453215272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe958103300 msgr2=0x7fe95819ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.502+0000 7fe9567fc640 1 --2- 172.21.15.70:0/3453215272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe958103300 0x7fe95819ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.502+0000 7fe9567fc640 1 -- 172.21.15.70:0/3453215272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe95819f500 con 0x7fe958102940 2024-04-01T01:31:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.502+0000 7fe9577fe640 1 --2- 172.21.15.70:0/3453215272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe958103300 0x7fe95819ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:31:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.502+0000 7fe9567fc640 1 --2- 172.21.15.70:0/3453215272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe958102940 0x7fe958197e20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00c920 tx=0x7fe94c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.502+0000 7fe93ffff640 1 -- 172.21.15.70:0/3453215272 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c007c70 con 0x7fe958102940 2024-04-01T01:31:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.502+0000 7fe95d6fb640 1 -- 172.21.15.70:0/3453215272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe95819f7f0 con 0x7fe958102940 2024-04-01T01:31:12.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.502+0000 7fe93ffff640 1 -- 172.21.15.70:0/3453215272 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe94c007e10 con 0x7fe958102940 2024-04-01T01:31:12.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.502+0000 7fe93ffff640 1 -- 172.21.15.70:0/3453215272 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94c018820 con 0x7fe958102940 2024-04-01T01:31:12.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.502+0000 7fe95d6fb640 1 -- 172.21.15.70:0/3453215272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe95819fd90 con 0x7fe958102940 2024-04-01T01:31:12.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.503+0000 7fe95d6fb640 1 -- 172.21.15.70:0/3453215272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe958108a10 con 0x7fe958102940 2024-04-01T01:31:12.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.504+0000 7fe93ffff640 1 -- 172.21.15.70:0/3453215272 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fe94c0189c0 con 0x7fe958102940 2024-04-01T01:31:12.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.504+0000 7fe93ffff640 1 --2- 172.21.15.70:0/3453215272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe9400a24a0 0x7fe9400a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:12.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.505+0000 7fe956ffd640 1 --2- 172.21.15.70:0/3453215272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe9400a24a0 0x7fe9400a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:12.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.505+0000 7fe93ffff640 1 -- 172.21.15.70:0/3453215272 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe94c0ce4a0 con 0x7fe958102940 2024-04-01T01:31:12.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.505+0000 7fe956ffd640 1 --2- 172.21.15.70:0/3453215272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe9400a24a0 0x7fe9400a4960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe948005fd0 tx=0x7fe948005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:12.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.509+0000 7fe93ffff640 1 -- 172.21.15.70:0/3453215272 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe94c014030 con 0x7fe958102940 2024-04-01T01:31:12.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:12 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] up:boot 2024-04-01T01:31:12.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:12 smithi081 ceph-mon[35142]: fsmap 12 up:standby 2024-04-01T01:31:12.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:12 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-01T01:31:12.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:12 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/648737914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:12.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.715+0000 7fe95d6fb640 1 -- 172.21.15.70:0/3453215272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe95802d360 con 0x7fe958102940 2024-04-01T01:31:12.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.715+0000 7fe93ffff640 1 -- 172.21.15.70:0/3453215272 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fe94c09bb00 con 0x7fe958102940 2024-04-01T01:31:12.715 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-01T01:31:12.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.718+0000 7fe95d6fb640 1 -- 172.21.15.70:0/3453215272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe9400a24a0 msgr2=0x7fe9400a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:12.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.718+0000 7fe95d6fb640 1 --2- 172.21.15.70:0/3453215272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe9400a24a0 0x7fe9400a4960 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe948005fd0 tx=0x7fe948005850 comp rx=0 tx=0).stop 2024-04-01T01:31:12.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.718+0000 7fe95d6fb640 1 -- 172.21.15.70:0/3453215272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe958102940 msgr2=0x7fe958197e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:12.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.718+0000 7fe95d6fb640 1 --2- 172.21.15.70:0/3453215272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe958102940 0x7fe958197e20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe94c00c920 tx=0x7fe94c00cde0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.718+0000 7fe95d6fb640 1 -- 172.21.15.70:0/3453215272 shutdown_connections 2024-04-01T01:31:12.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.718+0000 7fe95d6fb640 1 --2- 172.21.15.70:0/3453215272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe958103300 0x7fe95819ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.718+0000 7fe95d6fb640 1 --2- 172.21.15.70:0/3453215272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe958102940 0x7fe958197e20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.718+0000 7fe95d6fb640 1 --2- 172.21.15.70:0/3453215272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe9400a24a0 0x7fe9400a4960 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.718+0000 7fe95d6fb640 1 --2- 172.21.15.70:0/3453215272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe958101750 0x7fe9581978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.718+0000 7fe95d6fb640 1 -- 172.21.15.70:0/3453215272 >> 172.21.15.70:0/3453215272 conn(0x7fe9580fcce0 msgr2=0x7fe958105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:12.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.718+0000 7fe95d6fb640 1 -- 172.21.15.70:0/3453215272 shutdown_connections 2024-04-01T01:31:12.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.718+0000 7fe95d6fb640 1 -- 172.21.15.70:0/3453215272 wait complete. 2024-04-01T01:31:12.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:12 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] up:boot 2024-04-01T01:31:12.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:12 smithi070 ceph-mon[32179]: fsmap 12 up:standby 2024-04-01T01:31:12.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:12 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-01T01:31:12.729 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:12 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/648737914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:12.730 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-01T01:31:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.849+0000 7f50f8815640 1 -- 172.21.15.70:0/74392476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f4103330 msgr2=0x7f50f4109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.849+0000 7f50f8815640 1 --2- 172.21.15.70:0/74392476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f4103330 0x7f50f4109bc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f50ec009ca0 tx=0x7f50ec02c0f0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.849+0000 7f50f8815640 1 -- 172.21.15.70:0/74392476 shutdown_connections 2024-04-01T01:31:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.849+0000 7f50f8815640 1 --2- 172.21.15.70:0/74392476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f4103330 0x7f50f4109bc0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.850+0000 7f50f8815640 1 --2- 172.21.15.70:0/74392476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50f4102970 0x7f50f4102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.850+0000 7f50f8815640 1 --2- 172.21.15.70:0/74392476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50f4101780 0x7f50f4101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.850+0000 7f50f8815640 1 -- 172.21.15.70:0/74392476 >> 172.21.15.70:0/74392476 conn(0x7f50f40fccf0 msgr2=0x7f50f40ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.850+0000 7f50f8815640 1 -- 172.21.15.70:0/74392476 shutdown_connections 2024-04-01T01:31:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.850+0000 7f50f8815640 1 -- 172.21.15.70:0/74392476 wait complete. 2024-04-01T01:31:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.850+0000 7f50f8815640 1 Processor -- start 2024-04-01T01:31:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.850+0000 7f50f8815640 1 -- start start 2024-04-01T01:31:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.850+0000 7f50f8815640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50f4101780 0x7f50f41978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.850+0000 7f50f8815640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50f4102970 0x7f50f4197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.850+0000 7f50f8815640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f4103330 0x7f50f419ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.850+0000 7f50f8815640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f50f410d290 con 0x7f50f4103330 2024-04-01T01:31:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.850+0000 7f50f8815640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f50f410d110 con 0x7f50f4101780 2024-04-01T01:31:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.850+0000 7f50f8815640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f50f410d410 con 0x7f50f4102970 2024-04-01T01:31:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.851+0000 7f50f2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f4103330 0x7f50f419ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.851+0000 7f50f2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f4103330 0x7f50f419ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50988/0 (socket says 172.21.15.70:50988) 2024-04-01T01:31:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.851+0000 7f50f2d9d640 1 -- 172.21.15.70:0/2616476395 learned_addr learned my addr 172.21.15.70:0/2616476395 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.851+0000 7f50f1d9b640 1 --2- 172.21.15.70:0/2616476395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50f4102970 0x7f50f4197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.851+0000 7f50f259c640 1 --2- 172.21.15.70:0/2616476395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50f4101780 0x7f50f41978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.851+0000 7f50f2d9d640 1 -- 172.21.15.70:0/2616476395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50f4102970 msgr2=0x7f50f4197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.851+0000 7f50f2d9d640 1 --2- 172.21.15.70:0/2616476395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50f4102970 0x7f50f4197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.851+0000 7f50f2d9d640 1 -- 172.21.15.70:0/2616476395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50f4101780 msgr2=0x7f50f41978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.851+0000 7f50f2d9d640 1 --2- 172.21.15.70:0/2616476395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50f4101780 0x7f50f41978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.851+0000 7f50f2d9d640 1 -- 172.21.15.70:0/2616476395 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f419f500 con 0x7f50f4103330 2024-04-01T01:31:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.851+0000 7f50f2d9d640 1 --2- 172.21.15.70:0/2616476395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f4103330 0x7f50f419ee90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f50ec02c700 tx=0x7f50ec0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.851+0000 7f50e37fe640 1 -- 172.21.15.70:0/2616476395 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec041020 con 0x7f50f4103330 2024-04-01T01:31:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.851+0000 7f50f8815640 1 -- 172.21.15.70:0/2616476395 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50f419f790 con 0x7f50f4103330 2024-04-01T01:31:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.851+0000 7f50e37fe640 1 -- 172.21.15.70:0/2616476395 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f50ec003e20 con 0x7f50f4103330 2024-04-01T01:31:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.851+0000 7f50e37fe640 1 -- 172.21.15.70:0/2616476395 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ec039040 con 0x7f50f4103330 2024-04-01T01:31:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.851+0000 7f50f8815640 1 -- 172.21.15.70:0/2616476395 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50f419fcd0 con 0x7f50f4103330 2024-04-01T01:31:12.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.852+0000 7f50f8815640 1 -- 172.21.15.70:0/2616476395 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50f4108aa0 con 0x7f50f4103330 2024-04-01T01:31:12.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.853+0000 7f50e37fe640 1 -- 172.21.15.70:0/2616476395 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f50ec044020 con 0x7f50f4103330 2024-04-01T01:31:12.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.854+0000 7f50e37fe640 1 --2- 172.21.15.70:0/2616476395 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f50dc0a24a0 0x7f50dc0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:12.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.854+0000 7f50f259c640 1 --2- 172.21.15.70:0/2616476395 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f50dc0a24a0 0x7f50dc0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:12.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.854+0000 7f50e37fe640 1 -- 172.21.15.70:0/2616476395 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f50ec0f0500 con 0x7f50f4103330 2024-04-01T01:31:12.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.854+0000 7f50f259c640 1 --2- 172.21.15.70:0/2616476395 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f50dc0a24a0 0x7f50dc0a4960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f50e4005fd0 tx=0x7f50e400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:12.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:12.858+0000 7f50e37fe640 1 -- 172.21.15.70:0/2616476395 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f50ec04b020 con 0x7f50f4103330 2024-04-01T01:31:13.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.074+0000 7f50f8815640 1 -- 172.21.15.70:0/2616476395 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f50f4062350 con 0x7f50f4103330 2024-04-01T01:31:13.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.075+0000 7f50e37fe640 1 -- 172.21.15.70:0/2616476395 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f50ec0bdb60 con 0x7f50f4103330 2024-04-01T01:31:13.074 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:13.074 INFO:teuthology.orchestra.run.smithi070.stdout:[ 2024-04-01T01:31:13.074 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:31:13.075 INFO:teuthology.orchestra.run.smithi070.stdout: "poolnum": 1, 2024-04-01T01:31:13.075 INFO:teuthology.orchestra.run.smithi070.stdout: "poolname": ".mgr" 2024-04-01T01:31:13.075 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:31:13.075 INFO:teuthology.orchestra.run.smithi070.stdout:] 2024-04-01T01:31:13.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.077+0000 7f50f8815640 1 -- 172.21.15.70:0/2616476395 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f50dc0a24a0 msgr2=0x7f50dc0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:13.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.077+0000 7f50f8815640 1 --2- 172.21.15.70:0/2616476395 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f50dc0a24a0 0x7f50dc0a4960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f50e4005fd0 tx=0x7f50e400b410 comp rx=0 tx=0).stop 2024-04-01T01:31:13.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.077+0000 7f50f8815640 1 -- 172.21.15.70:0/2616476395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f4103330 msgr2=0x7f50f419ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:13.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.077+0000 7f50f8815640 1 --2- 172.21.15.70:0/2616476395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f4103330 0x7f50f419ee90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f50ec02c700 tx=0x7f50ec0328a0 comp rx=0 tx=0).stop 2024-04-01T01:31:13.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.077+0000 7f50f8815640 1 -- 172.21.15.70:0/2616476395 shutdown_connections 2024-04-01T01:31:13.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.077+0000 7f50f8815640 1 --2- 172.21.15.70:0/2616476395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f50f4103330 0x7f50f419ee90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:13.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.077+0000 7f50f8815640 1 --2- 172.21.15.70:0/2616476395 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f50dc0a24a0 0x7f50dc0a4960 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:13.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.077+0000 7f50f8815640 1 --2- 172.21.15.70:0/2616476395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50f4102970 0x7f50f4197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:13.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.077+0000 7f50f8815640 1 --2- 172.21.15.70:0/2616476395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50f4101780 0x7f50f41978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:13.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.077+0000 7f50f8815640 1 -- 172.21.15.70:0/2616476395 >> 172.21.15.70:0/2616476395 conn(0x7f50f40fccf0 msgr2=0x7f50f4105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:13.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.077+0000 7f50f8815640 1 -- 172.21.15.70:0/2616476395 shutdown_connections 2024-04-01T01:31:13.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.077+0000 7f50f8815640 1 -- 172.21.15.70:0/2616476395 wait complete. 2024-04-01T01:31:13.087 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-01T01:31:13.087 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-01T01:31:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.193+0000 7f792984e640 1 -- 172.21.15.70:0/2935200664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7924100670 msgr2=0x7f7924100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.193+0000 7f792984e640 1 --2- 172.21.15.70:0/2935200664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7924100670 0x7f7924100a90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7914009900 tx=0x7f791402bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.193+0000 7f792984e640 1 -- 172.21.15.70:0/2935200664 shutdown_connections 2024-04-01T01:31:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.193+0000 7f792984e640 1 --2- 172.21.15.70:0/2935200664 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7924102220 0x7f7924108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.193+0000 7f792984e640 1 --2- 172.21.15.70:0/2935200664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7924101860 0x7f7924101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.193+0000 7f792984e640 1 --2- 172.21.15.70:0/2935200664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7924100670 0x7f7924100a90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.193+0000 7f792984e640 1 -- 172.21.15.70:0/2935200664 >> 172.21.15.70:0/2935200664 conn(0x7f79240fbbe0 msgr2=0x7f79240fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.193+0000 7f792984e640 1 -- 172.21.15.70:0/2935200664 shutdown_connections 2024-04-01T01:31:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.193+0000 7f792984e640 1 -- 172.21.15.70:0/2935200664 wait complete. 2024-04-01T01:31:13.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.193+0000 7f792984e640 1 Processor -- start 2024-04-01T01:31:13.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f792984e640 1 -- start start 2024-04-01T01:31:13.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f792984e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7924100670 0x7f7924197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:13.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f792984e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7924101860 0x7f7924197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:13.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f792984e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7924102220 0x7f792419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:13.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f792984e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f792410c1a0 con 0x7f7924101860 2024-04-01T01:31:13.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f792984e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f792410c020 con 0x7f7924102220 2024-04-01T01:31:13.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f792984e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f792410c320 con 0x7f7924100670 2024-04-01T01:31:13.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f79227fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7924101860 0x7f7924197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:13.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f79227fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7924101860 0x7f7924197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51000/0 (socket says 172.21.15.70:51000) 2024-04-01T01:31:13.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f79227fc640 1 -- 172.21.15.70:0/2399033275 learned_addr learned my addr 172.21.15.70:0/2399033275 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:13.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f79237fe640 1 --2- 172.21.15.70:0/2399033275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7924102220 0x7f792419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:13.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f7922ffd640 1 --2- 172.21.15.70:0/2399033275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7924100670 0x7f7924197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:13.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f79227fc640 1 -- 172.21.15.70:0/2399033275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7924100670 msgr2=0x7f7924197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:13.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f79227fc640 1 --2- 172.21.15.70:0/2399033275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7924100670 0x7f7924197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:13.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f79227fc640 1 -- 172.21.15.70:0/2399033275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7924102220 msgr2=0x7f792419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:13.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f79227fc640 1 --2- 172.21.15.70:0/2399033275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7924102220 0x7f792419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:13.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f79227fc640 1 -- 172.21.15.70:0/2399033275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792419f680 con 0x7f7924101860 2024-04-01T01:31:13.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f79227fc640 1 --2- 172.21.15.70:0/2399033275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7924101860 0x7f7924197fa0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f791800d770 tx=0x7f791800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:13.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f792884c640 1 -- 172.21.15.70:0/2399033275 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7918017070 con 0x7f7924101860 2024-04-01T01:31:13.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f792884c640 1 -- 172.21.15.70:0/2399033275 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79180054d0 con 0x7f7924101860 2024-04-01T01:31:13.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f792984e640 1 -- 172.21.15.70:0/2399033275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f792419f970 con 0x7f7924101860 2024-04-01T01:31:13.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.194+0000 7f792884c640 1 -- 172.21.15.70:0/2399033275 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7918004cf0 con 0x7f7924101860 2024-04-01T01:31:13.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.195+0000 7f792984e640 1 -- 172.21.15.70:0/2399033275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f792419fe70 con 0x7f7924101860 2024-04-01T01:31:13.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.196+0000 7f792984e640 1 -- 172.21.15.70:0/2399033275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79241a00a0 con 0x7f7924101860 2024-04-01T01:31:13.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.197+0000 7f792884c640 1 -- 172.21.15.70:0/2399033275 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f7918004e90 con 0x7f7924101860 2024-04-01T01:31:13.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.197+0000 7f792884c640 1 --2- 172.21.15.70:0/2399033275 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f790c0a24a0 0x7f790c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:13.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.197+0000 7f7922ffd640 1 --2- 172.21.15.70:0/2399033275 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f790c0a24a0 0x7f790c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:13.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.197+0000 7f792884c640 1 -- 172.21.15.70:0/2399033275 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f79180cdce0 con 0x7f7924101860 2024-04-01T01:31:13.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.198+0000 7f7922ffd640 1 --2- 172.21.15.70:0/2399033275 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f790c0a24a0 0x7f790c0a4960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7914006010 tx=0x7f7914005dd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:13.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.202+0000 7f792884c640 1 -- 172.21.15.70:0/2399033275 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f791809b340 con 0x7f7924101860 2024-04-01T01:31:13.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:13.376+0000 7f792984e640 1 -- 172.21.15.70:0/2399033275 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f79241a0270 con 0x7f7924101860 2024-04-01T01:31:13.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:13 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2165353281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:13.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:13 smithi189 ceph-mon[35753]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:13.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:13 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3453215272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:31:13.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:13 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:13 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:13 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:13 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:13 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:13 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:13 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2616476395' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-01T01:31:13.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:13 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2165353281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:13.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:13 smithi070 ceph-mon[32179]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:13.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:13 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3453215272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:31:13.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:13 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:13 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:13 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:13 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:13 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:13 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:13 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2616476395' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-01T01:31:13.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:13 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2165353281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:13.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:13 smithi081 ceph-mon[35142]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:13.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:13 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3453215272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:31:13.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:13.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:13 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2616476395' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-01T01:31:14.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.352+0000 7f792884c640 1 -- 172.21.15.70:0/2399033275 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f791809ac40 con 0x7f7924101860 2024-04-01T01:31:14.352 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' created 2024-04-01T01:31:14.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.354+0000 7f792984e640 1 -- 172.21.15.70:0/2399033275 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f790c0a24a0 msgr2=0x7f790c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:14.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.354+0000 7f792984e640 1 --2- 172.21.15.70:0/2399033275 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f790c0a24a0 0x7f790c0a4960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7914006010 tx=0x7f7914005dd0 comp rx=0 tx=0).stop 2024-04-01T01:31:14.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.354+0000 7f792984e640 1 -- 172.21.15.70:0/2399033275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7924101860 msgr2=0x7f7924197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:14.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.354+0000 7f792984e640 1 --2- 172.21.15.70:0/2399033275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7924101860 0x7f7924197fa0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f791800d770 tx=0x7f791800dc30 comp rx=0 tx=0).stop 2024-04-01T01:31:14.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.355+0000 7f792984e640 1 -- 172.21.15.70:0/2399033275 shutdown_connections 2024-04-01T01:31:14.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.355+0000 7f792984e640 1 --2- 172.21.15.70:0/2399033275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7924102220 0x7f792419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:14.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.355+0000 7f792984e640 1 --2- 172.21.15.70:0/2399033275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7924101860 0x7f7924197fa0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:14.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.355+0000 7f792984e640 1 --2- 172.21.15.70:0/2399033275 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f790c0a24a0 0x7f790c0a4960 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:14.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.355+0000 7f792984e640 1 --2- 172.21.15.70:0/2399033275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7924100670 0x7f7924197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:14.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.355+0000 7f792984e640 1 -- 172.21.15.70:0/2399033275 >> 172.21.15.70:0/2399033275 conn(0x7f79240fbbe0 msgr2=0x7f7924069a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:14.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.355+0000 7f792984e640 1 -- 172.21.15.70:0/2399033275 shutdown_connections 2024-04-01T01:31:14.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.355+0000 7f792984e640 1 -- 172.21.15.70:0/2399033275 wait complete. 2024-04-01T01:31:14.369 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-01T01:31:14.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.477+0000 7f93568ed640 1 -- 172.21.15.70:0/337720934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9350104b20 msgr2=0x7f9350106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:14.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.477+0000 7f93568ed640 1 --2- 172.21.15.70:0/337720934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9350104b20 0x7f9350106f10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9344009900 tx=0x7f934402bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:14.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.477+0000 7f93568ed640 1 -- 172.21.15.70:0/337720934 shutdown_connections 2024-04-01T01:31:14.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.477+0000 7f93568ed640 1 --2- 172.21.15.70:0/337720934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9350107450 0x7f93501098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:14.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.477+0000 7f93568ed640 1 --2- 172.21.15.70:0/337720934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9350104b20 0x7f9350106f10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:14.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.477+0000 7f93568ed640 1 --2- 172.21.15.70:0/337720934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93501021f0 0x7f93501045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:14.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.477+0000 7f93568ed640 1 -- 172.21.15.70:0/337720934 >> 172.21.15.70:0/337720934 conn(0x7f93500fbc00 msgr2=0x7f93500fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:14.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.477+0000 7f93568ed640 1 -- 172.21.15.70:0/337720934 shutdown_connections 2024-04-01T01:31:14.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.477+0000 7f93568ed640 1 -- 172.21.15.70:0/337720934 wait complete. 2024-04-01T01:31:14.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f93568ed640 1 Processor -- start 2024-04-01T01:31:14.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f93568ed640 1 -- start start 2024-04-01T01:31:14.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f93568ed640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93501021f0 0x7f9350197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:14.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f93568ed640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9350104b20 0x7f9350197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:14.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f93568ed640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9350107450 0x7f935019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:14.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f93568ed640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f935010d2b0 con 0x7f9350104b20 2024-04-01T01:31:14.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f93568ed640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f935010d130 con 0x7f93501021f0 2024-04-01T01:31:14.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f93568ed640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f935010d430 con 0x7f9350107450 2024-04-01T01:31:14.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f934f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9350104b20 0x7f9350197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:14.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f934f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9350104b20 0x7f9350197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51014/0 (socket says 172.21.15.70:51014) 2024-04-01T01:31:14.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f934f7fe640 1 -- 172.21.15.70:0/2988163874 learned_addr learned my addr 172.21.15.70:0/2988163874 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:14.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f9354e8a640 1 --2- 172.21.15.70:0/2988163874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9350107450 0x7f935019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:14.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f934ffff640 1 --2- 172.21.15.70:0/2988163874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93501021f0 0x7f9350197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:14.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f934f7fe640 1 -- 172.21.15.70:0/2988163874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9350107450 msgr2=0x7f935019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:14.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f934f7fe640 1 --2- 172.21.15.70:0/2988163874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9350107450 0x7f935019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:14.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f934f7fe640 1 -- 172.21.15.70:0/2988163874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93501021f0 msgr2=0x7f9350197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:14.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f934f7fe640 1 --2- 172.21.15.70:0/2988163874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93501021f0 0x7f9350197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:14.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f934f7fe640 1 -- 172.21.15.70:0/2988163874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f935019f630 con 0x7f9350104b20 2024-04-01T01:31:14.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f934ffff640 1 --2- 172.21.15.70:0/2988163874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93501021f0 0x7f9350197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:31:14.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.478+0000 7f934f7fe640 1 --2- 172.21.15.70:0/2988163874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9350104b20 0x7f9350197f50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9344000c00 tx=0x7f9344004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:14.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.479+0000 7f934d7fa640 1 -- 172.21.15.70:0/2988163874 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344043020 con 0x7f9350104b20 2024-04-01T01:31:14.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.479+0000 7f93568ed640 1 -- 172.21.15.70:0/2988163874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f935019f8c0 con 0x7f9350104b20 2024-04-01T01:31:14.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.479+0000 7f934d7fa640 1 -- 172.21.15.70:0/2988163874 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9344004aa0 con 0x7f9350104b20 2024-04-01T01:31:14.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.479+0000 7f934d7fa640 1 -- 172.21.15.70:0/2988163874 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344003d00 con 0x7f9350104b20 2024-04-01T01:31:14.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.479+0000 7f93568ed640 1 -- 172.21.15.70:0/2988163874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f935019fe10 con 0x7f9350104b20 2024-04-01T01:31:14.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.480+0000 7f93568ed640 1 -- 172.21.15.70:0/2988163874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93500fd6f0 con 0x7f9350104b20 2024-04-01T01:31:14.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.481+0000 7f934d7fa640 1 -- 172.21.15.70:0/2988163874 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f9344046020 con 0x7f9350104b20 2024-04-01T01:31:14.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.481+0000 7f934d7fa640 1 --2- 172.21.15.70:0/2988163874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93380a24a0 0x7f93380a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:14.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.481+0000 7f934ffff640 1 --2- 172.21.15.70:0/2988163874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93380a24a0 0x7f93380a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:14.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.481+0000 7f934d7fa640 1 -- 172.21.15.70:0/2988163874 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f93440f0970 con 0x7f9350104b20 2024-04-01T01:31:14.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.482+0000 7f934ffff640 1 --2- 172.21.15.70:0/2988163874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93380a24a0 0x7f93380a4960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9340009b40 tx=0x7f9340005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:14.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.486+0000 7f934d7fa640 1 -- 172.21.15.70:0/2988163874 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f934404d020 con 0x7f9350104b20 2024-04-01T01:31:14.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:14.666+0000 7f93568ed640 1 -- 172.21.15.70:0/2988163874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f935002d170 con 0x7f9350104b20 2024-04-01T01:31:14.678 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:14 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2399033275' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-01T01:31:14.678 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:14 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:14.678 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:14 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:14.678 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:14 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:31:14.679 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:14 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:14.679 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:14 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:31:14.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:14 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2399033275' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-01T01:31:14.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:14 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:14.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:14 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:14.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:14 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:31:14.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:14 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:14.695 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:14 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:31:14.716 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:14 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2399033275' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-01T01:31:14.716 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:14 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:14.716 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:14 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:14.716 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:14 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:31:14.716 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:14 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:14.716 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:14 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:31:15.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.358+0000 7f934d7fa640 1 -- 172.21.15.70:0/2988163874 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7f93440bde60 con 0x7f9350104b20 2024-04-01T01:31:15.359 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' created 2024-04-01T01:31:15.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.362+0000 7f93568ed640 1 -- 172.21.15.70:0/2988163874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93380a24a0 msgr2=0x7f93380a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:15.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.362+0000 7f93568ed640 1 --2- 172.21.15.70:0/2988163874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93380a24a0 0x7f93380a4960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9340009b40 tx=0x7f9340005e50 comp rx=0 tx=0).stop 2024-04-01T01:31:15.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.362+0000 7f93568ed640 1 -- 172.21.15.70:0/2988163874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9350104b20 msgr2=0x7f9350197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:15.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.362+0000 7f93568ed640 1 --2- 172.21.15.70:0/2988163874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9350104b20 0x7f9350197f50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9344000c00 tx=0x7f9344004270 comp rx=0 tx=0).stop 2024-04-01T01:31:15.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.369+0000 7f93568ed640 1 -- 172.21.15.70:0/2988163874 shutdown_connections 2024-04-01T01:31:15.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.369+0000 7f93568ed640 1 --2- 172.21.15.70:0/2988163874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93380a24a0 0x7f93380a4960 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:15.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.369+0000 7f93568ed640 1 --2- 172.21.15.70:0/2988163874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9350107450 0x7f935019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:15.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.369+0000 7f93568ed640 1 --2- 172.21.15.70:0/2988163874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9350104b20 0x7f9350197f50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:15.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.369+0000 7f93568ed640 1 --2- 172.21.15.70:0/2988163874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93501021f0 0x7f9350197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:15.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.369+0000 7f93568ed640 1 -- 172.21.15.70:0/2988163874 >> 172.21.15.70:0/2988163874 conn(0x7f93500fbc00 msgr2=0x7f9350104060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:15.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.372+0000 7f93568ed640 1 -- 172.21.15.70:0/2988163874 shutdown_connections 2024-04-01T01:31:15.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.372+0000 7f93568ed640 1 -- 172.21.15.70:0/2988163874 wait complete. 2024-04-01T01:31:15.391 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-01T01:31:15.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.503+0000 7ff494979640 1 -- 172.21.15.70:0/2926970125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff490100670 msgr2=0x7ff490100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:15.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.503+0000 7ff494979640 1 --2- 172.21.15.70:0/2926970125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff490100670 0x7ff490100a90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff480009900 tx=0x7ff48002bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:15.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.503+0000 7ff494979640 1 -- 172.21.15.70:0/2926970125 shutdown_connections 2024-04-01T01:31:15.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.503+0000 7ff494979640 1 --2- 172.21.15.70:0/2926970125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff490102220 0x7ff490108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:15.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.503+0000 7ff494979640 1 --2- 172.21.15.70:0/2926970125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff490101860 0x7ff490101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:15.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.503+0000 7ff494979640 1 --2- 172.21.15.70:0/2926970125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff490100670 0x7ff490100a90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:15.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.503+0000 7ff494979640 1 -- 172.21.15.70:0/2926970125 >> 172.21.15.70:0/2926970125 conn(0x7ff4900fbc00 msgr2=0x7ff4900fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:15.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.503+0000 7ff494979640 1 -- 172.21.15.70:0/2926970125 shutdown_connections 2024-04-01T01:31:15.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.503+0000 7ff494979640 1 -- 172.21.15.70:0/2926970125 wait complete. 2024-04-01T01:31:15.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.503+0000 7ff494979640 1 Processor -- start 2024-04-01T01:31:15.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff494979640 1 -- start start 2024-04-01T01:31:15.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff494979640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff490100670 0x7ff490197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:15.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff494979640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff490101860 0x7ff490197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:15.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff494979640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff490102220 0x7ff49019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:15.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff494979640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff49010d320 con 0x7ff490100670 2024-04-01T01:31:15.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff494979640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff49010d1a0 con 0x7ff490101860 2024-04-01T01:31:15.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff494979640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff49010d4a0 con 0x7ff490102220 2024-04-01T01:31:15.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff48e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff490100670 0x7ff490197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:15.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff48e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff490100670 0x7ff490197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51030/0 (socket says 172.21.15.70:51030) 2024-04-01T01:31:15.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff48e59c640 1 -- 172.21.15.70:0/2767242737 learned_addr learned my addr 172.21.15.70:0/2767242737 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff487fff640 1 --2- 172.21.15.70:0/2767242737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff490101860 0x7ff490197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff48ed9d640 1 --2- 172.21.15.70:0/2767242737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff490102220 0x7ff49019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff48e59c640 1 -- 172.21.15.70:0/2767242737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff490102220 msgr2=0x7ff49019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff48e59c640 1 --2- 172.21.15.70:0/2767242737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff490102220 0x7ff49019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff48e59c640 1 -- 172.21.15.70:0/2767242737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff490101860 msgr2=0x7ff490197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff48e59c640 1 --2- 172.21.15.70:0/2767242737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff490101860 0x7ff490197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff48e59c640 1 -- 172.21.15.70:0/2767242737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff49019f660 con 0x7ff490100670 2024-04-01T01:31:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff487fff640 1 --2- 172.21.15.70:0/2767242737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff490101860 0x7ff490197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff48e59c640 1 --2- 172.21.15.70:0/2767242737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff490100670 0x7ff490197a40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff480000c00 tx=0x7ff4800040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.504+0000 7ff4877fe640 1 -- 172.21.15.70:0/2767242737 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480043020 con 0x7ff490100670 2024-04-01T01:31:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.505+0000 7ff494979640 1 -- 172.21.15.70:0/2767242737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff49019f8f0 con 0x7ff490100670 2024-04-01T01:31:15.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.505+0000 7ff4877fe640 1 -- 172.21.15.70:0/2767242737 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4800044c0 con 0x7ff490100670 2024-04-01T01:31:15.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.505+0000 7ff4877fe640 1 -- 172.21.15.70:0/2767242737 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480003b40 con 0x7ff490100670 2024-04-01T01:31:15.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.505+0000 7ff494979640 1 -- 172.21.15.70:0/2767242737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff49019fe40 con 0x7ff490100670 2024-04-01T01:31:15.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.506+0000 7ff494979640 1 -- 172.21.15.70:0/2767242737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff490107ab0 con 0x7ff490100670 2024-04-01T01:31:15.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.507+0000 7ff4877fe640 1 -- 172.21.15.70:0/2767242737 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7ff480046020 con 0x7ff490100670 2024-04-01T01:31:15.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.508+0000 7ff4877fe640 1 --2- 172.21.15.70:0/2767242737 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff47c0a24a0 0x7ff47c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:15.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.508+0000 7ff487fff640 1 --2- 172.21.15.70:0/2767242737 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff47c0a24a0 0x7ff47c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:15.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.508+0000 7ff4877fe640 1 -- 172.21.15.70:0/2767242737 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7ff4800f0bb0 con 0x7ff490100670 2024-04-01T01:31:15.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.508+0000 7ff487fff640 1 --2- 172.21.15.70:0/2767242737 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff47c0a24a0 0x7ff47c0a4960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff490198f50 tx=0x7ff47800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:15.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.512+0000 7ff4877fe640 1 -- 172.21.15.70:0/2767242737 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff48004d020 con 0x7ff490100670 2024-04-01T01:31:15.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:15.730+0000 7ff494979640 1 -- 172.21.15.70:0/2767242737 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff490062350 con 0x7ff490100670 2024-04-01T01:31:15.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:15 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2399033275' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-01T01:31:15.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:15 smithi189 ceph-mon[35753]: osdmap e82: 12 total, 12 up, 12 in 2024-04-01T01:31:15.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:15 smithi189 ceph-mon[35753]: pgmap v280: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:15.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:15 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2988163874' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-01T01:31:15.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:15 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2399033275' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-01T01:31:15.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:15 smithi070 ceph-mon[32179]: osdmap e82: 12 total, 12 up, 12 in 2024-04-01T01:31:15.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:15 smithi070 ceph-mon[32179]: pgmap v280: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:15.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:15 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2988163874' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-01T01:31:15.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:15 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2399033275' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-01T01:31:15.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:15 smithi081 ceph-mon[35142]: osdmap e82: 12 total, 12 up, 12 in 2024-04-01T01:31:15.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:15 smithi081 ceph-mon[35142]: pgmap v280: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:15.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:15 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2988163874' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-01T01:31:16.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.532+0000 7ff4877fe640 1 -- 172.21.15.70:0/2767242737 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7ff4800bdf00 con 0x7ff490100670 2024-04-01T01:31:16.533 INFO:teuthology.orchestra.run.smithi070.stderr:new fs with metadata pool 2 and data pool 3 2024-04-01T01:31:16.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.535+0000 7ff494979640 1 -- 172.21.15.70:0/2767242737 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff47c0a24a0 msgr2=0x7ff47c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:16.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.535+0000 7ff494979640 1 --2- 172.21.15.70:0/2767242737 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff47c0a24a0 0x7ff47c0a4960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff490198f50 tx=0x7ff47800b390 comp rx=0 tx=0).stop 2024-04-01T01:31:16.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.535+0000 7ff494979640 1 -- 172.21.15.70:0/2767242737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff490100670 msgr2=0x7ff490197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:16.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.535+0000 7ff494979640 1 --2- 172.21.15.70:0/2767242737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff490100670 0x7ff490197a40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff480000c00 tx=0x7ff4800040c0 comp rx=0 tx=0).stop 2024-04-01T01:31:16.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.535+0000 7ff494979640 1 -- 172.21.15.70:0/2767242737 shutdown_connections 2024-04-01T01:31:16.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.535+0000 7ff494979640 1 --2- 172.21.15.70:0/2767242737 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff47c0a24a0 0x7ff47c0a4960 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:16.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.535+0000 7ff494979640 1 --2- 172.21.15.70:0/2767242737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff490102220 0x7ff49019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:16.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.535+0000 7ff494979640 1 --2- 172.21.15.70:0/2767242737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff490101860 0x7ff490197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:16.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.535+0000 7ff494979640 1 --2- 172.21.15.70:0/2767242737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff490100670 0x7ff490197a40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:16.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.535+0000 7ff494979640 1 -- 172.21.15.70:0/2767242737 >> 172.21.15.70:0/2767242737 conn(0x7ff4900fbc00 msgr2=0x7ff490104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:16.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.535+0000 7ff494979640 1 -- 172.21.15.70:0/2767242737 shutdown_connections 2024-04-01T01:31:16.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.535+0000 7ff494979640 1 -- 172.21.15.70:0/2767242737 wait complete. 2024-04-01T01:31:16.548 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-01T01:31:16.549 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-01T01:31:16.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.656+0000 7f16b1bc7640 1 -- 172.21.15.70:0/4248276639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16ac100650 msgr2=0x7f16ac100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:16.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.656+0000 7f16b1bc7640 1 --2- 172.21.15.70:0/4248276639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16ac100650 0x7f16ac100a70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f169c009900 tx=0x7f169c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:16.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.656+0000 7f16b1bc7640 1 -- 172.21.15.70:0/4248276639 shutdown_connections 2024-04-01T01:31:16.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.656+0000 7f16b1bc7640 1 --2- 172.21.15.70:0/4248276639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16ac102200 0x7f16ac108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:16.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.656+0000 7f16b1bc7640 1 --2- 172.21.15.70:0/4248276639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16ac101840 0x7f16ac101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:16.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.656+0000 7f16b1bc7640 1 --2- 172.21.15.70:0/4248276639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16ac100650 0x7f16ac100a70 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:16.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.656+0000 7f16b1bc7640 1 -- 172.21.15.70:0/4248276639 >> 172.21.15.70:0/4248276639 conn(0x7f16ac0fbbc0 msgr2=0x7f16ac0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:16.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.657+0000 7f16b1bc7640 1 -- 172.21.15.70:0/4248276639 shutdown_connections 2024-04-01T01:31:16.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.657+0000 7f16b1bc7640 1 -- 172.21.15.70:0/4248276639 wait complete. 2024-04-01T01:31:16.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.657+0000 7f16b1bc7640 1 Processor -- start 2024-04-01T01:31:16.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.657+0000 7f16b1bc7640 1 -- start start 2024-04-01T01:31:16.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.657+0000 7f16b1bc7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16ac100650 0x7f16ac069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:16.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.657+0000 7f16b1bc7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16ac101840 0x7f16ac19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:16.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.657+0000 7f16b1bc7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16ac102200 0x7f16ac19ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:16.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.657+0000 7f16b1bc7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f16ac10c1a0 con 0x7f16ac102200 2024-04-01T01:31:16.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.657+0000 7f16b1bc7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f16ac10c020 con 0x7f16ac101840 2024-04-01T01:31:16.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.657+0000 7f16b1bc7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f16ac10c320 con 0x7f16ac100650 2024-04-01T01:31:16.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.658+0000 7f16abfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16ac102200 0x7f16ac19ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:16.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.658+0000 7f16abfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16ac102200 0x7f16ac19ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37196/0 (socket says 172.21.15.70:37196) 2024-04-01T01:31:16.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.658+0000 7f16abfff640 1 -- 172.21.15.70:0/3259802867 learned_addr learned my addr 172.21.15.70:0/3259802867 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:16.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.658+0000 7f16ab7fe640 1 --2- 172.21.15.70:0/3259802867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16ac100650 0x7f16ac069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:16.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.658+0000 7f16aaffd640 1 --2- 172.21.15.70:0/3259802867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16ac101840 0x7f16ac19c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:16.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.658+0000 7f16abfff640 1 -- 172.21.15.70:0/3259802867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16ac100650 msgr2=0x7f16ac069420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:16.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.658+0000 7f16abfff640 1 --2- 172.21.15.70:0/3259802867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16ac100650 0x7f16ac069420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:16.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.658+0000 7f16abfff640 1 -- 172.21.15.70:0/3259802867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16ac101840 msgr2=0x7f16ac19c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:16.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.658+0000 7f16abfff640 1 --2- 172.21.15.70:0/3259802867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16ac101840 0x7f16ac19c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:16.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.658+0000 7f16abfff640 1 -- 172.21.15.70:0/3259802867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16ac19f5c0 con 0x7f16ac102200 2024-04-01T01:31:16.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.658+0000 7f16abfff640 1 --2- 172.21.15.70:0/3259802867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16ac102200 0x7f16ac19ecb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f16a400cce0 tx=0x7f16a400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:16.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.658+0000 7f16a8ff9640 1 -- 172.21.15.70:0/3259802867 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a400f450 con 0x7f16ac102200 2024-04-01T01:31:16.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.658+0000 7f16a8ff9640 1 -- 172.21.15.70:0/3259802867 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16a400faa0 con 0x7f16ac102200 2024-04-01T01:31:16.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.658+0000 7f16b1bc7640 1 -- 172.21.15.70:0/3259802867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16ac19f850 con 0x7f16ac102200 2024-04-01T01:31:16.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.658+0000 7f16a8ff9640 1 -- 172.21.15.70:0/3259802867 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a4018b70 con 0x7f16ac102200 2024-04-01T01:31:16.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.658+0000 7f16b1bc7640 1 -- 172.21.15.70:0/3259802867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16ac19fd50 con 0x7f16ac102200 2024-04-01T01:31:16.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.659+0000 7f16b1bc7640 1 -- 172.21.15.70:0/3259802867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16ac107990 con 0x7f16ac102200 2024-04-01T01:31:16.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.660+0000 7f16a8ff9640 1 -- 172.21.15.70:0/3259802867 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f16a4018d10 con 0x7f16ac102200 2024-04-01T01:31:16.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.661+0000 7f16a8ff9640 1 --2- 172.21.15.70:0/3259802867 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f16940a24a0 0x7f16940a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:16.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.661+0000 7f16ab7fe640 1 --2- 172.21.15.70:0/3259802867 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f16940a24a0 0x7f16940a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:16.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.661+0000 7f16a8ff9640 1 -- 172.21.15.70:0/3259802867 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f16a40ceae0 con 0x7f16ac102200 2024-04-01T01:31:16.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.661+0000 7f16ab7fe640 1 --2- 172.21.15.70:0/3259802867 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f16940a24a0 0x7f16940a4960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f169c02c240 tx=0x7f169c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:16.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.665+0000 7f16a8ff9640 1 -- 172.21.15.70:0/3259802867 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f16a4014030 con 0x7f16ac102200 2024-04-01T01:31:16.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:16 smithi189 ceph-mon[35753]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-01T01:31:16.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:16 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2988163874' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-01T01:31:16.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:16 smithi189 ceph-mon[35753]: osdmap e83: 12 total, 12 up, 12 in 2024-04-01T01:31:16.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:16.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:31:16.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:16 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2767242737' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-01T01:31:16.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:16 smithi189 ceph-mon[35753]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-01T01:31:16.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:16 smithi070 ceph-mon[32179]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-01T01:31:16.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:16 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2988163874' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-01T01:31:16.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:16 smithi070 ceph-mon[32179]: osdmap e83: 12 total, 12 up, 12 in 2024-04-01T01:31:16.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:16.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:31:16.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:16 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2767242737' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-01T01:31:16.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:16 smithi070 ceph-mon[32179]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-01T01:31:16.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:16 smithi081 ceph-mon[35142]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-01T01:31:16.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:16 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2988163874' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-01T01:31:16.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:16 smithi081 ceph-mon[35142]: osdmap e83: 12 total, 12 up, 12 in 2024-04-01T01:31:16.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:31:16.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:31:16.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:31:16.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:16 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2767242737' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-01T01:31:16.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:16 smithi081 ceph-mon[35142]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-01T01:31:16.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:16.849+0000 7f16b1bc7640 1 -- 172.21.15.70:0/3259802867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f16ac02d370 con 0x7f16ac102200 2024-04-01T01:31:17.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.366+0000 7f16a8ff9640 1 -- 172.21.15.70:0/3259802867 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7f16a409bdf0 con 0x7f16ac102200 2024-04-01T01:31:17.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.369+0000 7f16b1bc7640 1 -- 172.21.15.70:0/3259802867 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f16940a24a0 msgr2=0x7f16940a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:17.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.369+0000 7f16b1bc7640 1 --2- 172.21.15.70:0/3259802867 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f16940a24a0 0x7f16940a4960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f169c02c240 tx=0x7f169c005fd0 comp rx=0 tx=0).stop 2024-04-01T01:31:17.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.369+0000 7f16b1bc7640 1 -- 172.21.15.70:0/3259802867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16ac102200 msgr2=0x7f16ac19ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:17.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.369+0000 7f16b1bc7640 1 --2- 172.21.15.70:0/3259802867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16ac102200 0x7f16ac19ecb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f16a400cce0 tx=0x7f16a400a620 comp rx=0 tx=0).stop 2024-04-01T01:31:17.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.369+0000 7f16b1bc7640 1 -- 172.21.15.70:0/3259802867 shutdown_connections 2024-04-01T01:31:17.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.369+0000 7f16b1bc7640 1 --2- 172.21.15.70:0/3259802867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16ac102200 0x7f16ac19ecb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:17.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.369+0000 7f16b1bc7640 1 --2- 172.21.15.70:0/3259802867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16ac101840 0x7f16ac19c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:17.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.369+0000 7f16b1bc7640 1 --2- 172.21.15.70:0/3259802867 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f16940a24a0 0x7f16940a4960 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:17.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.369+0000 7f16b1bc7640 1 --2- 172.21.15.70:0/3259802867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16ac100650 0x7f16ac069420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:17.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.369+0000 7f16b1bc7640 1 -- 172.21.15.70:0/3259802867 >> 172.21.15.70:0/3259802867 conn(0x7f16ac0fbbc0 msgr2=0x7f16ac104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:17.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.370+0000 7f16b1bc7640 1 -- 172.21.15.70:0/3259802867 shutdown_connections 2024-04-01T01:31:17.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.370+0000 7f16b1bc7640 1 -- 172.21.15.70:0/3259802867 wait complete. 2024-04-01T01:31:17.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-01T01:31:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.494+0000 7fb67045c640 1 -- 172.21.15.70:0/612523867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb668102970 msgr2=0x7fb668102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.494+0000 7fb67045c640 1 --2- 172.21.15.70:0/612523867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb668102970 0x7fb668102df0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb658009900 tx=0x7fb65802bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb67045c640 1 -- 172.21.15.70:0/612523867 shutdown_connections 2024-04-01T01:31:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb67045c640 1 --2- 172.21.15.70:0/612523867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb668103330 0x7fb668109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb67045c640 1 --2- 172.21.15.70:0/612523867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb668102970 0x7fb668102df0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb67045c640 1 --2- 172.21.15.70:0/612523867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb668101780 0x7fb668101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb67045c640 1 -- 172.21.15.70:0/612523867 >> 172.21.15.70:0/612523867 conn(0x7fb6680fcd30 msgr2=0x7fb6680ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb67045c640 1 -- 172.21.15.70:0/612523867 shutdown_connections 2024-04-01T01:31:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb67045c640 1 -- 172.21.15.70:0/612523867 wait complete. 2024-04-01T01:31:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb67045c640 1 Processor -- start 2024-04-01T01:31:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb67045c640 1 -- start start 2024-04-01T01:31:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb67045c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb668101780 0x7fb6681978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb67045c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb668102970 0x7fb668197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb67045c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb668103330 0x7fb66819ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb67045c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb66810d290 con 0x7fb668103330 2024-04-01T01:31:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb67045c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb66810d110 con 0x7fb668101780 2024-04-01T01:31:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb67045c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb66810d410 con 0x7fb668102970 2024-04-01T01:31:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb66e9f9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb668103330 0x7fb66819ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb66e9f9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb668103330 0x7fb66819ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37214/0 (socket says 172.21.15.70:37214) 2024-04-01T01:31:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.495+0000 7fb66e9f9640 1 -- 172.21.15.70:0/2932328871 learned_addr learned my addr 172.21.15.70:0/2932328871 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.496+0000 7fb66d9f7640 1 --2- 172.21.15.70:0/2932328871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb668102970 0x7fb668197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.496+0000 7fb66e1f8640 1 --2- 172.21.15.70:0/2932328871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb668101780 0x7fb6681978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.496+0000 7fb66e9f9640 1 -- 172.21.15.70:0/2932328871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb668102970 msgr2=0x7fb668197df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.496+0000 7fb66e9f9640 1 --2- 172.21.15.70:0/2932328871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb668102970 0x7fb668197df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.496+0000 7fb66e9f9640 1 -- 172.21.15.70:0/2932328871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb668101780 msgr2=0x7fb6681978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.496+0000 7fb66e9f9640 1 --2- 172.21.15.70:0/2932328871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb668101780 0x7fb6681978b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.496+0000 7fb66e9f9640 1 -- 172.21.15.70:0/2932328871 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66819f4d0 con 0x7fb668103330 2024-04-01T01:31:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.496+0000 7fb66e1f8640 1 --2- 172.21.15.70:0/2932328871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb668101780 0x7fb6681978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:31:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.496+0000 7fb66e9f9640 1 --2- 172.21.15.70:0/2932328871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb668103330 0x7fb66819ee60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb66000c970 tx=0x7fb66000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.496+0000 7fb65f7fe640 1 -- 172.21.15.70:0/2932328871 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb660007bb0 con 0x7fb668103330 2024-04-01T01:31:17.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.496+0000 7fb67045c640 1 -- 172.21.15.70:0/2932328871 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb66819f7c0 con 0x7fb668103330 2024-04-01T01:31:17.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.496+0000 7fb65f7fe640 1 -- 172.21.15.70:0/2932328871 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb660007d50 con 0x7fb668103330 2024-04-01T01:31:17.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.496+0000 7fb65f7fe640 1 -- 172.21.15.70:0/2932328871 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb660018700 con 0x7fb668103330 2024-04-01T01:31:17.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.496+0000 7fb67045c640 1 -- 172.21.15.70:0/2932328871 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb66819fd10 con 0x7fb668103330 2024-04-01T01:31:17.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.497+0000 7fb67045c640 1 -- 172.21.15.70:0/2932328871 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb668108a40 con 0x7fb668103330 2024-04-01T01:31:17.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.498+0000 7fb65f7fe640 1 -- 172.21.15.70:0/2932328871 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fb66000fa20 con 0x7fb668103330 2024-04-01T01:31:17.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.499+0000 7fb65f7fe640 1 --2- 172.21.15.70:0/2932328871 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6540a24a0 0x7fb6540a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:17.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.499+0000 7fb66e1f8640 1 --2- 172.21.15.70:0/2932328871 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6540a24a0 0x7fb6540a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:17.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.499+0000 7fb65f7fe640 1 -- 172.21.15.70:0/2932328871 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7fb6600ce8f0 con 0x7fb668103330 2024-04-01T01:31:17.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.499+0000 7fb66e1f8640 1 --2- 172.21.15.70:0/2932328871 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6540a24a0 0x7fb6540a4960 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb6681027d0 tx=0x7fb664008370 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:17.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.503+0000 7fb65f7fe640 1 -- 172.21.15.70:0/2932328871 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb660014030 con 0x7fb668103330 2024-04-01T01:31:17.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:17.680+0000 7fb67045c640 1 -- 172.21.15.70:0/2932328871 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb668062350 con 0x7fb668103330 2024-04-01T01:31:17.748 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:17 smithi189 ceph-mon[35753]: osdmap e84: 12 total, 12 up, 12 in 2024-04-01T01:31:17.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:17 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2767242737' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-01T01:31:17.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:17 smithi189 ceph-mon[35753]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-01T01:31:17.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:17 smithi189 ceph-mon[35753]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-01T01:31:17.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:17 smithi189 ceph-mon[35753]: pgmap v283: 97 pgs: 4 creating+activating, 25 creating+peering, 55 unknown, 13 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:17.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:17 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3259802867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-01T01:31:17.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:17 smithi070 ceph-mon[32179]: osdmap e84: 12 total, 12 up, 12 in 2024-04-01T01:31:17.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:17 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2767242737' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-01T01:31:17.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:17 smithi070 ceph-mon[32179]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-01T01:31:17.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:17 smithi070 ceph-mon[32179]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-01T01:31:17.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:17 smithi070 ceph-mon[32179]: pgmap v283: 97 pgs: 4 creating+activating, 25 creating+peering, 55 unknown, 13 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:17.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:17 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3259802867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-01T01:31:17.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:17 smithi081 ceph-mon[35142]: osdmap e84: 12 total, 12 up, 12 in 2024-04-01T01:31:17.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:17 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2767242737' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-01T01:31:17.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:17 smithi081 ceph-mon[35142]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-01T01:31:17.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:17 smithi081 ceph-mon[35142]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-01T01:31:17.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:17 smithi081 ceph-mon[35142]: pgmap v283: 97 pgs: 4 creating+activating, 25 creating+peering, 55 unknown, 13 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-01T01:31:17.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:17 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3259802867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-01T01:31:18.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:18 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3259802867' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-01T01:31:18.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:18 smithi189 ceph-mon[35753]: osdmap e85: 12 total, 12 up, 12 in 2024-04-01T01:31:18.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:18 smithi189 ceph-mon[35753]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-01T01:31:18.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:18 smithi189 ceph-mon[35753]: Cluster is now healthy 2024-04-01T01:31:18.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:18 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] up:active 2024-04-01T01:31:18.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:18 smithi189 ceph-mon[35753]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-01T01:31:18.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:18 smithi189 ceph-mon[35753]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-01T01:31:18.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:18 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2932328871' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-01T01:31:18.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:18 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3259802867' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-01T01:31:18.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:18 smithi070 ceph-mon[32179]: osdmap e85: 12 total, 12 up, 12 in 2024-04-01T01:31:18.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:18 smithi070 ceph-mon[32179]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-01T01:31:18.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:18 smithi070 ceph-mon[32179]: Cluster is now healthy 2024-04-01T01:31:18.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:18 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] up:active 2024-04-01T01:31:18.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:18 smithi070 ceph-mon[32179]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-01T01:31:18.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:18 smithi070 ceph-mon[32179]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-01T01:31:18.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:18 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2932328871' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-01T01:31:18.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:18 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3259802867' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-01T01:31:18.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:18 smithi081 ceph-mon[35142]: osdmap e85: 12 total, 12 up, 12 in 2024-04-01T01:31:18.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:18 smithi081 ceph-mon[35142]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-01T01:31:18.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:18 smithi081 ceph-mon[35142]: Cluster is now healthy 2024-04-01T01:31:18.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:18 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] up:active 2024-04-01T01:31:18.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:18 smithi081 ceph-mon[35142]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-01T01:31:18.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:18 smithi081 ceph-mon[35142]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-01T01:31:18.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:18 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2932328871' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-01T01:31:19.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.548+0000 7fb65f7fe640 1 -- 172.21.15.70:0/2932328871 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7fb66009ba80 con 0x7fb668103330 2024-04-01T01:31:19.549 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data_ec' created 2024-04-01T01:31:19.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.551+0000 7fb67045c640 1 -- 172.21.15.70:0/2932328871 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6540a24a0 msgr2=0x7fb6540a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:19.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.551+0000 7fb67045c640 1 --2- 172.21.15.70:0/2932328871 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6540a24a0 0x7fb6540a4960 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb6681027d0 tx=0x7fb664008370 comp rx=0 tx=0).stop 2024-04-01T01:31:19.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.552+0000 7fb67045c640 1 -- 172.21.15.70:0/2932328871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb668103330 msgr2=0x7fb66819ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:19.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.552+0000 7fb67045c640 1 --2- 172.21.15.70:0/2932328871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb668103330 0x7fb66819ee60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb66000c970 tx=0x7fb66000ce30 comp rx=0 tx=0).stop 2024-04-01T01:31:19.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.552+0000 7fb67045c640 1 -- 172.21.15.70:0/2932328871 shutdown_connections 2024-04-01T01:31:19.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.552+0000 7fb67045c640 1 --2- 172.21.15.70:0/2932328871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb668103330 0x7fb66819ee60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:19.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.552+0000 7fb67045c640 1 --2- 172.21.15.70:0/2932328871 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6540a24a0 0x7fb6540a4960 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:19.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.552+0000 7fb67045c640 1 --2- 172.21.15.70:0/2932328871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb668102970 0x7fb668197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:19.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.552+0000 7fb67045c640 1 --2- 172.21.15.70:0/2932328871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb668101780 0x7fb6681978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:19.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.552+0000 7fb67045c640 1 -- 172.21.15.70:0/2932328871 >> 172.21.15.70:0/2932328871 conn(0x7fb6680fcd30 msgr2=0x7fb668105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:19.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.552+0000 7fb67045c640 1 -- 172.21.15.70:0/2932328871 shutdown_connections 2024-04-01T01:31:19.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.552+0000 7fb67045c640 1 -- 172.21.15.70:0/2932328871 wait complete. 2024-04-01T01:31:19.570 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-01T01:31:19.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.680+0000 7f34bc42a640 1 -- 172.21.15.70:0/336476491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34b41021f0 msgr2=0x7f34b41045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:19.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.680+0000 7f34bc42a640 1 --2- 172.21.15.70:0/336476491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34b41021f0 0x7f34b41045e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009900 tx=0x7f34b002bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:19.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.680+0000 7f34bc42a640 1 -- 172.21.15.70:0/336476491 shutdown_connections 2024-04-01T01:31:19.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.680+0000 7f34bc42a640 1 --2- 172.21.15.70:0/336476491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b4107450 0x7f34b41098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:19.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.680+0000 7f34bc42a640 1 --2- 172.21.15.70:0/336476491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34b4104b20 0x7f34b4106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:19.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.680+0000 7f34bc42a640 1 --2- 172.21.15.70:0/336476491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34b41021f0 0x7f34b41045e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:19.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.680+0000 7f34bc42a640 1 -- 172.21.15.70:0/336476491 >> 172.21.15.70:0/336476491 conn(0x7f34b40fbc00 msgr2=0x7f34b40fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:19.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.680+0000 7f34bc42a640 1 -- 172.21.15.70:0/336476491 shutdown_connections 2024-04-01T01:31:19.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.680+0000 7f34bc42a640 1 -- 172.21.15.70:0/336476491 wait complete. 2024-04-01T01:31:19.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34bc42a640 1 Processor -- start 2024-04-01T01:31:19.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34bc42a640 1 -- start start 2024-04-01T01:31:19.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34bc42a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34b41021f0 0x7f34b4197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:19.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34bc42a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34b4104b20 0x7f34b4197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:19.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34bc42a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b4107450 0x7f34b419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:19.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34bc42a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f34b410d2c0 con 0x7f34b4104b20 2024-04-01T01:31:19.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34bc42a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f34b410d140 con 0x7f34b41021f0 2024-04-01T01:31:19.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34bc42a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f34b410d440 con 0x7f34b4107450 2024-04-01T01:31:19.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34b99c5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34b4104b20 0x7f34b4197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:19.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34b99c5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34b4104b20 0x7f34b4197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37232/0 (socket says 172.21.15.70:37232) 2024-04-01T01:31:19.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34b99c5640 1 -- 172.21.15.70:0/2370546956 learned_addr learned my addr 172.21.15.70:0/2370546956 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:19.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34ba1c6640 1 --2- 172.21.15.70:0/2370546956 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34b41021f0 0x7f34b4197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:19.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34b99c5640 1 -- 172.21.15.70:0/2370546956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b4107450 msgr2=0x7f34b419efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:19.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34b99c5640 1 --2- 172.21.15.70:0/2370546956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b4107450 0x7f34b419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:19.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34b99c5640 1 -- 172.21.15.70:0/2370546956 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34b41021f0 msgr2=0x7f34b4197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:19.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34b99c5640 1 --2- 172.21.15.70:0/2370546956 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34b41021f0 0x7f34b4197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:19.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34b99c5640 1 -- 172.21.15.70:0/2370546956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b419f650 con 0x7f34b4104b20 2024-04-01T01:31:19.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34b99c5640 1 --2- 172.21.15.70:0/2370546956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34b4104b20 0x7f34b4197f70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f34a400e9c0 tx=0x7f34a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:19.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34ab7fe640 1 -- 172.21.15.70:0/2370546956 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a400cd80 con 0x7f34b4104b20 2024-04-01T01:31:19.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34ab7fe640 1 -- 172.21.15.70:0/2370546956 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f34a40054d0 con 0x7f34b4104b20 2024-04-01T01:31:19.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34bc42a640 1 -- 172.21.15.70:0/2370546956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b419f940 con 0x7f34b4104b20 2024-04-01T01:31:19.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34ab7fe640 1 -- 172.21.15.70:0/2370546956 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a4010430 con 0x7f34b4104b20 2024-04-01T01:31:19.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.681+0000 7f34bc42a640 1 -- 172.21.15.70:0/2370546956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b419fee0 con 0x7f34b4104b20 2024-04-01T01:31:19.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.682+0000 7f34bc42a640 1 -- 172.21.15.70:0/2370546956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34b40fd6f0 con 0x7f34b4104b20 2024-04-01T01:31:19.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.683+0000 7f34ab7fe640 1 -- 172.21.15.70:0/2370546956 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f34a4004fb0 con 0x7f34b4104b20 2024-04-01T01:31:19.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.684+0000 7f34ab7fe640 1 --2- 172.21.15.70:0/2370546956 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f34a00a2520 0x7f34a00a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:19.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.684+0000 7f34ba1c6640 1 --2- 172.21.15.70:0/2370546956 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f34a00a2520 0x7f34a00a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:19.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.684+0000 7f34ab7fe640 1 -- 172.21.15.70:0/2370546956 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7f34a40cef50 con 0x7f34b4104b20 2024-04-01T01:31:19.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.685+0000 7f34ba1c6640 1 --2- 172.21.15.70:0/2370546956 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f34a00a2520 0x7f34a00a49e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009230 tx=0x7f34b0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:19.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.688+0000 7f34ab7fe640 1 -- 172.21.15.70:0/2370546956 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f34a4053280 con 0x7f34b4104b20 2024-04-01T01:31:19.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:19 smithi081 ceph-mon[35142]: pgmap v286: 97 pgs: 4 creating+activating, 25 creating+peering, 23 unknown, 45 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 6 op/s 2024-04-01T01:31:19.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:19 smithi081 ceph-mon[35142]: osdmap e86: 12 total, 12 up, 12 in 2024-04-01T01:31:19.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:19.861+0000 7f34bc42a640 1 -- 172.21.15.70:0/2370546956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f34b402d170 con 0x7f34b4104b20 2024-04-01T01:31:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:19 smithi189 ceph-mon[35753]: pgmap v286: 97 pgs: 4 creating+activating, 25 creating+peering, 23 unknown, 45 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 6 op/s 2024-04-01T01:31:19.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:19 smithi189 ceph-mon[35753]: osdmap e86: 12 total, 12 up, 12 in 2024-04-01T01:31:20.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:19 smithi070 ceph-mon[32179]: pgmap v286: 97 pgs: 4 creating+activating, 25 creating+peering, 23 unknown, 45 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 6 op/s 2024-04-01T01:31:20.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:19 smithi070 ceph-mon[32179]: osdmap e86: 12 total, 12 up, 12 in 2024-04-01T01:31:20.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.550+0000 7f34ab7fe640 1 -- 172.21.15.70:0/2370546956 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7f34a409bef0 con 0x7f34b4104b20 2024-04-01T01:31:20.550 INFO:teuthology.orchestra.run.smithi070.stderr:set pool 4 allow_ec_overwrites to true 2024-04-01T01:31:20.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.554+0000 7f34bc42a640 1 -- 172.21.15.70:0/2370546956 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f34a00a2520 msgr2=0x7f34a00a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:20.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.554+0000 7f34bc42a640 1 --2- 172.21.15.70:0/2370546956 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f34a00a2520 0x7f34a00a49e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009230 tx=0x7f34b0005fd0 comp rx=0 tx=0).stop 2024-04-01T01:31:20.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.554+0000 7f34bc42a640 1 -- 172.21.15.70:0/2370546956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34b4104b20 msgr2=0x7f34b4197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:20.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.554+0000 7f34bc42a640 1 --2- 172.21.15.70:0/2370546956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34b4104b20 0x7f34b4197f70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f34a400e9c0 tx=0x7f34a400ee80 comp rx=0 tx=0).stop 2024-04-01T01:31:20.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.554+0000 7f34bc42a640 1 -- 172.21.15.70:0/2370546956 shutdown_connections 2024-04-01T01:31:20.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.554+0000 7f34bc42a640 1 --2- 172.21.15.70:0/2370546956 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f34a00a2520 0x7f34a00a49e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:20.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.554+0000 7f34bc42a640 1 --2- 172.21.15.70:0/2370546956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b4107450 0x7f34b419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:20.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.554+0000 7f34bc42a640 1 --2- 172.21.15.70:0/2370546956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34b4104b20 0x7f34b4197f70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:20.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.554+0000 7f34bc42a640 1 --2- 172.21.15.70:0/2370546956 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34b41021f0 0x7f34b4197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:20.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.554+0000 7f34bc42a640 1 -- 172.21.15.70:0/2370546956 >> 172.21.15.70:0/2370546956 conn(0x7f34b40fbc00 msgr2=0x7f34b4102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:20.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.554+0000 7f34bc42a640 1 -- 172.21.15.70:0/2370546956 shutdown_connections 2024-04-01T01:31:20.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.554+0000 7f34bc42a640 1 -- 172.21.15.70:0/2370546956 wait complete. 2024-04-01T01:31:20.576 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-01T01:31:20.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.681+0000 7f5f94b02640 1 -- 172.21.15.70:0/2717176119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 msgr2=0x7f5f90108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:20.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.681+0000 7f5f94b02640 1 --2- 172.21.15.70:0/2717176119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 0x7f5f90108a90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5f88009ca0 tx=0x7f5f8802c0f0 comp rx=0 tx=0).stop 2024-04-01T01:31:20.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.682+0000 7f5f94b02640 1 -- 172.21.15.70:0/2717176119 shutdown_connections 2024-04-01T01:31:20.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.682+0000 7f5f94b02640 1 --2- 172.21.15.70:0/2717176119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 0x7f5f90108a90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:20.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.682+0000 7f5f94b02640 1 --2- 172.21.15.70:0/2717176119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f90101840 0x7f5f90101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:20.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.682+0000 7f5f94b02640 1 --2- 172.21.15.70:0/2717176119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f90100650 0x7f5f90100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:20.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.682+0000 7f5f94b02640 1 -- 172.21.15.70:0/2717176119 >> 172.21.15.70:0/2717176119 conn(0x7f5f900fbbe0 msgr2=0x7f5f900fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:20.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.682+0000 7f5f94b02640 1 -- 172.21.15.70:0/2717176119 shutdown_connections 2024-04-01T01:31:20.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.682+0000 7f5f94b02640 1 -- 172.21.15.70:0/2717176119 wait complete. 2024-04-01T01:31:20.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.682+0000 7f5f94b02640 1 Processor -- start 2024-04-01T01:31:20.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.682+0000 7f5f94b02640 1 -- start start 2024-04-01T01:31:20.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.682+0000 7f5f94b02640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f90100650 0x7f5f90069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:20.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.682+0000 7f5f94b02640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f90101840 0x7f5f9019c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:20.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.682+0000 7f5f94b02640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 0x7f5f9019ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:20.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.682+0000 7f5f94b02640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9010c180 con 0x7f5f90102200 2024-04-01T01:31:20.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.682+0000 7f5f94b02640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9010c000 con 0x7f5f90101840 2024-04-01T01:31:20.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.682+0000 7f5f94b02640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9010c300 con 0x7f5f90100650 2024-04-01T01:31:20.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.683+0000 7f5f8ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 0x7f5f9019ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:20.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.683+0000 7f5f8ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 0x7f5f9019ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37264/0 (socket says 172.21.15.70:37264) 2024-04-01T01:31:20.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.683+0000 7f5f8ed9d640 1 -- 172.21.15.70:0/1268795572 learned_addr learned my addr 172.21.15.70:0/1268795572 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:20.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.683+0000 7f5f8dd9b640 1 --2- 172.21.15.70:0/1268795572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f90101840 0x7f5f9019c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:20.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.683+0000 7f5f8ed9d640 1 -- 172.21.15.70:0/1268795572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f90100650 msgr2=0x7f5f90069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:20.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.683+0000 7f5f8ed9d640 1 --2- 172.21.15.70:0/1268795572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f90100650 0x7f5f90069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:20.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.683+0000 7f5f8ed9d640 1 -- 172.21.15.70:0/1268795572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f90101840 msgr2=0x7f5f9019c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:20.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.683+0000 7f5f8ed9d640 1 --2- 172.21.15.70:0/1268795572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f90101840 0x7f5f9019c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:20.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.683+0000 7f5f8ed9d640 1 -- 172.21.15.70:0/1268795572 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f9019f5d0 con 0x7f5f90102200 2024-04-01T01:31:20.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.683+0000 7f5f8dd9b640 1 --2- 172.21.15.70:0/1268795572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f90101840 0x7f5f9019c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:20.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.683+0000 7f5f8ed9d640 1 --2- 172.21.15.70:0/1268795572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 0x7f5f9019ed10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5f8802c700 tx=0x7f5f880328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:20.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.683+0000 7f5f7f7fe640 1 -- 172.21.15.70:0/1268795572 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f88041020 con 0x7f5f90102200 2024-04-01T01:31:20.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.683+0000 7f5f7f7fe640 1 -- 172.21.15.70:0/1268795572 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f88003e20 con 0x7f5f90102200 2024-04-01T01:31:20.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.683+0000 7f5f94b02640 1 -- 172.21.15.70:0/1268795572 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f9019f860 con 0x7f5f90102200 2024-04-01T01:31:20.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.683+0000 7f5f7f7fe640 1 -- 172.21.15.70:0/1268795572 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f88039040 con 0x7f5f90102200 2024-04-01T01:31:20.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.683+0000 7f5f94b02640 1 -- 172.21.15.70:0/1268795572 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f9019fd50 con 0x7f5f90102200 2024-04-01T01:31:20.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.684+0000 7f5f94b02640 1 -- 172.21.15.70:0/1268795572 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f90107a90 con 0x7f5f90102200 2024-04-01T01:31:20.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.685+0000 7f5f7f7fe640 1 -- 172.21.15.70:0/1268795572 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f5f88044020 con 0x7f5f90102200 2024-04-01T01:31:20.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.686+0000 7f5f7f7fe640 1 --2- 172.21.15.70:0/1268795572 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f780a24a0 0x7f5f780a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:20.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.686+0000 7f5f8e59c640 1 --2- 172.21.15.70:0/1268795572 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f780a24a0 0x7f5f780a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:20.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.686+0000 7f5f7f7fe640 1 -- 172.21.15.70:0/1268795572 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f5f880f0c30 con 0x7f5f90102200 2024-04-01T01:31:20.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.686+0000 7f5f8e59c640 1 --2- 172.21.15.70:0/1268795572 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f780a24a0 0x7f5f780a4960 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5f84005fd0 tx=0x7f5f8400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:20.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.690+0000 7f5f7f7fe640 1 -- 172.21.15.70:0/1268795572 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5f8804b020 con 0x7f5f90102200 2024-04-01T01:31:20.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:20 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2932328871' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-01T01:31:20.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:20 smithi081 ceph-mon[35142]: osdmap e87: 12 total, 12 up, 12 in 2024-04-01T01:31:20.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:20 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2370546956' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-01T01:31:20.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:20.901+0000 7f5f94b02640 1 -- 172.21.15.70:0/1268795572 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f5f90062350 con 0x7f5f90102200 2024-04-01T01:31:20.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:20 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2932328871' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-01T01:31:20.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:20 smithi189 ceph-mon[35753]: osdmap e87: 12 total, 12 up, 12 in 2024-04-01T01:31:20.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:20 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2370546956' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-01T01:31:21.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:20 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2932328871' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-01T01:31:21.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:20 smithi070 ceph-mon[32179]: osdmap e87: 12 total, 12 up, 12 in 2024-04-01T01:31:21.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:20 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2370546956' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-01T01:31:21.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.557+0000 7f5f7f7fe640 1 -- 172.21.15.70:0/1268795572 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f5f880bdbd0 con 0x7f5f90102200 2024-04-01T01:31:21.558 INFO:teuthology.orchestra.run.smithi070.stderr:added data pool 4 to fsmap 2024-04-01T01:31:21.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.560+0000 7f5f94b02640 1 -- 172.21.15.70:0/1268795572 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f780a24a0 msgr2=0x7f5f780a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:21.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.560+0000 7f5f94b02640 1 --2- 172.21.15.70:0/1268795572 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f780a24a0 0x7f5f780a4960 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5f84005fd0 tx=0x7f5f8400b410 comp rx=0 tx=0).stop 2024-04-01T01:31:21.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.560+0000 7f5f94b02640 1 -- 172.21.15.70:0/1268795572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 msgr2=0x7f5f9019ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:21.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.560+0000 7f5f94b02640 1 --2- 172.21.15.70:0/1268795572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 0x7f5f9019ed10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5f8802c700 tx=0x7f5f880328a0 comp rx=0 tx=0).stop 2024-04-01T01:31:21.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.561+0000 7f5f94b02640 1 -- 172.21.15.70:0/1268795572 shutdown_connections 2024-04-01T01:31:21.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.561+0000 7f5f94b02640 1 --2- 172.21.15.70:0/1268795572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 0x7f5f9019ed10 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:21.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.561+0000 7f5f94b02640 1 --2- 172.21.15.70:0/1268795572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f90101840 0x7f5f9019c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:21.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.561+0000 7f5f94b02640 1 --2- 172.21.15.70:0/1268795572 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f780a24a0 0x7f5f780a4960 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:21.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.561+0000 7f5f94b02640 1 --2- 172.21.15.70:0/1268795572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f90100650 0x7f5f90069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:21.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.561+0000 7f5f94b02640 1 -- 172.21.15.70:0/1268795572 >> 172.21.15.70:0/1268795572 conn(0x7f5f900fbbe0 msgr2=0x7f5f90104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:21.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.561+0000 7f5f94b02640 1 -- 172.21.15.70:0/1268795572 shutdown_connections 2024-04-01T01:31:21.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.561+0000 7f5f94b02640 1 -- 172.21.15.70:0/1268795572 wait complete. 2024-04-01T01:31:21.575 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:31:21.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.681+0000 7f7765a54640 1 -- 172.21.15.70:0/4095753097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77601074b0 msgr2=0x7f77601098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:21.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.681+0000 7f7765a54640 1 --2- 172.21.15.70:0/4095753097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77601074b0 0x7f77601098a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7758009900 tx=0x7f775802bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:21.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.682+0000 7f7765a54640 1 -- 172.21.15.70:0/4095753097 shutdown_connections 2024-04-01T01:31:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.682+0000 7f7765a54640 1 --2- 172.21.15.70:0/4095753097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77601074b0 0x7f77601098a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.682+0000 7f7765a54640 1 --2- 172.21.15.70:0/4095753097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7760104b80 0x7f7760106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.682+0000 7f7765a54640 1 --2- 172.21.15.70:0/4095753097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7760102250 0x7f7760104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.682+0000 7f7765a54640 1 -- 172.21.15.70:0/4095753097 >> 172.21.15.70:0/4095753097 conn(0x7f77600fbbe0 msgr2=0x7f77600fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.682+0000 7f7765a54640 1 -- 172.21.15.70:0/4095753097 shutdown_connections 2024-04-01T01:31:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.682+0000 7f7765a54640 1 -- 172.21.15.70:0/4095753097 wait complete. 2024-04-01T01:31:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.682+0000 7f7765a54640 1 Processor -- start 2024-04-01T01:31:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.682+0000 7f7765a54640 1 -- start start 2024-04-01T01:31:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.682+0000 7f7765a54640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7760102250 0x7f7760199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.682+0000 7f7765a54640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7760104b80 0x7f776019a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.682+0000 7f7765a54640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77601074b0 0x7f77601a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.682+0000 7f7765a54640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f776010d310 con 0x7f7760102250 2024-04-01T01:31:21.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.682+0000 7f7765a54640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f776010d190 con 0x7f77601074b0 2024-04-01T01:31:21.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.682+0000 7f7765a54640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f776010d490 con 0x7f7760104b80 2024-04-01T01:31:21.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.683+0000 7f775effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7760102250 0x7f7760199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:21.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.683+0000 7f775effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7760102250 0x7f7760199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37286/0 (socket says 172.21.15.70:37286) 2024-04-01T01:31:21.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.683+0000 7f775effd640 1 -- 172.21.15.70:0/382964281 learned_addr learned my addr 172.21.15.70:0/382964281 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:21.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.683+0000 7f775f7fe640 1 --2- 172.21.15.70:0/382964281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77601074b0 0x7f77601a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:21.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.683+0000 7f775e7fc640 1 --2- 172.21.15.70:0/382964281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7760104b80 0x7f776019a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:21.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.683+0000 7f775effd640 1 -- 172.21.15.70:0/382964281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7760104b80 msgr2=0x7f776019a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:21.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.683+0000 7f775effd640 1 --2- 172.21.15.70:0/382964281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7760104b80 0x7f776019a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:21.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.683+0000 7f775effd640 1 -- 172.21.15.70:0/382964281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77601074b0 msgr2=0x7f77601a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:21.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.683+0000 7f775effd640 1 --2- 172.21.15.70:0/382964281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77601074b0 0x7f77601a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:21.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.683+0000 7f775effd640 1 -- 172.21.15.70:0/382964281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77601a17c0 con 0x7f7760102250 2024-04-01T01:31:21.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.683+0000 7f775effd640 1 --2- 172.21.15.70:0/382964281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7760102250 0x7f7760199ba0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7750009d50 tx=0x7f775000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:21.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.683+0000 7f7764a52640 1 -- 172.21.15.70:0/382964281 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7750003b70 con 0x7f7760102250 2024-04-01T01:31:21.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.683+0000 7f7765a54640 1 -- 172.21.15.70:0/382964281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77601a1ab0 con 0x7f7760102250 2024-04-01T01:31:21.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.683+0000 7f7764a52640 1 -- 172.21.15.70:0/382964281 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7750003d10 con 0x7f7760102250 2024-04-01T01:31:21.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.683+0000 7f7764a52640 1 -- 172.21.15.70:0/382964281 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77500108f0 con 0x7f7760102250 2024-04-01T01:31:21.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.683+0000 7f7765a54640 1 -- 172.21.15.70:0/382964281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77601a2000 con 0x7f7760102250 2024-04-01T01:31:21.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.684+0000 7f7765a54640 1 -- 172.21.15.70:0/382964281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7760108fa0 con 0x7f7760102250 2024-04-01T01:31:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.685+0000 7f7764a52640 1 -- 172.21.15.70:0/382964281 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f7750003ec0 con 0x7f7760102250 2024-04-01T01:31:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.686+0000 7f7764a52640 1 --2- 172.21.15.70:0/382964281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77480a24a0 0x7f77480a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.686+0000 7f775e7fc640 1 --2- 172.21.15.70:0/382964281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77480a24a0 0x7f77480a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.686+0000 7f7764a52640 1 -- 172.21.15.70:0/382964281 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f77500cf070 con 0x7f7760102250 2024-04-01T01:31:21.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.686+0000 7f775e7fc640 1 --2- 172.21.15.70:0/382964281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77480a24a0 0x7f77480a4960 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f776019b0b0 tx=0x7f7754005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:21.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.690+0000 7f7764a52640 1 -- 172.21.15.70:0/382964281 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f775009bff0 con 0x7f7760102250 2024-04-01T01:31:21.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:21 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2370546956' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-01T01:31:21.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:21 smithi081 ceph-mon[35142]: osdmap e88: 12 total, 12 up, 12 in 2024-04-01T01:31:21.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:21 smithi081 ceph-mon[35142]: pgmap v289: 129 pgs: 4 creating+activating, 24 creating+peering, 24 unknown, 77 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 8 op/s 2024-04-01T01:31:21.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:21 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1268795572' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-01T01:31:21.828 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:31:21 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-b[65004]: 2024-04-01T01:31:21.343+0000 7ff9913f6700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-01T01:31:21.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.908+0000 7f7765a54640 1 -- 172.21.15.70:0/382964281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f776002d170 con 0x7f7760102250 2024-04-01T01:31:21.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.909+0000 7f7764a52640 1 -- 172.21.15.70:0/382964281 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11794 (secure 0 0 0) 0x7f775009b8f0 con 0x7f7760102250 2024-04-01T01:31:21.909 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:21.910 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24418,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24425,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24451,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:31:20.903788+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24464,"qdb_cluster":[24464]},"id":1}]} 2024-04-01T01:31:21.910 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 17 2024-04-01T01:31:21.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.911+0000 7f7765a54640 1 -- 172.21.15.70:0/382964281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77480a24a0 msgr2=0x7f77480a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:21.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.911+0000 7f7765a54640 1 --2- 172.21.15.70:0/382964281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77480a24a0 0x7f77480a4960 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f776019b0b0 tx=0x7f7754005f90 comp rx=0 tx=0).stop 2024-04-01T01:31:21.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.911+0000 7f7765a54640 1 -- 172.21.15.70:0/382964281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7760102250 msgr2=0x7f7760199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:21.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.911+0000 7f7765a54640 1 --2- 172.21.15.70:0/382964281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7760102250 0x7f7760199ba0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7750009d50 tx=0x7f775000c6a0 comp rx=0 tx=0).stop 2024-04-01T01:31:21.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.911+0000 7f7765a54640 1 -- 172.21.15.70:0/382964281 shutdown_connections 2024-04-01T01:31:21.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.911+0000 7f7765a54640 1 --2- 172.21.15.70:0/382964281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77601074b0 0x7f77601a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:21.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.911+0000 7f7765a54640 1 --2- 172.21.15.70:0/382964281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77480a24a0 0x7f77480a4960 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:21.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.911+0000 7f7765a54640 1 --2- 172.21.15.70:0/382964281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7760104b80 0x7f776019a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:21.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.911+0000 7f7765a54640 1 --2- 172.21.15.70:0/382964281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7760102250 0x7f7760199ba0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:21.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.911+0000 7f7765a54640 1 -- 172.21.15.70:0/382964281 >> 172.21.15.70:0/382964281 conn(0x7f77600fbbe0 msgr2=0x7f7760101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:21.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.911+0000 7f7765a54640 1 -- 172.21.15.70:0/382964281 shutdown_connections 2024-04-01T01:31:21.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:21.911+0000 7f7765a54640 1 -- 172.21.15.70:0/382964281 wait complete. 2024-04-01T01:31:21.921 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:21 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2370546956' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-01T01:31:21.922 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:21 smithi070 ceph-mon[32179]: osdmap e88: 12 total, 12 up, 12 in 2024-04-01T01:31:21.922 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:21 smithi070 ceph-mon[32179]: pgmap v289: 129 pgs: 4 creating+activating, 24 creating+peering, 24 unknown, 77 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 8 op/s 2024-04-01T01:31:21.922 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:21 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1268795572' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-01T01:31:21.923 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:31:21.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:21 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2370546956' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-01T01:31:21.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:21 smithi189 ceph-mon[35753]: osdmap e88: 12 total, 12 up, 12 in 2024-04-01T01:31:21.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:21 smithi189 ceph-mon[35753]: pgmap v289: 129 pgs: 4 creating+activating, 24 creating+peering, 24 unknown, 77 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 8 op/s 2024-04-01T01:31:21.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:21 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1268795572' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-01T01:31:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.024+0000 7f53e4c84640 1 -- 172.21.15.70:0/3905339651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53e0101860 msgr2=0x7f53e0101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.024+0000 7f53e4c84640 1 --2- 172.21.15.70:0/3905339651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53e0101860 0x7f53e0101ce0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009900 tx=0x7f53c802bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.024+0000 7f53e4c84640 1 -- 172.21.15.70:0/3905339651 shutdown_connections 2024-04-01T01:31:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.024+0000 7f53e4c84640 1 --2- 172.21.15.70:0/3905339651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53e0102220 0x7f53e0108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.024+0000 7f53e4c84640 1 --2- 172.21.15.70:0/3905339651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53e0101860 0x7f53e0101ce0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.024+0000 7f53e4c84640 1 --2- 172.21.15.70:0/3905339651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e0100670 0x7f53e0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.024+0000 7f53e4c84640 1 -- 172.21.15.70:0/3905339651 >> 172.21.15.70:0/3905339651 conn(0x7f53e00fbbe0 msgr2=0x7f53e00fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:22.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.024+0000 7f53e4c84640 1 -- 172.21.15.70:0/3905339651 shutdown_connections 2024-04-01T01:31:22.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.024+0000 7f53e4c84640 1 -- 172.21.15.70:0/3905339651 wait complete. 2024-04-01T01:31:22.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.025+0000 7f53e4c84640 1 Processor -- start 2024-04-01T01:31:22.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.025+0000 7f53e4c84640 1 -- start start 2024-04-01T01:31:22.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.025+0000 7f53e4c84640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e0100670 0x7f53e0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:22.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.025+0000 7f53e4c84640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53e0101860 0x7f53e019c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:22.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.025+0000 7f53e4c84640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53e0102220 0x7f53e019ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:22.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.025+0000 7f53e4c84640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f53e010c190 con 0x7f53e0102220 2024-04-01T01:31:22.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.025+0000 7f53e4c84640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f53e010c010 con 0x7f53e0101860 2024-04-01T01:31:22.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.025+0000 7f53e4c84640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f53e010c310 con 0x7f53e0100670 2024-04-01T01:31:22.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.025+0000 7f53ded9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53e0102220 0x7f53e019ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:22.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.025+0000 7f53ded9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53e0102220 0x7f53e019ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37312/0 (socket says 172.21.15.70:37312) 2024-04-01T01:31:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.025+0000 7f53ded9d640 1 -- 172.21.15.70:0/622994510 learned_addr learned my addr 172.21.15.70:0/622994510 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.025+0000 7f53d7fff640 1 --2- 172.21.15.70:0/622994510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53e0101860 0x7f53e019c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.026+0000 7f53de59c640 1 --2- 172.21.15.70:0/622994510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e0100670 0x7f53e0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.026+0000 7f53ded9d640 1 -- 172.21.15.70:0/622994510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e0100670 msgr2=0x7f53e0069420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.026+0000 7f53ded9d640 1 --2- 172.21.15.70:0/622994510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e0100670 0x7f53e0069420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.026+0000 7f53ded9d640 1 -- 172.21.15.70:0/622994510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53e0101860 msgr2=0x7f53e019c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.026+0000 7f53ded9d640 1 --2- 172.21.15.70:0/622994510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53e0101860 0x7f53e019c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.026+0000 7f53ded9d640 1 -- 172.21.15.70:0/622994510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e019f5d0 con 0x7f53e0102220 2024-04-01T01:31:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.026+0000 7f53ded9d640 1 --2- 172.21.15.70:0/622994510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53e0102220 0x7f53e019ecc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f53d8009fd0 tx=0x7f53d800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.026+0000 7f53d77fe640 1 -- 172.21.15.70:0/622994510 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d8017070 con 0x7f53e0102220 2024-04-01T01:31:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.026+0000 7f53e4c84640 1 -- 172.21.15.70:0/622994510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53e019f860 con 0x7f53e0102220 2024-04-01T01:31:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.026+0000 7f53d77fe640 1 -- 172.21.15.70:0/622994510 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53d8003e20 con 0x7f53e0102220 2024-04-01T01:31:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.026+0000 7f53d77fe640 1 -- 172.21.15.70:0/622994510 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d8010460 con 0x7f53e0102220 2024-04-01T01:31:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.026+0000 7f53e4c84640 1 -- 172.21.15.70:0/622994510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53e019fd30 con 0x7f53e0102220 2024-04-01T01:31:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.027+0000 7f53e4c84640 1 -- 172.21.15.70:0/622994510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53e0107a60 con 0x7f53e0102220 2024-04-01T01:31:22.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.028+0000 7f53d77fe640 1 -- 172.21.15.70:0/622994510 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f53d8003970 con 0x7f53e0102220 2024-04-01T01:31:22.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.028+0000 7f53d77fe640 1 --2- 172.21.15.70:0/622994510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53cc0a24a0 0x7f53cc0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:22.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.029+0000 7f53de59c640 1 --2- 172.21.15.70:0/622994510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53cc0a24a0 0x7f53cc0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:22.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.029+0000 7f53d77fe640 1 -- 172.21.15.70:0/622994510 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f53d80ce790 con 0x7f53e0102220 2024-04-01T01:31:22.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.029+0000 7f53de59c640 1 --2- 172.21.15.70:0/622994510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53cc0a24a0 0x7f53cc0a4960 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f53d00058c0 tx=0x7f53d0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:22.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.032+0000 7f53d77fe640 1 -- 172.21.15.70:0/622994510 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f53d809b710 con 0x7f53e0102220 2024-04-01T01:31:22.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.208+0000 7f53e4c84640 1 -- 172.21.15.70:0/622994510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f53e002d330 con 0x7f53e0102220 2024-04-01T01:31:22.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.209+0000 7f53d77fe640 1 -- 172.21.15.70:0/622994510 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24068 (secure 0 0 0) 0x7f53d809b010 con 0x7f53e0102220 2024-04-01T01:31:22.210 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:22.210 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":89,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:31:20.903623+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:31:22.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.211+0000 7f53e4c84640 1 -- 172.21.15.70:0/622994510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53cc0a24a0 msgr2=0x7f53cc0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.211+0000 7f53e4c84640 1 --2- 172.21.15.70:0/622994510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53cc0a24a0 0x7f53cc0a4960 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f53d00058c0 tx=0x7f53d0005850 comp rx=0 tx=0).stop 2024-04-01T01:31:22.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.211+0000 7f53e4c84640 1 -- 172.21.15.70:0/622994510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53e0102220 msgr2=0x7f53e019ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.211+0000 7f53e4c84640 1 --2- 172.21.15.70:0/622994510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53e0102220 0x7f53e019ecc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f53d8009fd0 tx=0x7f53d800b260 comp rx=0 tx=0).stop 2024-04-01T01:31:22.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.211+0000 7f53e4c84640 1 -- 172.21.15.70:0/622994510 shutdown_connections 2024-04-01T01:31:22.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.212+0000 7f53e4c84640 1 --2- 172.21.15.70:0/622994510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53e0102220 0x7f53e019ecc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.212+0000 7f53e4c84640 1 --2- 172.21.15.70:0/622994510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53e0101860 0x7f53e019c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.212+0000 7f53e4c84640 1 --2- 172.21.15.70:0/622994510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53cc0a24a0 0x7f53cc0a4960 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.212+0000 7f53e4c84640 1 --2- 172.21.15.70:0/622994510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e0100670 0x7f53e0069420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.212+0000 7f53e4c84640 1 -- 172.21.15.70:0/622994510 >> 172.21.15.70:0/622994510 conn(0x7f53e00fbbe0 msgr2=0x7f53e0104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:22.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.212+0000 7f53e4c84640 1 -- 172.21.15.70:0/622994510 shutdown_connections 2024-04-01T01:31:22.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.212+0000 7f53e4c84640 1 -- 172.21.15.70:0/622994510 wait complete. 2024-04-01T01:31:22.223 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:31:22.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.324+0000 7f19ad781640 1 -- 172.21.15.70:0/2247405321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19a81006f0 msgr2=0x7f19a8100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.324+0000 7f19ad781640 1 --2- 172.21.15.70:0/2247405321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19a81006f0 0x7f19a8100b10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1998009900 tx=0x7f199802bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:22.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.325+0000 7f19ad781640 1 -- 172.21.15.70:0/2247405321 shutdown_connections 2024-04-01T01:31:22.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.325+0000 7f19ad781640 1 --2- 172.21.15.70:0/2247405321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19a81022a0 0x7f19a8108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.325+0000 7f19ad781640 1 --2- 172.21.15.70:0/2247405321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19a81018e0 0x7f19a8101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.325+0000 7f19ad781640 1 --2- 172.21.15.70:0/2247405321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19a81006f0 0x7f19a8100b10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.325+0000 7f19ad781640 1 -- 172.21.15.70:0/2247405321 >> 172.21.15.70:0/2247405321 conn(0x7f19a80fbc80 msgr2=0x7f19a80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.325+0000 7f19ad781640 1 -- 172.21.15.70:0/2247405321 shutdown_connections 2024-04-01T01:31:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.325+0000 7f19ad781640 1 -- 172.21.15.70:0/2247405321 wait complete. 2024-04-01T01:31:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.325+0000 7f19ad781640 1 Processor -- start 2024-04-01T01:31:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.325+0000 7f19ad781640 1 -- start start 2024-04-01T01:31:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.325+0000 7f19ad781640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19a81006f0 0x7f19a8196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.325+0000 7f19ad781640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19a81018e0 0x7f19a8196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.325+0000 7f19ad781640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19a81022a0 0x7f19a819de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.325+0000 7f19ad781640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f19a810c180 con 0x7f19a81022a0 2024-04-01T01:31:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.325+0000 7f19ad781640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f19a810c000 con 0x7f19a81018e0 2024-04-01T01:31:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.325+0000 7f19ad781640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f19a810c300 con 0x7f19a81006f0 2024-04-01T01:31:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f19a77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19a81022a0 0x7f19a819de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:22.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f19a77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19a81022a0 0x7f19a819de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37332/0 (socket says 172.21.15.70:37332) 2024-04-01T01:31:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f19a77fe640 1 -- 172.21.15.70:0/2980277432 learned_addr learned my addr 172.21.15.70:0/2980277432 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f19a67fc640 1 --2- 172.21.15.70:0/2980277432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19a81018e0 0x7f19a8196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f19a6ffd640 1 --2- 172.21.15.70:0/2980277432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19a81006f0 0x7f19a8196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f19a77fe640 1 -- 172.21.15.70:0/2980277432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19a81006f0 msgr2=0x7f19a8196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f19a77fe640 1 --2- 172.21.15.70:0/2980277432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19a81006f0 0x7f19a8196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f19a77fe640 1 -- 172.21.15.70:0/2980277432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19a81018e0 msgr2=0x7f19a8196e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f19a77fe640 1 --2- 172.21.15.70:0/2980277432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19a81018e0 0x7f19a8196e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f19a77fe640 1 -- 172.21.15.70:0/2980277432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19a819e4f0 con 0x7f19a81022a0 2024-04-01T01:31:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f19a6ffd640 1 --2- 172.21.15.70:0/2980277432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19a81006f0 0x7f19a8196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:31:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f19a77fe640 1 --2- 172.21.15.70:0/2980277432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19a81022a0 0x7f19a819de80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f19a000cad0 tx=0x7f19a000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f198ffff640 1 -- 172.21.15.70:0/2980277432 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a000ae20 con 0x7f19a81022a0 2024-04-01T01:31:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f19ad781640 1 -- 172.21.15.70:0/2980277432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19a819e7e0 con 0x7f19a81022a0 2024-04-01T01:31:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f198ffff640 1 -- 172.21.15.70:0/2980277432 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19a0005020 con 0x7f19a81022a0 2024-04-01T01:31:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f198ffff640 1 -- 172.21.15.70:0/2980277432 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a000f6b0 con 0x7f19a81022a0 2024-04-01T01:31:22.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.326+0000 7f19ad781640 1 -- 172.21.15.70:0/2980277432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19a819ece0 con 0x7f19a81022a0 2024-04-01T01:31:22.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.327+0000 7f19ad781640 1 -- 172.21.15.70:0/2980277432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19a8107b80 con 0x7f19a81022a0 2024-04-01T01:31:22.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.328+0000 7f198ffff640 1 -- 172.21.15.70:0/2980277432 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f19a00051d0 con 0x7f19a81022a0 2024-04-01T01:31:22.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.328+0000 7f198ffff640 1 --2- 172.21.15.70:0/2980277432 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f19900a24a0 0x7f19900a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:22.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.329+0000 7f198ffff640 1 -- 172.21.15.70:0/2980277432 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f19a00ce9c0 con 0x7f19a81022a0 2024-04-01T01:31:22.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.329+0000 7f19a6ffd640 1 --2- 172.21.15.70:0/2980277432 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f19900a24a0 0x7f19900a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:22.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.329+0000 7f19a6ffd640 1 --2- 172.21.15.70:0/2980277432 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f19900a24a0 0x7f19900a4960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1998006010 tx=0x7f199802c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:22.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.333+0000 7f198ffff640 1 -- 172.21.15.70:0/2980277432 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f19a00d0e20 con 0x7f19a81022a0 2024-04-01T01:31:22.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.509+0000 7f19ad781640 1 -- 172.21.15.70:0/2980277432 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19a8062350 con 0x7f19a81022a0 2024-04-01T01:31:22.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.511+0000 7f198ffff640 1 -- 172.21.15.70:0/2980277432 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24068 (secure 0 0 0) 0x7f19a0014030 con 0x7f19a81022a0 2024-04-01T01:31:22.512 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:22.512 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":89,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:31:20.903623+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:31:22.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.513+0000 7f19ad781640 1 -- 172.21.15.70:0/2980277432 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f19900a24a0 msgr2=0x7f19900a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.513+0000 7f19ad781640 1 --2- 172.21.15.70:0/2980277432 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f19900a24a0 0x7f19900a4960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1998006010 tx=0x7f199802c7e0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.513+0000 7f19ad781640 1 -- 172.21.15.70:0/2980277432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19a81022a0 msgr2=0x7f19a819de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.513+0000 7f19ad781640 1 --2- 172.21.15.70:0/2980277432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19a81022a0 0x7f19a819de80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f19a000cad0 tx=0x7f19a000cf90 comp rx=0 tx=0).stop 2024-04-01T01:31:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.513+0000 7f19ad781640 1 -- 172.21.15.70:0/2980277432 shutdown_connections 2024-04-01T01:31:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.513+0000 7f19ad781640 1 --2- 172.21.15.70:0/2980277432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19a81022a0 0x7f19a819de80 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.513+0000 7f19ad781640 1 --2- 172.21.15.70:0/2980277432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19a81018e0 0x7f19a8196e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.513+0000 7f19ad781640 1 --2- 172.21.15.70:0/2980277432 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f19900a24a0 0x7f19900a4960 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.513+0000 7f19ad781640 1 --2- 172.21.15.70:0/2980277432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19a81006f0 0x7f19a8196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.513+0000 7f19ad781640 1 -- 172.21.15.70:0/2980277432 >> 172.21.15.70:0/2980277432 conn(0x7f19a80fbc80 msgr2=0x7f19a8104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.513+0000 7f19ad781640 1 -- 172.21.15.70:0/2980277432 shutdown_connections 2024-04-01T01:31:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.513+0000 7f19ad781640 1 -- 172.21.15.70:0/2980277432 wait complete. 2024-04-01T01:31:22.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:31:22.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.641+0000 7f53f2902640 1 -- 172.21.15.70:0/1005466178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53ec101860 msgr2=0x7f53ec101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.641+0000 7f53f2902640 1 --2- 172.21.15.70:0/1005466178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53ec101860 0x7f53ec101ce0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f53e0009900 tx=0x7f53e002bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:22.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.641+0000 7f53f2902640 1 -- 172.21.15.70:0/1005466178 shutdown_connections 2024-04-01T01:31:22.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.641+0000 7f53f2902640 1 --2- 172.21.15.70:0/1005466178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53ec102220 0x7f53ec108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.641+0000 7f53f2902640 1 --2- 172.21.15.70:0/1005466178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53ec101860 0x7f53ec101ce0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.641+0000 7f53f2902640 1 --2- 172.21.15.70:0/1005466178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53ec100670 0x7f53ec100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.641+0000 7f53f2902640 1 -- 172.21.15.70:0/1005466178 >> 172.21.15.70:0/1005466178 conn(0x7f53ec0fbbe0 msgr2=0x7f53ec0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:22.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.641+0000 7f53f2902640 1 -- 172.21.15.70:0/1005466178 shutdown_connections 2024-04-01T01:31:22.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.641+0000 7f53f2902640 1 -- 172.21.15.70:0/1005466178 wait complete. 2024-04-01T01:31:22.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.642+0000 7f53f2902640 1 Processor -- start 2024-04-01T01:31:22.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.642+0000 7f53f2902640 1 -- start start 2024-04-01T01:31:22.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.642+0000 7f53f2902640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53ec100670 0x7f53ec197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:22.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.642+0000 7f53f2902640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53ec101860 0x7f53ec197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:22.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.642+0000 7f53f2902640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53ec102220 0x7f53ec19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:22.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.642+0000 7f53f2902640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f53ec10c1f0 con 0x7f53ec101860 2024-04-01T01:31:22.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.642+0000 7f53f2902640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f53ec10c070 con 0x7f53ec102220 2024-04-01T01:31:22.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.642+0000 7f53f2902640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f53ec10c370 con 0x7f53ec100670 2024-04-01T01:31:22.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.642+0000 7f53eb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53ec101860 0x7f53ec197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:22.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.642+0000 7f53eb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53ec101860 0x7f53ec197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37354/0 (socket says 172.21.15.70:37354) 2024-04-01T01:31:22.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.642+0000 7f53eb7fe640 1 -- 172.21.15.70:0/1777784477 learned_addr learned my addr 172.21.15.70:0/1777784477 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:22.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.642+0000 7f53f0e9f640 1 --2- 172.21.15.70:0/1777784477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53ec102220 0x7f53ec19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:22.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.643+0000 7f53ebfff640 1 --2- 172.21.15.70:0/1777784477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53ec100670 0x7f53ec197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:22.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.643+0000 7f53eb7fe640 1 -- 172.21.15.70:0/1777784477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53ec100670 msgr2=0x7f53ec197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.643+0000 7f53eb7fe640 1 --2- 172.21.15.70:0/1777784477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53ec100670 0x7f53ec197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.643+0000 7f53eb7fe640 1 -- 172.21.15.70:0/1777784477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53ec102220 msgr2=0x7f53ec19ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.643+0000 7f53eb7fe640 1 --2- 172.21.15.70:0/1777784477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53ec102220 0x7f53ec19ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.643+0000 7f53eb7fe640 1 -- 172.21.15.70:0/1777784477 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ec19f5e0 con 0x7f53ec101860 2024-04-01T01:31:22.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.643+0000 7f53eb7fe640 1 --2- 172.21.15.70:0/1777784477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53ec101860 0x7f53ec197f50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f53e0000c00 tx=0x7f53e00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:22.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.643+0000 7f53e97fa640 1 -- 172.21.15.70:0/1777784477 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e0043020 con 0x7f53ec101860 2024-04-01T01:31:22.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.643+0000 7f53e97fa640 1 -- 172.21.15.70:0/1777784477 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53e0004aa0 con 0x7f53ec101860 2024-04-01T01:31:22.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.643+0000 7f53f2902640 1 -- 172.21.15.70:0/1777784477 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53ec19f870 con 0x7f53ec101860 2024-04-01T01:31:22.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.643+0000 7f53e97fa640 1 -- 172.21.15.70:0/1777784477 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e0003d70 con 0x7f53ec101860 2024-04-01T01:31:22.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.643+0000 7f53f2902640 1 -- 172.21.15.70:0/1777784477 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53ec19fe10 con 0x7f53ec101860 2024-04-01T01:31:22.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.644+0000 7f53f2902640 1 -- 172.21.15.70:0/1777784477 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53ec19ffe0 con 0x7f53ec101860 2024-04-01T01:31:22.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.645+0000 7f53e97fa640 1 -- 172.21.15.70:0/1777784477 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f53e0046020 con 0x7f53ec101860 2024-04-01T01:31:22.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.645+0000 7f53e97fa640 1 --2- 172.21.15.70:0/1777784477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53d40a2520 0x7f53d40a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:22.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.646+0000 7f53ebfff640 1 --2- 172.21.15.70:0/1777784477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53d40a2520 0x7f53d40a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:22.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.646+0000 7f53e97fa640 1 -- 172.21.15.70:0/1777784477 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f53e00f0c50 con 0x7f53ec101860 2024-04-01T01:31:22.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.646+0000 7f53ebfff640 1 --2- 172.21.15.70:0/1777784477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53d40a2520 0x7f53d40a49e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f53dc0058c0 tx=0x7f53dc005c80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:22.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.650+0000 7f53e97fa640 1 -- 172.21.15.70:0/1777784477 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f53e00f6020 con 0x7f53ec101860 2024-04-01T01:31:22.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:22 smithi081 ceph-mon[35142]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-01T01:31:22.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:22 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1268795572' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-01T01:31:22.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:22 smithi081 ceph-mon[35142]: osdmap e89: 12 total, 12 up, 12 in 2024-04-01T01:31:22.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:22 smithi081 ceph-mon[35142]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-01T01:31:22.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:22 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/382964281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:22.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:22 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/622994510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:22.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:22 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2980277432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:22.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.863+0000 7f53f2902640 1 -- 172.21.15.70:0/1777784477 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53ec1a0370 con 0x7f53ec101860 2024-04-01T01:31:22.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.863+0000 7f53e97fa640 1 -- 172.21.15.70:0/1777784477 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11794 (secure 0 0 0) 0x7f53e004d020 con 0x7f53ec101860 2024-04-01T01:31:22.864 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:22.864 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24371,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24379,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24404,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24418,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24425,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24451,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:31:20.903788+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24464},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24464,"qdb_cluster":[24464]},"id":1}]} 2024-04-01T01:31:22.864 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 17 2024-04-01T01:31:22.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.866+0000 7f53f2902640 1 -- 172.21.15.70:0/1777784477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53d40a2520 msgr2=0x7f53d40a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.866+0000 7f53f2902640 1 --2- 172.21.15.70:0/1777784477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53d40a2520 0x7f53d40a49e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f53dc0058c0 tx=0x7f53dc005c80 comp rx=0 tx=0).stop 2024-04-01T01:31:22.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.866+0000 7f53f2902640 1 -- 172.21.15.70:0/1777784477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53ec101860 msgr2=0x7f53ec197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.866+0000 7f53f2902640 1 --2- 172.21.15.70:0/1777784477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53ec101860 0x7f53ec197f50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f53e0000c00 tx=0x7f53e00042f0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.866+0000 7f53f2902640 1 -- 172.21.15.70:0/1777784477 shutdown_connections 2024-04-01T01:31:22.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.866+0000 7f53f2902640 1 --2- 172.21.15.70:0/1777784477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53ec102220 0x7f53ec19ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.866+0000 7f53f2902640 1 --2- 172.21.15.70:0/1777784477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53ec101860 0x7f53ec197f50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.866+0000 7f53f2902640 1 --2- 172.21.15.70:0/1777784477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53d40a2520 0x7f53d40a49e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.866+0000 7f53f2902640 1 --2- 172.21.15.70:0/1777784477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53ec100670 0x7f53ec197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.866+0000 7f53f2902640 1 -- 172.21.15.70:0/1777784477 >> 172.21.15.70:0/1777784477 conn(0x7f53ec0fbbe0 msgr2=0x7f53ec0fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:22.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.866+0000 7f53f2902640 1 -- 172.21.15.70:0/1777784477 shutdown_connections 2024-04-01T01:31:22.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.866+0000 7f53f2902640 1 -- 172.21.15.70:0/1777784477 wait complete. 2024-04-01T01:31:22.876 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:22 smithi070 ceph-mon[32179]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-01T01:31:22.876 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:22 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1268795572' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-01T01:31:22.876 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:22 smithi070 ceph-mon[32179]: osdmap e89: 12 total, 12 up, 12 in 2024-04-01T01:31:22.876 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:22 smithi070 ceph-mon[32179]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-01T01:31:22.876 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:22 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/382964281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:22.876 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:22 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/622994510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:22.876 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:22 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2980277432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:22.878 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:31:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.979+0000 7f25af920640 1 -- 172.21.15.70:0/1802575427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a81022a0 msgr2=0x7f25a8108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.979+0000 7f25af920640 1 --2- 172.21.15.70:0/1802575427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a81022a0 0x7f25a8108b30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f25a0009ca0 tx=0x7f25a002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.979+0000 7f25af920640 1 -- 172.21.15.70:0/1802575427 shutdown_connections 2024-04-01T01:31:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.979+0000 7f25af920640 1 --2- 172.21.15.70:0/1802575427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a81022a0 0x7f25a8108b30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.979+0000 7f25af920640 1 --2- 172.21.15.70:0/1802575427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25a81018e0 0x7f25a8101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.979+0000 7f25af920640 1 --2- 172.21.15.70:0/1802575427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25a81006f0 0x7f25a8100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.979+0000 7f25af920640 1 -- 172.21.15.70:0/1802575427 >> 172.21.15.70:0/1802575427 conn(0x7f25a80fbc80 msgr2=0x7f25a80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.979+0000 7f25af920640 1 -- 172.21.15.70:0/1802575427 shutdown_connections 2024-04-01T01:31:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.979+0000 7f25af920640 1 -- 172.21.15.70:0/1802575427 wait complete. 2024-04-01T01:31:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.979+0000 7f25af920640 1 Processor -- start 2024-04-01T01:31:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.979+0000 7f25af920640 1 -- start start 2024-04-01T01:31:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25af920640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25a81006f0 0x7f25a8196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25af920640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25a81018e0 0x7f25a8196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25af920640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a81022a0 0x7f25a819de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25af920640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f25a810c1b0 con 0x7f25a81022a0 2024-04-01T01:31:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25af920640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f25a810c030 con 0x7f25a81006f0 2024-04-01T01:31:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25af920640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f25a810c330 con 0x7f25a81018e0 2024-04-01T01:31:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25adebd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a81022a0 0x7f25a819de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25adebd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a81022a0 0x7f25a819de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37368/0 (socket says 172.21.15.70:37368) 2024-04-01T01:31:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25adebd640 1 -- 172.21.15.70:0/2168845406 learned_addr learned my addr 172.21.15.70:0/2168845406 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25ad6bc640 1 --2- 172.21.15.70:0/2168845406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25a81006f0 0x7f25a8196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25acebb640 1 --2- 172.21.15.70:0/2168845406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25a81018e0 0x7f25a8196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25adebd640 1 -- 172.21.15.70:0/2168845406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25a81018e0 msgr2=0x7f25a8196d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25adebd640 1 --2- 172.21.15.70:0/2168845406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25a81018e0 0x7f25a8196d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25adebd640 1 -- 172.21.15.70:0/2168845406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25a81006f0 msgr2=0x7f25a8196850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25adebd640 1 --2- 172.21.15.70:0/2168845406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25a81006f0 0x7f25a8196850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25adebd640 1 -- 172.21.15.70:0/2168845406 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a819e470 con 0x7f25a81022a0 2024-04-01T01:31:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25adebd640 1 --2- 172.21.15.70:0/2168845406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a81022a0 0x7f25a819de00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f25a002c700 tx=0x7f25a00328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f259e7fc640 1 -- 172.21.15.70:0/2168845406 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0041020 con 0x7f25a81022a0 2024-04-01T01:31:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25af920640 1 -- 172.21.15.70:0/2168845406 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a819e700 con 0x7f25a81022a0 2024-04-01T01:31:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f259e7fc640 1 -- 172.21.15.70:0/2168845406 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25a0003e20 con 0x7f25a81022a0 2024-04-01T01:31:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f259e7fc640 1 -- 172.21.15.70:0/2168845406 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0039040 con 0x7f25a81022a0 2024-04-01T01:31:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.980+0000 7f25af920640 1 -- 172.21.15.70:0/2168845406 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a819ec40 con 0x7f25a81022a0 2024-04-01T01:31:22.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.981+0000 7f25af920640 1 -- 172.21.15.70:0/2168845406 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a8107a60 con 0x7f25a81022a0 2024-04-01T01:31:22.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.982+0000 7f259e7fc640 1 -- 172.21.15.70:0/2168845406 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f25a0044020 con 0x7f25a81022a0 2024-04-01T01:31:22.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.983+0000 7f259e7fc640 1 --2- 172.21.15.70:0/2168845406 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f25940a24a0 0x7f25940a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:22.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.983+0000 7f25ad6bc640 1 --2- 172.21.15.70:0/2168845406 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f25940a24a0 0x7f25940a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:22.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.983+0000 7f259e7fc640 1 -- 172.21.15.70:0/2168845406 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f25a00f0c10 con 0x7f25a81022a0 2024-04-01T01:31:22.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.983+0000 7f25ad6bc640 1 --2- 172.21.15.70:0/2168845406 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f25940a24a0 0x7f25940a4960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f25a4005fd0 tx=0x7f25a400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:22.988+0000 7f259e7fc640 1 -- 172.21.15.70:0/2168845406 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f25a004b020 con 0x7f25a81022a0 2024-04-01T01:31:22.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:22 smithi189 ceph-mon[35753]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-01T01:31:22.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:22 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1268795572' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-01T01:31:22.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:22 smithi189 ceph-mon[35753]: osdmap e89: 12 total, 12 up, 12 in 2024-04-01T01:31:22.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:22 smithi189 ceph-mon[35753]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-01T01:31:22.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:22 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/382964281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:22.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:22 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/622994510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:22.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:22 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2980277432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:23.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.162+0000 7f25af920640 1 -- 172.21.15.70:0/2168845406 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f25a8062350 con 0x7f25a81022a0 2024-04-01T01:31:23.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.164+0000 7f259e7fc640 1 -- 172.21.15.70:0/2168845406 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24068 (secure 0 0 0) 0x7f25a00bdb90 con 0x7f25a81022a0 2024-04-01T01:31:23.164 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:23.164 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":89,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:31:20.903623+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:31:23.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.166+0000 7f25af920640 1 -- 172.21.15.70:0/2168845406 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f25940a24a0 msgr2=0x7f25940a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:23.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.166+0000 7f25af920640 1 --2- 172.21.15.70:0/2168845406 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f25940a24a0 0x7f25940a4960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f25a4005fd0 tx=0x7f25a400b390 comp rx=0 tx=0).stop 2024-04-01T01:31:23.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.166+0000 7f25af920640 1 -- 172.21.15.70:0/2168845406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a81022a0 msgr2=0x7f25a819de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:23.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.166+0000 7f25af920640 1 --2- 172.21.15.70:0/2168845406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a81022a0 0x7f25a819de00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f25a002c700 tx=0x7f25a00328a0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.166+0000 7f25af920640 1 -- 172.21.15.70:0/2168845406 shutdown_connections 2024-04-01T01:31:23.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.166+0000 7f25af920640 1 --2- 172.21.15.70:0/2168845406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25a81022a0 0x7f25a819de00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.166+0000 7f25af920640 1 --2- 172.21.15.70:0/2168845406 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f25940a24a0 0x7f25940a4960 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.166+0000 7f25af920640 1 --2- 172.21.15.70:0/2168845406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25a81018e0 0x7f25a8196d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.166+0000 7f25af920640 1 --2- 172.21.15.70:0/2168845406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25a81006f0 0x7f25a8196850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.166+0000 7f25af920640 1 -- 172.21.15.70:0/2168845406 >> 172.21.15.70:0/2168845406 conn(0x7f25a80fbc80 msgr2=0x7f25a8104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:23.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.166+0000 7f25af920640 1 -- 172.21.15.70:0/2168845406 shutdown_connections 2024-04-01T01:31:23.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.166+0000 7f25af920640 1 -- 172.21.15.70:0/2168845406 wait complete. 2024-04-01T01:31:23.178 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-01T01:31:15.732761+0000', 'modified': '2024-04-01T01:31:20.903788+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24464}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24464': {'gid': 24464, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6839/2805044611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6838', 'nonce': 2805044611}, {'type': 'v1', 'addr': '172.21.15.81:6839', 'nonce': 2805044611}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24464, 'qdb_cluster': [24464]} 2024-04-01T01:31:23.178 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-01T01:31:23.178 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-01T01:31:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.279+0000 7fcc4c155640 1 -- 172.21.15.70:0/2934697833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc441021f0 msgr2=0x7fcc441045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.279+0000 7fcc4c155640 1 --2- 172.21.15.70:0/2934697833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc441021f0 0x7fcc441045e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcc40009900 tx=0x7fcc4002bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.279+0000 7fcc4c155640 1 -- 172.21.15.70:0/2934697833 shutdown_connections 2024-04-01T01:31:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.279+0000 7fcc4c155640 1 --2- 172.21.15.70:0/2934697833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc44107450 0x7fcc441098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.279+0000 7fcc4c155640 1 --2- 172.21.15.70:0/2934697833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc44104b20 0x7fcc44106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.279+0000 7fcc4c155640 1 --2- 172.21.15.70:0/2934697833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc441021f0 0x7fcc441045e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.279+0000 7fcc4c155640 1 -- 172.21.15.70:0/2934697833 >> 172.21.15.70:0/2934697833 conn(0x7fcc440fbc00 msgr2=0x7fcc440fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.279+0000 7fcc4c155640 1 -- 172.21.15.70:0/2934697833 shutdown_connections 2024-04-01T01:31:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.279+0000 7fcc4c155640 1 -- 172.21.15.70:0/2934697833 wait complete. 2024-04-01T01:31:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc4c155640 1 Processor -- start 2024-04-01T01:31:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc4c155640 1 -- start start 2024-04-01T01:31:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc4c155640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc441021f0 0x7fcc44197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc4c155640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc44104b20 0x7fcc44197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc4c155640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc44107450 0x7fcc4419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc4c155640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4410d310 con 0x7fcc44104b20 2024-04-01T01:31:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc4c155640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4410d190 con 0x7fcc44107450 2024-04-01T01:31:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc4c155640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4410d490 con 0x7fcc441021f0 2024-04-01T01:31:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc496f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc44104b20 0x7fcc44197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc496f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc44104b20 0x7fcc44197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37394/0 (socket says 172.21.15.70:37394) 2024-04-01T01:31:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc496f0640 1 -- 172.21.15.70:0/3001148946 learned_addr learned my addr 172.21.15.70:0/3001148946 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc49ef1640 1 --2- 172.21.15.70:0/3001148946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc441021f0 0x7fcc44197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc4a6f2640 1 --2- 172.21.15.70:0/3001148946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc44107450 0x7fcc4419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc496f0640 1 -- 172.21.15.70:0/3001148946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc441021f0 msgr2=0x7fcc44197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:23.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc496f0640 1 --2- 172.21.15.70:0/3001148946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc441021f0 0x7fcc44197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc496f0640 1 -- 172.21.15.70:0/3001148946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc44107450 msgr2=0x7fcc4419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:23.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc496f0640 1 --2- 172.21.15.70:0/3001148946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc44107450 0x7fcc4419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.280+0000 7fcc496f0640 1 -- 172.21.15.70:0/3001148946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc4419f640 con 0x7fcc44104b20 2024-04-01T01:31:23.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.281+0000 7fcc496f0640 1 --2- 172.21.15.70:0/3001148946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc44104b20 0x7fcc44197f60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400e9a0 tx=0x7fcc3400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:23.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.281+0000 7fcc3affd640 1 -- 172.21.15.70:0/3001148946 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3400cd80 con 0x7fcc44104b20 2024-04-01T01:31:23.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.281+0000 7fcc3affd640 1 -- 172.21.15.70:0/3001148946 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc340035f0 con 0x7fcc44104b20 2024-04-01T01:31:23.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.281+0000 7fcc4c155640 1 -- 172.21.15.70:0/3001148946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc4419f930 con 0x7fcc44104b20 2024-04-01T01:31:23.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.281+0000 7fcc3affd640 1 -- 172.21.15.70:0/3001148946 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc34010690 con 0x7fcc44104b20 2024-04-01T01:31:23.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.281+0000 7fcc4c155640 1 -- 172.21.15.70:0/3001148946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc4419ff00 con 0x7fcc44104b20 2024-04-01T01:31:23.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.283+0000 7fcc3affd640 1 -- 172.21.15.70:0/3001148946 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fcc340037a0 con 0x7fcc44104b20 2024-04-01T01:31:23.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc4ee0c740 1 Processor -- start 2024-04-01T01:31:23.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc4ee0c740 1 -- start start 2024-04-01T01:31:23.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc4ee0c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfeb610 0x55682cfeba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc4ee0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682cfecc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc4ee0c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfed1a0 0x55682cff3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc4ee0c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55682cf4faf0 con 0x55682cfec7e0 2024-04-01T01:31:23.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc4ee0c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55682cef89d0 con 0x55682cfed1a0 2024-04-01T01:31:23.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc4ee0c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55682cf4bb90 con 0x55682cfeb610 2024-04-01T01:31:23.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc3affd640 1 --2- 172.21.15.70:0/3001148946 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcc300a24a0 0x7fcc300a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc49ef1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682cfecc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc49ef1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682cfecc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37398/0 (socket says 172.21.15.70:37398) 2024-04-01T01:31:23.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc49ef1640 1 -- 172.21.15.70:0/3046438116 learned_addr learned my addr 172.21.15.70:0/3046438116 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:23.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc496f0640 1 --2- 172.21.15.70:0/3046438116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfed1a0 0x55682cff3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc4a6f2640 1 --2- 172.21.15.70:0/3001148946 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcc300a24a0 0x7fcc300a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc4a6f2640 1 --2- 172.21.15.70:0/3046438116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfeb610 0x55682cfeba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc3affd640 1 -- 172.21.15.70:0/3001148946 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcc340ced40 con 0x7fcc44104b20 2024-04-01T01:31:23.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc49ef1640 1 -- 172.21.15.70:0/3046438116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfeb610 msgr2=0x55682cfeba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:31:23.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc49ef1640 1 --2- 172.21.15.70:0/3046438116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfeb610 0x55682cfeba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc49ef1640 1 -- 172.21.15.70:0/3046438116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfed1a0 msgr2=0x55682cff3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:31:23.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc49ef1640 1 --2- 172.21.15.70:0/3046438116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfed1a0 0x55682cff3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.284+0000 7fcc49ef1640 1 -- 172.21.15.70:0/3046438116 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55682cc642d0 con 0x55682cfec7e0 2024-04-01T01:31:23.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.285+0000 7fcc4a6f2640 1 --2- 172.21.15.70:0/3001148946 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcc300a24a0 0x7fcc300a4960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c009fd0 tx=0x7fcc3c009d80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:23.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.285+0000 7fcc49ef1640 1 --2- 172.21.15.70:0/3046438116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682cfecc60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcc40009900 tx=0x7fcc40005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c78db8dd37be3e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.285+0000 7fcc2ffff640 1 -- 172.21.15.70:0/3046438116 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc40044020 con 0x55682cfec7e0 2024-04-01T01:31:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.285+0000 7fcc2ffff640 1 -- 172.21.15.70:0/3046438116 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc40005040 con 0x55682cfec7e0 2024-04-01T01:31:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.285+0000 7fcc2ffff640 1 -- 172.21.15.70:0/3046438116 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc400048c0 con 0x55682cfec7e0 2024-04-01T01:31:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.286+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3046438116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 msgr2=0x55682cfecc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.286+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/3046438116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682cfecc60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcc40009900 tx=0x7fcc40005dd0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.286+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3046438116 shutdown_connections 2024-04-01T01:31:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.286+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/3046438116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfed1a0 0x55682cff3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.286+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/3046438116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682cfecc60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.286+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/3046438116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfeb610 0x55682cfeba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.286+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3046438116 >> 172.21.15.70:0/3046438116 conn(0x55682cfe6a00 msgr2=0x55682cfe6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.286+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3046438116 shutdown_connections 2024-04-01T01:31:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.286+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3046438116 wait complete. 2024-04-01T01:31:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.286+0000 7fcc4ee0c740 1 -- start start 2024-04-01T01:31:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.286+0000 7fcc4ee0c740 1 Processor -- start 2024-04-01T01:31:23.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.286+0000 7fcc4ee0c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfeb610 0x55682cff58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.286+0000 7fcc4ee0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682cff5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc4ee0c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfed1a0 0x55682cffce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc4ee0c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55682cff4150 con 0x55682cfec7e0 2024-04-01T01:31:23.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc4ee0c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55682cf0bf70 con 0x55682cfeb610 2024-04-01T01:31:23.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc4ee0c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55682cff42d0 con 0x55682cfed1a0 2024-04-01T01:31:23.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc4ee0c740 10 client.0 _ll_register_callbacks cb 0x55682cc63ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:31:23.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc496f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682cff5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc496f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682cff5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37406/0 (socket says 172.21.15.70:37406) 2024-04-01T01:31:23.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc496f0640 1 -- 172.21.15.70:0/3503312038 learned_addr learned my addr 172.21.15.70:0/3503312038 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:23.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc49ef1640 1 --2- 172.21.15.70:0/3503312038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfeb610 0x55682cff58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc4a6f2640 1 --2- 172.21.15.70:0/3503312038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfed1a0 0x55682cffce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc496f0640 1 -- 172.21.15.70:0/3503312038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfed1a0 msgr2=0x55682cffce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:31:23.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc496f0640 1 --2- 172.21.15.70:0/3503312038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfed1a0 0x55682cffce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc496f0640 1 -- 172.21.15.70:0/3503312038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfeb610 msgr2=0x55682cff58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:31:23.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc496f0640 1 --2- 172.21.15.70:0/3503312038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfeb610 0x55682cff58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc496f0640 1 -- 172.21.15.70:0/3503312038 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55682cd2e6b0 con 0x55682cfec7e0 2024-04-01T01:31:23.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc49ef1640 1 --2- 172.21.15.70:0/3503312038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfeb610 0x55682cff58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:23.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc4a6f2640 1 --2- 172.21.15.70:0/3503312038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfed1a0 0x55682cffce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:31:23.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc496f0640 1 --2- 172.21.15.70:0/3503312038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682cff5de0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcc340cc650 tx=0x7fcc34066890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440fd26e45426a35 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:23.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc2d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:31:23.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc2d7fa640 1 -- 172.21.15.70:0/3503312038 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc34003b10 con 0x55682cfec7e0 2024-04-01T01:31:23.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc2d7fa640 1 -- 172.21.15.70:0/3503312038 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcc3406a330 con 0x55682cfec7e0 2024-04-01T01:31:23.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3503312038 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc40005040 con 0x55682cfec7e0 2024-04-01T01:31:23.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc2d7fa640 1 -- 172.21.15.70:0/3503312038 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fcc34066900 con 0x55682cfec7e0 2024-04-01T01:31:23.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc4ee0c740 10 client.14745 fetch_fsmap learned FSMap version 17 2024-04-01T01:31:23.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3503312038 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55682cffff20 con 0x55682cfec7e0 2024-04-01T01:31:23.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.287+0000 7fcc2d7fa640 1 -- 172.21.15.70:0/3503312038 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc40005040 con 0x55682cfec7e0 2024-04-01T01:31:23.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.288+0000 7fcc2d7fa640 1 -- 172.21.15.70:0/3503312038 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fcc340a3b40 con 0x55682cfec7e0 2024-04-01T01:31:23.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.288+0000 7fcc4ee0c740 10 client.14745 fetch_fsmap finished waiting for FSMap version 17 2024-04-01T01:31:23.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.288+0000 7fcc4ee0c740 10 client.14745 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-01T01:31:23.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.288+0000 7fcc4ee0c740 20 client.14745 populate_metadata read hostname 'smithi070' 2024-04-01T01:31:23.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.288+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/3503312038 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55682d002560 0x55682d0039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.288+0000 7fcc4ee0c740 4 client.14745 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:31:23.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.288+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3503312038 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55682cf5d030 con 0x55682d002560 2024-04-01T01:31:23.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.288+0000 7fcc49ef1640 1 --2- 172.21.15.70:0/3503312038 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55682d002560 0x55682d0039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.288+0000 7fcc2d7fa640 10 client.14745 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:31:23.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.288+0000 7fcc49ef1640 1 --2- 172.21.15.70:0/3503312038 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55682d002560 0x55682d0039d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=371d595bf336a46d server_cookie=768ec1b75266a766 in_seq=0 out_seq=0 2024-04-01T01:31:23.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.289+0000 7fcc2d7fa640 1 -- 172.21.15.70:0/3503312038 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55682cf5d030 con 0x55682d002560 2024-04-01T01:31:23.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.289+0000 7fcc2d7fa640 10 client.14745 handle_command_reply: tid=0 2024-04-01T01:31:23.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.289+0000 7fcc4ee0c740 1 client.14745 shutdown 2024-04-01T01:31:23.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.289+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3503312038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 msgr2=0x55682cff5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:23.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.289+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/3503312038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682cff5de0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcc340cc650 tx=0x7fcc34066890 comp rx=0 tx=0).stop 2024-04-01T01:31:23.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.289+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3503312038 shutdown_connections 2024-04-01T01:31:23.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.289+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/3503312038 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55682d002560 0x55682d0039d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.289+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/3503312038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfed1a0 0x55682cffce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.289+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/3503312038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682cff5de0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.289+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/3503312038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfeb610 0x55682cff58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.289+0000 7fcc2d7fa640 0 client.14745 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:31:23.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.289+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3503312038 >> 172.21.15.70:0/3503312038 conn(0x55682cfe6a00 msgr2=0x55682cfe5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:23.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.289+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3503312038 shutdown_connections 2024-04-01T01:31:23.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.289+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3503312038 wait complete. 2024-04-01T01:31:23.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.289+0000 7fcc4ee0c740 20 client.14745 trim_cache size 0 max 16384 2024-04-01T01:31:23.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc4ee0c740 1 Processor -- start 2024-04-01T01:31:23.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc4ee0c740 1 -- start start 2024-04-01T01:31:23.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc4ee0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfeb610 0x55682cfe6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc4ee0c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfec7e0 0x55682cffcde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc4ee0c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfed1a0 0x55682cff54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc4ee0c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55682d001d10 con 0x55682cfeb610 2024-04-01T01:31:23.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc4ee0c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55682d0023e0 con 0x55682cfed1a0 2024-04-01T01:31:23.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc4ee0c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55682d002ab0 con 0x55682cfec7e0 2024-04-01T01:31:23.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc49ef1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfeb610 0x55682cfe6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc49ef1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfeb610 0x55682cfe6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37416/0 (socket says 172.21.15.70:37416) 2024-04-01T01:31:23.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc49ef1640 1 -- 172.21.15.70:0/4055842115 learned_addr learned my addr 172.21.15.70:0/4055842115 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:23.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc4a6f2640 1 --2- 172.21.15.70:0/4055842115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfed1a0 0x55682cff54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc49ef1640 1 -- 172.21.15.70:0/4055842115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfec7e0 msgr2=0x55682cffcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:31:23.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc49ef1640 1 --2- 172.21.15.70:0/4055842115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfec7e0 0x55682cffcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc49ef1640 1 -- 172.21.15.70:0/4055842115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfed1a0 msgr2=0x55682cff54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:31:23.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc49ef1640 1 --2- 172.21.15.70:0/4055842115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfed1a0 0x55682cff54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc49ef1640 1 -- 172.21.15.70:0/4055842115 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55682cff3390 con 0x55682cfeb610 2024-04-01T01:31:23.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.307+0000 7fcc4a6f2640 1 --2- 172.21.15.70:0/4055842115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfed1a0 0x55682cff54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:31:23.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.308+0000 7fcc49ef1640 1 --2- 172.21.15.70:0/4055842115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfeb610 0x55682cfe6de0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x55682cfe7970 tx=0x7fcc4002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d4661c3fa61fb86 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:23.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.308+0000 7fcc2f7fe640 1 -- 172.21.15.70:0/4055842115 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc40050070 con 0x55682cfeb610 2024-04-01T01:31:23.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.308+0000 7fcc2f7fe640 1 -- 172.21.15.70:0/4055842115 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc40005600 con 0x55682cfeb610 2024-04-01T01:31:23.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.308+0000 7fcc2f7fe640 1 -- 172.21.15.70:0/4055842115 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc40044040 con 0x55682cfeb610 2024-04-01T01:31:23.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.308+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/4055842115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfeb610 msgr2=0x55682cfe6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:23.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.308+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/4055842115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfeb610 0x55682cfe6de0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x55682cfe7970 tx=0x7fcc4002bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:23.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.308+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/4055842115 shutdown_connections 2024-04-01T01:31:23.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.308+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/4055842115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfed1a0 0x55682cff54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.308+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/4055842115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfec7e0 0x55682cffcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.308+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/4055842115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfeb610 0x55682cfe6de0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.308+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/4055842115 >> 172.21.15.70:0/4055842115 conn(0x55682cfe6a00 msgr2=0x55682cff3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:23.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.308+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/4055842115 shutdown_connections 2024-04-01T01:31:23.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.308+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/4055842115 wait complete. 2024-04-01T01:31:23.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc4ee0c740 1 -- start start 2024-04-01T01:31:23.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc4ee0c740 1 Processor -- start 2024-04-01T01:31:23.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc4ee0c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfeb610 0x55682cffadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc4ee0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682d0642b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc4ee0c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfed1a0 0x55682d066800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc4ee0c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55682d003850 con 0x55682cfec7e0 2024-04-01T01:31:23.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc4ee0c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55682d003180 con 0x55682cfeb610 2024-04-01T01:31:23.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc4ee0c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55682d003f20 con 0x55682cfed1a0 2024-04-01T01:31:23.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc4ee0c740 10 client.0 _ll_register_callbacks cb 0x7fcc3400ef90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:31:23.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc496f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682d0642b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc496f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682d0642b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:37426/0 (socket says 172.21.15.70:37426) 2024-04-01T01:31:23.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc496f0640 1 -- 172.21.15.70:0/3425822016 learned_addr learned my addr 172.21.15.70:0/3425822016 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:23.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc49ef1640 1 --2- 172.21.15.70:0/3425822016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfeb610 0x55682cffadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc4a6f2640 1 --2- 172.21.15.70:0/3425822016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfed1a0 0x55682d066800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc496f0640 1 -- 172.21.15.70:0/3425822016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfed1a0 msgr2=0x55682d066800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:31:23.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc496f0640 1 --2- 172.21.15.70:0/3425822016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfed1a0 0x55682d066800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc496f0640 1 -- 172.21.15.70:0/3425822016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfeb610 msgr2=0x55682cffadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:31:23.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc496f0640 1 --2- 172.21.15.70:0/3425822016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfeb610 0x55682cffadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc496f0640 1 -- 172.21.15.70:0/3425822016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55682d067010 con 0x55682cfec7e0 2024-04-01T01:31:23.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc49ef1640 1 --2- 172.21.15.70:0/3425822016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfeb610 0x55682cffadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:23.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.309+0000 7fcc4a6f2640 1 --2- 172.21.15.70:0/3425822016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfed1a0 0x55682d066800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:31:23.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc496f0640 1 --2- 172.21.15.70:0/3425822016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682d0642b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcc34009230 tx=0x7fcc3401ffb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2316e29ada933773 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:23.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc2d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:31:23.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc2d7fa640 1 -- 172.21.15.70:0/3425822016 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3409b100 con 0x55682cfec7e0 2024-04-01T01:31:23.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc2d7fa640 1 -- 172.21.15.70:0/3425822016 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcc34066d90 con 0x55682cfec7e0 2024-04-01T01:31:23.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3425822016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc40005600 con 0x55682cfec7e0 2024-04-01T01:31:23.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc2d7fa640 1 -- 172.21.15.70:0/3425822016 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fcc34003b10 con 0x55682cfec7e0 2024-04-01T01:31:23.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc4ee0c740 10 client.14751 fetch_fsmap learned FSMap version 17 2024-04-01T01:31:23.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3425822016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55682d068310 con 0x55682cfec7e0 2024-04-01T01:31:23.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc2d7fa640 1 -- 172.21.15.70:0/3425822016 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc40005600 con 0x55682cfec7e0 2024-04-01T01:31:23.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc2d7fa640 1 -- 172.21.15.70:0/3425822016 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fcc34066020 con 0x55682cfec7e0 2024-04-01T01:31:23.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc4ee0c740 10 client.14751 fetch_fsmap finished waiting for FSMap version 17 2024-04-01T01:31:23.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc4ee0c740 10 client.14751 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-01T01:31:23.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc4ee0c740 20 client.14751 populate_metadata read hostname 'smithi070' 2024-04-01T01:31:23.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/3425822016 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55682d06cef0 0x55682d06bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:23.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc4ee0c740 4 client.14751 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-01T01:31:23.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3425822016 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "status"}) -- 0x55682cf57380 con 0x55682d06cef0 2024-04-01T01:31:23.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.310+0000 7fcc49ef1640 1 --2- 172.21.15.70:0/3425822016 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55682d06cef0 0x55682d06bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:23.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc2d7fa640 10 client.14751 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:31:23.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc49ef1640 1 --2- 172.21.15.70:0/3425822016 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55682d06cef0 0x55682d06bc00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac7f3d50792aa8e6 server_cookie=7eb6602473ec4245 in_seq=0 out_seq=0 2024-04-01T01:31:23.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc2d7fa640 1 -- 172.21.15.70:0/3425822016 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55682cf57380 con 0x55682d06cef0 2024-04-01T01:31:23.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc2d7fa640 10 client.14751 handle_command_reply: tid=0 2024-04-01T01:31:23.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc4ee0c740 1 client.14751 shutdown 2024-04-01T01:31:23.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3425822016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 msgr2=0x55682d0642b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:23.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/3425822016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682d0642b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcc34009230 tx=0x7fcc3401ffb0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3425822016 shutdown_connections 2024-04-01T01:31:23.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/3425822016 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55682d06cef0 0x55682d06bc00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/3425822016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55682cfed1a0 0x55682d066800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc2d7fa640 0 client.14751 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:31:23.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/3425822016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55682cfec7e0 0x55682d0642b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc4ee0c740 1 --2- 172.21.15.70:0/3425822016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55682cfeb610 0x55682cffadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3425822016 >> 172.21.15.70:0/3425822016 conn(0x55682cfe6a00 msgr2=0x55682cfe6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:23.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3425822016 shutdown_connections 2024-04-01T01:31:23.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc4ee0c740 1 -- 172.21.15.70:0/3425822016 wait complete. 2024-04-01T01:31:23.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.311+0000 7fcc4ee0c740 20 client.14751 trim_cache size 0 max 16384 2024-04-01T01:31:23.321 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:31:23.321 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "36577394-efc6-11ee-b647-cb9ed24678a4", 2024-04-01T01:31:23.321 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-04-01T01:31:23.321 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24464, 2024-04-01T01:31:23.321 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-01T01:31:23.321 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-01T01:31:23.321 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-01T01:31:23.321 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 6.7774138920000002, 2024-04-01T01:31:23.321 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 17, 2024-04-01T01:31:23.322 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 89, 2024-04-01T01:31:23.322 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 88, 2024-04-01T01:31:23.322 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 12.280906469 2024-04-01T01:31:23.322 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:31:23.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.312+0000 7fcc4c155640 1 -- 172.21.15.70:0/3001148946 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcc300a24a0 msgr2=0x7fcc300a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:23.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.312+0000 7fcc4c155640 1 --2- 172.21.15.70:0/3001148946 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcc300a24a0 0x7fcc300a4960 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c009fd0 tx=0x7fcc3c009d80 comp rx=0 tx=0).stop 2024-04-01T01:31:23.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.312+0000 7fcc4c155640 1 -- 172.21.15.70:0/3001148946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc44104b20 msgr2=0x7fcc44197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:23.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.312+0000 7fcc4c155640 1 --2- 172.21.15.70:0/3001148946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc44104b20 0x7fcc44197f60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400e9a0 tx=0x7fcc3400ee60 comp rx=0 tx=0).stop 2024-04-01T01:31:23.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.312+0000 7fcc4c155640 1 -- 172.21.15.70:0/3001148946 shutdown_connections 2024-04-01T01:31:23.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.312+0000 7fcc4c155640 1 --2- 172.21.15.70:0/3001148946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc44107450 0x7fcc4419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.312+0000 7fcc4c155640 1 --2- 172.21.15.70:0/3001148946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc44104b20 0x7fcc44197f60 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.312+0000 7fcc4c155640 1 --2- 172.21.15.70:0/3001148946 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcc300a24a0 0x7fcc300a4960 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.312+0000 7fcc4c155640 1 --2- 172.21.15.70:0/3001148946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc441021f0 0x7fcc44197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:23.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.312+0000 7fcc4c155640 1 -- 172.21.15.70:0/3001148946 >> 172.21.15.70:0/3001148946 conn(0x7fcc440fbc00 msgr2=0x7fcc44102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:23.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.312+0000 7fcc4c155640 1 -- 172.21.15.70:0/3001148946 shutdown_connections 2024-04-01T01:31:23.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:23.312+0000 7fcc4c155640 1 -- 172.21.15.70:0/3001148946 wait complete. 2024-04-01T01:31:23.324 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-01T01:31:23.325 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-01T01:31:23.325 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph fs ls 2024-04-01T01:31:23.562 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:23.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:23 smithi081 ceph-mon[35142]: pgmap v291: 129 pgs: 8 creating+peering, 6 unknown, 115 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-04-01T01:31:23.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:23 smithi081 ceph-mon[35142]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-01T01:31:23.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:23 smithi081 ceph-mon[35142]: Cluster is now healthy 2024-04-01T01:31:23.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:23 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1777784477' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:23.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:23 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2168845406' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:23.841 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:23 smithi070 ceph-mon[32179]: pgmap v291: 129 pgs: 8 creating+peering, 6 unknown, 115 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-04-01T01:31:23.841 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:23 smithi070 ceph-mon[32179]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-01T01:31:23.841 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:23 smithi070 ceph-mon[32179]: Cluster is now healthy 2024-04-01T01:31:23.841 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:23 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1777784477' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:23.842 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:23 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2168845406' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:23.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:23 smithi189 ceph-mon[35753]: pgmap v291: 129 pgs: 8 creating+peering, 6 unknown, 115 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-04-01T01:31:23.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:23 smithi189 ceph-mon[35753]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-01T01:31:23.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:23 smithi189 ceph-mon[35753]: Cluster is now healthy 2024-04-01T01:31:23.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:23 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1777784477' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:23.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:23 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2168845406' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:25.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.221+0000 7fa0b7338700 1 -- 172.21.15.70:0/771763167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0b01077f0 msgr2=0x7fa0b010dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:25.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.221+0000 7fa0b7338700 1 --2- 172.21.15.70:0/771763167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0b01077f0 0x7fa0b010dfd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac0099c0 tx=0x7fa0ac009cd0 comp rx=0 tx=0).stop 2024-04-01T01:31:25.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.221+0000 7fa0b7338700 1 -- 172.21.15.70:0/771763167 shutdown_connections 2024-04-01T01:31:25.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.221+0000 7fa0b7338700 1 --2- 172.21.15.70:0/771763167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0b01077f0 0x7fa0b010dfd0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:25.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.221+0000 7fa0b7338700 1 --2- 172.21.15.70:0/771763167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b0106170 0x7fa0b0106550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:25.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.221+0000 7fa0b7338700 1 --2- 172.21.15.70:0/771763167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0b0106e30 0x7fa0b01072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:25.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.221+0000 7fa0b7338700 1 -- 172.21.15.70:0/771763167 >> 172.21.15.70:0/771763167 conn(0x7fa0b0077310 msgr2=0x7fa0b00fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:25.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.222+0000 7fa0b7338700 1 -- 172.21.15.70:0/771763167 shutdown_connections 2024-04-01T01:31:25.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.222+0000 7fa0b7338700 1 -- 172.21.15.70:0/771763167 wait complete. 2024-04-01T01:31:25.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.222+0000 7fa0b7338700 1 Processor -- start 2024-04-01T01:31:25.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.222+0000 7fa0b7338700 1 -- start start 2024-04-01T01:31:25.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b7338700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b0106170 0x7fa0b0197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:25.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b7338700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0b0106e30 0x7fa0b0197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:25.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b7338700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0b01077f0 0x7fa0b019bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:25.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b7338700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b01112a0 con 0x7fa0b01077f0 2024-04-01T01:31:25.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b7338700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b0111120 con 0x7fa0b0106e30 2024-04-01T01:31:25.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b7338700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b0111420 con 0x7fa0b0106170 2024-04-01T01:31:25.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b48d3700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0b0106e30 0x7fa0b0197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:25.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b48d3700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0b0106e30 0x7fa0b0197c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:44410/0 (socket says 172.21.15.70:44410) 2024-04-01T01:31:25.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b58d5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0b01077f0 0x7fa0b019bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:25.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b48d3700 1 -- 172.21.15.70:0/1891545964 learned_addr learned my addr 172.21.15.70:0/1891545964 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:25.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b50d4700 1 --2- 172.21.15.70:0/1891545964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b0106170 0x7fa0b0197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:25.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b48d3700 1 -- 172.21.15.70:0/1891545964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b0106170 msgr2=0x7fa0b0197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:25.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b48d3700 1 --2- 172.21.15.70:0/1891545964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b0106170 0x7fa0b0197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:25.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b48d3700 1 -- 172.21.15.70:0/1891545964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0b01077f0 msgr2=0x7fa0b019bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:25.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b48d3700 1 --2- 172.21.15.70:0/1891545964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0b01077f0 0x7fa0b019bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:25.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b48d3700 1 -- 172.21.15.70:0/1891545964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b019c6f0 con 0x7fa0b0106e30 2024-04-01T01:31:25.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b58d5700 1 --2- 172.21.15.70:0/1891545964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0b01077f0 0x7fa0b019bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:25.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b48d3700 1 --2- 172.21.15.70:0/1891545964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0b0106e30 0x7fa0b0197c70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000eab0 tx=0x7fa0a000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:25.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0a67fc700 1 -- 172.21.15.70:0/1891545964 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a001c070 con 0x7fa0b0106e30 2024-04-01T01:31:25.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0a67fc700 1 -- 172.21.15.70:0/1891545964 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0a0017410 con 0x7fa0b0106e30 2024-04-01T01:31:25.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0b7338700 1 -- 172.21.15.70:0/1891545964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0b019c9e0 con 0x7fa0b0106e30 2024-04-01T01:31:25.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.223+0000 7fa0a67fc700 1 -- 172.21.15.70:0/1891545964 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0016610 con 0x7fa0b0106e30 2024-04-01T01:31:25.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.224+0000 7fa0b7338700 1 -- 172.21.15.70:0/1891545964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0b0100bf0 con 0x7fa0b0106e30 2024-04-01T01:31:25.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.225+0000 7fa0b7338700 1 -- 172.21.15.70:0/1891545964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0b0068f00 con 0x7fa0b0106e30 2024-04-01T01:31:25.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.226+0000 7fa0a67fc700 1 -- 172.21.15.70:0/1891545964 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fa0a001f070 con 0x7fa0b0106e30 2024-04-01T01:31:25.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.226+0000 7fa0a67fc700 1 --2- 172.21.15.70:0/1891545964 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa09c0a2530 0x7fa09c0a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:25.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.226+0000 7fa0b50d4700 1 --2- 172.21.15.70:0/1891545964 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa09c0a2530 0x7fa09c0a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:25.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.226+0000 7fa0a67fc700 1 -- 172.21.15.70:0/1891545964 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa0a00cfd50 con 0x7fa0b0106e30 2024-04-01T01:31:25.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.227+0000 7fa0b50d4700 1 --2- 172.21.15.70:0/1891545964 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa09c0a2530 0x7fa09c0a49f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8009550 tx=0x7fa0a80091b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:25.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.234+0000 7fa0a67fc700 1 -- 172.21.15.70:0/1891545964 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa0a009ccd0 con 0x7fa0b0106e30 2024-04-01T01:31:25.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.472+0000 7fa0b7338700 1 -- 172.21.15.70:0/1891545964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa0b0198a40 con 0x7fa0b0106e30 2024-04-01T01:31:25.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.472+0000 7fa0a67fc700 1 -- 172.21.15.70:0/1891545964 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7fa0b0198a40 con 0x7fa0b0106e30 2024-04-01T01:31:25.473 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-01T01:31:25.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.474+0000 7fa0b7338700 1 -- 172.21.15.70:0/1891545964 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa09c0a2530 msgr2=0x7fa09c0a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:25.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.474+0000 7fa0b7338700 1 --2- 172.21.15.70:0/1891545964 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa09c0a2530 0x7fa09c0a49f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8009550 tx=0x7fa0a80091b0 comp rx=0 tx=0).stop 2024-04-01T01:31:25.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.474+0000 7fa0b7338700 1 -- 172.21.15.70:0/1891545964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0b0106e30 msgr2=0x7fa0b0197c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:25.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.474+0000 7fa0b7338700 1 --2- 172.21.15.70:0/1891545964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0b0106e30 0x7fa0b0197c70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000eab0 tx=0x7fa0a000edc0 comp rx=0 tx=0).stop 2024-04-01T01:31:25.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.475+0000 7fa0b7338700 1 -- 172.21.15.70:0/1891545964 shutdown_connections 2024-04-01T01:31:25.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.475+0000 7fa0b7338700 1 --2- 172.21.15.70:0/1891545964 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa09c0a2530 0x7fa09c0a49f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:25.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.475+0000 7fa0b7338700 1 --2- 172.21.15.70:0/1891545964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0b01077f0 0x7fa0b019bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:25.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.475+0000 7fa0b7338700 1 --2- 172.21.15.70:0/1891545964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b0106170 0x7fa0b0197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:25.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.475+0000 7fa0b7338700 1 --2- 172.21.15.70:0/1891545964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0b0106e30 0x7fa0b0197c70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:25.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.475+0000 7fa0b7338700 1 -- 172.21.15.70:0/1891545964 >> 172.21.15.70:0/1891545964 conn(0x7fa0b0077310 msgr2=0x7fa0b00feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:25.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.475+0000 7fa0b7338700 1 -- 172.21.15.70:0/1891545964 shutdown_connections 2024-04-01T01:31:25.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:25.475+0000 7fa0b7338700 1 -- 172.21.15.70:0/1891545964 wait complete. 2024-04-01T01:31:25.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:25 smithi070 ceph-mon[32179]: pgmap v292: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2024-04-01T01:31:25.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:25 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1891545964' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-01T01:31:25.827 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:25 smithi081 ceph-mon[35142]: pgmap v292: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2024-04-01T01:31:25.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:25 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1891545964' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-01T01:31:25.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:25 smithi189 ceph-mon[35753]: pgmap v292: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2024-04-01T01:31:25.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:25 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1891545964' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-01T01:31:26.255 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-01T01:31:26.255 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-01T01:31:26.255 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-01T01:31:26.255 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-01T01:31:26.255 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-01T01:31:26.256 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-01T01:31:26.256 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-01T01:31:26.256 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-01T01:31:26.256 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-01T01:31:26.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:31:26.256 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-01T01:31:26.274 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-01T01:31:26.274 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-01T01:31:26.274 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-01T01:31:26.274 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-01T01:31:26.274 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-01T01:31:26.274 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-01T01:31:26.275 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-01T01:31:26.275 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-01T01:31:26.275 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-01T01:31:26.275 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-01T01:31:26.275 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-01T01:31:26.275 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-01T01:31:26.275 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-01T01:31:26.275 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-01T01:31:26.275 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-01T01:31:26.275 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-01T01:31:26.275 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-01T01:31:26.276 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42187sec preferred_lft 42187sec 2024-04-01T01:31:26.276 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-01T01:31:26.276 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-01T01:31:26.276 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-01T01:31:26.277 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-01T01:31:26.277 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-01T01:31:26.277 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link add name ceph-brx type bridge 2024-04-01T01:31:26.277 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip addr flush dev ceph-brx 2024-04-01T01:31:26.277 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set ceph-brx up 2024-04-01T01:31:26.277 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-01T01:31:26.277 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-01T01:31:26.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-01T01:31:26.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-01T01:31:26.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:31:26.474 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-01T01:31:26.502 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-01T01:31:26.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:31:26.504 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-01T01:31:26.575 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-01T01:31:26.575 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-01T01:31:26.575 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-01T01:31:26.576 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-01T01:31:26.576 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-01T01:31:26.576 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-01T01:31:26.576 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-01T01:31:26.576 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-01T01:31:26.576 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-01T01:31:26.576 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-01T01:31:26.576 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-01T01:31:26.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-01T01:31:26.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-01T01:31:26.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:31:26.879 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-01T01:31:26.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:31:26.894 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list-id 2024-04-01T01:31:26.949 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-01T01:31:26.949 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-01T01:31:26.949 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-01T01:31:26.949 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-01T01:31:26.949 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-01T01:31:27.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-01T01:31:27.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-01T01:31:27.071 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-01T01:31:27.072 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-01T01:31:27.072 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-01T01:31:27.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-01T01:31:27.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-01T01:31:27.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-01T01:31:27.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-01T01:31:27.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-01T01:31:27.072 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-01T01:31:27.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-01T01:31:27.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-01T01:31:27.208 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-01T01:31:27.208 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-01T01:31:27.208 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.0 up 2024-04-01T01:31:27.208 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set dev brx.0 master ceph-brx 2024-04-01T01:31:27.209 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-01T01:31:27.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-01T01:31:27.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-01T01:31:27.290 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-01T01:31:27.290 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-01T01:31:27.290 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-01T01:31:27.305 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-01T01:31:27.305 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-01T01:31:27.305 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-01T01:31:27.359 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-01T01:31:27.428 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-01T01:31:27.485 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-01T01:31:27.485 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-01T01:31:27.485 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-01T01:31:27.485 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-01T01:31:27.485 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-01T01:31:27.485 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-01T01:31:27.485 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-01T01:31:27.485 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-01T01:31:27.485 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-01T01:31:27.485 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-01T01:31:27.486 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-01T01:31:27.486 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-01T01:31:27.486 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-01T01:31:27.486 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-01T01:31:27.486 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-01T01:31:27.486 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-01T01:31:27.486 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-01T01:31:27.486 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-01T01:31:27.486 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-01T01:31:27.486 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-01T01:31:27.486 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-01T01:31:27.486 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-01T01:31:27.486 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-01T01:31:27.487 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-01T01:31:27.487 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-01T01:31:27.487 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay 2024-04-01T01:31:27.487 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/585d4bb1e6cc6a226b096b7a454003dd2f25e52839ed75bbf063895bf9fd7647/merged 2024-04-01T01:31:27.487 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/363fe581d6f063986062f6da7b8ee4000839e42571fe570e38d6fbb8793740c8/merged 2024-04-01T01:31:27.487 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/0 2024-04-01T01:31:27.487 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-01T01:31:27.487 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/a12405395bb3d2147702e3fb40342d6133652c8521fdab4a47bc78ac3ac29f06/merged 2024-04-01T01:31:27.487 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/89ebda32a13bd3ddd2f34a1c254a0f6a4c573018cd0955b5136df44d8dffad5d/merged 2024-04-01T01:31:27.487 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/12b865e10303128d42ed6487fe430ce5108439f3bf4e03f50c67ec7752b3d01a/merged 2024-04-01T01:31:27.487 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/4540f4f9ae73641286821bdc45f5e67d404907e0f135fbcab13e7ee23b801382/merged 2024-04-01T01:31:27.487 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/2689de020410cdb1ff530095c72a11b0a47f70861c66a2e659bca46eca82dbbf/merged 2024-04-01T01:31:27.487 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/eed934015ed699d9040d7dc0a23865bfc758905371e49b8d844691ba7ff9042d/merged 2024-04-01T01:31:27.487 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/271f1491dd89220d7f4665be2342d2711e5b7bf0d13774eac80a21f0870129cf/merged 2024-04-01T01:31:27.487 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/b57c328c249e3f0fbe148520cbe162647f9204fc1e0663cb6c8be45b8c28466b/merged 2024-04-01T01:31:27.488 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-01T01:31:27.488 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-01T01:31:27.488 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-01T01:31:27.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:31:27.488 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-01T01:31:27.540 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-01T01:31:27.541 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-01T01:31:27.584 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-01T01:31:27.610 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-01T01:31:27.667 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-01T01:31:27.667 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-01T01:31:27.667 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-01T01:31:27.667 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-01T01:31:27.667 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-01T01:31:27.667 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-01T01:31:27.667 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-01T01:31:27.667 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-01T01:31:27.667 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-01T01:31:27.667 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-01T01:31:27.667 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-01T01:31:27.668 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-01T01:31:27.668 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-01T01:31:27.668 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-01T01:31:27.668 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-01T01:31:27.668 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-01T01:31:27.668 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-01T01:31:27.668 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-01T01:31:27.668 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-01T01:31:27.668 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-01T01:31:27.668 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-01T01:31:27.668 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-01T01:31:27.668 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-01T01:31:27.668 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-01T01:31:27.669 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-01T01:31:27.669 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay 2024-04-01T01:31:27.669 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/585d4bb1e6cc6a226b096b7a454003dd2f25e52839ed75bbf063895bf9fd7647/merged 2024-04-01T01:31:27.669 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/363fe581d6f063986062f6da7b8ee4000839e42571fe570e38d6fbb8793740c8/merged 2024-04-01T01:31:27.669 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/0 2024-04-01T01:31:27.669 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-01T01:31:27.669 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/a12405395bb3d2147702e3fb40342d6133652c8521fdab4a47bc78ac3ac29f06/merged 2024-04-01T01:31:27.669 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/89ebda32a13bd3ddd2f34a1c254a0f6a4c573018cd0955b5136df44d8dffad5d/merged 2024-04-01T01:31:27.669 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/12b865e10303128d42ed6487fe430ce5108439f3bf4e03f50c67ec7752b3d01a/merged 2024-04-01T01:31:27.669 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/4540f4f9ae73641286821bdc45f5e67d404907e0f135fbcab13e7ee23b801382/merged 2024-04-01T01:31:27.669 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/2689de020410cdb1ff530095c72a11b0a47f70861c66a2e659bca46eca82dbbf/merged 2024-04-01T01:31:27.669 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/eed934015ed699d9040d7dc0a23865bfc758905371e49b8d844691ba7ff9042d/merged 2024-04-01T01:31:27.670 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/271f1491dd89220d7f4665be2342d2711e5b7bf0d13774eac80a21f0870129cf/merged 2024-04-01T01:31:27.670 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/b57c328c249e3f0fbe148520cbe162647f9204fc1e0663cb6c8be45b8c28466b/merged 2024-04-01T01:31:27.670 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-01T01:31:27.670 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-01T01:31:27.670 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-01T01:31:27.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:31:27.670 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-01T01:31:27.680 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi070.stderr:2024-04-01T01:31:27.679+0000 7f09983f9480 -1 init, newargv = 0x5587a237dd20 newargc=15 2024-04-01T01:31:27.680 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi070.stderr:ceph-fuse[76219]: starting ceph client 2024-04-01T01:31:27.685 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi070.stderr:ceph-fuse[76219]: starting fuse 2024-04-01T01:31:27.698 INFO:teuthology.orchestra.run.smithi070.stdout:119 2024-04-01T01:31:27.698 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-04-01T01:31:27.699 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-01T01:31:27.699 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-01T01:31:27.699 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-01T01:31:27.699 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-01T01:31:27.699 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-01T01:31:27.699 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-01T01:31:27.699 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-01T01:31:27.699 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-01T01:31:27.699 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-01T01:31:27.699 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-01T01:31:27.699 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-01T01:31:27.700 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-01T01:31:27.700 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-01T01:31:27.700 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-01T01:31:27.700 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-01T01:31:27.700 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-01T01:31:27.700 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-01T01:31:27.700 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-01T01:31:27.700 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-01T01:31:27.700 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-01T01:31:27.700 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-01T01:31:27.700 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-01T01:31:27.700 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-01T01:31:27.701 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-01T01:31:27.701 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-01T01:31:27.701 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.admin")) 2024-04-01T01:31:27.701 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-01T01:31:27.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:27 smithi070 ceph-mon[32179]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 876 B/s wr, 2 op/s 2024-04-01T01:31:27.819 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.admin.76219.asok 2024-04-01T01:31:27.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-01T01:31:27.827 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:27 smithi081 ceph-mon[35142]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 876 B/s wr, 2 op/s 2024-04-01T01:31:27.829 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.76219.asok 2024-04-01T01:31:27.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:31:27.829 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.76219.asok status 2024-04-01T01:31:27.967 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:31:27.967 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-01T01:31:27.967 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "384f31f3d3aee7f8ba3f60846b8844c049165583", 2024-04-01T01:31:27.967 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev)", 2024-04-01T01:31:27.967 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "admin", 2024-04-01T01:31:27.967 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-01T01:31:27.968 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-01T01:31:27.968 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "76219", 2024-04-01T01:31:27.968 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/" 2024-04-01T01:31:27.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:31:27.968 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-01T01:31:27.968 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-01T01:31:27.968 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 14763, 2024-04-01T01:31:27.968 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-01T01:31:27.968 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-01T01:31:27.968 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-01T01:31:27.968 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 14763 2024-04-01T01:31:27.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:31:27.969 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-01T01:31:27.969 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-01T01:31:27.969 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-01T01:31:27.969 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 878326439 2024-04-01T01:31:27.969 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:31:27.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:31:27.969 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-01T01:31:27.969 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-01T01:31:27.969 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-01T01:31:27.969 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 878326439 2024-04-01T01:31:27.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:31:27.970 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.14763 192.168.0.1:0/878326439", 2024-04-01T01:31:27.970 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.1:0/878326439", 2024-04-01T01:31:27.970 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 1, 2024-04-01T01:31:27.970 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 17, 2024-04-01T01:31:27.970 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 89, 2024-04-01T01:31:27.970 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-01T01:31:27.970 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-01T01:31:27.970 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-01T01:31:27.970 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:31:27.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:31:27.976 DEBUG:teuthology.orchestra.run.smithi070:> stat --file-system '--printf=%T 2024-04-01T01:31:27.976 DEBUG:teuthology.orchestra.run.smithi070:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-01T01:31:27.991 INFO:teuthology.orchestra.run.smithi070.stdout:fuseblk 2024-04-01T01:31:27.991 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-01T01:31:27.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:31:27.992 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-01T01:31:27.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:27 smithi189 ceph-mon[35753]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 876 B/s wr, 2 op/s 2024-04-01T01:31:28.060 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-01T01:31:28.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-01T01:31:28.173 INFO:teuthology.orchestra.run.smithi070.stdout:# file: . 2024-04-01T01:31:28.173 INFO:teuthology.orchestra.run.smithi070.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-01T01:31:28.173 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:28.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-01T01:31:28.178 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:31:28.178 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-01T01:31:28.195 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:31:28.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.336+0000 7f6700c07640 1 -- 172.21.15.70:0/2393169603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66fc1021f0 msgr2=0x7f66fc1045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:28.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.336+0000 7f6700c07640 1 --2- 172.21.15.70:0/2393169603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66fc1021f0 0x7f66fc1045e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f66ec009ca0 tx=0x7f66ec02c0f0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.336+0000 7f6700c07640 1 -- 172.21.15.70:0/2393169603 shutdown_connections 2024-04-01T01:31:28.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.336+0000 7f6700c07640 1 --2- 172.21.15.70:0/2393169603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66fc107450 0x7f66fc1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.336+0000 7f6700c07640 1 --2- 172.21.15.70:0/2393169603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66fc104b20 0x7f66fc106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.336+0000 7f6700c07640 1 --2- 172.21.15.70:0/2393169603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66fc1021f0 0x7f66fc1045e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.336+0000 7f6700c07640 1 -- 172.21.15.70:0/2393169603 >> 172.21.15.70:0/2393169603 conn(0x7f66fc0fbbe0 msgr2=0x7f66fc0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:28.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.336+0000 7f6700c07640 1 -- 172.21.15.70:0/2393169603 shutdown_connections 2024-04-01T01:31:28.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.336+0000 7f6700c07640 1 -- 172.21.15.70:0/2393169603 wait complete. 2024-04-01T01:31:28.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.337+0000 7f6700c07640 1 Processor -- start 2024-04-01T01:31:28.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.337+0000 7f6700c07640 1 -- start start 2024-04-01T01:31:28.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.337+0000 7f6700c07640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66fc1021f0 0x7f66fc101b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:28.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.337+0000 7f6700c07640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66fc104b20 0x7f66fc1001a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:28.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.337+0000 7f6700c07640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66fc107450 0x7f66fc1006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:28.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.337+0000 7f6700c07640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f66fc10d2d0 con 0x7f66fc104b20 2024-04-01T01:31:28.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.337+0000 7f6700c07640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f66fc10d150 con 0x7f66fc107450 2024-04-01T01:31:28.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.337+0000 7f6700c07640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f66fc10d450 con 0x7f66fc1021f0 2024-04-01T01:31:28.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.337+0000 7f66f9d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66fc104b20 0x7f66fc1001a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:28.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.337+0000 7f66f9d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66fc104b20 0x7f66fc1001a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40772/0 (socket says 172.21.15.70:40772) 2024-04-01T01:31:28.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.337+0000 7f66f9d9b640 1 -- 172.21.15.70:0/3453357567 learned_addr learned my addr 172.21.15.70:0/3453357567 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:28.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.337+0000 7f66fad9d640 1 --2- 172.21.15.70:0/3453357567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66fc107450 0x7f66fc1006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:28.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.338+0000 7f66fa59c640 1 --2- 172.21.15.70:0/3453357567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66fc1021f0 0x7f66fc101b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:28.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.338+0000 7f66f9d9b640 1 -- 172.21.15.70:0/3453357567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66fc1021f0 msgr2=0x7f66fc101b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:28.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.338+0000 7f66f9d9b640 1 --2- 172.21.15.70:0/3453357567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66fc1021f0 0x7f66fc101b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.338+0000 7f66f9d9b640 1 -- 172.21.15.70:0/3453357567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66fc107450 msgr2=0x7f66fc1006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:28.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.338+0000 7f66f9d9b640 1 --2- 172.21.15.70:0/3453357567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66fc107450 0x7f66fc1006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.338+0000 7f66f9d9b640 1 -- 172.21.15.70:0/3453357567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66fc100f70 con 0x7f66fc104b20 2024-04-01T01:31:28.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.338+0000 7f66f9d9b640 1 --2- 172.21.15.70:0/3453357567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66fc104b20 0x7f66fc1001a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f66f000d740 tx=0x7f66f000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:28.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.338+0000 7f66eb7fe640 1 -- 172.21.15.70:0/3453357567 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f0017070 con 0x7f66fc104b20 2024-04-01T01:31:28.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.338+0000 7f66eb7fe640 1 -- 172.21.15.70:0/3453357567 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66f000de50 con 0x7f66fc104b20 2024-04-01T01:31:28.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.338+0000 7f6700c07640 1 -- 172.21.15.70:0/3453357567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66fc1a1b20 con 0x7f66fc104b20 2024-04-01T01:31:28.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.338+0000 7f66eb7fe640 1 -- 172.21.15.70:0/3453357567 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f0004d60 con 0x7f66fc104b20 2024-04-01T01:31:28.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.338+0000 7f6700c07640 1 -- 172.21.15.70:0/3453357567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66fc1a2020 con 0x7f66fc104b20 2024-04-01T01:31:28.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.339+0000 7f6700c07640 1 -- 172.21.15.70:0/3453357567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66fc0fd6d0 con 0x7f66fc104b20 2024-04-01T01:31:28.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.340+0000 7f66eb7fe640 1 -- 172.21.15.70:0/3453357567 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f66f0004f00 con 0x7f66fc104b20 2024-04-01T01:31:28.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.340+0000 7f66eb7fe640 1 --2- 172.21.15.70:0/3453357567 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66e40a24a0 0x7f66e40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:28.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.341+0000 7f66fa59c640 1 --2- 172.21.15.70:0/3453357567 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66e40a24a0 0x7f66e40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:28.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.341+0000 7f66eb7fe640 1 -- 172.21.15.70:0/3453357567 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f66f00ce320 con 0x7f66fc104b20 2024-04-01T01:31:28.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.341+0000 7f66fa59c640 1 --2- 172.21.15.70:0/3453357567 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66e40a24a0 0x7f66e40a4960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f66ec006010 tx=0x7f66ec032340 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:28.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.345+0000 7f66eb7fe640 1 -- 172.21.15.70:0/3453357567 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f66f009b2a0 con 0x7f66fc104b20 2024-04-01T01:31:28.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.520+0000 7f6700c07640 1 -- 172.21.15.70:0/3453357567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f66fc062380 con 0x7f66fc104b20 2024-04-01T01:31:28.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.522+0000 7f66eb7fe640 1 -- 172.21.15.70:0/3453357567 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24068 (secure 0 0 0) 0x7f66f009aba0 con 0x7f66fc104b20 2024-04-01T01:31:28.522 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:28.522 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":89,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:31:20.903623+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:31:28.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.524+0000 7f6700c07640 1 -- 172.21.15.70:0/3453357567 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66e40a24a0 msgr2=0x7f66e40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:28.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.524+0000 7f6700c07640 1 --2- 172.21.15.70:0/3453357567 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66e40a24a0 0x7f66e40a4960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f66ec006010 tx=0x7f66ec032340 comp rx=0 tx=0).stop 2024-04-01T01:31:28.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.524+0000 7f6700c07640 1 -- 172.21.15.70:0/3453357567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66fc104b20 msgr2=0x7f66fc1001a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:28.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.524+0000 7f6700c07640 1 --2- 172.21.15.70:0/3453357567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66fc104b20 0x7f66fc1001a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f66f000d740 tx=0x7f66f000dc00 comp rx=0 tx=0).stop 2024-04-01T01:31:28.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.524+0000 7f6700c07640 1 -- 172.21.15.70:0/3453357567 shutdown_connections 2024-04-01T01:31:28.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.524+0000 7f6700c07640 1 --2- 172.21.15.70:0/3453357567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66fc107450 0x7f66fc1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.524+0000 7f6700c07640 1 --2- 172.21.15.70:0/3453357567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66fc104b20 0x7f66fc1001a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.524+0000 7f6700c07640 1 --2- 172.21.15.70:0/3453357567 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66e40a24a0 0x7f66e40a4960 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.524+0000 7f6700c07640 1 --2- 172.21.15.70:0/3453357567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66fc1021f0 0x7f66fc101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.524+0000 7f6700c07640 1 -- 172.21.15.70:0/3453357567 >> 172.21.15.70:0/3453357567 conn(0x7f66fc0fbbe0 msgr2=0x7f66fc102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:28.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.524+0000 7f6700c07640 1 -- 172.21.15.70:0/3453357567 shutdown_connections 2024-04-01T01:31:28.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.524+0000 7f6700c07640 1 -- 172.21.15.70:0/3453357567 wait complete. 2024-04-01T01:31:28.535 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-01T01:26:42.809779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-01T01:31:13.612369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-01T01:31:14.910162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-01T01:31:18.783179+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-01T01:31:28.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-01T01:31:28.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.644+0000 7fdf652c9640 1 -- 172.21.15.70:0/3597226858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf60100650 msgr2=0x7fdf60100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:28.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.644+0000 7fdf652c9640 1 --2- 172.21.15.70:0/3597226858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf60100650 0x7fdf60100a70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdf50009900 tx=0x7fdf5002bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:28.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.644+0000 7fdf652c9640 1 -- 172.21.15.70:0/3597226858 shutdown_connections 2024-04-01T01:31:28.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.644+0000 7fdf652c9640 1 --2- 172.21.15.70:0/3597226858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf60102200 0x7fdf60108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.644+0000 7fdf652c9640 1 --2- 172.21.15.70:0/3597226858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf60101840 0x7fdf60101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.644+0000 7fdf652c9640 1 --2- 172.21.15.70:0/3597226858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf60100650 0x7fdf60100a70 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.644+0000 7fdf652c9640 1 -- 172.21.15.70:0/3597226858 >> 172.21.15.70:0/3597226858 conn(0x7fdf600fbbc0 msgr2=0x7fdf600fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:28.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.644+0000 7fdf652c9640 1 -- 172.21.15.70:0/3597226858 shutdown_connections 2024-04-01T01:31:28.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.644+0000 7fdf652c9640 1 -- 172.21.15.70:0/3597226858 wait complete. 2024-04-01T01:31:28.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.644+0000 7fdf652c9640 1 Processor -- start 2024-04-01T01:31:28.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.644+0000 7fdf652c9640 1 -- start start 2024-04-01T01:31:28.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf652c9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf60100650 0x7fdf601979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:28.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf652c9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf60101840 0x7fdf60197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:28.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf652c9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf60102200 0x7fdf6019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:28.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf652c9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6010c1d0 con 0x7fdf60100650 2024-04-01T01:31:28.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf652c9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6010c050 con 0x7fdf60102200 2024-04-01T01:31:28.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf652c9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6010c350 con 0x7fdf60101840 2024-04-01T01:31:28.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf5effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf60100650 0x7fdf601979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:28.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf5effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf60100650 0x7fdf601979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40782/0 (socket says 172.21.15.70:40782) 2024-04-01T01:31:28.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf5effd640 1 -- 172.21.15.70:0/1676972294 learned_addr learned my addr 172.21.15.70:0/1676972294 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:28.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf5f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf60102200 0x7fdf6019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:28.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf567fc640 1 --2- 172.21.15.70:0/1676972294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf60101840 0x7fdf60197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:28.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf5effd640 1 -- 172.21.15.70:0/1676972294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf60101840 msgr2=0x7fdf60197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:28.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf5effd640 1 --2- 172.21.15.70:0/1676972294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf60101840 0x7fdf60197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf5effd640 1 -- 172.21.15.70:0/1676972294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf60102200 msgr2=0x7fdf6019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:28.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf5effd640 1 --2- 172.21.15.70:0/1676972294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf60102200 0x7fdf6019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf5effd640 1 -- 172.21.15.70:0/1676972294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6019f5a0 con 0x7fdf60100650 2024-04-01T01:31:28.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf567fc640 1 --2- 172.21.15.70:0/1676972294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf60101840 0x7fdf60197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:31:28.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf5f7fe640 1 --2- 172.21.15.70:0/1676972294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf60102200 0x7fdf6019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:28.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf5effd640 1 --2- 172.21.15.70:0/1676972294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf60100650 0x7fdf601979d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdf50000c00 tx=0x7fdf500040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:28.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf5cff9640 1 -- 172.21.15.70:0/1676972294 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf50043020 con 0x7fdf60100650 2024-04-01T01:31:28.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf5cff9640 1 -- 172.21.15.70:0/1676972294 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdf500044c0 con 0x7fdf60100650 2024-04-01T01:31:28.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf652c9640 1 -- 172.21.15.70:0/1676972294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf6019f830 con 0x7fdf60100650 2024-04-01T01:31:28.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf5cff9640 1 -- 172.21.15.70:0/1676972294 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf50003c60 con 0x7fdf60100650 2024-04-01T01:31:28.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.645+0000 7fdf652c9640 1 -- 172.21.15.70:0/1676972294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf6019fdd0 con 0x7fdf60100650 2024-04-01T01:31:28.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.646+0000 7fdf652c9640 1 -- 172.21.15.70:0/1676972294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf601a0030 con 0x7fdf60100650 2024-04-01T01:31:28.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.647+0000 7fdf5cff9640 1 -- 172.21.15.70:0/1676972294 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fdf50046020 con 0x7fdf60100650 2024-04-01T01:31:28.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.648+0000 7fdf5cff9640 1 --2- 172.21.15.70:0/1676972294 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdf4c0a2520 0x7fdf4c0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:28.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.648+0000 7fdf567fc640 1 --2- 172.21.15.70:0/1676972294 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdf4c0a2520 0x7fdf4c0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:28.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.648+0000 7fdf5cff9640 1 -- 172.21.15.70:0/1676972294 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdf500f0bb0 con 0x7fdf60100650 2024-04-01T01:31:28.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.648+0000 7fdf567fc640 1 --2- 172.21.15.70:0/1676972294 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdf4c0a2520 0x7fdf4c0a49e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdf60198e90 tx=0x7fdf48005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:28.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.652+0000 7fdf5cff9640 1 -- 172.21.15.70:0/1676972294 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdf5004d020 con 0x7fdf60100650 2024-04-01T01:31:28.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.825+0000 7fdf652c9640 1 -- 172.21.15.70:0/1676972294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdf601a0200 con 0x7fdf60100650 2024-04-01T01:31:28.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.825+0000 7fdf5cff9640 1 -- 172.21.15.70:0/1676972294 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fdf500bdb30 con 0x7fdf60100650 2024-04-01T01:31:28.826 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-01T01:31:28.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.827+0000 7fdf652c9640 1 -- 172.21.15.70:0/1676972294 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdf4c0a2520 msgr2=0x7fdf4c0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:28.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.827+0000 7fdf652c9640 1 --2- 172.21.15.70:0/1676972294 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdf4c0a2520 0x7fdf4c0a49e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdf60198e90 tx=0x7fdf48005f90 comp rx=0 tx=0).stop 2024-04-01T01:31:28.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.827+0000 7fdf652c9640 1 -- 172.21.15.70:0/1676972294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf60100650 msgr2=0x7fdf601979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:28.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.827+0000 7fdf652c9640 1 --2- 172.21.15.70:0/1676972294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf60100650 0x7fdf601979d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdf50000c00 tx=0x7fdf500040c0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.827+0000 7fdf652c9640 1 -- 172.21.15.70:0/1676972294 shutdown_connections 2024-04-01T01:31:28.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.827+0000 7fdf652c9640 1 --2- 172.21.15.70:0/1676972294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf60102200 0x7fdf6019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.827+0000 7fdf652c9640 1 --2- 172.21.15.70:0/1676972294 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdf4c0a2520 0x7fdf4c0a49e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.827+0000 7fdf652c9640 1 --2- 172.21.15.70:0/1676972294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf60101840 0x7fdf60197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.827+0000 7fdf652c9640 1 --2- 172.21.15.70:0/1676972294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdf60100650 0x7fdf601979d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.827+0000 7fdf652c9640 1 -- 172.21.15.70:0/1676972294 >> 172.21.15.70:0/1676972294 conn(0x7fdf600fbbc0 msgr2=0x7fdf600fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:28.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.827+0000 7fdf652c9640 1 -- 172.21.15.70:0/1676972294 shutdown_connections 2024-04-01T01:31:28.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.828+0000 7fdf652c9640 1 -- 172.21.15.70:0/1676972294 wait complete. 2024-04-01T01:31:28.837 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:28 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3453357567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:28.838 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-01T01:31:28.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.943+0000 7f51974c0640 1 -- 172.21.15.70:0/3502443803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5190100650 msgr2=0x7f5190100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:28.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.943+0000 7f51974c0640 1 --2- 172.21.15.70:0/3502443803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5190100650 0x7f5190100a70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5188009ca0 tx=0x7f518802c0f0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.943+0000 7f51974c0640 1 -- 172.21.15.70:0/3502443803 shutdown_connections 2024-04-01T01:31:28.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.943+0000 7f51974c0640 1 --2- 172.21.15.70:0/3502443803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5190102200 0x7f5190108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.943+0000 7f51974c0640 1 --2- 172.21.15.70:0/3502443803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5190101840 0x7f5190101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.943+0000 7f51974c0640 1 --2- 172.21.15.70:0/3502443803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5190100650 0x7f5190100a70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.943+0000 7f51974c0640 1 -- 172.21.15.70:0/3502443803 >> 172.21.15.70:0/3502443803 conn(0x7f51900fbbe0 msgr2=0x7f51900fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:28.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.944+0000 7f51974c0640 1 -- 172.21.15.70:0/3502443803 shutdown_connections 2024-04-01T01:31:28.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.944+0000 7f51974c0640 1 -- 172.21.15.70:0/3502443803 wait complete. 2024-04-01T01:31:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.944+0000 7f51974c0640 1 Processor -- start 2024-04-01T01:31:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.944+0000 7f51974c0640 1 -- start start 2024-04-01T01:31:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.944+0000 7f51974c0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5190100650 0x7f5190196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.944+0000 7f51974c0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5190101840 0x7f5190196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.944+0000 7f51974c0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5190102200 0x7f519019df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.944+0000 7f51974c0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f519010c1d0 con 0x7f5190101840 2024-04-01T01:31:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.944+0000 7f51974c0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f519010c050 con 0x7f5190102200 2024-04-01T01:31:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.944+0000 7f51974c0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f519010c350 con 0x7f5190100650 2024-04-01T01:31:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.944+0000 7f5194a5b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5190101840 0x7f5190196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.944+0000 7f5194a5b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5190101840 0x7f5190196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40786/0 (socket says 172.21.15.70:40786) 2024-04-01T01:31:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.944+0000 7f5194a5b640 1 -- 172.21.15.70:0/410104752 learned_addr learned my addr 172.21.15.70:0/410104752 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.945+0000 7f5195a5d640 1 --2- 172.21.15.70:0/410104752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5190102200 0x7f519019df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.945+0000 7f519525c640 1 --2- 172.21.15.70:0/410104752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5190100650 0x7f5190196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:28.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.945+0000 7f5194a5b640 1 -- 172.21.15.70:0/410104752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5190100650 msgr2=0x7f5190196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:28.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.945+0000 7f5194a5b640 1 --2- 172.21.15.70:0/410104752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5190100650 0x7f5190196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.945+0000 7f5194a5b640 1 -- 172.21.15.70:0/410104752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5190102200 msgr2=0x7f519019df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:28.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.945+0000 7f5194a5b640 1 --2- 172.21.15.70:0/410104752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5190102200 0x7f519019df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:28.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.945+0000 7f5194a5b640 1 -- 172.21.15.70:0/410104752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f519019e570 con 0x7f5190101840 2024-04-01T01:31:28.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.945+0000 7f5194a5b640 1 --2- 172.21.15.70:0/410104752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5190101840 0x7f5190196e90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f518400e9f0 tx=0x7f518400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:28.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.945+0000 7f518e7fc640 1 -- 172.21.15.70:0/410104752 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518400cd80 con 0x7f5190101840 2024-04-01T01:31:28.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.945+0000 7f518e7fc640 1 -- 172.21.15.70:0/410104752 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f51840108e0 con 0x7f5190101840 2024-04-01T01:31:28.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.945+0000 7f51974c0640 1 -- 172.21.15.70:0/410104752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f519019e860 con 0x7f5190101840 2024-04-01T01:31:28.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.945+0000 7f518e7fc640 1 -- 172.21.15.70:0/410104752 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5184018920 con 0x7f5190101840 2024-04-01T01:31:28.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.945+0000 7f51974c0640 1 -- 172.21.15.70:0/410104752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f519019ee30 con 0x7f5190101840 2024-04-01T01:31:28.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.946+0000 7f51974c0640 1 -- 172.21.15.70:0/410104752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51901079e0 con 0x7f5190101840 2024-04-01T01:31:28.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.948+0000 7f518e7fc640 1 -- 172.21.15.70:0/410104752 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f5184010a90 con 0x7f5190101840 2024-04-01T01:31:28.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.948+0000 7f518e7fc640 1 --2- 172.21.15.70:0/410104752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f51780a24a0 0x7f51780a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:28.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.948+0000 7f519525c640 1 --2- 172.21.15.70:0/410104752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f51780a24a0 0x7f51780a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:28.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.948+0000 7f518e7fc640 1 -- 172.21.15.70:0/410104752 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f51840cf1a0 con 0x7f5190101840 2024-04-01T01:31:28.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.949+0000 7f519525c640 1 --2- 172.21.15.70:0/410104752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f51780a24a0 0x7f51780a4960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f518802c700 tx=0x7f5188000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:28.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:28.952+0000 7f518e7fc640 1 -- 172.21.15.70:0/410104752 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f518409c120 con 0x7f5190101840 2024-04-01T01:31:28.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:28 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3453357567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:28 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3453357567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:29.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.125+0000 7f51974c0640 1 -- 172.21.15.70:0/410104752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f519002d3e0 con 0x7f5190101840 2024-04-01T01:31:29.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.125+0000 7f518e7fc640 1 -- 172.21.15.70:0/410104752 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f518409ba20 con 0x7f5190101840 2024-04-01T01:31:29.126 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:31:29.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.127+0000 7f51974c0640 1 -- 172.21.15.70:0/410104752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f51780a24a0 msgr2=0x7f51780a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:29.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.127+0000 7f51974c0640 1 --2- 172.21.15.70:0/410104752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f51780a24a0 0x7f51780a4960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f518802c700 tx=0x7f5188000f40 comp rx=0 tx=0).stop 2024-04-01T01:31:29.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.127+0000 7f51974c0640 1 -- 172.21.15.70:0/410104752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5190101840 msgr2=0x7f5190196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:29.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.127+0000 7f51974c0640 1 --2- 172.21.15.70:0/410104752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5190101840 0x7f5190196e90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f518400e9f0 tx=0x7f518400eeb0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.127+0000 7f51974c0640 1 -- 172.21.15.70:0/410104752 shutdown_connections 2024-04-01T01:31:29.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.127+0000 7f51974c0640 1 --2- 172.21.15.70:0/410104752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5190102200 0x7f519019df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.127+0000 7f51974c0640 1 --2- 172.21.15.70:0/410104752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5190101840 0x7f5190196e90 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.127+0000 7f51974c0640 1 --2- 172.21.15.70:0/410104752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f51780a24a0 0x7f51780a4960 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.127+0000 7f51974c0640 1 --2- 172.21.15.70:0/410104752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5190100650 0x7f5190196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.127+0000 7f51974c0640 1 -- 172.21.15.70:0/410104752 >> 172.21.15.70:0/410104752 conn(0x7f51900fbbe0 msgr2=0x7f5190104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:29.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.127+0000 7f51974c0640 1 -- 172.21.15.70:0/410104752 shutdown_connections 2024-04-01T01:31:29.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.127+0000 7f51974c0640 1 -- 172.21.15.70:0/410104752 wait complete. 2024-04-01T01:31:29.138 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-01T01:31:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.243+0000 7f012b94b640 1 -- 172.21.15.70:0/1917965528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0124107440 msgr2=0x7f0124109890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.243+0000 7f012b94b640 1 --2- 172.21.15.70:0/1917965528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0124107440 0x7f0124109890 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f011c009ca0 tx=0x7f011c02c0f0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.243+0000 7f012b94b640 1 -- 172.21.15.70:0/1917965528 shutdown_connections 2024-04-01T01:31:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.243+0000 7f012b94b640 1 --2- 172.21.15.70:0/1917965528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0124107440 0x7f0124109890 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.243+0000 7f012b94b640 1 --2- 172.21.15.70:0/1917965528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0124104b10 0x7f0124106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.243+0000 7f012b94b640 1 --2- 172.21.15.70:0/1917965528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01241021e0 0x7f01241045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.243+0000 7f012b94b640 1 -- 172.21.15.70:0/1917965528 >> 172.21.15.70:0/1917965528 conn(0x7f01240fbbf0 msgr2=0x7f01240fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.243+0000 7f012b94b640 1 -- 172.21.15.70:0/1917965528 shutdown_connections 2024-04-01T01:31:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.243+0000 7f012b94b640 1 -- 172.21.15.70:0/1917965528 wait complete. 2024-04-01T01:31:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f012b94b640 1 Processor -- start 2024-04-01T01:31:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f012b94b640 1 -- start start 2024-04-01T01:31:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f012b94b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01241021e0 0x7f0124197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f012b94b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0124104b10 0x7f0124197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f012b94b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0124107440 0x7f012419eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f012b94b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f012410d290 con 0x7f0124104b10 2024-04-01T01:31:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f012b94b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f012410d110 con 0x7f01241021e0 2024-04-01T01:31:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f012b94b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f012410d410 con 0x7f0124107440 2024-04-01T01:31:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f0128ee6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0124104b10 0x7f0124197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f01296e7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01241021e0 0x7f0124197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f0128ee6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0124104b10 0x7f0124197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40802/0 (socket says 172.21.15.70:40802) 2024-04-01T01:31:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f0128ee6640 1 -- 172.21.15.70:0/2501673535 learned_addr learned my addr 172.21.15.70:0/2501673535 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f0129ee8640 1 --2- 172.21.15.70:0/2501673535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0124107440 0x7f012419eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f0128ee6640 1 -- 172.21.15.70:0/2501673535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0124107440 msgr2=0x7f012419eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f0128ee6640 1 --2- 172.21.15.70:0/2501673535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0124107440 0x7f012419eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f0128ee6640 1 -- 172.21.15.70:0/2501673535 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01241021e0 msgr2=0x7f0124197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f0128ee6640 1 --2- 172.21.15.70:0/2501673535 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01241021e0 0x7f0124197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f0128ee6640 1 -- 172.21.15.70:0/2501673535 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012419f550 con 0x7f0124104b10 2024-04-01T01:31:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f0129ee8640 1 --2- 172.21.15.70:0/2501673535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0124107440 0x7f012419eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:31:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f01296e7640 1 --2- 172.21.15.70:0/2501673535 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01241021e0 0x7f0124197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f0128ee6640 1 --2- 172.21.15.70:0/2501673535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0124104b10 0x7f0124197e70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f011400e9a0 tx=0x7f011400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.244+0000 7f011a7fc640 1 -- 172.21.15.70:0/2501673535 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011400cd80 con 0x7f0124104b10 2024-04-01T01:31:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.245+0000 7f011a7fc640 1 -- 172.21.15.70:0/2501673535 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01140035f0 con 0x7f0124104b10 2024-04-01T01:31:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.245+0000 7f012b94b640 1 -- 172.21.15.70:0/2501673535 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f012419f840 con 0x7f0124104b10 2024-04-01T01:31:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.245+0000 7f011a7fc640 1 -- 172.21.15.70:0/2501673535 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0114010690 con 0x7f0124104b10 2024-04-01T01:31:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.245+0000 7f012b94b640 1 -- 172.21.15.70:0/2501673535 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f012419fd90 con 0x7f0124104b10 2024-04-01T01:31:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.245+0000 7f012b94b640 1 -- 172.21.15.70:0/2501673535 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01240fd6e0 con 0x7f0124104b10 2024-04-01T01:31:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.247+0000 7f011a7fc640 1 -- 172.21.15.70:0/2501673535 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f01140037a0 con 0x7f0124104b10 2024-04-01T01:31:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.247+0000 7f011a7fc640 1 --2- 172.21.15.70:0/2501673535 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f01100a24a0 0x7f01100a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.248+0000 7f01296e7640 1 --2- 172.21.15.70:0/2501673535 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f01100a24a0 0x7f01100a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.248+0000 7f011a7fc640 1 -- 172.21.15.70:0/2501673535 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f01140cedc0 con 0x7f0124104b10 2024-04-01T01:31:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.248+0000 7f01296e7640 1 --2- 172.21.15.70:0/2501673535 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f01100a24a0 0x7f01100a4960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f012000bfd0 tx=0x7f012000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.251+0000 7f011a7fc640 1 -- 172.21.15.70:0/2501673535 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f011409bdd0 con 0x7f0124104b10 2024-04-01T01:31:29.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.424+0000 7f012b94b640 1 -- 172.21.15.70:0/2501673535 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f012402d300 con 0x7f0124104b10 2024-04-01T01:31:29.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.424+0000 7f011a7fc640 1 -- 172.21.15.70:0/2501673535 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f011409b6d0 con 0x7f0124104b10 2024-04-01T01:31:29.425 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-01T01:31:29.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.427+0000 7f012b94b640 1 -- 172.21.15.70:0/2501673535 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f01100a24a0 msgr2=0x7f01100a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:29.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.427+0000 7f012b94b640 1 --2- 172.21.15.70:0/2501673535 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f01100a24a0 0x7f01100a4960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f012000bfd0 tx=0x7f012000b390 comp rx=0 tx=0).stop 2024-04-01T01:31:29.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.427+0000 7f012b94b640 1 -- 172.21.15.70:0/2501673535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0124104b10 msgr2=0x7f0124197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:29.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.427+0000 7f012b94b640 1 --2- 172.21.15.70:0/2501673535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0124104b10 0x7f0124197e70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f011400e9a0 tx=0x7f011400ee60 comp rx=0 tx=0).stop 2024-04-01T01:31:29.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.427+0000 7f012b94b640 1 -- 172.21.15.70:0/2501673535 shutdown_connections 2024-04-01T01:31:29.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.427+0000 7f012b94b640 1 --2- 172.21.15.70:0/2501673535 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f01100a24a0 0x7f01100a4960 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.427+0000 7f012b94b640 1 --2- 172.21.15.70:0/2501673535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0124107440 0x7f012419eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.427+0000 7f012b94b640 1 --2- 172.21.15.70:0/2501673535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0124104b10 0x7f0124197e70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.427+0000 7f012b94b640 1 --2- 172.21.15.70:0/2501673535 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01241021e0 0x7f0124197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.427+0000 7f012b94b640 1 -- 172.21.15.70:0/2501673535 >> 172.21.15.70:0/2501673535 conn(0x7f01240fbbf0 msgr2=0x7f0124102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:29.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.427+0000 7f012b94b640 1 -- 172.21.15.70:0/2501673535 shutdown_connections 2024-04-01T01:31:29.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.427+0000 7f012b94b640 1 -- 172.21.15.70:0/2501673535 wait complete. 2024-04-01T01:31:29.438 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-01T01:31:29.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.542+0000 7f288f15c640 1 -- 172.21.15.70:0/3860861787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2888100650 msgr2=0x7f2888100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.542+0000 7f288f15c640 1 --2- 172.21.15.70:0/3860861787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2888100650 0x7f2888100a70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f287c009900 tx=0x7f287c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288f15c640 1 -- 172.21.15.70:0/3860861787 shutdown_connections 2024-04-01T01:31:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288f15c640 1 --2- 172.21.15.70:0/3860861787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2888102200 0x7f2888108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288f15c640 1 --2- 172.21.15.70:0/3860861787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888101840 0x7f2888101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288f15c640 1 --2- 172.21.15.70:0/3860861787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2888100650 0x7f2888100a70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288f15c640 1 -- 172.21.15.70:0/3860861787 >> 172.21.15.70:0/3860861787 conn(0x7f28880fbbe0 msgr2=0x7f28880fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288f15c640 1 -- 172.21.15.70:0/3860861787 shutdown_connections 2024-04-01T01:31:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288f15c640 1 -- 172.21.15.70:0/3860861787 wait complete. 2024-04-01T01:31:29.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288f15c640 1 Processor -- start 2024-04-01T01:31:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288f15c640 1 -- start start 2024-04-01T01:31:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288f15c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2888100650 0x7f2888197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288f15c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888101840 0x7f2888197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288f15c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2888102200 0x7f288819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288f15c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f288810c180 con 0x7f2888102200 2024-04-01T01:31:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288f15c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f288810c000 con 0x7f2888100650 2024-04-01T01:31:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288f15c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f288810c300 con 0x7f2888101840 2024-04-01T01:31:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288d6f9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2888102200 0x7f288819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288d6f9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2888102200 0x7f288819f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40834/0 (socket says 172.21.15.70:40834) 2024-04-01T01:31:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.543+0000 7f288d6f9640 1 -- 172.21.15.70:0/3208781804 learned_addr learned my addr 172.21.15.70:0/3208781804 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.544+0000 7f288cef8640 1 --2- 172.21.15.70:0/3208781804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2888100650 0x7f2888197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:29.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.544+0000 7f2887fff640 1 --2- 172.21.15.70:0/3208781804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888101840 0x7f2888197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.544+0000 7f288d6f9640 1 -- 172.21.15.70:0/3208781804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888101840 msgr2=0x7f2888197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.544+0000 7f288d6f9640 1 --2- 172.21.15.70:0/3208781804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888101840 0x7f2888197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.544+0000 7f288d6f9640 1 -- 172.21.15.70:0/3208781804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2888100650 msgr2=0x7f2888197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.544+0000 7f288d6f9640 1 --2- 172.21.15.70:0/3208781804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2888100650 0x7f2888197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.544+0000 7f288d6f9640 1 -- 172.21.15.70:0/3208781804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f288819f680 con 0x7f2888102200 2024-04-01T01:31:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.544+0000 7f2887fff640 1 --2- 172.21.15.70:0/3208781804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888101840 0x7f2888197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:31:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.544+0000 7f288d6f9640 1 --2- 172.21.15.70:0/3208781804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2888102200 0x7f288819f010 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f288000efd0 tx=0x7f288000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.544+0000 7f2885ffb640 1 -- 172.21.15.70:0/3208781804 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2880003b70 con 0x7f2888102200 2024-04-01T01:31:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.544+0000 7f2885ffb640 1 -- 172.21.15.70:0/3208781804 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2880003d10 con 0x7f2888102200 2024-04-01T01:31:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.544+0000 7f288f15c640 1 -- 172.21.15.70:0/3208781804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f288819f970 con 0x7f2888102200 2024-04-01T01:31:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.544+0000 7f2885ffb640 1 -- 172.21.15.70:0/3208781804 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2880010980 con 0x7f2888102200 2024-04-01T01:31:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.544+0000 7f288f15c640 1 -- 172.21.15.70:0/3208781804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f288819fe70 con 0x7f2888102200 2024-04-01T01:31:29.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.545+0000 7f288f15c640 1 -- 172.21.15.70:0/3208781804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28881a0100 con 0x7f2888102200 2024-04-01T01:31:29.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.546+0000 7f2885ffb640 1 -- 172.21.15.70:0/3208781804 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f2880004020 con 0x7f2888102200 2024-04-01T01:31:29.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.547+0000 7f2885ffb640 1 --2- 172.21.15.70:0/3208781804 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28780a24a0 0x7f28780a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:29.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.547+0000 7f288cef8640 1 --2- 172.21.15.70:0/3208781804 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28780a24a0 0x7f28780a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:29.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.547+0000 7f2885ffb640 1 -- 172.21.15.70:0/3208781804 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f28800cee50 con 0x7f2888102200 2024-04-01T01:31:29.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.547+0000 7f288cef8640 1 --2- 172.21.15.70:0/3208781804 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28780a24a0 0x7f28780a4960 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f287c005ab0 tx=0x7f287c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:29.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.551+0000 7f2885ffb640 1 -- 172.21.15.70:0/3208781804 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f288009bdd0 con 0x7f2888102200 2024-04-01T01:31:29.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.726+0000 7f288f15c640 1 -- 172.21.15.70:0/3208781804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f28881a02d0 con 0x7f2888102200 2024-04-01T01:31:29.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.726+0000 7f2885ffb640 1 -- 172.21.15.70:0/3208781804 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f288009b6d0 con 0x7f2888102200 2024-04-01T01:31:29.727 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:31:29.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.728+0000 7f288f15c640 1 -- 172.21.15.70:0/3208781804 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28780a24a0 msgr2=0x7f28780a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:29.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.728+0000 7f288f15c640 1 --2- 172.21.15.70:0/3208781804 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28780a24a0 0x7f28780a4960 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f287c005ab0 tx=0x7f287c005fd0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.728+0000 7f288f15c640 1 -- 172.21.15.70:0/3208781804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2888102200 msgr2=0x7f288819f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:29.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.728+0000 7f288f15c640 1 --2- 172.21.15.70:0/3208781804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2888102200 0x7f288819f010 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f288000efd0 tx=0x7f288000c6a0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.728+0000 7f288f15c640 1 -- 172.21.15.70:0/3208781804 shutdown_connections 2024-04-01T01:31:29.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.728+0000 7f288f15c640 1 --2- 172.21.15.70:0/3208781804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2888102200 0x7f288819f010 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.728+0000 7f288f15c640 1 --2- 172.21.15.70:0/3208781804 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28780a24a0 0x7f28780a4960 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.728+0000 7f288f15c640 1 --2- 172.21.15.70:0/3208781804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888101840 0x7f2888197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.728+0000 7f288f15c640 1 --2- 172.21.15.70:0/3208781804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2888100650 0x7f2888197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.728+0000 7f288f15c640 1 -- 172.21.15.70:0/3208781804 >> 172.21.15.70:0/3208781804 conn(0x7f28880fbbe0 msgr2=0x7f2888068960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:29.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.728+0000 7f288f15c640 1 -- 172.21.15.70:0/3208781804 shutdown_connections 2024-04-01T01:31:29.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.728+0000 7f288f15c640 1 -- 172.21.15.70:0/3208781804 wait complete. 2024-04-01T01:31:29.738 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:29 smithi070 ceph-mon[32179]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 1 op/s 2024-04-01T01:31:29.738 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:29 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1676972294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:31:29.738 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:29 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/410104752' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:31:29.738 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:29 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2501673535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:31:29.742 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-01T01:31:29.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.843+0000 7fe80f6eb640 1 -- 172.21.15.70:0/623681606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8080ffa50 msgr2=0x7fe8080ffe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:29.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.844+0000 7fe80f6eb640 1 --2- 172.21.15.70:0/623681606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8080ffa50 0x7fe8080ffe70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8009860 tx=0x7fe7f802bb70 comp rx=0 tx=0).stop 2024-04-01T01:31:29.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.844+0000 7fe80f6eb640 1 -- 172.21.15.70:0/623681606 shutdown_connections 2024-04-01T01:31:29.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.844+0000 7fe80f6eb640 1 --2- 172.21.15.70:0/623681606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8081033e0 0x7fe808109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.844+0000 7fe80f6eb640 1 --2- 172.21.15.70:0/623681606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe808102a20 0x7fe808102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.844+0000 7fe80f6eb640 1 --2- 172.21.15.70:0/623681606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8080ffa50 0x7fe8080ffe70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.844+0000 7fe80f6eb640 1 -- 172.21.15.70:0/623681606 >> 172.21.15.70:0/623681606 conn(0x7fe8080fb380 msgr2=0x7fe8080fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:29.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.844+0000 7fe80f6eb640 1 -- 172.21.15.70:0/623681606 shutdown_connections 2024-04-01T01:31:29.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.844+0000 7fe80f6eb640 1 -- 172.21.15.70:0/623681606 wait complete. 2024-04-01T01:31:29.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.844+0000 7fe80f6eb640 1 Processor -- start 2024-04-01T01:31:29.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.844+0000 7fe80f6eb640 1 -- start start 2024-04-01T01:31:29.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.844+0000 7fe80f6eb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8080ffa50 0x7fe8081979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:29.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.844+0000 7fe80f6eb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe808102a20 0x7fe808197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:29.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.844+0000 7fe80f6eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8081033e0 0x7fe80819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:29.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.844+0000 7fe80f6eb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe80810d2d0 con 0x7fe808102a20 2024-04-01T01:31:29.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.844+0000 7fe80f6eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe80810d150 con 0x7fe8081033e0 2024-04-01T01:31:29.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.844+0000 7fe80f6eb640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe80810d450 con 0x7fe8080ffa50 2024-04-01T01:31:29.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe80cc86640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe808102a20 0x7fe808197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:29.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe80cc86640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe808102a20 0x7fe808197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40864/0 (socket says 172.21.15.70:40864) 2024-04-01T01:31:29.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe80cc86640 1 -- 172.21.15.70:0/2347501238 learned_addr learned my addr 172.21.15.70:0/2347501238 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:29.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe80dc88640 1 --2- 172.21.15.70:0/2347501238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8081033e0 0x7fe80819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:29.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe80d487640 1 --2- 172.21.15.70:0/2347501238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8080ffa50 0x7fe8081979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:29.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe80cc86640 1 -- 172.21.15.70:0/2347501238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8080ffa50 msgr2=0x7fe8081979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:29.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe80cc86640 1 --2- 172.21.15.70:0/2347501238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8080ffa50 0x7fe8081979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe80cc86640 1 -- 172.21.15.70:0/2347501238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8081033e0 msgr2=0x7fe80819ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:29.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe80cc86640 1 --2- 172.21.15.70:0/2347501238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8081033e0 0x7fe80819ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:29.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe80cc86640 1 -- 172.21.15.70:0/2347501238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe80819f5f0 con 0x7fe808102a20 2024-04-01T01:31:29.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe80dc88640 1 --2- 172.21.15.70:0/2347501238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8081033e0 0x7fe80819ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:29.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe80cc86640 1 --2- 172.21.15.70:0/2347501238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe808102a20 0x7fe808197f10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe80400e9c0 tx=0x7fe80400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:29.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe7fe7fc640 1 -- 172.21.15.70:0/2347501238 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80400cde0 con 0x7fe808102a20 2024-04-01T01:31:29.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe7fe7fc640 1 -- 172.21.15.70:0/2347501238 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8040035f0 con 0x7fe808102a20 2024-04-01T01:31:29.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe80f6eb640 1 -- 172.21.15.70:0/2347501238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe80819f8e0 con 0x7fe808102a20 2024-04-01T01:31:29.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe7fe7fc640 1 -- 172.21.15.70:0/2347501238 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804010690 con 0x7fe808102a20 2024-04-01T01:31:29.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.845+0000 7fe80f6eb640 1 -- 172.21.15.70:0/2347501238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe80819fe30 con 0x7fe808102a20 2024-04-01T01:31:29.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.846+0000 7fe80f6eb640 1 -- 172.21.15.70:0/2347501238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe808108c20 con 0x7fe808102a20 2024-04-01T01:31:29.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.848+0000 7fe7fe7fc640 1 -- 172.21.15.70:0/2347501238 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fe8040037a0 con 0x7fe808102a20 2024-04-01T01:31:29.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.848+0000 7fe7fe7fc640 1 --2- 172.21.15.70:0/2347501238 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe7f40a24a0 0x7fe7f40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:29.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.849+0000 7fe80d487640 1 --2- 172.21.15.70:0/2347501238 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe7f40a24a0 0x7fe7f40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:29.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.849+0000 7fe7fe7fc640 1 -- 172.21.15.70:0/2347501238 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe8040cecf0 con 0x7fe808102a20 2024-04-01T01:31:29.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.849+0000 7fe80d487640 1 --2- 172.21.15.70:0/2347501238 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe7f40a24a0 0x7fe7f40a4960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8009830 tx=0x7fe7f80349d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:29.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:29.852+0000 7fe7fe7fc640 1 -- 172.21.15.70:0/2347501238 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe80409bd00 con 0x7fe808102a20 2024-04-01T01:31:29.900 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:29 smithi189 ceph-mon[35753]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 1 op/s 2024-04-01T01:31:29.900 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:29 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1676972294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:31:29.900 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:29 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/410104752' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:31:29.900 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:29 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2501673535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:31:30.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.041+0000 7fe80f6eb640 1 -- 172.21.15.70:0/2347501238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe80802d3c0 con 0x7fe808102a20 2024-04-01T01:31:30.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.042+0000 7fe7fe7fc640 1 -- 172.21.15.70:0/2347501238 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+590 (secure 0 0 0) 0x7fe80409b600 con 0x7fe808102a20 2024-04-01T01:31:30.042 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/950171900 2024-04-02T01:24:18.477610+0000 2024-04-01T01:31:30.043 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/2902831307 2024-04-02T01:24:18.477610+0000 2024-04-01T01:31:30.043 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6801/2836712828 2024-04-02T01:24:18.477610+0000 2024-04-01T01:31:30.043 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6800/2836712828 2024-04-02T01:24:18.477610+0000 2024-04-01T01:31:30.043 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/602656012 2024-04-02T01:23:52.091015+0000 2024-04-01T01:31:30.043 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6800/2121290979 2024-04-02T01:23:52.091015+0000 2024-04-01T01:31:30.043 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6801/2121290979 2024-04-02T01:23:52.091015+0000 2024-04-01T01:31:30.043 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/3734182136 2024-04-02T01:23:52.091015+0000 2024-04-01T01:31:30.043 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/2899206193 2024-04-02T01:23:52.091015+0000 2024-04-01T01:31:30.043 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/3948868469 2024-04-02T01:24:18.477610+0000 2024-04-01T01:31:30.043 INFO:teuthology.orchestra.run.smithi070.stderr:listed 10 entries 2024-04-01T01:31:30.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.044+0000 7fe80f6eb640 1 -- 172.21.15.70:0/2347501238 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe7f40a24a0 msgr2=0x7fe7f40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:30.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.044+0000 7fe80f6eb640 1 --2- 172.21.15.70:0/2347501238 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe7f40a24a0 0x7fe7f40a4960 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8009830 tx=0x7fe7f80349d0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.044+0000 7fe80f6eb640 1 -- 172.21.15.70:0/2347501238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe808102a20 msgr2=0x7fe808197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:30.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.044+0000 7fe80f6eb640 1 --2- 172.21.15.70:0/2347501238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe808102a20 0x7fe808197f10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe80400e9c0 tx=0x7fe80400ee80 comp rx=0 tx=0).stop 2024-04-01T01:31:30.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.044+0000 7fe80f6eb640 1 -- 172.21.15.70:0/2347501238 shutdown_connections 2024-04-01T01:31:30.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.044+0000 7fe80f6eb640 1 --2- 172.21.15.70:0/2347501238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8081033e0 0x7fe80819ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.044+0000 7fe80f6eb640 1 --2- 172.21.15.70:0/2347501238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe808102a20 0x7fe808197f10 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.044+0000 7fe80f6eb640 1 --2- 172.21.15.70:0/2347501238 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe7f40a24a0 0x7fe7f40a4960 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.044+0000 7fe80f6eb640 1 --2- 172.21.15.70:0/2347501238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8080ffa50 0x7fe8081979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.044+0000 7fe80f6eb640 1 -- 172.21.15.70:0/2347501238 >> 172.21.15.70:0/2347501238 conn(0x7fe8080fb380 msgr2=0x7fe808105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:30.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.044+0000 7fe80f6eb640 1 -- 172.21.15.70:0/2347501238 shutdown_connections 2024-04-01T01:31:30.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.044+0000 7fe80f6eb640 1 -- 172.21.15.70:0/2347501238 wait complete. 2024-04-01T01:31:30.055 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:31:30.055 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-01T01:31:30.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:31:30.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:29 smithi081 ceph-mon[35142]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s wr, 1 op/s 2024-04-01T01:31:30.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:29 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1676972294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:31:30.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:29 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/410104752' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:31:30.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:29 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2501673535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:31:30.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.213+0000 7f12d5725640 1 -- 172.21.15.70:0/1914258844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0100690 msgr2=0x7f12d0100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:30.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.213+0000 7f12d5725640 1 --2- 172.21.15.70:0/1914258844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0100690 0x7f12d0100ab0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f12c0009900 tx=0x7f12c002bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:30.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.214+0000 7f12d5725640 1 -- 172.21.15.70:0/1914258844 shutdown_connections 2024-04-01T01:31:30.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.214+0000 7f12d5725640 1 --2- 172.21.15.70:0/1914258844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d0102240 0x7f12d0108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.214+0000 7f12d5725640 1 --2- 172.21.15.70:0/1914258844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d0101880 0x7f12d0101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.214+0000 7f12d5725640 1 --2- 172.21.15.70:0/1914258844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0100690 0x7f12d0100ab0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.214+0000 7f12d5725640 1 -- 172.21.15.70:0/1914258844 >> 172.21.15.70:0/1914258844 conn(0x7f12d00fbc00 msgr2=0x7f12d00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:30.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.214+0000 7f12d5725640 1 -- 172.21.15.70:0/1914258844 shutdown_connections 2024-04-01T01:31:30.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.214+0000 7f12d5725640 1 -- 172.21.15.70:0/1914258844 wait complete. 2024-04-01T01:31:30.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.214+0000 7f12d5725640 1 Processor -- start 2024-04-01T01:31:30.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.214+0000 7f12d5725640 1 -- start start 2024-04-01T01:31:30.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.214+0000 7f12d5725640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d0100690 0x7f12d0196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:30.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.214+0000 7f12d5725640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0101880 0x7f12d0196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:30.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.214+0000 7f12d5725640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d0102240 0x7f12d019df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:30.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.214+0000 7f12d5725640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f12d010c190 con 0x7f12d0101880 2024-04-01T01:31:30.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.214+0000 7f12d5725640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f12d010c010 con 0x7f12d0102240 2024-04-01T01:31:30.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.214+0000 7f12d5725640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f12d010c310 con 0x7f12d0100690 2024-04-01T01:31:30.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.215+0000 7f12c67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0101880 0x7f12d0196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:30.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.215+0000 7f12c67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0101880 0x7f12d0196e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40878/0 (socket says 172.21.15.70:40878) 2024-04-01T01:31:30.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.215+0000 7f12c67fc640 1 -- 172.21.15.70:0/110981782 learned_addr learned my addr 172.21.15.70:0/110981782 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:30.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.215+0000 7f12ceffd640 1 --2- 172.21.15.70:0/110981782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d0100690 0x7f12d0196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:30.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.215+0000 7f12cf7fe640 1 --2- 172.21.15.70:0/110981782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d0102240 0x7f12d019df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:30.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.215+0000 7f12c67fc640 1 -- 172.21.15.70:0/110981782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d0100690 msgr2=0x7f12d0196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:30.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.215+0000 7f12c67fc640 1 --2- 172.21.15.70:0/110981782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d0100690 0x7f12d0196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.215+0000 7f12c67fc640 1 -- 172.21.15.70:0/110981782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d0102240 msgr2=0x7f12d019df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:30.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.215+0000 7f12c67fc640 1 --2- 172.21.15.70:0/110981782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d0102240 0x7f12d019df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.215+0000 7f12c67fc640 1 -- 172.21.15.70:0/110981782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d019e570 con 0x7f12d0101880 2024-04-01T01:31:30.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.215+0000 7f12c67fc640 1 --2- 172.21.15.70:0/110981782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0101880 0x7f12d0196e90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f12b800e9f0 tx=0x7f12b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:30.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.215+0000 7f12ccff9640 1 -- 172.21.15.70:0/110981782 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b800cd80 con 0x7f12d0101880 2024-04-01T01:31:30.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.215+0000 7f12ccff9640 1 -- 172.21.15.70:0/110981782 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12b80108e0 con 0x7f12d0101880 2024-04-01T01:31:30.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.215+0000 7f12ccff9640 1 -- 172.21.15.70:0/110981782 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b8018920 con 0x7f12d0101880 2024-04-01T01:31:30.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.215+0000 7f12d5725640 1 -- 172.21.15.70:0/110981782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12d019e860 con 0x7f12d0101880 2024-04-01T01:31:30.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.215+0000 7f12d5725640 1 -- 172.21.15.70:0/110981782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12d019ee00 con 0x7f12d0101880 2024-04-01T01:31:30.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.216+0000 7f12d5725640 1 -- 172.21.15.70:0/110981782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12d0107b10 con 0x7f12d0101880 2024-04-01T01:31:30.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.218+0000 7f12ccff9640 1 -- 172.21.15.70:0/110981782 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f12b8010a90 con 0x7f12d0101880 2024-04-01T01:31:30.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.218+0000 7f12ccff9640 1 --2- 172.21.15.70:0/110981782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12bc0a2520 0x7f12bc0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:30.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.219+0000 7f12ceffd640 1 --2- 172.21.15.70:0/110981782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12bc0a2520 0x7f12bc0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:30.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.219+0000 7f12ccff9640 1 -- 172.21.15.70:0/110981782 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f12b80cf080 con 0x7f12d0101880 2024-04-01T01:31:30.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.219+0000 7f12ceffd640 1 --2- 172.21.15.70:0/110981782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12bc0a2520 0x7f12bc0a49e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f12c0005ab0 tx=0x7f12c0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:30.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.222+0000 7f12ccff9640 1 -- 172.21.15.70:0/110981782 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f12b809b9b0 con 0x7f12d0101880 2024-04-01T01:31:30.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.397+0000 7f12d5725640 1 -- 172.21.15.70:0/110981782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f12d002d3c0 con 0x7f12d0101880 2024-04-01T01:31:30.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.399+0000 7f12ccff9640 1 -- 172.21.15.70:0/110981782 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24068 (secure 0 0 0) 0x7f12b809fa50 con 0x7f12d0101880 2024-04-01T01:31:30.399 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:30.400 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":89,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:31:20.903623+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:31:30.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.401+0000 7f12d5725640 1 -- 172.21.15.70:0/110981782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12bc0a2520 msgr2=0x7f12bc0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:30.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.401+0000 7f12d5725640 1 --2- 172.21.15.70:0/110981782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12bc0a2520 0x7f12bc0a49e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f12c0005ab0 tx=0x7f12c0005fd0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.401+0000 7f12d5725640 1 -- 172.21.15.70:0/110981782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0101880 msgr2=0x7f12d0196e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:30.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.401+0000 7f12d5725640 1 --2- 172.21.15.70:0/110981782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0101880 0x7f12d0196e90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f12b800e9f0 tx=0x7f12b800eeb0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.401+0000 7f12d5725640 1 -- 172.21.15.70:0/110981782 shutdown_connections 2024-04-01T01:31:30.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.401+0000 7f12d5725640 1 --2- 172.21.15.70:0/110981782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d0102240 0x7f12d019df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.401+0000 7f12d5725640 1 --2- 172.21.15.70:0/110981782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0101880 0x7f12d0196e90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.401+0000 7f12d5725640 1 --2- 172.21.15.70:0/110981782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12bc0a2520 0x7f12bc0a49e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.401+0000 7f12d5725640 1 --2- 172.21.15.70:0/110981782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d0100690 0x7f12d0196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.401+0000 7f12d5725640 1 -- 172.21.15.70:0/110981782 >> 172.21.15.70:0/110981782 conn(0x7f12d00fbc00 msgr2=0x7f12d0104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:30.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.401+0000 7f12d5725640 1 -- 172.21.15.70:0/110981782 shutdown_connections 2024-04-01T01:31:30.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.401+0000 7f12d5725640 1 -- 172.21.15.70:0/110981782 wait complete. 2024-04-01T01:31:30.413 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-01T01:26:42.809779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-01T01:31:13.612369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-01T01:31:14.910162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-01T01:31:18.783179+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-01T01:31:30.413 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-01T01:31:30.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.518+0000 7f9b022c9640 1 -- 172.21.15.70:0/1857682985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9afc1012d0 msgr2=0x7f9afc108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:30.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.518+0000 7f9b022c9640 1 --2- 172.21.15.70:0/1857682985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9afc1012d0 0x7f9afc108b70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9af4009cd0 tx=0x7f9af402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.518+0000 7f9b022c9640 1 -- 172.21.15.70:0/1857682985 shutdown_connections 2024-04-01T01:31:30.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.518+0000 7f9b022c9640 1 --2- 172.21.15.70:0/1857682985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9afc1012d0 0x7f9afc108b70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.518+0000 7f9b022c9640 1 --2- 172.21.15.70:0/1857682985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9afc100910 0x7f9afc100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.518+0000 7f9b022c9640 1 --2- 172.21.15.70:0/1857682985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9afc101c70 0x7f9afc102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.518+0000 7f9b022c9640 1 -- 172.21.15.70:0/1857682985 >> 172.21.15.70:0/1857682985 conn(0x7f9afc0fb360 msgr2=0x7f9afc0fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:30.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.518+0000 7f9b022c9640 1 -- 172.21.15.70:0/1857682985 shutdown_connections 2024-04-01T01:31:30.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.518+0000 7f9b022c9640 1 -- 172.21.15.70:0/1857682985 wait complete. 2024-04-01T01:31:30.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.518+0000 7f9b022c9640 1 Processor -- start 2024-04-01T01:31:30.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.518+0000 7f9b022c9640 1 -- start start 2024-04-01T01:31:30.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9b022c9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9afc100910 0x7f9afc100390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:30.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9b022c9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9afc1012d0 0x7f9afc19c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:30.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9b022c9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9afc101c70 0x7f9afc19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:30.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9b022c9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9afc10c190 con 0x7f9afc1012d0 2024-04-01T01:31:30.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9b022c9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9afc10c010 con 0x7f9afc100910 2024-04-01T01:31:30.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9b022c9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9afc10c310 con 0x7f9afc101c70 2024-04-01T01:31:30.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9afb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9afc1012d0 0x7f9afc19c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:30.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9afb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9afc1012d0 0x7f9afc19c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40900/0 (socket says 172.21.15.70:40900) 2024-04-01T01:31:30.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9afbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9afc100910 0x7f9afc100390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:30.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9afb7fe640 1 -- 172.21.15.70:0/2374037878 learned_addr learned my addr 172.21.15.70:0/2374037878 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:30.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9b00866640 1 --2- 172.21.15.70:0/2374037878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9afc101c70 0x7f9afc19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:30.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9afb7fe640 1 -- 172.21.15.70:0/2374037878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9afc101c70 msgr2=0x7f9afc19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:30.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9afb7fe640 1 --2- 172.21.15.70:0/2374037878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9afc101c70 0x7f9afc19ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9afb7fe640 1 -- 172.21.15.70:0/2374037878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9afc100910 msgr2=0x7f9afc100390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:30.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9afb7fe640 1 --2- 172.21.15.70:0/2374037878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9afc100910 0x7f9afc100390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9afb7fe640 1 -- 172.21.15.70:0/2374037878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9afc19f500 con 0x7f9afc1012d0 2024-04-01T01:31:30.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9afbfff640 1 --2- 172.21.15.70:0/2374037878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9afc100910 0x7f9afc100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:30.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.519+0000 7f9afb7fe640 1 --2- 172.21.15.70:0/2374037878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9afc1012d0 0x7f9afc19c600 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9af000e9f0 tx=0x7f9af000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:30.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.520+0000 7f9af97fa640 1 -- 172.21.15.70:0/2374037878 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af000cd80 con 0x7f9afc1012d0 2024-04-01T01:31:30.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.520+0000 7f9b022c9640 1 -- 172.21.15.70:0/2374037878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9afc19f7f0 con 0x7f9afc1012d0 2024-04-01T01:31:30.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.520+0000 7f9af97fa640 1 -- 172.21.15.70:0/2374037878 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9af00108e0 con 0x7f9afc1012d0 2024-04-01T01:31:30.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.520+0000 7f9af97fa640 1 -- 172.21.15.70:0/2374037878 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af0018920 con 0x7f9afc1012d0 2024-04-01T01:31:30.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.520+0000 7f9b022c9640 1 -- 172.21.15.70:0/2374037878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9afc19fd90 con 0x7f9afc1012d0 2024-04-01T01:31:30.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.521+0000 7f9b022c9640 1 -- 172.21.15.70:0/2374037878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9afc1079f0 con 0x7f9afc1012d0 2024-04-01T01:31:30.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.522+0000 7f9af97fa640 1 -- 172.21.15.70:0/2374037878 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f9af0010a90 con 0x7f9afc1012d0 2024-04-01T01:31:30.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.522+0000 7f9af97fa640 1 --2- 172.21.15.70:0/2374037878 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9ae40a24a0 0x7f9ae40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:30.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.522+0000 7f9afbfff640 1 --2- 172.21.15.70:0/2374037878 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9ae40a24a0 0x7f9ae40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:30.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.522+0000 7f9af97fa640 1 -- 172.21.15.70:0/2374037878 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9af00cec50 con 0x7f9afc1012d0 2024-04-01T01:31:30.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.523+0000 7f9afbfff640 1 --2- 172.21.15.70:0/2374037878 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9ae40a24a0 0x7f9ae40a4960 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9afc069ba0 tx=0x7f9aec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:30.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.527+0000 7f9af97fa640 1 -- 172.21.15.70:0/2374037878 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9af00d4020 con 0x7f9afc1012d0 2024-04-01T01:31:30.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.704+0000 7f9b022c9640 1 -- 172.21.15.70:0/2374037878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9afc062350 con 0x7f9afc1012d0 2024-04-01T01:31:30.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.704+0000 7f9af97fa640 1 -- 172.21.15.70:0/2374037878 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f9af009b580 con 0x7f9afc1012d0 2024-04-01T01:31:30.705 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-01T01:31:30.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.706+0000 7f9b022c9640 1 -- 172.21.15.70:0/2374037878 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9ae40a24a0 msgr2=0x7f9ae40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:30.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.706+0000 7f9b022c9640 1 --2- 172.21.15.70:0/2374037878 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9ae40a24a0 0x7f9ae40a4960 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9afc069ba0 tx=0x7f9aec005f90 comp rx=0 tx=0).stop 2024-04-01T01:31:30.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.706+0000 7f9b022c9640 1 -- 172.21.15.70:0/2374037878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9afc1012d0 msgr2=0x7f9afc19c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:30.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.706+0000 7f9b022c9640 1 --2- 172.21.15.70:0/2374037878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9afc1012d0 0x7f9afc19c600 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9af000e9f0 tx=0x7f9af000eeb0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.706+0000 7f9b022c9640 1 -- 172.21.15.70:0/2374037878 shutdown_connections 2024-04-01T01:31:30.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.706+0000 7f9b022c9640 1 --2- 172.21.15.70:0/2374037878 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9ae40a24a0 0x7f9ae40a4960 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.706+0000 7f9b022c9640 1 --2- 172.21.15.70:0/2374037878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9afc101c70 0x7f9afc19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.706+0000 7f9b022c9640 1 --2- 172.21.15.70:0/2374037878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9afc1012d0 0x7f9afc19c600 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.706+0000 7f9b022c9640 1 --2- 172.21.15.70:0/2374037878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9afc100910 0x7f9afc100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.706+0000 7f9b022c9640 1 -- 172.21.15.70:0/2374037878 >> 172.21.15.70:0/2374037878 conn(0x7f9afc0fb360 msgr2=0x7f9afc104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:30.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.706+0000 7f9b022c9640 1 -- 172.21.15.70:0/2374037878 shutdown_connections 2024-04-01T01:31:30.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.706+0000 7f9b022c9640 1 -- 172.21.15.70:0/2374037878 wait complete. 2024-04-01T01:31:30.716 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:30 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3208781804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:31:30.717 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:30 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2347501238' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-01T01:31:30.717 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:30 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/110981782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:30.718 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-01T01:31:30.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.822+0000 7fd604fc9640 1 -- 172.21.15.70:0/587357804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd600102220 msgr2=0x7fd600108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:30.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.822+0000 7fd604fc9640 1 --2- 172.21.15.70:0/587357804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd600102220 0x7fd600108ab0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8009ca0 tx=0x7fd5f802c0f0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.822+0000 7fd604fc9640 1 -- 172.21.15.70:0/587357804 shutdown_connections 2024-04-01T01:31:30.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.822+0000 7fd604fc9640 1 --2- 172.21.15.70:0/587357804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd600102220 0x7fd600108ab0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.822+0000 7fd604fc9640 1 --2- 172.21.15.70:0/587357804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd600101860 0x7fd600101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.822+0000 7fd604fc9640 1 --2- 172.21.15.70:0/587357804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd600100670 0x7fd600100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.822+0000 7fd604fc9640 1 -- 172.21.15.70:0/587357804 >> 172.21.15.70:0/587357804 conn(0x7fd6000fbc00 msgr2=0x7fd6000fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:30.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.822+0000 7fd604fc9640 1 -- 172.21.15.70:0/587357804 shutdown_connections 2024-04-01T01:31:30.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.822+0000 7fd604fc9640 1 -- 172.21.15.70:0/587357804 wait complete. 2024-04-01T01:31:30.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.823+0000 7fd604fc9640 1 Processor -- start 2024-04-01T01:31:30.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.823+0000 7fd604fc9640 1 -- start start 2024-04-01T01:31:30.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.823+0000 7fd604fc9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd600100670 0x7fd6001045b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:30.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.823+0000 7fd604fc9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd600101860 0x7fd600102c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.823+0000 7fd604fc9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd600102220 0x7fd600103180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.823+0000 7fd604fc9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd60010c1a0 con 0x7fd600101860 2024-04-01T01:31:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.823+0000 7fd604fc9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd60010c020 con 0x7fd600102220 2024-04-01T01:31:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.823+0000 7fd604fc9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd60010c320 con 0x7fd600100670 2024-04-01T01:31:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.823+0000 7fd5f5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd600101860 0x7fd600102c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.823+0000 7fd5f5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd600101860 0x7fd600102c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40906/0 (socket says 172.21.15.70:40906) 2024-04-01T01:31:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.823+0000 7fd5f5d9b640 1 -- 172.21.15.70:0/323877124 learned_addr learned my addr 172.21.15.70:0/323877124 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.823+0000 7fd5fed9d640 1 --2- 172.21.15.70:0/323877124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd600102220 0x7fd600103180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.823+0000 7fd5fe59c640 1 --2- 172.21.15.70:0/323877124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd600100670 0x7fd6001045b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.824+0000 7fd5f5d9b640 1 -- 172.21.15.70:0/323877124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd600100670 msgr2=0x7fd6001045b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.824+0000 7fd5f5d9b640 1 --2- 172.21.15.70:0/323877124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd600100670 0x7fd6001045b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.824+0000 7fd5f5d9b640 1 -- 172.21.15.70:0/323877124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd600102220 msgr2=0x7fd600103180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.824+0000 7fd5f5d9b640 1 --2- 172.21.15.70:0/323877124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd600102220 0x7fd600103180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:30.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.824+0000 7fd5f5d9b640 1 -- 172.21.15.70:0/323877124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd600103a40 con 0x7fd600101860 2024-04-01T01:31:30.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.824+0000 7fd5fed9d640 1 --2- 172.21.15.70:0/323877124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd600102220 0x7fd600103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:30.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.824+0000 7fd5f5d9b640 1 --2- 172.21.15.70:0/323877124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd600101860 0x7fd600102c40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd5e800ea60 tx=0x7fd5e800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:30.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.824+0000 7fd5f7fff640 1 -- 172.21.15.70:0/323877124 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e800cde0 con 0x7fd600101860 2024-04-01T01:31:30.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.824+0000 7fd5f7fff640 1 -- 172.21.15.70:0/323877124 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5e80035f0 con 0x7fd600101860 2024-04-01T01:31:30.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.824+0000 7fd604fc9640 1 -- 172.21.15.70:0/323877124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6001a1ac0 con 0x7fd600101860 2024-04-01T01:31:30.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.824+0000 7fd5f7fff640 1 -- 172.21.15.70:0/323877124 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8010690 con 0x7fd600101860 2024-04-01T01:31:30.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.824+0000 7fd604fc9640 1 -- 172.21.15.70:0/323877124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6001a1fe0 con 0x7fd600101860 2024-04-01T01:31:30.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.825+0000 7fd604fc9640 1 -- 172.21.15.70:0/323877124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6001a2270 con 0x7fd600101860 2024-04-01T01:31:30.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.826+0000 7fd5f7fff640 1 -- 172.21.15.70:0/323877124 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fd5e80037a0 con 0x7fd600101860 2024-04-01T01:31:30.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.826+0000 7fd5f7fff640 1 --2- 172.21.15.70:0/323877124 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5ec0a24a0 0x7fd5ec0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:30.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.827+0000 7fd5fe59c640 1 --2- 172.21.15.70:0/323877124 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5ec0a24a0 0x7fd5ec0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:30.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.827+0000 7fd5f7fff640 1 -- 172.21.15.70:0/323877124 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd5e80ceb90 con 0x7fd600101860 2024-04-01T01:31:30.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.827+0000 7fd5fe59c640 1 --2- 172.21.15.70:0/323877124 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5ec0a24a0 0x7fd5ec0a4960 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd6001016c0 tx=0x7fd5f0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:30.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:30.831+0000 7fd5f7fff640 1 -- 172.21.15.70:0/323877124 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd5e809bb10 con 0x7fd600101860 2024-04-01T01:31:30.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:30 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3208781804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:31:30.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:30 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2347501238' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-01T01:31:30.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:30 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/110981782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:31.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.005+0000 7fd604fc9640 1 -- 172.21.15.70:0/323877124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd6001a2440 con 0x7fd600101860 2024-04-01T01:31:31.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.005+0000 7fd5f7fff640 1 -- 172.21.15.70:0/323877124 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fd5e809b410 con 0x7fd600101860 2024-04-01T01:31:31.005 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:31:31.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.007+0000 7fd604fc9640 1 -- 172.21.15.70:0/323877124 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5ec0a24a0 msgr2=0x7fd5ec0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.007+0000 7fd604fc9640 1 --2- 172.21.15.70:0/323877124 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5ec0a24a0 0x7fd5ec0a4960 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd6001016c0 tx=0x7fd5f0005f90 comp rx=0 tx=0).stop 2024-04-01T01:31:31.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.007+0000 7fd604fc9640 1 -- 172.21.15.70:0/323877124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd600101860 msgr2=0x7fd600102c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.007+0000 7fd604fc9640 1 --2- 172.21.15.70:0/323877124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd600101860 0x7fd600102c40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd5e800ea60 tx=0x7fd5e800ef20 comp rx=0 tx=0).stop 2024-04-01T01:31:31.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.007+0000 7fd604fc9640 1 -- 172.21.15.70:0/323877124 shutdown_connections 2024-04-01T01:31:31.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.007+0000 7fd604fc9640 1 --2- 172.21.15.70:0/323877124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd600102220 0x7fd600103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.007+0000 7fd604fc9640 1 --2- 172.21.15.70:0/323877124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd600101860 0x7fd600102c40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.007+0000 7fd604fc9640 1 --2- 172.21.15.70:0/323877124 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5ec0a24a0 0x7fd5ec0a4960 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.007+0000 7fd604fc9640 1 --2- 172.21.15.70:0/323877124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd600100670 0x7fd6001045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.007+0000 7fd604fc9640 1 -- 172.21.15.70:0/323877124 >> 172.21.15.70:0/323877124 conn(0x7fd6000fbc00 msgr2=0x7fd6000fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:31.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.007+0000 7fd604fc9640 1 -- 172.21.15.70:0/323877124 shutdown_connections 2024-04-01T01:31:31.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.007+0000 7fd604fc9640 1 -- 172.21.15.70:0/323877124 wait complete. 2024-04-01T01:31:31.018 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-01T01:31:31.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:30 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3208781804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:31:31.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:30 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2347501238' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-01T01:31:31.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:30 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/110981782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:31.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 -- 172.21.15.70:0/81974566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f929c100660 msgr2=0x7f929c100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 --2- 172.21.15.70:0/81974566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f929c100660 0x7f929c100a80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9294009ca0 tx=0x7f929402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 -- 172.21.15.70:0/81974566 shutdown_connections 2024-04-01T01:31:31.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 --2- 172.21.15.70:0/81974566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f929c102210 0x7f929c108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 --2- 172.21.15.70:0/81974566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f929c101850 0x7f929c101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 --2- 172.21.15.70:0/81974566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f929c100660 0x7f929c100a80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 -- 172.21.15.70:0/81974566 >> 172.21.15.70:0/81974566 conn(0x7f929c0fbbf0 msgr2=0x7f929c0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:31.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 -- 172.21.15.70:0/81974566 shutdown_connections 2024-04-01T01:31:31.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 -- 172.21.15.70:0/81974566 wait complete. 2024-04-01T01:31:31.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 Processor -- start 2024-04-01T01:31:31.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 -- start start 2024-04-01T01:31:31.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f929c100660 0x7f929c196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:31.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f929c101850 0x7f929c196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:31.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f929c102210 0x7f929c19de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:31.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f929c10c190 con 0x7f929c102210 2024-04-01T01:31:31.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f929c10c010 con 0x7f929c101850 2024-04-01T01:31:31.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.122+0000 7f92a362e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f929c10c310 con 0x7f929c100660 2024-04-01T01:31:31.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a1bcb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f929c102210 0x7f929c19de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:31.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a0bc9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f929c101850 0x7f929c196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:31.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a13ca640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f929c100660 0x7f929c196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:31.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a0bc9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f929c101850 0x7f929c196dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:51288/0 (socket says 172.21.15.70:51288) 2024-04-01T01:31:31.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a13ca640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f929c100660 0x7f929c196890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:35024/0 (socket says 172.21.15.70:35024) 2024-04-01T01:31:31.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a1bcb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f929c102210 0x7f929c19de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40928/0 (socket says 172.21.15.70:40928) 2024-04-01T01:31:31.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a0bc9640 1 -- 172.21.15.70:0/3861550977 learned_addr learned my addr 172.21.15.70:0/3861550977 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:31.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a0bc9640 1 -- 172.21.15.70:0/3861550977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f929c100660 msgr2=0x7f929c196890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a0bc9640 1 --2- 172.21.15.70:0/3861550977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f929c100660 0x7f929c196890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a0bc9640 1 -- 172.21.15.70:0/3861550977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f929c102210 msgr2=0x7f929c19de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a0bc9640 1 --2- 172.21.15.70:0/3861550977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f929c102210 0x7f929c19de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a0bc9640 1 -- 172.21.15.70:0/3861550977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929c19e4b0 con 0x7f929c101850 2024-04-01T01:31:31.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a13ca640 1 --2- 172.21.15.70:0/3861550977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f929c100660 0x7f929c196890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:31.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a1bcb640 1 --2- 172.21.15.70:0/3861550977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f929c102210 0x7f929c19de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:31.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a0bc9640 1 --2- 172.21.15.70:0/3861550977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f929c101850 0x7f929c196dd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f928c00e9f0 tx=0x7f928c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:31.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92927fc640 1 -- 172.21.15.70:0/3861550977 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c00cd80 con 0x7f929c101850 2024-04-01T01:31:31.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a362e640 1 -- 172.21.15.70:0/3861550977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929c19e7a0 con 0x7f929c101850 2024-04-01T01:31:31.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92927fc640 1 -- 172.21.15.70:0/3861550977 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f928c0108e0 con 0x7f929c101850 2024-04-01T01:31:31.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92927fc640 1 -- 172.21.15.70:0/3861550977 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c0188c0 con 0x7f929c101850 2024-04-01T01:31:31.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.123+0000 7f92a362e640 1 -- 172.21.15.70:0/3861550977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f929c19ed40 con 0x7f929c101850 2024-04-01T01:31:31.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.124+0000 7f92a362e640 1 -- 172.21.15.70:0/3861550977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f929c107a50 con 0x7f929c101850 2024-04-01T01:31:31.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.125+0000 7f92927fc640 1 -- 172.21.15.70:0/3861550977 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f928c010a90 con 0x7f929c101850 2024-04-01T01:31:31.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.126+0000 7f92927fc640 1 --2- 172.21.15.70:0/3861550977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f92880a24a0 0x7f92880a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:31.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.126+0000 7f92a13ca640 1 --2- 172.21.15.70:0/3861550977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f92880a24a0 0x7f92880a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:31.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.126+0000 7f92927fc640 1 -- 172.21.15.70:0/3861550977 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f928c0cee10 con 0x7f929c101850 2024-04-01T01:31:31.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.126+0000 7f92a13ca640 1 --2- 172.21.15.70:0/3861550977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f92880a24a0 0x7f92880a4960 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9294032380 tx=0x7f9294000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:31.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.134+0000 7f92927fc640 1 -- 172.21.15.70:0/3861550977 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f928c09b740 con 0x7f929c101850 2024-04-01T01:31:31.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.307+0000 7f92a362e640 1 -- 172.21.15.70:0/3861550977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f929c02d3c0 con 0x7f929c101850 2024-04-01T01:31:31.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.308+0000 7f92927fc640 1 -- 172.21.15.70:0/3861550977 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f928c09f7e0 con 0x7f929c101850 2024-04-01T01:31:31.308 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-01T01:31:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.310+0000 7f92a362e640 1 -- 172.21.15.70:0/3861550977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f92880a24a0 msgr2=0x7f92880a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.310+0000 7f92a362e640 1 --2- 172.21.15.70:0/3861550977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f92880a24a0 0x7f92880a4960 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9294032380 tx=0x7f9294000f40 comp rx=0 tx=0).stop 2024-04-01T01:31:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.310+0000 7f92a362e640 1 -- 172.21.15.70:0/3861550977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f929c101850 msgr2=0x7f929c196dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.310+0000 7f92a362e640 1 --2- 172.21.15.70:0/3861550977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f929c101850 0x7f929c196dd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f928c00e9f0 tx=0x7f928c00eeb0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.310+0000 7f92a362e640 1 -- 172.21.15.70:0/3861550977 shutdown_connections 2024-04-01T01:31:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.310+0000 7f92a362e640 1 --2- 172.21.15.70:0/3861550977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f929c102210 0x7f929c19de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.310+0000 7f92a362e640 1 --2- 172.21.15.70:0/3861550977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f929c101850 0x7f929c196dd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.310+0000 7f92a362e640 1 --2- 172.21.15.70:0/3861550977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f92880a24a0 0x7f92880a4960 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.310+0000 7f92a362e640 1 --2- 172.21.15.70:0/3861550977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f929c100660 0x7f929c196890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.310+0000 7f92a362e640 1 -- 172.21.15.70:0/3861550977 >> 172.21.15.70:0/3861550977 conn(0x7f929c0fbbf0 msgr2=0x7f929c104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:31.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.310+0000 7f92a362e640 1 -- 172.21.15.70:0/3861550977 shutdown_connections 2024-04-01T01:31:31.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.310+0000 7f92a362e640 1 -- 172.21.15.70:0/3861550977 wait complete. 2024-04-01T01:31:31.321 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-01T01:31:31.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.427+0000 7f84c0f27640 1 -- 172.21.15.70:0/1443976201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84bc101840 msgr2=0x7f84bc101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.427+0000 7f84c0f27640 1 --2- 172.21.15.70:0/1443976201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84bc101840 0x7f84bc101cc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009900 tx=0x7f84b002bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:31.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.428+0000 7f84c0f27640 1 -- 172.21.15.70:0/1443976201 shutdown_connections 2024-04-01T01:31:31.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.428+0000 7f84c0f27640 1 --2- 172.21.15.70:0/1443976201 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84bc102200 0x7f84bc108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.428+0000 7f84c0f27640 1 --2- 172.21.15.70:0/1443976201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84bc101840 0x7f84bc101cc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.428+0000 7f84c0f27640 1 --2- 172.21.15.70:0/1443976201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84bc100650 0x7f84bc100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.428+0000 7f84c0f27640 1 -- 172.21.15.70:0/1443976201 >> 172.21.15.70:0/1443976201 conn(0x7f84bc0fbbe0 msgr2=0x7f84bc0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:31.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.428+0000 7f84c0f27640 1 -- 172.21.15.70:0/1443976201 shutdown_connections 2024-04-01T01:31:31.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.428+0000 7f84c0f27640 1 -- 172.21.15.70:0/1443976201 wait complete. 2024-04-01T01:31:31.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.428+0000 7f84c0f27640 1 Processor -- start 2024-04-01T01:31:31.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.428+0000 7f84c0f27640 1 -- start start 2024-04-01T01:31:31.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.428+0000 7f84c0f27640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84bc100650 0x7f84bc197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:31.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.428+0000 7f84c0f27640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84bc101840 0x7f84bc197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:31.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.428+0000 7f84c0f27640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84bc102200 0x7f84bc19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:31.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.428+0000 7f84c0f27640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc10c190 con 0x7f84bc100650 2024-04-01T01:31:31.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.428+0000 7f84c0f27640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc10c010 con 0x7f84bc101840 2024-04-01T01:31:31.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.428+0000 7f84c0f27640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f84bc10c310 con 0x7f84bc102200 2024-04-01T01:31:31.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84ba59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84bc100650 0x7f84bc197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:31.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84b9d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84bc101840 0x7f84bc197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:31.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84ba59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84bc100650 0x7f84bc197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40948/0 (socket says 172.21.15.70:40948) 2024-04-01T01:31:31.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84ba59c640 1 -- 172.21.15.70:0/91293488 learned_addr learned my addr 172.21.15.70:0/91293488 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:31.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84bad9d640 1 --2- 172.21.15.70:0/91293488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84bc102200 0x7f84bc19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:31.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84ba59c640 1 -- 172.21.15.70:0/91293488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84bc102200 msgr2=0x7f84bc19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84ba59c640 1 --2- 172.21.15.70:0/91293488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84bc102200 0x7f84bc19efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84ba59c640 1 -- 172.21.15.70:0/91293488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84bc101840 msgr2=0x7f84bc197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84ba59c640 1 --2- 172.21.15.70:0/91293488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84bc101840 0x7f84bc197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84ba59c640 1 -- 172.21.15.70:0/91293488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84bc19f630 con 0x7f84bc100650 2024-04-01T01:31:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84b9d9b640 1 --2- 172.21.15.70:0/91293488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84bc101840 0x7f84bc197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84ba59c640 1 --2- 172.21.15.70:0/91293488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84bc100650 0x7f84bc197a10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00eb70 tx=0x7f84ac00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84ab7fe640 1 -- 172.21.15.70:0/91293488 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ac003b70 con 0x7f84bc100650 2024-04-01T01:31:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84c0f27640 1 -- 172.21.15.70:0/91293488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84bc19f920 con 0x7f84bc100650 2024-04-01T01:31:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84ab7fe640 1 -- 172.21.15.70:0/91293488 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f84ac00ee70 con 0x7f84bc100650 2024-04-01T01:31:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84ab7fe640 1 -- 172.21.15.70:0/91293488 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ac010650 con 0x7f84bc100650 2024-04-01T01:31:31.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.429+0000 7f84c0f27640 1 -- 172.21.15.70:0/91293488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84bc19fdf0 con 0x7f84bc100650 2024-04-01T01:31:31.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.430+0000 7f84c0f27640 1 -- 172.21.15.70:0/91293488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84bc1a0050 con 0x7f84bc100650 2024-04-01T01:31:31.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.431+0000 7f84ab7fe640 1 -- 172.21.15.70:0/91293488 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f84ac004020 con 0x7f84bc100650 2024-04-01T01:31:31.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.432+0000 7f84ab7fe640 1 --2- 172.21.15.70:0/91293488 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f84a40a24a0 0x7f84a40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:31.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.432+0000 7f84b9d9b640 1 --2- 172.21.15.70:0/91293488 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f84a40a24a0 0x7f84a40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:31.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.432+0000 7f84ab7fe640 1 -- 172.21.15.70:0/91293488 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f84ac0cf060 con 0x7f84bc100650 2024-04-01T01:31:31.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.432+0000 7f84b9d9b640 1 --2- 172.21.15.70:0/91293488 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f84a40a24a0 0x7f84a40a4960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009230 tx=0x7f84b0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:31.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.436+0000 7f84ab7fe640 1 -- 172.21.15.70:0/91293488 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f84ac09bfe0 con 0x7f84bc100650 2024-04-01T01:31:31.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.614+0000 7f84c0f27640 1 -- 172.21.15.70:0/91293488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f84bc1a0220 con 0x7f84bc100650 2024-04-01T01:31:31.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.615+0000 7f84ab7fe640 1 -- 172.21.15.70:0/91293488 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f84ac09b8e0 con 0x7f84bc100650 2024-04-01T01:31:31.615 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:31:31.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.617+0000 7f84c0f27640 1 -- 172.21.15.70:0/91293488 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f84a40a24a0 msgr2=0x7f84a40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.617+0000 7f84c0f27640 1 --2- 172.21.15.70:0/91293488 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f84a40a24a0 0x7f84a40a4960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009230 tx=0x7f84b0005fd0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.617+0000 7f84c0f27640 1 -- 172.21.15.70:0/91293488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84bc100650 msgr2=0x7f84bc197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.617+0000 7f84c0f27640 1 --2- 172.21.15.70:0/91293488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84bc100650 0x7f84bc197a10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f84ac00eb70 tx=0x7f84ac00c6a0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.617+0000 7f84c0f27640 1 -- 172.21.15.70:0/91293488 shutdown_connections 2024-04-01T01:31:31.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.617+0000 7f84c0f27640 1 --2- 172.21.15.70:0/91293488 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f84a40a24a0 0x7f84a40a4960 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.617+0000 7f84c0f27640 1 --2- 172.21.15.70:0/91293488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84bc102200 0x7f84bc19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.617+0000 7f84c0f27640 1 --2- 172.21.15.70:0/91293488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84bc101840 0x7f84bc197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.617+0000 7f84c0f27640 1 --2- 172.21.15.70:0/91293488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f84bc100650 0x7f84bc197a10 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.617+0000 7f84c0f27640 1 -- 172.21.15.70:0/91293488 >> 172.21.15.70:0/91293488 conn(0x7f84bc0fbbe0 msgr2=0x7f84bc0688e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:31.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.617+0000 7f84c0f27640 1 -- 172.21.15.70:0/91293488 shutdown_connections 2024-04-01T01:31:31.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.617+0000 7f84c0f27640 1 -- 172.21.15.70:0/91293488 wait complete. 2024-04-01T01:31:31.628 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-01T01:31:31.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.732+0000 7f6e016fb640 1 -- 172.21.15.70:0/1693030037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dfc104b20 msgr2=0x7f6dfc106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.732+0000 7f6e016fb640 1 --2- 172.21.15.70:0/1693030037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dfc104b20 0x7f6dfc106f10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6df0009900 tx=0x7f6df002bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:31.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.732+0000 7f6e016fb640 1 -- 172.21.15.70:0/1693030037 shutdown_connections 2024-04-01T01:31:31.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.732+0000 7f6e016fb640 1 --2- 172.21.15.70:0/1693030037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dfc107450 0x7f6dfc1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.732+0000 7f6e016fb640 1 --2- 172.21.15.70:0/1693030037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dfc104b20 0x7f6dfc106f10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.732+0000 7f6e016fb640 1 --2- 172.21.15.70:0/1693030037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dfc1021f0 0x7f6dfc1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.732+0000 7f6e016fb640 1 -- 172.21.15.70:0/1693030037 >> 172.21.15.70:0/1693030037 conn(0x7f6dfc0fbc00 msgr2=0x7f6dfc0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:31.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.732+0000 7f6e016fb640 1 -- 172.21.15.70:0/1693030037 shutdown_connections 2024-04-01T01:31:31.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.732+0000 7f6e016fb640 1 -- 172.21.15.70:0/1693030037 wait complete. 2024-04-01T01:31:31.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.732+0000 7f6e016fb640 1 Processor -- start 2024-04-01T01:31:31.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.732+0000 7f6e016fb640 1 -- start start 2024-04-01T01:31:31.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6e016fb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dfc1021f0 0x7f6dfc197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:31.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6e016fb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dfc104b20 0x7f6dfc197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:31.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6e016fb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dfc107450 0x7f6dfc19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:31.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6e016fb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc10d2b0 con 0x7f6dfc1021f0 2024-04-01T01:31:31.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6e016fb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc10d130 con 0x7f6dfc107450 2024-04-01T01:31:31.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6e016fb640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc10d430 con 0x7f6dfc104b20 2024-04-01T01:31:31.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6dfaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dfc1021f0 0x7f6dfc197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:31.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6dfaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dfc1021f0 0x7f6dfc197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40966/0 (socket says 172.21.15.70:40966) 2024-04-01T01:31:31.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6dfaffd640 1 -- 172.21.15.70:0/2848504056 learned_addr learned my addr 172.21.15.70:0/2848504056 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:31.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6dfb7fe640 1 --2- 172.21.15.70:0/2848504056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dfc107450 0x7f6dfc19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:31.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6dfa7fc640 1 --2- 172.21.15.70:0/2848504056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dfc104b20 0x7f6dfc197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:31.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6dfaffd640 1 -- 172.21.15.70:0/2848504056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dfc104b20 msgr2=0x7f6dfc197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6dfaffd640 1 --2- 172.21.15.70:0/2848504056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dfc104b20 0x7f6dfc197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6dfaffd640 1 -- 172.21.15.70:0/2848504056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dfc107450 msgr2=0x7f6dfc19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6dfaffd640 1 --2- 172.21.15.70:0/2848504056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dfc107450 0x7f6dfc19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6dfaffd640 1 -- 172.21.15.70:0/2848504056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dfc19f620 con 0x7f6dfc1021f0 2024-04-01T01:31:31.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6dfb7fe640 1 --2- 172.21.15.70:0/2848504056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dfc107450 0x7f6dfc19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:31.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6dfaffd640 1 --2- 172.21.15.70:0/2848504056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dfc1021f0 0x7f6dfc197a50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6dec00eb90 tx=0x7f6dec00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:31.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6de3fff640 1 -- 172.21.15.70:0/2848504056 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec004d20 con 0x7f6dfc1021f0 2024-04-01T01:31:31.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6de3fff640 1 -- 172.21.15.70:0/2848504056 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dec00ee90 con 0x7f6dfc1021f0 2024-04-01T01:31:31.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6e016fb640 1 -- 172.21.15.70:0/2848504056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dfc19f910 con 0x7f6dfc1021f0 2024-04-01T01:31:31.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6de3fff640 1 -- 172.21.15.70:0/2848504056 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec010670 con 0x7f6dfc1021f0 2024-04-01T01:31:31.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.733+0000 7f6e016fb640 1 -- 172.21.15.70:0/2848504056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dfc19fde0 con 0x7f6dfc1021f0 2024-04-01T01:31:31.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.734+0000 7f6e016fb640 1 -- 172.21.15.70:0/2848504056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dfc0fd6f0 con 0x7f6dfc1021f0 2024-04-01T01:31:31.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.735+0000 7f6de3fff640 1 -- 172.21.15.70:0/2848504056 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f6dec005020 con 0x7f6dfc1021f0 2024-04-01T01:31:31.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.736+0000 7f6de3fff640 1 --2- 172.21.15.70:0/2848504056 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6de40a24a0 0x7f6de40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:31.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.736+0000 7f6dfa7fc640 1 --2- 172.21.15.70:0/2848504056 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6de40a24a0 0x7f6de40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:31.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.736+0000 7f6de3fff640 1 -- 172.21.15.70:0/2848504056 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6dec0ceff0 con 0x7f6dfc1021f0 2024-04-01T01:31:31.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.736+0000 7f6dfa7fc640 1 --2- 172.21.15.70:0/2848504056 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6de40a24a0 0x7f6de40a4960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6df0005ab0 tx=0x7f6df0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:31.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.740+0000 7f6de3fff640 1 -- 172.21.15.70:0/2848504056 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6dec09bf70 con 0x7f6dfc1021f0 2024-04-01T01:31:31.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.927+0000 7f6e016fb640 1 -- 172.21.15.70:0/2848504056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6dfc062380 con 0x7f6dfc1021f0 2024-04-01T01:31:31.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.927+0000 7f6de3fff640 1 -- 172.21.15.70:0/2848504056 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+590 (secure 0 0 0) 0x7f6dec09b870 con 0x7f6dfc1021f0 2024-04-01T01:31:31.927 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/950171900 2024-04-02T01:24:18.477610+0000 2024-04-01T01:31:31.927 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/2902831307 2024-04-02T01:24:18.477610+0000 2024-04-01T01:31:31.928 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6801/2836712828 2024-04-02T01:24:18.477610+0000 2024-04-01T01:31:31.928 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6800/2836712828 2024-04-02T01:24:18.477610+0000 2024-04-01T01:31:31.928 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/602656012 2024-04-02T01:23:52.091015+0000 2024-04-01T01:31:31.928 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6800/2121290979 2024-04-02T01:23:52.091015+0000 2024-04-01T01:31:31.928 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6801/2121290979 2024-04-02T01:23:52.091015+0000 2024-04-01T01:31:31.928 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/3734182136 2024-04-02T01:23:52.091015+0000 2024-04-01T01:31:31.928 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/2899206193 2024-04-02T01:23:52.091015+0000 2024-04-01T01:31:31.928 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/3948868469 2024-04-02T01:24:18.477610+0000 2024-04-01T01:31:31.928 INFO:teuthology.orchestra.run.smithi070.stderr:listed 10 entries 2024-04-01T01:31:31.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.929+0000 7f6e016fb640 1 -- 172.21.15.70:0/2848504056 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6de40a24a0 msgr2=0x7f6de40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.929+0000 7f6e016fb640 1 --2- 172.21.15.70:0/2848504056 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6de40a24a0 0x7f6de40a4960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6df0005ab0 tx=0x7f6df0005fd0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.929+0000 7f6e016fb640 1 -- 172.21.15.70:0/2848504056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dfc1021f0 msgr2=0x7f6dfc197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:31.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.929+0000 7f6e016fb640 1 --2- 172.21.15.70:0/2848504056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dfc1021f0 0x7f6dfc197a50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6dec00eb90 tx=0x7f6dec00c6a0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.929+0000 7f6e016fb640 1 -- 172.21.15.70:0/2848504056 shutdown_connections 2024-04-01T01:31:31.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.929+0000 7f6e016fb640 1 --2- 172.21.15.70:0/2848504056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dfc107450 0x7f6dfc19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.929+0000 7f6e016fb640 1 --2- 172.21.15.70:0/2848504056 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6de40a24a0 0x7f6de40a4960 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.929+0000 7f6e016fb640 1 --2- 172.21.15.70:0/2848504056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dfc104b20 0x7f6dfc197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.929+0000 7f6e016fb640 1 --2- 172.21.15.70:0/2848504056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dfc1021f0 0x7f6dfc197a50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:31.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.929+0000 7f6e016fb640 1 -- 172.21.15.70:0/2848504056 >> 172.21.15.70:0/2848504056 conn(0x7f6dfc0fbc00 msgr2=0x7f6dfc102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:31.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.929+0000 7f6e016fb640 1 -- 172.21.15.70:0/2848504056 shutdown_connections 2024-04-01T01:31:31.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:31.929+0000 7f6e016fb640 1 -- 172.21.15.70:0/2848504056 wait complete. 2024-04-01T01:31:31.939 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:31 smithi070 ceph-mon[32179]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-04-01T01:31:31.939 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:31 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2374037878' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:31:31.939 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:31 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/323877124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:31:31.939 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:31 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3861550977' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:31:31.940 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-01T01:31:31.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:31:31.941 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-01T01:31:31.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi070.stderr:ceph-fuse[76219]: fuse finished with error 0 and tester_r 0 2024-04-01T01:31:31.968 INFO:teuthology.orchestra.run:waiting for 300 2024-04-01T01:31:32.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:31 smithi189 ceph-mon[35753]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-04-01T01:31:32.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:31 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2374037878' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:31:32.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:31 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/323877124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:31:32.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:31 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3861550977' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:31:32.063 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-01T01:31:32.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:31:32.064 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-01T01:31:32.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:31 smithi081 ceph-mon[35142]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-04-01T01:31:32.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:31 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2374037878' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:31:32.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:31 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/323877124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:31:32.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:31 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3861550977' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:31:32.082 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:31:32.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.225+0000 7f35860eb640 1 -- 172.21.15.70:0/2708816653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3580100690 msgr2=0x7f3580100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:32.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.225+0000 7f35860eb640 1 --2- 172.21.15.70:0/2708816653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3580100690 0x7f3580100ab0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3570009900 tx=0x7f357002bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:32.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.225+0000 7f35860eb640 1 -- 172.21.15.70:0/2708816653 shutdown_connections 2024-04-01T01:31:32.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.225+0000 7f35860eb640 1 --2- 172.21.15.70:0/2708816653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3580102240 0x7f3580108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.225+0000 7f35860eb640 1 --2- 172.21.15.70:0/2708816653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3580101880 0x7f3580101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.225+0000 7f35860eb640 1 --2- 172.21.15.70:0/2708816653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3580100690 0x7f3580100ab0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.225+0000 7f35860eb640 1 -- 172.21.15.70:0/2708816653 >> 172.21.15.70:0/2708816653 conn(0x7f35800fbc00 msgr2=0x7f35800fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:32.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.225+0000 7f35860eb640 1 -- 172.21.15.70:0/2708816653 shutdown_connections 2024-04-01T01:31:32.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.225+0000 7f35860eb640 1 -- 172.21.15.70:0/2708816653 wait complete. 2024-04-01T01:31:32.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.225+0000 7f35860eb640 1 Processor -- start 2024-04-01T01:31:32.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.225+0000 7f35860eb640 1 -- start start 2024-04-01T01:31:32.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f35860eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3580100690 0x7f358006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:32.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f35860eb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3580101880 0x7f358019c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:32.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f35860eb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3580102240 0x7f358019ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:32.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f35860eb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f358010c1c0 con 0x7f3580101880 2024-04-01T01:31:32.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f35860eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f358010c040 con 0x7f3580100690 2024-04-01T01:31:32.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f35860eb640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f358010c340 con 0x7f3580102240 2024-04-01T01:31:32.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f3576ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3580101880 0x7f358019c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:32.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f3576ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3580101880 0x7f358019c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40990/0 (socket says 172.21.15.70:40990) 2024-04-01T01:31:32.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f3576ffd640 1 -- 172.21.15.70:0/2794484606 learned_addr learned my addr 172.21.15.70:0/2794484606 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:32.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f357f7fe640 1 --2- 172.21.15.70:0/2794484606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3580100690 0x7f358006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:32.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f357ffff640 1 --2- 172.21.15.70:0/2794484606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3580102240 0x7f358019ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:32.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f3576ffd640 1 -- 172.21.15.70:0/2794484606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3580102240 msgr2=0x7f358019ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:32.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f3576ffd640 1 --2- 172.21.15.70:0/2794484606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3580102240 0x7f358019ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f3576ffd640 1 -- 172.21.15.70:0/2794484606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3580100690 msgr2=0x7f358006a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:32.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f3576ffd640 1 --2- 172.21.15.70:0/2794484606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3580100690 0x7f358006a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f3576ffd640 1 -- 172.21.15.70:0/2794484606 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f358019f600 con 0x7f3580101880 2024-04-01T01:31:32.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f357ffff640 1 --2- 172.21.15.70:0/2794484606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3580102240 0x7f358019ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:31:32.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f3576ffd640 1 --2- 172.21.15.70:0/2794484606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3580101880 0x7f358019c6d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f356800d770 tx=0x7f356800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:32.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f357d7fa640 1 -- 172.21.15.70:0/2794484606 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3568017070 con 0x7f3580101880 2024-04-01T01:31:32.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f35860eb640 1 -- 172.21.15.70:0/2794484606 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f358019f8f0 con 0x7f3580101880 2024-04-01T01:31:32.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f357d7fa640 1 -- 172.21.15.70:0/2794484606 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35680054d0 con 0x7f3580101880 2024-04-01T01:31:32.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f357d7fa640 1 -- 172.21.15.70:0/2794484606 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3568004cf0 con 0x7f3580101880 2024-04-01T01:31:32.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.226+0000 7f35860eb640 1 -- 172.21.15.70:0/2794484606 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f358019fdf0 con 0x7f3580101880 2024-04-01T01:31:32.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.227+0000 7f35860eb640 1 -- 172.21.15.70:0/2794484606 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3580107a80 con 0x7f3580101880 2024-04-01T01:31:32.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.228+0000 7f357d7fa640 1 -- 172.21.15.70:0/2794484606 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f3568004e90 con 0x7f3580101880 2024-04-01T01:31:32.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.229+0000 7f357d7fa640 1 --2- 172.21.15.70:0/2794484606 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f356c0a24a0 0x7f356c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:32.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.229+0000 7f357f7fe640 1 --2- 172.21.15.70:0/2794484606 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f356c0a24a0 0x7f356c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:32.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.229+0000 7f357d7fa640 1 -- 172.21.15.70:0/2794484606 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f35680ce3c0 con 0x7f3580101880 2024-04-01T01:31:32.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.229+0000 7f357f7fe640 1 --2- 172.21.15.70:0/2794484606 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f356c0a24a0 0x7f356c0a4960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3570005ab0 tx=0x7f3570005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:32.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.233+0000 7f357d7fa640 1 -- 172.21.15.70:0/2794484606 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f356809b340 con 0x7f3580101880 2024-04-01T01:31:32.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.413+0000 7f35860eb640 1 -- 172.21.15.70:0/2794484606 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3580062350 con 0x7f3580101880 2024-04-01T01:31:32.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.415+0000 7f357d7fa640 1 -- 172.21.15.70:0/2794484606 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24068 (secure 0 0 0) 0x7f356809ac40 con 0x7f3580101880 2024-04-01T01:31:32.415 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:32.415 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":89,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:31:20.903623+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:31:32.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.417+0000 7f35860eb640 1 -- 172.21.15.70:0/2794484606 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f356c0a24a0 msgr2=0x7f356c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:32.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.417+0000 7f35860eb640 1 --2- 172.21.15.70:0/2794484606 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f356c0a24a0 0x7f356c0a4960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3570005ab0 tx=0x7f3570005fd0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.417+0000 7f35860eb640 1 -- 172.21.15.70:0/2794484606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3580101880 msgr2=0x7f358019c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:32.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.417+0000 7f35860eb640 1 --2- 172.21.15.70:0/2794484606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3580101880 0x7f358019c6d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f356800d770 tx=0x7f356800dc30 comp rx=0 tx=0).stop 2024-04-01T01:31:32.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.417+0000 7f35860eb640 1 -- 172.21.15.70:0/2794484606 shutdown_connections 2024-04-01T01:31:32.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.417+0000 7f35860eb640 1 --2- 172.21.15.70:0/2794484606 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f356c0a24a0 0x7f356c0a4960 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.417+0000 7f35860eb640 1 --2- 172.21.15.70:0/2794484606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3580102240 0x7f358019ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.417+0000 7f35860eb640 1 --2- 172.21.15.70:0/2794484606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3580101880 0x7f358019c6d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.417+0000 7f35860eb640 1 --2- 172.21.15.70:0/2794484606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3580100690 0x7f358006a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.417+0000 7f35860eb640 1 -- 172.21.15.70:0/2794484606 >> 172.21.15.70:0/2794484606 conn(0x7f35800fbc00 msgr2=0x7f3580104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:32.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.417+0000 7f35860eb640 1 -- 172.21.15.70:0/2794484606 shutdown_connections 2024-04-01T01:31:32.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.417+0000 7f35860eb640 1 -- 172.21.15.70:0/2794484606 wait complete. 2024-04-01T01:31:32.428 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:31:32.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 -- 172.21.15.70:0/3667356988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65b8102980 msgr2=0x7f65b8102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:32.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 --2- 172.21.15.70:0/3667356988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65b8102980 0x7f65b8102e00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f65b4009900 tx=0x7f65b402bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:32.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 -- 172.21.15.70:0/3667356988 shutdown_connections 2024-04-01T01:31:32.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 --2- 172.21.15.70:0/3667356988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65b8103340 0x7f65b8109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 --2- 172.21.15.70:0/3667356988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65b8102980 0x7f65b8102e00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 --2- 172.21.15.70:0/3667356988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65b8101790 0x7f65b8101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 -- 172.21.15.70:0/3667356988 >> 172.21.15.70:0/3667356988 conn(0x7f65b80fcd00 msgr2=0x7f65b80ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 -- 172.21.15.70:0/3667356988 shutdown_connections 2024-04-01T01:31:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 -- 172.21.15.70:0/3667356988 wait complete. 2024-04-01T01:31:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 Processor -- start 2024-04-01T01:31:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 -- start start 2024-04-01T01:31:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65b8101790 0x7f65b8197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65b8102980 0x7f65b8197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65b8103340 0x7f65b819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65b810d2c0 con 0x7f65b8101790 2024-04-01T01:31:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f65b810d140 con 0x7f65b8103340 2024-04-01T01:31:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.530+0000 7f65bf35c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f65b810d440 con 0x7f65b8102980 2024-04-01T01:31:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65bd0f8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65b8101790 0x7f65b8197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65bd0f8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65b8101790 0x7f65b8197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41010/0 (socket says 172.21.15.70:41010) 2024-04-01T01:31:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65bd0f8640 1 -- 172.21.15.70:0/1079480158 learned_addr learned my addr 172.21.15.70:0/1079480158 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65bd8f9640 1 --2- 172.21.15.70:0/1079480158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65b8103340 0x7f65b819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65bc8f7640 1 --2- 172.21.15.70:0/1079480158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65b8102980 0x7f65b8197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65bd0f8640 1 -- 172.21.15.70:0/1079480158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65b8102980 msgr2=0x7f65b8197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65bd0f8640 1 --2- 172.21.15.70:0/1079480158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65b8102980 0x7f65b8197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65bd0f8640 1 -- 172.21.15.70:0/1079480158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65b8103340 msgr2=0x7f65b819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65bd0f8640 1 --2- 172.21.15.70:0/1079480158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65b8103340 0x7f65b819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65bd0f8640 1 -- 172.21.15.70:0/1079480158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b819f620 con 0x7f65b8101790 2024-04-01T01:31:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65bd8f9640 1 --2- 172.21.15.70:0/1079480158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65b8103340 0x7f65b819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65bd0f8640 1 --2- 172.21.15.70:0/1079480158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65b8101790 0x7f65b8197a00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f65a800c970 tx=0x7f65a800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65ae7fc640 1 -- 172.21.15.70:0/1079480158 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8004220 con 0x7f65b8101790 2024-04-01T01:31:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65bf35c640 1 -- 172.21.15.70:0/1079480158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b819f910 con 0x7f65b8101790 2024-04-01T01:31:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65ae7fc640 1 -- 172.21.15.70:0/1079480158 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f65a80043c0 con 0x7f65b8101790 2024-04-01T01:31:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65ae7fc640 1 -- 172.21.15.70:0/1079480158 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8004a00 con 0x7f65b8101790 2024-04-01T01:31:32.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.531+0000 7f65bf35c640 1 -- 172.21.15.70:0/1079480158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b819fe10 con 0x7f65b8101790 2024-04-01T01:31:32.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.532+0000 7f65bf35c640 1 -- 172.21.15.70:0/1079480158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65b8108bd0 con 0x7f65b8101790 2024-04-01T01:31:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.533+0000 7f65ae7fc640 1 -- 172.21.15.70:0/1079480158 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f65a80079b0 con 0x7f65b8101790 2024-04-01T01:31:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.534+0000 7f65ae7fc640 1 --2- 172.21.15.70:0/1079480158 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65a40a24a0 0x7f65a40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.534+0000 7f65bc8f7640 1 --2- 172.21.15.70:0/1079480158 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65a40a24a0 0x7f65a40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.534+0000 7f65ae7fc640 1 -- 172.21.15.70:0/1079480158 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f65a80ce6e0 con 0x7f65b8101790 2024-04-01T01:31:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.534+0000 7f65bc8f7640 1 --2- 172.21.15.70:0/1079480158 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65a40a24a0 0x7f65a40a4960 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f65b4036040 tx=0x7f65b4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:32.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.538+0000 7f65ae7fc640 1 -- 172.21.15.70:0/1079480158 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f65a8014030 con 0x7f65b8101790 2024-04-01T01:31:32.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.718+0000 7f65bf35c640 1 -- 172.21.15.70:0/1079480158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f65b8062350 con 0x7f65b8101790 2024-04-01T01:31:32.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.720+0000 7f65ae7fc640 1 -- 172.21.15.70:0/1079480158 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24068 (secure 0 0 0) 0x7f65a809b660 con 0x7f65b8101790 2024-04-01T01:31:32.721 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:32.721 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":89,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:31:20.903623+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:31:32.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.722+0000 7f65bf35c640 1 -- 172.21.15.70:0/1079480158 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65a40a24a0 msgr2=0x7f65a40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:32.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.722+0000 7f65bf35c640 1 --2- 172.21.15.70:0/1079480158 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65a40a24a0 0x7f65a40a4960 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f65b4036040 tx=0x7f65b4000bc0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.722+0000 7f65bf35c640 1 -- 172.21.15.70:0/1079480158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65b8101790 msgr2=0x7f65b8197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:32.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.722+0000 7f65bf35c640 1 --2- 172.21.15.70:0/1079480158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65b8101790 0x7f65b8197a00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f65a800c970 tx=0x7f65a800ce30 comp rx=0 tx=0).stop 2024-04-01T01:31:32.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.722+0000 7f65bf35c640 1 -- 172.21.15.70:0/1079480158 shutdown_connections 2024-04-01T01:31:32.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.722+0000 7f65bf35c640 1 --2- 172.21.15.70:0/1079480158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65b8103340 0x7f65b819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.722+0000 7f65bf35c640 1 --2- 172.21.15.70:0/1079480158 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65a40a24a0 0x7f65a40a4960 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.722+0000 7f65bf35c640 1 --2- 172.21.15.70:0/1079480158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65b8102980 0x7f65b8197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.722+0000 7f65bf35c640 1 --2- 172.21.15.70:0/1079480158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65b8101790 0x7f65b8197a00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.722+0000 7f65bf35c640 1 -- 172.21.15.70:0/1079480158 >> 172.21.15.70:0/1079480158 conn(0x7f65b80fcd00 msgr2=0x7f65b8105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:32.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.722+0000 7f65bf35c640 1 -- 172.21.15.70:0/1079480158 shutdown_connections 2024-04-01T01:31:32.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.722+0000 7f65bf35c640 1 -- 172.21.15.70:0/1079480158 wait complete. 2024-04-01T01:31:32.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:32 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/91293488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:31:32.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:32 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2848504056' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-01T01:31:32.732 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:32 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2794484606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:32.734 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-01T01:31:32.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.838+0000 7f3107756640 1 -- 172.21.15.70:0/2388327569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3100100670 msgr2=0x7f3100100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:32.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.838+0000 7f3107756640 1 --2- 172.21.15.70:0/2388327569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3100100670 0x7f3100100a90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009900 tx=0x7f30fc02bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:32.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.838+0000 7f3107756640 1 -- 172.21.15.70:0/2388327569 shutdown_connections 2024-04-01T01:31:32.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.838+0000 7f3107756640 1 --2- 172.21.15.70:0/2388327569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3100102220 0x7f3100108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.838+0000 7f3107756640 1 --2- 172.21.15.70:0/2388327569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3100101860 0x7f3100101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.838+0000 7f3107756640 1 --2- 172.21.15.70:0/2388327569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3100100670 0x7f3100100a90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.838+0000 7f3107756640 1 -- 172.21.15.70:0/2388327569 >> 172.21.15.70:0/2388327569 conn(0x7f31000fbc00 msgr2=0x7f31000fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:32.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.838+0000 7f3107756640 1 -- 172.21.15.70:0/2388327569 shutdown_connections 2024-04-01T01:31:32.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.838+0000 7f3107756640 1 -- 172.21.15.70:0/2388327569 wait complete. 2024-04-01T01:31:32.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3107756640 1 Processor -- start 2024-04-01T01:31:32.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3107756640 1 -- start start 2024-04-01T01:31:32.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3107756640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3100100670 0x7f310006a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:32.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3107756640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3100101860 0x7f310019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:32.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3107756640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3100102220 0x7f31001a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:32.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3107756640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f310010c1a0 con 0x7f3100102220 2024-04-01T01:31:32.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3107756640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f310010c020 con 0x7f3100101860 2024-04-01T01:31:32.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3107756640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f310010c320 con 0x7f3100100670 2024-04-01T01:31:32.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3105cf3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3100102220 0x7f31001a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3105cf3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3100102220 0x7f31001a10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41038/0 (socket says 172.21.15.70:41038) 2024-04-01T01:31:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3105cf3640 1 -- 172.21.15.70:0/3811458233 learned_addr learned my addr 172.21.15.70:0/3811458233 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3104cf1640 1 --2- 172.21.15.70:0/3811458233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3100101860 0x7f310019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f31054f2640 1 --2- 172.21.15.70:0/3811458233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3100100670 0x7f310006a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3105cf3640 1 -- 172.21.15.70:0/3811458233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3100100670 msgr2=0x7f310006a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3105cf3640 1 --2- 172.21.15.70:0/3811458233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3100100670 0x7f310006a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3105cf3640 1 -- 172.21.15.70:0/3811458233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3100101860 msgr2=0x7f310019eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3105cf3640 1 --2- 172.21.15.70:0/3811458233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3100101860 0x7f310019eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.839+0000 7f3105cf3640 1 -- 172.21.15.70:0/3811458233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31001a18b0 con 0x7f3100102220 2024-04-01T01:31:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.840+0000 7f3105cf3640 1 --2- 172.21.15.70:0/3811458233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3100102220 0x7f31001a10c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f30f800eb10 tx=0x7f30f800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.840+0000 7f30f67fc640 1 -- 172.21.15.70:0/3811458233 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f8004b20 con 0x7f3100102220 2024-04-01T01:31:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.840+0000 7f30f67fc640 1 -- 172.21.15.70:0/3811458233 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30f8004cc0 con 0x7f3100102220 2024-04-01T01:31:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.840+0000 7f3107756640 1 -- 172.21.15.70:0/3811458233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31001a1b40 con 0x7f3100102220 2024-04-01T01:31:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.840+0000 7f30f67fc640 1 -- 172.21.15.70:0/3811458233 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f8010690 con 0x7f3100102220 2024-04-01T01:31:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.840+0000 7f3107756640 1 -- 172.21.15.70:0/3811458233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31001a2040 con 0x7f3100102220 2024-04-01T01:31:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.841+0000 7f3107756640 1 -- 172.21.15.70:0/3811458233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3100107ab0 con 0x7f3100102220 2024-04-01T01:31:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.842+0000 7f30f67fc640 1 -- 172.21.15.70:0/3811458233 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f30f8005020 con 0x7f3100102220 2024-04-01T01:31:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.842+0000 7f30f67fc640 1 --2- 172.21.15.70:0/3811458233 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30ec0a24d0 0x7f30ec0a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:32.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.842+0000 7f30f67fc640 1 -- 172.21.15.70:0/3811458233 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f30f80cef40 con 0x7f3100102220 2024-04-01T01:31:32.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.842+0000 7f31054f2640 1 --2- 172.21.15.70:0/3811458233 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30ec0a24d0 0x7f30ec0a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:32.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.843+0000 7f31054f2640 1 --2- 172.21.15.70:0/3811458233 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30ec0a24d0 0x7f30ec0a4990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f30fc036040 tx=0x7f30fc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:32.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:32.847+0000 7f30f67fc640 1 -- 172.21.15.70:0/3811458233 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f30f809bec0 con 0x7f3100102220 2024-04-01T01:31:32.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:32 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/91293488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:31:32.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:32 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2848504056' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-01T01:31:32.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:32 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2794484606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:33.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.058+0000 7f3107756640 1 -- 172.21.15.70:0/3811458233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f31000623f0 con 0x7f3100102220 2024-04-01T01:31:33.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:32 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/91293488' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:31:33.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:32 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2848504056' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-01T01:31:33.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:32 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2794484606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:33.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.594+0000 7f30f67fc640 1 -- 172.21.15.70:0/3811458233 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f30f809b7c0 con 0x7f3100102220 2024-04-01T01:31:33.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.596+0000 7f3107756640 1 -- 172.21.15.70:0/3811458233 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30ec0a24d0 msgr2=0x7f30ec0a4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:33.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.596+0000 7f3107756640 1 --2- 172.21.15.70:0/3811458233 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30ec0a24d0 0x7f30ec0a4990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f30fc036040 tx=0x7f30fc000bc0 comp rx=0 tx=0).stop 2024-04-01T01:31:33.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.596+0000 7f3107756640 1 -- 172.21.15.70:0/3811458233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3100102220 msgr2=0x7f31001a10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:33.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.596+0000 7f3107756640 1 --2- 172.21.15.70:0/3811458233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3100102220 0x7f31001a10c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f30f800eb10 tx=0x7f30f800c2d0 comp rx=0 tx=0).stop 2024-04-01T01:31:33.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.596+0000 7f3107756640 1 -- 172.21.15.70:0/3811458233 shutdown_connections 2024-04-01T01:31:33.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.596+0000 7f3107756640 1 --2- 172.21.15.70:0/3811458233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3100102220 0x7f31001a10c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:33.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.596+0000 7f3107756640 1 --2- 172.21.15.70:0/3811458233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3100101860 0x7f310019eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:33.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.596+0000 7f3107756640 1 --2- 172.21.15.70:0/3811458233 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30ec0a24d0 0x7f30ec0a4990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:33.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.596+0000 7f3107756640 1 --2- 172.21.15.70:0/3811458233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3100100670 0x7f310006a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:33.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.596+0000 7f3107756640 1 -- 172.21.15.70:0/3811458233 >> 172.21.15.70:0/3811458233 conn(0x7f31000fbc00 msgr2=0x7f3100104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:33.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.596+0000 7f3107756640 1 -- 172.21.15.70:0/3811458233 shutdown_connections 2024-04-01T01:31:33.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.597+0000 7f3107756640 1 -- 172.21.15.70:0/3811458233 wait complete. 2024-04-01T01:31:33.610 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-01T01:31:33.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-01T01:31:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.719+0000 7f9b8e720640 1 -- 172.21.15.70:0/156346311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b88101860 msgr2=0x7f9b88101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.719+0000 7f9b8e720640 1 --2- 172.21.15.70:0/156346311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b88101860 0x7f9b88101ce0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009900 tx=0x7f9b7002bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.719+0000 7f9b8e720640 1 -- 172.21.15.70:0/156346311 shutdown_connections 2024-04-01T01:31:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.719+0000 7f9b8e720640 1 --2- 172.21.15.70:0/156346311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b88102220 0x7f9b88108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.719+0000 7f9b8e720640 1 --2- 172.21.15.70:0/156346311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b88101860 0x7f9b88101ce0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.719+0000 7f9b8e720640 1 --2- 172.21.15.70:0/156346311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b88100670 0x7f9b88100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.719+0000 7f9b8e720640 1 -- 172.21.15.70:0/156346311 >> 172.21.15.70:0/156346311 conn(0x7f9b880fbc00 msgr2=0x7f9b880fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.719+0000 7f9b8e720640 1 -- 172.21.15.70:0/156346311 shutdown_connections 2024-04-01T01:31:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.719+0000 7f9b8e720640 1 -- 172.21.15.70:0/156346311 wait complete. 2024-04-01T01:31:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.720+0000 7f9b8e720640 1 Processor -- start 2024-04-01T01:31:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.720+0000 7f9b8e720640 1 -- start start 2024-04-01T01:31:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.720+0000 7f9b8e720640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b88100670 0x7f9b88197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.720+0000 7f9b8e720640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b88101860 0x7f9b88197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.720+0000 7f9b8e720640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b88102220 0x7f9b8819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.720+0000 7f9b8e720640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8810c1b0 con 0x7f9b88100670 2024-04-01T01:31:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.720+0000 7f9b8e720640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8810c030 con 0x7f9b88102220 2024-04-01T01:31:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.720+0000 7f9b8e720640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8810c330 con 0x7f9b88101860 2024-04-01T01:31:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.720+0000 7f9b87fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b88100670 0x7f9b88197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.720+0000 7f9b87fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b88100670 0x7f9b88197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41062/0 (socket says 172.21.15.70:41062) 2024-04-01T01:31:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.720+0000 7f9b87fff640 1 -- 172.21.15.70:0/4169456651 learned_addr learned my addr 172.21.15.70:0/4169456651 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.720+0000 7f9b8ccbd640 1 --2- 172.21.15.70:0/4169456651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b88102220 0x7f9b8819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.720+0000 7f9b877fe640 1 --2- 172.21.15.70:0/4169456651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b88101860 0x7f9b88197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.721+0000 7f9b87fff640 1 -- 172.21.15.70:0/4169456651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b88101860 msgr2=0x7f9b88197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:33.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.721+0000 7f9b87fff640 1 --2- 172.21.15.70:0/4169456651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b88101860 0x7f9b88197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:33.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.721+0000 7f9b87fff640 1 -- 172.21.15.70:0/4169456651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b88102220 msgr2=0x7f9b8819ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:33.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.721+0000 7f9b87fff640 1 --2- 172.21.15.70:0/4169456651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b88102220 0x7f9b8819ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:33.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.721+0000 7f9b87fff640 1 -- 172.21.15.70:0/4169456651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b8819f5b0 con 0x7f9b88100670 2024-04-01T01:31:33.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.721+0000 7f9b87fff640 1 --2- 172.21.15.70:0/4169456651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b88100670 0x7f9b88197990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800e9c0 tx=0x7f9b7800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:33.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.721+0000 7f9b857fa640 1 -- 172.21.15.70:0/4169456651 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7800cd80 con 0x7f9b88100670 2024-04-01T01:31:33.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.721+0000 7f9b8e720640 1 -- 172.21.15.70:0/4169456651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b8819f8a0 con 0x7f9b88100670 2024-04-01T01:31:33.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.721+0000 7f9b857fa640 1 -- 172.21.15.70:0/4169456651 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b780035f0 con 0x7f9b88100670 2024-04-01T01:31:33.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.721+0000 7f9b857fa640 1 -- 172.21.15.70:0/4169456651 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b78010690 con 0x7f9b88100670 2024-04-01T01:31:33.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.721+0000 7f9b8e720640 1 -- 172.21.15.70:0/4169456651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b8819fe10 con 0x7f9b88100670 2024-04-01T01:31:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.722+0000 7f9b8e720640 1 -- 172.21.15.70:0/4169456651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b881a0040 con 0x7f9b88100670 2024-04-01T01:31:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.723+0000 7f9b857fa640 1 -- 172.21.15.70:0/4169456651 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f9b780037a0 con 0x7f9b88100670 2024-04-01T01:31:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.724+0000 7f9b857fa640 1 --2- 172.21.15.70:0/4169456651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9b740a24a0 0x7f9b740a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.724+0000 7f9b877fe640 1 --2- 172.21.15.70:0/4169456651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9b740a24a0 0x7f9b740a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.724+0000 7f9b857fa640 1 -- 172.21.15.70:0/4169456651 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9b780cece0 con 0x7f9b88100670 2024-04-01T01:31:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.724+0000 7f9b877fe640 1 --2- 172.21.15.70:0/4169456651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9b740a24a0 0x7f9b740a4960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009230 tx=0x7f9b70005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:33.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.728+0000 7f9b857fa640 1 -- 172.21.15.70:0/4169456651 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9b7809bc60 con 0x7f9b88100670 2024-04-01T01:31:33.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:33.938+0000 7f9b8e720640 1 -- 172.21.15.70:0/4169456651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f9b881a0210 con 0x7f9b88100670 2024-04-01T01:31:33.978 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:33 smithi081 ceph-mon[35142]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 279 B/s wr, 0 op/s 2024-04-01T01:31:33.978 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:33 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1079480158' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:33.978 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:33 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3811458233' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-01T01:31:33.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:33 smithi189 ceph-mon[35753]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 279 B/s wr, 0 op/s 2024-04-01T01:31:33.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:33 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1079480158' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:33.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:33 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3811458233' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-01T01:31:34.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:33 smithi070 ceph-mon[32179]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 279 B/s wr, 0 op/s 2024-04-01T01:31:34.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:33 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1079480158' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:34.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:33 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3811458233' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-01T01:31:34.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.603+0000 7f9b857fa640 1 -- 172.21.15.70:0/4169456651 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f9b7809b560 con 0x7f9b88100670 2024-04-01T01:31:34.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.606+0000 7f9b8e720640 1 -- 172.21.15.70:0/4169456651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9b740a24a0 msgr2=0x7f9b740a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:34.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.606+0000 7f9b8e720640 1 --2- 172.21.15.70:0/4169456651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9b740a24a0 0x7f9b740a4960 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9b70009230 tx=0x7f9b70005fd0 comp rx=0 tx=0).stop 2024-04-01T01:31:34.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.606+0000 7f9b8e720640 1 -- 172.21.15.70:0/4169456651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b88100670 msgr2=0x7f9b88197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:34.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.606+0000 7f9b8e720640 1 --2- 172.21.15.70:0/4169456651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b88100670 0x7f9b88197990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800e9c0 tx=0x7f9b7800ee80 comp rx=0 tx=0).stop 2024-04-01T01:31:34.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.606+0000 7f9b8e720640 1 -- 172.21.15.70:0/4169456651 shutdown_connections 2024-04-01T01:31:34.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.606+0000 7f9b8e720640 1 --2- 172.21.15.70:0/4169456651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b88102220 0x7f9b8819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:34.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.606+0000 7f9b8e720640 1 --2- 172.21.15.70:0/4169456651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9b740a24a0 0x7f9b740a4960 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:34.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.606+0000 7f9b8e720640 1 --2- 172.21.15.70:0/4169456651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b88101860 0x7f9b88197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:34.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.606+0000 7f9b8e720640 1 --2- 172.21.15.70:0/4169456651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b88100670 0x7f9b88197990 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:34.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.606+0000 7f9b8e720640 1 -- 172.21.15.70:0/4169456651 >> 172.21.15.70:0/4169456651 conn(0x7f9b880fbc00 msgr2=0x7f9b88069d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:34.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.606+0000 7f9b8e720640 1 -- 172.21.15.70:0/4169456651 shutdown_connections 2024-04-01T01:31:34.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.606+0000 7f9b8e720640 1 -- 172.21.15.70:0/4169456651 wait complete. 2024-04-01T01:31:34.621 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-01T01:31:34.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 -- 172.21.15.70:0/2280571958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15781033b0 msgr2=0x7f1578109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:34.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 --2- 172.21.15.70:0/2280571958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15781033b0 0x7f1578109c40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1570009cd0 tx=0x7f157002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:31:34.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 -- 172.21.15.70:0/2280571958 shutdown_connections 2024-04-01T01:31:34.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 --2- 172.21.15.70:0/2280571958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15781033b0 0x7f1578109c40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:34.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 --2- 172.21.15.70:0/2280571958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15781029f0 0x7f1578102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:34.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 --2- 172.21.15.70:0/2280571958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15780ffa20 0x7f15780ffe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:34.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 -- 172.21.15.70:0/2280571958 >> 172.21.15.70:0/2280571958 conn(0x7f15780fb370 msgr2=0x7f15780fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:34.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 -- 172.21.15.70:0/2280571958 shutdown_connections 2024-04-01T01:31:34.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 -- 172.21.15.70:0/2280571958 wait complete. 2024-04-01T01:31:34.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 Processor -- start 2024-04-01T01:31:34.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 -- start start 2024-04-01T01:31:34.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15780ffa20 0x7f1578102360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:34.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15781029f0 0x7f15781009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:34.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15781033b0 0x7f1578100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:34.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f157810d320 con 0x7f15780ffa20 2024-04-01T01:31:34.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f157810d1a0 con 0x7f15781033b0 2024-04-01T01:31:34.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157ea84640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f157810d4a0 con 0x7f15781029f0 2024-04-01T01:31:34.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.727+0000 7f157c820640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15780ffa20 0x7f1578102360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:34.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.728+0000 7f157c820640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15780ffa20 0x7f1578102360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41084/0 (socket says 172.21.15.70:41084) 2024-04-01T01:31:34.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.728+0000 7f157c820640 1 -- 172.21.15.70:0/1293509173 learned_addr learned my addr 172.21.15.70:0/1293509173 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:34.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.728+0000 7f157d021640 1 --2- 172.21.15.70:0/1293509173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15781033b0 0x7f1578100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:34.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.728+0000 7f1577fff640 1 --2- 172.21.15.70:0/1293509173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15781029f0 0x7f15781009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:34.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.728+0000 7f157c820640 1 -- 172.21.15.70:0/1293509173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15781029f0 msgr2=0x7f15781009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:34.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.728+0000 7f157c820640 1 --2- 172.21.15.70:0/1293509173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15781029f0 0x7f15781009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:34.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.728+0000 7f157c820640 1 -- 172.21.15.70:0/1293509173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15781033b0 msgr2=0x7f1578100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:34.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.728+0000 7f157c820640 1 --2- 172.21.15.70:0/1293509173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15781033b0 0x7f1578100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:34.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.728+0000 7f157c820640 1 -- 172.21.15.70:0/1293509173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1578101770 con 0x7f15780ffa20 2024-04-01T01:31:34.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.728+0000 7f157c820640 1 --2- 172.21.15.70:0/1293509173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15780ffa20 0x7f1578102360 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f156c00b450 tx=0x7f156c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:34.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.728+0000 7f1575ffb640 1 -- 172.21.15.70:0/1293509173 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c015020 con 0x7f15780ffa20 2024-04-01T01:31:34.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.728+0000 7f1575ffb640 1 -- 172.21.15.70:0/1293509173 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f156c011d40 con 0x7f15780ffa20 2024-04-01T01:31:34.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.728+0000 7f157ea84640 1 -- 172.21.15.70:0/1293509173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15781a1b20 con 0x7f15780ffa20 2024-04-01T01:31:34.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.728+0000 7f1575ffb640 1 -- 172.21.15.70:0/1293509173 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c00f340 con 0x7f15780ffa20 2024-04-01T01:31:34.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.728+0000 7f157ea84640 1 -- 172.21.15.70:0/1293509173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15781a2020 con 0x7f15780ffa20 2024-04-01T01:31:34.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.729+0000 7f157ea84640 1 -- 172.21.15.70:0/1293509173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1578068990 con 0x7f15780ffa20 2024-04-01T01:31:34.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.730+0000 7f1575ffb640 1 -- 172.21.15.70:0/1293509173 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f156c018020 con 0x7f15780ffa20 2024-04-01T01:31:34.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.731+0000 7f1575ffb640 1 --2- 172.21.15.70:0/1293509173 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f15680a24a0 0x7f15680a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:34.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.731+0000 7f1575ffb640 1 -- 172.21.15.70:0/1293509173 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f156c0ce4c0 con 0x7f15780ffa20 2024-04-01T01:31:34.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.731+0000 7f1577fff640 1 --2- 172.21.15.70:0/1293509173 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f15680a24a0 0x7f15680a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:34.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.731+0000 7f1577fff640 1 --2- 172.21.15.70:0/1293509173 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f15680a24a0 0x7f15680a4960 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1564005d00 tx=0x7f1564005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:34.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.735+0000 7f1575ffb640 1 -- 172.21.15.70:0/1293509173 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f156c01f020 con 0x7f15780ffa20 2024-04-01T01:31:34.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:34.947+0000 7f157ea84640 1 -- 172.21.15.70:0/1293509173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f1578062350 con 0x7f15780ffa20 2024-04-01T01:31:34.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:34 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3811458233' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-01T01:31:34.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:34 smithi189 ceph-mon[35753]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-01T01:31:34.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:34 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/4169456651' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-01T01:31:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:34 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3811458233' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-01T01:31:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:34 smithi070 ceph-mon[32179]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-01T01:31:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:34 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/4169456651' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-01T01:31:35.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:34 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3811458233' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-01T01:31:35.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:34 smithi081 ceph-mon[35142]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-01T01:31:35.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:34 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/4169456651' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-01T01:31:35.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.536+0000 7f1575ffb640 1 -- 172.21.15.70:0/1293509173 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f156c00d040 con 0x7f15780ffa20 2024-04-01T01:31:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.539+0000 7f157ea84640 1 -- 172.21.15.70:0/1293509173 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f15680a24a0 msgr2=0x7f15680a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.539+0000 7f157ea84640 1 --2- 172.21.15.70:0/1293509173 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f15680a24a0 0x7f15680a4960 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1564005d00 tx=0x7f1564005c50 comp rx=0 tx=0).stop 2024-04-01T01:31:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.539+0000 7f157ea84640 1 -- 172.21.15.70:0/1293509173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15780ffa20 msgr2=0x7f1578102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.539+0000 7f157ea84640 1 --2- 172.21.15.70:0/1293509173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15780ffa20 0x7f1578102360 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f156c00b450 tx=0x7f156c00b910 comp rx=0 tx=0).stop 2024-04-01T01:31:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.539+0000 7f157ea84640 1 -- 172.21.15.70:0/1293509173 shutdown_connections 2024-04-01T01:31:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.539+0000 7f157ea84640 1 --2- 172.21.15.70:0/1293509173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15781033b0 0x7f1578100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.539+0000 7f157ea84640 1 --2- 172.21.15.70:0/1293509173 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f15680a24a0 0x7f15680a4960 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.539+0000 7f157ea84640 1 --2- 172.21.15.70:0/1293509173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15781029f0 0x7f15781009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.539+0000 7f157ea84640 1 --2- 172.21.15.70:0/1293509173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15780ffa20 0x7f1578102360 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.539+0000 7f157ea84640 1 -- 172.21.15.70:0/1293509173 >> 172.21.15.70:0/1293509173 conn(0x7f15780fb370 msgr2=0x7f1578105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.539+0000 7f157ea84640 1 -- 172.21.15.70:0/1293509173 shutdown_connections 2024-04-01T01:31:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.540+0000 7f157ea84640 1 -- 172.21.15.70:0/1293509173 wait complete. 2024-04-01T01:31:35.556 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-01T01:31:35.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.666+0000 7f77a83a1640 1 -- 172.21.15.70:0/3829462230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a0100670 msgr2=0x7f77a0100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:35.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.666+0000 7f77a83a1640 1 --2- 172.21.15.70:0/3829462230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a0100670 0x7f77a0100a90 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7790009900 tx=0x7f779002bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:35.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a83a1640 1 -- 172.21.15.70:0/3829462230 shutdown_connections 2024-04-01T01:31:35.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a83a1640 1 --2- 172.21.15.70:0/3829462230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77a0102220 0x7f77a0108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:35.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a83a1640 1 --2- 172.21.15.70:0/3829462230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77a0101860 0x7f77a0101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:35.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a83a1640 1 --2- 172.21.15.70:0/3829462230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a0100670 0x7f77a0100a90 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:35.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a83a1640 1 -- 172.21.15.70:0/3829462230 >> 172.21.15.70:0/3829462230 conn(0x7f77a00fbbe0 msgr2=0x7f77a00fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:35.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a83a1640 1 -- 172.21.15.70:0/3829462230 shutdown_connections 2024-04-01T01:31:35.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a83a1640 1 -- 172.21.15.70:0/3829462230 wait complete. 2024-04-01T01:31:35.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a83a1640 1 Processor -- start 2024-04-01T01:31:35.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a83a1640 1 -- start start 2024-04-01T01:31:35.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a83a1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a0100670 0x7f77a0197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:35.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a83a1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77a0101860 0x7f77a0197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:35.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a83a1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77a0102220 0x7f77a019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:35.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a83a1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f77a010c1a0 con 0x7f77a0100670 2024-04-01T01:31:35.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a83a1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f77a010c020 con 0x7f77a0101860 2024-04-01T01:31:35.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a83a1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f77a010c320 con 0x7f77a0102220 2024-04-01T01:31:35.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a613d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a0100670 0x7f77a0197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:35.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a613d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a0100670 0x7f77a0197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41094/0 (socket says 172.21.15.70:41094) 2024-04-01T01:31:35.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.667+0000 7f77a613d640 1 -- 172.21.15.70:0/92718226 learned_addr learned my addr 172.21.15.70:0/92718226 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:35.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.668+0000 7f77a593c640 1 --2- 172.21.15.70:0/92718226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77a0101860 0x7f77a0197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:35.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.668+0000 7f77a693e640 1 --2- 172.21.15.70:0/92718226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77a0102220 0x7f77a019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:35.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.668+0000 7f77a613d640 1 -- 172.21.15.70:0/92718226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77a0102220 msgr2=0x7f77a019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:35.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.668+0000 7f77a613d640 1 --2- 172.21.15.70:0/92718226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77a0102220 0x7f77a019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:35.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.668+0000 7f77a613d640 1 -- 172.21.15.70:0/92718226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77a0101860 msgr2=0x7f77a0197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:35.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.668+0000 7f77a613d640 1 --2- 172.21.15.70:0/92718226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77a0101860 0x7f77a0197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:35.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.668+0000 7f77a613d640 1 -- 172.21.15.70:0/92718226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77a019f660 con 0x7f77a0100670 2024-04-01T01:31:35.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.668+0000 7f77a693e640 1 --2- 172.21.15.70:0/92718226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77a0102220 0x7f77a019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:31:35.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.668+0000 7f77a613d640 1 --2- 172.21.15.70:0/92718226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a0100670 0x7f77a0197a90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7790000c00 tx=0x7f7790005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:35.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.668+0000 7f77977fe640 1 -- 172.21.15.70:0/92718226 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790043020 con 0x7f77a0100670 2024-04-01T01:31:35.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.668+0000 7f77a83a1640 1 -- 172.21.15.70:0/92718226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77a019f8f0 con 0x7f77a0100670 2024-04-01T01:31:35.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.668+0000 7f77977fe640 1 -- 172.21.15.70:0/92718226 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77900042d0 con 0x7f77a0100670 2024-04-01T01:31:35.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.668+0000 7f77977fe640 1 -- 172.21.15.70:0/92718226 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790003b10 con 0x7f77a0100670 2024-04-01T01:31:35.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.668+0000 7f77a83a1640 1 -- 172.21.15.70:0/92718226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77a019fe90 con 0x7f77a0100670 2024-04-01T01:31:35.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.669+0000 7f77a83a1640 1 -- 172.21.15.70:0/92718226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77a01a0060 con 0x7f77a0100670 2024-04-01T01:31:35.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.670+0000 7f77977fe640 1 -- 172.21.15.70:0/92718226 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f7790046020 con 0x7f77a0100670 2024-04-01T01:31:35.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.671+0000 7f77977fe640 1 --2- 172.21.15.70:0/92718226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f778c0a24a0 0x7f778c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:35.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.671+0000 7f77a593c640 1 --2- 172.21.15.70:0/92718226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f778c0a24a0 0x7f778c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:35.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.671+0000 7f77977fe640 1 -- 172.21.15.70:0/92718226 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f77900f0a50 con 0x7f77a0100670 2024-04-01T01:31:35.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.671+0000 7f77a593c640 1 --2- 172.21.15.70:0/92718226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f778c0a24a0 0x7f778c0a4960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f77a0198f50 tx=0x7f779c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:35.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.675+0000 7f77977fe640 1 -- 172.21.15.70:0/92718226 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f779004d020 con 0x7f77a0100670 2024-04-01T01:31:35.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:35.888+0000 7f77a83a1640 1 -- 172.21.15.70:0/92718226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f77a01a02d0 con 0x7f77a0100670 2024-04-01T01:31:35.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:35 smithi189 ceph-mon[35753]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-01T01:31:35.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:35 smithi189 ceph-mon[35753]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-01T01:31:35.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:35 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/4169456651' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-01T01:31:35.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:35 smithi189 ceph-mon[35753]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-01T01:31:35.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:35 smithi189 ceph-mon[35753]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-01T01:31:35.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:35 smithi189 ceph-mon[35753]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-01T01:31:35.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:35 smithi189 ceph-mon[35753]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-01T01:31:35.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:35 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1293509173' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-01T01:31:35.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:35 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1293509173' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-01T01:31:35.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:35 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] up:active 2024-04-01T01:31:35.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:35 smithi189 ceph-mon[35753]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-01T01:31:35.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:35 smithi189 ceph-mon[35753]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-04-01T01:31:35.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:35 smithi189 ceph-mon[35753]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-04-01T01:31:35.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:35 smithi189 ceph-mon[35753]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-01T01:31:36.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:35 smithi070 ceph-mon[32179]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-01T01:31:36.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:35 smithi070 ceph-mon[32179]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-01T01:31:36.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:35 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/4169456651' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-01T01:31:36.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:35 smithi070 ceph-mon[32179]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-01T01:31:36.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:35 smithi070 ceph-mon[32179]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-01T01:31:36.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:35 smithi070 ceph-mon[32179]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-01T01:31:36.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:35 smithi070 ceph-mon[32179]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-01T01:31:36.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:35 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1293509173' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-01T01:31:36.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:35 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1293509173' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-01T01:31:36.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:35 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] up:active 2024-04-01T01:31:36.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:35 smithi070 ceph-mon[32179]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-01T01:31:36.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:35 smithi070 ceph-mon[32179]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-04-01T01:31:36.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:35 smithi070 ceph-mon[32179]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-04-01T01:31:36.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:35 smithi070 ceph-mon[32179]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-01T01:31:36.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:35 smithi081 ceph-mon[35142]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-01T01:31:36.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:35 smithi081 ceph-mon[35142]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-01T01:31:36.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:35 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/4169456651' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-01T01:31:36.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:35 smithi081 ceph-mon[35142]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-01T01:31:36.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:35 smithi081 ceph-mon[35142]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-01T01:31:36.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:35 smithi081 ceph-mon[35142]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-01T01:31:36.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:35 smithi081 ceph-mon[35142]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-01T01:31:36.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:35 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1293509173' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-01T01:31:36.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:35 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1293509173' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-01T01:31:36.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:35 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] up:active 2024-04-01T01:31:36.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:35 smithi081 ceph-mon[35142]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-01T01:31:36.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:35 smithi081 ceph-mon[35142]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-04-01T01:31:36.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:35 smithi081 ceph-mon[35142]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-04-01T01:31:36.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:35 smithi081 ceph-mon[35142]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-04-01T01:31:36.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.544+0000 7f77977fe640 1 -- 172.21.15.70:0/92718226 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f77900bd9d0 con 0x7f77a0100670 2024-04-01T01:31:36.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.546+0000 7f77a83a1640 1 -- 172.21.15.70:0/92718226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f778c0a24a0 msgr2=0x7f778c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:36.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.546+0000 7f77a83a1640 1 --2- 172.21.15.70:0/92718226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f778c0a24a0 0x7f778c0a4960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f77a0198f50 tx=0x7f779c005f90 comp rx=0 tx=0).stop 2024-04-01T01:31:36.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.547+0000 7f77a83a1640 1 -- 172.21.15.70:0/92718226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a0100670 msgr2=0x7f77a0197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:36.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.547+0000 7f77a83a1640 1 --2- 172.21.15.70:0/92718226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a0100670 0x7f77a0197a90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7790000c00 tx=0x7f7790005d50 comp rx=0 tx=0).stop 2024-04-01T01:31:36.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.547+0000 7f77a83a1640 1 -- 172.21.15.70:0/92718226 shutdown_connections 2024-04-01T01:31:36.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.547+0000 7f77a83a1640 1 --2- 172.21.15.70:0/92718226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f778c0a24a0 0x7f778c0a4960 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:36.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.547+0000 7f77a83a1640 1 --2- 172.21.15.70:0/92718226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77a0102220 0x7f77a019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:36.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.547+0000 7f77a83a1640 1 --2- 172.21.15.70:0/92718226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77a0101860 0x7f77a0197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:36.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.547+0000 7f77a83a1640 1 --2- 172.21.15.70:0/92718226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77a0100670 0x7f77a0197a90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:36.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.547+0000 7f77a83a1640 1 -- 172.21.15.70:0/92718226 >> 172.21.15.70:0/92718226 conn(0x7f77a00fbbe0 msgr2=0x7f77a00699f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:36.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.547+0000 7f77a83a1640 1 -- 172.21.15.70:0/92718226 shutdown_connections 2024-04-01T01:31:36.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.547+0000 7f77a83a1640 1 -- 172.21.15.70:0/92718226 wait complete. 2024-04-01T01:31:36.563 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-01T01:31:36.564 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-01T01:31:36.606 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:31:36 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-b[65004]: 2024-04-01T01:31:36.345+0000 7ff9913f6700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-01T01:31:36.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.676+0000 7faa8fa8f640 1 -- 172.21.15.70:0/92965054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa88102200 msgr2=0x7faa88108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.676+0000 7faa8fa8f640 1 --2- 172.21.15.70:0/92965054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa88102200 0x7faa88108a90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faa84009be0 tx=0x7faa8402bef0 comp rx=0 tx=0).stop 2024-04-01T01:31:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.676+0000 7faa8fa8f640 1 -- 172.21.15.70:0/92965054 shutdown_connections 2024-04-01T01:31:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.676+0000 7faa8fa8f640 1 --2- 172.21.15.70:0/92965054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa88102200 0x7faa88108a90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.676+0000 7faa8fa8f640 1 --2- 172.21.15.70:0/92965054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa88101840 0x7faa88101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.676+0000 7faa8fa8f640 1 --2- 172.21.15.70:0/92965054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa88100650 0x7faa88100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.676+0000 7faa8fa8f640 1 -- 172.21.15.70:0/92965054 >> 172.21.15.70:0/92965054 conn(0x7faa880fbbe0 msgr2=0x7faa880fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.676+0000 7faa8fa8f640 1 -- 172.21.15.70:0/92965054 shutdown_connections 2024-04-01T01:31:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.676+0000 7faa8fa8f640 1 -- 172.21.15.70:0/92965054 wait complete. 2024-04-01T01:31:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.676+0000 7faa8fa8f640 1 Processor -- start 2024-04-01T01:31:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8fa8f640 1 -- start start 2024-04-01T01:31:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8fa8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa88100650 0x7faa881979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8fa8f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa88101840 0x7faa88197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:36.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8fa8f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa88102200 0x7faa8819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:36.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8fa8f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faa8810c1d0 con 0x7faa88100650 2024-04-01T01:31:36.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8fa8f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faa8810c050 con 0x7faa88102200 2024-04-01T01:31:36.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8fa8f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faa8810c350 con 0x7faa88101840 2024-04-01T01:31:36.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8d82b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa88100650 0x7faa881979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:36.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8d82b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa88100650 0x7faa881979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51500/0 (socket says 172.21.15.70:51500) 2024-04-01T01:31:36.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8d82b640 1 -- 172.21.15.70:0/3581531916 learned_addr learned my addr 172.21.15.70:0/3581531916 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:36.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8d02a640 1 --2- 172.21.15.70:0/3581531916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa88101840 0x7faa88197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:36.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8e02c640 1 --2- 172.21.15.70:0/3581531916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa88102200 0x7faa8819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:36.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8d82b640 1 -- 172.21.15.70:0/3581531916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa88101840 msgr2=0x7faa88197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:36.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8d82b640 1 --2- 172.21.15.70:0/3581531916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa88101840 0x7faa88197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:36.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8d82b640 1 -- 172.21.15.70:0/3581531916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa88102200 msgr2=0x7faa8819ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:36.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8d82b640 1 --2- 172.21.15.70:0/3581531916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa88102200 0x7faa8819ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:36.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8d82b640 1 -- 172.21.15.70:0/3581531916 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8819f5a0 con 0x7faa88100650 2024-04-01T01:31:36.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8d82b640 1 --2- 172.21.15.70:0/3581531916 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa88100650 0x7faa881979d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7faa8000e9a0 tx=0x7faa8000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:36.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa7effd640 1 -- 172.21.15.70:0/3581531916 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8000cd80 con 0x7faa88100650 2024-04-01T01:31:36.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa7effd640 1 -- 172.21.15.70:0/3581531916 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa800108e0 con 0x7faa88100650 2024-04-01T01:31:36.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa8fa8f640 1 -- 172.21.15.70:0/3581531916 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa8819f890 con 0x7faa88100650 2024-04-01T01:31:36.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.677+0000 7faa7effd640 1 -- 172.21.15.70:0/3581531916 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa80018950 con 0x7faa88100650 2024-04-01T01:31:36.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.678+0000 7faa8fa8f640 1 -- 172.21.15.70:0/3581531916 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa8819fde0 con 0x7faa88100650 2024-04-01T01:31:36.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.678+0000 7faa8fa8f640 1 -- 172.21.15.70:0/3581531916 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa881a0040 con 0x7faa88100650 2024-04-01T01:31:36.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.679+0000 7faa7effd640 1 -- 172.21.15.70:0/3581531916 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7faa80010a90 con 0x7faa88100650 2024-04-01T01:31:36.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.680+0000 7faa7effd640 1 --2- 172.21.15.70:0/3581531916 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faa740a24a0 0x7faa740a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:36.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.680+0000 7faa8d02a640 1 --2- 172.21.15.70:0/3581531916 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faa740a24a0 0x7faa740a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:36.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.680+0000 7faa7effd640 1 -- 172.21.15.70:0/3581531916 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7faa800ced50 con 0x7faa88100650 2024-04-01T01:31:36.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.680+0000 7faa8d02a640 1 --2- 172.21.15.70:0/3581531916 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faa740a24a0 0x7faa740a4960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faa88198e90 tx=0x7faa78005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:36.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.684+0000 7faa7effd640 1 -- 172.21.15.70:0/3581531916 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7faa80053280 con 0x7faa88100650 2024-04-01T01:31:36.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:36.884+0000 7faa8fa8f640 1 -- 172.21.15.70:0/3581531916 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7faa880fdd70 con 0x7faa740a24a0 2024-04-01T01:31:36.889 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:36 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/92718226' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-01T01:31:36.889 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:36 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/92718226' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-01T01:31:36.889 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:36 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] up:active 2024-04-01T01:31:36.889 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:36 smithi070 ceph-mon[32179]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-01T01:31:36.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:36 smithi070 ceph-mon[32179]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-01T01:31:36.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:36 smithi070 ceph-mon[32179]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:creating} 2 up:standby-replay 6 up:standby 2024-04-01T01:31:36.890 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:36 smithi070 ceph-mon[32179]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-04-01T01:31:36.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:36 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/92718226' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-01T01:31:36.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:36 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/92718226' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-01T01:31:36.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:36 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] up:active 2024-04-01T01:31:36.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:36 smithi189 ceph-mon[35753]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-01T01:31:36.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:36 smithi189 ceph-mon[35753]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-01T01:31:36.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:36 smithi189 ceph-mon[35753]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:creating} 2 up:standby-replay 6 up:standby 2024-04-01T01:31:36.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:36 smithi189 ceph-mon[35753]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-04-01T01:31:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:36 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/92718226' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-01T01:31:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:36 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/92718226' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-01T01:31:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:36 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] up:active 2024-04-01T01:31:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:36 smithi081 ceph-mon[35142]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-01T01:31:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:36 smithi081 ceph-mon[35142]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-04-01T01:31:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:36 smithi081 ceph-mon[35142]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:creating} 2 up:standby-replay 6 up:standby 2024-04-01T01:31:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:36 smithi081 ceph-mon[35142]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-04-01T01:31:37.254 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:31:36 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:31:36.889+0000 7f37ad4b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-01T01:31:37.254 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:31:36 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:31:36.889+0000 7f37ad4b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-01T01:31:37.254 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:31:36 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:31:36.889+0000 7f37ad4b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-01T01:31:37.254 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:31:36 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:31:36.889+0000 7f37ad4b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-01T01:31:37.254 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:31:36 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:31:36.889+0000 7f37ad4b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-01T01:31:37.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.578+0000 7faa7effd640 1 -- 172.21.15.70:0/3581531916 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7faa880fdd70 con 0x7faa740a24a0 2024-04-01T01:31:37.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.581+0000 7faa8fa8f640 1 -- 172.21.15.70:0/3581531916 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faa740a24a0 msgr2=0x7faa740a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:37.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.581+0000 7faa8fa8f640 1 --2- 172.21.15.70:0/3581531916 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faa740a24a0 0x7faa740a4960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faa88198e90 tx=0x7faa78005f90 comp rx=0 tx=0).stop 2024-04-01T01:31:37.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.581+0000 7faa8fa8f640 1 -- 172.21.15.70:0/3581531916 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa88100650 msgr2=0x7faa881979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:37.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.581+0000 7faa8fa8f640 1 --2- 172.21.15.70:0/3581531916 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa88100650 0x7faa881979d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7faa8000e9a0 tx=0x7faa8000ee60 comp rx=0 tx=0).stop 2024-04-01T01:31:37.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.581+0000 7faa8fa8f640 1 -- 172.21.15.70:0/3581531916 shutdown_connections 2024-04-01T01:31:37.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.581+0000 7faa8fa8f640 1 --2- 172.21.15.70:0/3581531916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa88102200 0x7faa8819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:37.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.581+0000 7faa8fa8f640 1 --2- 172.21.15.70:0/3581531916 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faa740a24a0 0x7faa740a4960 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:37.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.581+0000 7faa8fa8f640 1 --2- 172.21.15.70:0/3581531916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa88101840 0x7faa88197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:37.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.581+0000 7faa8fa8f640 1 --2- 172.21.15.70:0/3581531916 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa88100650 0x7faa881979d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:37.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.581+0000 7faa8fa8f640 1 -- 172.21.15.70:0/3581531916 >> 172.21.15.70:0/3581531916 conn(0x7faa880fbbe0 msgr2=0x7faa880fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:37.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.581+0000 7faa8fa8f640 1 -- 172.21.15.70:0/3581531916 shutdown_connections 2024-04-01T01:31:37.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.581+0000 7faa8fa8f640 1 -- 172.21.15.70:0/3581531916 wait complete. 2024-04-01T01:31:37.592 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-01T01:31:37.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.702+0000 7f30ffb65640 1 -- 172.21.15.70:0/1589163586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30f8103310 msgr2=0x7f30f8109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:37.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.702+0000 7f30ffb65640 1 --2- 172.21.15.70:0/1589163586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30f8103310 0x7f30f8109ba0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f30f0009ca0 tx=0x7f30f002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:31:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.702+0000 7f30ffb65640 1 -- 172.21.15.70:0/1589163586 shutdown_connections 2024-04-01T01:31:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.702+0000 7f30ffb65640 1 --2- 172.21.15.70:0/1589163586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30f8103310 0x7f30f8109ba0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.702+0000 7f30ffb65640 1 --2- 172.21.15.70:0/1589163586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30f8102950 0x7f30f8102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.702+0000 7f30ffb65640 1 --2- 172.21.15.70:0/1589163586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30f8101760 0x7f30f8101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.702+0000 7f30ffb65640 1 -- 172.21.15.70:0/1589163586 >> 172.21.15.70:0/1589163586 conn(0x7f30f80fccf0 msgr2=0x7f30f80ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.702+0000 7f30ffb65640 1 -- 172.21.15.70:0/1589163586 shutdown_connections 2024-04-01T01:31:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.702+0000 7f30ffb65640 1 -- 172.21.15.70:0/1589163586 wait complete. 2024-04-01T01:31:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.702+0000 7f30ffb65640 1 Processor -- start 2024-04-01T01:31:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30ffb65640 1 -- start start 2024-04-01T01:31:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30ffb65640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30f8101760 0x7f30f8197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30ffb65640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30f8102950 0x7f30f8197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30ffb65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30f8103310 0x7f30f819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30ffb65640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f30f810d270 con 0x7f30f8103310 2024-04-01T01:31:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30ffb65640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f30f810d0f0 con 0x7f30f8102950 2024-04-01T01:31:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30ffb65640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f30f810d3f0 con 0x7f30f8101760 2024-04-01T01:31:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30fe102640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30f8103310 0x7f30f819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30fe102640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30f8103310 0x7f30f819eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51544/0 (socket says 172.21.15.70:51544) 2024-04-01T01:31:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30fe102640 1 -- 172.21.15.70:0/4051254669 learned_addr learned my addr 172.21.15.70:0/4051254669 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30fd100640 1 --2- 172.21.15.70:0/4051254669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30f8102950 0x7f30f8197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30fd901640 1 --2- 172.21.15.70:0/4051254669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30f8101760 0x7f30f8197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30fe102640 1 -- 172.21.15.70:0/4051254669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30f8101760 msgr2=0x7f30f8197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30fe102640 1 --2- 172.21.15.70:0/4051254669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30f8101760 0x7f30f8197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30fe102640 1 -- 172.21.15.70:0/4051254669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30f8102950 msgr2=0x7f30f8197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30fe102640 1 --2- 172.21.15.70:0/4051254669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30f8102950 0x7f30f8197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30fe102640 1 -- 172.21.15.70:0/4051254669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f819f550 con 0x7f30f8103310 2024-04-01T01:31:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30fe102640 1 --2- 172.21.15.70:0/4051254669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30f8103310 0x7f30f819eee0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f30f002c700 tx=0x7f30f00328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:37.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30eeffd640 1 -- 172.21.15.70:0/4051254669 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f0041020 con 0x7f30f8103310 2024-04-01T01:31:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30eeffd640 1 -- 172.21.15.70:0/4051254669 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30f0003e20 con 0x7f30f8103310 2024-04-01T01:31:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30ffb65640 1 -- 172.21.15.70:0/4051254669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30f819f7e0 con 0x7f30f8103310 2024-04-01T01:31:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30eeffd640 1 -- 172.21.15.70:0/4051254669 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f0039040 con 0x7f30f8103310 2024-04-01T01:31:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.703+0000 7f30ffb65640 1 -- 172.21.15.70:0/4051254669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30f819fcd0 con 0x7f30f8103310 2024-04-01T01:31:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.705+0000 7f30ffb65640 1 -- 172.21.15.70:0/4051254669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30f8108b50 con 0x7f30f8103310 2024-04-01T01:31:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.706+0000 7f30eeffd640 1 -- 172.21.15.70:0/4051254669 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f30f0044020 con 0x7f30f8103310 2024-04-01T01:31:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.706+0000 7f30eeffd640 1 --2- 172.21.15.70:0/4051254669 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30e40a24a0 0x7f30e40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.706+0000 7f30fd901640 1 --2- 172.21.15.70:0/4051254669 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30e40a24a0 0x7f30e40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.707+0000 7f30eeffd640 1 -- 172.21.15.70:0/4051254669 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f30f00f0d20 con 0x7f30f8103310 2024-04-01T01:31:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.707+0000 7f30fd901640 1 --2- 172.21.15.70:0/4051254669 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30e40a24a0 0x7f30e40a4960 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f30f4005fd0 tx=0x7f30f400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.711+0000 7f30eeffd640 1 -- 172.21.15.70:0/4051254669 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f30f004b020 con 0x7f30f8103310 2024-04-01T01:31:37.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.910+0000 7f30ffb65640 1 -- 172.21.15.70:0/4051254669 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f30f8061b90 con 0x7f30e40a24a0 2024-04-01T01:31:37.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.934+0000 7f30eeffd640 1 -- 172.21.15.70:0/4051254669 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f30f8061b90 con 0x7f30e40a24a0 2024-04-01T01:31:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.936+0000 7f30ffb65640 1 -- 172.21.15.70:0/4051254669 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30e40a24a0 msgr2=0x7f30e40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.936+0000 7f30ffb65640 1 --2- 172.21.15.70:0/4051254669 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30e40a24a0 0x7f30e40a4960 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f30f4005fd0 tx=0x7f30f400b410 comp rx=0 tx=0).stop 2024-04-01T01:31:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.936+0000 7f30ffb65640 1 -- 172.21.15.70:0/4051254669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30f8103310 msgr2=0x7f30f819eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.936+0000 7f30ffb65640 1 --2- 172.21.15.70:0/4051254669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30f8103310 0x7f30f819eee0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f30f002c700 tx=0x7f30f00328a0 comp rx=0 tx=0).stop 2024-04-01T01:31:37.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.936+0000 7f30ffb65640 1 -- 172.21.15.70:0/4051254669 shutdown_connections 2024-04-01T01:31:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.936+0000 7f30ffb65640 1 --2- 172.21.15.70:0/4051254669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30f8103310 0x7f30f819eee0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.936+0000 7f30ffb65640 1 --2- 172.21.15.70:0/4051254669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30f8102950 0x7f30f8197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.936+0000 7f30ffb65640 1 --2- 172.21.15.70:0/4051254669 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30e40a24a0 0x7f30e40a4960 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.936+0000 7f30ffb65640 1 --2- 172.21.15.70:0/4051254669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30f8101760 0x7f30f8197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.936+0000 7f30ffb65640 1 -- 172.21.15.70:0/4051254669 >> 172.21.15.70:0/4051254669 conn(0x7f30f80fccf0 msgr2=0x7f30f8105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.936+0000 7f30ffb65640 1 -- 172.21.15.70:0/4051254669 shutdown_connections 2024-04-01T01:31:37.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:37.936+0000 7f30ffb65640 1 -- 172.21.15.70:0/4051254669 wait complete. 2024-04-01T01:31:37.945 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:37 smithi070 ceph-mon[32179]: pgmap v298: 129 pgs: 129 active+clean; 582 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 767 B/s wr, 2 op/s 2024-04-01T01:31:37.946 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:37 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] up:active 2024-04-01T01:31:37.946 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:37 smithi070 ceph-mon[32179]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-01T01:31:37.946 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:37 smithi070 ceph-mon[32179]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-01T01:31:37.946 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:37 smithi070 ceph-mon[32179]: Cluster is now healthy 2024-04-01T01:31:37.946 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:37 smithi070 ceph-mon[32179]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:active} 2 up:standby-replay 6 up:standby 2024-04-01T01:31:37.946 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:37 smithi070 ceph-mon[32179]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-04-01T01:31:37.946 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:37 smithi070 ceph-mon[32179]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-04-01T01:31:37.947 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-01T01:31:37.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:37 smithi189 ceph-mon[35753]: pgmap v298: 129 pgs: 129 active+clean; 582 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 767 B/s wr, 2 op/s 2024-04-01T01:31:37.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:37 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] up:active 2024-04-01T01:31:37.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:37 smithi189 ceph-mon[35753]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-01T01:31:37.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:37 smithi189 ceph-mon[35753]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-01T01:31:37.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:37 smithi189 ceph-mon[35753]: Cluster is now healthy 2024-04-01T01:31:37.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:37 smithi189 ceph-mon[35753]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:active} 2 up:standby-replay 6 up:standby 2024-04-01T01:31:37.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:37 smithi189 ceph-mon[35753]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-04-01T01:31:37.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:37 smithi189 ceph-mon[35753]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-04-01T01:31:38.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.054+0000 7f0f14be2640 1 -- 172.21.15.70:0/1680202761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10100650 msgr2=0x7f0f10100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:38.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.054+0000 7f0f14be2640 1 --2- 172.21.15.70:0/1680202761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10100650 0x7f0f10100a70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009ca0 tx=0x7f0f0002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.054+0000 7f0f14be2640 1 -- 172.21.15.70:0/1680202761 shutdown_connections 2024-04-01T01:31:38.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.054+0000 7f0f14be2640 1 --2- 172.21.15.70:0/1680202761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f10102200 0x7f0f10108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.054+0000 7f0f14be2640 1 --2- 172.21.15.70:0/1680202761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f10101840 0x7f0f10101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.054+0000 7f0f14be2640 1 --2- 172.21.15.70:0/1680202761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10100650 0x7f0f10100a70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.054+0000 7f0f14be2640 1 -- 172.21.15.70:0/1680202761 >> 172.21.15.70:0/1680202761 conn(0x7f0f100fbbe0 msgr2=0x7f0f100fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:38.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.055+0000 7f0f14be2640 1 -- 172.21.15.70:0/1680202761 shutdown_connections 2024-04-01T01:31:38.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.055+0000 7f0f14be2640 1 -- 172.21.15.70:0/1680202761 wait complete. 2024-04-01T01:31:38.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.055+0000 7f0f14be2640 1 Processor -- start 2024-04-01T01:31:38.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.055+0000 7f0f14be2640 1 -- start start 2024-04-01T01:31:38.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.055+0000 7f0f14be2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f10100650 0x7f0f10069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:38.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.055+0000 7f0f14be2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10101840 0x7f0f1019c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:38.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.055+0000 7f0f14be2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f10102200 0x7f0f1019ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:38.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.055+0000 7f0f14be2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1010c180 con 0x7f0f10101840 2024-04-01T01:31:38.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.055+0000 7f0f14be2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1010c000 con 0x7f0f10102200 2024-04-01T01:31:38.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.055+0000 7f0f14be2640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1010c300 con 0x7f0f10100650 2024-04-01T01:31:38.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.055+0000 7f0f0dd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10101840 0x7f0f1019c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:38.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.055+0000 7f0f0dd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10101840 0x7f0f1019c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51556/0 (socket says 172.21.15.70:51556) 2024-04-01T01:31:38.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.055+0000 7f0f0dd9b640 1 -- 172.21.15.70:0/3914399032 learned_addr learned my addr 172.21.15.70:0/3914399032 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:38.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.056+0000 7f0f0ed9d640 1 --2- 172.21.15.70:0/3914399032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f10102200 0x7f0f1019ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:38.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.056+0000 7f0f0e59c640 1 --2- 172.21.15.70:0/3914399032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f10100650 0x7f0f10069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:38.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.056+0000 7f0f0dd9b640 1 -- 172.21.15.70:0/3914399032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f10100650 msgr2=0x7f0f10069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:38.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.056+0000 7f0f0dd9b640 1 --2- 172.21.15.70:0/3914399032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f10100650 0x7f0f10069440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.056+0000 7f0f0dd9b640 1 -- 172.21.15.70:0/3914399032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f10102200 msgr2=0x7f0f1019ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:38.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.056+0000 7f0f0dd9b640 1 --2- 172.21.15.70:0/3914399032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f10102200 0x7f0f1019ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.056+0000 7f0f0dd9b640 1 -- 172.21.15.70:0/3914399032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1019f630 con 0x7f0f10101840 2024-04-01T01:31:38.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.056+0000 7f0f0dd9b640 1 --2- 172.21.15.70:0/3914399032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10101840 0x7f0f1019c730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400e9f0 tx=0x7f0f0400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:38.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.056+0000 7f0eff7fe640 1 -- 172.21.15.70:0/3914399032 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0400cd80 con 0x7f0f10101840 2024-04-01T01:31:38.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.056+0000 7f0eff7fe640 1 -- 172.21.15.70:0/3914399032 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0f040108e0 con 0x7f0f10101840 2024-04-01T01:31:38.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.056+0000 7f0f14be2640 1 -- 172.21.15.70:0/3914399032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f1019f920 con 0x7f0f10101840 2024-04-01T01:31:38.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.056+0000 7f0eff7fe640 1 -- 172.21.15.70:0/3914399032 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f04018920 con 0x7f0f10101840 2024-04-01T01:31:38.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.056+0000 7f0f14be2640 1 -- 172.21.15.70:0/3914399032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f1019fe70 con 0x7f0f10101840 2024-04-01T01:31:38.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.057+0000 7f0f14be2640 1 -- 172.21.15.70:0/3914399032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f10107a40 con 0x7f0f10101840 2024-04-01T01:31:38.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.058+0000 7f0eff7fe640 1 -- 172.21.15.70:0/3914399032 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f0f04010a90 con 0x7f0f10101840 2024-04-01T01:31:38.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.059+0000 7f0eff7fe640 1 --2- 172.21.15.70:0/3914399032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ef80a24a0 0x7f0ef80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:38.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.059+0000 7f0f0e59c640 1 --2- 172.21.15.70:0/3914399032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ef80a24a0 0x7f0ef80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:38.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.059+0000 7f0eff7fe640 1 -- 172.21.15.70:0/3914399032 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0f040cec30 con 0x7f0f10101840 2024-04-01T01:31:38.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.059+0000 7f0f0e59c640 1 --2- 172.21.15.70:0/3914399032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ef80a24a0 0x7f0ef80a4960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0f00032380 tx=0x7f0f00000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:38.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.063+0000 7f0eff7fe640 1 -- 172.21.15.70:0/3914399032 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0f0409b560 con 0x7f0f10101840 2024-04-01T01:31:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:37 smithi081 ceph-mon[35142]: pgmap v298: 129 pgs: 129 active+clean; 582 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 767 B/s wr, 2 op/s 2024-04-01T01:31:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:37 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] up:active 2024-04-01T01:31:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:37 smithi081 ceph-mon[35142]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-01T01:31:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:37 smithi081 ceph-mon[35142]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-01T01:31:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:37 smithi081 ceph-mon[35142]: Cluster is now healthy 2024-04-01T01:31:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:37 smithi081 ceph-mon[35142]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:active} 2 up:standby-replay 6 up:standby 2024-04-01T01:31:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:37 smithi081 ceph-mon[35142]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-04-01T01:31:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:37 smithi081 ceph-mon[35142]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-04-01T01:31:38.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.263+0000 7f0f14be2640 1 -- 172.21.15.70:0/3914399032 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f0f10061c20 con 0x7f0ef80a24a0 2024-04-01T01:31:38.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.269+0000 7f0eff7fe640 1 -- 172.21.15.70:0/3914399032 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f0f10061c20 con 0x7f0ef80a24a0 2024-04-01T01:31:38.269 INFO:teuthology.orchestra.run.smithi070.stdout:/volumes/qa/sv_0/79d38c5c-0128-46fd-a550-44920d4a703c 2024-04-01T01:31:38.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.271+0000 7f0f14be2640 1 -- 172.21.15.70:0/3914399032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ef80a24a0 msgr2=0x7f0ef80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:38.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.271+0000 7f0f14be2640 1 --2- 172.21.15.70:0/3914399032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ef80a24a0 0x7f0ef80a4960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0f00032380 tx=0x7f0f00000f40 comp rx=0 tx=0).stop 2024-04-01T01:31:38.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.271+0000 7f0f14be2640 1 -- 172.21.15.70:0/3914399032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10101840 msgr2=0x7f0f1019c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:38.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.271+0000 7f0f14be2640 1 --2- 172.21.15.70:0/3914399032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10101840 0x7f0f1019c730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400e9f0 tx=0x7f0f0400eeb0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.271+0000 7f0f14be2640 1 -- 172.21.15.70:0/3914399032 shutdown_connections 2024-04-01T01:31:38.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.271+0000 7f0f14be2640 1 --2- 172.21.15.70:0/3914399032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f10102200 0x7f0f1019ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.271+0000 7f0f14be2640 1 --2- 172.21.15.70:0/3914399032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f10101840 0x7f0f1019c730 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.271+0000 7f0f14be2640 1 --2- 172.21.15.70:0/3914399032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ef80a24a0 0x7f0ef80a4960 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.271+0000 7f0f14be2640 1 --2- 172.21.15.70:0/3914399032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f10100650 0x7f0f10069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.271+0000 7f0f14be2640 1 -- 172.21.15.70:0/3914399032 >> 172.21.15.70:0/3914399032 conn(0x7f0f100fbbe0 msgr2=0x7f0f10104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:38.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.271+0000 7f0f14be2640 1 -- 172.21.15.70:0/3914399032 shutdown_connections 2024-04-01T01:31:38.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.271+0000 7f0f14be2640 1 -- 172.21.15.70:0/3914399032 wait complete. 2024-04-01T01:31:38.281 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-01T01:31:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.388+0000 7ffb1853a640 1 -- 172.21.15.70:0/2841311084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb10102250 msgr2=0x7ffb10104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.388+0000 7ffb1853a640 1 --2- 172.21.15.70:0/2841311084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb10102250 0x7ffb10104640 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009900 tx=0x7ffb0c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.388+0000 7ffb1853a640 1 -- 172.21.15.70:0/2841311084 shutdown_connections 2024-04-01T01:31:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.388+0000 7ffb1853a640 1 --2- 172.21.15.70:0/2841311084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb101074b0 0x7ffb101098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.388+0000 7ffb1853a640 1 --2- 172.21.15.70:0/2841311084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb10104b80 0x7ffb10106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.388+0000 7ffb1853a640 1 --2- 172.21.15.70:0/2841311084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb10102250 0x7ffb10104640 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.388+0000 7ffb1853a640 1 -- 172.21.15.70:0/2841311084 >> 172.21.15.70:0/2841311084 conn(0x7ffb100fbbe0 msgr2=0x7ffb100fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.388+0000 7ffb1853a640 1 -- 172.21.15.70:0/2841311084 shutdown_connections 2024-04-01T01:31:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.388+0000 7ffb1853a640 1 -- 172.21.15.70:0/2841311084 wait complete. 2024-04-01T01:31:38.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.389+0000 7ffb1853a640 1 Processor -- start 2024-04-01T01:31:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.389+0000 7ffb1853a640 1 -- start start 2024-04-01T01:31:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.389+0000 7ffb1853a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb10102250 0x7ffb10197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.389+0000 7ffb1853a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb10104b80 0x7ffb10197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.389+0000 7ffb1853a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb101074b0 0x7ffb1019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.389+0000 7ffb1853a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1010d2a0 con 0x7ffb101074b0 2024-04-01T01:31:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.389+0000 7ffb1853a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1010d120 con 0x7ffb10104b80 2024-04-01T01:31:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.389+0000 7ffb1853a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffb1010d420 con 0x7ffb10102250 2024-04-01T01:31:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.389+0000 7ffb16ad7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb101074b0 0x7ffb1019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.389+0000 7ffb15ad5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb10104b80 0x7ffb10197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.389+0000 7ffb15ad5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb10104b80 0x7ffb10197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:50228/0 (socket says 172.21.15.70:50228) 2024-04-01T01:31:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.389+0000 7ffb15ad5640 1 -- 172.21.15.70:0/3850227437 learned_addr learned my addr 172.21.15.70:0/3850227437 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.389+0000 7ffb16ad7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb101074b0 0x7ffb1019efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51572/0 (socket says 172.21.15.70:51572) 2024-04-01T01:31:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.389+0000 7ffb162d6640 1 --2- 172.21.15.70:0/3850227437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb10102250 0x7ffb10197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:38.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.390+0000 7ffb15ad5640 1 -- 172.21.15.70:0/3850227437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb10102250 msgr2=0x7ffb10197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:38.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.390+0000 7ffb15ad5640 1 --2- 172.21.15.70:0/3850227437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb10102250 0x7ffb10197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.390+0000 7ffb15ad5640 1 -- 172.21.15.70:0/3850227437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb101074b0 msgr2=0x7ffb1019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:38.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.390+0000 7ffb15ad5640 1 --2- 172.21.15.70:0/3850227437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb101074b0 0x7ffb1019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.390+0000 7ffb15ad5640 1 -- 172.21.15.70:0/3850227437 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb1019f630 con 0x7ffb10104b80 2024-04-01T01:31:38.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.390+0000 7ffb16ad7640 1 --2- 172.21.15.70:0/3850227437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb101074b0 0x7ffb1019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:38.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.390+0000 7ffb15ad5640 1 --2- 172.21.15.70:0/3850227437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb10104b80 0x7ffb10197f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000e9c0 tx=0x7ffb0000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:38.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.390+0000 7ffb077fe640 1 -- 172.21.15.70:0/3850227437 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb0000cd80 con 0x7ffb10104b80 2024-04-01T01:31:38.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.390+0000 7ffb1853a640 1 -- 172.21.15.70:0/3850227437 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb1019f920 con 0x7ffb10104b80 2024-04-01T01:31:38.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.390+0000 7ffb077fe640 1 -- 172.21.15.70:0/3850227437 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffb000054d0 con 0x7ffb10104b80 2024-04-01T01:31:38.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.390+0000 7ffb1853a640 1 -- 172.21.15.70:0/3850227437 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb1019fec0 con 0x7ffb10104b80 2024-04-01T01:31:38.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.390+0000 7ffb077fe640 1 -- 172.21.15.70:0/3850227437 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb00010430 con 0x7ffb10104b80 2024-04-01T01:31:38.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.391+0000 7ffb1853a640 1 -- 172.21.15.70:0/3850227437 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb100fd6d0 con 0x7ffb10104b80 2024-04-01T01:31:38.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.392+0000 7ffb077fe640 1 -- 172.21.15.70:0/3850227437 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7ffb00005020 con 0x7ffb10104b80 2024-04-01T01:31:38.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.393+0000 7ffb077fe640 1 --2- 172.21.15.70:0/3850227437 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ffafc0a24a0 0x7ffafc0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:38.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.393+0000 7ffb162d6640 1 --2- 172.21.15.70:0/3850227437 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ffafc0a24a0 0x7ffafc0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:38.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.393+0000 7ffb077fe640 1 -- 172.21.15.70:0/3850227437 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ffb000cef90 con 0x7ffb10104b80 2024-04-01T01:31:38.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.393+0000 7ffb162d6640 1 --2- 172.21.15.70:0/3850227437 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ffafc0a24a0 0x7ffafc0a4960 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009230 tx=0x7ffb0c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:38.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.401+0000 7ffb077fe640 1 -- 172.21.15.70:0/3850227437 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ffb00053280 con 0x7ffb10104b80 2024-04-01T01:31:38.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.602+0000 7ffb1853a640 1 -- 172.21.15.70:0/3850227437 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ffb10061b90 con 0x7ffafc0a24a0 2024-04-01T01:31:38.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.631+0000 7ffb077fe640 1 -- 172.21.15.70:0/3850227437 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ffb10061b90 con 0x7ffafc0a24a0 2024-04-01T01:31:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.633+0000 7ffb1853a640 1 -- 172.21.15.70:0/3850227437 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ffafc0a24a0 msgr2=0x7ffafc0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.633+0000 7ffb1853a640 1 --2- 172.21.15.70:0/3850227437 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ffafc0a24a0 0x7ffafc0a4960 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009230 tx=0x7ffb0c005fd0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.633+0000 7ffb1853a640 1 -- 172.21.15.70:0/3850227437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb10104b80 msgr2=0x7ffb10197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.633+0000 7ffb1853a640 1 --2- 172.21.15.70:0/3850227437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb10104b80 0x7ffb10197f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000e9c0 tx=0x7ffb0000ee80 comp rx=0 tx=0).stop 2024-04-01T01:31:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.633+0000 7ffb1853a640 1 -- 172.21.15.70:0/3850227437 shutdown_connections 2024-04-01T01:31:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.633+0000 7ffb1853a640 1 --2- 172.21.15.70:0/3850227437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffb101074b0 0x7ffb1019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.633+0000 7ffb1853a640 1 --2- 172.21.15.70:0/3850227437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb10104b80 0x7ffb10197f50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.633+0000 7ffb1853a640 1 --2- 172.21.15.70:0/3850227437 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ffafc0a24a0 0x7ffafc0a4960 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.633+0000 7ffb1853a640 1 --2- 172.21.15.70:0/3850227437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb10102250 0x7ffb10197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.633+0000 7ffb1853a640 1 -- 172.21.15.70:0/3850227437 >> 172.21.15.70:0/3850227437 conn(0x7ffb100fbbe0 msgr2=0x7ffb10102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:38.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.633+0000 7ffb1853a640 1 -- 172.21.15.70:0/3850227437 shutdown_connections 2024-04-01T01:31:38.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.633+0000 7ffb1853a640 1 -- 172.21.15.70:0/3850227437 wait complete. 2024-04-01T01:31:38.643 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-01T01:31:38.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.750+0000 7f06c625c640 1 -- 172.21.15.70:0/12530458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c0101780 msgr2=0x7f06c0101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:38.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.750+0000 7f06c625c640 1 --2- 172.21.15.70:0/12530458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c0101780 0x7f06c0101ba0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f06b0009900 tx=0x7f06b002bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:38.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.750+0000 7f06c625c640 1 -- 172.21.15.70:0/12530458 shutdown_connections 2024-04-01T01:31:38.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.750+0000 7f06c625c640 1 --2- 172.21.15.70:0/12530458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c0103330 0x7f06c0109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.750+0000 7f06c625c640 1 --2- 172.21.15.70:0/12530458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c0102970 0x7f06c0102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.750+0000 7f06c625c640 1 --2- 172.21.15.70:0/12530458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c0101780 0x7f06c0101ba0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.750+0000 7f06c625c640 1 -- 172.21.15.70:0/12530458 >> 172.21.15.70:0/12530458 conn(0x7f06c00fccf0 msgr2=0x7f06c00ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:38.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.751+0000 7f06c625c640 1 -- 172.21.15.70:0/12530458 shutdown_connections 2024-04-01T01:31:38.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.751+0000 7f06c625c640 1 -- 172.21.15.70:0/12530458 wait complete. 2024-04-01T01:31:38.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.751+0000 7f06c625c640 1 Processor -- start 2024-04-01T01:31:38.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.751+0000 7f06c625c640 1 -- start start 2024-04-01T01:31:38.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.751+0000 7f06c625c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c0101780 0x7f06c0197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:38.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.751+0000 7f06c625c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c0102970 0x7f06c0197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:38.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.751+0000 7f06c625c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c0103330 0x7f06c019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:38.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.751+0000 7f06c625c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f06c010d280 con 0x7f06c0101780 2024-04-01T01:31:38.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.751+0000 7f06c625c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f06c010d100 con 0x7f06c0103330 2024-04-01T01:31:38.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.751+0000 7f06c625c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f06c010d400 con 0x7f06c0102970 2024-04-01T01:31:38.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.751+0000 7f06bf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c0101780 0x7f06c0197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:38.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.752+0000 7f06bf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c0101780 0x7f06c0197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51582/0 (socket says 172.21.15.70:51582) 2024-04-01T01:31:38.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.752+0000 7f06bf7fe640 1 -- 172.21.15.70:0/2144769251 learned_addr learned my addr 172.21.15.70:0/2144769251 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:38.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.752+0000 7f06bffff640 1 --2- 172.21.15.70:0/2144769251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c0103330 0x7f06c019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:38.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.752+0000 7f06b6ffd640 1 --2- 172.21.15.70:0/2144769251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c0102970 0x7f06c0197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:38.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.752+0000 7f06bf7fe640 1 -- 172.21.15.70:0/2144769251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c0102970 msgr2=0x7f06c0197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:38.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.752+0000 7f06bf7fe640 1 --2- 172.21.15.70:0/2144769251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c0102970 0x7f06c0197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.752+0000 7f06bf7fe640 1 -- 172.21.15.70:0/2144769251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c0103330 msgr2=0x7f06c019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:38.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.752+0000 7f06bf7fe640 1 --2- 172.21.15.70:0/2144769251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c0103330 0x7f06c019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.752+0000 7f06bf7fe640 1 -- 172.21.15.70:0/2144769251 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c019f630 con 0x7f06c0101780 2024-04-01T01:31:38.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.752+0000 7f06bf7fe640 1 --2- 172.21.15.70:0/2144769251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c0101780 0x7f06c0197a10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f06b0000c00 tx=0x7f06b0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:38.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.752+0000 7f06bd7fa640 1 -- 172.21.15.70:0/2144769251 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b0043020 con 0x7f06c0101780 2024-04-01T01:31:38.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.752+0000 7f06bd7fa640 1 -- 172.21.15.70:0/2144769251 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06b0004440 con 0x7f06c0101780 2024-04-01T01:31:38.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.752+0000 7f06c625c640 1 -- 172.21.15.70:0/2144769251 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06c019f8c0 con 0x7f06c0101780 2024-04-01T01:31:38.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.752+0000 7f06bd7fa640 1 -- 172.21.15.70:0/2144769251 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b0003c90 con 0x7f06c0101780 2024-04-01T01:31:38.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.752+0000 7f06c625c640 1 -- 172.21.15.70:0/2144769251 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06c019fe60 con 0x7f06c0101780 2024-04-01T01:31:38.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.753+0000 7f06c625c640 1 -- 172.21.15.70:0/2144769251 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06c0108bb0 con 0x7f06c0101780 2024-04-01T01:31:38.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.754+0000 7f06bd7fa640 1 -- 172.21.15.70:0/2144769251 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f06b0046020 con 0x7f06c0101780 2024-04-01T01:31:38.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.755+0000 7f06bd7fa640 1 --2- 172.21.15.70:0/2144769251 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06ac0a2520 0x7f06ac0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:38.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.755+0000 7f06b6ffd640 1 --2- 172.21.15.70:0/2144769251 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06ac0a2520 0x7f06ac0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:38.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.755+0000 7f06bd7fa640 1 -- 172.21.15.70:0/2144769251 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f06b00f0e90 con 0x7f06c0101780 2024-04-01T01:31:38.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.755+0000 7f06b6ffd640 1 --2- 172.21.15.70:0/2144769251 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06ac0a2520 0x7f06ac0a49e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f06c0198f20 tx=0x7f06a8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:38.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.758+0000 7f06bd7fa640 1 -- 172.21.15.70:0/2144769251 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f06b004d020 con 0x7f06c0101780 2024-04-01T01:31:38.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.961+0000 7f06c625c640 1 -- 172.21.15.70:0/2144769251 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f06c0061c20 con 0x7f06ac0a2520 2024-04-01T01:31:38.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.967+0000 7f06bd7fa640 1 -- 172.21.15.70:0/2144769251 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f06c0061c20 con 0x7f06ac0a2520 2024-04-01T01:31:38.967 INFO:teuthology.orchestra.run.smithi070.stdout:/volumes/qa/sv_1/9d5a19a7-6830-427d-8fc8-c531ebf6557b 2024-04-01T01:31:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.969+0000 7f06c625c640 1 -- 172.21.15.70:0/2144769251 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06ac0a2520 msgr2=0x7f06ac0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.969+0000 7f06c625c640 1 --2- 172.21.15.70:0/2144769251 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06ac0a2520 0x7f06ac0a49e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f06c0198f20 tx=0x7f06a8005f90 comp rx=0 tx=0).stop 2024-04-01T01:31:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.969+0000 7f06c625c640 1 -- 172.21.15.70:0/2144769251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c0101780 msgr2=0x7f06c0197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.969+0000 7f06c625c640 1 --2- 172.21.15.70:0/2144769251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c0101780 0x7f06c0197a10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f06b0000c00 tx=0x7f06b0004040 comp rx=0 tx=0).stop 2024-04-01T01:31:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.969+0000 7f06c625c640 1 -- 172.21.15.70:0/2144769251 shutdown_connections 2024-04-01T01:31:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.969+0000 7f06c625c640 1 --2- 172.21.15.70:0/2144769251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c0103330 0x7f06c019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.969+0000 7f06c625c640 1 --2- 172.21.15.70:0/2144769251 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06ac0a2520 0x7f06ac0a49e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.969+0000 7f06c625c640 1 --2- 172.21.15.70:0/2144769251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c0102970 0x7f06c0197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.969+0000 7f06c625c640 1 --2- 172.21.15.70:0/2144769251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c0101780 0x7f06c0197a10 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:38.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.969+0000 7f06c625c640 1 -- 172.21.15.70:0/2144769251 >> 172.21.15.70:0/2144769251 conn(0x7f06c00fccf0 msgr2=0x7f06c0105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:38.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.970+0000 7f06c625c640 1 -- 172.21.15.70:0/2144769251 shutdown_connections 2024-04-01T01:31:38.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:38.970+0000 7f06c625c640 1 -- 172.21.15.70:0/2144769251 wait complete. 2024-04-01T01:31:38.978 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:38 smithi070 ceph-mon[32179]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:31:38.979 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:38 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] up:active 2024-04-01T01:31:38.979 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:38 smithi070 ceph-mon[32179]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-04-01T01:31:38.979 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:38 smithi070 ceph-mon[32179]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-01T01:31:38.980 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:31:38.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:38 smithi189 ceph-mon[35753]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:31:38.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:38 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] up:active 2024-04-01T01:31:38.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:38 smithi189 ceph-mon[35753]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-04-01T01:31:38.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:38 smithi189 ceph-mon[35753]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-01T01:31:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:38 smithi081 ceph-mon[35142]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:31:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:38 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] up:active 2024-04-01T01:31:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:38 smithi081 ceph-mon[35142]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-04-01T01:31:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:38 smithi081 ceph-mon[35142]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-01T01:31:39.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.083+0000 7fca32b43640 1 -- 172.21.15.70:0/1159716565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca2c101770 msgr2=0x7fca2c101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:39.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.083+0000 7fca32b43640 1 --2- 172.21.15.70:0/1159716565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca2c101770 0x7fca2c101b90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fca20009900 tx=0x7fca2002bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:39.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.083+0000 7fca32b43640 1 -- 172.21.15.70:0/1159716565 shutdown_connections 2024-04-01T01:31:39.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.083+0000 7fca32b43640 1 --2- 172.21.15.70:0/1159716565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca2c103320 0x7fca2c109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.083+0000 7fca32b43640 1 --2- 172.21.15.70:0/1159716565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca2c102960 0x7fca2c102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.083+0000 7fca32b43640 1 --2- 172.21.15.70:0/1159716565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca2c101770 0x7fca2c101b90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.083+0000 7fca32b43640 1 -- 172.21.15.70:0/1159716565 >> 172.21.15.70:0/1159716565 conn(0x7fca2c0fcd00 msgr2=0x7fca2c0ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:39.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.083+0000 7fca32b43640 1 -- 172.21.15.70:0/1159716565 shutdown_connections 2024-04-01T01:31:39.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.083+0000 7fca32b43640 1 -- 172.21.15.70:0/1159716565 wait complete. 2024-04-01T01:31:39.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.083+0000 7fca32b43640 1 Processor -- start 2024-04-01T01:31:39.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca32b43640 1 -- start start 2024-04-01T01:31:39.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca32b43640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca2c101770 0x7fca2c1979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:39.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca32b43640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca2c102960 0x7fca2c197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:39.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca32b43640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca2c103320 0x7fca2c19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:39.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca32b43640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c10d2c0 con 0x7fca2c101770 2024-04-01T01:31:39.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca32b43640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c10d140 con 0x7fca2c102960 2024-04-01T01:31:39.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca32b43640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fca2c10d440 con 0x7fca2c103320 2024-04-01T01:31:39.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca308df640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca2c101770 0x7fca2c1979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:39.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca308df640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca2c101770 0x7fca2c1979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51604/0 (socket says 172.21.15.70:51604) 2024-04-01T01:31:39.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca308df640 1 -- 172.21.15.70:0/2170485849 learned_addr learned my addr 172.21.15.70:0/2170485849 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:39.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca310e0640 1 --2- 172.21.15.70:0/2170485849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca2c103320 0x7fca2c19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:39.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca2bfff640 1 --2- 172.21.15.70:0/2170485849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca2c102960 0x7fca2c197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:39.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca308df640 1 -- 172.21.15.70:0/2170485849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca2c103320 msgr2=0x7fca2c19ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:39.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca308df640 1 --2- 172.21.15.70:0/2170485849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca2c103320 0x7fca2c19ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca308df640 1 -- 172.21.15.70:0/2170485849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca2c102960 msgr2=0x7fca2c197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:39.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca308df640 1 --2- 172.21.15.70:0/2170485849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca2c102960 0x7fca2c197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca308df640 1 -- 172.21.15.70:0/2170485849 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca2c19f5e0 con 0x7fca2c101770 2024-04-01T01:31:39.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca2bfff640 1 --2- 172.21.15.70:0/2170485849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca2c102960 0x7fca2c197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:31:39.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca308df640 1 --2- 172.21.15.70:0/2170485849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca2c101770 0x7fca2c1979c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fca20000c00 tx=0x7fca20005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:39.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca29ffb640 1 -- 172.21.15.70:0/2170485849 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca20043020 con 0x7fca2c101770 2024-04-01T01:31:39.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca29ffb640 1 -- 172.21.15.70:0/2170485849 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca200043d0 con 0x7fca2c101770 2024-04-01T01:31:39.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca32b43640 1 -- 172.21.15.70:0/2170485849 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca2c19f870 con 0x7fca2c101770 2024-04-01T01:31:39.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.084+0000 7fca29ffb640 1 -- 172.21.15.70:0/2170485849 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca20003ab0 con 0x7fca2c101770 2024-04-01T01:31:39.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.085+0000 7fca32b43640 1 -- 172.21.15.70:0/2170485849 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca2c19fe10 con 0x7fca2c101770 2024-04-01T01:31:39.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.085+0000 7fca32b43640 1 -- 172.21.15.70:0/2170485849 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca2c108b60 con 0x7fca2c101770 2024-04-01T01:31:39.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.086+0000 7fca29ffb640 1 -- 172.21.15.70:0/2170485849 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7fca20046020 con 0x7fca2c101770 2024-04-01T01:31:39.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.087+0000 7fca29ffb640 1 --2- 172.21.15.70:0/2170485849 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fca1c0a2520 0x7fca1c0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:39.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.087+0000 7fca2bfff640 1 --2- 172.21.15.70:0/2170485849 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fca1c0a2520 0x7fca1c0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:39.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.087+0000 7fca29ffb640 1 -- 172.21.15.70:0/2170485849 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fca200f0ac0 con 0x7fca2c101770 2024-04-01T01:31:39.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.088+0000 7fca2bfff640 1 --2- 172.21.15.70:0/2170485849 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fca1c0a2520 0x7fca1c0a49e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fca2c198ed0 tx=0x7fca18005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:39.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.092+0000 7fca29ffb640 1 -- 172.21.15.70:0/2170485849 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fca2004d020 con 0x7fca2c101770 2024-04-01T01:31:39.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.302+0000 7fca32b43640 1 -- 172.21.15.70:0/2170485849 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca2c02d170 con 0x7fca2c101770 2024-04-01T01:31:39.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.302+0000 7fca29ffb640 1 -- 172.21.15.70:0/2170485849 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11929 (secure 0 0 0) 0x7fca2003c350 con 0x7fca2c101770 2024-04-01T01:31:39.302 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:39.302 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:31:38.563210+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24451,24464,24379,24404,24418]},"id":1}]} 2024-04-01T01:31:39.302 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 28 2024-04-01T01:31:39.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.304+0000 7fca32b43640 1 -- 172.21.15.70:0/2170485849 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fca1c0a2520 msgr2=0x7fca1c0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:39.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.304+0000 7fca32b43640 1 --2- 172.21.15.70:0/2170485849 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fca1c0a2520 0x7fca1c0a49e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fca2c198ed0 tx=0x7fca18005f90 comp rx=0 tx=0).stop 2024-04-01T01:31:39.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.304+0000 7fca32b43640 1 -- 172.21.15.70:0/2170485849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca2c101770 msgr2=0x7fca2c1979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:39.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.304+0000 7fca32b43640 1 --2- 172.21.15.70:0/2170485849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca2c101770 0x7fca2c1979c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fca20000c00 tx=0x7fca20005e60 comp rx=0 tx=0).stop 2024-04-01T01:31:39.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.305+0000 7fca32b43640 1 -- 172.21.15.70:0/2170485849 shutdown_connections 2024-04-01T01:31:39.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.305+0000 7fca32b43640 1 --2- 172.21.15.70:0/2170485849 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fca1c0a2520 0x7fca1c0a49e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.305+0000 7fca32b43640 1 --2- 172.21.15.70:0/2170485849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca2c103320 0x7fca2c19ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.305+0000 7fca32b43640 1 --2- 172.21.15.70:0/2170485849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca2c102960 0x7fca2c197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.305+0000 7fca32b43640 1 --2- 172.21.15.70:0/2170485849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca2c101770 0x7fca2c1979c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.305+0000 7fca32b43640 1 -- 172.21.15.70:0/2170485849 >> 172.21.15.70:0/2170485849 conn(0x7fca2c0fcd00 msgr2=0x7fca2c105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:39.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.305+0000 7fca32b43640 1 -- 172.21.15.70:0/2170485849 shutdown_connections 2024-04-01T01:31:39.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.305+0000 7fca32b43640 1 -- 172.21.15.70:0/2170485849 wait complete. 2024-04-01T01:31:39.315 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:31:39.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.419+0000 7f3909af4640 1 -- 172.21.15.70:0/3884492655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3904104b20 msgr2=0x7f3904106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:39.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.419+0000 7f3909af4640 1 --2- 172.21.15.70:0/3884492655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3904104b20 0x7f3904106f10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f38ec009900 tx=0x7f38ec02bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:39.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.419+0000 7f3909af4640 1 -- 172.21.15.70:0/3884492655 shutdown_connections 2024-04-01T01:31:39.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.419+0000 7f3909af4640 1 --2- 172.21.15.70:0/3884492655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3904107450 0x7f39041098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.419+0000 7f3909af4640 1 --2- 172.21.15.70:0/3884492655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3904104b20 0x7f3904106f10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.419+0000 7f3909af4640 1 --2- 172.21.15.70:0/3884492655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39041021f0 0x7f39041045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.419+0000 7f3909af4640 1 -- 172.21.15.70:0/3884492655 >> 172.21.15.70:0/3884492655 conn(0x7f39040fbbe0 msgr2=0x7f39040fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:39.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.419+0000 7f3909af4640 1 -- 172.21.15.70:0/3884492655 shutdown_connections 2024-04-01T01:31:39.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.419+0000 7f3909af4640 1 -- 172.21.15.70:0/3884492655 wait complete. 2024-04-01T01:31:39.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.419+0000 7f3909af4640 1 Processor -- start 2024-04-01T01:31:39.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.419+0000 7f3909af4640 1 -- start start 2024-04-01T01:31:39.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3909af4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39041021f0 0x7f39041979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:39.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3909af4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3904104b20 0x7f3904197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:39.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3909af4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3904107450 0x7f390419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:39.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3909af4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f390410d2b0 con 0x7f3904107450 2024-04-01T01:31:39.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3909af4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f390410d130 con 0x7f3904104b20 2024-04-01T01:31:39.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3909af4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f390410d430 con 0x7f39041021f0 2024-04-01T01:31:39.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3903fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3904107450 0x7f390419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:39.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f38faffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3904104b20 0x7f3904197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:39.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3903fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3904107450 0x7f390419ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51634/0 (socket says 172.21.15.70:51634) 2024-04-01T01:31:39.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3903fff640 1 -- 172.21.15.70:0/264341845 learned_addr learned my addr 172.21.15.70:0/264341845 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f39037fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39041021f0 0x7f39041979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3903fff640 1 -- 172.21.15.70:0/264341845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39041021f0 msgr2=0x7f39041979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3903fff640 1 --2- 172.21.15.70:0/264341845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39041021f0 0x7f39041979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3903fff640 1 -- 172.21.15.70:0/264341845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3904104b20 msgr2=0x7f3904197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3903fff640 1 --2- 172.21.15.70:0/264341845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3904104b20 0x7f3904197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3903fff640 1 -- 172.21.15.70:0/264341845 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390419f5d0 con 0x7f3904107450 2024-04-01T01:31:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f38faffd640 1 --2- 172.21.15.70:0/264341845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3904104b20 0x7f3904197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3903fff640 1 --2- 172.21.15.70:0/264341845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3904107450 0x7f390419ef60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00eb10 tx=0x7f38fc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f39017fa640 1 -- 172.21.15.70:0/264341845 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc004b20 con 0x7f3904107450 2024-04-01T01:31:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3909af4640 1 -- 172.21.15.70:0/264341845 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f390419f8c0 con 0x7f3904107450 2024-04-01T01:31:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f39017fa640 1 -- 172.21.15.70:0/264341845 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f38fc004cc0 con 0x7f3904107450 2024-04-01T01:31:39.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f39017fa640 1 -- 172.21.15.70:0/264341845 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc010690 con 0x7f3904107450 2024-04-01T01:31:39.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.420+0000 7f3909af4640 1 -- 172.21.15.70:0/264341845 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f390419fde0 con 0x7f3904107450 2024-04-01T01:31:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.421+0000 7f3909af4640 1 -- 172.21.15.70:0/264341845 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39040fd6d0 con 0x7f3904107450 2024-04-01T01:31:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.422+0000 7f39017fa640 1 -- 172.21.15.70:0/264341845 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 137320+0+0 (secure 0 0 0) 0x7f38fc005020 con 0x7f3904107450 2024-04-01T01:31:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.423+0000 7f39017fa640 1 --2- 172.21.15.70:0/264341845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f38e40a24a0 0x7f38e40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.423+0000 7f39017fa640 1 -- 172.21.15.70:0/264341845 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f38fc0ceef0 con 0x7f3904107450 2024-04-01T01:31:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.423+0000 7f39037fe640 1 --2- 172.21.15.70:0/264341845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f38e40a24a0 0x7f38e40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.423+0000 7f39037fe640 1 --2- 172.21.15.70:0/264341845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f38e40a24a0 0x7f38e40a4960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f38f4009bd0 tx=0x7f38f4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:39.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.427+0000 7f39017fa640 1 -- 172.21.15.70:0/264341845 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f38fc09be70 con 0x7f3904107450 2024-04-01T01:31:39.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.605+0000 7f3909af4640 1 -- 172.21.15.70:0/264341845 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3904062350 con 0x7f3904107450 2024-04-01T01:31:39.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.607+0000 7f39017fa640 1 -- 172.21.15.70:0/264341845 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24068 (secure 0 0 0) 0x7f38fc09b770 con 0x7f3904107450 2024-04-01T01:31:39.606 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:39.607 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":89,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:31:20.903623+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:31:39.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.609+0000 7f3909af4640 1 -- 172.21.15.70:0/264341845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f38e40a24a0 msgr2=0x7f38e40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:39.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.609+0000 7f3909af4640 1 --2- 172.21.15.70:0/264341845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f38e40a24a0 0x7f38e40a4960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f38f4009bd0 tx=0x7f38f4005f90 comp rx=0 tx=0).stop 2024-04-01T01:31:39.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.609+0000 7f3909af4640 1 -- 172.21.15.70:0/264341845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3904107450 msgr2=0x7f390419ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:39.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.609+0000 7f3909af4640 1 --2- 172.21.15.70:0/264341845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3904107450 0x7f390419ef60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00eb10 tx=0x7f38fc00c2d0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.609+0000 7f3909af4640 1 -- 172.21.15.70:0/264341845 shutdown_connections 2024-04-01T01:31:39.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.609+0000 7f3909af4640 1 --2- 172.21.15.70:0/264341845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3904107450 0x7f390419ef60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.609+0000 7f3909af4640 1 --2- 172.21.15.70:0/264341845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3904104b20 0x7f3904197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.609+0000 7f3909af4640 1 --2- 172.21.15.70:0/264341845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f38e40a24a0 0x7f38e40a4960 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.609+0000 7f3909af4640 1 --2- 172.21.15.70:0/264341845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39041021f0 0x7f39041979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.609+0000 7f3909af4640 1 -- 172.21.15.70:0/264341845 >> 172.21.15.70:0/264341845 conn(0x7f39040fbbe0 msgr2=0x7f3904102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:39.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.609+0000 7f3909af4640 1 -- 172.21.15.70:0/264341845 shutdown_connections 2024-04-01T01:31:39.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.609+0000 7f3909af4640 1 -- 172.21.15.70:0/264341845 wait complete. 2024-04-01T01:31:39.624 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-01T01:31:39.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T01:31:39.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.726+0000 7f2dbb3a1640 1 -- 172.21.15.70:0/3583389778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2db4103320 msgr2=0x7f2db4109bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:39.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.726+0000 7f2dbb3a1640 1 --2- 172.21.15.70:0/3583389778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2db4103320 0x7f2db4109bb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2dac009ca0 tx=0x7f2dac02c0f0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.726+0000 7f2dbb3a1640 1 -- 172.21.15.70:0/3583389778 shutdown_connections 2024-04-01T01:31:39.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.726+0000 7f2dbb3a1640 1 --2- 172.21.15.70:0/3583389778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2db4103320 0x7f2db4109bb0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.726+0000 7f2dbb3a1640 1 --2- 172.21.15.70:0/3583389778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2db4102960 0x7f2db4102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.726+0000 7f2dbb3a1640 1 --2- 172.21.15.70:0/3583389778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2db4101770 0x7f2db4101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.726+0000 7f2dbb3a1640 1 -- 172.21.15.70:0/3583389778 >> 172.21.15.70:0/3583389778 conn(0x7f2db40fcd00 msgr2=0x7f2db40ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:39.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.727+0000 7f2dbb3a1640 1 -- 172.21.15.70:0/3583389778 shutdown_connections 2024-04-01T01:31:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.727+0000 7f2dbb3a1640 1 -- 172.21.15.70:0/3583389778 wait complete. 2024-04-01T01:31:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.727+0000 7f2dbb3a1640 1 Processor -- start 2024-04-01T01:31:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.727+0000 7f2dbb3a1640 1 -- start start 2024-04-01T01:31:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.727+0000 7f2dbb3a1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2db4101770 0x7f2db4197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.727+0000 7f2dbb3a1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2db4102960 0x7f2db4197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.727+0000 7f2dbb3a1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2db4103320 0x7f2db419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.727+0000 7f2dbb3a1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2db410d2a0 con 0x7f2db4101770 2024-04-01T01:31:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.727+0000 7f2dbb3a1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2db410d120 con 0x7f2db4102960 2024-04-01T01:31:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.727+0000 7f2dbb3a1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2db410d420 con 0x7f2db4103320 2024-04-01T01:31:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.727+0000 7f2db913d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2db4101770 0x7f2db4197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.727+0000 7f2db913d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2db4101770 0x7f2db4197960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51654/0 (socket says 172.21.15.70:51654) 2024-04-01T01:31:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.727+0000 7f2db913d640 1 -- 172.21.15.70:0/2864589558 learned_addr learned my addr 172.21.15.70:0/2864589558 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.727+0000 7f2db993e640 1 --2- 172.21.15.70:0/2864589558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2db4103320 0x7f2db419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.727+0000 7f2db893c640 1 --2- 172.21.15.70:0/2864589558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2db4102960 0x7f2db4197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.728+0000 7f2db913d640 1 -- 172.21.15.70:0/2864589558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2db4103320 msgr2=0x7f2db419ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.728+0000 7f2db913d640 1 --2- 172.21.15.70:0/2864589558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2db4103320 0x7f2db419ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.728+0000 7f2db913d640 1 -- 172.21.15.70:0/2864589558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2db4102960 msgr2=0x7f2db4197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.728+0000 7f2db913d640 1 --2- 172.21.15.70:0/2864589558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2db4102960 0x7f2db4197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.728+0000 7f2db913d640 1 -- 172.21.15.70:0/2864589558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db419f580 con 0x7f2db4101770 2024-04-01T01:31:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.728+0000 7f2db913d640 1 --2- 172.21.15.70:0/2864589558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2db4101770 0x7f2db4197960 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2db000e9c0 tx=0x7f2db000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.728+0000 7f2daa7fc640 1 -- 172.21.15.70:0/2864589558 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db000cd80 con 0x7f2db4101770 2024-04-01T01:31:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.728+0000 7f2daa7fc640 1 -- 172.21.15.70:0/2864589558 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2db00035f0 con 0x7f2db4101770 2024-04-01T01:31:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.728+0000 7f2daa7fc640 1 -- 172.21.15.70:0/2864589558 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db0010690 con 0x7f2db4101770 2024-04-01T01:31:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.728+0000 7f2dbb3a1640 1 -- 172.21.15.70:0/2864589558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db419f870 con 0x7f2db4101770 2024-04-01T01:31:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.728+0000 7f2dbb3a1640 1 -- 172.21.15.70:0/2864589558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db419fe10 con 0x7f2db4101770 2024-04-01T01:31:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.729+0000 7f2dbb3a1640 1 -- 172.21.15.70:0/2864589558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2db4108a90 con 0x7f2db4101770 2024-04-01T01:31:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.730+0000 7f2daa7fc640 1 -- 172.21.15.70:0/2864589558 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f2db00037a0 con 0x7f2db4101770 2024-04-01T01:31:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.731+0000 7f2daa7fc640 1 --2- 172.21.15.70:0/2864589558 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2da00a25b0 0x7f2da00a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.731+0000 7f2db893c640 1 --2- 172.21.15.70:0/2864589558 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2da00a25b0 0x7f2da00a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.731+0000 7f2daa7fc640 1 -- 172.21.15.70:0/2864589558 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2db00cede0 con 0x7f2db4101770 2024-04-01T01:31:39.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.731+0000 7f2db893c640 1 --2- 172.21.15.70:0/2864589558 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2da00a25b0 0x7f2da00a4a70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2db4198e70 tx=0x7f2da4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:39.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.735+0000 7f2daa7fc640 1 -- 172.21.15.70:0/2864589558 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2db009bc00 con 0x7f2db4101770 2024-04-01T01:31:39.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.906+0000 7f2dbb3a1640 1 -- 172.21.15.70:0/2864589558 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2db4061b90 con 0x7f2da00a25b0 2024-04-01T01:31:39.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.916+0000 7f2daa7fc640 1 -- 172.21.15.70:0/2864589558 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340300 (secure 0 0 0) 0x7f2db4061b90 con 0x7f2da00a25b0 2024-04-01T01:31:39.920 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:39.922 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-01T01:31:39.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.919+0000 7f2dbb3a1640 1 -- 172.21.15.70:0/2864589558 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2da00a25b0 msgr2=0x7f2da00a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:39.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.919+0000 7f2dbb3a1640 1 --2- 172.21.15.70:0/2864589558 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2da00a25b0 0x7f2da00a4a70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2db4198e70 tx=0x7f2da4005f90 comp rx=0 tx=0).stop 2024-04-01T01:31:39.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.919+0000 7f2dbb3a1640 1 -- 172.21.15.70:0/2864589558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2db4101770 msgr2=0x7f2db4197960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:39.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.919+0000 7f2dbb3a1640 1 --2- 172.21.15.70:0/2864589558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2db4101770 0x7f2db4197960 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2db000e9c0 tx=0x7f2db000ee80 comp rx=0 tx=0).stop 2024-04-01T01:31:39.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.919+0000 7f2dbb3a1640 1 -- 172.21.15.70:0/2864589558 shutdown_connections 2024-04-01T01:31:39.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.919+0000 7f2dbb3a1640 1 --2- 172.21.15.70:0/2864589558 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2da00a25b0 0x7f2da00a4a70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.919+0000 7f2dbb3a1640 1 --2- 172.21.15.70:0/2864589558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2db4103320 0x7f2db419ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.919+0000 7f2dbb3a1640 1 --2- 172.21.15.70:0/2864589558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2db4102960 0x7f2db4197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.919+0000 7f2dbb3a1640 1 --2- 172.21.15.70:0/2864589558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2db4101770 0x7f2db4197960 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:39.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.919+0000 7f2dbb3a1640 1 -- 172.21.15.70:0/2864589558 >> 172.21.15.70:0/2864589558 conn(0x7f2db40fcd00 msgr2=0x7f2db4105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:39.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.919+0000 7f2dbb3a1640 1 -- 172.21.15.70:0/2864589558 shutdown_connections 2024-04-01T01:31:39.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:39.919+0000 7f2dbb3a1640 1 -- 172.21.15.70:0/2864589558 wait complete. 2024-04-01T01:31:39.928 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:39 smithi070 ceph-mon[32179]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:31:39.928 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:39 smithi070 ceph-mon[32179]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:31:39.928 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:39 smithi070 ceph-mon[32179]: pgmap v299: 129 pgs: 129 active+clean; 585 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-01T01:31:39.929 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:39 smithi070 ceph-mon[32179]: from='client.24716 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:31:39.929 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:39 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2170485849' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:39.929 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:39 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/264341845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:39.930 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-04-01T01:31:38.561581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":599046,"num_objects":69,"num_object_clones":0,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":143,"num_read_kb":127,"num_write":321,"num_write_kb":2985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261,"ondisk_log_size":261,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":358488,"kb_used_data":13476,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124583336,"statfs":{"total":1151940427776,"available":1151573336064,"internally_reserved":0,"allocated":13799424,"data_stored":7518684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6268,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":17,"num_read_kb":18,"num_write":95,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003192"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.728117+0000","last_change":"2024-04-01T01:31:16.385517+0000","last_active":"2024-04-01T01:31:22.728117+0000","last_peered":"2024-04-01T01:31:22.728117+0000","last_clean":"2024-04-01T01:31:22.728117+0000","last_became_active":"2024-04-01T01:31:16.384682+0000","last_became_peered":"2024-04-01T01:31:16.384682+0000","last_unstale":"2024-04-01T01:31:22.728117+0000","last_undegraded":"2024-04-01T01:31:22.728117+0000","last_fullsized":"2024-04-01T01:31:22.728117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:35:47.614041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726824+0000","last_change":"2024-04-01T01:31:16.376119+0000","last_active":"2024-04-01T01:31:22.726824+0000","last_peered":"2024-04-01T01:31:22.726824+0000","last_clean":"2024-04-01T01:31:22.726824+0000","last_became_active":"2024-04-01T01:31:16.375801+0000","last_became_peered":"2024-04-01T01:31:16.375801+0000","last_unstale":"2024-04-01T01:31:22.726824+0000","last_undegraded":"2024-04-01T01:31:22.726824+0000","last_fullsized":"2024-04-01T01:31:22.726824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:34.120487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726630+0000","last_change":"2024-04-01T01:31:16.378158+0000","last_active":"2024-04-01T01:31:22.726630+0000","last_peered":"2024-04-01T01:31:22.726630+0000","last_clean":"2024-04-01T01:31:22.726630+0000","last_became_active":"2024-04-01T01:31:16.377831+0000","last_became_peered":"2024-04-01T01:31:16.377831+0000","last_unstale":"2024-04-01T01:31:22.726630+0000","last_undegraded":"2024-04-01T01:31:22.726630+0000","last_fullsized":"2024-04-01T01:31:22.726630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:30:06.487545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726977+0000","last_change":"2024-04-01T01:31:16.389708+0000","last_active":"2024-04-01T01:31:22.726977+0000","last_peered":"2024-04-01T01:31:22.726977+0000","last_clean":"2024-04-01T01:31:22.726977+0000","last_became_active":"2024-04-01T01:31:16.389409+0000","last_became_peered":"2024-04-01T01:31:16.389409+0000","last_unstale":"2024-04-01T01:31:22.726977+0000","last_undegraded":"2024-04-01T01:31:22.726977+0000","last_fullsized":"2024-04-01T01:31:22.726977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:57:49.215988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.151166+0000","last_change":"2024-04-01T01:31:16.385996+0000","last_active":"2024-04-01T01:31:22.151166+0000","last_peered":"2024-04-01T01:31:22.151166+0000","last_clean":"2024-04-01T01:31:22.151166+0000","last_became_active":"2024-04-01T01:31:16.385648+0000","last_became_peered":"2024-04-01T01:31:16.385648+0000","last_unstale":"2024-04-01T01:31:22.151166+0000","last_undegraded":"2024-04-01T01:31:22.151166+0000","last_fullsized":"2024-04-01T01:31:22.151166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:14:56.595595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.961915+0000","last_change":"2024-04-01T01:31:16.376519+0000","last_active":"2024-04-01T01:31:21.961915+0000","last_peered":"2024-04-01T01:31:21.961915+0000","last_clean":"2024-04-01T01:31:21.961915+0000","last_became_active":"2024-04-01T01:31:16.376178+0000","last_became_peered":"2024-04-01T01:31:16.376178+0000","last_unstale":"2024-04-01T01:31:21.961915+0000","last_undegraded":"2024-04-01T01:31:21.961915+0000","last_fullsized":"2024-04-01T01:31:21.961915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:01:45.861899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563654+0000","last_change":"2024-04-01T01:31:16.383939+0000","last_active":"2024-04-01T01:31:21.563654+0000","last_peered":"2024-04-01T01:31:21.563654+0000","last_clean":"2024-04-01T01:31:21.563654+0000","last_became_active":"2024-04-01T01:31:16.383626+0000","last_became_peered":"2024-04-01T01:31:16.383626+0000","last_unstale":"2024-04-01T01:31:21.563654+0000","last_undegraded":"2024-04-01T01:31:21.563654+0000","last_fullsized":"2024-04-01T01:31:21.563654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:36:05.764031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.488942+0000","last_change":"2024-04-01T01:31:16.384007+0000","last_active":"2024-04-01T01:31:22.488942+0000","last_peered":"2024-04-01T01:31:22.488942+0000","last_clean":"2024-04-01T01:31:22.488942+0000","last_became_active":"2024-04-01T01:31:16.383687+0000","last_became_peered":"2024-04-01T01:31:16.383687+0000","last_unstale":"2024-04-01T01:31:22.488942+0000","last_undegraded":"2024-04-01T01:31:22.488942+0000","last_fullsized":"2024-04-01T01:31:22.488942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:14:17.593402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727748+0000","last_change":"2024-04-01T01:31:16.380357+0000","last_active":"2024-04-01T01:31:22.727748+0000","last_peered":"2024-04-01T01:31:22.727748+0000","last_clean":"2024-04-01T01:31:22.727748+0000","last_became_active":"2024-04-01T01:31:16.380046+0000","last_became_peered":"2024-04-01T01:31:16.380046+0000","last_unstale":"2024-04-01T01:31:22.727748+0000","last_undegraded":"2024-04-01T01:31:22.727748+0000","last_fullsized":"2024-04-01T01:31:22.727748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:51:53.887768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.962202+0000","last_change":"2024-04-01T01:31:16.378551+0000","last_active":"2024-04-01T01:31:21.962202+0000","last_peered":"2024-04-01T01:31:21.962202+0000","last_clean":"2024-04-01T01:31:21.962202+0000","last_became_active":"2024-04-01T01:31:16.378234+0000","last_became_peered":"2024-04-01T01:31:16.378234+0000","last_unstale":"2024-04-01T01:31:21.962202+0000","last_undegraded":"2024-04-01T01:31:21.962202+0000","last_fullsized":"2024-04-01T01:31:21.962202+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:37:45.998115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564002+0000","last_change":"2024-04-01T01:31:16.388755+0000","last_active":"2024-04-01T01:31:21.564002+0000","last_peered":"2024-04-01T01:31:21.564002+0000","last_clean":"2024-04-01T01:31:21.564002+0000","last_became_active":"2024-04-01T01:31:16.388434+0000","last_became_peered":"2024-04-01T01:31:16.388434+0000","last_unstale":"2024-04-01T01:31:21.564002+0000","last_undegraded":"2024-04-01T01:31:21.564002+0000","last_fullsized":"2024-04-01T01:31:21.564002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:03:07.168110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726864+0000","last_change":"2024-04-01T01:31:16.390478+0000","last_active":"2024-04-01T01:31:22.726864+0000","last_peered":"2024-04-01T01:31:22.726864+0000","last_clean":"2024-04-01T01:31:22.726864+0000","last_became_active":"2024-04-01T01:31:16.390013+0000","last_became_peered":"2024-04-01T01:31:16.390013+0000","last_unstale":"2024-04-01T01:31:22.726864+0000","last_undegraded":"2024-04-01T01:31:22.726864+0000","last_fullsized":"2024-04-01T01:31:22.726864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:28:21.677527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989589+0000","last_change":"2024-04-01T01:31:16.366921+0000","last_active":"2024-04-01T01:31:22.989589+0000","last_peered":"2024-04-01T01:31:22.989589+0000","last_clean":"2024-04-01T01:31:22.989589+0000","last_became_active":"2024-04-01T01:31:16.366098+0000","last_became_peered":"2024-04-01T01:31:16.366098+0000","last_unstale":"2024-04-01T01:31:22.989589+0000","last_undegraded":"2024-04-01T01:31:22.989589+0000","last_fullsized":"2024-04-01T01:31:22.989589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:29.456263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726607+0000","last_change":"2024-04-01T01:31:16.379178+0000","last_active":"2024-04-01T01:31:22.726607+0000","last_peered":"2024-04-01T01:31:22.726607+0000","last_clean":"2024-04-01T01:31:22.726607+0000","last_became_active":"2024-04-01T01:31:16.378877+0000","last_became_peered":"2024-04-01T01:31:16.378877+0000","last_unstale":"2024-04-01T01:31:22.726607+0000","last_undegraded":"2024-04-01T01:31:22.726607+0000","last_fullsized":"2024-04-01T01:31:22.726607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:19:16.941504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726585+0000","last_change":"2024-04-01T01:31:16.385373+0000","last_active":"2024-04-01T01:31:22.726585+0000","last_peered":"2024-04-01T01:31:22.726585+0000","last_clean":"2024-04-01T01:31:22.726585+0000","last_became_active":"2024-04-01T01:31:16.385046+0000","last_became_peered":"2024-04-01T01:31:16.385046+0000","last_unstale":"2024-04-01T01:31:22.726585+0000","last_undegraded":"2024-04-01T01:31:22.726585+0000","last_fullsized":"2024-04-01T01:31:22.726585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:40:32.521220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726525+0000","last_change":"2024-04-01T01:31:16.386267+0000","last_active":"2024-04-01T01:31:22.726525+0000","last_peered":"2024-04-01T01:31:22.726525+0000","last_clean":"2024-04-01T01:31:22.726525+0000","last_became_active":"2024-04-01T01:31:16.385917+0000","last_became_peered":"2024-04-01T01:31:16.385917+0000","last_unstale":"2024-04-01T01:31:22.726525+0000","last_undegraded":"2024-04-01T01:31:22.726525+0000","last_fullsized":"2024-04-01T01:31:22.726525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:58.249253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727779+0000","last_change":"2024-04-01T01:31:16.385609+0000","last_active":"2024-04-01T01:31:22.727779+0000","last_peered":"2024-04-01T01:31:22.727779+0000","last_clean":"2024-04-01T01:31:22.727779+0000","last_became_active":"2024-04-01T01:31:16.385128+0000","last_became_peered":"2024-04-01T01:31:16.385128+0000","last_unstale":"2024-04-01T01:31:22.727779+0000","last_undegraded":"2024-04-01T01:31:22.727779+0000","last_fullsized":"2024-04-01T01:31:22.727779+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:30:45.915587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.561736+0000","last_change":"2024-04-01T01:31:16.381209+0000","last_active":"2024-04-01T01:31:21.561736+0000","last_peered":"2024-04-01T01:31:21.561736+0000","last_clean":"2024-04-01T01:31:21.561736+0000","last_became_active":"2024-04-01T01:31:16.380721+0000","last_became_peered":"2024-04-01T01:31:16.380721+0000","last_unstale":"2024-04-01T01:31:21.561736+0000","last_undegraded":"2024-04-01T01:31:21.561736+0000","last_fullsized":"2024-04-01T01:31:21.561736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:22.438224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564687+0000","last_change":"2024-04-01T01:31:16.385235+0000","last_active":"2024-04-01T01:31:21.564687+0000","last_peered":"2024-04-01T01:31:21.564687+0000","last_clean":"2024-04-01T01:31:21.564687+0000","last_became_active":"2024-04-01T01:31:16.384856+0000","last_became_peered":"2024-04-01T01:31:16.384856+0000","last_unstale":"2024-04-01T01:31:21.564687+0000","last_undegraded":"2024-04-01T01:31:21.564687+0000","last_fullsized":"2024-04-01T01:31:21.564687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:35:30.214977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564290+0000","last_change":"2024-04-01T01:31:20.573626+0000","last_active":"2024-04-01T01:31:21.564290+0000","last_peered":"2024-04-01T01:31:21.564290+0000","last_clean":"2024-04-01T01:31:21.564290+0000","last_became_active":"2024-04-01T01:31:20.573182+0000","last_became_peered":"2024-04-01T01:31:20.573182+0000","last_unstale":"2024-04-01T01:31:21.564290+0000","last_undegraded":"2024-04-01T01:31:21.564290+0000","last_fullsized":"2024-04-01T01:31:21.564290+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:33:19.162613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.562044+0000","last_change":"2024-04-01T01:31:15.373100+0000","last_active":"2024-04-01T01:31:21.562044+0000","last_peered":"2024-04-01T01:31:21.562044+0000","last_clean":"2024-04-01T01:31:21.562044+0000","last_became_active":"2024-04-01T01:31:15.372710+0000","last_became_peered":"2024-04-01T01:31:15.372710+0000","last_unstale":"2024-04-01T01:31:21.562044+0000","last_undegraded":"2024-04-01T01:31:21.562044+0000","last_fullsized":"2024-04-01T01:31:21.562044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:29:42.303549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489391+0000","last_change":"2024-04-01T01:31:16.383690+0000","last_active":"2024-04-01T01:31:22.489391+0000","last_peered":"2024-04-01T01:31:22.489391+0000","last_clean":"2024-04-01T01:31:22.489391+0000","last_became_active":"2024-04-01T01:31:16.383248+0000","last_became_peered":"2024-04-01T01:31:16.383248+0000","last_unstale":"2024-04-01T01:31:22.489391+0000","last_undegraded":"2024-04-01T01:31:22.489391+0000","last_fullsized":"2024-04-01T01:31:22.489391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:34:49.112880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726880+0000","last_change":"2024-04-01T01:31:20.569469+0000","last_active":"2024-04-01T01:31:22.726880+0000","last_peered":"2024-04-01T01:31:22.726880+0000","last_clean":"2024-04-01T01:31:22.726880+0000","last_became_active":"2024-04-01T01:31:20.569060+0000","last_became_peered":"2024-04-01T01:31:20.569060+0000","last_unstale":"2024-04-01T01:31:22.726880+0000","last_undegraded":"2024-04-01T01:31:22.726880+0000","last_fullsized":"2024-04-01T01:31:22.726880+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:04:05.960140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489633+0000","last_change":"2024-04-01T01:31:15.381742+0000","last_active":"2024-04-01T01:31:22.489633+0000","last_peered":"2024-04-01T01:31:22.489633+0000","last_clean":"2024-04-01T01:31:22.489633+0000","last_became_active":"2024-04-01T01:31:15.377543+0000","last_became_peered":"2024-04-01T01:31:15.377543+0000","last_unstale":"2024-04-01T01:31:22.489633+0000","last_undegraded":"2024-04-01T01:31:22.489633+0000","last_fullsized":"2024-04-01T01:31:22.489633+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:51:42.911114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726871+0000","last_change":"2024-04-01T01:31:16.382001+0000","last_active":"2024-04-01T01:31:22.726871+0000","last_peered":"2024-04-01T01:31:22.726871+0000","last_clean":"2024-04-01T01:31:22.726871+0000","last_became_active":"2024-04-01T01:31:16.381694+0000","last_became_peered":"2024-04-01T01:31:16.381694+0000","last_unstale":"2024-04-01T01:31:22.726871+0000","last_undegraded":"2024-04-01T01:31:22.726871+0000","last_fullsized":"2024-04-01T01:31:22.726871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:55:07.366755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726847+0000","last_change":"2024-04-01T01:31:20.570080+0000","last_active":"2024-04-01T01:31:22.726847+0000","last_peered":"2024-04-01T01:31:22.726847+0000","last_clean":"2024-04-01T01:31:22.726847+0000","last_became_active":"2024-04-01T01:31:20.569712+0000","last_became_peered":"2024-04-01T01:31:20.569712+0000","last_unstale":"2024-04-01T01:31:22.726847+0000","last_undegraded":"2024-04-01T01:31:22.726847+0000","last_fullsized":"2024-04-01T01:31:22.726847+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:48:27.732831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'11","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:37.571665+0000","last_change":"2024-04-01T01:31:15.378381+0000","last_active":"2024-04-01T01:31:37.571665+0000","last_peered":"2024-04-01T01:31:37.571665+0000","last_clean":"2024-04-01T01:31:37.571665+0000","last_became_active":"2024-04-01T01:31:15.378081+0000","last_became_peered":"2024-04-01T01:31:15.378081+0000","last_unstale":"2024-04-01T01:31:37.571665+0000","last_undegraded":"2024-04-01T01:31:37.571665+0000","last_fullsized":"2024-04-01T01:31:37.571665+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:56:27.896123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727424+0000","last_change":"2024-04-01T01:31:16.383020+0000","last_active":"2024-04-01T01:31:22.727424+0000","last_peered":"2024-04-01T01:31:22.727424+0000","last_clean":"2024-04-01T01:31:22.727424+0000","last_became_active":"2024-04-01T01:31:16.382640+0000","last_became_peered":"2024-04-01T01:31:16.382640+0000","last_unstale":"2024-04-01T01:31:22.727424+0000","last_undegraded":"2024-04-01T01:31:22.727424+0000","last_fullsized":"2024-04-01T01:31:22.727424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:11:51.211929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726826+0000","last_change":"2024-04-01T01:31:20.573030+0000","last_active":"2024-04-01T01:31:22.726826+0000","last_peered":"2024-04-01T01:31:22.726826+0000","last_clean":"2024-04-01T01:31:22.726826+0000","last_became_active":"2024-04-01T01:31:20.572613+0000","last_became_peered":"2024-04-01T01:31:20.572613+0000","last_unstale":"2024-04-01T01:31:22.726826+0000","last_undegraded":"2024-04-01T01:31:22.726826+0000","last_fullsized":"2024-04-01T01:31:22.726826+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:42:17.086942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:37.570150+0000","last_change":"2024-04-01T01:31:15.373054+0000","last_active":"2024-04-01T01:31:37.570150+0000","last_peered":"2024-04-01T01:31:37.570150+0000","last_clean":"2024-04-01T01:31:37.570150+0000","last_became_active":"2024-04-01T01:31:15.372575+0000","last_became_peered":"2024-04-01T01:31:15.372575+0000","last_unstale":"2024-04-01T01:31:37.570150+0000","last_undegraded":"2024-04-01T01:31:37.570150+0000","last_fullsized":"2024-04-01T01:31:37.570150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:56:41.860423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.728277+0000","last_change":"2024-04-01T01:31:16.376037+0000","last_active":"2024-04-01T01:31:22.728277+0000","last_peered":"2024-04-01T01:31:22.728277+0000","last_clean":"2024-04-01T01:31:22.728277+0000","last_became_active":"2024-04-01T01:31:16.375463+0000","last_became_peered":"2024-04-01T01:31:16.375463+0000","last_unstale":"2024-04-01T01:31:22.728277+0000","last_undegraded":"2024-04-01T01:31:22.728277+0000","last_fullsized":"2024-04-01T01:31:22.728277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:30:40.098033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564643+0000","last_change":"2024-04-01T01:31:20.574288+0000","last_active":"2024-04-01T01:31:21.564643+0000","last_peered":"2024-04-01T01:31:21.564643+0000","last_clean":"2024-04-01T01:31:21.564643+0000","last_became_active":"2024-04-01T01:31:20.573812+0000","last_became_peered":"2024-04-01T01:31:20.573812+0000","last_unstale":"2024-04-01T01:31:21.564643+0000","last_undegraded":"2024-04-01T01:31:21.564643+0000","last_fullsized":"2024-04-01T01:31:21.564643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:43:46.126102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:35.543959+0000","last_change":"2024-04-01T01:31:15.380835+0000","last_active":"2024-04-01T01:31:35.543959+0000","last_peered":"2024-04-01T01:31:35.543959+0000","last_clean":"2024-04-01T01:31:35.543959+0000","last_became_active":"2024-04-01T01:31:15.380518+0000","last_became_peered":"2024-04-01T01:31:15.380518+0000","last_unstale":"2024-04-01T01:31:35.543959+0000","last_undegraded":"2024-04-01T01:31:35.543959+0000","last_fullsized":"2024-04-01T01:31:35.543959+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:51.452917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.151196+0000","last_change":"2024-04-01T01:31:16.377735+0000","last_active":"2024-04-01T01:31:22.151196+0000","last_peered":"2024-04-01T01:31:22.151196+0000","last_clean":"2024-04-01T01:31:22.151196+0000","last_became_active":"2024-04-01T01:31:16.377342+0000","last_became_peered":"2024-04-01T01:31:16.377342+0000","last_unstale":"2024-04-01T01:31:22.151196+0000","last_undegraded":"2024-04-01T01:31:22.151196+0000","last_fullsized":"2024-04-01T01:31:22.151196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:47.539223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564543+0000","last_change":"2024-04-01T01:31:20.573370+0000","last_active":"2024-04-01T01:31:21.564543+0000","last_peered":"2024-04-01T01:31:21.564543+0000","last_clean":"2024-04-01T01:31:21.564543+0000","last_became_active":"2024-04-01T01:31:20.572919+0000","last_became_peered":"2024-04-01T01:31:20.572919+0000","last_unstale":"2024-04-01T01:31:21.564543+0000","last_undegraded":"2024-04-01T01:31:21.564543+0000","last_fullsized":"2024-04-01T01:31:21.564543+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:42.190918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:35.557203+0000","last_change":"2024-04-01T01:31:15.372310+0000","last_active":"2024-04-01T01:31:35.557203+0000","last_peered":"2024-04-01T01:31:35.557203+0000","last_clean":"2024-04-01T01:31:35.557203+0000","last_became_active":"2024-04-01T01:31:15.371527+0000","last_became_peered":"2024-04-01T01:31:15.371527+0000","last_unstale":"2024-04-01T01:31:35.557203+0000","last_undegraded":"2024-04-01T01:31:35.557203+0000","last_fullsized":"2024-04-01T01:31:35.557203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:54:53.844991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727986+0000","last_change":"2024-04-01T01:31:16.384130+0000","last_active":"2024-04-01T01:31:22.727986+0000","last_peered":"2024-04-01T01:31:22.727986+0000","last_clean":"2024-04-01T01:31:22.727986+0000","last_became_active":"2024-04-01T01:31:16.383400+0000","last_became_peered":"2024-04-01T01:31:16.383400+0000","last_unstale":"2024-04-01T01:31:22.727986+0000","last_undegraded":"2024-04-01T01:31:22.727986+0000","last_fullsized":"2024-04-01T01:31:22.727986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:22:10.860325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727817+0000","last_change":"2024-04-01T01:31:20.571052+0000","last_active":"2024-04-01T01:31:22.727817+0000","last_peered":"2024-04-01T01:31:22.727817+0000","last_clean":"2024-04-01T01:31:22.727817+0000","last_became_active":"2024-04-01T01:31:20.570617+0000","last_became_peered":"2024-04-01T01:31:20.570617+0000","last_unstale":"2024-04-01T01:31:22.727817+0000","last_undegraded":"2024-04-01T01:31:22.727817+0000","last_fullsized":"2024-04-01T01:31:22.727817+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:53:51.195056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989391+0000","last_change":"2024-04-01T01:31:15.369820+0000","last_active":"2024-04-01T01:31:22.989391+0000","last_peered":"2024-04-01T01:31:22.989391+0000","last_clean":"2024-04-01T01:31:22.989391+0000","last_became_active":"2024-04-01T01:31:15.369378+0000","last_became_peered":"2024-04-01T01:31:15.369378+0000","last_unstale":"2024-04-01T01:31:22.989391+0000","last_undegraded":"2024-04-01T01:31:22.989391+0000","last_fullsized":"2024-04-01T01:31:22.989391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:37:34.351220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489268+0000","last_change":"2024-04-01T01:31:16.383352+0000","last_active":"2024-04-01T01:31:22.489268+0000","last_peered":"2024-04-01T01:31:22.489268+0000","last_clean":"2024-04-01T01:31:22.489268+0000","last_became_active":"2024-04-01T01:31:16.382898+0000","last_became_peered":"2024-04-01T01:31:16.382898+0000","last_unstale":"2024-04-01T01:31:22.489268+0000","last_undegraded":"2024-04-01T01:31:22.489268+0000","last_fullsized":"2024-04-01T01:31:22.489268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:27.854459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726904+0000","last_change":"2024-04-01T01:31:20.576578+0000","last_active":"2024-04-01T01:31:22.726904+0000","last_peered":"2024-04-01T01:31:22.726904+0000","last_clean":"2024-04-01T01:31:22.726904+0000","last_became_active":"2024-04-01T01:31:20.575996+0000","last_became_peered":"2024-04-01T01:31:20.575996+0000","last_unstale":"2024-04-01T01:31:22.726904+0000","last_undegraded":"2024-04-01T01:31:22.726904+0000","last_fullsized":"2024-04-01T01:31:22.726904+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:52:08.808357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727658+0000","last_change":"2024-04-01T01:31:15.377246+0000","last_active":"2024-04-01T01:31:22.727658+0000","last_peered":"2024-04-01T01:31:22.727658+0000","last_clean":"2024-04-01T01:31:22.727658+0000","last_became_active":"2024-04-01T01:31:15.376957+0000","last_became_peered":"2024-04-01T01:31:15.376957+0000","last_unstale":"2024-04-01T01:31:22.727658+0000","last_undegraded":"2024-04-01T01:31:22.727658+0000","last_fullsized":"2024-04-01T01:31:22.727658+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:59:34.313984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.561233+0000","last_change":"2024-04-01T01:31:16.379031+0000","last_active":"2024-04-01T01:31:21.561233+0000","last_peered":"2024-04-01T01:31:21.561233+0000","last_clean":"2024-04-01T01:31:21.561233+0000","last_became_active":"2024-04-01T01:31:16.378593+0000","last_became_peered":"2024-04-01T01:31:16.378593+0000","last_unstale":"2024-04-01T01:31:21.561233+0000","last_undegraded":"2024-04-01T01:31:21.561233+0000","last_fullsized":"2024-04-01T01:31:21.561233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:31.816103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489148+0000","last_change":"2024-04-01T01:31:20.578660+0000","last_active":"2024-04-01T01:31:22.489148+0000","last_peered":"2024-04-01T01:31:22.489148+0000","last_clean":"2024-04-01T01:31:22.489148+0000","last_became_active":"2024-04-01T01:31:20.578082+0000","last_became_peered":"2024-04-01T01:31:20.578082+0000","last_unstale":"2024-04-01T01:31:22.489148+0000","last_undegraded":"2024-04-01T01:31:22.489148+0000","last_fullsized":"2024-04-01T01:31:22.489148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:02:56.570816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:34.620860+0000","last_change":"2024-04-01T01:31:15.373428+0000","last_active":"2024-04-01T01:31:34.620860+0000","last_peered":"2024-04-01T01:31:34.620860+0000","last_clean":"2024-04-01T01:31:34.620860+0000","last_became_active":"2024-04-01T01:31:15.372965+0000","last_became_peered":"2024-04-01T01:31:15.372965+0000","last_unstale":"2024-04-01T01:31:34.620860+0000","last_undegraded":"2024-04-01T01:31:34.620860+0000","last_fullsized":"2024-04-01T01:31:34.620860+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:55:42.211055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727715+0000","last_change":"2024-04-01T01:31:16.382961+0000","last_active":"2024-04-01T01:31:22.727715+0000","last_peered":"2024-04-01T01:31:22.727715+0000","last_clean":"2024-04-01T01:31:22.727715+0000","last_became_active":"2024-04-01T01:31:16.382527+0000","last_became_peered":"2024-04-01T01:31:16.382527+0000","last_unstale":"2024-04-01T01:31:22.727715+0000","last_undegraded":"2024-04-01T01:31:22.727715+0000","last_fullsized":"2024-04-01T01:31:22.727715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:46:20.250064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.561270+0000","last_change":"2024-04-01T01:31:20.571842+0000","last_active":"2024-04-01T01:31:21.561270+0000","last_peered":"2024-04-01T01:31:21.561270+0000","last_clean":"2024-04-01T01:31:21.561270+0000","last_became_active":"2024-04-01T01:31:20.571394+0000","last_became_peered":"2024-04-01T01:31:20.571394+0000","last_unstale":"2024-04-01T01:31:21.561270+0000","last_undegraded":"2024-04-01T01:31:21.561270+0000","last_fullsized":"2024-04-01T01:31:21.561270+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:51:36.512277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:34.621119+0000","last_change":"2024-04-01T01:31:15.379737+0000","last_active":"2024-04-01T01:31:34.621119+0000","last_peered":"2024-04-01T01:31:34.621119+0000","last_clean":"2024-04-01T01:31:34.621119+0000","last_became_active":"2024-04-01T01:31:15.379158+0000","last_became_peered":"2024-04-01T01:31:15.379158+0000","last_unstale":"2024-04-01T01:31:34.621119+0000","last_undegraded":"2024-04-01T01:31:34.621119+0000","last_fullsized":"2024-04-01T01:31:34.621119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:12:44.474772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726929+0000","last_change":"2024-04-01T01:31:16.386059+0000","last_active":"2024-04-01T01:31:22.726929+0000","last_peered":"2024-04-01T01:31:22.726929+0000","last_clean":"2024-04-01T01:31:22.726929+0000","last_became_active":"2024-04-01T01:31:16.384551+0000","last_became_peered":"2024-04-01T01:31:16.384551+0000","last_unstale":"2024-04-01T01:31:22.726929+0000","last_undegraded":"2024-04-01T01:31:22.726929+0000","last_fullsized":"2024-04-01T01:31:22.726929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:52:16.594485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726990+0000","last_change":"2024-04-01T01:31:20.566937+0000","last_active":"2024-04-01T01:31:22.726990+0000","last_peered":"2024-04-01T01:31:22.726990+0000","last_clean":"2024-04-01T01:31:22.726990+0000","last_became_active":"2024-04-01T01:31:20.566447+0000","last_became_peered":"2024-04-01T01:31:20.566447+0000","last_unstale":"2024-04-01T01:31:22.726990+0000","last_undegraded":"2024-04-01T01:31:22.726990+0000","last_fullsized":"2024-04-01T01:31:22.726990+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:17:37.845166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:34.619695+0000","last_change":"2024-04-01T01:31:15.375669+0000","last_active":"2024-04-01T01:31:34.619695+0000","last_peered":"2024-04-01T01:31:34.619695+0000","last_clean":"2024-04-01T01:31:34.619695+0000","last_became_active":"2024-04-01T01:31:15.375353+0000","last_became_peered":"2024-04-01T01:31:15.375353+0000","last_unstale":"2024-04-01T01:31:34.619695+0000","last_undegraded":"2024-04-01T01:31:34.619695+0000","last_fullsized":"2024-04-01T01:31:34.619695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:47:43.096898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727521+0000","last_change":"2024-04-01T01:31:16.388399+0000","last_active":"2024-04-01T01:31:22.727521+0000","last_peered":"2024-04-01T01:31:22.727521+0000","last_clean":"2024-04-01T01:31:22.727521+0000","last_became_active":"2024-04-01T01:31:16.388059+0000","last_became_peered":"2024-04-01T01:31:16.388059+0000","last_unstale":"2024-04-01T01:31:22.727521+0000","last_undegraded":"2024-04-01T01:31:22.727521+0000","last_fullsized":"2024-04-01T01:31:22.727521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:59:50.907802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.561606+0000","last_change":"2024-04-01T01:31:20.569524+0000","last_active":"2024-04-01T01:31:21.561606+0000","last_peered":"2024-04-01T01:31:21.561606+0000","last_clean":"2024-04-01T01:31:21.561606+0000","last_became_active":"2024-04-01T01:31:20.569112+0000","last_became_peered":"2024-04-01T01:31:20.569112+0000","last_unstale":"2024-04-01T01:31:21.561606+0000","last_undegraded":"2024-04-01T01:31:21.561606+0000","last_fullsized":"2024-04-01T01:31:21.561606+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:06:53.651152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:35.548827+0000","last_change":"2024-04-01T01:31:15.376421+0000","last_active":"2024-04-01T01:31:35.548827+0000","last_peered":"2024-04-01T01:31:35.548827+0000","last_clean":"2024-04-01T01:31:35.548827+0000","last_became_active":"2024-04-01T01:31:15.375985+0000","last_became_peered":"2024-04-01T01:31:15.375985+0000","last_unstale":"2024-04-01T01:31:35.548827+0000","last_undegraded":"2024-04-01T01:31:35.548827+0000","last_fullsized":"2024-04-01T01:31:35.548827+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:37:29.121436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.151447+0000","last_change":"2024-04-01T01:31:16.380641+0000","last_active":"2024-04-01T01:31:22.151447+0000","last_peered":"2024-04-01T01:31:22.151447+0000","last_clean":"2024-04-01T01:31:22.151447+0000","last_became_active":"2024-04-01T01:31:16.380300+0000","last_became_peered":"2024-04-01T01:31:16.380300+0000","last_unstale":"2024-04-01T01:31:22.151447+0000","last_undegraded":"2024-04-01T01:31:22.151447+0000","last_fullsized":"2024-04-01T01:31:22.151447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:26:08.240694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.728572+0000","last_change":"2024-04-01T01:31:20.576094+0000","last_active":"2024-04-01T01:31:22.728572+0000","last_peered":"2024-04-01T01:31:22.728572+0000","last_clean":"2024-04-01T01:31:22.728572+0000","last_became_active":"2024-04-01T01:31:20.575674+0000","last_became_peered":"2024-04-01T01:31:20.575674+0000","last_unstale":"2024-04-01T01:31:22.728572+0000","last_undegraded":"2024-04-01T01:31:22.728572+0000","last_fullsized":"2024-04-01T01:31:22.728572+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:42:19.352398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:37.571173+0000","last_change":"2024-04-01T01:31:15.375136+0000","last_active":"2024-04-01T01:31:37.571173+0000","last_peered":"2024-04-01T01:31:37.571173+0000","last_clean":"2024-04-01T01:31:37.571173+0000","last_became_active":"2024-04-01T01:31:15.374813+0000","last_became_peered":"2024-04-01T01:31:15.374813+0000","last_unstale":"2024-04-01T01:31:37.571173+0000","last_undegraded":"2024-04-01T01:31:37.571173+0000","last_fullsized":"2024-04-01T01:31:37.571173+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:19:05.490103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989593+0000","last_change":"2024-04-01T01:31:16.380207+0000","last_active":"2024-04-01T01:31:22.989593+0000","last_peered":"2024-04-01T01:31:22.989593+0000","last_clean":"2024-04-01T01:31:22.989593+0000","last_became_active":"2024-04-01T01:31:16.379891+0000","last_became_peered":"2024-04-01T01:31:16.379891+0000","last_unstale":"2024-04-01T01:31:22.989593+0000","last_undegraded":"2024-04-01T01:31:22.989593+0000","last_fullsized":"2024-04-01T01:31:22.989593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:12:36.417162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727396+0000","last_change":"2024-04-01T01:31:20.569978+0000","last_active":"2024-04-01T01:31:22.727396+0000","last_peered":"2024-04-01T01:31:22.727396+0000","last_clean":"2024-04-01T01:31:22.727396+0000","last_became_active":"2024-04-01T01:31:20.568298+0000","last_became_peered":"2024-04-01T01:31:20.568298+0000","last_unstale":"2024-04-01T01:31:22.727396+0000","last_undegraded":"2024-04-01T01:31:22.727396+0000","last_fullsized":"2024-04-01T01:31:22.727396+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:42:16.439284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:34.621406+0000","last_change":"2024-04-01T01:31:15.380143+0000","last_active":"2024-04-01T01:31:34.621406+0000","last_peered":"2024-04-01T01:31:34.621406+0000","last_clean":"2024-04-01T01:31:34.621406+0000","last_became_active":"2024-04-01T01:31:15.379835+0000","last_became_peered":"2024-04-01T01:31:15.379835+0000","last_unstale":"2024-04-01T01:31:34.621406+0000","last_undegraded":"2024-04-01T01:31:34.621406+0000","last_fullsized":"2024-04-01T01:31:34.621406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:28:34.213626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563787+0000","last_change":"2024-04-01T01:31:16.380290+0000","last_active":"2024-04-01T01:31:21.563787+0000","last_peered":"2024-04-01T01:31:21.563787+0000","last_clean":"2024-04-01T01:31:21.563787+0000","last_became_active":"2024-04-01T01:31:16.379983+0000","last_became_peered":"2024-04-01T01:31:16.379983+0000","last_unstale":"2024-04-01T01:31:21.563787+0000","last_undegraded":"2024-04-01T01:31:21.563787+0000","last_fullsized":"2024-04-01T01:31:21.563787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:50.135454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.561681+0000","last_change":"2024-04-01T01:31:20.573500+0000","last_active":"2024-04-01T01:31:21.561681+0000","last_peered":"2024-04-01T01:31:21.561681+0000","last_clean":"2024-04-01T01:31:21.561681+0000","last_became_active":"2024-04-01T01:31:20.573101+0000","last_became_peered":"2024-04-01T01:31:20.573101+0000","last_unstale":"2024-04-01T01:31:21.561681+0000","last_undegraded":"2024-04-01T01:31:21.561681+0000","last_fullsized":"2024-04-01T01:31:21.561681+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:11:27.881703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:36.557382+0000","last_change":"2024-04-01T01:31:15.377265+0000","last_active":"2024-04-01T01:31:36.557382+0000","last_peered":"2024-04-01T01:31:36.557382+0000","last_clean":"2024-04-01T01:31:36.557382+0000","last_became_active":"2024-04-01T01:31:15.376889+0000","last_became_peered":"2024-04-01T01:31:15.376889+0000","last_unstale":"2024-04-01T01:31:36.557382+0000","last_undegraded":"2024-04-01T01:31:36.557382+0000","last_fullsized":"2024-04-01T01:31:36.557382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:43:31.582866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563600+0000","last_change":"2024-04-01T01:31:16.384982+0000","last_active":"2024-04-01T01:31:21.563600+0000","last_peered":"2024-04-01T01:31:21.563600+0000","last_clean":"2024-04-01T01:31:21.563600+0000","last_became_active":"2024-04-01T01:31:16.384663+0000","last_became_peered":"2024-04-01T01:31:16.384663+0000","last_unstale":"2024-04-01T01:31:21.563600+0000","last_undegraded":"2024-04-01T01:31:21.563600+0000","last_fullsized":"2024-04-01T01:31:21.563600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:14:57.351768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989927+0000","last_change":"2024-04-01T01:31:20.578761+0000","last_active":"2024-04-01T01:31:22.989927+0000","last_peered":"2024-04-01T01:31:22.989927+0000","last_clean":"2024-04-01T01:31:22.989927+0000","last_became_active":"2024-04-01T01:31:20.578204+0000","last_became_peered":"2024-04-01T01:31:20.578204+0000","last_unstale":"2024-04-01T01:31:22.989927+0000","last_undegraded":"2024-04-01T01:31:22.989927+0000","last_fullsized":"2024-04-01T01:31:22.989927+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:19:33.584872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:36.558677+0000","last_change":"2024-04-01T01:31:15.380773+0000","last_active":"2024-04-01T01:31:36.558677+0000","last_peered":"2024-04-01T01:31:36.558677+0000","last_clean":"2024-04-01T01:31:36.558677+0000","last_became_active":"2024-04-01T01:31:15.380472+0000","last_became_peered":"2024-04-01T01:31:15.380472+0000","last_unstale":"2024-04-01T01:31:36.558677+0000","last_undegraded":"2024-04-01T01:31:36.558677+0000","last_fullsized":"2024-04-01T01:31:36.558677+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:34:28.294500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727565+0000","last_change":"2024-04-01T01:31:16.374658+0000","last_active":"2024-04-01T01:31:22.727565+0000","last_peered":"2024-04-01T01:31:22.727565+0000","last_clean":"2024-04-01T01:31:22.727565+0000","last_became_active":"2024-04-01T01:31:16.373596+0000","last_became_peered":"2024-04-01T01:31:16.373596+0000","last_unstale":"2024-04-01T01:31:22.727565+0000","last_undegraded":"2024-04-01T01:31:22.727565+0000","last_fullsized":"2024-04-01T01:31:22.727565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:28:38.477719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727440+0000","last_change":"2024-04-01T01:31:20.577449+0000","last_active":"2024-04-01T01:31:22.727440+0000","last_peered":"2024-04-01T01:31:22.727440+0000","last_clean":"2024-04-01T01:31:22.727440+0000","last_became_active":"2024-04-01T01:31:20.577026+0000","last_became_peered":"2024-04-01T01:31:20.577026+0000","last_unstale":"2024-04-01T01:31:22.727440+0000","last_undegraded":"2024-04-01T01:31:22.727440+0000","last_fullsized":"2024-04-01T01:31:22.727440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:44:48.080304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:35.545691+0000","last_change":"2024-04-01T01:31:15.373885+0000","last_active":"2024-04-01T01:31:35.545691+0000","last_peered":"2024-04-01T01:31:35.545691+0000","last_clean":"2024-04-01T01:31:35.545691+0000","last_became_active":"2024-04-01T01:31:15.373578+0000","last_became_peered":"2024-04-01T01:31:15.373578+0000","last_unstale":"2024-04-01T01:31:35.545691+0000","last_undegraded":"2024-04-01T01:31:35.545691+0000","last_fullsized":"2024-04-01T01:31:35.545691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:52.641515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727785+0000","last_change":"2024-04-01T01:31:16.379997+0000","last_active":"2024-04-01T01:31:22.727785+0000","last_peered":"2024-04-01T01:31:22.727785+0000","last_clean":"2024-04-01T01:31:22.727785+0000","last_became_active":"2024-04-01T01:31:16.379664+0000","last_became_peered":"2024-04-01T01:31:16.379664+0000","last_unstale":"2024-04-01T01:31:22.727785+0000","last_undegraded":"2024-04-01T01:31:22.727785+0000","last_fullsized":"2024-04-01T01:31:22.727785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:37:17.432095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.151392+0000","last_change":"2024-04-01T01:31:20.569835+0000","last_active":"2024-04-01T01:31:22.151392+0000","last_peered":"2024-04-01T01:31:22.151392+0000","last_clean":"2024-04-01T01:31:22.151392+0000","last_became_active":"2024-04-01T01:31:20.569195+0000","last_became_peered":"2024-04-01T01:31:20.569195+0000","last_unstale":"2024-04-01T01:31:22.151392+0000","last_undegraded":"2024-04-01T01:31:22.151392+0000","last_fullsized":"2024-04-01T01:31:22.151392+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:10:18.278575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.561751+0000","last_change":"2024-04-01T01:31:15.377679+0000","last_active":"2024-04-01T01:31:21.561751+0000","last_peered":"2024-04-01T01:31:21.561751+0000","last_clean":"2024-04-01T01:31:21.561751+0000","last_became_active":"2024-04-01T01:31:15.377351+0000","last_became_peered":"2024-04-01T01:31:15.377351+0000","last_unstale":"2024-04-01T01:31:21.561751+0000","last_undegraded":"2024-04-01T01:31:21.561751+0000","last_fullsized":"2024-04-01T01:31:21.561751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:22:41.216563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.962094+0000","last_change":"2024-04-01T01:31:16.380997+0000","last_active":"2024-04-01T01:31:21.962094+0000","last_peered":"2024-04-01T01:31:21.962094+0000","last_clean":"2024-04-01T01:31:21.962094+0000","last_became_active":"2024-04-01T01:31:16.380692+0000","last_became_peered":"2024-04-01T01:31:16.380692+0000","last_unstale":"2024-04-01T01:31:21.962094+0000","last_undegraded":"2024-04-01T01:31:21.962094+0000","last_fullsized":"2024-04-01T01:31:21.962094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:09:51.220849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989517+0000","last_change":"2024-04-01T01:31:20.573191+0000","last_active":"2024-04-01T01:31:22.989517+0000","last_peered":"2024-04-01T01:31:22.989517+0000","last_clean":"2024-04-01T01:31:22.989517+0000","last_became_active":"2024-04-01T01:31:20.572669+0000","last_became_peered":"2024-04-01T01:31:20.572669+0000","last_unstale":"2024-04-01T01:31:22.989517+0000","last_undegraded":"2024-04-01T01:31:22.989517+0000","last_fullsized":"2024-04-01T01:31:22.989517+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:01:56.491271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563536+0000","last_change":"2024-04-01T01:31:15.379566+0000","last_active":"2024-04-01T01:31:21.563536+0000","last_peered":"2024-04-01T01:31:21.563536+0000","last_clean":"2024-04-01T01:31:21.563536+0000","last_became_active":"2024-04-01T01:31:15.379213+0000","last_became_peered":"2024-04-01T01:31:15.379213+0000","last_unstale":"2024-04-01T01:31:21.563536+0000","last_undegraded":"2024-04-01T01:31:21.563536+0000","last_fullsized":"2024-04-01T01:31:21.563536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:34:17.974422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.961932+0000","last_change":"2024-04-01T01:31:16.376218+0000","last_active":"2024-04-01T01:31:21.961932+0000","last_peered":"2024-04-01T01:31:21.961932+0000","last_clean":"2024-04-01T01:31:21.961932+0000","last_became_active":"2024-04-01T01:31:16.375887+0000","last_became_peered":"2024-04-01T01:31:16.375887+0000","last_unstale":"2024-04-01T01:31:21.961932+0000","last_undegraded":"2024-04-01T01:31:21.961932+0000","last_fullsized":"2024-04-01T01:31:21.961932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:27:03.723827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989853+0000","last_change":"2024-04-01T01:31:20.576461+0000","last_active":"2024-04-01T01:31:22.989853+0000","last_peered":"2024-04-01T01:31:22.989853+0000","last_clean":"2024-04-01T01:31:22.989853+0000","last_became_active":"2024-04-01T01:31:20.575985+0000","last_became_peered":"2024-04-01T01:31:20.575985+0000","last_unstale":"2024-04-01T01:31:22.989853+0000","last_undegraded":"2024-04-01T01:31:22.989853+0000","last_fullsized":"2024-04-01T01:31:22.989853+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:19:02.714455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727658+0000","last_change":"2024-04-01T01:31:16.376456+0000","last_active":"2024-04-01T01:31:22.727658+0000","last_peered":"2024-04-01T01:31:22.727658+0000","last_clean":"2024-04-01T01:31:22.727658+0000","last_became_active":"2024-04-01T01:31:16.375445+0000","last_became_peered":"2024-04-01T01:31:16.375445+0000","last_unstale":"2024-04-01T01:31:22.727658+0000","last_undegraded":"2024-04-01T01:31:22.727658+0000","last_fullsized":"2024-04-01T01:31:22.727658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:07:11.802799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989333+0000","last_change":"2024-04-01T01:31:15.382702+0000","last_active":"2024-04-01T01:31:22.989333+0000","last_peered":"2024-04-01T01:31:22.989333+0000","last_clean":"2024-04-01T01:31:22.989333+0000","last_became_active":"2024-04-01T01:31:15.382377+0000","last_became_peered":"2024-04-01T01:31:15.382377+0000","last_unstale":"2024-04-01T01:31:22.989333+0000","last_undegraded":"2024-04-01T01:31:22.989333+0000","last_fullsized":"2024-04-01T01:31:22.989333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:45.080992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563874+0000","last_change":"2024-04-01T01:31:20.573080+0000","last_active":"2024-04-01T01:31:21.563874+0000","last_peered":"2024-04-01T01:31:21.563874+0000","last_clean":"2024-04-01T01:31:21.563874+0000","last_became_active":"2024-04-01T01:31:20.572656+0000","last_became_peered":"2024-04-01T01:31:20.572656+0000","last_unstale":"2024-04-01T01:31:21.563874+0000","last_undegraded":"2024-04-01T01:31:21.563874+0000","last_fullsized":"2024-04-01T01:31:21.563874+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:09:54.154913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489197+0000","last_change":"2024-04-01T01:31:16.387981+0000","last_active":"2024-04-01T01:31:22.489197+0000","last_peered":"2024-04-01T01:31:22.489197+0000","last_clean":"2024-04-01T01:31:22.489197+0000","last_became_active":"2024-04-01T01:31:16.387594+0000","last_became_peered":"2024-04-01T01:31:16.387594+0000","last_unstale":"2024-04-01T01:31:22.489197+0000","last_undegraded":"2024-04-01T01:31:22.489197+0000","last_fullsized":"2024-04-01T01:31:22.489197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:17.148752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:36.557954+0000","last_change":"2024-04-01T01:31:15.375212+0000","last_active":"2024-04-01T01:31:36.557954+0000","last_peered":"2024-04-01T01:31:36.557954+0000","last_clean":"2024-04-01T01:31:36.557954+0000","last_became_active":"2024-04-01T01:31:15.374882+0000","last_became_peered":"2024-04-01T01:31:15.374882+0000","last_unstale":"2024-04-01T01:31:36.557954+0000","last_undegraded":"2024-04-01T01:31:36.557954+0000","last_fullsized":"2024-04-01T01:31:36.557954+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:56:23.412025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563494+0000","last_change":"2024-04-01T01:30:03.538590+0000","last_active":"2024-04-01T01:31:21.563494+0000","last_peered":"2024-04-01T01:31:21.563494+0000","last_clean":"2024-04-01T01:31:21.563494+0000","last_became_active":"2024-04-01T01:30:03.538098+0000","last_became_peered":"2024-04-01T01:30:03.538098+0000","last_unstale":"2024-04-01T01:31:21.563494+0000","last_undegraded":"2024-04-01T01:31:21.563494+0000","last_fullsized":"2024-04-01T01:31:21.563494+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:26:43.445198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:26:43.445198+0000","last_clean_scrub_stamp":"2024-04-01T01:26:43.445198+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:22.207795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.728450+0000","last_change":"2024-04-01T01:31:20.573260+0000","last_active":"2024-04-01T01:31:22.728450+0000","last_peered":"2024-04-01T01:31:22.728450+0000","last_clean":"2024-04-01T01:31:22.728450+0000","last_became_active":"2024-04-01T01:31:20.572846+0000","last_became_peered":"2024-04-01T01:31:20.572846+0000","last_unstale":"2024-04-01T01:31:22.728450+0000","last_undegraded":"2024-04-01T01:31:22.728450+0000","last_fullsized":"2024-04-01T01:31:22.728450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:24.986070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989259+0000","last_change":"2024-04-01T01:31:16.384782+0000","last_active":"2024-04-01T01:31:22.989259+0000","last_peered":"2024-04-01T01:31:22.989259+0000","last_clean":"2024-04-01T01:31:22.989259+0000","last_became_active":"2024-04-01T01:31:16.384420+0000","last_became_peered":"2024-04-01T01:31:16.384420+0000","last_unstale":"2024-04-01T01:31:22.989259+0000","last_undegraded":"2024-04-01T01:31:22.989259+0000","last_fullsized":"2024-04-01T01:31:22.989259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:00:20.902022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563606+0000","last_change":"2024-04-01T01:31:15.381853+0000","last_active":"2024-04-01T01:31:21.563606+0000","last_peered":"2024-04-01T01:31:21.563606+0000","last_clean":"2024-04-01T01:31:21.563606+0000","last_became_active":"2024-04-01T01:31:15.381496+0000","last_became_peered":"2024-04-01T01:31:15.381496+0000","last_unstale":"2024-04-01T01:31:21.563606+0000","last_undegraded":"2024-04-01T01:31:21.563606+0000","last_fullsized":"2024-04-01T01:31:21.563606+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:07:51.440351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727381+0000","last_change":"2024-04-01T01:31:20.568705+0000","last_active":"2024-04-01T01:31:22.727381+0000","last_peered":"2024-04-01T01:31:22.727381+0000","last_clean":"2024-04-01T01:31:22.727381+0000","last_became_active":"2024-04-01T01:31:20.568252+0000","last_became_peered":"2024-04-01T01:31:20.568252+0000","last_unstale":"2024-04-01T01:31:22.727381+0000","last_undegraded":"2024-04-01T01:31:22.727381+0000","last_fullsized":"2024-04-01T01:31:22.727381+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:22:28.509510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:36.560205+0000","last_change":"2024-04-01T01:31:15.385870+0000","last_active":"2024-04-01T01:31:36.560205+0000","last_peered":"2024-04-01T01:31:36.560205+0000","last_clean":"2024-04-01T01:31:36.560205+0000","last_became_active":"2024-04-01T01:31:15.385332+0000","last_became_peered":"2024-04-01T01:31:15.385332+0000","last_unstale":"2024-04-01T01:31:36.560205+0000","last_undegraded":"2024-04-01T01:31:36.560205+0000","last_fullsized":"2024-04-01T01:31:36.560205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:30:56.350707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989685+0000","last_change":"2024-04-01T01:31:16.377002+0000","last_active":"2024-04-01T01:31:22.989685+0000","last_peered":"2024-04-01T01:31:22.989685+0000","last_clean":"2024-04-01T01:31:22.989685+0000","last_became_active":"2024-04-01T01:31:16.376679+0000","last_became_peered":"2024-04-01T01:31:16.376679+0000","last_unstale":"2024-04-01T01:31:22.989685+0000","last_undegraded":"2024-04-01T01:31:22.989685+0000","last_fullsized":"2024-04-01T01:31:22.989685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:50:24.126192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489240+0000","last_change":"2024-04-01T01:31:20.577757+0000","last_active":"2024-04-01T01:31:22.489240+0000","last_peered":"2024-04-01T01:31:22.489240+0000","last_clean":"2024-04-01T01:31:22.489240+0000","last_became_active":"2024-04-01T01:31:20.577378+0000","last_became_peered":"2024-04-01T01:31:20.577378+0000","last_unstale":"2024-04-01T01:31:22.489240+0000","last_undegraded":"2024-04-01T01:31:22.489240+0000","last_fullsized":"2024-04-01T01:31:22.489240+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:48:04.511229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.561861+0000","last_change":"2024-04-01T01:31:15.383201+0000","last_active":"2024-04-01T01:31:21.561861+0000","last_peered":"2024-04-01T01:31:21.561861+0000","last_clean":"2024-04-01T01:31:21.561861+0000","last_became_active":"2024-04-01T01:31:15.382320+0000","last_became_peered":"2024-04-01T01:31:15.382320+0000","last_unstale":"2024-04-01T01:31:21.561861+0000","last_undegraded":"2024-04-01T01:31:21.561861+0000","last_fullsized":"2024-04-01T01:31:21.561861+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:15:47.138024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489376+0000","last_change":"2024-04-01T01:31:16.386936+0000","last_active":"2024-04-01T01:31:22.489376+0000","last_peered":"2024-04-01T01:31:22.489376+0000","last_clean":"2024-04-01T01:31:22.489376+0000","last_became_active":"2024-04-01T01:31:16.384448+0000","last_became_peered":"2024-04-01T01:31:16.384448+0000","last_unstale":"2024-04-01T01:31:22.489376+0000","last_undegraded":"2024-04-01T01:31:22.489376+0000","last_fullsized":"2024-04-01T01:31:22.489376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:16:42.243250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564083+0000","last_change":"2024-04-01T01:31:20.565570+0000","last_active":"2024-04-01T01:31:21.564083+0000","last_peered":"2024-04-01T01:31:21.564083+0000","last_clean":"2024-04-01T01:31:21.564083+0000","last_became_active":"2024-04-01T01:31:20.565141+0000","last_became_peered":"2024-04-01T01:31:20.565141+0000","last_unstale":"2024-04-01T01:31:21.564083+0000","last_undegraded":"2024-04-01T01:31:21.564083+0000","last_fullsized":"2024-04-01T01:31:21.564083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:47:12.844970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:34.620955+0000","last_change":"2024-04-01T01:31:15.381343+0000","last_active":"2024-04-01T01:31:34.620955+0000","last_peered":"2024-04-01T01:31:34.620955+0000","last_clean":"2024-04-01T01:31:34.620955+0000","last_became_active":"2024-04-01T01:31:15.380994+0000","last_became_peered":"2024-04-01T01:31:15.380994+0000","last_unstale":"2024-04-01T01:31:34.620955+0000","last_undegraded":"2024-04-01T01:31:34.620955+0000","last_fullsized":"2024-04-01T01:31:34.620955+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:19:10.710285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726919+0000","last_change":"2024-04-01T01:31:16.383951+0000","last_active":"2024-04-01T01:31:22.726919+0000","last_peered":"2024-04-01T01:31:22.726919+0000","last_clean":"2024-04-01T01:31:22.726919+0000","last_became_active":"2024-04-01T01:31:16.383657+0000","last_became_peered":"2024-04-01T01:31:16.383657+0000","last_unstale":"2024-04-01T01:31:22.726919+0000","last_undegraded":"2024-04-01T01:31:22.726919+0000","last_fullsized":"2024-04-01T01:31:22.726919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:55:45.216479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727059+0000","last_change":"2024-04-01T01:31:20.569416+0000","last_active":"2024-04-01T01:31:22.727059+0000","last_peered":"2024-04-01T01:31:22.727059+0000","last_clean":"2024-04-01T01:31:22.727059+0000","last_became_active":"2024-04-01T01:31:20.569007+0000","last_became_peered":"2024-04-01T01:31:20.569007+0000","last_unstale":"2024-04-01T01:31:22.727059+0000","last_undegraded":"2024-04-01T01:31:22.727059+0000","last_fullsized":"2024-04-01T01:31:22.727059+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:25:36.770919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726561+0000","last_change":"2024-04-01T01:31:15.373444+0000","last_active":"2024-04-01T01:31:22.726561+0000","last_peered":"2024-04-01T01:31:22.726561+0000","last_clean":"2024-04-01T01:31:22.726561+0000","last_became_active":"2024-04-01T01:31:15.373094+0000","last_became_peered":"2024-04-01T01:31:15.373094+0000","last_unstale":"2024-04-01T01:31:22.726561+0000","last_undegraded":"2024-04-01T01:31:22.726561+0000","last_fullsized":"2024-04-01T01:31:22.726561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:26:27.443832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.151031+0000","last_change":"2024-04-01T01:31:16.381441+0000","last_active":"2024-04-01T01:31:22.151031+0000","last_peered":"2024-04-01T01:31:22.151031+0000","last_clean":"2024-04-01T01:31:22.151031+0000","last_became_active":"2024-04-01T01:31:16.381092+0000","last_became_peered":"2024-04-01T01:31:16.381092+0000","last_unstale":"2024-04-01T01:31:22.151031+0000","last_undegraded":"2024-04-01T01:31:22.151031+0000","last_fullsized":"2024-04-01T01:31:22.151031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:47:00.229090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.562042+0000","last_change":"2024-04-01T01:31:20.569672+0000","last_active":"2024-04-01T01:31:21.562042+0000","last_peered":"2024-04-01T01:31:21.562042+0000","last_clean":"2024-04-01T01:31:21.562042+0000","last_became_active":"2024-04-01T01:31:20.569276+0000","last_became_peered":"2024-04-01T01:31:20.569276+0000","last_unstale":"2024-04-01T01:31:21.562042+0000","last_undegraded":"2024-04-01T01:31:21.562042+0000","last_fullsized":"2024-04-01T01:31:21.562042+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:39:03.331428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:36.556961+0000","last_change":"2024-04-01T01:31:15.376094+0000","last_active":"2024-04-01T01:31:36.556961+0000","last_peered":"2024-04-01T01:31:36.556961+0000","last_clean":"2024-04-01T01:31:36.556961+0000","last_became_active":"2024-04-01T01:31:15.375169+0000","last_became_peered":"2024-04-01T01:31:15.375169+0000","last_unstale":"2024-04-01T01:31:36.556961+0000","last_undegraded":"2024-04-01T01:31:36.556961+0000","last_fullsized":"2024-04-01T01:31:36.556961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:18:20.710644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726996+0000","last_change":"2024-04-01T01:31:16.378514+0000","last_active":"2024-04-01T01:31:22.726996+0000","last_peered":"2024-04-01T01:31:22.726996+0000","last_clean":"2024-04-01T01:31:22.726996+0000","last_became_active":"2024-04-01T01:31:16.377990+0000","last_became_peered":"2024-04-01T01:31:16.377990+0000","last_unstale":"2024-04-01T01:31:22.726996+0000","last_undegraded":"2024-04-01T01:31:22.726996+0000","last_fullsized":"2024-04-01T01:31:22.726996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:27:29.829775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726745+0000","last_change":"2024-04-01T01:31:20.567892+0000","last_active":"2024-04-01T01:31:22.726745+0000","last_peered":"2024-04-01T01:31:22.726745+0000","last_clean":"2024-04-01T01:31:22.726745+0000","last_became_active":"2024-04-01T01:31:20.567525+0000","last_became_peered":"2024-04-01T01:31:20.567525+0000","last_unstale":"2024-04-01T01:31:22.726745+0000","last_undegraded":"2024-04-01T01:31:22.726745+0000","last_fullsized":"2024-04-01T01:31:22.726745+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:57.828691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:35.615739+0000","last_change":"2024-04-01T01:31:15.373626+0000","last_active":"2024-04-01T01:31:35.615739+0000","last_peered":"2024-04-01T01:31:35.615739+0000","last_clean":"2024-04-01T01:31:35.615739+0000","last_became_active":"2024-04-01T01:31:15.373302+0000","last_became_peered":"2024-04-01T01:31:15.373302+0000","last_unstale":"2024-04-01T01:31:35.615739+0000","last_undegraded":"2024-04-01T01:31:35.615739+0000","last_fullsized":"2024-04-01T01:31:35.615739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:07:41.982481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489333+0000","last_change":"2024-04-01T01:31:16.387570+0000","last_active":"2024-04-01T01:31:22.489333+0000","last_peered":"2024-04-01T01:31:22.489333+0000","last_clean":"2024-04-01T01:31:22.489333+0000","last_became_active":"2024-04-01T01:31:16.387200+0000","last_became_peered":"2024-04-01T01:31:16.387200+0000","last_unstale":"2024-04-01T01:31:22.489333+0000","last_undegraded":"2024-04-01T01:31:22.489333+0000","last_fullsized":"2024-04-01T01:31:22.489333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:52:27.813722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727106+0000","last_change":"2024-04-01T01:31:20.569668+0000","last_active":"2024-04-01T01:31:22.727106+0000","last_peered":"2024-04-01T01:31:22.727106+0000","last_clean":"2024-04-01T01:31:22.727106+0000","last_became_active":"2024-04-01T01:31:20.569148+0000","last_became_peered":"2024-04-01T01:31:20.569148+0000","last_unstale":"2024-04-01T01:31:22.727106+0000","last_undegraded":"2024-04-01T01:31:22.727106+0000","last_fullsized":"2024-04-01T01:31:22.727106+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:20:56.048052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:34.621552+0000","last_change":"2024-04-01T01:31:15.379323+0000","last_active":"2024-04-01T01:31:34.621552+0000","last_peered":"2024-04-01T01:31:34.621552+0000","last_clean":"2024-04-01T01:31:34.621552+0000","last_became_active":"2024-04-01T01:31:15.379023+0000","last_became_peered":"2024-04-01T01:31:15.379023+0000","last_unstale":"2024-04-01T01:31:34.621552+0000","last_undegraded":"2024-04-01T01:31:34.621552+0000","last_fullsized":"2024-04-01T01:31:34.621552+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:05:37.709040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563654+0000","last_change":"2024-04-01T01:31:16.387250+0000","last_active":"2024-04-01T01:31:21.563654+0000","last_peered":"2024-04-01T01:31:21.563654+0000","last_clean":"2024-04-01T01:31:21.563654+0000","last_became_active":"2024-04-01T01:31:16.386940+0000","last_became_peered":"2024-04-01T01:31:16.386940+0000","last_unstale":"2024-04-01T01:31:21.563654+0000","last_undegraded":"2024-04-01T01:31:21.563654+0000","last_fullsized":"2024-04-01T01:31:21.563654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:19:34.886738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726827+0000","last_change":"2024-04-01T01:31:20.573228+0000","last_active":"2024-04-01T01:31:22.726827+0000","last_peered":"2024-04-01T01:31:22.726827+0000","last_clean":"2024-04-01T01:31:22.726827+0000","last_became_active":"2024-04-01T01:31:20.572681+0000","last_became_peered":"2024-04-01T01:31:20.572681+0000","last_unstale":"2024-04-01T01:31:22.726827+0000","last_undegraded":"2024-04-01T01:31:22.726827+0000","last_fullsized":"2024-04-01T01:31:22.726827+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:58:17.812966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:36.556687+0000","last_change":"2024-04-01T01:31:15.381477+0000","last_active":"2024-04-01T01:31:36.556687+0000","last_peered":"2024-04-01T01:31:36.556687+0000","last_clean":"2024-04-01T01:31:36.556687+0000","last_became_active":"2024-04-01T01:31:15.381166+0000","last_became_peered":"2024-04-01T01:31:15.381166+0000","last_unstale":"2024-04-01T01:31:36.556687+0000","last_undegraded":"2024-04-01T01:31:36.556687+0000","last_fullsized":"2024-04-01T01:31:36.556687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:19.029896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564006+0000","last_change":"2024-04-01T01:31:16.386234+0000","last_active":"2024-04-01T01:31:21.564006+0000","last_peered":"2024-04-01T01:31:21.564006+0000","last_clean":"2024-04-01T01:31:21.564006+0000","last_became_active":"2024-04-01T01:31:16.385876+0000","last_became_peered":"2024-04-01T01:31:16.385876+0000","last_unstale":"2024-04-01T01:31:21.564006+0000","last_undegraded":"2024-04-01T01:31:21.564006+0000","last_fullsized":"2024-04-01T01:31:21.564006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:34.187210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726675+0000","last_change":"2024-04-01T01:31:20.580499+0000","last_active":"2024-04-01T01:31:22.726675+0000","last_peered":"2024-04-01T01:31:22.726675+0000","last_clean":"2024-04-01T01:31:22.726675+0000","last_became_active":"2024-04-01T01:31:20.579796+0000","last_became_peered":"2024-04-01T01:31:20.579796+0000","last_unstale":"2024-04-01T01:31:22.726675+0000","last_undegraded":"2024-04-01T01:31:22.726675+0000","last_fullsized":"2024-04-01T01:31:22.726675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:59:10.852056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.728390+0000","last_change":"2024-04-01T01:31:15.378503+0000","last_active":"2024-04-01T01:31:22.728390+0000","last_peered":"2024-04-01T01:31:22.728390+0000","last_clean":"2024-04-01T01:31:22.728390+0000","last_became_active":"2024-04-01T01:31:15.378152+0000","last_became_peered":"2024-04-01T01:31:15.378152+0000","last_unstale":"2024-04-01T01:31:22.728390+0000","last_undegraded":"2024-04-01T01:31:22.728390+0000","last_fullsized":"2024-04-01T01:31:22.728390+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:15:42.118691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563902+0000","last_change":"2024-04-01T01:31:16.389676+0000","last_active":"2024-04-01T01:31:21.563902+0000","last_peered":"2024-04-01T01:31:21.563902+0000","last_clean":"2024-04-01T01:31:21.563902+0000","last_became_active":"2024-04-01T01:31:16.389304+0000","last_became_peered":"2024-04-01T01:31:16.389304+0000","last_unstale":"2024-04-01T01:31:21.563902+0000","last_undegraded":"2024-04-01T01:31:21.563902+0000","last_fullsized":"2024-04-01T01:31:21.563902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:20:24.021273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727031+0000","last_change":"2024-04-01T01:31:20.569461+0000","last_active":"2024-04-01T01:31:22.727031+0000","last_peered":"2024-04-01T01:31:22.727031+0000","last_clean":"2024-04-01T01:31:22.727031+0000","last_became_active":"2024-04-01T01:31:20.569011+0000","last_became_peered":"2024-04-01T01:31:20.569011+0000","last_unstale":"2024-04-01T01:31:22.727031+0000","last_undegraded":"2024-04-01T01:31:22.727031+0000","last_fullsized":"2024-04-01T01:31:22.727031+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:43:43.657065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989896+0000","last_change":"2024-04-01T01:31:15.384181+0000","last_active":"2024-04-01T01:31:22.989896+0000","last_peered":"2024-04-01T01:31:22.989896+0000","last_clean":"2024-04-01T01:31:22.989896+0000","last_became_active":"2024-04-01T01:31:15.383886+0000","last_became_peered":"2024-04-01T01:31:15.383886+0000","last_unstale":"2024-04-01T01:31:22.989896+0000","last_undegraded":"2024-04-01T01:31:22.989896+0000","last_fullsized":"2024-04-01T01:31:22.989896+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:43:28.414423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564097+0000","last_change":"2024-04-01T01:31:16.381877+0000","last_active":"2024-04-01T01:31:21.564097+0000","last_peered":"2024-04-01T01:31:21.564097+0000","last_clean":"2024-04-01T01:31:21.564097+0000","last_became_active":"2024-04-01T01:31:16.381558+0000","last_became_peered":"2024-04-01T01:31:16.381558+0000","last_unstale":"2024-04-01T01:31:21.564097+0000","last_undegraded":"2024-04-01T01:31:21.564097+0000","last_fullsized":"2024-04-01T01:31:21.564097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:24.120910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727030+0000","last_change":"2024-04-01T01:31:16.380116+0000","last_active":"2024-04-01T01:31:22.727030+0000","last_peered":"2024-04-01T01:31:22.727030+0000","last_clean":"2024-04-01T01:31:22.727030+0000","last_became_active":"2024-04-01T01:31:16.378936+0000","last_became_peered":"2024-04-01T01:31:16.378936+0000","last_unstale":"2024-04-01T01:31:22.727030+0000","last_undegraded":"2024-04-01T01:31:22.727030+0000","last_fullsized":"2024-04-01T01:31:22.727030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:18:16.211113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564540+0000","last_change":"2024-04-01T01:31:16.384872+0000","last_active":"2024-04-01T01:31:21.564540+0000","last_peered":"2024-04-01T01:31:21.564540+0000","last_clean":"2024-04-01T01:31:21.564540+0000","last_became_active":"2024-04-01T01:31:16.384406+0000","last_became_peered":"2024-04-01T01:31:16.384406+0000","last_unstale":"2024-04-01T01:31:21.564540+0000","last_undegraded":"2024-04-01T01:31:21.564540+0000","last_fullsized":"2024-04-01T01:31:21.564540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:13:36.567357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564110+0000","last_change":"2024-04-01T01:31:16.384561+0000","last_active":"2024-04-01T01:31:21.564110+0000","last_peered":"2024-04-01T01:31:21.564110+0000","last_clean":"2024-04-01T01:31:21.564110+0000","last_became_active":"2024-04-01T01:31:16.384221+0000","last_became_peered":"2024-04-01T01:31:16.384221+0000","last_unstale":"2024-04-01T01:31:21.564110+0000","last_undegraded":"2024-04-01T01:31:21.564110+0000","last_fullsized":"2024-04-01T01:31:21.564110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:17:56.490000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726795+0000","last_change":"2024-04-01T01:31:16.385464+0000","last_active":"2024-04-01T01:31:22.726795+0000","last_peered":"2024-04-01T01:31:22.726795+0000","last_clean":"2024-04-01T01:31:22.726795+0000","last_became_active":"2024-04-01T01:31:16.385159+0000","last_became_peered":"2024-04-01T01:31:16.385159+0000","last_unstale":"2024-04-01T01:31:22.726795+0000","last_undegraded":"2024-04-01T01:31:22.726795+0000","last_fullsized":"2024-04-01T01:31:22.726795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:33:53.852957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989615+0000","last_change":"2024-04-01T01:31:16.385688+0000","last_active":"2024-04-01T01:31:22.989615+0000","last_peered":"2024-04-01T01:31:22.989615+0000","last_clean":"2024-04-01T01:31:22.989615+0000","last_became_active":"2024-04-01T01:31:16.385381+0000","last_became_peered":"2024-04-01T01:31:16.385381+0000","last_unstale":"2024-04-01T01:31:22.989615+0000","last_undegraded":"2024-04-01T01:31:22.989615+0000","last_fullsized":"2024-04-01T01:31:22.989615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:39:36.810766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727844+0000","last_change":"2024-04-01T01:31:16.379976+0000","last_active":"2024-04-01T01:31:22.727844+0000","last_peered":"2024-04-01T01:31:22.727844+0000","last_clean":"2024-04-01T01:31:22.727844+0000","last_became_active":"2024-04-01T01:31:16.379553+0000","last_became_peered":"2024-04-01T01:31:16.379553+0000","last_unstale":"2024-04-01T01:31:22.727844+0000","last_undegraded":"2024-04-01T01:31:22.727844+0000","last_fullsized":"2024-04-01T01:31:22.727844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:48:23.555723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564765+0000","last_change":"2024-04-01T01:31:16.383995+0000","last_active":"2024-04-01T01:31:21.564765+0000","last_peered":"2024-04-01T01:31:21.564765+0000","last_clean":"2024-04-01T01:31:21.564765+0000","last_became_active":"2024-04-01T01:31:16.383673+0000","last_became_peered":"2024-04-01T01:31:16.383673+0000","last_unstale":"2024-04-01T01:31:21.564765+0000","last_undegraded":"2024-04-01T01:31:21.564765+0000","last_fullsized":"2024-04-01T01:31:21.564765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:03:16.192438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489485+0000","last_change":"2024-04-01T01:31:16.389969+0000","last_active":"2024-04-01T01:31:22.489485+0000","last_peered":"2024-04-01T01:31:22.489485+0000","last_clean":"2024-04-01T01:31:22.489485+0000","last_became_active":"2024-04-01T01:31:16.389640+0000","last_became_peered":"2024-04-01T01:31:16.389640+0000","last_unstale":"2024-04-01T01:31:22.489485+0000","last_undegraded":"2024-04-01T01:31:22.489485+0000","last_fullsized":"2024-04-01T01:31:22.489485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:35:15.226960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727211+0000","last_change":"2024-04-01T01:31:16.382962+0000","last_active":"2024-04-01T01:31:22.727211+0000","last_peered":"2024-04-01T01:31:22.727211+0000","last_clean":"2024-04-01T01:31:22.727211+0000","last_became_active":"2024-04-01T01:31:16.382653+0000","last_became_peered":"2024-04-01T01:31:16.382653+0000","last_unstale":"2024-04-01T01:31:22.727211+0000","last_undegraded":"2024-04-01T01:31:22.727211+0000","last_fullsized":"2024-04-01T01:31:22.727211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:53.054884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489203+0000","last_change":"2024-04-01T01:31:16.382997+0000","last_active":"2024-04-01T01:31:22.489203+0000","last_peered":"2024-04-01T01:31:22.489203+0000","last_clean":"2024-04-01T01:31:22.489203+0000","last_became_active":"2024-04-01T01:31:16.380180+0000","last_became_peered":"2024-04-01T01:31:16.380180+0000","last_unstale":"2024-04-01T01:31:22.489203+0000","last_undegraded":"2024-04-01T01:31:22.489203+0000","last_fullsized":"2024-04-01T01:31:22.489203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:29:51.990395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727267+0000","last_change":"2024-04-01T01:31:16.382948+0000","last_active":"2024-04-01T01:31:22.727267+0000","last_peered":"2024-04-01T01:31:22.727267+0000","last_clean":"2024-04-01T01:31:22.727267+0000","last_became_active":"2024-04-01T01:31:16.382134+0000","last_became_peered":"2024-04-01T01:31:16.382134+0000","last_unstale":"2024-04-01T01:31:22.727267+0000","last_undegraded":"2024-04-01T01:31:22.727267+0000","last_fullsized":"2024-04-01T01:31:22.727267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:24:08.515063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989688+0000","last_change":"2024-04-01T01:31:16.378841+0000","last_active":"2024-04-01T01:31:22.989688+0000","last_peered":"2024-04-01T01:31:22.989688+0000","last_clean":"2024-04-01T01:31:22.989688+0000","last_became_active":"2024-04-01T01:31:16.378501+0000","last_became_peered":"2024-04-01T01:31:16.378501+0000","last_unstale":"2024-04-01T01:31:22.989688+0000","last_undegraded":"2024-04-01T01:31:22.989688+0000","last_fullsized":"2024-04-01T01:31:22.989688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:50:20.881852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727402+0000","last_change":"2024-04-01T01:31:16.380979+0000","last_active":"2024-04-01T01:31:22.727402+0000","last_peered":"2024-04-01T01:31:22.727402+0000","last_clean":"2024-04-01T01:31:22.727402+0000","last_became_active":"2024-04-01T01:31:16.380625+0000","last_became_peered":"2024-04-01T01:31:16.380625+0000","last_unstale":"2024-04-01T01:31:22.727402+0000","last_undegraded":"2024-04-01T01:31:22.727402+0000","last_fullsized":"2024-04-01T01:31:22.727402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:09:03.246239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8678,"num_objects":67,"num_object_clones":0,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":17,"num_read_kb":18,"num_write":140,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":42864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383697,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36912,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708240,"statfs":{"total":95995035648,"available":95957237760,"internally_reserved":0,"allocated":1613824,"data_stored":1072381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808859,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":987136,"data_stored":477797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070496,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":478715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36888,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708264,"statfs":{"total":95995035648,"available":95957262336,"internally_reserved":0,"allocated":1581056,"data_stored":1068165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":481057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":478693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":483029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":477499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920819,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28584,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716568,"statfs":{"total":95995035648,"available":95965765632,"internally_reserved":0,"allocated":1605632,"data_stored":1069059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":25504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-01T01:31:39.937 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T01:31:39.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:39 smithi189 ceph-mon[35753]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:31:39.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:39 smithi189 ceph-mon[35753]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:31:39.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:39 smithi189 ceph-mon[35753]: pgmap v299: 129 pgs: 129 active+clean; 585 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-01T01:31:39.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:39 smithi189 ceph-mon[35753]: from='client.24716 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:31:39.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:39 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2170485849' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:39.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:39 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/264341845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.073+0000 7f205139a640 1 -- 172.21.15.70:0/3782572368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f204c101880 msgr2=0x7f204c101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.073+0000 7f205139a640 1 --2- 172.21.15.70:0/3782572368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f204c101880 0x7f204c101d00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2034009900 tx=0x7f203402bc30 comp rx=0 tx=0).stop 2024-04-01T01:31:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.074+0000 7f205139a640 1 -- 172.21.15.70:0/3782572368 shutdown_connections 2024-04-01T01:31:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.074+0000 7f205139a640 1 --2- 172.21.15.70:0/3782572368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f204c102240 0x7f204c108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.074+0000 7f205139a640 1 --2- 172.21.15.70:0/3782572368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f204c101880 0x7f204c101d00 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.074+0000 7f205139a640 1 --2- 172.21.15.70:0/3782572368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f204c100690 0x7f204c100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.074+0000 7f205139a640 1 -- 172.21.15.70:0/3782572368 >> 172.21.15.70:0/3782572368 conn(0x7f204c0fbc00 msgr2=0x7f204c0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.074+0000 7f205139a640 1 -- 172.21.15.70:0/3782572368 shutdown_connections 2024-04-01T01:31:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.074+0000 7f205139a640 1 -- 172.21.15.70:0/3782572368 wait complete. 2024-04-01T01:31:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.074+0000 7f205139a640 1 Processor -- start 2024-04-01T01:31:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.074+0000 7f205139a640 1 -- start start 2024-04-01T01:31:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.074+0000 7f205139a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f204c100690 0x7f204c1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.074+0000 7f205139a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f204c101880 0x7f204c197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.074+0000 7f205139a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f204c102240 0x7f204c19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.074+0000 7f205139a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f204c10c1a0 con 0x7f204c100690 2024-04-01T01:31:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.074+0000 7f205139a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f204c10c020 con 0x7f204c102240 2024-04-01T01:31:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.074+0000 7f205139a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f204c10c320 con 0x7f204c101880 2024-04-01T01:31:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f204affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f204c100690 0x7f204c1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f204affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f204c100690 0x7f204c1979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51672/0 (socket says 172.21.15.70:51672) 2024-04-01T01:31:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f204affd640 1 -- 172.21.15.70:0/3873689460 learned_addr learned my addr 172.21.15.70:0/3873689460 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f204b7fe640 1 --2- 172.21.15.70:0/3873689460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f204c102240 0x7f204c19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f204a7fc640 1 --2- 172.21.15.70:0/3873689460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f204c101880 0x7f204c197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f204affd640 1 -- 172.21.15.70:0/3873689460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f204c101880 msgr2=0x7f204c197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f204affd640 1 --2- 172.21.15.70:0/3873689460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f204c101880 0x7f204c197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:40.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f204affd640 1 -- 172.21.15.70:0/3873689460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f204c102240 msgr2=0x7f204c19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:40.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f204affd640 1 --2- 172.21.15.70:0/3873689460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f204c102240 0x7f204c19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:40.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f204affd640 1 -- 172.21.15.70:0/3873689460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f204c19f5c0 con 0x7f204c100690 2024-04-01T01:31:40.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f204b7fe640 1 --2- 172.21.15.70:0/3873689460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f204c102240 0x7f204c19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:31:40.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f204affd640 1 --2- 172.21.15.70:0/3873689460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f204c100690 0x7f204c1979a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f204400ebc0 tx=0x7f204400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:40.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f2043fff640 1 -- 172.21.15.70:0/3873689460 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044003b70 con 0x7f204c100690 2024-04-01T01:31:40.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f2043fff640 1 -- 172.21.15.70:0/3873689460 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2044003d10 con 0x7f204c100690 2024-04-01T01:31:40.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f205139a640 1 -- 172.21.15.70:0/3873689460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f204c19f8b0 con 0x7f204c100690 2024-04-01T01:31:40.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f2043fff640 1 -- 172.21.15.70:0/3873689460 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044019840 con 0x7f204c100690 2024-04-01T01:31:40.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.075+0000 7f205139a640 1 -- 172.21.15.70:0/3873689460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f204c19fdd0 con 0x7f204c100690 2024-04-01T01:31:40.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.076+0000 7f205139a640 1 -- 172.21.15.70:0/3873689460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f204c1a0030 con 0x7f204c100690 2024-04-01T01:31:40.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:39 smithi081 ceph-mon[35142]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:31:40.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:39 smithi081 ceph-mon[35142]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:31:40.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:39 smithi081 ceph-mon[35142]: pgmap v299: 129 pgs: 129 active+clean; 585 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-04-01T01:31:40.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:39 smithi081 ceph-mon[35142]: from='client.24716 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:31:40.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:39 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2170485849' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:31:40.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:39 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/264341845' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:40.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.077+0000 7f2043fff640 1 -- 172.21.15.70:0/3873689460 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f20440199e0 con 0x7f204c100690 2024-04-01T01:31:40.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.078+0000 7f2043fff640 1 --2- 172.21.15.70:0/3873689460 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f20380a2530 0x7f20380a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:40.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.078+0000 7f204a7fc640 1 --2- 172.21.15.70:0/3873689460 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f20380a2530 0x7f20380a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:40.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.078+0000 7f2043fff640 1 -- 172.21.15.70:0/3873689460 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f20440cf540 con 0x7f204c100690 2024-04-01T01:31:40.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.078+0000 7f204a7fc640 1 --2- 172.21.15.70:0/3873689460 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f20380a2530 0x7f20380a49f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2034009230 tx=0x7f2034005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:40.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.081+0000 7f2043fff640 1 -- 172.21.15.70:0/3873689460 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f204409c3f0 con 0x7f204c100690 2024-04-01T01:31:40.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.252+0000 7f205139a640 1 -- 172.21.15.70:0/3873689460 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f204c1a0200 con 0x7f20380a2530 2024-04-01T01:31:40.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.261+0000 7f2043fff640 1 -- 172.21.15.70:0/3873689460 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340300 (secure 0 0 0) 0x7f204c1a0200 con 0x7f20380a2530 2024-04-01T01:31:40.264 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:40.266 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-01T01:31:40.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.265+0000 7f205139a640 1 -- 172.21.15.70:0/3873689460 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f20380a2530 msgr2=0x7f20380a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:40.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.265+0000 7f205139a640 1 --2- 172.21.15.70:0/3873689460 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f20380a2530 0x7f20380a49f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2034009230 tx=0x7f2034005fd0 comp rx=0 tx=0).stop 2024-04-01T01:31:40.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.265+0000 7f205139a640 1 -- 172.21.15.70:0/3873689460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f204c100690 msgr2=0x7f204c1979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:40.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.265+0000 7f205139a640 1 --2- 172.21.15.70:0/3873689460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f204c100690 0x7f204c1979a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f204400ebc0 tx=0x7f204400c6a0 comp rx=0 tx=0).stop 2024-04-01T01:31:40.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.265+0000 7f205139a640 1 -- 172.21.15.70:0/3873689460 shutdown_connections 2024-04-01T01:31:40.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.265+0000 7f205139a640 1 --2- 172.21.15.70:0/3873689460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f204c102240 0x7f204c19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:40.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.265+0000 7f205139a640 1 --2- 172.21.15.70:0/3873689460 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f20380a2530 0x7f20380a49f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:40.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.265+0000 7f205139a640 1 --2- 172.21.15.70:0/3873689460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f204c101880 0x7f204c197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:40.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.265+0000 7f205139a640 1 --2- 172.21.15.70:0/3873689460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f204c100690 0x7f204c1979a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:40.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.265+0000 7f205139a640 1 -- 172.21.15.70:0/3873689460 >> 172.21.15.70:0/3873689460 conn(0x7f204c0fbc00 msgr2=0x7f204c0fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:40.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.265+0000 7f205139a640 1 -- 172.21.15.70:0/3873689460 shutdown_connections 2024-04-01T01:31:40.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:40.265+0000 7f205139a640 1 -- 172.21.15.70:0/3873689460 wait complete. 2024-04-01T01:31:40.275 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-04-01T01:31:38.561581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":599046,"num_objects":69,"num_object_clones":0,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":143,"num_read_kb":127,"num_write":321,"num_write_kb":2985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261,"ondisk_log_size":261,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":358488,"kb_used_data":13476,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124583336,"statfs":{"total":1151940427776,"available":1151573336064,"internally_reserved":0,"allocated":13799424,"data_stored":7518684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6268,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":17,"num_read_kb":18,"num_write":95,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003192"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.728117+0000","last_change":"2024-04-01T01:31:16.385517+0000","last_active":"2024-04-01T01:31:22.728117+0000","last_peered":"2024-04-01T01:31:22.728117+0000","last_clean":"2024-04-01T01:31:22.728117+0000","last_became_active":"2024-04-01T01:31:16.384682+0000","last_became_peered":"2024-04-01T01:31:16.384682+0000","last_unstale":"2024-04-01T01:31:22.728117+0000","last_undegraded":"2024-04-01T01:31:22.728117+0000","last_fullsized":"2024-04-01T01:31:22.728117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:35:47.614041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726824+0000","last_change":"2024-04-01T01:31:16.376119+0000","last_active":"2024-04-01T01:31:22.726824+0000","last_peered":"2024-04-01T01:31:22.726824+0000","last_clean":"2024-04-01T01:31:22.726824+0000","last_became_active":"2024-04-01T01:31:16.375801+0000","last_became_peered":"2024-04-01T01:31:16.375801+0000","last_unstale":"2024-04-01T01:31:22.726824+0000","last_undegraded":"2024-04-01T01:31:22.726824+0000","last_fullsized":"2024-04-01T01:31:22.726824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:34.120487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726630+0000","last_change":"2024-04-01T01:31:16.378158+0000","last_active":"2024-04-01T01:31:22.726630+0000","last_peered":"2024-04-01T01:31:22.726630+0000","last_clean":"2024-04-01T01:31:22.726630+0000","last_became_active":"2024-04-01T01:31:16.377831+0000","last_became_peered":"2024-04-01T01:31:16.377831+0000","last_unstale":"2024-04-01T01:31:22.726630+0000","last_undegraded":"2024-04-01T01:31:22.726630+0000","last_fullsized":"2024-04-01T01:31:22.726630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:30:06.487545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726977+0000","last_change":"2024-04-01T01:31:16.389708+0000","last_active":"2024-04-01T01:31:22.726977+0000","last_peered":"2024-04-01T01:31:22.726977+0000","last_clean":"2024-04-01T01:31:22.726977+0000","last_became_active":"2024-04-01T01:31:16.389409+0000","last_became_peered":"2024-04-01T01:31:16.389409+0000","last_unstale":"2024-04-01T01:31:22.726977+0000","last_undegraded":"2024-04-01T01:31:22.726977+0000","last_fullsized":"2024-04-01T01:31:22.726977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:57:49.215988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.151166+0000","last_change":"2024-04-01T01:31:16.385996+0000","last_active":"2024-04-01T01:31:22.151166+0000","last_peered":"2024-04-01T01:31:22.151166+0000","last_clean":"2024-04-01T01:31:22.151166+0000","last_became_active":"2024-04-01T01:31:16.385648+0000","last_became_peered":"2024-04-01T01:31:16.385648+0000","last_unstale":"2024-04-01T01:31:22.151166+0000","last_undegraded":"2024-04-01T01:31:22.151166+0000","last_fullsized":"2024-04-01T01:31:22.151166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:14:56.595595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.961915+0000","last_change":"2024-04-01T01:31:16.376519+0000","last_active":"2024-04-01T01:31:21.961915+0000","last_peered":"2024-04-01T01:31:21.961915+0000","last_clean":"2024-04-01T01:31:21.961915+0000","last_became_active":"2024-04-01T01:31:16.376178+0000","last_became_peered":"2024-04-01T01:31:16.376178+0000","last_unstale":"2024-04-01T01:31:21.961915+0000","last_undegraded":"2024-04-01T01:31:21.961915+0000","last_fullsized":"2024-04-01T01:31:21.961915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:01:45.861899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563654+0000","last_change":"2024-04-01T01:31:16.383939+0000","last_active":"2024-04-01T01:31:21.563654+0000","last_peered":"2024-04-01T01:31:21.563654+0000","last_clean":"2024-04-01T01:31:21.563654+0000","last_became_active":"2024-04-01T01:31:16.383626+0000","last_became_peered":"2024-04-01T01:31:16.383626+0000","last_unstale":"2024-04-01T01:31:21.563654+0000","last_undegraded":"2024-04-01T01:31:21.563654+0000","last_fullsized":"2024-04-01T01:31:21.563654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:36:05.764031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.488942+0000","last_change":"2024-04-01T01:31:16.384007+0000","last_active":"2024-04-01T01:31:22.488942+0000","last_peered":"2024-04-01T01:31:22.488942+0000","last_clean":"2024-04-01T01:31:22.488942+0000","last_became_active":"2024-04-01T01:31:16.383687+0000","last_became_peered":"2024-04-01T01:31:16.383687+0000","last_unstale":"2024-04-01T01:31:22.488942+0000","last_undegraded":"2024-04-01T01:31:22.488942+0000","last_fullsized":"2024-04-01T01:31:22.488942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:14:17.593402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727748+0000","last_change":"2024-04-01T01:31:16.380357+0000","last_active":"2024-04-01T01:31:22.727748+0000","last_peered":"2024-04-01T01:31:22.727748+0000","last_clean":"2024-04-01T01:31:22.727748+0000","last_became_active":"2024-04-01T01:31:16.380046+0000","last_became_peered":"2024-04-01T01:31:16.380046+0000","last_unstale":"2024-04-01T01:31:22.727748+0000","last_undegraded":"2024-04-01T01:31:22.727748+0000","last_fullsized":"2024-04-01T01:31:22.727748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:51:53.887768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.962202+0000","last_change":"2024-04-01T01:31:16.378551+0000","last_active":"2024-04-01T01:31:21.962202+0000","last_peered":"2024-04-01T01:31:21.962202+0000","last_clean":"2024-04-01T01:31:21.962202+0000","last_became_active":"2024-04-01T01:31:16.378234+0000","last_became_peered":"2024-04-01T01:31:16.378234+0000","last_unstale":"2024-04-01T01:31:21.962202+0000","last_undegraded":"2024-04-01T01:31:21.962202+0000","last_fullsized":"2024-04-01T01:31:21.962202+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:37:45.998115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564002+0000","last_change":"2024-04-01T01:31:16.388755+0000","last_active":"2024-04-01T01:31:21.564002+0000","last_peered":"2024-04-01T01:31:21.564002+0000","last_clean":"2024-04-01T01:31:21.564002+0000","last_became_active":"2024-04-01T01:31:16.388434+0000","last_became_peered":"2024-04-01T01:31:16.388434+0000","last_unstale":"2024-04-01T01:31:21.564002+0000","last_undegraded":"2024-04-01T01:31:21.564002+0000","last_fullsized":"2024-04-01T01:31:21.564002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:03:07.168110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726864+0000","last_change":"2024-04-01T01:31:16.390478+0000","last_active":"2024-04-01T01:31:22.726864+0000","last_peered":"2024-04-01T01:31:22.726864+0000","last_clean":"2024-04-01T01:31:22.726864+0000","last_became_active":"2024-04-01T01:31:16.390013+0000","last_became_peered":"2024-04-01T01:31:16.390013+0000","last_unstale":"2024-04-01T01:31:22.726864+0000","last_undegraded":"2024-04-01T01:31:22.726864+0000","last_fullsized":"2024-04-01T01:31:22.726864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:28:21.677527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989589+0000","last_change":"2024-04-01T01:31:16.366921+0000","last_active":"2024-04-01T01:31:22.989589+0000","last_peered":"2024-04-01T01:31:22.989589+0000","last_clean":"2024-04-01T01:31:22.989589+0000","last_became_active":"2024-04-01T01:31:16.366098+0000","last_became_peered":"2024-04-01T01:31:16.366098+0000","last_unstale":"2024-04-01T01:31:22.989589+0000","last_undegraded":"2024-04-01T01:31:22.989589+0000","last_fullsized":"2024-04-01T01:31:22.989589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:29.456263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726607+0000","last_change":"2024-04-01T01:31:16.379178+0000","last_active":"2024-04-01T01:31:22.726607+0000","last_peered":"2024-04-01T01:31:22.726607+0000","last_clean":"2024-04-01T01:31:22.726607+0000","last_became_active":"2024-04-01T01:31:16.378877+0000","last_became_peered":"2024-04-01T01:31:16.378877+0000","last_unstale":"2024-04-01T01:31:22.726607+0000","last_undegraded":"2024-04-01T01:31:22.726607+0000","last_fullsized":"2024-04-01T01:31:22.726607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:19:16.941504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726585+0000","last_change":"2024-04-01T01:31:16.385373+0000","last_active":"2024-04-01T01:31:22.726585+0000","last_peered":"2024-04-01T01:31:22.726585+0000","last_clean":"2024-04-01T01:31:22.726585+0000","last_became_active":"2024-04-01T01:31:16.385046+0000","last_became_peered":"2024-04-01T01:31:16.385046+0000","last_unstale":"2024-04-01T01:31:22.726585+0000","last_undegraded":"2024-04-01T01:31:22.726585+0000","last_fullsized":"2024-04-01T01:31:22.726585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:40:32.521220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726525+0000","last_change":"2024-04-01T01:31:16.386267+0000","last_active":"2024-04-01T01:31:22.726525+0000","last_peered":"2024-04-01T01:31:22.726525+0000","last_clean":"2024-04-01T01:31:22.726525+0000","last_became_active":"2024-04-01T01:31:16.385917+0000","last_became_peered":"2024-04-01T01:31:16.385917+0000","last_unstale":"2024-04-01T01:31:22.726525+0000","last_undegraded":"2024-04-01T01:31:22.726525+0000","last_fullsized":"2024-04-01T01:31:22.726525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:58.249253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727779+0000","last_change":"2024-04-01T01:31:16.385609+0000","last_active":"2024-04-01T01:31:22.727779+0000","last_peered":"2024-04-01T01:31:22.727779+0000","last_clean":"2024-04-01T01:31:22.727779+0000","last_became_active":"2024-04-01T01:31:16.385128+0000","last_became_peered":"2024-04-01T01:31:16.385128+0000","last_unstale":"2024-04-01T01:31:22.727779+0000","last_undegraded":"2024-04-01T01:31:22.727779+0000","last_fullsized":"2024-04-01T01:31:22.727779+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:30:45.915587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.561736+0000","last_change":"2024-04-01T01:31:16.381209+0000","last_active":"2024-04-01T01:31:21.561736+0000","last_peered":"2024-04-01T01:31:21.561736+0000","last_clean":"2024-04-01T01:31:21.561736+0000","last_became_active":"2024-04-01T01:31:16.380721+0000","last_became_peered":"2024-04-01T01:31:16.380721+0000","last_unstale":"2024-04-01T01:31:21.561736+0000","last_undegraded":"2024-04-01T01:31:21.561736+0000","last_fullsized":"2024-04-01T01:31:21.561736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:22.438224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564687+0000","last_change":"2024-04-01T01:31:16.385235+0000","last_active":"2024-04-01T01:31:21.564687+0000","last_peered":"2024-04-01T01:31:21.564687+0000","last_clean":"2024-04-01T01:31:21.564687+0000","last_became_active":"2024-04-01T01:31:16.384856+0000","last_became_peered":"2024-04-01T01:31:16.384856+0000","last_unstale":"2024-04-01T01:31:21.564687+0000","last_undegraded":"2024-04-01T01:31:21.564687+0000","last_fullsized":"2024-04-01T01:31:21.564687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:35:30.214977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564290+0000","last_change":"2024-04-01T01:31:20.573626+0000","last_active":"2024-04-01T01:31:21.564290+0000","last_peered":"2024-04-01T01:31:21.564290+0000","last_clean":"2024-04-01T01:31:21.564290+0000","last_became_active":"2024-04-01T01:31:20.573182+0000","last_became_peered":"2024-04-01T01:31:20.573182+0000","last_unstale":"2024-04-01T01:31:21.564290+0000","last_undegraded":"2024-04-01T01:31:21.564290+0000","last_fullsized":"2024-04-01T01:31:21.564290+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:33:19.162613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.562044+0000","last_change":"2024-04-01T01:31:15.373100+0000","last_active":"2024-04-01T01:31:21.562044+0000","last_peered":"2024-04-01T01:31:21.562044+0000","last_clean":"2024-04-01T01:31:21.562044+0000","last_became_active":"2024-04-01T01:31:15.372710+0000","last_became_peered":"2024-04-01T01:31:15.372710+0000","last_unstale":"2024-04-01T01:31:21.562044+0000","last_undegraded":"2024-04-01T01:31:21.562044+0000","last_fullsized":"2024-04-01T01:31:21.562044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:29:42.303549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489391+0000","last_change":"2024-04-01T01:31:16.383690+0000","last_active":"2024-04-01T01:31:22.489391+0000","last_peered":"2024-04-01T01:31:22.489391+0000","last_clean":"2024-04-01T01:31:22.489391+0000","last_became_active":"2024-04-01T01:31:16.383248+0000","last_became_peered":"2024-04-01T01:31:16.383248+0000","last_unstale":"2024-04-01T01:31:22.489391+0000","last_undegraded":"2024-04-01T01:31:22.489391+0000","last_fullsized":"2024-04-01T01:31:22.489391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:34:49.112880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726880+0000","last_change":"2024-04-01T01:31:20.569469+0000","last_active":"2024-04-01T01:31:22.726880+0000","last_peered":"2024-04-01T01:31:22.726880+0000","last_clean":"2024-04-01T01:31:22.726880+0000","last_became_active":"2024-04-01T01:31:20.569060+0000","last_became_peered":"2024-04-01T01:31:20.569060+0000","last_unstale":"2024-04-01T01:31:22.726880+0000","last_undegraded":"2024-04-01T01:31:22.726880+0000","last_fullsized":"2024-04-01T01:31:22.726880+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:04:05.960140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489633+0000","last_change":"2024-04-01T01:31:15.381742+0000","last_active":"2024-04-01T01:31:22.489633+0000","last_peered":"2024-04-01T01:31:22.489633+0000","last_clean":"2024-04-01T01:31:22.489633+0000","last_became_active":"2024-04-01T01:31:15.377543+0000","last_became_peered":"2024-04-01T01:31:15.377543+0000","last_unstale":"2024-04-01T01:31:22.489633+0000","last_undegraded":"2024-04-01T01:31:22.489633+0000","last_fullsized":"2024-04-01T01:31:22.489633+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:51:42.911114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726871+0000","last_change":"2024-04-01T01:31:16.382001+0000","last_active":"2024-04-01T01:31:22.726871+0000","last_peered":"2024-04-01T01:31:22.726871+0000","last_clean":"2024-04-01T01:31:22.726871+0000","last_became_active":"2024-04-01T01:31:16.381694+0000","last_became_peered":"2024-04-01T01:31:16.381694+0000","last_unstale":"2024-04-01T01:31:22.726871+0000","last_undegraded":"2024-04-01T01:31:22.726871+0000","last_fullsized":"2024-04-01T01:31:22.726871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:55:07.366755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726847+0000","last_change":"2024-04-01T01:31:20.570080+0000","last_active":"2024-04-01T01:31:22.726847+0000","last_peered":"2024-04-01T01:31:22.726847+0000","last_clean":"2024-04-01T01:31:22.726847+0000","last_became_active":"2024-04-01T01:31:20.569712+0000","last_became_peered":"2024-04-01T01:31:20.569712+0000","last_unstale":"2024-04-01T01:31:22.726847+0000","last_undegraded":"2024-04-01T01:31:22.726847+0000","last_fullsized":"2024-04-01T01:31:22.726847+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:48:27.732831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'11","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:37.571665+0000","last_change":"2024-04-01T01:31:15.378381+0000","last_active":"2024-04-01T01:31:37.571665+0000","last_peered":"2024-04-01T01:31:37.571665+0000","last_clean":"2024-04-01T01:31:37.571665+0000","last_became_active":"2024-04-01T01:31:15.378081+0000","last_became_peered":"2024-04-01T01:31:15.378081+0000","last_unstale":"2024-04-01T01:31:37.571665+0000","last_undegraded":"2024-04-01T01:31:37.571665+0000","last_fullsized":"2024-04-01T01:31:37.571665+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:56:27.896123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727424+0000","last_change":"2024-04-01T01:31:16.383020+0000","last_active":"2024-04-01T01:31:22.727424+0000","last_peered":"2024-04-01T01:31:22.727424+0000","last_clean":"2024-04-01T01:31:22.727424+0000","last_became_active":"2024-04-01T01:31:16.382640+0000","last_became_peered":"2024-04-01T01:31:16.382640+0000","last_unstale":"2024-04-01T01:31:22.727424+0000","last_undegraded":"2024-04-01T01:31:22.727424+0000","last_fullsized":"2024-04-01T01:31:22.727424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:11:51.211929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726826+0000","last_change":"2024-04-01T01:31:20.573030+0000","last_active":"2024-04-01T01:31:22.726826+0000","last_peered":"2024-04-01T01:31:22.726826+0000","last_clean":"2024-04-01T01:31:22.726826+0000","last_became_active":"2024-04-01T01:31:20.572613+0000","last_became_peered":"2024-04-01T01:31:20.572613+0000","last_unstale":"2024-04-01T01:31:22.726826+0000","last_undegraded":"2024-04-01T01:31:22.726826+0000","last_fullsized":"2024-04-01T01:31:22.726826+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:42:17.086942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:37.570150+0000","last_change":"2024-04-01T01:31:15.373054+0000","last_active":"2024-04-01T01:31:37.570150+0000","last_peered":"2024-04-01T01:31:37.570150+0000","last_clean":"2024-04-01T01:31:37.570150+0000","last_became_active":"2024-04-01T01:31:15.372575+0000","last_became_peered":"2024-04-01T01:31:15.372575+0000","last_unstale":"2024-04-01T01:31:37.570150+0000","last_undegraded":"2024-04-01T01:31:37.570150+0000","last_fullsized":"2024-04-01T01:31:37.570150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:56:41.860423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.728277+0000","last_change":"2024-04-01T01:31:16.376037+0000","last_active":"2024-04-01T01:31:22.728277+0000","last_peered":"2024-04-01T01:31:22.728277+0000","last_clean":"2024-04-01T01:31:22.728277+0000","last_became_active":"2024-04-01T01:31:16.375463+0000","last_became_peered":"2024-04-01T01:31:16.375463+0000","last_unstale":"2024-04-01T01:31:22.728277+0000","last_undegraded":"2024-04-01T01:31:22.728277+0000","last_fullsized":"2024-04-01T01:31:22.728277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:30:40.098033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564643+0000","last_change":"2024-04-01T01:31:20.574288+0000","last_active":"2024-04-01T01:31:21.564643+0000","last_peered":"2024-04-01T01:31:21.564643+0000","last_clean":"2024-04-01T01:31:21.564643+0000","last_became_active":"2024-04-01T01:31:20.573812+0000","last_became_peered":"2024-04-01T01:31:20.573812+0000","last_unstale":"2024-04-01T01:31:21.564643+0000","last_undegraded":"2024-04-01T01:31:21.564643+0000","last_fullsized":"2024-04-01T01:31:21.564643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:43:46.126102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:35.543959+0000","last_change":"2024-04-01T01:31:15.380835+0000","last_active":"2024-04-01T01:31:35.543959+0000","last_peered":"2024-04-01T01:31:35.543959+0000","last_clean":"2024-04-01T01:31:35.543959+0000","last_became_active":"2024-04-01T01:31:15.380518+0000","last_became_peered":"2024-04-01T01:31:15.380518+0000","last_unstale":"2024-04-01T01:31:35.543959+0000","last_undegraded":"2024-04-01T01:31:35.543959+0000","last_fullsized":"2024-04-01T01:31:35.543959+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:51.452917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.151196+0000","last_change":"2024-04-01T01:31:16.377735+0000","last_active":"2024-04-01T01:31:22.151196+0000","last_peered":"2024-04-01T01:31:22.151196+0000","last_clean":"2024-04-01T01:31:22.151196+0000","last_became_active":"2024-04-01T01:31:16.377342+0000","last_became_peered":"2024-04-01T01:31:16.377342+0000","last_unstale":"2024-04-01T01:31:22.151196+0000","last_undegraded":"2024-04-01T01:31:22.151196+0000","last_fullsized":"2024-04-01T01:31:22.151196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:47.539223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564543+0000","last_change":"2024-04-01T01:31:20.573370+0000","last_active":"2024-04-01T01:31:21.564543+0000","last_peered":"2024-04-01T01:31:21.564543+0000","last_clean":"2024-04-01T01:31:21.564543+0000","last_became_active":"2024-04-01T01:31:20.572919+0000","last_became_peered":"2024-04-01T01:31:20.572919+0000","last_unstale":"2024-04-01T01:31:21.564543+0000","last_undegraded":"2024-04-01T01:31:21.564543+0000","last_fullsized":"2024-04-01T01:31:21.564543+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:42.190918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:35.557203+0000","last_change":"2024-04-01T01:31:15.372310+0000","last_active":"2024-04-01T01:31:35.557203+0000","last_peered":"2024-04-01T01:31:35.557203+0000","last_clean":"2024-04-01T01:31:35.557203+0000","last_became_active":"2024-04-01T01:31:15.371527+0000","last_became_peered":"2024-04-01T01:31:15.371527+0000","last_unstale":"2024-04-01T01:31:35.557203+0000","last_undegraded":"2024-04-01T01:31:35.557203+0000","last_fullsized":"2024-04-01T01:31:35.557203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:54:53.844991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727986+0000","last_change":"2024-04-01T01:31:16.384130+0000","last_active":"2024-04-01T01:31:22.727986+0000","last_peered":"2024-04-01T01:31:22.727986+0000","last_clean":"2024-04-01T01:31:22.727986+0000","last_became_active":"2024-04-01T01:31:16.383400+0000","last_became_peered":"2024-04-01T01:31:16.383400+0000","last_unstale":"2024-04-01T01:31:22.727986+0000","last_undegraded":"2024-04-01T01:31:22.727986+0000","last_fullsized":"2024-04-01T01:31:22.727986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:22:10.860325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727817+0000","last_change":"2024-04-01T01:31:20.571052+0000","last_active":"2024-04-01T01:31:22.727817+0000","last_peered":"2024-04-01T01:31:22.727817+0000","last_clean":"2024-04-01T01:31:22.727817+0000","last_became_active":"2024-04-01T01:31:20.570617+0000","last_became_peered":"2024-04-01T01:31:20.570617+0000","last_unstale":"2024-04-01T01:31:22.727817+0000","last_undegraded":"2024-04-01T01:31:22.727817+0000","last_fullsized":"2024-04-01T01:31:22.727817+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:53:51.195056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989391+0000","last_change":"2024-04-01T01:31:15.369820+0000","last_active":"2024-04-01T01:31:22.989391+0000","last_peered":"2024-04-01T01:31:22.989391+0000","last_clean":"2024-04-01T01:31:22.989391+0000","last_became_active":"2024-04-01T01:31:15.369378+0000","last_became_peered":"2024-04-01T01:31:15.369378+0000","last_unstale":"2024-04-01T01:31:22.989391+0000","last_undegraded":"2024-04-01T01:31:22.989391+0000","last_fullsized":"2024-04-01T01:31:22.989391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:37:34.351220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489268+0000","last_change":"2024-04-01T01:31:16.383352+0000","last_active":"2024-04-01T01:31:22.489268+0000","last_peered":"2024-04-01T01:31:22.489268+0000","last_clean":"2024-04-01T01:31:22.489268+0000","last_became_active":"2024-04-01T01:31:16.382898+0000","last_became_peered":"2024-04-01T01:31:16.382898+0000","last_unstale":"2024-04-01T01:31:22.489268+0000","last_undegraded":"2024-04-01T01:31:22.489268+0000","last_fullsized":"2024-04-01T01:31:22.489268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:27.854459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726904+0000","last_change":"2024-04-01T01:31:20.576578+0000","last_active":"2024-04-01T01:31:22.726904+0000","last_peered":"2024-04-01T01:31:22.726904+0000","last_clean":"2024-04-01T01:31:22.726904+0000","last_became_active":"2024-04-01T01:31:20.575996+0000","last_became_peered":"2024-04-01T01:31:20.575996+0000","last_unstale":"2024-04-01T01:31:22.726904+0000","last_undegraded":"2024-04-01T01:31:22.726904+0000","last_fullsized":"2024-04-01T01:31:22.726904+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:52:08.808357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727658+0000","last_change":"2024-04-01T01:31:15.377246+0000","last_active":"2024-04-01T01:31:22.727658+0000","last_peered":"2024-04-01T01:31:22.727658+0000","last_clean":"2024-04-01T01:31:22.727658+0000","last_became_active":"2024-04-01T01:31:15.376957+0000","last_became_peered":"2024-04-01T01:31:15.376957+0000","last_unstale":"2024-04-01T01:31:22.727658+0000","last_undegraded":"2024-04-01T01:31:22.727658+0000","last_fullsized":"2024-04-01T01:31:22.727658+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:59:34.313984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.561233+0000","last_change":"2024-04-01T01:31:16.379031+0000","last_active":"2024-04-01T01:31:21.561233+0000","last_peered":"2024-04-01T01:31:21.561233+0000","last_clean":"2024-04-01T01:31:21.561233+0000","last_became_active":"2024-04-01T01:31:16.378593+0000","last_became_peered":"2024-04-01T01:31:16.378593+0000","last_unstale":"2024-04-01T01:31:21.561233+0000","last_undegraded":"2024-04-01T01:31:21.561233+0000","last_fullsized":"2024-04-01T01:31:21.561233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:31.816103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489148+0000","last_change":"2024-04-01T01:31:20.578660+0000","last_active":"2024-04-01T01:31:22.489148+0000","last_peered":"2024-04-01T01:31:22.489148+0000","last_clean":"2024-04-01T01:31:22.489148+0000","last_became_active":"2024-04-01T01:31:20.578082+0000","last_became_peered":"2024-04-01T01:31:20.578082+0000","last_unstale":"2024-04-01T01:31:22.489148+0000","last_undegraded":"2024-04-01T01:31:22.489148+0000","last_fullsized":"2024-04-01T01:31:22.489148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:02:56.570816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:34.620860+0000","last_change":"2024-04-01T01:31:15.373428+0000","last_active":"2024-04-01T01:31:34.620860+0000","last_peered":"2024-04-01T01:31:34.620860+0000","last_clean":"2024-04-01T01:31:34.620860+0000","last_became_active":"2024-04-01T01:31:15.372965+0000","last_became_peered":"2024-04-01T01:31:15.372965+0000","last_unstale":"2024-04-01T01:31:34.620860+0000","last_undegraded":"2024-04-01T01:31:34.620860+0000","last_fullsized":"2024-04-01T01:31:34.620860+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:55:42.211055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727715+0000","last_change":"2024-04-01T01:31:16.382961+0000","last_active":"2024-04-01T01:31:22.727715+0000","last_peered":"2024-04-01T01:31:22.727715+0000","last_clean":"2024-04-01T01:31:22.727715+0000","last_became_active":"2024-04-01T01:31:16.382527+0000","last_became_peered":"2024-04-01T01:31:16.382527+0000","last_unstale":"2024-04-01T01:31:22.727715+0000","last_undegraded":"2024-04-01T01:31:22.727715+0000","last_fullsized":"2024-04-01T01:31:22.727715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:46:20.250064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.561270+0000","last_change":"2024-04-01T01:31:20.571842+0000","last_active":"2024-04-01T01:31:21.561270+0000","last_peered":"2024-04-01T01:31:21.561270+0000","last_clean":"2024-04-01T01:31:21.561270+0000","last_became_active":"2024-04-01T01:31:20.571394+0000","last_became_peered":"2024-04-01T01:31:20.571394+0000","last_unstale":"2024-04-01T01:31:21.561270+0000","last_undegraded":"2024-04-01T01:31:21.561270+0000","last_fullsized":"2024-04-01T01:31:21.561270+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:51:36.512277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:34.621119+0000","last_change":"2024-04-01T01:31:15.379737+0000","last_active":"2024-04-01T01:31:34.621119+0000","last_peered":"2024-04-01T01:31:34.621119+0000","last_clean":"2024-04-01T01:31:34.621119+0000","last_became_active":"2024-04-01T01:31:15.379158+0000","last_became_peered":"2024-04-01T01:31:15.379158+0000","last_unstale":"2024-04-01T01:31:34.621119+0000","last_undegraded":"2024-04-01T01:31:34.621119+0000","last_fullsized":"2024-04-01T01:31:34.621119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:12:44.474772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726929+0000","last_change":"2024-04-01T01:31:16.386059+0000","last_active":"2024-04-01T01:31:22.726929+0000","last_peered":"2024-04-01T01:31:22.726929+0000","last_clean":"2024-04-01T01:31:22.726929+0000","last_became_active":"2024-04-01T01:31:16.384551+0000","last_became_peered":"2024-04-01T01:31:16.384551+0000","last_unstale":"2024-04-01T01:31:22.726929+0000","last_undegraded":"2024-04-01T01:31:22.726929+0000","last_fullsized":"2024-04-01T01:31:22.726929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:52:16.594485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726990+0000","last_change":"2024-04-01T01:31:20.566937+0000","last_active":"2024-04-01T01:31:22.726990+0000","last_peered":"2024-04-01T01:31:22.726990+0000","last_clean":"2024-04-01T01:31:22.726990+0000","last_became_active":"2024-04-01T01:31:20.566447+0000","last_became_peered":"2024-04-01T01:31:20.566447+0000","last_unstale":"2024-04-01T01:31:22.726990+0000","last_undegraded":"2024-04-01T01:31:22.726990+0000","last_fullsized":"2024-04-01T01:31:22.726990+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:17:37.845166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:34.619695+0000","last_change":"2024-04-01T01:31:15.375669+0000","last_active":"2024-04-01T01:31:34.619695+0000","last_peered":"2024-04-01T01:31:34.619695+0000","last_clean":"2024-04-01T01:31:34.619695+0000","last_became_active":"2024-04-01T01:31:15.375353+0000","last_became_peered":"2024-04-01T01:31:15.375353+0000","last_unstale":"2024-04-01T01:31:34.619695+0000","last_undegraded":"2024-04-01T01:31:34.619695+0000","last_fullsized":"2024-04-01T01:31:34.619695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:47:43.096898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727521+0000","last_change":"2024-04-01T01:31:16.388399+0000","last_active":"2024-04-01T01:31:22.727521+0000","last_peered":"2024-04-01T01:31:22.727521+0000","last_clean":"2024-04-01T01:31:22.727521+0000","last_became_active":"2024-04-01T01:31:16.388059+0000","last_became_peered":"2024-04-01T01:31:16.388059+0000","last_unstale":"2024-04-01T01:31:22.727521+0000","last_undegraded":"2024-04-01T01:31:22.727521+0000","last_fullsized":"2024-04-01T01:31:22.727521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:59:50.907802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.561606+0000","last_change":"2024-04-01T01:31:20.569524+0000","last_active":"2024-04-01T01:31:21.561606+0000","last_peered":"2024-04-01T01:31:21.561606+0000","last_clean":"2024-04-01T01:31:21.561606+0000","last_became_active":"2024-04-01T01:31:20.569112+0000","last_became_peered":"2024-04-01T01:31:20.569112+0000","last_unstale":"2024-04-01T01:31:21.561606+0000","last_undegraded":"2024-04-01T01:31:21.561606+0000","last_fullsized":"2024-04-01T01:31:21.561606+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:06:53.651152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:35.548827+0000","last_change":"2024-04-01T01:31:15.376421+0000","last_active":"2024-04-01T01:31:35.548827+0000","last_peered":"2024-04-01T01:31:35.548827+0000","last_clean":"2024-04-01T01:31:35.548827+0000","last_became_active":"2024-04-01T01:31:15.375985+0000","last_became_peered":"2024-04-01T01:31:15.375985+0000","last_unstale":"2024-04-01T01:31:35.548827+0000","last_undegraded":"2024-04-01T01:31:35.548827+0000","last_fullsized":"2024-04-01T01:31:35.548827+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:37:29.121436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.151447+0000","last_change":"2024-04-01T01:31:16.380641+0000","last_active":"2024-04-01T01:31:22.151447+0000","last_peered":"2024-04-01T01:31:22.151447+0000","last_clean":"2024-04-01T01:31:22.151447+0000","last_became_active":"2024-04-01T01:31:16.380300+0000","last_became_peered":"2024-04-01T01:31:16.380300+0000","last_unstale":"2024-04-01T01:31:22.151447+0000","last_undegraded":"2024-04-01T01:31:22.151447+0000","last_fullsized":"2024-04-01T01:31:22.151447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:26:08.240694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.728572+0000","last_change":"2024-04-01T01:31:20.576094+0000","last_active":"2024-04-01T01:31:22.728572+0000","last_peered":"2024-04-01T01:31:22.728572+0000","last_clean":"2024-04-01T01:31:22.728572+0000","last_became_active":"2024-04-01T01:31:20.575674+0000","last_became_peered":"2024-04-01T01:31:20.575674+0000","last_unstale":"2024-04-01T01:31:22.728572+0000","last_undegraded":"2024-04-01T01:31:22.728572+0000","last_fullsized":"2024-04-01T01:31:22.728572+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:42:19.352398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:37.571173+0000","last_change":"2024-04-01T01:31:15.375136+0000","last_active":"2024-04-01T01:31:37.571173+0000","last_peered":"2024-04-01T01:31:37.571173+0000","last_clean":"2024-04-01T01:31:37.571173+0000","last_became_active":"2024-04-01T01:31:15.374813+0000","last_became_peered":"2024-04-01T01:31:15.374813+0000","last_unstale":"2024-04-01T01:31:37.571173+0000","last_undegraded":"2024-04-01T01:31:37.571173+0000","last_fullsized":"2024-04-01T01:31:37.571173+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:19:05.490103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989593+0000","last_change":"2024-04-01T01:31:16.380207+0000","last_active":"2024-04-01T01:31:22.989593+0000","last_peered":"2024-04-01T01:31:22.989593+0000","last_clean":"2024-04-01T01:31:22.989593+0000","last_became_active":"2024-04-01T01:31:16.379891+0000","last_became_peered":"2024-04-01T01:31:16.379891+0000","last_unstale":"2024-04-01T01:31:22.989593+0000","last_undegraded":"2024-04-01T01:31:22.989593+0000","last_fullsized":"2024-04-01T01:31:22.989593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:12:36.417162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727396+0000","last_change":"2024-04-01T01:31:20.569978+0000","last_active":"2024-04-01T01:31:22.727396+0000","last_peered":"2024-04-01T01:31:22.727396+0000","last_clean":"2024-04-01T01:31:22.727396+0000","last_became_active":"2024-04-01T01:31:20.568298+0000","last_became_peered":"2024-04-01T01:31:20.568298+0000","last_unstale":"2024-04-01T01:31:22.727396+0000","last_undegraded":"2024-04-01T01:31:22.727396+0000","last_fullsized":"2024-04-01T01:31:22.727396+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:42:16.439284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:34.621406+0000","last_change":"2024-04-01T01:31:15.380143+0000","last_active":"2024-04-01T01:31:34.621406+0000","last_peered":"2024-04-01T01:31:34.621406+0000","last_clean":"2024-04-01T01:31:34.621406+0000","last_became_active":"2024-04-01T01:31:15.379835+0000","last_became_peered":"2024-04-01T01:31:15.379835+0000","last_unstale":"2024-04-01T01:31:34.621406+0000","last_undegraded":"2024-04-01T01:31:34.621406+0000","last_fullsized":"2024-04-01T01:31:34.621406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:28:34.213626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563787+0000","last_change":"2024-04-01T01:31:16.380290+0000","last_active":"2024-04-01T01:31:21.563787+0000","last_peered":"2024-04-01T01:31:21.563787+0000","last_clean":"2024-04-01T01:31:21.563787+0000","last_became_active":"2024-04-01T01:31:16.379983+0000","last_became_peered":"2024-04-01T01:31:16.379983+0000","last_unstale":"2024-04-01T01:31:21.563787+0000","last_undegraded":"2024-04-01T01:31:21.563787+0000","last_fullsized":"2024-04-01T01:31:21.563787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:50.135454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.561681+0000","last_change":"2024-04-01T01:31:20.573500+0000","last_active":"2024-04-01T01:31:21.561681+0000","last_peered":"2024-04-01T01:31:21.561681+0000","last_clean":"2024-04-01T01:31:21.561681+0000","last_became_active":"2024-04-01T01:31:20.573101+0000","last_became_peered":"2024-04-01T01:31:20.573101+0000","last_unstale":"2024-04-01T01:31:21.561681+0000","last_undegraded":"2024-04-01T01:31:21.561681+0000","last_fullsized":"2024-04-01T01:31:21.561681+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:11:27.881703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:36.557382+0000","last_change":"2024-04-01T01:31:15.377265+0000","last_active":"2024-04-01T01:31:36.557382+0000","last_peered":"2024-04-01T01:31:36.557382+0000","last_clean":"2024-04-01T01:31:36.557382+0000","last_became_active":"2024-04-01T01:31:15.376889+0000","last_became_peered":"2024-04-01T01:31:15.376889+0000","last_unstale":"2024-04-01T01:31:36.557382+0000","last_undegraded":"2024-04-01T01:31:36.557382+0000","last_fullsized":"2024-04-01T01:31:36.557382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:43:31.582866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563600+0000","last_change":"2024-04-01T01:31:16.384982+0000","last_active":"2024-04-01T01:31:21.563600+0000","last_peered":"2024-04-01T01:31:21.563600+0000","last_clean":"2024-04-01T01:31:21.563600+0000","last_became_active":"2024-04-01T01:31:16.384663+0000","last_became_peered":"2024-04-01T01:31:16.384663+0000","last_unstale":"2024-04-01T01:31:21.563600+0000","last_undegraded":"2024-04-01T01:31:21.563600+0000","last_fullsized":"2024-04-01T01:31:21.563600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:14:57.351768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989927+0000","last_change":"2024-04-01T01:31:20.578761+0000","last_active":"2024-04-01T01:31:22.989927+0000","last_peered":"2024-04-01T01:31:22.989927+0000","last_clean":"2024-04-01T01:31:22.989927+0000","last_became_active":"2024-04-01T01:31:20.578204+0000","last_became_peered":"2024-04-01T01:31:20.578204+0000","last_unstale":"2024-04-01T01:31:22.989927+0000","last_undegraded":"2024-04-01T01:31:22.989927+0000","last_fullsized":"2024-04-01T01:31:22.989927+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:19:33.584872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:36.558677+0000","last_change":"2024-04-01T01:31:15.380773+0000","last_active":"2024-04-01T01:31:36.558677+0000","last_peered":"2024-04-01T01:31:36.558677+0000","last_clean":"2024-04-01T01:31:36.558677+0000","last_became_active":"2024-04-01T01:31:15.380472+0000","last_became_peered":"2024-04-01T01:31:15.380472+0000","last_unstale":"2024-04-01T01:31:36.558677+0000","last_undegraded":"2024-04-01T01:31:36.558677+0000","last_fullsized":"2024-04-01T01:31:36.558677+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:34:28.294500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727565+0000","last_change":"2024-04-01T01:31:16.374658+0000","last_active":"2024-04-01T01:31:22.727565+0000","last_peered":"2024-04-01T01:31:22.727565+0000","last_clean":"2024-04-01T01:31:22.727565+0000","last_became_active":"2024-04-01T01:31:16.373596+0000","last_became_peered":"2024-04-01T01:31:16.373596+0000","last_unstale":"2024-04-01T01:31:22.727565+0000","last_undegraded":"2024-04-01T01:31:22.727565+0000","last_fullsized":"2024-04-01T01:31:22.727565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:28:38.477719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727440+0000","last_change":"2024-04-01T01:31:20.577449+0000","last_active":"2024-04-01T01:31:22.727440+0000","last_peered":"2024-04-01T01:31:22.727440+0000","last_clean":"2024-04-01T01:31:22.727440+0000","last_became_active":"2024-04-01T01:31:20.577026+0000","last_became_peered":"2024-04-01T01:31:20.577026+0000","last_unstale":"2024-04-01T01:31:22.727440+0000","last_undegraded":"2024-04-01T01:31:22.727440+0000","last_fullsized":"2024-04-01T01:31:22.727440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:44:48.080304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:35.545691+0000","last_change":"2024-04-01T01:31:15.373885+0000","last_active":"2024-04-01T01:31:35.545691+0000","last_peered":"2024-04-01T01:31:35.545691+0000","last_clean":"2024-04-01T01:31:35.545691+0000","last_became_active":"2024-04-01T01:31:15.373578+0000","last_became_peered":"2024-04-01T01:31:15.373578+0000","last_unstale":"2024-04-01T01:31:35.545691+0000","last_undegraded":"2024-04-01T01:31:35.545691+0000","last_fullsized":"2024-04-01T01:31:35.545691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:52.641515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727785+0000","last_change":"2024-04-01T01:31:16.379997+0000","last_active":"2024-04-01T01:31:22.727785+0000","last_peered":"2024-04-01T01:31:22.727785+0000","last_clean":"2024-04-01T01:31:22.727785+0000","last_became_active":"2024-04-01T01:31:16.379664+0000","last_became_peered":"2024-04-01T01:31:16.379664+0000","last_unstale":"2024-04-01T01:31:22.727785+0000","last_undegraded":"2024-04-01T01:31:22.727785+0000","last_fullsized":"2024-04-01T01:31:22.727785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:37:17.432095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.151392+0000","last_change":"2024-04-01T01:31:20.569835+0000","last_active":"2024-04-01T01:31:22.151392+0000","last_peered":"2024-04-01T01:31:22.151392+0000","last_clean":"2024-04-01T01:31:22.151392+0000","last_became_active":"2024-04-01T01:31:20.569195+0000","last_became_peered":"2024-04-01T01:31:20.569195+0000","last_unstale":"2024-04-01T01:31:22.151392+0000","last_undegraded":"2024-04-01T01:31:22.151392+0000","last_fullsized":"2024-04-01T01:31:22.151392+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:10:18.278575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.561751+0000","last_change":"2024-04-01T01:31:15.377679+0000","last_active":"2024-04-01T01:31:21.561751+0000","last_peered":"2024-04-01T01:31:21.561751+0000","last_clean":"2024-04-01T01:31:21.561751+0000","last_became_active":"2024-04-01T01:31:15.377351+0000","last_became_peered":"2024-04-01T01:31:15.377351+0000","last_unstale":"2024-04-01T01:31:21.561751+0000","last_undegraded":"2024-04-01T01:31:21.561751+0000","last_fullsized":"2024-04-01T01:31:21.561751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:22:41.216563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.962094+0000","last_change":"2024-04-01T01:31:16.380997+0000","last_active":"2024-04-01T01:31:21.962094+0000","last_peered":"2024-04-01T01:31:21.962094+0000","last_clean":"2024-04-01T01:31:21.962094+0000","last_became_active":"2024-04-01T01:31:16.380692+0000","last_became_peered":"2024-04-01T01:31:16.380692+0000","last_unstale":"2024-04-01T01:31:21.962094+0000","last_undegraded":"2024-04-01T01:31:21.962094+0000","last_fullsized":"2024-04-01T01:31:21.962094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:09:51.220849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989517+0000","last_change":"2024-04-01T01:31:20.573191+0000","last_active":"2024-04-01T01:31:22.989517+0000","last_peered":"2024-04-01T01:31:22.989517+0000","last_clean":"2024-04-01T01:31:22.989517+0000","last_became_active":"2024-04-01T01:31:20.572669+0000","last_became_peered":"2024-04-01T01:31:20.572669+0000","last_unstale":"2024-04-01T01:31:22.989517+0000","last_undegraded":"2024-04-01T01:31:22.989517+0000","last_fullsized":"2024-04-01T01:31:22.989517+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:01:56.491271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563536+0000","last_change":"2024-04-01T01:31:15.379566+0000","last_active":"2024-04-01T01:31:21.563536+0000","last_peered":"2024-04-01T01:31:21.563536+0000","last_clean":"2024-04-01T01:31:21.563536+0000","last_became_active":"2024-04-01T01:31:15.379213+0000","last_became_peered":"2024-04-01T01:31:15.379213+0000","last_unstale":"2024-04-01T01:31:21.563536+0000","last_undegraded":"2024-04-01T01:31:21.563536+0000","last_fullsized":"2024-04-01T01:31:21.563536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:34:17.974422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.961932+0000","last_change":"2024-04-01T01:31:16.376218+0000","last_active":"2024-04-01T01:31:21.961932+0000","last_peered":"2024-04-01T01:31:21.961932+0000","last_clean":"2024-04-01T01:31:21.961932+0000","last_became_active":"2024-04-01T01:31:16.375887+0000","last_became_peered":"2024-04-01T01:31:16.375887+0000","last_unstale":"2024-04-01T01:31:21.961932+0000","last_undegraded":"2024-04-01T01:31:21.961932+0000","last_fullsized":"2024-04-01T01:31:21.961932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:27:03.723827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989853+0000","last_change":"2024-04-01T01:31:20.576461+0000","last_active":"2024-04-01T01:31:22.989853+0000","last_peered":"2024-04-01T01:31:22.989853+0000","last_clean":"2024-04-01T01:31:22.989853+0000","last_became_active":"2024-04-01T01:31:20.575985+0000","last_became_peered":"2024-04-01T01:31:20.575985+0000","last_unstale":"2024-04-01T01:31:22.989853+0000","last_undegraded":"2024-04-01T01:31:22.989853+0000","last_fullsized":"2024-04-01T01:31:22.989853+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:19:02.714455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727658+0000","last_change":"2024-04-01T01:31:16.376456+0000","last_active":"2024-04-01T01:31:22.727658+0000","last_peered":"2024-04-01T01:31:22.727658+0000","last_clean":"2024-04-01T01:31:22.727658+0000","last_became_active":"2024-04-01T01:31:16.375445+0000","last_became_peered":"2024-04-01T01:31:16.375445+0000","last_unstale":"2024-04-01T01:31:22.727658+0000","last_undegraded":"2024-04-01T01:31:22.727658+0000","last_fullsized":"2024-04-01T01:31:22.727658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:07:11.802799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989333+0000","last_change":"2024-04-01T01:31:15.382702+0000","last_active":"2024-04-01T01:31:22.989333+0000","last_peered":"2024-04-01T01:31:22.989333+0000","last_clean":"2024-04-01T01:31:22.989333+0000","last_became_active":"2024-04-01T01:31:15.382377+0000","last_became_peered":"2024-04-01T01:31:15.382377+0000","last_unstale":"2024-04-01T01:31:22.989333+0000","last_undegraded":"2024-04-01T01:31:22.989333+0000","last_fullsized":"2024-04-01T01:31:22.989333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:45.080992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563874+0000","last_change":"2024-04-01T01:31:20.573080+0000","last_active":"2024-04-01T01:31:21.563874+0000","last_peered":"2024-04-01T01:31:21.563874+0000","last_clean":"2024-04-01T01:31:21.563874+0000","last_became_active":"2024-04-01T01:31:20.572656+0000","last_became_peered":"2024-04-01T01:31:20.572656+0000","last_unstale":"2024-04-01T01:31:21.563874+0000","last_undegraded":"2024-04-01T01:31:21.563874+0000","last_fullsized":"2024-04-01T01:31:21.563874+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:09:54.154913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489197+0000","last_change":"2024-04-01T01:31:16.387981+0000","last_active":"2024-04-01T01:31:22.489197+0000","last_peered":"2024-04-01T01:31:22.489197+0000","last_clean":"2024-04-01T01:31:22.489197+0000","last_became_active":"2024-04-01T01:31:16.387594+0000","last_became_peered":"2024-04-01T01:31:16.387594+0000","last_unstale":"2024-04-01T01:31:22.489197+0000","last_undegraded":"2024-04-01T01:31:22.489197+0000","last_fullsized":"2024-04-01T01:31:22.489197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:17.148752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:36.557954+0000","last_change":"2024-04-01T01:31:15.375212+0000","last_active":"2024-04-01T01:31:36.557954+0000","last_peered":"2024-04-01T01:31:36.557954+0000","last_clean":"2024-04-01T01:31:36.557954+0000","last_became_active":"2024-04-01T01:31:15.374882+0000","last_became_peered":"2024-04-01T01:31:15.374882+0000","last_unstale":"2024-04-01T01:31:36.557954+0000","last_undegraded":"2024-04-01T01:31:36.557954+0000","last_fullsized":"2024-04-01T01:31:36.557954+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:56:23.412025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563494+0000","last_change":"2024-04-01T01:30:03.538590+0000","last_active":"2024-04-01T01:31:21.563494+0000","last_peered":"2024-04-01T01:31:21.563494+0000","last_clean":"2024-04-01T01:31:21.563494+0000","last_became_active":"2024-04-01T01:30:03.538098+0000","last_became_peered":"2024-04-01T01:30:03.538098+0000","last_unstale":"2024-04-01T01:31:21.563494+0000","last_undegraded":"2024-04-01T01:31:21.563494+0000","last_fullsized":"2024-04-01T01:31:21.563494+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:26:43.445198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:26:43.445198+0000","last_clean_scrub_stamp":"2024-04-01T01:26:43.445198+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:22.207795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.728450+0000","last_change":"2024-04-01T01:31:20.573260+0000","last_active":"2024-04-01T01:31:22.728450+0000","last_peered":"2024-04-01T01:31:22.728450+0000","last_clean":"2024-04-01T01:31:22.728450+0000","last_became_active":"2024-04-01T01:31:20.572846+0000","last_became_peered":"2024-04-01T01:31:20.572846+0000","last_unstale":"2024-04-01T01:31:22.728450+0000","last_undegraded":"2024-04-01T01:31:22.728450+0000","last_fullsized":"2024-04-01T01:31:22.728450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:24.986070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989259+0000","last_change":"2024-04-01T01:31:16.384782+0000","last_active":"2024-04-01T01:31:22.989259+0000","last_peered":"2024-04-01T01:31:22.989259+0000","last_clean":"2024-04-01T01:31:22.989259+0000","last_became_active":"2024-04-01T01:31:16.384420+0000","last_became_peered":"2024-04-01T01:31:16.384420+0000","last_unstale":"2024-04-01T01:31:22.989259+0000","last_undegraded":"2024-04-01T01:31:22.989259+0000","last_fullsized":"2024-04-01T01:31:22.989259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:00:20.902022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563606+0000","last_change":"2024-04-01T01:31:15.381853+0000","last_active":"2024-04-01T01:31:21.563606+0000","last_peered":"2024-04-01T01:31:21.563606+0000","last_clean":"2024-04-01T01:31:21.563606+0000","last_became_active":"2024-04-01T01:31:15.381496+0000","last_became_peered":"2024-04-01T01:31:15.381496+0000","last_unstale":"2024-04-01T01:31:21.563606+0000","last_undegraded":"2024-04-01T01:31:21.563606+0000","last_fullsized":"2024-04-01T01:31:21.563606+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:07:51.440351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727381+0000","last_change":"2024-04-01T01:31:20.568705+0000","last_active":"2024-04-01T01:31:22.727381+0000","last_peered":"2024-04-01T01:31:22.727381+0000","last_clean":"2024-04-01T01:31:22.727381+0000","last_became_active":"2024-04-01T01:31:20.568252+0000","last_became_peered":"2024-04-01T01:31:20.568252+0000","last_unstale":"2024-04-01T01:31:22.727381+0000","last_undegraded":"2024-04-01T01:31:22.727381+0000","last_fullsized":"2024-04-01T01:31:22.727381+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:22:28.509510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:36.560205+0000","last_change":"2024-04-01T01:31:15.385870+0000","last_active":"2024-04-01T01:31:36.560205+0000","last_peered":"2024-04-01T01:31:36.560205+0000","last_clean":"2024-04-01T01:31:36.560205+0000","last_became_active":"2024-04-01T01:31:15.385332+0000","last_became_peered":"2024-04-01T01:31:15.385332+0000","last_unstale":"2024-04-01T01:31:36.560205+0000","last_undegraded":"2024-04-01T01:31:36.560205+0000","last_fullsized":"2024-04-01T01:31:36.560205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:30:56.350707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989685+0000","last_change":"2024-04-01T01:31:16.377002+0000","last_active":"2024-04-01T01:31:22.989685+0000","last_peered":"2024-04-01T01:31:22.989685+0000","last_clean":"2024-04-01T01:31:22.989685+0000","last_became_active":"2024-04-01T01:31:16.376679+0000","last_became_peered":"2024-04-01T01:31:16.376679+0000","last_unstale":"2024-04-01T01:31:22.989685+0000","last_undegraded":"2024-04-01T01:31:22.989685+0000","last_fullsized":"2024-04-01T01:31:22.989685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:50:24.126192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489240+0000","last_change":"2024-04-01T01:31:20.577757+0000","last_active":"2024-04-01T01:31:22.489240+0000","last_peered":"2024-04-01T01:31:22.489240+0000","last_clean":"2024-04-01T01:31:22.489240+0000","last_became_active":"2024-04-01T01:31:20.577378+0000","last_became_peered":"2024-04-01T01:31:20.577378+0000","last_unstale":"2024-04-01T01:31:22.489240+0000","last_undegraded":"2024-04-01T01:31:22.489240+0000","last_fullsized":"2024-04-01T01:31:22.489240+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:48:04.511229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.561861+0000","last_change":"2024-04-01T01:31:15.383201+0000","last_active":"2024-04-01T01:31:21.561861+0000","last_peered":"2024-04-01T01:31:21.561861+0000","last_clean":"2024-04-01T01:31:21.561861+0000","last_became_active":"2024-04-01T01:31:15.382320+0000","last_became_peered":"2024-04-01T01:31:15.382320+0000","last_unstale":"2024-04-01T01:31:21.561861+0000","last_undegraded":"2024-04-01T01:31:21.561861+0000","last_fullsized":"2024-04-01T01:31:21.561861+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:15:47.138024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489376+0000","last_change":"2024-04-01T01:31:16.386936+0000","last_active":"2024-04-01T01:31:22.489376+0000","last_peered":"2024-04-01T01:31:22.489376+0000","last_clean":"2024-04-01T01:31:22.489376+0000","last_became_active":"2024-04-01T01:31:16.384448+0000","last_became_peered":"2024-04-01T01:31:16.384448+0000","last_unstale":"2024-04-01T01:31:22.489376+0000","last_undegraded":"2024-04-01T01:31:22.489376+0000","last_fullsized":"2024-04-01T01:31:22.489376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:16:42.243250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564083+0000","last_change":"2024-04-01T01:31:20.565570+0000","last_active":"2024-04-01T01:31:21.564083+0000","last_peered":"2024-04-01T01:31:21.564083+0000","last_clean":"2024-04-01T01:31:21.564083+0000","last_became_active":"2024-04-01T01:31:20.565141+0000","last_became_peered":"2024-04-01T01:31:20.565141+0000","last_unstale":"2024-04-01T01:31:21.564083+0000","last_undegraded":"2024-04-01T01:31:21.564083+0000","last_fullsized":"2024-04-01T01:31:21.564083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:47:12.844970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:34.620955+0000","last_change":"2024-04-01T01:31:15.381343+0000","last_active":"2024-04-01T01:31:34.620955+0000","last_peered":"2024-04-01T01:31:34.620955+0000","last_clean":"2024-04-01T01:31:34.620955+0000","last_became_active":"2024-04-01T01:31:15.380994+0000","last_became_peered":"2024-04-01T01:31:15.380994+0000","last_unstale":"2024-04-01T01:31:34.620955+0000","last_undegraded":"2024-04-01T01:31:34.620955+0000","last_fullsized":"2024-04-01T01:31:34.620955+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:19:10.710285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726919+0000","last_change":"2024-04-01T01:31:16.383951+0000","last_active":"2024-04-01T01:31:22.726919+0000","last_peered":"2024-04-01T01:31:22.726919+0000","last_clean":"2024-04-01T01:31:22.726919+0000","last_became_active":"2024-04-01T01:31:16.383657+0000","last_became_peered":"2024-04-01T01:31:16.383657+0000","last_unstale":"2024-04-01T01:31:22.726919+0000","last_undegraded":"2024-04-01T01:31:22.726919+0000","last_fullsized":"2024-04-01T01:31:22.726919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:55:45.216479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727059+0000","last_change":"2024-04-01T01:31:20.569416+0000","last_active":"2024-04-01T01:31:22.727059+0000","last_peered":"2024-04-01T01:31:22.727059+0000","last_clean":"2024-04-01T01:31:22.727059+0000","last_became_active":"2024-04-01T01:31:20.569007+0000","last_became_peered":"2024-04-01T01:31:20.569007+0000","last_unstale":"2024-04-01T01:31:22.727059+0000","last_undegraded":"2024-04-01T01:31:22.727059+0000","last_fullsized":"2024-04-01T01:31:22.727059+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:25:36.770919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726561+0000","last_change":"2024-04-01T01:31:15.373444+0000","last_active":"2024-04-01T01:31:22.726561+0000","last_peered":"2024-04-01T01:31:22.726561+0000","last_clean":"2024-04-01T01:31:22.726561+0000","last_became_active":"2024-04-01T01:31:15.373094+0000","last_became_peered":"2024-04-01T01:31:15.373094+0000","last_unstale":"2024-04-01T01:31:22.726561+0000","last_undegraded":"2024-04-01T01:31:22.726561+0000","last_fullsized":"2024-04-01T01:31:22.726561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:26:27.443832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.151031+0000","last_change":"2024-04-01T01:31:16.381441+0000","last_active":"2024-04-01T01:31:22.151031+0000","last_peered":"2024-04-01T01:31:22.151031+0000","last_clean":"2024-04-01T01:31:22.151031+0000","last_became_active":"2024-04-01T01:31:16.381092+0000","last_became_peered":"2024-04-01T01:31:16.381092+0000","last_unstale":"2024-04-01T01:31:22.151031+0000","last_undegraded":"2024-04-01T01:31:22.151031+0000","last_fullsized":"2024-04-01T01:31:22.151031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:47:00.229090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.562042+0000","last_change":"2024-04-01T01:31:20.569672+0000","last_active":"2024-04-01T01:31:21.562042+0000","last_peered":"2024-04-01T01:31:21.562042+0000","last_clean":"2024-04-01T01:31:21.562042+0000","last_became_active":"2024-04-01T01:31:20.569276+0000","last_became_peered":"2024-04-01T01:31:20.569276+0000","last_unstale":"2024-04-01T01:31:21.562042+0000","last_undegraded":"2024-04-01T01:31:21.562042+0000","last_fullsized":"2024-04-01T01:31:21.562042+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:39:03.331428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:36.556961+0000","last_change":"2024-04-01T01:31:15.376094+0000","last_active":"2024-04-01T01:31:36.556961+0000","last_peered":"2024-04-01T01:31:36.556961+0000","last_clean":"2024-04-01T01:31:36.556961+0000","last_became_active":"2024-04-01T01:31:15.375169+0000","last_became_peered":"2024-04-01T01:31:15.375169+0000","last_unstale":"2024-04-01T01:31:36.556961+0000","last_undegraded":"2024-04-01T01:31:36.556961+0000","last_fullsized":"2024-04-01T01:31:36.556961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:18:20.710644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726996+0000","last_change":"2024-04-01T01:31:16.378514+0000","last_active":"2024-04-01T01:31:22.726996+0000","last_peered":"2024-04-01T01:31:22.726996+0000","last_clean":"2024-04-01T01:31:22.726996+0000","last_became_active":"2024-04-01T01:31:16.377990+0000","last_became_peered":"2024-04-01T01:31:16.377990+0000","last_unstale":"2024-04-01T01:31:22.726996+0000","last_undegraded":"2024-04-01T01:31:22.726996+0000","last_fullsized":"2024-04-01T01:31:22.726996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:27:29.829775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726745+0000","last_change":"2024-04-01T01:31:20.567892+0000","last_active":"2024-04-01T01:31:22.726745+0000","last_peered":"2024-04-01T01:31:22.726745+0000","last_clean":"2024-04-01T01:31:22.726745+0000","last_became_active":"2024-04-01T01:31:20.567525+0000","last_became_peered":"2024-04-01T01:31:20.567525+0000","last_unstale":"2024-04-01T01:31:22.726745+0000","last_undegraded":"2024-04-01T01:31:22.726745+0000","last_fullsized":"2024-04-01T01:31:22.726745+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:57.828691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:35.615739+0000","last_change":"2024-04-01T01:31:15.373626+0000","last_active":"2024-04-01T01:31:35.615739+0000","last_peered":"2024-04-01T01:31:35.615739+0000","last_clean":"2024-04-01T01:31:35.615739+0000","last_became_active":"2024-04-01T01:31:15.373302+0000","last_became_peered":"2024-04-01T01:31:15.373302+0000","last_unstale":"2024-04-01T01:31:35.615739+0000","last_undegraded":"2024-04-01T01:31:35.615739+0000","last_fullsized":"2024-04-01T01:31:35.615739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:07:41.982481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489333+0000","last_change":"2024-04-01T01:31:16.387570+0000","last_active":"2024-04-01T01:31:22.489333+0000","last_peered":"2024-04-01T01:31:22.489333+0000","last_clean":"2024-04-01T01:31:22.489333+0000","last_became_active":"2024-04-01T01:31:16.387200+0000","last_became_peered":"2024-04-01T01:31:16.387200+0000","last_unstale":"2024-04-01T01:31:22.489333+0000","last_undegraded":"2024-04-01T01:31:22.489333+0000","last_fullsized":"2024-04-01T01:31:22.489333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:52:27.813722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727106+0000","last_change":"2024-04-01T01:31:20.569668+0000","last_active":"2024-04-01T01:31:22.727106+0000","last_peered":"2024-04-01T01:31:22.727106+0000","last_clean":"2024-04-01T01:31:22.727106+0000","last_became_active":"2024-04-01T01:31:20.569148+0000","last_became_peered":"2024-04-01T01:31:20.569148+0000","last_unstale":"2024-04-01T01:31:22.727106+0000","last_undegraded":"2024-04-01T01:31:22.727106+0000","last_fullsized":"2024-04-01T01:31:22.727106+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:20:56.048052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:34.621552+0000","last_change":"2024-04-01T01:31:15.379323+0000","last_active":"2024-04-01T01:31:34.621552+0000","last_peered":"2024-04-01T01:31:34.621552+0000","last_clean":"2024-04-01T01:31:34.621552+0000","last_became_active":"2024-04-01T01:31:15.379023+0000","last_became_peered":"2024-04-01T01:31:15.379023+0000","last_unstale":"2024-04-01T01:31:34.621552+0000","last_undegraded":"2024-04-01T01:31:34.621552+0000","last_fullsized":"2024-04-01T01:31:34.621552+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:05:37.709040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563654+0000","last_change":"2024-04-01T01:31:16.387250+0000","last_active":"2024-04-01T01:31:21.563654+0000","last_peered":"2024-04-01T01:31:21.563654+0000","last_clean":"2024-04-01T01:31:21.563654+0000","last_became_active":"2024-04-01T01:31:16.386940+0000","last_became_peered":"2024-04-01T01:31:16.386940+0000","last_unstale":"2024-04-01T01:31:21.563654+0000","last_undegraded":"2024-04-01T01:31:21.563654+0000","last_fullsized":"2024-04-01T01:31:21.563654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:19:34.886738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726827+0000","last_change":"2024-04-01T01:31:20.573228+0000","last_active":"2024-04-01T01:31:22.726827+0000","last_peered":"2024-04-01T01:31:22.726827+0000","last_clean":"2024-04-01T01:31:22.726827+0000","last_became_active":"2024-04-01T01:31:20.572681+0000","last_became_peered":"2024-04-01T01:31:20.572681+0000","last_unstale":"2024-04-01T01:31:22.726827+0000","last_undegraded":"2024-04-01T01:31:22.726827+0000","last_fullsized":"2024-04-01T01:31:22.726827+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:58:17.812966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:36.556687+0000","last_change":"2024-04-01T01:31:15.381477+0000","last_active":"2024-04-01T01:31:36.556687+0000","last_peered":"2024-04-01T01:31:36.556687+0000","last_clean":"2024-04-01T01:31:36.556687+0000","last_became_active":"2024-04-01T01:31:15.381166+0000","last_became_peered":"2024-04-01T01:31:15.381166+0000","last_unstale":"2024-04-01T01:31:36.556687+0000","last_undegraded":"2024-04-01T01:31:36.556687+0000","last_fullsized":"2024-04-01T01:31:36.556687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:19.029896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564006+0000","last_change":"2024-04-01T01:31:16.386234+0000","last_active":"2024-04-01T01:31:21.564006+0000","last_peered":"2024-04-01T01:31:21.564006+0000","last_clean":"2024-04-01T01:31:21.564006+0000","last_became_active":"2024-04-01T01:31:16.385876+0000","last_became_peered":"2024-04-01T01:31:16.385876+0000","last_unstale":"2024-04-01T01:31:21.564006+0000","last_undegraded":"2024-04-01T01:31:21.564006+0000","last_fullsized":"2024-04-01T01:31:21.564006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:34.187210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726675+0000","last_change":"2024-04-01T01:31:20.580499+0000","last_active":"2024-04-01T01:31:22.726675+0000","last_peered":"2024-04-01T01:31:22.726675+0000","last_clean":"2024-04-01T01:31:22.726675+0000","last_became_active":"2024-04-01T01:31:20.579796+0000","last_became_peered":"2024-04-01T01:31:20.579796+0000","last_unstale":"2024-04-01T01:31:22.726675+0000","last_undegraded":"2024-04-01T01:31:22.726675+0000","last_fullsized":"2024-04-01T01:31:22.726675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:59:10.852056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.728390+0000","last_change":"2024-04-01T01:31:15.378503+0000","last_active":"2024-04-01T01:31:22.728390+0000","last_peered":"2024-04-01T01:31:22.728390+0000","last_clean":"2024-04-01T01:31:22.728390+0000","last_became_active":"2024-04-01T01:31:15.378152+0000","last_became_peered":"2024-04-01T01:31:15.378152+0000","last_unstale":"2024-04-01T01:31:22.728390+0000","last_undegraded":"2024-04-01T01:31:22.728390+0000","last_fullsized":"2024-04-01T01:31:22.728390+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:15:42.118691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.563902+0000","last_change":"2024-04-01T01:31:16.389676+0000","last_active":"2024-04-01T01:31:21.563902+0000","last_peered":"2024-04-01T01:31:21.563902+0000","last_clean":"2024-04-01T01:31:21.563902+0000","last_became_active":"2024-04-01T01:31:16.389304+0000","last_became_peered":"2024-04-01T01:31:16.389304+0000","last_unstale":"2024-04-01T01:31:21.563902+0000","last_undegraded":"2024-04-01T01:31:21.563902+0000","last_fullsized":"2024-04-01T01:31:21.563902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:20:24.021273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727031+0000","last_change":"2024-04-01T01:31:20.569461+0000","last_active":"2024-04-01T01:31:22.727031+0000","last_peered":"2024-04-01T01:31:22.727031+0000","last_clean":"2024-04-01T01:31:22.727031+0000","last_became_active":"2024-04-01T01:31:20.569011+0000","last_became_peered":"2024-04-01T01:31:20.569011+0000","last_unstale":"2024-04-01T01:31:22.727031+0000","last_undegraded":"2024-04-01T01:31:22.727031+0000","last_fullsized":"2024-04-01T01:31:22.727031+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:43:43.657065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989896+0000","last_change":"2024-04-01T01:31:15.384181+0000","last_active":"2024-04-01T01:31:22.989896+0000","last_peered":"2024-04-01T01:31:22.989896+0000","last_clean":"2024-04-01T01:31:22.989896+0000","last_became_active":"2024-04-01T01:31:15.383886+0000","last_became_peered":"2024-04-01T01:31:15.383886+0000","last_unstale":"2024-04-01T01:31:22.989896+0000","last_undegraded":"2024-04-01T01:31:22.989896+0000","last_fullsized":"2024-04-01T01:31:22.989896+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:43:28.414423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564097+0000","last_change":"2024-04-01T01:31:16.381877+0000","last_active":"2024-04-01T01:31:21.564097+0000","last_peered":"2024-04-01T01:31:21.564097+0000","last_clean":"2024-04-01T01:31:21.564097+0000","last_became_active":"2024-04-01T01:31:16.381558+0000","last_became_peered":"2024-04-01T01:31:16.381558+0000","last_unstale":"2024-04-01T01:31:21.564097+0000","last_undegraded":"2024-04-01T01:31:21.564097+0000","last_fullsized":"2024-04-01T01:31:21.564097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:24.120910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727030+0000","last_change":"2024-04-01T01:31:16.380116+0000","last_active":"2024-04-01T01:31:22.727030+0000","last_peered":"2024-04-01T01:31:22.727030+0000","last_clean":"2024-04-01T01:31:22.727030+0000","last_became_active":"2024-04-01T01:31:16.378936+0000","last_became_peered":"2024-04-01T01:31:16.378936+0000","last_unstale":"2024-04-01T01:31:22.727030+0000","last_undegraded":"2024-04-01T01:31:22.727030+0000","last_fullsized":"2024-04-01T01:31:22.727030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:18:16.211113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564540+0000","last_change":"2024-04-01T01:31:16.384872+0000","last_active":"2024-04-01T01:31:21.564540+0000","last_peered":"2024-04-01T01:31:21.564540+0000","last_clean":"2024-04-01T01:31:21.564540+0000","last_became_active":"2024-04-01T01:31:16.384406+0000","last_became_peered":"2024-04-01T01:31:16.384406+0000","last_unstale":"2024-04-01T01:31:21.564540+0000","last_undegraded":"2024-04-01T01:31:21.564540+0000","last_fullsized":"2024-04-01T01:31:21.564540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:13:36.567357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564110+0000","last_change":"2024-04-01T01:31:16.384561+0000","last_active":"2024-04-01T01:31:21.564110+0000","last_peered":"2024-04-01T01:31:21.564110+0000","last_clean":"2024-04-01T01:31:21.564110+0000","last_became_active":"2024-04-01T01:31:16.384221+0000","last_became_peered":"2024-04-01T01:31:16.384221+0000","last_unstale":"2024-04-01T01:31:21.564110+0000","last_undegraded":"2024-04-01T01:31:21.564110+0000","last_fullsized":"2024-04-01T01:31:21.564110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:17:56.490000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.726795+0000","last_change":"2024-04-01T01:31:16.385464+0000","last_active":"2024-04-01T01:31:22.726795+0000","last_peered":"2024-04-01T01:31:22.726795+0000","last_clean":"2024-04-01T01:31:22.726795+0000","last_became_active":"2024-04-01T01:31:16.385159+0000","last_became_peered":"2024-04-01T01:31:16.385159+0000","last_unstale":"2024-04-01T01:31:22.726795+0000","last_undegraded":"2024-04-01T01:31:22.726795+0000","last_fullsized":"2024-04-01T01:31:22.726795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:33:53.852957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989615+0000","last_change":"2024-04-01T01:31:16.385688+0000","last_active":"2024-04-01T01:31:22.989615+0000","last_peered":"2024-04-01T01:31:22.989615+0000","last_clean":"2024-04-01T01:31:22.989615+0000","last_became_active":"2024-04-01T01:31:16.385381+0000","last_became_peered":"2024-04-01T01:31:16.385381+0000","last_unstale":"2024-04-01T01:31:22.989615+0000","last_undegraded":"2024-04-01T01:31:22.989615+0000","last_fullsized":"2024-04-01T01:31:22.989615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:39:36.810766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727844+0000","last_change":"2024-04-01T01:31:16.379976+0000","last_active":"2024-04-01T01:31:22.727844+0000","last_peered":"2024-04-01T01:31:22.727844+0000","last_clean":"2024-04-01T01:31:22.727844+0000","last_became_active":"2024-04-01T01:31:16.379553+0000","last_became_peered":"2024-04-01T01:31:16.379553+0000","last_unstale":"2024-04-01T01:31:22.727844+0000","last_undegraded":"2024-04-01T01:31:22.727844+0000","last_fullsized":"2024-04-01T01:31:22.727844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:48:23.555723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:21.564765+0000","last_change":"2024-04-01T01:31:16.383995+0000","last_active":"2024-04-01T01:31:21.564765+0000","last_peered":"2024-04-01T01:31:21.564765+0000","last_clean":"2024-04-01T01:31:21.564765+0000","last_became_active":"2024-04-01T01:31:16.383673+0000","last_became_peered":"2024-04-01T01:31:16.383673+0000","last_unstale":"2024-04-01T01:31:21.564765+0000","last_undegraded":"2024-04-01T01:31:21.564765+0000","last_fullsized":"2024-04-01T01:31:21.564765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:03:16.192438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489485+0000","last_change":"2024-04-01T01:31:16.389969+0000","last_active":"2024-04-01T01:31:22.489485+0000","last_peered":"2024-04-01T01:31:22.489485+0000","last_clean":"2024-04-01T01:31:22.489485+0000","last_became_active":"2024-04-01T01:31:16.389640+0000","last_became_peered":"2024-04-01T01:31:16.389640+0000","last_unstale":"2024-04-01T01:31:22.489485+0000","last_undegraded":"2024-04-01T01:31:22.489485+0000","last_fullsized":"2024-04-01T01:31:22.489485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:35:15.226960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727211+0000","last_change":"2024-04-01T01:31:16.382962+0000","last_active":"2024-04-01T01:31:22.727211+0000","last_peered":"2024-04-01T01:31:22.727211+0000","last_clean":"2024-04-01T01:31:22.727211+0000","last_became_active":"2024-04-01T01:31:16.382653+0000","last_became_peered":"2024-04-01T01:31:16.382653+0000","last_unstale":"2024-04-01T01:31:22.727211+0000","last_undegraded":"2024-04-01T01:31:22.727211+0000","last_fullsized":"2024-04-01T01:31:22.727211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:53.054884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.489203+0000","last_change":"2024-04-01T01:31:16.382997+0000","last_active":"2024-04-01T01:31:22.489203+0000","last_peered":"2024-04-01T01:31:22.489203+0000","last_clean":"2024-04-01T01:31:22.489203+0000","last_became_active":"2024-04-01T01:31:16.380180+0000","last_became_peered":"2024-04-01T01:31:16.380180+0000","last_unstale":"2024-04-01T01:31:22.489203+0000","last_undegraded":"2024-04-01T01:31:22.489203+0000","last_fullsized":"2024-04-01T01:31:22.489203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:29:51.990395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727267+0000","last_change":"2024-04-01T01:31:16.382948+0000","last_active":"2024-04-01T01:31:22.727267+0000","last_peered":"2024-04-01T01:31:22.727267+0000","last_clean":"2024-04-01T01:31:22.727267+0000","last_became_active":"2024-04-01T01:31:16.382134+0000","last_became_peered":"2024-04-01T01:31:16.382134+0000","last_unstale":"2024-04-01T01:31:22.727267+0000","last_undegraded":"2024-04-01T01:31:22.727267+0000","last_fullsized":"2024-04-01T01:31:22.727267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:24:08.515063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.989688+0000","last_change":"2024-04-01T01:31:16.378841+0000","last_active":"2024-04-01T01:31:22.989688+0000","last_peered":"2024-04-01T01:31:22.989688+0000","last_clean":"2024-04-01T01:31:22.989688+0000","last_became_active":"2024-04-01T01:31:16.378501+0000","last_became_peered":"2024-04-01T01:31:16.378501+0000","last_unstale":"2024-04-01T01:31:22.989688+0000","last_undegraded":"2024-04-01T01:31:22.989688+0000","last_fullsized":"2024-04-01T01:31:22.989688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:50:20.881852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-01T01:31:22.727402+0000","last_change":"2024-04-01T01:31:16.380979+0000","last_active":"2024-04-01T01:31:22.727402+0000","last_peered":"2024-04-01T01:31:22.727402+0000","last_clean":"2024-04-01T01:31:22.727402+0000","last_became_active":"2024-04-01T01:31:16.380625+0000","last_became_peered":"2024-04-01T01:31:16.380625+0000","last_unstale":"2024-04-01T01:31:22.727402+0000","last_undegraded":"2024-04-01T01:31:22.727402+0000","last_fullsized":"2024-04-01T01:31:22.727402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:09:03.246239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8678,"num_objects":67,"num_object_clones":0,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":17,"num_read_kb":18,"num_write":140,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":42864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383697,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36912,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708240,"statfs":{"total":95995035648,"available":95957237760,"internally_reserved":0,"allocated":1613824,"data_stored":1072381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808859,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":987136,"data_stored":477797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070496,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":478715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36888,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708264,"statfs":{"total":95995035648,"available":95957262336,"internally_reserved":0,"allocated":1581056,"data_stored":1068165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495658,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":481057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":478693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":483029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":477499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920819,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":477859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28584,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716568,"statfs":{"total":95995035648,"available":95965765632,"internally_reserved":0,"allocated":1605632,"data_stored":1069059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":25504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-01T01:31:40.281 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-01T01:31:40.282 INFO:tasks.cephadm:Setting up client nodes... 2024-04-01T01:31:40.282 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-01T01:31:40.557 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:40.961 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:40 smithi070 ceph-mon[32179]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:31:40.961 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:40 smithi070 ceph-mon[32179]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-01T01:31:40.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:40 smithi189 ceph-mon[35753]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:31:40.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:40 smithi189 ceph-mon[35753]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-01T01:31:41.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:40 smithi081 ceph-mon[35142]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:31:41.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:40 smithi081 ceph-mon[35142]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-01T01:31:41.619 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:31:41 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-b[65004]: 2024-04-01T01:31:41.345+0000 7ff9913f6700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-01T01:31:41.620 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:31:41 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-b[65004]: 2024-04-01T01:31:41.345+0000 7ff9913f6700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-01T01:31:41.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:41 smithi189 ceph-mon[35753]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:31:41.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:41 smithi189 ceph-mon[35753]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:31:41.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:41 smithi189 ceph-mon[35753]: pgmap v300: 129 pgs: 129 active+clean; 586 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3.8 KiB/s wr, 14 op/s 2024-04-01T01:31:42.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:41 smithi070 ceph-mon[32179]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:31:42.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:41 smithi070 ceph-mon[32179]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:31:42.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:41 smithi070 ceph-mon[32179]: pgmap v300: 129 pgs: 129 active+clean; 586 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3.8 KiB/s wr, 14 op/s 2024-04-01T01:31:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:41 smithi081 ceph-mon[35142]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:31:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:41 smithi081 ceph-mon[35142]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:31:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:41 smithi081 ceph-mon[35142]: pgmap v300: 129 pgs: 129 active+clean; 586 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 3.8 KiB/s wr, 14 op/s 2024-04-01T01:31:42.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.447+0000 7f3f754d5700 1 -- 172.21.15.70:0/2194086063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f70102d10 msgr2=0x7f3f70105100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:42.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.447+0000 7f3f754d5700 1 --2- 172.21.15.70:0/2194086063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f70102d10 0x7f3f70105100 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3f58009990 tx=0x7f3f58009ca0 comp rx=0 tx=0).stop 2024-04-01T01:31:42.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.447+0000 7f3f754d5700 1 -- 172.21.15.70:0/2194086063 shutdown_connections 2024-04-01T01:31:42.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.447+0000 7f3f754d5700 1 --2- 172.21.15.70:0/2194086063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f70107f70 0x7f3f7010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:42.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.447+0000 7f3f754d5700 1 --2- 172.21.15.70:0/2194086063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f70102d10 0x7f3f70105100 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:42.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.447+0000 7f3f754d5700 1 --2- 172.21.15.70:0/2194086063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f70105640 0x7f3f70107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:42.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.447+0000 7f3f754d5700 1 -- 172.21.15.70:0/2194086063 >> 172.21.15.70:0/2194086063 conn(0x7f3f700fc740 msgr2=0x7f3f700feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:42.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.447+0000 7f3f754d5700 1 -- 172.21.15.70:0/2194086063 shutdown_connections 2024-04-01T01:31:42.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.447+0000 7f3f754d5700 1 -- 172.21.15.70:0/2194086063 wait complete. 2024-04-01T01:31:42.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.448+0000 7f3f754d5700 1 Processor -- start 2024-04-01T01:31:42.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.448+0000 7f3f754d5700 1 -- start start 2024-04-01T01:31:42.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.448+0000 7f3f754d5700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f70102d10 0x7f3f70102670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:42.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.448+0000 7f3f754d5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f70105640 0x7f3f70100cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:42.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.448+0000 7f3f754d5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f70107f70 0x7f3f70101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:42.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.448+0000 7f3f754d5700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3f7010dd90 con 0x7f3f70107f70 2024-04-01T01:31:42.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.448+0000 7f3f754d5700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3f7010dc10 con 0x7f3f70102d10 2024-04-01T01:31:42.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.448+0000 7f3f754d5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f7010df10 con 0x7f3f70105640 2024-04-01T01:31:42.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.448+0000 7f3f6effd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f70102d10 0x7f3f70102670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:42.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.448+0000 7f3f6e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f70105640 0x7f3f70100cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:42.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.448+0000 7f3f6f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f70107f70 0x7f3f70101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:42.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.448+0000 7f3f6effd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f70102d10 0x7f3f70102670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:50326/0 (socket says 172.21.15.70:50326) 2024-04-01T01:31:42.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.448+0000 7f3f6effd700 1 -- 172.21.15.70:0/574472752 learned_addr learned my addr 172.21.15.70:0/574472752 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:42.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.448+0000 7f3f6e7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f70105640 0x7f3f70100cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:35296/0 (socket says 172.21.15.70:35296) 2024-04-01T01:31:42.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.449+0000 7f3f6e7fc700 1 -- 172.21.15.70:0/574472752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f70102d10 msgr2=0x7f3f70102670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:42.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.449+0000 7f3f6e7fc700 1 --2- 172.21.15.70:0/574472752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f70102d10 0x7f3f70102670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:42.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.449+0000 7f3f6e7fc700 1 -- 172.21.15.70:0/574472752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f70107f70 msgr2=0x7f3f70101200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:42.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.449+0000 7f3f6e7fc700 1 --2- 172.21.15.70:0/574472752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f70107f70 0x7f3f70101200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:42.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.449+0000 7f3f6e7fc700 1 -- 172.21.15.70:0/574472752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f70101ab0 con 0x7f3f70105640 2024-04-01T01:31:42.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.449+0000 7f3f6effd700 1 --2- 172.21.15.70:0/574472752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f70102d10 0x7f3f70102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:42.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.449+0000 7f3f6e7fc700 1 --2- 172.21.15.70:0/574472752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f70105640 0x7f3f70100cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3f6000e9f0 tx=0x7f3f6000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:42.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.449+0000 7f3f67fff700 1 -- 172.21.15.70:0/574472752 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6000c9c0 con 0x7f3f70105640 2024-04-01T01:31:42.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.449+0000 7f3f67fff700 1 -- 172.21.15.70:0/574472752 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f6000cb60 con 0x7f3f70105640 2024-04-01T01:31:42.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.449+0000 7f3f754d5700 1 -- 172.21.15.70:0/574472752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f701a2490 con 0x7f3f70105640 2024-04-01T01:31:42.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.449+0000 7f3f67fff700 1 -- 172.21.15.70:0/574472752 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f60018680 con 0x7f3f70105640 2024-04-01T01:31:42.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.449+0000 7f3f754d5700 1 -- 172.21.15.70:0/574472752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f701a29a0 con 0x7f3f70105640 2024-04-01T01:31:42.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.450+0000 7f3f754d5700 1 -- 172.21.15.70:0/574472752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f70050750 con 0x7f3f70105640 2024-04-01T01:31:42.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.452+0000 7f3f67fff700 1 -- 172.21.15.70:0/574472752 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f3f60047ab0 con 0x7f3f70105640 2024-04-01T01:31:42.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.453+0000 7f3f67fff700 1 --2- 172.21.15.70:0/574472752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f5c0a2670 0x7f3f5c0a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:42.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.453+0000 7f3f6effd700 1 --2- 172.21.15.70:0/574472752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f5c0a2670 0x7f3f5c0a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:42.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.453+0000 7f3f67fff700 1 -- 172.21.15.70:0/574472752 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3f600ced30 con 0x7f3f70105640 2024-04-01T01:31:42.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.453+0000 7f3f6effd700 1 --2- 172.21.15.70:0/574472752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f5c0a2670 0x7f3f5c0a4b30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800b2e0 tx=0x7f3f5801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:42.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.456+0000 7f3f67fff700 1 -- 172.21.15.70:0/574472752 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3f6009bb50 con 0x7f3f70105640 2024-04-01T01:31:42.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.743+0000 7f3f754d5700 1 -- 172.21.15.70:0/574472752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f3f70062e50 con 0x7f3f70105640 2024-04-01T01:31:42.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.748+0000 7f3f67fff700 1 -- 172.21.15.70:0/574472752 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f3f6009b450 con 0x7f3f70105640 2024-04-01T01:31:42.748 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-04-01T01:31:42.748 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQB+DgpmzzZkLBAAMUukApdxgiJmnK/YJF/nWQ== 2024-04-01T01:31:42.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.750+0000 7f3f754d5700 1 -- 172.21.15.70:0/574472752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f5c0a2670 msgr2=0x7f3f5c0a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:42.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.750+0000 7f3f754d5700 1 --2- 172.21.15.70:0/574472752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f5c0a2670 0x7f3f5c0a4b30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800b2e0 tx=0x7f3f5801a040 comp rx=0 tx=0).stop 2024-04-01T01:31:42.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.750+0000 7f3f754d5700 1 -- 172.21.15.70:0/574472752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f70105640 msgr2=0x7f3f70100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:42.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.750+0000 7f3f754d5700 1 --2- 172.21.15.70:0/574472752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f70105640 0x7f3f70100cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3f6000e9f0 tx=0x7f3f6000ed00 comp rx=0 tx=0).stop 2024-04-01T01:31:42.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.750+0000 7f3f754d5700 1 -- 172.21.15.70:0/574472752 shutdown_connections 2024-04-01T01:31:42.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.750+0000 7f3f754d5700 1 --2- 172.21.15.70:0/574472752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f5c0a2670 0x7f3f5c0a4b30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:42.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.750+0000 7f3f754d5700 1 --2- 172.21.15.70:0/574472752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f70107f70 0x7f3f70101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:42.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.750+0000 7f3f754d5700 1 --2- 172.21.15.70:0/574472752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f70102d10 0x7f3f70102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:42.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.750+0000 7f3f754d5700 1 --2- 172.21.15.70:0/574472752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f70105640 0x7f3f70100cc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:42.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.750+0000 7f3f754d5700 1 -- 172.21.15.70:0/574472752 >> 172.21.15.70:0/574472752 conn(0x7f3f700fc740 msgr2=0x7f3f70103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:42.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.751+0000 7f3f754d5700 1 -- 172.21.15.70:0/574472752 shutdown_connections 2024-04-01T01:31:42.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:42.751+0000 7f3f754d5700 1 -- 172.21.15.70:0/574472752 wait complete. 2024-04-01T01:31:43.564 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:31:43.564 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-01T01:31:43.564 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-01T01:31:43.608 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-01T01:31:43.609 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-01T01:31:43.609 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph mgr dump --format=json 2024-04-01T01:31:43.673 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:43 smithi070 ceph-mon[32179]: pgmap v301: 129 pgs: 129 active+clean; 623 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 8.2 KiB/s wr, 20 op/s 2024-04-01T01:31:43.673 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:43 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/574472752' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-01T01:31:43.673 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:43 smithi070 ceph-mon[32179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-01T01:31:43.674 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:43 smithi070 ceph-mon[32179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-01T01:31:43.891 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:43.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:43 smithi189 ceph-mon[35753]: pgmap v301: 129 pgs: 129 active+clean; 623 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 8.2 KiB/s wr, 20 op/s 2024-04-01T01:31:43.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:43 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/574472752' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-01T01:31:43.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:43 smithi189 ceph-mon[35753]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-01T01:31:43.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:43 smithi189 ceph-mon[35753]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-01T01:31:44.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:43 smithi081 ceph-mon[35142]: pgmap v301: 129 pgs: 129 active+clean; 623 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 8.2 KiB/s wr, 20 op/s 2024-04-01T01:31:44.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:43 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/574472752' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-01T01:31:44.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:43 smithi081 ceph-mon[35142]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-01T01:31:44.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:43 smithi081 ceph-mon[35142]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-01T01:31:45.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.610+0000 7ff8c1438700 1 -- 172.21.15.70:0/1123078193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8bc101ae0 msgr2=0x7ff8bc10df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:45.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.610+0000 7ff8c1438700 1 --2- 172.21.15.70:0/1123078193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8bc101ae0 0x7ff8bc10df80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009990 tx=0x7ff8ac009ca0 comp rx=0 tx=0).stop 2024-04-01T01:31:45.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.610+0000 7ff8c1438700 1 -- 172.21.15.70:0/1123078193 shutdown_connections 2024-04-01T01:31:45.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.610+0000 7ff8c1438700 1 --2- 172.21.15.70:0/1123078193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8bc101ae0 0x7ff8bc10df80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:45.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.610+0000 7ff8c1438700 1 --2- 172.21.15.70:0/1123078193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8bc1071d0 0x7ff8bc1075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:45.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.610+0000 7ff8c1438700 1 --2- 172.21.15.70:0/1123078193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8bc101120 0x7ff8bc1015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:45.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.610+0000 7ff8c1438700 1 -- 172.21.15.70:0/1123078193 >> 172.21.15.70:0/1123078193 conn(0x7ff8bc078150 msgr2=0x7ff8bc078560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:45.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.610+0000 7ff8c1438700 1 -- 172.21.15.70:0/1123078193 shutdown_connections 2024-04-01T01:31:45.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.610+0000 7ff8c1438700 1 -- 172.21.15.70:0/1123078193 wait complete. 2024-04-01T01:31:45.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.611+0000 7ff8c1438700 1 Processor -- start 2024-04-01T01:31:45.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.611+0000 7ff8c1438700 1 -- start start 2024-04-01T01:31:45.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.611+0000 7ff8c1438700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8bc101120 0x7ff8bc19d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:45.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.611+0000 7ff8c1438700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8bc101ae0 0x7ff8bc19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:45.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.611+0000 7ff8c1438700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8bc1071d0 0x7ff8bc1975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:45.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.611+0000 7ff8c1438700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc1112b0 con 0x7ff8bc1071d0 2024-04-01T01:31:45.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.611+0000 7ff8c1438700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc111130 con 0x7ff8bc101120 2024-04-01T01:31:45.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.611+0000 7ff8c1438700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff8bc111430 con 0x7ff8bc101ae0 2024-04-01T01:31:45.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.611+0000 7ff8bb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8bc1071d0 0x7ff8bc1975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:45.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.611+0000 7ff8ba7fc700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8bc101ae0 0x7ff8bc19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:45.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.611+0000 7ff8bb7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8bc1071d0 0x7ff8bc1975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51718/0 (socket says 172.21.15.70:51718) 2024-04-01T01:31:45.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.611+0000 7ff8bb7fe700 1 -- 172.21.15.70:0/678449129 learned_addr learned my addr 172.21.15.70:0/678449129 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:45.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.611+0000 7ff8baffd700 1 --2- 172.21.15.70:0/678449129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8bc101120 0x7ff8bc19d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:45.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.612+0000 7ff8ba7fc700 1 -- 172.21.15.70:0/678449129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8bc101120 msgr2=0x7ff8bc19d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:45.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.612+0000 7ff8ba7fc700 1 --2- 172.21.15.70:0/678449129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8bc101120 0x7ff8bc19d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:45.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.612+0000 7ff8ba7fc700 1 -- 172.21.15.70:0/678449129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8bc1071d0 msgr2=0x7ff8bc1975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:45.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.612+0000 7ff8ba7fc700 1 --2- 172.21.15.70:0/678449129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8bc1071d0 0x7ff8bc1975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:45.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.612+0000 7ff8ba7fc700 1 -- 172.21.15.70:0/678449129 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8bc197d10 con 0x7ff8bc101ae0 2024-04-01T01:31:45.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.612+0000 7ff8bb7fe700 1 --2- 172.21.15.70:0/678449129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8bc1071d0 0x7ff8bc1975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:45.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.612+0000 7ff8ba7fc700 1 --2- 172.21.15.70:0/678449129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8bc101ae0 0x7ff8bc19d930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff8b000e9f0 tx=0x7ff8b000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:45.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.612+0000 7ff8a3fff700 1 -- 172.21.15.70:0/678449129 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b000c9c0 con 0x7ff8bc101ae0 2024-04-01T01:31:45.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.612+0000 7ff8c1438700 1 -- 172.21.15.70:0/678449129 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8bc198000 con 0x7ff8bc101ae0 2024-04-01T01:31:45.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.612+0000 7ff8a3fff700 1 -- 172.21.15.70:0/678449129 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8b000cb60 con 0x7ff8bc101ae0 2024-04-01T01:31:45.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.612+0000 7ff8a3fff700 1 -- 172.21.15.70:0/678449129 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0018680 con 0x7ff8bc101ae0 2024-04-01T01:31:45.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.612+0000 7ff8c1438700 1 -- 172.21.15.70:0/678449129 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8bc1a42c0 con 0x7ff8bc101ae0 2024-04-01T01:31:45.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.613+0000 7ff8c1438700 1 -- 172.21.15.70:0/678449129 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8bc068f00 con 0x7ff8bc101ae0 2024-04-01T01:31:45.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.614+0000 7ff8a3fff700 1 -- 172.21.15.70:0/678449129 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7ff8b0010430 con 0x7ff8bc101ae0 2024-04-01T01:31:45.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.615+0000 7ff8a3fff700 1 --2- 172.21.15.70:0/678449129 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff8a40a25c0 0x7ff8a40a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:45.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.615+0000 7ff8baffd700 1 --2- 172.21.15.70:0/678449129 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff8a40a25c0 0x7ff8a40a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:45.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.615+0000 7ff8a3fff700 1 -- 172.21.15.70:0/678449129 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff8b00cef70 con 0x7ff8bc101ae0 2024-04-01T01:31:45.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.615+0000 7ff8baffd700 1 --2- 172.21.15.70:0/678449129 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff8a40a25c0 0x7ff8a40a4a80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff8b400f540 tx=0x7ff8b400b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:45.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.618+0000 7ff8a3fff700 1 -- 172.21.15.70:0/678449129 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff8b009bd30 con 0x7ff8bc101ae0 2024-04-01T01:31:45.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.867+0000 7ff8c1438700 1 -- 172.21.15.70:0/678449129 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7ff8bc02db30 con 0x7ff8bc101ae0 2024-04-01T01:31:45.867 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:45 smithi070 ceph-mon[32179]: pgmap v302: 129 pgs: 129 active+clean; 623 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.0 KiB/s wr, 21 op/s 2024-04-01T01:31:45.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.875+0000 7ff8a3fff700 1 -- 172.21.15.70:0/678449129 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262361 (secure 0 0 0) 0x7ff8b009b630 con 0x7ff8bc101ae0 2024-04-01T01:31:45.881 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:45.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.878+0000 7ff8c1438700 1 -- 172.21.15.70:0/678449129 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff8a40a25c0 msgr2=0x7ff8a40a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:45.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.878+0000 7ff8c1438700 1 --2- 172.21.15.70:0/678449129 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff8a40a25c0 0x7ff8a40a4a80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff8b400f540 tx=0x7ff8b400b040 comp rx=0 tx=0).stop 2024-04-01T01:31:45.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.878+0000 7ff8c1438700 1 -- 172.21.15.70:0/678449129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8bc101ae0 msgr2=0x7ff8bc19d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:45.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.878+0000 7ff8c1438700 1 --2- 172.21.15.70:0/678449129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8bc101ae0 0x7ff8bc19d930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff8b000e9f0 tx=0x7ff8b000ed00 comp rx=0 tx=0).stop 2024-04-01T01:31:45.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.878+0000 7ff8c1438700 1 -- 172.21.15.70:0/678449129 shutdown_connections 2024-04-01T01:31:45.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.878+0000 7ff8c1438700 1 --2- 172.21.15.70:0/678449129 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff8a40a25c0 0x7ff8a40a4a80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:45.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.878+0000 7ff8c1438700 1 --2- 172.21.15.70:0/678449129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff8bc1071d0 0x7ff8bc1975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:45.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.878+0000 7ff8c1438700 1 --2- 172.21.15.70:0/678449129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8bc101120 0x7ff8bc19d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:45.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.878+0000 7ff8c1438700 1 --2- 172.21.15.70:0/678449129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8bc101ae0 0x7ff8bc19d930 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:45.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.878+0000 7ff8c1438700 1 -- 172.21.15.70:0/678449129 >> 172.21.15.70:0/678449129 conn(0x7ff8bc078150 msgr2=0x7ff8bc0fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:45.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.879+0000 7ff8c1438700 1 -- 172.21.15.70:0/678449129 shutdown_connections 2024-04-01T01:31:45.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:45.879+0000 7ff8c1438700 1 -- 172.21.15.70:0/678449129 wait complete. 2024-04-01T01:31:45.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:45 smithi189 ceph-mon[35753]: pgmap v302: 129 pgs: 129 active+clean; 623 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.0 KiB/s wr, 21 op/s 2024-04-01T01:31:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:45 smithi081 ceph-mon[35142]: pgmap v302: 129 pgs: 129 active+clean; 623 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.0 KiB/s wr, 21 op/s 2024-04-01T01:31:46.691 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":2070166741},{"type":"v1","addr":"172.21.15.70:6801","nonce":2070166741}]},"active_addr":"172.21.15.70:6801/2070166741","active_change":"2024-04-01T01:24:18.477720+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.70:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":3629482083}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":3255492857}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":2741070294}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":3639427196}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":1126005439}]}]} 2024-04-01T01:31:46.696 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-01T01:31:46.696 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-01T01:31:46.697 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-01T01:31:46.939 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:46.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:46 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/678449129' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-01T01:31:47.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:46 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/678449129' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-01T01:31:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:46 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/678449129' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-01T01:31:47.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:47 smithi189 ceph-mon[35753]: pgmap v303: 129 pgs: 129 active+clean; 628 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.6 KiB/s wr, 25 op/s 2024-04-01T01:31:48.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:47 smithi070 ceph-mon[32179]: pgmap v303: 129 pgs: 129 active+clean; 628 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.6 KiB/s wr, 25 op/s 2024-04-01T01:31:48.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:47 smithi081 ceph-mon[35142]: pgmap v303: 129 pgs: 129 active+clean; 628 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.6 KiB/s wr, 25 op/s 2024-04-01T01:31:48.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.789+0000 7fa48bec2700 1 -- 172.21.15.70:0/3270996572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa484102290 msgr2=0x7fa484102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:48.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.789+0000 7fa48bec2700 1 --2- 172.21.15.70:0/3270996572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa484102290 0x7fa484102710 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa4800099c0 tx=0x7fa480009cd0 comp rx=0 tx=0).stop 2024-04-01T01:31:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.789+0000 7fa48bec2700 1 -- 172.21.15.70:0/3270996572 shutdown_connections 2024-04-01T01:31:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.789+0000 7fa48bec2700 1 --2- 172.21.15.70:0/3270996572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa484102c50 0x7fa48410f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.789+0000 7fa48bec2700 1 --2- 172.21.15.70:0/3270996572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa484108270 0x7fa484108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.789+0000 7fa48bec2700 1 --2- 172.21.15.70:0/3270996572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa484102290 0x7fa484102710 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.789+0000 7fa48bec2700 1 -- 172.21.15.70:0/3270996572 >> 172.21.15.70:0/3270996572 conn(0x7fa484079370 msgr2=0x7fa484079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.789+0000 7fa48bec2700 1 -- 172.21.15.70:0/3270996572 shutdown_connections 2024-04-01T01:31:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.789+0000 7fa48bec2700 1 -- 172.21.15.70:0/3270996572 wait complete. 2024-04-01T01:31:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.790+0000 7fa48bec2700 1 Processor -- start 2024-04-01T01:31:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.790+0000 7fa48bec2700 1 -- start start 2024-04-01T01:31:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.790+0000 7fa48bec2700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa484102290 0x7fa48419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.790+0000 7fa48bec2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa484102c50 0x7fa48419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.790+0000 7fa48bec2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa484108270 0x7fa484198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.790+0000 7fa48bec2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa4841123d0 con 0x7fa484108270 2024-04-01T01:31:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.790+0000 7fa48bec2700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa484112250 con 0x7fa484102290 2024-04-01T01:31:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.790+0000 7fa48bec2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa484112550 con 0x7fa484102c50 2024-04-01T01:31:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.790+0000 7fa48a45f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa484108270 0x7fa484198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.790+0000 7fa48a45f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa484108270 0x7fa484198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39534/0 (socket says 172.21.15.70:39534) 2024-04-01T01:31:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.790+0000 7fa48a45f700 1 -- 172.21.15.70:0/1397612337 learned_addr learned my addr 172.21.15.70:0/1397612337 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.790+0000 7fa48945d700 1 --2- 172.21.15.70:0/1397612337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa484102c50 0x7fa48419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.790+0000 7fa489c5e700 1 --2- 172.21.15.70:0/1397612337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa484102290 0x7fa48419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.791+0000 7fa48a45f700 1 -- 172.21.15.70:0/1397612337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa484102c50 msgr2=0x7fa48419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.791+0000 7fa48a45f700 1 --2- 172.21.15.70:0/1397612337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa484102c50 0x7fa48419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.791+0000 7fa48a45f700 1 -- 172.21.15.70:0/1397612337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa484102290 msgr2=0x7fa48419e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.791+0000 7fa48a45f700 1 --2- 172.21.15.70:0/1397612337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa484102290 0x7fa48419e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.791+0000 7fa48a45f700 1 -- 172.21.15.70:0/1397612337 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa484199030 con 0x7fa484108270 2024-04-01T01:31:48.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.791+0000 7fa48a45f700 1 --2- 172.21.15.70:0/1397612337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa484108270 0x7fa484198750 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa47c00ef30 tx=0x7fa47c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:48.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.791+0000 7fa47affd700 1 -- 172.21.15.70:0/1397612337 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47c0092e0 con 0x7fa484108270 2024-04-01T01:31:48.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.791+0000 7fa47affd700 1 -- 172.21.15.70:0/1397612337 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa47c004bb0 con 0x7fa484108270 2024-04-01T01:31:48.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.791+0000 7fa48bec2700 1 -- 172.21.15.70:0/1397612337 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4841992c0 con 0x7fa484108270 2024-04-01T01:31:48.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.791+0000 7fa47affd700 1 -- 172.21.15.70:0/1397612337 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47c004020 con 0x7fa484108270 2024-04-01T01:31:48.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.791+0000 7fa48bec2700 1 -- 172.21.15.70:0/1397612337 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4841a5430 con 0x7fa484108270 2024-04-01T01:31:48.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.792+0000 7fa48bec2700 1 -- 172.21.15.70:0/1397612337 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa484103920 con 0x7fa484108270 2024-04-01T01:31:48.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.793+0000 7fa47affd700 1 -- 172.21.15.70:0/1397612337 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fa47c01b030 con 0x7fa484108270 2024-04-01T01:31:48.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.793+0000 7fa47affd700 1 --2- 172.21.15.70:0/1397612337 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4700a25c0 0x7fa4700a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:48.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.794+0000 7fa489c5e700 1 --2- 172.21.15.70:0/1397612337 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4700a25c0 0x7fa4700a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:48.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.794+0000 7fa47affd700 1 -- 172.21.15.70:0/1397612337 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa47c0ce5c0 con 0x7fa484108270 2024-04-01T01:31:48.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.794+0000 7fa489c5e700 1 --2- 172.21.15.70:0/1397612337 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4700a25c0 0x7fa4700a4a80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa474009be0 tx=0x7fa474009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:48.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.798+0000 7fa47affd700 1 -- 172.21.15.70:0/1397612337 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa47c022020 con 0x7fa484108270 2024-04-01T01:31:48.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.993+0000 7fa48bec2700 1 -- 172.21.15.70:0/1397612337 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa48402db30 con 0x7fa484108270 2024-04-01T01:31:48.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.995+0000 7fa47affd700 1 -- 172.21.15.70:0/1397612337 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24068 (secure 0 0 0) 0x7fa47c010040 con 0x7fa484108270 2024-04-01T01:31:48.995 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:48.996 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":89,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:31:20.903623+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:31:48.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.997+0000 7fa48bec2700 1 -- 172.21.15.70:0/1397612337 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4700a25c0 msgr2=0x7fa4700a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:48.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.997+0000 7fa48bec2700 1 --2- 172.21.15.70:0/1397612337 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4700a25c0 0x7fa4700a4a80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa474009be0 tx=0x7fa474009260 comp rx=0 tx=0).stop 2024-04-01T01:31:48.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.997+0000 7fa48bec2700 1 -- 172.21.15.70:0/1397612337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa484108270 msgr2=0x7fa484198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:48.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.997+0000 7fa48bec2700 1 --2- 172.21.15.70:0/1397612337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa484108270 0x7fa484198750 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa47c00ef30 tx=0x7fa47c00c3a0 comp rx=0 tx=0).stop 2024-04-01T01:31:48.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.997+0000 7fa48bec2700 1 -- 172.21.15.70:0/1397612337 shutdown_connections 2024-04-01T01:31:48.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.997+0000 7fa48bec2700 1 --2- 172.21.15.70:0/1397612337 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4700a25c0 0x7fa4700a4a80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:48.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.997+0000 7fa48bec2700 1 --2- 172.21.15.70:0/1397612337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa484108270 0x7fa484198750 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:48.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.997+0000 7fa48bec2700 1 --2- 172.21.15.70:0/1397612337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa484102290 0x7fa48419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:48.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.997+0000 7fa48bec2700 1 --2- 172.21.15.70:0/1397612337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa484102c50 0x7fa48419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:48.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.997+0000 7fa48bec2700 1 -- 172.21.15.70:0/1397612337 >> 172.21.15.70:0/1397612337 conn(0x7fa484079370 msgr2=0x7fa484101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:48.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.997+0000 7fa48bec2700 1 -- 172.21.15.70:0/1397612337 shutdown_connections 2024-04-01T01:31:48.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:48.997+0000 7fa48bec2700 1 -- 172.21.15.70:0/1397612337 wait complete. 2024-04-01T01:31:49.707 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-01T01:31:49.708 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-01T01:31:49.952 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:49.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:49 smithi189 ceph-mon[35753]: pgmap v304: 129 pgs: 129 active+clean; 628 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.1 KiB/s wr, 23 op/s 2024-04-01T01:31:49.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:49 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1397612337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:50.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:49 smithi070 ceph-mon[32179]: pgmap v304: 129 pgs: 129 active+clean; 628 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.1 KiB/s wr, 23 op/s 2024-04-01T01:31:50.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:49 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1397612337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:50.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:49 smithi081 ceph-mon[35142]: pgmap v304: 129 pgs: 129 active+clean; 628 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.1 KiB/s wr, 23 op/s 2024-04-01T01:31:50.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:49 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1397612337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:51.852 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:51 smithi070 ceph-mon[32179]: pgmap v305: 129 pgs: 129 active+clean; 628 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.1 KiB/s wr, 18 op/s 2024-04-01T01:31:51.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.850+0000 7f662824f700 1 -- 172.21.15.70:0/3669901059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6620102c50 msgr2=0x7f662010f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:51.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.850+0000 7f662824f700 1 --2- 172.21.15.70:0/3669901059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6620102c50 0x7f662010f090 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f66100099c0 tx=0x7f6610009cd0 comp rx=0 tx=0).stop 2024-04-01T01:31:51.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.850+0000 7f662824f700 1 -- 172.21.15.70:0/3669901059 shutdown_connections 2024-04-01T01:31:51.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.850+0000 7f662824f700 1 --2- 172.21.15.70:0/3669901059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6620102c50 0x7f662010f090 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:51.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.850+0000 7f662824f700 1 --2- 172.21.15.70:0/3669901059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6620108270 0x7f6620108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:51.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.850+0000 7f662824f700 1 --2- 172.21.15.70:0/3669901059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6620102290 0x7f6620102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:51.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.850+0000 7f662824f700 1 -- 172.21.15.70:0/3669901059 >> 172.21.15.70:0/3669901059 conn(0x7f6620079370 msgr2=0x7f6620079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:51.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.850+0000 7f662824f700 1 -- 172.21.15.70:0/3669901059 shutdown_connections 2024-04-01T01:31:51.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.850+0000 7f662824f700 1 -- 172.21.15.70:0/3669901059 wait complete. 2024-04-01T01:31:51.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.850+0000 7f662824f700 1 Processor -- start 2024-04-01T01:31:51.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.850+0000 7f662824f700 1 -- start start 2024-04-01T01:31:51.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f662824f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6620102290 0x7f662019e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:51.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f662824f700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6620102c50 0x7f662019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:51.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f662824f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6620108270 0x7f6620198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:51.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f662824f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f66201123d0 con 0x7f6620108270 2024-04-01T01:31:51.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f662824f700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6620112250 con 0x7f6620102c50 2024-04-01T01:31:51.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f662824f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6620112550 con 0x7f6620102290 2024-04-01T01:31:51.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f66257ea700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6620102c50 0x7f662019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:51.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f66267ec700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6620108270 0x7f6620198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:51.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f66257ea700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6620102c50 0x7f662019eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:58190/0 (socket says 172.21.15.70:58190) 2024-04-01T01:31:51.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f6625feb700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6620102290 0x7f662019e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:51.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f66257ea700 1 -- 172.21.15.70:0/107489255 learned_addr learned my addr 172.21.15.70:0/107489255 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:31:51.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f66267ec700 1 -- 172.21.15.70:0/107489255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6620102290 msgr2=0x7f662019e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:51.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f66267ec700 1 --2- 172.21.15.70:0/107489255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6620102290 0x7f662019e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:51.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f66267ec700 1 -- 172.21.15.70:0/107489255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6620102c50 msgr2=0x7f662019eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:51.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f66267ec700 1 --2- 172.21.15.70:0/107489255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6620102c50 0x7f662019eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:51.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f66267ec700 1 -- 172.21.15.70:0/107489255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6620198f10 con 0x7f6620108270 2024-04-01T01:31:51.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f66257ea700 1 --2- 172.21.15.70:0/107489255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6620102c50 0x7f662019eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:51.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f6625feb700 1 --2- 172.21.15.70:0/107489255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6620102290 0x7f662019e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:31:51.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f66267ec700 1 --2- 172.21.15.70:0/107489255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6620108270 0x7f6620198630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6610009990 tx=0x7f6610004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:51.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.851+0000 7f661effd700 1 -- 172.21.15.70:0/107489255 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6610009e60 con 0x7f6620108270 2024-04-01T01:31:51.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.852+0000 7f662824f700 1 -- 172.21.15.70:0/107489255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66201991a0 con 0x7f6620108270 2024-04-01T01:31:51.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.852+0000 7f661effd700 1 -- 172.21.15.70:0/107489255 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6610003c70 con 0x7f6620108270 2024-04-01T01:31:51.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.852+0000 7f661effd700 1 -- 172.21.15.70:0/107489255 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66100046b0 con 0x7f6620108270 2024-04-01T01:31:51.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.852+0000 7f662824f700 1 -- 172.21.15.70:0/107489255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66201a5310 con 0x7f6620108270 2024-04-01T01:31:51.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.853+0000 7f662824f700 1 -- 172.21.15.70:0/107489255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6620103920 con 0x7f6620108270 2024-04-01T01:31:51.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.853+0000 7f661effd700 1 -- 172.21.15.70:0/107489255 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f6610028060 con 0x7f6620108270 2024-04-01T01:31:51.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.854+0000 7f661effd700 1 --2- 172.21.15.70:0/107489255 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66080a2640 0x7f66080a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:31:51.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.854+0000 7f6625feb700 1 --2- 172.21.15.70:0/107489255 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66080a2640 0x7f66080a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:31:51.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.854+0000 7f661effd700 1 -- 172.21.15.70:0/107489255 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f66100cef50 con 0x7f6620108270 2024-04-01T01:31:51.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.854+0000 7f6625feb700 1 --2- 172.21.15.70:0/107489255 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66080a2640 0x7f66080a4b00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6618009480 tx=0x7f661800b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:31:51.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:51.859+0000 7f661effd700 1 -- 172.21.15.70:0/107489255 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6610013070 con 0x7f6620108270 2024-04-01T01:31:51.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:51 smithi189 ceph-mon[35753]: pgmap v305: 129 pgs: 129 active+clean; 628 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.1 KiB/s wr, 18 op/s 2024-04-01T01:31:52.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:52.051+0000 7f662824f700 1 -- 172.21.15.70:0/107489255 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f662002db30 con 0x7f6620108270 2024-04-01T01:31:52.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:52.053+0000 7f661effd700 1 -- 172.21.15.70:0/107489255 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24068 (secure 0 0 0) 0x7f661009be00 con 0x7f6620108270 2024-04-01T01:31:52.053 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:31:52.053 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":89,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:31:20.903623+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:31:52.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:52.055+0000 7f662824f700 1 -- 172.21.15.70:0/107489255 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66080a2640 msgr2=0x7f66080a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:52.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:52.055+0000 7f662824f700 1 --2- 172.21.15.70:0/107489255 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66080a2640 0x7f66080a4b00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6618009480 tx=0x7f661800b040 comp rx=0 tx=0).stop 2024-04-01T01:31:52.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:52.055+0000 7f662824f700 1 -- 172.21.15.70:0/107489255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6620108270 msgr2=0x7f6620198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:31:52.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:52.055+0000 7f662824f700 1 --2- 172.21.15.70:0/107489255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6620108270 0x7f6620198630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6610009990 tx=0x7f6610004270 comp rx=0 tx=0).stop 2024-04-01T01:31:52.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:52.055+0000 7f662824f700 1 -- 172.21.15.70:0/107489255 shutdown_connections 2024-04-01T01:31:52.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:52.055+0000 7f662824f700 1 --2- 172.21.15.70:0/107489255 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f66080a2640 0x7f66080a4b00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:52.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:52.055+0000 7f662824f700 1 --2- 172.21.15.70:0/107489255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6620108270 0x7f6620198630 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:52.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:52.055+0000 7f662824f700 1 --2- 172.21.15.70:0/107489255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6620102290 0x7f662019e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:52.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:52.055+0000 7f662824f700 1 --2- 172.21.15.70:0/107489255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6620102c50 0x7f662019eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:31:52.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:52.055+0000 7f662824f700 1 -- 172.21.15.70:0/107489255 >> 172.21.15.70:0/107489255 conn(0x7f6620079370 msgr2=0x7f6620101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:31:52.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:52.055+0000 7f662824f700 1 -- 172.21.15.70:0/107489255 shutdown_connections 2024-04-01T01:31:52.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:31:52.055+0000 7f662824f700 1 -- 172.21.15.70:0/107489255 wait complete. 2024-04-01T01:31:52.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:51 smithi081 ceph-mon[35142]: pgmap v305: 129 pgs: 129 active+clean; 628 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.1 KiB/s wr, 18 op/s 2024-04-01T01:31:52.866 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-04-01T01:31:52.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-04-01T01:31:52.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-04-01T01:31:52.868 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-04-01T01:31:52.869 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-04-01T01:31:52.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-04-01T01:31:52.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-04-01T01:31:52.871 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-04-01T01:31:52.872 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph tell osd.8 flush_pg_stats 2024-04-01T01:31:52.872 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph tell osd.9 flush_pg_stats 2024-04-01T01:31:52.873 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph tell osd.10 flush_pg_stats 2024-04-01T01:31:52.873 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph tell osd.11 flush_pg_stats 2024-04-01T01:31:52.921 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:52 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/107489255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:52.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:52 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/107489255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:53.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:52 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/107489255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:31:53.668 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:53.676 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:53.678 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:53.678 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:53.683 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:53.686 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:53.689 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:53.689 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:53.699 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:53.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:53 smithi070 ceph-mon[32179]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 16 op/s 2024-04-01T01:31:53.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:53 smithi189 ceph-mon[35753]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 16 op/s 2024-04-01T01:31:54.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:53 smithi081 ceph-mon[35142]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 16 op/s 2024-04-01T01:31:54.228 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:31:55.908 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:55 smithi070 ceph-mon[32179]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-01T01:31:55.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:55 smithi189 ceph-mon[35753]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-01T01:31:56.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:55 smithi081 ceph-mon[35142]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-01T01:31:57.903 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:57 smithi070 ceph-mon[32179]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-04-01T01:31:57.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:57 smithi189 ceph-mon[35753]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-04-01T01:31:58.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:57 smithi081 ceph-mon[35142]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-04-01T01:31:59.924 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:31:59 smithi070 ceph-mon[32179]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-01T01:31:59.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:31:59 smithi189 ceph-mon[35753]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-01T01:32:00.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:31:59 smithi081 ceph-mon[35142]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-01T01:32:01.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:01 smithi189 ceph-mon[35753]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-01T01:32:02.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:01 smithi070 ceph-mon[32179]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-01T01:32:02.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:01 smithi081 ceph-mon[35142]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-01T01:32:03.810 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:03 smithi070 ceph-mon[32179]: pgmap v311: 129 pgs: 129 active+clean; 644 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-01T01:32:03.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:03 smithi189 ceph-mon[35753]: pgmap v311: 129 pgs: 129 active+clean; 644 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-01T01:32:04.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:03 smithi081 ceph-mon[35142]: pgmap v311: 129 pgs: 129 active+clean; 644 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-01T01:32:05.974 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:05 smithi070 ceph-mon[32179]: pgmap v312: 129 pgs: 129 active+clean; 644 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-01T01:32:05.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:05 smithi189 ceph-mon[35753]: pgmap v312: 129 pgs: 129 active+clean; 644 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-01T01:32:06.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:05 smithi081 ceph-mon[35142]: pgmap v312: 129 pgs: 129 active+clean; 644 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-01T01:32:06.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.856+0000 7fc6665e6700 1 -- 172.21.15.70:0/1973211215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc660107180 msgr2=0x7fc660107560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:06.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.856+0000 7fc6665e6700 1 --2- 172.21.15.70:0/1973211215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc660107180 0x7fc660107560 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc65800b0a0 tx=0x7fc65800b3b0 comp rx=0 tx=0).stop 2024-04-01T01:32:06.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.856+0000 7fc6665e6700 1 -- 172.21.15.70:0/1973211215 shutdown_connections 2024-04-01T01:32:06.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.856+0000 7fc6665e6700 1 --2- 172.21.15.70:0/1973211215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc660101ae0 0x7fc66010dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:06.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.856+0000 7fc6665e6700 1 --2- 172.21.15.70:0/1973211215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc660107180 0x7fc660107560 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:06.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.856+0000 7fc6665e6700 1 --2- 172.21.15.70:0/1973211215 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc660101140 0x7fc6601015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:06.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.856+0000 7fc6665e6700 1 -- 172.21.15.70:0/1973211215 >> 172.21.15.70:0/1973211215 conn(0x7fc660078070 msgr2=0x7fc660078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:06.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.856+0000 7fc6665e6700 1 -- 172.21.15.70:0/1973211215 shutdown_connections 2024-04-01T01:32:06.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.856+0000 7fc6665e6700 1 -- 172.21.15.70:0/1973211215 wait complete. 2024-04-01T01:32:06.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc6665e6700 1 Processor -- start 2024-04-01T01:32:06.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc6665e6700 1 -- start start 2024-04-01T01:32:06.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc6665e6700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc660101140 0x7fc6601966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:06.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc6665e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc660101ae0 0x7fc660196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:06.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc6665e6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc660107180 0x7fc66019af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:06.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc6665e6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc660110200 con 0x7fc660107180 2024-04-01T01:32:06.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc6665e6700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc660110080 con 0x7fc660101140 2024-04-01T01:32:06.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc6665e6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc660110380 con 0x7fc660101ae0 2024-04-01T01:32:06.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc664b83700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc660107180 0x7fc66019af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:06.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc664b83700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc660107180 0x7fc66019af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32972/0 (socket says 172.21.15.70:32972) 2024-04-01T01:32:06.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc664b83700 1 -- 172.21.15.70:0/2398501353 learned_addr learned my addr 172.21.15.70:0/2398501353 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:06.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc65f7fe700 1 --2- 172.21.15.70:0/2398501353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc660101ae0 0x7fc660196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:06.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc65ffff700 1 --2- 172.21.15.70:0/2398501353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc660101140 0x7fc6601966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:06.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc664b83700 1 -- 172.21.15.70:0/2398501353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc660101ae0 msgr2=0x7fc660196bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:06.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc664b83700 1 --2- 172.21.15.70:0/2398501353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc660101ae0 0x7fc660196bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:06.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc664b83700 1 -- 172.21.15.70:0/2398501353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc660101140 msgr2=0x7fc6601966b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:06.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc664b83700 1 --2- 172.21.15.70:0/2398501353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc660101140 0x7fc6601966b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:06.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.857+0000 7fc664b83700 1 -- 172.21.15.70:0/2398501353 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc66019b6c0 con 0x7fc660107180 2024-04-01T01:32:06.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.858+0000 7fc664b83700 1 --2- 172.21.15.70:0/2398501353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc660107180 0x7fc66019af80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc65000e940 tx=0x7fc65000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:06.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.858+0000 7fc65d7fa700 1 -- 172.21.15.70:0/2398501353 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc650004a30 con 0x7fc660107180 2024-04-01T01:32:06.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.858+0000 7fc6665e6700 1 -- 172.21.15.70:0/2398501353 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc66019b9b0 con 0x7fc660107180 2024-04-01T01:32:06.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.858+0000 7fc65d7fa700 1 -- 172.21.15.70:0/2398501353 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6500054d0 con 0x7fc660107180 2024-04-01T01:32:06.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.858+0000 7fc65d7fa700 1 -- 172.21.15.70:0/2398501353 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65000ccf0 con 0x7fc660107180 2024-04-01T01:32:06.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.858+0000 7fc6665e6700 1 -- 172.21.15.70:0/2398501353 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6601a3230 con 0x7fc660107180 2024-04-01T01:32:06.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.858+0000 7fc6665e6700 1 -- 172.21.15.70:0/2398501353 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc6600629d0 con 0x7fc660107180 2024-04-01T01:32:06.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.860+0000 7fc65d7fa700 1 -- 172.21.15.70:0/2398501353 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fc650005000 con 0x7fc660107180 2024-04-01T01:32:06.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.860+0000 7fc65d7fa700 1 --2- 172.21.15.70:0/2398501353 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc6480a25c0 0x7fc6480a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:06.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.860+0000 7fc65ffff700 1 --2- 172.21.15.70:0/2398501353 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc6480a25c0 0x7fc6480a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:06.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.860+0000 7fc65d7fa700 1 -- 172.21.15.70:0/2398501353 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc6500cf000 con 0x7fc660107180 2024-04-01T01:32:06.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.860+0000 7fc65d7fa700 1 --2- 172.21.15.70:0/2398501353 >> [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716] conn(0x7fc6480ad280 0x7fc6480af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:06.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.861+0000 7fc65d7fa700 1 -- 172.21.15.70:0/2398501353 --> [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc6500090b0 con 0x7fc6480ad280 2024-04-01T01:32:06.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.861+0000 7fc65d7fa700 1 -- 172.21.15.70:0/2398501353 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fc6500cf450 con 0x7fc660107180 2024-04-01T01:32:06.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.861+0000 7fc65f7fe700 1 --2- 172.21.15.70:0/2398501353 >> [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716] conn(0x7fc6480ad280 0x7fc6480af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:06.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.861+0000 7fc65ffff700 1 --2- 172.21.15.70:0/2398501353 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc6480a25c0 0x7fc6480a4a80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc658009e60 tx=0x7fc65800b700 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:06.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.861+0000 7fc65f7fe700 1 --2- 172.21.15.70:0/2398501353 >> [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716] conn(0x7fc6480ad280 0x7fc6480af6c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:06.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.862+0000 7fc65d7fa700 1 -- 172.21.15.70:0/2398501353 <== osd.7 v2:172.21.15.81:6816/2088069716 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc6500090b0 con 0x7fc6480ad280 2024-04-01T01:32:06.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.878+0000 7fc6665e6700 1 -- 172.21.15.70:0/2398501353 --> [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc660062dd0 con 0x7fc6480ad280 2024-04-01T01:32:06.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.879+0000 7fc65d7fa700 1 -- 172.21.15.70:0/2398501353 <== osd.7 v2:172.21.15.81:6816/2088069716 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc660062dd0 con 0x7fc6480ad280 2024-04-01T01:32:06.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.879+0000 7fc6665e6700 1 -- 172.21.15.70:0/2398501353 >> [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716] conn(0x7fc6480ad280 msgr2=0x7fc6480af6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:06.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.879+0000 7fc6665e6700 1 --2- 172.21.15.70:0/2398501353 >> [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716] conn(0x7fc6480ad280 0x7fc6480af6c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:06.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.879+0000 7fc6665e6700 1 -- 172.21.15.70:0/2398501353 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc6480a25c0 msgr2=0x7fc6480a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:06.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.879+0000 7fc6665e6700 1 --2- 172.21.15.70:0/2398501353 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc6480a25c0 0x7fc6480a4a80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc658009e60 tx=0x7fc65800b700 comp rx=0 tx=0).stop 2024-04-01T01:32:06.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.879+0000 7fc6665e6700 1 -- 172.21.15.70:0/2398501353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc660107180 msgr2=0x7fc66019af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:06.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.879+0000 7fc6665e6700 1 --2- 172.21.15.70:0/2398501353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc660107180 0x7fc66019af80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc65000e940 tx=0x7fc65000ec50 comp rx=0 tx=0).stop 2024-04-01T01:32:06.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.879+0000 7fc664b83700 1 -- 172.21.15.70:0/2398501353 reap_dead start 2024-04-01T01:32:06.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.879+0000 7fc6665e6700 1 -- 172.21.15.70:0/2398501353 shutdown_connections 2024-04-01T01:32:06.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.879+0000 7fc6665e6700 1 -- 172.21.15.70:0/2398501353 >> 172.21.15.70:0/2398501353 conn(0x7fc660078070 msgr2=0x7fc660100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:06.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.879+0000 7fc6665e6700 1 -- 172.21.15.70:0/2398501353 shutdown_connections 2024-04-01T01:32:06.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:05.879+0000 7fc6665e6700 1 -- 172.21.15.70:0/2398501353 wait complete. 2024-04-01T01:32:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.288+0000 7fb3cc32b700 1 -- 172.21.15.70:0/1096694153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3c41013d0 msgr2=0x7fb3c41017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.288+0000 7fb3cc32b700 1 --2- 172.21.15.70:0/1096694153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3c41013d0 0x7fb3c41017b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4009990 tx=0x7fb3b4009ca0 comp rx=0 tx=0).stop 2024-04-01T01:32:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.288+0000 7fb3cc32b700 1 -- 172.21.15.70:0/1096694153 shutdown_connections 2024-04-01T01:32:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.288+0000 7fb3cc32b700 1 --2- 172.21.15.70:0/1096694153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3c4106450 0x7fb3c410e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.288+0000 7fb3cc32b700 1 --2- 172.21.15.70:0/1096694153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3c41013d0 0x7fb3c41017b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.288+0000 7fb3cc32b700 1 --2- 172.21.15.70:0/1096694153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3c4101d80 0x7fb3c4105d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.288+0000 7fb3cc32b700 1 -- 172.21.15.70:0/1096694153 >> 172.21.15.70:0/1096694153 conn(0x7fb3c4077180 msgr2=0x7fb3c4077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.288+0000 7fb3cc32b700 1 -- 172.21.15.70:0/1096694153 shutdown_connections 2024-04-01T01:32:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.288+0000 7fb3cc32b700 1 -- 172.21.15.70:0/1096694153 wait complete. 2024-04-01T01:32:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.288+0000 7fb3cc32b700 1 Processor -- start 2024-04-01T01:32:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.288+0000 7fb3cc32b700 1 -- start start 2024-04-01T01:32:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3cc32b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3c41013d0 0x7fb3c419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3cc32b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3c4101d80 0x7fb3c419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3cc32b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3c4106450 0x7fb3c4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3cc32b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c41112f0 con 0x7fb3c4101d80 2024-04-01T01:32:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3cc32b700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c4111170 con 0x7fb3c4106450 2024-04-01T01:32:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3cc32b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c4111470 con 0x7fb3c41013d0 2024-04-01T01:32:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3c98c6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3c4101d80 0x7fb3c419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3c98c6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3c4101d80 0x7fb3c419eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41906/0 (socket says 172.21.15.70:41906) 2024-04-01T01:32:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3c98c6700 1 -- 172.21.15.70:0/1202930369 learned_addr learned my addr 172.21.15.70:0/1202930369 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3ca0c7700 1 --2- 172.21.15.70:0/1202930369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3c41013d0 0x7fb3c419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3ca8c8700 1 --2- 172.21.15.70:0/1202930369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3c4106450 0x7fb3c4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3c98c6700 1 -- 172.21.15.70:0/1202930369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3c41013d0 msgr2=0x7fb3c419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3c98c6700 1 --2- 172.21.15.70:0/1202930369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3c41013d0 0x7fb3c419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3c98c6700 1 -- 172.21.15.70:0/1202930369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3c4106450 msgr2=0x7fb3c4198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3c98c6700 1 --2- 172.21.15.70:0/1202930369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3c4106450 0x7fb3c4198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3c98c6700 1 -- 172.21.15.70:0/1202930369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3c4199040 con 0x7fb3c4101d80 2024-04-01T01:32:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3c98c6700 1 --2- 172.21.15.70:0/1202930369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3c4101d80 0x7fb3c419eb80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb3c000ea30 tx=0x7fb3c000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3bb7fe700 1 -- 172.21.15.70:0/1202930369 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c000cbd0 con 0x7fb3c4101d80 2024-04-01T01:32:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3bb7fe700 1 -- 172.21.15.70:0/1202930369 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb3c00044d0 con 0x7fb3c4101d80 2024-04-01T01:32:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3bb7fe700 1 -- 172.21.15.70:0/1202930369 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0003db0 con 0x7fb3c4101d80 2024-04-01T01:32:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3cc32b700 1 -- 172.21.15.70:0/1202930369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3c4199330 con 0x7fb3c4101d80 2024-04-01T01:32:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.289+0000 7fb3cc32b700 1 -- 172.21.15.70:0/1202930369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3c41a55c0 con 0x7fb3c4101d80 2024-04-01T01:32:07.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.290+0000 7fb3cc32b700 1 -- 172.21.15.70:0/1202930369 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb3c406ac70 con 0x7fb3c4101d80 2024-04-01T01:32:07.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.292+0000 7fb3bb7fe700 1 -- 172.21.15.70:0/1202930369 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fb3c0003f50 con 0x7fb3c4101d80 2024-04-01T01:32:07.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.292+0000 7fb3bb7fe700 1 --2- 172.21.15.70:0/1202930369 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3b00a25c0 0x7fb3b00a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:07.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.292+0000 7fb3ca0c7700 1 --2- 172.21.15.70:0/1202930369 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3b00a25c0 0x7fb3b00a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:07.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.292+0000 7fb3bb7fe700 1 -- 172.21.15.70:0/1202930369 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb3c00cee60 con 0x7fb3c4101d80 2024-04-01T01:32:07.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.292+0000 7fb3bb7fe700 1 --2- 172.21.15.70:0/1202930369 >> [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424] conn(0x7fb3b00ad280 0x7fb3b00af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:07.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.292+0000 7fb3bb7fe700 1 -- 172.21.15.70:0/1202930369 --> [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb3c000be50 con 0x7fb3b00ad280 2024-04-01T01:32:07.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.292+0000 7fb3bb7fe700 1 -- 172.21.15.70:0/1202930369 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fb3c00cf2b0 con 0x7fb3c4101d80 2024-04-01T01:32:07.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.293+0000 7fb3ca0c7700 1 --2- 172.21.15.70:0/1202930369 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3b00a25c0 0x7fb3b00a4a80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb3b400b2e0 tx=0x7fb3b4005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:07.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.293+0000 7fb3ca8c8700 1 --2- 172.21.15.70:0/1202930369 >> [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424] conn(0x7fb3b00ad280 0x7fb3b00af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:07.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.293+0000 7fb3ca8c8700 1 --2- 172.21.15.70:0/1202930369 >> [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424] conn(0x7fb3b00ad280 0x7fb3b00af6c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:07.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.294+0000 7fb3bb7fe700 1 -- 172.21.15.70:0/1202930369 <== osd.1 v2:172.21.15.81:6800/4081087424 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb3c000be50 con 0x7fb3b00ad280 2024-04-01T01:32:07.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.310+0000 7fb3cc32b700 1 -- 172.21.15.70:0/1202930369 --> [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb3c4102850 con 0x7fb3b00ad280 2024-04-01T01:32:07.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.310+0000 7fb3bb7fe700 1 -- 172.21.15.70:0/1202930369 <== osd.1 v2:172.21.15.81:6800/4081087424 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb3c4102850 con 0x7fb3b00ad280 2024-04-01T01:32:07.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.310+0000 7fb3cc32b700 1 -- 172.21.15.70:0/1202930369 >> [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424] conn(0x7fb3b00ad280 msgr2=0x7fb3b00af6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:07.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.310+0000 7fb3cc32b700 1 --2- 172.21.15.70:0/1202930369 >> [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424] conn(0x7fb3b00ad280 0x7fb3b00af6c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:07.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.310+0000 7fb3cc32b700 1 -- 172.21.15.70:0/1202930369 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3b00a25c0 msgr2=0x7fb3b00a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:07.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.310+0000 7fb3cc32b700 1 --2- 172.21.15.70:0/1202930369 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3b00a25c0 0x7fb3b00a4a80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb3b400b2e0 tx=0x7fb3b4005960 comp rx=0 tx=0).stop 2024-04-01T01:32:07.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.310+0000 7fb3cc32b700 1 -- 172.21.15.70:0/1202930369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3c4101d80 msgr2=0x7fb3c419eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:07.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.310+0000 7fb3cc32b700 1 --2- 172.21.15.70:0/1202930369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3c4101d80 0x7fb3c419eb80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb3c000ea30 tx=0x7fb3c000ed40 comp rx=0 tx=0).stop 2024-04-01T01:32:07.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.311+0000 7fb3ca8c8700 1 -- 172.21.15.70:0/1202930369 reap_dead start 2024-04-01T01:32:07.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.311+0000 7fb3cc32b700 1 -- 172.21.15.70:0/1202930369 shutdown_connections 2024-04-01T01:32:07.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.311+0000 7fb3cc32b700 1 -- 172.21.15.70:0/1202930369 >> 172.21.15.70:0/1202930369 conn(0x7fb3c4077180 msgr2=0x7fb3c4105130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:07.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.311+0000 7fb3cc32b700 1 -- 172.21.15.70:0/1202930369 shutdown_connections 2024-04-01T01:32:07.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.311+0000 7fb3cc32b700 1 -- 172.21.15.70:0/1202930369 wait complete. 2024-04-01T01:32:07.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.497+0000 7f96ea0a3700 1 -- 172.21.15.70:0/237468024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96e4102290 msgr2=0x7f96e4102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:07.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.497+0000 7f96ea0a3700 1 --2- 172.21.15.70:0/237468024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96e4102290 0x7f96e4102710 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0099c0 tx=0x7f96cc009cd0 comp rx=0 tx=0).stop 2024-04-01T01:32:07.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.497+0000 7f96ea0a3700 1 -- 172.21.15.70:0/237468024 shutdown_connections 2024-04-01T01:32:07.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.497+0000 7f96ea0a3700 1 --2- 172.21.15.70:0/237468024 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96e4102c50 0x7f96e410f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:07.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.497+0000 7f96ea0a3700 1 --2- 172.21.15.70:0/237468024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96e4108270 0x7f96e4108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:07.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.497+0000 7f96ea0a3700 1 --2- 172.21.15.70:0/237468024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96e4102290 0x7f96e4102710 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:07.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.497+0000 7f96ea0a3700 1 -- 172.21.15.70:0/237468024 >> 172.21.15.70:0/237468024 conn(0x7f96e4079370 msgr2=0x7f96e4079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:07.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.497+0000 7f96ea0a3700 1 -- 172.21.15.70:0/237468024 shutdown_connections 2024-04-01T01:32:07.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.497+0000 7f96ea0a3700 1 -- 172.21.15.70:0/237468024 wait complete. 2024-04-01T01:32:07.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.497+0000 7f96ea0a3700 1 Processor -- start 2024-04-01T01:32:07.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.497+0000 7f96ea0a3700 1 -- start start 2024-04-01T01:32:07.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96ea0a3700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96e4102290 0x7f96e419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:07.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96ea0a3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96e4102c50 0x7f96e419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:07.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96ea0a3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96e4108270 0x7f96e41986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:07.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96ea0a3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f96e41123a0 con 0x7f96e4108270 2024-04-01T01:32:07.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96ea0a3700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f96e4112220 con 0x7f96e4102290 2024-04-01T01:32:07.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96ea0a3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f96e4112520 con 0x7f96e4102c50 2024-04-01T01:32:07.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96e3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96e4108270 0x7f96e41986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:07.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96e3fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96e4108270 0x7f96e41986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41926/0 (socket says 172.21.15.70:41926) 2024-04-01T01:32:07.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96daffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96e4102c50 0x7f96e419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:07.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96e37fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96e4102290 0x7f96e419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:07.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96e3fff700 1 -- 172.21.15.70:0/698579789 learned_addr learned my addr 172.21.15.70:0/698579789 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:07.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96e3fff700 1 -- 172.21.15.70:0/698579789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96e4102c50 msgr2=0x7f96e419eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:07.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96e3fff700 1 --2- 172.21.15.70:0/698579789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96e4102c50 0x7f96e419eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:07.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96e3fff700 1 -- 172.21.15.70:0/698579789 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96e4102290 msgr2=0x7f96e419e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:07.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96e3fff700 1 --2- 172.21.15.70:0/698579789 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96e4102290 0x7f96e419e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:07.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96e3fff700 1 -- 172.21.15.70:0/698579789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e4198fa0 con 0x7f96e4108270 2024-04-01T01:32:07.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96daffd700 1 --2- 172.21.15.70:0/698579789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96e4102c50 0x7f96e419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:07.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96e3fff700 1 --2- 172.21.15.70:0/698579789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96e4108270 0x7f96e41986c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00c370 tx=0x7f96dc00c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:07.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96e17fa700 1 -- 172.21.15.70:0/698579789 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96dc0092e0 con 0x7f96e4108270 2024-04-01T01:32:07.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.499+0000 7f96ea0a3700 1 -- 172.21.15.70:0/698579789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96e4199230 con 0x7f96e4108270 2024-04-01T01:32:07.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.498+0000 7f96e17fa700 1 -- 172.21.15.70:0/698579789 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96dc003a00 con 0x7f96e4108270 2024-04-01T01:32:07.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.499+0000 7f96e17fa700 1 -- 172.21.15.70:0/698579789 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96dc0046f0 con 0x7f96e4108270 2024-04-01T01:32:07.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.499+0000 7f96ea0a3700 1 -- 172.21.15.70:0/698579789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96e41a53a0 con 0x7f96e4108270 2024-04-01T01:32:07.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.499+0000 7f96ea0a3700 1 -- 172.21.15.70:0/698579789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f96e40629d0 con 0x7f96e4108270 2024-04-01T01:32:07.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.501+0000 7f96e17fa700 1 -- 172.21.15.70:0/698579789 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f96dc01a020 con 0x7f96e4108270 2024-04-01T01:32:07.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.501+0000 7f96e17fa700 1 --2- 172.21.15.70:0/698579789 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96d00a25c0 0x7f96d00a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:07.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.501+0000 7f96e37fe700 1 --2- 172.21.15.70:0/698579789 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96d00a25c0 0x7f96d00a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:07.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.501+0000 7f96e17fa700 1 -- 172.21.15.70:0/698579789 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f96dc0cee60 con 0x7f96e4108270 2024-04-01T01:32:07.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.501+0000 7f96e17fa700 1 --2- 172.21.15.70:0/698579789 >> [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673] conn(0x7f96d00ad280 0x7f96d00af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:07.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.501+0000 7f96e17fa700 1 -- 172.21.15.70:0/698579789 --> [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f96dc024e50 con 0x7f96d00ad280 2024-04-01T01:32:07.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.501+0000 7f96e17fa700 1 -- 172.21.15.70:0/698579789 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f96dc0cf2b0 con 0x7f96e4108270 2024-04-01T01:32:07.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.502+0000 7f96daffd700 1 --2- 172.21.15.70:0/698579789 >> [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673] conn(0x7f96d00ad280 0x7f96d00af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:07.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.502+0000 7f96e37fe700 1 --2- 172.21.15.70:0/698579789 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96d00a25c0 0x7f96d00a4a80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f96d4009680 tx=0x7f96d4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:07.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.502+0000 7f96daffd700 1 --2- 172.21.15.70:0/698579789 >> [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673] conn(0x7f96d00ad280 0x7f96d00af6c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:07.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.503+0000 7f96e17fa700 1 -- 172.21.15.70:0/698579789 <== osd.6 v2:172.21.15.70:6818/2689871673 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96dc024e50 con 0x7f96d00ad280 2024-04-01T01:32:07.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.519+0000 7f96ea0a3700 1 -- 172.21.15.70:0/698579789 --> [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f96e4062e50 con 0x7f96d00ad280 2024-04-01T01:32:07.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.519+0000 7f96e17fa700 1 -- 172.21.15.70:0/698579789 <== osd.6 v2:172.21.15.70:6818/2689871673 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f96e4062e50 con 0x7f96d00ad280 2024-04-01T01:32:07.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.520+0000 7f96ea0a3700 1 -- 172.21.15.70:0/698579789 >> [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673] conn(0x7f96d00ad280 msgr2=0x7f96d00af6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:07.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.520+0000 7f96ea0a3700 1 --2- 172.21.15.70:0/698579789 >> [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673] conn(0x7f96d00ad280 0x7f96d00af6c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:07.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.520+0000 7f96ea0a3700 1 -- 172.21.15.70:0/698579789 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96d00a25c0 msgr2=0x7f96d00a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:07.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.520+0000 7f96ea0a3700 1 --2- 172.21.15.70:0/698579789 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96d00a25c0 0x7f96d00a4a80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f96d4009680 tx=0x7f96d4009260 comp rx=0 tx=0).stop 2024-04-01T01:32:07.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.520+0000 7f96ea0a3700 1 -- 172.21.15.70:0/698579789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96e4108270 msgr2=0x7f96e41986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:07.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.520+0000 7f96ea0a3700 1 --2- 172.21.15.70:0/698579789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96e4108270 0x7f96e41986c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f96dc00c370 tx=0x7f96dc00c680 comp rx=0 tx=0).stop 2024-04-01T01:32:07.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.520+0000 7f96e3fff700 1 -- 172.21.15.70:0/698579789 reap_dead start 2024-04-01T01:32:07.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.520+0000 7f96ea0a3700 1 -- 172.21.15.70:0/698579789 shutdown_connections 2024-04-01T01:32:07.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.520+0000 7f96ea0a3700 1 -- 172.21.15.70:0/698579789 >> 172.21.15.70:0/698579789 conn(0x7f96e4079370 msgr2=0x7f96e41010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:07.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.520+0000 7f96ea0a3700 1 -- 172.21.15.70:0/698579789 shutdown_connections 2024-04-01T01:32:07.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:07.520+0000 7f96ea0a3700 1 -- 172.21.15.70:0/698579789 wait complete. 2024-04-01T01:32:07.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:07 smithi070 ceph-mon[32179]: pgmap v313: 129 pgs: 129 active+clean; 644 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 426 B/s wr, 12 op/s 2024-04-01T01:32:07.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:07 smithi189 ceph-mon[35753]: pgmap v313: 129 pgs: 129 active+clean; 644 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 426 B/s wr, 12 op/s 2024-04-01T01:32:08.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:07 smithi081 ceph-mon[35142]: pgmap v313: 129 pgs: 129 active+clean; 644 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 426 B/s wr, 12 op/s 2024-04-01T01:32:08.474 INFO:teuthology.orchestra.run.smithi070.stdout:219043332140 2024-04-01T01:32:08.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-04-01T01:32:08.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.136+0000 7f92dee26700 1 -- 172.21.15.70:0/3145494583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92d806a890 msgr2=0x7f92d810f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:08.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.136+0000 7f92dee26700 1 --2- 172.21.15.70:0/3145494583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92d806a890 0x7f92d810f0d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f92d000b0d0 tx=0x7f92d000b3e0 comp rx=0 tx=0).stop 2024-04-01T01:32:08.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.136+0000 7f92dee26700 1 -- 172.21.15.70:0/3145494583 shutdown_connections 2024-04-01T01:32:08.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.136+0000 7f92dee26700 1 --2- 172.21.15.70:0/3145494583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92d806a890 0x7f92d810f0d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:08.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.136+0000 7f92dee26700 1 --2- 172.21.15.70:0/3145494583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92d8108290 0x7f92d8108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:08.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.136+0000 7f92dee26700 1 --2- 172.21.15.70:0/3145494583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92d8069ef0 0x7f92d806a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:08.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.136+0000 7f92dee26700 1 -- 172.21.15.70:0/3145494583 >> 172.21.15.70:0/3145494583 conn(0x7f92d8078070 msgr2=0x7f92d8078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:08.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.136+0000 7f92dee26700 1 -- 172.21.15.70:0/3145494583 shutdown_connections 2024-04-01T01:32:08.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.136+0000 7f92dee26700 1 -- 172.21.15.70:0/3145494583 wait complete. 2024-04-01T01:32:08.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.137+0000 7f92dee26700 1 Processor -- start 2024-04-01T01:32:08.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.137+0000 7f92dee26700 1 -- start start 2024-04-01T01:32:08.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.137+0000 7f92dee26700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92d8069ef0 0x7f92d8101210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:08.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.137+0000 7f92dee26700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92d806a890 0x7f92d8101750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:08.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.137+0000 7f92dee26700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92d8108290 0x7f92d8101c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:08.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.137+0000 7f92dee26700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f92d81123c0 con 0x7f92d806a890 2024-04-01T01:32:08.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.137+0000 7f92dee26700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f92d8112240 con 0x7f92d8069ef0 2024-04-01T01:32:08.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.137+0000 7f92dee26700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f92d8112540 con 0x7f92d8108290 2024-04-01T01:32:08.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92d7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92d806a890 0x7f92d8101750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:08.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92d7fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92d806a890 0x7f92d8101750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41946/0 (socket says 172.21.15.70:41946) 2024-04-01T01:32:08.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92dcbc2700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92d8069ef0 0x7f92d8101210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:08.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92d7fff700 1 -- 172.21.15.70:0/2827838155 learned_addr learned my addr 172.21.15.70:0/2827838155 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:08.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92dd3c3700 1 --2- 172.21.15.70:0/2827838155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92d8108290 0x7f92d8101c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:08.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92d7fff700 1 -- 172.21.15.70:0/2827838155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92d8108290 msgr2=0x7f92d8101c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:08.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92d7fff700 1 --2- 172.21.15.70:0/2827838155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92d8108290 0x7f92d8101c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92d7fff700 1 -- 172.21.15.70:0/2827838155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92d8069ef0 msgr2=0x7f92d8101210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92d7fff700 1 --2- 172.21.15.70:0/2827838155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92d8069ef0 0x7f92d8101210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92d7fff700 1 -- 172.21.15.70:0/2827838155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92d81a6d90 con 0x7f92d806a890 2024-04-01T01:32:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92dcbc2700 1 --2- 172.21.15.70:0/2827838155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92d8069ef0 0x7f92d8101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92d7fff700 1 --2- 172.21.15.70:0/2827838155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92d806a890 0x7f92d8101750 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f92c400e9a0 tx=0x7f92c400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92d5ffb700 1 -- 172.21.15.70:0/2827838155 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c400ca10 con 0x7f92d806a890 2024-04-01T01:32:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92d5ffb700 1 -- 172.21.15.70:0/2827838155 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f92c400cbb0 con 0x7f92d806a890 2024-04-01T01:32:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92dee26700 1 -- 172.21.15.70:0/2827838155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92d81a7080 con 0x7f92d806a890 2024-04-01T01:32:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92d5ffb700 1 -- 172.21.15.70:0/2827838155 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c4018680 con 0x7f92d806a890 2024-04-01T01:32:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.138+0000 7f92dee26700 1 -- 172.21.15.70:0/2827838155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92d81a75c0 con 0x7f92d806a890 2024-04-01T01:32:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.139+0000 7f92dee26700 1 -- 172.21.15.70:0/2827838155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f92d8062950 con 0x7f92d806a890 2024-04-01T01:32:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.140+0000 7f92d5ffb700 1 -- 172.21.15.70:0/2827838155 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f92c4010430 con 0x7f92d806a890 2024-04-01T01:32:08.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.141+0000 7f92d5ffb700 1 --2- 172.21.15.70:0/2827838155 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f92c80a25c0 0x7f92c80a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.141+0000 7f92dcbc2700 1 --2- 172.21.15.70:0/2827838155 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f92c80a25c0 0x7f92c80a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.141+0000 7f92d5ffb700 1 -- 172.21.15.70:0/2827838155 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f92c40cf110 con 0x7f92d806a890 2024-04-01T01:32:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.141+0000 7f92d5ffb700 1 --2- 172.21.15.70:0/2827838155 >> [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377] conn(0x7f92c80ad280 0x7f92c80af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.141+0000 7f92d5ffb700 1 -- 172.21.15.70:0/2827838155 --> [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f92c4001d30 con 0x7f92c80ad280 2024-04-01T01:32:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.141+0000 7f92d5ffb700 1 -- 172.21.15.70:0/2827838155 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f92c40cf560 con 0x7f92d806a890 2024-04-01T01:32:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.141+0000 7f92dcbc2700 1 --2- 172.21.15.70:0/2827838155 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f92c80a25c0 0x7f92c80a4a80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f92cc009fd0 tx=0x7f92cc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.141+0000 7f92dd3c3700 1 --2- 172.21.15.70:0/2827838155 >> [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377] conn(0x7f92c80ad280 0x7f92c80af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.142+0000 7f92dd3c3700 1 --2- 172.21.15.70:0/2827838155 >> [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377] conn(0x7f92c80ad280 0x7f92c80af6c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.143+0000 7f92d5ffb700 1 -- 172.21.15.70:0/2827838155 <== osd.5 v2:172.21.15.189:6808/2305146377 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f92c4001d30 con 0x7f92c80ad280 2024-04-01T01:32:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.158+0000 7f92dee26700 1 -- 172.21.15.70:0/2827838155 --> [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f92d8062dd0 con 0x7f92c80ad280 2024-04-01T01:32:08.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.159+0000 7f92d5ffb700 1 -- 172.21.15.70:0/2827838155 <== osd.5 v2:172.21.15.189:6808/2305146377 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f92d8062dd0 con 0x7f92c80ad280 2024-04-01T01:32:08.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.159+0000 7f92dee26700 1 -- 172.21.15.70:0/2827838155 >> [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377] conn(0x7f92c80ad280 msgr2=0x7f92c80af6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:08.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.159+0000 7f92dee26700 1 --2- 172.21.15.70:0/2827838155 >> [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377] conn(0x7f92c80ad280 0x7f92c80af6c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:08.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.159+0000 7f92dee26700 1 -- 172.21.15.70:0/2827838155 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f92c80a25c0 msgr2=0x7f92c80a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:08.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.159+0000 7f92dee26700 1 --2- 172.21.15.70:0/2827838155 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f92c80a25c0 0x7f92c80a4a80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f92cc009fd0 tx=0x7f92cc009260 comp rx=0 tx=0).stop 2024-04-01T01:32:08.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.159+0000 7f92dee26700 1 -- 172.21.15.70:0/2827838155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92d806a890 msgr2=0x7f92d8101750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:08.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.159+0000 7f92dee26700 1 --2- 172.21.15.70:0/2827838155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92d806a890 0x7f92d8101750 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f92c400e9a0 tx=0x7f92c400ecb0 comp rx=0 tx=0).stop 2024-04-01T01:32:08.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.159+0000 7f92dd3c3700 1 -- 172.21.15.70:0/2827838155 reap_dead start 2024-04-01T01:32:08.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.159+0000 7f92dee26700 1 -- 172.21.15.70:0/2827838155 shutdown_connections 2024-04-01T01:32:08.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.159+0000 7f92dee26700 1 -- 172.21.15.70:0/2827838155 >> 172.21.15.70:0/2827838155 conn(0x7f92d8078070 msgr2=0x7f92d80ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:08.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.159+0000 7f92dee26700 1 -- 172.21.15.70:0/2827838155 shutdown_connections 2024-04-01T01:32:08.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:08.159+0000 7f92dee26700 1 -- 172.21.15.70:0/2827838155 wait complete. 2024-04-01T01:32:09.923 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:09 smithi070 ceph-mon[32179]: pgmap v314: 129 pgs: 129 active+clean; 644 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-01T01:32:09.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:09 smithi189 ceph-mon[35753]: pgmap v314: 129 pgs: 129 active+clean; 644 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-01T01:32:10.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:09 smithi081 ceph-mon[35142]: pgmap v314: 129 pgs: 129 active+clean; 644 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-01T01:32:10.376 INFO:teuthology.orchestra.run.smithi070.stdout:197568495665 2024-04-01T01:32:10.377 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-04-01T01:32:11.075 INFO:teuthology.orchestra.run.smithi070.stdout:55834574920 2024-04-01T01:32:11.075 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-04-01T01:32:11.996 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:11 smithi070 ceph-mon[32179]: pgmap v315: 129 pgs: 129 active+clean; 644 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-01T01:32:11.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:11 smithi189 ceph-mon[35753]: pgmap v315: 129 pgs: 129 active+clean; 644 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-01T01:32:12.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.002+0000 7f13e89da700 1 -- 172.21.15.70:0/641107563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13e41064c0 msgr2=0x7f13e410e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:12.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.002+0000 7f13e89da700 1 --2- 172.21.15.70:0/641107563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13e41064c0 0x7f13e410e090 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00b0d0 tx=0x7f13dc00b3e0 comp rx=0 tx=0).stop 2024-04-01T01:32:12.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.002+0000 7f13e89da700 1 -- 172.21.15.70:0/641107563 shutdown_connections 2024-04-01T01:32:12.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.002+0000 7f13e89da700 1 --2- 172.21.15.70:0/641107563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13e41064c0 0x7f13e410e090 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:12.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.002+0000 7f13e89da700 1 --2- 172.21.15.70:0/641107563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13e4101ba0 0x7f13e4101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:12.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.002+0000 7f13e89da700 1 --2- 172.21.15.70:0/641107563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13e4102550 0x7f13e4105de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:12.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.002+0000 7f13e89da700 1 -- 172.21.15.70:0/641107563 >> 172.21.15.70:0/641107563 conn(0x7f13e4077740 msgr2=0x7f13e40feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:12.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.002+0000 7f13e89da700 1 -- 172.21.15.70:0/641107563 shutdown_connections 2024-04-01T01:32:12.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.002+0000 7f13e89da700 1 -- 172.21.15.70:0/641107563 wait complete. 2024-04-01T01:32:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e89da700 1 Processor -- start 2024-04-01T01:32:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e89da700 1 -- start start 2024-04-01T01:32:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e89da700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13e4101ba0 0x7f13e419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e89da700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13e4102550 0x7f13e419ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e89da700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13e41064c0 0x7f13e4198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e89da700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f13e4111360 con 0x7f13e4101ba0 2024-04-01T01:32:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e89da700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f13e41111e0 con 0x7f13e4102550 2024-04-01T01:32:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e89da700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f13e41114e0 con 0x7f13e41064c0 2024-04-01T01:32:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13e4101ba0 0x7f13e419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13e4101ba0 0x7f13e419e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41966/0 (socket says 172.21.15.70:41966) 2024-04-01T01:32:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e259c700 1 -- 172.21.15.70:0/2509668864 learned_addr learned my addr 172.21.15.70:0/2509668864 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e1d9b700 1 --2- 172.21.15.70:0/2509668864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13e4102550 0x7f13e419ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e2d9d700 1 --2- 172.21.15.70:0/2509668864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13e41064c0 0x7f13e4198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e1d9b700 1 -- 172.21.15.70:0/2509668864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13e41064c0 msgr2=0x7f13e4198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e1d9b700 1 --2- 172.21.15.70:0/2509668864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13e41064c0 0x7f13e4198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e1d9b700 1 -- 172.21.15.70:0/2509668864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13e4101ba0 msgr2=0x7f13e419e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e1d9b700 1 --2- 172.21.15.70:0/2509668864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13e4101ba0 0x7f13e419e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e1d9b700 1 -- 172.21.15.70:0/2509668864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e4199070 con 0x7f13e4102550 2024-04-01T01:32:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.003+0000 7f13e259c700 1 --2- 172.21.15.70:0/2509668864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13e4101ba0 0x7f13e419e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.004+0000 7f13e1d9b700 1 --2- 172.21.15.70:0/2509668864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13e4102550 0x7f13e419ebb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f13d800e9c0 tx=0x7f13d800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.004+0000 7f13d37fe700 1 -- 172.21.15.70:0/2509668864 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d800c9c0 con 0x7f13e4102550 2024-04-01T01:32:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.004+0000 7f13d37fe700 1 -- 172.21.15.70:0/2509668864 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13d800cb60 con 0x7f13e4102550 2024-04-01T01:32:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.004+0000 7f13d37fe700 1 -- 172.21.15.70:0/2509668864 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d8018680 con 0x7f13e4102550 2024-04-01T01:32:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.004+0000 7f13e89da700 1 -- 172.21.15.70:0/2509668864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e4199360 con 0x7f13e4102550 2024-04-01T01:32:12.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.004+0000 7f13e89da700 1 -- 172.21.15.70:0/2509668864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e41a55f0 con 0x7f13e4102550 2024-04-01T01:32:12.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.005+0000 7f13e89da700 1 -- 172.21.15.70:0/2509668864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f13e4105630 con 0x7f13e4102550 2024-04-01T01:32:12.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.006+0000 7f13d37fe700 1 -- 172.21.15.70:0/2509668864 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f13d8010430 con 0x7f13e4102550 2024-04-01T01:32:12.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.006+0000 7f13d37fe700 1 --2- 172.21.15.70:0/2509668864 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f13cc0a25c0 0x7f13cc0a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:12.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.007+0000 7f13e259c700 1 --2- 172.21.15.70:0/2509668864 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f13cc0a25c0 0x7f13cc0a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:12.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.007+0000 7f13d37fe700 1 -- 172.21.15.70:0/2509668864 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f13d80cf6c0 con 0x7f13e4102550 2024-04-01T01:32:12.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.007+0000 7f13d37fe700 1 --2- 172.21.15.70:0/2509668864 >> [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617] conn(0x7f13cc0ad280 0x7f13cc0af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:12.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.007+0000 7f13d37fe700 1 -- 172.21.15.70:0/2509668864 --> [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13cc0afd90 con 0x7f13cc0ad280 2024-04-01T01:32:12.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.007+0000 7f13d37fe700 1 -- 172.21.15.70:0/2509668864 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f13d80cfa70 con 0x7f13e4102550 2024-04-01T01:32:12.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.007+0000 7f13e2d9d700 1 --2- 172.21.15.70:0/2509668864 >> [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617] conn(0x7f13cc0ad280 0x7f13cc0af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:12.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.007+0000 7f13e259c700 1 --2- 172.21.15.70:0/2509668864 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f13cc0a25c0 0x7f13cc0a4a80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f13d4011550 tx=0x7f13d4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:12.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.007+0000 7f13e2d9d700 1 --2- 172.21.15.70:0/2509668864 >> [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617] conn(0x7f13cc0ad280 0x7f13cc0af6c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:12.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.008+0000 7f13d37fe700 1 -- 172.21.15.70:0/2509668864 <== osd.3 v2:172.21.15.70:6810/627924617 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f13cc0afd90 con 0x7f13cc0ad280 2024-04-01T01:32:12.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.025+0000 7f13e89da700 1 -- 172.21.15.70:0/2509668864 --> [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f13e4102950 con 0x7f13cc0ad280 2024-04-01T01:32:12.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.025+0000 7f13d37fe700 1 -- 172.21.15.70:0/2509668864 <== osd.3 v2:172.21.15.70:6810/627924617 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f13e4102950 con 0x7f13cc0ad280 2024-04-01T01:32:12.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.026+0000 7f13e89da700 1 -- 172.21.15.70:0/2509668864 >> [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617] conn(0x7f13cc0ad280 msgr2=0x7f13cc0af6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:12.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.026+0000 7f13e89da700 1 --2- 172.21.15.70:0/2509668864 >> [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617] conn(0x7f13cc0ad280 0x7f13cc0af6c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:12.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.026+0000 7f13e89da700 1 -- 172.21.15.70:0/2509668864 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f13cc0a25c0 msgr2=0x7f13cc0a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:12.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.026+0000 7f13e89da700 1 --2- 172.21.15.70:0/2509668864 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f13cc0a25c0 0x7f13cc0a4a80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f13d4011550 tx=0x7f13d4009260 comp rx=0 tx=0).stop 2024-04-01T01:32:12.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.026+0000 7f13e89da700 1 -- 172.21.15.70:0/2509668864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13e4102550 msgr2=0x7f13e419ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:12.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.026+0000 7f13e89da700 1 --2- 172.21.15.70:0/2509668864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13e4102550 0x7f13e419ebb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f13d800e9c0 tx=0x7f13d800ecd0 comp rx=0 tx=0).stop 2024-04-01T01:32:12.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.026+0000 7f13e2d9d700 1 -- 172.21.15.70:0/2509668864 reap_dead start 2024-04-01T01:32:12.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.026+0000 7f13e89da700 1 -- 172.21.15.70:0/2509668864 shutdown_connections 2024-04-01T01:32:12.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.026+0000 7f13e89da700 1 -- 172.21.15.70:0/2509668864 >> 172.21.15.70:0/2509668864 conn(0x7f13e4077740 msgr2=0x7f13e4100ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:12.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.026+0000 7f13e89da700 1 -- 172.21.15.70:0/2509668864 shutdown_connections 2024-04-01T01:32:12.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:12.026+0000 7f13e89da700 1 -- 172.21.15.70:0/2509668864 wait complete. 2024-04-01T01:32:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:11 smithi081 ceph-mon[35142]: pgmap v315: 129 pgs: 129 active+clean; 644 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-01T01:32:13.271 INFO:teuthology.orchestra.run.smithi070.stdout:107374182464 2024-04-01T01:32:13.272 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-04-01T01:32:13.929 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:13 smithi070 ceph-mon[32179]: pgmap v316: 129 pgs: 129 active+clean; 646 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-01T01:32:13.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:13 smithi189 ceph-mon[35753]: pgmap v316: 129 pgs: 129 active+clean; 646 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-01T01:32:14.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:13 smithi081 ceph-mon[35142]: pgmap v316: 129 pgs: 129 active+clean; 646 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-01T01:32:14.410 INFO:teuthology.orchestra.run.smithi070.stdout:163208757302 2024-04-01T01:32:14.410 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-04-01T01:32:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.200+0000 7f88a34ec700 1 -- 172.21.15.70:0/3712920020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f889c102550 msgr2=0x7f889c105de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.200+0000 7f88a34ec700 1 --2- 172.21.15.70:0/3712920020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f889c102550 0x7f889c105de0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f88980099c0 tx=0x7f8898009cd0 comp rx=0 tx=0).stop 2024-04-01T01:32:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.201+0000 7f88a34ec700 1 -- 172.21.15.70:0/3712920020 shutdown_connections 2024-04-01T01:32:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.201+0000 7f88a34ec700 1 --2- 172.21.15.70:0/3712920020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f889c1064c0 0x7f889c10e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.201+0000 7f88a34ec700 1 --2- 172.21.15.70:0/3712920020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f889c101ba0 0x7f889c101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.201+0000 7f88a34ec700 1 --2- 172.21.15.70:0/3712920020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f889c102550 0x7f889c105de0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:15.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.201+0000 7f88a34ec700 1 -- 172.21.15.70:0/3712920020 >> 172.21.15.70:0/3712920020 conn(0x7f889c077740 msgr2=0x7f889c0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.201+0000 7f88a34ec700 1 -- 172.21.15.70:0/3712920020 shutdown_connections 2024-04-01T01:32:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.201+0000 7f88a34ec700 1 -- 172.21.15.70:0/3712920020 wait complete. 2024-04-01T01:32:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.201+0000 7f88a34ec700 1 Processor -- start 2024-04-01T01:32:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.201+0000 7f88a34ec700 1 -- start start 2024-04-01T01:32:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a34ec700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f889c101ba0 0x7f889c19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a34ec700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f889c102550 0x7f889c19ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a34ec700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f889c1064c0 0x7f889c198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a34ec700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f889c111350 con 0x7f889c1064c0 2024-04-01T01:32:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a34ec700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f889c1111d0 con 0x7f889c102550 2024-04-01T01:32:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a34ec700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f889c1114d0 con 0x7f889c101ba0 2024-04-01T01:32:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a1a89700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f889c1064c0 0x7f889c198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a1a89700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f889c1064c0 0x7f889c198850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41996/0 (socket says 172.21.15.70:41996) 2024-04-01T01:32:15.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a1288700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f889c101ba0 0x7f889c19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a0a87700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f889c102550 0x7f889c19ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a1a89700 1 -- 172.21.15.70:0/945829100 learned_addr learned my addr 172.21.15.70:0/945829100 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a1a89700 1 -- 172.21.15.70:0/945829100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f889c101ba0 msgr2=0x7f889c19e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a1a89700 1 --2- 172.21.15.70:0/945829100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f889c101ba0 0x7f889c19e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a1a89700 1 -- 172.21.15.70:0/945829100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f889c102550 msgr2=0x7f889c19ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a1a89700 1 --2- 172.21.15.70:0/945829100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f889c102550 0x7f889c19ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:15.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a1a89700 1 -- 172.21.15.70:0/945829100 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889c199030 con 0x7f889c1064c0 2024-04-01T01:32:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a1a89700 1 --2- 172.21.15.70:0/945829100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f889c1064c0 0x7f889c198850 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f889400ef30 tx=0x7f889400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88927fc700 1 -- 172.21.15.70:0/945829100 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88940092e0 con 0x7f889c1064c0 2024-04-01T01:32:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a34ec700 1 -- 172.21.15.70:0/945829100 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f889c1992c0 con 0x7f889c1064c0 2024-04-01T01:32:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88927fc700 1 -- 172.21.15.70:0/945829100 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8894004bb0 con 0x7f889c1064c0 2024-04-01T01:32:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88927fc700 1 -- 172.21.15.70:0/945829100 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8894004020 con 0x7f889c1064c0 2024-04-01T01:32:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.202+0000 7f88a34ec700 1 -- 172.21.15.70:0/945829100 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f889c1a54e0 con 0x7f889c1064c0 2024-04-01T01:32:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.203+0000 7f88a34ec700 1 -- 172.21.15.70:0/945829100 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f889c100b00 con 0x7f889c1064c0 2024-04-01T01:32:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.204+0000 7f88927fc700 1 -- 172.21.15.70:0/945829100 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f889401b030 con 0x7f889c1064c0 2024-04-01T01:32:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.205+0000 7f88927fc700 1 --2- 172.21.15.70:0/945829100 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f88880a2640 0x7f88880a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.205+0000 7f88a1288700 1 --2- 172.21.15.70:0/945829100 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f88880a2640 0x7f88880a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:15.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.205+0000 7f88927fc700 1 -- 172.21.15.70:0/945829100 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f88940cedc0 con 0x7f889c1064c0 2024-04-01T01:32:15.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.205+0000 7f88927fc700 1 --2- 172.21.15.70:0/945829100 >> [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510] conn(0x7f88880ad300 0x7f88880af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:15.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.205+0000 7f88927fc700 1 -- 172.21.15.70:0/945829100 --> [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f889401fe50 con 0x7f88880ad300 2024-04-01T01:32:15.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.205+0000 7f88927fc700 1 -- 172.21.15.70:0/945829100 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f88940cf210 con 0x7f889c1064c0 2024-04-01T01:32:15.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.205+0000 7f88a1288700 1 --2- 172.21.15.70:0/945829100 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f88880a2640 0x7f88880a4b00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f888c009c60 tx=0x7f888c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:15.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.205+0000 7f88a0a87700 1 --2- 172.21.15.70:0/945829100 >> [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510] conn(0x7f88880ad300 0x7f88880af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:15.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.206+0000 7f88a0a87700 1 --2- 172.21.15.70:0/945829100 >> [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510] conn(0x7f88880ad300 0x7f88880af740 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:15.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.207+0000 7f88927fc700 1 -- 172.21.15.70:0/945829100 <== osd.11 v2:172.21.15.189:6824/22739510 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f889401fe50 con 0x7f88880ad300 2024-04-01T01:32:15.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.223+0000 7f88a34ec700 1 -- 172.21.15.70:0/945829100 --> [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f889c068f00 con 0x7f88880ad300 2024-04-01T01:32:15.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.223+0000 7f88927fc700 1 -- 172.21.15.70:0/945829100 <== osd.11 v2:172.21.15.189:6824/22739510 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f889c068f00 con 0x7f88880ad300 2024-04-01T01:32:15.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.224+0000 7f88a34ec700 1 -- 172.21.15.70:0/945829100 >> [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510] conn(0x7f88880ad300 msgr2=0x7f88880af740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:15.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.224+0000 7f88a34ec700 1 --2- 172.21.15.70:0/945829100 >> [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510] conn(0x7f88880ad300 0x7f88880af740 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:15.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.224+0000 7f88a34ec700 1 -- 172.21.15.70:0/945829100 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f88880a2640 msgr2=0x7f88880a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:15.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.224+0000 7f88a34ec700 1 --2- 172.21.15.70:0/945829100 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f88880a2640 0x7f88880a4b00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f888c009c60 tx=0x7f888c009260 comp rx=0 tx=0).stop 2024-04-01T01:32:15.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.224+0000 7f88a34ec700 1 -- 172.21.15.70:0/945829100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f889c1064c0 msgr2=0x7f889c198850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:15.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.224+0000 7f88a34ec700 1 --2- 172.21.15.70:0/945829100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f889c1064c0 0x7f889c198850 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f889400ef30 tx=0x7f889400c3a0 comp rx=0 tx=0).stop 2024-04-01T01:32:15.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.224+0000 7f88a1a89700 1 -- 172.21.15.70:0/945829100 reap_dead start 2024-04-01T01:32:15.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.224+0000 7f88a34ec700 1 -- 172.21.15.70:0/945829100 shutdown_connections 2024-04-01T01:32:15.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.224+0000 7f88a34ec700 1 -- 172.21.15.70:0/945829100 >> 172.21.15.70:0/945829100 conn(0x7f889c077740 msgr2=0x7f889c104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:15.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.224+0000 7f88a34ec700 1 -- 172.21.15.70:0/945829100 shutdown_connections 2024-04-01T01:32:15.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.224+0000 7f88a34ec700 1 -- 172.21.15.70:0/945829100 wait complete. 2024-04-01T01:32:15.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.474+0000 7f2bf1ac7700 1 -- 172.21.15.70:0/3735943911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bec1024f0 msgr2=0x7f2bec1028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:15.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.474+0000 7f2bf1ac7700 1 --2- 172.21.15.70:0/3735943911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bec1024f0 0x7f2bec1028d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2be00099c0 tx=0x7f2be0009cd0 comp rx=0 tx=0).stop 2024-04-01T01:32:15.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.474+0000 7f2bf1ac7700 1 -- 172.21.15.70:0/3735943911 shutdown_connections 2024-04-01T01:32:15.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.474+0000 7f2bf1ac7700 1 --2- 172.21.15.70:0/3735943911 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bec1075c0 0x7f2bec10f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:15.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.474+0000 7f2bf1ac7700 1 --2- 172.21.15.70:0/3735943911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bec1024f0 0x7f2bec1028d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:15.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.474+0000 7f2bf1ac7700 1 --2- 172.21.15.70:0/3735943911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bec102ea0 0x7f2bec106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:15.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.474+0000 7f2bf1ac7700 1 -- 172.21.15.70:0/3735943911 >> 172.21.15.70:0/3735943911 conn(0x7f2bec0793a0 msgr2=0x7f2bec0797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:15.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.474+0000 7f2bf1ac7700 1 -- 172.21.15.70:0/3735943911 shutdown_connections 2024-04-01T01:32:15.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.474+0000 7f2bf1ac7700 1 -- 172.21.15.70:0/3735943911 wait complete. 2024-04-01T01:32:15.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.475+0000 7f2bf1ac7700 1 Processor -- start 2024-04-01T01:32:15.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.475+0000 7f2bf1ac7700 1 -- start start 2024-04-01T01:32:15.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.475+0000 7f2bf1ac7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bec1024f0 0x7f2bec19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:15.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.475+0000 7f2bf1ac7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bec102ea0 0x7f2bec19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:15.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.475+0000 7f2bf1ac7700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bec1075c0 0x7f2bec198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:15.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.475+0000 7f2bf1ac7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec112450 con 0x7f2bec102ea0 2024-04-01T01:32:15.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.475+0000 7f2bf1ac7700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec1122d0 con 0x7f2bec1075c0 2024-04-01T01:32:15.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.475+0000 7f2bf1ac7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec1125d0 con 0x7f2bec1024f0 2024-04-01T01:32:15.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.475+0000 7f2beaffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bec102ea0 0x7f2bec19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:15.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.475+0000 7f2beaffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bec102ea0 0x7f2bec19eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42018/0 (socket says 172.21.15.70:42018) 2024-04-01T01:32:15.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.475+0000 7f2beaffd700 1 -- 172.21.15.70:0/3235106105 learned_addr learned my addr 172.21.15.70:0/3235106105 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:15.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.475+0000 7f2beb7fe700 1 --2- 172.21.15.70:0/3235106105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bec1024f0 0x7f2bec19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:15.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.476+0000 7f2bebfff700 1 --2- 172.21.15.70:0/3235106105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bec1075c0 0x7f2bec198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:15.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.476+0000 7f2beaffd700 1 -- 172.21.15.70:0/3235106105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bec1024f0 msgr2=0x7f2bec19e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:15.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.476+0000 7f2beaffd700 1 --2- 172.21.15.70:0/3235106105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bec1024f0 0x7f2bec19e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:15.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.476+0000 7f2beaffd700 1 -- 172.21.15.70:0/3235106105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bec1075c0 msgr2=0x7f2bec198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:15.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.476+0000 7f2beaffd700 1 --2- 172.21.15.70:0/3235106105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bec1075c0 0x7f2bec198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:15.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.476+0000 7f2beaffd700 1 -- 172.21.15.70:0/3235106105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bec198ff0 con 0x7f2bec102ea0 2024-04-01T01:32:15.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.476+0000 7f2beb7fe700 1 --2- 172.21.15.70:0/3235106105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bec1024f0 0x7f2bec19e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:15.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.476+0000 7f2beaffd700 1 --2- 172.21.15.70:0/3235106105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bec102ea0 0x7f2bec19eb80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00e970 tx=0x7f2bdc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:15.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.476+0000 7f2be8ff9700 1 -- 172.21.15.70:0/3235106105 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc004a30 con 0x7f2bec102ea0 2024-04-01T01:32:15.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.476+0000 7f2bf1ac7700 1 -- 172.21.15.70:0/3235106105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bec1992e0 con 0x7f2bec102ea0 2024-04-01T01:32:15.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.476+0000 7f2be8ff9700 1 -- 172.21.15.70:0/3235106105 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2bdc00cc90 con 0x7f2bec102ea0 2024-04-01T01:32:15.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.476+0000 7f2be8ff9700 1 -- 172.21.15.70:0/3235106105 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc0055e0 con 0x7f2bec102ea0 2024-04-01T01:32:15.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.476+0000 7f2bf1ac7700 1 -- 172.21.15.70:0/3235106105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bec1a54a0 con 0x7f2bec102ea0 2024-04-01T01:32:15.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.477+0000 7f2bf1ac7700 1 -- 172.21.15.70:0/3235106105 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2bec062950 con 0x7f2bec102ea0 2024-04-01T01:32:15.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.478+0000 7f2be8ff9700 1 -- 172.21.15.70:0/3235106105 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f2bdc00c7e0 con 0x7f2bec102ea0 2024-04-01T01:32:15.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.479+0000 7f2be8ff9700 1 --2- 172.21.15.70:0/3235106105 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2bd40a25c0 0x7f2bd40a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:15.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.479+0000 7f2be8ff9700 1 -- 172.21.15.70:0/3235106105 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2bdc0cf0e0 con 0x7f2bec102ea0 2024-04-01T01:32:15.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.479+0000 7f2beb7fe700 1 --2- 172.21.15.70:0/3235106105 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2bd40a25c0 0x7f2bd40a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:15.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.479+0000 7f2be8ff9700 1 --2- 172.21.15.70:0/3235106105 >> [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091] conn(0x7f2bd40ad280 0x7f2bd40af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:15.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.479+0000 7f2be8ff9700 1 -- 172.21.15.70:0/3235106105 --> [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2bd40afd90 con 0x7f2bd40ad280 2024-04-01T01:32:15.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.479+0000 7f2be8ff9700 1 -- 172.21.15.70:0/3235106105 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f2bdc0cf530 con 0x7f2bec102ea0 2024-04-01T01:32:15.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.479+0000 7f2bebfff700 1 --2- 172.21.15.70:0/3235106105 >> [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091] conn(0x7f2bd40ad280 0x7f2bd40af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:15.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.479+0000 7f2beb7fe700 1 --2- 172.21.15.70:0/3235106105 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2bd40a25c0 0x7f2bd40a4a80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2be0003760 tx=0x7f2be0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:15.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.479+0000 7f2bebfff700 1 --2- 172.21.15.70:0/3235106105 >> [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091] conn(0x7f2bd40ad280 0x7f2bd40af6c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:15.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.480+0000 7f2be8ff9700 1 -- 172.21.15.70:0/3235106105 <== osd.9 v2:172.21.15.70:6826/1192094091 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2bd40afd90 con 0x7f2bd40ad280 2024-04-01T01:32:15.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.498+0000 7f2bf1ac7700 1 -- 172.21.15.70:0/3235106105 --> [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2bec062d50 con 0x7f2bd40ad280 2024-04-01T01:32:15.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.498+0000 7f2be8ff9700 1 -- 172.21.15.70:0/3235106105 <== osd.9 v2:172.21.15.70:6826/1192094091 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2bec062d50 con 0x7f2bd40ad280 2024-04-01T01:32:15.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.498+0000 7f2bf1ac7700 1 -- 172.21.15.70:0/3235106105 >> [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091] conn(0x7f2bd40ad280 msgr2=0x7f2bd40af6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:15.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.498+0000 7f2bf1ac7700 1 --2- 172.21.15.70:0/3235106105 >> [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091] conn(0x7f2bd40ad280 0x7f2bd40af6c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:15.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.498+0000 7f2bf1ac7700 1 -- 172.21.15.70:0/3235106105 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2bd40a25c0 msgr2=0x7f2bd40a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:15.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.499+0000 7f2bf1ac7700 1 --2- 172.21.15.70:0/3235106105 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2bd40a25c0 0x7f2bd40a4a80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2be0003760 tx=0x7f2be0005fb0 comp rx=0 tx=0).stop 2024-04-01T01:32:15.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.499+0000 7f2bf1ac7700 1 -- 172.21.15.70:0/3235106105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bec102ea0 msgr2=0x7f2bec19eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:15.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.499+0000 7f2bf1ac7700 1 --2- 172.21.15.70:0/3235106105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bec102ea0 0x7f2bec19eb80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00e970 tx=0x7f2bdc00ec80 comp rx=0 tx=0).stop 2024-04-01T01:32:15.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.499+0000 7f2bebfff700 1 -- 172.21.15.70:0/3235106105 reap_dead start 2024-04-01T01:32:15.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.499+0000 7f2bf1ac7700 1 -- 172.21.15.70:0/3235106105 shutdown_connections 2024-04-01T01:32:15.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.499+0000 7f2bf1ac7700 1 -- 172.21.15.70:0/3235106105 >> 172.21.15.70:0/3235106105 conn(0x7f2bec0793a0 msgr2=0x7f2bec106240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:15.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.499+0000 7f2bf1ac7700 1 -- 172.21.15.70:0/3235106105 shutdown_connections 2024-04-01T01:32:15.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:15.499+0000 7f2bf1ac7700 1 -- 172.21.15.70:0/3235106105 wait complete. 2024-04-01T01:32:15.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:15 smithi189 ceph-mon[35753]: pgmap v317: 129 pgs: 129 active+clean; 646 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-01T01:32:15.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:15 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:32:16.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:15 smithi070 ceph-mon[32179]: pgmap v317: 129 pgs: 129 active+clean; 646 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-01T01:32:16.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:15 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:32:16.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:15 smithi081 ceph-mon[35142]: pgmap v317: 129 pgs: 129 active+clean; 646 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-04-01T01:32:16.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:15 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:32:16.449 INFO:teuthology.orchestra.run.smithi070.stdout:343597383705 2024-04-01T01:32:16.449 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.11 2024-04-01T01:32:16.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.987+0000 7f2e8720e700 1 -- 172.21.15.70:0/3417101802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e80101d80 msgr2=0x7f2e80105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:16.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.987+0000 7f2e8720e700 1 --2- 172.21.15.70:0/3417101802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e80101d80 0x7f2e80105d70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c009990 tx=0x7f2e6c009ca0 comp rx=0 tx=0).stop 2024-04-01T01:32:16.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.987+0000 7f2e8720e700 1 -- 172.21.15.70:0/3417101802 shutdown_connections 2024-04-01T01:32:16.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.988+0000 7f2e8720e700 1 --2- 172.21.15.70:0/3417101802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e80106450 0x7f2e8010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:16.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.988+0000 7f2e8720e700 1 --2- 172.21.15.70:0/3417101802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e801013d0 0x7f2e801017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:16.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.988+0000 7f2e8720e700 1 --2- 172.21.15.70:0/3417101802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e80101d80 0x7f2e80105d70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:16.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.988+0000 7f2e8720e700 1 -- 172.21.15.70:0/3417101802 >> 172.21.15.70:0/3417101802 conn(0x7f2e80077150 msgr2=0x7f2e80077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:16.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.988+0000 7f2e8720e700 1 -- 172.21.15.70:0/3417101802 shutdown_connections 2024-04-01T01:32:16.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.988+0000 7f2e8720e700 1 -- 172.21.15.70:0/3417101802 wait complete. 2024-04-01T01:32:16.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.988+0000 7f2e8720e700 1 Processor -- start 2024-04-01T01:32:16.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.988+0000 7f2e8720e700 1 -- start start 2024-04-01T01:32:16.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.988+0000 7f2e8720e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e801013d0 0x7f2e80199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:16.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.988+0000 7f2e8720e700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e80101d80 0x7f2e80199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:16.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.988+0000 7f2e8720e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e80106450 0x7f2e801a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:16.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.988+0000 7f2e8720e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2e801112c0 con 0x7f2e801013d0 2024-04-01T01:32:16.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.988+0000 7f2e8720e700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2e80111140 con 0x7f2e80101d80 2024-04-01T01:32:16.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.988+0000 7f2e8720e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2e80111440 con 0x7f2e80106450 2024-04-01T01:32:16.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.988+0000 7f2e84faa700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e801013d0 0x7f2e80199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:16.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.988+0000 7f2e84faa700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e801013d0 0x7f2e80199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43674/0 (socket says 172.21.15.70:43674) 2024-04-01T01:32:16.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.989+0000 7f2e84faa700 1 -- 172.21.15.70:0/109558928 learned_addr learned my addr 172.21.15.70:0/109558928 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:16.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.989+0000 7f2e7ffff700 1 --2- 172.21.15.70:0/109558928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e80101d80 0x7f2e80199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:16.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.989+0000 7f2e857ab700 1 --2- 172.21.15.70:0/109558928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e80106450 0x7f2e801a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:16.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.989+0000 7f2e84faa700 1 -- 172.21.15.70:0/109558928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e80106450 msgr2=0x7f2e801a4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:16.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.989+0000 7f2e84faa700 1 --2- 172.21.15.70:0/109558928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e80106450 0x7f2e801a4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:16.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.989+0000 7f2e84faa700 1 -- 172.21.15.70:0/109558928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e80101d80 msgr2=0x7f2e80199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:16.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.989+0000 7f2e84faa700 1 --2- 172.21.15.70:0/109558928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e80101d80 0x7f2e80199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:16.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.989+0000 7f2e84faa700 1 -- 172.21.15.70:0/109558928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e801a4d40 con 0x7f2e801013d0 2024-04-01T01:32:16.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.989+0000 7f2e84faa700 1 --2- 172.21.15.70:0/109558928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e801013d0 0x7f2e80199780 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400d950 tx=0x7f2e7400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:16.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.989+0000 7f2e7dffb700 1 -- 172.21.15.70:0/109558928 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e74016070 con 0x7f2e801013d0 2024-04-01T01:32:16.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.989+0000 7f2e8720e700 1 -- 172.21.15.70:0/109558928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e801a4f70 con 0x7f2e801013d0 2024-04-01T01:32:16.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.989+0000 7f2e7dffb700 1 -- 172.21.15.70:0/109558928 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e74009df0 con 0x7f2e801013d0 2024-04-01T01:32:16.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.989+0000 7f2e7dffb700 1 -- 172.21.15.70:0/109558928 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7400f3c0 con 0x7f2e801013d0 2024-04-01T01:32:16.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.989+0000 7f2e8720e700 1 -- 172.21.15.70:0/109558928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e801a5370 con 0x7f2e801013d0 2024-04-01T01:32:16.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.990+0000 7f2e8720e700 1 -- 172.21.15.70:0/109558928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2e80062950 con 0x7f2e801013d0 2024-04-01T01:32:16.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.991+0000 7f2e7dffb700 1 -- 172.21.15.70:0/109558928 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f2e74019070 con 0x7f2e801013d0 2024-04-01T01:32:16.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.992+0000 7f2e7dffb700 1 --2- 172.21.15.70:0/109558928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2e700a25c0 0x7f2e700a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:16.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.992+0000 7f2e7ffff700 1 --2- 172.21.15.70:0/109558928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2e700a25c0 0x7f2e700a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:16.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.992+0000 7f2e7dffb700 1 -- 172.21.15.70:0/109558928 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2e740ceb70 con 0x7f2e801013d0 2024-04-01T01:32:16.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.992+0000 7f2e7dffb700 1 --2- 172.21.15.70:0/109558928 >> [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021] conn(0x7f2e700ad280 0x7f2e700af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:16.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.992+0000 7f2e7dffb700 1 -- 172.21.15.70:0/109558928 --> [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e74014e80 con 0x7f2e700ad280 2024-04-01T01:32:16.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.992+0000 7f2e7dffb700 1 -- 172.21.15.70:0/109558928 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f2e740cefc0 con 0x7f2e801013d0 2024-04-01T01:32:16.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.992+0000 7f2e7ffff700 1 --2- 172.21.15.70:0/109558928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2e700a25c0 0x7f2e700a4a80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c006010 tx=0x7f2e6c005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:16.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.992+0000 7f2e857ab700 1 --2- 172.21.15.70:0/109558928 >> [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021] conn(0x7f2e700ad280 0x7f2e700af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:16.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.993+0000 7f2e857ab700 1 --2- 172.21.15.70:0/109558928 >> [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021] conn(0x7f2e700ad280 0x7f2e700af6c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:16.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:16.994+0000 7f2e7dffb700 1 -- 172.21.15.70:0/109558928 <== osd.8 v2:172.21.15.189:6816/2097527021 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2e74014e80 con 0x7f2e700ad280 2024-04-01T01:32:17.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.010+0000 7f2e8720e700 1 -- 172.21.15.70:0/109558928 --> [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2e80062dd0 con 0x7f2e700ad280 2024-04-01T01:32:17.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.010+0000 7f2e7dffb700 1 -- 172.21.15.70:0/109558928 <== osd.8 v2:172.21.15.189:6816/2097527021 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2e80062dd0 con 0x7f2e700ad280 2024-04-01T01:32:17.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.011+0000 7f2e8720e700 1 -- 172.21.15.70:0/109558928 >> [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021] conn(0x7f2e700ad280 msgr2=0x7f2e700af6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:17.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.011+0000 7f2e8720e700 1 --2- 172.21.15.70:0/109558928 >> [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021] conn(0x7f2e700ad280 0x7f2e700af6c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.011+0000 7f2e8720e700 1 -- 172.21.15.70:0/109558928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2e700a25c0 msgr2=0x7f2e700a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:17.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.011+0000 7f2e8720e700 1 --2- 172.21.15.70:0/109558928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2e700a25c0 0x7f2e700a4a80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c006010 tx=0x7f2e6c005960 comp rx=0 tx=0).stop 2024-04-01T01:32:17.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.011+0000 7f2e8720e700 1 -- 172.21.15.70:0/109558928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e801013d0 msgr2=0x7f2e80199780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:17.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.011+0000 7f2e8720e700 1 --2- 172.21.15.70:0/109558928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e801013d0 0x7f2e80199780 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400d950 tx=0x7f2e7400dc60 comp rx=0 tx=0).stop 2024-04-01T01:32:17.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.011+0000 7f2e857ab700 1 -- 172.21.15.70:0/109558928 reap_dead start 2024-04-01T01:32:17.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.011+0000 7f2e8720e700 1 -- 172.21.15.70:0/109558928 shutdown_connections 2024-04-01T01:32:17.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.011+0000 7f2e8720e700 1 -- 172.21.15.70:0/109558928 >> 172.21.15.70:0/109558928 conn(0x7f2e80077150 msgr2=0x7f2e801075d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:17.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.011+0000 7f2e8720e700 1 -- 172.21.15.70:0/109558928 shutdown_connections 2024-04-01T01:32:17.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.011+0000 7f2e8720e700 1 -- 172.21.15.70:0/109558928 wait complete. 2024-04-01T01:32:17.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.448+0000 7f081a944700 1 -- 172.21.15.70:0/3884666382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0814102480 msgr2=0x7f0814102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:17.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.448+0000 7f081a944700 1 --2- 172.21.15.70:0/3884666382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0814102480 0x7f0814102860 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f08080099c0 tx=0x7f0808009cd0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.448+0000 7f081a944700 1 -- 172.21.15.70:0/3884666382 shutdown_connections 2024-04-01T01:32:17.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.448+0000 7f081a944700 1 --2- 172.21.15.70:0/3884666382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0814107560 0x7f081410f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.448+0000 7f081a944700 1 --2- 172.21.15.70:0/3884666382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0814102480 0x7f0814102860 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.448+0000 7f081a944700 1 --2- 172.21.15.70:0/3884666382 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0814102e30 0x7f0814106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.448+0000 7f081a944700 1 -- 172.21.15.70:0/3884666382 >> 172.21.15.70:0/3884666382 conn(0x7f08140fdcf0 msgr2=0x7f0814100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:17.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.448+0000 7f081a944700 1 -- 172.21.15.70:0/3884666382 shutdown_connections 2024-04-01T01:32:17.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.448+0000 7f081a944700 1 -- 172.21.15.70:0/3884666382 wait complete. 2024-04-01T01:32:17.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.448+0000 7f081a944700 1 Processor -- start 2024-04-01T01:32:17.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.448+0000 7f081a944700 1 -- start start 2024-04-01T01:32:17.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f081a944700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0814102480 0x7f081419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:17.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f081a944700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0814102e30 0x7f081419ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:17.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f081a944700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0814107560 0x7f0814198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:17.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f081a944700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08141123f0 con 0x7f0814102480 2024-04-01T01:32:17.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f081a944700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0814112270 con 0x7f0814107560 2024-04-01T01:32:17.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f081a944700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0814112570 con 0x7f0814102e30 2024-04-01T01:32:17.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f0813fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0814102480 0x7f081419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:17.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f0813fff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0814102480 0x7f081419e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43704/0 (socket says 172.21.15.70:43704) 2024-04-01T01:32:17.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f0813fff700 1 -- 172.21.15.70:0/1364233676 learned_addr learned my addr 172.21.15.70:0/1364233676 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:17.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f08137fe700 1 --2- 172.21.15.70:0/1364233676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0814102e30 0x7f081419ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:17.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f0818ee1700 1 --2- 172.21.15.70:0/1364233676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0814107560 0x7f0814198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:17.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f0813fff700 1 -- 172.21.15.70:0/1364233676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0814102e30 msgr2=0x7f081419ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:17.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f0813fff700 1 --2- 172.21.15.70:0/1364233676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0814102e30 0x7f081419ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f0813fff700 1 -- 172.21.15.70:0/1364233676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0814107560 msgr2=0x7f0814198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:17.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f0813fff700 1 --2- 172.21.15.70:0/1364233676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0814107560 0x7f0814198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f0813fff700 1 -- 172.21.15.70:0/1364233676 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0814199070 con 0x7f0814102480 2024-04-01T01:32:17.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f08137fe700 1 --2- 172.21.15.70:0/1364233676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0814102e30 0x7f081419ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:17.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f0813fff700 1 --2- 172.21.15.70:0/1364233676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0814102480 0x7f081419e670 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f08080059d0 tx=0x7f08080057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:17.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f08117fa700 1 -- 172.21.15.70:0/1364233676 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0808027070 con 0x7f0814102480 2024-04-01T01:32:17.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f08117fa700 1 -- 172.21.15.70:0/1364233676 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0808003f50 con 0x7f0814102480 2024-04-01T01:32:17.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.449+0000 7f081a944700 1 -- 172.21.15.70:0/1364233676 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0814199300 con 0x7f0814102480 2024-04-01T01:32:17.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.450+0000 7f08117fa700 1 -- 172.21.15.70:0/1364233676 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0808022c30 con 0x7f0814102480 2024-04-01T01:32:17.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.450+0000 7f081a944700 1 -- 172.21.15.70:0/1364233676 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08141a5420 con 0x7f0814102480 2024-04-01T01:32:17.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.450+0000 7f081a944700 1 -- 172.21.15.70:0/1364233676 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0814062990 con 0x7f0814102480 2024-04-01T01:32:17.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.452+0000 7f08117fa700 1 -- 172.21.15.70:0/1364233676 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f080802a070 con 0x7f0814102480 2024-04-01T01:32:17.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.452+0000 7f08117fa700 1 --2- 172.21.15.70:0/1364233676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07fc0a25c0 0x7f07fc0a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:17.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.452+0000 7f08137fe700 1 --2- 172.21.15.70:0/1364233676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07fc0a25c0 0x7f07fc0a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:17.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.452+0000 7f08117fa700 1 -- 172.21.15.70:0/1364233676 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f08080d0230 con 0x7f0814102480 2024-04-01T01:32:17.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.453+0000 7f08117fa700 1 --2- 172.21.15.70:0/1364233676 >> [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043] conn(0x7f07fc0ad280 0x7f07fc0af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:17.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.453+0000 7f08117fa700 1 -- 172.21.15.70:0/1364233676 --> [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0808019e80 con 0x7f07fc0ad280 2024-04-01T01:32:17.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.453+0000 7f08117fa700 1 -- 172.21.15.70:0/1364233676 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f08080d0710 con 0x7f0814102480 2024-04-01T01:32:17.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.453+0000 7f08137fe700 1 --2- 172.21.15.70:0/1364233676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07fc0a25c0 0x7f07fc0a4a80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0814199f00 tx=0x7f08040073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:17.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.453+0000 7f0818ee1700 1 --2- 172.21.15.70:0/1364233676 >> [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043] conn(0x7f07fc0ad280 0x7f07fc0af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:17.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.453+0000 7f0818ee1700 1 --2- 172.21.15.70:0/1364233676 >> [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043] conn(0x7f07fc0ad280 0x7f07fc0af6c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:17.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.454+0000 7f08117fa700 1 -- 172.21.15.70:0/1364233676 <== osd.0 v2:172.21.15.70:6802/3697496043 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0808019e80 con 0x7f07fc0ad280 2024-04-01T01:32:17.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.472+0000 7f081a944700 1 -- 172.21.15.70:0/1364233676 --> [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0814062e10 con 0x7f07fc0ad280 2024-04-01T01:32:17.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.473+0000 7f08117fa700 1 -- 172.21.15.70:0/1364233676 <== osd.0 v2:172.21.15.70:6802/3697496043 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0814062e10 con 0x7f07fc0ad280 2024-04-01T01:32:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.473+0000 7f081a944700 1 -- 172.21.15.70:0/1364233676 >> [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043] conn(0x7f07fc0ad280 msgr2=0x7f07fc0af6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.473+0000 7f081a944700 1 --2- 172.21.15.70:0/1364233676 >> [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043] conn(0x7f07fc0ad280 0x7f07fc0af6c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.473+0000 7f081a944700 1 -- 172.21.15.70:0/1364233676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07fc0a25c0 msgr2=0x7f07fc0a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.473+0000 7f081a944700 1 --2- 172.21.15.70:0/1364233676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07fc0a25c0 0x7f07fc0a4a80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0814199f00 tx=0x7f08040073d0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.473+0000 7f081a944700 1 -- 172.21.15.70:0/1364233676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0814102480 msgr2=0x7f081419e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.473+0000 7f081a944700 1 --2- 172.21.15.70:0/1364233676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0814102480 0x7f081419e670 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f08080059d0 tx=0x7f08080057c0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.473+0000 7f0818ee1700 1 -- 172.21.15.70:0/1364233676 reap_dead start 2024-04-01T01:32:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.473+0000 7f081a944700 1 -- 172.21.15.70:0/1364233676 shutdown_connections 2024-04-01T01:32:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.473+0000 7f081a944700 1 -- 172.21.15.70:0/1364233676 >> 172.21.15.70:0/1364233676 conn(0x7f08140fdcf0 msgr2=0x7f08140ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.473+0000 7f081a944700 1 -- 172.21.15.70:0/1364233676 shutdown_connections 2024-04-01T01:32:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.473+0000 7f081a944700 1 -- 172.21.15.70:0/1364233676 wait complete. 2024-04-01T01:32:17.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.643+0000 7fe006d22700 1 -- 172.21.15.70:0/3796291189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe000107170 msgr2=0x7fe000107550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:17.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.643+0000 7fe006d22700 1 --2- 172.21.15.70:0/3796291189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe000107170 0x7fe000107550 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdff4009990 tx=0x7fdff4009ca0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.643+0000 7fe006d22700 1 -- 172.21.15.70:0/3796291189 shutdown_connections 2024-04-01T01:32:17.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.643+0000 7fe006d22700 1 --2- 172.21.15.70:0/3796291189 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe000101af0 0x7fe00010dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.643+0000 7fe006d22700 1 --2- 172.21.15.70:0/3796291189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe000107170 0x7fe000107550 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.643+0000 7fe006d22700 1 --2- 172.21.15.70:0/3796291189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe000101130 0x7fe0001015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.643+0000 7fe006d22700 1 -- 172.21.15.70:0/3796291189 >> 172.21.15.70:0/3796291189 conn(0x7fe0000780e0 msgr2=0x7fe0000784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:17.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.643+0000 7fe006d22700 1 -- 172.21.15.70:0/3796291189 shutdown_connections 2024-04-01T01:32:17.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.643+0000 7fe006d22700 1 -- 172.21.15.70:0/3796291189 wait complete. 2024-04-01T01:32:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.644+0000 7fe006d22700 1 Processor -- start 2024-04-01T01:32:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.644+0000 7fe006d22700 1 -- start start 2024-04-01T01:32:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.644+0000 7fe006d22700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe000101130 0x7fe00019a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.644+0000 7fe006d22700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe000101af0 0x7fe000196760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.644+0000 7fe006d22700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe000107170 0x7fe000196ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.644+0000 7fe006d22700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe000111290 con 0x7fe000101af0 2024-04-01T01:32:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.644+0000 7fe006d22700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe000111110 con 0x7fe000101130 2024-04-01T01:32:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.644+0000 7fe006d22700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe000111410 con 0x7fe000107170 2024-04-01T01:32:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.645+0000 7fdffffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe000101af0 0x7fe000196760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:17.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.645+0000 7fe004abe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe000101130 0x7fe00019a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:17.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.645+0000 7fdffffff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe000101af0 0x7fe000196760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43720/0 (socket says 172.21.15.70:43720) 2024-04-01T01:32:17.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.645+0000 7fdffffff700 1 -- 172.21.15.70:0/1452624399 learned_addr learned my addr 172.21.15.70:0/1452624399 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:17.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.645+0000 7fe0052bf700 1 --2- 172.21.15.70:0/1452624399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe000107170 0x7fe000196ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:17.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.645+0000 7fe004abe700 1 -- 172.21.15.70:0/1452624399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe000107170 msgr2=0x7fe000196ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:17.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.645+0000 7fe004abe700 1 --2- 172.21.15.70:0/1452624399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe000107170 0x7fe000196ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.645+0000 7fe004abe700 1 -- 172.21.15.70:0/1452624399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe000101af0 msgr2=0x7fe000196760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:17.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.645+0000 7fe004abe700 1 --2- 172.21.15.70:0/1452624399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe000101af0 0x7fe000196760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.645+0000 7fe004abe700 1 -- 172.21.15.70:0/1452624399 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe000197520 con 0x7fe000101130 2024-04-01T01:32:17.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.645+0000 7fe004abe700 1 --2- 172.21.15.70:0/1452624399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe000101130 0x7fe00019a440 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdff4006010 tx=0x7fdff40057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:17.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.645+0000 7fdffdffb700 1 -- 172.21.15.70:0/1452624399 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4027070 con 0x7fe000101130 2024-04-01T01:32:17.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.646+0000 7fe006d22700 1 -- 172.21.15.70:0/1452624399 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0001ab430 con 0x7fe000101130 2024-04-01T01:32:17.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.646+0000 7fe006d22700 1 -- 172.21.15.70:0/1452624399 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0001ab880 con 0x7fe000101130 2024-04-01T01:32:17.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.645+0000 7fdffdffb700 1 -- 172.21.15.70:0/1452624399 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdff4003f10 con 0x7fe000101130 2024-04-01T01:32:17.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.646+0000 7fdffdffb700 1 -- 172.21.15.70:0/1452624399 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4022bd0 con 0x7fe000101130 2024-04-01T01:32:17.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.647+0000 7fe006d22700 1 -- 172.21.15.70:0/1452624399 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdfe4001030 con 0x7fe000101130 2024-04-01T01:32:17.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.649+0000 7fdffdffb700 1 -- 172.21.15.70:0/1452624399 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fdff402a070 con 0x7fe000101130 2024-04-01T01:32:17.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.649+0000 7fdffdffb700 1 --2- 172.21.15.70:0/1452624399 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdff00a25c0 0x7fdff00a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:17.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.649+0000 7fdffdffb700 1 -- 172.21.15.70:0/1452624399 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdff40cf8a0 con 0x7fe000101130 2024-04-01T01:32:17.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.649+0000 7fdffffff700 1 --2- 172.21.15.70:0/1452624399 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdff00a25c0 0x7fdff00a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:17.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.650+0000 7fdffdffb700 1 --2- 172.21.15.70:0/1452624399 >> [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460] conn(0x7fdff00ad140 0x7fdff00af580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:17.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.650+0000 7fdffdffb700 1 -- 172.21.15.70:0/1452624399 --> [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdff4001d30 con 0x7fdff00ad140 2024-04-01T01:32:17.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.650+0000 7fdffdffb700 1 -- 172.21.15.70:0/1452624399 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fdff40d42c0 con 0x7fe000101130 2024-04-01T01:32:17.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.650+0000 7fe0052bf700 1 --2- 172.21.15.70:0/1452624399 >> [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460] conn(0x7fdff00ad140 0x7fdff00af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:17.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.650+0000 7fdffffff700 1 --2- 172.21.15.70:0/1452624399 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdff00a25c0 0x7fdff00a4a80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdfec007c90 tx=0x7fdfec007480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:17.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.650+0000 7fe0052bf700 1 --2- 172.21.15.70:0/1452624399 >> [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460] conn(0x7fdff00ad140 0x7fdff00af580 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:17.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.652+0000 7fdffdffb700 1 -- 172.21.15.70:0/1452624399 <== osd.2 v2:172.21.15.189:6800/4191559460 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdff4001d30 con 0x7fdff00ad140 2024-04-01T01:32:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.672+0000 7fe006d22700 1 -- 172.21.15.70:0/1452624399 --> [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdfe4002e60 con 0x7fdff00ad140 2024-04-01T01:32:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.672+0000 7fdffdffb700 1 -- 172.21.15.70:0/1452624399 <== osd.2 v2:172.21.15.189:6800/4191559460 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdfe4002e60 con 0x7fdff00ad140 2024-04-01T01:32:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.673+0000 7fe006d22700 1 -- 172.21.15.70:0/1452624399 >> [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460] conn(0x7fdff00ad140 msgr2=0x7fdff00af580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.673+0000 7fe006d22700 1 --2- 172.21.15.70:0/1452624399 >> [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460] conn(0x7fdff00ad140 0x7fdff00af580 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.673+0000 7fe006d22700 1 -- 172.21.15.70:0/1452624399 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdff00a25c0 msgr2=0x7fdff00a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.673+0000 7fe006d22700 1 --2- 172.21.15.70:0/1452624399 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdff00a25c0 0x7fdff00a4a80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdfec007c90 tx=0x7fdfec007480 comp rx=0 tx=0).stop 2024-04-01T01:32:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.673+0000 7fe006d22700 1 -- 172.21.15.70:0/1452624399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe000101130 msgr2=0x7fe00019a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.673+0000 7fe006d22700 1 --2- 172.21.15.70:0/1452624399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe000101130 0x7fe00019a440 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdff4006010 tx=0x7fdff40057c0 comp rx=0 tx=0).stop 2024-04-01T01:32:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.673+0000 7fe0052bf700 1 -- 172.21.15.70:0/1452624399 reap_dead start 2024-04-01T01:32:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.673+0000 7fe006d22700 1 -- 172.21.15.70:0/1452624399 shutdown_connections 2024-04-01T01:32:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.673+0000 7fe006d22700 1 -- 172.21.15.70:0/1452624399 >> 172.21.15.70:0/1452624399 conn(0x7fe0000780e0 msgr2=0x7fe0000ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.673+0000 7fe006d22700 1 -- 172.21.15.70:0/1452624399 shutdown_connections 2024-04-01T01:32:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:17.673+0000 7fe006d22700 1 -- 172.21.15.70:0/1452624399 wait complete. 2024-04-01T01:32:17.993 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:17 smithi070 ceph-mon[32179]: pgmap v318: 129 pgs: 129 active+clean; 646 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 11 op/s 2024-04-01T01:32:17.994 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:17 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:17.994 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:17 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:17.994 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:17 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:17.994 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:17 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:17.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:17 smithi189 ceph-mon[35753]: pgmap v318: 129 pgs: 129 active+clean; 646 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 11 op/s 2024-04-01T01:32:17.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:17 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:17.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:17 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:17.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:17 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:17.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:17 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:18.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:17 smithi081 ceph-mon[35142]: pgmap v318: 129 pgs: 129 active+clean; 646 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 11 op/s 2024-04-01T01:32:18.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:17 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:18.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:17 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:18.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:17 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:18.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:17 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:18.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.095+0000 7f824076f700 1 -- 172.21.15.70:0/1954361561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8238102cb0 msgr2=0x7f823810f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:18.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.095+0000 7f824076f700 1 --2- 172.21.15.70:0/1954361561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8238102cb0 0x7f823810f0a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f823000b0a0 tx=0x7f823000b3b0 comp rx=0 tx=0).stop 2024-04-01T01:32:18.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.095+0000 7f824076f700 1 -- 172.21.15.70:0/1954361561 shutdown_connections 2024-04-01T01:32:18.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.095+0000 7f824076f700 1 --2- 172.21.15.70:0/1954361561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8238102cb0 0x7f823810f0a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:18.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.095+0000 7f824076f700 1 --2- 172.21.15.70:0/1954361561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8238108360 0x7f8238108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:18.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.095+0000 7f824076f700 1 --2- 172.21.15.70:0/1954361561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82381022f0 0x7f8238102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:18.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.095+0000 7f824076f700 1 -- 172.21.15.70:0/1954361561 >> 172.21.15.70:0/1954361561 conn(0x7f8238079310 msgr2=0x7f8238079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:18.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.095+0000 7f824076f700 1 -- 172.21.15.70:0/1954361561 shutdown_connections 2024-04-01T01:32:18.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.095+0000 7f824076f700 1 -- 172.21.15.70:0/1954361561 wait complete. 2024-04-01T01:32:18.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f824076f700 1 Processor -- start 2024-04-01T01:32:18.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f824076f700 1 -- start start 2024-04-01T01:32:18.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f824076f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82381022f0 0x7f823819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:18.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f824076f700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8238102cb0 0x7f823819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:18.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f824076f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8238108360 0x7f8238198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:18.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f824076f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8238112450 con 0x7f8238108360 2024-04-01T01:32:18.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f824076f700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f82381122d0 con 0x7f8238102cb0 2024-04-01T01:32:18.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f824076f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f82381125d0 con 0x7f82381022f0 2024-04-01T01:32:18.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f823ed0c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8238108360 0x7f8238198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:18.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f823ed0c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8238108360 0x7f8238198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43740/0 (socket says 172.21.15.70:43740) 2024-04-01T01:32:18.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f823ed0c700 1 -- 172.21.15.70:0/872991998 learned_addr learned my addr 172.21.15.70:0/872991998 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:18.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f823e50b700 1 --2- 172.21.15.70:0/872991998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82381022f0 0x7f823819e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:18.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f823dd0a700 1 --2- 172.21.15.70:0/872991998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8238102cb0 0x7f823819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:18.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f823ed0c700 1 -- 172.21.15.70:0/872991998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82381022f0 msgr2=0x7f823819e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:18.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f823ed0c700 1 --2- 172.21.15.70:0/872991998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82381022f0 0x7f823819e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:18.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f823ed0c700 1 -- 172.21.15.70:0/872991998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8238102cb0 msgr2=0x7f823819ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:18.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f823ed0c700 1 --2- 172.21.15.70:0/872991998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8238102cb0 0x7f823819ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:18.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f823ed0c700 1 -- 172.21.15.70:0/872991998 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8238199070 con 0x7f8238108360 2024-04-01T01:32:18.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f823dd0a700 1 --2- 172.21.15.70:0/872991998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8238102cb0 0x7f823819ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:18.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.096+0000 7f823ed0c700 1 --2- 172.21.15.70:0/872991998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8238108360 0x7f8238198790 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f82300062a0 tx=0x7f8230004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:18.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.097+0000 7f822b7fe700 1 -- 172.21.15.70:0/872991998 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8230021070 con 0x7f8238108360 2024-04-01T01:32:18.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.097+0000 7f824076f700 1 -- 172.21.15.70:0/872991998 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8238199300 con 0x7f8238108360 2024-04-01T01:32:18.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.097+0000 7f822b7fe700 1 -- 172.21.15.70:0/872991998 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8230007e30 con 0x7f8238108360 2024-04-01T01:32:18.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.097+0000 7f822b7fe700 1 -- 172.21.15.70:0/872991998 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82300046d0 con 0x7f8238108360 2024-04-01T01:32:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.097+0000 7f824076f700 1 -- 172.21.15.70:0/872991998 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82381a55f0 con 0x7f8238108360 2024-04-01T01:32:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.097+0000 7f824076f700 1 -- 172.21.15.70:0/872991998 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8238062910 con 0x7f8238108360 2024-04-01T01:32:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.099+0000 7f822b7fe700 1 -- 172.21.15.70:0/872991998 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f8230007980 con 0x7f8238108360 2024-04-01T01:32:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.099+0000 7f822b7fe700 1 --2- 172.21.15.70:0/872991998 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f82240a25c0 0x7f82240a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.099+0000 7f823e50b700 1 --2- 172.21.15.70:0/872991998 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f82240a25c0 0x7f82240a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.099+0000 7f822b7fe700 1 -- 172.21.15.70:0/872991998 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f823000f020 con 0x7f8238108360 2024-04-01T01:32:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.099+0000 7f822b7fe700 1 --2- 172.21.15.70:0/872991998 >> [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261] conn(0x7f82240ad280 0x7f82240af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.099+0000 7f822b7fe700 1 -- 172.21.15.70:0/872991998 --> [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8230026e50 con 0x7f82240ad280 2024-04-01T01:32:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.099+0000 7f822b7fe700 1 -- 172.21.15.70:0/872991998 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f82300d0530 con 0x7f8238108360 2024-04-01T01:32:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.100+0000 7f823dd0a700 1 --2- 172.21.15.70:0/872991998 >> [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261] conn(0x7f82240ad280 0x7f82240af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.100+0000 7f823e50b700 1 --2- 172.21.15.70:0/872991998 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f82240a25c0 0x7f82240a4a80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f82340045b0 tx=0x7f823400b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.100+0000 7f823dd0a700 1 --2- 172.21.15.70:0/872991998 >> [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261] conn(0x7f82240ad280 0x7f82240af6c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.101+0000 7f822b7fe700 1 -- 172.21.15.70:0/872991998 <== osd.4 v2:172.21.15.81:6808/404027261 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8230026e50 con 0x7f82240ad280 2024-04-01T01:32:18.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.117+0000 7f824076f700 1 -- 172.21.15.70:0/872991998 --> [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8238062d10 con 0x7f82240ad280 2024-04-01T01:32:18.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.118+0000 7f822b7fe700 1 -- 172.21.15.70:0/872991998 <== osd.4 v2:172.21.15.81:6808/404027261 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8238062d10 con 0x7f82240ad280 2024-04-01T01:32:18.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.118+0000 7f824076f700 1 -- 172.21.15.70:0/872991998 >> [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261] conn(0x7f82240ad280 msgr2=0x7f82240af6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:18.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.118+0000 7f824076f700 1 --2- 172.21.15.70:0/872991998 >> [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261] conn(0x7f82240ad280 0x7f82240af6c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:18.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.118+0000 7f824076f700 1 -- 172.21.15.70:0/872991998 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f82240a25c0 msgr2=0x7f82240a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:18.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.118+0000 7f824076f700 1 --2- 172.21.15.70:0/872991998 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f82240a25c0 0x7f82240a4a80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f82340045b0 tx=0x7f823400b3c0 comp rx=0 tx=0).stop 2024-04-01T01:32:18.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.118+0000 7f824076f700 1 -- 172.21.15.70:0/872991998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8238108360 msgr2=0x7f8238198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:18.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.118+0000 7f824076f700 1 --2- 172.21.15.70:0/872991998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8238108360 0x7f8238198790 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f82300062a0 tx=0x7f8230004c80 comp rx=0 tx=0).stop 2024-04-01T01:32:18.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.118+0000 7f823ed0c700 1 -- 172.21.15.70:0/872991998 reap_dead start 2024-04-01T01:32:18.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.118+0000 7f824076f700 1 -- 172.21.15.70:0/872991998 shutdown_connections 2024-04-01T01:32:18.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.118+0000 7f824076f700 1 -- 172.21.15.70:0/872991998 >> 172.21.15.70:0/872991998 conn(0x7f8238079310 msgr2=0x7f8238100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:18.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.118+0000 7f824076f700 1 -- 172.21.15.70:0/872991998 shutdown_connections 2024-04-01T01:32:18.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.118+0000 7f824076f700 1 -- 172.21.15.70:0/872991998 wait complete. 2024-04-01T01:32:18.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-01T01:32:18.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-01T01:32:18.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-04-01T01:32:18.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-04-01T01:32:18.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-04-01T01:32:18.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-04-01T01:32:18.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-04-01T01:32:18.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-04-01T01:32:18.715 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-04-01T01:32:18.716 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-04-01T01:32:18.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-01T01:32:18.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-01T01:32:18.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-04-01T01:32:18.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-04-01T01:32:18.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-04-01T01:32:18.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-04-01T01:32:18.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-04-01T01:32:18.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-04-01T01:32:18.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-04-01T01:32:18.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-04-01T01:32:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-01T01:32:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-01T01:32:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-04-01T01:32:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-04-01T01:32:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-04-01T01:32:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-04-01T01:32:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-04-01T01:32:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-04-01T01:32:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-04-01T01:32:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-04-01T01:32:19.471 INFO:teuthology.orchestra.run.smithi070.stdout:253403070504 2024-04-01T01:32:19.471 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.8 2024-04-01T01:32:19.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:19 smithi070 ceph-mon[32179]: pgmap v319: 129 pgs: 129 active+clean; 646 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-01T01:32:19.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-01T01:32:19.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-01T01:32:19.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-04-01T01:32:19.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-04-01T01:32:19.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-04-01T01:32:19.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-04-01T01:32:19.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-04-01T01:32:19.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-04-01T01:32:19.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-04-01T01:32:19.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-04-01T01:32:19.755 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:19 smithi070 ceph-mon[32179]: osdmap e90: 12 total, 12 up, 12 in 2024-04-01T01:32:19.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.557+0000 7ff813662700 1 -- 172.21.15.70:0/4152622301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c107560 msgr2=0x7ff80c10f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:19.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.557+0000 7ff813662700 1 --2- 172.21.15.70:0/4152622301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c107560 0x7ff80c10f130 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff80400b0d0 tx=0x7ff80400b3e0 comp rx=0 tx=0).stop 2024-04-01T01:32:19.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.557+0000 7ff813662700 1 -- 172.21.15.70:0/4152622301 shutdown_connections 2024-04-01T01:32:19.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.557+0000 7ff813662700 1 --2- 172.21.15.70:0/4152622301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c107560 0x7ff80c10f130 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:19.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.557+0000 7ff813662700 1 --2- 172.21.15.70:0/4152622301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff80c102480 0x7ff80c102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.557+0000 7ff813662700 1 --2- 172.21.15.70:0/4152622301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff80c102e30 0x7ff80c106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.557+0000 7ff813662700 1 -- 172.21.15.70:0/4152622301 >> 172.21.15.70:0/4152622301 conn(0x7ff80c0fdcf0 msgr2=0x7ff80c100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.557+0000 7ff813662700 1 -- 172.21.15.70:0/4152622301 shutdown_connections 2024-04-01T01:32:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.557+0000 7ff813662700 1 -- 172.21.15.70:0/4152622301 wait complete. 2024-04-01T01:32:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.557+0000 7ff813662700 1 Processor -- start 2024-04-01T01:32:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.557+0000 7ff813662700 1 -- start start 2024-04-01T01:32:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff813662700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff80c102480 0x7ff80c074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff813662700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c102e30 0x7ff80c074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff813662700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff80c107560 0x7ff80c074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff813662700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c112400 con 0x7ff80c102e30 2024-04-01T01:32:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff813662700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c112280 con 0x7ff80c107560 2024-04-01T01:32:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff813662700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c112580 con 0x7ff80c102480 2024-04-01T01:32:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff810bfd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c102e30 0x7ff80c074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff810bfd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c102e30 0x7ff80c074a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43762/0 (socket says 172.21.15.70:43762) 2024-04-01T01:32:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff8113fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff80c102480 0x7ff80c074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff810bfd700 1 -- 172.21.15.70:0/2984415383 learned_addr learned my addr 172.21.15.70:0/2984415383 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff811bff700 1 --2- 172.21.15.70:0/2984415383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff80c107560 0x7ff80c074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff810bfd700 1 -- 172.21.15.70:0/2984415383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff80c102480 msgr2=0x7ff80c074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff810bfd700 1 --2- 172.21.15.70:0/2984415383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff80c102480 0x7ff80c074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff810bfd700 1 -- 172.21.15.70:0/2984415383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff80c107560 msgr2=0x7ff80c074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff810bfd700 1 --2- 172.21.15.70:0/2984415383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff80c107560 0x7ff80c074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff810bfd700 1 -- 172.21.15.70:0/2984415383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80c1a6dd0 con 0x7ff80c102e30 2024-04-01T01:32:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff810bfd700 1 --2- 172.21.15.70:0/2984415383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c102e30 0x7ff80c074a80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00ea30 tx=0x7ff7fc00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff8027fc700 1 -- 172.21.15.70:0/2984415383 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc00cbd0 con 0x7ff80c102e30 2024-04-01T01:32:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff813662700 1 -- 172.21.15.70:0/2984415383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff80c1a70c0 con 0x7ff80c102e30 2024-04-01T01:32:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff8027fc700 1 -- 172.21.15.70:0/2984415383 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7fc0044d0 con 0x7ff80c102e30 2024-04-01T01:32:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff8027fc700 1 -- 172.21.15.70:0/2984415383 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc003db0 con 0x7ff80c102e30 2024-04-01T01:32:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.558+0000 7ff813662700 1 -- 172.21.15.70:0/2984415383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff80c1a7600 con 0x7ff80c102e30 2024-04-01T01:32:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.559+0000 7ff813662700 1 -- 172.21.15.70:0/2984415383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff80c062950 con 0x7ff80c102e30 2024-04-01T01:32:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.560+0000 7ff8027fc700 1 -- 172.21.15.70:0/2984415383 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7ff7fc004000 con 0x7ff80c102e30 2024-04-01T01:32:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.561+0000 7ff8027fc700 1 --2- 172.21.15.70:0/2984415383 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7f80a25c0 0x7ff7f80a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.561+0000 7ff8113fe700 1 --2- 172.21.15.70:0/2984415383 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7f80a25c0 0x7ff7f80a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.561+0000 7ff8027fc700 1 -- 172.21.15.70:0/2984415383 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff7fc0ceff0 con 0x7ff80c102e30 2024-04-01T01:32:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.561+0000 7ff8027fc700 1 --2- 172.21.15.70:0/2984415383 >> [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251] conn(0x7ff7f80ad280 0x7ff7f80af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.561+0000 7ff8027fc700 1 -- 172.21.15.70:0/2984415383 --> [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff7fc015e80 con 0x7ff7f80ad280 2024-04-01T01:32:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.561+0000 7ff8027fc700 1 -- 172.21.15.70:0/2984415383 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7ff7fc0cf440 con 0x7ff80c102e30 2024-04-01T01:32:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.561+0000 7ff8113fe700 1 --2- 172.21.15.70:0/2984415383 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7f80a25c0 0x7ff7f80a4a80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff808007be0 tx=0x7ff8080073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.562+0000 7ff811bff700 1 --2- 172.21.15.70:0/2984415383 >> [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251] conn(0x7ff7f80ad280 0x7ff7f80af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.562+0000 7ff811bff700 1 --2- 172.21.15.70:0/2984415383 >> [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251] conn(0x7ff7f80ad280 0x7ff7f80af6c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.563+0000 7ff8027fc700 1 -- 172.21.15.70:0/2984415383 <== osd.10 v2:172.21.15.81:6824/1556495251 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7fc015e80 con 0x7ff7f80ad280 2024-04-01T01:32:19.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.579+0000 7ff813662700 1 -- 172.21.15.70:0/2984415383 --> [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff80c062dd0 con 0x7ff7f80ad280 2024-04-01T01:32:19.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.579+0000 7ff8027fc700 1 -- 172.21.15.70:0/2984415383 <== osd.10 v2:172.21.15.81:6824/1556495251 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff80c062dd0 con 0x7ff7f80ad280 2024-04-01T01:32:19.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.579+0000 7ff813662700 1 -- 172.21.15.70:0/2984415383 >> [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251] conn(0x7ff7f80ad280 msgr2=0x7ff7f80af6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:19.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.579+0000 7ff813662700 1 --2- 172.21.15.70:0/2984415383 >> [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251] conn(0x7ff7f80ad280 0x7ff7f80af6c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:19.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.579+0000 7ff813662700 1 -- 172.21.15.70:0/2984415383 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7f80a25c0 msgr2=0x7ff7f80a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:19.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.579+0000 7ff813662700 1 --2- 172.21.15.70:0/2984415383 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7f80a25c0 0x7ff7f80a4a80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff808007be0 tx=0x7ff8080073d0 comp rx=0 tx=0).stop 2024-04-01T01:32:19.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.579+0000 7ff813662700 1 -- 172.21.15.70:0/2984415383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c102e30 msgr2=0x7ff80c074a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:19.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.579+0000 7ff813662700 1 --2- 172.21.15.70:0/2984415383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff80c102e30 0x7ff80c074a80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00ea30 tx=0x7ff7fc00ed40 comp rx=0 tx=0).stop 2024-04-01T01:32:19.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.579+0000 7ff811bff700 1 -- 172.21.15.70:0/2984415383 reap_dead start 2024-04-01T01:32:19.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.579+0000 7ff813662700 1 -- 172.21.15.70:0/2984415383 shutdown_connections 2024-04-01T01:32:19.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.579+0000 7ff813662700 1 -- 172.21.15.70:0/2984415383 >> 172.21.15.70:0/2984415383 conn(0x7ff80c0fdcf0 msgr2=0x7ff80c0ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:19.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.580+0000 7ff813662700 1 -- 172.21.15.70:0/2984415383 shutdown_connections 2024-04-01T01:32:19.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:18.580+0000 7ff813662700 1 -- 172.21.15.70:0/2984415383 wait complete. 2024-04-01T01:32:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:19 smithi189 ceph-mon[35753]: pgmap v319: 129 pgs: 129 active+clean; 646 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-01T01:32:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-01T01:32:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-01T01:32:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-04-01T01:32:19.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-04-01T01:32:19.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-04-01T01:32:19.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-04-01T01:32:19.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-04-01T01:32:19.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-04-01T01:32:19.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-04-01T01:32:19.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-04-01T01:32:19.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:19 smithi189 ceph-mon[35753]: osdmap e90: 12 total, 12 up, 12 in 2024-04-01T01:32:20.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:19 smithi081 ceph-mon[35142]: pgmap v319: 129 pgs: 129 active+clean; 646 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 9 op/s 2024-04-01T01:32:20.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-01T01:32:20.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-01T01:32:20.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-04-01T01:32:20.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-04-01T01:32:20.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-04-01T01:32:20.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-04-01T01:32:20.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-04-01T01:32:20.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-04-01T01:32:20.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-04-01T01:32:20.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-04-01T01:32:20.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:19 smithi081 ceph-mon[35142]: osdmap e90: 12 total, 12 up, 12 in 2024-04-01T01:32:20.546 INFO:teuthology.orchestra.run.smithi070.stdout:287762808868 2024-04-01T01:32:20.546 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.9 2024-04-01T01:32:20.657 INFO:teuthology.orchestra.run.smithi070.stdout:309237645343 2024-04-01T01:32:20.658 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.10 2024-04-01T01:32:20.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:20 smithi189 ceph-mon[35753]: osdmap e91: 12 total, 12 up, 12 in 2024-04-01T01:32:21.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:20 smithi070 ceph-mon[32179]: osdmap e91: 12 total, 12 up, 12 in 2024-04-01T01:32:21.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:20 smithi081 ceph-mon[35142]: osdmap e91: 12 total, 12 up, 12 in 2024-04-01T01:32:21.105 INFO:teuthology.orchestra.run.smithi070.stdout:77309411398 2024-04-01T01:32:21.105 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-04-01T01:32:21.495 INFO:teuthology.orchestra.run.smithi070.stdout:34359738447 2024-04-01T01:32:21.495 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-04-01T01:32:21.496 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:32:21.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:21 smithi070 ceph-mon[32179]: pgmap v322: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s; 14 B/s, 0 objects/s recovering 2024-04-01T01:32:21.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:21 smithi189 ceph-mon[35753]: pgmap v322: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s; 14 B/s, 0 objects/s recovering 2024-04-01T01:32:22.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:21 smithi081 ceph-mon[35142]: pgmap v322: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s; 14 B/s, 0 objects/s recovering 2024-04-01T01:32:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:32:22.682 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:32:22.731 INFO:teuthology.orchestra.run.smithi070.stdout:141733920828 2024-04-01T01:32:22.731 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-04-01T01:32:23.396 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:32:23.872 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:32:23.888 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:32:23.890 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:32:23.918 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:32:23.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:23 smithi189 ceph-mon[35753]: pgmap v323: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s; 155 B/s, 1 objects/s recovering 2024-04-01T01:32:24.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:23 smithi070 ceph-mon[32179]: pgmap v323: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s; 155 B/s, 1 objects/s recovering 2024-04-01T01:32:24.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:23 smithi081 ceph-mon[35142]: pgmap v323: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s; 155 B/s, 1 objects/s recovering 2024-04-01T01:32:24.510 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:32:24.514 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:32:25.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.320+0000 7f3759f39700 1 -- 172.21.15.70:0/1401684514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37541013d0 msgr2=0x7f37541017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.320+0000 7f3759f39700 1 --2- 172.21.15.70:0/1401684514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37541013d0 0x7f37541017b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f37440099c0 tx=0x7f3744009cd0 comp rx=0 tx=0).stop 2024-04-01T01:32:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.320+0000 7f3759f39700 1 -- 172.21.15.70:0/1401684514 shutdown_connections 2024-04-01T01:32:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.320+0000 7f3759f39700 1 --2- 172.21.15.70:0/1401684514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3754106420 0x7f375410dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.320+0000 7f3759f39700 1 --2- 172.21.15.70:0/1401684514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37541013d0 0x7f37541017b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.320+0000 7f3759f39700 1 --2- 172.21.15.70:0/1401684514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3754101d80 0x7f3754105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.320+0000 7f3759f39700 1 -- 172.21.15.70:0/1401684514 >> 172.21.15.70:0/1401684514 conn(0x7f3754077190 msgr2=0x7f37540775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.320+0000 7f3759f39700 1 -- 172.21.15.70:0/1401684514 shutdown_connections 2024-04-01T01:32:25.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.320+0000 7f3759f39700 1 -- 172.21.15.70:0/1401684514 wait complete. 2024-04-01T01:32:25.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.321+0000 7f3759f39700 1 Processor -- start 2024-04-01T01:32:25.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.321+0000 7f3759f39700 1 -- start start 2024-04-01T01:32:25.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.321+0000 7f3759f39700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37541013d0 0x7f375419d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:25.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.321+0000 7f3759f39700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3754101d80 0x7f375419dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:25.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.321+0000 7f3759f39700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3754106420 0x7f3754197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.321+0000 7f3759f39700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f37541112e0 con 0x7f3754101d80 2024-04-01T01:32:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.321+0000 7f3759f39700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3754111160 con 0x7f37541013d0 2024-04-01T01:32:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.321+0000 7f3759f39700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3754111460 con 0x7f3754106420 2024-04-01T01:32:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.321+0000 7f37537fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37541013d0 0x7f375419d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.321+0000 7f3752ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3754101d80 0x7f375419dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.322+0000 7f37537fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37541013d0 0x7f375419d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48296/0 (socket says 172.21.15.70:48296) 2024-04-01T01:32:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.322+0000 7f37537fe700 1 -- 172.21.15.70:0/2707344105 learned_addr learned my addr 172.21.15.70:0/2707344105 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.322+0000 7f3753fff700 1 --2- 172.21.15.70:0/2707344105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3754106420 0x7f3754197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.322+0000 7f37537fe700 1 -- 172.21.15.70:0/2707344105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3754106420 msgr2=0x7f3754197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.322+0000 7f37537fe700 1 --2- 172.21.15.70:0/2707344105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3754106420 0x7f3754197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.322+0000 7f37537fe700 1 -- 172.21.15.70:0/2707344105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3754101d80 msgr2=0x7f375419dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.322+0000 7f37537fe700 1 --2- 172.21.15.70:0/2707344105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3754101d80 0x7f375419dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.322+0000 7f37537fe700 1 -- 172.21.15.70:0/2707344105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3754197f20 con 0x7f37541013d0 2024-04-01T01:32:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.322+0000 7f3752ffd700 1 --2- 172.21.15.70:0/2707344105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3754101d80 0x7f375419dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.322+0000 7f37537fe700 1 --2- 172.21.15.70:0/2707344105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37541013d0 0x7f375419d570 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3744006010 tx=0x7f3744004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.322+0000 7f3750ff9700 1 -- 172.21.15.70:0/2707344105 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3744026070 con 0x7f37541013d0 2024-04-01T01:32:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.322+0000 7f3759f39700 1 -- 172.21.15.70:0/2707344105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37541981b0 con 0x7f37541013d0 2024-04-01T01:32:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.322+0000 7f3750ff9700 1 -- 172.21.15.70:0/2707344105 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3744004630 con 0x7f37541013d0 2024-04-01T01:32:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.322+0000 7f3750ff9700 1 -- 172.21.15.70:0/2707344105 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374400fb20 con 0x7f37541013d0 2024-04-01T01:32:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.322+0000 7f3759f39700 1 -- 172.21.15.70:0/2707344105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f375406a950 con 0x7f37541013d0 2024-04-01T01:32:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.323+0000 7f3759f39700 1 -- 172.21.15.70:0/2707344105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3754198510 con 0x7f37541013d0 2024-04-01T01:32:25.326 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:32:25.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.324+0000 7f3750ff9700 1 -- 172.21.15.70:0/2707344105 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f3744029070 con 0x7f37541013d0 2024-04-01T01:32:25.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.325+0000 7f3750ff9700 1 --2- 172.21.15.70:0/2707344105 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f373c0a2640 0x7f373c0a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:25.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.325+0000 7f3752ffd700 1 --2- 172.21.15.70:0/2707344105 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f373c0a2640 0x7f373c0a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:25.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.325+0000 7f3750ff9700 1 -- 172.21.15.70:0/2707344105 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f37440cf590 con 0x7f37541013d0 2024-04-01T01:32:25.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.326+0000 7f3752ffd700 1 --2- 172.21.15.70:0/2707344105 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f373c0a2640 0x7f373c0a4b00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3754198e00 tx=0x7f3748009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:25.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.333+0000 7f3750ff9700 1 -- 172.21.15.70:0/2707344105 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3744013070 con 0x7f37541013d0 2024-04-01T01:32:25.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.531+0000 7f3759f39700 1 -- 172.21.15.70:0/2707344105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f3754198b70 con 0x7f37541013d0 2024-04-01T01:32:25.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.532+0000 7f3750ff9700 1 -- 172.21.15.70:0/2707344105 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f374409c320 con 0x7f37541013d0 2024-04-01T01:32:25.532 INFO:teuthology.orchestra.run.smithi070.stdout:197568495668 2024-04-01T01:32:25.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.534+0000 7f3759f39700 1 -- 172.21.15.70:0/2707344105 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f373c0a2640 msgr2=0x7f373c0a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:25.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.534+0000 7f3759f39700 1 --2- 172.21.15.70:0/2707344105 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f373c0a2640 0x7f373c0a4b00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3754198e00 tx=0x7f3748009260 comp rx=0 tx=0).stop 2024-04-01T01:32:25.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.534+0000 7f3759f39700 1 -- 172.21.15.70:0/2707344105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37541013d0 msgr2=0x7f375419d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:25.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.534+0000 7f3759f39700 1 --2- 172.21.15.70:0/2707344105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37541013d0 0x7f375419d570 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3744006010 tx=0x7f3744004270 comp rx=0 tx=0).stop 2024-04-01T01:32:25.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.534+0000 7f3759f39700 1 -- 172.21.15.70:0/2707344105 shutdown_connections 2024-04-01T01:32:25.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.534+0000 7f3759f39700 1 --2- 172.21.15.70:0/2707344105 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f373c0a2640 0x7f373c0a4b00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:25.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.534+0000 7f3759f39700 1 --2- 172.21.15.70:0/2707344105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3754106420 0x7f3754197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:25.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.534+0000 7f3759f39700 1 --2- 172.21.15.70:0/2707344105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37541013d0 0x7f375419d570 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:25.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.534+0000 7f3759f39700 1 --2- 172.21.15.70:0/2707344105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3754101d80 0x7f375419dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:25.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.534+0000 7f3759f39700 1 -- 172.21.15.70:0/2707344105 >> 172.21.15.70:0/2707344105 conn(0x7f3754077190 msgr2=0x7f37541075f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:25.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.534+0000 7f3759f39700 1 -- 172.21.15.70:0/2707344105 shutdown_connections 2024-04-01T01:32:25.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:25.534+0000 7f3759f39700 1 -- 172.21.15.70:0/2707344105 wait complete. 2024-04-01T01:32:25.955 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:25 smithi070 ceph-mon[32179]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 155 B/s, 1 keys/s, 2 objects/s recovering 2024-04-01T01:32:25.956 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:25 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2707344105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-01T01:32:25.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:25 smithi189 ceph-mon[35753]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 155 B/s, 1 keys/s, 2 objects/s recovering 2024-04-01T01:32:25.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:25 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2707344105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-01T01:32:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:25 smithi081 ceph-mon[35142]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 155 B/s, 1 keys/s, 2 objects/s recovering 2024-04-01T01:32:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:25 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2707344105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-01T01:32:26.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.930+0000 7fbcfc950700 1 -- 172.21.15.70:0/1546883197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf806a890 msgr2=0x7fbcf810f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:26.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.930+0000 7fbcfc950700 1 --2- 172.21.15.70:0/1546883197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf806a890 0x7fbcf810f0d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fbcf000b0d0 tx=0x7fbcf000b3e0 comp rx=0 tx=0).stop 2024-04-01T01:32:26.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.930+0000 7fbcfc950700 1 -- 172.21.15.70:0/1546883197 shutdown_connections 2024-04-01T01:32:26.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.930+0000 7fbcfc950700 1 --2- 172.21.15.70:0/1546883197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf806a890 0x7fbcf810f0d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:26.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.930+0000 7fbcfc950700 1 --2- 172.21.15.70:0/1546883197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf8108290 0x7fbcf8108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:26.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.930+0000 7fbcfc950700 1 --2- 172.21.15.70:0/1546883197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcf8069ef0 0x7fbcf806a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:26.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.930+0000 7fbcfc950700 1 -- 172.21.15.70:0/1546883197 >> 172.21.15.70:0/1546883197 conn(0x7fbcf8078070 msgr2=0x7fbcf8078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:26.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.931+0000 7fbcfc950700 1 -- 172.21.15.70:0/1546883197 shutdown_connections 2024-04-01T01:32:26.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.931+0000 7fbcfc950700 1 -- 172.21.15.70:0/1546883197 wait complete. 2024-04-01T01:32:26.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.931+0000 7fbcfc950700 1 Processor -- start 2024-04-01T01:32:26.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.931+0000 7fbcfc950700 1 -- start start 2024-04-01T01:32:26.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.931+0000 7fbcfc950700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf8069ef0 0x7fbcf819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:26.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.931+0000 7fbcfc950700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcf806a890 0x7fbcf819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:26.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.931+0000 7fbcfc950700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf8108290 0x7fbcf8198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:26.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcfc950700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf81123c0 con 0x7fbcf8069ef0 2024-04-01T01:32:26.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcfc950700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf8112240 con 0x7fbcf8108290 2024-04-01T01:32:26.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcfc950700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf8112540 con 0x7fbcf806a890 2024-04-01T01:32:26.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcf659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf8069ef0 0x7fbcf819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:26.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcf659c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf8069ef0 0x7fbcf819e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53456/0 (socket says 172.21.15.70:53456) 2024-04-01T01:32:26.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcf659c700 1 -- 172.21.15.70:0/2481652543 learned_addr learned my addr 172.21.15.70:0/2481652543 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:26.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcf5d9b700 1 --2- 172.21.15.70:0/2481652543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcf806a890 0x7fbcf819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:26.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcf6d9d700 1 --2- 172.21.15.70:0/2481652543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf8108290 0x7fbcf8198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcf659c700 1 -- 172.21.15.70:0/2481652543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcf806a890 msgr2=0x7fbcf819eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcf659c700 1 --2- 172.21.15.70:0/2481652543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcf806a890 0x7fbcf819eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcf659c700 1 -- 172.21.15.70:0/2481652543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf8108290 msgr2=0x7fbcf8198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcf659c700 1 --2- 172.21.15.70:0/2481652543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf8108290 0x7fbcf8198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcf659c700 1 -- 172.21.15.70:0/2481652543 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf8199060 con 0x7fbcf8069ef0 2024-04-01T01:32:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcf5d9b700 1 --2- 172.21.15.70:0/2481652543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcf806a890 0x7fbcf819eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcf6d9d700 1 --2- 172.21.15.70:0/2481652543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf8108290 0x7fbcf8198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcf659c700 1 --2- 172.21.15.70:0/2481652543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf8069ef0 0x7fbcf819e660 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbce800e9a0 tx=0x7fbce800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbce77fe700 1 -- 172.21.15.70:0/2481652543 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce800ca10 con 0x7fbcf8069ef0 2024-04-01T01:32:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbce77fe700 1 -- 172.21.15.70:0/2481652543 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbce800cbb0 con 0x7fbcf8069ef0 2024-04-01T01:32:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcfc950700 1 -- 172.21.15.70:0/2481652543 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcf81992f0 con 0x7fbcf8069ef0 2024-04-01T01:32:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbce77fe700 1 -- 172.21.15.70:0/2481652543 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce8018680 con 0x7fbcf8069ef0 2024-04-01T01:32:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.932+0000 7fbcfc950700 1 -- 172.21.15.70:0/2481652543 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcf8101310 con 0x7fbcf8069ef0 2024-04-01T01:32:26.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.933+0000 7fbcfc950700 1 -- 172.21.15.70:0/2481652543 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcf8199660 con 0x7fbcf8069ef0 2024-04-01T01:32:26.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.935+0000 7fbce77fe700 1 -- 172.21.15.70:0/2481652543 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fbce8010430 con 0x7fbcf8069ef0 2024-04-01T01:32:26.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.935+0000 7fbce77fe700 1 --2- 172.21.15.70:0/2481652543 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbce00a25c0 0x7fbce00a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:26.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.935+0000 7fbcf5d9b700 1 --2- 172.21.15.70:0/2481652543 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbce00a25c0 0x7fbce00a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:26.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.935+0000 7fbce77fe700 1 -- 172.21.15.70:0/2481652543 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbce80cef40 con 0x7fbcf8069ef0 2024-04-01T01:32:26.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.936+0000 7fbcf5d9b700 1 --2- 172.21.15.70:0/2481652543 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbce00a25c0 0x7fbce00a4a80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8199ef0 tx=0x7fbcec009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:26.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:26.939+0000 7fbce77fe700 1 -- 172.21.15.70:0/2481652543 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbcf8199660 con 0x7fbcf8069ef0 2024-04-01T01:32:27.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:27.144+0000 7fbcfc950700 1 -- 172.21.15.70:0/2481652543 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fbcf8199c60 con 0x7fbcf8069ef0 2024-04-01T01:32:27.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:27.145+0000 7fbce77fe700 1 -- 172.21.15.70:0/2481652543 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbce809bc40 con 0x7fbcf8069ef0 2024-04-01T01:32:27.145 INFO:teuthology.orchestra.run.smithi070.stdout:219043332144 2024-04-01T01:32:27.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:27.147+0000 7fbcfc950700 1 -- 172.21.15.70:0/2481652543 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbce00a25c0 msgr2=0x7fbce00a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:27.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:27.147+0000 7fbcfc950700 1 --2- 172.21.15.70:0/2481652543 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbce00a25c0 0x7fbce00a4a80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8199ef0 tx=0x7fbcec009260 comp rx=0 tx=0).stop 2024-04-01T01:32:27.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:27.147+0000 7fbcfc950700 1 -- 172.21.15.70:0/2481652543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf8069ef0 msgr2=0x7fbcf819e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:27.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:27.147+0000 7fbcfc950700 1 --2- 172.21.15.70:0/2481652543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf8069ef0 0x7fbcf819e660 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbce800e9a0 tx=0x7fbce800ecb0 comp rx=0 tx=0).stop 2024-04-01T01:32:27.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:27.147+0000 7fbcfc950700 1 -- 172.21.15.70:0/2481652543 shutdown_connections 2024-04-01T01:32:27.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:27.147+0000 7fbcfc950700 1 --2- 172.21.15.70:0/2481652543 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbce00a25c0 0x7fbce00a4a80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:27.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:27.147+0000 7fbcfc950700 1 --2- 172.21.15.70:0/2481652543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcf8108290 0x7fbcf8198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:27.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:27.147+0000 7fbcfc950700 1 --2- 172.21.15.70:0/2481652543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbcf8069ef0 0x7fbcf819e660 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:27.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:27.147+0000 7fbcfc950700 1 --2- 172.21.15.70:0/2481652543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcf806a890 0x7fbcf819eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:27.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:27.147+0000 7fbcfc950700 1 -- 172.21.15.70:0/2481652543 >> 172.21.15.70:0/2481652543 conn(0x7fbcf8078070 msgr2=0x7fbcf80ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:27.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:27.147+0000 7fbcfc950700 1 -- 172.21.15.70:0/2481652543 shutdown_connections 2024-04-01T01:32:27.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:27.147+0000 7fbcfc950700 1 -- 172.21.15.70:0/2481652543 wait complete. 2024-04-01T01:32:27.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:27 smithi070 ceph-mon[32179]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s; 155 B/s, 1 keys/s, 2 objects/s recovering 2024-04-01T01:32:27.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:27 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2481652543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-01T01:32:27.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:27 smithi189 ceph-mon[35753]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s; 155 B/s, 1 keys/s, 2 objects/s recovering 2024-04-01T01:32:27.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:27 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2481652543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-01T01:32:28.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:27 smithi081 ceph-mon[35142]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s; 155 B/s, 1 keys/s, 2 objects/s recovering 2024-04-01T01:32:28.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:27 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2481652543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-01T01:32:28.829 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332140 got 219043332144 for osd.7 2024-04-01T01:32:28.829 DEBUG:teuthology.parallel:result is None 2024-04-01T01:32:30.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:29 smithi189 ceph-mon[35753]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s; 125 B/s, 1 keys/s, 1 objects/s recovering 2024-04-01T01:32:30.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:29 smithi070 ceph-mon[32179]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s; 125 B/s, 1 keys/s, 1 objects/s recovering 2024-04-01T01:32:30.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:29 smithi081 ceph-mon[35142]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s; 125 B/s, 1 keys/s, 1 objects/s recovering 2024-04-01T01:32:30.414 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495665 got 197568495668 for osd.6 2024-04-01T01:32:30.415 DEBUG:teuthology.parallel:result is None 2024-04-01T01:32:31.929 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:31 smithi070 ceph-mon[32179]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s; 113 B/s, 0 keys/s, 1 objects/s recovering 2024-04-01T01:32:31.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:31 smithi189 ceph-mon[35753]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s; 113 B/s, 0 keys/s, 1 objects/s recovering 2024-04-01T01:32:32.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:31 smithi081 ceph-mon[35142]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s; 113 B/s, 0 keys/s, 1 objects/s recovering 2024-04-01T01:32:33.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:33 smithi189 ceph-mon[35753]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 93 B/s, 0 keys/s, 1 objects/s recovering 2024-04-01T01:32:34.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:33 smithi070 ceph-mon[32179]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 93 B/s, 0 keys/s, 1 objects/s recovering 2024-04-01T01:32:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:33 smithi081 ceph-mon[35142]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 93 B/s, 0 keys/s, 1 objects/s recovering 2024-04-01T01:32:35.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.157+0000 7fd930c72700 1 -- 172.21.15.70:0/3391062070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd92c106170 msgr2=0x7fd92c106550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.157+0000 7fd930c72700 1 --2- 172.21.15.70:0/3391062070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd92c106170 0x7fd92c106550 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd91c009990 tx=0x7fd91c009ca0 comp rx=0 tx=0).stop 2024-04-01T01:32:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.157+0000 7fd930c72700 1 -- 172.21.15.70:0/3391062070 shutdown_connections 2024-04-01T01:32:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.157+0000 7fd930c72700 1 --2- 172.21.15.70:0/3391062070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd92c102640 0x7fd92c10e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.157+0000 7fd930c72700 1 --2- 172.21.15.70:0/3391062070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd92c106170 0x7fd92c106550 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.157+0000 7fd930c72700 1 --2- 172.21.15.70:0/3391062070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd92c101c80 0x7fd92c102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.157+0000 7fd930c72700 1 -- 172.21.15.70:0/3391062070 >> 172.21.15.70:0/3391062070 conn(0x7fd92c076470 msgr2=0x7fd92c076880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.157+0000 7fd930c72700 1 -- 172.21.15.70:0/3391062070 shutdown_connections 2024-04-01T01:32:35.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.157+0000 7fd930c72700 1 -- 172.21.15.70:0/3391062070 wait complete. 2024-04-01T01:32:35.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd930c72700 1 Processor -- start 2024-04-01T01:32:35.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd930c72700 1 -- start start 2024-04-01T01:32:35.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd930c72700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd92c101c80 0x7fd92c1975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:35.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd930c72700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd92c102640 0x7fd92c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:35.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd930c72700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd92c106170 0x7fd92c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:35.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd930c72700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c111370 con 0x7fd92c101c80 2024-04-01T01:32:35.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd930c72700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c1111f0 con 0x7fd92c102640 2024-04-01T01:32:35.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd930c72700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd92c1114f0 con 0x7fd92c106170 2024-04-01T01:32:35.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd92a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd92c101c80 0x7fd92c1975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:35.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd92a59c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd92c101c80 0x7fd92c1975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53476/0 (socket says 172.21.15.70:53476) 2024-04-01T01:32:35.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd92a59c700 1 -- 172.21.15.70:0/1543805277 learned_addr learned my addr 172.21.15.70:0/1543805277 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:35.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd92ad9d700 1 --2- 172.21.15.70:0/1543805277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd92c106170 0x7fd92c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:35.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd921d9b700 1 --2- 172.21.15.70:0/1543805277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd92c102640 0x7fd92c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:35.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd92a59c700 1 -- 172.21.15.70:0/1543805277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd92c106170 msgr2=0x7fd92c198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:35.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd92a59c700 1 --2- 172.21.15.70:0/1543805277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd92c106170 0x7fd92c198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd92a59c700 1 -- 172.21.15.70:0/1543805277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd92c102640 msgr2=0x7fd92c197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.158+0000 7fd92a59c700 1 --2- 172.21.15.70:0/1543805277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd92c102640 0x7fd92c197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.159+0000 7fd92a59c700 1 -- 172.21.15.70:0/1543805277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd92c1a6dd0 con 0x7fd92c101c80 2024-04-01T01:32:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.159+0000 7fd92a59c700 1 --2- 172.21.15.70:0/1543805277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd92c101c80 0x7fd92c1975c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd91c005910 tx=0x7fd91c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.159+0000 7fd923fff700 1 -- 172.21.15.70:0/1543805277 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c027070 con 0x7fd92c101c80 2024-04-01T01:32:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.159+0000 7fd923fff700 1 -- 172.21.15.70:0/1543805277 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd91c022470 con 0x7fd92c101c80 2024-04-01T01:32:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.159+0000 7fd930c72700 1 -- 172.21.15.70:0/1543805277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd92c1a7060 con 0x7fd92c101c80 2024-04-01T01:32:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.159+0000 7fd923fff700 1 -- 172.21.15.70:0/1543805277 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c00b610 con 0x7fd92c101c80 2024-04-01T01:32:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.159+0000 7fd930c72700 1 -- 172.21.15.70:0/1543805277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd92c1a74b0 con 0x7fd92c101c80 2024-04-01T01:32:35.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.160+0000 7fd930c72700 1 -- 172.21.15.70:0/1543805277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd92c1096d0 con 0x7fd92c101c80 2024-04-01T01:32:35.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.161+0000 7fd923fff700 1 -- 172.21.15.70:0/1543805277 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fd91c02a070 con 0x7fd92c101c80 2024-04-01T01:32:35.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.161+0000 7fd923fff700 1 --2- 172.21.15.70:0/1543805277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9180a25c0 0x7fd9180a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:35.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.162+0000 7fd921d9b700 1 --2- 172.21.15.70:0/1543805277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9180a25c0 0x7fd9180a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:35.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.162+0000 7fd923fff700 1 -- 172.21.15.70:0/1543805277 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd91c0cf830 con 0x7fd92c101c80 2024-04-01T01:32:35.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.162+0000 7fd921d9b700 1 --2- 172.21.15.70:0/1543805277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9180a25c0 0x7fd9180a4a80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd92c199720 tx=0x7fd914009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:35.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.166+0000 7fd923fff700 1 -- 172.21.15.70:0/1543805277 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd91c013070 con 0x7fd92c101c80 2024-04-01T01:32:35.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.370+0000 7fd930c72700 1 -- 172.21.15.70:0/1543805277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd92c101110 con 0x7fd92c101c80 2024-04-01T01:32:35.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.370+0000 7fd923fff700 1 -- 172.21.15.70:0/1543805277 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd91c09c5c0 con 0x7fd92c101c80 2024-04-01T01:32:35.371 INFO:teuthology.orchestra.run.smithi070.stdout:77309411402 2024-04-01T01:32:35.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.372+0000 7fd930c72700 1 -- 172.21.15.70:0/1543805277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9180a25c0 msgr2=0x7fd9180a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:35.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.372+0000 7fd930c72700 1 --2- 172.21.15.70:0/1543805277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9180a25c0 0x7fd9180a4a80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd92c199720 tx=0x7fd914009260 comp rx=0 tx=0).stop 2024-04-01T01:32:35.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.372+0000 7fd930c72700 1 -- 172.21.15.70:0/1543805277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd92c101c80 msgr2=0x7fd92c1975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:35.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.372+0000 7fd930c72700 1 --2- 172.21.15.70:0/1543805277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd92c101c80 0x7fd92c1975c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd91c005910 tx=0x7fd91c0047b0 comp rx=0 tx=0).stop 2024-04-01T01:32:35.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.373+0000 7fd930c72700 1 -- 172.21.15.70:0/1543805277 shutdown_connections 2024-04-01T01:32:35.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.373+0000 7fd930c72700 1 --2- 172.21.15.70:0/1543805277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9180a25c0 0x7fd9180a4a80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:35.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.373+0000 7fd930c72700 1 --2- 172.21.15.70:0/1543805277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd92c106170 0x7fd92c198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:35.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.373+0000 7fd930c72700 1 --2- 172.21.15.70:0/1543805277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd92c101c80 0x7fd92c1975c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:35.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.373+0000 7fd930c72700 1 --2- 172.21.15.70:0/1543805277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd92c102640 0x7fd92c197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:35.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.373+0000 7fd930c72700 1 -- 172.21.15.70:0/1543805277 >> 172.21.15.70:0/1543805277 conn(0x7fd92c076470 msgr2=0x7fd92c0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:35.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.373+0000 7fd930c72700 1 -- 172.21.15.70:0/1543805277 shutdown_connections 2024-04-01T01:32:35.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:35.373+0000 7fd930c72700 1 -- 172.21.15.70:0/1543805277 wait complete. 2024-04-01T01:32:35.978 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:32:35.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:35 smithi189 ceph-mon[35753]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-01T01:32:35.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:35 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1543805277' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-01T01:32:36.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:35 smithi070 ceph-mon[32179]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-01T01:32:36.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:35 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1543805277' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-01T01:32:36.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:35 smithi081 ceph-mon[35142]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-01T01:32:36.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:35 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1543805277' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-01T01:32:36.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.982+0000 7fad30c8b700 1 -- 172.21.15.70:0/94301636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad2c108270 msgr2=0x7fad2c108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:36.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.982+0000 7fad30c8b700 1 --2- 172.21.15.70:0/94301636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad2c108270 0x7fad2c108650 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fad1c0099c0 tx=0x7fad1c009cd0 comp rx=0 tx=0).stop 2024-04-01T01:32:36.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.982+0000 7fad30c8b700 1 -- 172.21.15.70:0/94301636 shutdown_connections 2024-04-01T01:32:36.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.982+0000 7fad30c8b700 1 --2- 172.21.15.70:0/94301636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad2c102c50 0x7fad2c10f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:36.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.982+0000 7fad30c8b700 1 --2- 172.21.15.70:0/94301636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad2c108270 0x7fad2c108650 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:36.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.982+0000 7fad30c8b700 1 --2- 172.21.15.70:0/94301636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad2c102290 0x7fad2c102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:36.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.982+0000 7fad30c8b700 1 -- 172.21.15.70:0/94301636 >> 172.21.15.70:0/94301636 conn(0x7fad2c079370 msgr2=0x7fad2c079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:36.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.982+0000 7fad30c8b700 1 -- 172.21.15.70:0/94301636 shutdown_connections 2024-04-01T01:32:36.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.982+0000 7fad30c8b700 1 -- 172.21.15.70:0/94301636 wait complete. 2024-04-01T01:32:36.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad30c8b700 1 Processor -- start 2024-04-01T01:32:36.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad30c8b700 1 -- start start 2024-04-01T01:32:36.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad30c8b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad2c102290 0x7fad2c074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:36.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad30c8b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad2c102c50 0x7fad2c074a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:36.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad30c8b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad2c108270 0x7fad2c074fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:36.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad30c8b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1123d0 con 0x7fad2c102c50 2024-04-01T01:32:36.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad30c8b700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c112250 con 0x7fad2c102290 2024-04-01T01:32:36.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad30c8b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c112550 con 0x7fad2c108270 2024-04-01T01:32:36.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad29d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad2c102c50 0x7fad2c074a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:36.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad29d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad2c102c50 0x7fad2c074a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45910/0 (socket says 172.21.15.70:45910) 2024-04-01T01:32:36.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad29d9b700 1 -- 172.21.15.70:0/3167160928 learned_addr learned my addr 172.21.15.70:0/3167160928 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:36.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad2ad9d700 1 --2- 172.21.15.70:0/3167160928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad2c108270 0x7fad2c074fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:36.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad2a59c700 1 --2- 172.21.15.70:0/3167160928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad2c102290 0x7fad2c074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:36.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad29d9b700 1 -- 172.21.15.70:0/3167160928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad2c108270 msgr2=0x7fad2c074fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:36.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad29d9b700 1 --2- 172.21.15.70:0/3167160928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad2c108270 0x7fad2c074fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:36.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad29d9b700 1 -- 172.21.15.70:0/3167160928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad2c102290 msgr2=0x7fad2c074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:36.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad29d9b700 1 --2- 172.21.15.70:0/3167160928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad2c102290 0x7fad2c074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:36.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad29d9b700 1 -- 172.21.15.70:0/3167160928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2c1a6df0 con 0x7fad2c102c50 2024-04-01T01:32:36.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.983+0000 7fad2a59c700 1 --2- 172.21.15.70:0/3167160928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad2c102290 0x7fad2c074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:36.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.984+0000 7fad29d9b700 1 --2- 172.21.15.70:0/3167160928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad2c102c50 0x7fad2c074a80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fad2000e9c0 tx=0x7fad2000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:36.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.984+0000 7fad1b7fe700 1 -- 172.21.15.70:0/3167160928 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2000cbd0 con 0x7fad2c102c50 2024-04-01T01:32:36.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.984+0000 7fad1b7fe700 1 -- 172.21.15.70:0/3167160928 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad2000ee50 con 0x7fad2c102c50 2024-04-01T01:32:36.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.984+0000 7fad30c8b700 1 -- 172.21.15.70:0/3167160928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2c1a70e0 con 0x7fad2c102c50 2024-04-01T01:32:36.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.984+0000 7fad1b7fe700 1 -- 172.21.15.70:0/3167160928 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20005550 con 0x7fad2c102c50 2024-04-01T01:32:36.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.984+0000 7fad30c8b700 1 -- 172.21.15.70:0/3167160928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad2c1a75d0 con 0x7fad2c102c50 2024-04-01T01:32:36.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.985+0000 7fad30c8b700 1 -- 172.21.15.70:0/3167160928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad2c068f00 con 0x7fad2c102c50 2024-04-01T01:32:36.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.986+0000 7fad1b7fe700 1 -- 172.21.15.70:0/3167160928 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fad20004750 con 0x7fad2c102c50 2024-04-01T01:32:36.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.986+0000 7fad1b7fe700 1 --2- 172.21.15.70:0/3167160928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fad140a25c0 0x7fad140a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:36.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.987+0000 7fad2a59c700 1 --2- 172.21.15.70:0/3167160928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fad140a25c0 0x7fad140a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:36.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.987+0000 7fad1b7fe700 1 -- 172.21.15.70:0/3167160928 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fad200cf090 con 0x7fad2c102c50 2024-04-01T01:32:36.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.987+0000 7fad2a59c700 1 --2- 172.21.15.70:0/3167160928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fad140a25c0 0x7fad140a4a80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fad1c003760 tx=0x7fad1c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:36.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:36.990+0000 7fad1b7fe700 1 -- 172.21.15.70:0/3167160928 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fad20053280 con 0x7fad2c102c50 2024-04-01T01:32:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:36 smithi081 ceph-mon[35142]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:32:37.090 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:36 smithi070 ceph-mon[32179]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:32:37.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.192+0000 7fb824c53700 1 -- 172.21.15.70:0/1149092814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb820101140 msgr2=0x7fb8201015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:37.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.192+0000 7fb824c53700 1 --2- 172.21.15.70:0/1149092814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb820101140 0x7fb8201015a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb810009990 tx=0x7fb810009ca0 comp rx=0 tx=0).stop 2024-04-01T01:32:37.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.192+0000 7fb824c53700 1 -- 172.21.15.70:0/1149092814 shutdown_connections 2024-04-01T01:32:37.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.192+0000 7fb824c53700 1 --2- 172.21.15.70:0/1149092814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb820101ae0 0x7fb82010dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:37.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.192+0000 7fb824c53700 1 --2- 172.21.15.70:0/1149092814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb820107180 0x7fb820107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:37.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.192+0000 7fb824c53700 1 --2- 172.21.15.70:0/1149092814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb820101140 0x7fb8201015a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:37.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.192+0000 7fb824c53700 1 -- 172.21.15.70:0/1149092814 >> 172.21.15.70:0/1149092814 conn(0x7fb820078070 msgr2=0x7fb820078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:37.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.192+0000 7fb824c53700 1 -- 172.21.15.70:0/1149092814 shutdown_connections 2024-04-01T01:32:37.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.192+0000 7fb824c53700 1 -- 172.21.15.70:0/1149092814 wait complete. 2024-04-01T01:32:37.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.193+0000 7fb824c53700 1 Processor -- start 2024-04-01T01:32:37.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.193+0000 7fb824c53700 1 -- start start 2024-04-01T01:32:37.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.193+0000 7fb824c53700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb820101140 0x7fb820197550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:37.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.193+0000 7fb824c53700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb820101ae0 0x7fb820197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:37.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.193+0000 7fb824c53700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb820107180 0x7fb820197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:37.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.193+0000 7fb824c53700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8201112e0 con 0x7fb820101ae0 2024-04-01T01:32:37.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.193+0000 7fb824c53700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb820111160 con 0x7fb820107180 2024-04-01T01:32:37.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.193+0000 7fb824c53700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb820111460 con 0x7fb820101140 2024-04-01T01:32:37.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.193+0000 7fb81dd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb820101ae0 0x7fb820197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:37.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.193+0000 7fb81dd9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb820101ae0 0x7fb820197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45918/0 (socket says 172.21.15.70:45918) 2024-04-01T01:32:37.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.193+0000 7fb81dd9b700 1 -- 172.21.15.70:0/3527905134 learned_addr learned my addr 172.21.15.70:0/3527905134 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:37.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.193+0000 7fb81e59c700 1 --2- 172.21.15.70:0/3527905134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb820101140 0x7fb820197550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:37.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.193+0000 7fb81ed9d700 1 --2- 172.21.15.70:0/3527905134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb820107180 0x7fb820197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:37.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.193+0000 7fb81e59c700 1 -- 172.21.15.70:0/3527905134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb820107180 msgr2=0x7fb820197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:37.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.194+0000 7fb81e59c700 1 --2- 172.21.15.70:0/3527905134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb820107180 0x7fb820197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:37.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.194+0000 7fb81e59c700 1 -- 172.21.15.70:0/3527905134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb820101ae0 msgr2=0x7fb820197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:37.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.194+0000 7fb81e59c700 1 --2- 172.21.15.70:0/3527905134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb820101ae0 0x7fb820197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:37.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.194+0000 7fb81e59c700 1 -- 172.21.15.70:0/3527905134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8201a6dc0 con 0x7fb820101140 2024-04-01T01:32:37.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.194+0000 7fb81dd9b700 1 --2- 172.21.15.70:0/3527905134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb820101ae0 0x7fb820197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:37.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.194+0000 7fb81e59c700 1 --2- 172.21.15.70:0/3527905134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb820101140 0x7fb820197550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb81400e9f0 tx=0x7fb81400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:37.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.194+0000 7fb80f7fe700 1 -- 172.21.15.70:0/3527905134 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81400cb70 con 0x7fb820101140 2024-04-01T01:32:37.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.194+0000 7fb80f7fe700 1 -- 172.21.15.70:0/3527905134 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb81400ee90 con 0x7fb820101140 2024-04-01T01:32:37.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.194+0000 7fb824c53700 1 -- 172.21.15.70:0/3527905134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8201a70b0 con 0x7fb820101140 2024-04-01T01:32:37.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.194+0000 7fb80f7fe700 1 -- 172.21.15.70:0/3527905134 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb814003d40 con 0x7fb820101140 2024-04-01T01:32:37.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.194+0000 7fb824c53700 1 -- 172.21.15.70:0/3527905134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8201a7640 con 0x7fb820101140 2024-04-01T01:32:37.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.195+0000 7fb824c53700 1 -- 172.21.15.70:0/3527905134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb820068f00 con 0x7fb820101140 2024-04-01T01:32:37.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.196+0000 7fb80f7fe700 1 -- 172.21.15.70:0/3527905134 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fb814003ee0 con 0x7fb820101140 2024-04-01T01:32:37.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.196+0000 7fb80f7fe700 1 --2- 172.21.15.70:0/3527905134 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8080a2640 0x7fb8080a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:37.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.197+0000 7fb81dd9b700 1 --2- 172.21.15.70:0/3527905134 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8080a2640 0x7fb8080a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:37.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.197+0000 7fb80f7fe700 1 -- 172.21.15.70:0/3527905134 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb8140cec30 con 0x7fb820101140 2024-04-01T01:32:37.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.197+0000 7fb81dd9b700 1 --2- 172.21.15.70:0/3527905134 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8080a2640 0x7fb8080a4b00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb8201996b0 tx=0x7fb81000b280 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:37.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.201+0000 7fad30c8b700 1 -- 172.21.15.70:0/3167160928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fad2c02db30 con 0x7fad2c102c50 2024-04-01T01:32:37.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.201+0000 7fad1b7fe700 1 -- 172.21.15.70:0/3167160928 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fad2009bd90 con 0x7fad2c102c50 2024-04-01T01:32:37.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.201+0000 7fb80f7fe700 1 -- 172.21.15.70:0/3527905134 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb81409b910 con 0x7fb820101140 2024-04-01T01:32:37.202 INFO:teuthology.orchestra.run.smithi070.stdout:107374182468 2024-04-01T01:32:37.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.203+0000 7fad30c8b700 1 -- 172.21.15.70:0/3167160928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fad140a25c0 msgr2=0x7fad140a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:37.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.203+0000 7fad30c8b700 1 --2- 172.21.15.70:0/3167160928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fad140a25c0 0x7fad140a4a80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fad1c003760 tx=0x7fad1c005fb0 comp rx=0 tx=0).stop 2024-04-01T01:32:37.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.203+0000 7fad30c8b700 1 -- 172.21.15.70:0/3167160928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad2c102c50 msgr2=0x7fad2c074a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:37.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.203+0000 7fad30c8b700 1 --2- 172.21.15.70:0/3167160928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad2c102c50 0x7fad2c074a80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fad2000e9c0 tx=0x7fad2000ecd0 comp rx=0 tx=0).stop 2024-04-01T01:32:37.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.203+0000 7fad30c8b700 1 -- 172.21.15.70:0/3167160928 shutdown_connections 2024-04-01T01:32:37.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.203+0000 7fad30c8b700 1 --2- 172.21.15.70:0/3167160928 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fad140a25c0 0x7fad140a4a80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:37.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.203+0000 7fad30c8b700 1 --2- 172.21.15.70:0/3167160928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad2c108270 0x7fad2c074fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:37.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.203+0000 7fad30c8b700 1 --2- 172.21.15.70:0/3167160928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad2c102290 0x7fad2c074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:37.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.203+0000 7fad30c8b700 1 --2- 172.21.15.70:0/3167160928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad2c102c50 0x7fad2c074a80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:37.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.203+0000 7fad30c8b700 1 -- 172.21.15.70:0/3167160928 >> 172.21.15.70:0/3167160928 conn(0x7fad2c079370 msgr2=0x7fad2c100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:37.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.203+0000 7fad30c8b700 1 -- 172.21.15.70:0/3167160928 shutdown_connections 2024-04-01T01:32:37.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.203+0000 7fad30c8b700 1 -- 172.21.15.70:0/3167160928 wait complete. 2024-04-01T01:32:37.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:36 smithi189 ceph-mon[35753]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:32:37.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.406+0000 7fb824c53700 1 -- 172.21.15.70:0/3527905134 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fb82002db30 con 0x7fb820101140 2024-04-01T01:32:37.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.406+0000 7fb80f7fe700 1 -- 172.21.15.70:0/3527905134 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fb81409b210 con 0x7fb820101140 2024-04-01T01:32:37.407 INFO:teuthology.orchestra.run.smithi070.stdout:343597383710 2024-04-01T01:32:37.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.408+0000 7fb824c53700 1 -- 172.21.15.70:0/3527905134 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8080a2640 msgr2=0x7fb8080a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:37.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.408+0000 7fb824c53700 1 --2- 172.21.15.70:0/3527905134 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8080a2640 0x7fb8080a4b00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb8201996b0 tx=0x7fb81000b280 comp rx=0 tx=0).stop 2024-04-01T01:32:37.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.408+0000 7fb824c53700 1 -- 172.21.15.70:0/3527905134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb820101140 msgr2=0x7fb820197550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:37.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.408+0000 7fb824c53700 1 --2- 172.21.15.70:0/3527905134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb820101140 0x7fb820197550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb81400e9f0 tx=0x7fb81400ed00 comp rx=0 tx=0).stop 2024-04-01T01:32:37.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.408+0000 7fb824c53700 1 -- 172.21.15.70:0/3527905134 shutdown_connections 2024-04-01T01:32:37.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.408+0000 7fb824c53700 1 --2- 172.21.15.70:0/3527905134 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8080a2640 0x7fb8080a4b00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:37.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.408+0000 7fb824c53700 1 --2- 172.21.15.70:0/3527905134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb820107180 0x7fb820197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:37.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.408+0000 7fb824c53700 1 --2- 172.21.15.70:0/3527905134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb820101140 0x7fb820197550 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:37.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.408+0000 7fb824c53700 1 --2- 172.21.15.70:0/3527905134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb820101ae0 0x7fb820197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:37.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.408+0000 7fb824c53700 1 -- 172.21.15.70:0/3527905134 >> 172.21.15.70:0/3527905134 conn(0x7fb820078070 msgr2=0x7fb8200ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:37.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.408+0000 7fb824c53700 1 -- 172.21.15.70:0/3527905134 shutdown_connections 2024-04-01T01:32:37.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:37.408+0000 7fb824c53700 1 -- 172.21.15.70:0/3527905134 wait complete. 2024-04-01T01:32:38.039 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:37 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3167160928' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-01T01:32:38.039 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:37 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3527905134' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-01T01:32:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:37 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3167160928' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-01T01:32:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:37 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3527905134' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-01T01:32:38.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.140+0000 7fc5b142e700 1 -- 172.21.15.70:0/3882541244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ac100830 msgr2=0x7fc5ac100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:38.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.140+0000 7fc5b142e700 1 --2- 172.21.15.70:0/3882541244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ac100830 0x7fc5ac100cb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc594009990 tx=0x7fc594009ca0 comp rx=0 tx=0).stop 2024-04-01T01:32:38.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.140+0000 7fc5b142e700 1 -- 172.21.15.70:0/3882541244 shutdown_connections 2024-04-01T01:32:38.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.140+0000 7fc5b142e700 1 --2- 172.21.15.70:0/3882541244 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5ac1011f0 0x7fc5ac10f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:38.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.140+0000 7fc5b142e700 1 --2- 172.21.15.70:0/3882541244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5ac1072b0 0x7fc5ac107690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:38.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.140+0000 7fc5b142e700 1 --2- 172.21.15.70:0/3882541244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ac100830 0x7fc5ac100cb0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:38.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.140+0000 7fc5b142e700 1 -- 172.21.15.70:0/3882541244 >> 172.21.15.70:0/3882541244 conn(0x7fc5ac077740 msgr2=0x7fc5ac0feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:38.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.140+0000 7fc5b142e700 1 -- 172.21.15.70:0/3882541244 shutdown_connections 2024-04-01T01:32:38.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.140+0000 7fc5b142e700 1 -- 172.21.15.70:0/3882541244 wait complete. 2024-04-01T01:32:38.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.141+0000 7fc5b142e700 1 Processor -- start 2024-04-01T01:32:38.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.141+0000 7fc5b142e700 1 -- start start 2024-04-01T01:32:38.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.141+0000 7fc5b142e700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5ac100830 0x7fc5ac19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:38.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.141+0000 7fc5b142e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5ac1011f0 0x7fc5ac19ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:38.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.141+0000 7fc5b142e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ac1072b0 0x7fc5ac1988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:38.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.141+0000 7fc5b142e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac112540 con 0x7fc5ac1072b0 2024-04-01T01:32:38.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.141+0000 7fc5b142e700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac1123c0 con 0x7fc5ac100830 2024-04-01T01:32:38.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.141+0000 7fc5b142e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac1126c0 con 0x7fc5ac1011f0 2024-04-01T01:32:38.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.141+0000 7fc5ab7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ac1072b0 0x7fc5ac1988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:38.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.141+0000 7fc5ab7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ac1072b0 0x7fc5ac1988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45926/0 (socket says 172.21.15.70:45926) 2024-04-01T01:32:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.141+0000 7fc5ab7fe700 1 -- 172.21.15.70:0/3260824989 learned_addr learned my addr 172.21.15.70:0/3260824989 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.141+0000 7fc5aa7fc700 1 --2- 172.21.15.70:0/3260824989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5ac1011f0 0x7fc5ac19ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.141+0000 7fc5aaffd700 1 --2- 172.21.15.70:0/3260824989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5ac100830 0x7fc5ac19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.141+0000 7fc5ab7fe700 1 -- 172.21.15.70:0/3260824989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5ac1011f0 msgr2=0x7fc5ac19ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.141+0000 7fc5ab7fe700 1 --2- 172.21.15.70:0/3260824989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5ac1011f0 0x7fc5ac19ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.142+0000 7fc5ab7fe700 1 -- 172.21.15.70:0/3260824989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5ac100830 msgr2=0x7fc5ac19e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.142+0000 7fc5ab7fe700 1 --2- 172.21.15.70:0/3260824989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5ac100830 0x7fc5ac19e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.142+0000 7fc5ab7fe700 1 -- 172.21.15.70:0/3260824989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ac199000 con 0x7fc5ac1072b0 2024-04-01T01:32:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.142+0000 7fc5ab7fe700 1 --2- 172.21.15.70:0/3260824989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ac1072b0 0x7fc5ac1988a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc5a400e970 tx=0x7fc5a400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.142+0000 7fc5a3fff700 1 -- 172.21.15.70:0/3260824989 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a4004a30 con 0x7fc5ac1072b0 2024-04-01T01:32:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.142+0000 7fc5a3fff700 1 -- 172.21.15.70:0/3260824989 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5a40054d0 con 0x7fc5ac1072b0 2024-04-01T01:32:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.142+0000 7fc5b142e700 1 -- 172.21.15.70:0/3260824989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5ac1992f0 con 0x7fc5ac1072b0 2024-04-01T01:32:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.142+0000 7fc5a3fff700 1 -- 172.21.15.70:0/3260824989 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a400cca0 con 0x7fc5ac1072b0 2024-04-01T01:32:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.142+0000 7fc5b142e700 1 -- 172.21.15.70:0/3260824989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5ac1a55b0 con 0x7fc5ac1072b0 2024-04-01T01:32:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.143+0000 7fc5b142e700 1 -- 172.21.15.70:0/3260824989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5ac10a810 con 0x7fc5ac1072b0 2024-04-01T01:32:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.144+0000 7fc5a3fff700 1 -- 172.21.15.70:0/3260824989 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7fc5a4005000 con 0x7fc5ac1072b0 2024-04-01T01:32:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.145+0000 7fc5a3fff700 1 --2- 172.21.15.70:0/3260824989 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5980a25c0 0x7fc5980a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.145+0000 7fc5aaffd700 1 --2- 172.21.15.70:0/3260824989 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5980a25c0 0x7fc5980a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.145+0000 7fc5a3fff700 1 -- 172.21.15.70:0/3260824989 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc5a40cf010 con 0x7fc5ac1072b0 2024-04-01T01:32:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.145+0000 7fc5aaffd700 1 --2- 172.21.15.70:0/3260824989 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5980a25c0 0x7fc5980a4a80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac1999e0 tx=0x7fc59c00b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.148+0000 7fc5a3fff700 1 -- 172.21.15.70:0/3260824989 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc5a4053280 con 0x7fc5ac1072b0 2024-04-01T01:32:38.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:37 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3167160928' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-01T01:32:38.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:37 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3527905134' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-01T01:32:38.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.348+0000 7fc5b142e700 1 -- 172.21.15.70:0/3260824989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fc5ac199dd0 con 0x7fc5ac1072b0 2024-04-01T01:32:38.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.348+0000 7fc5a3fff700 1 -- 172.21.15.70:0/3260824989 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc5a40052f0 con 0x7fc5ac1072b0 2024-04-01T01:32:38.349 INFO:teuthology.orchestra.run.smithi070.stdout:253403070508 2024-04-01T01:32:38.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.351+0000 7fc5b142e700 1 -- 172.21.15.70:0/3260824989 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5980a25c0 msgr2=0x7fc5980a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:38.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.351+0000 7fc5b142e700 1 --2- 172.21.15.70:0/3260824989 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5980a25c0 0x7fc5980a4a80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac1999e0 tx=0x7fc59c00b040 comp rx=0 tx=0).stop 2024-04-01T01:32:38.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.351+0000 7fc5b142e700 1 -- 172.21.15.70:0/3260824989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ac1072b0 msgr2=0x7fc5ac1988a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:38.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.351+0000 7fc5b142e700 1 --2- 172.21.15.70:0/3260824989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ac1072b0 0x7fc5ac1988a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc5a400e970 tx=0x7fc5a400ec80 comp rx=0 tx=0).stop 2024-04-01T01:32:38.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.351+0000 7fc5b142e700 1 -- 172.21.15.70:0/3260824989 shutdown_connections 2024-04-01T01:32:38.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.351+0000 7fc5b142e700 1 --2- 172.21.15.70:0/3260824989 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5980a25c0 0x7fc5980a4a80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:38.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.351+0000 7fc5b142e700 1 --2- 172.21.15.70:0/3260824989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ac1072b0 0x7fc5ac1988a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:38.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.351+0000 7fc5b142e700 1 --2- 172.21.15.70:0/3260824989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5ac100830 0x7fc5ac19e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:38.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.351+0000 7fc5b142e700 1 --2- 172.21.15.70:0/3260824989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5ac1011f0 0x7fc5ac19ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:38.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.351+0000 7fc5b142e700 1 -- 172.21.15.70:0/3260824989 >> 172.21.15.70:0/3260824989 conn(0x7fc5ac077740 msgr2=0x7fc5ac105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:38.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.351+0000 7fc5b142e700 1 -- 172.21.15.70:0/3260824989 shutdown_connections 2024-04-01T01:32:38.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:38.351+0000 7fc5b142e700 1 -- 172.21.15.70:0/3260824989 wait complete. 2024-04-01T01:32:39.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:38 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3260824989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-01T01:32:39.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:38 smithi070 ceph-mon[32179]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-01T01:32:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:38 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3260824989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-01T01:32:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:38 smithi081 ceph-mon[35142]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-01T01:32:39.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:38 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3260824989' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-01T01:32:39.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:38 smithi189 ceph-mon[35753]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-01T01:32:39.934 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383705 got 343597383710 for osd.11 2024-04-01T01:32:39.934 DEBUG:teuthology.parallel:result is None 2024-04-01T01:32:40.377 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411398 got 77309411402 for osd.2 2024-04-01T01:32:40.378 DEBUG:teuthology.parallel:result is None 2024-04-01T01:32:41.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.030+0000 7f93c908d700 1 -- 172.21.15.70:0/602961208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c4108270 msgr2=0x7f93c4108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:41.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.030+0000 7f93c908d700 1 --2- 172.21.15.70:0/602961208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c4108270 0x7f93c4108650 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f93b40099c0 tx=0x7f93b4009cd0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.030+0000 7f93c908d700 1 -- 172.21.15.70:0/602961208 shutdown_connections 2024-04-01T01:32:41.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.030+0000 7f93c908d700 1 --2- 172.21.15.70:0/602961208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93c4102c50 0x7f93c410f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.030+0000 7f93c908d700 1 --2- 172.21.15.70:0/602961208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c4108270 0x7f93c4108650 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.030+0000 7f93c908d700 1 --2- 172.21.15.70:0/602961208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93c4102290 0x7f93c4102710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.030+0000 7f93c908d700 1 -- 172.21.15.70:0/602961208 >> 172.21.15.70:0/602961208 conn(0x7f93c4079370 msgr2=0x7f93c4079780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:41.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.030+0000 7f93c908d700 1 -- 172.21.15.70:0/602961208 shutdown_connections 2024-04-01T01:32:41.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.030+0000 7f93c908d700 1 -- 172.21.15.70:0/602961208 wait complete. 2024-04-01T01:32:41.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.031+0000 7f93c908d700 1 Processor -- start 2024-04-01T01:32:41.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.031+0000 7f93c908d700 1 -- start start 2024-04-01T01:32:41.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c908d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93c4102290 0x7f93c419e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:41.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c908d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c4102c50 0x7f93c419eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:41.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c908d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93c4108270 0x7f93c41986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:41.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c908d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f93c41123d0 con 0x7f93c4102c50 2024-04-01T01:32:41.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c908d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f93c4112250 con 0x7f93c4102290 2024-04-01T01:32:41.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c908d700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f93c4112550 con 0x7f93c4108270 2024-04-01T01:32:41.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c4102c50 0x7f93c419eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:41.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c259c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c4102c50 0x7f93c419eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45942/0 (socket says 172.21.15.70:45942) 2024-04-01T01:32:41.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c2d9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93c4102290 0x7f93c419e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:41.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c259c700 1 -- 172.21.15.70:0/3238118901 learned_addr learned my addr 172.21.15.70:0/3238118901 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:41.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c359e700 1 --2- 172.21.15.70:0/3238118901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93c4108270 0x7f93c41986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:41.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c259c700 1 -- 172.21.15.70:0/3238118901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93c4108270 msgr2=0x7f93c41986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:41.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c259c700 1 --2- 172.21.15.70:0/3238118901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93c4108270 0x7f93c41986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c259c700 1 -- 172.21.15.70:0/3238118901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93c4102290 msgr2=0x7f93c419e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:41.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c259c700 1 --2- 172.21.15.70:0/3238118901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93c4102290 0x7f93c419e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c259c700 1 -- 172.21.15.70:0/3238118901 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c4198fb0 con 0x7f93c4102c50 2024-04-01T01:32:41.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c2d9d700 1 --2- 172.21.15.70:0/3238118901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93c4102290 0x7f93c419e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:41.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c259c700 1 --2- 172.21.15.70:0/3238118901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c4102c50 0x7f93c419eaf0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f93b800e9a0 tx=0x7f93b800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:41.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93abfff700 1 -- 172.21.15.70:0/3238118901 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b800ca10 con 0x7f93c4102c50 2024-04-01T01:32:41.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c908d700 1 -- 172.21.15.70:0/3238118901 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93c41992a0 con 0x7f93c4102c50 2024-04-01T01:32:41.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93abfff700 1 -- 172.21.15.70:0/3238118901 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93b800cbb0 con 0x7f93c4102c50 2024-04-01T01:32:41.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93abfff700 1 -- 172.21.15.70:0/3238118901 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b8018680 con 0x7f93c4102c50 2024-04-01T01:32:41.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.032+0000 7f93c908d700 1 -- 172.21.15.70:0/3238118901 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93c41a5530 con 0x7f93c4102c50 2024-04-01T01:32:41.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.033+0000 7f93c908d700 1 -- 172.21.15.70:0/3238118901 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93c4103920 con 0x7f93c4102c50 2024-04-01T01:32:41.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.034+0000 7f93abfff700 1 -- 172.21.15.70:0/3238118901 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f93b8010430 con 0x7f93c4102c50 2024-04-01T01:32:41.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.035+0000 7f93abfff700 1 --2- 172.21.15.70:0/3238118901 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93ac0a25c0 0x7f93ac0a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:41.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.035+0000 7f93c2d9d700 1 --2- 172.21.15.70:0/3238118901 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93ac0a25c0 0x7f93ac0a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:41.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.035+0000 7f93abfff700 1 -- 172.21.15.70:0/3238118901 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f93b80cefd0 con 0x7f93c4102c50 2024-04-01T01:32:41.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.035+0000 7f93c2d9d700 1 --2- 172.21.15.70:0/3238118901 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93ac0a25c0 0x7f93ac0a4a80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f93b4003760 tx=0x7f93b4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:41.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.039+0000 7f93abfff700 1 -- 172.21.15.70:0/3238118901 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f93b809bcd0 con 0x7f93c4102c50 2024-04-01T01:32:41.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.239+0000 7f93c908d700 1 -- 172.21.15.70:0/3238118901 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f93c4199bb0 con 0x7f93c4102c50 2024-04-01T01:32:41.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.239+0000 7f93abfff700 1 -- 172.21.15.70:0/3238118901 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f93b809b5d0 con 0x7f93c4102c50 2024-04-01T01:32:41.239 INFO:teuthology.orchestra.run.smithi070.stdout:163208757308 2024-04-01T01:32:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.241+0000 7f93c908d700 1 -- 172.21.15.70:0/3238118901 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93ac0a25c0 msgr2=0x7f93ac0a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.241+0000 7f93c908d700 1 --2- 172.21.15.70:0/3238118901 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93ac0a25c0 0x7f93ac0a4a80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f93b4003760 tx=0x7f93b4005fb0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.241+0000 7f93c908d700 1 -- 172.21.15.70:0/3238118901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c4102c50 msgr2=0x7f93c419eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.241+0000 7f93c908d700 1 --2- 172.21.15.70:0/3238118901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c4102c50 0x7f93c419eaf0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f93b800e9a0 tx=0x7f93b800ecb0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.242+0000 7f93c908d700 1 -- 172.21.15.70:0/3238118901 shutdown_connections 2024-04-01T01:32:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.242+0000 7f93c908d700 1 --2- 172.21.15.70:0/3238118901 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93ac0a25c0 0x7f93ac0a4a80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.242+0000 7f93c908d700 1 --2- 172.21.15.70:0/3238118901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93c4108270 0x7f93c41986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.242+0000 7f93c908d700 1 --2- 172.21.15.70:0/3238118901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93c4102290 0x7f93c419e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.242+0000 7f93c908d700 1 --2- 172.21.15.70:0/3238118901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93c4102c50 0x7f93c419eaf0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.242+0000 7f93c908d700 1 -- 172.21.15.70:0/3238118901 >> 172.21.15.70:0/3238118901 conn(0x7f93c4079370 msgr2=0x7f93c4100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:41.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.242+0000 7f93c908d700 1 -- 172.21.15.70:0/3238118901 shutdown_connections 2024-04-01T01:32:41.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.242+0000 7f93c908d700 1 -- 172.21.15.70:0/3238118901 wait complete. 2024-04-01T01:32:41.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.783+0000 7f9a62ed2700 1 -- 172.21.15.70:0/2138898728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a5c101140 msgr2=0x7f9a5c1015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:41.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.783+0000 7f9a62ed2700 1 --2- 172.21.15.70:0/2138898728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a5c101140 0x7f9a5c1015a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9a48009990 tx=0x7f9a48009ca0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.784 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:41 smithi070 ceph-mon[32179]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-01T01:32:41.784 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:41 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3238118901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-01T01:32:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.783+0000 7f9a62ed2700 1 -- 172.21.15.70:0/2138898728 shutdown_connections 2024-04-01T01:32:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.783+0000 7f9a62ed2700 1 --2- 172.21.15.70:0/2138898728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a5c101ae0 0x7f9a5c10dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.783+0000 7f9a62ed2700 1 --2- 172.21.15.70:0/2138898728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a5c107180 0x7f9a5c107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.783+0000 7f9a62ed2700 1 --2- 172.21.15.70:0/2138898728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a5c101140 0x7f9a5c1015a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.783+0000 7f9a62ed2700 1 -- 172.21.15.70:0/2138898728 >> 172.21.15.70:0/2138898728 conn(0x7f9a5c078070 msgr2=0x7f9a5c078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.783+0000 7f9a62ed2700 1 -- 172.21.15.70:0/2138898728 shutdown_connections 2024-04-01T01:32:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.783+0000 7f9a62ed2700 1 -- 172.21.15.70:0/2138898728 wait complete. 2024-04-01T01:32:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.784+0000 7f9a62ed2700 1 Processor -- start 2024-04-01T01:32:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.784+0000 7f9a62ed2700 1 -- start start 2024-04-01T01:32:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.784+0000 7f9a62ed2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a5c101140 0x7f9a5c19d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.784+0000 7f9a62ed2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a5c101ae0 0x7f9a5c19daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a62ed2700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a5c107180 0x7f9a5c197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a62ed2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c111290 con 0x7f9a5c101ae0 2024-04-01T01:32:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a62ed2700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c111110 con 0x7f9a5c107180 2024-04-01T01:32:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a62ed2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5c111410 con 0x7f9a5c101140 2024-04-01T01:32:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a60c6e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a5c101140 0x7f9a5c19d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a60c6e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a5c101140 0x7f9a5c19d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:59392/0 (socket says 172.21.15.70:59392) 2024-04-01T01:32:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a60c6e700 1 -- 172.21.15.70:0/2627982003 learned_addr learned my addr 172.21.15.70:0/2627982003 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a6146f700 1 --2- 172.21.15.70:0/2627982003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a5c107180 0x7f9a5c197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a5bfff700 1 --2- 172.21.15.70:0/2627982003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a5c101ae0 0x7f9a5c19daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a60c6e700 1 -- 172.21.15.70:0/2627982003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a5c107180 msgr2=0x7f9a5c197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a60c6e700 1 --2- 172.21.15.70:0/2627982003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a5c107180 0x7f9a5c197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a60c6e700 1 -- 172.21.15.70:0/2627982003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a5c101ae0 msgr2=0x7f9a5c19daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a60c6e700 1 --2- 172.21.15.70:0/2627982003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a5c101ae0 0x7f9a5c19daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a60c6e700 1 -- 172.21.15.70:0/2627982003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a5c197f10 con 0x7f9a5c101140 2024-04-01T01:32:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a6146f700 1 --2- 172.21.15.70:0/2627982003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a5c107180 0x7f9a5c197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a60c6e700 1 --2- 172.21.15.70:0/2627982003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a5c101140 0x7f9a5c19d560 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a5000ead0 tx=0x7f9a5000ede0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a59ffb700 1 -- 172.21.15.70:0/2627982003 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5000b9c0 con 0x7f9a5c101140 2024-04-01T01:32:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.785+0000 7f9a62ed2700 1 -- 172.21.15.70:0/2627982003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a5c1981a0 con 0x7f9a5c101140 2024-04-01T01:32:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.786+0000 7f9a59ffb700 1 -- 172.21.15.70:0/2627982003 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a50004ce0 con 0x7f9a5c101140 2024-04-01T01:32:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.786+0000 7f9a59ffb700 1 -- 172.21.15.70:0/2627982003 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a50005580 con 0x7f9a5c101140 2024-04-01T01:32:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.786+0000 7f9a62ed2700 1 -- 172.21.15.70:0/2627982003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a5c06a0c0 con 0x7f9a5c101140 2024-04-01T01:32:41.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.787+0000 7f9a62ed2700 1 -- 172.21.15.70:0/2627982003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a5c102830 con 0x7f9a5c101140 2024-04-01T01:32:41.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.788+0000 7f9a59ffb700 1 -- 172.21.15.70:0/2627982003 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f9a50004750 con 0x7f9a5c101140 2024-04-01T01:32:41.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.789+0000 7f9a59ffb700 1 --2- 172.21.15.70:0/2627982003 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9a4c0a2640 0x7f9a4c0a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:41.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.789+0000 7f9a5bfff700 1 --2- 172.21.15.70:0/2627982003 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9a4c0a2640 0x7f9a4c0a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:41.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.789+0000 7f9a59ffb700 1 -- 172.21.15.70:0/2627982003 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9a500d02a0 con 0x7f9a5c101140 2024-04-01T01:32:41.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.790+0000 7f9a5bfff700 1 --2- 172.21.15.70:0/2627982003 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9a4c0a2640 0x7f9a4c0a4b00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9a480098e0 tx=0x7f9a48005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:41.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:41.798+0000 7f9a59ffb700 1 -- 172.21.15.70:0/2627982003 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9a5009cfd0 con 0x7f9a5c101140 2024-04-01T01:32:41.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:41 smithi189 ceph-mon[35753]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-01T01:32:41.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:41 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3238118901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-01T01:32:42.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.014+0000 7f9a62ed2700 1 -- 172.21.15.70:0/2627982003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f9a5c1984e0 con 0x7f9a5c101140 2024-04-01T01:32:42.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.014+0000 7f9a59ffb700 1 -- 172.21.15.70:0/2627982003 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9a5009c8d0 con 0x7f9a5c101140 2024-04-01T01:32:42.014 INFO:teuthology.orchestra.run.smithi070.stdout:34359738451 2024-04-01T01:32:42.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.017+0000 7f9a62ed2700 1 -- 172.21.15.70:0/2627982003 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9a4c0a2640 msgr2=0x7f9a4c0a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:42.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.017+0000 7f9a62ed2700 1 --2- 172.21.15.70:0/2627982003 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9a4c0a2640 0x7f9a4c0a4b00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9a480098e0 tx=0x7f9a48005960 comp rx=0 tx=0).stop 2024-04-01T01:32:42.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.017+0000 7f9a62ed2700 1 -- 172.21.15.70:0/2627982003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a5c101140 msgr2=0x7f9a5c19d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:42.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.018+0000 7f9a62ed2700 1 --2- 172.21.15.70:0/2627982003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a5c101140 0x7f9a5c19d560 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9a5000ead0 tx=0x7f9a5000ede0 comp rx=0 tx=0).stop 2024-04-01T01:32:42.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.018+0000 7f9a62ed2700 1 -- 172.21.15.70:0/2627982003 shutdown_connections 2024-04-01T01:32:42.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.018+0000 7f9a62ed2700 1 --2- 172.21.15.70:0/2627982003 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9a4c0a2640 0x7f9a4c0a4b00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:42.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.018+0000 7f9a62ed2700 1 --2- 172.21.15.70:0/2627982003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a5c107180 0x7f9a5c197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:42.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.018+0000 7f9a62ed2700 1 --2- 172.21.15.70:0/2627982003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a5c101140 0x7f9a5c19d560 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:42.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.018+0000 7f9a62ed2700 1 --2- 172.21.15.70:0/2627982003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a5c101ae0 0x7f9a5c19daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:42.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.018+0000 7f9a62ed2700 1 -- 172.21.15.70:0/2627982003 >> 172.21.15.70:0/2627982003 conn(0x7f9a5c078070 msgr2=0x7f9a5c0fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:42.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.018+0000 7f9a62ed2700 1 -- 172.21.15.70:0/2627982003 shutdown_connections 2024-04-01T01:32:42.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.018+0000 7f9a62ed2700 1 -- 172.21.15.70:0/2627982003 wait complete. 2024-04-01T01:32:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:41 smithi081 ceph-mon[35142]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-01T01:32:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:41 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3238118901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-01T01:32:42.147 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182464 got 107374182468 for osd.3 2024-04-01T01:32:42.148 DEBUG:teuthology.parallel:result is None 2024-04-01T01:32:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.350+0000 7f00c7480700 1 -- 172.21.15.70:0/2406452914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00c0101140 msgr2=0x7f00c01015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.350+0000 7f00c7480700 1 --2- 172.21.15.70:0/2406452914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00c0101140 0x7f00c01015a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f00b0009990 tx=0x7f00b0009ca0 comp rx=0 tx=0).stop 2024-04-01T01:32:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.350+0000 7f00c7480700 1 -- 172.21.15.70:0/2406452914 shutdown_connections 2024-04-01T01:32:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.350+0000 7f00c7480700 1 --2- 172.21.15.70:0/2406452914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00c0101ae0 0x7f00c010dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.350+0000 7f00c7480700 1 --2- 172.21.15.70:0/2406452914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00c0107180 0x7f00c0107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.350+0000 7f00c7480700 1 --2- 172.21.15.70:0/2406452914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00c0101140 0x7f00c01015a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.350+0000 7f00c7480700 1 -- 172.21.15.70:0/2406452914 >> 172.21.15.70:0/2406452914 conn(0x7f00c0078070 msgr2=0x7f00c0078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.350+0000 7f00c7480700 1 -- 172.21.15.70:0/2406452914 shutdown_connections 2024-04-01T01:32:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.350+0000 7f00c7480700 1 -- 172.21.15.70:0/2406452914 wait complete. 2024-04-01T01:32:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.351+0000 7f00c7480700 1 Processor -- start 2024-04-01T01:32:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.351+0000 7f00c7480700 1 -- start start 2024-04-01T01:32:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.351+0000 7f00c7480700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00c0101140 0x7f00c0199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.351+0000 7f00c7480700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00c0101ae0 0x7f00c0199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.351+0000 7f00c7480700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00c0107180 0x7f00c01a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.351+0000 7f00c7480700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f00c01112b0 con 0x7f00c0101140 2024-04-01T01:32:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00c7480700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f00c0111130 con 0x7f00c0107180 2024-04-01T01:32:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00c7480700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f00c0111430 con 0x7f00c0101ae0 2024-04-01T01:32:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00c521c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00c0101140 0x7f00c0199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00c521c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00c0101140 0x7f00c0199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45970/0 (socket says 172.21.15.70:45970) 2024-04-01T01:32:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00c4a1b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00c0101ae0 0x7f00c0199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00c521c700 1 -- 172.21.15.70:0/2498233496 learned_addr learned my addr 172.21.15.70:0/2498233496 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00c5a1d700 1 --2- 172.21.15.70:0/2498233496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00c0107180 0x7f00c01a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:42.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00c4a1b700 1 -- 172.21.15.70:0/2498233496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00c0107180 msgr2=0x7f00c01a4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:42.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00c4a1b700 1 --2- 172.21.15.70:0/2498233496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00c0107180 0x7f00c01a4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:42.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00c4a1b700 1 -- 172.21.15.70:0/2498233496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00c0101140 msgr2=0x7f00c0199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:42.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00c4a1b700 1 --2- 172.21.15.70:0/2498233496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00c0101140 0x7f00c0199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:42.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00c4a1b700 1 -- 172.21.15.70:0/2498233496 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c01a4dc0 con 0x7f00c0101ae0 2024-04-01T01:32:42.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00c521c700 1 --2- 172.21.15.70:0/2498233496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00c0101140 0x7f00c0199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:42.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00c4a1b700 1 --2- 172.21.15.70:0/2498233496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00c0101ae0 0x7f00c0199d40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f00b00059d0 tx=0x7f00b0004790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:42.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00b67fc700 1 -- 172.21.15.70:0/2498233496 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b0027070 con 0x7f00c0101ae0 2024-04-01T01:32:42.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00c7480700 1 -- 172.21.15.70:0/2498233496 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c01a4f90 con 0x7f00c0101ae0 2024-04-01T01:32:42.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00b67fc700 1 -- 172.21.15.70:0/2498233496 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00b0022470 con 0x7f00c0101ae0 2024-04-01T01:32:42.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.352+0000 7f00b67fc700 1 -- 172.21.15.70:0/2498233496 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b000b610 con 0x7f00c0101ae0 2024-04-01T01:32:42.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.353+0000 7f00c7480700 1 -- 172.21.15.70:0/2498233496 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c01a53e0 con 0x7f00c0101ae0 2024-04-01T01:32:42.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.353+0000 7f00c7480700 1 -- 172.21.15.70:0/2498233496 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00c0068f00 con 0x7f00c0101ae0 2024-04-01T01:32:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.354+0000 7f00b67fc700 1 -- 172.21.15.70:0/2498233496 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f00b002a070 con 0x7f00c0101ae0 2024-04-01T01:32:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.355+0000 7f00b67fc700 1 --2- 172.21.15.70:0/2498233496 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00ac0a25c0 0x7f00ac0a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.355+0000 7f00c521c700 1 --2- 172.21.15.70:0/2498233496 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00ac0a25c0 0x7f00ac0a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.355+0000 7f00b67fc700 1 -- 172.21.15.70:0/2498233496 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f00b00cf7f0 con 0x7f00c0101ae0 2024-04-01T01:32:42.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.355+0000 7f00c521c700 1 --2- 172.21.15.70:0/2498233496 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00ac0a25c0 0x7f00ac0a4a80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009680 tx=0x7f00bc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:42.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.363+0000 7f00b67fc700 1 -- 172.21.15.70:0/2498233496 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f00b0013070 con 0x7f00c0101ae0 2024-04-01T01:32:42.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.570+0000 7f00c7480700 1 -- 172.21.15.70:0/2498233496 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f00c002db30 con 0x7f00c0101ae0 2024-04-01T01:32:42.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.570+0000 7f00b67fc700 1 -- 172.21.15.70:0/2498233496 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f00b009c520 con 0x7f00c0101ae0 2024-04-01T01:32:42.570 INFO:teuthology.orchestra.run.smithi070.stdout:309237645347 2024-04-01T01:32:42.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.572+0000 7f00c7480700 1 -- 172.21.15.70:0/2498233496 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00ac0a25c0 msgr2=0x7f00ac0a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:42.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.572+0000 7f00c7480700 1 --2- 172.21.15.70:0/2498233496 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00ac0a25c0 0x7f00ac0a4a80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009680 tx=0x7f00bc009260 comp rx=0 tx=0).stop 2024-04-01T01:32:42.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.572+0000 7f00c7480700 1 -- 172.21.15.70:0/2498233496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00c0101ae0 msgr2=0x7f00c0199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:42.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.572+0000 7f00c7480700 1 --2- 172.21.15.70:0/2498233496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00c0101ae0 0x7f00c0199d40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f00b00059d0 tx=0x7f00b0004790 comp rx=0 tx=0).stop 2024-04-01T01:32:42.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.572+0000 7f00c7480700 1 -- 172.21.15.70:0/2498233496 shutdown_connections 2024-04-01T01:32:42.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.572+0000 7f00c7480700 1 --2- 172.21.15.70:0/2498233496 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00ac0a25c0 0x7f00ac0a4a80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:42.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.572+0000 7f00c7480700 1 --2- 172.21.15.70:0/2498233496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00c0107180 0x7f00c01a4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:42.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.572+0000 7f00c7480700 1 --2- 172.21.15.70:0/2498233496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00c0101140 0x7f00c0199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:42.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.572+0000 7f00c7480700 1 --2- 172.21.15.70:0/2498233496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00c0101ae0 0x7f00c0199d40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:42.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.572+0000 7f00c7480700 1 -- 172.21.15.70:0/2498233496 >> 172.21.15.70:0/2498233496 conn(0x7f00c0078070 msgr2=0x7f00c0100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:42.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.573+0000 7f00c7480700 1 -- 172.21.15.70:0/2498233496 shutdown_connections 2024-04-01T01:32:42.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:42.573+0000 7f00c7480700 1 -- 172.21.15.70:0/2498233496 wait complete. 2024-04-01T01:32:42.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:42 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2627982003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-01T01:32:42.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:42 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2498233496' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-01T01:32:42.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:42 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2627982003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-01T01:32:42.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:42 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2498233496' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-01T01:32:43.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:42 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2627982003' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-01T01:32:43.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:42 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2498233496' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-01T01:32:43.271 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757302 got 163208757308 for osd.5 2024-04-01T01:32:43.271 DEBUG:teuthology.parallel:result is None 2024-04-01T01:32:43.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:43 smithi189 ceph-mon[35753]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-01T01:32:44.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:43 smithi070 ceph-mon[32179]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-01T01:32:44.063 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738447 got 34359738451 for osd.0 2024-04-01T01:32:44.063 DEBUG:teuthology.parallel:result is None 2024-04-01T01:32:44.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:43 smithi081 ceph-mon[35142]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-01T01:32:45.456 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070504 got 253403070508 for osd.8 2024-04-01T01:32:45.456 DEBUG:teuthology.parallel:result is None 2024-04-01T01:32:45.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:45 smithi070 ceph-mon[32179]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-01T01:32:45.935 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645343 got 309237645347 for osd.10 2024-04-01T01:32:45.935 DEBUG:teuthology.parallel:result is None 2024-04-01T01:32:45.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:45 smithi189 ceph-mon[35753]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-01T01:32:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:45 smithi081 ceph-mon[35142]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-01T01:32:47.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.324+0000 7f4a2d6bd700 1 -- 172.21.15.70:0/3808603172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a2806a9e0 msgr2=0x7f4a2806adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:47.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.324+0000 7f4a2d6bd700 1 --2- 172.21.15.70:0/3808603172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a2806a9e0 0x7f4a2806adc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4a180099c0 tx=0x7f4a18009cd0 comp rx=0 tx=0).stop 2024-04-01T01:32:47.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.324+0000 7f4a2d6bd700 1 -- 172.21.15.70:0/3808603172 shutdown_connections 2024-04-01T01:32:47.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.324+0000 7f4a2d6bd700 1 --2- 172.21.15.70:0/3808603172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a281075c0 0x7f4a2810f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:47.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.324+0000 7f4a2d6bd700 1 --2- 172.21.15.70:0/3808603172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a2806a9e0 0x7f4a2806adc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:47.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.324+0000 7f4a2d6bd700 1 --2- 172.21.15.70:0/3808603172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a2806b390 0x7f4a28106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:47.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.324+0000 7f4a2d6bd700 1 -- 172.21.15.70:0/3808603172 >> 172.21.15.70:0/3808603172 conn(0x7f4a28077190 msgr2=0x7f4a280775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:47.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.324+0000 7f4a2d6bd700 1 -- 172.21.15.70:0/3808603172 shutdown_connections 2024-04-01T01:32:47.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.324+0000 7f4a2d6bd700 1 -- 172.21.15.70:0/3808603172 wait complete. 2024-04-01T01:32:47.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.324+0000 7f4a2d6bd700 1 Processor -- start 2024-04-01T01:32:47.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a2d6bd700 1 -- start start 2024-04-01T01:32:47.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a2d6bd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a2806a9e0 0x7f4a28101240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:47.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a2d6bd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a2806b390 0x7f4a28101780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:47.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a2d6bd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a281075c0 0x7f4a28101cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:47.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a2d6bd700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4a28112400 con 0x7f4a2806a9e0 2024-04-01T01:32:47.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a2d6bd700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4a28112280 con 0x7f4a2806b390 2024-04-01T01:32:47.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a2d6bd700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4a28112580 con 0x7f4a281075c0 2024-04-01T01:32:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a26ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a2806a9e0 0x7f4a28101240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a26ffd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a2806a9e0 0x7f4a28101240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58532/0 (socket says 172.21.15.70:58532) 2024-04-01T01:32:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a26ffd700 1 -- 172.21.15.70:0/1294606865 learned_addr learned my addr 172.21.15.70:0/1294606865 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a1e7fc700 1 --2- 172.21.15.70:0/1294606865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a2806b390 0x7f4a28101780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a277fe700 1 --2- 172.21.15.70:0/1294606865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a281075c0 0x7f4a28101cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a26ffd700 1 -- 172.21.15.70:0/1294606865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a281075c0 msgr2=0x7f4a28101cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a26ffd700 1 --2- 172.21.15.70:0/1294606865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a281075c0 0x7f4a28101cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a26ffd700 1 -- 172.21.15.70:0/1294606865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a2806b390 msgr2=0x7f4a28101780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a26ffd700 1 --2- 172.21.15.70:0/1294606865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a2806b390 0x7f4a28101780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a26ffd700 1 -- 172.21.15.70:0/1294606865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a281a6e90 con 0x7f4a2806a9e0 2024-04-01T01:32:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.325+0000 7f4a1e7fc700 1 --2- 172.21.15.70:0/1294606865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a2806b390 0x7f4a28101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:47.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.326+0000 7f4a26ffd700 1 --2- 172.21.15.70:0/1294606865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a2806a9e0 0x7f4a28101240 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4a18006010 tx=0x7f4a18009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:47.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.326+0000 7f4a24ff9700 1 -- 172.21.15.70:0/1294606865 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a18026070 con 0x7f4a2806a9e0 2024-04-01T01:32:47.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.326+0000 7f4a24ff9700 1 -- 172.21.15.70:0/1294606865 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a18004ca0 con 0x7f4a2806a9e0 2024-04-01T01:32:47.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.326+0000 7f4a2d6bd700 1 -- 172.21.15.70:0/1294606865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a281a7120 con 0x7f4a2806a9e0 2024-04-01T01:32:47.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.326+0000 7f4a24ff9700 1 -- 172.21.15.70:0/1294606865 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a18003d20 con 0x7f4a2806a9e0 2024-04-01T01:32:47.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.326+0000 7f4a2d6bd700 1 -- 172.21.15.70:0/1294606865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a281a7570 con 0x7f4a2806a9e0 2024-04-01T01:32:47.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.327+0000 7f4a2d6bd700 1 -- 172.21.15.70:0/1294606865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a2810a770 con 0x7f4a2806a9e0 2024-04-01T01:32:47.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.328+0000 7f4a24ff9700 1 -- 172.21.15.70:0/1294606865 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f4a18029070 con 0x7f4a2806a9e0 2024-04-01T01:32:47.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.328+0000 7f4a24ff9700 1 --2- 172.21.15.70:0/1294606865 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4a140a25c0 0x7f4a140a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:47.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.328+0000 7f4a1e7fc700 1 --2- 172.21.15.70:0/1294606865 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4a140a25c0 0x7f4a140a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:47.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.329+0000 7f4a24ff9700 1 -- 172.21.15.70:0/1294606865 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4a180cf430 con 0x7f4a2806a9e0 2024-04-01T01:32:47.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.329+0000 7f4a1e7fc700 1 --2- 172.21.15.70:0/1294606865 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4a140a25c0 0x7f4a140a4a80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4a28100e10 tx=0x7f4a100073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:47.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.333+0000 7f4a24ff9700 1 -- 172.21.15.70:0/1294606865 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4a18013070 con 0x7f4a2806a9e0 2024-04-01T01:32:47.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.546+0000 7f4a2d6bd700 1 -- 172.21.15.70:0/1294606865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f4a2810a150 con 0x7f4a2806a9e0 2024-04-01T01:32:47.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.546+0000 7f4a24ff9700 1 -- 172.21.15.70:0/1294606865 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4a1809c1c0 con 0x7f4a2806a9e0 2024-04-01T01:32:47.545 INFO:teuthology.orchestra.run.smithi070.stdout:55834574928 2024-04-01T01:32:47.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.548+0000 7f4a2d6bd700 1 -- 172.21.15.70:0/1294606865 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4a140a25c0 msgr2=0x7f4a140a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:47.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.548+0000 7f4a2d6bd700 1 --2- 172.21.15.70:0/1294606865 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4a140a25c0 0x7f4a140a4a80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4a28100e10 tx=0x7f4a100073d0 comp rx=0 tx=0).stop 2024-04-01T01:32:47.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.548+0000 7f4a2d6bd700 1 -- 172.21.15.70:0/1294606865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a2806a9e0 msgr2=0x7f4a28101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:47.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.548+0000 7f4a2d6bd700 1 --2- 172.21.15.70:0/1294606865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a2806a9e0 0x7f4a28101240 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4a18006010 tx=0x7f4a18009f80 comp rx=0 tx=0).stop 2024-04-01T01:32:47.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.548+0000 7f4a2d6bd700 1 -- 172.21.15.70:0/1294606865 shutdown_connections 2024-04-01T01:32:47.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.548+0000 7f4a2d6bd700 1 --2- 172.21.15.70:0/1294606865 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4a140a25c0 0x7f4a140a4a80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:47.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.548+0000 7f4a2d6bd700 1 --2- 172.21.15.70:0/1294606865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4a281075c0 0x7f4a28101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:47.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.548+0000 7f4a2d6bd700 1 --2- 172.21.15.70:0/1294606865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a2806a9e0 0x7f4a28101240 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:47.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.548+0000 7f4a2d6bd700 1 --2- 172.21.15.70:0/1294606865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a2806b390 0x7f4a28101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:47.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.548+0000 7f4a2d6bd700 1 -- 172.21.15.70:0/1294606865 >> 172.21.15.70:0/1294606865 conn(0x7f4a28077190 msgr2=0x7f4a281086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:47.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.548+0000 7f4a2d6bd700 1 -- 172.21.15.70:0/1294606865 shutdown_connections 2024-04-01T01:32:47.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:47.548+0000 7f4a2d6bd700 1 -- 172.21.15.70:0/1294606865 wait complete. 2024-04-01T01:32:47.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:47 smithi189 ceph-mon[35753]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:32:47.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:47 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1294606865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-01T01:32:48.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:47 smithi070 ceph-mon[32179]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:32:48.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:47 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1294606865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-01T01:32:48.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:47 smithi081 ceph-mon[35142]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:32:48.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:47 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1294606865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-01T01:32:48.299 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574920 got 55834574928 for osd.1 2024-04-01T01:32:48.299 DEBUG:teuthology.parallel:result is None 2024-04-01T01:32:48.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.783+0000 7f0e5b655700 1 -- 172.21.15.70:0/3175274345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e541071d0 msgr2=0x7f0e541075b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:48.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.783+0000 7f0e5b655700 1 --2- 172.21.15.70:0/3175274345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e541071d0 0x7f0e541075b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0e50009990 tx=0x7f0e50009ca0 comp rx=0 tx=0).stop 2024-04-01T01:32:48.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.783+0000 7f0e5b655700 1 -- 172.21.15.70:0/3175274345 shutdown_connections 2024-04-01T01:32:48.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.783+0000 7f0e5b655700 1 --2- 172.21.15.70:0/3175274345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e54101ae0 0x7f0e5410df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:48.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.783+0000 7f0e5b655700 1 --2- 172.21.15.70:0/3175274345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e541071d0 0x7f0e541075b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:48.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.783+0000 7f0e5b655700 1 --2- 172.21.15.70:0/3175274345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e54101120 0x7f0e541015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:48.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.783+0000 7f0e5b655700 1 -- 172.21.15.70:0/3175274345 >> 172.21.15.70:0/3175274345 conn(0x7f0e54078150 msgr2=0x7f0e54078560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:48.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.783+0000 7f0e5b655700 1 -- 172.21.15.70:0/3175274345 shutdown_connections 2024-04-01T01:32:48.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.783+0000 7f0e5b655700 1 -- 172.21.15.70:0/3175274345 wait complete. 2024-04-01T01:32:48.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.783+0000 7f0e5b655700 1 Processor -- start 2024-04-01T01:32:48.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.783+0000 7f0e5b655700 1 -- start start 2024-04-01T01:32:48.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e5b655700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e54101120 0x7f0e54102dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:48.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e5b655700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e54101ae0 0x7f0e54103310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:48.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e5b655700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e541071d0 0x7f0e54103850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:48.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e5b655700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0e541112e0 con 0x7f0e541071d0 2024-04-01T01:32:48.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e5b655700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0e54111160 con 0x7f0e54101120 2024-04-01T01:32:48.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e5b655700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0e54111460 con 0x7f0e54101ae0 2024-04-01T01:32:48.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e593f1700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e54101120 0x7f0e54102dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:48.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e59bf2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e541071d0 0x7f0e54103850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:48.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e593f1700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e54101120 0x7f0e54102dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38844/0 (socket says 172.21.15.70:38844) 2024-04-01T01:32:48.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e593f1700 1 -- 172.21.15.70:0/2576313701 learned_addr learned my addr 172.21.15.70:0/2576313701 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:48.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e58bf0700 1 --2- 172.21.15.70:0/2576313701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e54101ae0 0x7f0e54103310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:48.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e593f1700 1 -- 172.21.15.70:0/2576313701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e54101ae0 msgr2=0x7f0e54103310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:48.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e593f1700 1 --2- 172.21.15.70:0/2576313701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e54101ae0 0x7f0e54103310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:48.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e593f1700 1 -- 172.21.15.70:0/2576313701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e541071d0 msgr2=0x7f0e54103850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:48.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e593f1700 1 --2- 172.21.15.70:0/2576313701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e541071d0 0x7f0e54103850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:48.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e593f1700 1 -- 172.21.15.70:0/2576313701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e541a6de0 con 0x7f0e54101120 2024-04-01T01:32:48.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e59bf2700 1 --2- 172.21.15.70:0/2576313701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e541071d0 0x7f0e54103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:48.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e593f1700 1 --2- 172.21.15.70:0/2576313701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e54101120 0x7f0e54102dd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0e50005910 tx=0x7f0e500047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:48.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.784+0000 7f0e4a7fc700 1 -- 172.21.15.70:0/2576313701 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e50027070 con 0x7f0e54101120 2024-04-01T01:32:48.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.785+0000 7f0e5b655700 1 -- 172.21.15.70:0/2576313701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e541a7070 con 0x7f0e54101120 2024-04-01T01:32:48.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.785+0000 7f0e4a7fc700 1 -- 172.21.15.70:0/2576313701 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0e50022470 con 0x7f0e54101120 2024-04-01T01:32:48.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.785+0000 7f0e4a7fc700 1 -- 172.21.15.70:0/2576313701 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5000b610 con 0x7f0e54101120 2024-04-01T01:32:48.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.785+0000 7f0e5b655700 1 -- 172.21.15.70:0/2576313701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e541a75a0 con 0x7f0e54101120 2024-04-01T01:32:48.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.785+0000 7f0e5b655700 1 -- 172.21.15.70:0/2576313701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e54068f00 con 0x7f0e54101120 2024-04-01T01:32:48.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.787+0000 7f0e4a7fc700 1 -- 172.21.15.70:0/2576313701 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f0e5002a070 con 0x7f0e54101120 2024-04-01T01:32:48.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.787+0000 7f0e4a7fc700 1 --2- 172.21.15.70:0/2576313701 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0e400a25c0 0x7f0e400a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:48.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.787+0000 7f0e58bf0700 1 --2- 172.21.15.70:0/2576313701 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0e400a25c0 0x7f0e400a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:48.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.787+0000 7f0e4a7fc700 1 -- 172.21.15.70:0/2576313701 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0e500cfb00 con 0x7f0e54101120 2024-04-01T01:32:48.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.788+0000 7f0e58bf0700 1 --2- 172.21.15.70:0/2576313701 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0e400a25c0 0x7f0e400a4a80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0e540786c0 tx=0x7f0e44009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:48.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:48.794+0000 7f0e4a7fc700 1 -- 172.21.15.70:0/2576313701 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0e50013070 con 0x7f0e54101120 2024-04-01T01:32:49.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.009+0000 7f0e5b655700 1 -- 172.21.15.70:0/2576313701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f0e541a78a0 con 0x7f0e54101120 2024-04-01T01:32:49.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.009+0000 7f0e4a7fc700 1 -- 172.21.15.70:0/2576313701 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0e5009c890 con 0x7f0e54101120 2024-04-01T01:32:49.009 INFO:teuthology.orchestra.run.smithi070.stdout:141733920834 2024-04-01T01:32:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.011+0000 7f0e5b655700 1 -- 172.21.15.70:0/2576313701 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0e400a25c0 msgr2=0x7f0e400a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.011+0000 7f0e5b655700 1 --2- 172.21.15.70:0/2576313701 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0e400a25c0 0x7f0e400a4a80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0e540786c0 tx=0x7f0e44009260 comp rx=0 tx=0).stop 2024-04-01T01:32:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.011+0000 7f0e5b655700 1 -- 172.21.15.70:0/2576313701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e54101120 msgr2=0x7f0e54102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.011+0000 7f0e5b655700 1 --2- 172.21.15.70:0/2576313701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e54101120 0x7f0e54102dd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0e50005910 tx=0x7f0e500047b0 comp rx=0 tx=0).stop 2024-04-01T01:32:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.012+0000 7f0e5b655700 1 -- 172.21.15.70:0/2576313701 shutdown_connections 2024-04-01T01:32:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.012+0000 7f0e5b655700 1 --2- 172.21.15.70:0/2576313701 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0e400a25c0 0x7f0e400a4a80 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.012+0000 7f0e5b655700 1 --2- 172.21.15.70:0/2576313701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e541071d0 0x7f0e54103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.012+0000 7f0e5b655700 1 --2- 172.21.15.70:0/2576313701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e54101120 0x7f0e54102dd0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.012+0000 7f0e5b655700 1 --2- 172.21.15.70:0/2576313701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e54101ae0 0x7f0e54103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:49.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.012+0000 7f0e5b655700 1 -- 172.21.15.70:0/2576313701 >> 172.21.15.70:0/2576313701 conn(0x7f0e54078150 msgr2=0x7f0e54100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:49.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.012+0000 7f0e5b655700 1 -- 172.21.15.70:0/2576313701 shutdown_connections 2024-04-01T01:32:49.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.012+0000 7f0e5b655700 1 -- 172.21.15.70:0/2576313701 wait complete. 2024-04-01T01:32:49.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.026+0000 7f2286c1c700 1 -- 172.21.15.70:0/3204215766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2280106900 msgr2=0x7f2280106ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:49.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.026+0000 7f2286c1c700 1 --2- 172.21.15.70:0/3204215766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2280106900 0x7f2280106ce0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2274009990 tx=0x7f2274009ca0 comp rx=0 tx=0).stop 2024-04-01T01:32:49.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.026+0000 7f2286c1c700 1 -- 172.21.15.70:0/3204215766 shutdown_connections 2024-04-01T01:32:49.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.026+0000 7f2286c1c700 1 --2- 172.21.15.70:0/3204215766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22801012c0 0x7f228010d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:49.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.026+0000 7f2286c1c700 1 --2- 172.21.15.70:0/3204215766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2280106900 0x7f2280106ce0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:49.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.026+0000 7f2286c1c700 1 --2- 172.21.15.70:0/3204215766 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2280100900 0x7f2280100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:49.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.026+0000 7f2286c1c700 1 -- 172.21.15.70:0/3204215766 >> 172.21.15.70:0/3204215766 conn(0x7f2280077740 msgr2=0x7f22800feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:49.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.026+0000 7f2286c1c700 1 -- 172.21.15.70:0/3204215766 shutdown_connections 2024-04-01T01:32:49.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.026+0000 7f2286c1c700 1 -- 172.21.15.70:0/3204215766 wait complete. 2024-04-01T01:32:49.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.026+0000 7f2286c1c700 1 Processor -- start 2024-04-01T01:32:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f2286c1c700 1 -- start start 2024-04-01T01:32:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f2286c1c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2280100900 0x7f228019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f2286c1c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22801012c0 0x7f228019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f2286c1c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2280106900 0x7f2280198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f2286c1c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2280110270 con 0x7f2280106900 2024-04-01T01:32:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f2286c1c700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f22801100f0 con 0x7f2280100900 2024-04-01T01:32:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f2286c1c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f22801103f0 con 0x7f22801012c0 2024-04-01T01:32:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f22851b9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2280106900 0x7f2280198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f227ffff700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22801012c0 0x7f228019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f22851b9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2280106900 0x7f2280198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58554/0 (socket says 172.21.15.70:58554) 2024-04-01T01:32:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f22851b9700 1 -- 172.21.15.70:0/3735815282 learned_addr learned my addr 172.21.15.70:0/3735815282 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f22849b8700 1 --2- 172.21.15.70:0/3735815282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2280100900 0x7f228019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f227ffff700 1 -- 172.21.15.70:0/3735815282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2280100900 msgr2=0x7f228019e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f227ffff700 1 --2- 172.21.15.70:0/3735815282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2280100900 0x7f228019e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f227ffff700 1 -- 172.21.15.70:0/3735815282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2280106900 msgr2=0x7f2280198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f227ffff700 1 --2- 172.21.15.70:0/3735815282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2280106900 0x7f2280198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f227ffff700 1 -- 172.21.15.70:0/3735815282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2280199030 con 0x7f22801012c0 2024-04-01T01:32:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.027+0000 7f227ffff700 1 --2- 172.21.15.70:0/3735815282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22801012c0 0x7f228019ebc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f226c00e9f0 tx=0x7f226c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.028+0000 7f227dffb700 1 -- 172.21.15.70:0/3735815282 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c00c9c0 con 0x7f22801012c0 2024-04-01T01:32:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.028+0000 7f2286c1c700 1 -- 172.21.15.70:0/3735815282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2280199320 con 0x7f22801012c0 2024-04-01T01:32:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.028+0000 7f227dffb700 1 -- 172.21.15.70:0/3735815282 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f226c00cb60 con 0x7f22801012c0 2024-04-01T01:32:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.028+0000 7f227dffb700 1 -- 172.21.15.70:0/3735815282 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c018680 con 0x7f22801012c0 2024-04-01T01:32:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.028+0000 7f2286c1c700 1 -- 172.21.15.70:0/3735815282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22801a5600 con 0x7f22801012c0 2024-04-01T01:32:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.029+0000 7f2286c1c700 1 -- 172.21.15.70:0/3735815282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2280102040 con 0x7f22801012c0 2024-04-01T01:32:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.030+0000 7f227dffb700 1 -- 172.21.15.70:0/3735815282 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 137371+0+0 (secure 0 0 0) 0x7f226c010430 con 0x7f22801012c0 2024-04-01T01:32:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.031+0000 7f227dffb700 1 --2- 172.21.15.70:0/3735815282 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22700a25c0 0x7f22700a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.031+0000 7f227dffb700 1 -- 172.21.15.70:0/3735815282 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f226c0cf050 con 0x7f22801012c0 2024-04-01T01:32:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.031+0000 7f22849b8700 1 --2- 172.21.15.70:0/3735815282 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22700a25c0 0x7f22700a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:49.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.031+0000 7f22849b8700 1 --2- 172.21.15.70:0/3735815282 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22700a25c0 0x7f22700a4a80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f22740098e0 tx=0x7f2274005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:49.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.039+0000 7f227dffb700 1 -- 172.21.15.70:0/3735815282 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f226c09bd30 con 0x7f22801012c0 2024-04-01T01:32:49.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.240+0000 7f2286c1c700 1 -- 172.21.15.70:0/3735815282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f2280068f30 con 0x7f22801012c0 2024-04-01T01:32:49.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.241+0000 7f227dffb700 1 -- 172.21.15.70:0/3735815282 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2280068f30 con 0x7f22801012c0 2024-04-01T01:32:49.240 INFO:teuthology.orchestra.run.smithi070.stdout:287762808874 2024-04-01T01:32:49.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.243+0000 7f2286c1c700 1 -- 172.21.15.70:0/3735815282 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22700a25c0 msgr2=0x7f22700a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:49.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.243+0000 7f2286c1c700 1 --2- 172.21.15.70:0/3735815282 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22700a25c0 0x7f22700a4a80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f22740098e0 tx=0x7f2274005960 comp rx=0 tx=0).stop 2024-04-01T01:32:49.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.243+0000 7f2286c1c700 1 -- 172.21.15.70:0/3735815282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22801012c0 msgr2=0x7f228019ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:49.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.243+0000 7f2286c1c700 1 --2- 172.21.15.70:0/3735815282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22801012c0 0x7f228019ebc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f226c00e9f0 tx=0x7f226c00ed00 comp rx=0 tx=0).stop 2024-04-01T01:32:49.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.243+0000 7f2286c1c700 1 -- 172.21.15.70:0/3735815282 shutdown_connections 2024-04-01T01:32:49.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.243+0000 7f2286c1c700 1 --2- 172.21.15.70:0/3735815282 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22700a25c0 0x7f22700a4a80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:49.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.243+0000 7f2286c1c700 1 --2- 172.21.15.70:0/3735815282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2280106900 0x7f2280198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:49.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.243+0000 7f2286c1c700 1 --2- 172.21.15.70:0/3735815282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2280100900 0x7f228019e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:49.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.243+0000 7f2286c1c700 1 --2- 172.21.15.70:0/3735815282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22801012c0 0x7f228019ebc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:49.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.243+0000 7f2286c1c700 1 -- 172.21.15.70:0/3735815282 >> 172.21.15.70:0/3735815282 conn(0x7f2280077740 msgr2=0x7f22800ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:49.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.243+0000 7f2286c1c700 1 -- 172.21.15.70:0/3735815282 shutdown_connections 2024-04-01T01:32:49.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:49.243+0000 7f2286c1c700 1 -- 172.21.15.70:0/3735815282 wait complete. 2024-04-01T01:32:49.953 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:49 smithi070 ceph-mon[32179]: pgmap v336: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-01T01:32:49.953 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:49 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2576313701' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-01T01:32:49.953 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:49 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3735815282' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-01T01:32:49.983 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920828 got 141733920834 for osd.4 2024-04-01T01:32:49.983 DEBUG:teuthology.parallel:result is None 2024-04-01T01:32:49.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:49 smithi189 ceph-mon[35753]: pgmap v336: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-01T01:32:49.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:49 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2576313701' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-01T01:32:49.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:49 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3735815282' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-01T01:32:50.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:49 smithi081 ceph-mon[35142]: pgmap v336: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-01T01:32:50.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:49 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2576313701' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-01T01:32:50.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:49 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3735815282' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-01T01:32:50.499 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808868 got 287762808874 for osd.9 2024-04-01T01:32:50.499 DEBUG:teuthology.parallel:result is None 2024-04-01T01:32:50.499 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-01T01:32:50.500 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-01T01:32:50.752 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:32:50.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:50 smithi070 ceph-mon[32179]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-01T01:32:50.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:50 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:50.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:50 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:50.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:50 smithi189 ceph-mon[35753]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-01T01:32:50.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:50 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:50.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:50 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:51.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:50 smithi081 ceph-mon[35142]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-01T01:32:51.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:50 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:51.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:50 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:51.930 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:51 smithi070 ceph-mon[32179]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-01T01:32:51.930 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:51 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:32:51.930 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:51 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:32:51.930 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:51 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:51.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:51 smithi189 ceph-mon[35753]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-01T01:32:51.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:51 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:32:51.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:51 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:32:51.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:51 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:52.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:51 smithi081 ceph-mon[35142]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-01T01:32:52.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:51 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:32:52.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:51 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:32:52.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:51 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:32:52.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.476+0000 7fcacc421700 1 -- 172.21.15.70:0/448875055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac406a890 msgr2=0x7fcac410f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:52.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.476+0000 7fcacc421700 1 --2- 172.21.15.70:0/448875055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac406a890 0x7fcac410f0d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcabc00b0d0 tx=0x7fcabc00b3e0 comp rx=0 tx=0).stop 2024-04-01T01:32:52.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.476+0000 7fcacc421700 1 -- 172.21.15.70:0/448875055 shutdown_connections 2024-04-01T01:32:52.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.476+0000 7fcacc421700 1 --2- 172.21.15.70:0/448875055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac406a890 0x7fcac410f0d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:52.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.476+0000 7fcacc421700 1 --2- 172.21.15.70:0/448875055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcac4108290 0x7fcac4108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:52.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.476+0000 7fcacc421700 1 --2- 172.21.15.70:0/448875055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac4069ef0 0x7fcac406a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:52.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.476+0000 7fcacc421700 1 -- 172.21.15.70:0/448875055 >> 172.21.15.70:0/448875055 conn(0x7fcac4078070 msgr2=0x7fcac4078480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:52.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.476+0000 7fcacc421700 1 -- 172.21.15.70:0/448875055 shutdown_connections 2024-04-01T01:32:52.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.476+0000 7fcacc421700 1 -- 172.21.15.70:0/448875055 wait complete. 2024-04-01T01:32:52.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcacc421700 1 Processor -- start 2024-04-01T01:32:52.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcacc421700 1 -- start start 2024-04-01T01:32:52.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcacc421700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcac4069ef0 0x7fcac4101210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:52.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcacc421700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac406a890 0x7fcac4101750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:52.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcacc421700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac4108290 0x7fcac4101c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:52.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcacc421700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcac41123c0 con 0x7fcac406a890 2024-04-01T01:32:52.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcacc421700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcac4112240 con 0x7fcac4069ef0 2024-04-01T01:32:52.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcacc421700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcac4112540 con 0x7fcac4108290 2024-04-01T01:32:52.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcac99bc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac406a890 0x7fcac4101750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:52.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcaca1bd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcac4069ef0 0x7fcac4101210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:52.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcac99bc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac406a890 0x7fcac4101750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58560/0 (socket says 172.21.15.70:58560) 2024-04-01T01:32:52.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcac99bc700 1 -- 172.21.15.70:0/101742661 learned_addr learned my addr 172.21.15.70:0/101742661 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:52.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcaca9be700 1 --2- 172.21.15.70:0/101742661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac4108290 0x7fcac4101c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:52.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcac99bc700 1 -- 172.21.15.70:0/101742661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac4108290 msgr2=0x7fcac4101c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:52.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcac99bc700 1 --2- 172.21.15.70:0/101742661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac4108290 0x7fcac4101c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:52.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcac99bc700 1 -- 172.21.15.70:0/101742661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcac4069ef0 msgr2=0x7fcac4101210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:52.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcac99bc700 1 --2- 172.21.15.70:0/101742661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcac4069ef0 0x7fcac4101210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:52.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcac99bc700 1 -- 172.21.15.70:0/101742661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac41a6e50 con 0x7fcac406a890 2024-04-01T01:32:52.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.477+0000 7fcaca1bd700 1 --2- 172.21.15.70:0/101742661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcac4069ef0 0x7fcac4101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:52.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.478+0000 7fcac99bc700 1 --2- 172.21.15.70:0/101742661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac406a890 0x7fcac4101750 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fcab400e9c0 tx=0x7fcab400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:52.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.478+0000 7fcabb7fe700 1 -- 172.21.15.70:0/101742661 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab400cbd0 con 0x7fcac406a890 2024-04-01T01:32:52.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.478+0000 7fcacc421700 1 -- 172.21.15.70:0/101742661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcac41a7140 con 0x7fcac406a890 2024-04-01T01:32:52.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.478+0000 7fcabb7fe700 1 -- 172.21.15.70:0/101742661 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcab400ee60 con 0x7fcac406a890 2024-04-01T01:32:52.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.478+0000 7fcacc421700 1 -- 172.21.15.70:0/101742661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcac41a7680 con 0x7fcac406a890 2024-04-01T01:32:52.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.478+0000 7fcabb7fe700 1 -- 172.21.15.70:0/101742661 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab40055b0 con 0x7fcac406a890 2024-04-01T01:32:52.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.479+0000 7fcacc421700 1 -- 172.21.15.70:0/101742661 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcac4068f00 con 0x7fcac406a890 2024-04-01T01:32:52.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.480+0000 7fcabb7fe700 1 -- 172.21.15.70:0/101742661 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fcab4004750 con 0x7fcac406a890 2024-04-01T01:32:52.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.480+0000 7fcabb7fe700 1 --2- 172.21.15.70:0/101742661 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcab00a2530 0x7fcab00a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:52.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.481+0000 7fcaca1bd700 1 --2- 172.21.15.70:0/101742661 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcab00a2530 0x7fcab00a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:52.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.481+0000 7fcabb7fe700 1 -- 172.21.15.70:0/101742661 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcab40ced80 con 0x7fcac406a890 2024-04-01T01:32:52.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.481+0000 7fcaca1bd700 1 --2- 172.21.15.70:0/101742661 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcab00a2530 0x7fcab00a49f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009fd0 tx=0x7fcac0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:52.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.485+0000 7fcabb7fe700 1 -- 172.21.15.70:0/101742661 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcab409bbe0 con 0x7fcac406a890 2024-04-01T01:32:52.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.676+0000 7fcacc421700 1 -- 172.21.15.70:0/101742661 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcac4062680 con 0x7fcab00a2530 2024-04-01T01:32:52.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.685+0000 7fcabb7fe700 1 -- 172.21.15.70:0/101742661 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340654 (secure 0 0 0) 0x7fcac4062680 con 0x7fcab00a2530 2024-04-01T01:32:52.691 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:32:52.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.689+0000 7fcacc421700 1 -- 172.21.15.70:0/101742661 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcab00a2530 msgr2=0x7fcab00a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:52.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.689+0000 7fcacc421700 1 --2- 172.21.15.70:0/101742661 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcab00a2530 0x7fcab00a49f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009fd0 tx=0x7fcac0009260 comp rx=0 tx=0).stop 2024-04-01T01:32:52.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.689+0000 7fcacc421700 1 -- 172.21.15.70:0/101742661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac406a890 msgr2=0x7fcac4101750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:52.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.689+0000 7fcacc421700 1 --2- 172.21.15.70:0/101742661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac406a890 0x7fcac4101750 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fcab400e9c0 tx=0x7fcab400ecd0 comp rx=0 tx=0).stop 2024-04-01T01:32:52.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.689+0000 7fcacc421700 1 -- 172.21.15.70:0/101742661 shutdown_connections 2024-04-01T01:32:52.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.689+0000 7fcacc421700 1 --2- 172.21.15.70:0/101742661 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcab00a2530 0x7fcab00a49f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:52.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.689+0000 7fcacc421700 1 --2- 172.21.15.70:0/101742661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcac4108290 0x7fcac4101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:52.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.689+0000 7fcacc421700 1 --2- 172.21.15.70:0/101742661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcac4069ef0 0x7fcac4101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:52.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.689+0000 7fcacc421700 1 --2- 172.21.15.70:0/101742661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcac406a890 0x7fcac4101750 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:52.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.689+0000 7fcacc421700 1 -- 172.21.15.70:0/101742661 >> 172.21.15.70:0/101742661 conn(0x7fcac4078070 msgr2=0x7fcac40ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.689+0000 7fcacc421700 1 -- 172.21.15.70:0/101742661 shutdown_connections 2024-04-01T01:32:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:52.689+0000 7fcacc421700 1 -- 172.21.15.70:0/101742661 wait complete. 2024-04-01T01:32:52.695 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-01T01:32:53.475 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-04-01T01:32:52.591324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661693,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":897,"num_read_kb":578,"num_write":433,"num_write_kb":3087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181978,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":796,"ondisk_log_size":796,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":359248,"kb_used_data":14240,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124582576,"statfs":{"total":1151940427776,"available":1151572557824,"internally_reserved":0,"allocated":14581760,"data_stored":7970560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,2,21,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":68,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003474"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.608412+0000","last_change":"2024-04-01T01:31:16.385517+0000","last_active":"2024-04-01T01:32:48.608412+0000","last_peered":"2024-04-01T01:32:48.608412+0000","last_clean":"2024-04-01T01:32:48.608412+0000","last_became_active":"2024-04-01T01:31:16.384682+0000","last_became_peered":"2024-04-01T01:31:16.384682+0000","last_unstale":"2024-04-01T01:32:48.608412+0000","last_undegraded":"2024-04-01T01:32:48.608412+0000","last_fullsized":"2024-04-01T01:32:48.608412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:35:47.614041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.604933+0000","last_change":"2024-04-01T01:31:16.376119+0000","last_active":"2024-04-01T01:32:48.604933+0000","last_peered":"2024-04-01T01:32:48.604933+0000","last_clean":"2024-04-01T01:32:48.604933+0000","last_became_active":"2024-04-01T01:31:16.375801+0000","last_became_peered":"2024-04-01T01:31:16.375801+0000","last_unstale":"2024-04-01T01:32:48.604933+0000","last_undegraded":"2024-04-01T01:32:48.604933+0000","last_fullsized":"2024-04-01T01:32:48.604933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:34.120487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.606902+0000","last_change":"2024-04-01T01:31:16.378158+0000","last_active":"2024-04-01T01:32:48.606902+0000","last_peered":"2024-04-01T01:32:48.606902+0000","last_clean":"2024-04-01T01:32:48.606902+0000","last_became_active":"2024-04-01T01:31:16.377831+0000","last_became_peered":"2024-04-01T01:31:16.377831+0000","last_unstale":"2024-04-01T01:32:48.606902+0000","last_undegraded":"2024-04-01T01:32:48.606902+0000","last_fullsized":"2024-04-01T01:32:48.606902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:30:06.487545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.592067+0000","last_change":"2024-04-01T01:31:16.389708+0000","last_active":"2024-04-01T01:32:48.592067+0000","last_peered":"2024-04-01T01:32:48.592067+0000","last_clean":"2024-04-01T01:32:48.592067+0000","last_became_active":"2024-04-01T01:31:16.389409+0000","last_became_peered":"2024-04-01T01:31:16.389409+0000","last_unstale":"2024-04-01T01:32:48.592067+0000","last_undegraded":"2024-04-01T01:32:48.592067+0000","last_fullsized":"2024-04-01T01:32:48.592067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:57:49.215988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.689907+0000","last_change":"2024-04-01T01:31:16.385996+0000","last_active":"2024-04-01T01:32:19.689907+0000","last_peered":"2024-04-01T01:32:19.689907+0000","last_clean":"2024-04-01T01:32:19.689907+0000","last_became_active":"2024-04-01T01:31:16.385648+0000","last_became_peered":"2024-04-01T01:31:16.385648+0000","last_unstale":"2024-04-01T01:32:19.689907+0000","last_undegraded":"2024-04-01T01:32:19.689907+0000","last_fullsized":"2024-04-01T01:32:19.689907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:14:56.595595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.576793+0000","last_change":"2024-04-01T01:32:19.693185+0000","last_active":"2024-04-01T01:32:48.576793+0000","last_peered":"2024-04-01T01:32:48.576793+0000","last_clean":"2024-04-01T01:32:48.576793+0000","last_became_active":"2024-04-01T01:32:19.692763+0000","last_became_peered":"2024-04-01T01:32:19.692763+0000","last_unstale":"2024-04-01T01:32:48.576793+0000","last_undegraded":"2024-04-01T01:32:48.576793+0000","last_fullsized":"2024-04-01T01:32:48.576793+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:37:54.712852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.600332+0000","last_change":"2024-04-01T01:31:16.383939+0000","last_active":"2024-04-01T01:32:48.600332+0000","last_peered":"2024-04-01T01:32:48.600332+0000","last_clean":"2024-04-01T01:32:48.600332+0000","last_became_active":"2024-04-01T01:31:16.383626+0000","last_became_peered":"2024-04-01T01:31:16.383626+0000","last_unstale":"2024-04-01T01:32:48.600332+0000","last_undegraded":"2024-04-01T01:32:48.600332+0000","last_fullsized":"2024-04-01T01:32:48.600332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:36:05.764031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.586784+0000","last_change":"2024-04-01T01:31:16.384007+0000","last_active":"2024-04-01T01:32:48.586784+0000","last_peered":"2024-04-01T01:32:48.586784+0000","last_clean":"2024-04-01T01:32:48.586784+0000","last_became_active":"2024-04-01T01:31:16.383687+0000","last_became_peered":"2024-04-01T01:31:16.383687+0000","last_unstale":"2024-04-01T01:32:48.586784+0000","last_undegraded":"2024-04-01T01:32:48.586784+0000","last_fullsized":"2024-04-01T01:32:48.586784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:14:17.593402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.598231+0000","last_change":"2024-04-01T01:31:16.380357+0000","last_active":"2024-04-01T01:32:48.598231+0000","last_peered":"2024-04-01T01:32:48.598231+0000","last_clean":"2024-04-01T01:32:48.598231+0000","last_became_active":"2024-04-01T01:31:16.380046+0000","last_became_peered":"2024-04-01T01:31:16.380046+0000","last_unstale":"2024-04-01T01:32:48.598231+0000","last_undegraded":"2024-04-01T01:32:48.598231+0000","last_fullsized":"2024-04-01T01:32:48.598231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:51:53.887768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.591699+0000","last_change":"2024-04-01T01:31:16.378551+0000","last_active":"2024-04-01T01:32:48.591699+0000","last_peered":"2024-04-01T01:32:48.591699+0000","last_clean":"2024-04-01T01:32:48.591699+0000","last_became_active":"2024-04-01T01:31:16.378234+0000","last_became_peered":"2024-04-01T01:31:16.378234+0000","last_unstale":"2024-04-01T01:32:48.591699+0000","last_undegraded":"2024-04-01T01:32:48.591699+0000","last_fullsized":"2024-04-01T01:32:48.591699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:37:45.998115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.594272+0000","last_change":"2024-04-01T01:31:16.388755+0000","last_active":"2024-04-01T01:32:48.594272+0000","last_peered":"2024-04-01T01:32:48.594272+0000","last_clean":"2024-04-01T01:32:48.594272+0000","last_became_active":"2024-04-01T01:31:16.388434+0000","last_became_peered":"2024-04-01T01:31:16.388434+0000","last_unstale":"2024-04-01T01:32:48.594272+0000","last_undegraded":"2024-04-01T01:32:48.594272+0000","last_fullsized":"2024-04-01T01:32:48.594272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:03:07.168110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.568940+0000","last_change":"2024-04-01T01:31:16.390478+0000","last_active":"2024-04-01T01:32:48.568940+0000","last_peered":"2024-04-01T01:32:48.568940+0000","last_clean":"2024-04-01T01:32:48.568940+0000","last_became_active":"2024-04-01T01:31:16.390013+0000","last_became_peered":"2024-04-01T01:31:16.390013+0000","last_unstale":"2024-04-01T01:32:48.568940+0000","last_undegraded":"2024-04-01T01:32:48.568940+0000","last_fullsized":"2024-04-01T01:32:48.568940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:28:21.677527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.595795+0000","last_change":"2024-04-01T01:31:16.366921+0000","last_active":"2024-04-01T01:32:48.595795+0000","last_peered":"2024-04-01T01:32:48.595795+0000","last_clean":"2024-04-01T01:32:48.595795+0000","last_became_active":"2024-04-01T01:31:16.366098+0000","last_became_peered":"2024-04-01T01:31:16.366098+0000","last_unstale":"2024-04-01T01:32:48.595795+0000","last_undegraded":"2024-04-01T01:32:48.595795+0000","last_fullsized":"2024-04-01T01:32:48.595795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:29.456263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.605838+0000","last_change":"2024-04-01T01:31:16.379178+0000","last_active":"2024-04-01T01:32:48.605838+0000","last_peered":"2024-04-01T01:32:48.605838+0000","last_clean":"2024-04-01T01:32:48.605838+0000","last_became_active":"2024-04-01T01:31:16.378877+0000","last_became_peered":"2024-04-01T01:31:16.378877+0000","last_unstale":"2024-04-01T01:32:48.605838+0000","last_undegraded":"2024-04-01T01:32:48.605838+0000","last_fullsized":"2024-04-01T01:32:48.605838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:19:16.941504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.606872+0000","last_change":"2024-04-01T01:31:16.385373+0000","last_active":"2024-04-01T01:32:48.606872+0000","last_peered":"2024-04-01T01:32:48.606872+0000","last_clean":"2024-04-01T01:32:48.606872+0000","last_became_active":"2024-04-01T01:31:16.385046+0000","last_became_peered":"2024-04-01T01:31:16.385046+0000","last_unstale":"2024-04-01T01:32:48.606872+0000","last_undegraded":"2024-04-01T01:32:48.606872+0000","last_fullsized":"2024-04-01T01:32:48.606872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:40:32.521220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.599246+0000","last_change":"2024-04-01T01:31:16.386267+0000","last_active":"2024-04-01T01:32:48.599246+0000","last_peered":"2024-04-01T01:32:48.599246+0000","last_clean":"2024-04-01T01:32:48.599246+0000","last_became_active":"2024-04-01T01:31:16.385917+0000","last_became_peered":"2024-04-01T01:31:16.385917+0000","last_unstale":"2024-04-01T01:32:48.599246+0000","last_undegraded":"2024-04-01T01:32:48.599246+0000","last_fullsized":"2024-04-01T01:32:48.599246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:58.249253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.608460+0000","last_change":"2024-04-01T01:31:16.385609+0000","last_active":"2024-04-01T01:32:48.608460+0000","last_peered":"2024-04-01T01:32:48.608460+0000","last_clean":"2024-04-01T01:32:48.608460+0000","last_became_active":"2024-04-01T01:31:16.385128+0000","last_became_peered":"2024-04-01T01:31:16.385128+0000","last_unstale":"2024-04-01T01:32:48.608460+0000","last_undegraded":"2024-04-01T01:32:48.608460+0000","last_fullsized":"2024-04-01T01:32:48.608460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:30:45.915587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.595736+0000","last_change":"2024-04-01T01:31:16.381209+0000","last_active":"2024-04-01T01:32:48.595736+0000","last_peered":"2024-04-01T01:32:48.595736+0000","last_clean":"2024-04-01T01:32:48.595736+0000","last_became_active":"2024-04-01T01:31:16.380721+0000","last_became_peered":"2024-04-01T01:31:16.380721+0000","last_unstale":"2024-04-01T01:32:48.595736+0000","last_undegraded":"2024-04-01T01:32:48.595736+0000","last_fullsized":"2024-04-01T01:32:48.595736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:22.438224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.684247+0000","last_change":"2024-04-01T01:31:16.385235+0000","last_active":"2024-04-01T01:32:19.684247+0000","last_peered":"2024-04-01T01:32:19.684247+0000","last_clean":"2024-04-01T01:32:19.684247+0000","last_became_active":"2024-04-01T01:31:16.384856+0000","last_became_peered":"2024-04-01T01:31:16.384856+0000","last_unstale":"2024-04-01T01:32:19.684247+0000","last_undegraded":"2024-04-01T01:32:19.684247+0000","last_fullsized":"2024-04-01T01:32:19.684247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:35:30.214977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683439+0000","last_change":"2024-04-01T01:31:20.573626+0000","last_active":"2024-04-01T01:32:19.683439+0000","last_peered":"2024-04-01T01:32:19.683439+0000","last_clean":"2024-04-01T01:32:19.683439+0000","last_became_active":"2024-04-01T01:31:20.573182+0000","last_became_peered":"2024-04-01T01:31:20.573182+0000","last_unstale":"2024-04-01T01:32:19.683439+0000","last_undegraded":"2024-04-01T01:32:19.683439+0000","last_fullsized":"2024-04-01T01:32:19.683439+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:33:19.162613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.654805+0000","last_change":"2024-04-01T01:31:15.373100+0000","last_active":"2024-04-01T01:32:48.654805+0000","last_peered":"2024-04-01T01:32:48.654805+0000","last_clean":"2024-04-01T01:32:48.654805+0000","last_became_active":"2024-04-01T01:31:15.372710+0000","last_became_peered":"2024-04-01T01:31:15.372710+0000","last_unstale":"2024-04-01T01:32:48.654805+0000","last_undegraded":"2024-04-01T01:32:48.654805+0000","last_fullsized":"2024-04-01T01:32:48.654805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:29:42.303549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.591687+0000","last_change":"2024-04-01T01:31:16.383690+0000","last_active":"2024-04-01T01:32:48.591687+0000","last_peered":"2024-04-01T01:32:48.591687+0000","last_clean":"2024-04-01T01:32:48.591687+0000","last_became_active":"2024-04-01T01:31:16.383248+0000","last_became_peered":"2024-04-01T01:31:16.383248+0000","last_unstale":"2024-04-01T01:32:48.591687+0000","last_undegraded":"2024-04-01T01:32:48.591687+0000","last_fullsized":"2024-04-01T01:32:48.591687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:34:49.112880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.691398+0000","last_change":"2024-04-01T01:32:19.691398+0000","last_active":"2024-04-01T01:32:19.691398+0000","last_peered":"2024-04-01T01:32:19.691398+0000","last_clean":"2024-04-01T01:32:19.691398+0000","last_became_active":"2024-04-01T01:32:19.690908+0000","last_became_peered":"2024-04-01T01:32:19.690908+0000","last_unstale":"2024-04-01T01:32:19.691398+0000","last_undegraded":"2024-04-01T01:32:19.691398+0000","last_fullsized":"2024-04-01T01:32:19.691398+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:58:54.712363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.687158+0000","last_change":"2024-04-01T01:31:15.381742+0000","last_active":"2024-04-01T01:32:19.687158+0000","last_peered":"2024-04-01T01:32:19.687158+0000","last_clean":"2024-04-01T01:32:19.687158+0000","last_became_active":"2024-04-01T01:31:15.377543+0000","last_became_peered":"2024-04-01T01:31:15.377543+0000","last_unstale":"2024-04-01T01:32:19.687158+0000","last_undegraded":"2024-04-01T01:32:19.687158+0000","last_fullsized":"2024-04-01T01:32:19.687158+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:51:42.911114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.605862+0000","last_change":"2024-04-01T01:31:16.382001+0000","last_active":"2024-04-01T01:32:48.605862+0000","last_peered":"2024-04-01T01:32:48.605862+0000","last_clean":"2024-04-01T01:32:48.605862+0000","last_became_active":"2024-04-01T01:31:16.381694+0000","last_became_peered":"2024-04-01T01:31:16.381694+0000","last_unstale":"2024-04-01T01:32:48.605862+0000","last_undegraded":"2024-04-01T01:32:48.605862+0000","last_fullsized":"2024-04-01T01:32:48.605862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:55:07.366755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683197+0000","last_change":"2024-04-01T01:31:20.570080+0000","last_active":"2024-04-01T01:32:19.683197+0000","last_peered":"2024-04-01T01:32:19.683197+0000","last_clean":"2024-04-01T01:32:19.683197+0000","last_became_active":"2024-04-01T01:31:20.569712+0000","last_became_peered":"2024-04-01T01:31:20.569712+0000","last_unstale":"2024-04-01T01:32:19.683197+0000","last_undegraded":"2024-04-01T01:32:19.683197+0000","last_fullsized":"2024-04-01T01:32:19.683197+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:48:27.732831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.703903+0000","last_change":"2024-04-01T01:32:19.703903+0000","last_active":"2024-04-01T01:32:19.703903+0000","last_peered":"2024-04-01T01:32:19.703903+0000","last_clean":"2024-04-01T01:32:19.703903+0000","last_became_active":"2024-04-01T01:32:19.691856+0000","last_became_peered":"2024-04-01T01:32:19.691856+0000","last_unstale":"2024-04-01T01:32:19.703903+0000","last_undegraded":"2024-04-01T01:32:19.703903+0000","last_fullsized":"2024-04-01T01:32:19.703903+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:38:41.284469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.596295+0000","last_change":"2024-04-01T01:31:16.383020+0000","last_active":"2024-04-01T01:32:48.596295+0000","last_peered":"2024-04-01T01:32:48.596295+0000","last_clean":"2024-04-01T01:32:48.596295+0000","last_became_active":"2024-04-01T01:31:16.382640+0000","last_became_peered":"2024-04-01T01:31:16.382640+0000","last_unstale":"2024-04-01T01:32:48.596295+0000","last_undegraded":"2024-04-01T01:32:48.596295+0000","last_fullsized":"2024-04-01T01:32:48.596295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:11:51.211929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683088+0000","last_change":"2024-04-01T01:31:20.573030+0000","last_active":"2024-04-01T01:32:19.683088+0000","last_peered":"2024-04-01T01:32:19.683088+0000","last_clean":"2024-04-01T01:32:19.683088+0000","last_became_active":"2024-04-01T01:31:20.572613+0000","last_became_peered":"2024-04-01T01:31:20.572613+0000","last_unstale":"2024-04-01T01:32:19.683088+0000","last_undegraded":"2024-04-01T01:32:19.683088+0000","last_fullsized":"2024-04-01T01:32:19.683088+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:42:17.086942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.689171+0000","last_change":"2024-04-01T01:31:15.373054+0000","last_active":"2024-04-01T01:32:19.689171+0000","last_peered":"2024-04-01T01:32:19.689171+0000","last_clean":"2024-04-01T01:32:19.689171+0000","last_became_active":"2024-04-01T01:31:15.372575+0000","last_became_peered":"2024-04-01T01:31:15.372575+0000","last_unstale":"2024-04-01T01:32:19.689171+0000","last_undegraded":"2024-04-01T01:32:19.689171+0000","last_fullsized":"2024-04-01T01:32:19.689171+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:56:41.860423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.599437+0000","last_change":"2024-04-01T01:31:16.376037+0000","last_active":"2024-04-01T01:32:48.599437+0000","last_peered":"2024-04-01T01:32:48.599437+0000","last_clean":"2024-04-01T01:32:48.599437+0000","last_became_active":"2024-04-01T01:31:16.375463+0000","last_became_peered":"2024-04-01T01:31:16.375463+0000","last_unstale":"2024-04-01T01:32:48.599437+0000","last_undegraded":"2024-04-01T01:32:48.599437+0000","last_fullsized":"2024-04-01T01:32:48.599437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:30:40.098033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.684137+0000","last_change":"2024-04-01T01:31:20.574288+0000","last_active":"2024-04-01T01:32:19.684137+0000","last_peered":"2024-04-01T01:32:19.684137+0000","last_clean":"2024-04-01T01:32:19.684137+0000","last_became_active":"2024-04-01T01:31:20.573812+0000","last_became_peered":"2024-04-01T01:31:20.573812+0000","last_unstale":"2024-04-01T01:32:19.684137+0000","last_undegraded":"2024-04-01T01:32:19.684137+0000","last_fullsized":"2024-04-01T01:32:19.684137+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:43:46.126102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":126,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:49.721894+0000","last_change":"2024-04-01T01:32:19.706645+0000","last_active":"2024-04-01T01:32:49.721894+0000","last_peered":"2024-04-01T01:32:49.721894+0000","last_clean":"2024-04-01T01:32:49.721894+0000","last_became_active":"2024-04-01T01:32:19.690510+0000","last_became_peered":"2024-04-01T01:32:19.690510+0000","last_unstale":"2024-04-01T01:32:49.721894+0000","last_undegraded":"2024-04-01T01:32:49.721894+0000","last_fullsized":"2024-04-01T01:32:49.721894+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:20:23.103229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.690462+0000","last_change":"2024-04-01T01:31:16.377735+0000","last_active":"2024-04-01T01:32:19.690462+0000","last_peered":"2024-04-01T01:32:19.690462+0000","last_clean":"2024-04-01T01:32:19.690462+0000","last_became_active":"2024-04-01T01:31:16.377342+0000","last_became_peered":"2024-04-01T01:31:16.377342+0000","last_unstale":"2024-04-01T01:32:19.690462+0000","last_undegraded":"2024-04-01T01:32:19.690462+0000","last_fullsized":"2024-04-01T01:32:19.690462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:47.539223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683899+0000","last_change":"2024-04-01T01:31:20.573370+0000","last_active":"2024-04-01T01:32:19.683899+0000","last_peered":"2024-04-01T01:32:19.683899+0000","last_clean":"2024-04-01T01:32:19.683899+0000","last_became_active":"2024-04-01T01:31:20.572919+0000","last_became_peered":"2024-04-01T01:31:20.572919+0000","last_unstale":"2024-04-01T01:32:19.683899+0000","last_undegraded":"2024-04-01T01:32:19.683899+0000","last_fullsized":"2024-04-01T01:32:19.683899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:42.190918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.810468+0000","last_change":"2024-04-01T01:32:19.810468+0000","last_active":"2024-04-01T01:32:19.810468+0000","last_peered":"2024-04-01T01:32:19.810468+0000","last_clean":"2024-04-01T01:32:19.810468+0000","last_became_active":"2024-04-01T01:32:19.805517+0000","last_became_peered":"2024-04-01T01:32:19.805517+0000","last_unstale":"2024-04-01T01:32:19.810468+0000","last_undegraded":"2024-04-01T01:32:19.810468+0000","last_fullsized":"2024-04-01T01:32:19.810468+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:54:35.355761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.588732+0000","last_change":"2024-04-01T01:31:16.384130+0000","last_active":"2024-04-01T01:32:48.588732+0000","last_peered":"2024-04-01T01:32:48.588732+0000","last_clean":"2024-04-01T01:32:48.588732+0000","last_became_active":"2024-04-01T01:31:16.383400+0000","last_became_peered":"2024-04-01T01:31:16.383400+0000","last_unstale":"2024-04-01T01:32:48.588732+0000","last_undegraded":"2024-04-01T01:32:48.588732+0000","last_fullsized":"2024-04-01T01:32:48.588732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:22:10.860325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.686306+0000","last_change":"2024-04-01T01:31:20.571052+0000","last_active":"2024-04-01T01:32:19.686306+0000","last_peered":"2024-04-01T01:32:19.686306+0000","last_clean":"2024-04-01T01:32:19.686306+0000","last_became_active":"2024-04-01T01:31:20.570617+0000","last_became_peered":"2024-04-01T01:31:20.570617+0000","last_unstale":"2024-04-01T01:32:19.686306+0000","last_undegraded":"2024-04-01T01:32:19.686306+0000","last_fullsized":"2024-04-01T01:32:19.686306+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:53:51.195056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.689102+0000","last_change":"2024-04-01T01:31:15.369820+0000","last_active":"2024-04-01T01:32:19.689102+0000","last_peered":"2024-04-01T01:32:19.689102+0000","last_clean":"2024-04-01T01:32:19.689102+0000","last_became_active":"2024-04-01T01:31:15.369378+0000","last_became_peered":"2024-04-01T01:31:15.369378+0000","last_unstale":"2024-04-01T01:32:19.689102+0000","last_undegraded":"2024-04-01T01:32:19.689102+0000","last_fullsized":"2024-04-01T01:32:19.689102+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:37:34.351220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.602880+0000","last_change":"2024-04-01T01:31:16.383352+0000","last_active":"2024-04-01T01:32:48.602880+0000","last_peered":"2024-04-01T01:32:48.602880+0000","last_clean":"2024-04-01T01:32:48.602880+0000","last_became_active":"2024-04-01T01:31:16.382898+0000","last_became_peered":"2024-04-01T01:31:16.382898+0000","last_unstale":"2024-04-01T01:32:48.602880+0000","last_undegraded":"2024-04-01T01:32:48.602880+0000","last_fullsized":"2024-04-01T01:32:48.602880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:27.854459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.684722+0000","last_change":"2024-04-01T01:31:20.576578+0000","last_active":"2024-04-01T01:32:19.684722+0000","last_peered":"2024-04-01T01:32:19.684722+0000","last_clean":"2024-04-01T01:32:19.684722+0000","last_became_active":"2024-04-01T01:31:20.575996+0000","last_became_peered":"2024-04-01T01:31:20.575996+0000","last_unstale":"2024-04-01T01:32:19.684722+0000","last_undegraded":"2024-04-01T01:32:19.684722+0000","last_fullsized":"2024-04-01T01:32:19.684722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:52:08.808357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685084+0000","last_change":"2024-04-01T01:31:15.377246+0000","last_active":"2024-04-01T01:32:19.685084+0000","last_peered":"2024-04-01T01:32:19.685084+0000","last_clean":"2024-04-01T01:32:19.685084+0000","last_became_active":"2024-04-01T01:31:15.376957+0000","last_became_peered":"2024-04-01T01:31:15.376957+0000","last_unstale":"2024-04-01T01:32:19.685084+0000","last_undegraded":"2024-04-01T01:32:19.685084+0000","last_fullsized":"2024-04-01T01:32:19.685084+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:59:34.313984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.604754+0000","last_change":"2024-04-01T01:31:16.379031+0000","last_active":"2024-04-01T01:32:48.604754+0000","last_peered":"2024-04-01T01:32:48.604754+0000","last_clean":"2024-04-01T01:32:48.604754+0000","last_became_active":"2024-04-01T01:31:16.378593+0000","last_became_peered":"2024-04-01T01:31:16.378593+0000","last_unstale":"2024-04-01T01:32:48.604754+0000","last_undegraded":"2024-04-01T01:32:48.604754+0000","last_fullsized":"2024-04-01T01:32:48.604754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:31.816103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.686568+0000","last_change":"2024-04-01T01:31:20.578660+0000","last_active":"2024-04-01T01:32:19.686568+0000","last_peered":"2024-04-01T01:32:19.686568+0000","last_clean":"2024-04-01T01:32:19.686568+0000","last_became_active":"2024-04-01T01:31:20.578082+0000","last_became_peered":"2024-04-01T01:31:20.578082+0000","last_unstale":"2024-04-01T01:32:19.686568+0000","last_undegraded":"2024-04-01T01:32:19.686568+0000","last_fullsized":"2024-04-01T01:32:19.686568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:02:56.570816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":181,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:50.667044+0000","last_change":"2024-04-01T01:31:15.373428+0000","last_active":"2024-04-01T01:32:50.667044+0000","last_peered":"2024-04-01T01:32:50.667044+0000","last_clean":"2024-04-01T01:32:50.667044+0000","last_became_active":"2024-04-01T01:31:15.372965+0000","last_became_peered":"2024-04-01T01:31:15.372965+0000","last_unstale":"2024-04-01T01:32:50.667044+0000","last_undegraded":"2024-04-01T01:32:50.667044+0000","last_fullsized":"2024-04-01T01:32:50.667044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:55:42.211055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":149,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.597812+0000","last_change":"2024-04-01T01:31:16.382961+0000","last_active":"2024-04-01T01:32:48.597812+0000","last_peered":"2024-04-01T01:32:48.597812+0000","last_clean":"2024-04-01T01:32:48.597812+0000","last_became_active":"2024-04-01T01:31:16.382527+0000","last_became_peered":"2024-04-01T01:31:16.382527+0000","last_unstale":"2024-04-01T01:32:48.597812+0000","last_undegraded":"2024-04-01T01:32:48.597812+0000","last_fullsized":"2024-04-01T01:32:48.597812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:46:20.250064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.682963+0000","last_change":"2024-04-01T01:31:20.571842+0000","last_active":"2024-04-01T01:32:19.682963+0000","last_peered":"2024-04-01T01:32:19.682963+0000","last_clean":"2024-04-01T01:32:19.682963+0000","last_became_active":"2024-04-01T01:31:20.571394+0000","last_became_peered":"2024-04-01T01:31:20.571394+0000","last_unstale":"2024-04-01T01:32:19.682963+0000","last_undegraded":"2024-04-01T01:32:19.682963+0000","last_fullsized":"2024-04-01T01:32:19.682963+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:51:36.512277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.686377+0000","last_change":"2024-04-01T01:31:15.379737+0000","last_active":"2024-04-01T01:32:19.686377+0000","last_peered":"2024-04-01T01:32:19.686377+0000","last_clean":"2024-04-01T01:32:19.686377+0000","last_became_active":"2024-04-01T01:31:15.379158+0000","last_became_peered":"2024-04-01T01:31:15.379158+0000","last_unstale":"2024-04-01T01:32:19.686377+0000","last_undegraded":"2024-04-01T01:32:19.686377+0000","last_fullsized":"2024-04-01T01:32:19.686377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:12:44.474772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.589078+0000","last_change":"2024-04-01T01:31:16.386059+0000","last_active":"2024-04-01T01:32:48.589078+0000","last_peered":"2024-04-01T01:32:48.589078+0000","last_clean":"2024-04-01T01:32:48.589078+0000","last_became_active":"2024-04-01T01:31:16.384551+0000","last_became_peered":"2024-04-01T01:31:16.384551+0000","last_unstale":"2024-04-01T01:32:48.589078+0000","last_undegraded":"2024-04-01T01:32:48.589078+0000","last_fullsized":"2024-04-01T01:32:48.589078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:52:16.594485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683310+0000","last_change":"2024-04-01T01:31:20.566937+0000","last_active":"2024-04-01T01:32:19.683310+0000","last_peered":"2024-04-01T01:32:19.683310+0000","last_clean":"2024-04-01T01:32:19.683310+0000","last_became_active":"2024-04-01T01:31:20.566447+0000","last_became_peered":"2024-04-01T01:31:20.566447+0000","last_unstale":"2024-04-01T01:32:19.683310+0000","last_undegraded":"2024-04-01T01:32:19.683310+0000","last_fullsized":"2024-04-01T01:32:19.683310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:17:37.845166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685433+0000","last_change":"2024-04-01T01:31:15.375669+0000","last_active":"2024-04-01T01:32:19.685433+0000","last_peered":"2024-04-01T01:32:19.685433+0000","last_clean":"2024-04-01T01:32:19.685433+0000","last_became_active":"2024-04-01T01:31:15.375353+0000","last_became_peered":"2024-04-01T01:31:15.375353+0000","last_unstale":"2024-04-01T01:32:19.685433+0000","last_undegraded":"2024-04-01T01:32:19.685433+0000","last_fullsized":"2024-04-01T01:32:19.685433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:47:43.096898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.599676+0000","last_change":"2024-04-01T01:31:16.388399+0000","last_active":"2024-04-01T01:32:48.599676+0000","last_peered":"2024-04-01T01:32:48.599676+0000","last_clean":"2024-04-01T01:32:48.599676+0000","last_became_active":"2024-04-01T01:31:16.388059+0000","last_became_peered":"2024-04-01T01:31:16.388059+0000","last_unstale":"2024-04-01T01:32:48.599676+0000","last_undegraded":"2024-04-01T01:32:48.599676+0000","last_fullsized":"2024-04-01T01:32:48.599676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:59:50.907802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.693040+0000","last_change":"2024-04-01T01:32:19.692098+0000","last_active":"2024-04-01T01:32:19.693040+0000","last_peered":"2024-04-01T01:32:19.693040+0000","last_clean":"2024-04-01T01:32:19.693040+0000","last_became_active":"2024-04-01T01:32:19.691673+0000","last_became_peered":"2024-04-01T01:32:19.691673+0000","last_unstale":"2024-04-01T01:32:19.693040+0000","last_undegraded":"2024-04-01T01:32:19.693040+0000","last_fullsized":"2024-04-01T01:32:19.693040+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:14:17.300517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"91'37","reported_seq":225,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:50.721396+0000","last_change":"2024-04-01T01:31:15.376421+0000","last_active":"2024-04-01T01:32:50.721396+0000","last_peered":"2024-04-01T01:32:50.721396+0000","last_clean":"2024-04-01T01:32:50.721396+0000","last_became_active":"2024-04-01T01:31:15.375985+0000","last_became_peered":"2024-04-01T01:31:15.375985+0000","last_unstale":"2024-04-01T01:32:50.721396+0000","last_undegraded":"2024-04-01T01:32:50.721396+0000","last_fullsized":"2024-04-01T01:32:50.721396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:37:29.121436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63605,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":161,"num_read_kb":143,"num_write":39,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.689863+0000","last_change":"2024-04-01T01:31:16.380641+0000","last_active":"2024-04-01T01:32:19.689863+0000","last_peered":"2024-04-01T01:32:19.689863+0000","last_clean":"2024-04-01T01:32:19.689863+0000","last_became_active":"2024-04-01T01:31:16.380300+0000","last_became_peered":"2024-04-01T01:31:16.380300+0000","last_unstale":"2024-04-01T01:32:19.689863+0000","last_undegraded":"2024-04-01T01:32:19.689863+0000","last_fullsized":"2024-04-01T01:32:19.689863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:26:08.240694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685977+0000","last_change":"2024-04-01T01:31:20.576094+0000","last_active":"2024-04-01T01:32:19.685977+0000","last_peered":"2024-04-01T01:32:19.685977+0000","last_clean":"2024-04-01T01:32:19.685977+0000","last_became_active":"2024-04-01T01:31:20.575674+0000","last_became_peered":"2024-04-01T01:31:20.575674+0000","last_unstale":"2024-04-01T01:32:19.685977+0000","last_undegraded":"2024-04-01T01:32:19.685977+0000","last_fullsized":"2024-04-01T01:32:19.685977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:42:19.352398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683520+0000","last_change":"2024-04-01T01:31:15.375136+0000","last_active":"2024-04-01T01:32:19.683520+0000","last_peered":"2024-04-01T01:32:19.683520+0000","last_clean":"2024-04-01T01:32:19.683520+0000","last_became_active":"2024-04-01T01:31:15.374813+0000","last_became_peered":"2024-04-01T01:31:15.374813+0000","last_unstale":"2024-04-01T01:32:19.683520+0000","last_undegraded":"2024-04-01T01:32:19.683520+0000","last_fullsized":"2024-04-01T01:32:19.683520+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:19:05.490103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.593781+0000","last_change":"2024-04-01T01:31:16.380207+0000","last_active":"2024-04-01T01:32:48.593781+0000","last_peered":"2024-04-01T01:32:48.593781+0000","last_clean":"2024-04-01T01:32:48.593781+0000","last_became_active":"2024-04-01T01:31:16.379891+0000","last_became_peered":"2024-04-01T01:31:16.379891+0000","last_unstale":"2024-04-01T01:32:48.593781+0000","last_undegraded":"2024-04-01T01:32:48.593781+0000","last_fullsized":"2024-04-01T01:32:48.593781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:12:36.417162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.686023+0000","last_change":"2024-04-01T01:31:20.569978+0000","last_active":"2024-04-01T01:32:19.686023+0000","last_peered":"2024-04-01T01:32:19.686023+0000","last_clean":"2024-04-01T01:32:19.686023+0000","last_became_active":"2024-04-01T01:31:20.568298+0000","last_became_peered":"2024-04-01T01:31:20.568298+0000","last_unstale":"2024-04-01T01:32:19.686023+0000","last_undegraded":"2024-04-01T01:32:19.686023+0000","last_fullsized":"2024-04-01T01:32:19.686023+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:42:16.439284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685854+0000","last_change":"2024-04-01T01:31:15.380143+0000","last_active":"2024-04-01T01:32:19.685854+0000","last_peered":"2024-04-01T01:32:19.685854+0000","last_clean":"2024-04-01T01:32:19.685854+0000","last_became_active":"2024-04-01T01:31:15.379835+0000","last_became_peered":"2024-04-01T01:31:15.379835+0000","last_unstale":"2024-04-01T01:32:19.685854+0000","last_undegraded":"2024-04-01T01:32:19.685854+0000","last_fullsized":"2024-04-01T01:32:19.685854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:28:34.213626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.601715+0000","last_change":"2024-04-01T01:31:16.380290+0000","last_active":"2024-04-01T01:32:48.601715+0000","last_peered":"2024-04-01T01:32:48.601715+0000","last_clean":"2024-04-01T01:32:48.601715+0000","last_became_active":"2024-04-01T01:31:16.379983+0000","last_became_peered":"2024-04-01T01:31:16.379983+0000","last_unstale":"2024-04-01T01:32:48.601715+0000","last_undegraded":"2024-04-01T01:32:48.601715+0000","last_fullsized":"2024-04-01T01:32:48.601715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:50.135454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683163+0000","last_change":"2024-04-01T01:31:20.573500+0000","last_active":"2024-04-01T01:32:19.683163+0000","last_peered":"2024-04-01T01:32:19.683163+0000","last_clean":"2024-04-01T01:32:19.683163+0000","last_became_active":"2024-04-01T01:31:20.573101+0000","last_became_peered":"2024-04-01T01:31:20.573101+0000","last_unstale":"2024-04-01T01:32:19.683163+0000","last_undegraded":"2024-04-01T01:32:19.683163+0000","last_fullsized":"2024-04-01T01:32:19.683163+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:11:27.881703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":111,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:50.673472+0000","last_change":"2024-04-01T01:31:15.377265+0000","last_active":"2024-04-01T01:32:50.673472+0000","last_peered":"2024-04-01T01:32:50.673472+0000","last_clean":"2024-04-01T01:32:50.673472+0000","last_became_active":"2024-04-01T01:31:15.376889+0000","last_became_peered":"2024-04-01T01:31:15.376889+0000","last_unstale":"2024-04-01T01:32:50.673472+0000","last_undegraded":"2024-04-01T01:32:50.673472+0000","last_fullsized":"2024-04-01T01:32:50.673472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:43:31.582866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.591850+0000","last_change":"2024-04-01T01:31:16.384982+0000","last_active":"2024-04-01T01:32:48.591850+0000","last_peered":"2024-04-01T01:32:48.591850+0000","last_clean":"2024-04-01T01:32:48.591850+0000","last_became_active":"2024-04-01T01:31:16.384663+0000","last_became_peered":"2024-04-01T01:31:16.384663+0000","last_unstale":"2024-04-01T01:32:48.591850+0000","last_undegraded":"2024-04-01T01:32:48.591850+0000","last_fullsized":"2024-04-01T01:32:48.591850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:14:57.351768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.689033+0000","last_change":"2024-04-01T01:31:20.578761+0000","last_active":"2024-04-01T01:32:19.689033+0000","last_peered":"2024-04-01T01:32:19.689033+0000","last_clean":"2024-04-01T01:32:19.689033+0000","last_became_active":"2024-04-01T01:31:20.578204+0000","last_became_peered":"2024-04-01T01:31:20.578204+0000","last_unstale":"2024-04-01T01:32:19.689033+0000","last_undegraded":"2024-04-01T01:32:19.689033+0000","last_fullsized":"2024-04-01T01:32:19.689033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:19:33.584872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.689295+0000","last_change":"2024-04-01T01:31:15.380773+0000","last_active":"2024-04-01T01:32:19.689295+0000","last_peered":"2024-04-01T01:32:19.689295+0000","last_clean":"2024-04-01T01:32:19.689295+0000","last_became_active":"2024-04-01T01:31:15.380472+0000","last_became_peered":"2024-04-01T01:31:15.380472+0000","last_unstale":"2024-04-01T01:32:19.689295+0000","last_undegraded":"2024-04-01T01:32:19.689295+0000","last_fullsized":"2024-04-01T01:32:19.689295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:34:28.294500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.599193+0000","last_change":"2024-04-01T01:31:16.374658+0000","last_active":"2024-04-01T01:32:48.599193+0000","last_peered":"2024-04-01T01:32:48.599193+0000","last_clean":"2024-04-01T01:32:48.599193+0000","last_became_active":"2024-04-01T01:31:16.373596+0000","last_became_peered":"2024-04-01T01:31:16.373596+0000","last_unstale":"2024-04-01T01:32:48.599193+0000","last_undegraded":"2024-04-01T01:32:48.599193+0000","last_fullsized":"2024-04-01T01:32:48.599193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:28:38.477719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685238+0000","last_change":"2024-04-01T01:31:20.577449+0000","last_active":"2024-04-01T01:32:19.685238+0000","last_peered":"2024-04-01T01:32:19.685238+0000","last_clean":"2024-04-01T01:32:19.685238+0000","last_became_active":"2024-04-01T01:31:20.577026+0000","last_became_peered":"2024-04-01T01:31:20.577026+0000","last_unstale":"2024-04-01T01:32:19.685238+0000","last_undegraded":"2024-04-01T01:32:19.685238+0000","last_fullsized":"2024-04-01T01:32:19.685238+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:44:48.080304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.689118+0000","last_change":"2024-04-01T01:31:15.373885+0000","last_active":"2024-04-01T01:32:19.689118+0000","last_peered":"2024-04-01T01:32:19.689118+0000","last_clean":"2024-04-01T01:32:19.689118+0000","last_became_active":"2024-04-01T01:31:15.373578+0000","last_became_peered":"2024-04-01T01:31:15.373578+0000","last_unstale":"2024-04-01T01:32:19.689118+0000","last_undegraded":"2024-04-01T01:32:19.689118+0000","last_fullsized":"2024-04-01T01:32:19.689118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:52.641515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.588417+0000","last_change":"2024-04-01T01:31:16.379997+0000","last_active":"2024-04-01T01:32:48.588417+0000","last_peered":"2024-04-01T01:32:48.588417+0000","last_clean":"2024-04-01T01:32:48.588417+0000","last_became_active":"2024-04-01T01:31:16.379664+0000","last_became_peered":"2024-04-01T01:31:16.379664+0000","last_unstale":"2024-04-01T01:32:48.588417+0000","last_undegraded":"2024-04-01T01:32:48.588417+0000","last_fullsized":"2024-04-01T01:32:48.588417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:37:17.432095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.690581+0000","last_change":"2024-04-01T01:31:20.569835+0000","last_active":"2024-04-01T01:32:19.690581+0000","last_peered":"2024-04-01T01:32:19.690581+0000","last_clean":"2024-04-01T01:32:19.690581+0000","last_became_active":"2024-04-01T01:31:20.569195+0000","last_became_peered":"2024-04-01T01:31:20.569195+0000","last_unstale":"2024-04-01T01:32:19.690581+0000","last_undegraded":"2024-04-01T01:32:19.690581+0000","last_fullsized":"2024-04-01T01:32:19.690581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:10:18.278575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683291+0000","last_change":"2024-04-01T01:31:15.377679+0000","last_active":"2024-04-01T01:32:19.683291+0000","last_peered":"2024-04-01T01:32:19.683291+0000","last_clean":"2024-04-01T01:32:19.683291+0000","last_became_active":"2024-04-01T01:31:15.377351+0000","last_became_peered":"2024-04-01T01:31:15.377351+0000","last_unstale":"2024-04-01T01:32:19.683291+0000","last_undegraded":"2024-04-01T01:32:19.683291+0000","last_fullsized":"2024-04-01T01:32:19.683291+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:22:41.216563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.593718+0000","last_change":"2024-04-01T01:31:16.380997+0000","last_active":"2024-04-01T01:32:48.593718+0000","last_peered":"2024-04-01T01:32:48.593718+0000","last_clean":"2024-04-01T01:32:48.593718+0000","last_became_active":"2024-04-01T01:31:16.380692+0000","last_became_peered":"2024-04-01T01:31:16.380692+0000","last_unstale":"2024-04-01T01:32:48.593718+0000","last_undegraded":"2024-04-01T01:32:48.593718+0000","last_fullsized":"2024-04-01T01:32:48.593718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:09:51.220849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.689069+0000","last_change":"2024-04-01T01:31:20.573191+0000","last_active":"2024-04-01T01:32:19.689069+0000","last_peered":"2024-04-01T01:32:19.689069+0000","last_clean":"2024-04-01T01:32:19.689069+0000","last_became_active":"2024-04-01T01:31:20.572669+0000","last_became_peered":"2024-04-01T01:31:20.572669+0000","last_unstale":"2024-04-01T01:32:19.689069+0000","last_undegraded":"2024-04-01T01:32:19.689069+0000","last_fullsized":"2024-04-01T01:32:19.689069+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:01:56.491271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.655261+0000","last_change":"2024-04-01T01:31:15.379566+0000","last_active":"2024-04-01T01:32:48.655261+0000","last_peered":"2024-04-01T01:32:48.655261+0000","last_clean":"2024-04-01T01:32:48.655261+0000","last_became_active":"2024-04-01T01:31:15.379213+0000","last_became_peered":"2024-04-01T01:31:15.379213+0000","last_unstale":"2024-04-01T01:32:48.655261+0000","last_undegraded":"2024-04-01T01:32:48.655261+0000","last_fullsized":"2024-04-01T01:32:48.655261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:34:17.974422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.599236+0000","last_change":"2024-04-01T01:31:16.376218+0000","last_active":"2024-04-01T01:32:48.599236+0000","last_peered":"2024-04-01T01:32:48.599236+0000","last_clean":"2024-04-01T01:32:48.599236+0000","last_became_active":"2024-04-01T01:31:16.375887+0000","last_became_peered":"2024-04-01T01:31:16.375887+0000","last_unstale":"2024-04-01T01:32:48.599236+0000","last_undegraded":"2024-04-01T01:32:48.599236+0000","last_fullsized":"2024-04-01T01:32:48.599236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:27:03.723827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.688754+0000","last_change":"2024-04-01T01:31:20.576461+0000","last_active":"2024-04-01T01:32:19.688754+0000","last_peered":"2024-04-01T01:32:19.688754+0000","last_clean":"2024-04-01T01:32:19.688754+0000","last_became_active":"2024-04-01T01:31:20.575985+0000","last_became_peered":"2024-04-01T01:31:20.575985+0000","last_unstale":"2024-04-01T01:32:19.688754+0000","last_undegraded":"2024-04-01T01:32:19.688754+0000","last_fullsized":"2024-04-01T01:32:19.688754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:19:02.714455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.592340+0000","last_change":"2024-04-01T01:31:16.376456+0000","last_active":"2024-04-01T01:32:48.592340+0000","last_peered":"2024-04-01T01:32:48.592340+0000","last_clean":"2024-04-01T01:32:48.592340+0000","last_became_active":"2024-04-01T01:31:16.375445+0000","last_became_peered":"2024-04-01T01:31:16.375445+0000","last_unstale":"2024-04-01T01:32:48.592340+0000","last_undegraded":"2024-04-01T01:32:48.592340+0000","last_fullsized":"2024-04-01T01:32:48.592340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:07:11.802799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.670194+0000","last_change":"2024-04-01T01:31:15.382702+0000","last_active":"2024-04-01T01:32:48.670194+0000","last_peered":"2024-04-01T01:32:48.670194+0000","last_clean":"2024-04-01T01:32:48.670194+0000","last_became_active":"2024-04-01T01:31:15.382377+0000","last_became_peered":"2024-04-01T01:31:15.382377+0000","last_unstale":"2024-04-01T01:32:48.670194+0000","last_undegraded":"2024-04-01T01:32:48.670194+0000","last_fullsized":"2024-04-01T01:32:48.670194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:45.080992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.704607+0000","last_change":"2024-04-01T01:32:19.704607+0000","last_active":"2024-04-01T01:32:19.704607+0000","last_peered":"2024-04-01T01:32:19.704607+0000","last_clean":"2024-04-01T01:32:19.704607+0000","last_became_active":"2024-04-01T01:32:19.694650+0000","last_became_peered":"2024-04-01T01:32:19.694650+0000","last_unstale":"2024-04-01T01:32:19.704607+0000","last_undegraded":"2024-04-01T01:32:19.704607+0000","last_fullsized":"2024-04-01T01:32:19.704607+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:48:55.439187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.581815+0000","last_change":"2024-04-01T01:31:16.387981+0000","last_active":"2024-04-01T01:32:48.581815+0000","last_peered":"2024-04-01T01:32:48.581815+0000","last_clean":"2024-04-01T01:32:48.581815+0000","last_became_active":"2024-04-01T01:31:16.387594+0000","last_became_peered":"2024-04-01T01:31:16.387594+0000","last_unstale":"2024-04-01T01:32:48.581815+0000","last_undegraded":"2024-04-01T01:32:48.581815+0000","last_fullsized":"2024-04-01T01:32:48.581815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:17.148752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683288+0000","last_change":"2024-04-01T01:31:15.375212+0000","last_active":"2024-04-01T01:32:19.683288+0000","last_peered":"2024-04-01T01:32:19.683288+0000","last_clean":"2024-04-01T01:32:19.683288+0000","last_became_active":"2024-04-01T01:31:15.374882+0000","last_became_peered":"2024-04-01T01:31:15.374882+0000","last_unstale":"2024-04-01T01:32:19.683288+0000","last_undegraded":"2024-04-01T01:32:19.683288+0000","last_fullsized":"2024-04-01T01:32:19.683288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:56:23.412025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685726+0000","last_change":"2024-04-01T01:30:03.538590+0000","last_active":"2024-04-01T01:32:19.685726+0000","last_peered":"2024-04-01T01:32:19.685726+0000","last_clean":"2024-04-01T01:32:19.685726+0000","last_became_active":"2024-04-01T01:30:03.538098+0000","last_became_peered":"2024-04-01T01:30:03.538098+0000","last_unstale":"2024-04-01T01:32:19.685726+0000","last_undegraded":"2024-04-01T01:32:19.685726+0000","last_fullsized":"2024-04-01T01:32:19.685726+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:26:43.445198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:26:43.445198+0000","last_clean_scrub_stamp":"2024-04-01T01:26:43.445198+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:22.207795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.686086+0000","last_change":"2024-04-01T01:31:20.573260+0000","last_active":"2024-04-01T01:32:19.686086+0000","last_peered":"2024-04-01T01:32:19.686086+0000","last_clean":"2024-04-01T01:32:19.686086+0000","last_became_active":"2024-04-01T01:31:20.572846+0000","last_became_peered":"2024-04-01T01:31:20.572846+0000","last_unstale":"2024-04-01T01:32:19.686086+0000","last_undegraded":"2024-04-01T01:32:19.686086+0000","last_fullsized":"2024-04-01T01:32:19.686086+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:24.986070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.593706+0000","last_change":"2024-04-01T01:31:16.384782+0000","last_active":"2024-04-01T01:32:48.593706+0000","last_peered":"2024-04-01T01:32:48.593706+0000","last_clean":"2024-04-01T01:32:48.593706+0000","last_became_active":"2024-04-01T01:31:16.384420+0000","last_became_peered":"2024-04-01T01:31:16.384420+0000","last_unstale":"2024-04-01T01:32:48.593706+0000","last_undegraded":"2024-04-01T01:32:48.593706+0000","last_fullsized":"2024-04-01T01:32:48.593706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:00:20.902022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685788+0000","last_change":"2024-04-01T01:31:15.381853+0000","last_active":"2024-04-01T01:32:19.685788+0000","last_peered":"2024-04-01T01:32:19.685788+0000","last_clean":"2024-04-01T01:32:19.685788+0000","last_became_active":"2024-04-01T01:31:15.381496+0000","last_became_peered":"2024-04-01T01:31:15.381496+0000","last_unstale":"2024-04-01T01:32:19.685788+0000","last_undegraded":"2024-04-01T01:32:19.685788+0000","last_fullsized":"2024-04-01T01:32:19.685788+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:07:51.440351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685308+0000","last_change":"2024-04-01T01:31:20.568705+0000","last_active":"2024-04-01T01:32:19.685308+0000","last_peered":"2024-04-01T01:32:19.685308+0000","last_clean":"2024-04-01T01:32:19.685308+0000","last_became_active":"2024-04-01T01:31:20.568252+0000","last_became_peered":"2024-04-01T01:31:20.568252+0000","last_unstale":"2024-04-01T01:32:19.685308+0000","last_undegraded":"2024-04-01T01:32:19.685308+0000","last_fullsized":"2024-04-01T01:32:19.685308+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:22:28.509510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.700251+0000","last_change":"2024-04-01T01:32:19.699781+0000","last_active":"2024-04-01T01:32:19.700251+0000","last_peered":"2024-04-01T01:32:19.700251+0000","last_clean":"2024-04-01T01:32:19.700251+0000","last_became_active":"2024-04-01T01:32:19.689593+0000","last_became_peered":"2024-04-01T01:32:19.689593+0000","last_unstale":"2024-04-01T01:32:19.700251+0000","last_undegraded":"2024-04-01T01:32:19.700251+0000","last_fullsized":"2024-04-01T01:32:19.700251+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:44:21.922097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.584119+0000","last_change":"2024-04-01T01:31:16.377002+0000","last_active":"2024-04-01T01:32:48.584119+0000","last_peered":"2024-04-01T01:32:48.584119+0000","last_clean":"2024-04-01T01:32:48.584119+0000","last_became_active":"2024-04-01T01:31:16.376679+0000","last_became_peered":"2024-04-01T01:31:16.376679+0000","last_unstale":"2024-04-01T01:32:48.584119+0000","last_undegraded":"2024-04-01T01:32:48.584119+0000","last_fullsized":"2024-04-01T01:32:48.584119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:50:24.126192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.686795+0000","last_change":"2024-04-01T01:31:20.577757+0000","last_active":"2024-04-01T01:32:19.686795+0000","last_peered":"2024-04-01T01:32:19.686795+0000","last_clean":"2024-04-01T01:32:19.686795+0000","last_became_active":"2024-04-01T01:31:20.577378+0000","last_became_peered":"2024-04-01T01:31:20.577378+0000","last_unstale":"2024-04-01T01:32:19.686795+0000","last_undegraded":"2024-04-01T01:32:19.686795+0000","last_fullsized":"2024-04-01T01:32:19.686795+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:48:04.511229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'13","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683655+0000","last_change":"2024-04-01T01:31:15.383201+0000","last_active":"2024-04-01T01:32:19.683655+0000","last_peered":"2024-04-01T01:32:19.683655+0000","last_clean":"2024-04-01T01:32:19.683655+0000","last_became_active":"2024-04-01T01:31:15.382320+0000","last_became_peered":"2024-04-01T01:31:15.382320+0000","last_unstale":"2024-04-01T01:32:19.683655+0000","last_undegraded":"2024-04-01T01:32:19.683655+0000","last_fullsized":"2024-04-01T01:32:19.683655+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:15:47.138024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.603330+0000","last_change":"2024-04-01T01:31:16.386936+0000","last_active":"2024-04-01T01:32:48.603330+0000","last_peered":"2024-04-01T01:32:48.603330+0000","last_clean":"2024-04-01T01:32:48.603330+0000","last_became_active":"2024-04-01T01:31:16.384448+0000","last_became_peered":"2024-04-01T01:31:16.384448+0000","last_unstale":"2024-04-01T01:32:48.603330+0000","last_undegraded":"2024-04-01T01:32:48.603330+0000","last_fullsized":"2024-04-01T01:32:48.603330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:16:42.243250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683611+0000","last_change":"2024-04-01T01:31:20.565570+0000","last_active":"2024-04-01T01:32:19.683611+0000","last_peered":"2024-04-01T01:32:19.683611+0000","last_clean":"2024-04-01T01:32:19.683611+0000","last_became_active":"2024-04-01T01:31:20.565141+0000","last_became_peered":"2024-04-01T01:31:20.565141+0000","last_unstale":"2024-04-01T01:32:19.683611+0000","last_undegraded":"2024-04-01T01:32:19.683611+0000","last_fullsized":"2024-04-01T01:32:19.683611+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:47:12.844970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683841+0000","last_change":"2024-04-01T01:31:15.381343+0000","last_active":"2024-04-01T01:32:19.683841+0000","last_peered":"2024-04-01T01:32:19.683841+0000","last_clean":"2024-04-01T01:32:19.683841+0000","last_became_active":"2024-04-01T01:31:15.380994+0000","last_became_peered":"2024-04-01T01:31:15.380994+0000","last_unstale":"2024-04-01T01:32:19.683841+0000","last_undegraded":"2024-04-01T01:32:19.683841+0000","last_fullsized":"2024-04-01T01:32:19.683841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:19:10.710285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.607597+0000","last_change":"2024-04-01T01:31:16.383951+0000","last_active":"2024-04-01T01:32:48.607597+0000","last_peered":"2024-04-01T01:32:48.607597+0000","last_clean":"2024-04-01T01:32:48.607597+0000","last_became_active":"2024-04-01T01:31:16.383657+0000","last_became_peered":"2024-04-01T01:31:16.383657+0000","last_unstale":"2024-04-01T01:32:48.607597+0000","last_undegraded":"2024-04-01T01:32:48.607597+0000","last_fullsized":"2024-04-01T01:32:48.607597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:55:45.216479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683439+0000","last_change":"2024-04-01T01:31:20.569416+0000","last_active":"2024-04-01T01:32:19.683439+0000","last_peered":"2024-04-01T01:32:19.683439+0000","last_clean":"2024-04-01T01:32:19.683439+0000","last_became_active":"2024-04-01T01:31:20.569007+0000","last_became_peered":"2024-04-01T01:31:20.569007+0000","last_unstale":"2024-04-01T01:32:19.683439+0000","last_undegraded":"2024-04-01T01:32:19.683439+0000","last_fullsized":"2024-04-01T01:32:19.683439+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:25:36.770919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.804186+0000","last_change":"2024-04-01T01:31:15.373444+0000","last_active":"2024-04-01T01:32:19.804186+0000","last_peered":"2024-04-01T01:32:19.804186+0000","last_clean":"2024-04-01T01:32:19.804186+0000","last_became_active":"2024-04-01T01:31:15.373094+0000","last_became_peered":"2024-04-01T01:31:15.373094+0000","last_unstale":"2024-04-01T01:32:19.804186+0000","last_undegraded":"2024-04-01T01:32:19.804186+0000","last_fullsized":"2024-04-01T01:32:19.804186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:26:27.443832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.688842+0000","last_change":"2024-04-01T01:31:16.381441+0000","last_active":"2024-04-01T01:32:19.688842+0000","last_peered":"2024-04-01T01:32:19.688842+0000","last_clean":"2024-04-01T01:32:19.688842+0000","last_became_active":"2024-04-01T01:31:16.381092+0000","last_became_peered":"2024-04-01T01:31:16.381092+0000","last_unstale":"2024-04-01T01:32:19.688842+0000","last_undegraded":"2024-04-01T01:32:19.688842+0000","last_fullsized":"2024-04-01T01:32:19.688842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:47:00.229090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683708+0000","last_change":"2024-04-01T01:31:20.569672+0000","last_active":"2024-04-01T01:32:19.683708+0000","last_peered":"2024-04-01T01:32:19.683708+0000","last_clean":"2024-04-01T01:32:19.683708+0000","last_became_active":"2024-04-01T01:31:20.569276+0000","last_became_peered":"2024-04-01T01:31:20.569276+0000","last_unstale":"2024-04-01T01:32:19.683708+0000","last_undegraded":"2024-04-01T01:32:19.683708+0000","last_fullsized":"2024-04-01T01:32:19.683708+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:39:03.331428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.686644+0000","last_change":"2024-04-01T01:31:15.376094+0000","last_active":"2024-04-01T01:32:19.686644+0000","last_peered":"2024-04-01T01:32:19.686644+0000","last_clean":"2024-04-01T01:32:19.686644+0000","last_became_active":"2024-04-01T01:31:15.375169+0000","last_became_peered":"2024-04-01T01:31:15.375169+0000","last_unstale":"2024-04-01T01:32:19.686644+0000","last_undegraded":"2024-04-01T01:32:19.686644+0000","last_fullsized":"2024-04-01T01:32:19.686644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:18:20.710644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.608411+0000","last_change":"2024-04-01T01:31:16.378514+0000","last_active":"2024-04-01T01:32:48.608411+0000","last_peered":"2024-04-01T01:32:48.608411+0000","last_clean":"2024-04-01T01:32:48.608411+0000","last_became_active":"2024-04-01T01:31:16.377990+0000","last_became_peered":"2024-04-01T01:31:16.377990+0000","last_unstale":"2024-04-01T01:32:48.608411+0000","last_undegraded":"2024-04-01T01:32:48.608411+0000","last_fullsized":"2024-04-01T01:32:48.608411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:27:29.829775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683420+0000","last_change":"2024-04-01T01:31:20.567892+0000","last_active":"2024-04-01T01:32:19.683420+0000","last_peered":"2024-04-01T01:32:19.683420+0000","last_clean":"2024-04-01T01:32:19.683420+0000","last_became_active":"2024-04-01T01:31:20.567525+0000","last_became_peered":"2024-04-01T01:31:20.567525+0000","last_unstale":"2024-04-01T01:32:19.683420+0000","last_undegraded":"2024-04-01T01:32:19.683420+0000","last_fullsized":"2024-04-01T01:32:19.683420+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:57.828691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.703872+0000","last_change":"2024-04-01T01:32:19.703414+0000","last_active":"2024-04-01T01:32:19.703872+0000","last_peered":"2024-04-01T01:32:19.703872+0000","last_clean":"2024-04-01T01:32:19.703872+0000","last_became_active":"2024-04-01T01:32:19.693581+0000","last_became_peered":"2024-04-01T01:32:19.693581+0000","last_unstale":"2024-04-01T01:32:19.703872+0000","last_undegraded":"2024-04-01T01:32:19.703872+0000","last_fullsized":"2024-04-01T01:32:19.703872+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:53:17.897154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.602653+0000","last_change":"2024-04-01T01:31:16.387570+0000","last_active":"2024-04-01T01:32:48.602653+0000","last_peered":"2024-04-01T01:32:48.602653+0000","last_clean":"2024-04-01T01:32:48.602653+0000","last_became_active":"2024-04-01T01:31:16.387200+0000","last_became_peered":"2024-04-01T01:31:16.387200+0000","last_unstale":"2024-04-01T01:32:48.602653+0000","last_undegraded":"2024-04-01T01:32:48.602653+0000","last_fullsized":"2024-04-01T01:32:48.602653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:52:27.813722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683718+0000","last_change":"2024-04-01T01:31:20.569668+0000","last_active":"2024-04-01T01:32:19.683718+0000","last_peered":"2024-04-01T01:32:19.683718+0000","last_clean":"2024-04-01T01:32:19.683718+0000","last_became_active":"2024-04-01T01:31:20.569148+0000","last_became_peered":"2024-04-01T01:31:20.569148+0000","last_unstale":"2024-04-01T01:32:19.683718+0000","last_undegraded":"2024-04-01T01:32:19.683718+0000","last_fullsized":"2024-04-01T01:32:19.683718+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:20:56.048052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.689799+0000","last_change":"2024-04-01T01:31:15.379323+0000","last_active":"2024-04-01T01:32:19.689799+0000","last_peered":"2024-04-01T01:32:19.689799+0000","last_clean":"2024-04-01T01:32:19.689799+0000","last_became_active":"2024-04-01T01:31:15.379023+0000","last_became_peered":"2024-04-01T01:31:15.379023+0000","last_unstale":"2024-04-01T01:32:19.689799+0000","last_undegraded":"2024-04-01T01:32:19.689799+0000","last_fullsized":"2024-04-01T01:32:19.689799+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:05:37.709040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.600667+0000","last_change":"2024-04-01T01:31:16.387250+0000","last_active":"2024-04-01T01:32:48.600667+0000","last_peered":"2024-04-01T01:32:48.600667+0000","last_clean":"2024-04-01T01:32:48.600667+0000","last_became_active":"2024-04-01T01:31:16.386940+0000","last_became_peered":"2024-04-01T01:31:16.386940+0000","last_unstale":"2024-04-01T01:32:48.600667+0000","last_undegraded":"2024-04-01T01:32:48.600667+0000","last_fullsized":"2024-04-01T01:32:48.600667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:19:34.886738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.804516+0000","last_change":"2024-04-01T01:31:20.573228+0000","last_active":"2024-04-01T01:32:19.804516+0000","last_peered":"2024-04-01T01:32:19.804516+0000","last_clean":"2024-04-01T01:32:19.804516+0000","last_became_active":"2024-04-01T01:31:20.572681+0000","last_became_peered":"2024-04-01T01:31:20.572681+0000","last_unstale":"2024-04-01T01:32:19.804516+0000","last_undegraded":"2024-04-01T01:32:19.804516+0000","last_fullsized":"2024-04-01T01:32:19.804516+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:58:17.812966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:46.662981+0000","last_change":"2024-04-01T01:31:15.381477+0000","last_active":"2024-04-01T01:32:46.662981+0000","last_peered":"2024-04-01T01:32:46.662981+0000","last_clean":"2024-04-01T01:32:46.662981+0000","last_became_active":"2024-04-01T01:31:15.381166+0000","last_became_peered":"2024-04-01T01:31:15.381166+0000","last_unstale":"2024-04-01T01:32:46.662981+0000","last_undegraded":"2024-04-01T01:32:46.662981+0000","last_fullsized":"2024-04-01T01:32:46.662981+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:19.029896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.594630+0000","last_change":"2024-04-01T01:31:16.386234+0000","last_active":"2024-04-01T01:32:48.594630+0000","last_peered":"2024-04-01T01:32:48.594630+0000","last_clean":"2024-04-01T01:32:48.594630+0000","last_became_active":"2024-04-01T01:31:16.385876+0000","last_became_peered":"2024-04-01T01:31:16.385876+0000","last_unstale":"2024-04-01T01:32:48.594630+0000","last_undegraded":"2024-04-01T01:32:48.594630+0000","last_fullsized":"2024-04-01T01:32:48.594630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:34.187210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.805473+0000","last_change":"2024-04-01T01:32:19.805473+0000","last_active":"2024-04-01T01:32:19.805473+0000","last_peered":"2024-04-01T01:32:19.805473+0000","last_clean":"2024-04-01T01:32:19.805473+0000","last_became_active":"2024-04-01T01:32:19.805151+0000","last_became_peered":"2024-04-01T01:32:19.805151+0000","last_unstale":"2024-04-01T01:32:19.805473+0000","last_undegraded":"2024-04-01T01:32:19.805473+0000","last_fullsized":"2024-04-01T01:32:19.805473+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:47:54.539407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685654+0000","last_change":"2024-04-01T01:31:15.378503+0000","last_active":"2024-04-01T01:32:19.685654+0000","last_peered":"2024-04-01T01:32:19.685654+0000","last_clean":"2024-04-01T01:32:19.685654+0000","last_became_active":"2024-04-01T01:31:15.378152+0000","last_became_peered":"2024-04-01T01:31:15.378152+0000","last_unstale":"2024-04-01T01:32:19.685654+0000","last_undegraded":"2024-04-01T01:32:19.685654+0000","last_fullsized":"2024-04-01T01:32:19.685654+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:15:42.118691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.601369+0000","last_change":"2024-04-01T01:31:16.389676+0000","last_active":"2024-04-01T01:32:48.601369+0000","last_peered":"2024-04-01T01:32:48.601369+0000","last_clean":"2024-04-01T01:32:48.601369+0000","last_became_active":"2024-04-01T01:31:16.389304+0000","last_became_peered":"2024-04-01T01:31:16.389304+0000","last_unstale":"2024-04-01T01:32:48.601369+0000","last_undegraded":"2024-04-01T01:32:48.601369+0000","last_fullsized":"2024-04-01T01:32:48.601369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:20:24.021273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685303+0000","last_change":"2024-04-01T01:31:20.569461+0000","last_active":"2024-04-01T01:32:19.685303+0000","last_peered":"2024-04-01T01:32:19.685303+0000","last_clean":"2024-04-01T01:32:19.685303+0000","last_became_active":"2024-04-01T01:31:20.569011+0000","last_became_peered":"2024-04-01T01:31:20.569011+0000","last_unstale":"2024-04-01T01:32:19.685303+0000","last_undegraded":"2024-04-01T01:32:19.685303+0000","last_fullsized":"2024-04-01T01:32:19.685303+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:43:43.657065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.688923+0000","last_change":"2024-04-01T01:31:15.384181+0000","last_active":"2024-04-01T01:32:19.688923+0000","last_peered":"2024-04-01T01:32:19.688923+0000","last_clean":"2024-04-01T01:32:19.688923+0000","last_became_active":"2024-04-01T01:31:15.383886+0000","last_became_peered":"2024-04-01T01:31:15.383886+0000","last_unstale":"2024-04-01T01:32:19.688923+0000","last_undegraded":"2024-04-01T01:32:19.688923+0000","last_fullsized":"2024-04-01T01:32:19.688923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:43:28.414423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.596612+0000","last_change":"2024-04-01T01:31:16.381877+0000","last_active":"2024-04-01T01:32:48.596612+0000","last_peered":"2024-04-01T01:32:48.596612+0000","last_clean":"2024-04-01T01:32:48.596612+0000","last_became_active":"2024-04-01T01:31:16.381558+0000","last_became_peered":"2024-04-01T01:31:16.381558+0000","last_unstale":"2024-04-01T01:32:48.596612+0000","last_undegraded":"2024-04-01T01:32:48.596612+0000","last_fullsized":"2024-04-01T01:32:48.596612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:24.120910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.594729+0000","last_change":"2024-04-01T01:31:16.380116+0000","last_active":"2024-04-01T01:32:48.594729+0000","last_peered":"2024-04-01T01:32:48.594729+0000","last_clean":"2024-04-01T01:32:48.594729+0000","last_became_active":"2024-04-01T01:31:16.378936+0000","last_became_peered":"2024-04-01T01:31:16.378936+0000","last_unstale":"2024-04-01T01:32:48.594729+0000","last_undegraded":"2024-04-01T01:32:48.594729+0000","last_fullsized":"2024-04-01T01:32:48.594729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:18:16.211113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683707+0000","last_change":"2024-04-01T01:31:16.384872+0000","last_active":"2024-04-01T01:32:19.683707+0000","last_peered":"2024-04-01T01:32:19.683707+0000","last_clean":"2024-04-01T01:32:19.683707+0000","last_became_active":"2024-04-01T01:31:16.384406+0000","last_became_peered":"2024-04-01T01:31:16.384406+0000","last_unstale":"2024-04-01T01:32:19.683707+0000","last_undegraded":"2024-04-01T01:32:19.683707+0000","last_fullsized":"2024-04-01T01:32:19.683707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:13:36.567357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683464+0000","last_change":"2024-04-01T01:31:16.384561+0000","last_active":"2024-04-01T01:32:19.683464+0000","last_peered":"2024-04-01T01:32:19.683464+0000","last_clean":"2024-04-01T01:32:19.683464+0000","last_became_active":"2024-04-01T01:31:16.384221+0000","last_became_peered":"2024-04-01T01:31:16.384221+0000","last_unstale":"2024-04-01T01:32:19.683464+0000","last_undegraded":"2024-04-01T01:32:19.683464+0000","last_fullsized":"2024-04-01T01:32:19.683464+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:17:56.490000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.603580+0000","last_change":"2024-04-01T01:31:16.385464+0000","last_active":"2024-04-01T01:32:48.603580+0000","last_peered":"2024-04-01T01:32:48.603580+0000","last_clean":"2024-04-01T01:32:48.603580+0000","last_became_active":"2024-04-01T01:31:16.385159+0000","last_became_peered":"2024-04-01T01:31:16.385159+0000","last_unstale":"2024-04-01T01:32:48.603580+0000","last_undegraded":"2024-04-01T01:32:48.603580+0000","last_fullsized":"2024-04-01T01:32:48.603580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:33:53.852957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.604151+0000","last_change":"2024-04-01T01:31:16.385688+0000","last_active":"2024-04-01T01:32:48.604151+0000","last_peered":"2024-04-01T01:32:48.604151+0000","last_clean":"2024-04-01T01:32:48.604151+0000","last_became_active":"2024-04-01T01:31:16.385381+0000","last_became_peered":"2024-04-01T01:31:16.385381+0000","last_unstale":"2024-04-01T01:32:48.604151+0000","last_undegraded":"2024-04-01T01:32:48.604151+0000","last_fullsized":"2024-04-01T01:32:48.604151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:39:36.810766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.593165+0000","last_change":"2024-04-01T01:31:16.379976+0000","last_active":"2024-04-01T01:32:48.593165+0000","last_peered":"2024-04-01T01:32:48.593165+0000","last_clean":"2024-04-01T01:32:48.593165+0000","last_became_active":"2024-04-01T01:31:16.379553+0000","last_became_peered":"2024-04-01T01:31:16.379553+0000","last_unstale":"2024-04-01T01:32:48.593165+0000","last_undegraded":"2024-04-01T01:32:48.593165+0000","last_fullsized":"2024-04-01T01:32:48.593165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:48:23.555723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.684459+0000","last_change":"2024-04-01T01:31:16.383995+0000","last_active":"2024-04-01T01:32:19.684459+0000","last_peered":"2024-04-01T01:32:19.684459+0000","last_clean":"2024-04-01T01:32:19.684459+0000","last_became_active":"2024-04-01T01:31:16.383673+0000","last_became_peered":"2024-04-01T01:31:16.383673+0000","last_unstale":"2024-04-01T01:32:19.684459+0000","last_undegraded":"2024-04-01T01:32:19.684459+0000","last_fullsized":"2024-04-01T01:32:19.684459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:03:16.192438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.597623+0000","last_change":"2024-04-01T01:31:16.389969+0000","last_active":"2024-04-01T01:32:48.597623+0000","last_peered":"2024-04-01T01:32:48.597623+0000","last_clean":"2024-04-01T01:32:48.597623+0000","last_became_active":"2024-04-01T01:31:16.389640+0000","last_became_peered":"2024-04-01T01:31:16.389640+0000","last_unstale":"2024-04-01T01:32:48.597623+0000","last_undegraded":"2024-04-01T01:32:48.597623+0000","last_fullsized":"2024-04-01T01:32:48.597623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:35:15.226960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.588985+0000","last_change":"2024-04-01T01:31:16.382962+0000","last_active":"2024-04-01T01:32:48.588985+0000","last_peered":"2024-04-01T01:32:48.588985+0000","last_clean":"2024-04-01T01:32:48.588985+0000","last_became_active":"2024-04-01T01:31:16.382653+0000","last_became_peered":"2024-04-01T01:31:16.382653+0000","last_unstale":"2024-04-01T01:32:48.588985+0000","last_undegraded":"2024-04-01T01:32:48.588985+0000","last_fullsized":"2024-04-01T01:32:48.588985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:53.054884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.602978+0000","last_change":"2024-04-01T01:31:16.382997+0000","last_active":"2024-04-01T01:32:48.602978+0000","last_peered":"2024-04-01T01:32:48.602978+0000","last_clean":"2024-04-01T01:32:48.602978+0000","last_became_active":"2024-04-01T01:31:16.380180+0000","last_became_peered":"2024-04-01T01:31:16.380180+0000","last_unstale":"2024-04-01T01:32:48.602978+0000","last_undegraded":"2024-04-01T01:32:48.602978+0000","last_fullsized":"2024-04-01T01:32:48.602978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:29:51.990395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.595157+0000","last_change":"2024-04-01T01:31:16.382948+0000","last_active":"2024-04-01T01:32:48.595157+0000","last_peered":"2024-04-01T01:32:48.595157+0000","last_clean":"2024-04-01T01:32:48.595157+0000","last_became_active":"2024-04-01T01:31:16.382134+0000","last_became_peered":"2024-04-01T01:31:16.382134+0000","last_unstale":"2024-04-01T01:32:48.595157+0000","last_undegraded":"2024-04-01T01:32:48.595157+0000","last_fullsized":"2024-04-01T01:32:48.595157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:24:08.515063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.599945+0000","last_change":"2024-04-01T01:31:16.378841+0000","last_active":"2024-04-01T01:32:48.599945+0000","last_peered":"2024-04-01T01:32:48.599945+0000","last_clean":"2024-04-01T01:32:48.599945+0000","last_became_active":"2024-04-01T01:31:16.378501+0000","last_became_peered":"2024-04-01T01:31:16.378501+0000","last_unstale":"2024-04-01T01:32:48.599945+0000","last_undegraded":"2024-04-01T01:32:48.599945+0000","last_fullsized":"2024-04-01T01:32:48.599945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:50:20.881852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.580110+0000","last_change":"2024-04-01T01:31:16.380979+0000","last_active":"2024-04-01T01:32:48.580110+0000","last_peered":"2024-04-01T01:32:48.580110+0000","last_clean":"2024-04-01T01:32:48.580110+0000","last_became_active":"2024-04-01T01:31:16.380625+0000","last_became_peered":"2024-04-01T01:31:16.380625+0000","last_unstale":"2024-04-01T01:32:48.580110+0000","last_undegraded":"2024-04-01T01:32:48.580110+0000","last_fullsized":"2024-04-01T01:32:48.580110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:09:03.246239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":439,"ondisk_log_size":439,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71091,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":771,"num_read_kb":469,"num_write":250,"num_write_kb":147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1125,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":213024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":218,"ondisk_log_size":218,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383713,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":501545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37024,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708128,"statfs":{"total":95995035648,"available":95957123072,"internally_reserved":0,"allocated":1728512,"data_stored":1152386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,2,21,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808875,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":505459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070511,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1048576,"data_stored":505725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332149,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36948,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708204,"statfs":{"total":95995035648,"available":95957200896,"internally_reserved":0,"allocated":1650688,"data_stored":1098911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":496867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":561426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":498323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1114112,"data_stored":560851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":496575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920835,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":502627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757311,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28632,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716520,"statfs":{"total":95995035648,"available":95965716480,"internally_reserved":0,"allocated":1654784,"data_stored":1089865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":65357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":65873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-01T01:32:53.480 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-01T01:32:53.724 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:32:53.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:53 smithi070 ceph-mon[32179]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-01T01:32:53.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:53 smithi070 ceph-mon[32179]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:32:53.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:53 smithi189 ceph-mon[35753]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-01T01:32:53.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:53 smithi189 ceph-mon[35753]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:32:54.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:53 smithi081 ceph-mon[35142]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-01T01:32:54.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:53 smithi081 ceph-mon[35142]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:32:55.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.600+0000 7f0a4b769700 1 -- 172.21.15.70:0/2185056639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a4406a9e0 msgr2=0x7f0a4406adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:55.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.600+0000 7f0a4b769700 1 --2- 172.21.15.70:0/2185056639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a4406a9e0 0x7f0a4406adc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0a400099c0 tx=0x7f0a40009cd0 comp rx=0 tx=0).stop 2024-04-01T01:32:55.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.600+0000 7f0a4b769700 1 -- 172.21.15.70:0/2185056639 shutdown_connections 2024-04-01T01:32:55.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.600+0000 7f0a4b769700 1 --2- 172.21.15.70:0/2185056639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a44107520 0x7f0a4410f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:55.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.600+0000 7f0a4b769700 1 --2- 172.21.15.70:0/2185056639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a4406a9e0 0x7f0a4406adc0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:55.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.600+0000 7f0a4b769700 1 --2- 172.21.15.70:0/2185056639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4406b390 0x7f0a44106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:55.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.600+0000 7f0a4b769700 1 -- 172.21.15.70:0/2185056639 >> 172.21.15.70:0/2185056639 conn(0x7f0a44077190 msgr2=0x7f0a440775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:55.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.600+0000 7f0a4b769700 1 -- 172.21.15.70:0/2185056639 shutdown_connections 2024-04-01T01:32:55.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.600+0000 7f0a4b769700 1 -- 172.21.15.70:0/2185056639 wait complete. 2024-04-01T01:32:55.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.601+0000 7f0a4b769700 1 Processor -- start 2024-04-01T01:32:55.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.601+0000 7f0a4b769700 1 -- start start 2024-04-01T01:32:55.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.601+0000 7f0a4b769700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4406a9e0 0x7f0a4419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:55.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.601+0000 7f0a4b769700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a4406b390 0x7f0a4419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:55.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.601+0000 7f0a4b769700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a44107520 0x7f0a44198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:55.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.601+0000 7f0a4b769700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0a441123d0 con 0x7f0a44107520 2024-04-01T01:32:55.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.601+0000 7f0a4b769700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0a44112250 con 0x7f0a4406b390 2024-04-01T01:32:55.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.601+0000 7f0a4b769700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a44112550 con 0x7f0a4406a9e0 2024-04-01T01:32:55.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.601+0000 7f0a49d06700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a44107520 0x7f0a44198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:55.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.601+0000 7f0a49505700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4406a9e0 0x7f0a4419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:55.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.601+0000 7f0a49d06700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a44107520 0x7f0a44198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58588/0 (socket says 172.21.15.70:58588) 2024-04-01T01:32:55.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.601+0000 7f0a49d06700 1 -- 172.21.15.70:0/3037256707 learned_addr learned my addr 172.21.15.70:0/3037256707 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:55.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.601+0000 7f0a48d04700 1 --2- 172.21.15.70:0/3037256707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a4406b390 0x7f0a4419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:55.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.601+0000 7f0a49d06700 1 -- 172.21.15.70:0/3037256707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4406a9e0 msgr2=0x7f0a4419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:55.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.601+0000 7f0a49d06700 1 --2- 172.21.15.70:0/3037256707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4406a9e0 0x7f0a4419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:55.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.602+0000 7f0a49d06700 1 -- 172.21.15.70:0/3037256707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a4406b390 msgr2=0x7f0a4419eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:55.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.602+0000 7f0a49d06700 1 --2- 172.21.15.70:0/3037256707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a4406b390 0x7f0a4419eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:55.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.602+0000 7f0a49d06700 1 -- 172.21.15.70:0/3037256707 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a44198ff0 con 0x7f0a44107520 2024-04-01T01:32:55.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.602+0000 7f0a49505700 1 --2- 172.21.15.70:0/3037256707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4406a9e0 0x7f0a4419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:55.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.602+0000 7f0a48d04700 1 --2- 172.21.15.70:0/3037256707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a4406b390 0x7f0a4419eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:32:55.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.602+0000 7f0a49d06700 1 --2- 172.21.15.70:0/3037256707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a44107520 0x7f0a44198710 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00ef30 tx=0x7f0a3c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:55.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.602+0000 7f0a3a7fc700 1 -- 172.21.15.70:0/3037256707 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c0092e0 con 0x7f0a44107520 2024-04-01T01:32:55.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.602+0000 7f0a3a7fc700 1 -- 172.21.15.70:0/3037256707 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a3c004bb0 con 0x7f0a44107520 2024-04-01T01:32:55.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.602+0000 7f0a4b769700 1 -- 172.21.15.70:0/3037256707 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a44199280 con 0x7f0a44107520 2024-04-01T01:32:55.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.602+0000 7f0a3a7fc700 1 -- 172.21.15.70:0/3037256707 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c004020 con 0x7f0a44107520 2024-04-01T01:32:55.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.602+0000 7f0a4b769700 1 -- 172.21.15.70:0/3037256707 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a441a53f0 con 0x7f0a44107520 2024-04-01T01:32:55.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.603+0000 7f0a4b769700 1 -- 172.21.15.70:0/3037256707 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a44103920 con 0x7f0a44107520 2024-04-01T01:32:55.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.604+0000 7f0a3a7fc700 1 -- 172.21.15.70:0/3037256707 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f0a3c01b030 con 0x7f0a44107520 2024-04-01T01:32:55.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.604+0000 7f0a3a7fc700 1 --2- 172.21.15.70:0/3037256707 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a300a2530 0x7f0a300a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:55.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.605+0000 7f0a3a7fc700 1 -- 172.21.15.70:0/3037256707 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0a3c0ce4d0 con 0x7f0a44107520 2024-04-01T01:32:55.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.605+0000 7f0a49505700 1 --2- 172.21.15.70:0/3037256707 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a300a2530 0x7f0a300a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:55.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.605+0000 7f0a49505700 1 --2- 172.21.15.70:0/3037256707 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a300a2530 0x7f0a300a49f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0a40003760 tx=0x7f0a40005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:55.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.609+0000 7f0a3a7fc700 1 -- 172.21.15.70:0/3037256707 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0a3c022020 con 0x7f0a44107520 2024-04-01T01:32:55.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.800+0000 7f0a4b769700 1 -- 172.21.15.70:0/3037256707 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0a44062680 con 0x7f0a300a2530 2024-04-01T01:32:55.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.809+0000 7f0a3a7fc700 1 -- 172.21.15.70:0/3037256707 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340668 (secure 0 0 0) 0x7f0a4007b2d0 con 0x7f0a300a2530 2024-04-01T01:32:55.814 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:32:55.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.813+0000 7f0a4b769700 1 -- 172.21.15.70:0/3037256707 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a300a2530 msgr2=0x7f0a300a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:55.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.813+0000 7f0a4b769700 1 --2- 172.21.15.70:0/3037256707 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a300a2530 0x7f0a300a49f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0a40003760 tx=0x7f0a40005fb0 comp rx=0 tx=0).stop 2024-04-01T01:32:55.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.813+0000 7f0a4b769700 1 -- 172.21.15.70:0/3037256707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a44107520 msgr2=0x7f0a44198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:55.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.813+0000 7f0a4b769700 1 --2- 172.21.15.70:0/3037256707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a44107520 0x7f0a44198710 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00ef30 tx=0x7f0a3c00c3a0 comp rx=0 tx=0).stop 2024-04-01T01:32:55.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.813+0000 7f0a4b769700 1 -- 172.21.15.70:0/3037256707 shutdown_connections 2024-04-01T01:32:55.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.813+0000 7f0a4b769700 1 --2- 172.21.15.70:0/3037256707 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a300a2530 0x7f0a300a49f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:55.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.813+0000 7f0a4b769700 1 --2- 172.21.15.70:0/3037256707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a44107520 0x7f0a44198710 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:55.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.813+0000 7f0a4b769700 1 --2- 172.21.15.70:0/3037256707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a4406a9e0 0x7f0a4419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:55.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.813+0000 7f0a4b769700 1 --2- 172.21.15.70:0/3037256707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a4406b390 0x7f0a4419eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:55.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.813+0000 7f0a4b769700 1 -- 172.21.15.70:0/3037256707 >> 172.21.15.70:0/3037256707 conn(0x7f0a44077190 msgr2=0x7f0a441061c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:55.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.814+0000 7f0a4b769700 1 -- 172.21.15.70:0/3037256707 shutdown_connections 2024-04-01T01:32:55.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:55.814+0000 7f0a4b769700 1 -- 172.21.15.70:0/3037256707 wait complete. 2024-04-01T01:32:55.819 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-01T01:32:55.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:55 smithi189 ceph-mon[35753]: pgmap v339: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-01T01:32:56.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:55 smithi070 ceph-mon[32179]: pgmap v339: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-01T01:32:56.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:55 smithi081 ceph-mon[35142]: pgmap v339: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-01T01:32:56.585 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-04-01T01:32:54.591819+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661693,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":902,"num_read_kb":578,"num_write":433,"num_write_kb":3087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181978,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":857,"ondisk_log_size":857,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":359232,"kb_used_data":14240,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124582592,"statfs":{"total":1151940427776,"available":1151572574208,"internally_reserved":0,"allocated":14581760,"data_stored":7970885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,2,21,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":108,"num_read_kb":53,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003423"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.608412+0000","last_change":"2024-04-01T01:31:16.385517+0000","last_active":"2024-04-01T01:32:48.608412+0000","last_peered":"2024-04-01T01:32:48.608412+0000","last_clean":"2024-04-01T01:32:48.608412+0000","last_became_active":"2024-04-01T01:31:16.384682+0000","last_became_peered":"2024-04-01T01:31:16.384682+0000","last_unstale":"2024-04-01T01:32:48.608412+0000","last_undegraded":"2024-04-01T01:32:48.608412+0000","last_fullsized":"2024-04-01T01:32:48.608412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:35:47.614041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.604933+0000","last_change":"2024-04-01T01:31:16.376119+0000","last_active":"2024-04-01T01:32:48.604933+0000","last_peered":"2024-04-01T01:32:48.604933+0000","last_clean":"2024-04-01T01:32:48.604933+0000","last_became_active":"2024-04-01T01:31:16.375801+0000","last_became_peered":"2024-04-01T01:31:16.375801+0000","last_unstale":"2024-04-01T01:32:48.604933+0000","last_undegraded":"2024-04-01T01:32:48.604933+0000","last_fullsized":"2024-04-01T01:32:48.604933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:34.120487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.606902+0000","last_change":"2024-04-01T01:31:16.378158+0000","last_active":"2024-04-01T01:32:48.606902+0000","last_peered":"2024-04-01T01:32:48.606902+0000","last_clean":"2024-04-01T01:32:48.606902+0000","last_became_active":"2024-04-01T01:31:16.377831+0000","last_became_peered":"2024-04-01T01:31:16.377831+0000","last_unstale":"2024-04-01T01:32:48.606902+0000","last_undegraded":"2024-04-01T01:32:48.606902+0000","last_fullsized":"2024-04-01T01:32:48.606902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:30:06.487545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.592067+0000","last_change":"2024-04-01T01:31:16.389708+0000","last_active":"2024-04-01T01:32:48.592067+0000","last_peered":"2024-04-01T01:32:48.592067+0000","last_clean":"2024-04-01T01:32:48.592067+0000","last_became_active":"2024-04-01T01:31:16.389409+0000","last_became_peered":"2024-04-01T01:31:16.389409+0000","last_unstale":"2024-04-01T01:32:48.592067+0000","last_undegraded":"2024-04-01T01:32:48.592067+0000","last_fullsized":"2024-04-01T01:32:48.592067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:57:49.215988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.567447+0000","last_change":"2024-04-01T01:31:16.385996+0000","last_active":"2024-04-01T01:32:48.567447+0000","last_peered":"2024-04-01T01:32:48.567447+0000","last_clean":"2024-04-01T01:32:48.567447+0000","last_became_active":"2024-04-01T01:31:16.385648+0000","last_became_peered":"2024-04-01T01:31:16.385648+0000","last_unstale":"2024-04-01T01:32:48.567447+0000","last_undegraded":"2024-04-01T01:32:48.567447+0000","last_fullsized":"2024-04-01T01:32:48.567447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:14:56.595595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.576793+0000","last_change":"2024-04-01T01:32:19.693185+0000","last_active":"2024-04-01T01:32:48.576793+0000","last_peered":"2024-04-01T01:32:48.576793+0000","last_clean":"2024-04-01T01:32:48.576793+0000","last_became_active":"2024-04-01T01:32:19.692763+0000","last_became_peered":"2024-04-01T01:32:19.692763+0000","last_unstale":"2024-04-01T01:32:48.576793+0000","last_undegraded":"2024-04-01T01:32:48.576793+0000","last_fullsized":"2024-04-01T01:32:48.576793+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:37:54.712852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.600332+0000","last_change":"2024-04-01T01:31:16.383939+0000","last_active":"2024-04-01T01:32:48.600332+0000","last_peered":"2024-04-01T01:32:48.600332+0000","last_clean":"2024-04-01T01:32:48.600332+0000","last_became_active":"2024-04-01T01:31:16.383626+0000","last_became_peered":"2024-04-01T01:31:16.383626+0000","last_unstale":"2024-04-01T01:32:48.600332+0000","last_undegraded":"2024-04-01T01:32:48.600332+0000","last_fullsized":"2024-04-01T01:32:48.600332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:36:05.764031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.586784+0000","last_change":"2024-04-01T01:31:16.384007+0000","last_active":"2024-04-01T01:32:48.586784+0000","last_peered":"2024-04-01T01:32:48.586784+0000","last_clean":"2024-04-01T01:32:48.586784+0000","last_became_active":"2024-04-01T01:31:16.383687+0000","last_became_peered":"2024-04-01T01:31:16.383687+0000","last_unstale":"2024-04-01T01:32:48.586784+0000","last_undegraded":"2024-04-01T01:32:48.586784+0000","last_fullsized":"2024-04-01T01:32:48.586784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:14:17.593402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.598231+0000","last_change":"2024-04-01T01:31:16.380357+0000","last_active":"2024-04-01T01:32:48.598231+0000","last_peered":"2024-04-01T01:32:48.598231+0000","last_clean":"2024-04-01T01:32:48.598231+0000","last_became_active":"2024-04-01T01:31:16.380046+0000","last_became_peered":"2024-04-01T01:31:16.380046+0000","last_unstale":"2024-04-01T01:32:48.598231+0000","last_undegraded":"2024-04-01T01:32:48.598231+0000","last_fullsized":"2024-04-01T01:32:48.598231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:51:53.887768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.591699+0000","last_change":"2024-04-01T01:31:16.378551+0000","last_active":"2024-04-01T01:32:48.591699+0000","last_peered":"2024-04-01T01:32:48.591699+0000","last_clean":"2024-04-01T01:32:48.591699+0000","last_became_active":"2024-04-01T01:31:16.378234+0000","last_became_peered":"2024-04-01T01:31:16.378234+0000","last_unstale":"2024-04-01T01:32:48.591699+0000","last_undegraded":"2024-04-01T01:32:48.591699+0000","last_fullsized":"2024-04-01T01:32:48.591699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:37:45.998115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.594272+0000","last_change":"2024-04-01T01:31:16.388755+0000","last_active":"2024-04-01T01:32:48.594272+0000","last_peered":"2024-04-01T01:32:48.594272+0000","last_clean":"2024-04-01T01:32:48.594272+0000","last_became_active":"2024-04-01T01:31:16.388434+0000","last_became_peered":"2024-04-01T01:31:16.388434+0000","last_unstale":"2024-04-01T01:32:48.594272+0000","last_undegraded":"2024-04-01T01:32:48.594272+0000","last_fullsized":"2024-04-01T01:32:48.594272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:03:07.168110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.568940+0000","last_change":"2024-04-01T01:31:16.390478+0000","last_active":"2024-04-01T01:32:48.568940+0000","last_peered":"2024-04-01T01:32:48.568940+0000","last_clean":"2024-04-01T01:32:48.568940+0000","last_became_active":"2024-04-01T01:31:16.390013+0000","last_became_peered":"2024-04-01T01:31:16.390013+0000","last_unstale":"2024-04-01T01:32:48.568940+0000","last_undegraded":"2024-04-01T01:32:48.568940+0000","last_fullsized":"2024-04-01T01:32:48.568940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:28:21.677527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.595795+0000","last_change":"2024-04-01T01:31:16.366921+0000","last_active":"2024-04-01T01:32:48.595795+0000","last_peered":"2024-04-01T01:32:48.595795+0000","last_clean":"2024-04-01T01:32:48.595795+0000","last_became_active":"2024-04-01T01:31:16.366098+0000","last_became_peered":"2024-04-01T01:31:16.366098+0000","last_unstale":"2024-04-01T01:32:48.595795+0000","last_undegraded":"2024-04-01T01:32:48.595795+0000","last_fullsized":"2024-04-01T01:32:48.595795+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:29.456263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.605838+0000","last_change":"2024-04-01T01:31:16.379178+0000","last_active":"2024-04-01T01:32:48.605838+0000","last_peered":"2024-04-01T01:32:48.605838+0000","last_clean":"2024-04-01T01:32:48.605838+0000","last_became_active":"2024-04-01T01:31:16.378877+0000","last_became_peered":"2024-04-01T01:31:16.378877+0000","last_unstale":"2024-04-01T01:32:48.605838+0000","last_undegraded":"2024-04-01T01:32:48.605838+0000","last_fullsized":"2024-04-01T01:32:48.605838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:19:16.941504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.606872+0000","last_change":"2024-04-01T01:31:16.385373+0000","last_active":"2024-04-01T01:32:48.606872+0000","last_peered":"2024-04-01T01:32:48.606872+0000","last_clean":"2024-04-01T01:32:48.606872+0000","last_became_active":"2024-04-01T01:31:16.385046+0000","last_became_peered":"2024-04-01T01:31:16.385046+0000","last_unstale":"2024-04-01T01:32:48.606872+0000","last_undegraded":"2024-04-01T01:32:48.606872+0000","last_fullsized":"2024-04-01T01:32:48.606872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:40:32.521220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.599246+0000","last_change":"2024-04-01T01:31:16.386267+0000","last_active":"2024-04-01T01:32:48.599246+0000","last_peered":"2024-04-01T01:32:48.599246+0000","last_clean":"2024-04-01T01:32:48.599246+0000","last_became_active":"2024-04-01T01:31:16.385917+0000","last_became_peered":"2024-04-01T01:31:16.385917+0000","last_unstale":"2024-04-01T01:32:48.599246+0000","last_undegraded":"2024-04-01T01:32:48.599246+0000","last_fullsized":"2024-04-01T01:32:48.599246+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:58.249253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.608460+0000","last_change":"2024-04-01T01:31:16.385609+0000","last_active":"2024-04-01T01:32:48.608460+0000","last_peered":"2024-04-01T01:32:48.608460+0000","last_clean":"2024-04-01T01:32:48.608460+0000","last_became_active":"2024-04-01T01:31:16.385128+0000","last_became_peered":"2024-04-01T01:31:16.385128+0000","last_unstale":"2024-04-01T01:32:48.608460+0000","last_undegraded":"2024-04-01T01:32:48.608460+0000","last_fullsized":"2024-04-01T01:32:48.608460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:30:45.915587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.595736+0000","last_change":"2024-04-01T01:31:16.381209+0000","last_active":"2024-04-01T01:32:48.595736+0000","last_peered":"2024-04-01T01:32:48.595736+0000","last_clean":"2024-04-01T01:32:48.595736+0000","last_became_active":"2024-04-01T01:31:16.380721+0000","last_became_peered":"2024-04-01T01:31:16.380721+0000","last_unstale":"2024-04-01T01:32:48.595736+0000","last_undegraded":"2024-04-01T01:32:48.595736+0000","last_fullsized":"2024-04-01T01:32:48.595736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:22.438224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.595863+0000","last_change":"2024-04-01T01:31:16.385235+0000","last_active":"2024-04-01T01:32:48.595863+0000","last_peered":"2024-04-01T01:32:48.595863+0000","last_clean":"2024-04-01T01:32:48.595863+0000","last_became_active":"2024-04-01T01:31:16.384856+0000","last_became_peered":"2024-04-01T01:31:16.384856+0000","last_unstale":"2024-04-01T01:32:48.595863+0000","last_undegraded":"2024-04-01T01:32:48.595863+0000","last_fullsized":"2024-04-01T01:32:48.595863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:35:30.214977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683439+0000","last_change":"2024-04-01T01:31:20.573626+0000","last_active":"2024-04-01T01:32:19.683439+0000","last_peered":"2024-04-01T01:32:19.683439+0000","last_clean":"2024-04-01T01:32:19.683439+0000","last_became_active":"2024-04-01T01:31:20.573182+0000","last_became_peered":"2024-04-01T01:31:20.573182+0000","last_unstale":"2024-04-01T01:32:19.683439+0000","last_undegraded":"2024-04-01T01:32:19.683439+0000","last_fullsized":"2024-04-01T01:32:19.683439+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:33:19.162613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.654805+0000","last_change":"2024-04-01T01:31:15.373100+0000","last_active":"2024-04-01T01:32:48.654805+0000","last_peered":"2024-04-01T01:32:48.654805+0000","last_clean":"2024-04-01T01:32:48.654805+0000","last_became_active":"2024-04-01T01:31:15.372710+0000","last_became_peered":"2024-04-01T01:31:15.372710+0000","last_unstale":"2024-04-01T01:32:48.654805+0000","last_undegraded":"2024-04-01T01:32:48.654805+0000","last_fullsized":"2024-04-01T01:32:48.654805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:29:42.303549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.591687+0000","last_change":"2024-04-01T01:31:16.383690+0000","last_active":"2024-04-01T01:32:48.591687+0000","last_peered":"2024-04-01T01:32:48.591687+0000","last_clean":"2024-04-01T01:32:48.591687+0000","last_became_active":"2024-04-01T01:31:16.383248+0000","last_became_peered":"2024-04-01T01:31:16.383248+0000","last_unstale":"2024-04-01T01:32:48.591687+0000","last_undegraded":"2024-04-01T01:32:48.591687+0000","last_fullsized":"2024-04-01T01:32:48.591687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:34:49.112880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.691398+0000","last_change":"2024-04-01T01:32:19.691398+0000","last_active":"2024-04-01T01:32:19.691398+0000","last_peered":"2024-04-01T01:32:19.691398+0000","last_clean":"2024-04-01T01:32:19.691398+0000","last_became_active":"2024-04-01T01:32:19.690908+0000","last_became_peered":"2024-04-01T01:32:19.690908+0000","last_unstale":"2024-04-01T01:32:19.691398+0000","last_undegraded":"2024-04-01T01:32:19.691398+0000","last_fullsized":"2024-04-01T01:32:19.691398+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:58:54.712363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.687158+0000","last_change":"2024-04-01T01:31:15.381742+0000","last_active":"2024-04-01T01:32:19.687158+0000","last_peered":"2024-04-01T01:32:19.687158+0000","last_clean":"2024-04-01T01:32:19.687158+0000","last_became_active":"2024-04-01T01:31:15.377543+0000","last_became_peered":"2024-04-01T01:31:15.377543+0000","last_unstale":"2024-04-01T01:32:19.687158+0000","last_undegraded":"2024-04-01T01:32:19.687158+0000","last_fullsized":"2024-04-01T01:32:19.687158+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:51:42.911114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.605862+0000","last_change":"2024-04-01T01:31:16.382001+0000","last_active":"2024-04-01T01:32:48.605862+0000","last_peered":"2024-04-01T01:32:48.605862+0000","last_clean":"2024-04-01T01:32:48.605862+0000","last_became_active":"2024-04-01T01:31:16.381694+0000","last_became_peered":"2024-04-01T01:31:16.381694+0000","last_unstale":"2024-04-01T01:32:48.605862+0000","last_undegraded":"2024-04-01T01:32:48.605862+0000","last_fullsized":"2024-04-01T01:32:48.605862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:55:07.366755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683197+0000","last_change":"2024-04-01T01:31:20.570080+0000","last_active":"2024-04-01T01:32:19.683197+0000","last_peered":"2024-04-01T01:32:19.683197+0000","last_clean":"2024-04-01T01:32:19.683197+0000","last_became_active":"2024-04-01T01:31:20.569712+0000","last_became_peered":"2024-04-01T01:31:20.569712+0000","last_unstale":"2024-04-01T01:32:19.683197+0000","last_undegraded":"2024-04-01T01:32:19.683197+0000","last_fullsized":"2024-04-01T01:32:19.683197+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:48:27.732831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":62,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.703903+0000","last_change":"2024-04-01T01:32:19.703903+0000","last_active":"2024-04-01T01:32:19.703903+0000","last_peered":"2024-04-01T01:32:19.703903+0000","last_clean":"2024-04-01T01:32:19.703903+0000","last_became_active":"2024-04-01T01:32:19.691856+0000","last_became_peered":"2024-04-01T01:32:19.691856+0000","last_unstale":"2024-04-01T01:32:19.703903+0000","last_undegraded":"2024-04-01T01:32:19.703903+0000","last_fullsized":"2024-04-01T01:32:19.703903+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:38:41.284469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.596295+0000","last_change":"2024-04-01T01:31:16.383020+0000","last_active":"2024-04-01T01:32:48.596295+0000","last_peered":"2024-04-01T01:32:48.596295+0000","last_clean":"2024-04-01T01:32:48.596295+0000","last_became_active":"2024-04-01T01:31:16.382640+0000","last_became_peered":"2024-04-01T01:31:16.382640+0000","last_unstale":"2024-04-01T01:32:48.596295+0000","last_undegraded":"2024-04-01T01:32:48.596295+0000","last_fullsized":"2024-04-01T01:32:48.596295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:11:51.211929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683088+0000","last_change":"2024-04-01T01:31:20.573030+0000","last_active":"2024-04-01T01:32:19.683088+0000","last_peered":"2024-04-01T01:32:19.683088+0000","last_clean":"2024-04-01T01:32:19.683088+0000","last_became_active":"2024-04-01T01:31:20.572613+0000","last_became_peered":"2024-04-01T01:31:20.572613+0000","last_unstale":"2024-04-01T01:32:19.683088+0000","last_undegraded":"2024-04-01T01:32:19.683088+0000","last_fullsized":"2024-04-01T01:32:19.683088+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:42:17.086942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.689171+0000","last_change":"2024-04-01T01:31:15.373054+0000","last_active":"2024-04-01T01:32:19.689171+0000","last_peered":"2024-04-01T01:32:19.689171+0000","last_clean":"2024-04-01T01:32:19.689171+0000","last_became_active":"2024-04-01T01:31:15.372575+0000","last_became_peered":"2024-04-01T01:31:15.372575+0000","last_unstale":"2024-04-01T01:32:19.689171+0000","last_undegraded":"2024-04-01T01:32:19.689171+0000","last_fullsized":"2024-04-01T01:32:19.689171+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:56:41.860423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.599437+0000","last_change":"2024-04-01T01:31:16.376037+0000","last_active":"2024-04-01T01:32:48.599437+0000","last_peered":"2024-04-01T01:32:48.599437+0000","last_clean":"2024-04-01T01:32:48.599437+0000","last_became_active":"2024-04-01T01:31:16.375463+0000","last_became_peered":"2024-04-01T01:31:16.375463+0000","last_unstale":"2024-04-01T01:32:48.599437+0000","last_undegraded":"2024-04-01T01:32:48.599437+0000","last_fullsized":"2024-04-01T01:32:48.599437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:30:40.098033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.684137+0000","last_change":"2024-04-01T01:31:20.574288+0000","last_active":"2024-04-01T01:32:19.684137+0000","last_peered":"2024-04-01T01:32:19.684137+0000","last_clean":"2024-04-01T01:32:19.684137+0000","last_became_active":"2024-04-01T01:31:20.573812+0000","last_became_peered":"2024-04-01T01:31:20.573812+0000","last_unstale":"2024-04-01T01:32:19.684137+0000","last_undegraded":"2024-04-01T01:32:19.684137+0000","last_fullsized":"2024-04-01T01:32:19.684137+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:43:46.126102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":126,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:49.721894+0000","last_change":"2024-04-01T01:32:19.706645+0000","last_active":"2024-04-01T01:32:49.721894+0000","last_peered":"2024-04-01T01:32:49.721894+0000","last_clean":"2024-04-01T01:32:49.721894+0000","last_became_active":"2024-04-01T01:32:19.690510+0000","last_became_peered":"2024-04-01T01:32:19.690510+0000","last_unstale":"2024-04-01T01:32:49.721894+0000","last_undegraded":"2024-04-01T01:32:49.721894+0000","last_fullsized":"2024-04-01T01:32:49.721894+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:20:23.103229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.576241+0000","last_change":"2024-04-01T01:31:16.377735+0000","last_active":"2024-04-01T01:32:48.576241+0000","last_peered":"2024-04-01T01:32:48.576241+0000","last_clean":"2024-04-01T01:32:48.576241+0000","last_became_active":"2024-04-01T01:31:16.377342+0000","last_became_peered":"2024-04-01T01:31:16.377342+0000","last_unstale":"2024-04-01T01:32:48.576241+0000","last_undegraded":"2024-04-01T01:32:48.576241+0000","last_fullsized":"2024-04-01T01:32:48.576241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:47.539223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683899+0000","last_change":"2024-04-01T01:31:20.573370+0000","last_active":"2024-04-01T01:32:19.683899+0000","last_peered":"2024-04-01T01:32:19.683899+0000","last_clean":"2024-04-01T01:32:19.683899+0000","last_became_active":"2024-04-01T01:31:20.572919+0000","last_became_peered":"2024-04-01T01:31:20.572919+0000","last_unstale":"2024-04-01T01:32:19.683899+0000","last_undegraded":"2024-04-01T01:32:19.683899+0000","last_fullsized":"2024-04-01T01:32:19.683899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:42.190918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.810468+0000","last_change":"2024-04-01T01:32:19.810468+0000","last_active":"2024-04-01T01:32:19.810468+0000","last_peered":"2024-04-01T01:32:19.810468+0000","last_clean":"2024-04-01T01:32:19.810468+0000","last_became_active":"2024-04-01T01:32:19.805517+0000","last_became_peered":"2024-04-01T01:32:19.805517+0000","last_unstale":"2024-04-01T01:32:19.810468+0000","last_undegraded":"2024-04-01T01:32:19.810468+0000","last_fullsized":"2024-04-01T01:32:19.810468+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:54:35.355761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.588732+0000","last_change":"2024-04-01T01:31:16.384130+0000","last_active":"2024-04-01T01:32:48.588732+0000","last_peered":"2024-04-01T01:32:48.588732+0000","last_clean":"2024-04-01T01:32:48.588732+0000","last_became_active":"2024-04-01T01:31:16.383400+0000","last_became_peered":"2024-04-01T01:31:16.383400+0000","last_unstale":"2024-04-01T01:32:48.588732+0000","last_undegraded":"2024-04-01T01:32:48.588732+0000","last_fullsized":"2024-04-01T01:32:48.588732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:22:10.860325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.686306+0000","last_change":"2024-04-01T01:31:20.571052+0000","last_active":"2024-04-01T01:32:19.686306+0000","last_peered":"2024-04-01T01:32:19.686306+0000","last_clean":"2024-04-01T01:32:19.686306+0000","last_became_active":"2024-04-01T01:31:20.570617+0000","last_became_peered":"2024-04-01T01:31:20.570617+0000","last_unstale":"2024-04-01T01:32:19.686306+0000","last_undegraded":"2024-04-01T01:32:19.686306+0000","last_fullsized":"2024-04-01T01:32:19.686306+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:53:51.195056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.689102+0000","last_change":"2024-04-01T01:31:15.369820+0000","last_active":"2024-04-01T01:32:19.689102+0000","last_peered":"2024-04-01T01:32:19.689102+0000","last_clean":"2024-04-01T01:32:19.689102+0000","last_became_active":"2024-04-01T01:31:15.369378+0000","last_became_peered":"2024-04-01T01:31:15.369378+0000","last_unstale":"2024-04-01T01:32:19.689102+0000","last_undegraded":"2024-04-01T01:32:19.689102+0000","last_fullsized":"2024-04-01T01:32:19.689102+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:37:34.351220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.602880+0000","last_change":"2024-04-01T01:31:16.383352+0000","last_active":"2024-04-01T01:32:48.602880+0000","last_peered":"2024-04-01T01:32:48.602880+0000","last_clean":"2024-04-01T01:32:48.602880+0000","last_became_active":"2024-04-01T01:31:16.382898+0000","last_became_peered":"2024-04-01T01:31:16.382898+0000","last_unstale":"2024-04-01T01:32:48.602880+0000","last_undegraded":"2024-04-01T01:32:48.602880+0000","last_fullsized":"2024-04-01T01:32:48.602880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:27.854459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.684722+0000","last_change":"2024-04-01T01:31:20.576578+0000","last_active":"2024-04-01T01:32:19.684722+0000","last_peered":"2024-04-01T01:32:19.684722+0000","last_clean":"2024-04-01T01:32:19.684722+0000","last_became_active":"2024-04-01T01:31:20.575996+0000","last_became_peered":"2024-04-01T01:31:20.575996+0000","last_unstale":"2024-04-01T01:32:19.684722+0000","last_undegraded":"2024-04-01T01:32:19.684722+0000","last_fullsized":"2024-04-01T01:32:19.684722+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:52:08.808357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685084+0000","last_change":"2024-04-01T01:31:15.377246+0000","last_active":"2024-04-01T01:32:19.685084+0000","last_peered":"2024-04-01T01:32:19.685084+0000","last_clean":"2024-04-01T01:32:19.685084+0000","last_became_active":"2024-04-01T01:31:15.376957+0000","last_became_peered":"2024-04-01T01:31:15.376957+0000","last_unstale":"2024-04-01T01:32:19.685084+0000","last_undegraded":"2024-04-01T01:32:19.685084+0000","last_fullsized":"2024-04-01T01:32:19.685084+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:59:34.313984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.604754+0000","last_change":"2024-04-01T01:31:16.379031+0000","last_active":"2024-04-01T01:32:48.604754+0000","last_peered":"2024-04-01T01:32:48.604754+0000","last_clean":"2024-04-01T01:32:48.604754+0000","last_became_active":"2024-04-01T01:31:16.378593+0000","last_became_peered":"2024-04-01T01:31:16.378593+0000","last_unstale":"2024-04-01T01:32:48.604754+0000","last_undegraded":"2024-04-01T01:32:48.604754+0000","last_fullsized":"2024-04-01T01:32:48.604754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:31.816103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.686568+0000","last_change":"2024-04-01T01:31:20.578660+0000","last_active":"2024-04-01T01:32:19.686568+0000","last_peered":"2024-04-01T01:32:19.686568+0000","last_clean":"2024-04-01T01:32:19.686568+0000","last_became_active":"2024-04-01T01:31:20.578082+0000","last_became_peered":"2024-04-01T01:31:20.578082+0000","last_unstale":"2024-04-01T01:32:19.686568+0000","last_undegraded":"2024-04-01T01:32:19.686568+0000","last_fullsized":"2024-04-01T01:32:19.686568+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:02:56.570816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":181,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:50.667044+0000","last_change":"2024-04-01T01:31:15.373428+0000","last_active":"2024-04-01T01:32:50.667044+0000","last_peered":"2024-04-01T01:32:50.667044+0000","last_clean":"2024-04-01T01:32:50.667044+0000","last_became_active":"2024-04-01T01:31:15.372965+0000","last_became_peered":"2024-04-01T01:31:15.372965+0000","last_unstale":"2024-04-01T01:32:50.667044+0000","last_undegraded":"2024-04-01T01:32:50.667044+0000","last_fullsized":"2024-04-01T01:32:50.667044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:55:42.211055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":149,"num_read_kb":149,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.597812+0000","last_change":"2024-04-01T01:31:16.382961+0000","last_active":"2024-04-01T01:32:48.597812+0000","last_peered":"2024-04-01T01:32:48.597812+0000","last_clean":"2024-04-01T01:32:48.597812+0000","last_became_active":"2024-04-01T01:31:16.382527+0000","last_became_peered":"2024-04-01T01:31:16.382527+0000","last_unstale":"2024-04-01T01:32:48.597812+0000","last_undegraded":"2024-04-01T01:32:48.597812+0000","last_fullsized":"2024-04-01T01:32:48.597812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:46:20.250064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.682963+0000","last_change":"2024-04-01T01:31:20.571842+0000","last_active":"2024-04-01T01:32:19.682963+0000","last_peered":"2024-04-01T01:32:19.682963+0000","last_clean":"2024-04-01T01:32:19.682963+0000","last_became_active":"2024-04-01T01:31:20.571394+0000","last_became_peered":"2024-04-01T01:31:20.571394+0000","last_unstale":"2024-04-01T01:32:19.682963+0000","last_undegraded":"2024-04-01T01:32:19.682963+0000","last_fullsized":"2024-04-01T01:32:19.682963+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:51:36.512277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.686377+0000","last_change":"2024-04-01T01:31:15.379737+0000","last_active":"2024-04-01T01:32:19.686377+0000","last_peered":"2024-04-01T01:32:19.686377+0000","last_clean":"2024-04-01T01:32:19.686377+0000","last_became_active":"2024-04-01T01:31:15.379158+0000","last_became_peered":"2024-04-01T01:31:15.379158+0000","last_unstale":"2024-04-01T01:32:19.686377+0000","last_undegraded":"2024-04-01T01:32:19.686377+0000","last_fullsized":"2024-04-01T01:32:19.686377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:12:44.474772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.589078+0000","last_change":"2024-04-01T01:31:16.386059+0000","last_active":"2024-04-01T01:32:48.589078+0000","last_peered":"2024-04-01T01:32:48.589078+0000","last_clean":"2024-04-01T01:32:48.589078+0000","last_became_active":"2024-04-01T01:31:16.384551+0000","last_became_peered":"2024-04-01T01:31:16.384551+0000","last_unstale":"2024-04-01T01:32:48.589078+0000","last_undegraded":"2024-04-01T01:32:48.589078+0000","last_fullsized":"2024-04-01T01:32:48.589078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:52:16.594485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683310+0000","last_change":"2024-04-01T01:31:20.566937+0000","last_active":"2024-04-01T01:32:19.683310+0000","last_peered":"2024-04-01T01:32:19.683310+0000","last_clean":"2024-04-01T01:32:19.683310+0000","last_became_active":"2024-04-01T01:31:20.566447+0000","last_became_peered":"2024-04-01T01:31:20.566447+0000","last_unstale":"2024-04-01T01:32:19.683310+0000","last_undegraded":"2024-04-01T01:32:19.683310+0000","last_fullsized":"2024-04-01T01:32:19.683310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:17:37.845166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685433+0000","last_change":"2024-04-01T01:31:15.375669+0000","last_active":"2024-04-01T01:32:19.685433+0000","last_peered":"2024-04-01T01:32:19.685433+0000","last_clean":"2024-04-01T01:32:19.685433+0000","last_became_active":"2024-04-01T01:31:15.375353+0000","last_became_peered":"2024-04-01T01:31:15.375353+0000","last_unstale":"2024-04-01T01:32:19.685433+0000","last_undegraded":"2024-04-01T01:32:19.685433+0000","last_fullsized":"2024-04-01T01:32:19.685433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:47:43.096898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.599676+0000","last_change":"2024-04-01T01:31:16.388399+0000","last_active":"2024-04-01T01:32:48.599676+0000","last_peered":"2024-04-01T01:32:48.599676+0000","last_clean":"2024-04-01T01:32:48.599676+0000","last_became_active":"2024-04-01T01:31:16.388059+0000","last_became_peered":"2024-04-01T01:31:16.388059+0000","last_unstale":"2024-04-01T01:32:48.599676+0000","last_undegraded":"2024-04-01T01:32:48.599676+0000","last_fullsized":"2024-04-01T01:32:48.599676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:59:50.907802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.693040+0000","last_change":"2024-04-01T01:32:19.692098+0000","last_active":"2024-04-01T01:32:19.693040+0000","last_peered":"2024-04-01T01:32:19.693040+0000","last_clean":"2024-04-01T01:32:19.693040+0000","last_became_active":"2024-04-01T01:32:19.691673+0000","last_became_peered":"2024-04-01T01:32:19.691673+0000","last_unstale":"2024-04-01T01:32:19.693040+0000","last_undegraded":"2024-04-01T01:32:19.693040+0000","last_fullsized":"2024-04-01T01:32:19.693040+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:14:17.300517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"91'37","reported_seq":225,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:50.721396+0000","last_change":"2024-04-01T01:31:15.376421+0000","last_active":"2024-04-01T01:32:50.721396+0000","last_peered":"2024-04-01T01:32:50.721396+0000","last_clean":"2024-04-01T01:32:50.721396+0000","last_became_active":"2024-04-01T01:31:15.375985+0000","last_became_peered":"2024-04-01T01:31:15.375985+0000","last_unstale":"2024-04-01T01:32:50.721396+0000","last_undegraded":"2024-04-01T01:32:50.721396+0000","last_fullsized":"2024-04-01T01:32:50.721396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:37:29.121436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63605,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":161,"num_read_kb":143,"num_write":39,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.578217+0000","last_change":"2024-04-01T01:31:16.380641+0000","last_active":"2024-04-01T01:32:48.578217+0000","last_peered":"2024-04-01T01:32:48.578217+0000","last_clean":"2024-04-01T01:32:48.578217+0000","last_became_active":"2024-04-01T01:31:16.380300+0000","last_became_peered":"2024-04-01T01:31:16.380300+0000","last_unstale":"2024-04-01T01:32:48.578217+0000","last_undegraded":"2024-04-01T01:32:48.578217+0000","last_fullsized":"2024-04-01T01:32:48.578217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:26:08.240694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685977+0000","last_change":"2024-04-01T01:31:20.576094+0000","last_active":"2024-04-01T01:32:19.685977+0000","last_peered":"2024-04-01T01:32:19.685977+0000","last_clean":"2024-04-01T01:32:19.685977+0000","last_became_active":"2024-04-01T01:31:20.575674+0000","last_became_peered":"2024-04-01T01:31:20.575674+0000","last_unstale":"2024-04-01T01:32:19.685977+0000","last_undegraded":"2024-04-01T01:32:19.685977+0000","last_fullsized":"2024-04-01T01:32:19.685977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:42:19.352398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683520+0000","last_change":"2024-04-01T01:31:15.375136+0000","last_active":"2024-04-01T01:32:19.683520+0000","last_peered":"2024-04-01T01:32:19.683520+0000","last_clean":"2024-04-01T01:32:19.683520+0000","last_became_active":"2024-04-01T01:31:15.374813+0000","last_became_peered":"2024-04-01T01:31:15.374813+0000","last_unstale":"2024-04-01T01:32:19.683520+0000","last_undegraded":"2024-04-01T01:32:19.683520+0000","last_fullsized":"2024-04-01T01:32:19.683520+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:19:05.490103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.593781+0000","last_change":"2024-04-01T01:31:16.380207+0000","last_active":"2024-04-01T01:32:48.593781+0000","last_peered":"2024-04-01T01:32:48.593781+0000","last_clean":"2024-04-01T01:32:48.593781+0000","last_became_active":"2024-04-01T01:31:16.379891+0000","last_became_peered":"2024-04-01T01:31:16.379891+0000","last_unstale":"2024-04-01T01:32:48.593781+0000","last_undegraded":"2024-04-01T01:32:48.593781+0000","last_fullsized":"2024-04-01T01:32:48.593781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:12:36.417162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.686023+0000","last_change":"2024-04-01T01:31:20.569978+0000","last_active":"2024-04-01T01:32:19.686023+0000","last_peered":"2024-04-01T01:32:19.686023+0000","last_clean":"2024-04-01T01:32:19.686023+0000","last_became_active":"2024-04-01T01:31:20.568298+0000","last_became_peered":"2024-04-01T01:31:20.568298+0000","last_unstale":"2024-04-01T01:32:19.686023+0000","last_undegraded":"2024-04-01T01:32:19.686023+0000","last_fullsized":"2024-04-01T01:32:19.686023+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:42:16.439284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685854+0000","last_change":"2024-04-01T01:31:15.380143+0000","last_active":"2024-04-01T01:32:19.685854+0000","last_peered":"2024-04-01T01:32:19.685854+0000","last_clean":"2024-04-01T01:32:19.685854+0000","last_became_active":"2024-04-01T01:31:15.379835+0000","last_became_peered":"2024-04-01T01:31:15.379835+0000","last_unstale":"2024-04-01T01:32:19.685854+0000","last_undegraded":"2024-04-01T01:32:19.685854+0000","last_fullsized":"2024-04-01T01:32:19.685854+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:28:34.213626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.601715+0000","last_change":"2024-04-01T01:31:16.380290+0000","last_active":"2024-04-01T01:32:48.601715+0000","last_peered":"2024-04-01T01:32:48.601715+0000","last_clean":"2024-04-01T01:32:48.601715+0000","last_became_active":"2024-04-01T01:31:16.379983+0000","last_became_peered":"2024-04-01T01:31:16.379983+0000","last_unstale":"2024-04-01T01:32:48.601715+0000","last_undegraded":"2024-04-01T01:32:48.601715+0000","last_fullsized":"2024-04-01T01:32:48.601715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:50.135454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683163+0000","last_change":"2024-04-01T01:31:20.573500+0000","last_active":"2024-04-01T01:32:19.683163+0000","last_peered":"2024-04-01T01:32:19.683163+0000","last_clean":"2024-04-01T01:32:19.683163+0000","last_became_active":"2024-04-01T01:31:20.573101+0000","last_became_peered":"2024-04-01T01:31:20.573101+0000","last_unstale":"2024-04-01T01:32:19.683163+0000","last_undegraded":"2024-04-01T01:32:19.683163+0000","last_fullsized":"2024-04-01T01:32:19.683163+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:11:27.881703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":111,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:50.673472+0000","last_change":"2024-04-01T01:31:15.377265+0000","last_active":"2024-04-01T01:32:50.673472+0000","last_peered":"2024-04-01T01:32:50.673472+0000","last_clean":"2024-04-01T01:32:50.673472+0000","last_became_active":"2024-04-01T01:31:15.376889+0000","last_became_peered":"2024-04-01T01:31:15.376889+0000","last_unstale":"2024-04-01T01:32:50.673472+0000","last_undegraded":"2024-04-01T01:32:50.673472+0000","last_fullsized":"2024-04-01T01:32:50.673472+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:43:31.582866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.591850+0000","last_change":"2024-04-01T01:31:16.384982+0000","last_active":"2024-04-01T01:32:48.591850+0000","last_peered":"2024-04-01T01:32:48.591850+0000","last_clean":"2024-04-01T01:32:48.591850+0000","last_became_active":"2024-04-01T01:31:16.384663+0000","last_became_peered":"2024-04-01T01:31:16.384663+0000","last_unstale":"2024-04-01T01:32:48.591850+0000","last_undegraded":"2024-04-01T01:32:48.591850+0000","last_fullsized":"2024-04-01T01:32:48.591850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:14:57.351768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.689033+0000","last_change":"2024-04-01T01:31:20.578761+0000","last_active":"2024-04-01T01:32:19.689033+0000","last_peered":"2024-04-01T01:32:19.689033+0000","last_clean":"2024-04-01T01:32:19.689033+0000","last_became_active":"2024-04-01T01:31:20.578204+0000","last_became_peered":"2024-04-01T01:31:20.578204+0000","last_unstale":"2024-04-01T01:32:19.689033+0000","last_undegraded":"2024-04-01T01:32:19.689033+0000","last_fullsized":"2024-04-01T01:32:19.689033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:19:33.584872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.689295+0000","last_change":"2024-04-01T01:31:15.380773+0000","last_active":"2024-04-01T01:32:19.689295+0000","last_peered":"2024-04-01T01:32:19.689295+0000","last_clean":"2024-04-01T01:32:19.689295+0000","last_became_active":"2024-04-01T01:31:15.380472+0000","last_became_peered":"2024-04-01T01:31:15.380472+0000","last_unstale":"2024-04-01T01:32:19.689295+0000","last_undegraded":"2024-04-01T01:32:19.689295+0000","last_fullsized":"2024-04-01T01:32:19.689295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:34:28.294500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.599193+0000","last_change":"2024-04-01T01:31:16.374658+0000","last_active":"2024-04-01T01:32:48.599193+0000","last_peered":"2024-04-01T01:32:48.599193+0000","last_clean":"2024-04-01T01:32:48.599193+0000","last_became_active":"2024-04-01T01:31:16.373596+0000","last_became_peered":"2024-04-01T01:31:16.373596+0000","last_unstale":"2024-04-01T01:32:48.599193+0000","last_undegraded":"2024-04-01T01:32:48.599193+0000","last_fullsized":"2024-04-01T01:32:48.599193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:28:38.477719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685238+0000","last_change":"2024-04-01T01:31:20.577449+0000","last_active":"2024-04-01T01:32:19.685238+0000","last_peered":"2024-04-01T01:32:19.685238+0000","last_clean":"2024-04-01T01:32:19.685238+0000","last_became_active":"2024-04-01T01:31:20.577026+0000","last_became_peered":"2024-04-01T01:31:20.577026+0000","last_unstale":"2024-04-01T01:32:19.685238+0000","last_undegraded":"2024-04-01T01:32:19.685238+0000","last_fullsized":"2024-04-01T01:32:19.685238+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:44:48.080304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.689118+0000","last_change":"2024-04-01T01:31:15.373885+0000","last_active":"2024-04-01T01:32:19.689118+0000","last_peered":"2024-04-01T01:32:19.689118+0000","last_clean":"2024-04-01T01:32:19.689118+0000","last_became_active":"2024-04-01T01:31:15.373578+0000","last_became_peered":"2024-04-01T01:31:15.373578+0000","last_unstale":"2024-04-01T01:32:19.689118+0000","last_undegraded":"2024-04-01T01:32:19.689118+0000","last_fullsized":"2024-04-01T01:32:19.689118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:52.641515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.588417+0000","last_change":"2024-04-01T01:31:16.379997+0000","last_active":"2024-04-01T01:32:48.588417+0000","last_peered":"2024-04-01T01:32:48.588417+0000","last_clean":"2024-04-01T01:32:48.588417+0000","last_became_active":"2024-04-01T01:31:16.379664+0000","last_became_peered":"2024-04-01T01:31:16.379664+0000","last_unstale":"2024-04-01T01:32:48.588417+0000","last_undegraded":"2024-04-01T01:32:48.588417+0000","last_fullsized":"2024-04-01T01:32:48.588417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:37:17.432095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.690581+0000","last_change":"2024-04-01T01:31:20.569835+0000","last_active":"2024-04-01T01:32:19.690581+0000","last_peered":"2024-04-01T01:32:19.690581+0000","last_clean":"2024-04-01T01:32:19.690581+0000","last_became_active":"2024-04-01T01:31:20.569195+0000","last_became_peered":"2024-04-01T01:31:20.569195+0000","last_unstale":"2024-04-01T01:32:19.690581+0000","last_undegraded":"2024-04-01T01:32:19.690581+0000","last_fullsized":"2024-04-01T01:32:19.690581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:10:18.278575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683291+0000","last_change":"2024-04-01T01:31:15.377679+0000","last_active":"2024-04-01T01:32:19.683291+0000","last_peered":"2024-04-01T01:32:19.683291+0000","last_clean":"2024-04-01T01:32:19.683291+0000","last_became_active":"2024-04-01T01:31:15.377351+0000","last_became_peered":"2024-04-01T01:31:15.377351+0000","last_unstale":"2024-04-01T01:32:19.683291+0000","last_undegraded":"2024-04-01T01:32:19.683291+0000","last_fullsized":"2024-04-01T01:32:19.683291+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:22:41.216563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.593718+0000","last_change":"2024-04-01T01:31:16.380997+0000","last_active":"2024-04-01T01:32:48.593718+0000","last_peered":"2024-04-01T01:32:48.593718+0000","last_clean":"2024-04-01T01:32:48.593718+0000","last_became_active":"2024-04-01T01:31:16.380692+0000","last_became_peered":"2024-04-01T01:31:16.380692+0000","last_unstale":"2024-04-01T01:32:48.593718+0000","last_undegraded":"2024-04-01T01:32:48.593718+0000","last_fullsized":"2024-04-01T01:32:48.593718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:09:51.220849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.689069+0000","last_change":"2024-04-01T01:31:20.573191+0000","last_active":"2024-04-01T01:32:19.689069+0000","last_peered":"2024-04-01T01:32:19.689069+0000","last_clean":"2024-04-01T01:32:19.689069+0000","last_became_active":"2024-04-01T01:31:20.572669+0000","last_became_peered":"2024-04-01T01:31:20.572669+0000","last_unstale":"2024-04-01T01:32:19.689069+0000","last_undegraded":"2024-04-01T01:32:19.689069+0000","last_fullsized":"2024-04-01T01:32:19.689069+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:01:56.491271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.655261+0000","last_change":"2024-04-01T01:31:15.379566+0000","last_active":"2024-04-01T01:32:48.655261+0000","last_peered":"2024-04-01T01:32:48.655261+0000","last_clean":"2024-04-01T01:32:48.655261+0000","last_became_active":"2024-04-01T01:31:15.379213+0000","last_became_peered":"2024-04-01T01:31:15.379213+0000","last_unstale":"2024-04-01T01:32:48.655261+0000","last_undegraded":"2024-04-01T01:32:48.655261+0000","last_fullsized":"2024-04-01T01:32:48.655261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:34:17.974422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.599236+0000","last_change":"2024-04-01T01:31:16.376218+0000","last_active":"2024-04-01T01:32:48.599236+0000","last_peered":"2024-04-01T01:32:48.599236+0000","last_clean":"2024-04-01T01:32:48.599236+0000","last_became_active":"2024-04-01T01:31:16.375887+0000","last_became_peered":"2024-04-01T01:31:16.375887+0000","last_unstale":"2024-04-01T01:32:48.599236+0000","last_undegraded":"2024-04-01T01:32:48.599236+0000","last_fullsized":"2024-04-01T01:32:48.599236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:27:03.723827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.688754+0000","last_change":"2024-04-01T01:31:20.576461+0000","last_active":"2024-04-01T01:32:19.688754+0000","last_peered":"2024-04-01T01:32:19.688754+0000","last_clean":"2024-04-01T01:32:19.688754+0000","last_became_active":"2024-04-01T01:31:20.575985+0000","last_became_peered":"2024-04-01T01:31:20.575985+0000","last_unstale":"2024-04-01T01:32:19.688754+0000","last_undegraded":"2024-04-01T01:32:19.688754+0000","last_fullsized":"2024-04-01T01:32:19.688754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:19:02.714455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.592340+0000","last_change":"2024-04-01T01:31:16.376456+0000","last_active":"2024-04-01T01:32:48.592340+0000","last_peered":"2024-04-01T01:32:48.592340+0000","last_clean":"2024-04-01T01:32:48.592340+0000","last_became_active":"2024-04-01T01:31:16.375445+0000","last_became_peered":"2024-04-01T01:31:16.375445+0000","last_unstale":"2024-04-01T01:32:48.592340+0000","last_undegraded":"2024-04-01T01:32:48.592340+0000","last_fullsized":"2024-04-01T01:32:48.592340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:07:11.802799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.670194+0000","last_change":"2024-04-01T01:31:15.382702+0000","last_active":"2024-04-01T01:32:48.670194+0000","last_peered":"2024-04-01T01:32:48.670194+0000","last_clean":"2024-04-01T01:32:48.670194+0000","last_became_active":"2024-04-01T01:31:15.382377+0000","last_became_peered":"2024-04-01T01:31:15.382377+0000","last_unstale":"2024-04-01T01:32:48.670194+0000","last_undegraded":"2024-04-01T01:32:48.670194+0000","last_fullsized":"2024-04-01T01:32:48.670194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:45.080992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.704607+0000","last_change":"2024-04-01T01:32:19.704607+0000","last_active":"2024-04-01T01:32:19.704607+0000","last_peered":"2024-04-01T01:32:19.704607+0000","last_clean":"2024-04-01T01:32:19.704607+0000","last_became_active":"2024-04-01T01:32:19.694650+0000","last_became_peered":"2024-04-01T01:32:19.694650+0000","last_unstale":"2024-04-01T01:32:19.704607+0000","last_undegraded":"2024-04-01T01:32:19.704607+0000","last_fullsized":"2024-04-01T01:32:19.704607+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:48:55.439187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.581815+0000","last_change":"2024-04-01T01:31:16.387981+0000","last_active":"2024-04-01T01:32:48.581815+0000","last_peered":"2024-04-01T01:32:48.581815+0000","last_clean":"2024-04-01T01:32:48.581815+0000","last_became_active":"2024-04-01T01:31:16.387594+0000","last_became_peered":"2024-04-01T01:31:16.387594+0000","last_unstale":"2024-04-01T01:32:48.581815+0000","last_undegraded":"2024-04-01T01:32:48.581815+0000","last_fullsized":"2024-04-01T01:32:48.581815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:17.148752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683288+0000","last_change":"2024-04-01T01:31:15.375212+0000","last_active":"2024-04-01T01:32:19.683288+0000","last_peered":"2024-04-01T01:32:19.683288+0000","last_clean":"2024-04-01T01:32:19.683288+0000","last_became_active":"2024-04-01T01:31:15.374882+0000","last_became_peered":"2024-04-01T01:31:15.374882+0000","last_unstale":"2024-04-01T01:32:19.683288+0000","last_undegraded":"2024-04-01T01:32:19.683288+0000","last_fullsized":"2024-04-01T01:32:19.683288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:56:23.412025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":352,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685726+0000","last_change":"2024-04-01T01:30:03.538590+0000","last_active":"2024-04-01T01:32:19.685726+0000","last_peered":"2024-04-01T01:32:19.685726+0000","last_clean":"2024-04-01T01:32:19.685726+0000","last_became_active":"2024-04-01T01:30:03.538098+0000","last_became_peered":"2024-04-01T01:30:03.538098+0000","last_unstale":"2024-04-01T01:32:19.685726+0000","last_undegraded":"2024-04-01T01:32:19.685726+0000","last_fullsized":"2024-04-01T01:32:19.685726+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:26:43.445198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:26:43.445198+0000","last_clean_scrub_stamp":"2024-04-01T01:26:43.445198+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:22.207795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.686086+0000","last_change":"2024-04-01T01:31:20.573260+0000","last_active":"2024-04-01T01:32:19.686086+0000","last_peered":"2024-04-01T01:32:19.686086+0000","last_clean":"2024-04-01T01:32:19.686086+0000","last_became_active":"2024-04-01T01:31:20.572846+0000","last_became_peered":"2024-04-01T01:31:20.572846+0000","last_unstale":"2024-04-01T01:32:19.686086+0000","last_undegraded":"2024-04-01T01:32:19.686086+0000","last_fullsized":"2024-04-01T01:32:19.686086+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:24.986070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.593706+0000","last_change":"2024-04-01T01:31:16.384782+0000","last_active":"2024-04-01T01:32:48.593706+0000","last_peered":"2024-04-01T01:32:48.593706+0000","last_clean":"2024-04-01T01:32:48.593706+0000","last_became_active":"2024-04-01T01:31:16.384420+0000","last_became_peered":"2024-04-01T01:31:16.384420+0000","last_unstale":"2024-04-01T01:32:48.593706+0000","last_undegraded":"2024-04-01T01:32:48.593706+0000","last_fullsized":"2024-04-01T01:32:48.593706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:00:20.902022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685788+0000","last_change":"2024-04-01T01:31:15.381853+0000","last_active":"2024-04-01T01:32:19.685788+0000","last_peered":"2024-04-01T01:32:19.685788+0000","last_clean":"2024-04-01T01:32:19.685788+0000","last_became_active":"2024-04-01T01:31:15.381496+0000","last_became_peered":"2024-04-01T01:31:15.381496+0000","last_unstale":"2024-04-01T01:32:19.685788+0000","last_undegraded":"2024-04-01T01:32:19.685788+0000","last_fullsized":"2024-04-01T01:32:19.685788+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:07:51.440351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685308+0000","last_change":"2024-04-01T01:31:20.568705+0000","last_active":"2024-04-01T01:32:19.685308+0000","last_peered":"2024-04-01T01:32:19.685308+0000","last_clean":"2024-04-01T01:32:19.685308+0000","last_became_active":"2024-04-01T01:31:20.568252+0000","last_became_peered":"2024-04-01T01:31:20.568252+0000","last_unstale":"2024-04-01T01:32:19.685308+0000","last_undegraded":"2024-04-01T01:32:19.685308+0000","last_fullsized":"2024-04-01T01:32:19.685308+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:22:28.509510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.700251+0000","last_change":"2024-04-01T01:32:19.699781+0000","last_active":"2024-04-01T01:32:19.700251+0000","last_peered":"2024-04-01T01:32:19.700251+0000","last_clean":"2024-04-01T01:32:19.700251+0000","last_became_active":"2024-04-01T01:32:19.689593+0000","last_became_peered":"2024-04-01T01:32:19.689593+0000","last_unstale":"2024-04-01T01:32:19.700251+0000","last_undegraded":"2024-04-01T01:32:19.700251+0000","last_fullsized":"2024-04-01T01:32:19.700251+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:44:21.922097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.584119+0000","last_change":"2024-04-01T01:31:16.377002+0000","last_active":"2024-04-01T01:32:48.584119+0000","last_peered":"2024-04-01T01:32:48.584119+0000","last_clean":"2024-04-01T01:32:48.584119+0000","last_became_active":"2024-04-01T01:31:16.376679+0000","last_became_peered":"2024-04-01T01:31:16.376679+0000","last_unstale":"2024-04-01T01:32:48.584119+0000","last_undegraded":"2024-04-01T01:32:48.584119+0000","last_fullsized":"2024-04-01T01:32:48.584119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:50:24.126192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.686795+0000","last_change":"2024-04-01T01:31:20.577757+0000","last_active":"2024-04-01T01:32:19.686795+0000","last_peered":"2024-04-01T01:32:19.686795+0000","last_clean":"2024-04-01T01:32:19.686795+0000","last_became_active":"2024-04-01T01:31:20.577378+0000","last_became_peered":"2024-04-01T01:31:20.577378+0000","last_unstale":"2024-04-01T01:32:19.686795+0000","last_undegraded":"2024-04-01T01:32:19.686795+0000","last_fullsized":"2024-04-01T01:32:19.686795+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:48:04.511229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'13","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683655+0000","last_change":"2024-04-01T01:31:15.383201+0000","last_active":"2024-04-01T01:32:19.683655+0000","last_peered":"2024-04-01T01:32:19.683655+0000","last_clean":"2024-04-01T01:32:19.683655+0000","last_became_active":"2024-04-01T01:31:15.382320+0000","last_became_peered":"2024-04-01T01:31:15.382320+0000","last_unstale":"2024-04-01T01:32:19.683655+0000","last_undegraded":"2024-04-01T01:32:19.683655+0000","last_fullsized":"2024-04-01T01:32:19.683655+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:15:47.138024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.603330+0000","last_change":"2024-04-01T01:31:16.386936+0000","last_active":"2024-04-01T01:32:48.603330+0000","last_peered":"2024-04-01T01:32:48.603330+0000","last_clean":"2024-04-01T01:32:48.603330+0000","last_became_active":"2024-04-01T01:31:16.384448+0000","last_became_peered":"2024-04-01T01:31:16.384448+0000","last_unstale":"2024-04-01T01:32:48.603330+0000","last_undegraded":"2024-04-01T01:32:48.603330+0000","last_fullsized":"2024-04-01T01:32:48.603330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:16:42.243250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683611+0000","last_change":"2024-04-01T01:31:20.565570+0000","last_active":"2024-04-01T01:32:19.683611+0000","last_peered":"2024-04-01T01:32:19.683611+0000","last_clean":"2024-04-01T01:32:19.683611+0000","last_became_active":"2024-04-01T01:31:20.565141+0000","last_became_peered":"2024-04-01T01:31:20.565141+0000","last_unstale":"2024-04-01T01:32:19.683611+0000","last_undegraded":"2024-04-01T01:32:19.683611+0000","last_fullsized":"2024-04-01T01:32:19.683611+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:47:12.844970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683841+0000","last_change":"2024-04-01T01:31:15.381343+0000","last_active":"2024-04-01T01:32:19.683841+0000","last_peered":"2024-04-01T01:32:19.683841+0000","last_clean":"2024-04-01T01:32:19.683841+0000","last_became_active":"2024-04-01T01:31:15.380994+0000","last_became_peered":"2024-04-01T01:31:15.380994+0000","last_unstale":"2024-04-01T01:32:19.683841+0000","last_undegraded":"2024-04-01T01:32:19.683841+0000","last_fullsized":"2024-04-01T01:32:19.683841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:19:10.710285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.607597+0000","last_change":"2024-04-01T01:31:16.383951+0000","last_active":"2024-04-01T01:32:48.607597+0000","last_peered":"2024-04-01T01:32:48.607597+0000","last_clean":"2024-04-01T01:32:48.607597+0000","last_became_active":"2024-04-01T01:31:16.383657+0000","last_became_peered":"2024-04-01T01:31:16.383657+0000","last_unstale":"2024-04-01T01:32:48.607597+0000","last_undegraded":"2024-04-01T01:32:48.607597+0000","last_fullsized":"2024-04-01T01:32:48.607597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:55:45.216479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683439+0000","last_change":"2024-04-01T01:31:20.569416+0000","last_active":"2024-04-01T01:32:19.683439+0000","last_peered":"2024-04-01T01:32:19.683439+0000","last_clean":"2024-04-01T01:32:19.683439+0000","last_became_active":"2024-04-01T01:31:20.569007+0000","last_became_peered":"2024-04-01T01:31:20.569007+0000","last_unstale":"2024-04-01T01:32:19.683439+0000","last_undegraded":"2024-04-01T01:32:19.683439+0000","last_fullsized":"2024-04-01T01:32:19.683439+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:25:36.770919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.804186+0000","last_change":"2024-04-01T01:31:15.373444+0000","last_active":"2024-04-01T01:32:19.804186+0000","last_peered":"2024-04-01T01:32:19.804186+0000","last_clean":"2024-04-01T01:32:19.804186+0000","last_became_active":"2024-04-01T01:31:15.373094+0000","last_became_peered":"2024-04-01T01:31:15.373094+0000","last_unstale":"2024-04-01T01:32:19.804186+0000","last_undegraded":"2024-04-01T01:32:19.804186+0000","last_fullsized":"2024-04-01T01:32:19.804186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:26:27.443832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.545765+0000","last_change":"2024-04-01T01:31:16.381441+0000","last_active":"2024-04-01T01:32:48.545765+0000","last_peered":"2024-04-01T01:32:48.545765+0000","last_clean":"2024-04-01T01:32:48.545765+0000","last_became_active":"2024-04-01T01:31:16.381092+0000","last_became_peered":"2024-04-01T01:31:16.381092+0000","last_unstale":"2024-04-01T01:32:48.545765+0000","last_undegraded":"2024-04-01T01:32:48.545765+0000","last_fullsized":"2024-04-01T01:32:48.545765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:47:00.229090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683708+0000","last_change":"2024-04-01T01:31:20.569672+0000","last_active":"2024-04-01T01:32:19.683708+0000","last_peered":"2024-04-01T01:32:19.683708+0000","last_clean":"2024-04-01T01:32:19.683708+0000","last_became_active":"2024-04-01T01:31:20.569276+0000","last_became_peered":"2024-04-01T01:31:20.569276+0000","last_unstale":"2024-04-01T01:32:19.683708+0000","last_undegraded":"2024-04-01T01:32:19.683708+0000","last_fullsized":"2024-04-01T01:32:19.683708+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:39:03.331428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.686644+0000","last_change":"2024-04-01T01:31:15.376094+0000","last_active":"2024-04-01T01:32:19.686644+0000","last_peered":"2024-04-01T01:32:19.686644+0000","last_clean":"2024-04-01T01:32:19.686644+0000","last_became_active":"2024-04-01T01:31:15.375169+0000","last_became_peered":"2024-04-01T01:31:15.375169+0000","last_unstale":"2024-04-01T01:32:19.686644+0000","last_undegraded":"2024-04-01T01:32:19.686644+0000","last_fullsized":"2024-04-01T01:32:19.686644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:18:20.710644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.608411+0000","last_change":"2024-04-01T01:31:16.378514+0000","last_active":"2024-04-01T01:32:48.608411+0000","last_peered":"2024-04-01T01:32:48.608411+0000","last_clean":"2024-04-01T01:32:48.608411+0000","last_became_active":"2024-04-01T01:31:16.377990+0000","last_became_peered":"2024-04-01T01:31:16.377990+0000","last_unstale":"2024-04-01T01:32:48.608411+0000","last_undegraded":"2024-04-01T01:32:48.608411+0000","last_fullsized":"2024-04-01T01:32:48.608411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:27:29.829775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683420+0000","last_change":"2024-04-01T01:31:20.567892+0000","last_active":"2024-04-01T01:32:19.683420+0000","last_peered":"2024-04-01T01:32:19.683420+0000","last_clean":"2024-04-01T01:32:19.683420+0000","last_became_active":"2024-04-01T01:31:20.567525+0000","last_became_peered":"2024-04-01T01:31:20.567525+0000","last_unstale":"2024-04-01T01:32:19.683420+0000","last_undegraded":"2024-04-01T01:32:19.683420+0000","last_fullsized":"2024-04-01T01:32:19.683420+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:57.828691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.703872+0000","last_change":"2024-04-01T01:32:19.703414+0000","last_active":"2024-04-01T01:32:19.703872+0000","last_peered":"2024-04-01T01:32:19.703872+0000","last_clean":"2024-04-01T01:32:19.703872+0000","last_became_active":"2024-04-01T01:32:19.693581+0000","last_became_peered":"2024-04-01T01:32:19.693581+0000","last_unstale":"2024-04-01T01:32:19.703872+0000","last_undegraded":"2024-04-01T01:32:19.703872+0000","last_fullsized":"2024-04-01T01:32:19.703872+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:53:17.897154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.602653+0000","last_change":"2024-04-01T01:31:16.387570+0000","last_active":"2024-04-01T01:32:48.602653+0000","last_peered":"2024-04-01T01:32:48.602653+0000","last_clean":"2024-04-01T01:32:48.602653+0000","last_became_active":"2024-04-01T01:31:16.387200+0000","last_became_peered":"2024-04-01T01:31:16.387200+0000","last_unstale":"2024-04-01T01:32:48.602653+0000","last_undegraded":"2024-04-01T01:32:48.602653+0000","last_fullsized":"2024-04-01T01:32:48.602653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:52:27.813722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.683718+0000","last_change":"2024-04-01T01:31:20.569668+0000","last_active":"2024-04-01T01:32:19.683718+0000","last_peered":"2024-04-01T01:32:19.683718+0000","last_clean":"2024-04-01T01:32:19.683718+0000","last_became_active":"2024-04-01T01:31:20.569148+0000","last_became_peered":"2024-04-01T01:31:20.569148+0000","last_unstale":"2024-04-01T01:32:19.683718+0000","last_undegraded":"2024-04-01T01:32:19.683718+0000","last_fullsized":"2024-04-01T01:32:19.683718+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:20:56.048052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.689799+0000","last_change":"2024-04-01T01:31:15.379323+0000","last_active":"2024-04-01T01:32:19.689799+0000","last_peered":"2024-04-01T01:32:19.689799+0000","last_clean":"2024-04-01T01:32:19.689799+0000","last_became_active":"2024-04-01T01:31:15.379023+0000","last_became_peered":"2024-04-01T01:31:15.379023+0000","last_unstale":"2024-04-01T01:32:19.689799+0000","last_undegraded":"2024-04-01T01:32:19.689799+0000","last_fullsized":"2024-04-01T01:32:19.689799+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:05:37.709040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.600667+0000","last_change":"2024-04-01T01:31:16.387250+0000","last_active":"2024-04-01T01:32:48.600667+0000","last_peered":"2024-04-01T01:32:48.600667+0000","last_clean":"2024-04-01T01:32:48.600667+0000","last_became_active":"2024-04-01T01:31:16.386940+0000","last_became_peered":"2024-04-01T01:31:16.386940+0000","last_unstale":"2024-04-01T01:32:48.600667+0000","last_undegraded":"2024-04-01T01:32:48.600667+0000","last_fullsized":"2024-04-01T01:32:48.600667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:19:34.886738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.804516+0000","last_change":"2024-04-01T01:31:20.573228+0000","last_active":"2024-04-01T01:32:19.804516+0000","last_peered":"2024-04-01T01:32:19.804516+0000","last_clean":"2024-04-01T01:32:19.804516+0000","last_became_active":"2024-04-01T01:31:20.572681+0000","last_became_peered":"2024-04-01T01:31:20.572681+0000","last_unstale":"2024-04-01T01:32:19.804516+0000","last_undegraded":"2024-04-01T01:32:19.804516+0000","last_fullsized":"2024-04-01T01:32:19.804516+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:58:17.812966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":109,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:51.668566+0000","last_change":"2024-04-01T01:31:15.381477+0000","last_active":"2024-04-01T01:32:51.668566+0000","last_peered":"2024-04-01T01:32:51.668566+0000","last_clean":"2024-04-01T01:32:51.668566+0000","last_became_active":"2024-04-01T01:31:15.381166+0000","last_became_peered":"2024-04-01T01:31:15.381166+0000","last_unstale":"2024-04-01T01:32:51.668566+0000","last_undegraded":"2024-04-01T01:32:51.668566+0000","last_fullsized":"2024-04-01T01:32:51.668566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:19.029896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":77,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.594630+0000","last_change":"2024-04-01T01:31:16.386234+0000","last_active":"2024-04-01T01:32:48.594630+0000","last_peered":"2024-04-01T01:32:48.594630+0000","last_clean":"2024-04-01T01:32:48.594630+0000","last_became_active":"2024-04-01T01:31:16.385876+0000","last_became_peered":"2024-04-01T01:31:16.385876+0000","last_unstale":"2024-04-01T01:32:48.594630+0000","last_undegraded":"2024-04-01T01:32:48.594630+0000","last_fullsized":"2024-04-01T01:32:48.594630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:34.187210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.805473+0000","last_change":"2024-04-01T01:32:19.805473+0000","last_active":"2024-04-01T01:32:19.805473+0000","last_peered":"2024-04-01T01:32:19.805473+0000","last_clean":"2024-04-01T01:32:19.805473+0000","last_became_active":"2024-04-01T01:32:19.805151+0000","last_became_peered":"2024-04-01T01:32:19.805151+0000","last_unstale":"2024-04-01T01:32:19.805473+0000","last_undegraded":"2024-04-01T01:32:19.805473+0000","last_fullsized":"2024-04-01T01:32:19.805473+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:47:54.539407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685654+0000","last_change":"2024-04-01T01:31:15.378503+0000","last_active":"2024-04-01T01:32:19.685654+0000","last_peered":"2024-04-01T01:32:19.685654+0000","last_clean":"2024-04-01T01:32:19.685654+0000","last_became_active":"2024-04-01T01:31:15.378152+0000","last_became_peered":"2024-04-01T01:31:15.378152+0000","last_unstale":"2024-04-01T01:32:19.685654+0000","last_undegraded":"2024-04-01T01:32:19.685654+0000","last_fullsized":"2024-04-01T01:32:19.685654+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:15:42.118691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.601369+0000","last_change":"2024-04-01T01:31:16.389676+0000","last_active":"2024-04-01T01:32:48.601369+0000","last_peered":"2024-04-01T01:32:48.601369+0000","last_clean":"2024-04-01T01:32:48.601369+0000","last_became_active":"2024-04-01T01:31:16.389304+0000","last_became_peered":"2024-04-01T01:31:16.389304+0000","last_unstale":"2024-04-01T01:32:48.601369+0000","last_undegraded":"2024-04-01T01:32:48.601369+0000","last_fullsized":"2024-04-01T01:32:48.601369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:20:24.021273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.685303+0000","last_change":"2024-04-01T01:31:20.569461+0000","last_active":"2024-04-01T01:32:19.685303+0000","last_peered":"2024-04-01T01:32:19.685303+0000","last_clean":"2024-04-01T01:32:19.685303+0000","last_became_active":"2024-04-01T01:31:20.569011+0000","last_became_peered":"2024-04-01T01:31:20.569011+0000","last_unstale":"2024-04-01T01:32:19.685303+0000","last_undegraded":"2024-04-01T01:32:19.685303+0000","last_fullsized":"2024-04-01T01:32:19.685303+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:43:43.657065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:19.688923+0000","last_change":"2024-04-01T01:31:15.384181+0000","last_active":"2024-04-01T01:32:19.688923+0000","last_peered":"2024-04-01T01:32:19.688923+0000","last_clean":"2024-04-01T01:32:19.688923+0000","last_became_active":"2024-04-01T01:31:15.383886+0000","last_became_peered":"2024-04-01T01:31:15.383886+0000","last_unstale":"2024-04-01T01:32:19.688923+0000","last_undegraded":"2024-04-01T01:32:19.688923+0000","last_fullsized":"2024-04-01T01:32:19.688923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:43:28.414423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.596612+0000","last_change":"2024-04-01T01:31:16.381877+0000","last_active":"2024-04-01T01:32:48.596612+0000","last_peered":"2024-04-01T01:32:48.596612+0000","last_clean":"2024-04-01T01:32:48.596612+0000","last_became_active":"2024-04-01T01:31:16.381558+0000","last_became_peered":"2024-04-01T01:31:16.381558+0000","last_unstale":"2024-04-01T01:32:48.596612+0000","last_undegraded":"2024-04-01T01:32:48.596612+0000","last_fullsized":"2024-04-01T01:32:48.596612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:24.120910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.594729+0000","last_change":"2024-04-01T01:31:16.380116+0000","last_active":"2024-04-01T01:32:48.594729+0000","last_peered":"2024-04-01T01:32:48.594729+0000","last_clean":"2024-04-01T01:32:48.594729+0000","last_became_active":"2024-04-01T01:31:16.378936+0000","last_became_peered":"2024-04-01T01:31:16.378936+0000","last_unstale":"2024-04-01T01:32:48.594729+0000","last_undegraded":"2024-04-01T01:32:48.594729+0000","last_fullsized":"2024-04-01T01:32:48.594729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:18:16.211113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.595068+0000","last_change":"2024-04-01T01:31:16.384872+0000","last_active":"2024-04-01T01:32:48.595068+0000","last_peered":"2024-04-01T01:32:48.595068+0000","last_clean":"2024-04-01T01:32:48.595068+0000","last_became_active":"2024-04-01T01:31:16.384406+0000","last_became_peered":"2024-04-01T01:31:16.384406+0000","last_unstale":"2024-04-01T01:32:48.595068+0000","last_undegraded":"2024-04-01T01:32:48.595068+0000","last_fullsized":"2024-04-01T01:32:48.595068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:13:36.567357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.600963+0000","last_change":"2024-04-01T01:31:16.384561+0000","last_active":"2024-04-01T01:32:48.600963+0000","last_peered":"2024-04-01T01:32:48.600963+0000","last_clean":"2024-04-01T01:32:48.600963+0000","last_became_active":"2024-04-01T01:31:16.384221+0000","last_became_peered":"2024-04-01T01:31:16.384221+0000","last_unstale":"2024-04-01T01:32:48.600963+0000","last_undegraded":"2024-04-01T01:32:48.600963+0000","last_fullsized":"2024-04-01T01:32:48.600963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:17:56.490000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.603580+0000","last_change":"2024-04-01T01:31:16.385464+0000","last_active":"2024-04-01T01:32:48.603580+0000","last_peered":"2024-04-01T01:32:48.603580+0000","last_clean":"2024-04-01T01:32:48.603580+0000","last_became_active":"2024-04-01T01:31:16.385159+0000","last_became_peered":"2024-04-01T01:31:16.385159+0000","last_unstale":"2024-04-01T01:32:48.603580+0000","last_undegraded":"2024-04-01T01:32:48.603580+0000","last_fullsized":"2024-04-01T01:32:48.603580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:33:53.852957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.604151+0000","last_change":"2024-04-01T01:31:16.385688+0000","last_active":"2024-04-01T01:32:48.604151+0000","last_peered":"2024-04-01T01:32:48.604151+0000","last_clean":"2024-04-01T01:32:48.604151+0000","last_became_active":"2024-04-01T01:31:16.385381+0000","last_became_peered":"2024-04-01T01:31:16.385381+0000","last_unstale":"2024-04-01T01:32:48.604151+0000","last_undegraded":"2024-04-01T01:32:48.604151+0000","last_fullsized":"2024-04-01T01:32:48.604151+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:39:36.810766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.593165+0000","last_change":"2024-04-01T01:31:16.379976+0000","last_active":"2024-04-01T01:32:48.593165+0000","last_peered":"2024-04-01T01:32:48.593165+0000","last_clean":"2024-04-01T01:32:48.593165+0000","last_became_active":"2024-04-01T01:31:16.379553+0000","last_became_peered":"2024-04-01T01:31:16.379553+0000","last_unstale":"2024-04-01T01:32:48.593165+0000","last_undegraded":"2024-04-01T01:32:48.593165+0000","last_fullsized":"2024-04-01T01:32:48.593165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:48:23.555723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.579674+0000","last_change":"2024-04-01T01:31:16.383995+0000","last_active":"2024-04-01T01:32:48.579674+0000","last_peered":"2024-04-01T01:32:48.579674+0000","last_clean":"2024-04-01T01:32:48.579674+0000","last_became_active":"2024-04-01T01:31:16.383673+0000","last_became_peered":"2024-04-01T01:31:16.383673+0000","last_unstale":"2024-04-01T01:32:48.579674+0000","last_undegraded":"2024-04-01T01:32:48.579674+0000","last_fullsized":"2024-04-01T01:32:48.579674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:03:16.192438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.597623+0000","last_change":"2024-04-01T01:31:16.389969+0000","last_active":"2024-04-01T01:32:48.597623+0000","last_peered":"2024-04-01T01:32:48.597623+0000","last_clean":"2024-04-01T01:32:48.597623+0000","last_became_active":"2024-04-01T01:31:16.389640+0000","last_became_peered":"2024-04-01T01:31:16.389640+0000","last_unstale":"2024-04-01T01:32:48.597623+0000","last_undegraded":"2024-04-01T01:32:48.597623+0000","last_fullsized":"2024-04-01T01:32:48.597623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:35:15.226960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.588985+0000","last_change":"2024-04-01T01:31:16.382962+0000","last_active":"2024-04-01T01:32:48.588985+0000","last_peered":"2024-04-01T01:32:48.588985+0000","last_clean":"2024-04-01T01:32:48.588985+0000","last_became_active":"2024-04-01T01:31:16.382653+0000","last_became_peered":"2024-04-01T01:31:16.382653+0000","last_unstale":"2024-04-01T01:32:48.588985+0000","last_undegraded":"2024-04-01T01:32:48.588985+0000","last_fullsized":"2024-04-01T01:32:48.588985+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:53.054884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.602978+0000","last_change":"2024-04-01T01:31:16.382997+0000","last_active":"2024-04-01T01:32:48.602978+0000","last_peered":"2024-04-01T01:32:48.602978+0000","last_clean":"2024-04-01T01:32:48.602978+0000","last_became_active":"2024-04-01T01:31:16.380180+0000","last_became_peered":"2024-04-01T01:31:16.380180+0000","last_unstale":"2024-04-01T01:32:48.602978+0000","last_undegraded":"2024-04-01T01:32:48.602978+0000","last_fullsized":"2024-04-01T01:32:48.602978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:29:51.990395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.595157+0000","last_change":"2024-04-01T01:31:16.382948+0000","last_active":"2024-04-01T01:32:48.595157+0000","last_peered":"2024-04-01T01:32:48.595157+0000","last_clean":"2024-04-01T01:32:48.595157+0000","last_became_active":"2024-04-01T01:31:16.382134+0000","last_became_peered":"2024-04-01T01:31:16.382134+0000","last_unstale":"2024-04-01T01:32:48.595157+0000","last_undegraded":"2024-04-01T01:32:48.595157+0000","last_fullsized":"2024-04-01T01:32:48.595157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:24:08.515063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.599945+0000","last_change":"2024-04-01T01:31:16.378841+0000","last_active":"2024-04-01T01:32:48.599945+0000","last_peered":"2024-04-01T01:32:48.599945+0000","last_clean":"2024-04-01T01:32:48.599945+0000","last_became_active":"2024-04-01T01:31:16.378501+0000","last_became_peered":"2024-04-01T01:31:16.378501+0000","last_unstale":"2024-04-01T01:32:48.599945+0000","last_undegraded":"2024-04-01T01:32:48.599945+0000","last_fullsized":"2024-04-01T01:32:48.599945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:50:20.881852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-01T01:32:48.580110+0000","last_change":"2024-04-01T01:31:16.380979+0000","last_active":"2024-04-01T01:32:48.580110+0000","last_peered":"2024-04-01T01:32:48.580110+0000","last_clean":"2024-04-01T01:32:48.580110+0000","last_became_active":"2024-04-01T01:31:16.380625+0000","last_became_peered":"2024-04-01T01:31:16.380625+0000","last_unstale":"2024-04-01T01:32:48.580110+0000","last_undegraded":"2024-04-01T01:32:48.580110+0000","last_fullsized":"2024-04-01T01:32:48.580110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:09:03.246239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71091,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":776,"num_read_kb":469,"num_write":250,"num_write_kb":147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1125,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":213273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":218,"ondisk_log_size":218,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383713,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":501545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37024,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708128,"statfs":{"total":95995035648,"available":95957123072,"internally_reserved":0,"allocated":1728512,"data_stored":1152386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,2,21,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808875,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":505459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070512,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":505725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332149,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36948,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708204,"statfs":{"total":95995035648,"available":95957200896,"internally_reserved":0,"allocated":1650688,"data_stored":1098911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":496867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":561502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":498323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":561100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":496575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920835,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":502627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757311,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28632,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716520,"statfs":{"total":95995035648,"available":95965716480,"internally_reserved":0,"allocated":1654784,"data_stored":1089865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":65357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":65873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-01T01:32:56.591 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-01T01:32:56.591 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-01T01:32:56.591 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-01T01:32:56.591 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph health --format=json 2024-04-01T01:32:56.834 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:32:57.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:57 smithi189 ceph-mon[35753]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:32:57.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:57 smithi189 ceph-mon[35753]: pgmap v340: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-01T01:32:58.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:57 smithi070 ceph-mon[32179]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:32:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:57 smithi070 ceph-mon[32179]: pgmap v340: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-01T01:32:58.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:57 smithi081 ceph-mon[35142]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:32:58.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:57 smithi081 ceph-mon[35142]: pgmap v340: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-01T01:32:58.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.581+0000 7f6bc93f7700 1 -- 172.21.15.70:0/808415607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bc406b390 msgr2=0x7f6bc4106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:58.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.581+0000 7f6bc93f7700 1 --2- 172.21.15.70:0/808415607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bc406b390 0x7f6bc4106e90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6bac0099c0 tx=0x7f6bac009cd0 comp rx=0 tx=0).stop 2024-04-01T01:32:58.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.581+0000 7f6bc93f7700 1 -- 172.21.15.70:0/808415607 shutdown_connections 2024-04-01T01:32:58.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.581+0000 7f6bc93f7700 1 --2- 172.21.15.70:0/808415607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bc41075c0 0x7f6bc410f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:58.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.581+0000 7f6bc93f7700 1 --2- 172.21.15.70:0/808415607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bc406a9e0 0x7f6bc406adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:58.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.581+0000 7f6bc93f7700 1 --2- 172.21.15.70:0/808415607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bc406b390 0x7f6bc4106e90 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:58.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.581+0000 7f6bc93f7700 1 -- 172.21.15.70:0/808415607 >> 172.21.15.70:0/808415607 conn(0x7f6bc4077190 msgr2=0x7f6bc40775a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:58.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.581+0000 7f6bc93f7700 1 -- 172.21.15.70:0/808415607 shutdown_connections 2024-04-01T01:32:58.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.581+0000 7f6bc93f7700 1 -- 172.21.15.70:0/808415607 wait complete. 2024-04-01T01:32:58.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc93f7700 1 Processor -- start 2024-04-01T01:32:58.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc93f7700 1 -- start start 2024-04-01T01:32:58.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc93f7700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bc406a9e0 0x7f6bc419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:58.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc93f7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bc406b390 0x7f6bc419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:58.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc93f7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bc41075c0 0x7f6bc4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:58.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc93f7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc41123c0 con 0x7f6bc41075c0 2024-04-01T01:32:58.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc93f7700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc4112240 con 0x7f6bc406a9e0 2024-04-01T01:32:58.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc93f7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc4112540 con 0x7f6bc406b390 2024-04-01T01:32:58.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bc41075c0 0x7f6bc4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:58.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc2ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bc406a9e0 0x7f6bc419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:58.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bc41075c0 0x7f6bc4198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39356/0 (socket says 172.21.15.70:39356) 2024-04-01T01:32:58.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc37fe700 1 -- 172.21.15.70:0/4140160342 learned_addr learned my addr 172.21.15.70:0/4140160342 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:58.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc27fc700 1 --2- 172.21.15.70:0/4140160342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bc406b390 0x7f6bc419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:58.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc37fe700 1 -- 172.21.15.70:0/4140160342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bc406b390 msgr2=0x7f6bc419ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:58.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc37fe700 1 --2- 172.21.15.70:0/4140160342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bc406b390 0x7f6bc419ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:58.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc37fe700 1 -- 172.21.15.70:0/4140160342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bc406a9e0 msgr2=0x7f6bc419e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:58.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc37fe700 1 --2- 172.21.15.70:0/4140160342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bc406a9e0 0x7f6bc419e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:58.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.582+0000 7f6bc37fe700 1 -- 172.21.15.70:0/4140160342 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bc4199040 con 0x7f6bc41075c0 2024-04-01T01:32:58.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.583+0000 7f6bc37fe700 1 --2- 172.21.15.70:0/4140160342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bc41075c0 0x7f6bc4198760 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6bb400e940 tx=0x7f6bb400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:58.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.583+0000 7f6bbbfff700 1 -- 172.21.15.70:0/4140160342 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb4004a30 con 0x7f6bc41075c0 2024-04-01T01:32:58.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.583+0000 7f6bbbfff700 1 -- 172.21.15.70:0/4140160342 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6bb40054d0 con 0x7f6bc41075c0 2024-04-01T01:32:58.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.583+0000 7f6bc93f7700 1 -- 172.21.15.70:0/4140160342 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bc4199330 con 0x7f6bc41075c0 2024-04-01T01:32:58.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.583+0000 7f6bbbfff700 1 -- 172.21.15.70:0/4140160342 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb400ccf0 con 0x7f6bc41075c0 2024-04-01T01:32:58.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.583+0000 7f6bc93f7700 1 -- 172.21.15.70:0/4140160342 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bc41a5610 con 0x7f6bc41075c0 2024-04-01T01:32:58.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.584+0000 7f6bc93f7700 1 -- 172.21.15.70:0/4140160342 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bc4103970 con 0x7f6bc41075c0 2024-04-01T01:32:58.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.585+0000 7f6bbbfff700 1 -- 172.21.15.70:0/4140160342 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f6bb4005000 con 0x7f6bc41075c0 2024-04-01T01:32:58.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.585+0000 7f6bbbfff700 1 --2- 172.21.15.70:0/4140160342 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6bb00a2530 0x7f6bb00a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:58.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.585+0000 7f6bc2ffd700 1 --2- 172.21.15.70:0/4140160342 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6bb00a2530 0x7f6bb00a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:58.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.586+0000 7f6bbbfff700 1 -- 172.21.15.70:0/4140160342 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6bb40cedd0 con 0x7f6bc41075c0 2024-04-01T01:32:58.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.586+0000 7f6bc2ffd700 1 --2- 172.21.15.70:0/4140160342 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6bb00a2530 0x7f6bb00a49f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00f540 tx=0x7f6bbc00b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:58.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.589+0000 7f6bbbfff700 1 -- 172.21.15.70:0/4140160342 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6bb4053280 con 0x7f6bc41075c0 2024-04-01T01:32:58.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.829+0000 7f6bc93f7700 1 -- 172.21.15.70:0/4140160342 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f6bc4199c90 con 0x7f6bc41075c0 2024-04-01T01:32:58.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.829+0000 7f6bbbfff700 1 -- 172.21.15.70:0/4140160342 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f6bb40052f0 con 0x7f6bc41075c0 2024-04-01T01:32:58.829 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:32:58.830 INFO:teuthology.orchestra.run.smithi070.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-01T01:32:58.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.831+0000 7f6bc93f7700 1 -- 172.21.15.70:0/4140160342 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6bb00a2530 msgr2=0x7f6bb00a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:58.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.831+0000 7f6bc93f7700 1 --2- 172.21.15.70:0/4140160342 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6bb00a2530 0x7f6bb00a49f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc00f540 tx=0x7f6bbc00b040 comp rx=0 tx=0).stop 2024-04-01T01:32:58.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.831+0000 7f6bc93f7700 1 -- 172.21.15.70:0/4140160342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bc41075c0 msgr2=0x7f6bc4198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:58.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.831+0000 7f6bc93f7700 1 --2- 172.21.15.70:0/4140160342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bc41075c0 0x7f6bc4198760 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6bb400e940 tx=0x7f6bb400ec50 comp rx=0 tx=0).stop 2024-04-01T01:32:58.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.831+0000 7f6bc93f7700 1 -- 172.21.15.70:0/4140160342 shutdown_connections 2024-04-01T01:32:58.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.831+0000 7f6bc93f7700 1 --2- 172.21.15.70:0/4140160342 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6bb00a2530 0x7f6bb00a49f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:58.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.831+0000 7f6bc93f7700 1 --2- 172.21.15.70:0/4140160342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6bc41075c0 0x7f6bc4198760 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:58.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.831+0000 7f6bc93f7700 1 --2- 172.21.15.70:0/4140160342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bc406a9e0 0x7f6bc419e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:58.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.831+0000 7f6bc93f7700 1 --2- 172.21.15.70:0/4140160342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bc406b390 0x7f6bc419ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:58.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.831+0000 7f6bc93f7700 1 -- 172.21.15.70:0/4140160342 >> 172.21.15.70:0/4140160342 conn(0x7f6bc4077190 msgr2=0x7f6bc40ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:58.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.832+0000 7f6bc93f7700 1 -- 172.21.15.70:0/4140160342 shutdown_connections 2024-04-01T01:32:58.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:58.832+0000 7f6bc93f7700 1 -- 172.21.15.70:0/4140160342 wait complete. 2024-04-01T01:32:59.648 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-01T01:32:59.649 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:32:59.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.758+0000 7f54c46a4640 1 -- 172.21.15.70:0/2983747113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54bc101930 msgr2=0x7f54bc108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:59.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.758+0000 7f54c46a4640 1 --2- 172.21.15.70:0/2983747113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54bc101930 0x7f54bc108250 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f54b4009ca0 tx=0x7f54b402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:32:59.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.758+0000 7f54c46a4640 1 -- 172.21.15.70:0/2983747113 shutdown_connections 2024-04-01T01:32:59.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.758+0000 7f54c46a4640 1 --2- 172.21.15.70:0/2983747113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54bc101930 0x7f54bc108250 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:59.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.758+0000 7f54c46a4640 1 --2- 172.21.15.70:0/2983747113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54bc100f70 0x7f54bc1013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:59.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.758+0000 7f54c46a4640 1 --2- 172.21.15.70:0/2983747113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54bc0ffe20 0x7f54bc100240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:59.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.758+0000 7f54c46a4640 1 -- 172.21.15.70:0/2983747113 >> 172.21.15.70:0/2983747113 conn(0x7f54bc0fb390 msgr2=0x7f54bc0fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:59.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.758+0000 7f54c46a4640 1 -- 172.21.15.70:0/2983747113 shutdown_connections 2024-04-01T01:32:59.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.758+0000 7f54c46a4640 1 -- 172.21.15.70:0/2983747113 wait complete. 2024-04-01T01:32:59.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c46a4640 1 Processor -- start 2024-04-01T01:32:59.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c46a4640 1 -- start start 2024-04-01T01:32:59.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c46a4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54bc0ffe20 0x7f54bc1097b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:59.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c46a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54bc100f70 0x7f54bc19c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:59.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c46a4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54bc101930 0x7f54bc19ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:59.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c46a4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc10c1e0 con 0x7f54bc100f70 2024-04-01T01:32:59.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c46a4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc10c060 con 0x7f54bc101930 2024-04-01T01:32:59.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c46a4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc10c360 con 0x7f54bc0ffe20 2024-04-01T01:32:59.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c1c3f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54bc100f70 0x7f54bc19c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:59.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c1c3f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54bc100f70 0x7f54bc19c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39370/0 (socket says 172.21.15.70:39370) 2024-04-01T01:32:59.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c1c3f640 1 -- 172.21.15.70:0/3386968397 learned_addr learned my addr 172.21.15.70:0/3386968397 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:32:59.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c2440640 1 --2- 172.21.15.70:0/3386968397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54bc0ffe20 0x7f54bc1097b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:59.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c2c41640 1 --2- 172.21.15.70:0/3386968397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54bc101930 0x7f54bc19ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:59.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c1c3f640 1 -- 172.21.15.70:0/3386968397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54bc0ffe20 msgr2=0x7f54bc1097b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:59.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c1c3f640 1 --2- 172.21.15.70:0/3386968397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54bc0ffe20 0x7f54bc1097b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:59.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c1c3f640 1 -- 172.21.15.70:0/3386968397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54bc101930 msgr2=0x7f54bc19ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:59.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c1c3f640 1 --2- 172.21.15.70:0/3386968397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54bc101930 0x7f54bc19ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:59.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.759+0000 7f54c1c3f640 1 -- 172.21.15.70:0/3386968397 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54bc19f5d0 con 0x7f54bc100f70 2024-04-01T01:32:59.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.760+0000 7f54c1c3f640 1 --2- 172.21.15.70:0/3386968397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54bc100f70 0x7f54bc19c6a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f54b800e9c0 tx=0x7f54b800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:59.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.760+0000 7f54b37fe640 1 -- 172.21.15.70:0/3386968397 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b800cd80 con 0x7f54bc100f70 2024-04-01T01:32:59.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.760+0000 7f54b37fe640 1 -- 172.21.15.70:0/3386968397 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54b80035f0 con 0x7f54bc100f70 2024-04-01T01:32:59.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.760+0000 7f54c46a4640 1 -- 172.21.15.70:0/3386968397 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54bc19f860 con 0x7f54bc100f70 2024-04-01T01:32:59.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.760+0000 7f54b37fe640 1 -- 172.21.15.70:0/3386968397 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b8010690 con 0x7f54bc100f70 2024-04-01T01:32:59.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.760+0000 7f54c46a4640 1 -- 172.21.15.70:0/3386968397 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54bc19fdb0 con 0x7f54bc100f70 2024-04-01T01:32:59.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.761+0000 7f54c46a4640 1 -- 172.21.15.70:0/3386968397 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54bc107130 con 0x7f54bc100f70 2024-04-01T01:32:59.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.762+0000 7f54b37fe640 1 -- 172.21.15.70:0/3386968397 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f54b80037a0 con 0x7f54bc100f70 2024-04-01T01:32:59.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.762+0000 7f54b37fe640 1 --2- 172.21.15.70:0/3386968397 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54a80a24a0 0x7f54a80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:32:59.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.762+0000 7f54c2440640 1 --2- 172.21.15.70:0/3386968397 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54a80a24a0 0x7f54a80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:32:59.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.762+0000 7f54b37fe640 1 -- 172.21.15.70:0/3386968397 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f54b80cee00 con 0x7f54bc100f70 2024-04-01T01:32:59.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.763+0000 7f54c2440640 1 --2- 172.21.15.70:0/3386968397 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54a80a24a0 0x7f54a80a4960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f54ac0095f0 tx=0x7f54ac005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:32:59.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.767+0000 7f54b37fe640 1 -- 172.21.15.70:0/3386968397 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f54b809bc60 con 0x7f54bc100f70 2024-04-01T01:32:59.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.941+0000 7f54c46a4640 1 -- 172.21.15.70:0/3386968397 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f54bc062350 con 0x7f54bc100f70 2024-04-01T01:32:59.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.943+0000 7f54b37fe640 1 -- 172.21.15.70:0/3386968397 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24595 (secure 0 0 0) 0x7f54b809b560 con 0x7f54bc100f70 2024-04-01T01:32:59.943 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:32:59.943 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":91,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:32:19.680117+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:32:59.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.945+0000 7f54c46a4640 1 -- 172.21.15.70:0/3386968397 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54a80a24a0 msgr2=0x7f54a80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:59.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.945+0000 7f54c46a4640 1 --2- 172.21.15.70:0/3386968397 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54a80a24a0 0x7f54a80a4960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f54ac0095f0 tx=0x7f54ac005f90 comp rx=0 tx=0).stop 2024-04-01T01:32:59.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.945+0000 7f54c46a4640 1 -- 172.21.15.70:0/3386968397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54bc100f70 msgr2=0x7f54bc19c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:32:59.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.945+0000 7f54c46a4640 1 --2- 172.21.15.70:0/3386968397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54bc100f70 0x7f54bc19c6a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f54b800e9c0 tx=0x7f54b800ee80 comp rx=0 tx=0).stop 2024-04-01T01:32:59.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.945+0000 7f54c46a4640 1 -- 172.21.15.70:0/3386968397 shutdown_connections 2024-04-01T01:32:59.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.945+0000 7f54c46a4640 1 --2- 172.21.15.70:0/3386968397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54bc101930 0x7f54bc19ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:59.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.945+0000 7f54c46a4640 1 --2- 172.21.15.70:0/3386968397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54bc100f70 0x7f54bc19c6a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:59.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.945+0000 7f54c46a4640 1 --2- 172.21.15.70:0/3386968397 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54a80a24a0 0x7f54a80a4960 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:59.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.945+0000 7f54c46a4640 1 --2- 172.21.15.70:0/3386968397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54bc0ffe20 0x7f54bc1097b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:32:59.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.945+0000 7f54c46a4640 1 -- 172.21.15.70:0/3386968397 >> 172.21.15.70:0/3386968397 conn(0x7f54bc0fb390 msgr2=0x7f54bc104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:32:59.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.945+0000 7f54c46a4640 1 -- 172.21.15.70:0/3386968397 shutdown_connections 2024-04-01T01:32:59.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:32:59.945+0000 7f54c46a4640 1 -- 172.21.15.70:0/3386968397 wait complete. 2024-04-01T01:32:59.955 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:59 smithi070 ceph-mon[32179]: pgmap v341: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-01T01:32:59.955 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:32:59 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/4140160342' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-01T01:32:59.957 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-01T01:26:42.809779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-01T01:31:13.612369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-01T01:31:14.910162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-01T01:31:18.783179+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-01T01:32:59.957 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-01T01:32:59.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:59 smithi189 ceph-mon[35753]: pgmap v341: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-01T01:32:59.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:32:59 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/4140160342' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-01T01:33:00.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.061+0000 7fe1bfeaf640 1 -- 172.21.15.70:0/642894277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b81017e0 msgr2=0x7fe1b8101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.061+0000 7fe1bfeaf640 1 --2- 172.21.15.70:0/642894277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b81017e0 0x7fe1b8101c00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4009ca0 tx=0x7fe1b402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.061+0000 7fe1bfeaf640 1 -- 172.21.15.70:0/642894277 shutdown_connections 2024-04-01T01:33:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.061+0000 7fe1bfeaf640 1 --2- 172.21.15.70:0/642894277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b8103390 0x7fe1b8109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.061+0000 7fe1bfeaf640 1 --2- 172.21.15.70:0/642894277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b81029d0 0x7fe1b8102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.061+0000 7fe1bfeaf640 1 --2- 172.21.15.70:0/642894277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b81017e0 0x7fe1b8101c00 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.061+0000 7fe1bfeaf640 1 -- 172.21.15.70:0/642894277 >> 172.21.15.70:0/642894277 conn(0x7fe1b80fcd70 msgr2=0x7fe1b80ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.061+0000 7fe1bfeaf640 1 -- 172.21.15.70:0/642894277 shutdown_connections 2024-04-01T01:33:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.061+0000 7fe1bfeaf640 1 -- 172.21.15.70:0/642894277 wait complete. 2024-04-01T01:33:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.062+0000 7fe1bfeaf640 1 Processor -- start 2024-04-01T01:33:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.062+0000 7fe1bfeaf640 1 -- start start 2024-04-01T01:33:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.062+0000 7fe1bfeaf640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b81017e0 0x7fe1b8197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.062+0000 7fe1bfeaf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b81029d0 0x7fe1b8197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.062+0000 7fe1bfeaf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b8103390 0x7fe1b819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:00.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.062+0000 7fe1bfeaf640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b810d270 con 0x7fe1b81029d0 2024-04-01T01:33:00.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.062+0000 7fe1bfeaf640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b810d0f0 con 0x7fe1b8103390 2024-04-01T01:33:00.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.062+0000 7fe1bfeaf640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b810d3f0 con 0x7fe1b81017e0 2024-04-01T01:33:00.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.062+0000 7fe1bd44a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b81029d0 0x7fe1b8197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:00.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.062+0000 7fe1bd44a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b81029d0 0x7fe1b8197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39388/0 (socket says 172.21.15.70:39388) 2024-04-01T01:33:00.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.062+0000 7fe1bd44a640 1 -- 172.21.15.70:0/3718267935 learned_addr learned my addr 172.21.15.70:0/3718267935 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:00.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.062+0000 7fe1bdc4b640 1 --2- 172.21.15.70:0/3718267935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b81017e0 0x7fe1b8197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:00.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.062+0000 7fe1be44c640 1 --2- 172.21.15.70:0/3718267935 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b8103390 0x7fe1b819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:00.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.062+0000 7fe1bd44a640 1 -- 172.21.15.70:0/3718267935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b81017e0 msgr2=0x7fe1b8197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.062+0000 7fe1bd44a640 1 --2- 172.21.15.70:0/3718267935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b81017e0 0x7fe1b8197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.063+0000 7fe1bd44a640 1 -- 172.21.15.70:0/3718267935 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b8103390 msgr2=0x7fe1b819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.063+0000 7fe1bd44a640 1 --2- 172.21.15.70:0/3718267935 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b8103390 0x7fe1b819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.063+0000 7fe1bd44a640 1 -- 172.21.15.70:0/3718267935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b819f590 con 0x7fe1b81029d0 2024-04-01T01:33:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.063+0000 7fe1be44c640 1 --2- 172.21.15.70:0/3718267935 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b8103390 0x7fe1b819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.063+0000 7fe1bd44a640 1 --2- 172.21.15.70:0/3718267935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b81029d0 0x7fe1b8197eb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800e9f0 tx=0x7fe1a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.063+0000 7fe1aeffd640 1 -- 172.21.15.70:0/3718267935 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a800cd80 con 0x7fe1b81029d0 2024-04-01T01:33:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.063+0000 7fe1aeffd640 1 -- 172.21.15.70:0/3718267935 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1a80108e0 con 0x7fe1b81029d0 2024-04-01T01:33:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.063+0000 7fe1bfeaf640 1 -- 172.21.15.70:0/3718267935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b819f880 con 0x7fe1b81029d0 2024-04-01T01:33:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.063+0000 7fe1aeffd640 1 -- 172.21.15.70:0/3718267935 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8018920 con 0x7fe1b81029d0 2024-04-01T01:33:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.063+0000 7fe1bfeaf640 1 -- 172.21.15.70:0/3718267935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1b819fdd0 con 0x7fe1b81029d0 2024-04-01T01:33:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.064+0000 7fe1bfeaf640 1 -- 172.21.15.70:0/3718267935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1b8108bd0 con 0x7fe1b81029d0 2024-04-01T01:33:00.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.065+0000 7fe1aeffd640 1 -- 172.21.15.70:0/3718267935 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fe1a8010a90 con 0x7fe1b81029d0 2024-04-01T01:33:00.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.066+0000 7fe1aeffd640 1 --2- 172.21.15.70:0/3718267935 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe1a40a24a0 0x7fe1a40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:00.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.066+0000 7fe1bdc4b640 1 --2- 172.21.15.70:0/3718267935 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe1a40a24a0 0x7fe1a40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:00.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.066+0000 7fe1aeffd640 1 -- 172.21.15.70:0/3718267935 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe1a80ced50 con 0x7fe1b81029d0 2024-04-01T01:33:00.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.066+0000 7fe1bdc4b640 1 --2- 172.21.15.70:0/3718267935 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe1a40a24a0 0x7fe1a40a4960 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe1b402c630 tx=0x7fe1b4032340 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:00.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.070+0000 7fe1aeffd640 1 -- 172.21.15.70:0/3718267935 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe1a80d2030 con 0x7fe1b81029d0 2024-04-01T01:33:00.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:59 smithi081 ceph-mon[35142]: pgmap v341: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-01T01:33:00.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:32:59 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/4140160342' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-01T01:33:00.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.244+0000 7fe1bfeaf640 1 -- 172.21.15.70:0/3718267935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe1b8062350 con 0x7fe1b81029d0 2024-04-01T01:33:00.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.244+0000 7fe1aeffd640 1 -- 172.21.15.70:0/3718267935 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7fe1a809b560 con 0x7fe1b81029d0 2024-04-01T01:33:00.245 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-01T01:33:00.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.246+0000 7fe1bfeaf640 1 -- 172.21.15.70:0/3718267935 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe1a40a24a0 msgr2=0x7fe1a40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.246+0000 7fe1bfeaf640 1 --2- 172.21.15.70:0/3718267935 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe1a40a24a0 0x7fe1a40a4960 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe1b402c630 tx=0x7fe1b4032340 comp rx=0 tx=0).stop 2024-04-01T01:33:00.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.246+0000 7fe1bfeaf640 1 -- 172.21.15.70:0/3718267935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b81029d0 msgr2=0x7fe1b8197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.246+0000 7fe1bfeaf640 1 --2- 172.21.15.70:0/3718267935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b81029d0 0x7fe1b8197eb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800e9f0 tx=0x7fe1a800eeb0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.246+0000 7fe1bfeaf640 1 -- 172.21.15.70:0/3718267935 shutdown_connections 2024-04-01T01:33:00.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.246+0000 7fe1bfeaf640 1 --2- 172.21.15.70:0/3718267935 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b8103390 0x7fe1b819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.246+0000 7fe1bfeaf640 1 --2- 172.21.15.70:0/3718267935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b81029d0 0x7fe1b8197eb0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.246+0000 7fe1bfeaf640 1 --2- 172.21.15.70:0/3718267935 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe1a40a24a0 0x7fe1a40a4960 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.246+0000 7fe1bfeaf640 1 --2- 172.21.15.70:0/3718267935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1b81017e0 0x7fe1b8197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.246+0000 7fe1bfeaf640 1 -- 172.21.15.70:0/3718267935 >> 172.21.15.70:0/3718267935 conn(0x7fe1b80fcd70 msgr2=0x7fe1b8105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:00.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.246+0000 7fe1bfeaf640 1 -- 172.21.15.70:0/3718267935 shutdown_connections 2024-04-01T01:33:00.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.246+0000 7fe1bfeaf640 1 -- 172.21.15.70:0/3718267935 wait complete. 2024-04-01T01:33:00.258 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-01T01:33:00.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.363+0000 7f493f58c640 1 -- 172.21.15.70:0/1561432319 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4938107450 msgr2=0x7f49381098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.363+0000 7f493f58c640 1 --2- 172.21.15.70:0/1561432319 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4938107450 0x7f49381098a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4930009ca0 tx=0x7f493002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.363+0000 7f493f58c640 1 -- 172.21.15.70:0/1561432319 shutdown_connections 2024-04-01T01:33:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.363+0000 7f493f58c640 1 --2- 172.21.15.70:0/1561432319 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4938107450 0x7f49381098a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.363+0000 7f493f58c640 1 --2- 172.21.15.70:0/1561432319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4938104b20 0x7f4938106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.363+0000 7f493f58c640 1 --2- 172.21.15.70:0/1561432319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49381021f0 0x7f49381045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.363+0000 7f493f58c640 1 -- 172.21.15.70:0/1561432319 >> 172.21.15.70:0/1561432319 conn(0x7f49380fbc00 msgr2=0x7f49380fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.364+0000 7f493f58c640 1 -- 172.21.15.70:0/1561432319 shutdown_connections 2024-04-01T01:33:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.364+0000 7f493f58c640 1 -- 172.21.15.70:0/1561432319 wait complete. 2024-04-01T01:33:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.364+0000 7f493f58c640 1 Processor -- start 2024-04-01T01:33:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.364+0000 7f493f58c640 1 -- start start 2024-04-01T01:33:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.364+0000 7f493f58c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49381021f0 0x7f4938101b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.364+0000 7f493f58c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4938104b20 0x7f49381001a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.364+0000 7f493f58c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4938107450 0x7f49381006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.364+0000 7f493f58c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f493810d2a0 con 0x7f4938107450 2024-04-01T01:33:00.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.364+0000 7f493f58c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f493810d120 con 0x7f49381021f0 2024-04-01T01:33:00.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.364+0000 7f493f58c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f493810d420 con 0x7f4938104b20 2024-04-01T01:33:00.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.364+0000 7f493db29640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4938107450 0x7f49381006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:00.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.364+0000 7f493d328640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49381021f0 0x7f4938101b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:00.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.364+0000 7f493db29640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4938107450 0x7f49381006e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39402/0 (socket says 172.21.15.70:39402) 2024-04-01T01:33:00.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.364+0000 7f493db29640 1 -- 172.21.15.70:0/2613309226 learned_addr learned my addr 172.21.15.70:0/2613309226 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:00.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.364+0000 7f493d328640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49381021f0 0x7f4938101b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:52566/0 (socket says 172.21.15.70:52566) 2024-04-01T01:33:00.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.364+0000 7f493cb27640 1 --2- 172.21.15.70:0/2613309226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4938104b20 0x7f49381001a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:00.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.365+0000 7f493db29640 1 -- 172.21.15.70:0/2613309226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4938104b20 msgr2=0x7f49381001a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.365+0000 7f493db29640 1 --2- 172.21.15.70:0/2613309226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4938104b20 0x7f49381001a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.365+0000 7f493db29640 1 -- 172.21.15.70:0/2613309226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49381021f0 msgr2=0x7f4938101b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.365+0000 7f493db29640 1 --2- 172.21.15.70:0/2613309226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49381021f0 0x7f4938101b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.365+0000 7f493db29640 1 -- 172.21.15.70:0/2613309226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4938100f70 con 0x7f4938107450 2024-04-01T01:33:00.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.365+0000 7f493d328640 1 --2- 172.21.15.70:0/2613309226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49381021f0 0x7f4938101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:00.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.365+0000 7f493db29640 1 --2- 172.21.15.70:0/2613309226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4938107450 0x7f49381006e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f493002c700 tx=0x7f49300328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:00.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.365+0000 7f492e7fc640 1 -- 172.21.15.70:0/2613309226 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4930041020 con 0x7f4938107450 2024-04-01T01:33:00.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.365+0000 7f493f58c640 1 -- 172.21.15.70:0/2613309226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49381a1a30 con 0x7f4938107450 2024-04-01T01:33:00.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.365+0000 7f492e7fc640 1 -- 172.21.15.70:0/2613309226 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4930003e20 con 0x7f4938107450 2024-04-01T01:33:00.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.365+0000 7f492e7fc640 1 -- 172.21.15.70:0/2613309226 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4930039040 con 0x7f4938107450 2024-04-01T01:33:00.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.365+0000 7f493f58c640 1 -- 172.21.15.70:0/2613309226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49381a1fd0 con 0x7f4938107450 2024-04-01T01:33:00.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.366+0000 7f493f58c640 1 -- 172.21.15.70:0/2613309226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4938068990 con 0x7f4938107450 2024-04-01T01:33:00.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.367+0000 7f492e7fc640 1 -- 172.21.15.70:0/2613309226 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f4930044020 con 0x7f4938107450 2024-04-01T01:33:00.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.368+0000 7f492e7fc640 1 --2- 172.21.15.70:0/2613309226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f49240a24a0 0x7f49240a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:00.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.368+0000 7f493d328640 1 --2- 172.21.15.70:0/2613309226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f49240a24a0 0x7f49240a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:00.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.368+0000 7f492e7fc640 1 -- 172.21.15.70:0/2613309226 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f49300f0ff0 con 0x7f4938107450 2024-04-01T01:33:00.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.368+0000 7f493d328640 1 --2- 172.21.15.70:0/2613309226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f49240a24a0 0x7f49240a4960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4934009fc0 tx=0x7f4934005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:00.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.371+0000 7f492e7fc640 1 -- 172.21.15.70:0/2613309226 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f493004b020 con 0x7f4938107450 2024-04-01T01:33:00.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.548+0000 7f493f58c640 1 -- 172.21.15.70:0/2613309226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4938062350 con 0x7f4938107450 2024-04-01T01:33:00.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.548+0000 7f492e7fc640 1 -- 172.21.15.70:0/2613309226 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f49300bde50 con 0x7f4938107450 2024-04-01T01:33:00.548 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:33:00.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.550+0000 7f493f58c640 1 -- 172.21.15.70:0/2613309226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f49240a24a0 msgr2=0x7f49240a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.550+0000 7f493f58c640 1 --2- 172.21.15.70:0/2613309226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f49240a24a0 0x7f49240a4960 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4934009fc0 tx=0x7f4934005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:00.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.550+0000 7f493f58c640 1 -- 172.21.15.70:0/2613309226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4938107450 msgr2=0x7f49381006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.550+0000 7f493f58c640 1 --2- 172.21.15.70:0/2613309226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4938107450 0x7f49381006e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f493002c700 tx=0x7f49300328a0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.550+0000 7f493f58c640 1 -- 172.21.15.70:0/2613309226 shutdown_connections 2024-04-01T01:33:00.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.550+0000 7f493f58c640 1 --2- 172.21.15.70:0/2613309226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4938107450 0x7f49381006e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.550+0000 7f493f58c640 1 --2- 172.21.15.70:0/2613309226 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f49240a24a0 0x7f49240a4960 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.550+0000 7f493f58c640 1 --2- 172.21.15.70:0/2613309226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4938104b20 0x7f49381001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.550+0000 7f493f58c640 1 --2- 172.21.15.70:0/2613309226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49381021f0 0x7f4938101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.550+0000 7f493f58c640 1 -- 172.21.15.70:0/2613309226 >> 172.21.15.70:0/2613309226 conn(0x7f49380fbc00 msgr2=0x7f4938102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:00.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.550+0000 7f493f58c640 1 -- 172.21.15.70:0/2613309226 shutdown_connections 2024-04-01T01:33:00.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.550+0000 7f493f58c640 1 -- 172.21.15.70:0/2613309226 wait complete. 2024-04-01T01:33:00.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-01T01:33:00.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.668+0000 7fbf8c8c0640 1 -- 172.21.15.70:0/1540574301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf881021e0 msgr2=0x7fbf881045d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.668+0000 7fbf8c8c0640 1 --2- 172.21.15.70:0/1540574301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf881021e0 0x7fbf881045d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbf78009900 tx=0x7fbf7802bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:00.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.668+0000 7fbf8c8c0640 1 -- 172.21.15.70:0/1540574301 shutdown_connections 2024-04-01T01:33:00.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.668+0000 7fbf8c8c0640 1 --2- 172.21.15.70:0/1540574301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf88107440 0x7fbf88109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.668+0000 7fbf8c8c0640 1 --2- 172.21.15.70:0/1540574301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf88104b10 0x7fbf88106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.668+0000 7fbf8c8c0640 1 --2- 172.21.15.70:0/1540574301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf881021e0 0x7fbf881045d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.668+0000 7fbf8c8c0640 1 -- 172.21.15.70:0/1540574301 >> 172.21.15.70:0/1540574301 conn(0x7fbf880fbbd0 msgr2=0x7fbf880fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:00.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.668+0000 7fbf8c8c0640 1 -- 172.21.15.70:0/1540574301 shutdown_connections 2024-04-01T01:33:00.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.668+0000 7fbf8c8c0640 1 -- 172.21.15.70:0/1540574301 wait complete. 2024-04-01T01:33:00.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.668+0000 7fbf8c8c0640 1 Processor -- start 2024-04-01T01:33:00.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf8c8c0640 1 -- start start 2024-04-01T01:33:00.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf8c8c0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf881021e0 0x7fbf881979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:00.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf8c8c0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf88104b10 0x7fbf88197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:00.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf8c8c0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf88107440 0x7fbf8819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:00.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf8c8c0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8810d310 con 0x7fbf88104b10 2024-04-01T01:33:00.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf8c8c0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8810d190 con 0x7fbf881021e0 2024-04-01T01:33:00.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf8c8c0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8810d490 con 0x7fbf88107440 2024-04-01T01:33:00.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf85d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf88104b10 0x7fbf88197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:00.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf8659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf881021e0 0x7fbf881979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:00.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf8659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf881021e0 0x7fbf881979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:52582/0 (socket says 172.21.15.70:52582) 2024-04-01T01:33:00.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf8659c640 1 -- 172.21.15.70:0/3208380498 learned_addr learned my addr 172.21.15.70:0/3208380498 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:00.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf86d9d640 1 --2- 172.21.15.70:0/3208380498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf88107440 0x7fbf8819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:00.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf85d9b640 1 -- 172.21.15.70:0/3208380498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf88107440 msgr2=0x7fbf8819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf85d9b640 1 --2- 172.21.15.70:0/3208380498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf88107440 0x7fbf8819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf85d9b640 1 -- 172.21.15.70:0/3208380498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf881021e0 msgr2=0x7fbf881979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf85d9b640 1 --2- 172.21.15.70:0/3208380498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf881021e0 0x7fbf881979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf85d9b640 1 -- 172.21.15.70:0/3208380498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8819f600 con 0x7fbf88104b10 2024-04-01T01:33:00.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf8659c640 1 --2- 172.21.15.70:0/3208380498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf881021e0 0x7fbf881979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:00.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf85d9b640 1 --2- 172.21.15.70:0/3208380498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf88104b10 0x7fbf88197f20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00e9c0 tx=0x7fbf7c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:00.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.669+0000 7fbf777fe640 1 -- 172.21.15.70:0/3208380498 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c00ccc0 con 0x7fbf88104b10 2024-04-01T01:33:00.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.670+0000 7fbf8c8c0640 1 -- 172.21.15.70:0/3208380498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf8819f8f0 con 0x7fbf88104b10 2024-04-01T01:33:00.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.670+0000 7fbf777fe640 1 -- 172.21.15.70:0/3208380498 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf7c005020 con 0x7fbf88104b10 2024-04-01T01:33:00.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.670+0000 7fbf777fe640 1 -- 172.21.15.70:0/3208380498 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c004a00 con 0x7fbf88104b10 2024-04-01T01:33:00.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.670+0000 7fbf8c8c0640 1 -- 172.21.15.70:0/3208380498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf8819fdf0 con 0x7fbf88104b10 2024-04-01T01:33:00.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.670+0000 7fbf8c8c0640 1 -- 172.21.15.70:0/3208380498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf880fd6c0 con 0x7fbf88104b10 2024-04-01T01:33:00.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.671+0000 7fbf777fe640 1 -- 172.21.15.70:0/3208380498 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fbf7c0051d0 con 0x7fbf88104b10 2024-04-01T01:33:00.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.672+0000 7fbf777fe640 1 --2- 172.21.15.70:0/3208380498 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbf700a24a0 0x7fbf700a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:00.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.672+0000 7fbf8659c640 1 --2- 172.21.15.70:0/3208380498 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbf700a24a0 0x7fbf700a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:00.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.672+0000 7fbf777fe640 1 -- 172.21.15.70:0/3208380498 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbf7c0cea40 con 0x7fbf88104b10 2024-04-01T01:33:00.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.673+0000 7fbf8659c640 1 --2- 172.21.15.70:0/3208380498 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbf700a24a0 0x7fbf700a4960 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbf780096c0 tx=0x7fbf7802c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:00.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.677+0000 7fbf777fe640 1 -- 172.21.15.70:0/3208380498 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbf7c09b8a0 con 0x7fbf88104b10 2024-04-01T01:33:00.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.854+0000 7fbf8c8c0640 1 -- 172.21.15.70:0/3208380498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbf88062350 con 0x7fbf88104b10 2024-04-01T01:33:00.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.855+0000 7fbf777fe640 1 -- 172.21.15.70:0/3208380498 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fbf7c09b1a0 con 0x7fbf88104b10 2024-04-01T01:33:00.855 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-01T01:33:00.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.856+0000 7fbf8c8c0640 1 -- 172.21.15.70:0/3208380498 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbf700a24a0 msgr2=0x7fbf700a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.856+0000 7fbf8c8c0640 1 --2- 172.21.15.70:0/3208380498 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbf700a24a0 0x7fbf700a4960 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbf780096c0 tx=0x7fbf7802c7e0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.856+0000 7fbf8c8c0640 1 -- 172.21.15.70:0/3208380498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf88104b10 msgr2=0x7fbf88197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.856+0000 7fbf8c8c0640 1 --2- 172.21.15.70:0/3208380498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf88104b10 0x7fbf88197f20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00e9c0 tx=0x7fbf7c00ee80 comp rx=0 tx=0).stop 2024-04-01T01:33:00.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.857+0000 7fbf8c8c0640 1 -- 172.21.15.70:0/3208380498 shutdown_connections 2024-04-01T01:33:00.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.857+0000 7fbf8c8c0640 1 --2- 172.21.15.70:0/3208380498 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbf700a24a0 0x7fbf700a4960 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.857+0000 7fbf8c8c0640 1 --2- 172.21.15.70:0/3208380498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf88107440 0x7fbf8819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.857+0000 7fbf8c8c0640 1 --2- 172.21.15.70:0/3208380498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf88104b10 0x7fbf88197f20 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.857+0000 7fbf8c8c0640 1 --2- 172.21.15.70:0/3208380498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf881021e0 0x7fbf881979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.857+0000 7fbf8c8c0640 1 -- 172.21.15.70:0/3208380498 >> 172.21.15.70:0/3208380498 conn(0x7fbf880fbbd0 msgr2=0x7fbf88102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:00.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.857+0000 7fbf8c8c0640 1 -- 172.21.15.70:0/3208380498 shutdown_connections 2024-04-01T01:33:00.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.857+0000 7fbf8c8c0640 1 -- 172.21.15.70:0/3208380498 wait complete. 2024-04-01T01:33:00.867 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:00 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3386968397' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:00.867 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:00 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3718267935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:00.867 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:00 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2613309226' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:00.868 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-01T01:33:00.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.973+0000 7fb77b8d4640 1 -- 172.21.15.70:0/3685976005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7740ffa50 msgr2=0x7fb7740ffe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.973+0000 7fb77b8d4640 1 --2- 172.21.15.70:0/3685976005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7740ffa50 0x7fb7740ffe70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb770009930 tx=0x7fb77002bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:00.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.973+0000 7fb77b8d4640 1 -- 172.21.15.70:0/3685976005 shutdown_connections 2024-04-01T01:33:00.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.973+0000 7fb77b8d4640 1 --2- 172.21.15.70:0/3685976005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb774103480 0x7fb774109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.973+0000 7fb77b8d4640 1 --2- 172.21.15.70:0/3685976005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb774102ac0 0x7fb774102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.973+0000 7fb77b8d4640 1 --2- 172.21.15.70:0/3685976005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7740ffa50 0x7fb7740ffe70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.973+0000 7fb77b8d4640 1 -- 172.21.15.70:0/3685976005 >> 172.21.15.70:0/3685976005 conn(0x7fb7740fb380 msgr2=0x7fb7740fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:00.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.973+0000 7fb77b8d4640 1 -- 172.21.15.70:0/3685976005 shutdown_connections 2024-04-01T01:33:00.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.973+0000 7fb77b8d4640 1 -- 172.21.15.70:0/3685976005 wait complete. 2024-04-01T01:33:00.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.973+0000 7fb77b8d4640 1 Processor -- start 2024-04-01T01:33:00.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.973+0000 7fb77b8d4640 1 -- start start 2024-04-01T01:33:00.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb77b8d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7740ffa50 0x7fb774197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:00.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb77b8d4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb774102ac0 0x7fb774197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:00.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb77b8d4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb774103480 0x7fb77419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:00.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb77b8d4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb77410d2b0 con 0x7fb774103480 2024-04-01T01:33:00.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb77b8d4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb77410d130 con 0x7fb7740ffa50 2024-04-01T01:33:00.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb77b8d4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb77410d430 con 0x7fb774102ac0 2024-04-01T01:33:00.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb779e71640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb774103480 0x7fb77419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:00.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb779e71640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb774103480 0x7fb77419efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39438/0 (socket says 172.21.15.70:39438) 2024-04-01T01:33:00.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb779e71640 1 -- 172.21.15.70:0/1753085032 learned_addr learned my addr 172.21.15.70:0/1753085032 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:00.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb779670640 1 --2- 172.21.15.70:0/1753085032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7740ffa50 0x7fb774197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:00.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb778e6f640 1 --2- 172.21.15.70:0/1753085032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb774102ac0 0x7fb774197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:00.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb779e71640 1 -- 172.21.15.70:0/1753085032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb774102ac0 msgr2=0x7fb774197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb779e71640 1 --2- 172.21.15.70:0/1753085032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb774102ac0 0x7fb774197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb779e71640 1 -- 172.21.15.70:0/1753085032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7740ffa50 msgr2=0x7fb774197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:00.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb779e71640 1 --2- 172.21.15.70:0/1753085032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7740ffa50 0x7fb774197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:00.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb779e71640 1 -- 172.21.15.70:0/1753085032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77419f630 con 0x7fb774103480 2024-04-01T01:33:00.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb779e71640 1 --2- 172.21.15.70:0/1753085032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb774103480 0x7fb77419efc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb76c00efd0 tx=0x7fb76c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:00.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb76a7fc640 1 -- 172.21.15.70:0/1753085032 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c003b70 con 0x7fb774103480 2024-04-01T01:33:00.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb76a7fc640 1 -- 172.21.15.70:0/1753085032 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb76c003d10 con 0x7fb774103480 2024-04-01T01:33:00.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb77b8d4640 1 -- 172.21.15.70:0/1753085032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb77419f920 con 0x7fb774103480 2024-04-01T01:33:00.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb76a7fc640 1 -- 172.21.15.70:0/1753085032 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c010980 con 0x7fb774103480 2024-04-01T01:33:00.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.974+0000 7fb77b8d4640 1 -- 172.21.15.70:0/1753085032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb77419fe20 con 0x7fb774103480 2024-04-01T01:33:00.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.975+0000 7fb77b8d4640 1 -- 172.21.15.70:0/1753085032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb774108d30 con 0x7fb774103480 2024-04-01T01:33:00.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.976+0000 7fb76a7fc640 1 -- 172.21.15.70:0/1753085032 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fb76c004020 con 0x7fb774103480 2024-04-01T01:33:00.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.977+0000 7fb76a7fc640 1 --2- 172.21.15.70:0/1753085032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7600a24a0 0x7fb7600a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:00.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.977+0000 7fb779670640 1 --2- 172.21.15.70:0/1753085032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7600a24a0 0x7fb7600a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:00.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.977+0000 7fb76a7fc640 1 -- 172.21.15.70:0/1753085032 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb76c0ceec0 con 0x7fb774103480 2024-04-01T01:33:00.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.978+0000 7fb779670640 1 --2- 172.21.15.70:0/1753085032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7600a24a0 0x7fb7600a4960 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb770000c00 tx=0x7fb770005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:00.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:00.982+0000 7fb76a7fc640 1 -- 172.21.15.70:0/1753085032 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb76c09bd20 con 0x7fb774103480 2024-04-01T01:33:00.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:00 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3386968397' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:00.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:00 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3718267935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:00.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:00 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2613309226' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:00 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3386968397' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:00 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3718267935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:00 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2613309226' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:01.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.155+0000 7fb77b8d4640 1 -- 172.21.15.70:0/1753085032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb774062350 con 0x7fb774103480 2024-04-01T01:33:01.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.156+0000 7fb76a7fc640 1 -- 172.21.15.70:0/1753085032 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7fb76c09b620 con 0x7fb774103480 2024-04-01T01:33:01.156 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:33:01.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.158+0000 7fb77b8d4640 1 -- 172.21.15.70:0/1753085032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7600a24a0 msgr2=0x7fb7600a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:01.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.158+0000 7fb77b8d4640 1 --2- 172.21.15.70:0/1753085032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7600a24a0 0x7fb7600a4960 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb770000c00 tx=0x7fb770005fd0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.158+0000 7fb77b8d4640 1 -- 172.21.15.70:0/1753085032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb774103480 msgr2=0x7fb77419efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:01.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.158+0000 7fb77b8d4640 1 --2- 172.21.15.70:0/1753085032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb774103480 0x7fb77419efc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb76c00efd0 tx=0x7fb76c00c6a0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.158+0000 7fb77b8d4640 1 -- 172.21.15.70:0/1753085032 shutdown_connections 2024-04-01T01:33:01.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.158+0000 7fb77b8d4640 1 --2- 172.21.15.70:0/1753085032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb774103480 0x7fb77419efc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.158+0000 7fb77b8d4640 1 --2- 172.21.15.70:0/1753085032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7600a24a0 0x7fb7600a4960 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.158+0000 7fb77b8d4640 1 --2- 172.21.15.70:0/1753085032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb774102ac0 0x7fb774197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.158+0000 7fb77b8d4640 1 --2- 172.21.15.70:0/1753085032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7740ffa50 0x7fb774197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.158+0000 7fb77b8d4640 1 -- 172.21.15.70:0/1753085032 >> 172.21.15.70:0/1753085032 conn(0x7fb7740fb380 msgr2=0x7fb774105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:01.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.158+0000 7fb77b8d4640 1 -- 172.21.15.70:0/1753085032 shutdown_connections 2024-04-01T01:33:01.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.158+0000 7fb77b8d4640 1 -- 172.21.15.70:0/1753085032 wait complete. 2024-04-01T01:33:01.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:33:01.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.271+0000 7fe65a6f4640 1 -- 172.21.15.70:0/991059351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6541018e0 msgr2=0x7fe654101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:01.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.271+0000 7fe65a6f4640 1 --2- 172.21.15.70:0/991059351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6541018e0 0x7fe654101d60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe648009900 tx=0x7fe64802bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:01.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.271+0000 7fe65a6f4640 1 -- 172.21.15.70:0/991059351 shutdown_connections 2024-04-01T01:33:01.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.271+0000 7fe65a6f4640 1 --2- 172.21.15.70:0/991059351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6541022a0 0x7fe654108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.271+0000 7fe65a6f4640 1 --2- 172.21.15.70:0/991059351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6541018e0 0x7fe654101d60 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.271+0000 7fe65a6f4640 1 --2- 172.21.15.70:0/991059351 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6541006f0 0x7fe654100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.271+0000 7fe65a6f4640 1 -- 172.21.15.70:0/991059351 >> 172.21.15.70:0/991059351 conn(0x7fe6540fbc80 msgr2=0x7fe6540fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:01.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.271+0000 7fe65a6f4640 1 -- 172.21.15.70:0/991059351 shutdown_connections 2024-04-01T01:33:01.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.271+0000 7fe65a6f4640 1 -- 172.21.15.70:0/991059351 wait complete. 2024-04-01T01:33:01.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.271+0000 7fe65a6f4640 1 Processor -- start 2024-04-01T01:33:01.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.271+0000 7fe65a6f4640 1 -- start start 2024-04-01T01:33:01.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe65a6f4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6541006f0 0x7fe6541968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:01.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe65a6f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6541018e0 0x7fe654196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:01.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe65a6f4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6541022a0 0x7fe65419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:01.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe65a6f4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe65410c1b0 con 0x7fe6541018e0 2024-04-01T01:33:01.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe65a6f4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe65410c030 con 0x7fe6541006f0 2024-04-01T01:33:01.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe65a6f4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe65410c330 con 0x7fe6541022a0 2024-04-01T01:33:01.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe6537fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6541018e0 0x7fe654196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe6537fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6541018e0 0x7fe654196df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39460/0 (socket says 172.21.15.70:39460) 2024-04-01T01:33:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe6537fe640 1 -- 172.21.15.70:0/3539656533 learned_addr learned my addr 172.21.15.70:0/3539656533 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe653fff640 1 --2- 172.21.15.70:0/3539656533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6541006f0 0x7fe6541968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe658c91640 1 --2- 172.21.15.70:0/3539656533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6541022a0 0x7fe65419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe6537fe640 1 -- 172.21.15.70:0/3539656533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6541022a0 msgr2=0x7fe65419de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe6537fe640 1 --2- 172.21.15.70:0/3539656533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6541022a0 0x7fe65419de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe6537fe640 1 -- 172.21.15.70:0/3539656533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6541006f0 msgr2=0x7fe6541968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe6537fe640 1 --2- 172.21.15.70:0/3539656533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6541006f0 0x7fe6541968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe6537fe640 1 -- 172.21.15.70:0/3539656533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe65419e480 con 0x7fe6541018e0 2024-04-01T01:33:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe653fff640 1 --2- 172.21.15.70:0/3539656533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6541006f0 0x7fe6541968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe6537fe640 1 --2- 172.21.15.70:0/3539656533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6541018e0 0x7fe654196df0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe648006010 tx=0x7fe648005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.272+0000 7fe6517fa640 1 -- 172.21.15.70:0/3539656533 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe648004e00 con 0x7fe6541018e0 2024-04-01T01:33:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.273+0000 7fe6517fa640 1 -- 172.21.15.70:0/3539656533 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe648005450 con 0x7fe6541018e0 2024-04-01T01:33:01.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.273+0000 7fe6517fa640 1 -- 172.21.15.70:0/3539656533 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64803c450 con 0x7fe6541018e0 2024-04-01T01:33:01.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.273+0000 7fe65a6f4640 1 -- 172.21.15.70:0/3539656533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe654069d80 con 0x7fe6541018e0 2024-04-01T01:33:01.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.273+0000 7fe65a6f4640 1 -- 172.21.15.70:0/3539656533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe65406a2d0 con 0x7fe6541018e0 2024-04-01T01:33:01.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.273+0000 7fe65a6f4640 1 -- 172.21.15.70:0/3539656533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe654068990 con 0x7fe6541018e0 2024-04-01T01:33:01.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.274+0000 7fe6517fa640 1 -- 172.21.15.70:0/3539656533 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fe6480036f0 con 0x7fe6541018e0 2024-04-01T01:33:01.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.275+0000 7fe6517fa640 1 --2- 172.21.15.70:0/3539656533 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe63c0a24a0 0x7fe63c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:01.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.275+0000 7fe653fff640 1 --2- 172.21.15.70:0/3539656533 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe63c0a24a0 0x7fe63c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:01.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.275+0000 7fe6517fa640 1 -- 172.21.15.70:0/3539656533 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe6480f2cf0 con 0x7fe6541018e0 2024-04-01T01:33:01.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.276+0000 7fe653fff640 1 --2- 172.21.15.70:0/3539656533 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe63c0a24a0 0x7fe63c0a4960 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe654101740 tx=0x7fe644005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:01.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.279+0000 7fe6517fa640 1 -- 172.21.15.70:0/3539656533 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe6480f7070 con 0x7fe6541018e0 2024-04-01T01:33:01.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.490+0000 7fe65a6f4640 1 -- 172.21.15.70:0/3539656533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe654107550 con 0x7fe6541018e0 2024-04-01T01:33:01.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.490+0000 7fe6517fa640 1 -- 172.21.15.70:0/3539656533 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11929 (secure 0 0 0) 0x7fe6480bfb50 con 0x7fe6541018e0 2024-04-01T01:33:01.491 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:01.491 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:31:38.563210+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24451,24464,24379,24404,24418]},"id":1}]} 2024-04-01T01:33:01.491 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 28 2024-04-01T01:33:01.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.492+0000 7fe65a6f4640 1 -- 172.21.15.70:0/3539656533 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe63c0a24a0 msgr2=0x7fe63c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:01.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.492+0000 7fe65a6f4640 1 --2- 172.21.15.70:0/3539656533 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe63c0a24a0 0x7fe63c0a4960 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe654101740 tx=0x7fe644005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:01.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.492+0000 7fe65a6f4640 1 -- 172.21.15.70:0/3539656533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6541018e0 msgr2=0x7fe654196df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:01.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.492+0000 7fe65a6f4640 1 --2- 172.21.15.70:0/3539656533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6541018e0 0x7fe654196df0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe648006010 tx=0x7fe648005e40 comp rx=0 tx=0).stop 2024-04-01T01:33:01.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.492+0000 7fe65a6f4640 1 -- 172.21.15.70:0/3539656533 shutdown_connections 2024-04-01T01:33:01.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.492+0000 7fe65a6f4640 1 --2- 172.21.15.70:0/3539656533 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe63c0a24a0 0x7fe63c0a4960 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.492+0000 7fe65a6f4640 1 --2- 172.21.15.70:0/3539656533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6541022a0 0x7fe65419de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.492+0000 7fe65a6f4640 1 --2- 172.21.15.70:0/3539656533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6541018e0 0x7fe654196df0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.492+0000 7fe65a6f4640 1 --2- 172.21.15.70:0/3539656533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6541006f0 0x7fe6541968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.492+0000 7fe65a6f4640 1 -- 172.21.15.70:0/3539656533 >> 172.21.15.70:0/3539656533 conn(0x7fe6540fbc80 msgr2=0x7fe654104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:01.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.492+0000 7fe65a6f4640 1 -- 172.21.15.70:0/3539656533 shutdown_connections 2024-04-01T01:33:01.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.492+0000 7fe65a6f4640 1 -- 172.21.15.70:0/3539656533 wait complete. 2024-04-01T01:33:01.504 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:33:01.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.607+0000 7faed7725640 1 -- 172.21.15.70:0/4260046839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed0100670 msgr2=0x7faed0100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:01.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.607+0000 7faed7725640 1 --2- 172.21.15.70:0/4260046839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed0100670 0x7faed0100a90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7faecc009900 tx=0x7faecc02bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:01.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.607+0000 7faed7725640 1 -- 172.21.15.70:0/4260046839 shutdown_connections 2024-04-01T01:33:01.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.607+0000 7faed7725640 1 --2- 172.21.15.70:0/4260046839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed0102220 0x7faed0108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.607+0000 7faed7725640 1 --2- 172.21.15.70:0/4260046839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faed0101860 0x7faed0101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.607+0000 7faed7725640 1 --2- 172.21.15.70:0/4260046839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed0100670 0x7faed0100a90 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.607+0000 7faed7725640 1 -- 172.21.15.70:0/4260046839 >> 172.21.15.70:0/4260046839 conn(0x7faed00fbbe0 msgr2=0x7faed00fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:01.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.607+0000 7faed7725640 1 -- 172.21.15.70:0/4260046839 shutdown_connections 2024-04-01T01:33:01.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.607+0000 7faed7725640 1 -- 172.21.15.70:0/4260046839 wait complete. 2024-04-01T01:33:01.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed7725640 1 Processor -- start 2024-04-01T01:33:01.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed7725640 1 -- start start 2024-04-01T01:33:01.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed7725640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed0100670 0x7faed006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:01.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed7725640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faed0101860 0x7faed019c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:01.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed7725640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed0102220 0x7faed019ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:01.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed7725640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faed010c190 con 0x7faed0100670 2024-04-01T01:33:01.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed7725640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faed010c010 con 0x7faed0101860 2024-04-01T01:33:01.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed7725640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faed010c310 con 0x7faed0102220 2024-04-01T01:33:01.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed54c1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed0100670 0x7faed006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:01.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed54c1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed0100670 0x7faed006a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39474/0 (socket says 172.21.15.70:39474) 2024-04-01T01:33:01.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed54c1640 1 -- 172.21.15.70:0/555860445 learned_addr learned my addr 172.21.15.70:0/555860445 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:01.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed4cc0640 1 --2- 172.21.15.70:0/555860445 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faed0101860 0x7faed019c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:01.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed5cc2640 1 --2- 172.21.15.70:0/555860445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed0102220 0x7faed019ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:01.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed54c1640 1 -- 172.21.15.70:0/555860445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed0102220 msgr2=0x7faed019ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:01.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed54c1640 1 --2- 172.21.15.70:0/555860445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed0102220 0x7faed019ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed54c1640 1 -- 172.21.15.70:0/555860445 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faed0101860 msgr2=0x7faed019c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:01.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed54c1640 1 --2- 172.21.15.70:0/555860445 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faed0101860 0x7faed019c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed54c1640 1 -- 172.21.15.70:0/555860445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed019f5c0 con 0x7faed0100670 2024-04-01T01:33:01.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.608+0000 7faed54c1640 1 --2- 172.21.15.70:0/555860445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed0100670 0x7faed006a950 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7faecc000c00 tx=0x7faecc005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:01.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.609+0000 7faec67fc640 1 -- 172.21.15.70:0/555860445 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc043020 con 0x7faed0100670 2024-04-01T01:33:01.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.609+0000 7faed7725640 1 -- 172.21.15.70:0/555860445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed019f850 con 0x7faed0100670 2024-04-01T01:33:01.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.609+0000 7faec67fc640 1 -- 172.21.15.70:0/555860445 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faecc004350 con 0x7faed0100670 2024-04-01T01:33:01.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.609+0000 7faec67fc640 1 -- 172.21.15.70:0/555860445 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc003b80 con 0x7faed0100670 2024-04-01T01:33:01.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.609+0000 7faed7725640 1 -- 172.21.15.70:0/555860445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed019fdf0 con 0x7faed0100670 2024-04-01T01:33:01.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.610+0000 7faed7725640 1 -- 172.21.15.70:0/555860445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faed0107a10 con 0x7faed0100670 2024-04-01T01:33:01.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.611+0000 7faec67fc640 1 -- 172.21.15.70:0/555860445 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7faecc046020 con 0x7faed0100670 2024-04-01T01:33:01.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.611+0000 7faec67fc640 1 --2- 172.21.15.70:0/555860445 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faebc0a24a0 0x7faebc0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:01.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.611+0000 7faed4cc0640 1 --2- 172.21.15.70:0/555860445 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faebc0a24a0 0x7faebc0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:01.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.611+0000 7faec67fc640 1 -- 172.21.15.70:0/555860445 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7faecc0f0bb0 con 0x7faed0100670 2024-04-01T01:33:01.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.611+0000 7faed4cc0640 1 --2- 172.21.15.70:0/555860445 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faebc0a24a0 0x7faebc0a4960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7faed006a6c0 tx=0x7faec0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:01.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.615+0000 7faec67fc640 1 -- 172.21.15.70:0/555860445 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7faecc04d020 con 0x7faed0100670 2024-04-01T01:33:01.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:01 smithi070 ceph-mon[32179]: pgmap v342: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-01T01:33:01.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:01 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3208380498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:01.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:01 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1753085032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:01.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:01 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3539656533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:01.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.793+0000 7faed7725640 1 -- 172.21.15.70:0/555860445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faed0062350 con 0x7faed0100670 2024-04-01T01:33:01.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.795+0000 7faec67fc640 1 -- 172.21.15.70:0/555860445 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24595 (secure 0 0 0) 0x7faecc0bda10 con 0x7faed0100670 2024-04-01T01:33:01.795 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:01.796 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":91,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:32:19.680117+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:33:01.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.797+0000 7faed7725640 1 -- 172.21.15.70:0/555860445 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faebc0a24a0 msgr2=0x7faebc0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:01.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.797+0000 7faed7725640 1 --2- 172.21.15.70:0/555860445 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faebc0a24a0 0x7faebc0a4960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7faed006a6c0 tx=0x7faec0005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:01.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.797+0000 7faed7725640 1 -- 172.21.15.70:0/555860445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed0100670 msgr2=0x7faed006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:01.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.797+0000 7faed7725640 1 --2- 172.21.15.70:0/555860445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed0100670 0x7faed006a950 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7faecc000c00 tx=0x7faecc005dc0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.797+0000 7faed7725640 1 -- 172.21.15.70:0/555860445 shutdown_connections 2024-04-01T01:33:01.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.797+0000 7faed7725640 1 --2- 172.21.15.70:0/555860445 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faebc0a24a0 0x7faebc0a4960 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.797+0000 7faed7725640 1 --2- 172.21.15.70:0/555860445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faed0102220 0x7faed019ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.797+0000 7faed7725640 1 --2- 172.21.15.70:0/555860445 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faed0101860 0x7faed019c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.797+0000 7faed7725640 1 --2- 172.21.15.70:0/555860445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faed0100670 0x7faed006a950 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.797+0000 7faed7725640 1 -- 172.21.15.70:0/555860445 >> 172.21.15.70:0/555860445 conn(0x7faed00fbbe0 msgr2=0x7faed0104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:01.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.797+0000 7faed7725640 1 -- 172.21.15.70:0/555860445 shutdown_connections 2024-04-01T01:33:01.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.797+0000 7faed7725640 1 -- 172.21.15.70:0/555860445 wait complete. 2024-04-01T01:33:01.809 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-01T01:26:42.809779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-01T01:31:13.612369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-01T01:31:14.910162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-01T01:31:18.783179+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-01T01:33:01.810 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-01T01:33:01.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.915+0000 7f105d86b640 1 -- 172.21.15.70:0/3840609878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1058102210 msgr2=0x7f1058108aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.915+0000 7f105d86b640 1 --2- 172.21.15.70:0/3840609878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1058102210 0x7f1058108aa0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1050009ca0 tx=0x7f105002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.915+0000 7f105d86b640 1 -- 172.21.15.70:0/3840609878 shutdown_connections 2024-04-01T01:33:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.915+0000 7f105d86b640 1 --2- 172.21.15.70:0/3840609878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1058102210 0x7f1058108aa0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.915+0000 7f105d86b640 1 --2- 172.21.15.70:0/3840609878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1058101850 0x7f1058101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.915+0000 7f105d86b640 1 --2- 172.21.15.70:0/3840609878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1058100660 0x7f1058100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.915+0000 7f105d86b640 1 -- 172.21.15.70:0/3840609878 >> 172.21.15.70:0/3840609878 conn(0x7f10580fbbb0 msgr2=0x7f10580fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.915+0000 7f105d86b640 1 -- 172.21.15.70:0/3840609878 shutdown_connections 2024-04-01T01:33:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.915+0000 7f105d86b640 1 -- 172.21.15.70:0/3840609878 wait complete. 2024-04-01T01:33:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.915+0000 7f105d86b640 1 Processor -- start 2024-04-01T01:33:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.915+0000 7f105d86b640 1 -- start start 2024-04-01T01:33:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f105d86b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1058100660 0x7f1058197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f105d86b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1058101850 0x7f1058197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f105d86b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1058102210 0x7f105819eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f105d86b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f105810d2b0 con 0x7f1058101850 2024-04-01T01:33:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f105d86b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f105810d130 con 0x7f1058100660 2024-04-01T01:33:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f105d86b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f105810d430 con 0x7f1058102210 2024-04-01T01:33:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f10567fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1058101850 0x7f1058197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f10567fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1058101850 0x7f1058197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39498/0 (socket says 172.21.15.70:39498) 2024-04-01T01:33:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f10567fc640 1 -- 172.21.15.70:0/53260362 learned_addr learned my addr 172.21.15.70:0/53260362 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f1056ffd640 1 --2- 172.21.15.70:0/53260362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1058100660 0x7f1058197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f10577fe640 1 --2- 172.21.15.70:0/53260362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1058102210 0x7f105819eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f10567fc640 1 -- 172.21.15.70:0/53260362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1058102210 msgr2=0x7f105819eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f10567fc640 1 --2- 172.21.15.70:0/53260362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1058102210 0x7f105819eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f10567fc640 1 -- 172.21.15.70:0/53260362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1058100660 msgr2=0x7f1058197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f10567fc640 1 --2- 172.21.15.70:0/53260362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1058100660 0x7f1058197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f10567fc640 1 -- 172.21.15.70:0/53260362 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105819f520 con 0x7f1058101850 2024-04-01T01:33:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f10577fe640 1 --2- 172.21.15.70:0/53260362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1058102210 0x7f105819eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:01.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f1056ffd640 1 --2- 172.21.15.70:0/53260362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1058100660 0x7f1058197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:01.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f10567fc640 1 --2- 172.21.15.70:0/53260362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1058101850 0x7f1058197e40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f104000e9f0 tx=0x7f104000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:01.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f105c869640 1 -- 172.21.15.70:0/53260362 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104000cd80 con 0x7f1058101850 2024-04-01T01:33:01.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f105c869640 1 -- 172.21.15.70:0/53260362 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f10400054d0 con 0x7f1058101850 2024-04-01T01:33:01.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f105d86b640 1 -- 172.21.15.70:0/53260362 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f105819f810 con 0x7f1058101850 2024-04-01T01:33:01.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f105c869640 1 -- 172.21.15.70:0/53260362 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1040010490 con 0x7f1058101850 2024-04-01T01:33:01.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.916+0000 7f105d86b640 1 -- 172.21.15.70:0/53260362 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f105819fd60 con 0x7f1058101850 2024-04-01T01:33:01.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.917+0000 7f105d86b640 1 -- 172.21.15.70:0/53260362 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1058107980 con 0x7f1058101850 2024-04-01T01:33:01.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.919+0000 7f105c869640 1 -- 172.21.15.70:0/53260362 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f1040005000 con 0x7f1058101850 2024-04-01T01:33:01.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.919+0000 7f105c869640 1 --2- 172.21.15.70:0/53260362 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f10440a2520 0x7f10440a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:01.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.919+0000 7f1056ffd640 1 --2- 172.21.15.70:0/53260362 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f10440a2520 0x7f10440a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:01.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.919+0000 7f105c869640 1 -- 172.21.15.70:0/53260362 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f10400cef90 con 0x7f1058101850 2024-04-01T01:33:01.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.920+0000 7f1056ffd640 1 --2- 172.21.15.70:0/53260362 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f10440a2520 0x7f10440a49e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f10581016b0 tx=0x7f1048005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:01.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:01.922+0000 7f105c869640 1 -- 172.21.15.70:0/53260362 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1040053280 con 0x7f1058101850 2024-04-01T01:33:01.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:01 smithi189 ceph-mon[35753]: pgmap v342: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-01T01:33:01.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:01 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3208380498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:01.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:01 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1753085032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:01.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:01 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3539656533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:02.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:01 smithi081 ceph-mon[35142]: pgmap v342: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-01T01:33:02.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:01 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3208380498' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:02.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:01 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1753085032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:02.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:01 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3539656533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:02.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.095+0000 7f105d86b640 1 -- 172.21.15.70:0/53260362 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f105802d300 con 0x7f1058101850 2024-04-01T01:33:02.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.096+0000 7f105c869640 1 -- 172.21.15.70:0/53260362 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f10400052f0 con 0x7f1058101850 2024-04-01T01:33:02.096 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-01T01:33:02.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.098+0000 7f105d86b640 1 -- 172.21.15.70:0/53260362 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f10440a2520 msgr2=0x7f10440a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:02.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.098+0000 7f105d86b640 1 --2- 172.21.15.70:0/53260362 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f10440a2520 0x7f10440a49e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f10581016b0 tx=0x7f1048005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:02.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.098+0000 7f105d86b640 1 -- 172.21.15.70:0/53260362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1058101850 msgr2=0x7f1058197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:02.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.098+0000 7f105d86b640 1 --2- 172.21.15.70:0/53260362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1058101850 0x7f1058197e40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f104000e9f0 tx=0x7f104000edf0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.098+0000 7f105d86b640 1 -- 172.21.15.70:0/53260362 shutdown_connections 2024-04-01T01:33:02.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.098+0000 7f105d86b640 1 --2- 172.21.15.70:0/53260362 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f10440a2520 0x7f10440a49e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.098+0000 7f105d86b640 1 --2- 172.21.15.70:0/53260362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1058102210 0x7f105819eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.098+0000 7f105d86b640 1 --2- 172.21.15.70:0/53260362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1058101850 0x7f1058197e40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.098+0000 7f105d86b640 1 --2- 172.21.15.70:0/53260362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1058100660 0x7f1058197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.098+0000 7f105d86b640 1 -- 172.21.15.70:0/53260362 >> 172.21.15.70:0/53260362 conn(0x7f10580fbbb0 msgr2=0x7f1058104b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:02.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.098+0000 7f105d86b640 1 -- 172.21.15.70:0/53260362 shutdown_connections 2024-04-01T01:33:02.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.098+0000 7f105d86b640 1 -- 172.21.15.70:0/53260362 wait complete. 2024-04-01T01:33:02.109 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-01T01:33:02.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.213+0000 7f14c57f6640 1 -- 172.21.15.70:0/4132159832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c0104b00 msgr2=0x7f14c0106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:02.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.213+0000 7f14c57f6640 1 --2- 172.21.15.70:0/4132159832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c0104b00 0x7f14c0106ef0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009900 tx=0x7f14b402bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:02.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.213+0000 7f14c57f6640 1 -- 172.21.15.70:0/4132159832 shutdown_connections 2024-04-01T01:33:02.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.213+0000 7f14c57f6640 1 --2- 172.21.15.70:0/4132159832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14c0107430 0x7f14c0109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.213+0000 7f14c57f6640 1 --2- 172.21.15.70:0/4132159832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c0104b00 0x7f14c0106ef0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.213+0000 7f14c57f6640 1 --2- 172.21.15.70:0/4132159832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14c01021d0 0x7f14c01045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.213+0000 7f14c57f6640 1 -- 172.21.15.70:0/4132159832 >> 172.21.15.70:0/4132159832 conn(0x7f14c00fbbe0 msgr2=0x7f14c00fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:02.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.213+0000 7f14c57f6640 1 -- 172.21.15.70:0/4132159832 shutdown_connections 2024-04-01T01:33:02.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.214+0000 7f14c57f6640 1 -- 172.21.15.70:0/4132159832 wait complete. 2024-04-01T01:33:02.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.214+0000 7f14c57f6640 1 Processor -- start 2024-04-01T01:33:02.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.214+0000 7f14c57f6640 1 -- start start 2024-04-01T01:33:02.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.214+0000 7f14c57f6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14c01021d0 0x7f14c0197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:02.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.214+0000 7f14c57f6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c0104b00 0x7f14c0197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:02.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.214+0000 7f14c57f6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14c0107430 0x7f14c019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:02.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.214+0000 7f14c57f6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f14c010d290 con 0x7f14c0107430 2024-04-01T01:33:02.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.214+0000 7f14c57f6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f14c010d110 con 0x7f14c0104b00 2024-04-01T01:33:02.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.214+0000 7f14c57f6640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f14c010d410 con 0x7f14c01021d0 2024-04-01T01:33:02.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.214+0000 7f14bf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14c0107430 0x7f14c019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:02.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.214+0000 7f14be7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c0104b00 0x7f14c0197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:02.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.214+0000 7f14beffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14c01021d0 0x7f14c0197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:02.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.215+0000 7f14be7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c0104b00 0x7f14c0197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:52646/0 (socket says 172.21.15.70:52646) 2024-04-01T01:33:02.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.215+0000 7f14bf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14c0107430 0x7f14c019ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39508/0 (socket says 172.21.15.70:39508) 2024-04-01T01:33:02.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.215+0000 7f14beffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14c01021d0 0x7f14c0197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39548/0 (socket says 172.21.15.70:39548) 2024-04-01T01:33:02.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.215+0000 7f14be7fc640 1 -- 172.21.15.70:0/2523060917 learned_addr learned my addr 172.21.15.70:0/2523060917 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:02.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.215+0000 7f14be7fc640 1 -- 172.21.15.70:0/2523060917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14c01021d0 msgr2=0x7f14c0197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:02.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.215+0000 7f14be7fc640 1 --2- 172.21.15.70:0/2523060917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14c01021d0 0x7f14c0197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.215+0000 7f14be7fc640 1 -- 172.21.15.70:0/2523060917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14c0107430 msgr2=0x7f14c019ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:02.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.215+0000 7f14be7fc640 1 --2- 172.21.15.70:0/2523060917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14c0107430 0x7f14c019ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.215+0000 7f14be7fc640 1 -- 172.21.15.70:0/2523060917 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14c019f600 con 0x7f14c0104b00 2024-04-01T01:33:02.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.215+0000 7f14bf7fe640 1 --2- 172.21.15.70:0/2523060917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14c0107430 0x7f14c019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:02.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.215+0000 7f14be7fc640 1 --2- 172.21.15.70:0/2523060917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c0104b00 0x7f14c0197f70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f14b4000c00 tx=0x7f14b4005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:02.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.215+0000 7f14a7fff640 1 -- 172.21.15.70:0/2523060917 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4043020 con 0x7f14c0104b00 2024-04-01T01:33:02.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.215+0000 7f14a7fff640 1 -- 172.21.15.70:0/2523060917 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f14b4004350 con 0x7f14c0104b00 2024-04-01T01:33:02.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.215+0000 7f14c57f6640 1 -- 172.21.15.70:0/2523060917 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14c019f890 con 0x7f14c0104b00 2024-04-01T01:33:02.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.215+0000 7f14a7fff640 1 -- 172.21.15.70:0/2523060917 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b4003b80 con 0x7f14c0104b00 2024-04-01T01:33:02.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.215+0000 7f14c57f6640 1 -- 172.21.15.70:0/2523060917 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14c019fde0 con 0x7f14c0104b00 2024-04-01T01:33:02.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.216+0000 7f14c57f6640 1 -- 172.21.15.70:0/2523060917 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14c00fd6d0 con 0x7f14c0104b00 2024-04-01T01:33:02.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.217+0000 7f14a7fff640 1 -- 172.21.15.70:0/2523060917 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f14b4046020 con 0x7f14c0104b00 2024-04-01T01:33:02.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.218+0000 7f14a7fff640 1 --2- 172.21.15.70:0/2523060917 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f14a80a24a0 0x7f14a80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:02.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.218+0000 7f14beffd640 1 --2- 172.21.15.70:0/2523060917 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f14a80a24a0 0x7f14a80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:02.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.218+0000 7f14a7fff640 1 -- 172.21.15.70:0/2523060917 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f14b40f1080 con 0x7f14c0104b00 2024-04-01T01:33:02.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.218+0000 7f14beffd640 1 --2- 172.21.15.70:0/2523060917 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f14a80a24a0 0x7f14a80a4960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009bd0 tx=0x7f14b0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:02.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.222+0000 7f14a7fff640 1 -- 172.21.15.70:0/2523060917 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f14b404d020 con 0x7f14c0104b00 2024-04-01T01:33:02.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.395+0000 7f14c57f6640 1 -- 172.21.15.70:0/2523060917 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f14c002d3c0 con 0x7f14c0104b00 2024-04-01T01:33:02.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.395+0000 7f14a7fff640 1 -- 172.21.15.70:0/2523060917 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f14b40bdee0 con 0x7f14c0104b00 2024-04-01T01:33:02.395 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:33:02.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.397+0000 7f14c57f6640 1 -- 172.21.15.70:0/2523060917 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f14a80a24a0 msgr2=0x7f14a80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:02.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.397+0000 7f14c57f6640 1 --2- 172.21.15.70:0/2523060917 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f14a80a24a0 0x7f14a80a4960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009bd0 tx=0x7f14b0005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:02.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.397+0000 7f14c57f6640 1 -- 172.21.15.70:0/2523060917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c0104b00 msgr2=0x7f14c0197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:02.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.397+0000 7f14c57f6640 1 --2- 172.21.15.70:0/2523060917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c0104b00 0x7f14c0197f70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f14b4000c00 tx=0x7f14b4005dc0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.397+0000 7f14c57f6640 1 -- 172.21.15.70:0/2523060917 shutdown_connections 2024-04-01T01:33:02.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.397+0000 7f14c57f6640 1 --2- 172.21.15.70:0/2523060917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f14c0107430 0x7f14c019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.397+0000 7f14c57f6640 1 --2- 172.21.15.70:0/2523060917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14c0104b00 0x7f14c0197f70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.397+0000 7f14c57f6640 1 --2- 172.21.15.70:0/2523060917 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f14a80a24a0 0x7f14a80a4960 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.397+0000 7f14c57f6640 1 --2- 172.21.15.70:0/2523060917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14c01021d0 0x7f14c0197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.397+0000 7f14c57f6640 1 -- 172.21.15.70:0/2523060917 >> 172.21.15.70:0/2523060917 conn(0x7f14c00fbbe0 msgr2=0x7f14c0104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:02.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.397+0000 7f14c57f6640 1 -- 172.21.15.70:0/2523060917 shutdown_connections 2024-04-01T01:33:02.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.397+0000 7f14c57f6640 1 -- 172.21.15.70:0/2523060917 wait complete. 2024-04-01T01:33:02.409 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-01T01:33:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.512+0000 7f8c8f67c640 1 -- 172.21.15.70:0/4215638957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c88107430 msgr2=0x7f8c88109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.512+0000 7f8c8f67c640 1 --2- 172.21.15.70:0/4215638957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c88107430 0x7f8c88109880 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8c80009ca0 tx=0x7f8c8002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.512+0000 7f8c8f67c640 1 -- 172.21.15.70:0/4215638957 shutdown_connections 2024-04-01T01:33:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.512+0000 7f8c8f67c640 1 --2- 172.21.15.70:0/4215638957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c88107430 0x7f8c88109880 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.512+0000 7f8c8f67c640 1 --2- 172.21.15.70:0/4215638957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c88104b00 0x7f8c88106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.512+0000 7f8c8f67c640 1 --2- 172.21.15.70:0/4215638957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c881021d0 0x7f8c881045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.512+0000 7f8c8f67c640 1 -- 172.21.15.70:0/4215638957 >> 172.21.15.70:0/4215638957 conn(0x7f8c880fbbc0 msgr2=0x7f8c880fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.512+0000 7f8c8f67c640 1 -- 172.21.15.70:0/4215638957 shutdown_connections 2024-04-01T01:33:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.513+0000 7f8c8f67c640 1 -- 172.21.15.70:0/4215638957 wait complete. 2024-04-01T01:33:02.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.513+0000 7f8c8f67c640 1 Processor -- start 2024-04-01T01:33:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.513+0000 7f8c8f67c640 1 -- start start 2024-04-01T01:33:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.513+0000 7f8c8f67c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c881021d0 0x7f8c88101b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.513+0000 7f8c8f67c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c88104b00 0x7f8c88100180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.513+0000 7f8c8f67c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c88107430 0x7f8c881006c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.513+0000 7f8c8f67c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8810d280 con 0x7f8c88107430 2024-04-01T01:33:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.513+0000 7f8c8f67c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8810d100 con 0x7f8c88104b00 2024-04-01T01:33:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.513+0000 7f8c8f67c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8810d400 con 0x7f8c881021d0 2024-04-01T01:33:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.513+0000 7f8c8dc19640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c88107430 0x7f8c881006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.513+0000 7f8c8dc19640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c88107430 0x7f8c881006c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39522/0 (socket says 172.21.15.70:39522) 2024-04-01T01:33:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.513+0000 7f8c8dc19640 1 -- 172.21.15.70:0/204231845 learned_addr learned my addr 172.21.15.70:0/204231845 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.513+0000 7f8c8d418640 1 --2- 172.21.15.70:0/204231845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c881021d0 0x7f8c88101b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.513+0000 7f8c8cc17640 1 --2- 172.21.15.70:0/204231845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c88104b00 0x7f8c88100180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.513+0000 7f8c8dc19640 1 -- 172.21.15.70:0/204231845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c881021d0 msgr2=0x7f8c88101b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.513+0000 7f8c8dc19640 1 --2- 172.21.15.70:0/204231845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c881021d0 0x7f8c88101b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.514+0000 7f8c8dc19640 1 -- 172.21.15.70:0/204231845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c88104b00 msgr2=0x7f8c88100180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.514+0000 7f8c8dc19640 1 --2- 172.21.15.70:0/204231845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c88104b00 0x7f8c88100180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.514+0000 7f8c8dc19640 1 -- 172.21.15.70:0/204231845 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c88100f50 con 0x7f8c88107430 2024-04-01T01:33:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.514+0000 7f8c8dc19640 1 --2- 172.21.15.70:0/204231845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c88107430 0x7f8c881006c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8c8002c700 tx=0x7f8c800328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.514+0000 7f8c7e7fc640 1 -- 172.21.15.70:0/204231845 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c80041020 con 0x7f8c88107430 2024-04-01T01:33:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.514+0000 7f8c7e7fc640 1 -- 172.21.15.70:0/204231845 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c80003e20 con 0x7f8c88107430 2024-04-01T01:33:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.514+0000 7f8c8f67c640 1 -- 172.21.15.70:0/204231845 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c881a19e0 con 0x7f8c88107430 2024-04-01T01:33:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.514+0000 7f8c7e7fc640 1 -- 172.21.15.70:0/204231845 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c80039040 con 0x7f8c88107430 2024-04-01T01:33:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.514+0000 7f8c8f67c640 1 -- 172.21.15.70:0/204231845 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c881a1f30 con 0x7f8c88107430 2024-04-01T01:33:02.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.515+0000 7f8c8f67c640 1 -- 172.21.15.70:0/204231845 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c880fd6b0 con 0x7f8c88107430 2024-04-01T01:33:02.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.516+0000 7f8c7e7fc640 1 -- 172.21.15.70:0/204231845 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f8c80044020 con 0x7f8c88107430 2024-04-01T01:33:02.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.517+0000 7f8c7e7fc640 1 --2- 172.21.15.70:0/204231845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c740a24a0 0x7f8c740a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:02.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.517+0000 7f8c8d418640 1 --2- 172.21.15.70:0/204231845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c740a24a0 0x7f8c740a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:02.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.517+0000 7f8c7e7fc640 1 -- 172.21.15.70:0/204231845 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8c800f1030 con 0x7f8c88107430 2024-04-01T01:33:02.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.517+0000 7f8c8d418640 1 --2- 172.21.15.70:0/204231845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c740a24a0 0x7f8c740a4960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8c840095f0 tx=0x7f8c84005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:02.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.521+0000 7f8c7e7fc640 1 -- 172.21.15.70:0/204231845 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8c8004b020 con 0x7f8c88107430 2024-04-01T01:33:02.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.696+0000 7f8c8f67c640 1 -- 172.21.15.70:0/204231845 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8c8802d300 con 0x7f8c88107430 2024-04-01T01:33:02.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.697+0000 7f8c7e7fc640 1 -- 172.21.15.70:0/204231845 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f8c800bde90 con 0x7f8c88107430 2024-04-01T01:33:02.697 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-01T01:33:02.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.699+0000 7f8c8f67c640 1 -- 172.21.15.70:0/204231845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c740a24a0 msgr2=0x7f8c740a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:02.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.699+0000 7f8c8f67c640 1 --2- 172.21.15.70:0/204231845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c740a24a0 0x7f8c740a4960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8c840095f0 tx=0x7f8c84005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:02.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.699+0000 7f8c8f67c640 1 -- 172.21.15.70:0/204231845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c88107430 msgr2=0x7f8c881006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:02.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.699+0000 7f8c8f67c640 1 --2- 172.21.15.70:0/204231845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c88107430 0x7f8c881006c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8c8002c700 tx=0x7f8c800328a0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.699+0000 7f8c8f67c640 1 -- 172.21.15.70:0/204231845 shutdown_connections 2024-04-01T01:33:02.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.699+0000 7f8c8f67c640 1 --2- 172.21.15.70:0/204231845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c88107430 0x7f8c881006c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.699+0000 7f8c8f67c640 1 --2- 172.21.15.70:0/204231845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c88104b00 0x7f8c88100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.699+0000 7f8c8f67c640 1 --2- 172.21.15.70:0/204231845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c740a24a0 0x7f8c740a4960 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.699+0000 7f8c8f67c640 1 --2- 172.21.15.70:0/204231845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c881021d0 0x7f8c88101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.699+0000 7f8c8f67c640 1 -- 172.21.15.70:0/204231845 >> 172.21.15.70:0/204231845 conn(0x7f8c880fbbc0 msgr2=0x7f8c88102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:02.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.699+0000 7f8c8f67c640 1 -- 172.21.15.70:0/204231845 shutdown_connections 2024-04-01T01:33:02.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.699+0000 7f8c8f67c640 1 -- 172.21.15.70:0/204231845 wait complete. 2024-04-01T01:33:02.710 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-01T01:33:02.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.815+0000 7f6a8d71e640 1 -- 172.21.15.70:0/1293581994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a881021d0 msgr2=0x7f6a881045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:02.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.815+0000 7f6a8d71e640 1 --2- 172.21.15.70:0/1293581994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a881021d0 0x7f6a881045c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009900 tx=0x7f6a7802bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:02.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.815+0000 7f6a8d71e640 1 -- 172.21.15.70:0/1293581994 shutdown_connections 2024-04-01T01:33:02.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.815+0000 7f6a8d71e640 1 --2- 172.21.15.70:0/1293581994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a88107430 0x7f6a88109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.815+0000 7f6a8d71e640 1 --2- 172.21.15.70:0/1293581994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a88104b00 0x7f6a88106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.815+0000 7f6a8d71e640 1 --2- 172.21.15.70:0/1293581994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a881021d0 0x7f6a881045c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.815+0000 7f6a8d71e640 1 -- 172.21.15.70:0/1293581994 >> 172.21.15.70:0/1293581994 conn(0x7f6a880fbbe0 msgr2=0x7f6a880fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:02.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.815+0000 7f6a8d71e640 1 -- 172.21.15.70:0/1293581994 shutdown_connections 2024-04-01T01:33:02.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.815+0000 7f6a8d71e640 1 -- 172.21.15.70:0/1293581994 wait complete. 2024-04-01T01:33:02.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.816+0000 7f6a8d71e640 1 Processor -- start 2024-04-01T01:33:02.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.816+0000 7f6a8d71e640 1 -- start start 2024-04-01T01:33:02.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.816+0000 7f6a8d71e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a881021d0 0x7f6a88101b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:02.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.816+0000 7f6a8d71e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a88104b00 0x7f6a88100180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:02.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.816+0000 7f6a8d71e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a88107430 0x7f6a881006c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:02.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.816+0000 7f6a8d71e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8810d2a0 con 0x7f6a88107430 2024-04-01T01:33:02.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.816+0000 7f6a8d71e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8810d120 con 0x7f6a88104b00 2024-04-01T01:33:02.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.816+0000 7f6a8d71e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8810d420 con 0x7f6a881021d0 2024-04-01T01:33:02.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.816+0000 7f6a877fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a88107430 0x7f6a881006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:02.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.816+0000 7f6a877fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a88107430 0x7f6a881006c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39548/0 (socket says 172.21.15.70:39548) 2024-04-01T01:33:02.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.816+0000 7f6a877fe640 1 -- 172.21.15.70:0/3353544063 learned_addr learned my addr 172.21.15.70:0/3353544063 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:02.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.816+0000 7f6a86ffd640 1 --2- 172.21.15.70:0/3353544063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a881021d0 0x7f6a88101b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:02.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.816+0000 7f6a867fc640 1 --2- 172.21.15.70:0/3353544063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a88104b00 0x7f6a88100180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:02.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.817+0000 7f6a877fe640 1 -- 172.21.15.70:0/3353544063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a881021d0 msgr2=0x7f6a88101b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:02.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.817+0000 7f6a877fe640 1 --2- 172.21.15.70:0/3353544063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a881021d0 0x7f6a88101b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.817+0000 7f6a877fe640 1 -- 172.21.15.70:0/3353544063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a88104b00 msgr2=0x7f6a88100180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:02.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.817+0000 7f6a877fe640 1 --2- 172.21.15.70:0/3353544063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a88104b00 0x7f6a88100180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:02.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.817+0000 7f6a877fe640 1 -- 172.21.15.70:0/3353544063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a88100f50 con 0x7f6a88107430 2024-04-01T01:33:02.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.817+0000 7f6a86ffd640 1 --2- 172.21.15.70:0/3353544063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a881021d0 0x7f6a88101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:02.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.817+0000 7f6a877fe640 1 --2- 172.21.15.70:0/3353544063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a88107430 0x7f6a881006c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6a8000eb10 tx=0x7f6a8000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:02.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.817+0000 7f6a6ffff640 1 -- 172.21.15.70:0/3353544063 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a80004b20 con 0x7f6a88107430 2024-04-01T01:33:02.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.817+0000 7f6a8d71e640 1 -- 172.21.15.70:0/3353544063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a881a1b30 con 0x7f6a88107430 2024-04-01T01:33:02.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.817+0000 7f6a6ffff640 1 -- 172.21.15.70:0/3353544063 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a80004cc0 con 0x7f6a88107430 2024-04-01T01:33:02.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.817+0000 7f6a6ffff640 1 -- 172.21.15.70:0/3353544063 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a80010690 con 0x7f6a88107430 2024-04-01T01:33:02.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.817+0000 7f6a8d71e640 1 -- 172.21.15.70:0/3353544063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a881a2080 con 0x7f6a88107430 2024-04-01T01:33:02.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.818+0000 7f6a8d71e640 1 -- 172.21.15.70:0/3353544063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a880fd6d0 con 0x7f6a88107430 2024-04-01T01:33:02.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.819+0000 7f6a6ffff640 1 -- 172.21.15.70:0/3353544063 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f6a80005020 con 0x7f6a88107430 2024-04-01T01:33:02.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.819+0000 7f6a6ffff640 1 --2- 172.21.15.70:0/3353544063 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6a700a24a0 0x7f6a700a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:02.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.820+0000 7f6a86ffd640 1 --2- 172.21.15.70:0/3353544063 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6a700a24a0 0x7f6a700a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:02.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.820+0000 7f6a6ffff640 1 -- 172.21.15.70:0/3353544063 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6a800cf190 con 0x7f6a88107430 2024-04-01T01:33:02.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.820+0000 7f6a86ffd640 1 --2- 172.21.15.70:0/3353544063 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6a700a24a0 0x7f6a700a4960 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009230 tx=0x7f6a78005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:02.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.824+0000 7f6a6ffff640 1 -- 172.21.15.70:0/3353544063 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6a8009bff0 con 0x7f6a88107430 2024-04-01T01:33:02.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.997+0000 7f6a8d71e640 1 -- 172.21.15.70:0/3353544063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6a880623f0 con 0x7f6a88107430 2024-04-01T01:33:02.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.997+0000 7f6a6ffff640 1 -- 172.21.15.70:0/3353544063 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f6a8009b8f0 con 0x7f6a88107430 2024-04-01T01:33:02.999 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:33:03.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.999+0000 7f6a8d71e640 1 -- 172.21.15.70:0/3353544063 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6a700a24a0 msgr2=0x7f6a700a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.999+0000 7f6a8d71e640 1 --2- 172.21.15.70:0/3353544063 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6a700a24a0 0x7f6a700a4960 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009230 tx=0x7f6a78005fd0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.999+0000 7f6a8d71e640 1 -- 172.21.15.70:0/3353544063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a88107430 msgr2=0x7f6a881006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.999+0000 7f6a8d71e640 1 --2- 172.21.15.70:0/3353544063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a88107430 0x7f6a881006c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6a8000eb10 tx=0x7f6a8000c2d0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.999+0000 7f6a8d71e640 1 -- 172.21.15.70:0/3353544063 shutdown_connections 2024-04-01T01:33:03.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.999+0000 7f6a8d71e640 1 --2- 172.21.15.70:0/3353544063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a88107430 0x7f6a881006c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.999+0000 7f6a8d71e640 1 --2- 172.21.15.70:0/3353544063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a88104b00 0x7f6a88100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.999+0000 7f6a8d71e640 1 --2- 172.21.15.70:0/3353544063 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6a700a24a0 0x7f6a700a4960 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.999+0000 7f6a8d71e640 1 --2- 172.21.15.70:0/3353544063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a881021d0 0x7f6a88101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.999+0000 7f6a8d71e640 1 -- 172.21.15.70:0/3353544063 >> 172.21.15.70:0/3353544063 conn(0x7f6a880fbbe0 msgr2=0x7f6a88102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:03.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.999+0000 7f6a8d71e640 1 -- 172.21.15.70:0/3353544063 shutdown_connections 2024-04-01T01:33:03.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:02.999+0000 7f6a8d71e640 1 -- 172.21.15.70:0/3353544063 wait complete. 2024-04-01T01:33:03.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:02 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/555860445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:03.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:02 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/53260362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:03.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:02 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2523060917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:03.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:02 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/555860445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:03.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:02 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/53260362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:03.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:02 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2523060917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:03.011 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:33:03.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:02 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/555860445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:03.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:02 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/53260362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:03.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:02 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2523060917' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:03.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.111+0000 7f5927615640 1 -- 172.21.15.70:0/82459271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59201018e0 msgr2=0x7f5920101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.111+0000 7f5927615640 1 --2- 172.21.15.70:0/82459271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59201018e0 0x7f5920101d60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f591c009900 tx=0x7f591c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:03.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5927615640 1 -- 172.21.15.70:0/82459271 shutdown_connections 2024-04-01T01:33:03.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5927615640 1 --2- 172.21.15.70:0/82459271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59201022a0 0x7f5920108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5927615640 1 --2- 172.21.15.70:0/82459271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59201018e0 0x7f5920101d60 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5927615640 1 --2- 172.21.15.70:0/82459271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59201006f0 0x7f5920100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5927615640 1 -- 172.21.15.70:0/82459271 >> 172.21.15.70:0/82459271 conn(0x7f59200fbc80 msgr2=0x7f59200fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5927615640 1 -- 172.21.15.70:0/82459271 shutdown_connections 2024-04-01T01:33:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5927615640 1 -- 172.21.15.70:0/82459271 wait complete. 2024-04-01T01:33:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5927615640 1 Processor -- start 2024-04-01T01:33:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5927615640 1 -- start start 2024-04-01T01:33:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5927615640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59201006f0 0x7f59201968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5927615640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59201018e0 0x7f5920196e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5927615640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59201022a0 0x7f592019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5927615640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f592010c190 con 0x7f59201018e0 2024-04-01T01:33:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5927615640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f592010c010 con 0x7f59201006f0 2024-04-01T01:33:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5927615640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f592010c310 con 0x7f59201022a0 2024-04-01T01:33:03.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5924bb0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59201018e0 0x7f5920196e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5924bb0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59201018e0 0x7f5920196e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39564/0 (socket says 172.21.15.70:39564) 2024-04-01T01:33:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.112+0000 7f5924bb0640 1 -- 172.21.15.70:0/3646089499 learned_addr learned my addr 172.21.15.70:0/3646089499 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.113+0000 7f59253b1640 1 --2- 172.21.15.70:0/3646089499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59201006f0 0x7f59201968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.113+0000 7f5925bb2640 1 --2- 172.21.15.70:0/3646089499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59201022a0 0x7f592019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.113+0000 7f5924bb0640 1 -- 172.21.15.70:0/3646089499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59201022a0 msgr2=0x7f592019de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.113+0000 7f5924bb0640 1 --2- 172.21.15.70:0/3646089499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59201022a0 0x7f592019de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.113+0000 7f5924bb0640 1 -- 172.21.15.70:0/3646089499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59201006f0 msgr2=0x7f59201968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.113+0000 7f5924bb0640 1 --2- 172.21.15.70:0/3646089499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59201006f0 0x7f59201968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.113+0000 7f5924bb0640 1 -- 172.21.15.70:0/3646089499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f592019e4f0 con 0x7f59201018e0 2024-04-01T01:33:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.113+0000 7f5924bb0640 1 --2- 172.21.15.70:0/3646089499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59201018e0 0x7f5920196e10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f591c006010 tx=0x7f591c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.113+0000 7f59167fc640 1 -- 172.21.15.70:0/3646089499 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c043020 con 0x7f59201018e0 2024-04-01T01:33:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.113+0000 7f59167fc640 1 -- 172.21.15.70:0/3646089499 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f591c0043d0 con 0x7f59201018e0 2024-04-01T01:33:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.113+0000 7f5927615640 1 -- 172.21.15.70:0/3646089499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f592019e780 con 0x7f59201018e0 2024-04-01T01:33:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.113+0000 7f59167fc640 1 -- 172.21.15.70:0/3646089499 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c003a80 con 0x7f59201018e0 2024-04-01T01:33:03.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.113+0000 7f5927615640 1 -- 172.21.15.70:0/3646089499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f592019ed20 con 0x7f59201018e0 2024-04-01T01:33:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.114+0000 7f5927615640 1 -- 172.21.15.70:0/3646089499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5920107b30 con 0x7f59201018e0 2024-04-01T01:33:03.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.115+0000 7f59167fc640 1 -- 172.21.15.70:0/3646089499 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f591c046020 con 0x7f59201018e0 2024-04-01T01:33:03.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.116+0000 7f59167fc640 1 --2- 172.21.15.70:0/3646089499 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f590c0a2520 0x7f590c0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:03.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.116+0000 7f59253b1640 1 --2- 172.21.15.70:0/3646089499 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f590c0a2520 0x7f590c0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:03.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.116+0000 7f59167fc640 1 -- 172.21.15.70:0/3646089499 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f591c0f1250 con 0x7f59201018e0 2024-04-01T01:33:03.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.116+0000 7f59253b1640 1 --2- 172.21.15.70:0/3646089499 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f590c0a2520 0x7f590c0a49e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5920101740 tx=0x7f5910005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:03.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.119+0000 7f59167fc640 1 -- 172.21.15.70:0/3646089499 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f591c04d020 con 0x7f59201018e0 2024-04-01T01:33:03.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.329+0000 7f5927615640 1 -- 172.21.15.70:0/3646089499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f592002d170 con 0x7f59201018e0 2024-04-01T01:33:03.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.330+0000 7f59167fc640 1 -- 172.21.15.70:0/3646089499 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11929 (secure 0 0 0) 0x7f591c0be0b0 con 0x7f59201018e0 2024-04-01T01:33:03.330 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:03.330 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:31:38.563210+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24451,24464,24379,24404,24418]},"id":1}]} 2024-04-01T01:33:03.330 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 28 2024-04-01T01:33:03.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.331+0000 7f5927615640 1 -- 172.21.15.70:0/3646089499 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f590c0a2520 msgr2=0x7f590c0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.331+0000 7f5927615640 1 --2- 172.21.15.70:0/3646089499 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f590c0a2520 0x7f590c0a49e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5920101740 tx=0x7f5910005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:03.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.331+0000 7f5927615640 1 -- 172.21.15.70:0/3646089499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59201018e0 msgr2=0x7f5920196e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.331+0000 7f5927615640 1 --2- 172.21.15.70:0/3646089499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59201018e0 0x7f5920196e10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f591c006010 tx=0x7f591c005e40 comp rx=0 tx=0).stop 2024-04-01T01:33:03.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.331+0000 7f5927615640 1 -- 172.21.15.70:0/3646089499 shutdown_connections 2024-04-01T01:33:03.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.331+0000 7f5927615640 1 --2- 172.21.15.70:0/3646089499 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f590c0a2520 0x7f590c0a49e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.331+0000 7f5927615640 1 --2- 172.21.15.70:0/3646089499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59201022a0 0x7f592019de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.331+0000 7f5927615640 1 --2- 172.21.15.70:0/3646089499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59201018e0 0x7f5920196e10 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.331+0000 7f5927615640 1 --2- 172.21.15.70:0/3646089499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59201006f0 0x7f59201968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.332+0000 7f5927615640 1 -- 172.21.15.70:0/3646089499 >> 172.21.15.70:0/3646089499 conn(0x7f59200fbc80 msgr2=0x7f5920104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:03.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.332+0000 7f5927615640 1 -- 172.21.15.70:0/3646089499 shutdown_connections 2024-04-01T01:33:03.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.332+0000 7f5927615640 1 -- 172.21.15.70:0/3646089499 wait complete. 2024-04-01T01:33:03.343 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:33:03.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.445+0000 7f792efb5640 1 -- 172.21.15.70:0/2458434279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7928102240 msgr2=0x7f7928108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.445+0000 7f792efb5640 1 --2- 172.21.15.70:0/2458434279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7928102240 0x7f7928108ad0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7918009900 tx=0x7f791802bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:03.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.445+0000 7f792efb5640 1 -- 172.21.15.70:0/2458434279 shutdown_connections 2024-04-01T01:33:03.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.445+0000 7f792efb5640 1 --2- 172.21.15.70:0/2458434279 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7928102240 0x7f7928108ad0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.445+0000 7f792efb5640 1 --2- 172.21.15.70:0/2458434279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7928101880 0x7f7928101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.445+0000 7f792efb5640 1 --2- 172.21.15.70:0/2458434279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7928100690 0x7f7928100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.445+0000 7f792efb5640 1 -- 172.21.15.70:0/2458434279 >> 172.21.15.70:0/2458434279 conn(0x7f79280fbc00 msgr2=0x7f79280fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:03.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.445+0000 7f792efb5640 1 -- 172.21.15.70:0/2458434279 shutdown_connections 2024-04-01T01:33:03.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.445+0000 7f792efb5640 1 -- 172.21.15.70:0/2458434279 wait complete. 2024-04-01T01:33:03.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.445+0000 7f792efb5640 1 Processor -- start 2024-04-01T01:33:03.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.445+0000 7f792efb5640 1 -- start start 2024-04-01T01:33:03.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.445+0000 7f792efb5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7928100690 0x7f792810b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:03.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.445+0000 7f792efb5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7928101880 0x7f7928109650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:03.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.445+0000 7f792efb5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7928102240 0x7f7928109b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:03.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.445+0000 7f792efb5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f792810d340 con 0x7f7928102240 2024-04-01T01:33:03.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f792efb5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f792810d1c0 con 0x7f7928101880 2024-04-01T01:33:03.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f792efb5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f792810d4c0 con 0x7f7928100690 2024-04-01T01:33:03.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f792d552640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7928102240 0x7f7928109b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:03.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f792d552640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7928102240 0x7f7928109b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39582/0 (socket says 172.21.15.70:39582) 2024-04-01T01:33:03.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f792d552640 1 -- 172.21.15.70:0/4081581941 learned_addr learned my addr 172.21.15.70:0/4081581941 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:03.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f7927fff640 1 --2- 172.21.15.70:0/4081581941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7928101880 0x7f7928109650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:03.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f792cd51640 1 --2- 172.21.15.70:0/4081581941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7928100690 0x7f792810b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:03.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f792d552640 1 -- 172.21.15.70:0/4081581941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7928100690 msgr2=0x7f792810b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f792d552640 1 --2- 172.21.15.70:0/4081581941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7928100690 0x7f792810b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f792d552640 1 -- 172.21.15.70:0/4081581941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7928101880 msgr2=0x7f7928109650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f792d552640 1 --2- 172.21.15.70:0/4081581941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7928101880 0x7f7928109650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f792d552640 1 -- 172.21.15.70:0/4081581941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792810a420 con 0x7f7928102240 2024-04-01T01:33:03.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f792d552640 1 --2- 172.21.15.70:0/4081581941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7928102240 0x7f7928109b90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7918036040 tx=0x7f7918005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:03.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f7925ffb640 1 -- 172.21.15.70:0/4081581941 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7918043020 con 0x7f7928102240 2024-04-01T01:33:03.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f7925ffb640 1 -- 172.21.15.70:0/4081581941 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7918004410 con 0x7f7928102240 2024-04-01T01:33:03.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f7925ffb640 1 -- 172.21.15.70:0/4081581941 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7918003b80 con 0x7f7928102240 2024-04-01T01:33:03.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f792efb5640 1 -- 172.21.15.70:0/4081581941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79281a1a70 con 0x7f7928102240 2024-04-01T01:33:03.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.446+0000 7f792efb5640 1 -- 172.21.15.70:0/4081581941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79281a2010 con 0x7f7928102240 2024-04-01T01:33:03.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.447+0000 7f792efb5640 1 -- 172.21.15.70:0/4081581941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79281079b0 con 0x7f7928102240 2024-04-01T01:33:03.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.448+0000 7f7925ffb640 1 -- 172.21.15.70:0/4081581941 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f7918046020 con 0x7f7928102240 2024-04-01T01:33:03.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.449+0000 7f7925ffb640 1 --2- 172.21.15.70:0/4081581941 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79140a2520 0x7f79140a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:03.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.449+0000 7f792cd51640 1 --2- 172.21.15.70:0/4081581941 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79140a2520 0x7f79140a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:03.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.449+0000 7f7925ffb640 1 -- 172.21.15.70:0/4081581941 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f79180f0bb0 con 0x7f7928102240 2024-04-01T01:33:03.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.449+0000 7f792cd51640 1 --2- 172.21.15.70:0/4081581941 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79140a2520 0x7f79140a49e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f79281016e0 tx=0x7f7920009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:03.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.453+0000 7f7925ffb640 1 -- 172.21.15.70:0/4081581941 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f791804d020 con 0x7f7928102240 2024-04-01T01:33:03.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.628+0000 7f792efb5640 1 -- 172.21.15.70:0/4081581941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7928062350 con 0x7f7928102240 2024-04-01T01:33:03.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.630+0000 7f7925ffb640 1 -- 172.21.15.70:0/4081581941 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24595 (secure 0 0 0) 0x7f79180bda10 con 0x7f7928102240 2024-04-01T01:33:03.630 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:03.630 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":91,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:32:19.680117+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:33:03.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.632+0000 7f792efb5640 1 -- 172.21.15.70:0/4081581941 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79140a2520 msgr2=0x7f79140a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.632+0000 7f792efb5640 1 --2- 172.21.15.70:0/4081581941 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79140a2520 0x7f79140a49e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f79281016e0 tx=0x7f7920009500 comp rx=0 tx=0).stop 2024-04-01T01:33:03.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.632+0000 7f792efb5640 1 -- 172.21.15.70:0/4081581941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7928102240 msgr2=0x7f7928109b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.632+0000 7f792efb5640 1 --2- 172.21.15.70:0/4081581941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7928102240 0x7f7928109b90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7918036040 tx=0x7f7918005dd0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.632+0000 7f792efb5640 1 -- 172.21.15.70:0/4081581941 shutdown_connections 2024-04-01T01:33:03.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.632+0000 7f792efb5640 1 --2- 172.21.15.70:0/4081581941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7928102240 0x7f7928109b90 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.632+0000 7f792efb5640 1 --2- 172.21.15.70:0/4081581941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7928101880 0x7f7928109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.632+0000 7f792efb5640 1 --2- 172.21.15.70:0/4081581941 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79140a2520 0x7f79140a49e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.632+0000 7f792efb5640 1 --2- 172.21.15.70:0/4081581941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7928100690 0x7f792810b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.632+0000 7f792efb5640 1 -- 172.21.15.70:0/4081581941 >> 172.21.15.70:0/4081581941 conn(0x7f79280fbc00 msgr2=0x7f7928104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:03.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.632+0000 7f792efb5640 1 -- 172.21.15.70:0/4081581941 shutdown_connections 2024-04-01T01:33:03.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.632+0000 7f792efb5640 1 -- 172.21.15.70:0/4081581941 wait complete. 2024-04-01T01:33:03.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:33:03.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.746+0000 7fa86c925640 1 -- 172.21.15.70:0/3674228970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa868101770 msgr2=0x7fa868101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.746+0000 7fa86c925640 1 --2- 172.21.15.70:0/3674228970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa868101770 0x7fa868101b90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa860009900 tx=0x7fa86002bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:03.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.746+0000 7fa86c925640 1 -- 172.21.15.70:0/3674228970 shutdown_connections 2024-04-01T01:33:03.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.746+0000 7fa86c925640 1 --2- 172.21.15.70:0/3674228970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa868103320 0x7fa868109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.746+0000 7fa86c925640 1 --2- 172.21.15.70:0/3674228970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa868102960 0x7fa868102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.746+0000 7fa86c925640 1 --2- 172.21.15.70:0/3674228970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa868101770 0x7fa868101b90 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.746+0000 7fa86c925640 1 -- 172.21.15.70:0/3674228970 >> 172.21.15.70:0/3674228970 conn(0x7fa8680fcd00 msgr2=0x7fa8680ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:03.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.746+0000 7fa86c925640 1 -- 172.21.15.70:0/3674228970 shutdown_connections 2024-04-01T01:33:03.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.746+0000 7fa86c925640 1 -- 172.21.15.70:0/3674228970 wait complete. 2024-04-01T01:33:03.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa86c925640 1 Processor -- start 2024-04-01T01:33:03.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa86c925640 1 -- start start 2024-04-01T01:33:03.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa86c925640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa868101770 0x7fa8681979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:03.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa86c925640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa868102960 0x7fa868197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:03.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa86c925640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa868103320 0x7fa86819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:03.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa86c925640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa86810d290 con 0x7fa868102960 2024-04-01T01:33:03.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa86c925640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa86810d110 con 0x7fa868103320 2024-04-01T01:33:03.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa86c925640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa86810d410 con 0x7fa868101770 2024-04-01T01:33:03.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa865d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa868102960 0x7fa868197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:03.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa865d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa868102960 0x7fa868197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39594/0 (socket says 172.21.15.70:39594) 2024-04-01T01:33:03.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa865d9b640 1 -- 172.21.15.70:0/3495912707 learned_addr learned my addr 172.21.15.70:0/3495912707 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:03.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa86659c640 1 --2- 172.21.15.70:0/3495912707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa868101770 0x7fa8681979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:03.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa866d9d640 1 --2- 172.21.15.70:0/3495912707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa868103320 0x7fa86819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:03.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa865d9b640 1 -- 172.21.15.70:0/3495912707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa868101770 msgr2=0x7fa8681979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa865d9b640 1 --2- 172.21.15.70:0/3495912707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa868101770 0x7fa8681979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa865d9b640 1 -- 172.21.15.70:0/3495912707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa868103320 msgr2=0x7fa86819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa865d9b640 1 --2- 172.21.15.70:0/3495912707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa868103320 0x7fa86819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa865d9b640 1 -- 172.21.15.70:0/3495912707 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86819f600 con 0x7fa868102960 2024-04-01T01:33:03.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.747+0000 7fa865d9b640 1 --2- 172.21.15.70:0/3495912707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa868102960 0x7fa868197f20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa85c00e9f0 tx=0x7fa85c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:03.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.748+0000 7fa8577fe640 1 -- 172.21.15.70:0/3495912707 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c00ccc0 con 0x7fa868102960 2024-04-01T01:33:03.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.748+0000 7fa86c925640 1 -- 172.21.15.70:0/3495912707 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8680739e0 con 0x7fa868102960 2024-04-01T01:33:03.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.748+0000 7fa8577fe640 1 -- 172.21.15.70:0/3495912707 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa85c00ce60 con 0x7fa868102960 2024-04-01T01:33:03.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.748+0000 7fa8577fe640 1 -- 172.21.15.70:0/3495912707 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c010430 con 0x7fa868102960 2024-04-01T01:33:03.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.748+0000 7fa86c925640 1 -- 172.21.15.70:0/3495912707 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa868073ee0 con 0x7fa868102960 2024-04-01T01:33:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.749+0000 7fa86c925640 1 -- 172.21.15.70:0/3495912707 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa868108b60 con 0x7fa868102960 2024-04-01T01:33:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.750+0000 7fa8577fe640 1 -- 172.21.15.70:0/3495912707 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fa85c005020 con 0x7fa868102960 2024-04-01T01:33:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.750+0000 7fa8577fe640 1 --2- 172.21.15.70:0/3495912707 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8500a24a0 0x7fa8500a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.750+0000 7fa86659c640 1 --2- 172.21.15.70:0/3495912707 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8500a24a0 0x7fa8500a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.750+0000 7fa8577fe640 1 -- 172.21.15.70:0/3495912707 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa85c0cee60 con 0x7fa868102960 2024-04-01T01:33:03.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.751+0000 7fa86659c640 1 --2- 172.21.15.70:0/3495912707 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8500a24a0 0x7fa8500a4960 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa860005ab0 tx=0x7fa860005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:03.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.754+0000 7fa8577fe640 1 -- 172.21.15.70:0/3495912707 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa85c09bcc0 con 0x7fa868102960 2024-04-01T01:33:03.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.964+0000 7fa86c925640 1 -- 172.21.15.70:0/3495912707 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa86802d170 con 0x7fa868102960 2024-04-01T01:33:03.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.965+0000 7fa8577fe640 1 -- 172.21.15.70:0/3495912707 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11929 (secure 0 0 0) 0x7fa85c09b5c0 con 0x7fa868102960 2024-04-01T01:33:03.965 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:03.965 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:31:38.563210+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24451,24464,24379,24404,24418]},"id":1}]} 2024-04-01T01:33:03.966 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 28 2024-04-01T01:33:03.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.967+0000 7fa86c925640 1 -- 172.21.15.70:0/3495912707 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8500a24a0 msgr2=0x7fa8500a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.967+0000 7fa86c925640 1 --2- 172.21.15.70:0/3495912707 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8500a24a0 0x7fa8500a4960 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa860005ab0 tx=0x7fa860005fd0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.967+0000 7fa86c925640 1 -- 172.21.15.70:0/3495912707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa868102960 msgr2=0x7fa868197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:03.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.967+0000 7fa86c925640 1 --2- 172.21.15.70:0/3495912707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa868102960 0x7fa868197f20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa85c00e9f0 tx=0x7fa85c00edf0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.967+0000 7fa86c925640 1 -- 172.21.15.70:0/3495912707 shutdown_connections 2024-04-01T01:33:03.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.967+0000 7fa86c925640 1 --2- 172.21.15.70:0/3495912707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa868103320 0x7fa86819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.967+0000 7fa86c925640 1 --2- 172.21.15.70:0/3495912707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa868102960 0x7fa868197f20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.967+0000 7fa86c925640 1 --2- 172.21.15.70:0/3495912707 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8500a24a0 0x7fa8500a4960 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.967+0000 7fa86c925640 1 --2- 172.21.15.70:0/3495912707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa868101770 0x7fa8681979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:03.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.967+0000 7fa86c925640 1 -- 172.21.15.70:0/3495912707 >> 172.21.15.70:0/3495912707 conn(0x7fa8680fcd00 msgr2=0x7fa868105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:03.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.967+0000 7fa86c925640 1 -- 172.21.15.70:0/3495912707 shutdown_connections 2024-04-01T01:33:03.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:03.967+0000 7fa86c925640 1 -- 172.21.15.70:0/3495912707 wait complete. 2024-04-01T01:33:03.977 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:03 smithi070 ceph-mon[32179]: pgmap v343: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-01T01:33:03.977 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:03 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/204231845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:03.978 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:03 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3353544063' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:03.978 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:03 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3646089499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:03.978 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:03 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/4081581941' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:03.978 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:03 smithi081 ceph-mon[35142]: pgmap v343: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-01T01:33:03.978 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:03 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/204231845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:03.978 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:03 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3353544063' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:03.978 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:03 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3646089499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:03.979 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:03 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/4081581941' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:03.980 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-01T01:31:15.732761+0000', 'modified': '2024-04-01T01:31:38.563210+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24464, 'mds_1': 24451, 'mds_2': 24418, 'mds_3': 24404, 'mds_4': 24379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14616': {'gid': 14616, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6839/1626804615', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6838', 'nonce': 1626804615}, {'type': 'v1', 'addr': '172.21.15.70:6839', 'nonce': 1626804615}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14628': {'gid': 14628, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6841/2380158677', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6840', 'nonce': 2380158677}, {'type': 'v1', 'addr': '172.21.15.70:6841', 'nonce': 2380158677}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24371': {'gid': 24371, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.81:6833/1246539452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6832', 'nonce': 1246539452}, {'type': 'v1', 'addr': '172.21.15.81:6833', 'nonce': 1246539452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24379': {'gid': 24379, 'name': 'l', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.189:6833/2161209704', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6832', 'nonce': 2161209704}, {'type': 'v1', 'addr': '172.21.15.189:6833', 'nonce': 2161209704}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24404': {'gid': 24404, 'name': 'e', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.81:6835/970944957', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 970944957}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 970944957}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24418': {'gid': 24418, 'name': 'f', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.189:6835/3861912911', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 3861912911}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 3861912911}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24425': {'gid': 24425, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.81:6837/3527001697', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3527001697}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3527001697}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24427': {'gid': 24427, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6837/2296387110', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6836', 'nonce': 2296387110}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 2296387110}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24451': {'gid': 24451, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.189:6839/2833086571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6838', 'nonce': 2833086571}, {'type': 'v1', 'addr': '172.21.15.189:6839', 'nonce': 2833086571}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24464': {'gid': 24464, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6839/2805044611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6838', 'nonce': 2805044611}, {'type': 'v1', 'addr': '172.21.15.81:6839', 'nonce': 2805044611}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24464, 'qdb_cluster': [24451, 24464, 24379, 24404, 24418]} 2024-04-01T01:33:03.980 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-01T01:33:03.980 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2024-04-01T01:33:03.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:03 smithi189 ceph-mon[35753]: pgmap v343: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-01T01:33:03.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:03 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/204231845' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:03.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:03 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3353544063' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:03.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:03 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3646089499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:03.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:03 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/4081581941' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:04.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.083+0000 7fd944055640 1 -- 172.21.15.70:0/499651070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c101770 msgr2=0x7fd93c101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.083+0000 7fd944055640 1 --2- 172.21.15.70:0/499651070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c101770 0x7fd93c101b90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd938009900 tx=0x7fd93802bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:04.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.083+0000 7fd944055640 1 -- 172.21.15.70:0/499651070 shutdown_connections 2024-04-01T01:33:04.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.083+0000 7fd944055640 1 --2- 172.21.15.70:0/499651070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd93c103320 0x7fd93c109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.083+0000 7fd944055640 1 --2- 172.21.15.70:0/499651070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd93c102960 0x7fd93c102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.083+0000 7fd944055640 1 --2- 172.21.15.70:0/499651070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c101770 0x7fd93c101b90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.083+0000 7fd944055640 1 -- 172.21.15.70:0/499651070 >> 172.21.15.70:0/499651070 conn(0x7fd93c0fcce0 msgr2=0x7fd93c0ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.083+0000 7fd944055640 1 -- 172.21.15.70:0/499651070 shutdown_connections 2024-04-01T01:33:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.083+0000 7fd944055640 1 -- 172.21.15.70:0/499651070 wait complete. 2024-04-01T01:33:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.083+0000 7fd944055640 1 Processor -- start 2024-04-01T01:33:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.083+0000 7fd944055640 1 -- start start 2024-04-01T01:33:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd944055640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c101770 0x7fd93c1979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd944055640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd93c102960 0x7fd93c197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd944055640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd93c103320 0x7fd93c19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd944055640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd93c10d270 con 0x7fd93c103320 2024-04-01T01:33:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd944055640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd93c10d0f0 con 0x7fd93c101770 2024-04-01T01:33:04.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd944055640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd93c10d3f0 con 0x7fd93c102960 2024-04-01T01:33:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd9425f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd93c103320 0x7fd93c19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd9425f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd93c103320 0x7fd93c19ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39600/0 (socket says 172.21.15.70:39600) 2024-04-01T01:33:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd9425f2640 1 -- 172.21.15.70:0/2001444468 learned_addr learned my addr 172.21.15.70:0/2001444468 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd941df1640 1 --2- 172.21.15.70:0/2001444468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c101770 0x7fd93c1979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd9415f0640 1 --2- 172.21.15.70:0/2001444468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd93c102960 0x7fd93c197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd9425f2640 1 -- 172.21.15.70:0/2001444468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd93c102960 msgr2=0x7fd93c197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd9425f2640 1 --2- 172.21.15.70:0/2001444468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd93c102960 0x7fd93c197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd9425f2640 1 -- 172.21.15.70:0/2001444468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c101770 msgr2=0x7fd93c1979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd9425f2640 1 --2- 172.21.15.70:0/2001444468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c101770 0x7fd93c1979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd9425f2640 1 -- 172.21.15.70:0/2001444468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd93c19f5e0 con 0x7fd93c103320 2024-04-01T01:33:04.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd9425f2640 1 --2- 172.21.15.70:0/2001444468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd93c103320 0x7fd93c19ef70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd93400eb10 tx=0x7fd93400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd932ffd640 1 -- 172.21.15.70:0/2001444468 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd934004b20 con 0x7fd93c103320 2024-04-01T01:33:04.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd944055640 1 -- 172.21.15.70:0/2001444468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd93c19f8d0 con 0x7fd93c103320 2024-04-01T01:33:04.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd932ffd640 1 -- 172.21.15.70:0/2001444468 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd934004cc0 con 0x7fd93c103320 2024-04-01T01:33:04.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd932ffd640 1 -- 172.21.15.70:0/2001444468 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd934010690 con 0x7fd93c103320 2024-04-01T01:33:04.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.084+0000 7fd944055640 1 -- 172.21.15.70:0/2001444468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd93c19fe20 con 0x7fd93c103320 2024-04-01T01:33:04.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.087+0000 7fd932ffd640 1 -- 172.21.15.70:0/2001444468 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fd934005020 con 0x7fd93c103320 2024-04-01T01:33:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.087+0000 7fd946c36740 1 Processor -- start 2024-04-01T01:33:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.087+0000 7fd946c36740 1 -- start start 2024-04-01T01:33:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd946c36740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c0610 0x55b3a48c0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd946c36740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c17e0 0x55b3a48c1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd946c36740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c21a0 0x55b3a48c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd946c36740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b3a4824af0 con 0x55b3a48c21a0 2024-04-01T01:33:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd946c36740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b3a47cd9d0 con 0x55b3a48c0610 2024-04-01T01:33:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd946c36740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b3a4820b90 con 0x55b3a48c17e0 2024-04-01T01:33:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd932ffd640 1 --2- 172.21.15.70:0/2001444468 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9280a24a0 0x7fd9280a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd9415f0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c0610 0x55b3a48c0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd9415f0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c0610 0x55b3a48c0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:52782/0 (socket says 172.21.15.70:52782) 2024-04-01T01:33:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd9415f0640 1 -- 172.21.15.70:0/480226989 learned_addr learned my addr 172.21.15.70:0/480226989 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd941df1640 1 --2- 172.21.15.70:0/480226989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c17e0 0x55b3a48c1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd9415f0640 1 --2- 172.21.15.70:0/480226989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c21a0 0x55b3a48c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd9425f2640 1 --2- 172.21.15.70:0/2001444468 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9280a24a0 0x7fd9280a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd932ffd640 1 -- 172.21.15.70:0/2001444468 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd9340cf070 con 0x7fd93c103320 2024-04-01T01:33:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd9415f0640 1 -- 172.21.15.70:0/480226989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c17e0 msgr2=0x55b3a48c1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd9415f0640 1 --2- 172.21.15.70:0/480226989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c17e0 0x55b3a48c1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd9415f0640 1 -- 172.21.15.70:0/480226989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c21a0 msgr2=0x55b3a48c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd9415f0640 1 --2- 172.21.15.70:0/480226989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c21a0 0x55b3a48c8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd9415f0640 1 -- 172.21.15.70:0/480226989 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3a45392d0 con 0x55b3a48c0610 2024-04-01T01:33:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd9425f2640 1 --2- 172.21.15.70:0/2001444468 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9280a24a0 0x7fd9280a4960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd9340a3c50 tx=0x7fd934090c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd9415f0640 1 --2- 172.21.15.70:0/480226989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c0610 0x55b3a48c0a30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd92c0058c0 tx=0x7fd92c005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44f5b5faab27173e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd927fff640 1 -- 172.21.15.70:0/480226989 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92c004030 con 0x55b3a48c0610 2024-04-01T01:33:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd927fff640 1 -- 172.21.15.70:0/480226989 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd92c005020 con 0x55b3a48c0610 2024-04-01T01:33:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.088+0000 7fd927fff640 1 -- 172.21.15.70:0/480226989 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92c004c60 con 0x55b3a48c0610 2024-04-01T01:33:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.089+0000 7fd946c36740 1 -- 172.21.15.70:0/480226989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c0610 msgr2=0x55b3a48c0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.089+0000 7fd946c36740 1 --2- 172.21.15.70:0/480226989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c0610 0x55b3a48c0a30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd92c0058c0 tx=0x7fd92c005850 comp rx=0 tx=0).stop 2024-04-01T01:33:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.089+0000 7fd946c36740 1 -- 172.21.15.70:0/480226989 shutdown_connections 2024-04-01T01:33:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.089+0000 7fd946c36740 1 --2- 172.21.15.70:0/480226989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c21a0 0x55b3a48c8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.089+0000 7fd946c36740 1 --2- 172.21.15.70:0/480226989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c17e0 0x55b3a48c1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.089+0000 7fd946c36740 1 --2- 172.21.15.70:0/480226989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c0610 0x55b3a48c0a30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.089+0000 7fd946c36740 1 -- 172.21.15.70:0/480226989 >> 172.21.15.70:0/480226989 conn(0x55b3a48bba00 msgr2=0x55b3a48bbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.089+0000 7fd946c36740 1 -- 172.21.15.70:0/480226989 shutdown_connections 2024-04-01T01:33:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.089+0000 7fd946c36740 1 -- 172.21.15.70:0/480226989 wait complete. 2024-04-01T01:33:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.089+0000 7fd946c36740 1 -- start start 2024-04-01T01:33:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd946c36740 1 Processor -- start 2024-04-01T01:33:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd946c36740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c0610 0x55b3a48caa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd946c36740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 0x55b3a48caf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd946c36740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c21a0 0x55b3a48d1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd946c36740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b3a48c91b0 con 0x55b3a48c0610 2024-04-01T01:33:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd946c36740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b3a47e0f70 con 0x55b3a48c17e0 2024-04-01T01:33:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd946c36740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b3a48c9330 con 0x55b3a48c21a0 2024-04-01T01:33:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd946c36740 10 client.0 _ll_register_callbacks cb 0x55b3a4538ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd9415f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c0610 0x55b3a48caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd941df1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 0x55b3a48caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd9415f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c0610 0x55b3a48caa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39610/0 (socket says 172.21.15.70:39610) 2024-04-01T01:33:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd9415f0640 1 -- 172.21.15.70:0/416703013 learned_addr learned my addr 172.21.15.70:0/416703013 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd9415f0640 1 --2- 172.21.15.70:0/416703013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c21a0 0x55b3a48d1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd941df1640 1 -- 172.21.15.70:0/416703013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c21a0 msgr2=0x55b3a48d1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd941df1640 1 --2- 172.21.15.70:0/416703013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c21a0 0x55b3a48d1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd941df1640 1 -- 172.21.15.70:0/416703013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c0610 msgr2=0x55b3a48caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd941df1640 1 --2- 172.21.15.70:0/416703013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c0610 0x55b3a48caa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd941df1640 1 -- 172.21.15.70:0/416703013 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3a46036b0 con 0x55b3a48c17e0 2024-04-01T01:33:04.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.090+0000 7fd9415f0640 1 --2- 172.21.15.70:0/416703013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c0610 0x55b3a48caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:04.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.091+0000 7fd941df1640 1 --2- 172.21.15.70:0/416703013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 0x55b3a48caf70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd938036040 tx=0x7fd938004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1e3b0f04f5ea89f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.091+0000 7fd9257fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:33:04.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.091+0000 7fd9257fa640 1 -- 172.21.15.70:0/416703013 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938005d50 con 0x55b3a48c17e0 2024-04-01T01:33:04.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.091+0000 7fd946c36740 1 -- 172.21.15.70:0/416703013 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd92c005020 con 0x55b3a48c17e0 2024-04-01T01:33:04.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.091+0000 7fd9257fa640 1 -- 172.21.15.70:0/416703013 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd938005030 con 0x55b3a48c17e0 2024-04-01T01:33:04.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.091+0000 7fd9257fa640 1 -- 172.21.15.70:0/416703013 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd938052050 con 0x55b3a48c17e0 2024-04-01T01:33:04.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.091+0000 7fd946c36740 10 client.25028 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:04.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.091+0000 7fd946c36740 1 -- 172.21.15.70:0/416703013 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b3a48d5050 con 0x55b3a48c17e0 2024-04-01T01:33:04.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.091+0000 7fd9257fa640 1 -- 172.21.15.70:0/416703013 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd92c005020 con 0x55b3a48c17e0 2024-04-01T01:33:04.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.091+0000 7fd9257fa640 1 -- 172.21.15.70:0/416703013 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd9380035f0 con 0x55b3a48c17e0 2024-04-01T01:33:04.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.091+0000 7fd946c36740 10 client.25028 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:04.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.091+0000 7fd946c36740 10 client.25028 resolve_mds: resolved name 'l' to daemon mds.l 2024-04-01T01:33:04.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.091+0000 7fd946c36740 20 client.25028 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:04.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.091+0000 7fd946c36740 1 --2- 172.21.15.70:0/416703013 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b3a48d76c0 0x55b3a48d8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.091+0000 7fd946c36740 4 client.25028 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:33:04.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.091+0000 7fd946c36740 1 -- 172.21.15.70:0/416703013 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3a4832030 con 0x55b3a48d76c0 2024-04-01T01:33:04.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.092+0000 7fd9415f0640 1 --2- 172.21.15.70:0/416703013 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b3a48d76c0 0x55b3a48d8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.092+0000 7fd9257fa640 10 client.25028 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:33:04.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.092+0000 7fd9415f0640 1 --2- 172.21.15.70:0/416703013 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b3a48d76c0 0x55b3a48d8b30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a033664a9c36628d server_cookie=89e9bcda7936bfab in_seq=0 out_seq=0 2024-04-01T01:33:04.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.093+0000 7fd9257fa640 1 -- 172.21.15.70:0/416703013 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b3a4832030 con 0x55b3a48d76c0 2024-04-01T01:33:04.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.093+0000 7fd9257fa640 10 client.25028 handle_command_reply: tid=0 2024-04-01T01:33:04.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.093+0000 7fd946c36740 1 client.25028 shutdown 2024-04-01T01:33:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.093+0000 7fd946c36740 1 -- 172.21.15.70:0/416703013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 msgr2=0x55b3a48caf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.093+0000 7fd946c36740 1 --2- 172.21.15.70:0/416703013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 0x55b3a48caf70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd938036040 tx=0x7fd938004270 comp rx=0 tx=0).stop 2024-04-01T01:33:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.093+0000 7fd946c36740 1 -- 172.21.15.70:0/416703013 shutdown_connections 2024-04-01T01:33:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.093+0000 7fd946c36740 1 --2- 172.21.15.70:0/416703013 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b3a48d76c0 0x55b3a48d8b30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.093+0000 7fd946c36740 1 --2- 172.21.15.70:0/416703013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c21a0 0x55b3a48d1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.093+0000 7fd9257fa640 0 client.25028 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:33:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.093+0000 7fd946c36740 1 --2- 172.21.15.70:0/416703013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 0x55b3a48caf70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.093+0000 7fd946c36740 1 --2- 172.21.15.70:0/416703013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c0610 0x55b3a48caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.093+0000 7fd946c36740 1 -- 172.21.15.70:0/416703013 >> 172.21.15.70:0/416703013 conn(0x55b3a48bba00 msgr2=0x55b3a48bab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.093+0000 7fd946c36740 1 -- 172.21.15.70:0/416703013 shutdown_connections 2024-04-01T01:33:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.093+0000 7fd946c36740 1 -- 172.21.15.70:0/416703013 wait complete. 2024-04-01T01:33:04.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.093+0000 7fd946c36740 20 client.25028 trim_cache size 0 max 16384 2024-04-01T01:33:04.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.110+0000 7fd946c36740 1 Processor -- start 2024-04-01T01:33:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd946c36740 1 -- start start 2024-04-01T01:33:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd946c36740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c0610 0x55b3a48c9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd946c36740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 0x55b3a48d1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd946c36740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c21a0 0x55b3a48c9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd946c36740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b3a48d6e70 con 0x55b3a48c0610 2024-04-01T01:33:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd946c36740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b3a48d7540 con 0x55b3a48c17e0 2024-04-01T01:33:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd946c36740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b3a48d7c10 con 0x55b3a48c21a0 2024-04-01T01:33:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd9415f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c0610 0x55b3a48c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd9415f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c0610 0x55b3a48c9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39624/0 (socket says 172.21.15.70:39624) 2024-04-01T01:33:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd9415f0640 1 -- 172.21.15.70:0/1234023423 learned_addr learned my addr 172.21.15.70:0/1234023423 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd941df1640 1 --2- 172.21.15.70:0/1234023423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 0x55b3a48d1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd9415f0640 1 --2- 172.21.15.70:0/1234023423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c21a0 0x55b3a48c9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd941df1640 1 -- 172.21.15.70:0/1234023423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c21a0 msgr2=0x55b3a48c9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd941df1640 1 --2- 172.21.15.70:0/1234023423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c21a0 0x55b3a48c9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd941df1640 1 -- 172.21.15.70:0/1234023423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c0610 msgr2=0x55b3a48c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd941df1640 1 --2- 172.21.15.70:0/1234023423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c0610 0x55b3a48c9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd941df1640 1 -- 172.21.15.70:0/1234023423 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3a48c1260 con 0x55b3a48c17e0 2024-04-01T01:33:04.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd9415f0640 1 --2- 172.21.15.70:0/1234023423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c0610 0x55b3a48c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:04.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd941df1640 1 --2- 172.21.15.70:0/1234023423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 0x55b3a48d1f70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd938003a60 tx=0x7fd9380042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=514ccef7ce1e15ae server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd9277fe640 1 -- 172.21.15.70:0/1234023423 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93805b070 con 0x55b3a48c17e0 2024-04-01T01:33:04.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd9277fe640 1 -- 172.21.15.70:0/1234023423 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd93802cc80 con 0x55b3a48c17e0 2024-04-01T01:33:04.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.111+0000 7fd9277fe640 1 -- 172.21.15.70:0/1234023423 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938000e60 con 0x55b3a48c17e0 2024-04-01T01:33:04.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.112+0000 7fd946c36740 1 -- 172.21.15.70:0/1234023423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 msgr2=0x55b3a48d1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.112+0000 7fd946c36740 1 --2- 172.21.15.70:0/1234023423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 0x55b3a48d1f70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd938003a60 tx=0x7fd9380042e0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.112+0000 7fd946c36740 1 -- 172.21.15.70:0/1234023423 shutdown_connections 2024-04-01T01:33:04.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.112+0000 7fd946c36740 1 --2- 172.21.15.70:0/1234023423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c21a0 0x55b3a48c9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.112+0000 7fd946c36740 1 --2- 172.21.15.70:0/1234023423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 0x55b3a48d1f70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.112+0000 7fd946c36740 1 --2- 172.21.15.70:0/1234023423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c0610 0x55b3a48c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.112+0000 7fd946c36740 1 -- 172.21.15.70:0/1234023423 >> 172.21.15.70:0/1234023423 conn(0x55b3a48bba00 msgr2=0x55b3a48bbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.112+0000 7fd946c36740 1 -- 172.21.15.70:0/1234023423 shutdown_connections 2024-04-01T01:33:04.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.112+0000 7fd946c36740 1 -- 172.21.15.70:0/1234023423 wait complete. 2024-04-01T01:33:04.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.112+0000 7fd946c36740 1 -- start start 2024-04-01T01:33:04.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.112+0000 7fd946c36740 1 Processor -- start 2024-04-01T01:33:04.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd946c36740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c0610 0x55b3a48cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd946c36740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 0x55b3a48cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd946c36740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c21a0 0x55b3a48d0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd946c36740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b3a48d89b0 con 0x55b3a48c21a0 2024-04-01T01:33:04.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd946c36740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b3a48d82e0 con 0x55b3a48c17e0 2024-04-01T01:33:04.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd946c36740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b3a48d9080 con 0x55b3a48c0610 2024-04-01T01:33:04.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd946c36740 10 client.0 _ll_register_callbacks cb 0x7fd938009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:04.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd9415f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c21a0 0x55b3a48d0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd9415f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c21a0 0x55b3a48d0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39632/0 (socket says 172.21.15.70:39632) 2024-04-01T01:33:04.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd9415f0640 1 -- 172.21.15.70:0/3909438652 learned_addr learned my addr 172.21.15.70:0/3909438652 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd9415f0640 1 --2- 172.21.15.70:0/3909438652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c0610 0x55b3a48cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd941df1640 1 --2- 172.21.15.70:0/3909438652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 0x55b3a48cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd9415f0640 1 -- 172.21.15.70:0/3909438652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c0610 msgr2=0x55b3a48cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd9415f0640 1 --2- 172.21.15.70:0/3909438652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c0610 0x55b3a48cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd9415f0640 1 -- 172.21.15.70:0/3909438652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 msgr2=0x55b3a48cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd9415f0640 1 --2- 172.21.15.70:0/3909438652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 0x55b3a48cde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd9415f0640 1 -- 172.21.15.70:0/3909438652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3a48d1420 con 0x55b3a48c21a0 2024-04-01T01:33:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd941df1640 1 --2- 172.21.15.70:0/3909438652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 0x55b3a48cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd9415f0640 1 --2- 172.21.15.70:0/3909438652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c21a0 0x55b3a48d0e30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd92c0057c0 tx=0x7fd92c009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e721b27911842 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd9257fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd9257fa640 1 -- 172.21.15.70:0/3909438652 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92c02f730 con 0x55b3a48c21a0 2024-04-01T01:33:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd9257fa640 1 -- 172.21.15.70:0/3909438652 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd92c011050 con 0x55b3a48c21a0 2024-04-01T01:33:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.113+0000 7fd946c36740 1 -- 172.21.15.70:0/3909438652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b3a482c380 con 0x55b3a48c21a0 2024-04-01T01:33:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.114+0000 7fd9257fa640 1 -- 172.21.15.70:0/3909438652 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd92c041050 con 0x55b3a48c21a0 2024-04-01T01:33:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.114+0000 7fd946c36740 10 client.15219 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.114+0000 7fd946c36740 1 -- 172.21.15.70:0/3909438652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b3a493d6e0 con 0x55b3a48c21a0 2024-04-01T01:33:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.114+0000 7fd9257fa640 1 -- 172.21.15.70:0/3909438652 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b3a482c380 con 0x55b3a48c21a0 2024-04-01T01:33:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.114+0000 7fd9257fa640 1 -- 172.21.15.70:0/3909438652 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd92c0035f0 con 0x55b3a48c21a0 2024-04-01T01:33:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.114+0000 7fd946c36740 10 client.15219 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.114+0000 7fd946c36740 10 client.15219 resolve_mds: resolved name 'l' to daemon mds.l 2024-04-01T01:33:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.114+0000 7fd946c36740 20 client.15219 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.114+0000 7fd946c36740 1 --2- 172.21.15.70:0/3909438652 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b3a4942130 0x55b3a4940e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.114+0000 7fd946c36740 4 client.15219 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-01T01:33:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.114+0000 7fd946c36740 1 -- 172.21.15.70:0/3909438652 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "status"}) -- 0x55b3a47aaa00 con 0x55b3a4942130 2024-04-01T01:33:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.114+0000 7fd941df1640 1 --2- 172.21.15.70:0/3909438652 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b3a4942130 0x55b3a4940e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd941df1640 1 --2- 172.21.15.70:0/3909438652 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b3a4942130 0x55b3a4940e40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=afebbf20fcf5e42e server_cookie=9ced74385f1eee58 in_seq=0 out_seq=0 2024-04-01T01:33:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd9257fa640 10 client.15219 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:33:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd9257fa640 1 -- 172.21.15.70:0/3909438652 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b3a47aaa00 con 0x55b3a4942130 2024-04-01T01:33:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd9257fa640 10 client.15219 handle_command_reply: tid=0 2024-04-01T01:33:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd946c36740 1 client.15219 shutdown 2024-04-01T01:33:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd946c36740 1 -- 172.21.15.70:0/3909438652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c21a0 msgr2=0x55b3a48d0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd946c36740 1 --2- 172.21.15.70:0/3909438652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c21a0 0x55b3a48d0e30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd92c0057c0 tx=0x7fd92c009fd0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd946c36740 1 -- 172.21.15.70:0/3909438652 shutdown_connections 2024-04-01T01:33:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd946c36740 1 --2- 172.21.15.70:0/3909438652 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b3a4942130 0x55b3a4940e40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd946c36740 1 --2- 172.21.15.70:0/3909438652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b3a48c21a0 0x55b3a48d0e30 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd946c36740 1 --2- 172.21.15.70:0/3909438652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b3a48c17e0 0x55b3a48cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd9257fa640 0 client.15219 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:33:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd946c36740 1 --2- 172.21.15.70:0/3909438652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3a48c0610 0x55b3a48cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd946c36740 1 -- 172.21.15.70:0/3909438652 >> 172.21.15.70:0/3909438652 conn(0x55b3a48bba00 msgr2=0x55b3a48c9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd946c36740 1 -- 172.21.15.70:0/3909438652 shutdown_connections 2024-04-01T01:33:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd946c36740 1 -- 172.21.15.70:0/3909438652 wait complete. 2024-04-01T01:33:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd946c36740 20 client.15219 trim_cache size 0 max 16384 2024-04-01T01:33:04.124 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:33:04.124 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "36577394-efc6-11ee-b647-cb9ed24678a4", 2024-04-01T01:33:04.124 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 4, 2024-04-01T01:33:04.124 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24379, 2024-04-01T01:33:04.124 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-01T01:33:04.124 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-01T01:33:04.124 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-01T01:33:04.124 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 86.554338337999994, 2024-04-01T01:33:04.124 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-04-01T01:33:04.124 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 91, 2024-04-01T01:33:04.124 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 89, 2024-04-01T01:33:04.124 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 144.252937229 2024-04-01T01:33:04.124 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:33:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd944055640 1 -- 172.21.15.70:0/2001444468 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9280a24a0 msgr2=0x7fd9280a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.115+0000 7fd944055640 1 --2- 172.21.15.70:0/2001444468 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9280a24a0 0x7fd9280a4960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd9340a3c50 tx=0x7fd934090c50 comp rx=0 tx=0).stop 2024-04-01T01:33:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.116+0000 7fd944055640 1 -- 172.21.15.70:0/2001444468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd93c103320 msgr2=0x7fd93c19ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.116+0000 7fd944055640 1 --2- 172.21.15.70:0/2001444468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd93c103320 0x7fd93c19ef70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd93400eb10 tx=0x7fd93400c2d0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.116+0000 7fd944055640 1 -- 172.21.15.70:0/2001444468 shutdown_connections 2024-04-01T01:33:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.116+0000 7fd944055640 1 --2- 172.21.15.70:0/2001444468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd93c103320 0x7fd93c19ef70 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.116+0000 7fd944055640 1 --2- 172.21.15.70:0/2001444468 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9280a24a0 0x7fd9280a4960 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.116+0000 7fd944055640 1 --2- 172.21.15.70:0/2001444468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd93c102960 0x7fd93c197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.116+0000 7fd944055640 1 --2- 172.21.15.70:0/2001444468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c101770 0x7fd93c1979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.116+0000 7fd944055640 1 -- 172.21.15.70:0/2001444468 >> 172.21.15.70:0/2001444468 conn(0x7fd93c0fcce0 msgr2=0x7fd93c105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.116+0000 7fd944055640 1 -- 172.21.15.70:0/2001444468 shutdown_connections 2024-04-01T01:33:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.116+0000 7fd944055640 1 -- 172.21.15.70:0/2001444468 wait complete. 2024-04-01T01:33:04.127 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-04-01T01:33:04.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.228+0000 7fc56c9e9640 1 -- 172.21.15.70:0/682484813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc568102220 msgr2=0x7fc568108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.228+0000 7fc56c9e9640 1 --2- 172.21.15.70:0/682484813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc568102220 0x7fc568108ab0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc560009be0 tx=0x7fc56002bef0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.229+0000 7fc56c9e9640 1 -- 172.21.15.70:0/682484813 shutdown_connections 2024-04-01T01:33:04.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.229+0000 7fc56c9e9640 1 --2- 172.21.15.70:0/682484813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc568102220 0x7fc568108ab0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.229+0000 7fc56c9e9640 1 --2- 172.21.15.70:0/682484813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc568101860 0x7fc568101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.229+0000 7fc56c9e9640 1 --2- 172.21.15.70:0/682484813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc568100670 0x7fc568100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.229+0000 7fc56c9e9640 1 -- 172.21.15.70:0/682484813 >> 172.21.15.70:0/682484813 conn(0x7fc5680fbc00 msgr2=0x7fc5680fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.229+0000 7fc56c9e9640 1 -- 172.21.15.70:0/682484813 shutdown_connections 2024-04-01T01:33:04.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.229+0000 7fc56c9e9640 1 -- 172.21.15.70:0/682484813 wait complete. 2024-04-01T01:33:04.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.229+0000 7fc56c9e9640 1 Processor -- start 2024-04-01T01:33:04.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.229+0000 7fc56c9e9640 1 -- start start 2024-04-01T01:33:04.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.229+0000 7fc56c9e9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc568100670 0x7fc568197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.229+0000 7fc56c9e9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc568101860 0x7fc568197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.229+0000 7fc56c9e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc568102220 0x7fc56819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.229+0000 7fc56c9e9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc56810c1b0 con 0x7fc568102220 2024-04-01T01:33:04.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.229+0000 7fc56c9e9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc56810c030 con 0x7fc568100670 2024-04-01T01:33:04.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.229+0000 7fc56c9e9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc56810c330 con 0x7fc568101860 2024-04-01T01:33:04.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc566d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc568102220 0x7fc56819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc566d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc568102220 0x7fc56819ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39650/0 (socket says 172.21.15.70:39650) 2024-04-01T01:33:04.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc566d9d640 1 -- 172.21.15.70:0/2159748153 learned_addr learned my addr 172.21.15.70:0/2159748153 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc56659c640 1 --2- 172.21.15.70:0/2159748153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc568100670 0x7fc568197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc565d9b640 1 --2- 172.21.15.70:0/2159748153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc568101860 0x7fc568197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc566d9d640 1 -- 172.21.15.70:0/2159748153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc568101860 msgr2=0x7fc568197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc566d9d640 1 --2- 172.21.15.70:0/2159748153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc568101860 0x7fc568197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc566d9d640 1 -- 172.21.15.70:0/2159748153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc568100670 msgr2=0x7fc568197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc566d9d640 1 --2- 172.21.15.70:0/2159748153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc568100670 0x7fc568197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc566d9d640 1 -- 172.21.15.70:0/2159748153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56819f590 con 0x7fc568102220 2024-04-01T01:33:04.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc56659c640 1 --2- 172.21.15.70:0/2159748153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc568100670 0x7fc568197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:04.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc566d9d640 1 --2- 172.21.15.70:0/2159748153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc568102220 0x7fc56819ef20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc560004af0 tx=0x7fc560033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc55f7fe640 1 -- 172.21.15.70:0/2159748153 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc560043020 con 0x7fc568102220 2024-04-01T01:33:04.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc56c9e9640 1 -- 172.21.15.70:0/2159748153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc56819f820 con 0x7fc568102220 2024-04-01T01:33:04.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc55f7fe640 1 -- 172.21.15.70:0/2159748153 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc56002ca60 con 0x7fc568102220 2024-04-01T01:33:04.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc55f7fe640 1 -- 172.21.15.70:0/2159748153 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc560034430 con 0x7fc568102220 2024-04-01T01:33:04.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.230+0000 7fc56c9e9640 1 -- 172.21.15.70:0/2159748153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc56819fdc0 con 0x7fc568102220 2024-04-01T01:33:04.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.232+0000 7fc55f7fe640 1 -- 172.21.15.70:0/2159748153 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fc560046020 con 0x7fc568102220 2024-04-01T01:33:04.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.233+0000 7fc56f5d3740 1 Processor -- start 2024-04-01T01:33:04.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.233+0000 7fc56f5d3740 1 -- start start 2024-04-01T01:33:04.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.233+0000 7fc56f5d3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647e610 0x55dbd647ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.233+0000 7fc56f5d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd647f7e0 0x55dbd647fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.233+0000 7fc56f5d3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd64801a0 0x55dbd6486a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.233+0000 7fc56f5d3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dbd63e2af0 con 0x55dbd647e610 2024-04-01T01:33:04.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.233+0000 7fc56f5d3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55dbd638b9d0 con 0x55dbd64801a0 2024-04-01T01:33:04.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.233+0000 7fc56f5d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dbd63deb90 con 0x55dbd647f7e0 2024-04-01T01:33:04.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.233+0000 7fc55f7fe640 1 --2- 172.21.15.70:0/2159748153 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5540a24a0 0x7fc5540a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc565d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647e610 0x55dbd647ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc565d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647e610 0x55dbd647ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39658/0 (socket says 172.21.15.70:39658) 2024-04-01T01:33:04.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc565d9b640 1 -- 172.21.15.70:0/288299677 learned_addr learned my addr 172.21.15.70:0/288299677 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc56659c640 1 --2- 172.21.15.70:0/288299677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd647f7e0 0x55dbd647fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc566d9d640 1 --2- 172.21.15.70:0/2159748153 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5540a24a0 0x7fc5540a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc565d9b640 1 --2- 172.21.15.70:0/288299677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd64801a0 0x55dbd6486a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc55f7fe640 1 -- 172.21.15.70:0/2159748153 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc5600f11b0 con 0x7fc568102220 2024-04-01T01:33:04.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc565d9b640 1 -- 172.21.15.70:0/288299677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd647f7e0 msgr2=0x55dbd647fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc565d9b640 1 --2- 172.21.15.70:0/288299677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd647f7e0 0x55dbd647fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc565d9b640 1 -- 172.21.15.70:0/288299677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd64801a0 msgr2=0x55dbd6486a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc565d9b640 1 --2- 172.21.15.70:0/288299677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd64801a0 0x55dbd6486a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc565d9b640 1 -- 172.21.15.70:0/288299677 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbd60f72d0 con 0x55dbd647e610 2024-04-01T01:33:04.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc56659c640 1 --2- 172.21.15.70:0/288299677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd647f7e0 0x55dbd647fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:04.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc565d9b640 1 --2- 172.21.15.70:0/288299677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647e610 0x55dbd647ea30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc558011680 tx=0x7fc558009950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd64bd254becb365 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc566d9d640 1 --2- 172.21.15.70:0/2159748153 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5540a24a0 0x7fc5540a4960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc5600c5d90 tx=0x7fc5600b2d90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc55cff9640 1 -- 172.21.15.70:0/288299677 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55801a070 con 0x55dbd647e610 2024-04-01T01:33:04.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc55cff9640 1 -- 172.21.15.70:0/288299677 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc558004250 con 0x55dbd647e610 2024-04-01T01:33:04.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.234+0000 7fc55cff9640 1 -- 172.21.15.70:0/288299677 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55800f940 con 0x55dbd647e610 2024-04-01T01:33:04.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.235+0000 7fc56f5d3740 1 -- 172.21.15.70:0/288299677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647e610 msgr2=0x55dbd647ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.235+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/288299677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647e610 0x55dbd647ea30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc558011680 tx=0x7fc558009950 comp rx=0 tx=0).stop 2024-04-01T01:33:04.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.235+0000 7fc56f5d3740 1 -- 172.21.15.70:0/288299677 shutdown_connections 2024-04-01T01:33:04.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.235+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/288299677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd64801a0 0x55dbd6486a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.235+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/288299677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd647f7e0 0x55dbd647fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.235+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/288299677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647e610 0x55dbd647ea30 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.235+0000 7fc56f5d3740 1 -- 172.21.15.70:0/288299677 >> 172.21.15.70:0/288299677 conn(0x55dbd6479a00 msgr2=0x55dbd6479e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.235+0000 7fc56f5d3740 1 -- 172.21.15.70:0/288299677 shutdown_connections 2024-04-01T01:33:04.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.235+0000 7fc56f5d3740 1 -- 172.21.15.70:0/288299677 wait complete. 2024-04-01T01:33:04.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.235+0000 7fc56f5d3740 1 -- start start 2024-04-01T01:33:04.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.235+0000 7fc56f5d3740 1 Processor -- start 2024-04-01T01:33:04.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc56f5d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd647e610 0x55dbd6488970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc56f5d3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd6488eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc56f5d3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd64801a0 0x55dbd648fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc56f5d3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dbd64871b0 con 0x55dbd647f7e0 2024-04-01T01:33:04.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc56f5d3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55dbd639ef70 con 0x55dbd64801a0 2024-04-01T01:33:04.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc56f5d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dbd6487330 con 0x55dbd647e610 2024-04-01T01:33:04.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc56f5d3740 10 client.0 _ll_register_callbacks cb 0x55dbd60f6ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:04.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc56659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd6488eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc565d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd647e610 0x55dbd6488970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc565d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd647e610 0x55dbd6488970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39738/0 (socket says 172.21.15.70:39738) 2024-04-01T01:33:04.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc565d9b640 1 -- 172.21.15.70:0/1229871808 learned_addr learned my addr 172.21.15.70:0/1229871808 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc56659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd6488eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39674/0 (socket says 172.21.15.70:39674) 2024-04-01T01:33:04.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc565d9b640 1 --2- 172.21.15.70:0/1229871808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd64801a0 0x55dbd648fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc565d9b640 1 -- 172.21.15.70:0/1229871808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd64801a0 msgr2=0x55dbd648fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc565d9b640 1 --2- 172.21.15.70:0/1229871808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd64801a0 0x55dbd648fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc565d9b640 1 -- 172.21.15.70:0/1229871808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 msgr2=0x55dbd6488eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc565d9b640 1 --2- 172.21.15.70:0/1229871808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd6488eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc565d9b640 1 -- 172.21.15.70:0/1229871808 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbd61c16b0 con 0x55dbd647e610 2024-04-01T01:33:04.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc56659c640 1 --2- 172.21.15.70:0/1229871808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd6488eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:04.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc565d9b640 1 --2- 172.21.15.70:0/1229871808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd647e610 0x55dbd6488970 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc5580044c0 tx=0x7fc558009c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e0a81284f21d225 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc54dffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:33:04.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.236+0000 7fc54dffb640 1 -- 172.21.15.70:0/1229871808 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55801a070 con 0x55dbd647e610 2024-04-01T01:33:04.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.237+0000 7fc56f5d3740 1 -- 172.21.15.70:0/1229871808 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc558004250 con 0x55dbd647e610 2024-04-01T01:33:04.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.237+0000 7fc54dffb640 1 -- 172.21.15.70:0/1229871808 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc55800f720 con 0x55dbd647e610 2024-04-01T01:33:04.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.237+0000 7fc54dffb640 1 -- 172.21.15.70:0/1229871808 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc558028040 con 0x55dbd647e610 2024-04-01T01:33:04.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.237+0000 7fc56f5d3740 10 client.24970 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:04.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.237+0000 7fc56f5d3740 1 -- 172.21.15.70:0/1229871808 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dbd6492eb0 con 0x55dbd647e610 2024-04-01T01:33:04.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.237+0000 7fc54dffb640 1 -- 172.21.15.70:0/1229871808 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc558004250 con 0x55dbd647e610 2024-04-01T01:33:04.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.237+0000 7fc54dffb640 1 -- 172.21.15.70:0/1229871808 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc55801f7b0 con 0x55dbd647e610 2024-04-01T01:33:04.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.237+0000 7fc56f5d3740 10 client.24970 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:04.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.237+0000 7fc56f5d3740 10 client.24970 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-01T01:33:04.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.237+0000 7fc56f5d3740 20 client.24970 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:04.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.237+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/1229871808 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55dbd6495350 0x55dbd64967c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.237+0000 7fc56f5d3740 4 client.24970 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:33:04.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.237+0000 7fc56f5d3740 1 -- 172.21.15.70:0/1229871808 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbd63f0030 con 0x55dbd6495350 2024-04-01T01:33:04.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.237+0000 7fc56659c640 1 --2- 172.21.15.70:0/1229871808 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55dbd6495350 0x55dbd64967c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.238+0000 7fc56659c640 1 --2- 172.21.15.70:0/1229871808 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55dbd6495350 0x55dbd64967c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b8c7da0d2e5cbb22 server_cookie=b3aad347d981b04c in_seq=0 out_seq=0 2024-04-01T01:33:04.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.238+0000 7fc54dffb640 10 client.24970 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:33:04.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.238+0000 7fc54dffb640 1 -- 172.21.15.70:0/1229871808 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dbd63f0030 con 0x55dbd6495350 2024-04-01T01:33:04.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.238+0000 7fc54dffb640 10 client.24970 handle_command_reply: tid=0 2024-04-01T01:33:04.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.238+0000 7fc56f5d3740 1 client.24970 shutdown 2024-04-01T01:33:04.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.238+0000 7fc56f5d3740 1 -- 172.21.15.70:0/1229871808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd647e610 msgr2=0x55dbd6488970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.238+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/1229871808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd647e610 0x55dbd6488970 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc5580044c0 tx=0x7fc558009c90 comp rx=0 tx=0).stop 2024-04-01T01:33:04.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.239+0000 7fc56f5d3740 1 -- 172.21.15.70:0/1229871808 shutdown_connections 2024-04-01T01:33:04.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.239+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/1229871808 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55dbd6495350 0x55dbd64967c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.239+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/1229871808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd64801a0 0x55dbd648fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.239+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/1229871808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd6488eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.239+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/1229871808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd647e610 0x55dbd6488970 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.239+0000 7fc54dffb640 0 client.24970 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:33:04.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.239+0000 7fc56f5d3740 1 -- 172.21.15.70:0/1229871808 >> 172.21.15.70:0/1229871808 conn(0x55dbd6479a00 msgr2=0x55dbd6478b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.239+0000 7fc56f5d3740 1 -- 172.21.15.70:0/1229871808 shutdown_connections 2024-04-01T01:33:04.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.239+0000 7fc56f5d3740 1 -- 172.21.15.70:0/1229871808 wait complete. 2024-04-01T01:33:04.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.239+0000 7fc56f5d3740 20 client.24970 trim_cache size 0 max 16384 2024-04-01T01:33:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.256+0000 7fc56f5d3740 1 Processor -- start 2024-04-01T01:33:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.256+0000 7fc56f5d3740 1 -- start start 2024-04-01T01:33:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.256+0000 7fc56f5d3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd647e610 0x55dbd6487760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.256+0000 7fc56f5d3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd6487ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.256+0000 7fc56f5d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd64801a0 0x55dbd6488580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.256+0000 7fc56f5d3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dbd6494b00 con 0x55dbd647f7e0 2024-04-01T01:33:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc56f5d3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55dbd64951d0 con 0x55dbd647e610 2024-04-01T01:33:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc56f5d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dbd64958a0 con 0x55dbd64801a0 2024-04-01T01:33:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc56659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd6487ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc565d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd647e610 0x55dbd6487760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc56659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd6487ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39690/0 (socket says 172.21.15.70:39690) 2024-04-01T01:33:04.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc565d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd647e610 0x55dbd6487760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:52870/0 (socket says 172.21.15.70:52870) 2024-04-01T01:33:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc56659c640 1 -- 172.21.15.70:0/1021369171 learned_addr learned my addr 172.21.15.70:0/1021369171 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc565d9b640 1 --2- 172.21.15.70:0/1021369171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd64801a0 0x55dbd6488580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc56659c640 1 -- 172.21.15.70:0/1021369171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd64801a0 msgr2=0x55dbd6488580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc56659c640 1 --2- 172.21.15.70:0/1021369171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd64801a0 0x55dbd6488580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc56659c640 1 -- 172.21.15.70:0/1021369171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd647e610 msgr2=0x55dbd6487760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc56659c640 1 --2- 172.21.15.70:0/1021369171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd647e610 0x55dbd6487760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc56659c640 1 -- 172.21.15.70:0/1021369171 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbd647a770 con 0x55dbd647f7e0 2024-04-01T01:33:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc565d9b640 1 --2- 172.21.15.70:0/1021369171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd647e610 0x55dbd6487760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc56659c640 1 --2- 172.21.15.70:0/1021369171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd6487ca0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc550005ec0 tx=0x7fc550009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f1703ea43e3bfa server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc54ffff640 1 -- 172.21.15.70:0/1021369171 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55000fad0 con 0x55dbd647f7e0 2024-04-01T01:33:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc54ffff640 1 -- 172.21.15.70:0/1021369171 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc55000fc70 con 0x55dbd647f7e0 2024-04-01T01:33:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.257+0000 7fc54ffff640 1 -- 172.21.15.70:0/1021369171 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc550022750 con 0x55dbd647f7e0 2024-04-01T01:33:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 -- 172.21.15.70:0/1021369171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 msgr2=0x55dbd6487ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/1021369171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd6487ca0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc550005ec0 tx=0x7fc550009260 comp rx=0 tx=0).stop 2024-04-01T01:33:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 -- 172.21.15.70:0/1021369171 shutdown_connections 2024-04-01T01:33:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/1021369171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd64801a0 0x55dbd6488580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/1021369171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd6487ca0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/1021369171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd647e610 0x55dbd6487760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 -- 172.21.15.70:0/1021369171 >> 172.21.15.70:0/1021369171 conn(0x55dbd6479a00 msgr2=0x55dbd6486750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 -- 172.21.15.70:0/1021369171 shutdown_connections 2024-04-01T01:33:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 -- 172.21.15.70:0/1021369171 wait complete. 2024-04-01T01:33:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 -- start start 2024-04-01T01:33:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 Processor -- start 2024-04-01T01:33:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd647e610 0x55dbd648f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd648f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd64801a0 0x55dbd64fd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dbd6496640 con 0x55dbd647f7e0 2024-04-01T01:33:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55dbd6495f70 con 0x55dbd647e610 2024-04-01T01:33:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dbd6496d10 con 0x55dbd64801a0 2024-04-01T01:33:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56f5d3740 10 client.0 _ll_register_callbacks cb 0x7fc550009b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd648f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd648f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39692/0 (socket says 172.21.15.70:39692) 2024-04-01T01:33:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc56659c640 1 -- 172.21.15.70:0/3673613946 learned_addr learned my addr 172.21.15.70:0/3673613946 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.258+0000 7fc565d9b640 1 --2- 172.21.15.70:0/3673613946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd64801a0 0x55dbd64fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.259+0000 7fc565d9b640 1 --2- 172.21.15.70:0/3673613946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd647e610 0x55dbd648f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.259+0000 7fc56659c640 1 -- 172.21.15.70:0/3673613946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd64801a0 msgr2=0x55dbd64fd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.259+0000 7fc56659c640 1 --2- 172.21.15.70:0/3673613946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd64801a0 0x55dbd64fd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.259+0000 7fc56659c640 1 -- 172.21.15.70:0/3673613946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd647e610 msgr2=0x55dbd648f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.259+0000 7fc56659c640 1 --2- 172.21.15.70:0/3673613946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd647e610 0x55dbd648f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.259+0000 7fc56659c640 1 -- 172.21.15.70:0/3673613946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbd64fd860 con 0x55dbd647f7e0 2024-04-01T01:33:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.259+0000 7fc56659c640 1 --2- 172.21.15.70:0/3673613946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd648f680 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc550012a30 tx=0x7fc550012870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e4a140b49ac609b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.259+0000 7fc54dffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.259+0000 7fc54dffb640 1 -- 172.21.15.70:0/3673613946 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55002f070 con 0x55dbd647f7e0 2024-04-01T01:33:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.259+0000 7fc56f5d3740 1 -- 172.21.15.70:0/3673613946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc55000fc70 con 0x55dbd647f7e0 2024-04-01T01:33:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.259+0000 7fc54dffb640 1 -- 172.21.15.70:0/3673613946 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc5500224d0 con 0x55dbd647f7e0 2024-04-01T01:33:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.259+0000 7fc54dffb640 1 -- 172.21.15.70:0/3673613946 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc550022110 con 0x55dbd647f7e0 2024-04-01T01:33:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.259+0000 7fc56f5d3740 10 client.15237 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.259+0000 7fc56f5d3740 1 -- 172.21.15.70:0/3673613946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dbd64feb60 con 0x55dbd647f7e0 2024-04-01T01:33:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.259+0000 7fc54dffb640 1 -- 172.21.15.70:0/3673613946 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc55000fc70 con 0x55dbd647f7e0 2024-04-01T01:33:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.259+0000 7fc54dffb640 1 -- 172.21.15.70:0/3673613946 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fc550004730 con 0x55dbd647f7e0 2024-04-01T01:33:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 10 client.15237 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 10 client.15237 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-01T01:33:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 20 client.15237 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/3673613946 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55dbd65033e0 0x55dbd65020f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 4 client.15237 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-01T01:33:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 1 -- 172.21.15.70:0/3673613946 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "status"}) -- 0x55dbd63ea380 con 0x55dbd65033e0 2024-04-01T01:33:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc565d9b640 1 --2- 172.21.15.70:0/3673613946 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55dbd65033e0 0x55dbd65020f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc54dffb640 10 client.15237 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:33:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc565d9b640 1 --2- 172.21.15.70:0/3673613946 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55dbd65033e0 0x55dbd65020f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f5fd66eb46ee1109 server_cookie=64da1abb42bb1abf in_seq=0 out_seq=0 2024-04-01T01:33:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc54dffb640 1 -- 172.21.15.70:0/3673613946 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55dbd63ea380 con 0x55dbd65033e0 2024-04-01T01:33:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc54dffb640 10 client.15237 handle_command_reply: tid=0 2024-04-01T01:33:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 1 client.15237 shutdown 2024-04-01T01:33:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 1 -- 172.21.15.70:0/3673613946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 msgr2=0x55dbd648f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/3673613946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd648f680 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc550012a30 tx=0x7fc550012870 comp rx=0 tx=0).stop 2024-04-01T01:33:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 1 -- 172.21.15.70:0/3673613946 shutdown_connections 2024-04-01T01:33:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/3673613946 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55dbd65033e0 0x55dbd65020f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/3673613946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd64801a0 0x55dbd64fd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc54dffb640 0 client.15237 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:33:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/3673613946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd647f7e0 0x55dbd648f680 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 1 --2- 172.21.15.70:0/3673613946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd647e610 0x55dbd648f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 1 -- 172.21.15.70:0/3673613946 >> 172.21.15.70:0/3673613946 conn(0x55dbd6479a00 msgr2=0x55dbd6487760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 1 -- 172.21.15.70:0/3673613946 shutdown_connections 2024-04-01T01:33:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.260+0000 7fc56f5d3740 1 -- 172.21.15.70:0/3673613946 wait complete. 2024-04-01T01:33:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.261+0000 7fc56f5d3740 20 client.15237 trim_cache size 0 max 16384 2024-04-01T01:33:04.270 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:33:04.270 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "36577394-efc6-11ee-b647-cb9ed24678a4", 2024-04-01T01:33:04.270 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 3, 2024-04-01T01:33:04.270 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24404, 2024-04-01T01:33:04.270 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-01T01:33:04.270 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-01T01:33:04.270 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-01T01:33:04.270 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 87.711669002999997, 2024-04-01T01:33:04.270 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-04-01T01:33:04.270 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 91, 2024-04-01T01:33:04.271 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 89, 2024-04-01T01:33:04.271 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 132.91111126199999 2024-04-01T01:33:04.271 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:33:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.261+0000 7fc56c9e9640 1 -- 172.21.15.70:0/2159748153 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5540a24a0 msgr2=0x7fc5540a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.261+0000 7fc56c9e9640 1 --2- 172.21.15.70:0/2159748153 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5540a24a0 0x7fc5540a4960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc5600c5d90 tx=0x7fc5600b2d90 comp rx=0 tx=0).stop 2024-04-01T01:33:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.261+0000 7fc56c9e9640 1 -- 172.21.15.70:0/2159748153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc568102220 msgr2=0x7fc56819ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.261+0000 7fc56c9e9640 1 --2- 172.21.15.70:0/2159748153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc568102220 0x7fc56819ef20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc560004af0 tx=0x7fc560033c90 comp rx=0 tx=0).stop 2024-04-01T01:33:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.261+0000 7fc56c9e9640 1 -- 172.21.15.70:0/2159748153 shutdown_connections 2024-04-01T01:33:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.261+0000 7fc56c9e9640 1 --2- 172.21.15.70:0/2159748153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc568102220 0x7fc56819ef20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.261+0000 7fc56c9e9640 1 --2- 172.21.15.70:0/2159748153 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5540a24a0 0x7fc5540a4960 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.261+0000 7fc56c9e9640 1 --2- 172.21.15.70:0/2159748153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc568101860 0x7fc568197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.261+0000 7fc56c9e9640 1 --2- 172.21.15.70:0/2159748153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc568100670 0x7fc568197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.261+0000 7fc56c9e9640 1 -- 172.21.15.70:0/2159748153 >> 172.21.15.70:0/2159748153 conn(0x7fc5680fbc00 msgr2=0x7fc568069d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.261+0000 7fc56c9e9640 1 -- 172.21.15.70:0/2159748153 shutdown_connections 2024-04-01T01:33:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.261+0000 7fc56c9e9640 1 -- 172.21.15.70:0/2159748153 wait complete. 2024-04-01T01:33:04.274 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-01T01:33:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.376+0000 7f37560fb640 1 -- 172.21.15.70:0/3745716837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3750102270 msgr2=0x7f3750104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.376+0000 7f37560fb640 1 --2- 172.21.15.70:0/3745716837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3750102270 0x7f3750104660 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3738009900 tx=0x7f373802bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.376+0000 7f37560fb640 1 -- 172.21.15.70:0/3745716837 shutdown_connections 2024-04-01T01:33:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.376+0000 7f37560fb640 1 --2- 172.21.15.70:0/3745716837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37501074d0 0x7f37501098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.376+0000 7f37560fb640 1 --2- 172.21.15.70:0/3745716837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3750104ba0 0x7f3750106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.376+0000 7f37560fb640 1 --2- 172.21.15.70:0/3745716837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3750102270 0x7f3750104660 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.376+0000 7f37560fb640 1 -- 172.21.15.70:0/3745716837 >> 172.21.15.70:0/3745716837 conn(0x7f37500fbbe0 msgr2=0x7f37500fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.376+0000 7f37560fb640 1 -- 172.21.15.70:0/3745716837 shutdown_connections 2024-04-01T01:33:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.376+0000 7f37560fb640 1 -- 172.21.15.70:0/3745716837 wait complete. 2024-04-01T01:33:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.376+0000 7f37560fb640 1 Processor -- start 2024-04-01T01:33:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.376+0000 7f37560fb640 1 -- start start 2024-04-01T01:33:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f37560fb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3750102270 0x7f3750197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f37560fb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3750104ba0 0x7f3750197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f37560fb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37501074d0 0x7f375019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f37560fb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f375010d290 con 0x7f3750102270 2024-04-01T01:33:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f37560fb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f375010d110 con 0x7f37501074d0 2024-04-01T01:33:04.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f37560fb640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f375010d410 con 0x7f3750104ba0 2024-04-01T01:33:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f374f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3750102270 0x7f3750197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f374f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3750102270 0x7f3750197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39718/0 (socket says 172.21.15.70:39718) 2024-04-01T01:33:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f374f7fe640 1 -- 172.21.15.70:0/918915083 learned_addr learned my addr 172.21.15.70:0/918915083 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f374effd640 1 --2- 172.21.15.70:0/918915083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3750104ba0 0x7f3750197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f374ffff640 1 --2- 172.21.15.70:0/918915083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37501074d0 0x7f375019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f374f7fe640 1 -- 172.21.15.70:0/918915083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3750104ba0 msgr2=0x7f3750197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f374f7fe640 1 --2- 172.21.15.70:0/918915083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3750104ba0 0x7f3750197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f374f7fe640 1 -- 172.21.15.70:0/918915083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37501074d0 msgr2=0x7f375019efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f374f7fe640 1 --2- 172.21.15.70:0/918915083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37501074d0 0x7f375019efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f374f7fe640 1 -- 172.21.15.70:0/918915083 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f375019f640 con 0x7f3750102270 2024-04-01T01:33:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f374f7fe640 1 --2- 172.21.15.70:0/918915083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3750102270 0x7f3750197a20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3738006010 tx=0x7f3738005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f374cff9640 1 -- 172.21.15.70:0/918915083 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3738043020 con 0x7f3750102270 2024-04-01T01:33:04.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f374cff9640 1 -- 172.21.15.70:0/918915083 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37380043d0 con 0x7f3750102270 2024-04-01T01:33:04.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f37560fb640 1 -- 172.21.15.70:0/918915083 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f375019f8d0 con 0x7f3750102270 2024-04-01T01:33:04.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f374cff9640 1 -- 172.21.15.70:0/918915083 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3738003b80 con 0x7f3750102270 2024-04-01T01:33:04.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.377+0000 7f37560fb640 1 -- 172.21.15.70:0/918915083 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f375019fe70 con 0x7f3750102270 2024-04-01T01:33:04.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.379+0000 7f374cff9640 1 -- 172.21.15.70:0/918915083 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f3738046020 con 0x7f3750102270 2024-04-01T01:33:04.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f3758dbc740 1 Processor -- start 2024-04-01T01:33:04.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f3758dbc740 1 -- start start 2024-04-01T01:33:04.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f3758dbc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79d610 0x560b4b79da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f3758dbc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79e7e0 0x560b4b79ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f3758dbc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79f1a0 0x560b4b7a5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f3758dbc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560b4b701af0 con 0x560b4b79e7e0 2024-04-01T01:33:04.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f3758dbc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560b4b6aa9d0 con 0x560b4b79d610 2024-04-01T01:33:04.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f3758dbc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560b4b6fdb90 con 0x560b4b79f1a0 2024-04-01T01:33:04.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f374f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79e7e0 0x560b4b79ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f374f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79e7e0 0x560b4b79ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39720/0 (socket says 172.21.15.70:39720) 2024-04-01T01:33:04.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f374f7fe640 1 -- 172.21.15.70:0/355429682 learned_addr learned my addr 172.21.15.70:0/355429682 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f374effd640 1 --2- 172.21.15.70:0/355429682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79f1a0 0x560b4b7a5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f374ffff640 1 --2- 172.21.15.70:0/355429682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79d610 0x560b4b79da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f374cff9640 1 --2- 172.21.15.70:0/918915083 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f373c0a2520 0x7f373c0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f374ffff640 1 --2- 172.21.15.70:0/918915083 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f373c0a2520 0x7f373c0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f374f7fe640 1 -- 172.21.15.70:0/355429682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79f1a0 msgr2=0x560b4b7a5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f374f7fe640 1 --2- 172.21.15.70:0/355429682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79f1a0 0x560b4b7a5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f374f7fe640 1 -- 172.21.15.70:0/355429682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79d610 msgr2=0x560b4b79da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f374f7fe640 1 --2- 172.21.15.70:0/355429682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79d610 0x560b4b79da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f374f7fe640 1 -- 172.21.15.70:0/355429682 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b4b4162d0 con 0x560b4b79e7e0 2024-04-01T01:33:04.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f374effd640 1 --2- 172.21.15.70:0/355429682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79f1a0 0x560b4b7a5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:04.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.380+0000 7f374cff9640 1 -- 172.21.15.70:0/918915083 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f37380f0bb0 con 0x7f3750102270 2024-04-01T01:33:04.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.381+0000 7f374f7fe640 1 --2- 172.21.15.70:0/355429682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79e7e0 0x560b4b79ec60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f37380c57f0 tx=0x7f373802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e125e7377acecc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.381+0000 7f374ffff640 1 --2- 172.21.15.70:0/918915083 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f373c0a2520 0x7f373c0a49e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x560b4b79e6e0 tx=0x7f3748004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.381+0000 7f3745ffb640 1 -- 172.21.15.70:0/355429682 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37380bcc70 con 0x560b4b79e7e0 2024-04-01T01:33:04.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.381+0000 7f3745ffb640 1 -- 172.21.15.70:0/355429682 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37380bda10 con 0x560b4b79e7e0 2024-04-01T01:33:04.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.381+0000 7f3745ffb640 1 -- 172.21.15.70:0/355429682 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37380c5d20 con 0x560b4b79e7e0 2024-04-01T01:33:04.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.381+0000 7f3758dbc740 1 -- 172.21.15.70:0/355429682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79e7e0 msgr2=0x560b4b79ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.381+0000 7f3758dbc740 1 --2- 172.21.15.70:0/355429682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79e7e0 0x560b4b79ec60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f37380c57f0 tx=0x7f373802bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:04.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.381+0000 7f3758dbc740 1 -- 172.21.15.70:0/355429682 shutdown_connections 2024-04-01T01:33:04.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.381+0000 7f3758dbc740 1 --2- 172.21.15.70:0/355429682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79f1a0 0x560b4b7a5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.381+0000 7f3758dbc740 1 --2- 172.21.15.70:0/355429682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79e7e0 0x560b4b79ec60 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.381+0000 7f3758dbc740 1 --2- 172.21.15.70:0/355429682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79d610 0x560b4b79da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.381+0000 7f3758dbc740 1 -- 172.21.15.70:0/355429682 >> 172.21.15.70:0/355429682 conn(0x560b4b798a00 msgr2=0x560b4b798e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.381+0000 7f3758dbc740 1 -- 172.21.15.70:0/355429682 shutdown_connections 2024-04-01T01:33:04.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.381+0000 7f3758dbc740 1 -- 172.21.15.70:0/355429682 wait complete. 2024-04-01T01:33:04.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.382+0000 7f3758dbc740 1 -- start start 2024-04-01T01:33:04.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.382+0000 7f3758dbc740 1 Processor -- start 2024-04-01T01:33:04.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.382+0000 7f3758dbc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79d610 0x560b4b7a7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.382+0000 7f3758dbc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79e7e0 0x560b4b7a7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.382+0000 7f3758dbc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79f1a0 0x560b4b7aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.382+0000 7f3758dbc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560b4b7a5f70 con 0x560b4b79d610 2024-04-01T01:33:04.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.382+0000 7f3758dbc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560b4b6bdf70 con 0x560b4b79f1a0 2024-04-01T01:33:04.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.382+0000 7f3758dbc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560b4b7a60f0 con 0x560b4b79e7e0 2024-04-01T01:33:04.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.382+0000 7f3758dbc740 10 client.0 _ll_register_callbacks cb 0x560b4b415ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:04.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.382+0000 7f374f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79d610 0x560b4b7a7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.382+0000 7f374f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79d610 0x560b4b7a7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39736/0 (socket says 172.21.15.70:39736) 2024-04-01T01:33:04.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.382+0000 7f374f7fe640 1 -- 172.21.15.70:0/4024550920 learned_addr learned my addr 172.21.15.70:0/4024550920 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.382+0000 7f374effd640 1 --2- 172.21.15.70:0/4024550920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79e7e0 0x560b4b7a7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.382+0000 7f374ffff640 1 --2- 172.21.15.70:0/4024550920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79f1a0 0x560b4b7aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f374f7fe640 1 -- 172.21.15.70:0/4024550920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79e7e0 msgr2=0x560b4b7a7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f374f7fe640 1 --2- 172.21.15.70:0/4024550920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79e7e0 0x560b4b7a7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f374f7fe640 1 -- 172.21.15.70:0/4024550920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79f1a0 msgr2=0x560b4b7aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f374f7fe640 1 --2- 172.21.15.70:0/4024550920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79f1a0 0x560b4b7aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f374f7fe640 1 -- 172.21.15.70:0/4024550920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b4b4e06b0 con 0x560b4b79d610 2024-04-01T01:33:04.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f374f7fe640 1 --2- 172.21.15.70:0/4024550920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79d610 0x560b4b7a7600 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f37380b9770 tx=0x7f3738005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d03047cce48ce1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f372effd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:04.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f372effd640 1 -- 172.21.15.70:0/4024550920 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37380d6060 con 0x560b4b79d610 2024-04-01T01:33:04.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f372effd640 1 -- 172.21.15.70:0/4024550920 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f37380bb610 con 0x560b4b79d610 2024-04-01T01:33:04.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f3758dbc740 1 -- 172.21.15.70:0/4024550920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37380bda10 con 0x560b4b79d610 2024-04-01T01:33:04.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f372effd640 1 -- 172.21.15.70:0/4024550920 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f37380bcc70 con 0x560b4b79d610 2024-04-01T01:33:04.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f3758dbc740 10 client.15249 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:04.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f3758dbc740 1 -- 172.21.15.70:0/4024550920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560b4b7a5f70 con 0x560b4b79d610 2024-04-01T01:33:04.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f372effd640 1 -- 172.21.15.70:0/4024550920 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f37380bda10 con 0x560b4b79d610 2024-04-01T01:33:04.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f372effd640 1 -- 172.21.15.70:0/4024550920 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f37380a0020 con 0x560b4b79d610 2024-04-01T01:33:04.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f3758dbc740 10 client.15249 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:04.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f3758dbc740 10 client.15249 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-01T01:33:04.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f3758dbc740 20 client.15249 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:04.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f3758dbc740 1 --2- 172.21.15.70:0/4024550920 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x560b4b7b3e00 0x560b4b7b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f3758dbc740 4 client.15249 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:33:04.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.383+0000 7f3758dbc740 1 -- 172.21.15.70:0/4024550920 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b4b70f030 con 0x560b4b7b3e00 2024-04-01T01:33:04.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.385+0000 7f374effd640 1 --2- 172.21.15.70:0/4024550920 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x560b4b7b3e00 0x560b4b7b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.386+0000 7f374effd640 1 --2- 172.21.15.70:0/4024550920 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x560b4b7b3e00 0x560b4b7b5270 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1c645fba12007a30 server_cookie=59e7447ad4e61f5f in_seq=0 out_seq=0 2024-04-01T01:33:04.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.386+0000 7f372effd640 10 client.15249 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:33:04.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.386+0000 7f372effd640 1 -- 172.21.15.70:0/4024550920 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560b4b70f030 con 0x560b4b7b3e00 2024-04-01T01:33:04.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.386+0000 7f372effd640 10 client.15249 handle_command_reply: tid=0 2024-04-01T01:33:04.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.386+0000 7f3758dbc740 1 client.15249 shutdown 2024-04-01T01:33:04.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.387+0000 7f3758dbc740 1 -- 172.21.15.70:0/4024550920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79d610 msgr2=0x560b4b7a7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.387+0000 7f3758dbc740 1 --2- 172.21.15.70:0/4024550920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79d610 0x560b4b7a7600 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f37380b9770 tx=0x7f3738005b50 comp rx=0 tx=0).stop 2024-04-01T01:33:04.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.387+0000 7f3758dbc740 1 -- 172.21.15.70:0/4024550920 shutdown_connections 2024-04-01T01:33:04.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.387+0000 7f3758dbc740 1 --2- 172.21.15.70:0/4024550920 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x560b4b7b3e00 0x560b4b7b5270 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.387+0000 7f3758dbc740 1 --2- 172.21.15.70:0/4024550920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79f1a0 0x560b4b7aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.387+0000 7f3758dbc740 1 --2- 172.21.15.70:0/4024550920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79e7e0 0x560b4b7a7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.387+0000 7f372effd640 0 client.15249 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:33:04.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.387+0000 7f3758dbc740 1 --2- 172.21.15.70:0/4024550920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79d610 0x560b4b7a7600 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.387+0000 7f3758dbc740 1 -- 172.21.15.70:0/4024550920 >> 172.21.15.70:0/4024550920 conn(0x560b4b798a00 msgr2=0x560b4b797b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.387+0000 7f3758dbc740 1 -- 172.21.15.70:0/4024550920 shutdown_connections 2024-04-01T01:33:04.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.387+0000 7f3758dbc740 1 -- 172.21.15.70:0/4024550920 wait complete. 2024-04-01T01:33:04.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.387+0000 7f3758dbc740 20 client.15249 trim_cache size 0 max 16384 2024-04-01T01:33:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f3758dbc740 1 Processor -- start 2024-04-01T01:33:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f3758dbc740 1 -- start start 2024-04-01T01:33:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f3758dbc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79d610 0x560b4b7a2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f3758dbc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79e7e0 0x560b4b7a2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f3758dbc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79f1a0 0x560b4b7aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f3758dbc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560b4b7b35b0 con 0x560b4b79d610 2024-04-01T01:33:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f3758dbc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560b4b7b3c80 con 0x560b4b79f1a0 2024-04-01T01:33:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f3758dbc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560b4b7b4350 con 0x560b4b79e7e0 2024-04-01T01:33:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f374f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79d610 0x560b4b7a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f374f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79d610 0x560b4b7a2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39738/0 (socket says 172.21.15.70:39738) 2024-04-01T01:33:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f374f7fe640 1 -- 172.21.15.70:0/3284905947 learned_addr learned my addr 172.21.15.70:0/3284905947 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f374ffff640 1 --2- 172.21.15.70:0/3284905947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79f1a0 0x560b4b7aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f374effd640 1 --2- 172.21.15.70:0/3284905947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79e7e0 0x560b4b7a2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f374f7fe640 1 -- 172.21.15.70:0/3284905947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79e7e0 msgr2=0x560b4b7a2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f374f7fe640 1 --2- 172.21.15.70:0/3284905947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79e7e0 0x560b4b7a2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f374f7fe640 1 -- 172.21.15.70:0/3284905947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79f1a0 msgr2=0x560b4b7aeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f374f7fe640 1 --2- 172.21.15.70:0/3284905947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79f1a0 0x560b4b7aeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f374f7fe640 1 -- 172.21.15.70:0/3284905947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b4b79e260 con 0x560b4b79d610 2024-04-01T01:33:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f374f7fe640 1 --2- 172.21.15.70:0/3284905947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79d610 0x560b4b7a2110 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f373802bc70 tx=0x7f37380d61e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=133ee726c0785d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f37457fa640 1 -- 172.21.15.70:0/3284905947 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37380d6040 con 0x560b4b79d610 2024-04-01T01:33:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f37457fa640 1 -- 172.21.15.70:0/3284905947 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f373809d020 con 0x560b4b79d610 2024-04-01T01:33:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.405+0000 7f37457fa640 1 -- 172.21.15.70:0/3284905947 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37380c5d20 con 0x560b4b79d610 2024-04-01T01:33:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.406+0000 7f3758dbc740 1 -- 172.21.15.70:0/3284905947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79d610 msgr2=0x560b4b7a2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.406+0000 7f3758dbc740 1 --2- 172.21.15.70:0/3284905947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79d610 0x560b4b7a2110 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f373802bc70 tx=0x7f37380d61e0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.406+0000 7f3758dbc740 1 -- 172.21.15.70:0/3284905947 shutdown_connections 2024-04-01T01:33:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.406+0000 7f3758dbc740 1 --2- 172.21.15.70:0/3284905947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79f1a0 0x560b4b7aeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.406+0000 7f3758dbc740 1 --2- 172.21.15.70:0/3284905947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79e7e0 0x560b4b7a2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.406+0000 7f3758dbc740 1 --2- 172.21.15.70:0/3284905947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79d610 0x560b4b7a2110 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.406+0000 7f3758dbc740 1 -- 172.21.15.70:0/3284905947 >> 172.21.15.70:0/3284905947 conn(0x560b4b798a00 msgr2=0x560b4b7a5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.406+0000 7f3758dbc740 1 -- 172.21.15.70:0/3284905947 shutdown_connections 2024-04-01T01:33:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.406+0000 7f3758dbc740 1 -- 172.21.15.70:0/3284905947 wait complete. 2024-04-01T01:33:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.406+0000 7f3758dbc740 1 -- start start 2024-04-01T01:33:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f3758dbc740 1 Processor -- start 2024-04-01T01:33:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f3758dbc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79d610 0x560b4b7aa760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f3758dbc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79e7e0 0x560b4b7aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f3758dbc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79f1a0 0x560b4b7adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f3758dbc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560b4b7b50f0 con 0x560b4b79e7e0 2024-04-01T01:33:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f3758dbc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560b4b7b4a20 con 0x560b4b79f1a0 2024-04-01T01:33:04.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f3758dbc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560b4b7b57c0 con 0x560b4b79d610 2024-04-01T01:33:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f3758dbc740 10 client.0 _ll_register_callbacks cb 0x7f3748009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f374effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79e7e0 0x560b4b7aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f374f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79d610 0x560b4b7aa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f374effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79e7e0 0x560b4b7aaca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39752/0 (socket says 172.21.15.70:39752) 2024-04-01T01:33:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f374effd640 1 -- 172.21.15.70:0/2473165006 learned_addr learned my addr 172.21.15.70:0/2473165006 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f374ffff640 1 --2- 172.21.15.70:0/2473165006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79f1a0 0x560b4b7adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f374effd640 1 -- 172.21.15.70:0/2473165006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79d610 msgr2=0x560b4b7aa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f374effd640 1 --2- 172.21.15.70:0/2473165006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79d610 0x560b4b7aa760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f374effd640 1 -- 172.21.15.70:0/2473165006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79f1a0 msgr2=0x560b4b7adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f374effd640 1 --2- 172.21.15.70:0/2473165006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79f1a0 0x560b4b7adcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f374effd640 1 -- 172.21.15.70:0/2473165006 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b4b7ae2b0 con 0x560b4b79e7e0 2024-04-01T01:33:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f374effd640 1 --2- 172.21.15.70:0/2473165006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79e7e0 0x560b4b7aaca0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3750198f30 tx=0x7f3740020570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62264492f42876c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f372effd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:04.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f372effd640 1 -- 172.21.15.70:0/2473165006 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3740024030 con 0x560b4b79e7e0 2024-04-01T01:33:04.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f372effd640 1 -- 172.21.15.70:0/2473165006 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3740005020 con 0x560b4b79e7e0 2024-04-01T01:33:04.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.407+0000 7f3758dbc740 1 -- 172.21.15.70:0/2473165006 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f373809d020 con 0x560b4b79e7e0 2024-04-01T01:33:04.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.408+0000 7f372effd640 1 -- 172.21.15.70:0/2473165006 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f374002e050 con 0x560b4b79e7e0 2024-04-01T01:33:04.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.408+0000 7f3758dbc740 10 client.15255 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:04.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.408+0000 7f3758dbc740 1 -- 172.21.15.70:0/2473165006 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560b4b81a1e0 con 0x560b4b79e7e0 2024-04-01T01:33:04.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.408+0000 7f372effd640 1 -- 172.21.15.70:0/2473165006 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f373809d020 con 0x560b4b79e7e0 2024-04-01T01:33:04.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.408+0000 7f372effd640 1 -- 172.21.15.70:0/2473165006 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f37400035f0 con 0x560b4b79e7e0 2024-04-01T01:33:04.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.408+0000 7f3758dbc740 10 client.15255 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:04.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.408+0000 7f3758dbc740 10 client.15255 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-01T01:33:04.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.408+0000 7f3758dbc740 20 client.15255 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:04.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.408+0000 7f3758dbc740 1 --2- 172.21.15.70:0/2473165006 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x560b4b81ee50 0x560b4b81db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.408+0000 7f3758dbc740 4 client.15255 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-01T01:33:04.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.408+0000 7f3758dbc740 1 -- 172.21.15.70:0/2473165006 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "status"}) -- 0x560b4b709380 con 0x560b4b81ee50 2024-04-01T01:33:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.408+0000 7f374f7fe640 1 --2- 172.21.15.70:0/2473165006 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x560b4b81ee50 0x560b4b81db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.408+0000 7f372effd640 10 client.15255 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:33:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.408+0000 7f374f7fe640 1 --2- 172.21.15.70:0/2473165006 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x560b4b81ee50 0x560b4b81db60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=692371a387683e8 server_cookie=b0567f593c2e1ccb in_seq=0 out_seq=0 2024-04-01T01:33:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f372effd640 1 -- 172.21.15.70:0/2473165006 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x560b4b709380 con 0x560b4b81ee50 2024-04-01T01:33:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f372effd640 10 client.15255 handle_command_reply: tid=0 2024-04-01T01:33:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f3758dbc740 1 client.15255 shutdown 2024-04-01T01:33:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f3758dbc740 1 -- 172.21.15.70:0/2473165006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79e7e0 msgr2=0x560b4b7aaca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f3758dbc740 1 --2- 172.21.15.70:0/2473165006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79e7e0 0x560b4b7aaca0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3750198f30 tx=0x7f3740020570 comp rx=0 tx=0).stop 2024-04-01T01:33:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f3758dbc740 1 -- 172.21.15.70:0/2473165006 shutdown_connections 2024-04-01T01:33:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f3758dbc740 1 --2- 172.21.15.70:0/2473165006 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x560b4b81ee50 0x560b4b81db60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f3758dbc740 1 --2- 172.21.15.70:0/2473165006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b4b79f1a0 0x560b4b7adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f372effd640 0 client.15255 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:33:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f3758dbc740 1 --2- 172.21.15.70:0/2473165006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b4b79e7e0 0x560b4b7aaca0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f3758dbc740 1 --2- 172.21.15.70:0/2473165006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b4b79d610 0x560b4b7aa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f3758dbc740 1 -- 172.21.15.70:0/2473165006 >> 172.21.15.70:0/2473165006 conn(0x560b4b798a00 msgr2=0x560b4b79bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f3758dbc740 1 -- 172.21.15.70:0/2473165006 shutdown_connections 2024-04-01T01:33:04.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f3758dbc740 1 -- 172.21.15.70:0/2473165006 wait complete. 2024-04-01T01:33:04.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f3758dbc740 20 client.15255 trim_cache size 0 max 16384 2024-04-01T01:33:04.417 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:33:04.417 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "36577394-efc6-11ee-b647-cb9ed24678a4", 2024-04-01T01:33:04.417 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 2, 2024-04-01T01:33:04.417 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24418, 2024-04-01T01:33:04.417 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-01T01:33:04.417 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-01T01:33:04.417 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-01T01:33:04.418 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 88.863791008000007, 2024-04-01T01:33:04.418 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-04-01T01:33:04.418 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 91, 2024-04-01T01:33:04.418 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 89, 2024-04-01T01:33:04.418 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 130.30830897300001 2024-04-01T01:33:04.418 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:33:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f37560fb640 1 -- 172.21.15.70:0/918915083 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f373c0a2520 msgr2=0x7f373c0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f37560fb640 1 --2- 172.21.15.70:0/918915083 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f373c0a2520 0x7f373c0a49e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x560b4b79e6e0 tx=0x7f3748004ab0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f37560fb640 1 -- 172.21.15.70:0/918915083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3750102270 msgr2=0x7f3750197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f37560fb640 1 --2- 172.21.15.70:0/918915083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3750102270 0x7f3750197a20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3738006010 tx=0x7f3738005e40 comp rx=0 tx=0).stop 2024-04-01T01:33:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f37560fb640 1 -- 172.21.15.70:0/918915083 shutdown_connections 2024-04-01T01:33:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f37560fb640 1 --2- 172.21.15.70:0/918915083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37501074d0 0x7f375019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f37560fb640 1 --2- 172.21.15.70:0/918915083 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f373c0a2520 0x7f373c0a49e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f37560fb640 1 --2- 172.21.15.70:0/918915083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3750104ba0 0x7f3750197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f37560fb640 1 --2- 172.21.15.70:0/918915083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3750102270 0x7f3750197a20 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.409+0000 7f37560fb640 1 -- 172.21.15.70:0/918915083 >> 172.21.15.70:0/918915083 conn(0x7f37500fbbe0 msgr2=0x7f37501040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.410+0000 7f37560fb640 1 -- 172.21.15.70:0/918915083 shutdown_connections 2024-04-01T01:33:04.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.410+0000 7f37560fb640 1 -- 172.21.15.70:0/918915083 wait complete. 2024-04-01T01:33:04.421 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-01T01:33:04.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.523+0000 7fcbe8a9d640 1 -- 172.21.15.70:0/2869221792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbe4107430 msgr2=0x7fcbe4109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.523+0000 7fcbe8a9d640 1 --2- 172.21.15.70:0/2869221792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbe4107430 0x7fcbe4109880 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc009900 tx=0x7fcbdc02bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:04.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.524+0000 7fcbe8a9d640 1 -- 172.21.15.70:0/2869221792 shutdown_connections 2024-04-01T01:33:04.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.524+0000 7fcbe8a9d640 1 --2- 172.21.15.70:0/2869221792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbe4107430 0x7fcbe4109880 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.524+0000 7fcbe8a9d640 1 --2- 172.21.15.70:0/2869221792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbe4104b00 0x7fcbe4106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.524+0000 7fcbe8a9d640 1 --2- 172.21.15.70:0/2869221792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbe41021d0 0x7fcbe41045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.524+0000 7fcbe8a9d640 1 -- 172.21.15.70:0/2869221792 >> 172.21.15.70:0/2869221792 conn(0x7fcbe40fbbe0 msgr2=0x7fcbe40fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.524+0000 7fcbe8a9d640 1 -- 172.21.15.70:0/2869221792 shutdown_connections 2024-04-01T01:33:04.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.524+0000 7fcbe8a9d640 1 -- 172.21.15.70:0/2869221792 wait complete. 2024-04-01T01:33:04.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.524+0000 7fcbe8a9d640 1 Processor -- start 2024-04-01T01:33:04.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.524+0000 7fcbe8a9d640 1 -- start start 2024-04-01T01:33:04.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe8a9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbe41021d0 0x7fcbe4197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe8a9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbe4104b00 0x7fcbe4197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe8a9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbe4107430 0x7fcbe419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe8a9d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe410d280 con 0x7fcbe4104b00 2024-04-01T01:33:04.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe8a9d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe410d100 con 0x7fcbe4107430 2024-04-01T01:33:04.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe8a9d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe410d400 con 0x7fcbe41021d0 2024-04-01T01:33:04.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbe4104b00 0x7fcbe4197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbe4104b00 0x7fcbe4197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39762/0 (socket says 172.21.15.70:39762) 2024-04-01T01:33:04.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe1d9b640 1 -- 172.21.15.70:0/31559641 learned_addr learned my addr 172.21.15.70:0/31559641 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe2d9d640 1 --2- 172.21.15.70:0/31559641 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbe4107430 0x7fcbe419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe259c640 1 --2- 172.21.15.70:0/31559641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbe41021d0 0x7fcbe4197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe1d9b640 1 -- 172.21.15.70:0/31559641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbe41021d0 msgr2=0x7fcbe4197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe1d9b640 1 --2- 172.21.15.70:0/31559641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbe41021d0 0x7fcbe4197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe1d9b640 1 -- 172.21.15.70:0/31559641 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbe4107430 msgr2=0x7fcbe419ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe1d9b640 1 --2- 172.21.15.70:0/31559641 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbe4107430 0x7fcbe419ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe1d9b640 1 -- 172.21.15.70:0/31559641 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe419f5a0 con 0x7fcbe4104b00 2024-04-01T01:33:04.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe259c640 1 --2- 172.21.15.70:0/31559641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbe41021d0 0x7fcbe4197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:04.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe2d9d640 1 --2- 172.21.15.70:0/31559641 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbe4107430 0x7fcbe419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:04.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe1d9b640 1 --2- 172.21.15.70:0/31559641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbe4104b00 0x7fcbe4197ec0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fcbd800e9a0 tx=0x7fcbd800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbd37fe640 1 -- 172.21.15.70:0/31559641 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd800cd80 con 0x7fcbe4104b00 2024-04-01T01:33:04.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbd37fe640 1 -- 172.21.15.70:0/31559641 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcbd80035f0 con 0x7fcbe4104b00 2024-04-01T01:33:04.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe8a9d640 1 -- 172.21.15.70:0/31559641 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbe419f890 con 0x7fcbe4104b00 2024-04-01T01:33:04.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbd37fe640 1 -- 172.21.15.70:0/31559641 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8010690 con 0x7fcbe4104b00 2024-04-01T01:33:04.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.525+0000 7fcbe8a9d640 1 -- 172.21.15.70:0/31559641 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbe419fde0 con 0x7fcbe4104b00 2024-04-01T01:33:04.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbd37fe640 1 -- 172.21.15.70:0/31559641 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fcbd80037a0 con 0x7fcbe4104b00 2024-04-01T01:33:04.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbeb684740 1 Processor -- start 2024-04-01T01:33:04.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbeb684740 1 -- start start 2024-04-01T01:33:04.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbeb684740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a83f610 0x560d0a83fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbeb684740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8407e0 0x560d0a840c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbeb684740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8411a0 0x560d0a847a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbeb684740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560d0a7a3af0 con 0x560d0a8411a0 2024-04-01T01:33:04.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbeb684740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560d0a74c9d0 con 0x560d0a83f610 2024-04-01T01:33:04.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbeb684740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560d0a79fb90 con 0x560d0a8407e0 2024-04-01T01:33:04.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbe1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8411a0 0x560d0a847a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbe2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a83f610 0x560d0a83fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbe259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8407e0 0x560d0a840c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbd37fe640 1 --2- 172.21.15.70:0/31559641 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcbcc0a24a0 0x7fcbcc0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbe1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8411a0 0x560d0a847a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39772/0 (socket says 172.21.15.70:39772) 2024-04-01T01:33:04.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbe259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8407e0 0x560d0a840c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39852/0 (socket says 172.21.15.70:39852) 2024-04-01T01:33:04.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbe2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a83f610 0x560d0a83fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:52966/0 (socket says 172.21.15.70:52966) 2024-04-01T01:33:04.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbe1d9b640 1 -- 172.21.15.70:0/503920480 learned_addr learned my addr 172.21.15.70:0/503920480 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.528+0000 7fcbe2d9d640 1 --2- 172.21.15.70:0/31559641 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcbcc0a24a0 0x7fcbcc0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.529+0000 7fcbd37fe640 1 -- 172.21.15.70:0/31559641 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcbd80cecd0 con 0x7fcbe4104b00 2024-04-01T01:33:04.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.529+0000 7fcbe1d9b640 1 -- 172.21.15.70:0/503920480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8407e0 msgr2=0x560d0a840c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.529+0000 7fcbe1d9b640 1 --2- 172.21.15.70:0/503920480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8407e0 0x560d0a840c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.529+0000 7fcbe1d9b640 1 -- 172.21.15.70:0/503920480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a83f610 msgr2=0x560d0a83fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.529+0000 7fcbe1d9b640 1 --2- 172.21.15.70:0/503920480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a83f610 0x560d0a83fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.529+0000 7fcbe1d9b640 1 -- 172.21.15.70:0/503920480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d0a4b82d0 con 0x560d0a8411a0 2024-04-01T01:33:04.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.529+0000 7fcbe2d9d640 1 --2- 172.21.15.70:0/503920480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a83f610 0x560d0a83fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:04.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.529+0000 7fcbe259c640 1 --2- 172.21.15.70:0/503920480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8407e0 0x560d0a840c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:33:04.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.529+0000 7fcbe1d9b640 1 --2- 172.21.15.70:0/503920480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8411a0 0x560d0a847a30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fcbd80a4000 tx=0x7fcbd8095e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47df0c03e9a1dc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.529+0000 7fcbd0ff9640 1 -- 172.21.15.70:0/503920480 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8003b10 con 0x560d0a8411a0 2024-04-01T01:33:04.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.529+0000 7fcbe2d9d640 1 --2- 172.21.15.70:0/31559641 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcbcc0a24a0 0x7fcbcc0a4960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc006010 tx=0x7fcbdc02bc30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.529+0000 7fcbd0ff9640 1 -- 172.21.15.70:0/503920480 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcbd80a3e40 con 0x560d0a8411a0 2024-04-01T01:33:04.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.529+0000 7fcbd0ff9640 1 -- 172.21.15.70:0/503920480 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd80bb0d0 con 0x560d0a8411a0 2024-04-01T01:33:04.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 -- 172.21.15.70:0/503920480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8411a0 msgr2=0x560d0a847a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 --2- 172.21.15.70:0/503920480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8411a0 0x560d0a847a30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fcbd80a4000 tx=0x7fcbd8095e90 comp rx=0 tx=0).stop 2024-04-01T01:33:04.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 -- 172.21.15.70:0/503920480 shutdown_connections 2024-04-01T01:33:04.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 --2- 172.21.15.70:0/503920480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8411a0 0x560d0a847a30 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 --2- 172.21.15.70:0/503920480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8407e0 0x560d0a840c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 --2- 172.21.15.70:0/503920480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a83f610 0x560d0a83fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 -- 172.21.15.70:0/503920480 >> 172.21.15.70:0/503920480 conn(0x560d0a83aa00 msgr2=0x560d0a83ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 -- 172.21.15.70:0/503920480 shutdown_connections 2024-04-01T01:33:04.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 -- 172.21.15.70:0/503920480 wait complete. 2024-04-01T01:33:04.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 -- start start 2024-04-01T01:33:04.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 Processor -- start 2024-04-01T01:33:04.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a83f610 0x560d0a849890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8407e0 0x560d0a849dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8411a0 0x560d0a850e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560d0a848150 con 0x560d0a8407e0 2024-04-01T01:33:04.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560d0a75ff70 con 0x560d0a83f610 2024-04-01T01:33:04.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560d0a8482d0 con 0x560d0a8411a0 2024-04-01T01:33:04.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbeb684740 10 client.0 _ll_register_callbacks cb 0x560d0a4b7ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:04.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.530+0000 7fcbe1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8407e0 0x560d0a849dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbe259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a83f610 0x560d0a849890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbe1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8407e0 0x560d0a849dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39782/0 (socket says 172.21.15.70:39782) 2024-04-01T01:33:04.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbe1d9b640 1 -- 172.21.15.70:0/300358130 learned_addr learned my addr 172.21.15.70:0/300358130 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbe2d9d640 1 --2- 172.21.15.70:0/300358130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8411a0 0x560d0a850e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbe1d9b640 1 -- 172.21.15.70:0/300358130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8411a0 msgr2=0x560d0a850e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbe1d9b640 1 --2- 172.21.15.70:0/300358130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8411a0 0x560d0a850e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbe1d9b640 1 -- 172.21.15.70:0/300358130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a83f610 msgr2=0x560d0a849890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbe1d9b640 1 --2- 172.21.15.70:0/300358130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a83f610 0x560d0a849890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbe1d9b640 1 -- 172.21.15.70:0/300358130 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d0a5826b0 con 0x560d0a8407e0 2024-04-01T01:33:04.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbe1d9b640 1 --2- 172.21.15.70:0/300358130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8407e0 0x560d0a849dd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8097e00 tx=0x7fcbd80bb380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e6cda165f1cebf server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbc9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:04.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbc9ffb640 1 -- 172.21.15.70:0/300358130 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd80b4060 con 0x560d0a8407e0 2024-04-01T01:33:04.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbc9ffb640 1 -- 172.21.15.70:0/300358130 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcbd809aa90 con 0x560d0a8407e0 2024-04-01T01:33:04.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbeb684740 1 -- 172.21.15.70:0/300358130 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560d0a7b1030 con 0x560d0a8407e0 2024-04-01T01:33:04.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbc9ffb640 1 -- 172.21.15.70:0/300358130 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcbd809b430 con 0x560d0a8407e0 2024-04-01T01:33:04.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbeb684740 10 client.15267 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:04.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbeb684740 1 -- 172.21.15.70:0/300358130 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560d0a853df0 con 0x560d0a8407e0 2024-04-01T01:33:04.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbc9ffb640 1 -- 172.21.15.70:0/300358130 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x560d0a7b1030 con 0x560d0a8407e0 2024-04-01T01:33:04.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.531+0000 7fcbc9ffb640 1 -- 172.21.15.70:0/300358130 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcbd809a2f0 con 0x560d0a8407e0 2024-04-01T01:33:04.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.532+0000 7fcbeb684740 10 client.15267 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:04.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.532+0000 7fcbeb684740 10 client.15267 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-01T01:33:04.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.532+0000 7fcbeb684740 20 client.15267 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:04.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.532+0000 7fcbeb684740 1 --2- 172.21.15.70:0/300358130 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560d0a855f50 0x560d0a8573c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.532+0000 7fcbeb684740 4 client.15267 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:33:04.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.532+0000 7fcbeb684740 1 -- 172.21.15.70:0/300358130 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d0a7ab380 con 0x560d0a855f50 2024-04-01T01:33:04.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.532+0000 7fcbe259c640 1 --2- 172.21.15.70:0/300358130 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560d0a855f50 0x560d0a8573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.532+0000 7fcbc9ffb640 10 client.15267 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:33:04.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.532+0000 7fcbe259c640 1 --2- 172.21.15.70:0/300358130 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560d0a855f50 0x560d0a8573c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=52a5441b75a11dd2 server_cookie=bd666341a466468c in_seq=0 out_seq=0 2024-04-01T01:33:04.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.533+0000 7fcbc9ffb640 1 -- 172.21.15.70:0/300358130 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560d0a7ab380 con 0x560d0a855f50 2024-04-01T01:33:04.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.533+0000 7fcbc9ffb640 10 client.15267 handle_command_reply: tid=0 2024-04-01T01:33:04.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.533+0000 7fcbeb684740 1 client.15267 shutdown 2024-04-01T01:33:04.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.533+0000 7fcbeb684740 1 -- 172.21.15.70:0/300358130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8407e0 msgr2=0x560d0a849dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.533+0000 7fcbeb684740 1 --2- 172.21.15.70:0/300358130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8407e0 0x560d0a849dd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8097e00 tx=0x7fcbd80bb380 comp rx=0 tx=0).stop 2024-04-01T01:33:04.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.533+0000 7fcbeb684740 1 -- 172.21.15.70:0/300358130 shutdown_connections 2024-04-01T01:33:04.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.533+0000 7fcbeb684740 1 --2- 172.21.15.70:0/300358130 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560d0a855f50 0x560d0a8573c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.533+0000 7fcbeb684740 1 --2- 172.21.15.70:0/300358130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8411a0 0x560d0a850e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.533+0000 7fcbeb684740 1 --2- 172.21.15.70:0/300358130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8407e0 0x560d0a849dd0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.533+0000 7fcbc9ffb640 0 client.15267 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:33:04.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.533+0000 7fcbeb684740 1 --2- 172.21.15.70:0/300358130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a83f610 0x560d0a849890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.533+0000 7fcbeb684740 1 -- 172.21.15.70:0/300358130 >> 172.21.15.70:0/300358130 conn(0x560d0a83aa00 msgr2=0x560d0a839b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.533+0000 7fcbeb684740 1 -- 172.21.15.70:0/300358130 shutdown_connections 2024-04-01T01:33:04.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.533+0000 7fcbeb684740 1 -- 172.21.15.70:0/300358130 wait complete. 2024-04-01T01:33:04.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.533+0000 7fcbeb684740 20 client.15267 trim_cache size 0 max 16384 2024-04-01T01:33:04.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbeb684740 1 Processor -- start 2024-04-01T01:33:04.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbeb684740 1 -- start start 2024-04-01T01:33:04.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbeb684740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a83f610 0x560d0a848670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbeb684740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a8407e0 0x560d0a850dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbeb684740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8411a0 0x560d0a83ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbeb684740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560d0a855700 con 0x560d0a83f610 2024-04-01T01:33:04.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbeb684740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560d0a855dd0 con 0x560d0a8407e0 2024-04-01T01:33:04.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbeb684740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560d0a8564a0 con 0x560d0a8411a0 2024-04-01T01:33:04.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbe259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a83f610 0x560d0a848670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbe259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a83f610 0x560d0a848670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39786/0 (socket says 172.21.15.70:39786) 2024-04-01T01:33:04.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbe259c640 1 -- 172.21.15.70:0/1797320028 learned_addr learned my addr 172.21.15.70:0/1797320028 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbe1d9b640 1 --2- 172.21.15.70:0/1797320028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a8407e0 0x560d0a850dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbe2d9d640 1 --2- 172.21.15.70:0/1797320028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8411a0 0x560d0a83ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbe259c640 1 -- 172.21.15.70:0/1797320028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8411a0 msgr2=0x560d0a83ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbe259c640 1 --2- 172.21.15.70:0/1797320028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8411a0 0x560d0a83ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbe259c640 1 -- 172.21.15.70:0/1797320028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a8407e0 msgr2=0x560d0a850dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbe259c640 1 --2- 172.21.15.70:0/1797320028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a8407e0 0x560d0a850dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbe259c640 1 -- 172.21.15.70:0/1797320028 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d0a839040 con 0x560d0a83f610 2024-04-01T01:33:04.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.551+0000 7fcbe259c640 1 --2- 172.21.15.70:0/1797320028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a83f610 0x560d0a848670 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4012740 tx=0x7fcbd4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e7e9ed19c1afc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.552+0000 7fcbcbfff640 1 -- 172.21.15.70:0/1797320028 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4004d20 con 0x560d0a83f610 2024-04-01T01:33:04.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.552+0000 7fcbcbfff640 1 -- 172.21.15.70:0/1797320028 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcbd4003e20 con 0x560d0a83f610 2024-04-01T01:33:04.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.552+0000 7fcbcbfff640 1 -- 172.21.15.70:0/1797320028 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd40222d0 con 0x560d0a83f610 2024-04-01T01:33:04.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.552+0000 7fcbeb684740 1 -- 172.21.15.70:0/1797320028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a83f610 msgr2=0x560d0a848670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.552+0000 7fcbeb684740 1 --2- 172.21.15.70:0/1797320028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a83f610 0x560d0a848670 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4012740 tx=0x7fcbd4009500 comp rx=0 tx=0).stop 2024-04-01T01:33:04.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.552+0000 7fcbeb684740 1 -- 172.21.15.70:0/1797320028 shutdown_connections 2024-04-01T01:33:04.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.552+0000 7fcbeb684740 1 --2- 172.21.15.70:0/1797320028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a8411a0 0x560d0a83ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.552+0000 7fcbeb684740 1 --2- 172.21.15.70:0/1797320028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a8407e0 0x560d0a850dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.552+0000 7fcbeb684740 1 --2- 172.21.15.70:0/1797320028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a83f610 0x560d0a848670 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.552+0000 7fcbeb684740 1 -- 172.21.15.70:0/1797320028 >> 172.21.15.70:0/1797320028 conn(0x560d0a83aa00 msgr2=0x560d0a847770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.552+0000 7fcbeb684740 1 -- 172.21.15.70:0/1797320028 shutdown_connections 2024-04-01T01:33:04.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.552+0000 7fcbeb684740 1 -- 172.21.15.70:0/1797320028 wait complete. 2024-04-01T01:33:04.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbeb684740 1 -- start start 2024-04-01T01:33:04.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbeb684740 1 Processor -- start 2024-04-01T01:33:04.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbeb684740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a83f610 0x560d0a84edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbeb684740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a8407e0 0x560d0a8b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbeb684740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8411a0 0x560d0a8ba210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbeb684740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560d0a857240 con 0x560d0a8411a0 2024-04-01T01:33:04.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbeb684740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560d0a856b70 con 0x560d0a8407e0 2024-04-01T01:33:04.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbeb684740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560d0a857910 con 0x560d0a83f610 2024-04-01T01:33:04.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbeb684740 10 client.0 _ll_register_callbacks cb 0x7fcbd40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:04.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbe2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8411a0 0x560d0a8ba210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbe259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a83f610 0x560d0a84edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbe2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8411a0 0x560d0a8ba210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39798/0 (socket says 172.21.15.70:39798) 2024-04-01T01:33:04.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbe2d9d640 1 -- 172.21.15.70:0/1144324466 learned_addr learned my addr 172.21.15.70:0/1144324466 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbe1d9b640 1 --2- 172.21.15.70:0/1144324466 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a8407e0 0x560d0a8b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbe2d9d640 1 -- 172.21.15.70:0/1144324466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a83f610 msgr2=0x560d0a84edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbe2d9d640 1 --2- 172.21.15.70:0/1144324466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a83f610 0x560d0a84edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbe2d9d640 1 -- 172.21.15.70:0/1144324466 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a8407e0 msgr2=0x560d0a8b7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbe2d9d640 1 --2- 172.21.15.70:0/1144324466 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a8407e0 0x560d0a8b7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbe2d9d640 1 -- 172.21.15.70:0/1144324466 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d0a8baa40 con 0x560d0a8411a0 2024-04-01T01:33:04.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbe2d9d640 1 --2- 172.21.15.70:0/1144324466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8411a0 0x560d0a8ba210 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc009900 tx=0x7fcbdc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c8085734179626 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbc9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:04.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbc9ffb640 1 -- 172.21.15.70:0/1144324466 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc054030 con 0x560d0a8411a0 2024-04-01T01:33:04.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbc9ffb640 1 -- 172.21.15.70:0/1144324466 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcbdc03a040 con 0x560d0a8411a0 2024-04-01T01:33:04.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.553+0000 7fcbeb684740 1 -- 172.21.15.70:0/1144324466 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbd4003e20 con 0x560d0a8411a0 2024-04-01T01:33:04.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.554+0000 7fcbc9ffb640 1 -- 172.21.15.70:0/1144324466 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcbdc049040 con 0x560d0a8411a0 2024-04-01T01:33:04.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.554+0000 7fcbeb684740 10 client.15273 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:04.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.554+0000 7fcbeb684740 1 -- 172.21.15.70:0/1144324466 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560d0a8bbdb0 con 0x560d0a8411a0 2024-04-01T01:33:04.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.554+0000 7fcbc9ffb640 1 -- 172.21.15.70:0/1144324466 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcbd4003e20 con 0x560d0a8411a0 2024-04-01T01:33:04.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.554+0000 7fcbc9ffb640 1 -- 172.21.15.70:0/1144324466 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcbdc040040 con 0x560d0a8411a0 2024-04-01T01:33:04.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.554+0000 7fcbeb684740 10 client.15273 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:04.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.554+0000 7fcbeb684740 10 client.15273 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-01T01:33:04.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.554+0000 7fcbeb684740 20 client.15273 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:04.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.554+0000 7fcbeb684740 1 --2- 172.21.15.70:0/1144324466 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560d0a8c0a30 0x560d0a8bf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.554+0000 7fcbeb684740 4 client.15273 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-01T01:33:04.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.554+0000 7fcbeb684740 1 -- 172.21.15.70:0/1144324466 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "status"}) -- 0x560d0a729a00 con 0x560d0a8c0a30 2024-04-01T01:33:04.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.554+0000 7fcbe259c640 1 --2- 172.21.15.70:0/1144324466 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560d0a8c0a30 0x560d0a8bf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbe259c640 1 --2- 172.21.15.70:0/1144324466 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560d0a8c0a30 0x560d0a8bf740 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=710df5f5d2e7b752 server_cookie=1c436903aa39c5cd in_seq=0 out_seq=0 2024-04-01T01:33:04.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbc9ffb640 10 client.15273 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:33:04.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbc9ffb640 1 -- 172.21.15.70:0/1144324466 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x560d0a729a00 con 0x560d0a8c0a30 2024-04-01T01:33:04.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbc9ffb640 10 client.15273 handle_command_reply: tid=0 2024-04-01T01:33:04.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbeb684740 1 client.15273 shutdown 2024-04-01T01:33:04.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbeb684740 1 -- 172.21.15.70:0/1144324466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8411a0 msgr2=0x560d0a8ba210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbeb684740 1 --2- 172.21.15.70:0/1144324466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8411a0 0x560d0a8ba210 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc009900 tx=0x7fcbdc009260 comp rx=0 tx=0).stop 2024-04-01T01:33:04.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbeb684740 1 -- 172.21.15.70:0/1144324466 shutdown_connections 2024-04-01T01:33:04.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbeb684740 1 --2- 172.21.15.70:0/1144324466 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560d0a8c0a30 0x560d0a8bf740 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbeb684740 1 --2- 172.21.15.70:0/1144324466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d0a8411a0 0x560d0a8ba210 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbeb684740 1 --2- 172.21.15.70:0/1144324466 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d0a8407e0 0x560d0a8b7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbc9ffb640 0 client.15273 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:33:04.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbeb684740 1 --2- 172.21.15.70:0/1144324466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d0a83f610 0x560d0a84edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbeb684740 1 -- 172.21.15.70:0/1144324466 >> 172.21.15.70:0/1144324466 conn(0x560d0a83aa00 msgr2=0x560d0a848670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbeb684740 1 -- 172.21.15.70:0/1144324466 shutdown_connections 2024-04-01T01:33:04.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbeb684740 1 -- 172.21.15.70:0/1144324466 wait complete. 2024-04-01T01:33:04.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbeb684740 20 client.15273 trim_cache size 0 max 16384 2024-04-01T01:33:04.563 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:33:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "36577394-efc6-11ee-b647-cb9ed24678a4", 2024-04-01T01:33:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 1, 2024-04-01T01:33:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24451, 2024-04-01T01:33:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-01T01:33:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-01T01:33:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-01T01:33:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 89.944642884000004, 2024-04-01T01:33:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-04-01T01:33:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 91, 2024-04-01T01:33:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 89, 2024-04-01T01:33:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 122.034649104 2024-04-01T01:33:04.564 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:33:04.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbe8a9d640 1 -- 172.21.15.70:0/31559641 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcbcc0a24a0 msgr2=0x7fcbcc0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbe8a9d640 1 --2- 172.21.15.70:0/31559641 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcbcc0a24a0 0x7fcbcc0a4960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc006010 tx=0x7fcbdc02bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:04.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbe8a9d640 1 -- 172.21.15.70:0/31559641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbe4104b00 msgr2=0x7fcbe4197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbe8a9d640 1 --2- 172.21.15.70:0/31559641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbe4104b00 0x7fcbe4197ec0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fcbd800e9a0 tx=0x7fcbd800ee60 comp rx=0 tx=0).stop 2024-04-01T01:33:04.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbe8a9d640 1 -- 172.21.15.70:0/31559641 shutdown_connections 2024-04-01T01:33:04.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbe8a9d640 1 --2- 172.21.15.70:0/31559641 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbe4107430 0x7fcbe419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbe8a9d640 1 --2- 172.21.15.70:0/31559641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbe4104b00 0x7fcbe4197ec0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.555+0000 7fcbe8a9d640 1 --2- 172.21.15.70:0/31559641 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcbcc0a24a0 0x7fcbcc0a4960 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.556+0000 7fcbe8a9d640 1 --2- 172.21.15.70:0/31559641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbe41021d0 0x7fcbe4197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.556+0000 7fcbe8a9d640 1 -- 172.21.15.70:0/31559641 >> 172.21.15.70:0/31559641 conn(0x7fcbe40fbbe0 msgr2=0x7fcbe4102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.556+0000 7fcbe8a9d640 1 -- 172.21.15.70:0/31559641 shutdown_connections 2024-04-01T01:33:04.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.556+0000 7fcbe8a9d640 1 -- 172.21.15.70:0/31559641 wait complete. 2024-04-01T01:33:04.567 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-01T01:33:04.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.670+0000 7f717c1b6640 1 -- 172.21.15.70:0/2262599762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7174100670 msgr2=0x7f7174100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.670+0000 7f717c1b6640 1 --2- 172.21.15.70:0/2262599762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7174100670 0x7f7174100a90 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7170009900 tx=0x7f717002bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:04.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.670+0000 7f717c1b6640 1 -- 172.21.15.70:0/2262599762 shutdown_connections 2024-04-01T01:33:04.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.670+0000 7f717c1b6640 1 --2- 172.21.15.70:0/2262599762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7174102220 0x7f7174108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.670+0000 7f717c1b6640 1 --2- 172.21.15.70:0/2262599762 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7174101860 0x7f7174101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.670+0000 7f717c1b6640 1 --2- 172.21.15.70:0/2262599762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7174100670 0x7f7174100a90 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.670+0000 7f717c1b6640 1 -- 172.21.15.70:0/2262599762 >> 172.21.15.70:0/2262599762 conn(0x7f71740fbc00 msgr2=0x7f71740fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.670+0000 7f717c1b6640 1 -- 172.21.15.70:0/2262599762 shutdown_connections 2024-04-01T01:33:04.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.670+0000 7f717c1b6640 1 -- 172.21.15.70:0/2262599762 wait complete. 2024-04-01T01:33:04.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f717c1b6640 1 Processor -- start 2024-04-01T01:33:04.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f717c1b6640 1 -- start start 2024-04-01T01:33:04.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f717c1b6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7174100670 0x7f71741979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f717c1b6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7174101860 0x7f7174197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f717c1b6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7174102220 0x7f717419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f717c1b6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f717410d2c0 con 0x7f7174100670 2024-04-01T01:33:04.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f717c1b6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f717410d140 con 0x7f7174101860 2024-04-01T01:33:04.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f717c1b6640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f717410d440 con 0x7f7174102220 2024-04-01T01:33:04.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f7179f52640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7174100670 0x7f71741979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f7179f52640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7174100670 0x7f71741979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39822/0 (socket says 172.21.15.70:39822) 2024-04-01T01:33:04.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f7179f52640 1 -- 172.21.15.70:0/1258298696 learned_addr learned my addr 172.21.15.70:0/1258298696 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f7179751640 1 --2- 172.21.15.70:0/1258298696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7174101860 0x7f7174197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f717a753640 1 --2- 172.21.15.70:0/1258298696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7174102220 0x7f717419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f7179f52640 1 -- 172.21.15.70:0/1258298696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7174102220 msgr2=0x7f717419ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f7179f52640 1 --2- 172.21.15.70:0/1258298696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7174102220 0x7f717419ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f7179f52640 1 -- 172.21.15.70:0/1258298696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7174101860 msgr2=0x7f7174197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f7179f52640 1 --2- 172.21.15.70:0/1258298696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7174101860 0x7f7174197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f7179f52640 1 -- 172.21.15.70:0/1258298696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f717419f5f0 con 0x7f7174100670 2024-04-01T01:33:04.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.671+0000 7f7179f52640 1 --2- 172.21.15.70:0/1258298696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7174100670 0x7f71741979d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7170000c00 tx=0x7f7170005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.672+0000 7f716affd640 1 -- 172.21.15.70:0/1258298696 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7170043020 con 0x7f7174100670 2024-04-01T01:33:04.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.672+0000 7f717c1b6640 1 -- 172.21.15.70:0/1258298696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f717419f880 con 0x7f7174100670 2024-04-01T01:33:04.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.672+0000 7f716affd640 1 -- 172.21.15.70:0/1258298696 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7170004350 con 0x7f7174100670 2024-04-01T01:33:04.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.672+0000 7f716affd640 1 -- 172.21.15.70:0/1258298696 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7170003b80 con 0x7f7174100670 2024-04-01T01:33:04.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.672+0000 7f717c1b6640 1 -- 172.21.15.70:0/1258298696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f717419fdd0 con 0x7f7174100670 2024-04-01T01:33:04.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.674+0000 7f716affd640 1 -- 172.21.15.70:0/1258298696 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f7170046020 con 0x7f7174100670 2024-04-01T01:33:04.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.674+0000 7f717ed91740 1 Processor -- start 2024-04-01T01:33:04.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.674+0000 7f717ed91740 1 -- start start 2024-04-01T01:33:04.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.674+0000 7f717ed91740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 0x556033ca6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.674+0000 7f717ed91740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca77e0 0x556033ca7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.674+0000 7f717ed91740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca81a0 0x556033caea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.674+0000 7f717ed91740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556033c0aaf0 con 0x556033ca77e0 2024-04-01T01:33:04.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.674+0000 7f716affd640 1 --2- 172.21.15.70:0/1258298696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f71600a24a0 0x7f71600a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f717ed91740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x556033bb39d0 con 0x556033ca81a0 2024-04-01T01:33:04.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f717ed91740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556033c06b90 con 0x556033ca6610 2024-04-01T01:33:04.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f7179f52640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca77e0 0x556033ca7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f717a753640 1 --2- 172.21.15.70:0/1258298696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f71600a24a0 0x7f71600a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f7179f52640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca77e0 0x556033ca7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39836/0 (socket says 172.21.15.70:39836) 2024-04-01T01:33:04.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f7179f52640 1 -- 172.21.15.70:0/3489513266 learned_addr learned my addr 172.21.15.70:0/3489513266 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f717a753640 1 --2- 172.21.15.70:0/3489513266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 0x556033ca6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f7179751640 1 --2- 172.21.15.70:0/3489513266 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca81a0 0x556033caea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f716affd640 1 -- 172.21.15.70:0/1258298696 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f71700f0bc0 con 0x7f7174100670 2024-04-01T01:33:04.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f717a753640 1 -- 172.21.15.70:0/3489513266 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca81a0 msgr2=0x556033caea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f717a753640 1 --2- 172.21.15.70:0/3489513266 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca81a0 0x556033caea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f717a753640 1 -- 172.21.15.70:0/3489513266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca77e0 msgr2=0x556033ca7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f717a753640 1 --2- 172.21.15.70:0/3489513266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca77e0 0x556033ca7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f717a753640 1 -- 172.21.15.70:0/3489513266 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55603391f2d0 con 0x556033ca6610 2024-04-01T01:33:04.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f717a753640 1 --2- 172.21.15.70:0/3489513266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 0x556033ca6a30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f716c010d80 tx=0x7f716c00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac8d1b0815a1a67f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f717a753640 1 --2- 172.21.15.70:0/1258298696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f71600a24a0 0x7f71600a4960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f716c00fc90 tx=0x7f716c018000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f715ffff640 1 -- 172.21.15.70:0/3489513266 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716c019040 con 0x556033ca6610 2024-04-01T01:33:04.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f715ffff640 1 -- 172.21.15.70:0/3489513266 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f716c00ee50 con 0x556033ca6610 2024-04-01T01:33:04.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.675+0000 7f715ffff640 1 -- 172.21.15.70:0/3489513266 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716c0043a0 con 0x556033ca6610 2024-04-01T01:33:04.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.676+0000 7f717ed91740 1 -- 172.21.15.70:0/3489513266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 msgr2=0x556033ca6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.676+0000 7f717ed91740 1 --2- 172.21.15.70:0/3489513266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 0x556033ca6a30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f716c010d80 tx=0x7f716c00f790 comp rx=0 tx=0).stop 2024-04-01T01:33:04.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.676+0000 7f717ed91740 1 -- 172.21.15.70:0/3489513266 shutdown_connections 2024-04-01T01:33:04.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.676+0000 7f717ed91740 1 --2- 172.21.15.70:0/3489513266 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca81a0 0x556033caea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.676+0000 7f717ed91740 1 --2- 172.21.15.70:0/3489513266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca77e0 0x556033ca7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.676+0000 7f717ed91740 1 --2- 172.21.15.70:0/3489513266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 0x556033ca6a30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.676+0000 7f717ed91740 1 -- 172.21.15.70:0/3489513266 >> 172.21.15.70:0/3489513266 conn(0x556033ca1a00 msgr2=0x556033ca1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.676+0000 7f717ed91740 1 -- 172.21.15.70:0/3489513266 shutdown_connections 2024-04-01T01:33:04.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.676+0000 7f717ed91740 1 -- 172.21.15.70:0/3489513266 wait complete. 2024-04-01T01:33:04.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.676+0000 7f717ed91740 1 -- start start 2024-04-01T01:33:04.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f717ed91740 1 Processor -- start 2024-04-01T01:33:04.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f717ed91740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 0x556033cb0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f717ed91740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca77e0 0x556033cb0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f717ed91740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca81a0 0x556033cb7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f717ed91740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556033caf150 con 0x556033ca81a0 2024-04-01T01:33:04.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f717ed91740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x556033bc6f70 con 0x556033ca77e0 2024-04-01T01:33:04.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f717ed91740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556033caf2d0 con 0x556033ca6610 2024-04-01T01:33:04.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f717ed91740 10 client.0 _ll_register_callbacks cb 0x55603391eac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:04.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f717a753640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca81a0 0x556033cb7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f7179751640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca77e0 0x556033cb0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f7179f52640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 0x556033cb0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f7179751640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca77e0 0x556033cb0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53028/0 (socket says 172.21.15.70:53028) 2024-04-01T01:33:04.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f717a753640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca81a0 0x556033cb7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39838/0 (socket says 172.21.15.70:39838) 2024-04-01T01:33:04.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f7179751640 1 -- 172.21.15.70:0/747569428 learned_addr learned my addr 172.21.15.70:0/747569428 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f7179f52640 1 -- 172.21.15.70:0/747569428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca77e0 msgr2=0x556033cb0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f7179f52640 1 --2- 172.21.15.70:0/747569428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca77e0 0x556033cb0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f7179f52640 1 -- 172.21.15.70:0/747569428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca81a0 msgr2=0x556033cb7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f7179f52640 1 --2- 172.21.15.70:0/747569428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca81a0 0x556033cb7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f7179f52640 1 -- 172.21.15.70:0/747569428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560339e96b0 con 0x556033ca6610 2024-04-01T01:33:04.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f7179751640 1 --2- 172.21.15.70:0/747569428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca77e0 0x556033cb0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:04.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.677+0000 7f717a753640 1 --2- 172.21.15.70:0/747569428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca81a0 0x556033cb7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:04.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f7179f52640 1 --2- 172.21.15.70:0/747569428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 0x556033cb0920 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f71700b9c90 tx=0x7f7170004500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2edbe8a0939c1342 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f715d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:33:04.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f715d7fa640 1 -- 172.21.15.70:0/747569428 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71700d5020 con 0x556033ca6610 2024-04-01T01:33:04.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f715d7fa640 1 -- 172.21.15.70:0/747569428 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f71700054d0 con 0x556033ca6610 2024-04-01T01:33:04.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f717ed91740 1 -- 172.21.15.70:0/747569428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f716c00ee50 con 0x556033ca6610 2024-04-01T01:33:04.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f715d7fa640 1 -- 172.21.15.70:0/747569428 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f71700bce00 con 0x556033ca6610 2024-04-01T01:33:04.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f717ed91740 10 client.25024 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:04.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f717ed91740 1 -- 172.21.15.70:0/747569428 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556033caf220 con 0x556033ca6610 2024-04-01T01:33:04.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f715d7fa640 1 -- 172.21.15.70:0/747569428 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f717009e030 con 0x556033ca6610 2024-04-01T01:33:04.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f715d7fa640 1 -- 172.21.15.70:0/747569428 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f71700bdc50 con 0x556033ca6610 2024-04-01T01:33:04.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f717ed91740 10 client.25024 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:04.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f717ed91740 10 client.25024 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-01T01:33:04.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f717ed91740 20 client.25024 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:04.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f717ed91740 1 --2- 172.21.15.70:0/747569428 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556033cbd680 0x556033cbeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f717ed91740 4 client.25024 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:33:04.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f717ed91740 1 -- 172.21.15.70:0/747569428 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556033c18030 con 0x556033cbd680 2024-04-01T01:33:04.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.678+0000 7f7179751640 1 --2- 172.21.15.70:0/747569428 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556033cbd680 0x556033cbeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.679+0000 7f715d7fa640 10 client.25024 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:04.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.679+0000 7f7179751640 1 --2- 172.21.15.70:0/747569428 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556033cbd680 0x556033cbeaf0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f62c69a9e2f1ba7 server_cookie=eb5cecb115fec1b9 in_seq=0 out_seq=0 2024-04-01T01:33:04.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.679+0000 7f715d7fa640 1 -- 172.21.15.70:0/747569428 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556033c18030 con 0x556033cbd680 2024-04-01T01:33:04.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.679+0000 7f715d7fa640 10 client.25024 handle_command_reply: tid=0 2024-04-01T01:33:04.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.679+0000 7f717ed91740 1 client.25024 shutdown 2024-04-01T01:33:04.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.680+0000 7f717ed91740 1 -- 172.21.15.70:0/747569428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 msgr2=0x556033cb0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.680+0000 7f717ed91740 1 --2- 172.21.15.70:0/747569428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 0x556033cb0920 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f71700b9c90 tx=0x7f7170004500 comp rx=0 tx=0).stop 2024-04-01T01:33:04.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.680+0000 7f717ed91740 1 -- 172.21.15.70:0/747569428 shutdown_connections 2024-04-01T01:33:04.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.680+0000 7f717ed91740 1 --2- 172.21.15.70:0/747569428 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556033cbd680 0x556033cbeaf0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.680+0000 7f717ed91740 1 --2- 172.21.15.70:0/747569428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca81a0 0x556033cb7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.680+0000 7f717ed91740 1 --2- 172.21.15.70:0/747569428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca77e0 0x556033cb0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.680+0000 7f715d7fa640 0 client.25024 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:04.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.680+0000 7f717ed91740 1 --2- 172.21.15.70:0/747569428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 0x556033cb0920 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.680+0000 7f717ed91740 1 -- 172.21.15.70:0/747569428 >> 172.21.15.70:0/747569428 conn(0x556033ca1a00 msgr2=0x556033ca0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.680+0000 7f717ed91740 1 -- 172.21.15.70:0/747569428 shutdown_connections 2024-04-01T01:33:04.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.680+0000 7f717ed91740 1 -- 172.21.15.70:0/747569428 wait complete. 2024-04-01T01:33:04.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.680+0000 7f717ed91740 20 client.25024 trim_cache size 0 max 16384 2024-04-01T01:33:04.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f717ed91740 1 Processor -- start 2024-04-01T01:33:04.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f717ed91740 1 -- start start 2024-04-01T01:33:04.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f717ed91740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 0x556033cae770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f717ed91740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca77e0 0x556033cb7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f717ed91740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca81a0 0x556033cbc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f717ed91740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556033cbce30 con 0x556033ca77e0 2024-04-01T01:33:04.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f717ed91740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x556033cbd500 con 0x556033ca81a0 2024-04-01T01:33:04.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f717ed91740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556033cbdbd0 con 0x556033ca6610 2024-04-01T01:33:04.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f7179751640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca77e0 0x556033cb7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f7179751640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca77e0 0x556033cb7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39840/0 (socket says 172.21.15.70:39840) 2024-04-01T01:33:04.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f7179751640 1 -- 172.21.15.70:0/1588823699 learned_addr learned my addr 172.21.15.70:0/1588823699 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f7179f52640 1 --2- 172.21.15.70:0/1588823699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 0x556033cae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f717a753640 1 --2- 172.21.15.70:0/1588823699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca81a0 0x556033cbc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f7179751640 1 -- 172.21.15.70:0/1588823699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 msgr2=0x556033cae770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f7179751640 1 --2- 172.21.15.70:0/1588823699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 0x556033cae770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f7179751640 1 -- 172.21.15.70:0/1588823699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca81a0 msgr2=0x556033cbc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f7179751640 1 --2- 172.21.15.70:0/1588823699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca81a0 0x556033cbc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.699+0000 7f7179751640 1 -- 172.21.15.70:0/1588823699 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556033ca7260 con 0x556033ca77e0 2024-04-01T01:33:04.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.700+0000 7f7179751640 1 --2- 172.21.15.70:0/1588823699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca77e0 0x556033cb7e60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x556033cbf900 tx=0x7f7164005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2cfde2dcecb2c07 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.700+0000 7f715f7fe640 1 -- 172.21.15.70:0/1588823699 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7164004230 con 0x556033ca77e0 2024-04-01T01:33:04.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.700+0000 7f715f7fe640 1 -- 172.21.15.70:0/1588823699 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71640043d0 con 0x556033ca77e0 2024-04-01T01:33:04.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.700+0000 7f715f7fe640 1 -- 172.21.15.70:0/1588823699 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7164003cd0 con 0x556033ca77e0 2024-04-01T01:33:04.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.700+0000 7f717ed91740 1 -- 172.21.15.70:0/1588823699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca77e0 msgr2=0x556033cb7e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.700+0000 7f717ed91740 1 --2- 172.21.15.70:0/1588823699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca77e0 0x556033cb7e60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x556033cbf900 tx=0x7f7164005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:04.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717ed91740 1 -- 172.21.15.70:0/1588823699 shutdown_connections 2024-04-01T01:33:04.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717ed91740 1 --2- 172.21.15.70:0/1588823699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca81a0 0x556033cbc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717ed91740 1 --2- 172.21.15.70:0/1588823699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca77e0 0x556033cb7e60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717ed91740 1 --2- 172.21.15.70:0/1588823699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca6610 0x556033cae770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717ed91740 1 -- 172.21.15.70:0/1588823699 >> 172.21.15.70:0/1588823699 conn(0x556033ca1a00 msgr2=0x556033c9c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717ed91740 1 -- 172.21.15.70:0/1588823699 shutdown_connections 2024-04-01T01:33:04.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717ed91740 1 -- 172.21.15.70:0/1588823699 wait complete. 2024-04-01T01:33:04.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717ed91740 1 -- start start 2024-04-01T01:33:04.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717ed91740 1 Processor -- start 2024-04-01T01:33:04.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717ed91740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca6610 0x556033cb5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717ed91740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca77e0 0x556033d1f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717ed91740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca81a0 0x556033d21920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717ed91740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556033cbe970 con 0x556033ca6610 2024-04-01T01:33:04.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717ed91740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x556033cbe2a0 con 0x556033ca81a0 2024-04-01T01:33:04.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717ed91740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556033cbf040 con 0x556033ca77e0 2024-04-01T01:33:04.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717ed91740 10 client.0 _ll_register_callbacks cb 0x7f71700057a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:04.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f7179f52640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca6610 0x556033cb5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f7179f52640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca6610 0x556033cb5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39848/0 (socket says 172.21.15.70:39848) 2024-04-01T01:33:04.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f7179f52640 1 -- 172.21.15.70:0/261656346 learned_addr learned my addr 172.21.15.70:0/261656346 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:04.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f717a753640 1 --2- 172.21.15.70:0/261656346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca81a0 0x556033d21920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.701+0000 7f7179751640 1 --2- 172.21.15.70:0/261656346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca77e0 0x556033d1f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f7179f52640 1 -- 172.21.15.70:0/261656346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca77e0 msgr2=0x556033d1f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f7179f52640 1 --2- 172.21.15.70:0/261656346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca77e0 0x556033d1f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f7179f52640 1 -- 172.21.15.70:0/261656346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca81a0 msgr2=0x556033d21920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:04.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f7179f52640 1 --2- 172.21.15.70:0/261656346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca81a0 0x556033d21920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f7179f52640 1 -- 172.21.15.70:0/261656346 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556033d22130 con 0x556033ca6610 2024-04-01T01:33:04.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f717a753640 1 --2- 172.21.15.70:0/261656346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca81a0 0x556033d21920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:04.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f7179f52640 1 --2- 172.21.15.70:0/261656346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca6610 0x556033cb5e50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f71700c14b0 tx=0x7f71700095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d564cfd34580b904 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:04.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f715d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:04.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f715d7fa640 1 -- 172.21.15.70:0/261656346 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71700d5040 con 0x556033ca6610 2024-04-01T01:33:04.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f715d7fa640 1 -- 172.21.15.70:0/261656346 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7170003d20 con 0x556033ca6610 2024-04-01T01:33:04.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f717ed91740 1 -- 172.21.15.70:0/261656346 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71640043d0 con 0x556033ca6610 2024-04-01T01:33:04.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f715d7fa640 1 -- 172.21.15.70:0/261656346 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f71700bcc20 con 0x556033ca6610 2024-04-01T01:33:04.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f717ed91740 10 client.15291 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:04.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f717ed91740 1 -- 172.21.15.70:0/261656346 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556033d23410 con 0x556033ca6610 2024-04-01T01:33:04.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f715d7fa640 1 -- 172.21.15.70:0/261656346 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71640043d0 con 0x556033ca6610 2024-04-01T01:33:04.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f715d7fa640 1 -- 172.21.15.70:0/261656346 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f71700bd520 con 0x556033ca6610 2024-04-01T01:33:04.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f717ed91740 10 client.15291 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:04.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f717ed91740 10 client.15291 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-01T01:33:04.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f717ed91740 20 client.15291 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:04.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f717ed91740 1 --2- 172.21.15.70:0/261656346 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556033d27d20 0x556033d26a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:04.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f717ed91740 4 client.15291 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-01T01:33:04.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.702+0000 7f717ed91740 1 -- 172.21.15.70:0/261656346 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "status"}) -- 0x556033c12380 con 0x556033d27d20 2024-04-01T01:33:04.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.703+0000 7f7179751640 1 --2- 172.21.15.70:0/261656346 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556033d27d20 0x556033d26a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:04.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.703+0000 7f715d7fa640 10 client.15291 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:04.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.703+0000 7f7179751640 1 --2- 172.21.15.70:0/261656346 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556033d27d20 0x556033d26a30 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecf7c87e4b3c33a0 server_cookie=91c32ea90350248 in_seq=0 out_seq=0 2024-04-01T01:33:04.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.703+0000 7f715d7fa640 1 -- 172.21.15.70:0/261656346 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x556033c12380 con 0x556033d27d20 2024-04-01T01:33:04.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.703+0000 7f715d7fa640 10 client.15291 handle_command_reply: tid=0 2024-04-01T01:33:04.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.703+0000 7f717ed91740 1 client.15291 shutdown 2024-04-01T01:33:04.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.703+0000 7f717ed91740 1 -- 172.21.15.70:0/261656346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca6610 msgr2=0x556033cb5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.703+0000 7f717ed91740 1 --2- 172.21.15.70:0/261656346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca6610 0x556033cb5e50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f71700c14b0 tx=0x7f71700095e0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.703+0000 7f717ed91740 1 -- 172.21.15.70:0/261656346 shutdown_connections 2024-04-01T01:33:04.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.703+0000 7f717ed91740 1 --2- 172.21.15.70:0/261656346 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556033d27d20 0x556033d26a30 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717ed91740 1 --2- 172.21.15.70:0/261656346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556033ca81a0 0x556033d21920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717ed91740 1 --2- 172.21.15.70:0/261656346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556033ca77e0 0x556033d1f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717ed91740 1 --2- 172.21.15.70:0/261656346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556033ca6610 0x556033cb5e50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f715d7fa640 0 client.15291 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:04.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717ed91740 1 -- 172.21.15.70:0/261656346 >> 172.21.15.70:0/261656346 conn(0x556033ca1a00 msgr2=0x556033cab1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717ed91740 1 -- 172.21.15.70:0/261656346 shutdown_connections 2024-04-01T01:33:04.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717ed91740 1 -- 172.21.15.70:0/261656346 wait complete. 2024-04-01T01:33:04.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717ed91740 20 client.15291 trim_cache size 0 max 16384 2024-04-01T01:33:04.711 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:33:04.711 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "36577394-efc6-11ee-b647-cb9ed24678a4", 2024-04-01T01:33:04.711 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-04-01T01:33:04.711 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24464, 2024-04-01T01:33:04.712 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-01T01:33:04.712 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-01T01:33:04.712 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-01T01:33:04.712 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 108.16978340599999, 2024-04-01T01:33:04.712 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-04-01T01:33:04.712 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 91, 2024-04-01T01:33:04.712 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 89, 2024-04-01T01:33:04.712 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 113.67327735000001 2024-04-01T01:33:04.712 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:33:04.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717c1b6640 1 -- 172.21.15.70:0/1258298696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f71600a24a0 msgr2=0x7f71600a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717c1b6640 1 --2- 172.21.15.70:0/1258298696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f71600a24a0 0x7f71600a4960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f716c00fc90 tx=0x7f716c018000 comp rx=0 tx=0).stop 2024-04-01T01:33:04.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717c1b6640 1 -- 172.21.15.70:0/1258298696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7174100670 msgr2=0x7f71741979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:04.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717c1b6640 1 --2- 172.21.15.70:0/1258298696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7174100670 0x7f71741979d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7170000c00 tx=0x7f7170005dc0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717c1b6640 1 -- 172.21.15.70:0/1258298696 shutdown_connections 2024-04-01T01:33:04.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717c1b6640 1 --2- 172.21.15.70:0/1258298696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f71600a24a0 0x7f71600a4960 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717c1b6640 1 --2- 172.21.15.70:0/1258298696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7174102220 0x7f717419ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717c1b6640 1 --2- 172.21.15.70:0/1258298696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7174101860 0x7f7174197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717c1b6640 1 --2- 172.21.15.70:0/1258298696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7174100670 0x7f71741979d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:04.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717c1b6640 1 -- 172.21.15.70:0/1258298696 >> 172.21.15.70:0/1258298696 conn(0x7f71740fbc00 msgr2=0x7f7174104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:04.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717c1b6640 1 -- 172.21.15.70:0/1258298696 shutdown_connections 2024-04-01T01:33:04.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:04.704+0000 7f717c1b6640 1 -- 172.21.15.70:0/1258298696 wait complete. 2024-04-01T01:33:04.715 INFO:tasks.cephadm:Setup complete, yielding 2024-04-01T01:33:04.716 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-01T01:33:04.726 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi070.front.sepia.ceph.com 2024-04-01T01:33:04.728 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch status' 2024-04-01T01:33:04.965 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:33:04.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:04 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3495912707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:05.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:04 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3495912707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:05.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:04 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3495912707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:05.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:05 smithi189 ceph-mon[35753]: pgmap v344: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-01T01:33:06.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:05 smithi070 ceph-mon[32179]: pgmap v344: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-01T01:33:06.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:05 smithi081 ceph-mon[35142]: pgmap v344: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-01T01:33:06.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.735+0000 7f931ae81700 1 -- 172.21.15.70:0/929970831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93141024b0 msgr2=0x7f9314108d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:06.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.735+0000 7f931ae81700 1 --2- 172.21.15.70:0/929970831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93141024b0 0x7f9314108d60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f930c00c240 tx=0x7f930c00c550 comp rx=0 tx=0).stop 2024-04-01T01:33:06.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.735+0000 7f931ae81700 1 -- 172.21.15.70:0/929970831 shutdown_connections 2024-04-01T01:33:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.735+0000 7f931ae81700 1 --2- 172.21.15.70:0/929970831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93141024b0 0x7f9314108d60 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.735+0000 7f931ae81700 1 --2- 172.21.15.70:0/929970831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9314100910 0x7f9314100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.735+0000 7f931ae81700 1 --2- 172.21.15.70:0/929970831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9314101b10 0x7f9314101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.735+0000 7f931ae81700 1 -- 172.21.15.70:0/929970831 >> 172.21.15.70:0/929970831 conn(0x7f93140fbe90 msgr2=0x7f93140fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.735+0000 7f931ae81700 1 -- 172.21.15.70:0/929970831 shutdown_connections 2024-04-01T01:33:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.735+0000 7f931ae81700 1 -- 172.21.15.70:0/929970831 wait complete. 2024-04-01T01:33:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f931ae81700 1 Processor -- start 2024-04-01T01:33:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f931ae81700 1 -- start start 2024-04-01T01:33:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f931ae81700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9314100910 0x7f9314198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f931ae81700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9314101b10 0x7f93141988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f931ae81700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93141024b0 0x7f931419f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f931ae81700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f931410baf0 con 0x7f93141024b0 2024-04-01T01:33:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f931ae81700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f931410b970 con 0x7f9314100910 2024-04-01T01:33:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f931ae81700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f931410bc70 con 0x7f9314101b10 2024-04-01T01:33:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f931941e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93141024b0 0x7f931419f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f9318c1d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9314100910 0x7f9314198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f931941e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93141024b0 0x7f931419f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42582/0 (socket says 172.21.15.70:42582) 2024-04-01T01:33:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f931941e700 1 -- 172.21.15.70:0/1265490423 learned_addr learned my addr 172.21.15.70:0/1265490423 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f9313fff700 1 --2- 172.21.15.70:0/1265490423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9314101b10 0x7f93141988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f931941e700 1 -- 172.21.15.70:0/1265490423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9314101b10 msgr2=0x7f93141988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f931941e700 1 --2- 172.21.15.70:0/1265490423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9314101b10 0x7f93141988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f931941e700 1 -- 172.21.15.70:0/1265490423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9314100910 msgr2=0x7f9314198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f931941e700 1 --2- 172.21.15.70:0/1265490423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9314100910 0x7f9314198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.736+0000 7f931941e700 1 -- 172.21.15.70:0/1265490423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f931419ffd0 con 0x7f93141024b0 2024-04-01T01:33:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.737+0000 7f931941e700 1 --2- 172.21.15.70:0/1265490423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93141024b0 0x7f931419f930 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f930c005a60 tx=0x7f930c005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.737+0000 7f9311ffb700 1 -- 172.21.15.70:0/1265490423 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c022020 con 0x7f93141024b0 2024-04-01T01:33:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.737+0000 7f931ae81700 1 -- 172.21.15.70:0/1265490423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93141a0260 con 0x7f93141024b0 2024-04-01T01:33:06.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.737+0000 7f9311ffb700 1 -- 172.21.15.70:0/1265490423 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f930c006040 con 0x7f93141024b0 2024-04-01T01:33:06.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.737+0000 7f9311ffb700 1 -- 172.21.15.70:0/1265490423 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c01e940 con 0x7f93141024b0 2024-04-01T01:33:06.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.737+0000 7f931ae81700 1 -- 172.21.15.70:0/1265490423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93141a06c0 con 0x7f93141024b0 2024-04-01T01:33:06.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.738+0000 7f931ae81700 1 -- 172.21.15.70:0/1265490423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9314068f00 con 0x7f93141024b0 2024-04-01T01:33:06.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.739+0000 7f9311ffb700 1 -- 172.21.15.70:0/1265490423 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f930c025020 con 0x7f93141024b0 2024-04-01T01:33:06.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.740+0000 7f9311ffb700 1 --2- 172.21.15.70:0/1265490423 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93040a2530 0x7f93040a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:06.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.740+0000 7f9318c1d700 1 --2- 172.21.15.70:0/1265490423 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93040a2530 0x7f93040a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:06.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.740+0000 7f9311ffb700 1 -- 172.21.15.70:0/1265490423 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f930c0d0930 con 0x7f93141024b0 2024-04-01T01:33:06.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.740+0000 7f9318c1d700 1 --2- 172.21.15.70:0/1265490423 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93040a2530 0x7f93040a49f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9314101970 tx=0x7f9308009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:06.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.744+0000 7f9311ffb700 1 -- 172.21.15.70:0/1265490423 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f930c02c020 con 0x7f93141024b0 2024-04-01T01:33:06.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.953+0000 7f931ae81700 1 -- 172.21.15.70:0/1265490423 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f93141086e0 con 0x7f93040a2530 2024-04-01T01:33:06.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.954+0000 7f9311ffb700 1 -- 172.21.15.70:0/1265490423 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f93141086e0 con 0x7f93040a2530 2024-04-01T01:33:06.954 INFO:teuthology.orchestra.run.smithi070.stdout:Backend: cephadm 2024-04-01T01:33:06.954 INFO:teuthology.orchestra.run.smithi070.stdout:Available: Yes 2024-04-01T01:33:06.954 INFO:teuthology.orchestra.run.smithi070.stdout:Paused: No 2024-04-01T01:33:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.956+0000 7f931ae81700 1 -- 172.21.15.70:0/1265490423 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93040a2530 msgr2=0x7f93040a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.956+0000 7f931ae81700 1 --2- 172.21.15.70:0/1265490423 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93040a2530 0x7f93040a49f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9314101970 tx=0x7f9308009260 comp rx=0 tx=0).stop 2024-04-01T01:33:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.956+0000 7f931ae81700 1 -- 172.21.15.70:0/1265490423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93141024b0 msgr2=0x7f931419f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.956+0000 7f931ae81700 1 --2- 172.21.15.70:0/1265490423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93141024b0 0x7f931419f930 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f930c005a60 tx=0x7f930c005a90 comp rx=0 tx=0).stop 2024-04-01T01:33:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.956+0000 7f931ae81700 1 -- 172.21.15.70:0/1265490423 shutdown_connections 2024-04-01T01:33:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.956+0000 7f931ae81700 1 --2- 172.21.15.70:0/1265490423 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93040a2530 0x7f93040a49f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.956+0000 7f931ae81700 1 --2- 172.21.15.70:0/1265490423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93141024b0 0x7f931419f930 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.956+0000 7f931ae81700 1 --2- 172.21.15.70:0/1265490423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9314100910 0x7f9314198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.956+0000 7f931ae81700 1 --2- 172.21.15.70:0/1265490423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9314101b10 0x7f93141988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.956+0000 7f931ae81700 1 -- 172.21.15.70:0/1265490423 >> 172.21.15.70:0/1265490423 conn(0x7f93140fbe90 msgr2=0x7f9314106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.956+0000 7f931ae81700 1 -- 172.21.15.70:0/1265490423 shutdown_connections 2024-04-01T01:33:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:06.956+0000 7f931ae81700 1 -- 172.21.15.70:0/1265490423 wait complete. 2024-04-01T01:33:07.784 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch ps' 2024-04-01T01:33:07.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:07 smithi189 ceph-mon[35753]: pgmap v345: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:33:08.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:07 smithi070 ceph-mon[32179]: pgmap v345: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:33:08.023 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:33:08.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:07 smithi081 ceph-mon[35142]: pgmap v345: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:33:08.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:08 smithi189 ceph-mon[35753]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:09.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:08 smithi070 ceph-mon[32179]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:09.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:08 smithi081 ceph-mon[35142]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.854+0000 7feec299e700 1 -- 172.21.15.70:0/3292885685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feebc1010a0 msgr2=0x7feebc1014c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.854+0000 7feec299e700 1 --2- 172.21.15.70:0/3292885685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feebc1010a0 0x7feebc1014c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7feeac0099c0 tx=0x7feeac009cd0 comp rx=0 tx=0).stop 2024-04-01T01:33:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.854+0000 7feec299e700 1 -- 172.21.15.70:0/3292885685 shutdown_connections 2024-04-01T01:33:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.854+0000 7feec299e700 1 --2- 172.21.15.70:0/3292885685 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feebc102c40 0x7feebc1094f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.854+0000 7feec299e700 1 --2- 172.21.15.70:0/3292885685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feebc1010a0 0x7feebc1014c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:09.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.854+0000 7feec299e700 1 --2- 172.21.15.70:0/3292885685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feebc1022a0 0x7feebc102700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:09.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.854+0000 7feec299e700 1 -- 172.21.15.70:0/3292885685 >> 172.21.15.70:0/3292885685 conn(0x7feebc0fc620 msgr2=0x7feebc0fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:09.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.855+0000 7feec299e700 1 -- 172.21.15.70:0/3292885685 shutdown_connections 2024-04-01T01:33:09.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.855+0000 7feec299e700 1 -- 172.21.15.70:0/3292885685 wait complete. 2024-04-01T01:33:09.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.855+0000 7feec299e700 1 Processor -- start 2024-04-01T01:33:09.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.855+0000 7feec299e700 1 -- start start 2024-04-01T01:33:09.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec299e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feebc1010a0 0x7feebc104fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:09.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec299e700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feebc1022a0 0x7feebc103680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:09.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec299e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feebc102c40 0x7feebc103bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:09.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec299e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7feebc10cb30 con 0x7feebc1010a0 2024-04-01T01:33:09.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec299e700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7feebc10c9b0 con 0x7feebc1022a0 2024-04-01T01:33:09.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec299e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7feebc10ccb0 con 0x7feebc102c40 2024-04-01T01:33:09.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec199c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feebc1010a0 0x7feebc104fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:09.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec199c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feebc1010a0 0x7feebc104fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42598/0 (socket says 172.21.15.70:42598) 2024-04-01T01:33:09.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec199c700 1 -- 172.21.15.70:0/3552322684 learned_addr learned my addr 172.21.15.70:0/3552322684 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:09.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec119b700 1 --2- 172.21.15.70:0/3552322684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feebc1022a0 0x7feebc103680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:09.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec219d700 1 --2- 172.21.15.70:0/3552322684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feebc102c40 0x7feebc103bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:09.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec199c700 1 -- 172.21.15.70:0/3552322684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feebc102c40 msgr2=0x7feebc103bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:09.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec199c700 1 --2- 172.21.15.70:0/3552322684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feebc102c40 0x7feebc103bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:09.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec199c700 1 -- 172.21.15.70:0/3552322684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feebc1022a0 msgr2=0x7feebc103680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:09.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec199c700 1 --2- 172.21.15.70:0/3552322684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feebc1022a0 0x7feebc103680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:09.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec199c700 1 -- 172.21.15.70:0/3552322684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feebc104490 con 0x7feebc1010a0 2024-04-01T01:33:09.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec119b700 1 --2- 172.21.15.70:0/3552322684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feebc1022a0 0x7feebc103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:09.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec199c700 1 --2- 172.21.15.70:0/3552322684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feebc1010a0 0x7feebc104fe0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7feeac000c00 tx=0x7feeac00fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:09.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feeb2ffd700 1 -- 172.21.15.70:0/3552322684 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac027070 con 0x7feebc1010a0 2024-04-01T01:33:09.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec299e700 1 -- 172.21.15.70:0/3552322684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feebc1a23b0 con 0x7feebc1010a0 2024-04-01T01:33:09.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feeb2ffd700 1 -- 172.21.15.70:0/3552322684 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feeac00fe00 con 0x7feebc1010a0 2024-04-01T01:33:09.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feeb2ffd700 1 -- 172.21.15.70:0/3552322684 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac017650 con 0x7feebc1010a0 2024-04-01T01:33:09.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.856+0000 7feec299e700 1 -- 172.21.15.70:0/3552322684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feebc1a2860 con 0x7feebc1010a0 2024-04-01T01:33:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.857+0000 7feec299e700 1 -- 172.21.15.70:0/3552322684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feebc068f00 con 0x7feebc1010a0 2024-04-01T01:33:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.858+0000 7feeb2ffd700 1 -- 172.21.15.70:0/3552322684 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7feeac02a070 con 0x7feebc1010a0 2024-04-01T01:33:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.859+0000 7feeb2ffd700 1 --2- 172.21.15.70:0/3552322684 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7feea80a2530 0x7feea80a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.859+0000 7feec119b700 1 --2- 172.21.15.70:0/3552322684 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7feea80a2530 0x7feea80a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.859+0000 7feeb2ffd700 1 -- 172.21.15.70:0/3552322684 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7feeac0d0000 con 0x7feebc1010a0 2024-04-01T01:33:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.859+0000 7feec119b700 1 --2- 172.21.15.70:0/3552322684 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7feea80a2530 0x7feea80a49f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7feeb8005950 tx=0x7feeb800a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:09.862+0000 7feeb2ffd700 1 -- 172.21.15.70:0/3552322684 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7feeac031070 con 0x7feebc1010a0 2024-04-01T01:33:09.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:09 smithi189 ceph-mon[35753]: pgmap v346: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-01T01:33:10.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:09 smithi070 ceph-mon[32179]: pgmap v346: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-01T01:33:10.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:10.066+0000 7feec299e700 1 -- 172.21.15.70:0/3552322684 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7feebc0fe840 con 0x7feea80a2530 2024-04-01T01:33:10.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:09 smithi081 ceph-mon[35142]: pgmap v346: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-01T01:33:10.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:10.080+0000 7feeb2ffd700 1 -- 172.21.15.70:0/3552322684 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7feebc0fe840 con 0x7feea80a2530 2024-04-01T01:33:10.080 INFO:teuthology.orchestra.run.smithi070.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-01T01:33:10.080 INFO:teuthology.orchestra.run.smithi070.stdout:mds.a smithi070 running (2m) 19s ago 2m 16.4M - 19.0.0-1977-g384f31f3 ec18d7baa60e 08971e5b39e2 2024-04-01T01:33:10.080 INFO:teuthology.orchestra.run.smithi070.stdout:mds.b smithi081 running (119s) 52s ago 118s 25.6M - 19.0.0-1977-g384f31f3 ec18d7baa60e de2b53065129 2024-04-01T01:33:10.080 INFO:teuthology.orchestra.run.smithi070.stdout:mds.c smithi189 running (2m) 52s ago 2m 16.3M - 19.0.0-1977-g384f31f3 ec18d7baa60e a3967370277d 2024-04-01T01:33:10.080 INFO:teuthology.orchestra.run.smithi070.stdout:mds.d smithi070 running (2m) 19s ago 2m 15.4M - 19.0.0-1977-g384f31f3 ec18d7baa60e c36c2d6d7d07 2024-04-01T01:33:10.081 INFO:teuthology.orchestra.run.smithi070.stdout:mds.e smithi081 running (2m) 52s ago 2m 17.1M - 19.0.0-1977-g384f31f3 ec18d7baa60e bb9382b3a3c1 2024-04-01T01:33:10.081 INFO:teuthology.orchestra.run.smithi070.stdout:mds.f smithi189 running (2m) 52s ago 2m 17.6M - 19.0.0-1977-g384f31f3 ec18d7baa60e bc506a7755a3 2024-04-01T01:33:10.081 INFO:teuthology.orchestra.run.smithi070.stdout:mds.g smithi070 running (2m) 19s ago 2m 18.9M - 19.0.0-1977-g384f31f3 ec18d7baa60e c02fe3619d88 2024-04-01T01:33:10.081 INFO:teuthology.orchestra.run.smithi070.stdout:mds.h smithi081 running (2m) 52s ago 2m 16.2M - 19.0.0-1977-g384f31f3 ec18d7baa60e 720bfad3a84a 2024-04-01T01:33:10.081 INFO:teuthology.orchestra.run.smithi070.stdout:mds.i smithi189 running (2m) 52s ago 2m 17.7M - 19.0.0-1977-g384f31f3 ec18d7baa60e 50ee14cddb4c 2024-04-01T01:33:10.081 INFO:teuthology.orchestra.run.smithi070.stdout:mds.j smithi070 running (2m) 19s ago 2m 16.1M - 19.0.0-1977-g384f31f3 ec18d7baa60e bfab699b9b08 2024-04-01T01:33:10.081 INFO:teuthology.orchestra.run.smithi070.stdout:mds.k smithi081 running (2m) 52s ago 2m 16.3M - 19.0.0-1977-g384f31f3 ec18d7baa60e 8da18107b3c1 2024-04-01T01:33:10.081 INFO:teuthology.orchestra.run.smithi070.stdout:mds.l smithi189 running (2m) 52s ago 2m 17.2M - 19.0.0-1977-g384f31f3 ec18d7baa60e d300843bb0a8 2024-04-01T01:33:10.081 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.x smithi070 *:9283,8765 running (9m) 19s ago 9m 513M - 19.0.0-1977-g384f31f3 ec18d7baa60e aed29d5f4240 2024-04-01T01:33:10.081 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.y smithi081 *:8443,8765 running (7m) 52s ago 7m 452M - 19.0.0-1977-g384f31f3 ec18d7baa60e 2e186bfb704a 2024-04-01T01:33:10.081 INFO:teuthology.orchestra.run.smithi070.stdout:mgr.z smithi189 *:8443,8765 running (7m) 52s ago 7m 450M - 19.0.0-1977-g384f31f3 ec18d7baa60e d852bc645259 2024-04-01T01:33:10.081 INFO:teuthology.orchestra.run.smithi070.stdout:mon.a smithi070 running (9m) 19s ago 9m 52.1M 2048M 19.0.0-1977-g384f31f3 ec18d7baa60e 3284b805c945 2024-04-01T01:33:10.081 INFO:teuthology.orchestra.run.smithi070.stdout:mon.b smithi081 running (7m) 52s ago 7m 41.6M 2048M 19.0.0-1977-g384f31f3 ec18d7baa60e be19c90532f5 2024-04-01T01:33:10.081 INFO:teuthology.orchestra.run.smithi070.stdout:mon.c smithi189 running (8m) 52s ago 8m 39.7M 2048M 19.0.0-1977-g384f31f3 ec18d7baa60e 72e7bf301c27 2024-04-01T01:33:10.082 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 smithi070 running (7m) 19s ago 7m 48.6M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e 47d3ee1d77ea 2024-04-01T01:33:10.082 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 smithi081 running (6m) 52s ago 6m 47.2M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e 86cb72dad9a0 2024-04-01T01:33:10.082 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 smithi189 running (6m) 52s ago 6m 46.8M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e aa94aee2b9ef 2024-04-01T01:33:10.082 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 smithi070 running (6m) 19s ago 6m 44.0M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e d1c1edad15b7 2024-04-01T01:33:10.082 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 smithi081 running (5m) 52s ago 5m 42.7M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e a1392b6f479d 2024-04-01T01:33:10.082 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 smithi189 running (5m) 52s ago 5m 44.7M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e 330634e99b31 2024-04-01T01:33:10.082 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 smithi070 running (4m) 19s ago 4m 45.6M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e 8b179eb1de43 2024-04-01T01:33:10.082 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 smithi081 running (4m) 52s ago 4m 55.3M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e bdc99f5b0b6c 2024-04-01T01:33:10.082 INFO:teuthology.orchestra.run.smithi070.stdout:osd.8 smithi189 running (4m) 52s ago 4m 43.4M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e 0b8ebf409a26 2024-04-01T01:33:10.082 INFO:teuthology.orchestra.run.smithi070.stdout:osd.9 smithi070 running (3m) 19s ago 3m 42.9M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e ae88f7ee8a94 2024-04-01T01:33:10.082 INFO:teuthology.orchestra.run.smithi070.stdout:osd.10 smithi081 running (3m) 52s ago 3m 52.0M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e c35bfefbd4ef 2024-04-01T01:33:10.082 INFO:teuthology.orchestra.run.smithi070.stdout:osd.11 smithi189 running (2m) 52s ago 2m 51.8M 4272M 19.0.0-1977-g384f31f3 ec18d7baa60e f64bcd3aeb89 2024-04-01T01:33:10.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:10.082+0000 7feec299e700 1 -- 172.21.15.70:0/3552322684 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7feea80a2530 msgr2=0x7feea80a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:10.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:10.082+0000 7feec299e700 1 --2- 172.21.15.70:0/3552322684 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7feea80a2530 0x7feea80a49f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7feeb8005950 tx=0x7feeb800a300 comp rx=0 tx=0).stop 2024-04-01T01:33:10.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:10.082+0000 7feec299e700 1 -- 172.21.15.70:0/3552322684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feebc1010a0 msgr2=0x7feebc104fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:10.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:10.082+0000 7feec299e700 1 --2- 172.21.15.70:0/3552322684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feebc1010a0 0x7feebc104fe0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7feeac000c00 tx=0x7feeac00fa00 comp rx=0 tx=0).stop 2024-04-01T01:33:10.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:10.082+0000 7feec299e700 1 -- 172.21.15.70:0/3552322684 shutdown_connections 2024-04-01T01:33:10.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:10.082+0000 7feec299e700 1 --2- 172.21.15.70:0/3552322684 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7feea80a2530 0x7feea80a49f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:10.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:10.082+0000 7feec299e700 1 --2- 172.21.15.70:0/3552322684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feebc102c40 0x7feebc103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:10.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:10.082+0000 7feec299e700 1 --2- 172.21.15.70:0/3552322684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feebc1010a0 0x7feebc104fe0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:10.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:10.082+0000 7feec299e700 1 --2- 172.21.15.70:0/3552322684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feebc1022a0 0x7feebc103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:10.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:10.082+0000 7feec299e700 1 -- 172.21.15.70:0/3552322684 >> 172.21.15.70:0/3552322684 conn(0x7feebc0fc620 msgr2=0x7feebc0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:10.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:10.082+0000 7feec299e700 1 -- 172.21.15.70:0/3552322684 shutdown_connections 2024-04-01T01:33:10.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:10.082+0000 7feec299e700 1 -- 172.21.15.70:0/3552322684 wait complete. 2024-04-01T01:33:10.878 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch ls' 2024-04-01T01:33:11.120 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:33:11.889 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:11 smithi070 ceph-mon[32179]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:11.889 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:11 smithi070 ceph-mon[32179]: pgmap v347: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-01T01:33:11.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:11 smithi189 ceph-mon[35753]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:11.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:11 smithi189 ceph-mon[35753]: pgmap v347: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-01T01:33:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:11 smithi081 ceph-mon[35142]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:11 smithi081 ceph-mon[35142]: pgmap v347: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-01T01:33:12.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.847+0000 7f85537e0700 1 -- 172.21.15.70:0/1186300278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f854c103e00 msgr2=0x7f854c10a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:12.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.847+0000 7f85537e0700 1 --2- 172.21.15.70:0/1186300278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f854c103e00 0x7f854c10a6b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f854400b0d0 tx=0x7f854400b3e0 comp rx=0 tx=0).stop 2024-04-01T01:33:12.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.847+0000 7f85537e0700 1 -- 172.21.15.70:0/1186300278 shutdown_connections 2024-04-01T01:33:12.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.847+0000 7f85537e0700 1 --2- 172.21.15.70:0/1186300278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f854c103e00 0x7f854c10a6b0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:12.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.847+0000 7f85537e0700 1 --2- 172.21.15.70:0/1186300278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c102260 0x7f854c102680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:12.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.847+0000 7f85537e0700 1 --2- 172.21.15.70:0/1186300278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854c103460 0x7f854c1038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:12.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.847+0000 7f85537e0700 1 -- 172.21.15.70:0/1186300278 >> 172.21.15.70:0/1186300278 conn(0x7f854c0fd820 msgr2=0x7f854c0ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:12.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.847+0000 7f85537e0700 1 -- 172.21.15.70:0/1186300278 shutdown_connections 2024-04-01T01:33:12.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.847+0000 7f85537e0700 1 -- 172.21.15.70:0/1186300278 wait complete. 2024-04-01T01:33:12.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.848+0000 7f85537e0700 1 Processor -- start 2024-04-01T01:33:12.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.848+0000 7f85537e0700 1 -- start start 2024-04-01T01:33:12.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.848+0000 7f85537e0700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c102260 0x7f854c198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:12.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.848+0000 7f85537e0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854c103460 0x7f854c198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:12.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.848+0000 7f85537e0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f854c103e00 0x7f854c19f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.848+0000 7f85537e0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f854c10dcf0 con 0x7f854c103e00 2024-04-01T01:33:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.848+0000 7f85537e0700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f854c10db70 con 0x7f854c102260 2024-04-01T01:33:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.848+0000 7f85537e0700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f854c10de70 con 0x7f854c103460 2024-04-01T01:33:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.848+0000 7f8551d7d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f854c103e00 0x7f854c19f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.848+0000 7f855157c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c102260 0x7f854c198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.848+0000 7f8550d7b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854c103460 0x7f854c198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.848+0000 7f855157c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c102260 0x7f854c198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47968/0 (socket says 172.21.15.70:47968) 2024-04-01T01:33:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.848+0000 7f855157c700 1 -- 172.21.15.70:0/3197625138 learned_addr learned my addr 172.21.15.70:0/3197625138 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.848+0000 7f8551d7d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f854c103e00 0x7f854c19f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42614/0 (socket says 172.21.15.70:42614) 2024-04-01T01:33:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.849+0000 7f855157c700 1 -- 172.21.15.70:0/3197625138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854c103460 msgr2=0x7f854c198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.849+0000 7f855157c700 1 --2- 172.21.15.70:0/3197625138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854c103460 0x7f854c198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.849+0000 7f855157c700 1 -- 172.21.15.70:0/3197625138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f854c103e00 msgr2=0x7f854c19f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.849+0000 7f855157c700 1 --2- 172.21.15.70:0/3197625138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f854c103e00 0x7f854c19f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:12.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.849+0000 7f855157c700 1 -- 172.21.15.70:0/3197625138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854c19ff60 con 0x7f854c102260 2024-04-01T01:33:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.849+0000 7f8551d7d700 1 --2- 172.21.15.70:0/3197625138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f854c103e00 0x7f854c19f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.849+0000 7f855157c700 1 --2- 172.21.15.70:0/3197625138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c102260 0x7f854c198300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8548009c80 tx=0x7f8548009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.849+0000 7f85427fc700 1 -- 172.21.15.70:0/3197625138 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854800bae0 con 0x7f854c102260 2024-04-01T01:33:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.849+0000 7f85427fc700 1 -- 172.21.15.70:0/3197625138 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85480044d0 con 0x7f854c102260 2024-04-01T01:33:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.849+0000 7f85537e0700 1 -- 172.21.15.70:0/3197625138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f854c0744d0 con 0x7f854c102260 2024-04-01T01:33:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.849+0000 7f85427fc700 1 -- 172.21.15.70:0/3197625138 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8548003e30 con 0x7f854c102260 2024-04-01T01:33:12.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.849+0000 7f85537e0700 1 -- 172.21.15.70:0/3197625138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f854c074a10 con 0x7f854c102260 2024-04-01T01:33:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.850+0000 7f85537e0700 1 -- 172.21.15.70:0/3197625138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f854c068f00 con 0x7f854c102260 2024-04-01T01:33:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.851+0000 7f85427fc700 1 -- 172.21.15.70:0/3197625138 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f8548020020 con 0x7f854c102260 2024-04-01T01:33:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.852+0000 7f85427fc700 1 --2- 172.21.15.70:0/3197625138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85380a2530 0x7f85380a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.852+0000 7f8550d7b700 1 --2- 172.21.15.70:0/3197625138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85380a2530 0x7f85380a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.852+0000 7f85427fc700 1 -- 172.21.15.70:0/3197625138 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f85480d2270 con 0x7f854c102260 2024-04-01T01:33:12.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.852+0000 7f8550d7b700 1 --2- 172.21.15.70:0/3197625138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85380a2530 0x7f85380a49f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f853c009c60 tx=0x7f853c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:12.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:12.856+0000 7f85427fc700 1 -- 172.21.15.70:0/3197625138 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f854809f070 con 0x7f854c102260 2024-04-01T01:33:13.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:13.091+0000 7f85537e0700 1 -- 172.21.15.70:0/3197625138 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f854c075e60 con 0x7f85380a2530 2024-04-01T01:33:13.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:13.097+0000 7f85427fc700 1 -- 172.21.15.70:0/3197625138 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f854c075e60 con 0x7f85380a2530 2024-04-01T01:33:13.097 INFO:teuthology.orchestra.run.smithi070.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-01T01:33:13.098 INFO:teuthology.orchestra.run.smithi070.stdout:mds.all 12/12 55s ago 2m smithi070=a;smithi070=d;smithi070=g;smithi070=j;smithi081=b;smithi081=e;smithi081=h;smithi081=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-04-01T01:33:13.098 INFO:teuthology.orchestra.run.smithi070.stdout:mgr 3/3 55s ago 7m smithi070=x;smithi081=y;smithi189=z;count:3 2024-04-01T01:33:13.098 INFO:teuthology.orchestra.run.smithi070.stdout:mon 3/3 55s ago 8m smithi070:172.21.15.70=a;smithi081:172.21.15.81=b;smithi189:172.21.15.189=c;count:3 2024-04-01T01:33:13.098 INFO:teuthology.orchestra.run.smithi070.stdout:osd 12 55s ago - 2024-04-01T01:33:13.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:13.099+0000 7f85537e0700 1 -- 172.21.15.70:0/3197625138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85380a2530 msgr2=0x7f85380a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:13.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:13.099+0000 7f85537e0700 1 --2- 172.21.15.70:0/3197625138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85380a2530 0x7f85380a49f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f853c009c60 tx=0x7f853c009260 comp rx=0 tx=0).stop 2024-04-01T01:33:13.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:13.099+0000 7f85537e0700 1 -- 172.21.15.70:0/3197625138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c102260 msgr2=0x7f854c198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:13.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:13.099+0000 7f85537e0700 1 --2- 172.21.15.70:0/3197625138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c102260 0x7f854c198300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8548009c80 tx=0x7f8548009f90 comp rx=0 tx=0).stop 2024-04-01T01:33:13.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:13.099+0000 7f85537e0700 1 -- 172.21.15.70:0/3197625138 shutdown_connections 2024-04-01T01:33:13.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:13.099+0000 7f85537e0700 1 --2- 172.21.15.70:0/3197625138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85380a2530 0x7f85380a49f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:13.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:13.099+0000 7f85537e0700 1 --2- 172.21.15.70:0/3197625138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f854c103e00 0x7f854c19f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:13.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:13.099+0000 7f85537e0700 1 --2- 172.21.15.70:0/3197625138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f854c102260 0x7f854c198300 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:13.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:13.099+0000 7f85537e0700 1 --2- 172.21.15.70:0/3197625138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854c103460 0x7f854c198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:13.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:13.099+0000 7f85537e0700 1 -- 172.21.15.70:0/3197625138 >> 172.21.15.70:0/3197625138 conn(0x7f854c0fd820 msgr2=0x7f854c106760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:13.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:13.100+0000 7f85537e0700 1 -- 172.21.15.70:0/3197625138 shutdown_connections 2024-04-01T01:33:13.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:13.100+0000 7f85537e0700 1 -- 172.21.15.70:0/3197625138 wait complete. 2024-04-01T01:33:13.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:13 smithi070 ceph-mon[32179]: pgmap v348: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-01T01:33:13.941 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch host ls' 2024-04-01T01:33:13.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:13 smithi189 ceph-mon[35753]: pgmap v348: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-01T01:33:14.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:13 smithi081 ceph-mon[35142]: pgmap v348: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-01T01:33:14.183 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:33:14.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:14 smithi189 ceph-mon[35753]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:15.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:14 smithi070 ceph-mon[32179]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:15.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:14 smithi081 ceph-mon[35142]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:15.959 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:15 smithi070 ceph-mon[32179]: pgmap v349: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-01T01:33:15.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:15 smithi189 ceph-mon[35753]: pgmap v349: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-01T01:33:16.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.061+0000 7f44d3cc2700 1 -- 172.21.15.70:0/437439687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc103460 msgr2=0x7f44cc1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:16.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.061+0000 7f44d3cc2700 1 --2- 172.21.15.70:0/437439687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc103460 0x7f44cc1038c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f44bc0099c0 tx=0x7f44bc009cd0 comp rx=0 tx=0).stop 2024-04-01T01:33:16.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.061+0000 7f44d3cc2700 1 -- 172.21.15.70:0/437439687 shutdown_connections 2024-04-01T01:33:16.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.061+0000 7f44d3cc2700 1 --2- 172.21.15.70:0/437439687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44cc103e00 0x7f44cc10a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:16.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.061+0000 7f44d3cc2700 1 --2- 172.21.15.70:0/437439687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44cc102260 0x7f44cc102680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:16.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.061+0000 7f44d3cc2700 1 --2- 172.21.15.70:0/437439687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc103460 0x7f44cc1038c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:16.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.061+0000 7f44d3cc2700 1 -- 172.21.15.70:0/437439687 >> 172.21.15.70:0/437439687 conn(0x7f44cc0fd820 msgr2=0x7f44cc0ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:16.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.061+0000 7f44d3cc2700 1 -- 172.21.15.70:0/437439687 shutdown_connections 2024-04-01T01:33:16.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.061+0000 7f44d3cc2700 1 -- 172.21.15.70:0/437439687 wait complete. 2024-04-01T01:33:16.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d3cc2700 1 Processor -- start 2024-04-01T01:33:16.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d3cc2700 1 -- start start 2024-04-01T01:33:16.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d3cc2700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44cc102260 0x7f44cc198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:16.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d3cc2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc103460 0x7f44cc1988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:16.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d3cc2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44cc103e00 0x7f44cc19f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:16.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d3cc2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc10dce0 con 0x7f44cc103460 2024-04-01T01:33:16.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d3cc2700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc10db60 con 0x7f44cc102260 2024-04-01T01:33:16.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d3cc2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc10de60 con 0x7f44cc103e00 2024-04-01T01:33:16.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d125d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc103460 0x7f44cc1988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:16.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d1a5e700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44cc102260 0x7f44cc198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:16.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d125d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc103460 0x7f44cc1988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42636/0 (socket says 172.21.15.70:42636) 2024-04-01T01:33:16.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d125d700 1 -- 172.21.15.70:0/2320305126 learned_addr learned my addr 172.21.15.70:0/2320305126 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:16.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d225f700 1 --2- 172.21.15.70:0/2320305126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44cc103e00 0x7f44cc19f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:16.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d125d700 1 -- 172.21.15.70:0/2320305126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44cc103e00 msgr2=0x7f44cc19f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:16.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d125d700 1 --2- 172.21.15.70:0/2320305126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44cc103e00 0x7f44cc19f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:16.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d125d700 1 -- 172.21.15.70:0/2320305126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44cc102260 msgr2=0x7f44cc198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:16.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d125d700 1 --2- 172.21.15.70:0/2320305126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44cc102260 0x7f44cc198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:16.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d125d700 1 -- 172.21.15.70:0/2320305126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44cc19ffd0 con 0x7f44cc103460 2024-04-01T01:33:16.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.062+0000 7f44d125d700 1 --2- 172.21.15.70:0/2320305126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc103460 0x7f44cc1988b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00b580 tx=0x7f44bc00fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:16.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.063+0000 7f44c2ffd700 1 -- 172.21.15.70:0/2320305126 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc027070 con 0x7f44cc103460 2024-04-01T01:33:16.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.063+0000 7f44c2ffd700 1 -- 172.21.15.70:0/2320305126 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44bc00fe30 con 0x7f44cc103460 2024-04-01T01:33:16.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.063+0000 7f44d3cc2700 1 -- 172.21.15.70:0/2320305126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44cc1a0260 con 0x7f44cc103460 2024-04-01T01:33:16.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.063+0000 7f44c2ffd700 1 -- 172.21.15.70:0/2320305126 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc0173f0 con 0x7f44cc103460 2024-04-01T01:33:16.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.063+0000 7f44d3cc2700 1 -- 172.21.15.70:0/2320305126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44cc1a0710 con 0x7f44cc103460 2024-04-01T01:33:16.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.064+0000 7f44d3cc2700 1 -- 172.21.15.70:0/2320305126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44cc068f00 con 0x7f44cc103460 2024-04-01T01:33:16.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.065+0000 7f44c2ffd700 1 -- 172.21.15.70:0/2320305126 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f44bc02a070 con 0x7f44cc103460 2024-04-01T01:33:16.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.065+0000 7f44c2ffd700 1 --2- 172.21.15.70:0/2320305126 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f44b80a25b0 0x7f44b80a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:16.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.065+0000 7f44d1a5e700 1 --2- 172.21.15.70:0/2320305126 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f44b80a25b0 0x7f44b80a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:16.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.065+0000 7f44c2ffd700 1 -- 172.21.15.70:0/2320305126 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f44bc0cf680 con 0x7f44cc103460 2024-04-01T01:33:16.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.066+0000 7f44d1a5e700 1 --2- 172.21.15.70:0/2320305126 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f44b80a25b0 0x7f44b80a4a70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009550 tx=0x7f44c80091b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:16.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.070+0000 7f44c2ffd700 1 -- 172.21.15.70:0/2320305126 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f44bc013070 con 0x7f44cc103460 2024-04-01T01:33:16.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:15 smithi081 ceph-mon[35142]: pgmap v349: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-01T01:33:16.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.267+0000 7f44d3cc2700 1 -- 172.21.15.70:0/2320305126 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f44cc062740 con 0x7f44b80a25b0 2024-04-01T01:33:16.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.268+0000 7f44c2ffd700 1 -- 172.21.15.70:0/2320305126 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f44cc062740 con 0x7f44b80a25b0 2024-04-01T01:33:16.268 INFO:teuthology.orchestra.run.smithi070.stdout:HOST ADDR LABELS STATUS 2024-04-01T01:33:16.268 INFO:teuthology.orchestra.run.smithi070.stdout:smithi070 172.21.15.70 2024-04-01T01:33:16.268 INFO:teuthology.orchestra.run.smithi070.stdout:smithi081 172.21.15.81 2024-04-01T01:33:16.268 INFO:teuthology.orchestra.run.smithi070.stdout:smithi189 172.21.15.189 2024-04-01T01:33:16.268 INFO:teuthology.orchestra.run.smithi070.stdout:3 hosts in cluster 2024-04-01T01:33:16.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.271+0000 7f44d3cc2700 1 -- 172.21.15.70:0/2320305126 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f44b80a25b0 msgr2=0x7f44b80a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:16.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.271+0000 7f44d3cc2700 1 --2- 172.21.15.70:0/2320305126 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f44b80a25b0 0x7f44b80a4a70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f44c8009550 tx=0x7f44c80091b0 comp rx=0 tx=0).stop 2024-04-01T01:33:16.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.271+0000 7f44d3cc2700 1 -- 172.21.15.70:0/2320305126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc103460 msgr2=0x7f44cc1988b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:16.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.271+0000 7f44d3cc2700 1 --2- 172.21.15.70:0/2320305126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc103460 0x7f44cc1988b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00b580 tx=0x7f44bc00fa00 comp rx=0 tx=0).stop 2024-04-01T01:33:16.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.271+0000 7f44d3cc2700 1 -- 172.21.15.70:0/2320305126 shutdown_connections 2024-04-01T01:33:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.271+0000 7f44d3cc2700 1 --2- 172.21.15.70:0/2320305126 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f44b80a25b0 0x7f44b80a4a70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.271+0000 7f44d3cc2700 1 --2- 172.21.15.70:0/2320305126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44cc103e00 0x7f44cc19f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.271+0000 7f44d3cc2700 1 --2- 172.21.15.70:0/2320305126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44cc102260 0x7f44cc198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.271+0000 7f44d3cc2700 1 --2- 172.21.15.70:0/2320305126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44cc103460 0x7f44cc1988b0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.271+0000 7f44d3cc2700 1 -- 172.21.15.70:0/2320305126 >> 172.21.15.70:0/2320305126 conn(0x7f44cc0fd820 msgr2=0x7f44cc106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.271+0000 7f44d3cc2700 1 -- 172.21.15.70:0/2320305126 shutdown_connections 2024-04-01T01:33:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:16.271+0000 7f44d3cc2700 1 -- 172.21.15.70:0/2320305126 wait complete. 2024-04-01T01:33:17.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch device ls' 2024-04-01T01:33:17.321 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:33:17.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:17 smithi189 ceph-mon[35753]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:17.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:17 smithi189 ceph-mon[35753]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:33:18.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:17 smithi070 ceph-mon[32179]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:18.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:17 smithi070 ceph-mon[32179]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:33:18.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:17 smithi081 ceph-mon[35142]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:18.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:17 smithi081 ceph-mon[35142]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:33:18.845 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-01T01:33:18.846 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-04-01T01:33:18.846 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-04-01T01:33:18.846 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-01T01:33:18.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-01T01:33:18.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-04-01T01:33:18.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-04-01T01:33:18.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-01T01:33:19.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.067+0000 7f02973f6700 1 -- 172.21.15.70:0/726525810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0290100b60 msgr2=0x7f0290102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.067+0000 7f02973f6700 1 --2- 172.21.15.70:0/726525810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0290100b60 0x7f0290102f90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f028c009990 tx=0x7f028c009ca0 comp rx=0 tx=0).stop 2024-04-01T01:33:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.068+0000 7f02973f6700 1 -- 172.21.15.70:0/726525810 shutdown_connections 2024-04-01T01:33:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.068+0000 7f02973f6700 1 --2- 172.21.15.70:0/726525810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0290105ea0 0x7f0290106320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.068+0000 7f02973f6700 1 --2- 172.21.15.70:0/726525810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0290100b60 0x7f0290102f90 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.068+0000 7f02973f6700 1 --2- 172.21.15.70:0/726525810 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02901034d0 0x7f0290105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.068+0000 7f02973f6700 1 -- 172.21.15.70:0/726525810 >> 172.21.15.70:0/726525810 conn(0x7f02900fc7a0 msgr2=0x7f02900febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.068+0000 7f02973f6700 1 -- 172.21.15.70:0/726525810 shutdown_connections 2024-04-01T01:33:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.068+0000 7f02973f6700 1 -- 172.21.15.70:0/726525810 wait complete. 2024-04-01T01:33:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.068+0000 7f02973f6700 1 Processor -- start 2024-04-01T01:33:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.068+0000 7f02973f6700 1 -- start start 2024-04-01T01:33:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.068+0000 7f02973f6700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0290100b60 0x7f029019e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.068+0000 7f02973f6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02901034d0 0x7f029019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:19.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.068+0000 7f02973f6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0290105ea0 0x7f02901a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.068+0000 7f02973f6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0290114280 con 0x7f02901034d0 2024-04-01T01:33:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.068+0000 7f02973f6700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0290114100 con 0x7f0290100b60 2024-04-01T01:33:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.068+0000 7f02973f6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0290114400 con 0x7f0290105ea0 2024-04-01T01:33:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.068+0000 7f0294991700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02901034d0 0x7f029019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.069+0000 7f0294991700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02901034d0 0x7f029019ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52198/0 (socket says 172.21.15.70:52198) 2024-04-01T01:33:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.069+0000 7f0294991700 1 -- 172.21.15.70:0/3180941967 learned_addr learned my addr 172.21.15.70:0/3180941967 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.069+0000 7f0295192700 1 --2- 172.21.15.70:0/3180941967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0290100b60 0x7f029019e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.069+0000 7f0295993700 1 --2- 172.21.15.70:0/3180941967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0290105ea0 0x7f02901a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.069+0000 7f0295192700 1 -- 172.21.15.70:0/3180941967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0290105ea0 msgr2=0x7f02901a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.069+0000 7f0295192700 1 --2- 172.21.15.70:0/3180941967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0290105ea0 0x7f02901a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.069+0000 7f0295192700 1 -- 172.21.15.70:0/3180941967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02901034d0 msgr2=0x7f029019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.069+0000 7f0295192700 1 --2- 172.21.15.70:0/3180941967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02901034d0 0x7f029019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.069+0000 7f0295192700 1 -- 172.21.15.70:0/3180941967 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02901a65b0 con 0x7f0290100b60 2024-04-01T01:33:19.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.069+0000 7f0294991700 1 --2- 172.21.15.70:0/3180941967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02901034d0 0x7f029019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:19.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.069+0000 7f0295192700 1 --2- 172.21.15.70:0/3180941967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0290100b60 0x7f029019e9f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f028c00b840 tx=0x7f028c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:19.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.069+0000 7f02867fc700 1 -- 172.21.15.70:0/3180941967 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028c026070 con 0x7f0290100b60 2024-04-01T01:33:19.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.069+0000 7f02973f6700 1 -- 172.21.15.70:0/3180941967 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02901a68a0 con 0x7f0290100b60 2024-04-01T01:33:19.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.069+0000 7f02867fc700 1 -- 172.21.15.70:0/3180941967 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f028c021410 con 0x7f0290100b60 2024-04-01T01:33:19.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.069+0000 7f02867fc700 1 -- 172.21.15.70:0/3180941967 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028c017650 con 0x7f0290100b60 2024-04-01T01:33:19.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.069+0000 7f02973f6700 1 -- 172.21.15.70:0/3180941967 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02901a6e30 con 0x7f0290100b60 2024-04-01T01:33:19.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.070+0000 7f02973f6700 1 -- 172.21.15.70:0/3180941967 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0290068f00 con 0x7f0290100b60 2024-04-01T01:33:19.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.071+0000 7f02867fc700 1 -- 172.21.15.70:0/3180941967 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f028c029070 con 0x7f0290100b60 2024-04-01T01:33:19.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.072+0000 7f02867fc700 1 --2- 172.21.15.70:0/3180941967 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f027c0a2530 0x7f027c0a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:19.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.072+0000 7f0294991700 1 --2- 172.21.15.70:0/3180941967 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f027c0a2530 0x7f027c0a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:19.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.072+0000 7f02867fc700 1 -- 172.21.15.70:0/3180941967 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9208+0+0 (secure 0 0 0) 0x7f028c0cff50 con 0x7f0290100b60 2024-04-01T01:33:19.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.072+0000 7f0294991700 1 --2- 172.21.15.70:0/3180941967 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f027c0a2530 0x7f027c0a49f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0280011590 tx=0x7f0280009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:19.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-01T01:33:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-04-01T01:33:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-04-01T01:33:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-04-01T01:33:19.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.080+0000 7f02867fc700 1 -- 172.21.15.70:0/3180941967 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f028c030070 con 0x7f0290100b60 2024-04-01T01:33:19.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.285+0000 7f02973f6700 1 -- 172.21.15.70:0/3180941967 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f0290062680 con 0x7f027c0a2530 2024-04-01T01:33:19.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.286+0000 7f02867fc700 1 -- 172.21.15.70:0/3180941967 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f0290062680 con 0x7f027c0a2530 2024-04-01T01:33:19.286 INFO:teuthology.orchestra.run.smithi070.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-01T01:33:19.286 INFO:teuthology.orchestra.run.smithi070.stdout:smithi070 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513002P480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-01T01:33:19.286 INFO:teuthology.orchestra.run.smithi070.stdout:smithi081 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400N7400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-01T01:33:19.286 INFO:teuthology.orchestra.run.smithi070.stdout:smithi189 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6232004L400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-01T01:33:19.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.288+0000 7f02973f6700 1 -- 172.21.15.70:0/3180941967 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f027c0a2530 msgr2=0x7f027c0a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:19.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.288+0000 7f02973f6700 1 --2- 172.21.15.70:0/3180941967 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f027c0a2530 0x7f027c0a49f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0280011590 tx=0x7f0280009260 comp rx=0 tx=0).stop 2024-04-01T01:33:19.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.288+0000 7f02973f6700 1 -- 172.21.15.70:0/3180941967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0290100b60 msgr2=0x7f029019e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:19.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.288+0000 7f02973f6700 1 --2- 172.21.15.70:0/3180941967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0290100b60 0x7f029019e9f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f028c00b840 tx=0x7f028c009580 comp rx=0 tx=0).stop 2024-04-01T01:33:19.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.289+0000 7f02973f6700 1 -- 172.21.15.70:0/3180941967 shutdown_connections 2024-04-01T01:33:19.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.289+0000 7f02973f6700 1 --2- 172.21.15.70:0/3180941967 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f027c0a2530 0x7f027c0a49f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:19.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.289+0000 7f02973f6700 1 --2- 172.21.15.70:0/3180941967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0290105ea0 0x7f02901a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:19.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.289+0000 7f02973f6700 1 --2- 172.21.15.70:0/3180941967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0290100b60 0x7f029019e9f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:19.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.289+0000 7f02973f6700 1 --2- 172.21.15.70:0/3180941967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f02901034d0 0x7f029019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:19.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.289+0000 7f02973f6700 1 -- 172.21.15.70:0/3180941967 >> 172.21.15.70:0/3180941967 conn(0x7f02900fc7a0 msgr2=0x7f0290104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:19.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.289+0000 7f02973f6700 1 -- 172.21.15.70:0/3180941967 shutdown_connections 2024-04-01T01:33:19.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:19.289+0000 7f02973f6700 1 -- 172.21.15.70:0/3180941967 wait complete. 2024-04-01T01:33:19.978 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:19 smithi070 ceph-mon[32179]: pgmap v351: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-01T01:33:19.978 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-01T01:33:19.978 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-04-01T01:33:19.978 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-04-01T01:33:19.978 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-01T01:33:19.978 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:19 smithi070 ceph-mon[32179]: osdmap e92: 12 total, 12 up, 12 in 2024-04-01T01:33:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:19 smithi189 ceph-mon[35753]: pgmap v351: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-01T01:33:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-01T01:33:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-04-01T01:33:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-04-01T01:33:19.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-01T01:33:19.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:19 smithi189 ceph-mon[35753]: osdmap e92: 12 total, 12 up, 12 in 2024-04-01T01:33:20.011 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-01T01:33:20.020 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi070.front.sepia.ceph.com 2024-04-01T01:33:20.020 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-04-01T01:33:20.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:19 smithi081 ceph-mon[35142]: pgmap v351: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-01T01:33:20.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-01T01:33:20.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-04-01T01:33:20.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-04-01T01:33:20.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-04-01T01:33:20.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:19 smithi081 ceph-mon[35142]: osdmap e92: 12 total, 12 up, 12 in 2024-04-01T01:33:20.264 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:33:20.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:20 smithi189 ceph-mon[35753]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:20.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:20 smithi189 ceph-mon[35753]: osdmap e93: 12 total, 12 up, 12 in 2024-04-01T01:33:20.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:20 smithi189 ceph-mon[35753]: pgmap v354: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-01T01:33:21.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:20 smithi070 ceph-mon[32179]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:21.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:20 smithi070 ceph-mon[32179]: osdmap e93: 12 total, 12 up, 12 in 2024-04-01T01:33:21.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:20 smithi070 ceph-mon[32179]: pgmap v354: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-01T01:33:21.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:20 smithi081 ceph-mon[35142]: from='client.25136 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:21.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:20 smithi081 ceph-mon[35142]: osdmap e93: 12 total, 12 up, 12 in 2024-04-01T01:33:21.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:20 smithi081 ceph-mon[35142]: pgmap v354: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-01T01:33:22.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.022+0000 7ff3cef1b700 1 -- 172.21.15.70:0/1690531221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3c8102290 msgr2=0x7ff3c81026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:22.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.022+0000 7ff3cef1b700 1 --2- 172.21.15.70:0/1690531221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3c8102290 0x7ff3c81026b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff3c400b0d0 tx=0x7ff3c400b3e0 comp rx=0 tx=0).stop 2024-04-01T01:33:22.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.022+0000 7ff3cef1b700 1 -- 172.21.15.70:0/1690531221 shutdown_connections 2024-04-01T01:33:22.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.022+0000 7ff3cef1b700 1 --2- 172.21.15.70:0/1690531221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3c8103e30 0x7ff3c810a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:22.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.022+0000 7ff3cef1b700 1 --2- 172.21.15.70:0/1690531221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3c8102290 0x7ff3c81026b0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:22.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.022+0000 7ff3cef1b700 1 --2- 172.21.15.70:0/1690531221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c8103490 0x7ff3c81038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:22.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.022+0000 7ff3cef1b700 1 -- 172.21.15.70:0/1690531221 >> 172.21.15.70:0/1690531221 conn(0x7ff3c80fd850 msgr2=0x7ff3c80ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:22.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.022+0000 7ff3cef1b700 1 -- 172.21.15.70:0/1690531221 shutdown_connections 2024-04-01T01:33:22.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.022+0000 7ff3cef1b700 1 -- 172.21.15.70:0/1690531221 wait complete. 2024-04-01T01:33:22.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cef1b700 1 Processor -- start 2024-04-01T01:33:22.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cef1b700 1 -- start start 2024-04-01T01:33:22.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cef1b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3c8102290 0x7ff3c81983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:22.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cef1b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3c8103490 0x7ff3c81988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:22.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cef1b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c8103e30 0x7ff3c819f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:22.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cef1b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c810dd30 con 0x7ff3c8102290 2024-04-01T01:33:22.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cef1b700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c810dbb0 con 0x7ff3c8103e30 2024-04-01T01:33:22.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cef1b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c810deb0 con 0x7ff3c8103490 2024-04-01T01:33:22.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cccb7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3c8102290 0x7ff3c81983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:22.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cd4b8700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c8103e30 0x7ff3c819f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:22.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cccb7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3c8102290 0x7ff3c81983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52214/0 (socket says 172.21.15.70:52214) 2024-04-01T01:33:22.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cccb7700 1 -- 172.21.15.70:0/346258806 learned_addr learned my addr 172.21.15.70:0/346258806 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:22.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3bffff700 1 --2- 172.21.15.70:0/346258806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3c8103490 0x7ff3c81988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cccb7700 1 -- 172.21.15.70:0/346258806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3c8103490 msgr2=0x7ff3c81988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cccb7700 1 --2- 172.21.15.70:0/346258806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3c8103490 0x7ff3c81988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cccb7700 1 -- 172.21.15.70:0/346258806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c8103e30 msgr2=0x7ff3c819f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cccb7700 1 --2- 172.21.15.70:0/346258806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c8103e30 0x7ff3c819f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cccb7700 1 -- 172.21.15.70:0/346258806 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c81a0010 con 0x7ff3c8102290 2024-04-01T01:33:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.023+0000 7ff3cd4b8700 1 --2- 172.21.15.70:0/346258806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c8103e30 0x7ff3c819f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.024+0000 7ff3cccb7700 1 --2- 172.21.15.70:0/346258806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3c8102290 0x7ff3c81983b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff3c400b770 tx=0x7ff3c4004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.024+0000 7ff3bdffb700 1 -- 172.21.15.70:0/346258806 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c401f070 con 0x7ff3c8102290 2024-04-01T01:33:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.024+0000 7ff3cef1b700 1 -- 172.21.15.70:0/346258806 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3c81a02a0 con 0x7ff3c8102290 2024-04-01T01:33:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.024+0000 7ff3bdffb700 1 -- 172.21.15.70:0/346258806 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff3c4007b80 con 0x7ff3c8102290 2024-04-01T01:33:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.024+0000 7ff3bdffb700 1 -- 172.21.15.70:0/346258806 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c401b670 con 0x7ff3c8102290 2024-04-01T01:33:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.024+0000 7ff3cef1b700 1 -- 172.21.15.70:0/346258806 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3c81a0750 con 0x7ff3c8102290 2024-04-01T01:33:22.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.025+0000 7ff3cef1b700 1 -- 172.21.15.70:0/346258806 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3c8068f00 con 0x7ff3c8102290 2024-04-01T01:33:22.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.026+0000 7ff3bdffb700 1 -- 172.21.15.70:0/346258806 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7ff3c4022070 con 0x7ff3c8102290 2024-04-01T01:33:22.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.027+0000 7ff3bdffb700 1 --2- 172.21.15.70:0/346258806 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff3b00a2530 0x7ff3b00a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.027+0000 7ff3bffff700 1 --2- 172.21.15.70:0/346258806 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff3b00a2530 0x7ff3b00a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.027+0000 7ff3bdffb700 1 -- 172.21.15.70:0/346258806 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff3c400f020 con 0x7ff3c8102290 2024-04-01T01:33:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.027+0000 7ff3bffff700 1 --2- 172.21.15.70:0/346258806 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff3b00a2530 0x7ff3b00a49f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009c60 tx=0x7ff3b8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:22.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.030+0000 7ff3bdffb700 1 -- 172.21.15.70:0/346258806 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff3c4029020 con 0x7ff3c8102290 2024-04-01T01:33:22.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.268+0000 7ff3cef1b700 1 -- 172.21.15.70:0/346258806 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7ff3c802db30 con 0x7ff3c8102290 2024-04-01T01:33:22.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.268+0000 7ff3bdffb700 1 -- 172.21.15.70:0/346258806 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3090 (secure 0 0 0) 0x7ff3c409cf90 con 0x7ff3c8102290 2024-04-01T01:33:22.268 INFO:teuthology.orchestra.run.smithi070.stdout:e28 2024-04-01T01:33:22.268 INFO:teuthology.orchestra.run.smithi070.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-01T01:33:22.268 INFO:teuthology.orchestra.run.smithi070.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-01T01:33:22.269 INFO:teuthology.orchestra.run.smithi070.stdout:legacy client fscid: 1 2024-04-01T01:33:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:22.269 INFO:teuthology.orchestra.run.smithi070.stdout:Filesystem 'cephfs' (1) 2024-04-01T01:33:22.269 INFO:teuthology.orchestra.run.smithi070.stdout:fs_name cephfs 2024-04-01T01:33:22.269 INFO:teuthology.orchestra.run.smithi070.stdout:epoch 28 2024-04-01T01:33:22.269 INFO:teuthology.orchestra.run.smithi070.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-01T01:33:22.269 INFO:teuthology.orchestra.run.smithi070.stdout:created 2024-04-01T01:31:15.732761+0000 2024-04-01T01:33:22.270 INFO:teuthology.orchestra.run.smithi070.stdout:modified 2024-04-01T01:31:38.563210+0000 2024-04-01T01:33:22.270 INFO:teuthology.orchestra.run.smithi070.stdout:tableserver 0 2024-04-01T01:33:22.270 INFO:teuthology.orchestra.run.smithi070.stdout:root 0 2024-04-01T01:33:22.270 INFO:teuthology.orchestra.run.smithi070.stdout:session_timeout 300 2024-04-01T01:33:22.270 INFO:teuthology.orchestra.run.smithi070.stdout:session_autoclose 300 2024-04-01T01:33:22.270 INFO:teuthology.orchestra.run.smithi070.stdout:max_file_size 1099511627776 2024-04-01T01:33:22.270 INFO:teuthology.orchestra.run.smithi070.stdout:max_xattr_size 65536 2024-04-01T01:33:22.271 INFO:teuthology.orchestra.run.smithi070.stdout:required_client_features {} 2024-04-01T01:33:22.271 INFO:teuthology.orchestra.run.smithi070.stdout:last_failure 0 2024-04-01T01:33:22.271 INFO:teuthology.orchestra.run.smithi070.stdout:last_failure_osd_epoch 0 2024-04-01T01:33:22.271 INFO:teuthology.orchestra.run.smithi070.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-01T01:33:22.271 INFO:teuthology.orchestra.run.smithi070.stdout:max_mds 5 2024-04-01T01:33:22.271 INFO:teuthology.orchestra.run.smithi070.stdout:in 0,1,2,3,4 2024-04-01T01:33:22.271 INFO:teuthology.orchestra.run.smithi070.stdout:up {0=24464,1=24451,2=24418,3=24404,4=24379} 2024-04-01T01:33:22.272 INFO:teuthology.orchestra.run.smithi070.stdout:failed 2024-04-01T01:33:22.272 INFO:teuthology.orchestra.run.smithi070.stdout:damaged 2024-04-01T01:33:22.272 INFO:teuthology.orchestra.run.smithi070.stdout:stopped 2024-04-01T01:33:22.272 INFO:teuthology.orchestra.run.smithi070.stdout:data_pools [3,4] 2024-04-01T01:33:22.272 INFO:teuthology.orchestra.run.smithi070.stdout:metadata_pool 2 2024-04-01T01:33:22.272 INFO:teuthology.orchestra.run.smithi070.stdout:inline_data disabled 2024-04-01T01:33:22.272 INFO:teuthology.orchestra.run.smithi070.stdout:balancer 2024-04-01T01:33:22.273 INFO:teuthology.orchestra.run.smithi070.stdout:bal_rank_mask -1 2024-04-01T01:33:22.273 INFO:teuthology.orchestra.run.smithi070.stdout:standby_count_wanted 0 2024-04-01T01:33:22.273 INFO:teuthology.orchestra.run.smithi070.stdout:qdb_cluster leader: 24464 members: 24451,24464,24379,24404,24418 2024-04-01T01:33:22.273 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.b{0:24464} state up:active seq 3 addr [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] compat {c=[1],r=[1],i=[1fff]}] 2024-04-01T01:33:22.273 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.c{0:24427} state up:standby-replay seq 1 addr [v2:172.21.15.189:6836/2296387110,v1:172.21.15.189:6837/2296387110] compat {c=[1],r=[1],i=[1fff]}] 2024-04-01T01:33:22.273 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.i{1:24451} state up:active seq 10 addr [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] compat {c=[1],r=[1],i=[1fff]}] 2024-04-01T01:33:22.273 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.h{1:24425} state up:standby-replay seq 1 addr [v2:172.21.15.81:6836/3527001697,v1:172.21.15.81:6837/3527001697] compat {c=[1],r=[1],i=[1fff]}] 2024-04-01T01:33:22.273 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.f{2:24418} state up:active seq 12 addr [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] compat {c=[1],r=[1],i=[1fff]}] 2024-04-01T01:33:22.274 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.k{2:24371} state up:standby-replay seq 1 addr [v2:172.21.15.81:6832/1246539452,v1:172.21.15.81:6833/1246539452] compat {c=[1],r=[1],i=[1fff]}] 2024-04-01T01:33:22.274 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.e{3:24404} state up:active seq 13 addr [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] compat {c=[1],r=[1],i=[1fff]}] 2024-04-01T01:33:22.274 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.a{3:14628} state up:standby-replay seq 1 addr [v2:172.21.15.70:6840/2380158677,v1:172.21.15.70:6841/2380158677] compat {c=[1],r=[1],i=[1fff]}] 2024-04-01T01:33:22.274 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.l{4:24379} state up:active seq 16 addr [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] compat {c=[1],r=[1],i=[1fff]}] 2024-04-01T01:33:22.274 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.g{4:14616} state up:standby-replay seq 1 addr [v2:172.21.15.70:6838/1626804615,v1:172.21.15.70:6839/1626804615] compat {c=[1],r=[1],i=[1fff]}] 2024-04-01T01:33:22.274 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:22.274 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:22.275 INFO:teuthology.orchestra.run.smithi070.stdout:Standby daemons: 2024-04-01T01:33:22.275 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:22.275 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.j{-1:14544} state up:standby seq 1 addr [v2:172.21.15.70:6834/33736871,v1:172.21.15.70:6835/33736871] compat {c=[1],r=[1],i=[1fff]}] 2024-04-01T01:33:22.275 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.d{-1:14562} state up:standby seq 1 addr [v2:172.21.15.70:6836/2674132555,v1:172.21.15.70:6837/2674132555] compat {c=[1],r=[1],i=[1fff]}] 2024-04-01T01:33:22.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.270+0000 7ff3cef1b700 1 -- 172.21.15.70:0/346258806 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff3b00a2530 msgr2=0x7ff3b00a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:22.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.270+0000 7ff3cef1b700 1 --2- 172.21.15.70:0/346258806 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff3b00a2530 0x7ff3b00a49f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009c60 tx=0x7ff3b8009260 comp rx=0 tx=0).stop 2024-04-01T01:33:22.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.270+0000 7ff3cef1b700 1 -- 172.21.15.70:0/346258806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3c8102290 msgr2=0x7ff3c81983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:22.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.270+0000 7ff3cef1b700 1 --2- 172.21.15.70:0/346258806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3c8102290 0x7ff3c81983b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff3c400b770 tx=0x7ff3c4004ce0 comp rx=0 tx=0).stop 2024-04-01T01:33:22.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.270+0000 7ff3cef1b700 1 -- 172.21.15.70:0/346258806 shutdown_connections 2024-04-01T01:33:22.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.270+0000 7ff3cef1b700 1 --2- 172.21.15.70:0/346258806 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff3b00a2530 0x7ff3b00a49f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:22.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.270+0000 7ff3cef1b700 1 --2- 172.21.15.70:0/346258806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3c8103e30 0x7ff3c819f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:22.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.270+0000 7ff3cef1b700 1 --2- 172.21.15.70:0/346258806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3c8102290 0x7ff3c81983b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:22.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.270+0000 7ff3cef1b700 1 --2- 172.21.15.70:0/346258806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3c8103490 0x7ff3c81988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:22.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.270+0000 7ff3cef1b700 1 -- 172.21.15.70:0/346258806 >> 172.21.15.70:0/346258806 conn(0x7ff3c80fd850 msgr2=0x7ff3c81067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:22.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.270+0000 7ff3cef1b700 1 -- 172.21.15.70:0/346258806 shutdown_connections 2024-04-01T01:33:22.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:22.270+0000 7ff3cef1b700 1 -- 172.21.15.70:0/346258806 wait complete. 2024-04-01T01:33:22.278 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 28 2024-04-01T01:33:22.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:22 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/346258806' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-01T01:33:22.968 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:22 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/346258806' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-01T01:33:22.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- bash -c 'ceph osd dump' 2024-04-01T01:33:22.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:22 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/346258806' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-01T01:33:23.237 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:33:23.609 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:23 smithi070 ceph-mon[32179]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:33:23.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:23 smithi081 ceph-mon[35142]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:33:23.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:23 smithi189 ceph-mon[35753]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:33:24.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.931+0000 7f282f9c2700 1 -- 172.21.15.70:0/643844423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2828103560 msgr2=0x7f28281039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:24.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.931+0000 7f282f9c2700 1 --2- 172.21.15.70:0/643844423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2828103560 0x7f28281039c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f2818009990 tx=0x7f2818009ca0 comp rx=0 tx=0).stop 2024-04-01T01:33:24.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.931+0000 7f282f9c2700 1 -- 172.21.15.70:0/643844423 shutdown_connections 2024-04-01T01:33:24.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.931+0000 7f282f9c2700 1 --2- 172.21.15.70:0/643844423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2828103f00 0x7f282810a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:24.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.931+0000 7f282f9c2700 1 --2- 172.21.15.70:0/643844423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2828100580 0x7f28281009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:24.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.931+0000 7f282f9c2700 1 --2- 172.21.15.70:0/643844423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2828103560 0x7f28281039c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:24.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.931+0000 7f282f9c2700 1 -- 172.21.15.70:0/643844423 >> 172.21.15.70:0/643844423 conn(0x7f28280fbee0 msgr2=0x7f28280fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:24.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.931+0000 7f282f9c2700 1 -- 172.21.15.70:0/643844423 shutdown_connections 2024-04-01T01:33:24.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.931+0000 7f282f9c2700 1 -- 172.21.15.70:0/643844423 wait complete. 2024-04-01T01:33:24.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.931+0000 7f282f9c2700 1 Processor -- start 2024-04-01T01:33:24.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.931+0000 7f282f9c2700 1 -- start start 2024-04-01T01:33:24.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282f9c2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2828100580 0x7f2828198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:24.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282f9c2700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2828103560 0x7f2828198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:24.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282f9c2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2828103f00 0x7f282819f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:24.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282f9c2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f282810dd30 con 0x7f2828100580 2024-04-01T01:33:24.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282f9c2700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f282810dbb0 con 0x7f2828103560 2024-04-01T01:33:24.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282f9c2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f282810deb0 con 0x7f2828103f00 2024-04-01T01:33:24.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282d75e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2828100580 0x7f2828198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:24.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282d75e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2828100580 0x7f2828198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52220/0 (socket says 172.21.15.70:52220) 2024-04-01T01:33:24.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282d75e700 1 -- 172.21.15.70:0/4082593886 learned_addr learned my addr 172.21.15.70:0/4082593886 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:24.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282cf5d700 1 --2- 172.21.15.70:0/4082593886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2828103560 0x7f2828198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:24.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282df5f700 1 --2- 172.21.15.70:0/4082593886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2828103f00 0x7f282819f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:24.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282d75e700 1 -- 172.21.15.70:0/4082593886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2828103f00 msgr2=0x7f282819f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:24.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282d75e700 1 --2- 172.21.15.70:0/4082593886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2828103f00 0x7f282819f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:24.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282d75e700 1 -- 172.21.15.70:0/4082593886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2828103560 msgr2=0x7f2828198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:24.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282d75e700 1 --2- 172.21.15.70:0/4082593886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2828103560 0x7f2828198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:24.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282d75e700 1 -- 172.21.15.70:0/4082593886 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f282819ff80 con 0x7f2828100580 2024-04-01T01:33:24.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282d75e700 1 --2- 172.21.15.70:0/4082593886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2828100580 0x7f2828198320 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f282400e9a0 tx=0x7f282400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:24.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f281e7fc700 1 -- 172.21.15.70:0/4082593886 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2824004220 con 0x7f2828100580 2024-04-01T01:33:24.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282f9c2700 1 -- 172.21.15.70:0/4082593886 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28281a0270 con 0x7f2828100580 2024-04-01T01:33:24.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f281e7fc700 1 -- 172.21.15.70:0/4082593886 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f28240043c0 con 0x7f2828100580 2024-04-01T01:33:24.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f281e7fc700 1 -- 172.21.15.70:0/4082593886 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2824003ce0 con 0x7f2828100580 2024-04-01T01:33:24.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.932+0000 7f282f9c2700 1 -- 172.21.15.70:0/4082593886 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28281a07b0 con 0x7f2828100580 2024-04-01T01:33:24.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.934+0000 7f282f9c2700 1 -- 172.21.15.70:0/4082593886 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2828068f00 con 0x7f2828100580 2024-04-01T01:33:24.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.934+0000 7f281e7fc700 1 -- 172.21.15.70:0/4082593886 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f282400c810 con 0x7f2828100580 2024-04-01T01:33:24.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.935+0000 7f281e7fc700 1 --2- 172.21.15.70:0/4082593886 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28140a2530 0x7f28140a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:24.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.935+0000 7f282cf5d700 1 --2- 172.21.15.70:0/4082593886 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28140a2530 0x7f28140a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:24.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.935+0000 7f281e7fc700 1 -- 172.21.15.70:0/4082593886 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f28240cece0 con 0x7f2828100580 2024-04-01T01:33:24.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.936+0000 7f282cf5d700 1 --2- 172.21.15.70:0/4082593886 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28140a2530 0x7f28140a49f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f281800b710 tx=0x7f2818005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:24.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:24.940+0000 7f281e7fc700 1 -- 172.21.15.70:0/4082593886 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f282409bad0 con 0x7f2828100580 2024-04-01T01:33:25.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.135+0000 7f282f9c2700 1 -- 172.21.15.70:0/4082593886 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f2828062e50 con 0x7f2828100580 2024-04-01T01:33:25.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.136+0000 7f281e7fc700 1 -- 172.21.15.70:0/4082593886 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5684 (secure 0 0 0) 0x7f282409b3d0 con 0x7f2828100580 2024-04-01T01:33:25.137 INFO:teuthology.orchestra.run.smithi070.stdout:epoch 93 2024-04-01T01:33:25.137 INFO:teuthology.orchestra.run.smithi070.stdout:fsid 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:33:25.137 INFO:teuthology.orchestra.run.smithi070.stdout:created 2024-04-01T01:23:20.696672+0000 2024-04-01T01:33:25.137 INFO:teuthology.orchestra.run.smithi070.stdout:modified 2024-04-01T01:33:19.716245+0000 2024-04-01T01:33:25.137 INFO:teuthology.orchestra.run.smithi070.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-01T01:33:25.137 INFO:teuthology.orchestra.run.smithi070.stdout:crush_version 27 2024-04-01T01:33:25.137 INFO:teuthology.orchestra.run.smithi070.stdout:full_ratio 0.95 2024-04-01T01:33:25.137 INFO:teuthology.orchestra.run.smithi070.stdout:backfillfull_ratio 0.9 2024-04-01T01:33:25.138 INFO:teuthology.orchestra.run.smithi070.stdout:nearfull_ratio 0.85 2024-04-01T01:33:25.138 INFO:teuthology.orchestra.run.smithi070.stdout:require_min_compat_client luminous 2024-04-01T01:33:25.138 INFO:teuthology.orchestra.run.smithi070.stdout:min_compat_client luminous 2024-04-01T01:33:25.138 INFO:teuthology.orchestra.run.smithi070.stdout:require_osd_release squid 2024-04-01T01:33:25.138 INFO:teuthology.orchestra.run.smithi070.stdout:stretch_mode_enabled false 2024-04-01T01:33:25.138 INFO:teuthology.orchestra.run.smithi070.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-01T01:33:25.138 INFO:teuthology.orchestra.run.smithi070.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-01T01:33:25.138 INFO:teuthology.orchestra.run.smithi070.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-01T01:33:25.138 INFO:teuthology.orchestra.run.smithi070.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-01T01:33:25.138 INFO:teuthology.orchestra.run.smithi070.stdout:max_osd 12 2024-04-01T01:33:25.138 INFO:teuthology.orchestra.run.smithi070.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6802/3697496043,v1:172.21.15.70:6803/3697496043] [v2:172.21.15.70:6804/3697496043,v1:172.21.15.70:6805/3697496043] exists,up a4e9041f-f0ad-4b4b-80f6-2726874d9231 2024-04-01T01:33:25.138 INFO:teuthology.orchestra.run.smithi070.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6800/4081087424,v1:172.21.15.81:6801/4081087424] [v2:172.21.15.81:6802/4081087424,v1:172.21.15.81:6803/4081087424] exists,up 9df24cdb-3f8d-427c-afd8-dd579612d782 2024-04-01T01:33:25.138 INFO:teuthology.orchestra.run.smithi070.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6800/4191559460,v1:172.21.15.189:6801/4191559460] [v2:172.21.15.189:6802/4191559460,v1:172.21.15.189:6803/4191559460] exists,up 800721f9-d63a-4bea-8bbb-ae09c8c161be 2024-04-01T01:33:25.139 INFO:teuthology.orchestra.run.smithi070.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6810/627924617,v1:172.21.15.70:6811/627924617] [v2:172.21.15.70:6812/627924617,v1:172.21.15.70:6813/627924617] exists,up 835bbe5b-ecc9-4ebb-b084-9707dce28d70 2024-04-01T01:33:25.139 INFO:teuthology.orchestra.run.smithi070.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6808/404027261,v1:172.21.15.81:6809/404027261] [v2:172.21.15.81:6810/404027261,v1:172.21.15.81:6811/404027261] exists,up ddf50f36-6795-440a-9890-2f20c63820fe 2024-04-01T01:33:25.139 INFO:teuthology.orchestra.run.smithi070.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6808/2305146377,v1:172.21.15.189:6809/2305146377] [v2:172.21.15.189:6810/2305146377,v1:172.21.15.189:6811/2305146377] exists,up 9cae9565-6f21-4686-a5e9-a8c08d5fe0bb 2024-04-01T01:33:25.139 INFO:teuthology.orchestra.run.smithi070.stdout:osd.6 up in weight 1 up_from 46 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6818/2689871673,v1:172.21.15.70:6819/2689871673] [v2:172.21.15.70:6820/2689871673,v1:172.21.15.70:6821/2689871673] exists,up 0446322b-061f-4134-aa65-2f4639238ce5 2024-04-01T01:33:25.139 INFO:teuthology.orchestra.run.smithi070.stdout:osd.7 up in weight 1 up_from 51 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6816/2088069716,v1:172.21.15.81:6817/2088069716] [v2:172.21.15.81:6818/2088069716,v1:172.21.15.81:6819/2088069716] exists,up d654db8e-5fdc-47ff-99d9-db10263fa237 2024-04-01T01:33:25.139 INFO:teuthology.orchestra.run.smithi070.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6816/2097527021,v1:172.21.15.189:6817/2097527021] [v2:172.21.15.189:6818/2097527021,v1:172.21.15.189:6819/2097527021] exists,up 073456b5-3f5d-491a-86a4-317161989697 2024-04-01T01:33:25.139 INFO:teuthology.orchestra.run.smithi070.stdout:osd.9 up in weight 1 up_from 67 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6826/1192094091,v1:172.21.15.70:6827/1192094091] [v2:172.21.15.70:6828/1192094091,v1:172.21.15.70:6829/1192094091] exists,up c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d 2024-04-01T01:33:25.139 INFO:teuthology.orchestra.run.smithi070.stdout:osd.10 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6824/1556495251,v1:172.21.15.81:6825/1556495251] [v2:172.21.15.81:6826/1556495251,v1:172.21.15.81:6827/1556495251] exists,up f58e117c-8f4b-4ca4-bddc-9ae37024d996 2024-04-01T01:33:25.139 INFO:teuthology.orchestra.run.smithi070.stdout:osd.11 up in weight 1 up_from 80 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6824/22739510,v1:172.21.15.189:6825/22739510] [v2:172.21.15.189:6826/22739510,v1:172.21.15.189:6827/22739510] exists,up 365b3275-80d3-4888-8a4e-e00b571599ca 2024-04-01T01:33:25.139 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 2.5 [8,11] 2024-04-01T01:33:25.139 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 2.a [2,11] 2024-04-01T01:33:25.139 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 2.1a [8,1] 2024-04-01T01:33:25.139 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 2.1b [2,4] 2024-04-01T01:33:25.139 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 2.1d [2,4] 2024-04-01T01:33:25.140 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 3.c [3,9] 2024-04-01T01:33:25.140 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 3.e [4,6] 2024-04-01T01:33:25.140 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 3.20 [5,9] 2024-04-01T01:33:25.140 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 3.34 [4,9] 2024-04-01T01:33:25.140 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 3.3a [4,8] 2024-04-01T01:33:25.140 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 4.5 [5,8] 2024-04-01T01:33:25.140 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 4.b [9,8] 2024-04-01T01:33:25.140 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 4.12 [9,10] 2024-04-01T01:33:25.140 INFO:teuthology.orchestra.run.smithi070.stdout:pg_upmap_items 4.18 [9,8] 2024-04-01T01:33:25.140 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/950171900 expires 2024-04-02T01:24:18.477610+0000 2024-04-01T01:33:25.140 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/2902831307 expires 2024-04-02T01:24:18.477610+0000 2024-04-01T01:33:25.140 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6801/2836712828 expires 2024-04-02T01:24:18.477610+0000 2024-04-01T01:33:25.140 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6800/2836712828 expires 2024-04-02T01:24:18.477610+0000 2024-04-01T01:33:25.140 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/602656012 expires 2024-04-02T01:23:52.091015+0000 2024-04-01T01:33:25.140 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6800/2121290979 expires 2024-04-02T01:23:52.091015+0000 2024-04-01T01:33:25.141 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:6801/2121290979 expires 2024-04-02T01:23:52.091015+0000 2024-04-01T01:33:25.141 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/3734182136 expires 2024-04-02T01:23:52.091015+0000 2024-04-01T01:33:25.141 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/2899206193 expires 2024-04-02T01:23:52.091015+0000 2024-04-01T01:33:25.141 INFO:teuthology.orchestra.run.smithi070.stdout:blocklist 172.21.15.70:0/3948868469 expires 2024-04-02T01:24:18.477610+0000 2024-04-01T01:33:25.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.139+0000 7f282f9c2700 1 -- 172.21.15.70:0/4082593886 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28140a2530 msgr2=0x7f28140a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:25.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.139+0000 7f282f9c2700 1 --2- 172.21.15.70:0/4082593886 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28140a2530 0x7f28140a49f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f281800b710 tx=0x7f2818005fd0 comp rx=0 tx=0).stop 2024-04-01T01:33:25.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.139+0000 7f282f9c2700 1 -- 172.21.15.70:0/4082593886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2828100580 msgr2=0x7f2828198320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:25.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.139+0000 7f282f9c2700 1 --2- 172.21.15.70:0/4082593886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2828100580 0x7f2828198320 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f282400e9a0 tx=0x7f282400ecb0 comp rx=0 tx=0).stop 2024-04-01T01:33:25.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.140+0000 7f282f9c2700 1 -- 172.21.15.70:0/4082593886 shutdown_connections 2024-04-01T01:33:25.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.140+0000 7f282f9c2700 1 --2- 172.21.15.70:0/4082593886 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28140a2530 0x7f28140a49f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:25.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.140+0000 7f282f9c2700 1 --2- 172.21.15.70:0/4082593886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2828103f00 0x7f282819f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:25.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.140+0000 7f282f9c2700 1 --2- 172.21.15.70:0/4082593886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2828100580 0x7f2828198320 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:25.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.140+0000 7f282f9c2700 1 --2- 172.21.15.70:0/4082593886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2828103560 0x7f2828198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:25.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.140+0000 7f282f9c2700 1 -- 172.21.15.70:0/4082593886 >> 172.21.15.70:0/4082593886 conn(0x7f28280fbee0 msgr2=0x7f2828106860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:25.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.140+0000 7f282f9c2700 1 -- 172.21.15.70:0/4082593886 shutdown_connections 2024-04-01T01:33:25.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.140+0000 7f282f9c2700 1 -- 172.21.15.70:0/4082593886 wait complete. 2024-04-01T01:33:25.841 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:25 smithi070 ceph-mon[32179]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-04-01T01:33:25.841 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:25 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/4082593886' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-01T01:33:25.870 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-01T01:33:25.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:33:25.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.987+0000 7fcbb9c07640 1 -- 172.21.15.70:0/416031847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbb41074b0 msgr2=0x7fcbb41098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:25.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.987+0000 7fcbb9c07640 1 --2- 172.21.15.70:0/416031847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbb41074b0 0x7fcbb41098a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009ca0 tx=0x7fcbac02c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:25.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.987+0000 7fcbb9c07640 1 -- 172.21.15.70:0/416031847 shutdown_connections 2024-04-01T01:33:25.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.987+0000 7fcbb9c07640 1 --2- 172.21.15.70:0/416031847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbb41074b0 0x7fcbb41098a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:25.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.987+0000 7fcbb9c07640 1 --2- 172.21.15.70:0/416031847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb4104b80 0x7fcbb4106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:25.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.987+0000 7fcbb9c07640 1 --2- 172.21.15.70:0/416031847 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbb4102250 0x7fcbb4104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:25.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.987+0000 7fcbb9c07640 1 -- 172.21.15.70:0/416031847 >> 172.21.15.70:0/416031847 conn(0x7fcbb40fbbe0 msgr2=0x7fcbb40fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:25.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.987+0000 7fcbb9c07640 1 -- 172.21.15.70:0/416031847 shutdown_connections 2024-04-01T01:33:25.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.987+0000 7fcbb9c07640 1 -- 172.21.15.70:0/416031847 wait complete. 2024-04-01T01:33:25.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.988+0000 7fcbb9c07640 1 Processor -- start 2024-04-01T01:33:25.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.988+0000 7fcbb9c07640 1 -- start start 2024-04-01T01:33:25.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.988+0000 7fcbb9c07640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbb4102250 0x7fcbb41979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:25.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.988+0000 7fcbb9c07640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbb4104b80 0x7fcbb4197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:25.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.988+0000 7fcbb9c07640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb41074b0 0x7fcbb419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:25.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.988+0000 7fcbb9c07640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb410d2c0 con 0x7fcbb4104b80 2024-04-01T01:33:25.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.988+0000 7fcbb9c07640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb410d140 con 0x7fcbb4102250 2024-04-01T01:33:25.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.988+0000 7fcbb9c07640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb410d440 con 0x7fcbb41074b0 2024-04-01T01:33:25.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.988+0000 7fcbb2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbb4104b80 0x7fcbb4197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:25.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.988+0000 7fcbb2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbb4104b80 0x7fcbb4197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52246/0 (socket says 172.21.15.70:52246) 2024-04-01T01:33:25.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.988+0000 7fcbb2ffd640 1 -- 172.21.15.70:0/2089416940 learned_addr learned my addr 172.21.15.70:0/2089416940 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:25.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.989+0000 7fcbb3fff640 1 --2- 172.21.15.70:0/2089416940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb41074b0 0x7fcbb419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:25.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.989+0000 7fcbb37fe640 1 --2- 172.21.15.70:0/2089416940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbb4102250 0x7fcbb41979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:25.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.989+0000 7fcbb2ffd640 1 -- 172.21.15.70:0/2089416940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb41074b0 msgr2=0x7fcbb419ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:25.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.989+0000 7fcbb2ffd640 1 --2- 172.21.15.70:0/2089416940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb41074b0 0x7fcbb419ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:25.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.989+0000 7fcbb2ffd640 1 -- 172.21.15.70:0/2089416940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbb4102250 msgr2=0x7fcbb41979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:25.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.989+0000 7fcbb2ffd640 1 --2- 172.21.15.70:0/2089416940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbb4102250 0x7fcbb41979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:25.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.989+0000 7fcbb2ffd640 1 -- 172.21.15.70:0/2089416940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb419f5a0 con 0x7fcbb4104b80 2024-04-01T01:33:25.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.989+0000 7fcbb2ffd640 1 --2- 172.21.15.70:0/2089416940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbb4104b80 0x7fcbb4197f10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c00e9f0 tx=0x7fcb9c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:25.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.989+0000 7fcbb0ff9640 1 -- 172.21.15.70:0/2089416940 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c00ccc0 con 0x7fcbb4104b80 2024-04-01T01:33:25.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.989+0000 7fcbb9c07640 1 -- 172.21.15.70:0/2089416940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbb419f890 con 0x7fcbb4104b80 2024-04-01T01:33:25.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.989+0000 7fcbb0ff9640 1 -- 172.21.15.70:0/2089416940 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb9c0054d0 con 0x7fcbb4104b80 2024-04-01T01:33:25.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.989+0000 7fcbb0ff9640 1 -- 172.21.15.70:0/2089416940 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c010430 con 0x7fcbb4104b80 2024-04-01T01:33:25.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.989+0000 7fcbb9c07640 1 -- 172.21.15.70:0/2089416940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbb419fd90 con 0x7fcbb4104b80 2024-04-01T01:33:25.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.990+0000 7fcbb9c07640 1 -- 172.21.15.70:0/2089416940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbb40fd6d0 con 0x7fcbb4104b80 2024-04-01T01:33:25.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.991+0000 7fcbb0ff9640 1 -- 172.21.15.70:0/2089416940 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fcb9c005000 con 0x7fcbb4104b80 2024-04-01T01:33:25.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.992+0000 7fcbb0ff9640 1 --2- 172.21.15.70:0/2089416940 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcba00a24a0 0x7fcba00a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:25.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.992+0000 7fcbb37fe640 1 --2- 172.21.15.70:0/2089416940 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcba00a24a0 0x7fcba00a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:25.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.992+0000 7fcbb0ff9640 1 -- 172.21.15.70:0/2089416940 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fcb9c0cf0c0 con 0x7fcbb4104b80 2024-04-01T01:33:25.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.992+0000 7fcbb37fe640 1 --2- 172.21.15.70:0/2089416940 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcba00a24a0 0x7fcba00a4960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009bd0 tx=0x7fcba4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:25.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:25.995+0000 7fcbb0ff9640 1 -- 172.21.15.70:0/2089416940 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcb9c053280 con 0x7fcbb4104b80 2024-04-01T01:33:25.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:25 smithi189 ceph-mon[35753]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-04-01T01:33:25.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:25 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/4082593886' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-01T01:33:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:25 smithi081 ceph-mon[35142]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-04-01T01:33:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:25 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/4082593886' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-01T01:33:26.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.172+0000 7fcbb9c07640 1 -- 172.21.15.70:0/2089416940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcbb402d3c0 con 0x7fcbb4104b80 2024-04-01T01:33:26.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.174+0000 7fcbb0ff9640 1 -- 172.21.15.70:0/2089416940 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24781 (secure 0 0 0) 0x7fcb9c0052f0 con 0x7fcbb4104b80 2024-04-01T01:33:26.175 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:26.175 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":93,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:33:19.716245+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:33:26.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.176+0000 7fcbb9c07640 1 -- 172.21.15.70:0/2089416940 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcba00a24a0 msgr2=0x7fcba00a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:26.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.176+0000 7fcbb9c07640 1 --2- 172.21.15.70:0/2089416940 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcba00a24a0 0x7fcba00a4960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009bd0 tx=0x7fcba4005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:26.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.176+0000 7fcbb9c07640 1 -- 172.21.15.70:0/2089416940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbb4104b80 msgr2=0x7fcbb4197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:26.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.176+0000 7fcbb9c07640 1 --2- 172.21.15.70:0/2089416940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbb4104b80 0x7fcbb4197f10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c00e9f0 tx=0x7fcb9c00eeb0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.176+0000 7fcbb9c07640 1 -- 172.21.15.70:0/2089416940 shutdown_connections 2024-04-01T01:33:26.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.176+0000 7fcbb9c07640 1 --2- 172.21.15.70:0/2089416940 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcba00a24a0 0x7fcba00a4960 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.176+0000 7fcbb9c07640 1 --2- 172.21.15.70:0/2089416940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb41074b0 0x7fcbb419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.176+0000 7fcbb9c07640 1 --2- 172.21.15.70:0/2089416940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcbb4104b80 0x7fcbb4197f10 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.176+0000 7fcbb9c07640 1 --2- 172.21.15.70:0/2089416940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcbb4102250 0x7fcbb41979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.176+0000 7fcbb9c07640 1 -- 172.21.15.70:0/2089416940 >> 172.21.15.70:0/2089416940 conn(0x7fcbb40fbbe0 msgr2=0x7fcbb4102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:26.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.176+0000 7fcbb9c07640 1 -- 172.21.15.70:0/2089416940 shutdown_connections 2024-04-01T01:33:26.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.176+0000 7fcbb9c07640 1 -- 172.21.15.70:0/2089416940 wait complete. 2024-04-01T01:33:26.188 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-01T01:26:42.809779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-01T01:31:13.612369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-01T01:31:14.910162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-01T01:31:18.783179+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-01T01:33:26.189 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-01T01:33:26.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.294+0000 7fc4ad88f640 1 -- 172.21.15.70:0/2999909240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4a8100660 msgr2=0x7fc4a8100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:26.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.294+0000 7fc4ad88f640 1 --2- 172.21.15.70:0/2999909240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4a8100660 0x7fc4a8100a80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc498009900 tx=0x7fc49802bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:26.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.294+0000 7fc4ad88f640 1 -- 172.21.15.70:0/2999909240 shutdown_connections 2024-04-01T01:33:26.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.294+0000 7fc4ad88f640 1 --2- 172.21.15.70:0/2999909240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4a8102210 0x7fc4a8108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.294+0000 7fc4ad88f640 1 --2- 172.21.15.70:0/2999909240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a8101850 0x7fc4a8101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.294+0000 7fc4ad88f640 1 --2- 172.21.15.70:0/2999909240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4a8100660 0x7fc4a8100a80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.294+0000 7fc4ad88f640 1 -- 172.21.15.70:0/2999909240 >> 172.21.15.70:0/2999909240 conn(0x7fc4a80fbbf0 msgr2=0x7fc4a80fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:26.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.294+0000 7fc4ad88f640 1 -- 172.21.15.70:0/2999909240 shutdown_connections 2024-04-01T01:33:26.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.294+0000 7fc4ad88f640 1 -- 172.21.15.70:0/2999909240 wait complete. 2024-04-01T01:33:26.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.294+0000 7fc4ad88f640 1 Processor -- start 2024-04-01T01:33:26.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4ad88f640 1 -- start start 2024-04-01T01:33:26.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4ad88f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4a8100660 0x7fc4a81979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:26.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4ad88f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4a8101850 0x7fc4a8197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:26.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4ad88f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a8102210 0x7fc4a819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:26.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4ad88f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a810c180 con 0x7fc4a8102210 2024-04-01T01:33:26.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4ad88f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a810c000 con 0x7fc4a8101850 2024-04-01T01:33:26.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4ad88f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a810c300 con 0x7fc4a8100660 2024-04-01T01:33:26.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4a77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a8102210 0x7fc4a819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:26.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4a6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4a8100660 0x7fc4a81979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:26.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4a6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4a8100660 0x7fc4a81979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46834/0 (socket says 172.21.15.70:46834) 2024-04-01T01:33:26.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4a77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a8102210 0x7fc4a819ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52264/0 (socket says 172.21.15.70:52264) 2024-04-01T01:33:26.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4a6ffd640 1 -- 172.21.15.70:0/3615800728 learned_addr learned my addr 172.21.15.70:0/3615800728 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:26.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4a67fc640 1 --2- 172.21.15.70:0/3615800728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4a8101850 0x7fc4a8197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:26.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4a6ffd640 1 -- 172.21.15.70:0/3615800728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4a8101850 msgr2=0x7fc4a8197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:26.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4a6ffd640 1 --2- 172.21.15.70:0/3615800728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4a8101850 0x7fc4a8197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4a6ffd640 1 -- 172.21.15.70:0/3615800728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a8102210 msgr2=0x7fc4a819ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:26.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4a6ffd640 1 --2- 172.21.15.70:0/3615800728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a8102210 0x7fc4a819ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4a6ffd640 1 -- 172.21.15.70:0/3615800728 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a819f5f0 con 0x7fc4a8100660 2024-04-01T01:33:26.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.295+0000 7fc4a6ffd640 1 --2- 172.21.15.70:0/3615800728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4a8100660 0x7fc4a81979d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc498000c00 tx=0x7fc498005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:26.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.296+0000 7fc4ac88d640 1 -- 172.21.15.70:0/3615800728 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498043020 con 0x7fc4a8100660 2024-04-01T01:33:26.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.296+0000 7fc4ad88f640 1 -- 172.21.15.70:0/3615800728 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a819f880 con 0x7fc4a8100660 2024-04-01T01:33:26.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.296+0000 7fc4ac88d640 1 -- 172.21.15.70:0/3615800728 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc498004350 con 0x7fc4a8100660 2024-04-01T01:33:26.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.296+0000 7fc4ac88d640 1 -- 172.21.15.70:0/3615800728 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498003b10 con 0x7fc4a8100660 2024-04-01T01:33:26.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.296+0000 7fc4ad88f640 1 -- 172.21.15.70:0/3615800728 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4a819fdd0 con 0x7fc4a8100660 2024-04-01T01:33:26.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.297+0000 7fc4ad88f640 1 -- 172.21.15.70:0/3615800728 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4a819ffd0 con 0x7fc4a8100660 2024-04-01T01:33:26.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.298+0000 7fc4ac88d640 1 -- 172.21.15.70:0/3615800728 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fc498046020 con 0x7fc4a8100660 2024-04-01T01:33:26.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.298+0000 7fc4ac88d640 1 --2- 172.21.15.70:0/3615800728 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc4900a24a0 0x7fc4900a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:26.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.299+0000 7fc4ac88d640 1 -- 172.21.15.70:0/3615800728 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fc4980f1000 con 0x7fc4a8100660 2024-04-01T01:33:26.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.299+0000 7fc4a67fc640 1 --2- 172.21.15.70:0/3615800728 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc4900a24a0 0x7fc4900a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:26.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.299+0000 7fc4a67fc640 1 --2- 172.21.15.70:0/3615800728 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc4900a24a0 0x7fc4900a4960 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8198ee0 tx=0x7fc49c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:26.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.302+0000 7fc4ac88d640 1 -- 172.21.15.70:0/3615800728 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc49804d020 con 0x7fc4a8100660 2024-04-01T01:33:26.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.477+0000 7fc4ad88f640 1 -- 172.21.15.70:0/3615800728 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc4a81a01a0 con 0x7fc4a8100660 2024-04-01T01:33:26.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.477+0000 7fc4ac88d640 1 -- 172.21.15.70:0/3615800728 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fc4980bddf0 con 0x7fc4a8100660 2024-04-01T01:33:26.478 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-01T01:33:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.479+0000 7fc4ad88f640 1 -- 172.21.15.70:0/3615800728 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc4900a24a0 msgr2=0x7fc4900a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.479+0000 7fc4ad88f640 1 --2- 172.21.15.70:0/3615800728 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc4900a24a0 0x7fc4900a4960 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8198ee0 tx=0x7fc49c005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.479+0000 7fc4ad88f640 1 -- 172.21.15.70:0/3615800728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4a8100660 msgr2=0x7fc4a81979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.479+0000 7fc4ad88f640 1 --2- 172.21.15.70:0/3615800728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4a8100660 0x7fc4a81979d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc498000c00 tx=0x7fc498005dc0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.479+0000 7fc4ad88f640 1 -- 172.21.15.70:0/3615800728 shutdown_connections 2024-04-01T01:33:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.479+0000 7fc4ad88f640 1 --2- 172.21.15.70:0/3615800728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4a8102210 0x7fc4a819ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.479+0000 7fc4ad88f640 1 --2- 172.21.15.70:0/3615800728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4a8101850 0x7fc4a8197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.479+0000 7fc4ad88f640 1 --2- 172.21.15.70:0/3615800728 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc4900a24a0 0x7fc4900a4960 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.479+0000 7fc4ad88f640 1 --2- 172.21.15.70:0/3615800728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4a8100660 0x7fc4a81979d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.479+0000 7fc4ad88f640 1 -- 172.21.15.70:0/3615800728 >> 172.21.15.70:0/3615800728 conn(0x7fc4a80fbbf0 msgr2=0x7fc4a8069d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.479+0000 7fc4ad88f640 1 -- 172.21.15.70:0/3615800728 shutdown_connections 2024-04-01T01:33:26.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.479+0000 7fc4ad88f640 1 -- 172.21.15.70:0/3615800728 wait complete. 2024-04-01T01:33:26.491 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-01T01:33:26.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 -- 172.21.15.70:0/1041218480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a8102200 msgr2=0x7fa7a8108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:26.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 --2- 172.21.15.70:0/1041218480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a8102200 0x7fa7a8108a90 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0009be0 tx=0x7fa7a002bef0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 -- 172.21.15.70:0/1041218480 shutdown_connections 2024-04-01T01:33:26.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 --2- 172.21.15.70:0/1041218480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a8102200 0x7fa7a8108a90 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 --2- 172.21.15.70:0/1041218480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7a8101840 0x7fa7a8101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 --2- 172.21.15.70:0/1041218480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7a8100650 0x7fa7a8100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 -- 172.21.15.70:0/1041218480 >> 172.21.15.70:0/1041218480 conn(0x7fa7a80fbbe0 msgr2=0x7fa7a80fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:26.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 -- 172.21.15.70:0/1041218480 shutdown_connections 2024-04-01T01:33:26.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 -- 172.21.15.70:0/1041218480 wait complete. 2024-04-01T01:33:26.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 Processor -- start 2024-04-01T01:33:26.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 -- start start 2024-04-01T01:33:26.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7a8100650 0x7fa7a81968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:26.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7a8101840 0x7fa7a8196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:26.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a8102200 0x7fa7a819de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:26.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a810c180 con 0x7fa7a8102200 2024-04-01T01:33:26.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a810c000 con 0x7fa7a8100650 2024-04-01T01:33:26.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7aeb65640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a810c300 con 0x7fa7a8101840 2024-04-01T01:33:26.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7ad102640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a8102200 0x7fa7a819de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:26.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.597+0000 7fa7ad102640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a8102200 0x7fa7a819de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52280/0 (socket says 172.21.15.70:52280) 2024-04-01T01:33:26.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.598+0000 7fa7ad102640 1 -- 172.21.15.70:0/1025097566 learned_addr learned my addr 172.21.15.70:0/1025097566 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:26.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.598+0000 7fa7a7fff640 1 --2- 172.21.15.70:0/1025097566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7a8101840 0x7fa7a8196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:26.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.598+0000 7fa7ac901640 1 --2- 172.21.15.70:0/1025097566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7a8100650 0x7fa7a81968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:26.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.598+0000 7fa7ad102640 1 -- 172.21.15.70:0/1025097566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7a8101840 msgr2=0x7fa7a8196de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:26.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.598+0000 7fa7ad102640 1 --2- 172.21.15.70:0/1025097566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7a8101840 0x7fa7a8196de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.598+0000 7fa7ad102640 1 -- 172.21.15.70:0/1025097566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7a8100650 msgr2=0x7fa7a81968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:26.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.598+0000 7fa7ad102640 1 --2- 172.21.15.70:0/1025097566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7a8100650 0x7fa7a81968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.598+0000 7fa7ad102640 1 -- 172.21.15.70:0/1025097566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a819e4c0 con 0x7fa7a8102200 2024-04-01T01:33:26.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.598+0000 7fa7ad102640 1 --2- 172.21.15.70:0/1025097566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a8102200 0x7fa7a819de50 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0004af0 tx=0x7fa7a0033d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:26.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.598+0000 7fa7a5ffb640 1 -- 172.21.15.70:0/1025097566 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0043070 con 0x7fa7a8102200 2024-04-01T01:33:26.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.598+0000 7fa7aeb65640 1 -- 172.21.15.70:0/1025097566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a80688b0 con 0x7fa7a8102200 2024-04-01T01:33:26.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.598+0000 7fa7a5ffb640 1 -- 172.21.15.70:0/1025097566 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa7a0003b80 con 0x7fa7a8102200 2024-04-01T01:33:26.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.598+0000 7fa7a5ffb640 1 -- 172.21.15.70:0/1025097566 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0034720 con 0x7fa7a8102200 2024-04-01T01:33:26.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.598+0000 7fa7aeb65640 1 -- 172.21.15.70:0/1025097566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a8068e50 con 0x7fa7a8102200 2024-04-01T01:33:26.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.599+0000 7fa7aeb65640 1 -- 172.21.15.70:0/1025097566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7a8068540 con 0x7fa7a8102200 2024-04-01T01:33:26.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.600+0000 7fa7a5ffb640 1 -- 172.21.15.70:0/1025097566 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fa7a0046070 con 0x7fa7a8102200 2024-04-01T01:33:26.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.601+0000 7fa7a5ffb640 1 --2- 172.21.15.70:0/1025097566 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa7980a24a0 0x7fa7980a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:26.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.601+0000 7fa7ac901640 1 --2- 172.21.15.70:0/1025097566 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa7980a24a0 0x7fa7980a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:26.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.601+0000 7fa7a5ffb640 1 -- 172.21.15.70:0/1025097566 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa7a00f34f0 con 0x7fa7a8102200 2024-04-01T01:33:26.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.602+0000 7fa7ac901640 1 --2- 172.21.15.70:0/1025097566 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa7980a24a0 0x7fa7980a4960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa7a81016a0 tx=0x7fa79c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:26.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.605+0000 7fa7a5ffb640 1 -- 172.21.15.70:0/1025097566 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa7a002cc00 con 0x7fa7a8102200 2024-04-01T01:33:26.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:26 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2089416940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:26.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:26 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3615800728' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:26.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.782+0000 7fa7aeb65640 1 -- 172.21.15.70:0/1025097566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa7a81076b0 con 0x7fa7a8102200 2024-04-01T01:33:26.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.783+0000 7fa7a5ffb640 1 -- 172.21.15.70:0/1025097566 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fa7a00bfc30 con 0x7fa7a8102200 2024-04-01T01:33:26.783 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:33:26.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.785+0000 7fa7aeb65640 1 -- 172.21.15.70:0/1025097566 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa7980a24a0 msgr2=0x7fa7980a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.785+0000 7fa7aeb65640 1 --2- 172.21.15.70:0/1025097566 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa7980a24a0 0x7fa7980a4960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa7a81016a0 tx=0x7fa79c005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.785+0000 7fa7aeb65640 1 -- 172.21.15.70:0/1025097566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a8102200 msgr2=0x7fa7a819de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.785+0000 7fa7aeb65640 1 --2- 172.21.15.70:0/1025097566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a8102200 0x7fa7a819de50 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0004af0 tx=0x7fa7a0033d00 comp rx=0 tx=0).stop 2024-04-01T01:33:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.785+0000 7fa7aeb65640 1 -- 172.21.15.70:0/1025097566 shutdown_connections 2024-04-01T01:33:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.785+0000 7fa7aeb65640 1 --2- 172.21.15.70:0/1025097566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa7a8102200 0x7fa7a819de50 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.785+0000 7fa7aeb65640 1 --2- 172.21.15.70:0/1025097566 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa7980a24a0 0x7fa7980a4960 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.785+0000 7fa7aeb65640 1 --2- 172.21.15.70:0/1025097566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7a8101840 0x7fa7a8196de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.785+0000 7fa7aeb65640 1 --2- 172.21.15.70:0/1025097566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7a8100650 0x7fa7a81968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.785+0000 7fa7aeb65640 1 -- 172.21.15.70:0/1025097566 >> 172.21.15.70:0/1025097566 conn(0x7fa7a80fbbe0 msgr2=0x7fa7a8104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.785+0000 7fa7aeb65640 1 -- 172.21.15.70:0/1025097566 shutdown_connections 2024-04-01T01:33:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.785+0000 7fa7aeb65640 1 -- 172.21.15.70:0/1025097566 wait complete. 2024-04-01T01:33:26.797 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-01T01:33:26.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.901+0000 7f409747c640 1 -- 172.21.15.70:0/1454147724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4090101850 msgr2=0x7f4090101cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:26.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.901+0000 7f409747c640 1 --2- 172.21.15.70:0/1454147724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4090101850 0x7f4090101cd0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f408c009900 tx=0x7f408c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:26.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.901+0000 7f409747c640 1 -- 172.21.15.70:0/1454147724 shutdown_connections 2024-04-01T01:33:26.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.901+0000 7f409747c640 1 --2- 172.21.15.70:0/1454147724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4090102210 0x7f4090108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.901+0000 7f409747c640 1 --2- 172.21.15.70:0/1454147724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4090101850 0x7f4090101cd0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.901+0000 7f409747c640 1 --2- 172.21.15.70:0/1454147724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4090100660 0x7f4090100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.901+0000 7f409747c640 1 -- 172.21.15.70:0/1454147724 >> 172.21.15.70:0/1454147724 conn(0x7f40900fbbf0 msgr2=0x7f40900fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:26.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.901+0000 7f409747c640 1 -- 172.21.15.70:0/1454147724 shutdown_connections 2024-04-01T01:33:26.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.901+0000 7f409747c640 1 -- 172.21.15.70:0/1454147724 wait complete. 2024-04-01T01:33:26.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.901+0000 7f409747c640 1 Processor -- start 2024-04-01T01:33:26.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f409747c640 1 -- start start 2024-04-01T01:33:26.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f409747c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4090100660 0x7f4090197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:26.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f409747c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4090101850 0x7f4090197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:26.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f409747c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4090102210 0x7f409019eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:26.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f409747c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f409010d2b0 con 0x7f4090101850 2024-04-01T01:33:26.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f409747c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f409010d130 con 0x7f4090100660 2024-04-01T01:33:26.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f409747c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f409010d430 con 0x7f4090102210 2024-04-01T01:33:26.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f4094a17640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4090101850 0x7f4090197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:26.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f4094a17640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4090101850 0x7f4090197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60972/0 (socket says 172.21.15.70:60972) 2024-04-01T01:33:26.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f4094a17640 1 -- 172.21.15.70:0/3960462168 learned_addr learned my addr 172.21.15.70:0/3960462168 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:26.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f4095218640 1 --2- 172.21.15.70:0/3960462168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4090100660 0x7f4090197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:26.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f4095a19640 1 --2- 172.21.15.70:0/3960462168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4090102210 0x7f409019eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:26.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f4094a17640 1 -- 172.21.15.70:0/3960462168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4090102210 msgr2=0x7f409019eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:26.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f4094a17640 1 --2- 172.21.15.70:0/3960462168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4090102210 0x7f409019eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f4094a17640 1 -- 172.21.15.70:0/3960462168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4090100660 msgr2=0x7f4090197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:26.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f4094a17640 1 --2- 172.21.15.70:0/3960462168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4090100660 0x7f4090197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:26.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f4094a17640 1 -- 172.21.15.70:0/3960462168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409019f560 con 0x7f4090101850 2024-04-01T01:33:26.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f4094a17640 1 --2- 172.21.15.70:0/3960462168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4090101850 0x7f4090197e80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f408c006010 tx=0x7f408c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:26.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f40867fc640 1 -- 172.21.15.70:0/3960462168 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c043020 con 0x7f4090101850 2024-04-01T01:33:26.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f40867fc640 1 -- 172.21.15.70:0/3960462168 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f408c0043d0 con 0x7f4090101850 2024-04-01T01:33:26.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f409747c640 1 -- 172.21.15.70:0/3960462168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f409019f7f0 con 0x7f4090101850 2024-04-01T01:33:26.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.902+0000 7f40867fc640 1 -- 172.21.15.70:0/3960462168 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c003b80 con 0x7f4090101850 2024-04-01T01:33:26.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.903+0000 7f409747c640 1 -- 172.21.15.70:0/3960462168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f409019fd40 con 0x7f4090101850 2024-04-01T01:33:26.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.904+0000 7f409747c640 1 -- 172.21.15.70:0/3960462168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4090107980 con 0x7f4090101850 2024-04-01T01:33:26.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.904+0000 7f40867fc640 1 -- 172.21.15.70:0/3960462168 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f408c046020 con 0x7f4090101850 2024-04-01T01:33:26.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.905+0000 7f40867fc640 1 --2- 172.21.15.70:0/3960462168 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f407c0a24a0 0x7f407c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:26.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.905+0000 7f4095218640 1 --2- 172.21.15.70:0/3960462168 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f407c0a24a0 0x7f407c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:26.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.905+0000 7f40867fc640 1 -- 172.21.15.70:0/3960462168 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f408c0f0ca0 con 0x7f4090101850 2024-04-01T01:33:26.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.905+0000 7f4095218640 1 --2- 172.21.15.70:0/3960462168 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f407c0a24a0 0x7f407c0a4960 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f40901016b0 tx=0x7f4080005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:26.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:26.909+0000 7f40867fc640 1 -- 172.21.15.70:0/3960462168 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f408c04d020 con 0x7f4090101850 2024-04-01T01:33:26.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:26 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2089416940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:26.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:26 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3615800728' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:27.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:26 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2089416940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:27.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:26 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3615800728' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:27.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.082+0000 7f409747c640 1 -- 172.21.15.70:0/3960462168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f409002d170 con 0x7f4090101850 2024-04-01T01:33:27.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.082+0000 7f40867fc640 1 -- 172.21.15.70:0/3960462168 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f408c0bda90 con 0x7f4090101850 2024-04-01T01:33:27.083 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-01T01:33:27.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.084+0000 7f409747c640 1 -- 172.21.15.70:0/3960462168 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f407c0a24a0 msgr2=0x7f407c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:27.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.084+0000 7f409747c640 1 --2- 172.21.15.70:0/3960462168 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f407c0a24a0 0x7f407c0a4960 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f40901016b0 tx=0x7f4080005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:27.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.084+0000 7f409747c640 1 -- 172.21.15.70:0/3960462168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4090101850 msgr2=0x7f4090197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:27.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.084+0000 7f409747c640 1 --2- 172.21.15.70:0/3960462168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4090101850 0x7f4090197e80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f408c006010 tx=0x7f408c005e40 comp rx=0 tx=0).stop 2024-04-01T01:33:27.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.084+0000 7f409747c640 1 -- 172.21.15.70:0/3960462168 shutdown_connections 2024-04-01T01:33:27.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.084+0000 7f409747c640 1 --2- 172.21.15.70:0/3960462168 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f407c0a24a0 0x7f407c0a4960 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.084+0000 7f409747c640 1 --2- 172.21.15.70:0/3960462168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4090102210 0x7f409019eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.084+0000 7f409747c640 1 --2- 172.21.15.70:0/3960462168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4090101850 0x7f4090197e80 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.084+0000 7f409747c640 1 --2- 172.21.15.70:0/3960462168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4090100660 0x7f4090197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.084+0000 7f409747c640 1 -- 172.21.15.70:0/3960462168 >> 172.21.15.70:0/3960462168 conn(0x7f40900fbbf0 msgr2=0x7f4090104b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:27.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.084+0000 7f409747c640 1 -- 172.21.15.70:0/3960462168 shutdown_connections 2024-04-01T01:33:27.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.084+0000 7f409747c640 1 -- 172.21.15.70:0/3960462168 wait complete. 2024-04-01T01:33:27.096 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-01T01:33:27.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.199+0000 7f7d39f56640 1 -- 172.21.15.70:0/469081731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d34103330 msgr2=0x7f7d34109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:27.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.199+0000 7f7d39f56640 1 --2- 172.21.15.70:0/469081731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d34103330 0x7f7d34109bc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009ca0 tx=0x7f7d2c02c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.199+0000 7f7d39f56640 1 -- 172.21.15.70:0/469081731 shutdown_connections 2024-04-01T01:33:27.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.199+0000 7f7d39f56640 1 --2- 172.21.15.70:0/469081731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d34103330 0x7f7d34109bc0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d39f56640 1 --2- 172.21.15.70:0/469081731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d34102970 0x7f7d34102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d39f56640 1 --2- 172.21.15.70:0/469081731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d34101780 0x7f7d34101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d39f56640 1 -- 172.21.15.70:0/469081731 >> 172.21.15.70:0/469081731 conn(0x7f7d340fccf0 msgr2=0x7f7d340ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:27.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d39f56640 1 -- 172.21.15.70:0/469081731 shutdown_connections 2024-04-01T01:33:27.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d39f56640 1 -- 172.21.15.70:0/469081731 wait complete. 2024-04-01T01:33:27.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d39f56640 1 Processor -- start 2024-04-01T01:33:27.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d39f56640 1 -- start start 2024-04-01T01:33:27.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d39f56640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d34101780 0x7f7d34197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:27.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d39f56640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d34102970 0x7f7d34197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:27.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d39f56640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d34103330 0x7f7d3419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:27.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d39f56640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3410d2b0 con 0x7f7d34101780 2024-04-01T01:33:27.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d39f56640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3410d130 con 0x7f7d34102970 2024-04-01T01:33:27.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d39f56640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3410d430 con 0x7f7d34103330 2024-04-01T01:33:27.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d337fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d34101780 0x7f7d34197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:27.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d337fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d34101780 0x7f7d34197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60986/0 (socket says 172.21.15.70:60986) 2024-04-01T01:33:27.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d337fe640 1 -- 172.21.15.70:0/4245249325 learned_addr learned my addr 172.21.15.70:0/4245249325 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:27.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d2bfff640 1 --2- 172.21.15.70:0/4245249325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d34102970 0x7f7d34197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:27.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.200+0000 7f7d33fff640 1 --2- 172.21.15.70:0/4245249325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d34103330 0x7f7d3419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:27.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.201+0000 7f7d337fe640 1 -- 172.21.15.70:0/4245249325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d34103330 msgr2=0x7f7d3419ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:27.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.201+0000 7f7d337fe640 1 --2- 172.21.15.70:0/4245249325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d34103330 0x7f7d3419ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.201+0000 7f7d337fe640 1 -- 172.21.15.70:0/4245249325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d34102970 msgr2=0x7f7d34197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:27.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.201+0000 7f7d337fe640 1 --2- 172.21.15.70:0/4245249325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d34102970 0x7f7d34197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.201+0000 7f7d337fe640 1 -- 172.21.15.70:0/4245249325 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3419f5b0 con 0x7f7d34101780 2024-04-01T01:33:27.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.201+0000 7f7d337fe640 1 --2- 172.21.15.70:0/4245249325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d34101780 0x7f7d34197990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400e9c0 tx=0x7f7d2400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:27.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.201+0000 7f7d317fa640 1 -- 172.21.15.70:0/4245249325 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2400cd80 con 0x7f7d34101780 2024-04-01T01:33:27.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.201+0000 7f7d317fa640 1 -- 172.21.15.70:0/4245249325 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7d240035f0 con 0x7f7d34101780 2024-04-01T01:33:27.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.201+0000 7f7d317fa640 1 -- 172.21.15.70:0/4245249325 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d24010690 con 0x7f7d34101780 2024-04-01T01:33:27.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.201+0000 7f7d39f56640 1 -- 172.21.15.70:0/4245249325 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d3419f8a0 con 0x7f7d34101780 2024-04-01T01:33:27.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.201+0000 7f7d39f56640 1 -- 172.21.15.70:0/4245249325 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d3419fdf0 con 0x7f7d34101780 2024-04-01T01:33:27.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.202+0000 7f7d39f56640 1 -- 172.21.15.70:0/4245249325 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d34108ac0 con 0x7f7d34101780 2024-04-01T01:33:27.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.203+0000 7f7d317fa640 1 -- 172.21.15.70:0/4245249325 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f7d240037a0 con 0x7f7d34101780 2024-04-01T01:33:27.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.203+0000 7f7d317fa640 1 --2- 172.21.15.70:0/4245249325 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7d200a24a0 0x7f7d200a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:27.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.204+0000 7f7d2bfff640 1 --2- 172.21.15.70:0/4245249325 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7d200a24a0 0x7f7d200a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:27.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.204+0000 7f7d317fa640 1 -- 172.21.15.70:0/4245249325 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f7d240cee70 con 0x7f7d34101780 2024-04-01T01:33:27.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.204+0000 7f7d2bfff640 1 --2- 172.21.15.70:0/4245249325 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7d200a24a0 0x7f7d200a4960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7d34198ea0 tx=0x7f7d1c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:27.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.208+0000 7f7d317fa640 1 -- 172.21.15.70:0/4245249325 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7d2409bc60 con 0x7f7d34101780 2024-04-01T01:33:27.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.380+0000 7f7d39f56640 1 -- 172.21.15.70:0/4245249325 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7d34062350 con 0x7f7d34101780 2024-04-01T01:33:27.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.381+0000 7f7d317fa640 1 -- 172.21.15.70:0/4245249325 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f7d2409b560 con 0x7f7d34101780 2024-04-01T01:33:27.381 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:33:27.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.383+0000 7f7d39f56640 1 -- 172.21.15.70:0/4245249325 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7d200a24a0 msgr2=0x7f7d200a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:27.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.383+0000 7f7d39f56640 1 --2- 172.21.15.70:0/4245249325 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7d200a24a0 0x7f7d200a4960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7d34198ea0 tx=0x7f7d1c005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:27.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.383+0000 7f7d39f56640 1 -- 172.21.15.70:0/4245249325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d34101780 msgr2=0x7f7d34197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:27.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.383+0000 7f7d39f56640 1 --2- 172.21.15.70:0/4245249325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d34101780 0x7f7d34197990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400e9c0 tx=0x7f7d2400ee80 comp rx=0 tx=0).stop 2024-04-01T01:33:27.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.383+0000 7f7d39f56640 1 -- 172.21.15.70:0/4245249325 shutdown_connections 2024-04-01T01:33:27.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.383+0000 7f7d39f56640 1 --2- 172.21.15.70:0/4245249325 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7d200a24a0 0x7f7d200a4960 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.383+0000 7f7d39f56640 1 --2- 172.21.15.70:0/4245249325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d34103330 0x7f7d3419ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.383+0000 7f7d39f56640 1 --2- 172.21.15.70:0/4245249325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d34102970 0x7f7d34197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.383+0000 7f7d39f56640 1 --2- 172.21.15.70:0/4245249325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d34101780 0x7f7d34197990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.383+0000 7f7d39f56640 1 -- 172.21.15.70:0/4245249325 >> 172.21.15.70:0/4245249325 conn(0x7f7d340fccf0 msgr2=0x7f7d34105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:27.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.383+0000 7f7d39f56640 1 -- 172.21.15.70:0/4245249325 shutdown_connections 2024-04-01T01:33:27.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.383+0000 7f7d39f56640 1 -- 172.21.15.70:0/4245249325 wait complete. 2024-04-01T01:33:27.398 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:33:27.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 -- 172.21.15.70:0/1775005875 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd30100690 msgr2=0x7fdd30100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:27.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 --2- 172.21.15.70:0/1775005875 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd30100690 0x7fdd30100ab0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009900 tx=0x7fdd2402bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:27.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 -- 172.21.15.70:0/1775005875 shutdown_connections 2024-04-01T01:33:27.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 --2- 172.21.15.70:0/1775005875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd30102240 0x7fdd30108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 --2- 172.21.15.70:0/1775005875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd30101880 0x7fdd30101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 --2- 172.21.15.70:0/1775005875 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd30100690 0x7fdd30100ab0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 -- 172.21.15.70:0/1775005875 >> 172.21.15.70:0/1775005875 conn(0x7fdd300fbc00 msgr2=0x7fdd300fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 -- 172.21.15.70:0/1775005875 shutdown_connections 2024-04-01T01:33:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 -- 172.21.15.70:0/1775005875 wait complete. 2024-04-01T01:33:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 Processor -- start 2024-04-01T01:33:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 -- start start 2024-04-01T01:33:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd30100690 0x7fdd301968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd30101880 0x7fdd30196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd30102240 0x7fdd3019de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3010c190 con 0x7fdd30102240 2024-04-01T01:33:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3010c010 con 0x7fdd30101880 2024-04-01T01:33:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.499+0000 7fdd363e2640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3010c310 con 0x7fdd30100690 2024-04-01T01:33:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd3497f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd30102240 0x7fdd3019de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd2f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd30101880 0x7fdd30196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd2ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd30100690 0x7fdd301968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd3497f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd30102240 0x7fdd3019de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32784/0 (socket says 172.21.15.70:32784) 2024-04-01T01:33:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd3497f640 1 -- 172.21.15.70:0/3315328084 learned_addr learned my addr 172.21.15.70:0/3315328084 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd3497f640 1 -- 172.21.15.70:0/3315328084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd30100690 msgr2=0x7fdd301968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd3497f640 1 --2- 172.21.15.70:0/3315328084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd30100690 0x7fdd301968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd3497f640 1 -- 172.21.15.70:0/3315328084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd30101880 msgr2=0x7fdd30196df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd3497f640 1 --2- 172.21.15.70:0/3315328084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd30101880 0x7fdd30196df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd3497f640 1 -- 172.21.15.70:0/3315328084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd3019e4d0 con 0x7fdd30102240 2024-04-01T01:33:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd2f7fe640 1 --2- 172.21.15.70:0/3315328084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd30101880 0x7fdd30196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd3497f640 1 --2- 172.21.15.70:0/3315328084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd30102240 0x7fdd3019de60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800eb10 tx=0x7fdd2800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd2d7fa640 1 -- 172.21.15.70:0/3315328084 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd28004b20 con 0x7fdd30102240 2024-04-01T01:33:27.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd2d7fa640 1 -- 172.21.15.70:0/3315328084 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd28004cc0 con 0x7fdd30102240 2024-04-01T01:33:27.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd363e2640 1 -- 172.21.15.70:0/3315328084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd30069de0 con 0x7fdd30102240 2024-04-01T01:33:27.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd2d7fa640 1 -- 172.21.15.70:0/3315328084 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd28010690 con 0x7fdd30102240 2024-04-01T01:33:27.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.500+0000 7fdd363e2640 1 -- 172.21.15.70:0/3315328084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd3006a2e0 con 0x7fdd30102240 2024-04-01T01:33:27.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.501+0000 7fdd363e2640 1 -- 172.21.15.70:0/3315328084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd30068990 con 0x7fdd30102240 2024-04-01T01:33:27.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.502+0000 7fdd2d7fa640 1 -- 172.21.15.70:0/3315328084 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fdd28005000 con 0x7fdd30102240 2024-04-01T01:33:27.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.503+0000 7fdd2d7fa640 1 --2- 172.21.15.70:0/3315328084 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdd180a24a0 0x7fdd180a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:27.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.503+0000 7fdd2d7fa640 1 -- 172.21.15.70:0/3315328084 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fdd280cf270 con 0x7fdd30102240 2024-04-01T01:33:27.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.503+0000 7fdd2ffff640 1 --2- 172.21.15.70:0/3315328084 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdd180a24a0 0x7fdd180a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:27.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.504+0000 7fdd2ffff640 1 --2- 172.21.15.70:0/3315328084 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdd180a24a0 0x7fdd180a4960 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdd24005ab0 tx=0x7fdd24005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:27.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.506+0000 7fdd2d7fa640 1 -- 172.21.15.70:0/3315328084 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdd28053280 con 0x7fdd30102240 2024-04-01T01:33:27.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.719+0000 7fdd363e2640 1 -- 172.21.15.70:0/3315328084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd301076b0 con 0x7fdd30102240 2024-04-01T01:33:27.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.719+0000 7fdd2d7fa640 1 -- 172.21.15.70:0/3315328084 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11929 (secure 0 0 0) 0x7fdd2809c060 con 0x7fdd30102240 2024-04-01T01:33:27.720 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:27.720 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:31:38.563210+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24451,24464,24379,24404,24418]},"id":1}]} 2024-04-01T01:33:27.720 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 28 2024-04-01T01:33:27.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.721+0000 7fdd363e2640 1 -- 172.21.15.70:0/3315328084 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdd180a24a0 msgr2=0x7fdd180a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:27.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.721+0000 7fdd363e2640 1 --2- 172.21.15.70:0/3315328084 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdd180a24a0 0x7fdd180a4960 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdd24005ab0 tx=0x7fdd24005fd0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.721+0000 7fdd363e2640 1 -- 172.21.15.70:0/3315328084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd30102240 msgr2=0x7fdd3019de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:27.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.721+0000 7fdd363e2640 1 --2- 172.21.15.70:0/3315328084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd30102240 0x7fdd3019de60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800eb10 tx=0x7fdd2800c2d0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.721+0000 7fdd363e2640 1 -- 172.21.15.70:0/3315328084 shutdown_connections 2024-04-01T01:33:27.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.721+0000 7fdd363e2640 1 --2- 172.21.15.70:0/3315328084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdd30102240 0x7fdd3019de60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.721+0000 7fdd363e2640 1 --2- 172.21.15.70:0/3315328084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd30101880 0x7fdd30196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.721+0000 7fdd363e2640 1 --2- 172.21.15.70:0/3315328084 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdd180a24a0 0x7fdd180a4960 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.721+0000 7fdd363e2640 1 --2- 172.21.15.70:0/3315328084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd30100690 0x7fdd301968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.721+0000 7fdd363e2640 1 -- 172.21.15.70:0/3315328084 >> 172.21.15.70:0/3315328084 conn(0x7fdd300fbc00 msgr2=0x7fdd30104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:27.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.721+0000 7fdd363e2640 1 -- 172.21.15.70:0/3315328084 shutdown_connections 2024-04-01T01:33:27.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.721+0000 7fdd363e2640 1 -- 172.21.15.70:0/3315328084 wait complete. 2024-04-01T01:33:27.733 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:33:27.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.835+0000 7fe55fd9a640 1 -- 172.21.15.70:0/3130796460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5581022a0 msgr2=0x7fe558108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:27.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.835+0000 7fe55fd9a640 1 --2- 172.21.15.70:0/3130796460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5581022a0 0x7fe558108b30 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe548009900 tx=0x7fe54802bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:27.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.835+0000 7fe55fd9a640 1 -- 172.21.15.70:0/3130796460 shutdown_connections 2024-04-01T01:33:27.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.835+0000 7fe55fd9a640 1 --2- 172.21.15.70:0/3130796460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5581022a0 0x7fe558108b30 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.835+0000 7fe55fd9a640 1 --2- 172.21.15.70:0/3130796460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5581018e0 0x7fe558101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.835+0000 7fe55fd9a640 1 --2- 172.21.15.70:0/3130796460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5581006f0 0x7fe558100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.835+0000 7fe55fd9a640 1 -- 172.21.15.70:0/3130796460 >> 172.21.15.70:0/3130796460 conn(0x7fe5580fbc80 msgr2=0x7fe5580fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:27.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.835+0000 7fe55fd9a640 1 -- 172.21.15.70:0/3130796460 shutdown_connections 2024-04-01T01:33:27.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.835+0000 7fe55fd9a640 1 -- 172.21.15.70:0/3130796460 wait complete. 2024-04-01T01:33:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.835+0000 7fe55fd9a640 1 Processor -- start 2024-04-01T01:33:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.835+0000 7fe55fd9a640 1 -- start start 2024-04-01T01:33:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55fd9a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5581006f0 0x7fe55806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55fd9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5581018e0 0x7fe55819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55fd9a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5581022a0 0x7fe55819ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55fd9a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe55810c1b0 con 0x7fe5581022a0 2024-04-01T01:33:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55fd9a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe55810c030 con 0x7fe5581018e0 2024-04-01T01:33:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55fd9a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe55810c330 con 0x7fe5581006f0 2024-04-01T01:33:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55e337640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5581022a0 0x7fe55819ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55e337640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5581022a0 0x7fe55819ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32806/0 (socket says 172.21.15.70:32806) 2024-04-01T01:33:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55e337640 1 -- 172.21.15.70:0/1232034818 learned_addr learned my addr 172.21.15.70:0/1232034818 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55d335640 1 --2- 172.21.15.70:0/1232034818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5581018e0 0x7fe55819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55db36640 1 --2- 172.21.15.70:0/1232034818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5581006f0 0x7fe55806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:27.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55e337640 1 -- 172.21.15.70:0/1232034818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5581006f0 msgr2=0x7fe55806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:27.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55e337640 1 --2- 172.21.15.70:0/1232034818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5581006f0 0x7fe55806a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55e337640 1 -- 172.21.15.70:0/1232034818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5581018e0 msgr2=0x7fe55819c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:27.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55e337640 1 --2- 172.21.15.70:0/1232034818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5581018e0 0x7fe55819c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:27.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55e337640 1 -- 172.21.15.70:0/1232034818 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe55819f5d0 con 0x7fe5581022a0 2024-04-01T01:33:27.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55d335640 1 --2- 172.21.15.70:0/1232034818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5581018e0 0x7fe55819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:27.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55e337640 1 --2- 172.21.15.70:0/1232034818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5581022a0 0x7fe55819ecc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe548009230 tx=0x7fe548005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:27.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe556ffd640 1 -- 172.21.15.70:0/1232034818 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe548043020 con 0x7fe5581022a0 2024-04-01T01:33:27.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe556ffd640 1 -- 172.21.15.70:0/1232034818 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5480043d0 con 0x7fe5581022a0 2024-04-01T01:33:27.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55fd9a640 1 -- 172.21.15.70:0/1232034818 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe55819f800 con 0x7fe5581022a0 2024-04-01T01:33:27.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe556ffd640 1 -- 172.21.15.70:0/1232034818 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe548003b40 con 0x7fe5581022a0 2024-04-01T01:33:27.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.836+0000 7fe55fd9a640 1 -- 172.21.15.70:0/1232034818 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe55819fd50 con 0x7fe5581022a0 2024-04-01T01:33:27.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.837+0000 7fe55fd9a640 1 -- 172.21.15.70:0/1232034818 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe558107a30 con 0x7fe5581022a0 2024-04-01T01:33:27.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.838+0000 7fe556ffd640 1 -- 172.21.15.70:0/1232034818 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fe548046020 con 0x7fe5581022a0 2024-04-01T01:33:27.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.839+0000 7fe556ffd640 1 --2- 172.21.15.70:0/1232034818 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe5400a24a0 0x7fe5400a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:27.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.839+0000 7fe55db36640 1 --2- 172.21.15.70:0/1232034818 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe5400a24a0 0x7fe5400a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:27.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.839+0000 7fe556ffd640 1 -- 172.21.15.70:0/1232034818 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe5480f10b0 con 0x7fe5581022a0 2024-04-01T01:33:27.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.839+0000 7fe55db36640 1 --2- 172.21.15.70:0/1232034818 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe5400a24a0 0x7fe5400a4960 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe558101740 tx=0x7fe550009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:27.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:27.842+0000 7fe556ffd640 1 -- 172.21.15.70:0/1232034818 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe54804d020 con 0x7fe5581022a0 2024-04-01T01:33:27.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:27 smithi189 ceph-mon[35753]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:33:27.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:27 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1025097566' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:27.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:27 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3960462168' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:27.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:27 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/4245249325' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:28.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:27 smithi070 ceph-mon[32179]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:33:28.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:27 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1025097566' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:28.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:27 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3960462168' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:28.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:27 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/4245249325' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:28.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.018+0000 7fe55fd9a640 1 -- 172.21.15.70:0/1232034818 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe55802d340 con 0x7fe5581022a0 2024-04-01T01:33:28.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.019+0000 7fe556ffd640 1 -- 172.21.15.70:0/1232034818 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24781 (secure 0 0 0) 0x7fe5480bdea0 con 0x7fe5581022a0 2024-04-01T01:33:28.020 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:28.020 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":93,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:33:19.716245+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:33:28.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.021+0000 7fe55fd9a640 1 -- 172.21.15.70:0/1232034818 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe5400a24a0 msgr2=0x7fe5400a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.021+0000 7fe55fd9a640 1 --2- 172.21.15.70:0/1232034818 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe5400a24a0 0x7fe5400a4960 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe558101740 tx=0x7fe550009500 comp rx=0 tx=0).stop 2024-04-01T01:33:28.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.021+0000 7fe55fd9a640 1 -- 172.21.15.70:0/1232034818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5581022a0 msgr2=0x7fe55819ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.021+0000 7fe55fd9a640 1 --2- 172.21.15.70:0/1232034818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5581022a0 0x7fe55819ecc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe548009230 tx=0x7fe548005e40 comp rx=0 tx=0).stop 2024-04-01T01:33:28.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.021+0000 7fe55fd9a640 1 -- 172.21.15.70:0/1232034818 shutdown_connections 2024-04-01T01:33:28.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.021+0000 7fe55fd9a640 1 --2- 172.21.15.70:0/1232034818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5581022a0 0x7fe55819ecc0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.021+0000 7fe55fd9a640 1 --2- 172.21.15.70:0/1232034818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5581018e0 0x7fe55819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.021+0000 7fe55fd9a640 1 --2- 172.21.15.70:0/1232034818 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe5400a24a0 0x7fe5400a4960 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.021+0000 7fe55fd9a640 1 --2- 172.21.15.70:0/1232034818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe5581006f0 0x7fe55806a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.021+0000 7fe55fd9a640 1 -- 172.21.15.70:0/1232034818 >> 172.21.15.70:0/1232034818 conn(0x7fe5580fbc80 msgr2=0x7fe558104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:28.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.021+0000 7fe55fd9a640 1 -- 172.21.15.70:0/1232034818 shutdown_connections 2024-04-01T01:33:28.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.021+0000 7fe55fd9a640 1 -- 172.21.15.70:0/1232034818 wait complete. 2024-04-01T01:33:28.033 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-01T01:26:42.809779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-01T01:31:13.612369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-01T01:31:14.910162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-01T01:31:18.783179+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-01T01:33:28.033 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-01T01:33:28.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:27 smithi081 ceph-mon[35142]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-01T01:33:28.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:27 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1025097566' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:28.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:27 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3960462168' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:28.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:27 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/4245249325' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:28.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.138+0000 7f65dd89d640 1 -- 172.21.15.70:0/3275076682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65d8102210 msgr2=0x7f65d8108aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.138+0000 7f65dd89d640 1 --2- 172.21.15.70:0/3275076682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65d8102210 0x7f65d8108aa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f65d0009ca0 tx=0x7f65d002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.138+0000 7f65dd89d640 1 -- 172.21.15.70:0/3275076682 shutdown_connections 2024-04-01T01:33:28.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.138+0000 7f65dd89d640 1 --2- 172.21.15.70:0/3275076682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65d8102210 0x7f65d8108aa0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.138+0000 7f65dd89d640 1 --2- 172.21.15.70:0/3275076682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d8101850 0x7f65d8101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.138+0000 7f65dd89d640 1 --2- 172.21.15.70:0/3275076682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65d8100660 0x7f65d8100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.138+0000 7f65dd89d640 1 -- 172.21.15.70:0/3275076682 >> 172.21.15.70:0/3275076682 conn(0x7f65d80fbbb0 msgr2=0x7f65d80fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:28.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.138+0000 7f65dd89d640 1 -- 172.21.15.70:0/3275076682 shutdown_connections 2024-04-01T01:33:28.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.138+0000 7f65dd89d640 1 -- 172.21.15.70:0/3275076682 wait complete. 2024-04-01T01:33:28.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.139+0000 7f65dd89d640 1 Processor -- start 2024-04-01T01:33:28.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.139+0000 7f65dd89d640 1 -- start start 2024-04-01T01:33:28.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.139+0000 7f65dd89d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65d8100660 0x7f65d8199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:28.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.139+0000 7f65dd89d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d8101850 0x7f65d819a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:28.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.139+0000 7f65dd89d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65d8102210 0x7f65d81a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:28.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.139+0000 7f65dd89d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65d810c170 con 0x7f65d8101850 2024-04-01T01:33:28.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.139+0000 7f65dd89d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f65d810bff0 con 0x7f65d8102210 2024-04-01T01:33:28.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.139+0000 7f65dd89d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f65d810c2f0 con 0x7f65d8100660 2024-04-01T01:33:28.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.139+0000 7f65d67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d8101850 0x7f65d819a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:28.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.139+0000 7f65d67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d8101850 0x7f65d819a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32822/0 (socket says 172.21.15.70:32822) 2024-04-01T01:33:28.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.139+0000 7f65d67fc640 1 -- 172.21.15.70:0/2013428851 learned_addr learned my addr 172.21.15.70:0/2013428851 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:28.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.139+0000 7f65d6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65d8100660 0x7f65d8199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:28.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.139+0000 7f65d77fe640 1 --2- 172.21.15.70:0/2013428851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65d8102210 0x7f65d81a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:28.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.140+0000 7f65d67fc640 1 -- 172.21.15.70:0/2013428851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65d8100660 msgr2=0x7f65d8199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.140+0000 7f65d67fc640 1 --2- 172.21.15.70:0/2013428851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65d8100660 0x7f65d8199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.140+0000 7f65d67fc640 1 -- 172.21.15.70:0/2013428851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65d8102210 msgr2=0x7f65d81a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.140+0000 7f65d67fc640 1 --2- 172.21.15.70:0/2013428851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65d8102210 0x7f65d81a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.140+0000 7f65d67fc640 1 -- 172.21.15.70:0/2013428851 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d81a1760 con 0x7f65d8101850 2024-04-01T01:33:28.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.140+0000 7f65d67fc640 1 --2- 172.21.15.70:0/2013428851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d8101850 0x7f65d819a080 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00ea60 tx=0x7f65cc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:28.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.140+0000 7f65dc89b640 1 -- 172.21.15.70:0/2013428851 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc00cde0 con 0x7f65d8101850 2024-04-01T01:33:28.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.140+0000 7f65dc89b640 1 -- 172.21.15.70:0/2013428851 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f65cc0035f0 con 0x7f65d8101850 2024-04-01T01:33:28.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.140+0000 7f65dd89d640 1 -- 172.21.15.70:0/2013428851 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65d81a1a50 con 0x7f65d8101850 2024-04-01T01:33:28.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.140+0000 7f65dc89b640 1 -- 172.21.15.70:0/2013428851 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc010690 con 0x7f65d8101850 2024-04-01T01:33:28.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.140+0000 7f65dd89d640 1 -- 172.21.15.70:0/2013428851 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65d81a1fa0 con 0x7f65d8101850 2024-04-01T01:33:28.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.141+0000 7f65dd89d640 1 -- 172.21.15.70:0/2013428851 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65d81a2290 con 0x7f65d8101850 2024-04-01T01:33:28.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.142+0000 7f65dc89b640 1 -- 172.21.15.70:0/2013428851 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f65cc0037a0 con 0x7f65d8101850 2024-04-01T01:33:28.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.142+0000 7f65dc89b640 1 --2- 172.21.15.70:0/2013428851 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65c00a24a0 0x7f65c00a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:28.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.143+0000 7f65d6ffd640 1 --2- 172.21.15.70:0/2013428851 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65c00a24a0 0x7f65c00a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:28.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.143+0000 7f65dc89b640 1 -- 172.21.15.70:0/2013428851 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f65cc0ced60 con 0x7f65d8101850 2024-04-01T01:33:28.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.143+0000 7f65d6ffd640 1 --2- 172.21.15.70:0/2013428851 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65c00a24a0 0x7f65c00a4960 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f65d81016b0 tx=0x7f65c8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:28.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.147+0000 7f65dc89b640 1 -- 172.21.15.70:0/2013428851 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f65cc09bb50 con 0x7f65d8101850 2024-04-01T01:33:28.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.321+0000 7f65dd89d640 1 -- 172.21.15.70:0/2013428851 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f65d81a2460 con 0x7f65d8101850 2024-04-01T01:33:28.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.322+0000 7f65dc89b640 1 -- 172.21.15.70:0/2013428851 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f65cc09b450 con 0x7f65d8101850 2024-04-01T01:33:28.322 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-01T01:33:28.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.323+0000 7f65dd89d640 1 -- 172.21.15.70:0/2013428851 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65c00a24a0 msgr2=0x7f65c00a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.323+0000 7f65dd89d640 1 --2- 172.21.15.70:0/2013428851 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65c00a24a0 0x7f65c00a4960 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f65d81016b0 tx=0x7f65c8005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:28.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.323+0000 7f65dd89d640 1 -- 172.21.15.70:0/2013428851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d8101850 msgr2=0x7f65d819a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.323+0000 7f65dd89d640 1 --2- 172.21.15.70:0/2013428851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d8101850 0x7f65d819a080 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00ea60 tx=0x7f65cc00ef20 comp rx=0 tx=0).stop 2024-04-01T01:33:28.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.323+0000 7f65dd89d640 1 -- 172.21.15.70:0/2013428851 shutdown_connections 2024-04-01T01:33:28.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.323+0000 7f65dd89d640 1 --2- 172.21.15.70:0/2013428851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65d8102210 0x7f65d81a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.323+0000 7f65dd89d640 1 --2- 172.21.15.70:0/2013428851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d8101850 0x7f65d819a080 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.323+0000 7f65dd89d640 1 --2- 172.21.15.70:0/2013428851 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f65c00a24a0 0x7f65c00a4960 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.324+0000 7f65dd89d640 1 --2- 172.21.15.70:0/2013428851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65d8100660 0x7f65d8199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.324+0000 7f65dd89d640 1 -- 172.21.15.70:0/2013428851 >> 172.21.15.70:0/2013428851 conn(0x7f65d80fbbb0 msgr2=0x7f65d80fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:28.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.324+0000 7f65dd89d640 1 -- 172.21.15.70:0/2013428851 shutdown_connections 2024-04-01T01:33:28.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.324+0000 7f65dd89d640 1 -- 172.21.15.70:0/2013428851 wait complete. 2024-04-01T01:33:28.336 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-01T01:33:28.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.440+0000 7fa4471c2640 1 -- 172.21.15.70:0/100373689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa440102950 msgr2=0x7fa440102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.440+0000 7fa4471c2640 1 --2- 172.21.15.70:0/100373689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa440102950 0x7fa440102dd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009900 tx=0x7fa42c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:28.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.440+0000 7fa4471c2640 1 -- 172.21.15.70:0/100373689 shutdown_connections 2024-04-01T01:33:28.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.440+0000 7fa4471c2640 1 --2- 172.21.15.70:0/100373689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa440103310 0x7fa440109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.440+0000 7fa4471c2640 1 --2- 172.21.15.70:0/100373689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa440102950 0x7fa440102dd0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.440+0000 7fa4471c2640 1 --2- 172.21.15.70:0/100373689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa440101760 0x7fa440101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.440+0000 7fa4471c2640 1 -- 172.21.15.70:0/100373689 >> 172.21.15.70:0/100373689 conn(0x7fa4400fccf0 msgr2=0x7fa4400ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:28.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.440+0000 7fa4471c2640 1 -- 172.21.15.70:0/100373689 shutdown_connections 2024-04-01T01:33:28.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.440+0000 7fa4471c2640 1 -- 172.21.15.70:0/100373689 wait complete. 2024-04-01T01:33:28.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa4471c2640 1 Processor -- start 2024-04-01T01:33:28.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa4471c2640 1 -- start start 2024-04-01T01:33:28.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa4471c2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa440101760 0x7fa4401979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:28.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa4471c2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa440102950 0x7fa440197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:28.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa4471c2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa440103310 0x7fa44019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:28.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa4471c2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa44010d270 con 0x7fa440101760 2024-04-01T01:33:28.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa4471c2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa44010d0f0 con 0x7fa440103310 2024-04-01T01:33:28.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa4471c2640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa44010d3f0 con 0x7fa440102950 2024-04-01T01:33:28.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa444f5e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa440101760 0x7fa4401979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:28.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa444f5e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa440101760 0x7fa4401979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32846/0 (socket says 172.21.15.70:32846) 2024-04-01T01:33:28.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa444f5e640 1 -- 172.21.15.70:0/3996775754 learned_addr learned my addr 172.21.15.70:0/3996775754 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:28.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa43ffff640 1 --2- 172.21.15.70:0/3996775754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa440102950 0x7fa440197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:28.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa44575f640 1 --2- 172.21.15.70:0/3996775754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa440103310 0x7fa44019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:28.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa444f5e640 1 -- 172.21.15.70:0/3996775754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa440102950 msgr2=0x7fa440197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa444f5e640 1 --2- 172.21.15.70:0/3996775754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa440102950 0x7fa440197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa444f5e640 1 -- 172.21.15.70:0/3996775754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa440103310 msgr2=0x7fa44019ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa444f5e640 1 --2- 172.21.15.70:0/3996775754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa440103310 0x7fa44019ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa444f5e640 1 -- 172.21.15.70:0/3996775754 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa44019f5e0 con 0x7fa440101760 2024-04-01T01:33:28.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.441+0000 7fa444f5e640 1 --2- 172.21.15.70:0/3996775754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa440101760 0x7fa4401979c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa43400e9c0 tx=0x7fa43400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:28.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.442+0000 7fa43dffb640 1 -- 172.21.15.70:0/3996775754 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43400cd80 con 0x7fa440101760 2024-04-01T01:33:28.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.442+0000 7fa43dffb640 1 -- 172.21.15.70:0/3996775754 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4340035f0 con 0x7fa440101760 2024-04-01T01:33:28.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.442+0000 7fa43dffb640 1 -- 172.21.15.70:0/3996775754 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa434010690 con 0x7fa440101760 2024-04-01T01:33:28.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.442+0000 7fa4471c2640 1 -- 172.21.15.70:0/3996775754 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa44019f8d0 con 0x7fa440101760 2024-04-01T01:33:28.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.442+0000 7fa4471c2640 1 -- 172.21.15.70:0/3996775754 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa44019fe40 con 0x7fa440101760 2024-04-01T01:33:28.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.443+0000 7fa4471c2640 1 -- 172.21.15.70:0/3996775754 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa440108b40 con 0x7fa440101760 2024-04-01T01:33:28.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.444+0000 7fa43dffb640 1 -- 172.21.15.70:0/3996775754 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fa4340037a0 con 0x7fa440101760 2024-04-01T01:33:28.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.444+0000 7fa43dffb640 1 --2- 172.21.15.70:0/3996775754 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4300a2520 0x7fa4300a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:28.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.444+0000 7fa43ffff640 1 --2- 172.21.15.70:0/3996775754 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4300a2520 0x7fa4300a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:28.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.444+0000 7fa43dffb640 1 -- 172.21.15.70:0/3996775754 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa4340cede0 con 0x7fa440101760 2024-04-01T01:33:28.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.445+0000 7fa43ffff640 1 --2- 172.21.15.70:0/3996775754 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4300a2520 0x7fa4300a49e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009230 tx=0x7fa42c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:28.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.449+0000 7fa43dffb640 1 -- 172.21.15.70:0/3996775754 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa43409bbd0 con 0x7fa440101760 2024-04-01T01:33:28.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.626+0000 7fa4471c2640 1 -- 172.21.15.70:0/3996775754 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa440062350 con 0x7fa440101760 2024-04-01T01:33:28.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.627+0000 7fa43dffb640 1 -- 172.21.15.70:0/3996775754 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fa43409b4d0 con 0x7fa440101760 2024-04-01T01:33:28.627 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:33:28.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.629+0000 7fa4471c2640 1 -- 172.21.15.70:0/3996775754 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4300a2520 msgr2=0x7fa4300a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.629+0000 7fa4471c2640 1 --2- 172.21.15.70:0/3996775754 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4300a2520 0x7fa4300a49e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009230 tx=0x7fa42c005fd0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.629+0000 7fa4471c2640 1 -- 172.21.15.70:0/3996775754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa440101760 msgr2=0x7fa4401979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.629+0000 7fa4471c2640 1 --2- 172.21.15.70:0/3996775754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa440101760 0x7fa4401979c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa43400e9c0 tx=0x7fa43400ee80 comp rx=0 tx=0).stop 2024-04-01T01:33:28.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.629+0000 7fa4471c2640 1 -- 172.21.15.70:0/3996775754 shutdown_connections 2024-04-01T01:33:28.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.629+0000 7fa4471c2640 1 --2- 172.21.15.70:0/3996775754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa440103310 0x7fa44019ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.629+0000 7fa4471c2640 1 --2- 172.21.15.70:0/3996775754 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4300a2520 0x7fa4300a49e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.629+0000 7fa4471c2640 1 --2- 172.21.15.70:0/3996775754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa440102950 0x7fa440197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.629+0000 7fa4471c2640 1 --2- 172.21.15.70:0/3996775754 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa440101760 0x7fa4401979c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.629+0000 7fa4471c2640 1 -- 172.21.15.70:0/3996775754 >> 172.21.15.70:0/3996775754 conn(0x7fa4400fccf0 msgr2=0x7fa440105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:28.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.629+0000 7fa4471c2640 1 -- 172.21.15.70:0/3996775754 shutdown_connections 2024-04-01T01:33:28.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.629+0000 7fa4471c2640 1 -- 172.21.15.70:0/3996775754 wait complete. 2024-04-01T01:33:28.641 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-01T01:33:28.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.747+0000 7fe145a8f640 1 -- 172.21.15.70:0/1128549214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1401009a0 msgr2=0x7fe140100e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.747+0000 7fe145a8f640 1 --2- 172.21.15.70:0/1128549214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1401009a0 0x7fe140100e00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe128009900 tx=0x7fe12802bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.747+0000 7fe145a8f640 1 -- 172.21.15.70:0/1128549214 shutdown_connections 2024-04-01T01:33:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.747+0000 7fe145a8f640 1 --2- 172.21.15.70:0/1128549214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe140101340 0x7fe1401017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.747+0000 7fe145a8f640 1 --2- 172.21.15.70:0/1128549214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1401009a0 0x7fe140100e00 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.747+0000 7fe145a8f640 1 --2- 172.21.15.70:0/1128549214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe140100040 0x7fe140100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.747+0000 7fe145a8f640 1 -- 172.21.15.70:0/1128549214 >> 172.21.15.70:0/1128549214 conn(0x7fe1400fbbe0 msgr2=0x7fe1400fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.747+0000 7fe145a8f640 1 -- 172.21.15.70:0/1128549214 shutdown_connections 2024-04-01T01:33:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.747+0000 7fe145a8f640 1 -- 172.21.15.70:0/1128549214 wait complete. 2024-04-01T01:33:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.748+0000 7fe145a8f640 1 Processor -- start 2024-04-01T01:33:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.748+0000 7fe145a8f640 1 -- start start 2024-04-01T01:33:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.748+0000 7fe145a8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe140100040 0x7fe140199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.748+0000 7fe145a8f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1401009a0 0x7fe14019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:28.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.748+0000 7fe145a8f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe140101340 0x7fe1401a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.748+0000 7fe145a8f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe14010f490 con 0x7fe140100040 2024-04-01T01:33:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.748+0000 7fe145a8f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe14010f310 con 0x7fe140101340 2024-04-01T01:33:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.748+0000 7fe145a8f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe14010f610 con 0x7fe1401009a0 2024-04-01T01:33:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.748+0000 7fe13f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe140100040 0x7fe140199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.748+0000 7fe13f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe140100040 0x7fe140199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32858/0 (socket says 172.21.15.70:32858) 2024-04-01T01:33:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.748+0000 7fe13f7fe640 1 -- 172.21.15.70:0/1983823009 learned_addr learned my addr 172.21.15.70:0/1983823009 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.748+0000 7fe13ffff640 1 --2- 172.21.15.70:0/1983823009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe140101340 0x7fe1401a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.748+0000 7fe13effd640 1 --2- 172.21.15.70:0/1983823009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1401009a0 0x7fe14019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.749+0000 7fe13f7fe640 1 -- 172.21.15.70:0/1983823009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1401009a0 msgr2=0x7fe14019a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.749+0000 7fe13f7fe640 1 --2- 172.21.15.70:0/1983823009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1401009a0 0x7fe14019a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.749+0000 7fe13f7fe640 1 -- 172.21.15.70:0/1983823009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe140101340 msgr2=0x7fe1401a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.749+0000 7fe13f7fe640 1 --2- 172.21.15.70:0/1983823009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe140101340 0x7fe1401a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.749+0000 7fe13f7fe640 1 -- 172.21.15.70:0/1983823009 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1401a1750 con 0x7fe140100040 2024-04-01T01:33:28.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.749+0000 7fe13f7fe640 1 --2- 172.21.15.70:0/1983823009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe140100040 0x7fe140199b30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe13000e9f0 tx=0x7fe13000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:28.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.749+0000 7fe13cff9640 1 -- 172.21.15.70:0/1983823009 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13000cd80 con 0x7fe140100040 2024-04-01T01:33:28.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.749+0000 7fe13cff9640 1 -- 172.21.15.70:0/1983823009 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1300108e0 con 0x7fe140100040 2024-04-01T01:33:28.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.749+0000 7fe13cff9640 1 -- 172.21.15.70:0/1983823009 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe130018920 con 0x7fe140100040 2024-04-01T01:33:28.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.749+0000 7fe145a8f640 1 -- 172.21.15.70:0/1983823009 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1401a1a40 con 0x7fe140100040 2024-04-01T01:33:28.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.749+0000 7fe145a8f640 1 -- 172.21.15.70:0/1983823009 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1401a1f60 con 0x7fe140100040 2024-04-01T01:33:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.750+0000 7fe145a8f640 1 -- 172.21.15.70:0/1983823009 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe140068990 con 0x7fe140100040 2024-04-01T01:33:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.751+0000 7fe13cff9640 1 -- 172.21.15.70:0/1983823009 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fe130010a90 con 0x7fe140100040 2024-04-01T01:33:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.751+0000 7fe13cff9640 1 --2- 172.21.15.70:0/1983823009 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe12c0a24a0 0x7fe12c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.752+0000 7fe13cff9640 1 -- 172.21.15.70:0/1983823009 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe1300cedc0 con 0x7fe140100040 2024-04-01T01:33:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.752+0000 7fe13effd640 1 --2- 172.21.15.70:0/1983823009 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe12c0a24a0 0x7fe12c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:28.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.752+0000 7fe13effd640 1 --2- 172.21.15.70:0/1983823009 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe12c0a24a0 0x7fe12c0a4960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe128009230 tx=0x7fe128005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:28.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.756+0000 7fe13cff9640 1 -- 172.21.15.70:0/1983823009 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe13009b560 con 0x7fe140100040 2024-04-01T01:33:28.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.929+0000 7fe145a8f640 1 -- 172.21.15.70:0/1983823009 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe140062380 con 0x7fe140100040 2024-04-01T01:33:28.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.930+0000 7fe13cff9640 1 -- 172.21.15.70:0/1983823009 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fe13009f600 con 0x7fe140100040 2024-04-01T01:33:28.930 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-01T01:33:28.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.932+0000 7fe145a8f640 1 -- 172.21.15.70:0/1983823009 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe12c0a24a0 msgr2=0x7fe12c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.932+0000 7fe145a8f640 1 --2- 172.21.15.70:0/1983823009 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe12c0a24a0 0x7fe12c0a4960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe128009230 tx=0x7fe128005fd0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.932+0000 7fe145a8f640 1 -- 172.21.15.70:0/1983823009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe140100040 msgr2=0x7fe140199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:28.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.932+0000 7fe145a8f640 1 --2- 172.21.15.70:0/1983823009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe140100040 0x7fe140199b30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe13000e9f0 tx=0x7fe13000eeb0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.932+0000 7fe145a8f640 1 -- 172.21.15.70:0/1983823009 shutdown_connections 2024-04-01T01:33:28.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.932+0000 7fe145a8f640 1 --2- 172.21.15.70:0/1983823009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe140101340 0x7fe1401a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.932+0000 7fe145a8f640 1 --2- 172.21.15.70:0/1983823009 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe12c0a24a0 0x7fe12c0a4960 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.932+0000 7fe145a8f640 1 --2- 172.21.15.70:0/1983823009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1401009a0 0x7fe14019a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.932+0000 7fe145a8f640 1 --2- 172.21.15.70:0/1983823009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe140100040 0x7fe140199b30 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:28.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.932+0000 7fe145a8f640 1 -- 172.21.15.70:0/1983823009 >> 172.21.15.70:0/1983823009 conn(0x7fe1400fbbe0 msgr2=0x7fe1400fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:28.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.932+0000 7fe145a8f640 1 -- 172.21.15.70:0/1983823009 shutdown_connections 2024-04-01T01:33:28.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:28.932+0000 7fe145a8f640 1 -- 172.21.15.70:0/1983823009 wait complete. 2024-04-01T01:33:28.942 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:28 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3315328084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:28.942 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:28 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1232034818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:28.942 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:28 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2013428851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:28.942 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:28 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3996775754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:28.943 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-01T01:33:28.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:28 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3315328084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:28.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:28 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1232034818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:28.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:28 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2013428851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:28.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:28 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3996775754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:29.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.048+0000 7fbbf17c9640 1 -- 172.21.15.70:0/2122190141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbec100650 msgr2=0x7fbbec100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.048+0000 7fbbf17c9640 1 --2- 172.21.15.70:0/2122190141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbec100650 0x7fbbec100a70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc009900 tx=0x7fbbdc02bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.048+0000 7fbbf17c9640 1 -- 172.21.15.70:0/2122190141 shutdown_connections 2024-04-01T01:33:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.048+0000 7fbbf17c9640 1 --2- 172.21.15.70:0/2122190141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbec102200 0x7fbbec108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.048+0000 7fbbf17c9640 1 --2- 172.21.15.70:0/2122190141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbec101840 0x7fbbec101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.048+0000 7fbbf17c9640 1 --2- 172.21.15.70:0/2122190141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbec100650 0x7fbbec100a70 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.048+0000 7fbbf17c9640 1 -- 172.21.15.70:0/2122190141 >> 172.21.15.70:0/2122190141 conn(0x7fbbec0fbbe0 msgr2=0x7fbbec0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.048+0000 7fbbf17c9640 1 -- 172.21.15.70:0/2122190141 shutdown_connections 2024-04-01T01:33:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.048+0000 7fbbf17c9640 1 -- 172.21.15.70:0/2122190141 wait complete. 2024-04-01T01:33:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.049+0000 7fbbf17c9640 1 Processor -- start 2024-04-01T01:33:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.049+0000 7fbbf17c9640 1 -- start start 2024-04-01T01:33:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.049+0000 7fbbf17c9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbec100650 0x7fbbec197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.049+0000 7fbbf17c9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbec101840 0x7fbbec197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.049+0000 7fbbf17c9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbec102200 0x7fbbec19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:29.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.049+0000 7fbbf17c9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec10d2a0 con 0x7fbbec102200 2024-04-01T01:33:29.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.049+0000 7fbbf17c9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec10d120 con 0x7fbbec101840 2024-04-01T01:33:29.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.049+0000 7fbbf17c9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec10d420 con 0x7fbbec100650 2024-04-01T01:33:29.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.049+0000 7fbbeb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbec102200 0x7fbbec19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:29.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.049+0000 7fbbeb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbec102200 0x7fbbec19efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32872/0 (socket says 172.21.15.70:32872) 2024-04-01T01:33:29.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.049+0000 7fbbeb7fe640 1 -- 172.21.15.70:0/1537160345 learned_addr learned my addr 172.21.15.70:0/1537160345 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:29.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.049+0000 7fbbeaffd640 1 --2- 172.21.15.70:0/1537160345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbec100650 0x7fbbec197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:29.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.049+0000 7fbbea7fc640 1 --2- 172.21.15.70:0/1537160345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbec101840 0x7fbbec197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:29.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.050+0000 7fbbeb7fe640 1 -- 172.21.15.70:0/1537160345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbec100650 msgr2=0x7fbbec197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.050+0000 7fbbeb7fe640 1 --2- 172.21.15.70:0/1537160345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbec100650 0x7fbbec197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.050+0000 7fbbeb7fe640 1 -- 172.21.15.70:0/1537160345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbec101840 msgr2=0x7fbbec197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.050+0000 7fbbeb7fe640 1 --2- 172.21.15.70:0/1537160345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbec101840 0x7fbbec197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.050+0000 7fbbeb7fe640 1 -- 172.21.15.70:0/1537160345 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbec19f630 con 0x7fbbec102200 2024-04-01T01:33:29.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.050+0000 7fbbea7fc640 1 --2- 172.21.15.70:0/1537160345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbec101840 0x7fbbec197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:29.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.050+0000 7fbbeaffd640 1 --2- 172.21.15.70:0/1537160345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbec100650 0x7fbbec197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:29.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.050+0000 7fbbeb7fe640 1 --2- 172.21.15.70:0/1537160345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbec102200 0x7fbbec19efc0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400eb10 tx=0x7fbbe400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:29.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.050+0000 7fbbd3fff640 1 -- 172.21.15.70:0/1537160345 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4004b20 con 0x7fbbec102200 2024-04-01T01:33:29.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.050+0000 7fbbf17c9640 1 -- 172.21.15.70:0/1537160345 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbec19f920 con 0x7fbbec102200 2024-04-01T01:33:29.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.050+0000 7fbbd3fff640 1 -- 172.21.15.70:0/1537160345 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbe4004cc0 con 0x7fbbec102200 2024-04-01T01:33:29.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.050+0000 7fbbd3fff640 1 -- 172.21.15.70:0/1537160345 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4010690 con 0x7fbbec102200 2024-04-01T01:33:29.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.050+0000 7fbbf17c9640 1 -- 172.21.15.70:0/1537160345 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbec19fe70 con 0x7fbbec102200 2024-04-01T01:33:29.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.051+0000 7fbbf17c9640 1 -- 172.21.15.70:0/1537160345 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbec107ae0 con 0x7fbbec102200 2024-04-01T01:33:29.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.052+0000 7fbbd3fff640 1 -- 172.21.15.70:0/1537160345 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fbbe4005020 con 0x7fbbec102200 2024-04-01T01:33:29.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.053+0000 7fbbd3fff640 1 --2- 172.21.15.70:0/1537160345 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbd40a24a0 0x7fbbd40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:29.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.053+0000 7fbbeaffd640 1 --2- 172.21.15.70:0/1537160345 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbd40a24a0 0x7fbbd40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:29.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.053+0000 7fbbd3fff640 1 -- 172.21.15.70:0/1537160345 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fbbe40cf1e0 con 0x7fbbec102200 2024-04-01T01:33:29.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.053+0000 7fbbeaffd640 1 --2- 172.21.15.70:0/1537160345 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbd40a24a0 0x7fbbd40a4960 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc005ab0 tx=0x7fbbdc005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:29.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.057+0000 7fbbd3fff640 1 -- 172.21.15.70:0/1537160345 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbbe409bfd0 con 0x7fbbec102200 2024-04-01T01:33:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:28 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3315328084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:28 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1232034818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:28 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2013428851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:28 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3996775754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:29.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.230+0000 7fbbf17c9640 1 -- 172.21.15.70:0/1537160345 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbbec062350 con 0x7fbbec102200 2024-04-01T01:33:29.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.230+0000 7fbbd3fff640 1 -- 172.21.15.70:0/1537160345 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fbbe409b8d0 con 0x7fbbec102200 2024-04-01T01:33:29.231 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:33:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.232+0000 7fbbf17c9640 1 -- 172.21.15.70:0/1537160345 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbd40a24a0 msgr2=0x7fbbd40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.232+0000 7fbbf17c9640 1 --2- 172.21.15.70:0/1537160345 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbd40a24a0 0x7fbbd40a4960 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc005ab0 tx=0x7fbbdc005fd0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.232+0000 7fbbf17c9640 1 -- 172.21.15.70:0/1537160345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbec102200 msgr2=0x7fbbec19efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.232+0000 7fbbf17c9640 1 --2- 172.21.15.70:0/1537160345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbec102200 0x7fbbec19efc0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400eb10 tx=0x7fbbe400c2d0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.232+0000 7fbbf17c9640 1 -- 172.21.15.70:0/1537160345 shutdown_connections 2024-04-01T01:33:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.232+0000 7fbbf17c9640 1 --2- 172.21.15.70:0/1537160345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbec102200 0x7fbbec19efc0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.232+0000 7fbbf17c9640 1 --2- 172.21.15.70:0/1537160345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbec101840 0x7fbbec197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.232+0000 7fbbf17c9640 1 --2- 172.21.15.70:0/1537160345 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbd40a24a0 0x7fbbd40a4960 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.232+0000 7fbbf17c9640 1 --2- 172.21.15.70:0/1537160345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbec100650 0x7fbbec197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.232+0000 7fbbf17c9640 1 -- 172.21.15.70:0/1537160345 >> 172.21.15.70:0/1537160345 conn(0x7fbbec0fbbe0 msgr2=0x7fbbec104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:29.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.233+0000 7fbbf17c9640 1 -- 172.21.15.70:0/1537160345 shutdown_connections 2024-04-01T01:33:29.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.233+0000 7fbbf17c9640 1 -- 172.21.15.70:0/1537160345 wait complete. 2024-04-01T01:33:29.244 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:33:29.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 -- 172.21.15.70:0/3681652626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc670102960 msgr2=0x7fc670102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 --2- 172.21.15.70:0/3681652626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc670102960 0x7fc670102de0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc660009ca0 tx=0x7fc66002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 -- 172.21.15.70:0/3681652626 shutdown_connections 2024-04-01T01:33:29.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 --2- 172.21.15.70:0/3681652626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc670103320 0x7fc670109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 --2- 172.21.15.70:0/3681652626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc670102960 0x7fc670102de0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 --2- 172.21.15.70:0/3681652626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc670101770 0x7fc670101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 -- 172.21.15.70:0/3681652626 >> 172.21.15.70:0/3681652626 conn(0x7fc6700fcd00 msgr2=0x7fc6700ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:29.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 -- 172.21.15.70:0/3681652626 shutdown_connections 2024-04-01T01:33:29.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 -- 172.21.15.70:0/3681652626 wait complete. 2024-04-01T01:33:29.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 Processor -- start 2024-04-01T01:33:29.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 -- start start 2024-04-01T01:33:29.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc670101770 0x7fc6701979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:29.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc670102960 0x7fc670197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:29.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc670103320 0x7fc67019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:29.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc67010d2a0 con 0x7fc670102960 2024-04-01T01:33:29.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc67010d120 con 0x7fc670101770 2024-04-01T01:33:29.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.352+0000 7fc676da7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc67010d420 con 0x7fc670103320 2024-04-01T01:33:29.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.353+0000 7fc66ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc670102960 0x7fc670197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:29.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.353+0000 7fc66ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc670102960 0x7fc670197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32890/0 (socket says 172.21.15.70:32890) 2024-04-01T01:33:29.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.353+0000 7fc66ffff640 1 -- 172.21.15.70:0/2009846262 learned_addr learned my addr 172.21.15.70:0/2009846262 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:29.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.353+0000 7fc674b43640 1 --2- 172.21.15.70:0/2009846262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc670101770 0x7fc6701979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:29.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.353+0000 7fc675344640 1 --2- 172.21.15.70:0/2009846262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc670103320 0x7fc67019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:29.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.353+0000 7fc66ffff640 1 -- 172.21.15.70:0/2009846262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc670103320 msgr2=0x7fc67019ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.353+0000 7fc66ffff640 1 --2- 172.21.15.70:0/2009846262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc670103320 0x7fc67019ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.353+0000 7fc66ffff640 1 -- 172.21.15.70:0/2009846262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc670101770 msgr2=0x7fc6701979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.353+0000 7fc66ffff640 1 --2- 172.21.15.70:0/2009846262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc670101770 0x7fc6701979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.353+0000 7fc66ffff640 1 -- 172.21.15.70:0/2009846262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc67019f5c0 con 0x7fc670102960 2024-04-01T01:33:29.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.353+0000 7fc66ffff640 1 --2- 172.21.15.70:0/2009846262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc670102960 0x7fc670197ee0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc660009dc0 tx=0x7fc6600328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:29.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.353+0000 7fc66dffb640 1 -- 172.21.15.70:0/2009846262 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc660041020 con 0x7fc670102960 2024-04-01T01:33:29.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.353+0000 7fc676da7640 1 -- 172.21.15.70:0/2009846262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc67019f850 con 0x7fc670102960 2024-04-01T01:33:29.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.353+0000 7fc66dffb640 1 -- 172.21.15.70:0/2009846262 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc660003e20 con 0x7fc670102960 2024-04-01T01:33:29.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.353+0000 7fc66dffb640 1 -- 172.21.15.70:0/2009846262 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc660039040 con 0x7fc670102960 2024-04-01T01:33:29.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.353+0000 7fc676da7640 1 -- 172.21.15.70:0/2009846262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc67019fd40 con 0x7fc670102960 2024-04-01T01:33:29.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.354+0000 7fc676da7640 1 -- 172.21.15.70:0/2009846262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc670108ae0 con 0x7fc670102960 2024-04-01T01:33:29.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.355+0000 7fc66dffb640 1 -- 172.21.15.70:0/2009846262 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fc660044020 con 0x7fc670102960 2024-04-01T01:33:29.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.356+0000 7fc66dffb640 1 --2- 172.21.15.70:0/2009846262 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc65c0a24a0 0x7fc65c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:29.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.356+0000 7fc674b43640 1 --2- 172.21.15.70:0/2009846262 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc65c0a24a0 0x7fc65c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:29.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.356+0000 7fc66dffb640 1 -- 172.21.15.70:0/2009846262 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fc6600f0fa0 con 0x7fc670102960 2024-04-01T01:33:29.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.357+0000 7fc674b43640 1 --2- 172.21.15.70:0/2009846262 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc65c0a24a0 0x7fc65c0a4960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc6701027c0 tx=0x7fc668005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:29.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.359+0000 7fc66dffb640 1 -- 172.21.15.70:0/2009846262 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc66004b020 con 0x7fc670102960 2024-04-01T01:33:29.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.571+0000 7fc676da7640 1 -- 172.21.15.70:0/2009846262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc67002d3c0 con 0x7fc670102960 2024-04-01T01:33:29.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.571+0000 7fc66dffb640 1 -- 172.21.15.70:0/2009846262 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11929 (secure 0 0 0) 0x7fc6600bdd90 con 0x7fc670102960 2024-04-01T01:33:29.572 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:29.572 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:31:38.563210+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24451,24464,24379,24404,24418]},"id":1}]} 2024-04-01T01:33:29.572 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 28 2024-04-01T01:33:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.573+0000 7fc676da7640 1 -- 172.21.15.70:0/2009846262 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc65c0a24a0 msgr2=0x7fc65c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.573+0000 7fc676da7640 1 --2- 172.21.15.70:0/2009846262 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc65c0a24a0 0x7fc65c0a4960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc6701027c0 tx=0x7fc668005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.573+0000 7fc676da7640 1 -- 172.21.15.70:0/2009846262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc670102960 msgr2=0x7fc670197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.573+0000 7fc676da7640 1 --2- 172.21.15.70:0/2009846262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc670102960 0x7fc670197ee0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc660009dc0 tx=0x7fc6600328a0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.573+0000 7fc676da7640 1 -- 172.21.15.70:0/2009846262 shutdown_connections 2024-04-01T01:33:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.573+0000 7fc676da7640 1 --2- 172.21.15.70:0/2009846262 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc65c0a24a0 0x7fc65c0a4960 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.573+0000 7fc676da7640 1 --2- 172.21.15.70:0/2009846262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc670103320 0x7fc67019ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.573+0000 7fc676da7640 1 --2- 172.21.15.70:0/2009846262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc670102960 0x7fc670197ee0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.573+0000 7fc676da7640 1 --2- 172.21.15.70:0/2009846262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc670101770 0x7fc6701979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.573+0000 7fc676da7640 1 -- 172.21.15.70:0/2009846262 >> 172.21.15.70:0/2009846262 conn(0x7fc6700fcd00 msgr2=0x7fc670105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:29.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.574+0000 7fc676da7640 1 -- 172.21.15.70:0/2009846262 shutdown_connections 2024-04-01T01:33:29.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.574+0000 7fc676da7640 1 -- 172.21.15.70:0/2009846262 wait complete. 2024-04-01T01:33:29.585 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:33:29.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.688+0000 7f3f9471e640 1 -- 172.21.15.70:0/2322252585 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f8c0ffe00 msgr2=0x7f3f8c100220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.688+0000 7f3f9471e640 1 --2- 172.21.15.70:0/2322252585 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f8c0ffe00 0x7f3f8c100220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3f88009860 tx=0x7f3f8802bb70 comp rx=0 tx=0).stop 2024-04-01T01:33:29.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.688+0000 7f3f9471e640 1 -- 172.21.15.70:0/2322252585 shutdown_connections 2024-04-01T01:33:29.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.688+0000 7f3f9471e640 1 --2- 172.21.15.70:0/2322252585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f8c1019b0 0x7f3f8c108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.688+0000 7f3f9471e640 1 --2- 172.21.15.70:0/2322252585 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f8c100ff0 0x7f3f8c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.688+0000 7f3f9471e640 1 --2- 172.21.15.70:0/2322252585 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f8c0ffe00 0x7f3f8c100220 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.688+0000 7f3f9471e640 1 -- 172.21.15.70:0/2322252585 >> 172.21.15.70:0/2322252585 conn(0x7f3f8c0fb370 msgr2=0x7f3f8c0fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:29.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.688+0000 7f3f9471e640 1 -- 172.21.15.70:0/2322252585 shutdown_connections 2024-04-01T01:33:29.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.688+0000 7f3f9471e640 1 -- 172.21.15.70:0/2322252585 wait complete. 2024-04-01T01:33:29.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f9471e640 1 Processor -- start 2024-04-01T01:33:29.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f9471e640 1 -- start start 2024-04-01T01:33:29.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f9471e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f8c0ffe00 0x7f3f8c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:29.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f9471e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f8c100ff0 0x7f3f8c197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:29.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f9471e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f8c1019b0 0x7f3f8c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:29.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f9471e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8c10b080 con 0x7f3f8c1019b0 2024-04-01T01:33:29.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f9471e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8c10af00 con 0x7f3f8c0ffe00 2024-04-01T01:33:29.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f9471e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8c10b200 con 0x7f3f8c100ff0 2024-04-01T01:33:29.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f92cbb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f8c1019b0 0x7f3f8c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:29.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f92cbb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f8c1019b0 0x7f3f8c19efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32908/0 (socket says 172.21.15.70:32908) 2024-04-01T01:33:29.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f92cbb640 1 -- 172.21.15.70:0/801213178 learned_addr learned my addr 172.21.15.70:0/801213178 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:29.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f924ba640 1 --2- 172.21.15.70:0/801213178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f8c0ffe00 0x7f3f8c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f91cb9640 1 --2- 172.21.15.70:0/801213178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f8c100ff0 0x7f3f8c197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f92cbb640 1 -- 172.21.15.70:0/801213178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f8c100ff0 msgr2=0x7f3f8c197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f92cbb640 1 --2- 172.21.15.70:0/801213178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f8c100ff0 0x7f3f8c197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f92cbb640 1 -- 172.21.15.70:0/801213178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f8c0ffe00 msgr2=0x7f3f8c197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f92cbb640 1 --2- 172.21.15.70:0/801213178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f8c0ffe00 0x7f3f8c197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f92cbb640 1 -- 172.21.15.70:0/801213178 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8c19f620 con 0x7f3f8c1019b0 2024-04-01T01:33:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.689+0000 7f3f924ba640 1 --2- 172.21.15.70:0/801213178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f8c0ffe00 0x7f3f8c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.690+0000 7f3f92cbb640 1 --2- 172.21.15.70:0/801213178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f8c1019b0 0x7f3f8c19efb0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400eb10 tx=0x7f3f8400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.690+0000 7f3f837fe640 1 -- 172.21.15.70:0/801213178 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8400ce20 con 0x7f3f8c1019b0 2024-04-01T01:33:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.690+0000 7f3f837fe640 1 -- 172.21.15.70:0/801213178 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f84005020 con 0x7f3f8c1019b0 2024-04-01T01:33:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.690+0000 7f3f9471e640 1 -- 172.21.15.70:0/801213178 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f8c19f910 con 0x7f3f8c1019b0 2024-04-01T01:33:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.690+0000 7f3f837fe640 1 -- 172.21.15.70:0/801213178 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84010690 con 0x7f3f8c1019b0 2024-04-01T01:33:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.690+0000 7f3f9471e640 1 -- 172.21.15.70:0/801213178 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f8c19fe10 con 0x7f3f8c1019b0 2024-04-01T01:33:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.691+0000 7f3f9471e640 1 -- 172.21.15.70:0/801213178 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f8c1a00a0 con 0x7f3f8c1019b0 2024-04-01T01:33:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.692+0000 7f3f837fe640 1 -- 172.21.15.70:0/801213178 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f3f840051d0 con 0x7f3f8c1019b0 2024-04-01T01:33:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.692+0000 7f3f837fe640 1 --2- 172.21.15.70:0/801213178 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f780a24a0 0x7f3f780a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.693+0000 7f3f924ba640 1 --2- 172.21.15.70:0/801213178 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f780a24a0 0x7f3f780a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.693+0000 7f3f837fe640 1 -- 172.21.15.70:0/801213178 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3f840cf0a0 con 0x7f3f8c1019b0 2024-04-01T01:33:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.693+0000 7f3f924ba640 1 --2- 172.21.15.70:0/801213178 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f780a24a0 0x7f3f780a4960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3f88009830 tx=0x7f3f88034980 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:29.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.697+0000 7f3f837fe640 1 -- 172.21.15.70:0/801213178 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3f8409be90 con 0x7f3f8c1019b0 2024-04-01T01:33:29.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.874+0000 7f3f9471e640 1 -- 172.21.15.70:0/801213178 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f8c1a0310 con 0x7f3f8c1019b0 2024-04-01T01:33:29.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.875+0000 7f3f837fe640 1 -- 172.21.15.70:0/801213178 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24781 (secure 0 0 0) 0x7f3f8409b790 con 0x7f3f8c1019b0 2024-04-01T01:33:29.876 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:29.876 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":93,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:33:19.716245+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:33:29.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.877+0000 7f3f9471e640 1 -- 172.21.15.70:0/801213178 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f780a24a0 msgr2=0x7f3f780a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.877+0000 7f3f9471e640 1 --2- 172.21.15.70:0/801213178 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f780a24a0 0x7f3f780a4960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3f88009830 tx=0x7f3f88034980 comp rx=0 tx=0).stop 2024-04-01T01:33:29.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.877+0000 7f3f9471e640 1 -- 172.21.15.70:0/801213178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f8c1019b0 msgr2=0x7f3f8c19efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.877+0000 7f3f9471e640 1 --2- 172.21.15.70:0/801213178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f8c1019b0 0x7f3f8c19efb0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400eb10 tx=0x7f3f8400ef10 comp rx=0 tx=0).stop 2024-04-01T01:33:29.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.877+0000 7f3f9471e640 1 -- 172.21.15.70:0/801213178 shutdown_connections 2024-04-01T01:33:29.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.877+0000 7f3f9471e640 1 --2- 172.21.15.70:0/801213178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f8c1019b0 0x7f3f8c19efb0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.877+0000 7f3f9471e640 1 --2- 172.21.15.70:0/801213178 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f780a24a0 0x7f3f780a4960 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.877+0000 7f3f9471e640 1 --2- 172.21.15.70:0/801213178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f8c100ff0 0x7f3f8c197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.877+0000 7f3f9471e640 1 --2- 172.21.15.70:0/801213178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f8c0ffe00 0x7f3f8c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.877+0000 7f3f9471e640 1 -- 172.21.15.70:0/801213178 >> 172.21.15.70:0/801213178 conn(0x7f3f8c0fb370 msgr2=0x7f3f8c106580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:29.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.877+0000 7f3f9471e640 1 -- 172.21.15.70:0/801213178 shutdown_connections 2024-04-01T01:33:29.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.877+0000 7f3f9471e640 1 -- 172.21.15.70:0/801213178 wait complete. 2024-04-01T01:33:29.888 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:29 smithi070 ceph-mon[32179]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-04-01T01:33:29.888 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:29 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1983823009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:29.888 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:29 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1537160345' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:29.888 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:29 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2009846262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:29.890 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-01T01:31:15.732761+0000', 'modified': '2024-04-01T01:31:38.563210+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24464, 'mds_1': 24451, 'mds_2': 24418, 'mds_3': 24404, 'mds_4': 24379}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14616': {'gid': 14616, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6839/1626804615', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6838', 'nonce': 1626804615}, {'type': 'v1', 'addr': '172.21.15.70:6839', 'nonce': 1626804615}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14628': {'gid': 14628, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6841/2380158677', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6840', 'nonce': 2380158677}, {'type': 'v1', 'addr': '172.21.15.70:6841', 'nonce': 2380158677}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24371': {'gid': 24371, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.81:6833/1246539452', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6832', 'nonce': 1246539452}, {'type': 'v1', 'addr': '172.21.15.81:6833', 'nonce': 1246539452}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24379': {'gid': 24379, 'name': 'l', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.189:6833/2161209704', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6832', 'nonce': 2161209704}, {'type': 'v1', 'addr': '172.21.15.189:6833', 'nonce': 2161209704}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24404': {'gid': 24404, 'name': 'e', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.81:6835/970944957', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 970944957}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 970944957}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24418': {'gid': 24418, 'name': 'f', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.189:6835/3861912911', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 3861912911}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 3861912911}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24425': {'gid': 24425, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.81:6837/3527001697', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3527001697}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3527001697}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24427': {'gid': 24427, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6837/2296387110', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6836', 'nonce': 2296387110}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 2296387110}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24451': {'gid': 24451, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.189:6839/2833086571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6838', 'nonce': 2833086571}, {'type': 'v1', 'addr': '172.21.15.189:6839', 'nonce': 2833086571}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24464': {'gid': 24464, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6839/2805044611', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6838', 'nonce': 2805044611}, {'type': 'v1', 'addr': '172.21.15.81:6839', 'nonce': 2805044611}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24464, 'qdb_cluster': [24451, 24464, 24379, 24404, 24418]} 2024-04-01T01:33:29.890 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-01T01:33:29.890 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2024-04-01T01:33:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.992+0000 7faba8a8f640 1 -- 172.21.15.70:0/1950385345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faba4101860 msgr2=0x7faba4101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.992+0000 7faba8a8f640 1 --2- 172.21.15.70:0/1950385345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faba4101860 0x7faba4101ce0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fab98009900 tx=0x7fab9802bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.992+0000 7faba8a8f640 1 -- 172.21.15.70:0/1950385345 shutdown_connections 2024-04-01T01:33:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.992+0000 7faba8a8f640 1 --2- 172.21.15.70:0/1950385345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faba4102220 0x7faba4108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.992+0000 7faba8a8f640 1 --2- 172.21.15.70:0/1950385345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faba4101860 0x7faba4101ce0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.992+0000 7faba8a8f640 1 --2- 172.21.15.70:0/1950385345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faba4100670 0x7faba4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.992+0000 7faba8a8f640 1 -- 172.21.15.70:0/1950385345 >> 172.21.15.70:0/1950385345 conn(0x7faba40fbc00 msgr2=0x7faba40fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.992+0000 7faba8a8f640 1 -- 172.21.15.70:0/1950385345 shutdown_connections 2024-04-01T01:33:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.992+0000 7faba8a8f640 1 -- 172.21.15.70:0/1950385345 wait complete. 2024-04-01T01:33:29.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.993+0000 7faba8a8f640 1 Processor -- start 2024-04-01T01:33:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.993+0000 7faba8a8f640 1 -- start start 2024-04-01T01:33:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.993+0000 7faba8a8f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faba4100670 0x7faba41968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.993+0000 7faba8a8f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faba4101860 0x7faba4196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.993+0000 7faba8a8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faba4102220 0x7faba419de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.993+0000 7faba8a8f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faba410c1b0 con 0x7faba4102220 2024-04-01T01:33:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.993+0000 7faba8a8f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faba410c030 con 0x7faba4100670 2024-04-01T01:33:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.993+0000 7faba8a8f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faba410c330 con 0x7faba4101860 2024-04-01T01:33:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.993+0000 7faba2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faba4102220 0x7faba419de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.993+0000 7faba259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faba4100670 0x7faba41968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.993+0000 7faba2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faba4102220 0x7faba419de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32916/0 (socket says 172.21.15.70:32916) 2024-04-01T01:33:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.993+0000 7faba2d9d640 1 -- 172.21.15.70:0/1458390947 learned_addr learned my addr 172.21.15.70:0/1458390947 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:29.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.993+0000 7faba1d9b640 1 --2- 172.21.15.70:0/1458390947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faba4101860 0x7faba4196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.994+0000 7faba2d9d640 1 -- 172.21.15.70:0/1458390947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faba4101860 msgr2=0x7faba4196de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.994+0000 7faba2d9d640 1 --2- 172.21.15.70:0/1458390947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faba4101860 0x7faba4196de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.994+0000 7faba2d9d640 1 -- 172.21.15.70:0/1458390947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faba4100670 msgr2=0x7faba41968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.994+0000 7faba2d9d640 1 --2- 172.21.15.70:0/1458390947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faba4100670 0x7faba41968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.994+0000 7faba2d9d640 1 -- 172.21.15.70:0/1458390947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba419e4c0 con 0x7faba4102220 2024-04-01T01:33:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.994+0000 7faba259c640 1 --2- 172.21.15.70:0/1458390947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faba4100670 0x7faba41968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.994+0000 7faba2d9d640 1 --2- 172.21.15.70:0/1458390947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faba4102220 0x7faba419de50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00e970 tx=0x7fab9c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.994+0000 7fab937fe640 1 -- 172.21.15.70:0/1458390947 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c00ccc0 con 0x7faba4102220 2024-04-01T01:33:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.994+0000 7faba8a8f640 1 -- 172.21.15.70:0/1458390947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faba4069d80 con 0x7faba4102220 2024-04-01T01:33:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.994+0000 7fab937fe640 1 -- 172.21.15.70:0/1458390947 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab9c00ce60 con 0x7faba4102220 2024-04-01T01:33:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.994+0000 7fab937fe640 1 -- 172.21.15.70:0/1458390947 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c010430 con 0x7faba4102220 2024-04-01T01:33:29.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.994+0000 7faba8a8f640 1 -- 172.21.15.70:0/1458390947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faba406a2f0 con 0x7faba4102220 2024-04-01T01:33:30.004 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:29 smithi189 ceph-mon[35753]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-04-01T01:33:30.004 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:29 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1983823009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:30.004 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:29 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1537160345' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:30.004 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:29 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2009846262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:30.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.996+0000 7fab937fe640 1 -- 172.21.15.70:0/1458390947 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fab9c004f90 con 0x7faba4102220 2024-04-01T01:33:30.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7fabab66e740 1 Processor -- start 2024-04-01T01:33:30.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7fabab66e740 1 -- start start 2024-04-01T01:33:30.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7fabab66e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363ef2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7fabab66e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef3bf0 0x558363ef4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7fabab66e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef45b0 0x558363efae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7fabab66e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558363e52af0 con 0x558363ef45b0 2024-04-01T01:33:30.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7fabab66e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x558363dfb9d0 con 0x558363ef3bf0 2024-04-01T01:33:30.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7fabab66e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558363e4eb90 con 0x558363ef2a20 2024-04-01T01:33:30.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7fab937fe640 1 --2- 172.21.15.70:0/1458390947 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fab8c0a24a0 0x7fab8c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7faba1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef45b0 0x558363efae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7faba1d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363ef2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7faba259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef3bf0 0x558363ef4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7faba2d9d640 1 --2- 172.21.15.70:0/1458390947 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fab8c0a24a0 0x7fab8c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7faba1d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363ef2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:48692/0 (socket says 172.21.15.70:48692) 2024-04-01T01:33:30.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7faba1d9b640 1 -- 172.21.15.70:0/4293437095 learned_addr learned my addr 172.21.15.70:0/4293437095 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7faba259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef3bf0 0x558363ef4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:58822/0 (socket says 172.21.15.70:58822) 2024-04-01T01:33:30.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7fab937fe640 1 -- 172.21.15.70:0/1458390947 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fab9c0ceea0 con 0x7faba4102220 2024-04-01T01:33:30.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7faba259c640 1 -- 172.21.15.70:0/4293437095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 msgr2=0x558363ef2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7faba259c640 1 --2- 172.21.15.70:0/4293437095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363ef2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7faba259c640 1 -- 172.21.15.70:0/4293437095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef45b0 msgr2=0x558363efae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7faba259c640 1 --2- 172.21.15.70:0/4293437095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef45b0 0x558363efae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7faba259c640 1 -- 172.21.15.70:0/4293437095 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558363b672d0 con 0x558363ef3bf0 2024-04-01T01:33:30.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.997+0000 7faba1d9b640 1 --2- 172.21.15.70:0/4293437095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363ef2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:30.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.998+0000 7faba2d9d640 1 --2- 172.21.15.70:0/1458390947 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fab8c0a24a0 0x7fab8c0a4960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fab9c097f60 tx=0x7fab9c086040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.998+0000 7faba259c640 1 --2- 172.21.15.70:0/4293437095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef3bf0 0x558363ef4070 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faba41016c0 tx=0x7fab94011c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4430d3aeb270c87c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.998+0000 7fab90ff9640 1 -- 172.21.15.70:0/4293437095 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94003c30 con 0x558363ef3bf0 2024-04-01T01:33:30.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.998+0000 7fab90ff9640 1 -- 172.21.15.70:0/4293437095 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab9400fe20 con 0x558363ef3bf0 2024-04-01T01:33:30.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.998+0000 7fab90ff9640 1 -- 172.21.15.70:0/4293437095 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94004810 con 0x558363ef3bf0 2024-04-01T01:33:30.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.998+0000 7fabab66e740 1 -- 172.21.15.70:0/4293437095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef3bf0 msgr2=0x558363ef4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 1 --2- 172.21.15.70:0/4293437095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef3bf0 0x558363ef4070 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faba41016c0 tx=0x7fab94011c70 comp rx=0 tx=0).stop 2024-04-01T01:33:30.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 1 -- 172.21.15.70:0/4293437095 shutdown_connections 2024-04-01T01:33:30.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 1 --2- 172.21.15.70:0/4293437095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef45b0 0x558363efae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 1 --2- 172.21.15.70:0/4293437095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef3bf0 0x558363ef4070 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 1 --2- 172.21.15.70:0/4293437095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363ef2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 1 -- 172.21.15.70:0/4293437095 >> 172.21.15.70:0/4293437095 conn(0x558363ee9a00 msgr2=0x558363ee9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 1 -- 172.21.15.70:0/4293437095 shutdown_connections 2024-04-01T01:33:30.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 1 -- 172.21.15.70:0/4293437095 wait complete. 2024-04-01T01:33:30.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 1 -- start start 2024-04-01T01:33:30.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 1 Processor -- start 2024-04-01T01:33:30.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363efccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef3bf0 0x558363efd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef45b0 0x558363f04270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558363efb5c0 con 0x558363ef3bf0 2024-04-01T01:33:30.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x558363e0ef70 con 0x558363ef45b0 2024-04-01T01:33:30.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558363efb740 con 0x558363ef2a20 2024-04-01T01:33:30.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7fabab66e740 10 client.0 _ll_register_callbacks cb 0x558363b66ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:30.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7faba259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef3bf0 0x558363efd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7faba259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef3bf0 0x558363efd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32946/0 (socket says 172.21.15.70:32946) 2024-04-01T01:33:30.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7faba259c640 1 -- 172.21.15.70:0/338234365 learned_addr learned my addr 172.21.15.70:0/338234365 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7faba1d9b640 1 --2- 172.21.15.70:0/338234365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef45b0 0x558363f04270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:29.999+0000 7faba1d9b640 1 --2- 172.21.15.70:0/338234365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363efccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7faba259c640 1 -- 172.21.15.70:0/338234365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 msgr2=0x558363efccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7faba259c640 1 --2- 172.21.15.70:0/338234365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363efccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7faba259c640 1 -- 172.21.15.70:0/338234365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef45b0 msgr2=0x558363f04270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7faba259c640 1 --2- 172.21.15.70:0/338234365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef45b0 0x558363f04270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7faba259c640 1 -- 172.21.15.70:0/338234365 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558363c316b0 con 0x558363ef3bf0 2024-04-01T01:33:30.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7faba1d9b640 1 --2- 172.21.15.70:0/338234365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef45b0 0x558363f04270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:30.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7faba259c640 1 --2- 172.21.15.70:0/338234365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef3bf0 0x558363efd230 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fab94004610 tx=0x7fab94004640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d7b040718a07791 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7fab89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:30.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7fab89ffb640 1 -- 172.21.15.70:0/338234365 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94027020 con 0x558363ef3bf0 2024-04-01T01:33:30.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7fab89ffb640 1 -- 172.21.15.70:0/338234365 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fab94011240 con 0x558363ef3bf0 2024-04-01T01:33:30.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7fabab66e740 1 -- 172.21.15.70:0/338234365 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab9400fe20 con 0x558363ef3bf0 2024-04-01T01:33:30.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7fab89ffb640 1 -- 172.21.15.70:0/338234365 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fab94032050 con 0x558363ef3bf0 2024-04-01T01:33:30.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7fabab66e740 10 client.15423 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:30.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7fabab66e740 1 -- 172.21.15.70:0/338234365 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558363f07250 con 0x558363ef3bf0 2024-04-01T01:33:30.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7fab89ffb640 1 -- 172.21.15.70:0/338234365 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab9400fe20 con 0x558363ef3bf0 2024-04-01T01:33:30.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7fab89ffb640 1 -- 172.21.15.70:0/338234365 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fab94018070 con 0x558363ef3bf0 2024-04-01T01:33:30.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.000+0000 7fabab66e740 10 client.15423 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:30.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.001+0000 7fabab66e740 10 client.15423 resolve_mds: resolved name 'l' to daemon mds.l 2024-04-01T01:33:30.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.001+0000 7fabab66e740 20 client.15423 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:30.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.001+0000 7fabab66e740 1 --2- 172.21.15.70:0/338234365 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x558363f09710 0x558363f0ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.001+0000 7fabab66e740 4 client.15423 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:33:30.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.001+0000 7fabab66e740 1 -- 172.21.15.70:0/338234365 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558363e60030 con 0x558363f09710 2024-04-01T01:33:30.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.001+0000 7faba1d9b640 1 --2- 172.21.15.70:0/338234365 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x558363f09710 0x558363f0ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.001+0000 7faba1d9b640 1 --2- 172.21.15.70:0/338234365 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x558363f09710 0x558363f0ab80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e6a8420946717055 server_cookie=1cd71d6e6a3e6cad in_seq=0 out_seq=0 2024-04-01T01:33:30.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.001+0000 7fab89ffb640 10 client.15423 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:33:30.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.002+0000 7fab89ffb640 1 -- 172.21.15.70:0/338234365 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558363e60030 con 0x558363f09710 2024-04-01T01:33:30.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.002+0000 7fab89ffb640 10 client.15423 handle_command_reply: tid=0 2024-04-01T01:33:30.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.002+0000 7fabab66e740 1 client.15423 shutdown 2024-04-01T01:33:30.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.002+0000 7fabab66e740 1 -- 172.21.15.70:0/338234365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef3bf0 msgr2=0x558363efd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.002+0000 7fabab66e740 1 --2- 172.21.15.70:0/338234365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef3bf0 0x558363efd230 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fab94004610 tx=0x7fab94004640 comp rx=0 tx=0).stop 2024-04-01T01:33:30.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.002+0000 7fabab66e740 1 -- 172.21.15.70:0/338234365 shutdown_connections 2024-04-01T01:33:30.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.002+0000 7fabab66e740 1 --2- 172.21.15.70:0/338234365 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x558363f09710 0x558363f0ab80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.002+0000 7fabab66e740 1 --2- 172.21.15.70:0/338234365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef45b0 0x558363f04270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.002+0000 7fabab66e740 1 --2- 172.21.15.70:0/338234365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef3bf0 0x558363efd230 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.002+0000 7fab89ffb640 0 client.15423 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:33:30.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.002+0000 7fabab66e740 1 --2- 172.21.15.70:0/338234365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363efccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.002+0000 7fabab66e740 1 -- 172.21.15.70:0/338234365 >> 172.21.15.70:0/338234365 conn(0x558363ee9a00 msgr2=0x558363ee8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.002+0000 7fabab66e740 1 -- 172.21.15.70:0/338234365 shutdown_connections 2024-04-01T01:33:30.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.002+0000 7fabab66e740 1 -- 172.21.15.70:0/338234365 wait complete. 2024-04-01T01:33:30.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.002+0000 7fabab66e740 20 client.15423 trim_cache size 0 max 16384 2024-04-01T01:33:30.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7fabab66e740 1 Processor -- start 2024-04-01T01:33:30.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7fabab66e740 1 -- start start 2024-04-01T01:33:30.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7fabab66e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363efaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7fabab66e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef3bf0 0x558363f04230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7fabab66e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef45b0 0x558363f08650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7fabab66e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558363f08ec0 con 0x558363ef45b0 2024-04-01T01:33:30.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7fabab66e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x558363f09590 con 0x558363ef3bf0 2024-04-01T01:33:30.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7fabab66e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558363f09c60 con 0x558363ef2a20 2024-04-01T01:33:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7faba1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef45b0 0x558363f08650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7faba1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef45b0 0x558363f08650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32948/0 (socket says 172.21.15.70:32948) 2024-04-01T01:33:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7faba1d9b640 1 -- 172.21.15.70:0/3663912190 learned_addr learned my addr 172.21.15.70:0/3663912190 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7faba1d9b640 1 --2- 172.21.15.70:0/3663912190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363efaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7faba259c640 1 --2- 172.21.15.70:0/3663912190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef3bf0 0x558363f04230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7faba1d9b640 1 -- 172.21.15.70:0/3663912190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 msgr2=0x558363efaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7faba1d9b640 1 --2- 172.21.15.70:0/3663912190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363efaad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7faba1d9b640 1 -- 172.21.15.70:0/3663912190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef3bf0 msgr2=0x558363f04230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7faba1d9b640 1 --2- 172.21.15.70:0/3663912190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef3bf0 0x558363f04230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.020+0000 7faba1d9b640 1 -- 172.21.15.70:0/3663912190 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558363eea770 con 0x558363ef45b0 2024-04-01T01:33:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.021+0000 7faba1d9b640 1 --2- 172.21.15.70:0/3663912190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef45b0 0x558363f08650 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fab98036040 tx=0x7fab9803e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719cb84309994926 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.021+0000 7fab8bfff640 1 -- 172.21.15.70:0/3663912190 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9804e680 con 0x558363ef45b0 2024-04-01T01:33:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.021+0000 7fab8bfff640 1 -- 172.21.15.70:0/3663912190 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab98003aa0 con 0x558363ef45b0 2024-04-01T01:33:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.021+0000 7fab8bfff640 1 -- 172.21.15.70:0/3663912190 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9803c650 con 0x558363ef45b0 2024-04-01T01:33:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.021+0000 7fabab66e740 1 -- 172.21.15.70:0/3663912190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef45b0 msgr2=0x558363f08650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.021+0000 7fabab66e740 1 --2- 172.21.15.70:0/3663912190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef45b0 0x558363f08650 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fab98036040 tx=0x7fab9803e040 comp rx=0 tx=0).stop 2024-04-01T01:33:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.021+0000 7fabab66e740 1 -- 172.21.15.70:0/3663912190 shutdown_connections 2024-04-01T01:33:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.021+0000 7fabab66e740 1 --2- 172.21.15.70:0/3663912190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef45b0 0x558363f08650 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.021+0000 7fabab66e740 1 --2- 172.21.15.70:0/3663912190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef3bf0 0x558363f04230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.021+0000 7fabab66e740 1 --2- 172.21.15.70:0/3663912190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363efaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.021+0000 7fabab66e740 1 -- 172.21.15.70:0/3663912190 >> 172.21.15.70:0/3663912190 conn(0x558363ee9a00 msgr2=0x558363ee9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.021+0000 7fabab66e740 1 -- 172.21.15.70:0/3663912190 shutdown_connections 2024-04-01T01:33:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.021+0000 7fabab66e740 1 -- 172.21.15.70:0/3663912190 wait complete. 2024-04-01T01:33:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.021+0000 7fabab66e740 1 -- start start 2024-04-01T01:33:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7fabab66e740 1 Processor -- start 2024-04-01T01:33:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7fabab66e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363eff870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7fabab66e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef3bf0 0x558363effdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7fabab66e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef45b0 0x558363f02d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7fabab66e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558363f0aa00 con 0x558363ef3bf0 2024-04-01T01:33:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7fabab66e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x558363f0a330 con 0x558363ef45b0 2024-04-01T01:33:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7fabab66e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558363f0b0d0 con 0x558363ef2a20 2024-04-01T01:33:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7fabab66e740 10 client.0 _ll_register_callbacks cb 0x7fab98009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7faba259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef3bf0 0x558363effdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7faba259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef3bf0 0x558363effdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32958/0 (socket says 172.21.15.70:32958) 2024-04-01T01:33:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7faba259c640 1 -- 172.21.15.70:0/1614898969 learned_addr learned my addr 172.21.15.70:0/1614898969 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7faba1d9b640 1 --2- 172.21.15.70:0/1614898969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363eff870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7faba1d9b640 1 --2- 172.21.15.70:0/1614898969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef45b0 0x558363f02d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7faba259c640 1 -- 172.21.15.70:0/1614898969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 msgr2=0x558363eff870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7faba259c640 1 --2- 172.21.15.70:0/1614898969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363eff870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7faba259c640 1 -- 172.21.15.70:0/1614898969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef45b0 msgr2=0x558363f02d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7faba259c640 1 --2- 172.21.15.70:0/1614898969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef45b0 0x558363f02d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7faba259c640 1 -- 172.21.15.70:0/1614898969 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558363f03370 con 0x558363ef3bf0 2024-04-01T01:33:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7faba1d9b640 1 --2- 172.21.15.70:0/1614898969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef45b0 0x558363f02d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7faba259c640 1 --2- 172.21.15.70:0/1614898969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef3bf0 0x558363effdb0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fab94004ec0 tx=0x7fab94004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fbd74d79f8eb0a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7fab89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7fab89ffb640 1 -- 172.21.15.70:0/1614898969 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9403b060 con 0x558363ef3bf0 2024-04-01T01:33:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7fab89ffb640 1 -- 172.21.15.70:0/1614898969 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fab94004f40 con 0x558363ef3bf0 2024-04-01T01:33:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.022+0000 7fabab66e740 1 -- 172.21.15.70:0/1614898969 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab98003aa0 con 0x558363ef3bf0 2024-04-01T01:33:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.023+0000 7fab89ffb640 1 -- 172.21.15.70:0/1614898969 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fab94042050 con 0x558363ef3bf0 2024-04-01T01:33:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.023+0000 7fabab66e740 10 client.15429 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.023+0000 7fabab66e740 1 -- 172.21.15.70:0/1614898969 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558363f6f480 con 0x558363ef3bf0 2024-04-01T01:33:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.023+0000 7fab89ffb640 1 -- 172.21.15.70:0/1614898969 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab98003aa0 con 0x558363ef3bf0 2024-04-01T01:33:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.023+0000 7fab89ffb640 1 -- 172.21.15.70:0/1614898969 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fab94018040 con 0x558363ef3bf0 2024-04-01T01:33:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.023+0000 7fabab66e740 10 client.15429 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.023+0000 7fabab66e740 10 client.15429 resolve_mds: resolved name 'l' to daemon mds.l 2024-04-01T01:33:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.023+0000 7fabab66e740 20 client.15429 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.023+0000 7fabab66e740 1 --2- 172.21.15.70:0/1614898969 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x558363f73ef0 0x558363f72c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.023+0000 7fabab66e740 4 client.15429 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-01T01:33:30.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.023+0000 7fabab66e740 1 -- 172.21.15.70:0/1614898969 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "status"}) -- 0x558363e5a380 con 0x558363f73ef0 2024-04-01T01:33:30.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.023+0000 7faba1d9b640 1 --2- 172.21.15.70:0/1614898969 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x558363f73ef0 0x558363f72c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7fab89ffb640 10 client.15429 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:33:30.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7faba1d9b640 1 --2- 172.21.15.70:0/1614898969 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x558363f73ef0 0x558363f72c00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9181ba2af1c6323a server_cookie=fa423fa0bdb99a61 in_seq=0 out_seq=0 2024-04-01T01:33:30.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7fab89ffb640 1 -- 172.21.15.70:0/1614898969 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x558363e5a380 con 0x558363f73ef0 2024-04-01T01:33:30.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7fab89ffb640 10 client.15429 handle_command_reply: tid=0 2024-04-01T01:33:30.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7fabab66e740 1 client.15429 shutdown 2024-04-01T01:33:30.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7fabab66e740 1 -- 172.21.15.70:0/1614898969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef3bf0 msgr2=0x558363effdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7fabab66e740 1 --2- 172.21.15.70:0/1614898969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef3bf0 0x558363effdb0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fab94004ec0 tx=0x7fab94004590 comp rx=0 tx=0).stop 2024-04-01T01:33:30.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7fabab66e740 1 -- 172.21.15.70:0/1614898969 shutdown_connections 2024-04-01T01:33:30.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7fabab66e740 1 --2- 172.21.15.70:0/1614898969 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x558363f73ef0 0x558363f72c00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7fabab66e740 1 --2- 172.21.15.70:0/1614898969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558363ef45b0 0x558363f02d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7fabab66e740 1 --2- 172.21.15.70:0/1614898969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558363ef3bf0 0x558363effdb0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7fab89ffb640 0 client.15429 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:33:30.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7fabab66e740 1 --2- 172.21.15.70:0/1614898969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558363ef2a20 0x558363eff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7fabab66e740 1 -- 172.21.15.70:0/1614898969 >> 172.21.15.70:0/1614898969 conn(0x558363ee9a00 msgr2=0x558363f08650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7fabab66e740 1 -- 172.21.15.70:0/1614898969 shutdown_connections 2024-04-01T01:33:30.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7fabab66e740 1 -- 172.21.15.70:0/1614898969 wait complete. 2024-04-01T01:33:30.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7fabab66e740 20 client.15429 trim_cache size 0 max 16384 2024-04-01T01:33:30.032 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:33:30.033 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "36577394-efc6-11ee-b647-cb9ed24678a4", 2024-04-01T01:33:30.033 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 4, 2024-04-01T01:33:30.033 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24379, 2024-04-01T01:33:30.033 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-01T01:33:30.033 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-01T01:33:30.033 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-01T01:33:30.033 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 112.46330026699999, 2024-04-01T01:33:30.033 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-04-01T01:33:30.033 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 93, 2024-04-01T01:33:30.033 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 89, 2024-04-01T01:33:30.033 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 170.161899632 2024-04-01T01:33:30.033 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:33:30.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7faba8a8f640 1 -- 172.21.15.70:0/1458390947 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fab8c0a24a0 msgr2=0x7fab8c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7faba8a8f640 1 --2- 172.21.15.70:0/1458390947 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fab8c0a24a0 0x7fab8c0a4960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fab9c097f60 tx=0x7fab9c086040 comp rx=0 tx=0).stop 2024-04-01T01:33:30.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7faba8a8f640 1 -- 172.21.15.70:0/1458390947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faba4102220 msgr2=0x7faba419de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7faba8a8f640 1 --2- 172.21.15.70:0/1458390947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faba4102220 0x7faba419de50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00e970 tx=0x7fab9c00ed70 comp rx=0 tx=0).stop 2024-04-01T01:33:30.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7faba8a8f640 1 -- 172.21.15.70:0/1458390947 shutdown_connections 2024-04-01T01:33:30.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7faba8a8f640 1 --2- 172.21.15.70:0/1458390947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faba4102220 0x7faba419de50 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7faba8a8f640 1 --2- 172.21.15.70:0/1458390947 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fab8c0a24a0 0x7fab8c0a4960 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7faba8a8f640 1 --2- 172.21.15.70:0/1458390947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faba4101860 0x7faba4196de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.024+0000 7faba8a8f640 1 --2- 172.21.15.70:0/1458390947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faba4100670 0x7faba41968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.025+0000 7faba8a8f640 1 -- 172.21.15.70:0/1458390947 >> 172.21.15.70:0/1458390947 conn(0x7faba40fbc00 msgr2=0x7faba4104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.025+0000 7faba8a8f640 1 -- 172.21.15.70:0/1458390947 shutdown_connections 2024-04-01T01:33:30.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.025+0000 7faba8a8f640 1 -- 172.21.15.70:0/1458390947 wait complete. 2024-04-01T01:33:30.036 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-04-01T01:33:30.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:29 smithi081 ceph-mon[35142]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-04-01T01:33:30.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:29 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1983823009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:30.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:29 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1537160345' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:30.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:29 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2009846262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:30.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.139+0000 7f867000e640 1 -- 172.21.15.70:0/108342514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8668103310 msgr2=0x7f8668109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.139+0000 7f867000e640 1 --2- 172.21.15.70:0/108342514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8668103310 0x7f8668109ba0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8660009ca0 tx=0x7f866002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.139+0000 7f867000e640 1 -- 172.21.15.70:0/108342514 shutdown_connections 2024-04-01T01:33:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.139+0000 7f867000e640 1 --2- 172.21.15.70:0/108342514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8668103310 0x7f8668109ba0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.139+0000 7f867000e640 1 --2- 172.21.15.70:0/108342514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8668102950 0x7f8668102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.139+0000 7f867000e640 1 --2- 172.21.15.70:0/108342514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8668101760 0x7f8668101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.139+0000 7f867000e640 1 -- 172.21.15.70:0/108342514 >> 172.21.15.70:0/108342514 conn(0x7f86680fccf0 msgr2=0x7f86680ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.139+0000 7f867000e640 1 -- 172.21.15.70:0/108342514 shutdown_connections 2024-04-01T01:33:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.139+0000 7f867000e640 1 -- 172.21.15.70:0/108342514 wait complete. 2024-04-01T01:33:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f867000e640 1 Processor -- start 2024-04-01T01:33:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f867000e640 1 -- start start 2024-04-01T01:33:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f867000e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8668101760 0x7f8668197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f867000e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8668102950 0x7f8668197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f867000e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8668103310 0x7f866819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f867000e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f866810d270 con 0x7f8668103310 2024-04-01T01:33:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f867000e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f866810d0f0 con 0x7f8668102950 2024-04-01T01:33:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f867000e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f866810d3f0 con 0x7f8668101760 2024-04-01T01:33:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f866e5ab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8668103310 0x7f866819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f866e5ab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8668103310 0x7f866819eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32988/0 (socket says 172.21.15.70:32988) 2024-04-01T01:33:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f866ddaa640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8668101760 0x7f8668197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f866d5a9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8668102950 0x7f8668197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f866e5ab640 1 -- 172.21.15.70:0/3575040508 learned_addr learned my addr 172.21.15.70:0/3575040508 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f866e5ab640 1 -- 172.21.15.70:0/3575040508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8668101760 msgr2=0x7f8668197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f866e5ab640 1 --2- 172.21.15.70:0/3575040508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8668101760 0x7f8668197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f866e5ab640 1 -- 172.21.15.70:0/3575040508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8668102950 msgr2=0x7f8668197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f866e5ab640 1 --2- 172.21.15.70:0/3575040508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8668102950 0x7f8668197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f866e5ab640 1 -- 172.21.15.70:0/3575040508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f866819f550 con 0x7f8668103310 2024-04-01T01:33:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f866d5a9640 1 --2- 172.21.15.70:0/3575040508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8668102950 0x7f8668197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:30.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f866e5ab640 1 --2- 172.21.15.70:0/3575040508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8668103310 0x7f866819eee0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f866002c700 tx=0x7f86600328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.140+0000 7f865affd640 1 -- 172.21.15.70:0/3575040508 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8660041020 con 0x7f8668103310 2024-04-01T01:33:30.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.141+0000 7f865affd640 1 -- 172.21.15.70:0/3575040508 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8660003e20 con 0x7f8668103310 2024-04-01T01:33:30.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.141+0000 7f867000e640 1 -- 172.21.15.70:0/3575040508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f866819f7e0 con 0x7f8668103310 2024-04-01T01:33:30.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.141+0000 7f865affd640 1 -- 172.21.15.70:0/3575040508 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8660039040 con 0x7f8668103310 2024-04-01T01:33:30.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.141+0000 7f867000e640 1 -- 172.21.15.70:0/3575040508 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f866819fcd0 con 0x7f8668103310 2024-04-01T01:33:30.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.143+0000 7f865affd640 1 -- 172.21.15.70:0/3575040508 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f8660044020 con 0x7f8668103310 2024-04-01T01:33:30.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.143+0000 7f8672bf2740 1 Processor -- start 2024-04-01T01:33:30.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.143+0000 7f8672bf2740 1 -- start start 2024-04-01T01:33:30.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.143+0000 7f865affd640 1 --2- 172.21.15.70:0/3575040508 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f86540a24a0 0x7f86540a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.143+0000 7f8672bf2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce25610 0x56104ce25a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.143+0000 7f8672bf2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 0x56104ce26c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.143+0000 7f8672bf2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce271a0 0x56104ce2da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.143+0000 7f8672bf2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56104cd89af0 con 0x56104ce271a0 2024-04-01T01:33:30.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.143+0000 7f8672bf2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56104cd329d0 con 0x56104ce267e0 2024-04-01T01:33:30.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.143+0000 7f8672bf2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56104cd85b90 con 0x56104ce25610 2024-04-01T01:33:30.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.143+0000 7f866d5a9640 1 --2- 172.21.15.70:0/3575040508 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f86540a24a0 0x7f86540a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.144+0000 7f865affd640 1 -- 172.21.15.70:0/3575040508 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f86600f0c80 con 0x7f8668103310 2024-04-01T01:33:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.144+0000 7f866e5ab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce271a0 0x56104ce2da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.144+0000 7f866e5ab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce271a0 0x56104ce2da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:32996/0 (socket says 172.21.15.70:32996) 2024-04-01T01:33:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.144+0000 7f866e5ab640 1 -- 172.21.15.70:0/1102413138 learned_addr learned my addr 172.21.15.70:0/1102413138 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.144+0000 7f866d5a9640 1 --2- 172.21.15.70:0/1102413138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 0x56104ce26c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.144+0000 7f866ddaa640 1 --2- 172.21.15.70:0/1102413138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce25610 0x56104ce25a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.144+0000 7f866e5ab640 1 -- 172.21.15.70:0/1102413138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce25610 msgr2=0x56104ce25a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.144+0000 7f866e5ab640 1 --2- 172.21.15.70:0/1102413138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce25610 0x56104ce25a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.144+0000 7f866e5ab640 1 -- 172.21.15.70:0/1102413138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 msgr2=0x56104ce26c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.144+0000 7f866e5ab640 1 --2- 172.21.15.70:0/1102413138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 0x56104ce26c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.144+0000 7f866e5ab640 1 -- 172.21.15.70:0/1102413138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56104ca9e2d0 con 0x56104ce271a0 2024-04-01T01:33:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.144+0000 7f866d5a9640 1 --2- 172.21.15.70:0/3575040508 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f86540a24a0 0x7f86540a4960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8668198e40 tx=0x7f865c009c00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.144+0000 7f866e5ab640 1 --2- 172.21.15.70:0/1102413138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce271a0 0x56104ce2da30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f86600b9d40 tx=0x7f86600bd200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b42d1a6f29a6312 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.144+0000 7f8653fff640 1 -- 172.21.15.70:0/1102413138 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86600a0e60 con 0x56104ce271a0 2024-04-01T01:33:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.144+0000 7f8653fff640 1 -- 172.21.15.70:0/1102413138 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86600bda70 con 0x56104ce271a0 2024-04-01T01:33:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.144+0000 7f8653fff640 1 -- 172.21.15.70:0/1102413138 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86600bb670 con 0x56104ce271a0 2024-04-01T01:33:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.145+0000 7f8672bf2740 1 -- 172.21.15.70:0/1102413138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce271a0 msgr2=0x56104ce2da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.145+0000 7f8672bf2740 1 --2- 172.21.15.70:0/1102413138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce271a0 0x56104ce2da30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f86600b9d40 tx=0x7f86600bd200 comp rx=0 tx=0).stop 2024-04-01T01:33:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.145+0000 7f8672bf2740 1 -- 172.21.15.70:0/1102413138 shutdown_connections 2024-04-01T01:33:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.145+0000 7f8672bf2740 1 --2- 172.21.15.70:0/1102413138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce271a0 0x56104ce2da30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.145+0000 7f8672bf2740 1 --2- 172.21.15.70:0/1102413138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 0x56104ce26c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.145+0000 7f8672bf2740 1 --2- 172.21.15.70:0/1102413138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce25610 0x56104ce25a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.145+0000 7f8672bf2740 1 -- 172.21.15.70:0/1102413138 >> 172.21.15.70:0/1102413138 conn(0x56104ce20a00 msgr2=0x56104ce20e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.145+0000 7f8672bf2740 1 -- 172.21.15.70:0/1102413138 shutdown_connections 2024-04-01T01:33:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.145+0000 7f8672bf2740 1 -- 172.21.15.70:0/1102413138 wait complete. 2024-04-01T01:33:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.145+0000 7f8672bf2740 1 -- start start 2024-04-01T01:33:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.145+0000 7f8672bf2740 1 Processor -- start 2024-04-01T01:33:30.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f8672bf2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce25610 0x56104ce2f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f8672bf2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce267e0 0x56104ce2fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f8672bf2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce271a0 0x56104ce36b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f8672bf2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56104ce2df70 con 0x56104ce267e0 2024-04-01T01:33:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f8672bf2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56104cd45f70 con 0x56104ce271a0 2024-04-01T01:33:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f8672bf2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56104ce2e0f0 con 0x56104ce25610 2024-04-01T01:33:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f8672bf2740 10 client.0 _ll_register_callbacks cb 0x56104ca9dac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f866d5a9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce267e0 0x56104ce2fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f866d5a9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce267e0 0x56104ce2fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33010/0 (socket says 172.21.15.70:33010) 2024-04-01T01:33:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f866d5a9640 1 -- 172.21.15.70:0/3810654780 learned_addr learned my addr 172.21.15.70:0/3810654780 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f866e5ab640 1 --2- 172.21.15.70:0/3810654780 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce271a0 0x56104ce36b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f866ddaa640 1 --2- 172.21.15.70:0/3810654780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce25610 0x56104ce2f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f866d5a9640 1 -- 172.21.15.70:0/3810654780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce25610 msgr2=0x56104ce2f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f866d5a9640 1 --2- 172.21.15.70:0/3810654780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce25610 0x56104ce2f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f866d5a9640 1 -- 172.21.15.70:0/3810654780 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce271a0 msgr2=0x56104ce36b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f866d5a9640 1 --2- 172.21.15.70:0/3810654780 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce271a0 0x56104ce36b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f866d5a9640 1 -- 172.21.15.70:0/3810654780 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56104cb686b0 con 0x56104ce267e0 2024-04-01T01:33:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f866e5ab640 1 --2- 172.21.15.70:0/3810654780 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce271a0 0x56104ce36b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f866d5a9640 1 --2- 172.21.15.70:0/3810654780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce267e0 0x56104ce2fb20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f865c00fcf0 tx=0x7f865c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1801ec22ec648565 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f86517fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f86517fa640 1 -- 172.21.15.70:0/3810654780 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c026070 con 0x56104ce267e0 2024-04-01T01:33:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f86517fa640 1 -- 172.21.15.70:0/3810654780 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f865c02a600 con 0x56104ce267e0 2024-04-01T01:33:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.146+0000 7f8672bf2740 1 -- 172.21.15.70:0/3810654780 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86600bda70 con 0x56104ce267e0 2024-04-01T01:33:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.147+0000 7f86517fa640 1 -- 172.21.15.70:0/3810654780 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f865c035050 con 0x56104ce267e0 2024-04-01T01:33:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.147+0000 7f8672bf2740 10 client.15441 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.147+0000 7f8672bf2740 1 -- 172.21.15.70:0/3810654780 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56104ce39bb0 con 0x56104ce267e0 2024-04-01T01:33:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.147+0000 7f86517fa640 1 -- 172.21.15.70:0/3810654780 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86600bda70 con 0x56104ce267e0 2024-04-01T01:33:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.147+0000 7f86517fa640 1 -- 172.21.15.70:0/3810654780 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f865c01f070 con 0x56104ce267e0 2024-04-01T01:33:30.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.147+0000 7f8672bf2740 10 client.15441 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.147+0000 7f8672bf2740 10 client.15441 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-01T01:33:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.147+0000 7f8672bf2740 20 client.15441 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.147+0000 7f8672bf2740 1 --2- 172.21.15.70:0/3810654780 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56104ce3c2c0 0x56104ce3d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.147+0000 7f8672bf2740 4 client.15441 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:33:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.147+0000 7f8672bf2740 1 -- 172.21.15.70:0/3810654780 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56104cd97030 con 0x56104ce3c2c0 2024-04-01T01:33:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.147+0000 7f866ddaa640 1 --2- 172.21.15.70:0/3810654780 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56104ce3c2c0 0x56104ce3d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.148+0000 7f866ddaa640 1 --2- 172.21.15.70:0/3810654780 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56104ce3c2c0 0x56104ce3d730 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=828fa48ebebcce28 server_cookie=8082c8ae12815ba9 in_seq=0 out_seq=0 2024-04-01T01:33:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.148+0000 7f86517fa640 10 client.15441 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:33:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.148+0000 7f86517fa640 1 -- 172.21.15.70:0/3810654780 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56104cd97030 con 0x56104ce3c2c0 2024-04-01T01:33:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.148+0000 7f86517fa640 10 client.15441 handle_command_reply: tid=0 2024-04-01T01:33:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.148+0000 7f8672bf2740 1 client.15441 shutdown 2024-04-01T01:33:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.148+0000 7f8672bf2740 1 -- 172.21.15.70:0/3810654780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce267e0 msgr2=0x56104ce2fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.148+0000 7f8672bf2740 1 --2- 172.21.15.70:0/3810654780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce267e0 0x56104ce2fb20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f865c00fcf0 tx=0x7f865c004730 comp rx=0 tx=0).stop 2024-04-01T01:33:30.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.148+0000 7f8672bf2740 1 -- 172.21.15.70:0/3810654780 shutdown_connections 2024-04-01T01:33:30.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.148+0000 7f8672bf2740 1 --2- 172.21.15.70:0/3810654780 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56104ce3c2c0 0x56104ce3d730 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.148+0000 7f8672bf2740 1 --2- 172.21.15.70:0/3810654780 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce271a0 0x56104ce36b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.148+0000 7f86517fa640 0 client.15441 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:33:30.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.148+0000 7f8672bf2740 1 --2- 172.21.15.70:0/3810654780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce267e0 0x56104ce2fb20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.148+0000 7f8672bf2740 1 --2- 172.21.15.70:0/3810654780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce25610 0x56104ce2f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.148+0000 7f8672bf2740 1 -- 172.21.15.70:0/3810654780 >> 172.21.15.70:0/3810654780 conn(0x56104ce20a00 msgr2=0x56104ce1fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.149+0000 7f8672bf2740 1 -- 172.21.15.70:0/3810654780 shutdown_connections 2024-04-01T01:33:30.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.149+0000 7f8672bf2740 1 -- 172.21.15.70:0/3810654780 wait complete. 2024-04-01T01:33:30.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.149+0000 7f8672bf2740 20 client.15441 trim_cache size 0 max 16384 2024-04-01T01:33:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.166+0000 7f8672bf2740 1 Processor -- start 2024-04-01T01:33:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.166+0000 7f8672bf2740 1 -- start start 2024-04-01T01:33:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.166+0000 7f8672bf2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce25610 0x56104ce20de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.166+0000 7f8672bf2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 0x56104ce36b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.166+0000 7f8672bf2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce271a0 0x56104ce2d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.166+0000 7f8672bf2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56104ce3ba70 con 0x56104ce25610 2024-04-01T01:33:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.166+0000 7f8672bf2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56104ce3c140 con 0x56104ce267e0 2024-04-01T01:33:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f8672bf2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56104ce3c810 con 0x56104ce271a0 2024-04-01T01:33:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f866ddaa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce25610 0x56104ce20de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f866d5a9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 0x56104ce36b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f866ddaa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce25610 0x56104ce20de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33016/0 (socket says 172.21.15.70:33016) 2024-04-01T01:33:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f866d5a9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 0x56104ce36b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:58906/0 (socket says 172.21.15.70:58906) 2024-04-01T01:33:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f866ddaa640 1 -- 172.21.15.70:0/3230427263 learned_addr learned my addr 172.21.15.70:0/3230427263 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f866e5ab640 1 --2- 172.21.15.70:0/3230427263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce271a0 0x56104ce2d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f866ddaa640 1 -- 172.21.15.70:0/3230427263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce271a0 msgr2=0x56104ce2d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f866ddaa640 1 --2- 172.21.15.70:0/3230427263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce271a0 0x56104ce2d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f866ddaa640 1 -- 172.21.15.70:0/3230427263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 msgr2=0x56104ce36b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f866ddaa640 1 --2- 172.21.15.70:0/3230427263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 0x56104ce36b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f866ddaa640 1 -- 172.21.15.70:0/3230427263 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56104ce1f040 con 0x56104ce25610 2024-04-01T01:33:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f866d5a9640 1 --2- 172.21.15.70:0/3230427263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 0x56104ce36b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f866ddaa640 1 --2- 172.21.15.70:0/3230427263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce25610 0x56104ce20de0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x56104ce266e0 tx=0x7f8664005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89579f5409de62b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f86537fe640 1 -- 172.21.15.70:0/3230427263 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8664004030 con 0x56104ce25610 2024-04-01T01:33:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f86537fe640 1 -- 172.21.15.70:0/3230427263 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f866400f870 con 0x56104ce25610 2024-04-01T01:33:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.167+0000 7f86537fe640 1 -- 172.21.15.70:0/3230427263 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8664020450 con 0x56104ce25610 2024-04-01T01:33:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.168+0000 7f8672bf2740 1 -- 172.21.15.70:0/3230427263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce25610 msgr2=0x56104ce20de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.168+0000 7f8672bf2740 1 --2- 172.21.15.70:0/3230427263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce25610 0x56104ce20de0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x56104ce266e0 tx=0x7f8664005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.168+0000 7f8672bf2740 1 -- 172.21.15.70:0/3230427263 shutdown_connections 2024-04-01T01:33:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.168+0000 7f8672bf2740 1 --2- 172.21.15.70:0/3230427263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce271a0 0x56104ce2d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.168+0000 7f8672bf2740 1 --2- 172.21.15.70:0/3230427263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 0x56104ce36b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.168+0000 7f8672bf2740 1 --2- 172.21.15.70:0/3230427263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce25610 0x56104ce20de0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.168+0000 7f8672bf2740 1 -- 172.21.15.70:0/3230427263 >> 172.21.15.70:0/3230427263 conn(0x56104ce20a00 msgr2=0x56104ce2f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.168+0000 7f8672bf2740 1 -- 172.21.15.70:0/3230427263 shutdown_connections 2024-04-01T01:33:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.168+0000 7f8672bf2740 1 -- 172.21.15.70:0/3230427263 wait complete. 2024-04-01T01:33:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.168+0000 7f8672bf2740 1 -- start start 2024-04-01T01:33:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.168+0000 7f8672bf2740 1 Processor -- start 2024-04-01T01:33:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f8672bf2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce25610 0x56104ce34b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f8672bf2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 0x56104ce9e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f8672bf2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce271a0 0x56104cea0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f8672bf2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56104ce3d5b0 con 0x56104ce271a0 2024-04-01T01:33:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f8672bf2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56104ce3cee0 con 0x56104ce267e0 2024-04-01T01:33:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f8672bf2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56104ce3dc80 con 0x56104ce25610 2024-04-01T01:33:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f8672bf2740 10 client.0 _ll_register_callbacks cb 0x7f865c013f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f866e5ab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce271a0 0x56104cea0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f866ddaa640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce25610 0x56104ce34b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f866e5ab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce271a0 0x56104cea0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33018/0 (socket says 172.21.15.70:33018) 2024-04-01T01:33:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f866e5ab640 1 -- 172.21.15.70:0/3430598868 learned_addr learned my addr 172.21.15.70:0/3430598868 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f866d5a9640 1 --2- 172.21.15.70:0/3430598868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 0x56104ce9e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f866e5ab640 1 -- 172.21.15.70:0/3430598868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce25610 msgr2=0x56104ce34b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f866e5ab640 1 --2- 172.21.15.70:0/3430598868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce25610 0x56104ce34b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f866e5ab640 1 -- 172.21.15.70:0/3430598868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 msgr2=0x56104ce9e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f866e5ab640 1 --2- 172.21.15.70:0/3430598868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 0x56104ce9e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f866e5ab640 1 -- 172.21.15.70:0/3430598868 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56104cea0d70 con 0x56104ce271a0 2024-04-01T01:33:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f866e5ab640 1 --2- 172.21.15.70:0/3430598868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce271a0 0x56104cea0560 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f86600bd070 tx=0x7f86600bcdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d46b969c7544d54 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f86517fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f86517fa640 1 -- 172.21.15.70:0/3430598868 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86600c14c0 con 0x56104ce271a0 2024-04-01T01:33:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f86517fa640 1 -- 172.21.15.70:0/3430598868 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f86600dd110 con 0x56104ce271a0 2024-04-01T01:33:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f8672bf2740 1 -- 172.21.15.70:0/3430598868 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f866400f870 con 0x56104ce271a0 2024-04-01T01:33:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f86517fa640 1 -- 172.21.15.70:0/3430598868 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f86600bd370 con 0x56104ce271a0 2024-04-01T01:33:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f8672bf2740 10 client.15447 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.169+0000 7f8672bf2740 1 -- 172.21.15.70:0/3430598868 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56104cea2070 con 0x56104ce271a0 2024-04-01T01:33:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.170+0000 7f86517fa640 1 -- 172.21.15.70:0/3430598868 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f866400f870 con 0x56104ce271a0 2024-04-01T01:33:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.170+0000 7f86517fa640 1 -- 172.21.15.70:0/3430598868 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f866003c030 con 0x56104ce271a0 2024-04-01T01:33:30.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.170+0000 7f8672bf2740 10 client.15447 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:30.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.170+0000 7f8672bf2740 10 client.15447 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-01T01:33:30.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.170+0000 7f8672bf2740 20 client.15447 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:30.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.170+0000 7f8672bf2740 1 --2- 172.21.15.70:0/3430598868 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56104cea6b30 0x56104cea5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.170+0000 7f8672bf2740 4 client.15447 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-01T01:33:30.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.170+0000 7f8672bf2740 1 -- 172.21.15.70:0/3430598868 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "status"}) -- 0x56104cd91380 con 0x56104cea6b30 2024-04-01T01:33:30.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.170+0000 7f866ddaa640 1 --2- 172.21.15.70:0/3430598868 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56104cea6b30 0x56104cea5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.170+0000 7f86517fa640 10 client.15447 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:33:30.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.170+0000 7f866ddaa640 1 --2- 172.21.15.70:0/3430598868 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56104cea6b30 0x56104cea5840 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d644850bffe705be server_cookie=35b05f7e45da1070 in_seq=0 out_seq=0 2024-04-01T01:33:30.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.170+0000 7f86517fa640 1 -- 172.21.15.70:0/3430598868 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56104cd91380 con 0x56104cea6b30 2024-04-01T01:33:30.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.170+0000 7f86517fa640 10 client.15447 handle_command_reply: tid=0 2024-04-01T01:33:30.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.170+0000 7f8672bf2740 1 client.15447 shutdown 2024-04-01T01:33:30.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f8672bf2740 1 -- 172.21.15.70:0/3430598868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce271a0 msgr2=0x56104cea0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f8672bf2740 1 --2- 172.21.15.70:0/3430598868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce271a0 0x56104cea0560 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f86600bd070 tx=0x7f86600bcdd0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f8672bf2740 1 -- 172.21.15.70:0/3430598868 shutdown_connections 2024-04-01T01:33:30.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f8672bf2740 1 --2- 172.21.15.70:0/3430598868 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56104cea6b30 0x56104cea5840 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f8672bf2740 1 --2- 172.21.15.70:0/3430598868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56104ce271a0 0x56104cea0560 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f86517fa640 0 client.15447 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:33:30.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f8672bf2740 1 --2- 172.21.15.70:0/3430598868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56104ce267e0 0x56104ce9e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f8672bf2740 1 --2- 172.21.15.70:0/3430598868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56104ce25610 0x56104ce34b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f8672bf2740 1 -- 172.21.15.70:0/3430598868 >> 172.21.15.70:0/3430598868 conn(0x56104ce20a00 msgr2=0x56104ce2e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f8672bf2740 1 -- 172.21.15.70:0/3430598868 shutdown_connections 2024-04-01T01:33:30.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f8672bf2740 1 -- 172.21.15.70:0/3430598868 wait complete. 2024-04-01T01:33:30.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f8672bf2740 20 client.15447 trim_cache size 0 max 16384 2024-04-01T01:33:30.178 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:33:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "36577394-efc6-11ee-b647-cb9ed24678a4", 2024-04-01T01:33:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 3, 2024-04-01T01:33:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24404, 2024-04-01T01:33:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-01T01:33:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-01T01:33:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-01T01:33:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 113.62185673099999, 2024-04-01T01:33:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-04-01T01:33:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 93, 2024-04-01T01:33:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 89, 2024-04-01T01:33:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 158.82129898700001 2024-04-01T01:33:30.179 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:33:30.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f867000e640 1 -- 172.21.15.70:0/3575040508 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f86540a24a0 msgr2=0x7f86540a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f867000e640 1 --2- 172.21.15.70:0/3575040508 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f86540a24a0 0x7f86540a4960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8668198e40 tx=0x7f865c009c00 comp rx=0 tx=0).stop 2024-04-01T01:33:30.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f867000e640 1 -- 172.21.15.70:0/3575040508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8668103310 msgr2=0x7f866819eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f867000e640 1 --2- 172.21.15.70:0/3575040508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8668103310 0x7f866819eee0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f866002c700 tx=0x7f86600328a0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f867000e640 1 -- 172.21.15.70:0/3575040508 shutdown_connections 2024-04-01T01:33:30.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f867000e640 1 --2- 172.21.15.70:0/3575040508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8668103310 0x7f866819eee0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f867000e640 1 --2- 172.21.15.70:0/3575040508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8668102950 0x7f8668197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f867000e640 1 --2- 172.21.15.70:0/3575040508 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f86540a24a0 0x7f86540a4960 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f867000e640 1 --2- 172.21.15.70:0/3575040508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8668101760 0x7f8668197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f867000e640 1 -- 172.21.15.70:0/3575040508 >> 172.21.15.70:0/3575040508 conn(0x7f86680fccf0 msgr2=0x7f8668105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f867000e640 1 -- 172.21.15.70:0/3575040508 shutdown_connections 2024-04-01T01:33:30.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.171+0000 7f867000e640 1 -- 172.21.15.70:0/3575040508 wait complete. 2024-04-01T01:33:30.183 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-01T01:33:30.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.284+0000 7f187c33e640 1 -- 172.21.15.70:0/1015202254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1874101840 msgr2=0x7f1874101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.284+0000 7f187c33e640 1 --2- 172.21.15.70:0/1015202254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1874101840 0x7f1874101cc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1870009860 tx=0x7f187002bb70 comp rx=0 tx=0).stop 2024-04-01T01:33:30.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187c33e640 1 -- 172.21.15.70:0/1015202254 shutdown_connections 2024-04-01T01:33:30.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187c33e640 1 --2- 172.21.15.70:0/1015202254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1874102200 0x7f1874108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187c33e640 1 --2- 172.21.15.70:0/1015202254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1874101840 0x7f1874101cc0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187c33e640 1 --2- 172.21.15.70:0/1015202254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1874100650 0x7f1874100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187c33e640 1 -- 172.21.15.70:0/1015202254 >> 172.21.15.70:0/1015202254 conn(0x7f18740fbbe0 msgr2=0x7f18740fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187c33e640 1 -- 172.21.15.70:0/1015202254 shutdown_connections 2024-04-01T01:33:30.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187c33e640 1 -- 172.21.15.70:0/1015202254 wait complete. 2024-04-01T01:33:30.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187c33e640 1 Processor -- start 2024-04-01T01:33:30.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187c33e640 1 -- start start 2024-04-01T01:33:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187c33e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1874100650 0x7f1874069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187c33e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1874101840 0x7f187419c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187c33e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1874102200 0x7f187419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187c33e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f187410c190 con 0x7f1874101840 2024-04-01T01:33:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187c33e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f187410c010 con 0x7f1874102200 2024-04-01T01:33:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187c33e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f187410c310 con 0x7f1874100650 2024-04-01T01:33:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f18798d9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1874101840 0x7f187419c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187a0da640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1874100650 0x7f1874069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f18798d9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1874101840 0x7f187419c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33034/0 (socket says 172.21.15.70:33034) 2024-04-01T01:33:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f18798d9640 1 -- 172.21.15.70:0/1965238355 learned_addr learned my addr 172.21.15.70:0/1965238355 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.285+0000 7f187a8db640 1 --2- 172.21.15.70:0/1965238355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1874102200 0x7f187419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.286+0000 7f18798d9640 1 -- 172.21.15.70:0/1965238355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1874100650 msgr2=0x7f1874069420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.286+0000 7f18798d9640 1 --2- 172.21.15.70:0/1965238355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1874100650 0x7f1874069420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.286+0000 7f18798d9640 1 -- 172.21.15.70:0/1965238355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1874102200 msgr2=0x7f187419ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.286+0000 7f18798d9640 1 --2- 172.21.15.70:0/1965238355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1874102200 0x7f187419ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.286+0000 7f18798d9640 1 -- 172.21.15.70:0/1965238355 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f187419f690 con 0x7f1874101840 2024-04-01T01:33:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.286+0000 7f18798d9640 1 --2- 172.21.15.70:0/1965238355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1874101840 0x7f187419c760 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1870005940 tx=0x7f1870004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.286+0000 7f186b7fe640 1 -- 172.21.15.70:0/1965238355 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1870041020 con 0x7f1874101840 2024-04-01T01:33:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.286+0000 7f187c33e640 1 -- 172.21.15.70:0/1965238355 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f187419f8c0 con 0x7f1874101840 2024-04-01T01:33:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.286+0000 7f186b7fe640 1 -- 172.21.15.70:0/1965238355 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f187002c9c0 con 0x7f1874101840 2024-04-01T01:33:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.286+0000 7f186b7fe640 1 -- 172.21.15.70:0/1965238355 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1870039040 con 0x7f1874101840 2024-04-01T01:33:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.286+0000 7f187c33e640 1 -- 172.21.15.70:0/1965238355 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f187419fe00 con 0x7f1874101840 2024-04-01T01:33:30.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.288+0000 7f186b7fe640 1 -- 172.21.15.70:0/1965238355 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f1870044020 con 0x7f1874101840 2024-04-01T01:33:30.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.288+0000 7f187f004740 1 Processor -- start 2024-04-01T01:33:30.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187f004740 1 -- start start 2024-04-01T01:33:30.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f186b7fe640 1 --2- 172.21.15.70:0/1965238355 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f18600a24a0 0x7f18600a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187f004740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc3610 0x5648b9fc3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187f004740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc47e0 0x5648b9fc4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187f004740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc51a0 0x5648b9fcba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187f004740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5648b9f27af0 con 0x5648b9fc51a0 2024-04-01T01:33:30.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187f004740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5648b9ed09d0 con 0x5648b9fc47e0 2024-04-01T01:33:30.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187f004740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5648b9f23b90 con 0x5648b9fc3610 2024-04-01T01:33:30.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187a8db640 1 --2- 172.21.15.70:0/1965238355 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f18600a24a0 0x7f18600a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187a0da640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc3610 0x5648b9fc3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f186b7fe640 1 -- 172.21.15.70:0/1965238355 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f18700f0380 con 0x7f1874101840 2024-04-01T01:33:30.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187a0da640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc3610 0x5648b9fc3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:48788/0 (socket says 172.21.15.70:48788) 2024-04-01T01:33:30.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187a0da640 1 -- 172.21.15.70:0/944852548 learned_addr learned my addr 172.21.15.70:0/944852548 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f18798d9640 1 --2- 172.21.15.70:0/944852548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc47e0 0x5648b9fc4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187a8db640 1 --2- 172.21.15.70:0/944852548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc51a0 0x5648b9fcba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187a0da640 1 -- 172.21.15.70:0/944852548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc47e0 msgr2=0x5648b9fc4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187a0da640 1 --2- 172.21.15.70:0/944852548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc47e0 0x5648b9fc4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187a0da640 1 -- 172.21.15.70:0/944852548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc51a0 msgr2=0x5648b9fcba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187a0da640 1 --2- 172.21.15.70:0/944852548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc51a0 0x5648b9fcba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f187a0da640 1 -- 172.21.15.70:0/944852548 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648b9c3c2d0 con 0x5648b9fc3610 2024-04-01T01:33:30.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.289+0000 7f18798d9640 1 --2- 172.21.15.70:0/944852548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc47e0 0x5648b9fc4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:30.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.290+0000 7f187a8db640 1 --2- 172.21.15.70:0/1965238355 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f18600a24a0 0x7f18600a4960 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f186c005fd0 tx=0x7f186c009e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.290+0000 7f187a0da640 1 --2- 172.21.15.70:0/944852548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc3610 0x5648b9fc3a30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1864009ca0 tx=0x7f1864011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8eeb20a9a1da16b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.290+0000 7f1868bfe640 1 -- 172.21.15.70:0/944852548 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1864004220 con 0x5648b9fc3610 2024-04-01T01:33:30.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.290+0000 7f1868bfe640 1 -- 172.21.15.70:0/944852548 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1864009e50 con 0x5648b9fc3610 2024-04-01T01:33:30.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.290+0000 7f1868bfe640 1 -- 172.21.15.70:0/944852548 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1864019690 con 0x5648b9fc3610 2024-04-01T01:33:30.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.290+0000 7f187f004740 1 -- 172.21.15.70:0/944852548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc3610 msgr2=0x5648b9fc3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.290+0000 7f187f004740 1 --2- 172.21.15.70:0/944852548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc3610 0x5648b9fc3a30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1864009ca0 tx=0x7f1864011810 comp rx=0 tx=0).stop 2024-04-01T01:33:30.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.290+0000 7f187f004740 1 -- 172.21.15.70:0/944852548 shutdown_connections 2024-04-01T01:33:30.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187f004740 1 --2- 172.21.15.70:0/944852548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc51a0 0x5648b9fcba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187f004740 1 --2- 172.21.15.70:0/944852548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc47e0 0x5648b9fc4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187f004740 1 --2- 172.21.15.70:0/944852548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc3610 0x5648b9fc3a30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187f004740 1 -- 172.21.15.70:0/944852548 >> 172.21.15.70:0/944852548 conn(0x5648b9fbea00 msgr2=0x5648b9fbee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187f004740 1 -- 172.21.15.70:0/944852548 shutdown_connections 2024-04-01T01:33:30.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187f004740 1 -- 172.21.15.70:0/944852548 wait complete. 2024-04-01T01:33:30.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187f004740 1 -- start start 2024-04-01T01:33:30.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187f004740 1 Processor -- start 2024-04-01T01:33:30.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187f004740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc3610 0x5648b9fcd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187f004740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc47e0 0x5648b9fcde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187f004740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc51a0 0x5648b9fd4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187f004740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5648b9fcc1b0 con 0x5648b9fc3610 2024-04-01T01:33:30.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187f004740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5648b9ee3f70 con 0x5648b9fc51a0 2024-04-01T01:33:30.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187f004740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5648b9fcc330 con 0x5648b9fc47e0 2024-04-01T01:33:30.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187f004740 10 client.0 _ll_register_callbacks cb 0x5648b9c3bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:30.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187a0da640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc3610 0x5648b9fcd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187a0da640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc3610 0x5648b9fcd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33058/0 (socket says 172.21.15.70:33058) 2024-04-01T01:33:30.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f18798d9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc47e0 0x5648b9fcde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187a0da640 1 -- 172.21.15.70:0/3632516122 learned_addr learned my addr 172.21.15.70:0/3632516122 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.291+0000 7f187a8db640 1 --2- 172.21.15.70:0/3632516122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc51a0 0x5648b9fd4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f187a0da640 1 -- 172.21.15.70:0/3632516122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc47e0 msgr2=0x5648b9fcde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f187a0da640 1 --2- 172.21.15.70:0/3632516122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc47e0 0x5648b9fcde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f187a0da640 1 -- 172.21.15.70:0/3632516122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc51a0 msgr2=0x5648b9fd4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f187a0da640 1 --2- 172.21.15.70:0/3632516122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc51a0 0x5648b9fd4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f187a0da640 1 -- 172.21.15.70:0/3632516122 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648b9d066b0 con 0x5648b9fc3610 2024-04-01T01:33:30.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f18798d9640 1 --2- 172.21.15.70:0/3632516122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc47e0 0x5648b9fcde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:30.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f187a8db640 1 --2- 172.21.15.70:0/3632516122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc51a0 0x5648b9fd4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:30.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f187a0da640 1 --2- 172.21.15.70:0/3632516122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc3610 0x5648b9fcd8e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1864009380 tx=0x7f1864004140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f09289f089b45f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f185dffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:30.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f185dffb640 1 -- 172.21.15.70:0/3632516122 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1864028020 con 0x5648b9fc3610 2024-04-01T01:33:30.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f185dffb640 1 -- 172.21.15.70:0/3632516122 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1864004330 con 0x5648b9fc3610 2024-04-01T01:33:30.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f187f004740 1 -- 172.21.15.70:0/3632516122 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1864009e50 con 0x5648b9fc3610 2024-04-01T01:33:30.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f185dffb640 1 -- 172.21.15.70:0/3632516122 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1864005050 con 0x5648b9fc3610 2024-04-01T01:33:30.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f187f004740 10 client.15459 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:30.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f187f004740 1 -- 172.21.15.70:0/3632516122 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5648b9fd7e40 con 0x5648b9fc3610 2024-04-01T01:33:30.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f185dffb640 1 -- 172.21.15.70:0/3632516122 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1864009e50 con 0x5648b9fc3610 2024-04-01T01:33:30.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f185dffb640 1 -- 172.21.15.70:0/3632516122 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1864004cd0 con 0x5648b9fc3610 2024-04-01T01:33:30.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.292+0000 7f187f004740 10 client.15459 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:30.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.293+0000 7f187f004740 10 client.15459 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-01T01:33:30.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.293+0000 7f187f004740 20 client.15459 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:30.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.293+0000 7f187f004740 1 --2- 172.21.15.70:0/3632516122 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5648b9fda2c0 0x5648b9fdb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.293+0000 7f187f004740 4 client.15459 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:33:30.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.293+0000 7f187f004740 1 -- 172.21.15.70:0/3632516122 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648b9f35030 con 0x5648b9fda2c0 2024-04-01T01:33:30.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.293+0000 7f18798d9640 1 --2- 172.21.15.70:0/3632516122 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5648b9fda2c0 0x5648b9fdb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.293+0000 7f18798d9640 1 --2- 172.21.15.70:0/3632516122 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5648b9fda2c0 0x5648b9fdb730 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dca67471f5b771c6 server_cookie=d3b799c926c7b8c2 in_seq=0 out_seq=0 2024-04-01T01:33:30.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.293+0000 7f185dffb640 10 client.15459 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:33:30.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.294+0000 7f185dffb640 1 -- 172.21.15.70:0/3632516122 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5648b9f35030 con 0x5648b9fda2c0 2024-04-01T01:33:30.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.294+0000 7f185dffb640 10 client.15459 handle_command_reply: tid=0 2024-04-01T01:33:30.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.294+0000 7f187f004740 1 client.15459 shutdown 2024-04-01T01:33:30.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.294+0000 7f187f004740 1 -- 172.21.15.70:0/3632516122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc3610 msgr2=0x5648b9fcd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.294+0000 7f187f004740 1 --2- 172.21.15.70:0/3632516122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc3610 0x5648b9fcd8e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1864009380 tx=0x7f1864004140 comp rx=0 tx=0).stop 2024-04-01T01:33:30.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.294+0000 7f187f004740 1 -- 172.21.15.70:0/3632516122 shutdown_connections 2024-04-01T01:33:30.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.294+0000 7f187f004740 1 --2- 172.21.15.70:0/3632516122 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5648b9fda2c0 0x5648b9fdb730 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.294+0000 7f187f004740 1 --2- 172.21.15.70:0/3632516122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc51a0 0x5648b9fd4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.294+0000 7f187f004740 1 --2- 172.21.15.70:0/3632516122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc47e0 0x5648b9fcde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.294+0000 7f187f004740 1 --2- 172.21.15.70:0/3632516122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc3610 0x5648b9fcd8e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.294+0000 7f185dffb640 0 client.15459 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:33:30.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.294+0000 7f187f004740 1 -- 172.21.15.70:0/3632516122 >> 172.21.15.70:0/3632516122 conn(0x5648b9fbea00 msgr2=0x5648b9fbdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.294+0000 7f187f004740 1 -- 172.21.15.70:0/3632516122 shutdown_connections 2024-04-01T01:33:30.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.294+0000 7f187f004740 1 -- 172.21.15.70:0/3632516122 wait complete. 2024-04-01T01:33:30.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.294+0000 7f187f004740 20 client.15459 trim_cache size 0 max 16384 2024-04-01T01:33:30.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.311+0000 7f187f004740 1 Processor -- start 2024-04-01T01:33:30.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187f004740 1 -- start start 2024-04-01T01:33:30.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187f004740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc3610 0x5648b9fc4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187f004740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc47e0 0x5648b9fd4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187f004740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc51a0 0x5648b9fcb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187f004740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5648b9fd9a70 con 0x5648b9fc3610 2024-04-01T01:33:30.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187f004740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5648b9fda140 con 0x5648b9fc47e0 2024-04-01T01:33:30.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187f004740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5648b9fda810 con 0x5648b9fc51a0 2024-04-01T01:33:30.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187a0da640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc3610 0x5648b9fc4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187a0da640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc3610 0x5648b9fc4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33068/0 (socket says 172.21.15.70:33068) 2024-04-01T01:33:30.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187a0da640 1 -- 172.21.15.70:0/973294078 learned_addr learned my addr 172.21.15.70:0/973294078 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f18798d9640 1 --2- 172.21.15.70:0/973294078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc47e0 0x5648b9fd4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187a8db640 1 --2- 172.21.15.70:0/973294078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc51a0 0x5648b9fcb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187a0da640 1 -- 172.21.15.70:0/973294078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc51a0 msgr2=0x5648b9fcb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187a0da640 1 --2- 172.21.15.70:0/973294078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc51a0 0x5648b9fcb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187a0da640 1 -- 172.21.15.70:0/973294078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc47e0 msgr2=0x5648b9fd4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187a0da640 1 --2- 172.21.15.70:0/973294078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc47e0 0x5648b9fd4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187a0da640 1 -- 172.21.15.70:0/973294078 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648b9fbd040 con 0x5648b9fc3610 2024-04-01T01:33:30.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f187a0da640 1 --2- 172.21.15.70:0/973294078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc3610 0x5648b9fc4bc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f18640245c0 tx=0x7f18640097c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b511170295b9b524 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f185ffff640 1 -- 172.21.15.70:0/973294078 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186403c070 con 0x5648b9fc3610 2024-04-01T01:33:30.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f185ffff640 1 -- 172.21.15.70:0/973294078 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18640056a0 con 0x5648b9fc3610 2024-04-01T01:33:30.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.312+0000 7f185ffff640 1 -- 172.21.15.70:0/973294078 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1864028070 con 0x5648b9fc3610 2024-04-01T01:33:30.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 -- 172.21.15.70:0/973294078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc3610 msgr2=0x5648b9fc4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 --2- 172.21.15.70:0/973294078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc3610 0x5648b9fc4bc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f18640245c0 tx=0x7f18640097c0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 -- 172.21.15.70:0/973294078 shutdown_connections 2024-04-01T01:33:30.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 --2- 172.21.15.70:0/973294078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc51a0 0x5648b9fcb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 --2- 172.21.15.70:0/973294078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc47e0 0x5648b9fd4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 --2- 172.21.15.70:0/973294078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc3610 0x5648b9fc4bc0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 -- 172.21.15.70:0/973294078 >> 172.21.15.70:0/973294078 conn(0x5648b9fbea00 msgr2=0x5648b9fc24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 -- 172.21.15.70:0/973294078 shutdown_connections 2024-04-01T01:33:30.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 -- 172.21.15.70:0/973294078 wait complete. 2024-04-01T01:33:30.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 -- start start 2024-04-01T01:33:30.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 Processor -- start 2024-04-01T01:33:30.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc3610 0x5648b9fcff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc47e0 0x5648b9fd04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc51a0 0x5648b9fd34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5648b9fdb5b0 con 0x5648b9fc51a0 2024-04-01T01:33:30.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5648b9fdaee0 con 0x5648b9fc3610 2024-04-01T01:33:30.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5648b9fdbc80 con 0x5648b9fc47e0 2024-04-01T01:33:30.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.313+0000 7f187f004740 10 client.0 _ll_register_callbacks cb 0x7f186c00e770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:30.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f187a8db640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc51a0 0x5648b9fd34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f187a8db640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc51a0 0x5648b9fd34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33072/0 (socket says 172.21.15.70:33072) 2024-04-01T01:33:30.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f187a8db640 1 -- 172.21.15.70:0/4142669740 learned_addr learned my addr 172.21.15.70:0/4142669740 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f187a0da640 1 --2- 172.21.15.70:0/4142669740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc3610 0x5648b9fcff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f18798d9640 1 --2- 172.21.15.70:0/4142669740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc47e0 0x5648b9fd04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f187a8db640 1 -- 172.21.15.70:0/4142669740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc47e0 msgr2=0x5648b9fd04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f187a8db640 1 --2- 172.21.15.70:0/4142669740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc47e0 0x5648b9fd04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f187a8db640 1 -- 172.21.15.70:0/4142669740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc3610 msgr2=0x5648b9fcff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f187a8db640 1 --2- 172.21.15.70:0/4142669740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc3610 0x5648b9fcff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f187a8db640 1 -- 172.21.15.70:0/4142669740 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648b9fd3ac0 con 0x5648b9fc51a0 2024-04-01T01:33:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f187a8db640 1 --2- 172.21.15.70:0/4142669740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc51a0 0x5648b9fd34d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f186c01ddc0 tx=0x7f186c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=851468ace978db69 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f185dffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f185dffb640 1 -- 172.21.15.70:0/4142669740 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c025040 con 0x5648b9fc51a0 2024-04-01T01:33:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f185dffb640 1 -- 172.21.15.70:0/4142669740 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f186c01ddf0 con 0x5648b9fc51a0 2024-04-01T01:33:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f187f004740 1 -- 172.21.15.70:0/4142669740 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18640056a0 con 0x5648b9fc51a0 2024-04-01T01:33:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f185dffb640 1 -- 172.21.15.70:0/4142669740 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f186c039050 con 0x5648b9fc51a0 2024-04-01T01:33:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f187f004740 10 client.15465 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:30.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f187f004740 1 -- 172.21.15.70:0/4142669740 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5648b9fd4e30 con 0x5648b9fc51a0 2024-04-01T01:33:30.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.314+0000 7f185dffb640 1 -- 172.21.15.70:0/4142669740 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18640056a0 con 0x5648b9fc51a0 2024-04-01T01:33:30.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.315+0000 7f185dffb640 1 -- 172.21.15.70:0/4142669740 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f186c028030 con 0x5648b9fc51a0 2024-04-01T01:33:30.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.315+0000 7f187f004740 10 client.15465 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:30.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.315+0000 7f187f004740 10 client.15465 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-01T01:33:30.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.315+0000 7f187f004740 20 client.15465 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:30.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.315+0000 7f187f004740 1 --2- 172.21.15.70:0/4142669740 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5648ba044730 0x5648ba040b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.315+0000 7f187f004740 4 client.15465 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-01T01:33:30.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.315+0000 7f187f004740 1 -- 172.21.15.70:0/4142669740 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "status"}) -- 0x5648b9f2f380 con 0x5648ba044730 2024-04-01T01:33:30.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.315+0000 7f187a0da640 1 --2- 172.21.15.70:0/4142669740 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5648ba044730 0x5648ba040b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.315+0000 7f187a0da640 1 --2- 172.21.15.70:0/4142669740 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5648ba044730 0x5648ba040b70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eac1f55f9ca48f0b server_cookie=9b68221db591a1a0 in_seq=0 out_seq=0 2024-04-01T01:33:30.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.315+0000 7f185dffb640 10 client.15465 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:33:30.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.315+0000 7f185dffb640 1 -- 172.21.15.70:0/4142669740 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5648b9f2f380 con 0x5648ba044730 2024-04-01T01:33:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.315+0000 7f185dffb640 10 client.15465 handle_command_reply: tid=0 2024-04-01T01:33:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.315+0000 7f187f004740 1 client.15465 shutdown 2024-04-01T01:33:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.315+0000 7f187f004740 1 -- 172.21.15.70:0/4142669740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc51a0 msgr2=0x5648b9fd34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.315+0000 7f187f004740 1 --2- 172.21.15.70:0/4142669740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc51a0 0x5648b9fd34d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f186c01ddc0 tx=0x7f186c01d000 comp rx=0 tx=0).stop 2024-04-01T01:33:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187f004740 1 -- 172.21.15.70:0/4142669740 shutdown_connections 2024-04-01T01:33:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187f004740 1 --2- 172.21.15.70:0/4142669740 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5648ba044730 0x5648ba040b70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187f004740 1 --2- 172.21.15.70:0/4142669740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5648b9fc51a0 0x5648b9fd34d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187f004740 1 --2- 172.21.15.70:0/4142669740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5648b9fc47e0 0x5648b9fd04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f185dffb640 0 client.15465 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:33:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187f004740 1 --2- 172.21.15.70:0/4142669740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5648b9fc3610 0x5648b9fcff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187f004740 1 -- 172.21.15.70:0/4142669740 >> 172.21.15.70:0/4142669740 conn(0x5648b9fbea00 msgr2=0x5648b9fd9250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187f004740 1 -- 172.21.15.70:0/4142669740 shutdown_connections 2024-04-01T01:33:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187f004740 1 -- 172.21.15.70:0/4142669740 wait complete. 2024-04-01T01:33:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187f004740 20 client.15465 trim_cache size 0 max 16384 2024-04-01T01:33:30.324 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:33:30.324 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "36577394-efc6-11ee-b647-cb9ed24678a4", 2024-04-01T01:33:30.324 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 2, 2024-04-01T01:33:30.324 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24418, 2024-04-01T01:33:30.324 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-01T01:33:30.324 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-01T01:33:30.324 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-01T01:33:30.324 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 114.77057137, 2024-04-01T01:33:30.324 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-04-01T01:33:30.324 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 93, 2024-04-01T01:33:30.324 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 89, 2024-04-01T01:33:30.325 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 156.21508736000001 2024-04-01T01:33:30.325 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:33:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187c33e640 1 -- 172.21.15.70:0/1965238355 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f18600a24a0 msgr2=0x7f18600a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187c33e640 1 --2- 172.21.15.70:0/1965238355 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f18600a24a0 0x7f18600a4960 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f186c005fd0 tx=0x7f186c009e30 comp rx=0 tx=0).stop 2024-04-01T01:33:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187c33e640 1 -- 172.21.15.70:0/1965238355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1874101840 msgr2=0x7f187419c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187c33e640 1 --2- 172.21.15.70:0/1965238355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1874101840 0x7f187419c760 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1870005940 tx=0x7f1870004270 comp rx=0 tx=0).stop 2024-04-01T01:33:30.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187c33e640 1 -- 172.21.15.70:0/1965238355 shutdown_connections 2024-04-01T01:33:30.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187c33e640 1 --2- 172.21.15.70:0/1965238355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1874102200 0x7f187419ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187c33e640 1 --2- 172.21.15.70:0/1965238355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1874101840 0x7f187419c760 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187c33e640 1 --2- 172.21.15.70:0/1965238355 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f18600a24a0 0x7f18600a4960 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187c33e640 1 --2- 172.21.15.70:0/1965238355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1874100650 0x7f1874069420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187c33e640 1 -- 172.21.15.70:0/1965238355 >> 172.21.15.70:0/1965238355 conn(0x7f18740fbbe0 msgr2=0x7f1874104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187c33e640 1 -- 172.21.15.70:0/1965238355 shutdown_connections 2024-04-01T01:33:30.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.316+0000 7f187c33e640 1 -- 172.21.15.70:0/1965238355 wait complete. 2024-04-01T01:33:30.327 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-01T01:33:30.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.430+0000 7fddbfe4e640 1 -- 172.21.15.70:0/234909631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddb8102210 msgr2=0x7fddb8108aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.430+0000 7fddbfe4e640 1 --2- 172.21.15.70:0/234909631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddb8102210 0x7fddb8108aa0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fddb0009ca0 tx=0x7fddb002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.430+0000 7fddbfe4e640 1 -- 172.21.15.70:0/234909631 shutdown_connections 2024-04-01T01:33:30.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.430+0000 7fddbfe4e640 1 --2- 172.21.15.70:0/234909631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddb8102210 0x7fddb8108aa0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.430+0000 7fddbfe4e640 1 --2- 172.21.15.70:0/234909631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddb8101850 0x7fddb8101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.430+0000 7fddbfe4e640 1 --2- 172.21.15.70:0/234909631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb8100660 0x7fddb8100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.430+0000 7fddbfe4e640 1 -- 172.21.15.70:0/234909631 >> 172.21.15.70:0/234909631 conn(0x7fddb80fbbf0 msgr2=0x7fddb80fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.430+0000 7fddbfe4e640 1 -- 172.21.15.70:0/234909631 shutdown_connections 2024-04-01T01:33:30.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.430+0000 7fddbfe4e640 1 -- 172.21.15.70:0/234909631 wait complete. 2024-04-01T01:33:30.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbfe4e640 1 Processor -- start 2024-04-01T01:33:30.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbfe4e640 1 -- start start 2024-04-01T01:33:30.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbfe4e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb8100660 0x7fddb8197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbfe4e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddb8101850 0x7fddb8197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbfe4e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddb8102210 0x7fddb819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbfe4e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fddb810d2b0 con 0x7fddb8101850 2024-04-01T01:33:30.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbfe4e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fddb810d130 con 0x7fddb8100660 2024-04-01T01:33:30.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbfe4e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fddb810d430 con 0x7fddb8102210 2024-04-01T01:33:30.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbd3e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddb8101850 0x7fddb8197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbd3e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddb8101850 0x7fddb8197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33100/0 (socket says 172.21.15.70:33100) 2024-04-01T01:33:30.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbd3e9640 1 -- 172.21.15.70:0/1423493995 learned_addr learned my addr 172.21.15.70:0/1423493995 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbe3eb640 1 --2- 172.21.15.70:0/1423493995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddb8102210 0x7fddb819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbdbea640 1 --2- 172.21.15.70:0/1423493995 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb8100660 0x7fddb8197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbd3e9640 1 -- 172.21.15.70:0/1423493995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddb8102210 msgr2=0x7fddb819ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbd3e9640 1 --2- 172.21.15.70:0/1423493995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddb8102210 0x7fddb819ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbd3e9640 1 -- 172.21.15.70:0/1423493995 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb8100660 msgr2=0x7fddb8197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbd3e9640 1 --2- 172.21.15.70:0/1423493995 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb8100660 0x7fddb8197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbd3e9640 1 -- 172.21.15.70:0/1423493995 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddb819f580 con 0x7fddb8101850 2024-04-01T01:33:30.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbd3e9640 1 --2- 172.21.15.70:0/1423493995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddb8101850 0x7fddb8197ea0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdda800e9f0 tx=0x7fdda800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddaeffd640 1 -- 172.21.15.70:0/1423493995 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda800cd80 con 0x7fddb8101850 2024-04-01T01:33:30.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddaeffd640 1 -- 172.21.15.70:0/1423493995 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdda80054d0 con 0x7fddb8101850 2024-04-01T01:33:30.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.431+0000 7fddbfe4e640 1 -- 172.21.15.70:0/1423493995 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddb819f870 con 0x7fddb8101850 2024-04-01T01:33:30.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.432+0000 7fddaeffd640 1 -- 172.21.15.70:0/1423493995 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda8010490 con 0x7fddb8101850 2024-04-01T01:33:30.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.432+0000 7fddbfe4e640 1 -- 172.21.15.70:0/1423493995 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddb819fd70 con 0x7fddb8101850 2024-04-01T01:33:30.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.434+0000 7fddaeffd640 1 -- 172.21.15.70:0/1423493995 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fdda8005020 con 0x7fddb8101850 2024-04-01T01:33:30.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.434+0000 7fddaeffd640 1 --2- 172.21.15.70:0/1423493995 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdda40a24a0 0x7fdda40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.434+0000 7fddc2a35740 1 Processor -- start 2024-04-01T01:33:30.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.434+0000 7fddbdbea640 1 --2- 172.21.15.70:0/1423493995 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdda40a24a0 0x7fdda40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.434+0000 7fddaeffd640 1 -- 172.21.15.70:0/1423493995 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fdda80cee60 con 0x7fddb8101850 2024-04-01T01:33:30.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.434+0000 7fddc2a35740 1 -- start start 2024-04-01T01:33:30.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddc2a35740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ed610 0x5578641eda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddc2a35740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ee7e0 0x5578641eec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddc2a35740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ef1a0 0x5578641f5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddc2a35740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557864151af0 con 0x5578641ee7e0 2024-04-01T01:33:30.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddc2a35740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5578640fa9d0 con 0x5578641ed610 2024-04-01T01:33:30.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddc2a35740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55786414db90 con 0x5578641ef1a0 2024-04-01T01:33:30.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddbd3e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ee7e0 0x5578641eec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddbdbea640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ed610 0x5578641eda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddbd3e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ee7e0 0x5578641eec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33104/0 (socket says 172.21.15.70:33104) 2024-04-01T01:33:30.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddbd3e9640 1 -- 172.21.15.70:0/2761877263 learned_addr learned my addr 172.21.15.70:0/2761877263 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddbe3eb640 1 --2- 172.21.15.70:0/2761877263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ef1a0 0x5578641f5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddbdbea640 1 --2- 172.21.15.70:0/1423493995 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdda40a24a0 0x7fdda40a4960 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fddb40095f0 tx=0x7fddb4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddbdbea640 1 -- 172.21.15.70:0/2761877263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ef1a0 msgr2=0x5578641f5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddbdbea640 1 --2- 172.21.15.70:0/2761877263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ef1a0 0x5578641f5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddbdbea640 1 -- 172.21.15.70:0/2761877263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ee7e0 msgr2=0x5578641eec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddbdbea640 1 --2- 172.21.15.70:0/2761877263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ee7e0 0x5578641eec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddbdbea640 1 -- 172.21.15.70:0/2761877263 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557863e662d0 con 0x5578641ed610 2024-04-01T01:33:30.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fddbdbea640 1 --2- 172.21.15.70:0/2761877263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ed610 0x5578641eda30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fddb4012e00 tx=0x7fddb4014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a00e7afdad21f725 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fdda3fff640 1 -- 172.21.15.70:0/2761877263 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb4004230 con 0x5578641ed610 2024-04-01T01:33:30.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fdda3fff640 1 -- 172.21.15.70:0/2761877263 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fddb401ce40 con 0x5578641ed610 2024-04-01T01:33:30.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.435+0000 7fdda3fff640 1 -- 172.21.15.70:0/2761877263 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb401b430 con 0x5578641ed610 2024-04-01T01:33:30.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.436+0000 7fddc2a35740 1 -- 172.21.15.70:0/2761877263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ed610 msgr2=0x5578641eda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.436+0000 7fddc2a35740 1 --2- 172.21.15.70:0/2761877263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ed610 0x5578641eda30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fddb4012e00 tx=0x7fddb4014000 comp rx=0 tx=0).stop 2024-04-01T01:33:30.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.436+0000 7fddc2a35740 1 -- 172.21.15.70:0/2761877263 shutdown_connections 2024-04-01T01:33:30.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.436+0000 7fddc2a35740 1 --2- 172.21.15.70:0/2761877263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ef1a0 0x5578641f5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.436+0000 7fddc2a35740 1 --2- 172.21.15.70:0/2761877263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ee7e0 0x5578641eec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.436+0000 7fddc2a35740 1 --2- 172.21.15.70:0/2761877263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ed610 0x5578641eda30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.436+0000 7fddc2a35740 1 -- 172.21.15.70:0/2761877263 >> 172.21.15.70:0/2761877263 conn(0x5578641e8a00 msgr2=0x5578641e8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.436+0000 7fddc2a35740 1 -- 172.21.15.70:0/2761877263 shutdown_connections 2024-04-01T01:33:30.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.436+0000 7fddc2a35740 1 -- 172.21.15.70:0/2761877263 wait complete. 2024-04-01T01:33:30.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddc2a35740 1 -- start start 2024-04-01T01:33:30.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddc2a35740 1 Processor -- start 2024-04-01T01:33:30.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddc2a35740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ed610 0x5578641f7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddc2a35740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ee7e0 0x5578641f7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddc2a35740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ef1a0 0x5578641feed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddc2a35740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5578641f61b0 con 0x5578641ee7e0 2024-04-01T01:33:30.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddc2a35740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55786410df70 con 0x5578641ef1a0 2024-04-01T01:33:30.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddc2a35740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5578641f6330 con 0x5578641ed610 2024-04-01T01:33:30.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddc2a35740 10 client.0 _ll_register_callbacks cb 0x557863e65ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:30.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddbd3e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ee7e0 0x5578641f7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddbd3e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ee7e0 0x5578641f7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33114/0 (socket says 172.21.15.70:33114) 2024-04-01T01:33:30.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddbd3e9640 1 -- 172.21.15.70:0/654585165 learned_addr learned my addr 172.21.15.70:0/654585165 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddbdbea640 1 --2- 172.21.15.70:0/654585165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ed610 0x5578641f7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddbe3eb640 1 --2- 172.21.15.70:0/654585165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ef1a0 0x5578641feed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddbd3e9640 1 -- 172.21.15.70:0/654585165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ed610 msgr2=0x5578641f7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddbd3e9640 1 --2- 172.21.15.70:0/654585165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ed610 0x5578641f7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddbd3e9640 1 -- 172.21.15.70:0/654585165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ef1a0 msgr2=0x5578641feed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddbd3e9640 1 --2- 172.21.15.70:0/654585165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ef1a0 0x5578641feed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.437+0000 7fddbd3e9640 1 -- 172.21.15.70:0/654585165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557863f306b0 con 0x5578641ee7e0 2024-04-01T01:33:30.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fddbd3e9640 1 --2- 172.21.15.70:0/654585165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ee7e0 0x5578641f7e90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fdda80097f0 tx=0x7fdda809af70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd54ccf29f676d3c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fdda17fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:30.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fdda17fa640 1 -- 172.21.15.70:0/654585165 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda809b550 con 0x5578641ee7e0 2024-04-01T01:33:30.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fdda17fa640 1 -- 172.21.15.70:0/654585165 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fdda80b4260 con 0x5578641ee7e0 2024-04-01T01:33:30.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fddc2a35740 1 -- 172.21.15.70:0/654585165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55786415f030 con 0x5578641ee7e0 2024-04-01T01:33:30.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fdda17fa640 1 -- 172.21.15.70:0/654585165 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdda809f6a0 con 0x5578641ee7e0 2024-04-01T01:33:30.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fddc2a35740 10 client.15477 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:30.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fddc2a35740 1 -- 172.21.15.70:0/654585165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557864201f80 con 0x5578641ee7e0 2024-04-01T01:33:30.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fdda17fa640 1 -- 172.21.15.70:0/654585165 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55786415f030 con 0x5578641ee7e0 2024-04-01T01:33:30.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fdda17fa640 1 -- 172.21.15.70:0/654585165 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fdda809a9a0 con 0x5578641ee7e0 2024-04-01T01:33:30.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fddc2a35740 10 client.15477 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:30.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fddc2a35740 10 client.15477 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-01T01:33:30.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fddc2a35740 20 client.15477 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:30.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fddc2a35740 1 --2- 172.21.15.70:0/654585165 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x557864204520 0x557864205990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fddc2a35740 4 client.15477 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:33:30.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fddc2a35740 1 -- 172.21.15.70:0/654585165 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557864159380 con 0x557864204520 2024-04-01T01:33:30.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.438+0000 7fddbdbea640 1 --2- 172.21.15.70:0/654585165 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x557864204520 0x557864205990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.439+0000 7fdda17fa640 10 client.15477 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:33:30.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.439+0000 7fddbdbea640 1 --2- 172.21.15.70:0/654585165 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x557864204520 0x557864205990 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6b0ba095e48aa1d server_cookie=a538d5e1d7e2ee59 in_seq=0 out_seq=0 2024-04-01T01:33:30.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.440+0000 7fdda17fa640 1 -- 172.21.15.70:0/654585165 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557864159380 con 0x557864204520 2024-04-01T01:33:30.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.440+0000 7fdda17fa640 10 client.15477 handle_command_reply: tid=0 2024-04-01T01:33:30.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.440+0000 7fddc2a35740 1 client.15477 shutdown 2024-04-01T01:33:30.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.440+0000 7fddc2a35740 1 -- 172.21.15.70:0/654585165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ee7e0 msgr2=0x5578641f7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.440+0000 7fddc2a35740 1 --2- 172.21.15.70:0/654585165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ee7e0 0x5578641f7e90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fdda80097f0 tx=0x7fdda809af70 comp rx=0 tx=0).stop 2024-04-01T01:33:30.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.440+0000 7fddc2a35740 1 -- 172.21.15.70:0/654585165 shutdown_connections 2024-04-01T01:33:30.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.440+0000 7fddc2a35740 1 --2- 172.21.15.70:0/654585165 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x557864204520 0x557864205990 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.440+0000 7fddc2a35740 1 --2- 172.21.15.70:0/654585165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ef1a0 0x5578641feed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.440+0000 7fddc2a35740 1 --2- 172.21.15.70:0/654585165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ee7e0 0x5578641f7e90 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.440+0000 7fdda17fa640 0 client.15477 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:33:30.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.440+0000 7fddc2a35740 1 --2- 172.21.15.70:0/654585165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ed610 0x5578641f7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.440+0000 7fddc2a35740 1 -- 172.21.15.70:0/654585165 >> 172.21.15.70:0/654585165 conn(0x5578641e8a00 msgr2=0x5578641e7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.440+0000 7fddc2a35740 1 -- 172.21.15.70:0/654585165 shutdown_connections 2024-04-01T01:33:30.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.440+0000 7fddc2a35740 1 -- 172.21.15.70:0/654585165 wait complete. 2024-04-01T01:33:30.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.440+0000 7fddc2a35740 20 client.15477 trim_cache size 0 max 16384 2024-04-01T01:33:30.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddc2a35740 1 Processor -- start 2024-04-01T01:33:30.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddc2a35740 1 -- start start 2024-04-01T01:33:30.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddc2a35740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ed610 0x5578641f5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddc2a35740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ee7e0 0x5578641fee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddc2a35740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ef1a0 0x5578641f7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddc2a35740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557864203cd0 con 0x5578641ef1a0 2024-04-01T01:33:30.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddc2a35740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5578642043a0 con 0x5578641ed610 2024-04-01T01:33:30.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddc2a35740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557864204a70 con 0x5578641ee7e0 2024-04-01T01:33:30.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddbe3eb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ef1a0 0x5578641f7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddbe3eb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ef1a0 0x5578641f7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33118/0 (socket says 172.21.15.70:33118) 2024-04-01T01:33:30.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddbe3eb640 1 -- 172.21.15.70:0/4231148494 learned_addr learned my addr 172.21.15.70:0/4231148494 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddbdbea640 1 --2- 172.21.15.70:0/4231148494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ed610 0x5578641f5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddbd3e9640 1 --2- 172.21.15.70:0/4231148494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ee7e0 0x5578641fee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddbe3eb640 1 -- 172.21.15.70:0/4231148494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ee7e0 msgr2=0x5578641fee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddbe3eb640 1 --2- 172.21.15.70:0/4231148494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ee7e0 0x5578641fee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddbe3eb640 1 -- 172.21.15.70:0/4231148494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ed610 msgr2=0x5578641f5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddbe3eb640 1 --2- 172.21.15.70:0/4231148494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ed610 0x5578641f5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddbe3eb640 1 -- 172.21.15.70:0/4231148494 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557864204f20 con 0x5578641ef1a0 2024-04-01T01:33:30.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.458+0000 7fddbe3eb640 1 --2- 172.21.15.70:0/4231148494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ef1a0 0x5578641f7950 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fddb0036000 tx=0x7fddb00324f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dbd3fdcf063f3a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.459+0000 7fdda37fe640 1 -- 172.21.15.70:0/4231148494 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb0039040 con 0x5578641ef1a0 2024-04-01T01:33:30.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.459+0000 7fdda37fe640 1 -- 172.21.15.70:0/4231148494 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fddb0045070 con 0x5578641ef1a0 2024-04-01T01:33:30.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.459+0000 7fdda37fe640 1 -- 172.21.15.70:0/4231148494 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb00042a0 con 0x5578641ef1a0 2024-04-01T01:33:30.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.459+0000 7fddc2a35740 1 -- 172.21.15.70:0/4231148494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ef1a0 msgr2=0x5578641f7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.459+0000 7fddc2a35740 1 --2- 172.21.15.70:0/4231148494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ef1a0 0x5578641f7950 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fddb0036000 tx=0x7fddb00324f0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.459+0000 7fddc2a35740 1 -- 172.21.15.70:0/4231148494 shutdown_connections 2024-04-01T01:33:30.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.459+0000 7fddc2a35740 1 --2- 172.21.15.70:0/4231148494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ef1a0 0x5578641f7950 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.459+0000 7fddc2a35740 1 --2- 172.21.15.70:0/4231148494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ee7e0 0x5578641fee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.459+0000 7fddc2a35740 1 --2- 172.21.15.70:0/4231148494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ed610 0x5578641f5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.459+0000 7fddc2a35740 1 -- 172.21.15.70:0/4231148494 >> 172.21.15.70:0/4231148494 conn(0x5578641e8a00 msgr2=0x5578641e3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.459+0000 7fddc2a35740 1 -- 172.21.15.70:0/4231148494 shutdown_connections 2024-04-01T01:33:30.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.459+0000 7fddc2a35740 1 -- 172.21.15.70:0/4231148494 wait complete. 2024-04-01T01:33:30.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddc2a35740 1 -- start start 2024-04-01T01:33:30.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddc2a35740 1 Processor -- start 2024-04-01T01:33:30.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddc2a35740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ed610 0x5578641fb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddc2a35740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ee7e0 0x5578641fb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddc2a35740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ef1a0 0x5578641fe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddc2a35740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557864205810 con 0x5578641ed610 2024-04-01T01:33:30.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddc2a35740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x557864205140 con 0x5578641ee7e0 2024-04-01T01:33:30.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddc2a35740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557864205ee0 con 0x5578641ef1a0 2024-04-01T01:33:30.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddc2a35740 10 client.0 _ll_register_callbacks cb 0x7fdda809b090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:30.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddbdbea640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ed610 0x5578641fb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddbdbea640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ed610 0x5578641fb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33134/0 (socket says 172.21.15.70:33134) 2024-04-01T01:33:30.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddbdbea640 1 -- 172.21.15.70:0/1711724111 learned_addr learned my addr 172.21.15.70:0/1711724111 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddbe3eb640 1 --2- 172.21.15.70:0/1711724111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ef1a0 0x5578641fe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddbd3e9640 1 --2- 172.21.15.70:0/1711724111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ee7e0 0x5578641fb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddbdbea640 1 -- 172.21.15.70:0/1711724111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ef1a0 msgr2=0x5578641fe2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddbdbea640 1 --2- 172.21.15.70:0/1711724111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ef1a0 0x5578641fe2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddbdbea640 1 -- 172.21.15.70:0/1711724111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ee7e0 msgr2=0x5578641fb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddbdbea640 1 --2- 172.21.15.70:0/1711724111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ee7e0 0x5578641fb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddbdbea640 1 -- 172.21.15.70:0/1711724111 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578641fe8b0 con 0x5578641ed610 2024-04-01T01:33:30.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddbe3eb640 1 --2- 172.21.15.70:0/1711724111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ef1a0 0x5578641fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:30.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.460+0000 7fddbd3e9640 1 --2- 172.21.15.70:0/1711724111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ee7e0 0x5578641fb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:30.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.461+0000 7fddbdbea640 1 --2- 172.21.15.70:0/1711724111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ed610 0x5578641fb0c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fddb4005780 tx=0x7fddb40119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b845ef300d9f7e15 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.461+0000 7fdda17fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:30.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.461+0000 7fdda17fa640 1 -- 172.21.15.70:0/1711724111 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb403a070 con 0x5578641ed610 2024-04-01T01:33:30.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.461+0000 7fdda17fa640 1 -- 172.21.15.70:0/1711724111 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fddb401cbb0 con 0x5578641ed610 2024-04-01T01:33:30.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.461+0000 7fddc2a35740 1 -- 172.21.15.70:0/1711724111 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddb0045070 con 0x5578641ed610 2024-04-01T01:33:30.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.461+0000 7fdda17fa640 1 -- 172.21.15.70:0/1711724111 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fddb4045050 con 0x5578641ed610 2024-04-01T01:33:30.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.461+0000 7fddc2a35740 10 client.15483 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:30.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.461+0000 7fddc2a35740 1 -- 172.21.15.70:0/1711724111 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55786426a9b0 con 0x5578641ed610 2024-04-01T01:33:30.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.461+0000 7fdda17fa640 1 -- 172.21.15.70:0/1711724111 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fddb0045070 con 0x5578641ed610 2024-04-01T01:33:30.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.461+0000 7fdda17fa640 1 -- 172.21.15.70:0/1711724111 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fddb4023070 con 0x5578641ed610 2024-04-01T01:33:30.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.461+0000 7fddc2a35740 10 client.15483 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:30.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.461+0000 7fddc2a35740 10 client.15483 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-01T01:33:30.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.461+0000 7fddc2a35740 20 client.15483 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:30.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.461+0000 7fddc2a35740 1 --2- 172.21.15.70:0/1711724111 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55786426f300 0x55786426b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.461+0000 7fddc2a35740 4 client.15483 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-01T01:33:30.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.461+0000 7fddc2a35740 1 -- 172.21.15.70:0/1711724111 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "status"}) -- 0x5578640d7a00 con 0x55786426f300 2024-04-01T01:33:30.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fddbd3e9640 1 --2- 172.21.15.70:0/1711724111 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55786426f300 0x55786426b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fdda17fa640 10 client.15483 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:33:30.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fddbd3e9640 1 --2- 172.21.15.70:0/1711724111 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55786426f300 0x55786426b740 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ae927ef1ed514836 server_cookie=f0c9b84d53a2b807 in_seq=0 out_seq=0 2024-04-01T01:33:30.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fdda17fa640 1 -- 172.21.15.70:0/1711724111 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5578640d7a00 con 0x55786426f300 2024-04-01T01:33:30.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fdda17fa640 10 client.15483 handle_command_reply: tid=0 2024-04-01T01:33:30.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fddc2a35740 1 client.15483 shutdown 2024-04-01T01:33:30.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fddc2a35740 1 -- 172.21.15.70:0/1711724111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ed610 msgr2=0x5578641fb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fddc2a35740 1 --2- 172.21.15.70:0/1711724111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ed610 0x5578641fb0c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fddb4005780 tx=0x7fddb40119d0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fddc2a35740 1 -- 172.21.15.70:0/1711724111 shutdown_connections 2024-04-01T01:33:30.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fddc2a35740 1 --2- 172.21.15.70:0/1711724111 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55786426f300 0x55786426b740 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fddc2a35740 1 --2- 172.21.15.70:0/1711724111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578641ef1a0 0x5578641fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fdda17fa640 0 client.15483 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:33:30.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fddc2a35740 1 --2- 172.21.15.70:0/1711724111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578641ee7e0 0x5578641fb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fddc2a35740 1 --2- 172.21.15.70:0/1711724111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578641ed610 0x5578641fb0c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fddc2a35740 1 -- 172.21.15.70:0/1711724111 >> 172.21.15.70:0/1711724111 conn(0x5578641e8a00 msgr2=0x5578641f55d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fddc2a35740 1 -- 172.21.15.70:0/1711724111 shutdown_connections 2024-04-01T01:33:30.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fddc2a35740 1 -- 172.21.15.70:0/1711724111 wait complete. 2024-04-01T01:33:30.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.462+0000 7fddc2a35740 20 client.15483 trim_cache size 0 max 16384 2024-04-01T01:33:30.470 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:33:30.470 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "36577394-efc6-11ee-b647-cb9ed24678a4", 2024-04-01T01:33:30.470 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 1, 2024-04-01T01:33:30.470 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24451, 2024-04-01T01:33:30.470 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-01T01:33:30.470 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-01T01:33:30.470 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-01T01:33:30.470 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 115.85198357900001, 2024-04-01T01:33:30.471 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-04-01T01:33:30.471 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 93, 2024-04-01T01:33:30.471 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 89, 2024-04-01T01:33:30.471 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 147.94199353799999 2024-04-01T01:33:30.471 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:33:30.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.463+0000 7fddbfe4e640 1 -- 172.21.15.70:0/1423493995 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdda40a24a0 msgr2=0x7fdda40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.463+0000 7fddbfe4e640 1 --2- 172.21.15.70:0/1423493995 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdda40a24a0 0x7fdda40a4960 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fddb40095f0 tx=0x7fddb4005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:30.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.463+0000 7fddbfe4e640 1 -- 172.21.15.70:0/1423493995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddb8101850 msgr2=0x7fddb8197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.463+0000 7fddbfe4e640 1 --2- 172.21.15.70:0/1423493995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddb8101850 0x7fddb8197ea0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdda800e9f0 tx=0x7fdda800edf0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.463+0000 7fddbfe4e640 1 -- 172.21.15.70:0/1423493995 shutdown_connections 2024-04-01T01:33:30.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.463+0000 7fddbfe4e640 1 --2- 172.21.15.70:0/1423493995 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdda40a24a0 0x7fdda40a4960 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.463+0000 7fddbfe4e640 1 --2- 172.21.15.70:0/1423493995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddb8102210 0x7fddb819ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.463+0000 7fddbfe4e640 1 --2- 172.21.15.70:0/1423493995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddb8101850 0x7fddb8197ea0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.463+0000 7fddbfe4e640 1 --2- 172.21.15.70:0/1423493995 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fddb8100660 0x7fddb8197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.463+0000 7fddbfe4e640 1 -- 172.21.15.70:0/1423493995 >> 172.21.15.70:0/1423493995 conn(0x7fddb80fbbf0 msgr2=0x7fddb8104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.463+0000 7fddbfe4e640 1 -- 172.21.15.70:0/1423493995 shutdown_connections 2024-04-01T01:33:30.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.463+0000 7fddbfe4e640 1 -- 172.21.15.70:0/1423493995 wait complete. 2024-04-01T01:33:30.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-01T01:33:30.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.576+0000 7fcf2268f640 1 -- 172.21.15.70:0/2057411014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf1c075730 msgr2=0x7fcf1c073b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.576+0000 7fcf2268f640 1 --2- 172.21.15.70:0/2057411014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf1c075730 0x7fcf1c073b80 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c009900 tx=0x7fcf0c02bcd0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.576+0000 7fcf2268f640 1 -- 172.21.15.70:0/2057411014 shutdown_connections 2024-04-01T01:33:30.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.576+0000 7fcf2268f640 1 --2- 172.21.15.70:0/2057411014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf1c109600 0x7fcf1c10baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.576+0000 7fcf2268f640 1 --2- 172.21.15.70:0/2057411014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c074150 0x7fcf1c0745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.576+0000 7fcf2268f640 1 --2- 172.21.15.70:0/2057411014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf1c075730 0x7fcf1c073b80 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.576+0000 7fcf2268f640 1 -- 172.21.15.70:0/2057411014 >> 172.21.15.70:0/2057411014 conn(0x7fcf1c0fcd00 msgr2=0x7fcf1c0ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.576+0000 7fcf2268f640 1 -- 172.21.15.70:0/2057411014 shutdown_connections 2024-04-01T01:33:30.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.576+0000 7fcf2268f640 1 -- 172.21.15.70:0/2057411014 wait complete. 2024-04-01T01:33:30.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.576+0000 7fcf2268f640 1 Processor -- start 2024-04-01T01:33:30.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.576+0000 7fcf2268f640 1 -- start start 2024-04-01T01:33:30.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf2268f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf1c074150 0x7fcf1c199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf2268f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c075730 0x7fcf1c19a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf2268f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf1c109600 0x7fcf1c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf2268f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c10f470 con 0x7fcf1c109600 2024-04-01T01:33:30.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf2268f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c10f2f0 con 0x7fcf1c074150 2024-04-01T01:33:30.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf2268f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c10f5f0 con 0x7fcf1c075730 2024-04-01T01:33:30.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf20c2c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf1c109600 0x7fcf1c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf20c2c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf1c109600 0x7fcf1c1a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33152/0 (socket says 172.21.15.70:33152) 2024-04-01T01:33:30.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf20c2c640 1 -- 172.21.15.70:0/3823094203 learned_addr learned my addr 172.21.15.70:0/3823094203 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf1b7fe640 1 --2- 172.21.15.70:0/3823094203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c075730 0x7fcf1c19a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf1bfff640 1 --2- 172.21.15.70:0/3823094203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf1c074150 0x7fcf1c199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf20c2c640 1 -- 172.21.15.70:0/3823094203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c075730 msgr2=0x7fcf1c19a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf20c2c640 1 --2- 172.21.15.70:0/3823094203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c075730 0x7fcf1c19a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf20c2c640 1 -- 172.21.15.70:0/3823094203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf1c074150 msgr2=0x7fcf1c199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf20c2c640 1 --2- 172.21.15.70:0/3823094203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf1c074150 0x7fcf1c199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf20c2c640 1 -- 172.21.15.70:0/3823094203 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1c1a1840 con 0x7fcf1c109600 2024-04-01T01:33:30.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf1b7fe640 1 --2- 172.21.15.70:0/3823094203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c075730 0x7fcf1c19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:30.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf20c2c640 1 --2- 172.21.15.70:0/3823094203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf1c109600 0x7fcf1c1a11a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400eb10 tx=0x7fcf1400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf197fa640 1 -- 172.21.15.70:0/3823094203 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf14004b20 con 0x7fcf1c109600 2024-04-01T01:33:30.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf2268f640 1 -- 172.21.15.70:0/3823094203 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf1c1a1b30 con 0x7fcf1c109600 2024-04-01T01:33:30.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf197fa640 1 -- 172.21.15.70:0/3823094203 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf14004cc0 con 0x7fcf1c109600 2024-04-01T01:33:30.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf197fa640 1 -- 172.21.15.70:0/3823094203 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf14010690 con 0x7fcf1c109600 2024-04-01T01:33:30.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.577+0000 7fcf2268f640 1 -- 172.21.15.70:0/3823094203 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf1c1a2080 con 0x7fcf1c109600 2024-04-01T01:33:30.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf197fa640 1 -- 172.21.15.70:0/3823094203 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fcf14005020 con 0x7fcf1c109600 2024-04-01T01:33:30.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf2526d740 1 Processor -- start 2024-04-01T01:33:30.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf2526d740 1 -- start start 2024-04-01T01:33:30.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf2526d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadc610 0x55c1ccadca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf2526d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccadd7e0 0x55c1ccaddc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf2526d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccade1a0 0x55c1ccae4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf2526d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1cca40af0 con 0x55c1ccadc610 2024-04-01T01:33:30.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf2526d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c1cc9e99d0 con 0x55c1ccade1a0 2024-04-01T01:33:30.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf2526d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c1cca3cb90 con 0x55c1ccadd7e0 2024-04-01T01:33:30.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf1b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadc610 0x55c1ccadca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf1b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadc610 0x55c1ccadca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33162/0 (socket says 172.21.15.70:33162) 2024-04-01T01:33:30.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf1b7fe640 1 -- 172.21.15.70:0/1877930637 learned_addr learned my addr 172.21.15.70:0/1877930637 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf1bfff640 1 --2- 172.21.15.70:0/1877930637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccadd7e0 0x55c1ccaddc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf197fa640 1 --2- 172.21.15.70:0/3823094203 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcf040a24a0 0x7fcf040a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf1b7fe640 1 --2- 172.21.15.70:0/1877930637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccade1a0 0x55c1ccae4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf20c2c640 1 --2- 172.21.15.70:0/3823094203 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcf040a24a0 0x7fcf040a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf1b7fe640 1 -- 172.21.15.70:0/1877930637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccadd7e0 msgr2=0x55c1ccaddc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf1b7fe640 1 --2- 172.21.15.70:0/1877930637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccadd7e0 0x55c1ccaddc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf1b7fe640 1 -- 172.21.15.70:0/1877930637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccade1a0 msgr2=0x55c1ccae4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf1b7fe640 1 --2- 172.21.15.70:0/1877930637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccade1a0 0x55c1ccae4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf1b7fe640 1 -- 172.21.15.70:0/1877930637 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1cc7552d0 con 0x55c1ccadc610 2024-04-01T01:33:30.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.580+0000 7fcf197fa640 1 -- 172.21.15.70:0/3823094203 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fcf140cf110 con 0x7fcf1c109600 2024-04-01T01:33:30.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.581+0000 7fcf1b7fe640 1 --2- 172.21.15.70:0/1877930637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadc610 0x55c1ccadca30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fcf100095f0 tx=0x7fcf10009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa56104c673bae88 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.581+0000 7fcf027fc640 1 -- 172.21.15.70:0/1877930637 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf10004230 con 0x55c1ccadc610 2024-04-01T01:33:30.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.581+0000 7fcf027fc640 1 -- 172.21.15.70:0/1877930637 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf10011e20 con 0x55c1ccadc610 2024-04-01T01:33:30.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.581+0000 7fcf027fc640 1 -- 172.21.15.70:0/1877930637 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf10003e10 con 0x55c1ccadc610 2024-04-01T01:33:30.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.581+0000 7fcf2526d740 1 -- 172.21.15.70:0/1877930637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadc610 msgr2=0x55c1ccadca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.581+0000 7fcf20c2c640 1 --2- 172.21.15.70:0/3823094203 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcf040a24a0 0x7fcf040a4960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcf1409b790 tx=0x7fcf1406a6f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.581+0000 7fcf2526d740 1 --2- 172.21.15.70:0/1877930637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadc610 0x55c1ccadca30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fcf100095f0 tx=0x7fcf10009cb0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.581+0000 7fcf2526d740 1 -- 172.21.15.70:0/1877930637 shutdown_connections 2024-04-01T01:33:30.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.581+0000 7fcf2526d740 1 --2- 172.21.15.70:0/1877930637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccade1a0 0x55c1ccae4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.581+0000 7fcf2526d740 1 --2- 172.21.15.70:0/1877930637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccadd7e0 0x55c1ccaddc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.581+0000 7fcf2526d740 1 --2- 172.21.15.70:0/1877930637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadc610 0x55c1ccadca30 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.581+0000 7fcf2526d740 1 -- 172.21.15.70:0/1877930637 >> 172.21.15.70:0/1877930637 conn(0x55c1ccad7a00 msgr2=0x55c1ccad7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.581+0000 7fcf2526d740 1 -- 172.21.15.70:0/1877930637 shutdown_connections 2024-04-01T01:33:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.581+0000 7fcf2526d740 1 -- 172.21.15.70:0/1877930637 wait complete. 2024-04-01T01:33:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.582+0000 7fcf2526d740 1 -- start start 2024-04-01T01:33:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.582+0000 7fcf2526d740 1 Processor -- start 2024-04-01T01:33:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.582+0000 7fcf2526d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccadc610 0x55c1ccae6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.582+0000 7fcf2526d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccae6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.582+0000 7fcf2526d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccade1a0 0x55c1ccaedef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.582+0000 7fcf2526d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1ccae51b0 con 0x55c1ccadd7e0 2024-04-01T01:33:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.582+0000 7fcf2526d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c1cc9fcf70 con 0x55c1ccadc610 2024-04-01T01:33:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.582+0000 7fcf2526d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c1ccae5330 con 0x55c1ccade1a0 2024-04-01T01:33:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.582+0000 7fcf2526d740 10 client.0 _ll_register_callbacks cb 0x55c1cc754ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.582+0000 7fcf1bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccae6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.582+0000 7fcf1bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccae6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33168/0 (socket says 172.21.15.70:33168) 2024-04-01T01:33:30.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.582+0000 7fcf1bfff640 1 -- 172.21.15.70:0/4140055265 learned_addr learned my addr 172.21.15.70:0/4140055265 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.582+0000 7fcf1b7fe640 1 --2- 172.21.15.70:0/4140055265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccade1a0 0x55c1ccaedef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.582+0000 7fcf1b7fe640 1 --2- 172.21.15.70:0/4140055265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccadc610 0x55c1ccae6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcf1bfff640 1 -- 172.21.15.70:0/4140055265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccade1a0 msgr2=0x55c1ccaedef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcf1bfff640 1 --2- 172.21.15.70:0/4140055265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccade1a0 0x55c1ccaedef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcf1bfff640 1 -- 172.21.15.70:0/4140055265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccadc610 msgr2=0x55c1ccae6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcf1bfff640 1 --2- 172.21.15.70:0/4140055265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccadc610 0x55c1ccae6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcf1bfff640 1 -- 172.21.15.70:0/4140055265 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1cc81f6b0 con 0x55c1ccadd7e0 2024-04-01T01:33:30.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcf1bfff640 1 --2- 172.21.15.70:0/4140055265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccae6eb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c009230 tx=0x7fcf0c02dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cee253f31e6501f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcefb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:30.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcefb7fe640 1 -- 172.21.15.70:0/4140055265 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c044030 con 0x55c1ccadd7e0 2024-04-01T01:33:30.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcefb7fe640 1 -- 172.21.15.70:0/4140055265 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fcf0c03a040 con 0x55c1ccadd7e0 2024-04-01T01:33:30.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcf2526d740 1 -- 172.21.15.70:0/4140055265 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf10011e20 con 0x55c1ccadd7e0 2024-04-01T01:33:30.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcefb7fe640 1 -- 172.21.15.70:0/4140055265 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcf0c051050 con 0x55c1ccadd7e0 2024-04-01T01:33:30.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcf2526d740 10 client.15495 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:30.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcf2526d740 1 -- 172.21.15.70:0/4140055265 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c1ccaf0fc0 con 0x55c1ccadd7e0 2024-04-01T01:33:30.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcefb7fe640 1 -- 172.21.15.70:0/4140055265 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf10011e20 con 0x55c1ccadd7e0 2024-04-01T01:33:30.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcefb7fe640 1 -- 172.21.15.70:0/4140055265 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcf0c0035f0 con 0x55c1ccadd7e0 2024-04-01T01:33:30.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcf2526d740 10 client.15495 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:30.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcf2526d740 10 client.15495 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-01T01:33:30.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcf2526d740 20 client.15495 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:30.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcf2526d740 1 --2- 172.21.15.70:0/4140055265 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c1ccaf36c0 0x55c1ccaf4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcf2526d740 4 client.15495 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:33:30.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.583+0000 7fcf2526d740 1 -- 172.21.15.70:0/4140055265 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1cca4e030 con 0x55c1ccaf36c0 2024-04-01T01:33:30.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.584+0000 7fcf1b7fe640 1 --2- 172.21.15.70:0/4140055265 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c1ccaf36c0 0x55c1ccaf4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.584+0000 7fcefb7fe640 10 client.15495 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:30.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.584+0000 7fcf1b7fe640 1 --2- 172.21.15.70:0/4140055265 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c1ccaf36c0 0x55c1ccaf4b30 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5d78424d8ede969 server_cookie=788d5d3a2f93a650 in_seq=0 out_seq=0 2024-04-01T01:33:30.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.585+0000 7fcefb7fe640 1 -- 172.21.15.70:0/4140055265 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c1cca4e030 con 0x55c1ccaf36c0 2024-04-01T01:33:30.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.585+0000 7fcefb7fe640 10 client.15495 handle_command_reply: tid=0 2024-04-01T01:33:30.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.585+0000 7fcf2526d740 1 client.15495 shutdown 2024-04-01T01:33:30.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.585+0000 7fcf2526d740 1 -- 172.21.15.70:0/4140055265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 msgr2=0x55c1ccae6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.585+0000 7fcf2526d740 1 --2- 172.21.15.70:0/4140055265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccae6eb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c009230 tx=0x7fcf0c02dbc0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.585+0000 7fcf2526d740 1 -- 172.21.15.70:0/4140055265 shutdown_connections 2024-04-01T01:33:30.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.585+0000 7fcf2526d740 1 --2- 172.21.15.70:0/4140055265 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c1ccaf36c0 0x55c1ccaf4b30 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.585+0000 7fcf2526d740 1 --2- 172.21.15.70:0/4140055265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccade1a0 0x55c1ccaedef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.585+0000 7fcefb7fe640 0 client.15495 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:30.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.585+0000 7fcf2526d740 1 --2- 172.21.15.70:0/4140055265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccae6eb0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.585+0000 7fcf2526d740 1 --2- 172.21.15.70:0/4140055265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccadc610 0x55c1ccae6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.585+0000 7fcf2526d740 1 -- 172.21.15.70:0/4140055265 >> 172.21.15.70:0/4140055265 conn(0x55c1ccad7a00 msgr2=0x55c1ccad6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.585+0000 7fcf2526d740 1 -- 172.21.15.70:0/4140055265 shutdown_connections 2024-04-01T01:33:30.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.585+0000 7fcf2526d740 1 -- 172.21.15.70:0/4140055265 wait complete. 2024-04-01T01:33:30.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.585+0000 7fcf2526d740 20 client.15495 trim_cache size 0 max 16384 2024-04-01T01:33:30.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf2526d740 1 Processor -- start 2024-04-01T01:33:30.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf2526d740 1 -- start start 2024-04-01T01:33:30.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf2526d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccadc610 0x55c1ccae4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf2526d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccaedeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf2526d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccade1a0 0x55c1ccae6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf2526d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1ccaf2e70 con 0x55c1ccadd7e0 2024-04-01T01:33:30.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf2526d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c1ccaf3540 con 0x55c1ccade1a0 2024-04-01T01:33:30.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf2526d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c1ccaf3c10 con 0x55c1ccadc610 2024-04-01T01:33:30.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf1bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccaedeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf1bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccaedeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33178/0 (socket says 172.21.15.70:33178) 2024-04-01T01:33:30.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf1bfff640 1 -- 172.21.15.70:0/319444009 learned_addr learned my addr 172.21.15.70:0/319444009 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf1b7fe640 1 --2- 172.21.15.70:0/319444009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccadc610 0x55c1ccae4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf1b7fe640 1 --2- 172.21.15.70:0/319444009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccade1a0 0x55c1ccae6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf1bfff640 1 -- 172.21.15.70:0/319444009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccadc610 msgr2=0x55c1ccae4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf1bfff640 1 --2- 172.21.15.70:0/319444009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccadc610 0x55c1ccae4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf1bfff640 1 -- 172.21.15.70:0/319444009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccade1a0 msgr2=0x55c1ccae6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf1bfff640 1 --2- 172.21.15.70:0/319444009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccade1a0 0x55c1ccae6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.603+0000 7fcf1bfff640 1 -- 172.21.15.70:0/319444009 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1ccaf0de0 con 0x55c1ccadd7e0 2024-04-01T01:33:30.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.604+0000 7fcf1bfff640 1 --2- 172.21.15.70:0/319444009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccaedeb0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c003a30 tx=0x7fcf0c02bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72cd4988bbfa1a9b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.604+0000 7fcf01ffb640 1 -- 172.21.15.70:0/319444009 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c044070 con 0x55c1ccadd7e0 2024-04-01T01:33:30.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.604+0000 7fcf01ffb640 1 -- 172.21.15.70:0/319444009 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf0c004b70 con 0x55c1ccadd7e0 2024-04-01T01:33:30.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.604+0000 7fcf01ffb640 1 -- 172.21.15.70:0/319444009 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c003e80 con 0x55c1ccadd7e0 2024-04-01T01:33:30.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.604+0000 7fcf2526d740 1 -- 172.21.15.70:0/319444009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 msgr2=0x55c1ccaedeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.604+0000 7fcf2526d740 1 --2- 172.21.15.70:0/319444009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccaedeb0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c003a30 tx=0x7fcf0c02bcd0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.604+0000 7fcf2526d740 1 -- 172.21.15.70:0/319444009 shutdown_connections 2024-04-01T01:33:30.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.604+0000 7fcf2526d740 1 --2- 172.21.15.70:0/319444009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccade1a0 0x55c1ccae6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.604+0000 7fcf2526d740 1 --2- 172.21.15.70:0/319444009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccaedeb0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.604+0000 7fcf2526d740 1 --2- 172.21.15.70:0/319444009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccadc610 0x55c1ccae4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.604+0000 7fcf2526d740 1 -- 172.21.15.70:0/319444009 >> 172.21.15.70:0/319444009 conn(0x55c1ccad7a00 msgr2=0x55c1ccad7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.604+0000 7fcf2526d740 1 -- 172.21.15.70:0/319444009 shutdown_connections 2024-04-01T01:33:30.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.604+0000 7fcf2526d740 1 -- 172.21.15.70:0/319444009 wait complete. 2024-04-01T01:33:30.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf2526d740 1 -- start start 2024-04-01T01:33:30.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf2526d740 1 Processor -- start 2024-04-01T01:33:30.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf2526d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccadc610 0x55c1ccae9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf2526d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccaea0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf2526d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccade1a0 0x55c1ccaed100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf2526d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1ccaf49b0 con 0x55c1ccadd7e0 2024-04-01T01:33:30.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf2526d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c1ccaf42e0 con 0x55c1ccadc610 2024-04-01T01:33:30.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf2526d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c1ccaf5080 con 0x55c1ccade1a0 2024-04-01T01:33:30.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf2526d740 10 client.0 _ll_register_callbacks cb 0x7fcf0c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:30.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf1bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccaea0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf1b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccadc610 0x55c1ccae9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf1bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccaea0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33188/0 (socket says 172.21.15.70:33188) 2024-04-01T01:33:30.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf1bfff640 1 -- 172.21.15.70:0/2699009076 learned_addr learned my addr 172.21.15.70:0/2699009076 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:30.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf1b7fe640 1 --2- 172.21.15.70:0/2699009076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccade1a0 0x55c1ccaed100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf1bfff640 1 -- 172.21.15.70:0/2699009076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccade1a0 msgr2=0x55c1ccaed100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf1bfff640 1 --2- 172.21.15.70:0/2699009076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccade1a0 0x55c1ccaed100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf1bfff640 1 -- 172.21.15.70:0/2699009076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccadc610 msgr2=0x55c1ccae9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:30.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.605+0000 7fcf1bfff640 1 --2- 172.21.15.70:0/2699009076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccadc610 0x55c1ccae9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcf1bfff640 1 -- 172.21.15.70:0/2699009076 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1ccaed6f0 con 0x55c1ccadd7e0 2024-04-01T01:33:30.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcf1b7fe640 1 --2- 172.21.15.70:0/2699009076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccadc610 0x55c1ccae9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:30.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcf1bfff640 1 --2- 172.21.15.70:0/2699009076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccaea0e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c02c210 tx=0x7fcf0c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56a4de49699dcf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:30.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcefb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:30.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcefb7fe640 1 -- 172.21.15.70:0/2699009076 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c062060 con 0x55c1ccadd7e0 2024-04-01T01:33:30.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcefb7fe640 1 -- 172.21.15.70:0/2699009076 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fcf0c0046c0 con 0x55c1ccadd7e0 2024-04-01T01:33:30.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcf2526d740 1 -- 172.21.15.70:0/2699009076 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c1cca48380 con 0x55c1ccadd7e0 2024-04-01T01:33:30.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcefb7fe640 1 -- 172.21.15.70:0/2699009076 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcf0c069050 con 0x55c1ccadd7e0 2024-04-01T01:33:30.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcf2526d740 10 client.15501 fetch_fsmap learned FSMap version 28 2024-04-01T01:33:30.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcf2526d740 1 -- 172.21.15.70:0/2699009076 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c1ccb59a30 con 0x55c1ccadd7e0 2024-04-01T01:33:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcefb7fe640 1 -- 172.21.15.70:0/2699009076 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55c1cca48380 con 0x55c1ccadd7e0 2024-04-01T01:33:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcefb7fe640 1 -- 172.21.15.70:0/2699009076 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcf0c0404e0 con 0x55c1ccadd7e0 2024-04-01T01:33:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcf2526d740 10 client.15501 fetch_fsmap finished waiting for FSMap version 28 2024-04-01T01:33:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcf2526d740 10 client.15501 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-01T01:33:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcf2526d740 20 client.15501 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcf2526d740 1 --2- 172.21.15.70:0/2699009076 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c1ccb5e280 0x55c1ccb5cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcf2526d740 4 client.15501 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-01T01:33:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.606+0000 7fcf2526d740 1 -- 172.21.15.70:0/2699009076 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "status"}) -- 0x55c1cc9c6a00 con 0x55c1ccb5e280 2024-04-01T01:33:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcf1b7fe640 1 --2- 172.21.15.70:0/2699009076 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c1ccb5e280 0x55c1ccb5cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcefb7fe640 10 client.15501 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcf1b7fe640 1 --2- 172.21.15.70:0/2699009076 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c1ccb5e280 0x55c1ccb5cf90 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eefd06260f4d0043 server_cookie=4c2b0786894b551b in_seq=0 out_seq=0 2024-04-01T01:33:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcefb7fe640 1 -- 172.21.15.70:0/2699009076 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55c1cc9c6a00 con 0x55c1ccb5e280 2024-04-01T01:33:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcefb7fe640 10 client.15501 handle_command_reply: tid=0 2024-04-01T01:33:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcf2526d740 1 client.15501 shutdown 2024-04-01T01:33:30.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcf2526d740 1 -- 172.21.15.70:0/2699009076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 msgr2=0x55c1ccaea0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcf2526d740 1 --2- 172.21.15.70:0/2699009076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccaea0e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c02c210 tx=0x7fcf0c005a20 comp rx=0 tx=0).stop 2024-04-01T01:33:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcf2526d740 1 -- 172.21.15.70:0/2699009076 shutdown_connections 2024-04-01T01:33:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcf2526d740 1 --2- 172.21.15.70:0/2699009076 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c1ccb5e280 0x55c1ccb5cf90 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcf2526d740 1 --2- 172.21.15.70:0/2699009076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1ccade1a0 0x55c1ccaed100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcefb7fe640 0 client.15501 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcf2526d740 1 --2- 172.21.15.70:0/2699009076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1ccadd7e0 0x55c1ccaea0e0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcf2526d740 1 --2- 172.21.15.70:0/2699009076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1ccadc610 0x55c1ccae9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcf2526d740 1 -- 172.21.15.70:0/2699009076 >> 172.21.15.70:0/2699009076 conn(0x55c1ccad7a00 msgr2=0x55c1ccae4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcf2526d740 1 -- 172.21.15.70:0/2699009076 shutdown_connections 2024-04-01T01:33:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcf2526d740 1 -- 172.21.15.70:0/2699009076 wait complete. 2024-04-01T01:33:30.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.607+0000 7fcf2526d740 20 client.15501 trim_cache size 0 max 16384 2024-04-01T01:33:30.615 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:33:30.615 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "36577394-efc6-11ee-b647-cb9ed24678a4", 2024-04-01T01:33:30.615 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-04-01T01:33:30.615 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 24464, 2024-04-01T01:33:30.615 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-01T01:33:30.615 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-01T01:33:30.615 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-01T01:33:30.615 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 134.07351785, 2024-04-01T01:33:30.615 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 28, 2024-04-01T01:33:30.615 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 93, 2024-04-01T01:33:30.615 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 89, 2024-04-01T01:33:30.615 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 139.57701085799999 2024-04-01T01:33:30.615 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:33:30.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.608+0000 7fcf2268f640 1 -- 172.21.15.70:0/3823094203 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcf040a24a0 msgr2=0x7fcf040a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.608+0000 7fcf2268f640 1 --2- 172.21.15.70:0/3823094203 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcf040a24a0 0x7fcf040a4960 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcf1409b790 tx=0x7fcf1406a6f0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.608+0000 7fcf2268f640 1 -- 172.21.15.70:0/3823094203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf1c109600 msgr2=0x7fcf1c1a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:30.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.608+0000 7fcf2268f640 1 --2- 172.21.15.70:0/3823094203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf1c109600 0x7fcf1c1a11a0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400eb10 tx=0x7fcf1400c2d0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.608+0000 7fcf2268f640 1 -- 172.21.15.70:0/3823094203 shutdown_connections 2024-04-01T01:33:30.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.608+0000 7fcf2268f640 1 --2- 172.21.15.70:0/3823094203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf1c109600 0x7fcf1c1a11a0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.608+0000 7fcf2268f640 1 --2- 172.21.15.70:0/3823094203 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcf040a24a0 0x7fcf040a4960 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.608+0000 7fcf2268f640 1 --2- 172.21.15.70:0/3823094203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c075730 0x7fcf1c19a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.608+0000 7fcf2268f640 1 --2- 172.21.15.70:0/3823094203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf1c074150 0x7fcf1c199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:30.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.608+0000 7fcf2268f640 1 -- 172.21.15.70:0/3823094203 >> 172.21.15.70:0/3823094203 conn(0x7fcf1c0fcd00 msgr2=0x7fcf1c105590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:30.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.608+0000 7fcf2268f640 1 -- 172.21.15.70:0/3823094203 shutdown_connections 2024-04-01T01:33:30.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:30.608+0000 7fcf2268f640 1 -- 172.21.15.70:0/3823094203 wait complete. 2024-04-01T01:33:30.619 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-01T01:33:30.630 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-01T01:33:30.631 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-04-01T01:33:30.631 INFO:tasks.ceph_fuse:client.0 config is {'mntopts': ['context=system_u:object_r:postgresql_db_t:s0'], 'mount_subvol_num': 0, 'mountpoint': '/tmp/cephfs'} 2024-04-01T01:33:30.631 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-01T01:33:30.631 INFO:tasks.cephfs.mount:using client_config["hostfs_mntpt"] = /tmp/cephfs 2024-04-01T01:33:30.631 INFO:tasks.cephfs.mount:hostfs_mntpt = /tmp/cephfs 2024-04-01T01:33:30.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:33:30.631 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-01T01:33:30.646 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-01T01:33:30.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:33:30.647 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-01T01:33:30.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:33:30.715 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link delete ceph-brx 2024-04-01T01:33:30.725 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:30 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/801213178' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:30.783 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-01T01:33:30.783 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph fs ls 2024-04-01T01:33:30.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:30 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/801213178' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:31.026 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:33:31.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:30 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/801213178' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:31.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:31 smithi189 ceph-mon[35753]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-04-01T01:33:32.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:31 smithi070 ceph-mon[32179]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-04-01T01:33:32.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:31 smithi081 ceph-mon[35142]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-04-01T01:33:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.711+0000 7f0206d77700 1 -- 172.21.15.70:0/780536546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0200107200 msgr2=0x7f020006a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.711+0000 7f0206d77700 1 --2- 172.21.15.70:0/780536546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0200107200 0x7f020006a9a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009990 tx=0x7f01f4009ca0 comp rx=0 tx=0).stop 2024-04-01T01:33:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.711+0000 7f0206d77700 1 -- 172.21.15.70:0/780536546 shutdown_connections 2024-04-01T01:33:32.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.711+0000 7f0206d77700 1 --2- 172.21.15.70:0/780536546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0200104c70 0x7f02001050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.711+0000 7f0206d77700 1 --2- 172.21.15.70:0/780536546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0200107200 0x7f020006a9a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.711+0000 7f0206d77700 1 --2- 172.21.15.70:0/780536546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f020006aee0 0x7f020006b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.711+0000 7f0206d77700 1 -- 172.21.15.70:0/780536546 >> 172.21.15.70:0/780536546 conn(0x7f0200077150 msgr2=0x7f0200077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.711+0000 7f0206d77700 1 -- 172.21.15.70:0/780536546 shutdown_connections 2024-04-01T01:33:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.711+0000 7f0206d77700 1 -- 172.21.15.70:0/780536546 wait complete. 2024-04-01T01:33:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.711+0000 7f0206d77700 1 Processor -- start 2024-04-01T01:33:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.711+0000 7f0206d77700 1 -- start start 2024-04-01T01:33:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0206d77700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f020006aee0 0x7f020019ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0206d77700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0200104c70 0x7f020019f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0206d77700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0200107200 0x7f02001a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0206d77700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0200118a10 con 0x7f020006aee0 2024-04-01T01:33:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0206d77700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0200118890 con 0x7f0200107200 2024-04-01T01:33:32.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0206d77700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0200118b90 con 0x7f0200104c70 2024-04-01T01:33:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0204b13700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f020006aee0 0x7f020019ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0204b13700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f020006aee0 0x7f020019ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33218/0 (socket says 172.21.15.70:33218) 2024-04-01T01:33:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0204b13700 1 -- 172.21.15.70:0/1700155165 learned_addr learned my addr 172.21.15.70:0/1700155165 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0205314700 1 --2- 172.21.15.70:0/1700155165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0200107200 0x7f02001a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f01fffff700 1 --2- 172.21.15.70:0/1700155165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0200104c70 0x7f020019f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0204b13700 1 -- 172.21.15.70:0/1700155165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0200104c70 msgr2=0x7f020019f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0204b13700 1 --2- 172.21.15.70:0/1700155165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0200104c70 0x7f020019f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0204b13700 1 -- 172.21.15.70:0/1700155165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0200107200 msgr2=0x7f02001a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0204b13700 1 --2- 172.21.15.70:0/1700155165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0200107200 0x7f02001a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0204b13700 1 -- 172.21.15.70:0/1700155165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02001a3e90 con 0x7f020006aee0 2024-04-01T01:33:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0204b13700 1 --2- 172.21.15.70:0/1700155165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f020006aee0 0x7f020019ee80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f01f4005910 tx=0x7f01f4009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f01fdffb700 1 -- 172.21.15.70:0/1700155165 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f4027070 con 0x7f020006aee0 2024-04-01T01:33:32.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f01fdffb700 1 -- 172.21.15.70:0/1700155165 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f01f4022470 con 0x7f020006aee0 2024-04-01T01:33:32.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f0206d77700 1 -- 172.21.15.70:0/1700155165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02001a4120 con 0x7f020006aee0 2024-04-01T01:33:32.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.712+0000 7f01fdffb700 1 -- 172.21.15.70:0/1700155165 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f400b5b0 con 0x7f020006aee0 2024-04-01T01:33:32.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.713+0000 7f0206d77700 1 -- 172.21.15.70:0/1700155165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02001aba00 con 0x7f020006aee0 2024-04-01T01:33:32.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.714+0000 7f0206d77700 1 -- 172.21.15.70:0/1700155165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0200101750 con 0x7f020006aee0 2024-04-01T01:33:32.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.715+0000 7f01fdffb700 1 -- 172.21.15.70:0/1700155165 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f01f402a070 con 0x7f020006aee0 2024-04-01T01:33:32.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.715+0000 7f01fdffb700 1 --2- 172.21.15.70:0/1700155165 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f01f00a2530 0x7f01f00a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:32.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.715+0000 7f01fffff700 1 --2- 172.21.15.70:0/1700155165 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f01f00a2530 0x7f01f00a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:32.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.715+0000 7f01fdffb700 1 -- 172.21.15.70:0/1700155165 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f01f40cf6f0 con 0x7f020006aee0 2024-04-01T01:33:32.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.716+0000 7f01fffff700 1 --2- 172.21.15.70:0/1700155165 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f01f00a2530 0x7f01f00a49f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0200077910 tx=0x7f01ec009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:32.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.720+0000 7f01fdffb700 1 -- 172.21.15.70:0/1700155165 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f01f4013070 con 0x7f020006aee0 2024-04-01T01:33:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.962+0000 7f0206d77700 1 -- 172.21.15.70:0/1700155165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f02001abbd0 con 0x7f020006aee0 2024-04-01T01:33:32.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.963+0000 7f01fdffb700 1 -- 172.21.15.70:0/1700155165 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+88 (secure 0 0 0) 0x7f01f409c570 con 0x7f020006aee0 2024-04-01T01:33:32.963 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-01T01:33:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.965+0000 7f0206d77700 1 -- 172.21.15.70:0/1700155165 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f01f00a2530 msgr2=0x7f01f00a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.965+0000 7f0206d77700 1 --2- 172.21.15.70:0/1700155165 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f01f00a2530 0x7f01f00a49f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0200077910 tx=0x7f01ec009260 comp rx=0 tx=0).stop 2024-04-01T01:33:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.965+0000 7f0206d77700 1 -- 172.21.15.70:0/1700155165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f020006aee0 msgr2=0x7f020019ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:32.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.965+0000 7f0206d77700 1 --2- 172.21.15.70:0/1700155165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f020006aee0 0x7f020019ee80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f01f4005910 tx=0x7f01f4009570 comp rx=0 tx=0).stop 2024-04-01T01:33:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.965+0000 7f0206d77700 1 -- 172.21.15.70:0/1700155165 shutdown_connections 2024-04-01T01:33:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.965+0000 7f0206d77700 1 --2- 172.21.15.70:0/1700155165 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f01f00a2530 0x7f01f00a49f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.965+0000 7f0206d77700 1 --2- 172.21.15.70:0/1700155165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0200107200 0x7f02001a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.965+0000 7f0206d77700 1 --2- 172.21.15.70:0/1700155165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f020006aee0 0x7f020019ee80 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.965+0000 7f0206d77700 1 --2- 172.21.15.70:0/1700155165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0200104c70 0x7f020019f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.965+0000 7f0206d77700 1 -- 172.21.15.70:0/1700155165 >> 172.21.15.70:0/1700155165 conn(0x7f0200077150 msgr2=0x7f0200102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.965+0000 7f0206d77700 1 -- 172.21.15.70:0/1700155165 shutdown_connections 2024-04-01T01:33:32.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:32.965+0000 7f0206d77700 1 -- 172.21.15.70:0/1700155165 wait complete. 2024-04-01T01:33:33.675 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-01T01:33:33.675 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-01T01:33:33.675 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-01T01:33:33.675 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-01T01:33:33.675 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /tmp/cephfs 2024-04-01T01:33:33.675 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-01T01:33:33.675 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-01T01:33:33.675 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-01T01:33:33.675 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/tmp/cephfs' 2024-04-01T01:33:33.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:33:33.676 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-01T01:33:33.692 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-01T01:33:33.692 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-01T01:33:33.692 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-01T01:33:33.692 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-01T01:33:33.692 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-01T01:33:33.692 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-01T01:33:33.692 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-01T01:33:33.693 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-01T01:33:33.693 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-01T01:33:33.693 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-01T01:33:33.693 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-01T01:33:33.693 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-01T01:33:33.693 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-01T01:33:33.693 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-01T01:33:33.693 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-01T01:33:33.693 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-01T01:33:33.693 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-01T01:33:33.693 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42059sec preferred_lft 42059sec 2024-04-01T01:33:33.693 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-01T01:33:33.693 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-01T01:33:33.694 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-01T01:33:33.694 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-01T01:33:33.694 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-01T01:33:33.694 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link add name ceph-brx type bridge 2024-04-01T01:33:33.694 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip addr flush dev ceph-brx 2024-04-01T01:33:33.694 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set ceph-brx up 2024-04-01T01:33:33.694 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-01T01:33:33.694 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-01T01:33:33.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-01T01:33:33.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-01T01:33:33.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:33:33.844 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-01T01:33:33.870 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-01T01:33:33.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:33:33.872 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-01T01:33:33.928 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-01T01:33:33.928 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-01T01:33:33.928 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-01T01:33:33.928 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-01T01:33:33.928 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-01T01:33:33.929 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-01T01:33:33.929 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-01T01:33:33.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-01T01:33:33.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-01T01:33:33.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-01T01:33:33.929 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-01T01:33:33.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:33 smithi189 ceph-mon[35753]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-01T01:33:33.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:33 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1700155165' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-01T01:33:34.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:33 smithi070 ceph-mon[32179]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-01T01:33:34.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:33 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1700155165' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-01T01:33:34.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-01T01:33:34.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-01T01:33:34.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:33:34.065 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-01T01:33:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:33 smithi081 ceph-mon[35142]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-01T01:33:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:33 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1700155165' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-01T01:33:34.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:33:34.120 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list-id 2024-04-01T01:33:34.176 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-01T01:33:34.176 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-01T01:33:34.176 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns add ceph-ns--tmp-cephfs 2024-04-01T01:33:34.176 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns set ceph-ns--tmp-cephfs 0 2024-04-01T01:33:34.176 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-01T01:33:34.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-01T01:33:34.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-01T01:33:34.298 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--tmp-cephfs' with 192.168.0.1/16 2024-04-01T01:33:34.299 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-01T01:33:34.299 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-01T01:33:34.299 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link add veth0 netns ceph-ns--tmp-cephfs type veth peer name brx.0 2024-04-01T01:33:34.299 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--tmp-cephfs ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-01T01:33:34.299 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set veth0 up 2024-04-01T01:33:34.299 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set lo up 2024-04-01T01:33:34.299 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--tmp-cephfs ip route add default via 192.168.255.254 2024-04-01T01:33:34.299 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-01T01:33:34.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-01T01:33:34.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-01T01:33:34.424 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-01T01:33:34.424 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-01T01:33:34.424 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.0 up 2024-04-01T01:33:34.424 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set dev brx.0 master ceph-brx 2024-04-01T01:33:34.425 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-01T01:33:34.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-01T01:33:34.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-01T01:33:34.507 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mntopts': ['context=system_u:object_r:postgresql_db_t:s0'], 'mount_subvol_num': 0, 'mountpoint': '/tmp/cephfs'} 2024-04-01T01:33:34.507 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-01T01:33:34.507 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /tmp/cephfs 2024-04-01T01:33:34.521 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/tmp/cephfs' 2024-04-01T01:33:34.522 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-01T01:33:34.522 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /tmp/cephfs 2024-04-01T01:33:34.577 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-01T01:33:34.646 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-01T01:33:34.702 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-01T01:33:34.703 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-01T01:33:34.703 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-01T01:33:34.703 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-01T01:33:34.703 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-01T01:33:34.703 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-01T01:33:34.703 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-01T01:33:34.704 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-01T01:33:34.704 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-01T01:33:34.704 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-01T01:33:34.704 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-01T01:33:34.704 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-01T01:33:34.704 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-01T01:33:34.705 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-01T01:33:34.705 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-01T01:33:34.705 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-01T01:33:34.705 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-01T01:33:34.705 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-01T01:33:34.705 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-01T01:33:34.705 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-01T01:33:34.706 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-01T01:33:34.706 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-01T01:33:34.706 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-01T01:33:34.706 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-01T01:33:34.706 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-01T01:33:34.706 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay 2024-04-01T01:33:34.707 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/585d4bb1e6cc6a226b096b7a454003dd2f25e52839ed75bbf063895bf9fd7647/merged 2024-04-01T01:33:34.707 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/363fe581d6f063986062f6da7b8ee4000839e42571fe570e38d6fbb8793740c8/merged 2024-04-01T01:33:34.707 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/0 2024-04-01T01:33:34.707 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-01T01:33:34.707 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/a12405395bb3d2147702e3fb40342d6133652c8521fdab4a47bc78ac3ac29f06/merged 2024-04-01T01:33:34.707 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/89ebda32a13bd3ddd2f34a1c254a0f6a4c573018cd0955b5136df44d8dffad5d/merged 2024-04-01T01:33:34.708 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/12b865e10303128d42ed6487fe430ce5108439f3bf4e03f50c67ec7752b3d01a/merged 2024-04-01T01:33:34.708 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/4540f4f9ae73641286821bdc45f5e67d404907e0f135fbcab13e7ee23b801382/merged 2024-04-01T01:33:34.708 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/2689de020410cdb1ff530095c72a11b0a47f70861c66a2e659bca46eca82dbbf/merged 2024-04-01T01:33:34.708 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/eed934015ed699d9040d7dc0a23865bfc758905371e49b8d844691ba7ff9042d/merged 2024-04-01T01:33:34.708 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/271f1491dd89220d7f4665be2342d2711e5b7bf0d13774eac80a21f0870129cf/merged 2024-04-01T01:33:34.708 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/b57c328c249e3f0fbe148520cbe162647f9204fc1e0663cb6c8be45b8c28466b/merged 2024-04-01T01:33:34.709 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-01T01:33:34.709 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--tmp-cephfs 2024-04-01T01:33:34.709 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--tmp-cephfs 2024-04-01T01:33:34.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:33:34.710 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-01T01:33:34.758 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-01T01:33:34.758 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--tmp-cephfs sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /tmp/cephfs --id 0 --client_mountpoint=/volumes/qa/sv_0/79d38c5c-0128-46fd-a550-44920d4a703c --client_fs=cephfs -o context=system_u:object_r:postgresql_db_t:s0) 2024-04-01T01:33:34.801 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-01T01:33:34.827 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-01T01:33:34.885 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-01T01:33:34.885 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-01T01:33:34.885 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-01T01:33:34.885 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-01T01:33:34.885 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-01T01:33:34.886 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-01T01:33:34.886 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-01T01:33:34.886 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-01T01:33:34.886 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-01T01:33:34.886 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-01T01:33:34.886 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-01T01:33:34.887 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-01T01:33:34.887 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-01T01:33:34.887 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-01T01:33:34.887 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-01T01:33:34.887 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-01T01:33:34.887 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-01T01:33:34.887 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-01T01:33:34.887 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-01T01:33:34.887 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-01T01:33:34.887 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-01T01:33:34.887 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-01T01:33:34.887 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-01T01:33:34.887 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-01T01:33:34.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-01T01:33:34.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay 2024-04-01T01:33:34.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/585d4bb1e6cc6a226b096b7a454003dd2f25e52839ed75bbf063895bf9fd7647/merged 2024-04-01T01:33:34.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/363fe581d6f063986062f6da7b8ee4000839e42571fe570e38d6fbb8793740c8/merged 2024-04-01T01:33:34.888 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/0 2024-04-01T01:33:34.888 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-01T01:33:34.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/a12405395bb3d2147702e3fb40342d6133652c8521fdab4a47bc78ac3ac29f06/merged 2024-04-01T01:33:34.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/89ebda32a13bd3ddd2f34a1c254a0f6a4c573018cd0955b5136df44d8dffad5d/merged 2024-04-01T01:33:34.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/12b865e10303128d42ed6487fe430ce5108439f3bf4e03f50c67ec7752b3d01a/merged 2024-04-01T01:33:34.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/4540f4f9ae73641286821bdc45f5e67d404907e0f135fbcab13e7ee23b801382/merged 2024-04-01T01:33:34.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/2689de020410cdb1ff530095c72a11b0a47f70861c66a2e659bca46eca82dbbf/merged 2024-04-01T01:33:34.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/eed934015ed699d9040d7dc0a23865bfc758905371e49b8d844691ba7ff9042d/merged 2024-04-01T01:33:34.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/271f1491dd89220d7f4665be2342d2711e5b7bf0d13774eac80a21f0870129cf/merged 2024-04-01T01:33:34.888 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/b57c328c249e3f0fbe148520cbe162647f9204fc1e0663cb6c8be45b8c28466b/merged 2024-04-01T01:33:34.888 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-01T01:33:34.889 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--tmp-cephfs 2024-04-01T01:33:34.889 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--tmp-cephfs 2024-04-01T01:33:34.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:33:34.889 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-01T01:33:34.890 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:2024-04-01T01:33:34.888+0000 7f2071439480 -1 init, newargv = 0x55fa7a8f24c0 newargc=17 2024-04-01T01:33:34.890 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[86412]: starting ceph client 2024-04-01T01:33:34.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[86412]: starting fuse 2024-04-01T01:33:35.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:35 smithi189 ceph-mon[35753]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-01T01:33:36.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:35 smithi070 ceph-mon[32179]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-01T01:33:36.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-01T01:33:36.040 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-01T01:33:36.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:35 smithi081 ceph-mon[35142]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-01T01:33:36.098 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-01T01:33:36.099 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-01T01:33:36.099 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-01T01:33:36.099 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-01T01:33:36.099 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-01T01:33:36.099 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-01T01:33:36.099 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-01T01:33:36.099 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-01T01:33:36.099 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-01T01:33:36.099 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-01T01:33:36.099 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-01T01:33:36.099 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-01T01:33:36.099 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-01T01:33:36.099 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-01T01:33:36.100 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-01T01:33:36.100 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-01T01:33:36.100 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-01T01:33:36.100 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-01T01:33:36.100 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-01T01:33:36.100 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-01T01:33:36.100 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-01T01:33:36.100 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-01T01:33:36.100 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-01T01:33:36.100 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-01T01:33:36.101 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-01T01:33:36.101 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay 2024-04-01T01:33:36.101 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/585d4bb1e6cc6a226b096b7a454003dd2f25e52839ed75bbf063895bf9fd7647/merged 2024-04-01T01:33:36.101 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/363fe581d6f063986062f6da7b8ee4000839e42571fe570e38d6fbb8793740c8/merged 2024-04-01T01:33:36.101 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/0 2024-04-01T01:33:36.101 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-01T01:33:36.101 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/a12405395bb3d2147702e3fb40342d6133652c8521fdab4a47bc78ac3ac29f06/merged 2024-04-01T01:33:36.101 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/89ebda32a13bd3ddd2f34a1c254a0f6a4c573018cd0955b5136df44d8dffad5d/merged 2024-04-01T01:33:36.101 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/12b865e10303128d42ed6487fe430ce5108439f3bf4e03f50c67ec7752b3d01a/merged 2024-04-01T01:33:36.101 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/4540f4f9ae73641286821bdc45f5e67d404907e0f135fbcab13e7ee23b801382/merged 2024-04-01T01:33:36.101 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/2689de020410cdb1ff530095c72a11b0a47f70861c66a2e659bca46eca82dbbf/merged 2024-04-01T01:33:36.101 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/eed934015ed699d9040d7dc0a23865bfc758905371e49b8d844691ba7ff9042d/merged 2024-04-01T01:33:36.101 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/271f1491dd89220d7f4665be2342d2711e5b7bf0d13774eac80a21f0870129cf/merged 2024-04-01T01:33:36.101 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/containers/storage/overlay/b57c328c249e3f0fbe148520cbe162647f9204fc1e0663cb6c8be45b8c28466b/merged 2024-04-01T01:33:36.102 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-01T01:33:36.102 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--tmp-cephfs 2024-04-01T01:33:36.102 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--tmp-cephfs 2024-04-01T01:33:36.102 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs 2024-04-01T01:33:36.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:33:36.102 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-01T01:33:36.157 INFO:teuthology.orchestra.run.smithi070.stdout:119 2024-04-01T01:33:36.157 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-04-01T01:33:36.158 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-01T01:33:36.158 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-01T01:33:36.158 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-01T01:33:36.158 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-01T01:33:36.158 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-01T01:33:36.159 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-01T01:33:36.159 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-01T01:33:36.159 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-01T01:33:36.159 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-01T01:33:36.159 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/tmp/cephfs" 2024-04-01T01:33:36.159 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-01T01:33:36.159 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-01T01:33:36.160 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-01T01:33:36.160 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-01T01:33:36.160 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-01T01:33:36.160 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-01T01:33:36.160 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-01T01:33:36.160 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-01T01:33:36.160 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-01T01:33:36.161 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-01T01:33:36.161 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-01T01:33:36.161 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-01T01:33:36.161 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-01T01:33:36.161 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-01T01:33:36.161 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-01T01:33:36.162 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.0")) 2024-04-01T01:33:36.162 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-01T01:33:36.283 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.0.86412.asok 2024-04-01T01:33:36.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-01T01:33:36.294 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.86412.asok 2024-04-01T01:33:36.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:33:36.294 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.86412.asok status 2024-04-01T01:33:36.432 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:33:36.432 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-01T01:33:36.432 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "384f31f3d3aee7f8ba3f60846b8844c049165583", 2024-04-01T01:33:36.432 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-1977-g384f31f3 (384f31f3d3aee7f8ba3f60846b8844c049165583) squid (dev)", 2024-04-01T01:33:36.432 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "0", 2024-04-01T01:33:36.432 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-01T01:33:36.432 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/tmp/cephfs", 2024-04-01T01:33:36.432 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "86412", 2024-04-01T01:33:36.432 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/volumes/qa/sv_0/79d38c5c-0128-46fd-a550-44920d4a703c" 2024-04-01T01:33:36.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:33:36.433 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-01T01:33:36.433 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-01T01:33:36.433 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 15513, 2024-04-01T01:33:36.433 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-01T01:33:36.433 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-01T01:33:36.433 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-01T01:33:36.433 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 15513 2024-04-01T01:33:36.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:33:36.433 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-01T01:33:36.433 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-01T01:33:36.433 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-01T01:33:36.433 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3309948499 2024-04-01T01:33:36.433 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:33:36.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:33:36.433 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-01T01:33:36.434 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-01T01:33:36.434 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-01T01:33:36.434 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3309948499 2024-04-01T01:33:36.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:33:36.434 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.15513 192.168.0.1:0/3309948499", 2024-04-01T01:33:36.434 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.1:0/3309948499", 2024-04-01T01:33:36.434 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 5, 2024-04-01T01:33:36.434 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 28, 2024-04-01T01:33:36.434 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 93, 2024-04-01T01:33:36.434 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-01T01:33:36.434 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-01T01:33:36.434 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-01T01:33:36.434 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:33:36.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:33:36.440 DEBUG:teuthology.orchestra.run.smithi070:> stat --file-system '--printf=%T 2024-04-01T01:33:36.441 DEBUG:teuthology.orchestra.run.smithi070:> ' -- /tmp/cephfs 2024-04-01T01:33:36.455 INFO:teuthology.orchestra.run.smithi070.stdout:fuseblk 2024-04-01T01:33:36.456 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /tmp/cephfs 2024-04-01T01:33:36.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:33:36.456 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 1777 /tmp/cephfs 2024-04-01T01:33:36.526 INFO:teuthology.run_tasks:Running task exec... 2024-04-01T01:33:36.536 INFO:teuthology.task.exec:Executing custom commands... 2024-04-01T01:33:36.536 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi070.front.sepia.ceph.com 2024-04-01T01:33:36.537 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-01T01:33:36.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.671+0000 7fe201855640 1 -- 172.21.15.70:0/4157250342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1fc101860 msgr2=0x7fe1fc101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:36.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.671+0000 7fe201855640 1 --2- 172.21.15.70:0/4157250342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1fc101860 0x7fe1fc101ce0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0009900 tx=0x7fe1f002bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:36.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.671+0000 7fe201855640 1 -- 172.21.15.70:0/4157250342 shutdown_connections 2024-04-01T01:33:36.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.671+0000 7fe201855640 1 --2- 172.21.15.70:0/4157250342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1fc102220 0x7fe1fc108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:36.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.671+0000 7fe201855640 1 --2- 172.21.15.70:0/4157250342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1fc101860 0x7fe1fc101ce0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:36.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.671+0000 7fe201855640 1 --2- 172.21.15.70:0/4157250342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1fc100670 0x7fe1fc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:36.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.671+0000 7fe201855640 1 -- 172.21.15.70:0/4157250342 >> 172.21.15.70:0/4157250342 conn(0x7fe1fc0fbbe0 msgr2=0x7fe1fc0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:36.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.671+0000 7fe201855640 1 -- 172.21.15.70:0/4157250342 shutdown_connections 2024-04-01T01:33:36.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.671+0000 7fe201855640 1 -- 172.21.15.70:0/4157250342 wait complete. 2024-04-01T01:33:36.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.671+0000 7fe201855640 1 Processor -- start 2024-04-01T01:33:36.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.671+0000 7fe201855640 1 -- start start 2024-04-01T01:33:36.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.671+0000 7fe201855640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1fc100670 0x7fe1fc196820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:36.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe201855640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1fc101860 0x7fe1fc196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:36.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe201855640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1fc102220 0x7fe1fc19ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:36.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe201855640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe1fc10c1b0 con 0x7fe1fc101860 2024-04-01T01:33:36.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe201855640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe1fc10c030 con 0x7fe1fc100670 2024-04-01T01:33:36.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe201855640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe1fc10c330 con 0x7fe1fc102220 2024-04-01T01:33:36.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe1fa7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1fc101860 0x7fe1fc196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:36.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe1faffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1fc100670 0x7fe1fc196820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:36.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe1faffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1fc100670 0x7fe1fc196820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34342/0 (socket says 172.21.15.70:34342) 2024-04-01T01:33:36.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe1fa7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1fc101860 0x7fe1fc196d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55176/0 (socket says 172.21.15.70:55176) 2024-04-01T01:33:36.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe1faffd640 1 -- 172.21.15.70:0/2032866280 learned_addr learned my addr 172.21.15.70:0/2032866280 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:36.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe1fb7fe640 1 --2- 172.21.15.70:0/2032866280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1fc102220 0x7fe1fc19ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:36.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe1fa7fc640 1 -- 172.21.15.70:0/2032866280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1fc102220 msgr2=0x7fe1fc19ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:36.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe1fa7fc640 1 --2- 172.21.15.70:0/2032866280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1fc102220 0x7fe1fc19ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:36.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe1fa7fc640 1 -- 172.21.15.70:0/2032866280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1fc100670 msgr2=0x7fe1fc196820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:36.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe1fa7fc640 1 --2- 172.21.15.70:0/2032866280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1fc100670 0x7fe1fc196820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:36.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe1fa7fc640 1 -- 172.21.15.70:0/2032866280 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1fc19e440 con 0x7fe1fc101860 2024-04-01T01:33:36.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe1faffd640 1 --2- 172.21.15.70:0/2032866280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1fc100670 0x7fe1fc196820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:36.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe1fa7fc640 1 --2- 172.21.15.70:0/2032866280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1fc101860 0x7fe1fc196d60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0000c00 tx=0x7fe1f0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:36.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe200853640 1 -- 172.21.15.70:0/2032866280 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0043020 con 0x7fe1fc101860 2024-04-01T01:33:36.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe200853640 1 -- 172.21.15.70:0/2032866280 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1f0004350 con 0x7fe1fc101860 2024-04-01T01:33:36.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe201855640 1 -- 172.21.15.70:0/2032866280 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1fc19e6d0 con 0x7fe1fc101860 2024-04-01T01:33:36.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe200853640 1 -- 172.21.15.70:0/2032866280 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0003b50 con 0x7fe1fc101860 2024-04-01T01:33:36.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.672+0000 7fe201855640 1 -- 172.21.15.70:0/2032866280 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1fc19ec20 con 0x7fe1fc101860 2024-04-01T01:33:36.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.673+0000 7fe201855640 1 -- 172.21.15.70:0/2032866280 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1fc1079b0 con 0x7fe1fc101860 2024-04-01T01:33:36.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.674+0000 7fe200853640 1 -- 172.21.15.70:0/2032866280 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fe1f0046020 con 0x7fe1fc101860 2024-04-01T01:33:36.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.675+0000 7fe200853640 1 --2- 172.21.15.70:0/2032866280 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe1e40a2540 0x7fe1e40a4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:36.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.675+0000 7fe1faffd640 1 --2- 172.21.15.70:0/2032866280 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe1e40a2540 0x7fe1e40a4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:36.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.675+0000 7fe200853640 1 -- 172.21.15.70:0/2032866280 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe1f00f0b40 con 0x7fe1fc101860 2024-04-01T01:33:36.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.675+0000 7fe1faffd640 1 --2- 172.21.15.70:0/2032866280 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe1e40a2540 0x7fe1e40a4a00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec0058c0 tx=0x7fe1ec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:36.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.679+0000 7fe200853640 1 -- 172.21.15.70:0/2032866280 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe1f004d020 con 0x7fe1fc101860 2024-04-01T01:33:36.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:36.893+0000 7fe201855640 1 -- 172.21.15.70:0/2032866280 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7fe1fc062350 con 0x7fe1fc101860 2024-04-01T01:33:37.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.682+0000 7fe200853640 1 -- 172.21.15.70:0/2032866280 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7fe1f00bd930 con 0x7fe1fc101860 2024-04-01T01:33:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.684+0000 7fe201855640 1 -- 172.21.15.70:0/2032866280 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe1e40a2540 msgr2=0x7fe1e40a4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.684+0000 7fe201855640 1 --2- 172.21.15.70:0/2032866280 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe1e40a2540 0x7fe1e40a4a00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec0058c0 tx=0x7fe1ec005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.684+0000 7fe201855640 1 -- 172.21.15.70:0/2032866280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1fc101860 msgr2=0x7fe1fc196d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.684+0000 7fe201855640 1 --2- 172.21.15.70:0/2032866280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1fc101860 0x7fe1fc196d60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0000c00 tx=0x7fe1f0005dc0 comp rx=0 tx=0).stop 2024-04-01T01:33:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.685+0000 7fe201855640 1 -- 172.21.15.70:0/2032866280 shutdown_connections 2024-04-01T01:33:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.685+0000 7fe201855640 1 --2- 172.21.15.70:0/2032866280 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe1e40a2540 0x7fe1e40a4a00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.685+0000 7fe201855640 1 --2- 172.21.15.70:0/2032866280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1fc102220 0x7fe1fc19ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.685+0000 7fe201855640 1 --2- 172.21.15.70:0/2032866280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1fc101860 0x7fe1fc196d60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.685+0000 7fe201855640 1 --2- 172.21.15.70:0/2032866280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1fc100670 0x7fe1fc196820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:37.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.685+0000 7fe201855640 1 -- 172.21.15.70:0/2032866280 >> 172.21.15.70:0/2032866280 conn(0x7fe1fc0fbbe0 msgr2=0x7fe1fc104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:37.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.685+0000 7fe201855640 1 -- 172.21.15.70:0/2032866280 shutdown_connections 2024-04-01T01:33:37.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.685+0000 7fe201855640 1 -- 172.21.15.70:0/2032866280 wait complete. 2024-04-01T01:33:37.699 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-01T01:33:37.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.808+0000 7fc1789e2640 1 -- 172.21.15.70:0/1082120463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc174104b20 msgr2=0x7fc174106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:37.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.808+0000 7fc1789e2640 1 --2- 172.21.15.70:0/1082120463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc174104b20 0x7fc174106f10 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc168009900 tx=0x7fc16802bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:37.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.809+0000 7fc1789e2640 1 -- 172.21.15.70:0/1082120463 shutdown_connections 2024-04-01T01:33:37.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.809+0000 7fc1789e2640 1 --2- 172.21.15.70:0/1082120463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc174107450 0x7fc1741098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:37.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.809+0000 7fc1789e2640 1 --2- 172.21.15.70:0/1082120463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc174104b20 0x7fc174106f10 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:37.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.809+0000 7fc1789e2640 1 --2- 172.21.15.70:0/1082120463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1741021f0 0x7fc1741045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:37.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.809+0000 7fc1789e2640 1 -- 172.21.15.70:0/1082120463 >> 172.21.15.70:0/1082120463 conn(0x7fc1740fbbe0 msgr2=0x7fc1740fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:37.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.809+0000 7fc1789e2640 1 -- 172.21.15.70:0/1082120463 shutdown_connections 2024-04-01T01:33:37.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.809+0000 7fc1789e2640 1 -- 172.21.15.70:0/1082120463 wait complete. 2024-04-01T01:33:37.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.809+0000 7fc1789e2640 1 Processor -- start 2024-04-01T01:33:37.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.809+0000 7fc1789e2640 1 -- start start 2024-04-01T01:33:37.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.809+0000 7fc1789e2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1741021f0 0x7fc174197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:37.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.809+0000 7fc1789e2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc174104b20 0x7fc174197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:37.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.809+0000 7fc1789e2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc174107450 0x7fc17419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:37.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.809+0000 7fc1789e2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc17410d2b0 con 0x7fc174107450 2024-04-01T01:33:37.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.809+0000 7fc1789e2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc17410d130 con 0x7fc174104b20 2024-04-01T01:33:37.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.809+0000 7fc1789e2640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc17410d430 con 0x7fc1741021f0 2024-04-01T01:33:37.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc172d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc174107450 0x7fc17419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:37.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc172d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc174107450 0x7fc17419ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55190/0 (socket says 172.21.15.70:55190) 2024-04-01T01:33:37.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc171d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc174104b20 0x7fc174197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc17259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1741021f0 0x7fc174197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc172d9d640 1 -- 172.21.15.70:0/3265203837 learned_addr learned my addr 172.21.15.70:0/3265203837 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc172d9d640 1 -- 172.21.15.70:0/3265203837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1741021f0 msgr2=0x7fc174197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc172d9d640 1 --2- 172.21.15.70:0/3265203837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1741021f0 0x7fc174197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc172d9d640 1 -- 172.21.15.70:0/3265203837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc174104b20 msgr2=0x7fc174197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc172d9d640 1 --2- 172.21.15.70:0/3265203837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc174104b20 0x7fc174197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc172d9d640 1 -- 172.21.15.70:0/3265203837 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17419f5a0 con 0x7fc174107450 2024-04-01T01:33:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc171d9b640 1 --2- 172.21.15.70:0/3265203837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc174104b20 0x7fc174197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc172d9d640 1 --2- 172.21.15.70:0/3265203837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc174107450 0x7fc17419ef30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc16c00eb10 tx=0x7fc16c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc1637fe640 1 -- 172.21.15.70:0/3265203837 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c004b20 con 0x7fc174107450 2024-04-01T01:33:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc1637fe640 1 -- 172.21.15.70:0/3265203837 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc16c004cc0 con 0x7fc174107450 2024-04-01T01:33:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc1789e2640 1 -- 172.21.15.70:0/3265203837 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc17419f890 con 0x7fc174107450 2024-04-01T01:33:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc1637fe640 1 -- 172.21.15.70:0/3265203837 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c010690 con 0x7fc174107450 2024-04-01T01:33:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.810+0000 7fc1789e2640 1 -- 172.21.15.70:0/3265203837 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc17419fd60 con 0x7fc174107450 2024-04-01T01:33:37.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.811+0000 7fc1789e2640 1 -- 172.21.15.70:0/3265203837 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1740fd6d0 con 0x7fc174107450 2024-04-01T01:33:37.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.812+0000 7fc1637fe640 1 -- 172.21.15.70:0/3265203837 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7fc16c005020 con 0x7fc174107450 2024-04-01T01:33:37.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.813+0000 7fc1637fe640 1 --2- 172.21.15.70:0/3265203837 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc15c0a2540 0x7fc15c0a4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:37.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.813+0000 7fc17259c640 1 --2- 172.21.15.70:0/3265203837 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc15c0a2540 0x7fc15c0a4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:37.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.813+0000 7fc1637fe640 1 -- 172.21.15.70:0/3265203837 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fc16c0cf080 con 0x7fc174107450 2024-04-01T01:33:37.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.813+0000 7fc17259c640 1 --2- 172.21.15.70:0/3265203837 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc15c0a2540 0x7fc15c0a4a00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc164009bd0 tx=0x7fc164005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:37.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:37.817+0000 7fc1637fe640 1 -- 172.21.15.70:0/3265203837 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc16c09be70 con 0x7fc174107450 2024-04-01T01:33:37.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:37 smithi189 ceph-mon[35753]: pgmap v362: 129 pgs: 129 active+clean; 648 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 12 op/s 2024-04-01T01:33:38.018 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:37 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2032866280' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-01T01:33:38.018 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:37 smithi070 ceph-mon[32179]: pgmap v362: 129 pgs: 129 active+clean; 648 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 12 op/s 2024-04-01T01:33:38.018 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:37 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2032866280' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-01T01:33:38.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.019+0000 7fc1789e2640 1 -- 172.21.15.70:0/3265203837 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7fc174061c30 con 0x7fc15c0a2540 2024-04-01T01:33:38.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.037+0000 7fc1637fe640 1 -- 172.21.15.70:0/3265203837 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7fc174061c30 con 0x7fc15c0a2540 2024-04-01T01:33:38.038 INFO:teuthology.orchestra.run.smithi070.stdout:{} 2024-04-01T01:33:38.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.040+0000 7fc1789e2640 1 -- 172.21.15.70:0/3265203837 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc15c0a2540 msgr2=0x7fc15c0a4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:38.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.040+0000 7fc1789e2640 1 --2- 172.21.15.70:0/3265203837 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc15c0a2540 0x7fc15c0a4a00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc164009bd0 tx=0x7fc164005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:38.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.040+0000 7fc1789e2640 1 -- 172.21.15.70:0/3265203837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc174107450 msgr2=0x7fc17419ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:38.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.040+0000 7fc1789e2640 1 --2- 172.21.15.70:0/3265203837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc174107450 0x7fc17419ef30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc16c00eb10 tx=0x7fc16c00c2d0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.040+0000 7fc1789e2640 1 -- 172.21.15.70:0/3265203837 shutdown_connections 2024-04-01T01:33:38.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.040+0000 7fc1789e2640 1 --2- 172.21.15.70:0/3265203837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc174107450 0x7fc17419ef30 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.040+0000 7fc1789e2640 1 --2- 172.21.15.70:0/3265203837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc174104b20 0x7fc174197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.040+0000 7fc1789e2640 1 --2- 172.21.15.70:0/3265203837 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc15c0a2540 0x7fc15c0a4a00 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.040+0000 7fc1789e2640 1 --2- 172.21.15.70:0/3265203837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1741021f0 0x7fc174197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.040+0000 7fc1789e2640 1 -- 172.21.15.70:0/3265203837 >> 172.21.15.70:0/3265203837 conn(0x7fc1740fbbe0 msgr2=0x7fc174102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:38.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.040+0000 7fc1789e2640 1 -- 172.21.15.70:0/3265203837 shutdown_connections 2024-04-01T01:33:38.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.040+0000 7fc1789e2640 1 -- 172.21.15.70:0/3265203837 wait complete. 2024-04-01T01:33:38.051 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-01T01:33:38.067 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-01T01:33:38.103 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:33:38.104 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:37 smithi081 ceph-mon[35142]: pgmap v362: 129 pgs: 129 active+clean; 648 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 12 op/s 2024-04-01T01:33:38.104 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:37 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2032866280' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-01T01:33:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.204+0000 7f28e51b5640 1 -- 172.21.15.70:0/365092806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28e0103330 msgr2=0x7f28e0109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.204+0000 7f28e51b5640 1 --2- 172.21.15.70:0/365092806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28e0103330 0x7f28e0109bc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f28d8009be0 tx=0x7f28d802bef0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.204+0000 7f28e51b5640 1 -- 172.21.15.70:0/365092806 shutdown_connections 2024-04-01T01:33:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.204+0000 7f28e51b5640 1 --2- 172.21.15.70:0/365092806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28e0103330 0x7f28e0109bc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.204+0000 7f28e51b5640 1 --2- 172.21.15.70:0/365092806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28e0102970 0x7f28e0102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.204+0000 7f28e51b5640 1 --2- 172.21.15.70:0/365092806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28e0101780 0x7f28e0101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.204+0000 7f28e51b5640 1 -- 172.21.15.70:0/365092806 >> 172.21.15.70:0/365092806 conn(0x7f28e00fcd30 msgr2=0x7f28e00ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.204+0000 7f28e51b5640 1 -- 172.21.15.70:0/365092806 shutdown_connections 2024-04-01T01:33:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.204+0000 7f28e51b5640 1 -- 172.21.15.70:0/365092806 wait complete. 2024-04-01T01:33:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28e51b5640 1 Processor -- start 2024-04-01T01:33:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28e51b5640 1 -- start start 2024-04-01T01:33:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28e51b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28e0101780 0x7f28e0197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28e51b5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28e0102970 0x7f28e0197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28e51b5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28e0103330 0x7f28e019ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28e51b5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f28e010d2b0 con 0x7f28e0101780 2024-04-01T01:33:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28e51b5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f28e010d130 con 0x7f28e0102970 2024-04-01T01:33:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28e51b5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f28e010d430 con 0x7f28e0103330 2024-04-01T01:33:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28ded9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28e0101780 0x7f28e0197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28ded9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28e0101780 0x7f28e0197960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55234/0 (socket says 172.21.15.70:55234) 2024-04-01T01:33:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28ded9d640 1 -- 172.21.15.70:0/3190837660 learned_addr learned my addr 172.21.15.70:0/3190837660 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28d7fff640 1 --2- 172.21.15.70:0/3190837660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28e0102970 0x7f28e0197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28df59e640 1 --2- 172.21.15.70:0/3190837660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28e0103330 0x7f28e019ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:38.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28ded9d640 1 -- 172.21.15.70:0/3190837660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28e0103330 msgr2=0x7f28e019ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28ded9d640 1 --2- 172.21.15.70:0/3190837660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28e0103330 0x7f28e019ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28ded9d640 1 -- 172.21.15.70:0/3190837660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28e0102970 msgr2=0x7f28e0197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28ded9d640 1 --2- 172.21.15.70:0/3190837660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28e0102970 0x7f28e0197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.205+0000 7f28ded9d640 1 -- 172.21.15.70:0/3190837660 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e019f580 con 0x7f28e0101780 2024-04-01T01:33:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.206+0000 7f28ded9d640 1 --2- 172.21.15.70:0/3190837660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28e0101780 0x7f28e0197960 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f28d000e9c0 tx=0x7f28d000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.206+0000 7f28dcd99640 1 -- 172.21.15.70:0/3190837660 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d000ccc0 con 0x7f28e0101780 2024-04-01T01:33:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.206+0000 7f28e51b5640 1 -- 172.21.15.70:0/3190837660 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28e019f870 con 0x7f28e0101780 2024-04-01T01:33:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.206+0000 7f28dcd99640 1 -- 172.21.15.70:0/3190837660 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f28d00035f0 con 0x7f28e0101780 2024-04-01T01:33:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.206+0000 7f28e51b5640 1 -- 172.21.15.70:0/3190837660 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28e019fdc0 con 0x7f28e0101780 2024-04-01T01:33:38.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.206+0000 7f28dcd99640 1 -- 172.21.15.70:0/3190837660 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d000ce60 con 0x7f28e0101780 2024-04-01T01:33:38.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.207+0000 7f28e51b5640 1 -- 172.21.15.70:0/3190837660 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28e0108b10 con 0x7f28e0101780 2024-04-01T01:33:38.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.208+0000 7f28dcd99640 1 -- 172.21.15.70:0/3190837660 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f28d00037a0 con 0x7f28e0101780 2024-04-01T01:33:38.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.209+0000 7f28dcd99640 1 --2- 172.21.15.70:0/3190837660 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28cc0a24a0 0x7f28cc0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:38.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.209+0000 7f28d7fff640 1 --2- 172.21.15.70:0/3190837660 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28cc0a24a0 0x7f28cc0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:38.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.209+0000 7f28dcd99640 1 -- 172.21.15.70:0/3190837660 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f28d00cefb0 con 0x7f28e0101780 2024-04-01T01:33:38.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.209+0000 7f28d7fff640 1 --2- 172.21.15.70:0/3190837660 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28cc0a24a0 0x7f28cc0a4960 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f28e0198e70 tx=0x7f28c8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:38.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.214+0000 7f28dcd99640 1 -- 172.21.15.70:0/3190837660 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f28d009bda0 con 0x7f28e0101780 2024-04-01T01:33:38.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.391+0000 7f28e51b5640 1 -- 172.21.15.70:0/3190837660 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28e0062350 con 0x7f28e0101780 2024-04-01T01:33:38.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.392+0000 7f28dcd99640 1 -- 172.21.15.70:0/3190837660 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24781 (secure 0 0 0) 0x7f28d009b6a0 con 0x7f28e0101780 2024-04-01T01:33:38.393 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:38.393 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":93,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:33:19.716245+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:33:38.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.395+0000 7f28e51b5640 1 -- 172.21.15.70:0/3190837660 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28cc0a24a0 msgr2=0x7f28cc0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:38.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.395+0000 7f28e51b5640 1 --2- 172.21.15.70:0/3190837660 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28cc0a24a0 0x7f28cc0a4960 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f28e0198e70 tx=0x7f28c8009260 comp rx=0 tx=0).stop 2024-04-01T01:33:38.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.395+0000 7f28e51b5640 1 -- 172.21.15.70:0/3190837660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28e0101780 msgr2=0x7f28e0197960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:38.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.395+0000 7f28e51b5640 1 --2- 172.21.15.70:0/3190837660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28e0101780 0x7f28e0197960 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f28d000e9c0 tx=0x7f28d000edc0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.395+0000 7f28e51b5640 1 -- 172.21.15.70:0/3190837660 shutdown_connections 2024-04-01T01:33:38.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.395+0000 7f28e51b5640 1 --2- 172.21.15.70:0/3190837660 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28cc0a24a0 0x7f28cc0a4960 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.395+0000 7f28e51b5640 1 --2- 172.21.15.70:0/3190837660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28e0103330 0x7f28e019ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.395+0000 7f28e51b5640 1 --2- 172.21.15.70:0/3190837660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28e0102970 0x7f28e0197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.395+0000 7f28e51b5640 1 --2- 172.21.15.70:0/3190837660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28e0101780 0x7f28e0197960 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.395+0000 7f28e51b5640 1 -- 172.21.15.70:0/3190837660 >> 172.21.15.70:0/3190837660 conn(0x7f28e00fcd30 msgr2=0x7f28e0105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:38.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.395+0000 7f28e51b5640 1 -- 172.21.15.70:0/3190837660 shutdown_connections 2024-04-01T01:33:38.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.395+0000 7f28e51b5640 1 -- 172.21.15.70:0/3190837660 wait complete. 2024-04-01T01:33:38.405 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:33:38 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:33:38.024+0000 7f37ad4b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-01T01:33:38.405 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:33:38 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:33:38.024+0000 7f37ad4b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-01T01:33:38.405 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:33:38 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:33:38.024+0000 7f37ad4b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-01T01:33:38.405 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:33:38 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:33:38.024+0000 7f37ad4b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-01T01:33:38.405 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:33:38 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:33:38.024+0000 7f37ad4b0700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-01T01:33:38.407 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-01T01:26:42.809779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-01T01:31:13.612369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-01T01:31:14.910162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-01T01:31:18.783179+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-01T01:33:38.407 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-01T01:33:38.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.512+0000 7f5f2911e640 1 -- 172.21.15.70:0/1102276028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f24100660 msgr2=0x7f5f24100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:38.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.512+0000 7f5f2911e640 1 --2- 172.21.15.70:0/1102276028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f24100660 0x7f5f24100a80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5f14009900 tx=0x7f5f1402bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:38.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.512+0000 7f5f2911e640 1 -- 172.21.15.70:0/1102276028 shutdown_connections 2024-04-01T01:33:38.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.512+0000 7f5f2911e640 1 --2- 172.21.15.70:0/1102276028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f24102210 0x7f5f24108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.512+0000 7f5f2911e640 1 --2- 172.21.15.70:0/1102276028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f24101850 0x7f5f24101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.512+0000 7f5f2911e640 1 --2- 172.21.15.70:0/1102276028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f24100660 0x7f5f24100a80 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.512+0000 7f5f2911e640 1 -- 172.21.15.70:0/1102276028 >> 172.21.15.70:0/1102276028 conn(0x7f5f240fbbd0 msgr2=0x7f5f240fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:38.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.512+0000 7f5f2911e640 1 -- 172.21.15.70:0/1102276028 shutdown_connections 2024-04-01T01:33:38.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.512+0000 7f5f2911e640 1 -- 172.21.15.70:0/1102276028 wait complete. 2024-04-01T01:33:38.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f2911e640 1 Processor -- start 2024-04-01T01:33:38.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f2911e640 1 -- start start 2024-04-01T01:33:38.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f2911e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f24100660 0x7f5f24197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:38.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f2911e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f24101850 0x7f5f24197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:38.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f2911e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f24102210 0x7f5f2419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:38.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f2911e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2410c1e0 con 0x7f5f24102210 2024-04-01T01:33:38.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f2911e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2410c060 con 0x7f5f24100660 2024-04-01T01:33:38.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f2911e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2410c360 con 0x7f5f24101850 2024-04-01T01:33:38.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f2359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f24102210 0x7f5f2419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:38.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f2359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f24102210 0x7f5f2419efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55250/0 (socket says 172.21.15.70:55250) 2024-04-01T01:33:38.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f2359e640 1 -- 172.21.15.70:0/3664309242 learned_addr learned my addr 172.21.15.70:0/3664309242 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:38.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f22d9d640 1 --2- 172.21.15.70:0/3664309242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f24100660 0x7f5f24197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:38.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f1a59c640 1 --2- 172.21.15.70:0/3664309242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f24101850 0x7f5f24197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:38.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f2359e640 1 -- 172.21.15.70:0/3664309242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f24101850 msgr2=0x7f5f24197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:38.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f2359e640 1 --2- 172.21.15.70:0/3664309242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f24101850 0x7f5f24197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f2359e640 1 -- 172.21.15.70:0/3664309242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f24100660 msgr2=0x7f5f24197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:38.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f2359e640 1 --2- 172.21.15.70:0/3664309242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f24100660 0x7f5f24197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.513+0000 7f5f2359e640 1 -- 172.21.15.70:0/3664309242 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f2419f620 con 0x7f5f24102210 2024-04-01T01:33:38.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.514+0000 7f5f2359e640 1 --2- 172.21.15.70:0/3664309242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f24102210 0x7f5f2419efb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c00eb10 tx=0x7f5f1c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:38.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.514+0000 7f5f20d99640 1 -- 172.21.15.70:0/3664309242 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c004b20 con 0x7f5f24102210 2024-04-01T01:33:38.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.514+0000 7f5f20d99640 1 -- 172.21.15.70:0/3664309242 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f1c004cc0 con 0x7f5f24102210 2024-04-01T01:33:38.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.514+0000 7f5f2911e640 1 -- 172.21.15.70:0/3664309242 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f2419f910 con 0x7f5f24102210 2024-04-01T01:33:38.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.514+0000 7f5f20d99640 1 -- 172.21.15.70:0/3664309242 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c010690 con 0x7f5f24102210 2024-04-01T01:33:38.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.514+0000 7f5f2911e640 1 -- 172.21.15.70:0/3664309242 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f2419fe60 con 0x7f5f24102210 2024-04-01T01:33:38.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.515+0000 7f5f2911e640 1 -- 172.21.15.70:0/3664309242 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f241a00f0 con 0x7f5f24102210 2024-04-01T01:33:38.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.516+0000 7f5f20d99640 1 -- 172.21.15.70:0/3664309242 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 137320+0+0 (secure 0 0 0) 0x7f5f1c005000 con 0x7f5f24102210 2024-04-01T01:33:38.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.517+0000 7f5f20d99640 1 --2- 172.21.15.70:0/3664309242 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f100a2520 0x7f5f100a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:38.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.517+0000 7f5f22d9d640 1 --2- 172.21.15.70:0/3664309242 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f100a2520 0x7f5f100a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:38.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.517+0000 7f5f20d99640 1 -- 172.21.15.70:0/3664309242 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5f1c0cf1f0 con 0x7f5f24102210 2024-04-01T01:33:38.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.517+0000 7f5f22d9d640 1 --2- 172.21.15.70:0/3664309242 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f100a2520 0x7f5f100a49e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5f1402c240 tx=0x7f5f1402c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:38.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.520+0000 7f5f20d99640 1 -- 172.21.15.70:0/3664309242 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5f1c053280 con 0x7f5f24102210 2024-04-01T01:33:38.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.697+0000 7f5f2911e640 1 -- 172.21.15.70:0/3664309242 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5f241a02c0 con 0x7f5f24102210 2024-04-01T01:33:38.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.702+0000 7f5f20d99640 1 -- 172.21.15.70:0/3664309242 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f5f1c0bb030 con 0x7f5f24102210 2024-04-01T01:33:38.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.702+0000 7f5f20d99640 1 -- 172.21.15.70:0/3664309242 <== mon.0 v2:172.21.15.70:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f5f1c0ae8c0 con 0x7f5f24102210 2024-04-01T01:33:38.702 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-01T01:33:38.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.704+0000 7f5f2911e640 1 -- 172.21.15.70:0/3664309242 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f100a2520 msgr2=0x7f5f100a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:38.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.704+0000 7f5f2911e640 1 --2- 172.21.15.70:0/3664309242 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f100a2520 0x7f5f100a49e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5f1402c240 tx=0x7f5f1402c7e0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.704+0000 7f5f2911e640 1 -- 172.21.15.70:0/3664309242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f24102210 msgr2=0x7f5f2419efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:38.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.704+0000 7f5f2911e640 1 --2- 172.21.15.70:0/3664309242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f24102210 0x7f5f2419efb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c00eb10 tx=0x7f5f1c00c2d0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.704+0000 7f5f2911e640 1 -- 172.21.15.70:0/3664309242 shutdown_connections 2024-04-01T01:33:38.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.704+0000 7f5f2911e640 1 --2- 172.21.15.70:0/3664309242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f24102210 0x7f5f2419efb0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.704+0000 7f5f2911e640 1 --2- 172.21.15.70:0/3664309242 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f100a2520 0x7f5f100a49e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.704+0000 7f5f2911e640 1 --2- 172.21.15.70:0/3664309242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f24101850 0x7f5f24197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.704+0000 7f5f2911e640 1 --2- 172.21.15.70:0/3664309242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f24100660 0x7f5f24197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.704+0000 7f5f2911e640 1 -- 172.21.15.70:0/3664309242 >> 172.21.15.70:0/3664309242 conn(0x7f5f240fbbd0 msgr2=0x7f5f240fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:38.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.705+0000 7f5f2911e640 1 -- 172.21.15.70:0/3664309242 shutdown_connections 2024-04-01T01:33:38.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.705+0000 7f5f2911e640 1 -- 172.21.15.70:0/3664309242 wait complete. 2024-04-01T01:33:38.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-01T01:33:38.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.821+0000 7f090dbb5640 1 -- 172.21.15.70:0/1458412842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0908102290 msgr2=0x7f0908104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:38.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.821+0000 7f090dbb5640 1 --2- 172.21.15.70:0/1458412842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0908102290 0x7f0908104680 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f08f8009900 tx=0x7f08f802bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:38.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.821+0000 7f090dbb5640 1 -- 172.21.15.70:0/1458412842 shutdown_connections 2024-04-01T01:33:38.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.821+0000 7f090dbb5640 1 --2- 172.21.15.70:0/1458412842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09081074f0 0x7f09081098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.821+0000 7f090dbb5640 1 --2- 172.21.15.70:0/1458412842 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0908104bc0 0x7f0908106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.821+0000 7f090dbb5640 1 --2- 172.21.15.70:0/1458412842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0908102290 0x7f0908104680 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.821+0000 7f090dbb5640 1 -- 172.21.15.70:0/1458412842 >> 172.21.15.70:0/1458412842 conn(0x7f09080fbc00 msgr2=0x7f09080fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:38.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.821+0000 7f090dbb5640 1 -- 172.21.15.70:0/1458412842 shutdown_connections 2024-04-01T01:33:38.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.821+0000 7f090dbb5640 1 -- 172.21.15.70:0/1458412842 wait complete. 2024-04-01T01:33:38.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.821+0000 7f090dbb5640 1 Processor -- start 2024-04-01T01:33:38.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f090dbb5640 1 -- start start 2024-04-01T01:33:38.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f090dbb5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0908102290 0x7f09081979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:38.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f090dbb5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0908104bc0 0x7f0908197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:38.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f090dbb5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09081074f0 0x7f090819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:38.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f090dbb5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f090810d2b0 con 0x7f0908102290 2024-04-01T01:33:38.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f090dbb5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f090810d130 con 0x7f09081074f0 2024-04-01T01:33:38.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f090dbb5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f090810d430 con 0x7f0908104bc0 2024-04-01T01:33:38.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f09077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0908102290 0x7f09081979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:38.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f09077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0908102290 0x7f09081979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55274/0 (socket says 172.21.15.70:55274) 2024-04-01T01:33:38.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f09077fe640 1 -- 172.21.15.70:0/3997698529 learned_addr learned my addr 172.21.15.70:0/3997698529 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:38.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f0906ffd640 1 --2- 172.21.15.70:0/3997698529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0908104bc0 0x7f0908197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:38.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f0907fff640 1 --2- 172.21.15.70:0/3997698529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09081074f0 0x7f090819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:38.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f09077fe640 1 -- 172.21.15.70:0/3997698529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0908104bc0 msgr2=0x7f0908197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:38.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f09077fe640 1 --2- 172.21.15.70:0/3997698529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0908104bc0 0x7f0908197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f09077fe640 1 -- 172.21.15.70:0/3997698529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09081074f0 msgr2=0x7f090819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:38.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f09077fe640 1 --2- 172.21.15.70:0/3997698529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09081074f0 0x7f090819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:38.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f09077fe640 1 -- 172.21.15.70:0/3997698529 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f090819f600 con 0x7f0908102290 2024-04-01T01:33:38.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f09077fe640 1 --2- 172.21.15.70:0/3997698529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0908102290 0x7f09081979e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f08f8006010 tx=0x7f08f8005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:38.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f0904ff9640 1 -- 172.21.15.70:0/3997698529 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f8043020 con 0x7f0908102290 2024-04-01T01:33:38.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f0904ff9640 1 -- 172.21.15.70:0/3997698529 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08f80043d0 con 0x7f0908102290 2024-04-01T01:33:38.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f090dbb5640 1 -- 172.21.15.70:0/3997698529 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f090819f890 con 0x7f0908102290 2024-04-01T01:33:38.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f0904ff9640 1 -- 172.21.15.70:0/3997698529 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f8003b80 con 0x7f0908102290 2024-04-01T01:33:38.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.822+0000 7f090dbb5640 1 -- 172.21.15.70:0/3997698529 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f090819fde0 con 0x7f0908102290 2024-04-01T01:33:38.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.823+0000 7f090dbb5640 1 -- 172.21.15.70:0/3997698529 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09080fd6f0 con 0x7f0908102290 2024-04-01T01:33:38.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.824+0000 7f0904ff9640 1 -- 172.21.15.70:0/3997698529 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f08f8046020 con 0x7f0908102290 2024-04-01T01:33:38.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.825+0000 7f0904ff9640 1 --2- 172.21.15.70:0/3997698529 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08f40a2530 0x7f08f40a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:38.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.825+0000 7f0906ffd640 1 --2- 172.21.15.70:0/3997698529 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08f40a2530 0x7f08f40a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:38.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.825+0000 7f0904ff9640 1 -- 172.21.15.70:0/3997698529 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f08f80f0d20 con 0x7f0908102290 2024-04-01T01:33:38.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.825+0000 7f0906ffd640 1 --2- 172.21.15.70:0/3997698529 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08f40a2530 0x7f08f40a49f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0908198ef0 tx=0x7f08f0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:38.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:38.829+0000 7f0904ff9640 1 -- 172.21.15.70:0/3997698529 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f08f804d020 con 0x7f0908102290 2024-04-01T01:33:38.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:38 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2032866280' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-01T01:33:38.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:38 smithi189 ceph-mon[35753]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-01T01:33:38.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:38 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3190837660' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:39.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.003+0000 7f090dbb5640 1 -- 172.21.15.70:0/3997698529 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0908062350 con 0x7f0908102290 2024-04-01T01:33:39.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:38 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2032866280' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-01T01:33:39.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:38 smithi070 ceph-mon[32179]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-01T01:33:39.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:38 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3190837660' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:39.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.003+0000 7f0904ff9640 1 -- 172.21.15.70:0/3997698529 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f08f80bdac0 con 0x7f0908102290 2024-04-01T01:33:39.004 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:33:39.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.005+0000 7f090dbb5640 1 -- 172.21.15.70:0/3997698529 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08f40a2530 msgr2=0x7f08f40a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.005+0000 7f090dbb5640 1 --2- 172.21.15.70:0/3997698529 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08f40a2530 0x7f08f40a49f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0908198ef0 tx=0x7f08f0005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:39.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.005+0000 7f090dbb5640 1 -- 172.21.15.70:0/3997698529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0908102290 msgr2=0x7f09081979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.005+0000 7f090dbb5640 1 --2- 172.21.15.70:0/3997698529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0908102290 0x7f09081979e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f08f8006010 tx=0x7f08f8005e40 comp rx=0 tx=0).stop 2024-04-01T01:33:39.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.005+0000 7f090dbb5640 1 -- 172.21.15.70:0/3997698529 shutdown_connections 2024-04-01T01:33:39.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.005+0000 7f090dbb5640 1 --2- 172.21.15.70:0/3997698529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09081074f0 0x7f090819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.005+0000 7f090dbb5640 1 --2- 172.21.15.70:0/3997698529 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08f40a2530 0x7f08f40a49f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.005+0000 7f090dbb5640 1 --2- 172.21.15.70:0/3997698529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0908104bc0 0x7f0908197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.005+0000 7f090dbb5640 1 --2- 172.21.15.70:0/3997698529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0908102290 0x7f09081979e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.005+0000 7f090dbb5640 1 -- 172.21.15.70:0/3997698529 >> 172.21.15.70:0/3997698529 conn(0x7f09080fbc00 msgr2=0x7f09081040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:39.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.005+0000 7f090dbb5640 1 -- 172.21.15.70:0/3997698529 shutdown_connections 2024-04-01T01:33:39.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.005+0000 7f090dbb5640 1 -- 172.21.15.70:0/3997698529 wait complete. 2024-04-01T01:33:39.017 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-01T01:33:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:38 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2032866280' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-01T01:33:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:38 smithi081 ceph-mon[35142]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-01T01:33:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:38 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3190837660' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:39.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.121+0000 7f10c21a0640 1 -- 172.21.15.70:0/4151844761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10bc100440 msgr2=0x7f10bc100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.121+0000 7f10c21a0640 1 --2- 172.21.15.70:0/4151844761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10bc100440 0x7f10bc100860 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f10b4009900 tx=0x7f10b402bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:39.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c21a0640 1 -- 172.21.15.70:0/4151844761 shutdown_connections 2024-04-01T01:33:39.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c21a0640 1 --2- 172.21.15.70:0/4151844761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10bc101ff0 0x7f10bc108880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c21a0640 1 --2- 172.21.15.70:0/4151844761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10bc101630 0x7f10bc101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c21a0640 1 --2- 172.21.15.70:0/4151844761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10bc100440 0x7f10bc100860 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c21a0640 1 -- 172.21.15.70:0/4151844761 >> 172.21.15.70:0/4151844761 conn(0x7f10bc0fb9f0 msgr2=0x7f10bc0fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:39.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c21a0640 1 -- 172.21.15.70:0/4151844761 shutdown_connections 2024-04-01T01:33:39.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c21a0640 1 -- 172.21.15.70:0/4151844761 wait complete. 2024-04-01T01:33:39.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c21a0640 1 Processor -- start 2024-04-01T01:33:39.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c21a0640 1 -- start start 2024-04-01T01:33:39.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c21a0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10bc100440 0x7f10bc0693d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:39.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c21a0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10bc101630 0x7f10bc19e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:39.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c21a0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10bc101ff0 0x7f10bc1a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:39.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c21a0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f10bc10bf80 con 0x7f10bc101ff0 2024-04-01T01:33:39.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c21a0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f10bc10be00 con 0x7f10bc101630 2024-04-01T01:33:39.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c21a0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f10bc10c100 con 0x7f10bc100440 2024-04-01T01:33:39.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c199f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10bc101ff0 0x7f10bc1a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:39.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c119e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10bc100440 0x7f10bc0693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:39.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c119e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10bc100440 0x7f10bc0693d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40854/0 (socket says 172.21.15.70:40854) 2024-04-01T01:33:39.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c199f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10bc101ff0 0x7f10bc1a0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55292/0 (socket says 172.21.15.70:55292) 2024-04-01T01:33:39.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c119e640 1 -- 172.21.15.70:0/10921874 learned_addr learned my addr 172.21.15.70:0/10921874 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:39.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.122+0000 7f10c099d640 1 --2- 172.21.15.70:0/10921874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10bc101630 0x7f10bc19e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:39.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.123+0000 7f10c119e640 1 -- 172.21.15.70:0/10921874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10bc101630 msgr2=0x7f10bc19e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.123+0000 7f10c119e640 1 --2- 172.21.15.70:0/10921874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10bc101630 0x7f10bc19e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.123+0000 7f10c119e640 1 -- 172.21.15.70:0/10921874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10bc101ff0 msgr2=0x7f10bc1a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.123+0000 7f10c119e640 1 --2- 172.21.15.70:0/10921874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10bc101ff0 0x7f10bc1a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.123+0000 7f10c119e640 1 -- 172.21.15.70:0/10921874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10bc1a1580 con 0x7f10bc100440 2024-04-01T01:33:39.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.123+0000 7f10c199f640 1 --2- 172.21.15.70:0/10921874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10bc101ff0 0x7f10bc1a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:39.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.123+0000 7f10c119e640 1 --2- 172.21.15.70:0/10921874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10bc100440 0x7f10bc0693d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f10b4000c00 tx=0x7f10b4005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:39.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.123+0000 7f10b27fc640 1 -- 172.21.15.70:0/10921874 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b4043020 con 0x7f10bc100440 2024-04-01T01:33:39.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.123+0000 7f10c21a0640 1 -- 172.21.15.70:0/10921874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10bc1a1810 con 0x7f10bc100440 2024-04-01T01:33:39.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.123+0000 7f10b27fc640 1 -- 172.21.15.70:0/10921874 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f10b4004350 con 0x7f10bc100440 2024-04-01T01:33:39.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.123+0000 7f10b27fc640 1 -- 172.21.15.70:0/10921874 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b4003b20 con 0x7f10bc100440 2024-04-01T01:33:39.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.123+0000 7f10c21a0640 1 -- 172.21.15.70:0/10921874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10bc1a1db0 con 0x7f10bc100440 2024-04-01T01:33:39.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.124+0000 7f10c21a0640 1 -- 172.21.15.70:0/10921874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10bc107780 con 0x7f10bc100440 2024-04-01T01:33:39.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.125+0000 7f10b27fc640 1 -- 172.21.15.70:0/10921874 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f10b4046020 con 0x7f10bc100440 2024-04-01T01:33:39.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.125+0000 7f10b27fc640 1 --2- 172.21.15.70:0/10921874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f10a80a2560 0x7f10a80a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:39.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.126+0000 7f10c099d640 1 --2- 172.21.15.70:0/10921874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f10a80a2560 0x7f10a80a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:39.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.126+0000 7f10b27fc640 1 -- 172.21.15.70:0/10921874 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f10b40f1130 con 0x7f10bc100440 2024-04-01T01:33:39.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.126+0000 7f10c099d640 1 --2- 172.21.15.70:0/10921874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f10a80a2560 0x7f10a80a4a20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f10bc069140 tx=0x7f10ac005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:39.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.129+0000 7f10b27fc640 1 -- 172.21.15.70:0/10921874 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f10b404d020 con 0x7f10bc100440 2024-04-01T01:33:39.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.301+0000 7f10c21a0640 1 -- 172.21.15.70:0/10921874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f10bc02d300 con 0x7f10bc100440 2024-04-01T01:33:39.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.302+0000 7f10b27fc640 1 -- 172.21.15.70:0/10921874 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f10b40dd030 con 0x7f10bc100440 2024-04-01T01:33:39.302 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-01T01:33:39.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.303+0000 7f10c21a0640 1 -- 172.21.15.70:0/10921874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f10a80a2560 msgr2=0x7f10a80a4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.304+0000 7f10c21a0640 1 --2- 172.21.15.70:0/10921874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f10a80a2560 0x7f10a80a4a20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f10bc069140 tx=0x7f10ac005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:39.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.304+0000 7f10c21a0640 1 -- 172.21.15.70:0/10921874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10bc100440 msgr2=0x7f10bc0693d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.304+0000 7f10c21a0640 1 --2- 172.21.15.70:0/10921874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10bc100440 0x7f10bc0693d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f10b4000c00 tx=0x7f10b4005dc0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.304+0000 7f10c21a0640 1 -- 172.21.15.70:0/10921874 shutdown_connections 2024-04-01T01:33:39.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.304+0000 7f10c21a0640 1 --2- 172.21.15.70:0/10921874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f10bc101ff0 0x7f10bc1a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.304+0000 7f10c21a0640 1 --2- 172.21.15.70:0/10921874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f10bc101630 0x7f10bc19e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.304+0000 7f10c21a0640 1 --2- 172.21.15.70:0/10921874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f10a80a2560 0x7f10a80a4a20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.304+0000 7f10c21a0640 1 --2- 172.21.15.70:0/10921874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10bc100440 0x7f10bc0693d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.304+0000 7f10c21a0640 1 -- 172.21.15.70:0/10921874 >> 172.21.15.70:0/10921874 conn(0x7f10bc0fb9f0 msgr2=0x7f10bc104950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:39.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.304+0000 7f10c21a0640 1 -- 172.21.15.70:0/10921874 shutdown_connections 2024-04-01T01:33:39.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.304+0000 7f10c21a0640 1 -- 172.21.15.70:0/10921874 wait complete. 2024-04-01T01:33:39.315 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-01T01:33:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.420+0000 7f15e260f640 1 -- 172.21.15.70:0/2137740782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15dc103310 msgr2=0x7f15dc109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.420+0000 7f15e260f640 1 --2- 172.21.15.70:0/2137740782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15dc103310 0x7f15dc109ba0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009ca0 tx=0x7f15d402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.420+0000 7f15e260f640 1 -- 172.21.15.70:0/2137740782 shutdown_connections 2024-04-01T01:33:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.420+0000 7f15e260f640 1 --2- 172.21.15.70:0/2137740782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15dc103310 0x7f15dc109ba0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.420+0000 7f15e260f640 1 --2- 172.21.15.70:0/2137740782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15dc102950 0x7f15dc102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.420+0000 7f15e260f640 1 --2- 172.21.15.70:0/2137740782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15dc101760 0x7f15dc101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.420+0000 7f15e260f640 1 -- 172.21.15.70:0/2137740782 >> 172.21.15.70:0/2137740782 conn(0x7f15dc0fccf0 msgr2=0x7f15dc0ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:39.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.420+0000 7f15e260f640 1 -- 172.21.15.70:0/2137740782 shutdown_connections 2024-04-01T01:33:39.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.420+0000 7f15e260f640 1 -- 172.21.15.70:0/2137740782 wait complete. 2024-04-01T01:33:39.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.420+0000 7f15e260f640 1 Processor -- start 2024-04-01T01:33:39.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.420+0000 7f15e260f640 1 -- start start 2024-04-01T01:33:39.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e260f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15dc101760 0x7f15dc075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:39.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e260f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15dc102950 0x7f15dc073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:39.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e260f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15dc103310 0x7f15dc074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:39.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e260f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f15dc10d270 con 0x7f15dc103310 2024-04-01T01:33:39.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e260f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f15dc10d0f0 con 0x7f15dc102950 2024-04-01T01:33:39.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e260f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f15dc10d3f0 con 0x7f15dc101760 2024-04-01T01:33:39.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e0bac640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15dc103310 0x7f15dc074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:39.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e0bac640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15dc103310 0x7f15dc074080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55312/0 (socket says 172.21.15.70:55312) 2024-04-01T01:33:39.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e0bac640 1 -- 172.21.15.70:0/1758661026 learned_addr learned my addr 172.21.15.70:0/1758661026 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15dbfff640 1 --2- 172.21.15.70:0/1758661026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15dc101760 0x7f15dc075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15db7fe640 1 --2- 172.21.15.70:0/1758661026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15dc102950 0x7f15dc073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e0bac640 1 -- 172.21.15.70:0/1758661026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15dc101760 msgr2=0x7f15dc075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e0bac640 1 --2- 172.21.15.70:0/1758661026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15dc101760 0x7f15dc075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e0bac640 1 -- 172.21.15.70:0/1758661026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15dc102950 msgr2=0x7f15dc073b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e0bac640 1 --2- 172.21.15.70:0/1758661026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15dc102950 0x7f15dc073b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e0bac640 1 -- 172.21.15.70:0/1758661026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15dc074910 con 0x7f15dc103310 2024-04-01T01:33:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e0bac640 1 --2- 172.21.15.70:0/1758661026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15dc103310 0x7f15dc074080 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f15d402c700 tx=0x7f15d40328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15d97fa640 1 -- 172.21.15.70:0/1758661026 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d4041020 con 0x7f15dc103310 2024-04-01T01:33:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e260f640 1 -- 172.21.15.70:0/1758661026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15dc1a1a00 con 0x7f15dc103310 2024-04-01T01:33:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15d97fa640 1 -- 172.21.15.70:0/1758661026 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f15d4003e20 con 0x7f15dc103310 2024-04-01T01:33:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15d97fa640 1 -- 172.21.15.70:0/1758661026 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d4039040 con 0x7f15dc103310 2024-04-01T01:33:39.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.421+0000 7f15e260f640 1 -- 172.21.15.70:0/1758661026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15dc1a1f50 con 0x7f15dc103310 2024-04-01T01:33:39.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.423+0000 7f15e260f640 1 -- 172.21.15.70:0/1758661026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15dc108b20 con 0x7f15dc103310 2024-04-01T01:33:39.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.423+0000 7f15d97fa640 1 -- 172.21.15.70:0/1758661026 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f15d4044020 con 0x7f15dc103310 2024-04-01T01:33:39.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.424+0000 7f15d97fa640 1 --2- 172.21.15.70:0/1758661026 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f15c40a2530 0x7f15c40a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:39.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.424+0000 7f15dbfff640 1 --2- 172.21.15.70:0/1758661026 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f15c40a2530 0x7f15c40a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:39.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.424+0000 7f15d97fa640 1 -- 172.21.15.70:0/1758661026 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f15d40f0d00 con 0x7f15dc103310 2024-04-01T01:33:39.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.425+0000 7f15dbfff640 1 --2- 172.21.15.70:0/1758661026 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f15c40a2530 0x7f15c40a49f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f15cc0095f0 tx=0x7f15cc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:39.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.429+0000 7f15d97fa640 1 -- 172.21.15.70:0/1758661026 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f15d404b020 con 0x7f15dc103310 2024-04-01T01:33:39.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.603+0000 7f15e260f640 1 -- 172.21.15.70:0/1758661026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f15dc062350 con 0x7f15dc103310 2024-04-01T01:33:39.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.604+0000 7f15d97fa640 1 -- 172.21.15.70:0/1758661026 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f15d40bdaa0 con 0x7f15dc103310 2024-04-01T01:33:39.604 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:33:39.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.606+0000 7f15e260f640 1 -- 172.21.15.70:0/1758661026 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f15c40a2530 msgr2=0x7f15c40a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.606+0000 7f15e260f640 1 --2- 172.21.15.70:0/1758661026 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f15c40a2530 0x7f15c40a49f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f15cc0095f0 tx=0x7f15cc005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:39.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.606+0000 7f15e260f640 1 -- 172.21.15.70:0/1758661026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15dc103310 msgr2=0x7f15dc074080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.606+0000 7f15e260f640 1 --2- 172.21.15.70:0/1758661026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15dc103310 0x7f15dc074080 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f15d402c700 tx=0x7f15d40328a0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.606+0000 7f15e260f640 1 -- 172.21.15.70:0/1758661026 shutdown_connections 2024-04-01T01:33:39.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.606+0000 7f15e260f640 1 --2- 172.21.15.70:0/1758661026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15dc103310 0x7f15dc074080 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.606+0000 7f15e260f640 1 --2- 172.21.15.70:0/1758661026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15dc102950 0x7f15dc073b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.606+0000 7f15e260f640 1 --2- 172.21.15.70:0/1758661026 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f15c40a2530 0x7f15c40a49f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.606+0000 7f15e260f640 1 --2- 172.21.15.70:0/1758661026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15dc101760 0x7f15dc075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.606+0000 7f15e260f640 1 -- 172.21.15.70:0/1758661026 >> 172.21.15.70:0/1758661026 conn(0x7f15dc0fccf0 msgr2=0x7f15dc105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:39.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.606+0000 7f15e260f640 1 -- 172.21.15.70:0/1758661026 shutdown_connections 2024-04-01T01:33:39.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.606+0000 7f15e260f640 1 -- 172.21.15.70:0/1758661026 wait complete. 2024-04-01T01:33:39.617 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:33:39.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.721+0000 7f36d086b640 1 -- 172.21.15.70:0/1667824919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36cc102220 msgr2=0x7f36cc108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.721+0000 7f36d086b640 1 --2- 172.21.15.70:0/1667824919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36cc102220 0x7f36cc108ab0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f36c4009ca0 tx=0x7f36c402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.721+0000 7f36d086b640 1 -- 172.21.15.70:0/1667824919 shutdown_connections 2024-04-01T01:33:39.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.721+0000 7f36d086b640 1 --2- 172.21.15.70:0/1667824919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36cc102220 0x7f36cc108ab0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.721+0000 7f36d086b640 1 --2- 172.21.15.70:0/1667824919 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36cc101860 0x7f36cc101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.721+0000 7f36d086b640 1 --2- 172.21.15.70:0/1667824919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36cc100670 0x7f36cc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.721+0000 7f36d086b640 1 -- 172.21.15.70:0/1667824919 >> 172.21.15.70:0/1667824919 conn(0x7f36cc0fbbe0 msgr2=0x7f36cc0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:39.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.721+0000 7f36d086b640 1 -- 172.21.15.70:0/1667824919 shutdown_connections 2024-04-01T01:33:39.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.721+0000 7f36d086b640 1 -- 172.21.15.70:0/1667824919 wait complete. 2024-04-01T01:33:39.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.722+0000 7f36d086b640 1 Processor -- start 2024-04-01T01:33:39.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.722+0000 7f36d086b640 1 -- start start 2024-04-01T01:33:39.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.722+0000 7f36d086b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36cc100670 0x7f36cc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:39.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.722+0000 7f36d086b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36cc101860 0x7f36cc19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:39.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.722+0000 7f36d086b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36cc102220 0x7f36cc1a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:39.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.722+0000 7f36d086b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f36cc10c1a0 con 0x7f36cc101860 2024-04-01T01:33:39.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.722+0000 7f36d086b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f36cc10c020 con 0x7f36cc100670 2024-04-01T01:33:39.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.722+0000 7f36d086b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f36cc10c320 con 0x7f36cc102220 2024-04-01T01:33:39.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.722+0000 7f36ca59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36cc100670 0x7f36cc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:39.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.722+0000 7f36c3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36cc101860 0x7f36cc19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:39.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.722+0000 7f36ca59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36cc100670 0x7f36cc069420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34486/0 (socket says 172.21.15.70:34486) 2024-04-01T01:33:39.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.722+0000 7f36ca59c640 1 -- 172.21.15.70:0/4120953937 learned_addr learned my addr 172.21.15.70:0/4120953937 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:39.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.722+0000 7f36cad9d640 1 --2- 172.21.15.70:0/4120953937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36cc102220 0x7f36cc1a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:39.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.723+0000 7f36ca59c640 1 -- 172.21.15.70:0/4120953937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36cc102220 msgr2=0x7f36cc1a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.723+0000 7f36ca59c640 1 --2- 172.21.15.70:0/4120953937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36cc102220 0x7f36cc1a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.723+0000 7f36ca59c640 1 -- 172.21.15.70:0/4120953937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36cc101860 msgr2=0x7f36cc19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.723+0000 7f36ca59c640 1 --2- 172.21.15.70:0/4120953937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36cc101860 0x7f36cc19eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.723+0000 7f36ca59c640 1 -- 172.21.15.70:0/4120953937 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36cc1a1800 con 0x7f36cc100670 2024-04-01T01:33:39.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.723+0000 7f36cad9d640 1 --2- 172.21.15.70:0/4120953937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36cc102220 0x7f36cc1a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:33:39.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.723+0000 7f36c3fff640 1 --2- 172.21.15.70:0/4120953937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36cc101860 0x7f36cc19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:39.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.723+0000 7f36ca59c640 1 --2- 172.21.15.70:0/4120953937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36cc100670 0x7f36cc069420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f36bc00e9c0 tx=0x7f36bc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:39.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.723+0000 7f36c37fe640 1 -- 172.21.15.70:0/4120953937 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc00cd80 con 0x7f36cc100670 2024-04-01T01:33:39.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.723+0000 7f36d086b640 1 -- 172.21.15.70:0/4120953937 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36cc1a1a90 con 0x7f36cc100670 2024-04-01T01:33:39.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.723+0000 7f36c37fe640 1 -- 172.21.15.70:0/4120953937 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36bc0035f0 con 0x7f36cc100670 2024-04-01T01:33:39.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.723+0000 7f36c37fe640 1 -- 172.21.15.70:0/4120953937 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc010630 con 0x7f36cc100670 2024-04-01T01:33:39.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.723+0000 7f36d086b640 1 -- 172.21.15.70:0/4120953937 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36cc1a1fe0 con 0x7f36cc100670 2024-04-01T01:33:39.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.724+0000 7f36d086b640 1 -- 172.21.15.70:0/4120953937 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36cc1079e0 con 0x7f36cc100670 2024-04-01T01:33:39.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.725+0000 7f36c37fe640 1 -- 172.21.15.70:0/4120953937 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f36bc0037a0 con 0x7f36cc100670 2024-04-01T01:33:39.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.726+0000 7f36c37fe640 1 --2- 172.21.15.70:0/4120953937 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36b80a2560 0x7f36b80a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:39.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.726+0000 7f36c3fff640 1 --2- 172.21.15.70:0/4120953937 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36b80a2560 0x7f36b80a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.726+0000 7f36c37fe640 1 -- 172.21.15.70:0/4120953937 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f36bc0cf1f0 con 0x7f36cc100670 2024-04-01T01:33:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.726+0000 7f36c3fff640 1 --2- 172.21.15.70:0/4120953937 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36b80a2560 0x7f36b80a4a20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f36b40095f0 tx=0x7f36b4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:39.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.734+0000 7f36c37fe640 1 -- 172.21.15.70:0/4120953937 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f36bc09be80 con 0x7f36cc100670 2024-04-01T01:33:39.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.912+0000 7f36d086b640 1 -- 172.21.15.70:0/4120953937 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f36cc02d330 con 0x7f36cc100670 2024-04-01T01:33:39.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.914+0000 7f36c37fe640 1 -- 172.21.15.70:0/4120953937 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24781 (secure 0 0 0) 0x7f36bc09b780 con 0x7f36cc100670 2024-04-01T01:33:39.914 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:39.915 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":93,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:33:19.716245+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:33:39.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.916+0000 7f36d086b640 1 -- 172.21.15.70:0/4120953937 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36b80a2560 msgr2=0x7f36b80a4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.916+0000 7f36d086b640 1 --2- 172.21.15.70:0/4120953937 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36b80a2560 0x7f36b80a4a20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f36b40095f0 tx=0x7f36b4005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:39.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.916+0000 7f36d086b640 1 -- 172.21.15.70:0/4120953937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36cc100670 msgr2=0x7f36cc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:39.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.916+0000 7f36d086b640 1 --2- 172.21.15.70:0/4120953937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36cc100670 0x7f36cc069420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f36bc00e9c0 tx=0x7f36bc00ee80 comp rx=0 tx=0).stop 2024-04-01T01:33:39.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.916+0000 7f36d086b640 1 -- 172.21.15.70:0/4120953937 shutdown_connections 2024-04-01T01:33:39.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.916+0000 7f36d086b640 1 --2- 172.21.15.70:0/4120953937 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36b80a2560 0x7f36b80a4a20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.916+0000 7f36d086b640 1 --2- 172.21.15.70:0/4120953937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36cc102220 0x7f36cc1a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.916+0000 7f36d086b640 1 --2- 172.21.15.70:0/4120953937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36cc101860 0x7f36cc19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.916+0000 7f36d086b640 1 --2- 172.21.15.70:0/4120953937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36cc100670 0x7f36cc069420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:39.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.916+0000 7f36d086b640 1 -- 172.21.15.70:0/4120953937 >> 172.21.15.70:0/4120953937 conn(0x7f36cc0fbbe0 msgr2=0x7f36cc104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:39.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.916+0000 7f36d086b640 1 -- 172.21.15.70:0/4120953937 shutdown_connections 2024-04-01T01:33:39.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:39.916+0000 7f36d086b640 1 -- 172.21.15.70:0/4120953937 wait complete. 2024-04-01T01:33:39.926 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:39 smithi070 ceph-mon[32179]: from='client.15525 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:39.926 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:39 smithi070 ceph-mon[32179]: pgmap v363: 129 pgs: 129 active+clean; 649 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-01T01:33:39.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:39 smithi070 ceph-mon[32179]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-01T01:33:39.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:39 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3664309242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:39.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:39 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3997698529' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:39.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:39 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/10921874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:39.927 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:39 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1758661026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:39.928 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-01T01:26:42.809779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-01T01:31:13.612369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-01T01:31:14.910162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-01T01:31:18.783179+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-01T01:33:39.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-01T01:33:39.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:39 smithi189 ceph-mon[35753]: from='client.15525 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:39.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:39 smithi189 ceph-mon[35753]: pgmap v363: 129 pgs: 129 active+clean; 649 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-01T01:33:39.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:39 smithi189 ceph-mon[35753]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-01T01:33:39.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:39 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3664309242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:39.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:39 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3997698529' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:39.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:39 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/10921874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:39.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:39 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1758661026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:40.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 -- 172.21.15.70:0/966217395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e4102220 msgr2=0x7f52e4108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 --2- 172.21.15.70:0/966217395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e4102220 0x7f52e4108ab0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009ca0 tx=0x7f52dc02c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 -- 172.21.15.70:0/966217395 shutdown_connections 2024-04-01T01:33:40.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 --2- 172.21.15.70:0/966217395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e4102220 0x7f52e4108ab0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 --2- 172.21.15.70:0/966217395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52e4101860 0x7f52e4101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 --2- 172.21.15.70:0/966217395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52e4100670 0x7f52e4100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 -- 172.21.15.70:0/966217395 >> 172.21.15.70:0/966217395 conn(0x7f52e40fbbe0 msgr2=0x7f52e40fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:40.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 -- 172.21.15.70:0/966217395 shutdown_connections 2024-04-01T01:33:40.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 -- 172.21.15.70:0/966217395 wait complete. 2024-04-01T01:33:40.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 Processor -- start 2024-04-01T01:33:40.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 -- start start 2024-04-01T01:33:40.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52e4100670 0x7f52e410aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:40.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52e4101860 0x7f52e4109630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:40.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e4102220 0x7f52e4109b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:40.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f52e410d2c0 con 0x7f52e4102220 2024-04-01T01:33:40.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f52e410d140 con 0x7f52e4101860 2024-04-01T01:33:40.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.034+0000 7f52ebc4e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f52e410d440 con 0x7f52e4100670 2024-04-01T01:33:40.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52ea1eb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e4102220 0x7f52e4109b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:40.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52e99ea640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52e4100670 0x7f52e410aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:40.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52e99ea640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52e4100670 0x7f52e410aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40914/0 (socket says 172.21.15.70:40914) 2024-04-01T01:33:40.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52e99ea640 1 -- 172.21.15.70:0/1702105690 learned_addr learned my addr 172.21.15.70:0/1702105690 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:40.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52ea1eb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e4102220 0x7f52e4109b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55344/0 (socket says 172.21.15.70:55344) 2024-04-01T01:33:40.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52e91e9640 1 --2- 172.21.15.70:0/1702105690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52e4101860 0x7f52e4109630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:40.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52ea1eb640 1 -- 172.21.15.70:0/1702105690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52e4100670 msgr2=0x7f52e410aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52ea1eb640 1 --2- 172.21.15.70:0/1702105690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52e4100670 0x7f52e410aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52ea1eb640 1 -- 172.21.15.70:0/1702105690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52e4101860 msgr2=0x7f52e4109630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52ea1eb640 1 --2- 172.21.15.70:0/1702105690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52e4101860 0x7f52e4109630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52ea1eb640 1 -- 172.21.15.70:0/1702105690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e410a400 con 0x7f52e4102220 2024-04-01T01:33:40.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52e91e9640 1 --2- 172.21.15.70:0/1702105690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52e4101860 0x7f52e4109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:40.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52ea1eb640 1 --2- 172.21.15.70:0/1702105690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e4102220 0x7f52e4109b70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f52dc02c630 tx=0x7f52dc0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:40.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52daffd640 1 -- 172.21.15.70:0/1702105690 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc041020 con 0x7f52e4102220 2024-04-01T01:33:40.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52daffd640 1 -- 172.21.15.70:0/1702105690 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f52dc003e20 con 0x7f52e4102220 2024-04-01T01:33:40.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52ebc4e640 1 -- 172.21.15.70:0/1702105690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52e41a19d0 con 0x7f52e4102220 2024-04-01T01:33:40.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52daffd640 1 -- 172.21.15.70:0/1702105690 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc039040 con 0x7f52e4102220 2024-04-01T01:33:40.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.035+0000 7f52ebc4e640 1 -- 172.21.15.70:0/1702105690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52e41a1f20 con 0x7f52e4102220 2024-04-01T01:33:40.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.036+0000 7f52ebc4e640 1 -- 172.21.15.70:0/1702105690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52e4107a10 con 0x7f52e4102220 2024-04-01T01:33:40.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.037+0000 7f52daffd640 1 -- 172.21.15.70:0/1702105690 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f52dc044020 con 0x7f52e4102220 2024-04-01T01:33:40.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.038+0000 7f52daffd640 1 --2- 172.21.15.70:0/1702105690 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f52d00a2530 0x7f52d00a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:40.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.038+0000 7f52e99ea640 1 --2- 172.21.15.70:0/1702105690 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f52d00a2530 0x7f52d00a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:40.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.038+0000 7f52daffd640 1 -- 172.21.15.70:0/1702105690 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f52dc0f0cb0 con 0x7f52e4102220 2024-04-01T01:33:40.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.038+0000 7f52e99ea640 1 --2- 172.21.15.70:0/1702105690 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f52d00a2530 0x7f52d00a49f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f52e00058c0 tx=0x7f52e0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:40.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.042+0000 7f52daffd640 1 -- 172.21.15.70:0/1702105690 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f52dc04b020 con 0x7f52e4102220 2024-04-01T01:33:40.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:39 smithi081 ceph-mon[35142]: from='client.15525 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-01T01:33:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:39 smithi081 ceph-mon[35142]: pgmap v363: 129 pgs: 129 active+clean; 649 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-01T01:33:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:39 smithi081 ceph-mon[35142]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-01T01:33:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:39 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3664309242' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:39 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3997698529' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:39 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/10921874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:39 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1758661026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:40.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.215+0000 7f52ebc4e640 1 -- 172.21.15.70:0/1702105690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f52e4062350 con 0x7f52e4102220 2024-04-01T01:33:40.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.216+0000 7f52daffd640 1 -- 172.21.15.70:0/1702105690 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f52dc0bda50 con 0x7f52e4102220 2024-04-01T01:33:40.216 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-01T01:33:40.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.218+0000 7f52ebc4e640 1 -- 172.21.15.70:0/1702105690 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f52d00a2530 msgr2=0x7f52d00a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.218+0000 7f52ebc4e640 1 --2- 172.21.15.70:0/1702105690 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f52d00a2530 0x7f52d00a49f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f52e00058c0 tx=0x7f52e0005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:40.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.218+0000 7f52ebc4e640 1 -- 172.21.15.70:0/1702105690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e4102220 msgr2=0x7f52e4109b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.218+0000 7f52ebc4e640 1 --2- 172.21.15.70:0/1702105690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e4102220 0x7f52e4109b70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f52dc02c630 tx=0x7f52dc0328a0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.218+0000 7f52ebc4e640 1 -- 172.21.15.70:0/1702105690 shutdown_connections 2024-04-01T01:33:40.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.218+0000 7f52ebc4e640 1 --2- 172.21.15.70:0/1702105690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52e4102220 0x7f52e4109b70 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.218+0000 7f52ebc4e640 1 --2- 172.21.15.70:0/1702105690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52e4101860 0x7f52e4109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.218+0000 7f52ebc4e640 1 --2- 172.21.15.70:0/1702105690 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f52d00a2530 0x7f52d00a49f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.218+0000 7f52ebc4e640 1 --2- 172.21.15.70:0/1702105690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52e4100670 0x7f52e410aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.218+0000 7f52ebc4e640 1 -- 172.21.15.70:0/1702105690 >> 172.21.15.70:0/1702105690 conn(0x7f52e40fbbe0 msgr2=0x7f52e4104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:40.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.218+0000 7f52ebc4e640 1 -- 172.21.15.70:0/1702105690 shutdown_connections 2024-04-01T01:33:40.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.218+0000 7f52ebc4e640 1 -- 172.21.15.70:0/1702105690 wait complete. 2024-04-01T01:33:40.229 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-01T01:33:40.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.334+0000 7fea8f440640 1 -- 172.21.15.70:0/4276375911 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea88100660 msgr2=0x7fea88100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.334+0000 7fea8f440640 1 --2- 172.21.15.70:0/4276375911 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea88100660 0x7fea88100a80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fea80009be0 tx=0x7fea8002bef0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.334+0000 7fea8f440640 1 -- 172.21.15.70:0/4276375911 shutdown_connections 2024-04-01T01:33:40.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.334+0000 7fea8f440640 1 --2- 172.21.15.70:0/4276375911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea88102210 0x7fea88108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.334+0000 7fea8f440640 1 --2- 172.21.15.70:0/4276375911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea88101850 0x7fea88101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.334+0000 7fea8f440640 1 --2- 172.21.15.70:0/4276375911 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea88100660 0x7fea88100a80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.334+0000 7fea8f440640 1 -- 172.21.15.70:0/4276375911 >> 172.21.15.70:0/4276375911 conn(0x7fea880fbbf0 msgr2=0x7fea880fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:40.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.334+0000 7fea8f440640 1 -- 172.21.15.70:0/4276375911 shutdown_connections 2024-04-01T01:33:40.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.334+0000 7fea8f440640 1 -- 172.21.15.70:0/4276375911 wait complete. 2024-04-01T01:33:40.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8f440640 1 Processor -- start 2024-04-01T01:33:40.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8f440640 1 -- start start 2024-04-01T01:33:40.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8f440640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea88100660 0x7fea881968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:40.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8f440640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea88101850 0x7fea88196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:40.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8f440640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea88102210 0x7fea8819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:40.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8f440640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fea8810c190 con 0x7fea88102210 2024-04-01T01:33:40.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8f440640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fea8810c010 con 0x7fea88100660 2024-04-01T01:33:40.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8f440640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fea8810c310 con 0x7fea88101850 2024-04-01T01:33:40.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8d1dc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea88100660 0x7fea881968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:40.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8d9dd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea88102210 0x7fea8819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:40.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8d1dc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea88100660 0x7fea881968f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34528/0 (socket says 172.21.15.70:34528) 2024-04-01T01:33:40.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8d1dc640 1 -- 172.21.15.70:0/28757300 learned_addr learned my addr 172.21.15.70:0/28757300 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:40.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8c9db640 1 --2- 172.21.15.70:0/28757300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea88101850 0x7fea88196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:40.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8d1dc640 1 -- 172.21.15.70:0/28757300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea88101850 msgr2=0x7fea88196e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8d1dc640 1 --2- 172.21.15.70:0/28757300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea88101850 0x7fea88196e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8d1dc640 1 -- 172.21.15.70:0/28757300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea88102210 msgr2=0x7fea8819dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8d1dc640 1 --2- 172.21.15.70:0/28757300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea88102210 0x7fea8819dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8d1dc640 1 -- 172.21.15.70:0/28757300 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8819e510 con 0x7fea88100660 2024-04-01T01:33:40.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.335+0000 7fea8d9dd640 1 --2- 172.21.15.70:0/28757300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea88102210 0x7fea8819dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:40.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.336+0000 7fea8d1dc640 1 --2- 172.21.15.70:0/28757300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea88100660 0x7fea881968f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fea80000f80 tx=0x7fea80033a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:40.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.336+0000 7fea867fc640 1 -- 172.21.15.70:0/28757300 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea80043020 con 0x7fea88100660 2024-04-01T01:33:40.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.336+0000 7fea867fc640 1 -- 172.21.15.70:0/28757300 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea80033e60 con 0x7fea88100660 2024-04-01T01:33:40.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.336+0000 7fea8f440640 1 -- 172.21.15.70:0/28757300 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea8819e7a0 con 0x7fea88100660 2024-04-01T01:33:40.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.336+0000 7fea867fc640 1 -- 172.21.15.70:0/28757300 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea80034430 con 0x7fea88100660 2024-04-01T01:33:40.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.336+0000 7fea8f440640 1 -- 172.21.15.70:0/28757300 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea8819ecf0 con 0x7fea88100660 2024-04-01T01:33:40.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.337+0000 7fea8f440640 1 -- 172.21.15.70:0/28757300 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea88107af0 con 0x7fea88100660 2024-04-01T01:33:40.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.338+0000 7fea867fc640 1 -- 172.21.15.70:0/28757300 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7fea80046020 con 0x7fea88100660 2024-04-01T01:33:40.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.338+0000 7fea867fc640 1 --2- 172.21.15.70:0/28757300 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fea700a2530 0x7fea700a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:40.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.338+0000 7fea8c9db640 1 --2- 172.21.15.70:0/28757300 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fea700a2530 0x7fea700a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:40.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.338+0000 7fea867fc640 1 -- 172.21.15.70:0/28757300 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fea800f1230 con 0x7fea88100660 2024-04-01T01:33:40.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.339+0000 7fea8c9db640 1 --2- 172.21.15.70:0/28757300 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fea700a2530 0x7fea700a49f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fea88197e00 tx=0x7fea7c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:40.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.343+0000 7fea867fc640 1 -- 172.21.15.70:0/28757300 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fea8004d020 con 0x7fea88100660 2024-04-01T01:33:40.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.520+0000 7fea8f440640 1 -- 172.21.15.70:0/28757300 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fea88062350 con 0x7fea88100660 2024-04-01T01:33:40.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.520+0000 7fea867fc640 1 -- 172.21.15.70:0/28757300 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fea800bdfd0 con 0x7fea88100660 2024-04-01T01:33:40.521 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:33:40.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.522+0000 7fea8f440640 1 -- 172.21.15.70:0/28757300 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fea700a2530 msgr2=0x7fea700a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.522+0000 7fea8f440640 1 --2- 172.21.15.70:0/28757300 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fea700a2530 0x7fea700a49f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fea88197e00 tx=0x7fea7c009260 comp rx=0 tx=0).stop 2024-04-01T01:33:40.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.522+0000 7fea8f440640 1 -- 172.21.15.70:0/28757300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea88100660 msgr2=0x7fea881968f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.522+0000 7fea8f440640 1 --2- 172.21.15.70:0/28757300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea88100660 0x7fea881968f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fea80000f80 tx=0x7fea80033a60 comp rx=0 tx=0).stop 2024-04-01T01:33:40.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.522+0000 7fea8f440640 1 -- 172.21.15.70:0/28757300 shutdown_connections 2024-04-01T01:33:40.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.522+0000 7fea8f440640 1 --2- 172.21.15.70:0/28757300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea88102210 0x7fea8819dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.522+0000 7fea8f440640 1 --2- 172.21.15.70:0/28757300 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fea700a2530 0x7fea700a49f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.522+0000 7fea8f440640 1 --2- 172.21.15.70:0/28757300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea88101850 0x7fea88196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.522+0000 7fea8f440640 1 --2- 172.21.15.70:0/28757300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea88100660 0x7fea881968f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.522+0000 7fea8f440640 1 -- 172.21.15.70:0/28757300 >> 172.21.15.70:0/28757300 conn(0x7fea880fbbf0 msgr2=0x7fea88104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:40.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.522+0000 7fea8f440640 1 -- 172.21.15.70:0/28757300 shutdown_connections 2024-04-01T01:33:40.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.522+0000 7fea8f440640 1 -- 172.21.15.70:0/28757300 wait complete. 2024-04-01T01:33:40.534 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-01T01:33:40.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.639+0000 7f409789d640 1 -- 172.21.15.70:0/2858030178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40900ffe20 msgr2=0x7f4090100240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.639+0000 7f409789d640 1 --2- 172.21.15.70:0/2858030178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40900ffe20 0x7f4090100240 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f408c009900 tx=0x7f408c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.639+0000 7f409789d640 1 -- 172.21.15.70:0/2858030178 shutdown_connections 2024-04-01T01:33:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.639+0000 7f409789d640 1 --2- 172.21.15.70:0/2858030178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4090101930 0x7f4090108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.639+0000 7f409789d640 1 --2- 172.21.15.70:0/2858030178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4090100f70 0x7f40901013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.639+0000 7f409789d640 1 --2- 172.21.15.70:0/2858030178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40900ffe20 0x7f4090100240 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.639+0000 7f409789d640 1 -- 172.21.15.70:0/2858030178 >> 172.21.15.70:0/2858030178 conn(0x7f40900fb390 msgr2=0x7f40900fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.639+0000 7f409789d640 1 -- 172.21.15.70:0/2858030178 shutdown_connections 2024-04-01T01:33:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.639+0000 7f409789d640 1 -- 172.21.15.70:0/2858030178 wait complete. 2024-04-01T01:33:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f409789d640 1 Processor -- start 2024-04-01T01:33:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f409789d640 1 -- start start 2024-04-01T01:33:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f409789d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40900ffe20 0x7f4090195750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f409789d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4090100f70 0x7f4090195c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f409789d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4090101930 0x7f409019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f409789d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f409010b0e0 con 0x7f40900ffe20 2024-04-01T01:33:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f409789d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f409010af60 con 0x7f4090101930 2024-04-01T01:33:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f409789d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f409010b260 con 0x7f4090100f70 2024-04-01T01:33:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f4095639640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40900ffe20 0x7f4090195750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f4095639640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40900ffe20 0x7f4090195750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55378/0 (socket says 172.21.15.70:55378) 2024-04-01T01:33:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f4095639640 1 -- 172.21.15.70:0/432891716 learned_addr learned my addr 172.21.15.70:0/432891716 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f4094e38640 1 --2- 172.21.15.70:0/432891716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4090100f70 0x7f4090195c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f4095e3a640 1 --2- 172.21.15.70:0/432891716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4090101930 0x7f409019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f4095639640 1 -- 172.21.15.70:0/432891716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4090100f70 msgr2=0x7f4090195c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f4095639640 1 --2- 172.21.15.70:0/432891716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4090100f70 0x7f4090195c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f4095639640 1 -- 172.21.15.70:0/432891716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4090101930 msgr2=0x7f409019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f4095639640 1 --2- 172.21.15.70:0/432891716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4090101930 0x7f409019cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f4095639640 1 -- 172.21.15.70:0/432891716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409019d370 con 0x7f40900ffe20 2024-04-01T01:33:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f4095e3a640 1 --2- 172.21.15.70:0/432891716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4090101930 0x7f409019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:33:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f4095639640 1 --2- 172.21.15.70:0/432891716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40900ffe20 0x7f4090195750 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f408c006010 tx=0x7f408c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.640+0000 7f40867fc640 1 -- 172.21.15.70:0/432891716 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c043020 con 0x7f40900ffe20 2024-04-01T01:33:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.641+0000 7f409789d640 1 -- 172.21.15.70:0/432891716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f409019d600 con 0x7f40900ffe20 2024-04-01T01:33:40.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.641+0000 7f40867fc640 1 -- 172.21.15.70:0/432891716 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f408c0043d0 con 0x7f40900ffe20 2024-04-01T01:33:40.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.641+0000 7f40867fc640 1 -- 172.21.15.70:0/432891716 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c003b80 con 0x7f40900ffe20 2024-04-01T01:33:40.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.641+0000 7f409789d640 1 -- 172.21.15.70:0/432891716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f409019dba0 con 0x7f40900ffe20 2024-04-01T01:33:40.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.641+0000 7f409789d640 1 -- 172.21.15.70:0/432891716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40901071b0 con 0x7f40900ffe20 2024-04-01T01:33:40.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.642+0000 7f40867fc640 1 -- 172.21.15.70:0/432891716 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f408c046020 con 0x7f40900ffe20 2024-04-01T01:33:40.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.643+0000 7f40867fc640 1 --2- 172.21.15.70:0/432891716 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f407c0a2530 0x7f407c0a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:40.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.643+0000 7f4094e38640 1 --2- 172.21.15.70:0/432891716 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f407c0a2530 0x7f407c0a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:40.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.643+0000 7f40867fc640 1 -- 172.21.15.70:0/432891716 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f408c0f0dd0 con 0x7f40900ffe20 2024-04-01T01:33:40.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.644+0000 7f4094e38640 1 --2- 172.21.15.70:0/432891716 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f407c0a2530 0x7f407c0a49f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4090196c60 tx=0x7f4080011040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:40.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.648+0000 7f40867fc640 1 -- 172.21.15.70:0/432891716 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f408c04d020 con 0x7f40900ffe20 2024-04-01T01:33:40.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.827+0000 7f409789d640 1 -- 172.21.15.70:0/432891716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4090062350 con 0x7f40900ffe20 2024-04-01T01:33:40.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.827+0000 7f40867fc640 1 -- 172.21.15.70:0/432891716 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f408c0bdb70 con 0x7f40900ffe20 2024-04-01T01:33:40.827 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-01T01:33:40.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.829+0000 7f409789d640 1 -- 172.21.15.70:0/432891716 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f407c0a2530 msgr2=0x7f407c0a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.829+0000 7f409789d640 1 --2- 172.21.15.70:0/432891716 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f407c0a2530 0x7f407c0a49f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4090196c60 tx=0x7f4080011040 comp rx=0 tx=0).stop 2024-04-01T01:33:40.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.829+0000 7f409789d640 1 -- 172.21.15.70:0/432891716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40900ffe20 msgr2=0x7f4090195750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.829+0000 7f409789d640 1 --2- 172.21.15.70:0/432891716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40900ffe20 0x7f4090195750 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f408c006010 tx=0x7f408c005e40 comp rx=0 tx=0).stop 2024-04-01T01:33:40.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.829+0000 7f409789d640 1 -- 172.21.15.70:0/432891716 shutdown_connections 2024-04-01T01:33:40.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.829+0000 7f409789d640 1 --2- 172.21.15.70:0/432891716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4090101930 0x7f409019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.829+0000 7f409789d640 1 --2- 172.21.15.70:0/432891716 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f407c0a2530 0x7f407c0a49f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.829+0000 7f409789d640 1 --2- 172.21.15.70:0/432891716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4090100f70 0x7f4090195c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.829+0000 7f409789d640 1 --2- 172.21.15.70:0/432891716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40900ffe20 0x7f4090195750 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.829+0000 7f409789d640 1 -- 172.21.15.70:0/432891716 >> 172.21.15.70:0/432891716 conn(0x7f40900fb390 msgr2=0x7f4090104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:40.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.829+0000 7f409789d640 1 -- 172.21.15.70:0/432891716 shutdown_connections 2024-04-01T01:33:40.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.829+0000 7f409789d640 1 -- 172.21.15.70:0/432891716 wait complete. 2024-04-01T01:33:40.839 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:40 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/4120953937' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:40.839 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:40 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1702105690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:40.840 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:40 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/28757300' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:40.840 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-01T01:33:40.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.946+0000 7fd103855640 1 -- 172.21.15.70:0/187834150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0fc101840 msgr2=0x7fd0fc101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.946+0000 7fd103855640 1 --2- 172.21.15.70:0/187834150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0fc101840 0x7fd0fc101cc0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec009900 tx=0x7fd0ec02bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:40.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.946+0000 7fd103855640 1 -- 172.21.15.70:0/187834150 shutdown_connections 2024-04-01T01:33:40.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.946+0000 7fd103855640 1 --2- 172.21.15.70:0/187834150 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0fc102200 0x7fd0fc108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.946+0000 7fd103855640 1 --2- 172.21.15.70:0/187834150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0fc101840 0x7fd0fc101cc0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.946+0000 7fd103855640 1 --2- 172.21.15.70:0/187834150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0fc100650 0x7fd0fc100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.946+0000 7fd103855640 1 -- 172.21.15.70:0/187834150 >> 172.21.15.70:0/187834150 conn(0x7fd0fc0fbbe0 msgr2=0x7fd0fc0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:40.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.946+0000 7fd103855640 1 -- 172.21.15.70:0/187834150 shutdown_connections 2024-04-01T01:33:40.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.946+0000 7fd103855640 1 -- 172.21.15.70:0/187834150 wait complete. 2024-04-01T01:33:40.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.946+0000 7fd103855640 1 Processor -- start 2024-04-01T01:33:40.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.946+0000 7fd103855640 1 -- start start 2024-04-01T01:33:40.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd103855640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0fc100650 0x7fd0fc1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:40.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd103855640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0fc101840 0x7fd0fc197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:40.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd103855640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0fc102200 0x7fd0fc19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:40.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd103855640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc10d2a0 con 0x7fd0fc102200 2024-04-01T01:33:40.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd103855640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc10d120 con 0x7fd0fc101840 2024-04-01T01:33:40.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd103855640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc10d420 con 0x7fd0fc100650 2024-04-01T01:33:40.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd101df2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0fc102200 0x7fd0fc19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:40.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd101df2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0fc102200 0x7fd0fc19ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55394/0 (socket says 172.21.15.70:55394) 2024-04-01T01:33:40.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd101df2640 1 -- 172.21.15.70:0/2449726824 learned_addr learned my addr 172.21.15.70:0/2449726824 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:40.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd100df0640 1 --2- 172.21.15.70:0/2449726824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0fc101840 0x7fd0fc197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:40.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd1015f1640 1 --2- 172.21.15.70:0/2449726824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0fc100650 0x7fd0fc1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:40.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd101df2640 1 -- 172.21.15.70:0/2449726824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0fc100650 msgr2=0x7fd0fc1979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd101df2640 1 --2- 172.21.15.70:0/2449726824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0fc100650 0x7fd0fc1979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd101df2640 1 -- 172.21.15.70:0/2449726824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0fc101840 msgr2=0x7fd0fc197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:40.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd101df2640 1 --2- 172.21.15.70:0/2449726824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0fc101840 0x7fd0fc197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:40.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd101df2640 1 -- 172.21.15.70:0/2449726824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0fc19f5f0 con 0x7fd0fc102200 2024-04-01T01:33:40.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd101df2640 1 --2- 172.21.15.70:0/2449726824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0fc102200 0x7fd0fc19ef80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd0f800e970 tx=0x7fd0f800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:40.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd0f27fc640 1 -- 172.21.15.70:0/2449726824 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f800ccc0 con 0x7fd0fc102200 2024-04-01T01:33:40.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd103855640 1 -- 172.21.15.70:0/2449726824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0fc19f8e0 con 0x7fd0fc102200 2024-04-01T01:33:40.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd0f27fc640 1 -- 172.21.15.70:0/2449726824 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0f800ce60 con 0x7fd0fc102200 2024-04-01T01:33:40.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd0f27fc640 1 -- 172.21.15.70:0/2449726824 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8010430 con 0x7fd0fc102200 2024-04-01T01:33:40.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.947+0000 7fd103855640 1 -- 172.21.15.70:0/2449726824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0fc19fe00 con 0x7fd0fc102200 2024-04-01T01:33:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.948+0000 7fd103855640 1 -- 172.21.15.70:0/2449726824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0fc107a10 con 0x7fd0fc102200 2024-04-01T01:33:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.949+0000 7fd0f27fc640 1 -- 172.21.15.70:0/2449726824 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7fd0f8004ff0 con 0x7fd0fc102200 2024-04-01T01:33:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.950+0000 7fd0f27fc640 1 --2- 172.21.15.70:0/2449726824 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0e80a2530 0x7fd0e80a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.950+0000 7fd1015f1640 1 --2- 172.21.15.70:0/2449726824 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0e80a2530 0x7fd0e80a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.950+0000 7fd0f27fc640 1 -- 172.21.15.70:0/2449726824 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd0f80cf070 con 0x7fd0fc102200 2024-04-01T01:33:40.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.950+0000 7fd1015f1640 1 --2- 172.21.15.70:0/2449726824 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0e80a2530 0x7fd0e80a49f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc1016a0 tx=0x7fd0f4009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:40.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:40.954+0000 7fd0f27fc640 1 -- 172.21.15.70:0/2449726824 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd0f809bd00 con 0x7fd0fc102200 2024-04-01T01:33:40.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:40 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/4120953937' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:40.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:40 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1702105690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:40.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:40 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/28757300' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:41.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:40 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/4120953937' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:41.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:40 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1702105690' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:41.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:40 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/28757300' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:41.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.129+0000 7fd103855640 1 -- 172.21.15.70:0/2449726824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd0fc062350 con 0x7fd0fc102200 2024-04-01T01:33:41.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.129+0000 7fd0f27fc640 1 -- 172.21.15.70:0/2449726824 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fd0f809b600 con 0x7fd0fc102200 2024-04-01T01:33:41.129 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:33:41.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.131+0000 7fd103855640 1 -- 172.21.15.70:0/2449726824 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0e80a2530 msgr2=0x7fd0e80a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:41.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.131+0000 7fd103855640 1 --2- 172.21.15.70:0/2449726824 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0e80a2530 0x7fd0e80a49f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc1016a0 tx=0x7fd0f4009680 comp rx=0 tx=0).stop 2024-04-01T01:33:41.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.131+0000 7fd103855640 1 -- 172.21.15.70:0/2449726824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0fc102200 msgr2=0x7fd0fc19ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:41.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.131+0000 7fd103855640 1 --2- 172.21.15.70:0/2449726824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0fc102200 0x7fd0fc19ef80 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd0f800e970 tx=0x7fd0f800ed70 comp rx=0 tx=0).stop 2024-04-01T01:33:41.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.131+0000 7fd103855640 1 -- 172.21.15.70:0/2449726824 shutdown_connections 2024-04-01T01:33:41.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.131+0000 7fd103855640 1 --2- 172.21.15.70:0/2449726824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0fc102200 0x7fd0fc19ef80 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.131+0000 7fd103855640 1 --2- 172.21.15.70:0/2449726824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0fc101840 0x7fd0fc197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.131+0000 7fd103855640 1 --2- 172.21.15.70:0/2449726824 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0e80a2530 0x7fd0e80a49f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.131+0000 7fd103855640 1 --2- 172.21.15.70:0/2449726824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0fc100650 0x7fd0fc1979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.131+0000 7fd103855640 1 -- 172.21.15.70:0/2449726824 >> 172.21.15.70:0/2449726824 conn(0x7fd0fc0fbbe0 msgr2=0x7fd0fc104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:41.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.131+0000 7fd103855640 1 -- 172.21.15.70:0/2449726824 shutdown_connections 2024-04-01T01:33:41.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.131+0000 7fd103855640 1 -- 172.21.15.70:0/2449726824 wait complete. 2024-04-01T01:33:41.142 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-01T01:33:41.142 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:33:41.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.246+0000 7f1b3ffc2640 1 -- 172.21.15.70:0/4019041122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b38102220 msgr2=0x7f1b38108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:41.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.246+0000 7f1b3ffc2640 1 --2- 172.21.15.70:0/4019041122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b38102220 0x7f1b38108ab0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1b30009ca0 tx=0x7f1b3002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.246+0000 7f1b3ffc2640 1 -- 172.21.15.70:0/4019041122 shutdown_connections 2024-04-01T01:33:41.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.246+0000 7f1b3ffc2640 1 --2- 172.21.15.70:0/4019041122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b38102220 0x7f1b38108ab0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.246+0000 7f1b3ffc2640 1 --2- 172.21.15.70:0/4019041122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b38101860 0x7f1b38101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.246+0000 7f1b3ffc2640 1 --2- 172.21.15.70:0/4019041122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b38100670 0x7f1b38100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.246+0000 7f1b3ffc2640 1 -- 172.21.15.70:0/4019041122 >> 172.21.15.70:0/4019041122 conn(0x7f1b380fbbe0 msgr2=0x7f1b380fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:41.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.246+0000 7f1b3ffc2640 1 -- 172.21.15.70:0/4019041122 shutdown_connections 2024-04-01T01:33:41.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.246+0000 7f1b3ffc2640 1 -- 172.21.15.70:0/4019041122 wait complete. 2024-04-01T01:33:41.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3ffc2640 1 Processor -- start 2024-04-01T01:33:41.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3ffc2640 1 -- start start 2024-04-01T01:33:41.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3ffc2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b38100670 0x7f1b381046d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:41.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3ffc2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b38101860 0x7f1b38102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:41.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3ffc2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b38102220 0x7f1b38103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:41.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3ffc2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3810c1b0 con 0x7f1b38101860 2024-04-01T01:33:41.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3ffc2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3810c030 con 0x7f1b38100670 2024-04-01T01:33:41.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3ffc2640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3810c330 con 0x7f1b38102220 2024-04-01T01:33:41.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3d55d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b38101860 0x7f1b38102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:41.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3dd5e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b38100670 0x7f1b381046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:41.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3d55d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b38101860 0x7f1b38102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55408/0 (socket says 172.21.15.70:55408) 2024-04-01T01:33:41.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3d55d640 1 -- 172.21.15.70:0/3992290788 learned_addr learned my addr 172.21.15.70:0/3992290788 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:41.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3e55f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b38102220 0x7f1b38103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:41.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3d55d640 1 -- 172.21.15.70:0/3992290788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b38102220 msgr2=0x7f1b38103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:41.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3d55d640 1 --2- 172.21.15.70:0/3992290788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b38102220 0x7f1b38103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3d55d640 1 -- 172.21.15.70:0/3992290788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b38100670 msgr2=0x7f1b381046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:41.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3d55d640 1 --2- 172.21.15.70:0/3992290788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b38100670 0x7f1b381046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3d55d640 1 -- 172.21.15.70:0/3992290788 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b38103ae0 con 0x7f1b38101860 2024-04-01T01:33:41.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3e55f640 1 --2- 172.21.15.70:0/3992290788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b38102220 0x7f1b38103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:41.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3dd5e640 1 --2- 172.21.15.70:0/3992290788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b38100670 0x7f1b381046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:41.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.247+0000 7f1b3d55d640 1 --2- 172.21.15.70:0/3992290788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b38101860 0x7f1b38102d10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800e9f0 tx=0x7f1b2800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:41.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.248+0000 7f1b2effd640 1 -- 172.21.15.70:0/3992290788 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2800cd80 con 0x7f1b38101860 2024-04-01T01:33:41.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.248+0000 7f1b2effd640 1 -- 172.21.15.70:0/3992290788 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b280054d0 con 0x7f1b38101860 2024-04-01T01:33:41.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.248+0000 7f1b3ffc2640 1 -- 172.21.15.70:0/3992290788 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b381a1a80 con 0x7f1b38101860 2024-04-01T01:33:41.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.248+0000 7f1b2effd640 1 -- 172.21.15.70:0/3992290788 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28010490 con 0x7f1b38101860 2024-04-01T01:33:41.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.248+0000 7f1b3ffc2640 1 -- 172.21.15.70:0/3992290788 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b381a1f80 con 0x7f1b38101860 2024-04-01T01:33:41.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.249+0000 7f1b3ffc2640 1 -- 172.21.15.70:0/3992290788 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b381a2210 con 0x7f1b38101860 2024-04-01T01:33:41.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.251+0000 7f1b2effd640 1 -- 172.21.15.70:0/3992290788 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f1b28005000 con 0x7f1b38101860 2024-04-01T01:33:41.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.251+0000 7f1b2effd640 1 --2- 172.21.15.70:0/3992290788 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1b240a2530 0x7f1b240a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:41.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.251+0000 7f1b2effd640 1 -- 172.21.15.70:0/3992290788 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1b280cf260 con 0x7f1b38101860 2024-04-01T01:33:41.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.251+0000 7f1b3dd5e640 1 --2- 172.21.15.70:0/3992290788 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1b240a2530 0x7f1b240a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:41.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.252+0000 7f1b3dd5e640 1 --2- 172.21.15.70:0/3992290788 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1b240a2530 0x7f1b240a49f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1b381016c0 tx=0x7f1b34005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:41.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.254+0000 7f1b2effd640 1 -- 172.21.15.70:0/3992290788 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1b28053280 con 0x7f1b38101860 2024-04-01T01:33:41.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.473+0000 7f1b3ffc2640 1 -- 172.21.15.70:0/3992290788 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b381a23e0 con 0x7f1b38101860 2024-04-01T01:33:41.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.473+0000 7f1b2effd640 1 -- 172.21.15.70:0/3992290788 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f1b280052f0 con 0x7f1b38101860 2024-04-01T01:33:41.474 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:41.474 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:33:36.894838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24418,24404,24379,24451,24464]},"id":1}]} 2024-04-01T01:33:41.474 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-04-01T01:33:41.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.475+0000 7f1b3ffc2640 1 -- 172.21.15.70:0/3992290788 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1b240a2530 msgr2=0x7f1b240a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:41.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.475+0000 7f1b3ffc2640 1 --2- 172.21.15.70:0/3992290788 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1b240a2530 0x7f1b240a49f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1b381016c0 tx=0x7f1b34005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:41.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.475+0000 7f1b3ffc2640 1 -- 172.21.15.70:0/3992290788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b38101860 msgr2=0x7f1b38102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:41.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.475+0000 7f1b3ffc2640 1 --2- 172.21.15.70:0/3992290788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b38101860 0x7f1b38102d10 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800e9f0 tx=0x7f1b2800edf0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.475+0000 7f1b3ffc2640 1 -- 172.21.15.70:0/3992290788 shutdown_connections 2024-04-01T01:33:41.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.475+0000 7f1b3ffc2640 1 --2- 172.21.15.70:0/3992290788 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1b240a2530 0x7f1b240a49f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.475+0000 7f1b3ffc2640 1 --2- 172.21.15.70:0/3992290788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b38102220 0x7f1b38103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.475+0000 7f1b3ffc2640 1 --2- 172.21.15.70:0/3992290788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b38101860 0x7f1b38102d10 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.475+0000 7f1b3ffc2640 1 --2- 172.21.15.70:0/3992290788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b38100670 0x7f1b381046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.475+0000 7f1b3ffc2640 1 -- 172.21.15.70:0/3992290788 >> 172.21.15.70:0/3992290788 conn(0x7f1b380fbbe0 msgr2=0x7f1b380688b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:41.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.476+0000 7f1b3ffc2640 1 -- 172.21.15.70:0/3992290788 shutdown_connections 2024-04-01T01:33:41.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.476+0000 7f1b3ffc2640 1 -- 172.21.15.70:0/3992290788 wait complete. 2024-04-01T01:33:41.487 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-01T01:33:41.487 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-01T01:33:41.487 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T01:33:41.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.593+0000 7fe5750ae640 1 -- 172.21.15.70:0/2612429458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe570102220 msgr2=0x7fe570108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:41.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.593+0000 7fe5750ae640 1 --2- 172.21.15.70:0/2612429458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe570102220 0x7fe570108ab0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe560009ca0 tx=0x7fe56002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.593+0000 7fe5750ae640 1 -- 172.21.15.70:0/2612429458 shutdown_connections 2024-04-01T01:33:41.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.593+0000 7fe5750ae640 1 --2- 172.21.15.70:0/2612429458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe570102220 0x7fe570108ab0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.593+0000 7fe5750ae640 1 --2- 172.21.15.70:0/2612429458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe570101860 0x7fe570101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.593+0000 7fe5750ae640 1 --2- 172.21.15.70:0/2612429458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe570100670 0x7fe570100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.593+0000 7fe5750ae640 1 -- 172.21.15.70:0/2612429458 >> 172.21.15.70:0/2612429458 conn(0x7fe5700fbbe0 msgr2=0x7fe5700fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:41.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.593+0000 7fe5750ae640 1 -- 172.21.15.70:0/2612429458 shutdown_connections 2024-04-01T01:33:41.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.593+0000 7fe5750ae640 1 -- 172.21.15.70:0/2612429458 wait complete. 2024-04-01T01:33:41.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.594+0000 7fe5750ae640 1 Processor -- start 2024-04-01T01:33:41.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.594+0000 7fe5750ae640 1 -- start start 2024-04-01T01:33:41.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.594+0000 7fe5750ae640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe570100670 0x7fe570196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:41.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.594+0000 7fe5750ae640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe570101860 0x7fe570196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:41.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.594+0000 7fe5750ae640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe570102220 0x7fe57019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:41.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.594+0000 7fe5750ae640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe57010c1a0 con 0x7fe570100670 2024-04-01T01:33:41.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.594+0000 7fe5750ae640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe57010c020 con 0x7fe570102220 2024-04-01T01:33:41.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.594+0000 7fe5750ae640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe57010c320 con 0x7fe570101860 2024-04-01T01:33:41.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.595+0000 7fe56f59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe570102220 0x7fe57019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:41.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.595+0000 7fe56ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe570100670 0x7fe570196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:41.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.595+0000 7fe56f59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe570102220 0x7fe57019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34598/0 (socket says 172.21.15.70:34598) 2024-04-01T01:33:41.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.595+0000 7fe56e59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe570101860 0x7fe570196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:41.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.595+0000 7fe56f59e640 1 -- 172.21.15.70:0/3892690743 learned_addr learned my addr 172.21.15.70:0/3892690743 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:41.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.595+0000 7fe56f59e640 1 -- 172.21.15.70:0/3892690743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe570101860 msgr2=0x7fe570196db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:41.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.595+0000 7fe56f59e640 1 --2- 172.21.15.70:0/3892690743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe570101860 0x7fe570196db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.595+0000 7fe56f59e640 1 -- 172.21.15.70:0/3892690743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe570100670 msgr2=0x7fe570196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:41.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.595+0000 7fe56f59e640 1 --2- 172.21.15.70:0/3892690743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe570100670 0x7fe570196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.595+0000 7fe56f59e640 1 -- 172.21.15.70:0/3892690743 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe57019e490 con 0x7fe570102220 2024-04-01T01:33:41.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.595+0000 7fe56f59e640 1 --2- 172.21.15.70:0/3892690743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe570102220 0x7fe57019de20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe56002c700 tx=0x7fe5600328a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:41.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.595+0000 7fe557fff640 1 -- 172.21.15.70:0/3892690743 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe560041020 con 0x7fe570102220 2024-04-01T01:33:41.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.595+0000 7fe557fff640 1 -- 172.21.15.70:0/3892690743 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe560003e20 con 0x7fe570102220 2024-04-01T01:33:41.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.595+0000 7fe557fff640 1 -- 172.21.15.70:0/3892690743 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe560039040 con 0x7fe570102220 2024-04-01T01:33:41.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.595+0000 7fe5750ae640 1 -- 172.21.15.70:0/3892690743 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe57019e720 con 0x7fe570102220 2024-04-01T01:33:41.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.595+0000 7fe5750ae640 1 -- 172.21.15.70:0/3892690743 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe57019ebc0 con 0x7fe570102220 2024-04-01T01:33:41.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.596+0000 7fe5750ae640 1 -- 172.21.15.70:0/3892690743 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe570107990 con 0x7fe570102220 2024-04-01T01:33:41.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.597+0000 7fe557fff640 1 -- 172.21.15.70:0/3892690743 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7fe560044020 con 0x7fe570102220 2024-04-01T01:33:41.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.598+0000 7fe557fff640 1 --2- 172.21.15.70:0/3892690743 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe5580a2530 0x7fe5580a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:41.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.598+0000 7fe56ed9d640 1 --2- 172.21.15.70:0/3892690743 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe5580a2530 0x7fe5580a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:41.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.598+0000 7fe557fff640 1 -- 172.21.15.70:0/3892690743 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe5600f10e0 con 0x7fe570102220 2024-04-01T01:33:41.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.598+0000 7fe56ed9d640 1 --2- 172.21.15.70:0/3892690743 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe5580a2530 0x7fe5580a49f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe5701016c0 tx=0x7fe568005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:41.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.605+0000 7fe557fff640 1 -- 172.21.15.70:0/3892690743 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe56004b020 con 0x7fe570102220 2024-04-01T01:33:41.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.784+0000 7fe5750ae640 1 -- 172.21.15.70:0/3892690743 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe570061b90 con 0x7fe5580a2530 2024-04-01T01:33:41.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.793+0000 7fe557fff640 1 -- 172.21.15.70:0/3892690743 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340534 (secure 0 0 0) 0x7fe570061b90 con 0x7fe5580a2530 2024-04-01T01:33:41.799 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:41.801 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-01T01:33:41.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.797+0000 7fe5750ae640 1 -- 172.21.15.70:0/3892690743 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe5580a2530 msgr2=0x7fe5580a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:41.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.797+0000 7fe5750ae640 1 --2- 172.21.15.70:0/3892690743 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe5580a2530 0x7fe5580a49f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe5701016c0 tx=0x7fe568005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:41.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.797+0000 7fe5750ae640 1 -- 172.21.15.70:0/3892690743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe570102220 msgr2=0x7fe57019de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:41.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.797+0000 7fe5750ae640 1 --2- 172.21.15.70:0/3892690743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe570102220 0x7fe57019de20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe56002c700 tx=0x7fe5600328a0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.797+0000 7fe5750ae640 1 -- 172.21.15.70:0/3892690743 shutdown_connections 2024-04-01T01:33:41.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.797+0000 7fe5750ae640 1 --2- 172.21.15.70:0/3892690743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe570102220 0x7fe57019de20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.797+0000 7fe5750ae640 1 --2- 172.21.15.70:0/3892690743 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe5580a2530 0x7fe5580a49f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.797+0000 7fe5750ae640 1 --2- 172.21.15.70:0/3892690743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe570101860 0x7fe570196db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.797+0000 7fe5750ae640 1 --2- 172.21.15.70:0/3892690743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe570100670 0x7fe570196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.797+0000 7fe5750ae640 1 -- 172.21.15.70:0/3892690743 >> 172.21.15.70:0/3892690743 conn(0x7fe5700fbbe0 msgr2=0x7fe570104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:41.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.797+0000 7fe5750ae640 1 -- 172.21.15.70:0/3892690743 shutdown_connections 2024-04-01T01:33:41.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.797+0000 7fe5750ae640 1 -- 172.21.15.70:0/3892690743 wait complete. 2024-04-01T01:33:41.809 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-04-01T01:33:40.604455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664860,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1373,"num_read_kb":819,"num_write":457,"num_write_kb":3105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181978,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":875,"ondisk_log_size":875,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":359628,"kb_used_data":14628,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124582196,"statfs":{"total":1151940427776,"available":1151572168704,"internally_reserved":0,"allocated":14979072,"data_stored":8211390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3167,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":121,"num_read_kb":66,"num_write":24,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003260"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725372+0000","last_change":"2024-04-01T01:31:16.385517+0000","last_active":"2024-04-01T01:33:19.725372+0000","last_peered":"2024-04-01T01:33:19.725372+0000","last_clean":"2024-04-01T01:33:19.725372+0000","last_became_active":"2024-04-01T01:31:16.384682+0000","last_became_peered":"2024-04-01T01:31:16.384682+0000","last_unstale":"2024-04-01T01:33:19.725372+0000","last_undegraded":"2024-04-01T01:33:19.725372+0000","last_fullsized":"2024-04-01T01:33:19.725372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:35:47.614041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838551+0000","last_change":"2024-04-01T01:31:16.376119+0000","last_active":"2024-04-01T01:33:20.838551+0000","last_peered":"2024-04-01T01:33:20.838551+0000","last_clean":"2024-04-01T01:33:20.838551+0000","last_became_active":"2024-04-01T01:31:16.375801+0000","last_became_peered":"2024-04-01T01:31:16.375801+0000","last_unstale":"2024-04-01T01:33:20.838551+0000","last_undegraded":"2024-04-01T01:33:20.838551+0000","last_fullsized":"2024-04-01T01:33:20.838551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:34.120487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838381+0000","last_change":"2024-04-01T01:31:16.378158+0000","last_active":"2024-04-01T01:33:20.838381+0000","last_peered":"2024-04-01T01:33:20.838381+0000","last_clean":"2024-04-01T01:33:20.838381+0000","last_became_active":"2024-04-01T01:31:16.377831+0000","last_became_peered":"2024-04-01T01:31:16.377831+0000","last_unstale":"2024-04-01T01:33:20.838381+0000","last_undegraded":"2024-04-01T01:33:20.838381+0000","last_fullsized":"2024-04-01T01:33:20.838381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:30:06.487545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721996+0000","last_change":"2024-04-01T01:31:16.389708+0000","last_active":"2024-04-01T01:33:19.721996+0000","last_peered":"2024-04-01T01:33:19.721996+0000","last_clean":"2024-04-01T01:33:19.721996+0000","last_became_active":"2024-04-01T01:31:16.389409+0000","last_became_peered":"2024-04-01T01:31:16.389409+0000","last_unstale":"2024-04-01T01:33:19.721996+0000","last_undegraded":"2024-04-01T01:33:19.721996+0000","last_fullsized":"2024-04-01T01:33:19.721996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:57:49.215988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725238+0000","last_change":"2024-04-01T01:31:16.385996+0000","last_active":"2024-04-01T01:33:19.725238+0000","last_peered":"2024-04-01T01:33:19.725238+0000","last_clean":"2024-04-01T01:33:19.725238+0000","last_became_active":"2024-04-01T01:31:16.385648+0000","last_became_peered":"2024-04-01T01:31:16.385648+0000","last_unstale":"2024-04-01T01:33:19.725238+0000","last_undegraded":"2024-04-01T01:33:19.725238+0000","last_fullsized":"2024-04-01T01:33:19.725238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:14:56.595595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.723900+0000","last_change":"2024-04-01T01:32:19.693185+0000","last_active":"2024-04-01T01:33:19.723900+0000","last_peered":"2024-04-01T01:33:19.723900+0000","last_clean":"2024-04-01T01:33:19.723900+0000","last_became_active":"2024-04-01T01:32:19.692763+0000","last_became_peered":"2024-04-01T01:32:19.692763+0000","last_unstale":"2024-04-01T01:33:19.723900+0000","last_undegraded":"2024-04-01T01:33:19.723900+0000","last_fullsized":"2024-04-01T01:33:19.723900+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:37:54.712852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724816+0000","last_change":"2024-04-01T01:31:16.383939+0000","last_active":"2024-04-01T01:33:19.724816+0000","last_peered":"2024-04-01T01:33:19.724816+0000","last_clean":"2024-04-01T01:33:19.724816+0000","last_became_active":"2024-04-01T01:31:16.383626+0000","last_became_peered":"2024-04-01T01:31:16.383626+0000","last_unstale":"2024-04-01T01:33:19.724816+0000","last_undegraded":"2024-04-01T01:33:19.724816+0000","last_fullsized":"2024-04-01T01:33:19.724816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:36:05.764031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.628778+0000","last_change":"2024-04-01T01:31:16.384007+0000","last_active":"2024-04-01T01:33:20.628778+0000","last_peered":"2024-04-01T01:33:20.628778+0000","last_clean":"2024-04-01T01:33:20.628778+0000","last_became_active":"2024-04-01T01:31:16.383687+0000","last_became_peered":"2024-04-01T01:31:16.383687+0000","last_unstale":"2024-04-01T01:33:20.628778+0000","last_undegraded":"2024-04-01T01:33:20.628778+0000","last_fullsized":"2024-04-01T01:33:20.628778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:14:17.593402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.609104+0000","last_change":"2024-04-01T01:31:16.380357+0000","last_active":"2024-04-01T01:33:20.609104+0000","last_peered":"2024-04-01T01:33:20.609104+0000","last_clean":"2024-04-01T01:33:20.609104+0000","last_became_active":"2024-04-01T01:31:16.380046+0000","last_became_peered":"2024-04-01T01:31:16.380046+0000","last_unstale":"2024-04-01T01:33:20.609104+0000","last_undegraded":"2024-04-01T01:33:20.609104+0000","last_fullsized":"2024-04-01T01:33:20.609104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:51:53.887768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724418+0000","last_change":"2024-04-01T01:31:16.378551+0000","last_active":"2024-04-01T01:33:19.724418+0000","last_peered":"2024-04-01T01:33:19.724418+0000","last_clean":"2024-04-01T01:33:19.724418+0000","last_became_active":"2024-04-01T01:31:16.378234+0000","last_became_peered":"2024-04-01T01:31:16.378234+0000","last_unstale":"2024-04-01T01:33:19.724418+0000","last_undegraded":"2024-04-01T01:33:19.724418+0000","last_fullsized":"2024-04-01T01:33:19.724418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:37:45.998115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725392+0000","last_change":"2024-04-01T01:31:16.388755+0000","last_active":"2024-04-01T01:33:19.725392+0000","last_peered":"2024-04-01T01:33:19.725392+0000","last_clean":"2024-04-01T01:33:19.725392+0000","last_became_active":"2024-04-01T01:31:16.388434+0000","last_became_peered":"2024-04-01T01:31:16.388434+0000","last_unstale":"2024-04-01T01:33:19.725392+0000","last_undegraded":"2024-04-01T01:33:19.725392+0000","last_fullsized":"2024-04-01T01:33:19.725392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:03:07.168110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727361+0000","last_change":"2024-04-01T01:33:19.727029+0000","last_active":"2024-04-01T01:33:19.727361+0000","last_peered":"2024-04-01T01:33:19.727361+0000","last_clean":"2024-04-01T01:33:19.727361+0000","last_became_active":"2024-04-01T01:33:19.726714+0000","last_became_peered":"2024-04-01T01:33:19.726714+0000","last_unstale":"2024-04-01T01:33:19.727361+0000","last_undegraded":"2024-04-01T01:33:19.727361+0000","last_fullsized":"2024-04-01T01:33:19.727361+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:35:03.224593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719204+0000","last_change":"2024-04-01T01:31:16.366921+0000","last_active":"2024-04-01T01:33:19.719204+0000","last_peered":"2024-04-01T01:33:19.719204+0000","last_clean":"2024-04-01T01:33:19.719204+0000","last_became_active":"2024-04-01T01:31:16.366098+0000","last_became_peered":"2024-04-01T01:31:16.366098+0000","last_unstale":"2024-04-01T01:33:19.719204+0000","last_undegraded":"2024-04-01T01:33:19.719204+0000","last_fullsized":"2024-04-01T01:33:19.719204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:29.456263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838282+0000","last_change":"2024-04-01T01:31:16.379178+0000","last_active":"2024-04-01T01:33:20.838282+0000","last_peered":"2024-04-01T01:33:20.838282+0000","last_clean":"2024-04-01T01:33:20.838282+0000","last_became_active":"2024-04-01T01:31:16.378877+0000","last_became_peered":"2024-04-01T01:31:16.378877+0000","last_unstale":"2024-04-01T01:33:20.838282+0000","last_undegraded":"2024-04-01T01:33:20.838282+0000","last_fullsized":"2024-04-01T01:33:20.838282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:19:16.941504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838274+0000","last_change":"2024-04-01T01:31:16.385373+0000","last_active":"2024-04-01T01:33:20.838274+0000","last_peered":"2024-04-01T01:33:20.838274+0000","last_clean":"2024-04-01T01:33:20.838274+0000","last_became_active":"2024-04-01T01:31:16.385046+0000","last_became_peered":"2024-04-01T01:31:16.385046+0000","last_unstale":"2024-04-01T01:33:20.838274+0000","last_undegraded":"2024-04-01T01:33:20.838274+0000","last_fullsized":"2024-04-01T01:33:20.838274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:40:32.521220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608501+0000","last_change":"2024-04-01T01:31:16.386267+0000","last_active":"2024-04-01T01:33:20.608501+0000","last_peered":"2024-04-01T01:33:20.608501+0000","last_clean":"2024-04-01T01:33:20.608501+0000","last_became_active":"2024-04-01T01:31:16.385917+0000","last_became_peered":"2024-04-01T01:31:16.385917+0000","last_unstale":"2024-04-01T01:33:20.608501+0000","last_undegraded":"2024-04-01T01:33:20.608501+0000","last_fullsized":"2024-04-01T01:33:20.608501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:58.249253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724843+0000","last_change":"2024-04-01T01:31:16.385609+0000","last_active":"2024-04-01T01:33:19.724843+0000","last_peered":"2024-04-01T01:33:19.724843+0000","last_clean":"2024-04-01T01:33:19.724843+0000","last_became_active":"2024-04-01T01:31:16.385128+0000","last_became_peered":"2024-04-01T01:31:16.385128+0000","last_unstale":"2024-04-01T01:33:19.724843+0000","last_undegraded":"2024-04-01T01:33:19.724843+0000","last_fullsized":"2024-04-01T01:33:19.724843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:30:45.915587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722053+0000","last_change":"2024-04-01T01:31:16.381209+0000","last_active":"2024-04-01T01:33:19.722053+0000","last_peered":"2024-04-01T01:33:19.722053+0000","last_clean":"2024-04-01T01:33:19.722053+0000","last_became_active":"2024-04-01T01:31:16.380721+0000","last_became_peered":"2024-04-01T01:31:16.380721+0000","last_unstale":"2024-04-01T01:33:19.722053+0000","last_undegraded":"2024-04-01T01:33:19.722053+0000","last_fullsized":"2024-04-01T01:33:19.722053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:22.438224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727542+0000","last_change":"2024-04-01T01:31:16.385235+0000","last_active":"2024-04-01T01:33:19.727542+0000","last_peered":"2024-04-01T01:33:19.727542+0000","last_clean":"2024-04-01T01:33:19.727542+0000","last_became_active":"2024-04-01T01:31:16.384856+0000","last_became_peered":"2024-04-01T01:31:16.384856+0000","last_unstale":"2024-04-01T01:33:19.727542+0000","last_undegraded":"2024-04-01T01:33:19.727542+0000","last_fullsized":"2024-04-01T01:33:19.727542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:35:30.214977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727267+0000","last_change":"2024-04-01T01:31:20.573626+0000","last_active":"2024-04-01T01:33:19.727267+0000","last_peered":"2024-04-01T01:33:19.727267+0000","last_clean":"2024-04-01T01:33:19.727267+0000","last_became_active":"2024-04-01T01:31:20.573182+0000","last_became_peered":"2024-04-01T01:31:20.573182+0000","last_unstale":"2024-04-01T01:33:19.727267+0000","last_undegraded":"2024-04-01T01:33:19.727267+0000","last_fullsized":"2024-04-01T01:33:19.727267+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:33:19.162613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'14","reported_seq":169,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:38.711759+0000","last_change":"2024-04-01T01:31:15.373100+0000","last_active":"2024-04-01T01:33:38.711759+0000","last_peered":"2024-04-01T01:33:38.711759+0000","last_clean":"2024-04-01T01:33:38.711759+0000","last_became_active":"2024-04-01T01:31:15.372710+0000","last_became_peered":"2024-04-01T01:31:15.372710+0000","last_unstale":"2024-04-01T01:33:38.711759+0000","last_undegraded":"2024-04-01T01:33:38.711759+0000","last_fullsized":"2024-04-01T01:33:38.711759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:29:42.303549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629686+0000","last_change":"2024-04-01T01:31:16.383690+0000","last_active":"2024-04-01T01:33:20.629686+0000","last_peered":"2024-04-01T01:33:20.629686+0000","last_clean":"2024-04-01T01:33:20.629686+0000","last_became_active":"2024-04-01T01:31:16.383248+0000","last_became_peered":"2024-04-01T01:31:16.383248+0000","last_unstale":"2024-04-01T01:33:20.629686+0000","last_undegraded":"2024-04-01T01:33:20.629686+0000","last_fullsized":"2024-04-01T01:33:20.629686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:34:49.112880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721649+0000","last_change":"2024-04-01T01:32:19.691398+0000","last_active":"2024-04-01T01:33:19.721649+0000","last_peered":"2024-04-01T01:33:19.721649+0000","last_clean":"2024-04-01T01:33:19.721649+0000","last_became_active":"2024-04-01T01:32:19.690908+0000","last_became_peered":"2024-04-01T01:32:19.690908+0000","last_unstale":"2024-04-01T01:33:19.721649+0000","last_undegraded":"2024-04-01T01:33:19.721649+0000","last_fullsized":"2024-04-01T01:33:19.721649+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:58:54.712363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629846+0000","last_change":"2024-04-01T01:31:15.381742+0000","last_active":"2024-04-01T01:33:20.629846+0000","last_peered":"2024-04-01T01:33:20.629846+0000","last_clean":"2024-04-01T01:33:20.629846+0000","last_became_active":"2024-04-01T01:31:15.377543+0000","last_became_peered":"2024-04-01T01:31:15.377543+0000","last_unstale":"2024-04-01T01:33:20.629846+0000","last_undegraded":"2024-04-01T01:33:20.629846+0000","last_fullsized":"2024-04-01T01:33:20.629846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:51:42.911114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838510+0000","last_change":"2024-04-01T01:31:16.382001+0000","last_active":"2024-04-01T01:33:20.838510+0000","last_peered":"2024-04-01T01:33:20.838510+0000","last_clean":"2024-04-01T01:33:20.838510+0000","last_became_active":"2024-04-01T01:31:16.381694+0000","last_became_peered":"2024-04-01T01:31:16.381694+0000","last_unstale":"2024-04-01T01:33:20.838510+0000","last_undegraded":"2024-04-01T01:33:20.838510+0000","last_fullsized":"2024-04-01T01:33:20.838510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:55:07.366755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721790+0000","last_change":"2024-04-01T01:31:20.570080+0000","last_active":"2024-04-01T01:33:19.721790+0000","last_peered":"2024-04-01T01:33:19.721790+0000","last_clean":"2024-04-01T01:33:19.721790+0000","last_became_active":"2024-04-01T01:31:20.569712+0000","last_became_peered":"2024-04-01T01:31:20.569712+0000","last_unstale":"2024-04-01T01:33:19.721790+0000","last_undegraded":"2024-04-01T01:33:19.721790+0000","last_fullsized":"2024-04-01T01:33:19.721790+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:48:27.732831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725670+0000","last_change":"2024-04-01T01:32:19.703903+0000","last_active":"2024-04-01T01:33:19.725670+0000","last_peered":"2024-04-01T01:33:19.725670+0000","last_clean":"2024-04-01T01:33:19.725670+0000","last_became_active":"2024-04-01T01:32:19.691856+0000","last_became_peered":"2024-04-01T01:32:19.691856+0000","last_unstale":"2024-04-01T01:33:19.725670+0000","last_undegraded":"2024-04-01T01:33:19.725670+0000","last_fullsized":"2024-04-01T01:33:19.725670+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:38:41.284469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608732+0000","last_change":"2024-04-01T01:31:16.383020+0000","last_active":"2024-04-01T01:33:20.608732+0000","last_peered":"2024-04-01T01:33:20.608732+0000","last_clean":"2024-04-01T01:33:20.608732+0000","last_became_active":"2024-04-01T01:31:16.382640+0000","last_became_peered":"2024-04-01T01:31:16.382640+0000","last_unstale":"2024-04-01T01:33:20.608732+0000","last_undegraded":"2024-04-01T01:33:20.608732+0000","last_fullsized":"2024-04-01T01:33:20.608732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:11:51.211929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721753+0000","last_change":"2024-04-01T01:31:20.573030+0000","last_active":"2024-04-01T01:33:19.721753+0000","last_peered":"2024-04-01T01:33:19.721753+0000","last_clean":"2024-04-01T01:33:19.721753+0000","last_became_active":"2024-04-01T01:31:20.572613+0000","last_became_peered":"2024-04-01T01:31:20.572613+0000","last_unstale":"2024-04-01T01:33:19.721753+0000","last_undegraded":"2024-04-01T01:33:19.721753+0000","last_fullsized":"2024-04-01T01:33:19.721753+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:42:17.086942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725015+0000","last_change":"2024-04-01T01:31:15.373054+0000","last_active":"2024-04-01T01:33:19.725015+0000","last_peered":"2024-04-01T01:33:19.725015+0000","last_clean":"2024-04-01T01:33:19.725015+0000","last_became_active":"2024-04-01T01:31:15.372575+0000","last_became_peered":"2024-04-01T01:31:15.372575+0000","last_unstale":"2024-04-01T01:33:19.725015+0000","last_undegraded":"2024-04-01T01:33:19.725015+0000","last_fullsized":"2024-04-01T01:33:19.725015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:56:41.860423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722460+0000","last_change":"2024-04-01T01:31:16.376037+0000","last_active":"2024-04-01T01:33:19.722460+0000","last_peered":"2024-04-01T01:33:19.722460+0000","last_clean":"2024-04-01T01:33:19.722460+0000","last_became_active":"2024-04-01T01:31:16.375463+0000","last_became_peered":"2024-04-01T01:31:16.375463+0000","last_unstale":"2024-04-01T01:33:19.722460+0000","last_undegraded":"2024-04-01T01:33:19.722460+0000","last_fullsized":"2024-04-01T01:33:19.722460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:30:40.098033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727397+0000","last_change":"2024-04-01T01:31:20.574288+0000","last_active":"2024-04-01T01:33:19.727397+0000","last_peered":"2024-04-01T01:33:19.727397+0000","last_clean":"2024-04-01T01:33:19.727397+0000","last_became_active":"2024-04-01T01:31:20.573812+0000","last_became_peered":"2024-04-01T01:31:20.573812+0000","last_unstale":"2024-04-01T01:33:19.727397+0000","last_undegraded":"2024-04-01T01:33:19.727397+0000","last_fullsized":"2024-04-01T01:33:19.727397+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:43:46.126102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":177,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:35.766196+0000","last_change":"2024-04-01T01:32:19.706645+0000","last_active":"2024-04-01T01:33:35.766196+0000","last_peered":"2024-04-01T01:33:35.766196+0000","last_clean":"2024-04-01T01:33:35.766196+0000","last_became_active":"2024-04-01T01:32:19.690510+0000","last_became_peered":"2024-04-01T01:32:19.690510+0000","last_unstale":"2024-04-01T01:33:35.766196+0000","last_undegraded":"2024-04-01T01:33:35.766196+0000","last_fullsized":"2024-04-01T01:33:35.766196+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:20:23.103229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1631,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724703+0000","last_change":"2024-04-01T01:31:16.377735+0000","last_active":"2024-04-01T01:33:19.724703+0000","last_peered":"2024-04-01T01:33:19.724703+0000","last_clean":"2024-04-01T01:33:19.724703+0000","last_became_active":"2024-04-01T01:31:16.377342+0000","last_became_peered":"2024-04-01T01:31:16.377342+0000","last_unstale":"2024-04-01T01:33:19.724703+0000","last_undegraded":"2024-04-01T01:33:19.724703+0000","last_fullsized":"2024-04-01T01:33:19.724703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:47.539223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727261+0000","last_change":"2024-04-01T01:31:20.573370+0000","last_active":"2024-04-01T01:33:19.727261+0000","last_peered":"2024-04-01T01:33:19.727261+0000","last_clean":"2024-04-01T01:33:19.727261+0000","last_became_active":"2024-04-01T01:31:20.572919+0000","last_became_peered":"2024-04-01T01:31:20.572919+0000","last_unstale":"2024-04-01T01:33:19.727261+0000","last_undegraded":"2024-04-01T01:33:19.727261+0000","last_fullsized":"2024-04-01T01:33:19.727261+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:42.190918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.723867+0000","last_change":"2024-04-01T01:32:19.810468+0000","last_active":"2024-04-01T01:33:19.723867+0000","last_peered":"2024-04-01T01:33:19.723867+0000","last_clean":"2024-04-01T01:33:19.723867+0000","last_became_active":"2024-04-01T01:32:19.805517+0000","last_became_peered":"2024-04-01T01:32:19.805517+0000","last_unstale":"2024-04-01T01:33:19.723867+0000","last_undegraded":"2024-04-01T01:33:19.723867+0000","last_fullsized":"2024-04-01T01:33:19.723867+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:54:35.355761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722977+0000","last_change":"2024-04-01T01:31:16.384130+0000","last_active":"2024-04-01T01:33:19.722977+0000","last_peered":"2024-04-01T01:33:19.722977+0000","last_clean":"2024-04-01T01:33:19.722977+0000","last_became_active":"2024-04-01T01:31:16.383400+0000","last_became_peered":"2024-04-01T01:31:16.383400+0000","last_unstale":"2024-04-01T01:33:19.722977+0000","last_undegraded":"2024-04-01T01:33:19.722977+0000","last_fullsized":"2024-04-01T01:33:19.722977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:22:10.860325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724729+0000","last_change":"2024-04-01T01:31:20.571052+0000","last_active":"2024-04-01T01:33:19.724729+0000","last_peered":"2024-04-01T01:33:19.724729+0000","last_clean":"2024-04-01T01:33:19.724729+0000","last_became_active":"2024-04-01T01:31:20.570617+0000","last_became_peered":"2024-04-01T01:31:20.570617+0000","last_unstale":"2024-04-01T01:33:19.724729+0000","last_undegraded":"2024-04-01T01:33:19.724729+0000","last_fullsized":"2024-04-01T01:33:19.724729+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:53:51.195056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719095+0000","last_change":"2024-04-01T01:31:15.369820+0000","last_active":"2024-04-01T01:33:19.719095+0000","last_peered":"2024-04-01T01:33:19.719095+0000","last_clean":"2024-04-01T01:33:19.719095+0000","last_became_active":"2024-04-01T01:31:15.369378+0000","last_became_peered":"2024-04-01T01:31:15.369378+0000","last_unstale":"2024-04-01T01:33:19.719095+0000","last_undegraded":"2024-04-01T01:33:19.719095+0000","last_fullsized":"2024-04-01T01:33:19.719095+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:37:34.351220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.628952+0000","last_change":"2024-04-01T01:31:16.383352+0000","last_active":"2024-04-01T01:33:20.628952+0000","last_peered":"2024-04-01T01:33:20.628952+0000","last_clean":"2024-04-01T01:33:20.628952+0000","last_became_active":"2024-04-01T01:31:16.382898+0000","last_became_peered":"2024-04-01T01:31:16.382898+0000","last_unstale":"2024-04-01T01:33:20.628952+0000","last_undegraded":"2024-04-01T01:33:20.628952+0000","last_fullsized":"2024-04-01T01:33:20.628952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:27.854459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.837895+0000","last_change":"2024-04-01T01:31:20.576578+0000","last_active":"2024-04-01T01:33:20.837895+0000","last_peered":"2024-04-01T01:33:20.837895+0000","last_clean":"2024-04-01T01:33:20.837895+0000","last_became_active":"2024-04-01T01:31:20.575996+0000","last_became_peered":"2024-04-01T01:31:20.575996+0000","last_unstale":"2024-04-01T01:33:20.837895+0000","last_undegraded":"2024-04-01T01:33:20.837895+0000","last_fullsized":"2024-04-01T01:33:20.837895+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:52:08.808357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722157+0000","last_change":"2024-04-01T01:31:15.377246+0000","last_active":"2024-04-01T01:33:19.722157+0000","last_peered":"2024-04-01T01:33:19.722157+0000","last_clean":"2024-04-01T01:33:19.722157+0000","last_became_active":"2024-04-01T01:31:15.376957+0000","last_became_peered":"2024-04-01T01:31:15.376957+0000","last_unstale":"2024-04-01T01:33:19.722157+0000","last_undegraded":"2024-04-01T01:33:19.722157+0000","last_fullsized":"2024-04-01T01:33:19.722157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:59:34.313984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721661+0000","last_change":"2024-04-01T01:31:16.379031+0000","last_active":"2024-04-01T01:33:19.721661+0000","last_peered":"2024-04-01T01:33:19.721661+0000","last_clean":"2024-04-01T01:33:19.721661+0000","last_became_active":"2024-04-01T01:31:16.378593+0000","last_became_peered":"2024-04-01T01:31:16.378593+0000","last_unstale":"2024-04-01T01:33:19.721661+0000","last_undegraded":"2024-04-01T01:33:19.721661+0000","last_fullsized":"2024-04-01T01:33:19.721661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:31.816103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.628494+0000","last_change":"2024-04-01T01:31:20.578660+0000","last_active":"2024-04-01T01:33:20.628494+0000","last_peered":"2024-04-01T01:33:20.628494+0000","last_clean":"2024-04-01T01:33:20.628494+0000","last_became_active":"2024-04-01T01:31:20.578082+0000","last_became_peered":"2024-04-01T01:31:20.578082+0000","last_unstale":"2024-04-01T01:33:20.628494+0000","last_undegraded":"2024-04-01T01:33:20.628494+0000","last_fullsized":"2024-04-01T01:33:20.628494+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:02:56.570816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'7","reported_seq":277,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:35.719505+0000","last_change":"2024-04-01T01:31:15.373428+0000","last_active":"2024-04-01T01:33:35.719505+0000","last_peered":"2024-04-01T01:33:35.719505+0000","last_clean":"2024-04-01T01:33:35.719505+0000","last_became_active":"2024-04-01T01:31:15.372965+0000","last_became_peered":"2024-04-01T01:31:15.372965+0000","last_unstale":"2024-04-01T01:33:35.719505+0000","last_undegraded":"2024-04-01T01:33:35.719505+0000","last_fullsized":"2024-04-01T01:33:35.719505+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:55:42.211055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":239,"num_read_kb":239,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.609010+0000","last_change":"2024-04-01T01:31:16.382961+0000","last_active":"2024-04-01T01:33:20.609010+0000","last_peered":"2024-04-01T01:33:20.609010+0000","last_clean":"2024-04-01T01:33:20.609010+0000","last_became_active":"2024-04-01T01:31:16.382527+0000","last_became_peered":"2024-04-01T01:31:16.382527+0000","last_unstale":"2024-04-01T01:33:20.609010+0000","last_undegraded":"2024-04-01T01:33:20.609010+0000","last_fullsized":"2024-04-01T01:33:20.609010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:46:20.250064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721640+0000","last_change":"2024-04-01T01:31:20.571842+0000","last_active":"2024-04-01T01:33:19.721640+0000","last_peered":"2024-04-01T01:33:19.721640+0000","last_clean":"2024-04-01T01:33:19.721640+0000","last_became_active":"2024-04-01T01:31:20.571394+0000","last_became_peered":"2024-04-01T01:31:20.571394+0000","last_unstale":"2024-04-01T01:33:19.721640+0000","last_undegraded":"2024-04-01T01:33:19.721640+0000","last_fullsized":"2024-04-01T01:33:19.721640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:51:36.512277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725497+0000","last_change":"2024-04-01T01:31:15.379737+0000","last_active":"2024-04-01T01:33:19.725497+0000","last_peered":"2024-04-01T01:33:19.725497+0000","last_clean":"2024-04-01T01:33:19.725497+0000","last_became_active":"2024-04-01T01:31:15.379158+0000","last_became_peered":"2024-04-01T01:31:15.379158+0000","last_unstale":"2024-04-01T01:33:19.725497+0000","last_undegraded":"2024-04-01T01:33:19.725497+0000","last_fullsized":"2024-04-01T01:33:19.725497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:12:44.474772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838564+0000","last_change":"2024-04-01T01:31:16.386059+0000","last_active":"2024-04-01T01:33:20.838564+0000","last_peered":"2024-04-01T01:33:20.838564+0000","last_clean":"2024-04-01T01:33:20.838564+0000","last_became_active":"2024-04-01T01:31:16.384551+0000","last_became_peered":"2024-04-01T01:31:16.384551+0000","last_unstale":"2024-04-01T01:33:20.838564+0000","last_undegraded":"2024-04-01T01:33:20.838564+0000","last_fullsized":"2024-04-01T01:33:20.838564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:52:16.594485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721972+0000","last_change":"2024-04-01T01:31:20.566937+0000","last_active":"2024-04-01T01:33:19.721972+0000","last_peered":"2024-04-01T01:33:19.721972+0000","last_clean":"2024-04-01T01:33:19.721972+0000","last_became_active":"2024-04-01T01:31:20.566447+0000","last_became_peered":"2024-04-01T01:31:20.566447+0000","last_unstale":"2024-04-01T01:33:19.721972+0000","last_undegraded":"2024-04-01T01:33:19.721972+0000","last_fullsized":"2024-04-01T01:33:19.721972+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:17:37.845166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722425+0000","last_change":"2024-04-01T01:31:15.375669+0000","last_active":"2024-04-01T01:33:19.722425+0000","last_peered":"2024-04-01T01:33:19.722425+0000","last_clean":"2024-04-01T01:33:19.722425+0000","last_became_active":"2024-04-01T01:31:15.375353+0000","last_became_peered":"2024-04-01T01:31:15.375353+0000","last_unstale":"2024-04-01T01:33:19.722425+0000","last_undegraded":"2024-04-01T01:33:19.722425+0000","last_fullsized":"2024-04-01T01:33:19.722425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:47:43.096898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608910+0000","last_change":"2024-04-01T01:31:16.388399+0000","last_active":"2024-04-01T01:33:20.608910+0000","last_peered":"2024-04-01T01:33:20.608910+0000","last_clean":"2024-04-01T01:33:20.608910+0000","last_became_active":"2024-04-01T01:31:16.388059+0000","last_became_peered":"2024-04-01T01:31:16.388059+0000","last_unstale":"2024-04-01T01:33:20.608910+0000","last_undegraded":"2024-04-01T01:33:20.608910+0000","last_fullsized":"2024-04-01T01:33:20.608910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:59:50.907802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.723472+0000","last_change":"2024-04-01T01:32:19.692098+0000","last_active":"2024-04-01T01:33:19.723472+0000","last_peered":"2024-04-01T01:33:19.723472+0000","last_clean":"2024-04-01T01:33:19.723472+0000","last_became_active":"2024-04-01T01:32:19.691673+0000","last_became_peered":"2024-04-01T01:32:19.691673+0000","last_unstale":"2024-04-01T01:33:19.723472+0000","last_undegraded":"2024-04-01T01:33:19.723472+0000","last_fullsized":"2024-04-01T01:33:19.723472+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:14:17.300517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"93'39","reported_seq":323,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:35.766454+0000","last_change":"2024-04-01T01:31:15.376421+0000","last_active":"2024-04-01T01:33:35.766454+0000","last_peered":"2024-04-01T01:33:35.766454+0000","last_clean":"2024-04-01T01:33:35.766454+0000","last_became_active":"2024-04-01T01:31:15.375985+0000","last_became_peered":"2024-04-01T01:31:15.375985+0000","last_unstale":"2024-04-01T01:33:35.766454+0000","last_undegraded":"2024-04-01T01:33:35.766454+0000","last_fullsized":"2024-04-01T01:33:35.766454+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:37:29.121436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64082,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":252,"num_read_kb":189,"num_write":41,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725852+0000","last_change":"2024-04-01T01:31:16.380641+0000","last_active":"2024-04-01T01:33:19.725852+0000","last_peered":"2024-04-01T01:33:19.725852+0000","last_clean":"2024-04-01T01:33:19.725852+0000","last_became_active":"2024-04-01T01:31:16.380300+0000","last_became_peered":"2024-04-01T01:31:16.380300+0000","last_unstale":"2024-04-01T01:33:19.725852+0000","last_undegraded":"2024-04-01T01:33:19.725852+0000","last_fullsized":"2024-04-01T01:33:19.725852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:26:08.240694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722617+0000","last_change":"2024-04-01T01:31:20.576094+0000","last_active":"2024-04-01T01:33:19.722617+0000","last_peered":"2024-04-01T01:33:19.722617+0000","last_clean":"2024-04-01T01:33:19.722617+0000","last_became_active":"2024-04-01T01:31:20.575674+0000","last_became_peered":"2024-04-01T01:31:20.575674+0000","last_unstale":"2024-04-01T01:33:19.722617+0000","last_undegraded":"2024-04-01T01:33:19.722617+0000","last_fullsized":"2024-04-01T01:33:19.722617+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:42:19.352398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727136+0000","last_change":"2024-04-01T01:31:15.375136+0000","last_active":"2024-04-01T01:33:19.727136+0000","last_peered":"2024-04-01T01:33:19.727136+0000","last_clean":"2024-04-01T01:33:19.727136+0000","last_became_active":"2024-04-01T01:31:15.374813+0000","last_became_peered":"2024-04-01T01:31:15.374813+0000","last_unstale":"2024-04-01T01:33:19.727136+0000","last_undegraded":"2024-04-01T01:33:19.727136+0000","last_fullsized":"2024-04-01T01:33:19.727136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:19:05.490103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719129+0000","last_change":"2024-04-01T01:31:16.380207+0000","last_active":"2024-04-01T01:33:19.719129+0000","last_peered":"2024-04-01T01:33:19.719129+0000","last_clean":"2024-04-01T01:33:19.719129+0000","last_became_active":"2024-04-01T01:31:16.379891+0000","last_became_peered":"2024-04-01T01:31:16.379891+0000","last_unstale":"2024-04-01T01:33:19.719129+0000","last_undegraded":"2024-04-01T01:33:19.719129+0000","last_fullsized":"2024-04-01T01:33:19.719129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:12:36.417162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724683+0000","last_change":"2024-04-01T01:31:20.569978+0000","last_active":"2024-04-01T01:33:19.724683+0000","last_peered":"2024-04-01T01:33:19.724683+0000","last_clean":"2024-04-01T01:33:19.724683+0000","last_became_active":"2024-04-01T01:31:20.568298+0000","last_became_peered":"2024-04-01T01:31:20.568298+0000","last_unstale":"2024-04-01T01:33:19.724683+0000","last_undegraded":"2024-04-01T01:33:19.724683+0000","last_fullsized":"2024-04-01T01:33:19.724683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:42:16.439284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722815+0000","last_change":"2024-04-01T01:31:15.380143+0000","last_active":"2024-04-01T01:33:19.722815+0000","last_peered":"2024-04-01T01:33:19.722815+0000","last_clean":"2024-04-01T01:33:19.722815+0000","last_became_active":"2024-04-01T01:31:15.379835+0000","last_became_peered":"2024-04-01T01:31:15.379835+0000","last_unstale":"2024-04-01T01:33:19.722815+0000","last_undegraded":"2024-04-01T01:33:19.722815+0000","last_fullsized":"2024-04-01T01:33:19.722815+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:28:34.213626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725248+0000","last_change":"2024-04-01T01:31:16.380290+0000","last_active":"2024-04-01T01:33:19.725248+0000","last_peered":"2024-04-01T01:33:19.725248+0000","last_clean":"2024-04-01T01:33:19.725248+0000","last_became_active":"2024-04-01T01:31:16.379983+0000","last_became_peered":"2024-04-01T01:31:16.379983+0000","last_unstale":"2024-04-01T01:33:19.725248+0000","last_undegraded":"2024-04-01T01:33:19.725248+0000","last_fullsized":"2024-04-01T01:33:19.725248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:50.135454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721926+0000","last_change":"2024-04-01T01:31:20.573500+0000","last_active":"2024-04-01T01:33:19.721926+0000","last_peered":"2024-04-01T01:33:19.721926+0000","last_clean":"2024-04-01T01:33:19.721926+0000","last_became_active":"2024-04-01T01:31:20.573101+0000","last_became_peered":"2024-04-01T01:31:20.573101+0000","last_unstale":"2024-04-01T01:33:19.721926+0000","last_undegraded":"2024-04-01T01:33:19.721926+0000","last_fullsized":"2024-04-01T01:33:19.721926+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:11:27.881703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'8","reported_seq":163,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:36.723420+0000","last_change":"2024-04-01T01:31:15.377265+0000","last_active":"2024-04-01T01:33:36.723420+0000","last_peered":"2024-04-01T01:33:36.723420+0000","last_clean":"2024-04-01T01:33:36.723420+0000","last_became_active":"2024-04-01T01:31:15.376889+0000","last_became_peered":"2024-04-01T01:31:15.376889+0000","last_unstale":"2024-04-01T01:33:36.723420+0000","last_undegraded":"2024-04-01T01:33:36.723420+0000","last_fullsized":"2024-04-01T01:33:36.723420+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:43:31.582866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1597,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724953+0000","last_change":"2024-04-01T01:31:16.384982+0000","last_active":"2024-04-01T01:33:19.724953+0000","last_peered":"2024-04-01T01:33:19.724953+0000","last_clean":"2024-04-01T01:33:19.724953+0000","last_became_active":"2024-04-01T01:31:16.384663+0000","last_became_peered":"2024-04-01T01:31:16.384663+0000","last_unstale":"2024-04-01T01:33:19.724953+0000","last_undegraded":"2024-04-01T01:33:19.724953+0000","last_fullsized":"2024-04-01T01:33:19.724953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:14:57.351768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719430+0000","last_change":"2024-04-01T01:31:20.578761+0000","last_active":"2024-04-01T01:33:19.719430+0000","last_peered":"2024-04-01T01:33:19.719430+0000","last_clean":"2024-04-01T01:33:19.719430+0000","last_became_active":"2024-04-01T01:31:20.578204+0000","last_became_peered":"2024-04-01T01:31:20.578204+0000","last_unstale":"2024-04-01T01:33:19.719430+0000","last_undegraded":"2024-04-01T01:33:19.719430+0000","last_fullsized":"2024-04-01T01:33:19.719430+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:19:33.584872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724403+0000","last_change":"2024-04-01T01:31:15.380773+0000","last_active":"2024-04-01T01:33:19.724403+0000","last_peered":"2024-04-01T01:33:19.724403+0000","last_clean":"2024-04-01T01:33:19.724403+0000","last_became_active":"2024-04-01T01:31:15.380472+0000","last_became_peered":"2024-04-01T01:31:15.380472+0000","last_unstale":"2024-04-01T01:33:19.724403+0000","last_undegraded":"2024-04-01T01:33:19.724403+0000","last_fullsized":"2024-04-01T01:33:19.724403+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:34:28.294500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722296+0000","last_change":"2024-04-01T01:31:16.374658+0000","last_active":"2024-04-01T01:33:19.722296+0000","last_peered":"2024-04-01T01:33:19.722296+0000","last_clean":"2024-04-01T01:33:19.722296+0000","last_became_active":"2024-04-01T01:31:16.373596+0000","last_became_peered":"2024-04-01T01:31:16.373596+0000","last_unstale":"2024-04-01T01:33:19.722296+0000","last_undegraded":"2024-04-01T01:33:19.722296+0000","last_fullsized":"2024-04-01T01:33:19.722296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:28:38.477719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722239+0000","last_change":"2024-04-01T01:31:20.577449+0000","last_active":"2024-04-01T01:33:19.722239+0000","last_peered":"2024-04-01T01:33:19.722239+0000","last_clean":"2024-04-01T01:33:19.722239+0000","last_became_active":"2024-04-01T01:31:20.577026+0000","last_became_peered":"2024-04-01T01:31:20.577026+0000","last_unstale":"2024-04-01T01:33:19.722239+0000","last_undegraded":"2024-04-01T01:33:19.722239+0000","last_fullsized":"2024-04-01T01:33:19.722239+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:44:48.080304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:33.356435+0000","last_change":"2024-04-01T01:31:15.373885+0000","last_active":"2024-04-01T01:33:33.356435+0000","last_peered":"2024-04-01T01:33:33.356435+0000","last_clean":"2024-04-01T01:33:33.356435+0000","last_became_active":"2024-04-01T01:31:15.373578+0000","last_became_peered":"2024-04-01T01:31:15.373578+0000","last_unstale":"2024-04-01T01:33:33.356435+0000","last_undegraded":"2024-04-01T01:33:33.356435+0000","last_fullsized":"2024-04-01T01:33:33.356435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:52.641515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.730680+0000","last_change":"2024-04-01T01:33:19.730088+0000","last_active":"2024-04-01T01:33:19.730680+0000","last_peered":"2024-04-01T01:33:19.730680+0000","last_clean":"2024-04-01T01:33:19.730680+0000","last_became_active":"2024-04-01T01:33:19.729627+0000","last_became_peered":"2024-04-01T01:33:19.729627+0000","last_unstale":"2024-04-01T01:33:19.730680+0000","last_undegraded":"2024-04-01T01:33:19.730680+0000","last_fullsized":"2024-04-01T01:33:19.730680+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:47:37.839420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725463+0000","last_change":"2024-04-01T01:31:20.569835+0000","last_active":"2024-04-01T01:33:19.725463+0000","last_peered":"2024-04-01T01:33:19.725463+0000","last_clean":"2024-04-01T01:33:19.725463+0000","last_became_active":"2024-04-01T01:31:20.569195+0000","last_became_peered":"2024-04-01T01:31:20.569195+0000","last_unstale":"2024-04-01T01:33:19.725463+0000","last_undegraded":"2024-04-01T01:33:19.725463+0000","last_fullsized":"2024-04-01T01:33:19.725463+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:10:18.278575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721958+0000","last_change":"2024-04-01T01:31:15.377679+0000","last_active":"2024-04-01T01:33:19.721958+0000","last_peered":"2024-04-01T01:33:19.721958+0000","last_clean":"2024-04-01T01:33:19.721958+0000","last_became_active":"2024-04-01T01:31:15.377351+0000","last_became_peered":"2024-04-01T01:31:15.377351+0000","last_unstale":"2024-04-01T01:33:19.721958+0000","last_undegraded":"2024-04-01T01:33:19.721958+0000","last_fullsized":"2024-04-01T01:33:19.721958+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:22:41.216563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724294+0000","last_change":"2024-04-01T01:31:16.380997+0000","last_active":"2024-04-01T01:33:19.724294+0000","last_peered":"2024-04-01T01:33:19.724294+0000","last_clean":"2024-04-01T01:33:19.724294+0000","last_became_active":"2024-04-01T01:31:16.380692+0000","last_became_peered":"2024-04-01T01:31:16.380692+0000","last_unstale":"2024-04-01T01:33:19.724294+0000","last_undegraded":"2024-04-01T01:33:19.724294+0000","last_fullsized":"2024-04-01T01:33:19.724294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:09:51.220849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719249+0000","last_change":"2024-04-01T01:31:20.573191+0000","last_active":"2024-04-01T01:33:19.719249+0000","last_peered":"2024-04-01T01:33:19.719249+0000","last_clean":"2024-04-01T01:33:19.719249+0000","last_became_active":"2024-04-01T01:31:20.572669+0000","last_became_peered":"2024-04-01T01:31:20.572669+0000","last_unstale":"2024-04-01T01:33:19.719249+0000","last_undegraded":"2024-04-01T01:33:19.719249+0000","last_fullsized":"2024-04-01T01:33:19.719249+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:01:56.491271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"93'7","reported_seq":164,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:38.712613+0000","last_change":"2024-04-01T01:31:15.379566+0000","last_active":"2024-04-01T01:33:38.712613+0000","last_peered":"2024-04-01T01:33:38.712613+0000","last_clean":"2024-04-01T01:33:38.712613+0000","last_became_active":"2024-04-01T01:31:15.379213+0000","last_became_peered":"2024-04-01T01:31:15.379213+0000","last_unstale":"2024-04-01T01:33:38.712613+0000","last_undegraded":"2024-04-01T01:33:38.712613+0000","last_fullsized":"2024-04-01T01:33:38.712613+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:34:17.974422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1456,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724071+0000","last_change":"2024-04-01T01:31:16.376218+0000","last_active":"2024-04-01T01:33:19.724071+0000","last_peered":"2024-04-01T01:33:19.724071+0000","last_clean":"2024-04-01T01:33:19.724071+0000","last_became_active":"2024-04-01T01:31:16.375887+0000","last_became_peered":"2024-04-01T01:31:16.375887+0000","last_unstale":"2024-04-01T01:33:19.724071+0000","last_undegraded":"2024-04-01T01:33:19.724071+0000","last_fullsized":"2024-04-01T01:33:19.724071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:27:03.723827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719394+0000","last_change":"2024-04-01T01:31:20.576461+0000","last_active":"2024-04-01T01:33:19.719394+0000","last_peered":"2024-04-01T01:33:19.719394+0000","last_clean":"2024-04-01T01:33:19.719394+0000","last_became_active":"2024-04-01T01:31:20.575985+0000","last_became_peered":"2024-04-01T01:31:20.575985+0000","last_unstale":"2024-04-01T01:33:19.719394+0000","last_undegraded":"2024-04-01T01:33:19.719394+0000","last_fullsized":"2024-04-01T01:33:19.719394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:19:02.714455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722259+0000","last_change":"2024-04-01T01:31:16.376456+0000","last_active":"2024-04-01T01:33:19.722259+0000","last_peered":"2024-04-01T01:33:19.722259+0000","last_clean":"2024-04-01T01:33:19.722259+0000","last_became_active":"2024-04-01T01:31:16.375445+0000","last_became_peered":"2024-04-01T01:31:16.375445+0000","last_unstale":"2024-04-01T01:33:19.722259+0000","last_undegraded":"2024-04-01T01:33:19.722259+0000","last_fullsized":"2024-04-01T01:33:19.722259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:07:11.802799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'7","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:38.725028+0000","last_change":"2024-04-01T01:31:15.382702+0000","last_active":"2024-04-01T01:33:38.725028+0000","last_peered":"2024-04-01T01:33:38.725028+0000","last_clean":"2024-04-01T01:33:38.725028+0000","last_became_active":"2024-04-01T01:31:15.382377+0000","last_became_peered":"2024-04-01T01:31:15.382377+0000","last_unstale":"2024-04-01T01:33:38.725028+0000","last_undegraded":"2024-04-01T01:33:38.725028+0000","last_fullsized":"2024-04-01T01:33:38.725028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:45.080992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725315+0000","last_change":"2024-04-01T01:32:19.704607+0000","last_active":"2024-04-01T01:33:19.725315+0000","last_peered":"2024-04-01T01:33:19.725315+0000","last_clean":"2024-04-01T01:33:19.725315+0000","last_became_active":"2024-04-01T01:32:19.694650+0000","last_became_peered":"2024-04-01T01:32:19.694650+0000","last_unstale":"2024-04-01T01:33:19.725315+0000","last_undegraded":"2024-04-01T01:33:19.725315+0000","last_fullsized":"2024-04-01T01:33:19.725315+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:48:55.439187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629394+0000","last_change":"2024-04-01T01:31:16.387981+0000","last_active":"2024-04-01T01:33:20.629394+0000","last_peered":"2024-04-01T01:33:20.629394+0000","last_clean":"2024-04-01T01:33:20.629394+0000","last_became_active":"2024-04-01T01:31:16.387594+0000","last_became_peered":"2024-04-01T01:31:16.387594+0000","last_unstale":"2024-04-01T01:33:20.629394+0000","last_undegraded":"2024-04-01T01:33:20.629394+0000","last_fullsized":"2024-04-01T01:33:20.629394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:17.148752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721940+0000","last_change":"2024-04-01T01:31:15.375212+0000","last_active":"2024-04-01T01:33:19.721940+0000","last_peered":"2024-04-01T01:33:19.721940+0000","last_clean":"2024-04-01T01:33:19.721940+0000","last_became_active":"2024-04-01T01:31:15.374882+0000","last_became_peered":"2024-04-01T01:31:15.374882+0000","last_unstale":"2024-04-01T01:33:19.721940+0000","last_undegraded":"2024-04-01T01:33:19.721940+0000","last_fullsized":"2024-04-01T01:33:19.721940+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:56:23.412025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724742+0000","last_change":"2024-04-01T01:30:03.538590+0000","last_active":"2024-04-01T01:33:19.724742+0000","last_peered":"2024-04-01T01:33:19.724742+0000","last_clean":"2024-04-01T01:33:19.724742+0000","last_became_active":"2024-04-01T01:30:03.538098+0000","last_became_peered":"2024-04-01T01:30:03.538098+0000","last_unstale":"2024-04-01T01:33:19.724742+0000","last_undegraded":"2024-04-01T01:33:19.724742+0000","last_fullsized":"2024-04-01T01:33:19.724742+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:26:43.445198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:26:43.445198+0000","last_clean_scrub_stamp":"2024-04-01T01:26:43.445198+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:22.207795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722391+0000","last_change":"2024-04-01T01:31:20.573260+0000","last_active":"2024-04-01T01:33:19.722391+0000","last_peered":"2024-04-01T01:33:19.722391+0000","last_clean":"2024-04-01T01:33:19.722391+0000","last_became_active":"2024-04-01T01:31:20.572846+0000","last_became_peered":"2024-04-01T01:31:20.572846+0000","last_unstale":"2024-04-01T01:33:19.722391+0000","last_undegraded":"2024-04-01T01:33:19.722391+0000","last_fullsized":"2024-04-01T01:33:19.722391+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:24.986070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.718978+0000","last_change":"2024-04-01T01:31:16.384782+0000","last_active":"2024-04-01T01:33:19.718978+0000","last_peered":"2024-04-01T01:33:19.718978+0000","last_clean":"2024-04-01T01:33:19.718978+0000","last_became_active":"2024-04-01T01:31:16.384420+0000","last_became_peered":"2024-04-01T01:31:16.384420+0000","last_unstale":"2024-04-01T01:33:19.718978+0000","last_undegraded":"2024-04-01T01:33:19.718978+0000","last_fullsized":"2024-04-01T01:33:19.718978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:00:20.902022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724778+0000","last_change":"2024-04-01T01:31:15.381853+0000","last_active":"2024-04-01T01:33:19.724778+0000","last_peered":"2024-04-01T01:33:19.724778+0000","last_clean":"2024-04-01T01:33:19.724778+0000","last_became_active":"2024-04-01T01:31:15.381496+0000","last_became_peered":"2024-04-01T01:31:15.381496+0000","last_unstale":"2024-04-01T01:33:19.724778+0000","last_undegraded":"2024-04-01T01:33:19.724778+0000","last_fullsized":"2024-04-01T01:33:19.724778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:07:51.440351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724645+0000","last_change":"2024-04-01T01:31:20.568705+0000","last_active":"2024-04-01T01:33:19.724645+0000","last_peered":"2024-04-01T01:33:19.724645+0000","last_clean":"2024-04-01T01:33:19.724645+0000","last_became_active":"2024-04-01T01:31:20.568252+0000","last_became_peered":"2024-04-01T01:31:20.568252+0000","last_unstale":"2024-04-01T01:33:19.724645+0000","last_undegraded":"2024-04-01T01:33:19.724645+0000","last_fullsized":"2024-04-01T01:33:19.724645+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:22:28.509510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724763+0000","last_change":"2024-04-01T01:32:19.699781+0000","last_active":"2024-04-01T01:33:19.724763+0000","last_peered":"2024-04-01T01:33:19.724763+0000","last_clean":"2024-04-01T01:33:19.724763+0000","last_became_active":"2024-04-01T01:32:19.689593+0000","last_became_peered":"2024-04-01T01:32:19.689593+0000","last_unstale":"2024-04-01T01:33:19.724763+0000","last_undegraded":"2024-04-01T01:33:19.724763+0000","last_fullsized":"2024-04-01T01:33:19.724763+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:44:21.922097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719384+0000","last_change":"2024-04-01T01:31:16.377002+0000","last_active":"2024-04-01T01:33:19.719384+0000","last_peered":"2024-04-01T01:33:19.719384+0000","last_clean":"2024-04-01T01:33:19.719384+0000","last_became_active":"2024-04-01T01:31:16.376679+0000","last_became_peered":"2024-04-01T01:31:16.376679+0000","last_unstale":"2024-04-01T01:33:19.719384+0000","last_undegraded":"2024-04-01T01:33:19.719384+0000","last_fullsized":"2024-04-01T01:33:19.719384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:50:24.126192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.628550+0000","last_change":"2024-04-01T01:31:20.577757+0000","last_active":"2024-04-01T01:33:20.628550+0000","last_peered":"2024-04-01T01:33:20.628550+0000","last_clean":"2024-04-01T01:33:20.628550+0000","last_became_active":"2024-04-01T01:31:20.577378+0000","last_became_peered":"2024-04-01T01:31:20.577378+0000","last_unstale":"2024-04-01T01:33:20.628550+0000","last_undegraded":"2024-04-01T01:33:20.628550+0000","last_fullsized":"2024-04-01T01:33:20.628550+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:48:04.511229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'16","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:36.559784+0000","last_change":"2024-04-01T01:31:15.383201+0000","last_active":"2024-04-01T01:33:36.559784+0000","last_peered":"2024-04-01T01:33:36.559784+0000","last_clean":"2024-04-01T01:33:36.559784+0000","last_became_active":"2024-04-01T01:31:15.382320+0000","last_became_peered":"2024-04-01T01:31:15.382320+0000","last_unstale":"2024-04-01T01:33:36.559784+0000","last_undegraded":"2024-04-01T01:33:36.559784+0000","last_fullsized":"2024-04-01T01:33:36.559784+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:15:47.138024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629854+0000","last_change":"2024-04-01T01:31:16.386936+0000","last_active":"2024-04-01T01:33:20.629854+0000","last_peered":"2024-04-01T01:33:20.629854+0000","last_clean":"2024-04-01T01:33:20.629854+0000","last_became_active":"2024-04-01T01:31:16.384448+0000","last_became_peered":"2024-04-01T01:31:16.384448+0000","last_unstale":"2024-04-01T01:33:20.629854+0000","last_undegraded":"2024-04-01T01:33:20.629854+0000","last_fullsized":"2024-04-01T01:33:20.629854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:16:42.243250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.726698+0000","last_change":"2024-04-01T01:31:20.565570+0000","last_active":"2024-04-01T01:33:19.726698+0000","last_peered":"2024-04-01T01:33:19.726698+0000","last_clean":"2024-04-01T01:33:19.726698+0000","last_became_active":"2024-04-01T01:31:20.565141+0000","last_became_peered":"2024-04-01T01:31:20.565141+0000","last_unstale":"2024-04-01T01:33:19.726698+0000","last_undegraded":"2024-04-01T01:33:19.726698+0000","last_fullsized":"2024-04-01T01:33:19.726698+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:47:12.844970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722422+0000","last_change":"2024-04-01T01:31:15.381343+0000","last_active":"2024-04-01T01:33:19.722422+0000","last_peered":"2024-04-01T01:33:19.722422+0000","last_clean":"2024-04-01T01:33:19.722422+0000","last_became_active":"2024-04-01T01:31:15.380994+0000","last_became_peered":"2024-04-01T01:31:15.380994+0000","last_unstale":"2024-04-01T01:33:19.722422+0000","last_undegraded":"2024-04-01T01:33:19.722422+0000","last_fullsized":"2024-04-01T01:33:19.722422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:19:10.710285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838567+0000","last_change":"2024-04-01T01:31:16.383951+0000","last_active":"2024-04-01T01:33:20.838567+0000","last_peered":"2024-04-01T01:33:20.838567+0000","last_clean":"2024-04-01T01:33:20.838567+0000","last_became_active":"2024-04-01T01:31:16.383657+0000","last_became_peered":"2024-04-01T01:31:16.383657+0000","last_unstale":"2024-04-01T01:33:20.838567+0000","last_undegraded":"2024-04-01T01:33:20.838567+0000","last_fullsized":"2024-04-01T01:33:20.838567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:55:45.216479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722156+0000","last_change":"2024-04-01T01:31:20.569416+0000","last_active":"2024-04-01T01:33:19.722156+0000","last_peered":"2024-04-01T01:33:19.722156+0000","last_clean":"2024-04-01T01:33:19.722156+0000","last_became_active":"2024-04-01T01:31:20.569007+0000","last_became_peered":"2024-04-01T01:31:20.569007+0000","last_unstale":"2024-04-01T01:33:19.722156+0000","last_undegraded":"2024-04-01T01:33:19.722156+0000","last_fullsized":"2024-04-01T01:33:19.722156+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:25:36.770919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608610+0000","last_change":"2024-04-01T01:31:15.373444+0000","last_active":"2024-04-01T01:33:20.608610+0000","last_peered":"2024-04-01T01:33:20.608610+0000","last_clean":"2024-04-01T01:33:20.608610+0000","last_became_active":"2024-04-01T01:31:15.373094+0000","last_became_peered":"2024-04-01T01:31:15.373094+0000","last_unstale":"2024-04-01T01:33:20.608610+0000","last_undegraded":"2024-04-01T01:33:20.608610+0000","last_fullsized":"2024-04-01T01:33:20.608610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:26:27.443832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724496+0000","last_change":"2024-04-01T01:31:16.381441+0000","last_active":"2024-04-01T01:33:19.724496+0000","last_peered":"2024-04-01T01:33:19.724496+0000","last_clean":"2024-04-01T01:33:19.724496+0000","last_became_active":"2024-04-01T01:31:16.381092+0000","last_became_peered":"2024-04-01T01:31:16.381092+0000","last_unstale":"2024-04-01T01:33:19.724496+0000","last_undegraded":"2024-04-01T01:33:19.724496+0000","last_fullsized":"2024-04-01T01:33:19.724496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:47:00.229090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722053+0000","last_change":"2024-04-01T01:31:20.569672+0000","last_active":"2024-04-01T01:33:19.722053+0000","last_peered":"2024-04-01T01:33:19.722053+0000","last_clean":"2024-04-01T01:33:19.722053+0000","last_became_active":"2024-04-01T01:31:20.569276+0000","last_became_peered":"2024-04-01T01:31:20.569276+0000","last_unstale":"2024-04-01T01:33:19.722053+0000","last_undegraded":"2024-04-01T01:33:19.722053+0000","last_fullsized":"2024-04-01T01:33:19.722053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:39:03.331428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629526+0000","last_change":"2024-04-01T01:31:15.376094+0000","last_active":"2024-04-01T01:33:20.629526+0000","last_peered":"2024-04-01T01:33:20.629526+0000","last_clean":"2024-04-01T01:33:20.629526+0000","last_became_active":"2024-04-01T01:31:15.375169+0000","last_became_peered":"2024-04-01T01:31:15.375169+0000","last_unstale":"2024-04-01T01:33:20.629526+0000","last_undegraded":"2024-04-01T01:33:20.629526+0000","last_fullsized":"2024-04-01T01:33:20.629526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:18:20.710644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724497+0000","last_change":"2024-04-01T01:31:16.378514+0000","last_active":"2024-04-01T01:33:19.724497+0000","last_peered":"2024-04-01T01:33:19.724497+0000","last_clean":"2024-04-01T01:33:19.724497+0000","last_became_active":"2024-04-01T01:31:16.377990+0000","last_became_peered":"2024-04-01T01:31:16.377990+0000","last_unstale":"2024-04-01T01:33:19.724497+0000","last_undegraded":"2024-04-01T01:33:19.724497+0000","last_fullsized":"2024-04-01T01:33:19.724497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:27:29.829775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721831+0000","last_change":"2024-04-01T01:31:20.567892+0000","last_active":"2024-04-01T01:33:19.721831+0000","last_peered":"2024-04-01T01:33:19.721831+0000","last_clean":"2024-04-01T01:33:19.721831+0000","last_became_active":"2024-04-01T01:31:20.567525+0000","last_became_peered":"2024-04-01T01:31:20.567525+0000","last_unstale":"2024-04-01T01:33:19.721831+0000","last_undegraded":"2024-04-01T01:33:19.721831+0000","last_fullsized":"2024-04-01T01:33:19.721831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:57.828691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'5","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:35.564105+0000","last_change":"2024-04-01T01:32:19.703414+0000","last_active":"2024-04-01T01:33:35.564105+0000","last_peered":"2024-04-01T01:33:35.564105+0000","last_clean":"2024-04-01T01:33:35.564105+0000","last_became_active":"2024-04-01T01:32:19.693581+0000","last_became_peered":"2024-04-01T01:32:19.693581+0000","last_unstale":"2024-04-01T01:33:35.564105+0000","last_undegraded":"2024-04-01T01:33:35.564105+0000","last_fullsized":"2024-04-01T01:33:35.564105+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:53:17.897154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629533+0000","last_change":"2024-04-01T01:31:16.387570+0000","last_active":"2024-04-01T01:33:20.629533+0000","last_peered":"2024-04-01T01:33:20.629533+0000","last_clean":"2024-04-01T01:33:20.629533+0000","last_became_active":"2024-04-01T01:31:16.387200+0000","last_became_peered":"2024-04-01T01:31:16.387200+0000","last_unstale":"2024-04-01T01:33:20.629533+0000","last_undegraded":"2024-04-01T01:33:20.629533+0000","last_fullsized":"2024-04-01T01:33:20.629533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:52:27.813722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722104+0000","last_change":"2024-04-01T01:31:20.569668+0000","last_active":"2024-04-01T01:33:19.722104+0000","last_peered":"2024-04-01T01:33:19.722104+0000","last_clean":"2024-04-01T01:33:19.722104+0000","last_became_active":"2024-04-01T01:31:20.569148+0000","last_became_peered":"2024-04-01T01:31:20.569148+0000","last_unstale":"2024-04-01T01:33:19.722104+0000","last_undegraded":"2024-04-01T01:33:19.722104+0000","last_fullsized":"2024-04-01T01:33:19.722104+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:20:56.048052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724566+0000","last_change":"2024-04-01T01:31:15.379323+0000","last_active":"2024-04-01T01:33:19.724566+0000","last_peered":"2024-04-01T01:33:19.724566+0000","last_clean":"2024-04-01T01:33:19.724566+0000","last_became_active":"2024-04-01T01:31:15.379023+0000","last_became_peered":"2024-04-01T01:31:15.379023+0000","last_unstale":"2024-04-01T01:33:19.724566+0000","last_undegraded":"2024-04-01T01:33:19.724566+0000","last_fullsized":"2024-04-01T01:33:19.724566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:05:37.709040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725164+0000","last_change":"2024-04-01T01:31:16.387250+0000","last_active":"2024-04-01T01:33:19.725164+0000","last_peered":"2024-04-01T01:33:19.725164+0000","last_clean":"2024-04-01T01:33:19.725164+0000","last_became_active":"2024-04-01T01:31:16.386940+0000","last_became_peered":"2024-04-01T01:31:16.386940+0000","last_unstale":"2024-04-01T01:33:19.725164+0000","last_undegraded":"2024-04-01T01:33:19.725164+0000","last_fullsized":"2024-04-01T01:33:19.725164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:19:34.886738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608300+0000","last_change":"2024-04-01T01:31:20.573228+0000","last_active":"2024-04-01T01:33:20.608300+0000","last_peered":"2024-04-01T01:33:20.608300+0000","last_clean":"2024-04-01T01:33:20.608300+0000","last_became_active":"2024-04-01T01:31:20.572681+0000","last_became_peered":"2024-04-01T01:31:20.572681+0000","last_unstale":"2024-04-01T01:33:20.608300+0000","last_undegraded":"2024-04-01T01:33:20.608300+0000","last_fullsized":"2024-04-01T01:33:20.608300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:58:17.812966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'7","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:37.723133+0000","last_change":"2024-04-01T01:31:15.381477+0000","last_active":"2024-04-01T01:33:37.723133+0000","last_peered":"2024-04-01T01:33:37.723133+0000","last_clean":"2024-04-01T01:33:37.723133+0000","last_became_active":"2024-04-01T01:31:15.381166+0000","last_became_peered":"2024-04-01T01:31:15.381166+0000","last_unstale":"2024-04-01T01:33:37.723133+0000","last_undegraded":"2024-04-01T01:33:37.723133+0000","last_fullsized":"2024-04-01T01:33:37.723133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:19.029896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2038,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725492+0000","last_change":"2024-04-01T01:31:16.386234+0000","last_active":"2024-04-01T01:33:19.725492+0000","last_peered":"2024-04-01T01:33:19.725492+0000","last_clean":"2024-04-01T01:33:19.725492+0000","last_became_active":"2024-04-01T01:31:16.385876+0000","last_became_peered":"2024-04-01T01:31:16.385876+0000","last_unstale":"2024-04-01T01:33:19.725492+0000","last_undegraded":"2024-04-01T01:33:19.725492+0000","last_fullsized":"2024-04-01T01:33:19.725492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:34.187210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838323+0000","last_change":"2024-04-01T01:32:19.805473+0000","last_active":"2024-04-01T01:33:20.838323+0000","last_peered":"2024-04-01T01:33:20.838323+0000","last_clean":"2024-04-01T01:33:20.838323+0000","last_became_active":"2024-04-01T01:32:19.805151+0000","last_became_peered":"2024-04-01T01:32:19.805151+0000","last_unstale":"2024-04-01T01:33:20.838323+0000","last_undegraded":"2024-04-01T01:33:20.838323+0000","last_fullsized":"2024-04-01T01:33:20.838323+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:47:54.539407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722570+0000","last_change":"2024-04-01T01:31:15.378503+0000","last_active":"2024-04-01T01:33:19.722570+0000","last_peered":"2024-04-01T01:33:19.722570+0000","last_clean":"2024-04-01T01:33:19.722570+0000","last_became_active":"2024-04-01T01:31:15.378152+0000","last_became_peered":"2024-04-01T01:31:15.378152+0000","last_unstale":"2024-04-01T01:33:19.722570+0000","last_undegraded":"2024-04-01T01:33:19.722570+0000","last_fullsized":"2024-04-01T01:33:19.722570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:15:42.118691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.728781+0000","last_change":"2024-04-01T01:33:19.728322+0000","last_active":"2024-04-01T01:33:19.728781+0000","last_peered":"2024-04-01T01:33:19.728781+0000","last_clean":"2024-04-01T01:33:19.728781+0000","last_became_active":"2024-04-01T01:33:19.727846+0000","last_became_peered":"2024-04-01T01:33:19.727846+0000","last_unstale":"2024-04-01T01:33:19.728781+0000","last_undegraded":"2024-04-01T01:33:19.728781+0000","last_fullsized":"2024-04-01T01:33:19.728781+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:54.899382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722019+0000","last_change":"2024-04-01T01:31:20.569461+0000","last_active":"2024-04-01T01:33:19.722019+0000","last_peered":"2024-04-01T01:33:19.722019+0000","last_clean":"2024-04-01T01:33:19.722019+0000","last_became_active":"2024-04-01T01:31:20.569011+0000","last_became_peered":"2024-04-01T01:31:20.569011+0000","last_unstale":"2024-04-01T01:33:19.722019+0000","last_undegraded":"2024-04-01T01:33:19.722019+0000","last_fullsized":"2024-04-01T01:33:19.722019+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:43:43.657065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719451+0000","last_change":"2024-04-01T01:31:15.384181+0000","last_active":"2024-04-01T01:33:19.719451+0000","last_peered":"2024-04-01T01:33:19.719451+0000","last_clean":"2024-04-01T01:33:19.719451+0000","last_became_active":"2024-04-01T01:31:15.383886+0000","last_became_peered":"2024-04-01T01:31:15.383886+0000","last_unstale":"2024-04-01T01:33:19.719451+0000","last_undegraded":"2024-04-01T01:33:19.719451+0000","last_fullsized":"2024-04-01T01:33:19.719451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:43:28.414423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725363+0000","last_change":"2024-04-01T01:31:16.381877+0000","last_active":"2024-04-01T01:33:19.725363+0000","last_peered":"2024-04-01T01:33:19.725363+0000","last_clean":"2024-04-01T01:33:19.725363+0000","last_became_active":"2024-04-01T01:31:16.381558+0000","last_became_peered":"2024-04-01T01:31:16.381558+0000","last_unstale":"2024-04-01T01:33:19.725363+0000","last_undegraded":"2024-04-01T01:33:19.725363+0000","last_fullsized":"2024-04-01T01:33:19.725363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:24.120910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.730654+0000","last_change":"2024-04-01T01:33:19.730079+0000","last_active":"2024-04-01T01:33:19.730654+0000","last_peered":"2024-04-01T01:33:19.730654+0000","last_clean":"2024-04-01T01:33:19.730654+0000","last_became_active":"2024-04-01T01:33:19.729631+0000","last_became_peered":"2024-04-01T01:33:19.729631+0000","last_unstale":"2024-04-01T01:33:19.730654+0000","last_undegraded":"2024-04-01T01:33:19.730654+0000","last_fullsized":"2024-04-01T01:33:19.730654+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:30.299413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727472+0000","last_change":"2024-04-01T01:31:16.384872+0000","last_active":"2024-04-01T01:33:19.727472+0000","last_peered":"2024-04-01T01:33:19.727472+0000","last_clean":"2024-04-01T01:33:19.727472+0000","last_became_active":"2024-04-01T01:31:16.384406+0000","last_became_peered":"2024-04-01T01:31:16.384406+0000","last_unstale":"2024-04-01T01:33:19.727472+0000","last_undegraded":"2024-04-01T01:33:19.727472+0000","last_fullsized":"2024-04-01T01:33:19.727472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:13:36.567357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727117+0000","last_change":"2024-04-01T01:31:16.384561+0000","last_active":"2024-04-01T01:33:19.727117+0000","last_peered":"2024-04-01T01:33:19.727117+0000","last_clean":"2024-04-01T01:33:19.727117+0000","last_became_active":"2024-04-01T01:31:16.384221+0000","last_became_peered":"2024-04-01T01:31:16.384221+0000","last_unstale":"2024-04-01T01:33:19.727117+0000","last_undegraded":"2024-04-01T01:33:19.727117+0000","last_fullsized":"2024-04-01T01:33:19.727117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:17:56.490000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838506+0000","last_change":"2024-04-01T01:31:16.385464+0000","last_active":"2024-04-01T01:33:20.838506+0000","last_peered":"2024-04-01T01:33:20.838506+0000","last_clean":"2024-04-01T01:33:20.838506+0000","last_became_active":"2024-04-01T01:31:16.385159+0000","last_became_peered":"2024-04-01T01:31:16.385159+0000","last_unstale":"2024-04-01T01:33:20.838506+0000","last_undegraded":"2024-04-01T01:33:20.838506+0000","last_fullsized":"2024-04-01T01:33:20.838506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:33:53.852957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719377+0000","last_change":"2024-04-01T01:31:16.385688+0000","last_active":"2024-04-01T01:33:19.719377+0000","last_peered":"2024-04-01T01:33:19.719377+0000","last_clean":"2024-04-01T01:33:19.719377+0000","last_became_active":"2024-04-01T01:31:16.385381+0000","last_became_peered":"2024-04-01T01:31:16.385381+0000","last_unstale":"2024-04-01T01:33:19.719377+0000","last_undegraded":"2024-04-01T01:33:19.719377+0000","last_fullsized":"2024-04-01T01:33:19.719377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:39:36.810766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725031+0000","last_change":"2024-04-01T01:31:16.379976+0000","last_active":"2024-04-01T01:33:19.725031+0000","last_peered":"2024-04-01T01:33:19.725031+0000","last_clean":"2024-04-01T01:33:19.725031+0000","last_became_active":"2024-04-01T01:31:16.379553+0000","last_became_peered":"2024-04-01T01:31:16.379553+0000","last_unstale":"2024-04-01T01:33:19.725031+0000","last_undegraded":"2024-04-01T01:33:19.725031+0000","last_fullsized":"2024-04-01T01:33:19.725031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:48:23.555723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727822+0000","last_change":"2024-04-01T01:31:16.383995+0000","last_active":"2024-04-01T01:33:19.727822+0000","last_peered":"2024-04-01T01:33:19.727822+0000","last_clean":"2024-04-01T01:33:19.727822+0000","last_became_active":"2024-04-01T01:31:16.383673+0000","last_became_peered":"2024-04-01T01:31:16.383673+0000","last_unstale":"2024-04-01T01:33:19.727822+0000","last_undegraded":"2024-04-01T01:33:19.727822+0000","last_fullsized":"2024-04-01T01:33:19.727822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:03:16.192438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629879+0000","last_change":"2024-04-01T01:31:16.389969+0000","last_active":"2024-04-01T01:33:20.629879+0000","last_peered":"2024-04-01T01:33:20.629879+0000","last_clean":"2024-04-01T01:33:20.629879+0000","last_became_active":"2024-04-01T01:31:16.389640+0000","last_became_peered":"2024-04-01T01:31:16.389640+0000","last_unstale":"2024-04-01T01:33:20.629879+0000","last_undegraded":"2024-04-01T01:33:20.629879+0000","last_fullsized":"2024-04-01T01:33:20.629879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:35:15.226960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722223+0000","last_change":"2024-04-01T01:31:16.382962+0000","last_active":"2024-04-01T01:33:19.722223+0000","last_peered":"2024-04-01T01:33:19.722223+0000","last_clean":"2024-04-01T01:33:19.722223+0000","last_became_active":"2024-04-01T01:31:16.382653+0000","last_became_peered":"2024-04-01T01:31:16.382653+0000","last_unstale":"2024-04-01T01:33:19.722223+0000","last_undegraded":"2024-04-01T01:33:19.722223+0000","last_fullsized":"2024-04-01T01:33:19.722223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:53.054884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629538+0000","last_change":"2024-04-01T01:31:16.382997+0000","last_active":"2024-04-01T01:33:20.629538+0000","last_peered":"2024-04-01T01:33:20.629538+0000","last_clean":"2024-04-01T01:33:20.629538+0000","last_became_active":"2024-04-01T01:31:16.380180+0000","last_became_peered":"2024-04-01T01:31:16.380180+0000","last_unstale":"2024-04-01T01:33:20.629538+0000","last_undegraded":"2024-04-01T01:33:20.629538+0000","last_fullsized":"2024-04-01T01:33:20.629538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:29:51.990395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608760+0000","last_change":"2024-04-01T01:31:16.382948+0000","last_active":"2024-04-01T01:33:20.608760+0000","last_peered":"2024-04-01T01:33:20.608760+0000","last_clean":"2024-04-01T01:33:20.608760+0000","last_became_active":"2024-04-01T01:31:16.382134+0000","last_became_peered":"2024-04-01T01:31:16.382134+0000","last_unstale":"2024-04-01T01:33:20.608760+0000","last_undegraded":"2024-04-01T01:33:20.608760+0000","last_fullsized":"2024-04-01T01:33:20.608760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:24:08.515063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719262+0000","last_change":"2024-04-01T01:31:16.378841+0000","last_active":"2024-04-01T01:33:19.719262+0000","last_peered":"2024-04-01T01:33:19.719262+0000","last_clean":"2024-04-01T01:33:19.719262+0000","last_became_active":"2024-04-01T01:31:16.378501+0000","last_became_peered":"2024-04-01T01:31:16.378501+0000","last_unstale":"2024-04-01T01:33:19.719262+0000","last_undegraded":"2024-04-01T01:33:19.719262+0000","last_fullsized":"2024-04-01T01:33:19.719262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:50:20.881852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724900+0000","last_change":"2024-04-01T01:31:16.380979+0000","last_active":"2024-04-01T01:33:19.724900+0000","last_peered":"2024-04-01T01:33:19.724900+0000","last_clean":"2024-04-01T01:33:19.724900+0000","last_became_active":"2024-04-01T01:31:16.380625+0000","last_became_peered":"2024-04-01T01:31:16.380625+0000","last_unstale":"2024-04-01T01:33:19.724900+0000","last_undegraded":"2024-04-01T01:33:19.724900+0000","last_fullsized":"2024-04-01T01:33:19.724900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:09:03.246239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74258,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1247,"num_read_kb":710,"num_write":274,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1125,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":585728,"data_stored":231402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1073152,"data_stored":520978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37056,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708096,"statfs":{"total":95995035648,"available":95957090304,"internally_reserved":0,"allocated":1761280,"data_stored":1172296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":524415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070521,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":524681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36980,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708172,"statfs":{"total":95995035648,"available":95957168128,"internally_reserved":0,"allocated":1683456,"data_stored":1119689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":516691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1138688,"data_stored":586508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":517756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1146880,"data_stored":581010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":515531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":522537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28664,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716488,"statfs":{"total":95995035648,"available":95965683712,"internally_reserved":0,"allocated":1687552,"data_stored":1109298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":71005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-01T01:33:41.815 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T01:33:41.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.953+0000 7ff71f9b5640 1 -- 172.21.15.70:0/35848536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff718102220 msgr2=0x7ff718108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:41.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.953+0000 7ff71f9b5640 1 --2- 172.21.15.70:0/35848536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff718102220 0x7ff718108ab0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff710009be0 tx=0x7ff71002bef0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.953+0000 7ff71f9b5640 1 -- 172.21.15.70:0/35848536 shutdown_connections 2024-04-01T01:33:41.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.953+0000 7ff71f9b5640 1 --2- 172.21.15.70:0/35848536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff718102220 0x7ff718108ab0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.953+0000 7ff71f9b5640 1 --2- 172.21.15.70:0/35848536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff718101860 0x7ff718101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.953+0000 7ff71f9b5640 1 --2- 172.21.15.70:0/35848536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff718100670 0x7ff718100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.953+0000 7ff71f9b5640 1 -- 172.21.15.70:0/35848536 >> 172.21.15.70:0/35848536 conn(0x7ff7180fbbe0 msgr2=0x7ff7180fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:41.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.953+0000 7ff71f9b5640 1 -- 172.21.15.70:0/35848536 shutdown_connections 2024-04-01T01:33:41.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.953+0000 7ff71f9b5640 1 -- 172.21.15.70:0/35848536 wait complete. 2024-04-01T01:33:41.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.953+0000 7ff71f9b5640 1 Processor -- start 2024-04-01T01:33:41.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.953+0000 7ff71f9b5640 1 -- start start 2024-04-01T01:33:41.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71f9b5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff718100670 0x7ff718197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:41.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71f9b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff718101860 0x7ff718197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:41.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71f9b5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff718102220 0x7ff71819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:41.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71f9b5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff71810c1b0 con 0x7ff718101860 2024-04-01T01:33:41.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71f9b5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff71810c030 con 0x7ff718102220 2024-04-01T01:33:41.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71f9b5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff71810c330 con 0x7ff718100670 2024-04-01T01:33:41.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71cf50640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff718101860 0x7ff718197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:41.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71cf50640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff718101860 0x7ff718197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55442/0 (socket says 172.21.15.70:55442) 2024-04-01T01:33:41.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71cf50640 1 -- 172.21.15.70:0/2230048018 learned_addr learned my addr 172.21.15.70:0/2230048018 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:41.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71df52640 1 --2- 172.21.15.70:0/2230048018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff718102220 0x7ff71819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:41.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71d751640 1 --2- 172.21.15.70:0/2230048018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff718100670 0x7ff718197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:41.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71cf50640 1 -- 172.21.15.70:0/2230048018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff718100670 msgr2=0x7ff718197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:41.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71cf50640 1 --2- 172.21.15.70:0/2230048018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff718100670 0x7ff718197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71cf50640 1 -- 172.21.15.70:0/2230048018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff718102220 msgr2=0x7ff71819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:41.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71cf50640 1 --2- 172.21.15.70:0/2230048018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff718102220 0x7ff71819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:41.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71cf50640 1 -- 172.21.15.70:0/2230048018 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff71819f590 con 0x7ff718101860 2024-04-01T01:33:41.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71cf50640 1 --2- 172.21.15.70:0/2230048018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff718101860 0x7ff718197eb0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff70800e9f0 tx=0x7ff70800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:41.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff70e7fc640 1 -- 172.21.15.70:0/2230048018 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70800cd80 con 0x7ff718101860 2024-04-01T01:33:41.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71f9b5640 1 -- 172.21.15.70:0/2230048018 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff71819f880 con 0x7ff718101860 2024-04-01T01:33:41.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff70e7fc640 1 -- 172.21.15.70:0/2230048018 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7080054d0 con 0x7ff718101860 2024-04-01T01:33:41.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff70e7fc640 1 -- 172.21.15.70:0/2230048018 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff708010490 con 0x7ff718101860 2024-04-01T01:33:41.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.954+0000 7ff71f9b5640 1 -- 172.21.15.70:0/2230048018 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff71819fd80 con 0x7ff718101860 2024-04-01T01:33:41.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.955+0000 7ff71f9b5640 1 -- 172.21.15.70:0/2230048018 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7181a0040 con 0x7ff718101860 2024-04-01T01:33:41.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.956+0000 7ff70e7fc640 1 -- 172.21.15.70:0/2230048018 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7ff708005020 con 0x7ff718101860 2024-04-01T01:33:41.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.957+0000 7ff70e7fc640 1 --2- 172.21.15.70:0/2230048018 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7040a2530 0x7ff7040a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:41.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.957+0000 7ff71d751640 1 --2- 172.21.15.70:0/2230048018 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7040a2530 0x7ff7040a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:41.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.957+0000 7ff70e7fc640 1 -- 172.21.15.70:0/2230048018 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff7080cefe0 con 0x7ff718101860 2024-04-01T01:33:41.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.957+0000 7ff71d751640 1 --2- 172.21.15.70:0/2230048018 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7040a2530 0x7ff7040a49f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff714009530 tx=0x7ff714009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:41.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:41.961+0000 7ff70e7fc640 1 -- 172.21.15.70:0/2230048018 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff70809bc70 con 0x7ff718101860 2024-04-01T01:33:41.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:41 smithi189 ceph-mon[35753]: pgmap v364: 129 pgs: 129 active+clean; 649 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-01T01:33:41.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:41 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/432891716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:41.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:41 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2449726824' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:41.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:41 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3992290788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:42.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:41 smithi070 ceph-mon[32179]: pgmap v364: 129 pgs: 129 active+clean; 649 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-01T01:33:42.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:41 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/432891716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:42.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:41 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2449726824' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:42.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:41 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3992290788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:41 smithi081 ceph-mon[35142]: pgmap v364: 129 pgs: 129 active+clean; 649 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-01T01:33:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:41 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/432891716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:41 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2449726824' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:41 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3992290788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:42.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.135+0000 7ff71f9b5640 1 -- 172.21.15.70:0/2230048018 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff718068fc0 con 0x7ff7040a2530 2024-04-01T01:33:42.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.144+0000 7ff70e7fc640 1 -- 172.21.15.70:0/2230048018 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340534 (secure 0 0 0) 0x7ff718068fc0 con 0x7ff7040a2530 2024-04-01T01:33:42.151 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:42.152 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-01T01:33:42.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.147+0000 7ff71f9b5640 1 -- 172.21.15.70:0/2230048018 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7040a2530 msgr2=0x7ff7040a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:42.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.147+0000 7ff71f9b5640 1 --2- 172.21.15.70:0/2230048018 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7040a2530 0x7ff7040a49f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff714009530 tx=0x7ff714009260 comp rx=0 tx=0).stop 2024-04-01T01:33:42.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.147+0000 7ff71f9b5640 1 -- 172.21.15.70:0/2230048018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff718101860 msgr2=0x7ff718197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:42.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.147+0000 7ff71f9b5640 1 --2- 172.21.15.70:0/2230048018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff718101860 0x7ff718197eb0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff70800e9f0 tx=0x7ff70800edf0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.148+0000 7ff71f9b5640 1 -- 172.21.15.70:0/2230048018 shutdown_connections 2024-04-01T01:33:42.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.148+0000 7ff71f9b5640 1 --2- 172.21.15.70:0/2230048018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff718102220 0x7ff71819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.148+0000 7ff71f9b5640 1 --2- 172.21.15.70:0/2230048018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff718101860 0x7ff718197eb0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.148+0000 7ff71f9b5640 1 --2- 172.21.15.70:0/2230048018 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7040a2530 0x7ff7040a49f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.148+0000 7ff71f9b5640 1 --2- 172.21.15.70:0/2230048018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff718100670 0x7ff718197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.148+0000 7ff71f9b5640 1 -- 172.21.15.70:0/2230048018 >> 172.21.15.70:0/2230048018 conn(0x7ff7180fbbe0 msgr2=0x7ff7180688b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:42.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.148+0000 7ff71f9b5640 1 -- 172.21.15.70:0/2230048018 shutdown_connections 2024-04-01T01:33:42.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.148+0000 7ff71f9b5640 1 -- 172.21.15.70:0/2230048018 wait complete. 2024-04-01T01:33:42.158 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-04-01T01:33:40.604455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664860,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1373,"num_read_kb":819,"num_write":457,"num_write_kb":3105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181978,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":875,"ondisk_log_size":875,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":359628,"kb_used_data":14628,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124582196,"statfs":{"total":1151940427776,"available":1151572168704,"internally_reserved":0,"allocated":14979072,"data_stored":8211390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3167,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":121,"num_read_kb":66,"num_write":24,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003260"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725372+0000","last_change":"2024-04-01T01:31:16.385517+0000","last_active":"2024-04-01T01:33:19.725372+0000","last_peered":"2024-04-01T01:33:19.725372+0000","last_clean":"2024-04-01T01:33:19.725372+0000","last_became_active":"2024-04-01T01:31:16.384682+0000","last_became_peered":"2024-04-01T01:31:16.384682+0000","last_unstale":"2024-04-01T01:33:19.725372+0000","last_undegraded":"2024-04-01T01:33:19.725372+0000","last_fullsized":"2024-04-01T01:33:19.725372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:35:47.614041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838551+0000","last_change":"2024-04-01T01:31:16.376119+0000","last_active":"2024-04-01T01:33:20.838551+0000","last_peered":"2024-04-01T01:33:20.838551+0000","last_clean":"2024-04-01T01:33:20.838551+0000","last_became_active":"2024-04-01T01:31:16.375801+0000","last_became_peered":"2024-04-01T01:31:16.375801+0000","last_unstale":"2024-04-01T01:33:20.838551+0000","last_undegraded":"2024-04-01T01:33:20.838551+0000","last_fullsized":"2024-04-01T01:33:20.838551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:34.120487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838381+0000","last_change":"2024-04-01T01:31:16.378158+0000","last_active":"2024-04-01T01:33:20.838381+0000","last_peered":"2024-04-01T01:33:20.838381+0000","last_clean":"2024-04-01T01:33:20.838381+0000","last_became_active":"2024-04-01T01:31:16.377831+0000","last_became_peered":"2024-04-01T01:31:16.377831+0000","last_unstale":"2024-04-01T01:33:20.838381+0000","last_undegraded":"2024-04-01T01:33:20.838381+0000","last_fullsized":"2024-04-01T01:33:20.838381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:30:06.487545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721996+0000","last_change":"2024-04-01T01:31:16.389708+0000","last_active":"2024-04-01T01:33:19.721996+0000","last_peered":"2024-04-01T01:33:19.721996+0000","last_clean":"2024-04-01T01:33:19.721996+0000","last_became_active":"2024-04-01T01:31:16.389409+0000","last_became_peered":"2024-04-01T01:31:16.389409+0000","last_unstale":"2024-04-01T01:33:19.721996+0000","last_undegraded":"2024-04-01T01:33:19.721996+0000","last_fullsized":"2024-04-01T01:33:19.721996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:57:49.215988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725238+0000","last_change":"2024-04-01T01:31:16.385996+0000","last_active":"2024-04-01T01:33:19.725238+0000","last_peered":"2024-04-01T01:33:19.725238+0000","last_clean":"2024-04-01T01:33:19.725238+0000","last_became_active":"2024-04-01T01:31:16.385648+0000","last_became_peered":"2024-04-01T01:31:16.385648+0000","last_unstale":"2024-04-01T01:33:19.725238+0000","last_undegraded":"2024-04-01T01:33:19.725238+0000","last_fullsized":"2024-04-01T01:33:19.725238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:14:56.595595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.723900+0000","last_change":"2024-04-01T01:32:19.693185+0000","last_active":"2024-04-01T01:33:19.723900+0000","last_peered":"2024-04-01T01:33:19.723900+0000","last_clean":"2024-04-01T01:33:19.723900+0000","last_became_active":"2024-04-01T01:32:19.692763+0000","last_became_peered":"2024-04-01T01:32:19.692763+0000","last_unstale":"2024-04-01T01:33:19.723900+0000","last_undegraded":"2024-04-01T01:33:19.723900+0000","last_fullsized":"2024-04-01T01:33:19.723900+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:37:54.712852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724816+0000","last_change":"2024-04-01T01:31:16.383939+0000","last_active":"2024-04-01T01:33:19.724816+0000","last_peered":"2024-04-01T01:33:19.724816+0000","last_clean":"2024-04-01T01:33:19.724816+0000","last_became_active":"2024-04-01T01:31:16.383626+0000","last_became_peered":"2024-04-01T01:31:16.383626+0000","last_unstale":"2024-04-01T01:33:19.724816+0000","last_undegraded":"2024-04-01T01:33:19.724816+0000","last_fullsized":"2024-04-01T01:33:19.724816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:36:05.764031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.628778+0000","last_change":"2024-04-01T01:31:16.384007+0000","last_active":"2024-04-01T01:33:20.628778+0000","last_peered":"2024-04-01T01:33:20.628778+0000","last_clean":"2024-04-01T01:33:20.628778+0000","last_became_active":"2024-04-01T01:31:16.383687+0000","last_became_peered":"2024-04-01T01:31:16.383687+0000","last_unstale":"2024-04-01T01:33:20.628778+0000","last_undegraded":"2024-04-01T01:33:20.628778+0000","last_fullsized":"2024-04-01T01:33:20.628778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:14:17.593402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.609104+0000","last_change":"2024-04-01T01:31:16.380357+0000","last_active":"2024-04-01T01:33:20.609104+0000","last_peered":"2024-04-01T01:33:20.609104+0000","last_clean":"2024-04-01T01:33:20.609104+0000","last_became_active":"2024-04-01T01:31:16.380046+0000","last_became_peered":"2024-04-01T01:31:16.380046+0000","last_unstale":"2024-04-01T01:33:20.609104+0000","last_undegraded":"2024-04-01T01:33:20.609104+0000","last_fullsized":"2024-04-01T01:33:20.609104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:51:53.887768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724418+0000","last_change":"2024-04-01T01:31:16.378551+0000","last_active":"2024-04-01T01:33:19.724418+0000","last_peered":"2024-04-01T01:33:19.724418+0000","last_clean":"2024-04-01T01:33:19.724418+0000","last_became_active":"2024-04-01T01:31:16.378234+0000","last_became_peered":"2024-04-01T01:31:16.378234+0000","last_unstale":"2024-04-01T01:33:19.724418+0000","last_undegraded":"2024-04-01T01:33:19.724418+0000","last_fullsized":"2024-04-01T01:33:19.724418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:37:45.998115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725392+0000","last_change":"2024-04-01T01:31:16.388755+0000","last_active":"2024-04-01T01:33:19.725392+0000","last_peered":"2024-04-01T01:33:19.725392+0000","last_clean":"2024-04-01T01:33:19.725392+0000","last_became_active":"2024-04-01T01:31:16.388434+0000","last_became_peered":"2024-04-01T01:31:16.388434+0000","last_unstale":"2024-04-01T01:33:19.725392+0000","last_undegraded":"2024-04-01T01:33:19.725392+0000","last_fullsized":"2024-04-01T01:33:19.725392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:03:07.168110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727361+0000","last_change":"2024-04-01T01:33:19.727029+0000","last_active":"2024-04-01T01:33:19.727361+0000","last_peered":"2024-04-01T01:33:19.727361+0000","last_clean":"2024-04-01T01:33:19.727361+0000","last_became_active":"2024-04-01T01:33:19.726714+0000","last_became_peered":"2024-04-01T01:33:19.726714+0000","last_unstale":"2024-04-01T01:33:19.727361+0000","last_undegraded":"2024-04-01T01:33:19.727361+0000","last_fullsized":"2024-04-01T01:33:19.727361+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:35:03.224593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719204+0000","last_change":"2024-04-01T01:31:16.366921+0000","last_active":"2024-04-01T01:33:19.719204+0000","last_peered":"2024-04-01T01:33:19.719204+0000","last_clean":"2024-04-01T01:33:19.719204+0000","last_became_active":"2024-04-01T01:31:16.366098+0000","last_became_peered":"2024-04-01T01:31:16.366098+0000","last_unstale":"2024-04-01T01:33:19.719204+0000","last_undegraded":"2024-04-01T01:33:19.719204+0000","last_fullsized":"2024-04-01T01:33:19.719204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:29.456263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838282+0000","last_change":"2024-04-01T01:31:16.379178+0000","last_active":"2024-04-01T01:33:20.838282+0000","last_peered":"2024-04-01T01:33:20.838282+0000","last_clean":"2024-04-01T01:33:20.838282+0000","last_became_active":"2024-04-01T01:31:16.378877+0000","last_became_peered":"2024-04-01T01:31:16.378877+0000","last_unstale":"2024-04-01T01:33:20.838282+0000","last_undegraded":"2024-04-01T01:33:20.838282+0000","last_fullsized":"2024-04-01T01:33:20.838282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:19:16.941504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838274+0000","last_change":"2024-04-01T01:31:16.385373+0000","last_active":"2024-04-01T01:33:20.838274+0000","last_peered":"2024-04-01T01:33:20.838274+0000","last_clean":"2024-04-01T01:33:20.838274+0000","last_became_active":"2024-04-01T01:31:16.385046+0000","last_became_peered":"2024-04-01T01:31:16.385046+0000","last_unstale":"2024-04-01T01:33:20.838274+0000","last_undegraded":"2024-04-01T01:33:20.838274+0000","last_fullsized":"2024-04-01T01:33:20.838274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:40:32.521220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608501+0000","last_change":"2024-04-01T01:31:16.386267+0000","last_active":"2024-04-01T01:33:20.608501+0000","last_peered":"2024-04-01T01:33:20.608501+0000","last_clean":"2024-04-01T01:33:20.608501+0000","last_became_active":"2024-04-01T01:31:16.385917+0000","last_became_peered":"2024-04-01T01:31:16.385917+0000","last_unstale":"2024-04-01T01:33:20.608501+0000","last_undegraded":"2024-04-01T01:33:20.608501+0000","last_fullsized":"2024-04-01T01:33:20.608501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:58.249253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724843+0000","last_change":"2024-04-01T01:31:16.385609+0000","last_active":"2024-04-01T01:33:19.724843+0000","last_peered":"2024-04-01T01:33:19.724843+0000","last_clean":"2024-04-01T01:33:19.724843+0000","last_became_active":"2024-04-01T01:31:16.385128+0000","last_became_peered":"2024-04-01T01:31:16.385128+0000","last_unstale":"2024-04-01T01:33:19.724843+0000","last_undegraded":"2024-04-01T01:33:19.724843+0000","last_fullsized":"2024-04-01T01:33:19.724843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:30:45.915587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722053+0000","last_change":"2024-04-01T01:31:16.381209+0000","last_active":"2024-04-01T01:33:19.722053+0000","last_peered":"2024-04-01T01:33:19.722053+0000","last_clean":"2024-04-01T01:33:19.722053+0000","last_became_active":"2024-04-01T01:31:16.380721+0000","last_became_peered":"2024-04-01T01:31:16.380721+0000","last_unstale":"2024-04-01T01:33:19.722053+0000","last_undegraded":"2024-04-01T01:33:19.722053+0000","last_fullsized":"2024-04-01T01:33:19.722053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:22.438224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727542+0000","last_change":"2024-04-01T01:31:16.385235+0000","last_active":"2024-04-01T01:33:19.727542+0000","last_peered":"2024-04-01T01:33:19.727542+0000","last_clean":"2024-04-01T01:33:19.727542+0000","last_became_active":"2024-04-01T01:31:16.384856+0000","last_became_peered":"2024-04-01T01:31:16.384856+0000","last_unstale":"2024-04-01T01:33:19.727542+0000","last_undegraded":"2024-04-01T01:33:19.727542+0000","last_fullsized":"2024-04-01T01:33:19.727542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:35:30.214977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727267+0000","last_change":"2024-04-01T01:31:20.573626+0000","last_active":"2024-04-01T01:33:19.727267+0000","last_peered":"2024-04-01T01:33:19.727267+0000","last_clean":"2024-04-01T01:33:19.727267+0000","last_became_active":"2024-04-01T01:31:20.573182+0000","last_became_peered":"2024-04-01T01:31:20.573182+0000","last_unstale":"2024-04-01T01:33:19.727267+0000","last_undegraded":"2024-04-01T01:33:19.727267+0000","last_fullsized":"2024-04-01T01:33:19.727267+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:33:19.162613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'14","reported_seq":169,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:38.711759+0000","last_change":"2024-04-01T01:31:15.373100+0000","last_active":"2024-04-01T01:33:38.711759+0000","last_peered":"2024-04-01T01:33:38.711759+0000","last_clean":"2024-04-01T01:33:38.711759+0000","last_became_active":"2024-04-01T01:31:15.372710+0000","last_became_peered":"2024-04-01T01:31:15.372710+0000","last_unstale":"2024-04-01T01:33:38.711759+0000","last_undegraded":"2024-04-01T01:33:38.711759+0000","last_fullsized":"2024-04-01T01:33:38.711759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:29:42.303549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629686+0000","last_change":"2024-04-01T01:31:16.383690+0000","last_active":"2024-04-01T01:33:20.629686+0000","last_peered":"2024-04-01T01:33:20.629686+0000","last_clean":"2024-04-01T01:33:20.629686+0000","last_became_active":"2024-04-01T01:31:16.383248+0000","last_became_peered":"2024-04-01T01:31:16.383248+0000","last_unstale":"2024-04-01T01:33:20.629686+0000","last_undegraded":"2024-04-01T01:33:20.629686+0000","last_fullsized":"2024-04-01T01:33:20.629686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:34:49.112880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721649+0000","last_change":"2024-04-01T01:32:19.691398+0000","last_active":"2024-04-01T01:33:19.721649+0000","last_peered":"2024-04-01T01:33:19.721649+0000","last_clean":"2024-04-01T01:33:19.721649+0000","last_became_active":"2024-04-01T01:32:19.690908+0000","last_became_peered":"2024-04-01T01:32:19.690908+0000","last_unstale":"2024-04-01T01:33:19.721649+0000","last_undegraded":"2024-04-01T01:33:19.721649+0000","last_fullsized":"2024-04-01T01:33:19.721649+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:58:54.712363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629846+0000","last_change":"2024-04-01T01:31:15.381742+0000","last_active":"2024-04-01T01:33:20.629846+0000","last_peered":"2024-04-01T01:33:20.629846+0000","last_clean":"2024-04-01T01:33:20.629846+0000","last_became_active":"2024-04-01T01:31:15.377543+0000","last_became_peered":"2024-04-01T01:31:15.377543+0000","last_unstale":"2024-04-01T01:33:20.629846+0000","last_undegraded":"2024-04-01T01:33:20.629846+0000","last_fullsized":"2024-04-01T01:33:20.629846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:51:42.911114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838510+0000","last_change":"2024-04-01T01:31:16.382001+0000","last_active":"2024-04-01T01:33:20.838510+0000","last_peered":"2024-04-01T01:33:20.838510+0000","last_clean":"2024-04-01T01:33:20.838510+0000","last_became_active":"2024-04-01T01:31:16.381694+0000","last_became_peered":"2024-04-01T01:31:16.381694+0000","last_unstale":"2024-04-01T01:33:20.838510+0000","last_undegraded":"2024-04-01T01:33:20.838510+0000","last_fullsized":"2024-04-01T01:33:20.838510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:55:07.366755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721790+0000","last_change":"2024-04-01T01:31:20.570080+0000","last_active":"2024-04-01T01:33:19.721790+0000","last_peered":"2024-04-01T01:33:19.721790+0000","last_clean":"2024-04-01T01:33:19.721790+0000","last_became_active":"2024-04-01T01:31:20.569712+0000","last_became_peered":"2024-04-01T01:31:20.569712+0000","last_unstale":"2024-04-01T01:33:19.721790+0000","last_undegraded":"2024-04-01T01:33:19.721790+0000","last_fullsized":"2024-04-01T01:33:19.721790+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:48:27.732831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725670+0000","last_change":"2024-04-01T01:32:19.703903+0000","last_active":"2024-04-01T01:33:19.725670+0000","last_peered":"2024-04-01T01:33:19.725670+0000","last_clean":"2024-04-01T01:33:19.725670+0000","last_became_active":"2024-04-01T01:32:19.691856+0000","last_became_peered":"2024-04-01T01:32:19.691856+0000","last_unstale":"2024-04-01T01:33:19.725670+0000","last_undegraded":"2024-04-01T01:33:19.725670+0000","last_fullsized":"2024-04-01T01:33:19.725670+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:38:41.284469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608732+0000","last_change":"2024-04-01T01:31:16.383020+0000","last_active":"2024-04-01T01:33:20.608732+0000","last_peered":"2024-04-01T01:33:20.608732+0000","last_clean":"2024-04-01T01:33:20.608732+0000","last_became_active":"2024-04-01T01:31:16.382640+0000","last_became_peered":"2024-04-01T01:31:16.382640+0000","last_unstale":"2024-04-01T01:33:20.608732+0000","last_undegraded":"2024-04-01T01:33:20.608732+0000","last_fullsized":"2024-04-01T01:33:20.608732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:11:51.211929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721753+0000","last_change":"2024-04-01T01:31:20.573030+0000","last_active":"2024-04-01T01:33:19.721753+0000","last_peered":"2024-04-01T01:33:19.721753+0000","last_clean":"2024-04-01T01:33:19.721753+0000","last_became_active":"2024-04-01T01:31:20.572613+0000","last_became_peered":"2024-04-01T01:31:20.572613+0000","last_unstale":"2024-04-01T01:33:19.721753+0000","last_undegraded":"2024-04-01T01:33:19.721753+0000","last_fullsized":"2024-04-01T01:33:19.721753+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:42:17.086942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725015+0000","last_change":"2024-04-01T01:31:15.373054+0000","last_active":"2024-04-01T01:33:19.725015+0000","last_peered":"2024-04-01T01:33:19.725015+0000","last_clean":"2024-04-01T01:33:19.725015+0000","last_became_active":"2024-04-01T01:31:15.372575+0000","last_became_peered":"2024-04-01T01:31:15.372575+0000","last_unstale":"2024-04-01T01:33:19.725015+0000","last_undegraded":"2024-04-01T01:33:19.725015+0000","last_fullsized":"2024-04-01T01:33:19.725015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:56:41.860423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722460+0000","last_change":"2024-04-01T01:31:16.376037+0000","last_active":"2024-04-01T01:33:19.722460+0000","last_peered":"2024-04-01T01:33:19.722460+0000","last_clean":"2024-04-01T01:33:19.722460+0000","last_became_active":"2024-04-01T01:31:16.375463+0000","last_became_peered":"2024-04-01T01:31:16.375463+0000","last_unstale":"2024-04-01T01:33:19.722460+0000","last_undegraded":"2024-04-01T01:33:19.722460+0000","last_fullsized":"2024-04-01T01:33:19.722460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:30:40.098033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727397+0000","last_change":"2024-04-01T01:31:20.574288+0000","last_active":"2024-04-01T01:33:19.727397+0000","last_peered":"2024-04-01T01:33:19.727397+0000","last_clean":"2024-04-01T01:33:19.727397+0000","last_became_active":"2024-04-01T01:31:20.573812+0000","last_became_peered":"2024-04-01T01:31:20.573812+0000","last_unstale":"2024-04-01T01:33:19.727397+0000","last_undegraded":"2024-04-01T01:33:19.727397+0000","last_fullsized":"2024-04-01T01:33:19.727397+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:43:46.126102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":177,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:35.766196+0000","last_change":"2024-04-01T01:32:19.706645+0000","last_active":"2024-04-01T01:33:35.766196+0000","last_peered":"2024-04-01T01:33:35.766196+0000","last_clean":"2024-04-01T01:33:35.766196+0000","last_became_active":"2024-04-01T01:32:19.690510+0000","last_became_peered":"2024-04-01T01:32:19.690510+0000","last_unstale":"2024-04-01T01:33:35.766196+0000","last_undegraded":"2024-04-01T01:33:35.766196+0000","last_fullsized":"2024-04-01T01:33:35.766196+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:20:23.103229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1631,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724703+0000","last_change":"2024-04-01T01:31:16.377735+0000","last_active":"2024-04-01T01:33:19.724703+0000","last_peered":"2024-04-01T01:33:19.724703+0000","last_clean":"2024-04-01T01:33:19.724703+0000","last_became_active":"2024-04-01T01:31:16.377342+0000","last_became_peered":"2024-04-01T01:31:16.377342+0000","last_unstale":"2024-04-01T01:33:19.724703+0000","last_undegraded":"2024-04-01T01:33:19.724703+0000","last_fullsized":"2024-04-01T01:33:19.724703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:47.539223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727261+0000","last_change":"2024-04-01T01:31:20.573370+0000","last_active":"2024-04-01T01:33:19.727261+0000","last_peered":"2024-04-01T01:33:19.727261+0000","last_clean":"2024-04-01T01:33:19.727261+0000","last_became_active":"2024-04-01T01:31:20.572919+0000","last_became_peered":"2024-04-01T01:31:20.572919+0000","last_unstale":"2024-04-01T01:33:19.727261+0000","last_undegraded":"2024-04-01T01:33:19.727261+0000","last_fullsized":"2024-04-01T01:33:19.727261+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:42.190918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.723867+0000","last_change":"2024-04-01T01:32:19.810468+0000","last_active":"2024-04-01T01:33:19.723867+0000","last_peered":"2024-04-01T01:33:19.723867+0000","last_clean":"2024-04-01T01:33:19.723867+0000","last_became_active":"2024-04-01T01:32:19.805517+0000","last_became_peered":"2024-04-01T01:32:19.805517+0000","last_unstale":"2024-04-01T01:33:19.723867+0000","last_undegraded":"2024-04-01T01:33:19.723867+0000","last_fullsized":"2024-04-01T01:33:19.723867+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:54:35.355761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722977+0000","last_change":"2024-04-01T01:31:16.384130+0000","last_active":"2024-04-01T01:33:19.722977+0000","last_peered":"2024-04-01T01:33:19.722977+0000","last_clean":"2024-04-01T01:33:19.722977+0000","last_became_active":"2024-04-01T01:31:16.383400+0000","last_became_peered":"2024-04-01T01:31:16.383400+0000","last_unstale":"2024-04-01T01:33:19.722977+0000","last_undegraded":"2024-04-01T01:33:19.722977+0000","last_fullsized":"2024-04-01T01:33:19.722977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:22:10.860325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724729+0000","last_change":"2024-04-01T01:31:20.571052+0000","last_active":"2024-04-01T01:33:19.724729+0000","last_peered":"2024-04-01T01:33:19.724729+0000","last_clean":"2024-04-01T01:33:19.724729+0000","last_became_active":"2024-04-01T01:31:20.570617+0000","last_became_peered":"2024-04-01T01:31:20.570617+0000","last_unstale":"2024-04-01T01:33:19.724729+0000","last_undegraded":"2024-04-01T01:33:19.724729+0000","last_fullsized":"2024-04-01T01:33:19.724729+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:53:51.195056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719095+0000","last_change":"2024-04-01T01:31:15.369820+0000","last_active":"2024-04-01T01:33:19.719095+0000","last_peered":"2024-04-01T01:33:19.719095+0000","last_clean":"2024-04-01T01:33:19.719095+0000","last_became_active":"2024-04-01T01:31:15.369378+0000","last_became_peered":"2024-04-01T01:31:15.369378+0000","last_unstale":"2024-04-01T01:33:19.719095+0000","last_undegraded":"2024-04-01T01:33:19.719095+0000","last_fullsized":"2024-04-01T01:33:19.719095+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:37:34.351220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.628952+0000","last_change":"2024-04-01T01:31:16.383352+0000","last_active":"2024-04-01T01:33:20.628952+0000","last_peered":"2024-04-01T01:33:20.628952+0000","last_clean":"2024-04-01T01:33:20.628952+0000","last_became_active":"2024-04-01T01:31:16.382898+0000","last_became_peered":"2024-04-01T01:31:16.382898+0000","last_unstale":"2024-04-01T01:33:20.628952+0000","last_undegraded":"2024-04-01T01:33:20.628952+0000","last_fullsized":"2024-04-01T01:33:20.628952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:27.854459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.837895+0000","last_change":"2024-04-01T01:31:20.576578+0000","last_active":"2024-04-01T01:33:20.837895+0000","last_peered":"2024-04-01T01:33:20.837895+0000","last_clean":"2024-04-01T01:33:20.837895+0000","last_became_active":"2024-04-01T01:31:20.575996+0000","last_became_peered":"2024-04-01T01:31:20.575996+0000","last_unstale":"2024-04-01T01:33:20.837895+0000","last_undegraded":"2024-04-01T01:33:20.837895+0000","last_fullsized":"2024-04-01T01:33:20.837895+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:52:08.808357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722157+0000","last_change":"2024-04-01T01:31:15.377246+0000","last_active":"2024-04-01T01:33:19.722157+0000","last_peered":"2024-04-01T01:33:19.722157+0000","last_clean":"2024-04-01T01:33:19.722157+0000","last_became_active":"2024-04-01T01:31:15.376957+0000","last_became_peered":"2024-04-01T01:31:15.376957+0000","last_unstale":"2024-04-01T01:33:19.722157+0000","last_undegraded":"2024-04-01T01:33:19.722157+0000","last_fullsized":"2024-04-01T01:33:19.722157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:59:34.313984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721661+0000","last_change":"2024-04-01T01:31:16.379031+0000","last_active":"2024-04-01T01:33:19.721661+0000","last_peered":"2024-04-01T01:33:19.721661+0000","last_clean":"2024-04-01T01:33:19.721661+0000","last_became_active":"2024-04-01T01:31:16.378593+0000","last_became_peered":"2024-04-01T01:31:16.378593+0000","last_unstale":"2024-04-01T01:33:19.721661+0000","last_undegraded":"2024-04-01T01:33:19.721661+0000","last_fullsized":"2024-04-01T01:33:19.721661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:31.816103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.628494+0000","last_change":"2024-04-01T01:31:20.578660+0000","last_active":"2024-04-01T01:33:20.628494+0000","last_peered":"2024-04-01T01:33:20.628494+0000","last_clean":"2024-04-01T01:33:20.628494+0000","last_became_active":"2024-04-01T01:31:20.578082+0000","last_became_peered":"2024-04-01T01:31:20.578082+0000","last_unstale":"2024-04-01T01:33:20.628494+0000","last_undegraded":"2024-04-01T01:33:20.628494+0000","last_fullsized":"2024-04-01T01:33:20.628494+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:02:56.570816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'7","reported_seq":277,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:35.719505+0000","last_change":"2024-04-01T01:31:15.373428+0000","last_active":"2024-04-01T01:33:35.719505+0000","last_peered":"2024-04-01T01:33:35.719505+0000","last_clean":"2024-04-01T01:33:35.719505+0000","last_became_active":"2024-04-01T01:31:15.372965+0000","last_became_peered":"2024-04-01T01:31:15.372965+0000","last_unstale":"2024-04-01T01:33:35.719505+0000","last_undegraded":"2024-04-01T01:33:35.719505+0000","last_fullsized":"2024-04-01T01:33:35.719505+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:55:42.211055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":239,"num_read_kb":239,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.609010+0000","last_change":"2024-04-01T01:31:16.382961+0000","last_active":"2024-04-01T01:33:20.609010+0000","last_peered":"2024-04-01T01:33:20.609010+0000","last_clean":"2024-04-01T01:33:20.609010+0000","last_became_active":"2024-04-01T01:31:16.382527+0000","last_became_peered":"2024-04-01T01:31:16.382527+0000","last_unstale":"2024-04-01T01:33:20.609010+0000","last_undegraded":"2024-04-01T01:33:20.609010+0000","last_fullsized":"2024-04-01T01:33:20.609010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:46:20.250064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721640+0000","last_change":"2024-04-01T01:31:20.571842+0000","last_active":"2024-04-01T01:33:19.721640+0000","last_peered":"2024-04-01T01:33:19.721640+0000","last_clean":"2024-04-01T01:33:19.721640+0000","last_became_active":"2024-04-01T01:31:20.571394+0000","last_became_peered":"2024-04-01T01:31:20.571394+0000","last_unstale":"2024-04-01T01:33:19.721640+0000","last_undegraded":"2024-04-01T01:33:19.721640+0000","last_fullsized":"2024-04-01T01:33:19.721640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:51:36.512277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725497+0000","last_change":"2024-04-01T01:31:15.379737+0000","last_active":"2024-04-01T01:33:19.725497+0000","last_peered":"2024-04-01T01:33:19.725497+0000","last_clean":"2024-04-01T01:33:19.725497+0000","last_became_active":"2024-04-01T01:31:15.379158+0000","last_became_peered":"2024-04-01T01:31:15.379158+0000","last_unstale":"2024-04-01T01:33:19.725497+0000","last_undegraded":"2024-04-01T01:33:19.725497+0000","last_fullsized":"2024-04-01T01:33:19.725497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:12:44.474772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838564+0000","last_change":"2024-04-01T01:31:16.386059+0000","last_active":"2024-04-01T01:33:20.838564+0000","last_peered":"2024-04-01T01:33:20.838564+0000","last_clean":"2024-04-01T01:33:20.838564+0000","last_became_active":"2024-04-01T01:31:16.384551+0000","last_became_peered":"2024-04-01T01:31:16.384551+0000","last_unstale":"2024-04-01T01:33:20.838564+0000","last_undegraded":"2024-04-01T01:33:20.838564+0000","last_fullsized":"2024-04-01T01:33:20.838564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:52:16.594485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721972+0000","last_change":"2024-04-01T01:31:20.566937+0000","last_active":"2024-04-01T01:33:19.721972+0000","last_peered":"2024-04-01T01:33:19.721972+0000","last_clean":"2024-04-01T01:33:19.721972+0000","last_became_active":"2024-04-01T01:31:20.566447+0000","last_became_peered":"2024-04-01T01:31:20.566447+0000","last_unstale":"2024-04-01T01:33:19.721972+0000","last_undegraded":"2024-04-01T01:33:19.721972+0000","last_fullsized":"2024-04-01T01:33:19.721972+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:17:37.845166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722425+0000","last_change":"2024-04-01T01:31:15.375669+0000","last_active":"2024-04-01T01:33:19.722425+0000","last_peered":"2024-04-01T01:33:19.722425+0000","last_clean":"2024-04-01T01:33:19.722425+0000","last_became_active":"2024-04-01T01:31:15.375353+0000","last_became_peered":"2024-04-01T01:31:15.375353+0000","last_unstale":"2024-04-01T01:33:19.722425+0000","last_undegraded":"2024-04-01T01:33:19.722425+0000","last_fullsized":"2024-04-01T01:33:19.722425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:47:43.096898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608910+0000","last_change":"2024-04-01T01:31:16.388399+0000","last_active":"2024-04-01T01:33:20.608910+0000","last_peered":"2024-04-01T01:33:20.608910+0000","last_clean":"2024-04-01T01:33:20.608910+0000","last_became_active":"2024-04-01T01:31:16.388059+0000","last_became_peered":"2024-04-01T01:31:16.388059+0000","last_unstale":"2024-04-01T01:33:20.608910+0000","last_undegraded":"2024-04-01T01:33:20.608910+0000","last_fullsized":"2024-04-01T01:33:20.608910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:59:50.907802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.723472+0000","last_change":"2024-04-01T01:32:19.692098+0000","last_active":"2024-04-01T01:33:19.723472+0000","last_peered":"2024-04-01T01:33:19.723472+0000","last_clean":"2024-04-01T01:33:19.723472+0000","last_became_active":"2024-04-01T01:32:19.691673+0000","last_became_peered":"2024-04-01T01:32:19.691673+0000","last_unstale":"2024-04-01T01:33:19.723472+0000","last_undegraded":"2024-04-01T01:33:19.723472+0000","last_fullsized":"2024-04-01T01:33:19.723472+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:14:17.300517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"93'39","reported_seq":323,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:35.766454+0000","last_change":"2024-04-01T01:31:15.376421+0000","last_active":"2024-04-01T01:33:35.766454+0000","last_peered":"2024-04-01T01:33:35.766454+0000","last_clean":"2024-04-01T01:33:35.766454+0000","last_became_active":"2024-04-01T01:31:15.375985+0000","last_became_peered":"2024-04-01T01:31:15.375985+0000","last_unstale":"2024-04-01T01:33:35.766454+0000","last_undegraded":"2024-04-01T01:33:35.766454+0000","last_fullsized":"2024-04-01T01:33:35.766454+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:37:29.121436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64082,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":252,"num_read_kb":189,"num_write":41,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725852+0000","last_change":"2024-04-01T01:31:16.380641+0000","last_active":"2024-04-01T01:33:19.725852+0000","last_peered":"2024-04-01T01:33:19.725852+0000","last_clean":"2024-04-01T01:33:19.725852+0000","last_became_active":"2024-04-01T01:31:16.380300+0000","last_became_peered":"2024-04-01T01:31:16.380300+0000","last_unstale":"2024-04-01T01:33:19.725852+0000","last_undegraded":"2024-04-01T01:33:19.725852+0000","last_fullsized":"2024-04-01T01:33:19.725852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:26:08.240694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722617+0000","last_change":"2024-04-01T01:31:20.576094+0000","last_active":"2024-04-01T01:33:19.722617+0000","last_peered":"2024-04-01T01:33:19.722617+0000","last_clean":"2024-04-01T01:33:19.722617+0000","last_became_active":"2024-04-01T01:31:20.575674+0000","last_became_peered":"2024-04-01T01:31:20.575674+0000","last_unstale":"2024-04-01T01:33:19.722617+0000","last_undegraded":"2024-04-01T01:33:19.722617+0000","last_fullsized":"2024-04-01T01:33:19.722617+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:42:19.352398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727136+0000","last_change":"2024-04-01T01:31:15.375136+0000","last_active":"2024-04-01T01:33:19.727136+0000","last_peered":"2024-04-01T01:33:19.727136+0000","last_clean":"2024-04-01T01:33:19.727136+0000","last_became_active":"2024-04-01T01:31:15.374813+0000","last_became_peered":"2024-04-01T01:31:15.374813+0000","last_unstale":"2024-04-01T01:33:19.727136+0000","last_undegraded":"2024-04-01T01:33:19.727136+0000","last_fullsized":"2024-04-01T01:33:19.727136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:19:05.490103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719129+0000","last_change":"2024-04-01T01:31:16.380207+0000","last_active":"2024-04-01T01:33:19.719129+0000","last_peered":"2024-04-01T01:33:19.719129+0000","last_clean":"2024-04-01T01:33:19.719129+0000","last_became_active":"2024-04-01T01:31:16.379891+0000","last_became_peered":"2024-04-01T01:31:16.379891+0000","last_unstale":"2024-04-01T01:33:19.719129+0000","last_undegraded":"2024-04-01T01:33:19.719129+0000","last_fullsized":"2024-04-01T01:33:19.719129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:12:36.417162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724683+0000","last_change":"2024-04-01T01:31:20.569978+0000","last_active":"2024-04-01T01:33:19.724683+0000","last_peered":"2024-04-01T01:33:19.724683+0000","last_clean":"2024-04-01T01:33:19.724683+0000","last_became_active":"2024-04-01T01:31:20.568298+0000","last_became_peered":"2024-04-01T01:31:20.568298+0000","last_unstale":"2024-04-01T01:33:19.724683+0000","last_undegraded":"2024-04-01T01:33:19.724683+0000","last_fullsized":"2024-04-01T01:33:19.724683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:42:16.439284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722815+0000","last_change":"2024-04-01T01:31:15.380143+0000","last_active":"2024-04-01T01:33:19.722815+0000","last_peered":"2024-04-01T01:33:19.722815+0000","last_clean":"2024-04-01T01:33:19.722815+0000","last_became_active":"2024-04-01T01:31:15.379835+0000","last_became_peered":"2024-04-01T01:31:15.379835+0000","last_unstale":"2024-04-01T01:33:19.722815+0000","last_undegraded":"2024-04-01T01:33:19.722815+0000","last_fullsized":"2024-04-01T01:33:19.722815+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:28:34.213626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725248+0000","last_change":"2024-04-01T01:31:16.380290+0000","last_active":"2024-04-01T01:33:19.725248+0000","last_peered":"2024-04-01T01:33:19.725248+0000","last_clean":"2024-04-01T01:33:19.725248+0000","last_became_active":"2024-04-01T01:31:16.379983+0000","last_became_peered":"2024-04-01T01:31:16.379983+0000","last_unstale":"2024-04-01T01:33:19.725248+0000","last_undegraded":"2024-04-01T01:33:19.725248+0000","last_fullsized":"2024-04-01T01:33:19.725248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:50.135454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721926+0000","last_change":"2024-04-01T01:31:20.573500+0000","last_active":"2024-04-01T01:33:19.721926+0000","last_peered":"2024-04-01T01:33:19.721926+0000","last_clean":"2024-04-01T01:33:19.721926+0000","last_became_active":"2024-04-01T01:31:20.573101+0000","last_became_peered":"2024-04-01T01:31:20.573101+0000","last_unstale":"2024-04-01T01:33:19.721926+0000","last_undegraded":"2024-04-01T01:33:19.721926+0000","last_fullsized":"2024-04-01T01:33:19.721926+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:11:27.881703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'8","reported_seq":163,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:36.723420+0000","last_change":"2024-04-01T01:31:15.377265+0000","last_active":"2024-04-01T01:33:36.723420+0000","last_peered":"2024-04-01T01:33:36.723420+0000","last_clean":"2024-04-01T01:33:36.723420+0000","last_became_active":"2024-04-01T01:31:15.376889+0000","last_became_peered":"2024-04-01T01:31:15.376889+0000","last_unstale":"2024-04-01T01:33:36.723420+0000","last_undegraded":"2024-04-01T01:33:36.723420+0000","last_fullsized":"2024-04-01T01:33:36.723420+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:43:31.582866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1597,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724953+0000","last_change":"2024-04-01T01:31:16.384982+0000","last_active":"2024-04-01T01:33:19.724953+0000","last_peered":"2024-04-01T01:33:19.724953+0000","last_clean":"2024-04-01T01:33:19.724953+0000","last_became_active":"2024-04-01T01:31:16.384663+0000","last_became_peered":"2024-04-01T01:31:16.384663+0000","last_unstale":"2024-04-01T01:33:19.724953+0000","last_undegraded":"2024-04-01T01:33:19.724953+0000","last_fullsized":"2024-04-01T01:33:19.724953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:14:57.351768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719430+0000","last_change":"2024-04-01T01:31:20.578761+0000","last_active":"2024-04-01T01:33:19.719430+0000","last_peered":"2024-04-01T01:33:19.719430+0000","last_clean":"2024-04-01T01:33:19.719430+0000","last_became_active":"2024-04-01T01:31:20.578204+0000","last_became_peered":"2024-04-01T01:31:20.578204+0000","last_unstale":"2024-04-01T01:33:19.719430+0000","last_undegraded":"2024-04-01T01:33:19.719430+0000","last_fullsized":"2024-04-01T01:33:19.719430+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:19:33.584872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724403+0000","last_change":"2024-04-01T01:31:15.380773+0000","last_active":"2024-04-01T01:33:19.724403+0000","last_peered":"2024-04-01T01:33:19.724403+0000","last_clean":"2024-04-01T01:33:19.724403+0000","last_became_active":"2024-04-01T01:31:15.380472+0000","last_became_peered":"2024-04-01T01:31:15.380472+0000","last_unstale":"2024-04-01T01:33:19.724403+0000","last_undegraded":"2024-04-01T01:33:19.724403+0000","last_fullsized":"2024-04-01T01:33:19.724403+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:34:28.294500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722296+0000","last_change":"2024-04-01T01:31:16.374658+0000","last_active":"2024-04-01T01:33:19.722296+0000","last_peered":"2024-04-01T01:33:19.722296+0000","last_clean":"2024-04-01T01:33:19.722296+0000","last_became_active":"2024-04-01T01:31:16.373596+0000","last_became_peered":"2024-04-01T01:31:16.373596+0000","last_unstale":"2024-04-01T01:33:19.722296+0000","last_undegraded":"2024-04-01T01:33:19.722296+0000","last_fullsized":"2024-04-01T01:33:19.722296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:28:38.477719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722239+0000","last_change":"2024-04-01T01:31:20.577449+0000","last_active":"2024-04-01T01:33:19.722239+0000","last_peered":"2024-04-01T01:33:19.722239+0000","last_clean":"2024-04-01T01:33:19.722239+0000","last_became_active":"2024-04-01T01:31:20.577026+0000","last_became_peered":"2024-04-01T01:31:20.577026+0000","last_unstale":"2024-04-01T01:33:19.722239+0000","last_undegraded":"2024-04-01T01:33:19.722239+0000","last_fullsized":"2024-04-01T01:33:19.722239+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:44:48.080304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:33.356435+0000","last_change":"2024-04-01T01:31:15.373885+0000","last_active":"2024-04-01T01:33:33.356435+0000","last_peered":"2024-04-01T01:33:33.356435+0000","last_clean":"2024-04-01T01:33:33.356435+0000","last_became_active":"2024-04-01T01:31:15.373578+0000","last_became_peered":"2024-04-01T01:31:15.373578+0000","last_unstale":"2024-04-01T01:33:33.356435+0000","last_undegraded":"2024-04-01T01:33:33.356435+0000","last_fullsized":"2024-04-01T01:33:33.356435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:52.641515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.730680+0000","last_change":"2024-04-01T01:33:19.730088+0000","last_active":"2024-04-01T01:33:19.730680+0000","last_peered":"2024-04-01T01:33:19.730680+0000","last_clean":"2024-04-01T01:33:19.730680+0000","last_became_active":"2024-04-01T01:33:19.729627+0000","last_became_peered":"2024-04-01T01:33:19.729627+0000","last_unstale":"2024-04-01T01:33:19.730680+0000","last_undegraded":"2024-04-01T01:33:19.730680+0000","last_fullsized":"2024-04-01T01:33:19.730680+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:47:37.839420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725463+0000","last_change":"2024-04-01T01:31:20.569835+0000","last_active":"2024-04-01T01:33:19.725463+0000","last_peered":"2024-04-01T01:33:19.725463+0000","last_clean":"2024-04-01T01:33:19.725463+0000","last_became_active":"2024-04-01T01:31:20.569195+0000","last_became_peered":"2024-04-01T01:31:20.569195+0000","last_unstale":"2024-04-01T01:33:19.725463+0000","last_undegraded":"2024-04-01T01:33:19.725463+0000","last_fullsized":"2024-04-01T01:33:19.725463+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:10:18.278575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721958+0000","last_change":"2024-04-01T01:31:15.377679+0000","last_active":"2024-04-01T01:33:19.721958+0000","last_peered":"2024-04-01T01:33:19.721958+0000","last_clean":"2024-04-01T01:33:19.721958+0000","last_became_active":"2024-04-01T01:31:15.377351+0000","last_became_peered":"2024-04-01T01:31:15.377351+0000","last_unstale":"2024-04-01T01:33:19.721958+0000","last_undegraded":"2024-04-01T01:33:19.721958+0000","last_fullsized":"2024-04-01T01:33:19.721958+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:22:41.216563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724294+0000","last_change":"2024-04-01T01:31:16.380997+0000","last_active":"2024-04-01T01:33:19.724294+0000","last_peered":"2024-04-01T01:33:19.724294+0000","last_clean":"2024-04-01T01:33:19.724294+0000","last_became_active":"2024-04-01T01:31:16.380692+0000","last_became_peered":"2024-04-01T01:31:16.380692+0000","last_unstale":"2024-04-01T01:33:19.724294+0000","last_undegraded":"2024-04-01T01:33:19.724294+0000","last_fullsized":"2024-04-01T01:33:19.724294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:09:51.220849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719249+0000","last_change":"2024-04-01T01:31:20.573191+0000","last_active":"2024-04-01T01:33:19.719249+0000","last_peered":"2024-04-01T01:33:19.719249+0000","last_clean":"2024-04-01T01:33:19.719249+0000","last_became_active":"2024-04-01T01:31:20.572669+0000","last_became_peered":"2024-04-01T01:31:20.572669+0000","last_unstale":"2024-04-01T01:33:19.719249+0000","last_undegraded":"2024-04-01T01:33:19.719249+0000","last_fullsized":"2024-04-01T01:33:19.719249+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:01:56.491271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"93'7","reported_seq":164,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:38.712613+0000","last_change":"2024-04-01T01:31:15.379566+0000","last_active":"2024-04-01T01:33:38.712613+0000","last_peered":"2024-04-01T01:33:38.712613+0000","last_clean":"2024-04-01T01:33:38.712613+0000","last_became_active":"2024-04-01T01:31:15.379213+0000","last_became_peered":"2024-04-01T01:31:15.379213+0000","last_unstale":"2024-04-01T01:33:38.712613+0000","last_undegraded":"2024-04-01T01:33:38.712613+0000","last_fullsized":"2024-04-01T01:33:38.712613+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:34:17.974422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1456,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724071+0000","last_change":"2024-04-01T01:31:16.376218+0000","last_active":"2024-04-01T01:33:19.724071+0000","last_peered":"2024-04-01T01:33:19.724071+0000","last_clean":"2024-04-01T01:33:19.724071+0000","last_became_active":"2024-04-01T01:31:16.375887+0000","last_became_peered":"2024-04-01T01:31:16.375887+0000","last_unstale":"2024-04-01T01:33:19.724071+0000","last_undegraded":"2024-04-01T01:33:19.724071+0000","last_fullsized":"2024-04-01T01:33:19.724071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:27:03.723827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719394+0000","last_change":"2024-04-01T01:31:20.576461+0000","last_active":"2024-04-01T01:33:19.719394+0000","last_peered":"2024-04-01T01:33:19.719394+0000","last_clean":"2024-04-01T01:33:19.719394+0000","last_became_active":"2024-04-01T01:31:20.575985+0000","last_became_peered":"2024-04-01T01:31:20.575985+0000","last_unstale":"2024-04-01T01:33:19.719394+0000","last_undegraded":"2024-04-01T01:33:19.719394+0000","last_fullsized":"2024-04-01T01:33:19.719394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:19:02.714455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722259+0000","last_change":"2024-04-01T01:31:16.376456+0000","last_active":"2024-04-01T01:33:19.722259+0000","last_peered":"2024-04-01T01:33:19.722259+0000","last_clean":"2024-04-01T01:33:19.722259+0000","last_became_active":"2024-04-01T01:31:16.375445+0000","last_became_peered":"2024-04-01T01:31:16.375445+0000","last_unstale":"2024-04-01T01:33:19.722259+0000","last_undegraded":"2024-04-01T01:33:19.722259+0000","last_fullsized":"2024-04-01T01:33:19.722259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:07:11.802799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'7","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:38.725028+0000","last_change":"2024-04-01T01:31:15.382702+0000","last_active":"2024-04-01T01:33:38.725028+0000","last_peered":"2024-04-01T01:33:38.725028+0000","last_clean":"2024-04-01T01:33:38.725028+0000","last_became_active":"2024-04-01T01:31:15.382377+0000","last_became_peered":"2024-04-01T01:31:15.382377+0000","last_unstale":"2024-04-01T01:33:38.725028+0000","last_undegraded":"2024-04-01T01:33:38.725028+0000","last_fullsized":"2024-04-01T01:33:38.725028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:45.080992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725315+0000","last_change":"2024-04-01T01:32:19.704607+0000","last_active":"2024-04-01T01:33:19.725315+0000","last_peered":"2024-04-01T01:33:19.725315+0000","last_clean":"2024-04-01T01:33:19.725315+0000","last_became_active":"2024-04-01T01:32:19.694650+0000","last_became_peered":"2024-04-01T01:32:19.694650+0000","last_unstale":"2024-04-01T01:33:19.725315+0000","last_undegraded":"2024-04-01T01:33:19.725315+0000","last_fullsized":"2024-04-01T01:33:19.725315+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:48:55.439187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629394+0000","last_change":"2024-04-01T01:31:16.387981+0000","last_active":"2024-04-01T01:33:20.629394+0000","last_peered":"2024-04-01T01:33:20.629394+0000","last_clean":"2024-04-01T01:33:20.629394+0000","last_became_active":"2024-04-01T01:31:16.387594+0000","last_became_peered":"2024-04-01T01:31:16.387594+0000","last_unstale":"2024-04-01T01:33:20.629394+0000","last_undegraded":"2024-04-01T01:33:20.629394+0000","last_fullsized":"2024-04-01T01:33:20.629394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:17.148752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721940+0000","last_change":"2024-04-01T01:31:15.375212+0000","last_active":"2024-04-01T01:33:19.721940+0000","last_peered":"2024-04-01T01:33:19.721940+0000","last_clean":"2024-04-01T01:33:19.721940+0000","last_became_active":"2024-04-01T01:31:15.374882+0000","last_became_peered":"2024-04-01T01:31:15.374882+0000","last_unstale":"2024-04-01T01:33:19.721940+0000","last_undegraded":"2024-04-01T01:33:19.721940+0000","last_fullsized":"2024-04-01T01:33:19.721940+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:56:23.412025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724742+0000","last_change":"2024-04-01T01:30:03.538590+0000","last_active":"2024-04-01T01:33:19.724742+0000","last_peered":"2024-04-01T01:33:19.724742+0000","last_clean":"2024-04-01T01:33:19.724742+0000","last_became_active":"2024-04-01T01:30:03.538098+0000","last_became_peered":"2024-04-01T01:30:03.538098+0000","last_unstale":"2024-04-01T01:33:19.724742+0000","last_undegraded":"2024-04-01T01:33:19.724742+0000","last_fullsized":"2024-04-01T01:33:19.724742+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:26:43.445198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:26:43.445198+0000","last_clean_scrub_stamp":"2024-04-01T01:26:43.445198+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:22.207795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722391+0000","last_change":"2024-04-01T01:31:20.573260+0000","last_active":"2024-04-01T01:33:19.722391+0000","last_peered":"2024-04-01T01:33:19.722391+0000","last_clean":"2024-04-01T01:33:19.722391+0000","last_became_active":"2024-04-01T01:31:20.572846+0000","last_became_peered":"2024-04-01T01:31:20.572846+0000","last_unstale":"2024-04-01T01:33:19.722391+0000","last_undegraded":"2024-04-01T01:33:19.722391+0000","last_fullsized":"2024-04-01T01:33:19.722391+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:24.986070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.718978+0000","last_change":"2024-04-01T01:31:16.384782+0000","last_active":"2024-04-01T01:33:19.718978+0000","last_peered":"2024-04-01T01:33:19.718978+0000","last_clean":"2024-04-01T01:33:19.718978+0000","last_became_active":"2024-04-01T01:31:16.384420+0000","last_became_peered":"2024-04-01T01:31:16.384420+0000","last_unstale":"2024-04-01T01:33:19.718978+0000","last_undegraded":"2024-04-01T01:33:19.718978+0000","last_fullsized":"2024-04-01T01:33:19.718978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:00:20.902022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724778+0000","last_change":"2024-04-01T01:31:15.381853+0000","last_active":"2024-04-01T01:33:19.724778+0000","last_peered":"2024-04-01T01:33:19.724778+0000","last_clean":"2024-04-01T01:33:19.724778+0000","last_became_active":"2024-04-01T01:31:15.381496+0000","last_became_peered":"2024-04-01T01:31:15.381496+0000","last_unstale":"2024-04-01T01:33:19.724778+0000","last_undegraded":"2024-04-01T01:33:19.724778+0000","last_fullsized":"2024-04-01T01:33:19.724778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:07:51.440351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724645+0000","last_change":"2024-04-01T01:31:20.568705+0000","last_active":"2024-04-01T01:33:19.724645+0000","last_peered":"2024-04-01T01:33:19.724645+0000","last_clean":"2024-04-01T01:33:19.724645+0000","last_became_active":"2024-04-01T01:31:20.568252+0000","last_became_peered":"2024-04-01T01:31:20.568252+0000","last_unstale":"2024-04-01T01:33:19.724645+0000","last_undegraded":"2024-04-01T01:33:19.724645+0000","last_fullsized":"2024-04-01T01:33:19.724645+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:22:28.509510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724763+0000","last_change":"2024-04-01T01:32:19.699781+0000","last_active":"2024-04-01T01:33:19.724763+0000","last_peered":"2024-04-01T01:33:19.724763+0000","last_clean":"2024-04-01T01:33:19.724763+0000","last_became_active":"2024-04-01T01:32:19.689593+0000","last_became_peered":"2024-04-01T01:32:19.689593+0000","last_unstale":"2024-04-01T01:33:19.724763+0000","last_undegraded":"2024-04-01T01:33:19.724763+0000","last_fullsized":"2024-04-01T01:33:19.724763+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:44:21.922097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719384+0000","last_change":"2024-04-01T01:31:16.377002+0000","last_active":"2024-04-01T01:33:19.719384+0000","last_peered":"2024-04-01T01:33:19.719384+0000","last_clean":"2024-04-01T01:33:19.719384+0000","last_became_active":"2024-04-01T01:31:16.376679+0000","last_became_peered":"2024-04-01T01:31:16.376679+0000","last_unstale":"2024-04-01T01:33:19.719384+0000","last_undegraded":"2024-04-01T01:33:19.719384+0000","last_fullsized":"2024-04-01T01:33:19.719384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:50:24.126192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.628550+0000","last_change":"2024-04-01T01:31:20.577757+0000","last_active":"2024-04-01T01:33:20.628550+0000","last_peered":"2024-04-01T01:33:20.628550+0000","last_clean":"2024-04-01T01:33:20.628550+0000","last_became_active":"2024-04-01T01:31:20.577378+0000","last_became_peered":"2024-04-01T01:31:20.577378+0000","last_unstale":"2024-04-01T01:33:20.628550+0000","last_undegraded":"2024-04-01T01:33:20.628550+0000","last_fullsized":"2024-04-01T01:33:20.628550+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:48:04.511229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'16","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:36.559784+0000","last_change":"2024-04-01T01:31:15.383201+0000","last_active":"2024-04-01T01:33:36.559784+0000","last_peered":"2024-04-01T01:33:36.559784+0000","last_clean":"2024-04-01T01:33:36.559784+0000","last_became_active":"2024-04-01T01:31:15.382320+0000","last_became_peered":"2024-04-01T01:31:15.382320+0000","last_unstale":"2024-04-01T01:33:36.559784+0000","last_undegraded":"2024-04-01T01:33:36.559784+0000","last_fullsized":"2024-04-01T01:33:36.559784+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:15:47.138024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629854+0000","last_change":"2024-04-01T01:31:16.386936+0000","last_active":"2024-04-01T01:33:20.629854+0000","last_peered":"2024-04-01T01:33:20.629854+0000","last_clean":"2024-04-01T01:33:20.629854+0000","last_became_active":"2024-04-01T01:31:16.384448+0000","last_became_peered":"2024-04-01T01:31:16.384448+0000","last_unstale":"2024-04-01T01:33:20.629854+0000","last_undegraded":"2024-04-01T01:33:20.629854+0000","last_fullsized":"2024-04-01T01:33:20.629854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:16:42.243250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.726698+0000","last_change":"2024-04-01T01:31:20.565570+0000","last_active":"2024-04-01T01:33:19.726698+0000","last_peered":"2024-04-01T01:33:19.726698+0000","last_clean":"2024-04-01T01:33:19.726698+0000","last_became_active":"2024-04-01T01:31:20.565141+0000","last_became_peered":"2024-04-01T01:31:20.565141+0000","last_unstale":"2024-04-01T01:33:19.726698+0000","last_undegraded":"2024-04-01T01:33:19.726698+0000","last_fullsized":"2024-04-01T01:33:19.726698+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:47:12.844970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722422+0000","last_change":"2024-04-01T01:31:15.381343+0000","last_active":"2024-04-01T01:33:19.722422+0000","last_peered":"2024-04-01T01:33:19.722422+0000","last_clean":"2024-04-01T01:33:19.722422+0000","last_became_active":"2024-04-01T01:31:15.380994+0000","last_became_peered":"2024-04-01T01:31:15.380994+0000","last_unstale":"2024-04-01T01:33:19.722422+0000","last_undegraded":"2024-04-01T01:33:19.722422+0000","last_fullsized":"2024-04-01T01:33:19.722422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:19:10.710285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838567+0000","last_change":"2024-04-01T01:31:16.383951+0000","last_active":"2024-04-01T01:33:20.838567+0000","last_peered":"2024-04-01T01:33:20.838567+0000","last_clean":"2024-04-01T01:33:20.838567+0000","last_became_active":"2024-04-01T01:31:16.383657+0000","last_became_peered":"2024-04-01T01:31:16.383657+0000","last_unstale":"2024-04-01T01:33:20.838567+0000","last_undegraded":"2024-04-01T01:33:20.838567+0000","last_fullsized":"2024-04-01T01:33:20.838567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:55:45.216479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722156+0000","last_change":"2024-04-01T01:31:20.569416+0000","last_active":"2024-04-01T01:33:19.722156+0000","last_peered":"2024-04-01T01:33:19.722156+0000","last_clean":"2024-04-01T01:33:19.722156+0000","last_became_active":"2024-04-01T01:31:20.569007+0000","last_became_peered":"2024-04-01T01:31:20.569007+0000","last_unstale":"2024-04-01T01:33:19.722156+0000","last_undegraded":"2024-04-01T01:33:19.722156+0000","last_fullsized":"2024-04-01T01:33:19.722156+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:25:36.770919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608610+0000","last_change":"2024-04-01T01:31:15.373444+0000","last_active":"2024-04-01T01:33:20.608610+0000","last_peered":"2024-04-01T01:33:20.608610+0000","last_clean":"2024-04-01T01:33:20.608610+0000","last_became_active":"2024-04-01T01:31:15.373094+0000","last_became_peered":"2024-04-01T01:31:15.373094+0000","last_unstale":"2024-04-01T01:33:20.608610+0000","last_undegraded":"2024-04-01T01:33:20.608610+0000","last_fullsized":"2024-04-01T01:33:20.608610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:26:27.443832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724496+0000","last_change":"2024-04-01T01:31:16.381441+0000","last_active":"2024-04-01T01:33:19.724496+0000","last_peered":"2024-04-01T01:33:19.724496+0000","last_clean":"2024-04-01T01:33:19.724496+0000","last_became_active":"2024-04-01T01:31:16.381092+0000","last_became_peered":"2024-04-01T01:31:16.381092+0000","last_unstale":"2024-04-01T01:33:19.724496+0000","last_undegraded":"2024-04-01T01:33:19.724496+0000","last_fullsized":"2024-04-01T01:33:19.724496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:47:00.229090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722053+0000","last_change":"2024-04-01T01:31:20.569672+0000","last_active":"2024-04-01T01:33:19.722053+0000","last_peered":"2024-04-01T01:33:19.722053+0000","last_clean":"2024-04-01T01:33:19.722053+0000","last_became_active":"2024-04-01T01:31:20.569276+0000","last_became_peered":"2024-04-01T01:31:20.569276+0000","last_unstale":"2024-04-01T01:33:19.722053+0000","last_undegraded":"2024-04-01T01:33:19.722053+0000","last_fullsized":"2024-04-01T01:33:19.722053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:39:03.331428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629526+0000","last_change":"2024-04-01T01:31:15.376094+0000","last_active":"2024-04-01T01:33:20.629526+0000","last_peered":"2024-04-01T01:33:20.629526+0000","last_clean":"2024-04-01T01:33:20.629526+0000","last_became_active":"2024-04-01T01:31:15.375169+0000","last_became_peered":"2024-04-01T01:31:15.375169+0000","last_unstale":"2024-04-01T01:33:20.629526+0000","last_undegraded":"2024-04-01T01:33:20.629526+0000","last_fullsized":"2024-04-01T01:33:20.629526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:18:20.710644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724497+0000","last_change":"2024-04-01T01:31:16.378514+0000","last_active":"2024-04-01T01:33:19.724497+0000","last_peered":"2024-04-01T01:33:19.724497+0000","last_clean":"2024-04-01T01:33:19.724497+0000","last_became_active":"2024-04-01T01:31:16.377990+0000","last_became_peered":"2024-04-01T01:31:16.377990+0000","last_unstale":"2024-04-01T01:33:19.724497+0000","last_undegraded":"2024-04-01T01:33:19.724497+0000","last_fullsized":"2024-04-01T01:33:19.724497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:27:29.829775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721831+0000","last_change":"2024-04-01T01:31:20.567892+0000","last_active":"2024-04-01T01:33:19.721831+0000","last_peered":"2024-04-01T01:33:19.721831+0000","last_clean":"2024-04-01T01:33:19.721831+0000","last_became_active":"2024-04-01T01:31:20.567525+0000","last_became_peered":"2024-04-01T01:31:20.567525+0000","last_unstale":"2024-04-01T01:33:19.721831+0000","last_undegraded":"2024-04-01T01:33:19.721831+0000","last_fullsized":"2024-04-01T01:33:19.721831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:57.828691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'5","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:35.564105+0000","last_change":"2024-04-01T01:32:19.703414+0000","last_active":"2024-04-01T01:33:35.564105+0000","last_peered":"2024-04-01T01:33:35.564105+0000","last_clean":"2024-04-01T01:33:35.564105+0000","last_became_active":"2024-04-01T01:32:19.693581+0000","last_became_peered":"2024-04-01T01:32:19.693581+0000","last_unstale":"2024-04-01T01:33:35.564105+0000","last_undegraded":"2024-04-01T01:33:35.564105+0000","last_fullsized":"2024-04-01T01:33:35.564105+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:53:17.897154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629533+0000","last_change":"2024-04-01T01:31:16.387570+0000","last_active":"2024-04-01T01:33:20.629533+0000","last_peered":"2024-04-01T01:33:20.629533+0000","last_clean":"2024-04-01T01:33:20.629533+0000","last_became_active":"2024-04-01T01:31:16.387200+0000","last_became_peered":"2024-04-01T01:31:16.387200+0000","last_unstale":"2024-04-01T01:33:20.629533+0000","last_undegraded":"2024-04-01T01:33:20.629533+0000","last_fullsized":"2024-04-01T01:33:20.629533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:52:27.813722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722104+0000","last_change":"2024-04-01T01:31:20.569668+0000","last_active":"2024-04-01T01:33:19.722104+0000","last_peered":"2024-04-01T01:33:19.722104+0000","last_clean":"2024-04-01T01:33:19.722104+0000","last_became_active":"2024-04-01T01:31:20.569148+0000","last_became_peered":"2024-04-01T01:31:20.569148+0000","last_unstale":"2024-04-01T01:33:19.722104+0000","last_undegraded":"2024-04-01T01:33:19.722104+0000","last_fullsized":"2024-04-01T01:33:19.722104+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:20:56.048052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724566+0000","last_change":"2024-04-01T01:31:15.379323+0000","last_active":"2024-04-01T01:33:19.724566+0000","last_peered":"2024-04-01T01:33:19.724566+0000","last_clean":"2024-04-01T01:33:19.724566+0000","last_became_active":"2024-04-01T01:31:15.379023+0000","last_became_peered":"2024-04-01T01:31:15.379023+0000","last_unstale":"2024-04-01T01:33:19.724566+0000","last_undegraded":"2024-04-01T01:33:19.724566+0000","last_fullsized":"2024-04-01T01:33:19.724566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:05:37.709040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725164+0000","last_change":"2024-04-01T01:31:16.387250+0000","last_active":"2024-04-01T01:33:19.725164+0000","last_peered":"2024-04-01T01:33:19.725164+0000","last_clean":"2024-04-01T01:33:19.725164+0000","last_became_active":"2024-04-01T01:31:16.386940+0000","last_became_peered":"2024-04-01T01:31:16.386940+0000","last_unstale":"2024-04-01T01:33:19.725164+0000","last_undegraded":"2024-04-01T01:33:19.725164+0000","last_fullsized":"2024-04-01T01:33:19.725164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:19:34.886738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608300+0000","last_change":"2024-04-01T01:31:20.573228+0000","last_active":"2024-04-01T01:33:20.608300+0000","last_peered":"2024-04-01T01:33:20.608300+0000","last_clean":"2024-04-01T01:33:20.608300+0000","last_became_active":"2024-04-01T01:31:20.572681+0000","last_became_peered":"2024-04-01T01:31:20.572681+0000","last_unstale":"2024-04-01T01:33:20.608300+0000","last_undegraded":"2024-04-01T01:33:20.608300+0000","last_fullsized":"2024-04-01T01:33:20.608300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:58:17.812966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'7","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:37.723133+0000","last_change":"2024-04-01T01:31:15.381477+0000","last_active":"2024-04-01T01:33:37.723133+0000","last_peered":"2024-04-01T01:33:37.723133+0000","last_clean":"2024-04-01T01:33:37.723133+0000","last_became_active":"2024-04-01T01:31:15.381166+0000","last_became_peered":"2024-04-01T01:31:15.381166+0000","last_unstale":"2024-04-01T01:33:37.723133+0000","last_undegraded":"2024-04-01T01:33:37.723133+0000","last_fullsized":"2024-04-01T01:33:37.723133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:19.029896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2038,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725492+0000","last_change":"2024-04-01T01:31:16.386234+0000","last_active":"2024-04-01T01:33:19.725492+0000","last_peered":"2024-04-01T01:33:19.725492+0000","last_clean":"2024-04-01T01:33:19.725492+0000","last_became_active":"2024-04-01T01:31:16.385876+0000","last_became_peered":"2024-04-01T01:31:16.385876+0000","last_unstale":"2024-04-01T01:33:19.725492+0000","last_undegraded":"2024-04-01T01:33:19.725492+0000","last_fullsized":"2024-04-01T01:33:19.725492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:34.187210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838323+0000","last_change":"2024-04-01T01:32:19.805473+0000","last_active":"2024-04-01T01:33:20.838323+0000","last_peered":"2024-04-01T01:33:20.838323+0000","last_clean":"2024-04-01T01:33:20.838323+0000","last_became_active":"2024-04-01T01:32:19.805151+0000","last_became_peered":"2024-04-01T01:32:19.805151+0000","last_unstale":"2024-04-01T01:33:20.838323+0000","last_undegraded":"2024-04-01T01:33:20.838323+0000","last_fullsized":"2024-04-01T01:33:20.838323+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:47:54.539407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722570+0000","last_change":"2024-04-01T01:31:15.378503+0000","last_active":"2024-04-01T01:33:19.722570+0000","last_peered":"2024-04-01T01:33:19.722570+0000","last_clean":"2024-04-01T01:33:19.722570+0000","last_became_active":"2024-04-01T01:31:15.378152+0000","last_became_peered":"2024-04-01T01:31:15.378152+0000","last_unstale":"2024-04-01T01:33:19.722570+0000","last_undegraded":"2024-04-01T01:33:19.722570+0000","last_fullsized":"2024-04-01T01:33:19.722570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:15:42.118691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.728781+0000","last_change":"2024-04-01T01:33:19.728322+0000","last_active":"2024-04-01T01:33:19.728781+0000","last_peered":"2024-04-01T01:33:19.728781+0000","last_clean":"2024-04-01T01:33:19.728781+0000","last_became_active":"2024-04-01T01:33:19.727846+0000","last_became_peered":"2024-04-01T01:33:19.727846+0000","last_unstale":"2024-04-01T01:33:19.728781+0000","last_undegraded":"2024-04-01T01:33:19.728781+0000","last_fullsized":"2024-04-01T01:33:19.728781+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:54.899382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722019+0000","last_change":"2024-04-01T01:31:20.569461+0000","last_active":"2024-04-01T01:33:19.722019+0000","last_peered":"2024-04-01T01:33:19.722019+0000","last_clean":"2024-04-01T01:33:19.722019+0000","last_became_active":"2024-04-01T01:31:20.569011+0000","last_became_peered":"2024-04-01T01:31:20.569011+0000","last_unstale":"2024-04-01T01:33:19.722019+0000","last_undegraded":"2024-04-01T01:33:19.722019+0000","last_fullsized":"2024-04-01T01:33:19.722019+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:43:43.657065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719451+0000","last_change":"2024-04-01T01:31:15.384181+0000","last_active":"2024-04-01T01:33:19.719451+0000","last_peered":"2024-04-01T01:33:19.719451+0000","last_clean":"2024-04-01T01:33:19.719451+0000","last_became_active":"2024-04-01T01:31:15.383886+0000","last_became_peered":"2024-04-01T01:31:15.383886+0000","last_unstale":"2024-04-01T01:33:19.719451+0000","last_undegraded":"2024-04-01T01:33:19.719451+0000","last_fullsized":"2024-04-01T01:33:19.719451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:43:28.414423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725363+0000","last_change":"2024-04-01T01:31:16.381877+0000","last_active":"2024-04-01T01:33:19.725363+0000","last_peered":"2024-04-01T01:33:19.725363+0000","last_clean":"2024-04-01T01:33:19.725363+0000","last_became_active":"2024-04-01T01:31:16.381558+0000","last_became_peered":"2024-04-01T01:31:16.381558+0000","last_unstale":"2024-04-01T01:33:19.725363+0000","last_undegraded":"2024-04-01T01:33:19.725363+0000","last_fullsized":"2024-04-01T01:33:19.725363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:24.120910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.730654+0000","last_change":"2024-04-01T01:33:19.730079+0000","last_active":"2024-04-01T01:33:19.730654+0000","last_peered":"2024-04-01T01:33:19.730654+0000","last_clean":"2024-04-01T01:33:19.730654+0000","last_became_active":"2024-04-01T01:33:19.729631+0000","last_became_peered":"2024-04-01T01:33:19.729631+0000","last_unstale":"2024-04-01T01:33:19.730654+0000","last_undegraded":"2024-04-01T01:33:19.730654+0000","last_fullsized":"2024-04-01T01:33:19.730654+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:30.299413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727472+0000","last_change":"2024-04-01T01:31:16.384872+0000","last_active":"2024-04-01T01:33:19.727472+0000","last_peered":"2024-04-01T01:33:19.727472+0000","last_clean":"2024-04-01T01:33:19.727472+0000","last_became_active":"2024-04-01T01:31:16.384406+0000","last_became_peered":"2024-04-01T01:31:16.384406+0000","last_unstale":"2024-04-01T01:33:19.727472+0000","last_undegraded":"2024-04-01T01:33:19.727472+0000","last_fullsized":"2024-04-01T01:33:19.727472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:13:36.567357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727117+0000","last_change":"2024-04-01T01:31:16.384561+0000","last_active":"2024-04-01T01:33:19.727117+0000","last_peered":"2024-04-01T01:33:19.727117+0000","last_clean":"2024-04-01T01:33:19.727117+0000","last_became_active":"2024-04-01T01:31:16.384221+0000","last_became_peered":"2024-04-01T01:31:16.384221+0000","last_unstale":"2024-04-01T01:33:19.727117+0000","last_undegraded":"2024-04-01T01:33:19.727117+0000","last_fullsized":"2024-04-01T01:33:19.727117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:17:56.490000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838506+0000","last_change":"2024-04-01T01:31:16.385464+0000","last_active":"2024-04-01T01:33:20.838506+0000","last_peered":"2024-04-01T01:33:20.838506+0000","last_clean":"2024-04-01T01:33:20.838506+0000","last_became_active":"2024-04-01T01:31:16.385159+0000","last_became_peered":"2024-04-01T01:31:16.385159+0000","last_unstale":"2024-04-01T01:33:20.838506+0000","last_undegraded":"2024-04-01T01:33:20.838506+0000","last_fullsized":"2024-04-01T01:33:20.838506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:33:53.852957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719377+0000","last_change":"2024-04-01T01:31:16.385688+0000","last_active":"2024-04-01T01:33:19.719377+0000","last_peered":"2024-04-01T01:33:19.719377+0000","last_clean":"2024-04-01T01:33:19.719377+0000","last_became_active":"2024-04-01T01:31:16.385381+0000","last_became_peered":"2024-04-01T01:31:16.385381+0000","last_unstale":"2024-04-01T01:33:19.719377+0000","last_undegraded":"2024-04-01T01:33:19.719377+0000","last_fullsized":"2024-04-01T01:33:19.719377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:39:36.810766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725031+0000","last_change":"2024-04-01T01:31:16.379976+0000","last_active":"2024-04-01T01:33:19.725031+0000","last_peered":"2024-04-01T01:33:19.725031+0000","last_clean":"2024-04-01T01:33:19.725031+0000","last_became_active":"2024-04-01T01:31:16.379553+0000","last_became_peered":"2024-04-01T01:31:16.379553+0000","last_unstale":"2024-04-01T01:33:19.725031+0000","last_undegraded":"2024-04-01T01:33:19.725031+0000","last_fullsized":"2024-04-01T01:33:19.725031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:48:23.555723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727822+0000","last_change":"2024-04-01T01:31:16.383995+0000","last_active":"2024-04-01T01:33:19.727822+0000","last_peered":"2024-04-01T01:33:19.727822+0000","last_clean":"2024-04-01T01:33:19.727822+0000","last_became_active":"2024-04-01T01:31:16.383673+0000","last_became_peered":"2024-04-01T01:31:16.383673+0000","last_unstale":"2024-04-01T01:33:19.727822+0000","last_undegraded":"2024-04-01T01:33:19.727822+0000","last_fullsized":"2024-04-01T01:33:19.727822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:03:16.192438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629879+0000","last_change":"2024-04-01T01:31:16.389969+0000","last_active":"2024-04-01T01:33:20.629879+0000","last_peered":"2024-04-01T01:33:20.629879+0000","last_clean":"2024-04-01T01:33:20.629879+0000","last_became_active":"2024-04-01T01:31:16.389640+0000","last_became_peered":"2024-04-01T01:31:16.389640+0000","last_unstale":"2024-04-01T01:33:20.629879+0000","last_undegraded":"2024-04-01T01:33:20.629879+0000","last_fullsized":"2024-04-01T01:33:20.629879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:35:15.226960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722223+0000","last_change":"2024-04-01T01:31:16.382962+0000","last_active":"2024-04-01T01:33:19.722223+0000","last_peered":"2024-04-01T01:33:19.722223+0000","last_clean":"2024-04-01T01:33:19.722223+0000","last_became_active":"2024-04-01T01:31:16.382653+0000","last_became_peered":"2024-04-01T01:31:16.382653+0000","last_unstale":"2024-04-01T01:33:19.722223+0000","last_undegraded":"2024-04-01T01:33:19.722223+0000","last_fullsized":"2024-04-01T01:33:19.722223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:53.054884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629538+0000","last_change":"2024-04-01T01:31:16.382997+0000","last_active":"2024-04-01T01:33:20.629538+0000","last_peered":"2024-04-01T01:33:20.629538+0000","last_clean":"2024-04-01T01:33:20.629538+0000","last_became_active":"2024-04-01T01:31:16.380180+0000","last_became_peered":"2024-04-01T01:31:16.380180+0000","last_unstale":"2024-04-01T01:33:20.629538+0000","last_undegraded":"2024-04-01T01:33:20.629538+0000","last_fullsized":"2024-04-01T01:33:20.629538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:29:51.990395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608760+0000","last_change":"2024-04-01T01:31:16.382948+0000","last_active":"2024-04-01T01:33:20.608760+0000","last_peered":"2024-04-01T01:33:20.608760+0000","last_clean":"2024-04-01T01:33:20.608760+0000","last_became_active":"2024-04-01T01:31:16.382134+0000","last_became_peered":"2024-04-01T01:31:16.382134+0000","last_unstale":"2024-04-01T01:33:20.608760+0000","last_undegraded":"2024-04-01T01:33:20.608760+0000","last_fullsized":"2024-04-01T01:33:20.608760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:24:08.515063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719262+0000","last_change":"2024-04-01T01:31:16.378841+0000","last_active":"2024-04-01T01:33:19.719262+0000","last_peered":"2024-04-01T01:33:19.719262+0000","last_clean":"2024-04-01T01:33:19.719262+0000","last_became_active":"2024-04-01T01:31:16.378501+0000","last_became_peered":"2024-04-01T01:31:16.378501+0000","last_unstale":"2024-04-01T01:33:19.719262+0000","last_undegraded":"2024-04-01T01:33:19.719262+0000","last_fullsized":"2024-04-01T01:33:19.719262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:50:20.881852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724900+0000","last_change":"2024-04-01T01:31:16.380979+0000","last_active":"2024-04-01T01:33:19.724900+0000","last_peered":"2024-04-01T01:33:19.724900+0000","last_clean":"2024-04-01T01:33:19.724900+0000","last_became_active":"2024-04-01T01:31:16.380625+0000","last_became_peered":"2024-04-01T01:31:16.380625+0000","last_unstale":"2024-04-01T01:33:19.724900+0000","last_undegraded":"2024-04-01T01:33:19.724900+0000","last_fullsized":"2024-04-01T01:33:19.724900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:09:03.246239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74258,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1247,"num_read_kb":710,"num_write":274,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1125,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":585728,"data_stored":231402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1073152,"data_stored":520978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37056,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708096,"statfs":{"total":95995035648,"available":95957090304,"internally_reserved":0,"allocated":1761280,"data_stored":1172296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":524415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070521,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":524681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36980,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708172,"statfs":{"total":95995035648,"available":95957168128,"internally_reserved":0,"allocated":1683456,"data_stored":1119689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":516691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1138688,"data_stored":586508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":517756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1146880,"data_stored":581010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":515531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":522537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28664,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716488,"statfs":{"total":95995035648,"available":95965683712,"internally_reserved":0,"allocated":1687552,"data_stored":1109298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":71005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-01T01:33:42.164 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-01T01:33:42.164 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-01T01:33:42.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.302+0000 7ff77f5e9640 1 -- 172.21.15.70:0/1915491801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff778101860 msgr2=0x7ff778101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:42.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.302+0000 7ff77f5e9640 1 --2- 172.21.15.70:0/1915491801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff778101860 0x7ff778101ce0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff768009900 tx=0x7ff76802bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:42.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.302+0000 7ff77f5e9640 1 -- 172.21.15.70:0/1915491801 shutdown_connections 2024-04-01T01:33:42.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.302+0000 7ff77f5e9640 1 --2- 172.21.15.70:0/1915491801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff778102220 0x7ff778108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.302+0000 7ff77f5e9640 1 --2- 172.21.15.70:0/1915491801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff778101860 0x7ff778101ce0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.302+0000 7ff77f5e9640 1 --2- 172.21.15.70:0/1915491801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff778100670 0x7ff778100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.302+0000 7ff77f5e9640 1 -- 172.21.15.70:0/1915491801 >> 172.21.15.70:0/1915491801 conn(0x7ff7780fbbe0 msgr2=0x7ff7780fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:42.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.302+0000 7ff77f5e9640 1 -- 172.21.15.70:0/1915491801 shutdown_connections 2024-04-01T01:33:42.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.302+0000 7ff77f5e9640 1 -- 172.21.15.70:0/1915491801 wait complete. 2024-04-01T01:33:42.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77f5e9640 1 Processor -- start 2024-04-01T01:33:42.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77f5e9640 1 -- start start 2024-04-01T01:33:42.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77f5e9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff778100670 0x7ff7781979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:42.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77f5e9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff778101860 0x7ff778197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:42.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77f5e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff778102220 0x7ff77819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:42.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77f5e9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff77810c1b0 con 0x7ff778102220 2024-04-01T01:33:42.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77f5e9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff77810c030 con 0x7ff778101860 2024-04-01T01:33:42.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77f5e9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff77810c330 con 0x7ff778100670 2024-04-01T01:33:42.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77db86640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff778102220 0x7ff77819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:42.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77db86640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff778102220 0x7ff77819efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55460/0 (socket says 172.21.15.70:55460) 2024-04-01T01:33:42.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77db86640 1 -- 172.21.15.70:0/242498240 learned_addr learned my addr 172.21.15.70:0/242498240 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:42.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77cb84640 1 --2- 172.21.15.70:0/242498240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff778101860 0x7ff778197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:42.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77d385640 1 --2- 172.21.15.70:0/242498240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff778100670 0x7ff7781979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:42.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77db86640 1 -- 172.21.15.70:0/242498240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff778100670 msgr2=0x7ff7781979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:42.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77db86640 1 --2- 172.21.15.70:0/242498240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff778100670 0x7ff7781979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77db86640 1 -- 172.21.15.70:0/242498240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff778101860 msgr2=0x7ff778197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:42.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77db86640 1 --2- 172.21.15.70:0/242498240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff778101860 0x7ff778197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77db86640 1 -- 172.21.15.70:0/242498240 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77819f610 con 0x7ff778102220 2024-04-01T01:33:42.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff77db86640 1 --2- 172.21.15.70:0/242498240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff778102220 0x7ff77819efa0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff77000efd0 tx=0x7ff77000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:42.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff76e7fc640 1 -- 172.21.15.70:0/242498240 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff770003b70 con 0x7ff778102220 2024-04-01T01:33:42.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.303+0000 7ff76e7fc640 1 -- 172.21.15.70:0/242498240 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff770003d10 con 0x7ff778102220 2024-04-01T01:33:42.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.304+0000 7ff76e7fc640 1 -- 172.21.15.70:0/242498240 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff770010980 con 0x7ff778102220 2024-04-01T01:33:42.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.304+0000 7ff77f5e9640 1 -- 172.21.15.70:0/242498240 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff77819f900 con 0x7ff778102220 2024-04-01T01:33:42.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.304+0000 7ff77f5e9640 1 -- 172.21.15.70:0/242498240 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff77819fe20 con 0x7ff778102220 2024-04-01T01:33:42.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.304+0000 7ff77f5e9640 1 -- 172.21.15.70:0/242498240 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7781a00b0 con 0x7ff778102220 2024-04-01T01:33:42.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.305+0000 7ff76e7fc640 1 -- 172.21.15.70:0/242498240 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7ff770004020 con 0x7ff778102220 2024-04-01T01:33:42.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.306+0000 7ff76e7fc640 1 --2- 172.21.15.70:0/242498240 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7640a25b0 0x7ff7640a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:42.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.306+0000 7ff76e7fc640 1 -- 172.21.15.70:0/242498240 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff7700cf020 con 0x7ff778102220 2024-04-01T01:33:42.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.306+0000 7ff77d385640 1 --2- 172.21.15.70:0/242498240 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7640a25b0 0x7ff7640a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:42.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.306+0000 7ff77d385640 1 --2- 172.21.15.70:0/242498240 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7640a25b0 0x7ff7640a4a70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff7781016c0 tx=0x7ff774005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:42.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.311+0000 7ff76e7fc640 1 -- 172.21.15.70:0/242498240 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff77009bcb0 con 0x7ff778102220 2024-04-01T01:33:42.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.486+0000 7ff77f5e9640 1 -- 172.21.15.70:0/242498240 --> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff7781a0280 con 0x7ff7640a25b0 2024-04-01T01:33:42.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.496+0000 7ff76e7fc640 1 -- 172.21.15.70:0/242498240 <== mgr.14150 v2:172.21.15.70:6800/2070166741 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340534 (secure 0 0 0) 0x7ff7781a0280 con 0x7ff7640a25b0 2024-04-01T01:33:42.500 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:42.501 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-01T01:33:42.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.499+0000 7ff77f5e9640 1 -- 172.21.15.70:0/242498240 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7640a25b0 msgr2=0x7ff7640a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:42.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.499+0000 7ff77f5e9640 1 --2- 172.21.15.70:0/242498240 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7640a25b0 0x7ff7640a4a70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff7781016c0 tx=0x7ff774005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:42.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.499+0000 7ff77f5e9640 1 -- 172.21.15.70:0/242498240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff778102220 msgr2=0x7ff77819efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:42.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.499+0000 7ff77f5e9640 1 --2- 172.21.15.70:0/242498240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff778102220 0x7ff77819efa0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff77000efd0 tx=0x7ff77000c6a0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.499+0000 7ff77f5e9640 1 -- 172.21.15.70:0/242498240 shutdown_connections 2024-04-01T01:33:42.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.499+0000 7ff77f5e9640 1 --2- 172.21.15.70:0/242498240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff778102220 0x7ff77819efa0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.499+0000 7ff77f5e9640 1 --2- 172.21.15.70:0/242498240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff778101860 0x7ff778197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.499+0000 7ff77f5e9640 1 --2- 172.21.15.70:0/242498240 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7640a25b0 0x7ff7640a4a70 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.499+0000 7ff77f5e9640 1 --2- 172.21.15.70:0/242498240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff778100670 0x7ff7781979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.499+0000 7ff77f5e9640 1 -- 172.21.15.70:0/242498240 >> 172.21.15.70:0/242498240 conn(0x7ff7780fbbe0 msgr2=0x7ff7780688b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:42.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.499+0000 7ff77f5e9640 1 -- 172.21.15.70:0/242498240 shutdown_connections 2024-04-01T01:33:42.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.499+0000 7ff77f5e9640 1 -- 172.21.15.70:0/242498240 wait complete. 2024-04-01T01:33:42.510 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-04-01T01:33:40.604455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664860,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1373,"num_read_kb":819,"num_write":457,"num_write_kb":3105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1181978,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":875,"ondisk_log_size":875,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":359628,"kb_used_data":14628,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124582196,"statfs":{"total":1151940427776,"available":1151572168704,"internally_reserved":0,"allocated":14979072,"data_stored":8211390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3167,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":121,"num_read_kb":66,"num_write":24,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003260"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725372+0000","last_change":"2024-04-01T01:31:16.385517+0000","last_active":"2024-04-01T01:33:19.725372+0000","last_peered":"2024-04-01T01:33:19.725372+0000","last_clean":"2024-04-01T01:33:19.725372+0000","last_became_active":"2024-04-01T01:31:16.384682+0000","last_became_peered":"2024-04-01T01:31:16.384682+0000","last_unstale":"2024-04-01T01:33:19.725372+0000","last_undegraded":"2024-04-01T01:33:19.725372+0000","last_fullsized":"2024-04-01T01:33:19.725372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:35:47.614041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838551+0000","last_change":"2024-04-01T01:31:16.376119+0000","last_active":"2024-04-01T01:33:20.838551+0000","last_peered":"2024-04-01T01:33:20.838551+0000","last_clean":"2024-04-01T01:33:20.838551+0000","last_became_active":"2024-04-01T01:31:16.375801+0000","last_became_peered":"2024-04-01T01:31:16.375801+0000","last_unstale":"2024-04-01T01:33:20.838551+0000","last_undegraded":"2024-04-01T01:33:20.838551+0000","last_fullsized":"2024-04-01T01:33:20.838551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:34.120487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838381+0000","last_change":"2024-04-01T01:31:16.378158+0000","last_active":"2024-04-01T01:33:20.838381+0000","last_peered":"2024-04-01T01:33:20.838381+0000","last_clean":"2024-04-01T01:33:20.838381+0000","last_became_active":"2024-04-01T01:31:16.377831+0000","last_became_peered":"2024-04-01T01:31:16.377831+0000","last_unstale":"2024-04-01T01:33:20.838381+0000","last_undegraded":"2024-04-01T01:33:20.838381+0000","last_fullsized":"2024-04-01T01:33:20.838381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:30:06.487545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721996+0000","last_change":"2024-04-01T01:31:16.389708+0000","last_active":"2024-04-01T01:33:19.721996+0000","last_peered":"2024-04-01T01:33:19.721996+0000","last_clean":"2024-04-01T01:33:19.721996+0000","last_became_active":"2024-04-01T01:31:16.389409+0000","last_became_peered":"2024-04-01T01:31:16.389409+0000","last_unstale":"2024-04-01T01:33:19.721996+0000","last_undegraded":"2024-04-01T01:33:19.721996+0000","last_fullsized":"2024-04-01T01:33:19.721996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:57:49.215988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725238+0000","last_change":"2024-04-01T01:31:16.385996+0000","last_active":"2024-04-01T01:33:19.725238+0000","last_peered":"2024-04-01T01:33:19.725238+0000","last_clean":"2024-04-01T01:33:19.725238+0000","last_became_active":"2024-04-01T01:31:16.385648+0000","last_became_peered":"2024-04-01T01:31:16.385648+0000","last_unstale":"2024-04-01T01:33:19.725238+0000","last_undegraded":"2024-04-01T01:33:19.725238+0000","last_fullsized":"2024-04-01T01:33:19.725238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:14:56.595595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.723900+0000","last_change":"2024-04-01T01:32:19.693185+0000","last_active":"2024-04-01T01:33:19.723900+0000","last_peered":"2024-04-01T01:33:19.723900+0000","last_clean":"2024-04-01T01:33:19.723900+0000","last_became_active":"2024-04-01T01:32:19.692763+0000","last_became_peered":"2024-04-01T01:32:19.692763+0000","last_unstale":"2024-04-01T01:33:19.723900+0000","last_undegraded":"2024-04-01T01:33:19.723900+0000","last_fullsized":"2024-04-01T01:33:19.723900+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:37:54.712852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724816+0000","last_change":"2024-04-01T01:31:16.383939+0000","last_active":"2024-04-01T01:33:19.724816+0000","last_peered":"2024-04-01T01:33:19.724816+0000","last_clean":"2024-04-01T01:33:19.724816+0000","last_became_active":"2024-04-01T01:31:16.383626+0000","last_became_peered":"2024-04-01T01:31:16.383626+0000","last_unstale":"2024-04-01T01:33:19.724816+0000","last_undegraded":"2024-04-01T01:33:19.724816+0000","last_fullsized":"2024-04-01T01:33:19.724816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:36:05.764031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.628778+0000","last_change":"2024-04-01T01:31:16.384007+0000","last_active":"2024-04-01T01:33:20.628778+0000","last_peered":"2024-04-01T01:33:20.628778+0000","last_clean":"2024-04-01T01:33:20.628778+0000","last_became_active":"2024-04-01T01:31:16.383687+0000","last_became_peered":"2024-04-01T01:31:16.383687+0000","last_unstale":"2024-04-01T01:33:20.628778+0000","last_undegraded":"2024-04-01T01:33:20.628778+0000","last_fullsized":"2024-04-01T01:33:20.628778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:14:17.593402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.609104+0000","last_change":"2024-04-01T01:31:16.380357+0000","last_active":"2024-04-01T01:33:20.609104+0000","last_peered":"2024-04-01T01:33:20.609104+0000","last_clean":"2024-04-01T01:33:20.609104+0000","last_became_active":"2024-04-01T01:31:16.380046+0000","last_became_peered":"2024-04-01T01:31:16.380046+0000","last_unstale":"2024-04-01T01:33:20.609104+0000","last_undegraded":"2024-04-01T01:33:20.609104+0000","last_fullsized":"2024-04-01T01:33:20.609104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:51:53.887768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724418+0000","last_change":"2024-04-01T01:31:16.378551+0000","last_active":"2024-04-01T01:33:19.724418+0000","last_peered":"2024-04-01T01:33:19.724418+0000","last_clean":"2024-04-01T01:33:19.724418+0000","last_became_active":"2024-04-01T01:31:16.378234+0000","last_became_peered":"2024-04-01T01:31:16.378234+0000","last_unstale":"2024-04-01T01:33:19.724418+0000","last_undegraded":"2024-04-01T01:33:19.724418+0000","last_fullsized":"2024-04-01T01:33:19.724418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:37:45.998115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725392+0000","last_change":"2024-04-01T01:31:16.388755+0000","last_active":"2024-04-01T01:33:19.725392+0000","last_peered":"2024-04-01T01:33:19.725392+0000","last_clean":"2024-04-01T01:33:19.725392+0000","last_became_active":"2024-04-01T01:31:16.388434+0000","last_became_peered":"2024-04-01T01:31:16.388434+0000","last_unstale":"2024-04-01T01:33:19.725392+0000","last_undegraded":"2024-04-01T01:33:19.725392+0000","last_fullsized":"2024-04-01T01:33:19.725392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:03:07.168110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727361+0000","last_change":"2024-04-01T01:33:19.727029+0000","last_active":"2024-04-01T01:33:19.727361+0000","last_peered":"2024-04-01T01:33:19.727361+0000","last_clean":"2024-04-01T01:33:19.727361+0000","last_became_active":"2024-04-01T01:33:19.726714+0000","last_became_peered":"2024-04-01T01:33:19.726714+0000","last_unstale":"2024-04-01T01:33:19.727361+0000","last_undegraded":"2024-04-01T01:33:19.727361+0000","last_fullsized":"2024-04-01T01:33:19.727361+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:35:03.224593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719204+0000","last_change":"2024-04-01T01:31:16.366921+0000","last_active":"2024-04-01T01:33:19.719204+0000","last_peered":"2024-04-01T01:33:19.719204+0000","last_clean":"2024-04-01T01:33:19.719204+0000","last_became_active":"2024-04-01T01:31:16.366098+0000","last_became_peered":"2024-04-01T01:31:16.366098+0000","last_unstale":"2024-04-01T01:33:19.719204+0000","last_undegraded":"2024-04-01T01:33:19.719204+0000","last_fullsized":"2024-04-01T01:33:19.719204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:23:29.456263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838282+0000","last_change":"2024-04-01T01:31:16.379178+0000","last_active":"2024-04-01T01:33:20.838282+0000","last_peered":"2024-04-01T01:33:20.838282+0000","last_clean":"2024-04-01T01:33:20.838282+0000","last_became_active":"2024-04-01T01:31:16.378877+0000","last_became_peered":"2024-04-01T01:31:16.378877+0000","last_unstale":"2024-04-01T01:33:20.838282+0000","last_undegraded":"2024-04-01T01:33:20.838282+0000","last_fullsized":"2024-04-01T01:33:20.838282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:19:16.941504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838274+0000","last_change":"2024-04-01T01:31:16.385373+0000","last_active":"2024-04-01T01:33:20.838274+0000","last_peered":"2024-04-01T01:33:20.838274+0000","last_clean":"2024-04-01T01:33:20.838274+0000","last_became_active":"2024-04-01T01:31:16.385046+0000","last_became_peered":"2024-04-01T01:31:16.385046+0000","last_unstale":"2024-04-01T01:33:20.838274+0000","last_undegraded":"2024-04-01T01:33:20.838274+0000","last_fullsized":"2024-04-01T01:33:20.838274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:40:32.521220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608501+0000","last_change":"2024-04-01T01:31:16.386267+0000","last_active":"2024-04-01T01:33:20.608501+0000","last_peered":"2024-04-01T01:33:20.608501+0000","last_clean":"2024-04-01T01:33:20.608501+0000","last_became_active":"2024-04-01T01:31:16.385917+0000","last_became_peered":"2024-04-01T01:31:16.385917+0000","last_unstale":"2024-04-01T01:33:20.608501+0000","last_undegraded":"2024-04-01T01:33:20.608501+0000","last_fullsized":"2024-04-01T01:33:20.608501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:58.249253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724843+0000","last_change":"2024-04-01T01:31:16.385609+0000","last_active":"2024-04-01T01:33:19.724843+0000","last_peered":"2024-04-01T01:33:19.724843+0000","last_clean":"2024-04-01T01:33:19.724843+0000","last_became_active":"2024-04-01T01:31:16.385128+0000","last_became_peered":"2024-04-01T01:31:16.385128+0000","last_unstale":"2024-04-01T01:33:19.724843+0000","last_undegraded":"2024-04-01T01:33:19.724843+0000","last_fullsized":"2024-04-01T01:33:19.724843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:30:45.915587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722053+0000","last_change":"2024-04-01T01:31:16.381209+0000","last_active":"2024-04-01T01:33:19.722053+0000","last_peered":"2024-04-01T01:33:19.722053+0000","last_clean":"2024-04-01T01:33:19.722053+0000","last_became_active":"2024-04-01T01:31:16.380721+0000","last_became_peered":"2024-04-01T01:31:16.380721+0000","last_unstale":"2024-04-01T01:33:19.722053+0000","last_undegraded":"2024-04-01T01:33:19.722053+0000","last_fullsized":"2024-04-01T01:33:19.722053+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:22.438224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727542+0000","last_change":"2024-04-01T01:31:16.385235+0000","last_active":"2024-04-01T01:33:19.727542+0000","last_peered":"2024-04-01T01:33:19.727542+0000","last_clean":"2024-04-01T01:33:19.727542+0000","last_became_active":"2024-04-01T01:31:16.384856+0000","last_became_peered":"2024-04-01T01:31:16.384856+0000","last_unstale":"2024-04-01T01:33:19.727542+0000","last_undegraded":"2024-04-01T01:33:19.727542+0000","last_fullsized":"2024-04-01T01:33:19.727542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:35:30.214977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727267+0000","last_change":"2024-04-01T01:31:20.573626+0000","last_active":"2024-04-01T01:33:19.727267+0000","last_peered":"2024-04-01T01:33:19.727267+0000","last_clean":"2024-04-01T01:33:19.727267+0000","last_became_active":"2024-04-01T01:31:20.573182+0000","last_became_peered":"2024-04-01T01:31:20.573182+0000","last_unstale":"2024-04-01T01:33:19.727267+0000","last_undegraded":"2024-04-01T01:33:19.727267+0000","last_fullsized":"2024-04-01T01:33:19.727267+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:33:19.162613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'14","reported_seq":169,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:38.711759+0000","last_change":"2024-04-01T01:31:15.373100+0000","last_active":"2024-04-01T01:33:38.711759+0000","last_peered":"2024-04-01T01:33:38.711759+0000","last_clean":"2024-04-01T01:33:38.711759+0000","last_became_active":"2024-04-01T01:31:15.372710+0000","last_became_peered":"2024-04-01T01:31:15.372710+0000","last_unstale":"2024-04-01T01:33:38.711759+0000","last_undegraded":"2024-04-01T01:33:38.711759+0000","last_fullsized":"2024-04-01T01:33:38.711759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:29:42.303549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629686+0000","last_change":"2024-04-01T01:31:16.383690+0000","last_active":"2024-04-01T01:33:20.629686+0000","last_peered":"2024-04-01T01:33:20.629686+0000","last_clean":"2024-04-01T01:33:20.629686+0000","last_became_active":"2024-04-01T01:31:16.383248+0000","last_became_peered":"2024-04-01T01:31:16.383248+0000","last_unstale":"2024-04-01T01:33:20.629686+0000","last_undegraded":"2024-04-01T01:33:20.629686+0000","last_fullsized":"2024-04-01T01:33:20.629686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:34:49.112880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721649+0000","last_change":"2024-04-01T01:32:19.691398+0000","last_active":"2024-04-01T01:33:19.721649+0000","last_peered":"2024-04-01T01:33:19.721649+0000","last_clean":"2024-04-01T01:33:19.721649+0000","last_became_active":"2024-04-01T01:32:19.690908+0000","last_became_peered":"2024-04-01T01:32:19.690908+0000","last_unstale":"2024-04-01T01:33:19.721649+0000","last_undegraded":"2024-04-01T01:33:19.721649+0000","last_fullsized":"2024-04-01T01:33:19.721649+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:58:54.712363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629846+0000","last_change":"2024-04-01T01:31:15.381742+0000","last_active":"2024-04-01T01:33:20.629846+0000","last_peered":"2024-04-01T01:33:20.629846+0000","last_clean":"2024-04-01T01:33:20.629846+0000","last_became_active":"2024-04-01T01:31:15.377543+0000","last_became_peered":"2024-04-01T01:31:15.377543+0000","last_unstale":"2024-04-01T01:33:20.629846+0000","last_undegraded":"2024-04-01T01:33:20.629846+0000","last_fullsized":"2024-04-01T01:33:20.629846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:51:42.911114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838510+0000","last_change":"2024-04-01T01:31:16.382001+0000","last_active":"2024-04-01T01:33:20.838510+0000","last_peered":"2024-04-01T01:33:20.838510+0000","last_clean":"2024-04-01T01:33:20.838510+0000","last_became_active":"2024-04-01T01:31:16.381694+0000","last_became_peered":"2024-04-01T01:31:16.381694+0000","last_unstale":"2024-04-01T01:33:20.838510+0000","last_undegraded":"2024-04-01T01:33:20.838510+0000","last_fullsized":"2024-04-01T01:33:20.838510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:55:07.366755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721790+0000","last_change":"2024-04-01T01:31:20.570080+0000","last_active":"2024-04-01T01:33:19.721790+0000","last_peered":"2024-04-01T01:33:19.721790+0000","last_clean":"2024-04-01T01:33:19.721790+0000","last_became_active":"2024-04-01T01:31:20.569712+0000","last_became_peered":"2024-04-01T01:31:20.569712+0000","last_unstale":"2024-04-01T01:33:19.721790+0000","last_undegraded":"2024-04-01T01:33:19.721790+0000","last_fullsized":"2024-04-01T01:33:19.721790+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:48:27.732831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725670+0000","last_change":"2024-04-01T01:32:19.703903+0000","last_active":"2024-04-01T01:33:19.725670+0000","last_peered":"2024-04-01T01:33:19.725670+0000","last_clean":"2024-04-01T01:33:19.725670+0000","last_became_active":"2024-04-01T01:32:19.691856+0000","last_became_peered":"2024-04-01T01:32:19.691856+0000","last_unstale":"2024-04-01T01:33:19.725670+0000","last_undegraded":"2024-04-01T01:33:19.725670+0000","last_fullsized":"2024-04-01T01:33:19.725670+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:38:41.284469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608732+0000","last_change":"2024-04-01T01:31:16.383020+0000","last_active":"2024-04-01T01:33:20.608732+0000","last_peered":"2024-04-01T01:33:20.608732+0000","last_clean":"2024-04-01T01:33:20.608732+0000","last_became_active":"2024-04-01T01:31:16.382640+0000","last_became_peered":"2024-04-01T01:31:16.382640+0000","last_unstale":"2024-04-01T01:33:20.608732+0000","last_undegraded":"2024-04-01T01:33:20.608732+0000","last_fullsized":"2024-04-01T01:33:20.608732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:11:51.211929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721753+0000","last_change":"2024-04-01T01:31:20.573030+0000","last_active":"2024-04-01T01:33:19.721753+0000","last_peered":"2024-04-01T01:33:19.721753+0000","last_clean":"2024-04-01T01:33:19.721753+0000","last_became_active":"2024-04-01T01:31:20.572613+0000","last_became_peered":"2024-04-01T01:31:20.572613+0000","last_unstale":"2024-04-01T01:33:19.721753+0000","last_undegraded":"2024-04-01T01:33:19.721753+0000","last_fullsized":"2024-04-01T01:33:19.721753+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:42:17.086942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725015+0000","last_change":"2024-04-01T01:31:15.373054+0000","last_active":"2024-04-01T01:33:19.725015+0000","last_peered":"2024-04-01T01:33:19.725015+0000","last_clean":"2024-04-01T01:33:19.725015+0000","last_became_active":"2024-04-01T01:31:15.372575+0000","last_became_peered":"2024-04-01T01:31:15.372575+0000","last_unstale":"2024-04-01T01:33:19.725015+0000","last_undegraded":"2024-04-01T01:33:19.725015+0000","last_fullsized":"2024-04-01T01:33:19.725015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:56:41.860423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722460+0000","last_change":"2024-04-01T01:31:16.376037+0000","last_active":"2024-04-01T01:33:19.722460+0000","last_peered":"2024-04-01T01:33:19.722460+0000","last_clean":"2024-04-01T01:33:19.722460+0000","last_became_active":"2024-04-01T01:31:16.375463+0000","last_became_peered":"2024-04-01T01:31:16.375463+0000","last_unstale":"2024-04-01T01:33:19.722460+0000","last_undegraded":"2024-04-01T01:33:19.722460+0000","last_fullsized":"2024-04-01T01:33:19.722460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:30:40.098033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727397+0000","last_change":"2024-04-01T01:31:20.574288+0000","last_active":"2024-04-01T01:33:19.727397+0000","last_peered":"2024-04-01T01:33:19.727397+0000","last_clean":"2024-04-01T01:33:19.727397+0000","last_became_active":"2024-04-01T01:31:20.573812+0000","last_became_peered":"2024-04-01T01:31:20.573812+0000","last_unstale":"2024-04-01T01:33:19.727397+0000","last_undegraded":"2024-04-01T01:33:19.727397+0000","last_fullsized":"2024-04-01T01:33:19.727397+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:43:46.126102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":177,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:35.766196+0000","last_change":"2024-04-01T01:32:19.706645+0000","last_active":"2024-04-01T01:33:35.766196+0000","last_peered":"2024-04-01T01:33:35.766196+0000","last_clean":"2024-04-01T01:33:35.766196+0000","last_became_active":"2024-04-01T01:32:19.690510+0000","last_became_peered":"2024-04-01T01:32:19.690510+0000","last_unstale":"2024-04-01T01:33:35.766196+0000","last_undegraded":"2024-04-01T01:33:35.766196+0000","last_fullsized":"2024-04-01T01:33:35.766196+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:20:23.103229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1631,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724703+0000","last_change":"2024-04-01T01:31:16.377735+0000","last_active":"2024-04-01T01:33:19.724703+0000","last_peered":"2024-04-01T01:33:19.724703+0000","last_clean":"2024-04-01T01:33:19.724703+0000","last_became_active":"2024-04-01T01:31:16.377342+0000","last_became_peered":"2024-04-01T01:31:16.377342+0000","last_unstale":"2024-04-01T01:33:19.724703+0000","last_undegraded":"2024-04-01T01:33:19.724703+0000","last_fullsized":"2024-04-01T01:33:19.724703+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:33:47.539223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727261+0000","last_change":"2024-04-01T01:31:20.573370+0000","last_active":"2024-04-01T01:33:19.727261+0000","last_peered":"2024-04-01T01:33:19.727261+0000","last_clean":"2024-04-01T01:33:19.727261+0000","last_became_active":"2024-04-01T01:31:20.572919+0000","last_became_peered":"2024-04-01T01:31:20.572919+0000","last_unstale":"2024-04-01T01:33:19.727261+0000","last_undegraded":"2024-04-01T01:33:19.727261+0000","last_fullsized":"2024-04-01T01:33:19.727261+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:59:42.190918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.723867+0000","last_change":"2024-04-01T01:32:19.810468+0000","last_active":"2024-04-01T01:33:19.723867+0000","last_peered":"2024-04-01T01:33:19.723867+0000","last_clean":"2024-04-01T01:33:19.723867+0000","last_became_active":"2024-04-01T01:32:19.805517+0000","last_became_peered":"2024-04-01T01:32:19.805517+0000","last_unstale":"2024-04-01T01:33:19.723867+0000","last_undegraded":"2024-04-01T01:33:19.723867+0000","last_fullsized":"2024-04-01T01:33:19.723867+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:54:35.355761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722977+0000","last_change":"2024-04-01T01:31:16.384130+0000","last_active":"2024-04-01T01:33:19.722977+0000","last_peered":"2024-04-01T01:33:19.722977+0000","last_clean":"2024-04-01T01:33:19.722977+0000","last_became_active":"2024-04-01T01:31:16.383400+0000","last_became_peered":"2024-04-01T01:31:16.383400+0000","last_unstale":"2024-04-01T01:33:19.722977+0000","last_undegraded":"2024-04-01T01:33:19.722977+0000","last_fullsized":"2024-04-01T01:33:19.722977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:22:10.860325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724729+0000","last_change":"2024-04-01T01:31:20.571052+0000","last_active":"2024-04-01T01:33:19.724729+0000","last_peered":"2024-04-01T01:33:19.724729+0000","last_clean":"2024-04-01T01:33:19.724729+0000","last_became_active":"2024-04-01T01:31:20.570617+0000","last_became_peered":"2024-04-01T01:31:20.570617+0000","last_unstale":"2024-04-01T01:33:19.724729+0000","last_undegraded":"2024-04-01T01:33:19.724729+0000","last_fullsized":"2024-04-01T01:33:19.724729+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:53:51.195056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719095+0000","last_change":"2024-04-01T01:31:15.369820+0000","last_active":"2024-04-01T01:33:19.719095+0000","last_peered":"2024-04-01T01:33:19.719095+0000","last_clean":"2024-04-01T01:33:19.719095+0000","last_became_active":"2024-04-01T01:31:15.369378+0000","last_became_peered":"2024-04-01T01:31:15.369378+0000","last_unstale":"2024-04-01T01:33:19.719095+0000","last_undegraded":"2024-04-01T01:33:19.719095+0000","last_fullsized":"2024-04-01T01:33:19.719095+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:37:34.351220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.628952+0000","last_change":"2024-04-01T01:31:16.383352+0000","last_active":"2024-04-01T01:33:20.628952+0000","last_peered":"2024-04-01T01:33:20.628952+0000","last_clean":"2024-04-01T01:33:20.628952+0000","last_became_active":"2024-04-01T01:31:16.382898+0000","last_became_peered":"2024-04-01T01:31:16.382898+0000","last_unstale":"2024-04-01T01:33:20.628952+0000","last_undegraded":"2024-04-01T01:33:20.628952+0000","last_fullsized":"2024-04-01T01:33:20.628952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:51:27.854459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.837895+0000","last_change":"2024-04-01T01:31:20.576578+0000","last_active":"2024-04-01T01:33:20.837895+0000","last_peered":"2024-04-01T01:33:20.837895+0000","last_clean":"2024-04-01T01:33:20.837895+0000","last_became_active":"2024-04-01T01:31:20.575996+0000","last_became_peered":"2024-04-01T01:31:20.575996+0000","last_unstale":"2024-04-01T01:33:20.837895+0000","last_undegraded":"2024-04-01T01:33:20.837895+0000","last_fullsized":"2024-04-01T01:33:20.837895+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:52:08.808357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722157+0000","last_change":"2024-04-01T01:31:15.377246+0000","last_active":"2024-04-01T01:33:19.722157+0000","last_peered":"2024-04-01T01:33:19.722157+0000","last_clean":"2024-04-01T01:33:19.722157+0000","last_became_active":"2024-04-01T01:31:15.376957+0000","last_became_peered":"2024-04-01T01:31:15.376957+0000","last_unstale":"2024-04-01T01:33:19.722157+0000","last_undegraded":"2024-04-01T01:33:19.722157+0000","last_fullsized":"2024-04-01T01:33:19.722157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:59:34.313984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721661+0000","last_change":"2024-04-01T01:31:16.379031+0000","last_active":"2024-04-01T01:33:19.721661+0000","last_peered":"2024-04-01T01:33:19.721661+0000","last_clean":"2024-04-01T01:33:19.721661+0000","last_became_active":"2024-04-01T01:31:16.378593+0000","last_became_peered":"2024-04-01T01:31:16.378593+0000","last_unstale":"2024-04-01T01:33:19.721661+0000","last_undegraded":"2024-04-01T01:33:19.721661+0000","last_fullsized":"2024-04-01T01:33:19.721661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:17:31.816103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.628494+0000","last_change":"2024-04-01T01:31:20.578660+0000","last_active":"2024-04-01T01:33:20.628494+0000","last_peered":"2024-04-01T01:33:20.628494+0000","last_clean":"2024-04-01T01:33:20.628494+0000","last_became_active":"2024-04-01T01:31:20.578082+0000","last_became_peered":"2024-04-01T01:31:20.578082+0000","last_unstale":"2024-04-01T01:33:20.628494+0000","last_undegraded":"2024-04-01T01:33:20.628494+0000","last_fullsized":"2024-04-01T01:33:20.628494+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:02:56.570816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'7","reported_seq":277,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:35.719505+0000","last_change":"2024-04-01T01:31:15.373428+0000","last_active":"2024-04-01T01:33:35.719505+0000","last_peered":"2024-04-01T01:33:35.719505+0000","last_clean":"2024-04-01T01:33:35.719505+0000","last_became_active":"2024-04-01T01:31:15.372965+0000","last_became_peered":"2024-04-01T01:31:15.372965+0000","last_unstale":"2024-04-01T01:33:35.719505+0000","last_undegraded":"2024-04-01T01:33:35.719505+0000","last_fullsized":"2024-04-01T01:33:35.719505+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:55:42.211055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":239,"num_read_kb":239,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.609010+0000","last_change":"2024-04-01T01:31:16.382961+0000","last_active":"2024-04-01T01:33:20.609010+0000","last_peered":"2024-04-01T01:33:20.609010+0000","last_clean":"2024-04-01T01:33:20.609010+0000","last_became_active":"2024-04-01T01:31:16.382527+0000","last_became_peered":"2024-04-01T01:31:16.382527+0000","last_unstale":"2024-04-01T01:33:20.609010+0000","last_undegraded":"2024-04-01T01:33:20.609010+0000","last_fullsized":"2024-04-01T01:33:20.609010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:46:20.250064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721640+0000","last_change":"2024-04-01T01:31:20.571842+0000","last_active":"2024-04-01T01:33:19.721640+0000","last_peered":"2024-04-01T01:33:19.721640+0000","last_clean":"2024-04-01T01:33:19.721640+0000","last_became_active":"2024-04-01T01:31:20.571394+0000","last_became_peered":"2024-04-01T01:31:20.571394+0000","last_unstale":"2024-04-01T01:33:19.721640+0000","last_undegraded":"2024-04-01T01:33:19.721640+0000","last_fullsized":"2024-04-01T01:33:19.721640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:51:36.512277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725497+0000","last_change":"2024-04-01T01:31:15.379737+0000","last_active":"2024-04-01T01:33:19.725497+0000","last_peered":"2024-04-01T01:33:19.725497+0000","last_clean":"2024-04-01T01:33:19.725497+0000","last_became_active":"2024-04-01T01:31:15.379158+0000","last_became_peered":"2024-04-01T01:31:15.379158+0000","last_unstale":"2024-04-01T01:33:19.725497+0000","last_undegraded":"2024-04-01T01:33:19.725497+0000","last_fullsized":"2024-04-01T01:33:19.725497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:12:44.474772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838564+0000","last_change":"2024-04-01T01:31:16.386059+0000","last_active":"2024-04-01T01:33:20.838564+0000","last_peered":"2024-04-01T01:33:20.838564+0000","last_clean":"2024-04-01T01:33:20.838564+0000","last_became_active":"2024-04-01T01:31:16.384551+0000","last_became_peered":"2024-04-01T01:31:16.384551+0000","last_unstale":"2024-04-01T01:33:20.838564+0000","last_undegraded":"2024-04-01T01:33:20.838564+0000","last_fullsized":"2024-04-01T01:33:20.838564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:52:16.594485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721972+0000","last_change":"2024-04-01T01:31:20.566937+0000","last_active":"2024-04-01T01:33:19.721972+0000","last_peered":"2024-04-01T01:33:19.721972+0000","last_clean":"2024-04-01T01:33:19.721972+0000","last_became_active":"2024-04-01T01:31:20.566447+0000","last_became_peered":"2024-04-01T01:31:20.566447+0000","last_unstale":"2024-04-01T01:33:19.721972+0000","last_undegraded":"2024-04-01T01:33:19.721972+0000","last_fullsized":"2024-04-01T01:33:19.721972+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:17:37.845166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722425+0000","last_change":"2024-04-01T01:31:15.375669+0000","last_active":"2024-04-01T01:33:19.722425+0000","last_peered":"2024-04-01T01:33:19.722425+0000","last_clean":"2024-04-01T01:33:19.722425+0000","last_became_active":"2024-04-01T01:31:15.375353+0000","last_became_peered":"2024-04-01T01:31:15.375353+0000","last_unstale":"2024-04-01T01:33:19.722425+0000","last_undegraded":"2024-04-01T01:33:19.722425+0000","last_fullsized":"2024-04-01T01:33:19.722425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:47:43.096898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608910+0000","last_change":"2024-04-01T01:31:16.388399+0000","last_active":"2024-04-01T01:33:20.608910+0000","last_peered":"2024-04-01T01:33:20.608910+0000","last_clean":"2024-04-01T01:33:20.608910+0000","last_became_active":"2024-04-01T01:31:16.388059+0000","last_became_peered":"2024-04-01T01:31:16.388059+0000","last_unstale":"2024-04-01T01:33:20.608910+0000","last_undegraded":"2024-04-01T01:33:20.608910+0000","last_fullsized":"2024-04-01T01:33:20.608910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:59:50.907802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.723472+0000","last_change":"2024-04-01T01:32:19.692098+0000","last_active":"2024-04-01T01:33:19.723472+0000","last_peered":"2024-04-01T01:33:19.723472+0000","last_clean":"2024-04-01T01:33:19.723472+0000","last_became_active":"2024-04-01T01:32:19.691673+0000","last_became_peered":"2024-04-01T01:32:19.691673+0000","last_unstale":"2024-04-01T01:33:19.723472+0000","last_undegraded":"2024-04-01T01:33:19.723472+0000","last_fullsized":"2024-04-01T01:33:19.723472+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:14:17.300517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"93'39","reported_seq":323,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:35.766454+0000","last_change":"2024-04-01T01:31:15.376421+0000","last_active":"2024-04-01T01:33:35.766454+0000","last_peered":"2024-04-01T01:33:35.766454+0000","last_clean":"2024-04-01T01:33:35.766454+0000","last_became_active":"2024-04-01T01:31:15.375985+0000","last_became_peered":"2024-04-01T01:31:15.375985+0000","last_unstale":"2024-04-01T01:33:35.766454+0000","last_undegraded":"2024-04-01T01:33:35.766454+0000","last_fullsized":"2024-04-01T01:33:35.766454+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:37:29.121436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64082,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":252,"num_read_kb":189,"num_write":41,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725852+0000","last_change":"2024-04-01T01:31:16.380641+0000","last_active":"2024-04-01T01:33:19.725852+0000","last_peered":"2024-04-01T01:33:19.725852+0000","last_clean":"2024-04-01T01:33:19.725852+0000","last_became_active":"2024-04-01T01:31:16.380300+0000","last_became_peered":"2024-04-01T01:31:16.380300+0000","last_unstale":"2024-04-01T01:33:19.725852+0000","last_undegraded":"2024-04-01T01:33:19.725852+0000","last_fullsized":"2024-04-01T01:33:19.725852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:26:08.240694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722617+0000","last_change":"2024-04-01T01:31:20.576094+0000","last_active":"2024-04-01T01:33:19.722617+0000","last_peered":"2024-04-01T01:33:19.722617+0000","last_clean":"2024-04-01T01:33:19.722617+0000","last_became_active":"2024-04-01T01:31:20.575674+0000","last_became_peered":"2024-04-01T01:31:20.575674+0000","last_unstale":"2024-04-01T01:33:19.722617+0000","last_undegraded":"2024-04-01T01:33:19.722617+0000","last_fullsized":"2024-04-01T01:33:19.722617+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:42:19.352398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727136+0000","last_change":"2024-04-01T01:31:15.375136+0000","last_active":"2024-04-01T01:33:19.727136+0000","last_peered":"2024-04-01T01:33:19.727136+0000","last_clean":"2024-04-01T01:33:19.727136+0000","last_became_active":"2024-04-01T01:31:15.374813+0000","last_became_peered":"2024-04-01T01:31:15.374813+0000","last_unstale":"2024-04-01T01:33:19.727136+0000","last_undegraded":"2024-04-01T01:33:19.727136+0000","last_fullsized":"2024-04-01T01:33:19.727136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:19:05.490103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719129+0000","last_change":"2024-04-01T01:31:16.380207+0000","last_active":"2024-04-01T01:33:19.719129+0000","last_peered":"2024-04-01T01:33:19.719129+0000","last_clean":"2024-04-01T01:33:19.719129+0000","last_became_active":"2024-04-01T01:31:16.379891+0000","last_became_peered":"2024-04-01T01:31:16.379891+0000","last_unstale":"2024-04-01T01:33:19.719129+0000","last_undegraded":"2024-04-01T01:33:19.719129+0000","last_fullsized":"2024-04-01T01:33:19.719129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:12:36.417162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724683+0000","last_change":"2024-04-01T01:31:20.569978+0000","last_active":"2024-04-01T01:33:19.724683+0000","last_peered":"2024-04-01T01:33:19.724683+0000","last_clean":"2024-04-01T01:33:19.724683+0000","last_became_active":"2024-04-01T01:31:20.568298+0000","last_became_peered":"2024-04-01T01:31:20.568298+0000","last_unstale":"2024-04-01T01:33:19.724683+0000","last_undegraded":"2024-04-01T01:33:19.724683+0000","last_fullsized":"2024-04-01T01:33:19.724683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:42:16.439284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722815+0000","last_change":"2024-04-01T01:31:15.380143+0000","last_active":"2024-04-01T01:33:19.722815+0000","last_peered":"2024-04-01T01:33:19.722815+0000","last_clean":"2024-04-01T01:33:19.722815+0000","last_became_active":"2024-04-01T01:31:15.379835+0000","last_became_peered":"2024-04-01T01:31:15.379835+0000","last_unstale":"2024-04-01T01:33:19.722815+0000","last_undegraded":"2024-04-01T01:33:19.722815+0000","last_fullsized":"2024-04-01T01:33:19.722815+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:28:34.213626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725248+0000","last_change":"2024-04-01T01:31:16.380290+0000","last_active":"2024-04-01T01:33:19.725248+0000","last_peered":"2024-04-01T01:33:19.725248+0000","last_clean":"2024-04-01T01:33:19.725248+0000","last_became_active":"2024-04-01T01:31:16.379983+0000","last_became_peered":"2024-04-01T01:31:16.379983+0000","last_unstale":"2024-04-01T01:33:19.725248+0000","last_undegraded":"2024-04-01T01:33:19.725248+0000","last_fullsized":"2024-04-01T01:33:19.725248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:55:50.135454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721926+0000","last_change":"2024-04-01T01:31:20.573500+0000","last_active":"2024-04-01T01:33:19.721926+0000","last_peered":"2024-04-01T01:33:19.721926+0000","last_clean":"2024-04-01T01:33:19.721926+0000","last_became_active":"2024-04-01T01:31:20.573101+0000","last_became_peered":"2024-04-01T01:31:20.573101+0000","last_unstale":"2024-04-01T01:33:19.721926+0000","last_undegraded":"2024-04-01T01:33:19.721926+0000","last_fullsized":"2024-04-01T01:33:19.721926+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:11:27.881703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'8","reported_seq":163,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:36.723420+0000","last_change":"2024-04-01T01:31:15.377265+0000","last_active":"2024-04-01T01:33:36.723420+0000","last_peered":"2024-04-01T01:33:36.723420+0000","last_clean":"2024-04-01T01:33:36.723420+0000","last_became_active":"2024-04-01T01:31:15.376889+0000","last_became_peered":"2024-04-01T01:31:15.376889+0000","last_unstale":"2024-04-01T01:33:36.723420+0000","last_undegraded":"2024-04-01T01:33:36.723420+0000","last_fullsized":"2024-04-01T01:33:36.723420+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:43:31.582866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1597,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724953+0000","last_change":"2024-04-01T01:31:16.384982+0000","last_active":"2024-04-01T01:33:19.724953+0000","last_peered":"2024-04-01T01:33:19.724953+0000","last_clean":"2024-04-01T01:33:19.724953+0000","last_became_active":"2024-04-01T01:31:16.384663+0000","last_became_peered":"2024-04-01T01:31:16.384663+0000","last_unstale":"2024-04-01T01:33:19.724953+0000","last_undegraded":"2024-04-01T01:33:19.724953+0000","last_fullsized":"2024-04-01T01:33:19.724953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:14:57.351768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719430+0000","last_change":"2024-04-01T01:31:20.578761+0000","last_active":"2024-04-01T01:33:19.719430+0000","last_peered":"2024-04-01T01:33:19.719430+0000","last_clean":"2024-04-01T01:33:19.719430+0000","last_became_active":"2024-04-01T01:31:20.578204+0000","last_became_peered":"2024-04-01T01:31:20.578204+0000","last_unstale":"2024-04-01T01:33:19.719430+0000","last_undegraded":"2024-04-01T01:33:19.719430+0000","last_fullsized":"2024-04-01T01:33:19.719430+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:19:33.584872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724403+0000","last_change":"2024-04-01T01:31:15.380773+0000","last_active":"2024-04-01T01:33:19.724403+0000","last_peered":"2024-04-01T01:33:19.724403+0000","last_clean":"2024-04-01T01:33:19.724403+0000","last_became_active":"2024-04-01T01:31:15.380472+0000","last_became_peered":"2024-04-01T01:31:15.380472+0000","last_unstale":"2024-04-01T01:33:19.724403+0000","last_undegraded":"2024-04-01T01:33:19.724403+0000","last_fullsized":"2024-04-01T01:33:19.724403+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:34:28.294500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722296+0000","last_change":"2024-04-01T01:31:16.374658+0000","last_active":"2024-04-01T01:33:19.722296+0000","last_peered":"2024-04-01T01:33:19.722296+0000","last_clean":"2024-04-01T01:33:19.722296+0000","last_became_active":"2024-04-01T01:31:16.373596+0000","last_became_peered":"2024-04-01T01:31:16.373596+0000","last_unstale":"2024-04-01T01:33:19.722296+0000","last_undegraded":"2024-04-01T01:33:19.722296+0000","last_fullsized":"2024-04-01T01:33:19.722296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:28:38.477719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722239+0000","last_change":"2024-04-01T01:31:20.577449+0000","last_active":"2024-04-01T01:33:19.722239+0000","last_peered":"2024-04-01T01:33:19.722239+0000","last_clean":"2024-04-01T01:33:19.722239+0000","last_became_active":"2024-04-01T01:31:20.577026+0000","last_became_peered":"2024-04-01T01:31:20.577026+0000","last_unstale":"2024-04-01T01:33:19.722239+0000","last_undegraded":"2024-04-01T01:33:19.722239+0000","last_fullsized":"2024-04-01T01:33:19.722239+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:44:48.080304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:33.356435+0000","last_change":"2024-04-01T01:31:15.373885+0000","last_active":"2024-04-01T01:33:33.356435+0000","last_peered":"2024-04-01T01:33:33.356435+0000","last_clean":"2024-04-01T01:33:33.356435+0000","last_became_active":"2024-04-01T01:31:15.373578+0000","last_became_peered":"2024-04-01T01:31:15.373578+0000","last_unstale":"2024-04-01T01:33:33.356435+0000","last_undegraded":"2024-04-01T01:33:33.356435+0000","last_fullsized":"2024-04-01T01:33:33.356435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:52.641515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.730680+0000","last_change":"2024-04-01T01:33:19.730088+0000","last_active":"2024-04-01T01:33:19.730680+0000","last_peered":"2024-04-01T01:33:19.730680+0000","last_clean":"2024-04-01T01:33:19.730680+0000","last_became_active":"2024-04-01T01:33:19.729627+0000","last_became_peered":"2024-04-01T01:33:19.729627+0000","last_unstale":"2024-04-01T01:33:19.730680+0000","last_undegraded":"2024-04-01T01:33:19.730680+0000","last_fullsized":"2024-04-01T01:33:19.730680+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:47:37.839420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725463+0000","last_change":"2024-04-01T01:31:20.569835+0000","last_active":"2024-04-01T01:33:19.725463+0000","last_peered":"2024-04-01T01:33:19.725463+0000","last_clean":"2024-04-01T01:33:19.725463+0000","last_became_active":"2024-04-01T01:31:20.569195+0000","last_became_peered":"2024-04-01T01:31:20.569195+0000","last_unstale":"2024-04-01T01:33:19.725463+0000","last_undegraded":"2024-04-01T01:33:19.725463+0000","last_fullsized":"2024-04-01T01:33:19.725463+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:10:18.278575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721958+0000","last_change":"2024-04-01T01:31:15.377679+0000","last_active":"2024-04-01T01:33:19.721958+0000","last_peered":"2024-04-01T01:33:19.721958+0000","last_clean":"2024-04-01T01:33:19.721958+0000","last_became_active":"2024-04-01T01:31:15.377351+0000","last_became_peered":"2024-04-01T01:31:15.377351+0000","last_unstale":"2024-04-01T01:33:19.721958+0000","last_undegraded":"2024-04-01T01:33:19.721958+0000","last_fullsized":"2024-04-01T01:33:19.721958+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:22:41.216563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724294+0000","last_change":"2024-04-01T01:31:16.380997+0000","last_active":"2024-04-01T01:33:19.724294+0000","last_peered":"2024-04-01T01:33:19.724294+0000","last_clean":"2024-04-01T01:33:19.724294+0000","last_became_active":"2024-04-01T01:31:16.380692+0000","last_became_peered":"2024-04-01T01:31:16.380692+0000","last_unstale":"2024-04-01T01:33:19.724294+0000","last_undegraded":"2024-04-01T01:33:19.724294+0000","last_fullsized":"2024-04-01T01:33:19.724294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:09:51.220849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719249+0000","last_change":"2024-04-01T01:31:20.573191+0000","last_active":"2024-04-01T01:33:19.719249+0000","last_peered":"2024-04-01T01:33:19.719249+0000","last_clean":"2024-04-01T01:33:19.719249+0000","last_became_active":"2024-04-01T01:31:20.572669+0000","last_became_peered":"2024-04-01T01:31:20.572669+0000","last_unstale":"2024-04-01T01:33:19.719249+0000","last_undegraded":"2024-04-01T01:33:19.719249+0000","last_fullsized":"2024-04-01T01:33:19.719249+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:01:56.491271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"93'7","reported_seq":164,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:38.712613+0000","last_change":"2024-04-01T01:31:15.379566+0000","last_active":"2024-04-01T01:33:38.712613+0000","last_peered":"2024-04-01T01:33:38.712613+0000","last_clean":"2024-04-01T01:33:38.712613+0000","last_became_active":"2024-04-01T01:31:15.379213+0000","last_became_peered":"2024-04-01T01:31:15.379213+0000","last_unstale":"2024-04-01T01:33:38.712613+0000","last_undegraded":"2024-04-01T01:33:38.712613+0000","last_fullsized":"2024-04-01T01:33:38.712613+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:34:17.974422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1456,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724071+0000","last_change":"2024-04-01T01:31:16.376218+0000","last_active":"2024-04-01T01:33:19.724071+0000","last_peered":"2024-04-01T01:33:19.724071+0000","last_clean":"2024-04-01T01:33:19.724071+0000","last_became_active":"2024-04-01T01:31:16.375887+0000","last_became_peered":"2024-04-01T01:31:16.375887+0000","last_unstale":"2024-04-01T01:33:19.724071+0000","last_undegraded":"2024-04-01T01:33:19.724071+0000","last_fullsized":"2024-04-01T01:33:19.724071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:27:03.723827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719394+0000","last_change":"2024-04-01T01:31:20.576461+0000","last_active":"2024-04-01T01:33:19.719394+0000","last_peered":"2024-04-01T01:33:19.719394+0000","last_clean":"2024-04-01T01:33:19.719394+0000","last_became_active":"2024-04-01T01:31:20.575985+0000","last_became_peered":"2024-04-01T01:31:20.575985+0000","last_unstale":"2024-04-01T01:33:19.719394+0000","last_undegraded":"2024-04-01T01:33:19.719394+0000","last_fullsized":"2024-04-01T01:33:19.719394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:19:02.714455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722259+0000","last_change":"2024-04-01T01:31:16.376456+0000","last_active":"2024-04-01T01:33:19.722259+0000","last_peered":"2024-04-01T01:33:19.722259+0000","last_clean":"2024-04-01T01:33:19.722259+0000","last_became_active":"2024-04-01T01:31:16.375445+0000","last_became_peered":"2024-04-01T01:31:16.375445+0000","last_unstale":"2024-04-01T01:33:19.722259+0000","last_undegraded":"2024-04-01T01:33:19.722259+0000","last_fullsized":"2024-04-01T01:33:19.722259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:07:11.802799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'7","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:38.725028+0000","last_change":"2024-04-01T01:31:15.382702+0000","last_active":"2024-04-01T01:33:38.725028+0000","last_peered":"2024-04-01T01:33:38.725028+0000","last_clean":"2024-04-01T01:33:38.725028+0000","last_became_active":"2024-04-01T01:31:15.382377+0000","last_became_peered":"2024-04-01T01:31:15.382377+0000","last_unstale":"2024-04-01T01:33:38.725028+0000","last_undegraded":"2024-04-01T01:33:38.725028+0000","last_fullsized":"2024-04-01T01:33:38.725028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:45.080992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725315+0000","last_change":"2024-04-01T01:32:19.704607+0000","last_active":"2024-04-01T01:33:19.725315+0000","last_peered":"2024-04-01T01:33:19.725315+0000","last_clean":"2024-04-01T01:33:19.725315+0000","last_became_active":"2024-04-01T01:32:19.694650+0000","last_became_peered":"2024-04-01T01:32:19.694650+0000","last_unstale":"2024-04-01T01:33:19.725315+0000","last_undegraded":"2024-04-01T01:33:19.725315+0000","last_fullsized":"2024-04-01T01:33:19.725315+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:48:55.439187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629394+0000","last_change":"2024-04-01T01:31:16.387981+0000","last_active":"2024-04-01T01:33:20.629394+0000","last_peered":"2024-04-01T01:33:20.629394+0000","last_clean":"2024-04-01T01:33:20.629394+0000","last_became_active":"2024-04-01T01:31:16.387594+0000","last_became_peered":"2024-04-01T01:31:16.387594+0000","last_unstale":"2024-04-01T01:33:20.629394+0000","last_undegraded":"2024-04-01T01:33:20.629394+0000","last_fullsized":"2024-04-01T01:33:20.629394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:17.148752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721940+0000","last_change":"2024-04-01T01:31:15.375212+0000","last_active":"2024-04-01T01:33:19.721940+0000","last_peered":"2024-04-01T01:33:19.721940+0000","last_clean":"2024-04-01T01:33:19.721940+0000","last_became_active":"2024-04-01T01:31:15.374882+0000","last_became_peered":"2024-04-01T01:31:15.374882+0000","last_unstale":"2024-04-01T01:33:19.721940+0000","last_undegraded":"2024-04-01T01:33:19.721940+0000","last_fullsized":"2024-04-01T01:33:19.721940+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:56:23.412025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724742+0000","last_change":"2024-04-01T01:30:03.538590+0000","last_active":"2024-04-01T01:33:19.724742+0000","last_peered":"2024-04-01T01:33:19.724742+0000","last_clean":"2024-04-01T01:33:19.724742+0000","last_became_active":"2024-04-01T01:30:03.538098+0000","last_became_peered":"2024-04-01T01:30:03.538098+0000","last_unstale":"2024-04-01T01:33:19.724742+0000","last_undegraded":"2024-04-01T01:33:19.724742+0000","last_fullsized":"2024-04-01T01:33:19.724742+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:26:43.445198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:26:43.445198+0000","last_clean_scrub_stamp":"2024-04-01T01:26:43.445198+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:22.207795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722391+0000","last_change":"2024-04-01T01:31:20.573260+0000","last_active":"2024-04-01T01:33:19.722391+0000","last_peered":"2024-04-01T01:33:19.722391+0000","last_clean":"2024-04-01T01:33:19.722391+0000","last_became_active":"2024-04-01T01:31:20.572846+0000","last_became_peered":"2024-04-01T01:31:20.572846+0000","last_unstale":"2024-04-01T01:33:19.722391+0000","last_undegraded":"2024-04-01T01:33:19.722391+0000","last_fullsized":"2024-04-01T01:33:19.722391+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:57:24.986070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.718978+0000","last_change":"2024-04-01T01:31:16.384782+0000","last_active":"2024-04-01T01:33:19.718978+0000","last_peered":"2024-04-01T01:33:19.718978+0000","last_clean":"2024-04-01T01:33:19.718978+0000","last_became_active":"2024-04-01T01:31:16.384420+0000","last_became_peered":"2024-04-01T01:31:16.384420+0000","last_unstale":"2024-04-01T01:33:19.718978+0000","last_undegraded":"2024-04-01T01:33:19.718978+0000","last_fullsized":"2024-04-01T01:33:19.718978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:00:20.902022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724778+0000","last_change":"2024-04-01T01:31:15.381853+0000","last_active":"2024-04-01T01:33:19.724778+0000","last_peered":"2024-04-01T01:33:19.724778+0000","last_clean":"2024-04-01T01:33:19.724778+0000","last_became_active":"2024-04-01T01:31:15.381496+0000","last_became_peered":"2024-04-01T01:31:15.381496+0000","last_unstale":"2024-04-01T01:33:19.724778+0000","last_undegraded":"2024-04-01T01:33:19.724778+0000","last_fullsized":"2024-04-01T01:33:19.724778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:07:51.440351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724645+0000","last_change":"2024-04-01T01:31:20.568705+0000","last_active":"2024-04-01T01:33:19.724645+0000","last_peered":"2024-04-01T01:33:19.724645+0000","last_clean":"2024-04-01T01:33:19.724645+0000","last_became_active":"2024-04-01T01:31:20.568252+0000","last_became_peered":"2024-04-01T01:31:20.568252+0000","last_unstale":"2024-04-01T01:33:19.724645+0000","last_undegraded":"2024-04-01T01:33:19.724645+0000","last_fullsized":"2024-04-01T01:33:19.724645+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:22:28.509510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724763+0000","last_change":"2024-04-01T01:32:19.699781+0000","last_active":"2024-04-01T01:33:19.724763+0000","last_peered":"2024-04-01T01:33:19.724763+0000","last_clean":"2024-04-01T01:33:19.724763+0000","last_became_active":"2024-04-01T01:32:19.689593+0000","last_became_peered":"2024-04-01T01:32:19.689593+0000","last_unstale":"2024-04-01T01:33:19.724763+0000","last_undegraded":"2024-04-01T01:33:19.724763+0000","last_fullsized":"2024-04-01T01:33:19.724763+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:44:21.922097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719384+0000","last_change":"2024-04-01T01:31:16.377002+0000","last_active":"2024-04-01T01:33:19.719384+0000","last_peered":"2024-04-01T01:33:19.719384+0000","last_clean":"2024-04-01T01:33:19.719384+0000","last_became_active":"2024-04-01T01:31:16.376679+0000","last_became_peered":"2024-04-01T01:31:16.376679+0000","last_unstale":"2024-04-01T01:33:19.719384+0000","last_undegraded":"2024-04-01T01:33:19.719384+0000","last_fullsized":"2024-04-01T01:33:19.719384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:50:24.126192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.628550+0000","last_change":"2024-04-01T01:31:20.577757+0000","last_active":"2024-04-01T01:33:20.628550+0000","last_peered":"2024-04-01T01:33:20.628550+0000","last_clean":"2024-04-01T01:33:20.628550+0000","last_became_active":"2024-04-01T01:31:20.577378+0000","last_became_peered":"2024-04-01T01:31:20.577378+0000","last_unstale":"2024-04-01T01:33:20.628550+0000","last_undegraded":"2024-04-01T01:33:20.628550+0000","last_fullsized":"2024-04-01T01:33:20.628550+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:48:04.511229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'16","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:36.559784+0000","last_change":"2024-04-01T01:31:15.383201+0000","last_active":"2024-04-01T01:33:36.559784+0000","last_peered":"2024-04-01T01:33:36.559784+0000","last_clean":"2024-04-01T01:33:36.559784+0000","last_became_active":"2024-04-01T01:31:15.382320+0000","last_became_peered":"2024-04-01T01:31:15.382320+0000","last_unstale":"2024-04-01T01:33:36.559784+0000","last_undegraded":"2024-04-01T01:33:36.559784+0000","last_fullsized":"2024-04-01T01:33:36.559784+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:15:47.138024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629854+0000","last_change":"2024-04-01T01:31:16.386936+0000","last_active":"2024-04-01T01:33:20.629854+0000","last_peered":"2024-04-01T01:33:20.629854+0000","last_clean":"2024-04-01T01:33:20.629854+0000","last_became_active":"2024-04-01T01:31:16.384448+0000","last_became_peered":"2024-04-01T01:31:16.384448+0000","last_unstale":"2024-04-01T01:33:20.629854+0000","last_undegraded":"2024-04-01T01:33:20.629854+0000","last_fullsized":"2024-04-01T01:33:20.629854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:16:42.243250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.726698+0000","last_change":"2024-04-01T01:31:20.565570+0000","last_active":"2024-04-01T01:33:19.726698+0000","last_peered":"2024-04-01T01:33:19.726698+0000","last_clean":"2024-04-01T01:33:19.726698+0000","last_became_active":"2024-04-01T01:31:20.565141+0000","last_became_peered":"2024-04-01T01:31:20.565141+0000","last_unstale":"2024-04-01T01:33:19.726698+0000","last_undegraded":"2024-04-01T01:33:19.726698+0000","last_fullsized":"2024-04-01T01:33:19.726698+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T11:47:12.844970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722422+0000","last_change":"2024-04-01T01:31:15.381343+0000","last_active":"2024-04-01T01:33:19.722422+0000","last_peered":"2024-04-01T01:33:19.722422+0000","last_clean":"2024-04-01T01:33:19.722422+0000","last_became_active":"2024-04-01T01:31:15.380994+0000","last_became_peered":"2024-04-01T01:31:15.380994+0000","last_unstale":"2024-04-01T01:33:19.722422+0000","last_undegraded":"2024-04-01T01:33:19.722422+0000","last_fullsized":"2024-04-01T01:33:19.722422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:19:10.710285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838567+0000","last_change":"2024-04-01T01:31:16.383951+0000","last_active":"2024-04-01T01:33:20.838567+0000","last_peered":"2024-04-01T01:33:20.838567+0000","last_clean":"2024-04-01T01:33:20.838567+0000","last_became_active":"2024-04-01T01:31:16.383657+0000","last_became_peered":"2024-04-01T01:31:16.383657+0000","last_unstale":"2024-04-01T01:33:20.838567+0000","last_undegraded":"2024-04-01T01:33:20.838567+0000","last_fullsized":"2024-04-01T01:33:20.838567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:55:45.216479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722156+0000","last_change":"2024-04-01T01:31:20.569416+0000","last_active":"2024-04-01T01:33:19.722156+0000","last_peered":"2024-04-01T01:33:19.722156+0000","last_clean":"2024-04-01T01:33:19.722156+0000","last_became_active":"2024-04-01T01:31:20.569007+0000","last_became_peered":"2024-04-01T01:31:20.569007+0000","last_unstale":"2024-04-01T01:33:19.722156+0000","last_undegraded":"2024-04-01T01:33:19.722156+0000","last_fullsized":"2024-04-01T01:33:19.722156+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T05:25:36.770919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608610+0000","last_change":"2024-04-01T01:31:15.373444+0000","last_active":"2024-04-01T01:33:20.608610+0000","last_peered":"2024-04-01T01:33:20.608610+0000","last_clean":"2024-04-01T01:33:20.608610+0000","last_became_active":"2024-04-01T01:31:15.373094+0000","last_became_peered":"2024-04-01T01:31:15.373094+0000","last_unstale":"2024-04-01T01:33:20.608610+0000","last_undegraded":"2024-04-01T01:33:20.608610+0000","last_fullsized":"2024-04-01T01:33:20.608610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:26:27.443832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724496+0000","last_change":"2024-04-01T01:31:16.381441+0000","last_active":"2024-04-01T01:33:19.724496+0000","last_peered":"2024-04-01T01:33:19.724496+0000","last_clean":"2024-04-01T01:33:19.724496+0000","last_became_active":"2024-04-01T01:31:16.381092+0000","last_became_peered":"2024-04-01T01:31:16.381092+0000","last_unstale":"2024-04-01T01:33:19.724496+0000","last_undegraded":"2024-04-01T01:33:19.724496+0000","last_fullsized":"2024-04-01T01:33:19.724496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:47:00.229090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722053+0000","last_change":"2024-04-01T01:31:20.569672+0000","last_active":"2024-04-01T01:33:19.722053+0000","last_peered":"2024-04-01T01:33:19.722053+0000","last_clean":"2024-04-01T01:33:19.722053+0000","last_became_active":"2024-04-01T01:31:20.569276+0000","last_became_peered":"2024-04-01T01:31:20.569276+0000","last_unstale":"2024-04-01T01:33:19.722053+0000","last_undegraded":"2024-04-01T01:33:19.722053+0000","last_fullsized":"2024-04-01T01:33:19.722053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:39:03.331428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629526+0000","last_change":"2024-04-01T01:31:15.376094+0000","last_active":"2024-04-01T01:33:20.629526+0000","last_peered":"2024-04-01T01:33:20.629526+0000","last_clean":"2024-04-01T01:33:20.629526+0000","last_became_active":"2024-04-01T01:31:15.375169+0000","last_became_peered":"2024-04-01T01:31:15.375169+0000","last_unstale":"2024-04-01T01:33:20.629526+0000","last_undegraded":"2024-04-01T01:33:20.629526+0000","last_fullsized":"2024-04-01T01:33:20.629526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T13:18:20.710644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724497+0000","last_change":"2024-04-01T01:31:16.378514+0000","last_active":"2024-04-01T01:33:19.724497+0000","last_peered":"2024-04-01T01:33:19.724497+0000","last_clean":"2024-04-01T01:33:19.724497+0000","last_became_active":"2024-04-01T01:31:16.377990+0000","last_became_peered":"2024-04-01T01:31:16.377990+0000","last_unstale":"2024-04-01T01:33:19.724497+0000","last_undegraded":"2024-04-01T01:33:19.724497+0000","last_fullsized":"2024-04-01T01:33:19.724497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:27:29.829775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.721831+0000","last_change":"2024-04-01T01:31:20.567892+0000","last_active":"2024-04-01T01:33:19.721831+0000","last_peered":"2024-04-01T01:33:19.721831+0000","last_clean":"2024-04-01T01:33:19.721831+0000","last_became_active":"2024-04-01T01:31:20.567525+0000","last_became_peered":"2024-04-01T01:31:20.567525+0000","last_unstale":"2024-04-01T01:33:19.721831+0000","last_undegraded":"2024-04-01T01:33:19.721831+0000","last_fullsized":"2024-04-01T01:33:19.721831+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:15:57.828691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'5","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:35.564105+0000","last_change":"2024-04-01T01:32:19.703414+0000","last_active":"2024-04-01T01:33:35.564105+0000","last_peered":"2024-04-01T01:33:35.564105+0000","last_clean":"2024-04-01T01:33:35.564105+0000","last_became_active":"2024-04-01T01:32:19.693581+0000","last_became_peered":"2024-04-01T01:32:19.693581+0000","last_unstale":"2024-04-01T01:33:35.564105+0000","last_undegraded":"2024-04-01T01:33:35.564105+0000","last_fullsized":"2024-04-01T01:33:35.564105+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:53:17.897154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629533+0000","last_change":"2024-04-01T01:31:16.387570+0000","last_active":"2024-04-01T01:33:20.629533+0000","last_peered":"2024-04-01T01:33:20.629533+0000","last_clean":"2024-04-01T01:33:20.629533+0000","last_became_active":"2024-04-01T01:31:16.387200+0000","last_became_peered":"2024-04-01T01:31:16.387200+0000","last_unstale":"2024-04-01T01:33:20.629533+0000","last_undegraded":"2024-04-01T01:33:20.629533+0000","last_fullsized":"2024-04-01T01:33:20.629533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:52:27.813722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722104+0000","last_change":"2024-04-01T01:31:20.569668+0000","last_active":"2024-04-01T01:33:19.722104+0000","last_peered":"2024-04-01T01:33:19.722104+0000","last_clean":"2024-04-01T01:33:19.722104+0000","last_became_active":"2024-04-01T01:31:20.569148+0000","last_became_peered":"2024-04-01T01:31:20.569148+0000","last_unstale":"2024-04-01T01:33:19.722104+0000","last_undegraded":"2024-04-01T01:33:19.722104+0000","last_fullsized":"2024-04-01T01:33:19.722104+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:20:56.048052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724566+0000","last_change":"2024-04-01T01:31:15.379323+0000","last_active":"2024-04-01T01:33:19.724566+0000","last_peered":"2024-04-01T01:33:19.724566+0000","last_clean":"2024-04-01T01:33:19.724566+0000","last_became_active":"2024-04-01T01:31:15.379023+0000","last_became_peered":"2024-04-01T01:31:15.379023+0000","last_unstale":"2024-04-01T01:33:19.724566+0000","last_undegraded":"2024-04-01T01:33:19.724566+0000","last_fullsized":"2024-04-01T01:33:19.724566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:05:37.709040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725164+0000","last_change":"2024-04-01T01:31:16.387250+0000","last_active":"2024-04-01T01:33:19.725164+0000","last_peered":"2024-04-01T01:33:19.725164+0000","last_clean":"2024-04-01T01:33:19.725164+0000","last_became_active":"2024-04-01T01:31:16.386940+0000","last_became_peered":"2024-04-01T01:31:16.386940+0000","last_unstale":"2024-04-01T01:33:19.725164+0000","last_undegraded":"2024-04-01T01:33:19.725164+0000","last_fullsized":"2024-04-01T01:33:19.725164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:19:34.886738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608300+0000","last_change":"2024-04-01T01:31:20.573228+0000","last_active":"2024-04-01T01:33:20.608300+0000","last_peered":"2024-04-01T01:33:20.608300+0000","last_clean":"2024-04-01T01:33:20.608300+0000","last_became_active":"2024-04-01T01:31:20.572681+0000","last_became_peered":"2024-04-01T01:31:20.572681+0000","last_unstale":"2024-04-01T01:33:20.608300+0000","last_undegraded":"2024-04-01T01:33:20.608300+0000","last_fullsized":"2024-04-01T01:33:20.608300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T07:58:17.812966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'7","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:37.723133+0000","last_change":"2024-04-01T01:31:15.381477+0000","last_active":"2024-04-01T01:33:37.723133+0000","last_peered":"2024-04-01T01:33:37.723133+0000","last_clean":"2024-04-01T01:33:37.723133+0000","last_became_active":"2024-04-01T01:31:15.381166+0000","last_became_peered":"2024-04-01T01:31:15.381166+0000","last_unstale":"2024-04-01T01:33:37.723133+0000","last_undegraded":"2024-04-01T01:33:37.723133+0000","last_fullsized":"2024-04-01T01:33:37.723133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:55:19.029896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2038,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725492+0000","last_change":"2024-04-01T01:31:16.386234+0000","last_active":"2024-04-01T01:33:19.725492+0000","last_peered":"2024-04-01T01:33:19.725492+0000","last_clean":"2024-04-01T01:33:19.725492+0000","last_became_active":"2024-04-01T01:31:16.385876+0000","last_became_peered":"2024-04-01T01:31:16.385876+0000","last_unstale":"2024-04-01T01:33:19.725492+0000","last_undegraded":"2024-04-01T01:33:19.725492+0000","last_fullsized":"2024-04-01T01:33:19.725492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:21:34.187210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838323+0000","last_change":"2024-04-01T01:32:19.805473+0000","last_active":"2024-04-01T01:33:20.838323+0000","last_peered":"2024-04-01T01:33:20.838323+0000","last_clean":"2024-04-01T01:33:20.838323+0000","last_became_active":"2024-04-01T01:32:19.805151+0000","last_became_peered":"2024-04-01T01:32:19.805151+0000","last_unstale":"2024-04-01T01:33:20.838323+0000","last_undegraded":"2024-04-01T01:33:20.838323+0000","last_fullsized":"2024-04-01T01:33:20.838323+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T03:47:54.539407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722570+0000","last_change":"2024-04-01T01:31:15.378503+0000","last_active":"2024-04-01T01:33:19.722570+0000","last_peered":"2024-04-01T01:33:19.722570+0000","last_clean":"2024-04-01T01:33:19.722570+0000","last_became_active":"2024-04-01T01:31:15.378152+0000","last_became_peered":"2024-04-01T01:31:15.378152+0000","last_unstale":"2024-04-01T01:33:19.722570+0000","last_undegraded":"2024-04-01T01:33:19.722570+0000","last_fullsized":"2024-04-01T01:33:19.722570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:15:42.118691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.728781+0000","last_change":"2024-04-01T01:33:19.728322+0000","last_active":"2024-04-01T01:33:19.728781+0000","last_peered":"2024-04-01T01:33:19.728781+0000","last_clean":"2024-04-01T01:33:19.728781+0000","last_became_active":"2024-04-01T01:33:19.727846+0000","last_became_peered":"2024-04-01T01:33:19.727846+0000","last_unstale":"2024-04-01T01:33:19.728781+0000","last_undegraded":"2024-04-01T01:33:19.728781+0000","last_fullsized":"2024-04-01T01:33:19.728781+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:42:54.899382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722019+0000","last_change":"2024-04-01T01:31:20.569461+0000","last_active":"2024-04-01T01:33:19.722019+0000","last_peered":"2024-04-01T01:33:19.722019+0000","last_clean":"2024-04-01T01:33:19.722019+0000","last_became_active":"2024-04-01T01:31:20.569011+0000","last_became_peered":"2024-04-01T01:31:20.569011+0000","last_unstale":"2024-04-01T01:33:19.722019+0000","last_undegraded":"2024-04-01T01:33:19.722019+0000","last_fullsized":"2024-04-01T01:33:19.722019+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:19.544444+0000","last_clean_scrub_stamp":"2024-04-01T01:31:19.544444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:43:43.657065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719451+0000","last_change":"2024-04-01T01:31:15.384181+0000","last_active":"2024-04-01T01:33:19.719451+0000","last_peered":"2024-04-01T01:33:19.719451+0000","last_clean":"2024-04-01T01:33:19.719451+0000","last_became_active":"2024-04-01T01:31:15.383886+0000","last_became_peered":"2024-04-01T01:31:15.383886+0000","last_unstale":"2024-04-01T01:33:19.719451+0000","last_undegraded":"2024-04-01T01:33:19.719451+0000","last_fullsized":"2024-04-01T01:33:19.719451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:14.346661+0000","last_clean_scrub_stamp":"2024-04-01T01:31:14.346661+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T09:43:28.414423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725363+0000","last_change":"2024-04-01T01:31:16.381877+0000","last_active":"2024-04-01T01:33:19.725363+0000","last_peered":"2024-04-01T01:33:19.725363+0000","last_clean":"2024-04-01T01:33:19.725363+0000","last_became_active":"2024-04-01T01:31:16.381558+0000","last_became_peered":"2024-04-01T01:31:16.381558+0000","last_unstale":"2024-04-01T01:33:19.725363+0000","last_undegraded":"2024-04-01T01:33:19.725363+0000","last_fullsized":"2024-04-01T01:33:19.725363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T01:48:24.120910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.730654+0000","last_change":"2024-04-01T01:33:19.730079+0000","last_active":"2024-04-01T01:33:19.730654+0000","last_peered":"2024-04-01T01:33:19.730654+0000","last_clean":"2024-04-01T01:33:19.730654+0000","last_became_active":"2024-04-01T01:33:19.729631+0000","last_became_peered":"2024-04-01T01:33:19.729631+0000","last_unstale":"2024-04-01T01:33:19.730654+0000","last_undegraded":"2024-04-01T01:33:19.730654+0000","last_fullsized":"2024-04-01T01:33:19.730654+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:49:30.299413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727472+0000","last_change":"2024-04-01T01:31:16.384872+0000","last_active":"2024-04-01T01:33:19.727472+0000","last_peered":"2024-04-01T01:33:19.727472+0000","last_clean":"2024-04-01T01:33:19.727472+0000","last_became_active":"2024-04-01T01:31:16.384406+0000","last_became_peered":"2024-04-01T01:31:16.384406+0000","last_unstale":"2024-04-01T01:33:19.727472+0000","last_undegraded":"2024-04-01T01:33:19.727472+0000","last_fullsized":"2024-04-01T01:33:19.727472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:13:36.567357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727117+0000","last_change":"2024-04-01T01:31:16.384561+0000","last_active":"2024-04-01T01:33:19.727117+0000","last_peered":"2024-04-01T01:33:19.727117+0000","last_clean":"2024-04-01T01:33:19.727117+0000","last_became_active":"2024-04-01T01:31:16.384221+0000","last_became_peered":"2024-04-01T01:31:16.384221+0000","last_unstale":"2024-04-01T01:33:19.727117+0000","last_undegraded":"2024-04-01T01:33:19.727117+0000","last_fullsized":"2024-04-01T01:33:19.727117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T10:17:56.490000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.838506+0000","last_change":"2024-04-01T01:31:16.385464+0000","last_active":"2024-04-01T01:33:20.838506+0000","last_peered":"2024-04-01T01:33:20.838506+0000","last_clean":"2024-04-01T01:33:20.838506+0000","last_became_active":"2024-04-01T01:31:16.385159+0000","last_became_peered":"2024-04-01T01:31:16.385159+0000","last_unstale":"2024-04-01T01:33:20.838506+0000","last_undegraded":"2024-04-01T01:33:20.838506+0000","last_fullsized":"2024-04-01T01:33:20.838506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:33:53.852957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719377+0000","last_change":"2024-04-01T01:31:16.385688+0000","last_active":"2024-04-01T01:33:19.719377+0000","last_peered":"2024-04-01T01:33:19.719377+0000","last_clean":"2024-04-01T01:33:19.719377+0000","last_became_active":"2024-04-01T01:31:16.385381+0000","last_became_peered":"2024-04-01T01:31:16.385381+0000","last_unstale":"2024-04-01T01:33:19.719377+0000","last_undegraded":"2024-04-01T01:33:19.719377+0000","last_fullsized":"2024-04-01T01:33:19.719377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:39:36.810766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.725031+0000","last_change":"2024-04-01T01:31:16.379976+0000","last_active":"2024-04-01T01:33:19.725031+0000","last_peered":"2024-04-01T01:33:19.725031+0000","last_clean":"2024-04-01T01:33:19.725031+0000","last_became_active":"2024-04-01T01:31:16.379553+0000","last_became_peered":"2024-04-01T01:31:16.379553+0000","last_unstale":"2024-04-01T01:33:19.725031+0000","last_undegraded":"2024-04-01T01:33:19.725031+0000","last_fullsized":"2024-04-01T01:33:19.725031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:48:23.555723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.727822+0000","last_change":"2024-04-01T01:31:16.383995+0000","last_active":"2024-04-01T01:33:19.727822+0000","last_peered":"2024-04-01T01:33:19.727822+0000","last_clean":"2024-04-01T01:33:19.727822+0000","last_became_active":"2024-04-01T01:31:16.383673+0000","last_became_peered":"2024-04-01T01:31:16.383673+0000","last_unstale":"2024-04-01T01:33:19.727822+0000","last_undegraded":"2024-04-01T01:33:19.727822+0000","last_fullsized":"2024-04-01T01:33:19.727822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:03:16.192438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629879+0000","last_change":"2024-04-01T01:31:16.389969+0000","last_active":"2024-04-01T01:33:20.629879+0000","last_peered":"2024-04-01T01:33:20.629879+0000","last_clean":"2024-04-01T01:33:20.629879+0000","last_became_active":"2024-04-01T01:31:16.389640+0000","last_became_peered":"2024-04-01T01:31:16.389640+0000","last_unstale":"2024-04-01T01:33:20.629879+0000","last_undegraded":"2024-04-01T01:33:20.629879+0000","last_fullsized":"2024-04-01T01:33:20.629879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T06:35:15.226960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.722223+0000","last_change":"2024-04-01T01:31:16.382962+0000","last_active":"2024-04-01T01:33:19.722223+0000","last_peered":"2024-04-01T01:33:19.722223+0000","last_clean":"2024-04-01T01:33:19.722223+0000","last_became_active":"2024-04-01T01:31:16.382653+0000","last_became_peered":"2024-04-01T01:31:16.382653+0000","last_unstale":"2024-04-01T01:33:19.722223+0000","last_undegraded":"2024-04-01T01:33:19.722223+0000","last_fullsized":"2024-04-01T01:33:19.722223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T02:58:53.054884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.629538+0000","last_change":"2024-04-01T01:31:16.382997+0000","last_active":"2024-04-01T01:33:20.629538+0000","last_peered":"2024-04-01T01:33:20.629538+0000","last_clean":"2024-04-01T01:33:20.629538+0000","last_became_active":"2024-04-01T01:31:16.380180+0000","last_became_peered":"2024-04-01T01:31:16.380180+0000","last_unstale":"2024-04-01T01:33:20.629538+0000","last_undegraded":"2024-04-01T01:33:20.629538+0000","last_fullsized":"2024-04-01T01:33:20.629538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T04:29:51.990395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:20.608760+0000","last_change":"2024-04-01T01:31:16.382948+0000","last_active":"2024-04-01T01:33:20.608760+0000","last_peered":"2024-04-01T01:33:20.608760+0000","last_clean":"2024-04-01T01:33:20.608760+0000","last_became_active":"2024-04-01T01:31:16.382134+0000","last_became_peered":"2024-04-01T01:31:16.382134+0000","last_unstale":"2024-04-01T01:33:20.608760+0000","last_undegraded":"2024-04-01T01:33:20.608760+0000","last_fullsized":"2024-04-01T01:33:20.608760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:24:08.515063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.719262+0000","last_change":"2024-04-01T01:31:16.378841+0000","last_active":"2024-04-01T01:33:19.719262+0000","last_peered":"2024-04-01T01:33:19.719262+0000","last_clean":"2024-04-01T01:33:19.719262+0000","last_became_active":"2024-04-01T01:31:16.378501+0000","last_became_peered":"2024-04-01T01:31:16.378501+0000","last_unstale":"2024-04-01T01:33:19.719262+0000","last_undegraded":"2024-04-01T01:33:19.719262+0000","last_fullsized":"2024-04-01T01:33:19.719262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T08:50:20.881852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-01T01:33:19.724900+0000","last_change":"2024-04-01T01:31:16.380979+0000","last_active":"2024-04-01T01:33:19.724900+0000","last_peered":"2024-04-01T01:33:19.724900+0000","last_clean":"2024-04-01T01:33:19.724900+0000","last_became_active":"2024-04-01T01:31:16.380625+0000","last_became_peered":"2024-04-01T01:31:16.380625+0000","last_unstale":"2024-04-01T01:33:19.724900+0000","last_undegraded":"2024-04-01T01:33:19.724900+0000","last_fullsized":"2024-04-01T01:33:19.724900+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-01T01:31:15.353768+0000","last_clean_scrub_stamp":"2024-04-01T01:31:15.353768+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-02T12:09:03.246239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74258,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1247,"num_read_kb":710,"num_write":274,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1125,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":585728,"data_stored":231402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1073152,"data_stored":520978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37056,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708096,"statfs":{"total":95995035648,"available":95957090304,"internally_reserved":0,"allocated":1761280,"data_stored":1172296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":524415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070521,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":524681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36980,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708172,"statfs":{"total":95995035648,"available":95957168128,"internally_reserved":0,"allocated":1683456,"data_stored":1119689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":516691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1138688,"data_stored":586508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":517756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1146880,"data_stored":581010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":515531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":522537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28664,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716488,"statfs":{"total":95995035648,"available":95965683712,"internally_reserved":0,"allocated":1687552,"data_stored":1109298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":71005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-01T01:33:42.515 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:33:42.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.653+0000 7efd0e69d640 1 -- 172.21.15.70:0/3564962814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd08101860 msgr2=0x7efd08101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:42.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.653+0000 7efd0e69d640 1 --2- 172.21.15.70:0/3564962814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd08101860 0x7efd08101ce0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009900 tx=0x7efcfc02bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:42.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.653+0000 7efd0e69d640 1 -- 172.21.15.70:0/3564962814 shutdown_connections 2024-04-01T01:33:42.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.653+0000 7efd0e69d640 1 --2- 172.21.15.70:0/3564962814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd08102220 0x7efd08108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.653+0000 7efd0e69d640 1 --2- 172.21.15.70:0/3564962814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd08101860 0x7efd08101ce0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.653+0000 7efd0e69d640 1 --2- 172.21.15.70:0/3564962814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd08100670 0x7efd08100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.653+0000 7efd0e69d640 1 -- 172.21.15.70:0/3564962814 >> 172.21.15.70:0/3564962814 conn(0x7efd080fbbe0 msgr2=0x7efd080fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:42.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.653+0000 7efd0e69d640 1 -- 172.21.15.70:0/3564962814 shutdown_connections 2024-04-01T01:33:42.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.653+0000 7efd0e69d640 1 -- 172.21.15.70:0/3564962814 wait complete. 2024-04-01T01:33:42.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.654+0000 7efd0e69d640 1 Processor -- start 2024-04-01T01:33:42.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.654+0000 7efd0e69d640 1 -- start start 2024-04-01T01:33:42.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.654+0000 7efd0e69d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd08100670 0x7efd08196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:42.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.654+0000 7efd0e69d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd08101860 0x7efd08196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:42.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.654+0000 7efd0e69d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd08102220 0x7efd0819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:42.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.654+0000 7efd0e69d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efd0810c1b0 con 0x7efd08100670 2024-04-01T01:33:42.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.654+0000 7efd0e69d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efd0810c030 con 0x7efd08102220 2024-04-01T01:33:42.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.654+0000 7efd0e69d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efd0810c330 con 0x7efd08101860 2024-04-01T01:33:42.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.654+0000 7efd07fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd08100670 0x7efd08196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.654+0000 7efd07fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd08100670 0x7efd08196890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55472/0 (socket says 172.21.15.70:55472) 2024-04-01T01:33:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.654+0000 7efd07fff640 1 -- 172.21.15.70:0/3427592543 learned_addr learned my addr 172.21.15.70:0/3427592543 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.654+0000 7efd077fe640 1 --2- 172.21.15.70:0/3427592543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd08101860 0x7efd08196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.654+0000 7efd0cc3a640 1 --2- 172.21.15.70:0/3427592543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd08102220 0x7efd0819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.655+0000 7efd07fff640 1 -- 172.21.15.70:0/3427592543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd08101860 msgr2=0x7efd08196dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.655+0000 7efd07fff640 1 --2- 172.21.15.70:0/3427592543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd08101860 0x7efd08196dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.655+0000 7efd07fff640 1 -- 172.21.15.70:0/3427592543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd08102220 msgr2=0x7efd0819de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.655+0000 7efd07fff640 1 --2- 172.21.15.70:0/3427592543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd08102220 0x7efd0819de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.655+0000 7efd07fff640 1 -- 172.21.15.70:0/3427592543 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0819e4b0 con 0x7efd08100670 2024-04-01T01:33:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.655+0000 7efd07fff640 1 --2- 172.21.15.70:0/3427592543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd08100670 0x7efd08196890 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7efcf800e9c0 tx=0x7efcf800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.655+0000 7efd057fa640 1 -- 172.21.15.70:0/3427592543 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf800cd80 con 0x7efd08100670 2024-04-01T01:33:42.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.655+0000 7efd057fa640 1 -- 172.21.15.70:0/3427592543 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efcf80035f0 con 0x7efd08100670 2024-04-01T01:33:42.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.655+0000 7efd0e69d640 1 -- 172.21.15.70:0/3427592543 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0819e7a0 con 0x7efd08100670 2024-04-01T01:33:42.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.655+0000 7efd057fa640 1 -- 172.21.15.70:0/3427592543 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf8010690 con 0x7efd08100670 2024-04-01T01:33:42.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.655+0000 7efd0e69d640 1 -- 172.21.15.70:0/3427592543 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd0819ecc0 con 0x7efd08100670 2024-04-01T01:33:42.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.656+0000 7efd0e69d640 1 -- 172.21.15.70:0/3427592543 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd08107a10 con 0x7efd08100670 2024-04-01T01:33:42.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.657+0000 7efd057fa640 1 -- 172.21.15.70:0/3427592543 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7efcf80037a0 con 0x7efd08100670 2024-04-01T01:33:42.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.657+0000 7efd057fa640 1 --2- 172.21.15.70:0/3427592543 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efcf00a2530 0x7efcf00a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:42.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.658+0000 7efd077fe640 1 --2- 172.21.15.70:0/3427592543 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efcf00a2530 0x7efcf00a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:42.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.658+0000 7efd057fa640 1 -- 172.21.15.70:0/3427592543 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7efcf80cf000 con 0x7efd08100670 2024-04-01T01:33:42.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.658+0000 7efd077fe640 1 --2- 172.21.15.70:0/3427592543 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efcf00a2530 0x7efcf00a49f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009230 tx=0x7efcfc005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:42.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.662+0000 7efd057fa640 1 -- 172.21.15.70:0/3427592543 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efcf809bc90 con 0x7efd08100670 2024-04-01T01:33:42.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.840+0000 7efd0e69d640 1 -- 172.21.15.70:0/3427592543 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd0802d420 con 0x7efd08100670 2024-04-01T01:33:42.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.842+0000 7efd057fa640 1 -- 172.21.15.70:0/3427592543 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24781 (secure 0 0 0) 0x7efcf809b590 con 0x7efd08100670 2024-04-01T01:33:42.843 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:42.843 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":93,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:33:19.716245+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:33:42.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.844+0000 7efd0e69d640 1 -- 172.21.15.70:0/3427592543 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efcf00a2530 msgr2=0x7efcf00a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:42.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.844+0000 7efd0e69d640 1 --2- 172.21.15.70:0/3427592543 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efcf00a2530 0x7efcf00a49f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009230 tx=0x7efcfc005fd0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.844+0000 7efd0e69d640 1 -- 172.21.15.70:0/3427592543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd08100670 msgr2=0x7efd08196890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:42.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.844+0000 7efd0e69d640 1 --2- 172.21.15.70:0/3427592543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd08100670 0x7efd08196890 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7efcf800e9c0 tx=0x7efcf800ee80 comp rx=0 tx=0).stop 2024-04-01T01:33:42.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.844+0000 7efd0e69d640 1 -- 172.21.15.70:0/3427592543 shutdown_connections 2024-04-01T01:33:42.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.844+0000 7efd0e69d640 1 --2- 172.21.15.70:0/3427592543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd08102220 0x7efd0819de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.844+0000 7efd0e69d640 1 --2- 172.21.15.70:0/3427592543 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efcf00a2530 0x7efcf00a49f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.844+0000 7efd0e69d640 1 --2- 172.21.15.70:0/3427592543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd08101860 0x7efd08196dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.844+0000 7efd0e69d640 1 --2- 172.21.15.70:0/3427592543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd08100670 0x7efd08196890 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.844+0000 7efd0e69d640 1 -- 172.21.15.70:0/3427592543 >> 172.21.15.70:0/3427592543 conn(0x7efd080fbbe0 msgr2=0x7efd08104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:42.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.844+0000 7efd0e69d640 1 -- 172.21.15.70:0/3427592543 shutdown_connections 2024-04-01T01:33:42.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.844+0000 7efd0e69d640 1 -- 172.21.15.70:0/3427592543 wait complete. 2024-04-01T01:33:42.856 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-01T01:26:42.809779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-01T01:31:13.612369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-01T01:31:14.910162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-01T01:31:18.783179+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-01T01:33:42.857 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-01T01:33:42.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.961+0000 7f0a1e239640 1 -- 172.21.15.70:0/3610667016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a18101840 msgr2=0x7f0a18101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:42.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.961+0000 7f0a1e239640 1 --2- 172.21.15.70:0/3610667016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a18101840 0x7f0a18101cc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c009860 tx=0x7f0a0c02bb70 comp rx=0 tx=0).stop 2024-04-01T01:33:42.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.961+0000 7f0a1e239640 1 -- 172.21.15.70:0/3610667016 shutdown_connections 2024-04-01T01:33:42.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.961+0000 7f0a1e239640 1 --2- 172.21.15.70:0/3610667016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a18102200 0x7f0a18108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.961+0000 7f0a1e239640 1 --2- 172.21.15.70:0/3610667016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a18101840 0x7f0a18101cc0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.961+0000 7f0a1e239640 1 --2- 172.21.15.70:0/3610667016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a18100650 0x7f0a18100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.961+0000 7f0a1e239640 1 -- 172.21.15.70:0/3610667016 >> 172.21.15.70:0/3610667016 conn(0x7f0a180fbbc0 msgr2=0x7f0a180fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:42.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.961+0000 7f0a1e239640 1 -- 172.21.15.70:0/3610667016 shutdown_connections 2024-04-01T01:33:42.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.961+0000 7f0a1e239640 1 -- 172.21.15.70:0/3610667016 wait complete. 2024-04-01T01:33:42.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.962+0000 7f0a1e239640 1 Processor -- start 2024-04-01T01:33:42.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.962+0000 7f0a1e239640 1 -- start start 2024-04-01T01:33:42.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.962+0000 7f0a1e239640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a18100650 0x7f0a18069480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:42.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.962+0000 7f0a1e239640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a18101840 0x7f0a1819c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:42.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.962+0000 7f0a1e239640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a18102200 0x7f0a1819ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:42.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.962+0000 7f0a1e239640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1810c190 con 0x7f0a18102200 2024-04-01T01:33:42.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.962+0000 7f0a1e239640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1810c010 con 0x7f0a18100650 2024-04-01T01:33:42.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.962+0000 7f0a1e239640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1810c310 con 0x7f0a18101840 2024-04-01T01:33:42.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.962+0000 7f0a17fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a18102200 0x7f0a1819ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:42.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.962+0000 7f0a177fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a18100650 0x7f0a18069480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:42.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.962+0000 7f0a17fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a18102200 0x7f0a1819ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55502/0 (socket says 172.21.15.70:55502) 2024-04-01T01:33:42.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.962+0000 7f0a17fff640 1 -- 172.21.15.70:0/3547644391 learned_addr learned my addr 172.21.15.70:0/3547644391 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:42.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.962+0000 7f0a16ffd640 1 --2- 172.21.15.70:0/3547644391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a18101840 0x7f0a1819c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:42.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.963+0000 7f0a177fe640 1 -- 172.21.15.70:0/3547644391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a18101840 msgr2=0x7f0a1819c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:42.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.963+0000 7f0a177fe640 1 --2- 172.21.15.70:0/3547644391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a18101840 0x7f0a1819c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.963+0000 7f0a177fe640 1 -- 172.21.15.70:0/3547644391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a18102200 msgr2=0x7f0a1819ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:42.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.963+0000 7f0a177fe640 1 --2- 172.21.15.70:0/3547644391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a18102200 0x7f0a1819ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:42.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.963+0000 7f0a177fe640 1 -- 172.21.15.70:0/3547644391 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a1819f590 con 0x7f0a18100650 2024-04-01T01:33:42.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.963+0000 7f0a177fe640 1 --2- 172.21.15.70:0/3547644391 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a18100650 0x7f0a18069480 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0a181016a0 tx=0x7f0a0800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:42.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.963+0000 7f0a14ff9640 1 -- 172.21.15.70:0/3547644391 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08004440 con 0x7f0a18100650 2024-04-01T01:33:42.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.963+0000 7f0a14ff9640 1 -- 172.21.15.70:0/3547644391 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a0800cd70 con 0x7f0a18100650 2024-04-01T01:33:42.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.963+0000 7f0a1e239640 1 -- 172.21.15.70:0/3547644391 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a1819f880 con 0x7f0a18100650 2024-04-01T01:33:42.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.963+0000 7f0a14ff9640 1 -- 172.21.15.70:0/3547644391 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08019480 con 0x7f0a18100650 2024-04-01T01:33:42.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.963+0000 7f0a1e239640 1 -- 172.21.15.70:0/3547644391 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a1819fda0 con 0x7f0a18100650 2024-04-01T01:33:42.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.964+0000 7f0a1e239640 1 -- 172.21.15.70:0/3547644391 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a18107990 con 0x7f0a18100650 2024-04-01T01:33:42.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.965+0000 7f0a14ff9640 1 -- 172.21.15.70:0/3547644391 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f0a08003710 con 0x7f0a18100650 2024-04-01T01:33:42.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.965+0000 7f0a14ff9640 1 --2- 172.21.15.70:0/3547644391 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a000a2530 0x7f0a000a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:42.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.966+0000 7f0a16ffd640 1 --2- 172.21.15.70:0/3547644391 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a000a2530 0x7f0a000a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:42.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.966+0000 7f0a14ff9640 1 -- 172.21.15.70:0/3547644391 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0a080cf050 con 0x7f0a18100650 2024-04-01T01:33:42.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.966+0000 7f0a16ffd640 1 --2- 172.21.15.70:0/3547644391 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a000a2530 0x7f0a000a49f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c02c180 tx=0x7f0a0c034980 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:42.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:42.970+0000 7f0a14ff9640 1 -- 172.21.15.70:0/3547644391 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0a08009510 con 0x7f0a18100650 2024-04-01T01:33:43.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.142+0000 7f0a1e239640 1 -- 172.21.15.70:0/3547644391 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0a1802d420 con 0x7f0a18100650 2024-04-01T01:33:43.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.142+0000 7f0a14ff9640 1 -- 172.21.15.70:0/3547644391 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f0a0809bce0 con 0x7f0a18100650 2024-04-01T01:33:43.143 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-01T01:33:43.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.145+0000 7f0a1e239640 1 -- 172.21.15.70:0/3547644391 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a000a2530 msgr2=0x7f0a000a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:43.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.145+0000 7f0a1e239640 1 --2- 172.21.15.70:0/3547644391 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a000a2530 0x7f0a000a49f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c02c180 tx=0x7f0a0c034980 comp rx=0 tx=0).stop 2024-04-01T01:33:43.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.145+0000 7f0a1e239640 1 -- 172.21.15.70:0/3547644391 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a18100650 msgr2=0x7f0a18069480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:43.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.145+0000 7f0a1e239640 1 --2- 172.21.15.70:0/3547644391 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a18100650 0x7f0a18069480 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0a181016a0 tx=0x7f0a0800ee50 comp rx=0 tx=0).stop 2024-04-01T01:33:43.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.145+0000 7f0a1e239640 1 -- 172.21.15.70:0/3547644391 shutdown_connections 2024-04-01T01:33:43.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.145+0000 7f0a1e239640 1 --2- 172.21.15.70:0/3547644391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a18102200 0x7f0a1819ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.145+0000 7f0a1e239640 1 --2- 172.21.15.70:0/3547644391 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a000a2530 0x7f0a000a49f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.145+0000 7f0a1e239640 1 --2- 172.21.15.70:0/3547644391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a18101840 0x7f0a1819c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.145+0000 7f0a1e239640 1 --2- 172.21.15.70:0/3547644391 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a18100650 0x7f0a18069480 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.145+0000 7f0a1e239640 1 -- 172.21.15.70:0/3547644391 >> 172.21.15.70:0/3547644391 conn(0x7f0a180fbbc0 msgr2=0x7f0a18104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:43.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.145+0000 7f0a1e239640 1 -- 172.21.15.70:0/3547644391 shutdown_connections 2024-04-01T01:33:43.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.145+0000 7f0a1e239640 1 -- 172.21.15.70:0/3547644391 wait complete. 2024-04-01T01:33:43.156 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-01T01:33:43.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.263+0000 7f0d40d79640 1 -- 172.21.15.70:0/2725020395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d3c100670 msgr2=0x7f0d3c100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:43.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.263+0000 7f0d40d79640 1 --2- 172.21.15.70:0/2725020395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d3c100670 0x7f0d3c100a90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c009900 tx=0x7f0d2c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:43.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.263+0000 7f0d40d79640 1 -- 172.21.15.70:0/2725020395 shutdown_connections 2024-04-01T01:33:43.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.263+0000 7f0d40d79640 1 --2- 172.21.15.70:0/2725020395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d3c102220 0x7f0d3c108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.263+0000 7f0d40d79640 1 --2- 172.21.15.70:0/2725020395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d3c101860 0x7f0d3c101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.263+0000 7f0d40d79640 1 --2- 172.21.15.70:0/2725020395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d3c100670 0x7f0d3c100a90 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.263+0000 7f0d40d79640 1 -- 172.21.15.70:0/2725020395 >> 172.21.15.70:0/2725020395 conn(0x7f0d3c0fbc00 msgr2=0x7f0d3c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:43.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.263+0000 7f0d40d79640 1 -- 172.21.15.70:0/2725020395 shutdown_connections 2024-04-01T01:33:43.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.263+0000 7f0d40d79640 1 -- 172.21.15.70:0/2725020395 wait complete. 2024-04-01T01:33:43.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d40d79640 1 Processor -- start 2024-04-01T01:33:43.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d40d79640 1 -- start start 2024-04-01T01:33:43.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d40d79640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d3c100670 0x7f0d3c196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:43.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d40d79640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d3c101860 0x7f0d3c196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:43.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d40d79640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d3c102220 0x7f0d3c19ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:43.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d40d79640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c10c190 con 0x7f0d3c100670 2024-04-01T01:33:43.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d40d79640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c10c010 con 0x7f0d3c102220 2024-04-01T01:33:43.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d40d79640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c10c310 con 0x7f0d3c101860 2024-04-01T01:33:43.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d3a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d3c100670 0x7f0d3c196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:43.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d3a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d3c100670 0x7f0d3c196920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55534/0 (socket says 172.21.15.70:55534) 2024-04-01T01:33:43.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d3a59c640 1 -- 172.21.15.70:0/39470284 learned_addr learned my addr 172.21.15.70:0/39470284 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:43.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d3ad9d640 1 --2- 172.21.15.70:0/39470284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d3c102220 0x7f0d3c19ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:43.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d33fff640 1 --2- 172.21.15.70:0/39470284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d3c101860 0x7f0d3c196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:43.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d3a59c640 1 -- 172.21.15.70:0/39470284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d3c101860 msgr2=0x7f0d3c196e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:43.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d3a59c640 1 --2- 172.21.15.70:0/39470284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d3c101860 0x7f0d3c196e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d3a59c640 1 -- 172.21.15.70:0/39470284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d3c102220 msgr2=0x7f0d3c19ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:43.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d3a59c640 1 --2- 172.21.15.70:0/39470284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d3c102220 0x7f0d3c19ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.264+0000 7f0d3a59c640 1 -- 172.21.15.70:0/39470284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d3c19e540 con 0x7f0d3c100670 2024-04-01T01:33:43.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.265+0000 7f0d3a59c640 1 --2- 172.21.15.70:0/39470284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d3c100670 0x7f0d3c196920 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c000c00 tx=0x7f0d2c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:43.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.265+0000 7f0d337fe640 1 -- 172.21.15.70:0/39470284 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c043020 con 0x7f0d3c100670 2024-04-01T01:33:43.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.265+0000 7f0d337fe640 1 -- 172.21.15.70:0/39470284 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d2c004350 con 0x7f0d3c100670 2024-04-01T01:33:43.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.265+0000 7f0d40d79640 1 -- 172.21.15.70:0/39470284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d3c19e7d0 con 0x7f0d3c100670 2024-04-01T01:33:43.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.265+0000 7f0d337fe640 1 -- 172.21.15.70:0/39470284 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c003b10 con 0x7f0d3c100670 2024-04-01T01:33:43.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.265+0000 7f0d40d79640 1 -- 172.21.15.70:0/39470284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d3c19ed20 con 0x7f0d3c100670 2024-04-01T01:33:43.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.266+0000 7f0d40d79640 1 -- 172.21.15.70:0/39470284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d3c107a50 con 0x7f0d3c100670 2024-04-01T01:33:43.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.267+0000 7f0d337fe640 1 -- 172.21.15.70:0/39470284 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f0d2c046020 con 0x7f0d3c100670 2024-04-01T01:33:43.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.267+0000 7f0d337fe640 1 --2- 172.21.15.70:0/39470284 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0d280a2530 0x7f0d280a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:43.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.268+0000 7f0d337fe640 1 -- 172.21.15.70:0/39470284 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0d2c0f0c70 con 0x7f0d3c100670 2024-04-01T01:33:43.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.268+0000 7f0d33fff640 1 --2- 172.21.15.70:0/39470284 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0d280a2530 0x7f0d280a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:43.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.268+0000 7f0d33fff640 1 --2- 172.21.15.70:0/39470284 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0d280a2530 0x7f0d280a49f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c197e30 tx=0x7f0d24005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:43.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.272+0000 7f0d337fe640 1 -- 172.21.15.70:0/39470284 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0d2c04d020 con 0x7f0d3c100670 2024-04-01T01:33:43.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.447+0000 7f0d40d79640 1 -- 172.21.15.70:0/39470284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0d3c062350 con 0x7f0d3c100670 2024-04-01T01:33:43.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.447+0000 7f0d337fe640 1 -- 172.21.15.70:0/39470284 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f0d2c0bda10 con 0x7f0d3c100670 2024-04-01T01:33:43.448 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:33:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.449+0000 7f0d40d79640 1 -- 172.21.15.70:0/39470284 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0d280a2530 msgr2=0x7f0d280a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.449+0000 7f0d40d79640 1 --2- 172.21.15.70:0/39470284 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0d280a2530 0x7f0d280a49f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c197e30 tx=0x7f0d24005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.449+0000 7f0d40d79640 1 -- 172.21.15.70:0/39470284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d3c100670 msgr2=0x7f0d3c196920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.449+0000 7f0d40d79640 1 --2- 172.21.15.70:0/39470284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d3c100670 0x7f0d3c196920 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c000c00 tx=0x7f0d2c005dc0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.449+0000 7f0d40d79640 1 -- 172.21.15.70:0/39470284 shutdown_connections 2024-04-01T01:33:43.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.449+0000 7f0d40d79640 1 --2- 172.21.15.70:0/39470284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d3c102220 0x7f0d3c19ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.449+0000 7f0d40d79640 1 --2- 172.21.15.70:0/39470284 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0d280a2530 0x7f0d280a49f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.449+0000 7f0d40d79640 1 --2- 172.21.15.70:0/39470284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d3c101860 0x7f0d3c196e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.449+0000 7f0d40d79640 1 --2- 172.21.15.70:0/39470284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d3c100670 0x7f0d3c196920 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.449+0000 7f0d40d79640 1 -- 172.21.15.70:0/39470284 >> 172.21.15.70:0/39470284 conn(0x7f0d3c0fbc00 msgr2=0x7f0d3c104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:43.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.449+0000 7f0d40d79640 1 -- 172.21.15.70:0/39470284 shutdown_connections 2024-04-01T01:33:43.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.449+0000 7f0d40d79640 1 -- 172.21.15.70:0/39470284 wait complete. 2024-04-01T01:33:43.461 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-01T01:33:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.564+0000 7f0b382c0640 1 -- 172.21.15.70:0/2541268809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b301017e0 msgr2=0x7f0b30101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.564+0000 7f0b382c0640 1 --2- 172.21.15.70:0/2541268809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b301017e0 0x7f0b30101c00 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0b20009ca0 tx=0x7f0b2002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.564+0000 7f0b382c0640 1 -- 172.21.15.70:0/2541268809 shutdown_connections 2024-04-01T01:33:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.564+0000 7f0b382c0640 1 --2- 172.21.15.70:0/2541268809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b30103390 0x7f0b30109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.564+0000 7f0b382c0640 1 --2- 172.21.15.70:0/2541268809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b301029d0 0x7f0b30102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.564+0000 7f0b382c0640 1 --2- 172.21.15.70:0/2541268809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b301017e0 0x7f0b30101c00 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.564+0000 7f0b382c0640 1 -- 172.21.15.70:0/2541268809 >> 172.21.15.70:0/2541268809 conn(0x7f0b300fcd70 msgr2=0x7f0b300ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:43.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.565+0000 7f0b382c0640 1 -- 172.21.15.70:0/2541268809 shutdown_connections 2024-04-01T01:33:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.565+0000 7f0b382c0640 1 -- 172.21.15.70:0/2541268809 wait complete. 2024-04-01T01:33:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.565+0000 7f0b382c0640 1 Processor -- start 2024-04-01T01:33:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.565+0000 7f0b382c0640 1 -- start start 2024-04-01T01:33:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.565+0000 7f0b382c0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b301017e0 0x7f0b30197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.565+0000 7f0b382c0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b301029d0 0x7f0b30197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.565+0000 7f0b382c0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b30103390 0x7f0b3019ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.565+0000 7f0b382c0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3010d2a0 con 0x7f0b30103390 2024-04-01T01:33:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.565+0000 7f0b382c0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3010d120 con 0x7f0b301029d0 2024-04-01T01:33:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.565+0000 7f0b382c0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3010d420 con 0x7f0b301017e0 2024-04-01T01:33:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.565+0000 7f0b3685d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b30103390 0x7f0b3019ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.565+0000 7f0b3605c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b301017e0 0x7f0b30197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.565+0000 7f0b3685d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b30103390 0x7f0b3019ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55554/0 (socket says 172.21.15.70:55554) 2024-04-01T01:33:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.565+0000 7f0b3605c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b301017e0 0x7f0b30197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41092/0 (socket says 172.21.15.70:41092) 2024-04-01T01:33:43.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.565+0000 7f0b3685d640 1 -- 172.21.15.70:0/2022537442 learned_addr learned my addr 172.21.15.70:0/2022537442 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.565+0000 7f0b3585b640 1 --2- 172.21.15.70:0/2022537442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b301029d0 0x7f0b30197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.566+0000 7f0b3685d640 1 -- 172.21.15.70:0/2022537442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b301017e0 msgr2=0x7f0b30197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.566+0000 7f0b3685d640 1 --2- 172.21.15.70:0/2022537442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b301017e0 0x7f0b30197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.566+0000 7f0b3685d640 1 -- 172.21.15.70:0/2022537442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b301029d0 msgr2=0x7f0b30197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.566+0000 7f0b3685d640 1 --2- 172.21.15.70:0/2022537442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b301029d0 0x7f0b30197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.566+0000 7f0b3685d640 1 -- 172.21.15.70:0/2022537442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b3019f590 con 0x7f0b30103390 2024-04-01T01:33:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.566+0000 7f0b3605c640 1 --2- 172.21.15.70:0/2022537442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b301017e0 0x7f0b30197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.566+0000 7f0b3585b640 1 --2- 172.21.15.70:0/2022537442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b301029d0 0x7f0b30197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.566+0000 7f0b3685d640 1 --2- 172.21.15.70:0/2022537442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b30103390 0x7f0b3019ef20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800e970 tx=0x7f0b2800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.566+0000 7f0b2f7fe640 1 -- 172.21.15.70:0/2022537442 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2800ccc0 con 0x7f0b30103390 2024-04-01T01:33:43.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.566+0000 7f0b2f7fe640 1 -- 172.21.15.70:0/2022537442 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b2800ce60 con 0x7f0b30103390 2024-04-01T01:33:43.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.566+0000 7f0b382c0640 1 -- 172.21.15.70:0/2022537442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b300739e0 con 0x7f0b30103390 2024-04-01T01:33:43.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.566+0000 7f0b2f7fe640 1 -- 172.21.15.70:0/2022537442 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b28010430 con 0x7f0b30103390 2024-04-01T01:33:43.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.566+0000 7f0b382c0640 1 -- 172.21.15.70:0/2022537442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b30073f30 con 0x7f0b30103390 2024-04-01T01:33:43.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.567+0000 7f0b382c0640 1 -- 172.21.15.70:0/2022537442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b30108ba0 con 0x7f0b30103390 2024-04-01T01:33:43.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.568+0000 7f0b2f7fe640 1 -- 172.21.15.70:0/2022537442 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f0b28004f70 con 0x7f0b30103390 2024-04-01T01:33:43.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.568+0000 7f0b2f7fe640 1 --2- 172.21.15.70:0/2022537442 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0b180a2530 0x7f0b180a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:43.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.569+0000 7f0b3605c640 1 --2- 172.21.15.70:0/2022537442 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0b180a2530 0x7f0b180a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:43.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.569+0000 7f0b2f7fe640 1 -- 172.21.15.70:0/2022537442 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0b280cf010 con 0x7f0b30103390 2024-04-01T01:33:43.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.569+0000 7f0b3605c640 1 --2- 172.21.15.70:0/2022537442 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0b180a2530 0x7f0b180a49f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0b20032380 tx=0x7f0b2003e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:43.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.573+0000 7f0b2f7fe640 1 -- 172.21.15.70:0/2022537442 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0b2809bca0 con 0x7f0b30103390 2024-04-01T01:33:43.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:43 smithi070 ceph-mon[32179]: from='client.25418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:33:43.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:43 smithi070 ceph-mon[32179]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:33:43.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:43 smithi070 ceph-mon[32179]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:33:43.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:43 smithi070 ceph-mon[32179]: pgmap v365: 129 pgs: 129 active+clean; 654 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-04-01T01:33:43.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:43 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3427592543' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:43.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:43 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3547644391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:43.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:43 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/39470284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:43.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.753+0000 7f0b382c0640 1 -- 172.21.15.70:0/2022537442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0b300753d0 con 0x7f0b30103390 2024-04-01T01:33:43.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.754+0000 7f0b2f7fe640 1 -- 172.21.15.70:0/2022537442 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f0b2809b5a0 con 0x7f0b30103390 2024-04-01T01:33:43.755 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-01T01:33:43.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.756+0000 7f0b382c0640 1 -- 172.21.15.70:0/2022537442 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0b180a2530 msgr2=0x7f0b180a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:43.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.756+0000 7f0b382c0640 1 --2- 172.21.15.70:0/2022537442 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0b180a2530 0x7f0b180a49f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0b20032380 tx=0x7f0b2003e040 comp rx=0 tx=0).stop 2024-04-01T01:33:43.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.756+0000 7f0b382c0640 1 -- 172.21.15.70:0/2022537442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b30103390 msgr2=0x7f0b3019ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:43.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.756+0000 7f0b382c0640 1 --2- 172.21.15.70:0/2022537442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b30103390 0x7f0b3019ef20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800e970 tx=0x7f0b2800ed70 comp rx=0 tx=0).stop 2024-04-01T01:33:43.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.756+0000 7f0b382c0640 1 -- 172.21.15.70:0/2022537442 shutdown_connections 2024-04-01T01:33:43.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.756+0000 7f0b382c0640 1 --2- 172.21.15.70:0/2022537442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b30103390 0x7f0b3019ef20 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.756+0000 7f0b382c0640 1 --2- 172.21.15.70:0/2022537442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b301029d0 0x7f0b30197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.756+0000 7f0b382c0640 1 --2- 172.21.15.70:0/2022537442 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0b180a2530 0x7f0b180a49f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.756+0000 7f0b382c0640 1 --2- 172.21.15.70:0/2022537442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b301017e0 0x7f0b30197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.756+0000 7f0b382c0640 1 -- 172.21.15.70:0/2022537442 >> 172.21.15.70:0/2022537442 conn(0x7f0b300fcd70 msgr2=0x7f0b30105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:43.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.756+0000 7f0b382c0640 1 -- 172.21.15.70:0/2022537442 shutdown_connections 2024-04-01T01:33:43.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.756+0000 7f0b382c0640 1 -- 172.21.15.70:0/2022537442 wait complete. 2024-04-01T01:33:43.767 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-01T01:33:43.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.872+0000 7f6022014640 1 -- 172.21.15.70:0/3994560403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f601c102270 msgr2=0x7f601c104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:43.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.872+0000 7f6022014640 1 --2- 172.21.15.70:0/3994560403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f601c102270 0x7f601c104660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f600c009900 tx=0x7f600c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:43.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.872+0000 7f6022014640 1 -- 172.21.15.70:0/3994560403 shutdown_connections 2024-04-01T01:33:43.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.872+0000 7f6022014640 1 --2- 172.21.15.70:0/3994560403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f601c1074d0 0x7f601c1098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.872+0000 7f6022014640 1 --2- 172.21.15.70:0/3994560403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f601c104ba0 0x7f601c106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.872+0000 7f6022014640 1 --2- 172.21.15.70:0/3994560403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f601c102270 0x7f601c104660 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.872+0000 7f6022014640 1 -- 172.21.15.70:0/3994560403 >> 172.21.15.70:0/3994560403 conn(0x7f601c0fbbe0 msgr2=0x7f601c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:43.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.872+0000 7f6022014640 1 -- 172.21.15.70:0/3994560403 shutdown_connections 2024-04-01T01:33:43.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.872+0000 7f6022014640 1 -- 172.21.15.70:0/3994560403 wait complete. 2024-04-01T01:33:43.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.872+0000 7f6022014640 1 Processor -- start 2024-04-01T01:33:43.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f6022014640 1 -- start start 2024-04-01T01:33:43.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f6022014640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f601c102270 0x7f601c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:43.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f6022014640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f601c104ba0 0x7f601c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:43.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f6022014640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f601c1074d0 0x7f601c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:43.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f6022014640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f601c10d2a0 con 0x7f601c104ba0 2024-04-01T01:33:43.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f6022014640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f601c10d120 con 0x7f601c102270 2024-04-01T01:33:43.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f6022014640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f601c10d420 con 0x7f601c1074d0 2024-04-01T01:33:43.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f601affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f601c104ba0 0x7f601c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:43.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f601affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f601c104ba0 0x7f601c197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55572/0 (socket says 172.21.15.70:55572) 2024-04-01T01:33:43.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f601affd640 1 -- 172.21.15.70:0/2248629100 learned_addr learned my addr 172.21.15.70:0/2248629100 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:43.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f601b7fe640 1 --2- 172.21.15.70:0/2248629100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f601c102270 0x7f601c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:43.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f601bfff640 1 --2- 172.21.15.70:0/2248629100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f601c1074d0 0x7f601c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:43.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f601affd640 1 -- 172.21.15.70:0/2248629100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f601c1074d0 msgr2=0x7f601c19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:43.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f601affd640 1 --2- 172.21.15.70:0/2248629100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f601c1074d0 0x7f601c19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f601affd640 1 -- 172.21.15.70:0/2248629100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f601c102270 msgr2=0x7f601c197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:43.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f601affd640 1 --2- 172.21.15.70:0/2248629100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f601c102270 0x7f601c197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:43.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f601affd640 1 -- 172.21.15.70:0/2248629100 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601c19f660 con 0x7f601c104ba0 2024-04-01T01:33:43.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.873+0000 7f601affd640 1 --2- 172.21.15.70:0/2248629100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f601c104ba0 0x7f601c197f80 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f601000e9f0 tx=0x7f601000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:43.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.874+0000 7f6018ff9640 1 -- 172.21.15.70:0/2248629100 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601000ccc0 con 0x7f601c104ba0 2024-04-01T01:33:43.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.874+0000 7f6022014640 1 -- 172.21.15.70:0/2248629100 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f601c19f950 con 0x7f601c104ba0 2024-04-01T01:33:43.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.874+0000 7f6018ff9640 1 -- 172.21.15.70:0/2248629100 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60100054d0 con 0x7f601c104ba0 2024-04-01T01:33:43.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.874+0000 7f6018ff9640 1 -- 172.21.15.70:0/2248629100 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6010010430 con 0x7f601c104ba0 2024-04-01T01:33:43.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.874+0000 7f6022014640 1 -- 172.21.15.70:0/2248629100 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f601c19fe50 con 0x7f601c104ba0 2024-04-01T01:33:43.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.875+0000 7f6022014640 1 -- 172.21.15.70:0/2248629100 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f601c0fd6d0 con 0x7f601c104ba0 2024-04-01T01:33:43.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.876+0000 7f6018ff9640 1 -- 172.21.15.70:0/2248629100 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f6010005020 con 0x7f601c104ba0 2024-04-01T01:33:43.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.876+0000 7f6018ff9640 1 --2- 172.21.15.70:0/2248629100 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60040a2530 0x7f60040a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:43.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.876+0000 7f601b7fe640 1 --2- 172.21.15.70:0/2248629100 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60040a2530 0x7f60040a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:43.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.876+0000 7f6018ff9640 1 -- 172.21.15.70:0/2248629100 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f60100cf030 con 0x7f601c104ba0 2024-04-01T01:33:43.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.877+0000 7f601b7fe640 1 --2- 172.21.15.70:0/2248629100 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60040a2530 0x7f60040a49f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f600c009230 tx=0x7f600c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:43.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:43.880+0000 7f6018ff9640 1 -- 172.21.15.70:0/2248629100 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f601009bcc0 con 0x7f601c104ba0 2024-04-01T01:33:43.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:43 smithi189 ceph-mon[35753]: from='client.25418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:33:43.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:43 smithi189 ceph-mon[35753]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:33:43.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:43 smithi189 ceph-mon[35753]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:33:43.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:43 smithi189 ceph-mon[35753]: pgmap v365: 129 pgs: 129 active+clean; 654 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-04-01T01:33:43.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:43 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3427592543' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:43.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:43 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3547644391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:43.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:43 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/39470284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:44.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.056+0000 7f6022014640 1 -- 172.21.15.70:0/2248629100 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f601c02d380 con 0x7f601c104ba0 2024-04-01T01:33:44.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.056+0000 7f6018ff9640 1 -- 172.21.15.70:0/2248629100 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f601009b5c0 con 0x7f601c104ba0 2024-04-01T01:33:44.056 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:33:44.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.058+0000 7f6022014640 1 -- 172.21.15.70:0/2248629100 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60040a2530 msgr2=0x7f60040a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.058+0000 7f6022014640 1 --2- 172.21.15.70:0/2248629100 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60040a2530 0x7f60040a49f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f600c009230 tx=0x7f600c005fd0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.058+0000 7f6022014640 1 -- 172.21.15.70:0/2248629100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f601c104ba0 msgr2=0x7f601c197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.058+0000 7f6022014640 1 --2- 172.21.15.70:0/2248629100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f601c104ba0 0x7f601c197f80 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f601000e9f0 tx=0x7f601000eeb0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.058+0000 7f6022014640 1 -- 172.21.15.70:0/2248629100 shutdown_connections 2024-04-01T01:33:44.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.058+0000 7f6022014640 1 --2- 172.21.15.70:0/2248629100 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60040a2530 0x7f60040a49f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.058+0000 7f6022014640 1 --2- 172.21.15.70:0/2248629100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f601c1074d0 0x7f601c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.058+0000 7f6022014640 1 --2- 172.21.15.70:0/2248629100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f601c104ba0 0x7f601c197f80 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.058+0000 7f6022014640 1 --2- 172.21.15.70:0/2248629100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f601c102270 0x7f601c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.058+0000 7f6022014640 1 -- 172.21.15.70:0/2248629100 >> 172.21.15.70:0/2248629100 conn(0x7f601c0fbbe0 msgr2=0x7f601c104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:44.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.058+0000 7f6022014640 1 -- 172.21.15.70:0/2248629100 shutdown_connections 2024-04-01T01:33:44.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.058+0000 7f6022014640 1 -- 172.21.15.70:0/2248629100 wait complete. 2024-04-01T01:33:44.069 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:33:44.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:43 smithi081 ceph-mon[35142]: from='client.25418 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:33:44.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:43 smithi081 ceph-mon[35142]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:33:44.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:43 smithi081 ceph-mon[35142]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-01T01:33:44.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:43 smithi081 ceph-mon[35142]: pgmap v365: 129 pgs: 129 active+clean; 654 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-04-01T01:33:44.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:43 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3427592543' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:44.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:43 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3547644391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:33:44.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:43 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/39470284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:33:44.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.171+0000 7f007721c640 1 -- 172.21.15.70:0/3298296577 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0070101870 msgr2=0x7f0070101cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.171+0000 7f007721c640 1 --2- 172.21.15.70:0/3298296577 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0070101870 0x7f0070101cf0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f005c009860 tx=0x7f005c02bb70 comp rx=0 tx=0).stop 2024-04-01T01:33:44.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.171+0000 7f007721c640 1 -- 172.21.15.70:0/3298296577 shutdown_connections 2024-04-01T01:33:44.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.171+0000 7f007721c640 1 --2- 172.21.15.70:0/3298296577 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0070102230 0x7f0070108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.171+0000 7f007721c640 1 --2- 172.21.15.70:0/3298296577 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0070101870 0x7f0070101cf0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.171+0000 7f007721c640 1 --2- 172.21.15.70:0/3298296577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0070100680 0x7f0070100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.171+0000 7f007721c640 1 -- 172.21.15.70:0/3298296577 >> 172.21.15.70:0/3298296577 conn(0x7f00700fbc30 msgr2=0x7f00700fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:44.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.171+0000 7f007721c640 1 -- 172.21.15.70:0/3298296577 shutdown_connections 2024-04-01T01:33:44.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.171+0000 7f007721c640 1 -- 172.21.15.70:0/3298296577 wait complete. 2024-04-01T01:33:44.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f007721c640 1 Processor -- start 2024-04-01T01:33:44.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f007721c640 1 -- start start 2024-04-01T01:33:44.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f007721c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0070100680 0x7f0070196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f007721c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0070101870 0x7f0070196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f007721c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0070102230 0x7f007019dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f007721c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f007010c1b0 con 0x7f0070102230 2024-04-01T01:33:44.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f007721c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f007010c030 con 0x7f0070101870 2024-04-01T01:33:44.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f007721c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f007010c330 con 0x7f0070100680 2024-04-01T01:33:44.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f0074fb8640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0070100680 0x7f0070196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f00757b9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0070102230 0x7f007019dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f0074fb8640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0070100680 0x7f0070196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41120/0 (socket says 172.21.15.70:41120) 2024-04-01T01:33:44.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f0074fb8640 1 -- 172.21.15.70:0/3204140646 learned_addr learned my addr 172.21.15.70:0/3204140646 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:44.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f00757b9640 1 --2- 172.21.15.70:0/3204140646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0070102230 0x7f007019dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55586/0 (socket says 172.21.15.70:55586) 2024-04-01T01:33:44.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f006ffff640 1 --2- 172.21.15.70:0/3204140646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0070101870 0x7f0070196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f00757b9640 1 -- 172.21.15.70:0/3204140646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0070100680 msgr2=0x7f0070196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f00757b9640 1 --2- 172.21.15.70:0/3204140646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0070100680 0x7f0070196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f00757b9640 1 -- 172.21.15.70:0/3204140646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0070101870 msgr2=0x7f0070196e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f00757b9640 1 --2- 172.21.15.70:0/3204140646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0070101870 0x7f0070196e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f00757b9640 1 -- 172.21.15.70:0/3204140646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f007019e550 con 0x7f0070102230 2024-04-01T01:33:44.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f006ffff640 1 --2- 172.21.15.70:0/3204140646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0070101870 0x7f0070196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:44.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f0074fb8640 1 --2- 172.21.15.70:0/3204140646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0070100680 0x7f0070196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:44.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f00757b9640 1 --2- 172.21.15.70:0/3204140646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0070102230 0x7f007019dee0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0068009fd0 tx=0x7f006800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:44.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.172+0000 7f006dffb640 1 -- 172.21.15.70:0/3204140646 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006800cd60 con 0x7f0070102230 2024-04-01T01:33:44.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.173+0000 7f006dffb640 1 -- 172.21.15.70:0/3204140646 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0068005020 con 0x7f0070102230 2024-04-01T01:33:44.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.173+0000 7f007721c640 1 -- 172.21.15.70:0/3204140646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0070069e00 con 0x7f0070102230 2024-04-01T01:33:44.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.173+0000 7f006dffb640 1 -- 172.21.15.70:0/3204140646 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0068010690 con 0x7f0070102230 2024-04-01T01:33:44.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.173+0000 7f007721c640 1 -- 172.21.15.70:0/3204140646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f007006a2d0 con 0x7f0070102230 2024-04-01T01:33:44.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.174+0000 7f007721c640 1 -- 172.21.15.70:0/3204140646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0070068990 con 0x7f0070102230 2024-04-01T01:33:44.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.175+0000 7f006dffb640 1 -- 172.21.15.70:0/3204140646 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f00680051d0 con 0x7f0070102230 2024-04-01T01:33:44.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.175+0000 7f006dffb640 1 --2- 172.21.15.70:0/3204140646 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00600a2530 0x7f00600a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.175+0000 7f006dffb640 1 -- 172.21.15.70:0/3204140646 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f00680cf380 con 0x7f0070102230 2024-04-01T01:33:44.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.175+0000 7f0074fb8640 1 --2- 172.21.15.70:0/3204140646 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00600a2530 0x7f00600a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.176+0000 7f0074fb8640 1 --2- 172.21.15.70:0/3204140646 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00600a2530 0x7f00600a49f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f006400e5f0 tx=0x7f0064010040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:44.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.180+0000 7f006dffb640 1 -- 172.21.15.70:0/3204140646 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f006809c010 con 0x7f0070102230 2024-04-01T01:33:44.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.392+0000 7f007721c640 1 -- 172.21.15.70:0/3204140646 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0070107530 con 0x7f0070102230 2024-04-01T01:33:44.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.392+0000 7f006dffb640 1 -- 172.21.15.70:0/3204140646 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f006809b910 con 0x7f0070102230 2024-04-01T01:33:44.393 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:44.393 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:33:36.894838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24418,24404,24379,24451,24464]},"id":1}]} 2024-04-01T01:33:44.393 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-04-01T01:33:44.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.395+0000 7f007721c640 1 -- 172.21.15.70:0/3204140646 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00600a2530 msgr2=0x7f00600a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.395+0000 7f007721c640 1 --2- 172.21.15.70:0/3204140646 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00600a2530 0x7f00600a49f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f006400e5f0 tx=0x7f0064010040 comp rx=0 tx=0).stop 2024-04-01T01:33:44.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.395+0000 7f007721c640 1 -- 172.21.15.70:0/3204140646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0070102230 msgr2=0x7f007019dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.395+0000 7f007721c640 1 --2- 172.21.15.70:0/3204140646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0070102230 0x7f007019dee0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0068009fd0 tx=0x7f006800ee50 comp rx=0 tx=0).stop 2024-04-01T01:33:44.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.395+0000 7f007721c640 1 -- 172.21.15.70:0/3204140646 shutdown_connections 2024-04-01T01:33:44.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.395+0000 7f007721c640 1 --2- 172.21.15.70:0/3204140646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0070102230 0x7f007019dee0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.395+0000 7f007721c640 1 --2- 172.21.15.70:0/3204140646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0070101870 0x7f0070196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.395+0000 7f007721c640 1 --2- 172.21.15.70:0/3204140646 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00600a2530 0x7f00600a49f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.395+0000 7f007721c640 1 --2- 172.21.15.70:0/3204140646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0070100680 0x7f0070196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.395+0000 7f007721c640 1 -- 172.21.15.70:0/3204140646 >> 172.21.15.70:0/3204140646 conn(0x7f00700fbc30 msgr2=0x7f0070104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:44.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.395+0000 7f007721c640 1 -- 172.21.15.70:0/3204140646 shutdown_connections 2024-04-01T01:33:44.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.395+0000 7f007721c640 1 -- 172.21.15.70:0/3204140646 wait complete. 2024-04-01T01:33:44.406 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:33:44.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.508+0000 7f4588a3b640 1 -- 172.21.15.70:0/2912838936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4584103320 msgr2=0x7f4584109bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.508+0000 7f4588a3b640 1 --2- 172.21.15.70:0/2912838936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4584103320 0x7f4584109bb0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f457c009ca0 tx=0x7f457c02c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.508+0000 7f4588a3b640 1 -- 172.21.15.70:0/2912838936 shutdown_connections 2024-04-01T01:33:44.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.508+0000 7f4588a3b640 1 --2- 172.21.15.70:0/2912838936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4584103320 0x7f4584109bb0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.508+0000 7f4588a3b640 1 --2- 172.21.15.70:0/2912838936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4584102960 0x7f4584102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.508+0000 7f4588a3b640 1 --2- 172.21.15.70:0/2912838936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4584101770 0x7f4584101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.508+0000 7f4588a3b640 1 -- 172.21.15.70:0/2912838936 >> 172.21.15.70:0/2912838936 conn(0x7f45840fcd00 msgr2=0x7f45840ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:44.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.508+0000 7f4588a3b640 1 -- 172.21.15.70:0/2912838936 shutdown_connections 2024-04-01T01:33:44.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.508+0000 7f4588a3b640 1 -- 172.21.15.70:0/2912838936 wait complete. 2024-04-01T01:33:44.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f4588a3b640 1 Processor -- start 2024-04-01T01:33:44.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f4588a3b640 1 -- start start 2024-04-01T01:33:44.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f4588a3b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4584101770 0x7f4584197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f4588a3b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4584102960 0x7f4584197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f4588a3b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4584103320 0x7f458419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f4588a3b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f458410d2a0 con 0x7f4584101770 2024-04-01T01:33:44.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f4588a3b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f458410d120 con 0x7f4584103320 2024-04-01T01:33:44.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f4588a3b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f458410d420 con 0x7f4584102960 2024-04-01T01:33:44.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f458259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4584101770 0x7f4584197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f458259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4584101770 0x7f4584197960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55602/0 (socket says 172.21.15.70:55602) 2024-04-01T01:33:44.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f458259c640 1 -- 172.21.15.70:0/3637139791 learned_addr learned my addr 172.21.15.70:0/3637139791 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:44.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f4581d9b640 1 --2- 172.21.15.70:0/3637139791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4584102960 0x7f4584197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f4582d9d640 1 --2- 172.21.15.70:0/3637139791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4584103320 0x7f458419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f458259c640 1 -- 172.21.15.70:0/3637139791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4584102960 msgr2=0x7f4584197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f458259c640 1 --2- 172.21.15.70:0/3637139791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4584102960 0x7f4584197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f458259c640 1 -- 172.21.15.70:0/3637139791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4584103320 msgr2=0x7f458419ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f458259c640 1 --2- 172.21.15.70:0/3637139791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4584103320 0x7f458419ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f458259c640 1 -- 172.21.15.70:0/3637139791 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f458419f580 con 0x7f4584101770 2024-04-01T01:33:44.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f458259c640 1 --2- 172.21.15.70:0/3637139791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4584101770 0x7f4584197960 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f457800e9c0 tx=0x7f457800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:44.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.509+0000 7f45737fe640 1 -- 172.21.15.70:0/3637139791 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457800cde0 con 0x7f4584101770 2024-04-01T01:33:44.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.510+0000 7f4588a3b640 1 -- 172.21.15.70:0/3637139791 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f458419f870 con 0x7f4584101770 2024-04-01T01:33:44.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.510+0000 7f45737fe640 1 -- 172.21.15.70:0/3637139791 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f45780108e0 con 0x7f4584101770 2024-04-01T01:33:44.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.510+0000 7f45737fe640 1 -- 172.21.15.70:0/3637139791 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45780188e0 con 0x7f4584101770 2024-04-01T01:33:44.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.510+0000 7f4588a3b640 1 -- 172.21.15.70:0/3637139791 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f458419fdc0 con 0x7f4584101770 2024-04-01T01:33:44.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.511+0000 7f4588a3b640 1 -- 172.21.15.70:0/3637139791 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4584108a90 con 0x7f4584101770 2024-04-01T01:33:44.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.511+0000 7f45737fe640 1 -- 172.21.15.70:0/3637139791 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f457801f020 con 0x7f4584101770 2024-04-01T01:33:44.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.512+0000 7f45737fe640 1 --2- 172.21.15.70:0/3637139791 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f456c0a2530 0x7f456c0a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.512+0000 7f4581d9b640 1 --2- 172.21.15.70:0/3637139791 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f456c0a2530 0x7f456c0a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.512+0000 7f45737fe640 1 -- 172.21.15.70:0/3637139791 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f45780cee40 con 0x7f4584101770 2024-04-01T01:33:44.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.512+0000 7f4581d9b640 1 --2- 172.21.15.70:0/3637139791 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f456c0a2530 0x7f456c0a49f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4584198e70 tx=0x7f4574005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:44.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.516+0000 7f45737fe640 1 -- 172.21.15.70:0/3637139791 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f457809bad0 con 0x7f4584101770 2024-04-01T01:33:44.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.697+0000 7f4588a3b640 1 -- 172.21.15.70:0/3637139791 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f458402d2f0 con 0x7f4584101770 2024-04-01T01:33:44.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.699+0000 7f45737fe640 1 -- 172.21.15.70:0/3637139791 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24781 (secure 0 0 0) 0x7f457809b3d0 con 0x7f4584101770 2024-04-01T01:33:44.699 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:44.700 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":93,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:33:19.716245+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:33:44.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.701+0000 7f4588a3b640 1 -- 172.21.15.70:0/3637139791 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f456c0a2530 msgr2=0x7f456c0a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.701+0000 7f4588a3b640 1 --2- 172.21.15.70:0/3637139791 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f456c0a2530 0x7f456c0a49f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4584198e70 tx=0x7f4574005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:44.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.701+0000 7f4588a3b640 1 -- 172.21.15.70:0/3637139791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4584101770 msgr2=0x7f4584197960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.701+0000 7f4588a3b640 1 --2- 172.21.15.70:0/3637139791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4584101770 0x7f4584197960 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f457800e9c0 tx=0x7f457800ee80 comp rx=0 tx=0).stop 2024-04-01T01:33:44.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.701+0000 7f4588a3b640 1 -- 172.21.15.70:0/3637139791 shutdown_connections 2024-04-01T01:33:44.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.701+0000 7f4588a3b640 1 --2- 172.21.15.70:0/3637139791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4584103320 0x7f458419ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.701+0000 7f4588a3b640 1 --2- 172.21.15.70:0/3637139791 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f456c0a2530 0x7f456c0a49f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.701+0000 7f4588a3b640 1 --2- 172.21.15.70:0/3637139791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4584102960 0x7f4584197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.701+0000 7f4588a3b640 1 --2- 172.21.15.70:0/3637139791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4584101770 0x7f4584197960 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.701+0000 7f4588a3b640 1 -- 172.21.15.70:0/3637139791 >> 172.21.15.70:0/3637139791 conn(0x7f45840fcd00 msgr2=0x7f4584105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:44.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.701+0000 7f4588a3b640 1 -- 172.21.15.70:0/3637139791 shutdown_connections 2024-04-01T01:33:44.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.701+0000 7f4588a3b640 1 -- 172.21.15.70:0/3637139791 wait complete. 2024-04-01T01:33:44.714 DEBUG:tasks.fwd_scrub:Yielding 2024-04-01T01:33:44.714 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-01T01:33:44.724 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi070.front.sepia.ceph.com 2024-04-01T01:33:44.724 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-01T01:33:44.725 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-01T01:33:44.725 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:33:44.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:33:44.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:33:44.727 INFO:teuthology.run_tasks:Running task exec... 2024-04-01T01:33:44.834 INFO:teuthology.task.exec:Executing custom commands... 2024-04-01T01:33:44.834 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi070.front.sepia.ceph.com 2024-04-01T01:33:44.834 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/' 2024-04-01T01:33:44.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.831+0000 7f6a40a1c640 1 -- 172.21.15.70:0/3015251873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c1019b0 msgr2=0x7f6a3c108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.831+0000 7f6a40a1c640 1 --2- 172.21.15.70:0/3015251873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c1019b0 0x7f6a3c108240 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f6a34009ca0 tx=0x7f6a3402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.831+0000 7f6a40a1c640 1 -- 172.21.15.70:0/3015251873 shutdown_connections 2024-04-01T01:33:44.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.831+0000 7f6a40a1c640 1 --2- 172.21.15.70:0/3015251873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c1019b0 0x7f6a3c108240 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.831+0000 7f6a40a1c640 1 --2- 172.21.15.70:0/3015251873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c100ff0 0x7f6a3c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.831+0000 7f6a40a1c640 1 --2- 172.21.15.70:0/3015251873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a3c0ffe00 0x7f6a3c100220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.831+0000 7f6a40a1c640 1 -- 172.21.15.70:0/3015251873 >> 172.21.15.70:0/3015251873 conn(0x7f6a3c0fb350 msgr2=0x7f6a3c0fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:44.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.831+0000 7f6a40a1c640 1 -- 172.21.15.70:0/3015251873 shutdown_connections 2024-04-01T01:33:44.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.831+0000 7f6a40a1c640 1 -- 172.21.15.70:0/3015251873 wait complete. 2024-04-01T01:33:44.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a40a1c640 1 Processor -- start 2024-04-01T01:33:44.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a40a1c640 1 -- start start 2024-04-01T01:33:44.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a40a1c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c0ffe00 0x7f6a3c195790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a40a1c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c100ff0 0x7f6a3c195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a40a1c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a3c1019b0 0x7f6a3c19ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a40a1c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c10b090 con 0x7f6a3c100ff0 2024-04-01T01:33:44.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a40a1c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c10af10 con 0x7f6a3c1019b0 2024-04-01T01:33:44.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a40a1c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c10b210 con 0x7f6a3c0ffe00 2024-04-01T01:33:44.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a39d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c100ff0 0x7f6a3c195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a39d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c100ff0 0x7f6a3c195cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55624/0 (socket says 172.21.15.70:55624) 2024-04-01T01:33:44.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a39d9b640 1 -- 172.21.15.70:0/3046995617 learned_addr learned my addr 172.21.15.70:0/3046995617 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:44.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a3ad9d640 1 --2- 172.21.15.70:0/3046995617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a3c1019b0 0x7f6a3c19ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a3a59c640 1 --2- 172.21.15.70:0/3046995617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c0ffe00 0x7f6a3c195790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a39d9b640 1 -- 172.21.15.70:0/3046995617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c0ffe00 msgr2=0x7f6a3c195790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a39d9b640 1 --2- 172.21.15.70:0/3046995617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c0ffe00 0x7f6a3c195790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a39d9b640 1 -- 172.21.15.70:0/3046995617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a3c1019b0 msgr2=0x7f6a3c19ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a39d9b640 1 --2- 172.21.15.70:0/3046995617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a3c1019b0 0x7f6a3c19ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.832+0000 7f6a39d9b640 1 -- 172.21.15.70:0/3046995617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3c19d360 con 0x7f6a3c100ff0 2024-04-01T01:33:44.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.833+0000 7f6a39d9b640 1 --2- 172.21.15.70:0/3046995617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c100ff0 0x7f6a3c195cd0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000e9f0 tx=0x7f6a3000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:44.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.833+0000 7f6a2b7fe640 1 -- 172.21.15.70:0/3046995617 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3000cd80 con 0x7f6a3c100ff0 2024-04-01T01:33:44.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.833+0000 7f6a2b7fe640 1 -- 172.21.15.70:0/3046995617 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a300108e0 con 0x7f6a3c100ff0 2024-04-01T01:33:44.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.833+0000 7f6a40a1c640 1 -- 172.21.15.70:0/3046995617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a3c19d650 con 0x7f6a3c100ff0 2024-04-01T01:33:44.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.833+0000 7f6a2b7fe640 1 -- 172.21.15.70:0/3046995617 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30018920 con 0x7f6a3c100ff0 2024-04-01T01:33:44.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.833+0000 7f6a40a1c640 1 -- 172.21.15.70:0/3046995617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a3c19dba0 con 0x7f6a3c100ff0 2024-04-01T01:33:44.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.835+0000 7f6a2b7fe640 1 -- 172.21.15.70:0/3046995617 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f6a30010a90 con 0x7f6a3c100ff0 2024-04-01T01:33:44.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.835+0000 7f6a2b7fe640 1 --2- 172.21.15.70:0/3046995617 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6a240a2530 0x7f6a240a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.835+0000 7f6a435ff740 1 Processor -- start 2024-04-01T01:33:44.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.835+0000 7f6a3a59c640 1 --2- 172.21.15.70:0/3046995617 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6a240a2530 0x7f6a240a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.835+0000 7f6a2b7fe640 1 -- 172.21.15.70:0/3046995617 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6a300cef50 con 0x7f6a3c100ff0 2024-04-01T01:33:44.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.835+0000 7f6a435ff740 1 -- start start 2024-04-01T01:33:44.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a435ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e349c20 0x555d0e34a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a435ff740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e34a610 0x555d0e34f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a435ff740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 0x555d0e351c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a435ff740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555d0e2490b0 con 0x555d0e34a610 2024-04-01T01:33:44.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a435ff740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555d0e2abea0 con 0x555d0e34f780 2024-04-01T01:33:44.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a435ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555d0e254c00 con 0x555d0e349c20 2024-04-01T01:33:44.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a3a59c640 1 --2- 172.21.15.70:0/3046995617 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6a240a2530 0x7f6a240a49f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c100e50 tx=0x7f6a2c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:44.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a39d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e34a610 0x555d0e34f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a39d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e34a610 0x555d0e34f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55634/0 (socket says 172.21.15.70:55634) 2024-04-01T01:33:44.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a39d9b640 1 -- 172.21.15.70:0/3378375438 learned_addr learned my addr 172.21.15.70:0/3378375438 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:44.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a3a59c640 1 --2- 172.21.15.70:0/3378375438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e349c20 0x555d0e34a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a3ad9d640 1 --2- 172.21.15.70:0/3378375438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 0x555d0e351c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a39d9b640 1 -- 172.21.15.70:0/3378375438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e349c20 msgr2=0x555d0e34a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:44.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a39d9b640 1 --2- 172.21.15.70:0/3378375438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e349c20 0x555d0e34a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a39d9b640 1 -- 172.21.15.70:0/3378375438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 msgr2=0x555d0e351c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:44.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a39d9b640 1 --2- 172.21.15.70:0/3378375438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 0x555d0e351c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a39d9b640 1 -- 172.21.15.70:0/3378375438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d0e0887e0 con 0x555d0e34a610 2024-04-01T01:33:44.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a39d9b640 1 --2- 172.21.15.70:0/3378375438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e34a610 0x555d0e34f110 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6a30097f60 tx=0x7f6a3009b590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b90d5a41d30ea816 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:44.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a28ff9640 1 -- 172.21.15.70:0/3378375438 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a300cc780 con 0x555d0e34a610 2024-04-01T01:33:44.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a28ff9640 1 -- 172.21.15.70:0/3378375438 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a30097c00 con 0x555d0e34a610 2024-04-01T01:33:44.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.836+0000 7f6a28ff9640 1 -- 172.21.15.70:0/3378375438 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a300ccd80 con 0x555d0e34a610 2024-04-01T01:33:44.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.837+0000 7f6a435ff740 1 -- 172.21.15.70:0/3378375438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e34a610 msgr2=0x555d0e34f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.837+0000 7f6a435ff740 1 --2- 172.21.15.70:0/3378375438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e34a610 0x555d0e34f110 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6a30097f60 tx=0x7f6a3009b590 comp rx=0 tx=0).stop 2024-04-01T01:33:44.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.837+0000 7f6a435ff740 1 -- 172.21.15.70:0/3378375438 shutdown_connections 2024-04-01T01:33:44.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.837+0000 7f6a435ff740 1 --2- 172.21.15.70:0/3378375438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 0x555d0e351c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.837+0000 7f6a435ff740 1 --2- 172.21.15.70:0/3378375438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e34a610 0x555d0e34f110 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.837+0000 7f6a435ff740 1 --2- 172.21.15.70:0/3378375438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e349c20 0x555d0e34a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.837+0000 7f6a435ff740 1 -- 172.21.15.70:0/3378375438 >> 172.21.15.70:0/3378375438 conn(0x555d0e345190 msgr2=0x555d0e345590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:44.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.837+0000 7f6a435ff740 1 -- 172.21.15.70:0/3378375438 shutdown_connections 2024-04-01T01:33:44.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.837+0000 7f6a435ff740 1 -- 172.21.15.70:0/3378375438 wait complete. 2024-04-01T01:33:44.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.837+0000 7f6a435ff740 1 -- start start 2024-04-01T01:33:44.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.837+0000 7f6a435ff740 1 Processor -- start 2024-04-01T01:33:44.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a435ff740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e349c20 0x555d0e353ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a435ff740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e34a610 0x555d0e354420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a435ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34f780 0x555d0e35b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a435ff740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555d0e24a3d0 con 0x555d0e34a610 2024-04-01T01:33:44.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a435ff740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555d0e2a7f40 con 0x555d0e349c20 2024-04-01T01:33:44.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a435ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555d0e352670 con 0x555d0e34f780 2024-04-01T01:33:44.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a435ff740 10 client.0 _ll_register_callbacks cb 0x555d0e267b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:44.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a39d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e34a610 0x555d0e354420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a3a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e349c20 0x555d0e353ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a3a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e349c20 0x555d0e353ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34784/0 (socket says 172.21.15.70:34784) 2024-04-01T01:33:44.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a3a59c640 1 -- 172.21.15.70:0/2225951089 learned_addr learned my addr 172.21.15.70:0/2225951089 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:44.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a3ad9d640 1 --2- 172.21.15.70:0/2225951089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34f780 0x555d0e35b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a3a59c640 1 -- 172.21.15.70:0/2225951089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34f780 msgr2=0x555d0e35b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:44.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a3a59c640 1 --2- 172.21.15.70:0/2225951089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34f780 0x555d0e35b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a3a59c640 1 -- 172.21.15.70:0/2225951089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e34a610 msgr2=0x555d0e354420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:44.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a3a59c640 1 --2- 172.21.15.70:0/2225951089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e34a610 0x555d0e354420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a3a59c640 1 -- 172.21.15.70:0/2225951089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d0e0e8f70 con 0x555d0e349c20 2024-04-01T01:33:44.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a39d9b640 1 --2- 172.21.15.70:0/2225951089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e34a610 0x555d0e354420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:44.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a3a59c640 1 --2- 172.21.15.70:0/2225951089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e349c20 0x555d0e353ee0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c01ca00 tx=0x7f6a2c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aae99c49f283c7e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:44.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:33:44.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a21ffb640 1 -- 172.21.15.70:0/2225951089 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c00db70 con 0x555d0e349c20 2024-04-01T01:33:44.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a21ffb640 1 -- 172.21.15.70:0/2225951089 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6a2c009050 con 0x555d0e349c20 2024-04-01T01:33:44.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.838+0000 7f6a435ff740 1 -- 172.21.15.70:0/2225951089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a30097c00 con 0x555d0e349c20 2024-04-01T01:33:44.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.839+0000 7f6a21ffb640 1 -- 172.21.15.70:0/2225951089 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6a2c032050 con 0x555d0e349c20 2024-04-01T01:33:44.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.839+0000 7f6a435ff740 10 client.25484 fetch_fsmap learned FSMap version 29 2024-04-01T01:33:44.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.839+0000 7f6a435ff740 1 -- 172.21.15.70:0/2225951089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555d0df80170 con 0x555d0e349c20 2024-04-01T01:33:44.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.839+0000 7f6a21ffb640 1 -- 172.21.15.70:0/2225951089 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a30097c00 con 0x555d0e349c20 2024-04-01T01:33:44.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.839+0000 7f6a21ffb640 1 -- 172.21.15.70:0/2225951089 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6a2c00b040 con 0x555d0e349c20 2024-04-01T01:33:44.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.839+0000 7f6a435ff740 10 client.25484 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:33:44.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.839+0000 7f6a435ff740 10 client.25484 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:33:44.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.839+0000 7f6a435ff740 20 client.25484 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:44.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.839+0000 7f6a435ff740 1 --2- 172.21.15.70:0/2225951089 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555d0e35fab0 0x555d0e361f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.839+0000 7f6a435ff740 4 client.25484 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:33:44.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.839+0000 7f6a435ff740 1 -- 172.21.15.70:0/2225951089 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d0e2b93e0 con 0x555d0e35fab0 2024-04-01T01:33:44.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.839+0000 7f6a39d9b640 1 --2- 172.21.15.70:0/2225951089 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555d0e35fab0 0x555d0e361f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.840+0000 7f6a21ffb640 10 client.25484 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:44.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.839+0000 7f6a39d9b640 1 --2- 172.21.15.70:0/2225951089 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555d0e35fab0 0x555d0e361f70 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f76f75af8f7733b server_cookie=40d26f4debca1dfd in_seq=0 out_seq=0 2024-04-01T01:33:44.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.840+0000 7f6a21ffb640 1 -- 172.21.15.70:0/2225951089 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555d0e2b93e0 con 0x555d0e35fab0 2024-04-01T01:33:44.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.840+0000 7f6a21ffb640 10 client.25484 handle_command_reply: tid=0 2024-04-01T01:33:44.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.840+0000 7f6a435ff740 1 client.25484 shutdown 2024-04-01T01:33:44.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.840+0000 7f6a435ff740 1 -- 172.21.15.70:0/2225951089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e349c20 msgr2=0x555d0e353ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.840+0000 7f6a435ff740 1 --2- 172.21.15.70:0/2225951089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e349c20 0x555d0e353ee0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c01ca00 tx=0x7f6a2c004730 comp rx=0 tx=0).stop 2024-04-01T01:33:44.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.840+0000 7f6a435ff740 1 -- 172.21.15.70:0/2225951089 shutdown_connections 2024-04-01T01:33:44.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.840+0000 7f6a435ff740 1 --2- 172.21.15.70:0/2225951089 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555d0e35fab0 0x555d0e361f70 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.840+0000 7f6a435ff740 1 --2- 172.21.15.70:0/2225951089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34f780 0x555d0e35b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.840+0000 7f6a435ff740 1 --2- 172.21.15.70:0/2225951089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e34a610 0x555d0e354420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.840+0000 7f6a435ff740 1 --2- 172.21.15.70:0/2225951089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e349c20 0x555d0e353ee0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.840+0000 7f6a21ffb640 0 client.25484 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:44.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.840+0000 7f6a435ff740 1 -- 172.21.15.70:0/2225951089 >> 172.21.15.70:0/2225951089 conn(0x555d0e345190 msgr2=0x555d0e342180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:44.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.840+0000 7f6a435ff740 1 -- 172.21.15.70:0/2225951089 shutdown_connections 2024-04-01T01:33:44.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.840+0000 7f6a435ff740 1 -- 172.21.15.70:0/2225951089 wait complete. 2024-04-01T01:33:44.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.840+0000 7f6a435ff740 20 client.25484 trim_cache size 0 max 16384 2024-04-01T01:33:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a435ff740 1 Processor -- start 2024-04-01T01:33:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a435ff740 1 -- start start 2024-04-01T01:33:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a435ff740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e349c20 0x555d0e342000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a435ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34a610 0x555d0e35b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a435ff740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 0x555d0e352a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a435ff740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c1a8170 con 0x555d0e349c20 2024-04-01T01:33:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a435ff740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c1a9cb0 con 0x555d0e34f780 2024-04-01T01:33:44.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a435ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c1a8840 con 0x555d0e34a610 2024-04-01T01:33:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a3a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e349c20 0x555d0e342000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a39d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34a610 0x555d0e35b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a39d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34a610 0x555d0e35b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41158/0 (socket says 172.21.15.70:41158) 2024-04-01T01:33:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a3a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e349c20 0x555d0e342000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55650/0 (socket says 172.21.15.70:55650) 2024-04-01T01:33:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a39d9b640 1 -- 172.21.15.70:0/1036314501 learned_addr learned my addr 172.21.15.70:0/1036314501 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a3ad9d640 1 --2- 172.21.15.70:0/1036314501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 0x555d0e352a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a39d9b640 1 -- 172.21.15.70:0/1036314501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 msgr2=0x555d0e352a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a39d9b640 1 --2- 172.21.15.70:0/1036314501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 0x555d0e352a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a39d9b640 1 -- 172.21.15.70:0/1036314501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e349c20 msgr2=0x555d0e342000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a39d9b640 1 --2- 172.21.15.70:0/1036314501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e349c20 0x555d0e342000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a39d9b640 1 -- 172.21.15.70:0/1036314501 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d0e34a9f0 con 0x555d0e34a610 2024-04-01T01:33:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a3ad9d640 1 --2- 172.21.15.70:0/1036314501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 0x555d0e352a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.858+0000 7f6a39d9b640 1 --2- 172.21.15.70:0/1036314501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34a610 0x555d0e35b3f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a30010f70 tx=0x7f6a30086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3df15fc46bc7ff23 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:44.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.859+0000 7f6a23fff640 1 -- 172.21.15.70:0/1036314501 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30097c00 con 0x555d0e34a610 2024-04-01T01:33:44.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.859+0000 7f6a23fff640 1 -- 172.21.15.70:0/1036314501 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a300bb0b0 con 0x555d0e34a610 2024-04-01T01:33:44.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.859+0000 7f6a23fff640 1 -- 172.21.15.70:0/1036314501 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30084b20 con 0x555d0e34a610 2024-04-01T01:33:44.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.859+0000 7f6a435ff740 1 -- 172.21.15.70:0/1036314501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34a610 msgr2=0x555d0e35b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.859+0000 7f6a435ff740 1 --2- 172.21.15.70:0/1036314501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34a610 0x555d0e35b3f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a30010f70 tx=0x7f6a30086040 comp rx=0 tx=0).stop 2024-04-01T01:33:44.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.859+0000 7f6a435ff740 1 -- 172.21.15.70:0/1036314501 shutdown_connections 2024-04-01T01:33:44.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.859+0000 7f6a435ff740 1 --2- 172.21.15.70:0/1036314501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 0x555d0e352a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.859+0000 7f6a435ff740 1 --2- 172.21.15.70:0/1036314501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34a610 0x555d0e35b3f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.859+0000 7f6a435ff740 1 --2- 172.21.15.70:0/1036314501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e349c20 0x555d0e342000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.859+0000 7f6a435ff740 1 -- 172.21.15.70:0/1036314501 >> 172.21.15.70:0/1036314501 conn(0x555d0e345190 msgr2=0x555d0e3418f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:44.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.859+0000 7f6a435ff740 1 -- 172.21.15.70:0/1036314501 shutdown_connections 2024-04-01T01:33:44.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.859+0000 7f6a435ff740 1 -- 172.21.15.70:0/1036314501 wait complete. 2024-04-01T01:33:44.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a435ff740 1 -- start start 2024-04-01T01:33:44.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a435ff740 1 Processor -- start 2024-04-01T01:33:44.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a435ff740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e349c20 0x555d0e358ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a435ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34a610 0x555d0e359400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a435ff740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 0x555d0e35a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a435ff740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c1a95e0 con 0x555d0e349c20 2024-04-01T01:33:44.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a435ff740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c1a8f10 con 0x555d0e34f780 2024-04-01T01:33:44.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a435ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c1a6150 con 0x555d0e34a610 2024-04-01T01:33:44.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a435ff740 10 client.0 _ll_register_callbacks cb 0x7f6a2c01c890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:44.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a3a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e349c20 0x555d0e358ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a3a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e349c20 0x555d0e358ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55662/0 (socket says 172.21.15.70:55662) 2024-04-01T01:33:44.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a3a59c640 1 -- 172.21.15.70:0/3962140775 learned_addr learned my addr 172.21.15.70:0/3962140775 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:44.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a39d9b640 1 --2- 172.21.15.70:0/3962140775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34a610 0x555d0e359400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a3ad9d640 1 --2- 172.21.15.70:0/3962140775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 0x555d0e35a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a3a59c640 1 -- 172.21.15.70:0/3962140775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34a610 msgr2=0x555d0e359400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:44.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a3a59c640 1 --2- 172.21.15.70:0/3962140775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34a610 0x555d0e359400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a3a59c640 1 -- 172.21.15.70:0/3962140775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 msgr2=0x555d0e35a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:44.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a3a59c640 1 --2- 172.21.15.70:0/3962140775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 0x555d0e35a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a3a59c640 1 -- 172.21.15.70:0/3962140775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d0e35aa00 con 0x555d0e349c20 2024-04-01T01:33:44.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.860+0000 7f6a3ad9d640 1 --2- 172.21.15.70:0/3962140775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 0x555d0e35a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:44.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.861+0000 7f6a3a59c640 1 --2- 172.21.15.70:0/3962140775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e349c20 0x555d0e358ec0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c003af0 tx=0x7f6a2c00de20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f9e60810f33f8a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:44.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.861+0000 7f6a21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:44.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.861+0000 7f6a21ffb640 1 -- 172.21.15.70:0/3962140775 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c03c070 con 0x555d0e349c20 2024-04-01T01:33:44.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.861+0000 7f6a21ffb640 1 -- 172.21.15.70:0/3962140775 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6a2c003800 con 0x555d0e349c20 2024-04-01T01:33:44.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.861+0000 7f6a435ff740 1 -- 172.21.15.70:0/3962140775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a300bb0b0 con 0x555d0e349c20 2024-04-01T01:33:44.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.861+0000 7f6a21ffb640 1 -- 172.21.15.70:0/3962140775 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6a2c045050 con 0x555d0e349c20 2024-04-01T01:33:44.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.861+0000 7f6a435ff740 10 client.15675 fetch_fsmap learned FSMap version 29 2024-04-01T01:33:44.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.861+0000 7f6a435ff740 1 -- 172.21.15.70:0/3962140775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555d0e3c6270 con 0x555d0e349c20 2024-04-01T01:33:44.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.861+0000 7f6a21ffb640 1 -- 172.21.15.70:0/3962140775 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a300bb0b0 con 0x555d0e349c20 2024-04-01T01:33:44.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.861+0000 7f6a21ffb640 1 -- 172.21.15.70:0/3962140775 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6a2c03f020 con 0x555d0e349c20 2024-04-01T01:33:44.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.861+0000 7f6a435ff740 10 client.15675 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:33:44.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.861+0000 7f6a435ff740 10 client.15675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:33:44.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.861+0000 7f6a435ff740 20 client.15675 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:44.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.861+0000 7f6a435ff740 1 --2- 172.21.15.70:0/3962140775 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555d0e3c6c60 0x555d0e3c9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.861+0000 7f6a435ff740 4 client.15675 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:33:44.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.861+0000 7f6a435ff740 1 -- 172.21.15.70:0/3962140775 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555d0e2b3730 con 0x555d0e3c6c60 2024-04-01T01:33:44.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.862+0000 7f6a39d9b640 1 --2- 172.21.15.70:0/3962140775 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555d0e3c6c60 0x555d0e3c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.862+0000 7f6a21ffb640 10 client.15675 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:44.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.862+0000 7f6a39d9b640 1 --2- 172.21.15.70:0/3962140775 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555d0e3c6c60 0x555d0e3c9080 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6521f68f9d33bae server_cookie=27f91c4207789d61 in_seq=0 out_seq=0 2024-04-01T01:33:44.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.863+0000 7f6a21ffb640 1 -- 172.21.15.70:0/3962140775 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555d0e2b3730 con 0x555d0e3c6c60 2024-04-01T01:33:44.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.863+0000 7f6a21ffb640 10 client.15675 handle_command_reply: tid=0 2024-04-01T01:33:44.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.863+0000 7f6a435ff740 1 client.15675 shutdown 2024-04-01T01:33:44.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.863+0000 7f6a435ff740 1 -- 172.21.15.70:0/3962140775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e349c20 msgr2=0x555d0e358ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.863+0000 7f6a435ff740 1 --2- 172.21.15.70:0/3962140775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e349c20 0x555d0e358ec0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c003af0 tx=0x7f6a2c00de20 comp rx=0 tx=0).stop 2024-04-01T01:33:44.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.863+0000 7f6a435ff740 1 -- 172.21.15.70:0/3962140775 shutdown_connections 2024-04-01T01:33:44.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.863+0000 7f6a435ff740 1 --2- 172.21.15.70:0/3962140775 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555d0e3c6c60 0x555d0e3c9080 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.863+0000 7f6a435ff740 1 --2- 172.21.15.70:0/3962140775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d0e34f780 0x555d0e35a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.863+0000 7f6a435ff740 1 --2- 172.21.15.70:0/3962140775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d0e34a610 0x555d0e359400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.863+0000 7f6a21ffb640 0 client.15675 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:44.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.863+0000 7f6a435ff740 1 --2- 172.21.15.70:0/3962140775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d0e349c20 0x555d0e358ec0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.863+0000 7f6a435ff740 1 -- 172.21.15.70:0/3962140775 >> 172.21.15.70:0/3962140775 conn(0x555d0e345190 msgr2=0x555d0e348c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:44.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.863+0000 7f6a435ff740 1 -- 172.21.15.70:0/3962140775 shutdown_connections 2024-04-01T01:33:44.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.863+0000 7f6a435ff740 1 -- 172.21.15.70:0/3962140775 wait complete. 2024-04-01T01:33:44.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.864+0000 7f6a435ff740 20 client.15675 trim_cache size 0 max 16384 2024-04-01T01:33:44.870 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:33:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:33:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "a1f420b3-deea-4c3f-a835-77ca2c26016e", 2024-04-01T01:33:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:33:44.870 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:33:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.864+0000 7f6a40a1c640 1 -- 172.21.15.70:0/3046995617 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6a240a2530 msgr2=0x7f6a240a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.864+0000 7f6a40a1c640 1 --2- 172.21.15.70:0/3046995617 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6a240a2530 0x7f6a240a49f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c100e50 tx=0x7f6a2c005f90 comp rx=0 tx=0).stop 2024-04-01T01:33:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.864+0000 7f6a40a1c640 1 -- 172.21.15.70:0/3046995617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c100ff0 msgr2=0x7f6a3c195cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.864+0000 7f6a40a1c640 1 --2- 172.21.15.70:0/3046995617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c100ff0 0x7f6a3c195cd0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000e9f0 tx=0x7f6a3000eeb0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.864+0000 7f6a40a1c640 1 -- 172.21.15.70:0/3046995617 shutdown_connections 2024-04-01T01:33:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.864+0000 7f6a40a1c640 1 --2- 172.21.15.70:0/3046995617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a3c1019b0 0x7f6a3c19ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.864+0000 7f6a40a1c640 1 --2- 172.21.15.70:0/3046995617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c100ff0 0x7f6a3c195cd0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.864+0000 7f6a40a1c640 1 --2- 172.21.15.70:0/3046995617 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6a240a2530 0x7f6a240a49f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.864+0000 7f6a40a1c640 1 --2- 172.21.15.70:0/3046995617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a3c0ffe00 0x7f6a3c195790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.864+0000 7f6a40a1c640 1 -- 172.21.15.70:0/3046995617 >> 172.21.15.70:0/3046995617 conn(0x7f6a3c0fb350 msgr2=0x7f6a3c104310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:44.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.864+0000 7f6a40a1c640 1 -- 172.21.15.70:0/3046995617 shutdown_connections 2024-04-01T01:33:44.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.864+0000 7f6a40a1c640 1 -- 172.21.15.70:0/3046995617 wait complete. 2024-04-01T01:33:44.875 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:33:44.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:33:44.879 INFO:teuthology.orchestra.run.smithi070.stdout:total 0 2024-04-01T01:33:44.893 INFO:teuthology.orchestra.run.smithi070.stdout:Filesystem Size Used Avail Use% Mounted on 2024-04-01T01:33:44.893 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-fuse 1.1T 352M 1.1T 1% /tmp/cephfs 2024-04-01T01:33:44.895 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres' 2024-04-01T01:33:44.959 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:44 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2022537442' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:44.959 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:44 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2248629100' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:44.959 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:44 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3204140646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:44.959 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:44 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3637139791' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:44.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.984+0000 7f8145649640 1 -- 172.21.15.70:0/556319144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8140101860 msgr2=0x7f8140101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.984+0000 7f8145649640 1 --2- 172.21.15.70:0/556319144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8140101860 0x7f8140101ce0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8134009900 tx=0x7f8134030040 comp rx=0 tx=0).stop 2024-04-01T01:33:44.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.984+0000 7f8145649640 1 -- 172.21.15.70:0/556319144 shutdown_connections 2024-04-01T01:33:44.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.984+0000 7f8145649640 1 --2- 172.21.15.70:0/556319144 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8140102220 0x7f8140108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.984+0000 7f8145649640 1 --2- 172.21.15.70:0/556319144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8140101860 0x7f8140101ce0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.984+0000 7f8145649640 1 --2- 172.21.15.70:0/556319144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8140100670 0x7f8140100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.984+0000 7f8145649640 1 -- 172.21.15.70:0/556319144 >> 172.21.15.70:0/556319144 conn(0x7f81400fbc00 msgr2=0x7f81400fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:44.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.984+0000 7f8145649640 1 -- 172.21.15.70:0/556319144 shutdown_connections 2024-04-01T01:33:44.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.984+0000 7f8145649640 1 -- 172.21.15.70:0/556319144 wait complete. 2024-04-01T01:33:44.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.984+0000 7f8145649640 1 Processor -- start 2024-04-01T01:33:44.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.984+0000 7f8145649640 1 -- start start 2024-04-01T01:33:44.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f8145649640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8140100670 0x7f8140195770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f8145649640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8140101860 0x7f8140195cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f8145649640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8140102220 0x7f814019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:44.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f8145649640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f814010b0a0 con 0x7f8140100670 2024-04-01T01:33:44.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f8145649640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f814010af20 con 0x7f8140101860 2024-04-01T01:33:44.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f8145649640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f814010b220 con 0x7f8140102220 2024-04-01T01:33:44.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f813effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8140100670 0x7f8140195770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f813effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8140100670 0x7f8140195770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55684/0 (socket says 172.21.15.70:55684) 2024-04-01T01:33:44.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f813effd640 1 -- 172.21.15.70:0/3819448107 learned_addr learned my addr 172.21.15.70:0/3819448107 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:44.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f813f7fe640 1 --2- 172.21.15.70:0/3819448107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8140102220 0x7f814019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f813e7fc640 1 --2- 172.21.15.70:0/3819448107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8140101860 0x7f8140195cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:44.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f813effd640 1 -- 172.21.15.70:0/3819448107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8140102220 msgr2=0x7f814019cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f813effd640 1 --2- 172.21.15.70:0/3819448107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8140102220 0x7f814019cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f813effd640 1 -- 172.21.15.70:0/3819448107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8140101860 msgr2=0x7f8140195cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:44.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f813effd640 1 --2- 172.21.15.70:0/3819448107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8140101860 0x7f8140195cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:44.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f813effd640 1 -- 172.21.15.70:0/3819448107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f814019d390 con 0x7f8140100670 2024-04-01T01:33:44.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f813effd640 1 --2- 172.21.15.70:0/3819448107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8140100670 0x7f8140195770 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8130009960 tx=0x7f8130009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:44.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f8127fff640 1 -- 172.21.15.70:0/3819448107 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813000ca60 con 0x7f8140100670 2024-04-01T01:33:44.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f8145649640 1 -- 172.21.15.70:0/3819448107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f814019d680 con 0x7f8140100670 2024-04-01T01:33:44.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f8127fff640 1 -- 172.21.15.70:0/3819448107 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f813000cc00 con 0x7f8140100670 2024-04-01T01:33:44.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f8127fff640 1 -- 172.21.15.70:0/3819448107 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8130003d60 con 0x7f8140100670 2024-04-01T01:33:44.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.985+0000 7f8145649640 1 -- 172.21.15.70:0/3819448107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f814019dbf0 con 0x7f8140100670 2024-04-01T01:33:45.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:44 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2022537442' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:45.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:44 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2248629100' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:45.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:44 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3204140646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:45.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:44 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3637139791' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:45.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.988+0000 7f8127fff640 1 -- 172.21.15.70:0/3819448107 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f813001d020 con 0x7f8140100670 2024-04-01T01:33:45.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.988+0000 7f814821f740 1 Processor -- start 2024-04-01T01:33:45.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.988+0000 7f814821f740 1 -- start start 2024-04-01T01:33:45.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.988+0000 7f814821f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x5605881606d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:45.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.988+0000 7f814821f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588160ca0 0x5605881657a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:45.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.988+0000 7f814821f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588165e10 0x5605881682a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:45.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.988+0000 7f814821f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5605880c5180 con 0x560588165e10 2024-04-01T01:33:45.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.988+0000 7f814821f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56058806db70 con 0x5605881602b0 2024-04-01T01:33:45.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.988+0000 7f814821f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5605880c1220 con 0x560588160ca0 2024-04-01T01:33:45.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.988+0000 7f813e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588165e10 0x5605881682a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:45.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.988+0000 7f813e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588165e10 0x5605881682a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55686/0 (socket says 172.21.15.70:55686) 2024-04-01T01:33:45.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.988+0000 7f813e7fc640 1 -- 172.21.15.70:0/3466989607 learned_addr learned my addr 172.21.15.70:0/3466989607 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:45.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.988+0000 7f813effd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588160ca0 0x5605881657a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:45.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.988+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x5605881606d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:45.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.989+0000 7f813e7fc640 1 -- 172.21.15.70:0/3466989607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588160ca0 msgr2=0x5605881657a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:45.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.989+0000 7f8127fff640 1 --2- 172.21.15.70:0/3819448107 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81280a2530 0x7f81280a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:45.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.989+0000 7f813e7fc640 1 --2- 172.21.15.70:0/3466989607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588160ca0 0x5605881657a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.989+0000 7f813e7fc640 1 -- 172.21.15.70:0/3466989607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 msgr2=0x5605881606d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:45.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.989+0000 7f813e7fc640 1 --2- 172.21.15.70:0/3466989607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x5605881606d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.989+0000 7f813e7fc640 1 -- 172.21.15.70:0/3466989607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605880817c0 con 0x560588165e10 2024-04-01T01:33:45.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.989+0000 7f813f7fe640 1 --2- 172.21.15.70:0/3466989607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x5605881606d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:45.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.989+0000 7f813f7fe640 1 --2- 172.21.15.70:0/3819448107 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81280a2530 0x7f81280a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:45.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.989+0000 7f8127fff640 1 -- 172.21.15.70:0/3819448107 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f81300ce130 con 0x7f8140100670 2024-04-01T01:33:45.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.989+0000 7f813e7fc640 1 --2- 172.21.15.70:0/3466989607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588165e10 0x5605881682a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8134009900 tx=0x7f8134005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ca5cb3dbeff17e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:45.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.989+0000 7f81257fa640 1 -- 172.21.15.70:0/3466989607 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8134047070 con 0x560588165e10 2024-04-01T01:33:45.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.989+0000 7f81257fa640 1 -- 172.21.15.70:0/3466989607 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f81340046c0 con 0x560588165e10 2024-04-01T01:33:45.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.989+0000 7f81257fa640 1 -- 172.21.15.70:0/3466989607 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8134038790 con 0x560588165e10 2024-04-01T01:33:45.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.990+0000 7f814821f740 1 -- 172.21.15.70:0/3466989607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588165e10 msgr2=0x5605881682a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:45.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.990+0000 7f814821f740 1 --2- 172.21.15.70:0/3466989607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588165e10 0x5605881682a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8134009900 tx=0x7f8134005be0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.990+0000 7f813f7fe640 1 --2- 172.21.15.70:0/3819448107 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81280a2530 0x7f81280a49f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x560588161500 tx=0x7f8138004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:45.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.990+0000 7f814821f740 1 -- 172.21.15.70:0/3466989607 shutdown_connections 2024-04-01T01:33:45.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.990+0000 7f814821f740 1 --2- 172.21.15.70:0/3466989607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588165e10 0x5605881682a0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.990+0000 7f814821f740 1 --2- 172.21.15.70:0/3466989607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588160ca0 0x5605881657a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.990+0000 7f814821f740 1 --2- 172.21.15.70:0/3466989607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x5605881606d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.990+0000 7f814821f740 1 -- 172.21.15.70:0/3466989607 >> 172.21.15.70:0/3466989607 conn(0x56058815b820 msgr2=0x56058815bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:45.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.990+0000 7f814821f740 1 -- 172.21.15.70:0/3466989607 shutdown_connections 2024-04-01T01:33:45.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.990+0000 7f814821f740 1 -- 172.21.15.70:0/3466989607 wait complete. 2024-04-01T01:33:45.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.990+0000 7f814821f740 1 -- start start 2024-04-01T01:33:45.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.990+0000 7f814821f740 1 Processor -- start 2024-04-01T01:33:45.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f814821f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x56058816a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:45.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f814821f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588160ca0 0x56058816aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:45.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f814821f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588165e10 0x560588171ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:45.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f814821f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560588168d60 con 0x560588160ca0 2024-04-01T01:33:45.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f814821f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5605880634a0 con 0x5605881602b0 2024-04-01T01:33:45.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f814821f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560588168ee0 con 0x560588165e10 2024-04-01T01:33:45.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f813e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588160ca0 0x56058816aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:45.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f813effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x56058816a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:45.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f814821f740 10 client.0 _ll_register_callbacks cb 0x560588066d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:45.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f813effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x56058816a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34872/0 (socket says 172.21.15.70:34872) 2024-04-01T01:33:45.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f813e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588160ca0 0x56058816aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55694/0 (socket says 172.21.15.70:55694) 2024-04-01T01:33:45.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f813effd640 1 -- 172.21.15.70:0/3716048416 learned_addr learned my addr 172.21.15.70:0/3716048416 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:45.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f813f7fe640 1 --2- 172.21.15.70:0/3716048416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588165e10 0x560588171ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:45.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f813effd640 1 -- 172.21.15.70:0/3716048416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588165e10 msgr2=0x560588171ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:45.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f813effd640 1 --2- 172.21.15.70:0/3716048416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588165e10 0x560588171ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f813effd640 1 -- 172.21.15.70:0/3716048416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588160ca0 msgr2=0x56058816aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:45.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f813effd640 1 --2- 172.21.15.70:0/3716048416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588160ca0 0x56058816aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f813effd640 1 -- 172.21.15.70:0/3716048416 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560587ea37d0 con 0x5605881602b0 2024-04-01T01:33:45.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f813effd640 1 --2- 172.21.15.70:0/3716048416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x56058816a760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f813009a3c0 tx=0x7f8130004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1b9c4e28f76282d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:45.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f8122ff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:33:45.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f8122ff5640 1 -- 172.21.15.70:0/3716048416 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813009a200 con 0x5605881602b0 2024-04-01T01:33:45.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f8122ff5640 1 -- 172.21.15.70:0/3716048416 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f81300989c0 con 0x5605881602b0 2024-04-01T01:33:45.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f814821f740 1 -- 172.21.15.70:0/3716048416 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81340046c0 con 0x5605881602b0 2024-04-01T01:33:45.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f8122ff5640 1 -- 172.21.15.70:0/3716048416 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f813009a020 con 0x5605881602b0 2024-04-01T01:33:45.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f814821f740 10 client.25502 fetch_fsmap learned FSMap version 29 2024-04-01T01:33:45.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.991+0000 7f814821f740 1 -- 172.21.15.70:0/3716048416 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560587dd93f0 con 0x5605881602b0 2024-04-01T01:33:45.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.992+0000 7f8122ff5640 1 -- 172.21.15.70:0/3716048416 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f81340046c0 con 0x5605881602b0 2024-04-01T01:33:45.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.992+0000 7f8122ff5640 1 -- 172.21.15.70:0/3716048416 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f81300b3230 con 0x5605881602b0 2024-04-01T01:33:45.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.992+0000 7f814821f740 10 client.25502 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:33:45.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.992+0000 7f814821f740 10 client.25502 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:33:45.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.992+0000 7f814821f740 20 client.25502 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:45.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.992+0000 7f814821f740 1 --2- 172.21.15.70:0/3716048416 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5605881765e0 0x560588178a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:45.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.992+0000 7f814821f740 4 client.25502 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:33:45.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.992+0000 7f814821f740 1 -- 172.21.15.70:0/3716048416 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605880d26c0 con 0x5605881765e0 2024-04-01T01:33:45.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.992+0000 7f813e7fc640 1 --2- 172.21.15.70:0/3716048416 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5605881765e0 0x560588178a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:45.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.992+0000 7f8122ff5640 10 client.25502 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:45.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.992+0000 7f813e7fc640 1 --2- 172.21.15.70:0/3716048416 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5605881765e0 0x560588178a50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5ebd4b284063f04 server_cookie=f33f2a1722cfbb4b in_seq=0 out_seq=0 2024-04-01T01:33:45.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.993+0000 7f8122ff5640 1 -- 172.21.15.70:0/3716048416 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5605880d26c0 con 0x5605881765e0 2024-04-01T01:33:45.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.993+0000 7f8122ff5640 10 client.25502 handle_command_reply: tid=0 2024-04-01T01:33:45.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.993+0000 7f814821f740 1 client.25502 shutdown 2024-04-01T01:33:45.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.993+0000 7f814821f740 1 -- 172.21.15.70:0/3716048416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 msgr2=0x56058816a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:45.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.993+0000 7f814821f740 1 --2- 172.21.15.70:0/3716048416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x56058816a760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f813009a3c0 tx=0x7f8130004730 comp rx=0 tx=0).stop 2024-04-01T01:33:45.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.993+0000 7f814821f740 1 -- 172.21.15.70:0/3716048416 shutdown_connections 2024-04-01T01:33:45.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.993+0000 7f814821f740 1 --2- 172.21.15.70:0/3716048416 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5605881765e0 0x560588178a50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.993+0000 7f814821f740 1 --2- 172.21.15.70:0/3716048416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588165e10 0x560588171ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.993+0000 7f8122ff5640 0 client.25502 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:45.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.993+0000 7f814821f740 1 --2- 172.21.15.70:0/3716048416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588160ca0 0x56058816aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.993+0000 7f814821f740 1 --2- 172.21.15.70:0/3716048416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x56058816a760 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.993+0000 7f814821f740 1 -- 172.21.15.70:0/3716048416 >> 172.21.15.70:0/3716048416 conn(0x56058815b820 msgr2=0x56058815ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:45.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.993+0000 7f814821f740 1 -- 172.21.15.70:0/3716048416 shutdown_connections 2024-04-01T01:33:45.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.993+0000 7f814821f740 1 -- 172.21.15.70:0/3716048416 wait complete. 2024-04-01T01:33:45.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:44.993+0000 7f814821f740 20 client.25502 trim_cache size 0 max 16384 2024-04-01T01:33:45.021 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data' 2024-04-01T01:33:45.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.010+0000 7f814821f740 1 Processor -- start 2024-04-01T01:33:45.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.010+0000 7f814821f740 1 -- start start 2024-04-01T01:33:45.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.010+0000 7f814821f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x56058816aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:45.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.010+0000 7f814821f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588160ca0 0x560588171ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:45.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.010+0000 7f814821f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588165e10 0x56058815a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.010+0000 7f814821f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f81401a80e0 con 0x560588165e10 2024-04-01T01:33:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.010+0000 7f814821f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f81401a9c20 con 0x5605881602b0 2024-04-01T01:33:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.011+0000 7f814821f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f81401a87b0 con 0x560588160ca0 2024-04-01T01:33:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.011+0000 7f813effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x56058816aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.011+0000 7f813effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x56058816aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34880/0 (socket says 172.21.15.70:34880) 2024-04-01T01:33:45.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.011+0000 7f813effd640 1 -- 172.21.15.70:0/3471735661 learned_addr learned my addr 172.21.15.70:0/3471735661 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.011+0000 7f813f7fe640 1 --2- 172.21.15.70:0/3471735661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588165e10 0x56058815a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.011+0000 7f813e7fc640 1 --2- 172.21.15.70:0/3471735661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588160ca0 0x560588171ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.011+0000 7f813effd640 1 -- 172.21.15.70:0/3471735661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588160ca0 msgr2=0x560588171ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.011+0000 7f813effd640 1 --2- 172.21.15.70:0/3471735661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588160ca0 0x560588171ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.011+0000 7f813effd640 1 -- 172.21.15.70:0/3471735661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588165e10 msgr2=0x56058815a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:45.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.011+0000 7f813effd640 1 --2- 172.21.15.70:0/3471735661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588165e10 0x56058815a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.011+0000 7f813effd640 1 -- 172.21.15.70:0/3471735661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560588169820 con 0x5605881602b0 2024-04-01T01:33:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.011+0000 7f813effd640 1 --2- 172.21.15.70:0/3471735661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x56058816aa80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f813000c640 tx=0x7f8130005e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24a6fac20a966af9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.011+0000 7f8124ff9640 1 -- 172.21.15.70:0/3471735661 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813009a200 con 0x5605881602b0 2024-04-01T01:33:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.011+0000 7f8124ff9640 1 -- 172.21.15.70:0/3471735661 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f81300a2e80 con 0x5605881602b0 2024-04-01T01:33:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.011+0000 7f8124ff9640 1 -- 172.21.15.70:0/3471735661 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8130096410 con 0x5605881602b0 2024-04-01T01:33:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 -- 172.21.15.70:0/3471735661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 msgr2=0x56058816aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:45.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 --2- 172.21.15.70:0/3471735661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x56058816aa80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f813000c640 tx=0x7f8130005e30 comp rx=0 tx=0).stop 2024-04-01T01:33:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 -- 172.21.15.70:0/3471735661 shutdown_connections 2024-04-01T01:33:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 --2- 172.21.15.70:0/3471735661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560588165e10 0x56058815a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 --2- 172.21.15.70:0/3471735661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588160ca0 0x560588171ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 --2- 172.21.15.70:0/3471735661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605881602b0 0x56058816aa80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 -- 172.21.15.70:0/3471735661 >> 172.21.15.70:0/3471735661 conn(0x56058815b820 msgr2=0x56058816a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:45.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 -- 172.21.15.70:0/3471735661 shutdown_connections 2024-04-01T01:33:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 -- 172.21.15.70:0/3471735661 wait complete. 2024-04-01T01:33:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 -- start start 2024-04-01T01:33:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 Processor -- start 2024-04-01T01:33:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605881602b0 0x56058816f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588160ca0 0x56058816f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:45.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560588165e10 0x5605881709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:45.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f81401a9550 con 0x5605881602b0 2024-04-01T01:33:45.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f81401a8e80 con 0x560588165e10 2024-04-01T01:33:45.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.012+0000 7f814821f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f81401a60c0 con 0x560588160ca0 2024-04-01T01:33:45.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f814821f740 10 client.0 _ll_register_callbacks cb 0x7f813000df50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:45.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560588165e10 0x5605881709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:45.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f813effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605881602b0 0x56058816f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:45.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f813f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560588165e10 0x5605881709c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34886/0 (socket says 172.21.15.70:34886) 2024-04-01T01:33:45.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f813f7fe640 1 -- 172.21.15.70:0/942907864 learned_addr learned my addr 172.21.15.70:0/942907864 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:45.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f813e7fc640 1 --2- 172.21.15.70:0/942907864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588160ca0 0x56058816f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:45.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f813f7fe640 1 -- 172.21.15.70:0/942907864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588160ca0 msgr2=0x56058816f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:45.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f813f7fe640 1 --2- 172.21.15.70:0/942907864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588160ca0 0x56058816f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f813f7fe640 1 -- 172.21.15.70:0/942907864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605881602b0 msgr2=0x56058816f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:45.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f813f7fe640 1 --2- 172.21.15.70:0/942907864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605881602b0 0x56058816f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f813f7fe640 1 -- 172.21.15.70:0/942907864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560587d990f0 con 0x560588165e10 2024-04-01T01:33:45.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f813f7fe640 1 --2- 172.21.15.70:0/942907864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560588165e10 0x5605881709c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f813800fff0 tx=0x7f8138004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92402ca19d65ecbe server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:45.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f8122ff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:33:45.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f8122ff5640 1 -- 172.21.15.70:0/942907864 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813802e060 con 0x560588165e10 2024-04-01T01:33:45.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f814821f740 1 -- 172.21.15.70:0/942907864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81300a2e80 con 0x560588165e10 2024-04-01T01:33:45.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f8122ff5640 1 -- 172.21.15.70:0/942907864 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f813801c410 con 0x560588165e10 2024-04-01T01:33:45.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f8122ff5640 1 -- 172.21.15.70:0/942907864 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8138035050 con 0x560588165e10 2024-04-01T01:33:45.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f814821f740 10 client.25508 fetch_fsmap learned FSMap version 29 2024-04-01T01:33:45.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.013+0000 7f814821f740 1 -- 172.21.15.70:0/942907864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5605881dc950 con 0x560588165e10 2024-04-01T01:33:45.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f8122ff5640 1 -- 172.21.15.70:0/942907864 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f81300a2e80 con 0x560588165e10 2024-04-01T01:33:45.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f8122ff5640 1 -- 172.21.15.70:0/942907864 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8138028070 con 0x560588165e10 2024-04-01T01:33:45.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f814821f740 10 client.25508 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:33:45.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f814821f740 10 client.25508 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:33:45.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f814821f740 20 client.25508 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:45.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f814821f740 1 --2- 172.21.15.70:0/942907864 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5605881dd290 0x5605881df6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:45.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f814821f740 4 client.25508 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:33:45.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f814821f740 1 -- 172.21.15.70:0/942907864 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5605880cca10 con 0x5605881dd290 2024-04-01T01:33:45.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f813effd640 1 --2- 172.21.15.70:0/942907864 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5605881dd290 0x5605881df6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:45.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f8122ff5640 10 client.25508 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:45.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f813effd640 1 --2- 172.21.15.70:0/942907864 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5605881dd290 0x5605881df6b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=509673a948ae84c2 server_cookie=e301c47f8613c658 in_seq=0 out_seq=0 2024-04-01T01:33:45.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f8122ff5640 1 -- 172.21.15.70:0/942907864 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5605880cca10 con 0x5605881dd290 2024-04-01T01:33:45.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f8122ff5640 10 client.25508 handle_command_reply: tid=0 2024-04-01T01:33:45.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f814821f740 1 client.25508 shutdown 2024-04-01T01:33:45.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f814821f740 1 -- 172.21.15.70:0/942907864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560588165e10 msgr2=0x5605881709c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:45.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f814821f740 1 --2- 172.21.15.70:0/942907864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560588165e10 0x5605881709c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f813800fff0 tx=0x7f8138004ab0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f814821f740 1 -- 172.21.15.70:0/942907864 shutdown_connections 2024-04-01T01:33:45.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f814821f740 1 --2- 172.21.15.70:0/942907864 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5605881dd290 0x5605881df6b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f814821f740 1 --2- 172.21.15.70:0/942907864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560588165e10 0x5605881709c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f8122ff5640 0 client.25508 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:45.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f814821f740 1 --2- 172.21.15.70:0/942907864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560588160ca0 0x56058816f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f814821f740 1 --2- 172.21.15.70:0/942907864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605881602b0 0x56058816f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.014+0000 7f814821f740 1 -- 172.21.15.70:0/942907864 >> 172.21.15.70:0/942907864 conn(0x56058815b820 msgr2=0x56058815ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:45.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.015+0000 7f814821f740 1 -- 172.21.15.70:0/942907864 shutdown_connections 2024-04-01T01:33:45.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.015+0000 7f814821f740 1 -- 172.21.15.70:0/942907864 wait complete. 2024-04-01T01:33:45.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.015+0000 7f814821f740 20 client.25508 trim_cache size 0 max 16384 2024-04-01T01:33:45.046 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:33:45.046 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:33:45.046 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:33:45.046 INFO:teuthology.orchestra.run.smithi070.stdout: "a1f420b3-deea-4c3f-a835-77ca2c26016e": { 2024-04-01T01:33:45.046 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:33:45.046 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "a1f420b3-deea-4c3f-a835-77ca2c26016e", 2024-04-01T01:33:45.047 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:33:45.047 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:33:45.047 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:33:45.047 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:33:45.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.015+0000 7f8145649640 1 -- 172.21.15.70:0/3819448107 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81280a2530 msgr2=0x7f81280a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:45.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.015+0000 7f8145649640 1 --2- 172.21.15.70:0/3819448107 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81280a2530 0x7f81280a49f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x560588161500 tx=0x7f8138004a40 comp rx=0 tx=0).stop 2024-04-01T01:33:45.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.015+0000 7f8145649640 1 -- 172.21.15.70:0/3819448107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8140100670 msgr2=0x7f8140195770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:45.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.015+0000 7f8145649640 1 --2- 172.21.15.70:0/3819448107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8140100670 0x7f8140195770 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8130009960 tx=0x7f8130009e20 comp rx=0 tx=0).stop 2024-04-01T01:33:45.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.015+0000 7f8145649640 1 -- 172.21.15.70:0/3819448107 shutdown_connections 2024-04-01T01:33:45.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.015+0000 7f8145649640 1 --2- 172.21.15.70:0/3819448107 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81280a2530 0x7f81280a49f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.015+0000 7f8145649640 1 --2- 172.21.15.70:0/3819448107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8140102220 0x7f814019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.015+0000 7f8145649640 1 --2- 172.21.15.70:0/3819448107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8140101860 0x7f8140195cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.015+0000 7f8145649640 1 --2- 172.21.15.70:0/3819448107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8140100670 0x7f8140195770 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:45.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.015+0000 7f8145649640 1 -- 172.21.15.70:0/3819448107 >> 172.21.15.70:0/3819448107 conn(0x7f81400fbc00 msgr2=0x7f8140104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:45.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.015+0000 7f8145649640 1 -- 172.21.15.70:0/3819448107 shutdown_connections 2024-04-01T01:33:45.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:33:45.015+0000 7f8145649640 1 -- 172.21.15.70:0/3819448107 wait complete. 2024-04-01T01:33:45.049 INFO:tasks.cephfs.filesystem:scrub status for tag:a1f420b3-deea-4c3f-a835-77ca2c26016e - {'path': '/', 'tag': 'a1f420b3-deea-4c3f-a835-77ca2c26016e', 'options': 'recursive,force'} 2024-04-01T01:33:45.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:44 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2022537442' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:33:45.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:44 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2248629100' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:33:45.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:44 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3204140646' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:33:45.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:44 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3637139791' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:33:45.132 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?"' 2024-04-01T01:33:45.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:45 smithi189 ceph-mon[35753]: pgmap v366: 129 pgs: 129 active+clean; 654 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-01T01:33:45.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:45 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:33:45.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:45 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:33:45.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:45 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:33:45.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:45 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:33:46.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:45 smithi070 ceph-mon[32179]: pgmap v366: 129 pgs: 129 active+clean; 654 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-01T01:33:46.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:45 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:33:46.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:45 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:33:46.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:45 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:33:46.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:45 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:33:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:45 smithi081 ceph-mon[35142]: pgmap v366: 129 pgs: 129 active+clean; 654 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-04-01T01:33:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:45 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:33:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:45 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:33:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:45 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:33:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:45 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:33:46.717 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local' 2024-04-01T01:33:46.758 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres(/.*)? system_u:object_r:postgresql_db_t:s0 2024-04-01T01:33:46.761 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo restorecon -R -v /tmp/cephfs/postgres' 2024-04-01T01:33:46.845 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-01T01:33:46.929 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/: 2024-04-01T01:33:46.929 INFO:teuthology.orchestra.run.smithi070.stdout:total 2 2024-04-01T01:33:46.930 INFO:teuthology.orchestra.run.smithi070.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:33:46.930 INFO:teuthology.orchestra.run.smithi070.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 .. 2024-04-01T01:33:46.930 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 data 2024-04-01T01:33:46.930 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:33:46.930 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data: 2024-04-01T01:33:46.930 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:33:46.930 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:33:46.930 INFO:teuthology.orchestra.run.smithi070.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 .. 2024-04-01T01:33:46.932 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '"'"'[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n'"'"' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf' 2024-04-01T01:33:46.981 INFO:teuthology.orchestra.run.smithi070.stdout:[Service] 2024-04-01T01:33:46.982 INFO:teuthology.orchestra.run.smithi070.stdout:Environment=PGDATA=/tmp/cephfs/postgres/data 2024-04-01T01:33:46.982 INFO:teuthology.orchestra.run.smithi070.stdout:Environment=PGLOG=/tmp/cephfs/postgres/pgstartup.log 2024-04-01T01:33:46.984 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- postgresql-setup --initdb' 2024-04-01T01:33:47.063 INFO:teuthology.orchestra.run.smithi070.stderr: * Initializing database in '/tmp/cephfs/postgres/data' 2024-04-01T01:33:47.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:47 smithi189 ceph-mon[35753]: pgmap v367: 129 pgs: 129 active+clean; 664 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.9 KiB/s wr, 15 op/s 2024-04-01T01:33:48.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:47 smithi070 ceph-mon[32179]: pgmap v367: 129 pgs: 129 active+clean; 664 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.9 KiB/s wr, 15 op/s 2024-04-01T01:33:48.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:47 smithi081 ceph-mon[35142]: pgmap v367: 129 pgs: 129 active+clean; 664 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.9 KiB/s wr, 15 op/s 2024-04-01T01:33:49.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:49 smithi189 ceph-mon[35753]: pgmap v368: 129 pgs: 129 active+clean; 952 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 26 KiB/s wr, 12 op/s 2024-04-01T01:33:50.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:49 smithi070 ceph-mon[32179]: pgmap v368: 129 pgs: 129 active+clean; 952 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 26 KiB/s wr, 12 op/s 2024-04-01T01:33:50.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:49 smithi081 ceph-mon[35142]: pgmap v368: 129 pgs: 129 active+clean; 952 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 26 KiB/s wr, 12 op/s 2024-04-01T01:33:51.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:51 smithi189 ceph-mon[35753]: pgmap v369: 129 pgs: 129 active+clean; 14 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 MiB/s wr, 20 op/s 2024-04-01T01:33:51.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:51 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:33:52.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:51 smithi070 ceph-mon[32179]: pgmap v369: 129 pgs: 129 active+clean; 14 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 MiB/s wr, 20 op/s 2024-04-01T01:33:52.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:51 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:33:52.040 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:51 smithi081 ceph-mon[35142]: pgmap v369: 129 pgs: 129 active+clean; 14 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.2 MiB/s wr, 20 op/s 2024-04-01T01:33:52.040 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:51 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:33:53.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:53 smithi189 ceph-mon[35753]: pgmap v370: 129 pgs: 129 active+clean; 26 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 2.6 MiB/s wr, 54 op/s 2024-04-01T01:33:53.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:53 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:53.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:53 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:53.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:53 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:53.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:53 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:53.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:53 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:53.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:53 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:54.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:53 smithi070 ceph-mon[32179]: pgmap v370: 129 pgs: 129 active+clean; 26 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 2.6 MiB/s wr, 54 op/s 2024-04-01T01:33:54.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:53 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:54.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:53 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:54.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:53 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:54.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:53 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:54.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:53 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:54.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:53 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:54.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:53 smithi081 ceph-mon[35142]: pgmap v370: 129 pgs: 129 active+clean; 26 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 2.6 MiB/s wr, 54 op/s 2024-04-01T01:33:54.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:53 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:54.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:53 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:54.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:53 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:54.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:53 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:54.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:53 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:54.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:53 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:54.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:54 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:33:54.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:54 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:33:54.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:54 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:55.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:54 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:33:55.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:54 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:33:55.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:54 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:55.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:54 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:33:55.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:54 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:33:55.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:54 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:33:55.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:55 smithi189 ceph-mon[35753]: pgmap v371: 129 pgs: 129 active+clean; 27 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 2.8 MiB/s wr, 60 op/s 2024-04-01T01:33:56.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:55 smithi070 ceph-mon[32179]: pgmap v371: 129 pgs: 129 active+clean; 27 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 2.8 MiB/s wr, 60 op/s 2024-04-01T01:33:56.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:55 smithi081 ceph-mon[35142]: pgmap v371: 129 pgs: 129 active+clean; 27 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 2.8 MiB/s wr, 60 op/s 2024-04-01T01:33:56.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:56 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:33:57.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:56 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:33:57.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:56 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:33:57.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.858+0000 7f59c6702640 1 -- 172.21.15.70:0/2763701358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c0101860 msgr2=0x7f59c0101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:57.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.858+0000 7f59c6702640 1 --2- 172.21.15.70:0/2763701358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c0101860 0x7f59c0101ce0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009900 tx=0x7f59b402bc30 comp rx=0 tx=0).stop 2024-04-01T01:33:57.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.858+0000 7f59c6702640 1 -- 172.21.15.70:0/2763701358 shutdown_connections 2024-04-01T01:33:57.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.858+0000 7f59c6702640 1 --2- 172.21.15.70:0/2763701358 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59c0102220 0x7f59c0108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.858+0000 7f59c6702640 1 --2- 172.21.15.70:0/2763701358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c0101860 0x7f59c0101ce0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.858+0000 7f59c6702640 1 --2- 172.21.15.70:0/2763701358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59c0100670 0x7f59c0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.858+0000 7f59c6702640 1 -- 172.21.15.70:0/2763701358 >> 172.21.15.70:0/2763701358 conn(0x7f59c00fbbe0 msgr2=0x7f59c00fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:57.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.858+0000 7f59c6702640 1 -- 172.21.15.70:0/2763701358 shutdown_connections 2024-04-01T01:33:57.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.858+0000 7f59c6702640 1 -- 172.21.15.70:0/2763701358 wait complete. 2024-04-01T01:33:57.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.858+0000 7f59c6702640 1 Processor -- start 2024-04-01T01:33:57.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.858+0000 7f59c6702640 1 -- start start 2024-04-01T01:33:57.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59c6702640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c0100670 0x7f59c0197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59c6702640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59c0101860 0x7f59c0197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59c6702640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59c0102220 0x7f59c019eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59c6702640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f59c010c190 con 0x7f59c0100670 2024-04-01T01:33:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59c6702640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f59c010c010 con 0x7f59c0101860 2024-04-01T01:33:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59c6702640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f59c010c310 con 0x7f59c0102220 2024-04-01T01:33:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59bffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c0100670 0x7f59c0197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59bffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c0100670 0x7f59c0197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57036/0 (socket says 172.21.15.70:57036) 2024-04-01T01:33:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59bffff640 1 -- 172.21.15.70:0/1708591486 learned_addr learned my addr 172.21.15.70:0/1708591486 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59bf7fe640 1 --2- 172.21.15.70:0/1708591486 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59c0101860 0x7f59c0197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59c4c9f640 1 --2- 172.21.15.70:0/1708591486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59c0102220 0x7f59c019eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59bffff640 1 -- 172.21.15.70:0/1708591486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59c0102220 msgr2=0x7f59c019eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59bffff640 1 --2- 172.21.15.70:0/1708591486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59c0102220 0x7f59c019eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59bffff640 1 -- 172.21.15.70:0/1708591486 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59c0101860 msgr2=0x7f59c0197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59bffff640 1 --2- 172.21.15.70:0/1708591486 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59c0101860 0x7f59c0197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59bffff640 1 -- 172.21.15.70:0/1708591486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c019f560 con 0x7f59c0100670 2024-04-01T01:33:57.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59bf7fe640 1 --2- 172.21.15.70:0/1708591486 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59c0101860 0x7f59c0197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:57.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59bffff640 1 --2- 172.21.15.70:0/1708591486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c0100670 0x7f59c0197990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f59b000e9c0 tx=0x7f59b000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:57.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59bd7fa640 1 -- 172.21.15.70:0/1708591486 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b000cd80 con 0x7f59c0100670 2024-04-01T01:33:57.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59bd7fa640 1 -- 172.21.15.70:0/1708591486 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59b00108e0 con 0x7f59c0100670 2024-04-01T01:33:57.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59bd7fa640 1 -- 172.21.15.70:0/1708591486 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b00188e0 con 0x7f59c0100670 2024-04-01T01:33:57.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59c6702640 1 -- 172.21.15.70:0/1708591486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59c019f850 con 0x7f59c0100670 2024-04-01T01:33:57.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.859+0000 7f59c6702640 1 -- 172.21.15.70:0/1708591486 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59c019fdc0 con 0x7f59c0100670 2024-04-01T01:33:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.862+0000 7f59bd7fa640 1 -- 172.21.15.70:0/1708591486 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f59b001f020 con 0x7f59c0100670 2024-04-01T01:33:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.862+0000 7f59c93c4740 1 Processor -- start 2024-04-01T01:33:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.862+0000 7f59c93c4740 1 -- start start 2024-04-01T01:33:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.862+0000 7f59c93c4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb12018d0 0x55abb1201cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.862+0000 7f59c93c4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12022c0 0x55abb1206dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.862+0000 7f59c93c4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb1207430 0x55abb12098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.862+0000 7f59c93c4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55abb11658f0 con 0x55abb12022c0 2024-04-01T01:33:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.862+0000 7f59c93c4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55abb110e8d0 con 0x55abb12018d0 2024-04-01T01:33:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.862+0000 7f59c93c4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55abb1161990 con 0x55abb1207430 2024-04-01T01:33:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.862+0000 7f59bffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12022c0 0x55abb1206dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.862+0000 7f59bffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12022c0 0x55abb1206dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57042/0 (socket says 172.21.15.70:57042) 2024-04-01T01:33:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.862+0000 7f59bf7fe640 1 --2- 172.21.15.70:0/1535176692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb1207430 0x55abb12098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.862+0000 7f59bffff640 1 -- 172.21.15.70:0/1535176692 learned_addr learned my addr 172.21.15.70:0/1535176692 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.862+0000 7f59c4c9f640 1 --2- 172.21.15.70:0/1535176692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb12018d0 0x55abb1201cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.862+0000 7f59bd7fa640 1 --2- 172.21.15.70:0/1708591486 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59a80a2660 0x7f59a80a4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.863+0000 7f59bf7fe640 1 -- 172.21.15.70:0/1535176692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb12018d0 msgr2=0x55abb1201cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.863+0000 7f59bf7fe640 1 --2- 172.21.15.70:0/1535176692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb12018d0 0x55abb1201cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.863+0000 7f59bf7fe640 1 -- 172.21.15.70:0/1535176692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12022c0 msgr2=0x55abb1206dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.863+0000 7f59bf7fe640 1 --2- 172.21.15.70:0/1535176692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12022c0 0x55abb1206dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.863+0000 7f59bf7fe640 1 -- 172.21.15.70:0/1535176692 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abb0e7a210 con 0x55abb1207430 2024-04-01T01:33:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.863+0000 7f59c4c9f640 1 --2- 172.21.15.70:0/1535176692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb12018d0 0x55abb1201cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:33:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.863+0000 7f59bffff640 1 --2- 172.21.15.70:0/1535176692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12022c0 0x55abb1206dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.863+0000 7f59c4c9f640 1 --2- 172.21.15.70:0/1708591486 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59a80a2660 0x7f59a80a4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.863+0000 7f59bd7fa640 1 -- 172.21.15.70:0/1708591486 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f59b00cec70 con 0x7f59c0100670 2024-04-01T01:33:57.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.863+0000 7f59bf7fe640 1 --2- 172.21.15.70:0/1535176692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb1207430 0x55abb12098c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f59b4000c00 tx=0x7f59b40095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=633df88d661badcc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.863+0000 7f59a67fc640 1 -- 172.21.15.70:0/1535176692 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b4045020 con 0x55abb1207430 2024-04-01T01:33:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.863+0000 7f59c4c9f640 1 --2- 172.21.15.70:0/1708591486 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59a80a2660 0x7f59a80a4b20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f59b8009fd0 tx=0x7f59b8014040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.863+0000 7f59a67fc640 1 -- 172.21.15.70:0/1535176692 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59b402cc90 con 0x55abb1207430 2024-04-01T01:33:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.863+0000 7f59a67fc640 1 -- 172.21.15.70:0/1535176692 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b4004920 con 0x55abb1207430 2024-04-01T01:33:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.864+0000 7f59c93c4740 1 -- 172.21.15.70:0/1535176692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb1207430 msgr2=0x55abb12098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.864+0000 7f59c93c4740 1 --2- 172.21.15.70:0/1535176692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb1207430 0x55abb12098c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f59b4000c00 tx=0x7f59b40095f0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.864+0000 7f59c93c4740 1 -- 172.21.15.70:0/1535176692 shutdown_connections 2024-04-01T01:33:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.864+0000 7f59c93c4740 1 --2- 172.21.15.70:0/1535176692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb1207430 0x55abb12098c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.864+0000 7f59c93c4740 1 --2- 172.21.15.70:0/1535176692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12022c0 0x55abb1206dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.864+0000 7f59c93c4740 1 --2- 172.21.15.70:0/1535176692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb12018d0 0x55abb1201cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.864+0000 7f59c93c4740 1 -- 172.21.15.70:0/1535176692 >> 172.21.15.70:0/1535176692 conn(0x55abb11fce40 msgr2=0x55abb11fd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.864+0000 7f59c93c4740 1 -- 172.21.15.70:0/1535176692 shutdown_connections 2024-04-01T01:33:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.864+0000 7f59c93c4740 1 -- 172.21.15.70:0/1535176692 wait complete. 2024-04-01T01:33:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.864+0000 7f59c93c4740 1 -- start start 2024-04-01T01:33:57.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.864+0000 7f59c93c4740 1 Processor -- start 2024-04-01T01:33:57.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.864+0000 7f59c93c4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb12018d0 0x55abb120bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59c93c4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12022c0 0x55abb120c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59c93c4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb1207430 0x55abb12130a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59c93c4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55abb120a380 con 0x55abb12022c0 2024-04-01T01:33:57.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59c93c4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55abb1103f80 con 0x55abb12018d0 2024-04-01T01:33:57.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59c93c4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55abb120a500 con 0x55abb1207430 2024-04-01T01:33:57.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59c93c4740 10 client.0 _ll_register_callbacks cb 0x55abb115c290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:57.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59bf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12022c0 0x55abb120c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59bf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12022c0 0x55abb120c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57044/0 (socket says 172.21.15.70:57044) 2024-04-01T01:33:57.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59bffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb12018d0 0x55abb120bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59bf7fe640 1 -- 172.21.15.70:0/3593398885 learned_addr learned my addr 172.21.15.70:0/3593398885 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:57.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59c4c9f640 1 --2- 172.21.15.70:0/3593398885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb1207430 0x55abb12130a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59bf7fe640 1 -- 172.21.15.70:0/3593398885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb1207430 msgr2=0x55abb12130a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59bf7fe640 1 --2- 172.21.15.70:0/3593398885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb1207430 0x55abb12130a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59bf7fe640 1 -- 172.21.15.70:0/3593398885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb12018d0 msgr2=0x55abb120bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59bf7fe640 1 --2- 172.21.15.70:0/3593398885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb12018d0 0x55abb120bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59bf7fe640 1 -- 172.21.15.70:0/3593398885 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abb0f46620 con 0x55abb12022c0 2024-04-01T01:33:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59bf7fe640 1 --2- 172.21.15.70:0/3593398885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12022c0 0x55abb120c060 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f59b4036040 tx=0x7f59b40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692c42008c9309c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f599f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:33:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f599f7fe640 1 -- 172.21.15.70:0/3593398885 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b4045040 con 0x55abb12022c0 2024-04-01T01:33:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59c93c4740 1 -- 172.21.15.70:0/3593398885 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59b402cc90 con 0x55abb12022c0 2024-04-01T01:33:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f599f7fe640 1 -- 172.21.15.70:0/3593398885 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f59b4041480 con 0x55abb12022c0 2024-04-01T01:33:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f599f7fe640 1 -- 172.21.15.70:0/3593398885 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f59b405a050 con 0x55abb12022c0 2024-04-01T01:33:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59c93c4740 10 client.15705 fetch_fsmap learned FSMap version 29 2024-04-01T01:33:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.865+0000 7f59c93c4740 1 -- 172.21.15.70:0/3593398885 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55abb1216080 con 0x55abb12022c0 2024-04-01T01:33:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.866+0000 7f599f7fe640 1 -- 172.21.15.70:0/3593398885 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59b402cc90 con 0x55abb12022c0 2024-04-01T01:33:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.866+0000 7f599f7fe640 1 -- 172.21.15.70:0/3593398885 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f59b404c460 con 0x55abb12022c0 2024-04-01T01:33:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.866+0000 7f59c93c4740 10 client.15705 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:33:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.866+0000 7f59c93c4740 10 client.15705 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:33:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.866+0000 7f59c93c4740 20 client.15705 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.866+0000 7f59c93c4740 1 --2- 172.21.15.70:0/3593398885 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55abb1217470 0x55abb12198e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.866+0000 7f59c93c4740 4 client.15705 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:33:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.866+0000 7f59c93c4740 1 -- 172.21.15.70:0/3593398885 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55abb1179120 con 0x55abb1217470 2024-04-01T01:33:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.866+0000 7f59bffff640 1 --2- 172.21.15.70:0/3593398885 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55abb1217470 0x55abb12198e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.866+0000 7f599f7fe640 10 client.15705 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.866+0000 7f59bffff640 1 --2- 172.21.15.70:0/3593398885 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55abb1217470 0x55abb12198e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dd301d238edaf01 server_cookie=6323033112c2ad19 in_seq=0 out_seq=0 2024-04-01T01:33:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.867+0000 7f599f7fe640 1 -- 172.21.15.70:0/3593398885 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55abb1179120 con 0x55abb1217470 2024-04-01T01:33:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.867+0000 7f599f7fe640 10 client.15705 handle_command_reply: tid=0 2024-04-01T01:33:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.867+0000 7f59c93c4740 1 client.15705 shutdown 2024-04-01T01:33:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.867+0000 7f59c93c4740 1 -- 172.21.15.70:0/3593398885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12022c0 msgr2=0x55abb120c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:57.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.867+0000 7f59c93c4740 1 --2- 172.21.15.70:0/3593398885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12022c0 0x55abb120c060 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f59b4036040 tx=0x7f59b40038a0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.867+0000 7f59c93c4740 1 -- 172.21.15.70:0/3593398885 shutdown_connections 2024-04-01T01:33:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.867+0000 7f59c93c4740 1 --2- 172.21.15.70:0/3593398885 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55abb1217470 0x55abb12198e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.867+0000 7f59c93c4740 1 --2- 172.21.15.70:0/3593398885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb1207430 0x55abb12130a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.867+0000 7f599f7fe640 0 client.15705 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.867+0000 7f59c93c4740 1 --2- 172.21.15.70:0/3593398885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12022c0 0x55abb120c060 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.867+0000 7f59c93c4740 1 --2- 172.21.15.70:0/3593398885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb12018d0 0x55abb120bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.867+0000 7f59c93c4740 1 -- 172.21.15.70:0/3593398885 >> 172.21.15.70:0/3593398885 conn(0x55abb11fce40 msgr2=0x55abb11fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.867+0000 7f59c93c4740 1 -- 172.21.15.70:0/3593398885 shutdown_connections 2024-04-01T01:33:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.867+0000 7f59c93c4740 1 -- 172.21.15.70:0/3593398885 wait complete. 2024-04-01T01:33:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.867+0000 7f59c93c4740 20 client.15705 trim_cache size 0 max 16384 2024-04-01T01:33:57.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59c93c4740 1 Processor -- start 2024-04-01T01:33:57.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59c93c4740 1 -- start start 2024-04-01T01:33:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59c93c4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12018d0 0x55abb120b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59c93c4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb12022c0 0x55abb1213060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59c93c4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb1207430 0x55abb11fd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59c93c4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f59c01aa1f0 con 0x55abb12018d0 2024-04-01T01:33:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59c93c4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f59c01abd30 con 0x55abb1207430 2024-04-01T01:33:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59c93c4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f59c01aa8c0 con 0x55abb12022c0 2024-04-01T01:33:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59bffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12018d0 0x55abb120b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59bffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12018d0 0x55abb120b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57058/0 (socket says 172.21.15.70:57058) 2024-04-01T01:33:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59bffff640 1 -- 172.21.15.70:0/2022109947 learned_addr learned my addr 172.21.15.70:0/2022109947 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59c4c9f640 1 --2- 172.21.15.70:0/2022109947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb1207430 0x55abb11fd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59bf7fe640 1 --2- 172.21.15.70:0/2022109947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb12022c0 0x55abb1213060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59bffff640 1 -- 172.21.15.70:0/2022109947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb12022c0 msgr2=0x55abb1213060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:57.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59bffff640 1 --2- 172.21.15.70:0/2022109947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb12022c0 0x55abb1213060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59bffff640 1 -- 172.21.15.70:0/2022109947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb1207430 msgr2=0x55abb11fd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:57.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59bffff640 1 --2- 172.21.15.70:0/2022109947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb1207430 0x55abb11fd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59bffff640 1 -- 172.21.15.70:0/2022109947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abb120c9c0 con 0x55abb12018d0 2024-04-01T01:33:57.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59bf7fe640 1 --2- 172.21.15.70:0/2022109947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb12022c0 0x55abb1213060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:33:57.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59bffff640 1 --2- 172.21.15.70:0/2022109947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12018d0 0x55abb120b140 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f59b000c6a0 tx=0x7f59b00d27c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6561b7724f200d30 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:57.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59a5ffb640 1 -- 172.21.15.70:0/2022109947 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b009ab60 con 0x55abb12018d0 2024-04-01T01:33:57.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59a5ffb640 1 -- 172.21.15.70:0/2022109947 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59b0010e20 con 0x55abb12018d0 2024-04-01T01:33:57.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.882+0000 7f59a5ffb640 1 -- 172.21.15.70:0/2022109947 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b0097870 con 0x55abb12018d0 2024-04-01T01:33:57.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.883+0000 7f59c93c4740 1 -- 172.21.15.70:0/2022109947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12018d0 msgr2=0x55abb120b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:57.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.883+0000 7f59c93c4740 1 --2- 172.21.15.70:0/2022109947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12018d0 0x55abb120b140 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f59b000c6a0 tx=0x7f59b00d27c0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.883+0000 7f59c93c4740 1 -- 172.21.15.70:0/2022109947 shutdown_connections 2024-04-01T01:33:57.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.883+0000 7f59c93c4740 1 --2- 172.21.15.70:0/2022109947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb1207430 0x55abb11fd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.883+0000 7f59c93c4740 1 --2- 172.21.15.70:0/2022109947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb12022c0 0x55abb1213060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.883+0000 7f59c93c4740 1 --2- 172.21.15.70:0/2022109947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb12018d0 0x55abb120b140 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.883+0000 7f59c93c4740 1 -- 172.21.15.70:0/2022109947 >> 172.21.15.70:0/2022109947 conn(0x55abb11fce40 msgr2=0x55abb1206bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.883+0000 7f59c93c4740 1 -- 172.21.15.70:0/2022109947 shutdown_connections 2024-04-01T01:33:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.883+0000 7f59c93c4740 1 -- 172.21.15.70:0/2022109947 wait complete. 2024-04-01T01:33:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.883+0000 7f59c93c4740 1 -- start start 2024-04-01T01:33:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59c93c4740 1 Processor -- start 2024-04-01T01:33:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59c93c4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb12018d0 0x55abb1278f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59c93c4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb12022c0 0x55abb12794d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59c93c4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb1207430 0x55abb1280510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59c93c4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f59c01ab660 con 0x55abb1207430 2024-04-01T01:33:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59c93c4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f59c01aaf90 con 0x55abb12022c0 2024-04-01T01:33:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59c93c4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f59c01a81d0 con 0x55abb12018d0 2024-04-01T01:33:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59c93c4740 10 client.0 _ll_register_callbacks cb 0x7f59b00d2690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:33:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59bffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb12018d0 0x55abb1278f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59c4c9f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb1207430 0x55abb1280510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59bffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb12018d0 0x55abb1278f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57438/0 (socket says 172.21.15.70:57438) 2024-04-01T01:33:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59bffff640 1 -- 172.21.15.70:0/3709733573 learned_addr learned my addr 172.21.15.70:0/3709733573 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:33:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59c4c9f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb1207430 0x55abb1280510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57060/0 (socket says 172.21.15.70:57060) 2024-04-01T01:33:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59bf7fe640 1 --2- 172.21.15.70:0/3709733573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb12022c0 0x55abb12794d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59bffff640 1 -- 172.21.15.70:0/3709733573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb12022c0 msgr2=0x55abb12794d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59bffff640 1 --2- 172.21.15.70:0/3709733573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb12022c0 0x55abb12794d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59bffff640 1 -- 172.21.15.70:0/3709733573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb1207430 msgr2=0x55abb1280510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:33:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59bffff640 1 --2- 172.21.15.70:0/3709733573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb1207430 0x55abb1280510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59bffff640 1 -- 172.21.15.70:0/3709733573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abb11fdbb0 con 0x55abb12018d0 2024-04-01T01:33:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59c4c9f640 1 --2- 172.21.15.70:0/3709733573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb1207430 0x55abb1280510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:33:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f59bffff640 1 --2- 172.21.15.70:0/3709733573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb12018d0 0x55abb1278f90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f59b0097bd0 tx=0x7f59b00cc6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b7e55fe788871b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:33:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f599f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:33:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.884+0000 7f599f7fe640 1 -- 172.21.15.70:0/3709733573 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b009b200 con 0x55abb12018d0 2024-04-01T01:33:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.885+0000 7f59c93c4740 1 -- 172.21.15.70:0/3709733573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55abb1172e30 con 0x55abb12018d0 2024-04-01T01:33:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.885+0000 7f599f7fe640 1 -- 172.21.15.70:0/3709733573 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f59b009f020 con 0x55abb12018d0 2024-04-01T01:33:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.885+0000 7f599f7fe640 1 -- 172.21.15.70:0/3709733573 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f59b009f350 con 0x55abb12018d0 2024-04-01T01:33:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.885+0000 7f59c93c4740 10 client.25447 fetch_fsmap learned FSMap version 29 2024-04-01T01:33:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.885+0000 7f59c93c4740 1 -- 172.21.15.70:0/3709733573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55abb1281c90 con 0x55abb12018d0 2024-04-01T01:33:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.885+0000 7f599f7fe640 1 -- 172.21.15.70:0/3709733573 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55abb1172e30 con 0x55abb12018d0 2024-04-01T01:33:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.885+0000 7f599f7fe640 1 -- 172.21.15.70:0/3709733573 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f59b00d3040 con 0x55abb12018d0 2024-04-01T01:33:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.885+0000 7f59c93c4740 10 client.25447 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:33:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.885+0000 7f59c93c4740 10 client.25447 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:33:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.885+0000 7f59c93c4740 20 client.25447 populate_metadata read hostname 'smithi070' 2024-04-01T01:33:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.885+0000 7f59c93c4740 1 --2- 172.21.15.70:0/3709733573 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55abb1282560 0x55abb1284980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:33:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.885+0000 7f59c93c4740 4 client.25447 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:33:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.885+0000 7f59c93c4740 1 -- 172.21.15.70:0/3709733573 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55abb116d180 con 0x55abb1282560 2024-04-01T01:33:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.885+0000 7f59bf7fe640 1 --2- 172.21.15.70:0/3709733573 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55abb1282560 0x55abb1284980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:33:57.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.886+0000 7f599f7fe640 10 client.25447 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:57.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:57.886+0000 7f59bf7fe640 1 --2- 172.21.15.70:0/3709733573 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55abb1282560 0x55abb1284980 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe5087b0b8baa32a server_cookie=1ebe0be1a3047684 in_seq=0 out_seq=0 2024-04-01T01:33:57.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:57 smithi189 ceph-mon[35753]: pgmap v372: 129 pgs: 129 active+clean; 30 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 3.2 MiB/s wr, 79 op/s 2024-04-01T01:33:58.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:57 smithi070 ceph-mon[32179]: pgmap v372: 129 pgs: 129 active+clean; 30 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 3.2 MiB/s wr, 79 op/s 2024-04-01T01:33:58.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:57 smithi081 ceph-mon[35142]: pgmap v372: 129 pgs: 129 active+clean; 30 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 3.2 MiB/s wr, 79 op/s 2024-04-01T01:33:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.426+0000 7f599f7fe640 1 -- 172.21.15.70:0/3709733573 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55abb116d180 con 0x55abb1282560 2024-04-01T01:33:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.426+0000 7f599f7fe640 10 client.25447 handle_command_reply: tid=0 2024-04-01T01:33:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.426+0000 7f59c93c4740 1 client.25447 shutdown 2024-04-01T01:33:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.426+0000 7f59c93c4740 1 -- 172.21.15.70:0/3709733573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb12018d0 msgr2=0x55abb1278f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.426+0000 7f59c93c4740 1 --2- 172.21.15.70:0/3709733573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb12018d0 0x55abb1278f90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f59b0097bd0 tx=0x7f59b00cc6d0 comp rx=0 tx=0).stop 2024-04-01T01:33:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.426+0000 7f59c93c4740 1 -- 172.21.15.70:0/3709733573 shutdown_connections 2024-04-01T01:33:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.426+0000 7f59c93c4740 1 --2- 172.21.15.70:0/3709733573 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55abb1282560 0x55abb1284980 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.426+0000 7f59c93c4740 1 --2- 172.21.15.70:0/3709733573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb1207430 0x55abb1280510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.426+0000 7f599f7fe640 0 client.25447 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:33:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.426+0000 7f59c93c4740 1 --2- 172.21.15.70:0/3709733573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb12022c0 0x55abb12794d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.426+0000 7f59c93c4740 1 --2- 172.21.15.70:0/3709733573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb12018d0 0x55abb1278f90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.426+0000 7f59c93c4740 1 -- 172.21.15.70:0/3709733573 >> 172.21.15.70:0/3709733573 conn(0x55abb11fce40 msgr2=0x55abb11fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.426+0000 7f59c93c4740 1 -- 172.21.15.70:0/3709733573 shutdown_connections 2024-04-01T01:33:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.426+0000 7f59c93c4740 1 -- 172.21.15.70:0/3709733573 wait complete. 2024-04-01T01:33:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.427+0000 7f59c93c4740 20 client.25447 trim_cache size 0 max 16384 2024-04-01T01:33:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:33:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:33:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:33:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:33:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.427+0000 7f59c6702640 1 -- 172.21.15.70:0/1708591486 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59a80a2660 msgr2=0x7f59a80a4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.427+0000 7f59c6702640 1 --2- 172.21.15.70:0/1708591486 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59a80a2660 0x7f59a80a4b20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f59b8009fd0 tx=0x7f59b8014040 comp rx=0 tx=0).stop 2024-04-01T01:33:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.427+0000 7f59c6702640 1 -- 172.21.15.70:0/1708591486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c0100670 msgr2=0x7f59c0197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:33:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.427+0000 7f59c6702640 1 --2- 172.21.15.70:0/1708591486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c0100670 0x7f59c0197990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f59b000e9c0 tx=0x7f59b000edc0 comp rx=0 tx=0).stop 2024-04-01T01:33:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.427+0000 7f59c6702640 1 -- 172.21.15.70:0/1708591486 shutdown_connections 2024-04-01T01:33:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.427+0000 7f59c6702640 1 --2- 172.21.15.70:0/1708591486 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59a80a2660 0x7f59a80a4b20 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.427+0000 7f59c6702640 1 --2- 172.21.15.70:0/1708591486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59c0102220 0x7f59c019eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.427+0000 7f59c6702640 1 --2- 172.21.15.70:0/1708591486 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59c0101860 0x7f59c0197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.427+0000 7f59c6702640 1 --2- 172.21.15.70:0/1708591486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59c0100670 0x7f59c0197990 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:33:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.427+0000 7f59c6702640 1 -- 172.21.15.70:0/1708591486 >> 172.21.15.70:0/1708591486 conn(0x7f59c00fbbe0 msgr2=0x7f59c00fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:33:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.427+0000 7f59c6702640 1 -- 172.21.15.70:0/1708591486 shutdown_connections 2024-04-01T01:33:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:33:58.427+0000 7f59c6702640 1 -- 172.21.15.70:0/1708591486 wait complete. 2024-04-01T01:33:59.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:33:59 smithi189 ceph-mon[35753]: pgmap v373: 129 pgs: 129 active+clean; 33 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 3.5 MiB/s wr, 109 op/s 2024-04-01T01:34:00.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:33:59 smithi070 ceph-mon[32179]: pgmap v373: 129 pgs: 129 active+clean; 33 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 3.5 MiB/s wr, 109 op/s 2024-04-01T01:34:00.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:33:59 smithi081 ceph-mon[35142]: pgmap v373: 129 pgs: 129 active+clean; 33 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 3.5 MiB/s wr, 109 op/s 2024-04-01T01:34:01.933 INFO:teuthology.orchestra.run.smithi070.stderr: * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log 2024-04-01T01:34:01.936 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-01T01:34:01.972 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:01 smithi070 ceph-mon[32179]: pgmap v374: 129 pgs: 129 active+clean; 36 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 4.0 MiB/s wr, 184 op/s 2024-04-01T01:34:01.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:01 smithi189 ceph-mon[35753]: pgmap v374: 129 pgs: 129 active+clean; 36 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 4.0 MiB/s wr, 184 op/s 2024-04-01T01:34:02.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:01 smithi081 ceph-mon[35142]: pgmap v374: 129 pgs: 129 active+clean; 36 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 4.0 MiB/s wr, 184 op/s 2024-04-01T01:34:02.222 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/: 2024-04-01T01:34:02.222 INFO:teuthology.orchestra.run.smithi070.stdout:total 2 2024-04-01T01:34:02.222 INFO:teuthology.orchestra.run.smithi070.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:33 . 2024-04-01T01:34:02.222 INFO:teuthology.orchestra.run.smithi070.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:33 .. 2024-04-01T01:34:02.222 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 data 2024-04-01T01:34:02.222 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.222 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data: 2024-04-01T01:34:02.222 INFO:teuthology.orchestra.run.smithi070.stdout:total 45 2024-04-01T01:34:02.222 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 . 2024-04-01T01:34:02.223 INFO:teuthology.orchestra.run.smithi070.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:33 .. 2024-04-01T01:34:02.223 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 23889417 Apr 1 01:33 base 2024-04-01T01:34:02.223 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 549376 Apr 1 01:33 global 2024-04-01T01:34:02.223 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:34 log 2024-04-01T01:34:02.223 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_commit_ts 2024-04-01T01:34:02.223 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_dynshmem 2024-04-01T01:34:02.223 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 1 01:33 pg_hba.conf 2024-04-01T01:34:02.223 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 1 01:33 pg_ident.conf 2024-04-01T01:34:02.223 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 1 01:33 pg_logical 2024-04-01T01:34:02.223 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 pg_multixact 2024-04-01T01:34:02.223 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_notify 2024-04-01T01:34:02.224 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_replslot 2024-04-01T01:34:02.224 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_serial 2024-04-01T01:34:02.224 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_snapshots 2024-04-01T01:34:02.224 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_stat 2024-04-01T01:34:02.224 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_stat_tmp 2024-04-01T01:34:02.224 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 pg_subtrans 2024-04-01T01:34:02.224 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_tblspc 2024-04-01T01:34:02.224 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_twophase 2024-04-01T01:34:02.224 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 1 01:33 PG_VERSION 2024-04-01T01:34:02.224 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:33 pg_wal 2024-04-01T01:34:02.224 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 pg_xact 2024-04-01T01:34:02.224 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 1 01:33 postgresql.auto.conf 2024-04-01T01:34:02.224 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 1 01:33 postgresql.conf 2024-04-01T01:34:02.225 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.225 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/base: 2024-04-01T01:34:02.225 INFO:teuthology.orchestra.run.smithi070.stdout:total 3 2024-04-01T01:34:02.225 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 23889417 Apr 1 01:33 . 2024-04-01T01:34:02.225 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.225 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Apr 1 01:33 1 2024-04-01T01:34:02.225 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Apr 1 01:33 13493 2024-04-01T01:34:02.225 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Apr 1 01:33 13494 2024-04-01T01:34:02.226 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.226 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-01T01:34:02.226 INFO:teuthology.orchestra.run.smithi070.stdout:total 7778 2024-04-01T01:34:02.226 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Apr 1 01:33 . 2024-04-01T01:34:02.226 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 23889417 Apr 1 01:33 .. 2024-04-01T01:34:02.226 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 112 2024-04-01T01:34:02.226 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 113 2024-04-01T01:34:02.226 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 1 01:33 1247 2024-04-01T01:34:02.226 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1247_fsm 2024-04-01T01:34:02.226 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1247_vm 2024-04-01T01:34:02.226 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 1 01:33 1249 2024-04-01T01:34:02.226 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1249_fsm 2024-04-01T01:34:02.227 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1249_vm 2024-04-01T01:34:02.227 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 1 01:33 1255 2024-04-01T01:34:02.227 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1255_fsm 2024-04-01T01:34:02.227 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1255_vm 2024-04-01T01:34:02.227 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 1 01:33 1259 2024-04-01T01:34:02.227 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1259_fsm 2024-04-01T01:34:02.227 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1259_vm 2024-04-01T01:34:02.227 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 1 01:33 13344 2024-04-01T01:34:02.227 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13344_fsm 2024-04-01T01:34:02.227 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13344_vm 2024-04-01T01:34:02.228 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13346 2024-04-01T01:34:02.228 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13348 2024-04-01T01:34:02.228 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13349 2024-04-01T01:34:02.228 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13349_fsm 2024-04-01T01:34:02.228 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13349_vm 2024-04-01T01:34:02.228 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13351 2024-04-01T01:34:02.228 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13353 2024-04-01T01:34:02.228 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13354 2024-04-01T01:34:02.228 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13354_fsm 2024-04-01T01:34:02.229 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13354_vm 2024-04-01T01:34:02.229 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13356 2024-04-01T01:34:02.229 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13358 2024-04-01T01:34:02.229 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13359 2024-04-01T01:34:02.229 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13359_fsm 2024-04-01T01:34:02.229 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13359_vm 2024-04-01T01:34:02.229 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13361 2024-04-01T01:34:02.229 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13363 2024-04-01T01:34:02.229 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 1417 2024-04-01T01:34:02.229 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 1418 2024-04-01T01:34:02.229 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 174 2024-04-01T01:34:02.229 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 175 2024-04-01T01:34:02.229 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2187 2024-04-01T01:34:02.230 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2224 2024-04-01T01:34:02.230 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2328 2024-04-01T01:34:02.230 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2336 2024-04-01T01:34:02.230 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2337 2024-04-01T01:34:02.230 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2579 2024-04-01T01:34:02.230 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2600 2024-04-01T01:34:02.230 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2600_fsm 2024-04-01T01:34:02.230 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2600_vm 2024-04-01T01:34:02.230 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2601 2024-04-01T01:34:02.230 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2601_fsm 2024-04-01T01:34:02.230 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2601_vm 2024-04-01T01:34:02.231 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2602 2024-04-01T01:34:02.231 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2602_fsm 2024-04-01T01:34:02.231 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2602_vm 2024-04-01T01:34:02.231 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2603 2024-04-01T01:34:02.231 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2603_fsm 2024-04-01T01:34:02.231 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2603_vm 2024-04-01T01:34:02.231 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2604 2024-04-01T01:34:02.231 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2605 2024-04-01T01:34:02.231 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2605_fsm 2024-04-01T01:34:02.231 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2605_vm 2024-04-01T01:34:02.231 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2606 2024-04-01T01:34:02.231 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2606_fsm 2024-04-01T01:34:02.232 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2606_vm 2024-04-01T01:34:02.232 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2607 2024-04-01T01:34:02.232 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2607_fsm 2024-04-01T01:34:02.232 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2607_vm 2024-04-01T01:34:02.232 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 1 01:33 2608 2024-04-01T01:34:02.232 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2608_fsm 2024-04-01T01:34:02.232 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2608_vm 2024-04-01T01:34:02.232 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 1 01:33 2609 2024-04-01T01:34:02.232 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2609_fsm 2024-04-01T01:34:02.232 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2609_vm 2024-04-01T01:34:02.232 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2610 2024-04-01T01:34:02.233 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2610_fsm 2024-04-01T01:34:02.233 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2610_vm 2024-04-01T01:34:02.233 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2611 2024-04-01T01:34:02.233 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2612 2024-04-01T01:34:02.233 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2612_fsm 2024-04-01T01:34:02.233 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2612_vm 2024-04-01T01:34:02.233 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2613 2024-04-01T01:34:02.233 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2615 2024-04-01T01:34:02.233 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2615_fsm 2024-04-01T01:34:02.233 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2615_vm 2024-04-01T01:34:02.233 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2616 2024-04-01T01:34:02.233 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2616_fsm 2024-04-01T01:34:02.233 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2616_vm 2024-04-01T01:34:02.234 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 1 01:33 2617 2024-04-01T01:34:02.234 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2617_fsm 2024-04-01T01:34:02.234 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2617_vm 2024-04-01T01:34:02.234 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 1 01:33 2618 2024-04-01T01:34:02.234 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2618_fsm 2024-04-01T01:34:02.234 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2618_vm 2024-04-01T01:34:02.234 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 1 01:33 2619 2024-04-01T01:34:02.234 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2619_fsm 2024-04-01T01:34:02.234 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2619_vm 2024-04-01T01:34:02.234 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2620 2024-04-01T01:34:02.234 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2650 2024-04-01T01:34:02.234 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2651 2024-04-01T01:34:02.235 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2652 2024-04-01T01:34:02.235 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2653 2024-04-01T01:34:02.235 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2654 2024-04-01T01:34:02.235 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2655 2024-04-01T01:34:02.235 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2656 2024-04-01T01:34:02.235 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2657 2024-04-01T01:34:02.235 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 1 01:33 2658 2024-04-01T01:34:02.235 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 1 01:33 2659 2024-04-01T01:34:02.235 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2660 2024-04-01T01:34:02.235 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2661 2024-04-01T01:34:02.235 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2662 2024-04-01T01:34:02.236 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2663 2024-04-01T01:34:02.236 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2664 2024-04-01T01:34:02.236 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2665 2024-04-01T01:34:02.236 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2666 2024-04-01T01:34:02.236 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2667 2024-04-01T01:34:02.236 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2668 2024-04-01T01:34:02.236 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2669 2024-04-01T01:34:02.236 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2670 2024-04-01T01:34:02.236 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 1 01:33 2673 2024-04-01T01:34:02.236 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 1 01:33 2674 2024-04-01T01:34:02.236 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 1 01:33 2675 2024-04-01T01:34:02.236 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2678 2024-04-01T01:34:02.237 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2679 2024-04-01T01:34:02.237 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2680 2024-04-01T01:34:02.237 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2681 2024-04-01T01:34:02.237 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2682 2024-04-01T01:34:02.237 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2683 2024-04-01T01:34:02.237 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2684 2024-04-01T01:34:02.237 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2685 2024-04-01T01:34:02.237 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2686 2024-04-01T01:34:02.237 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2687 2024-04-01T01:34:02.237 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2688 2024-04-01T01:34:02.237 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2689 2024-04-01T01:34:02.238 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 1 01:33 2690 2024-04-01T01:34:02.238 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 1 01:33 2691 2024-04-01T01:34:02.238 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2692 2024-04-01T01:34:02.238 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2693 2024-04-01T01:34:02.238 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2696 2024-04-01T01:34:02.238 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2699 2024-04-01T01:34:02.238 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2701 2024-04-01T01:34:02.238 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2702 2024-04-01T01:34:02.238 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2703 2024-04-01T01:34:02.238 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2704 2024-04-01T01:34:02.239 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2753 2024-04-01T01:34:02.239 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2753_fsm 2024-04-01T01:34:02.239 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2753_vm 2024-04-01T01:34:02.239 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2754 2024-04-01T01:34:02.239 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2755 2024-04-01T01:34:02.239 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2756 2024-04-01T01:34:02.239 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2757 2024-04-01T01:34:02.239 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2830 2024-04-01T01:34:02.239 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2831 2024-04-01T01:34:02.239 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2832 2024-04-01T01:34:02.239 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2833 2024-04-01T01:34:02.239 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2834 2024-04-01T01:34:02.239 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2835 2024-04-01T01:34:02.240 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2836 2024-04-01T01:34:02.240 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2837 2024-04-01T01:34:02.240 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 1 01:33 2838 2024-04-01T01:34:02.240 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2838_fsm 2024-04-01T01:34:02.240 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2838_vm 2024-04-01T01:34:02.240 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2839 2024-04-01T01:34:02.240 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2840 2024-04-01T01:34:02.240 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2840_fsm 2024-04-01T01:34:02.240 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2840_vm 2024-04-01T01:34:02.240 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2841 2024-04-01T01:34:02.240 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2995 2024-04-01T01:34:02.240 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2996 2024-04-01T01:34:02.240 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3079 2024-04-01T01:34:02.241 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3079_fsm 2024-04-01T01:34:02.241 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3079_vm 2024-04-01T01:34:02.241 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3080 2024-04-01T01:34:02.241 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3081 2024-04-01T01:34:02.241 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 3085 2024-04-01T01:34:02.241 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3118 2024-04-01T01:34:02.241 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3119 2024-04-01T01:34:02.241 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 1 01:33 3164 2024-04-01T01:34:02.241 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3256 2024-04-01T01:34:02.241 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3257 2024-04-01T01:34:02.241 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3258 2024-04-01T01:34:02.241 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3350 2024-04-01T01:34:02.242 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3351 2024-04-01T01:34:02.242 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3379 2024-04-01T01:34:02.242 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3380 2024-04-01T01:34:02.242 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3381 2024-04-01T01:34:02.242 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3394 2024-04-01T01:34:02.242 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3394_fsm 2024-04-01T01:34:02.242 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3394_vm 2024-04-01T01:34:02.242 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3395 2024-04-01T01:34:02.242 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3429 2024-04-01T01:34:02.242 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3430 2024-04-01T01:34:02.242 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3431 2024-04-01T01:34:02.242 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3433 2024-04-01T01:34:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3439 2024-04-01T01:34:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3440 2024-04-01T01:34:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 3455 2024-04-01T01:34:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 1 01:33 3456 2024-04-01T01:34:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3456_fsm 2024-04-01T01:34:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3456_vm 2024-04-01T01:34:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3466 2024-04-01T01:34:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3467 2024-04-01T01:34:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3468 2024-04-01T01:34:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3501 2024-04-01T01:34:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3502 2024-04-01T01:34:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3503 2024-04-01T01:34:02.243 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3534 2024-04-01T01:34:02.244 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3541 2024-04-01T01:34:02.244 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3541_fsm 2024-04-01T01:34:02.244 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3541_vm 2024-04-01T01:34:02.244 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3542 2024-04-01T01:34:02.244 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3574 2024-04-01T01:34:02.244 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3575 2024-04-01T01:34:02.244 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3576 2024-04-01T01:34:02.244 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3596 2024-04-01T01:34:02.244 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3597 2024-04-01T01:34:02.244 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3598 2024-04-01T01:34:02.244 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3599 2024-04-01T01:34:02.244 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3600 2024-04-01T01:34:02.245 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3600_fsm 2024-04-01T01:34:02.245 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3600_vm 2024-04-01T01:34:02.245 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3601 2024-04-01T01:34:02.245 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3601_fsm 2024-04-01T01:34:02.245 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3601_vm 2024-04-01T01:34:02.245 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3602 2024-04-01T01:34:02.245 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3602_fsm 2024-04-01T01:34:02.245 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3602_vm 2024-04-01T01:34:02.245 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3603 2024-04-01T01:34:02.245 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3603_fsm 2024-04-01T01:34:02.245 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3603_vm 2024-04-01T01:34:02.245 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3604 2024-04-01T01:34:02.246 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3605 2024-04-01T01:34:02.246 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3606 2024-04-01T01:34:02.246 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3607 2024-04-01T01:34:02.246 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3608 2024-04-01T01:34:02.246 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 3609 2024-04-01T01:34:02.246 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3712 2024-04-01T01:34:02.246 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3764 2024-04-01T01:34:02.246 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3764_fsm 2024-04-01T01:34:02.246 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3764_vm 2024-04-01T01:34:02.246 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3766 2024-04-01T01:34:02.246 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3767 2024-04-01T01:34:02.246 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3997 2024-04-01T01:34:02.246 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4143 2024-04-01T01:34:02.247 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4144 2024-04-01T01:34:02.247 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4145 2024-04-01T01:34:02.247 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4146 2024-04-01T01:34:02.247 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4147 2024-04-01T01:34:02.247 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4148 2024-04-01T01:34:02.247 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4149 2024-04-01T01:34:02.247 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4150 2024-04-01T01:34:02.247 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4151 2024-04-01T01:34:02.247 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4152 2024-04-01T01:34:02.247 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4153 2024-04-01T01:34:02.247 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4154 2024-04-01T01:34:02.247 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4155 2024-04-01T01:34:02.248 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4156 2024-04-01T01:34:02.248 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4157 2024-04-01T01:34:02.248 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4158 2024-04-01T01:34:02.444 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4159 2024-04-01T01:34:02.445 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4160 2024-04-01T01:34:02.445 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4161 2024-04-01T01:34:02.445 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4162 2024-04-01T01:34:02.445 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4163 2024-04-01T01:34:02.445 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4164 2024-04-01T01:34:02.445 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4165 2024-04-01T01:34:02.445 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4166 2024-04-01T01:34:02.445 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4167 2024-04-01T01:34:02.445 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4168 2024-04-01T01:34:02.445 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4169 2024-04-01T01:34:02.445 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4170 2024-04-01T01:34:02.445 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4171 2024-04-01T01:34:02.445 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4172 2024-04-01T01:34:02.446 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4173 2024-04-01T01:34:02.446 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4174 2024-04-01T01:34:02.446 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 5002 2024-04-01T01:34:02.446 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 548 2024-04-01T01:34:02.446 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 549 2024-04-01T01:34:02.446 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6102 2024-04-01T01:34:02.446 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6104 2024-04-01T01:34:02.446 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6106 2024-04-01T01:34:02.446 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6110 2024-04-01T01:34:02.446 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6111 2024-04-01T01:34:02.446 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6112 2024-04-01T01:34:02.446 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6113 2024-04-01T01:34:02.446 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6117 2024-04-01T01:34:02.446 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 826 2024-04-01T01:34:02.447 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 827 2024-04-01T01:34:02.447 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 828 2024-04-01T01:34:02.447 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 1 01:33 pg_filenode.map 2024-04-01T01:34:02.447 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 1 01:33 PG_VERSION 2024-04-01T01:34:02.447 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.447 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-01T01:34:02.447 INFO:teuthology.orchestra.run.smithi070.stdout:total 7778 2024-04-01T01:34:02.447 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Apr 1 01:33 . 2024-04-01T01:34:02.447 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 23889417 Apr 1 01:33 .. 2024-04-01T01:34:02.447 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 112 2024-04-01T01:34:02.447 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 113 2024-04-01T01:34:02.447 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 1 01:33 1247 2024-04-01T01:34:02.447 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1247_fsm 2024-04-01T01:34:02.448 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1247_vm 2024-04-01T01:34:02.448 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 1 01:33 1249 2024-04-01T01:34:02.448 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1249_fsm 2024-04-01T01:34:02.448 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1249_vm 2024-04-01T01:34:02.448 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 1 01:33 1255 2024-04-01T01:34:02.448 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1255_fsm 2024-04-01T01:34:02.448 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1255_vm 2024-04-01T01:34:02.448 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 1 01:33 1259 2024-04-01T01:34:02.448 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1259_fsm 2024-04-01T01:34:02.448 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1259_vm 2024-04-01T01:34:02.448 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 1 01:33 13344 2024-04-01T01:34:02.448 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13344_fsm 2024-04-01T01:34:02.448 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13344_vm 2024-04-01T01:34:02.448 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13346 2024-04-01T01:34:02.448 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13348 2024-04-01T01:34:02.449 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13349 2024-04-01T01:34:02.449 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13349_fsm 2024-04-01T01:34:02.449 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13349_vm 2024-04-01T01:34:02.449 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13351 2024-04-01T01:34:02.449 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13353 2024-04-01T01:34:02.449 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13354 2024-04-01T01:34:02.449 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13354_fsm 2024-04-01T01:34:02.449 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13354_vm 2024-04-01T01:34:02.449 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13356 2024-04-01T01:34:02.449 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13358 2024-04-01T01:34:02.449 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13359 2024-04-01T01:34:02.449 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13359_fsm 2024-04-01T01:34:02.449 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13359_vm 2024-04-01T01:34:02.449 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13361 2024-04-01T01:34:02.449 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13363 2024-04-01T01:34:02.449 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 1417 2024-04-01T01:34:02.450 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 1418 2024-04-01T01:34:02.450 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 174 2024-04-01T01:34:02.450 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 175 2024-04-01T01:34:02.450 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2187 2024-04-01T01:34:02.450 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2224 2024-04-01T01:34:02.450 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2328 2024-04-01T01:34:02.450 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2336 2024-04-01T01:34:02.450 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2337 2024-04-01T01:34:02.450 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2579 2024-04-01T01:34:02.450 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2600 2024-04-01T01:34:02.450 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2600_fsm 2024-04-01T01:34:02.450 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2600_vm 2024-04-01T01:34:02.450 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2601 2024-04-01T01:34:02.450 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2601_fsm 2024-04-01T01:34:02.451 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2601_vm 2024-04-01T01:34:02.451 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2602 2024-04-01T01:34:02.451 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2602_fsm 2024-04-01T01:34:02.451 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2602_vm 2024-04-01T01:34:02.451 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2603 2024-04-01T01:34:02.451 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2603_fsm 2024-04-01T01:34:02.451 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2603_vm 2024-04-01T01:34:02.451 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2604 2024-04-01T01:34:02.451 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2605 2024-04-01T01:34:02.451 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2605_fsm 2024-04-01T01:34:02.451 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2605_vm 2024-04-01T01:34:02.451 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2606 2024-04-01T01:34:02.451 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2606_fsm 2024-04-01T01:34:02.451 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2606_vm 2024-04-01T01:34:02.452 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2607 2024-04-01T01:34:02.452 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2607_fsm 2024-04-01T01:34:02.452 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2607_vm 2024-04-01T01:34:02.452 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 1 01:33 2608 2024-04-01T01:34:02.452 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2608_fsm 2024-04-01T01:34:02.452 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2608_vm 2024-04-01T01:34:02.452 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 1 01:33 2609 2024-04-01T01:34:02.452 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2609_fsm 2024-04-01T01:34:02.452 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2609_vm 2024-04-01T01:34:02.452 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2610 2024-04-01T01:34:02.452 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2610_fsm 2024-04-01T01:34:02.452 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2610_vm 2024-04-01T01:34:02.452 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2611 2024-04-01T01:34:02.452 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2612 2024-04-01T01:34:02.453 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2612_fsm 2024-04-01T01:34:02.453 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2612_vm 2024-04-01T01:34:02.453 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2613 2024-04-01T01:34:02.453 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2615 2024-04-01T01:34:02.453 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2615_fsm 2024-04-01T01:34:02.453 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2615_vm 2024-04-01T01:34:02.453 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2616 2024-04-01T01:34:02.453 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2616_fsm 2024-04-01T01:34:02.453 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2616_vm 2024-04-01T01:34:02.453 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 1 01:33 2617 2024-04-01T01:34:02.453 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2617_fsm 2024-04-01T01:34:02.453 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2617_vm 2024-04-01T01:34:02.453 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 1 01:33 2618 2024-04-01T01:34:02.453 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2618_fsm 2024-04-01T01:34:02.453 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2618_vm 2024-04-01T01:34:02.454 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 1 01:33 2619 2024-04-01T01:34:02.454 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2619_fsm 2024-04-01T01:34:02.454 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2619_vm 2024-04-01T01:34:02.454 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2620 2024-04-01T01:34:02.454 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2650 2024-04-01T01:34:02.454 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2651 2024-04-01T01:34:02.454 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2652 2024-04-01T01:34:02.454 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2653 2024-04-01T01:34:02.454 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2654 2024-04-01T01:34:02.454 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2655 2024-04-01T01:34:02.454 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2656 2024-04-01T01:34:02.454 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2657 2024-04-01T01:34:02.454 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 1 01:33 2658 2024-04-01T01:34:02.454 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 1 01:33 2659 2024-04-01T01:34:02.455 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2660 2024-04-01T01:34:02.455 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2661 2024-04-01T01:34:02.455 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2662 2024-04-01T01:34:02.455 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2663 2024-04-01T01:34:02.455 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2664 2024-04-01T01:34:02.455 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2665 2024-04-01T01:34:02.455 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2666 2024-04-01T01:34:02.455 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2667 2024-04-01T01:34:02.455 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2668 2024-04-01T01:34:02.455 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2669 2024-04-01T01:34:02.455 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2670 2024-04-01T01:34:02.455 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 1 01:33 2673 2024-04-01T01:34:02.455 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 1 01:33 2674 2024-04-01T01:34:02.455 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 1 01:33 2675 2024-04-01T01:34:02.456 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2678 2024-04-01T01:34:02.456 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2679 2024-04-01T01:34:02.456 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2680 2024-04-01T01:34:02.456 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2681 2024-04-01T01:34:02.456 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2682 2024-04-01T01:34:02.456 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2683 2024-04-01T01:34:02.456 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2684 2024-04-01T01:34:02.456 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2685 2024-04-01T01:34:02.456 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2686 2024-04-01T01:34:02.456 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2687 2024-04-01T01:34:02.456 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2688 2024-04-01T01:34:02.456 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2689 2024-04-01T01:34:02.456 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 1 01:33 2690 2024-04-01T01:34:02.456 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 1 01:33 2691 2024-04-01T01:34:02.457 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2692 2024-04-01T01:34:02.457 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2693 2024-04-01T01:34:02.457 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2696 2024-04-01T01:34:02.457 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2699 2024-04-01T01:34:02.457 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2701 2024-04-01T01:34:02.457 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2702 2024-04-01T01:34:02.457 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2703 2024-04-01T01:34:02.457 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2704 2024-04-01T01:34:02.457 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2753 2024-04-01T01:34:02.457 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2753_fsm 2024-04-01T01:34:02.457 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2753_vm 2024-04-01T01:34:02.457 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2754 2024-04-01T01:34:02.457 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2755 2024-04-01T01:34:02.457 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2756 2024-04-01T01:34:02.458 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2757 2024-04-01T01:34:02.458 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2830 2024-04-01T01:34:02.458 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2831 2024-04-01T01:34:02.458 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2832 2024-04-01T01:34:02.458 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2833 2024-04-01T01:34:02.458 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2834 2024-04-01T01:34:02.458 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2835 2024-04-01T01:34:02.458 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2836 2024-04-01T01:34:02.458 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2837 2024-04-01T01:34:02.458 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 1 01:33 2838 2024-04-01T01:34:02.458 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2838_fsm 2024-04-01T01:34:02.458 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2838_vm 2024-04-01T01:34:02.458 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2839 2024-04-01T01:34:02.458 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2840 2024-04-01T01:34:02.459 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2840_fsm 2024-04-01T01:34:02.459 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2840_vm 2024-04-01T01:34:02.459 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2841 2024-04-01T01:34:02.459 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2995 2024-04-01T01:34:02.459 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2996 2024-04-01T01:34:02.459 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3079 2024-04-01T01:34:02.459 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3079_fsm 2024-04-01T01:34:02.459 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3079_vm 2024-04-01T01:34:02.459 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3080 2024-04-01T01:34:02.459 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3081 2024-04-01T01:34:02.459 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 3085 2024-04-01T01:34:02.459 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3118 2024-04-01T01:34:02.459 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3119 2024-04-01T01:34:02.459 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 1 01:33 3164 2024-04-01T01:34:02.460 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3256 2024-04-01T01:34:02.460 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3257 2024-04-01T01:34:02.460 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3258 2024-04-01T01:34:02.460 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3350 2024-04-01T01:34:02.460 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3351 2024-04-01T01:34:02.460 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3379 2024-04-01T01:34:02.460 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3380 2024-04-01T01:34:02.460 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3381 2024-04-01T01:34:02.460 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3394 2024-04-01T01:34:02.460 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3394_fsm 2024-04-01T01:34:02.460 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3394_vm 2024-04-01T01:34:02.460 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3395 2024-04-01T01:34:02.460 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3429 2024-04-01T01:34:02.460 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3430 2024-04-01T01:34:02.460 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3431 2024-04-01T01:34:02.461 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3433 2024-04-01T01:34:02.461 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3439 2024-04-01T01:34:02.461 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3440 2024-04-01T01:34:02.461 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 3455 2024-04-01T01:34:02.461 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 1 01:33 3456 2024-04-01T01:34:02.461 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3456_fsm 2024-04-01T01:34:02.461 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3456_vm 2024-04-01T01:34:02.461 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3466 2024-04-01T01:34:02.461 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3467 2024-04-01T01:34:02.461 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3468 2024-04-01T01:34:02.461 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3501 2024-04-01T01:34:02.461 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3502 2024-04-01T01:34:02.461 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3503 2024-04-01T01:34:02.461 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3534 2024-04-01T01:34:02.461 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3541 2024-04-01T01:34:02.462 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3541_fsm 2024-04-01T01:34:02.462 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3541_vm 2024-04-01T01:34:02.462 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3542 2024-04-01T01:34:02.462 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3574 2024-04-01T01:34:02.462 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3575 2024-04-01T01:34:02.462 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3576 2024-04-01T01:34:02.462 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3596 2024-04-01T01:34:02.462 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3597 2024-04-01T01:34:02.462 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3598 2024-04-01T01:34:02.462 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3599 2024-04-01T01:34:02.462 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3600 2024-04-01T01:34:02.462 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3600_fsm 2024-04-01T01:34:02.463 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3600_vm 2024-04-01T01:34:02.463 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3601 2024-04-01T01:34:02.463 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3601_fsm 2024-04-01T01:34:02.463 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3601_vm 2024-04-01T01:34:02.463 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3602 2024-04-01T01:34:02.463 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3602_fsm 2024-04-01T01:34:02.463 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3602_vm 2024-04-01T01:34:02.463 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3603 2024-04-01T01:34:02.463 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3603_fsm 2024-04-01T01:34:02.463 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3603_vm 2024-04-01T01:34:02.463 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3604 2024-04-01T01:34:02.463 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3605 2024-04-01T01:34:02.463 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3606 2024-04-01T01:34:02.464 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3607 2024-04-01T01:34:02.464 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3608 2024-04-01T01:34:02.464 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 3609 2024-04-01T01:34:02.464 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3712 2024-04-01T01:34:02.464 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3764 2024-04-01T01:34:02.464 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3764_fsm 2024-04-01T01:34:02.464 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3764_vm 2024-04-01T01:34:02.464 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3766 2024-04-01T01:34:02.464 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3767 2024-04-01T01:34:02.464 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3997 2024-04-01T01:34:02.464 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4143 2024-04-01T01:34:02.464 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4144 2024-04-01T01:34:02.464 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4145 2024-04-01T01:34:02.465 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4146 2024-04-01T01:34:02.465 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4147 2024-04-01T01:34:02.465 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4148 2024-04-01T01:34:02.465 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4149 2024-04-01T01:34:02.465 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4150 2024-04-01T01:34:02.465 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4151 2024-04-01T01:34:02.465 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4152 2024-04-01T01:34:02.465 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4153 2024-04-01T01:34:02.465 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4154 2024-04-01T01:34:02.465 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4155 2024-04-01T01:34:02.465 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4156 2024-04-01T01:34:02.465 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4157 2024-04-01T01:34:02.466 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4158 2024-04-01T01:34:02.466 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4159 2024-04-01T01:34:02.466 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4160 2024-04-01T01:34:02.466 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4161 2024-04-01T01:34:02.466 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4162 2024-04-01T01:34:02.662 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4163 2024-04-01T01:34:02.662 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4164 2024-04-01T01:34:02.662 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4165 2024-04-01T01:34:02.662 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4166 2024-04-01T01:34:02.662 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4167 2024-04-01T01:34:02.662 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4168 2024-04-01T01:34:02.662 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4169 2024-04-01T01:34:02.662 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4170 2024-04-01T01:34:02.662 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4171 2024-04-01T01:34:02.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4172 2024-04-01T01:34:02.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4173 2024-04-01T01:34:02.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4174 2024-04-01T01:34:02.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 5002 2024-04-01T01:34:02.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 548 2024-04-01T01:34:02.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 549 2024-04-01T01:34:02.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6102 2024-04-01T01:34:02.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6104 2024-04-01T01:34:02.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6106 2024-04-01T01:34:02.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6110 2024-04-01T01:34:02.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6111 2024-04-01T01:34:02.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6112 2024-04-01T01:34:02.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6113 2024-04-01T01:34:02.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6117 2024-04-01T01:34:02.664 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 826 2024-04-01T01:34:02.664 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 827 2024-04-01T01:34:02.664 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 828 2024-04-01T01:34:02.664 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 1 01:33 pg_filenode.map 2024-04-01T01:34:02.664 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 1 01:33 PG_VERSION 2024-04-01T01:34:02.664 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.664 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-01T01:34:02.664 INFO:teuthology.orchestra.run.smithi070.stdout:total 7778 2024-04-01T01:34:02.664 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Apr 1 01:33 . 2024-04-01T01:34:02.664 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 23889417 Apr 1 01:33 .. 2024-04-01T01:34:02.664 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 112 2024-04-01T01:34:02.664 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 113 2024-04-01T01:34:02.665 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 1 01:33 1247 2024-04-01T01:34:02.665 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1247_fsm 2024-04-01T01:34:02.665 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1247_vm 2024-04-01T01:34:02.665 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 1 01:33 1249 2024-04-01T01:34:02.665 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1249_fsm 2024-04-01T01:34:02.665 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1249_vm 2024-04-01T01:34:02.665 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 1 01:33 1255 2024-04-01T01:34:02.665 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1255_fsm 2024-04-01T01:34:02.665 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1255_vm 2024-04-01T01:34:02.665 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 1 01:33 1259 2024-04-01T01:34:02.665 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1259_fsm 2024-04-01T01:34:02.666 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1259_vm 2024-04-01T01:34:02.666 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 1 01:33 13344 2024-04-01T01:34:02.666 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13344_fsm 2024-04-01T01:34:02.666 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13344_vm 2024-04-01T01:34:02.666 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13346 2024-04-01T01:34:02.666 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13348 2024-04-01T01:34:02.666 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13349 2024-04-01T01:34:02.666 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13349_fsm 2024-04-01T01:34:02.666 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13349_vm 2024-04-01T01:34:02.666 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13351 2024-04-01T01:34:02.666 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13353 2024-04-01T01:34:02.666 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13354 2024-04-01T01:34:02.667 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13354_fsm 2024-04-01T01:34:02.667 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13354_vm 2024-04-01T01:34:02.667 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13356 2024-04-01T01:34:02.667 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13358 2024-04-01T01:34:02.667 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13359 2024-04-01T01:34:02.667 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13359_fsm 2024-04-01T01:34:02.667 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13359_vm 2024-04-01T01:34:02.667 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13361 2024-04-01T01:34:02.667 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13363 2024-04-01T01:34:02.667 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 1417 2024-04-01T01:34:02.667 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 1418 2024-04-01T01:34:02.667 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 174 2024-04-01T01:34:02.667 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 175 2024-04-01T01:34:02.667 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2187 2024-04-01T01:34:02.668 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2224 2024-04-01T01:34:02.668 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2328 2024-04-01T01:34:02.668 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2336 2024-04-01T01:34:02.668 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2337 2024-04-01T01:34:02.668 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2579 2024-04-01T01:34:02.668 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2600 2024-04-01T01:34:02.668 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2600_fsm 2024-04-01T01:34:02.668 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2600_vm 2024-04-01T01:34:02.668 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2601 2024-04-01T01:34:02.668 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2601_fsm 2024-04-01T01:34:02.668 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2601_vm 2024-04-01T01:34:02.668 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2602 2024-04-01T01:34:02.668 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2602_fsm 2024-04-01T01:34:02.668 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2602_vm 2024-04-01T01:34:02.668 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2603 2024-04-01T01:34:02.669 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2603_fsm 2024-04-01T01:34:02.669 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2603_vm 2024-04-01T01:34:02.669 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2604 2024-04-01T01:34:02.669 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2605 2024-04-01T01:34:02.669 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2605_fsm 2024-04-01T01:34:02.669 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2605_vm 2024-04-01T01:34:02.669 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2606 2024-04-01T01:34:02.669 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2606_fsm 2024-04-01T01:34:02.669 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2606_vm 2024-04-01T01:34:02.669 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2607 2024-04-01T01:34:02.669 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2607_fsm 2024-04-01T01:34:02.669 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2607_vm 2024-04-01T01:34:02.669 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 1 01:33 2608 2024-04-01T01:34:02.669 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2608_fsm 2024-04-01T01:34:02.670 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2608_vm 2024-04-01T01:34:02.670 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 1 01:33 2609 2024-04-01T01:34:02.670 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2609_fsm 2024-04-01T01:34:02.670 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2609_vm 2024-04-01T01:34:02.670 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2610 2024-04-01T01:34:02.670 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2610_fsm 2024-04-01T01:34:02.670 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2610_vm 2024-04-01T01:34:02.670 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2611 2024-04-01T01:34:02.670 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2612 2024-04-01T01:34:02.670 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2612_fsm 2024-04-01T01:34:02.670 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2612_vm 2024-04-01T01:34:02.670 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2613 2024-04-01T01:34:02.670 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2615 2024-04-01T01:34:02.670 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2615_fsm 2024-04-01T01:34:02.671 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2615_vm 2024-04-01T01:34:02.671 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2616 2024-04-01T01:34:02.671 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2616_fsm 2024-04-01T01:34:02.671 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2616_vm 2024-04-01T01:34:02.671 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 1 01:33 2617 2024-04-01T01:34:02.671 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2617_fsm 2024-04-01T01:34:02.671 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2617_vm 2024-04-01T01:34:02.671 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 1 01:33 2618 2024-04-01T01:34:02.671 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2618_fsm 2024-04-01T01:34:02.671 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2618_vm 2024-04-01T01:34:02.671 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 1 01:33 2619 2024-04-01T01:34:02.671 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2619_fsm 2024-04-01T01:34:02.671 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2619_vm 2024-04-01T01:34:02.671 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2620 2024-04-01T01:34:02.671 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2650 2024-04-01T01:34:02.672 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2651 2024-04-01T01:34:02.672 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2652 2024-04-01T01:34:02.672 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2653 2024-04-01T01:34:02.672 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2654 2024-04-01T01:34:02.672 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2655 2024-04-01T01:34:02.672 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2656 2024-04-01T01:34:02.672 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2657 2024-04-01T01:34:02.672 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 1 01:33 2658 2024-04-01T01:34:02.672 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 1 01:33 2659 2024-04-01T01:34:02.672 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2660 2024-04-01T01:34:02.672 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2661 2024-04-01T01:34:02.672 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2662 2024-04-01T01:34:02.672 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2663 2024-04-01T01:34:02.672 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2664 2024-04-01T01:34:02.672 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2665 2024-04-01T01:34:02.673 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2666 2024-04-01T01:34:02.673 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2667 2024-04-01T01:34:02.673 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2668 2024-04-01T01:34:02.673 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2669 2024-04-01T01:34:02.673 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2670 2024-04-01T01:34:02.673 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 1 01:33 2673 2024-04-01T01:34:02.673 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 1 01:33 2674 2024-04-01T01:34:02.673 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 1 01:33 2675 2024-04-01T01:34:02.673 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2678 2024-04-01T01:34:02.673 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2679 2024-04-01T01:34:02.673 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2680 2024-04-01T01:34:02.673 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2681 2024-04-01T01:34:02.673 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2682 2024-04-01T01:34:02.673 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2683 2024-04-01T01:34:02.673 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2684 2024-04-01T01:34:02.674 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2685 2024-04-01T01:34:02.674 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2686 2024-04-01T01:34:02.674 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2687 2024-04-01T01:34:02.674 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2688 2024-04-01T01:34:02.674 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2689 2024-04-01T01:34:02.674 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 1 01:33 2690 2024-04-01T01:34:02.674 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 1 01:33 2691 2024-04-01T01:34:02.674 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2692 2024-04-01T01:34:02.674 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2693 2024-04-01T01:34:02.674 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2696 2024-04-01T01:34:02.674 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2699 2024-04-01T01:34:02.674 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2701 2024-04-01T01:34:02.674 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2702 2024-04-01T01:34:02.674 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2703 2024-04-01T01:34:02.675 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2704 2024-04-01T01:34:02.675 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2753 2024-04-01T01:34:02.675 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2753_fsm 2024-04-01T01:34:02.675 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2753_vm 2024-04-01T01:34:02.675 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2754 2024-04-01T01:34:02.675 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2755 2024-04-01T01:34:02.675 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2756 2024-04-01T01:34:02.675 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2757 2024-04-01T01:34:02.675 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2830 2024-04-01T01:34:02.675 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2831 2024-04-01T01:34:02.675 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2832 2024-04-01T01:34:02.675 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2833 2024-04-01T01:34:02.675 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2834 2024-04-01T01:34:02.675 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2835 2024-04-01T01:34:02.675 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2836 2024-04-01T01:34:02.676 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2837 2024-04-01T01:34:02.676 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 1 01:33 2838 2024-04-01T01:34:02.676 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2838_fsm 2024-04-01T01:34:02.676 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2838_vm 2024-04-01T01:34:02.676 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2839 2024-04-01T01:34:02.676 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2840 2024-04-01T01:34:02.676 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2840_fsm 2024-04-01T01:34:02.676 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2840_vm 2024-04-01T01:34:02.676 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2841 2024-04-01T01:34:02.676 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2995 2024-04-01T01:34:02.676 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2996 2024-04-01T01:34:02.676 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3079 2024-04-01T01:34:02.676 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3079_fsm 2024-04-01T01:34:02.676 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3079_vm 2024-04-01T01:34:02.677 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3080 2024-04-01T01:34:02.677 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3081 2024-04-01T01:34:02.677 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 3085 2024-04-01T01:34:02.677 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3118 2024-04-01T01:34:02.677 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3119 2024-04-01T01:34:02.677 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 1 01:33 3164 2024-04-01T01:34:02.677 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3256 2024-04-01T01:34:02.677 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3257 2024-04-01T01:34:02.677 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3258 2024-04-01T01:34:02.677 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3350 2024-04-01T01:34:02.677 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3351 2024-04-01T01:34:02.677 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3379 2024-04-01T01:34:02.677 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3380 2024-04-01T01:34:02.678 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3381 2024-04-01T01:34:02.678 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3394 2024-04-01T01:34:02.678 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3394_fsm 2024-04-01T01:34:02.678 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3394_vm 2024-04-01T01:34:02.678 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3395 2024-04-01T01:34:02.678 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3429 2024-04-01T01:34:02.678 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3430 2024-04-01T01:34:02.678 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3431 2024-04-01T01:34:02.678 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3433 2024-04-01T01:34:02.678 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3439 2024-04-01T01:34:02.678 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3440 2024-04-01T01:34:02.679 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 3455 2024-04-01T01:34:02.679 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 1 01:33 3456 2024-04-01T01:34:02.679 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3456_fsm 2024-04-01T01:34:02.679 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3456_vm 2024-04-01T01:34:02.679 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3466 2024-04-01T01:34:02.679 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3467 2024-04-01T01:34:02.679 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3468 2024-04-01T01:34:02.679 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3501 2024-04-01T01:34:02.679 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3502 2024-04-01T01:34:02.679 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3503 2024-04-01T01:34:02.679 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3534 2024-04-01T01:34:02.679 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3541 2024-04-01T01:34:02.679 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3541_fsm 2024-04-01T01:34:02.679 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3541_vm 2024-04-01T01:34:02.680 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3542 2024-04-01T01:34:02.680 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3574 2024-04-01T01:34:02.680 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3575 2024-04-01T01:34:02.680 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3576 2024-04-01T01:34:02.680 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3596 2024-04-01T01:34:02.680 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3597 2024-04-01T01:34:02.680 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3598 2024-04-01T01:34:02.680 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3599 2024-04-01T01:34:02.680 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3600 2024-04-01T01:34:02.680 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3600_fsm 2024-04-01T01:34:02.680 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3600_vm 2024-04-01T01:34:02.680 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3601 2024-04-01T01:34:02.680 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3601_fsm 2024-04-01T01:34:02.680 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3601_vm 2024-04-01T01:34:02.680 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3602 2024-04-01T01:34:02.681 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3602_fsm 2024-04-01T01:34:02.681 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3602_vm 2024-04-01T01:34:02.681 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3603 2024-04-01T01:34:02.681 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3603_fsm 2024-04-01T01:34:02.681 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3603_vm 2024-04-01T01:34:02.681 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3604 2024-04-01T01:34:02.681 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3605 2024-04-01T01:34:02.681 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3606 2024-04-01T01:34:02.681 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3607 2024-04-01T01:34:02.681 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3608 2024-04-01T01:34:02.681 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 3609 2024-04-01T01:34:02.681 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3712 2024-04-01T01:34:02.681 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3764 2024-04-01T01:34:02.681 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3764_fsm 2024-04-01T01:34:02.682 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3764_vm 2024-04-01T01:34:02.682 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3766 2024-04-01T01:34:02.682 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3767 2024-04-01T01:34:02.682 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3997 2024-04-01T01:34:02.682 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4143 2024-04-01T01:34:02.682 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4144 2024-04-01T01:34:02.682 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4145 2024-04-01T01:34:02.682 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4146 2024-04-01T01:34:02.682 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4147 2024-04-01T01:34:02.682 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4148 2024-04-01T01:34:02.682 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4149 2024-04-01T01:34:02.682 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4150 2024-04-01T01:34:02.682 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4151 2024-04-01T01:34:02.682 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4152 2024-04-01T01:34:02.683 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4153 2024-04-01T01:34:02.683 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4154 2024-04-01T01:34:02.683 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4155 2024-04-01T01:34:02.683 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4156 2024-04-01T01:34:02.683 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4157 2024-04-01T01:34:02.683 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4158 2024-04-01T01:34:02.683 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4159 2024-04-01T01:34:02.683 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4160 2024-04-01T01:34:02.683 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4161 2024-04-01T01:34:02.683 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4162 2024-04-01T01:34:02.683 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4163 2024-04-01T01:34:02.683 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4164 2024-04-01T01:34:02.683 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4165 2024-04-01T01:34:02.683 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4166 2024-04-01T01:34:02.683 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4167 2024-04-01T01:34:02.697 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4168 2024-04-01T01:34:02.697 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4169 2024-04-01T01:34:02.697 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4170 2024-04-01T01:34:02.697 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4171 2024-04-01T01:34:02.697 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4172 2024-04-01T01:34:02.697 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4173 2024-04-01T01:34:02.697 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4174 2024-04-01T01:34:02.697 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 5002 2024-04-01T01:34:02.697 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 548 2024-04-01T01:34:02.697 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 549 2024-04-01T01:34:02.697 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6102 2024-04-01T01:34:02.697 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6104 2024-04-01T01:34:02.698 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6106 2024-04-01T01:34:02.698 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6110 2024-04-01T01:34:02.698 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6111 2024-04-01T01:34:02.698 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6112 2024-04-01T01:34:02.698 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6113 2024-04-01T01:34:02.698 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6117 2024-04-01T01:34:02.698 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 826 2024-04-01T01:34:02.698 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 827 2024-04-01T01:34:02.698 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 828 2024-04-01T01:34:02.698 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 1 01:33 pg_filenode.map 2024-04-01T01:34:02.698 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 1 01:33 PG_VERSION 2024-04-01T01:34:02.698 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.698 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/global: 2024-04-01T01:34:02.698 INFO:teuthology.orchestra.run.smithi070.stdout:total 538 2024-04-01T01:34:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 549376 Apr 1 01:33 . 2024-04-01T01:34:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1213 2024-04-01T01:34:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1213_fsm 2024-04-01T01:34:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1213_vm 2024-04-01T01:34:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1214 2024-04-01T01:34:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1214_fsm 2024-04-01T01:34:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1214_vm 2024-04-01T01:34:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 1232 2024-04-01T01:34:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 1233 2024-04-01T01:34:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1260 2024-04-01T01:34:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1260_fsm 2024-04-01T01:34:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1260_vm 2024-04-01T01:34:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1261 2024-04-01T01:34:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1261_fsm 2024-04-01T01:34:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1261_vm 2024-04-01T01:34:02.700 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1262 2024-04-01T01:34:02.700 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1262_fsm 2024-04-01T01:34:02.700 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1262_vm 2024-04-01T01:34:02.700 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2396 2024-04-01T01:34:02.700 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2396_fsm 2024-04-01T01:34:02.700 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2396_vm 2024-04-01T01:34:02.700 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2397 2024-04-01T01:34:02.700 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2671 2024-04-01T01:34:02.700 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2672 2024-04-01T01:34:02.701 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2676 2024-04-01T01:34:02.701 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2677 2024-04-01T01:34:02.701 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2694 2024-04-01T01:34:02.701 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2695 2024-04-01T01:34:02.701 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2697 2024-04-01T01:34:02.701 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2698 2024-04-01T01:34:02.701 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2846 2024-04-01T01:34:02.701 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2847 2024-04-01T01:34:02.701 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2964 2024-04-01T01:34:02.701 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2965 2024-04-01T01:34:02.701 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2966 2024-04-01T01:34:02.701 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2967 2024-04-01T01:34:02.701 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3592 2024-04-01T01:34:02.701 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3593 2024-04-01T01:34:02.701 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4060 2024-04-01T01:34:02.702 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4061 2024-04-01T01:34:02.702 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4175 2024-04-01T01:34:02.702 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4176 2024-04-01T01:34:02.702 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4177 2024-04-01T01:34:02.702 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4178 2024-04-01T01:34:02.702 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4181 2024-04-01T01:34:02.702 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4182 2024-04-01T01:34:02.702 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4183 2024-04-01T01:34:02.702 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4184 2024-04-01T01:34:02.702 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4185 2024-04-01T01:34:02.702 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4186 2024-04-01T01:34:02.702 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6000 2024-04-01T01:34:02.702 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6001 2024-04-01T01:34:02.702 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6002 2024-04-01T01:34:02.703 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6100 2024-04-01T01:34:02.703 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6114 2024-04-01T01:34:02.703 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6115 2024-04-01T01:34:02.703 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 pg_control 2024-04-01T01:34:02.703 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 1 01:33 pg_filenode.map 2024-04-01T01:34:02.703 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.703 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/log: 2024-04-01T01:34:02.703 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:34:02.703 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:34 . 2024-04-01T01:34:02.744 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.745 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.745 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-01T01:34:02.745 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:34:02.745 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:34:02.745 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.745 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.745 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-01T01:34:02.745 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:34:02.745 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:34:02.745 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.745 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.745 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-01T01:34:02.745 INFO:teuthology.orchestra.run.smithi070.stdout:total 3 2024-04-01T01:34:02.745 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 1 01:33 . 2024-04-01T01:34:02.746 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.746 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 mappings 2024-04-01T01:34:02.746 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 1 01:33 replorigin_checkpoint 2024-04-01T01:34:02.746 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 snapshots 2024-04-01T01:34:02.746 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.746 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-01T01:34:02.746 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:34:02.746 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:34:02.746 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 1 01:33 .. 2024-04-01T01:34:02.746 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.746 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-01T01:34:02.746 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:34:02.746 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:34:02.747 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 1 01:33 .. 2024-04-01T01:34:02.747 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.747 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-01T01:34:02.747 INFO:teuthology.orchestra.run.smithi070.stdout:total 2 2024-04-01T01:34:02.747 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 . 2024-04-01T01:34:02.747 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.747 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 members 2024-04-01T01:34:02.747 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 offsets 2024-04-01T01:34:02.747 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.747 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-01T01:34:02.747 INFO:teuthology.orchestra.run.smithi070.stdout:total 9 2024-04-01T01:34:02.747 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 . 2024-04-01T01:34:02.747 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 .. 2024-04-01T01:34:02.747 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 0000 2024-04-01T01:34:02.748 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.748 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-01T01:34:02.748 INFO:teuthology.orchestra.run.smithi070.stdout:total 9 2024-04-01T01:34:02.748 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 . 2024-04-01T01:34:02.748 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 .. 2024-04-01T01:34:02.748 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 0000 2024-04-01T01:34:02.748 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.748 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-01T01:34:02.748 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:34:02.748 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:34:02.749 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.749 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.749 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-01T01:34:02.749 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:34:02.749 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:34:02.749 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.749 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.749 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-01T01:34:02.749 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:34:02.749 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:34:02.749 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.749 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.749 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-01T01:34:02.749 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:34:02.749 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:34:02.750 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.750 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.750 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-01T01:34:02.750 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:34:02.750 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:34:02.750 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.750 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.750 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-01T01:34:02.750 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:34:02.750 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:34:02.750 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.750 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.750 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-01T01:34:02.760 INFO:teuthology.orchestra.run.smithi070.stdout:total 9 2024-04-01T01:34:02.760 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 . 2024-04-01T01:34:02.760 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.760 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 0000 2024-04-01T01:34:02.760 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.760 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-01T01:34:02.760 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:34:02.760 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:34:02.760 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.760 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.760 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-01T01:34:02.760 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:34:02.761 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:34:02.761 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.761 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.761 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-01T01:34:02.761 INFO:teuthology.orchestra.run.smithi070.stdout:total 16386 2024-04-01T01:34:02.761 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:33 . 2024-04-01T01:34:02.761 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.761 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:33 000000010000000000000001 2024-04-01T01:34:02.761 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 archive_status 2024-04-01T01:34:02.761 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.761 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-01T01:34:02.761 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:34:02.761 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:34:02.761 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:33 .. 2024-04-01T01:34:02.762 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:02.762 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-01T01:34:02.762 INFO:teuthology.orchestra.run.smithi070.stdout:total 9 2024-04-01T01:34:02.762 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 . 2024-04-01T01:34:02.762 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 41283110 Apr 1 01:34 .. 2024-04-01T01:34:02.762 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 0000 2024-04-01T01:34:02.763 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl start postgresql' 2024-04-01T01:34:02.992 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 32 -i' 2024-04-01T01:34:03.072 INFO:teuthology.orchestra.run.smithi070.stderr:dropping old tables... 2024-04-01T01:34:03.079 INFO:teuthology.orchestra.run.smithi070.stderr:NOTICE: table "pgbench_accounts" does not exist, skipping 2024-04-01T01:34:03.079 INFO:teuthology.orchestra.run.smithi070.stderr:NOTICE: table "pgbench_branches" does not exist, skipping 2024-04-01T01:34:03.079 INFO:teuthology.orchestra.run.smithi070.stderr:NOTICE: table "pgbench_history" does not exist, skipping 2024-04-01T01:34:03.079 INFO:teuthology.orchestra.run.smithi070.stderr:NOTICE: table "pgbench_tellers" does not exist, skipping 2024-04-01T01:34:03.079 INFO:teuthology.orchestra.run.smithi070.stderr:creating tables... 2024-04-01T01:34:03.134 INFO:teuthology.orchestra.run.smithi070.stderr:generating data (client-side)... 2024-04-01T01:34:03.635 INFO:teuthology.orchestra.run.smithi070.stderr:100000 of 3200000 tuples (3%) done (elapsed 0.48 s, remaining 14.85 s) 2024-04-01T01:34:03.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:03 smithi189 ceph-mon[35753]: pgmap v375: 129 pgs: 129 active+clean; 47 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 3.9 MiB/s wr, 352 op/s 2024-04-01T01:34:04.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:03 smithi070 ceph-mon[32179]: pgmap v375: 129 pgs: 129 active+clean; 47 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 3.9 MiB/s wr, 352 op/s 2024-04-01T01:34:04.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:03 smithi081 ceph-mon[35142]: pgmap v375: 129 pgs: 129 active+clean; 47 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 3.9 MiB/s wr, 352 op/s 2024-04-01T01:34:04.686 INFO:teuthology.orchestra.run.smithi070.stderr:200000 of 3200000 tuples (6%) done (elapsed 1.53 s, remaining 22.95 s) 2024-04-01T01:34:05.814 INFO:teuthology.orchestra.run.smithi070.stderr:300000 of 3200000 tuples (9%) done (elapsed 2.66 s, remaining 25.70 s) 2024-04-01T01:34:05.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:05 smithi189 ceph-mon[35753]: pgmap v376: 129 pgs: 129 active+clean; 52 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-04-01T01:34:06.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:05 smithi070 ceph-mon[32179]: pgmap v376: 129 pgs: 129 active+clean; 52 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-04-01T01:34:06.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:05 smithi081 ceph-mon[35142]: pgmap v376: 129 pgs: 129 active+clean; 52 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-04-01T01:34:06.517 INFO:teuthology.orchestra.run.smithi070.stderr:400000 of 3200000 tuples (12%) done (elapsed 3.36 s, remaining 23.53 s) 2024-04-01T01:34:07.599 INFO:teuthology.orchestra.run.smithi070.stderr:500000 of 3200000 tuples (15%) done (elapsed 4.44 s, remaining 23.99 s) 2024-04-01T01:34:07.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:07 smithi189 ceph-mon[35753]: pgmap v377: 129 pgs: 129 active+clean; 85 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 7.4 MiB/s wr, 358 op/s 2024-04-01T01:34:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:07 smithi070 ceph-mon[32179]: pgmap v377: 129 pgs: 129 active+clean; 85 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 7.4 MiB/s wr, 358 op/s 2024-04-01T01:34:08.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:07 smithi081 ceph-mon[35142]: pgmap v377: 129 pgs: 129 active+clean; 85 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 7.4 MiB/s wr, 358 op/s 2024-04-01T01:34:08.376 INFO:teuthology.orchestra.run.smithi070.stderr:600000 of 3200000 tuples (18%) done (elapsed 5.22 s, remaining 22.62 s) 2024-04-01T01:34:09.418 INFO:teuthology.orchestra.run.smithi070.stderr:700000 of 3200000 tuples (21%) done (elapsed 6.26 s, remaining 22.37 s) 2024-04-01T01:34:09.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:09 smithi189 ceph-mon[35753]: pgmap v378: 129 pgs: 129 active+clean; 122 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 12 MiB/s wr, 345 op/s 2024-04-01T01:34:10.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:09 smithi070 ceph-mon[32179]: pgmap v378: 129 pgs: 129 active+clean; 122 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 12 MiB/s wr, 345 op/s 2024-04-01T01:34:10.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:09 smithi081 ceph-mon[35142]: pgmap v378: 129 pgs: 129 active+clean; 122 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 12 MiB/s wr, 345 op/s 2024-04-01T01:34:10.495 INFO:teuthology.orchestra.run.smithi070.stderr:800000 of 3200000 tuples (25%) done (elapsed 7.34 s, remaining 22.02 s) 2024-04-01T01:34:11.281 INFO:teuthology.orchestra.run.smithi070.stderr:900000 of 3200000 tuples (28%) done (elapsed 8.13 s, remaining 20.77 s) 2024-04-01T01:34:11.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.526+0000 7fe41f8ed640 1 -- 172.21.15.70:0/3028108683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418104b20 msgr2=0x7fe418106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:11.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.526+0000 7fe41f8ed640 1 --2- 172.21.15.70:0/3028108683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418104b20 0x7fe418106f10 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe408009900 tx=0x7fe40802bc30 comp rx=0 tx=0).stop 2024-04-01T01:34:11.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.527+0000 7fe41f8ed640 1 -- 172.21.15.70:0/3028108683 shutdown_connections 2024-04-01T01:34:11.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.527+0000 7fe41f8ed640 1 --2- 172.21.15.70:0/3028108683 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe418107450 0x7fe4181098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.527+0000 7fe41f8ed640 1 --2- 172.21.15.70:0/3028108683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418104b20 0x7fe418106f10 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.527+0000 7fe41f8ed640 1 --2- 172.21.15.70:0/3028108683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4181021f0 0x7fe4181045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.527+0000 7fe41f8ed640 1 -- 172.21.15.70:0/3028108683 >> 172.21.15.70:0/3028108683 conn(0x7fe4180fbbe0 msgr2=0x7fe4180fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:11.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.527+0000 7fe41f8ed640 1 -- 172.21.15.70:0/3028108683 shutdown_connections 2024-04-01T01:34:11.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.527+0000 7fe41f8ed640 1 -- 172.21.15.70:0/3028108683 wait complete. 2024-04-01T01:34:11.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.527+0000 7fe41f8ed640 1 Processor -- start 2024-04-01T01:34:11.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.527+0000 7fe41f8ed640 1 -- start start 2024-04-01T01:34:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.527+0000 7fe41f8ed640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4181021f0 0x7fe418101b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.527+0000 7fe41f8ed640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe418104b20 0x7fe4181001a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.527+0000 7fe41f8ed640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418107450 0x7fe4181006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.527+0000 7fe41f8ed640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe41810d2b0 con 0x7fe418107450 2024-04-01T01:34:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.527+0000 7fe41f8ed640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe41810d130 con 0x7fe4181021f0 2024-04-01T01:34:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.527+0000 7fe41f8ed640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe41810d430 con 0x7fe418104b20 2024-04-01T01:34:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe41d689640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4181021f0 0x7fe418101b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe41de8a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418107450 0x7fe4181006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe41ce88640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe418104b20 0x7fe4181001a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe41d689640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4181021f0 0x7fe418101b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38252/0 (socket says 172.21.15.70:38252) 2024-04-01T01:34:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe41de8a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418107450 0x7fe4181006e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56656/0 (socket says 172.21.15.70:56656) 2024-04-01T01:34:11.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe41d689640 1 -- 172.21.15.70:0/3229256015 learned_addr learned my addr 172.21.15.70:0/3229256015 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe41d689640 1 -- 172.21.15.70:0/3229256015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe418104b20 msgr2=0x7fe4181001a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe41d689640 1 --2- 172.21.15.70:0/3229256015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe418104b20 0x7fe4181001a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe41d689640 1 -- 172.21.15.70:0/3229256015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418107450 msgr2=0x7fe4181006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe41d689640 1 --2- 172.21.15.70:0/3229256015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418107450 0x7fe4181006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe41d689640 1 -- 172.21.15.70:0/3229256015 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe418100f70 con 0x7fe4181021f0 2024-04-01T01:34:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe41de8a640 1 --2- 172.21.15.70:0/3229256015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418107450 0x7fe4181006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe41ce88640 1 --2- 172.21.15.70:0/3229256015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe418104b20 0x7fe4181001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe41d689640 1 --2- 172.21.15.70:0/3229256015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4181021f0 0x7fe418101b60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe41400e9f0 tx=0x7fe41400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe40e7fc640 1 -- 172.21.15.70:0/3229256015 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41400cd80 con 0x7fe4181021f0 2024-04-01T01:34:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe41f8ed640 1 -- 172.21.15.70:0/3229256015 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4181a1aa0 con 0x7fe4181021f0 2024-04-01T01:34:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe40e7fc640 1 -- 172.21.15.70:0/3229256015 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4140054d0 con 0x7fe4181021f0 2024-04-01T01:34:11.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe40e7fc640 1 -- 172.21.15.70:0/3229256015 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe414010490 con 0x7fe4181021f0 2024-04-01T01:34:11.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.528+0000 7fe41f8ed640 1 -- 172.21.15.70:0/3229256015 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4181a1fc0 con 0x7fe4181021f0 2024-04-01T01:34:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.531+0000 7fe4225a9740 1 Processor -- start 2024-04-01T01:34:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.531+0000 7fe4225a9740 1 -- start start 2024-04-01T01:34:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.531+0000 7fe40e7fc640 1 -- 172.21.15.70:0/3229256015 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7fe414005020 con 0x7fe4181021f0 2024-04-01T01:34:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.531+0000 7fe4225a9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e918d0 0x55dd39e91cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe4225a9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e922c0 0x55dd39e96dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe4225a9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e97430 0x55dd39e998c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe4225a9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dd39df58f0 con 0x55dd39e922c0 2024-04-01T01:34:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe4225a9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55dd39d9e8d0 con 0x55dd39e918d0 2024-04-01T01:34:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe4225a9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dd39df1990 con 0x55dd39e97430 2024-04-01T01:34:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe41d689640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e922c0 0x55dd39e96dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe41de8a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e918d0 0x55dd39e91cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe41d689640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e922c0 0x55dd39e96dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56672/0 (socket says 172.21.15.70:56672) 2024-04-01T01:34:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe41de8a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e918d0 0x55dd39e91cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38264/0 (socket says 172.21.15.70:38264) 2024-04-01T01:34:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe41ce88640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e97430 0x55dd39e998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe41d689640 1 -- 172.21.15.70:0/2007822670 learned_addr learned my addr 172.21.15.70:0/2007822670 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe40e7fc640 1 --2- 172.21.15.70:0/3229256015 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe4040a25e0 0x7fe4040a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe41ce88640 1 -- 172.21.15.70:0/2007822670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e918d0 msgr2=0x55dd39e91cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe41ce88640 1 --2- 172.21.15.70:0/2007822670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e918d0 0x55dd39e91cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe41ce88640 1 -- 172.21.15.70:0/2007822670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e922c0 msgr2=0x55dd39e96dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe41ce88640 1 --2- 172.21.15.70:0/2007822670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e922c0 0x55dd39e96dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe41ce88640 1 -- 172.21.15.70:0/2007822670 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd39b0a210 con 0x55dd39e97430 2024-04-01T01:34:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe41de8a640 1 --2- 172.21.15.70:0/2007822670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e918d0 0x55dd39e91cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe41de8a640 1 --2- 172.21.15.70:0/3229256015 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe4040a25e0 0x7fe4040a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe40e7fc640 1 -- 172.21.15.70:0/3229256015 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe4140cf030 con 0x7fe4181021f0 2024-04-01T01:34:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.532+0000 7fe41ce88640 1 --2- 172.21.15.70:0/2007822670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e97430 0x55dd39e998c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe408036040 tx=0x7fe408009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bdb27b4b2ac7e24 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.533+0000 7fe4037fe640 1 -- 172.21.15.70:0/2007822670 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe408045020 con 0x55dd39e97430 2024-04-01T01:34:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.533+0000 7fe4037fe640 1 -- 172.21.15.70:0/2007822670 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe40802cc90 con 0x55dd39e97430 2024-04-01T01:34:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.533+0000 7fe4037fe640 1 -- 172.21.15.70:0/2007822670 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe408004920 con 0x55dd39e97430 2024-04-01T01:34:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.533+0000 7fe41de8a640 1 --2- 172.21.15.70:0/3229256015 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe4040a25e0 0x7fe4040a4aa0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x55dd39e92b20 tx=0x7fe410004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.533+0000 7fe4225a9740 1 -- 172.21.15.70:0/2007822670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e97430 msgr2=0x55dd39e998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.533+0000 7fe4225a9740 1 --2- 172.21.15.70:0/2007822670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e97430 0x55dd39e998c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe408036040 tx=0x7fe408009580 comp rx=0 tx=0).stop 2024-04-01T01:34:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.533+0000 7fe4225a9740 1 -- 172.21.15.70:0/2007822670 shutdown_connections 2024-04-01T01:34:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.533+0000 7fe4225a9740 1 --2- 172.21.15.70:0/2007822670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e97430 0x55dd39e998c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.533+0000 7fe4225a9740 1 --2- 172.21.15.70:0/2007822670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e922c0 0x55dd39e96dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.533+0000 7fe4225a9740 1 --2- 172.21.15.70:0/2007822670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e918d0 0x55dd39e91cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.533+0000 7fe4225a9740 1 -- 172.21.15.70:0/2007822670 >> 172.21.15.70:0/2007822670 conn(0x55dd39e8ce40 msgr2=0x55dd39e8d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.534+0000 7fe4225a9740 1 -- 172.21.15.70:0/2007822670 shutdown_connections 2024-04-01T01:34:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.534+0000 7fe4225a9740 1 -- 172.21.15.70:0/2007822670 wait complete. 2024-04-01T01:34:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.534+0000 7fe4225a9740 1 -- start start 2024-04-01T01:34:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.534+0000 7fe4225a9740 1 Processor -- start 2024-04-01T01:34:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.534+0000 7fe4225a9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e918d0 0x55dd39e9bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.534+0000 7fe4225a9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e922c0 0x55dd39e9c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.534+0000 7fe4225a9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e97430 0x55dd39ea3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.534+0000 7fe4225a9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dd39e9a380 con 0x55dd39e922c0 2024-04-01T01:34:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.534+0000 7fe4225a9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55dd39d93f80 con 0x55dd39e97430 2024-04-01T01:34:11.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.534+0000 7fe4225a9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dd39e9a500 con 0x55dd39e918d0 2024-04-01T01:34:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.534+0000 7fe4225a9740 10 client.0 _ll_register_callbacks cb 0x55dd39dec290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe41ce88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e922c0 0x55dd39e9c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe41d689640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e918d0 0x55dd39e9bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe41d689640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e918d0 0x55dd39e9bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:53684/0 (socket says 172.21.15.70:53684) 2024-04-01T01:34:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe41ce88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e922c0 0x55dd39e9c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56678/0 (socket says 172.21.15.70:56678) 2024-04-01T01:34:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe41d689640 1 -- 172.21.15.70:0/2548214026 learned_addr learned my addr 172.21.15.70:0/2548214026 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe41de8a640 1 --2- 172.21.15.70:0/2548214026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e97430 0x55dd39ea3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe41d689640 1 -- 172.21.15.70:0/2548214026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e97430 msgr2=0x55dd39ea3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe41d689640 1 --2- 172.21.15.70:0/2548214026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e97430 0x55dd39ea3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe41d689640 1 -- 172.21.15.70:0/2548214026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e922c0 msgr2=0x55dd39e9c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe41d689640 1 --2- 172.21.15.70:0/2548214026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e922c0 0x55dd39e9c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe41d689640 1 -- 172.21.15.70:0/2548214026 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd39bd6620 con 0x55dd39e918d0 2024-04-01T01:34:11.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe41de8a640 1 --2- 172.21.15.70:0/2548214026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e97430 0x55dd39ea3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe41ce88640 1 --2- 172.21.15.70:0/2548214026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e922c0 0x55dd39e9c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe41d689640 1 --2- 172.21.15.70:0/2548214026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e918d0 0x55dd39e9bbd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe414097a20 tx=0x7fe414097260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0e808a6ff834ba3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe400ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:34:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe400ff9640 1 -- 172.21.15.70:0/2548214026 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41409b100 con 0x55dd39e918d0 2024-04-01T01:34:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe400ff9640 1 -- 172.21.15.70:0/2548214026 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe41409f470 con 0x55dd39e918d0 2024-04-01T01:34:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe4225a9740 1 -- 172.21.15.70:0/2548214026 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe40802cc90 con 0x55dd39e918d0 2024-04-01T01:34:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe400ff9640 1 -- 172.21.15.70:0/2548214026 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe41409af20 con 0x55dd39e918d0 2024-04-01T01:34:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe4225a9740 10 client.25459 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.535+0000 7fe4225a9740 1 -- 172.21.15.70:0/2548214026 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dd39ea61a0 con 0x55dd39e918d0 2024-04-01T01:34:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.536+0000 7fe400ff9640 1 -- 172.21.15.70:0/2548214026 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe40802cc90 con 0x55dd39e918d0 2024-04-01T01:34:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.536+0000 7fe400ff9640 1 -- 172.21.15.70:0/2548214026 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe4140d10d0 con 0x55dd39e918d0 2024-04-01T01:34:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.536+0000 7fe4225a9740 10 client.25459 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.536+0000 7fe4225a9740 10 client.25459 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.536+0000 7fe4225a9740 20 client.25459 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.536+0000 7fe4225a9740 1 --2- 172.21.15.70:0/2548214026 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55dd39ea7890 0x55dd39ea9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.536+0000 7fe4225a9740 4 client.25459 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:11.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.536+0000 7fe4225a9740 1 -- 172.21.15.70:0/2548214026 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd39e09120 con 0x55dd39ea7890 2024-04-01T01:34:11.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.536+0000 7fe41ce88640 1 --2- 172.21.15.70:0/2548214026 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55dd39ea7890 0x55dd39ea9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.536+0000 7fe400ff9640 10 client.25459 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:11.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.536+0000 7fe41ce88640 1 --2- 172.21.15.70:0/2548214026 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55dd39ea7890 0x55dd39ea9d20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96e5727d87ebb71a server_cookie=8538671cdf13b7ee in_seq=0 out_seq=0 2024-04-01T01:34:11.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.537+0000 7fe400ff9640 1 -- 172.21.15.70:0/2548214026 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dd39e09120 con 0x55dd39ea7890 2024-04-01T01:34:11.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.537+0000 7fe400ff9640 10 client.25459 handle_command_reply: tid=0 2024-04-01T01:34:11.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.537+0000 7fe4225a9740 1 client.25459 shutdown 2024-04-01T01:34:11.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.537+0000 7fe4225a9740 1 -- 172.21.15.70:0/2548214026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e918d0 msgr2=0x55dd39e9bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:11.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.537+0000 7fe4225a9740 1 --2- 172.21.15.70:0/2548214026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e918d0 0x55dd39e9bbd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe414097a20 tx=0x7fe414097260 comp rx=0 tx=0).stop 2024-04-01T01:34:11.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.537+0000 7fe4225a9740 1 -- 172.21.15.70:0/2548214026 shutdown_connections 2024-04-01T01:34:11.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.537+0000 7fe4225a9740 1 --2- 172.21.15.70:0/2548214026 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55dd39ea7890 0x55dd39ea9d20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.537+0000 7fe4225a9740 1 --2- 172.21.15.70:0/2548214026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e97430 0x55dd39ea3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.537+0000 7fe4225a9740 1 --2- 172.21.15.70:0/2548214026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e922c0 0x55dd39e9c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.537+0000 7fe400ff9640 0 client.25459 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:11.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.537+0000 7fe4225a9740 1 --2- 172.21.15.70:0/2548214026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e918d0 0x55dd39e9bbd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.537+0000 7fe4225a9740 1 -- 172.21.15.70:0/2548214026 >> 172.21.15.70:0/2548214026 conn(0x55dd39e8ce40 msgr2=0x55dd39e8bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:11.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.537+0000 7fe4225a9740 1 -- 172.21.15.70:0/2548214026 shutdown_connections 2024-04-01T01:34:11.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.538+0000 7fe4225a9740 1 -- 172.21.15.70:0/2548214026 wait complete. 2024-04-01T01:34:11.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.538+0000 7fe4225a9740 20 client.25459 trim_cache size 0 max 16384 2024-04-01T01:34:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe4225a9740 1 Processor -- start 2024-04-01T01:34:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe4225a9740 1 -- start start 2024-04-01T01:34:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe4225a9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e8ce40 0x55dd39eaf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe4225a9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e918d0 0x55dd39ea3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe4225a9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e922c0 0x55dd39e900a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe4225a9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe4181ac4f0 con 0x55dd39e918d0 2024-04-01T01:34:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe4225a9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe4181ae030 con 0x55dd39e922c0 2024-04-01T01:34:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe4225a9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe4181acbc0 con 0x55dd39e8ce40 2024-04-01T01:34:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe41ce88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e918d0 0x55dd39ea3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe41ce88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e918d0 0x55dd39ea3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56690/0 (socket says 172.21.15.70:56690) 2024-04-01T01:34:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe41d689640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e8ce40 0x55dd39eaf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe41ce88640 1 -- 172.21.15.70:0/1248932594 learned_addr learned my addr 172.21.15.70:0/1248932594 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe41de8a640 1 --2- 172.21.15.70:0/1248932594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e922c0 0x55dd39e900a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe41ce88640 1 -- 172.21.15.70:0/1248932594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e8ce40 msgr2=0x55dd39eaf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe41ce88640 1 --2- 172.21.15.70:0/1248932594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e8ce40 0x55dd39eaf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe41ce88640 1 -- 172.21.15.70:0/1248932594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e922c0 msgr2=0x55dd39e900a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe41ce88640 1 --2- 172.21.15.70:0/1248932594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e922c0 0x55dd39e900a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe41ce88640 1 -- 172.21.15.70:0/1248932594 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd39e8dbb0 con 0x55dd39e918d0 2024-04-01T01:34:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe41d689640 1 --2- 172.21.15.70:0/1248932594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e8ce40 0x55dd39eaf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:34:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.555+0000 7fe41de8a640 1 --2- 172.21.15.70:0/1248932594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e922c0 0x55dd39e900a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:34:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.556+0000 7fe41ce88640 1 --2- 172.21.15.70:0/1248932594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e918d0 0x55dd39ea3110 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe408009900 tx=0x7fe408004350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d3947af626a12d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:11.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.556+0000 7fe402ffd640 1 -- 172.21.15.70:0/1248932594 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe408045040 con 0x55dd39e918d0 2024-04-01T01:34:11.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.556+0000 7fe402ffd640 1 -- 172.21.15.70:0/1248932594 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe40802cc90 con 0x55dd39e918d0 2024-04-01T01:34:11.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.556+0000 7fe402ffd640 1 -- 172.21.15.70:0/1248932594 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4080046c0 con 0x55dd39e918d0 2024-04-01T01:34:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.557+0000 7fe4225a9740 1 -- 172.21.15.70:0/1248932594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e918d0 msgr2=0x55dd39ea3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.557+0000 7fe4225a9740 1 --2- 172.21.15.70:0/1248932594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e918d0 0x55dd39ea3110 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe408009900 tx=0x7fe408004350 comp rx=0 tx=0).stop 2024-04-01T01:34:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.557+0000 7fe4225a9740 1 -- 172.21.15.70:0/1248932594 shutdown_connections 2024-04-01T01:34:11.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.557+0000 7fe4225a9740 1 --2- 172.21.15.70:0/1248932594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e922c0 0x55dd39e900a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.557+0000 7fe4225a9740 1 --2- 172.21.15.70:0/1248932594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e918d0 0x55dd39ea3110 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.557+0000 7fe4225a9740 1 --2- 172.21.15.70:0/1248932594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e8ce40 0x55dd39eaf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.557+0000 7fe4225a9740 1 -- 172.21.15.70:0/1248932594 >> 172.21.15.70:0/1248932594 conn(0x55dd39e92ec0 msgr2=0x55dd39e8d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.557+0000 7fe4225a9740 1 -- 172.21.15.70:0/1248932594 shutdown_connections 2024-04-01T01:34:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.557+0000 7fe4225a9740 1 -- 172.21.15.70:0/1248932594 wait complete. 2024-04-01T01:34:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.557+0000 7fe4225a9740 1 -- start start 2024-04-01T01:34:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.557+0000 7fe4225a9740 1 Processor -- start 2024-04-01T01:34:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.557+0000 7fe4225a9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e8ce40 0x55dd39ea04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.557+0000 7fe4225a9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e918d0 0x55dd39ea0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe4225a9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e922c0 0x55dd39ea1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe4225a9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe4181ad960 con 0x55dd39e918d0 2024-04-01T01:34:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe4225a9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe4181ad290 con 0x55dd39e922c0 2024-04-01T01:34:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe4225a9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe4181aa4d0 con 0x55dd39e8ce40 2024-04-01T01:34:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe4225a9740 10 client.0 _ll_register_callbacks cb 0x7fe41400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe41ce88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e918d0 0x55dd39ea0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe41ce88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e918d0 0x55dd39ea0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56692/0 (socket says 172.21.15.70:56692) 2024-04-01T01:34:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe41ce88640 1 -- 172.21.15.70:0/3292512053 learned_addr learned my addr 172.21.15.70:0/3292512053 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe41de8a640 1 --2- 172.21.15.70:0/3292512053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e922c0 0x55dd39ea1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe41d689640 1 --2- 172.21.15.70:0/3292512053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e8ce40 0x55dd39ea04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe41ce88640 1 -- 172.21.15.70:0/3292512053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e8ce40 msgr2=0x55dd39ea04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe41ce88640 1 --2- 172.21.15.70:0/3292512053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e8ce40 0x55dd39ea04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe41ce88640 1 -- 172.21.15.70:0/3292512053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e922c0 msgr2=0x55dd39ea1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe41ce88640 1 --2- 172.21.15.70:0/3292512053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e922c0 0x55dd39ea1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe41ce88640 1 -- 172.21.15.70:0/3292512053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd39ea2000 con 0x55dd39e918d0 2024-04-01T01:34:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe41de8a640 1 --2- 172.21.15.70:0/3292512053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e922c0 0x55dd39ea1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe41ce88640 1 --2- 172.21.15.70:0/3292512053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e918d0 0x55dd39ea0a00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe408009850 tx=0x7fe408004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67e20543b008c2d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe400ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe400ff9640 1 -- 172.21.15.70:0/3292512053 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40805c070 con 0x55dd39e918d0 2024-04-01T01:34:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe4225a9740 1 -- 172.21.15.70:0/3292512053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe40802cc90 con 0x55dd39e918d0 2024-04-01T01:34:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.558+0000 7fe400ff9640 1 -- 172.21.15.70:0/3292512053 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe408003e30 con 0x55dd39e918d0 2024-04-01T01:34:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.559+0000 7fe400ff9640 1 -- 172.21.15.70:0/3292512053 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe408068050 con 0x55dd39e918d0 2024-04-01T01:34:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.559+0000 7fe4225a9740 10 client.15729 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.559+0000 7fe4225a9740 1 -- 172.21.15.70:0/3292512053 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dd39ea3310 con 0x55dd39e918d0 2024-04-01T01:34:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.559+0000 7fe400ff9640 1 -- 172.21.15.70:0/3292512053 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe40802cc90 con 0x55dd39e918d0 2024-04-01T01:34:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.559+0000 7fe400ff9640 1 -- 172.21.15.70:0/3292512053 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe408004c00 con 0x55dd39e918d0 2024-04-01T01:34:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.559+0000 7fe4225a9740 10 client.15729 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.559+0000 7fe4225a9740 10 client.15729 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.559+0000 7fe4225a9740 20 client.15729 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.559+0000 7fe4225a9740 1 --2- 172.21.15.70:0/3292512053 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55dd39e97430 0x55dd39f0fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.559+0000 7fe4225a9740 4 client.15729 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:34:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.559+0000 7fe4225a9740 1 -- 172.21.15.70:0/3292512053 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd39e02e30 con 0x55dd39e97430 2024-04-01T01:34:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.560+0000 7fe41d689640 1 --2- 172.21.15.70:0/3292512053 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55dd39e97430 0x55dd39f0fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.560+0000 7fe400ff9640 10 client.15729 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.560+0000 7fe41d689640 1 --2- 172.21.15.70:0/3292512053 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55dd39e97430 0x55dd39f0fdc0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7aee54db7e52a04a server_cookie=fc75f39755c2c564 in_seq=0 out_seq=0 2024-04-01T01:34:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.673+0000 7fe400ff9640 1 -- 172.21.15.70:0/3292512053 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dd39e02e30 con 0x55dd39e97430 2024-04-01T01:34:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.673+0000 7fe400ff9640 10 client.15729 handle_command_reply: tid=0 2024-04-01T01:34:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.673+0000 7fe4225a9740 1 client.15729 shutdown 2024-04-01T01:34:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.673+0000 7fe4225a9740 1 -- 172.21.15.70:0/3292512053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e918d0 msgr2=0x55dd39ea0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.673+0000 7fe4225a9740 1 --2- 172.21.15.70:0/3292512053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e918d0 0x55dd39ea0a00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe408009850 tx=0x7fe408004580 comp rx=0 tx=0).stop 2024-04-01T01:34:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.673+0000 7fe4225a9740 1 -- 172.21.15.70:0/3292512053 shutdown_connections 2024-04-01T01:34:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.673+0000 7fe4225a9740 1 --2- 172.21.15.70:0/3292512053 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55dd39e97430 0x55dd39f0fdc0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.673+0000 7fe4225a9740 1 --2- 172.21.15.70:0/3292512053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dd39e922c0 0x55dd39ea1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.674+0000 7fe4225a9740 1 --2- 172.21.15.70:0/3292512053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dd39e918d0 0x55dd39ea0a00 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.674+0000 7fe4225a9740 1 --2- 172.21.15.70:0/3292512053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd39e8ce40 0x55dd39ea04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.674+0000 7fe400ff9640 0 client.15729 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.674+0000 7fe4225a9740 1 -- 172.21.15.70:0/3292512053 >> 172.21.15.70:0/3292512053 conn(0x55dd39e92ec0 msgr2=0x55dd39e900a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.674+0000 7fe4225a9740 1 -- 172.21.15.70:0/3292512053 shutdown_connections 2024-04-01T01:34:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.674+0000 7fe4225a9740 1 -- 172.21.15.70:0/3292512053 wait complete. 2024-04-01T01:34:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.674+0000 7fe4225a9740 20 client.15729 trim_cache size 0 max 16384 2024-04-01T01:34:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:34:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:34:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:34:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:34:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.675+0000 7fe41f8ed640 1 -- 172.21.15.70:0/3229256015 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe4040a25e0 msgr2=0x7fe4040a4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.675+0000 7fe41f8ed640 1 --2- 172.21.15.70:0/3229256015 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe4040a25e0 0x7fe4040a4aa0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x55dd39e92b20 tx=0x7fe410004a40 comp rx=0 tx=0).stop 2024-04-01T01:34:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.675+0000 7fe41f8ed640 1 -- 172.21.15.70:0/3229256015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4181021f0 msgr2=0x7fe418101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.675+0000 7fe41f8ed640 1 --2- 172.21.15.70:0/3229256015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4181021f0 0x7fe418101b60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe41400e9f0 tx=0x7fe41400edf0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.675+0000 7fe41f8ed640 1 -- 172.21.15.70:0/3229256015 shutdown_connections 2024-04-01T01:34:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.675+0000 7fe41f8ed640 1 --2- 172.21.15.70:0/3229256015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe418107450 0x7fe4181006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.675+0000 7fe41f8ed640 1 --2- 172.21.15.70:0/3229256015 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe4040a25e0 0x7fe4040a4aa0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.675+0000 7fe41f8ed640 1 --2- 172.21.15.70:0/3229256015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe418104b20 0x7fe4181001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.675+0000 7fe41f8ed640 1 --2- 172.21.15.70:0/3229256015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4181021f0 0x7fe418101b60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.675+0000 7fe41f8ed640 1 -- 172.21.15.70:0/3229256015 >> 172.21.15.70:0/3229256015 conn(0x7fe4180fbbe0 msgr2=0x7fe418102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.675+0000 7fe41f8ed640 1 -- 172.21.15.70:0/3229256015 shutdown_connections 2024-04-01T01:34:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:11.675+0000 7fe41f8ed640 1 -- 172.21.15.70:0/3229256015 wait complete. 2024-04-01T01:34:11.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:11 smithi189 ceph-mon[35753]: pgmap v379: 129 pgs: 129 active+clean; 154 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 18 MiB/s wr, 316 op/s 2024-04-01T01:34:12.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:11 smithi070 ceph-mon[32179]: pgmap v379: 129 pgs: 129 active+clean; 154 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 18 MiB/s wr, 316 op/s 2024-04-01T01:34:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:11 smithi081 ceph-mon[35142]: pgmap v379: 129 pgs: 129 active+clean; 154 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 18 MiB/s wr, 316 op/s 2024-04-01T01:34:12.430 INFO:teuthology.orchestra.run.smithi070.stderr:1000000 of 3200000 tuples (31%) done (elapsed 9.27 s, remaining 20.40 s) 2024-04-01T01:34:13.579 INFO:teuthology.orchestra.run.smithi070.stderr:1100000 of 3200000 tuples (34%) done (elapsed 10.42 s, remaining 19.90 s) 2024-04-01T01:34:13.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:13 smithi189 ceph-mon[35753]: pgmap v380: 129 pgs: 129 active+clean; 230 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 30 MiB/s wr, 254 op/s 2024-04-01T01:34:14.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:13 smithi070 ceph-mon[32179]: pgmap v380: 129 pgs: 129 active+clean; 230 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 30 MiB/s wr, 254 op/s 2024-04-01T01:34:14.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:13 smithi081 ceph-mon[35142]: pgmap v380: 129 pgs: 129 active+clean; 230 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 30 MiB/s wr, 254 op/s 2024-04-01T01:34:14.356 INFO:teuthology.orchestra.run.smithi070.stderr:1200000 of 3200000 tuples (37%) done (elapsed 11.20 s, remaining 18.67 s) 2024-04-01T01:34:14.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:14 smithi189 ceph-mon[35753]: pgmap v381: 129 pgs: 129 active+clean; 262 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 35 MiB/s wr, 94 op/s 2024-04-01T01:34:15.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:14 smithi070 ceph-mon[32179]: pgmap v381: 129 pgs: 129 active+clean; 262 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 35 MiB/s wr, 94 op/s 2024-04-01T01:34:15.051 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:34:15.051 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:34:15.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:14 smithi081 ceph-mon[35142]: pgmap v381: 129 pgs: 129 active+clean; 262 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 35 MiB/s wr, 94 op/s 2024-04-01T01:34:15.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.168+0000 7f006d01c640 1 -- 172.21.15.70:0/2993314635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0068101840 msgr2=0x7f0068101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.168+0000 7f006d01c640 1 --2- 172.21.15.70:0/2993314635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0068101840 0x7f0068101cc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f005c009900 tx=0x7f005c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:34:15.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.168+0000 7f006d01c640 1 -- 172.21.15.70:0/2993314635 shutdown_connections 2024-04-01T01:34:15.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.168+0000 7f006d01c640 1 --2- 172.21.15.70:0/2993314635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0068102200 0x7f0068108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.168+0000 7f006d01c640 1 --2- 172.21.15.70:0/2993314635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0068101840 0x7f0068101cc0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.168+0000 7f006d01c640 1 --2- 172.21.15.70:0/2993314635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0068100650 0x7f0068100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.168+0000 7f006d01c640 1 -- 172.21.15.70:0/2993314635 >> 172.21.15.70:0/2993314635 conn(0x7f00680fbbe0 msgr2=0x7f00680fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.168+0000 7f006d01c640 1 -- 172.21.15.70:0/2993314635 shutdown_connections 2024-04-01T01:34:15.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.168+0000 7f006d01c640 1 -- 172.21.15.70:0/2993314635 wait complete. 2024-04-01T01:34:15.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.169+0000 7f006d01c640 1 Processor -- start 2024-04-01T01:34:15.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.169+0000 7f006d01c640 1 -- start start 2024-04-01T01:34:15.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.169+0000 7f006d01c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0068100650 0x7f00681979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.169+0000 7f006d01c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0068101840 0x7f0068197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.169+0000 7f006d01c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0068102200 0x7f006819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.169+0000 7f0066d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0068100650 0x7f00681979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.169+0000 7f006759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0068102200 0x7f006819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.170+0000 7f0066d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0068100650 0x7f00681979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:53710/0 (socket says 172.21.15.70:53710) 2024-04-01T01:34:15.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.170+0000 7f006759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0068102200 0x7f006819efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56716/0 (socket says 172.21.15.70:56716) 2024-04-01T01:34:15.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.170+0000 7f0066d9d640 1 -- 172.21.15.70:0/2026080640 learned_addr learned my addr 172.21.15.70:0/2026080640 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:15.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.170+0000 7f006d01c640 1 -- 172.21.15.70:0/2026080640 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f006810c170 con 0x7f0068102200 2024-04-01T01:34:15.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.170+0000 7f006d01c640 1 -- 172.21.15.70:0/2026080640 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f006810bff0 con 0x7f0068101840 2024-04-01T01:34:15.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.170+0000 7f006d01c640 1 -- 172.21.15.70:0/2026080640 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f006810c2f0 con 0x7f0068100650 2024-04-01T01:34:15.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.170+0000 7f006659c640 1 --2- 172.21.15.70:0/2026080640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0068101840 0x7f0068197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.170+0000 7f0066d9d640 1 -- 172.21.15.70:0/2026080640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0068101840 msgr2=0x7f0068197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.170+0000 7f0066d9d640 1 --2- 172.21.15.70:0/2026080640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0068101840 0x7f0068197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.170+0000 7f0066d9d640 1 -- 172.21.15.70:0/2026080640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0068102200 msgr2=0x7f006819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.170+0000 7f0066d9d640 1 --2- 172.21.15.70:0/2026080640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0068102200 0x7f006819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.170+0000 7f0066d9d640 1 -- 172.21.15.70:0/2026080640 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0068102b90 con 0x7f0068100650 2024-04-01T01:34:15.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.170+0000 7f006659c640 1 --2- 172.21.15.70:0/2026080640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0068101840 0x7f0068197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:15.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.170+0000 7f006759e640 1 --2- 172.21.15.70:0/2026080640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0068102200 0x7f006819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:15.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.170+0000 7f0066d9d640 1 --2- 172.21.15.70:0/2026080640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0068100650 0x7f00681979f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f005800e9c0 tx=0x7f005800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.170+0000 7f004ffff640 1 -- 172.21.15.70:0/2026080640 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005800cd80 con 0x7f0068100650 2024-04-01T01:34:15.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.171+0000 7f004ffff640 1 -- 172.21.15.70:0/2026080640 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00580035f0 con 0x7f0068100650 2024-04-01T01:34:15.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.171+0000 7f004ffff640 1 -- 172.21.15.70:0/2026080640 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0058010690 con 0x7f0068100650 2024-04-01T01:34:15.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.171+0000 7f006d01c640 1 -- 172.21.15.70:0/2026080640 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0068102e80 con 0x7f0068100650 2024-04-01T01:34:15.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.171+0000 7f006d01c640 1 -- 172.21.15.70:0/2026080640 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00681033a0 con 0x7f0068100650 2024-04-01T01:34:15.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.173+0000 7f004ffff640 1 -- 172.21.15.70:0/2026080640 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f00580037a0 con 0x7f0068100650 2024-04-01T01:34:15.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.173+0000 7f006fc03740 1 Processor -- start 2024-04-01T01:34:15.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006fc03740 1 -- start start 2024-04-01T01:34:15.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006fc03740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab2312b0 0x561dab2316d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006fc03740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab231ca0 0x561dab2367a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006fc03740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab2392a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006fc03740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561dab196180 con 0x561dab236e10 2024-04-01T01:34:15.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006fc03740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561dab13eb70 con 0x561dab2312b0 2024-04-01T01:34:15.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006fc03740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561dab192220 con 0x561dab231ca0 2024-04-01T01:34:15.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006759e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab2312b0 0x561dab2316d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006759e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab2312b0 0x561dab2316d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38314/0 (socket says 172.21.15.70:38314) 2024-04-01T01:34:15.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f0066d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab231ca0 0x561dab2367a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006759e640 1 -- 172.21.15.70:0/3810942899 learned_addr learned my addr 172.21.15.70:0/3810942899 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:15.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006659c640 1 --2- 172.21.15.70:0/3810942899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab2392a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f004ffff640 1 --2- 172.21.15.70:0/2026080640 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00500a2530 0x7f00500a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006759e640 1 --2- 172.21.15.70:0/2026080640 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00500a2530 0x7f00500a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006759e640 1 -- 172.21.15.70:0/3810942899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab231ca0 msgr2=0x561dab2367a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f004ffff640 1 -- 172.21.15.70:0/2026080640 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f00580cf000 con 0x7f0068100650 2024-04-01T01:34:15.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006759e640 1 --2- 172.21.15.70:0/3810942899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab231ca0 0x561dab2367a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006759e640 1 -- 172.21.15.70:0/3810942899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 msgr2=0x561dab2392a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006759e640 1 --2- 172.21.15.70:0/3810942899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab2392a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006759e640 1 -- 172.21.15.70:0/3810942899 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dab1527c0 con 0x561dab2312b0 2024-04-01T01:34:15.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.174+0000 7f006659c640 1 --2- 172.21.15.70:0/3810942899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab2392a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:15.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.175+0000 7f006759e640 1 --2- 172.21.15.70:0/3810942899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab2312b0 0x561dab2316d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0060012e90 tx=0x7f006000e780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29e4efa1c2cd9774 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.175+0000 7f006759e640 1 --2- 172.21.15.70:0/2026080640 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00500a2530 0x7f00500a49f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f006001c000 tx=0x7f006001e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.175+0000 7f004d7fa640 1 -- 172.21.15.70:0/3810942899 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060023070 con 0x561dab2312b0 2024-04-01T01:34:15.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.175+0000 7f004d7fa640 1 -- 172.21.15.70:0/3810942899 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0060003e20 con 0x561dab2312b0 2024-04-01T01:34:15.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.175+0000 7f004d7fa640 1 -- 172.21.15.70:0/3810942899 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006001e470 con 0x561dab2312b0 2024-04-01T01:34:15.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 -- 172.21.15.70:0/3810942899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab2312b0 msgr2=0x561dab2316d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 --2- 172.21.15.70:0/3810942899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab2312b0 0x561dab2316d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0060012e90 tx=0x7f006000e780 comp rx=0 tx=0).stop 2024-04-01T01:34:15.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 -- 172.21.15.70:0/3810942899 shutdown_connections 2024-04-01T01:34:15.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 --2- 172.21.15.70:0/3810942899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab2392a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 --2- 172.21.15.70:0/3810942899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab231ca0 0x561dab2367a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 --2- 172.21.15.70:0/3810942899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab2312b0 0x561dab2316d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 -- 172.21.15.70:0/3810942899 >> 172.21.15.70:0/3810942899 conn(0x561dab22c820 msgr2=0x561dab22cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 -- 172.21.15.70:0/3810942899 shutdown_connections 2024-04-01T01:34:15.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 -- 172.21.15.70:0/3810942899 wait complete. 2024-04-01T01:34:15.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 -- start start 2024-04-01T01:34:15.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 Processor -- start 2024-04-01T01:34:15.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab2312b0 0x561dab23b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab231ca0 0x561dab23bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab242d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561dab239d60 con 0x561dab236e10 2024-04-01T01:34:15.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561dab1344a0 con 0x561dab2312b0 2024-04-01T01:34:15.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561dab239ee0 con 0x561dab231ca0 2024-04-01T01:34:15.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006fc03740 10 client.0 _ll_register_callbacks cb 0x561dab137d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:15.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab242d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f0066d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab2312b0 0x561dab23b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab242d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56730/0 (socket says 172.21.15.70:56730) 2024-04-01T01:34:15.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006659c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab231ca0 0x561dab23bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.176+0000 7f006759e640 1 -- 172.21.15.70:0/3193940541 learned_addr learned my addr 172.21.15.70:0/3193940541 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:15.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.177+0000 7f0066d9d640 1 -- 172.21.15.70:0/3193940541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab231ca0 msgr2=0x561dab23bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.177+0000 7f0066d9d640 1 --2- 172.21.15.70:0/3193940541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab231ca0 0x561dab23bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.177+0000 7f0066d9d640 1 -- 172.21.15.70:0/3193940541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 msgr2=0x561dab242d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.177+0000 7f0066d9d640 1 --2- 172.21.15.70:0/3193940541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab242d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.177+0000 7f0066d9d640 1 -- 172.21.15.70:0/3193940541 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561daaf747d0 con 0x561dab2312b0 2024-04-01T01:34:15.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.177+0000 7f006759e640 1 --2- 172.21.15.70:0/3193940541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab242d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:15.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.177+0000 7f0066d9d640 1 --2- 172.21.15.70:0/3193940541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab2312b0 0x561dab23b7a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f00580cc5e0 tx=0x7f00580971d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ae6319bb1ee9f58 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.177+0000 7f004aff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:34:15.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.177+0000 7f004aff5640 1 -- 172.21.15.70:0/3193940541 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005809aef0 con 0x561dab2312b0 2024-04-01T01:34:15.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.177+0000 7f004aff5640 1 -- 172.21.15.70:0/3193940541 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f00580a3b10 con 0x561dab2312b0 2024-04-01T01:34:15.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.177+0000 7f006fc03740 1 -- 172.21.15.70:0/3193940541 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0060003e20 con 0x561dab2312b0 2024-04-01T01:34:15.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.177+0000 7f004aff5640 1 -- 172.21.15.70:0/3193940541 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f00580d1310 con 0x561dab2312b0 2024-04-01T01:34:15.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.177+0000 7f006fc03740 10 client.25556 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:15.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.177+0000 7f006fc03740 1 -- 172.21.15.70:0/3193940541 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561daaeaa3f0 con 0x561dab2312b0 2024-04-01T01:34:15.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.177+0000 7f004aff5640 1 -- 172.21.15.70:0/3193940541 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0060003e20 con 0x561dab2312b0 2024-04-01T01:34:15.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.178+0000 7f004aff5640 1 -- 172.21.15.70:0/3193940541 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f00580b4140 con 0x561dab2312b0 2024-04-01T01:34:15.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.178+0000 7f006fc03740 10 client.25556 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:15.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.178+0000 7f006fc03740 10 client.25556 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:15.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.178+0000 7f006fc03740 20 client.25556 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:15.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.178+0000 7f006fc03740 1 --2- 172.21.15.70:0/3193940541 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561dab247520 0x561dab249970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.178+0000 7f006fc03740 4 client.25556 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:15.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.178+0000 7f006fc03740 1 -- 172.21.15.70:0/3193940541 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561dab1a36c0 con 0x561dab247520 2024-04-01T01:34:15.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.178+0000 7f006659c640 1 --2- 172.21.15.70:0/3193940541 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561dab247520 0x561dab249970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.178+0000 7f004aff5640 10 client.25556 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:15.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.178+0000 7f006659c640 1 --2- 172.21.15.70:0/3193940541 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561dab247520 0x561dab249970 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dec32b094373c23c server_cookie=2c98203abc77f173 in_seq=0 out_seq=0 2024-04-01T01:34:15.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.179+0000 7f004aff5640 1 -- 172.21.15.70:0/3193940541 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561dab1a36c0 con 0x561dab247520 2024-04-01T01:34:15.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.179+0000 7f004aff5640 10 client.25556 handle_command_reply: tid=0 2024-04-01T01:34:15.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.179+0000 7f006fc03740 1 client.25556 shutdown 2024-04-01T01:34:15.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.179+0000 7f006fc03740 1 -- 172.21.15.70:0/3193940541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab2312b0 msgr2=0x561dab23b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.179+0000 7f006fc03740 1 --2- 172.21.15.70:0/3193940541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab2312b0 0x561dab23b7a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f00580cc5e0 tx=0x7f00580971d0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.179+0000 7f006fc03740 1 -- 172.21.15.70:0/3193940541 shutdown_connections 2024-04-01T01:34:15.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.179+0000 7f006fc03740 1 --2- 172.21.15.70:0/3193940541 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561dab247520 0x561dab249970 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.179+0000 7f006fc03740 1 --2- 172.21.15.70:0/3193940541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab242d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.179+0000 7f004aff5640 0 client.25556 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:15.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.179+0000 7f006fc03740 1 --2- 172.21.15.70:0/3193940541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab231ca0 0x561dab23bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.179+0000 7f006fc03740 1 --2- 172.21.15.70:0/3193940541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab2312b0 0x561dab23b7a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.179+0000 7f006fc03740 1 -- 172.21.15.70:0/3193940541 >> 172.21.15.70:0/3193940541 conn(0x561dab22c820 msgr2=0x561dab22bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.179+0000 7f006fc03740 1 -- 172.21.15.70:0/3193940541 shutdown_connections 2024-04-01T01:34:15.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.179+0000 7f006fc03740 1 -- 172.21.15.70:0/3193940541 wait complete. 2024-04-01T01:34:15.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.179+0000 7f006fc03740 20 client.25556 trim_cache size 0 max 16384 2024-04-01T01:34:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.203+0000 7f006fc03740 1 Processor -- start 2024-04-01T01:34:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.203+0000 7f006fc03740 1 -- start start 2024-04-01T01:34:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f006fc03740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab2312b0 0x561dab23bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f006fc03740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab231ca0 0x561dab242ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f006fc03740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab22b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f006fc03740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f00681ac4f0 con 0x561dab236e10 2024-04-01T01:34:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f006fc03740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f00681ae030 con 0x561dab231ca0 2024-04-01T01:34:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f006fc03740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f00681acbc0 con 0x561dab2312b0 2024-04-01T01:34:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f006759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab22b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f006659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab231ca0 0x561dab242ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f0066d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab2312b0 0x561dab23bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f006759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab22b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56734/0 (socket says 172.21.15.70:56734) 2024-04-01T01:34:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f006759e640 1 -- 172.21.15.70:0/2668087790 learned_addr learned my addr 172.21.15.70:0/2668087790 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f0066d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab2312b0 0x561dab23bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:53734/0 (socket says 172.21.15.70:53734) 2024-04-01T01:34:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f006659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab231ca0 0x561dab242ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38342/0 (socket says 172.21.15.70:38342) 2024-04-01T01:34:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f0066d9d640 1 -- 172.21.15.70:0/2668087790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab231ca0 msgr2=0x561dab242ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f0066d9d640 1 --2- 172.21.15.70:0/2668087790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab231ca0 0x561dab242ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f0066d9d640 1 -- 172.21.15.70:0/2668087790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 msgr2=0x561dab22b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f0066d9d640 1 --2- 172.21.15.70:0/2668087790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab22b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f0066d9d640 1 -- 172.21.15.70:0/2668087790 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561daae6a0f0 con 0x561dab2312b0 2024-04-01T01:34:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f006759e640 1 --2- 172.21.15.70:0/2668087790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab22b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f006659c640 1 --2- 172.21.15.70:0/2668087790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab231ca0 0x561dab242ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f0066d9d640 1 --2- 172.21.15.70:0/2668087790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab2312b0 0x561dab23bac0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0058097f10 tx=0x7f00580cca40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=133363cb58ec50ac server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.204+0000 7f004cff9640 1 -- 172.21.15.70:0/2668087790 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005809aef0 con 0x561dab2312b0 2024-04-01T01:34:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.205+0000 7f004cff9640 1 -- 172.21.15.70:0/2668087790 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0058003b30 con 0x561dab2312b0 2024-04-01T01:34:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.205+0000 7f004cff9640 1 -- 172.21.15.70:0/2668087790 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0058097830 con 0x561dab2312b0 2024-04-01T01:34:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.205+0000 7f006fc03740 1 -- 172.21.15.70:0/2668087790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab2312b0 msgr2=0x561dab23bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.205+0000 7f006fc03740 1 --2- 172.21.15.70:0/2668087790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab2312b0 0x561dab23bac0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0058097f10 tx=0x7f00580cca40 comp rx=0 tx=0).stop 2024-04-01T01:34:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.205+0000 7f006fc03740 1 -- 172.21.15.70:0/2668087790 shutdown_connections 2024-04-01T01:34:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.205+0000 7f006fc03740 1 --2- 172.21.15.70:0/2668087790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab236e10 0x561dab22b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.205+0000 7f006fc03740 1 --2- 172.21.15.70:0/2668087790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab231ca0 0x561dab242ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.205+0000 7f006fc03740 1 --2- 172.21.15.70:0/2668087790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab2312b0 0x561dab23bac0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.205+0000 7f006fc03740 1 -- 172.21.15.70:0/2668087790 >> 172.21.15.70:0/2668087790 conn(0x561dab22c820 msgr2=0x561dab23b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.205+0000 7f006fc03740 1 -- 172.21.15.70:0/2668087790 shutdown_connections 2024-04-01T01:34:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.205+0000 7f006fc03740 1 -- 172.21.15.70:0/2668087790 wait complete. 2024-04-01T01:34:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f006fc03740 1 -- start start 2024-04-01T01:34:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f006fc03740 1 Processor -- start 2024-04-01T01:34:15.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f006fc03740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab2312b0 0x561dab240540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f006fc03740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab231ca0 0x561dab240a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f006fc03740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab236e10 0x561dab241a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f006fc03740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f00681ad960 con 0x561dab2312b0 2024-04-01T01:34:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f006fc03740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f00681ad290 con 0x561dab236e10 2024-04-01T01:34:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f006fc03740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f00681aa4d0 con 0x561dab231ca0 2024-04-01T01:34:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f006fc03740 10 client.0 _ll_register_callbacks cb 0x7f0058003d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f0066d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab2312b0 0x561dab240540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f0066d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab2312b0 0x561dab240540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56738/0 (socket says 172.21.15.70:56738) 2024-04-01T01:34:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f0066d9d640 1 -- 172.21.15.70:0/3420294147 learned_addr learned my addr 172.21.15.70:0/3420294147 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f006759e640 1 --2- 172.21.15.70:0/3420294147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab236e10 0x561dab241a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f006659c640 1 --2- 172.21.15.70:0/3420294147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab231ca0 0x561dab240a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f0066d9d640 1 -- 172.21.15.70:0/3420294147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab231ca0 msgr2=0x561dab240a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f0066d9d640 1 --2- 172.21.15.70:0/3420294147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab231ca0 0x561dab240a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f0066d9d640 1 -- 172.21.15.70:0/3420294147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab236e10 msgr2=0x561dab241a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f0066d9d640 1 --2- 172.21.15.70:0/3420294147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab236e10 0x561dab241a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f0066d9d640 1 -- 172.21.15.70:0/3420294147 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dab242010 con 0x561dab2312b0 2024-04-01T01:34:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f0066d9d640 1 --2- 172.21.15.70:0/3420294147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab2312b0 0x561dab240540 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f00580a3a70 tx=0x7f00580a3dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e1df46ba8bd62c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.206+0000 7f004aff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.207+0000 7f004aff5640 1 -- 172.21.15.70:0/3420294147 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0058097830 con 0x561dab2312b0 2024-04-01T01:34:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.207+0000 7f004aff5640 1 -- 172.21.15.70:0/3420294147 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f005809abf0 con 0x561dab2312b0 2024-04-01T01:34:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.207+0000 7f006fc03740 1 -- 172.21.15.70:0/3420294147 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561dab19da10 con 0x561dab2312b0 2024-04-01T01:34:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.207+0000 7f004aff5640 1 -- 172.21.15.70:0/3420294147 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f005809f730 con 0x561dab2312b0 2024-04-01T01:34:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.207+0000 7f006fc03740 10 client.15747 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.207+0000 7f006fc03740 1 -- 172.21.15.70:0/3420294147 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561dab2ad990 con 0x561dab2312b0 2024-04-01T01:34:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.207+0000 7f004aff5640 1 -- 172.21.15.70:0/3420294147 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x561dab19da10 con 0x561dab2312b0 2024-04-01T01:34:15.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.207+0000 7f004aff5640 1 -- 172.21.15.70:0/3420294147 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f00580972e0 con 0x561dab2312b0 2024-04-01T01:34:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.207+0000 7f006fc03740 10 client.15747 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.207+0000 7f006fc03740 10 client.15747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.207+0000 7f006fc03740 20 client.15747 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.207+0000 7f006fc03740 1 --2- 172.21.15.70:0/3420294147 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561dab2ae070 0x561dab2b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.207+0000 7f006fc03740 4 client.15747 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:34:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.207+0000 7f006fc03740 1 -- 172.21.15.70:0/3420294147 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561dab11b540 con 0x561dab2ae070 2024-04-01T01:34:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.207+0000 7f006659c640 1 --2- 172.21.15.70:0/3420294147 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561dab2ae070 0x561dab2b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f006659c640 1 --2- 172.21.15.70:0/3420294147 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561dab2ae070 0x561dab2b0490 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b711c7f9047b01f3 server_cookie=bed5e24c9ca3a233 in_seq=0 out_seq=0 2024-04-01T01:34:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f004aff5640 10 client.15747 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f004aff5640 1 -- 172.21.15.70:0/3420294147 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561dab11b540 con 0x561dab2ae070 2024-04-01T01:34:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f004aff5640 10 client.15747 handle_command_reply: tid=0 2024-04-01T01:34:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f006fc03740 1 client.15747 shutdown 2024-04-01T01:34:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f006fc03740 1 -- 172.21.15.70:0/3420294147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab2312b0 msgr2=0x561dab240540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f006fc03740 1 --2- 172.21.15.70:0/3420294147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab2312b0 0x561dab240540 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f00580a3a70 tx=0x7f00580a3dc0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f006fc03740 1 -- 172.21.15.70:0/3420294147 shutdown_connections 2024-04-01T01:34:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f006fc03740 1 --2- 172.21.15.70:0/3420294147 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561dab2ae070 0x561dab2b0490 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f006fc03740 1 --2- 172.21.15.70:0/3420294147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561dab236e10 0x561dab241a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f006fc03740 1 --2- 172.21.15.70:0/3420294147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561dab231ca0 0x561dab240a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f004aff5640 0 client.15747 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f006fc03740 1 --2- 172.21.15.70:0/3420294147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561dab2312b0 0x561dab240540 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f006fc03740 1 -- 172.21.15.70:0/3420294147 >> 172.21.15.70:0/3420294147 conn(0x561dab22c820 msgr2=0x561dab22b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f006fc03740 1 -- 172.21.15.70:0/3420294147 shutdown_connections 2024-04-01T01:34:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f006fc03740 1 -- 172.21.15.70:0/3420294147 wait complete. 2024-04-01T01:34:15.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.208+0000 7f006fc03740 20 client.15747 trim_cache size 0 max 16384 2024-04-01T01:34:15.215 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:34:15.216 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:34:15.216 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:34:15.216 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:34:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.209+0000 7f006d01c640 1 -- 172.21.15.70:0/2026080640 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00500a2530 msgr2=0x7f00500a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.209+0000 7f006d01c640 1 --2- 172.21.15.70:0/2026080640 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00500a2530 0x7f00500a49f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f006001c000 tx=0x7f006001e000 comp rx=0 tx=0).stop 2024-04-01T01:34:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.209+0000 7f006d01c640 1 -- 172.21.15.70:0/2026080640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0068100650 msgr2=0x7f00681979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.209+0000 7f006d01c640 1 --2- 172.21.15.70:0/2026080640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0068100650 0x7f00681979f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f005800e9c0 tx=0x7f005800ee80 comp rx=0 tx=0).stop 2024-04-01T01:34:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.209+0000 7f006d01c640 1 -- 172.21.15.70:0/2026080640 shutdown_connections 2024-04-01T01:34:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.209+0000 7f006d01c640 1 --2- 172.21.15.70:0/2026080640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0068102200 0x7f006819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.209+0000 7f006d01c640 1 --2- 172.21.15.70:0/2026080640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0068101840 0x7f0068197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.209+0000 7f006d01c640 1 --2- 172.21.15.70:0/2026080640 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00500a2530 0x7f00500a49f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.209+0000 7f006d01c640 1 --2- 172.21.15.70:0/2026080640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0068100650 0x7f00681979f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.209+0000 7f006d01c640 1 -- 172.21.15.70:0/2026080640 >> 172.21.15.70:0/2026080640 conn(0x7f00680fbbe0 msgr2=0x7f00680fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.209+0000 7f006d01c640 1 -- 172.21.15.70:0/2026080640 shutdown_connections 2024-04-01T01:34:15.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.209+0000 7f006d01c640 1 -- 172.21.15.70:0/2026080640 wait complete. 2024-04-01T01:34:15.221 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:34:15.221 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:34:15.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.340+0000 7f067c581640 1 -- 172.21.15.70:0/3782210498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674100670 msgr2=0x7f0674100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.340+0000 7f067c581640 1 --2- 172.21.15.70:0/3782210498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674100670 0x7f0674100a90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0670009900 tx=0x7f067002bc30 comp rx=0 tx=0).stop 2024-04-01T01:34:15.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.340+0000 7f067c581640 1 -- 172.21.15.70:0/3782210498 shutdown_connections 2024-04-01T01:34:15.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.340+0000 7f067c581640 1 --2- 172.21.15.70:0/3782210498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0674102220 0x7f0674108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.340+0000 7f067c581640 1 --2- 172.21.15.70:0/3782210498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0674101860 0x7f0674101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.340+0000 7f067c581640 1 --2- 172.21.15.70:0/3782210498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674100670 0x7f0674100a90 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.340+0000 7f067c581640 1 -- 172.21.15.70:0/3782210498 >> 172.21.15.70:0/3782210498 conn(0x7f06740fbbe0 msgr2=0x7f06740fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.340+0000 7f067c581640 1 -- 172.21.15.70:0/3782210498 shutdown_connections 2024-04-01T01:34:15.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.340+0000 7f067c581640 1 -- 172.21.15.70:0/3782210498 wait complete. 2024-04-01T01:34:15.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067c581640 1 Processor -- start 2024-04-01T01:34:15.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067c581640 1 -- start start 2024-04-01T01:34:15.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067c581640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0674100670 0x7f06741968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067c581640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0674101860 0x7f0674196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067c581640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102220 0x7f067419dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067c581640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f067410c1a0 con 0x7f0674102220 2024-04-01T01:34:15.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067c581640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f067410c020 con 0x7f0674101860 2024-04-01T01:34:15.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067c581640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f067410c320 con 0x7f0674100670 2024-04-01T01:34:15.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067ab1e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102220 0x7f067419dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067ab1e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102220 0x7f067419dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56752/0 (socket says 172.21.15.70:56752) 2024-04-01T01:34:15.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067ab1e640 1 -- 172.21.15.70:0/1945372160 learned_addr learned my addr 172.21.15.70:0/1945372160 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:15.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f0679b1c640 1 --2- 172.21.15.70:0/1945372160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0674101860 0x7f0674196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067a31d640 1 --2- 172.21.15.70:0/1945372160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0674100670 0x7f06741968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067ab1e640 1 -- 172.21.15.70:0/1945372160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0674100670 msgr2=0x7f06741968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067ab1e640 1 --2- 172.21.15.70:0/1945372160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0674100670 0x7f06741968f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067ab1e640 1 -- 172.21.15.70:0/1945372160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0674101860 msgr2=0x7f0674196e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067ab1e640 1 --2- 172.21.15.70:0/1945372160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0674101860 0x7f0674196e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067ab1e640 1 -- 172.21.15.70:0/1945372160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067419e510 con 0x7f0674102220 2024-04-01T01:34:15.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f0679b1c640 1 --2- 172.21.15.70:0/1945372160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0674101860 0x7f0674196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:15.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.341+0000 7f067ab1e640 1 --2- 172.21.15.70:0/1945372160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102220 0x7f067419dea0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f066c00eb10 tx=0x7f066c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.342+0000 7f066b7fe640 1 -- 172.21.15.70:0/1945372160 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c004b20 con 0x7f0674102220 2024-04-01T01:34:15.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.342+0000 7f066b7fe640 1 -- 172.21.15.70:0/1945372160 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f066c004cc0 con 0x7f0674102220 2024-04-01T01:34:15.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.342+0000 7f067c581640 1 -- 172.21.15.70:0/1945372160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f067419e800 con 0x7f0674102220 2024-04-01T01:34:15.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.342+0000 7f066b7fe640 1 -- 172.21.15.70:0/1945372160 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c010690 con 0x7f0674102220 2024-04-01T01:34:15.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.342+0000 7f067c581640 1 -- 172.21.15.70:0/1945372160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f067419ed50 con 0x7f0674102220 2024-04-01T01:34:15.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.343+0000 7f067c581640 1 -- 172.21.15.70:0/1945372160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06741074c0 con 0x7f0674102220 2024-04-01T01:34:15.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.344+0000 7f066b7fe640 1 -- 172.21.15.70:0/1945372160 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f066c005020 con 0x7f0674102220 2024-04-01T01:34:15.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.344+0000 7f066b7fe640 1 --2- 172.21.15.70:0/1945372160 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06600a2530 0x7f06600a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.345+0000 7f067a31d640 1 --2- 172.21.15.70:0/1945372160 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06600a2530 0x7f06600a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.345+0000 7f066b7fe640 1 -- 172.21.15.70:0/1945372160 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f066c0cf370 con 0x7f0674102220 2024-04-01T01:34:15.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.345+0000 7f067a31d640 1 --2- 172.21.15.70:0/1945372160 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06600a2530 0x7f06600a49f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0670005ab0 tx=0x7f0670005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.349+0000 7f066b7fe640 1 -- 172.21.15.70:0/1945372160 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f066c09c000 con 0x7f0674102220 2024-04-01T01:34:15.412 INFO:teuthology.orchestra.run.smithi070.stderr:1300000 of 3200000 tuples (40%) done (elapsed 12.26 s, remaining 17.91 s) 2024-04-01T01:34:15.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.620+0000 7f067c581640 1 -- 172.21.15.70:0/1945372160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0674107690 con 0x7f0674102220 2024-04-01T01:34:15.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.621+0000 7f066b7fe640 1 -- 172.21.15.70:0/1945372160 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f066c09b900 con 0x7f0674102220 2024-04-01T01:34:15.622 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:15.622 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:33:36.894838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24418,24404,24379,24451,24464]},"id":1}]} 2024-04-01T01:34:15.622 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-04-01T01:34:15.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.623+0000 7f067c581640 1 -- 172.21.15.70:0/1945372160 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06600a2530 msgr2=0x7f06600a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.623+0000 7f067c581640 1 --2- 172.21.15.70:0/1945372160 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06600a2530 0x7f06600a49f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f0670005ab0 tx=0x7f0670005fd0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.623+0000 7f067c581640 1 -- 172.21.15.70:0/1945372160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102220 msgr2=0x7f067419dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.623+0000 7f067c581640 1 --2- 172.21.15.70:0/1945372160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102220 0x7f067419dea0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f066c00eb10 tx=0x7f066c00c2d0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.623+0000 7f067c581640 1 -- 172.21.15.70:0/1945372160 shutdown_connections 2024-04-01T01:34:15.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.623+0000 7f067c581640 1 --2- 172.21.15.70:0/1945372160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0674102220 0x7f067419dea0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.624+0000 7f067c581640 1 --2- 172.21.15.70:0/1945372160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0674101860 0x7f0674196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.624+0000 7f067c581640 1 --2- 172.21.15.70:0/1945372160 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06600a2530 0x7f06600a49f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.624+0000 7f067c581640 1 --2- 172.21.15.70:0/1945372160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0674100670 0x7f06741968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.624+0000 7f067c581640 1 -- 172.21.15.70:0/1945372160 >> 172.21.15.70:0/1945372160 conn(0x7f06740fbbe0 msgr2=0x7f0674104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.624+0000 7f067c581640 1 -- 172.21.15.70:0/1945372160 shutdown_connections 2024-04-01T01:34:15.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.624+0000 7f067c581640 1 -- 172.21.15.70:0/1945372160 wait complete. 2024-04-01T01:34:15.635 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:34:15.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.758+0000 7f3cb1fa7640 1 -- 172.21.15.70:0/3584671482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3cac104ba0 msgr2=0x7f3cac106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.758+0000 7f3cb1fa7640 1 --2- 172.21.15.70:0/3584671482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3cac104ba0 0x7f3cac106f90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0009900 tx=0x7f3ca002bc30 comp rx=0 tx=0).stop 2024-04-01T01:34:15.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.758+0000 7f3cb1fa7640 1 -- 172.21.15.70:0/3584671482 shutdown_connections 2024-04-01T01:34:15.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.758+0000 7f3cb1fa7640 1 --2- 172.21.15.70:0/3584671482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cac1074d0 0x7f3cac1098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.758+0000 7f3cb1fa7640 1 --2- 172.21.15.70:0/3584671482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3cac104ba0 0x7f3cac106f90 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.758+0000 7f3cb1fa7640 1 --2- 172.21.15.70:0/3584671482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cac102270 0x7f3cac104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.758+0000 7f3cb1fa7640 1 -- 172.21.15.70:0/3584671482 >> 172.21.15.70:0/3584671482 conn(0x7f3cac0fbbe0 msgr2=0x7f3cac0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.758+0000 7f3cb1fa7640 1 -- 172.21.15.70:0/3584671482 shutdown_connections 2024-04-01T01:34:15.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.758+0000 7f3cb1fa7640 1 -- 172.21.15.70:0/3584671482 wait complete. 2024-04-01T01:34:15.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.758+0000 7f3cb1fa7640 1 Processor -- start 2024-04-01T01:34:15.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3cb1fa7640 1 -- start start 2024-04-01T01:34:15.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3cb1fa7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cac102270 0x7f3cac101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3cb1fa7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3cac104ba0 0x7f3cac100220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3cb1fa7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cac1074d0 0x7f3cac100760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3cb1fa7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3cac10d2a0 con 0x7f3cac104ba0 2024-04-01T01:34:15.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3cb1fa7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3cac10d120 con 0x7f3cac102270 2024-04-01T01:34:15.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3cb1fa7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3cac10d420 con 0x7f3cac1074d0 2024-04-01T01:34:15.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3caaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3cac104ba0 0x7f3cac100220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3cab7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cac102270 0x7f3cac101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3caaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3cac104ba0 0x7f3cac100220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56778/0 (socket says 172.21.15.70:56778) 2024-04-01T01:34:15.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3caaffd640 1 -- 172.21.15.70:0/3979590616 learned_addr learned my addr 172.21.15.70:0/3979590616 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:15.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3cabfff640 1 --2- 172.21.15.70:0/3979590616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cac1074d0 0x7f3cac100760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3caaffd640 1 -- 172.21.15.70:0/3979590616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cac1074d0 msgr2=0x7f3cac100760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3caaffd640 1 --2- 172.21.15.70:0/3979590616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cac1074d0 0x7f3cac100760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3caaffd640 1 -- 172.21.15.70:0/3979590616 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cac102270 msgr2=0x7f3cac101be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3caaffd640 1 --2- 172.21.15.70:0/3979590616 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cac102270 0x7f3cac101be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3caaffd640 1 -- 172.21.15.70:0/3979590616 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cac100ff0 con 0x7f3cac104ba0 2024-04-01T01:34:15.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.759+0000 7f3caaffd640 1 --2- 172.21.15.70:0/3979590616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3cac104ba0 0x7f3cac100220 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0000c00 tx=0x7f3ca0005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.760+0000 7f3ca8ff9640 1 -- 172.21.15.70:0/3979590616 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0043020 con 0x7f3cac104ba0 2024-04-01T01:34:15.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.760+0000 7f3ca8ff9640 1 -- 172.21.15.70:0/3979590616 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ca00043d0 con 0x7f3cac104ba0 2024-04-01T01:34:15.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.760+0000 7f3cb1fa7640 1 -- 172.21.15.70:0/3979590616 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cac1a1af0 con 0x7f3cac104ba0 2024-04-01T01:34:15.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.760+0000 7f3ca8ff9640 1 -- 172.21.15.70:0/3979590616 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0003b40 con 0x7f3cac104ba0 2024-04-01T01:34:15.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.760+0000 7f3cb1fa7640 1 -- 172.21.15.70:0/3979590616 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cac1a2040 con 0x7f3cac104ba0 2024-04-01T01:34:15.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.762+0000 7f3cb4b81740 1 Processor -- start 2024-04-01T01:34:15.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.762+0000 7f3cb4b81740 1 -- start start 2024-04-01T01:34:15.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.762+0000 7f3cb4b81740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6490 0x5609f8ca68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.762+0000 7f3cb4b81740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8ca6e80 0x5609f8cab980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.762+0000 7f3cb4b81740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8cabff0 0x5609f8cae480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.762+0000 7f3cb4b81740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5609f8c0b1b0 con 0x5609f8ca6e80 2024-04-01T01:34:15.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.762+0000 7f3cb4b81740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5609f8bb3c00 con 0x5609f8ca6490 2024-04-01T01:34:15.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.762+0000 7f3cb4b81740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5609f8c07250 con 0x5609f8cabff0 2024-04-01T01:34:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.762+0000 7f3cabfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6490 0x5609f8ca68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.762+0000 7f3cab7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8ca6e80 0x5609f8cab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.762+0000 7f3cabfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6490 0x5609f8ca68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38398/0 (socket says 172.21.15.70:38398) 2024-04-01T01:34:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.762+0000 7f3cabfff640 1 -- 172.21.15.70:0/4223877265 learned_addr learned my addr 172.21.15.70:0/4223877265 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.762+0000 7f3ca8ff9640 1 -- 172.21.15.70:0/3979590616 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f3ca0046020 con 0x7f3cac104ba0 2024-04-01T01:34:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.763+0000 7f3cabfff640 1 -- 172.21.15.70:0/4223877265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8cabff0 msgr2=0x5609f8cae480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.763+0000 7f3cabfff640 1 --2- 172.21.15.70:0/4223877265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8cabff0 0x5609f8cae480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.763+0000 7f3cabfff640 1 -- 172.21.15.70:0/4223877265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8ca6e80 msgr2=0x5609f8cab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.763+0000 7f3cabfff640 1 --2- 172.21.15.70:0/4223877265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8ca6e80 0x5609f8cab980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.763+0000 7f3cabfff640 1 -- 172.21.15.70:0/4223877265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609f8bd33d0 con 0x5609f8ca6490 2024-04-01T01:34:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.763+0000 7f3cabfff640 1 --2- 172.21.15.70:0/4223877265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6490 0x5609f8ca68b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4009fd0 tx=0x7f3ca40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d87a0c7907acf484 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.763+0000 7f3c91ffb640 1 -- 172.21.15.70:0/4223877265 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4004d20 con 0x5609f8ca6490 2024-04-01T01:34:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.763+0000 7f3c91ffb640 1 -- 172.21.15.70:0/4223877265 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ca4004050 con 0x5609f8ca6490 2024-04-01T01:34:15.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.763+0000 7f3c91ffb640 1 -- 172.21.15.70:0/4223877265 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4005280 con 0x5609f8ca6490 2024-04-01T01:34:15.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.763+0000 7f3ca8ff9640 1 --2- 172.21.15.70:0/3979590616 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3c940a2530 0x7f3c940a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.763+0000 7f3cab7fe640 1 --2- 172.21.15.70:0/3979590616 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3c940a2530 0x7f3c940a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.763+0000 7f3ca8ff9640 1 -- 172.21.15.70:0/3979590616 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3ca00f0ca0 con 0x7f3cac104ba0 2024-04-01T01:34:15.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.764+0000 7f3cb4b81740 1 -- 172.21.15.70:0/4223877265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6490 msgr2=0x5609f8ca68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.764+0000 7f3cab7fe640 1 --2- 172.21.15.70:0/3979590616 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3c940a2530 0x7f3c940a49f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x5609f8ca7280 tx=0x7f3c9c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.764+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/4223877265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6490 0x5609f8ca68b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4009fd0 tx=0x7f3ca40118d0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.764+0000 7f3cb4b81740 1 -- 172.21.15.70:0/4223877265 shutdown_connections 2024-04-01T01:34:15.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.764+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/4223877265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8cabff0 0x5609f8cae480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.764+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/4223877265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8ca6e80 0x5609f8cab980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.764+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/4223877265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6490 0x5609f8ca68b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.764+0000 7f3cb4b81740 1 -- 172.21.15.70:0/4223877265 >> 172.21.15.70:0/4223877265 conn(0x5609f8ca1a00 msgr2=0x5609f8ca1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.764+0000 7f3cb4b81740 1 -- 172.21.15.70:0/4223877265 shutdown_connections 2024-04-01T01:34:15.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.764+0000 7f3cb4b81740 1 -- 172.21.15.70:0/4223877265 wait complete. 2024-04-01T01:34:15.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.764+0000 7f3cb4b81740 1 -- start start 2024-04-01T01:34:15.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3cb4b81740 1 Processor -- start 2024-04-01T01:34:15.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3cb4b81740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8ca6490 0x5609f8cb0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3cb4b81740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 0x5609f8cb0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3cb4b81740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8cabff0 0x5609f8cb7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3cb4b81740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5609f8caefa0 con 0x5609f8cabff0 2024-04-01T01:34:15.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3cb4b81740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5609f8ba9470 con 0x5609f8ca6e80 2024-04-01T01:34:15.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3cb4b81740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5609f8caf120 con 0x5609f8ca6490 2024-04-01T01:34:15.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3cb4b81740 10 client.0 _ll_register_callbacks cb 0x5609f8bc6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:15.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3cabfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8cabff0 0x5609f8cb7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3cab7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8ca6490 0x5609f8cb0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3caaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 0x5609f8cb0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3cabfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8cabff0 0x5609f8cb7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56786/0 (socket says 172.21.15.70:56786) 2024-04-01T01:34:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3cab7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8ca6490 0x5609f8cb0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:53812/0 (socket says 172.21.15.70:53812) 2024-04-01T01:34:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3cabfff640 1 -- 172.21.15.70:0/2963486868 learned_addr learned my addr 172.21.15.70:0/2963486868 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3caaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 0x5609f8cb0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38404/0 (socket says 172.21.15.70:38404) 2024-04-01T01:34:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3caaffd640 1 -- 172.21.15.70:0/2963486868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8ca6490 msgr2=0x5609f8cb0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3caaffd640 1 --2- 172.21.15.70:0/2963486868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8ca6490 0x5609f8cb0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3caaffd640 1 -- 172.21.15.70:0/2963486868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8cabff0 msgr2=0x5609f8cb7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.765+0000 7f3caaffd640 1 --2- 172.21.15.70:0/2963486868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8cabff0 0x5609f8cb7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3caaffd640 1 -- 172.21.15.70:0/2963486868 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609f891f3f0 con 0x5609f8ca6e80 2024-04-01T01:34:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3caaffd640 1 --2- 172.21.15.70:0/2963486868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 0x5609f8cb0fa0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3ca00ee390 tx=0x7f3ca00bce80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=250e67d5154cabe1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3c8affd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:34:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3c8affd640 1 -- 172.21.15.70:0/2963486868 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca00bd340 con 0x5609f8ca6e80 2024-04-01T01:34:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2963486868 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ca4004050 con 0x5609f8ca6e80 2024-04-01T01:34:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3c8affd640 1 -- 172.21.15.70:0/2963486868 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3ca00c1120 con 0x5609f8ca6e80 2024-04-01T01:34:15.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3c8affd640 1 -- 172.21.15.70:0/2963486868 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3ca00d60c0 con 0x5609f8ca6e80 2024-04-01T01:34:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3cb4b81740 10 client.25580 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2963486868 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5609f89e97d0 con 0x5609f8ca6e80 2024-04-01T01:34:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3c8affd640 1 -- 172.21.15.70:0/2963486868 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ca4004050 con 0x5609f8ca6e80 2024-04-01T01:34:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3c8affd640 1 -- 172.21.15.70:0/2963486868 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3ca00c58c0 con 0x5609f8ca6e80 2024-04-01T01:34:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3cb4b81740 10 client.25580 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3cb4b81740 10 client.25580 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:34:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3cb4b81740 20 client.25580 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/2963486868 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5609f8cbc570 0x5609f8cbe960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3cb4b81740 4 client.25580 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.766+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2963486868 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609f8c186f0 con 0x5609f8cbc570 2024-04-01T01:34:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.767+0000 7f3cab7fe640 1 --2- 172.21.15.70:0/2963486868 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5609f8cbc570 0x5609f8cbe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.767+0000 7f3c8affd640 10 client.25580 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:34:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.767+0000 7f3cab7fe640 1 --2- 172.21.15.70:0/2963486868 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5609f8cbc570 0x5609f8cbe960 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5932a4a332e4a3db server_cookie=414c1282cd776acc in_seq=0 out_seq=0 2024-04-01T01:34:15.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.768+0000 7f3c8affd640 1 -- 172.21.15.70:0/2963486868 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5609f8c186f0 con 0x5609f8cbc570 2024-04-01T01:34:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.768+0000 7f3c8affd640 10 client.25580 handle_command_reply: tid=0 2024-04-01T01:34:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.768+0000 7f3cb4b81740 1 client.25580 shutdown 2024-04-01T01:34:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.768+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2963486868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 msgr2=0x5609f8cb0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.768+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/2963486868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 0x5609f8cb0fa0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3ca00ee390 tx=0x7f3ca00bce80 comp rx=0 tx=0).stop 2024-04-01T01:34:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.768+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2963486868 shutdown_connections 2024-04-01T01:34:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.768+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/2963486868 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5609f8cbc570 0x5609f8cbe960 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.768+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/2963486868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8cabff0 0x5609f8cb7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.768+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/2963486868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 0x5609f8cb0fa0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.768+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/2963486868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8ca6490 0x5609f8cb0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.768+0000 7f3c8affd640 0 client.25580 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:34:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.768+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2963486868 >> 172.21.15.70:0/2963486868 conn(0x5609f8ca1a00 msgr2=0x5609f8ca0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.768+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2963486868 shutdown_connections 2024-04-01T01:34:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.768+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2963486868 wait complete. 2024-04-01T01:34:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.768+0000 7f3cb4b81740 20 client.25580 trim_cache size 0 max 16384 2024-04-01T01:34:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.781+0000 7f3cb4b81740 1 Processor -- start 2024-04-01T01:34:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cb4b81740 1 -- start start 2024-04-01T01:34:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cb4b81740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8ca6490 0x5609f8ca8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cb4b81740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 0x5609f8cb7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cb4b81740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8cabff0 0x5609f8ca8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cb4b81740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3cac1ac580 con 0x5609f8cabff0 2024-04-01T01:34:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cb4b81740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3cac1ae0c0 con 0x5609f8ca6e80 2024-04-01T01:34:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cb4b81740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3cac1acc50 con 0x5609f8ca6490 2024-04-01T01:34:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cabfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8cabff0 0x5609f8ca8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cab7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8ca6490 0x5609f8ca8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3caaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 0x5609f8cb7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cabfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8cabff0 0x5609f8ca8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56796/0 (socket says 172.21.15.70:56796) 2024-04-01T01:34:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cabfff640 1 -- 172.21.15.70:0/2490730315 learned_addr learned my addr 172.21.15.70:0/2490730315 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cabfff640 1 -- 172.21.15.70:0/2490730315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8ca6490 msgr2=0x5609f8ca8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cabfff640 1 --2- 172.21.15.70:0/2490730315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8ca6490 0x5609f8ca8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cabfff640 1 -- 172.21.15.70:0/2490730315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 msgr2=0x5609f8cb7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cabfff640 1 --2- 172.21.15.70:0/2490730315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 0x5609f8cb7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cabfff640 1 -- 172.21.15.70:0/2490730315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609f8cab6e0 con 0x5609f8cabff0 2024-04-01T01:34:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3caaffd640 1 --2- 172.21.15.70:0/2490730315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 0x5609f8cb7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.782+0000 7f3cabfff640 1 --2- 172.21.15.70:0/2490730315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8cabff0 0x5609f8ca8950 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4005c10 tx=0x7f3ca4009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83bb2aa0daecc68f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.783+0000 7f3c917fa640 1 -- 172.21.15.70:0/2490730315 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4027070 con 0x5609f8cabff0 2024-04-01T01:34:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.783+0000 7f3c917fa640 1 -- 172.21.15.70:0/2490730315 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ca4003cf0 con 0x5609f8cabff0 2024-04-01T01:34:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.783+0000 7f3c917fa640 1 -- 172.21.15.70:0/2490730315 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca400f840 con 0x5609f8cabff0 2024-04-01T01:34:15.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.783+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2490730315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8cabff0 msgr2=0x5609f8ca8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.783+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/2490730315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8cabff0 0x5609f8ca8950 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4005c10 tx=0x7f3ca4009d70 comp rx=0 tx=0).stop 2024-04-01T01:34:15.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.783+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2490730315 shutdown_connections 2024-04-01T01:34:15.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.783+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/2490730315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8cabff0 0x5609f8ca8950 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.783+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/2490730315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 0x5609f8cb7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.783+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/2490730315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8ca6490 0x5609f8ca8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.783+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2490730315 >> 172.21.15.70:0/2490730315 conn(0x5609f8ca1a00 msgr2=0x5609f8ca1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.784+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2490730315 shutdown_connections 2024-04-01T01:34:15.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.784+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2490730315 wait complete. 2024-04-01T01:34:15.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.784+0000 7f3cb4b81740 1 -- start start 2024-04-01T01:34:15.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.784+0000 7f3cb4b81740 1 Processor -- start 2024-04-01T01:34:15.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.784+0000 7f3cb4b81740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8ca6490 0x5609f8cb5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.784+0000 7f3cb4b81740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 0x5609f8cb61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.784+0000 7f3cb4b81740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8cabff0 0x5609f8cb71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.784+0000 7f3cb4b81740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3cac1ad9f0 con 0x5609f8ca6490 2024-04-01T01:34:15.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.784+0000 7f3cb4b81740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3cac1ad320 con 0x5609f8ca6e80 2024-04-01T01:34:15.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.784+0000 7f3cb4b81740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3cac1aa560 con 0x5609f8cabff0 2024-04-01T01:34:15.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.784+0000 7f3cb4b81740 10 client.0 _ll_register_callbacks cb 0x7f3ca00b95e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:15.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.784+0000 7f3cab7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8ca6490 0x5609f8cb5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.784+0000 7f3cab7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8ca6490 0x5609f8cb5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56802/0 (socket says 172.21.15.70:56802) 2024-04-01T01:34:15.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.784+0000 7f3cab7fe640 1 -- 172.21.15.70:0/2408765897 learned_addr learned my addr 172.21.15.70:0/2408765897 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:15.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.784+0000 7f3caaffd640 1 --2- 172.21.15.70:0/2408765897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 0x5609f8cb61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.784+0000 7f3cabfff640 1 --2- 172.21.15.70:0/2408765897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8cabff0 0x5609f8cb71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.785+0000 7f3cab7fe640 1 -- 172.21.15.70:0/2408765897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8cabff0 msgr2=0x5609f8cb71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.785+0000 7f3cab7fe640 1 --2- 172.21.15.70:0/2408765897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8cabff0 0x5609f8cb71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.785+0000 7f3cab7fe640 1 -- 172.21.15.70:0/2408765897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 msgr2=0x5609f8cb61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.785+0000 7f3cab7fe640 1 --2- 172.21.15.70:0/2408765897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 0x5609f8cb61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.785+0000 7f3cab7fe640 1 -- 172.21.15.70:0/2408765897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609f8cb7790 con 0x5609f8ca6490 2024-04-01T01:34:15.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.785+0000 7f3cab7fe640 1 --2- 172.21.15.70:0/2408765897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8ca6490 0x5609f8cb5ca0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c013010 tx=0x7f3c9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7883f9905edcd922 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.785+0000 7f3c8affd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:15.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.785+0000 7f3c8affd640 1 -- 172.21.15.70:0/2408765897 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9c02bd10 con 0x5609f8ca6490 2024-04-01T01:34:15.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.785+0000 7f3c8affd640 1 -- 172.21.15.70:0/2408765897 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3c9c003b50 con 0x5609f8ca6490 2024-04-01T01:34:15.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.785+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2408765897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ca4003cf0 con 0x5609f8ca6490 2024-04-01T01:34:15.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.785+0000 7f3c8affd640 1 -- 172.21.15.70:0/2408765897 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3c9c03b050 con 0x5609f8ca6490 2024-04-01T01:34:15.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.785+0000 7f3cb4b81740 10 client.15771 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:15.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.785+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2408765897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5609f8d22df0 con 0x5609f8ca6490 2024-04-01T01:34:15.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.785+0000 7f3c8affd640 1 -- 172.21.15.70:0/2408765897 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3ca4003cf0 con 0x5609f8ca6490 2024-04-01T01:34:15.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.786+0000 7f3c8affd640 1 -- 172.21.15.70:0/2408765897 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3ca0003ce0 con 0x5609f8ca6490 2024-04-01T01:34:15.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.786+0000 7f3cb4b81740 10 client.15771 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:15.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.786+0000 7f3cb4b81740 10 client.15771 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:34:15.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.786+0000 7f3cb4b81740 20 client.15771 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:15.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.786+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/2408765897 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5609f8d23820 0x5609f8d25c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.786+0000 7f3cb4b81740 4 client.15771 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:34:15.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.786+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2408765897 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5609f8c12a40 con 0x5609f8d23820 2024-04-01T01:34:15.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.786+0000 7f3caaffd640 1 --2- 172.21.15.70:0/2408765897 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5609f8d23820 0x5609f8d25c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.786+0000 7f3c8affd640 10 client.15771 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:34:15.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.786+0000 7f3caaffd640 1 --2- 172.21.15.70:0/2408765897 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5609f8d23820 0x5609f8d25c40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a5238bc86cd0b80a server_cookie=13faf922b4094344 in_seq=0 out_seq=0 2024-04-01T01:34:15.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.786+0000 7f3c8affd640 1 -- 172.21.15.70:0/2408765897 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5609f8c12a40 con 0x5609f8d23820 2024-04-01T01:34:15.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.786+0000 7f3c8affd640 10 client.15771 handle_command_reply: tid=0 2024-04-01T01:34:15.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.786+0000 7f3cb4b81740 1 client.15771 shutdown 2024-04-01T01:34:15.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2408765897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8ca6490 msgr2=0x5609f8cb5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/2408765897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8ca6490 0x5609f8cb5ca0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c013010 tx=0x7f3c9c0046c0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2408765897 shutdown_connections 2024-04-01T01:34:15.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/2408765897 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5609f8d23820 0x5609f8d25c40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/2408765897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5609f8cabff0 0x5609f8cb71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/2408765897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5609f8ca6e80 0x5609f8cb61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3c8affd640 0 client.15771 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:34:15.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb4b81740 1 --2- 172.21.15.70:0/2408765897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5609f8ca6490 0x5609f8cb5ca0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2408765897 >> 172.21.15.70:0/2408765897 conn(0x5609f8ca1a00 msgr2=0x5609f8cb0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2408765897 shutdown_connections 2024-04-01T01:34:15.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb4b81740 1 -- 172.21.15.70:0/2408765897 wait complete. 2024-04-01T01:34:15.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb4b81740 20 client.15771 trim_cache size 0 max 16384 2024-04-01T01:34:15.793 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:34:15.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb1fa7640 1 -- 172.21.15.70:0/3979590616 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3c940a2530 msgr2=0x7f3c940a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb1fa7640 1 --2- 172.21.15.70:0/3979590616 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3c940a2530 0x7f3c940a49f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x5609f8ca7280 tx=0x7f3c9c005f90 comp rx=0 tx=0).stop 2024-04-01T01:34:15.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb1fa7640 1 -- 172.21.15.70:0/3979590616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3cac104ba0 msgr2=0x7f3cac100220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb1fa7640 1 --2- 172.21.15.70:0/3979590616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3cac104ba0 0x7f3cac100220 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0000c00 tx=0x7f3ca0005e60 comp rx=0 tx=0).stop 2024-04-01T01:34:15.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb1fa7640 1 -- 172.21.15.70:0/3979590616 shutdown_connections 2024-04-01T01:34:15.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb1fa7640 1 --2- 172.21.15.70:0/3979590616 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3c940a2530 0x7f3c940a49f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb1fa7640 1 --2- 172.21.15.70:0/3979590616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cac1074d0 0x7f3cac100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb1fa7640 1 --2- 172.21.15.70:0/3979590616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3cac104ba0 0x7f3cac100220 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb1fa7640 1 --2- 172.21.15.70:0/3979590616 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cac102270 0x7f3cac101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.787+0000 7f3cb1fa7640 1 -- 172.21.15.70:0/3979590616 >> 172.21.15.70:0/3979590616 conn(0x7f3cac0fbbe0 msgr2=0x7f3cac102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.788+0000 7f3cb1fa7640 1 -- 172.21.15.70:0/3979590616 shutdown_connections 2024-04-01T01:34:15.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.788+0000 7f3cb1fa7640 1 -- 172.21.15.70:0/3979590616 wait complete. 2024-04-01T01:34:15.800 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:34:15.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.915+0000 7f7715d02640 1 -- 172.21.15.70:0/2197910169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7710102200 msgr2=0x7f7710108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.915+0000 7f7715d02640 1 --2- 172.21.15.70:0/2197910169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7710102200 0x7f7710108a90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7708009ca0 tx=0x7f770802c0f0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.915+0000 7f7715d02640 1 -- 172.21.15.70:0/2197910169 shutdown_connections 2024-04-01T01:34:15.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.915+0000 7f7715d02640 1 --2- 172.21.15.70:0/2197910169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7710102200 0x7f7710108a90 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.915+0000 7f7715d02640 1 --2- 172.21.15.70:0/2197910169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7710101840 0x7f7710101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.915+0000 7f7715d02640 1 --2- 172.21.15.70:0/2197910169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7710100650 0x7f7710100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.915+0000 7f7715d02640 1 -- 172.21.15.70:0/2197910169 >> 172.21.15.70:0/2197910169 conn(0x7f77100fbbe0 msgr2=0x7f77100fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.915+0000 7f7715d02640 1 -- 172.21.15.70:0/2197910169 shutdown_connections 2024-04-01T01:34:15.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.915+0000 7f7715d02640 1 -- 172.21.15.70:0/2197910169 wait complete. 2024-04-01T01:34:15.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.915+0000 7f7715d02640 1 Processor -- start 2024-04-01T01:34:15.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.915+0000 7f7715d02640 1 -- start start 2024-04-01T01:34:15.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f7715d02640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7710100650 0x7f7710197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f7715d02640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7710101840 0x7f7710197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f7715d02640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7710102200 0x7f771019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f7715d02640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f771010c180 con 0x7f7710102200 2024-04-01T01:34:15.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f7715d02640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f771010c000 con 0x7f7710101840 2024-04-01T01:34:15.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f7715d02640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f771010c300 con 0x7f7710100650 2024-04-01T01:34:15.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f770f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7710100650 0x7f7710197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f770ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7710102200 0x7f771019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f770f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7710100650 0x7f7710197980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:53848/0 (socket says 172.21.15.70:53848) 2024-04-01T01:34:15.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f770f7fe640 1 -- 172.21.15.70:0/244271288 learned_addr learned my addr 172.21.15.70:0/244271288 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:15.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f770effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7710101840 0x7f7710197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f770ffff640 1 -- 172.21.15.70:0/244271288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7710100650 msgr2=0x7f7710197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f770ffff640 1 --2- 172.21.15.70:0/244271288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7710100650 0x7f7710197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f770ffff640 1 -- 172.21.15.70:0/244271288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7710101840 msgr2=0x7f7710197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f770ffff640 1 --2- 172.21.15.70:0/244271288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7710101840 0x7f7710197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f770ffff640 1 -- 172.21.15.70:0/244271288 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f771019f5a0 con 0x7f7710102200 2024-04-01T01:34:15.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f770f7fe640 1 --2- 172.21.15.70:0/244271288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7710100650 0x7f7710197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:15.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f770effd640 1 --2- 172.21.15.70:0/244271288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7710101840 0x7f7710197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:15.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.916+0000 7f770ffff640 1 --2- 172.21.15.70:0/244271288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7710102200 0x7f771019ef30 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f770802c630 tx=0x7f77080328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.917+0000 7f770cff9640 1 -- 172.21.15.70:0/244271288 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7708041020 con 0x7f7710102200 2024-04-01T01:34:15.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.917+0000 7f770cff9640 1 -- 172.21.15.70:0/244271288 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7708003e20 con 0x7f7710102200 2024-04-01T01:34:15.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.917+0000 7f7715d02640 1 -- 172.21.15.70:0/244271288 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f771019f830 con 0x7f7710102200 2024-04-01T01:34:15.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.917+0000 7f770cff9640 1 -- 172.21.15.70:0/244271288 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7708039040 con 0x7f7710102200 2024-04-01T01:34:15.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.917+0000 7f7715d02640 1 -- 172.21.15.70:0/244271288 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f771019fd20 con 0x7f7710102200 2024-04-01T01:34:15.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.919+0000 7f770cff9640 1 -- 172.21.15.70:0/244271288 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f7708044020 con 0x7f7710102200 2024-04-01T01:34:15.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.919+0000 7f77189c3740 1 Processor -- start 2024-04-01T01:34:15.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.919+0000 7f77189c3740 1 -- start start 2024-04-01T01:34:15.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.919+0000 7f770cff9640 1 --2- 172.21.15.70:0/244271288 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f76f80a2530 0x7f76f80a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.919+0000 7f77189c3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981490 0x56396d9818b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f77189c3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d981e80 0x56396d986980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f77189c3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 0x56396d989480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f77189c3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56396d8e61b0 con 0x56396d981490 2024-04-01T01:34:15.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f77189c3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56396d88ec00 con 0x56396d986ff0 2024-04-01T01:34:15.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f77189c3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56396d8e2250 con 0x56396d981e80 2024-04-01T01:34:15.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f770cff9640 1 -- 172.21.15.70:0/244271288 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f77080f0cb0 con 0x7f7710102200 2024-04-01T01:34:15.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f770effd640 1 --2- 172.21.15.70:0/244271288 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f76f80a2530 0x7f76f80a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f770f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981490 0x56396d9818b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f770ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 0x56396d989480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f770effd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d981e80 0x56396d986980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f770f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981490 0x56396d9818b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56824/0 (socket says 172.21.15.70:56824) 2024-04-01T01:34:15.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f770f7fe640 1 -- 172.21.15.70:0/1219674855 learned_addr learned my addr 172.21.15.70:0/1219674855 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:15.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f770ffff640 1 -- 172.21.15.70:0/1219674855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d981e80 msgr2=0x56396d986980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f770ffff640 1 --2- 172.21.15.70:0/1219674855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d981e80 0x56396d986980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f770ffff640 1 -- 172.21.15.70:0/1219674855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981490 msgr2=0x56396d9818b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f770ffff640 1 --2- 172.21.15.70:0/1219674855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981490 0x56396d9818b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f770ffff640 1 -- 172.21.15.70:0/1219674855 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56396d8ae3d0 con 0x56396d986ff0 2024-04-01T01:34:15.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f770f7fe640 1 --2- 172.21.15.70:0/1219674855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981490 0x56396d9818b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:34:15.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f770effd640 1 --2- 172.21.15.70:0/244271288 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f76f80a2530 0x7f76f80a49f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7704009fd0 tx=0x7f7704009ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.920+0000 7f770ffff640 1 --2- 172.21.15.70:0/1219674855 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 0x56396d989480 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f77080ee3a0 tx=0x7f770802c0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcc0cb0937a462a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.921+0000 7f76f5ffb640 1 -- 172.21.15.70:0/1219674855 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77080bd350 con 0x56396d986ff0 2024-04-01T01:34:15.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.921+0000 7f76f5ffb640 1 -- 172.21.15.70:0/1219674855 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77080c14a0 con 0x56396d986ff0 2024-04-01T01:34:15.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.921+0000 7f76f5ffb640 1 -- 172.21.15.70:0/1219674855 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77080b99c0 con 0x56396d986ff0 2024-04-01T01:34:15.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.921+0000 7f77189c3740 1 -- 172.21.15.70:0/1219674855 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 msgr2=0x56396d989480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.921+0000 7f77189c3740 1 --2- 172.21.15.70:0/1219674855 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 0x56396d989480 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f77080ee3a0 tx=0x7f770802c0f0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.921+0000 7f77189c3740 1 -- 172.21.15.70:0/1219674855 shutdown_connections 2024-04-01T01:34:15.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.921+0000 7f77189c3740 1 --2- 172.21.15.70:0/1219674855 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 0x56396d989480 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.921+0000 7f77189c3740 1 --2- 172.21.15.70:0/1219674855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d981e80 0x56396d986980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.921+0000 7f77189c3740 1 --2- 172.21.15.70:0/1219674855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981490 0x56396d9818b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.921+0000 7f77189c3740 1 -- 172.21.15.70:0/1219674855 >> 172.21.15.70:0/1219674855 conn(0x56396d97ca00 msgr2=0x56396d97ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.921+0000 7f77189c3740 1 -- 172.21.15.70:0/1219674855 shutdown_connections 2024-04-01T01:34:15.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.922+0000 7f77189c3740 1 -- 172.21.15.70:0/1219674855 wait complete. 2024-04-01T01:34:15.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.922+0000 7f77189c3740 1 -- start start 2024-04-01T01:34:15.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.922+0000 7f77189c3740 1 Processor -- start 2024-04-01T01:34:15.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.922+0000 7f77189c3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d981490 0x56396d98b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.922+0000 7f77189c3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981e80 0x56396d98bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.922+0000 7f77189c3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d986ff0 0x56396d992ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.922+0000 7f77189c3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56396d989d60 con 0x56396d981e80 2024-04-01T01:34:15.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.922+0000 7f77189c3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56396d884470 con 0x56396d981490 2024-04-01T01:34:15.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.922+0000 7f77189c3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56396d989ee0 con 0x56396d986ff0 2024-04-01T01:34:15.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f770effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981e80 0x56396d98bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f77189c3740 10 client.0 _ll_register_callbacks cb 0x56396d8a1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:15.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f770effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981e80 0x56396d98bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56834/0 (socket says 172.21.15.70:56834) 2024-04-01T01:34:15.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f770effd640 1 -- 172.21.15.70:0/1065620435 learned_addr learned my addr 172.21.15.70:0/1065620435 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:15.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f770f7fe640 1 --2- 172.21.15.70:0/1065620435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d981490 0x56396d98b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f770ffff640 1 --2- 172.21.15.70:0/1065620435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d986ff0 0x56396d992ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f770effd640 1 -- 172.21.15.70:0/1065620435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d986ff0 msgr2=0x56396d992ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f770effd640 1 --2- 172.21.15.70:0/1065620435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d986ff0 0x56396d992ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f770effd640 1 -- 172.21.15.70:0/1065620435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d981490 msgr2=0x56396d98b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f770effd640 1 --2- 172.21.15.70:0/1065620435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d981490 0x56396d98b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f770effd640 1 -- 172.21.15.70:0/1065620435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56396d5fa3f0 con 0x56396d981e80 2024-04-01T01:34:15.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f770f7fe640 1 --2- 172.21.15.70:0/1065620435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d981490 0x56396d98b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:34:15.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f770effd640 1 --2- 172.21.15.70:0/1065620435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981e80 0x56396d98bc60 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f770401f670 tx=0x7f770401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719ab0f07f43f3cb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f76eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:15.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f76eeffd640 1 -- 172.21.15.70:0/1065620435 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77040214d0 con 0x56396d981e80 2024-04-01T01:34:15.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f76eeffd640 1 -- 172.21.15.70:0/1065620435 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f770401c040 con 0x56396d981e80 2024-04-01T01:34:15.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.923+0000 7f77189c3740 1 -- 172.21.15.70:0/1065620435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77080c14a0 con 0x56396d981e80 2024-04-01T01:34:15.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.924+0000 7f76eeffd640 1 -- 172.21.15.70:0/1065620435 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7704032050 con 0x56396d981e80 2024-04-01T01:34:15.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.924+0000 7f77189c3740 10 client.15783 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:15.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.924+0000 7f77189c3740 1 -- 172.21.15.70:0/1065620435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56396d6c47d0 con 0x56396d981e80 2024-04-01T01:34:15.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.924+0000 7f76eeffd640 1 -- 172.21.15.70:0/1065620435 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77080c14a0 con 0x56396d981e80 2024-04-01T01:34:15.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.924+0000 7f76eeffd640 1 -- 172.21.15.70:0/1065620435 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7704020050 con 0x56396d981e80 2024-04-01T01:34:15.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.924+0000 7f77189c3740 10 client.15783 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:15.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.924+0000 7f77189c3740 10 client.15783 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:34:15.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.924+0000 7f77189c3740 20 client.15783 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:15.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.924+0000 7f77189c3740 1 --2- 172.21.15.70:0/1065620435 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56396d997160 0x56396d999620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.924+0000 7f77189c3740 4 client.15783 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:15.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.924+0000 7f77189c3740 1 -- 172.21.15.70:0/1065620435 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56396d8f36f0 con 0x56396d997160 2024-04-01T01:34:15.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.924+0000 7f770f7fe640 1 --2- 172.21.15.70:0/1065620435 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56396d997160 0x56396d999620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.925+0000 7f76eeffd640 10 client.15783 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:34:15.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.925+0000 7f770f7fe640 1 --2- 172.21.15.70:0/1065620435 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56396d997160 0x56396d999620 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=22db7dae64c30559 server_cookie=453b2c5ecad07c6d in_seq=0 out_seq=0 2024-04-01T01:34:15.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.925+0000 7f76eeffd640 1 -- 172.21.15.70:0/1065620435 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56396d8f36f0 con 0x56396d997160 2024-04-01T01:34:15.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.925+0000 7f76eeffd640 10 client.15783 handle_command_reply: tid=0 2024-04-01T01:34:15.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.925+0000 7f77189c3740 1 client.15783 shutdown 2024-04-01T01:34:15.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.926+0000 7f77189c3740 1 -- 172.21.15.70:0/1065620435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981e80 msgr2=0x56396d98bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.926+0000 7f77189c3740 1 --2- 172.21.15.70:0/1065620435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981e80 0x56396d98bc60 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f770401f670 tx=0x7f770401d000 comp rx=0 tx=0).stop 2024-04-01T01:34:15.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.926+0000 7f77189c3740 1 -- 172.21.15.70:0/1065620435 shutdown_connections 2024-04-01T01:34:15.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.926+0000 7f77189c3740 1 --2- 172.21.15.70:0/1065620435 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56396d997160 0x56396d999620 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.926+0000 7f77189c3740 1 --2- 172.21.15.70:0/1065620435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d986ff0 0x56396d992ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.926+0000 7f77189c3740 1 --2- 172.21.15.70:0/1065620435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981e80 0x56396d98bc60 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.926+0000 7f77189c3740 1 --2- 172.21.15.70:0/1065620435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d981490 0x56396d98b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.926+0000 7f77189c3740 1 -- 172.21.15.70:0/1065620435 >> 172.21.15.70:0/1065620435 conn(0x56396d97ca00 msgr2=0x56396d97bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.926+0000 7f76eeffd640 0 client.15783 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:34:15.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.926+0000 7f77189c3740 1 -- 172.21.15.70:0/1065620435 shutdown_connections 2024-04-01T01:34:15.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.926+0000 7f77189c3740 1 -- 172.21.15.70:0/1065620435 wait complete. 2024-04-01T01:34:15.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.926+0000 7f77189c3740 20 client.15783 trim_cache size 0 max 16384 2024-04-01T01:34:15.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.943+0000 7f77189c3740 1 Processor -- start 2024-04-01T01:34:15.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.943+0000 7f77189c3740 1 -- start start 2024-04-01T01:34:15.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.943+0000 7f77189c3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d981490 0x56396d97b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.943+0000 7f77189c3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981e80 0x56396d992c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.943+0000 7f77189c3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 0x56396d97b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.943+0000 7f77189c3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f77101aa380 con 0x56396d981e80 2024-04-01T01:34:15.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.943+0000 7f77189c3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f77101abec0 con 0x56396d986ff0 2024-04-01T01:34:15.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.943+0000 7f77189c3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f77101aaa50 con 0x56396d981490 2024-04-01T01:34:15.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.943+0000 7f770effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981e80 0x56396d992c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.943+0000 7f770effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981e80 0x56396d992c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56846/0 (socket says 172.21.15.70:56846) 2024-04-01T01:34:15.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.943+0000 7f770ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 0x56396d97b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.943+0000 7f770effd640 1 -- 172.21.15.70:0/2797993549 learned_addr learned my addr 172.21.15.70:0/2797993549 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:15.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.943+0000 7f770f7fe640 1 --2- 172.21.15.70:0/2797993549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d981490 0x56396d97b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.944+0000 7f770effd640 1 -- 172.21.15.70:0/2797993549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d981490 msgr2=0x56396d97b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.944+0000 7f770effd640 1 --2- 172.21.15.70:0/2797993549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d981490 0x56396d97b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.944+0000 7f770effd640 1 -- 172.21.15.70:0/2797993549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 msgr2=0x56396d97b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.944+0000 7f770effd640 1 --2- 172.21.15.70:0/2797993549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 0x56396d97b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.944+0000 7f770effd640 1 -- 172.21.15.70:0/2797993549 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56396d986670 con 0x56396d981e80 2024-04-01T01:34:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.944+0000 7f770ffff640 1 --2- 172.21.15.70:0/2797993549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 0x56396d97b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.944+0000 7f770f7fe640 1 --2- 172.21.15.70:0/2797993549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d981490 0x56396d97b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.944+0000 7f770effd640 1 --2- 172.21.15.70:0/2797993549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981e80 0x56396d992c60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7704009fa0 tx=0x7f7704005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce1b17f3f11a2c6f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.944+0000 7f76f57fa640 1 -- 172.21.15.70:0/2797993549 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770403b070 con 0x56396d981e80 2024-04-01T01:34:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.944+0000 7f76f57fa640 1 -- 172.21.15.70:0/2797993549 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77040139b0 con 0x56396d981e80 2024-04-01T01:34:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.944+0000 7f76f57fa640 1 -- 172.21.15.70:0/2797993549 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704021070 con 0x56396d981e80 2024-04-01T01:34:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.945+0000 7f77189c3740 1 -- 172.21.15.70:0/2797993549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981e80 msgr2=0x56396d992c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.945+0000 7f77189c3740 1 --2- 172.21.15.70:0/2797993549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981e80 0x56396d992c60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7704009fa0 tx=0x7f7704005b90 comp rx=0 tx=0).stop 2024-04-01T01:34:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.945+0000 7f77189c3740 1 -- 172.21.15.70:0/2797993549 shutdown_connections 2024-04-01T01:34:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.945+0000 7f77189c3740 1 --2- 172.21.15.70:0/2797993549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 0x56396d97b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.945+0000 7f77189c3740 1 --2- 172.21.15.70:0/2797993549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981e80 0x56396d992c60 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.945+0000 7f77189c3740 1 --2- 172.21.15.70:0/2797993549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d981490 0x56396d97b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.945+0000 7f77189c3740 1 -- 172.21.15.70:0/2797993549 >> 172.21.15.70:0/2797993549 conn(0x56396d97ca00 msgr2=0x56396d998f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.945+0000 7f77189c3740 1 -- 172.21.15.70:0/2797993549 shutdown_connections 2024-04-01T01:34:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.945+0000 7f77189c3740 1 -- 172.21.15.70:0/2797993549 wait complete. 2024-04-01T01:34:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.945+0000 7f77189c3740 1 -- start start 2024-04-01T01:34:15.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.945+0000 7f77189c3740 1 Processor -- start 2024-04-01T01:34:15.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f77189c3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981490 0x56396d990430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f77189c3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d981e80 0x56396d990970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f77189c3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 0x56396d991930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f77189c3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f77101ab7f0 con 0x56396d981490 2024-04-01T01:34:15.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f77189c3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f77101ab120 con 0x56396d986ff0 2024-04-01T01:34:15.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f77189c3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f77101a8360 con 0x56396d981e80 2024-04-01T01:34:15.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f77189c3740 10 client.0 _ll_register_callbacks cb 0x7f77040136c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:15.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f770f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981490 0x56396d990430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f770f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981490 0x56396d990430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56860/0 (socket says 172.21.15.70:56860) 2024-04-01T01:34:15.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f770f7fe640 1 -- 172.21.15.70:0/150599947 learned_addr learned my addr 172.21.15.70:0/150599947 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:15.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f770effd640 1 --2- 172.21.15.70:0/150599947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d981e80 0x56396d990970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f770ffff640 1 --2- 172.21.15.70:0/150599947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 0x56396d991930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f770f7fe640 1 -- 172.21.15.70:0/150599947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d981e80 msgr2=0x56396d990970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f770f7fe640 1 --2- 172.21.15.70:0/150599947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d981e80 0x56396d990970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f770f7fe640 1 -- 172.21.15.70:0/150599947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 msgr2=0x56396d991930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:15.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f770f7fe640 1 --2- 172.21.15.70:0/150599947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 0x56396d991930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f770f7fe640 1 -- 172.21.15.70:0/150599947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56396d991f20 con 0x56396d981490 2024-04-01T01:34:15.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f770f7fe640 1 --2- 172.21.15.70:0/150599947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981490 0x56396d990430 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x56396d999e90 tx=0x7f7700010e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a69264bb0ffbbc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:15.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f76eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:15.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f76eeffd640 1 -- 172.21.15.70:0/150599947 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770000fda0 con 0x56396d981490 2024-04-01T01:34:15.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f76eeffd640 1 -- 172.21.15.70:0/150599947 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f7700004030 con 0x56396d981490 2024-04-01T01:34:15.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.946+0000 7f77189c3740 1 -- 172.21.15.70:0/150599947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77040139b0 con 0x56396d981490 2024-04-01T01:34:15.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.947+0000 7f76eeffd640 1 -- 172.21.15.70:0/150599947 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f770002d050 con 0x56396d981490 2024-04-01T01:34:15.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.947+0000 7f77189c3740 10 client.15789 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:15.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.947+0000 7f77189c3740 1 -- 172.21.15.70:0/150599947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56396d9fd650 con 0x56396d981490 2024-04-01T01:34:15.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.947+0000 7f76eeffd640 1 -- 172.21.15.70:0/150599947 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77040139b0 con 0x56396d981490 2024-04-01T01:34:15.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.947+0000 7f76eeffd640 1 -- 172.21.15.70:0/150599947 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7700004830 con 0x56396d981490 2024-04-01T01:34:15.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.947+0000 7f77189c3740 10 client.15789 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:15.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.947+0000 7f77189c3740 10 client.15789 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:34:15.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.947+0000 7f77189c3740 20 client.15789 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:15.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.947+0000 7f77189c3740 1 --2- 172.21.15.70:0/150599947 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56396d9fe050 0x56396da00470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:15.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.947+0000 7f77189c3740 4 client.15789 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:34:15.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.947+0000 7f77189c3740 1 -- 172.21.15.70:0/150599947 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56396d8eda40 con 0x56396d9fe050 2024-04-01T01:34:15.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.947+0000 7f770effd640 1 --2- 172.21.15.70:0/150599947 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56396d9fe050 0x56396da00470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:15.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f76eeffd640 10 client.15789 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:34:15.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f770effd640 1 --2- 172.21.15.70:0/150599947 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56396d9fe050 0x56396da00470 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d08bbb4d4940334f server_cookie=a0af400da0721e6a in_seq=0 out_seq=0 2024-04-01T01:34:15.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f76eeffd640 1 -- 172.21.15.70:0/150599947 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56396d8eda40 con 0x56396d9fe050 2024-04-01T01:34:15.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f76eeffd640 10 client.15789 handle_command_reply: tid=0 2024-04-01T01:34:15.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f77189c3740 1 client.15789 shutdown 2024-04-01T01:34:15.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f77189c3740 1 -- 172.21.15.70:0/150599947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981490 msgr2=0x56396d990430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f77189c3740 1 --2- 172.21.15.70:0/150599947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981490 0x56396d990430 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x56396d999e90 tx=0x7f7700010e80 comp rx=0 tx=0).stop 2024-04-01T01:34:15.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f77189c3740 1 -- 172.21.15.70:0/150599947 shutdown_connections 2024-04-01T01:34:15.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f77189c3740 1 --2- 172.21.15.70:0/150599947 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56396d9fe050 0x56396da00470 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f77189c3740 1 --2- 172.21.15.70:0/150599947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56396d986ff0 0x56396d991930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f77189c3740 1 --2- 172.21.15.70:0/150599947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56396d981e80 0x56396d990970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f76eeffd640 0 client.15789 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:34:15.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f77189c3740 1 --2- 172.21.15.70:0/150599947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56396d981490 0x56396d990430 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f77189c3740 1 -- 172.21.15.70:0/150599947 >> 172.21.15.70:0/150599947 conn(0x56396d97ca00 msgr2=0x56396d97fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f77189c3740 1 -- 172.21.15.70:0/150599947 shutdown_connections 2024-04-01T01:34:15.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f77189c3740 1 -- 172.21.15.70:0/150599947 wait complete. 2024-04-01T01:34:15.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.948+0000 7f77189c3740 20 client.15789 trim_cache size 0 max 16384 2024-04-01T01:34:15.955 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:34:15.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.949+0000 7f7715d02640 1 -- 172.21.15.70:0/244271288 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f76f80a2530 msgr2=0x7f76f80a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.949+0000 7f7715d02640 1 --2- 172.21.15.70:0/244271288 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f76f80a2530 0x7f76f80a49f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7704009fd0 tx=0x7f7704009ec0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.949+0000 7f7715d02640 1 -- 172.21.15.70:0/244271288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7710102200 msgr2=0x7f771019ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:15.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.949+0000 7f7715d02640 1 --2- 172.21.15.70:0/244271288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7710102200 0x7f771019ef30 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f770802c630 tx=0x7f77080328a0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.949+0000 7f7715d02640 1 -- 172.21.15.70:0/244271288 shutdown_connections 2024-04-01T01:34:15.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.949+0000 7f7715d02640 1 --2- 172.21.15.70:0/244271288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7710102200 0x7f771019ef30 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.949+0000 7f7715d02640 1 --2- 172.21.15.70:0/244271288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7710101840 0x7f7710197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.949+0000 7f7715d02640 1 --2- 172.21.15.70:0/244271288 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f76f80a2530 0x7f76f80a49f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.949+0000 7f7715d02640 1 --2- 172.21.15.70:0/244271288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7710100650 0x7f7710197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:15.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.949+0000 7f7715d02640 1 -- 172.21.15.70:0/244271288 >> 172.21.15.70:0/244271288 conn(0x7f77100fbbe0 msgr2=0x7f77100688e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:15.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.949+0000 7f7715d02640 1 -- 172.21.15.70:0/244271288 shutdown_connections 2024-04-01T01:34:15.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:15.949+0000 7f7715d02640 1 -- 172.21.15.70:0/244271288 wait complete. 2024-04-01T01:34:15.959 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:15 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1945372160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:34:15.960 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:34:15.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:15 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1945372160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:34:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.077+0000 7f58f3548640 1 -- 172.21.15.70:0/2970387028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ec100650 msgr2=0x7f58ec100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.077+0000 7f58f3548640 1 --2- 172.21.15.70:0/2970387028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ec100650 0x7f58ec100a70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009900 tx=0x7f58e802bc30 comp rx=0 tx=0).stop 2024-04-01T01:34:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.077+0000 7f58f3548640 1 -- 172.21.15.70:0/2970387028 shutdown_connections 2024-04-01T01:34:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.077+0000 7f58f3548640 1 --2- 172.21.15.70:0/2970387028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ec102200 0x7f58ec108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.077+0000 7f58f3548640 1 --2- 172.21.15.70:0/2970387028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ec101840 0x7f58ec101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.077+0000 7f58f3548640 1 --2- 172.21.15.70:0/2970387028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ec100650 0x7f58ec100a70 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.077+0000 7f58f3548640 1 -- 172.21.15.70:0/2970387028 >> 172.21.15.70:0/2970387028 conn(0x7f58ec0fbbe0 msgr2=0x7f58ec0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.078+0000 7f58f3548640 1 -- 172.21.15.70:0/2970387028 shutdown_connections 2024-04-01T01:34:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.078+0000 7f58f3548640 1 -- 172.21.15.70:0/2970387028 wait complete. 2024-04-01T01:34:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.078+0000 7f58f3548640 1 Processor -- start 2024-04-01T01:34:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.078+0000 7f58f3548640 1 -- start start 2024-04-01T01:34:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.078+0000 7f58f3548640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ec100650 0x7f58ec1957d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.078+0000 7f58f3548640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ec101840 0x7f58ec195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.078+0000 7f58f3548640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ec102200 0x7f58ec19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.078+0000 7f58f3548640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec10b040 con 0x7f58ec100650 2024-04-01T01:34:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.078+0000 7f58f3548640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec10aec0 con 0x7f58ec102200 2024-04-01T01:34:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.078+0000 7f58f3548640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec10b1c0 con 0x7f58ec101840 2024-04-01T01:34:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.078+0000 7f58f12e4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ec100650 0x7f58ec1957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.079+0000 7f58f12e4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ec100650 0x7f58ec1957d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56886/0 (socket says 172.21.15.70:56886) 2024-04-01T01:34:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.079+0000 7f58f12e4640 1 -- 172.21.15.70:0/2247663612 learned_addr learned my addr 172.21.15.70:0/2247663612 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.079+0000 7f58f0ae3640 1 --2- 172.21.15.70:0/2247663612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ec101840 0x7f58ec195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.079+0000 7f58f1ae5640 1 --2- 172.21.15.70:0/2247663612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ec102200 0x7f58ec19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.079+0000 7f58f12e4640 1 -- 172.21.15.70:0/2247663612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ec101840 msgr2=0x7f58ec195d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.079+0000 7f58f12e4640 1 --2- 172.21.15.70:0/2247663612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ec101840 0x7f58ec195d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.079+0000 7f58f12e4640 1 -- 172.21.15.70:0/2247663612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ec102200 msgr2=0x7f58ec19cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.079+0000 7f58f12e4640 1 --2- 172.21.15.70:0/2247663612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ec102200 0x7f58ec19cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.079+0000 7f58f12e4640 1 -- 172.21.15.70:0/2247663612 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58ec19d3f0 con 0x7f58ec100650 2024-04-01T01:34:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.079+0000 7f58f12e4640 1 --2- 172.21.15.70:0/2247663612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ec100650 0x7f58ec1957d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f58e8000c00 tx=0x7f58e8005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.079+0000 7f58e27fc640 1 -- 172.21.15.70:0/2247663612 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e8043020 con 0x7f58ec100650 2024-04-01T01:34:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.079+0000 7f58f3548640 1 -- 172.21.15.70:0/2247663612 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58ec19d680 con 0x7f58ec100650 2024-04-01T01:34:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.079+0000 7f58e27fc640 1 -- 172.21.15.70:0/2247663612 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58e8004350 con 0x7f58ec100650 2024-04-01T01:34:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.079+0000 7f58e27fc640 1 -- 172.21.15.70:0/2247663612 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e8003b80 con 0x7f58ec100650 2024-04-01T01:34:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.079+0000 7f58f3548640 1 -- 172.21.15.70:0/2247663612 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58ec19dc20 con 0x7f58ec100650 2024-04-01T01:34:16.083 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:15 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1945372160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:34:16.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.081+0000 7f58e27fc640 1 -- 172.21.15.70:0/2247663612 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f58e8046020 con 0x7f58ec100650 2024-04-01T01:34:16.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.082+0000 7f58f6208740 1 Processor -- start 2024-04-01T01:34:16.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.082+0000 7f58f6208740 1 -- start start 2024-04-01T01:34:16.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.082+0000 7f58f6208740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df16490 0x56044df168b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.082+0000 7f58f6208740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16e80 0x56044df1b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.082+0000 7f58f6208740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df1bff0 0x56044df1e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.082+0000 7f58f6208740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56044de7b1b0 con 0x56044df16490 2024-04-01T01:34:16.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.082+0000 7f58f6208740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56044de23c00 con 0x56044df16e80 2024-04-01T01:34:16.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.082+0000 7f58f6208740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56044de77250 con 0x56044df1bff0 2024-04-01T01:34:16.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.082+0000 7f58e27fc640 1 --2- 172.21.15.70:0/2247663612 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f58d80a4740 0x7f58d80a6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.082+0000 7f58f0ae3640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df1bff0 0x56044df1e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.082+0000 7f58f12e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16e80 0x56044df1b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.082+0000 7f58f0ae3640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df1bff0 0x56044df1e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:53912/0 (socket says 172.21.15.70:53912) 2024-04-01T01:34:16.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.082+0000 7f58f0ae3640 1 -- 172.21.15.70:0/147545854 learned_addr learned my addr 172.21.15.70:0/147545854 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:16.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.082+0000 7f58f1ae5640 1 --2- 172.21.15.70:0/147545854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df16490 0x56044df168b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.083+0000 7f58f1ae5640 1 --2- 172.21.15.70:0/2247663612 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f58d80a4740 0x7f58d80a6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.083+0000 7f58e27fc640 1 -- 172.21.15.70:0/2247663612 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f58e80f0c50 con 0x7f58ec100650 2024-04-01T01:34:16.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.083+0000 7f58f0ae3640 1 -- 172.21.15.70:0/147545854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16e80 msgr2=0x56044df1b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.083+0000 7f58f0ae3640 1 --2- 172.21.15.70:0/147545854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16e80 0x56044df1b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.083+0000 7f58f0ae3640 1 -- 172.21.15.70:0/147545854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df16490 msgr2=0x56044df168b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.083+0000 7f58f0ae3640 1 --2- 172.21.15.70:0/147545854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df16490 0x56044df168b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.083+0000 7f58f0ae3640 1 -- 172.21.15.70:0/147545854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56044de433d0 con 0x56044df1bff0 2024-04-01T01:34:16.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.083+0000 7f58f1ae5640 1 --2- 172.21.15.70:0/2247663612 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f58d80a4740 0x7f58d80a6c00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x56044df176e0 tx=0x7f58e4009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.083+0000 7f58f0ae3640 1 --2- 172.21.15.70:0/147545854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df1bff0 0x56044df1e480 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f58dc009c90 tx=0x7f58dc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=444ef505a635f0c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.083+0000 7f58d77fe640 1 -- 172.21.15.70:0/147545854 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58dc01e070 con 0x56044df1bff0 2024-04-01T01:34:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.083+0000 7f58d77fe640 1 -- 172.21.15.70:0/147545854 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58dc009e40 con 0x56044df1bff0 2024-04-01T01:34:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.083+0000 7f58d77fe640 1 -- 172.21.15.70:0/147545854 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58dc00e8f0 con 0x56044df1bff0 2024-04-01T01:34:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.084+0000 7f58f6208740 1 -- 172.21.15.70:0/147545854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df1bff0 msgr2=0x56044df1e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.084+0000 7f58f6208740 1 --2- 172.21.15.70:0/147545854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df1bff0 0x56044df1e480 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f58dc009c90 tx=0x7f58dc011810 comp rx=0 tx=0).stop 2024-04-01T01:34:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.084+0000 7f58f6208740 1 -- 172.21.15.70:0/147545854 shutdown_connections 2024-04-01T01:34:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.084+0000 7f58f6208740 1 --2- 172.21.15.70:0/147545854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df1bff0 0x56044df1e480 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.084+0000 7f58f6208740 1 --2- 172.21.15.70:0/147545854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16e80 0x56044df1b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.084+0000 7f58f6208740 1 --2- 172.21.15.70:0/147545854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df16490 0x56044df168b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.084+0000 7f58f6208740 1 -- 172.21.15.70:0/147545854 >> 172.21.15.70:0/147545854 conn(0x56044df11a00 msgr2=0x56044df11e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.084+0000 7f58f6208740 1 -- 172.21.15.70:0/147545854 shutdown_connections 2024-04-01T01:34:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.084+0000 7f58f6208740 1 -- 172.21.15.70:0/147545854 wait complete. 2024-04-01T01:34:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.084+0000 7f58f6208740 1 -- start start 2024-04-01T01:34:16.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.084+0000 7f58f6208740 1 Processor -- start 2024-04-01T01:34:16.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f6208740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df16490 0x56044df20990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f6208740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df16e80 0x56044df20ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f6208740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df1bff0 0x56044df27f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f6208740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56044df1efa0 con 0x56044df16490 2024-04-01T01:34:16.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f6208740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56044de19470 con 0x56044df1bff0 2024-04-01T01:34:16.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f6208740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56044df1f120 con 0x56044df16e80 2024-04-01T01:34:16.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f12e4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df16490 0x56044df20990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f6208740 10 client.0 _ll_register_callbacks cb 0x56044de36cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:16.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f0ae3640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df16e80 0x56044df20ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f12e4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df16490 0x56044df20990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56894/0 (socket says 172.21.15.70:56894) 2024-04-01T01:34:16.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f1ae5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df1bff0 0x56044df27f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f12e4640 1 -- 172.21.15.70:0/1610104724 learned_addr learned my addr 172.21.15.70:0/1610104724 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:16.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f12e4640 1 -- 172.21.15.70:0/1610104724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df16e80 msgr2=0x56044df20ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f12e4640 1 --2- 172.21.15.70:0/1610104724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df16e80 0x56044df20ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f12e4640 1 -- 172.21.15.70:0/1610104724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df1bff0 msgr2=0x56044df27f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f12e4640 1 --2- 172.21.15.70:0/1610104724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df1bff0 0x56044df27f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f12e4640 1 -- 172.21.15.70:0/1610104724 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56044db8f3f0 con 0x56044df16490 2024-04-01T01:34:16.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f0ae3640 1 --2- 172.21.15.70:0/1610104724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df16e80 0x56044df20ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:16.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.085+0000 7f58f1ae5640 1 --2- 172.21.15.70:0/1610104724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df1bff0 0x56044df27f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:16.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.086+0000 7f58f12e4640 1 --2- 172.21.15.70:0/1610104724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df16490 0x56044df20990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f58e80036a0 tx=0x7f58e8088670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccabcd6193c5a51d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.086+0000 7f58d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:16.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.086+0000 7f58d4ff9640 1 -- 172.21.15.70:0/1610104724 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e80b7e60 con 0x56044df16490 2024-04-01T01:34:16.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.086+0000 7f58f6208740 1 -- 172.21.15.70:0/1610104724 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56044de886f0 con 0x56044df16490 2024-04-01T01:34:16.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.086+0000 7f58d4ff9640 1 -- 172.21.15.70:0/1610104724 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f58e80ee480 con 0x56044df16490 2024-04-01T01:34:16.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.086+0000 7f58d4ff9640 1 -- 172.21.15.70:0/1610104724 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f58e80bcc50 con 0x56044df16490 2024-04-01T01:34:16.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.086+0000 7f58f6208740 10 client.15801 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:16.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.086+0000 7f58f6208740 1 -- 172.21.15.70:0/1610104724 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56044dc597d0 con 0x56044df16490 2024-04-01T01:34:16.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.086+0000 7f58d4ff9640 1 -- 172.21.15.70:0/1610104724 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56044de886f0 con 0x56044df16490 2024-04-01T01:34:16.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.087+0000 7f58d4ff9640 1 -- 172.21.15.70:0/1610104724 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f58e8088240 con 0x56044df16490 2024-04-01T01:34:16.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.087+0000 7f58f6208740 10 client.15801 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:16.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.087+0000 7f58f6208740 10 client.15801 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:34:16.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.087+0000 7f58f6208740 20 client.15801 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:16.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.087+0000 7f58f6208740 1 --2- 172.21.15.70:0/1610104724 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x56044df2c3e0 0x56044df2e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.087+0000 7f58f6208740 4 client.15801 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:16.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.087+0000 7f58f6208740 1 -- 172.21.15.70:0/1610104724 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56044de82a40 con 0x56044df2c3e0 2024-04-01T01:34:16.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.087+0000 7f58f0ae3640 1 --2- 172.21.15.70:0/1610104724 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x56044df2c3e0 0x56044df2e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.088+0000 7f58d4ff9640 10 client.15801 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:34:16.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.088+0000 7f58f0ae3640 1 --2- 172.21.15.70:0/1610104724 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x56044df2c3e0 0x56044df2e850 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f979bee2e9e328a7 server_cookie=99efe46fc13fc1d2 in_seq=0 out_seq=0 2024-04-01T01:34:16.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.088+0000 7f58d4ff9640 1 -- 172.21.15.70:0/1610104724 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56044de82a40 con 0x56044df2c3e0 2024-04-01T01:34:16.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.088+0000 7f58d4ff9640 10 client.15801 handle_command_reply: tid=0 2024-04-01T01:34:16.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.088+0000 7f58f6208740 1 client.15801 shutdown 2024-04-01T01:34:16.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.088+0000 7f58f6208740 1 -- 172.21.15.70:0/1610104724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df16490 msgr2=0x56044df20990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.088+0000 7f58f6208740 1 --2- 172.21.15.70:0/1610104724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df16490 0x56044df20990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f58e80036a0 tx=0x7f58e8088670 comp rx=0 tx=0).stop 2024-04-01T01:34:16.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.088+0000 7f58f6208740 1 -- 172.21.15.70:0/1610104724 shutdown_connections 2024-04-01T01:34:16.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.088+0000 7f58f6208740 1 --2- 172.21.15.70:0/1610104724 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x56044df2c3e0 0x56044df2e850 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.088+0000 7f58f6208740 1 --2- 172.21.15.70:0/1610104724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df1bff0 0x56044df27f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.088+0000 7f58f6208740 1 --2- 172.21.15.70:0/1610104724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df16e80 0x56044df20ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.088+0000 7f58d4ff9640 0 client.15801 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:34:16.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.088+0000 7f58f6208740 1 --2- 172.21.15.70:0/1610104724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df16490 0x56044df20990 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.088+0000 7f58f6208740 1 -- 172.21.15.70:0/1610104724 >> 172.21.15.70:0/1610104724 conn(0x56044df11a00 msgr2=0x56044df10d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.089+0000 7f58f6208740 1 -- 172.21.15.70:0/1610104724 shutdown_connections 2024-04-01T01:34:16.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.089+0000 7f58f6208740 1 -- 172.21.15.70:0/1610104724 wait complete. 2024-04-01T01:34:16.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.089+0000 7f58f6208740 20 client.15801 trim_cache size 0 max 16384 2024-04-01T01:34:16.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f6208740 1 Processor -- start 2024-04-01T01:34:16.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f6208740 1 -- start start 2024-04-01T01:34:16.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f6208740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16490 0x56044df205a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f6208740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df16e80 0x56044df27ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f6208740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df1bff0 0x56044df20ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f6208740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec1a8250 con 0x56044df1bff0 2024-04-01T01:34:16.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f6208740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec1a9d90 con 0x56044df16490 2024-04-01T01:34:16.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f6208740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec1a8920 con 0x56044df16e80 2024-04-01T01:34:16.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f1ae5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df1bff0 0x56044df20ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f12e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16490 0x56044df205a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f12e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16490 0x56044df205a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38538/0 (socket says 172.21.15.70:38538) 2024-04-01T01:34:16.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f0ae3640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df16e80 0x56044df27ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f12e4640 1 -- 172.21.15.70:0/359253700 learned_addr learned my addr 172.21.15.70:0/359253700 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:16.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f1ae5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df1bff0 0x56044df20ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56906/0 (socket says 172.21.15.70:56906) 2024-04-01T01:34:16.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f12e4640 1 -- 172.21.15.70:0/359253700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df16e80 msgr2=0x56044df27ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f12e4640 1 --2- 172.21.15.70:0/359253700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df16e80 0x56044df27ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f12e4640 1 -- 172.21.15.70:0/359253700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df1bff0 msgr2=0x56044df20ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f12e4640 1 --2- 172.21.15.70:0/359253700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df1bff0 0x56044df20ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f12e4640 1 -- 172.21.15.70:0/359253700 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56044df12770 con 0x56044df16490 2024-04-01T01:34:16.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.104+0000 7f58f12e4640 1 --2- 172.21.15.70:0/359253700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16490 0x56044df205a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f58e80b9cc0 tx=0x7f58e80dd2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3508f85cebe77fb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.105+0000 7f58d6ffd640 1 -- 172.21.15.70:0/359253700 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e80b7e60 con 0x56044df16490 2024-04-01T01:34:16.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.105+0000 7f58d6ffd640 1 -- 172.21.15.70:0/359253700 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58e80bd2f0 con 0x56044df16490 2024-04-01T01:34:16.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.105+0000 7f58d6ffd640 1 -- 172.21.15.70:0/359253700 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e80c10d0 con 0x56044df16490 2024-04-01T01:34:16.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.106+0000 7f58f6208740 1 -- 172.21.15.70:0/359253700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16490 msgr2=0x56044df205a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.106+0000 7f58f6208740 1 --2- 172.21.15.70:0/359253700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16490 0x56044df205a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f58e80b9cc0 tx=0x7f58e80dd2c0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.106+0000 7f58f6208740 1 -- 172.21.15.70:0/359253700 shutdown_connections 2024-04-01T01:34:16.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.106+0000 7f58f6208740 1 --2- 172.21.15.70:0/359253700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df1bff0 0x56044df20ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.106+0000 7f58f6208740 1 --2- 172.21.15.70:0/359253700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df16e80 0x56044df27ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.106+0000 7f58f6208740 1 --2- 172.21.15.70:0/359253700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16490 0x56044df205a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.106+0000 7f58f6208740 1 -- 172.21.15.70:0/359253700 >> 172.21.15.70:0/359253700 conn(0x56044df11a00 msgr2=0x56044df18410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.106+0000 7f58f6208740 1 -- 172.21.15.70:0/359253700 shutdown_connections 2024-04-01T01:34:16.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.106+0000 7f58f6208740 1 -- 172.21.15.70:0/359253700 wait complete. 2024-04-01T01:34:16.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.106+0000 7f58f6208740 1 -- start start 2024-04-01T01:34:16.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.106+0000 7f58f6208740 1 Processor -- start 2024-04-01T01:34:16.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.106+0000 7f58f6208740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16490 0x56044df24e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f6208740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df16e80 0x56044df253d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f6208740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df1bff0 0x56044df90780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f6208740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec1a96c0 con 0x56044df1bff0 2024-04-01T01:34:16.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f6208740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec1a8ff0 con 0x56044df16490 2024-04-01T01:34:16.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f6208740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec1a6230 con 0x56044df16e80 2024-04-01T01:34:16.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f6208740 10 client.0 _ll_register_callbacks cb 0x7f58e80045d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:16.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f12e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16490 0x56044df24e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f1ae5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df1bff0 0x56044df90780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f12e4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16490 0x56044df24e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38548/0 (socket says 172.21.15.70:38548) 2024-04-01T01:34:16.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f12e4640 1 -- 172.21.15.70:0/439430723 learned_addr learned my addr 172.21.15.70:0/439430723 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:16.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f0ae3640 1 --2- 172.21.15.70:0/439430723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df16e80 0x56044df253d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f12e4640 1 -- 172.21.15.70:0/439430723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df16e80 msgr2=0x56044df253d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f12e4640 1 --2- 172.21.15.70:0/439430723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df16e80 0x56044df253d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f12e4640 1 -- 172.21.15.70:0/439430723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df1bff0 msgr2=0x56044df90780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f12e4640 1 --2- 172.21.15.70:0/439430723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df1bff0 0x56044df90780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f12e4640 1 -- 172.21.15.70:0/439430723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56044df90cf0 con 0x56044df16490 2024-04-01T01:34:16.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f1ae5640 1 --2- 172.21.15.70:0/439430723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df1bff0 0x56044df90780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:16.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f12e4640 1 --2- 172.21.15.70:0/439430723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16490 0x56044df24e90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f58e80bcff0 tx=0x7f58e802c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67c58df916c327aa server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:34:16.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58d4ff9640 1 -- 172.21.15.70:0/439430723 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58e80b7e60 con 0x56044df16490 2024-04-01T01:34:16.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58d4ff9640 1 -- 172.21.15.70:0/439430723 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f58e80c5870 con 0x56044df16490 2024-04-01T01:34:16.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.107+0000 7f58f6208740 1 -- 172.21.15.70:0/439430723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58e80bd2f0 con 0x56044df16490 2024-04-01T01:34:16.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.108+0000 7f58d4ff9640 1 -- 172.21.15.70:0/439430723 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f58e80b9590 con 0x56044df16490 2024-04-01T01:34:16.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.108+0000 7f58f6208740 10 client.25622 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:16.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.108+0000 7f58f6208740 1 -- 172.21.15.70:0/439430723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56044df90cf0 con 0x56044df16490 2024-04-01T01:34:16.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.108+0000 7f58d4ff9640 1 -- 172.21.15.70:0/439430723 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f58e80bd2f0 con 0x56044df16490 2024-04-01T01:34:16.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.108+0000 7f58d4ff9640 1 -- 172.21.15.70:0/439430723 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f58e80b9040 con 0x56044df16490 2024-04-01T01:34:16.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.108+0000 7f58f6208740 10 client.25622 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:16.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.108+0000 7f58f6208740 10 client.25622 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:34:16.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.108+0000 7f58f6208740 20 client.25622 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:16.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.108+0000 7f58f6208740 1 --2- 172.21.15.70:0/439430723 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x56044df924a0 0x56044df94890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.108+0000 7f58f6208740 4 client.25622 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:34:16.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.108+0000 7f58f6208740 1 -- 172.21.15.70:0/439430723 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56044de006a0 con 0x56044df924a0 2024-04-01T01:34:16.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.108+0000 7f58f0ae3640 1 --2- 172.21.15.70:0/439430723 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x56044df924a0 0x56044df94890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58f0ae3640 1 --2- 172.21.15.70:0/439430723 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x56044df924a0 0x56044df94890 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6012ebd9f42a2429 server_cookie=3cb790157bcb684b in_seq=0 out_seq=0 2024-04-01T01:34:16.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58d4ff9640 10 client.25622 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:34:16.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58d4ff9640 1 -- 172.21.15.70:0/439430723 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56044de006a0 con 0x56044df924a0 2024-04-01T01:34:16.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58d4ff9640 10 client.25622 handle_command_reply: tid=0 2024-04-01T01:34:16.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58f6208740 1 client.25622 shutdown 2024-04-01T01:34:16.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58f6208740 1 -- 172.21.15.70:0/439430723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16490 msgr2=0x56044df24e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58f6208740 1 --2- 172.21.15.70:0/439430723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16490 0x56044df24e90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f58e80bcff0 tx=0x7f58e802c800 comp rx=0 tx=0).stop 2024-04-01T01:34:16.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58f6208740 1 -- 172.21.15.70:0/439430723 shutdown_connections 2024-04-01T01:34:16.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58f6208740 1 --2- 172.21.15.70:0/439430723 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x56044df924a0 0x56044df94890 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58f6208740 1 --2- 172.21.15.70:0/439430723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56044df1bff0 0x56044df90780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58f6208740 1 --2- 172.21.15.70:0/439430723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56044df16e80 0x56044df253d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58f6208740 1 --2- 172.21.15.70:0/439430723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56044df16490 0x56044df24e90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58d4ff9640 0 client.25622 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:34:16.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58f6208740 1 -- 172.21.15.70:0/439430723 >> 172.21.15.70:0/439430723 conn(0x56044df11a00 msgr2=0x56044df20550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58f6208740 1 -- 172.21.15.70:0/439430723 shutdown_connections 2024-04-01T01:34:16.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58f6208740 1 -- 172.21.15.70:0/439430723 wait complete. 2024-04-01T01:34:16.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.109+0000 7f58f6208740 20 client.25622 trim_cache size 0 max 16384 2024-04-01T01:34:16.116 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:34:16.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.110+0000 7f58f3548640 1 -- 172.21.15.70:0/2247663612 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f58d80a4740 msgr2=0x7f58d80a6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.110+0000 7f58f3548640 1 --2- 172.21.15.70:0/2247663612 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f58d80a4740 0x7f58d80a6c00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x56044df176e0 tx=0x7f58e4009500 comp rx=0 tx=0).stop 2024-04-01T01:34:16.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.110+0000 7f58f3548640 1 -- 172.21.15.70:0/2247663612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ec100650 msgr2=0x7f58ec1957d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.110+0000 7f58f3548640 1 --2- 172.21.15.70:0/2247663612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ec100650 0x7f58ec1957d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f58e8000c00 tx=0x7f58e8005dc0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.110+0000 7f58f3548640 1 -- 172.21.15.70:0/2247663612 shutdown_connections 2024-04-01T01:34:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.110+0000 7f58f3548640 1 --2- 172.21.15.70:0/2247663612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ec102200 0x7f58ec19cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.110+0000 7f58f3548640 1 --2- 172.21.15.70:0/2247663612 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f58d80a4740 0x7f58d80a6c00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.110+0000 7f58f3548640 1 --2- 172.21.15.70:0/2247663612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ec101840 0x7f58ec195d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.110+0000 7f58f3548640 1 --2- 172.21.15.70:0/2247663612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ec100650 0x7f58ec1957d0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.110+0000 7f58f3548640 1 -- 172.21.15.70:0/2247663612 >> 172.21.15.70:0/2247663612 conn(0x7f58ec0fbbe0 msgr2=0x7f58ec104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.110+0000 7f58f3548640 1 -- 172.21.15.70:0/2247663612 shutdown_connections 2024-04-01T01:34:16.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.110+0000 7f58f3548640 1 -- 172.21.15.70:0/2247663612 wait complete. 2024-04-01T01:34:16.121 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:34:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.247+0000 7f8a3f319640 1 -- 172.21.15.70:0/2011905403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a38107430 msgr2=0x7f8a38109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.247+0000 7f8a3f319640 1 --2- 172.21.15.70:0/2011905403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a38107430 0x7f8a38109880 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8a30009900 tx=0x7f8a3002bc30 comp rx=0 tx=0).stop 2024-04-01T01:34:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.247+0000 7f8a3f319640 1 -- 172.21.15.70:0/2011905403 shutdown_connections 2024-04-01T01:34:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.247+0000 7f8a3f319640 1 --2- 172.21.15.70:0/2011905403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a38107430 0x7f8a38109880 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.247+0000 7f8a3f319640 1 --2- 172.21.15.70:0/2011905403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a38104b00 0x7f8a38106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.248+0000 7f8a3f319640 1 --2- 172.21.15.70:0/2011905403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a381021d0 0x7f8a381045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.248+0000 7f8a3f319640 1 -- 172.21.15.70:0/2011905403 >> 172.21.15.70:0/2011905403 conn(0x7f8a380fbbe0 msgr2=0x7f8a380fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.248+0000 7f8a3f319640 1 -- 172.21.15.70:0/2011905403 shutdown_connections 2024-04-01T01:34:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.248+0000 7f8a3f319640 1 -- 172.21.15.70:0/2011905403 wait complete. 2024-04-01T01:34:16.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.248+0000 7f8a3f319640 1 Processor -- start 2024-04-01T01:34:16.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.248+0000 7f8a3f319640 1 -- start start 2024-04-01T01:34:16.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.248+0000 7f8a3f319640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a381021d0 0x7f8a381979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.248+0000 7f8a3f319640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a38104b00 0x7f8a38197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.248+0000 7f8a3f319640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a38107430 0x7f8a3819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.248+0000 7f8a3f319640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3810d2a0 con 0x7f8a381021d0 2024-04-01T01:34:16.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.248+0000 7f8a3f319640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3810d120 con 0x7f8a38107430 2024-04-01T01:34:16.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.248+0000 7f8a3f319640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3810d420 con 0x7f8a38104b00 2024-04-01T01:34:16.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.248+0000 7f8a3d0b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a381021d0 0x7f8a381979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.249+0000 7f8a3d0b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a381021d0 0x7f8a381979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56938/0 (socket says 172.21.15.70:56938) 2024-04-01T01:34:16.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.249+0000 7f8a3d0b5640 1 -- 172.21.15.70:0/2035996239 learned_addr learned my addr 172.21.15.70:0/2035996239 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:16.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.249+0000 7f8a3c8b4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a38104b00 0x7f8a38197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.249+0000 7f8a3d8b6640 1 --2- 172.21.15.70:0/2035996239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a38107430 0x7f8a3819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.249+0000 7f8a3d0b5640 1 -- 172.21.15.70:0/2035996239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a38104b00 msgr2=0x7f8a38197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.249+0000 7f8a3d0b5640 1 --2- 172.21.15.70:0/2035996239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a38104b00 0x7f8a38197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.249+0000 7f8a3d0b5640 1 -- 172.21.15.70:0/2035996239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a38107430 msgr2=0x7f8a3819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.249+0000 7f8a3d0b5640 1 --2- 172.21.15.70:0/2035996239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a38107430 0x7f8a3819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.249+0000 7f8a3d0b5640 1 -- 172.21.15.70:0/2035996239 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a3819f5c0 con 0x7f8a381021d0 2024-04-01T01:34:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.249+0000 7f8a3d0b5640 1 --2- 172.21.15.70:0/2035996239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a381021d0 0x7f8a381979a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400eb90 tx=0x7f8a3400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.249+0000 7f8a2e7fc640 1 -- 172.21.15.70:0/2035996239 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a34004d20 con 0x7f8a381021d0 2024-04-01T01:34:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.249+0000 7f8a2e7fc640 1 -- 172.21.15.70:0/2035996239 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a3400ee90 con 0x7f8a381021d0 2024-04-01T01:34:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.249+0000 7f8a3f319640 1 -- 172.21.15.70:0/2035996239 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a3819f8b0 con 0x7f8a381021d0 2024-04-01T01:34:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.249+0000 7f8a2e7fc640 1 -- 172.21.15.70:0/2035996239 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a34010670 con 0x7f8a381021d0 2024-04-01T01:34:16.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.249+0000 7f8a3f319640 1 -- 172.21.15.70:0/2035996239 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a3819fdb0 con 0x7f8a381021d0 2024-04-01T01:34:16.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.252+0000 7f8a2e7fc640 1 -- 172.21.15.70:0/2035996239 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f8a34005020 con 0x7f8a381021d0 2024-04-01T01:34:16.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.252+0000 7f8a41fce740 1 Processor -- start 2024-04-01T01:34:16.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.252+0000 7f8a41fce740 1 -- start start 2024-04-01T01:34:16.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a41fce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375dd8a0 0x55ef375ddcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a41fce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375de290 0x55ef375e2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a41fce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375e3400 0x55ef375e5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a41fce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ef3753e1b0 con 0x55ef375de290 2024-04-01T01:34:16.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a41fce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ef374e6c00 con 0x55ef375e3400 2024-04-01T01:34:16.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a41fce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ef3753a250 con 0x55ef375dd8a0 2024-04-01T01:34:16.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a3d8b6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375dd8a0 0x55ef375ddcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a3d0b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375de290 0x55ef375e2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a3c8b4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375e3400 0x55ef375e5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a3d8b6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375dd8a0 0x55ef375ddcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:53982/0 (socket says 172.21.15.70:53982) 2024-04-01T01:34:16.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a3c8b4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375e3400 0x55ef375e5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38576/0 (socket says 172.21.15.70:38576) 2024-04-01T01:34:16.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a3d8b6640 1 -- 172.21.15.70:0/2961973659 learned_addr learned my addr 172.21.15.70:0/2961973659 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:16.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a2e7fc640 1 --2- 172.21.15.70:0/2035996239 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8a240a2530 0x7f8a240a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a3d8b6640 1 --2- 172.21.15.70:0/2035996239 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8a240a2530 0x7f8a240a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a3d8b6640 1 -- 172.21.15.70:0/2961973659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375e3400 msgr2=0x55ef375e5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a3d8b6640 1 --2- 172.21.15.70:0/2961973659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375e3400 0x55ef375e5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a2e7fc640 1 -- 172.21.15.70:0/2035996239 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8a340cf2f0 con 0x7f8a381021d0 2024-04-01T01:34:16.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a3d8b6640 1 -- 172.21.15.70:0/2961973659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375de290 msgr2=0x55ef375e2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a3d8b6640 1 --2- 172.21.15.70:0/2961973659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375de290 0x55ef375e2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a3d8b6640 1 -- 172.21.15.70:0/2961973659 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef375063d0 con 0x55ef375dd8a0 2024-04-01T01:34:16.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a3c8b4640 1 --2- 172.21.15.70:0/2961973659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375e3400 0x55ef375e5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:34:16.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.253+0000 7f8a3d0b5640 1 --2- 172.21.15.70:0/2961973659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375de290 0x55ef375e2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:16.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.254+0000 7f8a3d8b6640 1 --2- 172.21.15.70:0/2961973659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375dd8a0 0x55ef375ddcc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8a3002c640 tx=0x7f8a3002cc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74f96612ef54fab9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.254+0000 7f8a3d8b6640 1 --2- 172.21.15.70:0/2035996239 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8a240a2530 0x7f8a240a49f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8a30036040 tx=0x7f8a3004a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.254+0000 7f8a237fe640 1 -- 172.21.15.70:0/2961973659 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30005040 con 0x55ef375dd8a0 2024-04-01T01:34:16.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.254+0000 7f8a237fe640 1 -- 172.21.15.70:0/2961973659 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a300046c0 con 0x55ef375dd8a0 2024-04-01T01:34:16.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.254+0000 7f8a237fe640 1 -- 172.21.15.70:0/2961973659 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30003850 con 0x55ef375dd8a0 2024-04-01T01:34:16.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 -- 172.21.15.70:0/2961973659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375dd8a0 msgr2=0x55ef375ddcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 --2- 172.21.15.70:0/2961973659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375dd8a0 0x55ef375ddcc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8a3002c640 tx=0x7f8a3002cc90 comp rx=0 tx=0).stop 2024-04-01T01:34:16.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 -- 172.21.15.70:0/2961973659 shutdown_connections 2024-04-01T01:34:16.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 --2- 172.21.15.70:0/2961973659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375e3400 0x55ef375e5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 --2- 172.21.15.70:0/2961973659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375de290 0x55ef375e2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 --2- 172.21.15.70:0/2961973659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375dd8a0 0x55ef375ddcc0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 -- 172.21.15.70:0/2961973659 >> 172.21.15.70:0/2961973659 conn(0x55ef375d4a00 msgr2=0x55ef375d4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 -- 172.21.15.70:0/2961973659 shutdown_connections 2024-04-01T01:34:16.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 -- 172.21.15.70:0/2961973659 wait complete. 2024-04-01T01:34:16.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 -- start start 2024-04-01T01:34:16.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 Processor -- start 2024-04-01T01:34:16.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375dd8a0 0x55ef375e7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375de290 0x55ef375e8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375e3400 0x55ef375ef2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ef375e62f0 con 0x55ef375e3400 2024-04-01T01:34:16.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ef374dc470 con 0x55ef375de290 2024-04-01T01:34:16.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.255+0000 7f8a41fce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ef375e6470 con 0x55ef375dd8a0 2024-04-01T01:34:16.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a41fce740 10 client.0 _ll_register_callbacks cb 0x55ef374f9cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:16.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a3d8b6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375e3400 0x55ef375ef2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a3d0b5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375dd8a0 0x55ef375e7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a3d0b5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375dd8a0 0x55ef375e7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:53992/0 (socket says 172.21.15.70:53992) 2024-04-01T01:34:16.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a3d0b5640 1 -- 172.21.15.70:0/1305276312 learned_addr learned my addr 172.21.15.70:0/1305276312 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:16.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a3c8b4640 1 --2- 172.21.15.70:0/1305276312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375de290 0x55ef375e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a3d8b6640 1 -- 172.21.15.70:0/1305276312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375dd8a0 msgr2=0x55ef375e7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a3d8b6640 1 --2- 172.21.15.70:0/1305276312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375dd8a0 0x55ef375e7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a3d8b6640 1 -- 172.21.15.70:0/1305276312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375de290 msgr2=0x55ef375e8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a3d8b6640 1 --2- 172.21.15.70:0/1305276312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375de290 0x55ef375e8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a3d8b6640 1 -- 172.21.15.70:0/1305276312 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef372523f0 con 0x55ef375e3400 2024-04-01T01:34:16.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a3d8b6640 1 --2- 172.21.15.70:0/1305276312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375e3400 0x55ef375ef2b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8a3004f040 tx=0x7f8a3002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe3e1ba2c652174 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:16.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a20ff9640 1 -- 172.21.15.70:0/1305276312 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30003a40 con 0x55ef375e3400 2024-04-01T01:34:16.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a20ff9640 1 -- 172.21.15.70:0/1305276312 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8a30003be0 con 0x55ef375e3400 2024-04-01T01:34:16.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a41fce740 1 -- 172.21.15.70:0/1305276312 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a300046c0 con 0x55ef375e3400 2024-04-01T01:34:16.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a20ff9640 1 -- 172.21.15.70:0/1305276312 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8a30066050 con 0x55ef375e3400 2024-04-01T01:34:16.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a41fce740 10 client.15819 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:16.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.256+0000 7f8a41fce740 1 -- 172.21.15.70:0/1305276312 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ef3731c7d0 con 0x55ef375e3400 2024-04-01T01:34:16.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.257+0000 7f8a20ff9640 1 -- 172.21.15.70:0/1305276312 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a300046c0 con 0x55ef375e3400 2024-04-01T01:34:16.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.257+0000 7f8a20ff9640 1 -- 172.21.15.70:0/1305276312 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8a30040040 con 0x55ef375e3400 2024-04-01T01:34:16.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.257+0000 7f8a41fce740 10 client.15819 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:16.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.257+0000 7f8a41fce740 10 client.15819 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:34:16.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.257+0000 7f8a41fce740 20 client.15819 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:16.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.257+0000 7f8a41fce740 1 --2- 172.21.15.70:0/1305276312 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55ef375f3460 0x55ef375f5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.257+0000 7f8a41fce740 4 client.15819 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:16.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.257+0000 7f8a41fce740 1 -- 172.21.15.70:0/1305276312 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef3754b6f0 con 0x55ef375f3460 2024-04-01T01:34:16.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.257+0000 7f8a3d0b5640 1 --2- 172.21.15.70:0/1305276312 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55ef375f3460 0x55ef375f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.257+0000 7f8a20ff9640 10 client.15819 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:34:16.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.258+0000 7f8a3d0b5640 1 --2- 172.21.15.70:0/1305276312 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55ef375f3460 0x55ef375f5870 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fba1a483f3ecee67 server_cookie=7f39b42b4de221d2 in_seq=0 out_seq=0 2024-04-01T01:34:16.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.258+0000 7f8a20ff9640 1 -- 172.21.15.70:0/1305276312 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ef3754b6f0 con 0x55ef375f3460 2024-04-01T01:34:16.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.258+0000 7f8a20ff9640 10 client.15819 handle_command_reply: tid=0 2024-04-01T01:34:16.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.258+0000 7f8a41fce740 1 client.15819 shutdown 2024-04-01T01:34:16.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.258+0000 7f8a41fce740 1 -- 172.21.15.70:0/1305276312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375e3400 msgr2=0x55ef375ef2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.258+0000 7f8a41fce740 1 --2- 172.21.15.70:0/1305276312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375e3400 0x55ef375ef2b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8a3004f040 tx=0x7f8a3002bc30 comp rx=0 tx=0).stop 2024-04-01T01:34:16.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.258+0000 7f8a41fce740 1 -- 172.21.15.70:0/1305276312 shutdown_connections 2024-04-01T01:34:16.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.258+0000 7f8a41fce740 1 --2- 172.21.15.70:0/1305276312 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55ef375f3460 0x55ef375f5870 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.258+0000 7f8a41fce740 1 --2- 172.21.15.70:0/1305276312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375e3400 0x55ef375ef2b0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.258+0000 7f8a41fce740 1 --2- 172.21.15.70:0/1305276312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375de290 0x55ef375e8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.258+0000 7f8a41fce740 1 --2- 172.21.15.70:0/1305276312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375dd8a0 0x55ef375e7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.258+0000 7f8a41fce740 1 -- 172.21.15.70:0/1305276312 >> 172.21.15.70:0/1305276312 conn(0x55ef375d4a00 msgr2=0x55ef375d3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.258+0000 7f8a20ff9640 0 client.15819 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:34:16.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.259+0000 7f8a41fce740 1 -- 172.21.15.70:0/1305276312 shutdown_connections 2024-04-01T01:34:16.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.259+0000 7f8a41fce740 1 -- 172.21.15.70:0/1305276312 wait complete. 2024-04-01T01:34:16.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.259+0000 7f8a41fce740 20 client.15819 trim_cache size 0 max 16384 2024-04-01T01:34:16.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.274+0000 7f8a41fce740 1 Processor -- start 2024-04-01T01:34:16.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.274+0000 7f8a41fce740 1 -- start start 2024-04-01T01:34:16.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.274+0000 7f8a41fce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375dd8a0 0x55ef375de980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.274+0000 7f8a41fce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375de290 0x55ef375ef270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.274+0000 7f8a41fce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375e3400 0x55ef375deec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.274+0000 7f8a41fce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381a8280 con 0x55ef375dd8a0 2024-04-01T01:34:16.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.274+0000 7f8a41fce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381a9dc0 con 0x55ef375e3400 2024-04-01T01:34:16.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.274+0000 7f8a41fce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381a8950 con 0x55ef375de290 2024-04-01T01:34:16.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.275+0000 7f8a3d0b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375dd8a0 0x55ef375de980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.275+0000 7f8a3d8b6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375e3400 0x55ef375deec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.275+0000 7f8a3d8b6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375e3400 0x55ef375deec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38586/0 (socket says 172.21.15.70:38586) 2024-04-01T01:34:16.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.275+0000 7f8a3c8b4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375de290 0x55ef375ef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.275+0000 7f8a3d8b6640 1 -- 172.21.15.70:0/2180745485 learned_addr learned my addr 172.21.15.70:0/2180745485 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:16.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.275+0000 7f8a3d8b6640 1 -- 172.21.15.70:0/2180745485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375de290 msgr2=0x55ef375ef270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.275+0000 7f8a3d8b6640 1 --2- 172.21.15.70:0/2180745485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375de290 0x55ef375ef270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.275+0000 7f8a3d8b6640 1 -- 172.21.15.70:0/2180745485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375dd8a0 msgr2=0x55ef375de980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.275+0000 7f8a3d8b6640 1 --2- 172.21.15.70:0/2180745485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375dd8a0 0x55ef375de980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.275+0000 7f8a3d8b6640 1 -- 172.21.15.70:0/2180745485 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef375e2aa0 con 0x55ef375e3400 2024-04-01T01:34:16.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.275+0000 7f8a3d0b5640 1 --2- 172.21.15.70:0/2180745485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375dd8a0 0x55ef375de980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:16.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.275+0000 7f8a3d8b6640 1 --2- 172.21.15.70:0/2180745485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375e3400 0x55ef375deec0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8a3002c640 tx=0x7f8a30009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44e810822b8898db server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.275+0000 7f8a22ffd640 1 -- 172.21.15.70:0/2180745485 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3002cc90 con 0x55ef375e3400 2024-04-01T01:34:16.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.275+0000 7f8a22ffd640 1 -- 172.21.15.70:0/2180745485 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a30056470 con 0x55ef375e3400 2024-04-01T01:34:16.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.275+0000 7f8a22ffd640 1 -- 172.21.15.70:0/2180745485 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a30060690 con 0x55ef375e3400 2024-04-01T01:34:16.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.276+0000 7f8a41fce740 1 -- 172.21.15.70:0/2180745485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375e3400 msgr2=0x55ef375deec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.276+0000 7f8a41fce740 1 --2- 172.21.15.70:0/2180745485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375e3400 0x55ef375deec0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8a3002c640 tx=0x7f8a30009520 comp rx=0 tx=0).stop 2024-04-01T01:34:16.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.276+0000 7f8a41fce740 1 -- 172.21.15.70:0/2180745485 shutdown_connections 2024-04-01T01:34:16.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.276+0000 7f8a41fce740 1 --2- 172.21.15.70:0/2180745485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375e3400 0x55ef375deec0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.276+0000 7f8a41fce740 1 --2- 172.21.15.70:0/2180745485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375de290 0x55ef375ef270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.276+0000 7f8a41fce740 1 --2- 172.21.15.70:0/2180745485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375dd8a0 0x55ef375de980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.276+0000 7f8a41fce740 1 -- 172.21.15.70:0/2180745485 >> 172.21.15.70:0/2180745485 conn(0x55ef375d4a00 msgr2=0x55ef375e6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.276+0000 7f8a41fce740 1 -- 172.21.15.70:0/2180745485 shutdown_connections 2024-04-01T01:34:16.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.276+0000 7f8a41fce740 1 -- 172.21.15.70:0/2180745485 wait complete. 2024-04-01T01:34:16.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.276+0000 7f8a41fce740 1 -- start start 2024-04-01T01:34:16.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a41fce740 1 Processor -- start 2024-04-01T01:34:16.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a41fce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375dd8a0 0x55ef375ec690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a41fce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375de290 0x55ef375ecbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a41fce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375e3400 0x55ef375edbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a41fce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381a96f0 con 0x55ef375dd8a0 2024-04-01T01:34:16.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a41fce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381a9020 con 0x55ef375de290 2024-04-01T01:34:16.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a41fce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8a381a6260 con 0x55ef375e3400 2024-04-01T01:34:16.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a41fce740 10 client.0 _ll_register_callbacks cb 0x7f8a300035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:16.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a3d0b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375dd8a0 0x55ef375ec690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a3d0b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375dd8a0 0x55ef375ec690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56974/0 (socket says 172.21.15.70:56974) 2024-04-01T01:34:16.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a3d0b5640 1 -- 172.21.15.70:0/3332836716 learned_addr learned my addr 172.21.15.70:0/3332836716 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:16.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a3c8b4640 1 --2- 172.21.15.70:0/3332836716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375de290 0x55ef375ecbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a3d8b6640 1 --2- 172.21.15.70:0/3332836716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375e3400 0x55ef375edbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a3d0b5640 1 -- 172.21.15.70:0/3332836716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375e3400 msgr2=0x55ef375edbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a3d0b5640 1 --2- 172.21.15.70:0/3332836716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375e3400 0x55ef375edbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a3d0b5640 1 -- 172.21.15.70:0/3332836716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375de290 msgr2=0x55ef375ecbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a3d0b5640 1 --2- 172.21.15.70:0/3332836716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375de290 0x55ef375ecbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a3d0b5640 1 -- 172.21.15.70:0/3332836716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef375ee1d0 con 0x55ef375dd8a0 2024-04-01T01:34:16.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a3d0b5640 1 --2- 172.21.15.70:0/3332836716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375dd8a0 0x55ef375ec690 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400cba0 tx=0x7f8a34086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f67acd1c18c88c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.277+0000 7f8a20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:16.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.278+0000 7f8a20ff9640 1 -- 172.21.15.70:0/3332836716 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3409f9d0 con 0x55ef375dd8a0 2024-04-01T01:34:16.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.278+0000 7f8a20ff9640 1 -- 172.21.15.70:0/3332836716 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8a34005560 con 0x55ef375dd8a0 2024-04-01T01:34:16.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.278+0000 7f8a41fce740 1 -- 172.21.15.70:0/3332836716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a30056470 con 0x55ef375dd8a0 2024-04-01T01:34:16.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.278+0000 7f8a20ff9640 1 -- 172.21.15.70:0/3332836716 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8a340a4290 con 0x55ef375dd8a0 2024-04-01T01:34:16.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.278+0000 7f8a41fce740 10 client.15825 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:16.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.278+0000 7f8a41fce740 1 -- 172.21.15.70:0/3332836716 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ef375ef5a0 con 0x55ef375dd8a0 2024-04-01T01:34:16.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.278+0000 7f8a20ff9640 1 -- 172.21.15.70:0/3332836716 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a30056470 con 0x55ef375dd8a0 2024-04-01T01:34:16.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.278+0000 7f8a20ff9640 1 -- 172.21.15.70:0/3332836716 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8a340860b0 con 0x55ef375dd8a0 2024-04-01T01:34:16.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.278+0000 7f8a41fce740 10 client.15825 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:16.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.278+0000 7f8a41fce740 10 client.15825 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:34:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.278+0000 7f8a41fce740 20 client.15825 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.278+0000 7f8a41fce740 1 --2- 172.21.15.70:0/3332836716 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55ef37659f20 0x55ef3765c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.278+0000 7f8a41fce740 4 client.15825 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:34:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.278+0000 7f8a41fce740 1 -- 172.21.15.70:0/3332836716 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ef37545a40 con 0x55ef37659f20 2024-04-01T01:34:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.278+0000 7f8a3c8b4640 1 --2- 172.21.15.70:0/3332836716 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55ef37659f20 0x55ef3765c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.279+0000 7f8a20ff9640 10 client.15825 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:34:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.279+0000 7f8a3c8b4640 1 --2- 172.21.15.70:0/3332836716 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55ef37659f20 0x55ef3765c340 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d38cabe6a2a40d9d server_cookie=c9fc7a6795a1b818 in_seq=0 out_seq=0 2024-04-01T01:34:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.279+0000 7f8a20ff9640 1 -- 172.21.15.70:0/3332836716 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ef37545a40 con 0x55ef37659f20 2024-04-01T01:34:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.279+0000 7f8a20ff9640 10 client.15825 handle_command_reply: tid=0 2024-04-01T01:34:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.279+0000 7f8a41fce740 1 client.15825 shutdown 2024-04-01T01:34:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.279+0000 7f8a41fce740 1 -- 172.21.15.70:0/3332836716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375dd8a0 msgr2=0x55ef375ec690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.279+0000 7f8a41fce740 1 --2- 172.21.15.70:0/3332836716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375dd8a0 0x55ef375ec690 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400cba0 tx=0x7f8a34086040 comp rx=0 tx=0).stop 2024-04-01T01:34:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.279+0000 7f8a41fce740 1 -- 172.21.15.70:0/3332836716 shutdown_connections 2024-04-01T01:34:16.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.279+0000 7f8a41fce740 1 --2- 172.21.15.70:0/3332836716 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55ef37659f20 0x55ef3765c340 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.279+0000 7f8a41fce740 1 --2- 172.21.15.70:0/3332836716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef375e3400 0x55ef375edbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.279+0000 7f8a41fce740 1 --2- 172.21.15.70:0/3332836716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ef375de290 0x55ef375ecbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.279+0000 7f8a20ff9640 0 client.15825 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:34:16.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.279+0000 7f8a41fce740 1 --2- 172.21.15.70:0/3332836716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ef375dd8a0 0x55ef375ec690 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.279+0000 7f8a41fce740 1 -- 172.21.15.70:0/3332836716 >> 172.21.15.70:0/3332836716 conn(0x55ef375d4a00 msgr2=0x55ef375e6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.279+0000 7f8a41fce740 1 -- 172.21.15.70:0/3332836716 shutdown_connections 2024-04-01T01:34:16.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.280+0000 7f8a41fce740 1 -- 172.21.15.70:0/3332836716 wait complete. 2024-04-01T01:34:16.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.280+0000 7f8a41fce740 20 client.15825 trim_cache size 0 max 16384 2024-04-01T01:34:16.286 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:34:16.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.280+0000 7f8a3f319640 1 -- 172.21.15.70:0/2035996239 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8a240a2530 msgr2=0x7f8a240a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.280+0000 7f8a3f319640 1 --2- 172.21.15.70:0/2035996239 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8a240a2530 0x7f8a240a49f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8a30036040 tx=0x7f8a3004a000 comp rx=0 tx=0).stop 2024-04-01T01:34:16.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.280+0000 7f8a3f319640 1 -- 172.21.15.70:0/2035996239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a381021d0 msgr2=0x7f8a381979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.280+0000 7f8a3f319640 1 --2- 172.21.15.70:0/2035996239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a381021d0 0x7f8a381979a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400eb90 tx=0x7f8a3400c6a0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.280+0000 7f8a3f319640 1 -- 172.21.15.70:0/2035996239 shutdown_connections 2024-04-01T01:34:16.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.280+0000 7f8a3f319640 1 --2- 172.21.15.70:0/2035996239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a38107430 0x7f8a3819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.280+0000 7f8a3f319640 1 --2- 172.21.15.70:0/2035996239 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8a240a2530 0x7f8a240a49f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.280+0000 7f8a3f319640 1 --2- 172.21.15.70:0/2035996239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a38104b00 0x7f8a38197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.280+0000 7f8a3f319640 1 --2- 172.21.15.70:0/2035996239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a381021d0 0x7f8a381979a0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.280+0000 7f8a3f319640 1 -- 172.21.15.70:0/2035996239 >> 172.21.15.70:0/2035996239 conn(0x7f8a380fbbe0 msgr2=0x7f8a38102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.280+0000 7f8a3f319640 1 -- 172.21.15.70:0/2035996239 shutdown_connections 2024-04-01T01:34:16.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.280+0000 7f8a3f319640 1 -- 172.21.15.70:0/2035996239 wait complete. 2024-04-01T01:34:16.293 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:34:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.413+0000 7fe913e8f640 1 -- 172.21.15.70:0/3592120270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe90c101840 msgr2=0x7fe90c101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.413+0000 7fe913e8f640 1 --2- 172.21.15.70:0/3592120270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe90c101840 0x7fe90c101cc0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009900 tx=0x7fe8fc02bc30 comp rx=0 tx=0).stop 2024-04-01T01:34:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.413+0000 7fe913e8f640 1 -- 172.21.15.70:0/3592120270 shutdown_connections 2024-04-01T01:34:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.413+0000 7fe913e8f640 1 --2- 172.21.15.70:0/3592120270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe90c102200 0x7fe90c108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.413+0000 7fe913e8f640 1 --2- 172.21.15.70:0/3592120270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe90c101840 0x7fe90c101cc0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.413+0000 7fe913e8f640 1 --2- 172.21.15.70:0/3592120270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe90c100650 0x7fe90c100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.413+0000 7fe913e8f640 1 -- 172.21.15.70:0/3592120270 >> 172.21.15.70:0/3592120270 conn(0x7fe90c0fbbe0 msgr2=0x7fe90c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.413+0000 7fe913e8f640 1 -- 172.21.15.70:0/3592120270 shutdown_connections 2024-04-01T01:34:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.413+0000 7fe913e8f640 1 -- 172.21.15.70:0/3592120270 wait complete. 2024-04-01T01:34:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.414+0000 7fe913e8f640 1 Processor -- start 2024-04-01T01:34:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.414+0000 7fe913e8f640 1 -- start start 2024-04-01T01:34:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.414+0000 7fe913e8f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe90c100650 0x7fe90c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.414+0000 7fe913e8f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe90c101840 0x7fe90c19eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.414+0000 7fe913e8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe90c102200 0x7fe90c1a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.414+0000 7fe913e8f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c10c190 con 0x7fe90c102200 2024-04-01T01:34:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.414+0000 7fe913e8f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c10c010 con 0x7fe90c100650 2024-04-01T01:34:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.414+0000 7fe913e8f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c10c310 con 0x7fe90c101840 2024-04-01T01:34:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.414+0000 7fe91242c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe90c102200 0x7fe90c1a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.414+0000 7fe911c2b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe90c100650 0x7fe90c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.414+0000 7fe911c2b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe90c100650 0x7fe90c069420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38604/0 (socket says 172.21.15.70:38604) 2024-04-01T01:34:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.414+0000 7fe91242c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe90c102200 0x7fe90c1a1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56988/0 (socket says 172.21.15.70:56988) 2024-04-01T01:34:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.414+0000 7fe91142a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe90c101840 0x7fe90c19eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.414+0000 7fe91242c640 1 -- 172.21.15.70:0/1524377920 learned_addr learned my addr 172.21.15.70:0/1524377920 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.415+0000 7fe91242c640 1 -- 172.21.15.70:0/1524377920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe90c101840 msgr2=0x7fe90c19eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.415+0000 7fe91242c640 1 --2- 172.21.15.70:0/1524377920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe90c101840 0x7fe90c19eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.415+0000 7fe91242c640 1 -- 172.21.15.70:0/1524377920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe90c100650 msgr2=0x7fe90c069420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.415+0000 7fe91242c640 1 --2- 172.21.15.70:0/1524377920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe90c100650 0x7fe90c069420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.415+0000 7fe91242c640 1 -- 172.21.15.70:0/1524377920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe90c1a1880 con 0x7fe90c102200 2024-04-01T01:34:16.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.415+0000 7fe91242c640 1 --2- 172.21.15.70:0/1524377920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe90c102200 0x7fe90c1a1090 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fe90400eb10 tx=0x7fe90400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.415+0000 7fe902ffd640 1 -- 172.21.15.70:0/1524377920 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904004b20 con 0x7fe90c102200 2024-04-01T01:34:16.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.415+0000 7fe902ffd640 1 -- 172.21.15.70:0/1524377920 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe904004cc0 con 0x7fe90c102200 2024-04-01T01:34:16.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.415+0000 7fe913e8f640 1 -- 172.21.15.70:0/1524377920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe90c1a1b10 con 0x7fe90c102200 2024-04-01T01:34:16.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.415+0000 7fe902ffd640 1 -- 172.21.15.70:0/1524377920 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904010690 con 0x7fe90c102200 2024-04-01T01:34:16.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.415+0000 7fe913e8f640 1 -- 172.21.15.70:0/1524377920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe90c1a1fe0 con 0x7fe90c102200 2024-04-01T01:34:16.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.417+0000 7fe902ffd640 1 -- 172.21.15.70:0/1524377920 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7fe904005020 con 0x7fe90c102200 2024-04-01T01:34:16.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.418+0000 7fe916a73740 1 Processor -- start 2024-04-01T01:34:16.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.418+0000 7fe916a73740 1 -- start start 2024-04-01T01:34:16.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.418+0000 7fe902ffd640 1 --2- 172.21.15.70:0/1524377920 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8f80a2560 0x7fe8f80a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.418+0000 7fe916a73740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275b490 0x56063275b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.418+0000 7fe916a73740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063275be80 0x560632760980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.418+0000 7fe916a73740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560632760ff0 0x560632763480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.418+0000 7fe916a73740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5606326c01b0 con 0x56063275be80 2024-04-01T01:34:16.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.418+0000 7fe91142a640 1 --2- 172.21.15.70:0/1524377920 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8f80a2560 0x7fe8f80a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.418+0000 7fe916a73740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560632668c00 con 0x56063275b490 2024-04-01T01:34:16.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.418+0000 7fe916a73740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5606326bc250 con 0x560632760ff0 2024-04-01T01:34:16.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.418+0000 7fe902ffd640 1 -- 172.21.15.70:0/1524377920 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe9040cf1b0 con 0x7fe90c102200 2024-04-01T01:34:16.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.418+0000 7fe911c2b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275b490 0x56063275b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.418+0000 7fe911c2b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275b490 0x56063275b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38612/0 (socket says 172.21.15.70:38612) 2024-04-01T01:34:16.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.418+0000 7fe911c2b640 1 -- 172.21.15.70:0/3620666078 learned_addr learned my addr 172.21.15.70:0/3620666078 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:16.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.418+0000 7fe91142a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063275be80 0x560632760980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.418+0000 7fe91242c640 1 --2- 172.21.15.70:0/3620666078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560632760ff0 0x560632763480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.419+0000 7fe911c2b640 1 -- 172.21.15.70:0/3620666078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560632760ff0 msgr2=0x560632763480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.419+0000 7fe911c2b640 1 --2- 172.21.15.70:0/3620666078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560632760ff0 0x560632763480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.419+0000 7fe911c2b640 1 -- 172.21.15.70:0/3620666078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063275be80 msgr2=0x560632760980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.419+0000 7fe911c2b640 1 --2- 172.21.15.70:0/3620666078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063275be80 0x560632760980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.419+0000 7fe911c2b640 1 -- 172.21.15.70:0/3620666078 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606326883d0 con 0x56063275b490 2024-04-01T01:34:16.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.419+0000 7fe91142a640 1 --2- 172.21.15.70:0/3620666078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063275be80 0x560632760980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:16.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.419+0000 7fe91142a640 1 --2- 172.21.15.70:0/1524377920 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8f80a2560 0x7fe8f80a4a20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009230 tx=0x7fe8fc009560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.419+0000 7fe911c2b640 1 --2- 172.21.15.70:0/3620666078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275b490 0x56063275b8b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe908009ca0 tx=0x7fe908011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5244d23bff7ad454 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.419+0000 7fe8f7fff640 1 -- 172.21.15.70:0/3620666078 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe908004220 con 0x56063275b490 2024-04-01T01:34:16.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.419+0000 7fe8f7fff640 1 -- 172.21.15.70:0/3620666078 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe908009e50 con 0x56063275b490 2024-04-01T01:34:16.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.419+0000 7fe8f7fff640 1 -- 172.21.15.70:0/3620666078 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe908019690 con 0x56063275b490 2024-04-01T01:34:16.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.420+0000 7fe916a73740 1 -- 172.21.15.70:0/3620666078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275b490 msgr2=0x56063275b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.420+0000 7fe916a73740 1 --2- 172.21.15.70:0/3620666078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275b490 0x56063275b8b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe908009ca0 tx=0x7fe908011810 comp rx=0 tx=0).stop 2024-04-01T01:34:16.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.420+0000 7fe916a73740 1 -- 172.21.15.70:0/3620666078 shutdown_connections 2024-04-01T01:34:16.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.420+0000 7fe916a73740 1 --2- 172.21.15.70:0/3620666078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560632760ff0 0x560632763480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.420+0000 7fe916a73740 1 --2- 172.21.15.70:0/3620666078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063275be80 0x560632760980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.420+0000 7fe916a73740 1 --2- 172.21.15.70:0/3620666078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275b490 0x56063275b8b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.420+0000 7fe916a73740 1 -- 172.21.15.70:0/3620666078 >> 172.21.15.70:0/3620666078 conn(0x560632756a00 msgr2=0x560632756e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.420+0000 7fe916a73740 1 -- 172.21.15.70:0/3620666078 shutdown_connections 2024-04-01T01:34:16.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.420+0000 7fe916a73740 1 -- 172.21.15.70:0/3620666078 wait complete. 2024-04-01T01:34:16.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.420+0000 7fe916a73740 1 -- start start 2024-04-01T01:34:16.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.420+0000 7fe916a73740 1 Processor -- start 2024-04-01T01:34:16.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe916a73740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063275b490 0x560632765910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe916a73740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 0x560632765e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe916a73740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560632760ff0 0x56063276ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe916a73740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560632763fa0 con 0x56063275b490 2024-04-01T01:34:16.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe916a73740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56063265e470 con 0x56063275be80 2024-04-01T01:34:16.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe916a73740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560632764120 con 0x560632760ff0 2024-04-01T01:34:16.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe916a73740 10 client.0 _ll_register_callbacks cb 0x56063267bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:16.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe911c2b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063275b490 0x560632765910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe911c2b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063275b490 0x560632765910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57004/0 (socket says 172.21.15.70:57004) 2024-04-01T01:34:16.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe911c2b640 1 -- 172.21.15.70:0/2396932695 learned_addr learned my addr 172.21.15.70:0/2396932695 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:16.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe91142a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 0x560632765e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe91242c640 1 --2- 172.21.15.70:0/2396932695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560632760ff0 0x56063276ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe911c2b640 1 -- 172.21.15.70:0/2396932695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560632760ff0 msgr2=0x56063276ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe911c2b640 1 --2- 172.21.15.70:0/2396932695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560632760ff0 0x56063276ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe911c2b640 1 -- 172.21.15.70:0/2396932695 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 msgr2=0x560632765e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe911c2b640 1 --2- 172.21.15.70:0/2396932695 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 0x560632765e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe911c2b640 1 -- 172.21.15.70:0/2396932695 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606323d43f0 con 0x56063275b490 2024-04-01T01:34:16.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe911c2b640 1 --2- 172.21.15.70:0/2396932695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063275b490 0x560632765910 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe908009c70 tx=0x7fe9080038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8fc3a4aeac0fbcd server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.421+0000 7fe8f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:16.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.422+0000 7fe8f57fa640 1 -- 172.21.15.70:0/2396932695 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe908028020 con 0x56063275b490 2024-04-01T01:34:16.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.422+0000 7fe8f57fa640 1 -- 172.21.15.70:0/2396932695 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe908005050 con 0x56063275b490 2024-04-01T01:34:16.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.422+0000 7fe916a73740 1 -- 172.21.15.70:0/2396932695 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe908009e50 con 0x56063275b490 2024-04-01T01:34:16.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.422+0000 7fe8f57fa640 1 -- 172.21.15.70:0/2396932695 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe908033050 con 0x56063275b490 2024-04-01T01:34:16.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.422+0000 7fe916a73740 10 client.15837 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:16.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.422+0000 7fe916a73740 1 -- 172.21.15.70:0/2396932695 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56063249e7d0 con 0x56063275b490 2024-04-01T01:34:16.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.422+0000 7fe8f57fa640 1 -- 172.21.15.70:0/2396932695 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe908009e50 con 0x56063275b490 2024-04-01T01:34:16.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.422+0000 7fe8f57fa640 1 -- 172.21.15.70:0/2396932695 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe908004bb0 con 0x56063275b490 2024-04-01T01:34:16.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.422+0000 7fe916a73740 10 client.15837 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:16.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.422+0000 7fe916a73740 10 client.15837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:16.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.422+0000 7fe916a73740 20 client.15837 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:16.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.422+0000 7fe916a73740 1 --2- 172.21.15.70:0/2396932695 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5606327711c0 0x560632773660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.422+0000 7fe916a73740 4 client.15837 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:16.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.422+0000 7fe916a73740 1 -- 172.21.15.70:0/2396932695 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606326cd6f0 con 0x5606327711c0 2024-04-01T01:34:16.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.423+0000 7fe91142a640 1 --2- 172.21.15.70:0/2396932695 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5606327711c0 0x560632773660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.423+0000 7fe8f57fa640 10 client.15837 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:16.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.423+0000 7fe91142a640 1 --2- 172.21.15.70:0/2396932695 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5606327711c0 0x560632773660 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aaae55540c11628c server_cookie=89193d578bb46a8 in_seq=0 out_seq=0 2024-04-01T01:34:16.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.424+0000 7fe8f57fa640 1 -- 172.21.15.70:0/2396932695 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5606326cd6f0 con 0x5606327711c0 2024-04-01T01:34:16.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.424+0000 7fe8f57fa640 10 client.15837 handle_command_reply: tid=0 2024-04-01T01:34:16.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.424+0000 7fe916a73740 1 client.15837 shutdown 2024-04-01T01:34:16.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.424+0000 7fe916a73740 1 -- 172.21.15.70:0/2396932695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063275b490 msgr2=0x560632765910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.424+0000 7fe916a73740 1 --2- 172.21.15.70:0/2396932695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063275b490 0x560632765910 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe908009c70 tx=0x7fe9080038a0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.424+0000 7fe916a73740 1 -- 172.21.15.70:0/2396932695 shutdown_connections 2024-04-01T01:34:16.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.424+0000 7fe916a73740 1 --2- 172.21.15.70:0/2396932695 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5606327711c0 0x560632773660 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.424+0000 7fe916a73740 1 --2- 172.21.15.70:0/2396932695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560632760ff0 0x56063276ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.424+0000 7fe8f57fa640 0 client.15837 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:16.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.424+0000 7fe916a73740 1 --2- 172.21.15.70:0/2396932695 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 0x560632765e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.424+0000 7fe916a73740 1 --2- 172.21.15.70:0/2396932695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56063275b490 0x560632765910 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.424+0000 7fe916a73740 1 -- 172.21.15.70:0/2396932695 >> 172.21.15.70:0/2396932695 conn(0x560632756a00 msgr2=0x560632755d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.424+0000 7fe916a73740 1 -- 172.21.15.70:0/2396932695 shutdown_connections 2024-04-01T01:34:16.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.424+0000 7fe916a73740 1 -- 172.21.15.70:0/2396932695 wait complete. 2024-04-01T01:34:16.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.424+0000 7fe916a73740 20 client.15837 trim_cache size 0 max 16384 2024-04-01T01:34:16.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.438+0000 7fe916a73740 1 Processor -- start 2024-04-01T01:34:16.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.438+0000 7fe916a73740 1 -- start start 2024-04-01T01:34:16.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.438+0000 7fe916a73740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56063275b490 0x560632764f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.438+0000 7fe916a73740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 0x56063276ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.438+0000 7fe916a73740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560632760ff0 0x560632765470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.438+0000 7fe916a73740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c1ac700 con 0x560632760ff0 2024-04-01T01:34:16.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.438+0000 7fe916a73740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c1ae240 con 0x56063275be80 2024-04-01T01:34:16.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.438+0000 7fe916a73740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c1acdd0 con 0x56063275b490 2024-04-01T01:34:16.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.439+0000 7fe91242c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560632760ff0 0x560632765470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.439+0000 7fe911c2b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56063275b490 0x560632764f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.439+0000 7fe91242c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560632760ff0 0x560632765470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57018/0 (socket says 172.21.15.70:57018) 2024-04-01T01:34:16.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.439+0000 7fe91242c640 1 -- 172.21.15.70:0/4286478421 learned_addr learned my addr 172.21.15.70:0/4286478421 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:16.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.439+0000 7fe91142a640 1 --2- 172.21.15.70:0/4286478421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 0x56063276ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.439+0000 7fe91242c640 1 -- 172.21.15.70:0/4286478421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56063275b490 msgr2=0x560632764f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.439+0000 7fe91242c640 1 --2- 172.21.15.70:0/4286478421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56063275b490 0x560632764f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.439+0000 7fe91242c640 1 -- 172.21.15.70:0/4286478421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 msgr2=0x56063276ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.439+0000 7fe91242c640 1 --2- 172.21.15.70:0/4286478421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 0x56063276ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.439+0000 7fe91242c640 1 -- 172.21.15.70:0/4286478421 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56063275c260 con 0x560632760ff0 2024-04-01T01:34:16.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.439+0000 7fe91242c640 1 --2- 172.21.15.70:0/4286478421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560632760ff0 0x560632765470 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe904005850 tx=0x7fe90400efd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1867ec813c10fc9f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.439+0000 7fe8f77fe640 1 -- 172.21.15.70:0/4286478421 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9040b4070 con 0x560632760ff0 2024-04-01T01:34:16.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.439+0000 7fe8f77fe640 1 -- 172.21.15.70:0/4286478421 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe90409be40 con 0x560632760ff0 2024-04-01T01:34:16.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.439+0000 7fe8f77fe640 1 -- 172.21.15.70:0/4286478421 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9040a4150 con 0x560632760ff0 2024-04-01T01:34:16.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.440+0000 7fe916a73740 1 -- 172.21.15.70:0/4286478421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560632760ff0 msgr2=0x560632765470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.440+0000 7fe916a73740 1 --2- 172.21.15.70:0/4286478421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560632760ff0 0x560632765470 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe904005850 tx=0x7fe90400efd0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.440+0000 7fe916a73740 1 -- 172.21.15.70:0/4286478421 shutdown_connections 2024-04-01T01:34:16.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.440+0000 7fe916a73740 1 --2- 172.21.15.70:0/4286478421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560632760ff0 0x560632765470 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.440+0000 7fe916a73740 1 --2- 172.21.15.70:0/4286478421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 0x56063276ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.440+0000 7fe916a73740 1 --2- 172.21.15.70:0/4286478421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56063275b490 0x560632764f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.440+0000 7fe916a73740 1 -- 172.21.15.70:0/4286478421 >> 172.21.15.70:0/4286478421 conn(0x560632756a00 msgr2=0x560632755d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.440+0000 7fe916a73740 1 -- 172.21.15.70:0/4286478421 shutdown_connections 2024-04-01T01:34:16.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.440+0000 7fe916a73740 1 -- 172.21.15.70:0/4286478421 wait complete. 2024-04-01T01:34:16.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.440+0000 7fe916a73740 1 -- start start 2024-04-01T01:34:16.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.440+0000 7fe916a73740 1 Processor -- start 2024-04-01T01:34:16.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe916a73740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56063275b490 0x56063276a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe916a73740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 0x56063276a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe916a73740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560632760ff0 0x56063276b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe916a73740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c1adb70 con 0x560632760ff0 2024-04-01T01:34:16.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe916a73740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c1ad4a0 con 0x56063275be80 2024-04-01T01:34:16.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe916a73740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c1aa6e0 con 0x56063275b490 2024-04-01T01:34:16.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe916a73740 10 client.0 _ll_register_callbacks cb 0x7fe908004330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:16.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe91142a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 0x56063276a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe91242c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560632760ff0 0x56063276b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe911c2b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56063275b490 0x56063276a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe91142a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 0x56063276a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38628/0 (socket says 172.21.15.70:38628) 2024-04-01T01:34:16.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe911c2b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56063275b490 0x56063276a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:54084/0 (socket says 172.21.15.70:54084) 2024-04-01T01:34:16.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe91242c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560632760ff0 0x56063276b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:57024/0 (socket says 172.21.15.70:57024) 2024-04-01T01:34:16.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe91142a640 1 -- 172.21.15.70:0/102397284 learned_addr learned my addr 172.21.15.70:0/102397284 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:16.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe91242c640 1 -- 172.21.15.70:0/102397284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56063275b490 msgr2=0x56063276a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe91242c640 1 --2- 172.21.15.70:0/102397284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56063275b490 0x56063276a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe91242c640 1 -- 172.21.15.70:0/102397284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 msgr2=0x56063276a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:16.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe91242c640 1 --2- 172.21.15.70:0/102397284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 0x56063276a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe91242c640 1 -- 172.21.15.70:0/102397284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56063276bf10 con 0x560632760ff0 2024-04-01T01:34:16.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe91142a640 1 --2- 172.21.15.70:0/102397284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 0x56063276a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:16.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe91242c640 1 --2- 172.21.15.70:0/102397284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560632760ff0 0x56063276b920 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe904097ba0 tx=0x7fe904004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18fb83e1ed31209b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:16.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.441+0000 7fe8f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:16.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.442+0000 7fe8f57fa640 1 -- 172.21.15.70:0/102397284 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9040b4040 con 0x560632760ff0 2024-04-01T01:34:16.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.442+0000 7fe8f57fa640 1 -- 172.21.15.70:0/102397284 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe90409b9a0 con 0x560632760ff0 2024-04-01T01:34:16.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.442+0000 7fe916a73740 1 -- 172.21.15.70:0/102397284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe90409be40 con 0x560632760ff0 2024-04-01T01:34:16.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.442+0000 7fe8f57fa640 1 -- 172.21.15.70:0/102397284 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe9040a4150 con 0x560632760ff0 2024-04-01T01:34:16.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.442+0000 7fe916a73740 10 client.15843 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:16.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.442+0000 7fe916a73740 1 -- 172.21.15.70:0/102397284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56063276d210 con 0x560632760ff0 2024-04-01T01:34:16.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.442+0000 7fe8f57fa640 1 -- 172.21.15.70:0/102397284 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe90409be40 con 0x560632760ff0 2024-04-01T01:34:16.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.442+0000 7fe8f57fa640 1 -- 172.21.15.70:0/102397284 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe904005310 con 0x560632760ff0 2024-04-01T01:34:16.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.442+0000 7fe916a73740 10 client.15843 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:16.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.442+0000 7fe916a73740 10 client.15843 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:16.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.442+0000 7fe916a73740 20 client.15843 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:16.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.442+0000 7fe916a73740 1 --2- 172.21.15.70:0/102397284 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5606327d7db0 0x5606327da1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:16.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.442+0000 7fe916a73740 4 client.15843 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:34:16.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.442+0000 7fe916a73740 1 -- 172.21.15.70:0/102397284 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5606326c7a40 con 0x5606327d7db0 2024-04-01T01:34:16.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.442+0000 7fe911c2b640 1 --2- 172.21.15.70:0/102397284 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5606327d7db0 0x5606327da1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:16.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe911c2b640 1 --2- 172.21.15.70:0/102397284 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5606327d7db0 0x5606327da1a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c65a69999a5106d server_cookie=72d92f9656271a3 in_seq=0 out_seq=0 2024-04-01T01:34:16.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe8f57fa640 10 client.15843 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:16.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe8f57fa640 1 -- 172.21.15.70:0/102397284 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5606326c7a40 con 0x5606327d7db0 2024-04-01T01:34:16.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe8f57fa640 10 client.15843 handle_command_reply: tid=0 2024-04-01T01:34:16.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe916a73740 1 client.15843 shutdown 2024-04-01T01:34:16.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe916a73740 1 -- 172.21.15.70:0/102397284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560632760ff0 msgr2=0x56063276b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe916a73740 1 --2- 172.21.15.70:0/102397284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560632760ff0 0x56063276b920 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe904097ba0 tx=0x7fe904004a40 comp rx=0 tx=0).stop 2024-04-01T01:34:16.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe916a73740 1 -- 172.21.15.70:0/102397284 shutdown_connections 2024-04-01T01:34:16.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe916a73740 1 --2- 172.21.15.70:0/102397284 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5606327d7db0 0x5606327da1a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe916a73740 1 --2- 172.21.15.70:0/102397284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560632760ff0 0x56063276b920 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe916a73740 1 --2- 172.21.15.70:0/102397284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56063275be80 0x56063276a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe916a73740 1 --2- 172.21.15.70:0/102397284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56063275b490 0x56063276a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe8f57fa640 0 client.15843 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:16.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe916a73740 1 -- 172.21.15.70:0/102397284 >> 172.21.15.70:0/102397284 conn(0x560632756a00 msgr2=0x5606327554b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe916a73740 1 -- 172.21.15.70:0/102397284 shutdown_connections 2024-04-01T01:34:16.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe916a73740 1 -- 172.21.15.70:0/102397284 wait complete. 2024-04-01T01:34:16.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.443+0000 7fe916a73740 20 client.15843 trim_cache size 0 max 16384 2024-04-01T01:34:16.451 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:34:16.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.444+0000 7fe913e8f640 1 -- 172.21.15.70:0/1524377920 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8f80a2560 msgr2=0x7fe8f80a4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.444+0000 7fe913e8f640 1 --2- 172.21.15.70:0/1524377920 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8f80a2560 0x7fe8f80a4a20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009230 tx=0x7fe8fc009560 comp rx=0 tx=0).stop 2024-04-01T01:34:16.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.444+0000 7fe913e8f640 1 -- 172.21.15.70:0/1524377920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe90c102200 msgr2=0x7fe90c1a1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:16.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.444+0000 7fe913e8f640 1 --2- 172.21.15.70:0/1524377920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe90c102200 0x7fe90c1a1090 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fe90400eb10 tx=0x7fe90400c2d0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.444+0000 7fe913e8f640 1 -- 172.21.15.70:0/1524377920 shutdown_connections 2024-04-01T01:34:16.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.444+0000 7fe913e8f640 1 --2- 172.21.15.70:0/1524377920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe90c102200 0x7fe90c1a1090 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.444+0000 7fe913e8f640 1 --2- 172.21.15.70:0/1524377920 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8f80a2560 0x7fe8f80a4a20 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.444+0000 7fe913e8f640 1 --2- 172.21.15.70:0/1524377920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe90c101840 0x7fe90c19eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.444+0000 7fe913e8f640 1 --2- 172.21.15.70:0/1524377920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe90c100650 0x7fe90c069420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:16.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.444+0000 7fe913e8f640 1 -- 172.21.15.70:0/1524377920 >> 172.21.15.70:0/1524377920 conn(0x7fe90c0fbbe0 msgr2=0x7fe90c104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:16.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.444+0000 7fe913e8f640 1 -- 172.21.15.70:0/1524377920 shutdown_connections 2024-04-01T01:34:16.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:16.444+0000 7fe913e8f640 1 -- 172.21.15.70:0/1524377920 wait complete. 2024-04-01T01:34:16.544 INFO:teuthology.orchestra.run.smithi070.stderr:1400000 of 3200000 tuples (43%) done (elapsed 13.39 s, remaining 17.21 s) 2024-04-01T01:34:16.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:16 smithi189 ceph-mon[35753]: pgmap v382: 129 pgs: 129 active+clean; 310 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 43 MiB/s wr, 102 op/s 2024-04-01T01:34:17.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:16 smithi070 ceph-mon[32179]: pgmap v382: 129 pgs: 129 active+clean; 310 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 43 MiB/s wr, 102 op/s 2024-04-01T01:34:17.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:16 smithi081 ceph-mon[35142]: pgmap v382: 129 pgs: 129 active+clean; 310 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 43 MiB/s wr, 102 op/s 2024-04-01T01:34:17.309 INFO:teuthology.orchestra.run.smithi070.stderr:1500000 of 3200000 tuples (46%) done (elapsed 14.15 s, remaining 16.04 s) 2024-04-01T01:34:17.455 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:34:17.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:34:17.455 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:34:17.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.584+0000 7f94af22e640 1 -- 172.21.15.70:0/1856176290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94a8102210 msgr2=0x7f94a8108aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.584+0000 7f94af22e640 1 --2- 172.21.15.70:0/1856176290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94a8102210 0x7f94a8108aa0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f94a0009ca0 tx=0x7f94a002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.584+0000 7f94af22e640 1 -- 172.21.15.70:0/1856176290 shutdown_connections 2024-04-01T01:34:17.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.584+0000 7f94af22e640 1 --2- 172.21.15.70:0/1856176290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94a8102210 0x7f94a8108aa0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.584+0000 7f94af22e640 1 --2- 172.21.15.70:0/1856176290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94a8101850 0x7f94a8101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.584+0000 7f94af22e640 1 --2- 172.21.15.70:0/1856176290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a8100660 0x7f94a8100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.584+0000 7f94af22e640 1 -- 172.21.15.70:0/1856176290 >> 172.21.15.70:0/1856176290 conn(0x7f94a80fbbf0 msgr2=0x7f94a80fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:17.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.584+0000 7f94af22e640 1 -- 172.21.15.70:0/1856176290 shutdown_connections 2024-04-01T01:34:17.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.584+0000 7f94af22e640 1 -- 172.21.15.70:0/1856176290 wait complete. 2024-04-01T01:34:17.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.584+0000 7f94af22e640 1 Processor -- start 2024-04-01T01:34:17.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94af22e640 1 -- start start 2024-04-01T01:34:17.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94af22e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a8100660 0x7f94a81045f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94af22e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94a8101850 0x7f94a8102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94af22e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94a8102210 0x7f94a8103170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94af22e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f94a810c190 con 0x7f94a8102210 2024-04-01T01:34:17.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94af22e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f94a810c010 con 0x7f94a8101850 2024-04-01T01:34:17.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94af22e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f94a810c310 con 0x7f94a8100660 2024-04-01T01:34:17.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94ad7cb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94a8102210 0x7f94a8103170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94a7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94a8101850 0x7f94a8102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94ad7cb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94a8102210 0x7f94a8103170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45838/0 (socket says 172.21.15.70:45838) 2024-04-01T01:34:17.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94ad7cb640 1 -- 172.21.15.70:0/2839298952 learned_addr learned my addr 172.21.15.70:0/2839298952 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:17.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94acfca640 1 --2- 172.21.15.70:0/2839298952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a8100660 0x7f94a81045f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94ad7cb640 1 -- 172.21.15.70:0/2839298952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a8100660 msgr2=0x7f94a81045f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94ad7cb640 1 --2- 172.21.15.70:0/2839298952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a8100660 0x7f94a81045f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94ad7cb640 1 -- 172.21.15.70:0/2839298952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94a8101850 msgr2=0x7f94a8102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94ad7cb640 1 --2- 172.21.15.70:0/2839298952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94a8101850 0x7f94a8102c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94ad7cb640 1 -- 172.21.15.70:0/2839298952 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a8103a00 con 0x7f94a8102210 2024-04-01T01:34:17.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94a7fff640 1 --2- 172.21.15.70:0/2839298952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94a8101850 0x7f94a8102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:17.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94ad7cb640 1 --2- 172.21.15.70:0/2839298952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94a8102210 0x7f94a8103170 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f94a002c700 tx=0x7f94a00328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:17.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.585+0000 7f94a5ffb640 1 -- 172.21.15.70:0/2839298952 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a0041020 con 0x7f94a8102210 2024-04-01T01:34:17.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.586+0000 7f94af22e640 1 -- 172.21.15.70:0/2839298952 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94a81a1a50 con 0x7f94a8102210 2024-04-01T01:34:17.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.586+0000 7f94a5ffb640 1 -- 172.21.15.70:0/2839298952 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94a0003e20 con 0x7f94a8102210 2024-04-01T01:34:17.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.586+0000 7f94a5ffb640 1 -- 172.21.15.70:0/2839298952 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a0039040 con 0x7f94a8102210 2024-04-01T01:34:17.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.586+0000 7f94af22e640 1 -- 172.21.15.70:0/2839298952 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94a81a1fa0 con 0x7f94a8102210 2024-04-01T01:34:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94a5ffb640 1 -- 172.21.15.70:0/2839298952 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f94a0044020 con 0x7f94a8102210 2024-04-01T01:34:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94b1e12740 1 Processor -- start 2024-04-01T01:34:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94b1e12740 1 -- start start 2024-04-01T01:34:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94b1e12740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 0x55a870d51e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94b1e12740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d52400 0x55a870d56f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94b1e12740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d57570 0x55a870d59a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94b1e12740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a870c530b0 con 0x55a870d52400 2024-04-01T01:34:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94b1e12740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a870cb5ea0 con 0x55a870d57570 2024-04-01T01:34:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94b1e12740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a870c5ec00 con 0x55a870d51a10 2024-04-01T01:34:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94acfca640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d52400 0x55a870d56f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94a7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 0x55a870d51e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94acfca640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d52400 0x55a870d56f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45854/0 (socket says 172.21.15.70:45854) 2024-04-01T01:34:17.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94acfca640 1 -- 172.21.15.70:0/2925633775 learned_addr learned my addr 172.21.15.70:0/2925633775 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:17.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94a7fff640 1 --2- 172.21.15.70:0/2925633775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d57570 0x55a870d59a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94a7fff640 1 -- 172.21.15.70:0/2925633775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d57570 msgr2=0x55a870d59a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:17.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94a7fff640 1 --2- 172.21.15.70:0/2925633775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d57570 0x55a870d59a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94a7fff640 1 -- 172.21.15.70:0/2925633775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d52400 msgr2=0x55a870d56f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:17.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.589+0000 7f94a7fff640 1 --2- 172.21.15.70:0/2925633775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d52400 0x55a870d56f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.590+0000 7f94a7fff640 1 -- 172.21.15.70:0/2925633775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a870a927e0 con 0x55a870d51a10 2024-04-01T01:34:17.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.590+0000 7f94acfca640 1 --2- 172.21.15.70:0/2925633775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d52400 0x55a870d56f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:17.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.590+0000 7f94a5ffb640 1 --2- 172.21.15.70:0/2839298952 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94980a2530 0x7f94980a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.590+0000 7f94acfca640 1 --2- 172.21.15.70:0/2839298952 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94980a2530 0x7f94980a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.590+0000 7f94a5ffb640 1 -- 172.21.15.70:0/2839298952 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f94a00f0db0 con 0x7f94a8102210 2024-04-01T01:34:17.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.590+0000 7f94a7fff640 1 --2- 172.21.15.70:0/2925633775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 0x55a870d51e30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f94a8104360 tx=0x7f9494005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fae8c7aca57b1284 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:17.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.590+0000 7f9492ffd640 1 -- 172.21.15.70:0/2925633775 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9494011050 con 0x55a870d51a10 2024-04-01T01:34:17.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.590+0000 7f9492ffd640 1 -- 172.21.15.70:0/2925633775 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f949400fdf0 con 0x55a870d51a10 2024-04-01T01:34:17.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.590+0000 7f9492ffd640 1 -- 172.21.15.70:0/2925633775 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9494005620 con 0x55a870d51a10 2024-04-01T01:34:17.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.591+0000 7f94b1e12740 1 -- 172.21.15.70:0/2925633775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 msgr2=0x55a870d51e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.591+0000 7f94acfca640 1 --2- 172.21.15.70:0/2839298952 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94980a2530 0x7f94980a49f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x55a870d52800 tx=0x7f949c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:17.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.591+0000 7f94b1e12740 1 --2- 172.21.15.70:0/2925633775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 0x55a870d51e30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f94a8104360 tx=0x7f9494005850 comp rx=0 tx=0).stop 2024-04-01T01:34:17.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.591+0000 7f94b1e12740 1 -- 172.21.15.70:0/2925633775 shutdown_connections 2024-04-01T01:34:17.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.591+0000 7f94b1e12740 1 --2- 172.21.15.70:0/2925633775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d57570 0x55a870d59a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.591+0000 7f94b1e12740 1 --2- 172.21.15.70:0/2925633775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d52400 0x55a870d56f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.591+0000 7f94b1e12740 1 --2- 172.21.15.70:0/2925633775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 0x55a870d51e30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.591+0000 7f94b1e12740 1 -- 172.21.15.70:0/2925633775 >> 172.21.15.70:0/2925633775 conn(0x55a870d4cf80 msgr2=0x55a870d4d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:17.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.591+0000 7f94b1e12740 1 -- 172.21.15.70:0/2925633775 shutdown_connections 2024-04-01T01:34:17.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.591+0000 7f94b1e12740 1 -- 172.21.15.70:0/2925633775 wait complete. 2024-04-01T01:34:17.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.591+0000 7f94b1e12740 1 -- start start 2024-04-01T01:34:17.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.591+0000 7f94b1e12740 1 Processor -- start 2024-04-01T01:34:17.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94b1e12740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 0x55a870d5bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94b1e12740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d52400 0x55a870d5c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94b1e12740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d63360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94b1e12740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a870c543d0 con 0x55a870d57570 2024-04-01T01:34:17.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94b1e12740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a870cb1f40 con 0x55a870d52400 2024-04-01T01:34:17.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94b1e12740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a870d5a520 con 0x55a870d51a10 2024-04-01T01:34:17.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94b1e12740 10 client.0 _ll_register_callbacks cb 0x55a870c71b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:17.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94ad7cb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d63360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94a7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d52400 0x55a870d5c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94acfca640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 0x55a870d5bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94ad7cb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d63360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45868/0 (socket says 172.21.15.70:45868) 2024-04-01T01:34:17.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94ad7cb640 1 -- 172.21.15.70:0/1395680581 learned_addr learned my addr 172.21.15.70:0/1395680581 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:17.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94ad7cb640 1 -- 172.21.15.70:0/1395680581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 msgr2=0x55a870d5bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:17.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94ad7cb640 1 --2- 172.21.15.70:0/1395680581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 0x55a870d5bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94ad7cb640 1 -- 172.21.15.70:0/1395680581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d52400 msgr2=0x55a870d5c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:17.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94ad7cb640 1 --2- 172.21.15.70:0/1395680581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d52400 0x55a870d5c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94ad7cb640 1 -- 172.21.15.70:0/1395680581 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a870af2f70 con 0x55a870d57570 2024-04-01T01:34:17.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94acfca640 1 --2- 172.21.15.70:0/1395680581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 0x55a870d5bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:17.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94a7fff640 1 --2- 172.21.15.70:0/1395680581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d52400 0x55a870d5c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:17.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94ad7cb640 1 --2- 172.21.15.70:0/1395680581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d63360 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f94a002c6d0 tx=0x7f94a00bd310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0b33bb34b74eca server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:17.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f9487fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:17.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f9487fff640 1 -- 172.21.15.70:0/1395680581 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a00bdb50 con 0x55a870d57570 2024-04-01T01:34:17.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f9487fff640 1 -- 172.21.15.70:0/1395680581 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f94a00c1170 con 0x55a870d57570 2024-04-01T01:34:17.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.592+0000 7f94b1e12740 1 -- 172.21.15.70:0/1395680581 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f949400fdf0 con 0x55a870d57570 2024-04-01T01:34:17.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.593+0000 7f9487fff640 1 -- 172.21.15.70:0/1395680581 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f94a00c1370 con 0x55a870d57570 2024-04-01T01:34:17.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.593+0000 7f94b1e12740 10 client.15855 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:17.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.593+0000 7f94b1e12740 1 -- 172.21.15.70:0/1395680581 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a87098a170 con 0x55a870d57570 2024-04-01T01:34:17.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.593+0000 7f9487fff640 1 -- 172.21.15.70:0/1395680581 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f949400fdf0 con 0x55a870d57570 2024-04-01T01:34:17.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.593+0000 7f9487fff640 1 -- 172.21.15.70:0/1395680581 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f94a003c030 con 0x55a870d57570 2024-04-01T01:34:17.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.593+0000 7f94b1e12740 10 client.15855 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:17.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.593+0000 7f94b1e12740 10 client.15855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:17.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.593+0000 7f94b1e12740 20 client.15855 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:17.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.593+0000 7f94b1e12740 1 --2- 172.21.15.70:0/1395680581 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a870d67720 0x55a870d69b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.593+0000 7f94b1e12740 4 client.15855 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:17.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.593+0000 7f94b1e12740 1 -- 172.21.15.70:0/1395680581 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a870cc33e0 con 0x55a870d67720 2024-04-01T01:34:17.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.593+0000 7f94acfca640 1 --2- 172.21.15.70:0/1395680581 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a870d67720 0x55a870d69b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.594+0000 7f9487fff640 10 client.15855 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:17.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.594+0000 7f94acfca640 1 --2- 172.21.15.70:0/1395680581 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a870d67720 0x55a870d69b50 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eecb098aec96aa42 server_cookie=4a8408fc9ac52185 in_seq=0 out_seq=0 2024-04-01T01:34:17.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.594+0000 7f9487fff640 1 -- 172.21.15.70:0/1395680581 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a870cc33e0 con 0x55a870d67720 2024-04-01T01:34:17.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.594+0000 7f9487fff640 10 client.15855 handle_command_reply: tid=0 2024-04-01T01:34:17.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.594+0000 7f94b1e12740 1 client.15855 shutdown 2024-04-01T01:34:17.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.594+0000 7f94b1e12740 1 -- 172.21.15.70:0/1395680581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 msgr2=0x55a870d63360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.594+0000 7f94b1e12740 1 --2- 172.21.15.70:0/1395680581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d63360 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f94a002c6d0 tx=0x7f94a00bd310 comp rx=0 tx=0).stop 2024-04-01T01:34:17.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.594+0000 7f94b1e12740 1 -- 172.21.15.70:0/1395680581 shutdown_connections 2024-04-01T01:34:17.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.594+0000 7f94b1e12740 1 --2- 172.21.15.70:0/1395680581 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a870d67720 0x55a870d69b50 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.594+0000 7f94b1e12740 1 --2- 172.21.15.70:0/1395680581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d63360 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.594+0000 7f9487fff640 0 client.15855 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:17.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.594+0000 7f94b1e12740 1 --2- 172.21.15.70:0/1395680581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d52400 0x55a870d5c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.595+0000 7f94b1e12740 1 --2- 172.21.15.70:0/1395680581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 0x55a870d5bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.595+0000 7f94b1e12740 1 -- 172.21.15.70:0/1395680581 >> 172.21.15.70:0/1395680581 conn(0x55a870d4cf80 msgr2=0x55a870d4c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:17.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.595+0000 7f94b1e12740 1 -- 172.21.15.70:0/1395680581 shutdown_connections 2024-04-01T01:34:17.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.595+0000 7f94b1e12740 1 -- 172.21.15.70:0/1395680581 wait complete. 2024-04-01T01:34:17.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.595+0000 7f94b1e12740 20 client.15855 trim_cache size 0 max 16384 2024-04-01T01:34:17.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.617+0000 7f94b1e12740 1 Processor -- start 2024-04-01T01:34:17.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.617+0000 7f94b1e12740 1 -- start start 2024-04-01T01:34:17.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94b1e12740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 0x55a870d5a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94b1e12740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d52400 0x55a870d63320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94b1e12740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d5ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94b1e12740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f94a81ac600 con 0x55a870d57570 2024-04-01T01:34:17.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94b1e12740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f94a81ae140 con 0x55a870d52400 2024-04-01T01:34:17.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94b1e12740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f94a81accd0 con 0x55a870d51a10 2024-04-01T01:34:17.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94ad7cb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d5ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94ad7cb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d5ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45874/0 (socket says 172.21.15.70:45874) 2024-04-01T01:34:17.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94ad7cb640 1 -- 172.21.15.70:0/3337671136 learned_addr learned my addr 172.21.15.70:0/3337671136 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:17.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94a7fff640 1 --2- 172.21.15.70:0/3337671136 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d52400 0x55a870d63320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94acfca640 1 --2- 172.21.15.70:0/3337671136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 0x55a870d5a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94a7fff640 1 -- 172.21.15.70:0/3337671136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 msgr2=0x55a870d5a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:17.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94a7fff640 1 --2- 172.21.15.70:0/3337671136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 0x55a870d5a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94a7fff640 1 -- 172.21.15.70:0/3337671136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 msgr2=0x55a870d5ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:17.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94a7fff640 1 --2- 172.21.15.70:0/3337671136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d5ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94a7fff640 1 -- 172.21.15.70:0/3337671136 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a870d4d7c0 con 0x55a870d52400 2024-04-01T01:34:17.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94ad7cb640 1 --2- 172.21.15.70:0/3337671136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d5ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:17.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.618+0000 7f94a7fff640 1 --2- 172.21.15.70:0/3337671136 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d52400 0x55a870d63320 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9494009b70 tx=0x7f94940096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9aaf797701e36b9c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:17.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.619+0000 7f94927fc640 1 -- 172.21.15.70:0/3337671136 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9494004020 con 0x55a870d52400 2024-04-01T01:34:17.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.619+0000 7f94927fc640 1 -- 172.21.15.70:0/3337671136 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f949400fdf0 con 0x55a870d52400 2024-04-01T01:34:17.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.619+0000 7f94927fc640 1 -- 172.21.15.70:0/3337671136 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9494003c50 con 0x55a870d52400 2024-04-01T01:34:17.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.619+0000 7f94b1e12740 1 -- 172.21.15.70:0/3337671136 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d52400 msgr2=0x55a870d63320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.619+0000 7f94b1e12740 1 --2- 172.21.15.70:0/3337671136 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d52400 0x55a870d63320 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9494009b70 tx=0x7f94940096c0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.619+0000 7f94b1e12740 1 -- 172.21.15.70:0/3337671136 shutdown_connections 2024-04-01T01:34:17.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.619+0000 7f94b1e12740 1 --2- 172.21.15.70:0/3337671136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d5ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.619+0000 7f94b1e12740 1 --2- 172.21.15.70:0/3337671136 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d52400 0x55a870d63320 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.619+0000 7f94b1e12740 1 --2- 172.21.15.70:0/3337671136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d51a10 0x55a870d5a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.619+0000 7f94b1e12740 1 -- 172.21.15.70:0/3337671136 >> 172.21.15.70:0/3337671136 conn(0x55a870d4cf80 msgr2=0x55a870d50a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:17.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.619+0000 7f94b1e12740 1 -- 172.21.15.70:0/3337671136 shutdown_connections 2024-04-01T01:34:17.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.619+0000 7f94b1e12740 1 -- 172.21.15.70:0/3337671136 wait complete. 2024-04-01T01:34:17.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.620+0000 7f94b1e12740 1 -- start start 2024-04-01T01:34:17.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.620+0000 7f94b1e12740 1 Processor -- start 2024-04-01T01:34:17.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.620+0000 7f94b1e12740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d51a10 0x55a870d60ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.620+0000 7f94b1e12740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d52400 0x55a870d61430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.620+0000 7f94b1e12740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d62440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.620+0000 7f94b1e12740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f94a81ada70 con 0x55a870d57570 2024-04-01T01:34:17.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.620+0000 7f94b1e12740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f94a81ad3a0 con 0x55a870d51a10 2024-04-01T01:34:17.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.620+0000 7f94b1e12740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f94a81aa5e0 con 0x55a870d52400 2024-04-01T01:34:17.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.620+0000 7f94b1e12740 10 client.0 _ll_register_callbacks cb 0x7f94a00b97c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:17.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.620+0000 7f94acfca640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d51a10 0x55a870d60ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.620+0000 7f94ad7cb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d62440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.620+0000 7f94acfca640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d51a10 0x55a870d60ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37956/0 (socket says 172.21.15.70:37956) 2024-04-01T01:34:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.620+0000 7f94acfca640 1 -- 172.21.15.70:0/2310523361 learned_addr learned my addr 172.21.15.70:0/2310523361 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.620+0000 7f94a7fff640 1 --2- 172.21.15.70:0/2310523361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d52400 0x55a870d61430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.621+0000 7f94acfca640 1 -- 172.21.15.70:0/2310523361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d52400 msgr2=0x55a870d61430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.621+0000 7f94acfca640 1 --2- 172.21.15.70:0/2310523361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d52400 0x55a870d61430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.621+0000 7f94acfca640 1 -- 172.21.15.70:0/2310523361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 msgr2=0x55a870d62440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.621+0000 7f94acfca640 1 --2- 172.21.15.70:0/2310523361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d62440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.621+0000 7f94acfca640 1 -- 172.21.15.70:0/2310523361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a870d62a30 con 0x55a870d51a10 2024-04-01T01:34:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.621+0000 7f94ad7cb640 1 --2- 172.21.15.70:0/2310523361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d62440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.621+0000 7f94acfca640 1 --2- 172.21.15.70:0/2310523361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d51a10 0x55a870d60ef0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f949c005fd0 tx=0x7f949c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54030451d416927a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.621+0000 7f9487fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:34:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.621+0000 7f9487fff640 1 -- 172.21.15.70:0/2310523361 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949c02ab90 con 0x55a870d51a10 2024-04-01T01:34:17.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.621+0000 7f94b1e12740 1 -- 172.21.15.70:0/2310523361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f949400fdf0 con 0x55a870d51a10 2024-04-01T01:34:17.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.621+0000 7f9487fff640 1 -- 172.21.15.70:0/2310523361 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f949c02ad30 con 0x55a870d51a10 2024-04-01T01:34:17.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.621+0000 7f9487fff640 1 -- 172.21.15.70:0/2310523361 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f949c02a380 con 0x55a870d51a10 2024-04-01T01:34:17.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.621+0000 7f94b1e12740 10 client.25676 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:17.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.621+0000 7f94b1e12740 1 -- 172.21.15.70:0/2310523361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a870dcdeb0 con 0x55a870d51a10 2024-04-01T01:34:17.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.621+0000 7f9487fff640 1 -- 172.21.15.70:0/2310523361 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f949400fdf0 con 0x55a870d51a10 2024-04-01T01:34:17.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.622+0000 7f9487fff640 1 -- 172.21.15.70:0/2310523361 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f949c032b10 con 0x55a870d51a10 2024-04-01T01:34:17.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.622+0000 7f94b1e12740 10 client.25676 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:17.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.622+0000 7f94b1e12740 10 client.25676 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:17.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.622+0000 7f94b1e12740 20 client.25676 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:17.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.622+0000 7f94b1e12740 1 --2- 172.21.15.70:0/2310523361 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a870dce890 0x55a870dd0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.622+0000 7f94b1e12740 4 client.25676 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:34:17.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.622+0000 7f94b1e12740 1 -- 172.21.15.70:0/2310523361 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a870cbd730 con 0x55a870dce890 2024-04-01T01:34:17.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.622+0000 7f94a7fff640 1 --2- 172.21.15.70:0/2310523361 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a870dce890 0x55a870dd0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.622+0000 7f9487fff640 10 client.25676 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:17.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.622+0000 7f94a7fff640 1 --2- 172.21.15.70:0/2310523361 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a870dce890 0x55a870dd0cb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a746e3e0211c3786 server_cookie=363211dcc99dfbc6 in_seq=0 out_seq=0 2024-04-01T01:34:17.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.623+0000 7f9487fff640 1 -- 172.21.15.70:0/2310523361 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a870cbd730 con 0x55a870dce890 2024-04-01T01:34:17.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.623+0000 7f9487fff640 10 client.25676 handle_command_reply: tid=0 2024-04-01T01:34:17.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.623+0000 7f94b1e12740 1 client.25676 shutdown 2024-04-01T01:34:17.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.623+0000 7f94b1e12740 1 -- 172.21.15.70:0/2310523361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d51a10 msgr2=0x55a870d60ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.623+0000 7f94b1e12740 1 --2- 172.21.15.70:0/2310523361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d51a10 0x55a870d60ef0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f949c005fd0 tx=0x7f949c016000 comp rx=0 tx=0).stop 2024-04-01T01:34:17.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.623+0000 7f94b1e12740 1 -- 172.21.15.70:0/2310523361 shutdown_connections 2024-04-01T01:34:17.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.623+0000 7f94b1e12740 1 --2- 172.21.15.70:0/2310523361 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a870dce890 0x55a870dd0cb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.623+0000 7f94b1e12740 1 --2- 172.21.15.70:0/2310523361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a870d57570 0x55a870d62440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.623+0000 7f94b1e12740 1 --2- 172.21.15.70:0/2310523361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a870d52400 0x55a870d61430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.623+0000 7f9487fff640 0 client.25676 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:17.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.623+0000 7f94b1e12740 1 --2- 172.21.15.70:0/2310523361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a870d51a10 0x55a870d60ef0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.623+0000 7f94b1e12740 1 -- 172.21.15.70:0/2310523361 >> 172.21.15.70:0/2310523361 conn(0x55a870d4cf80 msgr2=0x55a870d4b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:17.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.624+0000 7f94b1e12740 1 -- 172.21.15.70:0/2310523361 shutdown_connections 2024-04-01T01:34:17.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.624+0000 7f94b1e12740 1 -- 172.21.15.70:0/2310523361 wait complete. 2024-04-01T01:34:17.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.624+0000 7f94b1e12740 20 client.25676 trim_cache size 0 max 16384 2024-04-01T01:34:17.631 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:34:17.631 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:34:17.631 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "f2d04175-bcbd-4182-8e27-f28d7070e1fb", 2024-04-01T01:34:17.631 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:34:17.631 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:34:17.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.624+0000 7f94af22e640 1 -- 172.21.15.70:0/2839298952 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94980a2530 msgr2=0x7f94980a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.624+0000 7f94af22e640 1 --2- 172.21.15.70:0/2839298952 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94980a2530 0x7f94980a49f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x55a870d52800 tx=0x7f949c00b390 comp rx=0 tx=0).stop 2024-04-01T01:34:17.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.624+0000 7f94af22e640 1 -- 172.21.15.70:0/2839298952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94a8102210 msgr2=0x7f94a8103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.624+0000 7f94af22e640 1 --2- 172.21.15.70:0/2839298952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94a8102210 0x7f94a8103170 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f94a002c700 tx=0x7f94a00328a0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.624+0000 7f94af22e640 1 -- 172.21.15.70:0/2839298952 shutdown_connections 2024-04-01T01:34:17.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.624+0000 7f94af22e640 1 --2- 172.21.15.70:0/2839298952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94a8102210 0x7f94a8103170 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.624+0000 7f94af22e640 1 --2- 172.21.15.70:0/2839298952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94a8101850 0x7f94a8102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.624+0000 7f94af22e640 1 --2- 172.21.15.70:0/2839298952 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f94980a2530 0x7f94980a49f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.624+0000 7f94af22e640 1 --2- 172.21.15.70:0/2839298952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a8100660 0x7f94a81045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.624+0000 7f94af22e640 1 -- 172.21.15.70:0/2839298952 >> 172.21.15.70:0/2839298952 conn(0x7f94a80fbbf0 msgr2=0x7f94a80fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:17.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.625+0000 7f94af22e640 1 -- 172.21.15.70:0/2839298952 shutdown_connections 2024-04-01T01:34:17.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.625+0000 7f94af22e640 1 -- 172.21.15.70:0/2839298952 wait complete. 2024-04-01T01:34:17.637 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:34:17.637 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:34:17.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.774+0000 7fac1283b640 1 -- 172.21.15.70:0/328849650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0c104ba0 msgr2=0x7fac0c106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.774+0000 7fac1283b640 1 --2- 172.21.15.70:0/328849650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0c104ba0 0x7fac0c106f90 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fac00009900 tx=0x7fac0002bc30 comp rx=0 tx=0).stop 2024-04-01T01:34:17.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.774+0000 7fac1283b640 1 -- 172.21.15.70:0/328849650 shutdown_connections 2024-04-01T01:34:17.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.774+0000 7fac1283b640 1 --2- 172.21.15.70:0/328849650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac0c1074d0 0x7fac0c1098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.774+0000 7fac1283b640 1 --2- 172.21.15.70:0/328849650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0c104ba0 0x7fac0c106f90 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.774+0000 7fac1283b640 1 --2- 172.21.15.70:0/328849650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac0c102270 0x7fac0c104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.774+0000 7fac1283b640 1 -- 172.21.15.70:0/328849650 >> 172.21.15.70:0/328849650 conn(0x7fac0c0fbbe0 msgr2=0x7fac0c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:17.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.774+0000 7fac1283b640 1 -- 172.21.15.70:0/328849650 shutdown_connections 2024-04-01T01:34:17.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.774+0000 7fac1283b640 1 -- 172.21.15.70:0/328849650 wait complete. 2024-04-01T01:34:17.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.774+0000 7fac1283b640 1 Processor -- start 2024-04-01T01:34:17.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.774+0000 7fac1283b640 1 -- start start 2024-04-01T01:34:17.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac1283b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0c102270 0x7fac0c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac1283b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac0c104ba0 0x7fac0c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac1283b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac0c1074d0 0x7fac0c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac1283b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c10d270 con 0x7fac0c102270 2024-04-01T01:34:17.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac1283b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c10d0f0 con 0x7fac0c104ba0 2024-04-01T01:34:17.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac1283b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c10d3f0 con 0x7fac0c1074d0 2024-04-01T01:34:17.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0c102270 0x7fac0c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac0b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac0c104ba0 0x7fac0c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac10dd8640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac0c1074d0 0x7fac0c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0c102270 0x7fac0c197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45902/0 (socket says 172.21.15.70:45902) 2024-04-01T01:34:17.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac0bfff640 1 -- 172.21.15.70:0/2498595936 learned_addr learned my addr 172.21.15.70:0/2498595936 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:17.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac0bfff640 1 -- 172.21.15.70:0/2498595936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac0c1074d0 msgr2=0x7fac0c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac0bfff640 1 --2- 172.21.15.70:0/2498595936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac0c1074d0 0x7fac0c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac0bfff640 1 -- 172.21.15.70:0/2498595936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac0c104ba0 msgr2=0x7fac0c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac0bfff640 1 --2- 172.21.15.70:0/2498595936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac0c104ba0 0x7fac0c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac0bfff640 1 -- 172.21.15.70:0/2498595936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac0c19f650 con 0x7fac0c102270 2024-04-01T01:34:17.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac0b7fe640 1 --2- 172.21.15.70:0/2498595936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac0c104ba0 0x7fac0c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:17.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac10dd8640 1 --2- 172.21.15.70:0/2498595936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac0c1074d0 0x7fac0c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:17.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.775+0000 7fac0bfff640 1 --2- 172.21.15.70:0/2498595936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0c102270 0x7fac0c197a30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fabfc00e9f0 tx=0x7fabfc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:17.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.776+0000 7fac097fa640 1 -- 172.21.15.70:0/2498595936 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc00ccc0 con 0x7fac0c102270 2024-04-01T01:34:17.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.776+0000 7fac1283b640 1 -- 172.21.15.70:0/2498595936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac0c19f940 con 0x7fac0c102270 2024-04-01T01:34:17.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.776+0000 7fac097fa640 1 -- 172.21.15.70:0/2498595936 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabfc00ce60 con 0x7fac0c102270 2024-04-01T01:34:17.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.776+0000 7fac097fa640 1 -- 172.21.15.70:0/2498595936 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc010430 con 0x7fac0c102270 2024-04-01T01:34:17.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.776+0000 7fac1283b640 1 -- 172.21.15.70:0/2498595936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac0c19fe10 con 0x7fac0c102270 2024-04-01T01:34:17.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.778+0000 7fac1541b740 1 Processor -- start 2024-04-01T01:34:17.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.778+0000 7fac1541b740 1 -- start start 2024-04-01T01:34:17.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.778+0000 7fac097fa640 1 -- 172.21.15.70:0/2498595936 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7fabfc005020 con 0x7fac0c102270 2024-04-01T01:34:17.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.778+0000 7fac1541b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acb2b0 0x561cc1acb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.778+0000 7fac1541b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1acbca0 0x561cc1ad07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.778+0000 7fac1541b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1ad0e10 0x561cc1ad32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.778+0000 7fac1541b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561cc1a30180 con 0x561cc1acbca0 2024-04-01T01:34:17.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.778+0000 7fac1541b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561cc19d8b70 con 0x561cc1ad0e10 2024-04-01T01:34:17.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.778+0000 7fac1541b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561cc1a2c220 con 0x561cc1acb2b0 2024-04-01T01:34:17.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1acbca0 0x561cc1ad07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fac0b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1ad0e10 0x561cc1ad32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1acbca0 0x561cc1ad07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45910/0 (socket says 172.21.15.70:45910) 2024-04-01T01:34:17.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fac0b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1ad0e10 0x561cc1ad32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37980/0 (socket says 172.21.15.70:37980) 2024-04-01T01:34:17.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fac0bfff640 1 -- 172.21.15.70:0/15423154 learned_addr learned my addr 172.21.15.70:0/15423154 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:17.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fac10dd8640 1 --2- 172.21.15.70:0/15423154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acb2b0 0x561cc1acb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fac0b7fe640 1 -- 172.21.15.70:0/15423154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acb2b0 msgr2=0x561cc1acb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:17.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fac0b7fe640 1 --2- 172.21.15.70:0/15423154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acb2b0 0x561cc1acb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fac0b7fe640 1 -- 172.21.15.70:0/15423154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1acbca0 msgr2=0x561cc1ad07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:17.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fac0b7fe640 1 --2- 172.21.15.70:0/15423154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1acbca0 0x561cc1ad07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fac0b7fe640 1 -- 172.21.15.70:0/15423154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cc19ec7c0 con 0x561cc1ad0e10 2024-04-01T01:34:17.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fac097fa640 1 --2- 172.21.15.70:0/2498595936 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fabf40a2530 0x7fabf40a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fac0bfff640 1 --2- 172.21.15.70:0/15423154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1acbca0 0x561cc1ad07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:17.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fac10dd8640 1 --2- 172.21.15.70:0/2498595936 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fabf40a2530 0x7fabf40a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fac097fa640 1 -- 172.21.15.70:0/2498595936 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fabfc0cf060 con 0x7fac0c102270 2024-04-01T01:34:17.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fac0b7fe640 1 --2- 172.21.15.70:0/15423154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1ad0e10 0x561cc1ad32a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fac00036040 tx=0x7fac00009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c69864a5639a1e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:17.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fabf27fc640 1 -- 172.21.15.70:0/15423154 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac00045020 con 0x561cc1ad0e10 2024-04-01T01:34:17.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fabf27fc640 1 -- 172.21.15.70:0/15423154 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac0002cc90 con 0x561cc1ad0e10 2024-04-01T01:34:17.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.779+0000 7fabf27fc640 1 -- 172.21.15.70:0/15423154 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac00004920 con 0x561cc1ad0e10 2024-04-01T01:34:17.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.780+0000 7fac1541b740 1 -- 172.21.15.70:0/15423154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1ad0e10 msgr2=0x561cc1ad32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.780+0000 7fac10dd8640 1 --2- 172.21.15.70:0/2498595936 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fabf40a2530 0x7fabf40a49f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x561cc1acc500 tx=0x7fac04009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:17.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.780+0000 7fac1541b740 1 --2- 172.21.15.70:0/15423154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1ad0e10 0x561cc1ad32a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fac00036040 tx=0x7fac00009580 comp rx=0 tx=0).stop 2024-04-01T01:34:17.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.780+0000 7fac1541b740 1 -- 172.21.15.70:0/15423154 shutdown_connections 2024-04-01T01:34:17.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.780+0000 7fac1541b740 1 --2- 172.21.15.70:0/15423154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1ad0e10 0x561cc1ad32a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.780+0000 7fac1541b740 1 --2- 172.21.15.70:0/15423154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1acbca0 0x561cc1ad07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.780+0000 7fac1541b740 1 --2- 172.21.15.70:0/15423154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acb2b0 0x561cc1acb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.780+0000 7fac1541b740 1 -- 172.21.15.70:0/15423154 >> 172.21.15.70:0/15423154 conn(0x561cc1ac6820 msgr2=0x561cc1ac6c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:17.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.780+0000 7fac1541b740 1 -- 172.21.15.70:0/15423154 shutdown_connections 2024-04-01T01:34:17.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.780+0000 7fac1541b740 1 -- 172.21.15.70:0/15423154 wait complete. 2024-04-01T01:34:17.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.781+0000 7fac1541b740 1 -- start start 2024-04-01T01:34:17.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.781+0000 7fac1541b740 1 Processor -- start 2024-04-01T01:34:17.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.781+0000 7fac1541b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acb2b0 0x561cc1ad5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.781+0000 7fac1541b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acbca0 0x561cc1ad5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.781+0000 7fac1541b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1ad0e10 0x561cc1adcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.781+0000 7fac1541b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561cc1ad3d60 con 0x561cc1ad0e10 2024-04-01T01:34:17.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.781+0000 7fac1541b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561cc19ce4a0 con 0x561cc1acbca0 2024-04-01T01:34:17.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.781+0000 7fac1541b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561cc1ad3ee0 con 0x561cc1acb2b0 2024-04-01T01:34:17.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.781+0000 7fac1541b740 10 client.0 _ll_register_callbacks cb 0x561cc19d1d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:17.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.781+0000 7fac0b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acbca0 0x561cc1ad5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.781+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acb2b0 0x561cc1ad5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.781+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acb2b0 0x561cc1ad5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36138/0 (socket says 172.21.15.70:36138) 2024-04-01T01:34:17.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.781+0000 7fac0bfff640 1 -- 172.21.15.70:0/3155527515 learned_addr learned my addr 172.21.15.70:0/3155527515 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:17.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.782+0000 7fac10dd8640 1 --2- 172.21.15.70:0/3155527515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1ad0e10 0x561cc1adcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.782+0000 7fac0bfff640 1 -- 172.21.15.70:0/3155527515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acbca0 msgr2=0x561cc1ad5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:17.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.782+0000 7fac0bfff640 1 --2- 172.21.15.70:0/3155527515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acbca0 0x561cc1ad5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.782+0000 7fac0bfff640 1 -- 172.21.15.70:0/3155527515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1ad0e10 msgr2=0x561cc1adcce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:17.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.782+0000 7fac0bfff640 1 --2- 172.21.15.70:0/3155527515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1ad0e10 0x561cc1adcce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.782+0000 7fac0bfff640 1 -- 172.21.15.70:0/3155527515 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cc180e7d0 con 0x561cc1acb2b0 2024-04-01T01:34:17.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.782+0000 7fac0bfff640 1 --2- 172.21.15.70:0/3155527515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acb2b0 0x561cc1ad5760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fabfc097fc0 tx=0x7fabfc004990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32009a54c327d8d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:17.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.782+0000 7fabeb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:34:17.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.782+0000 7fabeb7fe640 1 -- 172.21.15.70:0/3155527515 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc09b3c0 con 0x561cc1acb2b0 2024-04-01T01:34:17.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.783+0000 7fabeb7fe640 1 -- 172.21.15.70:0/3155527515 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fabfc09f470 con 0x561cc1acb2b0 2024-04-01T01:34:17.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.783+0000 7fac1541b740 1 -- 172.21.15.70:0/3155527515 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac0002cc90 con 0x561cc1acb2b0 2024-04-01T01:34:17.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.783+0000 7fabeb7fe640 1 -- 172.21.15.70:0/3155527515 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fabfc09af50 con 0x561cc1acb2b0 2024-04-01T01:34:17.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.783+0000 7fac1541b740 10 client.25606 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:17.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.783+0000 7fac1541b740 1 -- 172.21.15.70:0/3155527515 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561cc17443f0 con 0x561cc1acb2b0 2024-04-01T01:34:17.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.784+0000 7fabeb7fe640 1 -- 172.21.15.70:0/3155527515 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabfc09bcf0 con 0x561cc1acb2b0 2024-04-01T01:34:17.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.784+0000 7fabeb7fe640 1 -- 172.21.15.70:0/3155527515 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fabfc005310 con 0x561cc1acb2b0 2024-04-01T01:34:17.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.784+0000 7fac1541b740 10 client.25606 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:17.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.784+0000 7fac1541b740 10 client.25606 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:17.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.784+0000 7fac1541b740 20 client.25606 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:17.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.784+0000 7fac1541b740 1 --2- 172.21.15.70:0/3155527515 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561cc1ae16a0 0x561cc1ae3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.784+0000 7fac1541b740 4 client.25606 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:17.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.784+0000 7fac1541b740 1 -- 172.21.15.70:0/3155527515 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fac0002cc90 con 0x561cc1ae16a0 2024-04-01T01:34:17.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.785+0000 7fac0b7fe640 1 --2- 172.21.15.70:0/3155527515 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561cc1ae16a0 0x561cc1ae3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.785+0000 7fabeb7fe640 10 client.25606 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:17.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.785+0000 7fac0b7fe640 1 --2- 172.21.15.70:0/3155527515 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561cc1ae16a0 0x561cc1ae3b10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e31bc4dedbc87ee8 server_cookie=10906b17308c0c4a in_seq=0 out_seq=0 2024-04-01T01:34:17.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.786+0000 7fabeb7fe640 1 -- 172.21.15.70:0/3155527515 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fac0002cc90 con 0x561cc1ae16a0 2024-04-01T01:34:17.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.786+0000 7fabeb7fe640 10 client.25606 handle_command_reply: tid=0 2024-04-01T01:34:17.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.786+0000 7fac1541b740 1 client.25606 shutdown 2024-04-01T01:34:17.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.786+0000 7fac1541b740 1 -- 172.21.15.70:0/3155527515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acb2b0 msgr2=0x561cc1ad5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.786+0000 7fac1541b740 1 --2- 172.21.15.70:0/3155527515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acb2b0 0x561cc1ad5760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fabfc097fc0 tx=0x7fabfc004990 comp rx=0 tx=0).stop 2024-04-01T01:34:17.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.787+0000 7fac1541b740 1 -- 172.21.15.70:0/3155527515 shutdown_connections 2024-04-01T01:34:17.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.787+0000 7fac1541b740 1 --2- 172.21.15.70:0/3155527515 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561cc1ae16a0 0x561cc1ae3b10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.787+0000 7fac1541b740 1 --2- 172.21.15.70:0/3155527515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1ad0e10 0x561cc1adcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.787+0000 7fac1541b740 1 --2- 172.21.15.70:0/3155527515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acbca0 0x561cc1ad5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.787+0000 7fac1541b740 1 --2- 172.21.15.70:0/3155527515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acb2b0 0x561cc1ad5760 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.787+0000 7fac1541b740 1 -- 172.21.15.70:0/3155527515 >> 172.21.15.70:0/3155527515 conn(0x561cc1ac6820 msgr2=0x561cc1ac5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:17.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.787+0000 7fabeb7fe640 0 client.25606 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:17.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.787+0000 7fac1541b740 1 -- 172.21.15.70:0/3155527515 shutdown_connections 2024-04-01T01:34:17.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.787+0000 7fac1541b740 1 -- 172.21.15.70:0/3155527515 wait complete. 2024-04-01T01:34:17.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.787+0000 7fac1541b740 20 client.25606 trim_cache size 0 max 16384 2024-04-01T01:34:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.814+0000 7fac1541b740 1 Processor -- start 2024-04-01T01:34:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac1541b740 1 -- start start 2024-04-01T01:34:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac1541b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acb2b0 0x561cc1ac6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac1541b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acbca0 0x561cc1adcc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac1541b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1ad0e10 0x561cc1ad5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac1541b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1a80f0 con 0x561cc1ad0e10 2024-04-01T01:34:17.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac1541b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1a9c30 con 0x561cc1acb2b0 2024-04-01T01:34:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac1541b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1a87c0 con 0x561cc1acbca0 2024-04-01T01:34:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac10dd8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1ad0e10 0x561cc1ad5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac0b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acbca0 0x561cc1adcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac10dd8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1ad0e10 0x561cc1ad5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45928/0 (socket says 172.21.15.70:45928) 2024-04-01T01:34:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acb2b0 0x561cc1ac6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac0b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acbca0 0x561cc1adcc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36148/0 (socket says 172.21.15.70:36148) 2024-04-01T01:34:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac10dd8640 1 -- 172.21.15.70:0/334270657 learned_addr learned my addr 172.21.15.70:0/334270657 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac0bfff640 1 -- 172.21.15.70:0/334270657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acbca0 msgr2=0x561cc1adcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac0bfff640 1 --2- 172.21.15.70:0/334270657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acbca0 0x561cc1adcc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac0bfff640 1 -- 172.21.15.70:0/334270657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1ad0e10 msgr2=0x561cc1ad5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac0bfff640 1 --2- 172.21.15.70:0/334270657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1ad0e10 0x561cc1ad5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac0bfff640 1 -- 172.21.15.70:0/334270657 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cc1acd880 con 0x561cc1acb2b0 2024-04-01T01:34:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.815+0000 7fac10dd8640 1 --2- 172.21.15.70:0/334270657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1ad0e10 0x561cc1ad5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:17.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.816+0000 7fac0bfff640 1 --2- 172.21.15.70:0/334270657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acb2b0 0x561cc1ac6c00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0998a0 tx=0x7fabfc086170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dcea53dbc4b92af server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:17.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.816+0000 7fabf1ffb640 1 -- 172.21.15.70:0/334270657 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc09b130 con 0x561cc1acb2b0 2024-04-01T01:34:17.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.816+0000 7fabf1ffb640 1 -- 172.21.15.70:0/334270657 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabfc09bcf0 con 0x561cc1acb2b0 2024-04-01T01:34:17.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.816+0000 7fabf1ffb640 1 -- 172.21.15.70:0/334270657 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc09f740 con 0x561cc1acb2b0 2024-04-01T01:34:17.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.816+0000 7fac1541b740 1 -- 172.21.15.70:0/334270657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acb2b0 msgr2=0x561cc1ac6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.816+0000 7fac1541b740 1 --2- 172.21.15.70:0/334270657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acb2b0 0x561cc1ac6c00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0998a0 tx=0x7fabfc086170 comp rx=0 tx=0).stop 2024-04-01T01:34:17.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.817+0000 7fac1541b740 1 -- 172.21.15.70:0/334270657 shutdown_connections 2024-04-01T01:34:17.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.817+0000 7fac1541b740 1 --2- 172.21.15.70:0/334270657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1ad0e10 0x561cc1ad5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.817+0000 7fac1541b740 1 --2- 172.21.15.70:0/334270657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1acbca0 0x561cc1adcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.817+0000 7fac1541b740 1 --2- 172.21.15.70:0/334270657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acb2b0 0x561cc1ac6c00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.817+0000 7fac1541b740 1 -- 172.21.15.70:0/334270657 >> 172.21.15.70:0/334270657 conn(0x561cc1ac6820 msgr2=0x561cc1ad5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:17.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.817+0000 7fac1541b740 1 -- 172.21.15.70:0/334270657 shutdown_connections 2024-04-01T01:34:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.817+0000 7fac1541b740 1 -- 172.21.15.70:0/334270657 wait complete. 2024-04-01T01:34:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.817+0000 7fac1541b740 1 -- start start 2024-04-01T01:34:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.817+0000 7fac1541b740 1 Processor -- start 2024-04-01T01:34:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.817+0000 7fac1541b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acb2b0 0x561cc1ada520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.817+0000 7fac1541b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1acbca0 0x561cc1adaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.817+0000 7fac1541b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1ad0e10 0x561cc1adba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.817+0000 7fac1541b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1a9560 con 0x561cc1acbca0 2024-04-01T01:34:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.817+0000 7fac1541b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1a8e90 con 0x561cc1acb2b0 2024-04-01T01:34:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.817+0000 7fac1541b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c1a60d0 con 0x561cc1ad0e10 2024-04-01T01:34:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.817+0000 7fac1541b740 10 client.0 _ll_register_callbacks cb 0x7fabfc00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.818+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acb2b0 0x561cc1ada520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.818+0000 7fac0b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1acbca0 0x561cc1adaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.818+0000 7fac0bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acb2b0 0x561cc1ada520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38000/0 (socket says 172.21.15.70:38000) 2024-04-01T01:34:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.818+0000 7fac0bfff640 1 -- 172.21.15.70:0/2892504968 learned_addr learned my addr 172.21.15.70:0/2892504968 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.818+0000 7fac0bfff640 1 -- 172.21.15.70:0/2892504968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1ad0e10 msgr2=0x561cc1adba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.818+0000 7fac0bfff640 1 --2- 172.21.15.70:0/2892504968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1ad0e10 0x561cc1adba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.818+0000 7fac0bfff640 1 -- 172.21.15.70:0/2892504968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1acbca0 msgr2=0x561cc1adaa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.818+0000 7fac0bfff640 1 --2- 172.21.15.70:0/2892504968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1acbca0 0x561cc1adaa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.818+0000 7fac0bfff640 1 -- 172.21.15.70:0/2892504968 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cc1ad0b60 con 0x561cc1acb2b0 2024-04-01T01:34:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.818+0000 7fac0bfff640 1 --2- 172.21.15.70:0/2892504968 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acb2b0 0x561cc1ada520 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0b9fd0 tx=0x7fabfc00c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1276c14e1b3f1f23 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.818+0000 7fabeb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:34:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.818+0000 7fabeb7fe640 1 -- 172.21.15.70:0/2892504968 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0b9060 con 0x561cc1acb2b0 2024-04-01T01:34:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.818+0000 7fabeb7fe640 1 -- 172.21.15.70:0/2892504968 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fabfc0a3c60 con 0x561cc1acb2b0 2024-04-01T01:34:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.818+0000 7fac1541b740 1 -- 172.21.15.70:0/2892504968 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabfc09bcf0 con 0x561cc1acb2b0 2024-04-01T01:34:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.818+0000 7fabeb7fe640 1 -- 172.21.15.70:0/2892504968 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fabfc09b130 con 0x561cc1acb2b0 2024-04-01T01:34:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.819+0000 7fac1541b740 10 client.25694 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.819+0000 7fac1541b740 1 -- 172.21.15.70:0/2892504968 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561cc17040f0 con 0x561cc1acb2b0 2024-04-01T01:34:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.819+0000 7fabeb7fe640 1 -- 172.21.15.70:0/2892504968 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fabfc09bcf0 con 0x561cc1acb2b0 2024-04-01T01:34:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.819+0000 7fabeb7fe640 1 -- 172.21.15.70:0/2892504968 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fabfc097340 con 0x561cc1acb2b0 2024-04-01T01:34:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.819+0000 7fac1541b740 10 client.25694 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.819+0000 7fac1541b740 10 client.25694 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.819+0000 7fac1541b740 20 client.25694 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.819+0000 7fac1541b740 1 --2- 172.21.15.70:0/2892504968 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561cc1b47f50 0x561cc1b4a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.819+0000 7fac1541b740 4 client.25694 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:34:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.819+0000 7fac1541b740 1 -- 172.21.15.70:0/2892504968 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561cc1a3d6c0 con 0x561cc1b47f50 2024-04-01T01:34:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.819+0000 7fac0b7fe640 1 --2- 172.21.15.70:0/2892504968 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561cc1b47f50 0x561cc1b4a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fabeb7fe640 10 client.25694 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fac0b7fe640 1 --2- 172.21.15.70:0/2892504968 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561cc1b47f50 0x561cc1b4a370 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a01c868461171f94 server_cookie=7caf2610ab1d7c0e in_seq=0 out_seq=0 2024-04-01T01:34:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fabeb7fe640 1 -- 172.21.15.70:0/2892504968 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561cc1a3d6c0 con 0x561cc1b47f50 2024-04-01T01:34:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fabeb7fe640 10 client.25694 handle_command_reply: tid=0 2024-04-01T01:34:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fac1541b740 1 client.25694 shutdown 2024-04-01T01:34:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fac1541b740 1 -- 172.21.15.70:0/2892504968 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acb2b0 msgr2=0x561cc1ada520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fac1541b740 1 --2- 172.21.15.70:0/2892504968 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acb2b0 0x561cc1ada520 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0b9fd0 tx=0x7fabfc00c7d0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fac1541b740 1 -- 172.21.15.70:0/2892504968 shutdown_connections 2024-04-01T01:34:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fac1541b740 1 --2- 172.21.15.70:0/2892504968 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561cc1b47f50 0x561cc1b4a370 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fac1541b740 1 --2- 172.21.15.70:0/2892504968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cc1ad0e10 0x561cc1adba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fac1541b740 1 --2- 172.21.15.70:0/2892504968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cc1acbca0 0x561cc1adaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fabeb7fe640 0 client.25694 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fac1541b740 1 --2- 172.21.15.70:0/2892504968 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cc1acb2b0 0x561cc1ada520 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fac1541b740 1 -- 172.21.15.70:0/2892504968 >> 172.21.15.70:0/2892504968 conn(0x561cc1ac6820 msgr2=0x561cc1ac5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fac1541b740 1 -- 172.21.15.70:0/2892504968 shutdown_connections 2024-04-01T01:34:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fac1541b740 1 -- 172.21.15.70:0/2892504968 wait complete. 2024-04-01T01:34:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.820+0000 7fac1541b740 20 client.25694 trim_cache size 0 max 16384 2024-04-01T01:34:17.827 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:34:17.827 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:34:17.827 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:34:17.827 INFO:teuthology.orchestra.run.smithi070.stdout: "f2d04175-bcbd-4182-8e27-f28d7070e1fb": { 2024-04-01T01:34:17.827 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:34:17.827 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "f2d04175-bcbd-4182-8e27-f28d7070e1fb", 2024-04-01T01:34:17.827 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:34:17.827 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:34:17.827 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:34:17.828 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:34:17.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.821+0000 7fac1283b640 1 -- 172.21.15.70:0/2498595936 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fabf40a2530 msgr2=0x7fabf40a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.821+0000 7fac1283b640 1 --2- 172.21.15.70:0/2498595936 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fabf40a2530 0x7fabf40a49f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x561cc1acc500 tx=0x7fac04009500 comp rx=0 tx=0).stop 2024-04-01T01:34:17.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.821+0000 7fac1283b640 1 -- 172.21.15.70:0/2498595936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0c102270 msgr2=0x7fac0c197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:17.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.821+0000 7fac1283b640 1 --2- 172.21.15.70:0/2498595936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0c102270 0x7fac0c197a30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fabfc00e9f0 tx=0x7fabfc00edf0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.821+0000 7fac1283b640 1 -- 172.21.15.70:0/2498595936 shutdown_connections 2024-04-01T01:34:17.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.821+0000 7fac1283b640 1 --2- 172.21.15.70:0/2498595936 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fabf40a2530 0x7fabf40a49f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.821+0000 7fac1283b640 1 --2- 172.21.15.70:0/2498595936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac0c1074d0 0x7fac0c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.821+0000 7fac1283b640 1 --2- 172.21.15.70:0/2498595936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac0c104ba0 0x7fac0c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.821+0000 7fac1283b640 1 --2- 172.21.15.70:0/2498595936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0c102270 0x7fac0c197a30 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:17.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.821+0000 7fac1283b640 1 -- 172.21.15.70:0/2498595936 >> 172.21.15.70:0/2498595936 conn(0x7fac0c0fbbe0 msgr2=0x7fac0c104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:17.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.821+0000 7fac1283b640 1 -- 172.21.15.70:0/2498595936 shutdown_connections 2024-04-01T01:34:17.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:17.821+0000 7fac1283b640 1 -- 172.21.15.70:0/2498595936 wait complete. 2024-04-01T01:34:17.833 INFO:tasks.cephfs.filesystem:scrub status for tag:f2d04175-bcbd-4182-8e27-f28d7070e1fb - {'path': '/', 'tag': 'f2d04175-bcbd-4182-8e27-f28d7070e1fb', 'options': 'recursive,force'} 2024-04-01T01:34:18.385 INFO:teuthology.orchestra.run.smithi070.stderr:1600000 of 3200000 tuples (50%) done (elapsed 15.23 s, remaining 15.23 s) 2024-04-01T01:34:18.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:18 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:34:18.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:18 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:34:18.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:18 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:34:18.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:18 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:34:18.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:18 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:34:18.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:18 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:34:18.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:18 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:34:18.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:18 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:34:18.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:18 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:34:18.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:18 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:34:18.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:18 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:34:18.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:18 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:34:19.085 INFO:teuthology.orchestra.run.smithi070.stderr:1700000 of 3200000 tuples (53%) done (elapsed 15.93 s, remaining 14.05 s) 2024-04-01T01:34:19.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:19 smithi189 ceph-mon[35753]: pgmap v383: 129 pgs: 129 active+clean; 346 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 45 MiB/s wr, 102 op/s 2024-04-01T01:34:19.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:19 smithi070 ceph-mon[32179]: pgmap v383: 129 pgs: 129 active+clean; 346 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 45 MiB/s wr, 102 op/s 2024-04-01T01:34:19.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:19 smithi081 ceph-mon[35142]: pgmap v383: 129 pgs: 129 active+clean; 346 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 45 MiB/s wr, 102 op/s 2024-04-01T01:34:20.162 INFO:teuthology.orchestra.run.smithi070.stderr:1800000 of 3200000 tuples (56%) done (elapsed 17.01 s, remaining 13.23 s) 2024-04-01T01:34:21.196 INFO:teuthology.orchestra.run.smithi070.stderr:1900000 of 3200000 tuples (59%) done (elapsed 18.04 s, remaining 12.34 s) 2024-04-01T01:34:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2000000 of 3200000 tuples (62%) done (elapsed 18.84 s, remaining 11.30 s) 2024-04-01T01:34:21.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:21 smithi189 ceph-mon[35753]: pgmap v384: 129 pgs: 129 active+clean; 366 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 44 MiB/s wr, 120 op/s 2024-04-01T01:34:21.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:21 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:34:22.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:21 smithi070 ceph-mon[32179]: pgmap v384: 129 pgs: 129 active+clean; 366 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 44 MiB/s wr, 120 op/s 2024-04-01T01:34:22.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:21 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:34:22.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:21 smithi081 ceph-mon[35142]: pgmap v384: 129 pgs: 129 active+clean; 366 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 44 MiB/s wr, 120 op/s 2024-04-01T01:34:22.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:21 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:34:23.018 INFO:teuthology.orchestra.run.smithi070.stderr:2100000 of 3200000 tuples (65%) done (elapsed 19.86 s, remaining 10.40 s) 2024-04-01T01:34:23.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:23 smithi189 ceph-mon[35753]: pgmap v385: 129 pgs: 129 active+clean; 458 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 53 MiB/s wr, 174 op/s 2024-04-01T01:34:24.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:23 smithi070 ceph-mon[32179]: pgmap v385: 129 pgs: 129 active+clean; 458 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 53 MiB/s wr, 174 op/s 2024-04-01T01:34:24.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:23 smithi081 ceph-mon[35142]: pgmap v385: 129 pgs: 129 active+clean; 458 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 53 MiB/s wr, 174 op/s 2024-04-01T01:34:24.102 INFO:teuthology.orchestra.run.smithi070.stderr:2200000 of 3200000 tuples (68%) done (elapsed 20.95 s, remaining 9.52 s) 2024-04-01T01:34:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.777+0000 7fd5bdde9640 1 -- 172.21.15.70:0/926821580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5b8104b90 msgr2=0x7fd5b8105010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.777+0000 7fd5bdde9640 1 --2- 172.21.15.70:0/926821580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5b8104b90 0x7fd5b8105010 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009900 tx=0x7fd5ac02bc30 comp rx=0 tx=0).stop 2024-04-01T01:34:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.777+0000 7fd5bdde9640 1 -- 172.21.15.70:0/926821580 shutdown_connections 2024-04-01T01:34:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.777+0000 7fd5bdde9640 1 --2- 172.21.15.70:0/926821580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5b8105550 0x7fd5b810bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.777+0000 7fd5bdde9640 1 --2- 172.21.15.70:0/926821580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5b8104b90 0x7fd5b8105010 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.777+0000 7fd5bdde9640 1 --2- 172.21.15.70:0/926821580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b81039a0 0x7fd5b8103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.777+0000 7fd5bdde9640 1 -- 172.21.15.70:0/926821580 >> 172.21.15.70:0/926821580 conn(0x7fd5b80fef10 msgr2=0x7fd5b8101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.777+0000 7fd5bdde9640 1 -- 172.21.15.70:0/926821580 shutdown_connections 2024-04-01T01:34:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.777+0000 7fd5bdde9640 1 -- 172.21.15.70:0/926821580 wait complete. 2024-04-01T01:34:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.778+0000 7fd5bdde9640 1 Processor -- start 2024-04-01T01:34:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.778+0000 7fd5bdde9640 1 -- start start 2024-04-01T01:34:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.778+0000 7fd5bdde9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b81039a0 0x7fd5b8199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.778+0000 7fd5bdde9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5b8104b90 0x7fd5b819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.778+0000 7fd5bdde9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5b8105550 0x7fd5b81a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.778+0000 7fd5bdde9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b810f4d0 con 0x7fd5b8105550 2024-04-01T01:34:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.778+0000 7fd5bdde9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b810f350 con 0x7fd5b81039a0 2024-04-01T01:34:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.778+0000 7fd5bdde9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b810f650 con 0x7fd5b8104b90 2024-04-01T01:34:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.778+0000 7fd5b7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5b8105550 0x7fd5b81a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.779+0000 7fd5b6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5b8104b90 0x7fd5b819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.779+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b81039a0 0x7fd5b8199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.779+0000 7fd5b7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5b8105550 0x7fd5b81a10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45966/0 (socket says 172.21.15.70:45966) 2024-04-01T01:34:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.779+0000 7fd5b7fff640 1 -- 172.21.15.70:0/3337143127 learned_addr learned my addr 172.21.15.70:0/3337143127 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.779+0000 7fd5b6ffd640 1 -- 172.21.15.70:0/3337143127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b81039a0 msgr2=0x7fd5b8199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.779+0000 7fd5b6ffd640 1 --2- 172.21.15.70:0/3337143127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b81039a0 0x7fd5b8199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.779+0000 7fd5b6ffd640 1 -- 172.21.15.70:0/3337143127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5b8105550 msgr2=0x7fd5b81a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.779+0000 7fd5b6ffd640 1 --2- 172.21.15.70:0/3337143127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5b8105550 0x7fd5b81a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.779+0000 7fd5b6ffd640 1 -- 172.21.15.70:0/3337143127 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b81a1740 con 0x7fd5b8104b90 2024-04-01T01:34:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.779+0000 7fd5b6ffd640 1 --2- 172.21.15.70:0/3337143127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5b8104b90 0x7fd5b819a060 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac000c00 tx=0x7fd5ac005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.779+0000 7fd5b4ff9640 1 -- 172.21.15.70:0/3337143127 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac043020 con 0x7fd5b8104b90 2024-04-01T01:34:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.779+0000 7fd5b4ff9640 1 -- 172.21.15.70:0/3337143127 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5ac004350 con 0x7fd5b8104b90 2024-04-01T01:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.779+0000 7fd5bdde9640 1 -- 172.21.15.70:0/3337143127 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b81a19d0 con 0x7fd5b8104b90 2024-04-01T01:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.779+0000 7fd5b4ff9640 1 -- 172.21.15.70:0/3337143127 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac003b50 con 0x7fd5b8104b90 2024-04-01T01:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.779+0000 7fd5bdde9640 1 -- 172.21.15.70:0/3337143127 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b81a1f20 con 0x7fd5b8104b90 2024-04-01T01:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.781+0000 7fd5b4ff9640 1 -- 172.21.15.70:0/3337143127 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7fd5ac046020 con 0x7fd5b8104b90 2024-04-01T01:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.782+0000 7fd5c09d3740 1 Processor -- start 2024-04-01T01:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.782+0000 7fd5c09d3740 1 -- start start 2024-04-01T01:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.782+0000 7fd5c09d3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 0x55f1f2e59cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.782+0000 7fd5c09d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e5edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.782+0000 7fd5c09d3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5f430 0x55f1f2e618c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.782+0000 7fd5c09d3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f1f2dbd8f0 con 0x55f1f2e598d0 2024-04-01T01:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.782+0000 7fd5c09d3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f1f2d668d0 con 0x55f1f2e5f430 2024-04-01T01:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.782+0000 7fd5c09d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f1f2db9990 con 0x55f1f2e5a2c0 2024-04-01T01:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.782+0000 7fd5b4ff9640 1 --2- 172.21.15.70:0/3337143127 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5a00a25e0 0x7fd5a00a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.782+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e5edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.782+0000 7fd5b6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5f430 0x55f1f2e618c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.782+0000 7fd5b7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 0x55f1f2e59cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.782+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e5edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36176/0 (socket says 172.21.15.70:36176) 2024-04-01T01:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.782+0000 7fd5b77fe640 1 -- 172.21.15.70:0/2831308764 learned_addr learned my addr 172.21.15.70:0/2831308764 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.782+0000 7fd5b7fff640 1 --2- 172.21.15.70:0/3337143127 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5a00a25e0 0x7fd5a00a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.783+0000 7fd5b77fe640 1 -- 172.21.15.70:0/2831308764 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5f430 msgr2=0x55f1f2e618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.783+0000 7fd5b77fe640 1 --2- 172.21.15.70:0/2831308764 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5f430 0x55f1f2e618c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.783+0000 7fd5b4ff9640 1 -- 172.21.15.70:0/3337143127 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd5ac0f0bc0 con 0x7fd5b8104b90 2024-04-01T01:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.783+0000 7fd5b77fe640 1 -- 172.21.15.70:0/2831308764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 msgr2=0x55f1f2e59cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.783+0000 7fd5b77fe640 1 --2- 172.21.15.70:0/2831308764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 0x55f1f2e59cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.783+0000 7fd5b77fe640 1 -- 172.21.15.70:0/2831308764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1f2ad2210 con 0x55f1f2e5a2c0 2024-04-01T01:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.783+0000 7fd5b77fe640 1 --2- 172.21.15.70:0/2831308764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e5edc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009ca0 tx=0x7fd5a8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b04516b04eaade3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.783+0000 7fd5b7fff640 1 --2- 172.21.15.70:0/3337143127 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5a00a25e0 0x7fd5a00a4aa0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55f1f2e5ab20 tx=0x7fd5b0009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.783+0000 7fd59dffb640 1 -- 172.21.15.70:0/2831308764 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8004220 con 0x55f1f2e5a2c0 2024-04-01T01:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.783+0000 7fd59dffb640 1 -- 172.21.15.70:0/2831308764 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5a80043c0 con 0x55f1f2e5a2c0 2024-04-01T01:34:24.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.783+0000 7fd59dffb640 1 -- 172.21.15.70:0/2831308764 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a80197f0 con 0x55f1f2e5a2c0 2024-04-01T01:34:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.784+0000 7fd5c09d3740 1 -- 172.21.15.70:0/2831308764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 msgr2=0x55f1f2e5edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.784+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/2831308764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e5edc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009ca0 tx=0x7fd5a8011810 comp rx=0 tx=0).stop 2024-04-01T01:34:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.784+0000 7fd5c09d3740 1 -- 172.21.15.70:0/2831308764 shutdown_connections 2024-04-01T01:34:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.784+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/2831308764 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5f430 0x55f1f2e618c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.784+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/2831308764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e5edc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.784+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/2831308764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 0x55f1f2e59cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.784+0000 7fd5c09d3740 1 -- 172.21.15.70:0/2831308764 >> 172.21.15.70:0/2831308764 conn(0x55f1f2e54e40 msgr2=0x55f1f2e55240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.784+0000 7fd5c09d3740 1 -- 172.21.15.70:0/2831308764 shutdown_connections 2024-04-01T01:34:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.784+0000 7fd5c09d3740 1 -- 172.21.15.70:0/2831308764 wait complete. 2024-04-01T01:34:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.784+0000 7fd5c09d3740 1 -- start start 2024-04-01T01:34:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.784+0000 7fd5c09d3740 1 Processor -- start 2024-04-01T01:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5c09d3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 0x55f1f2e63c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5c09d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e64150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5c09d3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5f430 0x55f1f2e6b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5c09d3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f1f2e623e0 con 0x55f1f2e598d0 2024-04-01T01:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5c09d3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f1f2d5bf80 con 0x55f1f2e5f430 2024-04-01T01:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5c09d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f1f2e62560 con 0x55f1f2e5a2c0 2024-04-01T01:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5c09d3740 10 client.0 _ll_register_callbacks cb 0x55f1f2db4290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 0x55f1f2e63c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5b7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5f430 0x55f1f2e6b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5b6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e64150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 0x55f1f2e63c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45982/0 (socket says 172.21.15.70:45982) 2024-04-01T01:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5b7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5f430 0x55f1f2e6b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38044/0 (socket says 172.21.15.70:38044) 2024-04-01T01:34:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5b77fe640 1 -- 172.21.15.70:0/3267957857 learned_addr learned my addr 172.21.15.70:0/3267957857 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5b77fe640 1 -- 172.21.15.70:0/3267957857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 msgr2=0x55f1f2e64150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5b77fe640 1 --2- 172.21.15.70:0/3267957857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e64150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5b77fe640 1 -- 172.21.15.70:0/3267957857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5f430 msgr2=0x55f1f2e6b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5b77fe640 1 --2- 172.21.15.70:0/3267957857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5f430 0x55f1f2e6b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5b77fe640 1 -- 172.21.15.70:0/3267957857 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1f2b9e620 con 0x55f1f2e598d0 2024-04-01T01:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5b6ffd640 1 --2- 172.21.15.70:0/3267957857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e64150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5b7fff640 1 --2- 172.21.15.70:0/3267957857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5f430 0x55f1f2e6b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5b77fe640 1 --2- 172.21.15.70:0/3267957857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 0x55f1f2e63c10 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009380 tx=0x7fd5a8003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=313240bbefa7a01b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd596ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd596ffd640 1 -- 172.21.15.70:0/3267957857 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8028070 con 0x55f1f2e598d0 2024-04-01T01:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd596ffd640 1 -- 172.21.15.70:0/3267957857 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd5a8005040 con 0x55f1f2e598d0 2024-04-01T01:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.785+0000 7fd5c09d3740 1 -- 172.21.15.70:0/3267957857 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5a80043c0 con 0x55f1f2e598d0 2024-04-01T01:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.786+0000 7fd596ffd640 1 -- 172.21.15.70:0/3267957857 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd5a8034050 con 0x55f1f2e598d0 2024-04-01T01:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.786+0000 7fd5c09d3740 10 client.15891 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.786+0000 7fd5c09d3740 1 -- 172.21.15.70:0/3267957857 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f1f2e6e1c0 con 0x55f1f2e598d0 2024-04-01T01:34:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.786+0000 7fd596ffd640 1 -- 172.21.15.70:0/3267957857 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5a80043c0 con 0x55f1f2e598d0 2024-04-01T01:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.786+0000 7fd596ffd640 1 -- 172.21.15.70:0/3267957857 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd5a8004990 con 0x55f1f2e598d0 2024-04-01T01:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.786+0000 7fd5c09d3740 10 client.15891 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.786+0000 7fd5c09d3740 10 client.15891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.786+0000 7fd5c09d3740 20 client.15891 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.786+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/3267957857 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f1f2e6f690 0x55f1f2e71a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.786+0000 7fd5c09d3740 4 client.15891 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.786+0000 7fd5c09d3740 1 -- 172.21.15.70:0/3267957857 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1f2dd1120 con 0x55f1f2e6f690 2024-04-01T01:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.786+0000 7fd5b6ffd640 1 --2- 172.21.15.70:0/3267957857 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f1f2e6f690 0x55f1f2e71a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.787+0000 7fd596ffd640 10 client.15891 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.787+0000 7fd5b6ffd640 1 --2- 172.21.15.70:0/3267957857 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f1f2e6f690 0x55f1f2e71a80 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9234e6a9557f561 server_cookie=c43327d7e4302488 in_seq=0 out_seq=0 2024-04-01T01:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.787+0000 7fd596ffd640 1 -- 172.21.15.70:0/3267957857 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f1f2dd1120 con 0x55f1f2e6f690 2024-04-01T01:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.787+0000 7fd596ffd640 10 client.15891 handle_command_reply: tid=0 2024-04-01T01:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.787+0000 7fd5c09d3740 1 client.15891 shutdown 2024-04-01T01:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.788+0000 7fd5c09d3740 1 -- 172.21.15.70:0/3267957857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 msgr2=0x55f1f2e63c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.788+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/3267957857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 0x55f1f2e63c10 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009380 tx=0x7fd5a8003820 comp rx=0 tx=0).stop 2024-04-01T01:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.788+0000 7fd5c09d3740 1 -- 172.21.15.70:0/3267957857 shutdown_connections 2024-04-01T01:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.788+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/3267957857 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f1f2e6f690 0x55f1f2e71a80 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.788+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/3267957857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5f430 0x55f1f2e6b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.788+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/3267957857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e64150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.788+0000 7fd596ffd640 0 client.15891 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.788+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/3267957857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 0x55f1f2e63c10 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.788+0000 7fd5c09d3740 1 -- 172.21.15.70:0/3267957857 >> 172.21.15.70:0/3267957857 conn(0x55f1f2e54e40 msgr2=0x55f1f2e53f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.788+0000 7fd5c09d3740 1 -- 172.21.15.70:0/3267957857 shutdown_connections 2024-04-01T01:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.788+0000 7fd5c09d3740 1 -- 172.21.15.70:0/3267957857 wait complete. 2024-04-01T01:34:24.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.788+0000 7fd5c09d3740 20 client.15891 trim_cache size 0 max 16384 2024-04-01T01:34:24.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.804+0000 7fd5c09d3740 1 Processor -- start 2024-04-01T01:34:24.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.804+0000 7fd5c09d3740 1 -- start start 2024-04-01T01:34:24.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.804+0000 7fd5c09d3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 0x55f1f2e55220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.804+0000 7fd5c09d3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e6b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.804+0000 7fd5c09d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5f430 0x55f1f2e62f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.804+0000 7fd5c09d3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b81ac4e0 con 0x55f1f2e598d0 2024-04-01T01:34:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.804+0000 7fd5c09d3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b81ae020 con 0x55f1f2e5a2c0 2024-04-01T01:34:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.804+0000 7fd5c09d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b81acbb0 con 0x55f1f2e5f430 2024-04-01T01:34:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.804+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 0x55f1f2e55220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.804+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 0x55f1f2e55220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45998/0 (socket says 172.21.15.70:45998) 2024-04-01T01:34:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.804+0000 7fd5b77fe640 1 -- 172.21.15.70:0/4133140213 learned_addr learned my addr 172.21.15.70:0/4133140213 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.804+0000 7fd5b7fff640 1 --2- 172.21.15.70:0/4133140213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5f430 0x55f1f2e62f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.804+0000 7fd5b6ffd640 1 --2- 172.21.15.70:0/4133140213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e6b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.805+0000 7fd5b77fe640 1 -- 172.21.15.70:0/4133140213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5f430 msgr2=0x55f1f2e62f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.805+0000 7fd5b77fe640 1 --2- 172.21.15.70:0/4133140213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5f430 0x55f1f2e62f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.805+0000 7fd5b77fe640 1 -- 172.21.15.70:0/4133140213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5a2c0 msgr2=0x55f1f2e6b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.805+0000 7fd5b77fe640 1 --2- 172.21.15.70:0/4133140213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e6b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.805+0000 7fd5b77fe640 1 -- 172.21.15.70:0/4133140213 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1f2e55bb0 con 0x55f1f2e598d0 2024-04-01T01:34:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.805+0000 7fd5b77fe640 1 --2- 172.21.15.70:0/4133140213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 0x55f1f2e55220 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8011d10 tx=0x7fd5a8004f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b597263d56454dbc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.805+0000 7fd59d7fa640 1 -- 172.21.15.70:0/4133140213 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a803d070 con 0x55f1f2e598d0 2024-04-01T01:34:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.805+0000 7fd59d7fa640 1 -- 172.21.15.70:0/4133140213 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5a8031030 con 0x55f1f2e598d0 2024-04-01T01:34:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.805+0000 7fd59d7fa640 1 -- 172.21.15.70:0/4133140213 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8028040 con 0x55f1f2e598d0 2024-04-01T01:34:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.806+0000 7fd5c09d3740 1 -- 172.21.15.70:0/4133140213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 msgr2=0x55f1f2e55220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.806+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/4133140213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 0x55f1f2e55220 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8011d10 tx=0x7fd5a8004f50 comp rx=0 tx=0).stop 2024-04-01T01:34:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.806+0000 7fd5c09d3740 1 -- 172.21.15.70:0/4133140213 shutdown_connections 2024-04-01T01:34:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.806+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/4133140213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5f430 0x55f1f2e62f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.806+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/4133140213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e6b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.806+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/4133140213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e598d0 0x55f1f2e55220 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.806+0000 7fd5c09d3740 1 -- 172.21.15.70:0/4133140213 >> 172.21.15.70:0/4133140213 conn(0x55f1f2e54e40 msgr2=0x55f1f2e53f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.806+0000 7fd5c09d3740 1 -- 172.21.15.70:0/4133140213 shutdown_connections 2024-04-01T01:34:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.806+0000 7fd5c09d3740 1 -- 172.21.15.70:0/4133140213 wait complete. 2024-04-01T01:34:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.806+0000 7fd5c09d3740 1 -- start start 2024-04-01T01:34:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5c09d3740 1 Processor -- start 2024-04-01T01:34:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5c09d3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e598d0 0x55f1f2e683c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5c09d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e68900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5c09d3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e5f430 0x55f1f2e698a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5c09d3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b81ad950 con 0x55f1f2e5f430 2024-04-01T01:34:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5c09d3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b81ad280 con 0x55f1f2e598d0 2024-04-01T01:34:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5c09d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b81aa4c0 con 0x55f1f2e5a2c0 2024-04-01T01:34:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5c09d3740 10 client.0 _ll_register_callbacks cb 0x7fd5b00134e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e598d0 0x55f1f2e683c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5b7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e5f430 0x55f1f2e698a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e598d0 0x55f1f2e683c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38060/0 (socket says 172.21.15.70:38060) 2024-04-01T01:34:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5b77fe640 1 -- 172.21.15.70:0/2167787510 learned_addr learned my addr 172.21.15.70:0/2167787510 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5b6ffd640 1 --2- 172.21.15.70:0/2167787510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e68900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5b77fe640 1 -- 172.21.15.70:0/2167787510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 msgr2=0x55f1f2e68900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5b77fe640 1 --2- 172.21.15.70:0/2167787510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e68900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5b77fe640 1 -- 172.21.15.70:0/2167787510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e5f430 msgr2=0x55f1f2e698a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5b77fe640 1 --2- 172.21.15.70:0/2167787510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e5f430 0x55f1f2e698a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5b77fe640 1 -- 172.21.15.70:0/2167787510 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1f2e69e90 con 0x55f1f2e598d0 2024-04-01T01:34:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd5b77fe640 1 --2- 172.21.15.70:0/2167787510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e598d0 0x55f1f2e683c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009f60 tx=0x7fd5a80042d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bf573fd0f50593d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.807+0000 7fd596ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:34:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.808+0000 7fd596ffd640 1 -- 172.21.15.70:0/2167787510 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a803d020 con 0x55f1f2e598d0 2024-04-01T01:34:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.808+0000 7fd596ffd640 1 -- 172.21.15.70:0/2167787510 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd5a8019880 con 0x55f1f2e598d0 2024-04-01T01:34:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.808+0000 7fd5c09d3740 1 -- 172.21.15.70:0/2167787510 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5a8031030 con 0x55f1f2e598d0 2024-04-01T01:34:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.808+0000 7fd596ffd640 1 -- 172.21.15.70:0/2167787510 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd5a8048050 con 0x55f1f2e598d0 2024-04-01T01:34:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.808+0000 7fd5c09d3740 10 client.25712 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:24.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.808+0000 7fd5c09d3740 1 -- 172.21.15.70:0/2167787510 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f1f2e6b1a0 con 0x55f1f2e598d0 2024-04-01T01:34:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.808+0000 7fd596ffd640 1 -- 172.21.15.70:0/2167787510 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5a8031030 con 0x55f1f2e598d0 2024-04-01T01:34:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.808+0000 7fd596ffd640 1 -- 172.21.15.70:0/2167787510 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd5a8005040 con 0x55f1f2e598d0 2024-04-01T01:34:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.808+0000 7fd5c09d3740 10 client.25712 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.808+0000 7fd5c09d3740 10 client.25712 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.808+0000 7fd5c09d3740 20 client.25712 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.808+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/2167787510 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f1f2ed5d00 0x55f1f2ed8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.809+0000 7fd5c09d3740 4 client.25712 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:34:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.809+0000 7fd5c09d3740 1 -- 172.21.15.70:0/2167787510 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f1f2dcae30 con 0x55f1f2ed5d00 2024-04-01T01:34:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.809+0000 7fd5b6ffd640 1 --2- 172.21.15.70:0/2167787510 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f1f2ed5d00 0x55f1f2ed8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.810+0000 7fd596ffd640 10 client.25712 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:24.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.810+0000 7fd5b6ffd640 1 --2- 172.21.15.70:0/2167787510 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f1f2ed5d00 0x55f1f2ed8120 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e87606979782b8d server_cookie=4f34bf81e340396c in_seq=0 out_seq=0 2024-04-01T01:34:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.833+0000 7fd596ffd640 1 -- 172.21.15.70:0/2167787510 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f1f2dcae30 con 0x55f1f2ed5d00 2024-04-01T01:34:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.833+0000 7fd596ffd640 10 client.25712 handle_command_reply: tid=0 2024-04-01T01:34:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.833+0000 7fd5c09d3740 1 client.25712 shutdown 2024-04-01T01:34:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.833+0000 7fd5c09d3740 1 -- 172.21.15.70:0/2167787510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e598d0 msgr2=0x55f1f2e683c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.833+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/2167787510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e598d0 0x55f1f2e683c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009f60 tx=0x7fd5a80042d0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.833+0000 7fd5c09d3740 1 -- 172.21.15.70:0/2167787510 shutdown_connections 2024-04-01T01:34:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.833+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/2167787510 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f1f2ed5d00 0x55f1f2ed8120 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.833+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/2167787510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1f2e5f430 0x55f1f2e698a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.833+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/2167787510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1f2e5a2c0 0x55f1f2e68900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.833+0000 7fd596ffd640 0 client.25712 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.833+0000 7fd5c09d3740 1 --2- 172.21.15.70:0/2167787510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1f2e598d0 0x55f1f2e683c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.833+0000 7fd5c09d3740 1 -- 172.21.15.70:0/2167787510 >> 172.21.15.70:0/2167787510 conn(0x55f1f2e54e40 msgr2=0x55f1f2e53f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.833+0000 7fd5c09d3740 1 -- 172.21.15.70:0/2167787510 shutdown_connections 2024-04-01T01:34:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.833+0000 7fd5c09d3740 1 -- 172.21.15.70:0/2167787510 wait complete. 2024-04-01T01:34:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.833+0000 7fd5c09d3740 20 client.25712 trim_cache size 0 max 16384 2024-04-01T01:34:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:34:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:34:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:34:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:34:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.834+0000 7fd5bdde9640 1 -- 172.21.15.70:0/3337143127 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5a00a25e0 msgr2=0x7fd5a00a4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.834+0000 7fd5bdde9640 1 --2- 172.21.15.70:0/3337143127 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5a00a25e0 0x7fd5a00a4aa0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55f1f2e5ab20 tx=0x7fd5b0009500 comp rx=0 tx=0).stop 2024-04-01T01:34:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.834+0000 7fd5bdde9640 1 -- 172.21.15.70:0/3337143127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5b8104b90 msgr2=0x7fd5b819a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.834+0000 7fd5bdde9640 1 --2- 172.21.15.70:0/3337143127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5b8104b90 0x7fd5b819a060 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac000c00 tx=0x7fd5ac005dc0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.834+0000 7fd5bdde9640 1 -- 172.21.15.70:0/3337143127 shutdown_connections 2024-04-01T01:34:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.834+0000 7fd5bdde9640 1 --2- 172.21.15.70:0/3337143127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5b8105550 0x7fd5b81a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.834+0000 7fd5bdde9640 1 --2- 172.21.15.70:0/3337143127 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5a00a25e0 0x7fd5a00a4aa0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.834+0000 7fd5bdde9640 1 --2- 172.21.15.70:0/3337143127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5b8104b90 0x7fd5b819a060 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.834+0000 7fd5bdde9640 1 --2- 172.21.15.70:0/3337143127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b81039a0 0x7fd5b8199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.834+0000 7fd5bdde9640 1 -- 172.21.15.70:0/3337143127 >> 172.21.15.70:0/3337143127 conn(0x7fd5b80fef10 msgr2=0x7fd5b8107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.834+0000 7fd5bdde9640 1 -- 172.21.15.70:0/3337143127 shutdown_connections 2024-04-01T01:34:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:24.834+0000 7fd5bdde9640 1 -- 172.21.15.70:0/3337143127 wait complete. 2024-04-01T01:34:24.911 INFO:teuthology.orchestra.run.smithi070.stderr:2300000 of 3200000 tuples (71%) done (elapsed 21.75 s, remaining 8.51 s) 2024-04-01T01:34:25.986 INFO:teuthology.orchestra.run.smithi070.stderr:2400000 of 3200000 tuples (75%) done (elapsed 22.83 s, remaining 7.61 s) 2024-04-01T01:34:25.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:25 smithi189 ceph-mon[35753]: pgmap v386: 129 pgs: 129 active+clean; 474 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 43 MiB/s wr, 163 op/s 2024-04-01T01:34:26.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:25 smithi070 ceph-mon[32179]: pgmap v386: 129 pgs: 129 active+clean; 474 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 43 MiB/s wr, 163 op/s 2024-04-01T01:34:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:25 smithi081 ceph-mon[35142]: pgmap v386: 129 pgs: 129 active+clean; 474 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 43 MiB/s wr, 163 op/s 2024-04-01T01:34:27.058 INFO:teuthology.orchestra.run.smithi070.stderr:2500000 of 3200000 tuples (78%) done (elapsed 23.90 s, remaining 6.69 s) 2024-04-01T01:34:27.780 INFO:teuthology.orchestra.run.smithi070.stderr:2600000 of 3200000 tuples (81%) done (elapsed 24.62 s, remaining 5.68 s) 2024-04-01T01:34:27.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:27 smithi189 ceph-mon[35753]: pgmap v387: 129 pgs: 129 active+clean; 546 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 49 MiB/s wr, 158 op/s 2024-04-01T01:34:28.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:27 smithi070 ceph-mon[32179]: pgmap v387: 129 pgs: 129 active+clean; 546 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 49 MiB/s wr, 158 op/s 2024-04-01T01:34:28.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:27 smithi081 ceph-mon[35142]: pgmap v387: 129 pgs: 129 active+clean; 546 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 49 MiB/s wr, 158 op/s 2024-04-01T01:34:28.855 INFO:teuthology.orchestra.run.smithi070.stderr:2700000 of 3200000 tuples (84%) done (elapsed 25.70 s, remaining 4.76 s) 2024-04-01T01:34:29.613 INFO:teuthology.orchestra.run.smithi070.stderr:2800000 of 3200000 tuples (87%) done (elapsed 26.46 s, remaining 3.78 s) 2024-04-01T01:34:29.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:29 smithi189 ceph-mon[35753]: pgmap v388: 129 pgs: 129 active+clean; 592 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 48 MiB/s wr, 130 op/s 2024-04-01T01:34:30.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:29 smithi070 ceph-mon[32179]: pgmap v388: 129 pgs: 129 active+clean; 592 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 48 MiB/s wr, 130 op/s 2024-04-01T01:34:30.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:29 smithi081 ceph-mon[35142]: pgmap v388: 129 pgs: 129 active+clean; 592 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 48 MiB/s wr, 130 op/s 2024-04-01T01:34:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2900000 of 3200000 tuples (90%) done (elapsed 27.51 s, remaining 2.85 s) 2024-04-01T01:34:31.697 INFO:teuthology.orchestra.run.smithi070.stderr:3000000 of 3200000 tuples (93%) done (elapsed 28.54 s, remaining 1.90 s) 2024-04-01T01:34:31.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:31 smithi189 ceph-mon[35753]: pgmap v389: 129 pgs: 129 active+clean; 632 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 48 MiB/s wr, 112 op/s 2024-04-01T01:34:32.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:31 smithi070 ceph-mon[32179]: pgmap v389: 129 pgs: 129 active+clean; 632 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 48 MiB/s wr, 112 op/s 2024-04-01T01:34:32.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:31 smithi081 ceph-mon[35142]: pgmap v389: 129 pgs: 129 active+clean; 632 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 48 MiB/s wr, 112 op/s 2024-04-01T01:34:32.477 INFO:teuthology.orchestra.run.smithi070.stderr:3100000 of 3200000 tuples (96%) done (elapsed 29.32 s, remaining 0.95 s) 2024-04-01T01:34:33.545 INFO:teuthology.orchestra.run.smithi070.stderr:3200000 of 3200000 tuples (100%) done (elapsed 30.39 s, remaining 0.00 s) 2024-04-01T01:34:33.614 INFO:teuthology.orchestra.run.smithi070.stderr:vacuuming... 2024-04-01T01:34:33.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:33 smithi189 ceph-mon[35753]: pgmap v390: 129 pgs: 129 active+clean; 688 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 53 MiB/s wr, 93 op/s 2024-04-01T01:34:34.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:33 smithi070 ceph-mon[32179]: pgmap v390: 129 pgs: 129 active+clean; 688 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 53 MiB/s wr, 93 op/s 2024-04-01T01:34:34.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:33 smithi081 ceph-mon[35142]: pgmap v390: 129 pgs: 129 active+clean; 688 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 53 MiB/s wr, 93 op/s 2024-04-01T01:34:35.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:35 smithi189 ceph-mon[35753]: pgmap v391: 129 pgs: 129 active+clean; 728 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 45 MiB/s wr, 41 op/s 2024-04-01T01:34:36.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:35 smithi070 ceph-mon[32179]: pgmap v391: 129 pgs: 129 active+clean; 728 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 45 MiB/s wr, 41 op/s 2024-04-01T01:34:36.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:35 smithi081 ceph-mon[35142]: pgmap v391: 129 pgs: 129 active+clean; 728 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 45 MiB/s wr, 41 op/s 2024-04-01T01:34:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.940+0000 7f1ab1b8a640 1 -- 172.21.15.70:0/2649590250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1aac101860 msgr2=0x7f1aac101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.940+0000 7f1ab1b8a640 1 --2- 172.21.15.70:0/2649590250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1aac101860 0x7f1aac101ce0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0009900 tx=0x7f1aa002bc30 comp rx=0 tx=0).stop 2024-04-01T01:34:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.940+0000 7f1ab1b8a640 1 -- 172.21.15.70:0/2649590250 shutdown_connections 2024-04-01T01:34:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.940+0000 7f1ab1b8a640 1 --2- 172.21.15.70:0/2649590250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aac102220 0x7f1aac108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.940+0000 7f1ab1b8a640 1 --2- 172.21.15.70:0/2649590250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1aac101860 0x7f1aac101ce0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.940+0000 7f1ab1b8a640 1 --2- 172.21.15.70:0/2649590250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aac100670 0x7f1aac100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.940+0000 7f1ab1b8a640 1 -- 172.21.15.70:0/2649590250 >> 172.21.15.70:0/2649590250 conn(0x7f1aac0fbbe0 msgr2=0x7f1aac0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.940+0000 7f1ab1b8a640 1 -- 172.21.15.70:0/2649590250 shutdown_connections 2024-04-01T01:34:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.940+0000 7f1ab1b8a640 1 -- 172.21.15.70:0/2649590250 wait complete. 2024-04-01T01:34:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.941+0000 7f1ab1b8a640 1 Processor -- start 2024-04-01T01:34:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.941+0000 7f1ab1b8a640 1 -- start start 2024-04-01T01:34:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.941+0000 7f1ab1b8a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aac100670 0x7f1aac0698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.941+0000 7f1ab1b8a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1aac101860 0x7f1aac19c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.941+0000 7f1ab1b8a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aac102220 0x7f1aac19ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.941+0000 7f1ab1b8a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1aac10c190 con 0x7f1aac101860 2024-04-01T01:34:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.941+0000 7f1ab1b8a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1aac10c010 con 0x7f1aac102220 2024-04-01T01:34:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.941+0000 7f1ab1b8a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1aac10c310 con 0x7f1aac100670 2024-04-01T01:34:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.941+0000 7f1aaaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1aac101860 0x7f1aac19c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.941+0000 7f1aabfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aac102220 0x7f1aac19ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.941+0000 7f1aabfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aac102220 0x7f1aac19ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:44338/0 (socket says 172.21.15.70:44338) 2024-04-01T01:34:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.941+0000 7f1aab7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aac100670 0x7f1aac0698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.941+0000 7f1aabfff640 1 -- 172.21.15.70:0/1664751321 learned_addr learned my addr 172.21.15.70:0/1664751321 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.942+0000 7f1aaaffd640 1 -- 172.21.15.70:0/1664751321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aac100670 msgr2=0x7f1aac0698f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.942+0000 7f1aaaffd640 1 --2- 172.21.15.70:0/1664751321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aac100670 0x7f1aac0698f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.942+0000 7f1aaaffd640 1 -- 172.21.15.70:0/1664751321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aac102220 msgr2=0x7f1aac19ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.942+0000 7f1aaaffd640 1 --2- 172.21.15.70:0/1664751321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aac102220 0x7f1aac19ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.942+0000 7f1aaaffd640 1 -- 172.21.15.70:0/1664751321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aac19f570 con 0x7f1aac101860 2024-04-01T01:34:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.942+0000 7f1aaaffd640 1 --2- 172.21.15.70:0/1664751321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1aac101860 0x7f1aac19c6e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0000c00 tx=0x7f1aa0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.942+0000 7f1aa8ff9640 1 -- 172.21.15.70:0/1664751321 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0043020 con 0x7f1aac101860 2024-04-01T01:34:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.942+0000 7f1aa8ff9640 1 -- 172.21.15.70:0/1664751321 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1aa0004aa0 con 0x7f1aac101860 2024-04-01T01:34:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.942+0000 7f1ab1b8a640 1 -- 172.21.15.70:0/1664751321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aac19f800 con 0x7f1aac101860 2024-04-01T01:34:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.942+0000 7f1aa8ff9640 1 -- 172.21.15.70:0/1664751321 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0003cd0 con 0x7f1aac101860 2024-04-01T01:34:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.942+0000 7f1ab1b8a640 1 -- 172.21.15.70:0/1664751321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aac19fd50 con 0x7f1aac101860 2024-04-01T01:34:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.945+0000 7f1aa8ff9640 1 -- 172.21.15.70:0/1664751321 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f1aa0046020 con 0x7f1aac101860 2024-04-01T01:34:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.945+0000 7f1ab476e740 1 Processor -- start 2024-04-01T01:34:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.945+0000 7f1ab476e740 1 -- start start 2024-04-01T01:34:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.945+0000 7f1aa8ff9640 1 --2- 172.21.15.70:0/1664751321 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a940a25e0 0x7f1a940a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.945+0000 7f1ab476e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 0x5596b9baecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.945+0000 7f1ab476e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 0x5596b9bb3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.945+0000 7f1ab476e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 0x5596b9bb68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.945+0000 7f1ab476e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5596b9b128f0 con 0x5596b9bb4430 2024-04-01T01:34:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.945+0000 7f1ab476e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5596b9abb8d0 con 0x5596b9baf2c0 2024-04-01T01:34:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.945+0000 7f1ab476e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5596b9b0e990 con 0x5596b9bae8d0 2024-04-01T01:34:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.946+0000 7f1aabfff640 1 --2- 172.21.15.70:0/1664751321 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a940a25e0 0x7f1a940a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.946+0000 7f1aab7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 0x5596b9baecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.946+0000 7f1aab7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 0x5596b9baecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:60690/0 (socket says 172.21.15.70:60690) 2024-04-01T01:34:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.946+0000 7f1aab7fe640 1 -- 172.21.15.70:0/2205663983 learned_addr learned my addr 172.21.15.70:0/2205663983 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.946+0000 7f1aabfff640 1 --2- 172.21.15.70:0/2205663983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 0x5596b9bb68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.946+0000 7f1aa8ff9640 1 -- 172.21.15.70:0/1664751321 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1aa00f0ff0 con 0x7f1aac101860 2024-04-01T01:34:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.946+0000 7f1aaaffd640 1 --2- 172.21.15.70:0/2205663983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 0x5596b9bb3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.946+0000 7f1aab7fe640 1 -- 172.21.15.70:0/2205663983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 msgr2=0x5596b9bb3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.946+0000 7f1aab7fe640 1 --2- 172.21.15.70:0/2205663983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 0x5596b9bb3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.946+0000 7f1aab7fe640 1 -- 172.21.15.70:0/2205663983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 msgr2=0x5596b9bb68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.946+0000 7f1aab7fe640 1 --2- 172.21.15.70:0/2205663983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 0x5596b9bb68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.946+0000 7f1aab7fe640 1 -- 172.21.15.70:0/2205663983 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596b9827210 con 0x5596b9bae8d0 2024-04-01T01:34:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.946+0000 7f1aaaffd640 1 --2- 172.21.15.70:0/2205663983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 0x5596b9bb3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:34:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.946+0000 7f1aabfff640 1 --2- 172.21.15.70:0/1664751321 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a940a25e0 0x7f1a940a4aa0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4009a50 tx=0x7f1aa4009ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.946+0000 7f1aab7fe640 1 --2- 172.21.15.70:0/2205663983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 0x5596b9baecf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c004610 tx=0x7f1a9c012bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71ba136e97bce657 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.947+0000 7f1a91ffb640 1 -- 172.21.15.70:0/2205663983 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c00bc30 con 0x5596b9bae8d0 2024-04-01T01:34:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.947+0000 7f1a91ffb640 1 -- 172.21.15.70:0/2205663983 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a9c00bdd0 con 0x5596b9bae8d0 2024-04-01T01:34:37.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.947+0000 7f1a91ffb640 1 -- 172.21.15.70:0/2205663983 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c01b910 con 0x5596b9bae8d0 2024-04-01T01:34:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.948+0000 7f1ab476e740 1 -- 172.21.15.70:0/2205663983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 msgr2=0x5596b9baecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.948+0000 7f1ab476e740 1 --2- 172.21.15.70:0/2205663983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 0x5596b9baecf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c004610 tx=0x7f1a9c012bf0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.948+0000 7f1ab476e740 1 -- 172.21.15.70:0/2205663983 shutdown_connections 2024-04-01T01:34:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.948+0000 7f1ab476e740 1 --2- 172.21.15.70:0/2205663983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 0x5596b9bb68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.948+0000 7f1ab476e740 1 --2- 172.21.15.70:0/2205663983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 0x5596b9bb3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.948+0000 7f1ab476e740 1 --2- 172.21.15.70:0/2205663983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 0x5596b9baecf0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.948+0000 7f1ab476e740 1 -- 172.21.15.70:0/2205663983 >> 172.21.15.70:0/2205663983 conn(0x5596b9ba9e40 msgr2=0x5596b9baa240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.948+0000 7f1ab476e740 1 -- 172.21.15.70:0/2205663983 shutdown_connections 2024-04-01T01:34:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.948+0000 7f1ab476e740 1 -- 172.21.15.70:0/2205663983 wait complete. 2024-04-01T01:34:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.948+0000 7f1ab476e740 1 -- start start 2024-04-01T01:34:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1ab476e740 1 Processor -- start 2024-04-01T01:34:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1ab476e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9bae8d0 0x5596b9bb8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1ab476e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9baf2c0 0x5596b9bb9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1ab476e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bb4430 0x5596b9bc0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1ab476e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5596b9bb73e0 con 0x5596b9baf2c0 2024-04-01T01:34:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1ab476e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5596b9ab0f80 con 0x5596b9bae8d0 2024-04-01T01:34:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1ab476e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5596b9bb7560 con 0x5596b9bb4430 2024-04-01T01:34:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1ab476e740 10 client.0 _ll_register_callbacks cb 0x5596b9b09290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1aab7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9bae8d0 0x5596b9bb8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1aaaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9baf2c0 0x5596b9bb9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1aabfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bb4430 0x5596b9bc0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1aab7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9bae8d0 0x5596b9bb8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:44350/0 (socket says 172.21.15.70:44350) 2024-04-01T01:34:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1aaaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9baf2c0 0x5596b9bb9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58972/0 (socket says 172.21.15.70:58972) 2024-04-01T01:34:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1aab7fe640 1 -- 172.21.15.70:0/417933523 learned_addr learned my addr 172.21.15.70:0/417933523 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1aabfff640 1 -- 172.21.15.70:0/417933523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9bae8d0 msgr2=0x5596b9bb8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1aabfff640 1 --2- 172.21.15.70:0/417933523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9bae8d0 0x5596b9bb8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1aabfff640 1 -- 172.21.15.70:0/417933523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9baf2c0 msgr2=0x5596b9bb9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1aabfff640 1 --2- 172.21.15.70:0/417933523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9baf2c0 0x5596b9bb9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.949+0000 7f1aabfff640 1 -- 172.21.15.70:0/417933523 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596b98f3620 con 0x5596b9bb4430 2024-04-01T01:34:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.950+0000 7f1aabfff640 1 --2- 172.21.15.70:0/417933523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bb4430 0x5596b9bc0190 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4009fd0 tx=0x7f1aa401c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=367d0f80bf16397b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.950+0000 7f1a8affd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:34:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.950+0000 7f1a8affd640 1 -- 172.21.15.70:0/417933523 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4020610 con 0x5596b9bb4430 2024-04-01T01:34:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.950+0000 7f1a8affd640 1 -- 172.21.15.70:0/417933523 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1aa40207b0 con 0x5596b9bb4430 2024-04-01T01:34:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.950+0000 7f1ab476e740 1 -- 172.21.15.70:0/417933523 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a9c00bdd0 con 0x5596b9bb4430 2024-04-01T01:34:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.950+0000 7f1a8affd640 1 -- 172.21.15.70:0/417933523 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1aa40209b0 con 0x5596b9bb4430 2024-04-01T01:34:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.950+0000 7f1ab476e740 10 client.25639 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.950+0000 7f1ab476e740 1 -- 172.21.15.70:0/417933523 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5596b9bc3150 con 0x5596b9bb4430 2024-04-01T01:34:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.950+0000 7f1a8affd640 1 -- 172.21.15.70:0/417933523 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a9c00bdd0 con 0x5596b9bb4430 2024-04-01T01:34:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.950+0000 7f1a8affd640 1 -- 172.21.15.70:0/417933523 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1aa401f070 con 0x5596b9bb4430 2024-04-01T01:34:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.950+0000 7f1ab476e740 10 client.25639 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.950+0000 7f1ab476e740 10 client.25639 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.950+0000 7f1ab476e740 20 client.25639 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:37.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.951+0000 7f1ab476e740 1 --2- 172.21.15.70:0/417933523 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5596b9bc48b0 0x5596b9bc6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.951+0000 7f1ab476e740 4 client.25639 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.951+0000 7f1ab476e740 1 -- 172.21.15.70:0/417933523 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596b9b26120 con 0x5596b9bc48b0 2024-04-01T01:34:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.951+0000 7f1aab7fe640 1 --2- 172.21.15.70:0/417933523 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5596b9bc48b0 0x5596b9bc6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.951+0000 7f1aab7fe640 1 --2- 172.21.15.70:0/417933523 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5596b9bc48b0 0x5596b9bc6d00 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82f65547e8da8f0b server_cookie=f7983edefa321991 in_seq=0 out_seq=0 2024-04-01T01:34:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.951+0000 7f1a8affd640 10 client.25639 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.952+0000 7f1a8affd640 1 -- 172.21.15.70:0/417933523 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5596b9b26120 con 0x5596b9bc48b0 2024-04-01T01:34:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.952+0000 7f1a8affd640 10 client.25639 handle_command_reply: tid=0 2024-04-01T01:34:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.952+0000 7f1ab476e740 1 client.25639 shutdown 2024-04-01T01:34:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.952+0000 7f1ab476e740 1 -- 172.21.15.70:0/417933523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bb4430 msgr2=0x5596b9bc0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.952+0000 7f1ab476e740 1 --2- 172.21.15.70:0/417933523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bb4430 0x5596b9bc0190 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4009fd0 tx=0x7f1aa401c000 comp rx=0 tx=0).stop 2024-04-01T01:34:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.952+0000 7f1ab476e740 1 -- 172.21.15.70:0/417933523 shutdown_connections 2024-04-01T01:34:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.952+0000 7f1ab476e740 1 --2- 172.21.15.70:0/417933523 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5596b9bc48b0 0x5596b9bc6d00 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.952+0000 7f1ab476e740 1 --2- 172.21.15.70:0/417933523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bb4430 0x5596b9bc0190 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.952+0000 7f1ab476e740 1 --2- 172.21.15.70:0/417933523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9baf2c0 0x5596b9bb9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.952+0000 7f1a8affd640 0 client.25639 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.952+0000 7f1ab476e740 1 --2- 172.21.15.70:0/417933523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9bae8d0 0x5596b9bb8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.952+0000 7f1ab476e740 1 -- 172.21.15.70:0/417933523 >> 172.21.15.70:0/417933523 conn(0x5596b9ba9e40 msgr2=0x5596b9ba8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.953+0000 7f1ab476e740 1 -- 172.21.15.70:0/417933523 shutdown_connections 2024-04-01T01:34:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.953+0000 7f1ab476e740 1 -- 172.21.15.70:0/417933523 wait complete. 2024-04-01T01:34:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.953+0000 7f1ab476e740 20 client.25639 trim_cache size 0 max 16384 2024-04-01T01:34:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.969+0000 7f1ab476e740 1 Processor -- start 2024-04-01T01:34:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1ab476e740 1 -- start start 2024-04-01T01:34:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1ab476e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 0x5596b9bcc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1ab476e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 0x5596b9bc0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1ab476e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 0x5596b9bad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1ab476e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1aac1a80e0 con 0x5596b9bb4430 2024-04-01T01:34:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1ab476e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1aac1a9c20 con 0x5596b9baf2c0 2024-04-01T01:34:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1ab476e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1aac1a87b0 con 0x5596b9bae8d0 2024-04-01T01:34:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1aabfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 0x5596b9bad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1aabfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 0x5596b9bad0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58978/0 (socket says 172.21.15.70:58978) 2024-04-01T01:34:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1aabfff640 1 -- 172.21.15.70:0/3750509558 learned_addr learned my addr 172.21.15.70:0/3750509558 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1aaaffd640 1 --2- 172.21.15.70:0/3750509558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 0x5596b9bc0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1aab7fe640 1 --2- 172.21.15.70:0/3750509558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 0x5596b9bcc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1aabfff640 1 -- 172.21.15.70:0/3750509558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 msgr2=0x5596b9bcc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1aabfff640 1 --2- 172.21.15.70:0/3750509558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 0x5596b9bcc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1aabfff640 1 -- 172.21.15.70:0/3750509558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 msgr2=0x5596b9bc0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1aabfff640 1 --2- 172.21.15.70:0/3750509558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 0x5596b9bc0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1aabfff640 1 -- 172.21.15.70:0/3750509558 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596b9bc7380 con 0x5596b9bb4430 2024-04-01T01:34:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.970+0000 7f1aaaffd640 1 --2- 172.21.15.70:0/3750509558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 0x5596b9bc0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.971+0000 7f1aabfff640 1 --2- 172.21.15.70:0/3750509558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 0x5596b9bad0a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4009560 tx=0x7f1aa4010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2811c894aec8a4d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.971+0000 7f1a917fa640 1 -- 172.21.15.70:0/3750509558 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa40129a0 con 0x5596b9bb4430 2024-04-01T01:34:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.971+0000 7f1a917fa640 1 -- 172.21.15.70:0/3750509558 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1aa40120b0 con 0x5596b9bb4430 2024-04-01T01:34:37.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.971+0000 7f1a917fa640 1 -- 172.21.15.70:0/3750509558 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa402a8b0 con 0x5596b9bb4430 2024-04-01T01:34:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.972+0000 7f1ab476e740 1 -- 172.21.15.70:0/3750509558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 msgr2=0x5596b9bad0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.972+0000 7f1ab476e740 1 --2- 172.21.15.70:0/3750509558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 0x5596b9bad0a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4009560 tx=0x7f1aa4010f90 comp rx=0 tx=0).stop 2024-04-01T01:34:37.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.972+0000 7f1ab476e740 1 -- 172.21.15.70:0/3750509558 shutdown_connections 2024-04-01T01:34:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.972+0000 7f1ab476e740 1 --2- 172.21.15.70:0/3750509558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 0x5596b9bad0a0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.972+0000 7f1ab476e740 1 --2- 172.21.15.70:0/3750509558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 0x5596b9bc0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.972+0000 7f1ab476e740 1 --2- 172.21.15.70:0/3750509558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 0x5596b9bcc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.972+0000 7f1ab476e740 1 -- 172.21.15.70:0/3750509558 >> 172.21.15.70:0/3750509558 conn(0x5596b9ba9e40 msgr2=0x5596b9bb05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.972+0000 7f1ab476e740 1 -- 172.21.15.70:0/3750509558 shutdown_connections 2024-04-01T01:34:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.972+0000 7f1ab476e740 1 -- 172.21.15.70:0/3750509558 wait complete. 2024-04-01T01:34:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.972+0000 7f1ab476e740 1 -- start start 2024-04-01T01:34:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.972+0000 7f1ab476e740 1 Processor -- start 2024-04-01T01:34:37.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1ab476e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 0x5596b9bbde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1ab476e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 0x5596b9bbe3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1ab476e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 0x5596b9bbf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1ab476e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1aac1a9550 con 0x5596b9bb4430 2024-04-01T01:34:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1ab476e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1aac1a8e80 con 0x5596b9baf2c0 2024-04-01T01:34:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1ab476e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1aac1a60c0 con 0x5596b9bae8d0 2024-04-01T01:34:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1ab476e740 10 client.0 _ll_register_callbacks cb 0x7f1aa4012510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1aabfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 0x5596b9bbf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1aaaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 0x5596b9bbe3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1aabfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 0x5596b9bbf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58994/0 (socket says 172.21.15.70:58994) 2024-04-01T01:34:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1aabfff640 1 -- 172.21.15.70:0/2030919036 learned_addr learned my addr 172.21.15.70:0/2030919036 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:37.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1aab7fe640 1 --2- 172.21.15.70:0/2030919036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 0x5596b9bbde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1aabfff640 1 -- 172.21.15.70:0/2030919036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 msgr2=0x5596b9bbde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1aabfff640 1 --2- 172.21.15.70:0/2030919036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 0x5596b9bbde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1aabfff640 1 -- 172.21.15.70:0/2030919036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 msgr2=0x5596b9bbe3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1aabfff640 1 --2- 172.21.15.70:0/2030919036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 0x5596b9bbe3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1aabfff640 1 -- 172.21.15.70:0/2030919036 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596b9bbf9b0 con 0x5596b9bb4430 2024-04-01T01:34:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.973+0000 7f1aaaffd640 1 --2- 172.21.15.70:0/2030919036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 0x5596b9bbe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.974+0000 7f1aabfff640 1 --2- 172.21.15.70:0/2030919036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 0x5596b9bbf3c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40095d0 tx=0x7f1aa4020840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a16b8666b15bc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.974+0000 7f1a8affd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.974+0000 7f1a8affd640 1 -- 172.21.15.70:0/2030919036 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa40129a0 con 0x5596b9bb4430 2024-04-01T01:34:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.974+0000 7f1ab476e740 1 -- 172.21.15.70:0/2030919036 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1aa40120b0 con 0x5596b9bb4430 2024-04-01T01:34:37.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.974+0000 7f1a8affd640 1 -- 172.21.15.70:0/2030919036 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1aa402a630 con 0x5596b9bb4430 2024-04-01T01:34:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.974+0000 7f1a8affd640 1 -- 172.21.15.70:0/2030919036 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1aa404e050 con 0x5596b9bb4430 2024-04-01T01:34:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.974+0000 7f1ab476e740 10 client.15915 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.974+0000 7f1ab476e740 1 -- 172.21.15.70:0/2030919036 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5596b9c2b220 con 0x5596b9bb4430 2024-04-01T01:34:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.974+0000 7f1a8affd640 1 -- 172.21.15.70:0/2030919036 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1aa40120b0 con 0x5596b9bb4430 2024-04-01T01:34:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.974+0000 7f1a8affd640 1 -- 172.21.15.70:0/2030919036 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1aa4049030 con 0x5596b9bb4430 2024-04-01T01:34:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.975+0000 7f1ab476e740 10 client.15915 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.975+0000 7f1ab476e740 10 client.15915 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.975+0000 7f1ab476e740 20 client.15915 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.975+0000 7f1ab476e740 1 --2- 172.21.15.70:0/2030919036 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5596b9c2ba00 0x5596b9c2de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.975+0000 7f1ab476e740 4 client.15915 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:34:37.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.975+0000 7f1ab476e740 1 -- 172.21.15.70:0/2030919036 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5596b9b1fe30 con 0x5596b9c2ba00 2024-04-01T01:34:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.975+0000 7f1aab7fe640 1 --2- 172.21.15.70:0/2030919036 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5596b9c2ba00 0x5596b9c2de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.975+0000 7f1aab7fe640 1 --2- 172.21.15.70:0/2030919036 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5596b9c2ba00 0x5596b9c2de20 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c14868eaeca1aa08 server_cookie=4abbcedd35263623 in_seq=0 out_seq=0 2024-04-01T01:34:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:37.975+0000 7f1a8affd640 10 client.15915 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:37.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:37 smithi189 ceph-mon[35753]: pgmap v392: 129 pgs: 129 active+clean; 756 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 53 MiB/s wr, 45 op/s 2024-04-01T01:34:38.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:37 smithi070 ceph-mon[32179]: pgmap v392: 129 pgs: 129 active+clean; 756 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 53 MiB/s wr, 45 op/s 2024-04-01T01:34:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.004+0000 7f1a8affd640 1 -- 172.21.15.70:0/2030919036 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5596b9b1fe30 con 0x5596b9c2ba00 2024-04-01T01:34:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.004+0000 7f1a8affd640 10 client.15915 handle_command_reply: tid=0 2024-04-01T01:34:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.004+0000 7f1ab476e740 1 client.15915 shutdown 2024-04-01T01:34:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab476e740 1 -- 172.21.15.70:0/2030919036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 msgr2=0x5596b9bbf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab476e740 1 --2- 172.21.15.70:0/2030919036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 0x5596b9bbf3c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40095d0 tx=0x7f1aa4020840 comp rx=0 tx=0).stop 2024-04-01T01:34:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab476e740 1 -- 172.21.15.70:0/2030919036 shutdown_connections 2024-04-01T01:34:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab476e740 1 --2- 172.21.15.70:0/2030919036 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5596b9c2ba00 0x5596b9c2de20 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab476e740 1 --2- 172.21.15.70:0/2030919036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596b9bb4430 0x5596b9bbf3c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1a8affd640 0 client.15915 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab476e740 1 --2- 172.21.15.70:0/2030919036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596b9baf2c0 0x5596b9bbe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab476e740 1 --2- 172.21.15.70:0/2030919036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596b9bae8d0 0x5596b9bbde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab476e740 1 -- 172.21.15.70:0/2030919036 >> 172.21.15.70:0/2030919036 conn(0x5596b9ba9e40 msgr2=0x5596b9bad0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab476e740 1 -- 172.21.15.70:0/2030919036 shutdown_connections 2024-04-01T01:34:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab476e740 1 -- 172.21.15.70:0/2030919036 wait complete. 2024-04-01T01:34:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab476e740 20 client.15915 trim_cache size 0 max 16384 2024-04-01T01:34:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:34:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:34:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:34:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:34:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab1b8a640 1 -- 172.21.15.70:0/1664751321 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a940a25e0 msgr2=0x7f1a940a4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab1b8a640 1 --2- 172.21.15.70:0/1664751321 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a940a25e0 0x7f1a940a4aa0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4009a50 tx=0x7f1aa4009ea0 comp rx=0 tx=0).stop 2024-04-01T01:34:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab1b8a640 1 -- 172.21.15.70:0/1664751321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1aac101860 msgr2=0x7f1aac19c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab1b8a640 1 --2- 172.21.15.70:0/1664751321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1aac101860 0x7f1aac19c6e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0000c00 tx=0x7f1aa0004270 comp rx=0 tx=0).stop 2024-04-01T01:34:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab1b8a640 1 -- 172.21.15.70:0/1664751321 shutdown_connections 2024-04-01T01:34:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab1b8a640 1 --2- 172.21.15.70:0/1664751321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aac102220 0x7f1aac19ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab1b8a640 1 --2- 172.21.15.70:0/1664751321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1aac101860 0x7f1aac19c6e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab1b8a640 1 --2- 172.21.15.70:0/1664751321 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a940a25e0 0x7f1a940a4aa0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.005+0000 7f1ab1b8a640 1 --2- 172.21.15.70:0/1664751321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aac100670 0x7f1aac0698f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.006+0000 7f1ab1b8a640 1 -- 172.21.15.70:0/1664751321 >> 172.21.15.70:0/1664751321 conn(0x7f1aac0fbbe0 msgr2=0x7f1aac104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.006+0000 7f1ab1b8a640 1 -- 172.21.15.70:0/1664751321 shutdown_connections 2024-04-01T01:34:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:38.006+0000 7f1ab1b8a640 1 -- 172.21.15.70:0/1664751321 wait complete. 2024-04-01T01:34:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:37 smithi081 ceph-mon[35142]: pgmap v392: 129 pgs: 129 active+clean; 756 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 53 MiB/s wr, 45 op/s 2024-04-01T01:34:39.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:39 smithi189 ceph-mon[35753]: pgmap v393: 129 pgs: 129 active+clean; 772 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 51 MiB/s wr, 43 op/s 2024-04-01T01:34:40.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:39 smithi070 ceph-mon[32179]: pgmap v393: 129 pgs: 129 active+clean; 772 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 51 MiB/s wr, 43 op/s 2024-04-01T01:34:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:39 smithi081 ceph-mon[35142]: pgmap v393: 129 pgs: 129 active+clean; 772 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 51 MiB/s wr, 43 op/s 2024-04-01T01:34:40.767 INFO:teuthology.orchestra.run.smithi070.stderr:creating primary keys... 2024-04-01T01:34:41.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:41 smithi189 ceph-mon[35753]: pgmap v394: 129 pgs: 129 active+clean; 772 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 50 MiB/s wr, 40 op/s 2024-04-01T01:34:42.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:41 smithi070 ceph-mon[32179]: pgmap v394: 129 pgs: 129 active+clean; 772 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 50 MiB/s wr, 40 op/s 2024-04-01T01:34:42.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:41 smithi081 ceph-mon[35142]: pgmap v394: 129 pgs: 129 active+clean; 772 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 50 MiB/s wr, 40 op/s 2024-04-01T01:34:43.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:43 smithi189 ceph-mon[35753]: pgmap v395: 129 pgs: 129 active+clean; 807 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 56 MiB/s wr, 48 op/s 2024-04-01T01:34:44.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:43 smithi070 ceph-mon[32179]: pgmap v395: 129 pgs: 129 active+clean; 807 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 56 MiB/s wr, 48 op/s 2024-04-01T01:34:44.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:43 smithi081 ceph-mon[35142]: pgmap v395: 129 pgs: 129 active+clean; 807 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 56 MiB/s wr, 48 op/s 2024-04-01T01:34:45.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:45 smithi189 ceph-mon[35753]: pgmap v396: 129 pgs: 129 active+clean; 826 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 52 MiB/s wr, 50 op/s 2024-04-01T01:34:46.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:45 smithi070 ceph-mon[32179]: pgmap v396: 129 pgs: 129 active+clean; 826 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 52 MiB/s wr, 50 op/s 2024-04-01T01:34:46.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:45 smithi081 ceph-mon[35142]: pgmap v396: 129 pgs: 129 active+clean; 826 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 52 MiB/s wr, 50 op/s 2024-04-01T01:34:46.771 INFO:teuthology.orchestra.run.smithi070.stderr:done in 43.70 s (drop tables 0.01 s, create tables 0.06 s, client-side generate 30.48 s, vacuum 7.15 s, primary keys 6.00 s). 2024-04-01T01:34:46.775 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900' 2024-04-01T01:34:46.895 INFO:teuthology.orchestra.run.smithi070.stderr:starting vacuum...end. 2024-04-01T01:34:47.834 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:34:47.834 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:34:47.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.994+0000 7f8fda5e9640 1 -- 172.21.15.70:0/4145908059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd4108f60 msgr2=0x7f8fd41093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:47.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.994+0000 7f8fda5e9640 1 --2- 172.21.15.70:0/4145908059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd4108f60 0x7f8fd41093e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc009900 tx=0x7f8fbc02bc30 comp rx=0 tx=0).stop 2024-04-01T01:34:47.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.994+0000 7f8fda5e9640 1 -- 172.21.15.70:0/4145908059 shutdown_connections 2024-04-01T01:34:47.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.994+0000 7f8fda5e9640 1 --2- 172.21.15.70:0/4145908059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd4109920 0x7f8fd41101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:47.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.994+0000 7f8fda5e9640 1 --2- 172.21.15.70:0/4145908059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd4108f60 0x7f8fd41093e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:47.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.994+0000 7f8fda5e9640 1 --2- 172.21.15.70:0/4145908059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd4107d70 0x7f8fd4108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:47.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.994+0000 7f8fda5e9640 1 -- 172.21.15.70:0/4145908059 >> 172.21.15.70:0/4145908059 conn(0x7f8fd4076d10 msgr2=0x7f8fd4079150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:47.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.994+0000 7f8fda5e9640 1 -- 172.21.15.70:0/4145908059 shutdown_connections 2024-04-01T01:34:47.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.994+0000 7f8fda5e9640 1 -- 172.21.15.70:0/4145908059 wait complete. 2024-04-01T01:34:47.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.995+0000 7f8fda5e9640 1 Processor -- start 2024-04-01T01:34:47.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.995+0000 7f8fda5e9640 1 -- start start 2024-04-01T01:34:47.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.995+0000 7f8fda5e9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd4107d70 0x7f8fd419df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:47.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.995+0000 7f8fda5e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd4108f60 0x7f8fd419e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:47.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.995+0000 7f8fda5e9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd4109920 0x7f8fd41a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:47.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.995+0000 7f8fda5e9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd41138a0 con 0x7f8fd4108f60 2024-04-01T01:34:47.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.995+0000 7f8fda5e9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd4113720 con 0x7f8fd4107d70 2024-04-01T01:34:47.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.995+0000 7f8fda5e9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd4113a20 con 0x7f8fd4109920 2024-04-01T01:34:47.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.995+0000 7f8fcbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd4108f60 0x7f8fd419e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:47.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.995+0000 7f8fcbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd4108f60 0x7f8fd419e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41494/0 (socket says 172.21.15.70:41494) 2024-04-01T01:34:47.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.995+0000 7f8fcbfff640 1 -- 172.21.15.70:0/2958373470 learned_addr learned my addr 172.21.15.70:0/2958373470 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:47.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.995+0000 7f8fd3fff640 1 --2- 172.21.15.70:0/2958373470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd4107d70 0x7f8fd419df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:47.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.995+0000 7f8fd8b86640 1 --2- 172.21.15.70:0/2958373470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd4109920 0x7f8fd41a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:47.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.995+0000 7f8fd3fff640 1 -- 172.21.15.70:0/2958373470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd4109920 msgr2=0x7f8fd41a5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:47.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.996+0000 7f8fd3fff640 1 --2- 172.21.15.70:0/2958373470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd4109920 0x7f8fd41a5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:47.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.996+0000 7f8fd3fff640 1 -- 172.21.15.70:0/2958373470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd4108f60 msgr2=0x7f8fd419e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:47.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.996+0000 7f8fd3fff640 1 --2- 172.21.15.70:0/2958373470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd4108f60 0x7f8fd419e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:47.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.996+0000 7f8fd3fff640 1 -- 172.21.15.70:0/2958373470 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd41a5ba0 con 0x7f8fd4107d70 2024-04-01T01:34:47.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.996+0000 7f8fd8b86640 1 --2- 172.21.15.70:0/2958373470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd4109920 0x7f8fd41a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:34:47.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.996+0000 7f8fd3fff640 1 --2- 172.21.15.70:0/2958373470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd4107d70 0x7f8fd419df80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400e9c0 tx=0x7f8fc400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:47.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.996+0000 7f8fd1ffb640 1 -- 172.21.15.70:0/2958373470 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc400ccc0 con 0x7f8fd4107d70 2024-04-01T01:34:47.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.996+0000 7f8fda5e9640 1 -- 172.21.15.70:0/2958373470 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd41a5e90 con 0x7f8fd4107d70 2024-04-01T01:34:47.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.996+0000 7f8fda5e9640 1 -- 172.21.15.70:0/2958373470 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd41a6300 con 0x7f8fd4107d70 2024-04-01T01:34:47.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.996+0000 7f8fd1ffb640 1 -- 172.21.15.70:0/2958373470 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fc40035f0 con 0x7f8fd4107d70 2024-04-01T01:34:47.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.996+0000 7f8fd1ffb640 1 -- 172.21.15.70:0/2958373470 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc400ce60 con 0x7f8fd4107d70 2024-04-01T01:34:48.000 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:47 smithi189 ceph-mon[35753]: pgmap v397: 129 pgs: 129 active+clean; 886 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 55 MiB/s wr, 51 op/s 2024-04-01T01:34:48.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:47.999+0000 7f8fd1ffb640 1 -- 172.21.15.70:0/2958373470 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f8fc4010690 con 0x7f8fd4107d70 2024-04-01T01:34:48.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.000+0000 7f8fdd1d3740 1 Processor -- start 2024-04-01T01:34:48.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.000+0000 7f8fdd1d3740 1 -- start start 2024-04-01T01:34:48.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.000+0000 7f8fdd1d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240a92b0 0x5618240a96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.000+0000 7f8fdd1d3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a9ca0 0x5618240ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.000+0000 7f8fdd1d3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240aee10 0x5618240b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.000+0000 7f8fdd1d3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56182400e180 con 0x5618240a9ca0 2024-04-01T01:34:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.000+0000 7f8fdd1d3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561823fb6b70 con 0x5618240aee10 2024-04-01T01:34:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.000+0000 7f8fdd1d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56182400a220 con 0x5618240a92b0 2024-04-01T01:34:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.000+0000 7f8fd1ffb640 1 --2- 172.21.15.70:0/2958373470 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8fc00a25b0 0x7f8fc00a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.000+0000 7f8fd3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a9ca0 0x5618240ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.001+0000 7f8fd3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a9ca0 0x5618240ae7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41506/0 (socket says 172.21.15.70:41506) 2024-04-01T01:34:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.001+0000 7f8fd3fff640 1 -- 172.21.15.70:0/2731435958 learned_addr learned my addr 172.21.15.70:0/2731435958 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.001+0000 7f8fcbfff640 1 --2- 172.21.15.70:0/2731435958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240aee10 0x5618240b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.001+0000 7f8fd3fff640 1 -- 172.21.15.70:0/2731435958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240a92b0 msgr2=0x5618240a96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.001+0000 7f8fd3fff640 1 --2- 172.21.15.70:0/2731435958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240a92b0 0x5618240a96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.001+0000 7f8fd3fff640 1 -- 172.21.15.70:0/2731435958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240aee10 msgr2=0x5618240b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.001+0000 7f8fd3fff640 1 --2- 172.21.15.70:0/2731435958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240aee10 0x5618240b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.001+0000 7f8fd3fff640 1 -- 172.21.15.70:0/2731435958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561823fca7c0 con 0x5618240a9ca0 2024-04-01T01:34:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.001+0000 7f8fd8b86640 1 --2- 172.21.15.70:0/2958373470 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8fc00a25b0 0x7f8fc00a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.001+0000 7f8fd3fff640 1 --2- 172.21.15.70:0/2731435958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a9ca0 0x5618240ae7a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4017040 tx=0x7f8fc4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4edec69ecc4519a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.001+0000 7f8fca7fc640 1 -- 172.21.15.70:0/2731435958 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc409f840 con 0x5618240a9ca0 2024-04-01T01:34:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.002+0000 7f8fca7fc640 1 -- 172.21.15.70:0/2731435958 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fc40a90d0 con 0x5618240a9ca0 2024-04-01T01:34:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.002+0000 7f8fca7fc640 1 -- 172.21.15.70:0/2731435958 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4098b10 con 0x5618240a9ca0 2024-04-01T01:34:48.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.002+0000 7f8fcbfff640 1 --2- 172.21.15.70:0/2731435958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240aee10 0x5618240b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:34:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.002+0000 7f8fd1ffb640 1 -- 172.21.15.70:0/2958373470 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8fc40cf140 con 0x7f8fd4107d70 2024-04-01T01:34:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.002+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/2731435958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a9ca0 msgr2=0x5618240ae7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.002+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/2731435958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a9ca0 0x5618240ae7a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4017040 tx=0x7f8fc4014000 comp rx=0 tx=0).stop 2024-04-01T01:34:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.003+0000 7f8fd8b86640 1 --2- 172.21.15.70:0/2958373470 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8fc00a25b0 0x7f8fc00a4a70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc005c40 tx=0x7f8fcc0098e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.003+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/2731435958 shutdown_connections 2024-04-01T01:34:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.003+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/2731435958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240aee10 0x5618240b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.003+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/2731435958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a9ca0 0x5618240ae7a0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.003+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/2731435958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240a92b0 0x5618240a96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.003+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/2731435958 >> 172.21.15.70:0/2731435958 conn(0x5618240a4820 msgr2=0x5618240a4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.003+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/2731435958 shutdown_connections 2024-04-01T01:34:48.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:47 smithi070 ceph-mon[32179]: pgmap v397: 129 pgs: 129 active+clean; 886 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 55 MiB/s wr, 51 op/s 2024-04-01T01:34:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.003+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/2731435958 wait complete. 2024-04-01T01:34:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.003+0000 7f8fdd1d3740 1 -- start start 2024-04-01T01:34:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.004+0000 7f8fdd1d3740 1 Processor -- start 2024-04-01T01:34:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.004+0000 7f8fdd1d3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a92b0 0x5618240b37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.004+0000 7f8fd3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a92b0 0x5618240b37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.004+0000 7f8fd3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a92b0 0x5618240b37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41520/0 (socket says 172.21.15.70:41520) 2024-04-01T01:34:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.005+0000 7f8fd3fff640 1 -- 172.21.15.70:0/3363082475 learned_addr learned my addr 172.21.15.70:0/3363082475 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.005+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/3363082475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a9ca0 0x5618240b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.005+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/3363082475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240aee10 0x5618240bad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.005+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/3363082475 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5618240b1d60 con 0x5618240a92b0 2024-04-01T01:34:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.005+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/3363082475 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561823fac4a0 con 0x5618240a9ca0 2024-04-01T01:34:48.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.005+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/3363082475 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5618240b1ee0 con 0x5618240aee10 2024-04-01T01:34:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.005+0000 7f8fd8b86640 1 --2- 172.21.15.70:0/3363082475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240aee10 0x5618240bad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.005+0000 7f8fdd1d3740 10 client.0 _ll_register_callbacks cb 0x561823fafd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.005+0000 7f8fcbfff640 1 --2- 172.21.15.70:0/3363082475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a9ca0 0x5618240b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.005+0000 7f8fd3fff640 1 -- 172.21.15.70:0/3363082475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240aee10 msgr2=0x5618240bad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.005+0000 7f8fd3fff640 1 --2- 172.21.15.70:0/3363082475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240aee10 0x5618240bad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.005+0000 7f8fd3fff640 1 -- 172.21.15.70:0/3363082475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a9ca0 msgr2=0x5618240b3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.005+0000 7f8fd3fff640 1 --2- 172.21.15.70:0/3363082475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a9ca0 0x5618240b3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.005+0000 7f8fd3fff640 1 -- 172.21.15.70:0/3363082475 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561823dec7d0 con 0x5618240a92b0 2024-04-01T01:34:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.006+0000 7f8fd3fff640 1 --2- 172.21.15.70:0/3363082475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a92b0 0x5618240b37a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400c6a0 tx=0x7f8fc40039d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f4f83fbfdbfcd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.006+0000 7f8fb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.006+0000 7f8fb37fe640 1 -- 172.21.15.70:0/3363082475 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc409f840 con 0x5618240a92b0 2024-04-01T01:34:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.006+0000 7f8fb37fe640 1 -- 172.21.15.70:0/3363082475 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8fc40ab190 con 0x5618240a92b0 2024-04-01T01:34:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.006+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/3363082475 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fc40a90d0 con 0x5618240a92b0 2024-04-01T01:34:48.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.007+0000 7f8fb37fe640 1 -- 172.21.15.70:0/3363082475 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8fc409cba0 con 0x5618240a92b0 2024-04-01T01:34:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.007+0000 7f8fdd1d3740 10 client.15927 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.007+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/3363082475 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561823d223f0 con 0x5618240a92b0 2024-04-01T01:34:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.007+0000 7f8fb37fe640 1 -- 172.21.15.70:0/3363082475 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fc40ab390 con 0x5618240a92b0 2024-04-01T01:34:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.008+0000 7f8fb37fe640 1 -- 172.21.15.70:0/3363082475 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8fc40aa070 con 0x5618240a92b0 2024-04-01T01:34:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.008+0000 7f8fdd1d3740 10 client.15927 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.008+0000 7f8fdd1d3740 10 client.15927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.008+0000 7f8fdd1d3740 20 client.15927 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.008+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/3363082475 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5618240bf4c0 0x5618240c1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.008+0000 7f8fdd1d3740 4 client.15927 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.008+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/3363082475 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8fc40a90d0 con 0x5618240bf4c0 2024-04-01T01:34:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.008+0000 7f8fcbfff640 1 --2- 172.21.15.70:0/3363082475 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5618240bf4c0 0x5618240c1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.009+0000 7f8fb37fe640 10 client.15927 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:48.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.009+0000 7f8fcbfff640 1 --2- 172.21.15.70:0/3363082475 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5618240bf4c0 0x5618240c1920 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9a22d13a5f50f63 server_cookie=4211479970a34797 in_seq=0 out_seq=0 2024-04-01T01:34:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.010+0000 7f8fb37fe640 1 -- 172.21.15.70:0/3363082475 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f8fc40a90d0 con 0x5618240bf4c0 2024-04-01T01:34:48.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.010+0000 7f8fb37fe640 10 client.15927 handle_command_reply: tid=0 2024-04-01T01:34:48.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.010+0000 7f8fdd1d3740 1 client.15927 shutdown 2024-04-01T01:34:48.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.010+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/3363082475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a92b0 msgr2=0x5618240b37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.010+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/3363082475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a92b0 0x5618240b37a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400c6a0 tx=0x7f8fc40039d0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.010+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/3363082475 shutdown_connections 2024-04-01T01:34:48.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.010+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/3363082475 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5618240bf4c0 0x5618240c1920 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.010+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/3363082475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240aee10 0x5618240bad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.010+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/3363082475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a9ca0 0x5618240b3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.010+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/3363082475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a92b0 0x5618240b37a0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.010+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/3363082475 >> 172.21.15.70:0/3363082475 conn(0x5618240a4820 msgr2=0x5618240a3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.010+0000 7f8fb37fe640 0 client.15927 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:48.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.010+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/3363082475 shutdown_connections 2024-04-01T01:34:48.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.010+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/3363082475 wait complete. 2024-04-01T01:34:48.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.010+0000 7f8fdd1d3740 20 client.15927 trim_cache size 0 max 16384 2024-04-01T01:34:48.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.041+0000 7f8fdd1d3740 1 Processor -- start 2024-04-01T01:34:48.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.041+0000 7f8fdd1d3740 1 -- start start 2024-04-01T01:34:48.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.041+0000 7f8fdd1d3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a92b0 0x5618240a3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fdd1d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240a9ca0 0x5618240a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fdd1d3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240aee10 0x5618240b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fdd1d3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb400fd40 con 0x5618240aee10 2024-04-01T01:34:48.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fdd1d3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb4011880 con 0x5618240a92b0 2024-04-01T01:34:48.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fdd1d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb4010410 con 0x5618240a9ca0 2024-04-01T01:34:48.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fd3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a92b0 0x5618240a3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fd8b86640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240aee10 0x5618240b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fd3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a92b0 0x5618240a3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46320/0 (socket says 172.21.15.70:46320) 2024-04-01T01:34:48.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fd3fff640 1 -- 172.21.15.70:0/1196605482 learned_addr learned my addr 172.21.15.70:0/1196605482 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:48.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fcbfff640 1 --2- 172.21.15.70:0/1196605482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240a9ca0 0x5618240a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fd8b86640 1 -- 172.21.15.70:0/1196605482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240a9ca0 msgr2=0x5618240a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fd8b86640 1 --2- 172.21.15.70:0/1196605482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240a9ca0 0x5618240a38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fd8b86640 1 -- 172.21.15.70:0/1196605482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a92b0 msgr2=0x5618240a3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fd8b86640 1 --2- 172.21.15.70:0/1196605482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a92b0 0x5618240a3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fd8b86640 1 -- 172.21.15.70:0/1196605482 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561823ce20f0 con 0x5618240aee10 2024-04-01T01:34:48.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fd3fff640 1 --2- 172.21.15.70:0/1196605482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a92b0 0x5618240a3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:48.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.042+0000 7f8fd8b86640 1 --2- 172.21.15.70:0/1196605482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240aee10 0x5618240b2dc0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009e60 tx=0x7f8fcc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b06e6d1601c08c86 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.043+0000 7f8fc9ffb640 1 -- 172.21.15.70:0/1196605482 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc010d60 con 0x5618240aee10 2024-04-01T01:34:48.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.043+0000 7f8fc9ffb640 1 -- 172.21.15.70:0/1196605482 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fcc02b070 con 0x5618240aee10 2024-04-01T01:34:48.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.043+0000 7f8fc9ffb640 1 -- 172.21.15.70:0/1196605482 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc026c50 con 0x5618240aee10 2024-04-01T01:34:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.044+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/1196605482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240aee10 msgr2=0x5618240b2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.044+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/1196605482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240aee10 0x5618240b2dc0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009e60 tx=0x7f8fcc017000 comp rx=0 tx=0).stop 2024-04-01T01:34:48.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.044+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/1196605482 shutdown_connections 2024-04-01T01:34:48.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.044+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/1196605482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240aee10 0x5618240b2dc0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.044+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/1196605482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240a9ca0 0x5618240a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.044+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/1196605482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a92b0 0x5618240a3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.044+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/1196605482 >> 172.21.15.70:0/1196605482 conn(0x5618240a4820 msgr2=0x5618240a4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.044+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/1196605482 shutdown_connections 2024-04-01T01:34:48.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.044+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/1196605482 wait complete. 2024-04-01T01:34:48.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.044+0000 7f8fdd1d3740 1 -- start start 2024-04-01T01:34:48.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.045+0000 7f8fdd1d3740 1 Processor -- start 2024-04-01T01:34:48.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.045+0000 7f8fdd1d3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a92b0 0x5618240c6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.045+0000 7f8fdd1d3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a9ca0 0x5618240c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.045+0000 7f8fdd1d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240aee10 0x561824123790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.045+0000 7f8fdd1d3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb40111b0 con 0x5618240a9ca0 2024-04-01T01:34:48.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.045+0000 7f8fdd1d3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb4010ae0 con 0x5618240a92b0 2024-04-01T01:34:48.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.045+0000 7f8fdd1d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb400dd20 con 0x5618240aee10 2024-04-01T01:34:48.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.045+0000 7f8fcbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a9ca0 0x5618240c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.045+0000 7f8fd3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a92b0 0x5618240c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.045+0000 7f8fdd1d3740 10 client.0 _ll_register_callbacks cb 0x7f8fc409be90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.045+0000 7f8fcbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a9ca0 0x5618240c6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41540/0 (socket says 172.21.15.70:41540) 2024-04-01T01:34:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.045+0000 7f8fcbfff640 1 -- 172.21.15.70:0/2075553983 learned_addr learned my addr 172.21.15.70:0/2075553983 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.045+0000 7f8fd8b86640 1 --2- 172.21.15.70:0/2075553983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240aee10 0x561824123790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.046+0000 7f8fcbfff640 1 -- 172.21.15.70:0/2075553983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240aee10 msgr2=0x561824123790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.046+0000 7f8fcbfff640 1 --2- 172.21.15.70:0/2075553983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240aee10 0x561824123790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.046+0000 7f8fcbfff640 1 -- 172.21.15.70:0/2075553983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a92b0 msgr2=0x5618240c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.046+0000 7f8fcbfff640 1 --2- 172.21.15.70:0/2075553983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a92b0 0x5618240c6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.046+0000 7f8fcbfff640 1 -- 172.21.15.70:0/2075553983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561824123cd0 con 0x5618240a9ca0 2024-04-01T01:34:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.046+0000 7f8fcbfff640 1 --2- 172.21.15.70:0/2075553983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a9ca0 0x5618240c6a50 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc006010 tx=0x7f8fbc02c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784f157e3eddb5b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.046+0000 7f8fb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.046+0000 7f8fb37fe640 1 -- 172.21.15.70:0/2075553983 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc04f020 con 0x5618240a9ca0 2024-04-01T01:34:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.046+0000 7f8fb37fe640 1 -- 172.21.15.70:0/2075553983 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8fbc004250 con 0x5618240a9ca0 2024-04-01T01:34:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.046+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/2075553983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fcc02b070 con 0x5618240a9ca0 2024-04-01T01:34:48.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.047+0000 7f8fb37fe640 1 -- 172.21.15.70:0/2075553983 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8fbc057050 con 0x5618240a9ca0 2024-04-01T01:34:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.047+0000 7f8fdd1d3740 10 client.15933 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.047+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/2075553983 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561824125090 con 0x5618240a9ca0 2024-04-01T01:34:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.047+0000 7f8fb37fe640 1 -- 172.21.15.70:0/2075553983 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8fcc02b070 con 0x5618240a9ca0 2024-04-01T01:34:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.047+0000 7f8fb37fe640 1 -- 172.21.15.70:0/2075553983 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8fbc004a00 con 0x5618240a9ca0 2024-04-01T01:34:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.047+0000 7f8fdd1d3740 10 client.15933 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.047+0000 7f8fdd1d3740 10 client.15933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.047+0000 7f8fdd1d3740 20 client.15933 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.047+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/2075553983 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561824125ac0 0x561824127ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.047+0000 7f8fdd1d3740 4 client.15933 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:34:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.047+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/2075553983 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56182401b6c0 con 0x561824125ac0 2024-04-01T01:34:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.047+0000 7f8fd3fff640 1 --2- 172.21.15.70:0/2075553983 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561824125ac0 0x561824127ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.048+0000 7f8fb37fe640 10 client.15933 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:48.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.048+0000 7f8fd3fff640 1 --2- 172.21.15.70:0/2075553983 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561824125ac0 0x561824127ee0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d50a53125d81e5dd server_cookie=de7a99fc0c402d30 in_seq=0 out_seq=0 2024-04-01T01:34:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.048+0000 7f8fb37fe640 1 -- 172.21.15.70:0/2075553983 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56182401b6c0 con 0x561824125ac0 2024-04-01T01:34:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.048+0000 7f8fb37fe640 10 client.15933 handle_command_reply: tid=0 2024-04-01T01:34:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.048+0000 7f8fdd1d3740 1 client.15933 shutdown 2024-04-01T01:34:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.048+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/2075553983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a9ca0 msgr2=0x5618240c6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.048+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/2075553983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a9ca0 0x5618240c6a50 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc006010 tx=0x7f8fbc02c330 comp rx=0 tx=0).stop 2024-04-01T01:34:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.049+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/2075553983 shutdown_connections 2024-04-01T01:34:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.049+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/2075553983 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561824125ac0 0x561824127ee0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.049+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/2075553983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618240aee10 0x561824123790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.049+0000 7f8fb37fe640 0 client.15933 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:48.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.049+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/2075553983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618240a9ca0 0x5618240c6a50 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.049+0000 7f8fdd1d3740 1 --2- 172.21.15.70:0/2075553983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618240a92b0 0x5618240c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.049+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/2075553983 >> 172.21.15.70:0/2075553983 conn(0x5618240a4820 msgr2=0x5618240a3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.049+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/2075553983 shutdown_connections 2024-04-01T01:34:48.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.049+0000 7f8fdd1d3740 1 -- 172.21.15.70:0/2075553983 wait complete. 2024-04-01T01:34:48.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.049+0000 7f8fdd1d3740 20 client.15933 trim_cache size 0 max 16384 2024-04-01T01:34:48.053 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:34:48.053 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:34:48.053 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:34:48.053 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:34:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.050+0000 7f8fda5e9640 1 -- 172.21.15.70:0/2958373470 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8fc00a25b0 msgr2=0x7f8fc00a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.050+0000 7f8fda5e9640 1 --2- 172.21.15.70:0/2958373470 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8fc00a25b0 0x7f8fc00a4a70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc005c40 tx=0x7f8fcc0098e0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.050+0000 7f8fda5e9640 1 -- 172.21.15.70:0/2958373470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd4107d70 msgr2=0x7f8fd419df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.050+0000 7f8fda5e9640 1 --2- 172.21.15.70:0/2958373470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd4107d70 0x7f8fd419df80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400e9c0 tx=0x7f8fc400ee80 comp rx=0 tx=0).stop 2024-04-01T01:34:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.050+0000 7f8fda5e9640 1 -- 172.21.15.70:0/2958373470 shutdown_connections 2024-04-01T01:34:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.051+0000 7f8fda5e9640 1 --2- 172.21.15.70:0/2958373470 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8fc00a25b0 0x7f8fc00a4a70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.051+0000 7f8fda5e9640 1 --2- 172.21.15.70:0/2958373470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fd4109920 0x7f8fd41a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.051+0000 7f8fda5e9640 1 --2- 172.21.15.70:0/2958373470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fd4108f60 0x7f8fd419e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.051+0000 7f8fda5e9640 1 --2- 172.21.15.70:0/2958373470 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd4107d70 0x7f8fd419df80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.051+0000 7f8fda5e9640 1 -- 172.21.15.70:0/2958373470 >> 172.21.15.70:0/2958373470 conn(0x7f8fd4076d10 msgr2=0x7f8fd410c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.051+0000 7f8fda5e9640 1 -- 172.21.15.70:0/2958373470 shutdown_connections 2024-04-01T01:34:48.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.051+0000 7f8fda5e9640 1 -- 172.21.15.70:0/2958373470 wait complete. 2024-04-01T01:34:48.071 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:34:48.071 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:34:48.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:47 smithi081 ceph-mon[35142]: pgmap v397: 129 pgs: 129 active+clean; 886 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 55 MiB/s wr, 51 op/s 2024-04-01T01:34:48.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.216+0000 7f5ddd4a4640 1 -- 172.21.15.70:0/3736841001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd8103340 msgr2=0x7f5dd8109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.216+0000 7f5ddd4a4640 1 --2- 172.21.15.70:0/3736841001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd8103340 0x7f5dd8109bd0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009ca0 tx=0x7f5dd002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.216+0000 7f5ddd4a4640 1 -- 172.21.15.70:0/3736841001 shutdown_connections 2024-04-01T01:34:48.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.216+0000 7f5ddd4a4640 1 --2- 172.21.15.70:0/3736841001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd8103340 0x7f5dd8109bd0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.216+0000 7f5ddd4a4640 1 --2- 172.21.15.70:0/3736841001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dd8102980 0x7f5dd8102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.216+0000 7f5ddd4a4640 1 --2- 172.21.15.70:0/3736841001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5dd8101790 0x7f5dd8101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.216+0000 7f5ddd4a4640 1 -- 172.21.15.70:0/3736841001 >> 172.21.15.70:0/3736841001 conn(0x7f5dd80fcd00 msgr2=0x7f5dd80ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.216+0000 7f5ddd4a4640 1 -- 172.21.15.70:0/3736841001 shutdown_connections 2024-04-01T01:34:48.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.216+0000 7f5ddd4a4640 1 -- 172.21.15.70:0/3736841001 wait complete. 2024-04-01T01:34:48.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.216+0000 7f5ddd4a4640 1 Processor -- start 2024-04-01T01:34:48.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5ddd4a4640 1 -- start start 2024-04-01T01:34:48.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5ddd4a4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5dd8101790 0x7f5dd8197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5ddd4a4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dd8102980 0x7f5dd8197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5ddd4a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd8103340 0x7f5dd819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5ddd4a4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd810d2c0 con 0x7f5dd8103340 2024-04-01T01:34:48.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5ddd4a4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd810d140 con 0x7f5dd8101790 2024-04-01T01:34:48.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5ddd4a4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd810d440 con 0x7f5dd8102980 2024-04-01T01:34:48.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5dd77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd8103340 0x7f5dd819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5dd77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd8103340 0x7f5dd819ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41552/0 (socket says 172.21.15.70:41552) 2024-04-01T01:34:48.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5dd6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5dd8101790 0x7f5dd8197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5dd77fe640 1 -- 172.21.15.70:0/328550760 learned_addr learned my addr 172.21.15.70:0/328550760 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:48.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5dd67fc640 1 --2- 172.21.15.70:0/328550760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dd8102980 0x7f5dd8197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5dd77fe640 1 -- 172.21.15.70:0/328550760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dd8102980 msgr2=0x7f5dd8197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5dd77fe640 1 --2- 172.21.15.70:0/328550760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dd8102980 0x7f5dd8197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5dd77fe640 1 -- 172.21.15.70:0/328550760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5dd8101790 msgr2=0x7f5dd8197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5dd77fe640 1 --2- 172.21.15.70:0/328550760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5dd8101790 0x7f5dd8197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5dd77fe640 1 -- 172.21.15.70:0/328550760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd819f580 con 0x7f5dd8103340 2024-04-01T01:34:48.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.217+0000 7f5dd77fe640 1 --2- 172.21.15.70:0/328550760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd8103340 0x7f5dd819ef10 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5dd002c700 tx=0x7f5dd00328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.218+0000 7f5dbffff640 1 -- 172.21.15.70:0/328550760 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0041020 con 0x7f5dd8103340 2024-04-01T01:34:48.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.218+0000 7f5ddd4a4640 1 -- 172.21.15.70:0/328550760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd819f810 con 0x7f5dd8103340 2024-04-01T01:34:48.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.218+0000 7f5dbffff640 1 -- 172.21.15.70:0/328550760 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5dd0003e20 con 0x7f5dd8103340 2024-04-01T01:34:48.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.218+0000 7f5dbffff640 1 -- 172.21.15.70:0/328550760 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0039040 con 0x7f5dd8103340 2024-04-01T01:34:48.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.218+0000 7f5ddd4a4640 1 -- 172.21.15.70:0/328550760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd819fd50 con 0x7f5dd8103340 2024-04-01T01:34:48.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.219+0000 7f5ddd4a4640 1 -- 172.21.15.70:0/328550760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dd8108b30 con 0x7f5dd8103340 2024-04-01T01:34:48.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.220+0000 7f5dbffff640 1 -- 172.21.15.70:0/328550760 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7f5dd0044020 con 0x7f5dd8103340 2024-04-01T01:34:48.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.221+0000 7f5dbffff640 1 --2- 172.21.15.70:0/328550760 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5dc00a2530 0x7f5dc00a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.221+0000 7f5dd6ffd640 1 --2- 172.21.15.70:0/328550760 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5dc00a2530 0x7f5dc00a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.221+0000 7f5dbffff640 1 -- 172.21.15.70:0/328550760 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5dd00f0cf0 con 0x7f5dd8103340 2024-04-01T01:34:48.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.221+0000 7f5dd6ffd640 1 --2- 172.21.15.70:0/328550760 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5dc00a2530 0x7f5dc00a49f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5dd81027e0 tx=0x7f5dc8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.225+0000 7f5dbffff640 1 -- 172.21.15.70:0/328550760 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5dd004b020 con 0x7f5dd8103340 2024-04-01T01:34:48.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.480+0000 7f5ddd4a4640 1 -- 172.21.15.70:0/328550760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5dd802d340 con 0x7f5dd8103340 2024-04-01T01:34:48.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.481+0000 7f5dbffff640 1 -- 172.21.15.70:0/328550760 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f5dd00bda90 con 0x7f5dd8103340 2024-04-01T01:34:48.482 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:34:48.482 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:33:36.894838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24418,24404,24379,24451,24464]},"id":1}]} 2024-04-01T01:34:48.483 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-04-01T01:34:48.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.484+0000 7f5ddd4a4640 1 -- 172.21.15.70:0/328550760 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5dc00a2530 msgr2=0x7f5dc00a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.484+0000 7f5ddd4a4640 1 --2- 172.21.15.70:0/328550760 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5dc00a2530 0x7f5dc00a49f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5dd81027e0 tx=0x7f5dc8005f90 comp rx=0 tx=0).stop 2024-04-01T01:34:48.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.484+0000 7f5ddd4a4640 1 -- 172.21.15.70:0/328550760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd8103340 msgr2=0x7f5dd819ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.484+0000 7f5ddd4a4640 1 --2- 172.21.15.70:0/328550760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd8103340 0x7f5dd819ef10 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5dd002c700 tx=0x7f5dd00328a0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.484+0000 7f5ddd4a4640 1 -- 172.21.15.70:0/328550760 shutdown_connections 2024-04-01T01:34:48.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.484+0000 7f5ddd4a4640 1 --2- 172.21.15.70:0/328550760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd8103340 0x7f5dd819ef10 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.484+0000 7f5ddd4a4640 1 --2- 172.21.15.70:0/328550760 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5dc00a2530 0x7f5dc00a49f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.484+0000 7f5ddd4a4640 1 --2- 172.21.15.70:0/328550760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dd8102980 0x7f5dd8197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.484+0000 7f5ddd4a4640 1 --2- 172.21.15.70:0/328550760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5dd8101790 0x7f5dd8197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.484+0000 7f5ddd4a4640 1 -- 172.21.15.70:0/328550760 >> 172.21.15.70:0/328550760 conn(0x7f5dd80fcd00 msgr2=0x7f5dd8105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.485+0000 7f5ddd4a4640 1 -- 172.21.15.70:0/328550760 shutdown_connections 2024-04-01T01:34:48.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.485+0000 7f5ddd4a4640 1 -- 172.21.15.70:0/328550760 wait complete. 2024-04-01T01:34:48.497 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:34:48.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.676+0000 7fee51ac9640 1 -- 172.21.15.70:0/388500565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee4c0ffe40 msgr2=0x7fee4c106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.676+0000 7fee51ac9640 1 --2- 172.21.15.70:0/388500565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee4c0ffe40 0x7fee4c106e40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009900 tx=0x7fee3c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:34:48.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.676+0000 7fee51ac9640 1 -- 172.21.15.70:0/388500565 shutdown_connections 2024-04-01T01:34:48.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.676+0000 7fee51ac9640 1 --2- 172.21.15.70:0/388500565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee4c1073d0 0x7fee4c109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.676+0000 7fee51ac9640 1 --2- 172.21.15.70:0/388500565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee4c0ffe40 0x7fee4c106e40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.676+0000 7fee51ac9640 1 --2- 172.21.15.70:0/388500565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee4c0ff4e0 0x7fee4c0ff900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.676+0000 7fee51ac9640 1 -- 172.21.15.70:0/388500565 >> 172.21.15.70:0/388500565 conn(0x7fee4c0faf10 msgr2=0x7fee4c0fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.676+0000 7fee51ac9640 1 -- 172.21.15.70:0/388500565 shutdown_connections 2024-04-01T01:34:48.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.676+0000 7fee51ac9640 1 -- 172.21.15.70:0/388500565 wait complete. 2024-04-01T01:34:48.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.677+0000 7fee51ac9640 1 Processor -- start 2024-04-01T01:34:48.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.677+0000 7fee51ac9640 1 -- start start 2024-04-01T01:34:48.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.678+0000 7fee51ac9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee4c0ff4e0 0x7fee4c197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.678+0000 7fee51ac9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee4c0ffe40 0x7fee4c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.678+0000 7fee51ac9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee4c1073d0 0x7fee4c19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.678+0000 7fee4b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee4c0ff4e0 0x7fee4c197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.678+0000 7fee4b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee4c0ff4e0 0x7fee4c197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39018/0 (socket says 172.21.15.70:39018) 2024-04-01T01:34:48.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.678+0000 7fee4b7fe640 1 -- 172.21.15.70:0/3794406611 learned_addr learned my addr 172.21.15.70:0/3794406611 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:48.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.678+0000 7fee51ac9640 1 -- 172.21.15.70:0/3794406611 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c10d2b0 con 0x7fee4c0ffe40 2024-04-01T01:34:48.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.678+0000 7fee4bfff640 1 --2- 172.21.15.70:0/3794406611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee4c1073d0 0x7fee4c19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.678+0000 7fee51ac9640 1 -- 172.21.15.70:0/3794406611 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c10d130 con 0x7fee4c1073d0 2024-04-01T01:34:48.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.678+0000 7fee51ac9640 1 -- 172.21.15.70:0/3794406611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c10d430 con 0x7fee4c0ff4e0 2024-04-01T01:34:48.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.679+0000 7fee4b7fe640 1 -- 172.21.15.70:0/3794406611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee4c1073d0 msgr2=0x7fee4c19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.679+0000 7fee4b7fe640 1 --2- 172.21.15.70:0/3794406611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee4c1073d0 0x7fee4c19ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.679+0000 7fee4b7fe640 1 -- 172.21.15.70:0/3794406611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee4c0ffe40 msgr2=0x7fee4c197f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:34:48.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.679+0000 7fee4b7fe640 1 --2- 172.21.15.70:0/3794406611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee4c0ffe40 0x7fee4c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.679+0000 7fee4b7fe640 1 -- 172.21.15.70:0/3794406611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee4c19f5f0 con 0x7fee4c0ff4e0 2024-04-01T01:34:48.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.679+0000 7fee4bfff640 1 --2- 172.21.15.70:0/3794406611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee4c1073d0 0x7fee4c19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:48.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.679+0000 7fee4b7fe640 1 --2- 172.21.15.70:0/3794406611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee4c0ff4e0 0x7fee4c197a20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fee3400d950 tx=0x7fee3400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.680+0000 7fee497fa640 1 -- 172.21.15.70:0/3794406611 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee34017070 con 0x7fee4c0ff4e0 2024-04-01T01:34:48.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.680+0000 7fee497fa640 1 -- 172.21.15.70:0/3794406611 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee34009c30 con 0x7fee4c0ff4e0 2024-04-01T01:34:48.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.680+0000 7fee497fa640 1 -- 172.21.15.70:0/3794406611 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3400f620 con 0x7fee4c0ff4e0 2024-04-01T01:34:48.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.680+0000 7fee51ac9640 1 -- 172.21.15.70:0/3794406611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee4c19f8e0 con 0x7fee4c0ff4e0 2024-04-01T01:34:48.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.680+0000 7fee51ac9640 1 -- 172.21.15.70:0/3794406611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee4c19fde0 con 0x7fee4c0ff4e0 2024-04-01T01:34:48.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.683+0000 7fee497fa640 1 -- 172.21.15.70:0/3794406611 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 137371+0+0 (secure 0 0 0) 0x7fee340109a0 con 0x7fee4c0ff4e0 2024-04-01T01:34:48.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.684+0000 7fee497fa640 1 --2- 172.21.15.70:0/3794406611 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee380a2680 0x7fee380a4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.684+0000 7fee497fa640 1 -- 172.21.15.70:0/3794406611 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fee340ce860 con 0x7fee4c0ff4e0 2024-04-01T01:34:48.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.685+0000 7fee42ffd640 1 --2- 172.21.15.70:0/3794406611 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee380a2680 0x7fee380a4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.685+0000 7fee54785740 1 Processor -- start 2024-04-01T01:34:48.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.685+0000 7fee54785740 1 -- start start 2024-04-01T01:34:48.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.685+0000 7fee54785740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af165270 0x5580af165690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.685+0000 7fee54785740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165c60 0x5580af16a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.685+0000 7fee54785740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af16a860 0x5580af16ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.685+0000 7fee54785740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5580af0c11b0 con 0x5580af165c60 2024-04-01T01:34:48.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.685+0000 7fee54785740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5580af069c00 con 0x5580af16a860 2024-04-01T01:34:48.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.685+0000 7fee54785740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5580af0bd250 con 0x5580af165270 2024-04-01T01:34:48.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.685+0000 7fee4b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af165270 0x5580af165690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.686+0000 7fee4b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af165270 0x5580af165690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39032/0 (socket says 172.21.15.70:39032) 2024-04-01T01:34:48.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.686+0000 7fee4bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af16a860 0x5580af16ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.686+0000 7fee4b7fe640 1 -- 172.21.15.70:0/3202551607 learned_addr learned my addr 172.21.15.70:0/3202551607 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:48.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.686+0000 7fee42ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165c60 0x5580af16a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.686+0000 7fee42ffd640 1 --2- 172.21.15.70:0/3794406611 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee380a2680 0x7fee380a4b40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fee3c036040 tx=0x7fee3c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.686+0000 7fee42ffd640 1 -- 172.21.15.70:0/3202551607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af165270 msgr2=0x5580af165690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.686+0000 7fee42ffd640 1 --2- 172.21.15.70:0/3202551607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af165270 0x5580af165690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.686+0000 7fee42ffd640 1 -- 172.21.15.70:0/3202551607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af16a860 msgr2=0x5580af16ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.686+0000 7fee42ffd640 1 --2- 172.21.15.70:0/3202551607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af16a860 0x5580af16ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.686+0000 7fee42ffd640 1 -- 172.21.15.70:0/3202551607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580af0893d0 con 0x5580af165c60 2024-04-01T01:34:48.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.686+0000 7fee4bfff640 1 --2- 172.21.15.70:0/3202551607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af16a860 0x5580af16ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:48.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.687+0000 7fee42ffd640 1 --2- 172.21.15.70:0/3202551607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165c60 0x5580af16a1f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fee3c02c840 tx=0x7fee3c043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4254fdf8aa88ddf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.687+0000 7fee41ffb640 1 -- 172.21.15.70:0/3202551607 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c005050 con 0x5580af165c60 2024-04-01T01:34:48.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.687+0000 7fee41ffb640 1 -- 172.21.15.70:0/3202551607 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee3c04d070 con 0x5580af165c60 2024-04-01T01:34:48.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.687+0000 7fee41ffb640 1 -- 172.21.15.70:0/3202551607 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c0035f0 con 0x5580af165c60 2024-04-01T01:34:48.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.688+0000 7fee54785740 1 -- 172.21.15.70:0/3202551607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165c60 msgr2=0x5580af16a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.688+0000 7fee54785740 1 --2- 172.21.15.70:0/3202551607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165c60 0x5580af16a1f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fee3c02c840 tx=0x7fee3c043000 comp rx=0 tx=0).stop 2024-04-01T01:34:48.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.688+0000 7fee54785740 1 -- 172.21.15.70:0/3202551607 shutdown_connections 2024-04-01T01:34:48.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.688+0000 7fee54785740 1 --2- 172.21.15.70:0/3202551607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af16a860 0x5580af16ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.688+0000 7fee54785740 1 --2- 172.21.15.70:0/3202551607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165c60 0x5580af16a1f0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.688+0000 7fee54785740 1 --2- 172.21.15.70:0/3202551607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af165270 0x5580af165690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.688+0000 7fee54785740 1 -- 172.21.15.70:0/3202551607 >> 172.21.15.70:0/3202551607 conn(0x5580af157980 msgr2=0x5580af157d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.688+0000 7fee54785740 1 -- 172.21.15.70:0/3202551607 shutdown_connections 2024-04-01T01:34:48.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.688+0000 7fee54785740 1 -- 172.21.15.70:0/3202551607 wait complete. 2024-04-01T01:34:48.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.689+0000 7fee54785740 1 -- start start 2024-04-01T01:34:48.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.689+0000 7fee54785740 1 Processor -- start 2024-04-01T01:34:48.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.690+0000 7fee54785740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165270 0x5580af16f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.690+0000 7fee54785740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165c60 0x5580af16f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.690+0000 7fee54785740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af16a860 0x5580af176780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.690+0000 7fee54785740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5580af16d7b0 con 0x5580af165c60 2024-04-01T01:34:48.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.690+0000 7fee54785740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5580af05f470 con 0x5580af165270 2024-04-01T01:34:48.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.690+0000 7fee54785740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5580af16d930 con 0x5580af16a860 2024-04-01T01:34:48.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.690+0000 7fee4b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165270 0x5580af16f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.690+0000 7fee4b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165270 0x5580af16f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46400/0 (socket says 172.21.15.70:46400) 2024-04-01T01:34:48.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.690+0000 7fee4b7fe640 1 -- 172.21.15.70:0/638668529 learned_addr learned my addr 172.21.15.70:0/638668529 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:48.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.690+0000 7fee4bfff640 1 --2- 172.21.15.70:0/638668529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af16a860 0x5580af176780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.690+0000 7fee54785740 10 client.0 _ll_register_callbacks cb 0x5580af07ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:48.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.690+0000 7fee42ffd640 1 --2- 172.21.15.70:0/638668529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165c60 0x5580af16f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.690+0000 7fee4b7fe640 1 -- 172.21.15.70:0/638668529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af16a860 msgr2=0x5580af176780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.690+0000 7fee4b7fe640 1 --2- 172.21.15.70:0/638668529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af16a860 0x5580af176780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.690+0000 7fee4b7fe640 1 -- 172.21.15.70:0/638668529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165c60 msgr2=0x5580af16f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.691+0000 7fee4b7fe640 1 --2- 172.21.15.70:0/638668529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165c60 0x5580af16f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.691+0000 7fee4b7fe640 1 -- 172.21.15.70:0/638668529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580aedd53f0 con 0x5580af165270 2024-04-01T01:34:48.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.691+0000 7fee4bfff640 1 --2- 172.21.15.70:0/638668529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af16a860 0x5580af176780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:48.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.691+0000 7fee2affd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:34:48.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.691+0000 7fee4b7fe640 1 --2- 172.21.15.70:0/638668529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165270 0x5580af16f200 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fee34000c00 tx=0x7fee34009ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=424f32d431e877e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.691+0000 7fee2affd640 1 -- 172.21.15.70:0/638668529 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee340b3020 con 0x5580af165270 2024-04-01T01:34:48.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.691+0000 7fee54785740 1 -- 172.21.15.70:0/638668529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee3c04d070 con 0x5580af165270 2024-04-01T01:34:48.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.691+0000 7fee2affd640 1 -- 172.21.15.70:0/638668529 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fee3409a1e0 con 0x5580af165270 2024-04-01T01:34:48.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.691+0000 7fee2affd640 1 -- 172.21.15.70:0/638668529 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fee34020040 con 0x5580af165270 2024-04-01T01:34:48.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.691+0000 7fee54785740 10 client.25760 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:48.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.691+0000 7fee54785740 1 -- 172.21.15.70:0/638668529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5580aee9f7d0 con 0x5580af165270 2024-04-01T01:34:48.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.692+0000 7fee2affd640 1 -- 172.21.15.70:0/638668529 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee3c04d070 con 0x5580af165270 2024-04-01T01:34:48.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.692+0000 7fee2affd640 1 -- 172.21.15.70:0/638668529 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fee340b92b0 con 0x5580af165270 2024-04-01T01:34:48.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.692+0000 7fee54785740 10 client.25760 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:48.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.692+0000 7fee54785740 10 client.25760 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:34:48.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.692+0000 7fee54785740 20 client.25760 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:48.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.692+0000 7fee54785740 1 --2- 172.21.15.70:0/638668529 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5580af17acb0 0x5580af17d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.692+0000 7fee54785740 4 client.25760 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:48.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.692+0000 7fee54785740 1 -- 172.21.15.70:0/638668529 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580af0ce6f0 con 0x5580af17acb0 2024-04-01T01:34:48.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.692+0000 7fee42ffd640 1 --2- 172.21.15.70:0/638668529 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5580af17acb0 0x5580af17d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.693+0000 7fee42ffd640 1 --2- 172.21.15.70:0/638668529 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5580af17acb0 0x5580af17d170 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ab825192c36c48f6 server_cookie=cd40f45911d38fc1 in_seq=0 out_seq=0 2024-04-01T01:34:48.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.693+0000 7fee2affd640 10 client.25760 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:34:48.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.694+0000 7fee2affd640 1 -- 172.21.15.70:0/638668529 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5580af0ce6f0 con 0x5580af17acb0 2024-04-01T01:34:48.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.694+0000 7fee2affd640 10 client.25760 handle_command_reply: tid=0 2024-04-01T01:34:48.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.694+0000 7fee54785740 1 client.25760 shutdown 2024-04-01T01:34:48.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.694+0000 7fee54785740 1 -- 172.21.15.70:0/638668529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165270 msgr2=0x5580af16f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.695+0000 7fee54785740 1 --2- 172.21.15.70:0/638668529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165270 0x5580af16f200 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fee34000c00 tx=0x7fee34009ea0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.695+0000 7fee54785740 1 -- 172.21.15.70:0/638668529 shutdown_connections 2024-04-01T01:34:48.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.695+0000 7fee54785740 1 --2- 172.21.15.70:0/638668529 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5580af17acb0 0x5580af17d170 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.695+0000 7fee54785740 1 --2- 172.21.15.70:0/638668529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af16a860 0x5580af176780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.695+0000 7fee54785740 1 --2- 172.21.15.70:0/638668529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165c60 0x5580af16f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.695+0000 7fee54785740 1 --2- 172.21.15.70:0/638668529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165270 0x5580af16f200 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.695+0000 7fee54785740 1 -- 172.21.15.70:0/638668529 >> 172.21.15.70:0/638668529 conn(0x5580af157980 msgr2=0x5580af156d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.695+0000 7fee2affd640 0 client.25760 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:34:48.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.695+0000 7fee54785740 1 -- 172.21.15.70:0/638668529 shutdown_connections 2024-04-01T01:34:48.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.695+0000 7fee54785740 1 -- 172.21.15.70:0/638668529 wait complete. 2024-04-01T01:34:48.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.696+0000 7fee54785740 20 client.25760 trim_cache size 0 max 16384 2024-04-01T01:34:48.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.713+0000 7fee497fa640 1 -- 172.21.15.70:0/3794406611 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fee34097400 con 0x7fee4c0ff4e0 2024-04-01T01:34:48.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.720+0000 7fee54785740 1 Processor -- start 2024-04-01T01:34:48.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.720+0000 7fee54785740 1 -- start start 2024-04-01T01:34:48.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.720+0000 7fee54785740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165270 0x5580af16ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.720+0000 7fee54785740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165c60 0x5580af1766a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.720+0000 7fee4b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165270 0x5580af16ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.720+0000 7fee4b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165270 0x5580af16ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41598/0 (socket says 172.21.15.70:41598) 2024-04-01T01:34:48.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.720+0000 7fee4b7fe640 1 -- 172.21.15.70:0/1831473517 learned_addr learned my addr 172.21.15.70:0/1831473517 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:48.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.720+0000 7fee54785740 1 --2- 172.21.15.70:0/1831473517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af16a860 0x5580af16f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.720+0000 7fee54785740 1 -- 172.21.15.70:0/1831473517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c1a8160 con 0x5580af165270 2024-04-01T01:34:48.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.720+0000 7fee54785740 1 -- 172.21.15.70:0/1831473517 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c1a9ca0 con 0x5580af165c60 2024-04-01T01:34:48.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.720+0000 7fee54785740 1 -- 172.21.15.70:0/1831473517 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c1a8830 con 0x5580af16a860 2024-04-01T01:34:48.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.720+0000 7fee42ffd640 1 --2- 172.21.15.70:0/1831473517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165c60 0x5580af1766a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.721+0000 7fee4bfff640 1 --2- 172.21.15.70:0/1831473517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af16a860 0x5580af16f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.721+0000 7fee42ffd640 1 -- 172.21.15.70:0/1831473517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af16a860 msgr2=0x5580af16f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.721+0000 7fee42ffd640 1 --2- 172.21.15.70:0/1831473517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af16a860 0x5580af16f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.721+0000 7fee42ffd640 1 -- 172.21.15.70:0/1831473517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165270 msgr2=0x5580af16ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.721+0000 7fee42ffd640 1 --2- 172.21.15.70:0/1831473517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165270 0x5580af16ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.721+0000 7fee42ffd640 1 -- 172.21.15.70:0/1831473517 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580af179660 con 0x5580af165c60 2024-04-01T01:34:48.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.721+0000 7fee4b7fe640 1 --2- 172.21.15.70:0/1831473517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165270 0x5580af16ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:34:48.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.721+0000 7fee42ffd640 1 --2- 172.21.15.70:0/1831473517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165c60 0x5580af1766a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fee3c04a040 tx=0x7fee3c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf3a31288dfd0232 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.722+0000 7fee417fa640 1 -- 172.21.15.70:0/1831473517 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c05f070 con 0x5580af165c60 2024-04-01T01:34:48.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.722+0000 7fee417fa640 1 -- 172.21.15.70:0/1831473517 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee3c04d050 con 0x5580af165c60 2024-04-01T01:34:48.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.722+0000 7fee417fa640 1 -- 172.21.15.70:0/1831473517 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c0048e0 con 0x5580af165c60 2024-04-01T01:34:48.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.723+0000 7fee54785740 1 -- 172.21.15.70:0/1831473517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165c60 msgr2=0x5580af1766a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.723+0000 7fee54785740 1 --2- 172.21.15.70:0/1831473517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165c60 0x5580af1766a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fee3c04a040 tx=0x7fee3c005fd0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.723+0000 7fee54785740 1 -- 172.21.15.70:0/1831473517 shutdown_connections 2024-04-01T01:34:48.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.723+0000 7fee54785740 1 --2- 172.21.15.70:0/1831473517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af16a860 0x5580af16f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.723+0000 7fee54785740 1 --2- 172.21.15.70:0/1831473517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165c60 0x5580af1766a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.723+0000 7fee54785740 1 --2- 172.21.15.70:0/1831473517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af165270 0x5580af16ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.723+0000 7fee54785740 1 -- 172.21.15.70:0/1831473517 >> 172.21.15.70:0/1831473517 conn(0x5580af157980 msgr2=0x5580af16dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.723+0000 7fee54785740 1 -- 172.21.15.70:0/1831473517 shutdown_connections 2024-04-01T01:34:48.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.723+0000 7fee54785740 1 -- 172.21.15.70:0/1831473517 wait complete. 2024-04-01T01:34:48.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.724+0000 7fee54785740 1 -- start start 2024-04-01T01:34:48.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.724+0000 7fee54785740 1 Processor -- start 2024-04-01T01:34:48.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.725+0000 7fee54785740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165270 0x5580af173cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.725+0000 7fee54785740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af165c60 0x5580af174230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.725+0000 7fee4b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165270 0x5580af173cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.725+0000 7fee4b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165270 0x5580af173cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46414/0 (socket says 172.21.15.70:46414) 2024-04-01T01:34:48.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.725+0000 7fee4b7fe640 1 -- 172.21.15.70:0/2895672202 learned_addr learned my addr 172.21.15.70:0/2895672202 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:48.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.725+0000 7fee54785740 1 --2- 172.21.15.70:0/2895672202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af16a860 0x5580af1751f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.725+0000 7fee42ffd640 1 --2- 172.21.15.70:0/2895672202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af165c60 0x5580af174230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.725+0000 7fee54785740 1 -- 172.21.15.70:0/2895672202 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c1a95d0 con 0x5580af16a860 2024-04-01T01:34:48.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.725+0000 7fee54785740 1 -- 172.21.15.70:0/2895672202 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c1a8f00 con 0x5580af165270 2024-04-01T01:34:48.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.725+0000 7fee54785740 1 -- 172.21.15.70:0/2895672202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c1a6140 con 0x5580af165c60 2024-04-01T01:34:48.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.725+0000 7fee54785740 10 client.0 _ll_register_callbacks cb 0x7fee3409a6f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:48.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.725+0000 7fee4b7fe640 1 -- 172.21.15.70:0/2895672202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af165c60 msgr2=0x5580af174230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.725+0000 7fee4b7fe640 1 --2- 172.21.15.70:0/2895672202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af165c60 0x5580af174230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.725+0000 7fee4b7fe640 1 -- 172.21.15.70:0/2895672202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af16a860 msgr2=0x5580af1751f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.725+0000 7fee4b7fe640 1 --2- 172.21.15.70:0/2895672202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af16a860 0x5580af1751f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.726+0000 7fee4b7fe640 1 -- 172.21.15.70:0/2895672202 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580af1757e0 con 0x5580af165270 2024-04-01T01:34:48.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.726+0000 7fee54785740 1 -- 172.21.15.70:0/2895672202 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee3c04d050 con 0x5580af165270 2024-04-01T01:34:48.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.726+0000 7fee4b7fe640 1 --2- 172.21.15.70:0/2895672202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165270 0x5580af173cf0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fee340ca7b0 tx=0x7fee340c5630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dcfd355773a44a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.726+0000 7fee2affd640 10 client.25766 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:34:48.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.726+0000 7fee2affd640 1 -- 172.21.15.70:0/2895672202 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee340c7520 con 0x5580af165270 2024-04-01T01:34:48.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.726+0000 7fee2affd640 1 -- 172.21.15.70:0/2895672202 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fee3402b7a0 con 0x5580af165270 2024-04-01T01:34:48.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.726+0000 7fee2affd640 1 -- 172.21.15.70:0/2895672202 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fee340c76c0 con 0x5580af165270 2024-04-01T01:34:48.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.727+0000 7fee54785740 10 client.25766 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:48.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.727+0000 7fee54785740 1 -- 172.21.15.70:0/2895672202 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=29}) -- 0x5580af1769b0 con 0x5580af165270 2024-04-01T01:34:48.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.727+0000 7fee2affd640 1 -- 172.21.15.70:0/2895672202 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fee340b92b0 con 0x5580af165270 2024-04-01T01:34:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.727+0000 7fee54785740 10 client.25766 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.727+0000 7fee54785740 10 client.25766 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:34:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.727+0000 7fee54785740 20 client.25766 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.727+0000 7fee54785740 1 --2- 172.21.15.70:0/2895672202 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5580af1e15f0 0x5580af1e3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.728+0000 7fee54785740 4 client.25766 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:34:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.728+0000 7fee54785740 1 -- 172.21.15.70:0/2895672202 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5580af0c8a40 con 0x5580af1e15f0 2024-04-01T01:34:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.728+0000 7fee42ffd640 1 --2- 172.21.15.70:0/2895672202 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5580af1e15f0 0x5580af1e3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.728+0000 7fee2affd640 10 client.25766 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:34:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.728+0000 7fee42ffd640 1 --2- 172.21.15.70:0/2895672202 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5580af1e15f0 0x5580af1e3a30 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e489a5b37f987cf2 server_cookie=ea878af7274e531a in_seq=0 out_seq=0 2024-04-01T01:34:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.728+0000 7fee2affd640 1 -- 172.21.15.70:0/2895672202 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5580af0c8a40 con 0x5580af1e15f0 2024-04-01T01:34:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.728+0000 7fee2affd640 10 client.25766 handle_command_reply: tid=0 2024-04-01T01:34:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.728+0000 7fee54785740 1 client.25766 shutdown 2024-04-01T01:34:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.729+0000 7fee54785740 1 -- 172.21.15.70:0/2895672202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165270 msgr2=0x5580af173cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.729+0000 7fee54785740 1 --2- 172.21.15.70:0/2895672202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165270 0x5580af173cf0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fee340ca7b0 tx=0x7fee340c5630 comp rx=0 tx=0).stop 2024-04-01T01:34:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.729+0000 7fee54785740 1 -- 172.21.15.70:0/2895672202 shutdown_connections 2024-04-01T01:34:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.729+0000 7fee54785740 1 --2- 172.21.15.70:0/2895672202 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5580af1e15f0 0x5580af1e3a30 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.729+0000 7fee54785740 1 --2- 172.21.15.70:0/2895672202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580af16a860 0x5580af1751f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.729+0000 7fee54785740 1 --2- 172.21.15.70:0/2895672202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580af165c60 0x5580af174230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.729+0000 7fee54785740 1 --2- 172.21.15.70:0/2895672202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580af165270 0x5580af173cf0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.729+0000 7fee54785740 1 -- 172.21.15.70:0/2895672202 >> 172.21.15.70:0/2895672202 conn(0x5580af157980 msgr2=0x5580af16ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.730+0000 7fee2affd640 0 client.25766 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:34:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.730+0000 7fee54785740 1 -- 172.21.15.70:0/2895672202 shutdown_connections 2024-04-01T01:34:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.730+0000 7fee54785740 1 -- 172.21.15.70:0/2895672202 wait complete. 2024-04-01T01:34:48.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.730+0000 7fee54785740 20 client.25766 trim_cache size 0 max 16384 2024-04-01T01:34:48.734 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:34:48.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.731+0000 7fee51ac9640 1 -- 172.21.15.70:0/3794406611 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee380a2680 msgr2=0x7fee380a4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.731+0000 7fee51ac9640 1 --2- 172.21.15.70:0/3794406611 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee380a2680 0x7fee380a4b40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fee3c036040 tx=0x7fee3c000bc0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.731+0000 7fee51ac9640 1 -- 172.21.15.70:0/3794406611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee4c0ff4e0 msgr2=0x7fee4c197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.731+0000 7fee51ac9640 1 --2- 172.21.15.70:0/3794406611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee4c0ff4e0 0x7fee4c197a20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fee3400d950 tx=0x7fee3400de10 comp rx=0 tx=0).stop 2024-04-01T01:34:48.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.731+0000 7fee51ac9640 1 -- 172.21.15.70:0/3794406611 shutdown_connections 2024-04-01T01:34:48.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.731+0000 7fee51ac9640 1 --2- 172.21.15.70:0/3794406611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee4c1073d0 0x7fee4c19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.731+0000 7fee51ac9640 1 --2- 172.21.15.70:0/3794406611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee4c0ffe40 0x7fee4c197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.731+0000 7fee51ac9640 1 --2- 172.21.15.70:0/3794406611 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee380a2680 0x7fee380a4b40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.731+0000 7fee51ac9640 1 --2- 172.21.15.70:0/3794406611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee4c0ff4e0 0x7fee4c197a20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.731+0000 7fee51ac9640 1 -- 172.21.15.70:0/3794406611 >> 172.21.15.70:0/3794406611 conn(0x7fee4c0faf10 msgr2=0x7fee4c102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.731+0000 7fee51ac9640 1 -- 172.21.15.70:0/3794406611 shutdown_connections 2024-04-01T01:34:48.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.731+0000 7fee51ac9640 1 -- 172.21.15.70:0/3794406611 wait complete. 2024-04-01T01:34:48.748 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:48 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/328550760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:34:48.749 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:34:48.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.914+0000 7fb0b359e640 1 -- 172.21.15.70:0/2967993186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b4103850 msgr2=0x7fb0b4103c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.914+0000 7fb0b359e640 1 --2- 172.21.15.70:0/2967993186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b4103850 0x7fb0b4103c70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4009900 tx=0x7fb0a402bc30 comp rx=0 tx=0).stop 2024-04-01T01:34:48.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.914+0000 7fb0b359e640 1 -- 172.21.15.70:0/2967993186 shutdown_connections 2024-04-01T01:34:48.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.914+0000 7fb0b359e640 1 --2- 172.21.15.70:0/2967993186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0b4105400 0x7fb0b410bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.914+0000 7fb0b359e640 1 --2- 172.21.15.70:0/2967993186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0b4104a40 0x7fb0b4104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.915+0000 7fb0b359e640 1 --2- 172.21.15.70:0/2967993186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b4103850 0x7fb0b4103c70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.915+0000 7fb0b359e640 1 -- 172.21.15.70:0/2967993186 >> 172.21.15.70:0/2967993186 conn(0x7fb0b40fedc0 msgr2=0x7fb0b4101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.915+0000 7fb0b359e640 1 -- 172.21.15.70:0/2967993186 shutdown_connections 2024-04-01T01:34:48.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.915+0000 7fb0b359e640 1 -- 172.21.15.70:0/2967993186 wait complete. 2024-04-01T01:34:48.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.915+0000 7fb0b359e640 1 Processor -- start 2024-04-01T01:34:48.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.916+0000 7fb0b359e640 1 -- start start 2024-04-01T01:34:48.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.916+0000 7fb0b359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0b4103850 0x7fb0b4199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.916+0000 7fb0b359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0b4104a40 0x7fb0b4199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.916+0000 7fb0b259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0b4103850 0x7fb0b4199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.916+0000 7fb0b1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0b4104a40 0x7fb0b4199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.916+0000 7fb0b259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0b4103850 0x7fb0b4199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46440/0 (socket says 172.21.15.70:46440) 2024-04-01T01:34:48.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.916+0000 7fb0b1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0b4104a40 0x7fb0b4199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41614/0 (socket says 172.21.15.70:41614) 2024-04-01T01:34:48.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.916+0000 7fb0b259c640 1 -- 172.21.15.70:0/1305475665 learned_addr learned my addr 172.21.15.70:0/1305475665 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:48.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.916+0000 7fb0b359e640 1 --2- 172.21.15.70:0/1305475665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b4105400 0x7fb0b41a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.916+0000 7fb0b359e640 1 -- 172.21.15.70:0/1305475665 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b410f3a0 con 0x7fb0b4104a40 2024-04-01T01:34:48.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.916+0000 7fb0b359e640 1 -- 172.21.15.70:0/1305475665 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b410f220 con 0x7fb0b4103850 2024-04-01T01:34:48.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.916+0000 7fb0b359e640 1 -- 172.21.15.70:0/1305475665 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b410f520 con 0x7fb0b4105400 2024-04-01T01:34:48.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.916+0000 7fb0b2d9d640 1 --2- 172.21.15.70:0/1305475665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b4105400 0x7fb0b41a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.917+0000 7fb0b259c640 1 -- 172.21.15.70:0/1305475665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b4105400 msgr2=0x7fb0b41a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.917+0000 7fb0b259c640 1 --2- 172.21.15.70:0/1305475665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b4105400 0x7fb0b41a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.917+0000 7fb0b259c640 1 -- 172.21.15.70:0/1305475665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0b4104a40 msgr2=0x7fb0b4199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.917+0000 7fb0b259c640 1 --2- 172.21.15.70:0/1305475665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0b4104a40 0x7fb0b4199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.917+0000 7fb0b259c640 1 -- 172.21.15.70:0/1305475665 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b41a16c0 con 0x7fb0b4103850 2024-04-01T01:34:48.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.917+0000 7fb0b259c640 1 --2- 172.21.15.70:0/1305475665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0b4103850 0x7fb0b4199aa0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4000c00 tx=0x7fb0a4005e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.917+0000 7fb0a37fe640 1 -- 172.21.15.70:0/1305475665 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4043020 con 0x7fb0b4103850 2024-04-01T01:34:48.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.917+0000 7fb0a37fe640 1 -- 172.21.15.70:0/1305475665 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0a40043d0 con 0x7fb0b4103850 2024-04-01T01:34:48.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.917+0000 7fb0a37fe640 1 -- 172.21.15.70:0/1305475665 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4003bb0 con 0x7fb0b4103850 2024-04-01T01:34:48.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.917+0000 7fb0b359e640 1 -- 172.21.15.70:0/1305475665 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b41a1950 con 0x7fb0b4103850 2024-04-01T01:34:48.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.917+0000 7fb0b359e640 1 -- 172.21.15.70:0/1305475665 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b41a1e50 con 0x7fb0b4103850 2024-04-01T01:34:48.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.921+0000 7fb0a37fe640 1 -- 172.21.15.70:0/1305475665 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb0a4046020 con 0x7fb0b4103850 2024-04-01T01:34:48.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.922+0000 7fb0a37fe640 1 --2- 172.21.15.70:0/1305475665 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb09c0a2520 0x7fb09c0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.922+0000 7fb0a37fe640 1 -- 172.21.15.70:0/1305475665 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb0a40f1b70 con 0x7fb0b4103850 2024-04-01T01:34:48.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.922+0000 7fb0b1d9b640 1 --2- 172.21.15.70:0/1305475665 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb09c0a2520 0x7fb09c0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.923+0000 7fb0bb5fc740 1 Processor -- start 2024-04-01T01:34:48.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.923+0000 7fb0b1d9b640 1 --2- 172.21.15.70:0/1305475665 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb09c0a2520 0x7fb09c0a49e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80095f0 tx=0x7fb0a8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.923+0000 7fb0bb5fc740 1 -- start start 2024-04-01T01:34:48.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.923+0000 7fb0bb5fc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df3eb0 0x5630f2df42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.923+0000 7fb0bb5fc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df48a0 0x5630f2df93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.924+0000 7fb0bb5fc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df9a10 0x5630f2dfbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.924+0000 7fb0bb5fc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5630f2d501b0 con 0x5630f2df3eb0 2024-04-01T01:34:48.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.924+0000 7fb0bb5fc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5630f2cf8c00 con 0x5630f2df9a10 2024-04-01T01:34:48.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.924+0000 7fb0bb5fc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5630f2d4c250 con 0x5630f2df48a0 2024-04-01T01:34:48.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.924+0000 7fb0b259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df3eb0 0x5630f2df42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.924+0000 7fb0b259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df3eb0 0x5630f2df42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41618/0 (socket says 172.21.15.70:41618) 2024-04-01T01:34:48.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.924+0000 7fb0b259c640 1 -- 172.21.15.70:0/2095792159 learned_addr learned my addr 172.21.15.70:0/2095792159 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:48.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.924+0000 7fb0b2d9d640 1 --2- 172.21.15.70:0/2095792159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df9a10 0x5630f2dfbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.924+0000 7fb0b1d9b640 1 --2- 172.21.15.70:0/2095792159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df48a0 0x5630f2df93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.924+0000 7fb0b259c640 1 -- 172.21.15.70:0/2095792159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df48a0 msgr2=0x5630f2df93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.924+0000 7fb0b259c640 1 --2- 172.21.15.70:0/2095792159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df48a0 0x5630f2df93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.924+0000 7fb0b259c640 1 -- 172.21.15.70:0/2095792159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df9a10 msgr2=0x5630f2dfbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.924+0000 7fb0b259c640 1 --2- 172.21.15.70:0/2095792159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df9a10 0x5630f2dfbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.924+0000 7fb0b259c640 1 -- 172.21.15.70:0/2095792159 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630f2d183d0 con 0x5630f2df3eb0 2024-04-01T01:34:48.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.925+0000 7fb0b259c640 1 --2- 172.21.15.70:0/2095792159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df3eb0 0x5630f2df42d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb0a40be190 tx=0x7fb0a402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce8d803c1d93eef server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.925+0000 7fb0a0ff9640 1 -- 172.21.15.70:0/2095792159 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a40be200 con 0x5630f2df3eb0 2024-04-01T01:34:48.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.925+0000 7fb0a0ff9640 1 -- 172.21.15.70:0/2095792159 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0a40c2350 con 0x5630f2df3eb0 2024-04-01T01:34:48.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.925+0000 7fb0a0ff9640 1 -- 172.21.15.70:0/2095792159 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4005500 con 0x5630f2df3eb0 2024-04-01T01:34:48.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.926+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/2095792159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df3eb0 msgr2=0x5630f2df42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.926+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/2095792159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df3eb0 0x5630f2df42d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb0a40be190 tx=0x7fb0a402bc30 comp rx=0 tx=0).stop 2024-04-01T01:34:48.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.926+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/2095792159 shutdown_connections 2024-04-01T01:34:48.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.926+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/2095792159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df9a10 0x5630f2dfbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.926+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/2095792159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df48a0 0x5630f2df93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.926+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/2095792159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df3eb0 0x5630f2df42d0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.926+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/2095792159 >> 172.21.15.70:0/2095792159 conn(0x5630f2def420 msgr2=0x5630f2def820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.926+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/2095792159 shutdown_connections 2024-04-01T01:34:48.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.926+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/2095792159 wait complete. 2024-04-01T01:34:48.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.927+0000 7fb0bb5fc740 1 -- start start 2024-04-01T01:34:48.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.927+0000 7fb0bb5fc740 1 Processor -- start 2024-04-01T01:34:48.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.927+0000 7fb0bb5fc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df3eb0 0x5630f2dfdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.927+0000 7fb0bb5fc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df48a0 0x5630f2dfe4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.928+0000 7fb0bb5fc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df9a10 0x5630f2e05520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.928+0000 7fb0b259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df3eb0 0x5630f2dfdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.928+0000 7fb0b259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df3eb0 0x5630f2dfdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39104/0 (socket says 172.21.15.70:39104) 2024-04-01T01:34:48.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.928+0000 7fb0b1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df48a0 0x5630f2dfe4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.928+0000 7fb0b1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df48a0 0x5630f2dfe4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41632/0 (socket says 172.21.15.70:41632) 2024-04-01T01:34:48.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.928+0000 7fb0b259c640 1 -- 172.21.15.70:0/2621947563 learned_addr learned my addr 172.21.15.70:0/2621947563 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:48.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.928+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/2621947563 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5630f2dfc780 con 0x5630f2df48a0 2024-04-01T01:34:48.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.928+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/2621947563 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5630f2cee470 con 0x5630f2df9a10 2024-04-01T01:34:48.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.928+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/2621947563 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5630f2dfc900 con 0x5630f2df3eb0 2024-04-01T01:34:48.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.928+0000 7fb0bb5fc740 10 client.0 _ll_register_callbacks cb 0x5630f2d0bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.928+0000 7fb0b2d9d640 1 --2- 172.21.15.70:0/2621947563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df9a10 0x5630f2e05520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.928+0000 7fb0b259c640 1 -- 172.21.15.70:0/2621947563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df9a10 msgr2=0x5630f2e05520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.928+0000 7fb0b259c640 1 --2- 172.21.15.70:0/2621947563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df9a10 0x5630f2e05520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.928+0000 7fb0b259c640 1 -- 172.21.15.70:0/2621947563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df48a0 msgr2=0x5630f2dfe4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.928+0000 7fb0b259c640 1 --2- 172.21.15.70:0/2621947563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df48a0 0x5630f2dfe4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.928+0000 7fb0b259c640 1 -- 172.21.15.70:0/2621947563 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630f2a643f0 con 0x5630f2df3eb0 2024-04-01T01:34:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.929+0000 7fb0b259c640 1 --2- 172.21.15.70:0/2621947563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df3eb0 0x5630f2dfdfa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4003660 tx=0x7fb0a40c21f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7513828ad3a5225 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.929+0000 7fb099ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:34:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.929+0000 7fb099ffb640 1 -- 172.21.15.70:0/2621947563 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a40a0020 con 0x5630f2df3eb0 2024-04-01T01:34:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.929+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/2621947563 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0a40c2350 con 0x5630f2df3eb0 2024-04-01T01:34:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.929+0000 7fb099ffb640 1 -- 172.21.15.70:0/2621947563 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb0a40be660 con 0x5630f2df3eb0 2024-04-01T01:34:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.930+0000 7fb099ffb640 1 -- 172.21.15.70:0/2621947563 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb0a40c6e60 con 0x5630f2df3eb0 2024-04-01T01:34:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.930+0000 7fb0bb5fc740 10 client.25693 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:48.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.930+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/2621947563 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5630f2b2e7d0 con 0x5630f2df3eb0 2024-04-01T01:34:48.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.930+0000 7fb099ffb640 1 -- 172.21.15.70:0/2621947563 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0a40c2350 con 0x5630f2df3eb0 2024-04-01T01:34:48.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.930+0000 7fb099ffb640 1 -- 172.21.15.70:0/2621947563 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb0a40ba0e0 con 0x5630f2df3eb0 2024-04-01T01:34:48.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.930+0000 7fb0bb5fc740 10 client.25693 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:48.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.930+0000 7fb0bb5fc740 10 client.25693 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:34:48.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.931+0000 7fb0bb5fc740 20 client.25693 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:48.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.931+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/2621947563 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5630f2e09820 0x5630f2e0bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.931+0000 7fb0bb5fc740 4 client.25693 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:48.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.931+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/2621947563 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630f2d5d6f0 con 0x5630f2e09820 2024-04-01T01:34:48.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.931+0000 7fb0b1d9b640 1 --2- 172.21.15.70:0/2621947563 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5630f2e09820 0x5630f2e0bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.931+0000 7fb0b1d9b640 1 --2- 172.21.15.70:0/2621947563 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5630f2e09820 0x5630f2e0bc50 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=55596c4d6e4696c server_cookie=8e3067abdf0d591d in_seq=0 out_seq=0 2024-04-01T01:34:48.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.932+0000 7fb099ffb640 10 client.25693 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:34:48.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.932+0000 7fb099ffb640 1 -- 172.21.15.70:0/2621947563 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5630f2d5d6f0 con 0x5630f2e09820 2024-04-01T01:34:48.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.933+0000 7fb099ffb640 10 client.25693 handle_command_reply: tid=0 2024-04-01T01:34:48.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.933+0000 7fb0bb5fc740 1 client.25693 shutdown 2024-04-01T01:34:48.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.933+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/2621947563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df3eb0 msgr2=0x5630f2dfdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.933+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/2621947563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df3eb0 0x5630f2dfdfa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4003660 tx=0x7fb0a40c21f0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.933+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/2621947563 shutdown_connections 2024-04-01T01:34:48.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.933+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/2621947563 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5630f2e09820 0x5630f2e0bc50 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.933+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/2621947563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df9a10 0x5630f2e05520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.933+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/2621947563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df48a0 0x5630f2dfe4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.933+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/2621947563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df3eb0 0x5630f2dfdfa0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.933+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/2621947563 >> 172.21.15.70:0/2621947563 conn(0x5630f2def420 msgr2=0x5630f2df0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.933+0000 7fb099ffb640 0 client.25693 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:34:48.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.933+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/2621947563 shutdown_connections 2024-04-01T01:34:48.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.933+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/2621947563 wait complete. 2024-04-01T01:34:48.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.933+0000 7fb0bb5fc740 20 client.25693 trim_cache size 0 max 16384 2024-04-01T01:34:48.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.957+0000 7fb0bb5fc740 1 Processor -- start 2024-04-01T01:34:48.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.957+0000 7fb0bb5fc740 1 -- start start 2024-04-01T01:34:48.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.958+0000 7fb0bb5fc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df3eb0 0x5630f2df5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.958+0000 7fb0bb5fc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df48a0 0x5630f2df61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.958+0000 7fb0bb5fc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df9a10 0x5630f2e054b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.958+0000 7fb0bb5fc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b41a8020 con 0x5630f2df48a0 2024-04-01T01:34:48.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.958+0000 7fb0bb5fc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b41a9b60 con 0x5630f2df3eb0 2024-04-01T01:34:48.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.958+0000 7fb0bb5fc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b41a86f0 con 0x5630f2df9a10 2024-04-01T01:34:48.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.958+0000 7fb0b1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df48a0 0x5630f2df61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.958+0000 7fb0b259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df3eb0 0x5630f2df5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.958+0000 7fb0b2d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df9a10 0x5630f2e054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.958+0000 7fb0b259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df3eb0 0x5630f2df5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46464/0 (socket says 172.21.15.70:46464) 2024-04-01T01:34:48.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.958+0000 7fb0b259c640 1 -- 172.21.15.70:0/1209127708 learned_addr learned my addr 172.21.15.70:0/1209127708 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.958+0000 7fb0b2d9d640 1 -- 172.21.15.70:0/1209127708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df3eb0 msgr2=0x5630f2df5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.958+0000 7fb0b2d9d640 1 --2- 172.21.15.70:0/1209127708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df3eb0 0x5630f2df5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.958+0000 7fb0b2d9d640 1 -- 172.21.15.70:0/1209127708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df48a0 msgr2=0x5630f2df61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.959+0000 7fb0b2d9d640 1 --2- 172.21.15.70:0/1209127708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df48a0 0x5630f2df61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.959+0000 7fb0b2d9d640 1 -- 172.21.15.70:0/1209127708 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630f2dd7e60 con 0x5630f2df9a10 2024-04-01T01:34:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.959+0000 7fb0b259c640 1 --2- 172.21.15.70:0/1209127708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df3eb0 0x5630f2df5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.959+0000 7fb0b1d9b640 1 --2- 172.21.15.70:0/1209127708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df48a0 0x5630f2df61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:34:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.959+0000 7fb0b2d9d640 1 --2- 172.21.15.70:0/1209127708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df9a10 0x5630f2e054b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac005fd0 tx=0x7fb0ac0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56adcad5fd09835e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.959+0000 7fb09bfff640 1 -- 172.21.15.70:0/1209127708 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac004b20 con 0x5630f2df9a10 2024-04-01T01:34:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.959+0000 7fb09bfff640 1 -- 172.21.15.70:0/1209127708 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0ac004cc0 con 0x5630f2df9a10 2024-04-01T01:34:48.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.959+0000 7fb09bfff640 1 -- 172.21.15.70:0/1209127708 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac005330 con 0x5630f2df9a10 2024-04-01T01:34:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.960+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/1209127708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df9a10 msgr2=0x5630f2e054b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.960+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/1209127708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df9a10 0x5630f2e054b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac005fd0 tx=0x7fb0ac0118d0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.960+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/1209127708 shutdown_connections 2024-04-01T01:34:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.960+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/1209127708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df9a10 0x5630f2e054b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.960+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/1209127708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df48a0 0x5630f2df61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.960+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/1209127708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df3eb0 0x5630f2df5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.960+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/1209127708 >> 172.21.15.70:0/1209127708 conn(0x5630f2def420 msgr2=0x5630f2def800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.960+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/1209127708 shutdown_connections 2024-04-01T01:34:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.960+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/1209127708 wait complete. 2024-04-01T01:34:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.961+0000 7fb0bb5fc740 1 -- start start 2024-04-01T01:34:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.961+0000 7fb0bb5fc740 1 Processor -- start 2024-04-01T01:34:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.961+0000 7fb0bb5fc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df3eb0 0x5630f2e02d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.961+0000 7fb0bb5fc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df48a0 0x5630f2e032b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.961+0000 7fb0bb5fc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df9a10 0x5630f2e042c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.961+0000 7fb0bb5fc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b41a9490 con 0x5630f2df9a10 2024-04-01T01:34:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.961+0000 7fb0bb5fc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b41a8dc0 con 0x5630f2df48a0 2024-04-01T01:34:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.961+0000 7fb0bb5fc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b41a6000 con 0x5630f2df3eb0 2024-04-01T01:34:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.962+0000 7fb0bb5fc740 10 client.0 _ll_register_callbacks cb 0x7fb0a4005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.962+0000 7fb0b2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df9a10 0x5630f2e042c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.962+0000 7fb0b259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df3eb0 0x5630f2e02d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.962+0000 7fb0b2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df9a10 0x5630f2e042c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41650/0 (socket says 172.21.15.70:41650) 2024-04-01T01:34:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.962+0000 7fb0b2d9d640 1 -- 172.21.15.70:0/3865057276 learned_addr learned my addr 172.21.15.70:0/3865057276 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.962+0000 7fb0b1d9b640 1 --2- 172.21.15.70:0/3865057276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df48a0 0x5630f2e032b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.962+0000 7fb0b1d9b640 1 -- 172.21.15.70:0/3865057276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df3eb0 msgr2=0x5630f2e02d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.962+0000 7fb0b1d9b640 1 --2- 172.21.15.70:0/3865057276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df3eb0 0x5630f2e02d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.962+0000 7fb0b1d9b640 1 -- 172.21.15.70:0/3865057276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df9a10 msgr2=0x5630f2e042c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.962+0000 7fb0b1d9b640 1 --2- 172.21.15.70:0/3865057276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df9a10 0x5630f2e042c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.962+0000 7fb0b1d9b640 1 -- 172.21.15.70:0/3865057276 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630f2e048b0 con 0x5630f2df48a0 2024-04-01T01:34:48.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.962+0000 7fb0b2d9d640 1 --2- 172.21.15.70:0/3865057276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df9a10 0x5630f2e042c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.962+0000 7fb0b259c640 1 --2- 172.21.15.70:0/3865057276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df3eb0 0x5630f2e02d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.962+0000 7fb0b1d9b640 1 --2- 172.21.15.70:0/3865057276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df48a0 0x5630f2e032b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x5630f2e0c510 tx=0x7fb0a8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=630a8e0107b93d20 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.962+0000 7fb099ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:34:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.963+0000 7fb099ffb640 1 -- 172.21.15.70:0/3865057276 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8004360 con 0x5630f2df48a0 2024-04-01T01:34:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.963+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/3865057276 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0ac004cc0 con 0x5630f2df48a0 2024-04-01T01:34:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.963+0000 7fb099ffb640 1 -- 172.21.15.70:0/3865057276 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb0a802b070 con 0x5630f2df48a0 2024-04-01T01:34:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.963+0000 7fb099ffb640 1 -- 172.21.15.70:0/3865057276 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb0a802b270 con 0x5630f2df48a0 2024-04-01T01:34:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.963+0000 7fb0bb5fc740 10 client.25784 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.963+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/3865057276 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5630f2e6fd50 con 0x5630f2df48a0 2024-04-01T01:34:48.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.964+0000 7fb099ffb640 1 -- 172.21.15.70:0/3865057276 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0ac004cc0 con 0x5630f2df48a0 2024-04-01T01:34:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.964+0000 7fb099ffb640 1 -- 172.21.15.70:0/3865057276 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb0a8003bd0 con 0x5630f2df48a0 2024-04-01T01:34:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.964+0000 7fb0bb5fc740 10 client.25784 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.964+0000 7fb0bb5fc740 10 client.25784 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:34:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.964+0000 7fb0bb5fc740 20 client.25784 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.964+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/3865057276 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5630f2e706f0 0x5630f2e72b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.964+0000 7fb0bb5fc740 4 client.25784 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:34:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.964+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/3865057276 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5630f2d57a40 con 0x5630f2e706f0 2024-04-01T01:34:48.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.964+0000 7fb0b259c640 1 --2- 172.21.15.70:0/3865057276 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5630f2e706f0 0x5630f2e72b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.965+0000 7fb0b259c640 1 --2- 172.21.15.70:0/3865057276 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5630f2e706f0 0x5630f2e72b10 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f727cde94fe3d1a7 server_cookie=2696a4bd3dd90083 in_seq=0 out_seq=0 2024-04-01T01:34:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.965+0000 7fb099ffb640 10 client.25784 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:34:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.965+0000 7fb099ffb640 1 -- 172.21.15.70:0/3865057276 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5630f2d57a40 con 0x5630f2e706f0 2024-04-01T01:34:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.965+0000 7fb099ffb640 10 client.25784 handle_command_reply: tid=0 2024-04-01T01:34:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.965+0000 7fb0bb5fc740 1 client.25784 shutdown 2024-04-01T01:34:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.965+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/3865057276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df48a0 msgr2=0x5630f2e032b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.965+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/3865057276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df48a0 0x5630f2e032b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x5630f2e0c510 tx=0x7fb0a8004730 comp rx=0 tx=0).stop 2024-04-01T01:34:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.966+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/3865057276 shutdown_connections 2024-04-01T01:34:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.966+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/3865057276 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5630f2e706f0 0x5630f2e72b10 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.966+0000 7fb099ffb640 0 client.25784 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:34:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.966+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/3865057276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630f2df9a10 0x5630f2e042c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.966+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/3865057276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630f2df48a0 0x5630f2e032b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.966+0000 7fb0bb5fc740 1 --2- 172.21.15.70:0/3865057276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630f2df3eb0 0x5630f2e02d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.966+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/3865057276 >> 172.21.15.70:0/3865057276 conn(0x5630f2def420 msgr2=0x5630f2df5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.966+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/3865057276 shutdown_connections 2024-04-01T01:34:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.966+0000 7fb0bb5fc740 1 -- 172.21.15.70:0/3865057276 wait complete. 2024-04-01T01:34:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.966+0000 7fb0bb5fc740 20 client.25784 trim_cache size 0 max 16384 2024-04-01T01:34:48.970 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:34:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.967+0000 7fb0b359e640 1 -- 172.21.15.70:0/1305475665 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb09c0a2520 msgr2=0x7fb09c0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.967+0000 7fb0b359e640 1 --2- 172.21.15.70:0/1305475665 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb09c0a2520 0x7fb09c0a49e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80095f0 tx=0x7fb0a8005f90 comp rx=0 tx=0).stop 2024-04-01T01:34:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.967+0000 7fb0b359e640 1 -- 172.21.15.70:0/1305475665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0b4103850 msgr2=0x7fb0b4199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.967+0000 7fb0b359e640 1 --2- 172.21.15.70:0/1305475665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0b4103850 0x7fb0b4199aa0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4000c00 tx=0x7fb0a4005e60 comp rx=0 tx=0).stop 2024-04-01T01:34:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.967+0000 7fb0b359e640 1 -- 172.21.15.70:0/1305475665 shutdown_connections 2024-04-01T01:34:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.967+0000 7fb0b359e640 1 --2- 172.21.15.70:0/1305475665 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb09c0a2520 0x7fb09c0a49e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.967+0000 7fb0b359e640 1 --2- 172.21.15.70:0/1305475665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b4105400 0x7fb0b41a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.967+0000 7fb0b359e640 1 --2- 172.21.15.70:0/1305475665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0b4104a40 0x7fb0b4199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.967+0000 7fb0b359e640 1 --2- 172.21.15.70:0/1305475665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0b4103850 0x7fb0b4199aa0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:48.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.967+0000 7fb0b359e640 1 -- 172.21.15.70:0/1305475665 >> 172.21.15.70:0/1305475665 conn(0x7fb0b40fedc0 msgr2=0x7fb0b4107e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:48.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.967+0000 7fb0b359e640 1 -- 172.21.15.70:0/1305475665 shutdown_connections 2024-04-01T01:34:48.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:48.967+0000 7fb0b359e640 1 -- 172.21.15.70:0/1305475665 wait complete. 2024-04-01T01:34:48.984 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:34:48.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:48 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/328550760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:34:49.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:48 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/328550760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:34:49.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.158+0000 7f778e68f640 1 -- 172.21.15.70:0/153806579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7788102200 msgr2=0x7f7788108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.158+0000 7f778e68f640 1 --2- 172.21.15.70:0/153806579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7788102200 0x7f7788108a90 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7780009ca0 tx=0x7f778002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.158+0000 7f778e68f640 1 -- 172.21.15.70:0/153806579 shutdown_connections 2024-04-01T01:34:49.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.158+0000 7f778e68f640 1 --2- 172.21.15.70:0/153806579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7788102200 0x7f7788108a90 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.158+0000 7f778e68f640 1 --2- 172.21.15.70:0/153806579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7788101840 0x7f7788101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.158+0000 7f778e68f640 1 --2- 172.21.15.70:0/153806579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7788100650 0x7f7788100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.158+0000 7f778e68f640 1 -- 172.21.15.70:0/153806579 >> 172.21.15.70:0/153806579 conn(0x7f77880fbbe0 msgr2=0x7f77880fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.158+0000 7f778e68f640 1 -- 172.21.15.70:0/153806579 shutdown_connections 2024-04-01T01:34:49.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.158+0000 7f778e68f640 1 -- 172.21.15.70:0/153806579 wait complete. 2024-04-01T01:34:49.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.160+0000 7f778e68f640 1 Processor -- start 2024-04-01T01:34:49.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.160+0000 7f778e68f640 1 -- start start 2024-04-01T01:34:49.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.160+0000 7f778e68f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7788100650 0x7f77881957a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.161+0000 7f778e68f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7788101840 0x7f7788195ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.161+0000 7f7787fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7788100650 0x7f77881957a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.161+0000 7f7787fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7788100650 0x7f77881957a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41664/0 (socket says 172.21.15.70:41664) 2024-04-01T01:34:49.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.161+0000 7f7787fff640 1 -- 172.21.15.70:0/2512126787 learned_addr learned my addr 172.21.15.70:0/2512126787 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:49.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.161+0000 7f77877fe640 1 --2- 172.21.15.70:0/2512126787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7788101840 0x7f7788195ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.161+0000 7f778e68f640 1 --2- 172.21.15.70:0/2512126787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7788102200 0x7f778819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.161+0000 7f778e68f640 1 -- 172.21.15.70:0/2512126787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f778810b040 con 0x7f7788100650 2024-04-01T01:34:49.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.161+0000 7f778e68f640 1 -- 172.21.15.70:0/2512126787 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f778810aec0 con 0x7f7788102200 2024-04-01T01:34:49.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.161+0000 7f778e68f640 1 -- 172.21.15.70:0/2512126787 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f778810b1c0 con 0x7f7788101840 2024-04-01T01:34:49.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.161+0000 7f778cc2c640 1 --2- 172.21.15.70:0/2512126787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7788102200 0x7f778819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.161+0000 7f7787fff640 1 -- 172.21.15.70:0/2512126787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7788101840 msgr2=0x7f7788195ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.161+0000 7f7787fff640 1 --2- 172.21.15.70:0/2512126787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7788101840 0x7f7788195ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.161+0000 7f7787fff640 1 -- 172.21.15.70:0/2512126787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7788102200 msgr2=0x7f778819cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.161+0000 7f7787fff640 1 --2- 172.21.15.70:0/2512126787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7788102200 0x7f778819cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.161+0000 7f7787fff640 1 -- 172.21.15.70:0/2512126787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f778819d3c0 con 0x7f7788100650 2024-04-01T01:34:49.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.162+0000 7f77877fe640 1 --2- 172.21.15.70:0/2512126787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7788101840 0x7f7788195ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:49.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.162+0000 7f7787fff640 1 --2- 172.21.15.70:0/2512126787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7788100650 0x7f77881957a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7778012f90 tx=0x7f7778010490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.162+0000 7f77857fa640 1 -- 172.21.15.70:0/2512126787 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7778010da0 con 0x7f7788100650 2024-04-01T01:34:49.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.162+0000 7f778e68f640 1 -- 172.21.15.70:0/2512126787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f778819d6b0 con 0x7f7788100650 2024-04-01T01:34:49.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.162+0000 7f77857fa640 1 -- 172.21.15.70:0/2512126787 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f777800de50 con 0x7f7788100650 2024-04-01T01:34:49.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.162+0000 7f77857fa640 1 -- 172.21.15.70:0/2512126787 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7778018450 con 0x7f7788100650 2024-04-01T01:34:49.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.162+0000 7f778e68f640 1 -- 172.21.15.70:0/2512126787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f778819dc50 con 0x7f7788100650 2024-04-01T01:34:49.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.166+0000 7f77857fa640 1 -- 172.21.15.70:0/2512126787 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f77780035f0 con 0x7f7788100650 2024-04-01T01:34:49.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.167+0000 7f77857fa640 1 --2- 172.21.15.70:0/2512126787 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77700a25f0 0x7f77700a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.167+0000 7f7791277740 1 Processor -- start 2024-04-01T01:34:49.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.167+0000 7f7791277740 1 -- start start 2024-04-01T01:34:49.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f77857fa640 1 -- 172.21.15.70:0/2512126787 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f77780cee90 con 0x7f7788100650 2024-04-01T01:34:49.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f7791277740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4a490 0x56527db4a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f7791277740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4ae80 0x56527db4f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f7791277740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4fff0 0x56527db52480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f7791277740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56527daaf1b0 con 0x56527db4ae80 2024-04-01T01:34:49.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f7791277740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56527da57c00 con 0x56527db4a490 2024-04-01T01:34:49.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f7791277740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56527daab250 con 0x56527db4fff0 2024-04-01T01:34:49.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f7787fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4a490 0x56527db4a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f7787fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4a490 0x56527db4a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46500/0 (socket says 172.21.15.70:46500) 2024-04-01T01:34:49.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f7787fff640 1 -- 172.21.15.70:0/1077067041 learned_addr learned my addr 172.21.15.70:0/1077067041 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:49.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f778cc2c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4fff0 0x56527db52480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f7787fff640 1 -- 172.21.15.70:0/1077067041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4fff0 msgr2=0x56527db52480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f7787fff640 1 --2- 172.21.15.70:0/1077067041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4fff0 0x56527db52480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f7787fff640 1 -- 172.21.15.70:0/1077067041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4ae80 msgr2=0x56527db4f980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:34:49.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f7787fff640 1 --2- 172.21.15.70:0/1077067041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4ae80 0x56527db4f980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f7787fff640 1 -- 172.21.15.70:0/1077067041 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56527da773d0 con 0x56527db4a490 2024-04-01T01:34:49.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f77877fe640 1 --2- 172.21.15.70:0/2512126787 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77700a25f0 0x7f77700a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.168+0000 7f7787fff640 1 --2- 172.21.15.70:0/1077067041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4a490 0x56527db4a8b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f77780a3a00 tx=0x7f777809b190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=887d07df6f2dfca7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.169+0000 7f776e7fc640 1 -- 172.21.15.70:0/1077067041 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77780b3020 con 0x56527db4a490 2024-04-01T01:34:49.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.169+0000 7f776e7fc640 1 -- 172.21.15.70:0/1077067041 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f777809f670 con 0x56527db4a490 2024-04-01T01:34:49.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.169+0000 7f776e7fc640 1 -- 172.21.15.70:0/1077067041 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77780a3d00 con 0x56527db4a490 2024-04-01T01:34:49.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.170+0000 7f7791277740 1 -- 172.21.15.70:0/1077067041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4a490 msgr2=0x56527db4a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.170+0000 7f7791277740 1 --2- 172.21.15.70:0/1077067041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4a490 0x56527db4a8b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f77780a3a00 tx=0x7f777809b190 comp rx=0 tx=0).stop 2024-04-01T01:34:49.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.170+0000 7f77877fe640 1 --2- 172.21.15.70:0/2512126787 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77700a25f0 0x7f77700a4ab0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x56527db4b280 tx=0x7f777c00d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.170+0000 7f7791277740 1 -- 172.21.15.70:0/1077067041 shutdown_connections 2024-04-01T01:34:49.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.170+0000 7f7791277740 1 --2- 172.21.15.70:0/1077067041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4fff0 0x56527db52480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.170+0000 7f7791277740 1 --2- 172.21.15.70:0/1077067041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4ae80 0x56527db4f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.170+0000 7f7791277740 1 --2- 172.21.15.70:0/1077067041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4a490 0x56527db4a8b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.170+0000 7f7791277740 1 -- 172.21.15.70:0/1077067041 >> 172.21.15.70:0/1077067041 conn(0x56527db45a00 msgr2=0x56527db45e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.170+0000 7f7791277740 1 -- 172.21.15.70:0/1077067041 shutdown_connections 2024-04-01T01:34:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.170+0000 7f7791277740 1 -- 172.21.15.70:0/1077067041 wait complete. 2024-04-01T01:34:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.170+0000 7f7791277740 1 -- start start 2024-04-01T01:34:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.171+0000 7f7791277740 1 Processor -- start 2024-04-01T01:34:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.171+0000 7f7791277740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4a490 0x56527db65aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.171+0000 7f7791277740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4ae80 0x56527db65fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.171+0000 7f7791277740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4fff0 0x56527db6cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.171+0000 7f7791277740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56527db63f50 con 0x56527db4fff0 2024-04-01T01:34:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.171+0000 7f7791277740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56527da4d470 con 0x56527db4a490 2024-04-01T01:34:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.171+0000 7f7791277740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56527db640d0 con 0x56527db4ae80 2024-04-01T01:34:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.171+0000 7f7791277740 10 client.0 _ll_register_callbacks cb 0x56527da6acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.171+0000 7f7787fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4a490 0x56527db65aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.171+0000 7f77877fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4ae80 0x56527db65fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.171+0000 7f7787fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4a490 0x56527db65aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46512/0 (socket says 172.21.15.70:46512) 2024-04-01T01:34:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.171+0000 7f7787fff640 1 -- 172.21.15.70:0/4261472658 learned_addr learned my addr 172.21.15.70:0/4261472658 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.172+0000 7f778cc2c640 1 --2- 172.21.15.70:0/4261472658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4fff0 0x56527db6cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.172+0000 7f77877fe640 1 -- 172.21.15.70:0/4261472658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4a490 msgr2=0x56527db65aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.172+0000 7f77877fe640 1 --2- 172.21.15.70:0/4261472658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4a490 0x56527db65aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.172+0000 7f77877fe640 1 -- 172.21.15.70:0/4261472658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4fff0 msgr2=0x56527db6cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.172+0000 7f77877fe640 1 --2- 172.21.15.70:0/4261472658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4fff0 0x56527db6cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.172+0000 7f77877fe640 1 -- 172.21.15.70:0/4261472658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56527d7c33f0 con 0x56527db4ae80 2024-04-01T01:34:49.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.172+0000 7f77877fe640 1 --2- 172.21.15.70:0/4261472658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4ae80 0x56527db65fe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x56527db4b8f0 tx=0x7f777c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=231f1d2312e7fc71 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.172+0000 7f77677fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:34:49.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.173+0000 7f77677fe640 1 -- 172.21.15.70:0/4261472658 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777c029060 con 0x56527db4ae80 2024-04-01T01:34:49.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.173+0000 7f77677fe640 1 -- 172.21.15.70:0/4261472658 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f777c003ac0 con 0x56527db4ae80 2024-04-01T01:34:49.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.173+0000 7f7791277740 1 -- 172.21.15.70:0/4261472658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f777809f670 con 0x56527db4ae80 2024-04-01T01:34:49.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.173+0000 7f77677fe640 1 -- 172.21.15.70:0/4261472658 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f777c003cc0 con 0x56527db4ae80 2024-04-01T01:34:49.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.173+0000 7f7791277740 10 client.25711 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:49.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.173+0000 7f7791277740 1 -- 172.21.15.70:0/4261472658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56527d88d7d0 con 0x56527db4ae80 2024-04-01T01:34:49.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.174+0000 7f77677fe640 1 -- 172.21.15.70:0/4261472658 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f777809f670 con 0x56527db4ae80 2024-04-01T01:34:49.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.174+0000 7f77677fe640 1 -- 172.21.15.70:0/4261472658 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f777c016470 con 0x56527db4ae80 2024-04-01T01:34:49.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.174+0000 7f7791277740 10 client.25711 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.174+0000 7f7791277740 10 client.25711 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:34:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.174+0000 7f7791277740 20 client.25711 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.174+0000 7f7791277740 1 --2- 172.21.15.70:0/4261472658 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x56527db71620 0x56527db73a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.174+0000 7f7791277740 4 client.25711 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.174+0000 7f7791277740 1 -- 172.21.15.70:0/4261472658 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56527dabc6f0 con 0x56527db71620 2024-04-01T01:34:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.174+0000 7f7787fff640 1 --2- 172.21.15.70:0/4261472658 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x56527db71620 0x56527db73a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.175+0000 7f7787fff640 1 --2- 172.21.15.70:0/4261472658 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x56527db71620 0x56527db73a40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3b9a2036990a3556 server_cookie=aa5e8d8c21c64a6a in_seq=0 out_seq=0 2024-04-01T01:34:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.175+0000 7f77677fe640 10 client.25711 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:34:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.175+0000 7f77677fe640 1 -- 172.21.15.70:0/4261472658 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56527dabc6f0 con 0x56527db71620 2024-04-01T01:34:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.175+0000 7f77677fe640 10 client.25711 handle_command_reply: tid=0 2024-04-01T01:34:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.175+0000 7f7791277740 1 client.25711 shutdown 2024-04-01T01:34:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.176+0000 7f7791277740 1 -- 172.21.15.70:0/4261472658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4ae80 msgr2=0x56527db65fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.176+0000 7f7791277740 1 --2- 172.21.15.70:0/4261472658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4ae80 0x56527db65fe0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x56527db4b8f0 tx=0x7f777c0046c0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.176+0000 7f7791277740 1 -- 172.21.15.70:0/4261472658 shutdown_connections 2024-04-01T01:34:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.176+0000 7f7791277740 1 --2- 172.21.15.70:0/4261472658 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x56527db71620 0x56527db73a40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.176+0000 7f7791277740 1 --2- 172.21.15.70:0/4261472658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4fff0 0x56527db6cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.176+0000 7f7791277740 1 --2- 172.21.15.70:0/4261472658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4ae80 0x56527db65fe0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.176+0000 7f7791277740 1 --2- 172.21.15.70:0/4261472658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4a490 0x56527db65aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.176+0000 7f7791277740 1 -- 172.21.15.70:0/4261472658 >> 172.21.15.70:0/4261472658 conn(0x56527db45a00 msgr2=0x56527db44d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.176+0000 7f77677fe640 0 client.25711 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:34:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.176+0000 7f7791277740 1 -- 172.21.15.70:0/4261472658 shutdown_connections 2024-04-01T01:34:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.176+0000 7f7791277740 1 -- 172.21.15.70:0/4261472658 wait complete. 2024-04-01T01:34:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.176+0000 7f7791277740 20 client.25711 trim_cache size 0 max 16384 2024-04-01T01:34:49.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.200+0000 7f7791277740 1 Processor -- start 2024-04-01T01:34:49.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.200+0000 7f7791277740 1 -- start start 2024-04-01T01:34:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.201+0000 7f7791277740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4a490 0x56527db65480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.201+0000 7f7791277740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4ae80 0x56527db6cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.201+0000 7f7791277740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4fff0 0x56527db444b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.201+0000 7f7787fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4a490 0x56527db65480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.201+0000 7f7791277740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f777400fd90 con 0x56527db4ae80 2024-04-01T01:34:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.201+0000 7f7791277740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f77740118d0 con 0x56527db4fff0 2024-04-01T01:34:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.201+0000 7f7787fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4a490 0x56527db65480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39172/0 (socket says 172.21.15.70:39172) 2024-04-01T01:34:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.201+0000 7f7787fff640 1 -- 172.21.15.70:0/3712080243 learned_addr learned my addr 172.21.15.70:0/3712080243 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.201+0000 7f7791277740 1 -- 172.21.15.70:0/3712080243 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7774010460 con 0x56527db4a490 2024-04-01T01:34:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.201+0000 7f77877fe640 1 --2- 172.21.15.70:0/3712080243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4ae80 0x56527db6cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.201+0000 7f778cc2c640 1 --2- 172.21.15.70:0/3712080243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4fff0 0x56527db444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.201+0000 7f7787fff640 1 -- 172.21.15.70:0/3712080243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4fff0 msgr2=0x56527db444b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.201+0000 7f7787fff640 1 --2- 172.21.15.70:0/3712080243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4fff0 0x56527db444b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.201+0000 7f7787fff640 1 -- 172.21.15.70:0/3712080243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4ae80 msgr2=0x56527db6cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.201+0000 7f7787fff640 1 --2- 172.21.15.70:0/3712080243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4ae80 0x56527db6cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.202+0000 7f7787fff640 1 -- 172.21.15.70:0/3712080243 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56527db71380 con 0x56527db4a490 2024-04-01T01:34:49.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.202+0000 7f778cc2c640 1 --2- 172.21.15.70:0/3712080243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4fff0 0x56527db444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:34:49.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.202+0000 7f77877fe640 1 --2- 172.21.15.70:0/3712080243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4ae80 0x56527db6cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:34:49.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.203+0000 7f7787fff640 1 --2- 172.21.15.70:0/3712080243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4a490 0x56527db65480 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7778097980 tx=0x7f7778003900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9723a3f0ed6e537 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.203+0000 7f776dffb640 1 -- 172.21.15.70:0/3712080243 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77780b3040 con 0x56527db4a490 2024-04-01T01:34:49.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.203+0000 7f776dffb640 1 -- 172.21.15.70:0/3712080243 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77780b4220 con 0x56527db4a490 2024-04-01T01:34:49.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.203+0000 7f776dffb640 1 -- 172.21.15.70:0/3712080243 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77780bb340 con 0x56527db4a490 2024-04-01T01:34:49.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.204+0000 7f7791277740 1 -- 172.21.15.70:0/3712080243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4a490 msgr2=0x56527db65480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.204+0000 7f7791277740 1 --2- 172.21.15.70:0/3712080243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4a490 0x56527db65480 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7778097980 tx=0x7f7778003900 comp rx=0 tx=0).stop 2024-04-01T01:34:49.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.204+0000 7f7791277740 1 -- 172.21.15.70:0/3712080243 shutdown_connections 2024-04-01T01:34:49.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.204+0000 7f7791277740 1 --2- 172.21.15.70:0/3712080243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4fff0 0x56527db444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.204+0000 7f7791277740 1 --2- 172.21.15.70:0/3712080243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4ae80 0x56527db6cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.204+0000 7f7791277740 1 --2- 172.21.15.70:0/3712080243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4a490 0x56527db65480 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.204+0000 7f7791277740 1 -- 172.21.15.70:0/3712080243 >> 172.21.15.70:0/3712080243 conn(0x56527db45a00 msgr2=0x56527db64d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.204+0000 7f7791277740 1 -- 172.21.15.70:0/3712080243 shutdown_connections 2024-04-01T01:34:49.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.205+0000 7f7791277740 1 -- 172.21.15.70:0/3712080243 wait complete. 2024-04-01T01:34:49.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.205+0000 7f7791277740 1 -- start start 2024-04-01T01:34:49.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.205+0000 7f7791277740 1 Processor -- start 2024-04-01T01:34:49.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.206+0000 7f7791277740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4a490 0x56527db6a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.206+0000 7f7791277740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4ae80 0x56527db6ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.206+0000 7f7791277740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4fff0 0x56527db6bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.206+0000 7f77877fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4ae80 0x56527db6ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.206+0000 7f7787fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4a490 0x56527db6a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.206+0000 7f77877fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4ae80 0x56527db6ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39184/0 (socket says 172.21.15.70:39184) 2024-04-01T01:34:49.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.206+0000 7f77877fe640 1 -- 172.21.15.70:0/2154808613 learned_addr learned my addr 172.21.15.70:0/2154808613 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:49.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.206+0000 7f778cc2c640 1 --2- 172.21.15.70:0/2154808613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4fff0 0x56527db6bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.206+0000 7f7791277740 1 -- 172.21.15.70:0/2154808613 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7774011200 con 0x56527db4a490 2024-04-01T01:34:49.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.206+0000 7f7791277740 1 -- 172.21.15.70:0/2154808613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7774010b30 con 0x56527db4fff0 2024-04-01T01:34:49.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.206+0000 7f7791277740 1 -- 172.21.15.70:0/2154808613 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f777400dd70 con 0x56527db4ae80 2024-04-01T01:34:49.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.206+0000 7f7791277740 10 client.0 _ll_register_callbacks cb 0x7f777c00b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:49.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.207+0000 7f77877fe640 1 -- 172.21.15.70:0/2154808613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4fff0 msgr2=0x56527db6bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.207+0000 7f77877fe640 1 --2- 172.21.15.70:0/2154808613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4fff0 0x56527db6bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.207+0000 7f77877fe640 1 -- 172.21.15.70:0/2154808613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4a490 msgr2=0x56527db6a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.207+0000 7f77877fe640 1 --2- 172.21.15.70:0/2154808613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4a490 0x56527db6a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.207+0000 7f77877fe640 1 -- 172.21.15.70:0/2154808613 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56527db6c390 con 0x56527db4ae80 2024-04-01T01:34:49.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.207+0000 7f77877fe640 1 --2- 172.21.15.70:0/2154808613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4ae80 0x56527db6ad90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f777c017820 tx=0x7f777c003bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71f0edb0b9480549 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.207+0000 7f77677fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:34:49.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.207+0000 7f77677fe640 1 -- 172.21.15.70:0/2154808613 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777c03b060 con 0x56527db4ae80 2024-04-01T01:34:49.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.208+0000 7f77677fe640 1 -- 172.21.15.70:0/2154808613 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f777c020050 con 0x56527db4ae80 2024-04-01T01:34:49.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.208+0000 7f7791277740 1 -- 172.21.15.70:0/2154808613 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77780b4220 con 0x56527db4ae80 2024-04-01T01:34:49.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.208+0000 7f77677fe640 1 -- 172.21.15.70:0/2154808613 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f777c029020 con 0x56527db4ae80 2024-04-01T01:34:49.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.208+0000 7f7791277740 10 client.25717 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:49.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.208+0000 7f7791277740 1 -- 172.21.15.70:0/2154808613 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56527dbd7ab0 con 0x56527db4ae80 2024-04-01T01:34:49.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.208+0000 7f77677fe640 1 -- 172.21.15.70:0/2154808613 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f777c016d10 con 0x56527db4ae80 2024-04-01T01:34:49.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.208+0000 7f77677fe640 1 -- 172.21.15.70:0/2154808613 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f777c004730 con 0x56527db4ae80 2024-04-01T01:34:49.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.208+0000 7f7791277740 10 client.25717 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:49.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.208+0000 7f7791277740 10 client.25717 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:34:49.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.208+0000 7f7791277740 20 client.25717 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:49.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.208+0000 7f7791277740 1 --2- 172.21.15.70:0/2154808613 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x56527dbd84e0 0x56527dbda900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.208+0000 7f7791277740 4 client.25717 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:34:49.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.208+0000 7f7791277740 1 -- 172.21.15.70:0/2154808613 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f77780b4220 con 0x56527dbd84e0 2024-04-01T01:34:49.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.208+0000 7f7787fff640 1 --2- 172.21.15.70:0/2154808613 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x56527dbd84e0 0x56527dbda900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.209+0000 7f7787fff640 1 --2- 172.21.15.70:0/2154808613 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x56527dbd84e0 0x56527dbda900 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=446c87ead0ed9611 server_cookie=ce8a3ac0f0092323 in_seq=0 out_seq=0 2024-04-01T01:34:49.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.209+0000 7f77677fe640 10 client.25717 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:34:49.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.209+0000 7f77677fe640 1 -- 172.21.15.70:0/2154808613 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f77780b4220 con 0x56527dbd84e0 2024-04-01T01:34:49.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.209+0000 7f77677fe640 10 client.25717 handle_command_reply: tid=0 2024-04-01T01:34:49.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.209+0000 7f7791277740 1 client.25717 shutdown 2024-04-01T01:34:49.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.210+0000 7f7791277740 1 -- 172.21.15.70:0/2154808613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4ae80 msgr2=0x56527db6ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.210+0000 7f7791277740 1 --2- 172.21.15.70:0/2154808613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4ae80 0x56527db6ad90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f777c017820 tx=0x7f777c003bd0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.210+0000 7f7791277740 1 -- 172.21.15.70:0/2154808613 shutdown_connections 2024-04-01T01:34:49.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.210+0000 7f7791277740 1 --2- 172.21.15.70:0/2154808613 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x56527dbd84e0 0x56527dbda900 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.210+0000 7f7791277740 1 --2- 172.21.15.70:0/2154808613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56527db4fff0 0x56527db6bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.210+0000 7f7791277740 1 --2- 172.21.15.70:0/2154808613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56527db4ae80 0x56527db6ad90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.210+0000 7f7791277740 1 --2- 172.21.15.70:0/2154808613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56527db4a490 0x56527db6a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.210+0000 7f7791277740 1 -- 172.21.15.70:0/2154808613 >> 172.21.15.70:0/2154808613 conn(0x56527db45a00 msgr2=0x56527db48c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.210+0000 7f77677fe640 0 client.25717 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:34:49.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.210+0000 7f7791277740 1 -- 172.21.15.70:0/2154808613 shutdown_connections 2024-04-01T01:34:49.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.211+0000 7f7791277740 1 -- 172.21.15.70:0/2154808613 wait complete. 2024-04-01T01:34:49.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.211+0000 7f7791277740 20 client.25717 trim_cache size 0 max 16384 2024-04-01T01:34:49.213 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:34:49.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.211+0000 7f778e68f640 1 -- 172.21.15.70:0/2512126787 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77700a25f0 msgr2=0x7f77700a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.211+0000 7f778e68f640 1 --2- 172.21.15.70:0/2512126787 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77700a25f0 0x7f77700a4ab0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x56527db4b280 tx=0x7f777c00d040 comp rx=0 tx=0).stop 2024-04-01T01:34:49.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.212+0000 7f778e68f640 1 -- 172.21.15.70:0/2512126787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7788100650 msgr2=0x7f77881957a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.212+0000 7f778e68f640 1 --2- 172.21.15.70:0/2512126787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7788100650 0x7f77881957a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7778012f90 tx=0x7f7778010490 comp rx=0 tx=0).stop 2024-04-01T01:34:49.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.212+0000 7f778e68f640 1 -- 172.21.15.70:0/2512126787 shutdown_connections 2024-04-01T01:34:49.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.212+0000 7f778e68f640 1 --2- 172.21.15.70:0/2512126787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7788102200 0x7f778819cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.212+0000 7f778e68f640 1 --2- 172.21.15.70:0/2512126787 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77700a25f0 0x7f77700a4ab0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.212+0000 7f778e68f640 1 --2- 172.21.15.70:0/2512126787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7788101840 0x7f7788195ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.212+0000 7f778e68f640 1 --2- 172.21.15.70:0/2512126787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7788100650 0x7f77881957a0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.212+0000 7f778e68f640 1 -- 172.21.15.70:0/2512126787 >> 172.21.15.70:0/2512126787 conn(0x7f77880fbbe0 msgr2=0x7f7788104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.212+0000 7f778e68f640 1 -- 172.21.15.70:0/2512126787 shutdown_connections 2024-04-01T01:34:49.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.212+0000 7f778e68f640 1 -- 172.21.15.70:0/2512126787 wait complete. 2024-04-01T01:34:49.229 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:34:49.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.398+0000 7fe1a59d0640 1 -- 172.21.15.70:0/1382776213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1a00fffc0 msgr2=0x7fe1a01023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.398+0000 7fe1a59d0640 1 --2- 172.21.15.70:0/1382776213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1a00fffc0 0x7fe1a01023f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe190009860 tx=0x7fe19002bb70 comp rx=0 tx=0).stop 2024-04-01T01:34:49.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.398+0000 7fe1a59d0640 1 -- 172.21.15.70:0/1382776213 shutdown_connections 2024-04-01T01:34:49.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.398+0000 7fe1a59d0640 1 --2- 172.21.15.70:0/1382776213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a01052e0 0x7fe1a0105760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.398+0000 7fe1a59d0640 1 --2- 172.21.15.70:0/1382776213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1a0102930 0x7fe1a0104da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.398+0000 7fe1a59d0640 1 --2- 172.21.15.70:0/1382776213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1a00fffc0 0x7fe1a01023f0 secure :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe190009860 tx=0x7fe19002bb70 comp rx=0 tx=0).stop 2024-04-01T01:34:49.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.398+0000 7fe1a59d0640 1 -- 172.21.15.70:0/1382776213 >> 172.21.15.70:0/1382776213 conn(0x7fe1a00fbbe0 msgr2=0x7fe1a00fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.399+0000 7fe1a59d0640 1 -- 172.21.15.70:0/1382776213 shutdown_connections 2024-04-01T01:34:49.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.399+0000 7fe1a59d0640 1 -- 172.21.15.70:0/1382776213 wait complete. 2024-04-01T01:34:49.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.399+0000 7fe1a59d0640 1 Processor -- start 2024-04-01T01:34:49.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.399+0000 7fe1a59d0640 1 -- start start 2024-04-01T01:34:49.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.400+0000 7fe1a59d0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1a0102930 0x7fe1a019e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.400+0000 7fe1a59d0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a01052e0 0x7fe1a019e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.400+0000 7fe1a59d0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1a01a3750 0x7fe1a01a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.400+0000 7fe1a59d0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a0113850 con 0x7fe1a01a3750 2024-04-01T01:34:49.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.400+0000 7fe1a59d0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a01136d0 con 0x7fe1a0102930 2024-04-01T01:34:49.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.400+0000 7fe1a59d0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a01139d0 con 0x7fe1a01052e0 2024-04-01T01:34:49.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.400+0000 7fe19effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1a0102930 0x7fe1a019e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.400+0000 7fe19effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1a0102930 0x7fe1a019e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46536/0 (socket says 172.21.15.70:46536) 2024-04-01T01:34:49.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.400+0000 7fe19effd640 1 -- 172.21.15.70:0/2988093951 learned_addr learned my addr 172.21.15.70:0/2988093951 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:49.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.400+0000 7fe197fff640 1 --2- 172.21.15.70:0/2988093951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a01052e0 0x7fe1a019e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.400+0000 7fe19f7fe640 1 --2- 172.21.15.70:0/2988093951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1a01a3750 0x7fe1a01a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.400+0000 7fe19effd640 1 -- 172.21.15.70:0/2988093951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a01052e0 msgr2=0x7fe1a019e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.400+0000 7fe19effd640 1 --2- 172.21.15.70:0/2988093951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a01052e0 0x7fe1a019e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.400+0000 7fe19effd640 1 -- 172.21.15.70:0/2988093951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1a01a3750 msgr2=0x7fe1a01a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.400+0000 7fe19effd640 1 --2- 172.21.15.70:0/2988093951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1a01a3750 0x7fe1a01a5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.400+0000 7fe19effd640 1 -- 172.21.15.70:0/2988093951 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1a01a61b0 con 0x7fe1a0102930 2024-04-01T01:34:49.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.401+0000 7fe19effd640 1 --2- 172.21.15.70:0/2988093951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1a0102930 0x7fe1a019e1b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe190037040 tx=0x7fe190035d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.401+0000 7fe1a49ce640 1 -- 172.21.15.70:0/2988093951 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe190042070 con 0x7fe1a0102930 2024-04-01T01:34:49.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.401+0000 7fe1a49ce640 1 -- 172.21.15.70:0/2988093951 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1900054e0 con 0x7fe1a0102930 2024-04-01T01:34:49.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.401+0000 7fe1a49ce640 1 -- 172.21.15.70:0/2988093951 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe190039040 con 0x7fe1a0102930 2024-04-01T01:34:49.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.401+0000 7fe1a59d0640 1 -- 172.21.15.70:0/2988093951 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1a01a6440 con 0x7fe1a0102930 2024-04-01T01:34:49.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.401+0000 7fe1a59d0640 1 -- 172.21.15.70:0/2988093951 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1a01a68e0 con 0x7fe1a0102930 2024-04-01T01:34:49.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.405+0000 7fe1a85a8740 1 Processor -- start 2024-04-01T01:34:49.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.405+0000 7fe1a85a8740 1 -- start start 2024-04-01T01:34:49.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.406+0000 7fe1a85a8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.406+0000 7fe1a85a8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96cce80 0x562fc96d1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.406+0000 7fe1a85a8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96d1ff0 0x562fc96d4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.406+0000 7fe1a85a8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562fc96311b0 con 0x562fc96cce80 2024-04-01T01:34:49.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.406+0000 7fe1a85a8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562fc95d9c00 con 0x562fc96cc490 2024-04-01T01:34:49.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.406+0000 7fe1a85a8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562fc962d250 con 0x562fc96d1ff0 2024-04-01T01:34:49.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.406+0000 7fe19f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.406+0000 7fe19f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96cc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46552/0 (socket says 172.21.15.70:46552) 2024-04-01T01:34:49.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.406+0000 7fe19f7fe640 1 -- 172.21.15.70:0/2611959952 learned_addr learned my addr 172.21.15.70:0/2611959952 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:49.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.407+0000 7fe197fff640 1 --2- 172.21.15.70:0/2611959952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96d1ff0 0x562fc96d4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.407+0000 7fe1a49ce640 1 -- 172.21.15.70:0/2988093951 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fe190045070 con 0x7fe1a0102930 2024-04-01T01:34:49.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.407+0000 7fe19f7fe640 1 -- 172.21.15.70:0/2611959952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96d1ff0 msgr2=0x562fc96d4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.407+0000 7fe19f7fe640 1 --2- 172.21.15.70:0/2611959952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96d1ff0 0x562fc96d4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.407+0000 7fe19f7fe640 1 -- 172.21.15.70:0/2611959952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96cce80 msgr2=0x562fc96d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.407+0000 7fe19effd640 1 --2- 172.21.15.70:0/2611959952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96cce80 0x562fc96d1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.407+0000 7fe19f7fe640 1 --2- 172.21.15.70:0/2611959952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96cce80 0x562fc96d1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.408+0000 7fe19f7fe640 1 -- 172.21.15.70:0/2611959952 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fc95f93d0 con 0x562fc96cc490 2024-04-01T01:34:49.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.408+0000 7fe19effd640 1 --2- 172.21.15.70:0/2611959952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96cce80 0x562fc96d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:34:49.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.408+0000 7fe1a49ce640 1 --2- 172.21.15.70:0/2988093951 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe18c0a2670 0x7fe18c0a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.408+0000 7fe19f7fe640 1 --2- 172.21.15.70:0/2611959952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96cc8b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe198009f40 tx=0x7fe198010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8752724ece558c8a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.408+0000 7fe1a49ce640 1 -- 172.21.15.70:0/2988093951 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe1900ef910 con 0x7fe1a0102930 2024-04-01T01:34:49.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.408+0000 7fe19effd640 1 --2- 172.21.15.70:0/2988093951 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe18c0a2670 0x7fe18c0a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.409+0000 7fe195ffb640 1 -- 172.21.15.70:0/2611959952 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe198003b70 con 0x562fc96cc490 2024-04-01T01:34:49.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.409+0000 7fe19effd640 1 --2- 172.21.15.70:0/2988093951 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe18c0a2670 0x7fe18c0a4b30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe190009830 tx=0x7fe1900347b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.409+0000 7fe195ffb640 1 -- 172.21.15.70:0/2611959952 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe198003d10 con 0x562fc96cc490 2024-04-01T01:34:49.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.409+0000 7fe195ffb640 1 -- 172.21.15.70:0/2611959952 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1980119c0 con 0x562fc96cc490 2024-04-01T01:34:49.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.410+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2611959952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 msgr2=0x562fc96cc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.410+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/2611959952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96cc8b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe198009f40 tx=0x7fe198010860 comp rx=0 tx=0).stop 2024-04-01T01:34:49.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.411+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2611959952 shutdown_connections 2024-04-01T01:34:49.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.411+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/2611959952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96d1ff0 0x562fc96d4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.411+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/2611959952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96cce80 0x562fc96d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.411+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/2611959952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96cc8b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.411+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2611959952 >> 172.21.15.70:0/2611959952 conn(0x562fc96c7a00 msgr2=0x562fc96c7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.411+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2611959952 shutdown_connections 2024-04-01T01:34:49.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.411+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2611959952 wait complete. 2024-04-01T01:34:49.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.412+0000 7fe1a85a8740 1 -- start start 2024-04-01T01:34:49.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.412+0000 7fe1a85a8740 1 Processor -- start 2024-04-01T01:34:49.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.412+0000 7fe1a85a8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96d6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.412+0000 7fe1a85a8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96cce80 0x562fc96d6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.412+0000 7fe1a85a8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96d1ff0 0x562fc96ddfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.412+0000 7fe1a85a8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562fc96d4fa0 con 0x562fc96d1ff0 2024-04-01T01:34:49.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.412+0000 7fe1a85a8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562fc95cf470 con 0x562fc96cc490 2024-04-01T01:34:49.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.413+0000 7fe1a85a8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562fc96d5120 con 0x562fc96cce80 2024-04-01T01:34:49.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.413+0000 7fe1a85a8740 10 client.0 _ll_register_callbacks cb 0x562fc95eccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:49.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.413+0000 7fe19f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96d6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.413+0000 7fe19f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96d6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46566/0 (socket says 172.21.15.70:46566) 2024-04-01T01:34:49.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.413+0000 7fe19f7fe640 1 -- 172.21.15.70:0/2247644320 learned_addr learned my addr 172.21.15.70:0/2247644320 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:49.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.413+0000 7fe19f7fe640 1 --2- 172.21.15.70:0/2247644320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96d1ff0 0x562fc96ddfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.413+0000 7fe197fff640 1 --2- 172.21.15.70:0/2247644320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96cce80 0x562fc96d6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.413+0000 7fe19f7fe640 1 -- 172.21.15.70:0/2247644320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96cce80 msgr2=0x562fc96d6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.413+0000 7fe19f7fe640 1 --2- 172.21.15.70:0/2247644320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96cce80 0x562fc96d6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.413+0000 7fe19f7fe640 1 -- 172.21.15.70:0/2247644320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96d1ff0 msgr2=0x562fc96ddfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.413+0000 7fe19f7fe640 1 --2- 172.21.15.70:0/2247644320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96d1ff0 0x562fc96ddfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.413+0000 7fe19f7fe640 1 -- 172.21.15.70:0/2247644320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fc93453f0 con 0x562fc96cc490 2024-04-01T01:34:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.414+0000 7fe19f7fe640 1 --2- 172.21.15.70:0/2247644320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96d6a20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe1980044b0 tx=0x7fe198013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=795376be47e59343 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.414+0000 7fe17effd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:34:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.414+0000 7fe17effd640 1 -- 172.21.15.70:0/2247644320 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe198011740 con 0x562fc96cc490 2024-04-01T01:34:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.414+0000 7fe17effd640 1 -- 172.21.15.70:0/2247644320 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe1980118e0 con 0x562fc96cc490 2024-04-01T01:34:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.414+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2247644320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe198003d10 con 0x562fc96cc490 2024-04-01T01:34:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.415+0000 7fe17effd640 1 -- 172.21.15.70:0/2247644320 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe198034050 con 0x562fc96cc490 2024-04-01T01:34:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.415+0000 7fe1a85a8740 10 client.25811 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.415+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2247644320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562fc940f7d0 con 0x562fc96cc490 2024-04-01T01:34:49.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.415+0000 7fe17effd640 1 -- 172.21.15.70:0/2247644320 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe198003d10 con 0x562fc96cc490 2024-04-01T01:34:49.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.415+0000 7fe17effd640 1 -- 172.21.15.70:0/2247644320 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe198029040 con 0x562fc96cc490 2024-04-01T01:34:49.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.415+0000 7fe1a85a8740 10 client.25811 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:49.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.415+0000 7fe1a85a8740 10 client.25811 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:34:49.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.415+0000 7fe1a85a8740 20 client.25811 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:49.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.415+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/2247644320 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562fc96f31f0 0x562fc96f5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.415+0000 7fe1a85a8740 4 client.25811 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:49.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.415+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2247644320 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fc963e6f0 con 0x562fc96f31f0 2024-04-01T01:34:49.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.415+0000 7fe197fff640 1 --2- 172.21.15.70:0/2247644320 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562fc96f31f0 0x562fc96f5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.416+0000 7fe197fff640 1 --2- 172.21.15.70:0/2247644320 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562fc96f31f0 0x562fc96f5620 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=16626d965688099e server_cookie=65735f80dd3a0cf3 in_seq=0 out_seq=0 2024-04-01T01:34:49.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.416+0000 7fe17effd640 10 client.25811 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:34:49.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.417+0000 7fe17effd640 1 -- 172.21.15.70:0/2247644320 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562fc963e6f0 con 0x562fc96f31f0 2024-04-01T01:34:49.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.417+0000 7fe17effd640 10 client.25811 handle_command_reply: tid=0 2024-04-01T01:34:49.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.417+0000 7fe1a85a8740 1 client.25811 shutdown 2024-04-01T01:34:49.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.417+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2247644320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 msgr2=0x562fc96d6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.417+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/2247644320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96d6a20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe1980044b0 tx=0x7fe198013040 comp rx=0 tx=0).stop 2024-04-01T01:34:49.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.417+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2247644320 shutdown_connections 2024-04-01T01:34:49.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.417+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/2247644320 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562fc96f31f0 0x562fc96f5620 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.417+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/2247644320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96d1ff0 0x562fc96ddfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.417+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/2247644320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96cce80 0x562fc96d6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.417+0000 7fe17effd640 0 client.25811 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:34:49.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.417+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/2247644320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96d6a20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.417+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2247644320 >> 172.21.15.70:0/2247644320 conn(0x562fc96c7a00 msgr2=0x562fc96c6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.418+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2247644320 shutdown_connections 2024-04-01T01:34:49.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.418+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2247644320 wait complete. 2024-04-01T01:34:49.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.418+0000 7fe1a85a8740 20 client.25811 trim_cache size 0 max 16384 2024-04-01T01:34:49.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.441+0000 7fe1a85a8740 1 Processor -- start 2024-04-01T01:34:49.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.441+0000 7fe1a85a8740 1 -- start start 2024-04-01T01:34:49.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.441+0000 7fe1a85a8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96cc490 0x562fc96d6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.442+0000 7fe1a85a8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cce80 0x562fc96ddf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.442+0000 7fe1a85a8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96d1ff0 0x562fc96cb370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.442+0000 7fe1a85a8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a01a87f0 con 0x562fc96cc490 2024-04-01T01:34:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.442+0000 7fe1a85a8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a01aa330 con 0x562fc96cce80 2024-04-01T01:34:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.442+0000 7fe1a85a8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a01a8ec0 con 0x562fc96d1ff0 2024-04-01T01:34:49.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.442+0000 7fe19f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96cc490 0x562fc96d6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.442+0000 7fe19f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96cc490 0x562fc96d6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41722/0 (socket says 172.21.15.70:41722) 2024-04-01T01:34:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.442+0000 7fe19f7fe640 1 -- 172.21.15.70:0/2275912690 learned_addr learned my addr 172.21.15.70:0/2275912690 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.442+0000 7fe197fff640 1 --2- 172.21.15.70:0/2275912690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cce80 0x562fc96ddf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.442+0000 7fe19f7fe640 1 --2- 172.21.15.70:0/2275912690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96d1ff0 0x562fc96cb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.442+0000 7fe197fff640 1 -- 172.21.15.70:0/2275912690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96d1ff0 msgr2=0x562fc96cb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.442+0000 7fe197fff640 1 --2- 172.21.15.70:0/2275912690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96d1ff0 0x562fc96cb370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.442+0000 7fe197fff640 1 -- 172.21.15.70:0/2275912690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96cc490 msgr2=0x562fc96d6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.442+0000 7fe197fff640 1 --2- 172.21.15.70:0/2275912690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96cc490 0x562fc96d6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.442+0000 7fe197fff640 1 -- 172.21.15.70:0/2275912690 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fc96c8450 con 0x562fc96cce80 2024-04-01T01:34:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.442+0000 7fe19f7fe640 1 --2- 172.21.15.70:0/2275912690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96cc490 0x562fc96d6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.443+0000 7fe197fff640 1 --2- 172.21.15.70:0/2275912690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cce80 0x562fc96ddf30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x562fc96f5e60 tx=0x7fe1880044d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8a8eecbb309e384 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.443+0000 7fe1957fa640 1 -- 172.21.15.70:0/2275912690 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe188003ba0 con 0x562fc96cce80 2024-04-01T01:34:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.443+0000 7fe1957fa640 1 -- 172.21.15.70:0/2275912690 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe188003d40 con 0x562fc96cce80 2024-04-01T01:34:49.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.443+0000 7fe1957fa640 1 -- 172.21.15.70:0/2275912690 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe188011900 con 0x562fc96cce80 2024-04-01T01:34:49.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.444+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2275912690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cce80 msgr2=0x562fc96ddf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.444+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/2275912690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cce80 0x562fc96ddf30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x562fc96f5e60 tx=0x7fe1880044d0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.444+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2275912690 shutdown_connections 2024-04-01T01:34:49.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.444+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/2275912690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96d1ff0 0x562fc96cb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.444+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/2275912690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cce80 0x562fc96ddf30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.444+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/2275912690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96cc490 0x562fc96d6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.444+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2275912690 >> 172.21.15.70:0/2275912690 conn(0x562fc96c7a00 msgr2=0x562fc96cac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.445+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2275912690 shutdown_connections 2024-04-01T01:34:49.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.445+0000 7fe1a85a8740 1 -- 172.21.15.70:0/2275912690 wait complete. 2024-04-01T01:34:49.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.445+0000 7fe1a85a8740 1 -- start start 2024-04-01T01:34:49.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.446+0000 7fe1a85a8740 1 Processor -- start 2024-04-01T01:34:49.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.446+0000 7fe1a85a8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96dbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.446+0000 7fe1a85a8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96cce80 0x562fc96dc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.446+0000 7fe1a85a8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96d1ff0 0x562fc96dd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.446+0000 7fe19f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96dbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.447+0000 7fe1a85a8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a01a9c60 con 0x562fc96d1ff0 2024-04-01T01:34:49.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.447+0000 7fe1a85a8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a01a9590 con 0x562fc96cc490 2024-04-01T01:34:49.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.447+0000 7fe1a85a8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a010e9c0 con 0x562fc96cce80 2024-04-01T01:34:49.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.447+0000 7fe19f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96dbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46584/0 (socket says 172.21.15.70:46584) 2024-04-01T01:34:49.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.447+0000 7fe19f7fe640 1 -- 172.21.15.70:0/318684031 learned_addr learned my addr 172.21.15.70:0/318684031 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:49.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.447+0000 7fe1a85a8740 10 client.0 _ll_register_callbacks cb 0x7fe198004140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:49.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.447+0000 7fe19f7fe640 1 --2- 172.21.15.70:0/318684031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96d1ff0 0x562fc96dd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.447+0000 7fe197fff640 1 --2- 172.21.15.70:0/318684031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96cce80 0x562fc96dc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.447+0000 7fe19f7fe640 1 -- 172.21.15.70:0/318684031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96cce80 msgr2=0x562fc96dc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.447+0000 7fe19f7fe640 1 --2- 172.21.15.70:0/318684031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96cce80 0x562fc96dc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.447+0000 7fe19f7fe640 1 -- 172.21.15.70:0/318684031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 msgr2=0x562fc96dbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.447+0000 7fe19f7fe640 1 --2- 172.21.15.70:0/318684031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96dbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.447+0000 7fe19f7fe640 1 -- 172.21.15.70:0/318684031 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fc96dd790 con 0x562fc96d1ff0 2024-04-01T01:34:49.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.448+0000 7fe19f7fe640 1 --2- 172.21.15.70:0/318684031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96d1ff0 0x562fc96dd1a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe19802bbe0 tx=0x7fe198013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0c7554d9a7deda server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.448+0000 7fe17effd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:49.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.448+0000 7fe17effd640 1 -- 172.21.15.70:0/318684031 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1980099c0 con 0x562fc96d1ff0 2024-04-01T01:34:49.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.448+0000 7fe17effd640 1 -- 172.21.15.70:0/318684031 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe19802b460 con 0x562fc96d1ff0 2024-04-01T01:34:49.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.448+0000 7fe1a85a8740 1 -- 172.21.15.70:0/318684031 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe188003d40 con 0x562fc96d1ff0 2024-04-01T01:34:49.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.449+0000 7fe17effd640 1 -- 172.21.15.70:0/318684031 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe19802b870 con 0x562fc96d1ff0 2024-04-01T01:34:49.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.449+0000 7fe1a85a8740 10 client.15996 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:49.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.449+0000 7fe1a85a8740 1 -- 172.21.15.70:0/318684031 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562fc9759b10 con 0x562fc96d1ff0 2024-04-01T01:34:49.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.449+0000 7fe17effd640 1 -- 172.21.15.70:0/318684031 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe188003d40 con 0x562fc96d1ff0 2024-04-01T01:34:49.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.449+0000 7fe17effd640 1 -- 172.21.15.70:0/318684031 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe198044070 con 0x562fc96d1ff0 2024-04-01T01:34:49.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.449+0000 7fe1a85a8740 10 client.15996 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:49.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.449+0000 7fe1a85a8740 10 client.15996 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:34:49.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.449+0000 7fe1a85a8740 20 client.15996 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:49.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.449+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/318684031 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562fc975a3c0 0x562fc975c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.449+0000 7fe1a85a8740 4 client.15996 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:34:49.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.449+0000 7fe1a85a8740 1 -- 172.21.15.70:0/318684031 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562fc9638a40 con 0x562fc975a3c0 2024-04-01T01:34:49.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.449+0000 7fe197fff640 1 --2- 172.21.15.70:0/318684031 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562fc975a3c0 0x562fc975c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.450+0000 7fe197fff640 1 --2- 172.21.15.70:0/318684031 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562fc975a3c0 0x562fc975c7e0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=db2cbb28beb78629 server_cookie=c7fcd7da318e50a4 in_seq=0 out_seq=0 2024-04-01T01:34:49.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.450+0000 7fe17effd640 10 client.15996 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:34:49.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.450+0000 7fe17effd640 1 -- 172.21.15.70:0/318684031 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562fc9638a40 con 0x562fc975a3c0 2024-04-01T01:34:49.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.450+0000 7fe17effd640 10 client.15996 handle_command_reply: tid=0 2024-04-01T01:34:49.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.451+0000 7fe1a85a8740 1 client.15996 shutdown 2024-04-01T01:34:49.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.451+0000 7fe1a85a8740 1 -- 172.21.15.70:0/318684031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96d1ff0 msgr2=0x562fc96dd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.451+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/318684031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96d1ff0 0x562fc96dd1a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe19802bbe0 tx=0x7fe198013040 comp rx=0 tx=0).stop 2024-04-01T01:34:49.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.451+0000 7fe1a85a8740 1 -- 172.21.15.70:0/318684031 shutdown_connections 2024-04-01T01:34:49.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.451+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/318684031 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562fc975a3c0 0x562fc975c7e0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.451+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/318684031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562fc96d1ff0 0x562fc96dd1a0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.451+0000 7fe17effd640 0 client.15996 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:34:49.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.451+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/318684031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fc96cce80 0x562fc96dc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.451+0000 7fe1a85a8740 1 --2- 172.21.15.70:0/318684031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562fc96cc490 0x562fc96dbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.451+0000 7fe1a85a8740 1 -- 172.21.15.70:0/318684031 >> 172.21.15.70:0/318684031 conn(0x562fc96c7a00 msgr2=0x562fc96cb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.451+0000 7fe1a85a8740 1 -- 172.21.15.70:0/318684031 shutdown_connections 2024-04-01T01:34:49.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.451+0000 7fe1a85a8740 1 -- 172.21.15.70:0/318684031 wait complete. 2024-04-01T01:34:49.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.452+0000 7fe1a85a8740 20 client.15996 trim_cache size 0 max 16384 2024-04-01T01:34:49.455 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:34:49.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.452+0000 7fe1a59d0640 1 -- 172.21.15.70:0/2988093951 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe18c0a2670 msgr2=0x7fe18c0a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.452+0000 7fe1a59d0640 1 --2- 172.21.15.70:0/2988093951 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe18c0a2670 0x7fe18c0a4b30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe190009830 tx=0x7fe1900347b0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.452+0000 7fe1a59d0640 1 -- 172.21.15.70:0/2988093951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1a0102930 msgr2=0x7fe1a019e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.453+0000 7fe1a59d0640 1 --2- 172.21.15.70:0/2988093951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1a0102930 0x7fe1a019e1b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe190037040 tx=0x7fe190035d40 comp rx=0 tx=0).stop 2024-04-01T01:34:49.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.453+0000 7fe1a59d0640 1 -- 172.21.15.70:0/2988093951 shutdown_connections 2024-04-01T01:34:49.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.453+0000 7fe1a59d0640 1 --2- 172.21.15.70:0/2988093951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1a01a3750 0x7fe1a01a5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.453+0000 7fe1a59d0640 1 --2- 172.21.15.70:0/2988093951 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe18c0a2670 0x7fe18c0a4b30 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.453+0000 7fe1a59d0640 1 --2- 172.21.15.70:0/2988093951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a01052e0 0x7fe1a019e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.453+0000 7fe1a59d0640 1 --2- 172.21.15.70:0/2988093951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1a0102930 0x7fe1a019e1b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.453+0000 7fe1a59d0640 1 -- 172.21.15.70:0/2988093951 >> 172.21.15.70:0/2988093951 conn(0x7fe1a00fbbe0 msgr2=0x7fe1a0103440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.453+0000 7fe1a59d0640 1 -- 172.21.15.70:0/2988093951 shutdown_connections 2024-04-01T01:34:49.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.453+0000 7fe1a59d0640 1 -- 172.21.15.70:0/2988093951 wait complete. 2024-04-01T01:34:49.472 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:34:49.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.647+0000 7fd7002f4640 1 -- 172.21.15.70:0/3959738172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6f8101170 msgr2=0x7fd6f8105a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.647+0000 7fd7002f4640 1 --2- 172.21.15.70:0/3959738172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6f8101170 0x7fd6f8105a10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0009be0 tx=0x7fd6f002bef0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.647+0000 7fd7002f4640 1 -- 172.21.15.70:0/3959738172 shutdown_connections 2024-04-01T01:34:49.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.647+0000 7fd7002f4640 1 --2- 172.21.15.70:0/3959738172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6f8101170 0x7fd6f8105a10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.647+0000 7fd7002f4640 1 --2- 172.21.15.70:0/3959738172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6f81007b0 0x7fd6f8100c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.647+0000 7fd7002f4640 1 --2- 172.21.15.70:0/3959738172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f80ffa50 0x7fd6f80ffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.647+0000 7fd7002f4640 1 -- 172.21.15.70:0/3959738172 >> 172.21.15.70:0/3959738172 conn(0x7fd6f80fb380 msgr2=0x7fd6f80fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.647+0000 7fd7002f4640 1 -- 172.21.15.70:0/3959738172 shutdown_connections 2024-04-01T01:34:49.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.647+0000 7fd7002f4640 1 -- 172.21.15.70:0/3959738172 wait complete. 2024-04-01T01:34:49.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.648+0000 7fd7002f4640 1 Processor -- start 2024-04-01T01:34:49.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.648+0000 7fd7002f4640 1 -- start start 2024-04-01T01:34:49.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.648+0000 7fd7002f4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f80ffa50 0x7fd6f810ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.648+0000 7fd7002f4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6f81007b0 0x7fd6f810f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.649+0000 7fd6fe090640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f80ffa50 0x7fd6f810ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.649+0000 7fd6fe090640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f80ffa50 0x7fd6f810ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39234/0 (socket says 172.21.15.70:39234) 2024-04-01T01:34:49.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.649+0000 7fd6fe090640 1 -- 172.21.15.70:0/4236544238 learned_addr learned my addr 172.21.15.70:0/4236544238 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:49.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.649+0000 7fd6fd88f640 1 --2- 172.21.15.70:0/4236544238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6f81007b0 0x7fd6f810f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.649+0000 7fd7002f4640 1 --2- 172.21.15.70:0/4236544238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6f8101170 0x7fd6f8110330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.649+0000 7fd7002f4640 1 -- 172.21.15.70:0/4236544238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f81076d0 con 0x7fd6f8101170 2024-04-01T01:34:49.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.649+0000 7fd6fe891640 1 --2- 172.21.15.70:0/4236544238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6f8101170 0x7fd6f8110330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.649+0000 7fd7002f4640 1 -- 172.21.15.70:0/4236544238 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f8107550 con 0x7fd6f81007b0 2024-04-01T01:34:49.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.649+0000 7fd7002f4640 1 -- 172.21.15.70:0/4236544238 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f8107850 con 0x7fd6f80ffa50 2024-04-01T01:34:49.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.649+0000 7fd6fe090640 1 -- 172.21.15.70:0/4236544238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6f81007b0 msgr2=0x7fd6f810f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.649+0000 7fd6fe090640 1 --2- 172.21.15.70:0/4236544238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6f81007b0 0x7fd6f810f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.650+0000 7fd6fe090640 1 -- 172.21.15.70:0/4236544238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6f8101170 msgr2=0x7fd6f8110330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.650+0000 7fd6fe090640 1 --2- 172.21.15.70:0/4236544238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6f8101170 0x7fd6f8110330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.650+0000 7fd6fe090640 1 -- 172.21.15.70:0/4236544238 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f81109a0 con 0x7fd6f80ffa50 2024-04-01T01:34:49.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.651+0000 7fd6fe090640 1 --2- 172.21.15.70:0/4236544238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f80ffa50 0x7fd6f810ee00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd6f400e9c0 tx=0x7fd6f400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.651+0000 7fd6ef7fe640 1 -- 172.21.15.70:0/4236544238 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f400ccc0 con 0x7fd6f80ffa50 2024-04-01T01:34:49.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.651+0000 7fd7002f4640 1 -- 172.21.15.70:0/4236544238 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f8110c90 con 0x7fd6f80ffa50 2024-04-01T01:34:49.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.651+0000 7fd7002f4640 1 -- 172.21.15.70:0/4236544238 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f8111230 con 0x7fd6f80ffa50 2024-04-01T01:34:49.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.651+0000 7fd6ef7fe640 1 -- 172.21.15.70:0/4236544238 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6f40054d0 con 0x7fd6f80ffa50 2024-04-01T01:34:49.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.651+0000 7fd6ef7fe640 1 -- 172.21.15.70:0/4236544238 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4010430 con 0x7fd6f80ffa50 2024-04-01T01:34:49.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.655+0000 7fd702fb4740 1 Processor -- start 2024-04-01T01:34:49.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.656+0000 7fd702fb4740 1 -- start start 2024-04-01T01:34:49.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.656+0000 7fd6ef7fe640 1 -- 172.21.15.70:0/4236544238 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fd6f40105d0 con 0x7fd6f80ffa50 2024-04-01T01:34:49.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.656+0000 7fd702fb4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171490 0x55d6051718b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.656+0000 7fd702fb4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605171e80 0x55d605176980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.656+0000 7fd702fb4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605176ff0 0x55d605179480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.656+0000 7fd702fb4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d6050d61b0 con 0x55d605171490 2024-04-01T01:34:49.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.656+0000 7fd702fb4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d60507ec00 con 0x55d605176ff0 2024-04-01T01:34:49.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.656+0000 7fd702fb4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d6050d2250 con 0x55d605171e80 2024-04-01T01:34:49.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.656+0000 7fd6fe891640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171490 0x55d6051718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.656+0000 7fd6fe891640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171490 0x55d6051718b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41754/0 (socket says 172.21.15.70:41754) 2024-04-01T01:34:49.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.656+0000 7fd6fe891640 1 -- 172.21.15.70:0/194169214 learned_addr learned my addr 172.21.15.70:0/194169214 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:49.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.656+0000 7fd6fd88f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605176ff0 0x55d605179480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.656+0000 7fd6fe090640 1 --2- 172.21.15.70:0/194169214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605171e80 0x55d605176980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.657+0000 7fd6fe090640 1 -- 172.21.15.70:0/194169214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605176ff0 msgr2=0x55d605179480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.657+0000 7fd6fe090640 1 --2- 172.21.15.70:0/194169214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605176ff0 0x55d605179480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.657+0000 7fd6fe090640 1 -- 172.21.15.70:0/194169214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171490 msgr2=0x55d6051718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.657+0000 7fd6fe090640 1 --2- 172.21.15.70:0/194169214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171490 0x55d6051718b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.657+0000 7fd6fe090640 1 -- 172.21.15.70:0/194169214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d60509e3d0 con 0x55d605171e80 2024-04-01T01:34:49.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.657+0000 7fd6fe891640 1 --2- 172.21.15.70:0/194169214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171490 0x55d6051718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:49.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.658+0000 7fd6fd88f640 1 --2- 172.21.15.70:0/194169214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605176ff0 0x55d605179480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:49.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.658+0000 7fd6fe090640 1 --2- 172.21.15.70:0/194169214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605171e80 0x55d605176980 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd6f40cc2f0 tx=0x7fd6f40d2c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=676c2806bdc097b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.658+0000 7fd6ef7fe640 1 --2- 172.21.15.70:0/4236544238 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd6e40a25a0 0x7fd6e40a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.658+0000 7fd6ecbfe640 1 -- 172.21.15.70:0/194169214 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f40de070 con 0x55d605171e80 2024-04-01T01:34:49.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.658+0000 7fd6ecbfe640 1 -- 172.21.15.70:0/194169214 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6f409ba10 con 0x55d605171e80 2024-04-01T01:34:49.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.658+0000 7fd6ecbfe640 1 -- 172.21.15.70:0/194169214 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f40d3040 con 0x55d605171e80 2024-04-01T01:34:49.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.658+0000 7fd6fe891640 1 --2- 172.21.15.70:0/4236544238 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd6e40a25a0 0x7fd6e40a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.658+0000 7fd6ef7fe640 1 -- 172.21.15.70:0/4236544238 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd6f40cec80 con 0x7fd6f80ffa50 2024-04-01T01:34:49.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.659+0000 7fd702fb4740 1 -- 172.21.15.70:0/194169214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605171e80 msgr2=0x55d605176980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.659+0000 7fd702fb4740 1 --2- 172.21.15.70:0/194169214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605171e80 0x55d605176980 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd6f40cc2f0 tx=0x7fd6f40d2c30 comp rx=0 tx=0).stop 2024-04-01T01:34:49.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.659+0000 7fd6fe891640 1 --2- 172.21.15.70:0/4236544238 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd6e40a25a0 0x7fd6e40a4a60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x55d6051726e0 tx=0x7fd6f0009280 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.660+0000 7fd702fb4740 1 -- 172.21.15.70:0/194169214 shutdown_connections 2024-04-01T01:34:49.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.660+0000 7fd702fb4740 1 --2- 172.21.15.70:0/194169214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605176ff0 0x55d605179480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.660+0000 7fd702fb4740 1 --2- 172.21.15.70:0/194169214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605171e80 0x55d605176980 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.660+0000 7fd702fb4740 1 --2- 172.21.15.70:0/194169214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171490 0x55d6051718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.660+0000 7fd702fb4740 1 -- 172.21.15.70:0/194169214 >> 172.21.15.70:0/194169214 conn(0x55d60516ca00 msgr2=0x55d60516ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.660+0000 7fd702fb4740 1 -- 172.21.15.70:0/194169214 shutdown_connections 2024-04-01T01:34:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.660+0000 7fd702fb4740 1 -- 172.21.15.70:0/194169214 wait complete. 2024-04-01T01:34:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.660+0000 7fd702fb4740 1 -- start start 2024-04-01T01:34:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.660+0000 7fd702fb4740 1 Processor -- start 2024-04-01T01:34:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd702fb4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605171490 0x55d60517ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd702fb4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171e80 0x55d60517bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd702fb4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605176ff0 0x55d605182fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd702fb4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d605179fa0 con 0x55d605171e80 2024-04-01T01:34:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd702fb4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d605074470 con 0x55d605176ff0 2024-04-01T01:34:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd702fb4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d60517a120 con 0x55d605171490 2024-04-01T01:34:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd702fb4740 10 client.0 _ll_register_callbacks cb 0x55d605091cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd6fe891640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605176ff0 0x55d605182fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd6fe891640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605176ff0 0x55d605182fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46608/0 (socket says 172.21.15.70:46608) 2024-04-01T01:34:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd6fe891640 1 -- 172.21.15.70:0/3611975524 learned_addr learned my addr 172.21.15.70:0/3611975524 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd6fd88f640 1 --2- 172.21.15.70:0/3611975524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171e80 0x55d60517bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd6fe090640 1 --2- 172.21.15.70:0/3611975524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605171490 0x55d60517ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd6fe891640 1 -- 172.21.15.70:0/3611975524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605171490 msgr2=0x55d60517ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd6fe891640 1 --2- 172.21.15.70:0/3611975524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605171490 0x55d60517ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd6fe891640 1 -- 172.21.15.70:0/3611975524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171e80 msgr2=0x55d60517bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd6fe891640 1 --2- 172.21.15.70:0/3611975524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171e80 0x55d60517bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd6fe891640 1 -- 172.21.15.70:0/3611975524 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d604dea3f0 con 0x55d605176ff0 2024-04-01T01:34:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.661+0000 7fd6fd88f640 1 --2- 172.21.15.70:0/3611975524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171e80 0x55d60517bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:34:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.662+0000 7fd6e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:34:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.662+0000 7fd6fe891640 1 --2- 172.21.15.70:0/3611975524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605176ff0 0x55d605182fe0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0009b30 tx=0x7fd6f0005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be7fa16a6080f8e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.662+0000 7fd6e1ffb640 1 -- 172.21.15.70:0/3611975524 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0050070 con 0x55d605176ff0 2024-04-01T01:34:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.662+0000 7fd702fb4740 1 -- 172.21.15.70:0/3611975524 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6f409ba10 con 0x55d605176ff0 2024-04-01T01:34:49.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.663+0000 7fd6e1ffb640 1 -- 172.21.15.70:0/3611975524 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd6f0003fc0 con 0x55d605176ff0 2024-04-01T01:34:49.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.663+0000 7fd6e1ffb640 1 -- 172.21.15.70:0/3611975524 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd6f005c050 con 0x55d605176ff0 2024-04-01T01:34:49.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.663+0000 7fd702fb4740 10 client.25829 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:49.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.663+0000 7fd702fb4740 1 -- 172.21.15.70:0/3611975524 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d604eb47d0 con 0x55d605176ff0 2024-04-01T01:34:49.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.663+0000 7fd6e1ffb640 1 -- 172.21.15.70:0/3611975524 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6f409ba10 con 0x55d605176ff0 2024-04-01T01:34:49.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.663+0000 7fd6e1ffb640 1 -- 172.21.15.70:0/3611975524 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd6f0034bb0 con 0x55d605176ff0 2024-04-01T01:34:49.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.664+0000 7fd702fb4740 10 client.25829 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:49.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.664+0000 7fd702fb4740 10 client.25829 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:49.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.664+0000 7fd702fb4740 20 client.25829 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:49.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.664+0000 7fd702fb4740 1 --2- 172.21.15.70:0/3611975524 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d605187580 0x55d6051899c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.664+0000 7fd702fb4740 4 client.25829 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:49.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.664+0000 7fd702fb4740 1 -- 172.21.15.70:0/3611975524 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6050e36f0 con 0x55d605187580 2024-04-01T01:34:49.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.664+0000 7fd6fe090640 1 --2- 172.21.15.70:0/3611975524 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d605187580 0x55d6051899c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.665+0000 7fd6e1ffb640 10 client.25829 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:49.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.665+0000 7fd6fe090640 1 --2- 172.21.15.70:0/3611975524 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d605187580 0x55d6051899c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a972239c633a1bbd server_cookie=8af427151a956525 in_seq=0 out_seq=0 2024-04-01T01:34:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.666+0000 7fd6e1ffb640 1 -- 172.21.15.70:0/3611975524 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d6050e36f0 con 0x55d605187580 2024-04-01T01:34:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.666+0000 7fd6e1ffb640 10 client.25829 handle_command_reply: tid=0 2024-04-01T01:34:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.667+0000 7fd702fb4740 1 client.25829 shutdown 2024-04-01T01:34:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.667+0000 7fd702fb4740 1 -- 172.21.15.70:0/3611975524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605176ff0 msgr2=0x55d605182fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.667+0000 7fd702fb4740 1 --2- 172.21.15.70:0/3611975524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605176ff0 0x55d605182fe0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0009b30 tx=0x7fd6f0005fd0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.667+0000 7fd702fb4740 1 -- 172.21.15.70:0/3611975524 shutdown_connections 2024-04-01T01:34:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.667+0000 7fd702fb4740 1 --2- 172.21.15.70:0/3611975524 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d605187580 0x55d6051899c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.667+0000 7fd702fb4740 1 --2- 172.21.15.70:0/3611975524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605176ff0 0x55d605182fe0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.667+0000 7fd702fb4740 1 --2- 172.21.15.70:0/3611975524 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171e80 0x55d60517bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.667+0000 7fd702fb4740 1 --2- 172.21.15.70:0/3611975524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605171490 0x55d60517ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.667+0000 7fd702fb4740 1 -- 172.21.15.70:0/3611975524 >> 172.21.15.70:0/3611975524 conn(0x55d60516ca00 msgr2=0x55d60516bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.667+0000 7fd6e1ffb640 0 client.25829 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:49.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.667+0000 7fd702fb4740 1 -- 172.21.15.70:0/3611975524 shutdown_connections 2024-04-01T01:34:49.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.667+0000 7fd702fb4740 1 -- 172.21.15.70:0/3611975524 wait complete. 2024-04-01T01:34:49.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.667+0000 7fd702fb4740 20 client.25829 trim_cache size 0 max 16384 2024-04-01T01:34:49.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.692+0000 7fd702fb4740 1 Processor -- start 2024-04-01T01:34:49.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.692+0000 7fd702fb4740 1 -- start start 2024-04-01T01:34:49.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.693+0000 7fd702fb4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171490 0x55d60516bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.693+0000 7fd702fb4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605171e80 0x55d605182fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.693+0000 7fd702fb4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605176ff0 0x55d60516c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.693+0000 7fd702fb4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc00fcc0 con 0x55d605171490 2024-04-01T01:34:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.693+0000 7fd702fb4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc011800 con 0x55d605171e80 2024-04-01T01:34:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.693+0000 7fd702fb4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc010390 con 0x55d605176ff0 2024-04-01T01:34:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.693+0000 7fd6fe090640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171490 0x55d60516bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.693+0000 7fd6fe090640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171490 0x55d60516bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41778/0 (socket says 172.21.15.70:41778) 2024-04-01T01:34:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.693+0000 7fd6fe090640 1 -- 172.21.15.70:0/2707838220 learned_addr learned my addr 172.21.15.70:0/2707838220 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.693+0000 7fd6fd88f640 1 --2- 172.21.15.70:0/2707838220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605171e80 0x55d605182fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.693+0000 7fd6fe891640 1 --2- 172.21.15.70:0/2707838220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605176ff0 0x55d60516c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.693+0000 7fd6fe090640 1 -- 172.21.15.70:0/2707838220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605176ff0 msgr2=0x55d60516c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.693+0000 7fd6fe090640 1 --2- 172.21.15.70:0/2707838220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605176ff0 0x55d60516c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.693+0000 7fd6fe090640 1 -- 172.21.15.70:0/2707838220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605171e80 msgr2=0x55d605182fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.693+0000 7fd6fe090640 1 --2- 172.21.15.70:0/2707838220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605171e80 0x55d605182fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.693+0000 7fd6fe090640 1 -- 172.21.15.70:0/2707838220 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d60516c640 con 0x55d605171490 2024-04-01T01:34:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.694+0000 7fd6fe090640 1 --2- 172.21.15.70:0/2707838220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171490 0x55d60516bbc0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fd6f400c6a0 tx=0x7fd6f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc5f8d2118da62b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.694+0000 7fd6e3fff640 1 -- 172.21.15.70:0/2707838220 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f40de040 con 0x55d605171490 2024-04-01T01:34:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.694+0000 7fd6e3fff640 1 -- 172.21.15.70:0/2707838220 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6f409ba10 con 0x55d605171490 2024-04-01T01:34:49.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.694+0000 7fd6e3fff640 1 -- 172.21.15.70:0/2707838220 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f409b310 con 0x55d605171490 2024-04-01T01:34:49.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.695+0000 7fd702fb4740 1 -- 172.21.15.70:0/2707838220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171490 msgr2=0x55d60516bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.695+0000 7fd702fb4740 1 --2- 172.21.15.70:0/2707838220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171490 0x55d60516bbc0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fd6f400c6a0 tx=0x7fd6f4005880 comp rx=0 tx=0).stop 2024-04-01T01:34:49.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.695+0000 7fd702fb4740 1 -- 172.21.15.70:0/2707838220 shutdown_connections 2024-04-01T01:34:49.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.695+0000 7fd702fb4740 1 --2- 172.21.15.70:0/2707838220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605176ff0 0x55d60516c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.695+0000 7fd702fb4740 1 --2- 172.21.15.70:0/2707838220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605171e80 0x55d605182fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.695+0000 7fd702fb4740 1 --2- 172.21.15.70:0/2707838220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171490 0x55d60516bbc0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.695+0000 7fd702fb4740 1 -- 172.21.15.70:0/2707838220 >> 172.21.15.70:0/2707838220 conn(0x55d60516ca00 msgr2=0x55d60516b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.696+0000 7fd702fb4740 1 -- 172.21.15.70:0/2707838220 shutdown_connections 2024-04-01T01:34:49.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.696+0000 7fd702fb4740 1 -- 172.21.15.70:0/2707838220 wait complete. 2024-04-01T01:34:49.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.696+0000 7fd702fb4740 1 -- start start 2024-04-01T01:34:49.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.696+0000 7fd702fb4740 1 Processor -- start 2024-04-01T01:34:49.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.697+0000 7fd702fb4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605171490 0x55d605180cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.697+0000 7fd702fb4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171e80 0x55d605181200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.697+0000 7fd702fb4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605176ff0 0x55d605182210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.698+0000 7fd702fb4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc011130 con 0x55d605171e80 2024-04-01T01:34:49.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.698+0000 7fd702fb4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc010a60 con 0x55d605171490 2024-04-01T01:34:49.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.698+0000 7fd6fe090640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605171490 0x55d605180cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.698+0000 7fd702fb4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd6dc00dca0 con 0x55d605176ff0 2024-04-01T01:34:49.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.698+0000 7fd6fd88f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171e80 0x55d605181200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.698+0000 7fd6fe891640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605176ff0 0x55d605182210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.698+0000 7fd6fe090640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605171490 0x55d605180cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46622/0 (socket says 172.21.15.70:46622) 2024-04-01T01:34:49.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.698+0000 7fd6fe090640 1 -- 172.21.15.70:0/780538853 learned_addr learned my addr 172.21.15.70:0/780538853 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.698+0000 7fd702fb4740 10 client.0 _ll_register_callbacks cb 0x7fd6f0033e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.698+0000 7fd6fe090640 1 -- 172.21.15.70:0/780538853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605176ff0 msgr2=0x55d605182210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.698+0000 7fd6fe090640 1 --2- 172.21.15.70:0/780538853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605176ff0 0x55d605182210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.698+0000 7fd6fe090640 1 -- 172.21.15.70:0/780538853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171e80 msgr2=0x55d605181200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.698+0000 7fd6fe090640 1 --2- 172.21.15.70:0/780538853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171e80 0x55d605181200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.698+0000 7fd6fe090640 1 -- 172.21.15.70:0/780538853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d605182800 con 0x55d605171490 2024-04-01T01:34:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.698+0000 7fd6fe090640 1 --2- 172.21.15.70:0/780538853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605171490 0x55d605180cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd6f409b010 tx=0x7fd6f409b1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe00c312d00ef32c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.698+0000 7fd6e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:34:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.699+0000 7fd6e1ffb640 1 -- 172.21.15.70:0/780538853 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f40de070 con 0x55d605171490 2024-04-01T01:34:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.699+0000 7fd6e1ffb640 1 -- 172.21.15.70:0/780538853 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd6f40cc460 con 0x55d605171490 2024-04-01T01:34:49.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.699+0000 7fd702fb4740 1 -- 172.21.15.70:0/780538853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6f409ba10 con 0x55d605171490 2024-04-01T01:34:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.699+0000 7fd6e1ffb640 1 -- 172.21.15.70:0/780538853 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd6f409ac70 con 0x55d605171490 2024-04-01T01:34:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.699+0000 7fd702fb4740 10 client.25835 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.699+0000 7fd702fb4740 1 -- 172.21.15.70:0/780538853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d6051eddd0 con 0x55d605171490 2024-04-01T01:34:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.699+0000 7fd6e1ffb640 1 -- 172.21.15.70:0/780538853 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6f409f460 con 0x55d605171490 2024-04-01T01:34:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.699+0000 7fd6e1ffb640 1 -- 172.21.15.70:0/780538853 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd6f40d9040 con 0x55d605171490 2024-04-01T01:34:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.699+0000 7fd702fb4740 10 client.25835 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:49.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.700+0000 7fd702fb4740 10 client.25835 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.700+0000 7fd702fb4740 20 client.25835 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.700+0000 7fd702fb4740 1 --2- 172.21.15.70:0/780538853 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d6051ee320 0x55d6051f0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.700+0000 7fd702fb4740 4 client.25835 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:34:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.700+0000 7fd702fb4740 1 -- 172.21.15.70:0/780538853 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fd6f409ba10 con 0x55d6051ee320 2024-04-01T01:34:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.700+0000 7fd6fd88f640 1 --2- 172.21.15.70:0/780538853 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d6051ee320 0x55d6051f0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.700+0000 7fd6e1ffb640 10 client.25835 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.701+0000 7fd6fd88f640 1 --2- 172.21.15.70:0/780538853 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d6051ee320 0x55d6051f0710 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b95925aeb2103ce6 server_cookie=debb9bc462f40942 in_seq=0 out_seq=0 2024-04-01T01:34:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.701+0000 7fd6e1ffb640 1 -- 172.21.15.70:0/780538853 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fd6f409ba10 con 0x55d6051ee320 2024-04-01T01:34:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.701+0000 7fd6e1ffb640 10 client.25835 handle_command_reply: tid=0 2024-04-01T01:34:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.701+0000 7fd702fb4740 1 client.25835 shutdown 2024-04-01T01:34:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.701+0000 7fd702fb4740 1 -- 172.21.15.70:0/780538853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605171490 msgr2=0x55d605180cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.701+0000 7fd702fb4740 1 --2- 172.21.15.70:0/780538853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605171490 0x55d605180cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd6f409b010 tx=0x7fd6f409b1a0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.701+0000 7fd702fb4740 1 -- 172.21.15.70:0/780538853 shutdown_connections 2024-04-01T01:34:49.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.701+0000 7fd702fb4740 1 --2- 172.21.15.70:0/780538853 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d6051ee320 0x55d6051f0710 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.701+0000 7fd702fb4740 1 --2- 172.21.15.70:0/780538853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d605176ff0 0x55d605182210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.701+0000 7fd702fb4740 1 --2- 172.21.15.70:0/780538853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d605171e80 0x55d605181200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.701+0000 7fd702fb4740 1 --2- 172.21.15.70:0/780538853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d605171490 0x55d605180cc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.701+0000 7fd6e1ffb640 0 client.25835 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:49.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.701+0000 7fd702fb4740 1 -- 172.21.15.70:0/780538853 >> 172.21.15.70:0/780538853 conn(0x55d60516ca00 msgr2=0x55d60516bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.701+0000 7fd702fb4740 1 -- 172.21.15.70:0/780538853 shutdown_connections 2024-04-01T01:34:49.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.701+0000 7fd702fb4740 1 -- 172.21.15.70:0/780538853 wait complete. 2024-04-01T01:34:49.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.702+0000 7fd702fb4740 20 client.25835 trim_cache size 0 max 16384 2024-04-01T01:34:49.704 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:34:49.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.703+0000 7fd7002f4640 1 -- 172.21.15.70:0/4236544238 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd6e40a25a0 msgr2=0x7fd6e40a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.703+0000 7fd7002f4640 1 --2- 172.21.15.70:0/4236544238 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd6e40a25a0 0x7fd6e40a4a60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x55d6051726e0 tx=0x7fd6f0009280 comp rx=0 tx=0).stop 2024-04-01T01:34:49.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.703+0000 7fd7002f4640 1 -- 172.21.15.70:0/4236544238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f80ffa50 msgr2=0x7fd6f810ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:49.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.703+0000 7fd7002f4640 1 --2- 172.21.15.70:0/4236544238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f80ffa50 0x7fd6f810ee00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd6f400e9c0 tx=0x7fd6f400ee80 comp rx=0 tx=0).stop 2024-04-01T01:34:49.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.703+0000 7fd7002f4640 1 -- 172.21.15.70:0/4236544238 shutdown_connections 2024-04-01T01:34:49.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.703+0000 7fd7002f4640 1 --2- 172.21.15.70:0/4236544238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6f8101170 0x7fd6f8110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.703+0000 7fd7002f4640 1 --2- 172.21.15.70:0/4236544238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6f81007b0 0x7fd6f810f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.703+0000 7fd7002f4640 1 --2- 172.21.15.70:0/4236544238 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd6e40a25a0 0x7fd6e40a4a60 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.703+0000 7fd7002f4640 1 --2- 172.21.15.70:0/4236544238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f80ffa50 0x7fd6f810ee00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:49.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.703+0000 7fd7002f4640 1 -- 172.21.15.70:0/4236544238 >> 172.21.15.70:0/4236544238 conn(0x7fd6f80fb380 msgr2=0x7fd6f81039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:49.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.703+0000 7fd7002f4640 1 -- 172.21.15.70:0/4236544238 shutdown_connections 2024-04-01T01:34:49.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:49.704+0000 7fd7002f4640 1 -- 172.21.15.70:0/4236544238 wait complete. 2024-04-01T01:34:49.722 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:49 smithi070 ceph-mon[32179]: pgmap v398: 129 pgs: 129 active+clean; 946 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 54 MiB/s wr, 53 op/s 2024-04-01T01:34:49.722 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:49 smithi070 ceph-mon[32179]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-01T01:34:49.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:49 smithi189 ceph-mon[35753]: pgmap v398: 129 pgs: 129 active+clean; 946 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 54 MiB/s wr, 53 op/s 2024-04-01T01:34:49.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:49 smithi189 ceph-mon[35753]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-01T01:34:50.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:49 smithi081 ceph-mon[35142]: pgmap v398: 129 pgs: 129 active+clean; 946 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 54 MiB/s wr, 53 op/s 2024-04-01T01:34:50.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:49 smithi081 ceph-mon[35142]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-01T01:34:50.724 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:34:50.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:34:50.724 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:34:50.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.894+0000 7f901eaab640 1 -- 172.21.15.70:0/614705070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9010096830 msgr2=0x7f901009d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:50.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.894+0000 7f901eaab640 1 --2- 172.21.15.70:0/614705070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9010096830 0x7f901009d0c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9018067420 tx=0x7f90180977c0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.894+0000 7f901eaab640 1 -- 172.21.15.70:0/614705070 shutdown_connections 2024-04-01T01:34:50.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.894+0000 7f901eaab640 1 --2- 172.21.15.70:0/614705070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9010096830 0x7f901009d0c0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.894+0000 7f901eaab640 1 --2- 172.21.15.70:0/614705070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9010095e70 0x7f90100962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.894+0000 7f901eaab640 1 --2- 172.21.15.70:0/614705070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010094c80 0x7f90100950a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.894+0000 7f901eaab640 1 -- 172.21.15.70:0/614705070 >> 172.21.15.70:0/614705070 conn(0x7f90100901d0 msgr2=0x7f9010092630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:50.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.894+0000 7f901eaab640 1 -- 172.21.15.70:0/614705070 shutdown_connections 2024-04-01T01:34:50.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.894+0000 7f901eaab640 1 -- 172.21.15.70:0/614705070 wait complete. 2024-04-01T01:34:50.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.895+0000 7f901eaab640 1 Processor -- start 2024-04-01T01:34:50.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.895+0000 7f901eaab640 1 -- start start 2024-04-01T01:34:50.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.895+0000 7f901eaab640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010094c80 0x7f901013bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.895+0000 7f901eaab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9010095e70 0x7f901013c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.896+0000 7f901eaab640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9010096830 0x7f9010143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.896+0000 7f901eaab640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f90100a0660 con 0x7f9010095e70 2024-04-01T01:34:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.896+0000 7f901eaab640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f90100a04e0 con 0x7f9010094c80 2024-04-01T01:34:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.896+0000 7f901daa9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010094c80 0x7f901013bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.896+0000 7f901daa9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010094c80 0x7f901013bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46640/0 (socket says 172.21.15.70:46640) 2024-04-01T01:34:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.896+0000 7f901daa9640 1 -- 172.21.15.70:0/4115045184 learned_addr learned my addr 172.21.15.70:0/4115045184 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.896+0000 7f901eaab640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f90100a07e0 con 0x7f9010096830 2024-04-01T01:34:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.896+0000 7f901e2aa640 1 --2- 172.21.15.70:0/4115045184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9010096830 0x7f9010143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.896+0000 7f901d2a8640 1 --2- 172.21.15.70:0/4115045184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9010095e70 0x7f901013c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.896+0000 7f901daa9640 1 -- 172.21.15.70:0/4115045184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9010096830 msgr2=0x7f9010143110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.896+0000 7f901daa9640 1 --2- 172.21.15.70:0/4115045184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9010096830 0x7f9010143110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.896+0000 7f901daa9640 1 -- 172.21.15.70:0/4115045184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9010095e70 msgr2=0x7f901013c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.896+0000 7f901daa9640 1 --2- 172.21.15.70:0/4115045184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9010095e70 0x7f901013c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.896+0000 7f901daa9640 1 -- 172.21.15.70:0/4115045184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9010004df0 con 0x7f9010094c80 2024-04-01T01:34:50.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.896+0000 7f901daa9640 1 --2- 172.21.15.70:0/4115045184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010094c80 0x7f901013bb60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f901400e970 tx=0x7f901400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:50.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.897+0000 7f900effd640 1 -- 172.21.15.70:0/4115045184 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901400ccc0 con 0x7f9010094c80 2024-04-01T01:34:50.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.897+0000 7f901eaab640 1 -- 172.21.15.70:0/4115045184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90100050e0 con 0x7f9010094c80 2024-04-01T01:34:50.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.897+0000 7f901eaab640 1 -- 172.21.15.70:0/4115045184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9010005680 con 0x7f9010094c80 2024-04-01T01:34:50.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.897+0000 7f900effd640 1 -- 172.21.15.70:0/4115045184 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90140035f0 con 0x7f9010094c80 2024-04-01T01:34:50.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.897+0000 7f900effd640 1 -- 172.21.15.70:0/4115045184 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901400ce60 con 0x7f9010094c80 2024-04-01T01:34:50.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.900+0000 7f900effd640 1 -- 172.21.15.70:0/4115045184 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f9014010750 con 0x7f9010094c80 2024-04-01T01:34:50.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.900+0000 7f90229ca740 1 Processor -- start 2024-04-01T01:34:50.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.901+0000 7f90229ca740 1 -- start start 2024-04-01T01:34:50.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.901+0000 7f90229ca740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 0x56083b5f4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.901+0000 7f90229ca740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f45e0 0x56083b5f90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.901+0000 7f90229ca740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f9750 0x56083b5fbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.901+0000 7f90229ca740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56083b4f50b0 con 0x56083b5f45e0 2024-04-01T01:34:50.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.901+0000 7f90229ca740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56083b557ea0 con 0x56083b5f9750 2024-04-01T01:34:50.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.901+0000 7f90229ca740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56083b500c00 con 0x56083b5f3bf0 2024-04-01T01:34:50.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.901+0000 7f901e2aa640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 0x56083b5f4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.901+0000 7f901d2a8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f9750 0x56083b5fbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.902+0000 7f901d2a8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f9750 0x56083b5fbbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46652/0 (socket says 172.21.15.70:46652) 2024-04-01T01:34:50.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.902+0000 7f901d2a8640 1 -- 172.21.15.70:0/2464486385 learned_addr learned my addr 172.21.15.70:0/2464486385 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:50.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.902+0000 7f901daa9640 1 --2- 172.21.15.70:0/2464486385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f45e0 0x56083b5f90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.902+0000 7f901d2a8640 1 -- 172.21.15.70:0/2464486385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 msgr2=0x56083b5f4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:50.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.902+0000 7f901d2a8640 1 --2- 172.21.15.70:0/2464486385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 0x56083b5f4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.902+0000 7f901d2a8640 1 -- 172.21.15.70:0/2464486385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f45e0 msgr2=0x56083b5f90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:50.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.902+0000 7f901d2a8640 1 --2- 172.21.15.70:0/2464486385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f45e0 0x56083b5f90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.902+0000 7f901d2a8640 1 -- 172.21.15.70:0/2464486385 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56083b3347e0 con 0x56083b5f9750 2024-04-01T01:34:50.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.902+0000 7f901e2aa640 1 --2- 172.21.15.70:0/2464486385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 0x56083b5f4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:50.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.902+0000 7f900effd640 1 --2- 172.21.15.70:0/4115045184 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f90040a25f0 0x7f90040a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.902+0000 7f901d2a8640 1 --2- 172.21.15.70:0/2464486385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f9750 0x56083b5fbbe0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9008009c90 tx=0x7f9008011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec473f4041f9ccff server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:50.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.902+0000 7f900effd640 1 -- 172.21.15.70:0/4115045184 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f90140cec80 con 0x7f9010094c80 2024-04-01T01:34:50.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.902+0000 7f9003fff640 1 -- 172.21.15.70:0/2464486385 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f900801e070 con 0x56083b5f9750 2024-04-01T01:34:50.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.903+0000 7f9003fff640 1 -- 172.21.15.70:0/2464486385 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9008019ae0 con 0x56083b5f9750 2024-04-01T01:34:50.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.903+0000 7f9003fff640 1 -- 172.21.15.70:0/2464486385 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f900800ea40 con 0x56083b5f9750 2024-04-01T01:34:50.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.903+0000 7f901e2aa640 1 --2- 172.21.15.70:0/4115045184 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f90040a25f0 0x7f90040a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.904+0000 7f90229ca740 1 -- 172.21.15.70:0/2464486385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f9750 msgr2=0x56083b5fbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:50.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.904+0000 7f90229ca740 1 --2- 172.21.15.70:0/2464486385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f9750 0x56083b5fbbe0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9008009c90 tx=0x7f9008011810 comp rx=0 tx=0).stop 2024-04-01T01:34:50.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.904+0000 7f90229ca740 1 -- 172.21.15.70:0/2464486385 shutdown_connections 2024-04-01T01:34:50.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.904+0000 7f90229ca740 1 --2- 172.21.15.70:0/2464486385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f9750 0x56083b5fbbe0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.904+0000 7f90229ca740 1 --2- 172.21.15.70:0/2464486385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f45e0 0x56083b5f90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.904+0000 7f90229ca740 1 --2- 172.21.15.70:0/2464486385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 0x56083b5f4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.904+0000 7f90229ca740 1 -- 172.21.15.70:0/2464486385 >> 172.21.15.70:0/2464486385 conn(0x56083b5ef160 msgr2=0x56083b5ef560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:50.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.904+0000 7f901e2aa640 1 --2- 172.21.15.70:0/4115045184 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f90040a25f0 0x7f90040a4ab0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9018064c40 tx=0x7f901809ef70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:50.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.904+0000 7f90229ca740 1 -- 172.21.15.70:0/2464486385 shutdown_connections 2024-04-01T01:34:50.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.904+0000 7f90229ca740 1 -- 172.21.15.70:0/2464486385 wait complete. 2024-04-01T01:34:50.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.905+0000 7f90229ca740 1 -- start start 2024-04-01T01:34:50.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.905+0000 7f90229ca740 1 Processor -- start 2024-04-01T01:34:50.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.905+0000 7f90229ca740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f3bf0 0x56083b5fdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.905+0000 7f90229ca740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f45e0 0x56083b5fe530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f90229ca740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f9750 0x56083b605570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f90229ca740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56083b4f63d0 con 0x56083b5f45e0 2024-04-01T01:34:50.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f90229ca740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56083b553f40 con 0x56083b5f3bf0 2024-04-01T01:34:50.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f90229ca740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56083b5fc7a0 con 0x56083b5f9750 2024-04-01T01:34:50.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f90229ca740 10 client.0 _ll_register_callbacks cb 0x56083b513b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:50.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f901daa9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f3bf0 0x56083b5fdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f901daa9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f3bf0 0x56083b5fdff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46654/0 (socket says 172.21.15.70:46654) 2024-04-01T01:34:50.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f901daa9640 1 -- 172.21.15.70:0/1955969357 learned_addr learned my addr 172.21.15.70:0/1955969357 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:50.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f901e2aa640 1 --2- 172.21.15.70:0/1955969357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f9750 0x56083b605570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f901daa9640 1 -- 172.21.15.70:0/1955969357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f9750 msgr2=0x56083b605570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:50.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f901daa9640 1 --2- 172.21.15.70:0/1955969357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f9750 0x56083b605570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f901daa9640 1 -- 172.21.15.70:0/1955969357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f45e0 msgr2=0x56083b5fe530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:50.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f901daa9640 1 --2- 172.21.15.70:0/1955969357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f45e0 0x56083b5fe530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f901daa9640 1 -- 172.21.15.70:0/1955969357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56083b394f70 con 0x56083b5f3bf0 2024-04-01T01:34:50.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f901e2aa640 1 --2- 172.21.15.70:0/1955969357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f9750 0x56083b605570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:34:50.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f901daa9640 1 --2- 172.21.15.70:0/1955969357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f3bf0 0x56083b5fdff0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f90140a37f0 tx=0x7f9014096f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6fa39ea4ffb3ab3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:50.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f90017fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:34:50.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.906+0000 7f90017fa640 1 -- 172.21.15.70:0/1955969357 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90140d3040 con 0x56083b5f3bf0 2024-04-01T01:34:50.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.907+0000 7f90017fa640 1 -- 172.21.15.70:0/1955969357 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f901409f120 con 0x56083b5f3bf0 2024-04-01T01:34:50.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.907+0000 7f90229ca740 1 -- 172.21.15.70:0/1955969357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9008019ae0 con 0x56083b5f3bf0 2024-04-01T01:34:50.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.907+0000 7f90017fa640 1 -- 172.21.15.70:0/1955969357 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f901409ac70 con 0x56083b5f3bf0 2024-04-01T01:34:50.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.907+0000 7f90229ca740 10 client.25847 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:50.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.907+0000 7f90229ca740 1 -- 172.21.15.70:0/1955969357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56083b22c170 con 0x56083b5f3bf0 2024-04-01T01:34:50.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.907+0000 7f90017fa640 1 -- 172.21.15.70:0/1955969357 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9008019ae0 con 0x56083b5f3bf0 2024-04-01T01:34:50.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.907+0000 7f90017fa640 1 -- 172.21.15.70:0/1955969357 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9014097980 con 0x56083b5f3bf0 2024-04-01T01:34:50.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.907+0000 7f90229ca740 10 client.25847 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:50.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.907+0000 7f90229ca740 10 client.25847 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:50.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.907+0000 7f90229ca740 20 client.25847 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:50.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.907+0000 7f90229ca740 1 --2- 172.21.15.70:0/1955969357 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56083b609ac0 0x56083b60bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.907+0000 7f90229ca740 4 client.25847 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:50.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.907+0000 7f90229ca740 1 -- 172.21.15.70:0/1955969357 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56083b5653e0 con 0x56083b609ac0 2024-04-01T01:34:50.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.908+0000 7f901d2a8640 1 --2- 172.21.15.70:0/1955969357 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56083b609ac0 0x56083b60bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.908+0000 7f901d2a8640 1 --2- 172.21.15.70:0/1955969357 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56083b609ac0 0x56083b60bf80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e41e504473d9fd6e server_cookie=af09419ed54415b2 in_seq=0 out_seq=0 2024-04-01T01:34:50.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.908+0000 7f90017fa640 10 client.25847 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:50.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.909+0000 7f90017fa640 1 -- 172.21.15.70:0/1955969357 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56083b5653e0 con 0x56083b609ac0 2024-04-01T01:34:50.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.909+0000 7f90017fa640 10 client.25847 handle_command_reply: tid=0 2024-04-01T01:34:50.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.909+0000 7f90229ca740 1 client.25847 shutdown 2024-04-01T01:34:50.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.910+0000 7f90229ca740 1 -- 172.21.15.70:0/1955969357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f3bf0 msgr2=0x56083b5fdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:50.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.910+0000 7f90229ca740 1 --2- 172.21.15.70:0/1955969357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f3bf0 0x56083b5fdff0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f90140a37f0 tx=0x7f9014096f50 comp rx=0 tx=0).stop 2024-04-01T01:34:50.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.910+0000 7f90229ca740 1 -- 172.21.15.70:0/1955969357 shutdown_connections 2024-04-01T01:34:50.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.910+0000 7f90229ca740 1 --2- 172.21.15.70:0/1955969357 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56083b609ac0 0x56083b60bf80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.910+0000 7f90229ca740 1 --2- 172.21.15.70:0/1955969357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f9750 0x56083b605570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.910+0000 7f90229ca740 1 --2- 172.21.15.70:0/1955969357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f45e0 0x56083b5fe530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.910+0000 7f90229ca740 1 --2- 172.21.15.70:0/1955969357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f3bf0 0x56083b5fdff0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.910+0000 7f90229ca740 1 -- 172.21.15.70:0/1955969357 >> 172.21.15.70:0/1955969357 conn(0x56083b5ef160 msgr2=0x56083b5ee2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:50.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.910+0000 7f90017fa640 0 client.25847 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:50.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.910+0000 7f90229ca740 1 -- 172.21.15.70:0/1955969357 shutdown_connections 2024-04-01T01:34:50.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.911+0000 7f90229ca740 1 -- 172.21.15.70:0/1955969357 wait complete. 2024-04-01T01:34:50.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.911+0000 7f90229ca740 20 client.25847 trim_cache size 0 max 16384 2024-04-01T01:34:50.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.945+0000 7f90229ca740 1 Processor -- start 2024-04-01T01:34:50.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.945+0000 7f90229ca740 1 -- start start 2024-04-01T01:34:50.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.945+0000 7f90229ca740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 0x56083b5fcba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.945+0000 7f90229ca740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f45e0 0x56083b605530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.945+0000 7f90229ca740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f9750 0x56083b5fd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.946+0000 7f90229ca740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc00fc80 con 0x56083b5f9750 2024-04-01T01:34:50.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.946+0000 7f90229ca740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc0117c0 con 0x56083b5f45e0 2024-04-01T01:34:50.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.946+0000 7f90229ca740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc010350 con 0x56083b5f3bf0 2024-04-01T01:34:50.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.946+0000 7f901e2aa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f9750 0x56083b5fd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.946+0000 7f901e2aa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f9750 0x56083b5fd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41840/0 (socket says 172.21.15.70:41840) 2024-04-01T01:34:50.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.946+0000 7f901e2aa640 1 -- 172.21.15.70:0/3481333986 learned_addr learned my addr 172.21.15.70:0/3481333986 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:50.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.946+0000 7f901d2a8640 1 --2- 172.21.15.70:0/3481333986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f45e0 0x56083b605530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.946+0000 7f901daa9640 1 --2- 172.21.15.70:0/3481333986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 0x56083b5fcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.946+0000 7f901e2aa640 1 -- 172.21.15.70:0/3481333986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 msgr2=0x56083b5fcba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:50.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.946+0000 7f901e2aa640 1 --2- 172.21.15.70:0/3481333986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 0x56083b5fcba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.946+0000 7f901e2aa640 1 -- 172.21.15.70:0/3481333986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f45e0 msgr2=0x56083b605530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:50.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.946+0000 7f901e2aa640 1 --2- 172.21.15.70:0/3481333986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f45e0 0x56083b605530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.946+0000 7f901e2aa640 1 -- 172.21.15.70:0/3481333986 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56083b5e9440 con 0x56083b5f9750 2024-04-01T01:34:50.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.946+0000 7f901d2a8640 1 --2- 172.21.15.70:0/3481333986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f45e0 0x56083b605530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:50.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.947+0000 7f901e2aa640 1 --2- 172.21.15.70:0/3481333986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f9750 0x56083b5fd0e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9018050260 tx=0x7f90180ab000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=446dc122f19009e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:50.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.947+0000 7f90037fe640 1 -- 172.21.15.70:0/3481333986 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901809a040 con 0x56083b5f9750 2024-04-01T01:34:50.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.947+0000 7f90037fe640 1 -- 172.21.15.70:0/3481333986 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9018069830 con 0x56083b5f9750 2024-04-01T01:34:50.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.947+0000 7f90037fe640 1 -- 172.21.15.70:0/3481333986 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90180b23f0 con 0x56083b5f9750 2024-04-01T01:34:50.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.948+0000 7f90229ca740 1 -- 172.21.15.70:0/3481333986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f9750 msgr2=0x56083b5fd0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:50.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.948+0000 7f90229ca740 1 --2- 172.21.15.70:0/3481333986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f9750 0x56083b5fd0e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9018050260 tx=0x7f90180ab000 comp rx=0 tx=0).stop 2024-04-01T01:34:50.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.949+0000 7f90229ca740 1 -- 172.21.15.70:0/3481333986 shutdown_connections 2024-04-01T01:34:50.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.949+0000 7f90229ca740 1 --2- 172.21.15.70:0/3481333986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f9750 0x56083b5fd0e0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.949+0000 7f90229ca740 1 --2- 172.21.15.70:0/3481333986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f45e0 0x56083b605530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.949+0000 7f90229ca740 1 --2- 172.21.15.70:0/3481333986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 0x56083b5fcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.949+0000 7f90229ca740 1 -- 172.21.15.70:0/3481333986 >> 172.21.15.70:0/3481333986 conn(0x56083b5ef160 msgr2=0x56083b5ee2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:50.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.949+0000 7f90229ca740 1 -- 172.21.15.70:0/3481333986 shutdown_connections 2024-04-01T01:34:50.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.949+0000 7f90229ca740 1 -- 172.21.15.70:0/3481333986 wait complete. 2024-04-01T01:34:50.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.950+0000 7f90229ca740 1 -- start start 2024-04-01T01:34:50.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.950+0000 7f90229ca740 1 Processor -- start 2024-04-01T01:34:50.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.951+0000 7f90229ca740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 0x56083b602d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.951+0000 7f90229ca740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f45e0 0x56083b603250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.951+0000 7f901daa9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 0x56083b602d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.951+0000 7f901daa9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 0x56083b602d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39328/0 (socket says 172.21.15.70:39328) 2024-04-01T01:34:50.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.952+0000 7f901d2a8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f45e0 0x56083b603250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.952+0000 7f901d2a8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f45e0 0x56083b603250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41850/0 (socket says 172.21.15.70:41850) 2024-04-01T01:34:50.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.952+0000 7f90229ca740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f9750 0x56083b604170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.952+0000 7f901d2a8640 1 -- 172.21.15.70:0/3921239574 learned_addr learned my addr 172.21.15.70:0/3921239574 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:50.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.952+0000 7f901e2aa640 1 --2- 172.21.15.70:0/3921239574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f9750 0x56083b604170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.952+0000 7f90229ca740 1 -- 172.21.15.70:0/3921239574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc0110f0 con 0x56083b5f45e0 2024-04-01T01:34:50.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.952+0000 7f90229ca740 1 -- 172.21.15.70:0/3921239574 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc010a20 con 0x56083b5f9750 2024-04-01T01:34:50.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.952+0000 7f90229ca740 1 -- 172.21.15.70:0/3921239574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8ffc00dc60 con 0x56083b5f3bf0 2024-04-01T01:34:50.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.952+0000 7f90229ca740 10 client.0 _ll_register_callbacks cb 0x7f90140d2330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:50.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.953+0000 7f901e2aa640 1 -- 172.21.15.70:0/3921239574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 msgr2=0x56083b602d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:50.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.953+0000 7f901e2aa640 1 --2- 172.21.15.70:0/3921239574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 0x56083b602d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.953+0000 7f901e2aa640 1 -- 172.21.15.70:0/3921239574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f45e0 msgr2=0x56083b603250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:50.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.953+0000 7f901e2aa640 1 --2- 172.21.15.70:0/3921239574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f45e0 0x56083b603250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.953+0000 7f901e2aa640 1 -- 172.21.15.70:0/3921239574 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56083b604760 con 0x56083b5f9750 2024-04-01T01:34:50.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.953+0000 7f901daa9640 1 --2- 172.21.15.70:0/3921239574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 0x56083b602d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:50.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.953+0000 7f901e2aa640 1 --2- 172.21.15.70:0/3921239574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f9750 0x56083b604170 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9018064af0 tx=0x7f901809f820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e12133e5114e8d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:50.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.953+0000 7f90017fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:34:50.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.953+0000 7f90017fa640 1 -- 172.21.15.70:0/3921239574 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90180c5070 con 0x56083b5f9750 2024-04-01T01:34:50.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.953+0000 7f90229ca740 1 -- 172.21.15.70:0/3921239574 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9018069830 con 0x56083b5f9750 2024-04-01T01:34:50.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.953+0000 7f90017fa640 1 -- 172.21.15.70:0/3921239574 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f90180c1870 con 0x56083b5f9750 2024-04-01T01:34:50.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.953+0000 7f90017fa640 1 -- 172.21.15.70:0/3921239574 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f901809a040 con 0x56083b5f9750 2024-04-01T01:34:50.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.954+0000 7f90229ca740 10 client.25853 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:50.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.954+0000 7f90229ca740 1 -- 172.21.15.70:0/3921239574 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56083b66ff20 con 0x56083b5f9750 2024-04-01T01:34:50.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.954+0000 7f90017fa640 1 -- 172.21.15.70:0/3921239574 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9018069830 con 0x56083b5f9750 2024-04-01T01:34:50.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.954+0000 7f90017fa640 1 -- 172.21.15.70:0/3921239574 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f90180bf930 con 0x56083b5f9750 2024-04-01T01:34:50.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.954+0000 7f90229ca740 10 client.25853 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:50.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.954+0000 7f90229ca740 10 client.25853 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:50.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.954+0000 7f90229ca740 20 client.25853 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:50.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.954+0000 7f90229ca740 1 --2- 172.21.15.70:0/3921239574 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56083b6707d0 0x56083b672bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:50.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.954+0000 7f90229ca740 4 client.25853 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:34:50.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.954+0000 7f90229ca740 1 -- 172.21.15.70:0/3921239574 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56083b55f730 con 0x56083b6707d0 2024-04-01T01:34:50.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.954+0000 7f901daa9640 1 --2- 172.21.15.70:0/3921239574 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56083b6707d0 0x56083b672bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:50.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.955+0000 7f90017fa640 10 client.25853 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:50.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.955+0000 7f901daa9640 1 --2- 172.21.15.70:0/3921239574 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56083b6707d0 0x56083b672bf0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8903514a4d90b579 server_cookie=26b943fdff594c71 in_seq=0 out_seq=0 2024-04-01T01:34:50.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.957+0000 7f90017fa640 1 -- 172.21.15.70:0/3921239574 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56083b55f730 con 0x56083b6707d0 2024-04-01T01:34:50.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.957+0000 7f90017fa640 10 client.25853 handle_command_reply: tid=0 2024-04-01T01:34:50.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.957+0000 7f90229ca740 1 client.25853 shutdown 2024-04-01T01:34:50.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.957+0000 7f90229ca740 1 -- 172.21.15.70:0/3921239574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f9750 msgr2=0x56083b604170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:50.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.957+0000 7f90229ca740 1 --2- 172.21.15.70:0/3921239574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f9750 0x56083b604170 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9018064af0 tx=0x7f901809f820 comp rx=0 tx=0).stop 2024-04-01T01:34:50.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.958+0000 7f90229ca740 1 -- 172.21.15.70:0/3921239574 shutdown_connections 2024-04-01T01:34:50.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.958+0000 7f90229ca740 1 --2- 172.21.15.70:0/3921239574 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56083b6707d0 0x56083b672bf0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.958+0000 7f90229ca740 1 --2- 172.21.15.70:0/3921239574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56083b5f9750 0x56083b604170 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.958+0000 7f90229ca740 1 --2- 172.21.15.70:0/3921239574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56083b5f45e0 0x56083b603250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.958+0000 7f90229ca740 1 --2- 172.21.15.70:0/3921239574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56083b5f3bf0 0x56083b602d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.958+0000 7f90229ca740 1 -- 172.21.15.70:0/3921239574 >> 172.21.15.70:0/3921239574 conn(0x56083b5ef160 msgr2=0x56083b5f23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:50.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.958+0000 7f90017fa640 0 client.25853 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:50.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.958+0000 7f90229ca740 1 -- 172.21.15.70:0/3921239574 shutdown_connections 2024-04-01T01:34:50.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.958+0000 7f90229ca740 1 -- 172.21.15.70:0/3921239574 wait complete. 2024-04-01T01:34:50.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.958+0000 7f90229ca740 20 client.25853 trim_cache size 0 max 16384 2024-04-01T01:34:50.960 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:34:50.960 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:34:50.960 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "831920f1-e9d6-46aa-850b-e8b740c5b23b", 2024-04-01T01:34:50.960 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:34:50.960 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:34:50.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.959+0000 7f901eaab640 1 -- 172.21.15.70:0/4115045184 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f90040a25f0 msgr2=0x7f90040a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:50.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.959+0000 7f901eaab640 1 --2- 172.21.15.70:0/4115045184 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f90040a25f0 0x7f90040a4ab0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9018064c40 tx=0x7f901809ef70 comp rx=0 tx=0).stop 2024-04-01T01:34:50.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.959+0000 7f901eaab640 1 -- 172.21.15.70:0/4115045184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010094c80 msgr2=0x7f901013bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:50.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.959+0000 7f901eaab640 1 --2- 172.21.15.70:0/4115045184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010094c80 0x7f901013bb60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f901400e970 tx=0x7f901400ee30 comp rx=0 tx=0).stop 2024-04-01T01:34:50.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.959+0000 7f901eaab640 1 -- 172.21.15.70:0/4115045184 shutdown_connections 2024-04-01T01:34:50.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.959+0000 7f901eaab640 1 --2- 172.21.15.70:0/4115045184 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f90040a25f0 0x7f90040a4ab0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.959+0000 7f901eaab640 1 --2- 172.21.15.70:0/4115045184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9010096830 0x7f9010143110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.959+0000 7f901eaab640 1 --2- 172.21.15.70:0/4115045184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9010095e70 0x7f901013c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.959+0000 7f901eaab640 1 --2- 172.21.15.70:0/4115045184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010094c80 0x7f901013bb60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:50.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.959+0000 7f901eaab640 1 -- 172.21.15.70:0/4115045184 >> 172.21.15.70:0/4115045184 conn(0x7f90100901d0 msgr2=0x7f90100990a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:50.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.960+0000 7f901eaab640 1 -- 172.21.15.70:0/4115045184 shutdown_connections 2024-04-01T01:34:50.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:50.960+0000 7f901eaab640 1 -- 172.21.15.70:0/4115045184 wait complete. 2024-04-01T01:34:50.978 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:34:50.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:34:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.150+0000 7f61e1d3f640 1 -- 172.21.15.70:0/879816101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61dc1016f0 msgr2=0x7f61dc101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.150+0000 7f61e1d3f640 1 --2- 172.21.15.70:0/879816101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61dc1016f0 0x7f61dc101b70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009900 tx=0x7f61d002bc40 comp rx=0 tx=0).stop 2024-04-01T01:34:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.151+0000 7f61e1d3f640 1 -- 172.21.15.70:0/879816101 shutdown_connections 2024-04-01T01:34:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.151+0000 7f61e1d3f640 1 --2- 172.21.15.70:0/879816101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61dc1020b0 0x7f61dc108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.151+0000 7f61e1d3f640 1 --2- 172.21.15.70:0/879816101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61dc1016f0 0x7f61dc101b70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.151+0000 7f61e1d3f640 1 --2- 172.21.15.70:0/879816101 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61dc1009e0 0x7f61dc100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.151+0000 7f61e1d3f640 1 -- 172.21.15.70:0/879816101 >> 172.21.15.70:0/879816101 conn(0x7f61dc0fb1b0 msgr2=0x7f61dc0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.151+0000 7f61e1d3f640 1 -- 172.21.15.70:0/879816101 shutdown_connections 2024-04-01T01:34:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.151+0000 7f61e1d3f640 1 -- 172.21.15.70:0/879816101 wait complete. 2024-04-01T01:34:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.152+0000 7f61e1d3f640 1 Processor -- start 2024-04-01T01:34:51.152 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.152+0000 7f61e1d3f640 1 -- start start 2024-04-01T01:34:51.153 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.152+0000 7f61e1d3f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61dc1009e0 0x7f61dc1977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.153 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.152+0000 7f61e1d3f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61dc1016f0 0x7f61dc197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.153 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.152+0000 7f61e1d3f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61dc1020b0 0x7f61dc19eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.153+0000 7f61dbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61dc1016f0 0x7f61dc197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.153+0000 7f61e0d3d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61dc1009e0 0x7f61dc1977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.153+0000 7f61e1d3f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc10d180 con 0x7f61dc1016f0 2024-04-01T01:34:51.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.153+0000 7f61e1d3f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc10d000 con 0x7f61dc1009e0 2024-04-01T01:34:51.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.153+0000 7f61e0d3d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61dc1009e0 0x7f61dc1977f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46692/0 (socket says 172.21.15.70:46692) 2024-04-01T01:34:51.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.153+0000 7f61e0d3d640 1 -- 172.21.15.70:0/3369417944 learned_addr learned my addr 172.21.15.70:0/3369417944 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:51.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.153+0000 7f61e1d3f640 1 -- 172.21.15.70:0/3369417944 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc10d300 con 0x7f61dc1020b0 2024-04-01T01:34:51.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.153+0000 7f61e153e640 1 --2- 172.21.15.70:0/3369417944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61dc1020b0 0x7f61dc19eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.154+0000 7f61e153e640 1 -- 172.21.15.70:0/3369417944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61dc1009e0 msgr2=0x7f61dc1977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.154+0000 7f61e153e640 1 --2- 172.21.15.70:0/3369417944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61dc1009e0 0x7f61dc1977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.154+0000 7f61e153e640 1 -- 172.21.15.70:0/3369417944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61dc1016f0 msgr2=0x7f61dc197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.154+0000 7f61e153e640 1 --2- 172.21.15.70:0/3369417944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61dc1016f0 0x7f61dc197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.154+0000 7f61e153e640 1 -- 172.21.15.70:0/3369417944 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61dc19f410 con 0x7f61dc1020b0 2024-04-01T01:34:51.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.154+0000 7f61dbfff640 1 --2- 172.21.15.70:0/3369417944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61dc1016f0 0x7f61dc197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:34:51.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.154+0000 7f61e153e640 1 --2- 172.21.15.70:0/3369417944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61dc1020b0 0x7f61dc19eda0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f61d400e970 tx=0x7f61d400ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.154+0000 7f61d9ffb640 1 -- 172.21.15.70:0/3369417944 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d400ccc0 con 0x7f61dc1020b0 2024-04-01T01:34:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.154+0000 7f61d9ffb640 1 -- 172.21.15.70:0/3369417944 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61d4005020 con 0x7f61dc1020b0 2024-04-01T01:34:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.154+0000 7f61d9ffb640 1 -- 172.21.15.70:0/3369417944 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d4004a00 con 0x7f61dc1020b0 2024-04-01T01:34:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.154+0000 7f61e1d3f640 1 -- 172.21.15.70:0/3369417944 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61dc19f700 con 0x7f61dc1020b0 2024-04-01T01:34:51.156 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.155+0000 7f61e1d3f640 1 -- 172.21.15.70:0/3369417944 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61dc19fb70 con 0x7f61dc1020b0 2024-04-01T01:34:51.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.157+0000 7f64bcc07640 1 -- 172.21.15.70:0/2975309498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b810b0c0 msgr2=0x7f64b810d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.157+0000 7f64bcc07640 1 --2- 172.21.15.70:0/2975309498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b810b0c0 0x7f64b810d530 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f64b0009ca0 tx=0x7f64b002c100 comp rx=0 tx=0).stop 2024-04-01T01:34:51.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.157+0000 7f64bcc07640 1 -- 172.21.15.70:0/2975309498 shutdown_connections 2024-04-01T01:34:51.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.157+0000 7f64bcc07640 1 --2- 172.21.15.70:0/2975309498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64b810da70 0x7f64b810def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.157+0000 7f64bcc07640 1 --2- 172.21.15.70:0/2975309498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b810b0c0 0x7f64b810d530 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.157+0000 7f64bcc07640 1 --2- 172.21.15.70:0/2975309498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b8108750 0x7f64b810ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.157+0000 7f64bcc07640 1 -- 172.21.15.70:0/2975309498 >> 172.21.15.70:0/2975309498 conn(0x7f64b806e5b0 msgr2=0x7f64b80709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:51.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.157+0000 7f61d9ffb640 1 -- 172.21.15.70:0/3369417944 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f61d4004ba0 con 0x7f61dc1020b0 2024-04-01T01:34:51.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.158+0000 7f64bcc07640 1 -- 172.21.15.70:0/2975309498 shutdown_connections 2024-04-01T01:34:51.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.158+0000 7f64bcc07640 1 -- 172.21.15.70:0/2975309498 wait complete. 2024-04-01T01:34:51.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.158+0000 7f61d9ffb640 1 --2- 172.21.15.70:0/3369417944 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61c40a2590 0x7f61c40a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f64bcc07640 1 Processor -- start 2024-04-01T01:34:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f61e5b84740 1 Processor -- start 2024-04-01T01:34:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f61e5b84740 1 -- start start 2024-04-01T01:34:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f61e5b84740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c51a60 0x5556a5c51e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f61e5b84740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c52450 0x5556a5c54f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f61e5b84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c555b0 0x5556a5c62420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f61e5b84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5556a5bb58f0 con 0x5556a5c555b0 2024-04-01T01:34:51.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f61e5b84740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5556a5b5e8d0 con 0x5556a5c52450 2024-04-01T01:34:51.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f61e5b84740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5556a5bb1990 con 0x5556a5c51a60 2024-04-01T01:34:51.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f61d9ffb640 1 -- 172.21.15.70:0/3369417944 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f61d40ceb40 con 0x7f61dc1020b0 2024-04-01T01:34:51.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f64bcc07640 1 -- start start 2024-04-01T01:34:51.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f64bcc07640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8108750 0x7f64b81a6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f64bcc07640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64b810b0c0 0x7f64b81a6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f64bcc07640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b810da70 0x7f64b81addc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f64bcc07640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f64b811c100 con 0x7f64b810b0c0 2024-04-01T01:34:51.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f64bcc07640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f64b811bf80 con 0x7f64b8108750 2024-04-01T01:34:51.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f64bcc07640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f64b811c280 con 0x7f64b810da70 2024-04-01T01:34:51.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f61e0d3d640 1 --2- 172.21.15.70:0/3369417944 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61c40a2590 0x7f61c40a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f61dbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c52450 0x5556a5c54f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f61dbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c52450 0x5556a5c54f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46708/0 (socket says 172.21.15.70:46708) 2024-04-01T01:34:51.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f61dbfff640 1 -- 172.21.15.70:0/3665995190 learned_addr learned my addr 172.21.15.70:0/3665995190 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:51.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f64b659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8108750 0x7f64b81a6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f64b6d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b810da70 0x7f64b81addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f64b659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8108750 0x7f64b81a6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46714/0 (socket says 172.21.15.70:46714) 2024-04-01T01:34:51.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f64b659c640 1 -- 172.21.15.70:0/3338422312 learned_addr learned my addr 172.21.15.70:0/3338422312 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f61e153e640 1 --2- 172.21.15.70:0/3665995190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c555b0 0x5556a5c62420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.159+0000 7f61e0d3d640 1 --2- 172.21.15.70:0/3665995190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c51a60 0x5556a5c51e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f61dbfff640 1 -- 172.21.15.70:0/3665995190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c51a60 msgr2=0x5556a5c51e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f61dbfff640 1 --2- 172.21.15.70:0/3665995190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c51a60 0x5556a5c51e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f61dbfff640 1 -- 172.21.15.70:0/3665995190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c555b0 msgr2=0x5556a5c62420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f61dbfff640 1 --2- 172.21.15.70:0/3665995190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c555b0 0x5556a5c62420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f61dbfff640 1 -- 172.21.15.70:0/3665995190 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556a58ca210 con 0x5556a5c52450 2024-04-01T01:34:51.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f61e153e640 1 --2- 172.21.15.70:0/3665995190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c555b0 0x5556a5c62420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:34:51.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f64b5d9b640 1 --2- 172.21.15.70:0/3338422312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64b810b0c0 0x7f64b81a6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f64b659c640 1 -- 172.21.15.70:0/3338422312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b810da70 msgr2=0x7f64b81addc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f64b659c640 1 --2- 172.21.15.70:0/3338422312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b810da70 0x7f64b81addc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f64b659c640 1 -- 172.21.15.70:0/3338422312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64b810b0c0 msgr2=0x7f64b81a6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f64b659c640 1 --2- 172.21.15.70:0/3338422312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64b810b0c0 0x7f64b81a6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f64b659c640 1 -- 172.21.15.70:0/3338422312 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b810ffb0 con 0x7f64b8108750 2024-04-01T01:34:51.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f64b6d9d640 1 --2- 172.21.15.70:0/3338422312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b810da70 0x7f64b81addc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:34:51.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f64b5d9b640 1 --2- 172.21.15.70:0/3338422312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64b810b0c0 0x7f64b81a6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f61dbfff640 1 --2- 172.21.15.70:0/3665995190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c52450 0x5556a5c54f40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f61d0000c00 tx=0x7f61d0005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27a0820a96f1a27b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:51.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f61e0d3d640 1 --2- 172.21.15.70:0/3369417944 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61c40a2590 0x7f61c40a4a50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x5556a5c52cb0 tx=0x7f61cc005bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f61c2ffd640 1 -- 172.21.15.70:0/3665995190 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d0044020 con 0x5556a5c52450 2024-04-01T01:34:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f61c2ffd640 1 -- 172.21.15.70:0/3665995190 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61d002cc30 con 0x5556a5c52450 2024-04-01T01:34:51.167 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.160+0000 7f61c2ffd640 1 -- 172.21.15.70:0/3665995190 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d0004920 con 0x5556a5c52450 2024-04-01T01:34:51.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.161+0000 7f64b659c640 1 --2- 172.21.15.70:0/3338422312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8108750 0x7f64b81a6810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f64a800d950 tx=0x7f64a800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:51.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.161+0000 7f64a77fe640 1 -- 172.21.15.70:0/3338422312 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a8017070 con 0x7f64b8108750 2024-04-01T01:34:51.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.161+0000 7f64bcc07640 1 -- 172.21.15.70:0/3338422312 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64b81102a0 con 0x7f64b8108750 2024-04-01T01:34:51.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.161+0000 7f64a77fe640 1 -- 172.21.15.70:0/3338422312 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64a8009c30 con 0x7f64b8108750 2024-04-01T01:34:51.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.161+0000 7f64a77fe640 1 -- 172.21.15.70:0/3338422312 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a8017070 con 0x7f64b8108750 2024-04-01T01:34:51.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.161+0000 7f64bcc07640 1 -- 172.21.15.70:0/3338422312 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64b8110810 con 0x7f64b8108750 2024-04-01T01:34:51.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.161+0000 7f61e5b84740 1 -- 172.21.15.70:0/3665995190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c52450 msgr2=0x5556a5c54f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.162+0000 7f61e5b84740 1 --2- 172.21.15.70:0/3665995190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c52450 0x5556a5c54f40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f61d0000c00 tx=0x7f61d0005d50 comp rx=0 tx=0).stop 2024-04-01T01:34:51.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.162+0000 7f61e5b84740 1 -- 172.21.15.70:0/3665995190 shutdown_connections 2024-04-01T01:34:51.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.162+0000 7f61e5b84740 1 --2- 172.21.15.70:0/3665995190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c555b0 0x5556a5c62420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.162+0000 7f61e5b84740 1 --2- 172.21.15.70:0/3665995190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c52450 0x5556a5c54f40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.162+0000 7f61e5b84740 1 --2- 172.21.15.70:0/3665995190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c51a60 0x5556a5c51e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.162+0000 7f61e5b84740 1 -- 172.21.15.70:0/3665995190 >> 172.21.15.70:0/3665995190 conn(0x5556a5c4d050 msgr2=0x5556a5c4d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:51.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.162+0000 7f61e5b84740 1 -- 172.21.15.70:0/3665995190 shutdown_connections 2024-04-01T01:34:51.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.162+0000 7f61e5b84740 1 -- 172.21.15.70:0/3665995190 wait complete. 2024-04-01T01:34:51.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.163+0000 7f61e5b84740 1 -- start start 2024-04-01T01:34:51.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.163+0000 7f61e5b84740 1 Processor -- start 2024-04-01T01:34:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.164+0000 7f61e5b84740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c51a60 0x5556a5c5bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.164+0000 7f61e5b84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c52450 0x5556a5c5c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.164+0000 7f61e5b84740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c555b0 0x5556a5c612a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.164+0000 7f61e5b84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5556a5c55f90 con 0x5556a5c52450 2024-04-01T01:34:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.164+0000 7f61dbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c52450 0x5556a5c5c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.164+0000 7f61e0d3d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c51a60 0x5556a5c5bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.164+0000 7f61e5b84740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5556a5b53f80 con 0x5556a5c555b0 2024-04-01T01:34:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.164+0000 7f61e5b84740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5556a5c56110 con 0x5556a5c51a60 2024-04-01T01:34:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.164+0000 7f61e153e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c555b0 0x5556a5c612a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.164+0000 7f61e0d3d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c51a60 0x5556a5c5bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39368/0 (socket says 172.21.15.70:39368) 2024-04-01T01:34:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.164+0000 7f61e153e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c555b0 0x5556a5c612a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46728/0 (socket says 172.21.15.70:46728) 2024-04-01T01:34:51.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.164+0000 7f61e0d3d640 1 -- 172.21.15.70:0/2993019933 learned_addr learned my addr 172.21.15.70:0/2993019933 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:51.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.165+0000 7f61e5b84740 10 client.0 _ll_register_callbacks cb 0x5556a5bac290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:51.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.165+0000 7f64bf7ef740 1 Processor -- start 2024-04-01T01:34:51.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.165+0000 7f64bf7ef740 1 -- start start 2024-04-01T01:34:51.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.165+0000 7f64bf7ef740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55905933d2b0 0x55905933d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.165+0000 7f64bf7ef740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55905933dca0 0x5590593427a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.165+0000 7f64bf7ef740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559059342e10 0x5590593452a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.165+0000 7f64bf7ef740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5590592a2180 con 0x55905933d2b0 2024-04-01T01:34:51.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.165+0000 7f64bf7ef740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55905924ab70 con 0x559059342e10 2024-04-01T01:34:51.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.165+0000 7f64bf7ef740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55905929e220 con 0x55905933dca0 2024-04-01T01:34:51.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.165+0000 7f61dbfff640 1 -- 172.21.15.70:0/2993019933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c51a60 msgr2=0x5556a5c5bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:51.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.165+0000 7f61dbfff640 1 --2- 172.21.15.70:0/2993019933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c51a60 0x5556a5c5bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.165+0000 7f61dbfff640 1 -- 172.21.15.70:0/2993019933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c555b0 msgr2=0x5556a5c612a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:51.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.165+0000 7f61dbfff640 1 --2- 172.21.15.70:0/2993019933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c555b0 0x5556a5c612a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.165+0000 7f61dbfff640 1 -- 172.21.15.70:0/2993019933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556a5996620 con 0x5556a5c52450 2024-04-01T01:34:51.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.166+0000 7f61dbfff640 1 --2- 172.21.15.70:0/2993019933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c52450 0x5556a5c5c270 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x5556a5c4dfc0 tx=0x7f61d0003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb159faa24f733e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:51.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.166+0000 7f61bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:51.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.166+0000 7f64b6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55905933d2b0 0x55905933d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.166+0000 7f64b6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55905933d2b0 0x55905933d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41930/0 (socket says 172.21.15.70:41930) 2024-04-01T01:34:51.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.166+0000 7f64b6d9d640 1 -- 172.21.15.70:0/2576297158 learned_addr learned my addr 172.21.15.70:0/2576297158 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:51.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.166+0000 7f64a77fe640 1 -- 172.21.15.70:0/3338422312 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f64a8010460 con 0x7f64b8108750 2024-04-01T01:34:51.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.166+0000 7f61bbfff640 1 -- 172.21.15.70:0/2993019933 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d0044040 con 0x5556a5c52450 2024-04-01T01:34:51.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.166+0000 7f61e5b84740 1 -- 172.21.15.70:0/2993019933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61d002cc30 con 0x5556a5c52450 2024-04-01T01:34:51.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.166+0000 7f61bbfff640 1 -- 172.21.15.70:0/2993019933 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f61d002ca30 con 0x5556a5c52450 2024-04-01T01:34:51.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.166+0000 7f64b5d9b640 1 --2- 172.21.15.70:0/2576297158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559059342e10 0x5590593452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.167+0000 7f61bbfff640 1 -- 172.21.15.70:0/2993019933 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f61d0058050 con 0x5556a5c52450 2024-04-01T01:34:51.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.167+0000 7f61e5b84740 10 client.16044 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:51.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.167+0000 7f61e5b84740 1 -- 172.21.15.70:0/2993019933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5556a5c772d0 con 0x5556a5c52450 2024-04-01T01:34:51.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.167+0000 7f64a77fe640 1 --2- 172.21.15.70:0/3338422312 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64a00a2960 0x7f64a00a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.167+0000 7f64b6d9d640 1 -- 172.21.15.70:0/2576297158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55905933dca0 msgr2=0x5590593427a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:34:51.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.167+0000 7f64b6d9d640 1 --2- 172.21.15.70:0/2576297158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55905933dca0 0x5590593427a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.167+0000 7f64b6d9d640 1 -- 172.21.15.70:0/2576297158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559059342e10 msgr2=0x5590593452a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:51.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.167+0000 7f64b6d9d640 1 --2- 172.21.15.70:0/2576297158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559059342e10 0x5590593452a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.167+0000 7f64b6d9d640 1 -- 172.21.15.70:0/2576297158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55905925e7c0 con 0x55905933d2b0 2024-04-01T01:34:51.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.167+0000 7f64a77fe640 1 -- 172.21.15.70:0/3338422312 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f64a8009a30 con 0x7f64b8108750 2024-04-01T01:34:51.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.167+0000 7f64b5d9b640 1 --2- 172.21.15.70:0/2576297158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559059342e10 0x5590593452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:51.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.167+0000 7f61bbfff640 1 -- 172.21.15.70:0/2993019933 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61d0003ba0 con 0x5556a5c52450 2024-04-01T01:34:51.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.167+0000 7f61bbfff640 1 -- 172.21.15.70:0/2993019933 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f61d004b460 con 0x5556a5c52450 2024-04-01T01:34:51.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.167+0000 7f64b6d9d640 1 --2- 172.21.15.70:0/3338422312 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64a00a2960 0x7f64a00a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.168+0000 7f64b6d9d640 1 --2- 172.21.15.70:0/2576297158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55905933d2b0 0x55905933d6d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f64ac009c60 tx=0x7f64ac0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd5fee9d74dccdf server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:51.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.168+0000 7f61e5b84740 10 client.16044 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:51.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.168+0000 7f61e5b84740 10 client.16044 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:51.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.168+0000 7f61e5b84740 20 client.16044 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:51.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.168+0000 7f61e5b84740 1 --2- 172.21.15.70:0/2993019933 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5556a5c78aa0 0x5556a5c7aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.168+0000 7f64a4ff9640 1 -- 172.21.15.70:0/2576297158 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac01b6e0 con 0x55905933d2b0 2024-04-01T01:34:51.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.168+0000 7f61e5b84740 4 client.16044 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:51.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.168+0000 7f64b6d9d640 1 --2- 172.21.15.70:0/3338422312 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64a00a2960 0x7f64a00a4e20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f64ac022000 tx=0x7f64ac023000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:51.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.168+0000 7f64a4ff9640 1 -- 172.21.15.70:0/2576297158 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64ac00f8e0 con 0x55905933d2b0 2024-04-01T01:34:51.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.168+0000 7f64a4ff9640 1 -- 172.21.15.70:0/2576297158 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac00e730 con 0x55905933d2b0 2024-04-01T01:34:51.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.168+0000 7f61e0d3d640 1 --2- 172.21.15.70:0/2993019933 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5556a5c78aa0 0x5556a5c7aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.168+0000 7f61e5b84740 1 -- 172.21.15.70:0/2993019933 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f61d002cc30 con 0x5556a5c78aa0 2024-04-01T01:34:51.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.169+0000 7f61e0d3d640 1 --2- 172.21.15.70:0/2993019933 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5556a5c78aa0 0x5556a5c7aeb0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75120be693502475 server_cookie=3fd62ec98ddba92c in_seq=0 out_seq=0 2024-04-01T01:34:51.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.169+0000 7f61bbfff640 10 client.16044 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:51.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.169+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2576297158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55905933d2b0 msgr2=0x55905933d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.169+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/2576297158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55905933d2b0 0x55905933d6d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f64ac009c60 tx=0x7f64ac0117b0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.169+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2576297158 shutdown_connections 2024-04-01T01:34:51.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.169+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/2576297158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559059342e10 0x5590593452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.169+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/2576297158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55905933dca0 0x5590593427a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.169+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/2576297158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55905933d2b0 0x55905933d6d0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.169+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2576297158 >> 172.21.15.70:0/2576297158 conn(0x559059338820 msgr2=0x559059338c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:51.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.170+0000 7f61bbfff640 1 -- 172.21.15.70:0/2993019933 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f61d002cc30 con 0x5556a5c78aa0 2024-04-01T01:34:51.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.170+0000 7f61bbfff640 10 client.16044 handle_command_reply: tid=0 2024-04-01T01:34:51.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.170+0000 7f61e5b84740 1 client.16044 shutdown 2024-04-01T01:34:51.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.170+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2576297158 shutdown_connections 2024-04-01T01:34:51.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.170+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2576297158 wait complete. 2024-04-01T01:34:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.170+0000 7f61e5b84740 1 -- 172.21.15.70:0/2993019933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c52450 msgr2=0x5556a5c5c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.170+0000 7f61e5b84740 1 --2- 172.21.15.70:0/2993019933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c52450 0x5556a5c5c270 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x5556a5c4dfc0 tx=0x7f61d0003820 comp rx=0 tx=0).stop 2024-04-01T01:34:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.170+0000 7f61e5b84740 1 -- 172.21.15.70:0/2993019933 shutdown_connections 2024-04-01T01:34:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.170+0000 7f61e5b84740 1 --2- 172.21.15.70:0/2993019933 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5556a5c78aa0 0x5556a5c7aeb0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.170+0000 7f61e5b84740 1 --2- 172.21.15.70:0/2993019933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c555b0 0x5556a5c612a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.170+0000 7f61bbfff640 0 client.16044 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:51.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.170+0000 7f61e5b84740 1 --2- 172.21.15.70:0/2993019933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c52450 0x5556a5c5c270 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.170+0000 7f61e5b84740 1 --2- 172.21.15.70:0/2993019933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c51a60 0x5556a5c5bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.170+0000 7f61e5b84740 1 -- 172.21.15.70:0/2993019933 >> 172.21.15.70:0/2993019933 conn(0x5556a5c4d050 msgr2=0x5556a5c4c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.170+0000 7f61e5b84740 1 -- 172.21.15.70:0/2993019933 shutdown_connections 2024-04-01T01:34:51.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.170+0000 7f61e5b84740 1 -- 172.21.15.70:0/2993019933 wait complete. 2024-04-01T01:34:51.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.171+0000 7f64bf7ef740 1 -- start start 2024-04-01T01:34:51.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.171+0000 7f64bf7ef740 1 Processor -- start 2024-04-01T01:34:51.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.171+0000 7f64bf7ef740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933d2b0 0x5590593478b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.171+0000 7f61e5b84740 20 client.16044 trim_cache size 0 max 16384 2024-04-01T01:34:51.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.171+0000 7f64bf7ef740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55905933dca0 0x559059347df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.171+0000 7f64b659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933d2b0 0x5590593478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.171+0000 7f64bf7ef740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559059342e10 0x55905934ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.172+0000 7f64b659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933d2b0 0x5590593478b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46754/0 (socket says 172.21.15.70:46754) 2024-04-01T01:34:51.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.172+0000 7f64b659c640 1 -- 172.21.15.70:0/3354936220 learned_addr learned my addr 172.21.15.70:0/3354936220 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:51.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.172+0000 7f64bf7ef740 1 -- 172.21.15.70:0/3354936220 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559059345dc0 con 0x559059342e10 2024-04-01T01:34:51.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.172+0000 7f64bf7ef740 1 -- 172.21.15.70:0/3354936220 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5590592404a0 con 0x55905933d2b0 2024-04-01T01:34:51.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.172+0000 7f64bf7ef740 1 -- 172.21.15.70:0/3354936220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559059345f40 con 0x55905933dca0 2024-04-01T01:34:51.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.172+0000 7f64bf7ef740 10 client.0 _ll_register_callbacks cb 0x559059243d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:51.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.172+0000 7f64b5d9b640 1 --2- 172.21.15.70:0/3354936220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55905933dca0 0x559059347df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.172+0000 7f64b6d9d640 1 --2- 172.21.15.70:0/3354936220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559059342e10 0x55905934ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.172+0000 7f64b659c640 1 -- 172.21.15.70:0/3354936220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55905933dca0 msgr2=0x559059347df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:51.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.172+0000 7f64b659c640 1 --2- 172.21.15.70:0/3354936220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55905933dca0 0x559059347df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.172+0000 7f64b659c640 1 -- 172.21.15.70:0/3354936220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559059342e10 msgr2=0x55905934ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:51.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.172+0000 7f64b659c640 1 --2- 172.21.15.70:0/3354936220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559059342e10 0x55905934ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.172+0000 7f64b659c640 1 -- 172.21.15.70:0/3354936220 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590590807d0 con 0x55905933d2b0 2024-04-01T01:34:51.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.173+0000 7f64b659c640 1 --2- 172.21.15.70:0/3354936220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933d2b0 0x5590593478b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f64a80a2c50 tx=0x7f64a809a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe4de06d5c5ebf24 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:51.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.173+0000 7f649dffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:34:51.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.173+0000 7f649dffb640 1 -- 172.21.15.70:0/3354936220 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a8017070 con 0x55905933d2b0 2024-04-01T01:34:51.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.173+0000 7f649dffb640 1 -- 172.21.15.70:0/3354936220 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f64a80cb8c0 con 0x55905933d2b0 2024-04-01T01:34:51.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.173+0000 7f64bf7ef740 1 -- 172.21.15.70:0/3354936220 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64ac00f8e0 con 0x55905933d2b0 2024-04-01T01:34:51.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.173+0000 7f649dffb640 1 -- 172.21.15.70:0/3354936220 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f64a8020070 con 0x55905933d2b0 2024-04-01T01:34:51.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.173+0000 7f64bf7ef740 10 client.25877 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:51.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.173+0000 7f64bf7ef740 1 -- 172.21.15.70:0/3354936220 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559058fb63f0 con 0x55905933d2b0 2024-04-01T01:34:51.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.174+0000 7f649dffb640 1 -- 172.21.15.70:0/3354936220 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64ac00f8e0 con 0x55905933d2b0 2024-04-01T01:34:51.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.174+0000 7f649dffb640 1 -- 172.21.15.70:0/3354936220 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f64a80109a0 con 0x55905933d2b0 2024-04-01T01:34:51.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.174+0000 7f64bf7ef740 10 client.25877 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:51.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.174+0000 7f64bf7ef740 10 client.25877 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:51.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.174+0000 7f64bf7ef740 20 client.25877 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:51.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.174+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/3354936220 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5590593536c0 0x559059355b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.174+0000 7f64bf7ef740 4 client.25877 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:34:51.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.174+0000 7f64bf7ef740 1 -- 172.21.15.70:0/3354936220 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590592af6c0 con 0x5590593536c0 2024-04-01T01:34:51.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.175+0000 7f64b5d9b640 1 --2- 172.21.15.70:0/3354936220 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5590593536c0 0x559059355b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.175+0000 7f64b5d9b640 1 --2- 172.21.15.70:0/3354936220 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5590593536c0 0x559059355b00 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=463973dab2e59b4a server_cookie=99a3e50fc799cfc4 in_seq=0 out_seq=0 2024-04-01T01:34:51.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.175+0000 7f649dffb640 10 client.25877 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:51.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.176+0000 7f649dffb640 1 -- 172.21.15.70:0/3354936220 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5590592af6c0 con 0x5590593536c0 2024-04-01T01:34:51.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.176+0000 7f649dffb640 10 client.25877 handle_command_reply: tid=0 2024-04-01T01:34:51.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.176+0000 7f64bf7ef740 1 client.25877 shutdown 2024-04-01T01:34:51.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.176+0000 7f64bf7ef740 1 -- 172.21.15.70:0/3354936220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933d2b0 msgr2=0x5590593478b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.176+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/3354936220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933d2b0 0x5590593478b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f64a80a2c50 tx=0x7f64a809a0d0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.177+0000 7f64bf7ef740 1 -- 172.21.15.70:0/3354936220 shutdown_connections 2024-04-01T01:34:51.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.177+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/3354936220 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5590593536c0 0x559059355b00 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.177+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/3354936220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559059342e10 0x55905934ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.177+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/3354936220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55905933dca0 0x559059347df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.177+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/3354936220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933d2b0 0x5590593478b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.177+0000 7f64bf7ef740 1 -- 172.21.15.70:0/3354936220 >> 172.21.15.70:0/3354936220 conn(0x559059338820 msgr2=0x559059337c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:51.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.177+0000 7f649dffb640 0 client.25877 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:51.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.177+0000 7f64bf7ef740 1 -- 172.21.15.70:0/3354936220 shutdown_connections 2024-04-01T01:34:51.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.177+0000 7f64bf7ef740 1 -- 172.21.15.70:0/3354936220 wait complete. 2024-04-01T01:34:51.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.177+0000 7f64bf7ef740 20 client.25877 trim_cache size 0 max 16384 2024-04-01T01:34:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.199+0000 7f61e5b84740 1 Processor -- start 2024-04-01T01:34:51.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.199+0000 7f61e5b84740 1 -- start start 2024-04-01T01:34:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.199+0000 7f61e5b84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c51a60 0x5556a5c807c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.200+0000 7f61e5b84740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c52450 0x5556a5c502b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.200+0000 7f61e5b84740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c555b0 0x5556a5c507f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.200+0000 7f61e5b84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc1a8150 con 0x5556a5c51a60 2024-04-01T01:34:51.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.200+0000 7f61e5b84740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc1a9c90 con 0x5556a5c555b0 2024-04-01T01:34:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.200+0000 7f61e5b84740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc1a8820 con 0x5556a5c52450 2024-04-01T01:34:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.200+0000 7f61e153e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c555b0 0x5556a5c507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.200+0000 7f61dbfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c52450 0x5556a5c502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.200+0000 7f61dbfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c52450 0x5556a5c502b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39404/0 (socket says 172.21.15.70:39404) 2024-04-01T01:34:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.200+0000 7f61dbfff640 1 -- 172.21.15.70:0/3170849650 learned_addr learned my addr 172.21.15.70:0/3170849650 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.200+0000 7f61e0d3d640 1 --2- 172.21.15.70:0/3170849650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c51a60 0x5556a5c807c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.201+0000 7f61e0d3d640 1 -- 172.21.15.70:0/3170849650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c52450 msgr2=0x5556a5c502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.201+0000 7f61e0d3d640 1 --2- 172.21.15.70:0/3170849650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c52450 0x5556a5c502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.201+0000 7f61e0d3d640 1 -- 172.21.15.70:0/3170849650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c555b0 msgr2=0x5556a5c507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.201+0000 7f61e0d3d640 1 --2- 172.21.15.70:0/3170849650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c555b0 0x5556a5c507f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.201+0000 7f61e0d3d640 1 -- 172.21.15.70:0/3170849650 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556a5c78850 con 0x5556a5c51a60 2024-04-01T01:34:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.201+0000 7f61e153e640 1 --2- 172.21.15.70:0/3170849650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c555b0 0x5556a5c507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:34:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.201+0000 7f61e0d3d640 1 --2- 172.21.15.70:0/3170849650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c51a60 0x5556a5c807c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x5556a5c7b700 tx=0x7f61cc015070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3155ef93bc70b2b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:51.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.202+0000 7f61c27fc640 1 -- 172.21.15.70:0/3170849650 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61cc02d6b0 con 0x5556a5c51a60 2024-04-01T01:34:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.202+0000 7f61c27fc640 1 -- 172.21.15.70:0/3170849650 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61cc033070 con 0x5556a5c51a60 2024-04-01T01:34:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.202+0000 7f61c27fc640 1 -- 172.21.15.70:0/3170849650 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61cc01f6a0 con 0x5556a5c51a60 2024-04-01T01:34:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.203+0000 7f61e5b84740 1 -- 172.21.15.70:0/3170849650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c51a60 msgr2=0x5556a5c807c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.203+0000 7f61e5b84740 1 --2- 172.21.15.70:0/3170849650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c51a60 0x5556a5c807c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x5556a5c7b700 tx=0x7f61cc015070 comp rx=0 tx=0).stop 2024-04-01T01:34:51.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.203+0000 7f61e5b84740 1 -- 172.21.15.70:0/3170849650 shutdown_connections 2024-04-01T01:34:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.203+0000 7f61e5b84740 1 --2- 172.21.15.70:0/3170849650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c555b0 0x5556a5c507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.203+0000 7f61e5b84740 1 --2- 172.21.15.70:0/3170849650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c52450 0x5556a5c502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.203+0000 7f61e5b84740 1 --2- 172.21.15.70:0/3170849650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c51a60 0x5556a5c807c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.203+0000 7f61e5b84740 1 -- 172.21.15.70:0/3170849650 >> 172.21.15.70:0/3170849650 conn(0x5556a5c4d050 msgr2=0x5556a5c4d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.203+0000 7f61e5b84740 1 -- 172.21.15.70:0/3170849650 shutdown_connections 2024-04-01T01:34:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.203+0000 7f61e5b84740 1 -- 172.21.15.70:0/3170849650 wait complete. 2024-04-01T01:34:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.204+0000 7f61e5b84740 1 -- start start 2024-04-01T01:34:51.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.204+0000 7f61e5b84740 1 Processor -- start 2024-04-01T01:34:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.204+0000 7f61e5b84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c51a60 0x5556a5c89c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.204+0000 7f61e5b84740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c52450 0x5556a5c8a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.204+0000 7f61e5b84740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c555b0 0x5556a5c8b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.204+0000 7f61e5b84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc1a95c0 con 0x5556a5c51a60 2024-04-01T01:34:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.204+0000 7f61e5b84740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc1a8ef0 con 0x5556a5c52450 2024-04-01T01:34:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.204+0000 7f61e5b84740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc1a6130 con 0x5556a5c555b0 2024-04-01T01:34:51.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.204+0000 7f61e5b84740 10 client.0 _ll_register_callbacks cb 0x7f61d0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.205+0000 7f61e0d3d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c51a60 0x5556a5c89c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.206 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.205+0000 7f61e153e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c555b0 0x5556a5c8b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.205+0000 7f61e0d3d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c51a60 0x5556a5c89c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41954/0 (socket says 172.21.15.70:41954) 2024-04-01T01:34:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.205+0000 7f61e0d3d640 1 -- 172.21.15.70:0/3788157778 learned_addr learned my addr 172.21.15.70:0/3788157778 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.205+0000 7f61dbfff640 1 --2- 172.21.15.70:0/3788157778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c52450 0x5556a5c8a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.205+0000 7f61e0d3d640 1 -- 172.21.15.70:0/3788157778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c555b0 msgr2=0x5556a5c8b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.205+0000 7f61e0d3d640 1 --2- 172.21.15.70:0/3788157778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c555b0 0x5556a5c8b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.205+0000 7f61e0d3d640 1 -- 172.21.15.70:0/3788157778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c52450 msgr2=0x5556a5c8a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.205+0000 7f61e0d3d640 1 --2- 172.21.15.70:0/3788157778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c52450 0x5556a5c8a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.205+0000 7f61e0d3d640 1 -- 172.21.15.70:0/3788157778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556a5c8b780 con 0x5556a5c51a60 2024-04-01T01:34:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.205+0000 7f61e153e640 1 --2- 172.21.15.70:0/3788157778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c555b0 0x5556a5c8b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:34:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.206+0000 7f61e0d3d640 1 --2- 172.21.15.70:0/3788157778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c51a60 0x5556a5c89c40 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f61cc009b20 tx=0x7f61cc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d737f2a3d75f20f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.206+0000 7f61bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.206+0000 7f61bbfff640 1 -- 172.21.15.70:0/3788157778 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61cc03e070 con 0x5556a5c51a60 2024-04-01T01:34:51.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.206+0000 7f61bbfff640 1 -- 172.21.15.70:0/3788157778 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f61cc004a40 con 0x5556a5c51a60 2024-04-01T01:34:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.206+0000 7f61e5b84740 1 -- 172.21.15.70:0/3788157778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61cc033070 con 0x5556a5c51a60 2024-04-01T01:34:51.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.207+0000 7f61bbfff640 1 -- 172.21.15.70:0/3788157778 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f61cc048050 con 0x5556a5c51a60 2024-04-01T01:34:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.207+0000 7f61e5b84740 10 client.16056 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.207+0000 7f61e5b84740 1 -- 172.21.15.70:0/3788157778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5556a5cded30 con 0x5556a5c51a60 2024-04-01T01:34:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.208+0000 7f61bbfff640 1 -- 172.21.15.70:0/3788157778 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61cc033070 con 0x5556a5c51a60 2024-04-01T01:34:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.208+0000 7f61bbfff640 1 -- 172.21.15.70:0/3788157778 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f61cc01d040 con 0x5556a5c51a60 2024-04-01T01:34:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.208+0000 7f61e5b84740 10 client.16056 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.208+0000 7f61e5b84740 10 client.16056 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.208+0000 7f61e5b84740 20 client.16056 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.208+0000 7f61e5b84740 1 --2- 172.21.15.70:0/3788157778 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5556a5c78aa0 0x5556a5ce16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.208+0000 7f61e5b84740 4 client.16056 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:34:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.208+0000 7f61e5b84740 1 -- 172.21.15.70:0/3788157778 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5556a5bc9120 con 0x5556a5c78aa0 2024-04-01T01:34:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.208+0000 7f61dbfff640 1 --2- 172.21.15.70:0/3788157778 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5556a5c78aa0 0x5556a5ce16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.208+0000 7f61bbfff640 10 client.16056 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:51.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.209+0000 7f61dbfff640 1 --2- 172.21.15.70:0/3788157778 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5556a5c78aa0 0x5556a5ce16a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fed190a6e59ee86 server_cookie=5ede2b1e1c4a7a3a in_seq=0 out_seq=0 2024-04-01T01:34:51.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.211+0000 7f64bf7ef740 1 Processor -- start 2024-04-01T01:34:51.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.211+0000 7f64bf7ef740 1 -- start start 2024-04-01T01:34:51.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.211+0000 7f64bf7ef740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55905933d2b0 0x5590593474c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.212+0000 7f64bf7ef740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933dca0 0x55905934edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.212+0000 7f64b659c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55905933d2b0 0x5590593474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.212+0000 7f64b659c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55905933d2b0 0x5590593474c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39416/0 (socket says 172.21.15.70:39416) 2024-04-01T01:34:51.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.212+0000 7f64bf7ef740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559059342e10 0x559059347a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.212+0000 7f64b5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933dca0 0x55905934edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.212+0000 7f64b659c640 1 -- 172.21.15.70:0/2984192695 learned_addr learned my addr 172.21.15.70:0/2984192695 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:51.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.212+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2984192695 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f649800fcf0 con 0x559059342e10 2024-04-01T01:34:51.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.212+0000 7f64b6d9d640 1 --2- 172.21.15.70:0/2984192695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559059342e10 0x559059347a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.212+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2984192695 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6498011830 con 0x55905933dca0 2024-04-01T01:34:51.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.212+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2984192695 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f64980103c0 con 0x55905933d2b0 2024-04-01T01:34:51.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.213+0000 7f64b6d9d640 1 -- 172.21.15.70:0/2984192695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55905933d2b0 msgr2=0x5590593474c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:51.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.213+0000 7f64b6d9d640 1 --2- 172.21.15.70:0/2984192695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55905933d2b0 0x5590593474c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.213+0000 7f64b6d9d640 1 -- 172.21.15.70:0/2984192695 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933dca0 msgr2=0x55905934edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:51.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.213+0000 7f64b6d9d640 1 --2- 172.21.15.70:0/2984192695 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933dca0 0x55905934edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.213+0000 7f64b6d9d640 1 -- 172.21.15.70:0/2984192695 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559058f760f0 con 0x559059342e10 2024-04-01T01:34:51.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.214+0000 7f64b6d9d640 1 --2- 172.21.15.70:0/2984192695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559059342e10 0x559059347a00 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f64ac005890 tx=0x7f64ac027000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28fcd50bba1b252a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:51.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.214+0000 7f649ffff640 1 -- 172.21.15.70:0/2984192695 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac00eaa0 con 0x559059342e10 2024-04-01T01:34:51.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.214+0000 7f649ffff640 1 -- 172.21.15.70:0/2984192695 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64ac02d050 con 0x559059342e10 2024-04-01T01:34:51.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.214+0000 7f649ffff640 1 -- 172.21.15.70:0/2984192695 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac0052a0 con 0x559059342e10 2024-04-01T01:34:51.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.216+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2984192695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559059342e10 msgr2=0x559059347a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.216+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/2984192695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559059342e10 0x559059347a00 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f64ac005890 tx=0x7f64ac027000 comp rx=0 tx=0).stop 2024-04-01T01:34:51.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.216+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2984192695 shutdown_connections 2024-04-01T01:34:51.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.216+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/2984192695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559059342e10 0x559059347a00 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.216+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/2984192695 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933dca0 0x55905934edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.216+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/2984192695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55905933d2b0 0x5590593474c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.216+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2984192695 >> 172.21.15.70:0/2984192695 conn(0x559059338820 msgr2=0x55905933f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:51.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.216+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2984192695 shutdown_connections 2024-04-01T01:34:51.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.216+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2984192695 wait complete. 2024-04-01T01:34:51.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.217+0000 7f64bf7ef740 1 -- start start 2024-04-01T01:34:51.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.217+0000 7f64bf7ef740 1 Processor -- start 2024-04-01T01:34:51.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.217+0000 7f64bf7ef740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55905933d2b0 0x55905934bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.218+0000 7f64bf7ef740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933dca0 0x55905934c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.218+0000 7f64bf7ef740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559059342e10 0x55905934d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.218+0000 7f64bf7ef740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6498011160 con 0x55905933d2b0 2024-04-01T01:34:51.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.218+0000 7f64bf7ef740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6498010a90 con 0x55905933dca0 2024-04-01T01:34:51.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.218+0000 7f64bf7ef740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f649800dcd0 con 0x559059342e10 2024-04-01T01:34:51.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.218+0000 7f64b659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55905933d2b0 0x55905934bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.218+0000 7f64bf7ef740 10 client.0 _ll_register_callbacks cb 0x7f64a809a5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:34:51.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.218+0000 7f64b659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55905933d2b0 0x55905934bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41972/0 (socket says 172.21.15.70:41972) 2024-04-01T01:34:51.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.218+0000 7f64b659c640 1 -- 172.21.15.70:0/2762370236 learned_addr learned my addr 172.21.15.70:0/2762370236 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:34:51.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.218+0000 7f64b6d9d640 1 --2- 172.21.15.70:0/2762370236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559059342e10 0x55905934d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.218+0000 7f64b5d9b640 1 --2- 172.21.15.70:0/2762370236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933dca0 0x55905934c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.218+0000 7f64b659c640 1 -- 172.21.15.70:0/2762370236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559059342e10 msgr2=0x55905934d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:51.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.218+0000 7f64b659c640 1 --2- 172.21.15.70:0/2762370236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559059342e10 0x55905934d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.218+0000 7f64b659c640 1 -- 172.21.15.70:0/2762370236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933dca0 msgr2=0x55905934c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:34:51.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.218+0000 7f64b659c640 1 --2- 172.21.15.70:0/2762370236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933dca0 0x55905934c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.218+0000 7f64b659c640 1 -- 172.21.15.70:0/2762370236 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55905934d9e0 con 0x55905933d2b0 2024-04-01T01:34:51.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.219+0000 7f64b659c640 1 --2- 172.21.15.70:0/2762370236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55905933d2b0 0x55905934bef0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f64a809a700 tx=0x7f64a8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ea40bc7cd9e778 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:34:51.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.219+0000 7f649dffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:34:51.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.220+0000 7f649dffb640 1 -- 172.21.15.70:0/2762370236 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a80b7070 con 0x55905933d2b0 2024-04-01T01:34:51.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.220+0000 7f649dffb640 1 -- 172.21.15.70:0/2762370236 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f64a80b31f0 con 0x55905933d2b0 2024-04-01T01:34:51.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.221+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2762370236 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64ac02d050 con 0x55905933d2b0 2024-04-01T01:34:51.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.223+0000 7f649dffb640 1 -- 172.21.15.70:0/2762370236 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f64a8081050 con 0x55905933d2b0 2024-04-01T01:34:51.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.223+0000 7f64bf7ef740 10 client.16062 fetch_fsmap learned FSMap version 29 2024-04-01T01:34:51.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.223+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2762370236 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55905934ed60 con 0x55905933d2b0 2024-04-01T01:34:51.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.226+0000 7f649dffb640 1 -- 172.21.15.70:0/2762370236 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64a809e8c0 con 0x55905933d2b0 2024-04-01T01:34:51.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.227+0000 7f649dffb640 1 -- 172.21.15.70:0/2762370236 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f64a8099630 con 0x55905933d2b0 2024-04-01T01:34:51.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.227+0000 7f64bf7ef740 10 client.16062 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:34:51.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.227+0000 7f64bf7ef740 10 client.16062 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:34:51.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.227+0000 7f64bf7ef740 20 client.16062 populate_metadata read hostname 'smithi070' 2024-04-01T01:34:51.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.227+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/2762370236 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5590593b9c10 0x5590593bc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:34:51.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.227+0000 7f64bf7ef740 4 client.16062 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:34:51.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.228+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2762370236 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f64ac02d050 con 0x5590593b9c10 2024-04-01T01:34:51.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.228+0000 7f64b5d9b640 1 --2- 172.21.15.70:0/2762370236 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5590593b9c10 0x5590593bc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:34:51.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.231+0000 7f64b5d9b640 1 --2- 172.21.15.70:0/2762370236 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5590593b9c10 0x5590593bc000 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae179d4fe85628e4 server_cookie=c4934077ac91232b in_seq=0 out_seq=0 2024-04-01T01:34:51.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.231+0000 7f649dffb640 10 client.16062 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.234+0000 7f61bbfff640 1 -- 172.21.15.70:0/3788157778 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5556a5bc9120 con 0x5556a5c78aa0 2024-04-01T01:34:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.234+0000 7f61bbfff640 10 client.16056 handle_command_reply: tid=0 2024-04-01T01:34:51.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.234+0000 7f61e5b84740 1 client.16056 shutdown 2024-04-01T01:34:51.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.235+0000 7f649dffb640 1 -- 172.21.15.70:0/2762370236 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f64ac02d050 con 0x5590593b9c10 2024-04-01T01:34:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.235+0000 7f61e5b84740 1 -- 172.21.15.70:0/3788157778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c51a60 msgr2=0x5556a5c89c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.235+0000 7f61e5b84740 1 --2- 172.21.15.70:0/3788157778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c51a60 0x5556a5c89c40 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f61cc009b20 tx=0x7f61cc009b50 comp rx=0 tx=0).stop 2024-04-01T01:34:51.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.235+0000 7f649dffb640 10 client.16062 handle_command_reply: tid=0 2024-04-01T01:34:51.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.235+0000 7f64bf7ef740 1 client.16062 shutdown 2024-04-01T01:34:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.235+0000 7f61e5b84740 1 -- 172.21.15.70:0/3788157778 shutdown_connections 2024-04-01T01:34:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.235+0000 7f61e5b84740 1 --2- 172.21.15.70:0/3788157778 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5556a5c78aa0 0x5556a5ce16a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.235+0000 7f61e5b84740 1 --2- 172.21.15.70:0/3788157778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556a5c555b0 0x5556a5c8b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.235+0000 7f61bbfff640 0 client.16056 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.235+0000 7f61e5b84740 1 --2- 172.21.15.70:0/3788157778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556a5c52450 0x5556a5c8a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.235+0000 7f61e5b84740 1 --2- 172.21.15.70:0/3788157778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556a5c51a60 0x5556a5c89c40 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.235+0000 7f61e5b84740 1 -- 172.21.15.70:0/3788157778 >> 172.21.15.70:0/3788157778 conn(0x5556a5c4d050 msgr2=0x5556a5c56d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.235+0000 7f61e5b84740 1 -- 172.21.15.70:0/3788157778 shutdown_connections 2024-04-01T01:34:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.235+0000 7f61e5b84740 1 -- 172.21.15.70:0/3788157778 wait complete. 2024-04-01T01:34:51.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.235+0000 7f61e5b84740 20 client.16056 trim_cache size 0 max 16384 2024-04-01T01:34:51.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.236+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2762370236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55905933d2b0 msgr2=0x55905934bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.236+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/2762370236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55905933d2b0 0x55905934bef0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f64a809a700 tx=0x7f64a8000bc0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:34:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:34:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:34:51.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:34:51.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.236+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2762370236 shutdown_connections 2024-04-01T01:34:51.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.236+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/2762370236 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5590593b9c10 0x5590593bc000 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.236+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/2762370236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559059342e10 0x55905934d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.236+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/2762370236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55905933dca0 0x55905934c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.236+0000 7f64bf7ef740 1 --2- 172.21.15.70:0/2762370236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55905933d2b0 0x55905934bef0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.236+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2762370236 >> 172.21.15.70:0/2762370236 conn(0x559059338820 msgr2=0x559059337390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:51.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.236+0000 7f649dffb640 0 client.16062 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:34:51.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.236+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2762370236 shutdown_connections 2024-04-01T01:34:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.236+0000 7f61e1d3f640 1 -- 172.21.15.70:0/3369417944 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61c40a2590 msgr2=0x7f61c40a4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.236+0000 7f61e1d3f640 1 --2- 172.21.15.70:0/3369417944 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61c40a2590 0x7f61c40a4a50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x5556a5c52cb0 tx=0x7f61cc005bd0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.236+0000 7f64bf7ef740 1 -- 172.21.15.70:0/2762370236 wait complete. 2024-04-01T01:34:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.237+0000 7f61e1d3f640 1 -- 172.21.15.70:0/3369417944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61dc1020b0 msgr2=0x7f61dc19eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.237+0000 7f61e1d3f640 1 --2- 172.21.15.70:0/3369417944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61dc1020b0 0x7f61dc19eda0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f61d400e970 tx=0x7f61d400ee30 comp rx=0 tx=0).stop 2024-04-01T01:34:51.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.237+0000 7f64bf7ef740 20 client.16062 trim_cache size 0 max 16384 2024-04-01T01:34:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.237+0000 7f61e1d3f640 1 -- 172.21.15.70:0/3369417944 shutdown_connections 2024-04-01T01:34:51.242 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:34:51.242 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:34:51.243 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:34:51.243 INFO:teuthology.orchestra.run.smithi070.stdout: "831920f1-e9d6-46aa-850b-e8b740c5b23b": { 2024-04-01T01:34:51.243 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:34:51.243 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "831920f1-e9d6-46aa-850b-e8b740c5b23b", 2024-04-01T01:34:51.243 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:34:51.243 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:34:51.243 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:34:51.243 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:34:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.237+0000 7f61e1d3f640 1 --2- 172.21.15.70:0/3369417944 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61c40a2590 0x7f61c40a4a50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.237+0000 7f61e1d3f640 1 --2- 172.21.15.70:0/3369417944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61dc1020b0 0x7f61dc19eda0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.237+0000 7f61e1d3f640 1 --2- 172.21.15.70:0/3369417944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61dc1016f0 0x7f61dc197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.237+0000 7f61e1d3f640 1 --2- 172.21.15.70:0/3369417944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61dc1009e0 0x7f61dc1977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.237+0000 7f61e1d3f640 1 -- 172.21.15.70:0/3369417944 >> 172.21.15.70:0/3369417944 conn(0x7f61dc0fb1b0 msgr2=0x7f61dc104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.237+0000 7f61e1d3f640 1 -- 172.21.15.70:0/3369417944 shutdown_connections 2024-04-01T01:34:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:34:51.237+0000 7f61e1d3f640 1 -- 172.21.15.70:0/3369417944 wait complete. 2024-04-01T01:34:51.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.238+0000 7f64bcc07640 1 -- 172.21.15.70:0/3338422312 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64a00a2960 msgr2=0x7f64a00a4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.238+0000 7f64bcc07640 1 --2- 172.21.15.70:0/3338422312 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64a00a2960 0x7f64a00a4e20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f64ac022000 tx=0x7f64ac023000 comp rx=0 tx=0).stop 2024-04-01T01:34:51.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.238+0000 7f64bcc07640 1 -- 172.21.15.70:0/3338422312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8108750 msgr2=0x7f64b81a6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:34:51.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.238+0000 7f64bcc07640 1 --2- 172.21.15.70:0/3338422312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8108750 0x7f64b81a6810 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f64a800d950 tx=0x7f64a800de10 comp rx=0 tx=0).stop 2024-04-01T01:34:51.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.238+0000 7f64bcc07640 1 -- 172.21.15.70:0/3338422312 shutdown_connections 2024-04-01T01:34:51.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.238+0000 7f64bcc07640 1 --2- 172.21.15.70:0/3338422312 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64a00a2960 0x7f64a00a4e20 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.238+0000 7f64bcc07640 1 --2- 172.21.15.70:0/3338422312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64b810da70 0x7f64b81addc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.239+0000 7f64bcc07640 1 --2- 172.21.15.70:0/3338422312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64b810b0c0 0x7f64b81a6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.239+0000 7f64bcc07640 1 --2- 172.21.15.70:0/3338422312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b8108750 0x7f64b81a6810 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:34:51.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.239+0000 7f64bcc07640 1 -- 172.21.15.70:0/3338422312 >> 172.21.15.70:0/3338422312 conn(0x7f64b806e5b0 msgr2=0x7f64b8108b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:34:51.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.239+0000 7f64bcc07640 1 -- 172.21.15.70:0/3338422312 shutdown_connections 2024-04-01T01:34:51.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:34:51.239+0000 7f64bcc07640 1 -- 172.21.15.70:0/3338422312 wait complete. 2024-04-01T01:34:51.260 INFO:tasks.cephfs.filesystem:scrub status for tag:831920f1-e9d6-46aa-850b-e8b740c5b23b - {'path': '/', 'tag': '831920f1-e9d6-46aa-850b-e8b740c5b23b', 'options': 'recursive,force'} 2024-04-01T01:34:51.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 5.0 s, 1670.2 tps, lat 56.377 ms stddev 84.723 2024-04-01T01:34:51.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:51 smithi189 ceph-mon[35753]: pgmap v399: 129 pgs: 129 active+clean; 947 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 MiB/s wr, 48 op/s 2024-04-01T01:34:51.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:51 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:34:51.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:51 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:34:51.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:51 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:34:52.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:51 smithi070 ceph-mon[32179]: pgmap v399: 129 pgs: 129 active+clean; 947 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 MiB/s wr, 48 op/s 2024-04-01T01:34:52.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:51 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:34:52.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:51 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:34:52.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:51 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:34:52.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:51 smithi081 ceph-mon[35142]: pgmap v399: 129 pgs: 129 active+clean; 947 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 MiB/s wr, 48 op/s 2024-04-01T01:34:52.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:51 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:34:52.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:51 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:34:52.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:51 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:34:52.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:52 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:34:53.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:52 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:34:53.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:52 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:34:53.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:53 smithi189 ceph-mon[35753]: pgmap v400: 129 pgs: 129 active+clean; 930 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 43 MiB/s wr, 201 op/s 2024-04-01T01:34:54.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:53 smithi070 ceph-mon[32179]: pgmap v400: 129 pgs: 129 active+clean; 930 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 43 MiB/s wr, 201 op/s 2024-04-01T01:34:54.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:53 smithi081 ceph-mon[35142]: pgmap v400: 129 pgs: 129 active+clean; 930 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 43 MiB/s wr, 201 op/s 2024-04-01T01:34:54.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:54 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:34:54.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:54 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:54.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:54 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:54.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:54 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:54.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:54 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:55.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:54 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:34:55.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:54 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:55.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:54 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:55.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:54 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:55.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:54 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:55.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:54 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:34:55.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:54 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:55.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:54 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:55.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:54 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:55.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:54 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:55.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:55 smithi189 ceph-mon[35753]: pgmap v401: 129 pgs: 129 active+clean; 927 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 31 MiB/s wr, 247 op/s 2024-04-01T01:34:55.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:55.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:55 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:56.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:55 smithi070 ceph-mon[32179]: pgmap v401: 129 pgs: 129 active+clean; 927 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 31 MiB/s wr, 247 op/s 2024-04-01T01:34:56.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:56.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:55 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:56.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:55 smithi081 ceph-mon[35142]: pgmap v401: 129 pgs: 129 active+clean; 927 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 31 MiB/s wr, 247 op/s 2024-04-01T01:34:56.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:56.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:55 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 10.0 s, 1159.2 tps, lat 85.996 ms stddev 50.035 2024-04-01T01:34:56.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:34:56.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:34:56.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:56.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:56 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:34:57.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:34:57.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:34:57.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:57.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:56 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:34:57.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:34:57.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:34:57.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:34:57.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:56 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:34:57.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:57 smithi189 ceph-mon[35753]: pgmap v402: 129 pgs: 129 active+clean; 927 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 26 MiB/s wr, 404 op/s 2024-04-01T01:34:58.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:57 smithi070 ceph-mon[32179]: pgmap v402: 129 pgs: 129 active+clean; 927 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 26 MiB/s wr, 404 op/s 2024-04-01T01:34:58.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:57 smithi081 ceph-mon[35142]: pgmap v402: 129 pgs: 129 active+clean; 927 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 26 MiB/s wr, 404 op/s 2024-04-01T01:34:59.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:34:59 smithi189 ceph-mon[35753]: pgmap v403: 129 pgs: 129 active+clean; 927 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 20 MiB/s wr, 638 op/s 2024-04-01T01:35:00.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:34:59 smithi070 ceph-mon[32179]: pgmap v403: 129 pgs: 129 active+clean; 927 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 20 MiB/s wr, 638 op/s 2024-04-01T01:35:00.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:34:59 smithi081 ceph-mon[35142]: pgmap v403: 129 pgs: 129 active+clean; 927 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 20 MiB/s wr, 638 op/s 2024-04-01T01:35:01.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 15.0 s, 1312.9 tps, lat 76.096 ms stddev 23.700 2024-04-01T01:35:01.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:01 smithi189 ceph-mon[35753]: pgmap v404: 129 pgs: 129 active+clean; 927 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 11 MiB/s wr, 765 op/s 2024-04-01T01:35:02.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:01 smithi070 ceph-mon[32179]: pgmap v404: 129 pgs: 129 active+clean; 927 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 11 MiB/s wr, 765 op/s 2024-04-01T01:35:02.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:01 smithi081 ceph-mon[35142]: pgmap v404: 129 pgs: 129 active+clean; 927 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 11 MiB/s wr, 765 op/s 2024-04-01T01:35:03.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:02 smithi081 ceph-mon[35142]: pgmap v405: 129 pgs: 129 active+clean; 930 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-04-01T01:35:03.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:02 smithi189 ceph-mon[35753]: pgmap v405: 129 pgs: 129 active+clean; 930 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-04-01T01:35:03.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:02 smithi070 ceph-mon[32179]: pgmap v405: 129 pgs: 129 active+clean; 930 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-04-01T01:35:04.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.386+0000 7f345d5a0640 1 -- 172.21.15.70:0/3166422396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f345810ad30 msgr2=0x7f345810b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:04.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.386+0000 7f345d5a0640 1 --2- 172.21.15.70:0/3166422396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f345810ad30 0x7f345810b1b0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f344c009900 tx=0x7f344c02bc40 comp rx=0 tx=0).stop 2024-04-01T01:35:04.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.387+0000 7f345d5a0640 1 -- 172.21.15.70:0/3166422396 shutdown_connections 2024-04-01T01:35:04.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.387+0000 7f345d5a0640 1 --2- 172.21.15.70:0/3166422396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3458111e90 0x7f3458114280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.387+0000 7f345d5a0640 1 --2- 172.21.15.70:0/3166422396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f345810ad30 0x7f345810b1b0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.387+0000 7f345d5a0640 1 --2- 172.21.15.70:0/3166422396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3458073590 0x7f345810a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.387+0000 7f345d5a0640 1 -- 172.21.15.70:0/3166422396 >> 172.21.15.70:0/3166422396 conn(0x7f345806e780 msgr2=0x7f3458070bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:04.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.387+0000 7f345d5a0640 1 -- 172.21.15.70:0/3166422396 shutdown_connections 2024-04-01T01:35:04.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.387+0000 7f345d5a0640 1 -- 172.21.15.70:0/3166422396 wait complete. 2024-04-01T01:35:04.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.388+0000 7f345d5a0640 1 Processor -- start 2024-04-01T01:35:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.388+0000 7f345d5a0640 1 -- start start 2024-04-01T01:35:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.388+0000 7f345d5a0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345810ad30 0x7f345810f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.388+0000 7f345d5a0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3458111e90 0x7f345810f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.389+0000 7f345d5a0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f345810be50 0x7f345810c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.389+0000 7f345d5a0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3458117b60 con 0x7f3458111e90 2024-04-01T01:35:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.389+0000 7f345d5a0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f34581179e0 con 0x7f345810be50 2024-04-01T01:35:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.389+0000 7f345d5a0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3458117ce0 con 0x7f345810ad30 2024-04-01T01:35:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.389+0000 7f3457fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345810ad30 0x7f345810f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.389+0000 7f3457fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345810ad30 0x7f345810f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57436/0 (socket says 172.21.15.70:57436) 2024-04-01T01:35:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.389+0000 7f3457fff640 1 -- 172.21.15.70:0/1005737736 learned_addr learned my addr 172.21.15.70:0/1005737736 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.389+0000 7f345cd9f640 1 --2- 172.21.15.70:0/1005737736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f345810be50 0x7f345810c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.389+0000 7f34577fe640 1 --2- 172.21.15.70:0/1005737736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3458111e90 0x7f345810f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.389+0000 7f345cd9f640 1 -- 172.21.15.70:0/1005737736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345810ad30 msgr2=0x7f345810f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.389+0000 7f345cd9f640 1 --2- 172.21.15.70:0/1005737736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345810ad30 0x7f345810f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.389+0000 7f345cd9f640 1 -- 172.21.15.70:0/1005737736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3458111e90 msgr2=0x7f345810f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.389+0000 7f345cd9f640 1 --2- 172.21.15.70:0/1005737736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3458111e90 0x7f345810f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.389+0000 7f345cd9f640 1 -- 172.21.15.70:0/1005737736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34581ae6e0 con 0x7f345810be50 2024-04-01T01:35:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.389+0000 7f3457fff640 1 --2- 172.21.15.70:0/1005737736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345810ad30 0x7f345810f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.390+0000 7f345cd9f640 1 --2- 172.21.15.70:0/1005737736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f345810be50 0x7f345810c2e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3450009fd0 tx=0x7f345000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.390+0000 7f34557fa640 1 -- 172.21.15.70:0/1005737736 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345000ce20 con 0x7f345810be50 2024-04-01T01:35:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.390+0000 7f34557fa640 1 -- 172.21.15.70:0/1005737736 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3450005020 con 0x7f345810be50 2024-04-01T01:35:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.390+0000 7f34557fa640 1 -- 172.21.15.70:0/1005737736 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3450010690 con 0x7f345810be50 2024-04-01T01:35:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.390+0000 7f345d5a0640 1 -- 172.21.15.70:0/1005737736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34581ae970 con 0x7f345810be50 2024-04-01T01:35:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.390+0000 7f345d5a0640 1 -- 172.21.15.70:0/1005737736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34581aede0 con 0x7f345810be50 2024-04-01T01:35:04.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.393+0000 7f34557fa640 1 -- 172.21.15.70:0/1005737736 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f34500051d0 con 0x7f345810be50 2024-04-01T01:35:04.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.396+0000 7f34557fa640 1 --2- 172.21.15.70:0/1005737736 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f34400a25e0 0x7f34400a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.396+0000 7f34557fa640 1 -- 172.21.15.70:0/1005737736 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f34500cf1a0 con 0x7f345810be50 2024-04-01T01:35:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.397+0000 7f3457fff640 1 --2- 172.21.15.70:0/1005737736 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f34400a25e0 0x7f34400a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.397+0000 7f34613e9740 1 Processor -- start 2024-04-01T01:35:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.397+0000 7f34613e9740 1 -- start start 2024-04-01T01:35:04.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.398+0000 7f34613e9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a557fae0 0x55f4a557ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.398+0000 7f34613e9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a55804d0 0x55f4a5584fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.398+0000 7f34613e9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a5585640 0x55f4a5587ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.398+0000 7f34613e9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f4a54e38f0 con 0x55f4a557fae0 2024-04-01T01:35:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.398+0000 7f34613e9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f4a548c8d0 con 0x55f4a5585640 2024-04-01T01:35:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.398+0000 7f34577fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a55804d0 0x55f4a5584fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.398+0000 7f345cd9f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a5585640 0x55f4a5587ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.398+0000 7f345cd9f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a5585640 0x55f4a5587ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38202/0 (socket says 172.21.15.70:38202) 2024-04-01T01:35:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.398+0000 7f345cd9f640 1 -- 172.21.15.70:0/844663823 learned_addr learned my addr 172.21.15.70:0/844663823 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.398+0000 7f34613e9740 1 -- 172.21.15.70:0/844663823 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f4a54df990 con 0x55f4a55804d0 2024-04-01T01:35:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.398+0000 7f3457fff640 1 --2- 172.21.15.70:0/844663823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a557fae0 0x55f4a557ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.398+0000 7f3457fff640 1 --2- 172.21.15.70:0/1005737736 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f34400a25e0 0x7f34400a4aa0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f34480104e0 tx=0x7f344800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.399+0000 7f345cd9f640 1 -- 172.21.15.70:0/844663823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a55804d0 msgr2=0x55f4a5584fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.399+0000 7f345cd9f640 1 --2- 172.21.15.70:0/844663823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a55804d0 0x55f4a5584fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.399+0000 7f345cd9f640 1 -- 172.21.15.70:0/844663823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a557fae0 msgr2=0x55f4a557ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.399+0000 7f345cd9f640 1 --2- 172.21.15.70:0/844663823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a557fae0 0x55f4a557ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.399+0000 7f345cd9f640 1 -- 172.21.15.70:0/844663823 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4a51f8210 con 0x55f4a5585640 2024-04-01T01:35:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.399+0000 7f345cd9f640 1 --2- 172.21.15.70:0/844663823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a5585640 0x55f4a5587ad0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f345009b7c0 tx=0x7f345009c070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c71b98a22d18618d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:04.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.399+0000 7f343e7fc640 1 -- 172.21.15.70:0/844663823 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34500998d0 con 0x55f4a5585640 2024-04-01T01:35:04.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.399+0000 7f343e7fc640 1 -- 172.21.15.70:0/844663823 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3450099b30 con 0x55f4a5585640 2024-04-01T01:35:04.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.399+0000 7f343e7fc640 1 -- 172.21.15.70:0/844663823 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345009bcf0 con 0x55f4a5585640 2024-04-01T01:35:04.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.400+0000 7f34613e9740 1 -- 172.21.15.70:0/844663823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a5585640 msgr2=0x55f4a5587ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:04.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.400+0000 7f34613e9740 1 --2- 172.21.15.70:0/844663823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a5585640 0x55f4a5587ad0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f345009b7c0 tx=0x7f345009c070 comp rx=0 tx=0).stop 2024-04-01T01:35:04.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.400+0000 7f34613e9740 1 -- 172.21.15.70:0/844663823 shutdown_connections 2024-04-01T01:35:04.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.400+0000 7f34613e9740 1 --2- 172.21.15.70:0/844663823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a5585640 0x55f4a5587ad0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.400+0000 7f34613e9740 1 --2- 172.21.15.70:0/844663823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a55804d0 0x55f4a5584fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.400+0000 7f34613e9740 1 --2- 172.21.15.70:0/844663823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a557fae0 0x55f4a557ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.400+0000 7f34613e9740 1 -- 172.21.15.70:0/844663823 >> 172.21.15.70:0/844663823 conn(0x55f4a557b050 msgr2=0x55f4a557b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.400+0000 7f34613e9740 1 -- 172.21.15.70:0/844663823 shutdown_connections 2024-04-01T01:35:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.400+0000 7f34613e9740 1 -- 172.21.15.70:0/844663823 wait complete. 2024-04-01T01:35:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.401+0000 7f34613e9740 1 -- start start 2024-04-01T01:35:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.401+0000 7f34613e9740 1 Processor -- start 2024-04-01T01:35:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.401+0000 7f34613e9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 0x55f4a5589d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.401+0000 7f34613e9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a55804d0 0x55f4a558a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.401+0000 7f34613e9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a5585640 0x55f4a55912b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.401+0000 7f34613e9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f4a5588640 con 0x55f4a55804d0 2024-04-01T01:35:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.401+0000 7f34613e9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f4a5481f80 con 0x55f4a557fae0 2024-04-01T01:35:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.401+0000 7f34613e9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f4a55887c0 con 0x55f4a5585640 2024-04-01T01:35:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.401+0000 7f34613e9740 10 client.0 _ll_register_callbacks cb 0x55f4a54da290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.402+0000 7f34577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a55804d0 0x55f4a558a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.403 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.402+0000 7f345cd9f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a5585640 0x55f4a55912b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.402+0000 7f34577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a55804d0 0x55f4a558a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50328/0 (socket says 172.21.15.70:50328) 2024-04-01T01:35:04.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.402+0000 7f34577fe640 1 -- 172.21.15.70:0/3416155966 learned_addr learned my addr 172.21.15.70:0/3416155966 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:04.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.402+0000 7f3457fff640 1 --2- 172.21.15.70:0/3416155966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 0x55f4a5589d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.402+0000 7f345cd9f640 1 -- 172.21.15.70:0/3416155966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 msgr2=0x55f4a5589d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:04.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.402+0000 7f345cd9f640 1 --2- 172.21.15.70:0/3416155966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 0x55f4a5589d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.402+0000 7f345cd9f640 1 -- 172.21.15.70:0/3416155966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a55804d0 msgr2=0x55f4a558a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.402+0000 7f345cd9f640 1 --2- 172.21.15.70:0/3416155966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a55804d0 0x55f4a558a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.402+0000 7f345cd9f640 1 -- 172.21.15.70:0/3416155966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4a52c4620 con 0x55f4a5585640 2024-04-01T01:35:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.403+0000 7f345cd9f640 1 --2- 172.21.15.70:0/3416155966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a5585640 0x55f4a55912b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f34500a3d10 tx=0x7f3450099ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51102b77b4ec10ca server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.403+0000 7f34377fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:35:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.403+0000 7f34377fe640 1 -- 172.21.15.70:0/3416155966 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34500b8070 con 0x55f4a5585640 2024-04-01T01:35:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.403+0000 7f34377fe640 1 -- 172.21.15.70:0/3416155966 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f34500a3fc0 con 0x55f4a5585640 2024-04-01T01:35:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.403+0000 7f34613e9740 1 -- 172.21.15.70:0/3416155966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3450099b30 con 0x55f4a5585640 2024-04-01T01:35:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.403+0000 7f34377fe640 1 -- 172.21.15.70:0/3416155966 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f345009cb60 con 0x55f4a5585640 2024-04-01T01:35:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.404+0000 7f34613e9740 10 client.25813 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.404+0000 7f34613e9740 1 -- 172.21.15.70:0/3416155966 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f4a5588640 con 0x55f4a5585640 2024-04-01T01:35:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.404+0000 7f34377fe640 1 -- 172.21.15.70:0/3416155966 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3450099b30 con 0x55f4a5585640 2024-04-01T01:35:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.404+0000 7f34377fe640 1 -- 172.21.15.70:0/3416155966 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3450097580 con 0x55f4a5585640 2024-04-01T01:35:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.404+0000 7f34613e9740 10 client.25813 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.404+0000 7f34613e9740 10 client.25813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.404+0000 7f34613e9740 20 client.25813 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.404+0000 7f34613e9740 1 --2- 172.21.15.70:0/3416155966 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f4a5595540 0x55f4a5597930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.404+0000 7f34613e9740 4 client.25813 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.404+0000 7f34613e9740 1 -- 172.21.15.70:0/3416155966 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4a54f7120 con 0x55f4a5595540 2024-04-01T01:35:04.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.405+0000 7f3457fff640 1 --2- 172.21.15.70:0/3416155966 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f4a5595540 0x55f4a5597930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.406+0000 7f3457fff640 1 --2- 172.21.15.70:0/3416155966 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f4a5595540 0x55f4a5597930 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76d4365af12a0970 server_cookie=3b36a6a095114d94 in_seq=0 out_seq=0 2024-04-01T01:35:04.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.406+0000 7f34377fe640 10 client.25813 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:04.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.407+0000 7f34377fe640 1 -- 172.21.15.70:0/3416155966 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f4a54f7120 con 0x55f4a5595540 2024-04-01T01:35:04.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.407+0000 7f34377fe640 10 client.25813 handle_command_reply: tid=0 2024-04-01T01:35:04.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.407+0000 7f34613e9740 1 client.25813 shutdown 2024-04-01T01:35:04.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.408+0000 7f34613e9740 1 -- 172.21.15.70:0/3416155966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a5585640 msgr2=0x55f4a55912b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.408+0000 7f34613e9740 1 --2- 172.21.15.70:0/3416155966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a5585640 0x55f4a55912b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f34500a3d10 tx=0x7f3450099ce0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.408+0000 7f34613e9740 1 -- 172.21.15.70:0/3416155966 shutdown_connections 2024-04-01T01:35:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.408+0000 7f34613e9740 1 --2- 172.21.15.70:0/3416155966 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f4a5595540 0x55f4a5597930 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.408+0000 7f34613e9740 1 --2- 172.21.15.70:0/3416155966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a5585640 0x55f4a55912b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.408+0000 7f34613e9740 1 --2- 172.21.15.70:0/3416155966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a55804d0 0x55f4a558a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.408+0000 7f34377fe640 0 client.25813 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.408+0000 7f34613e9740 1 --2- 172.21.15.70:0/3416155966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 0x55f4a5589d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.408+0000 7f34613e9740 1 -- 172.21.15.70:0/3416155966 >> 172.21.15.70:0/3416155966 conn(0x55f4a557b050 msgr2=0x55f4a557a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.408+0000 7f34613e9740 1 -- 172.21.15.70:0/3416155966 shutdown_connections 2024-04-01T01:35:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.408+0000 7f34613e9740 1 -- 172.21.15.70:0/3416155966 wait complete. 2024-04-01T01:35:04.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.408+0000 7f34613e9740 20 client.25813 trim_cache size 0 max 16384 2024-04-01T01:35:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.435+0000 7f34613e9740 1 Processor -- start 2024-04-01T01:35:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.435+0000 7f34613e9740 1 -- start start 2024-04-01T01:35:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.436+0000 7f34613e9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 0x55f4a559d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.436+0000 7f34613e9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a55804d0 0x55f4a5591180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.436+0000 7f34613e9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a5585640 0x55f4a557e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.436+0000 7f34613e9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f34581b9040 con 0x55f4a5585640 2024-04-01T01:35:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.436+0000 7f34613e9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f34581bab80 con 0x55f4a557fae0 2024-04-01T01:35:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.436+0000 7f34613e9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f34581b9710 con 0x55f4a55804d0 2024-04-01T01:35:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.436+0000 7f3457fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 0x55f4a559d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.436+0000 7f3457fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 0x55f4a559d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38226/0 (socket says 172.21.15.70:38226) 2024-04-01T01:35:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.436+0000 7f3457fff640 1 -- 172.21.15.70:0/195633639 learned_addr learned my addr 172.21.15.70:0/195633639 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.436+0000 7f345cd9f640 1 --2- 172.21.15.70:0/195633639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a5585640 0x55f4a557e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.436+0000 7f34577fe640 1 --2- 172.21.15.70:0/195633639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a55804d0 0x55f4a5591180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.437+0000 7f3457fff640 1 -- 172.21.15.70:0/195633639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a55804d0 msgr2=0x55f4a5591180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.437+0000 7f3457fff640 1 --2- 172.21.15.70:0/195633639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a55804d0 0x55f4a5591180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.437+0000 7f3457fff640 1 -- 172.21.15.70:0/195633639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a5585640 msgr2=0x55f4a557e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.437+0000 7f3457fff640 1 --2- 172.21.15.70:0/195633639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a5585640 0x55f4a557e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.437+0000 7f3457fff640 1 -- 172.21.15.70:0/195633639 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4a557bdc0 con 0x55f4a557fae0 2024-04-01T01:35:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.437+0000 7f3457fff640 1 --2- 172.21.15.70:0/195633639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 0x55f4a559d1b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3448010d70 tx=0x7f3448013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=264be0fd66112ab6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.437+0000 7f343dffb640 1 -- 172.21.15.70:0/195633639 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344802ae60 con 0x55f4a557fae0 2024-04-01T01:35:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.437+0000 7f343dffb640 1 -- 172.21.15.70:0/195633639 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3448030070 con 0x55f4a557fae0 2024-04-01T01:35:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.437+0000 7f343dffb640 1 -- 172.21.15.70:0/195633639 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344801e8a0 con 0x55f4a557fae0 2024-04-01T01:35:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.438+0000 7f34613e9740 1 -- 172.21.15.70:0/195633639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 msgr2=0x55f4a559d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.438+0000 7f34613e9740 1 --2- 172.21.15.70:0/195633639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 0x55f4a559d1b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3448010d70 tx=0x7f3448013000 comp rx=0 tx=0).stop 2024-04-01T01:35:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.438+0000 7f34613e9740 1 -- 172.21.15.70:0/195633639 shutdown_connections 2024-04-01T01:35:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.438+0000 7f34613e9740 1 --2- 172.21.15.70:0/195633639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a5585640 0x55f4a557e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.438+0000 7f34613e9740 1 --2- 172.21.15.70:0/195633639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a55804d0 0x55f4a5591180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.438+0000 7f34613e9740 1 --2- 172.21.15.70:0/195633639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 0x55f4a559d1b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.438+0000 7f34613e9740 1 -- 172.21.15.70:0/195633639 >> 172.21.15.70:0/195633639 conn(0x55f4a557b050 msgr2=0x55f4a5584d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.439+0000 7f34613e9740 1 -- 172.21.15.70:0/195633639 shutdown_connections 2024-04-01T01:35:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.439+0000 7f34613e9740 1 -- 172.21.15.70:0/195633639 wait complete. 2024-04-01T01:35:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.439+0000 7f34613e9740 1 -- start start 2024-04-01T01:35:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.440+0000 7f34613e9740 1 Processor -- start 2024-04-01T01:35:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.440+0000 7f34613e9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 0x55f4a558e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.440+0000 7f34613e9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a55804d0 0x55f4a558ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.440+0000 7f34613e9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a5585640 0x55f4a558fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.440+0000 7f34613e9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f34581ba4b0 con 0x55f4a5585640 2024-04-01T01:35:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.440+0000 7f34613e9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f34581b9de0 con 0x55f4a557fae0 2024-04-01T01:35:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.440+0000 7f34613e9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f34581b7020 con 0x55f4a55804d0 2024-04-01T01:35:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.440+0000 7f34613e9740 10 client.0 _ll_register_callbacks cb 0x7f344800bcc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.441+0000 7f34577fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a55804d0 0x55f4a558ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.441+0000 7f3457fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 0x55f4a558e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.441+0000 7f3457fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 0x55f4a558e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38232/0 (socket says 172.21.15.70:38232) 2024-04-01T01:35:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.441+0000 7f3457fff640 1 -- 172.21.15.70:0/3436004919 learned_addr learned my addr 172.21.15.70:0/3436004919 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.441+0000 7f345cd9f640 1 --2- 172.21.15.70:0/3436004919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a5585640 0x55f4a558fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.441+0000 7f345cd9f640 1 -- 172.21.15.70:0/3436004919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a55804d0 msgr2=0x55f4a558ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.441+0000 7f345cd9f640 1 --2- 172.21.15.70:0/3436004919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a55804d0 0x55f4a558ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.441+0000 7f345cd9f640 1 -- 172.21.15.70:0/3436004919 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 msgr2=0x55f4a558e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.441+0000 7f345cd9f640 1 --2- 172.21.15.70:0/3436004919 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 0x55f4a558e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.442+0000 7f345cd9f640 1 -- 172.21.15.70:0/3436004919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4a55916c0 con 0x55f4a5585640 2024-04-01T01:35:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.442+0000 7f345cd9f640 1 --2- 172.21.15.70:0/3436004919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a5585640 0x55f4a558fc90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f3450098200 tx=0x7f3450005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2aef5d62e625655 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.442+0000 7f34377fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:35:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.442+0000 7f34377fe640 1 -- 172.21.15.70:0/3436004919 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34500b8070 con 0x55f4a5585640 2024-04-01T01:35:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.442+0000 7f34613e9740 1 -- 172.21.15.70:0/3436004919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3448030070 con 0x55f4a5585640 2024-04-01T01:35:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.442+0000 7f34377fe640 1 -- 172.21.15.70:0/3436004919 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3450097ea0 con 0x55f4a5585640 2024-04-01T01:35:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.443+0000 7f34377fe640 1 -- 172.21.15.70:0/3436004919 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f345009b600 con 0x55f4a5585640 2024-04-01T01:35:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.443+0000 7f34613e9740 10 client.16080 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.443+0000 7f34613e9740 1 -- 172.21.15.70:0/3436004919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f4a55913c0 con 0x55f4a5585640 2024-04-01T01:35:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.443+0000 7f34377fe640 1 -- 172.21.15.70:0/3436004919 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3448030070 con 0x55f4a5585640 2024-04-01T01:35:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.443+0000 7f34377fe640 1 -- 172.21.15.70:0/3436004919 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f34500afaf0 con 0x55f4a5585640 2024-04-01T01:35:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.443+0000 7f34613e9740 10 client.16080 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.443+0000 7f34613e9740 10 client.16080 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.443+0000 7f34613e9740 20 client.16080 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.443+0000 7f34613e9740 1 --2- 172.21.15.70:0/3436004919 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f4a55fbd10 0x55f4a55fe130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.443+0000 7f34613e9740 4 client.16080 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:35:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.443+0000 7f34613e9740 1 -- 172.21.15.70:0/3436004919 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f4a54f0e30 con 0x55f4a55fbd10 2024-04-01T01:35:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.444+0000 7f3457fff640 1 --2- 172.21.15.70:0/3436004919 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f4a55fbd10 0x55f4a55fe130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.445+0000 7f3457fff640 1 --2- 172.21.15.70:0/3436004919 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f4a55fbd10 0x55f4a55fe130 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b67abebf8f1d3fb server_cookie=73d40c607da2b473 in_seq=0 out_seq=0 2024-04-01T01:35:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.445+0000 7f34377fe640 10 client.16080 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.488+0000 7f34377fe640 1 -- 172.21.15.70:0/3436004919 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f4a54f0e30 con 0x55f4a55fbd10 2024-04-01T01:35:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.488+0000 7f34377fe640 10 client.16080 handle_command_reply: tid=0 2024-04-01T01:35:04.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.488+0000 7f34613e9740 1 client.16080 shutdown 2024-04-01T01:35:04.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.489+0000 7f34613e9740 1 -- 172.21.15.70:0/3436004919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a5585640 msgr2=0x55f4a558fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:04.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.489+0000 7f34613e9740 1 --2- 172.21.15.70:0/3436004919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a5585640 0x55f4a558fc90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f3450098200 tx=0x7f3450005880 comp rx=0 tx=0).stop 2024-04-01T01:35:04.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.489+0000 7f34613e9740 1 -- 172.21.15.70:0/3436004919 shutdown_connections 2024-04-01T01:35:04.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.489+0000 7f34613e9740 1 --2- 172.21.15.70:0/3436004919 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f4a55fbd10 0x55f4a55fe130 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.489+0000 7f34613e9740 1 --2- 172.21.15.70:0/3436004919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4a5585640 0x55f4a558fc90 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.489+0000 7f34613e9740 1 --2- 172.21.15.70:0/3436004919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4a55804d0 0x55f4a558ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.489+0000 7f34613e9740 1 --2- 172.21.15.70:0/3436004919 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4a557fae0 0x55f4a558e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.489+0000 7f34613e9740 1 -- 172.21.15.70:0/3436004919 >> 172.21.15.70:0/3436004919 conn(0x55f4a557b050 msgr2=0x55f4a5589cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:04.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.489+0000 7f34377fe640 0 client.16080 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:04.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.489+0000 7f34613e9740 1 -- 172.21.15.70:0/3436004919 shutdown_connections 2024-04-01T01:35:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.490+0000 7f34613e9740 1 -- 172.21.15.70:0/3436004919 wait complete. 2024-04-01T01:35:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.490+0000 7f34613e9740 20 client.16080 trim_cache size 0 max 16384 2024-04-01T01:35:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:35:04.491 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:35:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:35:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:35:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.492+0000 7f345d5a0640 1 -- 172.21.15.70:0/1005737736 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f34400a25e0 msgr2=0x7f34400a4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.492+0000 7f345d5a0640 1 --2- 172.21.15.70:0/1005737736 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f34400a25e0 0x7f34400a4aa0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f34480104e0 tx=0x7f344800b410 comp rx=0 tx=0).stop 2024-04-01T01:35:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.492+0000 7f345d5a0640 1 -- 172.21.15.70:0/1005737736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f345810be50 msgr2=0x7f345810c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:04.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.492+0000 7f345d5a0640 1 --2- 172.21.15.70:0/1005737736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f345810be50 0x7f345810c2e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3450009fd0 tx=0x7f345000ef10 comp rx=0 tx=0).stop 2024-04-01T01:35:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.493+0000 7f345d5a0640 1 -- 172.21.15.70:0/1005737736 shutdown_connections 2024-04-01T01:35:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.493+0000 7f345d5a0640 1 --2- 172.21.15.70:0/1005737736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f345810be50 0x7f345810c2e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.493+0000 7f345d5a0640 1 --2- 172.21.15.70:0/1005737736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3458111e90 0x7f345810f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.493+0000 7f345d5a0640 1 --2- 172.21.15.70:0/1005737736 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f34400a25e0 0x7f34400a4aa0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.493+0000 7f345d5a0640 1 --2- 172.21.15.70:0/1005737736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f345810ad30 0x7f345810f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:04.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.493+0000 7f345d5a0640 1 -- 172.21.15.70:0/1005737736 >> 172.21.15.70:0/1005737736 conn(0x7f345806e780 msgr2=0x7f3458108cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.494+0000 7f345d5a0640 1 -- 172.21.15.70:0/1005737736 shutdown_connections 2024-04-01T01:35:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:04.494+0000 7f345d5a0640 1 -- 172.21.15.70:0/1005737736 wait complete. 2024-04-01T01:35:05.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:05 smithi189 ceph-mon[35753]: pgmap v406: 129 pgs: 129 active+clean; 930 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 10 MiB/s wr, 1.08k op/s 2024-04-01T01:35:06.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:05 smithi070 ceph-mon[32179]: pgmap v406: 129 pgs: 129 active+clean; 930 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 10 MiB/s wr, 1.08k op/s 2024-04-01T01:35:06.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:05 smithi081 ceph-mon[35142]: pgmap v406: 129 pgs: 129 active+clean; 930 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 10 MiB/s wr, 1.08k op/s 2024-04-01T01:35:06.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 20.0 s, 1145.4 tps, lat 87.266 ms stddev 29.619 2024-04-01T01:35:07.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:07 smithi189 ceph-mon[35753]: pgmap v407: 129 pgs: 129 active+clean; 931 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 12 MiB/s wr, 1.32k op/s 2024-04-01T01:35:08.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:07 smithi070 ceph-mon[32179]: pgmap v407: 129 pgs: 129 active+clean; 931 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 12 MiB/s wr, 1.32k op/s 2024-04-01T01:35:08.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:07 smithi081 ceph-mon[35142]: pgmap v407: 129 pgs: 129 active+clean; 931 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 12 MiB/s wr, 1.32k op/s 2024-04-01T01:35:09.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:09 smithi189 ceph-mon[35753]: pgmap v408: 129 pgs: 129 active+clean; 931 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 12 MiB/s wr, 1.41k op/s 2024-04-01T01:35:10.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:09 smithi070 ceph-mon[32179]: pgmap v408: 129 pgs: 129 active+clean; 931 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 12 MiB/s wr, 1.41k op/s 2024-04-01T01:35:10.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:09 smithi081 ceph-mon[35142]: pgmap v408: 129 pgs: 129 active+clean; 931 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 12 MiB/s wr, 1.41k op/s 2024-04-01T01:35:11.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 25.0 s, 1144.4 tps, lat 83.546 ms stddev 21.594 2024-04-01T01:35:11.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:11 smithi189 ceph-mon[35753]: pgmap v409: 129 pgs: 129 active+clean; 931 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-04-01T01:35:12.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:11 smithi070 ceph-mon[32179]: pgmap v409: 129 pgs: 129 active+clean; 931 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-04-01T01:35:12.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:11 smithi081 ceph-mon[35142]: pgmap v409: 129 pgs: 129 active+clean; 931 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-04-01T01:35:13.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:13 smithi189 ceph-mon[35753]: pgmap v410: 129 pgs: 129 active+clean; 931 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-04-01T01:35:14.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:13 smithi070 ceph-mon[32179]: pgmap v410: 129 pgs: 129 active+clean; 931 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-04-01T01:35:14.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:13 smithi081 ceph-mon[35142]: pgmap v410: 129 pgs: 129 active+clean; 931 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 13 MiB/s wr, 1.52k op/s 2024-04-01T01:35:15.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:15 smithi189 ceph-mon[35753]: pgmap v411: 129 pgs: 129 active+clean; 931 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 11 MiB/s wr, 1.28k op/s 2024-04-01T01:35:16.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:15 smithi070 ceph-mon[32179]: pgmap v411: 129 pgs: 129 active+clean; 931 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 11 MiB/s wr, 1.28k op/s 2024-04-01T01:35:16.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:15 smithi081 ceph-mon[35142]: pgmap v411: 129 pgs: 129 active+clean; 931 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 11 MiB/s wr, 1.28k op/s 2024-04-01T01:35:16.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 30.0 s, 1184.6 tps, lat 88.165 ms stddev 84.598 2024-04-01T01:35:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.644+0000 7fac3ddac640 1 -- 172.21.15.70:0/2271237848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac381097a0 msgr2=0x7fac38110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.644+0000 7fac3ddac640 1 --2- 172.21.15.70:0/2271237848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac381097a0 0x7fac38110030 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fac30009ca0 tx=0x7fac3002c120 comp rx=0 tx=0).stop 2024-04-01T01:35:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.644+0000 7fac3ddac640 1 -- 172.21.15.70:0/2271237848 shutdown_connections 2024-04-01T01:35:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.645+0000 7fac3ddac640 1 --2- 172.21.15.70:0/2271237848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac381097a0 0x7fac38110030 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.645+0000 7fac3ddac640 1 --2- 172.21.15.70:0/2271237848 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac38108de0 0x7fac38109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.645+0000 7fac3ddac640 1 --2- 172.21.15.70:0/2271237848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac38107bf0 0x7fac38108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.645+0000 7fac3ddac640 1 -- 172.21.15.70:0/2271237848 >> 172.21.15.70:0/2271237848 conn(0x7fac38076b80 msgr2=0x7fac38078fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.645+0000 7fac3ddac640 1 -- 172.21.15.70:0/2271237848 shutdown_connections 2024-04-01T01:35:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.645+0000 7fac3ddac640 1 -- 172.21.15.70:0/2271237848 wait complete. 2024-04-01T01:35:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.646+0000 7fac3ddac640 1 Processor -- start 2024-04-01T01:35:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.646+0000 7fac3ddac640 1 -- start start 2024-04-01T01:35:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.646+0000 7fac3ddac640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac38107bf0 0x7fac3819dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.646+0000 7fac3ddac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac38108de0 0x7fac3819e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.646+0000 7fac3ddac640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac381097a0 0x7fac381a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.646+0000 7fac3ddac640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fac38113700 con 0x7fac38107bf0 2024-04-01T01:35:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.646+0000 7fac3ddac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fac38113580 con 0x7fac38108de0 2024-04-01T01:35:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.646+0000 7fac3ddac640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fac38113880 con 0x7fac381097a0 2024-04-01T01:35:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.646+0000 7fac3cdaa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac38107bf0 0x7fac3819dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.646+0000 7fac37fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac38108de0 0x7fac3819e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.646+0000 7fac37fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac38108de0 0x7fac3819e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35042/0 (socket says 172.21.15.70:35042) 2024-04-01T01:35:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.646+0000 7fac37fff640 1 -- 172.21.15.70:0/1401829957 learned_addr learned my addr 172.21.15.70:0/1401829957 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.646+0000 7fac3d5ab640 1 --2- 172.21.15.70:0/1401829957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac381097a0 0x7fac381a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.647+0000 7fac37fff640 1 -- 172.21.15.70:0/1401829957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac381097a0 msgr2=0x7fac381a52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.647+0000 7fac37fff640 1 --2- 172.21.15.70:0/1401829957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac381097a0 0x7fac381a52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.647+0000 7fac37fff640 1 -- 172.21.15.70:0/1401829957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac38107bf0 msgr2=0x7fac3819dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.647+0000 7fac37fff640 1 --2- 172.21.15.70:0/1401829957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac38107bf0 0x7fac3819dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.647+0000 7fac37fff640 1 -- 172.21.15.70:0/1401829957 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac381a5960 con 0x7fac38108de0 2024-04-01T01:35:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.647+0000 7fac3d5ab640 1 --2- 172.21.15.70:0/1401829957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac381097a0 0x7fac381a52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:35:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.647+0000 7fac37fff640 1 --2- 172.21.15.70:0/1401829957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac38108de0 0x7fac3819e280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fac2400e9f0 tx=0x7fac2400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.647+0000 7fac35ffb640 1 -- 172.21.15.70:0/1401829957 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2400cd80 con 0x7fac38108de0 2024-04-01T01:35:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.647+0000 7fac35ffb640 1 -- 172.21.15.70:0/1401829957 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac240108e0 con 0x7fac38108de0 2024-04-01T01:35:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.647+0000 7fac35ffb640 1 -- 172.21.15.70:0/1401829957 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac24018920 con 0x7fac38108de0 2024-04-01T01:35:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.648+0000 7fac3ddac640 1 -- 172.21.15.70:0/1401829957 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac381a5c50 con 0x7fac38108de0 2024-04-01T01:35:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.648+0000 7fac3ddac640 1 -- 172.21.15.70:0/1401829957 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac381a6090 con 0x7fac38108de0 2024-04-01T01:35:17.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.652+0000 7fac35ffb640 1 -- 172.21.15.70:0/1401829957 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fac24010a90 con 0x7fac38108de0 2024-04-01T01:35:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.653+0000 7fac35ffb640 1 --2- 172.21.15.70:0/1401829957 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac280a2590 0x7fac280a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.653+0000 7fac35ffb640 1 -- 172.21.15.70:0/1401829957 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fac240cee10 con 0x7fac38108de0 2024-04-01T01:35:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.653+0000 7fac3cdaa640 1 --2- 172.21.15.70:0/1401829957 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac280a2590 0x7fac280a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.653+0000 7fac41bf2740 1 Processor -- start 2024-04-01T01:35:17.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.654+0000 7fac41bf2740 1 -- start start 2024-04-01T01:35:17.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.654+0000 7fac41bf2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c2ae0 0x56186d2c2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.654+0000 7fac41bf2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c34d0 0x56186d2c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.654+0000 7fac41bf2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c8640 0x56186d2caad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.654+0000 7fac41bf2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56186d2268f0 con 0x56186d2c2ae0 2024-04-01T01:35:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.654+0000 7fac41bf2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56186d1cf8d0 con 0x56186d2c34d0 2024-04-01T01:35:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.654+0000 7fac41bf2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56186d222990 con 0x56186d2c8640 2024-04-01T01:35:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.654+0000 7fac3cdaa640 1 --2- 172.21.15.70:0/1401829957 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac280a2590 0x7fac280a4a50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fac2c005fd0 tx=0x7fac2c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.654+0000 7fac37fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c34d0 0x56186d2c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.654+0000 7fac37fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c34d0 0x56186d2c7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35056/0 (socket says 172.21.15.70:35056) 2024-04-01T01:35:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.654+0000 7fac37fff640 1 -- 172.21.15.70:0/2500907632 learned_addr learned my addr 172.21.15.70:0/2500907632 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.654+0000 7fac3cdaa640 1 --2- 172.21.15.70:0/2500907632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c2ae0 0x56186d2c2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.655+0000 7fac37fff640 1 -- 172.21.15.70:0/2500907632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c8640 msgr2=0x56186d2caad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.655+0000 7fac3d5ab640 1 --2- 172.21.15.70:0/2500907632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c8640 0x56186d2caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.655+0000 7fac37fff640 1 --2- 172.21.15.70:0/2500907632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c8640 0x56186d2caad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.655+0000 7fac37fff640 1 -- 172.21.15.70:0/2500907632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c2ae0 msgr2=0x56186d2c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.655+0000 7fac37fff640 1 --2- 172.21.15.70:0/2500907632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c2ae0 0x56186d2c2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.655+0000 7fac37fff640 1 -- 172.21.15.70:0/2500907632 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56186cf3b210 con 0x56186d2c34d0 2024-04-01T01:35:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.655+0000 7fac3cdaa640 1 --2- 172.21.15.70:0/2500907632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c2ae0 0x56186d2c2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:17.657 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.655+0000 7fac37fff640 1 --2- 172.21.15.70:0/2500907632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c34d0 0x56186d2c7fd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fac240a3980 tx=0x7fac24010de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89c9ff93b22eb404 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.656+0000 7fac22ffd640 1 -- 172.21.15.70:0/2500907632 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2409f5f0 con 0x56186d2c34d0 2024-04-01T01:35:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.656+0000 7fac22ffd640 1 -- 172.21.15.70:0/2500907632 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac240a3eb0 con 0x56186d2c34d0 2024-04-01T01:35:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.656+0000 7fac22ffd640 1 -- 172.21.15.70:0/2500907632 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2409f130 con 0x56186d2c34d0 2024-04-01T01:35:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.657+0000 7fac41bf2740 1 -- 172.21.15.70:0/2500907632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c34d0 msgr2=0x56186d2c7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.657+0000 7fac41bf2740 1 --2- 172.21.15.70:0/2500907632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c34d0 0x56186d2c7fd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fac240a3980 tx=0x7fac24010de0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.657+0000 7fac41bf2740 1 -- 172.21.15.70:0/2500907632 shutdown_connections 2024-04-01T01:35:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.657+0000 7fac41bf2740 1 --2- 172.21.15.70:0/2500907632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c8640 0x56186d2caad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.657+0000 7fac41bf2740 1 --2- 172.21.15.70:0/2500907632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c34d0 0x56186d2c7fd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.657+0000 7fac41bf2740 1 --2- 172.21.15.70:0/2500907632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c2ae0 0x56186d2c2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.657+0000 7fac41bf2740 1 -- 172.21.15.70:0/2500907632 >> 172.21.15.70:0/2500907632 conn(0x56186d2be050 msgr2=0x56186d2be450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.658+0000 7fac41bf2740 1 -- 172.21.15.70:0/2500907632 shutdown_connections 2024-04-01T01:35:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.658+0000 7fac41bf2740 1 -- 172.21.15.70:0/2500907632 wait complete. 2024-04-01T01:35:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.658+0000 7fac41bf2740 1 -- start start 2024-04-01T01:35:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.658+0000 7fac41bf2740 1 Processor -- start 2024-04-01T01:35:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.659+0000 7fac41bf2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.659+0000 7fac41bf2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c34d0 0x56186d2cd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.659+0000 7fac41bf2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c8640 0x56186d2d4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.659+0000 7fac41bf2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56186d2cb590 con 0x56186d2c34d0 2024-04-01T01:35:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.659+0000 7fac3cdaa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.660+0000 7fac3cdaa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35060/0 (socket says 172.21.15.70:35060) 2024-04-01T01:35:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.660+0000 7fac3cdaa640 1 -- 172.21.15.70:0/1076585894 learned_addr learned my addr 172.21.15.70:0/1076585894 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.660+0000 7fac3d5ab640 1 --2- 172.21.15.70:0/1076585894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c8640 0x56186d2d4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.660+0000 7fac41bf2740 1 -- 172.21.15.70:0/1076585894 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56186d1c4f80 con 0x56186d2c2ae0 2024-04-01T01:35:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.660+0000 7fac37fff640 1 --2- 172.21.15.70:0/1076585894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c34d0 0x56186d2cd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.660+0000 7fac41bf2740 1 -- 172.21.15.70:0/1076585894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56186d2cb710 con 0x56186d2c8640 2024-04-01T01:35:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.660+0000 7fac41bf2740 10 client.0 _ll_register_callbacks cb 0x56186d21d290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.660+0000 7fac3d5ab640 1 -- 172.21.15.70:0/1076585894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 msgr2=0x56186d2ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.660+0000 7fac3d5ab640 1 --2- 172.21.15.70:0/1076585894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.660+0000 7fac3d5ab640 1 -- 172.21.15.70:0/1076585894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c34d0 msgr2=0x56186d2cd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.660+0000 7fac3d5ab640 1 --2- 172.21.15.70:0/1076585894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c34d0 0x56186d2cd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.660+0000 7fac3d5ab640 1 -- 172.21.15.70:0/1076585894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56186d007620 con 0x56186d2c8640 2024-04-01T01:35:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.660+0000 7fac3cdaa640 1 --2- 172.21.15.70:0/1076585894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.660+0000 7fac37fff640 1 --2- 172.21.15.70:0/1076585894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c34d0 0x56186d2cd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.661+0000 7fac3d5ab640 1 --2- 172.21.15.70:0/1076585894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c8640 0x56186d2d4360 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fac30037040 tx=0x7fac30009a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55843fe2e4ecfeef server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.661+0000 7fac17fff640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:35:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.661+0000 7fac17fff640 1 -- 172.21.15.70:0/1076585894 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac30044070 con 0x56186d2c8640 2024-04-01T01:35:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.661+0000 7fac17fff640 1 -- 172.21.15.70:0/1076585894 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fac30003970 con 0x56186d2c8640 2024-04-01T01:35:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.662+0000 7fac41bf2740 1 -- 172.21.15.70:0/1076585894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac240a3eb0 con 0x56186d2c8640 2024-04-01T01:35:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.662+0000 7fac17fff640 1 -- 172.21.15.70:0/1076585894 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fac3004d050 con 0x56186d2c8640 2024-04-01T01:35:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.662+0000 7fac41bf2740 10 client.25828 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.662+0000 7fac41bf2740 1 -- 172.21.15.70:0/1076585894 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56186d2d7390 con 0x56186d2c8640 2024-04-01T01:35:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.663+0000 7fac17fff640 1 -- 172.21.15.70:0/1076585894 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac240a3eb0 con 0x56186d2c8640 2024-04-01T01:35:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.663+0000 7fac17fff640 1 -- 172.21.15.70:0/1076585894 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fac30047020 con 0x56186d2c8640 2024-04-01T01:35:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.664+0000 7fac41bf2740 10 client.25828 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.664+0000 7fac41bf2740 10 client.25828 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.664+0000 7fac41bf2740 20 client.25828 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.664+0000 7fac41bf2740 1 --2- 172.21.15.70:0/1076585894 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56186d2d8b20 0x56186d2daf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.664+0000 7fac41bf2740 4 client.25828 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.664+0000 7fac41bf2740 1 -- 172.21.15.70:0/1076585894 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56186d23a120 con 0x56186d2d8b20 2024-04-01T01:35:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.664+0000 7fac3cdaa640 1 --2- 172.21.15.70:0/1076585894 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56186d2d8b20 0x56186d2daf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.665+0000 7fac3cdaa640 1 --2- 172.21.15.70:0/1076585894 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56186d2d8b20 0x56186d2daf10 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bd1a63ad2aa9c3e server_cookie=f95b42c1478e2259 in_seq=0 out_seq=0 2024-04-01T01:35:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.665+0000 7fac17fff640 10 client.25828 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:17.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.666+0000 7fac17fff640 1 -- 172.21.15.70:0/1076585894 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56186d23a120 con 0x56186d2d8b20 2024-04-01T01:35:17.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.666+0000 7fac17fff640 10 client.25828 handle_command_reply: tid=0 2024-04-01T01:35:17.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.666+0000 7fac41bf2740 1 client.25828 shutdown 2024-04-01T01:35:17.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.667+0000 7fac41bf2740 1 -- 172.21.15.70:0/1076585894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c8640 msgr2=0x56186d2d4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:17.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.667+0000 7fac41bf2740 1 --2- 172.21.15.70:0/1076585894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c8640 0x56186d2d4360 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fac30037040 tx=0x7fac30009a10 comp rx=0 tx=0).stop 2024-04-01T01:35:17.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.667+0000 7fac41bf2740 1 -- 172.21.15.70:0/1076585894 shutdown_connections 2024-04-01T01:35:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.667+0000 7fac41bf2740 1 --2- 172.21.15.70:0/1076585894 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56186d2d8b20 0x56186d2daf10 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.667+0000 7fac41bf2740 1 --2- 172.21.15.70:0/1076585894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c8640 0x56186d2d4360 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.667+0000 7fac41bf2740 1 --2- 172.21.15.70:0/1076585894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c34d0 0x56186d2cd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.667+0000 7fac17fff640 0 client.25828 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.667+0000 7fac41bf2740 1 --2- 172.21.15.70:0/1076585894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.667+0000 7fac41bf2740 1 -- 172.21.15.70:0/1076585894 >> 172.21.15.70:0/1076585894 conn(0x56186d2be050 msgr2=0x56186d2bd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.667+0000 7fac41bf2740 1 -- 172.21.15.70:0/1076585894 shutdown_connections 2024-04-01T01:35:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.667+0000 7fac41bf2740 1 -- 172.21.15.70:0/1076585894 wait complete. 2024-04-01T01:35:17.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.667+0000 7fac41bf2740 20 client.25828 trim_cache size 0 max 16384 2024-04-01T01:35:17.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.696+0000 7fac41bf2740 1 Processor -- start 2024-04-01T01:35:17.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.696+0000 7fac41bf2740 1 -- start start 2024-04-01T01:35:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.696+0000 7fac41bf2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2caa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.696+0000 7fac41bf2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c34d0 0x56186d2d4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.696+0000 7fac3cdaa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.696+0000 7fac41bf2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c8640 0x56186d2ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.696+0000 7fac3cdaa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2caa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35072/0 (socket says 172.21.15.70:35072) 2024-04-01T01:35:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.696+0000 7fac3cdaa640 1 -- 172.21.15.70:0/2136401239 learned_addr learned my addr 172.21.15.70:0/2136401239 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.696+0000 7fac41bf2740 1 -- 172.21.15.70:0/2136401239 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fac381b07b0 con 0x56186d2c34d0 2024-04-01T01:35:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.696+0000 7fac41bf2740 1 -- 172.21.15.70:0/2136401239 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fac381b22f0 con 0x56186d2c2ae0 2024-04-01T01:35:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.696+0000 7fac41bf2740 1 -- 172.21.15.70:0/2136401239 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fac381b0e80 con 0x56186d2c8640 2024-04-01T01:35:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.697+0000 7fac3d5ab640 1 --2- 172.21.15.70:0/2136401239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c8640 0x56186d2ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.697+0000 7fac37fff640 1 --2- 172.21.15.70:0/2136401239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c34d0 0x56186d2d4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.697+0000 7fac3cdaa640 1 -- 172.21.15.70:0/2136401239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c8640 msgr2=0x56186d2ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.697+0000 7fac3cdaa640 1 --2- 172.21.15.70:0/2136401239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c8640 0x56186d2ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.697+0000 7fac3cdaa640 1 -- 172.21.15.70:0/2136401239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c34d0 msgr2=0x56186d2d4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.697+0000 7fac3cdaa640 1 --2- 172.21.15.70:0/2136401239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c34d0 0x56186d2d4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.697+0000 7fac3cdaa640 1 -- 172.21.15.70:0/2136401239 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56186d2db590 con 0x56186d2c2ae0 2024-04-01T01:35:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.697+0000 7fac3cdaa640 1 --2- 172.21.15.70:0/2136401239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2caa30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00b6e0 tx=0x7fac2c013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0d4e3320dd00b9f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.698+0000 7fac227fc640 1 -- 172.21.15.70:0/2136401239 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c030070 con 0x56186d2c2ae0 2024-04-01T01:35:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.698+0000 7fac227fc640 1 -- 172.21.15.70:0/2136401239 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac2c02ac40 con 0x56186d2c2ae0 2024-04-01T01:35:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.698+0000 7fac227fc640 1 -- 172.21.15.70:0/2136401239 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0119d0 con 0x56186d2c2ae0 2024-04-01T01:35:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.699+0000 7fac41bf2740 1 -- 172.21.15.70:0/2136401239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 msgr2=0x56186d2caa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.699+0000 7fac41bf2740 1 --2- 172.21.15.70:0/2136401239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2caa30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00b6e0 tx=0x7fac2c013000 comp rx=0 tx=0).stop 2024-04-01T01:35:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.699+0000 7fac41bf2740 1 -- 172.21.15.70:0/2136401239 shutdown_connections 2024-04-01T01:35:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.699+0000 7fac41bf2740 1 --2- 172.21.15.70:0/2136401239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c8640 0x56186d2ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.699+0000 7fac41bf2740 1 --2- 172.21.15.70:0/2136401239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c34d0 0x56186d2d4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.699+0000 7fac41bf2740 1 --2- 172.21.15.70:0/2136401239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2caa30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.699+0000 7fac41bf2740 1 -- 172.21.15.70:0/2136401239 >> 172.21.15.70:0/2136401239 conn(0x56186d2be050 msgr2=0x56186d2be430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.700+0000 7fac41bf2740 1 -- 172.21.15.70:0/2136401239 shutdown_connections 2024-04-01T01:35:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.700+0000 7fac41bf2740 1 -- 172.21.15.70:0/2136401239 wait complete. 2024-04-01T01:35:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.700+0000 7fac41bf2740 1 -- start start 2024-04-01T01:35:17.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.700+0000 7fac41bf2740 1 Processor -- start 2024-04-01T01:35:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.701+0000 7fac41bf2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2d1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.701+0000 7fac41bf2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c34d0 0x56186d2d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.701+0000 7fac3cdaa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2d1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.701+0000 7fac3cdaa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2d1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35086/0 (socket says 172.21.15.70:35086) 2024-04-01T01:35:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.701+0000 7fac3cdaa640 1 -- 172.21.15.70:0/4196158734 learned_addr learned my addr 172.21.15.70:0/4196158734 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.701+0000 7fac41bf2740 1 --2- 172.21.15.70:0/4196158734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c8640 0x56186d33ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.701+0000 7fac37fff640 1 --2- 172.21.15.70:0/4196158734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c34d0 0x56186d2d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.701+0000 7fac41bf2740 1 -- 172.21.15.70:0/4196158734 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fac381b1c20 con 0x56186d2c8640 2024-04-01T01:35:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.701+0000 7fac41bf2740 1 -- 172.21.15.70:0/4196158734 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fac381b1550 con 0x56186d2c2ae0 2024-04-01T01:35:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.701+0000 7fac41bf2740 1 -- 172.21.15.70:0/4196158734 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fac381ae790 con 0x56186d2c34d0 2024-04-01T01:35:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.701+0000 7fac41bf2740 10 client.0 _ll_register_callbacks cb 0x7fac30037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.701+0000 7fac3d5ab640 1 --2- 172.21.15.70:0/4196158734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c8640 0x56186d33ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.701+0000 7fac3cdaa640 1 -- 172.21.15.70:0/4196158734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c34d0 msgr2=0x56186d2d16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.702+0000 7fac3cdaa640 1 --2- 172.21.15.70:0/4196158734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c34d0 0x56186d2d16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.702+0000 7fac3cdaa640 1 -- 172.21.15.70:0/4196158734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c8640 msgr2=0x56186d33ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.702+0000 7fac3cdaa640 1 --2- 172.21.15.70:0/4196158734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c8640 0x56186d33ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.702+0000 7fac3cdaa640 1 -- 172.21.15.70:0/4196158734 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56186d2d1d70 con 0x56186d2c2ae0 2024-04-01T01:35:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.702+0000 7fac3d5ab640 1 --2- 172.21.15.70:0/4196158734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c8640 0x56186d33ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:35:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.702+0000 7fac41bf2740 1 -- 172.21.15.70:0/4196158734 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac2c02ac40 con 0x56186d2c2ae0 2024-04-01T01:35:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.702+0000 7fac3cdaa640 1 --2- 172.21.15.70:0/4196158734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2d1190 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0043f0 tx=0x7fac2c003d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7859dce3460407f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.702+0000 7fac17fff640 10 client.25925 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.703+0000 7fac17fff640 1 -- 172.21.15.70:0/4196158734 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c030070 con 0x56186d2c2ae0 2024-04-01T01:35:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.703+0000 7fac17fff640 1 -- 172.21.15.70:0/4196158734 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fac2c02a430 con 0x56186d2c2ae0 2024-04-01T01:35:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.703+0000 7fac17fff640 1 -- 172.21.15.70:0/4196158734 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fac2c02a670 con 0x56186d2c2ae0 2024-04-01T01:35:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.703+0000 7fac41bf2740 10 client.25925 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.703+0000 7fac41bf2740 1 -- 172.21.15.70:0/4196158734 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=29}) -- 0x56186d33e3d0 con 0x56186d2c2ae0 2024-04-01T01:35:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.705+0000 7fac17fff640 1 -- 172.21.15.70:0/4196158734 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fac2c03d030 con 0x56186d2c2ae0 2024-04-01T01:35:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.705+0000 7fac41bf2740 10 client.25925 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.705+0000 7fac41bf2740 10 client.25925 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.705+0000 7fac41bf2740 20 client.25925 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.705+0000 7fac41bf2740 1 --2- 172.21.15.70:0/4196158734 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56186d33ecd0 0x56186d3410f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.705+0000 7fac41bf2740 4 client.25925 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:35:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.705+0000 7fac41bf2740 1 -- 172.21.15.70:0/4196158734 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56186d233e30 con 0x56186d33ecd0 2024-04-01T01:35:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.705+0000 7fac37fff640 1 --2- 172.21.15.70:0/4196158734 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56186d33ecd0 0x56186d3410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.706+0000 7fac37fff640 1 --2- 172.21.15.70:0/4196158734 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56186d33ecd0 0x56186d3410f0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5aaf258c56d923bb server_cookie=47e491f63ec46373 in_seq=0 out_seq=0 2024-04-01T01:35:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.706+0000 7fac17fff640 10 client.25925 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.741+0000 7fac17fff640 1 -- 172.21.15.70:0/4196158734 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56186d233e30 con 0x56186d33ecd0 2024-04-01T01:35:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.741+0000 7fac17fff640 10 client.25925 handle_command_reply: tid=0 2024-04-01T01:35:17.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.741+0000 7fac41bf2740 1 client.25925 shutdown 2024-04-01T01:35:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.742+0000 7fac41bf2740 1 -- 172.21.15.70:0/4196158734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 msgr2=0x56186d2d1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.742+0000 7fac41bf2740 1 --2- 172.21.15.70:0/4196158734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2d1190 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0043f0 tx=0x7fac2c003d50 comp rx=0 tx=0).stop 2024-04-01T01:35:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.742+0000 7fac41bf2740 1 -- 172.21.15.70:0/4196158734 shutdown_connections 2024-04-01T01:35:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.742+0000 7fac41bf2740 1 --2- 172.21.15.70:0/4196158734 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56186d33ecd0 0x56186d3410f0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.742+0000 7fac41bf2740 1 --2- 172.21.15.70:0/4196158734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56186d2c8640 0x56186d33ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.742+0000 7fac17fff640 0 client.25925 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.742+0000 7fac41bf2740 1 --2- 172.21.15.70:0/4196158734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56186d2c34d0 0x56186d2d16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.742+0000 7fac41bf2740 1 --2- 172.21.15.70:0/4196158734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56186d2c2ae0 0x56186d2d1190 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.743+0000 7fac41bf2740 1 -- 172.21.15.70:0/4196158734 >> 172.21.15.70:0/4196158734 conn(0x56186d2be050 msgr2=0x56186d2ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.743+0000 7fac41bf2740 1 -- 172.21.15.70:0/4196158734 shutdown_connections 2024-04-01T01:35:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.744+0000 7fac41bf2740 1 -- 172.21.15.70:0/4196158734 wait complete. 2024-04-01T01:35:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.744+0000 7fac41bf2740 20 client.25925 trim_cache size 0 max 16384 2024-04-01T01:35:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:35:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:35:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:35:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:35:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.746+0000 7fac3ddac640 1 -- 172.21.15.70:0/1401829957 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac280a2590 msgr2=0x7fac280a4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.746+0000 7fac3ddac640 1 --2- 172.21.15.70:0/1401829957 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac280a2590 0x7fac280a4a50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fac2c005fd0 tx=0x7fac2c00b410 comp rx=0 tx=0).stop 2024-04-01T01:35:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.747+0000 7fac3ddac640 1 -- 172.21.15.70:0/1401829957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac38108de0 msgr2=0x7fac3819e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.747+0000 7fac3ddac640 1 --2- 172.21.15.70:0/1401829957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac38108de0 0x7fac3819e280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fac2400e9f0 tx=0x7fac2400eeb0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.748+0000 7fac3ddac640 1 -- 172.21.15.70:0/1401829957 shutdown_connections 2024-04-01T01:35:17.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.748+0000 7fac3ddac640 1 --2- 172.21.15.70:0/1401829957 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac280a2590 0x7fac280a4a50 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.748+0000 7fac3ddac640 1 --2- 172.21.15.70:0/1401829957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac381097a0 0x7fac381a52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.748+0000 7fac3ddac640 1 --2- 172.21.15.70:0/1401829957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac38108de0 0x7fac3819e280 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.748+0000 7fac3ddac640 1 --2- 172.21.15.70:0/1401829957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac38107bf0 0x7fac3819dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:17.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.748+0000 7fac3ddac640 1 -- 172.21.15.70:0/1401829957 >> 172.21.15.70:0/1401829957 conn(0x7fac38076b80 msgr2=0x7fac3810c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:17.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.750+0000 7fac3ddac640 1 -- 172.21.15.70:0/1401829957 shutdown_connections 2024-04-01T01:35:17.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:17.750+0000 7fac3ddac640 1 -- 172.21.15.70:0/1401829957 wait complete. 2024-04-01T01:35:17.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:17 smithi189 ceph-mon[35753]: pgmap v412: 129 pgs: 129 active+clean; 943 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 14 MiB/s wr, 1.49k op/s 2024-04-01T01:35:18.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:17 smithi070 ceph-mon[32179]: pgmap v412: 129 pgs: 129 active+clean; 943 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 14 MiB/s wr, 1.49k op/s 2024-04-01T01:35:18.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:17 smithi081 ceph-mon[35142]: pgmap v412: 129 pgs: 129 active+clean; 943 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 14 MiB/s wr, 1.49k op/s 2024-04-01T01:35:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:19 smithi189 ceph-mon[35753]: pgmap v413: 129 pgs: 129 active+clean; 943 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 13 MiB/s wr, 1.39k op/s 2024-04-01T01:35:20.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:19 smithi070 ceph-mon[32179]: pgmap v413: 129 pgs: 129 active+clean; 943 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 13 MiB/s wr, 1.39k op/s 2024-04-01T01:35:20.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:19 smithi081 ceph-mon[35142]: pgmap v413: 129 pgs: 129 active+clean; 943 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 13 MiB/s wr, 1.39k op/s 2024-04-01T01:35:21.261 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:35:21.261 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:35:21.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.437+0000 7f0ac67fc640 1 -- 172.21.15.70:0/1950181998 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0034d80 con 0x7f0ac81089c0 2024-04-01T01:35:21.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.438+0000 7f0acdf8c640 1 -- 172.21.15.70:0/1950181998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac81089c0 msgr2=0x7f0ac810adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:21.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.438+0000 7f0acdf8c640 1 --2- 172.21.15.70:0/1950181998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac81089c0 0x7f0ac810adb0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009860 tx=0x7f0ab002bb70 comp rx=0 tx=0).stop 2024-04-01T01:35:21.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.439+0000 7f0acdf8c640 1 -- 172.21.15.70:0/1950181998 shutdown_connections 2024-04-01T01:35:21.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.439+0000 7f0acdf8c640 1 --2- 172.21.15.70:0/1950181998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ac810de00 0x7f0ac81102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.439+0000 7f0acdf8c640 1 --2- 172.21.15.70:0/1950181998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ac810b380 0x7f0ac810d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.439+0000 7f0acdf8c640 1 --2- 172.21.15.70:0/1950181998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac81089c0 0x7f0ac810adb0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.439+0000 7f0acdf8c640 1 -- 172.21.15.70:0/1950181998 >> 172.21.15.70:0/1950181998 conn(0x7f0ac806e950 msgr2=0x7f0ac8070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:21.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.439+0000 7f0acdf8c640 1 -- 172.21.15.70:0/1950181998 shutdown_connections 2024-04-01T01:35:21.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.439+0000 7f0acdf8c640 1 -- 172.21.15.70:0/1950181998 wait complete. 2024-04-01T01:35:21.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.440+0000 7f0acdf8c640 1 Processor -- start 2024-04-01T01:35:21.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.441+0000 7f0acdf8c640 1 -- start start 2024-04-01T01:35:21.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.441+0000 7f0acdf8c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ac81089c0 0x7f0ac8119320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.441+0000 7f0acdf8c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ac810b380 0x7f0ac8119860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.441+0000 7f0acdf8c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac810de00 0x7f0ac81208d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.441+0000 7f0acdf8c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81139d0 con 0x7f0ac810de00 2024-04-01T01:35:21.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.441+0000 7f0acdf8c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac8113850 con 0x7f0ac810b380 2024-04-01T01:35:21.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.441+0000 7f0acdf8c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac8113b50 con 0x7f0ac81089c0 2024-04-01T01:35:21.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.441+0000 7f0ac77fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ac81089c0 0x7f0ac8119320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.441+0000 7f0ac7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac810de00 0x7f0ac81208d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.441+0000 7f0ac7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac810de00 0x7f0ac81208d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38690/0 (socket says 172.21.15.70:38690) 2024-04-01T01:35:21.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.441+0000 7f0ac7fff640 1 -- 172.21.15.70:0/1428001280 learned_addr learned my addr 172.21.15.70:0/1428001280 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:21.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.441+0000 7f0ac6ffd640 1 --2- 172.21.15.70:0/1428001280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ac810b380 0x7f0ac8119860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.442+0000 7f0ac7fff640 1 -- 172.21.15.70:0/1428001280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ac81089c0 msgr2=0x7f0ac8119320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:21.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.442+0000 7f0ac7fff640 1 --2- 172.21.15.70:0/1428001280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ac81089c0 0x7f0ac8119320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.442+0000 7f0ac7fff640 1 -- 172.21.15.70:0/1428001280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ac810b380 msgr2=0x7f0ac8119860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:21.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.442+0000 7f0ac7fff640 1 --2- 172.21.15.70:0/1428001280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ac810b380 0x7f0ac8119860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.442+0000 7f0ac7fff640 1 -- 172.21.15.70:0/1428001280 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac8120f40 con 0x7f0ac810de00 2024-04-01T01:35:21.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.442+0000 7f0ac7fff640 1 --2- 172.21.15.70:0/1428001280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac810de00 0x7f0ac81208d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000d750 tx=0x7f0ac000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:21.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.443+0000 7f0ac4ff9640 1 -- 172.21.15.70:0/1428001280 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0017070 con 0x7f0ac810de00 2024-04-01T01:35:21.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.443+0000 7f0ac4ff9640 1 -- 172.21.15.70:0/1428001280 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ac00054d0 con 0x7f0ac810de00 2024-04-01T01:35:21.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.443+0000 7f0acdf8c640 1 -- 172.21.15.70:0/1428001280 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac81b6f60 con 0x7f0ac810de00 2024-04-01T01:35:21.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.443+0000 7f0acdf8c640 1 -- 172.21.15.70:0/1428001280 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ac81b73d0 con 0x7f0ac810de00 2024-04-01T01:35:21.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.444+0000 7f0ac4ff9640 1 -- 172.21.15.70:0/1428001280 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0004c60 con 0x7f0ac810de00 2024-04-01T01:35:21.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.448+0000 7f0ad0b72740 1 Processor -- start 2024-04-01T01:35:21.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.448+0000 7f0ac4ff9640 1 -- 172.21.15.70:0/1428001280 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f0ac0004e00 con 0x7f0ac810de00 2024-04-01T01:35:21.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.448+0000 7f0ad0b72740 1 -- start start 2024-04-01T01:35:21.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.448+0000 7f0ad0b72740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c82b0 0x560f4a6c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.448+0000 7f0ad0b72740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6c8ca0 0x560f4a6cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.448+0000 7f0ad0b72740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6cde10 0x560f4a6d02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.448+0000 7f0ad0b72740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560f4a62d180 con 0x560f4a6c8ca0 2024-04-01T01:35:21.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.448+0000 7f0ad0b72740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560f4a5d5b70 con 0x560f4a6c82b0 2024-04-01T01:35:21.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.448+0000 7f0ad0b72740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560f4a629220 con 0x560f4a6cde10 2024-04-01T01:35:21.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.448+0000 7f0ac6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c82b0 0x560f4a6c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.448+0000 7f0ac6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c82b0 0x560f4a6c86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35104/0 (socket says 172.21.15.70:35104) 2024-04-01T01:35:21.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.448+0000 7f0ac6ffd640 1 -- 172.21.15.70:0/919497799 learned_addr learned my addr 172.21.15.70:0/919497799 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:21.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.448+0000 7f0ac6ffd640 1 --2- 172.21.15.70:0/919497799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6cde10 0x560f4a6d02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.449+0000 7f0ac77fe640 1 --2- 172.21.15.70:0/919497799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6c8ca0 0x560f4a6cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.449+0000 7f0ac6ffd640 1 -- 172.21.15.70:0/919497799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6cde10 msgr2=0x560f4a6d02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:21.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.449+0000 7f0ac6ffd640 1 --2- 172.21.15.70:0/919497799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6cde10 0x560f4a6d02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.449+0000 7f0ac6ffd640 1 -- 172.21.15.70:0/919497799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6c8ca0 msgr2=0x560f4a6cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:21.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.449+0000 7f0ac6ffd640 1 --2- 172.21.15.70:0/919497799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6c8ca0 0x560f4a6cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.449+0000 7f0ac6ffd640 1 -- 172.21.15.70:0/919497799 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f4a5e97c0 con 0x560f4a6c82b0 2024-04-01T01:35:21.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.449+0000 7f0ac4ff9640 1 --2- 172.21.15.70:0/1428001280 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ab40a26a0 0x7f0ab40a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.450+0000 7f0ac6ffd640 1 --2- 172.21.15.70:0/919497799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c82b0 0x560f4a6c86d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800ec90 tx=0x7f0ab8010040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e47a8adbc07e3272 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:21.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.450+0000 7f0ac4ff9640 1 -- 172.21.15.70:0/1428001280 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0ac00ce300 con 0x7f0ac810de00 2024-04-01T01:35:21.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.450+0000 7f0abdffb640 1 -- 172.21.15.70:0/919497799 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab800ee40 con 0x560f4a6c82b0 2024-04-01T01:35:21.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.450+0000 7f0abdffb640 1 -- 172.21.15.70:0/919497799 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ab800ce10 con 0x560f4a6c82b0 2024-04-01T01:35:21.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.450+0000 7f0ac77fe640 1 --2- 172.21.15.70:0/1428001280 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ab40a26a0 0x7f0ab40a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.451+0000 7f0abdffb640 1 -- 172.21.15.70:0/919497799 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab801c640 con 0x560f4a6c82b0 2024-04-01T01:35:21.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.452+0000 7f0ad0b72740 1 -- 172.21.15.70:0/919497799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c82b0 msgr2=0x560f4a6c86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:21.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.452+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/919497799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c82b0 0x560f4a6c86d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800ec90 tx=0x7f0ab8010040 comp rx=0 tx=0).stop 2024-04-01T01:35:21.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.452+0000 7f0ac77fe640 1 --2- 172.21.15.70:0/1428001280 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ab40a26a0 0x7f0ab40a4b60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0000c00 tx=0x7f0ab0034980 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:21.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.452+0000 7f0ad0b72740 1 -- 172.21.15.70:0/919497799 shutdown_connections 2024-04-01T01:35:21.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.452+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/919497799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6cde10 0x560f4a6d02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.452+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/919497799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6c8ca0 0x560f4a6cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.453+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/919497799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c82b0 0x560f4a6c86d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.453+0000 7f0ad0b72740 1 -- 172.21.15.70:0/919497799 >> 172.21.15.70:0/919497799 conn(0x560f4a6c3820 msgr2=0x560f4a6c3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:21.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.453+0000 7f0ad0b72740 1 -- 172.21.15.70:0/919497799 shutdown_connections 2024-04-01T01:35:21.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.454+0000 7f0ad0b72740 1 -- 172.21.15.70:0/919497799 wait complete. 2024-04-01T01:35:21.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.454+0000 7f0ad0b72740 1 -- start start 2024-04-01T01:35:21.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.454+0000 7f0ad0b72740 1 Processor -- start 2024-04-01T01:35:21.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.455+0000 7f0ad0b72740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c82b0 0x560f4a6d28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.455+0000 7f0ad0b72740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c8ca0 0x560f4a6d2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.455+0000 7f0ad0b72740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6cde10 0x560f4a6d9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.455+0000 7f0ac77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c82b0 0x560f4a6d28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.455+0000 7f0ac77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c82b0 0x560f4a6d28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35120/0 (socket says 172.21.15.70:35120) 2024-04-01T01:35:21.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.455+0000 7f0ac7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6cde10 0x560f4a6d9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.456+0000 7f0ac7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6cde10 0x560f4a6d9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38704/0 (socket says 172.21.15.70:38704) 2024-04-01T01:35:21.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.456+0000 7f0ac6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c8ca0 0x560f4a6d2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.456+0000 7f0ac77fe640 1 -- 172.21.15.70:0/2871054797 learned_addr learned my addr 172.21.15.70:0/2871054797 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:21.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.456+0000 7f0ad0b72740 1 -- 172.21.15.70:0/2871054797 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560f4a6d0db0 con 0x560f4a6cde10 2024-04-01T01:35:21.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.456+0000 7f0ad0b72740 1 -- 172.21.15.70:0/2871054797 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560f4a5cb4a0 con 0x560f4a6c82b0 2024-04-01T01:35:21.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.456+0000 7f0ad0b72740 1 -- 172.21.15.70:0/2871054797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560f4a6d0f30 con 0x560f4a6c8ca0 2024-04-01T01:35:21.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.456+0000 7f0ad0b72740 10 client.0 _ll_register_callbacks cb 0x560f4a5ced90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:21.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.458+0000 7f0ac77fe640 1 -- 172.21.15.70:0/2871054797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c8ca0 msgr2=0x560f4a6d2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:21.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.458+0000 7f0ac77fe640 1 --2- 172.21.15.70:0/2871054797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c8ca0 0x560f4a6d2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.458+0000 7f0ac77fe640 1 -- 172.21.15.70:0/2871054797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6cde10 msgr2=0x560f4a6d9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:21.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.458+0000 7f0ac77fe640 1 --2- 172.21.15.70:0/2871054797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6cde10 0x560f4a6d9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.458+0000 7f0ac77fe640 1 -- 172.21.15.70:0/2871054797 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f4a40b7d0 con 0x560f4a6c82b0 2024-04-01T01:35:21.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.459+0000 7f0aa6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:21.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.459+0000 7f0ac77fe640 1 --2- 172.21.15.70:0/2871054797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c82b0 0x560f4a6d28f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009860 tx=0x7f0ab002bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e69f869a4c017536 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:21.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.459+0000 7f0aa6ffd640 1 -- 172.21.15.70:0/2871054797 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab004e020 con 0x560f4a6c82b0 2024-04-01T01:35:21.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.459+0000 7f0aa6ffd640 1 -- 172.21.15.70:0/2871054797 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0ab0034670 con 0x560f4a6c82b0 2024-04-01T01:35:21.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.460+0000 7f0ad0b72740 1 -- 172.21.15.70:0/2871054797 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ab800ce10 con 0x560f4a6c82b0 2024-04-01T01:35:21.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.460+0000 7f0aa6ffd640 1 -- 172.21.15.70:0/2871054797 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0ab004e020 con 0x560f4a6c82b0 2024-04-01T01:35:21.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.461+0000 7f0ad0b72740 10 client.25937 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:21.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.461+0000 7f0ad0b72740 1 -- 172.21.15.70:0/2871054797 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560f4a3413f0 con 0x560f4a6c82b0 2024-04-01T01:35:21.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.461+0000 7f0aa6ffd640 1 -- 172.21.15.70:0/2871054797 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ab800ce10 con 0x560f4a6c82b0 2024-04-01T01:35:21.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.462+0000 7f0aa6ffd640 1 -- 172.21.15.70:0/2871054797 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0ab005b020 con 0x560f4a6c82b0 2024-04-01T01:35:21.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.462+0000 7f0ad0b72740 10 client.25937 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:21.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.462+0000 7f0ad0b72740 10 client.25937 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:21.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.462+0000 7f0ad0b72740 20 client.25937 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:21.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.462+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/2871054797 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560f4a6de7e0 0x560f4a6e0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.462+0000 7f0ad0b72740 4 client.25937 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:21.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.462+0000 7f0ad0b72740 1 -- 172.21.15.70:0/2871054797 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f4a63a6c0 con 0x560f4a6de7e0 2024-04-01T01:35:21.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.462+0000 7f0ac6ffd640 1 --2- 172.21.15.70:0/2871054797 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560f4a6de7e0 0x560f4a6e0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.463+0000 7f0ac6ffd640 1 --2- 172.21.15.70:0/2871054797 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560f4a6de7e0 0x560f4a6e0c30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e94d246669f0ba93 server_cookie=116cf28f499ddf82 in_seq=0 out_seq=0 2024-04-01T01:35:21.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.463+0000 7f0aa6ffd640 10 client.25937 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.465+0000 7f0aa6ffd640 1 -- 172.21.15.70:0/2871054797 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560f4a63a6c0 con 0x560f4a6de7e0 2024-04-01T01:35:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.465+0000 7f0aa6ffd640 10 client.25937 handle_command_reply: tid=0 2024-04-01T01:35:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.465+0000 7f0ad0b72740 1 client.25937 shutdown 2024-04-01T01:35:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.465+0000 7f0ad0b72740 1 -- 172.21.15.70:0/2871054797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c82b0 msgr2=0x560f4a6d28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.465+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/2871054797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c82b0 0x560f4a6d28f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009860 tx=0x7f0ab002bb70 comp rx=0 tx=0).stop 2024-04-01T01:35:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.465+0000 7f0ad0b72740 1 -- 172.21.15.70:0/2871054797 shutdown_connections 2024-04-01T01:35:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.465+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/2871054797 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560f4a6de7e0 0x560f4a6e0c30 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.465+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/2871054797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6cde10 0x560f4a6d9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.465+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/2871054797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c8ca0 0x560f4a6d2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.465+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/2871054797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c82b0 0x560f4a6d28f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.465+0000 7f0ad0b72740 1 -- 172.21.15.70:0/2871054797 >> 172.21.15.70:0/2871054797 conn(0x560f4a6c3820 msgr2=0x560f4a6c2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:21.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.465+0000 7f0aa6ffd640 0 client.25937 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:21.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.466+0000 7f0ad0b72740 1 -- 172.21.15.70:0/2871054797 shutdown_connections 2024-04-01T01:35:21.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.466+0000 7f0ad0b72740 1 -- 172.21.15.70:0/2871054797 wait complete. 2024-04-01T01:35:21.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.466+0000 7f0ad0b72740 20 client.25937 trim_cache size 0 max 16384 2024-04-01T01:35:21.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.504+0000 7f0ad0b72740 1 Processor -- start 2024-04-01T01:35:21.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.504+0000 7f0ad0b72740 1 -- start start 2024-04-01T01:35:21.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.504+0000 7f0ad0b72740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c82b0 0x560f4a6d2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.504+0000 7f0ad0b72740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c8ca0 0x560f4a6d9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.504+0000 7f0ad0b72740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6cde10 0x560f4a6c2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.504+0000 7f0ad0b72740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81c1a80 con 0x560f4a6cde10 2024-04-01T01:35:21.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.504+0000 7f0ad0b72740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81c35c0 con 0x560f4a6c8ca0 2024-04-01T01:35:21.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.504+0000 7f0ad0b72740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81c2150 con 0x560f4a6c82b0 2024-04-01T01:35:21.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.505+0000 7f0ac77fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c82b0 0x560f4a6d2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.505+0000 7f0ac77fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c82b0 0x560f4a6d2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37606/0 (socket says 172.21.15.70:37606) 2024-04-01T01:35:21.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.505+0000 7f0ac77fe640 1 -- 172.21.15.70:0/1709840863 learned_addr learned my addr 172.21.15.70:0/1709840863 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:21.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.505+0000 7f0ac7fff640 1 --2- 172.21.15.70:0/1709840863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6cde10 0x560f4a6c2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.505+0000 7f0ac6ffd640 1 --2- 172.21.15.70:0/1709840863 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c8ca0 0x560f4a6d9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.505+0000 7f0ac77fe640 1 -- 172.21.15.70:0/1709840863 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c8ca0 msgr2=0x560f4a6d9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:21.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.505+0000 7f0ac77fe640 1 --2- 172.21.15.70:0/1709840863 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c8ca0 0x560f4a6d9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.505+0000 7f0ac77fe640 1 -- 172.21.15.70:0/1709840863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6cde10 msgr2=0x560f4a6c2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:21.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.505+0000 7f0ac77fe640 1 --2- 172.21.15.70:0/1709840863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6cde10 0x560f4a6c2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.505+0000 7f0ac77fe640 1 -- 172.21.15.70:0/1709840863 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f4a3010f0 con 0x560f4a6c82b0 2024-04-01T01:35:21.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.505+0000 7f0ac77fe640 1 --2- 172.21.15.70:0/1709840863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c82b0 0x560f4a6d2d90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009830 tx=0x7f0ab0034440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11813eb932752015 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:21.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.506+0000 7f0abd7fa640 1 -- 172.21.15.70:0/1709840863 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0043070 con 0x560f4a6c82b0 2024-04-01T01:35:21.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.506+0000 7f0abd7fa640 1 -- 172.21.15.70:0/1709840863 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ab0060020 con 0x560f4a6c82b0 2024-04-01T01:35:21.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.506+0000 7f0abd7fa640 1 -- 172.21.15.70:0/1709840863 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0039040 con 0x560f4a6c82b0 2024-04-01T01:35:21.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.507+0000 7f0ad0b72740 1 -- 172.21.15.70:0/1709840863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c82b0 msgr2=0x560f4a6d2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:21.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.507+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/1709840863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c82b0 0x560f4a6d2d90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009830 tx=0x7f0ab0034440 comp rx=0 tx=0).stop 2024-04-01T01:35:21.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.507+0000 7f0ad0b72740 1 -- 172.21.15.70:0/1709840863 shutdown_connections 2024-04-01T01:35:21.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.507+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/1709840863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6cde10 0x560f4a6c2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.507+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/1709840863 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6c8ca0 0x560f4a6d9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.507+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/1709840863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c82b0 0x560f4a6d2d90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.507+0000 7f0ad0b72740 1 -- 172.21.15.70:0/1709840863 >> 172.21.15.70:0/1709840863 conn(0x560f4a6c3820 msgr2=0x560f4a6c2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:21.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.510+0000 7f0ad0b72740 1 -- 172.21.15.70:0/1709840863 shutdown_connections 2024-04-01T01:35:21.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.510+0000 7f0ad0b72740 1 -- 172.21.15.70:0/1709840863 wait complete. 2024-04-01T01:35:21.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.510+0000 7f0ad0b72740 1 -- start start 2024-04-01T01:35:21.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.511+0000 7f0ad0b72740 1 Processor -- start 2024-04-01T01:35:21.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.511+0000 7f0ad0b72740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6c82b0 0x560f4a6d74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.511+0000 7f0ad0b72740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c8ca0 0x560f4a6d7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.511+0000 7f0ad0b72740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6cde10 0x560f4a6d8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.511+0000 7f0ad0b72740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81c2ef0 con 0x560f4a6c82b0 2024-04-01T01:35:21.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.512+0000 7f0ad0b72740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81c2820 con 0x560f4a6cde10 2024-04-01T01:35:21.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.512+0000 7f0ad0b72740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81bfa60 con 0x560f4a6c8ca0 2024-04-01T01:35:21.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.512+0000 7f0ad0b72740 10 client.0 _ll_register_callbacks cb 0x7f0ab8009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:21.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.512+0000 7f0ac6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c8ca0 0x560f4a6d7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.512+0000 7f0ac6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c8ca0 0x560f4a6d7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37616/0 (socket says 172.21.15.70:37616) 2024-04-01T01:35:21.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.512+0000 7f0ac6ffd640 1 -- 172.21.15.70:0/3030791048 learned_addr learned my addr 172.21.15.70:0/3030791048 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:21.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.512+0000 7f0ac77fe640 1 --2- 172.21.15.70:0/3030791048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6c82b0 0x560f4a6d74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.512+0000 7f0ac7fff640 1 --2- 172.21.15.70:0/3030791048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6cde10 0x560f4a6d8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.512+0000 7f0ac6ffd640 1 -- 172.21.15.70:0/3030791048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6cde10 msgr2=0x560f4a6d8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:21.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.512+0000 7f0ac6ffd640 1 --2- 172.21.15.70:0/3030791048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6cde10 0x560f4a6d8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.512+0000 7f0ac6ffd640 1 -- 172.21.15.70:0/3030791048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6c82b0 msgr2=0x560f4a6d74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:21.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.512+0000 7f0ac6ffd640 1 --2- 172.21.15.70:0/3030791048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6c82b0 0x560f4a6d74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.512+0000 7f0ac6ffd640 1 -- 172.21.15.70:0/3030791048 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f4a6d9020 con 0x560f4a6c8ca0 2024-04-01T01:35:21.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.513+0000 7f0ac6ffd640 1 --2- 172.21.15.70:0/3030791048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c8ca0 0x560f4a6d7a20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8010010 tx=0x7f0ab8009d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12e5cab90d299983 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:21.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.513+0000 7f0aa6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:35:21.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.513+0000 7f0aa6ffd640 1 -- 172.21.15.70:0/3030791048 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8009280 con 0x560f4a6c8ca0 2024-04-01T01:35:21.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.513+0000 7f0aa6ffd640 1 -- 172.21.15.70:0/3030791048 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0ab800ce10 con 0x560f4a6c8ca0 2024-04-01T01:35:21.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.513+0000 7f0ad0b72740 1 -- 172.21.15.70:0/3030791048 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ab0060020 con 0x560f4a6c8ca0 2024-04-01T01:35:21.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.514+0000 7f0aa6ffd640 1 -- 172.21.15.70:0/3030791048 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0ab8040050 con 0x560f4a6c8ca0 2024-04-01T01:35:21.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.514+0000 7f0ad0b72740 10 client.25852 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:21.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.514+0000 7f0ad0b72740 1 -- 172.21.15.70:0/3030791048 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560f4a744b30 con 0x560f4a6c8ca0 2024-04-01T01:35:21.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.515+0000 7f0aa6ffd640 1 -- 172.21.15.70:0/3030791048 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ab8003970 con 0x560f4a6c8ca0 2024-04-01T01:35:21.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.515+0000 7f0aa6ffd640 1 -- 172.21.15.70:0/3030791048 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0ab800b040 con 0x560f4a6c8ca0 2024-04-01T01:35:21.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.515+0000 7f0ad0b72740 10 client.25852 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:21.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.515+0000 7f0ad0b72740 10 client.25852 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:21.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.515+0000 7f0ad0b72740 20 client.25852 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:21.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.515+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/3030791048 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560f4a7451d0 0x560f4a7475f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.515+0000 7f0ad0b72740 4 client.25852 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:35:21.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.515+0000 7f0ad0b72740 1 -- 172.21.15.70:0/3030791048 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f0ab0060020 con 0x560f4a7451d0 2024-04-01T01:35:21.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.515+0000 7f0ac77fe640 1 --2- 172.21.15.70:0/3030791048 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560f4a7451d0 0x560f4a7475f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.516+0000 7f0ac77fe640 1 --2- 172.21.15.70:0/3030791048 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560f4a7451d0 0x560f4a7475f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42912d9c884f8f9e server_cookie=aa6d13419464d170 in_seq=0 out_seq=0 2024-04-01T01:35:21.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.516+0000 7f0aa6ffd640 10 client.25852 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:21.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.516+0000 7f0aa6ffd640 1 -- 172.21.15.70:0/3030791048 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f0ab0060020 con 0x560f4a7451d0 2024-04-01T01:35:21.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.516+0000 7f0aa6ffd640 10 client.25852 handle_command_reply: tid=0 2024-04-01T01:35:21.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.516+0000 7f0ad0b72740 1 client.25852 shutdown 2024-04-01T01:35:21.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.517+0000 7f0ad0b72740 1 -- 172.21.15.70:0/3030791048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c8ca0 msgr2=0x560f4a6d7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:21.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.517+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/3030791048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c8ca0 0x560f4a6d7a20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8010010 tx=0x7f0ab8009d00 comp rx=0 tx=0).stop 2024-04-01T01:35:21.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.517+0000 7f0ad0b72740 1 -- 172.21.15.70:0/3030791048 shutdown_connections 2024-04-01T01:35:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.517+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/3030791048 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560f4a7451d0 0x560f4a7475f0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.517+0000 7f0aa6ffd640 0 client.25852 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.517+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/3030791048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4a6cde10 0x560f4a6d8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.517+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/3030791048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4a6c8ca0 0x560f4a6d7a20 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8010010 tx=0x7f0ab8009d00 comp rx=0 tx=0).stop 2024-04-01T01:35:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.517+0000 7f0ad0b72740 1 --2- 172.21.15.70:0/3030791048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4a6c82b0 0x560f4a6d74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.517+0000 7f0ad0b72740 1 -- 172.21.15.70:0/3030791048 >> 172.21.15.70:0/3030791048 conn(0x560f4a6c3820 msgr2=0x560f4a6c2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.517+0000 7f0ad0b72740 1 -- 172.21.15.70:0/3030791048 shutdown_connections 2024-04-01T01:35:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.518+0000 7f0ad0b72740 1 -- 172.21.15.70:0/3030791048 wait complete. 2024-04-01T01:35:21.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.518+0000 7f0ad0b72740 20 client.25852 trim_cache size 0 max 16384 2024-04-01T01:35:21.520 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:35:21.520 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:35:21.520 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:35:21.520 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:35:21.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.519+0000 7f0acdf8c640 1 -- 172.21.15.70:0/1428001280 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ab40a26a0 msgr2=0x7f0ab40a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:21.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.519+0000 7f0acdf8c640 1 --2- 172.21.15.70:0/1428001280 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ab40a26a0 0x7f0ab40a4b60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0000c00 tx=0x7f0ab0034980 comp rx=0 tx=0).stop 2024-04-01T01:35:21.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.519+0000 7f0acdf8c640 1 -- 172.21.15.70:0/1428001280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac810de00 msgr2=0x7f0ac81208d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:21.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.519+0000 7f0acdf8c640 1 --2- 172.21.15.70:0/1428001280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac810de00 0x7f0ac81208d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000d750 tx=0x7f0ac000dc10 comp rx=0 tx=0).stop 2024-04-01T01:35:21.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.519+0000 7f0acdf8c640 1 -- 172.21.15.70:0/1428001280 shutdown_connections 2024-04-01T01:35:21.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.519+0000 7f0acdf8c640 1 --2- 172.21.15.70:0/1428001280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac810de00 0x7f0ac81208d0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.519+0000 7f0acdf8c640 1 --2- 172.21.15.70:0/1428001280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ac810b380 0x7f0ac8119860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.519+0000 7f0acdf8c640 1 --2- 172.21.15.70:0/1428001280 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ab40a26a0 0x7f0ab40a4b60 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.519+0000 7f0acdf8c640 1 --2- 172.21.15.70:0/1428001280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ac81089c0 0x7f0ac8119320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.519+0000 7f0acdf8c640 1 -- 172.21.15.70:0/1428001280 >> 172.21.15.70:0/1428001280 conn(0x7f0ac806e950 msgr2=0x7f0ac8109440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.519+0000 7f0acdf8c640 1 -- 172.21.15.70:0/1428001280 shutdown_connections 2024-04-01T01:35:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.519+0000 7f0acdf8c640 1 -- 172.21.15.70:0/1428001280 wait complete. 2024-04-01T01:35:21.538 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:35:21.538 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:35:21.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.716+0000 7f6f95ffb640 1 -- 172.21.15.70:0/858066385 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f900328c0 con 0x7f6f98077cf0 2024-04-01T01:35:21.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.716+0000 7f6f9d2c9640 1 -- 172.21.15.70:0/858066385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f98077cf0 msgr2=0x7f6f98078590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:21.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.716+0000 7f6f9d2c9640 1 --2- 172.21.15.70:0/858066385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f98077cf0 0x7f6f98078590 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6f90009ca0 tx=0x7f6f9002c100 comp rx=0 tx=0).stop 2024-04-01T01:35:21.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.717+0000 7f6f9d2c9640 1 -- 172.21.15.70:0/858066385 shutdown_connections 2024-04-01T01:35:21.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.717+0000 7f6f9d2c9640 1 --2- 172.21.15.70:0/858066385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f98077cf0 0x7f6f98078590 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.717+0000 7f6f9d2c9640 1 --2- 172.21.15.70:0/858066385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f98077350 0x7f6f980777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.717+0000 7f6f9d2c9640 1 --2- 172.21.15.70:0/858066385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f98076110 0x7f6f98076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.717+0000 7f6f9d2c9640 1 -- 172.21.15.70:0/858066385 >> 172.21.15.70:0/858066385 conn(0x7f6f980fef00 msgr2=0x7f6f98101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:21.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.717+0000 7f6f9d2c9640 1 -- 172.21.15.70:0/858066385 shutdown_connections 2024-04-01T01:35:21.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.717+0000 7f6f9d2c9640 1 -- 172.21.15.70:0/858066385 wait complete. 2024-04-01T01:35:21.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.718+0000 7f6f9d2c9640 1 Processor -- start 2024-04-01T01:35:21.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.718+0000 7f6f9d2c9640 1 -- start start 2024-04-01T01:35:21.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.718+0000 7f6f9d2c9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f98076110 0x7f6f9819df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.718+0000 7f6f9d2c9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f98077350 0x7f6f9819e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.718+0000 7f6f9d2c9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f98077cf0 0x7f6f981a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.718+0000 7f6f9d2c9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6f98113910 con 0x7f6f98076110 2024-04-01T01:35:21.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.718+0000 7f6f9d2c9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f98113790 con 0x7f6f98077cf0 2024-04-01T01:35:21.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.718+0000 7f6f9d2c9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6f98113a90 con 0x7f6f98077350 2024-04-01T01:35:21.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.719+0000 7f6f977fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f98077cf0 0x7f6f981a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.719+0000 7f6f977fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f98077cf0 0x7f6f981a54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35154/0 (socket says 172.21.15.70:35154) 2024-04-01T01:35:21.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.719+0000 7f6f977fe640 1 -- 172.21.15.70:0/3177810315 learned_addr learned my addr 172.21.15.70:0/3177810315 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:21.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.719+0000 7f6f967fc640 1 --2- 172.21.15.70:0/3177810315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f98077350 0x7f6f9819e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.719+0000 7f6f96ffd640 1 --2- 172.21.15.70:0/3177810315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f98076110 0x7f6f9819df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.719+0000 7f6f977fe640 1 -- 172.21.15.70:0/3177810315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f98077350 msgr2=0x7f6f9819e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:21.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.719+0000 7f6f977fe640 1 --2- 172.21.15.70:0/3177810315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f98077350 0x7f6f9819e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.719+0000 7f6f977fe640 1 -- 172.21.15.70:0/3177810315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f98076110 msgr2=0x7f6f9819df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:21.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.719+0000 7f6f977fe640 1 --2- 172.21.15.70:0/3177810315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f98076110 0x7f6f9819df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:21.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.719+0000 7f6f977fe640 1 -- 172.21.15.70:0/3177810315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f981a5b20 con 0x7f6f98077cf0 2024-04-01T01:35:21.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.720+0000 7f6f977fe640 1 --2- 172.21.15.70:0/3177810315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f98077cf0 0x7f6f981a54b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6f90032520 tx=0x7f6f90035ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:21.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.722+0000 7f6f7ffff640 1 -- 172.21.15.70:0/3177810315 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f90039040 con 0x7f6f98077cf0 2024-04-01T01:35:21.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.722+0000 7f6f9d2c9640 1 -- 172.21.15.70:0/3177810315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f981a5db0 con 0x7f6f98077cf0 2024-04-01T01:35:21.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.722+0000 7f6f9d2c9640 1 -- 172.21.15.70:0/3177810315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f981a6240 con 0x7f6f98077cf0 2024-04-01T01:35:21.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.723+0000 7f6f7ffff640 1 -- 172.21.15.70:0/3177810315 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f90044070 con 0x7f6f98077cf0 2024-04-01T01:35:21.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.723+0000 7f6f7ffff640 1 -- 172.21.15.70:0/3177810315 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f90004330 con 0x7f6f98077cf0 2024-04-01T01:35:21.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.724+0000 7f6f9d2c9640 1 -- 172.21.15.70:0/3177810315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f840052c0 con 0x7f6f98077cf0 2024-04-01T01:35:21.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.726+0000 7f6f7ffff640 1 -- 172.21.15.70:0/3177810315 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f6f900044d0 con 0x7f6f98077cf0 2024-04-01T01:35:21.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.727+0000 7f6f7ffff640 1 --2- 172.21.15.70:0/3177810315 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f800a2670 0x7f6f800a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:21.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.727+0000 7f6f7ffff640 1 -- 172.21.15.70:0/3177810315 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6f900f08c0 con 0x7f6f98077cf0 2024-04-01T01:35:21.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.728+0000 7f6f96ffd640 1 --2- 172.21.15.70:0/3177810315 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f800a2670 0x7f6f800a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:21.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.729+0000 7f6f96ffd640 1 --2- 172.21.15.70:0/3177810315 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f800a2670 0x7f6f800a4b30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6f8800abe0 tx=0x7f6f8800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:21.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:21.736+0000 7f6f7ffff640 1 -- 172.21.15.70:0/3177810315 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6f9004d020 con 0x7f6f98077cf0 2024-04-01T01:35:21.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 35.0 s, 1237.4 tps, lat 80.840 ms stddev 23.666 2024-04-01T01:35:21.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:21 smithi189 ceph-mon[35753]: pgmap v414: 129 pgs: 129 active+clean; 943 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.30k op/s 2024-04-01T01:35:22.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:21 smithi070 ceph-mon[32179]: pgmap v414: 129 pgs: 129 active+clean; 943 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.30k op/s 2024-04-01T01:35:22.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:21 smithi081 ceph-mon[35142]: pgmap v414: 129 pgs: 129 active+clean; 943 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.30k op/s 2024-04-01T01:35:22.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.164+0000 7f6f9d2c9640 1 -- 172.21.15.70:0/3177810315 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f840055b0 con 0x7f6f98077cf0 2024-04-01T01:35:22.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.166+0000 7f6f7ffff640 1 -- 172.21.15.70:0/3177810315 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f6f900bd650 con 0x7f6f98077cf0 2024-04-01T01:35:22.167 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:35:22.167 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:33:36.894838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24418,24404,24379,24451,24464]},"id":1}]} 2024-04-01T01:35:22.167 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-04-01T01:35:22.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.169+0000 7f6f9d2c9640 1 -- 172.21.15.70:0/3177810315 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f800a2670 msgr2=0x7f6f800a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.169+0000 7f6f9d2c9640 1 --2- 172.21.15.70:0/3177810315 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f800a2670 0x7f6f800a4b30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6f8800abe0 tx=0x7f6f8800a2d0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.170+0000 7f6f9d2c9640 1 -- 172.21.15.70:0/3177810315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f98077cf0 msgr2=0x7f6f981a54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.170+0000 7f6f9d2c9640 1 --2- 172.21.15.70:0/3177810315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f98077cf0 0x7f6f981a54b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6f90032520 tx=0x7f6f90035ce0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.170+0000 7f6f9d2c9640 1 -- 172.21.15.70:0/3177810315 shutdown_connections 2024-04-01T01:35:22.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.170+0000 7f6f9d2c9640 1 --2- 172.21.15.70:0/3177810315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f98077cf0 0x7f6f981a54b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.170+0000 7f6f9d2c9640 1 --2- 172.21.15.70:0/3177810315 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f800a2670 0x7f6f800a4b30 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.170+0000 7f6f9d2c9640 1 --2- 172.21.15.70:0/3177810315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f98077350 0x7f6f9819e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.170+0000 7f6f9d2c9640 1 --2- 172.21.15.70:0/3177810315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f98076110 0x7f6f9819df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.170+0000 7f6f9d2c9640 1 -- 172.21.15.70:0/3177810315 >> 172.21.15.70:0/3177810315 conn(0x7f6f980fef00 msgr2=0x7f6f981005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.170+0000 7f6f9d2c9640 1 -- 172.21.15.70:0/3177810315 shutdown_connections 2024-04-01T01:35:22.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.170+0000 7f6f9d2c9640 1 -- 172.21.15.70:0/3177810315 wait complete. 2024-04-01T01:35:22.189 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:35:22.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.378+0000 7fc53ca55640 1 -- 172.21.15.70:0/3114263238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5381089c0 msgr2=0x7fc53810adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.378+0000 7fc53ca55640 1 --2- 172.21.15.70:0/3114263238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5381089c0 0x7fc53810adb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc52c009370 tx=0x7fc52c030180 comp rx=0 tx=0).stop 2024-04-01T01:35:22.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.378+0000 7fc53ca55640 1 -- 172.21.15.70:0/3114263238 shutdown_connections 2024-04-01T01:35:22.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.378+0000 7fc53ca55640 1 --2- 172.21.15.70:0/3114263238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53810de20 0x7fc5381102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.378+0000 7fc53ca55640 1 --2- 172.21.15.70:0/3114263238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc53810b380 0x7fc53810d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.378+0000 7fc53ca55640 1 --2- 172.21.15.70:0/3114263238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5381089c0 0x7fc53810adb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.378+0000 7fc53ca55640 1 -- 172.21.15.70:0/3114263238 >> 172.21.15.70:0/3114263238 conn(0x7fc53806e5b0 msgr2=0x7fc5380709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.378+0000 7fc53ca55640 1 -- 172.21.15.70:0/3114263238 shutdown_connections 2024-04-01T01:35:22.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.379+0000 7fc53ca55640 1 -- 172.21.15.70:0/3114263238 wait complete. 2024-04-01T01:35:22.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.380+0000 7fc53ca55640 1 Processor -- start 2024-04-01T01:35:22.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.380+0000 7fc53ca55640 1 -- start start 2024-04-01T01:35:22.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.381+0000 7fc53ca55640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5381089c0 0x7fc53819e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.381+0000 7fc53ca55640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc53810b380 0x7fc53819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.381+0000 7fc53ca55640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53810de20 0x7fc5381a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.381+0000 7fc53ca55640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc538113960 con 0x7fc53810b380 2024-04-01T01:35:22.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.381+0000 7fc53ca55640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc5381137e0 con 0x7fc5381089c0 2024-04-01T01:35:22.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.381+0000 7fc53ca55640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc538113ae0 con 0x7fc53810de20 2024-04-01T01:35:22.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.381+0000 7fc536d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53810de20 0x7fc5381a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.381+0000 7fc536d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53810de20 0x7fc5381a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37652/0 (socket says 172.21.15.70:37652) 2024-04-01T01:35:22.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.381+0000 7fc536d9d640 1 -- 172.21.15.70:0/3946917752 learned_addr learned my addr 172.21.15.70:0/3946917752 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:22.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.381+0000 7fc535d9b640 1 --2- 172.21.15.70:0/3946917752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc53810b380 0x7fc53819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.382+0000 7fc536d9d640 1 -- 172.21.15.70:0/3946917752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5381089c0 msgr2=0x7fc53819e110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:35:22.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.382+0000 7fc536d9d640 1 --2- 172.21.15.70:0/3946917752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5381089c0 0x7fc53819e110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.382+0000 7fc536d9d640 1 -- 172.21.15.70:0/3946917752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc53810b380 msgr2=0x7fc53819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.382+0000 7fc536d9d640 1 --2- 172.21.15.70:0/3946917752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc53810b380 0x7fc53819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.382+0000 7fc536d9d640 1 -- 172.21.15.70:0/3946917752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5381a5d30 con 0x7fc53810de20 2024-04-01T01:35:22.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.383+0000 7fc536d9d640 1 --2- 172.21.15.70:0/3946917752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53810de20 0x7fc5381a56c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc52800a820 tx=0x7fc52800ace0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.383+0000 7fc5277fe640 1 -- 172.21.15.70:0/3946917752 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5280079b0 con 0x7fc53810de20 2024-04-01T01:35:22.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.383+0000 7fc5277fe640 1 -- 172.21.15.70:0/3946917752 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc528005020 con 0x7fc53810de20 2024-04-01T01:35:22.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.383+0000 7fc53ca55640 1 -- 172.21.15.70:0/3946917752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5381a5fc0 con 0x7fc53810de20 2024-04-01T01:35:22.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.384+0000 7fc53ca55640 1 -- 172.21.15.70:0/3946917752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5381a6560 con 0x7fc53810de20 2024-04-01T01:35:22.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.384+0000 7fc5277fe640 1 -- 172.21.15.70:0/3946917752 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc528004cf0 con 0x7fc53810de20 2024-04-01T01:35:22.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.387+0000 7fc5277fe640 1 -- 172.21.15.70:0/3946917752 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fc5280051d0 con 0x7fc53810de20 2024-04-01T01:35:22.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.388+0000 7fc53f63a740 1 Processor -- start 2024-04-01T01:35:22.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.388+0000 7fc5277fe640 1 --2- 172.21.15.70:0/3946917752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5200a25f0 0x7fc5200a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.388+0000 7fc53f63a740 1 -- start start 2024-04-01T01:35:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.388+0000 7fc535d9b640 1 --2- 172.21.15.70:0/3946917752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5200a25f0 0x7fc5200a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc53f63a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96dc490 0x5641e96dc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc53f63a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96dce80 0x5641e96e1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc53f63a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96e1ff0 0x5641e96e4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc53f63a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5641e96411b0 con 0x5641e96e1ff0 2024-04-01T01:35:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc53f63a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5641e95e9c00 con 0x5641e96dc490 2024-04-01T01:35:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc53f63a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5641e963d250 con 0x5641e96dce80 2024-04-01T01:35:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc535d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96dce80 0x5641e96e1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc535d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96dce80 0x5641e96e1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37654/0 (socket says 172.21.15.70:37654) 2024-04-01T01:35:22.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc535d9b640 1 -- 172.21.15.70:0/3432042541 learned_addr learned my addr 172.21.15.70:0/3432042541 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:22.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc53659c640 1 --2- 172.21.15.70:0/3432042541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96dc490 0x5641e96dc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc536d9d640 1 --2- 172.21.15.70:0/3432042541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96e1ff0 0x5641e96e4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc53659c640 1 -- 172.21.15.70:0/3432042541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96dce80 msgr2=0x5641e96e1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc53659c640 1 --2- 172.21.15.70:0/3432042541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96dce80 0x5641e96e1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc53659c640 1 -- 172.21.15.70:0/3432042541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96e1ff0 msgr2=0x5641e96e4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc53659c640 1 --2- 172.21.15.70:0/3432042541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96e1ff0 0x5641e96e4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc53659c640 1 -- 172.21.15.70:0/3432042541 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641e96093d0 con 0x5641e96dc490 2024-04-01T01:35:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.389+0000 7fc5277fe640 1 -- 172.21.15.70:0/3946917752 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fc5280ce020 con 0x7fc53810de20 2024-04-01T01:35:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.390+0000 7fc53659c640 1 --2- 172.21.15.70:0/3432042541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96dc490 0x5641e96dc8b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc52c009370 tx=0x7fc52c009db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3021f21548eef96 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.390+0000 7fc535d9b640 1 --2- 172.21.15.70:0/3946917752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5200a25f0 0x7fc5200a4ab0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc530009fd0 tx=0x7fc53000cb80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.390+0000 7fc524ff9640 1 -- 172.21.15.70:0/3432042541 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c045070 con 0x5641e96dc490 2024-04-01T01:35:22.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.390+0000 7fc524ff9640 1 -- 172.21.15.70:0/3432042541 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc52c038640 con 0x5641e96dc490 2024-04-01T01:35:22.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.390+0000 7fc524ff9640 1 -- 172.21.15.70:0/3432042541 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c0044a0 con 0x5641e96dc490 2024-04-01T01:35:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.391+0000 7fc53f63a740 1 -- 172.21.15.70:0/3432042541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96dc490 msgr2=0x5641e96dc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.391+0000 7fc53f63a740 1 --2- 172.21.15.70:0/3432042541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96dc490 0x5641e96dc8b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc52c009370 tx=0x7fc52c009db0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.391+0000 7fc53f63a740 1 -- 172.21.15.70:0/3432042541 shutdown_connections 2024-04-01T01:35:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.391+0000 7fc53f63a740 1 --2- 172.21.15.70:0/3432042541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96e1ff0 0x5641e96e4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.391+0000 7fc53f63a740 1 --2- 172.21.15.70:0/3432042541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96dce80 0x5641e96e1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.391+0000 7fc53f63a740 1 --2- 172.21.15.70:0/3432042541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96dc490 0x5641e96dc8b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.391+0000 7fc53f63a740 1 -- 172.21.15.70:0/3432042541 >> 172.21.15.70:0/3432042541 conn(0x5641e96d7a00 msgr2=0x5641e96d7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.392+0000 7fc53f63a740 1 -- 172.21.15.70:0/3432042541 shutdown_connections 2024-04-01T01:35:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.392+0000 7fc53f63a740 1 -- 172.21.15.70:0/3432042541 wait complete. 2024-04-01T01:35:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.392+0000 7fc53f63a740 1 -- start start 2024-04-01T01:35:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.392+0000 7fc53f63a740 1 Processor -- start 2024-04-01T01:35:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.393+0000 7fc53f63a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96dc490 0x5641e96ef190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.393+0000 7fc53f63a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96dce80 0x5641e96ef6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.393+0000 7fc53f63a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96e1ff0 0x5641e96f6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.393+0000 7fc53f63a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5641e96ed750 con 0x5641e96dce80 2024-04-01T01:35:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.393+0000 7fc53f63a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5641e95df470 con 0x5641e96e1ff0 2024-04-01T01:35:22.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.393+0000 7fc53f63a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5641e96ed8d0 con 0x5641e96dc490 2024-04-01T01:35:22.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.393+0000 7fc53f63a740 10 client.0 _ll_register_callbacks cb 0x5641e95fccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:22.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.393+0000 7fc535d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96dce80 0x5641e96ef6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.393+0000 7fc535d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96dce80 0x5641e96ef6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38802/0 (socket says 172.21.15.70:38802) 2024-04-01T01:35:22.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.393+0000 7fc535d9b640 1 -- 172.21.15.70:0/3008390580 learned_addr learned my addr 172.21.15.70:0/3008390580 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:22.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.393+0000 7fc53659c640 1 --2- 172.21.15.70:0/3008390580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96dc490 0x5641e96ef190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.394+0000 7fc536d9d640 1 --2- 172.21.15.70:0/3008390580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96e1ff0 0x5641e96f6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.394+0000 7fc53659c640 1 -- 172.21.15.70:0/3008390580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96e1ff0 msgr2=0x5641e96f6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.394+0000 7fc53659c640 1 --2- 172.21.15.70:0/3008390580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96e1ff0 0x5641e96f6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.394+0000 7fc53659c640 1 -- 172.21.15.70:0/3008390580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96dce80 msgr2=0x5641e96ef6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.394+0000 7fc53659c640 1 --2- 172.21.15.70:0/3008390580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96dce80 0x5641e96ef6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.394+0000 7fc53659c640 1 -- 172.21.15.70:0/3008390580 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641e93553f0 con 0x5641e96dc490 2024-04-01T01:35:22.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.395+0000 7fc53659c640 1 --2- 172.21.15.70:0/3008390580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96dc490 0x5641e96ef190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc52c030760 tx=0x7fc52c031d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0e8964ec82b437 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.395+0000 7fc51dffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:35:22.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.395+0000 7fc51dffb640 1 -- 172.21.15.70:0/3008390580 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c045040 con 0x5641e96dc490 2024-04-01T01:35:22.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.395+0000 7fc53f63a740 1 -- 172.21.15.70:0/3008390580 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc52c038640 con 0x5641e96dc490 2024-04-01T01:35:22.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.395+0000 7fc51dffb640 1 -- 172.21.15.70:0/3008390580 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fc52c0046c0 con 0x5641e96dc490 2024-04-01T01:35:22.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.395+0000 7fc51dffb640 1 -- 172.21.15.70:0/3008390580 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc52c058050 con 0x5641e96dc490 2024-04-01T01:35:22.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.395+0000 7fc53f63a740 10 client.25867 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:22.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.395+0000 7fc53f63a740 1 -- 172.21.15.70:0/3008390580 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5641e941f7d0 con 0x5641e96dc490 2024-04-01T01:35:22.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.395+0000 7fc51dffb640 1 -- 172.21.15.70:0/3008390580 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc52c038640 con 0x5641e96dc490 2024-04-01T01:35:22.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.396+0000 7fc51dffb640 1 -- 172.21.15.70:0/3008390580 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc52c053030 con 0x5641e96dc490 2024-04-01T01:35:22.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.396+0000 7fc53f63a740 10 client.25867 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:22.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.396+0000 7fc53f63a740 10 client.25867 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:35:22.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.396+0000 7fc53f63a740 20 client.25867 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:22.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.396+0000 7fc53f63a740 1 --2- 172.21.15.70:0/3008390580 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5641e96fad30 0x5641e96fd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.396+0000 7fc53f63a740 4 client.25867 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:22.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.396+0000 7fc53f63a740 1 -- 172.21.15.70:0/3008390580 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641e964e6f0 con 0x5641e96fad30 2024-04-01T01:35:22.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.396+0000 7fc535d9b640 1 --2- 172.21.15.70:0/3008390580 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5641e96fad30 0x5641e96fd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.397+0000 7fc535d9b640 1 --2- 172.21.15.70:0/3008390580 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5641e96fad30 0x5641e96fd1a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=593335e27d61dd11 server_cookie=59fc28c10bbfedb0 in_seq=0 out_seq=0 2024-04-01T01:35:22.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.397+0000 7fc51dffb640 10 client.25867 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:35:22.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.398+0000 7fc51dffb640 1 -- 172.21.15.70:0/3008390580 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5641e964e6f0 con 0x5641e96fad30 2024-04-01T01:35:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.398+0000 7fc51dffb640 10 client.25867 handle_command_reply: tid=0 2024-04-01T01:35:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.398+0000 7fc53f63a740 1 client.25867 shutdown 2024-04-01T01:35:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.398+0000 7fc53f63a740 1 -- 172.21.15.70:0/3008390580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96dc490 msgr2=0x5641e96ef190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.398+0000 7fc53f63a740 1 --2- 172.21.15.70:0/3008390580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96dc490 0x5641e96ef190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc52c030760 tx=0x7fc52c031d60 comp rx=0 tx=0).stop 2024-04-01T01:35:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.398+0000 7fc53f63a740 1 -- 172.21.15.70:0/3008390580 shutdown_connections 2024-04-01T01:35:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.398+0000 7fc53f63a740 1 --2- 172.21.15.70:0/3008390580 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5641e96fad30 0x5641e96fd1a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.398+0000 7fc53f63a740 1 --2- 172.21.15.70:0/3008390580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96e1ff0 0x5641e96f6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.398+0000 7fc53f63a740 1 --2- 172.21.15.70:0/3008390580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96dce80 0x5641e96ef6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.398+0000 7fc53f63a740 1 --2- 172.21.15.70:0/3008390580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96dc490 0x5641e96ef190 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc52c030760 tx=0x7fc52c031d60 comp rx=0 tx=0).stop 2024-04-01T01:35:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.398+0000 7fc53f63a740 1 -- 172.21.15.70:0/3008390580 >> 172.21.15.70:0/3008390580 conn(0x5641e96d7a00 msgr2=0x5641e96d6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.398+0000 7fc51dffb640 0 client.25867 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:35:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.399+0000 7fc53f63a740 1 -- 172.21.15.70:0/3008390580 shutdown_connections 2024-04-01T01:35:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.399+0000 7fc53f63a740 1 -- 172.21.15.70:0/3008390580 wait complete. 2024-04-01T01:35:22.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.399+0000 7fc53f63a740 20 client.25867 trim_cache size 0 max 16384 2024-04-01T01:35:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.429+0000 7fc53f63a740 1 Processor -- start 2024-04-01T01:35:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.429+0000 7fc53f63a740 1 -- start start 2024-04-01T01:35:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.429+0000 7fc53f63a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96dce80 0x5641e96ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.429+0000 7fc53f63a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96e1ff0 0x5641e96eed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.429+0000 7fc53f63a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96d3030 0x5641e96ef260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.430+0000 7fc53f63a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc5381b09e0 con 0x5641e96e1ff0 2024-04-01T01:35:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.430+0000 7fc53f63a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc5381b2520 con 0x5641e96dce80 2024-04-01T01:35:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.430+0000 7fc53f63a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc5381b10b0 con 0x5641e96d3030 2024-04-01T01:35:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.430+0000 7fc535d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96e1ff0 0x5641e96eed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.430+0000 7fc535d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96e1ff0 0x5641e96eed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38808/0 (socket says 172.21.15.70:38808) 2024-04-01T01:35:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.430+0000 7fc535d9b640 1 -- 172.21.15.70:0/798616747 learned_addr learned my addr 172.21.15.70:0/798616747 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.430+0000 7fc53659c640 1 --2- 172.21.15.70:0/798616747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96dce80 0x5641e96ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.430+0000 7fc536d9d640 1 --2- 172.21.15.70:0/798616747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96d3030 0x5641e96ef260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.430+0000 7fc53659c640 1 -- 172.21.15.70:0/798616747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96d3030 msgr2=0x5641e96ef260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.430+0000 7fc53659c640 1 --2- 172.21.15.70:0/798616747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96d3030 0x5641e96ef260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.430+0000 7fc53659c640 1 -- 172.21.15.70:0/798616747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96e1ff0 msgr2=0x5641e96eed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.430+0000 7fc53659c640 1 --2- 172.21.15.70:0/798616747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96e1ff0 0x5641e96eed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.430+0000 7fc53659c640 1 -- 172.21.15.70:0/798616747 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641e96dd260 con 0x5641e96dce80 2024-04-01T01:35:22.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.430+0000 7fc535d9b640 1 --2- 172.21.15.70:0/798616747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96e1ff0 0x5641e96eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:22.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.431+0000 7fc53659c640 1 --2- 172.21.15.70:0/798616747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96dce80 0x5641e96ee7e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc52c011290 tx=0x7fc52c009e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=216099a3148b77a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.432+0000 7fc51ffff640 1 -- 172.21.15.70:0/798616747 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c045040 con 0x5641e96dce80 2024-04-01T01:35:22.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.432+0000 7fc51ffff640 1 -- 172.21.15.70:0/798616747 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc52c011310 con 0x5641e96dce80 2024-04-01T01:35:22.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.432+0000 7fc51ffff640 1 -- 172.21.15.70:0/798616747 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c004a30 con 0x5641e96dce80 2024-04-01T01:35:22.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.433+0000 7fc53f63a740 1 -- 172.21.15.70:0/798616747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96dce80 msgr2=0x5641e96ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.433+0000 7fc53f63a740 1 --2- 172.21.15.70:0/798616747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96dce80 0x5641e96ee7e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc52c011290 tx=0x7fc52c009e30 comp rx=0 tx=0).stop 2024-04-01T01:35:22.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.434+0000 7fc53f63a740 1 -- 172.21.15.70:0/798616747 shutdown_connections 2024-04-01T01:35:22.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.434+0000 7fc53f63a740 1 --2- 172.21.15.70:0/798616747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96d3030 0x5641e96ef260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.434+0000 7fc53f63a740 1 --2- 172.21.15.70:0/798616747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96e1ff0 0x5641e96eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.434+0000 7fc53f63a740 1 --2- 172.21.15.70:0/798616747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96dce80 0x5641e96ee7e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.434+0000 7fc53f63a740 1 -- 172.21.15.70:0/798616747 >> 172.21.15.70:0/798616747 conn(0x5641e96d7a00 msgr2=0x5641e96db4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.435+0000 7fc53f63a740 1 -- 172.21.15.70:0/798616747 shutdown_connections 2024-04-01T01:35:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.435+0000 7fc53f63a740 1 -- 172.21.15.70:0/798616747 wait complete. 2024-04-01T01:35:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.435+0000 7fc53f63a740 1 -- start start 2024-04-01T01:35:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.435+0000 7fc53f63a740 1 Processor -- start 2024-04-01T01:35:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.436+0000 7fc53f63a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96d3030 0x5641e96f3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.436+0000 7fc53f63a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96dce80 0x5641e96f3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.436+0000 7fc53f63a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96e1ff0 0x5641e96f4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.436+0000 7fc53659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96d3030 0x5641e96f3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.436+0000 7fc53f63a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc5381b1e50 con 0x5641e96dce80 2024-04-01T01:35:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.436+0000 7fc53f63a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc5381b1780 con 0x5641e96d3030 2024-04-01T01:35:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.436+0000 7fc53659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96d3030 0x5641e96f3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35196/0 (socket says 172.21.15.70:35196) 2024-04-01T01:35:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.436+0000 7fc53659c640 1 -- 172.21.15.70:0/689154603 learned_addr learned my addr 172.21.15.70:0/689154603 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.436+0000 7fc535d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96dce80 0x5641e96f3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.437+0000 7fc53f63a740 1 -- 172.21.15.70:0/689154603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc5381ae9c0 con 0x5641e96e1ff0 2024-04-01T01:35:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.437+0000 7fc53f63a740 10 client.0 _ll_register_callbacks cb 0x7fc52c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.437+0000 7fc536d9d640 1 --2- 172.21.15.70:0/689154603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96e1ff0 0x5641e96f4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.437+0000 7fc53659c640 1 -- 172.21.15.70:0/689154603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96e1ff0 msgr2=0x5641e96f4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.437+0000 7fc53659c640 1 --2- 172.21.15.70:0/689154603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96e1ff0 0x5641e96f4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.437+0000 7fc53659c640 1 -- 172.21.15.70:0/689154603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96dce80 msgr2=0x5641e96f3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.437+0000 7fc53659c640 1 --2- 172.21.15.70:0/689154603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96dce80 0x5641e96f3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.437+0000 7fc53659c640 1 -- 172.21.15.70:0/689154603 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641e96f5590 con 0x5641e96d3030 2024-04-01T01:35:22.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.438+0000 7fc53659c640 1 --2- 172.21.15.70:0/689154603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96d3030 0x5641e96f3aa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc52c011260 tx=0x7fc52c0306a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27749b4e932ebbf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.438+0000 7fc51dffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:22.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.438+0000 7fc51dffb640 1 -- 172.21.15.70:0/689154603 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c062030 con 0x5641e96d3030 2024-04-01T01:35:22.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.438+0000 7fc51dffb640 1 -- 172.21.15.70:0/689154603 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fc52c038440 con 0x5641e96d3030 2024-04-01T01:35:22.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.438+0000 7fc53f63a740 1 -- 172.21.15.70:0/689154603 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc52c011310 con 0x5641e96d3030 2024-04-01T01:35:22.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.439+0000 7fc51dffb640 1 -- 172.21.15.70:0/689154603 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc52c06d050 con 0x5641e96d3030 2024-04-01T01:35:22.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.439+0000 7fc53f63a740 10 client.25964 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:22.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.439+0000 7fc53f63a740 1 -- 172.21.15.70:0/689154603 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5641e97610e0 con 0x5641e96d3030 2024-04-01T01:35:22.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.440+0000 7fc51dffb640 1 -- 172.21.15.70:0/689154603 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc52c011310 con 0x5641e96d3030 2024-04-01T01:35:22.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.440+0000 7fc51dffb640 1 -- 172.21.15.70:0/689154603 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc52c053070 con 0x5641e96d3030 2024-04-01T01:35:22.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.440+0000 7fc53f63a740 10 client.25964 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:22.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.440+0000 7fc53f63a740 10 client.25964 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:35:22.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.440+0000 7fc53f63a740 20 client.25964 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:22.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.440+0000 7fc53f63a740 1 --2- 172.21.15.70:0/689154603 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5641e9761860 0x5641e9763c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.440+0000 7fc53f63a740 4 client.25964 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:35:22.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.441+0000 7fc53f63a740 1 -- 172.21.15.70:0/689154603 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5641e9648a40 con 0x5641e9761860 2024-04-01T01:35:22.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.441+0000 7fc535d9b640 1 --2- 172.21.15.70:0/689154603 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5641e9761860 0x5641e9763c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.442+0000 7fc535d9b640 1 --2- 172.21.15.70:0/689154603 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5641e9761860 0x5641e9763c80 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dec0056aa64ac6e8 server_cookie=f984cfdc16c57bac in_seq=0 out_seq=0 2024-04-01T01:35:22.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.442+0000 7fc51dffb640 10 client.25964 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:35:22.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.442+0000 7fc51dffb640 1 -- 172.21.15.70:0/689154603 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5641e9648a40 con 0x5641e9761860 2024-04-01T01:35:22.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.442+0000 7fc51dffb640 10 client.25964 handle_command_reply: tid=0 2024-04-01T01:35:22.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.443+0000 7fc53f63a740 1 client.25964 shutdown 2024-04-01T01:35:22.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.444+0000 7fc53f63a740 1 -- 172.21.15.70:0/689154603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96d3030 msgr2=0x5641e96f3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.444+0000 7fc53f63a740 1 --2- 172.21.15.70:0/689154603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96d3030 0x5641e96f3aa0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc52c011260 tx=0x7fc52c0306a0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.444+0000 7fc53f63a740 1 -- 172.21.15.70:0/689154603 shutdown_connections 2024-04-01T01:35:22.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.444+0000 7fc53f63a740 1 --2- 172.21.15.70:0/689154603 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5641e9761860 0x5641e9763c80 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.444+0000 7fc53f63a740 1 --2- 172.21.15.70:0/689154603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641e96e1ff0 0x5641e96f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.444+0000 7fc53f63a740 1 --2- 172.21.15.70:0/689154603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5641e96dce80 0x5641e96f3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.444+0000 7fc51dffb640 0 client.25964 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:35:22.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.444+0000 7fc53f63a740 1 --2- 172.21.15.70:0/689154603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5641e96d3030 0x5641e96f3aa0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.444+0000 7fc53f63a740 1 -- 172.21.15.70:0/689154603 >> 172.21.15.70:0/689154603 conn(0x5641e96d7a00 msgr2=0x5641e96d64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.445+0000 7fc53f63a740 1 -- 172.21.15.70:0/689154603 shutdown_connections 2024-04-01T01:35:22.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.446+0000 7fc53f63a740 1 -- 172.21.15.70:0/689154603 wait complete. 2024-04-01T01:35:22.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.446+0000 7fc53f63a740 20 client.25964 trim_cache size 0 max 16384 2024-04-01T01:35:22.446 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:35:22.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.447+0000 7fc53ca55640 1 -- 172.21.15.70:0/3946917752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5200a25f0 msgr2=0x7fc5200a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.447+0000 7fc53ca55640 1 --2- 172.21.15.70:0/3946917752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5200a25f0 0x7fc5200a4ab0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc530009fd0 tx=0x7fc53000cb80 comp rx=0 tx=0).stop 2024-04-01T01:35:22.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.447+0000 7fc53ca55640 1 -- 172.21.15.70:0/3946917752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53810de20 msgr2=0x7fc5381a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.447+0000 7fc53ca55640 1 --2- 172.21.15.70:0/3946917752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53810de20 0x7fc5381a56c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc52800a820 tx=0x7fc52800ace0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.447+0000 7fc53ca55640 1 -- 172.21.15.70:0/3946917752 shutdown_connections 2024-04-01T01:35:22.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.447+0000 7fc53ca55640 1 --2- 172.21.15.70:0/3946917752 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc5200a25f0 0x7fc5200a4ab0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.447+0000 7fc53ca55640 1 --2- 172.21.15.70:0/3946917752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc53810de20 0x7fc5381a56c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.447+0000 7fc53ca55640 1 --2- 172.21.15.70:0/3946917752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc53810b380 0x7fc53819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.447+0000 7fc53ca55640 1 --2- 172.21.15.70:0/3946917752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5381089c0 0x7fc53819e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.447+0000 7fc53ca55640 1 -- 172.21.15.70:0/3946917752 >> 172.21.15.70:0/3946917752 conn(0x7fc53806e5b0 msgr2=0x7fc538108da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.448+0000 7fc53ca55640 1 -- 172.21.15.70:0/3946917752 shutdown_connections 2024-04-01T01:35:22.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.448+0000 7fc53ca55640 1 -- 172.21.15.70:0/3946917752 wait complete. 2024-04-01T01:35:22.465 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:35:22.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.653+0000 7f32c378a640 1 -- 172.21.15.70:0/1967190145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32bc107450 msgr2=0x7f32bc1098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.653+0000 7f32c378a640 1 --2- 172.21.15.70:0/1967190145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32bc107450 0x7f32bc1098a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f32b4009ca0 tx=0x7f32b402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.653+0000 7f32c378a640 1 -- 172.21.15.70:0/1967190145 shutdown_connections 2024-04-01T01:35:22.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.653+0000 7f32c378a640 1 --2- 172.21.15.70:0/1967190145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32bc107450 0x7f32bc1098a0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.653+0000 7f32c378a640 1 --2- 172.21.15.70:0/1967190145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32bc104b20 0x7f32bc106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.653+0000 7f32c378a640 1 --2- 172.21.15.70:0/1967190145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32bc1021f0 0x7f32bc1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.653+0000 7f32c378a640 1 -- 172.21.15.70:0/1967190145 >> 172.21.15.70:0/1967190145 conn(0x7f32bc0fbc00 msgr2=0x7f32bc0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.654+0000 7f32c378a640 1 -- 172.21.15.70:0/1967190145 shutdown_connections 2024-04-01T01:35:22.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.654+0000 7f32c378a640 1 -- 172.21.15.70:0/1967190145 wait complete. 2024-04-01T01:35:22.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.654+0000 7f32c378a640 1 Processor -- start 2024-04-01T01:35:22.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.654+0000 7f32c378a640 1 -- start start 2024-04-01T01:35:22.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c378a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32bc1021f0 0x7f32bc1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c378a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32bc104b20 0x7f32bc197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c378a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32bc107450 0x7f32bc19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c378a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f32bc10d2a0 con 0x7f32bc104b20 2024-04-01T01:35:22.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c378a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f32bc10d120 con 0x7f32bc1021f0 2024-04-01T01:35:22.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c378a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f32bc10d420 con 0x7f32bc107450 2024-04-01T01:35:22.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c0d25640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32bc104b20 0x7f32bc197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c1526640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32bc1021f0 0x7f32bc1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c0d25640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32bc104b20 0x7f32bc197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38832/0 (socket says 172.21.15.70:38832) 2024-04-01T01:35:22.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c0d25640 1 -- 172.21.15.70:0/2142199347 learned_addr learned my addr 172.21.15.70:0/2142199347 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:22.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c1d27640 1 --2- 172.21.15.70:0/2142199347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32bc107450 0x7f32bc19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c1526640 1 -- 172.21.15.70:0/2142199347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32bc107450 msgr2=0x7f32bc19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c1526640 1 --2- 172.21.15.70:0/2142199347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32bc107450 0x7f32bc19ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c1526640 1 -- 172.21.15.70:0/2142199347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32bc104b20 msgr2=0x7f32bc197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c1526640 1 --2- 172.21.15.70:0/2142199347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32bc104b20 0x7f32bc197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c1526640 1 -- 172.21.15.70:0/2142199347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32bc19f5d0 con 0x7f32bc1021f0 2024-04-01T01:35:22.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.655+0000 7f32c0d25640 1 --2- 172.21.15.70:0/2142199347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32bc104b20 0x7f32bc197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:22.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.656+0000 7f32c1526640 1 --2- 172.21.15.70:0/2142199347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32bc1021f0 0x7f32bc1979b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f32b800e9f0 tx=0x7f32b800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.656+0000 7f32b27fc640 1 -- 172.21.15.70:0/2142199347 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b800ccc0 con 0x7f32bc1021f0 2024-04-01T01:35:22.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.656+0000 7f32c378a640 1 -- 172.21.15.70:0/2142199347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32bc100040 con 0x7f32bc1021f0 2024-04-01T01:35:22.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.656+0000 7f32b27fc640 1 -- 172.21.15.70:0/2142199347 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32b80108e0 con 0x7f32bc1021f0 2024-04-01T01:35:22.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.657+0000 7f32b27fc640 1 -- 172.21.15.70:0/2142199347 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b8018880 con 0x7f32bc1021f0 2024-04-01T01:35:22.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.657+0000 7f32c378a640 1 -- 172.21.15.70:0/2142199347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32bc1004d0 con 0x7f32bc1021f0 2024-04-01T01:35:22.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.662+0000 7f32c636f740 1 Processor -- start 2024-04-01T01:35:22.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.663+0000 7f32c636f740 1 -- start start 2024-04-01T01:35:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.663+0000 7f32c636f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b3ca0 0x55562f3b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.663+0000 7f32c636f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b4690 0x55562f3b9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.663+0000 7f32b27fc640 1 -- 172.21.15.70:0/2142199347 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f32b8010430 con 0x7f32bc1021f0 2024-04-01T01:35:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.663+0000 7f32c636f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b9800 0x55562f3bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.663+0000 7f32c636f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55562f3101b0 con 0x55562f3b3ca0 2024-04-01T01:35:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.663+0000 7f32c636f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55562f2b8c00 con 0x55562f3b9800 2024-04-01T01:35:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.663+0000 7f32c636f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55562f30c250 con 0x55562f3b4690 2024-04-01T01:35:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.664+0000 7f32c0d25640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b9800 0x55562f3bbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.664+0000 7f32c0d25640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b9800 0x55562f3bbc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35204/0 (socket says 172.21.15.70:35204) 2024-04-01T01:35:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.664+0000 7f32c0d25640 1 -- 172.21.15.70:0/2377629584 learned_addr learned my addr 172.21.15.70:0/2377629584 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.664+0000 7f32c1526640 1 --2- 172.21.15.70:0/2377629584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b4690 0x55562f3b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.664+0000 7f32c1d27640 1 --2- 172.21.15.70:0/2377629584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b3ca0 0x55562f3b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.664+0000 7f32c0d25640 1 -- 172.21.15.70:0/2377629584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b4690 msgr2=0x55562f3b9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.664+0000 7f32c0d25640 1 --2- 172.21.15.70:0/2377629584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b4690 0x55562f3b9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.664+0000 7f32c0d25640 1 -- 172.21.15.70:0/2377629584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b3ca0 msgr2=0x55562f3b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.664+0000 7f32c0d25640 1 --2- 172.21.15.70:0/2377629584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b3ca0 0x55562f3b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.664+0000 7f32c0d25640 1 -- 172.21.15.70:0/2377629584 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55562f2d83d0 con 0x55562f3b9800 2024-04-01T01:35:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.664+0000 7f32c1526640 1 --2- 172.21.15.70:0/2377629584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b4690 0x55562f3b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:35:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.664+0000 7f32c0d25640 1 --2- 172.21.15.70:0/2377629584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b9800 0x55562f3bbc90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f32ac009850 tx=0x7f32ac005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74d33fefdeca0b74 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.665+0000 7f32a77fe640 1 -- 172.21.15.70:0/2377629584 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac00e930 con 0x55562f3b9800 2024-04-01T01:35:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.665+0000 7f32b27fc640 1 --2- 172.21.15.70:0/2142199347 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32a80a2490 0x7f32a80a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.665+0000 7f32a77fe640 1 -- 172.21.15.70:0/2377629584 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32ac019420 con 0x55562f3b9800 2024-04-01T01:35:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.665+0000 7f32a77fe640 1 -- 172.21.15.70:0/2377629584 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac018620 con 0x55562f3b9800 2024-04-01T01:35:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.665+0000 7f32c1d27640 1 --2- 172.21.15.70:0/2142199347 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32a80a2490 0x7f32a80a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.666+0000 7f32c636f740 1 -- 172.21.15.70:0/2377629584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b9800 msgr2=0x55562f3bbc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.666+0000 7f32c636f740 1 --2- 172.21.15.70:0/2377629584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b9800 0x55562f3bbc90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f32ac009850 tx=0x7f32ac005f90 comp rx=0 tx=0).stop 2024-04-01T01:35:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.666+0000 7f32b27fc640 1 -- 172.21.15.70:0/2142199347 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f32b80cef00 con 0x7f32bc1021f0 2024-04-01T01:35:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.666+0000 7f32c1d27640 1 --2- 172.21.15.70:0/2142199347 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32a80a2490 0x7f32a80a4950 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f32b4006010 tx=0x7f32b402c610 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.666+0000 7f32c636f740 1 -- 172.21.15.70:0/2377629584 shutdown_connections 2024-04-01T01:35:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.666+0000 7f32c636f740 1 --2- 172.21.15.70:0/2377629584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b9800 0x55562f3bbc90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.666+0000 7f32c636f740 1 --2- 172.21.15.70:0/2377629584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b4690 0x55562f3b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.666+0000 7f32c636f740 1 --2- 172.21.15.70:0/2377629584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b3ca0 0x55562f3b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.666+0000 7f32c636f740 1 -- 172.21.15.70:0/2377629584 >> 172.21.15.70:0/2377629584 conn(0x55562f3af210 msgr2=0x55562f3af610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.667+0000 7f32c636f740 1 -- 172.21.15.70:0/2377629584 shutdown_connections 2024-04-01T01:35:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.667+0000 7f32c636f740 1 -- 172.21.15.70:0/2377629584 wait complete. 2024-04-01T01:35:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.667+0000 7f32c636f740 1 -- start start 2024-04-01T01:35:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.667+0000 7f32c636f740 1 Processor -- start 2024-04-01T01:35:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.668+0000 7f32c636f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b3ca0 0x55562f3bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.668+0000 7f32c636f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b4690 0x55562f3be4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.668+0000 7f32c636f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b9800 0x55562f3c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.668+0000 7f32c1526640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b3ca0 0x55562f3bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.668+0000 7f32c0d25640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b4690 0x55562f3be4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.668+0000 7f32c0d25640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b4690 0x55562f3be4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38848/0 (socket says 172.21.15.70:38848) 2024-04-01T01:35:22.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.668+0000 7f32c636f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55562f3bc800 con 0x55562f3b4690 2024-04-01T01:35:22.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.668+0000 7f32c1526640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b3ca0 0x55562f3bdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37728/0 (socket says 172.21.15.70:37728) 2024-04-01T01:35:22.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.668+0000 7f32c0d25640 1 -- 172.21.15.70:0/3461629512 learned_addr learned my addr 172.21.15.70:0/3461629512 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:22.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.668+0000 7f32c636f740 1 -- 172.21.15.70:0/3461629512 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55562f2ae470 con 0x55562f3b9800 2024-04-01T01:35:22.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.668+0000 7f32c636f740 1 -- 172.21.15.70:0/3461629512 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55562f3bc980 con 0x55562f3b3ca0 2024-04-01T01:35:22.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.668+0000 7f32c1d27640 1 --2- 172.21.15.70:0/3461629512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b9800 0x55562f3c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.669+0000 7f32c636f740 10 client.0 _ll_register_callbacks cb 0x55562f2cbcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:22.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.669+0000 7f32c1d27640 1 -- 172.21.15.70:0/3461629512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b3ca0 msgr2=0x55562f3bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.669+0000 7f32c1d27640 1 --2- 172.21.15.70:0/3461629512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b3ca0 0x55562f3bdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.669+0000 7f32c1d27640 1 -- 172.21.15.70:0/3461629512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b4690 msgr2=0x55562f3be4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.669+0000 7f32c1d27640 1 --2- 172.21.15.70:0/3461629512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b4690 0x55562f3be4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.669+0000 7f32c1d27640 1 -- 172.21.15.70:0/3461629512 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55562f0243f0 con 0x55562f3b9800 2024-04-01T01:35:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.669+0000 7f32c1d27640 1 --2- 172.21.15.70:0/3461629512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b9800 0x55562f3c54b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f32b4037040 tx=0x7f32b40095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=474a8f27c2cfe7a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.669+0000 7f32a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.670+0000 7f32a4ff9640 1 -- 172.21.15.70:0/3461629512 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b4052060 con 0x55562f3b9800 2024-04-01T01:35:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.670+0000 7f32a4ff9640 1 -- 172.21.15.70:0/3461629512 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f32b4003b70 con 0x55562f3b9800 2024-04-01T01:35:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.670+0000 7f32c636f740 1 -- 172.21.15.70:0/3461629512 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32ac019420 con 0x55562f3b9800 2024-04-01T01:35:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.670+0000 7f32a4ff9640 1 -- 172.21.15.70:0/3461629512 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f32b4059050 con 0x55562f3b9800 2024-04-01T01:35:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.670+0000 7f32c636f740 10 client.25976 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.670+0000 7f32c636f740 1 -- 172.21.15.70:0/3461629512 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55562f0ee7d0 con 0x55562f3b9800 2024-04-01T01:35:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.670+0000 7f32a4ff9640 1 -- 172.21.15.70:0/3461629512 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32ac019420 con 0x55562f3b9800 2024-04-01T01:35:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.671+0000 7f32a4ff9640 1 -- 172.21.15.70:0/3461629512 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f32b404d500 con 0x55562f3b9800 2024-04-01T01:35:22.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.671+0000 7f32c636f740 10 client.25976 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:22.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.671+0000 7f32c636f740 10 client.25976 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:35:22.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.671+0000 7f32c636f740 20 client.25976 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:22.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.671+0000 7f32c636f740 1 --2- 172.21.15.70:0/3461629512 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55562f3c99d0 0x55562f3cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.671+0000 7f32c636f740 4 client.25976 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.671+0000 7f32c636f740 1 -- 172.21.15.70:0/3461629512 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55562f31d6f0 con 0x55562f3c99d0 2024-04-01T01:35:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.671+0000 7f32c1526640 1 --2- 172.21.15.70:0/3461629512 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55562f3c99d0 0x55562f3cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.672+0000 7f32c1526640 1 --2- 172.21.15.70:0/3461629512 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55562f3c99d0 0x55562f3cbe70 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=266b3b4bfa593ca server_cookie=4d5ba815c382eca9 in_seq=0 out_seq=0 2024-04-01T01:35:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.672+0000 7f32a4ff9640 10 client.25976 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:35:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.673+0000 7f32a4ff9640 1 -- 172.21.15.70:0/3461629512 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55562f31d6f0 con 0x55562f3c99d0 2024-04-01T01:35:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.673+0000 7f32a4ff9640 10 client.25976 handle_command_reply: tid=0 2024-04-01T01:35:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.673+0000 7f32c636f740 1 client.25976 shutdown 2024-04-01T01:35:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.673+0000 7f32c636f740 1 -- 172.21.15.70:0/3461629512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b9800 msgr2=0x55562f3c54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.673+0000 7f32c636f740 1 --2- 172.21.15.70:0/3461629512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b9800 0x55562f3c54b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f32b4037040 tx=0x7f32b40095c0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.674+0000 7f32c636f740 1 -- 172.21.15.70:0/3461629512 shutdown_connections 2024-04-01T01:35:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.674+0000 7f32c636f740 1 --2- 172.21.15.70:0/3461629512 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55562f3c99d0 0x55562f3cbe70 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.674+0000 7f32c636f740 1 --2- 172.21.15.70:0/3461629512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b9800 0x55562f3c54b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.674+0000 7f32c636f740 1 --2- 172.21.15.70:0/3461629512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b4690 0x55562f3be4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.674+0000 7f32c636f740 1 --2- 172.21.15.70:0/3461629512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b3ca0 0x55562f3bdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.674+0000 7f32c636f740 1 -- 172.21.15.70:0/3461629512 >> 172.21.15.70:0/3461629512 conn(0x55562f3af210 msgr2=0x55562f3a5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.674+0000 7f32a4ff9640 0 client.25976 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:35:22.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.674+0000 7f32c636f740 1 -- 172.21.15.70:0/3461629512 shutdown_connections 2024-04-01T01:35:22.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.674+0000 7f32c636f740 1 -- 172.21.15.70:0/3461629512 wait complete. 2024-04-01T01:35:22.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.674+0000 7f32c636f740 20 client.25976 trim_cache size 0 max 16384 2024-04-01T01:35:22.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.699+0000 7f32c636f740 1 Processor -- start 2024-04-01T01:35:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.699+0000 7f32c636f740 1 -- start start 2024-04-01T01:35:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.700+0000 7f32c636f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b3ca0 0x55562f3bda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.700+0000 7f32c636f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b4690 0x55562f3c5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.700+0000 7f32c636f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b9800 0x55562f3d1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.700+0000 7f32c636f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f32bc1a80d0 con 0x55562f3b4690 2024-04-01T01:35:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.700+0000 7f32c636f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f32bc1a9c10 con 0x55562f3b3ca0 2024-04-01T01:35:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.700+0000 7f32c636f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f32bc1a87a0 con 0x55562f3b9800 2024-04-01T01:35:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.700+0000 7f32c1d27640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b9800 0x55562f3d1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.700+0000 7f32c0d25640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b4690 0x55562f3c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.700+0000 7f32c0d25640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b4690 0x55562f3c5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38856/0 (socket says 172.21.15.70:38856) 2024-04-01T01:35:22.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.700+0000 7f32c0d25640 1 -- 172.21.15.70:0/3103883499 learned_addr learned my addr 172.21.15.70:0/3103883499 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.701+0000 7f32c0d25640 1 -- 172.21.15.70:0/3103883499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b9800 msgr2=0x55562f3d1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.701+0000 7f32c0d25640 1 --2- 172.21.15.70:0/3103883499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b9800 0x55562f3d1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.701+0000 7f32c0d25640 1 -- 172.21.15.70:0/3103883499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b3ca0 msgr2=0x55562f3bda20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:35:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.701+0000 7f32c0d25640 1 --2- 172.21.15.70:0/3103883499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b3ca0 0x55562f3bda20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.701+0000 7f32c0d25640 1 -- 172.21.15.70:0/3103883499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55562f3be830 con 0x55562f3b4690 2024-04-01T01:35:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.701+0000 7f32c0d25640 1 --2- 172.21.15.70:0/3103883499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b4690 0x55562f3c5470 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f32ac009250 tx=0x7f32ac000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33268fd384c1510 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.701+0000 7f32a6ffd640 1 -- 172.21.15.70:0/3103883499 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac019bf0 con 0x55562f3b4690 2024-04-01T01:35:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.702+0000 7f32a6ffd640 1 -- 172.21.15.70:0/3103883499 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32ac019d90 con 0x55562f3b4690 2024-04-01T01:35:22.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.702+0000 7f32a6ffd640 1 -- 172.21.15.70:0/3103883499 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac005040 con 0x55562f3b4690 2024-04-01T01:35:22.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.702+0000 7f32c636f740 1 -- 172.21.15.70:0/3103883499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b4690 msgr2=0x55562f3c5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.702+0000 7f32c636f740 1 --2- 172.21.15.70:0/3103883499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b4690 0x55562f3c5470 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f32ac009250 tx=0x7f32ac000bc0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.703+0000 7f32c636f740 1 -- 172.21.15.70:0/3103883499 shutdown_connections 2024-04-01T01:35:22.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.703+0000 7f32c636f740 1 --2- 172.21.15.70:0/3103883499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b9800 0x55562f3d1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.703+0000 7f32c636f740 1 --2- 172.21.15.70:0/3103883499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b4690 0x55562f3c5470 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.703+0000 7f32c636f740 1 --2- 172.21.15.70:0/3103883499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b3ca0 0x55562f3bda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.703+0000 7f32c636f740 1 -- 172.21.15.70:0/3103883499 >> 172.21.15.70:0/3103883499 conn(0x55562f3af210 msgr2=0x55562f3b59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.703+0000 7f32c636f740 1 -- 172.21.15.70:0/3103883499 shutdown_connections 2024-04-01T01:35:22.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.703+0000 7f32c636f740 1 -- 172.21.15.70:0/3103883499 wait complete. 2024-04-01T01:35:22.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.703+0000 7f32c636f740 1 -- start start 2024-04-01T01:35:22.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.703+0000 7f32c636f740 1 Processor -- start 2024-04-01T01:35:22.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.704+0000 7f32c636f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b3ca0 0x55562f3c2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.704+0000 7f32c636f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b4690 0x55562f3c3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.704+0000 7f32c636f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b9800 0x55562f3c4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.704+0000 7f32c636f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f32bc1a9540 con 0x55562f3b9800 2024-04-01T01:35:22.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.704+0000 7f32c636f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f32bc1a8e70 con 0x55562f3b4690 2024-04-01T01:35:22.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.704+0000 7f32c636f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f32bc1a60b0 con 0x55562f3b3ca0 2024-04-01T01:35:22.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.704+0000 7f32c636f740 10 client.0 _ll_register_callbacks cb 0x7f32b4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:22.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.705+0000 7f32c0d25640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b4690 0x55562f3c3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.705+0000 7f32c1526640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b3ca0 0x55562f3c2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.705+0000 7f32c0d25640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b4690 0x55562f3c3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35240/0 (socket says 172.21.15.70:35240) 2024-04-01T01:35:22.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.705+0000 7f32c0d25640 1 -- 172.21.15.70:0/3831347402 learned_addr learned my addr 172.21.15.70:0/3831347402 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:22.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.705+0000 7f32c1d27640 1 --2- 172.21.15.70:0/3831347402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b9800 0x55562f3c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.706+0000 7f32c0d25640 1 -- 172.21.15.70:0/3831347402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b3ca0 msgr2=0x55562f3c2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.706+0000 7f32c0d25640 1 --2- 172.21.15.70:0/3831347402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b3ca0 0x55562f3c2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.706+0000 7f32c0d25640 1 -- 172.21.15.70:0/3831347402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b9800 msgr2=0x55562f3c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.706+0000 7f32c0d25640 1 --2- 172.21.15.70:0/3831347402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b9800 0x55562f3c4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.706+0000 7f32c0d25640 1 -- 172.21.15.70:0/3831347402 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55562f3c4850 con 0x55562f3b4690 2024-04-01T01:35:22.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.706+0000 7f32c1526640 1 --2- 172.21.15.70:0/3831347402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b3ca0 0x55562f3c2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:22.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.706+0000 7f32c0d25640 1 --2- 172.21.15.70:0/3831347402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b4690 0x55562f3c3310 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f32ac01b010 tx=0x7f32ac019c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d6e17d45806075c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.706+0000 7f32a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:22.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.706+0000 7f32a4ff9640 1 -- 172.21.15.70:0/3831347402 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac00ed40 con 0x55562f3b4690 2024-04-01T01:35:22.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.706+0000 7f32c636f740 1 -- 172.21.15.70:0/3831347402 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32ac019d90 con 0x55562f3b4690 2024-04-01T01:35:22.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.707+0000 7f32a4ff9640 1 -- 172.21.15.70:0/3831347402 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f32ac005040 con 0x55562f3b4690 2024-04-01T01:35:22.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.708+0000 7f32a4ff9640 1 -- 172.21.15.70:0/3831347402 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f32ac041050 con 0x55562f3b4690 2024-04-01T01:35:22.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.708+0000 7f32c636f740 10 client.25979 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:22.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.708+0000 7f32c636f740 1 -- 172.21.15.70:0/3831347402 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55562f42ff10 con 0x55562f3b4690 2024-04-01T01:35:22.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.709+0000 7f32a4ff9640 1 -- 172.21.15.70:0/3831347402 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32ac019d90 con 0x55562f3b4690 2024-04-01T01:35:22.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.709+0000 7f32a4ff9640 1 -- 172.21.15.70:0/3831347402 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f32ac004cb0 con 0x55562f3b4690 2024-04-01T01:35:22.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.709+0000 7f32c636f740 10 client.25979 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:22.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.709+0000 7f32c636f740 10 client.25979 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:35:22.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.709+0000 7f32c636f740 20 client.25979 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:22.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.709+0000 7f32c636f740 1 --2- 172.21.15.70:0/3831347402 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55562f430640 0x55562f432a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.709+0000 7f32c636f740 4 client.25979 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:35:22.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.709+0000 7f32c636f740 1 -- 172.21.15.70:0/3831347402 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55562f317a40 con 0x55562f430640 2024-04-01T01:35:22.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.709+0000 7f32c1526640 1 --2- 172.21.15.70:0/3831347402 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55562f430640 0x55562f432a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.713+0000 7f32c1526640 1 --2- 172.21.15.70:0/3831347402 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55562f430640 0x55562f432a60 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=98091f9e1dd3652a server_cookie=9ae185792c802c09 in_seq=0 out_seq=0 2024-04-01T01:35:22.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.713+0000 7f32a4ff9640 10 client.25979 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:35:22.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.714+0000 7f32a4ff9640 1 -- 172.21.15.70:0/3831347402 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55562f317a40 con 0x55562f430640 2024-04-01T01:35:22.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.714+0000 7f32a4ff9640 10 client.25979 handle_command_reply: tid=0 2024-04-01T01:35:22.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.714+0000 7f32c636f740 1 client.25979 shutdown 2024-04-01T01:35:22.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.714+0000 7f32c636f740 1 -- 172.21.15.70:0/3831347402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b4690 msgr2=0x55562f3c3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.714+0000 7f32c636f740 1 --2- 172.21.15.70:0/3831347402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b4690 0x55562f3c3310 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f32ac01b010 tx=0x7f32ac019c80 comp rx=0 tx=0).stop 2024-04-01T01:35:22.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.714+0000 7f32c636f740 1 -- 172.21.15.70:0/3831347402 shutdown_connections 2024-04-01T01:35:22.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.714+0000 7f32c636f740 1 --2- 172.21.15.70:0/3831347402 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55562f430640 0x55562f432a60 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.714+0000 7f32a4ff9640 0 client.25979 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:35:22.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.714+0000 7f32c636f740 1 --2- 172.21.15.70:0/3831347402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55562f3b9800 0x55562f3c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.714+0000 7f32c636f740 1 --2- 172.21.15.70:0/3831347402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55562f3b4690 0x55562f3c3310 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.715+0000 7f32c636f740 1 --2- 172.21.15.70:0/3831347402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55562f3b3ca0 0x55562f3c2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.715+0000 7f32c636f740 1 -- 172.21.15.70:0/3831347402 >> 172.21.15.70:0/3831347402 conn(0x55562f3af210 msgr2=0x55562f3b2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.715+0000 7f32c636f740 1 -- 172.21.15.70:0/3831347402 shutdown_connections 2024-04-01T01:35:22.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.715+0000 7f32c636f740 1 -- 172.21.15.70:0/3831347402 wait complete. 2024-04-01T01:35:22.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.715+0000 7f32c636f740 20 client.25979 trim_cache size 0 max 16384 2024-04-01T01:35:22.716 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:35:22.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.716+0000 7f32c378a640 1 -- 172.21.15.70:0/2142199347 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32a80a2490 msgr2=0x7f32a80a4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.716+0000 7f32c378a640 1 --2- 172.21.15.70:0/2142199347 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32a80a2490 0x7f32a80a4950 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f32b4006010 tx=0x7f32b402c610 comp rx=0 tx=0).stop 2024-04-01T01:35:22.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.716+0000 7f32c378a640 1 -- 172.21.15.70:0/2142199347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32bc1021f0 msgr2=0x7f32bc1979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.716+0000 7f32c378a640 1 --2- 172.21.15.70:0/2142199347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32bc1021f0 0x7f32bc1979b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f32b800e9f0 tx=0x7f32b800eeb0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.716+0000 7f32c378a640 1 -- 172.21.15.70:0/2142199347 shutdown_connections 2024-04-01T01:35:22.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.716+0000 7f32c378a640 1 --2- 172.21.15.70:0/2142199347 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32a80a2490 0x7f32a80a4950 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.716+0000 7f32c378a640 1 --2- 172.21.15.70:0/2142199347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32bc107450 0x7f32bc19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.716+0000 7f32c378a640 1 --2- 172.21.15.70:0/2142199347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f32bc104b20 0x7f32bc197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.716+0000 7f32c378a640 1 --2- 172.21.15.70:0/2142199347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32bc1021f0 0x7f32bc1979b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.716+0000 7f32c378a640 1 -- 172.21.15.70:0/2142199347 >> 172.21.15.70:0/2142199347 conn(0x7f32bc0fbc00 msgr2=0x7f32bc102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.717+0000 7f32c378a640 1 -- 172.21.15.70:0/2142199347 shutdown_connections 2024-04-01T01:35:22.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.717+0000 7f32c378a640 1 -- 172.21.15.70:0/2142199347 wait complete. 2024-04-01T01:35:22.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:35:22.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.922+0000 7fbe0ea3b640 1 -- 172.21.15.70:0/4108650289 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe081097e0 msgr2=0x7fbe0810e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.922+0000 7fbe0ea3b640 1 --2- 172.21.15.70:0/4108650289 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe081097e0 0x7fbe0810e080 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbe00009ca0 tx=0x7fbe0002c100 comp rx=0 tx=0).stop 2024-04-01T01:35:22.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.922+0000 7fbe0ea3b640 1 -- 172.21.15.70:0/4108650289 shutdown_connections 2024-04-01T01:35:22.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.922+0000 7fbe0ea3b640 1 --2- 172.21.15.70:0/4108650289 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe081097e0 0x7fbe0810e080 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.922+0000 7fbe0ea3b640 1 --2- 172.21.15.70:0/4108650289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe08108e20 0x7fbe081092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.922+0000 7fbe0ea3b640 1 --2- 172.21.15.70:0/4108650289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe08072b60 0x7fbe08072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.922+0000 7fbe0ea3b640 1 -- 172.21.15.70:0/4108650289 >> 172.21.15.70:0/4108650289 conn(0x7fbe0806e630 msgr2=0x7fbe08070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.923+0000 7fbe0ea3b640 1 -- 172.21.15.70:0/4108650289 shutdown_connections 2024-04-01T01:35:22.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.923+0000 7fbe0ea3b640 1 -- 172.21.15.70:0/4108650289 wait complete. 2024-04-01T01:35:22.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.923+0000 7fbe0ea3b640 1 Processor -- start 2024-04-01T01:35:22.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.923+0000 7fbe0ea3b640 1 -- start start 2024-04-01T01:35:22.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.924+0000 7fbe0ea3b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe08072b60 0x7fbe081a65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.924+0000 7fbe0ea3b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe08108e20 0x7fbe081a6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.924+0000 7fbe0ea3b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe081097e0 0x7fbe081adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.924+0000 7fbe0ea3b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0811bf90 con 0x7fbe081097e0 2024-04-01T01:35:22.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.924+0000 7fbe0ea3b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0811be10 con 0x7fbe08108e20 2024-04-01T01:35:22.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.924+0000 7fbe0ea3b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0811c110 con 0x7fbe08072b60 2024-04-01T01:35:22.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.924+0000 7fbe0d238640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe08108e20 0x7fbe081a6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.924+0000 7fbe0da39640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe08072b60 0x7fbe081a65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.924+0000 7fbe0d238640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe08108e20 0x7fbe081a6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35254/0 (socket says 172.21.15.70:35254) 2024-04-01T01:35:22.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.924+0000 7fbe0d238640 1 -- 172.21.15.70:0/2303136598 learned_addr learned my addr 172.21.15.70:0/2303136598 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:22.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.924+0000 7fbe0e23a640 1 --2- 172.21.15.70:0/2303136598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe081097e0 0x7fbe081adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.924+0000 7fbe0d238640 1 -- 172.21.15.70:0/2303136598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe08072b60 msgr2=0x7fbe081a65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.924+0000 7fbe0d238640 1 --2- 172.21.15.70:0/2303136598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe08072b60 0x7fbe081a65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.924+0000 7fbe0d238640 1 -- 172.21.15.70:0/2303136598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe081097e0 msgr2=0x7fbe081adb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.924+0000 7fbe0d238640 1 --2- 172.21.15.70:0/2303136598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe081097e0 0x7fbe081adb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.924+0000 7fbe0d238640 1 -- 172.21.15.70:0/2303136598 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe081ae1f0 con 0x7fbe08108e20 2024-04-01T01:35:22.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.925+0000 7fbe0d238640 1 --2- 172.21.15.70:0/2303136598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe08108e20 0x7fbe081a6b10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400e9f0 tx=0x7fbe0400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.925+0000 7fbdfeffd640 1 -- 172.21.15.70:0/2303136598 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0400cd80 con 0x7fbe08108e20 2024-04-01T01:35:22.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.925+0000 7fbe0ea3b640 1 -- 172.21.15.70:0/2303136598 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe081ae4e0 con 0x7fbe08108e20 2024-04-01T01:35:22.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.925+0000 7fbe0ea3b640 1 -- 172.21.15.70:0/2303136598 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe081aea80 con 0x7fbe08108e20 2024-04-01T01:35:22.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.925+0000 7fbdfeffd640 1 -- 172.21.15.70:0/2303136598 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe040108e0 con 0x7fbe08108e20 2024-04-01T01:35:22.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.925+0000 7fbdfeffd640 1 -- 172.21.15.70:0/2303136598 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe04018920 con 0x7fbe08108e20 2024-04-01T01:35:22.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.929+0000 7fbe12881740 1 Processor -- start 2024-04-01T01:35:22.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.929+0000 7fbe12881740 1 -- start start 2024-04-01T01:35:22.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.929+0000 7fbdfeffd640 1 -- 172.21.15.70:0/2303136598 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fbe04010430 con 0x7fbe08108e20 2024-04-01T01:35:22.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe12881740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf56b6a0 0x55fdaf56bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe12881740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 0x55fdaf570b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe12881740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf571200 0x55fdaf573690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe12881740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fdaf4d01b0 con 0x55fdaf56c090 2024-04-01T01:35:22.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe12881740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55fdaf478c00 con 0x55fdaf56b6a0 2024-04-01T01:35:22.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe12881740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fdaf4cc250 con 0x55fdaf571200 2024-04-01T01:35:22.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe0da39640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 0x55fdaf570b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe0e23a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf56b6a0 0x55fdaf56bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe0e23a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf56b6a0 0x55fdaf56bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35262/0 (socket says 172.21.15.70:35262) 2024-04-01T01:35:22.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe0da39640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 0x55fdaf570b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38904/0 (socket says 172.21.15.70:38904) 2024-04-01T01:35:22.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe0e23a640 1 -- 172.21.15.70:0/665455083 learned_addr learned my addr 172.21.15.70:0/665455083 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:22.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe0d238640 1 --2- 172.21.15.70:0/665455083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf571200 0x55fdaf573690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe0e23a640 1 -- 172.21.15.70:0/665455083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf571200 msgr2=0x55fdaf573690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe0e23a640 1 --2- 172.21.15.70:0/665455083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf571200 0x55fdaf573690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe0e23a640 1 -- 172.21.15.70:0/665455083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 msgr2=0x55fdaf570b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe0e23a640 1 --2- 172.21.15.70:0/665455083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 0x55fdaf570b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.930+0000 7fbe0e23a640 1 -- 172.21.15.70:0/665455083 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdaf4983d0 con 0x55fdaf56b6a0 2024-04-01T01:35:22.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.931+0000 7fbe0e23a640 1 --2- 172.21.15.70:0/665455083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf56b6a0 0x55fdaf56bac0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbe00005cc0 tx=0x7fbe000093e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15946bcd3011a9b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.931+0000 7fbdfeffd640 1 --2- 172.21.15.70:0/2303136598 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbdf40a2520 0x7fbdf40a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.931+0000 7fbdf3fff640 1 -- 172.21.15.70:0/665455083 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe00043020 con 0x55fdaf56b6a0 2024-04-01T01:35:22.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.931+0000 7fbdf3fff640 1 -- 172.21.15.70:0/665455083 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe00032dc0 con 0x55fdaf56b6a0 2024-04-01T01:35:22.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.931+0000 7fbdf3fff640 1 -- 172.21.15.70:0/665455083 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe00039040 con 0x55fdaf56b6a0 2024-04-01T01:35:22.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.931+0000 7fbe0da39640 1 --2- 172.21.15.70:0/2303136598 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbdf40a2520 0x7fbdf40a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.931+0000 7fbdfeffd640 1 -- 172.21.15.70:0/2303136598 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fbe040ced80 con 0x7fbe08108e20 2024-04-01T01:35:22.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.932+0000 7fbe12881740 1 -- 172.21.15.70:0/665455083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf56b6a0 msgr2=0x55fdaf56bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.932+0000 7fbe0da39640 1 --2- 172.21.15.70:0/2303136598 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbdf40a2520 0x7fbdf40a49e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x55fdaf56c490 tx=0x7fbdf8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.932+0000 7fbe12881740 1 --2- 172.21.15.70:0/665455083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf56b6a0 0x55fdaf56bac0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbe00005cc0 tx=0x7fbe000093e0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.932+0000 7fbe12881740 1 -- 172.21.15.70:0/665455083 shutdown_connections 2024-04-01T01:35:22.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.932+0000 7fbe12881740 1 --2- 172.21.15.70:0/665455083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf571200 0x55fdaf573690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.932+0000 7fbe12881740 1 --2- 172.21.15.70:0/665455083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 0x55fdaf570b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.932+0000 7fbe12881740 1 --2- 172.21.15.70:0/665455083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf56b6a0 0x55fdaf56bac0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.932+0000 7fbe12881740 1 -- 172.21.15.70:0/665455083 >> 172.21.15.70:0/665455083 conn(0x55fdaf566c10 msgr2=0x55fdaf567010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.932+0000 7fbe12881740 1 -- 172.21.15.70:0/665455083 shutdown_connections 2024-04-01T01:35:22.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.933+0000 7fbe12881740 1 -- 172.21.15.70:0/665455083 wait complete. 2024-04-01T01:35:22.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.933+0000 7fbe12881740 1 -- start start 2024-04-01T01:35:22.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.933+0000 7fbe12881740 1 Processor -- start 2024-04-01T01:35:22.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.934+0000 7fbe12881740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56b6a0 0x55fdaf575be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.934+0000 7fbe12881740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 0x55fdaf576120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.934+0000 7fbe12881740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf571200 0x55fdaf57d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.934+0000 7fbe12881740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fdaf574150 con 0x55fdaf56c090 2024-04-01T01:35:22.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.934+0000 7fbe12881740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55fdaf46e470 con 0x55fdaf571200 2024-04-01T01:35:22.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.934+0000 7fbe12881740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fdaf5742d0 con 0x55fdaf56b6a0 2024-04-01T01:35:22.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.934+0000 7fbe0da39640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56b6a0 0x55fdaf575be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.934+0000 7fbe0e23a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf571200 0x55fdaf57d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.934+0000 7fbe0da39640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56b6a0 0x55fdaf575be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37786/0 (socket says 172.21.15.70:37786) 2024-04-01T01:35:22.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.934+0000 7fbe0da39640 1 -- 172.21.15.70:0/1280092839 learned_addr learned my addr 172.21.15.70:0/1280092839 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:22.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.935+0000 7fbe12881740 10 client.0 _ll_register_callbacks cb 0x55fdaf48bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:22.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.935+0000 7fbe0d238640 1 --2- 172.21.15.70:0/1280092839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 0x55fdaf576120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.935+0000 7fbe0da39640 1 -- 172.21.15.70:0/1280092839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf571200 msgr2=0x55fdaf57d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.935+0000 7fbe0da39640 1 --2- 172.21.15.70:0/1280092839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf571200 0x55fdaf57d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.935+0000 7fbe0da39640 1 -- 172.21.15.70:0/1280092839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 msgr2=0x55fdaf576120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.935+0000 7fbe0da39640 1 --2- 172.21.15.70:0/1280092839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 0x55fdaf576120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.935+0000 7fbe0da39640 1 -- 172.21.15.70:0/1280092839 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdaf1e43f0 con 0x55fdaf56b6a0 2024-04-01T01:35:22.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.936+0000 7fbe0da39640 1 --2- 172.21.15.70:0/1280092839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56b6a0 0x55fdaf575be0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8004600 tx=0x7fbdf8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b5af0eb71938ae1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.936+0000 7fbdf17fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:35:22.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.936+0000 7fbdf17fa640 1 -- 172.21.15.70:0/1280092839 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf801f630 con 0x55fdaf56b6a0 2024-04-01T01:35:22.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.936+0000 7fbdf17fa640 1 -- 172.21.15.70:0/1280092839 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fbdf801f7d0 con 0x55fdaf56b6a0 2024-04-01T01:35:22.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.936+0000 7fbe12881740 1 -- 172.21.15.70:0/1280092839 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe00032dc0 con 0x55fdaf56b6a0 2024-04-01T01:35:22.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.937+0000 7fbdf17fa640 1 -- 172.21.15.70:0/1280092839 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbdf801f9d0 con 0x55fdaf56b6a0 2024-04-01T01:35:22.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.937+0000 7fbe12881740 10 client.25903 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:22.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.937+0000 7fbe12881740 1 -- 172.21.15.70:0/1280092839 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fdaf2ae7d0 con 0x55fdaf56b6a0 2024-04-01T01:35:22.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.937+0000 7fbdf17fa640 1 -- 172.21.15.70:0/1280092839 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbdf801fb70 con 0x55fdaf56b6a0 2024-04-01T01:35:22.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.937+0000 7fbdf17fa640 1 -- 172.21.15.70:0/1280092839 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fbdf8027070 con 0x55fdaf56b6a0 2024-04-01T01:35:22.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.937+0000 7fbe12881740 10 client.25903 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:22.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.937+0000 7fbe12881740 10 client.25903 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:35:22.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.937+0000 7fbe12881740 20 client.25903 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:22.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.937+0000 7fbe12881740 1 --2- 172.21.15.70:0/1280092839 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55fdaf581810 0x55fdaf583c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.937+0000 7fbe12881740 4 client.25903 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:22.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.937+0000 7fbe12881740 1 -- 172.21.15.70:0/1280092839 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbe00032dc0 con 0x55fdaf581810 2024-04-01T01:35:22.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.937+0000 7fbe0d238640 1 --2- 172.21.15.70:0/1280092839 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55fdaf581810 0x55fdaf583c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.938+0000 7fbdf17fa640 10 client.25903 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:35:22.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.939+0000 7fbe0d238640 1 --2- 172.21.15.70:0/1280092839 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55fdaf581810 0x55fdaf583c70 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4bfbd51e54db8adb server_cookie=9242e66f0343caa8 in_seq=0 out_seq=0 2024-04-01T01:35:22.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.940+0000 7fbdf17fa640 1 -- 172.21.15.70:0/1280092839 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fbe00032dc0 con 0x55fdaf581810 2024-04-01T01:35:22.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.940+0000 7fbdf17fa640 10 client.25903 handle_command_reply: tid=0 2024-04-01T01:35:22.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.940+0000 7fbe12881740 1 client.25903 shutdown 2024-04-01T01:35:22.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.941+0000 7fbe12881740 1 -- 172.21.15.70:0/1280092839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56b6a0 msgr2=0x55fdaf575be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.941+0000 7fbe12881740 1 --2- 172.21.15.70:0/1280092839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56b6a0 0x55fdaf575be0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8004600 tx=0x7fbdf8004730 comp rx=0 tx=0).stop 2024-04-01T01:35:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.941+0000 7fbe12881740 1 -- 172.21.15.70:0/1280092839 shutdown_connections 2024-04-01T01:35:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.941+0000 7fbe12881740 1 --2- 172.21.15.70:0/1280092839 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55fdaf581810 0x55fdaf583c70 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.941+0000 7fbe12881740 1 --2- 172.21.15.70:0/1280092839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf571200 0x55fdaf57d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.941+0000 7fbe12881740 1 --2- 172.21.15.70:0/1280092839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 0x55fdaf576120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.941+0000 7fbe12881740 1 --2- 172.21.15.70:0/1280092839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56b6a0 0x55fdaf575be0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.941+0000 7fbe12881740 1 -- 172.21.15.70:0/1280092839 >> 172.21.15.70:0/1280092839 conn(0x55fdaf566c10 msgr2=0x55fdaf565f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.941+0000 7fbdf17fa640 0 client.25903 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:35:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.941+0000 7fbe12881740 1 -- 172.21.15.70:0/1280092839 shutdown_connections 2024-04-01T01:35:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.941+0000 7fbe12881740 1 -- 172.21.15.70:0/1280092839 wait complete. 2024-04-01T01:35:22.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.941+0000 7fbe12881740 20 client.25903 trim_cache size 0 max 16384 2024-04-01T01:35:22.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.963+0000 7fbe12881740 1 Processor -- start 2024-04-01T01:35:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.963+0000 7fbe12881740 1 -- start start 2024-04-01T01:35:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe12881740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56b6a0 0x55fdaf57df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe12881740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 0x55fdaf57d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe12881740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf571200 0x55fdaf583820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe0d238640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 0x55fdaf57d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe0d238640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 0x55fdaf57d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38918/0 (socket says 172.21.15.70:38918) 2024-04-01T01:35:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe0da39640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56b6a0 0x55fdaf57df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe0d238640 1 -- 172.21.15.70:0/1161657991 learned_addr learned my addr 172.21.15.70:0/1161657991 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe12881740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec00fd90 con 0x55fdaf56c090 2024-04-01T01:35:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe12881740 1 -- 172.21.15.70:0/1161657991 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec0118d0 con 0x55fdaf571200 2024-04-01T01:35:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe12881740 1 -- 172.21.15.70:0/1161657991 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec010460 con 0x55fdaf56b6a0 2024-04-01T01:35:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe0e23a640 1 --2- 172.21.15.70:0/1161657991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf571200 0x55fdaf583820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe0e23a640 1 -- 172.21.15.70:0/1161657991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56b6a0 msgr2=0x55fdaf57df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe0e23a640 1 --2- 172.21.15.70:0/1161657991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56b6a0 0x55fdaf57df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe0e23a640 1 -- 172.21.15.70:0/1161657991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 msgr2=0x55fdaf57d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe0e23a640 1 --2- 172.21.15.70:0/1161657991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 0x55fdaf57d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe0e23a640 1 -- 172.21.15.70:0/1161657991 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdaf567b80 con 0x55fdaf571200 2024-04-01T01:35:22.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.964+0000 7fbe0d238640 1 --2- 172.21.15.70:0/1161657991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 0x55fdaf57d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.965+0000 7fbe0e23a640 1 --2- 172.21.15.70:0/1161657991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf571200 0x55fdaf583820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbe0002c710 tx=0x7fbe00005fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2898ce9bee1e217 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.965+0000 7fbdf37fe640 1 -- 172.21.15.70:0/1161657991 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0004c070 con 0x55fdaf571200 2024-04-01T01:35:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.965+0000 7fbdf37fe640 1 -- 172.21.15.70:0/1161657991 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe00032dc0 con 0x55fdaf571200 2024-04-01T01:35:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.965+0000 7fbdf37fe640 1 -- 172.21.15.70:0/1161657991 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe00043070 con 0x55fdaf571200 2024-04-01T01:35:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.966+0000 7fbe12881740 1 -- 172.21.15.70:0/1161657991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf571200 msgr2=0x55fdaf583820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.966+0000 7fbe12881740 1 --2- 172.21.15.70:0/1161657991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf571200 0x55fdaf583820 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbe0002c710 tx=0x7fbe00005fa0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.966+0000 7fbe12881740 1 -- 172.21.15.70:0/1161657991 shutdown_connections 2024-04-01T01:35:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.966+0000 7fbe12881740 1 --2- 172.21.15.70:0/1161657991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf571200 0x55fdaf583820 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.966+0000 7fbe12881740 1 --2- 172.21.15.70:0/1161657991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56c090 0x55fdaf57d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.966+0000 7fbe12881740 1 --2- 172.21.15.70:0/1161657991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56b6a0 0x55fdaf57df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.966+0000 7fbe12881740 1 -- 172.21.15.70:0/1161657991 >> 172.21.15.70:0/1161657991 conn(0x55fdaf566c10 msgr2=0x55fdaf57d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.966+0000 7fbe12881740 1 -- 172.21.15.70:0/1161657991 shutdown_connections 2024-04-01T01:35:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.967+0000 7fbe12881740 1 -- 172.21.15.70:0/1161657991 wait complete. 2024-04-01T01:35:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.967+0000 7fbe12881740 1 -- start start 2024-04-01T01:35:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.967+0000 7fbe12881740 1 Processor -- start 2024-04-01T01:35:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.968+0000 7fbe12881740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56b6a0 0x55fdaf57a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.968+0000 7fbe12881740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56c090 0x55fdaf57a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.968+0000 7fbe12881740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf571200 0x55fdaf57b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.968+0000 7fbe0da39640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56b6a0 0x55fdaf57a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.968+0000 7fbe0da39640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56b6a0 0x55fdaf57a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38934/0 (socket says 172.21.15.70:38934) 2024-04-01T01:35:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.968+0000 7fbe0d238640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56c090 0x55fdaf57a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.968+0000 7fbe0d238640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56c090 0x55fdaf57a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37802/0 (socket says 172.21.15.70:37802) 2024-04-01T01:35:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.968+0000 7fbe12881740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec011200 con 0x55fdaf56b6a0 2024-04-01T01:35:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.968+0000 7fbe12881740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec010b30 con 0x55fdaf571200 2024-04-01T01:35:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.968+0000 7fbe12881740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec00dd70 con 0x55fdaf56c090 2024-04-01T01:35:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.969+0000 7fbe0da39640 1 -- 172.21.15.70:0/2177412684 learned_addr learned my addr 172.21.15.70:0/2177412684 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.969+0000 7fbe12881740 10 client.0 _ll_register_callbacks cb 0x7fbdf80044c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.969+0000 7fbe0e23a640 1 --2- 172.21.15.70:0/2177412684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf571200 0x55fdaf57b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.969+0000 7fbe0d238640 1 -- 172.21.15.70:0/2177412684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf571200 msgr2=0x55fdaf57b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.969+0000 7fbe0d238640 1 --2- 172.21.15.70:0/2177412684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf571200 0x55fdaf57b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.969+0000 7fbe0d238640 1 -- 172.21.15.70:0/2177412684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56b6a0 msgr2=0x55fdaf57a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.969+0000 7fbe0d238640 1 --2- 172.21.15.70:0/2177412684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56b6a0 0x55fdaf57a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.969+0000 7fbe0d238640 1 -- 172.21.15.70:0/2177412684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdaf57bde0 con 0x55fdaf56c090 2024-04-01T01:35:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.969+0000 7fbe0d238640 1 --2- 172.21.15.70:0/2177412684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56c090 0x55fdaf57a850 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbe040cc4b0 tx=0x7fbe040cc380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b80ce583cd2fae3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.969+0000 7fbdf17fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:35:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.969+0000 7fbdf17fa640 1 -- 172.21.15.70:0/2177412684 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe04010720 con 0x55fdaf56c090 2024-04-01T01:35:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.970+0000 7fbdf17fa640 1 -- 172.21.15.70:0/2177412684 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fbe04097ae0 con 0x55fdaf56c090 2024-04-01T01:35:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.970+0000 7fbe12881740 1 -- 172.21.15.70:0/2177412684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe00032dc0 con 0x55fdaf56c090 2024-04-01T01:35:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.970+0000 7fbdf17fa640 1 -- 172.21.15.70:0/2177412684 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbe0409bb70 con 0x55fdaf56c090 2024-04-01T01:35:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.970+0000 7fbe12881740 10 client.25909 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.970+0000 7fbe12881740 1 -- 172.21.15.70:0/2177412684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fdaf57d1b0 con 0x55fdaf56c090 2024-04-01T01:35:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.970+0000 7fbdf17fa640 1 -- 172.21.15.70:0/2177412684 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe00032dc0 con 0x55fdaf56c090 2024-04-01T01:35:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.971+0000 7fbdf17fa640 1 -- 172.21.15.70:0/2177412684 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fbe040971c0 con 0x55fdaf56c090 2024-04-01T01:35:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.971+0000 7fbe12881740 10 client.25909 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.971+0000 7fbe12881740 10 client.25909 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:35:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.971+0000 7fbe12881740 20 client.25909 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.971+0000 7fbe12881740 1 --2- 172.21.15.70:0/2177412684 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55fdaf5e80e0 0x55fdaf5ea500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.971+0000 7fbe12881740 4 client.25909 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:35:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.972+0000 7fbe12881740 1 -- 172.21.15.70:0/2177412684 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fdaf4dd6f0 con 0x55fdaf5e80e0 2024-04-01T01:35:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.972+0000 7fbe0da39640 1 --2- 172.21.15.70:0/2177412684 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55fdaf5e80e0 0x55fdaf5ea500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.972+0000 7fbe0da39640 1 --2- 172.21.15.70:0/2177412684 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55fdaf5e80e0 0x55fdaf5ea500 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9387d2e4a630466c server_cookie=2c8bb95d603f1e85 in_seq=0 out_seq=0 2024-04-01T01:35:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.972+0000 7fbdf17fa640 10 client.25909 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:35:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.972+0000 7fbdf17fa640 1 -- 172.21.15.70:0/2177412684 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fdaf4dd6f0 con 0x55fdaf5e80e0 2024-04-01T01:35:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.972+0000 7fbdf17fa640 10 client.25909 handle_command_reply: tid=0 2024-04-01T01:35:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.972+0000 7fbe12881740 1 client.25909 shutdown 2024-04-01T01:35:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.973+0000 7fbe12881740 1 -- 172.21.15.70:0/2177412684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56c090 msgr2=0x55fdaf57a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.973+0000 7fbe12881740 1 --2- 172.21.15.70:0/2177412684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56c090 0x55fdaf57a850 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbe040cc4b0 tx=0x7fbe040cc380 comp rx=0 tx=0).stop 2024-04-01T01:35:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.973+0000 7fbe12881740 1 -- 172.21.15.70:0/2177412684 shutdown_connections 2024-04-01T01:35:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.973+0000 7fbe12881740 1 --2- 172.21.15.70:0/2177412684 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55fdaf5e80e0 0x55fdaf5ea500 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.973+0000 7fbe12881740 1 --2- 172.21.15.70:0/2177412684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fdaf571200 0x55fdaf57b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.973+0000 7fbe12881740 1 --2- 172.21.15.70:0/2177412684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fdaf56c090 0x55fdaf57a850 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.973+0000 7fbe12881740 1 --2- 172.21.15.70:0/2177412684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fdaf56b6a0 0x55fdaf57a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.973+0000 7fbe12881740 1 -- 172.21.15.70:0/2177412684 >> 172.21.15.70:0/2177412684 conn(0x55fdaf566c10 msgr2=0x55fdaf5757d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.973+0000 7fbdf17fa640 0 client.25909 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:35:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.973+0000 7fbe12881740 1 -- 172.21.15.70:0/2177412684 shutdown_connections 2024-04-01T01:35:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.973+0000 7fbe12881740 1 -- 172.21.15.70:0/2177412684 wait complete. 2024-04-01T01:35:22.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.973+0000 7fbe12881740 20 client.25909 trim_cache size 0 max 16384 2024-04-01T01:35:22.976 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:35:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.974+0000 7fbe0ea3b640 1 -- 172.21.15.70:0/2303136598 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbdf40a2520 msgr2=0x7fbdf40a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.974+0000 7fbe0ea3b640 1 --2- 172.21.15.70:0/2303136598 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbdf40a2520 0x7fbdf40a49e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x55fdaf56c490 tx=0x7fbdf8005f90 comp rx=0 tx=0).stop 2024-04-01T01:35:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.974+0000 7fbe0ea3b640 1 -- 172.21.15.70:0/2303136598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe08108e20 msgr2=0x7fbe081a6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:22.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.974+0000 7fbe0ea3b640 1 --2- 172.21.15.70:0/2303136598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe08108e20 0x7fbe081a6b10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400e9f0 tx=0x7fbe0400eeb0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.975+0000 7fbe0ea3b640 1 -- 172.21.15.70:0/2303136598 shutdown_connections 2024-04-01T01:35:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.975+0000 7fbe0ea3b640 1 --2- 172.21.15.70:0/2303136598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe081097e0 0x7fbe081adb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.975+0000 7fbe0ea3b640 1 --2- 172.21.15.70:0/2303136598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe08108e20 0x7fbe081a6b10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.975+0000 7fbe0ea3b640 1 --2- 172.21.15.70:0/2303136598 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbdf40a2520 0x7fbdf40a49e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.975+0000 7fbe0ea3b640 1 --2- 172.21.15.70:0/2303136598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe08072b60 0x7fbe081a65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.975+0000 7fbe0ea3b640 1 -- 172.21.15.70:0/2303136598 >> 172.21.15.70:0/2303136598 conn(0x7fbe0806e630 msgr2=0x7fbe08070270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.975+0000 7fbe0ea3b640 1 -- 172.21.15.70:0/2303136598 shutdown_connections 2024-04-01T01:35:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:22.975+0000 7fbe0ea3b640 1 -- 172.21.15.70:0/2303136598 wait complete. 2024-04-01T01:35:22.993 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:22 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3177810315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:35:22.994 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:35:22.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:22 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3177810315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:35:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:22 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3177810315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:35:23.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.147+0000 7f6758bb5640 1 -- 172.21.15.70:0/2648291472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6754104b00 msgr2=0x7f6754106ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.147+0000 7f6758bb5640 1 --2- 172.21.15.70:0/2648291472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6754104b00 0x7f6754106ef0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6748009900 tx=0x7f674802bc30 comp rx=0 tx=0).stop 2024-04-01T01:35:23.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.147+0000 7f6758bb5640 1 -- 172.21.15.70:0/2648291472 shutdown_connections 2024-04-01T01:35:23.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.147+0000 7f6758bb5640 1 --2- 172.21.15.70:0/2648291472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6754107430 0x7f6754109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.147+0000 7f6758bb5640 1 --2- 172.21.15.70:0/2648291472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6754104b00 0x7f6754106ef0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.147+0000 7f6758bb5640 1 --2- 172.21.15.70:0/2648291472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67541021d0 0x7f67541045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.147+0000 7f6758bb5640 1 -- 172.21.15.70:0/2648291472 >> 172.21.15.70:0/2648291472 conn(0x7f67540fbbe0 msgr2=0x7f67540fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:23.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.147+0000 7f6758bb5640 1 -- 172.21.15.70:0/2648291472 shutdown_connections 2024-04-01T01:35:23.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.147+0000 7f6758bb5640 1 -- 172.21.15.70:0/2648291472 wait complete. 2024-04-01T01:35:23.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.148+0000 7f6758bb5640 1 Processor -- start 2024-04-01T01:35:23.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.148+0000 7f6758bb5640 1 -- start start 2024-04-01T01:35:23.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.148+0000 7f6758bb5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67541021d0 0x7f6754101b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.148+0000 7f6758bb5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6754104b00 0x7f6754100180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.148+0000 7f6758bb5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6754107430 0x7f67541006c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.148+0000 7f6758bb5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f675410d280 con 0x7f67541021d0 2024-04-01T01:35:23.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.148+0000 7f6758bb5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f675410d100 con 0x7f6754104b00 2024-04-01T01:35:23.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.148+0000 7f6758bb5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f675410d400 con 0x7f6754107430 2024-04-01T01:35:23.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.148+0000 7f675259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67541021d0 0x7f6754101b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.148+0000 7f675259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67541021d0 0x7f6754101b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38942/0 (socket says 172.21.15.70:38942) 2024-04-01T01:35:23.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.148+0000 7f675259c640 1 -- 172.21.15.70:0/834177795 learned_addr learned my addr 172.21.15.70:0/834177795 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:23.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.149+0000 7f6752d9d640 1 --2- 172.21.15.70:0/834177795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6754107430 0x7f67541006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.149+0000 7f6751d9b640 1 --2- 172.21.15.70:0/834177795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6754104b00 0x7f6754100180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.149+0000 7f675259c640 1 -- 172.21.15.70:0/834177795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6754107430 msgr2=0x7f67541006c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.149+0000 7f675259c640 1 --2- 172.21.15.70:0/834177795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6754107430 0x7f67541006c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.149+0000 7f675259c640 1 -- 172.21.15.70:0/834177795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6754104b00 msgr2=0x7f6754100180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.149+0000 7f675259c640 1 --2- 172.21.15.70:0/834177795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6754104b00 0x7f6754100180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.149+0000 7f675259c640 1 -- 172.21.15.70:0/834177795 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6754100c00 con 0x7f67541021d0 2024-04-01T01:35:23.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.149+0000 7f6751d9b640 1 --2- 172.21.15.70:0/834177795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6754104b00 0x7f6754100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:35:23.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.149+0000 7f675259c640 1 --2- 172.21.15.70:0/834177795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67541021d0 0x7f6754101b40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f674400ebc0 tx=0x7f674400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:23.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.149+0000 7f67437fe640 1 -- 172.21.15.70:0/834177795 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744010630 con 0x7f67541021d0 2024-04-01T01:35:23.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.149+0000 7f6758bb5640 1 -- 172.21.15.70:0/834177795 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6754100ef0 con 0x7f67541021d0 2024-04-01T01:35:23.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.149+0000 7f67437fe640 1 -- 172.21.15.70:0/834177795 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6744010c80 con 0x7f67541021d0 2024-04-01T01:35:23.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.149+0000 7f67437fe640 1 -- 172.21.15.70:0/834177795 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744018b30 con 0x7f67541021d0 2024-04-01T01:35:23.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.149+0000 7f6758bb5640 1 -- 172.21.15.70:0/834177795 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67541a6520 con 0x7f67541021d0 2024-04-01T01:35:23.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.151+0000 7f67437fe640 1 -- 172.21.15.70:0/834177795 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f6744018cd0 con 0x7f67541021d0 2024-04-01T01:35:23.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.151+0000 7f675b796740 1 Processor -- start 2024-04-01T01:35:23.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f675b796740 1 -- start start 2024-04-01T01:35:23.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f675b796740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56349185b490 0x56349185b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f675b796740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185be80 0x563491860980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f675b796740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x563491863480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f675b796740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5634917c01b0 con 0x563491860ff0 2024-04-01T01:35:23.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f675b796740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563491768c00 con 0x56349185be80 2024-04-01T01:35:23.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f675b796740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5634917bc250 con 0x56349185b490 2024-04-01T01:35:23.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f67437fe640 1 --2- 172.21.15.70:0/834177795 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f673c0a24a0 0x7f673c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f6751d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x563491863480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f6751d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x563491863480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38950/0 (socket says 172.21.15.70:38950) 2024-04-01T01:35:23.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f6751d9b640 1 -- 172.21.15.70:0/3911348704 learned_addr learned my addr 172.21.15.70:0/3911348704 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:23.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f6752d9d640 1 --2- 172.21.15.70:0/3911348704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56349185b490 0x56349185b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f675259c640 1 --2- 172.21.15.70:0/3911348704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185be80 0x563491860980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f67437fe640 1 -- 172.21.15.70:0/834177795 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f67440cf150 con 0x7f67541021d0 2024-04-01T01:35:23.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f6752d9d640 1 --2- 172.21.15.70:0/834177795 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f673c0a24a0 0x7f673c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f6752d9d640 1 -- 172.21.15.70:0/3911348704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185be80 msgr2=0x563491860980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:23.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f6752d9d640 1 --2- 172.21.15.70:0/3911348704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185be80 0x563491860980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f6752d9d640 1 -- 172.21.15.70:0/3911348704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 msgr2=0x563491863480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:23.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f6752d9d640 1 --2- 172.21.15.70:0/3911348704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x563491863480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f6752d9d640 1 -- 172.21.15.70:0/3911348704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634917883d0 con 0x56349185b490 2024-04-01T01:35:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f675259c640 1 --2- 172.21.15.70:0/3911348704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185be80 0x563491860980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.152+0000 7f6751d9b640 1 --2- 172.21.15.70:0/3911348704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x563491863480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.153+0000 7f6752d9d640 1 --2- 172.21.15.70:0/3911348704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56349185b490 0x56349185b8b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f674c0119b0 tx=0x7f674c011db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f539c688f34f4c47 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.153+0000 7f6752d9d640 1 --2- 172.21.15.70:0/834177795 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f673c0a24a0 0x7f673c0a4960 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f674c009710 tx=0x7f674c01e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.153+0000 7f6740ff9640 1 -- 172.21.15.70:0/3911348704 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c004250 con 0x56349185b490 2024-04-01T01:35:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.153+0000 7f6740ff9640 1 -- 172.21.15.70:0/3911348704 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f674c011560 con 0x56349185b490 2024-04-01T01:35:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.153+0000 7f6740ff9640 1 -- 172.21.15.70:0/3911348704 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c0049e0 con 0x56349185b490 2024-04-01T01:35:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.153+0000 7f675b796740 1 -- 172.21.15.70:0/3911348704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56349185b490 msgr2=0x56349185b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.153+0000 7f675b796740 1 --2- 172.21.15.70:0/3911348704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56349185b490 0x56349185b8b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f674c0119b0 tx=0x7f674c011db0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675b796740 1 -- 172.21.15.70:0/3911348704 shutdown_connections 2024-04-01T01:35:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675b796740 1 --2- 172.21.15.70:0/3911348704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x563491863480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675b796740 1 --2- 172.21.15.70:0/3911348704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185be80 0x563491860980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675b796740 1 --2- 172.21.15.70:0/3911348704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56349185b490 0x56349185b8b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675b796740 1 -- 172.21.15.70:0/3911348704 >> 172.21.15.70:0/3911348704 conn(0x563491856a00 msgr2=0x563491856e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675b796740 1 -- 172.21.15.70:0/3911348704 shutdown_connections 2024-04-01T01:35:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675b796740 1 -- 172.21.15.70:0/3911348704 wait complete. 2024-04-01T01:35:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675b796740 1 -- start start 2024-04-01T01:35:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675b796740 1 Processor -- start 2024-04-01T01:35:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675b796740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185b490 0x563491865a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675b796740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56349185be80 0x563491865f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675b796740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563491860ff0 0x56349186cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675b796740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563491863f40 con 0x56349185be80 2024-04-01T01:35:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675b796740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56349175e470 con 0x56349185b490 2024-04-01T01:35:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675b796740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5634918640c0 con 0x563491860ff0 2024-04-01T01:35:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675b796740 10 client.0 _ll_register_callbacks cb 0x56349177bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f6751d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56349185be80 0x563491865f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f6751d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56349185be80 0x563491865f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38966/0 (socket says 172.21.15.70:38966) 2024-04-01T01:35:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f6751d9b640 1 -- 172.21.15.70:0/1317181636 learned_addr learned my addr 172.21.15.70:0/1317181636 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f6752d9d640 1 --2- 172.21.15.70:0/1317181636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563491860ff0 0x56349186cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.154+0000 7f675259c640 1 --2- 172.21.15.70:0/1317181636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185b490 0x563491865a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.155+0000 7f6751d9b640 1 -- 172.21.15.70:0/1317181636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563491860ff0 msgr2=0x56349186cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:23.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.155+0000 7f6751d9b640 1 --2- 172.21.15.70:0/1317181636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563491860ff0 0x56349186cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.155+0000 7f6751d9b640 1 -- 172.21.15.70:0/1317181636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185b490 msgr2=0x563491865a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:23.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.155+0000 7f6751d9b640 1 --2- 172.21.15.70:0/1317181636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185b490 0x563491865a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.155+0000 7f6751d9b640 1 -- 172.21.15.70:0/1317181636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634914d43f0 con 0x56349185be80 2024-04-01T01:35:23.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.155+0000 7f6751d9b640 1 --2- 172.21.15.70:0/1317181636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56349185be80 0x563491865f40 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6748009230 tx=0x7f6748004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4e75f7690d2760 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:23.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.155+0000 7f6739ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:35:23.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.155+0000 7f6739ffb640 1 -- 172.21.15.70:0/1317181636 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748005d50 con 0x56349185be80 2024-04-01T01:35:23.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.155+0000 7f6739ffb640 1 -- 172.21.15.70:0/1317181636 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6748005230 con 0x56349185be80 2024-04-01T01:35:23.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.155+0000 7f675b796740 1 -- 172.21.15.70:0/1317181636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5634917cd6f0 con 0x56349185be80 2024-04-01T01:35:23.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.155+0000 7f6739ffb640 1 -- 172.21.15.70:0/1317181636 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6748050050 con 0x56349185be80 2024-04-01T01:35:23.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.155+0000 7f675b796740 10 client.16182 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:23.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.155+0000 7f675b796740 1 -- 172.21.15.70:0/1317181636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56349159e7d0 con 0x56349185be80 2024-04-01T01:35:23.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.155+0000 7f6739ffb640 1 -- 172.21.15.70:0/1317181636 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5634917cd6f0 con 0x56349185be80 2024-04-01T01:35:23.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.156+0000 7f6739ffb640 1 -- 172.21.15.70:0/1317181636 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6748003680 con 0x56349185be80 2024-04-01T01:35:23.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.156+0000 7f675b796740 10 client.16182 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:23.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.156+0000 7f675b796740 10 client.16182 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:35:23.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.156+0000 7f675b796740 20 client.16182 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:23.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.156+0000 7f675b796740 1 --2- 172.21.15.70:0/1317181636 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5634918714d0 0x563491873920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.156+0000 7f675b796740 4 client.16182 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:23.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.156+0000 7f675b796740 1 -- 172.21.15.70:0/1317181636 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634917c7a40 con 0x5634918714d0 2024-04-01T01:35:23.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.156+0000 7f675259c640 1 --2- 172.21.15.70:0/1317181636 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5634918714d0 0x563491873920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.156+0000 7f6739ffb640 10 client.16182 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:35:23.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.156+0000 7f675259c640 1 --2- 172.21.15.70:0/1317181636 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5634918714d0 0x563491873920 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eaed78b2ed6c3fdb server_cookie=2de26b79a072effc in_seq=0 out_seq=0 2024-04-01T01:35:23.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.157+0000 7f6739ffb640 1 -- 172.21.15.70:0/1317181636 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5634917c7a40 con 0x5634918714d0 2024-04-01T01:35:23.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.157+0000 7f6739ffb640 10 client.16182 handle_command_reply: tid=0 2024-04-01T01:35:23.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.157+0000 7f675b796740 1 client.16182 shutdown 2024-04-01T01:35:23.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.157+0000 7f675b796740 1 -- 172.21.15.70:0/1317181636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56349185be80 msgr2=0x563491865f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.157+0000 7f675b796740 1 --2- 172.21.15.70:0/1317181636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56349185be80 0x563491865f40 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6748009230 tx=0x7f6748004270 comp rx=0 tx=0).stop 2024-04-01T01:35:23.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.158+0000 7f675b796740 1 -- 172.21.15.70:0/1317181636 shutdown_connections 2024-04-01T01:35:23.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.158+0000 7f675b796740 1 --2- 172.21.15.70:0/1317181636 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5634918714d0 0x563491873920 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.158+0000 7f675b796740 1 --2- 172.21.15.70:0/1317181636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563491860ff0 0x56349186cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.158+0000 7f6739ffb640 0 client.16182 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:35:23.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.158+0000 7f675b796740 1 --2- 172.21.15.70:0/1317181636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56349185be80 0x563491865f40 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.158+0000 7f675b796740 1 --2- 172.21.15.70:0/1317181636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185b490 0x563491865a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.158+0000 7f675b796740 1 -- 172.21.15.70:0/1317181636 >> 172.21.15.70:0/1317181636 conn(0x563491856a00 msgr2=0x563491855d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:23.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.158+0000 7f675b796740 1 -- 172.21.15.70:0/1317181636 shutdown_connections 2024-04-01T01:35:23.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.158+0000 7f675b796740 1 -- 172.21.15.70:0/1317181636 wait complete. 2024-04-01T01:35:23.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.158+0000 7f675b796740 20 client.16182 trim_cache size 0 max 16384 2024-04-01T01:35:23.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f675b796740 1 Processor -- start 2024-04-01T01:35:23.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f675b796740 1 -- start start 2024-04-01T01:35:23.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f675b796740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185b490 0x56349185a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f675b796740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56349185be80 0x56349186cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f675b796740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x5634918781e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f675b796740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f67541b0a60 con 0x563491860ff0 2024-04-01T01:35:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f675b796740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f67541b25a0 con 0x56349185b490 2024-04-01T01:35:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f675b796740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f67541b1130 con 0x56349185be80 2024-04-01T01:35:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f6752d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x5634918781e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f6752d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x5634918781e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38982/0 (socket says 172.21.15.70:38982) 2024-04-01T01:35:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f6752d9d640 1 -- 172.21.15.70:0/364908415 learned_addr learned my addr 172.21.15.70:0/364908415 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f675259c640 1 --2- 172.21.15.70:0/364908415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185b490 0x56349185a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f6751d9b640 1 --2- 172.21.15.70:0/364908415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56349185be80 0x56349186cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f6752d9d640 1 -- 172.21.15.70:0/364908415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56349185be80 msgr2=0x56349186cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f6752d9d640 1 --2- 172.21.15.70:0/364908415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56349185be80 0x56349186cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f6752d9d640 1 -- 172.21.15.70:0/364908415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185b490 msgr2=0x56349185a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f6752d9d640 1 --2- 172.21.15.70:0/364908415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185b490 0x56349185a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f6752d9d640 1 -- 172.21.15.70:0/364908415 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563491857970 con 0x563491860ff0 2024-04-01T01:35:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f6752d9d640 1 --2- 172.21.15.70:0/364908415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x5634918781e0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x56349185c6e0 tx=0x7f674c005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a1942db78860f8f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.171+0000 7f673bfff640 1 -- 172.21.15.70:0/364908415 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c037070 con 0x563491860ff0 2024-04-01T01:35:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.172+0000 7f673bfff640 1 -- 172.21.15.70:0/364908415 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f674c004250 con 0x563491860ff0 2024-04-01T01:35:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.172+0000 7f673bfff640 1 -- 172.21.15.70:0/364908415 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c003850 con 0x563491860ff0 2024-04-01T01:35:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.172+0000 7f675b796740 1 -- 172.21.15.70:0/364908415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 msgr2=0x5634918781e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.172+0000 7f675b796740 1 --2- 172.21.15.70:0/364908415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x5634918781e0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x56349185c6e0 tx=0x7f674c005710 comp rx=0 tx=0).stop 2024-04-01T01:35:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.172+0000 7f675b796740 1 -- 172.21.15.70:0/364908415 shutdown_connections 2024-04-01T01:35:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.172+0000 7f675b796740 1 --2- 172.21.15.70:0/364908415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x5634918781e0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.172+0000 7f675b796740 1 --2- 172.21.15.70:0/364908415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56349185be80 0x56349186cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.172+0000 7f675b796740 1 --2- 172.21.15.70:0/364908415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185b490 0x56349185a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.172+0000 7f675b796740 1 -- 172.21.15.70:0/364908415 >> 172.21.15.70:0/364908415 conn(0x563491856a00 msgr2=0x563491859c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.172+0000 7f675b796740 1 -- 172.21.15.70:0/364908415 shutdown_connections 2024-04-01T01:35:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.172+0000 7f675b796740 1 -- 172.21.15.70:0/364908415 wait complete. 2024-04-01T01:35:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.172+0000 7f675b796740 1 -- start start 2024-04-01T01:35:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.172+0000 7f675b796740 1 Processor -- start 2024-04-01T01:35:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f675b796740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185b490 0x56349186a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f675b796740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56349185be80 0x56349186adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f675b796740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x56349186bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f675b796740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f67541b1ed0 con 0x563491860ff0 2024-04-01T01:35:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f675b796740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f67541b1800 con 0x56349185b490 2024-04-01T01:35:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f675b796740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f67541aea40 con 0x56349185be80 2024-04-01T01:35:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f675b796740 10 client.0 _ll_register_callbacks cb 0x7f6748009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f6752d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x56349186bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f6752d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x56349186bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38998/0 (socket says 172.21.15.70:38998) 2024-04-01T01:35:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f6752d9d640 1 -- 172.21.15.70:0/1572277102 learned_addr learned my addr 172.21.15.70:0/1572277102 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f675259c640 1 --2- 172.21.15.70:0/1572277102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185b490 0x56349186a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f6751d9b640 1 --2- 172.21.15.70:0/1572277102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56349185be80 0x56349186adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f6752d9d640 1 -- 172.21.15.70:0/1572277102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56349185be80 msgr2=0x56349186adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f6752d9d640 1 --2- 172.21.15.70:0/1572277102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56349185be80 0x56349186adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f6752d9d640 1 -- 172.21.15.70:0/1572277102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185b490 msgr2=0x56349186a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f6752d9d640 1 --2- 172.21.15.70:0/1572277102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185b490 0x56349186a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f6752d9d640 1 -- 172.21.15.70:0/1572277102 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56349186c3c0 con 0x563491860ff0 2024-04-01T01:35:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f6752d9d640 1 --2- 172.21.15.70:0/1572277102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x56349186bdd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f674c0059f0 tx=0x7f674c01df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c81e6665674c4ad server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f6739ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:35:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f6739ffb640 1 -- 172.21.15.70:0/1572277102 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c037030 con 0x563491860ff0 2024-04-01T01:35:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f6739ffb640 1 -- 172.21.15.70:0/1572277102 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f674c005250 con 0x563491860ff0 2024-04-01T01:35:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f675b796740 1 -- 172.21.15.70:0/1572277102 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f674c004250 con 0x563491860ff0 2024-04-01T01:35:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f6739ffb640 1 -- 172.21.15.70:0/1572277102 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f674c005450 con 0x563491860ff0 2024-04-01T01:35:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.173+0000 7f675b796740 10 client.16188 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.174+0000 7f675b796740 1 -- 172.21.15.70:0/1572277102 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5634918d79a0 con 0x563491860ff0 2024-04-01T01:35:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.174+0000 7f6739ffb640 1 -- 172.21.15.70:0/1572277102 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f674c004250 con 0x563491860ff0 2024-04-01T01:35:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.174+0000 7f6739ffb640 1 -- 172.21.15.70:0/1572277102 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f674c026920 con 0x563491860ff0 2024-04-01T01:35:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.174+0000 7f675b796740 10 client.16188 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:23.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.174+0000 7f675b796740 10 client.16188 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:35:23.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.174+0000 7f675b796740 20 client.16188 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:23.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.174+0000 7f675b796740 1 --2- 172.21.15.70:0/1572277102 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5634918d8170 0x5634918da590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.174+0000 7f675b796740 4 client.16188 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:35:23.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.174+0000 7f675b796740 1 -- 172.21.15.70:0/1572277102 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5634917456a0 con 0x5634918d8170 2024-04-01T01:35:23.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.174+0000 7f675259c640 1 --2- 172.21.15.70:0/1572277102 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5634918d8170 0x5634918da590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.174+0000 7f6739ffb640 10 client.16188 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:35:23.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.174+0000 7f675259c640 1 --2- 172.21.15.70:0/1572277102 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5634918d8170 0x5634918da590 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=707da778d1095e8 server_cookie=2c60911d3e2c7a33 in_seq=0 out_seq=0 2024-04-01T01:35:23.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.174+0000 7f6739ffb640 1 -- 172.21.15.70:0/1572277102 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5634917456a0 con 0x5634918d8170 2024-04-01T01:35:23.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.174+0000 7f6739ffb640 10 client.16188 handle_command_reply: tid=0 2024-04-01T01:35:23.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.174+0000 7f675b796740 1 client.16188 shutdown 2024-04-01T01:35:23.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f675b796740 1 -- 172.21.15.70:0/1572277102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 msgr2=0x56349186bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f675b796740 1 --2- 172.21.15.70:0/1572277102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x56349186bdd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f674c0059f0 tx=0x7f674c01df80 comp rx=0 tx=0).stop 2024-04-01T01:35:23.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f675b796740 1 -- 172.21.15.70:0/1572277102 shutdown_connections 2024-04-01T01:35:23.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f675b796740 1 --2- 172.21.15.70:0/1572277102 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5634918d8170 0x5634918da590 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f675b796740 1 --2- 172.21.15.70:0/1572277102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563491860ff0 0x56349186bdd0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f6739ffb640 0 client.16188 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:35:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f675b796740 1 --2- 172.21.15.70:0/1572277102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56349185be80 0x56349186adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f675b796740 1 --2- 172.21.15.70:0/1572277102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56349185b490 0x56349186a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f675b796740 1 -- 172.21.15.70:0/1572277102 >> 172.21.15.70:0/1572277102 conn(0x563491856a00 msgr2=0x56349185a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f675b796740 1 -- 172.21.15.70:0/1572277102 shutdown_connections 2024-04-01T01:35:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f675b796740 1 -- 172.21.15.70:0/1572277102 wait complete. 2024-04-01T01:35:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f675b796740 20 client.16188 trim_cache size 0 max 16384 2024-04-01T01:35:23.182 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:35:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f6758bb5640 1 -- 172.21.15.70:0/834177795 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f673c0a24a0 msgr2=0x7f673c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f6758bb5640 1 --2- 172.21.15.70:0/834177795 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f673c0a24a0 0x7f673c0a4960 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f674c009710 tx=0x7f674c01e000 comp rx=0 tx=0).stop 2024-04-01T01:35:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f6758bb5640 1 -- 172.21.15.70:0/834177795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67541021d0 msgr2=0x7f6754101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f6758bb5640 1 --2- 172.21.15.70:0/834177795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67541021d0 0x7f6754101b40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f674400ebc0 tx=0x7f674400c6a0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f6758bb5640 1 -- 172.21.15.70:0/834177795 shutdown_connections 2024-04-01T01:35:23.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f6758bb5640 1 --2- 172.21.15.70:0/834177795 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f673c0a24a0 0x7f673c0a4960 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f6758bb5640 1 --2- 172.21.15.70:0/834177795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6754107430 0x7f67541006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f6758bb5640 1 --2- 172.21.15.70:0/834177795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6754104b00 0x7f6754100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f6758bb5640 1 --2- 172.21.15.70:0/834177795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67541021d0 0x7f6754101b40 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f6758bb5640 1 -- 172.21.15.70:0/834177795 >> 172.21.15.70:0/834177795 conn(0x7f67540fbbe0 msgr2=0x7f6754104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:23.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f6758bb5640 1 -- 172.21.15.70:0/834177795 shutdown_connections 2024-04-01T01:35:23.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.175+0000 7f6758bb5640 1 -- 172.21.15.70:0/834177795 wait complete. 2024-04-01T01:35:23.186 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:35:23.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.361+0000 7ff293acd640 1 -- 172.21.15.70:0/2654323545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff28c10eb50 msgr2=0x7ff28c110ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.361+0000 7ff293acd640 1 --2- 172.21.15.70:0/2654323545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff28c10eb50 0x7ff28c110ff0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff280009860 tx=0x7ff28002bb90 comp rx=0 tx=0).stop 2024-04-01T01:35:23.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.362+0000 7ff293acd640 1 -- 172.21.15.70:0/2654323545 shutdown_connections 2024-04-01T01:35:23.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.362+0000 7ff293acd640 1 --2- 172.21.15.70:0/2654323545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff28c10eb50 0x7ff28c110ff0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.362+0000 7ff293acd640 1 --2- 172.21.15.70:0/2654323545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff28c10c190 0x7ff28c10e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.362+0000 7ff293acd640 1 --2- 172.21.15.70:0/2654323545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff28c109860 0x7ff28c10bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.362+0000 7ff293acd640 1 -- 172.21.15.70:0/2654323545 >> 172.21.15.70:0/2654323545 conn(0x7ff28c06e950 msgr2=0x7ff28c070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:23.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.362+0000 7ff293acd640 1 -- 172.21.15.70:0/2654323545 shutdown_connections 2024-04-01T01:35:23.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.363+0000 7ff293acd640 1 -- 172.21.15.70:0/2654323545 wait complete. 2024-04-01T01:35:23.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.363+0000 7ff293acd640 1 Processor -- start 2024-04-01T01:35:23.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.364+0000 7ff293acd640 1 -- start start 2024-04-01T01:35:23.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.364+0000 7ff293acd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff28c109860 0x7ff28c1a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.364+0000 7ff293acd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff28c10c190 0x7ff28c1a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.364+0000 7ff293acd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff28c10eb50 0x7ff28c1a7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.364+0000 7ff293acd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c115aa0 con 0x7ff28c10eb50 2024-04-01T01:35:23.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.364+0000 7ff293acd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c115920 con 0x7ff28c109860 2024-04-01T01:35:23.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.364+0000 7ff293acd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c115c20 con 0x7ff28c10c190 2024-04-01T01:35:23.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.365+0000 7ff291068640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff28c10c190 0x7ff28c1a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.365+0000 7ff291869640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff28c109860 0x7ff28c1a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.365+0000 7ff291068640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff28c10c190 0x7ff28c1a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37876/0 (socket says 172.21.15.70:37876) 2024-04-01T01:35:23.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.365+0000 7ff291068640 1 -- 172.21.15.70:0/253251104 learned_addr learned my addr 172.21.15.70:0/253251104 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:23.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.365+0000 7ff29206a640 1 --2- 172.21.15.70:0/253251104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff28c10eb50 0x7ff28c1a7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.365+0000 7ff291068640 1 -- 172.21.15.70:0/253251104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff28c109860 msgr2=0x7ff28c1a0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.365+0000 7ff291068640 1 --2- 172.21.15.70:0/253251104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff28c109860 0x7ff28c1a0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.365+0000 7ff291068640 1 -- 172.21.15.70:0/253251104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff28c10eb50 msgr2=0x7ff28c1a7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.365+0000 7ff291068640 1 --2- 172.21.15.70:0/253251104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff28c10eb50 0x7ff28c1a7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.365+0000 7ff291068640 1 -- 172.21.15.70:0/253251104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff28c111dc0 con 0x7ff28c10c190 2024-04-01T01:35:23.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.366+0000 7ff291869640 1 --2- 172.21.15.70:0/253251104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff28c109860 0x7ff28c1a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:23.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.366+0000 7ff291068640 1 --2- 172.21.15.70:0/253251104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff28c10c190 0x7ff28c1a06b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00eb90 tx=0x7ff27c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:23.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.366+0000 7ff28affd640 1 -- 172.21.15.70:0/253251104 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c010630 con 0x7ff28c10c190 2024-04-01T01:35:23.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.366+0000 7ff28affd640 1 -- 172.21.15.70:0/253251104 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff27c00ee90 con 0x7ff28c10c190 2024-04-01T01:35:23.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.366+0000 7ff28affd640 1 -- 172.21.15.70:0/253251104 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c018ac0 con 0x7ff28c10c190 2024-04-01T01:35:23.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.367+0000 7ff293acd640 1 -- 172.21.15.70:0/253251104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff28c1120b0 con 0x7ff28c10c190 2024-04-01T01:35:23.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.367+0000 7ff293acd640 1 -- 172.21.15.70:0/253251104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff28c1125b0 con 0x7ff28c10c190 2024-04-01T01:35:23.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.370+0000 7ff296790740 1 Processor -- start 2024-04-01T01:35:23.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.371+0000 7ff28affd640 1 -- 172.21.15.70:0/253251104 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7ff27c018c60 con 0x7ff28c10c190 2024-04-01T01:35:23.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.371+0000 7ff296790740 1 -- start start 2024-04-01T01:35:23.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.371+0000 7ff296790740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf34490 0x563aeaf348b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.371+0000 7ff296790740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34e80 0x563aeaf39980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.371+0000 7ff296790740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf39ff0 0x563aeaf3c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.371+0000 7ff296790740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563aeae991b0 con 0x563aeaf34490 2024-04-01T01:35:23.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.371+0000 7ff296790740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563aeae41c00 con 0x563aeaf34e80 2024-04-01T01:35:23.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.371+0000 7ff296790740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563aeae95250 con 0x563aeaf39ff0 2024-04-01T01:35:23.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.372+0000 7ff291869640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34e80 0x563aeaf39980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.372+0000 7ff291869640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34e80 0x563aeaf39980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35374/0 (socket says 172.21.15.70:35374) 2024-04-01T01:35:23.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.372+0000 7ff291869640 1 -- 172.21.15.70:0/4181201703 learned_addr learned my addr 172.21.15.70:0/4181201703 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:23.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.372+0000 7ff28affd640 1 --2- 172.21.15.70:0/253251104 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2740a2520 0x7ff2740a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.372+0000 7ff29206a640 1 --2- 172.21.15.70:0/4181201703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf34490 0x563aeaf348b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.372+0000 7ff291068640 1 --2- 172.21.15.70:0/4181201703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf39ff0 0x563aeaf3c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.372+0000 7ff28affd640 1 -- 172.21.15.70:0/253251104 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff27c0cefc0 con 0x7ff28c10c190 2024-04-01T01:35:23.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.373+0000 7ff291869640 1 -- 172.21.15.70:0/4181201703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf39ff0 msgr2=0x563aeaf3c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:23.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.373+0000 7ff291869640 1 --2- 172.21.15.70:0/4181201703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf39ff0 0x563aeaf3c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.373+0000 7ff291869640 1 -- 172.21.15.70:0/4181201703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf34490 msgr2=0x563aeaf348b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:23.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.373+0000 7ff291869640 1 --2- 172.21.15.70:0/4181201703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf34490 0x563aeaf348b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.373+0000 7ff291869640 1 -- 172.21.15.70:0/4181201703 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563aeae613d0 con 0x563aeaf34e80 2024-04-01T01:35:23.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.373+0000 7ff29206a640 1 --2- 172.21.15.70:0/4181201703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf34490 0x563aeaf348b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:23.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.373+0000 7ff29206a640 1 --2- 172.21.15.70:0/253251104 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2740a2520 0x7ff2740a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.373+0000 7ff291869640 1 --2- 172.21.15.70:0/4181201703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34e80 0x563aeaf39980 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff2840114f0 tx=0x7ff2840119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20498d22f87c17e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:23.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.373+0000 7ff273fff640 1 -- 172.21.15.70:0/4181201703 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff284005020 con 0x563aeaf34e80 2024-04-01T01:35:23.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.374+0000 7ff273fff640 1 -- 172.21.15.70:0/4181201703 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2840047a0 con 0x563aeaf34e80 2024-04-01T01:35:23.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.374+0000 7ff273fff640 1 -- 172.21.15.70:0/4181201703 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28400f9e0 con 0x563aeaf34e80 2024-04-01T01:35:23.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.375+0000 7ff296790740 1 -- 172.21.15.70:0/4181201703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34e80 msgr2=0x563aeaf39980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.375+0000 7ff296790740 1 --2- 172.21.15.70:0/4181201703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34e80 0x563aeaf39980 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff2840114f0 tx=0x7ff2840119b0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.375+0000 7ff29206a640 1 --2- 172.21.15.70:0/253251104 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2740a2520 0x7ff2740a49e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff28002c1a0 tx=0x7ff280005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:23.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.376+0000 7ff296790740 1 -- 172.21.15.70:0/4181201703 shutdown_connections 2024-04-01T01:35:23.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.376+0000 7ff296790740 1 --2- 172.21.15.70:0/4181201703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf39ff0 0x563aeaf3c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.376+0000 7ff296790740 1 --2- 172.21.15.70:0/4181201703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34e80 0x563aeaf39980 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.376+0000 7ff296790740 1 --2- 172.21.15.70:0/4181201703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf34490 0x563aeaf348b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.376+0000 7ff296790740 1 -- 172.21.15.70:0/4181201703 >> 172.21.15.70:0/4181201703 conn(0x563aeaf2fa00 msgr2=0x563aeaf2fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:23.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.376+0000 7ff296790740 1 -- 172.21.15.70:0/4181201703 shutdown_connections 2024-04-01T01:35:23.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.377+0000 7ff296790740 1 -- 172.21.15.70:0/4181201703 wait complete. 2024-04-01T01:35:23.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.378+0000 7ff296790740 1 -- start start 2024-04-01T01:35:23.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.378+0000 7ff296790740 1 Processor -- start 2024-04-01T01:35:23.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.378+0000 7ff296790740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34490 0x563aeaf4f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.378+0000 7ff296790740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf34e80 0x563aeaf4ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.378+0000 7ff296790740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf39ff0 0x563aeaf56f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.378+0000 7ff296790740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563aeaf4e000 con 0x563aeaf34e80 2024-04-01T01:35:23.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.378+0000 7ff296790740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563aeae37470 con 0x563aeaf39ff0 2024-04-01T01:35:23.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.378+0000 7ff296790740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563aeaf4e180 con 0x563aeaf34490 2024-04-01T01:35:23.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.378+0000 7ff296790740 10 client.0 _ll_register_callbacks cb 0x563aeae54cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:23.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.379+0000 7ff291869640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34490 0x563aeaf4f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.379+0000 7ff291869640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34490 0x563aeaf4f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37898/0 (socket says 172.21.15.70:37898) 2024-04-01T01:35:23.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.379+0000 7ff291869640 1 -- 172.21.15.70:0/2008574397 learned_addr learned my addr 172.21.15.70:0/2008574397 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:23.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.379+0000 7ff291068640 1 --2- 172.21.15.70:0/2008574397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf34e80 0x563aeaf4ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.379+0000 7ff29206a640 1 --2- 172.21.15.70:0/2008574397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf39ff0 0x563aeaf56f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.380+0000 7ff291869640 1 -- 172.21.15.70:0/2008574397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf39ff0 msgr2=0x563aeaf56f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:23.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.380+0000 7ff291869640 1 --2- 172.21.15.70:0/2008574397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf39ff0 0x563aeaf56f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.380+0000 7ff291869640 1 -- 172.21.15.70:0/2008574397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf34e80 msgr2=0x563aeaf4ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:23.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.380+0000 7ff291869640 1 --2- 172.21.15.70:0/2008574397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf34e80 0x563aeaf4ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.380+0000 7ff291869640 1 -- 172.21.15.70:0/2008574397 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563aeabad3f0 con 0x563aeaf34490 2024-04-01T01:35:23.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.380+0000 7ff29206a640 1 --2- 172.21.15.70:0/2008574397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf39ff0 0x563aeaf56f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:35:23.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.380+0000 7ff291869640 1 --2- 172.21.15.70:0/2008574397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34490 0x563aeaf4f9e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff2840114c0 tx=0x7ff2840051d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a29cf9a3d52187c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:23.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.380+0000 7ff2717fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:35:23.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.380+0000 7ff2717fa640 1 -- 172.21.15.70:0/2008574397 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff284028070 con 0x563aeaf34490 2024-04-01T01:35:23.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.381+0000 7ff2717fa640 1 -- 172.21.15.70:0/2008574397 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff2840054d0 con 0x563aeaf34490 2024-04-01T01:35:23.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.381+0000 7ff296790740 1 -- 172.21.15.70:0/2008574397 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2840047a0 con 0x563aeaf34490 2024-04-01T01:35:23.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.382+0000 7ff2717fa640 1 -- 172.21.15.70:0/2008574397 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff284032050 con 0x563aeaf34490 2024-04-01T01:35:23.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.382+0000 7ff296790740 10 client.25939 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:23.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.382+0000 7ff296790740 1 -- 172.21.15.70:0/2008574397 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563aeac777d0 con 0x563aeaf34490 2024-04-01T01:35:23.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.383+0000 7ff2717fa640 1 -- 172.21.15.70:0/2008574397 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2840047a0 con 0x563aeaf34490 2024-04-01T01:35:23.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.383+0000 7ff2717fa640 1 -- 172.21.15.70:0/2008574397 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff284024750 con 0x563aeaf34490 2024-04-01T01:35:23.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.383+0000 7ff296790740 10 client.25939 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:23.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.383+0000 7ff296790740 10 client.25939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:23.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.383+0000 7ff296790740 20 client.25939 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:23.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.383+0000 7ff296790740 1 --2- 172.21.15.70:0/2008574397 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563aeaf5b820 0x563aeaf5dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.383+0000 7ff296790740 4 client.25939 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:23.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.383+0000 7ff296790740 1 -- 172.21.15.70:0/2008574397 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563aeaea66f0 con 0x563aeaf5b820 2024-04-01T01:35:23.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.383+0000 7ff291068640 1 --2- 172.21.15.70:0/2008574397 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563aeaf5b820 0x563aeaf5dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.384+0000 7ff2717fa640 10 client.25939 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:23.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.385+0000 7ff291068640 1 --2- 172.21.15.70:0/2008574397 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563aeaf5b820 0x563aeaf5dc80 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=696420f633d11d75 server_cookie=d6dce46ca4c13b65 in_seq=0 out_seq=0 2024-04-01T01:35:23.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.386+0000 7ff2717fa640 1 -- 172.21.15.70:0/2008574397 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563aeaea66f0 con 0x563aeaf5b820 2024-04-01T01:35:23.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.386+0000 7ff2717fa640 10 client.25939 handle_command_reply: tid=0 2024-04-01T01:35:23.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.387+0000 7ff296790740 1 client.25939 shutdown 2024-04-01T01:35:23.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.387+0000 7ff296790740 1 -- 172.21.15.70:0/2008574397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34490 msgr2=0x563aeaf4f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.387+0000 7ff296790740 1 --2- 172.21.15.70:0/2008574397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34490 0x563aeaf4f9e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff2840114c0 tx=0x7ff2840051d0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.388+0000 7ff296790740 1 -- 172.21.15.70:0/2008574397 shutdown_connections 2024-04-01T01:35:23.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.388+0000 7ff296790740 1 --2- 172.21.15.70:0/2008574397 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563aeaf5b820 0x563aeaf5dc80 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.388+0000 7ff296790740 1 --2- 172.21.15.70:0/2008574397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf39ff0 0x563aeaf56f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.388+0000 7ff296790740 1 --2- 172.21.15.70:0/2008574397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf34e80 0x563aeaf4ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.388+0000 7ff296790740 1 --2- 172.21.15.70:0/2008574397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34490 0x563aeaf4f9e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.388+0000 7ff296790740 1 -- 172.21.15.70:0/2008574397 >> 172.21.15.70:0/2008574397 conn(0x563aeaf2fa00 msgr2=0x563aeaf2ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:23.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.388+0000 7ff2717fa640 0 client.25939 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:23.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.388+0000 7ff296790740 1 -- 172.21.15.70:0/2008574397 shutdown_connections 2024-04-01T01:35:23.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.388+0000 7ff296790740 1 -- 172.21.15.70:0/2008574397 wait complete. 2024-04-01T01:35:23.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.389+0000 7ff296790740 20 client.25939 trim_cache size 0 max 16384 2024-04-01T01:35:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.417+0000 7ff296790740 1 Processor -- start 2024-04-01T01:35:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.417+0000 7ff296790740 1 -- start start 2024-04-01T01:35:23.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.417+0000 7ff296790740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34490 0x563aeaf2e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.417+0000 7ff296790740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34e80 0x563aeaf56f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.417+0000 7ff296790740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf39ff0 0x563aeaf2e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.417+0000 7ff296790740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c1b0ba0 con 0x563aeaf39ff0 2024-04-01T01:35:23.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.417+0000 7ff296790740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c1b26e0 con 0x563aeaf34490 2024-04-01T01:35:23.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.417+0000 7ff296790740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c1b1270 con 0x563aeaf34e80 2024-04-01T01:35:23.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.418+0000 7ff291068640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34e80 0x563aeaf56f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.418+0000 7ff291068640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34e80 0x563aeaf56f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37910/0 (socket says 172.21.15.70:37910) 2024-04-01T01:35:23.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.418+0000 7ff291068640 1 -- 172.21.15.70:0/1204672107 learned_addr learned my addr 172.21.15.70:0/1204672107 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:23.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.418+0000 7ff291869640 1 --2- 172.21.15.70:0/1204672107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34490 0x563aeaf2e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.418+0000 7ff29206a640 1 --2- 172.21.15.70:0/1204672107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf39ff0 0x563aeaf2e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.418+0000 7ff291869640 1 -- 172.21.15.70:0/1204672107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34e80 msgr2=0x563aeaf56f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:23.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.418+0000 7ff291869640 1 --2- 172.21.15.70:0/1204672107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34e80 0x563aeaf56f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.418+0000 7ff291869640 1 -- 172.21.15.70:0/1204672107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf39ff0 msgr2=0x563aeaf2e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:23.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.418+0000 7ff291869640 1 --2- 172.21.15.70:0/1204672107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf39ff0 0x563aeaf2e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.418+0000 7ff291869640 1 -- 172.21.15.70:0/1204672107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563aeaf36c70 con 0x563aeaf34490 2024-04-01T01:35:23.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.418+0000 7ff291068640 1 --2- 172.21.15.70:0/1204672107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34e80 0x563aeaf56f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:23.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.419+0000 7ff291869640 1 --2- 172.21.15.70:0/1204672107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34490 0x563aeaf2e4b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff284024b50 tx=0x7ff2840119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6c3b9bf9ca40c3a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:23.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.419+0000 7ff2737fe640 1 -- 172.21.15.70:0/1204672107 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff284028040 con 0x563aeaf34490 2024-04-01T01:35:23.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.419+0000 7ff2737fe640 1 -- 172.21.15.70:0/1204672107 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2840047a0 con 0x563aeaf34490 2024-04-01T01:35:23.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.419+0000 7ff2737fe640 1 -- 172.21.15.70:0/1204672107 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28400f640 con 0x563aeaf34490 2024-04-01T01:35:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.420+0000 7ff296790740 1 -- 172.21.15.70:0/1204672107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34490 msgr2=0x563aeaf2e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.420+0000 7ff296790740 1 --2- 172.21.15.70:0/1204672107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34490 0x563aeaf2e4b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff284024b50 tx=0x7ff2840119b0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.420+0000 7ff296790740 1 -- 172.21.15.70:0/1204672107 shutdown_connections 2024-04-01T01:35:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.420+0000 7ff296790740 1 --2- 172.21.15.70:0/1204672107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf39ff0 0x563aeaf2e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.420+0000 7ff296790740 1 --2- 172.21.15.70:0/1204672107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34e80 0x563aeaf56f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.420+0000 7ff296790740 1 --2- 172.21.15.70:0/1204672107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34490 0x563aeaf2e4b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.420+0000 7ff296790740 1 -- 172.21.15.70:0/1204672107 >> 172.21.15.70:0/1204672107 conn(0x563aeaf2fa00 msgr2=0x563aeaf2fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.420+0000 7ff296790740 1 -- 172.21.15.70:0/1204672107 shutdown_connections 2024-04-01T01:35:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.420+0000 7ff296790740 1 -- 172.21.15.70:0/1204672107 wait complete. 2024-04-01T01:35:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.421+0000 7ff296790740 1 -- start start 2024-04-01T01:35:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.421+0000 7ff296790740 1 Processor -- start 2024-04-01T01:35:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.421+0000 7ff296790740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34490 0x563aeaf53ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.421+0000 7ff296790740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34e80 0x563aeaf54530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.421+0000 7ff296790740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf39ff0 0x563aeaf55540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.421+0000 7ff296790740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c1b2010 con 0x563aeaf39ff0 2024-04-01T01:35:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.421+0000 7ff296790740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c1b1940 con 0x563aeaf34e80 2024-04-01T01:35:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.421+0000 7ff296790740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c1aeb80 con 0x563aeaf34490 2024-04-01T01:35:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.421+0000 7ff296790740 10 client.0 _ll_register_callbacks cb 0x7ff284004d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.421+0000 7ff291869640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34490 0x563aeaf53ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.421+0000 7ff291068640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34e80 0x563aeaf54530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.422+0000 7ff291869640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34490 0x563aeaf53ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37920/0 (socket says 172.21.15.70:37920) 2024-04-01T01:35:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.422+0000 7ff291068640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34e80 0x563aeaf54530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35398/0 (socket says 172.21.15.70:35398) 2024-04-01T01:35:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.422+0000 7ff291068640 1 -- 172.21.15.70:0/4231895780 learned_addr learned my addr 172.21.15.70:0/4231895780 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.422+0000 7ff29206a640 1 --2- 172.21.15.70:0/4231895780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf39ff0 0x563aeaf55540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.422+0000 7ff29206a640 1 -- 172.21.15.70:0/4231895780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34490 msgr2=0x563aeaf53ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.422+0000 7ff29206a640 1 --2- 172.21.15.70:0/4231895780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34490 0x563aeaf53ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.422+0000 7ff29206a640 1 -- 172.21.15.70:0/4231895780 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34e80 msgr2=0x563aeaf54530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:23.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.422+0000 7ff29206a640 1 --2- 172.21.15.70:0/4231895780 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34e80 0x563aeaf54530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.422+0000 7ff29206a640 1 -- 172.21.15.70:0/4231895780 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563aeaf55b30 con 0x563aeaf39ff0 2024-04-01T01:35:23.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.422+0000 7ff29206a640 1 --2- 172.21.15.70:0/4231895780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf39ff0 0x563aeaf55540 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff280009780 tx=0x7ff2800056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32106e0d61116f0e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:23.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.422+0000 7ff2717fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:35:23.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.423+0000 7ff2717fa640 1 -- 172.21.15.70:0/4231895780 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28004f070 con 0x563aeaf39ff0 2024-04-01T01:35:23.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.423+0000 7ff296790740 1 -- 172.21.15.70:0/4231895780 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2840047a0 con 0x563aeaf39ff0 2024-04-01T01:35:23.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.423+0000 7ff2717fa640 1 -- 172.21.15.70:0/4231895780 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff28002c940 con 0x563aeaf39ff0 2024-04-01T01:35:23.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.423+0000 7ff2717fa640 1 -- 172.21.15.70:0/4231895780 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff28005b050 con 0x563aeaf39ff0 2024-04-01T01:35:23.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.423+0000 7ff296790740 10 client.16206 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:23.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.423+0000 7ff296790740 1 -- 172.21.15.70:0/4231895780 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563aeaf56ef0 con 0x563aeaf39ff0 2024-04-01T01:35:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.424+0000 7ff2717fa640 1 -- 172.21.15.70:0/4231895780 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2840047a0 con 0x563aeaf39ff0 2024-04-01T01:35:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.424+0000 7ff2717fa640 1 -- 172.21.15.70:0/4231895780 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff28004b690 con 0x563aeaf39ff0 2024-04-01T01:35:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.424+0000 7ff296790740 10 client.16206 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.424+0000 7ff296790740 10 client.16206 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.424+0000 7ff296790740 20 client.16206 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.424+0000 7ff296790740 1 --2- 172.21.15.70:0/4231895780 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563aeafc1fd0 0x563aeafc43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.424+0000 7ff296790740 4 client.16206 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:35:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.424+0000 7ff296790740 1 -- 172.21.15.70:0/4231895780 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563aeaea0a40 con 0x563aeafc1fd0 2024-04-01T01:35:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.425+0000 7ff291869640 1 --2- 172.21.15.70:0/4231895780 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563aeafc1fd0 0x563aeafc43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.425+0000 7ff291869640 1 --2- 172.21.15.70:0/4231895780 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563aeafc1fd0 0x563aeafc43f0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=393aec0e31d465b2 server_cookie=3a9239eac1e8be4f in_seq=0 out_seq=0 2024-04-01T01:35:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.425+0000 7ff2717fa640 10 client.16206 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:23.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.425+0000 7ff2717fa640 1 -- 172.21.15.70:0/4231895780 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563aeaea0a40 con 0x563aeafc1fd0 2024-04-01T01:35:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.425+0000 7ff2717fa640 10 client.16206 handle_command_reply: tid=0 2024-04-01T01:35:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.425+0000 7ff296790740 1 client.16206 shutdown 2024-04-01T01:35:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.426+0000 7ff296790740 1 -- 172.21.15.70:0/4231895780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf39ff0 msgr2=0x563aeaf55540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.426+0000 7ff296790740 1 --2- 172.21.15.70:0/4231895780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf39ff0 0x563aeaf55540 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff280009780 tx=0x7ff2800056d0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.426+0000 7ff296790740 1 -- 172.21.15.70:0/4231895780 shutdown_connections 2024-04-01T01:35:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.426+0000 7ff296790740 1 --2- 172.21.15.70:0/4231895780 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563aeafc1fd0 0x563aeafc43f0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.426+0000 7ff296790740 1 --2- 172.21.15.70:0/4231895780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563aeaf39ff0 0x563aeaf55540 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.426+0000 7ff296790740 1 --2- 172.21.15.70:0/4231895780 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563aeaf34e80 0x563aeaf54530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.426+0000 7ff296790740 1 --2- 172.21.15.70:0/4231895780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563aeaf34490 0x563aeaf53ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.426+0000 7ff296790740 1 -- 172.21.15.70:0/4231895780 >> 172.21.15.70:0/4231895780 conn(0x563aeaf2fa00 msgr2=0x563aeaf32c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.426+0000 7ff2717fa640 0 client.16206 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.426+0000 7ff296790740 1 -- 172.21.15.70:0/4231895780 shutdown_connections 2024-04-01T01:35:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.426+0000 7ff296790740 1 -- 172.21.15.70:0/4231895780 wait complete. 2024-04-01T01:35:23.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.426+0000 7ff296790740 20 client.16206 trim_cache size 0 max 16384 2024-04-01T01:35:23.429 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:35:23.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.427+0000 7ff293acd640 1 -- 172.21.15.70:0/253251104 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2740a2520 msgr2=0x7ff2740a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.427+0000 7ff293acd640 1 --2- 172.21.15.70:0/253251104 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2740a2520 0x7ff2740a49e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff28002c1a0 tx=0x7ff280005f90 comp rx=0 tx=0).stop 2024-04-01T01:35:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.428+0000 7ff293acd640 1 -- 172.21.15.70:0/253251104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff28c10c190 msgr2=0x7ff28c1a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.428+0000 7ff293acd640 1 --2- 172.21.15.70:0/253251104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff28c10c190 0x7ff28c1a06b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00eb90 tx=0x7ff27c00c6a0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.428+0000 7ff293acd640 1 -- 172.21.15.70:0/253251104 shutdown_connections 2024-04-01T01:35:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.428+0000 7ff293acd640 1 --2- 172.21.15.70:0/253251104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff28c10eb50 0x7ff28c1a7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.428+0000 7ff293acd640 1 --2- 172.21.15.70:0/253251104 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2740a2520 0x7ff2740a49e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.428+0000 7ff293acd640 1 --2- 172.21.15.70:0/253251104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff28c10c190 0x7ff28c1a06b0 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00eb90 tx=0x7ff27c00c6a0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.428+0000 7ff293acd640 1 --2- 172.21.15.70:0/253251104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff28c109860 0x7ff28c1a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.428+0000 7ff293acd640 1 -- 172.21.15.70:0/253251104 >> 172.21.15.70:0/253251104 conn(0x7ff28c06e950 msgr2=0x7ff28c10a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.429+0000 7ff293acd640 1 -- 172.21.15.70:0/253251104 shutdown_connections 2024-04-01T01:35:23.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:23.429+0000 7ff293acd640 1 -- 172.21.15.70:0/253251104 wait complete. 2024-04-01T01:35:23.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:23 smithi189 ceph-mon[35753]: pgmap v415: 129 pgs: 129 active+clean; 943 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 13 MiB/s wr, 1.41k op/s 2024-04-01T01:35:24.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:23 smithi070 ceph-mon[32179]: pgmap v415: 129 pgs: 129 active+clean; 943 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 13 MiB/s wr, 1.41k op/s 2024-04-01T01:35:24.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:23 smithi081 ceph-mon[35142]: pgmap v415: 129 pgs: 129 active+clean; 943 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 13 MiB/s wr, 1.41k op/s 2024-04-01T01:35:24.448 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:35:24.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:35:24.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:35:24.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.628+0000 7f8e30fd0640 1 -- 172.21.15.70:0/236330157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e24093fe0 msgr2=0x7f8e24094400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.628+0000 7f8e30fd0640 1 --2- 172.21.15.70:0/236330157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e24093fe0 0x7f8e24094400 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8e2000d910 tx=0x7f8e2002b9a0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.628+0000 7f8e30fd0640 1 -- 172.21.15.70:0/236330157 shutdown_connections 2024-04-01T01:35:24.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.628+0000 7f8e30fd0640 1 --2- 172.21.15.70:0/236330157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e24095710 0x7f8e24095b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.628+0000 7f8e30fd0640 1 --2- 172.21.15.70:0/236330157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e24004850 0x7f8e240951d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.628+0000 7f8e30fd0640 1 --2- 172.21.15.70:0/236330157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e24093fe0 0x7f8e24094400 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.628+0000 7f8e30fd0640 1 -- 172.21.15.70:0/236330157 >> 172.21.15.70:0/236330157 conn(0x7f8e2408f530 msgr2=0x7f8e24091990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:24.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.628+0000 7f8e30fd0640 1 -- 172.21.15.70:0/236330157 shutdown_connections 2024-04-01T01:35:24.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.628+0000 7f8e30fd0640 1 -- 172.21.15.70:0/236330157 wait complete. 2024-04-01T01:35:24.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.629+0000 7f8e30fd0640 1 Processor -- start 2024-04-01T01:35:24.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.629+0000 7f8e30fd0640 1 -- start start 2024-04-01T01:35:24.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.630+0000 7f8e30fd0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e24004850 0x7f8e24129e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.630+0000 7f8e30fd0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e24093fe0 0x7f8e2412a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.630+0000 7f8e30fd0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e24095710 0x7f8e241426e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.630+0000 7f8e30fd0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2409f7c0 con 0x7f8e24095710 2024-04-01T01:35:24.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.630+0000 7f8e30fd0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2409f640 con 0x7f8e24004850 2024-04-01T01:35:24.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.630+0000 7f8e30fd0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2409f940 con 0x7f8e24093fe0 2024-04-01T01:35:24.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.630+0000 7f8e29d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e24093fe0 0x7f8e2412a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.630+0000 7f8e29d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e24093fe0 0x7f8e2412a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37942/0 (socket says 172.21.15.70:37942) 2024-04-01T01:35:24.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.630+0000 7f8e29d9b640 1 -- 172.21.15.70:0/1388050063 learned_addr learned my addr 172.21.15.70:0/1388050063 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:24.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.630+0000 7f8e2a59c640 1 --2- 172.21.15.70:0/1388050063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e24004850 0x7f8e24129e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.630+0000 7f8e29d9b640 1 -- 172.21.15.70:0/1388050063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e24004850 msgr2=0x7f8e24129e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.630+0000 7f8e29d9b640 1 --2- 172.21.15.70:0/1388050063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e24004850 0x7f8e24129e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.630+0000 7f8e29d9b640 1 -- 172.21.15.70:0/1388050063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e24095710 msgr2=0x7f8e241426e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.630+0000 7f8e29d9b640 1 --2- 172.21.15.70:0/1388050063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e24095710 0x7f8e241426e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.631+0000 7f8e29d9b640 1 -- 172.21.15.70:0/1388050063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e24142c20 con 0x7f8e24093fe0 2024-04-01T01:35:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.631+0000 7f8e29d9b640 1 --2- 172.21.15.70:0/1388050063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e24093fe0 0x7f8e2412a380 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c005fd0 tx=0x7f8e1c00df80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.631+0000 7f8e1b7fe640 1 -- 172.21.15.70:0/1388050063 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c00cc40 con 0x7f8e24093fe0 2024-04-01T01:35:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.631+0000 7f8e30fd0640 1 -- 172.21.15.70:0/1388050063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e24142df0 con 0x7f8e24093fe0 2024-04-01T01:35:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.631+0000 7f8e30fd0640 1 -- 172.21.15.70:0/1388050063 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e24143250 con 0x7f8e24093fe0 2024-04-01T01:35:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.632+0000 7f8e1b7fe640 1 -- 172.21.15.70:0/1388050063 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e1c00cde0 con 0x7f8e24093fe0 2024-04-01T01:35:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.632+0000 7f8e1b7fe640 1 -- 172.21.15.70:0/1388050063 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c015410 con 0x7f8e24093fe0 2024-04-01T01:35:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.635+0000 7f8e1b7fe640 1 -- 172.21.15.70:0/1388050063 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f8e1c004000 con 0x7f8e24093fe0 2024-04-01T01:35:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.635+0000 7f8e33bb9740 1 Processor -- start 2024-04-01T01:35:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.636+0000 7f8e33bb9740 1 -- start start 2024-04-01T01:35:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.636+0000 7f8e33bb9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 0x55895d4b0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.636+0000 7f8e33bb9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b05e0 0x55895d4b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.636+0000 7f8e33bb9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b5750 0x55895d4b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.636+0000 7f8e33bb9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55895d3b10b0 con 0x55895d4b05e0 2024-04-01T01:35:24.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.636+0000 7f8e33bb9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55895d413ea0 con 0x55895d4b5750 2024-04-01T01:35:24.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.636+0000 7f8e33bb9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55895d3bcc00 con 0x55895d4afbf0 2024-04-01T01:35:24.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.636+0000 7f8e2a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b05e0 0x55895d4b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.636+0000 7f8e2ad9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 0x55895d4b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.636+0000 7f8e2a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b05e0 0x55895d4b50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39064/0 (socket says 172.21.15.70:39064) 2024-04-01T01:35:24.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.636+0000 7f8e2a59c640 1 -- 172.21.15.70:0/613275216 learned_addr learned my addr 172.21.15.70:0/613275216 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:24.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.636+0000 7f8e29d9b640 1 --2- 172.21.15.70:0/613275216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b5750 0x55895d4b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.637+0000 7f8e2ad9d640 1 -- 172.21.15.70:0/613275216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b5750 msgr2=0x55895d4b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:24.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.637+0000 7f8e2ad9d640 1 --2- 172.21.15.70:0/613275216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b5750 0x55895d4b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.637+0000 7f8e2ad9d640 1 -- 172.21.15.70:0/613275216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b05e0 msgr2=0x55895d4b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:24.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.637+0000 7f8e2ad9d640 1 --2- 172.21.15.70:0/613275216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b05e0 0x55895d4b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.637+0000 7f8e2ad9d640 1 -- 172.21.15.70:0/613275216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55895d1f07e0 con 0x55895d4afbf0 2024-04-01T01:35:24.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.637+0000 7f8e29d9b640 1 --2- 172.21.15.70:0/613275216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b5750 0x55895d4b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:24.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.637+0000 7f8e1b7fe640 1 --2- 172.21.15.70:0/1388050063 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e140a25f0 0x7f8e140a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.637+0000 7f8e1b7fe640 1 -- 172.21.15.70:0/1388050063 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8e1c0ce260 con 0x7f8e24093fe0 2024-04-01T01:35:24.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.637+0000 7f8e2a59c640 1 --2- 172.21.15.70:0/1388050063 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e140a25f0 0x7f8e140a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.637+0000 7f8e2ad9d640 1 --2- 172.21.15.70:0/613275216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 0x55895d4b0010 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c000b60 tx=0x7f8e2c074d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94095eb3962e212 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:24.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.638+0000 7f8e2a59c640 1 --2- 172.21.15.70:0/1388050063 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e140a25f0 0x7f8e140a4ab0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8e20005ec0 tx=0x7f8e20005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:24.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.638+0000 7f8e18ff9640 1 -- 172.21.15.70:0/613275216 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c084070 con 0x55895d4afbf0 2024-04-01T01:35:24.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.638+0000 7f8e18ff9640 1 -- 172.21.15.70:0/613275216 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e2c07f400 con 0x55895d4afbf0 2024-04-01T01:35:24.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.638+0000 7f8e18ff9640 1 -- 172.21.15.70:0/613275216 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c07e620 con 0x55895d4afbf0 2024-04-01T01:35:24.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.639+0000 7f8e33bb9740 1 -- 172.21.15.70:0/613275216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 msgr2=0x55895d4b0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.639+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/613275216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 0x55895d4b0010 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c000b60 tx=0x7f8e2c074d60 comp rx=0 tx=0).stop 2024-04-01T01:35:24.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.639+0000 7f8e33bb9740 1 -- 172.21.15.70:0/613275216 shutdown_connections 2024-04-01T01:35:24.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.639+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/613275216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b5750 0x55895d4b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.639+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/613275216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b05e0 0x55895d4b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.639+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/613275216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 0x55895d4b0010 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.639+0000 7f8e33bb9740 1 -- 172.21.15.70:0/613275216 >> 172.21.15.70:0/613275216 conn(0x55895d4ab160 msgr2=0x55895d4ab560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:24.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.640+0000 7f8e33bb9740 1 -- 172.21.15.70:0/613275216 shutdown_connections 2024-04-01T01:35:24.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.640+0000 7f8e33bb9740 1 -- 172.21.15.70:0/613275216 wait complete. 2024-04-01T01:35:24.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.640+0000 7f8e33bb9740 1 -- start start 2024-04-01T01:35:24.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.640+0000 7f8e33bb9740 1 Processor -- start 2024-04-01T01:35:24.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.641+0000 7f8e33bb9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 0x55895d4b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.641+0000 7f8e33bb9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b05e0 0x55895d4ba530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.641+0000 7f8e33bb9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b5750 0x55895d4c1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.641+0000 7f8e33bb9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55895d3b23d0 con 0x55895d4b05e0 2024-04-01T01:35:24.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.641+0000 7f8e33bb9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55895d40ff40 con 0x55895d4b5750 2024-04-01T01:35:24.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.641+0000 7f8e33bb9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55895d4b8700 con 0x55895d4afbf0 2024-04-01T01:35:24.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.641+0000 7f8e33bb9740 10 client.0 _ll_register_callbacks cb 0x55895d3cfb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:24.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.641+0000 7f8e2ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b5750 0x55895d4c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.641+0000 7f8e2ad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b5750 0x55895d4c1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35436/0 (socket says 172.21.15.70:35436) 2024-04-01T01:35:24.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.641+0000 7f8e2ad9d640 1 -- 172.21.15.70:0/164254267 learned_addr learned my addr 172.21.15.70:0/164254267 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:24.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.641+0000 7f8e29d9b640 1 --2- 172.21.15.70:0/164254267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b05e0 0x55895d4ba530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.641+0000 7f8e2ad9d640 1 -- 172.21.15.70:0/164254267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 msgr2=0x55895d4b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.641+0000 7f8e2a59c640 1 --2- 172.21.15.70:0/164254267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 0x55895d4b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.642+0000 7f8e2ad9d640 1 --2- 172.21.15.70:0/164254267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 0x55895d4b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.642+0000 7f8e2ad9d640 1 -- 172.21.15.70:0/164254267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b05e0 msgr2=0x55895d4ba530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.642+0000 7f8e2ad9d640 1 --2- 172.21.15.70:0/164254267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b05e0 0x55895d4ba530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.642+0000 7f8e2ad9d640 1 -- 172.21.15.70:0/164254267 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55895d250f70 con 0x55895d4b5750 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.642+0000 7f8e29d9b640 1 --2- 172.21.15.70:0/164254267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b05e0 0x55895d4ba530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.642+0000 7f8e2a59c640 1 --2- 172.21.15.70:0/164254267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 0x55895d4b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.642+0000 7f8e2ad9d640 1 --2- 172.21.15.70:0/164254267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b5750 0x55895d4c1570 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c050720 tx=0x7f8e2c069b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81ab4a9aa2b2f0b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.642+0000 7f8e11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.642+0000 7f8e11ffb640 1 -- 172.21.15.70:0/164254267 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c084040 con 0x55895d4b5750 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.642+0000 7f8e33bb9740 1 -- 172.21.15.70:0/164254267 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e2c07f400 con 0x55895d4b5750 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.642+0000 7f8e11ffb640 1 -- 172.21.15.70:0/164254267 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8e2c07f670 con 0x55895d4b5750 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.643+0000 7f8e11ffb640 1 -- 172.21.15.70:0/164254267 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8e2c07f8b0 con 0x55895d4b5750 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.643+0000 7f8e33bb9740 10 client.26042 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.643+0000 7f8e33bb9740 1 -- 172.21.15.70:0/164254267 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55895d0e8170 con 0x55895d4b5750 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.643+0000 7f8e11ffb640 1 -- 172.21.15.70:0/164254267 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e2c07f400 con 0x55895d4b5750 2024-04-01T01:35:24.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.643+0000 7f8e11ffb640 1 -- 172.21.15.70:0/164254267 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8e2c087070 con 0x55895d4b5750 2024-04-01T01:35:24.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.643+0000 7f8e33bb9740 10 client.26042 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:24.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.643+0000 7f8e33bb9740 10 client.26042 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:24.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.643+0000 7f8e33bb9740 20 client.26042 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:24.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.643+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/164254267 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55895d4c5d70 0x55895d4c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.643+0000 7f8e33bb9740 4 client.26042 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:24.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.643+0000 7f8e33bb9740 1 -- 172.21.15.70:0/164254267 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55895d4213e0 con 0x55895d4c5d70 2024-04-01T01:35:24.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.643+0000 7f8e2a59c640 1 --2- 172.21.15.70:0/164254267 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55895d4c5d70 0x55895d4c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.644+0000 7f8e2a59c640 1 --2- 172.21.15.70:0/164254267 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55895d4c5d70 0x55895d4c81f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c062591e700b727f server_cookie=17b502e87ad02aad in_seq=0 out_seq=0 2024-04-01T01:35:24.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.644+0000 7f8e11ffb640 10 client.26042 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:24.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.646+0000 7f8e11ffb640 1 -- 172.21.15.70:0/164254267 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55895d4213e0 con 0x55895d4c5d70 2024-04-01T01:35:24.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.646+0000 7f8e11ffb640 10 client.26042 handle_command_reply: tid=0 2024-04-01T01:35:24.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.646+0000 7f8e33bb9740 1 client.26042 shutdown 2024-04-01T01:35:24.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.646+0000 7f8e33bb9740 1 -- 172.21.15.70:0/164254267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b5750 msgr2=0x55895d4c1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.646+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/164254267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b5750 0x55895d4c1570 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c050720 tx=0x7f8e2c069b20 comp rx=0 tx=0).stop 2024-04-01T01:35:24.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.647+0000 7f8e33bb9740 1 -- 172.21.15.70:0/164254267 shutdown_connections 2024-04-01T01:35:24.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.647+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/164254267 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55895d4c5d70 0x55895d4c81f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.647+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/164254267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b5750 0x55895d4c1570 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.647+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/164254267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b05e0 0x55895d4ba530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.647+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/164254267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 0x55895d4b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.647+0000 7f8e33bb9740 1 -- 172.21.15.70:0/164254267 >> 172.21.15.70:0/164254267 conn(0x55895d4ab160 msgr2=0x55895d4aa2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:24.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.647+0000 7f8e11ffb640 0 client.26042 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:24.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.647+0000 7f8e33bb9740 1 -- 172.21.15.70:0/164254267 shutdown_connections 2024-04-01T01:35:24.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.647+0000 7f8e33bb9740 1 -- 172.21.15.70:0/164254267 wait complete. 2024-04-01T01:35:24.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.647+0000 7f8e33bb9740 20 client.26042 trim_cache size 0 max 16384 2024-04-01T01:35:24.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.681+0000 7f8e33bb9740 1 Processor -- start 2024-04-01T01:35:24.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.681+0000 7f8e33bb9740 1 -- start start 2024-04-01T01:35:24.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.681+0000 7f8e33bb9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4afbf0 0x55895d4b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.681+0000 7f8e33bb9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b05e0 0x55895d4c1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.681+0000 7f8e33bb9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4b5750 0x55895d4ba530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.681+0000 7f8e33bb9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c020cd0 con 0x55895d4afbf0 2024-04-01T01:35:24.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.681+0000 7f8e33bb9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c022810 con 0x55895d4b05e0 2024-04-01T01:35:24.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.681+0000 7f8e33bb9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c0213a0 con 0x55895d4b5750 2024-04-01T01:35:24.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.682+0000 7f8e2a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4afbf0 0x55895d4b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.682+0000 7f8e2a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4afbf0 0x55895d4b9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39092/0 (socket says 172.21.15.70:39092) 2024-04-01T01:35:24.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.682+0000 7f8e2a59c640 1 -- 172.21.15.70:0/3726052619 learned_addr learned my addr 172.21.15.70:0/3726052619 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:24.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.682+0000 7f8e29d9b640 1 --2- 172.21.15.70:0/3726052619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b05e0 0x55895d4c1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.682+0000 7f8e2ad9d640 1 --2- 172.21.15.70:0/3726052619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4b5750 0x55895d4ba530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.682+0000 7f8e2a59c640 1 -- 172.21.15.70:0/3726052619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4b5750 msgr2=0x55895d4ba530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:24.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.682+0000 7f8e2a59c640 1 --2- 172.21.15.70:0/3726052619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4b5750 0x55895d4ba530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.682+0000 7f8e2a59c640 1 -- 172.21.15.70:0/3726052619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b05e0 msgr2=0x55895d4c1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:24.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.682+0000 7f8e2a59c640 1 --2- 172.21.15.70:0/3726052619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b05e0 0x55895d4c1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.682+0000 7f8e2a59c640 1 -- 172.21.15.70:0/3726052619 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55895d4abed0 con 0x55895d4afbf0 2024-04-01T01:35:24.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.682+0000 7f8e2a59c640 1 --2- 172.21.15.70:0/3726052619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4afbf0 0x55895d4b9ff0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8e2002bee0 tx=0x7f8e20042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21af42476b94235f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:24.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.683+0000 7f8e13fff640 1 -- 172.21.15.70:0/3726052619 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e20004020 con 0x55895d4afbf0 2024-04-01T01:35:24.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.683+0000 7f8e13fff640 1 -- 172.21.15.70:0/3726052619 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e2000cce0 con 0x55895d4afbf0 2024-04-01T01:35:24.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.683+0000 7f8e13fff640 1 -- 172.21.15.70:0/3726052619 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e20055540 con 0x55895d4afbf0 2024-04-01T01:35:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.684+0000 7f8e33bb9740 1 -- 172.21.15.70:0/3726052619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4afbf0 msgr2=0x55895d4b9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.684+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/3726052619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4afbf0 0x55895d4b9ff0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8e2002bee0 tx=0x7f8e20042000 comp rx=0 tx=0).stop 2024-04-01T01:35:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.684+0000 7f8e33bb9740 1 -- 172.21.15.70:0/3726052619 shutdown_connections 2024-04-01T01:35:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.684+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/3726052619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4b5750 0x55895d4ba530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.684+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/3726052619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b05e0 0x55895d4c1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.684+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/3726052619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4afbf0 0x55895d4b9ff0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.684+0000 7f8e33bb9740 1 -- 172.21.15.70:0/3726052619 >> 172.21.15.70:0/3726052619 conn(0x55895d4ab160 msgr2=0x55895d4b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.684+0000 7f8e33bb9740 1 -- 172.21.15.70:0/3726052619 shutdown_connections 2024-04-01T01:35:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.684+0000 7f8e33bb9740 1 -- 172.21.15.70:0/3726052619 wait complete. 2024-04-01T01:35:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.685+0000 7f8e33bb9740 1 -- start start 2024-04-01T01:35:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.685+0000 7f8e33bb9740 1 Processor -- start 2024-04-01T01:35:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.685+0000 7f8e33bb9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 0x55895d4be5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.685+0000 7f8e33bb9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b05e0 0x55895d4beb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.686+0000 7f8e33bb9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b5750 0x55895d4bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.686+0000 7f8e33bb9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c022140 con 0x55895d4b5750 2024-04-01T01:35:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.686+0000 7f8e33bb9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c021a70 con 0x55895d4b05e0 2024-04-01T01:35:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.686+0000 7f8e33bb9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c01ecb0 con 0x55895d4afbf0 2024-04-01T01:35:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.686+0000 7f8e33bb9740 10 client.0 _ll_register_callbacks cb 0x7f8e2000d790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.686+0000 7f8e29d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b05e0 0x55895d4beb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.686+0000 7f8e29d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b05e0 0x55895d4beb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35460/0 (socket says 172.21.15.70:35460) 2024-04-01T01:35:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.686+0000 7f8e29d9b640 1 -- 172.21.15.70:0/1444657568 learned_addr learned my addr 172.21.15.70:0/1444657568 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.686+0000 7f8e2a59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 0x55895d4be5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.686+0000 7f8e2ad9d640 1 --2- 172.21.15.70:0/1444657568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b5750 0x55895d4bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.686+0000 7f8e29d9b640 1 -- 172.21.15.70:0/1444657568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 msgr2=0x55895d4be5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.686+0000 7f8e29d9b640 1 --2- 172.21.15.70:0/1444657568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 0x55895d4be5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.686+0000 7f8e29d9b640 1 -- 172.21.15.70:0/1444657568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b5750 msgr2=0x55895d4bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.686+0000 7f8e29d9b640 1 --2- 172.21.15.70:0/1444657568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b5750 0x55895d4bfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.686+0000 7f8e29d9b640 1 -- 172.21.15.70:0/1444657568 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55895d4c0120 con 0x55895d4b05e0 2024-04-01T01:35:24.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.687+0000 7f8e29d9b640 1 --2- 172.21.15.70:0/1444657568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b05e0 0x55895d4beb20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c0a2e30 tx=0x7f8e1c0d1f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b313fb9efd8bddd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:24.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.687+0000 7f8e11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:24.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.687+0000 7f8e11ffb640 1 -- 172.21.15.70:0/1444657568 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c0042f0 con 0x55895d4b05e0 2024-04-01T01:35:24.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.687+0000 7f8e11ffb640 1 -- 172.21.15.70:0/1444657568 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8e1c099da0 con 0x55895d4b05e0 2024-04-01T01:35:24.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.687+0000 7f8e33bb9740 1 -- 172.21.15.70:0/1444657568 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e2000cce0 con 0x55895d4b05e0 2024-04-01T01:35:24.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.687+0000 7f8e11ffb640 1 -- 172.21.15.70:0/1444657568 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8e1c09b050 con 0x55895d4b05e0 2024-04-01T01:35:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.688+0000 7f8e33bb9740 10 client.26048 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.688+0000 7f8e33bb9740 1 -- 172.21.15.70:0/1444657568 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55895d4c14d0 con 0x55895d4b05e0 2024-04-01T01:35:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.689+0000 7f8e11ffb640 1 -- 172.21.15.70:0/1444657568 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e2000cce0 con 0x55895d4b05e0 2024-04-01T01:35:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.689+0000 7f8e11ffb640 1 -- 172.21.15.70:0/1444657568 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8e1c099810 con 0x55895d4b05e0 2024-04-01T01:35:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.689+0000 7f8e33bb9740 10 client.26048 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.689+0000 7f8e33bb9740 10 client.26048 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.689+0000 7f8e33bb9740 20 client.26048 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.689+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/1444657568 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55895d52c570 0x55895d52e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.689+0000 7f8e33bb9740 4 client.26048 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:35:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.689+0000 7f8e33bb9740 1 -- 172.21.15.70:0/1444657568 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55895d41b730 con 0x55895d52c570 2024-04-01T01:35:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.689+0000 7f8e2a59c640 1 --2- 172.21.15.70:0/1444657568 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55895d52c570 0x55895d52e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.690+0000 7f8e11ffb640 10 client.26048 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.690+0000 7f8e2a59c640 1 --2- 172.21.15.70:0/1444657568 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55895d52c570 0x55895d52e990 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46d3c89ad8e2270 server_cookie=7cdcca799370bd97 in_seq=0 out_seq=0 2024-04-01T01:35:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.692+0000 7f8e11ffb640 1 -- 172.21.15.70:0/1444657568 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55895d41b730 con 0x55895d52c570 2024-04-01T01:35:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.692+0000 7f8e11ffb640 10 client.26048 handle_command_reply: tid=0 2024-04-01T01:35:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.692+0000 7f8e33bb9740 1 client.26048 shutdown 2024-04-01T01:35:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.692+0000 7f8e33bb9740 1 -- 172.21.15.70:0/1444657568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b05e0 msgr2=0x55895d4beb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.692+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/1444657568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b05e0 0x55895d4beb20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c0a2e30 tx=0x7f8e1c0d1f80 comp rx=0 tx=0).stop 2024-04-01T01:35:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.692+0000 7f8e33bb9740 1 -- 172.21.15.70:0/1444657568 shutdown_connections 2024-04-01T01:35:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.692+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/1444657568 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55895d52c570 0x55895d52e990 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.692+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/1444657568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55895d4b5750 0x55895d4bfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.692+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/1444657568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55895d4b05e0 0x55895d4beb20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.692+0000 7f8e33bb9740 1 --2- 172.21.15.70:0/1444657568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55895d4afbf0 0x55895d4be5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.692+0000 7f8e11ffb640 0 client.26048 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.692+0000 7f8e33bb9740 1 -- 172.21.15.70:0/1444657568 >> 172.21.15.70:0/1444657568 conn(0x55895d4ab160 msgr2=0x55895d4b9fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.693+0000 7f8e33bb9740 1 -- 172.21.15.70:0/1444657568 shutdown_connections 2024-04-01T01:35:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.693+0000 7f8e33bb9740 1 -- 172.21.15.70:0/1444657568 wait complete. 2024-04-01T01:35:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.693+0000 7f8e33bb9740 20 client.26048 trim_cache size 0 max 16384 2024-04-01T01:35:24.694 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:35:24.694 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:35:24.694 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "ba8b3a7e-cda6-4ff4-b01f-793d064d40a9", 2024-04-01T01:35:24.694 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:35:24.694 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:35:24.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.694+0000 7f8e30fd0640 1 -- 172.21.15.70:0/1388050063 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e140a25f0 msgr2=0x7f8e140a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.694+0000 7f8e30fd0640 1 --2- 172.21.15.70:0/1388050063 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e140a25f0 0x7f8e140a4ab0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8e20005ec0 tx=0x7f8e20005a20 comp rx=0 tx=0).stop 2024-04-01T01:35:24.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.694+0000 7f8e30fd0640 1 -- 172.21.15.70:0/1388050063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e24093fe0 msgr2=0x7f8e2412a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.694+0000 7f8e30fd0640 1 --2- 172.21.15.70:0/1388050063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e24093fe0 0x7f8e2412a380 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c005fd0 tx=0x7f8e1c00df80 comp rx=0 tx=0).stop 2024-04-01T01:35:24.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.694+0000 7f8e30fd0640 1 -- 172.21.15.70:0/1388050063 shutdown_connections 2024-04-01T01:35:24.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.694+0000 7f8e30fd0640 1 --2- 172.21.15.70:0/1388050063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e24095710 0x7f8e241426e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.694+0000 7f8e30fd0640 1 --2- 172.21.15.70:0/1388050063 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e140a25f0 0x7f8e140a4ab0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.694+0000 7f8e30fd0640 1 --2- 172.21.15.70:0/1388050063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e24093fe0 0x7f8e2412a380 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.694+0000 7f8e30fd0640 1 --2- 172.21.15.70:0/1388050063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e24004850 0x7f8e24129e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.694+0000 7f8e30fd0640 1 -- 172.21.15.70:0/1388050063 >> 172.21.15.70:0/1388050063 conn(0x7f8e2408f530 msgr2=0x7f8e24090d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:24.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.695+0000 7f8e30fd0640 1 -- 172.21.15.70:0/1388050063 shutdown_connections 2024-04-01T01:35:24.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.695+0000 7f8e30fd0640 1 -- 172.21.15.70:0/1388050063 wait complete. 2024-04-01T01:35:24.716 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:35:24.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:35:24.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.909+0000 7fe3e4c9c640 1 -- 172.21.15.70:0/3524215329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e0100650 msgr2=0x7fe3e0100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.909+0000 7fe3e4c9c640 1 --2- 172.21.15.70:0/3524215329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e0100650 0x7fe3e0100a70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009900 tx=0x7fe3d002bc40 comp rx=0 tx=0).stop 2024-04-01T01:35:24.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.910+0000 7fe3e4c9c640 1 -- 172.21.15.70:0/3524215329 shutdown_connections 2024-04-01T01:35:24.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.910+0000 7fe3e4c9c640 1 --2- 172.21.15.70:0/3524215329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e0102200 0x7fe3e0108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.910+0000 7fe3e4c9c640 1 --2- 172.21.15.70:0/3524215329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3e0101840 0x7fe3e0101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.910+0000 7fe3e4c9c640 1 --2- 172.21.15.70:0/3524215329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e0100650 0x7fe3e0100a70 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.910+0000 7fe3e4c9c640 1 -- 172.21.15.70:0/3524215329 >> 172.21.15.70:0/3524215329 conn(0x7fe3e00fbbe0 msgr2=0x7fe3e00fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:24.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.910+0000 7fe3e4c9c640 1 -- 172.21.15.70:0/3524215329 shutdown_connections 2024-04-01T01:35:24.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.910+0000 7fe3e4c9c640 1 -- 172.21.15.70:0/3524215329 wait complete. 2024-04-01T01:35:24.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.911+0000 7fe3e4c9c640 1 Processor -- start 2024-04-01T01:35:24.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.911+0000 7fe3e4c9c640 1 -- start start 2024-04-01T01:35:24.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.911+0000 7fe3e4c9c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e0100650 0x7fe3e0199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.911+0000 7fe3e4c9c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e0101840 0x7fe3e019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.911+0000 7fe3e4c9c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3e0102200 0x7fe3e01a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.911+0000 7fe3e4c9c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e010b090 con 0x7fe3e0102200 2024-04-01T01:35:24.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.911+0000 7fe3e4c9c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e010af10 con 0x7fe3e0101840 2024-04-01T01:35:24.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.911+0000 7fe3e4c9c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e010b210 con 0x7fe3e0100650 2024-04-01T01:35:24.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.911+0000 7fe3ded9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3e0102200 0x7fe3e01a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.911+0000 7fe3ded9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3e0102200 0x7fe3e01a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39110/0 (socket says 172.21.15.70:39110) 2024-04-01T01:35:24.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.911+0000 7fe3ded9d640 1 -- 172.21.15.70:0/3430963340 learned_addr learned my addr 172.21.15.70:0/3430963340 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:24.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.912+0000 7fe3ddd9b640 1 --2- 172.21.15.70:0/3430963340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e0101840 0x7fe3e019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.912+0000 7fe3de59c640 1 --2- 172.21.15.70:0/3430963340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e0100650 0x7fe3e0199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.912+0000 7fe3ded9d640 1 -- 172.21.15.70:0/3430963340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e0100650 msgr2=0x7fe3e0199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.912+0000 7fe3ded9d640 1 --2- 172.21.15.70:0/3430963340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e0100650 0x7fe3e0199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.912+0000 7fe3ded9d640 1 -- 172.21.15.70:0/3430963340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e0101840 msgr2=0x7fe3e019a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.912+0000 7fe3ded9d640 1 --2- 172.21.15.70:0/3430963340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e0101840 0x7fe3e019a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.912+0000 7fe3ded9d640 1 -- 172.21.15.70:0/3430963340 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e01a17e0 con 0x7fe3e0102200 2024-04-01T01:35:24.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.912+0000 7fe3de59c640 1 --2- 172.21.15.70:0/3430963340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e0100650 0x7fe3e0199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:35:24.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.912+0000 7fe3ddd9b640 1 --2- 172.21.15.70:0/3430963340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e0101840 0x7fe3e019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:24.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.912+0000 7fe3ded9d640 1 --2- 172.21.15.70:0/3430963340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3e0102200 0x7fe3e01a1170 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800cad0 tx=0x7fe3d800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:24.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.912+0000 7fe3cf7fe640 1 -- 172.21.15.70:0/3430963340 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d800ae20 con 0x7fe3e0102200 2024-04-01T01:35:24.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.912+0000 7fe3e4c9c640 1 -- 172.21.15.70:0/3430963340 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e01a1ad0 con 0x7fe3e0102200 2024-04-01T01:35:24.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.912+0000 7fe3cf7fe640 1 -- 172.21.15.70:0/3430963340 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3d8005020 con 0x7fe3e0102200 2024-04-01T01:35:24.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.912+0000 7fe3cf7fe640 1 -- 172.21.15.70:0/3430963340 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d800f6b0 con 0x7fe3e0102200 2024-04-01T01:35:24.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.912+0000 7fe3e4c9c640 1 -- 172.21.15.70:0/3430963340 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e01a2020 con 0x7fe3e0102200 2024-04-01T01:35:24.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.916+0000 7fe3e787d740 1 Processor -- start 2024-04-01T01:35:24.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.916+0000 7fe3cf7fe640 1 -- 172.21.15.70:0/3430963340 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fe3d80051d0 con 0x7fe3e0102200 2024-04-01T01:35:24.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.917+0000 7fe3e787d740 1 -- start start 2024-04-01T01:35:24.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.917+0000 7fe3cf7fe640 1 --2- 172.21.15.70:0/3430963340 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe3c80a2570 0x7fe3c80a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.917+0000 7fe3e787d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270a2b0 0x560e2270a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.917+0000 7fe3e787d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e2270f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.917+0000 7fe3e787d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270fe10 0x560e227122a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.917+0000 7fe3e787d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560e2266f180 con 0x560e2270aca0 2024-04-01T01:35:24.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.917+0000 7fe3e787d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560e22617b70 con 0x560e2270a2b0 2024-04-01T01:35:24.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.917+0000 7fe3e787d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560e2266b220 con 0x560e2270fe10 2024-04-01T01:35:24.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.917+0000 7fe3cf7fe640 1 -- 172.21.15.70:0/3430963340 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe3d80ceaf0 con 0x7fe3e0102200 2024-04-01T01:35:24.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.917+0000 7fe3ded9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270fe10 0x560e227122a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.917+0000 7fe3de59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270a2b0 0x560e2270a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.917+0000 7fe3ded9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270fe10 0x560e227122a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:38002/0 (socket says 172.21.15.70:38002) 2024-04-01T01:35:24.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.917+0000 7fe3ded9d640 1 -- 172.21.15.70:0/3486913918 learned_addr learned my addr 172.21.15.70:0/3486913918 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:24.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.917+0000 7fe3ddd9b640 1 --2- 172.21.15.70:0/3486913918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e2270f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.917+0000 7fe3de59c640 1 -- 172.21.15.70:0/3486913918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270fe10 msgr2=0x560e227122a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:24.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.918+0000 7fe3de59c640 1 --2- 172.21.15.70:0/3486913918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270fe10 0x560e227122a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.918+0000 7fe3de59c640 1 -- 172.21.15.70:0/3486913918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 msgr2=0x560e2270f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:24.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.918+0000 7fe3de59c640 1 --2- 172.21.15.70:0/3486913918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e2270f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.918+0000 7fe3de59c640 1 -- 172.21.15.70:0/3486913918 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e2262b7c0 con 0x560e2270a2b0 2024-04-01T01:35:24.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.918+0000 7fe3ddd9b640 1 --2- 172.21.15.70:0/3486913918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e2270f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:35:24.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.918+0000 7fe3ddd9b640 1 --2- 172.21.15.70:0/3430963340 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe3c80a2570 0x7fe3c80a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.918+0000 7fe3de59c640 1 --2- 172.21.15.70:0/3486913918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270a2b0 0x560e2270a6d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe3d002c1d0 tx=0x7fe3d0009630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2d061991230e9f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:24.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.918+0000 7fe3ccff9640 1 -- 172.21.15.70:0/3486913918 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0043020 con 0x560e2270a2b0 2024-04-01T01:35:24.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.918+0000 7fe3ccff9640 1 -- 172.21.15.70:0/3486913918 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3d002c9f0 con 0x560e2270a2b0 2024-04-01T01:35:24.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.919+0000 7fe3ccff9640 1 -- 172.21.15.70:0/3486913918 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d00046c0 con 0x560e2270a2b0 2024-04-01T01:35:24.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.919+0000 7fe3e787d740 1 -- 172.21.15.70:0/3486913918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270a2b0 msgr2=0x560e2270a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.919+0000 7fe3ddd9b640 1 --2- 172.21.15.70:0/3430963340 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe3c80a2570 0x7fe3c80a4a30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x560e2270b0a0 tx=0x7fe3d4004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:24.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.919+0000 7fe3e787d740 1 --2- 172.21.15.70:0/3486913918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270a2b0 0x560e2270a6d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe3d002c1d0 tx=0x7fe3d0009630 comp rx=0 tx=0).stop 2024-04-01T01:35:24.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.919+0000 7fe3e787d740 1 -- 172.21.15.70:0/3486913918 shutdown_connections 2024-04-01T01:35:24.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.919+0000 7fe3e787d740 1 --2- 172.21.15.70:0/3486913918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270fe10 0x560e227122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.919+0000 7fe3e787d740 1 --2- 172.21.15.70:0/3486913918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e2270f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.919+0000 7fe3e787d740 1 --2- 172.21.15.70:0/3486913918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270a2b0 0x560e2270a6d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.919+0000 7fe3e787d740 1 -- 172.21.15.70:0/3486913918 >> 172.21.15.70:0/3486913918 conn(0x560e22705820 msgr2=0x560e22705c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:24.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.920+0000 7fe3e787d740 1 -- 172.21.15.70:0/3486913918 shutdown_connections 2024-04-01T01:35:24.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.920+0000 7fe3e787d740 1 -- 172.21.15.70:0/3486913918 wait complete. 2024-04-01T01:35:24.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.920+0000 7fe3e787d740 1 -- start start 2024-04-01T01:35:24.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.920+0000 7fe3e787d740 1 Processor -- start 2024-04-01T01:35:24.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.920+0000 7fe3e787d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270a2b0 0x560e22714810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.920+0000 7fe3e787d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e22714d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.920+0000 7fe3e787d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270fe10 0x560e2271bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.920+0000 7fe3e787d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560e22712db0 con 0x560e2270aca0 2024-04-01T01:35:24.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.920+0000 7fe3e787d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560e2260d4a0 con 0x560e2270fe10 2024-04-01T01:35:24.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.920+0000 7fe3e787d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560e22712f30 con 0x560e2270a2b0 2024-04-01T01:35:24.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.920+0000 7fe3e787d740 10 client.0 _ll_register_callbacks cb 0x560e22610d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:24.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.921+0000 7fe3de59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270a2b0 0x560e22714810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.921+0000 7fe3de59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270a2b0 0x560e22714810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:38008/0 (socket says 172.21.15.70:38008) 2024-04-01T01:35:24.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.921+0000 7fe3de59c640 1 -- 172.21.15.70:0/450799132 learned_addr learned my addr 172.21.15.70:0/450799132 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:24.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.921+0000 7fe3ded9d640 1 --2- 172.21.15.70:0/450799132 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270fe10 0x560e2271bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.921+0000 7fe3ddd9b640 1 --2- 172.21.15.70:0/450799132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e22714d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.921+0000 7fe3de59c640 1 -- 172.21.15.70:0/450799132 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270fe10 msgr2=0x560e2271bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:24.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.921+0000 7fe3de59c640 1 --2- 172.21.15.70:0/450799132 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270fe10 0x560e2271bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.921+0000 7fe3de59c640 1 -- 172.21.15.70:0/450799132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 msgr2=0x560e22714d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:24.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.921+0000 7fe3de59c640 1 --2- 172.21.15.70:0/450799132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e22714d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.921+0000 7fe3de59c640 1 -- 172.21.15.70:0/450799132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e2244d7d0 con 0x560e2270a2b0 2024-04-01T01:35:24.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.922+0000 7fe3de59c640 1 --2- 172.21.15.70:0/450799132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270a2b0 0x560e22714810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009850 tx=0x7fe3d0004510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5c9aedea0cb1f10 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:24.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.922+0000 7fe3c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:35:24.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.922+0000 7fe3c5ffb640 1 -- 172.21.15.70:0/450799132 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d004e020 con 0x560e2270a2b0 2024-04-01T01:35:24.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.922+0000 7fe3c5ffb640 1 -- 172.21.15.70:0/450799132 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe3d00037f0 con 0x560e2270a2b0 2024-04-01T01:35:24.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.922+0000 7fe3e787d740 1 -- 172.21.15.70:0/450799132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3d002c9f0 con 0x560e2270a2b0 2024-04-01T01:35:24.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.923+0000 7fe3c5ffb640 1 -- 172.21.15.70:0/450799132 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe3d0059050 con 0x560e2270a2b0 2024-04-01T01:35:24.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.923+0000 7fe3e787d740 10 client.25963 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:24.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.923+0000 7fe3e787d740 1 -- 172.21.15.70:0/450799132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560e223833f0 con 0x560e2270a2b0 2024-04-01T01:35:24.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.923+0000 7fe3c5ffb640 1 -- 172.21.15.70:0/450799132 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3d002c9f0 con 0x560e2270a2b0 2024-04-01T01:35:24.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.923+0000 7fe3c5ffb640 1 -- 172.21.15.70:0/450799132 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe3d004aab0 con 0x560e2270a2b0 2024-04-01T01:35:24.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.923+0000 7fe3e787d740 10 client.25963 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:24.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.923+0000 7fe3e787d740 10 client.25963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:24.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.923+0000 7fe3e787d740 20 client.25963 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:24.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.923+0000 7fe3e787d740 1 --2- 172.21.15.70:0/450799132 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560e22720750 0x560e22722ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.923+0000 7fe3e787d740 4 client.25963 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:24.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.923+0000 7fe3e787d740 1 -- 172.21.15.70:0/450799132 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e2267c6c0 con 0x560e22720750 2024-04-01T01:35:24.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.924+0000 7fe3ddd9b640 1 --2- 172.21.15.70:0/450799132 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560e22720750 0x560e22722ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.925+0000 7fe3ddd9b640 1 --2- 172.21.15.70:0/450799132 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560e22720750 0x560e22722ba0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af39d1e13d1f8ef7 server_cookie=fc8b8da4e9d635e1 in_seq=0 out_seq=0 2024-04-01T01:35:24.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.925+0000 7fe3c5ffb640 10 client.25963 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:24.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.926+0000 7fe3c5ffb640 1 -- 172.21.15.70:0/450799132 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560e2267c6c0 con 0x560e22720750 2024-04-01T01:35:24.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.926+0000 7fe3c5ffb640 10 client.25963 handle_command_reply: tid=0 2024-04-01T01:35:24.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.926+0000 7fe3e787d740 1 client.25963 shutdown 2024-04-01T01:35:24.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.926+0000 7fe3e787d740 1 -- 172.21.15.70:0/450799132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270a2b0 msgr2=0x560e22714810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.926+0000 7fe3e787d740 1 --2- 172.21.15.70:0/450799132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270a2b0 0x560e22714810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009850 tx=0x7fe3d0004510 comp rx=0 tx=0).stop 2024-04-01T01:35:24.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.926+0000 7fe3e787d740 1 -- 172.21.15.70:0/450799132 shutdown_connections 2024-04-01T01:35:24.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.926+0000 7fe3e787d740 1 --2- 172.21.15.70:0/450799132 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560e22720750 0x560e22722ba0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.926+0000 7fe3e787d740 1 --2- 172.21.15.70:0/450799132 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270fe10 0x560e2271bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.926+0000 7fe3e787d740 1 --2- 172.21.15.70:0/450799132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e22714d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.926+0000 7fe3e787d740 1 --2- 172.21.15.70:0/450799132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270a2b0 0x560e22714810 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.926+0000 7fe3e787d740 1 -- 172.21.15.70:0/450799132 >> 172.21.15.70:0/450799132 conn(0x560e22705820 msgr2=0x560e22704c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:24.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.926+0000 7fe3c5ffb640 0 client.25963 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:24.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.927+0000 7fe3e787d740 1 -- 172.21.15.70:0/450799132 shutdown_connections 2024-04-01T01:35:24.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.927+0000 7fe3e787d740 1 -- 172.21.15.70:0/450799132 wait complete. 2024-04-01T01:35:24.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.927+0000 7fe3e787d740 20 client.25963 trim_cache size 0 max 16384 2024-04-01T01:35:24.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.961+0000 7fe3e787d740 1 Processor -- start 2024-04-01T01:35:24.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.961+0000 7fe3e787d740 1 -- start start 2024-04-01T01:35:24.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.961+0000 7fe3e787d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270a2b0 0x560e22704390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.961+0000 7fe3e787d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e2271bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.961+0000 7fe3e787d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270fe10 0x560e227048d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.961+0000 7fe3de59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270a2b0 0x560e22704390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.961+0000 7fe3de59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270a2b0 0x560e22704390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35522/0 (socket says 172.21.15.70:35522) 2024-04-01T01:35:24.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.961+0000 7fe3de59c640 1 -- 172.21.15.70:0/3488578582 learned_addr learned my addr 172.21.15.70:0/3488578582 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:24.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.962+0000 7fe3ded9d640 1 --2- 172.21.15.70:0/3488578582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270fe10 0x560e227048d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.962+0000 7fe3e787d740 1 -- 172.21.15.70:0/3488578582 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e01ac670 con 0x560e2270aca0 2024-04-01T01:35:24.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.962+0000 7fe3e787d740 1 -- 172.21.15.70:0/3488578582 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e01ae1b0 con 0x560e2270a2b0 2024-04-01T01:35:24.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.962+0000 7fe3e787d740 1 -- 172.21.15.70:0/3488578582 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e01acd40 con 0x560e2270fe10 2024-04-01T01:35:24.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.962+0000 7fe3ddd9b640 1 --2- 172.21.15.70:0/3488578582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e2271bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.962+0000 7fe3de59c640 1 -- 172.21.15.70:0/3488578582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270fe10 msgr2=0x560e227048d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:24.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.962+0000 7fe3de59c640 1 --2- 172.21.15.70:0/3488578582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270fe10 0x560e227048d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.962+0000 7fe3de59c640 1 -- 172.21.15.70:0/3488578582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 msgr2=0x560e2271bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:24.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.962+0000 7fe3de59c640 1 --2- 172.21.15.70:0/3488578582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e2271bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.962+0000 7fe3de59c640 1 -- 172.21.15.70:0/3488578582 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e223430f0 con 0x560e2270a2b0 2024-04-01T01:35:24.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.962+0000 7fe3de59c640 1 --2- 172.21.15.70:0/3488578582 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270a2b0 0x560e22704390 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe3d004ae50 tx=0x7fe3d0005cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6a247fb15c1ac4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:24.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.963+0000 7fe3c7fff640 1 -- 172.21.15.70:0/3488578582 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0062070 con 0x560e2270a2b0 2024-04-01T01:35:24.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.963+0000 7fe3c7fff640 1 -- 172.21.15.70:0/3488578582 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe3d0004350 con 0x560e2270a2b0 2024-04-01T01:35:24.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.963+0000 7fe3c7fff640 1 -- 172.21.15.70:0/3488578582 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d004e070 con 0x560e2270a2b0 2024-04-01T01:35:24.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.964+0000 7fe3e787d740 1 -- 172.21.15.70:0/3488578582 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270a2b0 msgr2=0x560e22704390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.964+0000 7fe3e787d740 1 --2- 172.21.15.70:0/3488578582 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270a2b0 0x560e22704390 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe3d004ae50 tx=0x7fe3d0005cb0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.964+0000 7fe3e787d740 1 -- 172.21.15.70:0/3488578582 shutdown_connections 2024-04-01T01:35:24.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.964+0000 7fe3e787d740 1 --2- 172.21.15.70:0/3488578582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270fe10 0x560e227048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.964+0000 7fe3e787d740 1 --2- 172.21.15.70:0/3488578582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e2271bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.964+0000 7fe3e787d740 1 --2- 172.21.15.70:0/3488578582 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270a2b0 0x560e22704390 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.964+0000 7fe3e787d740 1 -- 172.21.15.70:0/3488578582 >> 172.21.15.70:0/3488578582 conn(0x560e22705820 msgr2=0x560e22705c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:24.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.965+0000 7fe3e787d740 1 -- 172.21.15.70:0/3488578582 shutdown_connections 2024-04-01T01:35:24.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.966+0000 7fe3e787d740 1 -- 172.21.15.70:0/3488578582 wait complete. 2024-04-01T01:35:24.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.966+0000 7fe3e787d740 1 -- start start 2024-04-01T01:35:24.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.966+0000 7fe3e787d740 1 Processor -- start 2024-04-01T01:35:24.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.966+0000 7fe3e787d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270a2b0 0x560e22718a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.966+0000 7fe3e787d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e22718fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.967+0000 7fe3de59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270a2b0 0x560e22718a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.967+0000 7fe3de59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270a2b0 0x560e22718a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:38026/0 (socket says 172.21.15.70:38026) 2024-04-01T01:35:24.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.967+0000 7fe3e787d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270fe10 0x560e2278d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.967+0000 7fe3e787d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e01adae0 con 0x560e2270aca0 2024-04-01T01:35:24.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.967+0000 7fe3e787d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e01ad410 con 0x560e2270fe10 2024-04-01T01:35:24.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.967+0000 7fe3e787d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e01aa650 con 0x560e2270a2b0 2024-04-01T01:35:24.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.967+0000 7fe3de59c640 1 -- 172.21.15.70:0/4125681624 learned_addr learned my addr 172.21.15.70:0/4125681624 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:24.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.967+0000 7fe3e787d740 10 client.0 _ll_register_callbacks cb 0x7fe3d0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:24.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.967+0000 7fe3ddd9b640 1 --2- 172.21.15.70:0/4125681624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e22718fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.967+0000 7fe3ded9d640 1 --2- 172.21.15.70:0/4125681624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270fe10 0x560e2278d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.968+0000 7fe3de59c640 1 -- 172.21.15.70:0/4125681624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270fe10 msgr2=0x560e2278d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:24.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.968+0000 7fe3de59c640 1 --2- 172.21.15.70:0/4125681624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270fe10 0x560e2278d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.968+0000 7fe3de59c640 1 -- 172.21.15.70:0/4125681624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 msgr2=0x560e22718fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:24.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.968+0000 7fe3de59c640 1 --2- 172.21.15.70:0/4125681624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e22718fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.968+0000 7fe3de59c640 1 -- 172.21.15.70:0/4125681624 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e2271c080 con 0x560e2270a2b0 2024-04-01T01:35:24.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.968+0000 7fe3de59c640 1 --2- 172.21.15.70:0/4125681624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270a2b0 0x560e22718a80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009900 tx=0x7fe3d0034ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa34086ff35f9328 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:24.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.968+0000 7fe3c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:35:24.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.969+0000 7fe3c5ffb640 1 -- 172.21.15.70:0/4125681624 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0062070 con 0x560e2270a2b0 2024-04-01T01:35:24.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.969+0000 7fe3c5ffb640 1 -- 172.21.15.70:0/4125681624 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe3d004a670 con 0x560e2270a2b0 2024-04-01T01:35:24.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.969+0000 7fe3e787d740 1 -- 172.21.15.70:0/4125681624 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560e22676a10 con 0x560e2270a2b0 2024-04-01T01:35:24.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.969+0000 7fe3c5ffb640 1 -- 172.21.15.70:0/4125681624 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe3d004e040 con 0x560e2270a2b0 2024-04-01T01:35:24.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.969+0000 7fe3e787d740 10 client.25969 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:24.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.969+0000 7fe3e787d740 1 -- 172.21.15.70:0/4125681624 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560e2278e7d0 con 0x560e2270a2b0 2024-04-01T01:35:24.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.970+0000 7fe3c5ffb640 1 -- 172.21.15.70:0/4125681624 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x560e22676a10 con 0x560e2270a2b0 2024-04-01T01:35:24.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.971+0000 7fe3c5ffb640 1 -- 172.21.15.70:0/4125681624 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe3d0003b50 con 0x560e2270a2b0 2024-04-01T01:35:24.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.971+0000 7fe3e787d740 10 client.25969 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:24.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.971+0000 7fe3e787d740 10 client.25969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:24.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.971+0000 7fe3e787d740 20 client.25969 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:24.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.971+0000 7fe3e787d740 1 --2- 172.21.15.70:0/4125681624 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560e2278eee0 0x560e22791300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:24.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.971+0000 7fe3e787d740 4 client.25969 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:35:24.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.971+0000 7fe3e787d740 1 -- 172.21.15.70:0/4125681624 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560e225f4540 con 0x560e2278eee0 2024-04-01T01:35:24.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.971+0000 7fe3ddd9b640 1 --2- 172.21.15.70:0/4125681624 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560e2278eee0 0x560e22791300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:24.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.972+0000 7fe3ddd9b640 1 --2- 172.21.15.70:0/4125681624 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560e2278eee0 0x560e22791300 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffa50500d746cda7 server_cookie=492e3251cb1b91b8 in_seq=0 out_seq=0 2024-04-01T01:35:24.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.972+0000 7fe3c5ffb640 10 client.25969 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:24.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.972+0000 7fe3c5ffb640 1 -- 172.21.15.70:0/4125681624 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560e225f4540 con 0x560e2278eee0 2024-04-01T01:35:24.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.972+0000 7fe3c5ffb640 10 client.25969 handle_command_reply: tid=0 2024-04-01T01:35:24.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.973+0000 7fe3e787d740 1 client.25969 shutdown 2024-04-01T01:35:24.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.973+0000 7fe3e787d740 1 -- 172.21.15.70:0/4125681624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270a2b0 msgr2=0x560e22718a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.973+0000 7fe3e787d740 1 --2- 172.21.15.70:0/4125681624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270a2b0 0x560e22718a80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009900 tx=0x7fe3d0034ac0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.973+0000 7fe3e787d740 1 -- 172.21.15.70:0/4125681624 shutdown_connections 2024-04-01T01:35:24.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.973+0000 7fe3e787d740 1 --2- 172.21.15.70:0/4125681624 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560e2278eee0 0x560e22791300 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.973+0000 7fe3e787d740 1 --2- 172.21.15.70:0/4125681624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e2270fe10 0x560e2278d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.973+0000 7fe3e787d740 1 --2- 172.21.15.70:0/4125681624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e2270aca0 0x560e22718fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.973+0000 7fe3e787d740 1 --2- 172.21.15.70:0/4125681624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2270a2b0 0x560e22718a80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.973+0000 7fe3e787d740 1 -- 172.21.15.70:0/4125681624 >> 172.21.15.70:0/4125681624 conn(0x560e22705820 msgr2=0x560e22704390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:24.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.973+0000 7fe3c5ffb640 0 client.25969 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:24.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.973+0000 7fe3e787d740 1 -- 172.21.15.70:0/4125681624 shutdown_connections 2024-04-01T01:35:24.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.974+0000 7fe3e787d740 1 -- 172.21.15.70:0/4125681624 wait complete. 2024-04-01T01:35:24.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.974+0000 7fe3e787d740 20 client.25969 trim_cache size 0 max 16384 2024-04-01T01:35:24.974 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:35:24.974 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:35:24.974 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:35:24.974 INFO:teuthology.orchestra.run.smithi070.stdout: "ba8b3a7e-cda6-4ff4-b01f-793d064d40a9": { 2024-04-01T01:35:24.974 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:35:24.974 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "ba8b3a7e-cda6-4ff4-b01f-793d064d40a9", 2024-04-01T01:35:24.974 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:35:24.975 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:35:24.975 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:35:24.975 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:35:24.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.975+0000 7fe3e4c9c640 1 -- 172.21.15.70:0/3430963340 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe3c80a2570 msgr2=0x7fe3c80a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.975+0000 7fe3e4c9c640 1 --2- 172.21.15.70:0/3430963340 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe3c80a2570 0x7fe3c80a4a30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x560e2270b0a0 tx=0x7fe3d4004730 comp rx=0 tx=0).stop 2024-04-01T01:35:24.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.975+0000 7fe3e4c9c640 1 -- 172.21.15.70:0/3430963340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3e0102200 msgr2=0x7fe3e01a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:24.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.975+0000 7fe3e4c9c640 1 --2- 172.21.15.70:0/3430963340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3e0102200 0x7fe3e01a1170 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe3d800cad0 tx=0x7fe3d800cf90 comp rx=0 tx=0).stop 2024-04-01T01:35:24.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.975+0000 7fe3e4c9c640 1 -- 172.21.15.70:0/3430963340 shutdown_connections 2024-04-01T01:35:24.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.975+0000 7fe3e4c9c640 1 --2- 172.21.15.70:0/3430963340 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3e0102200 0x7fe3e01a1170 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.975+0000 7fe3e4c9c640 1 --2- 172.21.15.70:0/3430963340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e0101840 0x7fe3e019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.975+0000 7fe3e4c9c640 1 --2- 172.21.15.70:0/3430963340 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe3c80a2570 0x7fe3c80a4a30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.975+0000 7fe3e4c9c640 1 --2- 172.21.15.70:0/3430963340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3e0100650 0x7fe3e0199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:24.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.975+0000 7fe3e4c9c640 1 -- 172.21.15.70:0/3430963340 >> 172.21.15.70:0/3430963340 conn(0x7fe3e00fbbe0 msgr2=0x7fe3e00fc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:24.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.975+0000 7fe3e4c9c640 1 -- 172.21.15.70:0/3430963340 shutdown_connections 2024-04-01T01:35:24.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:24.975+0000 7fe3e4c9c640 1 -- 172.21.15.70:0/3430963340 wait complete. 2024-04-01T01:35:24.997 INFO:tasks.cephfs.filesystem:scrub status for tag:ba8b3a7e-cda6-4ff4-b01f-793d064d40a9 - {'path': '/', 'tag': 'ba8b3a7e-cda6-4ff4-b01f-793d064d40a9', 'options': 'recursive,force'} 2024-04-01T01:35:25.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:25 smithi189 ceph-mon[35753]: pgmap v416: 129 pgs: 129 active+clean; 943 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 12 MiB/s wr, 1.24k op/s 2024-04-01T01:35:25.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:25 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:35:25.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:25 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:35:25.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:25 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:35:25.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:25 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:35:26.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:25 smithi070 ceph-mon[32179]: pgmap v416: 129 pgs: 129 active+clean; 943 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 12 MiB/s wr, 1.24k op/s 2024-04-01T01:35:26.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:25 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:35:26.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:25 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:35:26.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:25 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:35:26.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:25 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:35:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:25 smithi081 ceph-mon[35142]: pgmap v416: 129 pgs: 129 active+clean; 943 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 12 MiB/s wr, 1.24k op/s 2024-04-01T01:35:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:25 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:35:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:25 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:35:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:25 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:35:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:25 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:35:26.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 40.0 s, 1160.1 tps, lat 74.085 ms stddev 25.715 2024-04-01T01:35:26.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:26 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:35:27.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:26 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:35:27.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:26 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:35:27.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:27 smithi189 ceph-mon[35753]: pgmap v417: 129 pgs: 129 active+clean; 944 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 14 MiB/s wr, 1.55k op/s 2024-04-01T01:35:28.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:27 smithi070 ceph-mon[32179]: pgmap v417: 129 pgs: 129 active+clean; 944 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 14 MiB/s wr, 1.55k op/s 2024-04-01T01:35:28.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:27 smithi081 ceph-mon[35142]: pgmap v417: 129 pgs: 129 active+clean; 944 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 14 MiB/s wr, 1.55k op/s 2024-04-01T01:35:29.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:28 smithi070 ceph-mon[32179]: pgmap v418: 129 pgs: 129 active+clean; 944 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 1.43k op/s 2024-04-01T01:35:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:28 smithi081 ceph-mon[35142]: pgmap v418: 129 pgs: 129 active+clean; 944 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 1.43k op/s 2024-04-01T01:35:29.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:28 smithi189 ceph-mon[35753]: pgmap v418: 129 pgs: 129 active+clean; 944 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 1.43k op/s 2024-04-01T01:35:30.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.909+0000 7f39e90df640 1 -- 172.21.15.70:0/1111279914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39e410b380 msgr2=0x7f39e410d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:30.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.909+0000 7f39e90df640 1 --2- 172.21.15.70:0/1111279914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39e410b380 0x7f39e410d790 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f39d40074e0 tx=0x7f39d402c8d0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.910+0000 7f39e90df640 1 -- 172.21.15.70:0/1111279914 shutdown_connections 2024-04-01T01:35:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.910+0000 7f39e90df640 1 --2- 172.21.15.70:0/1111279914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39e410de20 0x7f39e41102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.910+0000 7f39e90df640 1 --2- 172.21.15.70:0/1111279914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39e410b380 0x7f39e410d790 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.910+0000 7f39e90df640 1 --2- 172.21.15.70:0/1111279914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39e41089c0 0x7f39e410adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.910+0000 7f39e90df640 1 -- 172.21.15.70:0/1111279914 >> 172.21.15.70:0/1111279914 conn(0x7f39e406e790 msgr2=0x7f39e4070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.910+0000 7f39e90df640 1 -- 172.21.15.70:0/1111279914 shutdown_connections 2024-04-01T01:35:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.910+0000 7f39e90df640 1 -- 172.21.15.70:0/1111279914 wait complete. 2024-04-01T01:35:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.911+0000 7f39e90df640 1 Processor -- start 2024-04-01T01:35:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.911+0000 7f39e90df640 1 -- start start 2024-04-01T01:35:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.911+0000 7f39e90df640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39e41089c0 0x7f39e41a00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.911+0000 7f39e2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39e41089c0 0x7f39e41a00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:30.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.912+0000 7f39e2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39e41089c0 0x7f39e41a00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49182/0 (socket says 172.21.15.70:49182) 2024-04-01T01:35:30.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.912+0000 7f39e90df640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39e410b380 0x7f39e41a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.912+0000 7f39e2d9d640 1 -- 172.21.15.70:0/1382914254 learned_addr learned my addr 172.21.15.70:0/1382914254 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:30.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.912+0000 7f39e90df640 1 --2- 172.21.15.70:0/1382914254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39e410de20 0x7f39e41a7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.912+0000 7f39e90df640 1 -- 172.21.15.70:0/1382914254 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39e4113820 con 0x7f39e410de20 2024-04-01T01:35:30.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.912+0000 7f39e90df640 1 -- 172.21.15.70:0/1382914254 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f39e41136a0 con 0x7f39e41089c0 2024-04-01T01:35:30.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.912+0000 7f39e90df640 1 -- 172.21.15.70:0/1382914254 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f39e41139a0 con 0x7f39e410b380 2024-04-01T01:35:30.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.912+0000 7f39e359e640 1 --2- 172.21.15.70:0/1382914254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39e410de20 0x7f39e41a7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:30.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.912+0000 7f39e259c640 1 --2- 172.21.15.70:0/1382914254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39e410b380 0x7f39e41a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.912+0000 7f39e2d9d640 1 -- 172.21.15.70:0/1382914254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39e410b380 msgr2=0x7f39e41a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.913+0000 7f39e2d9d640 1 --2- 172.21.15.70:0/1382914254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39e410b380 0x7f39e41a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.913+0000 7f39e2d9d640 1 -- 172.21.15.70:0/1382914254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39e410de20 msgr2=0x7f39e41a7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.913+0000 7f39e2d9d640 1 --2- 172.21.15.70:0/1382914254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39e410de20 0x7f39e41a7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.913+0000 7f39e2d9d640 1 -- 172.21.15.70:0/1382914254 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e41a7cc0 con 0x7f39e41089c0 2024-04-01T01:35:30.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.913+0000 7f39e2d9d640 1 --2- 172.21.15.70:0/1382914254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39e41089c0 0x7f39e41a00a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00ebc0 tx=0x7f39dc00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.913+0000 7f39cbfff640 1 -- 172.21.15.70:0/1382914254 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39dc010630 con 0x7f39e41089c0 2024-04-01T01:35:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.913+0000 7f39e90df640 1 -- 172.21.15.70:0/1382914254 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39e4108e00 con 0x7f39e41089c0 2024-04-01T01:35:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.913+0000 7f39cbfff640 1 -- 172.21.15.70:0/1382914254 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39dc010c80 con 0x7f39e41089c0 2024-04-01T01:35:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.913+0000 7f39cbfff640 1 -- 172.21.15.70:0/1382914254 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39dc018b30 con 0x7f39e41089c0 2024-04-01T01:35:30.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.914+0000 7f39e90df640 1 -- 172.21.15.70:0/1382914254 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39e41092e0 con 0x7f39e41089c0 2024-04-01T01:35:30.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.916+0000 7f39cbfff640 1 -- 172.21.15.70:0/1382914254 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f39dc018cd0 con 0x7f39e41089c0 2024-04-01T01:35:30.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.917+0000 7f39cbfff640 1 --2- 172.21.15.70:0/1382914254 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39cc0a2640 0x7f39cc0a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.917+0000 7f39cbfff640 1 -- 172.21.15.70:0/1382914254 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f39dc0cf140 con 0x7f39e41089c0 2024-04-01T01:35:30.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.918+0000 7f39ebd9d740 1 Processor -- start 2024-04-01T01:35:30.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.918+0000 7f39ebd9d740 1 -- start start 2024-04-01T01:35:30.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.918+0000 7f39e259c640 1 --2- 172.21.15.70:0/1382914254 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39cc0a2640 0x7f39cc0a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:30.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.918+0000 7f39ebd9d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb3c0e0 0x56112eb3c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.918+0000 7f39ebd9d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb415d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.918+0000 7f39ebd9d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb41c40 0x56112eb440d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.918+0000 7f39ebd9d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56112ea978f0 con 0x56112eb3c0e0 2024-04-01T01:35:30.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.918+0000 7f39ebd9d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56112ea408d0 con 0x56112eb41c40 2024-04-01T01:35:30.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.918+0000 7f39ebd9d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56112ea93990 con 0x56112eb3cad0 2024-04-01T01:35:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.919+0000 7f39e2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb3c0e0 0x56112eb3c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.919+0000 7f39e2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb3c0e0 0x56112eb3c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41802/0 (socket says 172.21.15.70:41802) 2024-04-01T01:35:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.919+0000 7f39e2d9d640 1 -- 172.21.15.70:0/2304502204 learned_addr learned my addr 172.21.15.70:0/2304502204 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.919+0000 7f39e259c640 1 --2- 172.21.15.70:0/2304502204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:30.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.919+0000 7f39e359e640 1 --2- 172.21.15.70:0/2304502204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb41c40 0x56112eb440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.919+0000 7f39e259c640 1 -- 172.21.15.70:0/2304502204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb41c40 msgr2=0x56112eb440d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.919+0000 7f39e259c640 1 --2- 172.21.15.70:0/2304502204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb41c40 0x56112eb440d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.919+0000 7f39e259c640 1 -- 172.21.15.70:0/2304502204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb3c0e0 msgr2=0x56112eb3c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.919+0000 7f39e259c640 1 --2- 172.21.15.70:0/2304502204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb3c0e0 0x56112eb3c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.919+0000 7f39e259c640 1 -- 172.21.15.70:0/2304502204 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56112e7ac210 con 0x56112eb3cad0 2024-04-01T01:35:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.919+0000 7f39e359e640 1 --2- 172.21.15.70:0/2304502204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb41c40 0x56112eb440d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:35:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.920+0000 7f39e259c640 1 --2- 172.21.15.70:0/2304502204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb415d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f39d402ce10 tx=0x7f39d4047000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9bd7dd3682d0979 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.920+0000 7f39e259c640 1 --2- 172.21.15.70:0/1382914254 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39cc0a2640 0x7f39cc0a4b00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f39d4006010 tx=0x7f39d40047a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.920+0000 7f39c97fa640 1 -- 172.21.15.70:0/2304502204 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d40044d0 con 0x56112eb3cad0 2024-04-01T01:35:30.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.920+0000 7f39c97fa640 1 -- 172.21.15.70:0/2304502204 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39d4005480 con 0x56112eb3cad0 2024-04-01T01:35:30.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.921+0000 7f39c97fa640 1 -- 172.21.15.70:0/2304502204 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d400a970 con 0x56112eb3cad0 2024-04-01T01:35:30.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.921+0000 7f39ebd9d740 1 -- 172.21.15.70:0/2304502204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 msgr2=0x56112eb415d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:30.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.921+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/2304502204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb415d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f39d402ce10 tx=0x7f39d4047000 comp rx=0 tx=0).stop 2024-04-01T01:35:30.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.921+0000 7f39ebd9d740 1 -- 172.21.15.70:0/2304502204 shutdown_connections 2024-04-01T01:35:30.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.921+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/2304502204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb41c40 0x56112eb440d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.921+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/2304502204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb415d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.921+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/2304502204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb3c0e0 0x56112eb3c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.921+0000 7f39ebd9d740 1 -- 172.21.15.70:0/2304502204 >> 172.21.15.70:0/2304502204 conn(0x56112eb2ee40 msgr2=0x56112eb2f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:30.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.922+0000 7f39ebd9d740 1 -- 172.21.15.70:0/2304502204 shutdown_connections 2024-04-01T01:35:30.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.922+0000 7f39ebd9d740 1 -- 172.21.15.70:0/2304502204 wait complete. 2024-04-01T01:35:30.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.922+0000 7f39ebd9d740 1 -- start start 2024-04-01T01:35:30.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.922+0000 7f39ebd9d740 1 Processor -- start 2024-04-01T01:35:30.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.923+0000 7f39ebd9d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb3c0e0 0x56112eb462f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.923+0000 7f39ebd9d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb46830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.923+0000 7f39ebd9d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb41c40 0x56112eb4d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.923+0000 7f39ebd9d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56112eb44a60 con 0x56112eb41c40 2024-04-01T01:35:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.923+0000 7f39ebd9d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56112ea35f80 con 0x56112eb3c0e0 2024-04-01T01:35:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.923+0000 7f39ebd9d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56112eb44be0 con 0x56112eb3cad0 2024-04-01T01:35:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.923+0000 7f39ebd9d740 10 client.0 _ll_register_callbacks cb 0x56112ea8e290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.923+0000 7f39e359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb41c40 0x56112eb4d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.923+0000 7f39e359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb41c40 0x56112eb4d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41818/0 (socket says 172.21.15.70:41818) 2024-04-01T01:35:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.923+0000 7f39e359e640 1 -- 172.21.15.70:0/184185871 learned_addr learned my addr 172.21.15.70:0/184185871 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.923+0000 7f39e259c640 1 --2- 172.21.15.70:0/184185871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb46830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.923+0000 7f39e2d9d640 1 --2- 172.21.15.70:0/184185871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb3c0e0 0x56112eb462f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.924+0000 7f39e259c640 1 -- 172.21.15.70:0/184185871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb3c0e0 msgr2=0x56112eb462f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.924+0000 7f39e259c640 1 --2- 172.21.15.70:0/184185871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb3c0e0 0x56112eb462f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.924+0000 7f39e259c640 1 -- 172.21.15.70:0/184185871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb41c40 msgr2=0x56112eb4d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.924+0000 7f39e259c640 1 --2- 172.21.15.70:0/184185871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb41c40 0x56112eb4d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.924+0000 7f39e259c640 1 -- 172.21.15.70:0/184185871 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56112e878620 con 0x56112eb3cad0 2024-04-01T01:35:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.924+0000 7f39e359e640 1 --2- 172.21.15.70:0/184185871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb41c40 0x56112eb4d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.925+0000 7f39e259c640 1 --2- 172.21.15.70:0/184185871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb46830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f39d4042040 tx=0x7f39d4005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49f4cab6247f0d11 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.925+0000 7f39c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:35:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.926+0000 7f39c6ff5640 1 -- 172.21.15.70:0/184185871 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d4050050 con 0x56112eb3cad0 2024-04-01T01:35:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.926+0000 7f39ebd9d740 1 -- 172.21.15.70:0/184185871 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39d4005480 con 0x56112eb3cad0 2024-04-01T01:35:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.926+0000 7f39c6ff5640 1 -- 172.21.15.70:0/184185871 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f39d40051e0 con 0x56112eb3cad0 2024-04-01T01:35:30.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.926+0000 7f39c6ff5640 1 -- 172.21.15.70:0/184185871 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f39d405e050 con 0x56112eb3cad0 2024-04-01T01:35:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.927+0000 7f39ebd9d740 10 client.25981 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.927+0000 7f39ebd9d740 1 -- 172.21.15.70:0/184185871 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56112eb44a60 con 0x56112eb3cad0 2024-04-01T01:35:30.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.928+0000 7f39c6ff5640 1 -- 172.21.15.70:0/184185871 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39d4005480 con 0x56112eb3cad0 2024-04-01T01:35:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.929+0000 7f39c6ff5640 1 -- 172.21.15.70:0/184185871 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f39d4008050 con 0x56112eb3cad0 2024-04-01T01:35:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.929+0000 7f39ebd9d740 10 client.25981 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.929+0000 7f39ebd9d740 10 client.25981 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.929+0000 7f39ebd9d740 20 client.25981 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.929+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/184185871 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56112eb373c0 0x56112eb397b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.929+0000 7f39ebd9d740 4 client.25981 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.929+0000 7f39ebd9d740 1 -- 172.21.15.70:0/184185871 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56112eaab120 con 0x56112eb373c0 2024-04-01T01:35:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.930+0000 7f39e2d9d640 1 --2- 172.21.15.70:0/184185871 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56112eb373c0 0x56112eb397b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.931+0000 7f39e2d9d640 1 --2- 172.21.15.70:0/184185871 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56112eb373c0 0x56112eb397b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c58ce87235b0e214 server_cookie=d2f4eb5dcf408c0d in_seq=0 out_seq=0 2024-04-01T01:35:30.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.931+0000 7f39c6ff5640 10 client.25981 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:30.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.933+0000 7f39c6ff5640 1 -- 172.21.15.70:0/184185871 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56112eaab120 con 0x56112eb373c0 2024-04-01T01:35:30.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.933+0000 7f39c6ff5640 10 client.25981 handle_command_reply: tid=0 2024-04-01T01:35:30.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.933+0000 7f39ebd9d740 1 client.25981 shutdown 2024-04-01T01:35:30.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.934+0000 7f39ebd9d740 1 -- 172.21.15.70:0/184185871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 msgr2=0x56112eb46830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:30.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.934+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/184185871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb46830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f39d4042040 tx=0x7f39d4005d50 comp rx=0 tx=0).stop 2024-04-01T01:35:30.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.934+0000 7f39ebd9d740 1 -- 172.21.15.70:0/184185871 shutdown_connections 2024-04-01T01:35:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.934+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/184185871 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56112eb373c0 0x56112eb397b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.934+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/184185871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb41c40 0x56112eb4d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.934+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/184185871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb46830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.934+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/184185871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb3c0e0 0x56112eb462f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.934+0000 7f39ebd9d740 1 -- 172.21.15.70:0/184185871 >> 172.21.15.70:0/184185871 conn(0x56112eb2ee40 msgr2=0x56112eb2df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.934+0000 7f39c6ff5640 0 client.25981 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.935+0000 7f39ebd9d740 1 -- 172.21.15.70:0/184185871 shutdown_connections 2024-04-01T01:35:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.935+0000 7f39ebd9d740 1 -- 172.21.15.70:0/184185871 wait complete. 2024-04-01T01:35:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.935+0000 7f39ebd9d740 20 client.25981 trim_cache size 0 max 16384 2024-04-01T01:35:30.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.966+0000 7f39ebd9d740 1 Processor -- start 2024-04-01T01:35:30.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.967+0000 7f39ebd9d740 1 -- start start 2024-04-01T01:35:30.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.967+0000 7f39ebd9d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb3c0e0 0x56112eb45910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.967+0000 7f39ebd9d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb4d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.967+0000 7f39ebd9d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb41c40 0x56112eb462f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.967+0000 7f39ebd9d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39e41b0990 con 0x56112eb41c40 2024-04-01T01:35:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.967+0000 7f39ebd9d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f39e41b24d0 con 0x56112eb3c0e0 2024-04-01T01:35:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.967+0000 7f39ebd9d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f39e41b1060 con 0x56112eb3cad0 2024-04-01T01:35:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.967+0000 7f39e2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb3c0e0 0x56112eb45910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.967+0000 7f39e2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb3c0e0 0x56112eb45910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49220/0 (socket says 172.21.15.70:49220) 2024-04-01T01:35:30.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.967+0000 7f39e2d9d640 1 -- 172.21.15.70:0/3913259200 learned_addr learned my addr 172.21.15.70:0/3913259200 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.967+0000 7f39e259c640 1 --2- 172.21.15.70:0/3913259200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb4d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.968+0000 7f39e2d9d640 1 -- 172.21.15.70:0/3913259200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 msgr2=0x56112eb4d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.968+0000 7f39e2d9d640 1 --2- 172.21.15.70:0/3913259200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb4d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.968+0000 7f39e2d9d640 1 -- 172.21.15.70:0/3913259200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb41c40 msgr2=0x56112eb462f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.968+0000 7f39e2d9d640 1 --2- 172.21.15.70:0/3913259200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb41c40 0x56112eb462f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.968+0000 7f39e2d9d640 1 -- 172.21.15.70:0/3913259200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56112eb3ceb0 con 0x56112eb3c0e0 2024-04-01T01:35:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.968+0000 7f39e2d9d640 1 --2- 172.21.15.70:0/3913259200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb3c0e0 0x56112eb45910 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00eb90 tx=0x7f39dc086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76165a12928f4238 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.968+0000 7f39c8ff9640 1 -- 172.21.15.70:0/3913259200 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39dc0b9070 con 0x56112eb3c0e0 2024-04-01T01:35:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.968+0000 7f39c8ff9640 1 -- 172.21.15.70:0/3913259200 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39dc09f920 con 0x56112eb3c0e0 2024-04-01T01:35:30.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.968+0000 7f39c8ff9640 1 -- 172.21.15.70:0/3913259200 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39dc0b3030 con 0x56112eb3c0e0 2024-04-01T01:35:30.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.970+0000 7f39ebd9d740 1 -- 172.21.15.70:0/3913259200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb3c0e0 msgr2=0x56112eb45910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:30.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.970+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/3913259200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb3c0e0 0x56112eb45910 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00eb90 tx=0x7f39dc086040 comp rx=0 tx=0).stop 2024-04-01T01:35:30.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.972+0000 7f39ebd9d740 1 -- 172.21.15.70:0/3913259200 shutdown_connections 2024-04-01T01:35:30.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.972+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/3913259200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb41c40 0x56112eb462f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.972+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/3913259200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb4d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.972+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/3913259200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb3c0e0 0x56112eb45910 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.972+0000 7f39ebd9d740 1 -- 172.21.15.70:0/3913259200 >> 172.21.15.70:0/3913259200 conn(0x56112eb2ee40 msgr2=0x56112eb2f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:30.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.973+0000 7f39ebd9d740 1 -- 172.21.15.70:0/3913259200 shutdown_connections 2024-04-01T01:35:30.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.973+0000 7f39ebd9d740 1 -- 172.21.15.70:0/3913259200 wait complete. 2024-04-01T01:35:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.974+0000 7f39ebd9d740 1 -- start start 2024-04-01T01:35:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.974+0000 7f39ebd9d740 1 Processor -- start 2024-04-01T01:35:30.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.974+0000 7f39ebd9d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb3c0e0 0x56112eb48130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.974+0000 7f39ebd9d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb4d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.974+0000 7f39ebd9d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb41c40 0x56112eb48670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.975+0000 7f39ebd9d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39e41b1e00 con 0x56112eb3c0e0 2024-04-01T01:35:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.975+0000 7f39e359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb41c40 0x56112eb48670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.975+0000 7f39e359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb41c40 0x56112eb48670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49226/0 (socket says 172.21.15.70:49226) 2024-04-01T01:35:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.975+0000 7f39e359e640 1 -- 172.21.15.70:0/2967526064 learned_addr learned my addr 172.21.15.70:0/2967526064 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.975+0000 7f39ebd9d740 1 -- 172.21.15.70:0/2967526064 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f39e41b1730 con 0x56112eb41c40 2024-04-01T01:35:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.975+0000 7f39ebd9d740 1 -- 172.21.15.70:0/2967526064 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f39e41ae970 con 0x56112eb3cad0 2024-04-01T01:35:30.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.975+0000 7f39e259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb4d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.975+0000 7f39ebd9d740 10 client.0 _ll_register_callbacks cb 0x7f39d4042070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.975+0000 7f39e259c640 1 -- 172.21.15.70:0/2967526064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb41c40 msgr2=0x56112eb48670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.975+0000 7f39e259c640 1 --2- 172.21.15.70:0/2967526064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb41c40 0x56112eb48670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.975+0000 7f39e259c640 1 -- 172.21.15.70:0/2967526064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb3c0e0 msgr2=0x56112eb48130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:35:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.975+0000 7f39e259c640 1 --2- 172.21.15.70:0/2967526064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb3c0e0 0x56112eb48130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.975+0000 7f39e259c640 1 -- 172.21.15.70:0/2967526064 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56112ebbe400 con 0x56112eb3cad0 2024-04-01T01:35:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.976+0000 7f39e259c640 1 --2- 172.21.15.70:0/2967526064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb4d6a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f39d4004830 tx=0x7f39d400d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b2dd0600ce12258 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.976+0000 7f39c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:35:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.976+0000 7f39c6ff5640 1 -- 172.21.15.70:0/2967526064 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d40051e0 con 0x56112eb3cad0 2024-04-01T01:35:30.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.976+0000 7f39c6ff5640 1 -- 172.21.15.70:0/2967526064 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f39d4005380 con 0x56112eb3cad0 2024-04-01T01:35:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.976+0000 7f39ebd9d740 1 -- 172.21.15.70:0/2967526064 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39dc09f920 con 0x56112eb3cad0 2024-04-01T01:35:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.977+0000 7f39c6ff5640 1 -- 172.21.15.70:0/2967526064 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f39d4008590 con 0x56112eb3cad0 2024-04-01T01:35:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.977+0000 7f39ebd9d740 10 client.25987 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.977+0000 7f39ebd9d740 1 -- 172.21.15.70:0/2967526064 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56112ebbf620 con 0x56112eb3cad0 2024-04-01T01:35:30.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.977+0000 7f39c6ff5640 1 -- 172.21.15.70:0/2967526064 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39dc09f920 con 0x56112eb3cad0 2024-04-01T01:35:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.977+0000 7f39c6ff5640 1 -- 172.21.15.70:0/2967526064 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f39d40035f0 con 0x56112eb3cad0 2024-04-01T01:35:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.977+0000 7f39ebd9d740 10 client.25987 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.977+0000 7f39ebd9d740 10 client.25987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.978+0000 7f39ebd9d740 20 client.25987 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.978+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/2967526064 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56112ebbfe70 0x56112ebc2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.978+0000 7f39ebd9d740 4 client.25987 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:35:30.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.978+0000 7f39ebd9d740 1 -- 172.21.15.70:0/2967526064 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56112eaa4e30 con 0x56112ebbfe70 2024-04-01T01:35:30.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.978+0000 7f39e2d9d640 1 --2- 172.21.15.70:0/2967526064 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56112ebbfe70 0x56112ebc2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.982+0000 7f39e2d9d640 1 --2- 172.21.15.70:0/2967526064 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56112ebbfe70 0x56112ebc2290 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbb5aec0baf51b1e server_cookie=e2ed72769a3974a9 in_seq=0 out_seq=0 2024-04-01T01:35:30.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:30.982+0000 7f39c6ff5640 10 client.25987 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:31.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.008+0000 7f39c6ff5640 1 -- 172.21.15.70:0/2967526064 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56112eaa4e30 con 0x56112ebbfe70 2024-04-01T01:35:31.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.008+0000 7f39c6ff5640 10 client.25987 handle_command_reply: tid=0 2024-04-01T01:35:31.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.008+0000 7f39ebd9d740 1 client.25987 shutdown 2024-04-01T01:35:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.009+0000 7f39ebd9d740 1 -- 172.21.15.70:0/2967526064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 msgr2=0x56112eb4d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.009+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/2967526064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb4d6a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f39d4004830 tx=0x7f39d400d000 comp rx=0 tx=0).stop 2024-04-01T01:35:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.009+0000 7f39ebd9d740 1 -- 172.21.15.70:0/2967526064 shutdown_connections 2024-04-01T01:35:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.009+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/2967526064 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56112ebbfe70 0x56112ebc2290 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.009+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/2967526064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56112eb41c40 0x56112eb48670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.009+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/2967526064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56112eb3cad0 0x56112eb4d6a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.009+0000 7f39ebd9d740 1 --2- 172.21.15.70:0/2967526064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56112eb3c0e0 0x56112eb48130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.009+0000 7f39ebd9d740 1 -- 172.21.15.70:0/2967526064 >> 172.21.15.70:0/2967526064 conn(0x56112eb2ee40 msgr2=0x56112eb3dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.009+0000 7f39c6ff5640 0 client.25987 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.009+0000 7f39ebd9d740 1 -- 172.21.15.70:0/2967526064 shutdown_connections 2024-04-01T01:35:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.009+0000 7f39ebd9d740 1 -- 172.21.15.70:0/2967526064 wait complete. 2024-04-01T01:35:31.009 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.009+0000 7f39ebd9d740 20 client.25987 trim_cache size 0 max 16384 2024-04-01T01:35:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:35:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:35:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:35:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:35:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.014+0000 7f39e90df640 1 -- 172.21.15.70:0/1382914254 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39cc0a2640 msgr2=0x7f39cc0a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.014+0000 7f39e90df640 1 --2- 172.21.15.70:0/1382914254 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39cc0a2640 0x7f39cc0a4b00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f39d4006010 tx=0x7f39d40047a0 comp rx=0 tx=0).stop 2024-04-01T01:35:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.014+0000 7f39e90df640 1 -- 172.21.15.70:0/1382914254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39e41089c0 msgr2=0x7f39e41a00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:31.013 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.014+0000 7f39e90df640 1 --2- 172.21.15.70:0/1382914254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39e41089c0 0x7f39e41a00a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00ebc0 tx=0x7f39dc00c6a0 comp rx=0 tx=0).stop 2024-04-01T01:35:31.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.015+0000 7f39e90df640 1 -- 172.21.15.70:0/1382914254 shutdown_connections 2024-04-01T01:35:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.015+0000 7f39e90df640 1 --2- 172.21.15.70:0/1382914254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39e410de20 0x7f39e41a7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.015+0000 7f39e90df640 1 --2- 172.21.15.70:0/1382914254 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39cc0a2640 0x7f39cc0a4b00 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.015+0000 7f39e90df640 1 --2- 172.21.15.70:0/1382914254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39e410b380 0x7f39e41a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.015+0000 7f39e90df640 1 --2- 172.21.15.70:0/1382914254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39e41089c0 0x7f39e41a00a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.015+0000 7f39e90df640 1 -- 172.21.15.70:0/1382914254 >> 172.21.15.70:0/1382914254 conn(0x7f39e406e790 msgr2=0x7f39e410e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.016+0000 7f39e90df640 1 -- 172.21.15.70:0/1382914254 shutdown_connections 2024-04-01T01:35:31.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:31.016+0000 7f39e90df640 1 -- 172.21.15.70:0/1382914254 wait complete. 2024-04-01T01:35:31.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 45.0 s, 1035.3 tps, lat 104.336 ms stddev 517.702 2024-04-01T01:35:31.949 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:31 smithi189 ceph-mon[35753]: pgmap v419: 129 pgs: 129 active+clean; 946 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 12 MiB/s wr, 1.44k op/s 2024-04-01T01:35:32.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:31 smithi070 ceph-mon[32179]: pgmap v419: 129 pgs: 129 active+clean; 946 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 12 MiB/s wr, 1.44k op/s 2024-04-01T01:35:32.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:31 smithi081 ceph-mon[35142]: pgmap v419: 129 pgs: 129 active+clean; 946 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 12 MiB/s wr, 1.44k op/s 2024-04-01T01:35:33.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:33 smithi189 ceph-mon[35753]: pgmap v420: 129 pgs: 129 active+clean; 949 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 13 MiB/s wr, 1.55k op/s 2024-04-01T01:35:34.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:33 smithi070 ceph-mon[32179]: pgmap v420: 129 pgs: 129 active+clean; 949 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 13 MiB/s wr, 1.55k op/s 2024-04-01T01:35:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:33 smithi081 ceph-mon[35142]: pgmap v420: 129 pgs: 129 active+clean; 949 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 13 MiB/s wr, 1.55k op/s 2024-04-01T01:35:35.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:35 smithi189 ceph-mon[35753]: pgmap v421: 129 pgs: 129 active+clean; 953 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 11 MiB/s wr, 1.35k op/s 2024-04-01T01:35:36.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:35 smithi070 ceph-mon[32179]: pgmap v421: 129 pgs: 129 active+clean; 953 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 11 MiB/s wr, 1.35k op/s 2024-04-01T01:35:36.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:35 smithi081 ceph-mon[35142]: pgmap v421: 129 pgs: 129 active+clean; 953 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 11 MiB/s wr, 1.35k op/s 2024-04-01T01:35:36.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 50.0 s, 1138.8 tps, lat 93.117 ms stddev 226.204 2024-04-01T01:35:37.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:37 smithi189 ceph-mon[35753]: pgmap v422: 129 pgs: 129 active+clean; 953 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 13 MiB/s wr, 1.56k op/s 2024-04-01T01:35:38.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:37 smithi070 ceph-mon[32179]: pgmap v422: 129 pgs: 129 active+clean; 953 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 13 MiB/s wr, 1.56k op/s 2024-04-01T01:35:38.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:37 smithi081 ceph-mon[35142]: pgmap v422: 129 pgs: 129 active+clean; 953 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 13 MiB/s wr, 1.56k op/s 2024-04-01T01:35:39.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:39 smithi189 ceph-mon[35753]: pgmap v423: 129 pgs: 129 active+clean; 953 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-04-01T01:35:40.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:39 smithi070 ceph-mon[32179]: pgmap v423: 129 pgs: 129 active+clean; 953 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-04-01T01:35:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:39 smithi081 ceph-mon[35142]: pgmap v423: 129 pgs: 129 active+clean; 953 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 12 MiB/s wr, 1.39k op/s 2024-04-01T01:35:41.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 55.0 s, 1095.6 tps, lat 91.355 ms stddev 89.931 2024-04-01T01:35:41.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:41 smithi189 ceph-mon[35753]: pgmap v424: 129 pgs: 129 active+clean; 953 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-04-01T01:35:42.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:41 smithi070 ceph-mon[32179]: pgmap v424: 129 pgs: 129 active+clean; 953 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-04-01T01:35:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:41 smithi081 ceph-mon[35142]: pgmap v424: 129 pgs: 129 active+clean; 953 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-04-01T01:35:43.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:43 smithi189 ceph-mon[35753]: pgmap v425: 129 pgs: 129 active+clean; 961 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 13 MiB/s wr, 1.38k op/s 2024-04-01T01:35:44.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:43 smithi070 ceph-mon[32179]: pgmap v425: 129 pgs: 129 active+clean; 961 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 13 MiB/s wr, 1.38k op/s 2024-04-01T01:35:44.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:43 smithi081 ceph-mon[35142]: pgmap v425: 129 pgs: 129 active+clean; 961 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 13 MiB/s wr, 1.38k op/s 2024-04-01T01:35:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.171+0000 7fb8c6e72640 1 -- 172.21.15.70:0/1126939089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8c0101860 msgr2=0x7fb8c0101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.171+0000 7fb8c6e72640 1 --2- 172.21.15.70:0/1126939089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8c0101860 0x7fb8c0101ce0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009860 tx=0x7fb8b402bb90 comp rx=0 tx=0).stop 2024-04-01T01:35:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.171+0000 7fb8c6e72640 1 -- 172.21.15.70:0/1126939089 shutdown_connections 2024-04-01T01:35:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.171+0000 7fb8c6e72640 1 --2- 172.21.15.70:0/1126939089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8c0102220 0x7fb8c0108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.171+0000 7fb8c6e72640 1 --2- 172.21.15.70:0/1126939089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8c0101860 0x7fb8c0101ce0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.171+0000 7fb8c6e72640 1 --2- 172.21.15.70:0/1126939089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8c0100670 0x7fb8c0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.171+0000 7fb8c6e72640 1 -- 172.21.15.70:0/1126939089 >> 172.21.15.70:0/1126939089 conn(0x7fb8c00fbbe0 msgr2=0x7fb8c00fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.171+0000 7fb8c6e72640 1 -- 172.21.15.70:0/1126939089 shutdown_connections 2024-04-01T01:35:44.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.171+0000 7fb8c6e72640 1 -- 172.21.15.70:0/1126939089 wait complete. 2024-04-01T01:35:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.173+0000 7fb8c6e72640 1 Processor -- start 2024-04-01T01:35:44.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.173+0000 7fb8c6e72640 1 -- start start 2024-04-01T01:35:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.174+0000 7fb8c6e72640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8c0100670 0x7fb8c0199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.174+0000 7fb8c6e72640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8c0101860 0x7fb8c019a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.174+0000 7fb8c4c0e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8c0100670 0x7fb8c0199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.174+0000 7fb8c4c0e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8c0100670 0x7fb8c0199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:60450/0 (socket says 172.21.15.70:60450) 2024-04-01T01:35:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.174+0000 7fb8c6e72640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8c0102220 0x7fb8c01a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.174+0000 7fb8c6e72640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c010c190 con 0x7fb8c0102220 2024-04-01T01:35:44.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.174+0000 7fb8c6e72640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c010c010 con 0x7fb8c0101860 2024-04-01T01:35:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.174+0000 7fb8c6e72640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c010c310 con 0x7fb8c0100670 2024-04-01T01:35:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.175+0000 7fb8c4c0e640 1 -- 172.21.15.70:0/319513296 learned_addr learned my addr 172.21.15.70:0/319513296 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.175+0000 7fb8c540f640 1 --2- 172.21.15.70:0/319513296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8c0102220 0x7fb8c01a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:44.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.175+0000 7fb8bffff640 1 --2- 172.21.15.70:0/319513296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8c0101860 0x7fb8c019a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.175+0000 7fb8c4c0e640 1 -- 172.21.15.70:0/319513296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8c0101860 msgr2=0x7fb8c019a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.175+0000 7fb8c4c0e640 1 --2- 172.21.15.70:0/319513296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8c0101860 0x7fb8c019a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.175+0000 7fb8c4c0e640 1 -- 172.21.15.70:0/319513296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8c0102220 msgr2=0x7fb8c01a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.175+0000 7fb8c4c0e640 1 --2- 172.21.15.70:0/319513296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8c0102220 0x7fb8c01a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.175+0000 7fb8c4c0e640 1 -- 172.21.15.70:0/319513296 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8c01a1700 con 0x7fb8c0100670 2024-04-01T01:35:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.175+0000 7fb8c540f640 1 --2- 172.21.15.70:0/319513296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8c0102220 0x7fb8c01a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.176+0000 7fb8c4c0e640 1 --2- 172.21.15.70:0/319513296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8c0100670 0x7fb8c0199ae0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000c970 tx=0x7fb8b000ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.176+0000 7fb8bdffb640 1 -- 172.21.15.70:0/319513296 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0004220 con 0x7fb8c0100670 2024-04-01T01:35:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.176+0000 7fb8bdffb640 1 -- 172.21.15.70:0/319513296 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8b00043c0 con 0x7fb8c0100670 2024-04-01T01:35:44.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.176+0000 7fb8bdffb640 1 -- 172.21.15.70:0/319513296 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b000f450 con 0x7fb8c0100670 2024-04-01T01:35:44.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.177+0000 7fb8c6e72640 1 -- 172.21.15.70:0/319513296 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8c01a19f0 con 0x7fb8c0100670 2024-04-01T01:35:44.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.177+0000 7fb8c6e72640 1 -- 172.21.15.70:0/319513296 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8c01a1ec0 con 0x7fb8c0100670 2024-04-01T01:35:44.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.180+0000 7fb8c9a50740 1 Processor -- start 2024-04-01T01:35:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.181+0000 7fb8c9a50740 1 -- start start 2024-04-01T01:35:44.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.181+0000 7fb8c9a50740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef190e0 0x5634bef19500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.181+0000 7fb8c9a50740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef19ad0 0x5634bef1e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.181+0000 7fb8c9a50740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef1ec40 0x5634bef210d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.181+0000 7fb8c9a50740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5634bee748f0 con 0x5634bef190e0 2024-04-01T01:35:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.181+0000 7fb8c9a50740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5634bee1d8d0 con 0x5634bef19ad0 2024-04-01T01:35:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.181+0000 7fb8c9a50740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5634bee70990 con 0x5634bef1ec40 2024-04-01T01:35:44.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.182+0000 7fb8c540f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef190e0 0x5634bef19500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.182+0000 7fb8c540f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef190e0 0x5634bef19500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56726/0 (socket says 172.21.15.70:56726) 2024-04-01T01:35:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.182+0000 7fb8c540f640 1 -- 172.21.15.70:0/592969343 learned_addr learned my addr 172.21.15.70:0/592969343 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.182+0000 7fb8c540f640 1 -- 172.21.15.70:0/592969343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef1ec40 msgr2=0x5634bef210d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:35:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.182+0000 7fb8c540f640 1 --2- 172.21.15.70:0/592969343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef1ec40 0x5634bef210d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.182+0000 7fb8bdffb640 1 -- 172.21.15.70:0/319513296 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb8b00079b0 con 0x7fb8c0100670 2024-04-01T01:35:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.182+0000 7fb8c540f640 1 -- 172.21.15.70:0/592969343 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef19ad0 msgr2=0x5634bef1e5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:35:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.182+0000 7fb8c540f640 1 --2- 172.21.15.70:0/592969343 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef19ad0 0x5634bef1e5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.183+0000 7fb8c540f640 1 -- 172.21.15.70:0/592969343 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634beb89210 con 0x5634bef190e0 2024-04-01T01:35:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.183+0000 7fb8c540f640 1 --2- 172.21.15.70:0/592969343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef190e0 0x5634bef19500 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800e8f0 tx=0x7fb8b80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b846d4918354629 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:44.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.184+0000 7fb8a6ffd640 1 -- 172.21.15.70:0/592969343 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8017420 con 0x5634bef190e0 2024-04-01T01:35:44.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.184+0000 7fb8bdffb640 1 --2- 172.21.15.70:0/319513296 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8a80a25c0 0x7fb8a80a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.184 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.184+0000 7fb8a6ffd640 1 -- 172.21.15.70:0/592969343 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8b801c070 con 0x5634bef190e0 2024-04-01T01:35:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.184+0000 7fb8bdffb640 1 -- 172.21.15.70:0/319513296 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb8b00ce870 con 0x7fb8c0100670 2024-04-01T01:35:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.185+0000 7fb8c4c0e640 1 --2- 172.21.15.70:0/319513296 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8a80a25c0 0x7fb8a80a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.185+0000 7fb8a6ffd640 1 -- 172.21.15.70:0/592969343 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b80166f0 con 0x5634bef190e0 2024-04-01T01:35:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.185+0000 7fb8c9a50740 1 -- 172.21.15.70:0/592969343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef190e0 msgr2=0x5634bef19500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.185+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/592969343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef190e0 0x5634bef19500 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800e8f0 tx=0x7fb8b80095c0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.185+0000 7fb8c4c0e640 1 --2- 172.21.15.70:0/319513296 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8a80a25c0 0x7fb8a80a4a80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb8b00a33e0 tx=0x7fb8b000a5d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.186+0000 7fb8c9a50740 1 -- 172.21.15.70:0/592969343 shutdown_connections 2024-04-01T01:35:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.186+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/592969343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef1ec40 0x5634bef210d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.186+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/592969343 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef19ad0 0x5634bef1e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.186+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/592969343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef190e0 0x5634bef19500 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.186+0000 7fb8c9a50740 1 -- 172.21.15.70:0/592969343 >> 172.21.15.70:0/592969343 conn(0x5634bef0be40 msgr2=0x5634bef0c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.186+0000 7fb8c9a50740 1 -- 172.21.15.70:0/592969343 shutdown_connections 2024-04-01T01:35:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.186+0000 7fb8c9a50740 1 -- 172.21.15.70:0/592969343 wait complete. 2024-04-01T01:35:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.187+0000 7fb8c9a50740 1 -- start start 2024-04-01T01:35:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.187+0000 7fb8c9a50740 1 Processor -- start 2024-04-01T01:35:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.188+0000 7fb8c9a50740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef190e0 0x5634bef12290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.188+0000 7fb8c9a50740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef19ad0 0x5634bef127d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.188+0000 7fb8c540f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef190e0 0x5634bef12290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.188+0000 7fb8c540f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef190e0 0x5634bef12290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:39964/0 (socket says 172.21.15.70:39964) 2024-04-01T01:35:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.188+0000 7fb8c540f640 1 -- 172.21.15.70:0/2170780297 learned_addr learned my addr 172.21.15.70:0/2170780297 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.188+0000 7fb8bffff640 1 --2- 172.21.15.70:0/2170780297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef19ad0 0x5634bef127d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.188+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/2170780297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef1ec40 0x5634bef17800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.188+0000 7fb8c9a50740 1 -- 172.21.15.70:0/2170780297 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5634bef10a20 con 0x5634bef1ec40 2024-04-01T01:35:44.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.189+0000 7fb8c9a50740 1 -- 172.21.15.70:0/2170780297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5634bee12f80 con 0x5634bef190e0 2024-04-01T01:35:44.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.189+0000 7fb8c9a50740 1 -- 172.21.15.70:0/2170780297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5634bef10ba0 con 0x5634bef19ad0 2024-04-01T01:35:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.189+0000 7fb8c540f640 1 -- 172.21.15.70:0/2170780297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef19ad0 msgr2=0x5634bef127d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.189+0000 7fb8c540f640 1 --2- 172.21.15.70:0/2170780297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef19ad0 0x5634bef127d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.189+0000 7fb8c540f640 1 -- 172.21.15.70:0/2170780297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef1ec40 msgr2=0x5634bef17800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.189+0000 7fb8c540f640 1 --2- 172.21.15.70:0/2170780297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef1ec40 0x5634bef17800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.189+0000 7fb8c540f640 1 -- 172.21.15.70:0/2170780297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634bec55620 con 0x5634bef190e0 2024-04-01T01:35:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.190+0000 7fb8c540f640 1 --2- 172.21.15.70:0/2170780297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef190e0 0x5634bef12290 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800e000 tx=0x7fb8b8000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18d243c845778c92 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.190+0000 7fb8c9a50740 10 client.0 _ll_register_callbacks cb 0x5634bee6b290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.190+0000 7fb8c9a50740 1 -- 172.21.15.70:0/2170780297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8b801c070 con 0x5634bef190e0 2024-04-01T01:35:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.190+0000 7fb89ffff640 10 client.26087 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.190+0000 7fb89ffff640 1 -- 172.21.15.70:0/2170780297 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8029020 con 0x5634bef190e0 2024-04-01T01:35:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.190+0000 7fb89ffff640 1 -- 172.21.15.70:0/2170780297 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb8b8017830 con 0x5634bef190e0 2024-04-01T01:35:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.190+0000 7fb89ffff640 1 -- 172.21.15.70:0/2170780297 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb8b8034050 con 0x5634bef190e0 2024-04-01T01:35:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.191+0000 7fb8c9a50740 10 client.26087 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.191+0000 7fb8c9a50740 1 -- 172.21.15.70:0/2170780297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=29}) -- 0x5634bef35e60 con 0x5634bef190e0 2024-04-01T01:35:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.191+0000 7fb89ffff640 1 -- 172.21.15.70:0/2170780297 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb8b800b040 con 0x5634bef190e0 2024-04-01T01:35:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.192+0000 7fb8c9a50740 10 client.26087 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.193+0000 7fb8c9a50740 10 client.26087 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.193+0000 7fb8c9a50740 20 client.26087 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.193+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/2170780297 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5634bef371b0 0x5634bef395c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.193+0000 7fb8c9a50740 4 client.26087 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.193+0000 7fb8c9a50740 1 -- 172.21.15.70:0/2170780297 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb8b801c070 con 0x5634bef371b0 2024-04-01T01:35:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.193+0000 7fb8bffff640 1 --2- 172.21.15.70:0/2170780297 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5634bef371b0 0x5634bef395c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.194+0000 7fb8bffff640 1 --2- 172.21.15.70:0/2170780297 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5634bef371b0 0x5634bef395c0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74cc6e965fb0bb0e server_cookie=4c0d10042afa01c3 in_seq=0 out_seq=0 2024-04-01T01:35:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.194+0000 7fb89ffff640 10 client.26087 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.195+0000 7fb89ffff640 1 -- 172.21.15.70:0/2170780297 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fb8b801c070 con 0x5634bef371b0 2024-04-01T01:35:44.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.195+0000 7fb89ffff640 10 client.26087 handle_command_reply: tid=0 2024-04-01T01:35:44.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.195+0000 7fb8c9a50740 1 client.26087 shutdown 2024-04-01T01:35:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.196+0000 7fb8c9a50740 1 -- 172.21.15.70:0/2170780297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef190e0 msgr2=0x5634bef12290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.196+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/2170780297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef190e0 0x5634bef12290 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800e000 tx=0x7fb8b8000f40 comp rx=0 tx=0).stop 2024-04-01T01:35:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.197+0000 7fb8c9a50740 1 -- 172.21.15.70:0/2170780297 shutdown_connections 2024-04-01T01:35:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.197+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/2170780297 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5634bef371b0 0x5634bef395c0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.197+0000 7fb89ffff640 0 client.26087 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.197+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/2170780297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef1ec40 0x5634bef17800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.197+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/2170780297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef19ad0 0x5634bef127d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.197+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/2170780297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef190e0 0x5634bef12290 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.197+0000 7fb8c9a50740 1 -- 172.21.15.70:0/2170780297 >> 172.21.15.70:0/2170780297 conn(0x5634bef0be40 msgr2=0x5634bef0af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.198+0000 7fb8c9a50740 1 -- 172.21.15.70:0/2170780297 shutdown_connections 2024-04-01T01:35:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.198+0000 7fb8c9a50740 1 -- 172.21.15.70:0/2170780297 wait complete. 2024-04-01T01:35:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.198+0000 7fb8c9a50740 20 client.26087 trim_cache size 0 max 16384 2024-04-01T01:35:44.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.232+0000 7fb8c9a50740 1 Processor -- start 2024-04-01T01:35:44.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.232+0000 7fb8c9a50740 1 -- start start 2024-04-01T01:35:44.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.232+0000 7fb8c9a50740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef190e0 0x5634bef3ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.232+0000 7fb8c9a50740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef19ad0 0x5634bef0f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.232+0000 7fb8c9a50740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef1ec40 0x5634bef0f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.232+0000 7fb8c9a50740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c01a8200 con 0x5634bef1ec40 2024-04-01T01:35:44.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.232+0000 7fb8c9a50740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c01a9d40 con 0x5634bef190e0 2024-04-01T01:35:44.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.232+0000 7fb8c9a50740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c01a88d0 con 0x5634bef19ad0 2024-04-01T01:35:44.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.232+0000 7fb8c540f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef190e0 0x5634bef3ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:44.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.232+0000 7fb8c540f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef190e0 0x5634bef3ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:39976/0 (socket says 172.21.15.70:39976) 2024-04-01T01:35:44.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.233+0000 7fb8c540f640 1 -- 172.21.15.70:0/2996244546 learned_addr learned my addr 172.21.15.70:0/2996244546 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:44.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.233+0000 7fb8bffff640 1 --2- 172.21.15.70:0/2996244546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef19ad0 0x5634bef0f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:44.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.233+0000 7fb8c540f640 1 --2- 172.21.15.70:0/2996244546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef1ec40 0x5634bef0f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:44.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.233+0000 7fb8c540f640 1 -- 172.21.15.70:0/2996244546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef19ad0 msgr2=0x5634bef0f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:44.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.233+0000 7fb8c540f640 1 --2- 172.21.15.70:0/2996244546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef19ad0 0x5634bef0f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.233+0000 7fb8c540f640 1 -- 172.21.15.70:0/2996244546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef1ec40 msgr2=0x5634bef0f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:44.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.233+0000 7fb8c540f640 1 --2- 172.21.15.70:0/2996244546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef1ec40 0x5634bef0f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.233+0000 7fb8c540f640 1 -- 172.21.15.70:0/2996244546 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634bef10ae0 con 0x5634bef190e0 2024-04-01T01:35:44.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.233+0000 7fb8c540f640 1 --2- 172.21.15.70:0/2996244546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef190e0 0x5634bef3ef20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800e4d0 tx=0x7fb8b8026000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bea8021095a21cac server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:44.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.234+0000 7fb8a67fc640 1 -- 172.21.15.70:0/2996244546 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8017630 con 0x5634bef190e0 2024-04-01T01:35:44.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.234+0000 7fb8a67fc640 1 -- 172.21.15.70:0/2996244546 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8b801c070 con 0x5634bef190e0 2024-04-01T01:35:44.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.234+0000 7fb8a67fc640 1 -- 172.21.15.70:0/2996244546 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8005950 con 0x5634bef190e0 2024-04-01T01:35:44.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.235+0000 7fb8c9a50740 1 -- 172.21.15.70:0/2996244546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef190e0 msgr2=0x5634bef3ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:44.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.235+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/2996244546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef190e0 0x5634bef3ef20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800e4d0 tx=0x7fb8b8026000 comp rx=0 tx=0).stop 2024-04-01T01:35:44.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.235+0000 7fb8c9a50740 1 -- 172.21.15.70:0/2996244546 shutdown_connections 2024-04-01T01:35:44.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.235+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/2996244546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef1ec40 0x5634bef0f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.235+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/2996244546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef19ad0 0x5634bef0f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.235+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/2996244546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef190e0 0x5634bef3ef20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.235+0000 7fb8c9a50740 1 -- 172.21.15.70:0/2996244546 >> 172.21.15.70:0/2996244546 conn(0x5634bef0be40 msgr2=0x5634bef0c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:44.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.235+0000 7fb8c9a50740 1 -- 172.21.15.70:0/2996244546 shutdown_connections 2024-04-01T01:35:44.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.235+0000 7fb8c9a50740 1 -- 172.21.15.70:0/2996244546 wait complete. 2024-04-01T01:35:44.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.236+0000 7fb8c9a50740 1 -- start start 2024-04-01T01:35:44.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.236+0000 7fb8c9a50740 1 Processor -- start 2024-04-01T01:35:44.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.236+0000 7fb8c9a50740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef190e0 0x5634bef4a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.236+0000 7fb8c9a50740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef19ad0 0x5634bef4a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.236+0000 7fb8c9a50740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef1ec40 0x5634bef48890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.237+0000 7fb8c9a50740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c01a9670 con 0x5634bef190e0 2024-04-01T01:35:44.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.237+0000 7fb8c9a50740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c01a8fa0 con 0x5634bef1ec40 2024-04-01T01:35:44.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.237+0000 7fb8c9a50740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c01a61e0 con 0x5634bef19ad0 2024-04-01T01:35:44.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.237+0000 7fb8c540f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef190e0 0x5634bef4a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:44.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.237+0000 7fb8c540f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef190e0 0x5634bef4a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56756/0 (socket says 172.21.15.70:56756) 2024-04-01T01:35:44.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.237+0000 7fb8c540f640 1 -- 172.21.15.70:0/523518965 learned_addr learned my addr 172.21.15.70:0/523518965 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:44.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.237+0000 7fb8c9a50740 10 client.0 _ll_register_callbacks cb 0x7fb8b800e7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:44.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.237+0000 7fb8bffff640 1 --2- 172.21.15.70:0/523518965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef19ad0 0x5634bef4a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.237+0000 7fb8c540f640 1 --2- 172.21.15.70:0/523518965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef1ec40 0x5634bef48890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.238+0000 7fb8c540f640 1 -- 172.21.15.70:0/523518965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef19ad0 msgr2=0x5634bef4a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.238+0000 7fb8c540f640 1 --2- 172.21.15.70:0/523518965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef19ad0 0x5634bef4a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.238+0000 7fb8c540f640 1 -- 172.21.15.70:0/523518965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef190e0 msgr2=0x5634bef4a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.238+0000 7fb8c540f640 1 --2- 172.21.15.70:0/523518965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef190e0 0x5634bef4a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.238+0000 7fb8c540f640 1 -- 172.21.15.70:0/523518965 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634bef49090 con 0x5634bef1ec40 2024-04-01T01:35:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.239+0000 7fb8c540f640 1 --2- 172.21.15.70:0/523518965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef1ec40 0x5634bef48890 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800e8f0 tx=0x7fb8b8017f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=390733caab96977a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.239+0000 7fb89ffff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.239+0000 7fb89ffff640 1 -- 172.21.15.70:0/523518965 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b80058a0 con 0x5634bef1ec40 2024-04-01T01:35:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.240+0000 7fb89ffff640 1 -- 172.21.15.70:0/523518965 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb8b8017630 con 0x5634bef1ec40 2024-04-01T01:35:44.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.240+0000 7fb8c9a50740 1 -- 172.21.15.70:0/523518965 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8b801c070 con 0x5634bef1ec40 2024-04-01T01:35:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.240+0000 7fb89ffff640 1 -- 172.21.15.70:0/523518965 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb8b8017830 con 0x5634bef1ec40 2024-04-01T01:35:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.240+0000 7fb8c9a50740 10 client.26093 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.240+0000 7fb8c9a50740 1 -- 172.21.15.70:0/523518965 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5634bef9dac0 con 0x5634bef1ec40 2024-04-01T01:35:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.241+0000 7fb89ffff640 1 -- 172.21.15.70:0/523518965 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8b8003970 con 0x5634bef1ec40 2024-04-01T01:35:44.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.241+0000 7fb89ffff640 1 -- 172.21.15.70:0/523518965 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb8b800b040 con 0x5634bef1ec40 2024-04-01T01:35:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.241+0000 7fb8c9a50740 10 client.26093 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.241+0000 7fb8c9a50740 10 client.26093 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.241+0000 7fb8c9a50740 20 client.26093 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.241+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/523518965 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5634bef371b0 0x5634befa0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.241+0000 7fb8c9a50740 4 client.26093 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:35:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.241+0000 7fb8c9a50740 1 -- 172.21.15.70:0/523518965 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fb8b801c070 con 0x5634bef371b0 2024-04-01T01:35:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.241+0000 7fb8bffff640 1 --2- 172.21.15.70:0/523518965 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5634bef371b0 0x5634befa0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.242+0000 7fb8bffff640 1 --2- 172.21.15.70:0/523518965 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5634bef371b0 0x5634befa0260 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e83676b0fe7ea27e server_cookie=ab978a7525aaa205 in_seq=0 out_seq=0 2024-04-01T01:35:44.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.242+0000 7fb89ffff640 10 client.26093 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.313+0000 7fb89ffff640 1 -- 172.21.15.70:0/523518965 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fb8b801c070 con 0x5634bef371b0 2024-04-01T01:35:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.313+0000 7fb89ffff640 10 client.26093 handle_command_reply: tid=0 2024-04-01T01:35:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.313+0000 7fb8c9a50740 1 client.26093 shutdown 2024-04-01T01:35:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.315+0000 7fb8c9a50740 1 -- 172.21.15.70:0/523518965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef1ec40 msgr2=0x5634bef48890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.315+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/523518965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef1ec40 0x5634bef48890 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800e8f0 tx=0x7fb8b8017f90 comp rx=0 tx=0).stop 2024-04-01T01:35:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.315+0000 7fb8c9a50740 1 -- 172.21.15.70:0/523518965 shutdown_connections 2024-04-01T01:35:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.315+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/523518965 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5634bef371b0 0x5634befa0260 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.315+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/523518965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634bef1ec40 0x5634bef48890 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.315+0000 7fb89ffff640 0 client.26093 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.315+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/523518965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634bef19ad0 0x5634bef4a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.315+0000 7fb8c9a50740 1 --2- 172.21.15.70:0/523518965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634bef190e0 0x5634bef4a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.315+0000 7fb8c9a50740 1 -- 172.21.15.70:0/523518965 >> 172.21.15.70:0/523518965 conn(0x5634bef0be40 msgr2=0x5634bef0f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.317+0000 7fb8c9a50740 1 -- 172.21.15.70:0/523518965 shutdown_connections 2024-04-01T01:35:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.317+0000 7fb8c9a50740 1 -- 172.21.15.70:0/523518965 wait complete. 2024-04-01T01:35:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.317+0000 7fb8c9a50740 20 client.26093 trim_cache size 0 max 16384 2024-04-01T01:35:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:35:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:35:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:35:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:35:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.319+0000 7fb8c6e72640 1 -- 172.21.15.70:0/319513296 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8a80a25c0 msgr2=0x7fb8a80a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.319+0000 7fb8c6e72640 1 --2- 172.21.15.70:0/319513296 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8a80a25c0 0x7fb8a80a4a80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb8b00a33e0 tx=0x7fb8b000a5d0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.319+0000 7fb8c6e72640 1 -- 172.21.15.70:0/319513296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8c0100670 msgr2=0x7fb8c0199ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.319+0000 7fb8c6e72640 1 --2- 172.21.15.70:0/319513296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8c0100670 0x7fb8c0199ae0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000c970 tx=0x7fb8b000ce30 comp rx=0 tx=0).stop 2024-04-01T01:35:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.319+0000 7fb8c6e72640 1 -- 172.21.15.70:0/319513296 shutdown_connections 2024-04-01T01:35:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.319+0000 7fb8c6e72640 1 --2- 172.21.15.70:0/319513296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8c0102220 0x7fb8c01a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.319+0000 7fb8c6e72640 1 --2- 172.21.15.70:0/319513296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8c0101860 0x7fb8c019a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.319+0000 7fb8c6e72640 1 --2- 172.21.15.70:0/319513296 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8a80a25c0 0x7fb8a80a4a80 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.319+0000 7fb8c6e72640 1 --2- 172.21.15.70:0/319513296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8c0100670 0x7fb8c0199ae0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.319+0000 7fb8c6e72640 1 -- 172.21.15.70:0/319513296 >> 172.21.15.70:0/319513296 conn(0x7fb8c00fbbe0 msgr2=0x7fb8c00fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:44.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.320+0000 7fb8c6e72640 1 -- 172.21.15.70:0/319513296 shutdown_connections 2024-04-01T01:35:44.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:44.320+0000 7fb8c6e72640 1 -- 172.21.15.70:0/319513296 wait complete. 2024-04-01T01:35:45.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:45 smithi189 ceph-mon[35753]: pgmap v426: 129 pgs: 129 active+clean; 961 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 11 MiB/s wr, 1.20k op/s 2024-04-01T01:35:46.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:45 smithi070 ceph-mon[32179]: pgmap v426: 129 pgs: 129 active+clean; 961 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 11 MiB/s wr, 1.20k op/s 2024-04-01T01:35:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:45 smithi081 ceph-mon[35142]: pgmap v426: 129 pgs: 129 active+clean; 961 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 11 MiB/s wr, 1.20k op/s 2024-04-01T01:35:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 60.0 s, 1168.4 tps, lat 85.490 ms stddev 30.526 2024-04-01T01:35:47.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:47 smithi189 ceph-mon[35753]: pgmap v427: 129 pgs: 129 active+clean; 970 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-04-01T01:35:48.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:47 smithi070 ceph-mon[32179]: pgmap v427: 129 pgs: 129 active+clean; 970 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-04-01T01:35:48.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:47 smithi081 ceph-mon[35142]: pgmap v427: 129 pgs: 129 active+clean; 970 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-04-01T01:35:49.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:49 smithi189 ceph-mon[35753]: pgmap v428: 129 pgs: 129 active+clean; 970 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 12 MiB/s wr, 1.32k op/s 2024-04-01T01:35:50.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:49 smithi070 ceph-mon[32179]: pgmap v428: 129 pgs: 129 active+clean; 970 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 12 MiB/s wr, 1.32k op/s 2024-04-01T01:35:50.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:49 smithi081 ceph-mon[35142]: pgmap v428: 129 pgs: 129 active+clean; 970 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 12 MiB/s wr, 1.32k op/s 2024-04-01T01:35:51.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 65.0 s, 1307.0 tps, lat 76.422 ms stddev 21.364 2024-04-01T01:35:51.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:51 smithi189 ceph-mon[35753]: pgmap v429: 129 pgs: 129 active+clean; 970 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 11 MiB/s wr, 1.20k op/s 2024-04-01T01:35:52.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:51 smithi070 ceph-mon[32179]: pgmap v429: 129 pgs: 129 active+clean; 970 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 11 MiB/s wr, 1.20k op/s 2024-04-01T01:35:52.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:51 smithi081 ceph-mon[35142]: pgmap v429: 129 pgs: 129 active+clean; 970 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 11 MiB/s wr, 1.20k op/s 2024-04-01T01:35:53.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:53 smithi189 ceph-mon[35753]: pgmap v430: 129 pgs: 129 active+clean; 970 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-04-01T01:35:54.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:53 smithi070 ceph-mon[32179]: pgmap v430: 129 pgs: 129 active+clean; 970 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-04-01T01:35:54.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:53 smithi081 ceph-mon[35142]: pgmap v430: 129 pgs: 129 active+clean; 970 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-04-01T01:35:54.998 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:35:54.999 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:35:55.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.172+0000 7f5a0dd9b640 1 -- 172.21.15.70:0/780466965 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a1009ac60 con 0x7f5a080a6160 2024-04-01T01:35:55.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.173+0000 7f5a1493d640 1 -- 172.21.15.70:0/780466965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a080a6160 msgr2=0x7f5a080a6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:55.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.173+0000 7f5a1493d640 1 --2- 172.21.15.70:0/780466965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a080a6160 0x7f5a080a6580 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5a1006efd0 tx=0x7f5a10093470 comp rx=0 tx=0).stop 2024-04-01T01:35:55.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.174+0000 7f5a1493d640 1 -- 172.21.15.70:0/780466965 shutdown_connections 2024-04-01T01:35:55.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.174+0000 7f5a1493d640 1 --2- 172.21.15.70:0/780466965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a080a5150 0x7f5a0800bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.174+0000 7f5a1493d640 1 --2- 172.21.15.70:0/780466965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a080a4790 0x7f5a080a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.174+0000 7f5a1493d640 1 --2- 172.21.15.70:0/780466965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a080a6160 0x7f5a080a6580 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.174+0000 7f5a1493d640 1 -- 172.21.15.70:0/780466965 >> 172.21.15.70:0/780466965 conn(0x7f5a080a00d0 msgr2=0x7f5a080a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:55.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.174+0000 7f5a1493d640 1 -- 172.21.15.70:0/780466965 shutdown_connections 2024-04-01T01:35:55.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.175+0000 7f5a1493d640 1 -- 172.21.15.70:0/780466965 wait complete. 2024-04-01T01:35:55.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.176+0000 7f5a1493d640 1 Processor -- start 2024-04-01T01:35:55.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.176+0000 7f5a1493d640 1 -- start start 2024-04-01T01:35:55.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.176+0000 7f5a1493d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a080a4790 0x7f5a080165e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.176+0000 7f5a1493d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a080a5150 0x7f5a08014c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.176+0000 7f5a1493d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a080a6160 0x7f5a080151b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.176+0000 7f5a1493d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0800f450 con 0x7f5a080a4790 2024-04-01T01:35:55.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.176+0000 7f5a1493d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0800f2d0 con 0x7f5a080a5150 2024-04-01T01:35:55.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.176+0000 7f5a1493d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0800f5d0 con 0x7f5a080a6160 2024-04-01T01:35:55.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.176+0000 7f5a0f59e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a080a6160 0x7f5a080151b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.176+0000 7f5a0f59e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a080a6160 0x7f5a080151b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52568/0 (socket says 172.21.15.70:52568) 2024-04-01T01:35:55.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.176+0000 7f5a0f59e640 1 -- 172.21.15.70:0/611149709 learned_addr learned my addr 172.21.15.70:0/611149709 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:55.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.176+0000 7f5a0e59c640 1 --2- 172.21.15.70:0/611149709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a080a5150 0x7f5a08014c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.177+0000 7f5a0f59e640 1 -- 172.21.15.70:0/611149709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a080a5150 msgr2=0x7f5a08014c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:55.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.177+0000 7f5a0f59e640 1 --2- 172.21.15.70:0/611149709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a080a5150 0x7f5a08014c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.177+0000 7f5a0f59e640 1 -- 172.21.15.70:0/611149709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a080a4790 msgr2=0x7f5a080165e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:55.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.177+0000 7f5a0f59e640 1 --2- 172.21.15.70:0/611149709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a080a4790 0x7f5a080165e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.177+0000 7f5a0f59e640 1 -- 172.21.15.70:0/611149709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a08015930 con 0x7f5a080a6160 2024-04-01T01:35:55.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.177+0000 7f5a0f59e640 1 --2- 172.21.15.70:0/611149709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a080a6160 0x7f5a080151b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009990 tx=0x7f5a04009e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:55.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.177+0000 7f59f7fff640 1 -- 172.21.15.70:0/611149709 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0400c960 con 0x7f5a080a6160 2024-04-01T01:35:55.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.178+0000 7f5a1493d640 1 -- 172.21.15.70:0/611149709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a08153ce0 con 0x7f5a080a6160 2024-04-01T01:35:55.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.178+0000 7f5a1493d640 1 -- 172.21.15.70:0/611149709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a081541d0 con 0x7f5a080a6160 2024-04-01T01:35:55.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.178+0000 7f59f7fff640 1 -- 172.21.15.70:0/611149709 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a04005020 con 0x7f5a080a6160 2024-04-01T01:35:55.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.178+0000 7f59f7fff640 1 -- 172.21.15.70:0/611149709 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a04004ba0 con 0x7f5a080a6160 2024-04-01T01:35:55.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.181+0000 7f5a17df7740 1 Processor -- start 2024-04-01T01:35:55.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.184+0000 7f59f7fff640 1 -- 172.21.15.70:0/611149709 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f5a04016410 con 0x7f5a080a6160 2024-04-01T01:35:55.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.184+0000 7f5a17df7740 1 -- start start 2024-04-01T01:35:55.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.184+0000 7f5a17df7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6490 0x55af54de68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.184+0000 7f5a17df7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54de6e80 0x55af54deb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.184+0000 7f5a17df7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54debff0 0x55af54dee480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.184+0000 7f5a17df7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55af54d4b180 con 0x55af54de6490 2024-04-01T01:35:55.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.184+0000 7f5a17df7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55af54cf3b70 con 0x55af54debff0 2024-04-01T01:35:55.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.184+0000 7f5a17df7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55af54d47220 con 0x55af54de6e80 2024-04-01T01:35:55.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.185+0000 7f5a0e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54debff0 0x55af54dee480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.185+0000 7f5a0e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6490 0x55af54de68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.185+0000 7f5a0e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6490 0x55af54de68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58538/0 (socket says 172.21.15.70:58538) 2024-04-01T01:35:55.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.185+0000 7f5a0e59c640 1 -- 172.21.15.70:0/1085004402 learned_addr learned my addr 172.21.15.70:0/1085004402 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:55.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.186+0000 7f5a0e59c640 1 -- 172.21.15.70:0/1085004402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54de6e80 msgr2=0x55af54deb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:55.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.186+0000 7f5a0e59c640 1 --2- 172.21.15.70:0/1085004402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54de6e80 0x55af54deb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.186+0000 7f5a0e59c640 1 -- 172.21.15.70:0/1085004402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6490 msgr2=0x55af54de68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:55.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.186+0000 7f5a0e59c640 1 --2- 172.21.15.70:0/1085004402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6490 0x55af54de68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.186+0000 7f5a0e59c640 1 -- 172.21.15.70:0/1085004402 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af54d077c0 con 0x55af54debff0 2024-04-01T01:35:55.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.186+0000 7f59f7fff640 1 --2- 172.21.15.70:0/611149709 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59f80a25f0 0x7f59f80a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.186+0000 7f5a0e59c640 1 --2- 172.21.15.70:0/1085004402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54debff0 0x55af54dee480 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5a00005d00 tx=0x7f5a00015b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c8660df626bb433 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:55.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.186+0000 7f59f7fff640 1 -- 172.21.15.70:0/611149709 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5a040ce130 con 0x7f5a080a6160 2024-04-01T01:35:55.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.186+0000 7f5a0ed9d640 1 --2- 172.21.15.70:0/611149709 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59f80a25f0 0x7f59f80a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.186+0000 7f59f57fa640 1 -- 172.21.15.70:0/1085004402 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00005220 con 0x55af54debff0 2024-04-01T01:35:55.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.187+0000 7f59f57fa640 1 -- 172.21.15.70:0/1085004402 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a00004c50 con 0x55af54debff0 2024-04-01T01:35:55.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.188+0000 7f5a17df7740 1 -- 172.21.15.70:0/1085004402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54debff0 msgr2=0x55af54dee480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:55.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.188+0000 7f5a17df7740 1 --2- 172.21.15.70:0/1085004402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54debff0 0x55af54dee480 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5a00005d00 tx=0x7f5a00015b30 comp rx=0 tx=0).stop 2024-04-01T01:35:55.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.188+0000 7f5a17df7740 1 -- 172.21.15.70:0/1085004402 shutdown_connections 2024-04-01T01:35:55.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.188+0000 7f5a17df7740 1 --2- 172.21.15.70:0/1085004402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54debff0 0x55af54dee480 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.188+0000 7f5a17df7740 1 --2- 172.21.15.70:0/1085004402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54de6e80 0x55af54deb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.188+0000 7f5a17df7740 1 --2- 172.21.15.70:0/1085004402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6490 0x55af54de68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.188+0000 7f5a17df7740 1 -- 172.21.15.70:0/1085004402 >> 172.21.15.70:0/1085004402 conn(0x55af54de1a00 msgr2=0x55af54de1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:55.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.188+0000 7f5a0ed9d640 1 --2- 172.21.15.70:0/611149709 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59f80a25f0 0x7f59f80a4ab0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x55af54de7280 tx=0x7f5a1009a860 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:55.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.189+0000 7f5a17df7740 1 -- 172.21.15.70:0/1085004402 shutdown_connections 2024-04-01T01:35:55.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.189+0000 7f5a17df7740 1 -- 172.21.15.70:0/1085004402 wait complete. 2024-04-01T01:35:55.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.189+0000 7f5a17df7740 1 -- start start 2024-04-01T01:35:55.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.189+0000 7f5a17df7740 1 Processor -- start 2024-04-01T01:35:55.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.189+0000 7f5a17df7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54de6490 0x55af54e017e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.189+0000 7f5a17df7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6e80 0x55af54e01d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.189+0000 7f5a17df7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54e06d50 0x55af54e09140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.189+0000 7f5a17df7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55af54dfffa0 con 0x55af54e06d50 2024-04-01T01:35:55.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.190+0000 7f5a17df7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55af54ce94a0 con 0x55af54de6e80 2024-04-01T01:35:55.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.190+0000 7f5a17df7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55af54e00120 con 0x55af54de6490 2024-04-01T01:35:55.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.190+0000 7f5a17df7740 10 client.0 _ll_register_callbacks cb 0x55af54cecd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:55.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.190+0000 7f5a0e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6e80 0x55af54e01d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.190+0000 7f5a0e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6e80 0x55af54e01d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33182/0 (socket says 172.21.15.70:33182) 2024-04-01T01:35:55.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.190+0000 7f5a0e59c640 1 -- 172.21.15.70:0/478045805 learned_addr learned my addr 172.21.15.70:0/478045805 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:55.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.190+0000 7f5a0e59c640 1 -- 172.21.15.70:0/478045805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54de6490 msgr2=0x55af54e017e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:35:55.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.190+0000 7f5a0e59c640 1 --2- 172.21.15.70:0/478045805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54de6490 0x55af54e017e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.190+0000 7f5a0f59e640 1 --2- 172.21.15.70:0/478045805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54e06d50 0x55af54e09140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.190+0000 7f5a0e59c640 1 -- 172.21.15.70:0/478045805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54e06d50 msgr2=0x55af54e09140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:55.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.190+0000 7f5a0e59c640 1 --2- 172.21.15.70:0/478045805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54e06d50 0x55af54e09140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.190+0000 7f5a0e59c640 1 -- 172.21.15.70:0/478045805 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af54b297d0 con 0x55af54de6e80 2024-04-01T01:35:55.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.191+0000 7f5a0f59e640 1 --2- 172.21.15.70:0/478045805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54e06d50 0x55af54e09140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:35:55.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.191+0000 7f5a0e59c640 1 --2- 172.21.15.70:0/478045805 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6e80 0x55af54e01d20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5a00001eb0 tx=0x7f5a0000b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37863574ba270f13 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:55.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.191+0000 7f59f2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:55.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.191+0000 7f59f2ff5640 1 -- 172.21.15.70:0/478045805 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00010c70 con 0x55af54de6e80 2024-04-01T01:35:55.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.191+0000 7f59f2ff5640 1 -- 172.21.15.70:0/478045805 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5a00005020 con 0x55af54de6e80 2024-04-01T01:35:55.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.191+0000 7f5a17df7740 1 -- 172.21.15.70:0/478045805 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55af54d586c0 con 0x55af54de6e80 2024-04-01T01:35:55.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.192+0000 7f59f2ff5640 1 -- 172.21.15.70:0/478045805 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5a00031050 con 0x55af54de6e80 2024-04-01T01:35:55.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.192+0000 7f5a17df7740 10 client.26105 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:55.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.192+0000 7f5a17df7740 1 -- 172.21.15.70:0/478045805 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55af54a5f3f0 con 0x55af54de6e80 2024-04-01T01:35:55.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.192+0000 7f59f2ff5640 1 -- 172.21.15.70:0/478045805 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55af54d586c0 con 0x55af54de6e80 2024-04-01T01:35:55.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.193+0000 7f59f2ff5640 1 -- 172.21.15.70:0/478045805 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5a00004a40 con 0x55af54de6e80 2024-04-01T01:35:55.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.193+0000 7f5a17df7740 10 client.26105 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:55.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.193+0000 7f5a17df7740 10 client.26105 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:55.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.193+0000 7f5a17df7740 20 client.26105 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:55.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.193+0000 7f5a17df7740 1 --2- 172.21.15.70:0/478045805 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55af54df1710 0x55af54df3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.193+0000 7f5a17df7740 4 client.26105 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:55.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.193+0000 7f5a17df7740 1 -- 172.21.15.70:0/478045805 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af54d52a10 con 0x55af54df1710 2024-04-01T01:35:55.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.193+0000 7f5a0ed9d640 1 --2- 172.21.15.70:0/478045805 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55af54df1710 0x55af54df3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.194+0000 7f5a0ed9d640 1 --2- 172.21.15.70:0/478045805 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55af54df1710 0x55af54df3b90 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1de5ef03cdc6837a server_cookie=22a48eca38a34b39 in_seq=0 out_seq=0 2024-04-01T01:35:55.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.194+0000 7f59f2ff5640 10 client.26105 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:55.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.196+0000 7f59f2ff5640 1 -- 172.21.15.70:0/478045805 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55af54d52a10 con 0x55af54df1710 2024-04-01T01:35:55.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.196+0000 7f59f2ff5640 10 client.26105 handle_command_reply: tid=0 2024-04-01T01:35:55.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.196+0000 7f5a17df7740 1 client.26105 shutdown 2024-04-01T01:35:55.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.197+0000 7f5a17df7740 1 -- 172.21.15.70:0/478045805 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6e80 msgr2=0x55af54e01d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:55.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.197+0000 7f5a17df7740 1 --2- 172.21.15.70:0/478045805 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6e80 0x55af54e01d20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5a00001eb0 tx=0x7f5a0000b390 comp rx=0 tx=0).stop 2024-04-01T01:35:55.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.197+0000 7f5a17df7740 1 -- 172.21.15.70:0/478045805 shutdown_connections 2024-04-01T01:35:55.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.197+0000 7f5a17df7740 1 --2- 172.21.15.70:0/478045805 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55af54df1710 0x55af54df3b90 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.197+0000 7f5a17df7740 1 --2- 172.21.15.70:0/478045805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54e06d50 0x55af54e09140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.197+0000 7f5a17df7740 1 --2- 172.21.15.70:0/478045805 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6e80 0x55af54e01d20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.197+0000 7f5a17df7740 1 --2- 172.21.15.70:0/478045805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54de6490 0x55af54e017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.197+0000 7f5a17df7740 1 -- 172.21.15.70:0/478045805 >> 172.21.15.70:0/478045805 conn(0x55af54de1a00 msgr2=0x55af54de0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:55.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.197+0000 7f59f2ff5640 0 client.26105 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:55.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.197+0000 7f5a17df7740 1 -- 172.21.15.70:0/478045805 shutdown_connections 2024-04-01T01:35:55.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.198+0000 7f5a17df7740 1 -- 172.21.15.70:0/478045805 wait complete. 2024-04-01T01:35:55.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.198+0000 7f5a17df7740 20 client.26105 trim_cache size 0 max 16384 2024-04-01T01:35:55.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.230+0000 7f5a17df7740 1 Processor -- start 2024-04-01T01:35:55.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.230+0000 7f5a17df7740 1 -- start start 2024-04-01T01:35:55.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.230+0000 7f5a17df7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54de1a00 0x55af54de8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.231+0000 7f5a17df7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6490 0x55af54de8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.231+0000 7f5a17df7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6e80 0x55af54de4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.231+0000 7f5a17df7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc00fd10 con 0x55af54de6e80 2024-04-01T01:35:55.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.231+0000 7f5a17df7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc011850 con 0x55af54de6490 2024-04-01T01:35:55.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.231+0000 7f5a0f59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6e80 0x55af54de4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.231+0000 7f5a0e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6490 0x55af54de8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.231+0000 7f5a0f59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6e80 0x55af54de4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58558/0 (socket says 172.21.15.70:58558) 2024-04-01T01:35:55.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.231+0000 7f5a0f59e640 1 -- 172.21.15.70:0/3673402050 learned_addr learned my addr 172.21.15.70:0/3673402050 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:55.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.231+0000 7f5a17df7740 1 -- 172.21.15.70:0/3673402050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc0103e0 con 0x55af54de1a00 2024-04-01T01:35:55.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.232+0000 7f5a0e59c640 1 -- 172.21.15.70:0/3673402050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54de1a00 msgr2=0x55af54de8100 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:35:55.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.232+0000 7f5a0e59c640 1 --2- 172.21.15.70:0/3673402050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54de1a00 0x55af54de8100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.232+0000 7f5a0e59c640 1 -- 172.21.15.70:0/3673402050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6e80 msgr2=0x55af54de4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:55.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.232+0000 7f5a0e59c640 1 --2- 172.21.15.70:0/3673402050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6e80 0x55af54de4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.232+0000 7f5a0e59c640 1 -- 172.21.15.70:0/3673402050 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af54a1f0f0 con 0x55af54de6490 2024-04-01T01:35:55.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.232+0000 7f5a0e59c640 1 --2- 172.21.15.70:0/3673402050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6490 0x55af54de8640 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5a00005870 tx=0x7f5a00015c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f4c9510bc6c3244 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:55.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.232+0000 7f59f4ff9640 1 -- 172.21.15.70:0/3673402050 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0002b040 con 0x55af54de6490 2024-04-01T01:35:55.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.233+0000 7f59f4ff9640 1 -- 172.21.15.70:0/3673402050 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a0002b1e0 con 0x55af54de6490 2024-04-01T01:35:55.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.233+0000 7f59f4ff9640 1 -- 172.21.15.70:0/3673402050 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00003850 con 0x55af54de6490 2024-04-01T01:35:55.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.234+0000 7f5a17df7740 1 -- 172.21.15.70:0/3673402050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6490 msgr2=0x55af54de8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:55.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.234+0000 7f5a17df7740 1 --2- 172.21.15.70:0/3673402050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6490 0x55af54de8640 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5a00005870 tx=0x7f5a00015c90 comp rx=0 tx=0).stop 2024-04-01T01:35:55.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.235+0000 7f5a17df7740 1 -- 172.21.15.70:0/3673402050 shutdown_connections 2024-04-01T01:35:55.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.235+0000 7f5a17df7740 1 --2- 172.21.15.70:0/3673402050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6e80 0x55af54de4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.235+0000 7f5a17df7740 1 --2- 172.21.15.70:0/3673402050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6490 0x55af54de8640 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.235+0000 7f5a17df7740 1 --2- 172.21.15.70:0/3673402050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54de1a00 0x55af54de8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.236+0000 7f5a17df7740 1 -- 172.21.15.70:0/3673402050 >> 172.21.15.70:0/3673402050 conn(0x55af54de2150 msgr2=0x55af54e017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:55.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.236+0000 7f5a17df7740 1 -- 172.21.15.70:0/3673402050 shutdown_connections 2024-04-01T01:35:55.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.237+0000 7f5a17df7740 1 -- 172.21.15.70:0/3673402050 wait complete. 2024-04-01T01:35:55.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.238+0000 7f5a17df7740 1 -- start start 2024-04-01T01:35:55.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.239+0000 7f5a17df7740 1 Processor -- start 2024-04-01T01:35:55.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.239+0000 7f5a17df7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6490 0x55af54e2f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.239+0000 7f5a17df7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6e80 0x55af54e2f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.239+0000 7f5a17df7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54e06d50 0x55af54e82950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.239+0000 7f5a17df7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc011180 con 0x55af54de6490 2024-04-01T01:35:55.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.239+0000 7f5a17df7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc010ab0 con 0x55af54de6e80 2024-04-01T01:35:55.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.239+0000 7f5a17df7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc00dcf0 con 0x55af54e06d50 2024-04-01T01:35:55.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.239+0000 7f5a17df7740 10 client.0 _ll_register_callbacks cb 0x7f5a0000ba40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:55.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.239+0000 7f5a0e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6e80 0x55af54e2f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.239+0000 7f5a0f59e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54e06d50 0x55af54e82950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.239+0000 7f5a0e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6e80 0x55af54e2f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33206/0 (socket says 172.21.15.70:33206) 2024-04-01T01:35:55.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.239+0000 7f5a0e59c640 1 -- 172.21.15.70:0/3263583028 learned_addr learned my addr 172.21.15.70:0/3263583028 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:55.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.240+0000 7f5a0ed9d640 1 --2- 172.21.15.70:0/3263583028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6490 0x55af54e2f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.240+0000 7f5a0e59c640 1 -- 172.21.15.70:0/3263583028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54e06d50 msgr2=0x55af54e82950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:55.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.240+0000 7f5a0e59c640 1 --2- 172.21.15.70:0/3263583028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54e06d50 0x55af54e82950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.240+0000 7f5a0e59c640 1 -- 172.21.15.70:0/3263583028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6490 msgr2=0x55af54e2f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:55.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.240+0000 7f5a0e59c640 1 --2- 172.21.15.70:0/3263583028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6490 0x55af54e2f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.240+0000 7f5a0e59c640 1 -- 172.21.15.70:0/3263583028 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af54e82f40 con 0x55af54de6e80 2024-04-01T01:35:55.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.240+0000 7f5a0e59c640 1 --2- 172.21.15.70:0/3263583028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6e80 0x55af54e2f6c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5a00005960 tx=0x7f5a0000b570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2167fe7bdf8190d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:55.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.240+0000 7f59f2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:55.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.241+0000 7f59f2ff5640 1 -- 172.21.15.70:0/3263583028 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00010c70 con 0x55af54de6e80 2024-04-01T01:35:55.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.241+0000 7f59f2ff5640 1 -- 172.21.15.70:0/3263583028 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5a000035f0 con 0x55af54de6e80 2024-04-01T01:35:55.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.241+0000 7f5a17df7740 1 -- 172.21.15.70:0/3263583028 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a0002b1e0 con 0x55af54de6e80 2024-04-01T01:35:55.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.241+0000 7f59f2ff5640 1 -- 172.21.15.70:0/3263583028 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5a00010e10 con 0x55af54de6e80 2024-04-01T01:35:55.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.241+0000 7f5a17df7740 10 client.26111 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:55.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.241+0000 7f5a17df7740 1 -- 172.21.15.70:0/3263583028 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55af54e841f0 con 0x55af54de6e80 2024-04-01T01:35:55.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.241+0000 7f59f2ff5640 1 -- 172.21.15.70:0/3263583028 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a0002b1e0 con 0x55af54de6e80 2024-04-01T01:35:55.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.242+0000 7f59f2ff5640 1 -- 172.21.15.70:0/3263583028 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5a00036700 con 0x55af54de6e80 2024-04-01T01:35:55.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.242+0000 7f5a17df7740 10 client.26111 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:55.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.242+0000 7f5a17df7740 10 client.26111 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:55.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.242+0000 7f5a17df7740 20 client.26111 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:55.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.242+0000 7f5a17df7740 1 --2- 172.21.15.70:0/3263583028 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55af54e84ab0 0x55af54e86ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.242+0000 7f5a17df7740 4 client.26111 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:35:55.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.242+0000 7f5a17df7740 1 -- 172.21.15.70:0/3263583028 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55af54cd0540 con 0x55af54e84ab0 2024-04-01T01:35:55.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.243+0000 7f5a0ed9d640 1 --2- 172.21.15.70:0/3263583028 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55af54e84ab0 0x55af54e86ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.243+0000 7f5a0ed9d640 1 --2- 172.21.15.70:0/3263583028 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55af54e84ab0 0x55af54e86ed0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47d376abf8dba1fe server_cookie=ccf3809459328b25 in_seq=0 out_seq=0 2024-04-01T01:35:55.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.243+0000 7f59f2ff5640 10 client.26111 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:55.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.244+0000 7f59f2ff5640 1 -- 172.21.15.70:0/3263583028 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55af54cd0540 con 0x55af54e84ab0 2024-04-01T01:35:55.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.244+0000 7f59f2ff5640 10 client.26111 handle_command_reply: tid=0 2024-04-01T01:35:55.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.244+0000 7f5a17df7740 1 client.26111 shutdown 2024-04-01T01:35:55.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.244+0000 7f5a17df7740 1 -- 172.21.15.70:0/3263583028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6e80 msgr2=0x55af54e2f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:55.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.244+0000 7f5a17df7740 1 --2- 172.21.15.70:0/3263583028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6e80 0x55af54e2f6c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5a00005960 tx=0x7f5a0000b570 comp rx=0 tx=0).stop 2024-04-01T01:35:55.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.245+0000 7f5a17df7740 1 -- 172.21.15.70:0/3263583028 shutdown_connections 2024-04-01T01:35:55.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.245+0000 7f5a17df7740 1 --2- 172.21.15.70:0/3263583028 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55af54e84ab0 0x55af54e86ed0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.245+0000 7f5a17df7740 1 --2- 172.21.15.70:0/3263583028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55af54e06d50 0x55af54e82950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.245+0000 7f5a17df7740 1 --2- 172.21.15.70:0/3263583028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55af54de6e80 0x55af54e2f6c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.245+0000 7f5a17df7740 1 --2- 172.21.15.70:0/3263583028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af54de6490 0x55af54e2f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.245+0000 7f5a17df7740 1 -- 172.21.15.70:0/3263583028 >> 172.21.15.70:0/3263583028 conn(0x55af54de2150 msgr2=0x55af54de4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.245+0000 7f59f2ff5640 0 client.26111 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.246+0000 7f5a17df7740 1 -- 172.21.15.70:0/3263583028 shutdown_connections 2024-04-01T01:35:55.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.246+0000 7f5a17df7740 1 -- 172.21.15.70:0/3263583028 wait complete. 2024-04-01T01:35:55.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.248+0000 7f5a17df7740 20 client.26111 trim_cache size 0 max 16384 2024-04-01T01:35:55.249 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:35:55.249 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:35:55.249 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:35:55.249 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:35:55.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.249+0000 7f5a1493d640 1 -- 172.21.15.70:0/611149709 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59f80a25f0 msgr2=0x7f59f80a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:55.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.249+0000 7f5a1493d640 1 --2- 172.21.15.70:0/611149709 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59f80a25f0 0x7f59f80a4ab0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x55af54de7280 tx=0x7f5a1009a860 comp rx=0 tx=0).stop 2024-04-01T01:35:55.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.249+0000 7f5a1493d640 1 -- 172.21.15.70:0/611149709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a080a6160 msgr2=0x7f5a080151b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:55.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.249+0000 7f5a1493d640 1 --2- 172.21.15.70:0/611149709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a080a6160 0x7f5a080151b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009990 tx=0x7f5a04009e50 comp rx=0 tx=0).stop 2024-04-01T01:35:55.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.249+0000 7f5a1493d640 1 -- 172.21.15.70:0/611149709 shutdown_connections 2024-04-01T01:35:55.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.249+0000 7f5a1493d640 1 --2- 172.21.15.70:0/611149709 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59f80a25f0 0x7f59f80a4ab0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.249+0000 7f5a1493d640 1 --2- 172.21.15.70:0/611149709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a080a6160 0x7f5a080151b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.249+0000 7f5a1493d640 1 --2- 172.21.15.70:0/611149709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a080a5150 0x7f5a08014c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.249+0000 7f5a1493d640 1 --2- 172.21.15.70:0/611149709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5a080a4790 0x7f5a080165e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.249+0000 7f5a1493d640 1 -- 172.21.15.70:0/611149709 >> 172.21.15.70:0/611149709 conn(0x7f5a080a00d0 msgr2=0x7f5a080079a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:55.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.249+0000 7f5a1493d640 1 -- 172.21.15.70:0/611149709 shutdown_connections 2024-04-01T01:35:55.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.249+0000 7f5a1493d640 1 -- 172.21.15.70:0/611149709 wait complete. 2024-04-01T01:35:55.266 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:35:55.266 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:35:55.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.441+0000 7fba7ef7a640 1 -- 172.21.15.70:0/3933972536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba78101780 msgr2=0x7fba78101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:55.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.441+0000 7fba7ef7a640 1 --2- 172.21.15.70:0/3933972536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba78101780 0x7fba78101ba0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fba68009860 tx=0x7fba6802bb90 comp rx=0 tx=0).stop 2024-04-01T01:35:55.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.441+0000 7fba7ef7a640 1 -- 172.21.15.70:0/3933972536 shutdown_connections 2024-04-01T01:35:55.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.441+0000 7fba7ef7a640 1 --2- 172.21.15.70:0/3933972536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba78103330 0x7fba78109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.441+0000 7fba7ef7a640 1 --2- 172.21.15.70:0/3933972536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba78102970 0x7fba78102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.441+0000 7fba7ef7a640 1 --2- 172.21.15.70:0/3933972536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba78101780 0x7fba78101ba0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.441+0000 7fba7ef7a640 1 -- 172.21.15.70:0/3933972536 >> 172.21.15.70:0/3933972536 conn(0x7fba780fccf0 msgr2=0x7fba780ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:55.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.442+0000 7fba7ef7a640 1 -- 172.21.15.70:0/3933972536 shutdown_connections 2024-04-01T01:35:55.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.442+0000 7fba7ef7a640 1 -- 172.21.15.70:0/3933972536 wait complete. 2024-04-01T01:35:55.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.443+0000 7fba7ef7a640 1 Processor -- start 2024-04-01T01:35:55.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.443+0000 7fba7ef7a640 1 -- start start 2024-04-01T01:35:55.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.444+0000 7fba7ef7a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba78101780 0x7fba78195880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.444+0000 7fba7ef7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba78102970 0x7fba78195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.444+0000 7fba7ef7a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba78103330 0x7fba7819ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.444+0000 7fba7ef7a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fba7810d320 con 0x7fba78103330 2024-04-01T01:35:55.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.444+0000 7fba7ef7a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fba7810d1a0 con 0x7fba78102970 2024-04-01T01:35:55.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.444+0000 7fba7ef7a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fba7810d4a0 con 0x7fba78101780 2024-04-01T01:35:55.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.444+0000 7fba77fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba78102970 0x7fba78195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.444+0000 7fba77fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba78102970 0x7fba78195dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33224/0 (socket says 172.21.15.70:33224) 2024-04-01T01:35:55.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.444+0000 7fba77fff640 1 -- 172.21.15.70:0/652568723 learned_addr learned my addr 172.21.15.70:0/652568723 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:55.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.444+0000 7fba7cd16640 1 --2- 172.21.15.70:0/652568723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba78101780 0x7fba78195880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.444+0000 7fba7d517640 1 --2- 172.21.15.70:0/652568723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba78103330 0x7fba7819ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.445+0000 7fba77fff640 1 -- 172.21.15.70:0/652568723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba78101780 msgr2=0x7fba78195880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:55.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.445+0000 7fba77fff640 1 --2- 172.21.15.70:0/652568723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba78101780 0x7fba78195880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.445+0000 7fba77fff640 1 -- 172.21.15.70:0/652568723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba78103330 msgr2=0x7fba7819ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:55.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.445+0000 7fba77fff640 1 --2- 172.21.15.70:0/652568723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba78103330 0x7fba7819ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.445+0000 7fba77fff640 1 -- 172.21.15.70:0/652568723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba7819d510 con 0x7fba78102970 2024-04-01T01:35:55.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.445+0000 7fba77fff640 1 --2- 172.21.15.70:0/652568723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba78102970 0x7fba78195dc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fba7000ba50 tx=0x7fba7000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:55.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.445+0000 7fba75ffb640 1 -- 172.21.15.70:0/652568723 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba70019070 con 0x7fba78102970 2024-04-01T01:35:55.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.445+0000 7fba75ffb640 1 -- 172.21.15.70:0/652568723 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba7000cc10 con 0x7fba78102970 2024-04-01T01:35:55.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.445+0000 7fba7ef7a640 1 -- 172.21.15.70:0/652568723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba7819d800 con 0x7fba78102970 2024-04-01T01:35:55.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.446+0000 7fba75ffb640 1 -- 172.21.15.70:0/652568723 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba700075d0 con 0x7fba78102970 2024-04-01T01:35:55.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.446+0000 7fba7ef7a640 1 -- 172.21.15.70:0/652568723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba7819dda0 con 0x7fba78102970 2024-04-01T01:35:55.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.447+0000 7fba7ef7a640 1 -- 172.21.15.70:0/652568723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba7819e060 con 0x7fba78102970 2024-04-01T01:35:55.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.449+0000 7fba75ffb640 1 -- 172.21.15.70:0/652568723 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fba7001c070 con 0x7fba78102970 2024-04-01T01:35:55.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.450+0000 7fba75ffb640 1 --2- 172.21.15.70:0/652568723 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba640a25f0 0x7fba640a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:55.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.450+0000 7fba75ffb640 1 -- 172.21.15.70:0/652568723 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fba700ce070 con 0x7fba78102970 2024-04-01T01:35:55.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.450+0000 7fba7cd16640 1 --2- 172.21.15.70:0/652568723 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba640a25f0 0x7fba640a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:55.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.452+0000 7fba7cd16640 1 --2- 172.21.15.70:0/652568723 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba640a25f0 0x7fba640a4ab0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fba680348e0 tx=0x7fba680349b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:55.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.458+0000 7fba75ffb640 1 -- 172.21.15.70:0/652568723 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fba7009a830 con 0x7fba78102970 2024-04-01T01:35:55.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.875+0000 7fba7ef7a640 1 -- 172.21.15.70:0/652568723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba7819e480 con 0x7fba78102970 2024-04-01T01:35:55.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.876+0000 7fba75ffb640 1 -- 172.21.15.70:0/652568723 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7fba7009e8d0 con 0x7fba78102970 2024-04-01T01:35:55.876 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:35:55.877 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:33:36.894838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24418,24404,24379,24451,24464]},"id":1}]} 2024-04-01T01:35:55.877 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-04-01T01:35:55.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.880+0000 7fba7ef7a640 1 -- 172.21.15.70:0/652568723 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba640a25f0 msgr2=0x7fba640a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:55.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.880+0000 7fba7ef7a640 1 --2- 172.21.15.70:0/652568723 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba640a25f0 0x7fba640a4ab0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fba680348e0 tx=0x7fba680349b0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.880+0000 7fba7ef7a640 1 -- 172.21.15.70:0/652568723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba78102970 msgr2=0x7fba78195dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:55.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.880+0000 7fba7ef7a640 1 --2- 172.21.15.70:0/652568723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba78102970 0x7fba78195dc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fba7000ba50 tx=0x7fba7000bf10 comp rx=0 tx=0).stop 2024-04-01T01:35:55.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.881+0000 7fba7ef7a640 1 -- 172.21.15.70:0/652568723 shutdown_connections 2024-04-01T01:35:55.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.881+0000 7fba7ef7a640 1 --2- 172.21.15.70:0/652568723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba78103330 0x7fba7819ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.881+0000 7fba7ef7a640 1 --2- 172.21.15.70:0/652568723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba78102970 0x7fba78195dc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.881+0000 7fba7ef7a640 1 --2- 172.21.15.70:0/652568723 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba640a25f0 0x7fba640a4ab0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.881+0000 7fba7ef7a640 1 --2- 172.21.15.70:0/652568723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba78101780 0x7fba78195880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:55.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.881+0000 7fba7ef7a640 1 -- 172.21.15.70:0/652568723 >> 172.21.15.70:0/652568723 conn(0x7fba780fccf0 msgr2=0x7fba78073980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:55.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.881+0000 7fba7ef7a640 1 -- 172.21.15.70:0/652568723 shutdown_connections 2024-04-01T01:35:55.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:55.881+0000 7fba7ef7a640 1 -- 172.21.15.70:0/652568723 wait complete. 2024-04-01T01:35:55.898 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:35:55.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:55 smithi189 ceph-mon[35753]: pgmap v431: 129 pgs: 129 active+clean; 970 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-04-01T01:35:56.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:55 smithi070 ceph-mon[32179]: pgmap v431: 129 pgs: 129 active+clean; 970 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-04-01T01:35:56.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.071+0000 7fd0b10df640 1 -- 172.21.15.70:0/3451620567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0a409d5b0 msgr2=0x7fd0a409da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.071+0000 7fd0b10df640 1 --2- 172.21.15.70:0/3451620567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0a409d5b0 0x7fd0a409da30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0009860 tx=0x7fd0a002bb70 comp rx=0 tx=0).stop 2024-04-01T01:35:56.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.071+0000 7fd0b10df640 1 -- 172.21.15.70:0/3451620567 shutdown_connections 2024-04-01T01:35:56.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.071+0000 7fd0b10df640 1 --2- 172.21.15.70:0/3451620567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0a409df70 0x7fd0a40a4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.071+0000 7fd0b10df640 1 --2- 172.21.15.70:0/3451620567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0a409d5b0 0x7fd0a409da30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.071+0000 7fd0b10df640 1 --2- 172.21.15.70:0/3451620567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0a409c3c0 0x7fd0a409c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.071+0000 7fd0b10df640 1 -- 172.21.15.70:0/3451620567 >> 172.21.15.70:0/3451620567 conn(0x7fd0a400b560 msgr2=0x7fd0a409a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.071+0000 7fd0b10df640 1 -- 172.21.15.70:0/3451620567 shutdown_connections 2024-04-01T01:35:56.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.071+0000 7fd0b10df640 1 -- 172.21.15.70:0/3451620567 wait complete. 2024-04-01T01:35:56.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.072+0000 7fd0b10df640 1 Processor -- start 2024-04-01T01:35:56.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.073+0000 7fd0b10df640 1 -- start start 2024-04-01T01:35:56.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.073+0000 7fd0b10df640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0a409c3c0 0x7fd0a4132610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.073+0000 7fd0b10df640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0a409d5b0 0x7fd0a4132b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.073+0000 7fd0b10df640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0a409df70 0x7fd0a4139bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.073+0000 7fd0aad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0a409c3c0 0x7fd0a4132610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.074+0000 7fd0aad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0a409c3c0 0x7fd0a4132610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33240/0 (socket says 172.21.15.70:33240) 2024-04-01T01:35:56.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.074+0000 7fd0b10df640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a40a7f50 con 0x7fd0a409d5b0 2024-04-01T01:35:56.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.074+0000 7fd0b10df640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a40a7dd0 con 0x7fd0a409c3c0 2024-04-01T01:35:56.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.074+0000 7fd0ab59e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0a409df70 0x7fd0a4139bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.074+0000 7fd0ab59e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0a409df70 0x7fd0a4139bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52644/0 (socket says 172.21.15.70:52644) 2024-04-01T01:35:56.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.074+0000 7fd0ab59e640 1 -- 172.21.15.70:0/1606194856 learned_addr learned my addr 172.21.15.70:0/1606194856 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.074+0000 7fd0b10df640 1 -- 172.21.15.70:0/1606194856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a40a80d0 con 0x7fd0a409df70 2024-04-01T01:35:56.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.074+0000 7fd0aa59c640 1 --2- 172.21.15.70:0/1606194856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0a409d5b0 0x7fd0a4132b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.074+0000 7fd0aa59c640 1 -- 172.21.15.70:0/1606194856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0a409df70 msgr2=0x7fd0a4139bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.074+0000 7fd0aa59c640 1 --2- 172.21.15.70:0/1606194856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0a409df70 0x7fd0a4139bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.074+0000 7fd0aa59c640 1 -- 172.21.15.70:0/1606194856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0a409c3c0 msgr2=0x7fd0a4132610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.074+0000 7fd0aa59c640 1 --2- 172.21.15.70:0/1606194856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0a409c3c0 0x7fd0a4132610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.074+0000 7fd0aa59c640 1 -- 172.21.15.70:0/1606194856 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a413a230 con 0x7fd0a409d5b0 2024-04-01T01:35:56.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.075+0000 7fd0aa59c640 1 --2- 172.21.15.70:0/1606194856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0a409d5b0 0x7fd0a4132b50 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0009780 tx=0x7fd0a0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.075+0000 7fd0ab59e640 1 --2- 172.21.15.70:0/1606194856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0a409df70 0x7fd0a4139bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:35:56.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.075+0000 7fd0aad9d640 1 --2- 172.21.15.70:0/1606194856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0a409c3c0 0x7fd0a4132610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:35:56.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.076+0000 7fd093fff640 1 -- 172.21.15.70:0/1606194856 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0039040 con 0x7fd0a409d5b0 2024-04-01T01:35:56.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.076+0000 7fd093fff640 1 -- 172.21.15.70:0/1606194856 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0a0042070 con 0x7fd0a409d5b0 2024-04-01T01:35:56.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.076+0000 7fd093fff640 1 -- 172.21.15.70:0/1606194856 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0004920 con 0x7fd0a409d5b0 2024-04-01T01:35:56.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.076+0000 7fd0b10df640 1 -- 172.21.15.70:0/1606194856 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0a413a4c0 con 0x7fd0a409d5b0 2024-04-01T01:35:56.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.077+0000 7fd0b10df640 1 -- 172.21.15.70:0/1606194856 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0a413a960 con 0x7fd0a409d5b0 2024-04-01T01:35:56.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:55 smithi081 ceph-mon[35142]: pgmap v431: 129 pgs: 129 active+clean; 970 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 1.23k op/s 2024-04-01T01:35:56.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.080+0000 7fd0b3d9d740 1 Processor -- start 2024-04-01T01:35:56.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.080+0000 7fd0b3d9d740 1 -- start start 2024-04-01T01:35:56.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.081+0000 7fd0b3d9d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a36a0 0x55b6332a3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.081+0000 7fd0b3d9d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a4090 0x55b6332a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.081+0000 7fd0b3d9d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a9200 0x55b6332ab690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.081+0000 7fd0b3d9d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b6332081b0 con 0x55b6332a36a0 2024-04-01T01:35:56.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.081+0000 7fd0b3d9d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b6331b0c00 con 0x55b6332a9200 2024-04-01T01:35:56.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.082+0000 7fd0b3d9d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b633204250 con 0x55b6332a4090 2024-04-01T01:35:56.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.081+0000 7fd0ab59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a36a0 0x55b6332a3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.082+0000 7fd0ab59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a36a0 0x55b6332a3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58602/0 (socket says 172.21.15.70:58602) 2024-04-01T01:35:56.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.082+0000 7fd0ab59e640 1 -- 172.21.15.70:0/3590477860 learned_addr learned my addr 172.21.15.70:0/3590477860 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.083+0000 7fd0aa59c640 1 --2- 172.21.15.70:0/3590477860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a9200 0x55b6332ab690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.082+0000 7fd0aad9d640 1 --2- 172.21.15.70:0/3590477860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a4090 0x55b6332a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.083+0000 7fd0ab59e640 1 -- 172.21.15.70:0/3590477860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a4090 msgr2=0x55b6332a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.083+0000 7fd093fff640 1 -- 172.21.15.70:0/1606194856 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fd0a00050e0 con 0x7fd0a409d5b0 2024-04-01T01:35:56.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.083+0000 7fd0ab59e640 1 --2- 172.21.15.70:0/3590477860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a4090 0x55b6332a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.083+0000 7fd0ab59e640 1 -- 172.21.15.70:0/3590477860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a9200 msgr2=0x55b6332ab690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.083+0000 7fd0ab59e640 1 --2- 172.21.15.70:0/3590477860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a9200 0x55b6332ab690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.084+0000 7fd0ab59e640 1 -- 172.21.15.70:0/3590477860 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6331d03d0 con 0x55b6332a36a0 2024-04-01T01:35:56.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.084+0000 7fd0aa59c640 1 --2- 172.21.15.70:0/3590477860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a9200 0x55b6332ab690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:35:56.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.084+0000 7fd093fff640 1 --2- 172.21.15.70:0/1606194856 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0940a26a0 0x7fd0940a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.084+0000 7fd0aad9d640 1 --2- 172.21.15.70:0/3590477860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a4090 0x55b6332a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:35:56.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.084+0000 7fd093fff640 1 -- 172.21.15.70:0/1606194856 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd0a00f0330 con 0x7fd0a409d5b0 2024-04-01T01:35:56.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.085+0000 7fd0ab59e640 1 --2- 172.21.15.70:0/3590477860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a36a0 0x55b6332a3ac0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac066f10 tx=0x7fd0ac050420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ed62e4243d5400 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.085+0000 7fd0aad9d640 1 --2- 172.21.15.70:0/1606194856 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0940a26a0 0x7fd0940a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.086+0000 7fd0917fa640 1 -- 172.21.15.70:0/3590477860 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac06c070 con 0x55b6332a36a0 2024-04-01T01:35:56.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.086+0000 7fd0917fa640 1 -- 172.21.15.70:0/3590477860 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0ac06d040 con 0x55b6332a36a0 2024-04-01T01:35:56.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.087+0000 7fd0917fa640 1 -- 172.21.15.70:0/3590477860 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac0728c0 con 0x55b6332a36a0 2024-04-01T01:35:56.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.087+0000 7fd0aad9d640 1 --2- 172.21.15.70:0/1606194856 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0940a26a0 0x7fd0940a4b60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x55b6332a4490 tx=0x7fd09c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.087+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3590477860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a36a0 msgr2=0x55b6332a3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.087+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/3590477860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a36a0 0x55b6332a3ac0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac066f10 tx=0x7fd0ac050420 comp rx=0 tx=0).stop 2024-04-01T01:35:56.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.088+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3590477860 shutdown_connections 2024-04-01T01:35:56.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.088+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/3590477860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a9200 0x55b6332ab690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.088+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/3590477860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a4090 0x55b6332a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.088+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/3590477860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a36a0 0x55b6332a3ac0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.088+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3590477860 >> 172.21.15.70:0/3590477860 conn(0x55b63329ec10 msgr2=0x55b63329f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.088+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3590477860 shutdown_connections 2024-04-01T01:35:56.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.088+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3590477860 wait complete. 2024-04-01T01:35:56.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.088+0000 7fd0b3d9d740 1 -- start start 2024-04-01T01:35:56.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.089+0000 7fd0b3d9d740 1 Processor -- start 2024-04-01T01:35:56.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.090+0000 7fd0b3d9d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332bebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.090+0000 7fd0aad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332bebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.090+0000 7fd0aad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332bebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33248/0 (socket says 172.21.15.70:33248) 2024-04-01T01:35:56.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.091+0000 7fd0aad9d640 1 -- 172.21.15.70:0/3566882070 learned_addr learned my addr 172.21.15.70:0/3566882070 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.091+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/3566882070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a4090 0x55b6332bf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.091+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/3566882070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a9200 0x55b6332c6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.091+0000 7fd0aa59c640 1 --2- 172.21.15.70:0/3566882070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a4090 0x55b6332bf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.091+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3566882070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b6332bd1c0 con 0x55b6332a9200 2024-04-01T01:35:56.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.091+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3566882070 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b6331a6470 con 0x55b6332a36a0 2024-04-01T01:35:56.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.091+0000 7fd0ab59e640 1 --2- 172.21.15.70:0/3566882070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a9200 0x55b6332c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.091+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3566882070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b6332bd340 con 0x55b6332a4090 2024-04-01T01:35:56.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.092+0000 7fd0b3d9d740 10 client.0 _ll_register_callbacks cb 0x55b6331c3cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:56.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.092+0000 7fd0aad9d640 1 -- 172.21.15.70:0/3566882070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a4090 msgr2=0x55b6332bf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.092+0000 7fd0aad9d640 1 --2- 172.21.15.70:0/3566882070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a4090 0x55b6332bf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.092+0000 7fd0aad9d640 1 -- 172.21.15.70:0/3566882070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a9200 msgr2=0x55b6332c6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.092+0000 7fd0aad9d640 1 --2- 172.21.15.70:0/3566882070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a9200 0x55b6332c6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.092+0000 7fd0aad9d640 1 -- 172.21.15.70:0/3566882070 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b632f1c3f0 con 0x55b6332a36a0 2024-04-01T01:35:56.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.093+0000 7fd0aad9d640 1 --2- 172.21.15.70:0/3566882070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332bebd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd09c01eb70 tx=0x7fd09c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2521cdd6081aab3f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.093+0000 7fd08eff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:56.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.093+0000 7fd08eff5640 1 -- 172.21.15.70:0/3566882070 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c010ae0 con 0x55b6332a36a0 2024-04-01T01:35:56.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.093+0000 7fd08eff5640 1 -- 172.21.15.70:0/3566882070 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd09c010c80 con 0x55b6332a36a0 2024-04-01T01:35:56.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.094+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3566882070 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0ac06d040 con 0x55b6332a36a0 2024-04-01T01:35:56.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.095+0000 7fd08eff5640 1 -- 172.21.15.70:0/3566882070 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd09c005020 con 0x55b6332a36a0 2024-04-01T01:35:56.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.095+0000 7fd0b3d9d740 10 client.26123 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:56.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.095+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3566882070 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b632fe67d0 con 0x55b6332a36a0 2024-04-01T01:35:56.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.095+0000 7fd08eff5640 1 -- 172.21.15.70:0/3566882070 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd09c0047a0 con 0x55b6332a36a0 2024-04-01T01:35:56.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.096+0000 7fd08eff5640 1 -- 172.21.15.70:0/3566882070 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd09c00f040 con 0x55b6332a36a0 2024-04-01T01:35:56.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.096+0000 7fd0b3d9d740 10 client.26123 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:56.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.096+0000 7fd0b3d9d740 10 client.26123 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:35:56.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.096+0000 7fd0b3d9d740 20 client.26123 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:56.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.096+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/3566882070 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b6332caa50 0x55b6332cced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.096+0000 7fd0b3d9d740 4 client.26123 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:56.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.096+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3566882070 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd0ac06d040 con 0x55b6332caa50 2024-04-01T01:35:56.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.097+0000 7fd0aa59c640 1 --2- 172.21.15.70:0/3566882070 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b6332caa50 0x55b6332cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.097+0000 7fd08eff5640 10 client.26123 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:35:56.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.097+0000 7fd0aa59c640 1 --2- 172.21.15.70:0/3566882070 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b6332caa50 0x55b6332cced0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ed3bdc80b527441f server_cookie=db24bbfda4d4c2b3 in_seq=0 out_seq=0 2024-04-01T01:35:56.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.098+0000 7fd08eff5640 1 -- 172.21.15.70:0/3566882070 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fd0ac06d040 con 0x55b6332caa50 2024-04-01T01:35:56.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.098+0000 7fd08eff5640 10 client.26123 handle_command_reply: tid=0 2024-04-01T01:35:56.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.098+0000 7fd0b3d9d740 1 client.26123 shutdown 2024-04-01T01:35:56.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.099+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3566882070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 msgr2=0x55b6332bebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.099+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/3566882070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332bebd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd09c01eb70 tx=0x7fd09c0046c0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.099+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3566882070 shutdown_connections 2024-04-01T01:35:56.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.099+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/3566882070 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b6332caa50 0x55b6332cced0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.099+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/3566882070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a9200 0x55b6332c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.099+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/3566882070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a4090 0x55b6332bf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.099+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/3566882070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332bebd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.099+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3566882070 >> 172.21.15.70:0/3566882070 conn(0x55b63329ec10 msgr2=0x55b63329df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.099+0000 7fd08eff5640 0 client.26123 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:35:56.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.099+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3566882070 shutdown_connections 2024-04-01T01:35:56.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.099+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3566882070 wait complete. 2024-04-01T01:35:56.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.100+0000 7fd0b3d9d740 20 client.26123 trim_cache size 0 max 16384 2024-04-01T01:35:56.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.124+0000 7fd0b3d9d740 1 Processor -- start 2024-04-01T01:35:56.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.125+0000 7fd0b3d9d740 1 -- start start 2024-04-01T01:35:56.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.125+0000 7fd0b3d9d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332a5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.125+0000 7fd0b3d9d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a4090 0x55b6332a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.125+0000 7fd0b3d9d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a9200 0x55b6332c6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.125+0000 7fd0b3d9d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a4145170 con 0x55b6332a9200 2024-04-01T01:35:56.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.125+0000 7fd0b3d9d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a4146cb0 con 0x55b6332a36a0 2024-04-01T01:35:56.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.125+0000 7fd0b3d9d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a4145840 con 0x55b6332a4090 2024-04-01T01:35:56.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.125+0000 7fd0aa59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a4090 0x55b6332a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.125+0000 7fd0ab59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a9200 0x55b6332c6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.125+0000 7fd0ab59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a9200 0x55b6332c6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58628/0 (socket says 172.21.15.70:58628) 2024-04-01T01:35:56.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.125+0000 7fd0ab59e640 1 -- 172.21.15.70:0/3627517533 learned_addr learned my addr 172.21.15.70:0/3627517533 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.125+0000 7fd0aad9d640 1 --2- 172.21.15.70:0/3627517533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.126+0000 7fd0aad9d640 1 -- 172.21.15.70:0/3627517533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a4090 msgr2=0x55b6332a5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.126+0000 7fd0aad9d640 1 --2- 172.21.15.70:0/3627517533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a4090 0x55b6332a5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.126+0000 7fd0aad9d640 1 -- 172.21.15.70:0/3627517533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a9200 msgr2=0x55b6332c6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.126+0000 7fd0aad9d640 1 --2- 172.21.15.70:0/3627517533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a9200 0x55b6332c6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.126+0000 7fd0aad9d640 1 -- 172.21.15.70:0/3627517533 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b63328fe60 con 0x55b6332a36a0 2024-04-01T01:35:56.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.126+0000 7fd0ab59e640 1 --2- 172.21.15.70:0/3627517533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a9200 0x55b6332c6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:56.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.126+0000 7fd0aad9d640 1 --2- 172.21.15.70:0/3627517533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332a5550 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd09c005e00 tx=0x7fd09c0058f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aadd48071fd5237a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.127+0000 7fd090ff9640 1 -- 172.21.15.70:0/3627517533 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c0053c0 con 0x55b6332a36a0 2024-04-01T01:35:56.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.127+0000 7fd090ff9640 1 -- 172.21.15.70:0/3627517533 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd09c035070 con 0x55b6332a36a0 2024-04-01T01:35:56.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.127+0000 7fd090ff9640 1 -- 172.21.15.70:0/3627517533 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c01fd70 con 0x55b6332a36a0 2024-04-01T01:35:56.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.128+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3627517533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 msgr2=0x55b6332a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.128+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/3627517533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332a5550 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd09c005e00 tx=0x7fd09c0058f0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.128+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3627517533 shutdown_connections 2024-04-01T01:35:56.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.128+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/3627517533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a9200 0x55b6332c6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.128+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/3627517533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a4090 0x55b6332a5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.128+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/3627517533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332a5550 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.128+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3627517533 >> 172.21.15.70:0/3627517533 conn(0x55b63329ec10 msgr2=0x55b633299e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.128+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3627517533 shutdown_connections 2024-04-01T01:35:56.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.128+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/3627517533 wait complete. 2024-04-01T01:35:56.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.129+0000 7fd0b3d9d740 1 -- start start 2024-04-01T01:35:56.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.129+0000 7fd0b3d9d740 1 Processor -- start 2024-04-01T01:35:56.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.129+0000 7fd0b3d9d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332c3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.129+0000 7fd0b3d9d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a4090 0x55b6332c3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.129+0000 7fd0b3d9d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a9200 0x55b6332c4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.129+0000 7fd0b3d9d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a41465e0 con 0x55b6332a4090 2024-04-01T01:35:56.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.129+0000 7fd0b3d9d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a4145f10 con 0x55b6332a36a0 2024-04-01T01:35:56.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.129+0000 7fd0b3d9d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a4143150 con 0x55b6332a9200 2024-04-01T01:35:56.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.129+0000 7fd0b3d9d740 10 client.0 _ll_register_callbacks cb 0x7fd09c01e820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:56.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.129+0000 7fd0aad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332c3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.129+0000 7fd0aad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332c3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33268/0 (socket says 172.21.15.70:33268) 2024-04-01T01:35:56.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.129+0000 7fd0aad9d640 1 -- 172.21.15.70:0/51069548 learned_addr learned my addr 172.21.15.70:0/51069548 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.129+0000 7fd0aa59c640 1 --2- 172.21.15.70:0/51069548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a4090 0x55b6332c3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.129+0000 7fd0ab59e640 1 --2- 172.21.15.70:0/51069548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a9200 0x55b6332c4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.130+0000 7fd0aad9d640 1 -- 172.21.15.70:0/51069548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a9200 msgr2=0x55b6332c4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.130+0000 7fd0aad9d640 1 --2- 172.21.15.70:0/51069548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a9200 0x55b6332c4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.130+0000 7fd0aad9d640 1 -- 172.21.15.70:0/51069548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a4090 msgr2=0x55b6332c3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.130+0000 7fd0aad9d640 1 --2- 172.21.15.70:0/51069548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a4090 0x55b6332c3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.130+0000 7fd0aad9d640 1 -- 172.21.15.70:0/51069548 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6332c5090 con 0x55b6332a36a0 2024-04-01T01:35:56.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.130+0000 7fd0aa59c640 1 --2- 172.21.15.70:0/51069548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a4090 0x55b6332c3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:35:56.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.130+0000 7fd0aad9d640 1 --2- 172.21.15.70:0/51069548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332c3550 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0109e0 tx=0x7fd09c010a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6eb4786a59f38c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.130+0000 7fd08eff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:56.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.130+0000 7fd08eff5640 1 -- 172.21.15.70:0/51069548 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c041020 con 0x55b6332a36a0 2024-04-01T01:35:56.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.130+0000 7fd08eff5640 1 -- 172.21.15.70:0/51069548 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd09c01fa70 con 0x55b6332a36a0 2024-04-01T01:35:56.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.130+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/51069548 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd09c035070 con 0x55b6332a36a0 2024-04-01T01:35:56.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.130+0000 7fd08eff5640 1 -- 172.21.15.70:0/51069548 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd09c04c050 con 0x55b6332a36a0 2024-04-01T01:35:56.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.130+0000 7fd0b3d9d740 10 client.26129 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:56.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.130+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/51069548 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b6332c6390 con 0x55b6332a36a0 2024-04-01T01:35:56.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.131+0000 7fd08eff5640 1 -- 172.21.15.70:0/51069548 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd09c035070 con 0x55b6332a36a0 2024-04-01T01:35:56.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.131+0000 7fd08eff5640 1 -- 172.21.15.70:0/51069548 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd09c031480 con 0x55b6332a36a0 2024-04-01T01:35:56.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.131+0000 7fd0b3d9d740 10 client.26129 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:56.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.131+0000 7fd0b3d9d740 10 client.26129 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:35:56.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.131+0000 7fd0b3d9d740 20 client.26129 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:56.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.131+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/51069548 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b633331480 0x55b6333338a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.131+0000 7fd0b3d9d740 4 client.26129 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:35:56.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.131+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/51069548 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b6332156f0 con 0x55b633331480 2024-04-01T01:35:56.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.132+0000 7fd0aa59c640 1 --2- 172.21.15.70:0/51069548 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b633331480 0x55b6333338a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.132+0000 7fd08eff5640 10 client.26129 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:35:56.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.132+0000 7fd0aa59c640 1 --2- 172.21.15.70:0/51069548 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b633331480 0x55b6333338a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a5103ab784bf960c server_cookie=5c3065fd6a2f98f0 in_seq=0 out_seq=0 2024-04-01T01:35:56.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.133+0000 7fd08eff5640 1 -- 172.21.15.70:0/51069548 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b6332156f0 con 0x55b633331480 2024-04-01T01:35:56.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.133+0000 7fd08eff5640 10 client.26129 handle_command_reply: tid=0 2024-04-01T01:35:56.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.133+0000 7fd0b3d9d740 1 client.26129 shutdown 2024-04-01T01:35:56.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.133+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/51069548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 msgr2=0x55b6332c3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.133+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/51069548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332c3550 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0109e0 tx=0x7fd09c010a10 comp rx=0 tx=0).stop 2024-04-01T01:35:56.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.133+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/51069548 shutdown_connections 2024-04-01T01:35:56.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.133+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/51069548 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b633331480 0x55b6333338a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.133+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/51069548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6332a9200 0x55b6332c4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.133+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/51069548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b6332a4090 0x55b6332c3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.133+0000 7fd0b3d9d740 1 --2- 172.21.15.70:0/51069548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b6332a36a0 0x55b6332c3550 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.133+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/51069548 >> 172.21.15.70:0/51069548 conn(0x55b63329ec10 msgr2=0x55b6332a5550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.133+0000 7fd08eff5640 0 client.26129 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:35:56.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.133+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/51069548 shutdown_connections 2024-04-01T01:35:56.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.133+0000 7fd0b3d9d740 1 -- 172.21.15.70:0/51069548 wait complete. 2024-04-01T01:35:56.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.133+0000 7fd0b3d9d740 20 client.26129 trim_cache size 0 max 16384 2024-04-01T01:35:56.137 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:35:56.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.135+0000 7fd0b10df640 1 -- 172.21.15.70:0/1606194856 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0940a26a0 msgr2=0x7fd0940a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.135+0000 7fd0b10df640 1 --2- 172.21.15.70:0/1606194856 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0940a26a0 0x7fd0940a4b60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x55b6332a4490 tx=0x7fd09c005f90 comp rx=0 tx=0).stop 2024-04-01T01:35:56.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.135+0000 7fd0b10df640 1 -- 172.21.15.70:0/1606194856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0a409d5b0 msgr2=0x7fd0a4132b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.135+0000 7fd0b10df640 1 --2- 172.21.15.70:0/1606194856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0a409d5b0 0x7fd0a4132b50 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0009780 tx=0x7fd0a0004270 comp rx=0 tx=0).stop 2024-04-01T01:35:56.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.135+0000 7fd0b10df640 1 -- 172.21.15.70:0/1606194856 shutdown_connections 2024-04-01T01:35:56.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.135+0000 7fd0b10df640 1 --2- 172.21.15.70:0/1606194856 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0940a26a0 0x7fd0940a4b60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.135+0000 7fd0b10df640 1 --2- 172.21.15.70:0/1606194856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0a409df70 0x7fd0a4139bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.135+0000 7fd0b10df640 1 --2- 172.21.15.70:0/1606194856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0a409d5b0 0x7fd0a4132b50 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.135+0000 7fd0b10df640 1 --2- 172.21.15.70:0/1606194856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0a409c3c0 0x7fd0a4132610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.135+0000 7fd0b10df640 1 -- 172.21.15.70:0/1606194856 >> 172.21.15.70:0/1606194856 conn(0x7fd0a400b560 msgr2=0x7fd0a40a0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.136+0000 7fd0b10df640 1 -- 172.21.15.70:0/1606194856 shutdown_connections 2024-04-01T01:35:56.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.136+0000 7fd0b10df640 1 -- 172.21.15.70:0/1606194856 wait complete. 2024-04-01T01:35:56.153 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:35:56.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.314+0000 7fb5a690d640 1 -- 172.21.15.70:0/215716318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5a0104b60 msgr2=0x7fb5a0104fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.314+0000 7fb5a690d640 1 --2- 172.21.15.70:0/215716318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5a0104b60 0x7fb5a0104fe0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb598009950 tx=0x7fb59802bca0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.315+0000 7fb5a690d640 1 -- 172.21.15.70:0/215716318 shutdown_connections 2024-04-01T01:35:56.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.315+0000 7fb5a690d640 1 --2- 172.21.15.70:0/215716318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5a0105520 0x7fb5a010bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.315+0000 7fb5a690d640 1 --2- 172.21.15.70:0/215716318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5a0104b60 0x7fb5a0104fe0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.315+0000 7fb5a690d640 1 --2- 172.21.15.70:0/215716318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5a0103970 0x7fb5a0103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.315+0000 7fb5a690d640 1 -- 172.21.15.70:0/215716318 >> 172.21.15.70:0/215716318 conn(0x7fb5a00fef00 msgr2=0x7fb5a0101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.315+0000 7fb5a690d640 1 -- 172.21.15.70:0/215716318 shutdown_connections 2024-04-01T01:35:56.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.315+0000 7fb5a690d640 1 -- 172.21.15.70:0/215716318 wait complete. 2024-04-01T01:35:56.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.316+0000 7fb5a690d640 1 Processor -- start 2024-04-01T01:35:56.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.316+0000 7fb5a690d640 1 -- start start 2024-04-01T01:35:56.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.317+0000 7fb5a690d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5a0103970 0x7fb5a019e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.317+0000 7fb5a690d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5a0105520 0x7fb5a019e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.317+0000 7fb5a690d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5a01a37b0 0x7fb5a01a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.317+0000 7fb5a690d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a010f520 con 0x7fb5a01a37b0 2024-04-01T01:35:56.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.317+0000 7fb5a690d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a010f3a0 con 0x7fb5a0103970 2024-04-01T01:35:56.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.317+0000 7fb5a690d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a010f6a0 con 0x7fb5a0105520 2024-04-01T01:35:56.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.317+0000 7fb5a4eaa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5a01a37b0 0x7fb5a01a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.317+0000 7fb59ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5a0103970 0x7fb5a019e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.317+0000 7fb5a4eaa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5a01a37b0 0x7fb5a01a5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58646/0 (socket says 172.21.15.70:58646) 2024-04-01T01:35:56.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.317+0000 7fb5a4eaa640 1 -- 172.21.15.70:0/353622594 learned_addr learned my addr 172.21.15.70:0/353622594 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.317+0000 7fb59f7fe640 1 --2- 172.21.15.70:0/353622594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5a0105520 0x7fb5a019e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.317+0000 7fb59ffff640 1 -- 172.21.15.70:0/353622594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5a0105520 msgr2=0x7fb5a019e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.317+0000 7fb59ffff640 1 --2- 172.21.15.70:0/353622594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5a0105520 0x7fb5a019e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.318+0000 7fb59ffff640 1 -- 172.21.15.70:0/353622594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5a01a37b0 msgr2=0x7fb5a01a5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.318+0000 7fb59ffff640 1 --2- 172.21.15.70:0/353622594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5a01a37b0 0x7fb5a01a5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.318+0000 7fb59ffff640 1 -- 172.21.15.70:0/353622594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a01a6210 con 0x7fb5a0103970 2024-04-01T01:35:56.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.318+0000 7fb59f7fe640 1 --2- 172.21.15.70:0/353622594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5a0105520 0x7fb5a019e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:35:56.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.318+0000 7fb59ffff640 1 --2- 172.21.15.70:0/353622594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5a0103970 0x7fb5a019e210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb588005fd0 tx=0x7fb58800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.318+0000 7fb59d7fa640 1 -- 172.21.15.70:0/353622594 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58800cd80 con 0x7fb5a0103970 2024-04-01T01:35:56.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.318+0000 7fb5a690d640 1 -- 172.21.15.70:0/353622594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5a01a6500 con 0x7fb5a0103970 2024-04-01T01:35:56.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.318+0000 7fb5a690d640 1 -- 172.21.15.70:0/353622594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5a01a6af0 con 0x7fb5a0103970 2024-04-01T01:35:56.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.318+0000 7fb59d7fa640 1 -- 172.21.15.70:0/353622594 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb588011920 con 0x7fb5a0103970 2024-04-01T01:35:56.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.318+0000 7fb59d7fa640 1 -- 172.21.15.70:0/353622594 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb58801aa20 con 0x7fb5a0103970 2024-04-01T01:35:56.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.322+0000 7fb59d7fa640 1 -- 172.21.15.70:0/353622594 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb588011ad0 con 0x7fb5a0103970 2024-04-01T01:35:56.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.322+0000 7fb5a95c7740 1 Processor -- start 2024-04-01T01:35:56.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.323+0000 7fb5a95c7740 1 -- start start 2024-04-01T01:35:56.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.323+0000 7fb5a95c7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366ce490 0x557f366ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.323+0000 7fb5a95c7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366cee80 0x557f366d3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.323+0000 7fb5a95c7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366d3ff0 0x557f366d6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.323+0000 7fb5a95c7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557f366331b0 con 0x557f366d3ff0 2024-04-01T01:35:56.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.323+0000 7fb5a95c7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x557f365dbc00 con 0x557f366cee80 2024-04-01T01:35:56.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.323+0000 7fb5a95c7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557f3662f250 con 0x557f366ce490 2024-04-01T01:35:56.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.323+0000 7fb59f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366d3ff0 0x557f366d6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.323+0000 7fb59f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366d3ff0 0x557f366d6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58654/0 (socket says 172.21.15.70:58654) 2024-04-01T01:35:56.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.323+0000 7fb59f7fe640 1 -- 172.21.15.70:0/3396578743 learned_addr learned my addr 172.21.15.70:0/3396578743 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.323+0000 7fb5a4eaa640 1 --2- 172.21.15.70:0/3396578743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366ce490 0x557f366ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.324+0000 7fb59ffff640 1 --2- 172.21.15.70:0/3396578743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366cee80 0x557f366d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.324+0000 7fb59d7fa640 1 --2- 172.21.15.70:0/353622594 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb5800a25a0 0x7fb5800a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.324+0000 7fb59f7fe640 1 -- 172.21.15.70:0/3396578743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366ce490 msgr2=0x557f366ce8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.324+0000 7fb59f7fe640 1 --2- 172.21.15.70:0/3396578743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366ce490 0x557f366ce8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.324+0000 7fb59f7fe640 1 -- 172.21.15.70:0/3396578743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366cee80 msgr2=0x557f366d3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.324+0000 7fb59f7fe640 1 --2- 172.21.15.70:0/3396578743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366cee80 0x557f366d3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.324+0000 7fb59f7fe640 1 -- 172.21.15.70:0/3396578743 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f365fb3d0 con 0x557f366d3ff0 2024-04-01T01:35:56.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.324+0000 7fb59ffff640 1 --2- 172.21.15.70:0/3396578743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366cee80 0x557f366d3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:56.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.324+0000 7fb5a4eaa640 1 --2- 172.21.15.70:0/353622594 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb5800a25a0 0x7fb5800a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.325+0000 7fb59f7fe640 1 --2- 172.21.15.70:0/3396578743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366d3ff0 0x557f366d6480 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb598036040 tx=0x7fb598005440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34745acc34199db8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.326+0000 7fb5a4eaa640 1 --2- 172.21.15.70:0/353622594 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb5800a25a0 0x7fb5800a4a60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb590015350 tx=0x7fb5900095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.326+0000 7fb59d7fa640 1 -- 172.21.15.70:0/353622594 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb5880ceef0 con 0x7fb5a0103970 2024-04-01T01:35:56.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.326+0000 7fb5967fc640 1 -- 172.21.15.70:0/3396578743 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598042070 con 0x557f366d3ff0 2024-04-01T01:35:56.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.326+0000 7fb5967fc640 1 -- 172.21.15.70:0/3396578743 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb59802cb80 con 0x557f366d3ff0 2024-04-01T01:35:56.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.327+0000 7fb5967fc640 1 -- 172.21.15.70:0/3396578743 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598005220 con 0x557f366d3ff0 2024-04-01T01:35:56.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.328+0000 7fb5a95c7740 1 -- 172.21.15.70:0/3396578743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366d3ff0 msgr2=0x557f366d6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.328+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/3396578743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366d3ff0 0x557f366d6480 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb598036040 tx=0x7fb598005440 comp rx=0 tx=0).stop 2024-04-01T01:35:56.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.328+0000 7fb5a95c7740 1 -- 172.21.15.70:0/3396578743 shutdown_connections 2024-04-01T01:35:56.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.328+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/3396578743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366d3ff0 0x557f366d6480 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.328+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/3396578743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366cee80 0x557f366d3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.328+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/3396578743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366ce490 0x557f366ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.328+0000 7fb5a95c7740 1 -- 172.21.15.70:0/3396578743 >> 172.21.15.70:0/3396578743 conn(0x557f366c9a00 msgr2=0x557f366c9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.328+0000 7fb5a95c7740 1 -- 172.21.15.70:0/3396578743 shutdown_connections 2024-04-01T01:35:56.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.328+0000 7fb5a95c7740 1 -- 172.21.15.70:0/3396578743 wait complete. 2024-04-01T01:35:56.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.329+0000 7fb5a95c7740 1 -- start start 2024-04-01T01:35:56.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.330+0000 7fb5a95c7740 1 Processor -- start 2024-04-01T01:35:56.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.330+0000 7fb5a95c7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 0x557f366d8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.330+0000 7fb5a95c7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366cee80 0x557f366d8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.330+0000 7fb5a95c7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366d3ff0 0x557f366dfee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.330+0000 7fb5a95c7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557f366d6f40 con 0x557f366ce490 2024-04-01T01:35:56.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.330+0000 7fb5a95c7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x557f365d1470 con 0x557f366d3ff0 2024-04-01T01:35:56.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.330+0000 7fb5a95c7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557f366d70c0 con 0x557f366cee80 2024-04-01T01:35:56.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.330+0000 7fb5a95c7740 10 client.0 _ll_register_callbacks cb 0x557f365eecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:56.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.330+0000 7fb5a4eaa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366d3ff0 0x557f366dfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.330+0000 7fb5a4eaa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366d3ff0 0x557f366dfee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33306/0 (socket says 172.21.15.70:33306) 2024-04-01T01:35:56.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.330+0000 7fb5a4eaa640 1 -- 172.21.15.70:0/78882456 learned_addr learned my addr 172.21.15.70:0/78882456 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.330+0000 7fb59f7fe640 1 --2- 172.21.15.70:0/78882456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366cee80 0x557f366d8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.330+0000 7fb59ffff640 1 --2- 172.21.15.70:0/78882456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 0x557f366d8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.331+0000 7fb5a4eaa640 1 -- 172.21.15.70:0/78882456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366cee80 msgr2=0x557f366d8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.331+0000 7fb5a4eaa640 1 --2- 172.21.15.70:0/78882456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366cee80 0x557f366d8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.331+0000 7fb5a4eaa640 1 -- 172.21.15.70:0/78882456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 msgr2=0x557f366d8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.331+0000 7fb5a4eaa640 1 --2- 172.21.15.70:0/78882456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 0x557f366d8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.331+0000 7fb5a4eaa640 1 -- 172.21.15.70:0/78882456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f363473f0 con 0x557f366d3ff0 2024-04-01T01:35:56.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.331+0000 7fb59ffff640 1 --2- 172.21.15.70:0/78882456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 0x557f366d8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:56.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.331+0000 7fb5a4eaa640 1 --2- 172.21.15.70:0/78882456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366d3ff0 0x557f366dfee0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb590018d70 tx=0x7fb590009430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=705390e5bb4f0e4d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.332+0000 7fb58f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:56.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.332+0000 7fb58f7fe640 1 -- 172.21.15.70:0/78882456 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb590017cb0 con 0x557f366d3ff0 2024-04-01T01:35:56.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.332+0000 7fb58f7fe640 1 -- 172.21.15.70:0/78882456 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb59000f030 con 0x557f366d3ff0 2024-04-01T01:35:56.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.332+0000 7fb5a95c7740 1 -- 172.21.15.70:0/78882456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb59802cb80 con 0x557f366d3ff0 2024-04-01T01:35:56.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.333+0000 7fb58f7fe640 1 -- 172.21.15.70:0/78882456 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb59002e050 con 0x557f366d3ff0 2024-04-01T01:35:56.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.333+0000 7fb5a95c7740 10 client.26141 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:56.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.333+0000 7fb5a95c7740 1 -- 172.21.15.70:0/78882456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557f364117d0 con 0x557f366d3ff0 2024-04-01T01:35:56.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.333+0000 7fb58f7fe640 1 -- 172.21.15.70:0/78882456 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb59001c050 con 0x557f366d3ff0 2024-04-01T01:35:56.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.333+0000 7fb58f7fe640 1 -- 172.21.15.70:0/78882456 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb590006420 con 0x557f366d3ff0 2024-04-01T01:35:56.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.333+0000 7fb5a95c7740 10 client.26141 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:56.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.333+0000 7fb5a95c7740 10 client.26141 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:35:56.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.333+0000 7fb5a95c7740 20 client.26141 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:56.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.333+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/78882456 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x557f366e4490 0x557f366e68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.333+0000 7fb5a95c7740 4 client.26141 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:56.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.333+0000 7fb5a95c7740 1 -- 172.21.15.70:0/78882456 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb59802cb80 con 0x557f366e4490 2024-04-01T01:35:56.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.334+0000 7fb59ffff640 1 --2- 172.21.15.70:0/78882456 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x557f366e4490 0x557f366e68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.335+0000 7fb59ffff640 1 --2- 172.21.15.70:0/78882456 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x557f366e4490 0x557f366e68f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5a322761bca9f8b server_cookie=903c86a1aa65eead in_seq=0 out_seq=0 2024-04-01T01:35:56.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.335+0000 7fb58f7fe640 10 client.26141 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:35:56.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.337+0000 7fb58f7fe640 1 -- 172.21.15.70:0/78882456 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fb59802cb80 con 0x557f366e4490 2024-04-01T01:35:56.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.337+0000 7fb58f7fe640 10 client.26141 handle_command_reply: tid=0 2024-04-01T01:35:56.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.337+0000 7fb5a95c7740 1 client.26141 shutdown 2024-04-01T01:35:56.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.337+0000 7fb5a95c7740 1 -- 172.21.15.70:0/78882456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366d3ff0 msgr2=0x557f366dfee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.337+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/78882456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366d3ff0 0x557f366dfee0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb590018d70 tx=0x7fb590009430 comp rx=0 tx=0).stop 2024-04-01T01:35:56.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.337+0000 7fb5a95c7740 1 -- 172.21.15.70:0/78882456 shutdown_connections 2024-04-01T01:35:56.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.337+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/78882456 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x557f366e4490 0x557f366e68f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.337+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/78882456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366d3ff0 0x557f366dfee0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.337+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/78882456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366cee80 0x557f366d8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.337+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/78882456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 0x557f366d8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.337+0000 7fb5a95c7740 1 -- 172.21.15.70:0/78882456 >> 172.21.15.70:0/78882456 conn(0x557f366c9a00 msgr2=0x557f366c8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.338+0000 7fb58f7fe640 0 client.26141 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:35:56.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.338+0000 7fb5a95c7740 1 -- 172.21.15.70:0/78882456 shutdown_connections 2024-04-01T01:35:56.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.338+0000 7fb5a95c7740 1 -- 172.21.15.70:0/78882456 wait complete. 2024-04-01T01:35:56.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.338+0000 7fb5a95c7740 20 client.26141 trim_cache size 0 max 16384 2024-04-01T01:35:56.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.366+0000 7fb5a95c7740 1 Processor -- start 2024-04-01T01:35:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.366+0000 7fb5a95c7740 1 -- start start 2024-04-01T01:35:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.366+0000 7fb5a95c7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 0x557f366d63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.366+0000 7fb5a95c7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366cee80 0x557f366dfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.366+0000 7fb5a95c7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366d3ff0 0x557f366d8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.366+0000 7fb5a95c7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb58400fd90 con 0x557f366ce490 2024-04-01T01:35:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.366+0000 7fb5a95c7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb5840118d0 con 0x557f366d3ff0 2024-04-01T01:35:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.366+0000 7fb5a95c7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb584010460 con 0x557f366cee80 2024-04-01T01:35:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.366+0000 7fb5a4eaa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366d3ff0 0x557f366d8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.366+0000 7fb5a4eaa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366d3ff0 0x557f366d8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33318/0 (socket says 172.21.15.70:33318) 2024-04-01T01:35:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.366+0000 7fb5a4eaa640 1 -- 172.21.15.70:0/3920897572 learned_addr learned my addr 172.21.15.70:0/3920897572 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.367+0000 7fb59ffff640 1 --2- 172.21.15.70:0/3920897572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 0x557f366d63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.367+0000 7fb59f7fe640 1 --2- 172.21.15.70:0/3920897572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366cee80 0x557f366dfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.367+0000 7fb59ffff640 1 -- 172.21.15.70:0/3920897572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366cee80 msgr2=0x557f366dfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.367+0000 7fb59ffff640 1 --2- 172.21.15.70:0/3920897572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366cee80 0x557f366dfe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.367+0000 7fb59ffff640 1 -- 172.21.15.70:0/3920897572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366d3ff0 msgr2=0x557f366d8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.367+0000 7fb59ffff640 1 --2- 172.21.15.70:0/3920897572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366d3ff0 0x557f366d8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.367+0000 7fb59ffff640 1 -- 172.21.15.70:0/3920897572 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f366cfd80 con 0x557f366ce490 2024-04-01T01:35:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.368+0000 7fb59ffff640 1 --2- 172.21.15.70:0/3920897572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 0x557f366d63e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x557f366cf8f0 tx=0x7fb588086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae68d7f8110f8da server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.368+0000 7fb595ffb640 1 -- 172.21.15.70:0/3920897572 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb588011e60 con 0x557f366ce490 2024-04-01T01:35:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.368+0000 7fb595ffb640 1 -- 172.21.15.70:0/3920897572 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb58809bc80 con 0x557f366ce490 2024-04-01T01:35:56.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.368+0000 7fb595ffb640 1 -- 172.21.15.70:0/3920897572 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5880a3f90 con 0x557f366ce490 2024-04-01T01:35:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.369+0000 7fb5a95c7740 1 -- 172.21.15.70:0/3920897572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 msgr2=0x557f366d63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.369+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/3920897572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 0x557f366d63e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x557f366cf8f0 tx=0x7fb588086040 comp rx=0 tx=0).stop 2024-04-01T01:35:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.370+0000 7fb5a95c7740 1 -- 172.21.15.70:0/3920897572 shutdown_connections 2024-04-01T01:35:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.370+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/3920897572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366d3ff0 0x557f366d8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.370+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/3920897572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366cee80 0x557f366dfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.370+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/3920897572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 0x557f366d63e0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.370+0000 7fb5a95c7740 1 -- 172.21.15.70:0/3920897572 >> 172.21.15.70:0/3920897572 conn(0x557f366c9a00 msgr2=0x557f366c4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.370+0000 7fb5a95c7740 1 -- 172.21.15.70:0/3920897572 shutdown_connections 2024-04-01T01:35:56.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.370+0000 7fb5a95c7740 1 -- 172.21.15.70:0/3920897572 wait complete. 2024-04-01T01:35:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.370+0000 7fb5a95c7740 1 -- start start 2024-04-01T01:35:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.371+0000 7fb5a95c7740 1 Processor -- start 2024-04-01T01:35:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.371+0000 7fb5a95c7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 0x557f366dcee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.371+0000 7fb5a95c7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366cee80 0x557f366dd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.371+0000 7fb5a95c7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366d3ff0 0x557f366de430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.371+0000 7fb5a95c7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb584011200 con 0x557f366ce490 2024-04-01T01:35:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.371+0000 7fb5a95c7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb584010b30 con 0x557f366cee80 2024-04-01T01:35:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.371+0000 7fb5a95c7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb58400dd70 con 0x557f366d3ff0 2024-04-01T01:35:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.371+0000 7fb5a95c7740 10 client.0 _ll_register_callbacks cb 0x7fb590018790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:56.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.371+0000 7fb59f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366cee80 0x557f366dd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.371+0000 7fb59f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366cee80 0x557f366dd420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33332/0 (socket says 172.21.15.70:33332) 2024-04-01T01:35:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.371+0000 7fb59f7fe640 1 -- 172.21.15.70:0/348670187 learned_addr learned my addr 172.21.15.70:0/348670187 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.371+0000 7fb59ffff640 1 --2- 172.21.15.70:0/348670187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 0x557f366dcee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.372+0000 7fb59f7fe640 1 -- 172.21.15.70:0/348670187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366d3ff0 msgr2=0x557f366de430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.372+0000 7fb59f7fe640 1 --2- 172.21.15.70:0/348670187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366d3ff0 0x557f366de430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.372+0000 7fb59f7fe640 1 -- 172.21.15.70:0/348670187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 msgr2=0x557f366dcee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.372+0000 7fb59f7fe640 1 --2- 172.21.15.70:0/348670187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 0x557f366dcee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.372+0000 7fb59f7fe640 1 -- 172.21.15.70:0/348670187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f366dea20 con 0x557f366cee80 2024-04-01T01:35:56.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.372+0000 7fb59f7fe640 1 --2- 172.21.15.70:0/348670187 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366cee80 0x557f366dd420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb598009950 tx=0x7fb59802c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34d1d38c23e883f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.372+0000 7fb58f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:56.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.372+0000 7fb58f7fe640 1 -- 172.21.15.70:0/348670187 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598042040 con 0x557f366cee80 2024-04-01T01:35:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.372+0000 7fb58f7fe640 1 -- 172.21.15.70:0/348670187 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb598004220 con 0x557f366cee80 2024-04-01T01:35:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.373+0000 7fb5a95c7740 1 -- 172.21.15.70:0/348670187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb58809bc80 con 0x557f366cee80 2024-04-01T01:35:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.373+0000 7fb58f7fe640 1 -- 172.21.15.70:0/348670187 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb598057050 con 0x557f366cee80 2024-04-01T01:35:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.373+0000 7fb5a95c7740 10 client.26147 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.373+0000 7fb5a95c7740 1 -- 172.21.15.70:0/348670187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557f366dfd90 con 0x557f366cee80 2024-04-01T01:35:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.373+0000 7fb58f7fe640 1 -- 172.21.15.70:0/348670187 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb58809bc80 con 0x557f366cee80 2024-04-01T01:35:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.374+0000 7fb58f7fe640 1 -- 172.21.15.70:0/348670187 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb5980037b0 con 0x557f366cee80 2024-04-01T01:35:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.374+0000 7fb5a95c7740 10 client.26147 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.374+0000 7fb5a95c7740 10 client.26147 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:35:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.374+0000 7fb5a95c7740 20 client.26147 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.374+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/348670187 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x557f3674a830 0x557f3674cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.374+0000 7fb5a95c7740 4 client.26147 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:35:56.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.374+0000 7fb5a95c7740 1 -- 172.21.15.70:0/348670187 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557f366406f0 con 0x557f3674a830 2024-04-01T01:35:56.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.374+0000 7fb59ffff640 1 --2- 172.21.15.70:0/348670187 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x557f3674a830 0x557f3674cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.375+0000 7fb58f7fe640 10 client.26147 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:35:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.375+0000 7fb59ffff640 1 --2- 172.21.15.70:0/348670187 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x557f3674a830 0x557f3674cc20 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ecafb07d521d01b2 server_cookie=fab45c20434b1631 in_seq=0 out_seq=0 2024-04-01T01:35:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.375+0000 7fb58f7fe640 1 -- 172.21.15.70:0/348670187 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557f366406f0 con 0x557f3674a830 2024-04-01T01:35:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.375+0000 7fb58f7fe640 10 client.26147 handle_command_reply: tid=0 2024-04-01T01:35:56.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.375+0000 7fb5a95c7740 1 client.26147 shutdown 2024-04-01T01:35:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.376+0000 7fb5a95c7740 1 -- 172.21.15.70:0/348670187 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366cee80 msgr2=0x557f366dd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.376+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/348670187 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366cee80 0x557f366dd420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb598009950 tx=0x7fb59802c3b0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.376+0000 7fb5a95c7740 1 -- 172.21.15.70:0/348670187 shutdown_connections 2024-04-01T01:35:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.376+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/348670187 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x557f3674a830 0x557f3674cc20 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.376+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/348670187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557f366d3ff0 0x557f366de430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.376+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/348670187 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f366cee80 0x557f366dd420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.376+0000 7fb58f7fe640 0 client.26147 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:35:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.376+0000 7fb5a95c7740 1 --2- 172.21.15.70:0/348670187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557f366ce490 0x557f366dcee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.376+0000 7fb5a95c7740 1 -- 172.21.15.70:0/348670187 >> 172.21.15.70:0/348670187 conn(0x557f366c9a00 msgr2=0x557f366d8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.376+0000 7fb5a95c7740 1 -- 172.21.15.70:0/348670187 shutdown_connections 2024-04-01T01:35:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.376+0000 7fb5a95c7740 1 -- 172.21.15.70:0/348670187 wait complete. 2024-04-01T01:35:56.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.376+0000 7fb5a95c7740 20 client.26147 trim_cache size 0 max 16384 2024-04-01T01:35:56.379 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:35:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.377+0000 7fb5a690d640 1 -- 172.21.15.70:0/353622594 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb5800a25a0 msgr2=0x7fb5800a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.377+0000 7fb5a690d640 1 --2- 172.21.15.70:0/353622594 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb5800a25a0 0x7fb5800a4a60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb590015350 tx=0x7fb5900095c0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.377+0000 7fb5a690d640 1 -- 172.21.15.70:0/353622594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5a0103970 msgr2=0x7fb5a019e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.377+0000 7fb5a690d640 1 --2- 172.21.15.70:0/353622594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5a0103970 0x7fb5a019e210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb588005fd0 tx=0x7fb58800ee80 comp rx=0 tx=0).stop 2024-04-01T01:35:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.378+0000 7fb5a690d640 1 -- 172.21.15.70:0/353622594 shutdown_connections 2024-04-01T01:35:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.378+0000 7fb5a690d640 1 --2- 172.21.15.70:0/353622594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5a01a37b0 0x7fb5a01a5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.378+0000 7fb5a690d640 1 --2- 172.21.15.70:0/353622594 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb5800a25a0 0x7fb5800a4a60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.378+0000 7fb5a690d640 1 --2- 172.21.15.70:0/353622594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5a0105520 0x7fb5a019e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.378+0000 7fb5a690d640 1 --2- 172.21.15.70:0/353622594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5a0103970 0x7fb5a019e210 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.378+0000 7fb5a690d640 1 -- 172.21.15.70:0/353622594 >> 172.21.15.70:0/353622594 conn(0x7fb5a00fef00 msgr2=0x7fb5a0077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.378+0000 7fb5a690d640 1 -- 172.21.15.70:0/353622594 shutdown_connections 2024-04-01T01:35:56.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.378+0000 7fb5a690d640 1 -- 172.21.15.70:0/353622594 wait complete. 2024-04-01T01:35:56.403 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:35:56.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.593+0000 7fe684e10640 1 -- 172.21.15.70:0/2651228915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe680109820 msgr2=0x7fe6801100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.593+0000 7fe684e10640 1 --2- 172.21.15.70:0/2651228915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe680109820 0x7fe6801100b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe678009be0 tx=0x7fe67802bf10 comp rx=0 tx=0).stop 2024-04-01T01:35:56.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.593+0000 7fe684e10640 1 -- 172.21.15.70:0/2651228915 shutdown_connections 2024-04-01T01:35:56.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.594+0000 7fe684e10640 1 --2- 172.21.15.70:0/2651228915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe680109820 0x7fe6801100b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.594+0000 7fe684e10640 1 --2- 172.21.15.70:0/2651228915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe680108e60 0x7fe6801092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.594+0000 7fe684e10640 1 --2- 172.21.15.70:0/2651228915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe680107c70 0x7fe680108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.594+0000 7fe684e10640 1 -- 172.21.15.70:0/2651228915 >> 172.21.15.70:0/2651228915 conn(0x7fe680076c20 msgr2=0x7fe680079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.594+0000 7fe684e10640 1 -- 172.21.15.70:0/2651228915 shutdown_connections 2024-04-01T01:35:56.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.594+0000 7fe684e10640 1 -- 172.21.15.70:0/2651228915 wait complete. 2024-04-01T01:35:56.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.595+0000 7fe684e10640 1 Processor -- start 2024-04-01T01:35:56.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.595+0000 7fe684e10640 1 -- start start 2024-04-01T01:35:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.595+0000 7fe684e10640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe680107c70 0x7fe6800700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.595+0000 7fe684e10640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe680108e60 0x7fe680070600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.596+0000 7fe67f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe680107c70 0x7fe6800700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.596+0000 7fe67effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe680108e60 0x7fe680070600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.596+0000 7fe67effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe680108e60 0x7fe680070600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33356/0 (socket says 172.21.15.70:33356) 2024-04-01T01:35:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.596+0000 7fe67effd640 1 -- 172.21.15.70:0/4241946229 learned_addr learned my addr 172.21.15.70:0/4241946229 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.596+0000 7fe684e10640 1 --2- 172.21.15.70:0/4241946229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe680109820 0x7fe6801adee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.596+0000 7fe684e10640 1 -- 172.21.15.70:0/4241946229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe6801137d0 con 0x7fe680109820 2024-04-01T01:35:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.596+0000 7fe684e10640 1 -- 172.21.15.70:0/4241946229 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe680113650 con 0x7fe680108e60 2024-04-01T01:35:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.596+0000 7fe684e10640 1 -- 172.21.15.70:0/4241946229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe680113950 con 0x7fe680107c70 2024-04-01T01:35:56.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.597+0000 7fe67ffff640 1 --2- 172.21.15.70:0/4241946229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe680109820 0x7fe6801adee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.597+0000 7fe67f7fe640 1 -- 172.21.15.70:0/4241946229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe680108e60 msgr2=0x7fe680070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.597+0000 7fe67f7fe640 1 --2- 172.21.15.70:0/4241946229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe680108e60 0x7fe680070600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.597+0000 7fe67f7fe640 1 -- 172.21.15.70:0/4241946229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe680109820 msgr2=0x7fe6801adee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.597+0000 7fe67f7fe640 1 --2- 172.21.15.70:0/4241946229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe680109820 0x7fe6801adee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.597+0000 7fe67f7fe640 1 -- 172.21.15.70:0/4241946229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6801ae420 con 0x7fe680107c70 2024-04-01T01:35:56.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.597+0000 7fe67f7fe640 1 --2- 172.21.15.70:0/4241946229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe680107c70 0x7fe6800700c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe67000d6a0 tx=0x7fe67000db60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.598+0000 7fe67cff9640 1 -- 172.21.15.70:0/4241946229 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe670017070 con 0x7fe680107c70 2024-04-01T01:35:56.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.598+0000 7fe67cff9640 1 -- 172.21.15.70:0/4241946229 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe670009cd0 con 0x7fe680107c70 2024-04-01T01:35:56.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.598+0000 7fe67cff9640 1 -- 172.21.15.70:0/4241946229 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe670003b80 con 0x7fe680107c70 2024-04-01T01:35:56.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.598+0000 7fe684e10640 1 -- 172.21.15.70:0/4241946229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6801ae5f0 con 0x7fe680107c70 2024-04-01T01:35:56.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.599+0000 7fe684e10640 1 -- 172.21.15.70:0/4241946229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6801aea60 con 0x7fe680107c70 2024-04-01T01:35:56.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.603+0000 7fe67cff9640 1 -- 172.21.15.70:0/4241946229 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fe670003d20 con 0x7fe680107c70 2024-04-01T01:35:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.604+0000 7fe688c4e740 1 Processor -- start 2024-04-01T01:35:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.604+0000 7fe688c4e740 1 -- start start 2024-04-01T01:35:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.604+0000 7fe688c4e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff4476a0 0x55ccff447ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.605+0000 7fe688c4e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff448090 0x55ccff44cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.605+0000 7fe688c4e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff44d200 0x55ccff44f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.605+0000 7fe688c4e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ccff3ac1b0 con 0x55ccff4476a0 2024-04-01T01:35:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.605+0000 7fe688c4e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ccff354c00 con 0x55ccff448090 2024-04-01T01:35:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.605+0000 7fe688c4e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ccff3a8250 con 0x55ccff44d200 2024-04-01T01:35:56.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.605+0000 7fe67cff9640 1 --2- 172.21.15.70:0/4241946229 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6680a2670 0x7fe6680a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.605+0000 7fe67ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff4476a0 0x55ccff447ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.605+0000 7fe67ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff4476a0 0x55ccff447ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58696/0 (socket says 172.21.15.70:58696) 2024-04-01T01:35:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.605+0000 7fe67ffff640 1 -- 172.21.15.70:0/3845447177 learned_addr learned my addr 172.21.15.70:0/3845447177 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.605+0000 7fe67effd640 1 --2- 172.21.15.70:0/3845447177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff44d200 0x55ccff44f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.605+0000 7fe67f7fe640 1 --2- 172.21.15.70:0/3845447177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff448090 0x55ccff44cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.605+0000 7fe67cff9640 1 -- 172.21.15.70:0/4241946229 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe6700ce3f0 con 0x7fe680107c70 2024-04-01T01:35:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.606+0000 7fe67ffff640 1 --2- 172.21.15.70:0/4241946229 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6680a2670 0x7fe6680a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.606+0000 7fe67effd640 1 -- 172.21.15.70:0/3845447177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff448090 msgr2=0x55ccff44cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.606+0000 7fe67effd640 1 --2- 172.21.15.70:0/3845447177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff448090 0x55ccff44cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.606+0000 7fe67effd640 1 -- 172.21.15.70:0/3845447177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff4476a0 msgr2=0x55ccff447ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.606+0000 7fe67effd640 1 --2- 172.21.15.70:0/3845447177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff4476a0 0x55ccff447ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.606+0000 7fe67effd640 1 -- 172.21.15.70:0/3845447177 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccff3743d0 con 0x55ccff44d200 2024-04-01T01:35:56.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.606+0000 7fe67ffff640 1 --2- 172.21.15.70:0/4241946229 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6680a2670 0x7fe6680a4b30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe678004af0 tx=0x7fe678005e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.607+0000 7fe67effd640 1 --2- 172.21.15.70:0/3845447177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff44d200 0x55ccff44f690 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe674005fd0 tx=0x7fe67400bc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d95f43ae952f4b66 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.607+0000 7fe665ffb640 1 -- 172.21.15.70:0/3845447177 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674003da0 con 0x55ccff44d200 2024-04-01T01:35:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.607+0000 7fe665ffb640 1 -- 172.21.15.70:0/3845447177 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe674015070 con 0x55ccff44d200 2024-04-01T01:35:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.608+0000 7fe665ffb640 1 -- 172.21.15.70:0/3845447177 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6740116a0 con 0x55ccff44d200 2024-04-01T01:35:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.608+0000 7fe688c4e740 1 -- 172.21.15.70:0/3845447177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff44d200 msgr2=0x55ccff44f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.608+0000 7fe688c4e740 1 --2- 172.21.15.70:0/3845447177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff44d200 0x55ccff44f690 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe674005fd0 tx=0x7fe67400bc70 comp rx=0 tx=0).stop 2024-04-01T01:35:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.609+0000 7fe688c4e740 1 -- 172.21.15.70:0/3845447177 shutdown_connections 2024-04-01T01:35:56.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.609+0000 7fe688c4e740 1 --2- 172.21.15.70:0/3845447177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff44d200 0x55ccff44f690 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.609+0000 7fe688c4e740 1 --2- 172.21.15.70:0/3845447177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff448090 0x55ccff44cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.609+0000 7fe688c4e740 1 --2- 172.21.15.70:0/3845447177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff4476a0 0x55ccff447ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.609+0000 7fe688c4e740 1 -- 172.21.15.70:0/3845447177 >> 172.21.15.70:0/3845447177 conn(0x55ccff442c10 msgr2=0x55ccff443010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.609+0000 7fe688c4e740 1 -- 172.21.15.70:0/3845447177 shutdown_connections 2024-04-01T01:35:56.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.609+0000 7fe688c4e740 1 -- 172.21.15.70:0/3845447177 wait complete. 2024-04-01T01:35:56.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.610+0000 7fe688c4e740 1 -- start start 2024-04-01T01:35:56.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.610+0000 7fe688c4e740 1 Processor -- start 2024-04-01T01:35:56.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.611+0000 7fe688c4e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff4476a0 0x55ccff451af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.611+0000 7fe688c4e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff448090 0x55ccff452030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.611+0000 7fe688c4e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff44d200 0x55ccff459070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.611+0000 7fe67f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff4476a0 0x55ccff451af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.611+0000 7fe67f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff4476a0 0x55ccff451af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33364/0 (socket says 172.21.15.70:33364) 2024-04-01T01:35:56.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.611+0000 7fe67f7fe640 1 -- 172.21.15.70:0/3971572204 learned_addr learned my addr 172.21.15.70:0/3971572204 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.611+0000 7fe67effd640 1 --2- 172.21.15.70:0/3971572204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff448090 0x55ccff452030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.611+0000 7fe67ffff640 1 --2- 172.21.15.70:0/3971572204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff44d200 0x55ccff459070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.611+0000 7fe688c4e740 1 -- 172.21.15.70:0/3971572204 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ccff450250 con 0x55ccff44d200 2024-04-01T01:35:56.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.611+0000 7fe688c4e740 1 -- 172.21.15.70:0/3971572204 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ccff34a470 con 0x55ccff4476a0 2024-04-01T01:35:56.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.612+0000 7fe688c4e740 1 -- 172.21.15.70:0/3971572204 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ccff4503d0 con 0x55ccff448090 2024-04-01T01:35:56.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.612+0000 7fe67f7fe640 1 -- 172.21.15.70:0/3971572204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff448090 msgr2=0x55ccff452030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.612+0000 7fe67f7fe640 1 --2- 172.21.15.70:0/3971572204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff448090 0x55ccff452030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.612+0000 7fe67f7fe640 1 -- 172.21.15.70:0/3971572204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff44d200 msgr2=0x55ccff459070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.612+0000 7fe67f7fe640 1 --2- 172.21.15.70:0/3971572204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff44d200 0x55ccff459070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.612+0000 7fe67f7fe640 1 -- 172.21.15.70:0/3971572204 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccff0c03f0 con 0x55ccff4476a0 2024-04-01T01:35:56.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.612+0000 7fe688c4e740 10 client.0 _ll_register_callbacks cb 0x55ccff367cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:56.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.612+0000 7fe67f7fe640 1 --2- 172.21.15.70:0/3971572204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff4476a0 0x55ccff451af0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe670009610 tx=0x7fe670096e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39923c967947455b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.612+0000 7fe65effd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:56.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.612+0000 7fe65effd640 1 -- 172.21.15.70:0/3971572204 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6700b3060 con 0x55ccff4476a0 2024-04-01T01:35:56.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.612+0000 7fe688c4e740 1 -- 172.21.15.70:0/3971572204 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe674015070 con 0x55ccff4476a0 2024-04-01T01:35:56.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.613+0000 7fe65effd640 1 -- 172.21.15.70:0/3971572204 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe670004340 con 0x55ccff4476a0 2024-04-01T01:35:56.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.613+0000 7fe65effd640 1 -- 172.21.15.70:0/3971572204 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe670020040 con 0x55ccff4476a0 2024-04-01T01:35:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.613+0000 7fe688c4e740 10 client.26159 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.613+0000 7fe688c4e740 1 -- 172.21.15.70:0/3971572204 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55ccff18a7d0 con 0x55ccff4476a0 2024-04-01T01:35:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.613+0000 7fe65effd640 1 -- 172.21.15.70:0/3971572204 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe6700970f0 con 0x55ccff4476a0 2024-04-01T01:35:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.613+0000 7fe688c4e740 10 client.26159 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.613+0000 7fe688c4e740 10 client.26159 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:35:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.614+0000 7fe688c4e740 20 client.26159 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.614+0000 7fe688c4e740 1 --2- 172.21.15.70:0/3971572204 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55ccff45d300 0x55ccff45f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.614+0000 7fe688c4e740 4 client.26159 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.614+0000 7fe688c4e740 1 -- 172.21.15.70:0/3971572204 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe674015070 con 0x55ccff45d300 2024-04-01T01:35:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.614+0000 7fe67effd640 1 --2- 172.21.15.70:0/3971572204 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55ccff45d300 0x55ccff45f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.615+0000 7fe67effd640 1 --2- 172.21.15.70:0/3971572204 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55ccff45d300 0x55ccff45f7c0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=59e13a09bbcf47af server_cookie=ed48b749ccd39eec in_seq=0 out_seq=0 2024-04-01T01:35:56.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.615+0000 7fe65effd640 10 client.26159 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:35:56.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.616+0000 7fe65effd640 1 -- 172.21.15.70:0/3971572204 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fe674015070 con 0x55ccff45d300 2024-04-01T01:35:56.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.616+0000 7fe65effd640 10 client.26159 handle_command_reply: tid=0 2024-04-01T01:35:56.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.616+0000 7fe688c4e740 1 client.26159 shutdown 2024-04-01T01:35:56.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.616+0000 7fe688c4e740 1 -- 172.21.15.70:0/3971572204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff4476a0 msgr2=0x55ccff451af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.617+0000 7fe688c4e740 1 --2- 172.21.15.70:0/3971572204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff4476a0 0x55ccff451af0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe670009610 tx=0x7fe670096e20 comp rx=0 tx=0).stop 2024-04-01T01:35:56.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.618+0000 7fe688c4e740 1 -- 172.21.15.70:0/3971572204 shutdown_connections 2024-04-01T01:35:56.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.618+0000 7fe688c4e740 1 --2- 172.21.15.70:0/3971572204 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55ccff45d300 0x55ccff45f7c0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.618+0000 7fe688c4e740 1 --2- 172.21.15.70:0/3971572204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff44d200 0x55ccff459070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.618+0000 7fe688c4e740 1 --2- 172.21.15.70:0/3971572204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff448090 0x55ccff452030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.618+0000 7fe688c4e740 1 --2- 172.21.15.70:0/3971572204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff4476a0 0x55ccff451af0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.618+0000 7fe65effd640 0 client.26159 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:35:56.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.618+0000 7fe688c4e740 1 -- 172.21.15.70:0/3971572204 >> 172.21.15.70:0/3971572204 conn(0x55ccff442c10 msgr2=0x55ccff441f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.618+0000 7fe688c4e740 1 -- 172.21.15.70:0/3971572204 shutdown_connections 2024-04-01T01:35:56.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.619+0000 7fe688c4e740 1 -- 172.21.15.70:0/3971572204 wait complete. 2024-04-01T01:35:56.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.619+0000 7fe688c4e740 20 client.26159 trim_cache size 0 max 16384 2024-04-01T01:35:56.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.649+0000 7fe688c4e740 1 Processor -- start 2024-04-01T01:35:56.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.649+0000 7fe688c4e740 1 -- start start 2024-04-01T01:35:56.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.649+0000 7fe688c4e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff4476a0 0x55ccff451f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.649+0000 7fe688c4e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff448090 0x55ccff459000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.649+0000 7fe688c4e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff44d200 0x55ccff441700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.649+0000 7fe688c4e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe6801b8f30 con 0x55ccff44d200 2024-04-01T01:35:56.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.649+0000 7fe688c4e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe6801baa70 con 0x55ccff4476a0 2024-04-01T01:35:56.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.649+0000 7fe688c4e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe6801b9600 con 0x55ccff448090 2024-04-01T01:35:56.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.650+0000 7fe67effd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff448090 0x55ccff459000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.650+0000 7fe67effd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff448090 0x55ccff459000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49564/0 (socket says 172.21.15.70:49564) 2024-04-01T01:35:56.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.650+0000 7fe67effd640 1 -- 172.21.15.70:0/3296432455 learned_addr learned my addr 172.21.15.70:0/3296432455 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.650+0000 7fe67f7fe640 1 --2- 172.21.15.70:0/3296432455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff4476a0 0x55ccff451f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.650+0000 7fe67ffff640 1 --2- 172.21.15.70:0/3296432455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff44d200 0x55ccff441700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.650+0000 7fe67effd640 1 -- 172.21.15.70:0/3296432455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff4476a0 msgr2=0x55ccff451f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.650+0000 7fe67effd640 1 --2- 172.21.15.70:0/3296432455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff4476a0 0x55ccff451f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.650+0000 7fe67effd640 1 -- 172.21.15.70:0/3296432455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff44d200 msgr2=0x55ccff441700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.650+0000 7fe67effd640 1 --2- 172.21.15.70:0/3296432455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff44d200 0x55ccff441700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.650+0000 7fe67effd640 1 -- 172.21.15.70:0/3296432455 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccff433e60 con 0x55ccff448090 2024-04-01T01:35:56.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.650+0000 7fe67f7fe640 1 --2- 172.21.15.70:0/3296432455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff4476a0 0x55ccff451f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:35:56.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.650+0000 7fe67effd640 1 --2- 172.21.15.70:0/3296432455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff448090 0x55ccff459000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe674005fd0 tx=0x7fe67400b480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e27477950f18bb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.651+0000 7fe6657fa640 1 -- 172.21.15.70:0/3296432455 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674003d60 con 0x55ccff448090 2024-04-01T01:35:56.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.651+0000 7fe6657fa640 1 -- 172.21.15.70:0/3296432455 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe67402e070 con 0x55ccff448090 2024-04-01T01:35:56.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.651+0000 7fe6657fa640 1 -- 172.21.15.70:0/3296432455 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674029720 con 0x55ccff448090 2024-04-01T01:35:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.652+0000 7fe688c4e740 1 -- 172.21.15.70:0/3296432455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff448090 msgr2=0x55ccff459000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.652+0000 7fe688c4e740 1 --2- 172.21.15.70:0/3296432455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff448090 0x55ccff459000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe674005fd0 tx=0x7fe67400b480 comp rx=0 tx=0).stop 2024-04-01T01:35:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.652+0000 7fe688c4e740 1 -- 172.21.15.70:0/3296432455 shutdown_connections 2024-04-01T01:35:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.652+0000 7fe688c4e740 1 --2- 172.21.15.70:0/3296432455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff44d200 0x55ccff441700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.652+0000 7fe688c4e740 1 --2- 172.21.15.70:0/3296432455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff448090 0x55ccff459000 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.652+0000 7fe688c4e740 1 --2- 172.21.15.70:0/3296432455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff4476a0 0x55ccff451f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.652+0000 7fe688c4e740 1 -- 172.21.15.70:0/3296432455 >> 172.21.15.70:0/3296432455 conn(0x55ccff442c10 msgr2=0x55ccff43de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.653+0000 7fe688c4e740 1 -- 172.21.15.70:0/3296432455 shutdown_connections 2024-04-01T01:35:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.653+0000 7fe688c4e740 1 -- 172.21.15.70:0/3296432455 wait complete. 2024-04-01T01:35:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.653+0000 7fe688c4e740 1 -- start start 2024-04-01T01:35:56.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.654+0000 7fe688c4e740 1 Processor -- start 2024-04-01T01:35:56.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.654+0000 7fe688c4e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff4476a0 0x55ccff456b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.655+0000 7fe688c4e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff448090 0x55ccff457080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.655+0000 7fe67f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff4476a0 0x55ccff456b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.655+0000 7fe67f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff4476a0 0x55ccff456b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49578/0 (socket says 172.21.15.70:49578) 2024-04-01T01:35:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.655+0000 7fe67f7fe640 1 -- 172.21.15.70:0/1499772845 learned_addr learned my addr 172.21.15.70:0/1499772845 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.655+0000 7fe67effd640 1 --2- 172.21.15.70:0/1499772845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff448090 0x55ccff457080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.655+0000 7fe688c4e740 1 --2- 172.21.15.70:0/1499772845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff44d200 0x55ccff458090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.655+0000 7fe688c4e740 1 -- 172.21.15.70:0/1499772845 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe6801ba3a0 con 0x55ccff44d200 2024-04-01T01:35:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.656+0000 7fe688c4e740 1 -- 172.21.15.70:0/1499772845 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe6801b9cd0 con 0x55ccff448090 2024-04-01T01:35:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.656+0000 7fe688c4e740 1 -- 172.21.15.70:0/1499772845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe6801b6f10 con 0x55ccff4476a0 2024-04-01T01:35:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.656+0000 7fe67ffff640 1 --2- 172.21.15.70:0/1499772845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff44d200 0x55ccff458090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.656+0000 7fe67f7fe640 1 -- 172.21.15.70:0/1499772845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff448090 msgr2=0x55ccff457080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.656+0000 7fe67f7fe640 1 --2- 172.21.15.70:0/1499772845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff448090 0x55ccff457080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.656+0000 7fe67f7fe640 1 -- 172.21.15.70:0/1499772845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff44d200 msgr2=0x55ccff458090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.656+0000 7fe67f7fe640 1 --2- 172.21.15.70:0/1499772845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff44d200 0x55ccff458090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.656+0000 7fe67f7fe640 1 -- 172.21.15.70:0/1499772845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccff458650 con 0x55ccff4476a0 2024-04-01T01:35:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.656+0000 7fe688c4e740 10 client.0 _ll_register_callbacks cb 0x55ccff367cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:56.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.656+0000 7fe688c4e740 1 -- 172.21.15.70:0/1499772845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe67402e070 con 0x55ccff4476a0 2024-04-01T01:35:56.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.657+0000 7fe67f7fe640 1 --2- 172.21.15.70:0/1499772845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff4476a0 0x55ccff456b40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe6700cbb10 tx=0x7fe6700cb990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2d4369fe94ec0f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.658+0000 7fe65effd640 10 client.26059 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:35:56.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.658+0000 7fe65effd640 1 -- 172.21.15.70:0/1499772845 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6700a3490 con 0x55ccff4476a0 2024-04-01T01:35:56.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.658+0000 7fe65effd640 1 -- 172.21.15.70:0/1499772845 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe670098d80 con 0x55ccff4476a0 2024-04-01T01:35:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.658+0000 7fe65effd640 1 -- 172.21.15.70:0/1499772845 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe67009a0e0 con 0x55ccff4476a0 2024-04-01T01:35:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.658+0000 7fe688c4e740 10 client.26059 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.658+0000 7fe688c4e740 1 -- 172.21.15.70:0/1499772845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55ccff4c39a0 con 0x55ccff4476a0 2024-04-01T01:35:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.659+0000 7fe65effd640 1 -- 172.21.15.70:0/1499772845 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe6700967d0 con 0x55ccff4476a0 2024-04-01T01:35:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.659+0000 7fe688c4e740 10 client.26059 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.659+0000 7fe688c4e740 10 client.26059 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:35:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.659+0000 7fe688c4e740 20 client.26059 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.659+0000 7fe688c4e740 1 --2- 172.21.15.70:0/1499772845 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55ccff45d300 0x55ccff4c6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.659+0000 7fe688c4e740 4 client.26059 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:35:56.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.659+0000 7fe688c4e740 1 -- 172.21.15.70:0/1499772845 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ccff3b96f0 con 0x55ccff45d300 2024-04-01T01:35:56.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.661+0000 7fe67effd640 1 --2- 172.21.15.70:0/1499772845 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55ccff45d300 0x55ccff4c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.661+0000 7fe67effd640 1 --2- 172.21.15.70:0/1499772845 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55ccff45d300 0x55ccff4c6060 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d01e74e43ea1dbdf server_cookie=caec543c176e6dfb in_seq=0 out_seq=0 2024-04-01T01:35:56.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.661+0000 7fe65effd640 10 client.26059 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:35:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.662+0000 7fe65effd640 1 -- 172.21.15.70:0/1499772845 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ccff3b96f0 con 0x55ccff45d300 2024-04-01T01:35:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.662+0000 7fe65effd640 10 client.26059 handle_command_reply: tid=0 2024-04-01T01:35:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.662+0000 7fe688c4e740 1 client.26059 shutdown 2024-04-01T01:35:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.662+0000 7fe688c4e740 1 -- 172.21.15.70:0/1499772845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff4476a0 msgr2=0x55ccff456b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.662+0000 7fe688c4e740 1 --2- 172.21.15.70:0/1499772845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff4476a0 0x55ccff456b40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe6700cbb10 tx=0x7fe6700cb990 comp rx=0 tx=0).stop 2024-04-01T01:35:56.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.662+0000 7fe688c4e740 1 -- 172.21.15.70:0/1499772845 shutdown_connections 2024-04-01T01:35:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.662+0000 7fe688c4e740 1 --2- 172.21.15.70:0/1499772845 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55ccff45d300 0x55ccff4c6060 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.662+0000 7fe688c4e740 1 --2- 172.21.15.70:0/1499772845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccff44d200 0x55ccff458090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.662+0000 7fe65effd640 0 client.26059 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:35:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.662+0000 7fe688c4e740 1 --2- 172.21.15.70:0/1499772845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ccff448090 0x55ccff457080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.662+0000 7fe688c4e740 1 --2- 172.21.15.70:0/1499772845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ccff4476a0 0x55ccff456b40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.663+0000 7fe688c4e740 1 -- 172.21.15.70:0/1499772845 >> 172.21.15.70:0/1499772845 conn(0x55ccff442c10 msgr2=0x55ccff43de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.663+0000 7fe688c4e740 1 -- 172.21.15.70:0/1499772845 shutdown_connections 2024-04-01T01:35:56.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.663+0000 7fe688c4e740 1 -- 172.21.15.70:0/1499772845 wait complete. 2024-04-01T01:35:56.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.663+0000 7fe688c4e740 20 client.26059 trim_cache size 0 max 16384 2024-04-01T01:35:56.665 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:35:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.665+0000 7fe684e10640 1 -- 172.21.15.70:0/4241946229 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6680a2670 msgr2=0x7fe6680a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.665+0000 7fe684e10640 1 --2- 172.21.15.70:0/4241946229 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6680a2670 0x7fe6680a4b30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe678004af0 tx=0x7fe678005e70 comp rx=0 tx=0).stop 2024-04-01T01:35:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.665+0000 7fe684e10640 1 -- 172.21.15.70:0/4241946229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe680107c70 msgr2=0x7fe6800700c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.665+0000 7fe684e10640 1 --2- 172.21.15.70:0/4241946229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe680107c70 0x7fe6800700c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe67000d6a0 tx=0x7fe67000db60 comp rx=0 tx=0).stop 2024-04-01T01:35:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.665+0000 7fe684e10640 1 -- 172.21.15.70:0/4241946229 shutdown_connections 2024-04-01T01:35:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.665+0000 7fe684e10640 1 --2- 172.21.15.70:0/4241946229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe680109820 0x7fe6801adee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.665+0000 7fe684e10640 1 --2- 172.21.15.70:0/4241946229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe680108e60 0x7fe680070600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.665+0000 7fe684e10640 1 --2- 172.21.15.70:0/4241946229 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6680a2670 0x7fe6680a4b30 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.665+0000 7fe684e10640 1 --2- 172.21.15.70:0/4241946229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe680107c70 0x7fe6800700c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.665+0000 7fe684e10640 1 -- 172.21.15.70:0/4241946229 >> 172.21.15.70:0/4241946229 conn(0x7fe680076c20 msgr2=0x7fe68010c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.666+0000 7fe684e10640 1 -- 172.21.15.70:0/4241946229 shutdown_connections 2024-04-01T01:35:56.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.666+0000 7fe684e10640 1 -- 172.21.15.70:0/4241946229 wait complete. 2024-04-01T01:35:56.684 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:35:56.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.874+0000 7f6337fff640 1 -- 172.21.15.70:0/217493516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63300a6700 msgr2=0x7f63300aafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.874+0000 7f6337fff640 1 --2- 172.21.15.70:0/217493516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63300a6700 0x7f63300aafc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6338067420 tx=0x7f633808f4c0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.875+0000 7f6337fff640 1 -- 172.21.15.70:0/217493516 shutdown_connections 2024-04-01T01:35:56.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.875+0000 7f6337fff640 1 --2- 172.21.15.70:0/217493516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63300a6700 0x7f63300aafc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.875+0000 7f6337fff640 1 --2- 172.21.15.70:0/217493516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63300a5d60 0x7f63300a61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.875+0000 7f6337fff640 1 --2- 172.21.15.70:0/217493516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63300a4c30 0x7f63300a5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.875+0000 7f6337fff640 1 -- 172.21.15.70:0/217493516 >> 172.21.15.70:0/217493516 conn(0x7f63300a00c0 msgr2=0x7f63300a2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.876+0000 7f6337fff640 1 -- 172.21.15.70:0/217493516 shutdown_connections 2024-04-01T01:35:56.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.877+0000 7f6337fff640 1 -- 172.21.15.70:0/217493516 wait complete. 2024-04-01T01:35:56.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.878+0000 7f6337fff640 1 Processor -- start 2024-04-01T01:35:56.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.878+0000 7f6337fff640 1 -- start start 2024-04-01T01:35:56.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.878+0000 7f6337fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63300a4c30 0x7f63301434f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.879+0000 7f6337fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63300a5d60 0x7f6330143a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.879+0000 7f6337fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63300a6700 0x7f633014aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.879+0000 7f63367fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63300a5d60 0x7f6330143a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.879+0000 7f63367fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63300a5d60 0x7f6330143a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52668/0 (socket says 172.21.15.70:52668) 2024-04-01T01:35:56.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.879+0000 7f6336ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63300a4c30 0x7f63301434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.879+0000 7f6337fff640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f63300b8e30 con 0x7f63300a5d60 2024-04-01T01:35:56.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.879+0000 7f6337fff640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f63300b8cb0 con 0x7f63300a6700 2024-04-01T01:35:56.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.879+0000 7f63367fc640 1 -- 172.21.15.70:0/1452681202 learned_addr learned my addr 172.21.15.70:0/1452681202 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.879+0000 7f6337fff640 1 -- 172.21.15.70:0/1452681202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f63300b8fb0 con 0x7f63300a4c30 2024-04-01T01:35:56.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.879+0000 7f63377fe640 1 --2- 172.21.15.70:0/1452681202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63300a6700 0x7f633014aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.880+0000 7f63377fe640 1 -- 172.21.15.70:0/1452681202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63300a4c30 msgr2=0x7f63301434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.880+0000 7f63377fe640 1 --2- 172.21.15.70:0/1452681202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63300a4c30 0x7f63301434f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.880+0000 7f63377fe640 1 -- 172.21.15.70:0/1452681202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63300a5d60 msgr2=0x7f6330143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.880+0000 7f63377fe640 1 --2- 172.21.15.70:0/1452681202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63300a5d60 0x7f6330143a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.880+0000 7f63377fe640 1 -- 172.21.15.70:0/1452681202 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f633014b0c0 con 0x7f63300a6700 2024-04-01T01:35:56.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.881+0000 7f63377fe640 1 --2- 172.21.15.70:0/1452681202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63300a6700 0x7f633014aa50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6338000b60 tx=0x7f6338067c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.882+0000 7f631ffff640 1 -- 172.21.15.70:0/1452681202 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63380a8070 con 0x7f63300a6700 2024-04-01T01:35:56.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.882+0000 7f631ffff640 1 -- 172.21.15.70:0/1452681202 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f633809d040 con 0x7f63300a6700 2024-04-01T01:35:56.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.882+0000 7f631ffff640 1 -- 172.21.15.70:0/1452681202 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63380a3870 con 0x7f63300a6700 2024-04-01T01:35:56.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.882+0000 7f6337fff640 1 -- 172.21.15.70:0/1452681202 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f633014b350 con 0x7f63300a6700 2024-04-01T01:35:56.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.882+0000 7f6337fff640 1 -- 172.21.15.70:0/1452681202 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f633014b700 con 0x7f63300a6700 2024-04-01T01:35:56.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.885+0000 7f631ffff640 1 -- 172.21.15.70:0/1452681202 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f63380ab070 con 0x7f63300a6700 2024-04-01T01:35:56.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.886+0000 7f631ffff640 1 --2- 172.21.15.70:0/1452681202 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f63200a25f0 0x7f63200a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.886+0000 7f63401de740 1 Processor -- start 2024-04-01T01:35:56.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.886+0000 7f63401de740 1 -- start start 2024-04-01T01:35:56.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.886+0000 7f631ffff640 1 -- 172.21.15.70:0/1452681202 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f63381538c0 con 0x7f63300a6700 2024-04-01T01:35:56.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f6336ffd640 1 --2- 172.21.15.70:0/1452681202 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f63200a25f0 0x7f63200a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f63401de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f766a0 0x55bd64f76ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f63401de740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f77090 0x55bd64f7bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f63401de740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f7c200 0x55bd64f7e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f63401de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bd64edb1b0 con 0x55bd64f766a0 2024-04-01T01:35:56.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f63401de740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55bd64e83c00 con 0x55bd64f7c200 2024-04-01T01:35:56.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f63401de740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bd64ed7250 con 0x55bd64f77090 2024-04-01T01:35:56.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f63377fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f7c200 0x55bd64f7e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f63377fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f7c200 0x55bd64f7e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34708/0 (socket says 172.21.15.70:34708) 2024-04-01T01:35:56.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f63377fe640 1 -- 172.21.15.70:0/3471505593 learned_addr learned my addr 172.21.15.70:0/3471505593 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f6336ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f766a0 0x55bd64f76ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f63367fc640 1 --2- 172.21.15.70:0/3471505593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f77090 0x55bd64f7bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f63377fe640 1 -- 172.21.15.70:0/3471505593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f77090 msgr2=0x55bd64f7bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f63377fe640 1 --2- 172.21.15.70:0/3471505593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f77090 0x55bd64f7bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f63377fe640 1 -- 172.21.15.70:0/3471505593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f766a0 msgr2=0x55bd64f76ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f63377fe640 1 --2- 172.21.15.70:0/3471505593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f766a0 0x55bd64f76ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.887+0000 7f63377fe640 1 -- 172.21.15.70:0/3471505593 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd64ea33d0 con 0x55bd64f7c200 2024-04-01T01:35:56.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.888+0000 7f6336ffd640 1 --2- 172.21.15.70:0/3471505593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f766a0 0x55bd64f76ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:35:56.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.888+0000 7f6336ffd640 1 --2- 172.21.15.70:0/1452681202 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f63200a25f0 0x7f63200a4ab0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f632c00bc30 tx=0x7f632c0107a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.888+0000 7f63377fe640 1 --2- 172.21.15.70:0/3471505593 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f7c200 0x55bd64f7e690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f633806f450 tx=0x7f633811fd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f127756f6683ca3b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.888+0000 7f631d7fa640 1 -- 172.21.15.70:0/3471505593 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63381206b0 con 0x55bd64f7c200 2024-04-01T01:35:56.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.888+0000 7f631d7fa640 1 -- 172.21.15.70:0/3471505593 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f633811ffb0 con 0x55bd64f7c200 2024-04-01T01:35:56.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.888+0000 7f631d7fa640 1 -- 172.21.15.70:0/3471505593 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633811c620 con 0x55bd64f7c200 2024-04-01T01:35:56.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.891+0000 7f63401de740 1 -- 172.21.15.70:0/3471505593 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f7c200 msgr2=0x55bd64f7e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.891+0000 7f63401de740 1 --2- 172.21.15.70:0/3471505593 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f7c200 0x55bd64f7e690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f633806f450 tx=0x7f633811fd80 comp rx=0 tx=0).stop 2024-04-01T01:35:56.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.891+0000 7f63401de740 1 -- 172.21.15.70:0/3471505593 shutdown_connections 2024-04-01T01:35:56.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.891+0000 7f63401de740 1 --2- 172.21.15.70:0/3471505593 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f7c200 0x55bd64f7e690 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.891+0000 7f63401de740 1 --2- 172.21.15.70:0/3471505593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f77090 0x55bd64f7bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.891+0000 7f63401de740 1 --2- 172.21.15.70:0/3471505593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f766a0 0x55bd64f76ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.891+0000 7f63401de740 1 -- 172.21.15.70:0/3471505593 >> 172.21.15.70:0/3471505593 conn(0x55bd64f71c10 msgr2=0x55bd64f72010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.892+0000 7f63401de740 1 -- 172.21.15.70:0/3471505593 shutdown_connections 2024-04-01T01:35:56.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.892+0000 7f63401de740 1 -- 172.21.15.70:0/3471505593 wait complete. 2024-04-01T01:35:56.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.893+0000 7f63401de740 1 -- start start 2024-04-01T01:35:56.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.893+0000 7f63401de740 1 Processor -- start 2024-04-01T01:35:56.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.893+0000 7f63401de740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f766a0 0x55bd64f80870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.894+0000 7f63401de740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f77090 0x55bd64f80db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.894+0000 7f63401de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f7c200 0x55bd64f87df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.894+0000 7f6336ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f766a0 0x55bd64f80870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.894+0000 7f6336ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f766a0 0x55bd64f80870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49602/0 (socket says 172.21.15.70:49602) 2024-04-01T01:35:56.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.894+0000 7f6336ffd640 1 -- 172.21.15.70:0/389481372 learned_addr learned my addr 172.21.15.70:0/389481372 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.894+0000 7f63401de740 1 -- 172.21.15.70:0/389481372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bd64f7ef70 con 0x55bd64f7c200 2024-04-01T01:35:56.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.894+0000 7f63401de740 1 -- 172.21.15.70:0/389481372 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55bd64e79470 con 0x55bd64f77090 2024-04-01T01:35:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.894+0000 7f63401de740 1 -- 172.21.15.70:0/389481372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bd64f7f0f0 con 0x55bd64f766a0 2024-04-01T01:35:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.894+0000 7f63401de740 10 client.0 _ll_register_callbacks cb 0x55bd64e96cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.894+0000 7f6336ffd640 1 -- 172.21.15.70:0/389481372 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f77090 msgr2=0x55bd64f80db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:35:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.894+0000 7f6336ffd640 1 --2- 172.21.15.70:0/389481372 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f77090 0x55bd64f80db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.894+0000 7f6336ffd640 1 -- 172.21.15.70:0/389481372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f7c200 msgr2=0x55bd64f87df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.894+0000 7f6336ffd640 1 --2- 172.21.15.70:0/389481372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f7c200 0x55bd64f87df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.894+0000 7f6336ffd640 1 -- 172.21.15.70:0/389481372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd64bef3f0 con 0x55bd64f766a0 2024-04-01T01:35:56.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.895+0000 7f6336ffd640 1 --2- 172.21.15.70:0/389481372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f766a0 0x55bd64f80870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f632c013740 tx=0x7f632c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e215ace671da467 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.895+0000 7f631aff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:35:56.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.895+0000 7f631aff5640 1 -- 172.21.15.70:0/389481372 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632c01e800 con 0x55bd64f766a0 2024-04-01T01:35:56.897 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 70.0 s, 1183.3 tps, lat 84.159 ms stddev 52.335 2024-04-01T01:35:56.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.896+0000 7f631aff5640 1 -- 172.21.15.70:0/389481372 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f632c01e9a0 con 0x55bd64f766a0 2024-04-01T01:35:56.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.896+0000 7f63401de740 1 -- 172.21.15.70:0/389481372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f633811ffb0 con 0x55bd64f766a0 2024-04-01T01:35:56.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.896+0000 7f631aff5640 1 -- 172.21.15.70:0/389481372 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f632c01eba0 con 0x55bd64f766a0 2024-04-01T01:35:56.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.896+0000 7f63401de740 10 client.26071 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:56.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.896+0000 7f63401de740 1 -- 172.21.15.70:0/389481372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bd64cb97d0 con 0x55bd64f766a0 2024-04-01T01:35:56.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.897+0000 7f631aff5640 1 -- 172.21.15.70:0/389481372 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f633811ffb0 con 0x55bd64f766a0 2024-04-01T01:35:56.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.897+0000 7f631aff5640 1 -- 172.21.15.70:0/389481372 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f632c01f430 con 0x55bd64f766a0 2024-04-01T01:35:56.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.897+0000 7f63401de740 10 client.26071 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:56.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.897+0000 7f63401de740 10 client.26071 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:35:56.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.897+0000 7f63401de740 20 client.26071 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:56.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.897+0000 7f63401de740 1 --2- 172.21.15.70:0/389481372 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bd64f8c4b0 0x55bd64f8e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.897+0000 7f63401de740 4 client.26071 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:56.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.897+0000 7f63401de740 1 -- 172.21.15.70:0/389481372 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd64ee86f0 con 0x55bd64f8c4b0 2024-04-01T01:35:56.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.898+0000 7f63367fc640 1 --2- 172.21.15.70:0/389481372 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bd64f8c4b0 0x55bd64f8e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.898+0000 7f63367fc640 1 --2- 172.21.15.70:0/389481372 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bd64f8c4b0 0x55bd64f8e950 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d44cb0a69c93ade2 server_cookie=9d55453c1d1665eb in_seq=0 out_seq=0 2024-04-01T01:35:56.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.898+0000 7f631aff5640 10 client.26071 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:35:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.900+0000 7f631aff5640 1 -- 172.21.15.70:0/389481372 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bd64ee86f0 con 0x55bd64f8c4b0 2024-04-01T01:35:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.900+0000 7f631aff5640 10 client.26071 handle_command_reply: tid=0 2024-04-01T01:35:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.900+0000 7f63401de740 1 client.26071 shutdown 2024-04-01T01:35:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.900+0000 7f63401de740 1 -- 172.21.15.70:0/389481372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f766a0 msgr2=0x55bd64f80870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.900+0000 7f63401de740 1 --2- 172.21.15.70:0/389481372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f766a0 0x55bd64f80870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f632c013740 tx=0x7f632c004730 comp rx=0 tx=0).stop 2024-04-01T01:35:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.900+0000 7f63401de740 1 -- 172.21.15.70:0/389481372 shutdown_connections 2024-04-01T01:35:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.900+0000 7f63401de740 1 --2- 172.21.15.70:0/389481372 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bd64f8c4b0 0x55bd64f8e950 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.901+0000 7f63401de740 1 --2- 172.21.15.70:0/389481372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f7c200 0x55bd64f87df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.901+0000 7f63401de740 1 --2- 172.21.15.70:0/389481372 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f77090 0x55bd64f80db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.901+0000 7f63401de740 1 --2- 172.21.15.70:0/389481372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f766a0 0x55bd64f80870 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.901+0000 7f63401de740 1 -- 172.21.15.70:0/389481372 >> 172.21.15.70:0/389481372 conn(0x55bd64f71c10 msgr2=0x55bd64f70f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.901+0000 7f631aff5640 0 client.26071 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:35:56.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.901+0000 7f63401de740 1 -- 172.21.15.70:0/389481372 shutdown_connections 2024-04-01T01:35:56.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.901+0000 7f63401de740 1 -- 172.21.15.70:0/389481372 wait complete. 2024-04-01T01:35:56.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.901+0000 7f63401de740 20 client.26071 trim_cache size 0 max 16384 2024-04-01T01:35:56.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.926+0000 7f63401de740 1 Processor -- start 2024-04-01T01:35:56.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.926+0000 7f63401de740 1 -- start start 2024-04-01T01:35:56.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.926+0000 7f63401de740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f766a0 0x55bd64f80d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.926+0000 7f63401de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f77090 0x55bd64f75580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.927+0000 7f63401de740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f7c200 0x55bd64f7f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.927+0000 7f63401de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f632400fcd0 con 0x55bd64f77090 2024-04-01T01:35:56.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.927+0000 7f63401de740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6324011810 con 0x55bd64f7c200 2024-04-01T01:35:56.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.927+0000 7f63401de740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f63240103a0 con 0x55bd64f766a0 2024-04-01T01:35:56.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.927+0000 7f63367fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f77090 0x55bd64f75580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.927+0000 7f63377fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f7c200 0x55bd64f7f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.927+0000 7f63367fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f77090 0x55bd64f75580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52702/0 (socket says 172.21.15.70:52702) 2024-04-01T01:35:56.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.927+0000 7f63367fc640 1 -- 172.21.15.70:0/145627211 learned_addr learned my addr 172.21.15.70:0/145627211 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.927+0000 7f6336ffd640 1 --2- 172.21.15.70:0/145627211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f766a0 0x55bd64f80d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.928+0000 7f63377fe640 1 -- 172.21.15.70:0/145627211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f766a0 msgr2=0x55bd64f80d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.928+0000 7f63377fe640 1 --2- 172.21.15.70:0/145627211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f766a0 0x55bd64f80d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.928+0000 7f63377fe640 1 -- 172.21.15.70:0/145627211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f77090 msgr2=0x55bd64f75580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.928+0000 7f63377fe640 1 --2- 172.21.15.70:0/145627211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f77090 0x55bd64f75580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.928+0000 7f63377fe640 1 -- 172.21.15.70:0/145627211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd64f77920 con 0x55bd64f7c200 2024-04-01T01:35:56.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.928+0000 7f6336ffd640 1 --2- 172.21.15.70:0/145627211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f766a0 0x55bd64f80d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:56.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.928+0000 7f63377fe640 1 --2- 172.21.15.70:0/145627211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f7c200 0x55bd64f7f420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6338064c40 tx=0x7f633806f310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eba6ad65e8ed1557 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.929+0000 7f631cff9640 1 -- 172.21.15.70:0/145627211 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633811e050 con 0x55bd64f7c200 2024-04-01T01:35:56.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.930+0000 7f631cff9640 1 -- 172.21.15.70:0/145627211 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63381206b0 con 0x55bd64f7c200 2024-04-01T01:35:56.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.930+0000 7f631cff9640 1 -- 172.21.15.70:0/145627211 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6338128530 con 0x55bd64f7c200 2024-04-01T01:35:56.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.931+0000 7f63401de740 1 -- 172.21.15.70:0/145627211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f7c200 msgr2=0x55bd64f7f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.931+0000 7f63401de740 1 --2- 172.21.15.70:0/145627211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f7c200 0x55bd64f7f420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6338064c40 tx=0x7f633806f310 comp rx=0 tx=0).stop 2024-04-01T01:35:56.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.932+0000 7f63401de740 1 -- 172.21.15.70:0/145627211 shutdown_connections 2024-04-01T01:35:56.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.932+0000 7f63401de740 1 --2- 172.21.15.70:0/145627211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f7c200 0x55bd64f7f420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.932+0000 7f63401de740 1 --2- 172.21.15.70:0/145627211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f77090 0x55bd64f75580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.932+0000 7f63401de740 1 --2- 172.21.15.70:0/145627211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f766a0 0x55bd64f80d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.932+0000 7f63401de740 1 -- 172.21.15.70:0/145627211 >> 172.21.15.70:0/145627211 conn(0x55bd64f71c10 msgr2=0x55bd64f74e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.932+0000 7f63401de740 1 -- 172.21.15.70:0/145627211 shutdown_connections 2024-04-01T01:35:56.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.932+0000 7f63401de740 1 -- 172.21.15.70:0/145627211 wait complete. 2024-04-01T01:35:56.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.933+0000 7f63401de740 1 -- start start 2024-04-01T01:35:56.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.933+0000 7f63401de740 1 Processor -- start 2024-04-01T01:35:56.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.933+0000 7f63401de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f766a0 0x55bd64f85240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.933+0000 7f63401de740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f77090 0x55bd64f85780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.933+0000 7f63401de740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f7c200 0x55bd64f86790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.933+0000 7f63401de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6324011140 con 0x55bd64f766a0 2024-04-01T01:35:56.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.933+0000 7f63401de740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6324010a70 con 0x55bd64f77090 2024-04-01T01:35:56.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.933+0000 7f63401de740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f632400dcb0 con 0x55bd64f7c200 2024-04-01T01:35:56.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.933+0000 7f63401de740 10 client.0 _ll_register_callbacks cb 0x7f632c00be60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:56.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.933+0000 7f63377fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f7c200 0x55bd64f86790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.933+0000 7f6336ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f766a0 0x55bd64f85240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.934+0000 7f63377fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f7c200 0x55bd64f86790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49610/0 (socket says 172.21.15.70:49610) 2024-04-01T01:35:56.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.934+0000 7f6336ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f766a0 0x55bd64f85240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52704/0 (socket says 172.21.15.70:52704) 2024-04-01T01:35:56.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.934+0000 7f63377fe640 1 -- 172.21.15.70:0/1857895262 learned_addr learned my addr 172.21.15.70:0/1857895262 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:56.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.933+0000 7f63367fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f77090 0x55bd64f85780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.934+0000 7f6336ffd640 1 -- 172.21.15.70:0/1857895262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f7c200 msgr2=0x55bd64f86790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.934+0000 7f6336ffd640 1 --2- 172.21.15.70:0/1857895262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f7c200 0x55bd64f86790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.934+0000 7f6336ffd640 1 -- 172.21.15.70:0/1857895262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f77090 msgr2=0x55bd64f85780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:56.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.934+0000 7f6336ffd640 1 --2- 172.21.15.70:0/1857895262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f77090 0x55bd64f85780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.934+0000 7f6336ffd640 1 -- 172.21.15.70:0/1857895262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd64f86d80 con 0x55bd64f766a0 2024-04-01T01:35:56.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.934+0000 7f63367fc640 1 --2- 172.21.15.70:0/1857895262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f77090 0x55bd64f85780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:56.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.935+0000 7f63377fe640 1 --2- 172.21.15.70:0/1857895262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f7c200 0x55bd64f86790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:56.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.935+0000 7f6336ffd640 1 --2- 172.21.15.70:0/1857895262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f766a0 0x55bd64f85240 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f632c01f970 tx=0x7f632c01fe70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e799c8ad242848b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:56.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.935+0000 7f631aff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:35:56.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.935+0000 7f631aff5640 1 -- 172.21.15.70:0/1857895262 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632c01e600 con 0x55bd64f766a0 2024-04-01T01:35:56.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.935+0000 7f631aff5640 1 -- 172.21.15.70:0/1857895262 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f632c01e7a0 con 0x55bd64f766a0 2024-04-01T01:35:56.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.935+0000 7f63401de740 1 -- 172.21.15.70:0/1857895262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63381206b0 con 0x55bd64f766a0 2024-04-01T01:35:56.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.935+0000 7f631aff5640 1 -- 172.21.15.70:0/1857895262 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f632c01e9c0 con 0x55bd64f766a0 2024-04-01T01:35:56.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.935+0000 7f63401de740 10 client.16344 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:56.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.935+0000 7f63401de740 1 -- 172.21.15.70:0/1857895262 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bd64ff2960 con 0x55bd64f766a0 2024-04-01T01:35:56.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.936+0000 7f631aff5640 1 -- 172.21.15.70:0/1857895262 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63381206b0 con 0x55bd64f766a0 2024-04-01T01:35:56.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.936+0000 7f631aff5640 1 -- 172.21.15.70:0/1857895262 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f632c0054d0 con 0x55bd64f766a0 2024-04-01T01:35:56.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.936+0000 7f63401de740 10 client.16344 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:56.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.936+0000 7f63401de740 10 client.16344 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:35:56.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.936+0000 7f63401de740 20 client.16344 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:56.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.936+0000 7f63401de740 1 --2- 172.21.15.70:0/1857895262 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bd64ff3390 0x55bd64ff57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:56.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.936+0000 7f63401de740 4 client.16344 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:35:56.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.936+0000 7f63401de740 1 -- 172.21.15.70:0/1857895262 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bd64ee2a40 con 0x55bd64ff3390 2024-04-01T01:35:56.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.937+0000 7f63367fc640 1 --2- 172.21.15.70:0/1857895262 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bd64ff3390 0x55bd64ff57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:56.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.937+0000 7f63367fc640 1 --2- 172.21.15.70:0/1857895262 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bd64ff3390 0x55bd64ff57b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e0140d8d88bf0b63 server_cookie=df706b881d8964a4 in_seq=0 out_seq=0 2024-04-01T01:35:56.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.937+0000 7f631aff5640 10 client.16344 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:35:56.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.938+0000 7f631aff5640 1 -- 172.21.15.70:0/1857895262 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bd64ee2a40 con 0x55bd64ff3390 2024-04-01T01:35:56.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.938+0000 7f631aff5640 10 client.16344 handle_command_reply: tid=0 2024-04-01T01:35:56.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.938+0000 7f63401de740 1 client.16344 shutdown 2024-04-01T01:35:56.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.938+0000 7f63401de740 1 -- 172.21.15.70:0/1857895262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f766a0 msgr2=0x55bd64f85240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.938+0000 7f63401de740 1 --2- 172.21.15.70:0/1857895262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f766a0 0x55bd64f85240 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f632c01f970 tx=0x7f632c01fe70 comp rx=0 tx=0).stop 2024-04-01T01:35:56.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.938+0000 7f63401de740 1 -- 172.21.15.70:0/1857895262 shutdown_connections 2024-04-01T01:35:56.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.938+0000 7f63401de740 1 --2- 172.21.15.70:0/1857895262 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bd64ff3390 0x55bd64ff57b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.938+0000 7f631aff5640 0 client.16344 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:35:56.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.938+0000 7f63401de740 1 --2- 172.21.15.70:0/1857895262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bd64f7c200 0x55bd64f86790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.938+0000 7f63401de740 1 --2- 172.21.15.70:0/1857895262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bd64f77090 0x55bd64f85780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.938+0000 7f63401de740 1 --2- 172.21.15.70:0/1857895262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bd64f766a0 0x55bd64f85240 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.938+0000 7f63401de740 1 -- 172.21.15.70:0/1857895262 >> 172.21.15.70:0/1857895262 conn(0x55bd64f71c10 msgr2=0x55bd64f70700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.938+0000 7f63401de740 1 -- 172.21.15.70:0/1857895262 shutdown_connections 2024-04-01T01:35:56.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.938+0000 7f63401de740 1 -- 172.21.15.70:0/1857895262 wait complete. 2024-04-01T01:35:56.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.939+0000 7f63401de740 20 client.16344 trim_cache size 0 max 16384 2024-04-01T01:35:56.941 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:35:56.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.939+0000 7f6337fff640 1 -- 172.21.15.70:0/1452681202 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f63200a25f0 msgr2=0x7f63200a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.939+0000 7f6337fff640 1 --2- 172.21.15.70:0/1452681202 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f63200a25f0 0x7f63200a4ab0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f632c00bc30 tx=0x7f632c0107a0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.939+0000 7f6337fff640 1 -- 172.21.15.70:0/1452681202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63300a6700 msgr2=0x7f633014aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:56.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.939+0000 7f6337fff640 1 --2- 172.21.15.70:0/1452681202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63300a6700 0x7f633014aa50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6338000b60 tx=0x7f6338067c50 comp rx=0 tx=0).stop 2024-04-01T01:35:56.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.940+0000 7f6337fff640 1 -- 172.21.15.70:0/1452681202 shutdown_connections 2024-04-01T01:35:56.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.940+0000 7f6337fff640 1 --2- 172.21.15.70:0/1452681202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63300a6700 0x7f633014aa50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.940+0000 7f6337fff640 1 --2- 172.21.15.70:0/1452681202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63300a5d60 0x7f6330143a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.940+0000 7f6337fff640 1 --2- 172.21.15.70:0/1452681202 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f63200a25f0 0x7f63200a4ab0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.940+0000 7f6337fff640 1 --2- 172.21.15.70:0/1452681202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63300a4c30 0x7f63301434f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:56.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.940+0000 7f6337fff640 1 -- 172.21.15.70:0/1452681202 >> 172.21.15.70:0/1452681202 conn(0x7f63300a00c0 msgr2=0x7f63300a6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:56.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.940+0000 7f6337fff640 1 -- 172.21.15.70:0/1452681202 shutdown_connections 2024-04-01T01:35:56.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:56.940+0000 7f6337fff640 1 -- 172.21.15.70:0/1452681202 wait complete. 2024-04-01T01:35:56.955 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:56 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/652568723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:35:56.955 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:56 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:35:56.956 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:35:56.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:56 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/652568723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:35:56.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:56 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:35:57.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:56 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/652568723' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:35:57.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:56 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:35:57.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.134+0000 7f6639cc0640 1 -- 172.21.15.70:0/368512411 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6634108750 msgr2=0x7f663410ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.134+0000 7f6639cc0640 1 --2- 172.21.15.70:0/368512411 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6634108750 0x7f663410ab80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f662c009900 tx=0x7f662c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:35:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.134+0000 7f6639cc0640 1 -- 172.21.15.70:0/368512411 shutdown_connections 2024-04-01T01:35:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.134+0000 7f6639cc0640 1 --2- 172.21.15.70:0/368512411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f663410da70 0x7f663410def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.134+0000 7f6639cc0640 1 --2- 172.21.15.70:0/368512411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f663410b0c0 0x7f663410d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.134+0000 7f6639cc0640 1 --2- 172.21.15.70:0/368512411 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6634108750 0x7f663410ab80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.134+0000 7f6639cc0640 1 -- 172.21.15.70:0/368512411 >> 172.21.15.70:0/368512411 conn(0x7f663406e5b0 msgr2=0x7f66340709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.134+0000 7f6639cc0640 1 -- 172.21.15.70:0/368512411 shutdown_connections 2024-04-01T01:35:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.134+0000 7f6639cc0640 1 -- 172.21.15.70:0/368512411 wait complete. 2024-04-01T01:35:57.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.135+0000 7f6639cc0640 1 Processor -- start 2024-04-01T01:35:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.136+0000 7f6639cc0640 1 -- start start 2024-04-01T01:35:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.136+0000 7f6639cc0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6634108750 0x7f66341a6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.136+0000 7f6639cc0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f663410b0c0 0x7f66341a6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.136+0000 7f6639cc0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f663410da70 0x7f66341adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.136+0000 7f6639cc0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f663411c0c0 con 0x7f663410da70 2024-04-01T01:35:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.136+0000 7f6639cc0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f663411bf40 con 0x7f6634108750 2024-04-01T01:35:57.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.136+0000 7f6639cc0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f663411c240 con 0x7f663410b0c0 2024-04-01T01:35:57.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.136+0000 7f66337fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6634108750 0x7f66341a6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.136+0000 7f66337fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6634108750 0x7f66341a6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34748/0 (socket says 172.21.15.70:34748) 2024-04-01T01:35:57.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.136+0000 7f66337fe640 1 -- 172.21.15.70:0/3495675277 learned_addr learned my addr 172.21.15.70:0/3495675277 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:57.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.136+0000 7f6632ffd640 1 --2- 172.21.15.70:0/3495675277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f663410b0c0 0x7f66341a6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.137+0000 7f6633fff640 1 --2- 172.21.15.70:0/3495675277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f663410da70 0x7f66341adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.137+0000 7f6632ffd640 1 -- 172.21.15.70:0/3495675277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6634108750 msgr2=0x7f66341a6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.137+0000 7f6632ffd640 1 --2- 172.21.15.70:0/3495675277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6634108750 0x7f66341a6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.137+0000 7f6632ffd640 1 -- 172.21.15.70:0/3495675277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f663410da70 msgr2=0x7f66341adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.137+0000 7f6632ffd640 1 --2- 172.21.15.70:0/3495675277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f663410da70 0x7f66341adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.137+0000 7f6632ffd640 1 -- 172.21.15.70:0/3495675277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f663410ffb0 con 0x7f663410b0c0 2024-04-01T01:35:57.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.137+0000 7f66337fe640 1 --2- 172.21.15.70:0/3495675277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6634108750 0x7f66341a6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:57.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.137+0000 7f6632ffd640 1 --2- 172.21.15.70:0/3495675277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f663410b0c0 0x7f66341a6d80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f662800d6d0 tx=0x7f662800db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:57.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.138+0000 7f6630ff9640 1 -- 172.21.15.70:0/3495675277 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6628017070 con 0x7f663410b0c0 2024-04-01T01:35:57.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.138+0000 7f6630ff9640 1 -- 172.21.15.70:0/3495675277 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6628009820 con 0x7f663410b0c0 2024-04-01T01:35:57.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.138+0000 7f6639cc0640 1 -- 172.21.15.70:0/3495675277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66341102a0 con 0x7f663410b0c0 2024-04-01T01:35:57.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.138+0000 7f6630ff9640 1 -- 172.21.15.70:0/3495675277 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6628004c40 con 0x7f663410b0c0 2024-04-01T01:35:57.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.138+0000 7f6639cc0640 1 -- 172.21.15.70:0/3495675277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66341107f0 con 0x7f663410b0c0 2024-04-01T01:35:57.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.141+0000 7f6630ff9640 1 -- 172.21.15.70:0/3495675277 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f66280099d0 con 0x7f663410b0c0 2024-04-01T01:35:57.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.142+0000 7f663c979740 1 Processor -- start 2024-04-01T01:35:57.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.142+0000 7f663c979740 1 -- start start 2024-04-01T01:35:57.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.142+0000 7f663c979740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f143490 0x56553f1438b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.142+0000 7f663c979740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f143e80 0x56553f148980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.142+0000 7f663c979740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f148ff0 0x56553f14b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.142+0000 7f663c979740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56553f0a81b0 con 0x56553f143490 2024-04-01T01:35:57.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.142+0000 7f663c979740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56553f050c00 con 0x56553f143e80 2024-04-01T01:35:57.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.142+0000 7f663c979740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56553f0a4250 con 0x56553f148ff0 2024-04-01T01:35:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.142+0000 7f6633fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f143490 0x56553f1438b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.142+0000 7f6633fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f143490 0x56553f1438b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52730/0 (socket says 172.21.15.70:52730) 2024-04-01T01:35:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.142+0000 7f6633fff640 1 -- 172.21.15.70:0/2061594260 learned_addr learned my addr 172.21.15.70:0/2061594260 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.142+0000 7f66337fe640 1 --2- 172.21.15.70:0/2061594260 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f143e80 0x56553f148980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.142+0000 7f6630ff9640 1 --2- 172.21.15.70:0/3495675277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f661c0a2620 0x7f661c0a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.142+0000 7f6632ffd640 1 --2- 172.21.15.70:0/2061594260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f148ff0 0x56553f14b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.143+0000 7f66337fe640 1 -- 172.21.15.70:0/2061594260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f148ff0 msgr2=0x56553f14b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.143+0000 7f66337fe640 1 --2- 172.21.15.70:0/2061594260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f148ff0 0x56553f14b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.143+0000 7f66337fe640 1 -- 172.21.15.70:0/2061594260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f143490 msgr2=0x56553f1438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.143+0000 7f66337fe640 1 --2- 172.21.15.70:0/2061594260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f143490 0x56553f1438b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.143+0000 7f66337fe640 1 -- 172.21.15.70:0/2061594260 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56553f0703d0 con 0x56553f143e80 2024-04-01T01:35:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.143+0000 7f6633fff640 1 --2- 172.21.15.70:0/2061594260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f143490 0x56553f1438b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:35:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.143+0000 7f6633fff640 1 --2- 172.21.15.70:0/3495675277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f661c0a2620 0x7f661c0a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.143+0000 7f6632ffd640 1 --2- 172.21.15.70:0/2061594260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f148ff0 0x56553f14b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:35:57.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.143+0000 7f6630ff9640 1 -- 172.21.15.70:0/3495675277 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f66280ce450 con 0x7f663410b0c0 2024-04-01T01:35:57.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.143+0000 7f66337fe640 1 --2- 172.21.15.70:0/2061594260 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f143e80 0x56553f148980 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f662c006010 tx=0x7f662c02cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6534e3e1d1e7f3bd server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.143+0000 7f6619ffb640 1 -- 172.21.15.70:0/2061594260 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c045020 con 0x56553f143e80 2024-04-01T01:35:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.143+0000 7f6619ffb640 1 -- 172.21.15.70:0/2061594260 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f662c005040 con 0x56553f143e80 2024-04-01T01:35:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.143+0000 7f6619ffb640 1 -- 172.21.15.70:0/2061594260 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c004aa0 con 0x56553f143e80 2024-04-01T01:35:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.144+0000 7f663c979740 1 -- 172.21.15.70:0/2061594260 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f143e80 msgr2=0x56553f148980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.144+0000 7f663c979740 1 --2- 172.21.15.70:0/2061594260 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f143e80 0x56553f148980 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f662c006010 tx=0x7f662c02cc90 comp rx=0 tx=0).stop 2024-04-01T01:35:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.144+0000 7f6633fff640 1 --2- 172.21.15.70:0/3495675277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f661c0a2620 0x7f661c0a4ae0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x56553f1446e0 tx=0x7f6624004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.145+0000 7f663c979740 1 -- 172.21.15.70:0/2061594260 shutdown_connections 2024-04-01T01:35:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.145+0000 7f663c979740 1 --2- 172.21.15.70:0/2061594260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f148ff0 0x56553f14b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.145+0000 7f663c979740 1 --2- 172.21.15.70:0/2061594260 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f143e80 0x56553f148980 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.145+0000 7f663c979740 1 --2- 172.21.15.70:0/2061594260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f143490 0x56553f1438b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.145+0000 7f663c979740 1 -- 172.21.15.70:0/2061594260 >> 172.21.15.70:0/2061594260 conn(0x56553f13ea00 msgr2=0x56553f13ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:57.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.145+0000 7f663c979740 1 -- 172.21.15.70:0/2061594260 shutdown_connections 2024-04-01T01:35:57.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.145+0000 7f663c979740 1 -- 172.21.15.70:0/2061594260 wait complete. 2024-04-01T01:35:57.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.145+0000 7f663c979740 1 -- start start 2024-04-01T01:35:57.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.146+0000 7f663c979740 1 Processor -- start 2024-04-01T01:35:57.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.146+0000 7f663c979740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f143490 0x56553f14d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.146+0000 7f663c979740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f143e80 0x56553f14de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.146+0000 7f663c979740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f148ff0 0x56553f154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.146+0000 7f663c979740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56553f14bf40 con 0x56553f148ff0 2024-04-01T01:35:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.146+0000 7f663c979740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56553f046470 con 0x56553f143e80 2024-04-01T01:35:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.146+0000 7f663c979740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56553f14c0c0 con 0x56553f143490 2024-04-01T01:35:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.146+0000 7f663c979740 10 client.0 _ll_register_callbacks cb 0x56553f063cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.146+0000 7f66337fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f143490 0x56553f14d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.146+0000 7f66337fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f143490 0x56553f14d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49654/0 (socket says 172.21.15.70:49654) 2024-04-01T01:35:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.147+0000 7f66337fe640 1 -- 172.21.15.70:0/1039933117 learned_addr learned my addr 172.21.15.70:0/1039933117 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.147+0000 7f6632ffd640 1 --2- 172.21.15.70:0/1039933117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f143e80 0x56553f14de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.147+0000 7f66337fe640 1 -- 172.21.15.70:0/1039933117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f143e80 msgr2=0x56553f14de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.147+0000 7f66337fe640 1 --2- 172.21.15.70:0/1039933117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f143e80 0x56553f14de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.147+0000 7f66337fe640 1 -- 172.21.15.70:0/1039933117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f148ff0 msgr2=0x56553f154ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.147+0000 7f66337fe640 1 --2- 172.21.15.70:0/1039933117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f148ff0 0x56553f154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.147+0000 7f66337fe640 1 -- 172.21.15.70:0/1039933117 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56553edbc3f0 con 0x56553f143490 2024-04-01T01:35:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.147+0000 7f66337fe640 1 --2- 172.21.15.70:0/1039933117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f143490 0x56553f14d950 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f662c005ab0 tx=0x7f662c004af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78e32f6d32f28266 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.147+0000 7f6612ffd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:35:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.148+0000 7f6612ffd640 1 -- 172.21.15.70:0/1039933117 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c045040 con 0x56553f143490 2024-04-01T01:35:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.148+0000 7f6612ffd640 1 -- 172.21.15.70:0/1039933117 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f662c041680 con 0x56553f143490 2024-04-01T01:35:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.148+0000 7f663c979740 1 -- 172.21.15.70:0/1039933117 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f662c005040 con 0x56553f143490 2024-04-01T01:35:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.148+0000 7f6612ffd640 1 -- 172.21.15.70:0/1039933117 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f662c05b050 con 0x56553f143490 2024-04-01T01:35:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.148+0000 7f663c979740 10 client.26089 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.148+0000 7f663c979740 1 -- 172.21.15.70:0/1039933117 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56553ee867d0 con 0x56553f143490 2024-04-01T01:35:57.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.148+0000 7f6612ffd640 1 -- 172.21.15.70:0/1039933117 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f662c005040 con 0x56553f143490 2024-04-01T01:35:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.149+0000 7f6612ffd640 1 -- 172.21.15.70:0/1039933117 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f662c04a870 con 0x56553f143490 2024-04-01T01:35:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.149+0000 7f663c979740 10 client.26089 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.149+0000 7f663c979740 10 client.26089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.149+0000 7f663c979740 20 client.26089 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.149+0000 7f663c979740 1 --2- 172.21.15.70:0/1039933117 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56553f159030 0x56553f15b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.149+0000 7f663c979740 4 client.26089 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.149+0000 7f663c979740 1 -- 172.21.15.70:0/1039933117 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56553f0b56f0 con 0x56553f159030 2024-04-01T01:35:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.149+0000 7f6632ffd640 1 --2- 172.21.15.70:0/1039933117 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56553f159030 0x56553f15b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.150+0000 7f6632ffd640 1 --2- 172.21.15.70:0/1039933117 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56553f159030 0x56553f15b440 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60952ca8c0facd85 server_cookie=6d665d73016a94b4 in_seq=0 out_seq=0 2024-04-01T01:35:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.150+0000 7f6612ffd640 10 client.26089 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.150+0000 7f6612ffd640 1 -- 172.21.15.70:0/1039933117 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56553f0b56f0 con 0x56553f159030 2024-04-01T01:35:57.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.150+0000 7f6612ffd640 10 client.26089 handle_command_reply: tid=0 2024-04-01T01:35:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.151+0000 7f663c979740 1 client.26089 shutdown 2024-04-01T01:35:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.151+0000 7f663c979740 1 -- 172.21.15.70:0/1039933117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f143490 msgr2=0x56553f14d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.151+0000 7f663c979740 1 --2- 172.21.15.70:0/1039933117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f143490 0x56553f14d950 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f662c005ab0 tx=0x7f662c004af0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.151+0000 7f663c979740 1 -- 172.21.15.70:0/1039933117 shutdown_connections 2024-04-01T01:35:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.151+0000 7f663c979740 1 --2- 172.21.15.70:0/1039933117 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56553f159030 0x56553f15b440 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.151+0000 7f663c979740 1 --2- 172.21.15.70:0/1039933117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f148ff0 0x56553f154ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.151+0000 7f663c979740 1 --2- 172.21.15.70:0/1039933117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f143e80 0x56553f14de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.151+0000 7f663c979740 1 --2- 172.21.15.70:0/1039933117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f143490 0x56553f14d950 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f662c005ab0 tx=0x7f662c004af0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.151+0000 7f663c979740 1 -- 172.21.15.70:0/1039933117 >> 172.21.15.70:0/1039933117 conn(0x56553f13ea00 msgr2=0x56553f13dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.151+0000 7f6612ffd640 0 client.26089 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.151+0000 7f663c979740 1 -- 172.21.15.70:0/1039933117 shutdown_connections 2024-04-01T01:35:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.151+0000 7f663c979740 1 -- 172.21.15.70:0/1039933117 wait complete. 2024-04-01T01:35:57.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.152+0000 7f663c979740 20 client.26089 trim_cache size 0 max 16384 2024-04-01T01:35:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.181+0000 7f663c979740 1 Processor -- start 2024-04-01T01:35:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.181+0000 7f663c979740 1 -- start start 2024-04-01T01:35:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.182+0000 7f663c979740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f143e80 0x56553f14cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.182+0000 7f663c979740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f148ff0 0x56553f154e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.182+0000 7f663c979740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f13f240 0x56553f14d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.182+0000 7f663c979740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f66341c1980 con 0x56553f143e80 2024-04-01T01:35:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.182+0000 7f663c979740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f66341c34c0 con 0x56553f148ff0 2024-04-01T01:35:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.182+0000 7f663c979740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f66341c2050 con 0x56553f13f240 2024-04-01T01:35:57.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.183+0000 7f6633fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f13f240 0x56553f14d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.183+0000 7f66337fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f143e80 0x56553f14cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.183+0000 7f6632ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f148ff0 0x56553f154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.183+0000 7f66337fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f143e80 0x56553f14cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52746/0 (socket says 172.21.15.70:52746) 2024-04-01T01:35:57.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.183+0000 7f66337fe640 1 -- 172.21.15.70:0/1380473213 learned_addr learned my addr 172.21.15.70:0/1380473213 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:57.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.183+0000 7f6633fff640 1 -- 172.21.15.70:0/1380473213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f148ff0 msgr2=0x56553f154e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:57.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.183+0000 7f6633fff640 1 --2- 172.21.15.70:0/1380473213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f148ff0 0x56553f154e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.183+0000 7f6633fff640 1 -- 172.21.15.70:0/1380473213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f143e80 msgr2=0x56553f14cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:57.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.183+0000 7f6633fff640 1 --2- 172.21.15.70:0/1380473213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f143e80 0x56553f14cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.183+0000 7f6633fff640 1 -- 172.21.15.70:0/1380473213 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56553f14ead0 con 0x56553f13f240 2024-04-01T01:35:57.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.184+0000 7f6633fff640 1 --2- 172.21.15.70:0/1380473213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f13f240 0x56553f14d4e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6624009360 tx=0x7f6624009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31f10704c6444cf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:57.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.185+0000 7f66197fa640 1 -- 172.21.15.70:0/1380473213 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662400fd30 con 0x56553f13f240 2024-04-01T01:35:57.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.185+0000 7f66197fa640 1 -- 172.21.15.70:0/1380473213 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6624025030 con 0x56553f13f240 2024-04-01T01:35:57.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.185+0000 7f66197fa640 1 -- 172.21.15.70:0/1380473213 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6624020670 con 0x56553f13f240 2024-04-01T01:35:57.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.187+0000 7f663c979740 1 -- 172.21.15.70:0/1380473213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f13f240 msgr2=0x56553f14d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.187+0000 7f663c979740 1 --2- 172.21.15.70:0/1380473213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f13f240 0x56553f14d4e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6624009360 tx=0x7f6624009500 comp rx=0 tx=0).stop 2024-04-01T01:35:57.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.188+0000 7f663c979740 1 -- 172.21.15.70:0/1380473213 shutdown_connections 2024-04-01T01:35:57.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.188+0000 7f663c979740 1 --2- 172.21.15.70:0/1380473213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f13f240 0x56553f14d4e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.188+0000 7f663c979740 1 --2- 172.21.15.70:0/1380473213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f148ff0 0x56553f154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.188+0000 7f663c979740 1 --2- 172.21.15.70:0/1380473213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f143e80 0x56553f14cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.188+0000 7f663c979740 1 -- 172.21.15.70:0/1380473213 >> 172.21.15.70:0/1380473213 conn(0x56553f13ea00 msgr2=0x56553f13dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:57.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.189+0000 7f663c979740 1 -- 172.21.15.70:0/1380473213 shutdown_connections 2024-04-01T01:35:57.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.190+0000 7f663c979740 1 -- 172.21.15.70:0/1380473213 wait complete. 2024-04-01T01:35:57.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.191+0000 7f663c979740 1 -- start start 2024-04-01T01:35:57.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.191+0000 7f663c979740 1 Processor -- start 2024-04-01T01:35:57.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.192+0000 7f663c979740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f13f240 0x56553f152b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.192+0000 7f663c979740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f143e80 0x56553f153070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.192+0000 7f663c979740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f148ff0 0x56553f154030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.192+0000 7f663c979740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f66341c2df0 con 0x56553f148ff0 2024-04-01T01:35:57.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.192+0000 7f663c979740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f66341c2720 con 0x56553f13f240 2024-04-01T01:35:57.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.192+0000 7f663c979740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f66341183f0 con 0x56553f143e80 2024-04-01T01:35:57.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.192+0000 7f663c979740 10 client.0 _ll_register_callbacks cb 0x7f662c0055f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.193+0000 7f6632ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f143e80 0x56553f153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.193+0000 7f6632ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f143e80 0x56553f153070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49674/0 (socket says 172.21.15.70:49674) 2024-04-01T01:35:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.193+0000 7f6632ffd640 1 -- 172.21.15.70:0/605891877 learned_addr learned my addr 172.21.15.70:0/605891877 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.193+0000 7f66337fe640 1 --2- 172.21.15.70:0/605891877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f13f240 0x56553f152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.193+0000 7f6633fff640 1 --2- 172.21.15.70:0/605891877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f148ff0 0x56553f154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.193+0000 7f66337fe640 1 -- 172.21.15.70:0/605891877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f143e80 msgr2=0x56553f153070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.193+0000 7f66337fe640 1 --2- 172.21.15.70:0/605891877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f143e80 0x56553f153070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.193+0000 7f66337fe640 1 -- 172.21.15.70:0/605891877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f148ff0 msgr2=0x56553f154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.193+0000 7f66337fe640 1 --2- 172.21.15.70:0/605891877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f148ff0 0x56553f154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.193+0000 7f66337fe640 1 -- 172.21.15.70:0/605891877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56553f154620 con 0x56553f13f240 2024-04-01T01:35:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.193+0000 7f6632ffd640 1 --2- 172.21.15.70:0/605891877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f143e80 0x56553f153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.193+0000 7f6633fff640 1 --2- 172.21.15.70:0/605891877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f148ff0 0x56553f154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:35:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.194+0000 7f66337fe640 1 --2- 172.21.15.70:0/605891877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f13f240 0x56553f152b30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x56553f13a540 tx=0x7f662c005120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f54f4dfe7dec0b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.194+0000 7f6612ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.194+0000 7f6612ffd640 1 -- 172.21.15.70:0/605891877 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662c045040 con 0x56553f13f240 2024-04-01T01:35:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.194+0000 7f6612ffd640 1 -- 172.21.15.70:0/605891877 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f662c004450 con 0x56553f13f240 2024-04-01T01:35:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.194+0000 7f663c979740 1 -- 172.21.15.70:0/605891877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6624025030 con 0x56553f13f240 2024-04-01T01:35:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.195+0000 7f6612ffd640 1 -- 172.21.15.70:0/605891877 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f662c02c2b0 con 0x56553f13f240 2024-04-01T01:35:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.195+0000 7f663c979740 10 client.26195 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.195+0000 7f663c979740 1 -- 172.21.15.70:0/605891877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56553f1bf890 con 0x56553f13f240 2024-04-01T01:35:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.195+0000 7f6612ffd640 1 -- 172.21.15.70:0/605891877 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6624025030 con 0x56553f13f240 2024-04-01T01:35:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.197+0000 7f6612ffd640 1 -- 172.21.15.70:0/605891877 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f662c04a450 con 0x56553f13f240 2024-04-01T01:35:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.197+0000 7f663c979740 10 client.26195 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.197+0000 7f663c979740 10 client.26195 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.197+0000 7f663c979740 20 client.26195 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.197+0000 7f663c979740 1 --2- 172.21.15.70:0/605891877 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56553f1c00a0 0x56553f1c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.197+0000 7f663c979740 4 client.26195 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:35:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.197+0000 7f663c979740 1 -- 172.21.15.70:0/605891877 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56553f0afa40 con 0x56553f1c00a0 2024-04-01T01:35:57.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.197+0000 7f6632ffd640 1 --2- 172.21.15.70:0/605891877 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56553f1c00a0 0x56553f1c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.198+0000 7f6632ffd640 1 --2- 172.21.15.70:0/605891877 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56553f1c00a0 0x56553f1c24c0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61d435665eac4e13 server_cookie=8e90d3d4524556b5 in_seq=0 out_seq=0 2024-04-01T01:35:57.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.198+0000 7f6612ffd640 10 client.26195 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.198+0000 7f6612ffd640 1 -- 172.21.15.70:0/605891877 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56553f0afa40 con 0x56553f1c00a0 2024-04-01T01:35:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.198+0000 7f6612ffd640 10 client.26195 handle_command_reply: tid=0 2024-04-01T01:35:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.198+0000 7f663c979740 1 client.26195 shutdown 2024-04-01T01:35:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.198+0000 7f663c979740 1 -- 172.21.15.70:0/605891877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f13f240 msgr2=0x56553f152b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.199+0000 7f663c979740 1 --2- 172.21.15.70:0/605891877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f13f240 0x56553f152b30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x56553f13a540 tx=0x7f662c005120 comp rx=0 tx=0).stop 2024-04-01T01:35:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.199+0000 7f663c979740 1 -- 172.21.15.70:0/605891877 shutdown_connections 2024-04-01T01:35:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.199+0000 7f663c979740 1 --2- 172.21.15.70:0/605891877 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56553f1c00a0 0x56553f1c24c0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.199+0000 7f663c979740 1 --2- 172.21.15.70:0/605891877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56553f148ff0 0x56553f154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.199+0000 7f663c979740 1 --2- 172.21.15.70:0/605891877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56553f143e80 0x56553f153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.199+0000 7f663c979740 1 --2- 172.21.15.70:0/605891877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56553f13f240 0x56553f152b30 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.199+0000 7f663c979740 1 -- 172.21.15.70:0/605891877 >> 172.21.15.70:0/605891877 conn(0x56553f13ea00 msgr2=0x56553f13d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.199+0000 7f6612ffd640 0 client.26195 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:57.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.199+0000 7f663c979740 1 -- 172.21.15.70:0/605891877 shutdown_connections 2024-04-01T01:35:57.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.199+0000 7f663c979740 1 -- 172.21.15.70:0/605891877 wait complete. 2024-04-01T01:35:57.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.199+0000 7f663c979740 20 client.26195 trim_cache size 0 max 16384 2024-04-01T01:35:57.201 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:35:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.201+0000 7f6639cc0640 1 -- 172.21.15.70:0/3495675277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f661c0a2620 msgr2=0x7f661c0a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.201+0000 7f6639cc0640 1 --2- 172.21.15.70:0/3495675277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f661c0a2620 0x7f661c0a4ae0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x56553f1446e0 tx=0x7f6624004ab0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.201+0000 7f6639cc0640 1 -- 172.21.15.70:0/3495675277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f663410b0c0 msgr2=0x7f66341a6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.201+0000 7f6639cc0640 1 --2- 172.21.15.70:0/3495675277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f663410b0c0 0x7f66341a6d80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f662800d6d0 tx=0x7f662800db90 comp rx=0 tx=0).stop 2024-04-01T01:35:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.201+0000 7f6639cc0640 1 -- 172.21.15.70:0/3495675277 shutdown_connections 2024-04-01T01:35:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.201+0000 7f6639cc0640 1 --2- 172.21.15.70:0/3495675277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f663410da70 0x7f66341adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.201+0000 7f6639cc0640 1 --2- 172.21.15.70:0/3495675277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f661c0a2620 0x7f661c0a4ae0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.201+0000 7f6639cc0640 1 --2- 172.21.15.70:0/3495675277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f663410b0c0 0x7f66341a6d80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.201+0000 7f6639cc0640 1 --2- 172.21.15.70:0/3495675277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6634108750 0x7f66341a6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.201+0000 7f6639cc0640 1 -- 172.21.15.70:0/3495675277 >> 172.21.15.70:0/3495675277 conn(0x7f663406e5b0 msgr2=0x7f663410b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.201+0000 7f6639cc0640 1 -- 172.21.15.70:0/3495675277 shutdown_connections 2024-04-01T01:35:57.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:57.201+0000 7f6639cc0640 1 -- 172.21.15.70:0/3495675277 wait complete. 2024-04-01T01:35:57.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.454+0000 7fe6d4f6c640 1 -- 172.21.15.70:0/64081664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6d0100b90 msgr2=0x7fe6d0100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.454+0000 7fe6d4f6c640 1 --2- 172.21.15.70:0/64081664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6d0100b90 0x7fe6d0100fb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0009900 tx=0x7fe6c002bc30 comp rx=0 tx=0).stop 2024-04-01T01:35:57.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.455+0000 7fe6d4f6c640 1 -- 172.21.15.70:0/64081664 shutdown_connections 2024-04-01T01:35:57.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.455+0000 7fe6d4f6c640 1 --2- 172.21.15.70:0/64081664 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6d0102260 0x7fe6d0108af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.455+0000 7fe6d4f6c640 1 --2- 172.21.15.70:0/64081664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d01018a0 0x7fe6d0101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.455+0000 7fe6d4f6c640 1 --2- 172.21.15.70:0/64081664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6d0100b90 0x7fe6d0100fb0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.455+0000 7fe6d4f6c640 1 -- 172.21.15.70:0/64081664 >> 172.21.15.70:0/64081664 conn(0x7fe6d00fb3a0 msgr2=0x7fe6d00fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:57.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.455+0000 7fe6d4f6c640 1 -- 172.21.15.70:0/64081664 shutdown_connections 2024-04-01T01:35:57.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.455+0000 7fe6d4f6c640 1 -- 172.21.15.70:0/64081664 wait complete. 2024-04-01T01:35:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.456+0000 7fe6d4f6c640 1 Processor -- start 2024-04-01T01:35:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.456+0000 7fe6d4f6c640 1 -- start start 2024-04-01T01:35:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.456+0000 7fe6d4f6c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6d0100b90 0x7fe6d0199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.456+0000 7fe6d4f6c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6d01018a0 0x7fe6d019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.456+0000 7fe6d4f6c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d0102260 0x7fe6d01a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.456+0000 7fe6d4f6c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d010f650 con 0x7fe6d0100b90 2024-04-01T01:35:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.456+0000 7fe6d4f6c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d010afa0 con 0x7fe6d01018a0 2024-04-01T01:35:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.456+0000 7fe6d4f6c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d010f7d0 con 0x7fe6d0102260 2024-04-01T01:35:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.457+0000 7fe6cdd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6d01018a0 0x7fe6d019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.457+0000 7fe6ced9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d0102260 0x7fe6d01a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.457+0000 7fe6cdd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6d01018a0 0x7fe6d019a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34808/0 (socket says 172.21.15.70:34808) 2024-04-01T01:35:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.457+0000 7fe6cdd9b640 1 -- 172.21.15.70:0/3902967006 learned_addr learned my addr 172.21.15.70:0/3902967006 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.457+0000 7fe6ce59c640 1 --2- 172.21.15.70:0/3902967006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6d0100b90 0x7fe6d0199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.457+0000 7fe6cdd9b640 1 -- 172.21.15.70:0/3902967006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d0102260 msgr2=0x7fe6d01a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.457+0000 7fe6cdd9b640 1 --2- 172.21.15.70:0/3902967006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d0102260 0x7fe6d01a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.457+0000 7fe6cdd9b640 1 -- 172.21.15.70:0/3902967006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6d0100b90 msgr2=0x7fe6d0199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.457+0000 7fe6cdd9b640 1 --2- 172.21.15.70:0/3902967006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6d0100b90 0x7fe6d0199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.457+0000 7fe6cdd9b640 1 -- 172.21.15.70:0/3902967006 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d01a1740 con 0x7fe6d01018a0 2024-04-01T01:35:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.458+0000 7fe6cdd9b640 1 --2- 172.21.15.70:0/3902967006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6d01018a0 0x7fe6d019a0b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400e9f0 tx=0x7fe6c400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.458+0000 7fe6bf7fe640 1 -- 172.21.15.70:0/3902967006 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c400cd80 con 0x7fe6d01018a0 2024-04-01T01:35:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.458+0000 7fe6bf7fe640 1 -- 172.21.15.70:0/3902967006 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6c40108e0 con 0x7fe6d01018a0 2024-04-01T01:35:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.458+0000 7fe6d4f6c640 1 -- 172.21.15.70:0/3902967006 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d01a1a30 con 0x7fe6d01018a0 2024-04-01T01:35:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.459+0000 7fe6bf7fe640 1 -- 172.21.15.70:0/3902967006 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4018920 con 0x7fe6d01018a0 2024-04-01T01:35:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.459+0000 7fe6d4f6c640 1 -- 172.21.15.70:0/3902967006 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d01a1fd0 con 0x7fe6d01018a0 2024-04-01T01:35:57.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.462+0000 7fe6bf7fe640 1 -- 172.21.15.70:0/3902967006 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fe6c4010a90 con 0x7fe6d01018a0 2024-04-01T01:35:57.461 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.462+0000 7fe6d7c2e740 1 Processor -- start 2024-04-01T01:35:57.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.463+0000 7fe6d7c2e740 1 -- start start 2024-04-01T01:35:57.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.463+0000 7fe6d7c2e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef0210e0 0x5596ef021500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.463+0000 7fe6d7c2e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef021ad0 0x5596ef0265d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.463+0000 7fe6d7c2e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef026c40 0x5596ef0290d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.463+0000 7fe6d7c2e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5596eef7c8f0 con 0x5596ef021ad0 2024-04-01T01:35:57.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.463+0000 7fe6d7c2e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5596eef258d0 con 0x5596ef026c40 2024-04-01T01:35:57.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.463+0000 7fe6d7c2e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5596eef78990 con 0x5596ef0210e0 2024-04-01T01:35:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.463+0000 7fe6bf7fe640 1 --2- 172.21.15.70:0/3902967006 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6b80a2640 0x7fe6b80a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.463+0000 7fe6ce59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef021ad0 0x5596ef0265d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.463+0000 7fe6ce59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef021ad0 0x5596ef0265d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52788/0 (socket says 172.21.15.70:52788) 2024-04-01T01:35:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.463+0000 7fe6ce59c640 1 -- 172.21.15.70:0/900356509 learned_addr learned my addr 172.21.15.70:0/900356509 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.463+0000 7fe6ced9d640 1 --2- 172.21.15.70:0/900356509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef0210e0 0x5596ef021500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.463+0000 7fe6cdd9b640 1 --2- 172.21.15.70:0/900356509 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef026c40 0x5596ef0290d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.463+0000 7fe6ced9d640 1 --2- 172.21.15.70:0/3902967006 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6b80a2640 0x7fe6b80a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.464+0000 7fe6cdd9b640 1 -- 172.21.15.70:0/900356509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef0210e0 msgr2=0x5596ef021500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.464+0000 7fe6bf7fe640 1 -- 172.21.15.70:0/3902967006 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe6c40ceda0 con 0x7fe6d01018a0 2024-04-01T01:35:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.464+0000 7fe6cdd9b640 1 --2- 172.21.15.70:0/900356509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef0210e0 0x5596ef021500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.464+0000 7fe6cdd9b640 1 -- 172.21.15.70:0/900356509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef021ad0 msgr2=0x5596ef0265d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.464+0000 7fe6cdd9b640 1 --2- 172.21.15.70:0/900356509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef021ad0 0x5596ef0265d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.464+0000 7fe6cdd9b640 1 -- 172.21.15.70:0/900356509 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596eec91210 con 0x5596ef026c40 2024-04-01T01:35:57.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.464+0000 7fe6ced9d640 1 --2- 172.21.15.70:0/900356509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef0210e0 0x5596ef021500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:57.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.465+0000 7fe6ced9d640 1 --2- 172.21.15.70:0/3902967006 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6b80a2640 0x7fe6b80a4b00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x5596ef022330 tx=0x7fe6c8009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:57.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.466+0000 7fe6cdd9b640 1 --2- 172.21.15.70:0/900356509 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef026c40 0x5596ef0290d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40cc440 tx=0x7fe6c4066920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfcfbe4960700366 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.466+0000 7fe6bcff9640 1 -- 172.21.15.70:0/900356509 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4026040 con 0x5596ef026c40 2024-04-01T01:35:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.466+0000 7fe6bcff9640 1 -- 172.21.15.70:0/900356509 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6c4010de0 con 0x5596ef026c40 2024-04-01T01:35:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.467+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/900356509 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef026c40 msgr2=0x5596ef0290d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.467+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/900356509 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef026c40 0x5596ef0290d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40cc440 tx=0x7fe6c4066920 comp rx=0 tx=0).stop 2024-04-01T01:35:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.468+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/900356509 shutdown_connections 2024-04-01T01:35:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.468+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/900356509 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef026c40 0x5596ef0290d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.468+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/900356509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef021ad0 0x5596ef0265d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.468+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/900356509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef0210e0 0x5596ef021500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.468+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/900356509 >> 172.21.15.70:0/900356509 conn(0x5596ef013e40 msgr2=0x5596ef014240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.469+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/900356509 shutdown_connections 2024-04-01T01:35:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.469+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/900356509 wait complete. 2024-04-01T01:35:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.469+0000 7fe6d7c2e740 1 -- start start 2024-04-01T01:35:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.469+0000 7fe6d7c2e740 1 Processor -- start 2024-04-01T01:35:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.469+0000 7fe6d7c2e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef0210e0 0x5596ef02b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.469+0000 7fe6d7c2e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef021ad0 0x5596ef02b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.469+0000 7fe6d7c2e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef026c40 0x5596ef032840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.470+0000 7fe6d7c2e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5596ef029b90 con 0x5596ef026c40 2024-04-01T01:35:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.470+0000 7fe6d7c2e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5596eef1af80 con 0x5596ef021ad0 2024-04-01T01:35:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.470+0000 7fe6d7c2e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5596ef029d10 con 0x5596ef0210e0 2024-04-01T01:35:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.470+0000 7fe6d7c2e740 10 client.0 _ll_register_callbacks cb 0x5596eef73290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.470+0000 7fe6ced9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef026c40 0x5596ef032840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.470+0000 7fe6ce59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef0210e0 0x5596ef02b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.470+0000 7fe6ced9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef026c40 0x5596ef032840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52792/0 (socket says 172.21.15.70:52792) 2024-04-01T01:35:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.470+0000 7fe6ced9d640 1 -- 172.21.15.70:0/2829814404 learned_addr learned my addr 172.21.15.70:0/2829814404 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.470+0000 7fe6cdd9b640 1 --2- 172.21.15.70:0/2829814404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef021ad0 0x5596ef02b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.470+0000 7fe6ced9d640 1 -- 172.21.15.70:0/2829814404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef0210e0 msgr2=0x5596ef02b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.470+0000 7fe6ced9d640 1 --2- 172.21.15.70:0/2829814404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef0210e0 0x5596ef02b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.470+0000 7fe6ced9d640 1 -- 172.21.15.70:0/2829814404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef021ad0 msgr2=0x5596ef02b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.470+0000 7fe6ced9d640 1 --2- 172.21.15.70:0/2829814404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef021ad0 0x5596ef02b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.470+0000 7fe6ced9d640 1 -- 172.21.15.70:0/2829814404 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596eed5d620 con 0x5596ef026c40 2024-04-01T01:35:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.471+0000 7fe6ced9d640 1 --2- 172.21.15.70:0/2829814404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef026c40 0x5596ef032840 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8019180 tx=0x7fe6c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8491ed487e9d03fc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.471+0000 7fe6b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:35:57.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.471+0000 7fe6b5ffb640 1 -- 172.21.15.70:0/2829814404 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c802d060 con 0x5596ef026c40 2024-04-01T01:35:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.471+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2829814404 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6c40998b0 con 0x5596ef026c40 2024-04-01T01:35:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.471+0000 7fe6b5ffb640 1 -- 172.21.15.70:0/2829814404 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe6c8004730 con 0x5596ef026c40 2024-04-01T01:35:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.472+0000 7fe6b5ffb640 1 -- 172.21.15.70:0/2829814404 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe6c8034050 con 0x5596ef026c40 2024-04-01T01:35:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.472+0000 7fe6d7c2e740 10 client.16368 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.472+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2829814404 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5596ef0358e0 con 0x5596ef026c40 2024-04-01T01:35:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.473+0000 7fe6b5ffb640 1 -- 172.21.15.70:0/2829814404 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6c8020b20 con 0x5596ef026c40 2024-04-01T01:35:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.473+0000 7fe6b5ffb640 1 -- 172.21.15.70:0/2829814404 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe6c8027030 con 0x5596ef026c40 2024-04-01T01:35:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.473+0000 7fe6d7c2e740 10 client.16368 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.473+0000 7fe6d7c2e740 10 client.16368 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.473+0000 7fe6d7c2e740 20 client.16368 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.473+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/2829814404 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5596ef037000 0x5596ef0393f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.473+0000 7fe6d7c2e740 4 client.16368 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.474+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2829814404 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe6c40998b0 con 0x5596ef037000 2024-04-01T01:35:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.474+0000 7fe6ce59c640 1 --2- 172.21.15.70:0/2829814404 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5596ef037000 0x5596ef0393f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.475+0000 7fe6b5ffb640 10 client.16368 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:57.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.475+0000 7fe6ce59c640 1 --2- 172.21.15.70:0/2829814404 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5596ef037000 0x5596ef0393f0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5fae27e9c6f8fa2 server_cookie=c239e487c4b0bd62 in_seq=0 out_seq=0 2024-04-01T01:35:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.476+0000 7fe6b5ffb640 1 -- 172.21.15.70:0/2829814404 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fe6c40998b0 con 0x5596ef037000 2024-04-01T01:35:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.476+0000 7fe6b5ffb640 10 client.16368 handle_command_reply: tid=0 2024-04-01T01:35:57.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.476+0000 7fe6d7c2e740 1 client.16368 shutdown 2024-04-01T01:35:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.476+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2829814404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef026c40 msgr2=0x5596ef032840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.476+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/2829814404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef026c40 0x5596ef032840 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8019180 tx=0x7fe6c8004a40 comp rx=0 tx=0).stop 2024-04-01T01:35:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.476+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2829814404 shutdown_connections 2024-04-01T01:35:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.477+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/2829814404 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5596ef037000 0x5596ef0393f0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.477+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/2829814404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef026c40 0x5596ef032840 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.477+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/2829814404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef021ad0 0x5596ef02b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.477+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/2829814404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef0210e0 0x5596ef02b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.477+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2829814404 >> 172.21.15.70:0/2829814404 conn(0x5596ef013e40 msgr2=0x5596ef012f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.477+0000 7fe6b5ffb640 0 client.16368 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.477+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2829814404 shutdown_connections 2024-04-01T01:35:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.477+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2829814404 wait complete. 2024-04-01T01:35:57.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.478+0000 7fe6d7c2e740 20 client.16368 trim_cache size 0 max 16384 2024-04-01T01:35:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.502+0000 7fe6d7c2e740 1 Processor -- start 2024-04-01T01:35:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.502+0000 7fe6d7c2e740 1 -- start start 2024-04-01T01:35:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.503+0000 7fe6d7c2e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef0210e0 0x5596ef014220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.503+0000 7fe6d7c2e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef021ad0 0x5596ef032800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.503+0000 7fe6d7c2e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef026c40 0x5596ef03ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.503+0000 7fe6d7c2e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01a81f0 con 0x5596ef0210e0 2024-04-01T01:35:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.503+0000 7fe6d7c2e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01a9d30 con 0x5596ef026c40 2024-04-01T01:35:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.504+0000 7fe6d7c2e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01a88c0 con 0x5596ef021ad0 2024-04-01T01:35:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.504+0000 7fe6ced9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef026c40 0x5596ef03ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.504+0000 7fe6cdd9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef021ad0 0x5596ef032800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.504+0000 7fe6ced9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef026c40 0x5596ef03ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34840/0 (socket says 172.21.15.70:34840) 2024-04-01T01:35:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.504+0000 7fe6ced9d640 1 -- 172.21.15.70:0/2999778623 learned_addr learned my addr 172.21.15.70:0/2999778623 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.505+0000 7fe6ce59c640 1 --2- 172.21.15.70:0/2999778623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef0210e0 0x5596ef014220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.505+0000 7fe6ced9d640 1 -- 172.21.15.70:0/2999778623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef021ad0 msgr2=0x5596ef032800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.505+0000 7fe6ced9d640 1 --2- 172.21.15.70:0/2999778623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef021ad0 0x5596ef032800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.505+0000 7fe6ced9d640 1 -- 172.21.15.70:0/2999778623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef0210e0 msgr2=0x5596ef014220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.505+0000 7fe6ced9d640 1 --2- 172.21.15.70:0/2999778623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef0210e0 0x5596ef014220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.506 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.505+0000 7fe6ced9d640 1 -- 172.21.15.70:0/2999778623 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596ef02c440 con 0x5596ef026c40 2024-04-01T01:35:57.506 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.507+0000 7fe6ced9d640 1 --2- 172.21.15.70:0/2999778623 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef026c40 0x5596ef03ed00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe6c801e920 tx=0x7fe6c800ffd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12e320b72f656eec server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:57.506 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.507+0000 7fe6b7fff640 1 -- 172.21.15.70:0/2999778623 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c803d070 con 0x5596ef026c40 2024-04-01T01:35:57.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.508+0000 7fe6b7fff640 1 -- 172.21.15.70:0/2999778623 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6c8009570 con 0x5596ef026c40 2024-04-01T01:35:57.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.508+0000 7fe6b7fff640 1 -- 172.21.15.70:0/2999778623 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c802d040 con 0x5596ef026c40 2024-04-01T01:35:57.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.509+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2999778623 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef026c40 msgr2=0x5596ef03ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.509+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/2999778623 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef026c40 0x5596ef03ed00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe6c801e920 tx=0x7fe6c800ffd0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.510+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2999778623 shutdown_connections 2024-04-01T01:35:57.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.510+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/2999778623 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef026c40 0x5596ef03ed00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.510+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/2999778623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef021ad0 0x5596ef032800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.510+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/2999778623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef0210e0 0x5596ef014220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.510+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2999778623 >> 172.21.15.70:0/2999778623 conn(0x5596ef013e40 msgr2=0x5596ef026320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:57.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.510+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2999778623 shutdown_connections 2024-04-01T01:35:57.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.510+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2999778623 wait complete. 2024-04-01T01:35:57.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.511+0000 7fe6d7c2e740 1 -- start start 2024-04-01T01:35:57.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.511+0000 7fe6d7c2e740 1 Processor -- start 2024-04-01T01:35:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.512+0000 7fe6d7c2e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef0210e0 0x5596ef02f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.513+0000 7fe6d7c2e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef021ad0 0x5596ef02fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.513+0000 7fe6d7c2e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef026c40 0x5596ef09b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.513+0000 7fe6d7c2e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01a9660 con 0x5596ef026c40 2024-04-01T01:35:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.513+0000 7fe6cdd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef021ad0 0x5596ef02fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.513+0000 7fe6cdd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef021ad0 0x5596ef02fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34842/0 (socket says 172.21.15.70:34842) 2024-04-01T01:35:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.513+0000 7fe6cdd9b640 1 -- 172.21.15.70:0/2790687792 learned_addr learned my addr 172.21.15.70:0/2790687792 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.513+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2790687792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01a8f90 con 0x5596ef021ad0 2024-04-01T01:35:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.513+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2790687792 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01a61d0 con 0x5596ef0210e0 2024-04-01T01:35:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.513+0000 7fe6d7c2e740 10 client.0 _ll_register_callbacks cb 0x7fe6c8019040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.513+0000 7fe6ce59c640 1 --2- 172.21.15.70:0/2790687792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef0210e0 0x5596ef02f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.514+0000 7fe6cdd9b640 1 -- 172.21.15.70:0/2790687792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef0210e0 msgr2=0x5596ef02f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.514+0000 7fe6cdd9b640 1 --2- 172.21.15.70:0/2790687792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef0210e0 0x5596ef02f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.514+0000 7fe6cdd9b640 1 -- 172.21.15.70:0/2790687792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef026c40 msgr2=0x5596ef09b320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:35:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.514+0000 7fe6cdd9b640 1 --2- 172.21.15.70:0/2790687792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef026c40 0x5596ef09b320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.514+0000 7fe6cdd9b640 1 -- 172.21.15.70:0/2790687792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596ef09b860 con 0x5596ef021ad0 2024-04-01T01:35:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.514+0000 7fe6cdd9b640 1 --2- 172.21.15.70:0/2790687792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef021ad0 0x5596ef02fbe0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40a3910 tx=0x7fe6c409b020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77806300bdc7e41d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.514+0000 7fe6b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.514+0000 7fe6b5ffb640 1 -- 172.21.15.70:0/2790687792 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4097040 con 0x5596ef021ad0 2024-04-01T01:35:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.515+0000 7fe6b5ffb640 1 -- 172.21.15.70:0/2790687792 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe6c409f170 con 0x5596ef021ad0 2024-04-01T01:35:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.515+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2790687792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6c8009570 con 0x5596ef021ad0 2024-04-01T01:35:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.515+0000 7fe6b5ffb640 1 -- 172.21.15.70:0/2790687792 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe6c409f370 con 0x5596ef021ad0 2024-04-01T01:35:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.515+0000 7fe6d7c2e740 10 client.26213 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.515+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2790687792 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5596ef09ca80 con 0x5596ef021ad0 2024-04-01T01:35:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.516+0000 7fe6b5ffb640 1 -- 172.21.15.70:0/2790687792 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6c8009570 con 0x5596ef021ad0 2024-04-01T01:35:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.516+0000 7fe6b5ffb640 1 -- 172.21.15.70:0/2790687792 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe6c40a39b0 con 0x5596ef021ad0 2024-04-01T01:35:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.516+0000 7fe6d7c2e740 10 client.26213 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.516+0000 7fe6d7c2e740 10 client.26213 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.516+0000 7fe6d7c2e740 20 client.26213 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.516+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/2790687792 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5596ef09cff0 0x5596ef09f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.516+0000 7fe6d7c2e740 4 client.26213 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:35:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.516+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2790687792 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5596eef90120 con 0x5596ef09cff0 2024-04-01T01:35:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.516+0000 7fe6ce59c640 1 --2- 172.21.15.70:0/2790687792 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5596ef09cff0 0x5596ef09f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.518+0000 7fe6ce59c640 1 --2- 172.21.15.70:0/2790687792 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5596ef09cff0 0x5596ef09f3e0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d536d6ccb1e20e5 server_cookie=abd2c18983f80baf in_seq=0 out_seq=0 2024-04-01T01:35:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.518+0000 7fe6b5ffb640 10 client.26213 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:57.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.539+0000 7fe6b5ffb640 1 -- 172.21.15.70:0/2790687792 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5596eef90120 con 0x5596ef09cff0 2024-04-01T01:35:57.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.539+0000 7fe6b5ffb640 10 client.26213 handle_command_reply: tid=0 2024-04-01T01:35:57.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.539+0000 7fe6d7c2e740 1 client.26213 shutdown 2024-04-01T01:35:57.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.539+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2790687792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef021ad0 msgr2=0x5596ef02fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.539+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/2790687792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef021ad0 0x5596ef02fbe0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40a3910 tx=0x7fe6c409b020 comp rx=0 tx=0).stop 2024-04-01T01:35:57.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.540+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2790687792 shutdown_connections 2024-04-01T01:35:57.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.540+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/2790687792 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5596ef09cff0 0x5596ef09f3e0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.540+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/2790687792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5596ef026c40 0x5596ef09b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.540+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/2790687792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5596ef021ad0 0x5596ef02fbe0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.540+0000 7fe6d7c2e740 1 --2- 172.21.15.70:0/2790687792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5596ef0210e0 0x5596ef02f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.540+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2790687792 >> 172.21.15.70:0/2790687792 conn(0x5596ef013e40 msgr2=0x5596ef02b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:57.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.540+0000 7fe6b5ffb640 0 client.26213 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:57.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.540+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2790687792 shutdown_connections 2024-04-01T01:35:57.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.540+0000 7fe6d7c2e740 1 -- 172.21.15.70:0/2790687792 wait complete. 2024-04-01T01:35:57.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.540+0000 7fe6d7c2e740 20 client.26213 trim_cache size 0 max 16384 2024-04-01T01:35:57.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:35:57.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:35:57.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:35:57.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:35:57.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.541+0000 7fe6d4f6c640 1 -- 172.21.15.70:0/3902967006 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6b80a2640 msgr2=0x7fe6b80a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.541+0000 7fe6d4f6c640 1 --2- 172.21.15.70:0/3902967006 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6b80a2640 0x7fe6b80a4b00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x5596ef022330 tx=0x7fe6c8009500 comp rx=0 tx=0).stop 2024-04-01T01:35:57.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.541+0000 7fe6d4f6c640 1 -- 172.21.15.70:0/3902967006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6d01018a0 msgr2=0x7fe6d019a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:57.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.541+0000 7fe6d4f6c640 1 --2- 172.21.15.70:0/3902967006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6d01018a0 0x7fe6d019a0b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400e9f0 tx=0x7fe6c400eeb0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.541+0000 7fe6d4f6c640 1 -- 172.21.15.70:0/3902967006 shutdown_connections 2024-04-01T01:35:57.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.541+0000 7fe6d4f6c640 1 --2- 172.21.15.70:0/3902967006 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6b80a2640 0x7fe6b80a4b00 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.541+0000 7fe6d4f6c640 1 --2- 172.21.15.70:0/3902967006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d0102260 0x7fe6d01a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.541+0000 7fe6d4f6c640 1 --2- 172.21.15.70:0/3902967006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6d01018a0 0x7fe6d019a0b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.541+0000 7fe6d4f6c640 1 --2- 172.21.15.70:0/3902967006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6d0100b90 0x7fe6d0199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:57.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.541+0000 7fe6d4f6c640 1 -- 172.21.15.70:0/3902967006 >> 172.21.15.70:0/3902967006 conn(0x7fe6d00fb3a0 msgr2=0x7fe6d0104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:57.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.542+0000 7fe6d4f6c640 1 -- 172.21.15.70:0/3902967006 shutdown_connections 2024-04-01T01:35:57.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:35:57.542+0000 7fe6d4f6c640 1 -- 172.21.15.70:0/3902967006 wait complete. 2024-04-01T01:35:57.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:57 smithi189 ceph-mon[35753]: pgmap v432: 129 pgs: 129 active+clean; 970 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 13 MiB/s wr, 1.51k op/s 2024-04-01T01:35:57.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:57 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:35:57.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:57 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:35:57.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:57 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:35:58.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:57 smithi070 ceph-mon[32179]: pgmap v432: 129 pgs: 129 active+clean; 970 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 13 MiB/s wr, 1.51k op/s 2024-04-01T01:35:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:57 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:35:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:57 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:35:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:57 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:35:58.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:57 smithi081 ceph-mon[35142]: pgmap v432: 129 pgs: 129 active+clean; 970 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 13 MiB/s wr, 1.51k op/s 2024-04-01T01:35:58.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:57 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:35:58.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:57 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:35:58.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:57 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:35:58.219 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:35:58.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:35:58.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:35:58.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.410+0000 7fef2957a640 1 -- 172.21.15.70:0/2502434254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef241123d0 msgr2=0x7fef241147c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.410+0000 7fef2957a640 1 --2- 172.21.15.70:0/2502434254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef241123d0 0x7fef241147c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fef1c009ca0 tx=0x7fef1c02c120 comp rx=0 tx=0).stop 2024-04-01T01:35:58.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.411+0000 7fef2957a640 1 -- 172.21.15.70:0/2502434254 shutdown_connections 2024-04-01T01:35:58.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.411+0000 7fef2957a640 1 --2- 172.21.15.70:0/2502434254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef241123d0 0x7fef241147c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.411+0000 7fef2957a640 1 --2- 172.21.15.70:0/2502434254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef241090b0 0x7fef24111e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.411+0000 7fef2957a640 1 --2- 172.21.15.70:0/2502434254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef24108750 0x7fef24108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.411+0000 7fef2957a640 1 -- 172.21.15.70:0/2502434254 >> 172.21.15.70:0/2502434254 conn(0x7fef2406e950 msgr2=0x7fef24070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:58.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.411+0000 7fef2957a640 1 -- 172.21.15.70:0/2502434254 shutdown_connections 2024-04-01T01:35:58.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.411+0000 7fef2957a640 1 -- 172.21.15.70:0/2502434254 wait complete. 2024-04-01T01:35:58.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.411+0000 7fef2957a640 1 Processor -- start 2024-04-01T01:35:58.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.412+0000 7fef2957a640 1 -- start start 2024-04-01T01:35:58.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.412+0000 7fef2957a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef24108750 0x7fef2411c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.412+0000 7fef2957a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef241090b0 0x7fef2411cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.412+0000 7fef2957a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef241123d0 0x7fef24119300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.412+0000 7fef2957a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef2410ea50 con 0x7fef241090b0 2024-04-01T01:35:58.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.412+0000 7fef2957a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fef2410e8d0 con 0x7fef241123d0 2024-04-01T01:35:58.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.412+0000 7fef2957a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fef2410ebd0 con 0x7fef24108750 2024-04-01T01:35:58.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.412+0000 7fef227fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef241090b0 0x7fef2411cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.412+0000 7fef227fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef241090b0 0x7fef2411cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52820/0 (socket says 172.21.15.70:52820) 2024-04-01T01:35:58.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.412+0000 7fef227fc640 1 -- 172.21.15.70:0/3074075133 learned_addr learned my addr 172.21.15.70:0/3074075133 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:58.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.413+0000 7fef237fe640 1 --2- 172.21.15.70:0/3074075133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef241123d0 0x7fef24119300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.413+0000 7fef22ffd640 1 --2- 172.21.15.70:0/3074075133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef24108750 0x7fef2411c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.413+0000 7fef227fc640 1 -- 172.21.15.70:0/3074075133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef24108750 msgr2=0x7fef2411c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.413+0000 7fef227fc640 1 --2- 172.21.15.70:0/3074075133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef24108750 0x7fef2411c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.413+0000 7fef227fc640 1 -- 172.21.15.70:0/3074075133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef241123d0 msgr2=0x7fef24119300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.413+0000 7fef227fc640 1 --2- 172.21.15.70:0/3074075133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef241123d0 0x7fef24119300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.413+0000 7fef227fc640 1 -- 172.21.15.70:0/3074075133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef24119ad0 con 0x7fef241090b0 2024-04-01T01:35:58.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.413+0000 7fef227fc640 1 --2- 172.21.15.70:0/3074075133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef241090b0 0x7fef2411cd80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fef0c00b450 tx=0x7fef0c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:58.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.414+0000 7fef1bfff640 1 -- 172.21.15.70:0/3074075133 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c015020 con 0x7fef241090b0 2024-04-01T01:35:58.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.414+0000 7fef2957a640 1 -- 172.21.15.70:0/3074075133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef24119dc0 con 0x7fef241090b0 2024-04-01T01:35:58.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.414+0000 7fef1bfff640 1 -- 172.21.15.70:0/3074075133 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef0c011d40 con 0x7fef241090b0 2024-04-01T01:35:58.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.414+0000 7fef1bfff640 1 -- 172.21.15.70:0/3074075133 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c00f340 con 0x7fef241090b0 2024-04-01T01:35:58.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.415+0000 7fef2957a640 1 -- 172.21.15.70:0/3074075133 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef241b74e0 con 0x7fef241090b0 2024-04-01T01:35:58.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.418+0000 7fef1bfff640 1 -- 172.21.15.70:0/3074075133 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fef0c018020 con 0x7fef241090b0 2024-04-01T01:35:58.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.419+0000 7fef2c238740 1 Processor -- start 2024-04-01T01:35:58.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.419+0000 7fef2c238740 1 -- start start 2024-04-01T01:35:58.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.419+0000 7fef1bfff640 1 --2- 172.21.15.70:0/3074075133 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fef100a2520 0x7fef100a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.419+0000 7fef1bfff640 1 -- 172.21.15.70:0/3074075133 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fef0c0ce480 con 0x7fef241090b0 2024-04-01T01:35:58.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.419+0000 7fef2c238740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa79a10 0x55f0baa79e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.419+0000 7fef2c238740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa7a400 0x55f0baa7ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.420+0000 7fef2c238740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa7f570 0x55f0baa81a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.420+0000 7fef2c238740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f0ba97b0b0 con 0x55f0baa7a400 2024-04-01T01:35:58.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.420+0000 7fef2c238740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f0ba9ddea0 con 0x55f0baa7f570 2024-04-01T01:35:58.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.420+0000 7fef2c238740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f0ba986c00 con 0x55f0baa79a10 2024-04-01T01:35:58.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.420+0000 7fef237fe640 1 --2- 172.21.15.70:0/3074075133 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fef100a2520 0x7fef100a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.420+0000 7fef227fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa7a400 0x55f0baa7ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.420+0000 7fef227fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa7a400 0x55f0baa7ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52822/0 (socket says 172.21.15.70:52822) 2024-04-01T01:35:58.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.420+0000 7fef227fc640 1 -- 172.21.15.70:0/2704722500 learned_addr learned my addr 172.21.15.70:0/2704722500 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:58.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.420+0000 7fef22ffd640 1 --2- 172.21.15.70:0/2704722500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa79a10 0x55f0baa79e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.420+0000 7fef237fe640 1 --2- 172.21.15.70:0/2704722500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa7f570 0x55f0baa81a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.420+0000 7fef22ffd640 1 -- 172.21.15.70:0/2704722500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa7f570 msgr2=0x55f0baa81a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:58.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.420+0000 7fef22ffd640 1 --2- 172.21.15.70:0/2704722500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa7f570 0x55f0baa81a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.420+0000 7fef22ffd640 1 -- 172.21.15.70:0/2704722500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa7a400 msgr2=0x55f0baa7ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:58.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.421+0000 7fef22ffd640 1 --2- 172.21.15.70:0/2704722500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa7a400 0x55f0baa7ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.421+0000 7fef22ffd640 1 -- 172.21.15.70:0/2704722500 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0ba7ba7e0 con 0x55f0baa79a10 2024-04-01T01:35:58.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.421+0000 7fef237fe640 1 --2- 172.21.15.70:0/2704722500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa7f570 0x55f0baa81a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:35:58.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.421+0000 7fef227fc640 1 --2- 172.21.15.70:0/2704722500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa7a400 0x55f0baa7ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:58.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.421+0000 7fef237fe640 1 --2- 172.21.15.70:0/3074075133 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fef100a2520 0x7fef100a49e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fef1c02c660 tx=0x7fef1c032d40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:58.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.422+0000 7fef22ffd640 1 --2- 172.21.15.70:0/2704722500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa79a10 0x55f0baa79e30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fef1400bbf0 tx=0x7fef140107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29e399c1937924b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:58.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.422+0000 7fef197fa640 1 -- 172.21.15.70:0/2704722500 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef14004030 con 0x55f0baa79a10 2024-04-01T01:35:58.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.423+0000 7fef197fa640 1 -- 172.21.15.70:0/2704722500 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef140035f0 con 0x55f0baa79a10 2024-04-01T01:35:58.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.423+0000 7fef197fa640 1 -- 172.21.15.70:0/2704722500 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef14011680 con 0x55f0baa79a10 2024-04-01T01:35:58.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.424+0000 7fef2c238740 1 -- 172.21.15.70:0/2704722500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa79a10 msgr2=0x55f0baa79e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.424+0000 7fef2c238740 1 --2- 172.21.15.70:0/2704722500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa79a10 0x55f0baa79e30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fef1400bbf0 tx=0x7fef140107a0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.425+0000 7fef2c238740 1 -- 172.21.15.70:0/2704722500 shutdown_connections 2024-04-01T01:35:58.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.425+0000 7fef2c238740 1 --2- 172.21.15.70:0/2704722500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa7f570 0x55f0baa81a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.425+0000 7fef2c238740 1 --2- 172.21.15.70:0/2704722500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa7a400 0x55f0baa7ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.425+0000 7fef2c238740 1 --2- 172.21.15.70:0/2704722500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa79a10 0x55f0baa79e30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.425+0000 7fef2c238740 1 -- 172.21.15.70:0/2704722500 >> 172.21.15.70:0/2704722500 conn(0x55f0baa74f80 msgr2=0x55f0baa75380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:58.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.425+0000 7fef2c238740 1 -- 172.21.15.70:0/2704722500 shutdown_connections 2024-04-01T01:35:58.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.425+0000 7fef2c238740 1 -- 172.21.15.70:0/2704722500 wait complete. 2024-04-01T01:35:58.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.426+0000 7fef2c238740 1 -- start start 2024-04-01T01:35:58.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.426+0000 7fef2c238740 1 Processor -- start 2024-04-01T01:35:58.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.427+0000 7fef2c238740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa79a10 0x55f0baa83d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.427+0000 7fef2c238740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa7a400 0x55f0baa842c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.427+0000 7fef2c238740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa7f570 0x55f0baa8b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.427+0000 7fef2c238740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f0ba97c3d0 con 0x55f0baa79a10 2024-04-01T01:35:58.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.427+0000 7fef227fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa7a400 0x55f0baa842c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.427+0000 7fef227fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa7a400 0x55f0baa842c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49772/0 (socket says 172.21.15.70:49772) 2024-04-01T01:35:58.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.427+0000 7fef227fc640 1 -- 172.21.15.70:0/3327408050 learned_addr learned my addr 172.21.15.70:0/3327408050 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:58.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.427+0000 7fef237fe640 1 --2- 172.21.15.70:0/3327408050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa7f570 0x55f0baa8b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.427+0000 7fef2c238740 1 -- 172.21.15.70:0/3327408050 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f0ba9d9f40 con 0x55f0baa7f570 2024-04-01T01:35:58.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.427+0000 7fef22ffd640 1 --2- 172.21.15.70:0/3327408050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa79a10 0x55f0baa83d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.428+0000 7fef2c238740 1 -- 172.21.15.70:0/3327408050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f0baa82520 con 0x55f0baa7a400 2024-04-01T01:35:58.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.428+0000 7fef2c238740 10 client.0 _ll_register_callbacks cb 0x55f0ba999b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:58.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.428+0000 7fef227fc640 1 -- 172.21.15.70:0/3327408050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa7f570 msgr2=0x55f0baa8b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:58.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.428+0000 7fef227fc640 1 --2- 172.21.15.70:0/3327408050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa7f570 0x55f0baa8b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.428+0000 7fef227fc640 1 -- 172.21.15.70:0/3327408050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa79a10 msgr2=0x55f0baa83d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:58.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.428+0000 7fef227fc640 1 --2- 172.21.15.70:0/3327408050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa79a10 0x55f0baa83d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.428+0000 7fef227fc640 1 -- 172.21.15.70:0/3327408050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0ba81af70 con 0x55f0baa7a400 2024-04-01T01:35:58.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.428+0000 7fef227fc640 1 --2- 172.21.15.70:0/3327408050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa7a400 0x55f0baa842c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fef0c0cbf20 tx=0x7fef0c09a600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdc7332a6ac6ed5a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:58.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.429+0000 7fef0a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:35:58.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.429+0000 7fef0a7fc640 1 -- 172.21.15.70:0/3327408050 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c0b2020 con 0x55f0baa7a400 2024-04-01T01:35:58.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.429+0000 7fef0a7fc640 1 -- 172.21.15.70:0/3327408050 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fef0c011890 con 0x55f0baa7a400 2024-04-01T01:35:58.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.430+0000 7fef2c238740 1 -- 172.21.15.70:0/3327408050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef140035f0 con 0x55f0baa7a400 2024-04-01T01:35:58.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.430+0000 7fef0a7fc640 1 -- 172.21.15.70:0/3327408050 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fef0c09ab70 con 0x55f0baa7a400 2024-04-01T01:35:58.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.430+0000 7fef2c238740 10 client.26122 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:58.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.430+0000 7fef2c238740 1 -- 172.21.15.70:0/3327408050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f0ba6b2170 con 0x55f0baa7a400 2024-04-01T01:35:58.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.431+0000 7fef0a7fc640 1 -- 172.21.15.70:0/3327408050 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef0c0b9300 con 0x55f0baa7a400 2024-04-01T01:35:58.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.431+0000 7fef0a7fc640 1 -- 172.21.15.70:0/3327408050 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fef0c0b3b10 con 0x55f0baa7a400 2024-04-01T01:35:58.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.431+0000 7fef2c238740 10 client.26122 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:58.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.431+0000 7fef2c238740 10 client.26122 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:58.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.431+0000 7fef2c238740 20 client.26122 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:58.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.431+0000 7fef2c238740 1 --2- 172.21.15.70:0/3327408050 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f0baa8f7d0 0x55f0baa91c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.431+0000 7fef2c238740 4 client.26122 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:58.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.431+0000 7fef2c238740 1 -- 172.21.15.70:0/3327408050 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fef140035f0 con 0x55f0baa8f7d0 2024-04-01T01:35:58.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.431+0000 7fef22ffd640 1 --2- 172.21.15.70:0/3327408050 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f0baa8f7d0 0x55f0baa91c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.433+0000 7fef22ffd640 1 --2- 172.21.15.70:0/3327408050 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f0baa8f7d0 0x55f0baa91c40 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cea6c5399324d395 server_cookie=2702a431a0c9761f in_seq=0 out_seq=0 2024-04-01T01:35:58.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.433+0000 7fef0a7fc640 10 client.26122 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:58.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.434+0000 7fef0a7fc640 1 -- 172.21.15.70:0/3327408050 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fef140035f0 con 0x55f0baa8f7d0 2024-04-01T01:35:58.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.434+0000 7fef0a7fc640 10 client.26122 handle_command_reply: tid=0 2024-04-01T01:35:58.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.434+0000 7fef2c238740 1 client.26122 shutdown 2024-04-01T01:35:58.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.434+0000 7fef2c238740 1 -- 172.21.15.70:0/3327408050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa7a400 msgr2=0x55f0baa842c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.434+0000 7fef2c238740 1 --2- 172.21.15.70:0/3327408050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa7a400 0x55f0baa842c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fef0c0cbf20 tx=0x7fef0c09a600 comp rx=0 tx=0).stop 2024-04-01T01:35:58.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.434+0000 7fef2c238740 1 -- 172.21.15.70:0/3327408050 shutdown_connections 2024-04-01T01:35:58.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.434+0000 7fef2c238740 1 --2- 172.21.15.70:0/3327408050 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f0baa8f7d0 0x55f0baa91c40 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.434+0000 7fef2c238740 1 --2- 172.21.15.70:0/3327408050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa7f570 0x55f0baa8b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.434+0000 7fef2c238740 1 --2- 172.21.15.70:0/3327408050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa7a400 0x55f0baa842c0 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fef0c0cbf20 tx=0x7fef0c09a600 comp rx=0 tx=0).stop 2024-04-01T01:35:58.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.434+0000 7fef2c238740 1 --2- 172.21.15.70:0/3327408050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa79a10 0x55f0baa83d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.434+0000 7fef2c238740 1 -- 172.21.15.70:0/3327408050 >> 172.21.15.70:0/3327408050 conn(0x55f0baa74f80 msgr2=0x55f0baa74180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:58.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.434+0000 7fef0a7fc640 0 client.26122 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:58.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.436+0000 7fef2c238740 1 -- 172.21.15.70:0/3327408050 shutdown_connections 2024-04-01T01:35:58.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.436+0000 7fef2c238740 1 -- 172.21.15.70:0/3327408050 wait complete. 2024-04-01T01:35:58.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.436+0000 7fef2c238740 20 client.26122 trim_cache size 0 max 16384 2024-04-01T01:35:58.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.469+0000 7fef2c238740 1 Processor -- start 2024-04-01T01:35:58.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.470+0000 7fef2c238740 1 -- start start 2024-04-01T01:35:58.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.470+0000 7fef2c238740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa79a10 0x55f0baa828c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.470+0000 7fef2c238740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa7f570 0x55f0baa8b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.471+0000 7fef2c238740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa82e00 0x55f0baa83270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.471+0000 7fef22ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa79a10 0x55f0baa828c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.471+0000 7fef22ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa79a10 0x55f0baa828c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34906/0 (socket says 172.21.15.70:34906) 2024-04-01T01:35:58.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.471+0000 7fef22ffd640 1 -- 172.21.15.70:0/3017987574 learned_addr learned my addr 172.21.15.70:0/3017987574 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:58.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.471+0000 7fef2c238740 1 -- 172.21.15.70:0/3017987574 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef241c1ab0 con 0x55f0baa82e00 2024-04-01T01:35:58.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.471+0000 7fef237fe640 1 --2- 172.21.15.70:0/3017987574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa82e00 0x55f0baa83270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.471+0000 7fef227fc640 1 --2- 172.21.15.70:0/3017987574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa7f570 0x55f0baa8b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.471+0000 7fef2c238740 1 -- 172.21.15.70:0/3017987574 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fef241c35f0 con 0x55f0baa79a10 2024-04-01T01:35:58.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.471+0000 7fef2c238740 1 -- 172.21.15.70:0/3017987574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fef241c2180 con 0x55f0baa7f570 2024-04-01T01:35:58.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.472+0000 7fef22ffd640 1 -- 172.21.15.70:0/3017987574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa7f570 msgr2=0x55f0baa8b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:58.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.472+0000 7fef22ffd640 1 --2- 172.21.15.70:0/3017987574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa7f570 0x55f0baa8b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.472+0000 7fef22ffd640 1 -- 172.21.15.70:0/3017987574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa82e00 msgr2=0x55f0baa83270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:58.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.472+0000 7fef22ffd640 1 --2- 172.21.15.70:0/3017987574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa82e00 0x55f0baa83270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.472+0000 7fef22ffd640 1 -- 172.21.15.70:0/3017987574 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0baa759f0 con 0x55f0baa79a10 2024-04-01T01:35:58.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.472+0000 7fef227fc640 1 --2- 172.21.15.70:0/3017987574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa7f570 0x55f0baa8b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:35:58.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.472+0000 7fef22ffd640 1 --2- 172.21.15.70:0/3017987574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa79a10 0x55f0baa828c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fef1400bbf0 tx=0x7fef1400b590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c636b465ca2371a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:58.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.472+0000 7fef18ff9640 1 -- 172.21.15.70:0/3017987574 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef140038a0 con 0x55f0baa79a10 2024-04-01T01:35:58.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.473+0000 7fef18ff9640 1 -- 172.21.15.70:0/3017987574 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef1402a070 con 0x55f0baa79a10 2024-04-01T01:35:58.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.473+0000 7fef18ff9640 1 -- 172.21.15.70:0/3017987574 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef140055a0 con 0x55f0baa79a10 2024-04-01T01:35:58.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.474+0000 7fef2c238740 1 -- 172.21.15.70:0/3017987574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa79a10 msgr2=0x55f0baa828c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.474+0000 7fef2c238740 1 --2- 172.21.15.70:0/3017987574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa79a10 0x55f0baa828c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fef1400bbf0 tx=0x7fef1400b590 comp rx=0 tx=0).stop 2024-04-01T01:35:58.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.474+0000 7fef2c238740 1 -- 172.21.15.70:0/3017987574 shutdown_connections 2024-04-01T01:35:58.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.474+0000 7fef2c238740 1 --2- 172.21.15.70:0/3017987574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa82e00 0x55f0baa83270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.474+0000 7fef2c238740 1 --2- 172.21.15.70:0/3017987574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa7f570 0x55f0baa8b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.474+0000 7fef2c238740 1 --2- 172.21.15.70:0/3017987574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa79a10 0x55f0baa828c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.474+0000 7fef2c238740 1 -- 172.21.15.70:0/3017987574 >> 172.21.15.70:0/3017987574 conn(0x55f0baa74f80 msgr2=0x55f0baa74180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:58.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.474+0000 7fef2c238740 1 -- 172.21.15.70:0/3017987574 shutdown_connections 2024-04-01T01:35:58.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.475+0000 7fef2c238740 1 -- 172.21.15.70:0/3017987574 wait complete. 2024-04-01T01:35:58.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.475+0000 7fef2c238740 1 -- start start 2024-04-01T01:35:58.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.475+0000 7fef2c238740 1 Processor -- start 2024-04-01T01:35:58.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef2c238740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa79a10 0x55f0baa8ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef2c238740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa7f570 0x55f0baa893f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef2c238740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa82e00 0x55f0baa89930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef2c238740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef241c2f20 con 0x55f0baa7f570 2024-04-01T01:35:58.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef2c238740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fef241c2850 con 0x55f0baa82e00 2024-04-01T01:35:58.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef2c238740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fef241bfa90 con 0x55f0baa79a10 2024-04-01T01:35:58.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef237fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa82e00 0x55f0baa89930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef22ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa79a10 0x55f0baa8ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef237fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa82e00 0x55f0baa89930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34910/0 (socket says 172.21.15.70:34910) 2024-04-01T01:35:58.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef237fe640 1 -- 172.21.15.70:0/1453467331 learned_addr learned my addr 172.21.15.70:0/1453467331 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:58.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef2c238740 10 client.0 _ll_register_callbacks cb 0x7fef0c0ce040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:58.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef227fc640 1 --2- 172.21.15.70:0/1453467331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa7f570 0x55f0baa893f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef237fe640 1 -- 172.21.15.70:0/1453467331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa79a10 msgr2=0x55f0baa8ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:58.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef237fe640 1 --2- 172.21.15.70:0/1453467331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa79a10 0x55f0baa8ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef237fe640 1 -- 172.21.15.70:0/1453467331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa7f570 msgr2=0x55f0baa893f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:58.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef237fe640 1 --2- 172.21.15.70:0/1453467331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa7f570 0x55f0baa893f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.476+0000 7fef237fe640 1 -- 172.21.15.70:0/1453467331 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0baa89e70 con 0x55f0baa82e00 2024-04-01T01:35:58.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.477+0000 7fef237fe640 1 --2- 172.21.15.70:0/1453467331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa82e00 0x55f0baa89930 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fef1c032380 tx=0x7fef1c02c120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c302218a75cb2bc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:58.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.477+0000 7fef0a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:58.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.477+0000 7fef0a7fc640 1 -- 172.21.15.70:0/1453467331 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c039040 con 0x55f0baa82e00 2024-04-01T01:35:58.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.477+0000 7fef2c238740 1 -- 172.21.15.70:0/1453467331 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef1402a070 con 0x55f0baa82e00 2024-04-01T01:35:58.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.477+0000 7fef0a7fc640 1 -- 172.21.15.70:0/1453467331 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fef1c004240 con 0x55f0baa82e00 2024-04-01T01:35:58.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.477+0000 7fef0a7fc640 1 -- 172.21.15.70:0/1453467331 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fef1c05b050 con 0x55f0baa82e00 2024-04-01T01:35:58.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.477+0000 7fef2c238740 10 client.26228 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:58.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.477+0000 7fef2c238740 1 -- 172.21.15.70:0/1453467331 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f0baaf6890 con 0x55f0baa82e00 2024-04-01T01:35:58.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.478+0000 7fef0a7fc640 1 -- 172.21.15.70:0/1453467331 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef1402a070 con 0x55f0baa82e00 2024-04-01T01:35:58.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.478+0000 7fef0a7fc640 1 -- 172.21.15.70:0/1453467331 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fef1c034a20 con 0x55f0baa82e00 2024-04-01T01:35:58.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.478+0000 7fef2c238740 10 client.26228 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:58.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.478+0000 7fef2c238740 10 client.26228 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:58.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.478+0000 7fef2c238740 20 client.26228 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:58.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.478+0000 7fef2c238740 1 --2- 172.21.15.70:0/1453467331 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f0baaf7270 0x55f0baaf9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.478+0000 7fef2c238740 4 client.26228 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:35:58.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.478+0000 7fef2c238740 1 -- 172.21.15.70:0/1453467331 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f0ba9eb3e0 con 0x55f0baaf7270 2024-04-01T01:35:58.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.478+0000 7fef22ffd640 1 --2- 172.21.15.70:0/1453467331 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f0baaf7270 0x55f0baaf9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.479+0000 7fef22ffd640 1 --2- 172.21.15.70:0/1453467331 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f0baaf7270 0x55f0baaf9690 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8772aeb756907c60 server_cookie=5229268acea12a0 in_seq=0 out_seq=0 2024-04-01T01:35:58.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.479+0000 7fef0a7fc640 10 client.26228 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:58.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.480+0000 7fef0a7fc640 1 -- 172.21.15.70:0/1453467331 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f0ba9eb3e0 con 0x55f0baaf7270 2024-04-01T01:35:58.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.480+0000 7fef0a7fc640 10 client.26228 handle_command_reply: tid=0 2024-04-01T01:35:58.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.480+0000 7fef2c238740 1 client.26228 shutdown 2024-04-01T01:35:58.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.481+0000 7fef2c238740 1 -- 172.21.15.70:0/1453467331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa82e00 msgr2=0x55f0baa89930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.481+0000 7fef2c238740 1 --2- 172.21.15.70:0/1453467331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa82e00 0x55f0baa89930 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fef1c032380 tx=0x7fef1c02c120 comp rx=0 tx=0).stop 2024-04-01T01:35:58.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.481+0000 7fef2c238740 1 -- 172.21.15.70:0/1453467331 shutdown_connections 2024-04-01T01:35:58.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.481+0000 7fef2c238740 1 --2- 172.21.15.70:0/1453467331 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f0baaf7270 0x55f0baaf9690 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.481+0000 7fef2c238740 1 --2- 172.21.15.70:0/1453467331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f0baa82e00 0x55f0baa89930 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.481+0000 7fef2c238740 1 --2- 172.21.15.70:0/1453467331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f0baa7f570 0x55f0baa893f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.481+0000 7fef2c238740 1 --2- 172.21.15.70:0/1453467331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0baa79a10 0x55f0baa8ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.481+0000 7fef2c238740 1 -- 172.21.15.70:0/1453467331 >> 172.21.15.70:0/1453467331 conn(0x55f0baa74f80 msgr2=0x55f0baa781e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:58.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.481+0000 7fef0a7fc640 0 client.26228 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:58.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.481+0000 7fef2c238740 1 -- 172.21.15.70:0/1453467331 shutdown_connections 2024-04-01T01:35:58.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.481+0000 7fef2c238740 1 -- 172.21.15.70:0/1453467331 wait complete. 2024-04-01T01:35:58.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.481+0000 7fef2c238740 20 client.26228 trim_cache size 0 max 16384 2024-04-01T01:35:58.482 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:35:58.482 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:35:58.482 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "4341346d-3ac8-463c-9010-6169bfdac470", 2024-04-01T01:35:58.482 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:35:58.482 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:35:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.482+0000 7fef2957a640 1 -- 172.21.15.70:0/3074075133 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fef100a2520 msgr2=0x7fef100a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.482+0000 7fef2957a640 1 --2- 172.21.15.70:0/3074075133 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fef100a2520 0x7fef100a49e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fef1c02c660 tx=0x7fef1c032d40 comp rx=0 tx=0).stop 2024-04-01T01:35:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.483+0000 7fef2957a640 1 -- 172.21.15.70:0/3074075133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef241090b0 msgr2=0x7fef2411cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.483+0000 7fef2957a640 1 --2- 172.21.15.70:0/3074075133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef241090b0 0x7fef2411cd80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fef0c00b450 tx=0x7fef0c00b910 comp rx=0 tx=0).stop 2024-04-01T01:35:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.483+0000 7fef2957a640 1 -- 172.21.15.70:0/3074075133 shutdown_connections 2024-04-01T01:35:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.483+0000 7fef2957a640 1 --2- 172.21.15.70:0/3074075133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef241123d0 0x7fef24119300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.483+0000 7fef2957a640 1 --2- 172.21.15.70:0/3074075133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef241090b0 0x7fef2411cd80 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.483+0000 7fef2957a640 1 --2- 172.21.15.70:0/3074075133 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fef100a2520 0x7fef100a49e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.483+0000 7fef2957a640 1 --2- 172.21.15.70:0/3074075133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef24108750 0x7fef2411c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.483+0000 7fef2957a640 1 -- 172.21.15.70:0/3074075133 >> 172.21.15.70:0/3074075133 conn(0x7fef2406e950 msgr2=0x7fef2406f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:58.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.484+0000 7fef2957a640 1 -- 172.21.15.70:0/3074075133 shutdown_connections 2024-04-01T01:35:58.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.484+0000 7fef2957a640 1 -- 172.21.15.70:0/3074075133 wait complete. 2024-04-01T01:35:58.505 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:35:58.505 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:35:58.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.690+0000 7fac8619a640 1 -- 172.21.15.70:0/276050000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac80100000 msgr2=0x7fac80102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.690+0000 7fac8619a640 1 --2- 172.21.15.70:0/276050000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac80100000 0x7fac80102430 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fac68009860 tx=0x7fac6802bb70 comp rx=0 tx=0).stop 2024-04-01T01:35:58.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.690+0000 7fac8619a640 1 -- 172.21.15.70:0/276050000 shutdown_connections 2024-04-01T01:35:58.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.690+0000 7fac8619a640 1 --2- 172.21.15.70:0/276050000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac80105320 0x7fac801057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.690+0000 7fac8619a640 1 --2- 172.21.15.70:0/276050000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac80102970 0x7fac80104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.690+0000 7fac8619a640 1 --2- 172.21.15.70:0/276050000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac80100000 0x7fac80102430 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.690+0000 7fac8619a640 1 -- 172.21.15.70:0/276050000 >> 172.21.15.70:0/276050000 conn(0x7fac800fbc00 msgr2=0x7fac800fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:58.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.690+0000 7fac8619a640 1 -- 172.21.15.70:0/276050000 shutdown_connections 2024-04-01T01:35:58.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.690+0000 7fac8619a640 1 -- 172.21.15.70:0/276050000 wait complete. 2024-04-01T01:35:58.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.691+0000 7fac8619a640 1 Processor -- start 2024-04-01T01:35:58.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.691+0000 7fac8619a640 1 -- start start 2024-04-01T01:35:58.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.692+0000 7fac8619a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac80100000 0x7fac8019e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.692+0000 7fac8619a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac80102970 0x7fac8019e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.692+0000 7fac8619a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac80105320 0x7fac801a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.692+0000 7fac8619a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fac801138e0 con 0x7fac80100000 2024-04-01T01:35:58.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.692+0000 7fac8619a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fac80113760 con 0x7fac80105320 2024-04-01T01:35:58.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.692+0000 7fac8619a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fac80113a60 con 0x7fac80102970 2024-04-01T01:35:58.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.692+0000 7fac7f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac80100000 0x7fac8019e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.692+0000 7fac7f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac80100000 0x7fac8019e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52856/0 (socket says 172.21.15.70:52856) 2024-04-01T01:35:58.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.692+0000 7fac7f7fe640 1 -- 172.21.15.70:0/3609336507 learned_addr learned my addr 172.21.15.70:0/3609336507 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:58.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.692+0000 7fac7ffff640 1 --2- 172.21.15.70:0/3609336507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac80105320 0x7fac801a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.692+0000 7fac77fff640 1 --2- 172.21.15.70:0/3609336507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac80102970 0x7fac8019e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.693+0000 7fac77fff640 1 -- 172.21.15.70:0/3609336507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac80105320 msgr2=0x7fac801a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.693+0000 7fac77fff640 1 --2- 172.21.15.70:0/3609336507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac80105320 0x7fac801a5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.693+0000 7fac77fff640 1 -- 172.21.15.70:0/3609336507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac80100000 msgr2=0x7fac8019e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.693+0000 7fac77fff640 1 --2- 172.21.15.70:0/3609336507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac80100000 0x7fac8019e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.693+0000 7fac77fff640 1 -- 172.21.15.70:0/3609336507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac801a5c80 con 0x7fac80102970 2024-04-01T01:35:58.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.693+0000 7fac7f7fe640 1 --2- 172.21.15.70:0/3609336507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac80100000 0x7fac8019e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:58.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.693+0000 7fac77fff640 1 --2- 172.21.15.70:0/3609336507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac80102970 0x7fac8019e5a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fac7000e790 tx=0x7fac7000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:58.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.693+0000 7fac7d7fa640 1 -- 172.21.15.70:0/3609336507 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7000eea0 con 0x7fac80102970 2024-04-01T01:35:58.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.694+0000 7fac7d7fa640 1 -- 172.21.15.70:0/3609336507 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac7000ccc0 con 0x7fac80102970 2024-04-01T01:35:58.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.694+0000 7fac8619a640 1 -- 172.21.15.70:0/3609336507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac801a5f70 con 0x7fac80102970 2024-04-01T01:35:58.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.694+0000 7fac7d7fa640 1 -- 172.21.15.70:0/3609336507 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7000eea0 con 0x7fac80102970 2024-04-01T01:35:58.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.694+0000 7fac8619a640 1 -- 172.21.15.70:0/3609336507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac801a63e0 con 0x7fac80102970 2024-04-01T01:35:58.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.698+0000 7fac88d7e740 1 Processor -- start 2024-04-01T01:35:58.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.698+0000 7fac88d7e740 1 -- start start 2024-04-01T01:35:58.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.698+0000 7fac88d7e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd92b0 0x561042cd96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.698+0000 7fac88d7e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 0x561042cde7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.698+0000 7fac88d7e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cdee10 0x561042ce12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.698+0000 7fac88d7e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561042c3e180 con 0x561042cdee10 2024-04-01T01:35:58.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.698+0000 7fac88d7e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561042be6b70 con 0x561042cd9ca0 2024-04-01T01:35:58.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.698+0000 7fac88d7e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561042c3a220 con 0x561042cd92b0 2024-04-01T01:35:58.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.698+0000 7fac7d7fa640 1 -- 172.21.15.70:0/3609336507 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fac70010430 con 0x7fac80102970 2024-04-01T01:35:58.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.699+0000 7fac7ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd92b0 0x561042cd96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.699+0000 7fac7ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd92b0 0x561042cd96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49818/0 (socket says 172.21.15.70:49818) 2024-04-01T01:35:58.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.699+0000 7fac7ffff640 1 -- 172.21.15.70:0/2106559427 learned_addr learned my addr 172.21.15.70:0/2106559427 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:58.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.699+0000 7fac7f7fe640 1 --2- 172.21.15.70:0/2106559427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 0x561042cde7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.699+0000 7fac7f7fe640 1 -- 172.21.15.70:0/2106559427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd92b0 msgr2=0x561042cd96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:58.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.699+0000 7fac7f7fe640 1 --2- 172.21.15.70:0/2106559427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd92b0 0x561042cd96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.699+0000 7fac7f7fe640 1 -- 172.21.15.70:0/2106559427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cdee10 msgr2=0x561042ce12a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:35:58.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.699+0000 7fac7d7fa640 1 --2- 172.21.15.70:0/3609336507 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac6c0a26e0 0x7fac6c0a4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.699+0000 7fac7f7fe640 1 --2- 172.21.15.70:0/2106559427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cdee10 0x561042ce12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.699+0000 7fac7f7fe640 1 -- 172.21.15.70:0/2106559427 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561042bfa7c0 con 0x561042cd9ca0 2024-04-01T01:35:58.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.699+0000 7fac7ffff640 1 --2- 172.21.15.70:0/2106559427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd92b0 0x561042cd96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:58.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.699+0000 7fac7d7fa640 1 -- 172.21.15.70:0/3609336507 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fac700ce9e0 con 0x7fac80102970 2024-04-01T01:35:58.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.700+0000 7fac7f7fe640 1 --2- 172.21.15.70:0/2106559427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 0x561042cde7a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fac6802c180 tx=0x7fac68009490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d819a4ad96d67c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:58.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.700+0000 7fac7ffff640 1 --2- 172.21.15.70:0/3609336507 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac6c0a26e0 0x7fac6c0a4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.700+0000 7fac75ffb640 1 -- 172.21.15.70:0/2106559427 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac68043020 con 0x561042cd9ca0 2024-04-01T01:35:58.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.700+0000 7fac75ffb640 1 -- 172.21.15.70:0/2106559427 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac68034dc0 con 0x561042cd9ca0 2024-04-01T01:35:58.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.700+0000 7fac75ffb640 1 -- 172.21.15.70:0/2106559427 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac68039040 con 0x561042cd9ca0 2024-04-01T01:35:58.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.701+0000 7fac88d7e740 1 -- 172.21.15.70:0/2106559427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 msgr2=0x561042cde7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.701+0000 7fac7ffff640 1 --2- 172.21.15.70:0/3609336507 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac6c0a26e0 0x7fac6c0a4ba0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x561042cda500 tx=0x7fac78009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:58.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.701+0000 7fac88d7e740 1 --2- 172.21.15.70:0/2106559427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 0x561042cde7a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fac6802c180 tx=0x7fac68009490 comp rx=0 tx=0).stop 2024-04-01T01:35:58.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.701+0000 7fac88d7e740 1 -- 172.21.15.70:0/2106559427 shutdown_connections 2024-04-01T01:35:58.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.701+0000 7fac88d7e740 1 --2- 172.21.15.70:0/2106559427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cdee10 0x561042ce12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.701+0000 7fac88d7e740 1 --2- 172.21.15.70:0/2106559427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 0x561042cde7a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.701+0000 7fac88d7e740 1 --2- 172.21.15.70:0/2106559427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd92b0 0x561042cd96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.701+0000 7fac88d7e740 1 -- 172.21.15.70:0/2106559427 >> 172.21.15.70:0/2106559427 conn(0x561042cd4820 msgr2=0x561042cd4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:58.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.701+0000 7fac88d7e740 1 -- 172.21.15.70:0/2106559427 shutdown_connections 2024-04-01T01:35:58.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.701+0000 7fac88d7e740 1 -- 172.21.15.70:0/2106559427 wait complete. 2024-04-01T01:35:58.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.702+0000 7fac88d7e740 1 -- start start 2024-04-01T01:35:58.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.702+0000 7fac88d7e740 1 Processor -- start 2024-04-01T01:35:58.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.703+0000 7fac88d7e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd92b0 0x561042cf4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.703+0000 7fac88d7e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 0x561042cf4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.703+0000 7fac88d7e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cdee10 0x561042cfbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.703+0000 7fac88d7e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561042cf2d70 con 0x561042cdee10 2024-04-01T01:35:58.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.703+0000 7fac77fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 0x561042cf4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.703+0000 7fac88d7e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561042bdc4a0 con 0x561042cd9ca0 2024-04-01T01:35:58.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.703+0000 7fac88d7e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561042cf2ef0 con 0x561042cd92b0 2024-04-01T01:35:58.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.703+0000 7fac77fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 0x561042cf4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34948/0 (socket says 172.21.15.70:34948) 2024-04-01T01:35:58.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.703+0000 7fac77fff640 1 -- 172.21.15.70:0/2400023813 learned_addr learned my addr 172.21.15.70:0/2400023813 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:58.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.703+0000 7fac88d7e740 10 client.0 _ll_register_callbacks cb 0x561042bdfd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:58.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.704+0000 7fac7ffff640 1 --2- 172.21.15.70:0/2400023813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cdee10 0x561042cfbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.704+0000 7fac7f7fe640 1 --2- 172.21.15.70:0/2400023813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd92b0 0x561042cf4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.704+0000 7fac7ffff640 1 -- 172.21.15.70:0/2400023813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd92b0 msgr2=0x561042cf4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:58.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.704+0000 7fac7ffff640 1 --2- 172.21.15.70:0/2400023813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd92b0 0x561042cf4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.704+0000 7fac7ffff640 1 -- 172.21.15.70:0/2400023813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 msgr2=0x561042cf4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:58.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.704+0000 7fac7ffff640 1 --2- 172.21.15.70:0/2400023813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 0x561042cf4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.704+0000 7fac7ffff640 1 -- 172.21.15.70:0/2400023813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561042a1c7d0 con 0x561042cdee10 2024-04-01T01:35:58.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.704+0000 7fac7f7fe640 1 --2- 172.21.15.70:0/2400023813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd92b0 0x561042cf4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:35:58.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.705+0000 7fac77fff640 1 --2- 172.21.15.70:0/2400023813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 0x561042cf4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:35:58.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.705+0000 7fac7ffff640 1 --2- 172.21.15.70:0/2400023813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cdee10 0x561042cfbe10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fac78009fd0 tx=0x7fac78004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c236ed70bf648bb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:58.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.705+0000 7fac5effd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:35:58.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.705+0000 7fac5effd640 1 -- 172.21.15.70:0/2400023813 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac78004310 con 0x561042cdee10 2024-04-01T01:35:58.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.705+0000 7fac88d7e740 1 -- 172.21.15.70:0/2400023813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac68034dc0 con 0x561042cdee10 2024-04-01T01:35:58.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.705+0000 7fac5effd640 1 -- 172.21.15.70:0/2400023813 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fac780044b0 con 0x561042cdee10 2024-04-01T01:35:58.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.706+0000 7fac5effd640 1 -- 172.21.15.70:0/2400023813 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fac780046b0 con 0x561042cdee10 2024-04-01T01:35:58.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.708+0000 7fac88d7e740 10 client.16398 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:58.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.708+0000 7fac88d7e740 1 -- 172.21.15.70:0/2400023813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5610429523f0 con 0x561042cdee10 2024-04-01T01:35:58.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.709+0000 7fac5effd640 1 -- 172.21.15.70:0/2400023813 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac68034dc0 con 0x561042cdee10 2024-04-01T01:35:58.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.709+0000 7fac5effd640 1 -- 172.21.15.70:0/2400023813 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fac78026070 con 0x561042cdee10 2024-04-01T01:35:58.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.709+0000 7fac88d7e740 10 client.16398 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:58.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.709+0000 7fac88d7e740 10 client.16398 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:58.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.709+0000 7fac88d7e740 20 client.16398 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:58.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.709+0000 7fac88d7e740 1 --2- 172.21.15.70:0/2400023813 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561042d00c30 0x561042d030b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.709+0000 7fac88d7e740 4 client.16398 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:35:58.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.709+0000 7fac88d7e740 1 -- 172.21.15.70:0/2400023813 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561042c4b6c0 con 0x561042d00c30 2024-04-01T01:35:58.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.709+0000 7fac7f7fe640 1 --2- 172.21.15.70:0/2400023813 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561042d00c30 0x561042d030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.710+0000 7fac7f7fe640 1 --2- 172.21.15.70:0/2400023813 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561042d00c30 0x561042d030b0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ba9423d7bd4f914 server_cookie=a19cf32707cdf13d in_seq=0 out_seq=0 2024-04-01T01:35:58.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.710+0000 7fac5effd640 10 client.16398 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:58.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.711+0000 7fac5effd640 1 -- 172.21.15.70:0/2400023813 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561042c4b6c0 con 0x561042d00c30 2024-04-01T01:35:58.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.711+0000 7fac5effd640 10 client.16398 handle_command_reply: tid=0 2024-04-01T01:35:58.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.711+0000 7fac88d7e740 1 client.16398 shutdown 2024-04-01T01:35:58.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.711+0000 7fac88d7e740 1 -- 172.21.15.70:0/2400023813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cdee10 msgr2=0x561042cfbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.711+0000 7fac88d7e740 1 --2- 172.21.15.70:0/2400023813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cdee10 0x561042cfbe10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fac78009fd0 tx=0x7fac78004ab0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.711+0000 7fac88d7e740 1 -- 172.21.15.70:0/2400023813 shutdown_connections 2024-04-01T01:35:58.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.711+0000 7fac88d7e740 1 --2- 172.21.15.70:0/2400023813 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561042d00c30 0x561042d030b0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.711+0000 7fac88d7e740 1 --2- 172.21.15.70:0/2400023813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cdee10 0x561042cfbe10 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.711+0000 7fac88d7e740 1 --2- 172.21.15.70:0/2400023813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 0x561042cf4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.711+0000 7fac88d7e740 1 --2- 172.21.15.70:0/2400023813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd92b0 0x561042cf4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.711+0000 7fac88d7e740 1 -- 172.21.15.70:0/2400023813 >> 172.21.15.70:0/2400023813 conn(0x561042cd4820 msgr2=0x561042cd3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:58.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.712+0000 7fac5effd640 0 client.16398 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:58.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.712+0000 7fac88d7e740 1 -- 172.21.15.70:0/2400023813 shutdown_connections 2024-04-01T01:35:58.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.712+0000 7fac88d7e740 1 -- 172.21.15.70:0/2400023813 wait complete. 2024-04-01T01:35:58.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.712+0000 7fac88d7e740 20 client.16398 trim_cache size 0 max 16384 2024-04-01T01:35:58.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.743+0000 7fac88d7e740 1 Processor -- start 2024-04-01T01:35:58.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.743+0000 7fac88d7e740 1 -- start start 2024-04-01T01:35:58.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.743+0000 7fac88d7e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cd92b0 0x561042cf3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.743+0000 7fac88d7e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 0x561042cfbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.743+0000 7fac88d7e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cdee10 0x561042cf37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.743+0000 7fac88d7e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fac801a8370 con 0x561042cd92b0 2024-04-01T01:35:58.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.743+0000 7fac88d7e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fac801a9eb0 con 0x561042cd9ca0 2024-04-01T01:35:58.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.743+0000 7fac88d7e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fac801a8a40 con 0x561042cdee10 2024-04-01T01:35:58.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.744+0000 7fac7ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cdee10 0x561042cf37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.744+0000 7fac7ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cdee10 0x561042cf37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49830/0 (socket says 172.21.15.70:49830) 2024-04-01T01:35:58.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.744+0000 7fac7ffff640 1 -- 172.21.15.70:0/447492482 learned_addr learned my addr 172.21.15.70:0/447492482 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:58.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.744+0000 7fac77fff640 1 --2- 172.21.15.70:0/447492482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 0x561042cfbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.744+0000 7fac7f7fe640 1 --2- 172.21.15.70:0/447492482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cd92b0 0x561042cf3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.744+0000 7fac77fff640 1 -- 172.21.15.70:0/447492482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cdee10 msgr2=0x561042cf37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:58.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.744+0000 7fac77fff640 1 --2- 172.21.15.70:0/447492482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cdee10 0x561042cf37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.744+0000 7fac77fff640 1 -- 172.21.15.70:0/447492482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cd92b0 msgr2=0x561042cf3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:58.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.744+0000 7fac77fff640 1 --2- 172.21.15.70:0/447492482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cd92b0 0x561042cf3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.744+0000 7fac77fff640 1 -- 172.21.15.70:0/447492482 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610429120f0 con 0x561042cd9ca0 2024-04-01T01:35:58.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.745+0000 7fac77fff640 1 --2- 172.21.15.70:0/447492482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 0x561042cfbdd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fac7009b060 tx=0x7fac7009aaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=210220cca166c5c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:58.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.745+0000 7fac757fa640 1 -- 172.21.15.70:0/447492482 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7009f220 con 0x561042cd9ca0 2024-04-01T01:35:58.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.745+0000 7fac757fa640 1 -- 172.21.15.70:0/447492482 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac700a3ae0 con 0x561042cd9ca0 2024-04-01T01:35:58.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.745+0000 7fac757fa640 1 -- 172.21.15.70:0/447492482 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac700971a0 con 0x561042cd9ca0 2024-04-01T01:35:58.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.746+0000 7fac88d7e740 1 -- 172.21.15.70:0/447492482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 msgr2=0x561042cfbdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.746+0000 7fac88d7e740 1 --2- 172.21.15.70:0/447492482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 0x561042cfbdd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fac7009b060 tx=0x7fac7009aaf0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.746+0000 7fac88d7e740 1 -- 172.21.15.70:0/447492482 shutdown_connections 2024-04-01T01:35:58.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.746+0000 7fac88d7e740 1 --2- 172.21.15.70:0/447492482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cdee10 0x561042cf37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.746+0000 7fac88d7e740 1 --2- 172.21.15.70:0/447492482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd9ca0 0x561042cfbdd0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.746+0000 7fac88d7e740 1 --2- 172.21.15.70:0/447492482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cd92b0 0x561042cf3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.746+0000 7fac88d7e740 1 -- 172.21.15.70:0/447492482 >> 172.21.15.70:0/447492482 conn(0x561042cd4820 msgr2=0x561042cf4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:58.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.747+0000 7fac88d7e740 1 -- 172.21.15.70:0/447492482 shutdown_connections 2024-04-01T01:35:58.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.747+0000 7fac88d7e740 1 -- 172.21.15.70:0/447492482 wait complete. 2024-04-01T01:35:58.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.747+0000 7fac88d7e740 1 -- start start 2024-04-01T01:35:58.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.747+0000 7fac88d7e740 1 Processor -- start 2024-04-01T01:35:58.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.748+0000 7fac88d7e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd92b0 0x561042cf8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.748+0000 7fac88d7e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd9ca0 0x561042cf9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.748+0000 7fac88d7e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cdee10 0x561042d64fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.748+0000 7fac88d7e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fac801a97e0 con 0x561042cdee10 2024-04-01T01:35:58.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.748+0000 7fac88d7e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fac801a9110 con 0x561042cd92b0 2024-04-01T01:35:58.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.748+0000 7fac88d7e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fac800736c0 con 0x561042cd9ca0 2024-04-01T01:35:58.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.748+0000 7fac88d7e740 10 client.0 _ll_register_callbacks cb 0x7fac78011e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:35:58.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.748+0000 7fac7f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd92b0 0x561042cf8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.748+0000 7fac77fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd9ca0 0x561042cf9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.748+0000 7fac77fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd9ca0 0x561042cf9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49832/0 (socket says 172.21.15.70:49832) 2024-04-01T01:35:58.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.748+0000 7fac77fff640 1 -- 172.21.15.70:0/1567247670 learned_addr learned my addr 172.21.15.70:0/1567247670 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:35:58.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.749+0000 7fac7ffff640 1 --2- 172.21.15.70:0/1567247670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cdee10 0x561042d64fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.749+0000 7fac7f7fe640 1 -- 172.21.15.70:0/1567247670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd9ca0 msgr2=0x561042cf9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:58.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.749+0000 7fac7f7fe640 1 --2- 172.21.15.70:0/1567247670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd9ca0 0x561042cf9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.749+0000 7fac7f7fe640 1 -- 172.21.15.70:0/1567247670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cdee10 msgr2=0x561042d64fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:35:58.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.749+0000 7fac7f7fe640 1 --2- 172.21.15.70:0/1567247670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cdee10 0x561042d64fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.749+0000 7fac7f7fe640 1 -- 172.21.15.70:0/1567247670 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561042d76530 con 0x561042cd92b0 2024-04-01T01:35:58.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.749+0000 7fac7f7fe640 1 --2- 172.21.15.70:0/1567247670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd92b0 0x561042cf8ce0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x561042cd5790 tx=0x7fac68005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6c19de587c77134 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:35:58.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.749+0000 7fac5effd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:35:58.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.750+0000 7fac5effd640 1 -- 172.21.15.70:0/1567247670 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac68039040 con 0x561042cd92b0 2024-04-01T01:35:58.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.750+0000 7fac5effd640 1 -- 172.21.15.70:0/1567247670 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fac68052cb0 con 0x561042cd92b0 2024-04-01T01:35:58.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.752+0000 7fac88d7e740 1 -- 172.21.15.70:0/1567247670 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac700a3ae0 con 0x561042cd92b0 2024-04-01T01:35:58.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.753+0000 7fac5effd640 1 -- 172.21.15.70:0/1567247670 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fac68059050 con 0x561042cd92b0 2024-04-01T01:35:58.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.753+0000 7fac88d7e740 10 client.26246 fetch_fsmap learned FSMap version 29 2024-04-01T01:35:58.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.753+0000 7fac88d7e740 1 -- 172.21.15.70:0/1567247670 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561042d77910 con 0x561042cd92b0 2024-04-01T01:35:58.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.754+0000 7fac5effd640 1 -- 172.21.15.70:0/1567247670 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac700a3ae0 con 0x561042cd92b0 2024-04-01T01:35:58.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.754+0000 7fac5effd640 1 -- 172.21.15.70:0/1567247670 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fac68005070 con 0x561042cd92b0 2024-04-01T01:35:58.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.754+0000 7fac88d7e740 10 client.26246 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:35:58.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.754+0000 7fac88d7e740 10 client.26246 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:35:58.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.754+0000 7fac88d7e740 20 client.26246 populate_metadata read hostname 'smithi070' 2024-04-01T01:35:58.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.754+0000 7fac88d7e740 1 --2- 172.21.15.70:0/1567247670 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561042d77e10 0x561042d7a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:35:58.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.754+0000 7fac88d7e740 4 client.26246 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:35:58.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.754+0000 7fac88d7e740 1 -- 172.21.15.70:0/1567247670 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561042c45a10 con 0x561042d77e10 2024-04-01T01:35:58.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.755+0000 7fac77fff640 1 --2- 172.21.15.70:0/1567247670 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561042d77e10 0x561042d7a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:35:58.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.763+0000 7fac77fff640 1 --2- 172.21.15.70:0/1567247670 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561042d77e10 0x561042d7a200 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f275a23a480dc04 server_cookie=50dc5fcaa4b9a993 in_seq=0 out_seq=0 2024-04-01T01:35:58.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.763+0000 7fac5effd640 10 client.26246 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:58.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.764+0000 7fac5effd640 1 -- 172.21.15.70:0/1567247670 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561042c45a10 con 0x561042d77e10 2024-04-01T01:35:58.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.764+0000 7fac5effd640 10 client.26246 handle_command_reply: tid=0 2024-04-01T01:35:58.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.764+0000 7fac88d7e740 1 client.26246 shutdown 2024-04-01T01:35:58.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.764+0000 7fac88d7e740 1 -- 172.21.15.70:0/1567247670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd92b0 msgr2=0x561042cf8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.764+0000 7fac88d7e740 1 --2- 172.21.15.70:0/1567247670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd92b0 0x561042cf8ce0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x561042cd5790 tx=0x7fac68005f90 comp rx=0 tx=0).stop 2024-04-01T01:35:58.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.764+0000 7fac88d7e740 1 -- 172.21.15.70:0/1567247670 shutdown_connections 2024-04-01T01:35:58.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.764+0000 7fac88d7e740 1 --2- 172.21.15.70:0/1567247670 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561042d77e10 0x561042d7a200 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.764+0000 7fac88d7e740 1 --2- 172.21.15.70:0/1567247670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561042cdee10 0x561042d64fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.764+0000 7fac88d7e740 1 --2- 172.21.15.70:0/1567247670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561042cd9ca0 0x561042cf9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.764+0000 7fac88d7e740 1 --2- 172.21.15.70:0/1567247670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561042cd92b0 0x561042cf8ce0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.764+0000 7fac5effd640 0 client.26246 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:35:58.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.764+0000 7fac88d7e740 1 -- 172.21.15.70:0/1567247670 >> 172.21.15.70:0/1567247670 conn(0x561042cd4820 msgr2=0x561042cf4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:58.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.764+0000 7fac88d7e740 1 -- 172.21.15.70:0/1567247670 shutdown_connections 2024-04-01T01:35:58.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.764+0000 7fac88d7e740 1 -- 172.21.15.70:0/1567247670 wait complete. 2024-04-01T01:35:58.765 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:35:58.765 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:35:58.766 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:35:58.766 INFO:teuthology.orchestra.run.smithi070.stdout: "4341346d-3ac8-463c-9010-6169bfdac470": { 2024-04-01T01:35:58.766 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:35:58.766 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "4341346d-3ac8-463c-9010-6169bfdac470", 2024-04-01T01:35:58.766 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:35:58.766 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:35:58.766 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:35:58.766 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:35:58.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.765+0000 7fac88d7e740 20 client.26246 trim_cache size 0 max 16384 2024-04-01T01:35:58.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.766+0000 7fac8619a640 1 -- 172.21.15.70:0/3609336507 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac6c0a26e0 msgr2=0x7fac6c0a4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.766+0000 7fac8619a640 1 --2- 172.21.15.70:0/3609336507 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac6c0a26e0 0x7fac6c0a4ba0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x561042cda500 tx=0x7fac78009500 comp rx=0 tx=0).stop 2024-04-01T01:35:58.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.767+0000 7fac8619a640 1 -- 172.21.15.70:0/3609336507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac80102970 msgr2=0x7fac8019e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:35:58.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.767+0000 7fac8619a640 1 --2- 172.21.15.70:0/3609336507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac80102970 0x7fac8019e5a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fac7000e790 tx=0x7fac7000ec50 comp rx=0 tx=0).stop 2024-04-01T01:35:58.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.767+0000 7fac8619a640 1 -- 172.21.15.70:0/3609336507 shutdown_connections 2024-04-01T01:35:58.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.767+0000 7fac8619a640 1 --2- 172.21.15.70:0/3609336507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac80105320 0x7fac801a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.767+0000 7fac8619a640 1 --2- 172.21.15.70:0/3609336507 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fac6c0a26e0 0x7fac6c0a4ba0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.767+0000 7fac8619a640 1 --2- 172.21.15.70:0/3609336507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac80102970 0x7fac8019e5a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.767+0000 7fac8619a640 1 --2- 172.21.15.70:0/3609336507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac80100000 0x7fac8019e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:35:58.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.767+0000 7fac8619a640 1 -- 172.21.15.70:0/3609336507 >> 172.21.15.70:0/3609336507 conn(0x7fac800fbc00 msgr2=0x7fac801034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:35:58.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.767+0000 7fac8619a640 1 -- 172.21.15.70:0/3609336507 shutdown_connections 2024-04-01T01:35:58.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:35:58.767+0000 7fac8619a640 1 -- 172.21.15.70:0/3609336507 wait complete. 2024-04-01T01:35:58.786 INFO:tasks.cephfs.filesystem:scrub status for tag:4341346d-3ac8-463c-9010-6169bfdac470 - {'path': '/', 'tag': '4341346d-3ac8-463c-9010-6169bfdac470', 'options': 'recursive,force'} 2024-04-01T01:35:58.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:58 smithi189 ceph-mon[35753]: pgmap v433: 129 pgs: 129 active+clean; 970 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 12 MiB/s wr, 1.44k op/s 2024-04-01T01:35:59.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:58 smithi070 ceph-mon[32179]: pgmap v433: 129 pgs: 129 active+clean; 970 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 12 MiB/s wr, 1.44k op/s 2024-04-01T01:35:59.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:58 smithi081 ceph-mon[35142]: pgmap v433: 129 pgs: 129 active+clean; 970 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 12 MiB/s wr, 1.44k op/s 2024-04-01T01:36:00.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:59 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:36:00.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:59 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:36:00.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:59 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:36:00.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:35:59 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:36:00.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:59 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:36:00.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:59 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:36:00.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:59 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:36:00.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:35:59 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:36:00.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:59 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:36:00.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:59 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:36:00.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:59 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:36:00.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:35:59 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:36:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:00 smithi081 ceph-mon[35142]: pgmap v434: 129 pgs: 129 active+clean; 973 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 1.31k op/s 2024-04-01T01:36:01.149 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:00 smithi189 ceph-mon[35753]: pgmap v434: 129 pgs: 129 active+clean; 973 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 1.31k op/s 2024-04-01T01:36:01.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:00 smithi070 ceph-mon[32179]: pgmap v434: 129 pgs: 129 active+clean; 973 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 1.31k op/s 2024-04-01T01:36:01.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 75.0 s, 1302.5 tps, lat 77.238 ms stddev 27.185 2024-04-01T01:36:02.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:01 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:36:02.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:01 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:36:02.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:01 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:36:03.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:02 smithi081 ceph-mon[35142]: pgmap v435: 129 pgs: 129 active+clean; 973 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 13 MiB/s wr, 1.58k op/s 2024-04-01T01:36:03.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:02 smithi189 ceph-mon[35753]: pgmap v435: 129 pgs: 129 active+clean; 973 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 13 MiB/s wr, 1.58k op/s 2024-04-01T01:36:03.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:02 smithi070 ceph-mon[32179]: pgmap v435: 129 pgs: 129 active+clean; 973 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 13 MiB/s wr, 1.58k op/s 2024-04-01T01:36:05.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:05 smithi189 ceph-mon[35753]: pgmap v436: 129 pgs: 129 active+clean; 973 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 11 MiB/s wr, 1.38k op/s 2024-04-01T01:36:06.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:05 smithi070 ceph-mon[32179]: pgmap v436: 129 pgs: 129 active+clean; 973 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 11 MiB/s wr, 1.38k op/s 2024-04-01T01:36:06.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:05 smithi081 ceph-mon[35142]: pgmap v436: 129 pgs: 129 active+clean; 973 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 11 MiB/s wr, 1.38k op/s 2024-04-01T01:36:06.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 80.0 s, 1190.1 tps, lat 83.947 ms stddev 29.659 2024-04-01T01:36:07.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:07 smithi189 ceph-mon[35753]: pgmap v437: 129 pgs: 129 active+clean; 974 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 13 MiB/s wr, 1.57k op/s 2024-04-01T01:36:08.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:07 smithi070 ceph-mon[32179]: pgmap v437: 129 pgs: 129 active+clean; 974 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 13 MiB/s wr, 1.57k op/s 2024-04-01T01:36:08.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:07 smithi081 ceph-mon[35142]: pgmap v437: 129 pgs: 129 active+clean; 974 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 13 MiB/s wr, 1.57k op/s 2024-04-01T01:36:09.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:09 smithi189 ceph-mon[35753]: pgmap v438: 129 pgs: 129 active+clean; 974 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 12 MiB/s wr, 1.46k op/s 2024-04-01T01:36:10.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:09 smithi070 ceph-mon[32179]: pgmap v438: 129 pgs: 129 active+clean; 974 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 12 MiB/s wr, 1.46k op/s 2024-04-01T01:36:10.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:09 smithi081 ceph-mon[35142]: pgmap v438: 129 pgs: 129 active+clean; 974 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 12 MiB/s wr, 1.46k op/s 2024-04-01T01:36:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.667+0000 7f49374a4640 1 -- 172.21.15.70:0/3785147092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4930101c80 msgr2=0x7f49301020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.667+0000 7f49374a4640 1 --2- 172.21.15.70:0/3785147092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4930101c80 0x7f49301020a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f492c009900 tx=0x7f492c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:36:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.667+0000 7f49374a4640 1 -- 172.21.15.70:0/3785147092 shutdown_connections 2024-04-01T01:36:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.667+0000 7f49374a4640 1 --2- 172.21.15.70:0/3785147092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4930101290 0x7f4930108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.667+0000 7f49374a4640 1 --2- 172.21.15.70:0/3785147092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49301008d0 0x7f4930100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.667+0000 7f49374a4640 1 --2- 172.21.15.70:0/3785147092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4930101c80 0x7f49301020a0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.667+0000 7f49374a4640 1 -- 172.21.15.70:0/3785147092 >> 172.21.15.70:0/3785147092 conn(0x7f49300fb3a0 msgr2=0x7f49300fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.667+0000 7f49374a4640 1 -- 172.21.15.70:0/3785147092 shutdown_connections 2024-04-01T01:36:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.667+0000 7f49374a4640 1 -- 172.21.15.70:0/3785147092 wait complete. 2024-04-01T01:36:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.668+0000 7f49374a4640 1 Processor -- start 2024-04-01T01:36:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.668+0000 7f49374a4640 1 -- start start 2024-04-01T01:36:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.668+0000 7f49374a4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49301008d0 0x7f4930197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.668+0000 7f49374a4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4930101290 0x7f4930197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.668+0000 7f49374a4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4930101c80 0x7f493019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.668+0000 7f49374a4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f493010c230 con 0x7f4930101290 2024-04-01T01:36:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.668+0000 7f49374a4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f493010c0b0 con 0x7f4930101c80 2024-04-01T01:36:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.668+0000 7f49374a4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f493010c3b0 con 0x7f49301008d0 2024-04-01T01:36:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.668+0000 7f4934a3f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4930101290 0x7f4930197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.668+0000 7f4934a3f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4930101290 0x7f4930197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49448/0 (socket says 172.21.15.70:49448) 2024-04-01T01:36:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.668+0000 7f4934a3f640 1 -- 172.21.15.70:0/3735412753 learned_addr learned my addr 172.21.15.70:0/3735412753 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.668+0000 7f4935240640 1 --2- 172.21.15.70:0/3735412753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49301008d0 0x7f4930197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.668+0000 7f4935a41640 1 --2- 172.21.15.70:0/3735412753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4930101c80 0x7f493019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.669+0000 7f4935240640 1 -- 172.21.15.70:0/3735412753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4930101c80 msgr2=0x7f493019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.669+0000 7f4935240640 1 --2- 172.21.15.70:0/3735412753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4930101c80 0x7f493019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.669+0000 7f4935240640 1 -- 172.21.15.70:0/3735412753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4930101290 msgr2=0x7f4930197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.669+0000 7f4935240640 1 --2- 172.21.15.70:0/3735412753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4930101290 0x7f4930197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.669+0000 7f4935240640 1 -- 172.21.15.70:0/3735412753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4930102b80 con 0x7f49301008d0 2024-04-01T01:36:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.669+0000 7f4935240640 1 --2- 172.21.15.70:0/3735412753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49301008d0 0x7f4930197a70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f492c036040 tx=0x7f492c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.669+0000 7f49267fc640 1 -- 172.21.15.70:0/3735412753 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f492c043020 con 0x7f49301008d0 2024-04-01T01:36:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.669+0000 7f49374a4640 1 -- 172.21.15.70:0/3735412753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4930102e10 con 0x7f49301008d0 2024-04-01T01:36:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.669+0000 7f49267fc640 1 -- 172.21.15.70:0/3735412753 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f492c004aa0 con 0x7f49301008d0 2024-04-01T01:36:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.669+0000 7f49267fc640 1 -- 172.21.15.70:0/3735412753 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f492c003bf0 con 0x7f49301008d0 2024-04-01T01:36:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.669+0000 7f49374a4640 1 -- 172.21.15.70:0/3735412753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4930103400 con 0x7f49301008d0 2024-04-01T01:36:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.672+0000 7f493a163740 1 Processor -- start 2024-04-01T01:36:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.672+0000 7f49267fc640 1 -- 172.21.15.70:0/3735412753 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f492c046020 con 0x7f49301008d0 2024-04-01T01:36:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.672+0000 7f493a163740 1 -- start start 2024-04-01T01:36:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.672+0000 7f493a163740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e028d0 0x563300e02cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.672+0000 7f493a163740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e032c0 0x563300e07dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.672+0000 7f493a163740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e08430 0x563300e0a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.672+0000 7f493a163740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563300d668f0 con 0x563300e028d0 2024-04-01T01:36:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.672+0000 7f493a163740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563300d0f8d0 con 0x563300e032c0 2024-04-01T01:36:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.672+0000 7f493a163740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563300d62990 con 0x563300e08430 2024-04-01T01:36:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.672+0000 7f4934a3f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e08430 0x563300e0a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.672+0000 7f4935240640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e032c0 0x563300e07dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.672+0000 7f4934a3f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e08430 0x563300e0a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41418/0 (socket says 172.21.15.70:41418) 2024-04-01T01:36:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.672+0000 7f4934a3f640 1 -- 172.21.15.70:0/1031014929 learned_addr learned my addr 172.21.15.70:0/1031014929 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.672+0000 7f4935a41640 1 --2- 172.21.15.70:0/1031014929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e028d0 0x563300e02cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.673+0000 7f4934a3f640 1 -- 172.21.15.70:0/1031014929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e032c0 msgr2=0x563300e07dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.673+0000 7f4934a3f640 1 --2- 172.21.15.70:0/1031014929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e032c0 0x563300e07dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.673+0000 7f4934a3f640 1 -- 172.21.15.70:0/1031014929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e028d0 msgr2=0x563300e02cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.673+0000 7f4934a3f640 1 --2- 172.21.15.70:0/1031014929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e028d0 0x563300e02cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.673+0000 7f49267fc640 1 --2- 172.21.15.70:0/3735412753 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f491c0a2540 0x7f491c0a4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.673+0000 7f4934a3f640 1 -- 172.21.15.70:0/1031014929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563300a7b210 con 0x563300e08430 2024-04-01T01:36:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.673+0000 7f4935240640 1 --2- 172.21.15.70:0/1031014929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e032c0 0x563300e07dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:36:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.673+0000 7f4935240640 1 --2- 172.21.15.70:0/3735412753 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f491c0a2540 0x7f491c0a4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.673+0000 7f4934a3f640 1 --2- 172.21.15.70:0/1031014929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e08430 0x563300e0a8c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4920009ac0 tx=0x7f4920009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac50429be14bd309 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.673+0000 7f49267fc640 1 -- 172.21.15.70:0/3735412753 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f492c0f0e30 con 0x7f49301008d0 2024-04-01T01:36:10.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.673+0000 7f491b7fe640 1 -- 172.21.15.70:0/1031014929 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4920017070 con 0x563300e08430 2024-04-01T01:36:10.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.673+0000 7f491b7fe640 1 -- 172.21.15.70:0/1031014929 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f492000eba0 con 0x563300e08430 2024-04-01T01:36:10.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.673+0000 7f491b7fe640 1 -- 172.21.15.70:0/1031014929 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49200125f0 con 0x563300e08430 2024-04-01T01:36:10.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.674+0000 7f493a163740 1 -- 172.21.15.70:0/1031014929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e08430 msgr2=0x563300e0a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:10.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.674+0000 7f4935240640 1 --2- 172.21.15.70:0/3735412753 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f491c0a2540 0x7f491c0a4a00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f492c0ee8d0 tx=0x7f492c0f4910 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:10.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.674+0000 7f493a163740 1 --2- 172.21.15.70:0/1031014929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e08430 0x563300e0a8c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4920009ac0 tx=0x7f4920009f80 comp rx=0 tx=0).stop 2024-04-01T01:36:10.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.674+0000 7f493a163740 1 -- 172.21.15.70:0/1031014929 shutdown_connections 2024-04-01T01:36:10.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.674+0000 7f493a163740 1 --2- 172.21.15.70:0/1031014929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e08430 0x563300e0a8c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.674+0000 7f493a163740 1 --2- 172.21.15.70:0/1031014929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e032c0 0x563300e07dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.674+0000 7f493a163740 1 --2- 172.21.15.70:0/1031014929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e028d0 0x563300e02cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.674+0000 7f493a163740 1 -- 172.21.15.70:0/1031014929 >> 172.21.15.70:0/1031014929 conn(0x563300dfde40 msgr2=0x563300dfe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:10.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.674+0000 7f493a163740 1 -- 172.21.15.70:0/1031014929 shutdown_connections 2024-04-01T01:36:10.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.674+0000 7f493a163740 1 -- 172.21.15.70:0/1031014929 wait complete. 2024-04-01T01:36:10.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.675+0000 7f493a163740 1 -- start start 2024-04-01T01:36:10.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.675+0000 7f493a163740 1 Processor -- start 2024-04-01T01:36:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.675+0000 7f493a163740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e028d0 0x563300e0cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.675+0000 7f493a163740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e032c0 0x563300e0d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.675+0000 7f493a163740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e08430 0x563300e140c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.675+0000 7f493a163740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563300e0b430 con 0x563300e028d0 2024-04-01T01:36:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.675+0000 7f493a163740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563300d04f80 con 0x563300e032c0 2024-04-01T01:36:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.675+0000 7f493a163740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563300e0b5b0 con 0x563300e08430 2024-04-01T01:36:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.675+0000 7f493a163740 10 client.0 _ll_register_callbacks cb 0x563300d5d290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.675+0000 7f4935a41640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e028d0 0x563300e0cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.675+0000 7f4935a41640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e028d0 0x563300e0cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49458/0 (socket says 172.21.15.70:49458) 2024-04-01T01:36:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.675+0000 7f4935a41640 1 -- 172.21.15.70:0/2370228518 learned_addr learned my addr 172.21.15.70:0/2370228518 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.675+0000 7f4935a41640 1 --2- 172.21.15.70:0/2370228518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e08430 0x563300e140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.675+0000 7f4934a3f640 1 --2- 172.21.15.70:0/2370228518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e032c0 0x563300e0d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.676+0000 7f4935a41640 1 -- 172.21.15.70:0/2370228518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e032c0 msgr2=0x563300e0d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.676+0000 7f4935a41640 1 --2- 172.21.15.70:0/2370228518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e032c0 0x563300e0d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.676+0000 7f4935a41640 1 -- 172.21.15.70:0/2370228518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e028d0 msgr2=0x563300e0cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.676+0000 7f4935a41640 1 --2- 172.21.15.70:0/2370228518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e028d0 0x563300e0cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.676+0000 7f4935a41640 1 -- 172.21.15.70:0/2370228518 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563300b47620 con 0x563300e08430 2024-04-01T01:36:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.676+0000 7f4935a41640 1 --2- 172.21.15.70:0/2370228518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e08430 0x563300e140c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4928010fd0 tx=0x7f4928009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bd98e5720baac7c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.676+0000 7f4918ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:36:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.676+0000 7f4918ff9640 1 -- 172.21.15.70:0/2370228518 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4928019040 con 0x563300e08430 2024-04-01T01:36:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.676+0000 7f4918ff9640 1 -- 172.21.15.70:0/2370228518 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f4928004b20 con 0x563300e08430 2024-04-01T01:36:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.676+0000 7f493a163740 1 -- 172.21.15.70:0/2370228518 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f492000eba0 con 0x563300e08430 2024-04-01T01:36:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.676+0000 7f4918ff9640 1 -- 172.21.15.70:0/2370228518 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4928029050 con 0x563300e08430 2024-04-01T01:36:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.676+0000 7f493a163740 10 client.26155 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.676+0000 7f493a163740 1 -- 172.21.15.70:0/2370228518 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563300e17160 con 0x563300e08430 2024-04-01T01:36:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.677+0000 7f4918ff9640 1 -- 172.21.15.70:0/2370228518 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f492000eba0 con 0x563300e08430 2024-04-01T01:36:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.677+0000 7f4918ff9640 1 -- 172.21.15.70:0/2370228518 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4928020740 con 0x563300e08430 2024-04-01T01:36:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.677+0000 7f493a163740 10 client.26155 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.677+0000 7f493a163740 10 client.26155 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:36:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.677+0000 7f493a163740 20 client.26155 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.677+0000 7f493a163740 1 --2- 172.21.15.70:0/2370228518 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563300e18830 0x563300e1ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.677+0000 7f493a163740 4 client.26155 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:36:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.677+0000 7f493a163740 1 -- 172.21.15.70:0/2370228518 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563300d7a120 con 0x563300e18830 2024-04-01T01:36:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.677+0000 7f4934a3f640 1 --2- 172.21.15.70:0/2370228518 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563300e18830 0x563300e1ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.677+0000 7f4918ff9640 10 client.26155 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.678+0000 7f4934a3f640 1 --2- 172.21.15.70:0/2370228518 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563300e18830 0x563300e1ac80 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bdf4b1131f94ae server_cookie=6b9cd0845a09ca6e in_seq=0 out_seq=0 2024-04-01T01:36:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.679+0000 7f4918ff9640 1 -- 172.21.15.70:0/2370228518 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563300d7a120 con 0x563300e18830 2024-04-01T01:36:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.679+0000 7f4918ff9640 10 client.26155 handle_command_reply: tid=0 2024-04-01T01:36:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.679+0000 7f493a163740 1 client.26155 shutdown 2024-04-01T01:36:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.679+0000 7f493a163740 1 -- 172.21.15.70:0/2370228518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e08430 msgr2=0x563300e140c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.679+0000 7f493a163740 1 --2- 172.21.15.70:0/2370228518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e08430 0x563300e140c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4928010fd0 tx=0x7f4928009500 comp rx=0 tx=0).stop 2024-04-01T01:36:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.679+0000 7f493a163740 1 -- 172.21.15.70:0/2370228518 shutdown_connections 2024-04-01T01:36:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.679+0000 7f493a163740 1 --2- 172.21.15.70:0/2370228518 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563300e18830 0x563300e1ac80 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.679+0000 7f493a163740 1 --2- 172.21.15.70:0/2370228518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e08430 0x563300e140c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.679+0000 7f4918ff9640 0 client.26155 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.679+0000 7f493a163740 1 --2- 172.21.15.70:0/2370228518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e032c0 0x563300e0d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.679+0000 7f493a163740 1 --2- 172.21.15.70:0/2370228518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e028d0 0x563300e0cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.679+0000 7f493a163740 1 -- 172.21.15.70:0/2370228518 >> 172.21.15.70:0/2370228518 conn(0x563300dfde40 msgr2=0x563300dfcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.679+0000 7f493a163740 1 -- 172.21.15.70:0/2370228518 shutdown_connections 2024-04-01T01:36:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.679+0000 7f493a163740 1 -- 172.21.15.70:0/2370228518 wait complete. 2024-04-01T01:36:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.680+0000 7f493a163740 20 client.26155 trim_cache size 0 max 16384 2024-04-01T01:36:10.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.696+0000 7f493a163740 1 Processor -- start 2024-04-01T01:36:10.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.696+0000 7f493a163740 1 -- start start 2024-04-01T01:36:10.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.696+0000 7f493a163740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e028d0 0x563300e07b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.696+0000 7f493a163740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e032c0 0x563300e14080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.696+0000 7f493a163740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e08430 0x563300dfe220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.696+0000 7f493a163740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f49301ac520 con 0x563300e08430 2024-04-01T01:36:10.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.696+0000 7f493a163740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f49301ae060 con 0x563300e028d0 2024-04-01T01:36:10.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.696+0000 7f493a163740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f49301acbf0 con 0x563300e032c0 2024-04-01T01:36:10.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.696+0000 7f4935a41640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e08430 0x563300dfe220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.697+0000 7f4935a41640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e08430 0x563300dfe220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49462/0 (socket says 172.21.15.70:49462) 2024-04-01T01:36:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.697+0000 7f4935a41640 1 -- 172.21.15.70:0/3238481516 learned_addr learned my addr 172.21.15.70:0/3238481516 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.697+0000 7f4934a3f640 1 --2- 172.21.15.70:0/3238481516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e032c0 0x563300e14080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.697+0000 7f4935a41640 1 --2- 172.21.15.70:0/3238481516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e028d0 0x563300e07b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.697+0000 7f4935a41640 1 -- 172.21.15.70:0/3238481516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e032c0 msgr2=0x563300e14080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.697+0000 7f4935a41640 1 --2- 172.21.15.70:0/3238481516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e032c0 0x563300e14080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.697+0000 7f4935a41640 1 -- 172.21.15.70:0/3238481516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e028d0 msgr2=0x563300e07b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.697+0000 7f4935a41640 1 --2- 172.21.15.70:0/3238481516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e028d0 0x563300e07b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.697+0000 7f4935a41640 1 -- 172.21.15.70:0/3238481516 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563300dfebb0 con 0x563300e08430 2024-04-01T01:36:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.697+0000 7f4935a41640 1 --2- 172.21.15.70:0/3238481516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e08430 0x563300dfe220 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f492800f460 tx=0x7f49280104e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cbbe169f2a8c5c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.697+0000 7f491affd640 1 -- 172.21.15.70:0/3238481516 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4928010800 con 0x563300e08430 2024-04-01T01:36:10.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.697+0000 7f491affd640 1 -- 172.21.15.70:0/3238481516 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4928014070 con 0x563300e08430 2024-04-01T01:36:10.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.697+0000 7f491affd640 1 -- 172.21.15.70:0/3238481516 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49280205b0 con 0x563300e08430 2024-04-01T01:36:10.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.698+0000 7f493a163740 1 -- 172.21.15.70:0/3238481516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e08430 msgr2=0x563300dfe220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:10.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.698+0000 7f493a163740 1 --2- 172.21.15.70:0/3238481516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e08430 0x563300dfe220 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f492800f460 tx=0x7f49280104e0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.698+0000 7f493a163740 1 -- 172.21.15.70:0/3238481516 shutdown_connections 2024-04-01T01:36:10.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.698+0000 7f493a163740 1 --2- 172.21.15.70:0/3238481516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e08430 0x563300dfe220 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.698+0000 7f493a163740 1 --2- 172.21.15.70:0/3238481516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e032c0 0x563300e14080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.698+0000 7f493a163740 1 --2- 172.21.15.70:0/3238481516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e028d0 0x563300e07b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.698+0000 7f493a163740 1 -- 172.21.15.70:0/3238481516 >> 172.21.15.70:0/3238481516 conn(0x563300dfde40 msgr2=0x563300df8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:10.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.698+0000 7f493a163740 1 -- 172.21.15.70:0/3238481516 shutdown_connections 2024-04-01T01:36:10.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.698+0000 7f493a163740 1 -- 172.21.15.70:0/3238481516 wait complete. 2024-04-01T01:36:10.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.698+0000 7f493a163740 1 -- start start 2024-04-01T01:36:10.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.698+0000 7f493a163740 1 Processor -- start 2024-04-01T01:36:10.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f493a163740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e028d0 0x563300e11100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f493a163740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e032c0 0x563300e11640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f493a163740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e08430 0x563300e12600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f493a163740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f49301ad990 con 0x563300e08430 2024-04-01T01:36:10.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f493a163740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f49301ad2c0 con 0x563300e028d0 2024-04-01T01:36:10.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f493a163740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f49301aa500 con 0x563300e032c0 2024-04-01T01:36:10.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f493a163740 10 client.0 _ll_register_callbacks cb 0x7f4928010750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:10.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f4935a41640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e08430 0x563300e12600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f4935a41640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e08430 0x563300e12600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49464/0 (socket says 172.21.15.70:49464) 2024-04-01T01:36:10.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f4935a41640 1 -- 172.21.15.70:0/2738035384 learned_addr learned my addr 172.21.15.70:0/2738035384 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:10.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f4934a3f640 1 --2- 172.21.15.70:0/2738035384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e032c0 0x563300e11640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f4935a41640 1 --2- 172.21.15.70:0/2738035384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e028d0 0x563300e11100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f4935a41640 1 -- 172.21.15.70:0/2738035384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e032c0 msgr2=0x563300e11640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:10.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f4935a41640 1 --2- 172.21.15.70:0/2738035384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e032c0 0x563300e11640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f4935a41640 1 -- 172.21.15.70:0/2738035384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e028d0 msgr2=0x563300e11100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:10.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f4935a41640 1 --2- 172.21.15.70:0/2738035384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e028d0 0x563300e11100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f4935a41640 1 -- 172.21.15.70:0/2738035384 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563300e12bf0 con 0x563300e08430 2024-04-01T01:36:10.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f4935a41640 1 --2- 172.21.15.70:0/2738035384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e08430 0x563300e12600 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f492800eea0 tx=0x7f4928026040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a11120f652edc06 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:10.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f4918ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:36:10.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f4918ff9640 1 -- 172.21.15.70:0/2738035384 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f492800e780 con 0x563300e08430 2024-04-01T01:36:10.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f4918ff9640 1 -- 172.21.15.70:0/2738035384 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f492800e920 con 0x563300e08430 2024-04-01T01:36:10.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.699+0000 7f493a163740 1 -- 172.21.15.70:0/2738035384 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4928014070 con 0x563300e08430 2024-04-01T01:36:10.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.700+0000 7f4918ff9640 1 -- 172.21.15.70:0/2738035384 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f492800eb20 con 0x563300e08430 2024-04-01T01:36:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.700+0000 7f493a163740 10 client.16422 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.700+0000 7f493a163740 1 -- 172.21.15.70:0/2738035384 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563300e13ee0 con 0x563300e08430 2024-04-01T01:36:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.700+0000 7f4918ff9640 1 -- 172.21.15.70:0/2738035384 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4928014070 con 0x563300e08430 2024-04-01T01:36:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.700+0000 7f4918ff9640 1 -- 172.21.15.70:0/2738035384 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f492803b020 con 0x563300e08430 2024-04-01T01:36:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.700+0000 7f493a163740 10 client.16422 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.700+0000 7f493a163740 10 client.16422 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:36:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.700+0000 7f493a163740 20 client.16422 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.700+0000 7f493a163740 1 --2- 172.21.15.70:0/2738035384 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563300e18830 0x563300e80e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.700+0000 7f493a163740 4 client.16422 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:36:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.700+0000 7f493a163740 1 -- 172.21.15.70:0/2738035384 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563300d73e30 con 0x563300e18830 2024-04-01T01:36:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.700+0000 7f4934a3f640 1 --2- 172.21.15.70:0/2738035384 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563300e18830 0x563300e80e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:10.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.700+0000 7f4918ff9640 10 client.16422 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:10.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.700+0000 7f4934a3f640 1 --2- 172.21.15.70:0/2738035384 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563300e18830 0x563300e80e50 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c658c3b441d109f9 server_cookie=95d87e53993ba3e2 in_seq=0 out_seq=0 2024-04-01T01:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.731+0000 7f4918ff9640 1 -- 172.21.15.70:0/2738035384 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563300d73e30 con 0x563300e18830 2024-04-01T01:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.731+0000 7f4918ff9640 10 client.16422 handle_command_reply: tid=0 2024-04-01T01:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.731+0000 7f493a163740 1 client.16422 shutdown 2024-04-01T01:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.731+0000 7f493a163740 1 -- 172.21.15.70:0/2738035384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e08430 msgr2=0x563300e12600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.731+0000 7f493a163740 1 --2- 172.21.15.70:0/2738035384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e08430 0x563300e12600 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f492800eea0 tx=0x7f4928026040 comp rx=0 tx=0).stop 2024-04-01T01:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.731+0000 7f493a163740 1 -- 172.21.15.70:0/2738035384 shutdown_connections 2024-04-01T01:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.731+0000 7f493a163740 1 --2- 172.21.15.70:0/2738035384 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563300e18830 0x563300e80e50 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.731+0000 7f493a163740 1 --2- 172.21.15.70:0/2738035384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563300e08430 0x563300e12600 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.731+0000 7f493a163740 1 --2- 172.21.15.70:0/2738035384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563300e032c0 0x563300e11640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.731+0000 7f4918ff9640 0 client.16422 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.731+0000 7f493a163740 1 --2- 172.21.15.70:0/2738035384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563300e028d0 0x563300e11100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.731+0000 7f493a163740 1 -- 172.21.15.70:0/2738035384 >> 172.21.15.70:0/2738035384 conn(0x563300dfde40 msgr2=0x563300dfe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.731+0000 7f493a163740 1 -- 172.21.15.70:0/2738035384 shutdown_connections 2024-04-01T01:36:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.731+0000 7f493a163740 1 -- 172.21.15.70:0/2738035384 wait complete. 2024-04-01T01:36:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.732+0000 7f493a163740 20 client.16422 trim_cache size 0 max 16384 2024-04-01T01:36:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:36:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:36:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:36:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:36:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.733+0000 7f49374a4640 1 -- 172.21.15.70:0/3735412753 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f491c0a2540 msgr2=0x7f491c0a4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.733+0000 7f49374a4640 1 --2- 172.21.15.70:0/3735412753 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f491c0a2540 0x7f491c0a4a00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f492c0ee8d0 tx=0x7f492c0f4910 comp rx=0 tx=0).stop 2024-04-01T01:36:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.733+0000 7f49374a4640 1 -- 172.21.15.70:0/3735412753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49301008d0 msgr2=0x7f4930197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.733+0000 7f49374a4640 1 --2- 172.21.15.70:0/3735412753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49301008d0 0x7f4930197a70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f492c036040 tx=0x7f492c004270 comp rx=0 tx=0).stop 2024-04-01T01:36:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.733+0000 7f49374a4640 1 -- 172.21.15.70:0/3735412753 shutdown_connections 2024-04-01T01:36:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.733+0000 7f49374a4640 1 --2- 172.21.15.70:0/3735412753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4930101c80 0x7f493019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.733+0000 7f49374a4640 1 --2- 172.21.15.70:0/3735412753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4930101290 0x7f4930197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.733+0000 7f49374a4640 1 --2- 172.21.15.70:0/3735412753 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f491c0a2540 0x7f491c0a4a00 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.733+0000 7f49374a4640 1 --2- 172.21.15.70:0/3735412753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49301008d0 0x7f4930197a70 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.734+0000 7f49374a4640 1 -- 172.21.15.70:0/3735412753 >> 172.21.15.70:0/3735412753 conn(0x7f49300fb3a0 msgr2=0x7f49300fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.734+0000 7f49374a4640 1 -- 172.21.15.70:0/3735412753 shutdown_connections 2024-04-01T01:36:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:10.734+0000 7f49374a4640 1 -- 172.21.15.70:0/3735412753 wait complete. 2024-04-01T01:36:11.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 85.0 s, 1196.7 tps, lat 83.675 ms stddev 87.104 2024-04-01T01:36:11.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:11 smithi189 ceph-mon[35753]: pgmap v439: 129 pgs: 129 active+clean; 974 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 11 MiB/s wr, 1.39k op/s 2024-04-01T01:36:12.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:11 smithi070 ceph-mon[32179]: pgmap v439: 129 pgs: 129 active+clean; 974 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 11 MiB/s wr, 1.39k op/s 2024-04-01T01:36:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:11 smithi081 ceph-mon[35142]: pgmap v439: 129 pgs: 129 active+clean; 974 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 11 MiB/s wr, 1.39k op/s 2024-04-01T01:36:13.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:13 smithi189 ceph-mon[35753]: pgmap v440: 129 pgs: 129 active+clean; 986 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 14 MiB/s wr, 1.57k op/s 2024-04-01T01:36:14.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:13 smithi070 ceph-mon[32179]: pgmap v440: 129 pgs: 129 active+clean; 986 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 14 MiB/s wr, 1.57k op/s 2024-04-01T01:36:14.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:13 smithi081 ceph-mon[35142]: pgmap v440: 129 pgs: 129 active+clean; 986 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 14 MiB/s wr, 1.57k op/s 2024-04-01T01:36:15.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:15 smithi189 ceph-mon[35753]: pgmap v441: 129 pgs: 129 active+clean; 990 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 1.27k op/s 2024-04-01T01:36:16.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:15 smithi070 ceph-mon[32179]: pgmap v441: 129 pgs: 129 active+clean; 990 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 1.27k op/s 2024-04-01T01:36:16.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:15 smithi081 ceph-mon[35142]: pgmap v441: 129 pgs: 129 active+clean; 990 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 12 MiB/s wr, 1.27k op/s 2024-04-01T01:36:16.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 90.0 s, 1231.6 tps, lat 80.922 ms stddev 24.260 2024-04-01T01:36:17.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:17 smithi189 ceph-mon[35753]: pgmap v442: 129 pgs: 129 active+clean; 990 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 13 MiB/s wr, 1.40k op/s 2024-04-01T01:36:18.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:17 smithi070 ceph-mon[32179]: pgmap v442: 129 pgs: 129 active+clean; 990 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 13 MiB/s wr, 1.40k op/s 2024-04-01T01:36:18.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:17 smithi081 ceph-mon[35142]: pgmap v442: 129 pgs: 129 active+clean; 990 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 13 MiB/s wr, 1.40k op/s 2024-04-01T01:36:19.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:19 smithi189 ceph-mon[35753]: pgmap v443: 129 pgs: 129 active+clean; 990 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-04-01T01:36:20.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:19 smithi070 ceph-mon[32179]: pgmap v443: 129 pgs: 129 active+clean; 990 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-04-01T01:36:20.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:19 smithi081 ceph-mon[35142]: pgmap v443: 129 pgs: 129 active+clean; 990 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-04-01T01:36:21.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 95.0 s, 1227.0 tps, lat 81.692 ms stddev 25.946 2024-04-01T01:36:21.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:21 smithi189 ceph-mon[35753]: pgmap v444: 129 pgs: 129 active+clean; 990 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 12 MiB/s wr, 1.21k op/s 2024-04-01T01:36:22.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:21 smithi070 ceph-mon[32179]: pgmap v444: 129 pgs: 129 active+clean; 990 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 12 MiB/s wr, 1.21k op/s 2024-04-01T01:36:22.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:21 smithi081 ceph-mon[35142]: pgmap v444: 129 pgs: 129 active+clean; 990 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 12 MiB/s wr, 1.21k op/s 2024-04-01T01:36:23.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.879+0000 7f807b0fb640 1 -- 172.21.15.70:0/4065060464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80741022d0 msgr2=0x7f80741046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:23.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.879+0000 7f807b0fb640 1 --2- 172.21.15.70:0/4065060464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80741022d0 0x7f80741046c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8068009900 tx=0x7f806802bc30 comp rx=0 tx=0).stop 2024-04-01T01:36:23.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.879+0000 7f807b0fb640 1 -- 172.21.15.70:0/4065060464 shutdown_connections 2024-04-01T01:36:23.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.879+0000 7f807b0fb640 1 --2- 172.21.15.70:0/4065060464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8074107530 0x7f8074109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.879+0000 7f807b0fb640 1 --2- 172.21.15.70:0/4065060464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8074104c00 0x7f8074106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.879+0000 7f807b0fb640 1 --2- 172.21.15.70:0/4065060464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80741022d0 0x7f80741046c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.879+0000 7f807b0fb640 1 -- 172.21.15.70:0/4065060464 >> 172.21.15.70:0/4065060464 conn(0x7f80740fbbe0 msgr2=0x7f80740fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:23.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.879+0000 7f807b0fb640 1 -- 172.21.15.70:0/4065060464 shutdown_connections 2024-04-01T01:36:23.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.879+0000 7f807b0fb640 1 -- 172.21.15.70:0/4065060464 wait complete. 2024-04-01T01:36:23.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.880+0000 7f807b0fb640 1 Processor -- start 2024-04-01T01:36:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.880+0000 7f807b0fb640 1 -- start start 2024-04-01T01:36:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.880+0000 7f807b0fb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80741022d0 0x7f8074196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.881+0000 7f807b0fb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8074104c00 0x7f80741968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.881+0000 7f807b0fb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8074107530 0x7f8074192f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.881+0000 7f807b0fb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f807410d340 con 0x7f80741022d0 2024-04-01T01:36:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.881+0000 7f8078e97640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80741022d0 0x7f8074196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.881+0000 7f8078e97640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80741022d0 0x7f8074196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33342/0 (socket says 172.21.15.70:33342) 2024-04-01T01:36:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.881+0000 7f8078e97640 1 -- 172.21.15.70:0/1284557497 learned_addr learned my addr 172.21.15.70:0/1284557497 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.881+0000 7f807b0fb640 1 -- 172.21.15.70:0/1284557497 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f807410d1c0 con 0x7f8074107530 2024-04-01T01:36:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.881+0000 7f8073fff640 1 --2- 172.21.15.70:0/1284557497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8074104c00 0x7f80741968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.881+0000 7f807b0fb640 1 -- 172.21.15.70:0/1284557497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f807410d4c0 con 0x7f8074104c00 2024-04-01T01:36:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.881+0000 7f8079698640 1 --2- 172.21.15.70:0/1284557497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8074107530 0x7f8074192f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.881+0000 7f8073fff640 1 -- 172.21.15.70:0/1284557497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8074107530 msgr2=0x7f8074192f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.881+0000 7f8073fff640 1 --2- 172.21.15.70:0/1284557497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8074107530 0x7f8074192f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.881+0000 7f8073fff640 1 -- 172.21.15.70:0/1284557497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80741022d0 msgr2=0x7f8074196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.881+0000 7f8073fff640 1 --2- 172.21.15.70:0/1284557497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80741022d0 0x7f8074196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.881+0000 7f8073fff640 1 -- 172.21.15.70:0/1284557497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80741937c0 con 0x7f8074104c00 2024-04-01T01:36:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.881+0000 7f8078e97640 1 --2- 172.21.15.70:0/1284557497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80741022d0 0x7f8074196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:36:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.882+0000 7f8073fff640 1 --2- 172.21.15.70:0/1284557497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8074104c00 0x7f80741968d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8060005fd0 tx=0x7f806000dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.882+0000 7f8071ffb640 1 -- 172.21.15.70:0/1284557497 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8060009c00 con 0x7f8074104c00 2024-04-01T01:36:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.882+0000 7f807b0fb640 1 -- 172.21.15.70:0/1284557497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80741a5f40 con 0x7f8074104c00 2024-04-01T01:36:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.882+0000 7f8071ffb640 1 -- 172.21.15.70:0/1284557497 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8060009da0 con 0x7f8074104c00 2024-04-01T01:36:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.882+0000 7f8071ffb640 1 -- 172.21.15.70:0/1284557497 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8060003e40 con 0x7f8074104c00 2024-04-01T01:36:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.882+0000 7f807b0fb640 1 -- 172.21.15.70:0/1284557497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80741a6340 con 0x7f8074104c00 2024-04-01T01:36:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.886+0000 7f807ddb8740 1 Processor -- start 2024-04-01T01:36:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.886+0000 7f807ddb8740 1 -- start start 2024-04-01T01:36:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.887+0000 7f807ddb8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d3968d0 0x560f4d396cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.887+0000 7f807ddb8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3972c0 0x560f4d39bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.887+0000 7f807ddb8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d39c430 0x560f4d39e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.887+0000 7f807ddb8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560f4d2fa8f0 con 0x560f4d3972c0 2024-04-01T01:36:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.887+0000 7f807ddb8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560f4d2a38d0 con 0x560f4d3968d0 2024-04-01T01:36:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.887+0000 7f807ddb8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560f4d2f6990 con 0x560f4d39c430 2024-04-01T01:36:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.887+0000 7f8071ffb640 1 -- 172.21.15.70:0/1284557497 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f8060003fe0 con 0x7f8074104c00 2024-04-01T01:36:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.887+0000 7f8078e97640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3972c0 0x560f4d39bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.887+0000 7f8078e97640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3972c0 0x560f4d39bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33358/0 (socket says 172.21.15.70:33358) 2024-04-01T01:36:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.887+0000 7f8078e97640 1 -- 172.21.15.70:0/590738629 learned_addr learned my addr 172.21.15.70:0/590738629 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.887+0000 7f8079698640 1 --2- 172.21.15.70:0/590738629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d3968d0 0x560f4d396cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.887+0000 7f8073fff640 1 --2- 172.21.15.70:0/590738629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d39c430 0x560f4d39e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.887+0000 7f8078e97640 1 -- 172.21.15.70:0/590738629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d39c430 msgr2=0x560f4d39e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.887+0000 7f8078e97640 1 --2- 172.21.15.70:0/590738629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d39c430 0x560f4d39e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.887+0000 7f8078e97640 1 -- 172.21.15.70:0/590738629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d3968d0 msgr2=0x560f4d396cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.888+0000 7f8078e97640 1 --2- 172.21.15.70:0/590738629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d3968d0 0x560f4d396cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.888+0000 7f8078e97640 1 -- 172.21.15.70:0/590738629 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f4d00f210 con 0x560f4d3972c0 2024-04-01T01:36:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.888+0000 7f8079698640 1 --2- 172.21.15.70:0/590738629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d3968d0 0x560f4d396cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:36:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.888+0000 7f8071ffb640 1 --2- 172.21.15.70:0/1284557497 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f80640a25c0 0x7f80640a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.888+0000 7f8078e97640 1 --2- 172.21.15.70:0/590738629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3972c0 0x560f4d39bdc0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8068009690 tx=0x7f8068005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ec0b888a26715 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.888+0000 7f8071ffb640 1 -- 172.21.15.70:0/1284557497 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f80600ce860 con 0x7f8074104c00 2024-04-01T01:36:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.888+0000 7f8079698640 1 --2- 172.21.15.70:0/1284557497 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f80640a25c0 0x7f80640a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.888+0000 7f805effd640 1 -- 172.21.15.70:0/590738629 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068045020 con 0x560f4d3972c0 2024-04-01T01:36:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.889+0000 7f805effd640 1 -- 172.21.15.70:0/590738629 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8068005040 con 0x560f4d3972c0 2024-04-01T01:36:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.889+0000 7f805effd640 1 -- 172.21.15.70:0/590738629 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068004b20 con 0x560f4d3972c0 2024-04-01T01:36:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.890+0000 7f807ddb8740 1 -- 172.21.15.70:0/590738629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3972c0 msgr2=0x560f4d39bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.890+0000 7f807ddb8740 1 --2- 172.21.15.70:0/590738629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3972c0 0x560f4d39bdc0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8068009690 tx=0x7f8068005be0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.890+0000 7f8079698640 1 --2- 172.21.15.70:0/1284557497 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f80640a25c0 0x7f80640a4a80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x560f4d397b20 tx=0x7f806c009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.890+0000 7f807ddb8740 1 -- 172.21.15.70:0/590738629 shutdown_connections 2024-04-01T01:36:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.890+0000 7f807ddb8740 1 --2- 172.21.15.70:0/590738629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d39c430 0x560f4d39e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.890+0000 7f807ddb8740 1 --2- 172.21.15.70:0/590738629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3972c0 0x560f4d39bdc0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.890+0000 7f807ddb8740 1 --2- 172.21.15.70:0/590738629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d3968d0 0x560f4d396cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.890+0000 7f807ddb8740 1 -- 172.21.15.70:0/590738629 >> 172.21.15.70:0/590738629 conn(0x560f4d391e40 msgr2=0x560f4d392240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.890+0000 7f807ddb8740 1 -- 172.21.15.70:0/590738629 shutdown_connections 2024-04-01T01:36:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.890+0000 7f807ddb8740 1 -- 172.21.15.70:0/590738629 wait complete. 2024-04-01T01:36:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.891+0000 7f807ddb8740 1 -- start start 2024-04-01T01:36:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.891+0000 7f807ddb8740 1 Processor -- start 2024-04-01T01:36:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.892+0000 7f807ddb8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d3968d0 0x560f4d3a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.892+0000 7f807ddb8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3972c0 0x560f4d3a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.892+0000 7f807ddb8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d39c430 0x560f4d3a80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.892+0000 7f807ddb8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560f4d39f3d0 con 0x560f4d3972c0 2024-04-01T01:36:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.892+0000 7f807ddb8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560f4d298f80 con 0x560f4d3968d0 2024-04-01T01:36:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.892+0000 7f807ddb8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560f4d39f550 con 0x560f4d39c430 2024-04-01T01:36:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.892+0000 7f8078e97640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d3968d0 0x560f4d3a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.892+0000 7f8078e97640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d3968d0 0x560f4d3a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:36904/0 (socket says 172.21.15.70:36904) 2024-04-01T01:36:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.892+0000 7f8078e97640 1 -- 172.21.15.70:0/1873067701 learned_addr learned my addr 172.21.15.70:0/1873067701 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.892+0000 7f807ddb8740 10 client.0 _ll_register_callbacks cb 0x560f4d2f1290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.892+0000 7f8073fff640 1 --2- 172.21.15.70:0/1873067701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3972c0 0x560f4d3a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.892+0000 7f8079698640 1 --2- 172.21.15.70:0/1873067701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d39c430 0x560f4d3a80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.893+0000 7f8078e97640 1 -- 172.21.15.70:0/1873067701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d39c430 msgr2=0x560f4d3a80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.893+0000 7f8078e97640 1 --2- 172.21.15.70:0/1873067701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d39c430 0x560f4d3a80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.893+0000 7f8078e97640 1 -- 172.21.15.70:0/1873067701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3972c0 msgr2=0x560f4d3a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.893+0000 7f8078e97640 1 --2- 172.21.15.70:0/1873067701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3972c0 0x560f4d3a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.893+0000 7f8078e97640 1 -- 172.21.15.70:0/1873067701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f4d0db620 con 0x560f4d3968d0 2024-04-01T01:36:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.893+0000 7f8078e97640 1 --2- 172.21.15.70:0/1873067701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d3968d0 0x560f4d3a0b60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8068036040 tx=0x7f8068005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5da10ea07b358f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.893+0000 7f8053fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:36:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.893+0000 7f8053fff640 1 -- 172.21.15.70:0/1873067701 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068045040 con 0x560f4d3968d0 2024-04-01T01:36:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.894+0000 7f807ddb8740 1 -- 172.21.15.70:0/1873067701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8068005040 con 0x560f4d3968d0 2024-04-01T01:36:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.894+0000 7f8053fff640 1 -- 172.21.15.70:0/1873067701 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8068041680 con 0x560f4d3968d0 2024-04-01T01:36:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.894+0000 7f8053fff640 1 -- 172.21.15.70:0/1873067701 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f806805c050 con 0x560f4d3968d0 2024-04-01T01:36:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.894+0000 7f807ddb8740 10 client.26276 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.894+0000 7f807ddb8740 1 -- 172.21.15.70:0/1873067701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560f4d3ab0a0 con 0x560f4d3968d0 2024-04-01T01:36:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.894+0000 7f8053fff640 1 -- 172.21.15.70:0/1873067701 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8068005040 con 0x560f4d3968d0 2024-04-01T01:36:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.895+0000 7f8053fff640 1 -- 172.21.15.70:0/1873067701 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f806804a710 con 0x560f4d3968d0 2024-04-01T01:36:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.895+0000 7f807ddb8740 10 client.26276 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.895+0000 7f807ddb8740 10 client.26276 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:36:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.895+0000 7f807ddb8740 20 client.26276 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.895+0000 7f807ddb8740 1 --2- 172.21.15.70:0/1873067701 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560f4d3ac3f0 0x560f4d3ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.895+0000 7f807ddb8740 4 client.26276 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:36:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.895+0000 7f807ddb8740 1 -- 172.21.15.70:0/1873067701 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f4d30e120 con 0x560f4d3ac3f0 2024-04-01T01:36:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.895+0000 7f8073fff640 1 --2- 172.21.15.70:0/1873067701 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560f4d3ac3f0 0x560f4d3ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.896+0000 7f8073fff640 1 --2- 172.21.15.70:0/1873067701 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560f4d3ac3f0 0x560f4d3ae7e0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f55a41991bac0f08 server_cookie=34bc8fff3c3217c0 in_seq=0 out_seq=0 2024-04-01T01:36:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.896+0000 7f8053fff640 10 client.26276 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.897+0000 7f8053fff640 1 -- 172.21.15.70:0/1873067701 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560f4d30e120 con 0x560f4d3ac3f0 2024-04-01T01:36:23.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.897+0000 7f8053fff640 10 client.26276 handle_command_reply: tid=0 2024-04-01T01:36:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.897+0000 7f807ddb8740 1 client.26276 shutdown 2024-04-01T01:36:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.897+0000 7f807ddb8740 1 -- 172.21.15.70:0/1873067701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d3968d0 msgr2=0x560f4d3a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.897+0000 7f807ddb8740 1 --2- 172.21.15.70:0/1873067701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d3968d0 0x560f4d3a0b60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8068036040 tx=0x7f8068005fd0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.897+0000 7f807ddb8740 1 -- 172.21.15.70:0/1873067701 shutdown_connections 2024-04-01T01:36:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.897+0000 7f807ddb8740 1 --2- 172.21.15.70:0/1873067701 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560f4d3ac3f0 0x560f4d3ae7e0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.897+0000 7f807ddb8740 1 --2- 172.21.15.70:0/1873067701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d39c430 0x560f4d3a80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.897+0000 7f807ddb8740 1 --2- 172.21.15.70:0/1873067701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3972c0 0x560f4d3a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.897+0000 7f8053fff640 0 client.26276 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.897+0000 7f807ddb8740 1 --2- 172.21.15.70:0/1873067701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d3968d0 0x560f4d3a0b60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.897+0000 7f807ddb8740 1 -- 172.21.15.70:0/1873067701 >> 172.21.15.70:0/1873067701 conn(0x560f4d391e40 msgr2=0x560f4d390f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.898+0000 7f807ddb8740 1 -- 172.21.15.70:0/1873067701 shutdown_connections 2024-04-01T01:36:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.898+0000 7f807ddb8740 1 -- 172.21.15.70:0/1873067701 wait complete. 2024-04-01T01:36:23.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.898+0000 7f807ddb8740 20 client.26276 trim_cache size 0 max 16384 2024-04-01T01:36:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.924+0000 7f807ddb8740 1 Processor -- start 2024-04-01T01:36:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.925+0000 7f807ddb8740 1 -- start start 2024-04-01T01:36:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.925+0000 7f807ddb8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3968d0 0x560f4d3a0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.925+0000 7f807ddb8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d3972c0 0x560f4d3a8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.926+0000 7f807ddb8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d39c430 0x560f4d3a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.926+0000 7f8073fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d3972c0 0x560f4d3a8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.926+0000 7f8073fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d3972c0 0x560f4d3a8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57868/0 (socket says 172.21.15.70:57868) 2024-04-01T01:36:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.926+0000 7f8078e97640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3968d0 0x560f4d3a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.926+0000 7f8078e97640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3968d0 0x560f4d3a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33386/0 (socket says 172.21.15.70:33386) 2024-04-01T01:36:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.926+0000 7f807ddb8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f80741b0860 con 0x560f4d3968d0 2024-04-01T01:36:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.926+0000 7f807ddb8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f80741b23a0 con 0x560f4d39c430 2024-04-01T01:36:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.926+0000 7f807ddb8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f80741b0f30 con 0x560f4d3972c0 2024-04-01T01:36:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.926+0000 7f8073fff640 1 -- 172.21.15.70:0/3093233729 learned_addr learned my addr 172.21.15.70:0/3093233729 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.926+0000 7f8079698640 1 --2- 172.21.15.70:0/3093233729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d39c430 0x560f4d3a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.926+0000 7f8073fff640 1 -- 172.21.15.70:0/3093233729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d39c430 msgr2=0x560f4d3a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.926+0000 7f8073fff640 1 --2- 172.21.15.70:0/3093233729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d39c430 0x560f4d3a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.926+0000 7f8073fff640 1 -- 172.21.15.70:0/3093233729 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3968d0 msgr2=0x560f4d3a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.926+0000 7f8073fff640 1 --2- 172.21.15.70:0/3093233729 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3968d0 0x560f4d3a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.927+0000 7f8073fff640 1 -- 172.21.15.70:0/3093233729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f4d3aeb50 con 0x560f4d3972c0 2024-04-01T01:36:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.928+0000 7f8073fff640 1 --2- 172.21.15.70:0/3093233729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d3972c0 0x560f4d3a8070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8060003710 tx=0x7f806000bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fd41ec63f10db2e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.928+0000 7f805e7fc640 1 -- 172.21.15.70:0/3093233729 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8060020040 con 0x560f4d3972c0 2024-04-01T01:36:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.928+0000 7f805e7fc640 1 -- 172.21.15.70:0/3093233729 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f806009aef0 con 0x560f4d3972c0 2024-04-01T01:36:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.928+0000 7f805e7fc640 1 -- 172.21.15.70:0/3093233729 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8060097560 con 0x560f4d3972c0 2024-04-01T01:36:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.929+0000 7f807ddb8740 1 -- 172.21.15.70:0/3093233729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d3972c0 msgr2=0x560f4d3a8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.929+0000 7f807ddb8740 1 --2- 172.21.15.70:0/3093233729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d3972c0 0x560f4d3a8070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8060003710 tx=0x7f806000bef0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.929+0000 7f807ddb8740 1 -- 172.21.15.70:0/3093233729 shutdown_connections 2024-04-01T01:36:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.929+0000 7f807ddb8740 1 --2- 172.21.15.70:0/3093233729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d39c430 0x560f4d3a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.929+0000 7f807ddb8740 1 --2- 172.21.15.70:0/3093233729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d3972c0 0x560f4d3a8070 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.929+0000 7f807ddb8740 1 --2- 172.21.15.70:0/3093233729 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3968d0 0x560f4d3a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.929+0000 7f807ddb8740 1 -- 172.21.15.70:0/3093233729 >> 172.21.15.70:0/3093233729 conn(0x560f4d391e40 msgr2=0x560f4d390f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.930+0000 7f807ddb8740 1 -- 172.21.15.70:0/3093233729 shutdown_connections 2024-04-01T01:36:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.930+0000 7f807ddb8740 1 -- 172.21.15.70:0/3093233729 wait complete. 2024-04-01T01:36:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.930+0000 7f807ddb8740 1 -- start start 2024-04-01T01:36:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.930+0000 7f807ddb8740 1 Processor -- start 2024-04-01T01:36:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f807ddb8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3968d0 0x560f4d3a5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f807ddb8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d3972c0 0x560f4d3a5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f807ddb8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d39c430 0x560f4d3a6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f807ddb8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f80741b1cd0 con 0x560f4d3968d0 2024-04-01T01:36:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f807ddb8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f80741b1600 con 0x560f4d39c430 2024-04-01T01:36:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f807ddb8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f80741ae840 con 0x560f4d3972c0 2024-04-01T01:36:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f8078e97640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3968d0 0x560f4d3a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f807ddb8740 10 client.0 _ll_register_callbacks cb 0x7f806c0099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f8078e97640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3968d0 0x560f4d3a5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33398/0 (socket says 172.21.15.70:33398) 2024-04-01T01:36:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f8078e97640 1 -- 172.21.15.70:0/196514654 learned_addr learned my addr 172.21.15.70:0/196514654 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f8079698640 1 --2- 172.21.15.70:0/196514654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d39c430 0x560f4d3a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f8073fff640 1 --2- 172.21.15.70:0/196514654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d3972c0 0x560f4d3a5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f8078e97640 1 -- 172.21.15.70:0/196514654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d3972c0 msgr2=0x560f4d3a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:23.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f8078e97640 1 --2- 172.21.15.70:0/196514654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d3972c0 0x560f4d3a5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f8078e97640 1 -- 172.21.15.70:0/196514654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d39c430 msgr2=0x560f4d3a6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:23.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f8078e97640 1 --2- 172.21.15.70:0/196514654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d39c430 0x560f4d3a6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.931+0000 7f8078e97640 1 -- 172.21.15.70:0/196514654 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f4d3a6e60 con 0x560f4d3968d0 2024-04-01T01:36:23.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.932+0000 7f8078e97640 1 --2- 172.21.15.70:0/196514654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3968d0 0x560f4d3a5320 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x560f4d397be0 tx=0x7f8068005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c1bce2a1a92d0d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:23.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.932+0000 7f8053fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:36:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.932+0000 7f8053fff640 1 -- 172.21.15.70:0/196514654 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068045040 con 0x560f4d3968d0 2024-04-01T01:36:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.932+0000 7f8053fff640 1 -- 172.21.15.70:0/196514654 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8068004230 con 0x560f4d3968d0 2024-04-01T01:36:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.932+0000 7f807ddb8740 1 -- 172.21.15.70:0/196514654 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f806009aef0 con 0x560f4d3968d0 2024-04-01T01:36:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.933+0000 7f8053fff640 1 -- 172.21.15.70:0/196514654 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8068034440 con 0x560f4d3968d0 2024-04-01T01:36:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.933+0000 7f807ddb8740 10 client.16440 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.933+0000 7f807ddb8740 1 -- 172.21.15.70:0/196514654 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560f4d3a8170 con 0x560f4d3968d0 2024-04-01T01:36:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.933+0000 7f8053fff640 1 -- 172.21.15.70:0/196514654 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8068004430 con 0x560f4d3968d0 2024-04-01T01:36:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.933+0000 7f8053fff640 1 -- 172.21.15.70:0/196514654 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f80680349e0 con 0x560f4d3968d0 2024-04-01T01:36:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.934+0000 7f807ddb8740 10 client.16440 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.934+0000 7f807ddb8740 10 client.16440 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:36:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.934+0000 7f807ddb8740 20 client.16440 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.934+0000 7f807ddb8740 1 --2- 172.21.15.70:0/196514654 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560f4d412c80 0x560f4d4150a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.934+0000 7f807ddb8740 4 client.16440 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:36:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.934+0000 7f807ddb8740 1 -- 172.21.15.70:0/196514654 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f806009aef0 con 0x560f4d412c80 2024-04-01T01:36:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.934+0000 7f8073fff640 1 --2- 172.21.15.70:0/196514654 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560f4d412c80 0x560f4d4150a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.935+0000 7f8073fff640 1 --2- 172.21.15.70:0/196514654 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560f4d412c80 0x560f4d4150a0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cc3ef3a140f7c2a server_cookie=6ccaff3653486118 in_seq=0 out_seq=0 2024-04-01T01:36:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.935+0000 7f8053fff640 10 client.16440 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.971+0000 7f8053fff640 1 -- 172.21.15.70:0/196514654 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f806009aef0 con 0x560f4d412c80 2024-04-01T01:36:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.971+0000 7f8053fff640 10 client.16440 handle_command_reply: tid=0 2024-04-01T01:36:23.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.971+0000 7f807ddb8740 1 client.16440 shutdown 2024-04-01T01:36:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.972+0000 7f807ddb8740 1 -- 172.21.15.70:0/196514654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3968d0 msgr2=0x560f4d3a5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.972+0000 7f807ddb8740 1 --2- 172.21.15.70:0/196514654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3968d0 0x560f4d3a5320 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x560f4d397be0 tx=0x7f8068005750 comp rx=0 tx=0).stop 2024-04-01T01:36:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.972+0000 7f807ddb8740 1 -- 172.21.15.70:0/196514654 shutdown_connections 2024-04-01T01:36:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.972+0000 7f807ddb8740 1 --2- 172.21.15.70:0/196514654 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560f4d412c80 0x560f4d4150a0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.972+0000 7f8053fff640 0 client.16440 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.972+0000 7f807ddb8740 1 --2- 172.21.15.70:0/196514654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f4d39c430 0x560f4d3a6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.972+0000 7f807ddb8740 1 --2- 172.21.15.70:0/196514654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f4d3972c0 0x560f4d3a5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.972+0000 7f807ddb8740 1 --2- 172.21.15.70:0/196514654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4d3968d0 0x560f4d3a5320 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.972+0000 7f807ddb8740 1 -- 172.21.15.70:0/196514654 >> 172.21.15.70:0/196514654 conn(0x560f4d391e40 msgr2=0x560f4d3950a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.972+0000 7f807ddb8740 1 -- 172.21.15.70:0/196514654 shutdown_connections 2024-04-01T01:36:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.973+0000 7f807ddb8740 1 -- 172.21.15.70:0/196514654 wait complete. 2024-04-01T01:36:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.973+0000 7f807ddb8740 20 client.16440 trim_cache size 0 max 16384 2024-04-01T01:36:23.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:36:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:36:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:36:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:36:23.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.974+0000 7f807b0fb640 1 -- 172.21.15.70:0/1284557497 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f80640a25c0 msgr2=0x7f80640a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.974+0000 7f807b0fb640 1 --2- 172.21.15.70:0/1284557497 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f80640a25c0 0x7f80640a4a80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x560f4d397b20 tx=0x7f806c009500 comp rx=0 tx=0).stop 2024-04-01T01:36:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.975+0000 7f807b0fb640 1 -- 172.21.15.70:0/1284557497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8074104c00 msgr2=0x7f80741968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.975+0000 7f807b0fb640 1 --2- 172.21.15.70:0/1284557497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8074104c00 0x7f80741968d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8060005fd0 tx=0x7f806000dde0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.975+0000 7f807b0fb640 1 -- 172.21.15.70:0/1284557497 shutdown_connections 2024-04-01T01:36:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.975+0000 7f807b0fb640 1 --2- 172.21.15.70:0/1284557497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8074107530 0x7f8074192f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.975+0000 7f807b0fb640 1 --2- 172.21.15.70:0/1284557497 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f80640a25c0 0x7f80640a4a80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.975+0000 7f807b0fb640 1 --2- 172.21.15.70:0/1284557497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8074104c00 0x7f80741968d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.975+0000 7f807b0fb640 1 --2- 172.21.15.70:0/1284557497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f80741022d0 0x7f8074196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.975+0000 7f807b0fb640 1 -- 172.21.15.70:0/1284557497 >> 172.21.15.70:0/1284557497 conn(0x7f80740fbbe0 msgr2=0x7f8074101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.975+0000 7f807b0fb640 1 -- 172.21.15.70:0/1284557497 shutdown_connections 2024-04-01T01:36:23.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:23.975+0000 7f807b0fb640 1 -- 172.21.15.70:0/1284557497 wait complete. 2024-04-01T01:36:23.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:23 smithi189 ceph-mon[35753]: pgmap v445: 129 pgs: 129 active+clean; 990 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-04-01T01:36:24.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:23 smithi070 ceph-mon[32179]: pgmap v445: 129 pgs: 129 active+clean; 990 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-04-01T01:36:24.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:23 smithi081 ceph-mon[35142]: pgmap v445: 129 pgs: 129 active+clean; 990 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 13 MiB/s wr, 1.35k op/s 2024-04-01T01:36:25.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:24 smithi081 ceph-mon[35142]: pgmap v446: 129 pgs: 129 active+clean; 990 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-04-01T01:36:25.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:24 smithi189 ceph-mon[35753]: pgmap v446: 129 pgs: 129 active+clean; 990 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-04-01T01:36:25.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:24 smithi070 ceph-mon[32179]: pgmap v446: 129 pgs: 129 active+clean; 990 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 10 MiB/s wr, 1.20k op/s 2024-04-01T01:36:26.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 100.0 s, 1203.0 tps, lat 83.064 ms stddev 34.546 2024-04-01T01:36:27.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:27 smithi189 ceph-mon[35753]: pgmap v447: 129 pgs: 129 active+clean; 990 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-04-01T01:36:28.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:27 smithi070 ceph-mon[32179]: pgmap v447: 129 pgs: 129 active+clean; 990 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-04-01T01:36:28.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:27 smithi081 ceph-mon[35142]: pgmap v447: 129 pgs: 129 active+clean; 990 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-04-01T01:36:28.787 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:36:28.788 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:36:28.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.971+0000 7fee63884640 1 -- 172.21.15.70:0/2790966035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee5c100060 msgr2=0x7fee5c100480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:28.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.971+0000 7fee63884640 1 --2- 172.21.15.70:0/2790966035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee5c100060 0x7fee5c100480 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fee58009860 tx=0x7fee5802bb70 comp rx=0 tx=0).stop 2024-04-01T01:36:28.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.972+0000 7fee63884640 1 -- 172.21.15.70:0/2790966035 shutdown_connections 2024-04-01T01:36:28.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.972+0000 7fee63884640 1 --2- 172.21.15.70:0/2790966035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee5c101360 0x7fee5c1017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:28.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.972+0000 7fee63884640 1 --2- 172.21.15.70:0/2790966035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee5c1009c0 0x7fee5c100e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:28.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.972+0000 7fee63884640 1 --2- 172.21.15.70:0/2790966035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee5c100060 0x7fee5c100480 secure :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fee58009860 tx=0x7fee5802bb70 comp rx=0 tx=0).stop 2024-04-01T01:36:28.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.972+0000 7fee63884640 1 -- 172.21.15.70:0/2790966035 >> 172.21.15.70:0/2790966035 conn(0x7fee5c0fbbe0 msgr2=0x7fee5c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:28.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.972+0000 7fee63884640 1 -- 172.21.15.70:0/2790966035 shutdown_connections 2024-04-01T01:36:28.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.972+0000 7fee63884640 1 -- 172.21.15.70:0/2790966035 wait complete. 2024-04-01T01:36:28.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.973+0000 7fee63884640 1 Processor -- start 2024-04-01T01:36:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.974+0000 7fee63884640 1 -- start start 2024-04-01T01:36:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.974+0000 7fee63884640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee5c1009c0 0x7fee5c110ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.974+0000 7fee63884640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee5c101360 0x7fee5c111220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.974+0000 7fee63884640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee5c114270 0x7fee5c114720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.974+0000 7fee63884640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fee5c103f70 con 0x7fee5c114270 2024-04-01T01:36:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.974+0000 7fee63884640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fee5c103df0 con 0x7fee5c101360 2024-04-01T01:36:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.974+0000 7fee63884640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fee5c1040f0 con 0x7fee5c1009c0 2024-04-01T01:36:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.975+0000 7fee61620640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee5c1009c0 0x7fee5c110ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.975+0000 7fee61620640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee5c1009c0 0x7fee5c110ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37750/0 (socket says 172.21.15.70:37750) 2024-04-01T01:36:28.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.975+0000 7fee61620640 1 -- 172.21.15.70:0/1717795448 learned_addr learned my addr 172.21.15.70:0/1717795448 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:28.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.975+0000 7fee60e1f640 1 --2- 172.21.15.70:0/1717795448 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee5c101360 0x7fee5c111220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:28.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.975+0000 7fee61e21640 1 --2- 172.21.15.70:0/1717795448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee5c114270 0x7fee5c114720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:28.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.975+0000 7fee61e21640 1 -- 172.21.15.70:0/1717795448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee5c1009c0 msgr2=0x7fee5c110ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:28.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.975+0000 7fee61e21640 1 --2- 172.21.15.70:0/1717795448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee5c1009c0 0x7fee5c110ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:28.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.976+0000 7fee61e21640 1 -- 172.21.15.70:0/1717795448 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee5c101360 msgr2=0x7fee5c111220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:28.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.976+0000 7fee61e21640 1 --2- 172.21.15.70:0/1717795448 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee5c101360 0x7fee5c111220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:28.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.976+0000 7fee61e21640 1 -- 172.21.15.70:0/1717795448 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee5c114ce0 con 0x7fee5c114270 2024-04-01T01:36:28.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.976+0000 7fee61e21640 1 --2- 172.21.15.70:0/1717795448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee5c114270 0x7fee5c114720 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fee4c00e970 tx=0x7fee4c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:28.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.977+0000 7fee527fc640 1 -- 172.21.15.70:0/1717795448 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c00ccc0 con 0x7fee5c114270 2024-04-01T01:36:28.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.977+0000 7fee527fc640 1 -- 172.21.15.70:0/1717795448 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee4c005020 con 0x7fee5c114270 2024-04-01T01:36:28.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.977+0000 7fee527fc640 1 -- 172.21.15.70:0/1717795448 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c004a00 con 0x7fee5c114270 2024-04-01T01:36:28.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.977+0000 7fee63884640 1 -- 172.21.15.70:0/1717795448 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee5c114fd0 con 0x7fee5c114270 2024-04-01T01:36:28.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.977+0000 7fee63884640 1 -- 172.21.15.70:0/1717795448 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee5c1154d0 con 0x7fee5c114270 2024-04-01T01:36:28.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.981+0000 7fee66464740 1 Processor -- start 2024-04-01T01:36:28.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.981+0000 7fee527fc640 1 -- 172.21.15.70:0/1717795448 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fee4c0051d0 con 0x7fee5c114270 2024-04-01T01:36:28.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.981+0000 7fee66464740 1 -- start start 2024-04-01T01:36:28.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.981+0000 7fee66464740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 0x561c17de06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:28.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.981+0000 7fee66464740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de0ca0 0x561c17de57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:28.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.981+0000 7fee66464740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de5e10 0x561c17de82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:28.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.981+0000 7fee66464740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561c17d45180 con 0x561c17de0ca0 2024-04-01T01:36:28.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.981+0000 7fee66464740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561c17cedb70 con 0x561c17de02b0 2024-04-01T01:36:28.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.981+0000 7fee66464740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561c17d41220 con 0x561c17de5e10 2024-04-01T01:36:28.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.982+0000 7fee61620640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de0ca0 0x561c17de57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:28.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.982+0000 7fee61620640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de0ca0 0x561c17de57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59632/0 (socket says 172.21.15.70:59632) 2024-04-01T01:36:28.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.982+0000 7fee61620640 1 -- 172.21.15.70:0/4011337889 learned_addr learned my addr 172.21.15.70:0/4011337889 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:28.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.982+0000 7fee60e1f640 1 --2- 172.21.15.70:0/4011337889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 0x561c17de06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:28.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.982+0000 7fee527fc640 1 --2- 172.21.15.70:0/1717795448 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee480a2620 0x7fee480a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:28.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.982+0000 7fee60e1f640 1 --2- 172.21.15.70:0/4011337889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de5e10 0x561c17de82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:28.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.982+0000 7fee61620640 1 -- 172.21.15.70:0/4011337889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de5e10 msgr2=0x561c17de82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:28.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.982+0000 7fee61620640 1 --2- 172.21.15.70:0/4011337889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de5e10 0x561c17de82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:28.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.982+0000 7fee61620640 1 -- 172.21.15.70:0/4011337889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 msgr2=0x561c17de06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:28.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.982+0000 7fee61620640 1 --2- 172.21.15.70:0/4011337889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 0x561c17de06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:28.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.982+0000 7fee61620640 1 -- 172.21.15.70:0/4011337889 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c17d017c0 con 0x561c17de0ca0 2024-04-01T01:36:28.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.982+0000 7fee527fc640 1 -- 172.21.15.70:0/1717795448 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fee4c0ce960 con 0x7fee5c114270 2024-04-01T01:36:28.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.982+0000 7fee61e21640 1 --2- 172.21.15.70:0/1717795448 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee480a2620 0x7fee480a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:28.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.983+0000 7fee61620640 1 --2- 172.21.15.70:0/4011337889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de0ca0 0x561c17de57a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fee5802c100 tx=0x7fee58009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3938754671f43c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:28.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.983+0000 7fee61e21640 1 --2- 172.21.15.70:0/1717795448 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee480a2620 0x7fee480a4ae0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fee4c097a20 tx=0x7fee4c009620 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:28.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.983+0000 7fee477fe640 1 -- 172.21.15.70:0/4011337889 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee58043020 con 0x561c17de0ca0 2024-04-01T01:36:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.983+0000 7fee477fe640 1 -- 172.21.15.70:0/4011337889 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee58034640 con 0x561c17de0ca0 2024-04-01T01:36:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.983+0000 7fee477fe640 1 -- 172.21.15.70:0/4011337889 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee58039040 con 0x561c17de0ca0 2024-04-01T01:36:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.984+0000 7fee66464740 1 -- 172.21.15.70:0/4011337889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de0ca0 msgr2=0x561c17de57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.984+0000 7fee66464740 1 --2- 172.21.15.70:0/4011337889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de0ca0 0x561c17de57a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fee5802c100 tx=0x7fee58009520 comp rx=0 tx=0).stop 2024-04-01T01:36:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.985+0000 7fee66464740 1 -- 172.21.15.70:0/4011337889 shutdown_connections 2024-04-01T01:36:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.985+0000 7fee66464740 1 --2- 172.21.15.70:0/4011337889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de5e10 0x561c17de82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.985+0000 7fee66464740 1 --2- 172.21.15.70:0/4011337889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de0ca0 0x561c17de57a0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.985+0000 7fee66464740 1 --2- 172.21.15.70:0/4011337889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 0x561c17de06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.985+0000 7fee66464740 1 -- 172.21.15.70:0/4011337889 >> 172.21.15.70:0/4011337889 conn(0x561c17ddb820 msgr2=0x561c17ddbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.985+0000 7fee66464740 1 -- 172.21.15.70:0/4011337889 shutdown_connections 2024-04-01T01:36:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.985+0000 7fee66464740 1 -- 172.21.15.70:0/4011337889 wait complete. 2024-04-01T01:36:28.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.985+0000 7fee66464740 1 -- start start 2024-04-01T01:36:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.986+0000 7fee66464740 1 Processor -- start 2024-04-01T01:36:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.986+0000 7fee66464740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 0x561c17dea740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.986+0000 7fee66464740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de0ca0 0x561c17deac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.987+0000 7fee66464740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de5e10 0x561c17df1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.987+0000 7fee66464740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561c17de8d60 con 0x561c17de5e10 2024-04-01T01:36:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.987+0000 7fee66464740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561c17ce34a0 con 0x561c17de02b0 2024-04-01T01:36:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.987+0000 7fee66464740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561c17de8ee0 con 0x561c17de0ca0 2024-04-01T01:36:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.987+0000 7fee66464740 10 client.0 _ll_register_callbacks cb 0x561c17ce6d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.987+0000 7fee60e1f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 0x561c17dea740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.987+0000 7fee61620640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de0ca0 0x561c17deac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.987+0000 7fee60e1f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 0x561c17dea740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35346/0 (socket says 172.21.15.70:35346) 2024-04-01T01:36:28.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.987+0000 7fee61620640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de0ca0 0x561c17deac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37762/0 (socket says 172.21.15.70:37762) 2024-04-01T01:36:28.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.987+0000 7fee60e1f640 1 -- 172.21.15.70:0/1683999700 learned_addr learned my addr 172.21.15.70:0/1683999700 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:28.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.987+0000 7fee60e1f640 1 --2- 172.21.15.70:0/1683999700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de5e10 0x561c17df1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:28.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.987+0000 7fee60e1f640 1 -- 172.21.15.70:0/1683999700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de0ca0 msgr2=0x561c17deac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:28.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.987+0000 7fee60e1f640 1 --2- 172.21.15.70:0/1683999700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de0ca0 0x561c17deac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:28.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.987+0000 7fee60e1f640 1 -- 172.21.15.70:0/1683999700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de5e10 msgr2=0x561c17df1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:28.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.987+0000 7fee60e1f640 1 --2- 172.21.15.70:0/1683999700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de5e10 0x561c17df1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:28.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.987+0000 7fee60e1f640 1 -- 172.21.15.70:0/1683999700 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c17b237d0 con 0x561c17de02b0 2024-04-01T01:36:28.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.988+0000 7fee60e1f640 1 --2- 172.21.15.70:0/1683999700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 0x561c17dea740 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fee54010010 tx=0x7fee54004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7303a1331a8b6ff6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:28.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.988+0000 7fee44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:36:28.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.988+0000 7fee44ff9640 1 -- 172.21.15.70:0/1683999700 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee54024070 con 0x561c17de02b0 2024-04-01T01:36:28.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.988+0000 7fee44ff9640 1 -- 172.21.15.70:0/1683999700 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fee54003970 con 0x561c17de02b0 2024-04-01T01:36:28.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.988+0000 7fee66464740 1 -- 172.21.15.70:0/1683999700 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee58034640 con 0x561c17de02b0 2024-04-01T01:36:28.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.989+0000 7fee44ff9640 1 -- 172.21.15.70:0/1683999700 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fee54003b70 con 0x561c17de02b0 2024-04-01T01:36:28.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.989+0000 7fee66464740 10 client.26294 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:28.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.989+0000 7fee66464740 1 -- 172.21.15.70:0/1683999700 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561c17a593f0 con 0x561c17de02b0 2024-04-01T01:36:28.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.990+0000 7fee44ff9640 1 -- 172.21.15.70:0/1683999700 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee58034640 con 0x561c17de02b0 2024-04-01T01:36:28.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.990+0000 7fee44ff9640 1 -- 172.21.15.70:0/1683999700 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fee5400b040 con 0x561c17de02b0 2024-04-01T01:36:28.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.990+0000 7fee66464740 10 client.26294 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:28.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.990+0000 7fee66464740 10 client.26294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:36:28.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.990+0000 7fee66464740 20 client.26294 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:28.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.990+0000 7fee66464740 1 --2- 172.21.15.70:0/1683999700 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561c17df65b0 0x561c17df8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:28.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.990+0000 7fee66464740 4 client.26294 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:36:28.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.990+0000 7fee66464740 1 -- 172.21.15.70:0/1683999700 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c17d526c0 con 0x561c17df65b0 2024-04-01T01:36:28.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.991+0000 7fee61620640 1 --2- 172.21.15.70:0/1683999700 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561c17df65b0 0x561c17df8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:28.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.991+0000 7fee44ff9640 10 client.26294 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:28.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.991+0000 7fee61620640 1 --2- 172.21.15.70:0/1683999700 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561c17df65b0 0x561c17df8a10 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb2c8dc3c76fc860 server_cookie=bf7fffd8daeed159 in_seq=0 out_seq=0 2024-04-01T01:36:28.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.992+0000 7fee44ff9640 1 -- 172.21.15.70:0/1683999700 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561c17d526c0 con 0x561c17df65b0 2024-04-01T01:36:28.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.992+0000 7fee44ff9640 10 client.26294 handle_command_reply: tid=0 2024-04-01T01:36:28.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.992+0000 7fee66464740 1 client.26294 shutdown 2024-04-01T01:36:28.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.992+0000 7fee66464740 1 -- 172.21.15.70:0/1683999700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 msgr2=0x561c17dea740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:28.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.992+0000 7fee66464740 1 --2- 172.21.15.70:0/1683999700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 0x561c17dea740 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fee54010010 tx=0x7fee54004ab0 comp rx=0 tx=0).stop 2024-04-01T01:36:28.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.993+0000 7fee66464740 1 -- 172.21.15.70:0/1683999700 shutdown_connections 2024-04-01T01:36:28.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.993+0000 7fee66464740 1 --2- 172.21.15.70:0/1683999700 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561c17df65b0 0x561c17df8a10 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:28.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.993+0000 7fee66464740 1 --2- 172.21.15.70:0/1683999700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de5e10 0x561c17df1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:28.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.993+0000 7fee66464740 1 --2- 172.21.15.70:0/1683999700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de0ca0 0x561c17deac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:28.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.993+0000 7fee66464740 1 --2- 172.21.15.70:0/1683999700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 0x561c17dea740 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:28.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.993+0000 7fee66464740 1 -- 172.21.15.70:0/1683999700 >> 172.21.15.70:0/1683999700 conn(0x561c17ddb820 msgr2=0x561c17ddac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:28.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.993+0000 7fee44ff9640 0 client.26294 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:28.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.993+0000 7fee66464740 1 -- 172.21.15.70:0/1683999700 shutdown_connections 2024-04-01T01:36:28.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.993+0000 7fee66464740 1 -- 172.21.15.70:0/1683999700 wait complete. 2024-04-01T01:36:28.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:28.993+0000 7fee66464740 20 client.26294 trim_cache size 0 max 16384 2024-04-01T01:36:29.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.027+0000 7fee66464740 1 Processor -- start 2024-04-01T01:36:29.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.027+0000 7fee66464740 1 -- start start 2024-04-01T01:36:29.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.027+0000 7fee66464740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 0x561c17dea350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.027+0000 7fee66464740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de0ca0 0x561c17df1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.027+0000 7fee66464740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de5e10 0x561c17dea890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.027+0000 7fee66464740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fee5c1c1ff0 con 0x561c17de5e10 2024-04-01T01:36:29.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.027+0000 7fee66464740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fee5c1c3b30 con 0x561c17de02b0 2024-04-01T01:36:29.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.027+0000 7fee66464740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fee5c1c26c0 con 0x561c17de0ca0 2024-04-01T01:36:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.028+0000 7fee60e1f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 0x561c17dea350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.028+0000 7fee60e1f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 0x561c17dea350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35350/0 (socket says 172.21.15.70:35350) 2024-04-01T01:36:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.028+0000 7fee60e1f640 1 -- 172.21.15.70:0/1300520485 learned_addr learned my addr 172.21.15.70:0/1300520485 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.028+0000 7fee60e1f640 1 --2- 172.21.15.70:0/1300520485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de5e10 0x561c17dea890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.028+0000 7fee61620640 1 --2- 172.21.15.70:0/1300520485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de0ca0 0x561c17df1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.028+0000 7fee60e1f640 1 -- 172.21.15.70:0/1300520485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de0ca0 msgr2=0x561c17df1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.028+0000 7fee60e1f640 1 --2- 172.21.15.70:0/1300520485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de0ca0 0x561c17df1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.028+0000 7fee60e1f640 1 -- 172.21.15.70:0/1300520485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de5e10 msgr2=0x561c17dea890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.028+0000 7fee60e1f640 1 --2- 172.21.15.70:0/1300520485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de5e10 0x561c17dea890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.028+0000 7fee60e1f640 1 -- 172.21.15.70:0/1300520485 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c17a190f0 con 0x561c17de02b0 2024-04-01T01:36:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.028+0000 7fee60e1f640 1 --2- 172.21.15.70:0/1300520485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 0x561c17dea350 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fee5400e450 tx=0x7fee54000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe540f088801ca9e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.029+0000 7fee46ffd640 1 -- 172.21.15.70:0/1300520485 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee54024040 con 0x561c17de02b0 2024-04-01T01:36:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.029+0000 7fee46ffd640 1 -- 172.21.15.70:0/1300520485 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee54004b20 con 0x561c17de02b0 2024-04-01T01:36:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.029+0000 7fee46ffd640 1 -- 172.21.15.70:0/1300520485 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5401db70 con 0x561c17de02b0 2024-04-01T01:36:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.030+0000 7fee66464740 1 -- 172.21.15.70:0/1300520485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 msgr2=0x561c17dea350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.030+0000 7fee66464740 1 --2- 172.21.15.70:0/1300520485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 0x561c17dea350 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fee5400e450 tx=0x7fee54000f40 comp rx=0 tx=0).stop 2024-04-01T01:36:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.030+0000 7fee66464740 1 -- 172.21.15.70:0/1300520485 shutdown_connections 2024-04-01T01:36:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.030+0000 7fee66464740 1 --2- 172.21.15.70:0/1300520485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de5e10 0x561c17dea890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.030+0000 7fee66464740 1 --2- 172.21.15.70:0/1300520485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de0ca0 0x561c17df1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.030+0000 7fee66464740 1 --2- 172.21.15.70:0/1300520485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de02b0 0x561c17dea350 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.030+0000 7fee66464740 1 -- 172.21.15.70:0/1300520485 >> 172.21.15.70:0/1300520485 conn(0x561c17ddb820 msgr2=0x561c17de2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.030+0000 7fee66464740 1 -- 172.21.15.70:0/1300520485 shutdown_connections 2024-04-01T01:36:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.030+0000 7fee66464740 1 -- 172.21.15.70:0/1300520485 wait complete. 2024-04-01T01:36:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.030+0000 7fee66464740 1 -- start start 2024-04-01T01:36:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.031+0000 7fee66464740 1 Processor -- start 2024-04-01T01:36:29.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.031+0000 7fee66464740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de02b0 0x561c17deedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.031+0000 7fee66464740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de0ca0 0x561c17def330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.031+0000 7fee66464740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de5e10 0x561c17df0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.031+0000 7fee66464740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fee5c1c3460 con 0x561c17de02b0 2024-04-01T01:36:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.031+0000 7fee66464740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fee5c1c2d90 con 0x561c17de0ca0 2024-04-01T01:36:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.031+0000 7fee66464740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fee5c072c90 con 0x561c17de5e10 2024-04-01T01:36:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.031+0000 7fee66464740 10 client.0 _ll_register_callbacks cb 0x7fee5400f6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee60e1f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de02b0 0x561c17deedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee61620640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de0ca0 0x561c17def330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee60e1f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de02b0 0x561c17deedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59652/0 (socket says 172.21.15.70:59652) 2024-04-01T01:36:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee60e1f640 1 -- 172.21.15.70:0/3341557457 learned_addr learned my addr 172.21.15.70:0/3341557457 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee61620640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de0ca0 0x561c17def330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35358/0 (socket says 172.21.15.70:35358) 2024-04-01T01:36:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee60e1f640 1 --2- 172.21.15.70:0/3341557457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de5e10 0x561c17df0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee61620640 1 -- 172.21.15.70:0/3341557457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de5e10 msgr2=0x561c17df0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee61620640 1 --2- 172.21.15.70:0/3341557457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de5e10 0x561c17df0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee61620640 1 -- 172.21.15.70:0/3341557457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de02b0 msgr2=0x561c17deedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee61620640 1 --2- 172.21.15.70:0/3341557457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de02b0 0x561c17deedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee61620640 1 -- 172.21.15.70:0/3341557457 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c17df0930 con 0x561c17de0ca0 2024-04-01T01:36:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee60e1f640 1 --2- 172.21.15.70:0/3341557457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de5e10 0x561c17df0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:36:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee61620640 1 --2- 172.21.15.70:0/3341557457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de0ca0 0x561c17def330 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x561c17ddc790 tx=0x7fee5802bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b63cbf4443721f6c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:36:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee44ff9640 1 -- 172.21.15.70:0/3341557457 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee58043040 con 0x561c17de0ca0 2024-04-01T01:36:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee66464740 1 -- 172.21.15.70:0/3341557457 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee54004b20 con 0x561c17de0ca0 2024-04-01T01:36:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.032+0000 7fee44ff9640 1 -- 172.21.15.70:0/3341557457 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fee58034a80 con 0x561c17de0ca0 2024-04-01T01:36:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.033+0000 7fee44ff9640 1 -- 172.21.15.70:0/3341557457 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fee58059050 con 0x561c17de0ca0 2024-04-01T01:36:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.033+0000 7fee66464740 10 client.26300 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.033+0000 7fee66464740 1 -- 172.21.15.70:0/3341557457 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561c17df1d00 con 0x561c17de0ca0 2024-04-01T01:36:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.033+0000 7fee44ff9640 1 -- 172.21.15.70:0/3341557457 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee54004b20 con 0x561c17de0ca0 2024-04-01T01:36:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.033+0000 7fee44ff9640 1 -- 172.21.15.70:0/3341557457 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fee58050cc0 con 0x561c17de0ca0 2024-04-01T01:36:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.033+0000 7fee66464740 10 client.26300 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.033+0000 7fee66464740 10 client.26300 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:36:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.033+0000 7fee66464740 20 client.26300 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.034+0000 7fee66464740 1 --2- 172.21.15.70:0/3341557457 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561c17e5c910 0x561c17e5ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.034+0000 7fee66464740 4 client.26300 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:36:29.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.034+0000 7fee66464740 1 -- 172.21.15.70:0/3341557457 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561c17d4ca10 con 0x561c17e5c910 2024-04-01T01:36:29.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.034+0000 7fee60e1f640 1 --2- 172.21.15.70:0/3341557457 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561c17e5c910 0x561c17e5ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.035+0000 7fee60e1f640 1 --2- 172.21.15.70:0/3341557457 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561c17e5c910 0x561c17e5ed00 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e851e4390f7fd07 server_cookie=6b1629d14bcfbf80 in_seq=0 out_seq=0 2024-04-01T01:36:29.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.035+0000 7fee44ff9640 10 client.26300 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:29.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.035+0000 7fee44ff9640 1 -- 172.21.15.70:0/3341557457 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561c17d4ca10 con 0x561c17e5c910 2024-04-01T01:36:29.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.035+0000 7fee44ff9640 10 client.26300 handle_command_reply: tid=0 2024-04-01T01:36:29.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.035+0000 7fee66464740 1 client.26300 shutdown 2024-04-01T01:36:29.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.035+0000 7fee66464740 1 -- 172.21.15.70:0/3341557457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de0ca0 msgr2=0x561c17def330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.035+0000 7fee66464740 1 --2- 172.21.15.70:0/3341557457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de0ca0 0x561c17def330 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x561c17ddc790 tx=0x7fee5802bb70 comp rx=0 tx=0).stop 2024-04-01T01:36:29.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.035+0000 7fee66464740 1 -- 172.21.15.70:0/3341557457 shutdown_connections 2024-04-01T01:36:29.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.035+0000 7fee66464740 1 --2- 172.21.15.70:0/3341557457 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561c17e5c910 0x561c17e5ed00 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.035+0000 7fee66464740 1 --2- 172.21.15.70:0/3341557457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561c17de5e10 0x561c17df0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.035+0000 7fee66464740 1 --2- 172.21.15.70:0/3341557457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c17de0ca0 0x561c17def330 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.035+0000 7fee66464740 1 --2- 172.21.15.70:0/3341557457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561c17de02b0 0x561c17deedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.035+0000 7fee66464740 1 -- 172.21.15.70:0/3341557457 >> 172.21.15.70:0/3341557457 conn(0x561c17ddb820 msgr2=0x561c17dea2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:29.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.035+0000 7fee44ff9640 0 client.26300 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:29.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.035+0000 7fee66464740 1 -- 172.21.15.70:0/3341557457 shutdown_connections 2024-04-01T01:36:29.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.035+0000 7fee66464740 1 -- 172.21.15.70:0/3341557457 wait complete. 2024-04-01T01:36:29.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.036+0000 7fee66464740 20 client.26300 trim_cache size 0 max 16384 2024-04-01T01:36:29.039 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:36:29.039 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:36:29.039 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:36:29.039 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:36:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.036+0000 7fee63884640 1 -- 172.21.15.70:0/1717795448 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee480a2620 msgr2=0x7fee480a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.036+0000 7fee63884640 1 --2- 172.21.15.70:0/1717795448 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee480a2620 0x7fee480a4ae0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fee4c097a20 tx=0x7fee4c009620 comp rx=0 tx=0).stop 2024-04-01T01:36:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.036+0000 7fee63884640 1 -- 172.21.15.70:0/1717795448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee5c114270 msgr2=0x7fee5c114720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.036+0000 7fee63884640 1 --2- 172.21.15.70:0/1717795448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee5c114270 0x7fee5c114720 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fee4c00e970 tx=0x7fee4c00ee30 comp rx=0 tx=0).stop 2024-04-01T01:36:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.037+0000 7fee63884640 1 -- 172.21.15.70:0/1717795448 shutdown_connections 2024-04-01T01:36:29.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.037+0000 7fee63884640 1 --2- 172.21.15.70:0/1717795448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee5c114270 0x7fee5c114720 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.037+0000 7fee63884640 1 --2- 172.21.15.70:0/1717795448 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee5c101360 0x7fee5c111220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.037+0000 7fee63884640 1 --2- 172.21.15.70:0/1717795448 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee480a2620 0x7fee480a4ae0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.037+0000 7fee63884640 1 --2- 172.21.15.70:0/1717795448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee5c1009c0 0x7fee5c110ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.037+0000 7fee63884640 1 -- 172.21.15.70:0/1717795448 >> 172.21.15.70:0/1717795448 conn(0x7fee5c0fbbe0 msgr2=0x7fee5c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.037+0000 7fee63884640 1 -- 172.21.15.70:0/1717795448 shutdown_connections 2024-04-01T01:36:29.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.037+0000 7fee63884640 1 -- 172.21.15.70:0/1717795448 wait complete. 2024-04-01T01:36:29.054 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:36:29.054 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:36:29.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.230+0000 7ffa063ec640 1 -- 172.21.15.70:0/3093040597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa00104930 msgr2=0x7ffa00106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.230+0000 7ffa063ec640 1 --2- 172.21.15.70:0/3093040597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa00104930 0x7ffa00106d20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc009860 tx=0x7ff9fc02bb90 comp rx=0 tx=0).stop 2024-04-01T01:36:29.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.230+0000 7ffa063ec640 1 -- 172.21.15.70:0/3093040597 shutdown_connections 2024-04-01T01:36:29.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.230+0000 7ffa063ec640 1 --2- 172.21.15.70:0/3093040597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa00107260 0x7ffa001096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.230+0000 7ffa063ec640 1 --2- 172.21.15.70:0/3093040597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa00104930 0x7ffa00106d20 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.230+0000 7ffa063ec640 1 --2- 172.21.15.70:0/3093040597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa00102000 0x7ffa001043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.230+0000 7ffa063ec640 1 -- 172.21.15.70:0/3093040597 >> 172.21.15.70:0/3093040597 conn(0x7ffa000fb9d0 msgr2=0x7ffa000fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.231+0000 7ffa063ec640 1 -- 172.21.15.70:0/3093040597 shutdown_connections 2024-04-01T01:36:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.231+0000 7ffa063ec640 1 -- 172.21.15.70:0/3093040597 wait complete. 2024-04-01T01:36:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.231+0000 7ffa063ec640 1 Processor -- start 2024-04-01T01:36:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa063ec640 1 -- start start 2024-04-01T01:36:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa063ec640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa00102000 0x7ffa001999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa063ec640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa00104930 0x7ffa00199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa063ec640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa00107260 0x7ffa001a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa063ec640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0010d120 con 0x7ffa00104930 2024-04-01T01:36:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa063ec640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0010cfa0 con 0x7ffa00107260 2024-04-01T01:36:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa063ec640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0010d2a0 con 0x7ffa00102000 2024-04-01T01:36:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa05beb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa00107260 0x7ffa001a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa05beb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa00107260 0x7ffa001a0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35374/0 (socket says 172.21.15.70:35374) 2024-04-01T01:36:29.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa05beb640 1 -- 172.21.15.70:0/1126190022 learned_addr learned my addr 172.21.15.70:0/1126190022 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:29.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa053ea640 1 --2- 172.21.15.70:0/1126190022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa00102000 0x7ffa001999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa04be9640 1 --2- 172.21.15.70:0/1126190022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa00104930 0x7ffa00199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa05beb640 1 -- 172.21.15.70:0/1126190022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa00102000 msgr2=0x7ffa001999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa05beb640 1 --2- 172.21.15.70:0/1126190022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa00102000 0x7ffa001999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa05beb640 1 -- 172.21.15.70:0/1126190022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa00104930 msgr2=0x7ffa00199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa05beb640 1 --2- 172.21.15.70:0/1126190022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa00104930 0x7ffa00199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa05beb640 1 -- 172.21.15.70:0/1126190022 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa001a15f0 con 0x7ffa00107260 2024-04-01T01:36:29.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.232+0000 7ffa04be9640 1 --2- 172.21.15.70:0/1126190022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa00104930 0x7ffa00199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:36:29.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.233+0000 7ffa05beb640 1 --2- 172.21.15.70:0/1126190022 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa00107260 0x7ffa001a0f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800cce0 tx=0x7ff9f800a620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:29.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.233+0000 7ff9f67fc640 1 -- 172.21.15.70:0/1126190022 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f800f450 con 0x7ffa00107260 2024-04-01T01:36:29.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.233+0000 7ffa063ec640 1 -- 172.21.15.70:0/1126190022 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa001a18e0 con 0x7ffa00107260 2024-04-01T01:36:29.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.233+0000 7ff9f67fc640 1 -- 172.21.15.70:0/1126190022 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9f800faa0 con 0x7ffa00107260 2024-04-01T01:36:29.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.233+0000 7ff9f67fc640 1 -- 172.21.15.70:0/1126190022 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8018b70 con 0x7ffa00107260 2024-04-01T01:36:29.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.233+0000 7ffa063ec640 1 -- 172.21.15.70:0/1126190022 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa001a1e80 con 0x7ffa00107260 2024-04-01T01:36:29.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.235+0000 7ffa063ec640 1 -- 172.21.15.70:0/1126190022 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa00108d90 con 0x7ffa00107260 2024-04-01T01:36:29.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.236+0000 7ff9f67fc640 1 -- 172.21.15.70:0/1126190022 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7ff9f8018d10 con 0x7ffa00107260 2024-04-01T01:36:29.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.237+0000 7ff9f67fc640 1 --2- 172.21.15.70:0/1126190022 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff9ec0a25a0 0x7ff9ec0a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.237+0000 7ff9f67fc640 1 -- 172.21.15.70:0/1126190022 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff9f80cef10 con 0x7ffa00107260 2024-04-01T01:36:29.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.237+0000 7ffa053ea640 1 --2- 172.21.15.70:0/1126190022 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff9ec0a25a0 0x7ff9ec0a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.238+0000 7ffa053ea640 1 --2- 172.21.15.70:0/1126190022 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff9ec0a25a0 0x7ff9ec0a4a60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0006fd0 tx=0x7ff9f00090e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.244+0000 7ff9f67fc640 1 -- 172.21.15.70:0/1126190022 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff9f8014030 con 0x7ffa00107260 2024-04-01T01:36:29.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.671+0000 7ffa063ec640 1 -- 172.21.15.70:0/1126190022 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa0002d170 con 0x7ffa00107260 2024-04-01T01:36:29.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.672+0000 7ff9f67fc640 1 -- 172.21.15.70:0/1126190022 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7ff9f809bca0 con 0x7ffa00107260 2024-04-01T01:36:29.673 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:36:29.673 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:33:36.894838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24418,24404,24379,24451,24464]},"id":1}]} 2024-04-01T01:36:29.673 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-04-01T01:36:29.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.676+0000 7ffa063ec640 1 -- 172.21.15.70:0/1126190022 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff9ec0a25a0 msgr2=0x7ff9ec0a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.677+0000 7ffa063ec640 1 --2- 172.21.15.70:0/1126190022 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff9ec0a25a0 0x7ff9ec0a4a60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0006fd0 tx=0x7ff9f00090e0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.677+0000 7ffa063ec640 1 -- 172.21.15.70:0/1126190022 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa00107260 msgr2=0x7ffa001a0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.677+0000 7ffa063ec640 1 --2- 172.21.15.70:0/1126190022 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa00107260 0x7ffa001a0f80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800cce0 tx=0x7ff9f800a620 comp rx=0 tx=0).stop 2024-04-01T01:36:29.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.677+0000 7ffa063ec640 1 -- 172.21.15.70:0/1126190022 shutdown_connections 2024-04-01T01:36:29.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.677+0000 7ffa063ec640 1 --2- 172.21.15.70:0/1126190022 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa00107260 0x7ffa001a0f80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.677+0000 7ffa063ec640 1 --2- 172.21.15.70:0/1126190022 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa00104930 0x7ffa00199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.677+0000 7ffa063ec640 1 --2- 172.21.15.70:0/1126190022 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff9ec0a25a0 0x7ff9ec0a4a60 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.677+0000 7ffa063ec640 1 --2- 172.21.15.70:0/1126190022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa00102000 0x7ffa001999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.677+0000 7ffa063ec640 1 -- 172.21.15.70:0/1126190022 >> 172.21.15.70:0/1126190022 conn(0x7ffa000fb9d0 msgr2=0x7ffa00101680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:29.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.677+0000 7ffa063ec640 1 -- 172.21.15.70:0/1126190022 shutdown_connections 2024-04-01T01:36:29.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.679+0000 7ffa063ec640 1 -- 172.21.15.70:0/1126190022 wait complete. 2024-04-01T01:36:29.697 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:36:29.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.875+0000 7fb2aea15640 1 -- 172.21.15.70:0/2715883056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2a81010e0 msgr2=0x7fb2a81059a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.875+0000 7fb2aea15640 1 --2- 172.21.15.70:0/2715883056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2a81010e0 0x7fb2a81059a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb298009900 tx=0x7fb29802bc40 comp rx=0 tx=0).stop 2024-04-01T01:36:29.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.875+0000 7fb2aea15640 1 -- 172.21.15.70:0/2715883056 shutdown_connections 2024-04-01T01:36:29.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.875+0000 7fb2aea15640 1 --2- 172.21.15.70:0/2715883056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2a81010e0 0x7fb2a81059a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.875+0000 7fb2aea15640 1 --2- 172.21.15.70:0/2715883056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2a8100740 0x7fb2a8100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.875+0000 7fb2aea15640 1 --2- 172.21.15.70:0/2715883056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a80ff610 0x7fb2a80ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.875+0000 7fb2aea15640 1 -- 172.21.15.70:0/2715883056 >> 172.21.15.70:0/2715883056 conn(0x7fb2a80faf10 msgr2=0x7fb2a80fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:29.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.876+0000 7fb2aea15640 1 -- 172.21.15.70:0/2715883056 shutdown_connections 2024-04-01T01:36:29.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.876+0000 7fb2aea15640 1 -- 172.21.15.70:0/2715883056 wait complete. 2024-04-01T01:36:29.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.877+0000 7fb2aea15640 1 Processor -- start 2024-04-01T01:36:29.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.877+0000 7fb2aea15640 1 -- start start 2024-04-01T01:36:29.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.877+0000 7fb2aea15640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2a80ff610 0x7fb2a819df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.877+0000 7fb2aea15640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a8100740 0x7fb2a819e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.877+0000 7fb2aea15640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2a81010e0 0x7fb2a81a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.877+0000 7fb2aea15640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81138a0 con 0x7fb2a80ff610 2024-04-01T01:36:29.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.877+0000 7fb2aea15640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a8113720 con 0x7fb2a8100740 2024-04-01T01:36:29.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.877+0000 7fb2aea15640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a8113a20 con 0x7fb2a81010e0 2024-04-01T01:36:29.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.877+0000 7fb2a77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a8100740 0x7fb2a819e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.878+0000 7fb2acfb2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2a81010e0 0x7fb2a81a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.878+0000 7fb2a7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2a80ff610 0x7fb2a819df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.878+0000 7fb2acfb2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2a81010e0 0x7fb2a81a54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37816/0 (socket says 172.21.15.70:37816) 2024-04-01T01:36:29.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.878+0000 7fb2a7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2a80ff610 0x7fb2a819df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59690/0 (socket says 172.21.15.70:59690) 2024-04-01T01:36:29.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.878+0000 7fb2acfb2640 1 -- 172.21.15.70:0/2231835811 learned_addr learned my addr 172.21.15.70:0/2231835811 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:29.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.878+0000 7fb2a77fe640 1 -- 172.21.15.70:0/2231835811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2a81010e0 msgr2=0x7fb2a81a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.878+0000 7fb2a77fe640 1 --2- 172.21.15.70:0/2231835811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2a81010e0 0x7fb2a81a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.878+0000 7fb2a77fe640 1 -- 172.21.15.70:0/2231835811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2a80ff610 msgr2=0x7fb2a819df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.878+0000 7fb2a77fe640 1 --2- 172.21.15.70:0/2231835811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2a80ff610 0x7fb2a819df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.878+0000 7fb2a77fe640 1 -- 172.21.15.70:0/2231835811 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a81a5b10 con 0x7fb2a8100740 2024-04-01T01:36:29.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.878+0000 7fb2acfb2640 1 --2- 172.21.15.70:0/2231835811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2a81010e0 0x7fb2a81a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:36:29.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.878+0000 7fb2a77fe640 1 --2- 172.21.15.70:0/2231835811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a8100740 0x7fb2a819e480 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb29c00e9f0 tx=0x7fb29c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:29.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.879+0000 7fb2a57fa640 1 -- 172.21.15.70:0/2231835811 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c00ccc0 con 0x7fb2a8100740 2024-04-01T01:36:29.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.879+0000 7fb2a57fa640 1 -- 172.21.15.70:0/2231835811 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb29c0108e0 con 0x7fb2a8100740 2024-04-01T01:36:29.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.879+0000 7fb2a57fa640 1 -- 172.21.15.70:0/2231835811 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c0188c0 con 0x7fb2a8100740 2024-04-01T01:36:29.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.879+0000 7fb2aea15640 1 -- 172.21.15.70:0/2231835811 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2a81a5e00 con 0x7fb2a8100740 2024-04-01T01:36:29.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.879+0000 7fb2aea15640 1 -- 172.21.15.70:0/2231835811 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2a81a6310 con 0x7fb2a8100740 2024-04-01T01:36:29.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.882+0000 7fb2a57fa640 1 -- 172.21.15.70:0/2231835811 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb29c010a90 con 0x7fb2a8100740 2024-04-01T01:36:29.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.882+0000 7fb2b15fb740 1 Processor -- start 2024-04-01T01:36:29.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.883+0000 7fb2a57fa640 1 --2- 172.21.15.70:0/2231835811 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2900a2520 0x7fb2900a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.883+0000 7fb2b15fb740 1 -- start start 2024-04-01T01:36:29.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.883+0000 7fb2a57fa640 1 -- 172.21.15.70:0/2231835811 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb29c097bc0 con 0x7fb2a8100740 2024-04-01T01:36:29.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2b15fb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2b15fb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61e80 0x564912f66980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2b15fb740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f66ff0 0x564912f69480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2b15fb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564912ec61b0 con 0x564912f61e80 2024-04-01T01:36:29.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2b15fb740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x564912e6ec00 con 0x564912f66ff0 2024-04-01T01:36:29.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2b15fb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564912ec2250 con 0x564912f61490 2024-04-01T01:36:29.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2a7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2a7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f618b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37824/0 (socket says 172.21.15.70:37824) 2024-04-01T01:36:29.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2a7fff640 1 -- 172.21.15.70:0/1452673696 learned_addr learned my addr 172.21.15.70:0/1452673696 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:29.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2a77fe640 1 --2- 172.21.15.70:0/1452673696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61e80 0x564912f66980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2acfb2640 1 --2- 172.21.15.70:0/1452673696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f66ff0 0x564912f69480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2acfb2640 1 --2- 172.21.15.70:0/2231835811 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2900a2520 0x7fb2900a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2a7fff640 1 -- 172.21.15.70:0/1452673696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f66ff0 msgr2=0x564912f69480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:29.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2a7fff640 1 --2- 172.21.15.70:0/1452673696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f66ff0 0x564912f69480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2a7fff640 1 -- 172.21.15.70:0/1452673696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61e80 msgr2=0x564912f66980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:29.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2a7fff640 1 --2- 172.21.15.70:0/1452673696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61e80 0x564912f66980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.884+0000 7fb2a7fff640 1 -- 172.21.15.70:0/1452673696 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564912e8e3d0 con 0x564912f61490 2024-04-01T01:36:29.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.885+0000 7fb2acfb2640 1 --2- 172.21.15.70:0/2231835811 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2900a2520 0x7fb2900a49e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb29802c910 tx=0x7fb29802c5e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:29.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.885+0000 7fb2a7fff640 1 --2- 172.21.15.70:0/1452673696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f618b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb2a00114f0 tx=0x7fb2a00119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cd437e6edf4c397 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:29.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.885+0000 7fb28e7fc640 1 -- 172.21.15.70:0/1452673696 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a001e070 con 0x564912f61490 2024-04-01T01:36:29.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.885+0000 7fb28e7fc640 1 -- 172.21.15.70:0/1452673696 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2a000fad0 con 0x564912f61490 2024-04-01T01:36:29.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.885+0000 7fb28e7fc640 1 -- 172.21.15.70:0/1452673696 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0019b10 con 0x564912f61490 2024-04-01T01:36:29.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.886+0000 7fb2b15fb740 1 -- 172.21.15.70:0/1452673696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 msgr2=0x564912f618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.886+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/1452673696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f618b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb2a00114f0 tx=0x7fb2a00119b0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.886+0000 7fb2b15fb740 1 -- 172.21.15.70:0/1452673696 shutdown_connections 2024-04-01T01:36:29.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.886+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/1452673696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f66ff0 0x564912f69480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.886+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/1452673696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61e80 0x564912f66980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.886+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/1452673696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f618b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.886+0000 7fb2b15fb740 1 -- 172.21.15.70:0/1452673696 >> 172.21.15.70:0/1452673696 conn(0x564912f5ca00 msgr2=0x564912f5ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:29.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.887+0000 7fb2b15fb740 1 -- 172.21.15.70:0/1452673696 shutdown_connections 2024-04-01T01:36:29.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.887+0000 7fb2b15fb740 1 -- 172.21.15.70:0/1452673696 wait complete. 2024-04-01T01:36:29.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.887+0000 7fb2b15fb740 1 -- start start 2024-04-01T01:36:29.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.888+0000 7fb2b15fb740 1 Processor -- start 2024-04-01T01:36:29.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.888+0000 7fb2b15fb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f8d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.888+0000 7fb2b15fb740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f61e80 0x564912f8df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.888+0000 7fb2b15fb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f66ff0 0x564912f94f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.888+0000 7fb2b15fb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564912f7afb0 con 0x564912f66ff0 2024-04-01T01:36:29.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.888+0000 7fb2b15fb740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x564912e64470 con 0x564912f61e80 2024-04-01T01:36:29.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.888+0000 7fb2b15fb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564912f7b130 con 0x564912f61490 2024-04-01T01:36:29.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.888+0000 7fb2b15fb740 10 client.0 _ll_register_callbacks cb 0x564912e81cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:29.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.889+0000 7fb2a7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f8d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.889+0000 7fb2a7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f8d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37830/0 (socket says 172.21.15.70:37830) 2024-04-01T01:36:29.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.889+0000 7fb2a7fff640 1 -- 172.21.15.70:0/3958915721 learned_addr learned my addr 172.21.15.70:0/3958915721 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:29.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.889+0000 7fb2acfb2640 1 --2- 172.21.15.70:0/3958915721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f66ff0 0x564912f94f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.889+0000 7fb2a7fff640 1 -- 172.21.15.70:0/3958915721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f61e80 msgr2=0x564912f8df30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:36:29.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.889+0000 7fb2a7fff640 1 --2- 172.21.15.70:0/3958915721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f61e80 0x564912f8df30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.889+0000 7fb2a7fff640 1 -- 172.21.15.70:0/3958915721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f66ff0 msgr2=0x564912f94f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:29.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.889+0000 7fb2a7fff640 1 --2- 172.21.15.70:0/3958915721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f66ff0 0x564912f94f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.889+0000 7fb2a7fff640 1 -- 172.21.15.70:0/3958915721 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564912bda3f0 con 0x564912f61490 2024-04-01T01:36:29.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.890+0000 7fb2a7fff640 1 --2- 172.21.15.70:0/3958915721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f8d9f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb2a00114c0 tx=0x7fb2a000f8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d317cdb4334d6bc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:29.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.890+0000 7fb2877fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:36:29.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.890+0000 7fb2877fe640 1 -- 172.21.15.70:0/3958915721 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a001e040 con 0x564912f61490 2024-04-01T01:36:29.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.890+0000 7fb2877fe640 1 -- 172.21.15.70:0/3958915721 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb2a0016040 con 0x564912f61490 2024-04-01T01:36:29.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.890+0000 7fb2b15fb740 1 -- 172.21.15.70:0/3958915721 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2a000fad0 con 0x564912f61490 2024-04-01T01:36:29.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.891+0000 7fb2877fe640 1 -- 172.21.15.70:0/3958915721 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb2a0019410 con 0x564912f61490 2024-04-01T01:36:29.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.891+0000 7fb2b15fb740 10 client.26212 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:29.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.891+0000 7fb2b15fb740 1 -- 172.21.15.70:0/3958915721 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564912ca47d0 con 0x564912f61490 2024-04-01T01:36:29.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.892+0000 7fb2877fe640 1 -- 172.21.15.70:0/3958915721 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2a000fad0 con 0x564912f61490 2024-04-01T01:36:29.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.892+0000 7fb2877fe640 1 -- 172.21.15.70:0/3958915721 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb2a0005040 con 0x564912f61490 2024-04-01T01:36:29.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.892+0000 7fb2b15fb740 10 client.26212 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:29.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.892+0000 7fb2b15fb740 10 client.26212 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:36:29.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.892+0000 7fb2b15fb740 20 client.26212 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:29.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.892+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/3958915721 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x564912f7f1e0 0x564912f81680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.892+0000 7fb2b15fb740 4 client.26212 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:36:29.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.892+0000 7fb2b15fb740 1 -- 172.21.15.70:0/3958915721 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564912ed36f0 con 0x564912f7f1e0 2024-04-01T01:36:29.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.893+0000 7fb2a77fe640 1 --2- 172.21.15.70:0/3958915721 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x564912f7f1e0 0x564912f81680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.894+0000 7fb2a77fe640 1 --2- 172.21.15.70:0/3958915721 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x564912f7f1e0 0x564912f81680 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=95defff7ef744af6 server_cookie=28db35c17be03ba4 in_seq=0 out_seq=0 2024-04-01T01:36:29.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.894+0000 7fb2877fe640 10 client.26212 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:36:29.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.895+0000 7fb2877fe640 1 -- 172.21.15.70:0/3958915721 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564912ed36f0 con 0x564912f7f1e0 2024-04-01T01:36:29.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.895+0000 7fb2877fe640 10 client.26212 handle_command_reply: tid=0 2024-04-01T01:36:29.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.895+0000 7fb2b15fb740 1 client.26212 shutdown 2024-04-01T01:36:29.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.896+0000 7fb2b15fb740 1 -- 172.21.15.70:0/3958915721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 msgr2=0x564912f8d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.896+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/3958915721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f8d9f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb2a00114c0 tx=0x7fb2a000f8d0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.896+0000 7fb2b15fb740 1 -- 172.21.15.70:0/3958915721 shutdown_connections 2024-04-01T01:36:29.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.896+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/3958915721 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x564912f7f1e0 0x564912f81680 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.896+0000 7fb2877fe640 0 client.26212 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:36:29.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.896+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/3958915721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f66ff0 0x564912f94f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.896+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/3958915721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f61e80 0x564912f8df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.896+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/3958915721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f8d9f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.896+0000 7fb2b15fb740 1 -- 172.21.15.70:0/3958915721 >> 172.21.15.70:0/3958915721 conn(0x564912f5ca00 msgr2=0x564912f5bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:29.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.896+0000 7fb2b15fb740 1 -- 172.21.15.70:0/3958915721 shutdown_connections 2024-04-01T01:36:29.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.896+0000 7fb2b15fb740 1 -- 172.21.15.70:0/3958915721 wait complete. 2024-04-01T01:36:29.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.896+0000 7fb2b15fb740 20 client.26212 trim_cache size 0 max 16384 2024-04-01T01:36:29.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.924+0000 7fb2b15fb740 1 Processor -- start 2024-04-01T01:36:29.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.924+0000 7fb2b15fb740 1 -- start start 2024-04-01T01:36:29.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.924+0000 7fb2b15fb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61490 0x564912f8d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.925+0000 7fb2b15fb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61e80 0x564912f94f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.925+0000 7fb2b15fb740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f66ff0 0x564912f8d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.925+0000 7fb2a7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61490 0x564912f8d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.925+0000 7fb2a7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61490 0x564912f8d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59712/0 (socket says 172.21.15.70:59712) 2024-04-01T01:36:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.925+0000 7fb2a7fff640 1 -- 172.21.15.70:0/645319451 learned_addr learned my addr 172.21.15.70:0/645319451 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.925+0000 7fb2a77fe640 1 --2- 172.21.15.70:0/645319451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61e80 0x564912f94f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.926+0000 7fb2acfb2640 1 --2- 172.21.15.70:0/645319451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f66ff0 0x564912f8d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.926+0000 7fb2b15fb740 1 -- 172.21.15.70:0/645319451 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a806cd70 con 0x564912f61490 2024-04-01T01:36:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.926+0000 7fb2b15fb740 1 -- 172.21.15.70:0/645319451 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a806e8b0 con 0x564912f66ff0 2024-04-01T01:36:29.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.926+0000 7fb2b15fb740 1 -- 172.21.15.70:0/645319451 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a806d440 con 0x564912f61e80 2024-04-01T01:36:29.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.926+0000 7fb2a7fff640 1 -- 172.21.15.70:0/645319451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61e80 msgr2=0x564912f94f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:29.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.926+0000 7fb2a7fff640 1 --2- 172.21.15.70:0/645319451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61e80 0x564912f94f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.926+0000 7fb2a7fff640 1 -- 172.21.15.70:0/645319451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f66ff0 msgr2=0x564912f8d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:29.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.926+0000 7fb2a7fff640 1 --2- 172.21.15.70:0/645319451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f66ff0 0x564912f8d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.926+0000 7fb2a7fff640 1 -- 172.21.15.70:0/645319451 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564912f7ae90 con 0x564912f61490 2024-04-01T01:36:29.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.927+0000 7fb2acfb2640 1 --2- 172.21.15.70:0/645319451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f66ff0 0x564912f8d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:36:29.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.927+0000 7fb2a7fff640 1 --2- 172.21.15.70:0/645319451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61490 0x564912f8d2c0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb2a00114f0 tx=0x7fb2a0005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a4b28172f79ca7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:29.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.927+0000 7fb28dffb640 1 -- 172.21.15.70:0/645319451 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0035070 con 0x564912f61490 2024-04-01T01:36:29.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.927+0000 7fb28dffb640 1 -- 172.21.15.70:0/645319451 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2a0003e20 con 0x564912f61490 2024-04-01T01:36:29.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.927+0000 7fb28dffb640 1 -- 172.21.15.70:0/645319451 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a001e030 con 0x564912f61490 2024-04-01T01:36:29.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.928+0000 7fb2b15fb740 1 -- 172.21.15.70:0/645319451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61490 msgr2=0x564912f8d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.928+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/645319451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61490 0x564912f8d2c0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb2a00114f0 tx=0x7fb2a0005f10 comp rx=0 tx=0).stop 2024-04-01T01:36:29.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.928+0000 7fb2b15fb740 1 -- 172.21.15.70:0/645319451 shutdown_connections 2024-04-01T01:36:29.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.928+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/645319451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f66ff0 0x564912f8d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.928+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/645319451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61e80 0x564912f94f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.928+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/645319451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61490 0x564912f8d2c0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.928+0000 7fb2b15fb740 1 -- 172.21.15.70:0/645319451 >> 172.21.15.70:0/645319451 conn(0x564912f5ca00 msgr2=0x564912f604f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.929+0000 7fb2b15fb740 1 -- 172.21.15.70:0/645319451 shutdown_connections 2024-04-01T01:36:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.929+0000 7fb2b15fb740 1 -- 172.21.15.70:0/645319451 wait complete. 2024-04-01T01:36:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.929+0000 7fb2b15fb740 1 -- start start 2024-04-01T01:36:29.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.929+0000 7fb2b15fb740 1 Processor -- start 2024-04-01T01:36:29.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.930+0000 7fb2b15fb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f921e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.930+0000 7fb2b15fb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61e80 0x564912f92720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.930+0000 7fb2b15fb740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f66ff0 0x564912f93730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.930+0000 7fb2b15fb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a806e1e0 con 0x564912f61e80 2024-04-01T01:36:29.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.930+0000 7fb2b15fb740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a806db10 con 0x564912f66ff0 2024-04-01T01:36:29.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.930+0000 7fb2b15fb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a810f620 con 0x564912f61490 2024-04-01T01:36:29.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.930+0000 7fb2b15fb740 10 client.0 _ll_register_callbacks cb 0x7fb2a0009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:29.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.930+0000 7fb2a77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61e80 0x564912f92720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.930+0000 7fb2a77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61e80 0x564912f92720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59726/0 (socket says 172.21.15.70:59726) 2024-04-01T01:36:29.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.930+0000 7fb2a77fe640 1 -- 172.21.15.70:0/387333716 learned_addr learned my addr 172.21.15.70:0/387333716 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:29.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.930+0000 7fb2acfb2640 1 --2- 172.21.15.70:0/387333716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f66ff0 0x564912f93730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.930+0000 7fb2a7fff640 1 --2- 172.21.15.70:0/387333716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f921e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.931+0000 7fb2a7fff640 1 -- 172.21.15.70:0/387333716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f66ff0 msgr2=0x564912f93730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:29.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.931+0000 7fb2a7fff640 1 --2- 172.21.15.70:0/387333716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f66ff0 0x564912f93730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.931+0000 7fb2a7fff640 1 -- 172.21.15.70:0/387333716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61e80 msgr2=0x564912f92720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:29.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.931+0000 7fb2a7fff640 1 --2- 172.21.15.70:0/387333716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61e80 0x564912f92720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.931+0000 7fb2a7fff640 1 -- 172.21.15.70:0/387333716 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564912f93d20 con 0x564912f61490 2024-04-01T01:36:29.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.931+0000 7fb2a7fff640 1 --2- 172.21.15.70:0/387333716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f921e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x564912f8e6c0 tx=0x7fb2a0005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb063d87e14dc20e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:29.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.931+0000 7fb2877fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:36:29.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.932+0000 7fb2877fe640 1 -- 172.21.15.70:0/387333716 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0035070 con 0x564912f61490 2024-04-01T01:36:29.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.932+0000 7fb2b15fb740 1 -- 172.21.15.70:0/387333716 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2a0003e20 con 0x564912f61490 2024-04-01T01:36:29.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.932+0000 7fb2877fe640 1 -- 172.21.15.70:0/387333716 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb2a0003b80 con 0x564912f61490 2024-04-01T01:36:29.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.932+0000 7fb2877fe640 1 -- 172.21.15.70:0/387333716 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb2a001e070 con 0x564912f61490 2024-04-01T01:36:29.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.932+0000 7fb2b15fb740 10 client.26218 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:29.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.932+0000 7fb2b15fb740 1 -- 172.21.15.70:0/387333716 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564912f94f90 con 0x564912f61490 2024-04-01T01:36:29.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.933+0000 7fb2877fe640 1 -- 172.21.15.70:0/387333716 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2a0019a60 con 0x564912f61490 2024-04-01T01:36:29.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.933+0000 7fb2877fe640 1 -- 172.21.15.70:0/387333716 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb2a00246c0 con 0x564912f61490 2024-04-01T01:36:29.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.934+0000 7fb2b15fb740 10 client.26218 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:29.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.934+0000 7fb2b15fb740 10 client.26218 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:36:29.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.934+0000 7fb2b15fb740 20 client.26218 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:29.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.934+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/387333716 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x564913010d40 0x564913013130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:29.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.934+0000 7fb2b15fb740 4 client.26218 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:36:29.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.934+0000 7fb2b15fb740 1 -- 172.21.15.70:0/387333716 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fb2a0003e20 con 0x564913010d40 2024-04-01T01:36:29.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.934+0000 7fb2a77fe640 1 --2- 172.21.15.70:0/387333716 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x564913010d40 0x564913013130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:29.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.935+0000 7fb2a77fe640 1 --2- 172.21.15.70:0/387333716 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x564913010d40 0x564913013130 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9ee9f61428900dfa server_cookie=bcbe294c5e1ce468 in_seq=0 out_seq=0 2024-04-01T01:36:29.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.935+0000 7fb2877fe640 10 client.26218 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:36:29.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.935+0000 7fb2877fe640 1 -- 172.21.15.70:0/387333716 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fb2a0003e20 con 0x564913010d40 2024-04-01T01:36:29.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.935+0000 7fb2877fe640 10 client.26218 handle_command_reply: tid=0 2024-04-01T01:36:29.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.935+0000 7fb2b15fb740 1 client.26218 shutdown 2024-04-01T01:36:29.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.935+0000 7fb2b15fb740 1 -- 172.21.15.70:0/387333716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 msgr2=0x564912f921e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.936+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/387333716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f921e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x564912f8e6c0 tx=0x7fb2a0005270 comp rx=0 tx=0).stop 2024-04-01T01:36:29.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.936+0000 7fb2b15fb740 1 -- 172.21.15.70:0/387333716 shutdown_connections 2024-04-01T01:36:29.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.936+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/387333716 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x564913010d40 0x564913013130 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.936+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/387333716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564912f66ff0 0x564912f93730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.936+0000 7fb2877fe640 0 client.26218 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:36:29.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.936+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/387333716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564912f61e80 0x564912f92720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.936+0000 7fb2b15fb740 1 --2- 172.21.15.70:0/387333716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564912f61490 0x564912f921e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.936+0000 7fb2b15fb740 1 -- 172.21.15.70:0/387333716 >> 172.21.15.70:0/387333716 conn(0x564912f5ca00 msgr2=0x564912f5b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:29.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.936+0000 7fb2b15fb740 1 -- 172.21.15.70:0/387333716 shutdown_connections 2024-04-01T01:36:29.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.936+0000 7fb2b15fb740 1 -- 172.21.15.70:0/387333716 wait complete. 2024-04-01T01:36:29.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.936+0000 7fb2b15fb740 20 client.26218 trim_cache size 0 max 16384 2024-04-01T01:36:29.939 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:36:29.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.938+0000 7fb2aea15640 1 -- 172.21.15.70:0/2231835811 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2900a2520 msgr2=0x7fb2900a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.938+0000 7fb2aea15640 1 --2- 172.21.15.70:0/2231835811 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2900a2520 0x7fb2900a49e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb29802c910 tx=0x7fb29802c5e0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.938+0000 7fb2aea15640 1 -- 172.21.15.70:0/2231835811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a8100740 msgr2=0x7fb2a819e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:29.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.938+0000 7fb2aea15640 1 --2- 172.21.15.70:0/2231835811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a8100740 0x7fb2a819e480 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb29c00e9f0 tx=0x7fb29c00eeb0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.938+0000 7fb2aea15640 1 -- 172.21.15.70:0/2231835811 shutdown_connections 2024-04-01T01:36:29.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.938+0000 7fb2aea15640 1 --2- 172.21.15.70:0/2231835811 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2900a2520 0x7fb2900a49e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.938+0000 7fb2aea15640 1 --2- 172.21.15.70:0/2231835811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2a81010e0 0x7fb2a81a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.938+0000 7fb2aea15640 1 --2- 172.21.15.70:0/2231835811 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a8100740 0x7fb2a819e480 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.938+0000 7fb2aea15640 1 --2- 172.21.15.70:0/2231835811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2a80ff610 0x7fb2a819df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:29.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.938+0000 7fb2aea15640 1 -- 172.21.15.70:0/2231835811 >> 172.21.15.70:0/2231835811 conn(0x7fb2a80faf10 msgr2=0x7fb2a8101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:29.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.938+0000 7fb2aea15640 1 -- 172.21.15.70:0/2231835811 shutdown_connections 2024-04-01T01:36:29.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:29.938+0000 7fb2aea15640 1 -- 172.21.15.70:0/2231835811 wait complete. 2024-04-01T01:36:29.959 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:36:29.960 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:29 smithi070 ceph-mon[32179]: pgmap v448: 129 pgs: 129 active+clean; 990 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-04-01T01:36:29.960 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:29 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1126190022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:36:29.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:29 smithi189 ceph-mon[35753]: pgmap v448: 129 pgs: 129 active+clean; 990 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-04-01T01:36:29.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:29 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1126190022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:36:30.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:29 smithi081 ceph-mon[35142]: pgmap v448: 129 pgs: 129 active+clean; 990 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 1.33k op/s 2024-04-01T01:36:30.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:29 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1126190022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:36:30.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.128+0000 7f7556a8d640 1 -- 172.21.15.70:0/3216631687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7550101840 msgr2=0x7f7550101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.128+0000 7f7556a8d640 1 --2- 172.21.15.70:0/3216631687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7550101840 0x7f7550101cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f753c009860 tx=0x7f753c02bb70 comp rx=0 tx=0).stop 2024-04-01T01:36:30.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.129+0000 7f7556a8d640 1 -- 172.21.15.70:0/3216631687 shutdown_connections 2024-04-01T01:36:30.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.129+0000 7f7556a8d640 1 --2- 172.21.15.70:0/3216631687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7550102200 0x7f7550108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.129+0000 7f7556a8d640 1 --2- 172.21.15.70:0/3216631687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7550101840 0x7f7550101cc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.129+0000 7f7556a8d640 1 --2- 172.21.15.70:0/3216631687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7550100650 0x7f7550100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.129+0000 7f7556a8d640 1 -- 172.21.15.70:0/3216631687 >> 172.21.15.70:0/3216631687 conn(0x7f75500fbbe0 msgr2=0x7f75500fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.129+0000 7f7556a8d640 1 -- 172.21.15.70:0/3216631687 shutdown_connections 2024-04-01T01:36:30.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.129+0000 7f7556a8d640 1 -- 172.21.15.70:0/3216631687 wait complete. 2024-04-01T01:36:30.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.130+0000 7f7556a8d640 1 Processor -- start 2024-04-01T01:36:30.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.130+0000 7f7556a8d640 1 -- start start 2024-04-01T01:36:30.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.130+0000 7f7556a8d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7550100650 0x7f7550110030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.130+0000 7f7556a8d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7550101840 0x7f7550110570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.131+0000 7f7556a8d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7550102200 0x7f755010cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.131+0000 7f7556a8d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f755010b060 con 0x7f7550101840 2024-04-01T01:36:30.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.131+0000 7f7556a8d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f755010aee0 con 0x7f7550100650 2024-04-01T01:36:30.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.131+0000 7f7556a8d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f755010b1e0 con 0x7f7550102200 2024-04-01T01:36:30.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.131+0000 7f755502a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7550102200 0x7f755010cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.131+0000 7f755502a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7550102200 0x7f755010cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37858/0 (socket says 172.21.15.70:37858) 2024-04-01T01:36:30.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.131+0000 7f755502a640 1 -- 172.21.15.70:0/1990136943 learned_addr learned my addr 172.21.15.70:0/1990136943 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.131+0000 7f7554829640 1 --2- 172.21.15.70:0/1990136943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7550100650 0x7f7550110030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.131+0000 7f754ffff640 1 --2- 172.21.15.70:0/1990136943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7550101840 0x7f7550110570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.132+0000 7f7554829640 1 -- 172.21.15.70:0/1990136943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7550102200 msgr2=0x7f755010cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.132+0000 7f7554829640 1 --2- 172.21.15.70:0/1990136943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7550102200 0x7f755010cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.132+0000 7f7554829640 1 -- 172.21.15.70:0/1990136943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7550101840 msgr2=0x7f7550110570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.132+0000 7f7554829640 1 --2- 172.21.15.70:0/1990136943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7550101840 0x7f7550110570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.132+0000 7f7554829640 1 -- 172.21.15.70:0/1990136943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f755010d460 con 0x7f7550100650 2024-04-01T01:36:30.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.132+0000 7f754ffff640 1 --2- 172.21.15.70:0/1990136943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7550101840 0x7f7550110570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:36:30.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.132+0000 7f7554829640 1 --2- 172.21.15.70:0/1990136943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7550100650 0x7f7550110030 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f754400e9c0 tx=0x7f754400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.132+0000 7f754dffb640 1 -- 172.21.15.70:0/1990136943 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754400ccc0 con 0x7f7550100650 2024-04-01T01:36:30.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.132+0000 7f754dffb640 1 -- 172.21.15.70:0/1990136943 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75440035f0 con 0x7f7550100650 2024-04-01T01:36:30.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.132+0000 7f754dffb640 1 -- 172.21.15.70:0/1990136943 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754400ce60 con 0x7f7550100650 2024-04-01T01:36:30.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.132+0000 7f7556a8d640 1 -- 172.21.15.70:0/1990136943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75501a6000 con 0x7f7550100650 2024-04-01T01:36:30.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.133+0000 7f7556a8d640 1 -- 172.21.15.70:0/1990136943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75501a63b0 con 0x7f7550100650 2024-04-01T01:36:30.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.136+0000 7f754dffb640 1 -- 172.21.15.70:0/1990136943 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f75440037a0 con 0x7f7550100650 2024-04-01T01:36:30.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.137+0000 7f754dffb640 1 --2- 172.21.15.70:0/1990136943 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f75400a2520 0x7f75400a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.137+0000 7f754dffb640 1 -- 172.21.15.70:0/1990136943 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f75440cef30 con 0x7f7550100650 2024-04-01T01:36:30.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.138+0000 7f754ffff640 1 --2- 172.21.15.70:0/1990136943 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f75400a2520 0x7f75400a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.138+0000 7f755966a740 1 Processor -- start 2024-04-01T01:36:30.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.138+0000 7f755966a740 1 -- start start 2024-04-01T01:36:30.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.138+0000 7f754ffff640 1 --2- 172.21.15.70:0/1990136943 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f75400a2520 0x7f75400a49e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f753c02c180 tx=0x7f753c034980 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.138+0000 7f755966a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364ea490 0x5643364ea8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.138+0000 7f755966a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364eae80 0x5643364ef980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.138+0000 7f755966a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364efff0 0x5643364f2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.138+0000 7f7554829640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364ea490 0x5643364ea8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.138+0000 7f755966a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56433644f1b0 con 0x5643364efff0 2024-04-01T01:36:30.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.138+0000 7f755966a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5643363f7c00 con 0x5643364ea490 2024-04-01T01:36:30.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.138+0000 7f755966a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56433644b250 con 0x5643364eae80 2024-04-01T01:36:30.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.138+0000 7f7554829640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364ea490 0x5643364ea8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35456/0 (socket says 172.21.15.70:35456) 2024-04-01T01:36:30.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.138+0000 7f7554829640 1 -- 172.21.15.70:0/2634465753 learned_addr learned my addr 172.21.15.70:0/2634465753 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.139+0000 7f755502a640 1 --2- 172.21.15.70:0/2634465753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364efff0 0x5643364f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.139+0000 7f754ffff640 1 --2- 172.21.15.70:0/2634465753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364eae80 0x5643364ef980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.139+0000 7f7554829640 1 -- 172.21.15.70:0/2634465753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364eae80 msgr2=0x5643364ef980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.139+0000 7f7554829640 1 --2- 172.21.15.70:0/2634465753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364eae80 0x5643364ef980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.139+0000 7f7554829640 1 -- 172.21.15.70:0/2634465753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364efff0 msgr2=0x5643364f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.139+0000 7f7554829640 1 --2- 172.21.15.70:0/2634465753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364efff0 0x5643364f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.139+0000 7f7554829640 1 -- 172.21.15.70:0/2634465753 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643364173d0 con 0x5643364ea490 2024-04-01T01:36:30.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.139+0000 7f7554829640 1 --2- 172.21.15.70:0/2634465753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364ea490 0x5643364ea8b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f754409b5b0 tx=0x7f754409b0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69baf6e9bc6690b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.139+0000 7f753affd640 1 -- 172.21.15.70:0/2634465753 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75440b3020 con 0x5643364ea490 2024-04-01T01:36:30.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.140+0000 7f753affd640 1 -- 172.21.15.70:0/2634465753 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f754409b620 con 0x5643364ea490 2024-04-01T01:36:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.140+0000 7f753affd640 1 -- 172.21.15.70:0/2634465753 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7544084bb0 con 0x5643364ea490 2024-04-01T01:36:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.140+0000 7f755966a740 1 -- 172.21.15.70:0/2634465753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364ea490 msgr2=0x5643364ea8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.140+0000 7f755966a740 1 --2- 172.21.15.70:0/2634465753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364ea490 0x5643364ea8b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f754409b5b0 tx=0x7f754409b0b0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.141+0000 7f755966a740 1 -- 172.21.15.70:0/2634465753 shutdown_connections 2024-04-01T01:36:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.141+0000 7f755966a740 1 --2- 172.21.15.70:0/2634465753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364efff0 0x5643364f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.141+0000 7f755966a740 1 --2- 172.21.15.70:0/2634465753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364eae80 0x5643364ef980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.141+0000 7f755966a740 1 --2- 172.21.15.70:0/2634465753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364ea490 0x5643364ea8b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.141+0000 7f755966a740 1 -- 172.21.15.70:0/2634465753 >> 172.21.15.70:0/2634465753 conn(0x5643364e5a00 msgr2=0x5643364e5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.141+0000 7f755966a740 1 -- 172.21.15.70:0/2634465753 shutdown_connections 2024-04-01T01:36:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.141+0000 7f755966a740 1 -- 172.21.15.70:0/2634465753 wait complete. 2024-04-01T01:36:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.141+0000 7f755966a740 1 -- start start 2024-04-01T01:36:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.141+0000 7f755966a740 1 Processor -- start 2024-04-01T01:36:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.142+0000 7f755966a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364ea490 0x5643364f49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.142+0000 7f755966a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364eae80 0x5643364f4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.142+0000 7f755966a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364efff0 0x5643364fbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.142+0000 7f755966a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5643364f2ff0 con 0x5643364efff0 2024-04-01T01:36:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.142+0000 7f755966a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5643363ed470 con 0x5643364eae80 2024-04-01T01:36:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.142+0000 7f755966a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5643364f3170 con 0x5643364ea490 2024-04-01T01:36:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.142+0000 7f755966a740 10 client.0 _ll_register_callbacks cb 0x56433640acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.142+0000 7f754ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364eae80 0x5643364f4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.142+0000 7f754ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364eae80 0x5643364f4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35466/0 (socket says 172.21.15.70:35466) 2024-04-01T01:36:30.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.142+0000 7f754ffff640 1 -- 172.21.15.70:0/1232041207 learned_addr learned my addr 172.21.15.70:0/1232041207 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.142+0000 7f755502a640 1 --2- 172.21.15.70:0/1232041207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364efff0 0x5643364fbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.142+0000 7f7554829640 1 --2- 172.21.15.70:0/1232041207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364ea490 0x5643364f49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.143+0000 7f754ffff640 1 -- 172.21.15.70:0/1232041207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364ea490 msgr2=0x5643364f49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.143+0000 7f754ffff640 1 --2- 172.21.15.70:0/1232041207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364ea490 0x5643364f49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.143+0000 7f754ffff640 1 -- 172.21.15.70:0/1232041207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364efff0 msgr2=0x5643364fbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.143+0000 7f754ffff640 1 --2- 172.21.15.70:0/1232041207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364efff0 0x5643364fbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.143+0000 7f754ffff640 1 -- 172.21.15.70:0/1232041207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643361633f0 con 0x5643364eae80 2024-04-01T01:36:30.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.143+0000 7f754ffff640 1 --2- 172.21.15.70:0/1232041207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364eae80 0x5643364f4f20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f753c009780 tx=0x7f753c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b46f341b0fe7b4e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.143+0000 7f752ffff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:36:30.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.143+0000 7f752ffff640 1 -- 172.21.15.70:0/1232041207 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c04f020 con 0x5643364eae80 2024-04-01T01:36:30.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.144+0000 7f755966a740 1 -- 172.21.15.70:0/1232041207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f754409b620 con 0x5643364eae80 2024-04-01T01:36:30.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.144+0000 7f752ffff640 1 -- 172.21.15.70:0/1232041207 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f753c02c710 con 0x5643364eae80 2024-04-01T01:36:30.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.144+0000 7f752ffff640 1 -- 172.21.15.70:0/1232041207 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f753c05a050 con 0x5643364eae80 2024-04-01T01:36:30.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.144+0000 7f755966a740 10 client.26327 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:30.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.144+0000 7f755966a740 1 -- 172.21.15.70:0/1232041207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56433622d7d0 con 0x5643364eae80 2024-04-01T01:36:30.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.144+0000 7f752ffff640 1 -- 172.21.15.70:0/1232041207 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f754409b620 con 0x5643364eae80 2024-04-01T01:36:30.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.145+0000 7f752ffff640 1 -- 172.21.15.70:0/1232041207 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f753c0035f0 con 0x5643364eae80 2024-04-01T01:36:30.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.145+0000 7f755966a740 10 client.26327 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:30.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.145+0000 7f755966a740 10 client.26327 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:36:30.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.145+0000 7f755966a740 20 client.26327 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:30.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.145+0000 7f755966a740 1 --2- 172.21.15.70:0/1232041207 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5643365008c0 0x564336502d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.145+0000 7f755966a740 4 client.26327 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:36:30.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.145+0000 7f755966a740 1 -- 172.21.15.70:0/1232041207 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56433645c6f0 con 0x5643365008c0 2024-04-01T01:36:30.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.145+0000 7f7554829640 1 --2- 172.21.15.70:0/1232041207 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5643365008c0 0x564336502d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.146+0000 7f7554829640 1 --2- 172.21.15.70:0/1232041207 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5643365008c0 0x564336502d20 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1046fde90877483 server_cookie=cd64a9b461f3ced9 in_seq=0 out_seq=0 2024-04-01T01:36:30.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.146+0000 7f752ffff640 10 client.26327 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:36:30.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.147+0000 7f752ffff640 1 -- 172.21.15.70:0/1232041207 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56433645c6f0 con 0x5643365008c0 2024-04-01T01:36:30.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.147+0000 7f752ffff640 10 client.26327 handle_command_reply: tid=0 2024-04-01T01:36:30.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.147+0000 7f755966a740 1 client.26327 shutdown 2024-04-01T01:36:30.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.148+0000 7f755966a740 1 -- 172.21.15.70:0/1232041207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364eae80 msgr2=0x5643364f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.148+0000 7f755966a740 1 --2- 172.21.15.70:0/1232041207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364eae80 0x5643364f4f20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f753c009780 tx=0x7f753c005f90 comp rx=0 tx=0).stop 2024-04-01T01:36:30.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.148+0000 7f755966a740 1 -- 172.21.15.70:0/1232041207 shutdown_connections 2024-04-01T01:36:30.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.149+0000 7f755966a740 1 --2- 172.21.15.70:0/1232041207 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5643365008c0 0x564336502d20 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.149+0000 7f755966a740 1 --2- 172.21.15.70:0/1232041207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364efff0 0x5643364fbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.149+0000 7f755966a740 1 --2- 172.21.15.70:0/1232041207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364eae80 0x5643364f4f20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.149+0000 7f755966a740 1 --2- 172.21.15.70:0/1232041207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364ea490 0x5643364f49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.149+0000 7f755966a740 1 -- 172.21.15.70:0/1232041207 >> 172.21.15.70:0/1232041207 conn(0x5643364e5a00 msgr2=0x5643364e4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.149+0000 7f752ffff640 0 client.26327 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:36:30.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.149+0000 7f755966a740 1 -- 172.21.15.70:0/1232041207 shutdown_connections 2024-04-01T01:36:30.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.149+0000 7f755966a740 1 -- 172.21.15.70:0/1232041207 wait complete. 2024-04-01T01:36:30.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.149+0000 7f755966a740 20 client.26327 trim_cache size 0 max 16384 2024-04-01T01:36:30.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.163+0000 7f755966a740 1 Processor -- start 2024-04-01T01:36:30.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.163+0000 7f755966a740 1 -- start start 2024-04-01T01:36:30.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.163+0000 7f755966a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364ea490 0x5643364f4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.163+0000 7f755966a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364eae80 0x5643364fbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.163+0000 7f755966a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364efff0 0x5643364e44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f754ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364eae80 0x5643364fbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f7554829640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364ea490 0x5643364f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f754ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364eae80 0x5643364fbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59756/0 (socket says 172.21.15.70:59756) 2024-04-01T01:36:30.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f7554829640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364ea490 0x5643364f4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35470/0 (socket says 172.21.15.70:35470) 2024-04-01T01:36:30.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f754ffff640 1 -- 172.21.15.70:0/2947087861 learned_addr learned my addr 172.21.15.70:0/2947087861 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f755966a740 1 -- 172.21.15.70:0/2947087861 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f75501b0a40 con 0x5643364eae80 2024-04-01T01:36:30.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f755966a740 1 -- 172.21.15.70:0/2947087861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f75501b2580 con 0x5643364ea490 2024-04-01T01:36:30.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f755966a740 1 -- 172.21.15.70:0/2947087861 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f75501b1110 con 0x5643364efff0 2024-04-01T01:36:30.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f755502a640 1 --2- 172.21.15.70:0/2947087861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364efff0 0x5643364e44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f754ffff640 1 -- 172.21.15.70:0/2947087861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364efff0 msgr2=0x5643364e44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f754ffff640 1 --2- 172.21.15.70:0/2947087861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364efff0 0x5643364e44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f754ffff640 1 -- 172.21.15.70:0/2947087861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364ea490 msgr2=0x5643364f4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f754ffff640 1 --2- 172.21.15.70:0/2947087861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364ea490 0x5643364f4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f754ffff640 1 -- 172.21.15.70:0/2947087861 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643364e6470 con 0x5643364eae80 2024-04-01T01:36:30.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f754ffff640 1 --2- 172.21.15.70:0/2947087861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364eae80 0x5643364fbf20 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f753c003ae0 tx=0x7f753c0045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22a5da8ae2f1026 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f753a7fc640 1 -- 172.21.15.70:0/2947087861 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c063070 con 0x5643364eae80 2024-04-01T01:36:30.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f753a7fc640 1 -- 172.21.15.70:0/2947087861 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f753c046040 con 0x5643364eae80 2024-04-01T01:36:30.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.164+0000 7f753a7fc640 1 -- 172.21.15.70:0/2947087861 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c04f040 con 0x5643364eae80 2024-04-01T01:36:30.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.165+0000 7f755966a740 1 -- 172.21.15.70:0/2947087861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364eae80 msgr2=0x5643364fbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.165+0000 7f755966a740 1 --2- 172.21.15.70:0/2947087861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364eae80 0x5643364fbf20 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f753c003ae0 tx=0x7f753c0045a0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.165+0000 7f755966a740 1 -- 172.21.15.70:0/2947087861 shutdown_connections 2024-04-01T01:36:30.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.165+0000 7f755966a740 1 --2- 172.21.15.70:0/2947087861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364efff0 0x5643364e44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.165+0000 7f755966a740 1 --2- 172.21.15.70:0/2947087861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364eae80 0x5643364fbf20 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.165+0000 7f755966a740 1 --2- 172.21.15.70:0/2947087861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364ea490 0x5643364f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.165+0000 7f755966a740 1 -- 172.21.15.70:0/2947087861 >> 172.21.15.70:0/2947087861 conn(0x5643364e5a00 msgr2=0x5643364ec340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.165+0000 7f755966a740 1 -- 172.21.15.70:0/2947087861 shutdown_connections 2024-04-01T01:36:30.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.165+0000 7f755966a740 1 -- 172.21.15.70:0/2947087861 wait complete. 2024-04-01T01:36:30.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.165+0000 7f755966a740 1 -- start start 2024-04-01T01:36:30.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.165+0000 7f755966a740 1 Processor -- start 2024-04-01T01:36:30.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f755966a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364ea490 0x5643364f9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f755966a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364eae80 0x5643364f97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f755966a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364efff0 0x5643364fa7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f755966a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f75501b1eb0 con 0x5643364ea490 2024-04-01T01:36:30.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f755966a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f75501b17e0 con 0x5643364efff0 2024-04-01T01:36:30.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f755966a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f75501aea20 con 0x5643364eae80 2024-04-01T01:36:30.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f7554829640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364ea490 0x5643364f9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f755966a740 10 client.0 _ll_register_callbacks cb 0x7f753c004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:30.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f754ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364eae80 0x5643364f97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f754ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364eae80 0x5643364f97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37892/0 (socket says 172.21.15.70:37892) 2024-04-01T01:36:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f754ffff640 1 -- 172.21.15.70:0/508450384 learned_addr learned my addr 172.21.15.70:0/508450384 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f7554829640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364ea490 0x5643364f9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59768/0 (socket says 172.21.15.70:59768) 2024-04-01T01:36:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f755502a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364efff0 0x5643364fa7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f7554829640 1 -- 172.21.15.70:0/508450384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364eae80 msgr2=0x5643364f97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f7554829640 1 --2- 172.21.15.70:0/508450384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364eae80 0x5643364f97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f7554829640 1 -- 172.21.15.70:0/508450384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364efff0 msgr2=0x5643364fa7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f7554829640 1 --2- 172.21.15.70:0/508450384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364efff0 0x5643364fa7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.166+0000 7f7554829640 1 -- 172.21.15.70:0/508450384 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643364fadb0 con 0x5643364ea490 2024-04-01T01:36:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f755502a640 1 --2- 172.21.15.70:0/508450384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364efff0 0x5643364fa7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:36:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f7554829640 1 --2- 172.21.15.70:0/508450384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364ea490 0x5643364f9270 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x5643364eb6e0 tx=0x7f754409b160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f011224c2984965a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f752ffff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:36:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f752ffff640 1 -- 172.21.15.70:0/508450384 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75440b3070 con 0x5643364ea490 2024-04-01T01:36:30.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f752ffff640 1 -- 172.21.15.70:0/508450384 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f75440b4260 con 0x5643364ea490 2024-04-01T01:36:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f755966a740 1 -- 172.21.15.70:0/508450384 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f753c046040 con 0x5643364ea490 2024-04-01T01:36:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f752ffff640 1 -- 172.21.15.70:0/508450384 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f75440a4030 con 0x5643364ea490 2024-04-01T01:36:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f755966a740 10 client.16500 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f755966a740 1 -- 172.21.15.70:0/508450384 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5643364fc120 con 0x5643364ea490 2024-04-01T01:36:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f752ffff640 1 -- 172.21.15.70:0/508450384 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f753c046040 con 0x5643364ea490 2024-04-01T01:36:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f752ffff640 1 -- 172.21.15.70:0/508450384 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f75440ccb20 con 0x5643364ea490 2024-04-01T01:36:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f755966a740 10 client.16500 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f755966a740 10 client.16500 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:36:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f755966a740 20 client.16500 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f755966a740 1 --2- 172.21.15.70:0/508450384 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x564336566e70 0x564336569290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f755966a740 4 client.16500 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:36:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.167+0000 7f755966a740 1 -- 172.21.15.70:0/508450384 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564336456a40 con 0x564336566e70 2024-04-01T01:36:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.168+0000 7f754ffff640 1 --2- 172.21.15.70:0/508450384 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x564336566e70 0x564336569290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.168+0000 7f752ffff640 10 client.16500 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:36:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.168+0000 7f754ffff640 1 --2- 172.21.15.70:0/508450384 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x564336566e70 0x564336569290 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bbebe2bf873a57ce server_cookie=72608bda777afa3a in_seq=0 out_seq=0 2024-04-01T01:36:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.168+0000 7f752ffff640 1 -- 172.21.15.70:0/508450384 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564336456a40 con 0x564336566e70 2024-04-01T01:36:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.168+0000 7f752ffff640 10 client.16500 handle_command_reply: tid=0 2024-04-01T01:36:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.168+0000 7f755966a740 1 client.16500 shutdown 2024-04-01T01:36:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.168+0000 7f755966a740 1 -- 172.21.15.70:0/508450384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364ea490 msgr2=0x5643364f9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.168+0000 7f755966a740 1 --2- 172.21.15.70:0/508450384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364ea490 0x5643364f9270 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x5643364eb6e0 tx=0x7f754409b160 comp rx=0 tx=0).stop 2024-04-01T01:36:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.168+0000 7f755966a740 1 -- 172.21.15.70:0/508450384 shutdown_connections 2024-04-01T01:36:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.168+0000 7f755966a740 1 --2- 172.21.15.70:0/508450384 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x564336566e70 0x564336569290 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.168+0000 7f755966a740 1 --2- 172.21.15.70:0/508450384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643364efff0 0x5643364fa7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.168+0000 7f752ffff640 0 client.16500 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:36:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.168+0000 7f755966a740 1 --2- 172.21.15.70:0/508450384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643364eae80 0x5643364f97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.168+0000 7f755966a740 1 --2- 172.21.15.70:0/508450384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643364ea490 0x5643364f9270 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.168+0000 7f755966a740 1 -- 172.21.15.70:0/508450384 >> 172.21.15.70:0/508450384 conn(0x5643364e5a00 msgr2=0x5643364ec340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.169+0000 7f755966a740 1 -- 172.21.15.70:0/508450384 shutdown_connections 2024-04-01T01:36:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.169+0000 7f755966a740 1 -- 172.21.15.70:0/508450384 wait complete. 2024-04-01T01:36:30.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.169+0000 7f755966a740 20 client.16500 trim_cache size 0 max 16384 2024-04-01T01:36:30.175 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:36:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.169+0000 7f7556a8d640 1 -- 172.21.15.70:0/1990136943 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f75400a2520 msgr2=0x7f75400a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.169+0000 7f7556a8d640 1 --2- 172.21.15.70:0/1990136943 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f75400a2520 0x7f75400a49e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f753c02c180 tx=0x7f753c034980 comp rx=0 tx=0).stop 2024-04-01T01:36:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.169+0000 7f7556a8d640 1 -- 172.21.15.70:0/1990136943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7550100650 msgr2=0x7f7550110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.169+0000 7f7556a8d640 1 --2- 172.21.15.70:0/1990136943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7550100650 0x7f7550110030 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f754400e9c0 tx=0x7f754400ee80 comp rx=0 tx=0).stop 2024-04-01T01:36:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.169+0000 7f7556a8d640 1 -- 172.21.15.70:0/1990136943 shutdown_connections 2024-04-01T01:36:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.169+0000 7f7556a8d640 1 --2- 172.21.15.70:0/1990136943 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f75400a2520 0x7f75400a49e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.169+0000 7f7556a8d640 1 --2- 172.21.15.70:0/1990136943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7550102200 0x7f755010cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.169+0000 7f7556a8d640 1 --2- 172.21.15.70:0/1990136943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7550101840 0x7f7550110570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.169+0000 7f7556a8d640 1 --2- 172.21.15.70:0/1990136943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7550100650 0x7f7550110030 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.169+0000 7f7556a8d640 1 -- 172.21.15.70:0/1990136943 >> 172.21.15.70:0/1990136943 conn(0x7f75500fbbe0 msgr2=0x7f75500fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.169+0000 7f7556a8d640 1 -- 172.21.15.70:0/1990136943 shutdown_connections 2024-04-01T01:36:30.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.169+0000 7f7556a8d640 1 -- 172.21.15.70:0/1990136943 wait complete. 2024-04-01T01:36:30.179 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:36:30.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.321+0000 7f291b6f4640 1 -- 172.21.15.70:0/3724114086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2914109920 msgr2=0x7f29141101b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.321+0000 7f291b6f4640 1 --2- 172.21.15.70:0/3724114086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2914109920 0x7f29141101b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2910009900 tx=0x7f291002bc30 comp rx=0 tx=0).stop 2024-04-01T01:36:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.321+0000 7f291b6f4640 1 -- 172.21.15.70:0/3724114086 shutdown_connections 2024-04-01T01:36:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.321+0000 7f291b6f4640 1 --2- 172.21.15.70:0/3724114086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2914109920 0x7f29141101b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.321+0000 7f291b6f4640 1 --2- 172.21.15.70:0/3724114086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2914108f60 0x7f29141093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.321+0000 7f291b6f4640 1 --2- 172.21.15.70:0/3724114086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2914107d70 0x7f2914108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.321+0000 7f291b6f4640 1 -- 172.21.15.70:0/3724114086 >> 172.21.15.70:0/3724114086 conn(0x7f2914076d10 msgr2=0x7f2914079150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.321+0000 7f291b6f4640 1 -- 172.21.15.70:0/3724114086 shutdown_connections 2024-04-01T01:36:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.321+0000 7f291b6f4640 1 -- 172.21.15.70:0/3724114086 wait complete. 2024-04-01T01:36:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.322+0000 7f291b6f4640 1 Processor -- start 2024-04-01T01:36:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.322+0000 7f291b6f4640 1 -- start start 2024-04-01T01:36:30.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.322+0000 7f291b6f4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2914107d70 0x7f291419df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.322+0000 7f291b6f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2914108f60 0x7f291419e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.322+0000 7f291b6f4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2914109920 0x7f29141a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.322+0000 7f291b6f4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f29141138d0 con 0x7f2914108f60 2024-04-01T01:36:30.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.322+0000 7f291b6f4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2914113750 con 0x7f2914109920 2024-04-01T01:36:30.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.322+0000 7f291b6f4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2914113a50 con 0x7f2914107d70 2024-04-01T01:36:30.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.322+0000 7f2919490640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2914107d70 0x7f291419df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.322+0000 7f2919490640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2914107d70 0x7f291419df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37908/0 (socket says 172.21.15.70:37908) 2024-04-01T01:36:30.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.322+0000 7f2919490640 1 -- 172.21.15.70:0/890039688 learned_addr learned my addr 172.21.15.70:0/890039688 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.323+0000 7f2918c8f640 1 --2- 172.21.15.70:0/890039688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2914108f60 0x7f291419e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.323+0000 7f2919c91640 1 --2- 172.21.15.70:0/890039688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2914109920 0x7f29141a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.323+0000 7f2919490640 1 -- 172.21.15.70:0/890039688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2914109920 msgr2=0x7f29141a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.323+0000 7f2919490640 1 --2- 172.21.15.70:0/890039688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2914109920 0x7f29141a54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.323+0000 7f2919490640 1 -- 172.21.15.70:0/890039688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2914108f60 msgr2=0x7f291419e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.323+0000 7f2919490640 1 --2- 172.21.15.70:0/890039688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2914108f60 0x7f291419e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.323+0000 7f2919490640 1 -- 172.21.15.70:0/890039688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29141a5b30 con 0x7f2914107d70 2024-04-01T01:36:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.323+0000 7f2918c8f640 1 --2- 172.21.15.70:0/890039688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2914108f60 0x7f291419e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:36:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.324+0000 7f2919490640 1 --2- 172.21.15.70:0/890039688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2914107d70 0x7f291419df10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f290400d950 tx=0x7f290400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.324+0000 7f290a7fc640 1 -- 172.21.15.70:0/890039688 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904017020 con 0x7f2914107d70 2024-04-01T01:36:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.324+0000 7f291b6f4640 1 -- 172.21.15.70:0/890039688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29141a5e20 con 0x7f2914107d70 2024-04-01T01:36:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.324+0000 7f291b6f4640 1 -- 172.21.15.70:0/890039688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29141a62e0 con 0x7f2914107d70 2024-04-01T01:36:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.324+0000 7f290a7fc640 1 -- 172.21.15.70:0/890039688 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2904009c30 con 0x7f2914107d70 2024-04-01T01:36:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.324+0000 7f290a7fc640 1 -- 172.21.15.70:0/890039688 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904005220 con 0x7f2914107d70 2024-04-01T01:36:30.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.327+0000 7f291e3b4740 1 Processor -- start 2024-04-01T01:36:30.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.327+0000 7f291e3b4740 1 -- start start 2024-04-01T01:36:30.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.328+0000 7f291e3b4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038490 0x55aeea0388b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.328+0000 7f291e3b4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038e80 0x55aeea03d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.328+0000 7f291e3b4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea03dff0 0x55aeea040480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.328+0000 7f291e3b4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55aee9f9d1b0 con 0x55aeea038490 2024-04-01T01:36:30.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.328+0000 7f291e3b4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55aee9f45c00 con 0x55aeea038e80 2024-04-01T01:36:30.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.328+0000 7f291e3b4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55aee9f99250 con 0x55aeea03dff0 2024-04-01T01:36:30.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.328+0000 7f290a7fc640 1 -- 172.21.15.70:0/890039688 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f290401a020 con 0x7f2914107d70 2024-04-01T01:36:30.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.328+0000 7f2919c91640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038490 0x55aeea0388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.328+0000 7f2919c91640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038490 0x55aeea0388b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59806/0 (socket says 172.21.15.70:59806) 2024-04-01T01:36:30.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.328+0000 7f2919c91640 1 -- 172.21.15.70:0/2024319586 learned_addr learned my addr 172.21.15.70:0/2024319586 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.328+0000 7f2918c8f640 1 --2- 172.21.15.70:0/2024319586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea03dff0 0x55aeea040480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.329+0000 7f2918c8f640 1 -- 172.21.15.70:0/2024319586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038e80 msgr2=0x55aeea03d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.329+0000 7f290a7fc640 1 --2- 172.21.15.70:0/890039688 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f29000a25a0 0x7f29000a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.329+0000 7f290a7fc640 1 -- 172.21.15.70:0/890039688 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f29040ce6e0 con 0x7f2914107d70 2024-04-01T01:36:30.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.330+0000 7f2919490640 1 --2- 172.21.15.70:0/2024319586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038e80 0x55aeea03d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.330+0000 7f2918c8f640 1 --2- 172.21.15.70:0/2024319586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038e80 0x55aeea03d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.330+0000 7f2918c8f640 1 -- 172.21.15.70:0/2024319586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038490 msgr2=0x55aeea0388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.330+0000 7f2918c8f640 1 --2- 172.21.15.70:0/2024319586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038490 0x55aeea0388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.330+0000 7f2918c8f640 1 -- 172.21.15.70:0/2024319586 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aee9f653d0 con 0x55aeea03dff0 2024-04-01T01:36:30.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.330+0000 7f2919c91640 1 --2- 172.21.15.70:0/2024319586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038490 0x55aeea0388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:36:30.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.330+0000 7f2919490640 1 --2- 172.21.15.70:0/2024319586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038e80 0x55aeea03d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:36:30.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.330+0000 7f2918c8f640 1 --2- 172.21.15.70:0/2024319586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea03dff0 0x55aeea040480 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f291419f420 tx=0x7f290c011c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64d0e71a8aa70cd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.331+0000 7f2919c91640 1 --2- 172.21.15.70:0/890039688 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f29000a25a0 0x7f29000a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.331+0000 7f28ff7fe640 1 -- 172.21.15.70:0/2024319586 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290c011e60 con 0x55aeea03dff0 2024-04-01T01:36:30.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.331+0000 7f28ff7fe640 1 -- 172.21.15.70:0/2024319586 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f290c00fe20 con 0x55aeea03dff0 2024-04-01T01:36:30.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.331+0000 7f28ff7fe640 1 -- 172.21.15.70:0/2024319586 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290c004790 con 0x55aeea03dff0 2024-04-01T01:36:30.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.332+0000 7f291e3b4740 1 -- 172.21.15.70:0/2024319586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea03dff0 msgr2=0x55aeea040480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.332+0000 7f291e3b4740 1 --2- 172.21.15.70:0/2024319586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea03dff0 0x55aeea040480 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f291419f420 tx=0x7f290c011c70 comp rx=0 tx=0).stop 2024-04-01T01:36:30.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.333+0000 7f291e3b4740 1 -- 172.21.15.70:0/2024319586 shutdown_connections 2024-04-01T01:36:30.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.333+0000 7f291e3b4740 1 --2- 172.21.15.70:0/2024319586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea03dff0 0x55aeea040480 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.333+0000 7f291e3b4740 1 --2- 172.21.15.70:0/2024319586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038e80 0x55aeea03d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.333+0000 7f291e3b4740 1 --2- 172.21.15.70:0/2024319586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038490 0x55aeea0388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.333+0000 7f291e3b4740 1 -- 172.21.15.70:0/2024319586 >> 172.21.15.70:0/2024319586 conn(0x55aeea033a00 msgr2=0x55aeea033e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.333+0000 7f291e3b4740 1 -- 172.21.15.70:0/2024319586 shutdown_connections 2024-04-01T01:36:30.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.333+0000 7f2919c91640 1 --2- 172.21.15.70:0/890039688 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f29000a25a0 0x7f29000a4a60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f291002c840 tx=0x7f2910009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.334+0000 7f291e3b4740 1 -- 172.21.15.70:0/2024319586 wait complete. 2024-04-01T01:36:30.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.334+0000 7f291e3b4740 1 -- start start 2024-04-01T01:36:30.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.334+0000 7f291e3b4740 1 Processor -- start 2024-04-01T01:36:30.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.335+0000 7f291e3b4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea038490 0x55aeea0427f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.335+0000 7f291e3b4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 0x55aeea042d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.335+0000 7f291e3b4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea03dff0 0x55aeea049d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.335+0000 7f291e3b4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55aeea040fa0 con 0x55aeea038e80 2024-04-01T01:36:30.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.335+0000 7f291e3b4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55aee9f3b470 con 0x55aeea03dff0 2024-04-01T01:36:30.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.335+0000 7f291e3b4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55aeea041120 con 0x55aeea038490 2024-04-01T01:36:30.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.335+0000 7f291e3b4740 10 client.0 _ll_register_callbacks cb 0x55aee9f58cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:30.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.335+0000 7f2919490640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea038490 0x55aeea0427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.335+0000 7f2918c8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 0x55aeea042d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.335+0000 7f2919c91640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea03dff0 0x55aeea049d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.335+0000 7f2918c8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 0x55aeea042d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59818/0 (socket says 172.21.15.70:59818) 2024-04-01T01:36:30.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.335+0000 7f2919c91640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea03dff0 0x55aeea049d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35516/0 (socket says 172.21.15.70:35516) 2024-04-01T01:36:30.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.335+0000 7f2918c8f640 1 -- 172.21.15.70:0/518605890 learned_addr learned my addr 172.21.15.70:0/518605890 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.336+0000 7f2919490640 1 -- 172.21.15.70:0/518605890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea03dff0 msgr2=0x55aeea049d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.336+0000 7f2919490640 1 --2- 172.21.15.70:0/518605890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea03dff0 0x55aeea049d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.336+0000 7f2919490640 1 -- 172.21.15.70:0/518605890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 msgr2=0x55aeea042d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.336+0000 7f2919490640 1 --2- 172.21.15.70:0/518605890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 0x55aeea042d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.336+0000 7f2919490640 1 -- 172.21.15.70:0/518605890 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aee9cb13f0 con 0x55aeea038490 2024-04-01T01:36:30.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.336+0000 7f2918c8f640 1 --2- 172.21.15.70:0/518605890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 0x55aeea042d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:36:30.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.336+0000 7f2919490640 1 --2- 172.21.15.70:0/518605890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea038490 0x55aeea0427f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2904010fd0 tx=0x7f290409a7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb21c1ec5e00e101 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.336+0000 7f28fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:36:30.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.336+0000 7f28fcff9640 1 -- 172.21.15.70:0/518605890 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29040b8060 con 0x55aeea038490 2024-04-01T01:36:30.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.336+0000 7f28fcff9640 1 -- 172.21.15.70:0/518605890 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f29040053c0 con 0x55aeea038490 2024-04-01T01:36:30.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.336+0000 7f291e3b4740 1 -- 172.21.15.70:0/518605890 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f290c00fe20 con 0x55aeea038490 2024-04-01T01:36:30.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.336+0000 7f28fcff9640 1 -- 172.21.15.70:0/518605890 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f29040b2030 con 0x55aeea038490 2024-04-01T01:36:30.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.337+0000 7f291e3b4740 10 client.26248 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:30.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.337+0000 7f291e3b4740 1 -- 172.21.15.70:0/518605890 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55aee9d7b7d0 con 0x55aeea038490 2024-04-01T01:36:30.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.337+0000 7f28fcff9640 1 -- 172.21.15.70:0/518605890 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f290c00fe20 con 0x55aeea038490 2024-04-01T01:36:30.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.337+0000 7f28fcff9640 1 -- 172.21.15.70:0/518605890 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2904024060 con 0x55aeea038490 2024-04-01T01:36:30.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.337+0000 7f291e3b4740 10 client.26248 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:30.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.337+0000 7f291e3b4740 10 client.26248 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:36:30.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.337+0000 7f291e3b4740 20 client.26248 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:30.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.337+0000 7f291e3b4740 1 --2- 172.21.15.70:0/518605890 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55aeea04e450 0x55aeea0508f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.337+0000 7f291e3b4740 4 client.26248 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:36:30.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.337+0000 7f291e3b4740 1 -- 172.21.15.70:0/518605890 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aee9faa6f0 con 0x55aeea04e450 2024-04-01T01:36:30.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.337+0000 7f2918c8f640 1 --2- 172.21.15.70:0/518605890 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55aeea04e450 0x55aeea0508f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.338+0000 7f2918c8f640 1 --2- 172.21.15.70:0/518605890 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55aeea04e450 0x55aeea0508f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d4c4594bfa771eb4 server_cookie=feba86291495dbec in_seq=0 out_seq=0 2024-04-01T01:36:30.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.338+0000 7f28fcff9640 10 client.26248 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:36:30.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.339+0000 7f28fcff9640 1 -- 172.21.15.70:0/518605890 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55aee9faa6f0 con 0x55aeea04e450 2024-04-01T01:36:30.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.339+0000 7f28fcff9640 10 client.26248 handle_command_reply: tid=0 2024-04-01T01:36:30.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.339+0000 7f291e3b4740 1 client.26248 shutdown 2024-04-01T01:36:30.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.339+0000 7f291e3b4740 1 -- 172.21.15.70:0/518605890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea038490 msgr2=0x55aeea0427f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.339+0000 7f291e3b4740 1 --2- 172.21.15.70:0/518605890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea038490 0x55aeea0427f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2904010fd0 tx=0x7f290409a7e0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.339+0000 7f291e3b4740 1 -- 172.21.15.70:0/518605890 shutdown_connections 2024-04-01T01:36:30.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.339+0000 7f291e3b4740 1 --2- 172.21.15.70:0/518605890 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55aeea04e450 0x55aeea0508f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.339+0000 7f291e3b4740 1 --2- 172.21.15.70:0/518605890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea03dff0 0x55aeea049d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.339+0000 7f291e3b4740 1 --2- 172.21.15.70:0/518605890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 0x55aeea042d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.339+0000 7f291e3b4740 1 --2- 172.21.15.70:0/518605890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea038490 0x55aeea0427f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.339+0000 7f291e3b4740 1 -- 172.21.15.70:0/518605890 >> 172.21.15.70:0/518605890 conn(0x55aeea033a00 msgr2=0x55aeea032d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.339+0000 7f28fcff9640 0 client.26248 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:36:30.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.340+0000 7f291e3b4740 1 -- 172.21.15.70:0/518605890 shutdown_connections 2024-04-01T01:36:30.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.340+0000 7f291e3b4740 1 -- 172.21.15.70:0/518605890 wait complete. 2024-04-01T01:36:30.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.340+0000 7f291e3b4740 20 client.26248 trim_cache size 0 max 16384 2024-04-01T01:36:30.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.364+0000 7f291e3b4740 1 Processor -- start 2024-04-01T01:36:30.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.364+0000 7f291e3b4740 1 -- start start 2024-04-01T01:36:30.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.364+0000 7f291e3b4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038490 0x55aeea0561c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.364+0000 7f291e3b4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 0x55aeea036c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.364+0000 7f291e3b4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea03dff0 0x55aeea0371a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.364+0000 7f291e3b4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f28f800fc30 con 0x55aeea038e80 2024-04-01T01:36:30.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.365+0000 7f2918c8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 0x55aeea036c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.365+0000 7f2919490640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038490 0x55aeea0561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.365+0000 7f2918c8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 0x55aeea036c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59826/0 (socket says 172.21.15.70:59826) 2024-04-01T01:36:30.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.365+0000 7f2918c8f640 1 -- 172.21.15.70:0/154747735 learned_addr learned my addr 172.21.15.70:0/154747735 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.364+0000 7f291e3b4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f28f8011770 con 0x55aeea038490 2024-04-01T01:36:30.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.365+0000 7f291e3b4740 1 -- 172.21.15.70:0/154747735 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f28f8010300 con 0x55aeea03dff0 2024-04-01T01:36:30.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.365+0000 7f2919c91640 1 --2- 172.21.15.70:0/154747735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea03dff0 0x55aeea0371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.365+0000 7f2919490640 1 -- 172.21.15.70:0/154747735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea03dff0 msgr2=0x55aeea0371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.365+0000 7f2919490640 1 --2- 172.21.15.70:0/154747735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea03dff0 0x55aeea0371a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.365+0000 7f2919490640 1 -- 172.21.15.70:0/154747735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 msgr2=0x55aeea036c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.365+0000 7f2919490640 1 --2- 172.21.15.70:0/154747735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 0x55aeea036c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.365+0000 7f2919490640 1 -- 172.21.15.70:0/154747735 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aeea039260 con 0x55aeea038490 2024-04-01T01:36:30.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.367+0000 7f2919490640 1 --2- 172.21.15.70:0/154747735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038490 0x55aeea0561c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2904021000 tx=0x7f2904099280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7422b8bc56939b5c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.367+0000 7f28feffd640 1 -- 172.21.15.70:0/154747735 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29040b8070 con 0x55aeea038490 2024-04-01T01:36:30.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.367+0000 7f28feffd640 1 -- 172.21.15.70:0/154747735 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29040973e0 con 0x55aeea038490 2024-04-01T01:36:30.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.367+0000 7f28feffd640 1 -- 172.21.15.70:0/154747735 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904096ac0 con 0x55aeea038490 2024-04-01T01:36:30.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.369+0000 7f291e3b4740 1 -- 172.21.15.70:0/154747735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038490 msgr2=0x55aeea0561c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.370+0000 7f291e3b4740 1 --2- 172.21.15.70:0/154747735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038490 0x55aeea0561c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2904021000 tx=0x7f2904099280 comp rx=0 tx=0).stop 2024-04-01T01:36:30.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.370+0000 7f291e3b4740 1 -- 172.21.15.70:0/154747735 shutdown_connections 2024-04-01T01:36:30.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.370+0000 7f291e3b4740 1 --2- 172.21.15.70:0/154747735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea03dff0 0x55aeea0371a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.370+0000 7f291e3b4740 1 --2- 172.21.15.70:0/154747735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 0x55aeea036c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.370+0000 7f291e3b4740 1 --2- 172.21.15.70:0/154747735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038490 0x55aeea0561c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.370+0000 7f291e3b4740 1 -- 172.21.15.70:0/154747735 >> 172.21.15.70:0/154747735 conn(0x55aeea033a00 msgr2=0x55aeea02ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.370+0000 7f291e3b4740 1 -- 172.21.15.70:0/154747735 shutdown_connections 2024-04-01T01:36:30.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.371+0000 7f291e3b4740 1 -- 172.21.15.70:0/154747735 wait complete. 2024-04-01T01:36:30.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.371+0000 7f291e3b4740 1 -- start start 2024-04-01T01:36:30.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.371+0000 7f291e3b4740 1 Processor -- start 2024-04-01T01:36:30.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.372+0000 7f291e3b4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038490 0x55aeea044820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.372+0000 7f291e3b4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 0x55aeea044d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.372+0000 7f291e3b4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea03dff0 0x55aeea0c37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.372+0000 7f291e3b4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f28f80110a0 con 0x55aeea038e80 2024-04-01T01:36:30.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.372+0000 7f291e3b4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f28f80109d0 con 0x55aeea038490 2024-04-01T01:36:30.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.372+0000 7f291e3b4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f28f800dc10 con 0x55aeea03dff0 2024-04-01T01:36:30.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.372+0000 7f2919490640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038490 0x55aeea044820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.372+0000 7f291e3b4740 10 client.0 _ll_register_callbacks cb 0x7f2904009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:30.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.372+0000 7f2919490640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038490 0x55aeea044820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35534/0 (socket says 172.21.15.70:35534) 2024-04-01T01:36:30.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.372+0000 7f2919490640 1 -- 172.21.15.70:0/1305464471 learned_addr learned my addr 172.21.15.70:0/1305464471 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.373+0000 7f2919c91640 1 --2- 172.21.15.70:0/1305464471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea03dff0 0x55aeea0c37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.373+0000 7f2918c8f640 1 --2- 172.21.15.70:0/1305464471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 0x55aeea044d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.373+0000 7f2919490640 1 -- 172.21.15.70:0/1305464471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea03dff0 msgr2=0x55aeea0c37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.373+0000 7f2919490640 1 --2- 172.21.15.70:0/1305464471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea03dff0 0x55aeea0c37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.373+0000 7f2919490640 1 -- 172.21.15.70:0/1305464471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 msgr2=0x55aeea044d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.374+0000 7f2919490640 1 --2- 172.21.15.70:0/1305464471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 0x55aeea044d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.374+0000 7f2919490640 1 -- 172.21.15.70:0/1305464471 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aeea045400 con 0x55aeea038490 2024-04-01T01:36:30.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.374+0000 7f2918c8f640 1 --2- 172.21.15.70:0/1305464471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 0x55aeea044d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:36:30.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.375+0000 7f2919490640 1 --2- 172.21.15.70:0/1305464471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038490 0x55aeea044820 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2904099020 tx=0x7f2904097590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e5acbbe3c83ce8b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.375+0000 7f28fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:36:30.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.375+0000 7f28fcff9640 1 -- 172.21.15.70:0/1305464471 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29040b8040 con 0x55aeea038490 2024-04-01T01:36:30.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.375+0000 7f28fcff9640 1 -- 172.21.15.70:0/1305464471 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f2904096ac0 con 0x55aeea038490 2024-04-01T01:36:30.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.375+0000 7f291e3b4740 1 -- 172.21.15.70:0/1305464471 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29040973e0 con 0x55aeea038490 2024-04-01T01:36:30.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.375+0000 7f28fcff9640 1 -- 172.21.15.70:0/1305464471 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f29040ce4e0 con 0x55aeea038490 2024-04-01T01:36:30.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.375+0000 7f291e3b4740 10 client.26348 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:30.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.375+0000 7f291e3b4740 1 -- 172.21.15.70:0/1305464471 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55aeea0c4e60 con 0x55aeea038490 2024-04-01T01:36:30.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.376+0000 7f28fcff9640 1 -- 172.21.15.70:0/1305464471 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29040973e0 con 0x55aeea038490 2024-04-01T01:36:30.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.376+0000 7f28fcff9640 1 -- 172.21.15.70:0/1305464471 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2904085a80 con 0x55aeea038490 2024-04-01T01:36:30.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.376+0000 7f291e3b4740 10 client.26348 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:30.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.376+0000 7f291e3b4740 10 client.26348 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:36:30.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.376+0000 7f291e3b4740 20 client.26348 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:30.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.376+0000 7f291e3b4740 1 --2- 172.21.15.70:0/1305464471 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55aeea0c56d0 0x55aeea0c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.376+0000 7f291e3b4740 4 client.26348 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:36:30.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.376+0000 7f291e3b4740 1 -- 172.21.15.70:0/1305464471 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55aee9fa4a40 con 0x55aeea0c56d0 2024-04-01T01:36:30.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.376+0000 7f2918c8f640 1 --2- 172.21.15.70:0/1305464471 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55aeea0c56d0 0x55aeea0c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.377+0000 7f2918c8f640 1 --2- 172.21.15.70:0/1305464471 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55aeea0c56d0 0x55aeea0c7af0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=adda1b3fdd006005 server_cookie=a55bd215c911c70 in_seq=0 out_seq=0 2024-04-01T01:36:30.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.377+0000 7f28fcff9640 10 client.26348 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:36:30.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.377+0000 7f28fcff9640 1 -- 172.21.15.70:0/1305464471 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55aee9fa4a40 con 0x55aeea0c56d0 2024-04-01T01:36:30.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.377+0000 7f28fcff9640 10 client.26348 handle_command_reply: tid=0 2024-04-01T01:36:30.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.377+0000 7f291e3b4740 1 client.26348 shutdown 2024-04-01T01:36:30.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.378+0000 7f291e3b4740 1 -- 172.21.15.70:0/1305464471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038490 msgr2=0x55aeea044820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.378+0000 7f291e3b4740 1 --2- 172.21.15.70:0/1305464471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038490 0x55aeea044820 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2904099020 tx=0x7f2904097590 comp rx=0 tx=0).stop 2024-04-01T01:36:30.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.378+0000 7f291e3b4740 1 -- 172.21.15.70:0/1305464471 shutdown_connections 2024-04-01T01:36:30.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.378+0000 7f291e3b4740 1 --2- 172.21.15.70:0/1305464471 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55aeea0c56d0 0x55aeea0c7af0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.378+0000 7f28fcff9640 0 client.26348 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:36:30.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.378+0000 7f291e3b4740 1 --2- 172.21.15.70:0/1305464471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aeea03dff0 0x55aeea0c37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.378+0000 7f291e3b4740 1 --2- 172.21.15.70:0/1305464471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aeea038e80 0x55aeea044d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.378+0000 7f291e3b4740 1 --2- 172.21.15.70:0/1305464471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aeea038490 0x55aeea044820 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.378+0000 7f291e3b4740 1 -- 172.21.15.70:0/1305464471 >> 172.21.15.70:0/1305464471 conn(0x55aeea033a00 msgr2=0x55aeea036c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.378+0000 7f291e3b4740 1 -- 172.21.15.70:0/1305464471 shutdown_connections 2024-04-01T01:36:30.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.378+0000 7f291e3b4740 1 -- 172.21.15.70:0/1305464471 wait complete. 2024-04-01T01:36:30.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.379+0000 7f291e3b4740 20 client.26348 trim_cache size 0 max 16384 2024-04-01T01:36:30.379 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:36:30.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.380+0000 7f291b6f4640 1 -- 172.21.15.70:0/890039688 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f29000a25a0 msgr2=0x7f29000a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.380+0000 7f291b6f4640 1 --2- 172.21.15.70:0/890039688 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f29000a25a0 0x7f29000a4a60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f291002c840 tx=0x7f2910009260 comp rx=0 tx=0).stop 2024-04-01T01:36:30.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.380+0000 7f291b6f4640 1 -- 172.21.15.70:0/890039688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2914107d70 msgr2=0x7f291419df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.380+0000 7f291b6f4640 1 --2- 172.21.15.70:0/890039688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2914107d70 0x7f291419df10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f290400d950 tx=0x7f290400de10 comp rx=0 tx=0).stop 2024-04-01T01:36:30.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.380+0000 7f291b6f4640 1 -- 172.21.15.70:0/890039688 shutdown_connections 2024-04-01T01:36:30.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.380+0000 7f291b6f4640 1 --2- 172.21.15.70:0/890039688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2914109920 0x7f29141a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.380+0000 7f291b6f4640 1 --2- 172.21.15.70:0/890039688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2914108f60 0x7f291419e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.380+0000 7f291b6f4640 1 --2- 172.21.15.70:0/890039688 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f29000a25a0 0x7f29000a4a60 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.380+0000 7f291b6f4640 1 --2- 172.21.15.70:0/890039688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2914107d70 0x7f291419df10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.380+0000 7f291b6f4640 1 -- 172.21.15.70:0/890039688 >> 172.21.15.70:0/890039688 conn(0x7f2914076d10 msgr2=0x7f291410c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.380+0000 7f291b6f4640 1 -- 172.21.15.70:0/890039688 shutdown_connections 2024-04-01T01:36:30.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.380+0000 7f291b6f4640 1 -- 172.21.15.70:0/890039688 wait complete. 2024-04-01T01:36:30.398 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:36:30.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.580+0000 7f9d0b230640 1 -- 172.21.15.70:0/14895210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cfc0990f0 msgr2=0x7f9cfc099570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.580+0000 7f9d0b230640 1 --2- 172.21.15.70:0/14895210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cfc0990f0 0x7f9cfc099570 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0009900 tx=0x7f9cf002bc30 comp rx=0 tx=0).stop 2024-04-01T01:36:30.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.581+0000 7f9d0b230640 1 -- 172.21.15.70:0/14895210 shutdown_connections 2024-04-01T01:36:30.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.581+0000 7f9d0b230640 1 --2- 172.21.15.70:0/14895210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cfc099ab0 0x7f9cfc0a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.581+0000 7f9d0b230640 1 --2- 172.21.15.70:0/14895210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cfc0990f0 0x7f9cfc099570 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.581+0000 7f9d0b230640 1 --2- 172.21.15.70:0/14895210 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cfc097f20 0x7f9cfc098360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.581+0000 7f9d0b230640 1 -- 172.21.15.70:0/14895210 >> 172.21.15.70:0/14895210 conn(0x7f9cfc0934e0 msgr2=0x7f9cfc095920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.581+0000 7f9d0b230640 1 -- 172.21.15.70:0/14895210 shutdown_connections 2024-04-01T01:36:30.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.582+0000 7f9d0b230640 1 -- 172.21.15.70:0/14895210 wait complete. 2024-04-01T01:36:30.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.583+0000 7f9d0b230640 1 Processor -- start 2024-04-01T01:36:30.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.583+0000 7f9d0b230640 1 -- start start 2024-04-01T01:36:30.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.583+0000 7f9d0b230640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cfc097f20 0x7f9cfc12e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.583+0000 7f9d0b230640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cfc0990f0 0x7f9cfc12e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.583+0000 7f9d0b230640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cfc099ab0 0x7f9cfc135780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.583+0000 7f9d0b230640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9cfc0a3ad0 con 0x7f9cfc097f20 2024-04-01T01:36:30.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.583+0000 7f9d0b230640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9cfc0a3950 con 0x7f9cfc099ab0 2024-04-01T01:36:30.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.583+0000 7f9d0b230640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9cfc0a3c50 con 0x7f9cfc0990f0 2024-04-01T01:36:30.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.583+0000 7f9d097cd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cfc099ab0 0x7f9cfc135780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.583+0000 7f9d097cd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cfc099ab0 0x7f9cfc135780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35560/0 (socket says 172.21.15.70:35560) 2024-04-01T01:36:30.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.583+0000 7f9d08fcc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cfc097f20 0x7f9cfc12e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.583+0000 7f9d097cd640 1 -- 172.21.15.70:0/2947932782 learned_addr learned my addr 172.21.15.70:0/2947932782 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.583+0000 7f9d03fff640 1 --2- 172.21.15.70:0/2947932782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cfc0990f0 0x7f9cfc12e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.584+0000 7f9d097cd640 1 -- 172.21.15.70:0/2947932782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cfc0990f0 msgr2=0x7f9cfc12e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.584+0000 7f9d097cd640 1 --2- 172.21.15.70:0/2947932782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cfc0990f0 0x7f9cfc12e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.584+0000 7f9d097cd640 1 -- 172.21.15.70:0/2947932782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cfc097f20 msgr2=0x7f9cfc12e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.584+0000 7f9d097cd640 1 --2- 172.21.15.70:0/2947932782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cfc097f20 0x7f9cfc12e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.584+0000 7f9d097cd640 1 -- 172.21.15.70:0/2947932782 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cfc135df0 con 0x7f9cfc099ab0 2024-04-01T01:36:30.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.584+0000 7f9d097cd640 1 --2- 172.21.15.70:0/2947932782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cfc099ab0 0x7f9cfc135780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800d750 tx=0x7f9cf800db50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.585+0000 7f9d01ffb640 1 -- 172.21.15.70:0/2947932782 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8017070 con 0x7f9cfc099ab0 2024-04-01T01:36:30.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.585+0000 7f9d01ffb640 1 -- 172.21.15.70:0/2947932782 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9cf80097c0 con 0x7f9cfc099ab0 2024-04-01T01:36:30.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.585+0000 7f9d01ffb640 1 -- 172.21.15.70:0/2947932782 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8004b80 con 0x7f9cfc099ab0 2024-04-01T01:36:30.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.585+0000 7f9d0b230640 1 -- 172.21.15.70:0/2947932782 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cfc1360e0 con 0x7f9cfc099ab0 2024-04-01T01:36:30.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.586+0000 7f9d0b230640 1 -- 172.21.15.70:0/2947932782 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cfc1365b0 con 0x7f9cfc099ab0 2024-04-01T01:36:30.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.589+0000 7f9d01ffb640 1 -- 172.21.15.70:0/2947932782 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f9cf8004d20 con 0x7f9cfc099ab0 2024-04-01T01:36:30.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.590+0000 7f9d0de0c740 1 Processor -- start 2024-04-01T01:36:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.590+0000 7f9d0de0c740 1 -- start start 2024-04-01T01:36:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.590+0000 7f9d01ffb640 1 --2- 172.21.15.70:0/2947932782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9cf40a25a0 0x7f9cf40a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.590+0000 7f9d0de0c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe56a0 0x5617f7fe5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.590+0000 7f9d0de0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7fe6090 0x5617f7ff33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.590+0000 7f9d0de0c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7ff3a10 0x5617f7ff5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.590+0000 7f9d0de0c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5617f7f4a1b0 con 0x5617f7fe6090 2024-04-01T01:36:30.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.590+0000 7f9d0de0c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5617f7ef2c00 con 0x5617f7ff3a10 2024-04-01T01:36:30.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.590+0000 7f9d0de0c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5617f7f46250 con 0x5617f7fe56a0 2024-04-01T01:36:30.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.590+0000 7f9d01ffb640 1 -- 172.21.15.70:0/2947932782 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9cf80ce430 con 0x7f9cfc099ab0 2024-04-01T01:36:30.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.590+0000 7f9d03fff640 1 --2- 172.21.15.70:0/2947932782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9cf40a25a0 0x7f9cf40a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.590+0000 7f9d08fcc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe56a0 0x5617f7fe5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.590+0000 7f9d097cd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7ff3a10 0x5617f7ff5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.590+0000 7f9d097cd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7ff3a10 0x5617f7ff5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35568/0 (socket says 172.21.15.70:35568) 2024-04-01T01:36:30.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.590+0000 7f9d097cd640 1 -- 172.21.15.70:0/2431109074 learned_addr learned my addr 172.21.15.70:0/2431109074 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.591+0000 7f9d03fff640 1 --2- 172.21.15.70:0/2431109074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7fe6090 0x5617f7ff33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.591+0000 7f9d097cd640 1 -- 172.21.15.70:0/2431109074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe56a0 msgr2=0x5617f7fe5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.591+0000 7f9d097cd640 1 --2- 172.21.15.70:0/2431109074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe56a0 0x5617f7fe5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.591+0000 7f9d097cd640 1 -- 172.21.15.70:0/2431109074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7fe6090 msgr2=0x5617f7ff33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.591+0000 7f9d097cd640 1 --2- 172.21.15.70:0/2431109074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7fe6090 0x5617f7ff33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.591+0000 7f9d097cd640 1 -- 172.21.15.70:0/2431109074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617f7f123d0 con 0x5617f7ff3a10 2024-04-01T01:36:30.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.591+0000 7f9d03fff640 1 --2- 172.21.15.70:0/2431109074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7fe6090 0x5617f7ff33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:36:30.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.591+0000 7f9d097cd640 1 --2- 172.21.15.70:0/2431109074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7ff3a10 0x5617f7ff5ea0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8096f20 tx=0x7f9cf8049890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=313666c41050b66f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.591+0000 7f9d03fff640 1 --2- 172.21.15.70:0/2947932782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9cf40a25a0 0x7f9cf40a4a60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0000c00 tx=0x7f9cf0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.592+0000 7f9ceeffd640 1 -- 172.21.15.70:0/2431109074 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8020070 con 0x5617f7ff3a10 2024-04-01T01:36:30.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.592+0000 7f9ceeffd640 1 -- 172.21.15.70:0/2431109074 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9cf809aac0 con 0x5617f7ff3a10 2024-04-01T01:36:30.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.592+0000 7f9ceeffd640 1 -- 172.21.15.70:0/2431109074 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8097130 con 0x5617f7ff3a10 2024-04-01T01:36:30.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.593+0000 7f9d0de0c740 1 -- 172.21.15.70:0/2431109074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7ff3a10 msgr2=0x5617f7ff5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.593+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/2431109074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7ff3a10 0x5617f7ff5ea0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8096f20 tx=0x7f9cf8049890 comp rx=0 tx=0).stop 2024-04-01T01:36:30.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.593+0000 7f9d0de0c740 1 -- 172.21.15.70:0/2431109074 shutdown_connections 2024-04-01T01:36:30.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.593+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/2431109074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7ff3a10 0x5617f7ff5ea0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.593+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/2431109074 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7fe6090 0x5617f7ff33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.593+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/2431109074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe56a0 0x5617f7fe5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.593+0000 7f9d0de0c740 1 -- 172.21.15.70:0/2431109074 >> 172.21.15.70:0/2431109074 conn(0x5617f7fe0c10 msgr2=0x5617f7fe1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.593+0000 7f9d0de0c740 1 -- 172.21.15.70:0/2431109074 shutdown_connections 2024-04-01T01:36:30.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.593+0000 7f9d0de0c740 1 -- 172.21.15.70:0/2431109074 wait complete. 2024-04-01T01:36:30.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.594+0000 7f9d0de0c740 1 -- start start 2024-04-01T01:36:30.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.594+0000 7f9d0de0c740 1 Processor -- start 2024-04-01T01:36:30.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.594+0000 7f9d0de0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7fe56a0 0x5617f7fecc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.594+0000 7f9d0de0c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe6090 0x5617f7fed1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.594+0000 7f9d0de0c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7ff3a10 0x5617f7ff2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.594+0000 7f9d0de0c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5617f7fe6d40 con 0x5617f7fe56a0 2024-04-01T01:36:30.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.594+0000 7f9d0de0c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5617f7ee8470 con 0x5617f7ff3a10 2024-04-01T01:36:30.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.594+0000 7f9d0de0c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5617f7fe6ec0 con 0x5617f7fe6090 2024-04-01T01:36:30.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.594+0000 7f9d03fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe6090 0x5617f7fed1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.594+0000 7f9d03fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe6090 0x5617f7fed1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37978/0 (socket says 172.21.15.70:37978) 2024-04-01T01:36:30.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.594+0000 7f9d03fff640 1 -- 172.21.15.70:0/1325871320 learned_addr learned my addr 172.21.15.70:0/1325871320 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.595+0000 7f9d097cd640 1 --2- 172.21.15.70:0/1325871320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7ff3a10 0x5617f7ff2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.594+0000 7f9d0de0c740 10 client.0 _ll_register_callbacks cb 0x5617f7f05cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:30.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.595+0000 7f9d08fcc640 1 --2- 172.21.15.70:0/1325871320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7fe56a0 0x5617f7fecc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.595+0000 7f9d097cd640 1 -- 172.21.15.70:0/1325871320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe6090 msgr2=0x5617f7fed1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.595+0000 7f9d097cd640 1 --2- 172.21.15.70:0/1325871320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe6090 0x5617f7fed1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.595+0000 7f9d097cd640 1 -- 172.21.15.70:0/1325871320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7fe56a0 msgr2=0x5617f7fecc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.595+0000 7f9d097cd640 1 --2- 172.21.15.70:0/1325871320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7fe56a0 0x5617f7fecc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.595+0000 7f9d097cd640 1 -- 172.21.15.70:0/1325871320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617f7c5e3f0 con 0x5617f7ff3a10 2024-04-01T01:36:30.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.595+0000 7f9d097cd640 1 --2- 172.21.15.70:0/1325871320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7ff3a10 0x5617f7ff2200 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8096f20 tx=0x7f9cf80ce350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b3880aa882e376c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.595+0000 7f9ce3fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:36:30.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.595+0000 7f9ce3fff640 1 -- 172.21.15.70:0/1325871320 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8020050 con 0x5617f7ff3a10 2024-04-01T01:36:30.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.596+0000 7f9ce3fff640 1 -- 172.21.15.70:0/1325871320 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9cf8097130 con 0x5617f7ff3a10 2024-04-01T01:36:30.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.596+0000 7f9d0de0c740 1 -- 172.21.15.70:0/1325871320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cf809aac0 con 0x5617f7ff3a10 2024-04-01T01:36:30.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.596+0000 7f9ce3fff640 1 -- 172.21.15.70:0/1325871320 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9cf8009970 con 0x5617f7ff3a10 2024-04-01T01:36:30.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.597+0000 7f9d0de0c740 10 client.26360 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:30.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.597+0000 7f9d0de0c740 1 -- 172.21.15.70:0/1325871320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5617f7d287d0 con 0x5617f7ff3a10 2024-04-01T01:36:30.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.597+0000 7f9ce3fff640 1 -- 172.21.15.70:0/1325871320 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9cf809aac0 con 0x5617f7ff3a10 2024-04-01T01:36:30.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.597+0000 7f9ce3fff640 1 -- 172.21.15.70:0/1325871320 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9cf8096810 con 0x5617f7ff3a10 2024-04-01T01:36:30.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.597+0000 7f9d0de0c740 10 client.26360 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:30.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.597+0000 7f9d0de0c740 10 client.26360 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:36:30.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.597+0000 7f9d0de0c740 20 client.26360 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:30.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.597+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/1325871320 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5617f7ffb130 0x5617f7ffd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.598+0000 7f9d0de0c740 4 client.26360 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:36:30.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.598+0000 7f9d0de0c740 1 -- 172.21.15.70:0/1325871320 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617f7f576f0 con 0x5617f7ffb130 2024-04-01T01:36:30.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.598+0000 7f9d08fcc640 1 --2- 172.21.15.70:0/1325871320 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5617f7ffb130 0x5617f7ffd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.598+0000 7f9d08fcc640 1 --2- 172.21.15.70:0/1325871320 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5617f7ffb130 0x5617f7ffd520 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5c13d8c7c5ac66dc server_cookie=c989ae0a35decc1b in_seq=0 out_seq=0 2024-04-01T01:36:30.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.599+0000 7f9ce3fff640 10 client.26360 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:36:30.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.600+0000 7f9ce3fff640 1 -- 172.21.15.70:0/1325871320 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5617f7f576f0 con 0x5617f7ffb130 2024-04-01T01:36:30.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.600+0000 7f9ce3fff640 10 client.26360 handle_command_reply: tid=0 2024-04-01T01:36:30.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.600+0000 7f9d0de0c740 1 client.26360 shutdown 2024-04-01T01:36:30.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.600+0000 7f9d0de0c740 1 -- 172.21.15.70:0/1325871320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7ff3a10 msgr2=0x5617f7ff2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.600+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/1325871320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7ff3a10 0x5617f7ff2200 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8096f20 tx=0x7f9cf80ce350 comp rx=0 tx=0).stop 2024-04-01T01:36:30.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.600+0000 7f9d0de0c740 1 -- 172.21.15.70:0/1325871320 shutdown_connections 2024-04-01T01:36:30.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.600+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/1325871320 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5617f7ffb130 0x5617f7ffd520 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.600+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/1325871320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7ff3a10 0x5617f7ff2200 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.600+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/1325871320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe6090 0x5617f7fed1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.600+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/1325871320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7fe56a0 0x5617f7fecc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.600+0000 7f9d0de0c740 1 -- 172.21.15.70:0/1325871320 >> 172.21.15.70:0/1325871320 conn(0x5617f7fe0c10 msgr2=0x5617f7fdff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.601+0000 7f9ce3fff640 0 client.26360 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:36:30.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.601+0000 7f9d0de0c740 1 -- 172.21.15.70:0/1325871320 shutdown_connections 2024-04-01T01:36:30.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.601+0000 7f9d0de0c740 1 -- 172.21.15.70:0/1325871320 wait complete. 2024-04-01T01:36:30.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.601+0000 7f9d0de0c740 20 client.26360 trim_cache size 0 max 16384 2024-04-01T01:36:30.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.626+0000 7f9d0de0c740 1 Processor -- start 2024-04-01T01:36:30.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.626+0000 7f9d0de0c740 1 -- start start 2024-04-01T01:36:30.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.627+0000 7f9d0de0c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7fe56a0 0x5617f7ff21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.627+0000 7f9d0de0c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe6090 0x5617f7ffd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.627+0000 7f9d08fcc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7fe56a0 0x5617f7ff21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.627+0000 7f9d08fcc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7fe56a0 0x5617f7ff21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35584/0 (socket says 172.21.15.70:35584) 2024-04-01T01:36:30.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.627+0000 7f9d03fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe6090 0x5617f7ffd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.627+0000 7f9d03fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe6090 0x5617f7ffd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37982/0 (socket says 172.21.15.70:37982) 2024-04-01T01:36:30.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.627+0000 7f9d03fff640 1 -- 172.21.15.70:0/265907375 learned_addr learned my addr 172.21.15.70:0/265907375 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.627+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/265907375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7ff3a10 0x5617f7ffd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.628+0000 7f9d0de0c740 1 -- 172.21.15.70:0/265907375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9cfc13c730 con 0x5617f7ff3a10 2024-04-01T01:36:30.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.628+0000 7f9d0de0c740 1 -- 172.21.15.70:0/265907375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9cfc13e270 con 0x5617f7fe56a0 2024-04-01T01:36:30.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.628+0000 7f9d097cd640 1 --2- 172.21.15.70:0/265907375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7ff3a10 0x5617f7ffd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.628+0000 7f9d0de0c740 1 -- 172.21.15.70:0/265907375 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9cfc13ce00 con 0x5617f7fe6090 2024-04-01T01:36:30.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.628+0000 7f9d097cd640 1 -- 172.21.15.70:0/265907375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe6090 msgr2=0x5617f7ffd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.628+0000 7f9d097cd640 1 --2- 172.21.15.70:0/265907375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe6090 0x5617f7ffd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.628+0000 7f9d097cd640 1 -- 172.21.15.70:0/265907375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7fe56a0 msgr2=0x5617f7ff21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.628+0000 7f9d097cd640 1 --2- 172.21.15.70:0/265907375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7fe56a0 0x5617f7ff21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.628+0000 7f9d097cd640 1 -- 172.21.15.70:0/265907375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617f7fd1e60 con 0x5617f7ff3a10 2024-04-01T01:36:30.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.628+0000 7f9d03fff640 1 --2- 172.21.15.70:0/265907375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe6090 0x5617f7ffd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:36:30.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.629+0000 7f9d08fcc640 1 --2- 172.21.15.70:0/265907375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7fe56a0 0x5617f7ff21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:36:30.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.629+0000 7f9d097cd640 1 --2- 172.21.15.70:0/265907375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7ff3a10 0x5617f7ffd680 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8098d70 tx=0x7f9cf80b9350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8afd33cb3eb433ca server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.629+0000 7f9cee7fc640 1 -- 172.21.15.70:0/265907375 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8020050 con 0x5617f7ff3a10 2024-04-01T01:36:30.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.629+0000 7f9cee7fc640 1 -- 172.21.15.70:0/265907375 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9cf809aac0 con 0x5617f7ff3a10 2024-04-01T01:36:30.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.629+0000 7f9cee7fc640 1 -- 172.21.15.70:0/265907375 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf80503f0 con 0x5617f7ff3a10 2024-04-01T01:36:30.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.630+0000 7f9d0de0c740 1 -- 172.21.15.70:0/265907375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7ff3a10 msgr2=0x5617f7ffd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.630+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/265907375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7ff3a10 0x5617f7ffd680 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8098d70 tx=0x7f9cf80b9350 comp rx=0 tx=0).stop 2024-04-01T01:36:30.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.630+0000 7f9d0de0c740 1 -- 172.21.15.70:0/265907375 shutdown_connections 2024-04-01T01:36:30.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.630+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/265907375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7ff3a10 0x5617f7ffd680 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.630+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/265907375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7fe6090 0x5617f7ffd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.630+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/265907375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7fe56a0 0x5617f7ff21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.630+0000 7f9d0de0c740 1 -- 172.21.15.70:0/265907375 >> 172.21.15.70:0/265907375 conn(0x5617f7fe0c10 msgr2=0x5617f7fe0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.630+0000 7f9d0de0c740 1 -- 172.21.15.70:0/265907375 shutdown_connections 2024-04-01T01:36:30.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.631+0000 7f9d0de0c740 1 -- 172.21.15.70:0/265907375 wait complete. 2024-04-01T01:36:30.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.631+0000 7f9d0de0c740 1 -- start start 2024-04-01T01:36:30.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.631+0000 7f9d0de0c740 1 Processor -- start 2024-04-01T01:36:30.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.632+0000 7f9d0de0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7fe56a0 0x5617f800bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.632+0000 7f9d0de0c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7fe6090 0x5617f800c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.632+0000 7f9d0de0c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7ff3a10 0x5617f800d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.632+0000 7f9d03fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7fe6090 0x5617f800c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.632+0000 7f9d03fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7fe6090 0x5617f800c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35600/0 (socket says 172.21.15.70:35600) 2024-04-01T01:36:30.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.632+0000 7f9d03fff640 1 -- 172.21.15.70:0/2539262011 learned_addr learned my addr 172.21.15.70:0/2539262011 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.632+0000 7f9d0de0c740 1 -- 172.21.15.70:0/2539262011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9cfc13dba0 con 0x5617f7fe56a0 2024-04-01T01:36:30.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.632+0000 7f9d097cd640 1 --2- 172.21.15.70:0/2539262011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7ff3a10 0x5617f800d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.633+0000 7f9d0de0c740 1 -- 172.21.15.70:0/2539262011 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9cfc13d4d0 con 0x5617f7fe6090 2024-04-01T01:36:30.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.633+0000 7f9d0de0c740 1 -- 172.21.15.70:0/2539262011 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9cfc13a750 con 0x5617f7ff3a10 2024-04-01T01:36:30.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.633+0000 7f9d03fff640 1 -- 172.21.15.70:0/2539262011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7ff3a10 msgr2=0x5617f800d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.633+0000 7f9d03fff640 1 --2- 172.21.15.70:0/2539262011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7ff3a10 0x5617f800d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.633+0000 7f9d03fff640 1 -- 172.21.15.70:0/2539262011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7fe56a0 msgr2=0x5617f800bc40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:36:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.633+0000 7f9d03fff640 1 --2- 172.21.15.70:0/2539262011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7fe56a0 0x5617f800bc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.633+0000 7f9d03fff640 1 -- 172.21.15.70:0/2539262011 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617f800d780 con 0x5617f7fe6090 2024-04-01T01:36:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.633+0000 7f9d0de0c740 10 client.0 _ll_register_callbacks cb 0x7f9cf809ec10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.633+0000 7f9d03fff640 1 --2- 172.21.15.70:0/2539262011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7fe6090 0x5617f800c180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0009900 tx=0x7f9cf0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1f48d0214db8c4e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.634+0000 7f9d0de0c740 1 -- 172.21.15.70:0/2539262011 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cf809aac0 con 0x5617f7fe6090 2024-04-01T01:36:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.634+0000 7f9ce3fff640 10 client.26366 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:36:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.634+0000 7f9ce3fff640 1 -- 172.21.15.70:0/2539262011 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0050070 con 0x5617f7fe6090 2024-04-01T01:36:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.634+0000 7f9ce3fff640 1 -- 172.21.15.70:0/2539262011 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9cf00046c0 con 0x5617f7fe6090 2024-04-01T01:36:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.634+0000 7f9ce3fff640 1 -- 172.21.15.70:0/2539262011 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9cf005e050 con 0x5617f7fe6090 2024-04-01T01:36:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.634+0000 7f9d0de0c740 10 client.26366 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.634+0000 7f9d0de0c740 1 -- 172.21.15.70:0/2539262011 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=29}) -- 0x5617f8060d20 con 0x5617f7fe6090 2024-04-01T01:36:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.635+0000 7f9ce3fff640 1 -- 172.21.15.70:0/2539262011 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9cf00048c0 con 0x5617f7fe6090 2024-04-01T01:36:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.635+0000 7f9d0de0c740 10 client.26366 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.635+0000 7f9d0de0c740 10 client.26366 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:36:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.635+0000 7f9d0de0c740 20 client.26366 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.635+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/2539262011 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5617f8061610 0x5617f8063a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.635+0000 7f9d0de0c740 4 client.26366 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:36:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.635+0000 7f9d0de0c740 1 -- 172.21.15.70:0/2539262011 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f9cf809aac0 con 0x5617f8061610 2024-04-01T01:36:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.635+0000 7f9d08fcc640 1 --2- 172.21.15.70:0/2539262011 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5617f8061610 0x5617f8063a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.636+0000 7f9d08fcc640 1 --2- 172.21.15.70:0/2539262011 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5617f8061610 0x5617f8063a30 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1702a78b82812a14 server_cookie=7b9588b73e31328e in_seq=0 out_seq=0 2024-04-01T01:36:30.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.636+0000 7f9ce3fff640 10 client.26366 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:36:30.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.638+0000 7f9ce3fff640 1 -- 172.21.15.70:0/2539262011 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f9cf809aac0 con 0x5617f8061610 2024-04-01T01:36:30.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.638+0000 7f9ce3fff640 10 client.26366 handle_command_reply: tid=0 2024-04-01T01:36:30.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.638+0000 7f9d0de0c740 1 client.26366 shutdown 2024-04-01T01:36:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.638+0000 7f9d0de0c740 1 -- 172.21.15.70:0/2539262011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7fe6090 msgr2=0x5617f800c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.638+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/2539262011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7fe6090 0x5617f800c180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0009900 tx=0x7f9cf0009260 comp rx=0 tx=0).stop 2024-04-01T01:36:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.638+0000 7f9d0de0c740 1 -- 172.21.15.70:0/2539262011 shutdown_connections 2024-04-01T01:36:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.638+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/2539262011 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5617f8061610 0x5617f8063a30 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.638+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/2539262011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617f7ff3a10 0x5617f800d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.638+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/2539262011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617f7fe6090 0x5617f800c180 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.638+0000 7f9d0de0c740 1 --2- 172.21.15.70:0/2539262011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617f7fe56a0 0x5617f800bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.638+0000 7f9d0de0c740 1 -- 172.21.15.70:0/2539262011 >> 172.21.15.70:0/2539262011 conn(0x5617f7fe0c10 msgr2=0x5617f7fe7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.638+0000 7f9ce3fff640 0 client.26366 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:36:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.639+0000 7f9d0de0c740 1 -- 172.21.15.70:0/2539262011 shutdown_connections 2024-04-01T01:36:30.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.639+0000 7f9d0de0c740 1 -- 172.21.15.70:0/2539262011 wait complete. 2024-04-01T01:36:30.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.640+0000 7f9d0de0c740 20 client.26366 trim_cache size 0 max 16384 2024-04-01T01:36:30.639 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:36:30.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.641+0000 7f9d0b230640 1 -- 172.21.15.70:0/2947932782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9cf40a25a0 msgr2=0x7f9cf40a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.641+0000 7f9d0b230640 1 --2- 172.21.15.70:0/2947932782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9cf40a25a0 0x7f9cf40a4a60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0000c00 tx=0x7f9cf0005fd0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.641+0000 7f9d0b230640 1 -- 172.21.15.70:0/2947932782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cfc099ab0 msgr2=0x7f9cfc135780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.641+0000 7f9d0b230640 1 --2- 172.21.15.70:0/2947932782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cfc099ab0 0x7f9cfc135780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800d750 tx=0x7f9cf800db50 comp rx=0 tx=0).stop 2024-04-01T01:36:30.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.642+0000 7f9d0b230640 1 -- 172.21.15.70:0/2947932782 shutdown_connections 2024-04-01T01:36:30.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.642+0000 7f9d0b230640 1 --2- 172.21.15.70:0/2947932782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9cfc099ab0 0x7f9cfc135780 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.642+0000 7f9d0b230640 1 --2- 172.21.15.70:0/2947932782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9cf40a25a0 0x7f9cf40a4a60 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.642+0000 7f9d0b230640 1 --2- 172.21.15.70:0/2947932782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cfc0990f0 0x7f9cfc12e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.642+0000 7f9d0b230640 1 --2- 172.21.15.70:0/2947932782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cfc097f20 0x7f9cfc12e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.642+0000 7f9d0b230640 1 -- 172.21.15.70:0/2947932782 >> 172.21.15.70:0/2947932782 conn(0x7f9cfc0934e0 msgr2=0x7f9cfc09c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.643+0000 7f9d0b230640 1 -- 172.21.15.70:0/2947932782 shutdown_connections 2024-04-01T01:36:30.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.643+0000 7f9d0b230640 1 -- 172.21.15.70:0/2947932782 wait complete. 2024-04-01T01:36:30.662 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:36:30.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.840+0000 7fcea1ffb640 1 -- 172.21.15.70:0/3424160413 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea4096a40 con 0x7fce9c0a5ef0 2024-04-01T01:36:30.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.840+0000 7fcea3fff640 1 -- 172.21.15.70:0/3424160413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce9c0a5ef0 msgr2=0x7fce9c00c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.840+0000 7fcea3fff640 1 --2- 172.21.15.70:0/3424160413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce9c0a5ef0 0x7fce9c00c530 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcea4067420 tx=0x7fcea408f440 comp rx=0 tx=0).stop 2024-04-01T01:36:30.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.840+0000 7fcea3fff640 1 -- 172.21.15.70:0/3424160413 shutdown_connections 2024-04-01T01:36:30.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.840+0000 7fcea3fff640 1 --2- 172.21.15.70:0/3424160413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce9c0a5ef0 0x7fce9c00c530 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.840+0000 7fcea3fff640 1 --2- 172.21.15.70:0/3424160413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce9c0a4cd0 0x7fce9c0a5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.840+0000 7fcea3fff640 1 --2- 172.21.15.70:0/3424160413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce9c0a65b0 0x7fce9c0a4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.840+0000 7fcea3fff640 1 -- 172.21.15.70:0/3424160413 >> 172.21.15.70:0/3424160413 conn(0x7fce9c0a00d0 msgr2=0x7fce9c0a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.840+0000 7fcea3fff640 1 -- 172.21.15.70:0/3424160413 shutdown_connections 2024-04-01T01:36:30.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.840+0000 7fcea3fff640 1 -- 172.21.15.70:0/3424160413 wait complete. 2024-04-01T01:36:30.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.841+0000 7fcea3fff640 1 Processor -- start 2024-04-01T01:36:30.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.841+0000 7fcea3fff640 1 -- start start 2024-04-01T01:36:30.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.842+0000 7fcea3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce9c0a4cd0 0x7fce9c0164e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.842+0000 7fcea3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce9c0a5ef0 0x7fce9c014b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.842+0000 7fcea3fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce9c0a65b0 0x7fce9c015060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.842+0000 7fcea3fff640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fce9c00f330 con 0x7fce9c0a5ef0 2024-04-01T01:36:30.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.842+0000 7fcea3fff640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fce9c00f1b0 con 0x7fce9c0a4cd0 2024-04-01T01:36:30.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.842+0000 7fcea3fff640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fce9c00f4b0 con 0x7fce9c0a65b0 2024-04-01T01:36:30.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.842+0000 7fcea37fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce9c0a65b0 0x7fce9c015060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.843+0000 7fcea37fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce9c0a65b0 0x7fce9c015060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:38026/0 (socket says 172.21.15.70:38026) 2024-04-01T01:36:30.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.843+0000 7fcea37fe640 1 -- 172.21.15.70:0/1671307528 learned_addr learned my addr 172.21.15.70:0/1671307528 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.843+0000 7fcea27fc640 1 --2- 172.21.15.70:0/1671307528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce9c0a5ef0 0x7fce9c014b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.843+0000 7fcea27fc640 1 -- 172.21.15.70:0/1671307528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce9c0a65b0 msgr2=0x7fce9c015060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.843+0000 7fcea27fc640 1 --2- 172.21.15.70:0/1671307528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce9c0a65b0 0x7fce9c015060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.843+0000 7fcea27fc640 1 -- 172.21.15.70:0/1671307528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce9c0a4cd0 msgr2=0x7fce9c0164e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.843+0000 7fcea27fc640 1 --2- 172.21.15.70:0/1671307528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce9c0a4cd0 0x7fce9c0164e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.843+0000 7fcea27fc640 1 -- 172.21.15.70:0/1671307528 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce9c015700 con 0x7fce9c0a5ef0 2024-04-01T01:36:30.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.844+0000 7fcea27fc640 1 --2- 172.21.15.70:0/1671307528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce9c0a5ef0 0x7fce9c014b20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fce9800e790 tx=0x7fce9800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.844+0000 7fce8bfff640 1 -- 172.21.15.70:0/1671307528 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9800eea0 con 0x7fce9c0a5ef0 2024-04-01T01:36:30.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.844+0000 7fcea3fff640 1 -- 172.21.15.70:0/1671307528 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce9c15c4f0 con 0x7fce9c0a5ef0 2024-04-01T01:36:30.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.844+0000 7fcea3fff640 1 -- 172.21.15.70:0/1671307528 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce9c15c950 con 0x7fce9c0a5ef0 2024-04-01T01:36:30.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.844+0000 7fce8bfff640 1 -- 172.21.15.70:0/1671307528 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce9800ccc0 con 0x7fce9c0a5ef0 2024-04-01T01:36:30.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.844+0000 7fce8bfff640 1 -- 172.21.15.70:0/1671307528 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce98018710 con 0x7fce9c0a5ef0 2024-04-01T01:36:30.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.848+0000 7fceac377740 1 Processor -- start 2024-04-01T01:36:30.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.848+0000 7fceac377740 1 -- start start 2024-04-01T01:36:30.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.848+0000 7fceac377740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 0x55cdf686eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.848+0000 7fceac377740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 0x55cdf6873b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.848+0000 7fceac377740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 0x55cdf6876690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.848+0000 7fceac377740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cdf67d31b0 con 0x55cdf686f090 2024-04-01T01:36:30.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.848+0000 7fceac377740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55cdf677bc00 con 0x55cdf6874200 2024-04-01T01:36:30.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.848+0000 7fceac377740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cdf67cf250 con 0x55cdf686e6a0 2024-04-01T01:36:30.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.848+0000 7fcea2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 0x55cdf6873b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.848+0000 7fcea2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 0x55cdf6873b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59906/0 (socket says 172.21.15.70:59906) 2024-04-01T01:36:30.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.848+0000 7fcea2ffd640 1 -- 172.21.15.70:0/3024401615 learned_addr learned my addr 172.21.15.70:0/3024401615 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.848+0000 7fce8bfff640 1 -- 172.21.15.70:0/1671307528 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fce980188b0 con 0x7fce9c0a5ef0 2024-04-01T01:36:30.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.849+0000 7fcea37fe640 1 --2- 172.21.15.70:0/3024401615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 0x55cdf686eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.849+0000 7fcea27fc640 1 --2- 172.21.15.70:0/3024401615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 0x55cdf6876690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.850+0000 7fcea2ffd640 1 -- 172.21.15.70:0/3024401615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 msgr2=0x55cdf686eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.850+0000 7fcea2ffd640 1 --2- 172.21.15.70:0/3024401615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 0x55cdf686eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.850+0000 7fcea2ffd640 1 -- 172.21.15.70:0/3024401615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 msgr2=0x55cdf6876690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.850+0000 7fcea2ffd640 1 --2- 172.21.15.70:0/3024401615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 0x55cdf6876690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.850+0000 7fcea2ffd640 1 -- 172.21.15.70:0/3024401615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdf679b3d0 con 0x55cdf686f090 2024-04-01T01:36:30.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.850+0000 7fcea37fe640 1 --2- 172.21.15.70:0/3024401615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 0x55cdf686eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:36:30.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.850+0000 7fce8bfff640 1 --2- 172.21.15.70:0/1671307528 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fce8c0a25f0 0x7fce8c0a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.850+0000 7fcea2ffd640 1 --2- 172.21.15.70:0/3024401615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 0x55cdf6873b90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fce9400ab60 tx=0x7fce9400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=496954a61ec4bbe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.851+0000 7fcea37fe640 1 --2- 172.21.15.70:0/1671307528 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fce8c0a25f0 0x7fce8c0a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.851+0000 7fce8bfff640 1 -- 172.21.15.70:0/1671307528 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fce980ced80 con 0x7fce9c0a5ef0 2024-04-01T01:36:30.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.851+0000 7fce897fa640 1 -- 172.21.15.70:0/3024401615 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9400ae60 con 0x55cdf686f090 2024-04-01T01:36:30.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.852+0000 7fce897fa640 1 -- 172.21.15.70:0/3024401615 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce9400fb00 con 0x55cdf686f090 2024-04-01T01:36:30.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.852+0000 7fce897fa640 1 -- 172.21.15.70:0/3024401615 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce94018ac0 con 0x55cdf686f090 2024-04-01T01:36:30.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.852+0000 7fcea37fe640 1 --2- 172.21.15.70:0/1671307528 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fce8c0a25f0 0x7fce8c0a4ab0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcea408fa50 tx=0x7fcea40675f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.853+0000 7fceac377740 1 -- 172.21.15.70:0/3024401615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 msgr2=0x55cdf6873b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.853+0000 7fceac377740 1 --2- 172.21.15.70:0/3024401615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 0x55cdf6873b90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fce9400ab60 tx=0x7fce9400c6a0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.853+0000 7fceac377740 1 -- 172.21.15.70:0/3024401615 shutdown_connections 2024-04-01T01:36:30.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.853+0000 7fceac377740 1 --2- 172.21.15.70:0/3024401615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 0x55cdf6876690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.853+0000 7fceac377740 1 --2- 172.21.15.70:0/3024401615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 0x55cdf6873b90 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.853+0000 7fceac377740 1 --2- 172.21.15.70:0/3024401615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 0x55cdf686eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.853+0000 7fceac377740 1 -- 172.21.15.70:0/3024401615 >> 172.21.15.70:0/3024401615 conn(0x55cdf6869c10 msgr2=0x55cdf686a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.853+0000 7fceac377740 1 -- 172.21.15.70:0/3024401615 shutdown_connections 2024-04-01T01:36:30.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.853+0000 7fceac377740 1 -- 172.21.15.70:0/3024401615 wait complete. 2024-04-01T01:36:30.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.854+0000 7fceac377740 1 -- start start 2024-04-01T01:36:30.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.854+0000 7fceac377740 1 Processor -- start 2024-04-01T01:36:30.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.855+0000 7fceac377740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 0x55cdf68789f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.855+0000 7fceac377740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 0x55cdf6878f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.855+0000 7fceac377740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 0x55cdf687ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.855+0000 7fceac377740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cdf68771b0 con 0x55cdf686f090 2024-04-01T01:36:30.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.855+0000 7fcea27fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 0x55cdf6878f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.855+0000 7fcea2ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 0x55cdf68789f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.855+0000 7fceac377740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55cdf6771470 con 0x55cdf6874200 2024-04-01T01:36:30.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.855+0000 7fcea2ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 0x55cdf68789f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:38046/0 (socket says 172.21.15.70:38046) 2024-04-01T01:36:30.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.855+0000 7fceac377740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cdf6877330 con 0x55cdf686e6a0 2024-04-01T01:36:30.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.855+0000 7fcea2ffd640 1 -- 172.21.15.70:0/891579386 learned_addr learned my addr 172.21.15.70:0/891579386 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.855+0000 7fceac377740 10 client.0 _ll_register_callbacks cb 0x55cdf678ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:30.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.855+0000 7fcea37fe640 1 --2- 172.21.15.70:0/891579386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 0x55cdf687ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.856+0000 7fcea2ffd640 1 -- 172.21.15.70:0/891579386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 msgr2=0x55cdf687ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.856+0000 7fcea2ffd640 1 --2- 172.21.15.70:0/891579386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 0x55cdf687ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.856+0000 7fcea2ffd640 1 -- 172.21.15.70:0/891579386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 msgr2=0x55cdf6878f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.856+0000 7fcea2ffd640 1 --2- 172.21.15.70:0/891579386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 0x55cdf6878f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.856+0000 7fcea2ffd640 1 -- 172.21.15.70:0/891579386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdf64e73f0 con 0x55cdf686e6a0 2024-04-01T01:36:30.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.856+0000 7fcea2ffd640 1 --2- 172.21.15.70:0/891579386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 0x55cdf68789f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fce94017000 tx=0x7fce9400f940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78642c2b8f90cacf server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.856+0000 7fce86ff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:36:30.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.856+0000 7fce86ff5640 1 -- 172.21.15.70:0/891579386 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce94027020 con 0x55cdf686e6a0 2024-04-01T01:36:30.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.856+0000 7fce86ff5640 1 -- 172.21.15.70:0/891579386 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fce94018bd0 con 0x55cdf686e6a0 2024-04-01T01:36:30.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.856+0000 7fceac377740 1 -- 172.21.15.70:0/891579386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce9400fb00 con 0x55cdf686e6a0 2024-04-01T01:36:30.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.857+0000 7fce86ff5640 1 -- 172.21.15.70:0/891579386 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fce9402f050 con 0x55cdf686e6a0 2024-04-01T01:36:30.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.857+0000 7fceac377740 10 client.26281 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:30.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.857+0000 7fceac377740 1 -- 172.21.15.70:0/891579386 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cdf65b17d0 con 0x55cdf686e6a0 2024-04-01T01:36:30.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.857+0000 7fce86ff5640 1 -- 172.21.15.70:0/891579386 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce9400fb00 con 0x55cdf686e6a0 2024-04-01T01:36:30.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.857+0000 7fce86ff5640 1 -- 172.21.15.70:0/891579386 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fce94023b30 con 0x55cdf686e6a0 2024-04-01T01:36:30.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.857+0000 7fceac377740 10 client.26281 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:30.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.857+0000 7fceac377740 10 client.26281 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:36:30.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.857+0000 7fceac377740 20 client.26281 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:30.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.857+0000 7fceac377740 1 --2- 172.21.15.70:0/891579386 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cdf68840b0 0x55cdf68864a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.857+0000 7fceac377740 4 client.26281 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:36:30.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.857+0000 7fceac377740 1 -- 172.21.15.70:0/891579386 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cdf67e06f0 con 0x55cdf68840b0 2024-04-01T01:36:30.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.858+0000 7fcea27fc640 1 --2- 172.21.15.70:0/891579386 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cdf68840b0 0x55cdf68864a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.858+0000 7fce86ff5640 10 client.26281 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:30.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.858+0000 7fcea27fc640 1 --2- 172.21.15.70:0/891579386 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cdf68840b0 0x55cdf68864a0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9a0d16573a9da9 server_cookie=70fe57ee50863297 in_seq=0 out_seq=0 2024-04-01T01:36:30.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.859+0000 7fce86ff5640 1 -- 172.21.15.70:0/891579386 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cdf67e06f0 con 0x55cdf68840b0 2024-04-01T01:36:30.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.859+0000 7fce86ff5640 10 client.26281 handle_command_reply: tid=0 2024-04-01T01:36:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.859+0000 7fceac377740 1 client.26281 shutdown 2024-04-01T01:36:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.860+0000 7fceac377740 1 -- 172.21.15.70:0/891579386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 msgr2=0x55cdf68789f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.860+0000 7fceac377740 1 --2- 172.21.15.70:0/891579386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 0x55cdf68789f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fce94017000 tx=0x7fce9400f940 comp rx=0 tx=0).stop 2024-04-01T01:36:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.860+0000 7fceac377740 1 -- 172.21.15.70:0/891579386 shutdown_connections 2024-04-01T01:36:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.860+0000 7fceac377740 1 --2- 172.21.15.70:0/891579386 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cdf68840b0 0x55cdf68864a0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.860+0000 7fceac377740 1 --2- 172.21.15.70:0/891579386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 0x55cdf687ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.860+0000 7fceac377740 1 --2- 172.21.15.70:0/891579386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 0x55cdf6878f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.860+0000 7fce86ff5640 0 client.26281 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.860+0000 7fceac377740 1 --2- 172.21.15.70:0/891579386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 0x55cdf68789f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.860+0000 7fceac377740 1 -- 172.21.15.70:0/891579386 >> 172.21.15.70:0/891579386 conn(0x55cdf6869c10 msgr2=0x55cdf6868f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.860+0000 7fceac377740 1 -- 172.21.15.70:0/891579386 shutdown_connections 2024-04-01T01:36:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.860+0000 7fceac377740 1 -- 172.21.15.70:0/891579386 wait complete. 2024-04-01T01:36:30.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.860+0000 7fceac377740 20 client.26281 trim_cache size 0 max 16384 2024-04-01T01:36:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.885+0000 7fceac377740 1 Processor -- start 2024-04-01T01:36:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.885+0000 7fceac377740 1 -- start start 2024-04-01T01:36:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.886+0000 7fceac377740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 0x55cdf6878600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.886+0000 7fceac377740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 0x55cdf687ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.886+0000 7fceac377740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 0x55cdf6878b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.886+0000 7fceac377740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fce9000fcc0 con 0x55cdf686f090 2024-04-01T01:36:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.886+0000 7fceac377740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fce90011800 con 0x55cdf6874200 2024-04-01T01:36:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.886+0000 7fceac377740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fce90010390 con 0x55cdf686e6a0 2024-04-01T01:36:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.886+0000 7fcea27fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 0x55cdf687ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.886+0000 7fcea27fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 0x55cdf687ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59924/0 (socket says 172.21.15.70:59924) 2024-04-01T01:36:30.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.886+0000 7fcea27fc640 1 -- 172.21.15.70:0/1889980670 learned_addr learned my addr 172.21.15.70:0/1889980670 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.886+0000 7fcea37fe640 1 --2- 172.21.15.70:0/1889980670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 0x55cdf6878b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.886+0000 7fcea27fc640 1 -- 172.21.15.70:0/1889980670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 msgr2=0x55cdf6878600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.886+0000 7fcea27fc640 1 --2- 172.21.15.70:0/1889980670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 0x55cdf6878600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.886+0000 7fcea27fc640 1 -- 172.21.15.70:0/1889980670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 msgr2=0x55cdf6878b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.886+0000 7fcea27fc640 1 --2- 172.21.15.70:0/1889980670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 0x55cdf6878b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.887+0000 7fcea27fc640 1 -- 172.21.15.70:0/1889980670 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdf6886ad0 con 0x55cdf686f090 2024-04-01T01:36:30.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.887+0000 7fcea27fc640 1 --2- 172.21.15.70:0/1889980670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 0x55cdf687ff30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fce9800cf90 tx=0x7fce98097040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df793769a068b85c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.888+0000 7fce88ff9640 1 -- 172.21.15.70:0/1889980670 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce9809bb10 con 0x55cdf686f090 2024-04-01T01:36:30.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.888+0000 7fce88ff9640 1 -- 172.21.15.70:0/1889980670 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce9809b410 con 0x55cdf686f090 2024-04-01T01:36:30.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.888+0000 7fce88ff9640 1 -- 172.21.15.70:0/1889980670 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce980997a0 con 0x55cdf686f090 2024-04-01T01:36:30.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.889+0000 7fceac377740 1 -- 172.21.15.70:0/1889980670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 msgr2=0x55cdf687ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.889+0000 7fceac377740 1 --2- 172.21.15.70:0/1889980670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 0x55cdf687ff30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fce9800cf90 tx=0x7fce98097040 comp rx=0 tx=0).stop 2024-04-01T01:36:30.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.889+0000 7fceac377740 1 -- 172.21.15.70:0/1889980670 shutdown_connections 2024-04-01T01:36:30.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.889+0000 7fceac377740 1 --2- 172.21.15.70:0/1889980670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 0x55cdf6878b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.889+0000 7fceac377740 1 --2- 172.21.15.70:0/1889980670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686f090 0x55cdf687ff30 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.889+0000 7fceac377740 1 --2- 172.21.15.70:0/1889980670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686e6a0 0x55cdf6878600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.889+0000 7fceac377740 1 -- 172.21.15.70:0/1889980670 >> 172.21.15.70:0/1889980670 conn(0x55cdf6869c10 msgr2=0x55cdf68739e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.889+0000 7fceac377740 1 -- 172.21.15.70:0/1889980670 shutdown_connections 2024-04-01T01:36:30.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.889+0000 7fceac377740 1 -- 172.21.15.70:0/1889980670 wait complete. 2024-04-01T01:36:30.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.890+0000 7fceac377740 1 -- start start 2024-04-01T01:36:30.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.890+0000 7fceac377740 1 Processor -- start 2024-04-01T01:36:30.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.890+0000 7fceac377740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686e6a0 0x55cdf687d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.890+0000 7fceac377740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686f090 0x55cdf687dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.890+0000 7fceac377740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 0x55cdf687ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.890+0000 7fceac377740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fce90011130 con 0x55cdf686e6a0 2024-04-01T01:36:30.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.890+0000 7fceac377740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fce90010a60 con 0x55cdf6874200 2024-04-01T01:36:30.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.890+0000 7fceac377740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fce9000dca0 con 0x55cdf686f090 2024-04-01T01:36:30.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.890+0000 7fceac377740 10 client.0 _ll_register_callbacks cb 0x7fce9400a9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:30.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.891+0000 7fcea2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686e6a0 0x55cdf687d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.891+0000 7fcea2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686e6a0 0x55cdf687d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59930/0 (socket says 172.21.15.70:59930) 2024-04-01T01:36:30.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.891+0000 7fcea2ffd640 1 -- 172.21.15.70:0/3787420137 learned_addr learned my addr 172.21.15.70:0/3787420137 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:30.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.891+0000 7fcea27fc640 1 --2- 172.21.15.70:0/3787420137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686f090 0x55cdf687dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.891+0000 7fcea37fe640 1 --2- 172.21.15.70:0/3787420137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 0x55cdf687ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.891+0000 7fcea2ffd640 1 -- 172.21.15.70:0/3787420137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686f090 msgr2=0x55cdf687dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.891+0000 7fcea2ffd640 1 --2- 172.21.15.70:0/3787420137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686f090 0x55cdf687dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.891+0000 7fcea2ffd640 1 -- 172.21.15.70:0/3787420137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 msgr2=0x55cdf687ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:30.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.891+0000 7fcea2ffd640 1 --2- 172.21.15.70:0/3787420137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 0x55cdf687ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.891+0000 7fcea2ffd640 1 -- 172.21.15.70:0/3787420137 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdf687f220 con 0x55cdf686e6a0 2024-04-01T01:36:30.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.891+0000 7fcea2ffd640 1 --2- 172.21.15.70:0/3787420137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686e6a0 0x55cdf687d6e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fce94025000 tx=0x7fce94018c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7520c0c99e53e0e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:30.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.891+0000 7fce86ff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:36:30.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.892+0000 7fce86ff5640 1 -- 172.21.15.70:0/3787420137 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce94037020 con 0x55cdf686e6a0 2024-04-01T01:36:30.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.892+0000 7fce86ff5640 1 -- 172.21.15.70:0/3787420137 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fce94022ce0 con 0x55cdf686e6a0 2024-04-01T01:36:30.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.892+0000 7fceac377740 1 -- 172.21.15.70:0/3787420137 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce9809b410 con 0x55cdf686e6a0 2024-04-01T01:36:30.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.892+0000 7fce86ff5640 1 -- 172.21.15.70:0/3787420137 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fce94027040 con 0x55cdf686e6a0 2024-04-01T01:36:30.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.893+0000 7fceac377740 10 client.16542 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:30.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.893+0000 7fceac377740 1 -- 172.21.15.70:0/3787420137 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cdf68fb340 con 0x55cdf686e6a0 2024-04-01T01:36:30.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.894+0000 7fce86ff5640 1 -- 172.21.15.70:0/3787420137 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce9400fb00 con 0x55cdf686e6a0 2024-04-01T01:36:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.894+0000 7fce86ff5640 1 -- 172.21.15.70:0/3787420137 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fce940238f0 con 0x55cdf686e6a0 2024-04-01T01:36:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.894+0000 7fceac377740 10 client.16542 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.894+0000 7fceac377740 10 client.16542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:36:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.894+0000 7fceac377740 20 client.16542 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.894+0000 7fceac377740 1 --2- 172.21.15.70:0/3787420137 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cdf68fbc50 0x55cdf68fe070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.894+0000 7fceac377740 4 client.16542 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:36:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.894+0000 7fceac377740 1 -- 172.21.15.70:0/3787420137 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fce9809b410 con 0x55cdf68fbc50 2024-04-01T01:36:30.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.895+0000 7fcea27fc640 1 --2- 172.21.15.70:0/3787420137 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cdf68fbc50 0x55cdf68fe070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:30.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.896+0000 7fcea27fc640 1 --2- 172.21.15.70:0/3787420137 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cdf68fbc50 0x55cdf68fe070 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=353838afa742eb1e server_cookie=bd565d6fecca21c7 in_seq=0 out_seq=0 2024-04-01T01:36:30.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.896+0000 7fce86ff5640 10 client.16542 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:30.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.896+0000 7fce86ff5640 1 -- 172.21.15.70:0/3787420137 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fce9809b410 con 0x55cdf68fbc50 2024-04-01T01:36:30.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.896+0000 7fce86ff5640 10 client.16542 handle_command_reply: tid=0 2024-04-01T01:36:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.896+0000 7fceac377740 1 client.16542 shutdown 2024-04-01T01:36:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.896+0000 7fceac377740 1 -- 172.21.15.70:0/3787420137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686e6a0 msgr2=0x55cdf687d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.896+0000 7fceac377740 1 --2- 172.21.15.70:0/3787420137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686e6a0 0x55cdf687d6e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fce94025000 tx=0x7fce94018c80 comp rx=0 tx=0).stop 2024-04-01T01:36:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.896+0000 7fceac377740 1 -- 172.21.15.70:0/3787420137 shutdown_connections 2024-04-01T01:36:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.896+0000 7fceac377740 1 --2- 172.21.15.70:0/3787420137 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cdf68fbc50 0x55cdf68fe070 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.896+0000 7fceac377740 1 --2- 172.21.15.70:0/3787420137 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdf6874200 0x55cdf687ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.896+0000 7fceac377740 1 --2- 172.21.15.70:0/3787420137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cdf686f090 0x55cdf687dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.896+0000 7fceac377740 1 --2- 172.21.15.70:0/3787420137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cdf686e6a0 0x55cdf687d6e0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.896+0000 7fceac377740 1 -- 172.21.15.70:0/3787420137 >> 172.21.15.70:0/3787420137 conn(0x55cdf6869c10 msgr2=0x55cdf68785b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.896+0000 7fce86ff5640 0 client.16542 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:30.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.896+0000 7fceac377740 1 -- 172.21.15.70:0/3787420137 shutdown_connections 2024-04-01T01:36:30.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.897+0000 7fceac377740 1 -- 172.21.15.70:0/3787420137 wait complete. 2024-04-01T01:36:30.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.897+0000 7fceac377740 20 client.16542 trim_cache size 0 max 16384 2024-04-01T01:36:30.897 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:36:30.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.898+0000 7fcea3fff640 1 -- 172.21.15.70:0/1671307528 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fce8c0a25f0 msgr2=0x7fce8c0a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.898+0000 7fcea3fff640 1 --2- 172.21.15.70:0/1671307528 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fce8c0a25f0 0x7fce8c0a4ab0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcea408fa50 tx=0x7fcea40675f0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.898+0000 7fcea3fff640 1 -- 172.21.15.70:0/1671307528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce9c0a5ef0 msgr2=0x7fce9c014b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:30.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.898+0000 7fcea3fff640 1 --2- 172.21.15.70:0/1671307528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce9c0a5ef0 0x7fce9c014b20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fce9800e790 tx=0x7fce9800ec50 comp rx=0 tx=0).stop 2024-04-01T01:36:30.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.898+0000 7fcea3fff640 1 -- 172.21.15.70:0/1671307528 shutdown_connections 2024-04-01T01:36:30.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.898+0000 7fcea3fff640 1 --2- 172.21.15.70:0/1671307528 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fce8c0a25f0 0x7fce8c0a4ab0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.898+0000 7fcea3fff640 1 --2- 172.21.15.70:0/1671307528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce9c0a65b0 0x7fce9c015060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.898+0000 7fcea3fff640 1 --2- 172.21.15.70:0/1671307528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fce9c0a5ef0 0x7fce9c014b20 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.898+0000 7fcea3fff640 1 --2- 172.21.15.70:0/1671307528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce9c0a4cd0 0x7fce9c0164e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:30.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.898+0000 7fcea3fff640 1 -- 172.21.15.70:0/1671307528 >> 172.21.15.70:0/1671307528 conn(0x7fce9c0a00d0 msgr2=0x7fce9c0a0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:30.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.898+0000 7fcea3fff640 1 -- 172.21.15.70:0/1671307528 shutdown_connections 2024-04-01T01:36:30.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:30.899+0000 7fcea3fff640 1 -- 172.21.15.70:0/1671307528 wait complete. 2024-04-01T01:36:31.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 105.0 s, 1226.3 tps, lat 81.498 ms stddev 33.376 2024-04-01T01:36:31.918 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:36:31.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:36:31.919 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:36:31.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:31 smithi189 ceph-mon[35753]: pgmap v449: 129 pgs: 129 active+clean; 990 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 9.8 MiB/s wr, 1.18k op/s 2024-04-01T01:36:32.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:31 smithi070 ceph-mon[32179]: pgmap v449: 129 pgs: 129 active+clean; 990 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 9.8 MiB/s wr, 1.18k op/s 2024-04-01T01:36:32.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:31 smithi081 ceph-mon[35142]: pgmap v449: 129 pgs: 129 active+clean; 990 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 9.8 MiB/s wr, 1.18k op/s 2024-04-01T01:36:32.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.102+0000 7fb9a1299640 1 -- 172.21.15.70:0/2875867148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb99c1097e0 msgr2=0x7fb99c10e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.102+0000 7fb9a1299640 1 --2- 172.21.15.70:0/2875867148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb99c1097e0 0x7fb99c10e080 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb98c009900 tx=0x7fb98c02bc60 comp rx=0 tx=0).stop 2024-04-01T01:36:32.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.102+0000 7fb9a1299640 1 -- 172.21.15.70:0/2875867148 shutdown_connections 2024-04-01T01:36:32.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.102+0000 7fb9a1299640 1 --2- 172.21.15.70:0/2875867148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb99c1097e0 0x7fb99c10e080 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.102+0000 7fb9a1299640 1 --2- 172.21.15.70:0/2875867148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb99c108e20 0x7fb99c1092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.102+0000 7fb9a1299640 1 --2- 172.21.15.70:0/2875867148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb99c072b60 0x7fb99c072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.102+0000 7fb9a1299640 1 -- 172.21.15.70:0/2875867148 >> 172.21.15.70:0/2875867148 conn(0x7fb99c06e630 msgr2=0x7fb99c070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:32.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.104+0000 7fb9a1299640 1 -- 172.21.15.70:0/2875867148 shutdown_connections 2024-04-01T01:36:32.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.104+0000 7fb9a1299640 1 -- 172.21.15.70:0/2875867148 wait complete. 2024-04-01T01:36:32.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.105+0000 7fb9a1299640 1 Processor -- start 2024-04-01T01:36:32.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.105+0000 7fb9a1299640 1 -- start start 2024-04-01T01:36:32.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.106+0000 7fb9a1299640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb99c072b60 0x7fb99c1a6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.106+0000 7fb9a1299640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb99c108e20 0x7fb99c1a6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.106+0000 7fb9a1299640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb99c1097e0 0x7fb99c1adc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.106+0000 7fb9a1299640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c11c000 con 0x7fb99c108e20 2024-04-01T01:36:32.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.106+0000 7fb9a1299640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c11be80 con 0x7fb99c1097e0 2024-04-01T01:36:32.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.106+0000 7fb9a1299640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c11c180 con 0x7fb99c072b60 2024-04-01T01:36:32.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.106+0000 7fb9a0a98640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb99c1097e0 0x7fb99c1adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.106+0000 7fb9a0a98640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb99c1097e0 0x7fb99c1adc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35672/0 (socket says 172.21.15.70:35672) 2024-04-01T01:36:32.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.106+0000 7fb9a0a98640 1 -- 172.21.15.70:0/1428965797 learned_addr learned my addr 172.21.15.70:0/1428965797 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:32.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.106+0000 7fb99bfff640 1 --2- 172.21.15.70:0/1428965797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb99c072b60 0x7fb99c1a6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.106+0000 7fb9a0a98640 1 -- 172.21.15.70:0/1428965797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb99c072b60 msgr2=0x7fb99c1a6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.106+0000 7fb9a0a98640 1 --2- 172.21.15.70:0/1428965797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb99c072b60 0x7fb99c1a6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.106+0000 7fb9a0a98640 1 -- 172.21.15.70:0/1428965797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb99c108e20 msgr2=0x7fb99c1a6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.106+0000 7fb9a0a98640 1 --2- 172.21.15.70:0/1428965797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb99c108e20 0x7fb99c1a6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.106+0000 7fb9a0a98640 1 -- 172.21.15.70:0/1428965797 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb99c1ae290 con 0x7fb99c1097e0 2024-04-01T01:36:32.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.107+0000 7fb9a0a98640 1 --2- 172.21.15.70:0/1428965797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb99c1097e0 0x7fb99c1adc20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb98c02c1a0 tx=0x7fb98c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:32.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.107+0000 7fb9997fa640 1 -- 172.21.15.70:0/1428965797 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c043020 con 0x7fb99c1097e0 2024-04-01T01:36:32.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.107+0000 7fb9997fa640 1 -- 172.21.15.70:0/1428965797 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb98c0052b0 con 0x7fb99c1097e0 2024-04-01T01:36:32.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.108+0000 7fb9997fa640 1 -- 172.21.15.70:0/1428965797 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c003b90 con 0x7fb99c1097e0 2024-04-01T01:36:32.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.108+0000 7fb9a1299640 1 -- 172.21.15.70:0/1428965797 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb99c1ae520 con 0x7fb99c1097e0 2024-04-01T01:36:32.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.108+0000 7fb9a1299640 1 -- 172.21.15.70:0/1428965797 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb99c1ae9f0 con 0x7fb99c1097e0 2024-04-01T01:36:32.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.112+0000 7fb9a51ba740 1 Processor -- start 2024-04-01T01:36:32.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.113+0000 7fb9997fa640 1 -- 172.21.15.70:0/1428965797 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb98c046020 con 0x7fb99c1097e0 2024-04-01T01:36:32.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.113+0000 7fb9a51ba740 1 -- start start 2024-04-01T01:36:32.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.113+0000 7fb9a51ba740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e0bf0 0x5608734e1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.114+0000 7fb9a51ba740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e15e0 0x5608734e60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.114+0000 7fb9a51ba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e6750 0x5608734e8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.114+0000 7fb9a51ba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5608733e20b0 con 0x5608734e6750 2024-04-01T01:36:32.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.114+0000 7fb9a51ba740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560873444ea0 con 0x5608734e15e0 2024-04-01T01:36:32.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.114+0000 7fb9a51ba740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5608733edc00 con 0x5608734e0bf0 2024-04-01T01:36:32.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.114+0000 7fb99bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e15e0 0x5608734e60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.114+0000 7fb99bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e15e0 0x5608734e60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35680/0 (socket says 172.21.15.70:35680) 2024-04-01T01:36:32.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.114+0000 7fb99bfff640 1 -- 172.21.15.70:0/2251690241 learned_addr learned my addr 172.21.15.70:0/2251690241 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:32.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.114+0000 7fb99bfff640 1 -- 172.21.15.70:0/2251690241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e0bf0 msgr2=0x5608734e1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:32.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.114+0000 7fb99b7fe640 1 --2- 172.21.15.70:0/2251690241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e0bf0 0x5608734e1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.114+0000 7fb99bfff640 1 --2- 172.21.15.70:0/2251690241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e0bf0 0x5608734e1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.114+0000 7fb99bfff640 1 -- 172.21.15.70:0/2251690241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e6750 msgr2=0x5608734e8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:32.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.114+0000 7fb99bfff640 1 --2- 172.21.15.70:0/2251690241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e6750 0x5608734e8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.114+0000 7fb99bfff640 1 -- 172.21.15.70:0/2251690241 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608732217e0 con 0x5608734e15e0 2024-04-01T01:36:32.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.115+0000 7fb99b7fe640 1 --2- 172.21.15.70:0/2251690241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e0bf0 0x5608734e1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:36:32.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.115+0000 7fb99bfff640 1 --2- 172.21.15.70:0/2251690241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e15e0 0x5608734e60e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb9940103d0 tx=0x7fb994010890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d983ad03c19c7ec3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:32.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.115+0000 7fb9997fa640 1 --2- 172.21.15.70:0/1428965797 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9800a2380 0x7fb9800a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.115+0000 7fb9927fc640 1 -- 172.21.15.70:0/2251690241 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb994011700 con 0x5608734e15e0 2024-04-01T01:36:32.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.115+0000 7fb9927fc640 1 -- 172.21.15.70:0/2251690241 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb994011d50 con 0x5608734e15e0 2024-04-01T01:36:32.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.116+0000 7fb9997fa640 1 -- 172.21.15.70:0/1428965797 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb98c0f0870 con 0x7fb99c1097e0 2024-04-01T01:36:32.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.116+0000 7fb9927fc640 1 -- 172.21.15.70:0/2251690241 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99400d340 con 0x5608734e15e0 2024-04-01T01:36:32.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.116+0000 7fb99b7fe640 1 --2- 172.21.15.70:0/1428965797 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9800a2380 0x7fb9800a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.116+0000 7fb9a51ba740 1 -- 172.21.15.70:0/2251690241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e15e0 msgr2=0x5608734e60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.116+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/2251690241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e15e0 0x5608734e60e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb9940103d0 tx=0x7fb994010890 comp rx=0 tx=0).stop 2024-04-01T01:36:32.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.117+0000 7fb9a51ba740 1 -- 172.21.15.70:0/2251690241 shutdown_connections 2024-04-01T01:36:32.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.117+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/2251690241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e6750 0x5608734e8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.117+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/2251690241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e15e0 0x5608734e60e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.117+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/2251690241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e0bf0 0x5608734e1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.117+0000 7fb9a51ba740 1 -- 172.21.15.70:0/2251690241 >> 172.21.15.70:0/2251690241 conn(0x5608734dc160 msgr2=0x5608734dc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:32.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.117+0000 7fb9a51ba740 1 -- 172.21.15.70:0/2251690241 shutdown_connections 2024-04-01T01:36:32.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.118+0000 7fb9a51ba740 1 -- 172.21.15.70:0/2251690241 wait complete. 2024-04-01T01:36:32.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.118+0000 7fb9a51ba740 1 -- start start 2024-04-01T01:36:32.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.119+0000 7fb9a51ba740 1 Processor -- start 2024-04-01T01:36:32.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.119+0000 7fb9a51ba740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e0bf0 0x5608734eb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.119+0000 7fb9a51ba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e15e0 0x5608734eb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.119+0000 7fb9a51ba740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e6750 0x5608734f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.119+0000 7fb9a51ba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5608733e33d0 con 0x5608734e15e0 2024-04-01T01:36:32.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.119+0000 7fb9a51ba740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560873440f40 con 0x5608734e0bf0 2024-04-01T01:36:32.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.119+0000 7fb9a51ba740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5608734e9750 con 0x5608734e6750 2024-04-01T01:36:32.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.119+0000 7fb9a51ba740 10 client.0 _ll_register_callbacks cb 0x560873400b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:32.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.119+0000 7fb9a0a98640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e6750 0x5608734f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.119+0000 7fb99bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e0bf0 0x5608734eb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.120+0000 7fb99bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e0bf0 0x5608734eb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35696/0 (socket says 172.21.15.70:35696) 2024-04-01T01:36:32.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.120+0000 7fb99bfff640 1 -- 172.21.15.70:0/1599831868 learned_addr learned my addr 172.21.15.70:0/1599831868 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:32.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.120+0000 7fb99b7fe640 1 --2- 172.21.15.70:0/1599831868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e15e0 0x5608734eb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.120+0000 7fb99b7fe640 1 --2- 172.21.15.70:0/1428965797 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9800a2380 0x7fb9800a4840 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x5608734e1e40 tx=0x7fb988004a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:32.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.120+0000 7fb99bfff640 1 -- 172.21.15.70:0/1599831868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e6750 msgr2=0x5608734f25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:32.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.120+0000 7fb99bfff640 1 --2- 172.21.15.70:0/1599831868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e6750 0x5608734f25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.120+0000 7fb99bfff640 1 -- 172.21.15.70:0/1599831868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e15e0 msgr2=0x5608734eb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:32.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.120+0000 7fb99bfff640 1 --2- 172.21.15.70:0/1599831868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e15e0 0x5608734eb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.120+0000 7fb99bfff640 1 -- 172.21.15.70:0/1599831868 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560873281f70 con 0x5608734e0bf0 2024-04-01T01:36:32.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.120+0000 7fb9a0a98640 1 --2- 172.21.15.70:0/1599831868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e6750 0x5608734f25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:36:32.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.120+0000 7fb99bfff640 1 --2- 172.21.15.70:0/1599831868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e0bf0 0x5608734eb020 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb99400f010 tx=0x7fb99400e5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0f4cd63cbe03d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:32.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.120+0000 7fb97f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:36:32.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.120+0000 7fb97f7fe640 1 -- 172.21.15.70:0/1599831868 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99402a060 con 0x5608734e0bf0 2024-04-01T01:36:32.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.120+0000 7fb9a51ba740 1 -- 172.21.15.70:0/1599831868 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb994011d50 con 0x5608734e0bf0 2024-04-01T01:36:32.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.120+0000 7fb97f7fe640 1 -- 172.21.15.70:0/1599831868 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb994011500 con 0x5608734e0bf0 2024-04-01T01:36:32.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.121+0000 7fb97f7fe640 1 -- 172.21.15.70:0/1599831868 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb994031050 con 0x5608734e0bf0 2024-04-01T01:36:32.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.121+0000 7fb9a51ba740 10 client.26393 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:32.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.121+0000 7fb9a51ba740 1 -- 172.21.15.70:0/1599831868 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560873119170 con 0x5608734e0bf0 2024-04-01T01:36:32.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.121+0000 7fb97f7fe640 1 -- 172.21.15.70:0/1599831868 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb994011d50 con 0x5608734e0bf0 2024-04-01T01:36:32.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.121+0000 7fb97f7fe640 1 -- 172.21.15.70:0/1599831868 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb994005470 con 0x5608734e0bf0 2024-04-01T01:36:32.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.121+0000 7fb9a51ba740 10 client.26393 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:32.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.121+0000 7fb9a51ba740 10 client.26393 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:36:32.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.121+0000 7fb9a51ba740 20 client.26393 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:32.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.121+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/1599831868 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5608734f6670 0x5608734f8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.121+0000 7fb9a51ba740 4 client.26393 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:36:32.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.121+0000 7fb9a51ba740 1 -- 172.21.15.70:0/1599831868 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608734523e0 con 0x5608734f6670 2024-04-01T01:36:32.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.122+0000 7fb99b7fe640 1 --2- 172.21.15.70:0/1599831868 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5608734f6670 0x5608734f8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.123+0000 7fb97f7fe640 10 client.26393 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:32.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.123+0000 7fb99b7fe640 1 --2- 172.21.15.70:0/1599831868 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5608734f6670 0x5608734f8a60 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9eb264c7294f319 server_cookie=21fb9e9194e0f0b7 in_seq=0 out_seq=0 2024-04-01T01:36:32.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.124+0000 7fb97f7fe640 1 -- 172.21.15.70:0/1599831868 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5608734523e0 con 0x5608734f6670 2024-04-01T01:36:32.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.124+0000 7fb97f7fe640 10 client.26393 handle_command_reply: tid=0 2024-04-01T01:36:32.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.124+0000 7fb9a51ba740 1 client.26393 shutdown 2024-04-01T01:36:32.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.124+0000 7fb9a51ba740 1 -- 172.21.15.70:0/1599831868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e0bf0 msgr2=0x5608734eb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.124+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/1599831868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e0bf0 0x5608734eb020 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb99400f010 tx=0x7fb99400e5f0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.124+0000 7fb9a51ba740 1 -- 172.21.15.70:0/1599831868 shutdown_connections 2024-04-01T01:36:32.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.124+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/1599831868 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5608734f6670 0x5608734f8a60 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.124+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/1599831868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e6750 0x5608734f25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.124+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/1599831868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e15e0 0x5608734eb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.124+0000 7fb97f7fe640 0 client.26393 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:32.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.124+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/1599831868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e0bf0 0x5608734eb020 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.124+0000 7fb9a51ba740 1 -- 172.21.15.70:0/1599831868 >> 172.21.15.70:0/1599831868 conn(0x5608734dc160 msgr2=0x5608734db2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:32.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.124+0000 7fb9a51ba740 1 -- 172.21.15.70:0/1599831868 shutdown_connections 2024-04-01T01:36:32.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.124+0000 7fb9a51ba740 1 -- 172.21.15.70:0/1599831868 wait complete. 2024-04-01T01:36:32.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.124+0000 7fb9a51ba740 20 client.26393 trim_cache size 0 max 16384 2024-04-01T01:36:32.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.158+0000 7fb9a51ba740 1 Processor -- start 2024-04-01T01:36:32.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.158+0000 7fb9a51ba740 1 -- start start 2024-04-01T01:36:32.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.159+0000 7fb9a51ba740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e0bf0 0x5608734ea660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.159+0000 7fb9a51ba740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e15e0 0x5608734f2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.159+0000 7fb9a51ba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e6750 0x5608734eaba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.159+0000 7fb99bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e0bf0 0x5608734ea660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.159+0000 7fb9a51ba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c1b8e80 con 0x5608734e6750 2024-04-01T01:36:32.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.159+0000 7fb9a51ba740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c1ba9c0 con 0x5608734e0bf0 2024-04-01T01:36:32.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.159+0000 7fb9a51ba740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c1b9550 con 0x5608734e15e0 2024-04-01T01:36:32.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.159+0000 7fb99bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e0bf0 0x5608734ea660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35700/0 (socket says 172.21.15.70:35700) 2024-04-01T01:36:32.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.159+0000 7fb99bfff640 1 -- 172.21.15.70:0/3220253434 learned_addr learned my addr 172.21.15.70:0/3220253434 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:32.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.159+0000 7fb99b7fe640 1 --2- 172.21.15.70:0/3220253434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e15e0 0x5608734f2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.160+0000 7fb99bfff640 1 -- 172.21.15.70:0/3220253434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e15e0 msgr2=0x5608734f2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:32.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.160+0000 7fb99bfff640 1 --2- 172.21.15.70:0/3220253434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e15e0 0x5608734f2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.160+0000 7fb99bfff640 1 -- 172.21.15.70:0/3220253434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e6750 msgr2=0x5608734eaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:32.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.160+0000 7fb99bfff640 1 --2- 172.21.15.70:0/3220253434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e6750 0x5608734eaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.160+0000 7fb99bfff640 1 -- 172.21.15.70:0/3220253434 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608734e95e0 con 0x5608734e0bf0 2024-04-01T01:36:32.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.160+0000 7fb99bfff640 1 --2- 172.21.15.70:0/3220253434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e0bf0 0x5608734ea660 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb9940103d0 tx=0x7fb994025670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=289d934adfeca1a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:32.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.160+0000 7fb991ffb640 1 -- 172.21.15.70:0/3220253434 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99402a040 con 0x5608734e0bf0 2024-04-01T01:36:32.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.160+0000 7fb991ffb640 1 -- 172.21.15.70:0/3220253434 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb994025c80 con 0x5608734e0bf0 2024-04-01T01:36:32.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.161+0000 7fb9a51ba740 1 -- 172.21.15.70:0/3220253434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e0bf0 msgr2=0x5608734ea660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.161+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/3220253434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e0bf0 0x5608734ea660 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb9940103d0 tx=0x7fb994025670 comp rx=0 tx=0).stop 2024-04-01T01:36:32.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.162+0000 7fb9a51ba740 1 -- 172.21.15.70:0/3220253434 shutdown_connections 2024-04-01T01:36:32.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.162+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/3220253434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e6750 0x5608734eaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.162+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/3220253434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e15e0 0x5608734f2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.162+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/3220253434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e0bf0 0x5608734ea660 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.162+0000 7fb9a51ba740 1 -- 172.21.15.70:0/3220253434 >> 172.21.15.70:0/3220253434 conn(0x5608734dc160 msgr2=0x5608734db2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:32.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.163+0000 7fb9a51ba740 1 -- 172.21.15.70:0/3220253434 shutdown_connections 2024-04-01T01:36:32.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.163+0000 7fb9a51ba740 1 -- 172.21.15.70:0/3220253434 wait complete. 2024-04-01T01:36:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.163+0000 7fb9a51ba740 1 -- start start 2024-04-01T01:36:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.163+0000 7fb9a51ba740 1 Processor -- start 2024-04-01T01:36:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.164+0000 7fb9a51ba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e0bf0 0x5608734efed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.164+0000 7fb9a51ba740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e15e0 0x5608734f0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.164+0000 7fb9a51ba740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e6750 0x5608734f1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.164+0000 7fb9a51ba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c1ba2f0 con 0x5608734e0bf0 2024-04-01T01:36:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.164+0000 7fb9a51ba740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c1b9c20 con 0x5608734e6750 2024-04-01T01:36:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.164+0000 7fb9a51ba740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c1b6e60 con 0x5608734e15e0 2024-04-01T01:36:32.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.164+0000 7fb9a51ba740 10 client.0 _ll_register_callbacks cb 0x7fb994005bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:32.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.164+0000 7fb9a0a98640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e6750 0x5608734f1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.164+0000 7fb9a0a98640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e6750 0x5608734f1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35704/0 (socket says 172.21.15.70:35704) 2024-04-01T01:36:32.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.164+0000 7fb9a0a98640 1 -- 172.21.15.70:0/2176689938 learned_addr learned my addr 172.21.15.70:0/2176689938 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:32.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.164+0000 7fb99bfff640 1 --2- 172.21.15.70:0/2176689938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e0bf0 0x5608734efed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.164+0000 7fb9a0a98640 1 -- 172.21.15.70:0/2176689938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e15e0 msgr2=0x5608734f0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:32.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.164+0000 7fb9a0a98640 1 --2- 172.21.15.70:0/2176689938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e15e0 0x5608734f0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.164+0000 7fb9a0a98640 1 -- 172.21.15.70:0/2176689938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e0bf0 msgr2=0x5608734efed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:32.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.164+0000 7fb9a0a98640 1 --2- 172.21.15.70:0/2176689938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e0bf0 0x5608734efed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.164+0000 7fb9a0a98640 1 -- 172.21.15.70:0/2176689938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608734f1a10 con 0x5608734e6750 2024-04-01T01:36:32.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.165+0000 7fb99bfff640 1 --2- 172.21.15.70:0/2176689938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e0bf0 0x5608734efed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:36:32.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.165+0000 7fb9a0a98640 1 --2- 172.21.15.70:0/2176689938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e6750 0x5608734f1420 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb98c0094b0 tx=0x7fb98c0b9260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67f3b2af40e2e675 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:32.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.165+0000 7fb97f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:36:32.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.166+0000 7fb97f7fe640 1 -- 172.21.15.70:0/2176689938 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c0da060 con 0x5608734e6750 2024-04-01T01:36:32.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.166+0000 7fb97f7fe640 1 -- 172.21.15.70:0/2176689938 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb98c0bd160 con 0x5608734e6750 2024-04-01T01:36:32.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.166+0000 7fb9a51ba740 1 -- 172.21.15.70:0/2176689938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb994025c80 con 0x5608734e6750 2024-04-01T01:36:32.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.167+0000 7fb97f7fe640 1 -- 172.21.15.70:0/2176689938 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb98c0c1050 con 0x5608734e6750 2024-04-01T01:36:32.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.168+0000 7fb9a51ba740 10 client.26399 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:32.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.168+0000 7fb9a51ba740 1 -- 172.21.15.70:0/2176689938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56087355cc70 con 0x5608734e6750 2024-04-01T01:36:32.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.169+0000 7fb97f7fe640 1 -- 172.21.15.70:0/2176689938 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb994025c80 con 0x5608734e6750 2024-04-01T01:36:32.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.169+0000 7fb97f7fe640 1 -- 172.21.15.70:0/2176689938 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb98c0b8c50 con 0x5608734e6750 2024-04-01T01:36:32.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.169+0000 7fb9a51ba740 10 client.26399 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:32.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.169+0000 7fb9a51ba740 10 client.26399 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:36:32.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.169+0000 7fb9a51ba740 20 client.26399 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:32.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.169+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/2176689938 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56087355d5d0 0x56087355f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.169+0000 7fb9a51ba740 4 client.26399 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:36:32.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.169+0000 7fb9a51ba740 1 -- 172.21.15.70:0/2176689938 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56087344c730 con 0x56087355d5d0 2024-04-01T01:36:32.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.169+0000 7fb99bfff640 1 --2- 172.21.15.70:0/2176689938 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56087355d5d0 0x56087355f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.170+0000 7fb99bfff640 1 --2- 172.21.15.70:0/2176689938 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56087355d5d0 0x56087355f9f0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=768d62031e079fb7 server_cookie=47b8d3c94fce6344 in_seq=0 out_seq=0 2024-04-01T01:36:32.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.170+0000 7fb97f7fe640 10 client.26399 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:32.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.172+0000 7fb97f7fe640 1 -- 172.21.15.70:0/2176689938 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56087344c730 con 0x56087355d5d0 2024-04-01T01:36:32.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.172+0000 7fb97f7fe640 10 client.26399 handle_command_reply: tid=0 2024-04-01T01:36:32.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.172+0000 7fb9a51ba740 1 client.26399 shutdown 2024-04-01T01:36:32.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.172+0000 7fb9a51ba740 1 -- 172.21.15.70:0/2176689938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e6750 msgr2=0x5608734f1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.172+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/2176689938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e6750 0x5608734f1420 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb98c0094b0 tx=0x7fb98c0b9260 comp rx=0 tx=0).stop 2024-04-01T01:36:32.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.172+0000 7fb9a51ba740 1 -- 172.21.15.70:0/2176689938 shutdown_connections 2024-04-01T01:36:32.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.172+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/2176689938 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56087355d5d0 0x56087355f9f0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.172+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/2176689938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608734e6750 0x5608734f1420 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.172+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/2176689938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608734e15e0 0x5608734f0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.172+0000 7fb9a51ba740 1 --2- 172.21.15.70:0/2176689938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608734e0bf0 0x5608734efed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.172+0000 7fb9a51ba740 1 -- 172.21.15.70:0/2176689938 >> 172.21.15.70:0/2176689938 conn(0x5608734dc160 msgr2=0x5608734df3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:32.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.172+0000 7fb97f7fe640 0 client.26399 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:32.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.174+0000 7fb9a51ba740 1 -- 172.21.15.70:0/2176689938 shutdown_connections 2024-04-01T01:36:32.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.178+0000 7fb9a51ba740 1 -- 172.21.15.70:0/2176689938 wait complete. 2024-04-01T01:36:32.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.180+0000 7fb9a51ba740 20 client.26399 trim_cache size 0 max 16384 2024-04-01T01:36:32.180 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:36:32.180 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:36:32.180 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "4f67cf82-3341-43cd-916e-4592e3fd6586", 2024-04-01T01:36:32.180 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:36:32.180 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:36:32.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.181+0000 7fb9a1299640 1 -- 172.21.15.70:0/1428965797 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9800a2380 msgr2=0x7fb9800a4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.181+0000 7fb9a1299640 1 --2- 172.21.15.70:0/1428965797 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9800a2380 0x7fb9800a4840 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x5608734e1e40 tx=0x7fb988004a20 comp rx=0 tx=0).stop 2024-04-01T01:36:32.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.181+0000 7fb9a1299640 1 -- 172.21.15.70:0/1428965797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb99c1097e0 msgr2=0x7fb99c1adc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.181+0000 7fb9a1299640 1 --2- 172.21.15.70:0/1428965797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb99c1097e0 0x7fb99c1adc20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb98c02c1a0 tx=0x7fb98c004270 comp rx=0 tx=0).stop 2024-04-01T01:36:32.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.181+0000 7fb9a1299640 1 -- 172.21.15.70:0/1428965797 shutdown_connections 2024-04-01T01:36:32.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.181+0000 7fb9a1299640 1 --2- 172.21.15.70:0/1428965797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb99c1097e0 0x7fb99c1adc20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.181+0000 7fb9a1299640 1 --2- 172.21.15.70:0/1428965797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb99c108e20 0x7fb99c1a6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.181+0000 7fb9a1299640 1 --2- 172.21.15.70:0/1428965797 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9800a2380 0x7fb9800a4840 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.181+0000 7fb9a1299640 1 --2- 172.21.15.70:0/1428965797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb99c072b60 0x7fb99c1a6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.181+0000 7fb9a1299640 1 -- 172.21.15.70:0/1428965797 >> 172.21.15.70:0/1428965797 conn(0x7fb99c06e630 msgr2=0x7fb99c0702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:32.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.181+0000 7fb9a1299640 1 -- 172.21.15.70:0/1428965797 shutdown_connections 2024-04-01T01:36:32.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.182+0000 7fb9a1299640 1 -- 172.21.15.70:0/1428965797 wait complete. 2024-04-01T01:36:32.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:36:32.200 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:36:32.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.377+0000 7f6acb9f0640 1 -- 172.21.15.70:0/276721420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4101760 msgr2=0x7f6ac4101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.377+0000 7f6acb9f0640 1 --2- 172.21.15.70:0/276721420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4101760 0x7f6ac4101b80 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f6abc009ca0 tx=0x7f6abc02c100 comp rx=0 tx=0).stop 2024-04-01T01:36:32.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.378+0000 7f6acb9f0640 1 -- 172.21.15.70:0/276721420 shutdown_connections 2024-04-01T01:36:32.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.378+0000 7f6acb9f0640 1 --2- 172.21.15.70:0/276721420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac4103310 0x7f6ac4109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.378+0000 7f6acb9f0640 1 --2- 172.21.15.70:0/276721420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac4102950 0x7f6ac4102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.378+0000 7f6acb9f0640 1 --2- 172.21.15.70:0/276721420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4101760 0x7f6ac4101b80 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.378+0000 7f6acb9f0640 1 -- 172.21.15.70:0/276721420 >> 172.21.15.70:0/276721420 conn(0x7f6ac40fccf0 msgr2=0x7f6ac40ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:32.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.378+0000 7f6acb9f0640 1 -- 172.21.15.70:0/276721420 shutdown_connections 2024-04-01T01:36:32.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.378+0000 7f6acb9f0640 1 -- 172.21.15.70:0/276721420 wait complete. 2024-04-01T01:36:32.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.379+0000 7f6acb9f0640 1 Processor -- start 2024-04-01T01:36:32.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.379+0000 7f6acb9f0640 1 -- start start 2024-04-01T01:36:32.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.379+0000 7f6acb9f0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac4101760 0x7f6ac419ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.379+0000 7f6acb9f0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac4102950 0x7f6ac419e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.380+0000 7f6ac978c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac4101760 0x7f6ac419ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.380+0000 7f6acb9f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4103310 0x7f6ac41a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.380+0000 7f6ac978c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac4101760 0x7f6ac419ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35716/0 (socket says 172.21.15.70:35716) 2024-04-01T01:36:32.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.380+0000 7f6ac978c640 1 -- 172.21.15.70:0/737881826 learned_addr learned my addr 172.21.15.70:0/737881826 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:32.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.380+0000 7f6acb9f0640 1 -- 172.21.15.70:0/737881826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac4074ce0 con 0x7f6ac4103310 2024-04-01T01:36:32.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.380+0000 7f6acb9f0640 1 -- 172.21.15.70:0/737881826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac4074b60 con 0x7f6ac4101760 2024-04-01T01:36:32.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.380+0000 7f6acb9f0640 1 -- 172.21.15.70:0/737881826 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac4074e60 con 0x7f6ac4102950 2024-04-01T01:36:32.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.380+0000 7f6ac9f8d640 1 --2- 172.21.15.70:0/737881826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4103310 0x7f6ac41a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.380+0000 7f6ac8f8b640 1 --2- 172.21.15.70:0/737881826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac4102950 0x7f6ac419e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.381+0000 7f6ac9f8d640 1 -- 172.21.15.70:0/737881826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac4102950 msgr2=0x7f6ac419e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.381+0000 7f6ac9f8d640 1 --2- 172.21.15.70:0/737881826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac4102950 0x7f6ac419e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.381+0000 7f6ac9f8d640 1 -- 172.21.15.70:0/737881826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac4101760 msgr2=0x7f6ac419ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.381+0000 7f6ac9f8d640 1 --2- 172.21.15.70:0/737881826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac4101760 0x7f6ac419ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.381+0000 7f6ac9f8d640 1 -- 172.21.15.70:0/737881826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac41a59e0 con 0x7f6ac4103310 2024-04-01T01:36:32.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.381+0000 7f6ac8f8b640 1 --2- 172.21.15.70:0/737881826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac4102950 0x7f6ac419e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:36:32.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.381+0000 7f6ac9f8d640 1 --2- 172.21.15.70:0/737881826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4103310 0x7f6ac41a5370 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400d680 tx=0x7f6ab400db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:32.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.381+0000 7f6aba7fc640 1 -- 172.21.15.70:0/737881826 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4017070 con 0x7f6ac4103310 2024-04-01T01:36:32.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.381+0000 7f6aba7fc640 1 -- 172.21.15.70:0/737881826 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ab4009820 con 0x7f6ac4103310 2024-04-01T01:36:32.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.381+0000 7f6aba7fc640 1 -- 172.21.15.70:0/737881826 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4004cf0 con 0x7f6ac4103310 2024-04-01T01:36:32.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.381+0000 7f6acb9f0640 1 -- 172.21.15.70:0/737881826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac41a5cd0 con 0x7f6ac4103310 2024-04-01T01:36:32.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.382+0000 7f6acb9f0640 1 -- 172.21.15.70:0/737881826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac41a61d0 con 0x7f6ac4103310 2024-04-01T01:36:32.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.385+0000 7f6ace5cd740 1 Processor -- start 2024-04-01T01:36:32.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.386+0000 7f6aba7fc640 1 -- 172.21.15.70:0/737881826 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f6ab40099d0 con 0x7f6ac4103310 2024-04-01T01:36:32.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.386+0000 7f6ace5cd740 1 -- start start 2024-04-01T01:36:32.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.386+0000 7f6ace5cd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab8482b0 0x55acab8486d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.386+0000 7f6ace5cd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab848ca0 0x55acab84d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.386+0000 7f6ace5cd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab84de10 0x55acab8502a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.386+0000 7f6ace5cd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55acab7ad180 con 0x55acab84de10 2024-04-01T01:36:32.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.386+0000 7f6ace5cd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55acab755b70 con 0x55acab848ca0 2024-04-01T01:36:32.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.386+0000 7f6ace5cd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55acab7a9220 con 0x55acab8482b0 2024-04-01T01:36:32.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.387+0000 7f6ac978c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab848ca0 0x55acab84d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.387+0000 7f6ac978c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab848ca0 0x55acab84d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35732/0 (socket says 172.21.15.70:35732) 2024-04-01T01:36:32.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.387+0000 7f6ac978c640 1 -- 172.21.15.70:0/3394489646 learned_addr learned my addr 172.21.15.70:0/3394489646 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:32.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.387+0000 7f6ac8f8b640 1 --2- 172.21.15.70:0/3394489646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab8482b0 0x55acab8486d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.387+0000 7f6ac8f8b640 1 --2- 172.21.15.70:0/3394489646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab84de10 0x55acab8502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.387+0000 7f6ac978c640 1 -- 172.21.15.70:0/3394489646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab8482b0 msgr2=0x55acab8486d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:32.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.387+0000 7f6ac978c640 1 --2- 172.21.15.70:0/3394489646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab8482b0 0x55acab8486d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.387+0000 7f6ac978c640 1 -- 172.21.15.70:0/3394489646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab84de10 msgr2=0x55acab8502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:32.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.387+0000 7f6ac978c640 1 --2- 172.21.15.70:0/3394489646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab84de10 0x55acab8502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.387+0000 7f6ac978c640 1 -- 172.21.15.70:0/3394489646 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acab7697c0 con 0x55acab848ca0 2024-04-01T01:36:32.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.387+0000 7f6aba7fc640 1 --2- 172.21.15.70:0/737881826 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6ab00a2520 0x7f6ab00a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.387+0000 7f6ac978c640 1 --2- 172.21.15.70:0/3394489646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab848ca0 0x55acab84d7a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6abc009880 tx=0x7f6abc0098b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a04a29a9c8c4791a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:32.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.388+0000 7f6aba7fc640 1 -- 172.21.15.70:0/737881826 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6ab40ce2f0 con 0x7f6ac4103310 2024-04-01T01:36:32.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.388+0000 7f6aaf7fe640 1 -- 172.21.15.70:0/3394489646 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6abc043070 con 0x55acab848ca0 2024-04-01T01:36:32.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.388+0000 7f6aaf7fe640 1 -- 172.21.15.70:0/3394489646 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6abc003970 con 0x55acab848ca0 2024-04-01T01:36:32.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.388+0000 7f6aaf7fe640 1 -- 172.21.15.70:0/3394489646 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6abc039040 con 0x55acab848ca0 2024-04-01T01:36:32.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.388+0000 7f6ac8f8b640 1 --2- 172.21.15.70:0/737881826 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6ab00a2520 0x7f6ab00a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.389+0000 7f6ace5cd740 1 -- 172.21.15.70:0/3394489646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab848ca0 msgr2=0x55acab84d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.389+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/3394489646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab848ca0 0x55acab84d7a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6abc009880 tx=0x7f6abc0098b0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.389+0000 7f6ac8f8b640 1 --2- 172.21.15.70:0/737881826 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6ab00a2520 0x7f6ab00a49e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x55acab849500 tx=0x7f6ac000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:32.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.389+0000 7f6ace5cd740 1 -- 172.21.15.70:0/3394489646 shutdown_connections 2024-04-01T01:36:32.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.389+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/3394489646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab84de10 0x55acab8502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.389+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/3394489646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab848ca0 0x55acab84d7a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.389+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/3394489646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab8482b0 0x55acab8486d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.389+0000 7f6ace5cd740 1 -- 172.21.15.70:0/3394489646 >> 172.21.15.70:0/3394489646 conn(0x55acab843820 msgr2=0x55acab843c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:32.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.391+0000 7f6ace5cd740 1 -- 172.21.15.70:0/3394489646 shutdown_connections 2024-04-01T01:36:32.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.393+0000 7f6ace5cd740 1 -- 172.21.15.70:0/3394489646 wait complete. 2024-04-01T01:36:32.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.393+0000 7f6ace5cd740 1 -- start start 2024-04-01T01:36:32.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.393+0000 7f6ace5cd740 1 Processor -- start 2024-04-01T01:36:32.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.394+0000 7f6ace5cd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab8482b0 0x55acab852830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.394+0000 7f6ace5cd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab848ca0 0x55acab852d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.394+0000 7f6ace5cd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab84de10 0x55acab859db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.394+0000 7f6ace5cd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55acab850d60 con 0x55acab848ca0 2024-04-01T01:36:32.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.394+0000 7f6ace5cd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55acab74b4a0 con 0x55acab84de10 2024-04-01T01:36:32.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.394+0000 7f6ace5cd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55acab850ee0 con 0x55acab8482b0 2024-04-01T01:36:32.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.394+0000 7f6ace5cd740 10 client.0 _ll_register_callbacks cb 0x55acab74ed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:32.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.394+0000 7f6ac978c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab8482b0 0x55acab852830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.394+0000 7f6ac978c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab8482b0 0x55acab852830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:38156/0 (socket says 172.21.15.70:38156) 2024-04-01T01:36:32.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.394+0000 7f6ac978c640 1 -- 172.21.15.70:0/565201896 learned_addr learned my addr 172.21.15.70:0/565201896 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:32.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.394+0000 7f6ac9f8d640 1 --2- 172.21.15.70:0/565201896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab84de10 0x55acab859db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.394+0000 7f6ac8f8b640 1 --2- 172.21.15.70:0/565201896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab848ca0 0x55acab852d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.395+0000 7f6ac8f8b640 1 -- 172.21.15.70:0/565201896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab8482b0 msgr2=0x55acab852830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:32.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.395+0000 7f6ac8f8b640 1 --2- 172.21.15.70:0/565201896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab8482b0 0x55acab852830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.395+0000 7f6ac8f8b640 1 -- 172.21.15.70:0/565201896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab84de10 msgr2=0x55acab859db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:32.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.395+0000 7f6ac8f8b640 1 --2- 172.21.15.70:0/565201896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab84de10 0x55acab859db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.395+0000 7f6ac8f8b640 1 -- 172.21.15.70:0/565201896 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acab58b7d0 con 0x55acab848ca0 2024-04-01T01:36:32.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.395+0000 7f6ac9f8d640 1 --2- 172.21.15.70:0/565201896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab84de10 0x55acab859db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:36:32.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.395+0000 7f6ac8f8b640 1 --2- 172.21.15.70:0/565201896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab848ca0 0x55acab852d70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00125a0 tx=0x7f6ac00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b682426b6d7ce6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:32.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.395+0000 7f6aacff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:36:32.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.395+0000 7f6aacff9640 1 -- 172.21.15.70:0/565201896 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0010b90 con 0x55acab848ca0 2024-04-01T01:36:32.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.395+0000 7f6aacff9640 1 -- 172.21.15.70:0/565201896 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6ac0010d30 con 0x55acab848ca0 2024-04-01T01:36:32.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.396+0000 7f6ace5cd740 1 -- 172.21.15.70:0/565201896 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6abc003970 con 0x55acab848ca0 2024-04-01T01:36:32.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.396+0000 7f6aacff9640 1 -- 172.21.15.70:0/565201896 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6ac001f480 con 0x55acab848ca0 2024-04-01T01:36:32.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.396+0000 7f6ace5cd740 10 client.16560 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:32.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.396+0000 7f6ace5cd740 1 -- 172.21.15.70:0/565201896 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55acab4c13f0 con 0x55acab848ca0 2024-04-01T01:36:32.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.396+0000 7f6aacff9640 1 -- 172.21.15.70:0/565201896 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6abc003970 con 0x55acab848ca0 2024-04-01T01:36:32.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.396+0000 7f6aacff9640 1 -- 172.21.15.70:0/565201896 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6ac0028070 con 0x55acab848ca0 2024-04-01T01:36:32.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.397+0000 7f6ace5cd740 10 client.16560 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:32.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.397+0000 7f6ace5cd740 10 client.16560 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:36:32.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.397+0000 7f6ace5cd740 20 client.16560 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:32.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.397+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/565201896 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55acab85e6a0 0x55acab860b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.397+0000 7f6ace5cd740 4 client.16560 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:36:32.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.397+0000 7f6ace5cd740 1 -- 172.21.15.70:0/565201896 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55acab7ba6c0 con 0x55acab85e6a0 2024-04-01T01:36:32.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.397+0000 7f6ac978c640 1 --2- 172.21.15.70:0/565201896 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55acab85e6a0 0x55acab860b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.398+0000 7f6ac978c640 1 --2- 172.21.15.70:0/565201896 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55acab85e6a0 0x55acab860b20 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=231098d72102da6a server_cookie=f64674fcbe3f16e3 in_seq=0 out_seq=0 2024-04-01T01:36:32.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.398+0000 7f6aacff9640 10 client.16560 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:32.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.399+0000 7f6aacff9640 1 -- 172.21.15.70:0/565201896 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55acab7ba6c0 con 0x55acab85e6a0 2024-04-01T01:36:32.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.399+0000 7f6aacff9640 10 client.16560 handle_command_reply: tid=0 2024-04-01T01:36:32.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.399+0000 7f6ace5cd740 1 client.16560 shutdown 2024-04-01T01:36:32.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.400+0000 7f6ace5cd740 1 -- 172.21.15.70:0/565201896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab848ca0 msgr2=0x55acab852d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.400+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/565201896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab848ca0 0x55acab852d70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00125a0 tx=0x7f6ac00046c0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.400+0000 7f6ace5cd740 1 -- 172.21.15.70:0/565201896 shutdown_connections 2024-04-01T01:36:32.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.400+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/565201896 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55acab85e6a0 0x55acab860b20 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.400+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/565201896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab84de10 0x55acab859db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.400+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/565201896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab848ca0 0x55acab852d70 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.400+0000 7f6aacff9640 0 client.16560 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:32.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.400+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/565201896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab8482b0 0x55acab852830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.400+0000 7f6ace5cd740 1 -- 172.21.15.70:0/565201896 >> 172.21.15.70:0/565201896 conn(0x55acab843820 msgr2=0x55acab842c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:32.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.400+0000 7f6ace5cd740 1 -- 172.21.15.70:0/565201896 shutdown_connections 2024-04-01T01:36:32.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.400+0000 7f6ace5cd740 1 -- 172.21.15.70:0/565201896 wait complete. 2024-04-01T01:36:32.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.400+0000 7f6ace5cd740 20 client.16560 trim_cache size 0 max 16384 2024-04-01T01:36:32.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.432+0000 7f6ace5cd740 1 Processor -- start 2024-04-01T01:36:32.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.432+0000 7f6ace5cd740 1 -- start start 2024-04-01T01:36:32.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.433+0000 7f6ace5cd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab8482b0 0x55acab8512e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.433+0000 7f6ace5cd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab848ca0 0x55acab859d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.433+0000 7f6ace5cd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab84de10 0x55acab842390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.433+0000 7f6ace5cd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac41b0870 con 0x55acab84de10 2024-04-01T01:36:32.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.433+0000 7f6ace5cd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac41b23b0 con 0x55acab8482b0 2024-04-01T01:36:32.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.433+0000 7f6ace5cd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac41b0f40 con 0x55acab848ca0 2024-04-01T01:36:32.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.433+0000 7f6ac978c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab8482b0 0x55acab8512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.433+0000 7f6ac978c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab8482b0 0x55acab8512e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35744/0 (socket says 172.21.15.70:35744) 2024-04-01T01:36:32.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.433+0000 7f6ac978c640 1 -- 172.21.15.70:0/4244970324 learned_addr learned my addr 172.21.15.70:0/4244970324 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:32.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.433+0000 7f6ac9f8d640 1 --2- 172.21.15.70:0/4244970324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab84de10 0x55acab842390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.433+0000 7f6ac8f8b640 1 --2- 172.21.15.70:0/4244970324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab848ca0 0x55acab859d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.433+0000 7f6ac978c640 1 -- 172.21.15.70:0/4244970324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab848ca0 msgr2=0x55acab859d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:32.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.433+0000 7f6ac978c640 1 --2- 172.21.15.70:0/4244970324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab848ca0 0x55acab859d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.433+0000 7f6ac978c640 1 -- 172.21.15.70:0/4244970324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab84de10 msgr2=0x55acab842390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:32.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.433+0000 7f6ac978c640 1 --2- 172.21.15.70:0/4244970324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab84de10 0x55acab842390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.433+0000 7f6ac978c640 1 -- 172.21.15.70:0/4244970324 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55acab4810f0 con 0x55acab8482b0 2024-04-01T01:36:32.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.434+0000 7f6ac8f8b640 1 --2- 172.21.15.70:0/4244970324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab848ca0 0x55acab859d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:36:32.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.434+0000 7f6ac978c640 1 --2- 172.21.15.70:0/4244970324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab8482b0 0x55acab8512e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6abc037040 tx=0x7f6abc02c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83dcdae121dabad7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:32.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.434+0000 7f6aaeffd640 1 -- 172.21.15.70:0/4244970324 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6abc057070 con 0x55acab8482b0 2024-04-01T01:36:32.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.434+0000 7f6aaeffd640 1 -- 172.21.15.70:0/4244970324 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6abc003970 con 0x55acab8482b0 2024-04-01T01:36:32.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.435+0000 7f6aaeffd640 1 -- 172.21.15.70:0/4244970324 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6abc039040 con 0x55acab8482b0 2024-04-01T01:36:32.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.436+0000 7f6ace5cd740 1 -- 172.21.15.70:0/4244970324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab8482b0 msgr2=0x55acab8512e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.436+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/4244970324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab8482b0 0x55acab8512e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6abc037040 tx=0x7f6abc02c100 comp rx=0 tx=0).stop 2024-04-01T01:36:32.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.436+0000 7f6ace5cd740 1 -- 172.21.15.70:0/4244970324 shutdown_connections 2024-04-01T01:36:32.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.436+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/4244970324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab84de10 0x55acab842390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.436+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/4244970324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab848ca0 0x55acab859d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.436+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/4244970324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab8482b0 0x55acab8512e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.436+0000 7f6ace5cd740 1 -- 172.21.15.70:0/4244970324 >> 172.21.15.70:0/4244970324 conn(0x55acab843820 msgr2=0x55acab84a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:32.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.436+0000 7f6ace5cd740 1 -- 172.21.15.70:0/4244970324 shutdown_connections 2024-04-01T01:36:32.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.436+0000 7f6ace5cd740 1 -- 172.21.15.70:0/4244970324 wait complete. 2024-04-01T01:36:32.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.437+0000 7f6ace5cd740 1 -- start start 2024-04-01T01:36:32.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.437+0000 7f6ace5cd740 1 Processor -- start 2024-04-01T01:36:32.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ace5cd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab8482b0 0x55acab8573c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ace5cd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab848ca0 0x55acab857900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ace5cd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab84de10 0x55acab858780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ace5cd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac41b1ce0 con 0x55acab848ca0 2024-04-01T01:36:32.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ace5cd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac41b1610 con 0x55acab8482b0 2024-04-01T01:36:32.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ace5cd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac41ae850 con 0x55acab84de10 2024-04-01T01:36:32.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ace5cd740 10 client.0 _ll_register_callbacks cb 0x7f6ac00124f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:32.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ac8f8b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab848ca0 0x55acab857900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ac8f8b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab848ca0 0x55acab857900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60064/0 (socket says 172.21.15.70:60064) 2024-04-01T01:36:32.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ac8f8b640 1 -- 172.21.15.70:0/2234015980 learned_addr learned my addr 172.21.15.70:0/2234015980 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:32.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ac978c640 1 --2- 172.21.15.70:0/2234015980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab8482b0 0x55acab8573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ac9f8d640 1 --2- 172.21.15.70:0/2234015980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab84de10 0x55acab858780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ac978c640 1 -- 172.21.15.70:0/2234015980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab84de10 msgr2=0x55acab858780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:32.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ac978c640 1 --2- 172.21.15.70:0/2234015980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab84de10 0x55acab858780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ac978c640 1 -- 172.21.15.70:0/2234015980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab848ca0 msgr2=0x55acab857900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:32.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ac978c640 1 --2- 172.21.15.70:0/2234015980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab848ca0 0x55acab857900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.438+0000 7f6ac978c640 1 -- 172.21.15.70:0/2234015980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acab858dc0 con 0x55acab8482b0 2024-04-01T01:36:32.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.439+0000 7f6ac978c640 1 --2- 172.21.15.70:0/2234015980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab8482b0 0x55acab8573c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6abc054000 tx=0x7f6abc009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3be0dadd5f315ec server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:32.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.439+0000 7f6aacff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:36:32.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.439+0000 7f6aacff9640 1 -- 172.21.15.70:0/2234015980 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6abc057030 con 0x55acab8482b0 2024-04-01T01:36:32.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.439+0000 7f6aacff9640 1 -- 172.21.15.70:0/2234015980 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f6abc003ba0 con 0x55acab8482b0 2024-04-01T01:36:32.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.439+0000 7f6ace5cd740 1 -- 172.21.15.70:0/2234015980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6abc003970 con 0x55acab8482b0 2024-04-01T01:36:32.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.439+0000 7f6aacff9640 1 -- 172.21.15.70:0/2234015980 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6abc061050 con 0x55acab8482b0 2024-04-01T01:36:32.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.439+0000 7f6ace5cd740 10 client.26417 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:32.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.439+0000 7f6ace5cd740 1 -- 172.21.15.70:0/2234015980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55acab85a000 con 0x55acab8482b0 2024-04-01T01:36:32.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.440+0000 7f6aacff9640 1 -- 172.21.15.70:0/2234015980 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6abc032610 con 0x55acab8482b0 2024-04-01T01:36:32.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.440+0000 7f6aacff9640 1 -- 172.21.15.70:0/2234015980 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6abc003f80 con 0x55acab8482b0 2024-04-01T01:36:32.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.440+0000 7f6ace5cd740 10 client.26417 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:32.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.440+0000 7f6ace5cd740 10 client.26417 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:36:32.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.440+0000 7f6ace5cd740 20 client.26417 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:32.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.440+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/2234015980 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55acab8c4c70 0x55acab8c7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:32.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.440+0000 7f6ace5cd740 4 client.26417 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:36:32.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.440+0000 7f6ace5cd740 1 -- 172.21.15.70:0/2234015980 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f6abc003970 con 0x55acab8c4c70 2024-04-01T01:36:32.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.441+0000 7f6ac8f8b640 1 --2- 172.21.15.70:0/2234015980 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55acab8c4c70 0x55acab8c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:32.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.441+0000 7f6ac8f8b640 1 --2- 172.21.15.70:0/2234015980 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55acab8c4c70 0x55acab8c7060 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45042d4696617745 server_cookie=e7369a2646adc049 in_seq=0 out_seq=0 2024-04-01T01:36:32.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.441+0000 7f6aacff9640 10 client.26417 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:32.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.441+0000 7f6aacff9640 1 -- 172.21.15.70:0/2234015980 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f6abc003970 con 0x55acab8c4c70 2024-04-01T01:36:32.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.441+0000 7f6aacff9640 10 client.26417 handle_command_reply: tid=0 2024-04-01T01:36:32.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.441+0000 7f6ace5cd740 1 client.26417 shutdown 2024-04-01T01:36:32.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.442+0000 7f6ace5cd740 1 -- 172.21.15.70:0/2234015980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab8482b0 msgr2=0x55acab8573c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.442+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/2234015980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab8482b0 0x55acab8573c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6abc054000 tx=0x7f6abc009bb0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.442+0000 7f6ace5cd740 1 -- 172.21.15.70:0/2234015980 shutdown_connections 2024-04-01T01:36:32.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.442+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/2234015980 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55acab8c4c70 0x55acab8c7060 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.442+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/2234015980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55acab84de10 0x55acab858780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.442+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/2234015980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55acab848ca0 0x55acab857900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.442+0000 7f6ace5cd740 1 --2- 172.21.15.70:0/2234015980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55acab8482b0 0x55acab8573c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.442+0000 7f6ace5cd740 1 -- 172.21.15.70:0/2234015980 >> 172.21.15.70:0/2234015980 conn(0x55acab843820 msgr2=0x55acab842390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:32.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.443+0000 7f6aacff9640 0 client.26417 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:32.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.444+0000 7f6ace5cd740 1 -- 172.21.15.70:0/2234015980 shutdown_connections 2024-04-01T01:36:32.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.445+0000 7f6ace5cd740 1 -- 172.21.15.70:0/2234015980 wait complete. 2024-04-01T01:36:32.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.446+0000 7f6ace5cd740 20 client.26417 trim_cache size 0 max 16384 2024-04-01T01:36:32.446 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:36:32.446 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:36:32.446 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:36:32.446 INFO:teuthology.orchestra.run.smithi070.stdout: "4f67cf82-3341-43cd-916e-4592e3fd6586": { 2024-04-01T01:36:32.446 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:36:32.447 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "4f67cf82-3341-43cd-916e-4592e3fd6586", 2024-04-01T01:36:32.447 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:36:32.447 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:36:32.447 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:36:32.447 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:36:32.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.447+0000 7f6acb9f0640 1 -- 172.21.15.70:0/737881826 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6ab00a2520 msgr2=0x7f6ab00a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.447+0000 7f6acb9f0640 1 --2- 172.21.15.70:0/737881826 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6ab00a2520 0x7f6ab00a49e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x55acab849500 tx=0x7f6ac000b390 comp rx=0 tx=0).stop 2024-04-01T01:36:32.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.447+0000 7f6acb9f0640 1 -- 172.21.15.70:0/737881826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4103310 msgr2=0x7f6ac41a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:32.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.447+0000 7f6acb9f0640 1 --2- 172.21.15.70:0/737881826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4103310 0x7f6ac41a5370 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400d680 tx=0x7f6ab400db40 comp rx=0 tx=0).stop 2024-04-01T01:36:32.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.447+0000 7f6acb9f0640 1 -- 172.21.15.70:0/737881826 shutdown_connections 2024-04-01T01:36:32.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.447+0000 7f6acb9f0640 1 --2- 172.21.15.70:0/737881826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ac4103310 0x7f6ac41a5370 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.447+0000 7f6acb9f0640 1 --2- 172.21.15.70:0/737881826 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6ab00a2520 0x7f6ab00a49e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.448+0000 7f6acb9f0640 1 --2- 172.21.15.70:0/737881826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ac4102950 0x7f6ac419e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.448+0000 7f6acb9f0640 1 --2- 172.21.15.70:0/737881826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ac4101760 0x7f6ac419ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:32.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.448+0000 7f6acb9f0640 1 -- 172.21.15.70:0/737881826 >> 172.21.15.70:0/737881826 conn(0x7f6ac40fccf0 msgr2=0x7f6ac40fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:32.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.449+0000 7f6acb9f0640 1 -- 172.21.15.70:0/737881826 shutdown_connections 2024-04-01T01:36:32.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:36:32.449+0000 7f6acb9f0640 1 -- 172.21.15.70:0/737881826 wait complete. 2024-04-01T01:36:32.471 INFO:tasks.cephfs.filesystem:scrub status for tag:4f67cf82-3341-43cd-916e-4592e3fd6586 - {'path': '/', 'tag': '4f67cf82-3341-43cd-916e-4592e3fd6586', 'options': 'recursive,force'} 2024-04-01T01:36:33.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:33 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:36:33.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:33 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:36:33.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:33 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:36:33.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:33 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:36:33.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:33 smithi189 ceph-mon[35753]: pgmap v450: 129 pgs: 129 active+clean; 990 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-04-01T01:36:34.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:33 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:36:34.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:33 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:36:34.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:33 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:36:34.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:33 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:36:34.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:33 smithi070 ceph-mon[32179]: pgmap v450: 129 pgs: 129 active+clean; 990 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-04-01T01:36:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:33 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:36:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:33 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:36:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:33 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:36:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:33 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:36:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:33 smithi081 ceph-mon[35142]: pgmap v450: 129 pgs: 129 active+clean; 990 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.38k op/s 2024-04-01T01:36:35.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:35 smithi189 ceph-mon[35753]: pgmap v451: 129 pgs: 129 active+clean; 990 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 10 MiB/s wr, 1.23k op/s 2024-04-01T01:36:36.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:35 smithi070 ceph-mon[32179]: pgmap v451: 129 pgs: 129 active+clean; 990 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 10 MiB/s wr, 1.23k op/s 2024-04-01T01:36:36.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:35 smithi081 ceph-mon[35142]: pgmap v451: 129 pgs: 129 active+clean; 990 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 10 MiB/s wr, 1.23k op/s 2024-04-01T01:36:36.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 110.0 s, 1179.3 tps, lat 84.754 ms stddev 30.033 2024-04-01T01:36:36.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:36 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:36:37.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:36 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:36:37.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:36 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:36:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.127+0000 7f1a1d260640 1 -- 172.21.15.70:0/1108879663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a18111c70 msgr2=0x7f1a18114100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.127+0000 7f1a1d260640 1 --2- 172.21.15.70:0/1108879663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a18111c70 0x7f1a18114100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c009900 tx=0x7f1a0c02bc40 comp rx=0 tx=0).stop 2024-04-01T01:36:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.127+0000 7f1a1d260640 1 -- 172.21.15.70:0/1108879663 shutdown_connections 2024-04-01T01:36:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.127+0000 7f1a1d260640 1 --2- 172.21.15.70:0/1108879663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a18111c70 0x7f1a18114100 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.127+0000 7f1a1d260640 1 --2- 172.21.15.70:0/1108879663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a1806b500 0x7f1a1806b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.127+0000 7f1a1d260640 1 --2- 172.21.15.70:0/1108879663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a1806cae0 0x7f1a1806af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.127+0000 7f1a1d260640 1 -- 172.21.15.70:0/1108879663 >> 172.21.15.70:0/1108879663 conn(0x7f1a18070540 msgr2=0x7f1a18072980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.128+0000 7f1a1d260640 1 -- 172.21.15.70:0/1108879663 shutdown_connections 2024-04-01T01:36:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.128+0000 7f1a1d260640 1 -- 172.21.15.70:0/1108879663 wait complete. 2024-04-01T01:36:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.129+0000 7f1a1d260640 1 Processor -- start 2024-04-01T01:36:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.129+0000 7f1a1d260640 1 -- start start 2024-04-01T01:36:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.130+0000 7f1a1d260640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a1806b500 0x7f1a181a2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.130+0000 7f1a1d260640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a1806cae0 0x7f1a181a2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.130+0000 7f1a1d260640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a18111c70 0x7f1a181a97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.130+0000 7f1a1d260640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1a18117b20 con 0x7f1a1806cae0 2024-04-01T01:36:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.130+0000 7f1a1d260640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1a181179a0 con 0x7f1a1806b500 2024-04-01T01:36:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.130+0000 7f1a1d260640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1a18117ca0 con 0x7f1a18111c70 2024-04-01T01:36:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.130+0000 7f1a177fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a1806cae0 0x7f1a181a2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.130+0000 7f1a177fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a1806cae0 0x7f1a181a2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42868/0 (socket says 172.21.15.70:42868) 2024-04-01T01:36:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.130+0000 7f1a177fe640 1 -- 172.21.15.70:0/3704358455 learned_addr learned my addr 172.21.15.70:0/3704358455 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.130+0000 7f1a1ca5f640 1 --2- 172.21.15.70:0/3704358455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a18111c70 0x7f1a181a97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.131+0000 7f1a17fff640 1 --2- 172.21.15.70:0/3704358455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a1806b500 0x7f1a181a2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.131+0000 7f1a1ca5f640 1 -- 172.21.15.70:0/3704358455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a1806b500 msgr2=0x7f1a181a2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.131+0000 7f1a1ca5f640 1 --2- 172.21.15.70:0/3704358455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a1806b500 0x7f1a181a2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.131+0000 7f1a1ca5f640 1 -- 172.21.15.70:0/3704358455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a1806cae0 msgr2=0x7f1a181a2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.131+0000 7f1a1ca5f640 1 --2- 172.21.15.70:0/3704358455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a1806cae0 0x7f1a181a2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.131+0000 7f1a1ca5f640 1 -- 172.21.15.70:0/3704358455 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a181a9e30 con 0x7f1a18111c70 2024-04-01T01:36:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.131+0000 7f1a17fff640 1 --2- 172.21.15.70:0/3704358455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a1806b500 0x7f1a181a2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:36:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.131+0000 7f1a1ca5f640 1 --2- 172.21.15.70:0/3704358455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a18111c70 0x7f1a181a97c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c036040 tx=0x7f1a0c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.131+0000 7f1a157fa640 1 -- 172.21.15.70:0/3704358455 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c043020 con 0x7f1a18111c70 2024-04-01T01:36:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.131+0000 7f1a157fa640 1 -- 172.21.15.70:0/3704358455 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a0c0052b0 con 0x7f1a18111c70 2024-04-01T01:36:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.131+0000 7f1a157fa640 1 -- 172.21.15.70:0/3704358455 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c003bb0 con 0x7f1a18111c70 2024-04-01T01:36:37.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.132+0000 7f1a1d260640 1 -- 172.21.15.70:0/3704358455 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a181aa0c0 con 0x7f1a18111c70 2024-04-01T01:36:37.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.132+0000 7f1a1d260640 1 -- 172.21.15.70:0/3704358455 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a18109760 con 0x7f1a18111c70 2024-04-01T01:36:37.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.135+0000 7f1a21176740 1 Processor -- start 2024-04-01T01:36:37.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.135+0000 7f1a157fa640 1 -- 172.21.15.70:0/3704358455 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f1a0c046020 con 0x7f1a18111c70 2024-04-01T01:36:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.137+0000 7f1a157fa640 1 --2- 172.21.15.70:0/3704358455 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a000a26a0 0x7f1a000a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.137+0000 7f1a157fa640 1 -- 172.21.15.70:0/3704358455 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1a0c0f0d10 con 0x7f1a18111c70 2024-04-01T01:36:37.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.137+0000 7f1a21176740 1 -- start start 2024-04-01T01:36:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.137+0000 7f1a21176740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3dae0 0x55739ec3df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.137+0000 7f1a21176740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec3e4d0 0x55739ec42fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.137+0000 7f1a21176740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec43640 0x55739ec45ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.137+0000 7f1a21176740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55739eba18f0 con 0x55739ec3e4d0 2024-04-01T01:36:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.137+0000 7f1a21176740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55739eb4a8d0 con 0x55739ec3dae0 2024-04-01T01:36:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.137+0000 7f1a21176740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55739eb9d990 con 0x55739ec43640 2024-04-01T01:36:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.137+0000 7f1a177fe640 1 --2- 172.21.15.70:0/3704358455 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a000a26a0 0x7f1a000a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.138+0000 7f1a17fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3dae0 0x55739ec3df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.138+0000 7f1a17fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3dae0 0x55739ec3df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60578/0 (socket says 172.21.15.70:60578) 2024-04-01T01:36:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.138+0000 7f1a17fff640 1 -- 172.21.15.70:0/887051745 learned_addr learned my addr 172.21.15.70:0/887051745 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.138+0000 7f1a177fe640 1 --2- 172.21.15.70:0/887051745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec3e4d0 0x55739ec42fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.139+0000 7f1a1ca5f640 1 --2- 172.21.15.70:0/887051745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec43640 0x55739ec45ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.139+0000 7f1a17fff640 1 -- 172.21.15.70:0/887051745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec43640 msgr2=0x55739ec45ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.139+0000 7f1a17fff640 1 --2- 172.21.15.70:0/887051745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec43640 0x55739ec45ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.139+0000 7f1a17fff640 1 -- 172.21.15.70:0/887051745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec3e4d0 msgr2=0x55739ec42fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.139+0000 7f1a17fff640 1 --2- 172.21.15.70:0/887051745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec3e4d0 0x55739ec42fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.139+0000 7f1a17fff640 1 -- 172.21.15.70:0/887051745 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55739e8b6210 con 0x55739ec3dae0 2024-04-01T01:36:37.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.139+0000 7f1a177fe640 1 --2- 172.21.15.70:0/3704358455 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a000a26a0 0x7f1a000a4b60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1a181a3720 tx=0x7f1a080129b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.140+0000 7f1a17fff640 1 --2- 172.21.15.70:0/887051745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3dae0 0x55739ec3df00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1a10007fb0 tx=0x7f1a1000c690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8298be71e99d72cf server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.140+0000 7f19fe7fc640 1 -- 172.21.15.70:0/887051745 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a10014070 con 0x55739ec3dae0 2024-04-01T01:36:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.140+0000 7f19fe7fc640 1 -- 172.21.15.70:0/887051745 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a1000fd00 con 0x55739ec3dae0 2024-04-01T01:36:37.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.140+0000 7f19fe7fc640 1 -- 172.21.15.70:0/887051745 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1000ec20 con 0x55739ec3dae0 2024-04-01T01:36:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.141+0000 7f1a21176740 1 -- 172.21.15.70:0/887051745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3dae0 msgr2=0x55739ec3df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.141+0000 7f1a21176740 1 --2- 172.21.15.70:0/887051745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3dae0 0x55739ec3df00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1a10007fb0 tx=0x7f1a1000c690 comp rx=0 tx=0).stop 2024-04-01T01:36:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.141+0000 7f1a21176740 1 -- 172.21.15.70:0/887051745 shutdown_connections 2024-04-01T01:36:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.141+0000 7f1a21176740 1 --2- 172.21.15.70:0/887051745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec43640 0x55739ec45ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.141+0000 7f1a21176740 1 --2- 172.21.15.70:0/887051745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec3e4d0 0x55739ec42fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.141+0000 7f1a21176740 1 --2- 172.21.15.70:0/887051745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3dae0 0x55739ec3df00 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.141+0000 7f1a21176740 1 -- 172.21.15.70:0/887051745 >> 172.21.15.70:0/887051745 conn(0x55739ec39050 msgr2=0x55739ec39450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.142+0000 7f1a21176740 1 -- 172.21.15.70:0/887051745 shutdown_connections 2024-04-01T01:36:37.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.142+0000 7f1a21176740 1 -- 172.21.15.70:0/887051745 wait complete. 2024-04-01T01:36:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.142+0000 7f1a21176740 1 -- start start 2024-04-01T01:36:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.142+0000 7f1a21176740 1 Processor -- start 2024-04-01T01:36:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.143+0000 7f1a21176740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec3dae0 0x55739ec47e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.143+0000 7f1a21176740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3e4d0 0x55739ec48360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.143+0000 7f1a21176740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec43640 0x55739ec4f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.143+0000 7f1a21176740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55739ec465f0 con 0x55739ec43640 2024-04-01T01:36:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.143+0000 7f1a21176740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55739eb3ff80 con 0x55739ec3e4d0 2024-04-01T01:36:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.143+0000 7f1a21176740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55739ec46770 con 0x55739ec3dae0 2024-04-01T01:36:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.143+0000 7f1a21176740 10 client.0 _ll_register_callbacks cb 0x55739eb98290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.143+0000 7f1a17fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec3dae0 0x55739ec47e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.143+0000 7f1a177fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3e4d0 0x55739ec48360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.143+0000 7f1a17fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec3dae0 0x55739ec47e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57230/0 (socket says 172.21.15.70:57230) 2024-04-01T01:36:37.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.143+0000 7f1a17fff640 1 -- 172.21.15.70:0/1339272851 learned_addr learned my addr 172.21.15.70:0/1339272851 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.143+0000 7f1a1ca5f640 1 --2- 172.21.15.70:0/1339272851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec43640 0x55739ec4f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.144+0000 7f1a17fff640 1 -- 172.21.15.70:0/1339272851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3e4d0 msgr2=0x55739ec48360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.144+0000 7f1a17fff640 1 --2- 172.21.15.70:0/1339272851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3e4d0 0x55739ec48360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.144+0000 7f1a17fff640 1 -- 172.21.15.70:0/1339272851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec43640 msgr2=0x55739ec4f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.144+0000 7f1a17fff640 1 --2- 172.21.15.70:0/1339272851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec43640 0x55739ec4f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.144+0000 7f1a17fff640 1 -- 172.21.15.70:0/1339272851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55739e982620 con 0x55739ec3dae0 2024-04-01T01:36:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.144+0000 7f1a17fff640 1 --2- 172.21.15.70:0/1339272851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec3dae0 0x55739ec47e20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1a1000fc80 tx=0x7f1a1000e600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78dad56f6a7b84f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.144+0000 7f19f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:36:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.144+0000 7f19f77fe640 1 -- 172.21.15.70:0/1339272851 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a10014040 con 0x55739ec3dae0 2024-04-01T01:36:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.144+0000 7f19f77fe640 1 -- 172.21.15.70:0/1339272851 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1a1000eb20 con 0x55739ec3dae0 2024-04-01T01:36:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.144+0000 7f1a21176740 1 -- 172.21.15.70:0/1339272851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a1000fd00 con 0x55739ec3dae0 2024-04-01T01:36:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.145+0000 7f19f77fe640 1 -- 172.21.15.70:0/1339272851 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1a1000ed20 con 0x55739ec3dae0 2024-04-01T01:36:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.145+0000 7f1a21176740 10 client.26314 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.145+0000 7f1a21176740 1 -- 172.21.15.70:0/1339272851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55739ec52380 con 0x55739ec3dae0 2024-04-01T01:36:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.145+0000 7f19f77fe640 1 -- 172.21.15.70:0/1339272851 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a1000fd00 con 0x55739ec3dae0 2024-04-01T01:36:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.145+0000 7f19f77fe640 1 -- 172.21.15.70:0/1339272851 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1a10021420 con 0x55739ec3dae0 2024-04-01T01:36:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.145+0000 7f1a21176740 10 client.26314 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.145+0000 7f1a21176740 10 client.26314 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:36:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.145+0000 7f1a21176740 20 client.26314 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.145+0000 7f1a21176740 1 --2- 172.21.15.70:0/1339272851 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55739ec53780 0x55739ec55bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.145+0000 7f1a21176740 4 client.26314 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:36:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.145+0000 7f1a21176740 1 -- 172.21.15.70:0/1339272851 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55739ebb5120 con 0x55739ec53780 2024-04-01T01:36:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.146+0000 7f1a177fe640 1 --2- 172.21.15.70:0/1339272851 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55739ec53780 0x55739ec55bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.147+0000 7f1a177fe640 1 --2- 172.21.15.70:0/1339272851 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55739ec53780 0x55739ec55bb0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32d3b48806947ebf server_cookie=20ab24ca054ca533 in_seq=0 out_seq=0 2024-04-01T01:36:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.147+0000 7f19f77fe640 10 client.26314 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.148+0000 7f19f77fe640 1 -- 172.21.15.70:0/1339272851 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55739ebb5120 con 0x55739ec53780 2024-04-01T01:36:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.148+0000 7f19f77fe640 10 client.26314 handle_command_reply: tid=0 2024-04-01T01:36:37.147 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.148+0000 7f1a21176740 1 client.26314 shutdown 2024-04-01T01:36:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.149+0000 7f1a21176740 1 -- 172.21.15.70:0/1339272851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec3dae0 msgr2=0x55739ec47e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:37.148 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.149+0000 7f1a21176740 1 --2- 172.21.15.70:0/1339272851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec3dae0 0x55739ec47e20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1a1000fc80 tx=0x7f1a1000e600 comp rx=0 tx=0).stop 2024-04-01T01:36:37.149 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.149+0000 7f1a21176740 1 -- 172.21.15.70:0/1339272851 shutdown_connections 2024-04-01T01:36:37.149 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.149+0000 7f1a21176740 1 --2- 172.21.15.70:0/1339272851 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55739ec53780 0x55739ec55bb0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.149+0000 7f1a21176740 1 --2- 172.21.15.70:0/1339272851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec43640 0x55739ec4f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.149+0000 7f1a21176740 1 --2- 172.21.15.70:0/1339272851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3e4d0 0x55739ec48360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.149+0000 7f1a21176740 1 --2- 172.21.15.70:0/1339272851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec3dae0 0x55739ec47e20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.149+0000 7f1a21176740 1 -- 172.21.15.70:0/1339272851 >> 172.21.15.70:0/1339272851 conn(0x55739ec39050 msgr2=0x55739ec38190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:37.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.149+0000 7f19f77fe640 0 client.26314 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:37.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.150+0000 7f1a21176740 1 -- 172.21.15.70:0/1339272851 shutdown_connections 2024-04-01T01:36:37.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.150+0000 7f1a21176740 1 -- 172.21.15.70:0/1339272851 wait complete. 2024-04-01T01:36:37.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.150+0000 7f1a21176740 20 client.26314 trim_cache size 0 max 16384 2024-04-01T01:36:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.183+0000 7f1a21176740 1 Processor -- start 2024-04-01T01:36:37.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.184+0000 7f1a21176740 1 -- start start 2024-04-01T01:36:37.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.184+0000 7f1a21176740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec3dae0 0x55739ec37900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a21176740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3e4d0 0x55739ec4f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a21176740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec43640 0x55739ec37e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a21176740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0400fcd0 con 0x55739ec3dae0 2024-04-01T01:36:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a21176740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1a04011810 con 0x55739ec3e4d0 2024-04-01T01:36:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a21176740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1a040103a0 con 0x55739ec43640 2024-04-01T01:36:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a1ca5f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec43640 0x55739ec37e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a17fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec3dae0 0x55739ec37900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a1ca5f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec43640 0x55739ec37e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57232/0 (socket says 172.21.15.70:57232) 2024-04-01T01:36:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a17fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec3dae0 0x55739ec37900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42900/0 (socket says 172.21.15.70:42900) 2024-04-01T01:36:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a17fff640 1 -- 172.21.15.70:0/3812335775 learned_addr learned my addr 172.21.15.70:0/3812335775 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a177fe640 1 --2- 172.21.15.70:0/3812335775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3e4d0 0x55739ec4f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a17fff640 1 -- 172.21.15.70:0/3812335775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec43640 msgr2=0x55739ec37e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a17fff640 1 --2- 172.21.15.70:0/3812335775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec43640 0x55739ec37e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a17fff640 1 -- 172.21.15.70:0/3812335775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3e4d0 msgr2=0x55739ec4f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:37.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a17fff640 1 --2- 172.21.15.70:0/3812335775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3e4d0 0x55739ec4f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a17fff640 1 -- 172.21.15.70:0/3812335775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55739ec48be0 con 0x55739ec3dae0 2024-04-01T01:36:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.185+0000 7f1a177fe640 1 --2- 172.21.15.70:0/3812335775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3e4d0 0x55739ec4f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:36:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.186+0000 7f1a17fff640 1 --2- 172.21.15.70:0/3812335775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec3dae0 0x55739ec37900 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1a10021840 tx=0x7f1a10007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31264a2a4d05e580 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.186+0000 7f19fdffb640 1 -- 172.21.15.70:0/3812335775 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a10014050 con 0x55739ec3dae0 2024-04-01T01:36:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.187+0000 7f19fdffb640 1 -- 172.21.15.70:0/3812335775 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a10026070 con 0x55739ec3dae0 2024-04-01T01:36:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.187+0000 7f19fdffb640 1 -- 172.21.15.70:0/3812335775 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a10021ab0 con 0x55739ec3dae0 2024-04-01T01:36:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.188+0000 7f1a21176740 1 -- 172.21.15.70:0/3812335775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec3dae0 msgr2=0x55739ec37900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.188+0000 7f1a21176740 1 --2- 172.21.15.70:0/3812335775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec3dae0 0x55739ec37900 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1a10021840 tx=0x7f1a10007e00 comp rx=0 tx=0).stop 2024-04-01T01:36:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.188+0000 7f1a21176740 1 -- 172.21.15.70:0/3812335775 shutdown_connections 2024-04-01T01:36:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.188+0000 7f1a21176740 1 --2- 172.21.15.70:0/3812335775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec43640 0x55739ec37e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.188+0000 7f1a21176740 1 --2- 172.21.15.70:0/3812335775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3e4d0 0x55739ec4f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.188+0000 7f1a21176740 1 --2- 172.21.15.70:0/3812335775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec3dae0 0x55739ec37900 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.188+0000 7f1a21176740 1 -- 172.21.15.70:0/3812335775 >> 172.21.15.70:0/3812335775 conn(0x55739ec39050 msgr2=0x55739ec39430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.188+0000 7f1a21176740 1 -- 172.21.15.70:0/3812335775 shutdown_connections 2024-04-01T01:36:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.188+0000 7f1a21176740 1 -- 172.21.15.70:0/3812335775 wait complete. 2024-04-01T01:36:37.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.188+0000 7f1a21176740 1 -- start start 2024-04-01T01:36:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.189+0000 7f1a21176740 1 Processor -- start 2024-04-01T01:36:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.189+0000 7f1a21176740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3dae0 0x55739ec4c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.189+0000 7f1a21176740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec3e4d0 0x55739ec4c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.189+0000 7f1a21176740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec43640 0x55739ecb7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.189+0000 7f1a21176740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1a04011140 con 0x55739ec43640 2024-04-01T01:36:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.189+0000 7f1a21176740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1a04010a70 con 0x55739ec3dae0 2024-04-01T01:36:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.189+0000 7f1a21176740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0400dcb0 con 0x55739ec3e4d0 2024-04-01T01:36:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.189+0000 7f1a21176740 10 client.0 _ll_register_callbacks cb 0x7f1a0800bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.190+0000 7f1a17fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3dae0 0x55739ec4c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.190+0000 7f1a1ca5f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec43640 0x55739ecb7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.190+0000 7f1a17fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3dae0 0x55739ec4c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60592/0 (socket says 172.21.15.70:60592) 2024-04-01T01:36:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.190+0000 7f1a17fff640 1 -- 172.21.15.70:0/2829762253 learned_addr learned my addr 172.21.15.70:0/2829762253 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.190+0000 7f1a177fe640 1 --2- 172.21.15.70:0/2829762253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec3e4d0 0x55739ec4c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.190+0000 7f1a17fff640 1 -- 172.21.15.70:0/2829762253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec3e4d0 msgr2=0x55739ec4c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.190+0000 7f1a17fff640 1 --2- 172.21.15.70:0/2829762253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec3e4d0 0x55739ec4c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.190+0000 7f1a17fff640 1 -- 172.21.15.70:0/2829762253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec43640 msgr2=0x55739ecb7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.190+0000 7f1a17fff640 1 --2- 172.21.15.70:0/2829762253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec43640 0x55739ecb7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.190+0000 7f1a17fff640 1 -- 172.21.15.70:0/2829762253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55739ec4cb80 con 0x55739ec3dae0 2024-04-01T01:36:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.190+0000 7f1a1ca5f640 1 --2- 172.21.15.70:0/2829762253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec43640 0x55739ecb7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:36:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.190+0000 7f1a17fff640 1 --2- 172.21.15.70:0/2829762253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3dae0 0x55739ec4c050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1a10021840 tx=0x7f1a10007700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cadafefcb3cc4b0e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.190+0000 7f19f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:36:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.191+0000 7f19f77fe640 1 -- 172.21.15.70:0/2829762253 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a10014040 con 0x55739ec3dae0 2024-04-01T01:36:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.191+0000 7f19f77fe640 1 -- 172.21.15.70:0/2829762253 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1a10002030 con 0x55739ec3dae0 2024-04-01T01:36:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.191+0000 7f1a21176740 1 -- 172.21.15.70:0/2829762253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a10026070 con 0x55739ec3dae0 2024-04-01T01:36:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.191+0000 7f19f77fe640 1 -- 172.21.15.70:0/2829762253 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1a10041050 con 0x55739ec3dae0 2024-04-01T01:36:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.191+0000 7f1a21176740 10 client.26432 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.191+0000 7f1a21176740 1 -- 172.21.15.70:0/2829762253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55739ecb90a0 con 0x55739ec3dae0 2024-04-01T01:36:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.192+0000 7f19f77fe640 1 -- 172.21.15.70:0/2829762253 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a10026070 con 0x55739ec3dae0 2024-04-01T01:36:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.192+0000 7f19f77fe640 1 -- 172.21.15.70:0/2829762253 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1a10019450 con 0x55739ec3dae0 2024-04-01T01:36:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.192+0000 7f1a21176740 10 client.26432 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.192+0000 7f1a21176740 10 client.26432 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:36:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.192+0000 7f1a21176740 20 client.26432 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.192+0000 7f1a21176740 1 --2- 172.21.15.70:0/2829762253 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55739ecb9970 0x55739ecbbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.192+0000 7f1a21176740 4 client.26432 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:36:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.193+0000 7f1a21176740 1 -- 172.21.15.70:0/2829762253 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55739ebaee30 con 0x55739ecb9970 2024-04-01T01:36:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.193+0000 7f1a177fe640 1 --2- 172.21.15.70:0/2829762253 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55739ecb9970 0x55739ecbbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:37.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.194+0000 7f19f77fe640 10 client.26432 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:37.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.194+0000 7f1a177fe640 1 --2- 172.21.15.70:0/2829762253 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55739ecb9970 0x55739ecbbd90 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=321f34d2f50df4b4 server_cookie=f67867f45c03f921 in_seq=0 out_seq=0 2024-04-01T01:36:37.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.220+0000 7f19f77fe640 1 -- 172.21.15.70:0/2829762253 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55739ebaee30 con 0x55739ecb9970 2024-04-01T01:36:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.220+0000 7f19f77fe640 10 client.26432 handle_command_reply: tid=0 2024-04-01T01:36:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.220+0000 7f1a21176740 1 client.26432 shutdown 2024-04-01T01:36:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.222+0000 7f1a21176740 1 -- 172.21.15.70:0/2829762253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3dae0 msgr2=0x55739ec4c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:37.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.222+0000 7f1a21176740 1 --2- 172.21.15.70:0/2829762253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3dae0 0x55739ec4c050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1a10021840 tx=0x7f1a10007700 comp rx=0 tx=0).stop 2024-04-01T01:36:37.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.222+0000 7f1a21176740 1 -- 172.21.15.70:0/2829762253 shutdown_connections 2024-04-01T01:36:37.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.223+0000 7f1a21176740 1 --2- 172.21.15.70:0/2829762253 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55739ecb9970 0x55739ecbbd90 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.223+0000 7f19f77fe640 0 client.26432 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:37.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.223+0000 7f1a21176740 1 --2- 172.21.15.70:0/2829762253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55739ec43640 0x55739ecb7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.223+0000 7f1a21176740 1 --2- 172.21.15.70:0/2829762253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55739ec3e4d0 0x55739ec4c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.223+0000 7f1a21176740 1 --2- 172.21.15.70:0/2829762253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55739ec3dae0 0x55739ec4c050 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.223+0000 7f1a21176740 1 -- 172.21.15.70:0/2829762253 >> 172.21.15.70:0/2829762253 conn(0x55739ec39050 msgr2=0x55739ec37900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:37.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.223+0000 7f1a21176740 1 -- 172.21.15.70:0/2829762253 shutdown_connections 2024-04-01T01:36:37.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.223+0000 7f1a21176740 1 -- 172.21.15.70:0/2829762253 wait complete. 2024-04-01T01:36:37.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.223+0000 7f1a21176740 20 client.26432 trim_cache size 0 max 16384 2024-04-01T01:36:37.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:36:37.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:36:37.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:36:37.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:36:37.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.225+0000 7f1a1d260640 1 -- 172.21.15.70:0/3704358455 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a000a26a0 msgr2=0x7f1a000a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.225+0000 7f1a1d260640 1 --2- 172.21.15.70:0/3704358455 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a000a26a0 0x7f1a000a4b60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1a181a3720 tx=0x7f1a080129b0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.225+0000 7f1a1d260640 1 -- 172.21.15.70:0/3704358455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a18111c70 msgr2=0x7f1a181a97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.225+0000 7f1a1d260640 1 --2- 172.21.15.70:0/3704358455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a18111c70 0x7f1a181a97c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c036040 tx=0x7f1a0c004270 comp rx=0 tx=0).stop 2024-04-01T01:36:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.225+0000 7f1a1d260640 1 -- 172.21.15.70:0/3704358455 shutdown_connections 2024-04-01T01:36:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.225+0000 7f1a1d260640 1 --2- 172.21.15.70:0/3704358455 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a000a26a0 0x7f1a000a4b60 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.225+0000 7f1a1d260640 1 --2- 172.21.15.70:0/3704358455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a18111c70 0x7f1a181a97c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.225+0000 7f1a1d260640 1 --2- 172.21.15.70:0/3704358455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a1806cae0 0x7f1a181a2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.225+0000 7f1a1d260640 1 --2- 172.21.15.70:0/3704358455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a1806b500 0x7f1a181a2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.225+0000 7f1a1d260640 1 -- 172.21.15.70:0/3704358455 >> 172.21.15.70:0/3704358455 conn(0x7f1a18070540 msgr2=0x7f1a1810db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.225+0000 7f1a1d260640 1 -- 172.21.15.70:0/3704358455 shutdown_connections 2024-04-01T01:36:37.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:37.225+0000 7f1a1d260640 1 -- 172.21.15.70:0/3704358455 wait complete. 2024-04-01T01:36:37.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:37 smithi189 ceph-mon[35753]: pgmap v452: 129 pgs: 129 active+clean; 990 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 12 MiB/s wr, 1.46k op/s 2024-04-01T01:36:38.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:37 smithi070 ceph-mon[32179]: pgmap v452: 129 pgs: 129 active+clean; 990 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 12 MiB/s wr, 1.46k op/s 2024-04-01T01:36:38.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:37 smithi081 ceph-mon[35142]: pgmap v452: 129 pgs: 129 active+clean; 990 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 12 MiB/s wr, 1.46k op/s 2024-04-01T01:36:39.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:39 smithi189 ceph-mon[35753]: pgmap v453: 129 pgs: 129 active+clean; 990 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 11 MiB/s wr, 1.38k op/s 2024-04-01T01:36:40.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:39 smithi070 ceph-mon[32179]: pgmap v453: 129 pgs: 129 active+clean; 990 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 11 MiB/s wr, 1.38k op/s 2024-04-01T01:36:40.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:39 smithi081 ceph-mon[35142]: pgmap v453: 129 pgs: 129 active+clean; 990 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 11 MiB/s wr, 1.38k op/s 2024-04-01T01:36:41.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 115.0 s, 1073.6 tps, lat 93.349 ms stddev 95.514 2024-04-01T01:36:41.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:41 smithi189 ceph-mon[35753]: pgmap v454: 129 pgs: 129 active+clean; 990 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 10 MiB/s wr, 1.28k op/s 2024-04-01T01:36:42.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:41 smithi070 ceph-mon[32179]: pgmap v454: 129 pgs: 129 active+clean; 990 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 10 MiB/s wr, 1.28k op/s 2024-04-01T01:36:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:41 smithi081 ceph-mon[35142]: pgmap v454: 129 pgs: 129 active+clean; 990 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 10 MiB/s wr, 1.28k op/s 2024-04-01T01:36:44.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:43 smithi070 ceph-mon[32179]: pgmap v455: 129 pgs: 129 active+clean; 998 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 12 MiB/s wr, 1.49k op/s 2024-04-01T01:36:44.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:43 smithi081 ceph-mon[35142]: pgmap v455: 129 pgs: 129 active+clean; 998 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 12 MiB/s wr, 1.49k op/s 2024-04-01T01:36:44.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:43 smithi189 ceph-mon[35753]: pgmap v455: 129 pgs: 129 active+clean; 998 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 12 MiB/s wr, 1.49k op/s 2024-04-01T01:36:45.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:44 smithi081 ceph-mon[35142]: pgmap v456: 129 pgs: 129 active+clean; 1002 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-04-01T01:36:45.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:44 smithi189 ceph-mon[35753]: pgmap v456: 129 pgs: 129 active+clean; 1002 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-04-01T01:36:45.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:44 smithi070 ceph-mon[32179]: pgmap v456: 129 pgs: 129 active+clean; 1002 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-04-01T01:36:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 120.0 s, 1060.8 tps, lat 94.274 ms stddev 70.336 2024-04-01T01:36:47.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:47 smithi189 ceph-mon[35753]: pgmap v457: 129 pgs: 129 active+clean; 1006 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-04-01T01:36:48.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:47 smithi070 ceph-mon[32179]: pgmap v457: 129 pgs: 129 active+clean; 1006 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-04-01T01:36:48.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:47 smithi081 ceph-mon[35142]: pgmap v457: 129 pgs: 129 active+clean; 1006 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 12 MiB/s wr, 1.37k op/s 2024-04-01T01:36:49.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:49 smithi189 ceph-mon[35753]: pgmap v458: 129 pgs: 129 active+clean; 1006 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 1.20k op/s 2024-04-01T01:36:50.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:49 smithi070 ceph-mon[32179]: pgmap v458: 129 pgs: 129 active+clean; 1006 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 1.20k op/s 2024-04-01T01:36:50.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:49 smithi081 ceph-mon[35142]: pgmap v458: 129 pgs: 129 active+clean; 1006 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 1.20k op/s 2024-04-01T01:36:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.374+0000 7f852def4640 1 -- 172.21.15.70:0/4018163194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8528108fa0 msgr2=0x7f8528109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.374+0000 7f852def4640 1 --2- 172.21.15.70:0/4018163194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8528108fa0 0x7f8528109420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8518009900 tx=0x7f851802bc30 comp rx=0 tx=0).stop 2024-04-01T01:36:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.374+0000 7f852def4640 1 -- 172.21.15.70:0/4018163194 shutdown_connections 2024-04-01T01:36:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.374+0000 7f852def4640 1 --2- 172.21.15.70:0/4018163194 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8528109960 0x7f85281101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.374+0000 7f852def4640 1 --2- 172.21.15.70:0/4018163194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8528108fa0 0x7f8528109420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.374+0000 7f852def4640 1 --2- 172.21.15.70:0/4018163194 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8528107db0 0x7f85281081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.374+0000 7f852def4640 1 -- 172.21.15.70:0/4018163194 >> 172.21.15.70:0/4018163194 conn(0x7f8528076d70 msgr2=0x7f8528079190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.374+0000 7f852def4640 1 -- 172.21.15.70:0/4018163194 shutdown_connections 2024-04-01T01:36:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.375+0000 7f852def4640 1 -- 172.21.15.70:0/4018163194 wait complete. 2024-04-01T01:36:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.375+0000 7f852def4640 1 Processor -- start 2024-04-01T01:36:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.376+0000 7f852def4640 1 -- start start 2024-04-01T01:36:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.376+0000 7f852def4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8528107db0 0x7f852819df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.376+0000 7f852def4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8528108fa0 0x7f852819e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.376+0000 7f852def4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8528109960 0x7f85281a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.376+0000 7f852def4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8528113930 con 0x7f8528109960 2024-04-01T01:36:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.376+0000 7f852def4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f85281137b0 con 0x7f8528108fa0 2024-04-01T01:36:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.376+0000 7f852def4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8528113ab0 con 0x7f8528107db0 2024-04-01T01:36:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.376+0000 7f8526ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8528108fa0 0x7f852819e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.376+0000 7f8526ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8528108fa0 0x7f852819e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37146/0 (socket says 172.21.15.70:37146) 2024-04-01T01:36:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.376+0000 7f8526ffd640 1 -- 172.21.15.70:0/1171144275 learned_addr learned my addr 172.21.15.70:0/1171144275 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.377+0000 7f85277fe640 1 --2- 172.21.15.70:0/1171144275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8528107db0 0x7f852819df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.377+0000 7f8527fff640 1 --2- 172.21.15.70:0/1171144275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8528109960 0x7f85281a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.377+0000 7f8526ffd640 1 -- 172.21.15.70:0/1171144275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8528107db0 msgr2=0x7f852819df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.377+0000 7f8526ffd640 1 --2- 172.21.15.70:0/1171144275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8528107db0 0x7f852819df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.377+0000 7f8526ffd640 1 -- 172.21.15.70:0/1171144275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8528109960 msgr2=0x7f85281a54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.377+0000 7f8526ffd640 1 --2- 172.21.15.70:0/1171144275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8528109960 0x7f85281a54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.377+0000 7f8526ffd640 1 -- 172.21.15.70:0/1171144275 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85281a5b60 con 0x7f8528108fa0 2024-04-01T01:36:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.377+0000 7f8526ffd640 1 --2- 172.21.15.70:0/1171144275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8528108fa0 0x7f852819e480 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8518000c00 tx=0x7f85180040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.378+0000 7f8524ff9640 1 -- 172.21.15.70:0/1171144275 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8518043020 con 0x7f8528108fa0 2024-04-01T01:36:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.378+0000 7f8524ff9640 1 -- 172.21.15.70:0/1171144275 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85180044c0 con 0x7f8528108fa0 2024-04-01T01:36:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.378+0000 7f8524ff9640 1 -- 172.21.15.70:0/1171144275 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8518003c60 con 0x7f8528108fa0 2024-04-01T01:36:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.378+0000 7f852def4640 1 -- 172.21.15.70:0/1171144275 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85281a5df0 con 0x7f8528108fa0 2024-04-01T01:36:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.379+0000 7f852def4640 1 -- 172.21.15.70:0/1171144275 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85281a61a0 con 0x7f8528108fa0 2024-04-01T01:36:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.382+0000 7f8524ff9640 1 -- 172.21.15.70:0/1171144275 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f8518046020 con 0x7f8528108fa0 2024-04-01T01:36:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.383+0000 7f8530bb2740 1 Processor -- start 2024-04-01T01:36:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.383+0000 7f8530bb2740 1 -- start start 2024-04-01T01:36:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.383+0000 7f8530bb2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d2198d0 0x55771d219cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.384+0000 7f8530bb2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d21a2c0 0x55771d21edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.384+0000 7f8530bb2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21f430 0x55771d2218c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.384+0000 7f8530bb2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55771d17d8f0 con 0x55771d2198d0 2024-04-01T01:36:50.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.384+0000 7f8530bb2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55771d1268d0 con 0x55771d21a2c0 2024-04-01T01:36:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.384+0000 7f8530bb2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55771d179990 con 0x55771d21f430 2024-04-01T01:36:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.384+0000 7f8527fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d2198d0 0x55771d219cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.384+0000 7f8526ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21f430 0x55771d2218c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.384+0000 7f8527fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d2198d0 0x55771d219cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48856/0 (socket says 172.21.15.70:48856) 2024-04-01T01:36:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.384+0000 7f8527fff640 1 -- 172.21.15.70:0/1186615771 learned_addr learned my addr 172.21.15.70:0/1186615771 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.384+0000 7f8526ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21f430 0x55771d2218c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:60688/0 (socket says 172.21.15.70:60688) 2024-04-01T01:36:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.384+0000 7f85277fe640 1 --2- 172.21.15.70:0/1186615771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d21a2c0 0x55771d21edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.384+0000 7f8524ff9640 1 --2- 172.21.15.70:0/1171144275 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85140a26b0 0x7f85140a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.385+0000 7f8524ff9640 1 -- 172.21.15.70:0/1171144275 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f85180f0d50 con 0x7f8528108fa0 2024-04-01T01:36:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.385+0000 7f8527fff640 1 --2- 172.21.15.70:0/1171144275 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85140a26b0 0x7f85140a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.385+0000 7f8527fff640 1 -- 172.21.15.70:0/1186615771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21f430 msgr2=0x55771d2218c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.385+0000 7f8527fff640 1 --2- 172.21.15.70:0/1186615771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21f430 0x55771d2218c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.385+0000 7f8527fff640 1 -- 172.21.15.70:0/1186615771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d21a2c0 msgr2=0x55771d21edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.385+0000 7f8527fff640 1 --2- 172.21.15.70:0/1186615771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d21a2c0 0x55771d21edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.385+0000 7f8527fff640 1 -- 172.21.15.70:0/1186615771 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55771ce92210 con 0x55771d2198d0 2024-04-01T01:36:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.385+0000 7f85277fe640 1 --2- 172.21.15.70:0/1186615771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d21a2c0 0x55771d21edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:36:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.385+0000 7f8527fff640 1 --2- 172.21.15.70:0/1186615771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d2198d0 0x55771d219cf0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f8520014720 tx=0x7f8520014b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38cf0d6a5fba6cc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.386+0000 7f8527fff640 1 --2- 172.21.15.70:0/1171144275 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85140a26b0 0x7f85140a4b70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f852001b000 tx=0x7f852001d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.386+0000 7f851dffb640 1 -- 172.21.15.70:0/1186615771 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8520003970 con 0x55771d2198d0 2024-04-01T01:36:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.386+0000 7f851dffb640 1 -- 172.21.15.70:0/1186615771 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f852000ed00 con 0x55771d2198d0 2024-04-01T01:36:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.386+0000 7f851dffb640 1 -- 172.21.15.70:0/1186615771 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8520011ba0 con 0x55771d2198d0 2024-04-01T01:36:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.387+0000 7f8530bb2740 1 -- 172.21.15.70:0/1186615771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d2198d0 msgr2=0x55771d219cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.387+0000 7f8530bb2740 1 --2- 172.21.15.70:0/1186615771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d2198d0 0x55771d219cf0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f8520014720 tx=0x7f8520014b20 comp rx=0 tx=0).stop 2024-04-01T01:36:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.387+0000 7f8530bb2740 1 -- 172.21.15.70:0/1186615771 shutdown_connections 2024-04-01T01:36:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.387+0000 7f8530bb2740 1 --2- 172.21.15.70:0/1186615771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21f430 0x55771d2218c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.387+0000 7f8530bb2740 1 --2- 172.21.15.70:0/1186615771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d21a2c0 0x55771d21edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.387+0000 7f8530bb2740 1 --2- 172.21.15.70:0/1186615771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d2198d0 0x55771d219cf0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.387+0000 7f8530bb2740 1 -- 172.21.15.70:0/1186615771 >> 172.21.15.70:0/1186615771 conn(0x55771d214e40 msgr2=0x55771d215240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.388+0000 7f8530bb2740 1 -- 172.21.15.70:0/1186615771 shutdown_connections 2024-04-01T01:36:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.388+0000 7f8530bb2740 1 -- 172.21.15.70:0/1186615771 wait complete. 2024-04-01T01:36:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.388+0000 7f8530bb2740 1 -- start start 2024-04-01T01:36:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.388+0000 7f8530bb2740 1 Processor -- start 2024-04-01T01:36:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.389+0000 7f8530bb2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d2198d0 0x55771d223c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.389+0000 7f8530bb2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21a2c0 0x55771d2241c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.389+0000 7f8530bb2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d21f430 0x55771d22b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.389+0000 7f8530bb2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55771d222380 con 0x55771d21f430 2024-04-01T01:36:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.389+0000 7f8530bb2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55771d11bf80 con 0x55771d2198d0 2024-04-01T01:36:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.389+0000 7f8530bb2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55771d222500 con 0x55771d21a2c0 2024-04-01T01:36:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.389+0000 7f8530bb2740 10 client.0 _ll_register_callbacks cb 0x55771d174290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.389+0000 7f85277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d2198d0 0x55771d223c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.389+0000 7f85277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d2198d0 0x55771d223c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37160/0 (socket says 172.21.15.70:37160) 2024-04-01T01:36:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.389+0000 7f85277fe640 1 -- 172.21.15.70:0/2555797939 learned_addr learned my addr 172.21.15.70:0/2555797939 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.389+0000 7f8526ffd640 1 --2- 172.21.15.70:0/2555797939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21a2c0 0x55771d2241c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.389+0000 7f8527fff640 1 --2- 172.21.15.70:0/2555797939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d21f430 0x55771d22b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.389+0000 7f85277fe640 1 -- 172.21.15.70:0/2555797939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21a2c0 msgr2=0x55771d2241c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:50.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.389+0000 7f85277fe640 1 --2- 172.21.15.70:0/2555797939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21a2c0 0x55771d2241c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.389+0000 7f85277fe640 1 -- 172.21.15.70:0/2555797939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d21f430 msgr2=0x55771d22b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.389+0000 7f85277fe640 1 --2- 172.21.15.70:0/2555797939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d21f430 0x55771d22b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.390+0000 7f85277fe640 1 -- 172.21.15.70:0/2555797939 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55771cf5e620 con 0x55771d2198d0 2024-04-01T01:36:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.390+0000 7f85277fe640 1 --2- 172.21.15.70:0/2555797939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d2198d0 0x55771d223c80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f85100124d0 tx=0x7f8510012990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b637571a7d948542 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.390+0000 7f8506ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:36:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.390+0000 7f8506ffd640 1 -- 172.21.15.70:0/2555797939 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851001f060 con 0x55771d2198d0 2024-04-01T01:36:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.390+0000 7f8530bb2740 1 -- 172.21.15.70:0/2555797939 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f852000ed00 con 0x55771d2198d0 2024-04-01T01:36:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.390+0000 7f8506ffd640 1 -- 172.21.15.70:0/2555797939 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8510004460 con 0x55771d2198d0 2024-04-01T01:36:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.391+0000 7f8506ffd640 1 -- 172.21.15.70:0/2555797939 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8510026050 con 0x55771d2198d0 2024-04-01T01:36:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.391+0000 7f8530bb2740 10 client.26441 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:50.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.391+0000 7f8530bb2740 1 -- 172.21.15.70:0/2555797939 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55771d22e250 con 0x55771d2198d0 2024-04-01T01:36:50.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.391+0000 7f8506ffd640 1 -- 172.21.15.70:0/2555797939 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f852000ed00 con 0x55771d2198d0 2024-04-01T01:36:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.391+0000 7f8506ffd640 1 -- 172.21.15.70:0/2555797939 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8510010a50 con 0x55771d2198d0 2024-04-01T01:36:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.391+0000 7f8530bb2740 10 client.26441 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.391+0000 7f8530bb2740 10 client.26441 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:36:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.391+0000 7f8530bb2740 20 client.26441 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.391+0000 7f8530bb2740 1 --2- 172.21.15.70:0/2555797939 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55771d22f8d0 0x55771d231d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.391+0000 7f8530bb2740 4 client.26441 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:36:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.391+0000 7f8530bb2740 1 -- 172.21.15.70:0/2555797939 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55771d191120 con 0x55771d22f8d0 2024-04-01T01:36:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.392+0000 7f8526ffd640 1 --2- 172.21.15.70:0/2555797939 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55771d22f8d0 0x55771d231d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.392+0000 7f8526ffd640 1 --2- 172.21.15.70:0/2555797939 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55771d22f8d0 0x55771d231d20 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=896a5fc927c5e399 server_cookie=cda2359d72567b72 in_seq=0 out_seq=0 2024-04-01T01:36:50.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.392+0000 7f8506ffd640 10 client.26441 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:50.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.394+0000 7f8506ffd640 1 -- 172.21.15.70:0/2555797939 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55771d191120 con 0x55771d22f8d0 2024-04-01T01:36:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.394+0000 7f8506ffd640 10 client.26441 handle_command_reply: tid=0 2024-04-01T01:36:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.394+0000 7f8530bb2740 1 client.26441 shutdown 2024-04-01T01:36:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.395+0000 7f8530bb2740 1 -- 172.21.15.70:0/2555797939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d2198d0 msgr2=0x55771d223c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.395+0000 7f8530bb2740 1 --2- 172.21.15.70:0/2555797939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d2198d0 0x55771d223c80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f85100124d0 tx=0x7f8510012990 comp rx=0 tx=0).stop 2024-04-01T01:36:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.395+0000 7f8530bb2740 1 -- 172.21.15.70:0/2555797939 shutdown_connections 2024-04-01T01:36:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.395+0000 7f8530bb2740 1 --2- 172.21.15.70:0/2555797939 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55771d22f8d0 0x55771d231d20 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.395+0000 7f8530bb2740 1 --2- 172.21.15.70:0/2555797939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d21f430 0x55771d22b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.395+0000 7f8530bb2740 1 --2- 172.21.15.70:0/2555797939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21a2c0 0x55771d2241c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.395+0000 7f8530bb2740 1 --2- 172.21.15.70:0/2555797939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d2198d0 0x55771d223c80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.395+0000 7f8530bb2740 1 -- 172.21.15.70:0/2555797939 >> 172.21.15.70:0/2555797939 conn(0x55771d214e40 msgr2=0x55771d213f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.395+0000 7f8506ffd640 0 client.26441 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.395+0000 7f8530bb2740 1 -- 172.21.15.70:0/2555797939 shutdown_connections 2024-04-01T01:36:50.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.395+0000 7f8530bb2740 1 -- 172.21.15.70:0/2555797939 wait complete. 2024-04-01T01:36:50.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.395+0000 7f8530bb2740 20 client.26441 trim_cache size 0 max 16384 2024-04-01T01:36:50.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.421+0000 7f8530bb2740 1 Processor -- start 2024-04-01T01:36:50.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.421+0000 7f8530bb2740 1 -- start start 2024-04-01T01:36:50.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.421+0000 7f8530bb2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d2198d0 0x55771d223a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.421+0000 7f8530bb2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21a2c0 0x55771d22b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.422+0000 7f8530bb2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d21f430 0x55771d223f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.422+0000 7f85277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d2198d0 0x55771d223a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.422+0000 7f85277fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d2198d0 0x55771d223a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37174/0 (socket says 172.21.15.70:37174) 2024-04-01T01:36:50.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.422+0000 7f85277fe640 1 -- 172.21.15.70:0/3451958981 learned_addr learned my addr 172.21.15.70:0/3451958981 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:50.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.422+0000 7f8530bb2740 1 -- 172.21.15.70:0/3451958981 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f85281b08f0 con 0x55771d21f430 2024-04-01T01:36:50.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.422+0000 7f8530bb2740 1 -- 172.21.15.70:0/3451958981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f85281b2430 con 0x55771d2198d0 2024-04-01T01:36:50.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.422+0000 7f8530bb2740 1 -- 172.21.15.70:0/3451958981 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f85281b0fc0 con 0x55771d21a2c0 2024-04-01T01:36:50.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.422+0000 7f8527fff640 1 --2- 172.21.15.70:0/3451958981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d21f430 0x55771d223f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.422+0000 7f85277fe640 1 -- 172.21.15.70:0/3451958981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21a2c0 msgr2=0x55771d22b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:50.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.422+0000 7f85277fe640 1 --2- 172.21.15.70:0/3451958981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21a2c0 0x55771d22b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.423+0000 7f85277fe640 1 -- 172.21.15.70:0/3451958981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d21f430 msgr2=0x55771d223f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:50.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.423+0000 7f85277fe640 1 --2- 172.21.15.70:0/3451958981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d21f430 0x55771d223f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.423+0000 7f85277fe640 1 -- 172.21.15.70:0/3451958981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55771d219cb0 con 0x55771d2198d0 2024-04-01T01:36:50.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.423+0000 7f85277fe640 1 --2- 172.21.15.70:0/3451958981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d2198d0 0x55771d223a10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8510004020 tx=0x7f851001a640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5f43e82749990a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:50.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.423+0000 7f851d7fa640 1 -- 172.21.15.70:0/3451958981 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85100054e0 con 0x55771d2198d0 2024-04-01T01:36:50.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.423+0000 7f851d7fa640 1 -- 172.21.15.70:0/3451958981 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f851001ad50 con 0x55771d2198d0 2024-04-01T01:36:50.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.423+0000 7f851d7fa640 1 -- 172.21.15.70:0/3451958981 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85100035f0 con 0x55771d2198d0 2024-04-01T01:36:50.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.425+0000 7f8530bb2740 1 -- 172.21.15.70:0/3451958981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d2198d0 msgr2=0x55771d223a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:50.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.425+0000 7f8530bb2740 1 --2- 172.21.15.70:0/3451958981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d2198d0 0x55771d223a10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8510004020 tx=0x7f851001a640 comp rx=0 tx=0).stop 2024-04-01T01:36:50.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.425+0000 7f8530bb2740 1 -- 172.21.15.70:0/3451958981 shutdown_connections 2024-04-01T01:36:50.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.425+0000 7f8530bb2740 1 --2- 172.21.15.70:0/3451958981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d21f430 0x55771d223f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.425+0000 7f8530bb2740 1 --2- 172.21.15.70:0/3451958981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21a2c0 0x55771d22b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.425+0000 7f8530bb2740 1 --2- 172.21.15.70:0/3451958981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d2198d0 0x55771d223a10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.426+0000 7f8530bb2740 1 -- 172.21.15.70:0/3451958981 >> 172.21.15.70:0/3451958981 conn(0x55771d214e40 msgr2=0x55771d215220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:50.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.426+0000 7f8530bb2740 1 -- 172.21.15.70:0/3451958981 shutdown_connections 2024-04-01T01:36:50.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.426+0000 7f8530bb2740 1 -- 172.21.15.70:0/3451958981 wait complete. 2024-04-01T01:36:50.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.426+0000 7f8530bb2740 1 -- start start 2024-04-01T01:36:50.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.427+0000 7f8530bb2740 1 Processor -- start 2024-04-01T01:36:50.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.427+0000 7f8530bb2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d2198d0 0x55771d228b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.427+0000 7f8530bb2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d21a2c0 0x55771d2290d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.427+0000 7f8530bb2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21f430 0x55771d22a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.427+0000 7f8530bb2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f85281b1d60 con 0x55771d2198d0 2024-04-01T01:36:50.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.427+0000 7f8530bb2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f85281b1690 con 0x55771d21a2c0 2024-04-01T01:36:50.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.427+0000 7f8530bb2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f85281ae8d0 con 0x55771d21f430 2024-04-01T01:36:50.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.427+0000 7f85277fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d2198d0 0x55771d228b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.427+0000 7f8526ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d21a2c0 0x55771d2290d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.427+0000 7f85277fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d2198d0 0x55771d228b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48884/0 (socket says 172.21.15.70:48884) 2024-04-01T01:36:50.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.428+0000 7f85277fe640 1 -- 172.21.15.70:0/4193731549 learned_addr learned my addr 172.21.15.70:0/4193731549 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:36:50.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.427+0000 7f8526ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d21a2c0 0x55771d2290d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37176/0 (socket says 172.21.15.70:37176) 2024-04-01T01:36:50.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.428+0000 7f8527fff640 1 --2- 172.21.15.70:0/4193731549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21f430 0x55771d22a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.428+0000 7f8530bb2740 10 client.0 _ll_register_callbacks cb 0x7f8510009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:36:50.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.428+0000 7f8526ffd640 1 -- 172.21.15.70:0/4193731549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21f430 msgr2=0x55771d22a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:50.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.428+0000 7f8526ffd640 1 --2- 172.21.15.70:0/4193731549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21f430 0x55771d22a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.428+0000 7f8526ffd640 1 -- 172.21.15.70:0/4193731549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d2198d0 msgr2=0x55771d228b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:36:50.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.428+0000 7f8526ffd640 1 --2- 172.21.15.70:0/4193731549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d2198d0 0x55771d228b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.428+0000 7f8526ffd640 1 -- 172.21.15.70:0/4193731549 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55771d22a6d0 con 0x55771d21a2c0 2024-04-01T01:36:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.428+0000 7f8526ffd640 1 --2- 172.21.15.70:0/4193731549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d21a2c0 0x55771d2290d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f85180ee4d0 tx=0x7f85180bcec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b909d941870f8c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:36:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.429+0000 7f8506ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:36:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.429+0000 7f8506ffd640 1 -- 172.21.15.70:0/4193731549 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851803c350 con 0x55771d21a2c0 2024-04-01T01:36:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.429+0000 7f8506ffd640 1 -- 172.21.15.70:0/4193731549 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8518003e00 con 0x55771d21a2c0 2024-04-01T01:36:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.429+0000 7f8530bb2740 1 -- 172.21.15.70:0/4193731549 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f851001ad50 con 0x55771d21a2c0 2024-04-01T01:36:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.429+0000 7f8506ffd640 1 -- 172.21.15.70:0/4193731549 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f85180bdc60 con 0x55771d21a2c0 2024-04-01T01:36:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.429+0000 7f8530bb2740 10 client.26447 fetch_fsmap learned FSMap version 29 2024-04-01T01:36:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.429+0000 7f8530bb2740 1 -- 172.21.15.70:0/4193731549 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55771d2a6fe0 con 0x55771d21a2c0 2024-04-01T01:36:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.430+0000 7f8506ffd640 1 -- 172.21.15.70:0/4193731549 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85180bd560 con 0x55771d21a2c0 2024-04-01T01:36:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.430+0000 7f8506ffd640 1 -- 172.21.15.70:0/4193731549 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f85180b9bd0 con 0x55771d21a2c0 2024-04-01T01:36:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.430+0000 7f8530bb2740 10 client.26447 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:36:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.430+0000 7f8530bb2740 10 client.26447 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:36:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.430+0000 7f8530bb2740 20 client.26447 populate_metadata read hostname 'smithi070' 2024-04-01T01:36:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.430+0000 7f8530bb2740 1 --2- 172.21.15.70:0/4193731549 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55771d2a7940 0x55771d2a9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:36:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.430+0000 7f8530bb2740 4 client.26447 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:36:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.430+0000 7f8530bb2740 1 -- 172.21.15.70:0/4193731549 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f851001ad50 con 0x55771d2a7940 2024-04-01T01:36:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.431+0000 7f85277fe640 1 --2- 172.21.15.70:0/4193731549 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55771d2a7940 0x55771d2a9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:36:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.431+0000 7f85277fe640 1 --2- 172.21.15.70:0/4193731549 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55771d2a7940 0x55771d2a9d60 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f58cb2d47a323cef server_cookie=ed12e089b05f098f in_seq=0 out_seq=0 2024-04-01T01:36:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.431+0000 7f8506ffd640 10 client.26447 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:50.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.470+0000 7f8506ffd640 1 -- 172.21.15.70:0/4193731549 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f851001ad50 con 0x55771d2a7940 2024-04-01T01:36:50.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.470+0000 7f8506ffd640 10 client.26447 handle_command_reply: tid=0 2024-04-01T01:36:50.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.470+0000 7f8530bb2740 1 client.26447 shutdown 2024-04-01T01:36:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.471+0000 7f8530bb2740 1 -- 172.21.15.70:0/4193731549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d21a2c0 msgr2=0x55771d2290d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.471+0000 7f8530bb2740 1 --2- 172.21.15.70:0/4193731549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d21a2c0 0x55771d2290d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f85180ee4d0 tx=0x7f85180bcec0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.471+0000 7f8530bb2740 1 -- 172.21.15.70:0/4193731549 shutdown_connections 2024-04-01T01:36:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.471+0000 7f8530bb2740 1 --2- 172.21.15.70:0/4193731549 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55771d2a7940 0x55771d2a9d60 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.471+0000 7f8530bb2740 1 --2- 172.21.15.70:0/4193731549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55771d21f430 0x55771d22a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.471+0000 7f8506ffd640 0 client.26447 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:36:50.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.471+0000 7f8530bb2740 1 --2- 172.21.15.70:0/4193731549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55771d21a2c0 0x55771d2290d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.471+0000 7f8530bb2740 1 --2- 172.21.15.70:0/4193731549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55771d2198d0 0x55771d228b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.471+0000 7f8530bb2740 1 -- 172.21.15.70:0/4193731549 >> 172.21.15.70:0/4193731549 conn(0x55771d214e40 msgr2=0x55771d2239f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:50.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.473+0000 7f8530bb2740 1 -- 172.21.15.70:0/4193731549 shutdown_connections 2024-04-01T01:36:50.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.473+0000 7f8530bb2740 1 -- 172.21.15.70:0/4193731549 wait complete. 2024-04-01T01:36:50.474 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.474+0000 7f8530bb2740 20 client.26447 trim_cache size 0 max 16384 2024-04-01T01:36:50.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:36:50.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:36:50.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:36:50.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:36:50.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.475+0000 7f852def4640 1 -- 172.21.15.70:0/1171144275 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85140a26b0 msgr2=0x7f85140a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:50.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.475+0000 7f852def4640 1 --2- 172.21.15.70:0/1171144275 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85140a26b0 0x7f85140a4b70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f852001b000 tx=0x7f852001d000 comp rx=0 tx=0).stop 2024-04-01T01:36:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.475+0000 7f852def4640 1 -- 172.21.15.70:0/1171144275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8528108fa0 msgr2=0x7f852819e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:36:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.475+0000 7f852def4640 1 --2- 172.21.15.70:0/1171144275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8528108fa0 0x7f852819e480 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8518000c00 tx=0x7f85180040c0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.476+0000 7f852def4640 1 -- 172.21.15.70:0/1171144275 shutdown_connections 2024-04-01T01:36:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.476+0000 7f852def4640 1 --2- 172.21.15.70:0/1171144275 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8528109960 0x7f85281a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.476+0000 7f852def4640 1 --2- 172.21.15.70:0/1171144275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8528108fa0 0x7f852819e480 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.476+0000 7f852def4640 1 --2- 172.21.15.70:0/1171144275 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85140a26b0 0x7f85140a4b70 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.476+0000 7f852def4640 1 --2- 172.21.15.70:0/1171144275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8528107db0 0x7f852819df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:36:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.476+0000 7f852def4640 1 -- 172.21.15.70:0/1171144275 >> 172.21.15.70:0/1171144275 conn(0x7f8528076d70 msgr2=0x7f852810c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:36:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.477+0000 7f852def4640 1 -- 172.21.15.70:0/1171144275 shutdown_connections 2024-04-01T01:36:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:36:50.477+0000 7f852def4640 1 -- 172.21.15.70:0/1171144275 wait complete. 2024-04-01T01:36:51.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 125.0 s, 1221.4 tps, lat 81.744 ms stddev 28.001 2024-04-01T01:36:51.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:51 smithi189 ceph-mon[35753]: pgmap v459: 129 pgs: 129 active+clean; 1008 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 11 MiB/s wr, 1.12k op/s 2024-04-01T01:36:52.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:51 smithi070 ceph-mon[32179]: pgmap v459: 129 pgs: 129 active+clean; 1008 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 11 MiB/s wr, 1.12k op/s 2024-04-01T01:36:52.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:51 smithi081 ceph-mon[35142]: pgmap v459: 129 pgs: 129 active+clean; 1008 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 11 MiB/s wr, 1.12k op/s 2024-04-01T01:36:53.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:53 smithi189 ceph-mon[35753]: pgmap v460: 129 pgs: 129 active+clean; 1011 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 12 MiB/s wr, 1.26k op/s 2024-04-01T01:36:54.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:53 smithi070 ceph-mon[32179]: pgmap v460: 129 pgs: 129 active+clean; 1011 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 12 MiB/s wr, 1.26k op/s 2024-04-01T01:36:54.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:53 smithi081 ceph-mon[35142]: pgmap v460: 129 pgs: 129 active+clean; 1011 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 12 MiB/s wr, 1.26k op/s 2024-04-01T01:36:55.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:55 smithi189 ceph-mon[35753]: pgmap v461: 129 pgs: 129 active+clean; 1011 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 1.09k op/s 2024-04-01T01:36:56.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:55 smithi070 ceph-mon[32179]: pgmap v461: 129 pgs: 129 active+clean; 1011 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 1.09k op/s 2024-04-01T01:36:56.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:55 smithi081 ceph-mon[35142]: pgmap v461: 129 pgs: 129 active+clean; 1011 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 10 MiB/s wr, 1.09k op/s 2024-04-01T01:36:56.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 130.0 s, 1312.4 tps, lat 76.125 ms stddev 24.971 2024-04-01T01:36:57.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:57 smithi189 ceph-mon[35753]: pgmap v462: 129 pgs: 129 active+clean; 1011 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-04-01T01:36:57.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:57 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:36:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:57 smithi070 ceph-mon[32179]: pgmap v462: 129 pgs: 129 active+clean; 1011 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-04-01T01:36:58.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:57 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:36:58.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:57 smithi081 ceph-mon[35142]: pgmap v462: 129 pgs: 129 active+clean; 1011 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-04-01T01:36:58.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:57 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:36:59.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:58 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:36:59.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:58 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:36:59.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:58 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:36:59.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:58 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:36:59.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:58 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:36:59.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:58 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:36:59.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:58 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:36:59.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:58 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:36:59.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:58 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:37:00.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:36:59 smithi070 ceph-mon[32179]: pgmap v463: 129 pgs: 129 active+clean; 1011 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-04-01T01:37:00.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:36:59 smithi081 ceph-mon[35142]: pgmap v463: 129 pgs: 129 active+clean; 1011 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-04-01T01:37:00.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:36:59 smithi189 ceph-mon[35753]: pgmap v463: 129 pgs: 129 active+clean; 1011 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-04-01T01:37:01.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 135.0 s, 1113.8 tps, lat 89.752 ms stddev 29.736 2024-04-01T01:37:02.049 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:01 smithi189 ceph-mon[35753]: pgmap v464: 129 pgs: 129 active+clean; 1011 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 10 MiB/s wr, 1.15k op/s 2024-04-01T01:37:02.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:01 smithi081 ceph-mon[35142]: pgmap v464: 129 pgs: 129 active+clean; 1011 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 10 MiB/s wr, 1.15k op/s 2024-04-01T01:37:02.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:01 smithi070 ceph-mon[32179]: pgmap v464: 129 pgs: 129 active+clean; 1011 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 10 MiB/s wr, 1.15k op/s 2024-04-01T01:37:02.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:37:02.472 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:37:02.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.652+0000 7f860755e640 1 -- 172.21.15.70:0/3190711211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8600104a80 msgr2=0x7f8600104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:02.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.652+0000 7f860755e640 1 --2- 172.21.15.70:0/3190711211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8600104a80 0x7f8600104f00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009900 tx=0x7f85f002bc30 comp rx=0 tx=0).stop 2024-04-01T01:37:02.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.653+0000 7f860755e640 1 -- 172.21.15.70:0/3190711211 shutdown_connections 2024-04-01T01:37:02.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.653+0000 7f860755e640 1 --2- 172.21.15.70:0/3190711211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8600105440 0x7f860010bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.653+0000 7f860755e640 1 --2- 172.21.15.70:0/3190711211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8600104a80 0x7f8600104f00 secure :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f85f0009900 tx=0x7f85f002bc30 comp rx=0 tx=0).stop 2024-04-01T01:37:02.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.654+0000 7f860755e640 1 --2- 172.21.15.70:0/3190711211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8600103890 0x7f8600103cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.654+0000 7f860755e640 1 -- 172.21.15.70:0/3190711211 >> 172.21.15.70:0/3190711211 conn(0x7f86000fee00 msgr2=0x7f8600101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:02.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.655+0000 7f860755e640 1 -- 172.21.15.70:0/3190711211 shutdown_connections 2024-04-01T01:37:02.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.655+0000 7f860755e640 1 -- 172.21.15.70:0/3190711211 wait complete. 2024-04-01T01:37:02.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.656+0000 7f860755e640 1 Processor -- start 2024-04-01T01:37:02.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.656+0000 7f860755e640 1 -- start start 2024-04-01T01:37:02.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.657+0000 7f860755e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8600103890 0x7f86001958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.657+0000 7f860755e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8600105440 0x7f8600195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.657+0000 7f86052fa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8600103890 0x7f86001958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.657+0000 7f86052fa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8600103890 0x7f86001958a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56520/0 (socket says 172.21.15.70:56520) 2024-04-01T01:37:02.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.657+0000 7f86052fa640 1 -- 172.21.15.70:0/586259608 learned_addr learned my addr 172.21.15.70:0/586259608 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:02.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.657+0000 7f860755e640 1 --2- 172.21.15.70:0/586259608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f860019ae40 0x7f860019d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.657+0000 7f8604af9640 1 --2- 172.21.15.70:0/586259608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8600105440 0x7f8600195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.657+0000 7f860755e640 1 -- 172.21.15.70:0/586259608 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8600078690 con 0x7f8600103890 2024-04-01T01:37:02.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.657+0000 7f860755e640 1 -- 172.21.15.70:0/586259608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8600078510 con 0x7f860019ae40 2024-04-01T01:37:02.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.657+0000 7f860755e640 1 -- 172.21.15.70:0/586259608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8600078810 con 0x7f8600105440 2024-04-01T01:37:02.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.658+0000 7f8605afb640 1 --2- 172.21.15.70:0/586259608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f860019ae40 0x7f860019d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.658+0000 7f86052fa640 1 -- 172.21.15.70:0/586259608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8600105440 msgr2=0x7f8600195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:02.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.658+0000 7f86052fa640 1 --2- 172.21.15.70:0/586259608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8600105440 0x7f8600195de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.658+0000 7f86052fa640 1 -- 172.21.15.70:0/586259608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f860019ae40 msgr2=0x7f860019d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:02.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.658+0000 7f86052fa640 1 --2- 172.21.15.70:0/586259608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f860019ae40 0x7f860019d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.658+0000 7f86052fa640 1 -- 172.21.15.70:0/586259608 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860019d8a0 con 0x7f8600103890 2024-04-01T01:37:02.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.658+0000 7f86052fa640 1 --2- 172.21.15.70:0/586259608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8600103890 0x7f86001958a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009fd0 tx=0x7f85fc007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:02.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.659+0000 7f85f67fc640 1 -- 172.21.15.70:0/586259608 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc004020 con 0x7f8600103890 2024-04-01T01:37:02.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.659+0000 7f85f67fc640 1 -- 172.21.15.70:0/586259608 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85fc0041c0 con 0x7f8600103890 2024-04-01T01:37:02.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.659+0000 7f85f67fc640 1 -- 172.21.15.70:0/586259608 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc01c7c0 con 0x7f8600103890 2024-04-01T01:37:02.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.659+0000 7f860755e640 1 -- 172.21.15.70:0/586259608 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860019db90 con 0x7f8600103890 2024-04-01T01:37:02.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.659+0000 7f860755e640 1 -- 172.21.15.70:0/586259608 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f860019dfa0 con 0x7f8600103890 2024-04-01T01:37:02.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.663+0000 7f860a218740 1 Processor -- start 2024-04-01T01:37:02.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.664+0000 7f860a218740 1 -- start start 2024-04-01T01:37:02.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.664+0000 7f860a218740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be8434a0 0x55a0be8438c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.664+0000 7f860a218740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be843e90 0x55a0be848990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.664+0000 7f860a218740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be849000 0x55a0be84b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.664+0000 7f860a218740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a0be797180 con 0x55a0be8434a0 2024-04-01T01:37:02.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.664+0000 7f860a218740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a0be73fb70 con 0x55a0be843e90 2024-04-01T01:37:02.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.664+0000 7f860a218740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a0be793220 con 0x55a0be849000 2024-04-01T01:37:02.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.665+0000 7f8605afb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be8434a0 0x55a0be8438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.665+0000 7f8605afb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be8434a0 0x55a0be8438c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56522/0 (socket says 172.21.15.70:56522) 2024-04-01T01:37:02.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.665+0000 7f8605afb640 1 -- 172.21.15.70:0/3465410310 learned_addr learned my addr 172.21.15.70:0/3465410310 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:02.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.665+0000 7f86052fa640 1 --2- 172.21.15.70:0/3465410310 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be843e90 0x55a0be848990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.665+0000 7f8605afb640 1 -- 172.21.15.70:0/3465410310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be849000 msgr2=0x55a0be84b490 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:37:02.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.665+0000 7f8605afb640 1 --2- 172.21.15.70:0/3465410310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be849000 0x55a0be84b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.665+0000 7f8605afb640 1 -- 172.21.15.70:0/3465410310 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be843e90 msgr2=0x55a0be848990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:02.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.665+0000 7f8605afb640 1 --2- 172.21.15.70:0/3465410310 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be843e90 0x55a0be848990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.665+0000 7f8605afb640 1 -- 172.21.15.70:0/3465410310 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0be7537c0 con 0x55a0be8434a0 2024-04-01T01:37:02.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.665+0000 7f85f67fc640 1 -- 172.21.15.70:0/586259608 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f85fc024030 con 0x7f8600103890 2024-04-01T01:37:02.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.666+0000 7f8605afb640 1 --2- 172.21.15.70:0/3465410310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be8434a0 0x55a0be8438c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f85f8032380 tx=0x7f85f8032840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a223f3ad454ef6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:02.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.666+0000 7f85eb7fe640 1 -- 172.21.15.70:0/3465410310 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f8004d20 con 0x55a0be8434a0 2024-04-01T01:37:02.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.666+0000 7f85eb7fe640 1 -- 172.21.15.70:0/3465410310 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85f80054d0 con 0x55a0be8434a0 2024-04-01T01:37:02.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.666+0000 7f85eb7fe640 1 -- 172.21.15.70:0/3465410310 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f80339a0 con 0x55a0be8434a0 2024-04-01T01:37:02.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.667+0000 7f85f67fc640 1 --2- 172.21.15.70:0/586259608 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85ec0a2670 0x7f85ec0a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.667+0000 7f85f67fc640 1 -- 172.21.15.70:0/586259608 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f85fc0d6720 con 0x7f8600103890 2024-04-01T01:37:02.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.667+0000 7f860a218740 1 -- 172.21.15.70:0/3465410310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be8434a0 msgr2=0x55a0be8438c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:02.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.667+0000 7f860a218740 1 --2- 172.21.15.70:0/3465410310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be8434a0 0x55a0be8438c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f85f8032380 tx=0x7f85f8032840 comp rx=0 tx=0).stop 2024-04-01T01:37:02.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.667+0000 7f86052fa640 1 --2- 172.21.15.70:0/586259608 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85ec0a2670 0x7f85ec0a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.668+0000 7f860a218740 1 -- 172.21.15.70:0/3465410310 shutdown_connections 2024-04-01T01:37:02.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.668+0000 7f860a218740 1 --2- 172.21.15.70:0/3465410310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be849000 0x55a0be84b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.668+0000 7f860a218740 1 --2- 172.21.15.70:0/3465410310 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be843e90 0x55a0be848990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.668+0000 7f860a218740 1 --2- 172.21.15.70:0/3465410310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be8434a0 0x55a0be8438c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.668+0000 7f860a218740 1 -- 172.21.15.70:0/3465410310 >> 172.21.15.70:0/3465410310 conn(0x55a0be82da00 msgr2=0x55a0be82de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:02.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.668+0000 7f86052fa640 1 --2- 172.21.15.70:0/586259608 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85ec0a2670 0x7f85ec0a4b30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f85fc0a2b40 tx=0x7f85fc0dc530 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:02.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.668+0000 7f860a218740 1 -- 172.21.15.70:0/3465410310 shutdown_connections 2024-04-01T01:37:02.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.669+0000 7f860a218740 1 -- 172.21.15.70:0/3465410310 wait complete. 2024-04-01T01:37:02.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.669+0000 7f860a218740 1 -- start start 2024-04-01T01:37:02.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.670+0000 7f860a218740 1 Processor -- start 2024-04-01T01:37:02.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.670+0000 7f860a218740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be8434a0 0x55a0be833d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.670+0000 7f860a218740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be843e90 0x55a0be834250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.671+0000 7f860a218740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be849000 0x55a0be83b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.671+0000 7f860a218740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a0be832370 con 0x55a0be849000 2024-04-01T01:37:02.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.671+0000 7f860a218740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a0be7354a0 con 0x55a0be843e90 2024-04-01T01:37:02.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.671+0000 7f860a218740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a0be8324f0 con 0x55a0be8434a0 2024-04-01T01:37:02.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.671+0000 7f860a218740 10 client.0 _ll_register_callbacks cb 0x55a0be738d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:02.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.671+0000 7f8605afb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be8434a0 0x55a0be833d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.671+0000 7f8605afb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be8434a0 0x55a0be833d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45640/0 (socket says 172.21.15.70:45640) 2024-04-01T01:37:02.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.671+0000 7f8605afb640 1 -- 172.21.15.70:0/2635563717 learned_addr learned my addr 172.21.15.70:0/2635563717 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:02.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.671+0000 7f8604af9640 1 --2- 172.21.15.70:0/2635563717 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be843e90 0x55a0be834250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.671+0000 7f8605afb640 1 --2- 172.21.15.70:0/2635563717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be849000 0x55a0be83b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.671+0000 7f8604af9640 1 -- 172.21.15.70:0/2635563717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be8434a0 msgr2=0x55a0be833d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:02.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.671+0000 7f8604af9640 1 --2- 172.21.15.70:0/2635563717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be8434a0 0x55a0be833d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.671+0000 7f8604af9640 1 -- 172.21.15.70:0/2635563717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be849000 msgr2=0x55a0be83b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:02.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.671+0000 7f8604af9640 1 --2- 172.21.15.70:0/2635563717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be849000 0x55a0be83b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.671+0000 7f8604af9640 1 -- 172.21.15.70:0/2635563717 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0be5757d0 con 0x55a0be843e90 2024-04-01T01:37:02.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.672+0000 7f8604af9640 1 --2- 172.21.15.70:0/2635563717 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be843e90 0x55a0be834250 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f85f0008000 tx=0x7f85f0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5234a47cb38c5bf server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:02.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.672+0000 7f85e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:02.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.672+0000 7f85e8ff9640 1 -- 172.21.15.70:0/2635563717 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f0045020 con 0x55a0be843e90 2024-04-01T01:37:02.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.672+0000 7f85e8ff9640 1 -- 172.21.15.70:0/2635563717 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f85f0005040 con 0x55a0be843e90 2024-04-01T01:37:02.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.672+0000 7f860a218740 1 -- 172.21.15.70:0/2635563717 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85f80054d0 con 0x55a0be843e90 2024-04-01T01:37:02.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.673+0000 7f85e8ff9640 1 -- 172.21.15.70:0/2635563717 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f85f0005240 con 0x55a0be843e90 2024-04-01T01:37:02.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.673+0000 7f860a218740 10 client.26459 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:02.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.673+0000 7f860a218740 1 -- 172.21.15.70:0/2635563717 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a0be4ab3f0 con 0x55a0be843e90 2024-04-01T01:37:02.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.673+0000 7f85e8ff9640 1 -- 172.21.15.70:0/2635563717 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85f00053e0 con 0x55a0be843e90 2024-04-01T01:37:02.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.673+0000 7f85e8ff9640 1 -- 172.21.15.70:0/2635563717 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f85f00035f0 con 0x55a0be843e90 2024-04-01T01:37:02.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.673+0000 7f860a218740 10 client.26459 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:02.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.673+0000 7f860a218740 10 client.26459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:02.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.673+0000 7f860a218740 20 client.26459 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:02.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.674+0000 7f860a218740 1 --2- 172.21.15.70:0/2635563717 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a0be83fb60 0x55a0be841f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.674+0000 7f860a218740 4 client.26459 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:02.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.674+0000 7f860a218740 1 -- 172.21.15.70:0/2635563717 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f85f80054d0 con 0x55a0be83fb60 2024-04-01T01:37:02.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.674+0000 7f8605afb640 1 --2- 172.21.15.70:0/2635563717 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a0be83fb60 0x55a0be841f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.675+0000 7f8605afb640 1 --2- 172.21.15.70:0/2635563717 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a0be83fb60 0x55a0be841f80 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abb74b306e7a167 server_cookie=97c95e01eb5299d1 in_seq=0 out_seq=0 2024-04-01T01:37:02.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.675+0000 7f85e8ff9640 10 client.26459 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:02.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.676+0000 7f85e8ff9640 1 -- 172.21.15.70:0/2635563717 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f85f80054d0 con 0x55a0be83fb60 2024-04-01T01:37:02.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.676+0000 7f85e8ff9640 10 client.26459 handle_command_reply: tid=0 2024-04-01T01:37:02.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.677+0000 7f860a218740 1 client.26459 shutdown 2024-04-01T01:37:02.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.677+0000 7f860a218740 1 -- 172.21.15.70:0/2635563717 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be843e90 msgr2=0x55a0be834250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:02.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.677+0000 7f860a218740 1 --2- 172.21.15.70:0/2635563717 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be843e90 0x55a0be834250 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f85f0008000 tx=0x7f85f0004270 comp rx=0 tx=0).stop 2024-04-01T01:37:02.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.677+0000 7f860a218740 1 -- 172.21.15.70:0/2635563717 shutdown_connections 2024-04-01T01:37:02.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.677+0000 7f860a218740 1 --2- 172.21.15.70:0/2635563717 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a0be83fb60 0x55a0be841f80 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.677+0000 7f860a218740 1 --2- 172.21.15.70:0/2635563717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be849000 0x55a0be83b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.677+0000 7f85e8ff9640 0 client.26459 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:02.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.677+0000 7f860a218740 1 --2- 172.21.15.70:0/2635563717 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be843e90 0x55a0be834250 secure :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f85f0008000 tx=0x7f85f0004270 comp rx=0 tx=0).stop 2024-04-01T01:37:02.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.677+0000 7f860a218740 1 --2- 172.21.15.70:0/2635563717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be8434a0 0x55a0be833d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.677+0000 7f860a218740 1 -- 172.21.15.70:0/2635563717 >> 172.21.15.70:0/2635563717 conn(0x55a0be82da00 msgr2=0x55a0be82cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:02.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.678+0000 7f860a218740 1 -- 172.21.15.70:0/2635563717 shutdown_connections 2024-04-01T01:37:02.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.678+0000 7f860a218740 1 -- 172.21.15.70:0/2635563717 wait complete. 2024-04-01T01:37:02.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.678+0000 7f860a218740 20 client.26459 trim_cache size 0 max 16384 2024-04-01T01:37:02.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.718+0000 7f860a218740 1 Processor -- start 2024-04-01T01:37:02.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.718+0000 7f860a218740 1 -- start start 2024-04-01T01:37:02.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.718+0000 7f860a218740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be83fb60 0x55a0be830c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.718+0000 7f860a218740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be8434a0 0x55a0be83b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.718+0000 7f860a218740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be849000 0x55a0be8311a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.719+0000 7f860a218740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f860019fe00 con 0x55a0be8434a0 2024-04-01T01:37:02.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.719+0000 7f860a218740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f86001a1940 con 0x55a0be849000 2024-04-01T01:37:02.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.719+0000 7f860a218740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f86001a04d0 con 0x55a0be83fb60 2024-04-01T01:37:02.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.719+0000 7f8605afb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be83fb60 0x55a0be830c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.719+0000 7f8605afb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be83fb60 0x55a0be830c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45646/0 (socket says 172.21.15.70:45646) 2024-04-01T01:37:02.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.719+0000 7f8604af9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be8434a0 0x55a0be83b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.719+0000 7f8605afb640 1 -- 172.21.15.70:0/3303537699 learned_addr learned my addr 172.21.15.70:0/3303537699 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:02.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.719+0000 7f8605afb640 1 --2- 172.21.15.70:0/3303537699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be849000 0x55a0be8311a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.719+0000 7f8605afb640 1 -- 172.21.15.70:0/3303537699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be849000 msgr2=0x55a0be8311a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:02.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.719+0000 7f8605afb640 1 --2- 172.21.15.70:0/3303537699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be849000 0x55a0be8311a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.719+0000 7f8605afb640 1 -- 172.21.15.70:0/3303537699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be8434a0 msgr2=0x55a0be83b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:02.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.719+0000 7f8605afb640 1 --2- 172.21.15.70:0/3303537699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be8434a0 0x55a0be83b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.719+0000 7f8605afb640 1 -- 172.21.15.70:0/3303537699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0be46b0f0 con 0x55a0be83fb60 2024-04-01T01:37:02.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.719+0000 7f8604af9640 1 --2- 172.21.15.70:0/3303537699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be8434a0 0x55a0be83b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:02.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.720+0000 7f8605afb640 1 --2- 172.21.15.70:0/3303537699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be83fb60 0x55a0be830c60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f85f8033010 tx=0x7f85f8032840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35f2c0355087aa9a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:02.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.721+0000 7f85eaffd640 1 -- 172.21.15.70:0/3303537699 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f804ebe0 con 0x55a0be83fb60 2024-04-01T01:37:02.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.721+0000 7f85eaffd640 1 -- 172.21.15.70:0/3303537699 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85f8048040 con 0x55a0be83fb60 2024-04-01T01:37:02.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.721+0000 7f85eaffd640 1 -- 172.21.15.70:0/3303537699 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f8033630 con 0x55a0be83fb60 2024-04-01T01:37:02.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.722+0000 7f860a218740 1 -- 172.21.15.70:0/3303537699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be83fb60 msgr2=0x55a0be830c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:02.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.722+0000 7f860a218740 1 --2- 172.21.15.70:0/3303537699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be83fb60 0x55a0be830c60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f85f8033010 tx=0x7f85f8032840 comp rx=0 tx=0).stop 2024-04-01T01:37:02.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.722+0000 7f860a218740 1 -- 172.21.15.70:0/3303537699 shutdown_connections 2024-04-01T01:37:02.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.722+0000 7f860a218740 1 --2- 172.21.15.70:0/3303537699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be849000 0x55a0be8311a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.722+0000 7f860a218740 1 --2- 172.21.15.70:0/3303537699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be8434a0 0x55a0be83b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.722+0000 7f860a218740 1 --2- 172.21.15.70:0/3303537699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be83fb60 0x55a0be830c60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.722+0000 7f860a218740 1 -- 172.21.15.70:0/3303537699 >> 172.21.15.70:0/3303537699 conn(0x55a0be82da00 msgr2=0x55a0be8427e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:02.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.723+0000 7f860a218740 1 -- 172.21.15.70:0/3303537699 shutdown_connections 2024-04-01T01:37:02.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.723+0000 7f860a218740 1 -- 172.21.15.70:0/3303537699 wait complete. 2024-04-01T01:37:02.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.723+0000 7f860a218740 1 -- start start 2024-04-01T01:37:02.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.723+0000 7f860a218740 1 Processor -- start 2024-04-01T01:37:02.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.724+0000 7f860a218740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be83fb60 0x55a0be839270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.724+0000 7f860a218740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be8434a0 0x55a0be8397b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.725+0000 7f860a218740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be849000 0x55a0be83a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.725+0000 7f860a218740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f86001a1270 con 0x55a0be849000 2024-04-01T01:37:02.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.725+0000 7f860a218740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f86001a0ba0 con 0x55a0be83fb60 2024-04-01T01:37:02.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.725+0000 7f860a218740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8600073530 con 0x55a0be8434a0 2024-04-01T01:37:02.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.725+0000 7f860a218740 10 client.0 _ll_register_callbacks cb 0x7f85f0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:02.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.725+0000 7f8604af9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be8434a0 0x55a0be8397b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.725+0000 7f8604af9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be8434a0 0x55a0be8397b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45652/0 (socket says 172.21.15.70:45652) 2024-04-01T01:37:02.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.725+0000 7f8604af9640 1 -- 172.21.15.70:0/2891905728 learned_addr learned my addr 172.21.15.70:0/2891905728 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:02.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.725+0000 7f8605afb640 1 --2- 172.21.15.70:0/2891905728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be83fb60 0x55a0be839270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.726+0000 7f8605afb640 1 -- 172.21.15.70:0/2891905728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be8434a0 msgr2=0x55a0be8397b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:02.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.726+0000 7f8605afb640 1 --2- 172.21.15.70:0/2891905728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be8434a0 0x55a0be8397b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.726+0000 7f8605afb640 1 -- 172.21.15.70:0/2891905728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be849000 msgr2=0x55a0be83a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:02.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.726+0000 7f8605afb640 1 --2- 172.21.15.70:0/2891905728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be849000 0x55a0be83a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.726+0000 7f8605afb640 1 -- 172.21.15.70:0/2891905728 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0be83adb0 con 0x55a0be83fb60 2024-04-01T01:37:02.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.726+0000 7f8605afb640 1 --2- 172.21.15.70:0/2891905728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be83fb60 0x55a0be839270 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x55a0be83e590 tx=0x7f85f8005ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=606b9133ea66fe88 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:02.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.726+0000 7f85e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:02.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.730+0000 7f85e8ff9640 1 -- 172.21.15.70:0/2891905728 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f8009c20 con 0x55a0be83fb60 2024-04-01T01:37:02.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.730+0000 7f85e8ff9640 1 -- 172.21.15.70:0/2891905728 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f85f804ebe0 con 0x55a0be83fb60 2024-04-01T01:37:02.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.730+0000 7f860a218740 1 -- 172.21.15.70:0/2891905728 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85f8048040 con 0x55a0be83fb60 2024-04-01T01:37:02.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.731+0000 7f85e8ff9640 1 -- 172.21.15.70:0/2891905728 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f85f804e7f0 con 0x55a0be83fb60 2024-04-01T01:37:02.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.731+0000 7f860a218740 10 client.26462 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:02.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.731+0000 7f860a218740 1 -- 172.21.15.70:0/2891905728 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a0be8afcc0 con 0x55a0be83fb60 2024-04-01T01:37:02.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.731+0000 7f85e8ff9640 1 -- 172.21.15.70:0/2891905728 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85f8048040 con 0x55a0be83fb60 2024-04-01T01:37:02.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.731+0000 7f85e8ff9640 1 -- 172.21.15.70:0/2891905728 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f85f800f040 con 0x55a0be83fb60 2024-04-01T01:37:02.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.731+0000 7f860a218740 10 client.26462 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:02.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.731+0000 7f860a218740 10 client.26462 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:02.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.731+0000 7f860a218740 20 client.26462 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:02.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.731+0000 7f860a218740 1 --2- 172.21.15.70:0/2891905728 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a0be8b0430 0x55a0be8b2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.731+0000 7f860a218740 4 client.26462 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:37:02.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.731+0000 7f860a218740 1 -- 172.21.15.70:0/2891905728 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a0be7a46c0 con 0x55a0be8b0430 2024-04-01T01:37:02.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.732+0000 7f8604af9640 1 --2- 172.21.15.70:0/2891905728 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a0be8b0430 0x55a0be8b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.732+0000 7f85e8ff9640 10 client.26462 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:02.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.732+0000 7f8604af9640 1 --2- 172.21.15.70:0/2891905728 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a0be8b0430 0x55a0be8b2850 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16629a336d694600 server_cookie=2d7bba0975f75533 in_seq=0 out_seq=0 2024-04-01T01:37:02.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.733+0000 7f85e8ff9640 1 -- 172.21.15.70:0/2891905728 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a0be7a46c0 con 0x55a0be8b0430 2024-04-01T01:37:02.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.733+0000 7f85e8ff9640 10 client.26462 handle_command_reply: tid=0 2024-04-01T01:37:02.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.733+0000 7f860a218740 1 client.26462 shutdown 2024-04-01T01:37:02.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.733+0000 7f860a218740 1 -- 172.21.15.70:0/2891905728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be83fb60 msgr2=0x55a0be839270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:02.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.733+0000 7f860a218740 1 --2- 172.21.15.70:0/2891905728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be83fb60 0x55a0be839270 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x55a0be83e590 tx=0x7f85f8005ab0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.733+0000 7f860a218740 1 -- 172.21.15.70:0/2891905728 shutdown_connections 2024-04-01T01:37:02.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.733+0000 7f860a218740 1 --2- 172.21.15.70:0/2891905728 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a0be8b0430 0x55a0be8b2850 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.733+0000 7f860a218740 1 --2- 172.21.15.70:0/2891905728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0be849000 0x55a0be83a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.733+0000 7f860a218740 1 --2- 172.21.15.70:0/2891905728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0be8434a0 0x55a0be8397b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.733+0000 7f860a218740 1 --2- 172.21.15.70:0/2891905728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0be83fb60 0x55a0be839270 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.733+0000 7f860a218740 1 -- 172.21.15.70:0/2891905728 >> 172.21.15.70:0/2891905728 conn(0x55a0be82da00 msgr2=0x55a0be830c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:02.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.733+0000 7f85e8ff9640 0 client.26462 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:02.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.734+0000 7f860a218740 1 -- 172.21.15.70:0/2891905728 shutdown_connections 2024-04-01T01:37:02.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.734+0000 7f860a218740 1 -- 172.21.15.70:0/2891905728 wait complete. 2024-04-01T01:37:02.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.734+0000 7f860a218740 20 client.26462 trim_cache size 0 max 16384 2024-04-01T01:37:02.735 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:37:02.735 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:37:02.735 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:37:02.735 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:37:02.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.735+0000 7f860755e640 1 -- 172.21.15.70:0/586259608 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85ec0a2670 msgr2=0x7f85ec0a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:02.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.736+0000 7f860755e640 1 --2- 172.21.15.70:0/586259608 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85ec0a2670 0x7f85ec0a4b30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f85fc0a2b40 tx=0x7f85fc0dc530 comp rx=0 tx=0).stop 2024-04-01T01:37:02.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.736+0000 7f860755e640 1 -- 172.21.15.70:0/586259608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8600103890 msgr2=0x7f86001958a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:02.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.736+0000 7f860755e640 1 --2- 172.21.15.70:0/586259608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8600103890 0x7f86001958a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009fd0 tx=0x7f85fc007650 comp rx=0 tx=0).stop 2024-04-01T01:37:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.738+0000 7f860755e640 1 -- 172.21.15.70:0/586259608 shutdown_connections 2024-04-01T01:37:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.738+0000 7f860755e640 1 --2- 172.21.15.70:0/586259608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f860019ae40 0x7f860019d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.738+0000 7f860755e640 1 --2- 172.21.15.70:0/586259608 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85ec0a2670 0x7f85ec0a4b30 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.738+0000 7f860755e640 1 --2- 172.21.15.70:0/586259608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8600105440 0x7f8600195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.738+0000 7f860755e640 1 --2- 172.21.15.70:0/586259608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8600103890 0x7f86001958a0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.738+0000 7f860755e640 1 -- 172.21.15.70:0/586259608 >> 172.21.15.70:0/586259608 conn(0x7f86000fee00 msgr2=0x7f86000ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.739+0000 7f860755e640 1 -- 172.21.15.70:0/586259608 shutdown_connections 2024-04-01T01:37:02.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.740+0000 7f860755e640 1 -- 172.21.15.70:0/586259608 wait complete. 2024-04-01T01:37:02.762 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:37:02.803 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:37:02.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.980+0000 7f41c359e640 1 -- 172.21.15.70:0/1801080579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41c40ffc50 msgr2=0x7f41c4100070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:02.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.980+0000 7f41c359e640 1 --2- 172.21.15.70:0/1801080579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41c40ffc50 0x7f41c4100070 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f41b400a910 tx=0x7f41b402be40 comp rx=0 tx=0).stop 2024-04-01T01:37:02.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.980+0000 7f41c359e640 1 -- 172.21.15.70:0/1801080579 shutdown_connections 2024-04-01T01:37:02.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.980+0000 7f41c359e640 1 --2- 172.21.15.70:0/1801080579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c4101800 0x7f41c4108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.980+0000 7f41c359e640 1 --2- 172.21.15.70:0/1801080579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c4100e40 0x7f41c41012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.980+0000 7f41c359e640 1 --2- 172.21.15.70:0/1801080579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41c40ffc50 0x7f41c4100070 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.980+0000 7f41c359e640 1 -- 172.21.15.70:0/1801080579 >> 172.21.15.70:0/1801080579 conn(0x7f41c40fb1c0 msgr2=0x7f41c40fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:02.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.982+0000 7f41c359e640 1 -- 172.21.15.70:0/1801080579 shutdown_connections 2024-04-01T01:37:02.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.982+0000 7f41c359e640 1 -- 172.21.15.70:0/1801080579 wait complete. 2024-04-01T01:37:02.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.983+0000 7f41c359e640 1 Processor -- start 2024-04-01T01:37:02.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.983+0000 7f41c359e640 1 -- start start 2024-04-01T01:37:02.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.983+0000 7f41c359e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c40ffc50 0x7f41c4199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.983+0000 7f41c359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41c4100e40 0x7f41c4199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.983+0000 7f41c359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c4101800 0x7f41c41a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.983+0000 7f41c359e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f41c410f370 con 0x7f41c4100e40 2024-04-01T01:37:02.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.983+0000 7f41c359e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f41c410f1f0 con 0x7f41c4101800 2024-04-01T01:37:02.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.983+0000 7f41c359e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f41c410f4f0 con 0x7f41c40ffc50 2024-04-01T01:37:02.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.984+0000 7f41c259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c40ffc50 0x7f41c4199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.984+0000 7f41c2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c4101800 0x7f41c41a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.984+0000 7f41c2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c4101800 0x7f41c41a0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:50354/0 (socket says 172.21.15.70:50354) 2024-04-01T01:37:02.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.984+0000 7f41c2d9d640 1 -- 172.21.15.70:0/1505750126 learned_addr learned my addr 172.21.15.70:0/1505750126 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:02.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.984+0000 7f41c2d9d640 1 -- 172.21.15.70:0/1505750126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c40ffc50 msgr2=0x7f41c4199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:02.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.984+0000 7f41c2d9d640 1 --2- 172.21.15.70:0/1505750126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c40ffc50 0x7f41c4199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.984+0000 7f41c2d9d640 1 -- 172.21.15.70:0/1505750126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41c4100e40 msgr2=0x7f41c4199d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:37:02.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.984+0000 7f41c2d9d640 1 --2- 172.21.15.70:0/1505750126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41c4100e40 0x7f41c4199d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:02.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.984+0000 7f41c2d9d640 1 -- 172.21.15.70:0/1505750126 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c41a1420 con 0x7f41c4101800 2024-04-01T01:37:02.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.985+0000 7f41c2d9d640 1 --2- 172.21.15.70:0/1505750126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c4101800 0x7f41c41a0db0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f41bc00eb10 tx=0x7f41bc00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:02.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.985+0000 7f41b37fe640 1 -- 172.21.15.70:0/1505750126 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc004b20 con 0x7f41c4101800 2024-04-01T01:37:02.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.985+0000 7f41c359e640 1 -- 172.21.15.70:0/1505750126 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c41a1710 con 0x7f41c4101800 2024-04-01T01:37:02.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.985+0000 7f41c359e640 1 -- 172.21.15.70:0/1505750126 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c41a1c60 con 0x7f41c4101800 2024-04-01T01:37:02.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.985+0000 7f41b37fe640 1 -- 172.21.15.70:0/1505750126 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41bc004cc0 con 0x7f41c4101800 2024-04-01T01:37:02.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.985+0000 7f41b37fe640 1 -- 172.21.15.70:0/1505750126 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc010690 con 0x7f41c4101800 2024-04-01T01:37:02.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.989+0000 7f41b37fe640 1 -- 172.21.15.70:0/1505750126 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f41bc005020 con 0x7f41c4101800 2024-04-01T01:37:02.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.990+0000 7f41c359e640 1 -- 172.21.15.70:0/1505750126 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41a40052c0 con 0x7f41c4101800 2024-04-01T01:37:02.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.991+0000 7f41b37fe640 1 --2- 172.21.15.70:0/1505750126 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41ac0a25a0 0x7f41ac0a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:02.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.992+0000 7f41c259c640 1 --2- 172.21.15.70:0/1505750126 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41ac0a25a0 0x7f41ac0a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:02.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.992+0000 7f41b37fe640 1 -- 172.21.15.70:0/1505750126 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f41bc0cf0e0 con 0x7f41c4101800 2024-04-01T01:37:02.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.992+0000 7f41c259c640 1 --2- 172.21.15.70:0/1505750126 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41ac0a25a0 0x7f41ac0a4a60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f41b402c380 tx=0x7f41b4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:02.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:02.999+0000 7f41b37fe640 1 -- 172.21.15.70:0/1505750126 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f41bc09be70 con 0x7f41c4101800 2024-04-01T01:37:03.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:02 smithi081 ceph-mon[35142]: pgmap v465: 129 pgs: 129 active+clean; 1011 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 1.32k op/s 2024-04-01T01:37:03.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:02 smithi189 ceph-mon[35753]: pgmap v465: 129 pgs: 129 active+clean; 1011 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 1.32k op/s 2024-04-01T01:37:03.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:02 smithi070 ceph-mon[32179]: pgmap v465: 129 pgs: 129 active+clean; 1011 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 11 MiB/s wr, 1.32k op/s 2024-04-01T01:37:03.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.430+0000 7f41c359e640 1 -- 172.21.15.70:0/1505750126 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41a40055b0 con 0x7f41c4101800 2024-04-01T01:37:03.432 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:37:03.432 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:33:36.894838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24418,24404,24379,24451,24464]},"id":1}]} 2024-04-01T01:37:03.432 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-04-01T01:37:03.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.432+0000 7f41b37fe640 1 -- 172.21.15.70:0/1505750126 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f41bc09b770 con 0x7f41c4101800 2024-04-01T01:37:03.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.434+0000 7f41c359e640 1 -- 172.21.15.70:0/1505750126 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41ac0a25a0 msgr2=0x7f41ac0a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.434+0000 7f41c359e640 1 --2- 172.21.15.70:0/1505750126 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41ac0a25a0 0x7f41ac0a4a60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f41b402c380 tx=0x7f41b4000bc0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.434+0000 7f41c359e640 1 -- 172.21.15.70:0/1505750126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c4101800 msgr2=0x7f41c41a0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.434+0000 7f41c359e640 1 --2- 172.21.15.70:0/1505750126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c4101800 0x7f41c41a0db0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f41bc00eb10 tx=0x7f41bc00c2d0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.435+0000 7f41c359e640 1 -- 172.21.15.70:0/1505750126 shutdown_connections 2024-04-01T01:37:03.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.435+0000 7f41c359e640 1 --2- 172.21.15.70:0/1505750126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41c4101800 0x7f41c41a0db0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.435+0000 7f41c359e640 1 --2- 172.21.15.70:0/1505750126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41c4100e40 0x7f41c4199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.435+0000 7f41c359e640 1 --2- 172.21.15.70:0/1505750126 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41ac0a25a0 0x7f41ac0a4a60 secure :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f41b402c380 tx=0x7f41b4000bc0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.435+0000 7f41c359e640 1 --2- 172.21.15.70:0/1505750126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41c40ffc50 0x7f41c4199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.435+0000 7f41c359e640 1 -- 172.21.15.70:0/1505750126 >> 172.21.15.70:0/1505750126 conn(0x7f41c40fb1c0 msgr2=0x7f41c4104070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:03.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.436+0000 7f41c359e640 1 -- 172.21.15.70:0/1505750126 shutdown_connections 2024-04-01T01:37:03.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.436+0000 7f41c359e640 1 -- 172.21.15.70:0/1505750126 wait complete. 2024-04-01T01:37:03.453 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:37:03.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.631+0000 7f2bf39fd640 1 -- 172.21.15.70:0/1134817574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bec116560 msgr2=0x7f2bec118950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.631+0000 7f2bf39fd640 1 --2- 172.21.15.70:0/1134817574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bec116560 0x7f2bec118950 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc009900 tx=0x7f2bdc02bc30 comp rx=0 tx=0).stop 2024-04-01T01:37:03.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.632+0000 7f2bf39fd640 1 -- 172.21.15.70:0/1134817574 shutdown_connections 2024-04-01T01:37:03.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.632+0000 7f2bf39fd640 1 --2- 172.21.15.70:0/1134817574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bec116560 0x7f2bec118950 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.632+0000 7f2bf39fd640 1 --2- 172.21.15.70:0/1134817574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bec06b660 0x7f2bec06bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.632+0000 7f2bf39fd640 1 --2- 172.21.15.70:0/1134817574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bec06cc40 0x7f2bec06b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.632+0000 7f2bf39fd640 1 -- 172.21.15.70:0/1134817574 >> 172.21.15.70:0/1134817574 conn(0x7f2bec0706f0 msgr2=0x7f2bec072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:03.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.632+0000 7f2bf39fd640 1 -- 172.21.15.70:0/1134817574 shutdown_connections 2024-04-01T01:37:03.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.632+0000 7f2bf39fd640 1 -- 172.21.15.70:0/1134817574 wait complete. 2024-04-01T01:37:03.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.633+0000 7f2bf39fd640 1 Processor -- start 2024-04-01T01:37:03.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.633+0000 7f2bf39fd640 1 -- start start 2024-04-01T01:37:03.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.633+0000 7f2bf39fd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bec06b660 0x7f2bec1a6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.633+0000 7f2bf39fd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bec06cc40 0x7f2bec1a6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.633+0000 7f2bf39fd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bec116560 0x7f2bec1adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.633+0000 7f2bf39fd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec11c070 con 0x7f2bec06b660 2024-04-01T01:37:03.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.633+0000 7f2bf39fd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec11bef0 con 0x7f2bec06cc40 2024-04-01T01:37:03.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.633+0000 7f2bf39fd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec11c1f0 con 0x7f2bec116560 2024-04-01T01:37:03.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.633+0000 7f2bf1f9a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bec116560 0x7f2bec1adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.634+0000 7f2bf1f9a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bec116560 0x7f2bec1adcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45708/0 (socket says 172.21.15.70:45708) 2024-04-01T01:37:03.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.634+0000 7f2bf1f9a640 1 -- 172.21.15.70:0/3939361321 learned_addr learned my addr 172.21.15.70:0/3939361321 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:03.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.634+0000 7f2bf0f98640 1 --2- 172.21.15.70:0/3939361321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bec06cc40 0x7f2bec1a6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.634+0000 7f2bf1799640 1 --2- 172.21.15.70:0/3939361321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bec06b660 0x7f2bec1a6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.634+0000 7f2bf1f9a640 1 -- 172.21.15.70:0/3939361321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bec06cc40 msgr2=0x7f2bec1a6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.634+0000 7f2bf1f9a640 1 --2- 172.21.15.70:0/3939361321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bec06cc40 0x7f2bec1a6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.634+0000 7f2bf1f9a640 1 -- 172.21.15.70:0/3939361321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bec06b660 msgr2=0x7f2bec1a6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.634+0000 7f2bf1f9a640 1 --2- 172.21.15.70:0/3939361321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bec06b660 0x7f2bec1a6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.634+0000 7f2bf1f9a640 1 -- 172.21.15.70:0/3939361321 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bec10de80 con 0x7f2bec116560 2024-04-01T01:37:03.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.635+0000 7f2bf1f9a640 1 --2- 172.21.15.70:0/3939361321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bec116560 0x7f2bec1adcc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc036040 tx=0x7f2bdc004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:03.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.638+0000 7f2be27fc640 1 -- 172.21.15.70:0/3939361321 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc043020 con 0x7f2bec116560 2024-04-01T01:37:03.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.638+0000 7f2bf39fd640 1 -- 172.21.15.70:0/3939361321 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bec10e110 con 0x7f2bec116560 2024-04-01T01:37:03.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.638+0000 7f2bf39fd640 1 -- 172.21.15.70:0/3939361321 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bec10e660 con 0x7f2bec116560 2024-04-01T01:37:03.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.638+0000 7f2be27fc640 1 -- 172.21.15.70:0/3939361321 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2bdc0052b0 con 0x7f2bec116560 2024-04-01T01:37:03.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.638+0000 7f2be27fc640 1 -- 172.21.15.70:0/3939361321 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc003bc0 con 0x7f2bec116560 2024-04-01T01:37:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.641+0000 7fd63f042640 1 -- 172.21.15.70:0/1425827841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd638077cf0 msgr2=0x7fd638078590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.641+0000 7fd63f042640 1 --2- 172.21.15.70:0/1425827841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd638077cf0 0x7fd638078590 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd630009be0 tx=0x7fd63002bef0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.641+0000 7fd63f042640 1 -- 172.21.15.70:0/1425827841 shutdown_connections 2024-04-01T01:37:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.641+0000 7fd63f042640 1 --2- 172.21.15.70:0/1425827841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd638077cf0 0x7fd638078590 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.641+0000 7fd63f042640 1 --2- 172.21.15.70:0/1425827841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd638077350 0x7fd6380777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.641+0000 7fd63f042640 1 --2- 172.21.15.70:0/1425827841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd638076110 0x7fd638076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.641+0000 7fd63f042640 1 -- 172.21.15.70:0/1425827841 >> 172.21.15.70:0/1425827841 conn(0x7fd6380fed60 msgr2=0x7fd638101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:03.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.642+0000 7f2bf65d9740 1 Processor -- start 2024-04-01T01:37:03.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.644+0000 7f2bf65d9740 1 -- start start 2024-04-01T01:37:03.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.644+0000 7fd63f042640 1 -- 172.21.15.70:0/1425827841 shutdown_connections 2024-04-01T01:37:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.644+0000 7f2bf65d9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dd490 0x5556b66dd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.644+0000 7f2bf65d9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 0x5556b66e2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.644+0000 7f2bf65d9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66e2ff0 0x5556b66e5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.644+0000 7f2bf65d9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5556b66421b0 con 0x5556b66dde80 2024-04-01T01:37:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.644+0000 7f2bf65d9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5556b65eac00 con 0x5556b66e2ff0 2024-04-01T01:37:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.644+0000 7f2bf65d9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5556b663e250 con 0x5556b66dd490 2024-04-01T01:37:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.644+0000 7fd63f042640 1 -- 172.21.15.70:0/1425827841 wait complete. 2024-04-01T01:37:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.644+0000 7f2bf0f98640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dd490 0x5556b66dd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.644+0000 7f2bf0f98640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dd490 0x5556b66dd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45724/0 (socket says 172.21.15.70:45724) 2024-04-01T01:37:03.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.644+0000 7f2bf0f98640 1 -- 172.21.15.70:0/2767516152 learned_addr learned my addr 172.21.15.70:0/2767516152 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:03.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.644+0000 7f2bf0f98640 1 --2- 172.21.15.70:0/2767516152 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66e2ff0 0x5556b66e5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.644+0000 7f2bf1799640 1 --2- 172.21.15.70:0/2767516152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 0x5556b66e2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.644+0000 7f2be27fc640 1 -- 172.21.15.70:0/3939361321 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f2bdc046020 con 0x7f2bec116560 2024-04-01T01:37:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.645+0000 7fd63f042640 1 Processor -- start 2024-04-01T01:37:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.645+0000 7fd63f042640 1 -- start start 2024-04-01T01:37:03.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.645+0000 7f2bf1799640 1 -- 172.21.15.70:0/2767516152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dd490 msgr2=0x5556b66dd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.645+0000 7f2bf1799640 1 --2- 172.21.15.70:0/2767516152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dd490 0x5556b66dd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.645+0000 7f2bf1799640 1 -- 172.21.15.70:0/2767516152 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66e2ff0 msgr2=0x5556b66e5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.645+0000 7f2bf1799640 1 --2- 172.21.15.70:0/2767516152 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66e2ff0 0x5556b66e5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.645+0000 7f2bf1799640 1 -- 172.21.15.70:0/2767516152 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556b660a3d0 con 0x5556b66dde80 2024-04-01T01:37:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.645+0000 7fd63f042640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd638076110 0x7fd638070090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.646+0000 7fd63f042640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd638077350 0x7fd6380705d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.645+0000 7f2be27fc640 1 --2- 172.21.15.70:0/3939361321 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2bd80a26a0 0x7f2bd80a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.646+0000 7f2be27fc640 1 -- 172.21.15.70:0/3939361321 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f2bdc0f0d00 con 0x7f2bec116560 2024-04-01T01:37:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.646+0000 7fd63f042640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd638077cf0 0x7fd6381adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.646+0000 7fd63f042640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd638113710 con 0x7fd638077350 2024-04-01T01:37:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.646+0000 7fd63f042640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd638113590 con 0x7fd638076110 2024-04-01T01:37:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.646+0000 7fd63f042640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd638113890 con 0x7fd638077cf0 2024-04-01T01:37:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.646+0000 7fd63e841640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd638077cf0 0x7fd6381adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.646+0000 7fd63e841640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd638077cf0 0x7fd6381adde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45726/0 (socket says 172.21.15.70:45726) 2024-04-01T01:37:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.646+0000 7fd63e841640 1 -- 172.21.15.70:0/3612589256 learned_addr learned my addr 172.21.15.70:0/3612589256 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.646+0000 7fd63d83f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd638077350 0x7fd6380705d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.646+0000 7fd63e040640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd638076110 0x7fd638070090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.646+0000 7f2bf1799640 1 --2- 172.21.15.70:0/2767516152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 0x5556b66e2980 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2be4007ed0 tx=0x7f2be400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=751f8ffb81cf1195 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:03.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.646+0000 7f2bf0f98640 1 --2- 172.21.15.70:0/3939361321 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2bd80a26a0 0x7f2bd80a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.646+0000 7fd63d83f640 1 -- 172.21.15.70:0/3612589256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd638077cf0 msgr2=0x7fd6381adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.646+0000 7fd63d83f640 1 --2- 172.21.15.70:0/3612589256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd638077cf0 0x7fd6381adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.646+0000 7fd63d83f640 1 -- 172.21.15.70:0/3612589256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd638076110 msgr2=0x7fd638070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.646+0000 7fd63d83f640 1 --2- 172.21.15.70:0/3612589256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd638076110 0x7fd638070090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.646+0000 7fd63d83f640 1 -- 172.21.15.70:0/3612589256 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd638070c70 con 0x7fd638077350 2024-04-01T01:37:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.646+0000 7fd63e841640 1 --2- 172.21.15.70:0/3612589256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd638077cf0 0x7fd6381adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.647+0000 7fd63d83f640 1 --2- 172.21.15.70:0/3612589256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd638077350 0x7fd6380705d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd634005fd0 tx=0x7fd63400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.647+0000 7f2bf0f98640 1 --2- 172.21.15.70:0/3939361321 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2bd80a26a0 0x7f2bd80a4b60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x5556b66de6e0 tx=0x7f2be800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.647+0000 7f2bd77fe640 1 -- 172.21.15.70:0/2767516152 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be4004d20 con 0x5556b66dde80 2024-04-01T01:37:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.647+0000 7f2bd77fe640 1 -- 172.21.15.70:0/2767516152 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2be4010af0 con 0x5556b66dde80 2024-04-01T01:37:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.647+0000 7f2bd77fe640 1 -- 172.21.15.70:0/2767516152 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be40056a0 con 0x5556b66dde80 2024-04-01T01:37:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.649+0000 7fd62f7fe640 1 -- 172.21.15.70:0/3612589256 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63400cc80 con 0x7fd638077350 2024-04-01T01:37:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.649+0000 7fd63f042640 1 -- 172.21.15.70:0/3612589256 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6381ae440 con 0x7fd638077350 2024-04-01T01:37:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.649+0000 7fd62f7fe640 1 -- 172.21.15.70:0/3612589256 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd63400ce20 con 0x7fd638077350 2024-04-01T01:37:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.649+0000 7fd62f7fe640 1 -- 172.21.15.70:0/3612589256 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd634010430 con 0x7fd638077350 2024-04-01T01:37:03.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.650+0000 7f2bf65d9740 1 -- 172.21.15.70:0/2767516152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 msgr2=0x5556b66e2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.650+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/2767516152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 0x5556b66e2980 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2be4007ed0 tx=0x7f2be400f790 comp rx=0 tx=0).stop 2024-04-01T01:37:03.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.650+0000 7f2bf65d9740 1 -- 172.21.15.70:0/2767516152 shutdown_connections 2024-04-01T01:37:03.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.650+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/2767516152 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66e2ff0 0x5556b66e5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.650+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/2767516152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 0x5556b66e2980 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.650+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/2767516152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dd490 0x5556b66dd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.650+0000 7f2bf65d9740 1 -- 172.21.15.70:0/2767516152 >> 172.21.15.70:0/2767516152 conn(0x5556b66d8a00 msgr2=0x5556b66d8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.649+0000 7fd63f042640 1 -- 172.21.15.70:0/3612589256 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6381ae960 con 0x7fd638077350 2024-04-01T01:37:03.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.652+0000 7f2bf65d9740 1 -- 172.21.15.70:0/2767516152 shutdown_connections 2024-04-01T01:37:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.652+0000 7fd642f63740 1 Processor -- start 2024-04-01T01:37:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.653+0000 7fd642f63740 1 -- start start 2024-04-01T01:37:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.653+0000 7fd642f63740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbaae0 0x5580ffcbaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.653+0000 7fd642f63740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbb4d0 0x5580ffcbffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.653+0000 7fd642f63740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcc0640 0x5580ffcc2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.653+0000 7fd642f63740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5580ffc1e8f0 con 0x5580ffcc0640 2024-04-01T01:37:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.653+0000 7fd642f63740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5580ffbc78d0 con 0x5580ffcbb4d0 2024-04-01T01:37:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.653+0000 7fd642f63740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5580ffc1a990 con 0x5580ffcbaae0 2024-04-01T01:37:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.653+0000 7fd63e841640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbaae0 0x5580ffcbaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.653+0000 7fd63e841640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbaae0 0x5580ffcbaf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45736/0 (socket says 172.21.15.70:45736) 2024-04-01T01:37:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.653+0000 7fd63e841640 1 -- 172.21.15.70:0/1245972774 learned_addr learned my addr 172.21.15.70:0/1245972774 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.654+0000 7fd63e841640 1 -- 172.21.15.70:0/1245972774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbb4d0 msgr2=0x5580ffcbffd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:37:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.654+0000 7fd63e841640 1 --2- 172.21.15.70:0/1245972774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbb4d0 0x5580ffcbffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.654+0000 7fd63e841640 1 -- 172.21.15.70:0/1245972774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcc0640 msgr2=0x5580ffcc2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.654+0000 7fd63e841640 1 --2- 172.21.15.70:0/1245972774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcc0640 0x5580ffcc2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.654+0000 7fd63e841640 1 -- 172.21.15.70:0/1245972774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580ff933210 con 0x5580ffcbaae0 2024-04-01T01:37:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.654+0000 7fd62f7fe640 1 -- 172.21.15.70:0/3612589256 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fd6340105d0 con 0x7fd638077350 2024-04-01T01:37:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.654+0000 7fd63e841640 1 --2- 172.21.15.70:0/1245972774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbaae0 0x5580ffcbaf00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd630033ac0 tx=0x7fd630033af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bc39d291ff90f39 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.654+0000 7fd62cbfe640 1 -- 172.21.15.70:0/1245972774 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd630043030 con 0x5580ffcbaae0 2024-04-01T01:37:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.655+0000 7fd62cbfe640 1 -- 172.21.15.70:0/1245972774 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd630003e20 con 0x5580ffcbaae0 2024-04-01T01:37:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.655+0000 7fd62cbfe640 1 -- 172.21.15.70:0/1245972774 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63002cda0 con 0x5580ffcbaae0 2024-04-01T01:37:03.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.655+0000 7f2bf65d9740 1 -- 172.21.15.70:0/2767516152 wait complete. 2024-04-01T01:37:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.655+0000 7fd62f7fe640 1 --2- 172.21.15.70:0/3612589256 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd6240a26a0 0x7fd6240a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.655+0000 7f2bf65d9740 1 -- start start 2024-04-01T01:37:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.655+0000 7fd62f7fe640 1 -- 172.21.15.70:0/3612589256 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd6340cec10 con 0x7fd638077350 2024-04-01T01:37:03.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.655+0000 7f2bf65d9740 1 Processor -- start 2024-04-01T01:37:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.656+0000 7fd642f63740 1 -- 172.21.15.70:0/1245972774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbaae0 msgr2=0x5580ffcbaf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.656+0000 7fd642f63740 1 --2- 172.21.15.70:0/1245972774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbaae0 0x5580ffcbaf00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd630033ac0 tx=0x7fd630033af0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.656+0000 7fd642f63740 1 -- 172.21.15.70:0/1245972774 shutdown_connections 2024-04-01T01:37:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.656+0000 7fd642f63740 1 --2- 172.21.15.70:0/1245972774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcc0640 0x5580ffcc2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.656+0000 7fd642f63740 1 --2- 172.21.15.70:0/1245972774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbb4d0 0x5580ffcbffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.656+0000 7fd642f63740 1 --2- 172.21.15.70:0/1245972774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbaae0 0x5580ffcbaf00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.656+0000 7fd642f63740 1 -- 172.21.15.70:0/1245972774 >> 172.21.15.70:0/1245972774 conn(0x5580ffcb6050 msgr2=0x5580ffcb6450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.656+0000 7f2bf65d9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dd490 0x5556b66e7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.656+0000 7f2bf65d9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dde80 0x5556b66e7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.656+0000 7f2bf65d9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66e2ff0 0x5556b66eedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.656+0000 7f2bf65d9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5556b66e5fa0 con 0x5556b66dd490 2024-04-01T01:37:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.656+0000 7f2bf65d9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5556b65e0470 con 0x5556b66e2ff0 2024-04-01T01:37:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.656+0000 7f2bf65d9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5556b66e6120 con 0x5556b66dde80 2024-04-01T01:37:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.657+0000 7f2bf0f98640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dde80 0x5556b66e7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.657+0000 7f2bf1f9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66e2ff0 0x5556b66eedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.657+0000 7f2bf0f98640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dde80 0x5556b66e7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45738/0 (socket says 172.21.15.70:45738) 2024-04-01T01:37:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.657+0000 7f2bf0f98640 1 -- 172.21.15.70:0/382867600 learned_addr learned my addr 172.21.15.70:0/382867600 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.657+0000 7f2bf65d9740 10 client.0 _ll_register_callbacks cb 0x5556b65fdcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.657+0000 7f2bf1799640 1 --2- 172.21.15.70:0/382867600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dd490 0x5556b66e7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.657+0000 7f2bf0f98640 1 -- 172.21.15.70:0/382867600 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66e2ff0 msgr2=0x5556b66eedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.657+0000 7f2bf0f98640 1 --2- 172.21.15.70:0/382867600 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66e2ff0 0x5556b66eedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.657+0000 7f2bf0f98640 1 -- 172.21.15.70:0/382867600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dd490 msgr2=0x5556b66e7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.657+0000 7f2bf0f98640 1 --2- 172.21.15.70:0/382867600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dd490 0x5556b66e7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.657+0000 7f2bf0f98640 1 -- 172.21.15.70:0/382867600 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556b63563f0 con 0x5556b66dde80 2024-04-01T01:37:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.657+0000 7fd642f63740 1 -- 172.21.15.70:0/1245972774 shutdown_connections 2024-04-01T01:37:03.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.657+0000 7f2bf1f9a640 1 --2- 172.21.15.70:0/382867600 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66e2ff0 0x5556b66eedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.657+0000 7fd63e040640 1 --2- 172.21.15.70:0/3612589256 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd6240a26a0 0x7fd6240a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.657+0000 7fd642f63740 1 -- 172.21.15.70:0/1245972774 wait complete. 2024-04-01T01:37:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.658+0000 7fd642f63740 1 -- start start 2024-04-01T01:37:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.658+0000 7fd642f63740 1 Processor -- start 2024-04-01T01:37:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.658+0000 7fd63e040640 1 --2- 172.21.15.70:0/3612589256 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd6240a26a0 0x7fd6240a4b60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x5580ffcb6fc0 tx=0x7fd62800c3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.658+0000 7f2bf0f98640 1 --2- 172.21.15.70:0/382867600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dde80 0x5556b66e7d70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2be8004390 tx=0x7f2be80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3a8a4f8db2e3284 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.658+0000 7fd642f63740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbaae0 0x5580ffcc4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.658+0000 7fd642f63740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbb4d0 0x5580ffcc5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.658+0000 7fd642f63740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcc0640 0x5580ffccc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.658+0000 7fd642f63740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5580ffcc3590 con 0x5580ffcc0640 2024-04-01T01:37:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.658+0000 7fd642f63740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5580ffbbcf80 con 0x5580ffcbb4d0 2024-04-01T01:37:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.658+0000 7fd642f63740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5580ffcc3710 con 0x5580ffcbaae0 2024-04-01T01:37:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.658+0000 7fd642f63740 10 client.0 _ll_register_callbacks cb 0x5580ffc15290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.659+0000 7fd63d83f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbb4d0 0x5580ffcc5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.659+0000 7fd63e040640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbaae0 0x5580ffcc4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.659+0000 7fd63d83f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbb4d0 0x5580ffcc5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:50420/0 (socket says 172.21.15.70:50420) 2024-04-01T01:37:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.659+0000 7fd63d83f640 1 -- 172.21.15.70:0/2266179514 learned_addr learned my addr 172.21.15.70:0/2266179514 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.659+0000 7fd63d83f640 1 -- 172.21.15.70:0/2266179514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbaae0 msgr2=0x5580ffcc4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.659+0000 7fd63d83f640 1 --2- 172.21.15.70:0/2266179514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbaae0 0x5580ffcc4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.659+0000 7fd63d83f640 1 -- 172.21.15.70:0/2266179514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcc0640 msgr2=0x5580ffccc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.659+0000 7fd63d83f640 1 --2- 172.21.15.70:0/2266179514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcc0640 0x5580ffccc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.659+0000 7fd63d83f640 1 -- 172.21.15.70:0/2266179514 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580ff9ff620 con 0x5580ffcbb4d0 2024-04-01T01:37:03.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.659+0000 7f2bd4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:37:03.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.659+0000 7f2bd4ff9640 1 -- 172.21.15.70:0/382867600 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be8003800 con 0x5556b66dde80 2024-04-01T01:37:03.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.659+0000 7f2bd4ff9640 1 -- 172.21.15.70:0/382867600 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f2be80039a0 con 0x5556b66dde80 2024-04-01T01:37:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.660+0000 7fd63d83f640 1 --2- 172.21.15.70:0/2266179514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbb4d0 0x5580ffcc5390 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd634097700 tx=0x7fd63409aee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a69ff74785c9c946 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.660+0000 7fd621ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:03.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.660+0000 7f2bf65d9740 1 -- 172.21.15.70:0/382867600 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2be4010af0 con 0x5556b66dde80 2024-04-01T01:37:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.660+0000 7fd621ffb640 1 -- 172.21.15.70:0/2266179514 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63409b2a0 con 0x5580ffcbb4d0 2024-04-01T01:37:03.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.660+0000 7fd621ffb640 1 -- 172.21.15.70:0/2266179514 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd6340cc3f0 con 0x5580ffcbb4d0 2024-04-01T01:37:03.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.660+0000 7f2bd4ff9640 1 -- 172.21.15.70:0/382867600 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2be8003ba0 con 0x5556b66dde80 2024-04-01T01:37:03.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.660+0000 7fd642f63740 1 -- 172.21.15.70:0/2266179514 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd630003e20 con 0x5580ffcbb4d0 2024-04-01T01:37:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.660+0000 7f2bf65d9740 10 client.26377 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.660+0000 7f2bf65d9740 1 -- 172.21.15.70:0/382867600 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5556b64207d0 con 0x5556b66dde80 2024-04-01T01:37:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.660+0000 7fd621ffb640 1 -- 172.21.15.70:0/2266179514 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd6340b4020 con 0x5580ffcbb4d0 2024-04-01T01:37:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.660+0000 7fd642f63740 10 client.26489 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.661+0000 7fd642f63740 1 -- 172.21.15.70:0/2266179514 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5580ffccf210 con 0x5580ffcbb4d0 2024-04-01T01:37:03.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.661+0000 7fd621ffb640 1 -- 172.21.15.70:0/2266179514 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd630003e20 con 0x5580ffcbb4d0 2024-04-01T01:37:03.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.662+0000 7f2bd4ff9640 1 -- 172.21.15.70:0/382867600 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2be4010af0 con 0x5556b66dde80 2024-04-01T01:37:03.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.662+0000 7fd621ffb640 1 -- 172.21.15.70:0/2266179514 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd634097100 con 0x5580ffcbb4d0 2024-04-01T01:37:03.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.662+0000 7fd642f63740 10 client.26489 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:03.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.662+0000 7fd642f63740 10 client.26489 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:03.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.662+0000 7fd642f63740 20 client.26489 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:03.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.662+0000 7fd642f63740 1 --2- 172.21.15.70:0/2266179514 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5580ffcd08e0 0x5580ffcd2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.662+0000 7fd642f63740 4 client.26489 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:03.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.662+0000 7fd642f63740 1 -- 172.21.15.70:0/2266179514 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580ffc32120 con 0x5580ffcd08e0 2024-04-01T01:37:03.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.663+0000 7f2bd4ff9640 1 -- 172.21.15.70:0/382867600 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2be8028070 con 0x5556b66dde80 2024-04-01T01:37:03.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.663+0000 7f2bf65d9740 10 client.26377 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:03.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.663+0000 7fd63e040640 1 --2- 172.21.15.70:0/2266179514 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5580ffcd08e0 0x5580ffcd2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.663+0000 7f2bf65d9740 10 client.26377 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:37:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.663+0000 7f2bf65d9740 20 client.26377 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.663+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/382867600 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5556b66f3340 0x5556b66f57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.663+0000 7f2bf65d9740 4 client.26377 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.663+0000 7f2bf65d9740 1 -- 172.21.15.70:0/382867600 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556b664f6f0 con 0x5556b66f3340 2024-04-01T01:37:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.663+0000 7f2bf1799640 1 --2- 172.21.15.70:0/382867600 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5556b66f3340 0x5556b66f57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.664+0000 7fd621ffb640 10 client.26489 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:03.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.664+0000 7fd63e040640 1 --2- 172.21.15.70:0/2266179514 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5580ffcd08e0 0x5580ffcd2d80 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8ebe7f60be6a992 server_cookie=3ebf50bc099e8fa7 in_seq=0 out_seq=0 2024-04-01T01:37:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.664+0000 7f2bf1799640 1 --2- 172.21.15.70:0/382867600 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5556b66f3340 0x5556b66f57a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8319945afc51c553 server_cookie=c81c10f234cc19b4 in_seq=0 out_seq=0 2024-04-01T01:37:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.664+0000 7f2bd4ff9640 10 client.26377 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:37:03.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.664+0000 7fd621ffb640 1 -- 172.21.15.70:0/2266179514 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5580ffc32120 con 0x5580ffcd08e0 2024-04-01T01:37:03.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.664+0000 7fd621ffb640 10 client.26489 handle_command_reply: tid=0 2024-04-01T01:37:03.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.665+0000 7fd642f63740 1 client.26489 shutdown 2024-04-01T01:37:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.665+0000 7f2bd4ff9640 1 -- 172.21.15.70:0/382867600 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5556b664f6f0 con 0x5556b66f3340 2024-04-01T01:37:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.665+0000 7f2bd4ff9640 10 client.26377 handle_command_reply: tid=0 2024-04-01T01:37:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.665+0000 7f2bf65d9740 1 client.26377 shutdown 2024-04-01T01:37:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.666+0000 7f2bf65d9740 1 -- 172.21.15.70:0/382867600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dde80 msgr2=0x5556b66e7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.666+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/382867600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dde80 0x5556b66e7d70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2be8004390 tx=0x7f2be80046c0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.666+0000 7f2bf65d9740 1 -- 172.21.15.70:0/382867600 shutdown_connections 2024-04-01T01:37:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.666+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/382867600 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5556b66f3340 0x5556b66f57a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.666+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/382867600 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66e2ff0 0x5556b66eedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.667+0000 7f2bd4ff9640 0 client.26377 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:37:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.667+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/382867600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dde80 0x5556b66e7d70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.667+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/382867600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dd490 0x5556b66e7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.667+0000 7f2bf65d9740 1 -- 172.21.15.70:0/382867600 >> 172.21.15.70:0/382867600 conn(0x5556b66d8a00 msgr2=0x5556b66d7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:03.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.667+0000 7fd642f63740 1 -- 172.21.15.70:0/2266179514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbb4d0 msgr2=0x5580ffcc5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.667+0000 7fd642f63740 1 --2- 172.21.15.70:0/2266179514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbb4d0 0x5580ffcc5390 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd634097700 tx=0x7fd63409aee0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.667+0000 7fd642f63740 1 -- 172.21.15.70:0/2266179514 shutdown_connections 2024-04-01T01:37:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.667+0000 7fd642f63740 1 --2- 172.21.15.70:0/2266179514 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5580ffcd08e0 0x5580ffcd2d80 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.667+0000 7fd642f63740 1 --2- 172.21.15.70:0/2266179514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcc0640 0x5580ffccc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.667+0000 7fd642f63740 1 --2- 172.21.15.70:0/2266179514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbb4d0 0x5580ffcc5390 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.667+0000 7fd642f63740 1 --2- 172.21.15.70:0/2266179514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbaae0 0x5580ffcc4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.667+0000 7fd642f63740 1 -- 172.21.15.70:0/2266179514 >> 172.21.15.70:0/2266179514 conn(0x5580ffcb6050 msgr2=0x5580ffcb5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:03.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.667+0000 7f2bf65d9740 1 -- 172.21.15.70:0/382867600 shutdown_connections 2024-04-01T01:37:03.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.667+0000 7fd621ffb640 0 client.26489 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.667+0000 7f2bf65d9740 1 -- 172.21.15.70:0/382867600 wait complete. 2024-04-01T01:37:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.667+0000 7f2bf65d9740 20 client.26377 trim_cache size 0 max 16384 2024-04-01T01:37:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.667+0000 7fd642f63740 1 -- 172.21.15.70:0/2266179514 shutdown_connections 2024-04-01T01:37:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.670+0000 7fd642f63740 1 -- 172.21.15.70:0/2266179514 wait complete. 2024-04-01T01:37:03.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.670+0000 7fd642f63740 20 client.26489 trim_cache size 0 max 16384 2024-04-01T01:37:03.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.693+0000 7f2bf65d9740 1 Processor -- start 2024-04-01T01:37:03.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.693+0000 7f2bf65d9740 1 -- start start 2024-04-01T01:37:03.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.693+0000 7f2bf65d9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66dd490 0x5556b66d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.693+0000 7f2bf65d9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 0x5556b66eed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.693+0000 7f2bf65d9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66e2ff0 0x5556b66dbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.693+0000 7f2bf1799640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66dd490 0x5556b66d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.693+0000 7f2bf1799640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66dd490 0x5556b66d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:50428/0 (socket says 172.21.15.70:50428) 2024-04-01T01:37:03.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.694+0000 7f2bf1799640 1 -- 172.21.15.70:0/3174007664 learned_addr learned my addr 172.21.15.70:0/3174007664 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:03.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.694+0000 7f2bf65d9740 1 -- 172.21.15.70:0/3174007664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd000fc60 con 0x5556b66dde80 2024-04-01T01:37:03.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.694+0000 7f2bf65d9740 1 -- 172.21.15.70:0/3174007664 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd00117a0 con 0x5556b66dd490 2024-04-01T01:37:03.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.694+0000 7f2bf65d9740 1 -- 172.21.15.70:0/3174007664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd0010330 con 0x5556b66e2ff0 2024-04-01T01:37:03.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.694+0000 7f2bf1f9a640 1 --2- 172.21.15.70:0/3174007664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66e2ff0 0x5556b66dbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.694+0000 7f2bf0f98640 1 --2- 172.21.15.70:0/3174007664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 0x5556b66eed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.694+0000 7f2bf1f9a640 1 -- 172.21.15.70:0/3174007664 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66dd490 msgr2=0x5556b66d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.694+0000 7f2bf1f9a640 1 --2- 172.21.15.70:0/3174007664 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66dd490 0x5556b66d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.694+0000 7f2bf1f9a640 1 -- 172.21.15.70:0/3174007664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 msgr2=0x5556b66eed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.694+0000 7f2bf1f9a640 1 --2- 172.21.15.70:0/3174007664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 0x5556b66eed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.694+0000 7f2bf1f9a640 1 -- 172.21.15.70:0/3174007664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556b66d4460 con 0x5556b66e2ff0 2024-04-01T01:37:03.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.695+0000 7f2bf1f9a640 1 --2- 172.21.15.70:0/3174007664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66e2ff0 0x5556b66dbc60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc0ee3d0 tx=0x7f2bdc0ee2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3914e4a83142a8a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:03.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.695+0000 7f2bd6ffd640 1 -- 172.21.15.70:0/3174007664 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc0bda90 con 0x5556b66e2ff0 2024-04-01T01:37:03.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.695+0000 7f2bd6ffd640 1 -- 172.21.15.70:0/3174007664 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2bdc0bd390 con 0x5556b66e2ff0 2024-04-01T01:37:03.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.695+0000 7f2bd6ffd640 1 -- 172.21.15.70:0/3174007664 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc0c10d0 con 0x5556b66e2ff0 2024-04-01T01:37:03.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.696+0000 7f2bf65d9740 1 -- 172.21.15.70:0/3174007664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66e2ff0 msgr2=0x5556b66dbc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.696+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/3174007664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66e2ff0 0x5556b66dbc60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc0ee3d0 tx=0x7f2bdc0ee2a0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.696+0000 7f2bf65d9740 1 -- 172.21.15.70:0/3174007664 shutdown_connections 2024-04-01T01:37:03.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.696+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/3174007664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66e2ff0 0x5556b66dbc60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.696+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/3174007664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 0x5556b66eed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.696+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/3174007664 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66dd490 0x5556b66d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.696+0000 7f2bf65d9740 1 -- 172.21.15.70:0/3174007664 >> 172.21.15.70:0/3174007664 conn(0x5556b66d8a00 msgr2=0x5556b66d8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:03.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.697+0000 7f2bf65d9740 1 -- 172.21.15.70:0/3174007664 shutdown_connections 2024-04-01T01:37:03.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.697+0000 7f2bf65d9740 1 -- 172.21.15.70:0/3174007664 wait complete. 2024-04-01T01:37:03.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.697+0000 7f2bf65d9740 1 -- start start 2024-04-01T01:37:03.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.697+0000 7f2bf65d9740 1 Processor -- start 2024-04-01T01:37:03.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.698+0000 7f2bf65d9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dd490 0x5556b66ee810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.698+0000 7f2bf65d9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 0x5556b66eed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.698+0000 7f2bf65d9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66e2ff0 0x5556b66ecea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.698+0000 7f2bf65d9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd00110d0 con 0x5556b66dde80 2024-04-01T01:37:03.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.698+0000 7f2bf65d9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd0010a00 con 0x5556b66e2ff0 2024-04-01T01:37:03.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.698+0000 7f2bf65d9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd000dcf0 con 0x5556b66dd490 2024-04-01T01:37:03.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.698+0000 7f2bf65d9740 10 client.0 _ll_register_callbacks cb 0x7f2be8004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:03.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.698+0000 7f2bf1799640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dd490 0x5556b66ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.698+0000 7f2bf1799640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dd490 0x5556b66ee810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45764/0 (socket says 172.21.15.70:45764) 2024-04-01T01:37:03.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.698+0000 7f2bf1799640 1 -- 172.21.15.70:0/155891926 learned_addr learned my addr 172.21.15.70:0/155891926 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:03.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.698+0000 7f2bf0f98640 1 --2- 172.21.15.70:0/155891926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 0x5556b66eed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.699+0000 7fd642f63740 1 Processor -- start 2024-04-01T01:37:03.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.699+0000 7f2bf1f9a640 1 --2- 172.21.15.70:0/155891926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66e2ff0 0x5556b66ecea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.699+0000 7fd642f63740 1 -- start start 2024-04-01T01:37:03.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.699+0000 7fd642f63740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbaae0 0x5580ffcc43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.699+0000 7fd642f63740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbb4d0 0x5580ffccc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.699+0000 7fd642f63740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcc0640 0x5580ffcb5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.699+0000 7fd642f63740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c00fb60 con 0x5580ffcc0640 2024-04-01T01:37:03.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.699+0000 7fd642f63740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c0116a0 con 0x5580ffcbaae0 2024-04-01T01:37:03.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.699+0000 7fd642f63740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c010230 con 0x5580ffcbb4d0 2024-04-01T01:37:03.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.699+0000 7f2bf0f98640 1 -- 172.21.15.70:0/155891926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dd490 msgr2=0x5556b66ee810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.699+0000 7f2bf0f98640 1 --2- 172.21.15.70:0/155891926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dd490 0x5556b66ee810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.699+0000 7f2bf0f98640 1 -- 172.21.15.70:0/155891926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66e2ff0 msgr2=0x5556b66ecea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.699+0000 7f2bf0f98640 1 --2- 172.21.15.70:0/155891926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66e2ff0 0x5556b66ecea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.699+0000 7f2bf0f98640 1 -- 172.21.15.70:0/155891926 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556b66ed6d0 con 0x5556b66dde80 2024-04-01T01:37:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.699+0000 7fd63e040640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbaae0 0x5580ffcc43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.699+0000 7fd63e841640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcc0640 0x5580ffcb5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.699+0000 7fd63e040640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbaae0 0x5580ffcc43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:50450/0 (socket says 172.21.15.70:50450) 2024-04-01T01:37:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.699+0000 7fd63e040640 1 -- 172.21.15.70:0/2682806881 learned_addr learned my addr 172.21.15.70:0/2682806881 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.700+0000 7fd63d83f640 1 --2- 172.21.15.70:0/2682806881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbb4d0 0x5580ffccc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.700+0000 7fd63d83f640 1 -- 172.21.15.70:0/2682806881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbaae0 msgr2=0x5580ffcc43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.700+0000 7fd63d83f640 1 --2- 172.21.15.70:0/2682806881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbaae0 0x5580ffcc43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.700+0000 7fd63d83f640 1 -- 172.21.15.70:0/2682806881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcc0640 msgr2=0x5580ffcb5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.700+0000 7fd63d83f640 1 --2- 172.21.15.70:0/2682806881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcc0640 0x5580ffcb5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.700+0000 7fd63d83f640 1 -- 172.21.15.70:0/2682806881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580ffcbaee0 con 0x5580ffcbb4d0 2024-04-01T01:37:03.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.700+0000 7f2bf0f98640 1 --2- 172.21.15.70:0/155891926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 0x5556b66eed50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f2be800de00 tx=0x7f2be802a810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53b786db1d1068c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:03.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.700+0000 7f2bd4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:37:03.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.700+0000 7fd63d83f640 1 --2- 172.21.15.70:0/2682806881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbb4d0 0x5580ffccc390 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd63409a200 tx=0x7fd634000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e96ca2b410b7693 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:03.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.701+0000 7f2bd4ff9640 1 -- 172.21.15.70:0/155891926 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be8038020 con 0x5556b66dde80 2024-04-01T01:37:03.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.701+0000 7f2bf65d9740 1 -- 172.21.15.70:0/155891926 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bdc0bd390 con 0x5556b66dde80 2024-04-01T01:37:03.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.701+0000 7f2bd4ff9640 1 -- 172.21.15.70:0/155891926 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f2be802ac10 con 0x5556b66dde80 2024-04-01T01:37:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.701+0000 7fd623fff640 1 -- 172.21.15.70:0/2682806881 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6340a7030 con 0x5580ffcbb4d0 2024-04-01T01:37:03.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.701+0000 7fd623fff640 1 -- 172.21.15.70:0/2682806881 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd63409f3f0 con 0x5580ffcbb4d0 2024-04-01T01:37:03.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.701+0000 7f2bd4ff9640 1 -- 172.21.15.70:0/155891926 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2be8040050 con 0x5556b66dde80 2024-04-01T01:37:03.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.702+0000 7f2bf65d9740 10 client.16638 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:03.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.702+0000 7f2bf65d9740 1 -- 172.21.15.70:0/155891926 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5556b675a350 con 0x5556b66dde80 2024-04-01T01:37:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.702+0000 7fd623fff640 1 -- 172.21.15.70:0/2682806881 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63409a4c0 con 0x5580ffcbb4d0 2024-04-01T01:37:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.702+0000 7fd642f63740 1 -- 172.21.15.70:0/2682806881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbb4d0 msgr2=0x5580ffccc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.702+0000 7fd642f63740 1 --2- 172.21.15.70:0/2682806881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbb4d0 0x5580ffccc390 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd63409a200 tx=0x7fd634000bc0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.703+0000 7f2bd4ff9640 1 -- 172.21.15.70:0/155891926 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2bdc0bd390 con 0x5556b66dde80 2024-04-01T01:37:03.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.703+0000 7f2bd4ff9640 1 -- 172.21.15.70:0/155891926 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2be8028040 con 0x5556b66dde80 2024-04-01T01:37:03.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.703+0000 7f2bf65d9740 10 client.16638 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:03.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.703+0000 7f2bf65d9740 10 client.16638 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:37:03.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.703+0000 7f2bf65d9740 20 client.16638 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:03.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.703+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/155891926 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5556b66f3340 0x5556b675cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.703+0000 7f2bf65d9740 4 client.16638 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:37:03.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.703+0000 7f2bf65d9740 1 -- 172.21.15.70:0/155891926 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5556b6649a40 con 0x5556b66f3340 2024-04-01T01:37:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.703+0000 7fd642f63740 1 -- 172.21.15.70:0/2682806881 shutdown_connections 2024-04-01T01:37:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.703+0000 7fd642f63740 1 --2- 172.21.15.70:0/2682806881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcc0640 0x5580ffcb5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.703+0000 7fd642f63740 1 --2- 172.21.15.70:0/2682806881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcbb4d0 0x5580ffccc390 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.703+0000 7fd642f63740 1 --2- 172.21.15.70:0/2682806881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbaae0 0x5580ffcc43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.703+0000 7fd642f63740 1 -- 172.21.15.70:0/2682806881 >> 172.21.15.70:0/2682806881 conn(0x5580ffcb6050 msgr2=0x5580ffcb4900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.703+0000 7fd642f63740 1 -- 172.21.15.70:0/2682806881 shutdown_connections 2024-04-01T01:37:03.706 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.703+0000 7fd642f63740 1 -- 172.21.15.70:0/2682806881 wait complete. 2024-04-01T01:37:03.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.703+0000 7f2bf1799640 1 --2- 172.21.15.70:0/155891926 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5556b66f3340 0x5556b675cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.704+0000 7fd642f63740 1 -- start start 2024-04-01T01:37:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.704+0000 7fd642f63740 1 Processor -- start 2024-04-01T01:37:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.704+0000 7fd642f63740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcbaae0 0x5580ffcc99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.704+0000 7fd642f63740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbb4d0 0x5580ffcc9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.704+0000 7fd642f63740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcc0640 0x5580ffccaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.704+0000 7fd642f63740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c010fd0 con 0x5580ffcbaae0 2024-04-01T01:37:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.704+0000 7fd642f63740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c010900 con 0x5580ffcbb4d0 2024-04-01T01:37:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.704+0000 7fd642f63740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd61c00dbf0 con 0x5580ffcc0640 2024-04-01T01:37:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.704+0000 7fd642f63740 10 client.0 _ll_register_callbacks cb 0x7fd63409b0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:03.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.704+0000 7f2bd4ff9640 10 client.16638 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:37:03.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.705+0000 7fd63e040640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcbaae0 0x5580ffcc99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.705+0000 7f2bf1799640 1 --2- 172.21.15.70:0/155891926 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5556b66f3340 0x5556b675cdc0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ca20cdec36deaa0a server_cookie=5154539aa929942e in_seq=0 out_seq=0 2024-04-01T01:37:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.705+0000 7fd63e040640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcbaae0 0x5580ffcc99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56652/0 (socket says 172.21.15.70:56652) 2024-04-01T01:37:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.705+0000 7fd63e040640 1 -- 172.21.15.70:0/844817117 learned_addr learned my addr 172.21.15.70:0/844817117 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.705+0000 7fd63e841640 1 --2- 172.21.15.70:0/844817117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcc0640 0x5580ffccaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.705+0000 7fd63d83f640 1 --2- 172.21.15.70:0/844817117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbb4d0 0x5580ffcc9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.705+0000 7f2bd4ff9640 1 -- 172.21.15.70:0/155891926 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5556b6649a40 con 0x5556b66f3340 2024-04-01T01:37:03.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.705+0000 7f2bd4ff9640 10 client.16638 handle_command_reply: tid=0 2024-04-01T01:37:03.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.705+0000 7f2bf65d9740 1 client.16638 shutdown 2024-04-01T01:37:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.705+0000 7fd63e040640 1 -- 172.21.15.70:0/844817117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcc0640 msgr2=0x5580ffccaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.705+0000 7fd63e040640 1 --2- 172.21.15.70:0/844817117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcc0640 0x5580ffccaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.705+0000 7fd63e040640 1 -- 172.21.15.70:0/844817117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbb4d0 msgr2=0x5580ffcc9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.705+0000 7fd63e040640 1 --2- 172.21.15.70:0/844817117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbb4d0 0x5580ffcc9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.710 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.705+0000 7fd63e040640 1 -- 172.21.15.70:0/844817117 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580ffccb4a0 con 0x5580ffcbaae0 2024-04-01T01:37:03.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.705+0000 7f2bf65d9740 1 -- 172.21.15.70:0/155891926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 msgr2=0x5556b66eed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.705+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/155891926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 0x5556b66eed50 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f2be800de00 tx=0x7f2be802a810 comp rx=0 tx=0).stop 2024-04-01T01:37:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.705+0000 7fd63e040640 1 --2- 172.21.15.70:0/844817117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcbaae0 0x5580ffcc99b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd62800efe0 tx=0x7fd628011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66620a8143b8bd8d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:03.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.705+0000 7fd621ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:37:03.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.706+0000 7f2bf65d9740 1 -- 172.21.15.70:0/155891926 shutdown_connections 2024-04-01T01:37:03.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.706+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/155891926 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5556b66f3340 0x5556b675cdc0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.706+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/155891926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5556b66e2ff0 0x5556b66ecea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.706+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/155891926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5556b66dde80 0x5556b66eed50 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.706+0000 7f2bf65d9740 1 --2- 172.21.15.70:0/155891926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556b66dd490 0x5556b66ee810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.706+0000 7f2bf65d9740 1 -- 172.21.15.70:0/155891926 >> 172.21.15.70:0/155891926 conn(0x5556b66d8a00 msgr2=0x5556b66e23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:03.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.706+0000 7f2bd4ff9640 0 client.16638 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:37:03.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.706+0000 7fd621ffb640 1 -- 172.21.15.70:0/844817117 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd628025600 con 0x5580ffcbaae0 2024-04-01T01:37:03.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.706+0000 7f2bf65d9740 1 -- 172.21.15.70:0/155891926 shutdown_connections 2024-04-01T01:37:03.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.706+0000 7f2bf65d9740 1 -- 172.21.15.70:0/155891926 wait complete. 2024-04-01T01:37:03.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.706+0000 7fd621ffb640 1 -- 172.21.15.70:0/844817117 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd6280257a0 con 0x5580ffcbaae0 2024-04-01T01:37:03.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.706+0000 7f2bf65d9740 20 client.16638 trim_cache size 0 max 16384 2024-04-01T01:37:03.713 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.707+0000 7fd642f63740 1 -- 172.21.15.70:0/844817117 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd63409f3f0 con 0x5580ffcbaae0 2024-04-01T01:37:03.714 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:37:03.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.708+0000 7f2bf39fd640 1 -- 172.21.15.70:0/3939361321 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2bd80a26a0 msgr2=0x7f2bd80a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.708+0000 7f2bf39fd640 1 --2- 172.21.15.70:0/3939361321 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2bd80a26a0 0x7f2bd80a4b60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x5556b66de6e0 tx=0x7f2be800b390 comp rx=0 tx=0).stop 2024-04-01T01:37:03.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.708+0000 7f2bf39fd640 1 -- 172.21.15.70:0/3939361321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bec116560 msgr2=0x7f2bec1adcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.708+0000 7f2bf39fd640 1 --2- 172.21.15.70:0/3939361321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bec116560 0x7f2bec1adcc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc036040 tx=0x7f2bdc004270 comp rx=0 tx=0).stop 2024-04-01T01:37:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.708+0000 7fd621ffb640 1 -- 172.21.15.70:0/844817117 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd62802e480 con 0x5580ffcbaae0 2024-04-01T01:37:03.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.708+0000 7fd642f63740 10 client.16644 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:03.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.708+0000 7f2bf39fd640 1 -- 172.21.15.70:0/3939361321 shutdown_connections 2024-04-01T01:37:03.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.708+0000 7f2bf39fd640 1 --2- 172.21.15.70:0/3939361321 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2bd80a26a0 0x7f2bd80a4b60 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.708+0000 7f2bf39fd640 1 --2- 172.21.15.70:0/3939361321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bec116560 0x7f2bec1adcc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.708+0000 7f2bf39fd640 1 --2- 172.21.15.70:0/3939361321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bec06cc40 0x7f2bec1a6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.708+0000 7f2bf39fd640 1 --2- 172.21.15.70:0/3939361321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2bec06b660 0x7f2bec1a6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.708+0000 7f2bf39fd640 1 -- 172.21.15.70:0/3939361321 >> 172.21.15.70:0/3939361321 conn(0x7f2bec0706f0 msgr2=0x7f2bec109940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:03.716 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.708+0000 7fd642f63740 1 -- 172.21.15.70:0/844817117 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5580ffd47c90 con 0x5580ffcbaae0 2024-04-01T01:37:03.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.709+0000 7f2bf39fd640 1 -- 172.21.15.70:0/3939361321 shutdown_connections 2024-04-01T01:37:03.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.709+0000 7f2bf39fd640 1 -- 172.21.15.70:0/3939361321 wait complete. 2024-04-01T01:37:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.709+0000 7fd621ffb640 1 -- 172.21.15.70:0/844817117 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd63409f3f0 con 0x5580ffcbaae0 2024-04-01T01:37:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.709+0000 7fd621ffb640 1 -- 172.21.15.70:0/844817117 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd6280107e0 con 0x5580ffcbaae0 2024-04-01T01:37:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.711+0000 7fd642f63740 10 client.16644 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.711+0000 7fd642f63740 10 client.16644 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.711+0000 7fd642f63740 20 client.16644 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.711+0000 7fd642f63740 1 --2- 172.21.15.70:0/844817117 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5580ffd48670 0x5580ffd4aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.711+0000 7fd642f63740 4 client.16644 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:37:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.711+0000 7fd642f63740 1 -- 172.21.15.70:0/844817117 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5580ffc2be30 con 0x5580ffd48670 2024-04-01T01:37:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.711+0000 7fd63d83f640 1 --2- 172.21.15.70:0/844817117 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5580ffd48670 0x5580ffd4aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.712+0000 7fd621ffb640 10 client.16644 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:03.718 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.712+0000 7fd63d83f640 1 --2- 172.21.15.70:0/844817117 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5580ffd48670 0x5580ffd4aa90 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2c1ad6de7be8269 server_cookie=314ac96f84b2fe97 in_seq=0 out_seq=0 2024-04-01T01:37:03.736 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:37:03.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.750+0000 7fd621ffb640 1 -- 172.21.15.70:0/844817117 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5580ffc2be30 con 0x5580ffd48670 2024-04-01T01:37:03.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.750+0000 7fd621ffb640 10 client.16644 handle_command_reply: tid=0 2024-04-01T01:37:03.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.750+0000 7fd642f63740 1 client.16644 shutdown 2024-04-01T01:37:03.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.750+0000 7fd642f63740 1 -- 172.21.15.70:0/844817117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcbaae0 msgr2=0x5580ffcc99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.750+0000 7fd642f63740 1 --2- 172.21.15.70:0/844817117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcbaae0 0x5580ffcc99b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd62800efe0 tx=0x7fd628011000 comp rx=0 tx=0).stop 2024-04-01T01:37:03.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.751+0000 7fd642f63740 1 -- 172.21.15.70:0/844817117 shutdown_connections 2024-04-01T01:37:03.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.751+0000 7fd642f63740 1 --2- 172.21.15.70:0/844817117 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5580ffd48670 0x5580ffd4aa90 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.751+0000 7fd642f63740 1 --2- 172.21.15.70:0/844817117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580ffcc0640 0x5580ffccaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.751+0000 7fd642f63740 1 --2- 172.21.15.70:0/844817117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580ffcbb4d0 0x5580ffcc9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.751+0000 7fd642f63740 1 --2- 172.21.15.70:0/844817117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580ffcbaae0 0x5580ffcc99b0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.751+0000 7fd642f63740 1 -- 172.21.15.70:0/844817117 >> 172.21.15.70:0/844817117 conn(0x5580ffcb6050 msgr2=0x5580ffcb5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:03.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.752+0000 7fd621ffb640 0 client.16644 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:03.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.752+0000 7fd642f63740 1 -- 172.21.15.70:0/844817117 shutdown_connections 2024-04-01T01:37:03.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.752+0000 7fd642f63740 1 -- 172.21.15.70:0/844817117 wait complete. 2024-04-01T01:37:03.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.753+0000 7fd642f63740 20 client.16644 trim_cache size 0 max 16384 2024-04-01T01:37:03.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:37:03.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:37:03.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:37:03.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:37:03.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.755+0000 7fd63f042640 1 -- 172.21.15.70:0/3612589256 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd6240a26a0 msgr2=0x7fd6240a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.755+0000 7fd63f042640 1 --2- 172.21.15.70:0/3612589256 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd6240a26a0 0x7fd6240a4b60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x5580ffcb6fc0 tx=0x7fd62800c3c0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.756+0000 7fd63f042640 1 -- 172.21.15.70:0/3612589256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd638077350 msgr2=0x7fd6380705d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.756+0000 7fd63f042640 1 --2- 172.21.15.70:0/3612589256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd638077350 0x7fd6380705d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd634005fd0 tx=0x7fd63400ee50 comp rx=0 tx=0).stop 2024-04-01T01:37:03.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.756+0000 7fd63f042640 1 -- 172.21.15.70:0/3612589256 shutdown_connections 2024-04-01T01:37:03.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.756+0000 7fd63f042640 1 --2- 172.21.15.70:0/3612589256 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd6240a26a0 0x7fd6240a4b60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.756+0000 7fd63f042640 1 --2- 172.21.15.70:0/3612589256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd638077cf0 0x7fd6381adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.756+0000 7fd63f042640 1 --2- 172.21.15.70:0/3612589256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd638077350 0x7fd6380705d0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.756+0000 7fd63f042640 1 --2- 172.21.15.70:0/3612589256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd638076110 0x7fd638070090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.755 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.756+0000 7fd63f042640 1 -- 172.21.15.70:0/3612589256 >> 172.21.15.70:0/3612589256 conn(0x7fd6380fed60 msgr2=0x7fd638100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:03.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.756+0000 7fd63f042640 1 -- 172.21.15.70:0/3612589256 shutdown_connections 2024-04-01T01:37:03.756 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:03.756+0000 7fd63f042640 1 -- 172.21.15.70:0/3612589256 wait complete. 2024-04-01T01:37:03.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.944+0000 7ff30f167640 1 -- 172.21.15.70:0/2839224847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3000a6710 msgr2=0x7ff3000a6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.944+0000 7ff30f167640 1 --2- 172.21.15.70:0/2839224847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3000a6710 0x7ff3000a6b90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff2f800aa10 tx=0x7ff2f802cf60 comp rx=0 tx=0).stop 2024-04-01T01:37:03.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.944+0000 7ff30f167640 1 -- 172.21.15.70:0/2839224847 shutdown_connections 2024-04-01T01:37:03.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.944+0000 7ff30f167640 1 --2- 172.21.15.70:0/2839224847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3000ab410 0x7ff3000ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.944+0000 7ff30f167640 1 --2- 172.21.15.70:0/2839224847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3000a6710 0x7ff3000a6b90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.944+0000 7ff30f167640 1 --2- 172.21.15.70:0/2839224847 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3000182a0 0x7ff3000a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.944+0000 7ff30f167640 1 -- 172.21.15.70:0/2839224847 >> 172.21.15.70:0/2839224847 conn(0x7ff3000a0530 msgr2=0x7ff3000a2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:03.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.945+0000 7ff30f167640 1 -- 172.21.15.70:0/2839224847 shutdown_connections 2024-04-01T01:37:03.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.945+0000 7ff30f167640 1 -- 172.21.15.70:0/2839224847 wait complete. 2024-04-01T01:37:03.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.946+0000 7ff30f167640 1 Processor -- start 2024-04-01T01:37:03.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.946+0000 7ff30f167640 1 -- start start 2024-04-01T01:37:03.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.946+0000 7ff30f167640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3000182a0 0x7ff3001435a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.946+0000 7ff30f167640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3000a6710 0x7ff300143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.946+0000 7ff30f167640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3000ab410 0x7ff30014ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.946+0000 7ff30f167640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff3000b4b70 con 0x7ff3000182a0 2024-04-01T01:37:03.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.946+0000 7ff30f167640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff3000b49f0 con 0x7ff3000ab410 2024-04-01T01:37:03.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.946+0000 7ff30f167640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff3000b4cf0 con 0x7ff3000a6710 2024-04-01T01:37:03.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.947+0000 7ff30d964640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3000a6710 0x7ff300143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.947+0000 7ff30d964640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3000a6710 0x7ff300143ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45804/0 (socket says 172.21.15.70:45804) 2024-04-01T01:37:03.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.947+0000 7ff30d964640 1 -- 172.21.15.70:0/2125837826 learned_addr learned my addr 172.21.15.70:0/2125837826 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:03.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.947+0000 7ff30e165640 1 --2- 172.21.15.70:0/2125837826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3000182a0 0x7ff3001435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.947+0000 7ff30e966640 1 --2- 172.21.15.70:0/2125837826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3000ab410 0x7ff30014ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.947+0000 7ff30d964640 1 -- 172.21.15.70:0/2125837826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3000ab410 msgr2=0x7ff30014ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.947+0000 7ff30d964640 1 --2- 172.21.15.70:0/2125837826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3000ab410 0x7ff30014ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.947+0000 7ff30d964640 1 -- 172.21.15.70:0/2125837826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3000182a0 msgr2=0x7ff3001435a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.947+0000 7ff30d964640 1 --2- 172.21.15.70:0/2125837826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3000182a0 0x7ff3001435a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.947+0000 7ff30d964640 1 -- 172.21.15.70:0/2125837826 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3000b5670 con 0x7ff3000a6710 2024-04-01T01:37:03.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.948+0000 7ff30d964640 1 --2- 172.21.15.70:0/2125837826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3000a6710 0x7ff300143ae0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8000c00 tx=0x7ff2f8005200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:03.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.948+0000 7ff2ff7fe640 1 -- 172.21.15.70:0/2125837826 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8042020 con 0x7ff3000a6710 2024-04-01T01:37:03.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.948+0000 7ff2ff7fe640 1 -- 172.21.15.70:0/2125837826 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2f8004450 con 0x7ff3000a6710 2024-04-01T01:37:03.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.948+0000 7ff2ff7fe640 1 -- 172.21.15.70:0/2125837826 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f803e730 con 0x7ff3000a6710 2024-04-01T01:37:03.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.948+0000 7ff30f167640 1 -- 172.21.15.70:0/2125837826 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3000b5900 con 0x7ff3000a6710 2024-04-01T01:37:03.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.948+0000 7ff30f167640 1 -- 172.21.15.70:0/2125837826 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3000b5e00 con 0x7ff3000a6710 2024-04-01T01:37:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.952+0000 7ff312faf740 1 Processor -- start 2024-04-01T01:37:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.952+0000 7ff2ff7fe640 1 -- 172.21.15.70:0/2125837826 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7ff2f8045020 con 0x7ff3000a6710 2024-04-01T01:37:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.953+0000 7ff312faf740 1 -- start start 2024-04-01T01:37:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.953+0000 7ff312faf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f1046a0 0x55787f104ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.953+0000 7ff312faf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f105090 0x55787f109b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.953+0000 7ff312faf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f10a200 0x55787f10c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.953+0000 7ff312faf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55787f0691b0 con 0x55787f10a200 2024-04-01T01:37:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.953+0000 7ff312faf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55787f011c00 con 0x55787f1046a0 2024-04-01T01:37:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.953+0000 7ff312faf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55787f065250 con 0x55787f105090 2024-04-01T01:37:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.953+0000 7ff30d964640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f10a200 0x55787f10c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.953+0000 7ff30d964640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f10a200 0x55787f10c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56688/0 (socket says 172.21.15.70:56688) 2024-04-01T01:37:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.953+0000 7ff30d964640 1 -- 172.21.15.70:0/47744851 learned_addr learned my addr 172.21.15.70:0/47744851 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.954+0000 7ff2ff7fe640 1 --2- 172.21.15.70:0/2125837826 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2f40a2480 0x7ff2f40a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.954+0000 7ff30e165640 1 --2- 172.21.15.70:0/47744851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f105090 0x55787f109b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.954+0000 7ff30e966640 1 --2- 172.21.15.70:0/47744851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f1046a0 0x55787f104ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.954+0000 7ff30d964640 1 -- 172.21.15.70:0/47744851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f105090 msgr2=0x55787f109b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.954+0000 7ff30d964640 1 --2- 172.21.15.70:0/47744851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f105090 0x55787f109b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.954+0000 7ff30d964640 1 -- 172.21.15.70:0/47744851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f1046a0 msgr2=0x55787f104ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.954+0000 7ff30d964640 1 --2- 172.21.15.70:0/47744851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f1046a0 0x55787f104ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.954+0000 7ff30d964640 1 -- 172.21.15.70:0/47744851 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55787f0313d0 con 0x55787f10a200 2024-04-01T01:37:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.954+0000 7ff2ff7fe640 1 -- 172.21.15.70:0/2125837826 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff2f80f10b0 con 0x7ff3000a6710 2024-04-01T01:37:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.954+0000 7ff30e966640 1 --2- 172.21.15.70:0/2125837826 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2f40a2480 0x7ff2f40a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.954+0000 7ff30d964640 1 --2- 172.21.15.70:0/47744851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f10a200 0x55787f10c690 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff2f80d6150 tx=0x7ff2f80b94f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d561d3f4bf1390 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.955+0000 7ff2fcbfe640 1 -- 172.21.15.70:0/47744851 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f80bd570 con 0x55787f10a200 2024-04-01T01:37:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.955+0000 7ff2fcbfe640 1 -- 172.21.15.70:0/47744851 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2f80dd070 con 0x55787f10a200 2024-04-01T01:37:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.955+0000 7ff2fcbfe640 1 -- 172.21.15.70:0/47744851 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f80c6400 con 0x55787f10a200 2024-04-01T01:37:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.956+0000 7ff312faf740 1 -- 172.21.15.70:0/47744851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f10a200 msgr2=0x55787f10c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.956+0000 7ff312faf740 1 --2- 172.21.15.70:0/47744851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f10a200 0x55787f10c690 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff2f80d6150 tx=0x7ff2f80b94f0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.956+0000 7ff30e966640 1 --2- 172.21.15.70:0/2125837826 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2f40a2480 0x7ff2f40a4940 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff308000b60 tx=0x7ff3080675f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.956+0000 7ff312faf740 1 -- 172.21.15.70:0/47744851 shutdown_connections 2024-04-01T01:37:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.956+0000 7ff312faf740 1 --2- 172.21.15.70:0/47744851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f10a200 0x55787f10c690 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.956+0000 7ff312faf740 1 --2- 172.21.15.70:0/47744851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f105090 0x55787f109b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.956+0000 7ff312faf740 1 --2- 172.21.15.70:0/47744851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f1046a0 0x55787f104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.956+0000 7ff312faf740 1 -- 172.21.15.70:0/47744851 >> 172.21.15.70:0/47744851 conn(0x55787f0ffc10 msgr2=0x55787f100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.957+0000 7ff312faf740 1 -- 172.21.15.70:0/47744851 shutdown_connections 2024-04-01T01:37:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.958+0000 7ff312faf740 1 -- 172.21.15.70:0/47744851 wait complete. 2024-04-01T01:37:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.958+0000 7ff312faf740 1 -- start start 2024-04-01T01:37:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.959+0000 7ff312faf740 1 Processor -- start 2024-04-01T01:37:03.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.960+0000 7ff312faf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1046a0 0x55787f10e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.960+0000 7ff30e165640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1046a0 0x55787f10e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.960+0000 7ff312faf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f105090 0x55787f10ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.960+0000 7ff30e165640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1046a0 0x55787f10e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45814/0 (socket says 172.21.15.70:45814) 2024-04-01T01:37:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.961+0000 7ff312faf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 0x55787f115e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.961+0000 7ff312faf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55787f10cf70 con 0x55787f105090 2024-04-01T01:37:03.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.961+0000 7ff312faf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55787f007470 con 0x55787f10a200 2024-04-01T01:37:03.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.961+0000 7ff30e966640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 0x55787f115e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.961+0000 7ff30e966640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 0x55787f115e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:50512/0 (socket says 172.21.15.70:50512) 2024-04-01T01:37:03.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.961+0000 7ff30e966640 1 -- 172.21.15.70:0/1503538498 learned_addr learned my addr 172.21.15.70:0/1503538498 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:03.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.961+0000 7ff312faf740 1 -- 172.21.15.70:0/1503538498 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55787f10d0f0 con 0x55787f1046a0 2024-04-01T01:37:03.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.961+0000 7ff312faf740 10 client.0 _ll_register_callbacks cb 0x55787f024cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.961+0000 7ff30d964640 1 --2- 172.21.15.70:0/1503538498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f105090 0x55787f10ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.962+0000 7ff30e966640 1 -- 172.21.15.70:0/1503538498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1046a0 msgr2=0x55787f10e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.962+0000 7ff30e966640 1 --2- 172.21.15.70:0/1503538498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1046a0 0x55787f10e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.962+0000 7ff30e966640 1 -- 172.21.15.70:0/1503538498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f105090 msgr2=0x55787f10ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.962+0000 7ff30e966640 1 --2- 172.21.15.70:0/1503538498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f105090 0x55787f10ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.962+0000 7ff30e966640 1 -- 172.21.15.70:0/1503538498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55787ed7d3f0 con 0x55787f10a200 2024-04-01T01:37:03.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.965+0000 7ff30e966640 1 --2- 172.21.15.70:0/1503538498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 0x55787f115e50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff308062d10 tx=0x7ff308067a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bf84f9706829b2c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:03.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.967+0000 7ff2f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:03.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.967+0000 7ff2f1ffb640 1 -- 172.21.15.70:0/1503538498 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff308074b30 con 0x55787f10a200 2024-04-01T01:37:03.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.967+0000 7ff312faf740 1 -- 172.21.15.70:0/1503538498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2f80dd070 con 0x55787f10a200 2024-04-01T01:37:03.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.967+0000 7ff2f1ffb640 1 -- 172.21.15.70:0/1503538498 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff308074cd0 con 0x55787f10a200 2024-04-01T01:37:03.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.967+0000 7ff2f1ffb640 1 -- 172.21.15.70:0/1503538498 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff308091050 con 0x55787f10a200 2024-04-01T01:37:03.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.968+0000 7ff312faf740 10 client.26510 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:03.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.968+0000 7ff312faf740 1 -- 172.21.15.70:0/1503538498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55787ee477d0 con 0x55787f10a200 2024-04-01T01:37:03.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.968+0000 7ff2f1ffb640 1 -- 172.21.15.70:0/1503538498 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2f80dd070 con 0x55787f10a200 2024-04-01T01:37:03.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.969+0000 7ff2f1ffb640 1 -- 172.21.15.70:0/1503538498 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff30808c410 con 0x55787f10a200 2024-04-01T01:37:03.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.969+0000 7ff312faf740 10 client.26510 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:03.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.969+0000 7ff312faf740 10 client.26510 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:37:03.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.969+0000 7ff312faf740 20 client.26510 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:03.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.969+0000 7ff312faf740 1 --2- 172.21.15.70:0/1503538498 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55787f11a780 0x55787f11cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:03.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.969+0000 7ff312faf740 4 client.26510 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:03.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.969+0000 7ff312faf740 1 -- 172.21.15.70:0/1503538498 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55787f0766f0 con 0x55787f11a780 2024-04-01T01:37:03.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.969+0000 7ff30e165640 1 --2- 172.21.15.70:0/1503538498 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55787f11a780 0x55787f11cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:03.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.971+0000 7ff30e165640 1 --2- 172.21.15.70:0/1503538498 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55787f11a780 0x55787f11cbd0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f54d1e0f5eb06cbc server_cookie=433693b4db5d5b37 in_seq=0 out_seq=0 2024-04-01T01:37:03.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.971+0000 7ff2f1ffb640 10 client.26510 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:37:03.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.973+0000 7ff2f1ffb640 1 -- 172.21.15.70:0/1503538498 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55787f0766f0 con 0x55787f11a780 2024-04-01T01:37:03.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.973+0000 7ff2f1ffb640 10 client.26510 handle_command_reply: tid=0 2024-04-01T01:37:03.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.973+0000 7ff312faf740 1 client.26510 shutdown 2024-04-01T01:37:03.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.974+0000 7ff312faf740 1 -- 172.21.15.70:0/1503538498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 msgr2=0x55787f115e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:03.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.974+0000 7ff312faf740 1 --2- 172.21.15.70:0/1503538498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 0x55787f115e50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff308062d10 tx=0x7ff308067a20 comp rx=0 tx=0).stop 2024-04-01T01:37:03.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.974+0000 7ff312faf740 1 -- 172.21.15.70:0/1503538498 shutdown_connections 2024-04-01T01:37:03.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.974+0000 7ff312faf740 1 --2- 172.21.15.70:0/1503538498 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55787f11a780 0x55787f11cbd0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.974+0000 7ff312faf740 1 --2- 172.21.15.70:0/1503538498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 0x55787f115e50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.974+0000 7ff312faf740 1 --2- 172.21.15.70:0/1503538498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f105090 0x55787f10ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.974+0000 7ff312faf740 1 --2- 172.21.15.70:0/1503538498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1046a0 0x55787f10e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:03.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.974+0000 7ff312faf740 1 -- 172.21.15.70:0/1503538498 >> 172.21.15.70:0/1503538498 conn(0x55787f0ffc10 msgr2=0x55787f0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:03.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.975+0000 7ff2f1ffb640 0 client.26510 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:37:03.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.975+0000 7ff312faf740 1 -- 172.21.15.70:0/1503538498 shutdown_connections 2024-04-01T01:37:03.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.975+0000 7ff312faf740 1 -- 172.21.15.70:0/1503538498 wait complete. 2024-04-01T01:37:03.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:03.975+0000 7ff312faf740 20 client.26510 trim_cache size 0 max 16384 2024-04-01T01:37:04.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:03 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1505750126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:37:04.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.011+0000 7ff312faf740 1 Processor -- start 2024-04-01T01:37:04.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.011+0000 7ff312faf740 1 -- start start 2024-04-01T01:37:04.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.012+0000 7ff312faf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1046a0 0x55787f0fe700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.012+0000 7ff312faf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f105090 0x55787f115e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.012+0000 7ff312faf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 0x55787f0fec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.012+0000 7ff312faf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff30015e8e0 con 0x55787f105090 2024-04-01T01:37:04.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.012+0000 7ff312faf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff300160420 con 0x55787f10a200 2024-04-01T01:37:04.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.012+0000 7ff312faf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff30015efb0 con 0x55787f1046a0 2024-04-01T01:37:04.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.012+0000 7ff30e165640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1046a0 0x55787f0fe700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.012+0000 7ff30e165640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1046a0 0x55787f0fe700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45828/0 (socket says 172.21.15.70:45828) 2024-04-01T01:37:04.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.012+0000 7ff30e165640 1 -- 172.21.15.70:0/3304831778 learned_addr learned my addr 172.21.15.70:0/3304831778 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.012+0000 7ff30d964640 1 --2- 172.21.15.70:0/3304831778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f105090 0x55787f115e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.012+0000 7ff30e966640 1 --2- 172.21.15.70:0/3304831778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 0x55787f0fec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.013+0000 7ff30e165640 1 -- 172.21.15.70:0/3304831778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 msgr2=0x55787f0fec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.013+0000 7ff30e165640 1 --2- 172.21.15.70:0/3304831778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 0x55787f0fec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.013+0000 7ff30e165640 1 -- 172.21.15.70:0/3304831778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f105090 msgr2=0x55787f115e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.013+0000 7ff30e165640 1 --2- 172.21.15.70:0/3304831778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f105090 0x55787f115e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.013+0000 7ff30e165640 1 -- 172.21.15.70:0/3304831778 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55787f100980 con 0x55787f1046a0 2024-04-01T01:37:04.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.013+0000 7ff30e165640 1 --2- 172.21.15.70:0/3304831778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1046a0 0x55787f0fe700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55787f105490 tx=0x7ff304005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5e25525e501d6be server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.013+0000 7ff2f3fff640 1 -- 172.21.15.70:0/3304831778 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff304003d20 con 0x55787f1046a0 2024-04-01T01:37:04.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.014+0000 7ff2f3fff640 1 -- 172.21.15.70:0/3304831778 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff304010430 con 0x55787f1046a0 2024-04-01T01:37:04.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.014+0000 7ff2f3fff640 1 -- 172.21.15.70:0/3304831778 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30401e5f0 con 0x55787f1046a0 2024-04-01T01:37:04.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.015+0000 7ff312faf740 1 -- 172.21.15.70:0/3304831778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1046a0 msgr2=0x55787f0fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.015+0000 7ff312faf740 1 --2- 172.21.15.70:0/3304831778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1046a0 0x55787f0fe700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55787f105490 tx=0x7ff304005f90 comp rx=0 tx=0).stop 2024-04-01T01:37:04.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.015+0000 7ff312faf740 1 -- 172.21.15.70:0/3304831778 shutdown_connections 2024-04-01T01:37:04.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.015+0000 7ff312faf740 1 --2- 172.21.15.70:0/3304831778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 0x55787f0fec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.015+0000 7ff312faf740 1 --2- 172.21.15.70:0/3304831778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f105090 0x55787f115e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.015+0000 7ff312faf740 1 --2- 172.21.15.70:0/3304831778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1046a0 0x55787f0fe700 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55787f105490 tx=0x7ff304005f90 comp rx=0 tx=0).stop 2024-04-01T01:37:04.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.015+0000 7ff312faf740 1 -- 172.21.15.70:0/3304831778 >> 172.21.15.70:0/3304831778 conn(0x55787f0ffc10 msgr2=0x55787f106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.016+0000 7ff312faf740 1 -- 172.21.15.70:0/3304831778 shutdown_connections 2024-04-01T01:37:04.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.017+0000 7ff312faf740 1 -- 172.21.15.70:0/3304831778 wait complete. 2024-04-01T01:37:04.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.017+0000 7ff312faf740 1 -- start start 2024-04-01T01:37:04.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.017+0000 7ff312faf740 1 Processor -- start 2024-04-01T01:37:04.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.018+0000 7ff312faf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f105090 0x55787f113470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.018+0000 7ff312faf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 0x55787f1139b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.018+0000 7ff312faf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1149c0 0x55787f114e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.018+0000 7ff312faf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff30015fd50 con 0x55787f105090 2024-04-01T01:37:04.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.018+0000 7ff312faf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff30015f680 con 0x55787f10a200 2024-04-01T01:37:04.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.018+0000 7ff312faf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff30015c9b0 con 0x55787f1149c0 2024-04-01T01:37:04.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.018+0000 7ff312faf740 10 client.0 _ll_register_callbacks cb 0x7ff308075720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:04.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.018+0000 7ff30e966640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1149c0 0x55787f114e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.018+0000 7ff30e966640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1149c0 0x55787f114e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45830/0 (socket says 172.21.15.70:45830) 2024-04-01T01:37:04.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.018+0000 7ff30e966640 1 -- 172.21.15.70:0/224231617 learned_addr learned my addr 172.21.15.70:0/224231617 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.018+0000 7ff30d964640 1 --2- 172.21.15.70:0/224231617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 0x55787f1139b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.018+0000 7ff30e165640 1 --2- 172.21.15.70:0/224231617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f105090 0x55787f113470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.019+0000 7ff30e165640 1 -- 172.21.15.70:0/224231617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1149c0 msgr2=0x55787f114e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.019+0000 7ff30e165640 1 --2- 172.21.15.70:0/224231617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1149c0 0x55787f114e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.019+0000 7ff30e165640 1 -- 172.21.15.70:0/224231617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 msgr2=0x55787f1139b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.019+0000 7ff30e165640 1 --2- 172.21.15.70:0/224231617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 0x55787f1139b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.019+0000 7ff30e165640 1 -- 172.21.15.70:0/224231617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55787f115380 con 0x55787f105090 2024-04-01T01:37:04.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.019+0000 7ff30e165640 1 --2- 172.21.15.70:0/224231617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f105090 0x55787f113470 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x55787f118dd0 tx=0x7ff3040056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f19a7ffce62b4018 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.020+0000 7ff2f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:37:04.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.020+0000 7ff2f1ffb640 1 -- 172.21.15.70:0/224231617 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30401f430 con 0x55787f105090 2024-04-01T01:37:04.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.020+0000 7ff312faf740 1 -- 172.21.15.70:0/224231617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff304010430 con 0x55787f105090 2024-04-01T01:37:04.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.020+0000 7ff2f1ffb640 1 -- 172.21.15.70:0/224231617 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff30401f5d0 con 0x55787f105090 2024-04-01T01:37:04.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.020+0000 7ff2f1ffb640 1 -- 172.21.15.70:0/224231617 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff30403c050 con 0x55787f105090 2024-04-01T01:37:04.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.020+0000 7ff312faf740 10 client.16659 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:04.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.020+0000 7ff312faf740 1 -- 172.21.15.70:0/224231617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55787f180e50 con 0x55787f105090 2024-04-01T01:37:04.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.021+0000 7ff2f1ffb640 1 -- 172.21.15.70:0/224231617 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff304010430 con 0x55787f105090 2024-04-01T01:37:04.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.021+0000 7ff2f1ffb640 1 -- 172.21.15.70:0/224231617 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff304031020 con 0x55787f105090 2024-04-01T01:37:04.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.022+0000 7ff312faf740 10 client.16659 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:04.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.022+0000 7ff312faf740 10 client.16659 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:37:04.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.022+0000 7ff312faf740 20 client.16659 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:04.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.022+0000 7ff312faf740 1 --2- 172.21.15.70:0/224231617 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55787f181630 0x55787f183a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.022+0000 7ff312faf740 4 client.16659 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:37:04.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.022+0000 7ff312faf740 1 -- 172.21.15.70:0/224231617 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55787f070a40 con 0x55787f181630 2024-04-01T01:37:04.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.022+0000 7ff30d964640 1 --2- 172.21.15.70:0/224231617 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55787f181630 0x55787f183a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.023+0000 7ff30d964640 1 --2- 172.21.15.70:0/224231617 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55787f181630 0x55787f183a50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d5d633646c3ab462 server_cookie=5ec0b543807757cd in_seq=0 out_seq=0 2024-04-01T01:37:04.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.024+0000 7ff2f1ffb640 10 client.16659 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:37:04.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.024+0000 7ff2f1ffb640 1 -- 172.21.15.70:0/224231617 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55787f070a40 con 0x55787f181630 2024-04-01T01:37:04.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.024+0000 7ff2f1ffb640 10 client.16659 handle_command_reply: tid=0 2024-04-01T01:37:04.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.024+0000 7ff312faf740 1 client.16659 shutdown 2024-04-01T01:37:04.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.028+0000 7ff312faf740 1 -- 172.21.15.70:0/224231617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f105090 msgr2=0x55787f113470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.028+0000 7ff312faf740 1 --2- 172.21.15.70:0/224231617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f105090 0x55787f113470 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x55787f118dd0 tx=0x7ff3040056d0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.028+0000 7ff312faf740 1 -- 172.21.15.70:0/224231617 shutdown_connections 2024-04-01T01:37:04.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.028+0000 7ff312faf740 1 --2- 172.21.15.70:0/224231617 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55787f181630 0x55787f183a50 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.029+0000 7ff312faf740 1 --2- 172.21.15.70:0/224231617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55787f1149c0 0x55787f114e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.029+0000 7ff2f1ffb640 0 client.16659 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:37:04.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.029+0000 7ff312faf740 1 --2- 172.21.15.70:0/224231617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55787f10a200 0x55787f1139b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.029+0000 7ff312faf740 1 --2- 172.21.15.70:0/224231617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55787f105090 0x55787f113470 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.029+0000 7ff312faf740 1 -- 172.21.15.70:0/224231617 >> 172.21.15.70:0/224231617 conn(0x55787f0ffc10 msgr2=0x55787f102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.031+0000 7ff312faf740 1 -- 172.21.15.70:0/224231617 shutdown_connections 2024-04-01T01:37:04.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.032+0000 7ff312faf740 1 -- 172.21.15.70:0/224231617 wait complete. 2024-04-01T01:37:04.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.032+0000 7ff312faf740 20 client.16659 trim_cache size 0 max 16384 2024-04-01T01:37:04.032 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:37:04.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.034+0000 7ff30f167640 1 -- 172.21.15.70:0/2125837826 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2f40a2480 msgr2=0x7ff2f40a4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.034+0000 7ff30f167640 1 --2- 172.21.15.70:0/2125837826 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2f40a2480 0x7ff2f40a4940 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff308000b60 tx=0x7ff3080675f0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.034+0000 7ff30f167640 1 -- 172.21.15.70:0/2125837826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3000a6710 msgr2=0x7ff300143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.034+0000 7ff30f167640 1 --2- 172.21.15.70:0/2125837826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3000a6710 0x7ff300143ae0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8000c00 tx=0x7ff2f8005200 comp rx=0 tx=0).stop 2024-04-01T01:37:04.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.037+0000 7ff30f167640 1 -- 172.21.15.70:0/2125837826 shutdown_connections 2024-04-01T01:37:04.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.037+0000 7ff30f167640 1 --2- 172.21.15.70:0/2125837826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3000ab410 0x7ff30014ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.037+0000 7ff30f167640 1 --2- 172.21.15.70:0/2125837826 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2f40a2480 0x7ff2f40a4940 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.037+0000 7ff30f167640 1 --2- 172.21.15.70:0/2125837826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3000a6710 0x7ff300143ae0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.037+0000 7ff30f167640 1 --2- 172.21.15.70:0/2125837826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3000182a0 0x7ff3001435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.037+0000 7ff30f167640 1 -- 172.21.15.70:0/2125837826 >> 172.21.15.70:0/2125837826 conn(0x7ff3000a0530 msgr2=0x7ff3000a73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.040+0000 7ff30f167640 1 -- 172.21.15.70:0/2125837826 shutdown_connections 2024-04-01T01:37:04.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.040+0000 7ff30f167640 1 -- 172.21.15.70:0/2125837826 wait complete. 2024-04-01T01:37:04.065 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:37:04.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:03 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1505750126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:37:04.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:03 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1505750126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:37:04.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.290+0000 7f9a8ad8c640 1 -- 172.21.15.70:0/1215734435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a8406cc40 msgr2=0x7f9a8406b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.290+0000 7f9a8ad8c640 1 --2- 172.21.15.70:0/1215734435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a8406cc40 0x7f9a8406b090 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f9a74009900 tx=0x7f9a7402bc50 comp rx=0 tx=0).stop 2024-04-01T01:37:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.290+0000 7f9a8ad8c640 1 -- 172.21.15.70:0/1215734435 shutdown_connections 2024-04-01T01:37:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.290+0000 7f9a8ad8c640 1 --2- 172.21.15.70:0/1215734435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a84116560 0x7f9a84118950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.290+0000 7f9a8ad8c640 1 --2- 172.21.15.70:0/1215734435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8406b660 0x7f9a8406bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.290+0000 7f9a8ad8c640 1 --2- 172.21.15.70:0/1215734435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a8406cc40 0x7f9a8406b090 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.290+0000 7f9a8ad8c640 1 -- 172.21.15.70:0/1215734435 >> 172.21.15.70:0/1215734435 conn(0x7f9a840706f0 msgr2=0x7f9a84072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.293+0000 7f9a8ad8c640 1 -- 172.21.15.70:0/1215734435 shutdown_connections 2024-04-01T01:37:04.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.293+0000 7f9a8ad8c640 1 -- 172.21.15.70:0/1215734435 wait complete. 2024-04-01T01:37:04.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.294+0000 7f9a8ad8c640 1 Processor -- start 2024-04-01T01:37:04.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.295+0000 7f9a8ad8c640 1 -- start start 2024-04-01T01:37:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.295+0000 7f9a8ad8c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a8406b660 0x7f9a841a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.295+0000 7f9a8ad8c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8406cc40 0x7f9a841a6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.295+0000 7f9a8ad8c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a84116560 0x7f9a841ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.295+0000 7f9a88b28640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a8406b660 0x7f9a841a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.295+0000 7f9a88b28640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a8406b660 0x7f9a841a6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45836/0 (socket says 172.21.15.70:45836) 2024-04-01T01:37:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.296+0000 7f9a88b28640 1 -- 172.21.15.70:0/1945015580 learned_addr learned my addr 172.21.15.70:0/1945015580 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.296+0000 7f9a8ad8c640 1 -- 172.21.15.70:0/1945015580 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8411c160 con 0x7f9a84116560 2024-04-01T01:37:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.296+0000 7f9a8ad8c640 1 -- 172.21.15.70:0/1945015580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8411bfe0 con 0x7f9a8406cc40 2024-04-01T01:37:04.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.296+0000 7f9a83fff640 1 --2- 172.21.15.70:0/1945015580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8406cc40 0x7f9a841a6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.296+0000 7f9a8ad8c640 1 -- 172.21.15.70:0/1945015580 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8411c2e0 con 0x7f9a8406b660 2024-04-01T01:37:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.296+0000 7f9a83fff640 1 -- 172.21.15.70:0/1945015580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a8406b660 msgr2=0x7f9a841a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.296+0000 7f9a83fff640 1 --2- 172.21.15.70:0/1945015580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a8406b660 0x7f9a841a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.296+0000 7f9a83fff640 1 -- 172.21.15.70:0/1945015580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a84116560 msgr2=0x7f9a841ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.296+0000 7f9a83fff640 1 --2- 172.21.15.70:0/1945015580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a84116560 0x7f9a841ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.296+0000 7f9a83fff640 1 -- 172.21.15.70:0/1945015580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a841ae4a0 con 0x7f9a8406cc40 2024-04-01T01:37:04.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.297+0000 7f9a83fff640 1 --2- 172.21.15.70:0/1945015580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8406cc40 0x7f9a841a6dc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009fd0 tx=0x7f9a6c00b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.298+0000 7f9a81ffb640 1 -- 172.21.15.70:0/1945015580 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c015020 con 0x7f9a8406cc40 2024-04-01T01:37:04.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.298+0000 7f9a81ffb640 1 -- 172.21.15.70:0/1945015580 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a6c011ce0 con 0x7f9a8406cc40 2024-04-01T01:37:04.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.298+0000 7f9a81ffb640 1 -- 172.21.15.70:0/1945015580 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c010de0 con 0x7f9a8406cc40 2024-04-01T01:37:04.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.298+0000 7f9a8ad8c640 1 -- 172.21.15.70:0/1945015580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a841ae790 con 0x7f9a8406cc40 2024-04-01T01:37:04.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.298+0000 7f9a8ad8c640 1 -- 172.21.15.70:0/1945015580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a841aed10 con 0x7f9a8406cc40 2024-04-01T01:37:04.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.302+0000 7f9a81ffb640 1 -- 172.21.15.70:0/1945015580 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f9a6c018020 con 0x7f9a8406cc40 2024-04-01T01:37:04.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.302+0000 7f9a8d971740 1 Processor -- start 2024-04-01T01:37:04.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.302+0000 7f9a8d971740 1 -- start start 2024-04-01T01:37:04.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.302+0000 7f9a8d971740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10eb490 0x5636d10eb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.302+0000 7f9a8d971740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10ebe80 0x5636d10f0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.302+0000 7f9a8d971740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 0x5636d10f3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.302+0000 7f9a8d971740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5636d10501b0 con 0x5636d10ebe80 2024-04-01T01:37:04.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.302+0000 7f9a8d971740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5636d0ff8c00 con 0x5636d10eb490 2024-04-01T01:37:04.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.302+0000 7f9a8d971740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5636d104c250 con 0x5636d10f0ff0 2024-04-01T01:37:04.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.303+0000 7f9a83fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 0x5636d10f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.303+0000 7f9a83fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 0x5636d10f3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45846/0 (socket says 172.21.15.70:45846) 2024-04-01T01:37:04.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.303+0000 7f9a83fff640 1 -- 172.21.15.70:0/353503191 learned_addr learned my addr 172.21.15.70:0/353503191 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.303+0000 7f9a88b28640 1 --2- 172.21.15.70:0/353503191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10ebe80 0x5636d10f0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.303+0000 7f9a89329640 1 --2- 172.21.15.70:0/353503191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10eb490 0x5636d10eb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.303+0000 7f9a83fff640 1 -- 172.21.15.70:0/353503191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10eb490 msgr2=0x5636d10eb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.303+0000 7f9a83fff640 1 --2- 172.21.15.70:0/353503191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10eb490 0x5636d10eb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.303+0000 7f9a83fff640 1 -- 172.21.15.70:0/353503191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10ebe80 msgr2=0x5636d10f0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.303+0000 7f9a83fff640 1 --2- 172.21.15.70:0/353503191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10ebe80 0x5636d10f0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.303+0000 7f9a83fff640 1 -- 172.21.15.70:0/353503191 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636d10183d0 con 0x5636d10f0ff0 2024-04-01T01:37:04.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.303+0000 7f9a88b28640 1 --2- 172.21.15.70:0/353503191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10ebe80 0x5636d10f0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:37:04.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.303+0000 7f9a83fff640 1 --2- 172.21.15.70:0/353503191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 0x5636d10f3480 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0d64b0 tx=0x7f9a6c0d6970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6432944ed8bfb46 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.304+0000 7f9a7affd640 1 -- 172.21.15.70:0/353503191 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c0e1070 con 0x5636d10f0ff0 2024-04-01T01:37:04.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.304+0000 7f9a7affd640 1 -- 172.21.15.70:0/353503191 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a6c0ceb10 con 0x5636d10f0ff0 2024-04-01T01:37:04.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.305+0000 7f9a81ffb640 1 --2- 172.21.15.70:0/1945015580 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9a700a26a0 0x7f9a700a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.305+0000 7f9a7affd640 1 -- 172.21.15.70:0/353503191 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c0dcac0 con 0x5636d10f0ff0 2024-04-01T01:37:04.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.305+0000 7f9a8d971740 1 -- 172.21.15.70:0/353503191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 msgr2=0x5636d10f3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.305+0000 7f9a8d971740 1 --2- 172.21.15.70:0/353503191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 0x5636d10f3480 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0d64b0 tx=0x7f9a6c0d6970 comp rx=0 tx=0).stop 2024-04-01T01:37:04.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.305+0000 7f9a81ffb640 1 -- 172.21.15.70:0/1945015580 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9a6c0ce2b0 con 0x7f9a8406cc40 2024-04-01T01:37:04.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.306+0000 7f9a8d971740 1 -- 172.21.15.70:0/353503191 shutdown_connections 2024-04-01T01:37:04.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.306+0000 7f9a8d971740 1 --2- 172.21.15.70:0/353503191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 0x5636d10f3480 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.306+0000 7f9a8d971740 1 --2- 172.21.15.70:0/353503191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10ebe80 0x5636d10f0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.306+0000 7f9a8d971740 1 --2- 172.21.15.70:0/353503191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10eb490 0x5636d10eb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.306+0000 7f9a8d971740 1 -- 172.21.15.70:0/353503191 >> 172.21.15.70:0/353503191 conn(0x5636d10e6a00 msgr2=0x5636d10e6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.306+0000 7f9a8d971740 1 -- 172.21.15.70:0/353503191 shutdown_connections 2024-04-01T01:37:04.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.306+0000 7f9a89329640 1 --2- 172.21.15.70:0/1945015580 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9a700a26a0 0x7f9a700a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.306+0000 7f9a8d971740 1 -- 172.21.15.70:0/353503191 wait complete. 2024-04-01T01:37:04.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.306+0000 7f9a8d971740 1 -- start start 2024-04-01T01:37:04.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.307+0000 7f9a8d971740 1 Processor -- start 2024-04-01T01:37:04.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a8d971740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10eb490 0x5636d11068d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a8d971740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10ebe80 0x5636d1106e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a8d971740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 0x5636d110de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a8d971740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5636d1104e30 con 0x5636d10ebe80 2024-04-01T01:37:04.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a8d971740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5636d0fee470 con 0x5636d10eb490 2024-04-01T01:37:04.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a8d971740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5636d1104fb0 con 0x5636d10f0ff0 2024-04-01T01:37:04.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a8d971740 10 client.0 _ll_register_callbacks cb 0x5636d100bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:04.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a89329640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 0x5636d110de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a89329640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 0x5636d110de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45856/0 (socket says 172.21.15.70:45856) 2024-04-01T01:37:04.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a89329640 1 -- 172.21.15.70:0/1588022547 learned_addr learned my addr 172.21.15.70:0/1588022547 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a83fff640 1 --2- 172.21.15.70:0/1588022547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10ebe80 0x5636d1106e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a89329640 1 --2- 172.21.15.70:0/1945015580 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9a700a26a0 0x7f9a700a4b60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x5636d10ec6e0 tx=0x7f9a7c009aa0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a89329640 1 -- 172.21.15.70:0/1588022547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10eb490 msgr2=0x5636d11068d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:37:04.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a89329640 1 --2- 172.21.15.70:0/1588022547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10eb490 0x5636d11068d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a89329640 1 -- 172.21.15.70:0/1588022547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10ebe80 msgr2=0x5636d1106e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a89329640 1 --2- 172.21.15.70:0/1588022547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10ebe80 0x5636d1106e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a89329640 1 -- 172.21.15.70:0/1588022547 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636d0d643f0 con 0x5636d10f0ff0 2024-04-01T01:37:04.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.308+0000 7f9a83fff640 1 --2- 172.21.15.70:0/1588022547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10ebe80 0x5636d1106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:37:04.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.309+0000 7f9a89329640 1 --2- 172.21.15.70:0/1588022547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 0x5636d110de50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c011c50 tx=0x7f9a7c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77fa63cbcf740048 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.309+0000 7f9a63fff640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:37:04.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.310+0000 7f9a63fff640 1 -- 172.21.15.70:0/1588022547 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c003b70 con 0x5636d10f0ff0 2024-04-01T01:37:04.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.310+0000 7f9a8d971740 1 -- 172.21.15.70:0/1588022547 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a6c0ceb10 con 0x5636d10f0ff0 2024-04-01T01:37:04.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.310+0000 7f9a63fff640 1 -- 172.21.15.70:0/1588022547 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9a7c003d10 con 0x5636d10f0ff0 2024-04-01T01:37:04.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.310+0000 7f9a63fff640 1 -- 172.21.15.70:0/1588022547 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9a7c02d050 con 0x5636d10f0ff0 2024-04-01T01:37:04.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.310+0000 7f9a8d971740 10 client.26422 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:04.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.311+0000 7f9a8d971740 1 -- 172.21.15.70:0/1588022547 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5636d0e2e7d0 con 0x5636d10f0ff0 2024-04-01T01:37:04.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.311+0000 7f9a63fff640 1 -- 172.21.15.70:0/1588022547 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a6c0ceb10 con 0x5636d10f0ff0 2024-04-01T01:37:04.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.312+0000 7f9a63fff640 1 -- 172.21.15.70:0/1588022547 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9a7c01e540 con 0x5636d10f0ff0 2024-04-01T01:37:04.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.312+0000 7f9a8d971740 10 client.26422 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:04.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.312+0000 7f9a8d971740 10 client.26422 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:37:04.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.312+0000 7f9a8d971740 20 client.26422 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:04.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.312+0000 7f9a8d971740 1 --2- 172.21.15.70:0/1588022547 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5636d1112820 0x5636d1114ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.312+0000 7f9a8d971740 4 client.26422 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:04.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.312+0000 7f9a8d971740 1 -- 172.21.15.70:0/1588022547 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636d105d6f0 con 0x5636d1112820 2024-04-01T01:37:04.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.312+0000 7f9a88b28640 1 --2- 172.21.15.70:0/1588022547 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5636d1112820 0x5636d1114ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.313+0000 7f9a88b28640 1 --2- 172.21.15.70:0/1588022547 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5636d1112820 0x5636d1114ce0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9df5188794691ce2 server_cookie=98dd4c2fedf6fe38 in_seq=0 out_seq=0 2024-04-01T01:37:04.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.313+0000 7f9a63fff640 10 client.26422 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:37:04.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.314+0000 7f9a63fff640 1 -- 172.21.15.70:0/1588022547 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5636d105d6f0 con 0x5636d1112820 2024-04-01T01:37:04.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.314+0000 7f9a63fff640 10 client.26422 handle_command_reply: tid=0 2024-04-01T01:37:04.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.314+0000 7f9a8d971740 1 client.26422 shutdown 2024-04-01T01:37:04.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.315+0000 7f9a8d971740 1 -- 172.21.15.70:0/1588022547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 msgr2=0x5636d110de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.315+0000 7f9a8d971740 1 --2- 172.21.15.70:0/1588022547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 0x5636d110de50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c011c50 tx=0x7f9a7c004a40 comp rx=0 tx=0).stop 2024-04-01T01:37:04.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.315+0000 7f9a8d971740 1 -- 172.21.15.70:0/1588022547 shutdown_connections 2024-04-01T01:37:04.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.315+0000 7f9a8d971740 1 --2- 172.21.15.70:0/1588022547 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5636d1112820 0x5636d1114ce0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.315+0000 7f9a8d971740 1 --2- 172.21.15.70:0/1588022547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 0x5636d110de50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.315+0000 7f9a63fff640 0 client.26422 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:37:04.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.315+0000 7f9a8d971740 1 --2- 172.21.15.70:0/1588022547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10ebe80 0x5636d1106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.315+0000 7f9a8d971740 1 --2- 172.21.15.70:0/1588022547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10eb490 0x5636d11068d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.315+0000 7f9a8d971740 1 -- 172.21.15.70:0/1588022547 >> 172.21.15.70:0/1588022547 conn(0x5636d10e6a00 msgr2=0x5636d10e5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.316+0000 7f9a8d971740 1 -- 172.21.15.70:0/1588022547 shutdown_connections 2024-04-01T01:37:04.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.316+0000 7f9a8d971740 1 -- 172.21.15.70:0/1588022547 wait complete. 2024-04-01T01:37:04.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.316+0000 7f9a8d971740 20 client.26422 trim_cache size 0 max 16384 2024-04-01T01:37:04.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.344+0000 7f9a8d971740 1 Processor -- start 2024-04-01T01:37:04.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.345+0000 7f9a8d971740 1 -- start start 2024-04-01T01:37:04.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.345+0000 7f9a8d971740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10eb490 0x5636d10ed320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.345+0000 7f9a8d971740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10ebe80 0x5636d10ed860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.345+0000 7f9a8d971740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10f0ff0 0x5636d110de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.345+0000 7f9a8d971740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9a841b0c90 con 0x5636d10f0ff0 2024-04-01T01:37:04.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.346+0000 7f9a8d971740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9a841b27d0 con 0x5636d10eb490 2024-04-01T01:37:04.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.346+0000 7f9a8d971740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9a841b1360 con 0x5636d10ebe80 2024-04-01T01:37:04.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.346+0000 7f9a88b28640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10eb490 0x5636d10ed320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.346+0000 7f9a88b28640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10eb490 0x5636d10ed320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:50598/0 (socket says 172.21.15.70:50598) 2024-04-01T01:37:04.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.346+0000 7f9a88b28640 1 -- 172.21.15.70:0/691313851 learned_addr learned my addr 172.21.15.70:0/691313851 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.346+0000 7f9a83fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10ebe80 0x5636d10ed860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.346+0000 7f9a89329640 1 --2- 172.21.15.70:0/691313851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10f0ff0 0x5636d110de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.346+0000 7f9a88b28640 1 -- 172.21.15.70:0/691313851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10ebe80 msgr2=0x5636d10ed860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.346+0000 7f9a88b28640 1 --2- 172.21.15.70:0/691313851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10ebe80 0x5636d10ed860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.346+0000 7f9a88b28640 1 -- 172.21.15.70:0/691313851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10f0ff0 msgr2=0x5636d110de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.346+0000 7f9a88b28640 1 --2- 172.21.15.70:0/691313851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10f0ff0 0x5636d110de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.346+0000 7f9a88b28640 1 -- 172.21.15.70:0/691313851 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636d10ec260 con 0x5636d10eb490 2024-04-01T01:37:04.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.346+0000 7f9a88b28640 1 --2- 172.21.15.70:0/691313851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10eb490 0x5636d10ed320 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9a7402c260 tx=0x7f9a7402bc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14101339867c515a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.347+0000 7f9a7a7fc640 1 -- 172.21.15.70:0/691313851 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7402cbf0 con 0x5636d10eb490 2024-04-01T01:37:04.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.347+0000 7f9a7a7fc640 1 -- 172.21.15.70:0/691313851 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a74004040 con 0x5636d10eb490 2024-04-01T01:37:04.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.347+0000 7f9a7a7fc640 1 -- 172.21.15.70:0/691313851 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74005030 con 0x5636d10eb490 2024-04-01T01:37:04.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.348+0000 7f9a8d971740 1 -- 172.21.15.70:0/691313851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10eb490 msgr2=0x5636d10ed320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.348+0000 7f9a8d971740 1 --2- 172.21.15.70:0/691313851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10eb490 0x5636d10ed320 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9a7402c260 tx=0x7f9a7402bc50 comp rx=0 tx=0).stop 2024-04-01T01:37:04.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.348+0000 7f9a8d971740 1 -- 172.21.15.70:0/691313851 shutdown_connections 2024-04-01T01:37:04.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.348+0000 7f9a8d971740 1 --2- 172.21.15.70:0/691313851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d10f0ff0 0x5636d110de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.348+0000 7f9a8d971740 1 --2- 172.21.15.70:0/691313851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10ebe80 0x5636d10ed860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.348+0000 7f9a8d971740 1 --2- 172.21.15.70:0/691313851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10eb490 0x5636d10ed320 secure :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9a7402c260 tx=0x7f9a7402bc50 comp rx=0 tx=0).stop 2024-04-01T01:37:04.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.348+0000 7f9a8d971740 1 -- 172.21.15.70:0/691313851 >> 172.21.15.70:0/691313851 conn(0x5636d10e6a00 msgr2=0x5636d10e6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.349+0000 7f9a8d971740 1 -- 172.21.15.70:0/691313851 shutdown_connections 2024-04-01T01:37:04.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.349+0000 7f9a8d971740 1 -- 172.21.15.70:0/691313851 wait complete. 2024-04-01T01:37:04.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.350+0000 7f9a8d971740 1 -- start start 2024-04-01T01:37:04.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.350+0000 7f9a8d971740 1 Processor -- start 2024-04-01T01:37:04.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.350+0000 7f9a8d971740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10ebe80 0x5636d110b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.350+0000 7f9a8d971740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 0x5636d110bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.350+0000 7f9a8d971740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d1112820 0x5636d110cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.351+0000 7f9a8d971740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9a841b2100 con 0x5636d1112820 2024-04-01T01:37:04.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.351+0000 7f9a8d971740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9a841b1a30 con 0x5636d10ebe80 2024-04-01T01:37:04.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.351+0000 7f9a8d971740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9a84116190 con 0x5636d10f0ff0 2024-04-01T01:37:04.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.351+0000 7f9a8d971740 10 client.0 _ll_register_callbacks cb 0x7f9a7c0118d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:04.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.351+0000 7f9a89329640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d1112820 0x5636d110cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.351+0000 7f9a89329640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d1112820 0x5636d110cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56766/0 (socket says 172.21.15.70:56766) 2024-04-01T01:37:04.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.351+0000 7f9a89329640 1 -- 172.21.15.70:0/546245977 learned_addr learned my addr 172.21.15.70:0/546245977 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.351+0000 7f9a88b28640 1 --2- 172.21.15.70:0/546245977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10ebe80 0x5636d110b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.351+0000 7f9a83fff640 1 --2- 172.21.15.70:0/546245977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 0x5636d110bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.352+0000 7f9a88b28640 1 -- 172.21.15.70:0/546245977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 msgr2=0x5636d110bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.352+0000 7f9a88b28640 1 --2- 172.21.15.70:0/546245977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 0x5636d110bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.352+0000 7f9a88b28640 1 -- 172.21.15.70:0/546245977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d1112820 msgr2=0x5636d110cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.352+0000 7f9a88b28640 1 --2- 172.21.15.70:0/546245977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d1112820 0x5636d110cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.352+0000 7f9a88b28640 1 -- 172.21.15.70:0/546245977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636d110d4b0 con 0x5636d10ebe80 2024-04-01T01:37:04.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.352+0000 7f9a89329640 1 --2- 172.21.15.70:0/546245977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d1112820 0x5636d110cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:04.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.352+0000 7f9a88b28640 1 --2- 172.21.15.70:0/546245977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10ebe80 0x5636d110b9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x5636d10f38a0 tx=0x7f9a74005cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7520d0c72653eee9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.352+0000 7f9a63fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:04.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.352+0000 7f9a63fff640 1 -- 172.21.15.70:0/546245977 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74055070 con 0x5636d10ebe80 2024-04-01T01:37:04.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.352+0000 7f9a63fff640 1 -- 172.21.15.70:0/546245977 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9a74005040 con 0x5636d10ebe80 2024-04-01T01:37:04.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.352+0000 7f9a8d971740 1 -- 172.21.15.70:0/546245977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a74004040 con 0x5636d10ebe80 2024-04-01T01:37:04.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.354+0000 7f9a63fff640 1 -- 172.21.15.70:0/546245977 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9a7405f050 con 0x5636d10ebe80 2024-04-01T01:37:04.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.354+0000 7f9a8d971740 10 client.26528 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:04.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.354+0000 7f9a8d971740 1 -- 172.21.15.70:0/546245977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5636d11790b0 con 0x5636d10ebe80 2024-04-01T01:37:04.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.354+0000 7f9a63fff640 1 -- 172.21.15.70:0/546245977 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a74004040 con 0x5636d10ebe80 2024-04-01T01:37:04.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.354+0000 7f9a63fff640 1 -- 172.21.15.70:0/546245977 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9a740035f0 con 0x5636d10ebe80 2024-04-01T01:37:04.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.355+0000 7f9a8d971740 10 client.26528 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:04.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.355+0000 7f9a8d971740 10 client.26528 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:37:04.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.355+0000 7f9a8d971740 20 client.26528 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:04.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.355+0000 7f9a8d971740 1 --2- 172.21.15.70:0/546245977 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5636d1179960 0x5636d117bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.355+0000 7f9a8d971740 4 client.26528 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:37:04.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.355+0000 7f9a8d971740 1 -- 172.21.15.70:0/546245977 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5636d1057a40 con 0x5636d1179960 2024-04-01T01:37:04.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.355+0000 7f9a83fff640 1 --2- 172.21.15.70:0/546245977 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5636d1179960 0x5636d117bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.356+0000 7f9a83fff640 1 --2- 172.21.15.70:0/546245977 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5636d1179960 0x5636d117bd80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ad4a943f416dd627 server_cookie=3f1a059c8e65ba69 in_seq=0 out_seq=0 2024-04-01T01:37:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.356+0000 7f9a63fff640 10 client.26528 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:37:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.356+0000 7f9a63fff640 1 -- 172.21.15.70:0/546245977 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5636d1057a40 con 0x5636d1179960 2024-04-01T01:37:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.356+0000 7f9a63fff640 10 client.26528 handle_command_reply: tid=0 2024-04-01T01:37:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.356+0000 7f9a8d971740 1 client.26528 shutdown 2024-04-01T01:37:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.356+0000 7f9a8d971740 1 -- 172.21.15.70:0/546245977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10ebe80 msgr2=0x5636d110b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.357+0000 7f9a8d971740 1 --2- 172.21.15.70:0/546245977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10ebe80 0x5636d110b9c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x5636d10f38a0 tx=0x7f9a74005cd0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.357+0000 7f9a8d971740 1 -- 172.21.15.70:0/546245977 shutdown_connections 2024-04-01T01:37:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.357+0000 7f9a8d971740 1 --2- 172.21.15.70:0/546245977 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5636d1179960 0x5636d117bd80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.357+0000 7f9a8d971740 1 --2- 172.21.15.70:0/546245977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636d1112820 0x5636d110cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.357+0000 7f9a63fff640 0 client.26528 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:37:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.357+0000 7f9a8d971740 1 --2- 172.21.15.70:0/546245977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5636d10f0ff0 0x5636d110bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.357+0000 7f9a8d971740 1 --2- 172.21.15.70:0/546245977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636d10ebe80 0x5636d110b9c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.357+0000 7f9a8d971740 1 -- 172.21.15.70:0/546245977 >> 172.21.15.70:0/546245977 conn(0x5636d10e6a00 msgr2=0x5636d11067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.357+0000 7f9a8d971740 1 -- 172.21.15.70:0/546245977 shutdown_connections 2024-04-01T01:37:04.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.357+0000 7f9a8d971740 1 -- 172.21.15.70:0/546245977 wait complete. 2024-04-01T01:37:04.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.358+0000 7f9a8d971740 20 client.26528 trim_cache size 0 max 16384 2024-04-01T01:37:04.358 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:37:04.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.359+0000 7f9a8ad8c640 1 -- 172.21.15.70:0/1945015580 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9a700a26a0 msgr2=0x7f9a700a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.359+0000 7f9a8ad8c640 1 --2- 172.21.15.70:0/1945015580 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9a700a26a0 0x7f9a700a4b60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x5636d10ec6e0 tx=0x7f9a7c009aa0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.359+0000 7f9a8ad8c640 1 -- 172.21.15.70:0/1945015580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8406cc40 msgr2=0x7f9a841a6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.359+0000 7f9a8ad8c640 1 --2- 172.21.15.70:0/1945015580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8406cc40 0x7f9a841a6dc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c009fd0 tx=0x7f9a6c00b870 comp rx=0 tx=0).stop 2024-04-01T01:37:04.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.360+0000 7f9a8ad8c640 1 -- 172.21.15.70:0/1945015580 shutdown_connections 2024-04-01T01:37:04.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.360+0000 7f9a8ad8c640 1 --2- 172.21.15.70:0/1945015580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9a84116560 0x7f9a841ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.360+0000 7f9a8ad8c640 1 --2- 172.21.15.70:0/1945015580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a8406cc40 0x7f9a841a6dc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.360+0000 7f9a8ad8c640 1 --2- 172.21.15.70:0/1945015580 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9a700a26a0 0x7f9a700a4b60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.360+0000 7f9a8ad8c640 1 --2- 172.21.15.70:0/1945015580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a8406b660 0x7f9a841a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.360+0000 7f9a8ad8c640 1 -- 172.21.15.70:0/1945015580 >> 172.21.15.70:0/1945015580 conn(0x7f9a840706f0 msgr2=0x7f9a84109940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.362+0000 7f9a8ad8c640 1 -- 172.21.15.70:0/1945015580 shutdown_connections 2024-04-01T01:37:04.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.362+0000 7f9a8ad8c640 1 -- 172.21.15.70:0/1945015580 wait complete. 2024-04-01T01:37:04.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:37:04.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.567+0000 7fcb6a313640 1 -- 172.21.15.70:0/3977682828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb640ff880 msgr2=0x7fcb641006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.567+0000 7fcb6a313640 1 --2- 172.21.15.70:0/3977682828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb640ff880 0x7fcb641006a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009900 tx=0x7fcb4c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:37:04.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.567+0000 7fcb6a313640 1 -- 172.21.15.70:0/3977682828 shutdown_connections 2024-04-01T01:37:04.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.567+0000 7fcb6a313640 1 --2- 172.21.15.70:0/3977682828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb64100fa0 0x7fcb64101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.567+0000 7fcb6a313640 1 --2- 172.21.15.70:0/3977682828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb640ff880 0x7fcb641006a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.567+0000 7fcb6a313640 1 --2- 172.21.15.70:0/3977682828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb640ff3d0 0x7fcb64100100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.567+0000 7fcb6a313640 1 -- 172.21.15.70:0/3977682828 >> 172.21.15.70:0/3977682828 conn(0x7fcb640faf10 msgr2=0x7fcb640fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.568+0000 7fcb6a313640 1 -- 172.21.15.70:0/3977682828 shutdown_connections 2024-04-01T01:37:04.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.568+0000 7fcb6a313640 1 -- 172.21.15.70:0/3977682828 wait complete. 2024-04-01T01:37:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.569+0000 7fcb6a313640 1 Processor -- start 2024-04-01T01:37:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.569+0000 7fcb6a313640 1 -- start start 2024-04-01T01:37:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.569+0000 7fcb6a313640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb640ff3d0 0x7fcb64195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.569+0000 7fcb6a313640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb640ff880 0x7fcb64195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.569+0000 7fcb6a313640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb64100fa0 0x7fcb6419cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.569+0000 7fcb6a313640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6410b0b0 con 0x7fcb640ff3d0 2024-04-01T01:37:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.569+0000 7fcb6a313640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6410af30 con 0x7fcb64100fa0 2024-04-01T01:37:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.569+0000 7fcb6a313640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6410b230 con 0x7fcb640ff880 2024-04-01T01:37:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.570+0000 7fcb637fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb640ff880 0x7fcb64195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.570+0000 7fcb637fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb640ff880 0x7fcb64195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45896/0 (socket says 172.21.15.70:45896) 2024-04-01T01:37:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.570+0000 7fcb637fe640 1 -- 172.21.15.70:0/3379656020 learned_addr learned my addr 172.21.15.70:0/3379656020 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.570+0000 7fcb688b0640 1 --2- 172.21.15.70:0/3379656020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb64100fa0 0x7fcb6419cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.571+0000 7fcb63fff640 1 --2- 172.21.15.70:0/3379656020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb640ff3d0 0x7fcb64195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.571+0000 7fcb688b0640 1 -- 172.21.15.70:0/3379656020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb640ff880 msgr2=0x7fcb64195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.571+0000 7fcb688b0640 1 --2- 172.21.15.70:0/3379656020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb640ff880 0x7fcb64195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.571+0000 7fcb688b0640 1 -- 172.21.15.70:0/3379656020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb640ff3d0 msgr2=0x7fcb64195870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.571+0000 7fcb688b0640 1 --2- 172.21.15.70:0/3379656020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb640ff3d0 0x7fcb64195870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.571+0000 7fcb688b0640 1 -- 172.21.15.70:0/3379656020 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6419d490 con 0x7fcb64100fa0 2024-04-01T01:37:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.571+0000 7fcb688b0640 1 --2- 172.21.15.70:0/3379656020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb64100fa0 0x7fcb6419cdf0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00d6b0 tx=0x7fcb5c00db70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.572+0000 7fcb617fa640 1 -- 172.21.15.70:0/3379656020 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c017070 con 0x7fcb64100fa0 2024-04-01T01:37:04.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.572+0000 7fcb617fa640 1 -- 172.21.15.70:0/3379656020 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb5c009cd0 con 0x7fcb64100fa0 2024-04-01T01:37:04.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.572+0000 7fcb617fa640 1 -- 172.21.15.70:0/3379656020 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c003cc0 con 0x7fcb64100fa0 2024-04-01T01:37:04.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.572+0000 7fcb6a313640 1 -- 172.21.15.70:0/3379656020 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb6419d780 con 0x7fcb64100fa0 2024-04-01T01:37:04.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.573+0000 7fcb6a313640 1 -- 172.21.15.70:0/3379656020 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb6419dc80 con 0x7fcb64100fa0 2024-04-01T01:37:04.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.577+0000 7fcb617fa640 1 -- 172.21.15.70:0/3379656020 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fcb5c010460 con 0x7fcb64100fa0 2024-04-01T01:37:04.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.577+0000 7fcb6cfd3740 1 Processor -- start 2024-04-01T01:37:04.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.577+0000 7fcb6cfd3740 1 -- start start 2024-04-01T01:37:04.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.577+0000 7fcb6cfd3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e124a0 0x55c1f8e128c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.577+0000 7fcb6cfd3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e12e90 0x55c1f8e17990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.577+0000 7fcb6cfd3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e18000 0x55c1f8e1a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.577+0000 7fcb6cfd3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1f8d661b0 con 0x55c1f8e18000 2024-04-01T01:37:04.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.577+0000 7fcb6cfd3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c1f8d0ec00 con 0x55c1f8e124a0 2024-04-01T01:37:04.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.577+0000 7fcb6cfd3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c1f8d62250 con 0x55c1f8e12e90 2024-04-01T01:37:04.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.577+0000 7fcb63fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e12e90 0x55c1f8e17990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.577+0000 7fcb63fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e12e90 0x55c1f8e17990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45910/0 (socket says 172.21.15.70:45910) 2024-04-01T01:37:04.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.577+0000 7fcb63fff640 1 -- 172.21.15.70:0/461495320 learned_addr learned my addr 172.21.15.70:0/461495320 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.578+0000 7fcb637fe640 1 --2- 172.21.15.70:0/461495320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e124a0 0x55c1f8e128c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.578+0000 7fcb617fa640 1 --2- 172.21.15.70:0/3379656020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcb500a2650 0x7fcb500a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.578+0000 7fcb617fa640 1 -- 172.21.15.70:0/3379656020 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fcb5c0ce160 con 0x7fcb64100fa0 2024-04-01T01:37:04.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.579+0000 7fcb688b0640 1 --2- 172.21.15.70:0/3379656020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcb500a2650 0x7fcb500a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.579+0000 7fcb637fe640 1 --2- 172.21.15.70:0/461495320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e18000 0x55c1f8e1a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.579+0000 7fcb63fff640 1 -- 172.21.15.70:0/461495320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e124a0 msgr2=0x55c1f8e128c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.579+0000 7fcb63fff640 1 --2- 172.21.15.70:0/461495320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e124a0 0x55c1f8e128c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.579+0000 7fcb63fff640 1 -- 172.21.15.70:0/461495320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e18000 msgr2=0x55c1f8e1a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.579+0000 7fcb63fff640 1 --2- 172.21.15.70:0/461495320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e18000 0x55c1f8e1a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.579+0000 7fcb63fff640 1 -- 172.21.15.70:0/461495320 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1f8d2e3d0 con 0x55c1f8e12e90 2024-04-01T01:37:04.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.580+0000 7fcb637fe640 1 --2- 172.21.15.70:0/461495320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e18000 0x55c1f8e1a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:37:04.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.580+0000 7fcb63fff640 1 --2- 172.21.15.70:0/461495320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e12e90 0x55c1f8e17990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400aed0 tx=0x7fcb540317a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9945b4627976f4bc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.580+0000 7fcb688b0640 1 --2- 172.21.15.70:0/3379656020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcb500a2650 0x7fcb500a4b10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00dfd0 tx=0x7fcb5c005b90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.580+0000 7fcb5a7fc640 1 -- 172.21.15.70:0/461495320 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb54032850 con 0x55c1f8e12e90 2024-04-01T01:37:04.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.581+0000 7fcb5a7fc640 1 -- 172.21.15.70:0/461495320 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb5403a3f0 con 0x55c1f8e12e90 2024-04-01T01:37:04.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.581+0000 7fcb5a7fc640 1 -- 172.21.15.70:0/461495320 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb54039600 con 0x55c1f8e12e90 2024-04-01T01:37:04.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.582+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/461495320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e12e90 msgr2=0x55c1f8e17990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.582+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/461495320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e12e90 0x55c1f8e17990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400aed0 tx=0x7fcb540317a0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.582+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/461495320 shutdown_connections 2024-04-01T01:37:04.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.583+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/461495320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e18000 0x55c1f8e1a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.583+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/461495320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e12e90 0x55c1f8e17990 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.583+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/461495320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e124a0 0x55c1f8e128c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.583+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/461495320 >> 172.21.15.70:0/461495320 conn(0x55c1f8dfca00 msgr2=0x55c1f8dfce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.583+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/461495320 shutdown_connections 2024-04-01T01:37:04.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.584+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/461495320 wait complete. 2024-04-01T01:37:04.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.584+0000 7fcb6cfd3740 1 -- start start 2024-04-01T01:37:04.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.584+0000 7fcb6cfd3740 1 Processor -- start 2024-04-01T01:37:04.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.585+0000 7fcb6cfd3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e124a0 0x55c1f8e02e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.585+0000 7fcb6cfd3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e12e90 0x55c1f8e033c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.585+0000 7fcb6cfd3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e18000 0x55c1f8e0a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.586+0000 7fcb6cfd3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1f8e01350 con 0x55c1f8e12e90 2024-04-01T01:37:04.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.586+0000 7fcb6cfd3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c1f8d04470 con 0x55c1f8e124a0 2024-04-01T01:37:04.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.586+0000 7fcb6cfd3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c1f8e014d0 con 0x55c1f8e18000 2024-04-01T01:37:04.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.586+0000 7fcb6cfd3740 10 client.0 _ll_register_callbacks cb 0x55c1f8d21cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:04.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.586+0000 7fcb637fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e124a0 0x55c1f8e02e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.586+0000 7fcb637fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e124a0 0x55c1f8e02e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:50630/0 (socket says 172.21.15.70:50630) 2024-04-01T01:37:04.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.587+0000 7fcb637fe640 1 -- 172.21.15.70:0/390999556 learned_addr learned my addr 172.21.15.70:0/390999556 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.587+0000 7fcb63fff640 1 --2- 172.21.15.70:0/390999556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e12e90 0x55c1f8e033c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.587+0000 7fcb637fe640 1 --2- 172.21.15.70:0/390999556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e18000 0x55c1f8e0a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.588+0000 7fcb637fe640 1 -- 172.21.15.70:0/390999556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e18000 msgr2=0x55c1f8e0a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.588+0000 7fcb637fe640 1 --2- 172.21.15.70:0/390999556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e18000 0x55c1f8e0a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.588+0000 7fcb637fe640 1 -- 172.21.15.70:0/390999556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e12e90 msgr2=0x55c1f8e033c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.588+0000 7fcb637fe640 1 --2- 172.21.15.70:0/390999556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e12e90 0x55c1f8e033c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.588+0000 7fcb637fe640 1 -- 172.21.15.70:0/390999556 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1f8a7a3f0 con 0x55c1f8e124a0 2024-04-01T01:37:04.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.589+0000 7fcb637fe640 1 --2- 172.21.15.70:0/390999556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e124a0 0x55c1f8e02e80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c000c00 tx=0x7fcb4c02c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce6f68a5aec45494 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.589+0000 7fcb437fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:04.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.589+0000 7fcb437fe640 1 -- 172.21.15.70:0/390999556 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c005480 con 0x55c1f8e124a0 2024-04-01T01:37:04.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.589+0000 7fcb437fe640 1 -- 172.21.15.70:0/390999556 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fcb4c0046c0 con 0x55c1f8e124a0 2024-04-01T01:37:04.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.589+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/390999556 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb5403a3f0 con 0x55c1f8e124a0 2024-04-01T01:37:04.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.590+0000 7fcb437fe640 1 -- 172.21.15.70:0/390999556 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcb4c054050 con 0x55c1f8e124a0 2024-04-01T01:37:04.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.590+0000 7fcb6cfd3740 10 client.26540 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:04.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.590+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/390999556 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c1f8b447d0 con 0x55c1f8e124a0 2024-04-01T01:37:04.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.590+0000 7fcb437fe640 1 -- 172.21.15.70:0/390999556 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb5403a3f0 con 0x55c1f8e124a0 2024-04-01T01:37:04.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.591+0000 7fcb437fe640 1 -- 172.21.15.70:0/390999556 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fcb4c003990 con 0x55c1f8e124a0 2024-04-01T01:37:04.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.592+0000 7fcb6cfd3740 10 client.26540 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:04.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.592+0000 7fcb6cfd3740 10 client.26540 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:37:04.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.592+0000 7fcb6cfd3740 20 client.26540 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:04.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.592+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/390999556 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55c1f8e0e830 0x55c1f8e10cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.592+0000 7fcb6cfd3740 4 client.26540 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:04.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.592+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/390999556 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1f8d736f0 con 0x55c1f8e0e830 2024-04-01T01:37:04.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.592+0000 7fcb63fff640 1 --2- 172.21.15.70:0/390999556 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55c1f8e0e830 0x55c1f8e10cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.594+0000 7fcb63fff640 1 --2- 172.21.15.70:0/390999556 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55c1f8e0e830 0x55c1f8e10cb0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6a04aa61888ad9ad server_cookie=7c1dd57f24e1bcea in_seq=0 out_seq=0 2024-04-01T01:37:04.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.594+0000 7fcb437fe640 10 client.26540 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:37:04.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.596+0000 7fcb437fe640 1 -- 172.21.15.70:0/390999556 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c1f8d736f0 con 0x55c1f8e0e830 2024-04-01T01:37:04.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.596+0000 7fcb437fe640 10 client.26540 handle_command_reply: tid=0 2024-04-01T01:37:04.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.597+0000 7fcb6cfd3740 1 client.26540 shutdown 2024-04-01T01:37:04.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.598+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/390999556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e124a0 msgr2=0x55c1f8e02e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.598+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/390999556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e124a0 0x55c1f8e02e80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c000c00 tx=0x7fcb4c02c8e0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.599+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/390999556 shutdown_connections 2024-04-01T01:37:04.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.599+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/390999556 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55c1f8e0e830 0x55c1f8e10cb0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.599+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/390999556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e18000 0x55c1f8e0a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.599+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/390999556 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e12e90 0x55c1f8e033c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.599+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/390999556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e124a0 0x55c1f8e02e80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.599+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/390999556 >> 172.21.15.70:0/390999556 conn(0x55c1f8dfca00 msgr2=0x55c1f8dfbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.599+0000 7fcb437fe640 0 client.26540 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:37:04.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.599+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/390999556 shutdown_connections 2024-04-01T01:37:04.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.599+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/390999556 wait complete. 2024-04-01T01:37:04.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.599+0000 7fcb6cfd3740 20 client.26540 trim_cache size 0 max 16384 2024-04-01T01:37:04.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.626+0000 7fcb6cfd3740 1 Processor -- start 2024-04-01T01:37:04.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.626+0000 7fcb6cfd3740 1 -- start start 2024-04-01T01:37:04.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.626+0000 7fcb6cfd3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e124a0 0x55c1f8e11550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.627+0000 7fcb6cfd3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e12e90 0x55c1f8e0a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.627+0000 7fcb6cfd3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e18000 0x55c1f8e11a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.627+0000 7fcb6cfd3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4400fd90 con 0x55c1f8e12e90 2024-04-01T01:37:04.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.627+0000 7fcb6cfd3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcb440118d0 con 0x55c1f8e18000 2024-04-01T01:37:04.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.627+0000 7fcb6cfd3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcb44010460 con 0x55c1f8e124a0 2024-04-01T01:37:04.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.627+0000 7fcb63fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e12e90 0x55c1f8e0a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.627+0000 7fcb63fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e12e90 0x55c1f8e0a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56790/0 (socket says 172.21.15.70:56790) 2024-04-01T01:37:04.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.627+0000 7fcb63fff640 1 -- 172.21.15.70:0/1431567896 learned_addr learned my addr 172.21.15.70:0/1431567896 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.627+0000 7fcb63fff640 1 -- 172.21.15.70:0/1431567896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e124a0 msgr2=0x55c1f8e11550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.627+0000 7fcb63fff640 1 --2- 172.21.15.70:0/1431567896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e124a0 0x55c1f8e11550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.627+0000 7fcb63fff640 1 -- 172.21.15.70:0/1431567896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e18000 msgr2=0x55c1f8e11a90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:37:04.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.627+0000 7fcb63fff640 1 --2- 172.21.15.70:0/1431567896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e18000 0x55c1f8e11a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.627+0000 7fcb63fff640 1 -- 172.21.15.70:0/1431567896 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1f8e17750 con 0x55c1f8e12e90 2024-04-01T01:37:04.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.628+0000 7fcb63fff640 1 --2- 172.21.15.70:0/1431567896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e12e90 0x55c1f8e0a3c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fcb54031ca0 tx=0x7fcb5403aab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524cf8d12d943a6e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.629+0000 7fcb59ffb640 1 -- 172.21.15.70:0/1431567896 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5400a590 con 0x55c1f8e12e90 2024-04-01T01:37:04.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.629+0000 7fcb59ffb640 1 -- 172.21.15.70:0/1431567896 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb5403aca0 con 0x55c1f8e12e90 2024-04-01T01:37:04.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.629+0000 7fcb59ffb640 1 -- 172.21.15.70:0/1431567896 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb54004a00 con 0x55c1f8e12e90 2024-04-01T01:37:04.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.630+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/1431567896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e12e90 msgr2=0x55c1f8e0a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.630+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/1431567896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e12e90 0x55c1f8e0a3c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fcb54031ca0 tx=0x7fcb5403aab0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.630+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/1431567896 shutdown_connections 2024-04-01T01:37:04.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.630+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/1431567896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e18000 0x55c1f8e11a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.630+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/1431567896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e12e90 0x55c1f8e0a3c0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.630+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/1431567896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e124a0 0x55c1f8e11550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.630+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/1431567896 >> 172.21.15.70:0/1431567896 conn(0x55c1f8dfca00 msgr2=0x55c1f8e144a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.631+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/1431567896 shutdown_connections 2024-04-01T01:37:04.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.631+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/1431567896 wait complete. 2024-04-01T01:37:04.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.631+0000 7fcb6cfd3740 1 -- start start 2024-04-01T01:37:04.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.631+0000 7fcb6cfd3740 1 Processor -- start 2024-04-01T01:37:04.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.632+0000 7fcb6cfd3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e124a0 0x55c1f8e0a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.632+0000 7fcb6cfd3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e12e90 0x55c1f8e06b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.632+0000 7fcb6cfd3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e18000 0x55c1f8e070c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.632+0000 7fcb6cfd3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb44011200 con 0x55c1f8e124a0 2024-04-01T01:37:04.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.632+0000 7fcb6cfd3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcb44010b30 con 0x55c1f8e18000 2024-04-01T01:37:04.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.632+0000 7fcb6cfd3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4400dd70 con 0x55c1f8e12e90 2024-04-01T01:37:04.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.632+0000 7fcb6cfd3740 10 client.0 _ll_register_callbacks cb 0x7fcb4c02cc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:04.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.632+0000 7fcb637fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e124a0 0x55c1f8e0a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.632+0000 7fcb63fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e12e90 0x55c1f8e06b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.632+0000 7fcb63fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e12e90 0x55c1f8e06b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45946/0 (socket says 172.21.15.70:45946) 2024-04-01T01:37:04.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.632+0000 7fcb63fff640 1 -- 172.21.15.70:0/339753736 learned_addr learned my addr 172.21.15.70:0/339753736 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.632+0000 7fcb637fe640 1 --2- 172.21.15.70:0/339753736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e18000 0x55c1f8e070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.633+0000 7fcb63fff640 1 -- 172.21.15.70:0/339753736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e18000 msgr2=0x55c1f8e070c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.633+0000 7fcb63fff640 1 --2- 172.21.15.70:0/339753736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e18000 0x55c1f8e070c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.633+0000 7fcb63fff640 1 -- 172.21.15.70:0/339753736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e124a0 msgr2=0x55c1f8e0a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.633+0000 7fcb63fff640 1 --2- 172.21.15.70:0/339753736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e124a0 0x55c1f8e0a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.633+0000 7fcb63fff640 1 -- 172.21.15.70:0/339753736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1f8e07680 con 0x55c1f8e12e90 2024-04-01T01:37:04.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.633+0000 7fcb63fff640 1 --2- 172.21.15.70:0/339753736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e12e90 0x55c1f8e06b80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400aea0 tx=0x7fcb5400a590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2fd51871df50eef server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.633+0000 7fcb437fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:37:04.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.634+0000 7fcb437fe640 1 -- 172.21.15.70:0/339753736 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb54004cd0 con 0x55c1f8e12e90 2024-04-01T01:37:04.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.634+0000 7fcb437fe640 1 -- 172.21.15.70:0/339753736 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fcb5404b040 con 0x55c1f8e12e90 2024-04-01T01:37:04.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.634+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/339753736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb5403aca0 con 0x55c1f8e12e90 2024-04-01T01:37:04.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.634+0000 7fcb437fe640 1 -- 172.21.15.70:0/339753736 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcb54003970 con 0x55c1f8e12e90 2024-04-01T01:37:04.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.634+0000 7fcb6cfd3740 10 client.26440 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:04.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.634+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/339753736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c1f8e7d3e0 con 0x55c1f8e12e90 2024-04-01T01:37:04.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.635+0000 7fcb437fe640 1 -- 172.21.15.70:0/339753736 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcb5403aca0 con 0x55c1f8e12e90 2024-04-01T01:37:04.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.635+0000 7fcb437fe640 1 -- 172.21.15.70:0/339753736 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fcb54032650 con 0x55c1f8e12e90 2024-04-01T01:37:04.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.635+0000 7fcb6cfd3740 10 client.26440 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:04.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.635+0000 7fcb6cfd3740 10 client.26440 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:37:04.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.635+0000 7fcb6cfd3740 20 client.26440 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:04.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.635+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/339753736 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55c1f8e7db80 0x55c1f8e7ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.635+0000 7fcb6cfd3740 4 client.26440 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:37:04.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.635+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/339753736 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c1f8d6da40 con 0x55c1f8e7db80 2024-04-01T01:37:04.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.636+0000 7fcb637fe640 1 --2- 172.21.15.70:0/339753736 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55c1f8e7db80 0x55c1f8e7ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.637+0000 7fcb637fe640 1 --2- 172.21.15.70:0/339753736 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55c1f8e7db80 0x55c1f8e7ffa0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=749f8f0f5c5608da server_cookie=3e73cd944fad2dc9 in_seq=0 out_seq=0 2024-04-01T01:37:04.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.637+0000 7fcb437fe640 10 client.26440 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:37:04.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.637+0000 7fcb437fe640 1 -- 172.21.15.70:0/339753736 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c1f8d6da40 con 0x55c1f8e7db80 2024-04-01T01:37:04.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.637+0000 7fcb437fe640 10 client.26440 handle_command_reply: tid=0 2024-04-01T01:37:04.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.638+0000 7fcb6cfd3740 1 client.26440 shutdown 2024-04-01T01:37:04.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.638+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/339753736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e12e90 msgr2=0x55c1f8e06b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.638+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/339753736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e12e90 0x55c1f8e06b80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400aea0 tx=0x7fcb5400a590 comp rx=0 tx=0).stop 2024-04-01T01:37:04.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.638+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/339753736 shutdown_connections 2024-04-01T01:37:04.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.639+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/339753736 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55c1f8e7db80 0x55c1f8e7ffa0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.639+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/339753736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c1f8e18000 0x55c1f8e070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.639+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/339753736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c1f8e12e90 0x55c1f8e06b80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.639+0000 7fcb437fe640 0 client.26440 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:37:04.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.639+0000 7fcb6cfd3740 1 --2- 172.21.15.70:0/339753736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f8e124a0 0x55c1f8e0a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.639+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/339753736 >> 172.21.15.70:0/339753736 conn(0x55c1f8dfca00 msgr2=0x55c1f8e114f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.640+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/339753736 shutdown_connections 2024-04-01T01:37:04.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.641+0000 7fcb6cfd3740 1 -- 172.21.15.70:0/339753736 wait complete. 2024-04-01T01:37:04.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.641+0000 7fcb6cfd3740 20 client.26440 trim_cache size 0 max 16384 2024-04-01T01:37:04.641 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:37:04.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.642+0000 7fcb6a313640 1 -- 172.21.15.70:0/3379656020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcb500a2650 msgr2=0x7fcb500a4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.642+0000 7fcb6a313640 1 --2- 172.21.15.70:0/3379656020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcb500a2650 0x7fcb500a4b10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00dfd0 tx=0x7fcb5c005b90 comp rx=0 tx=0).stop 2024-04-01T01:37:04.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.643+0000 7fcb6a313640 1 -- 172.21.15.70:0/3379656020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb64100fa0 msgr2=0x7fcb6419cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.643+0000 7fcb6a313640 1 --2- 172.21.15.70:0/3379656020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb64100fa0 0x7fcb6419cdf0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00d6b0 tx=0x7fcb5c00db70 comp rx=0 tx=0).stop 2024-04-01T01:37:04.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.643+0000 7fcb6a313640 1 -- 172.21.15.70:0/3379656020 shutdown_connections 2024-04-01T01:37:04.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.643+0000 7fcb6a313640 1 --2- 172.21.15.70:0/3379656020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb64100fa0 0x7fcb6419cdf0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.643+0000 7fcb6a313640 1 --2- 172.21.15.70:0/3379656020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcb500a2650 0x7fcb500a4b10 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.643+0000 7fcb6a313640 1 --2- 172.21.15.70:0/3379656020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb640ff880 0x7fcb64195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.643+0000 7fcb6a313640 1 --2- 172.21.15.70:0/3379656020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb640ff3d0 0x7fcb64195870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.643+0000 7fcb6a313640 1 -- 172.21.15.70:0/3379656020 >> 172.21.15.70:0/3379656020 conn(0x7fcb640faf10 msgr2=0x7fcb640fc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.643+0000 7fcb6a313640 1 -- 172.21.15.70:0/3379656020 shutdown_connections 2024-04-01T01:37:04.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.643+0000 7fcb6a313640 1 -- 172.21.15.70:0/3379656020 wait complete. 2024-04-01T01:37:04.662 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:37:04.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.844+0000 7fb649e2d640 1 -- 172.21.15.70:0/2183659213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb644108e20 msgr2=0x7fb6441092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.844+0000 7fb649e2d640 1 --2- 172.21.15.70:0/2183659213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb644108e20 0x7fb6441092a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb63401acb0 tx=0x7fb63403d130 comp rx=0 tx=0).stop 2024-04-01T01:37:04.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.844+0000 7fb649e2d640 1 -- 172.21.15.70:0/2183659213 shutdown_connections 2024-04-01T01:37:04.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.844+0000 7fb649e2d640 1 --2- 172.21.15.70:0/2183659213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6441097e0 0x7fb64410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.845+0000 7fb649e2d640 1 --2- 172.21.15.70:0/2183659213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb644108e20 0x7fb6441092a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.845+0000 7fb649e2d640 1 --2- 172.21.15.70:0/2183659213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb644072b60 0x7fb644072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.845+0000 7fb649e2d640 1 -- 172.21.15.70:0/2183659213 >> 172.21.15.70:0/2183659213 conn(0x7fb64406e630 msgr2=0x7fb644070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.845+0000 7fb649e2d640 1 -- 172.21.15.70:0/2183659213 shutdown_connections 2024-04-01T01:37:04.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.845+0000 7fb649e2d640 1 -- 172.21.15.70:0/2183659213 wait complete. 2024-04-01T01:37:04.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.847+0000 7fb649e2d640 1 Processor -- start 2024-04-01T01:37:04.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.848+0000 7fb649e2d640 1 -- start start 2024-04-01T01:37:04.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.849+0000 7fb649e2d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb644072b60 0x7fb64419de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.849+0000 7fb648e2b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb644072b60 0x7fb64419de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.849+0000 7fb648e2b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb644072b60 0x7fb64419de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:50666/0 (socket says 172.21.15.70:50666) 2024-04-01T01:37:04.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.849+0000 7fb648e2b640 1 -- 172.21.15.70:0/3488679918 learned_addr learned my addr 172.21.15.70:0/3488679918 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.850+0000 7fb649e2d640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb644108e20 0x7fb64419e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.850+0000 7fb649e2d640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6441097e0 0x7fb6441a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.850+0000 7fb649e2d640 1 -- 172.21.15.70:0/3488679918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb6441118e0 con 0x7fb644108e20 2024-04-01T01:37:04.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.850+0000 7fb649e2d640 1 -- 172.21.15.70:0/3488679918 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb644111760 con 0x7fb644072b60 2024-04-01T01:37:04.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.850+0000 7fb649e2d640 1 -- 172.21.15.70:0/3488679918 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb644111a60 con 0x7fb6441097e0 2024-04-01T01:37:04.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.851+0000 7fb64962c640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6441097e0 0x7fb6441a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.851+0000 7fb643fff640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb644108e20 0x7fb64419e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.851+0000 7fb648e2b640 1 -- 172.21.15.70:0/3488679918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6441097e0 msgr2=0x7fb6441a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.851+0000 7fb648e2b640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6441097e0 0x7fb6441a5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.851+0000 7fb648e2b640 1 -- 172.21.15.70:0/3488679918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb644108e20 msgr2=0x7fb64419e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.852+0000 7fb648e2b640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb644108e20 0x7fb64419e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.852+0000 7fb648e2b640 1 -- 172.21.15.70:0/3488679918 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6441a5a70 con 0x7fb644072b60 2024-04-01T01:37:04.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.852+0000 7fb648e2b640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb644072b60 0x7fb64419de50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb62c00d770 tx=0x7fb62c00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.853+0000 7fb64962c640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6441097e0 0x7fb6441a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:37:04.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.853+0000 7fb641ffb640 1 -- 172.21.15.70:0/3488679918 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62c017070 con 0x7fb644072b60 2024-04-01T01:37:04.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.853+0000 7fb641ffb640 1 -- 172.21.15.70:0/3488679918 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb62c009c30 con 0x7fb644072b60 2024-04-01T01:37:04.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.853+0000 7fb641ffb640 1 -- 172.21.15.70:0/3488679918 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62c003d00 con 0x7fb644072b60 2024-04-01T01:37:04.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.853+0000 7fb649e2d640 1 -- 172.21.15.70:0/3488679918 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6441a5d60 con 0x7fb644072b60 2024-04-01T01:37:04.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.854+0000 7fb649e2d640 1 -- 172.21.15.70:0/3488679918 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6441a61d0 con 0x7fb644072b60 2024-04-01T01:37:04.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.857+0000 7fb64dc7c740 1 Processor -- start 2024-04-01T01:37:04.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.857+0000 7fb64dc7c740 1 -- start start 2024-04-01T01:37:04.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.857+0000 7fb64dc7c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb5216a0 0x5618bb521ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.857+0000 7fb64dc7c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb522090 0x5618bb526b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.857+0000 7fb64dc7c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb527200 0x5618bb529690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.857+0000 7fb64dc7c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5618bb4861b0 con 0x5618bb5216a0 2024-04-01T01:37:04.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.857+0000 7fb64dc7c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5618bb42ec00 con 0x5618bb527200 2024-04-01T01:37:04.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.857+0000 7fb64dc7c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5618bb482250 con 0x5618bb522090 2024-04-01T01:37:04.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.859+0000 7fb641ffb640 1 -- 172.21.15.70:0/3488679918 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb62c004020 con 0x7fb644072b60 2024-04-01T01:37:04.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.859+0000 7fb64962c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb5216a0 0x5618bb521ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.860+0000 7fb648e2b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb522090 0x5618bb526b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.860+0000 7fb648e2b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb522090 0x5618bb526b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45978/0 (socket says 172.21.15.70:45978) 2024-04-01T01:37:04.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.860+0000 7fb648e2b640 1 -- 172.21.15.70:0/3997215540 learned_addr learned my addr 172.21.15.70:0/3997215540 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.860+0000 7fb643fff640 1 --2- 172.21.15.70:0/3997215540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb527200 0x5618bb529690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.861+0000 7fb648e2b640 1 -- 172.21.15.70:0/3997215540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb527200 msgr2=0x5618bb529690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.861+0000 7fb648e2b640 1 --2- 172.21.15.70:0/3997215540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb527200 0x5618bb529690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.861+0000 7fb648e2b640 1 -- 172.21.15.70:0/3997215540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb5216a0 msgr2=0x5618bb521ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.861+0000 7fb648e2b640 1 --2- 172.21.15.70:0/3997215540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb5216a0 0x5618bb521ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.861+0000 7fb648e2b640 1 -- 172.21.15.70:0/3997215540 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618bb44e3d0 con 0x5618bb522090 2024-04-01T01:37:04.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.861+0000 7fb641ffb640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6300a2670 0x7fb6300a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.862+0000 7fb641ffb640 1 -- 172.21.15.70:0/3488679918 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb62c0ce3f0 con 0x7fb644072b60 2024-04-01T01:37:04.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.862+0000 7fb64962c640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6300a2670 0x7fb6300a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.862+0000 7fb648e2b640 1 --2- 172.21.15.70:0/3997215540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb522090 0x5618bb526b90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0d6510 tx=0x7fb62c0d69d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e83712154e39b9b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.862+0000 7fb63affd640 1 -- 172.21.15.70:0/3997215540 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62c020070 con 0x5618bb522090 2024-04-01T01:37:04.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.862+0000 7fb63affd640 1 -- 172.21.15.70:0/3997215540 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb62c09b180 con 0x5618bb522090 2024-04-01T01:37:04.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.862+0000 7fb63affd640 1 -- 172.21.15.70:0/3997215540 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62c09a3e0 con 0x5618bb522090 2024-04-01T01:37:04.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.863+0000 7fb64dc7c740 1 -- 172.21.15.70:0/3997215540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb522090 msgr2=0x5618bb526b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.863+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/3997215540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb522090 0x5618bb526b90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0d6510 tx=0x7fb62c0d69d0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.864+0000 7fb64962c640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6300a2670 0x7fb6300a4b30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb63c005d00 tx=0x7fb63c005ed0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.864+0000 7fb64dc7c740 1 -- 172.21.15.70:0/3997215540 shutdown_connections 2024-04-01T01:37:04.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.864+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/3997215540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb527200 0x5618bb529690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.864+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/3997215540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb522090 0x5618bb526b90 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0d6510 tx=0x7fb62c0d69d0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.864+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/3997215540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb5216a0 0x5618bb521ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.864+0000 7fb64dc7c740 1 -- 172.21.15.70:0/3997215540 >> 172.21.15.70:0/3997215540 conn(0x5618bb51cc10 msgr2=0x5618bb51d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.865+0000 7fb64dc7c740 1 -- 172.21.15.70:0/3997215540 shutdown_connections 2024-04-01T01:37:04.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.865+0000 7fb64dc7c740 1 -- 172.21.15.70:0/3997215540 wait complete. 2024-04-01T01:37:04.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.866+0000 7fb64dc7c740 1 -- start start 2024-04-01T01:37:04.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.866+0000 7fb64dc7c740 1 Processor -- start 2024-04-01T01:37:04.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.867+0000 7fb64dc7c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb5216a0 0x5618bb52bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.867+0000 7fb64dc7c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb527200 0x5618bb52c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.867+0000 7fb64dc7c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb531200 0x5618bb5335f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.867+0000 7fb64dc7c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5618bb52a150 con 0x5618bb5216a0 2024-04-01T01:37:04.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.867+0000 7fb64dc7c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5618bb424470 con 0x5618bb527200 2024-04-01T01:37:04.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.867+0000 7fb64dc7c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5618bb52a2d0 con 0x5618bb531200 2024-04-01T01:37:04.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.867+0000 7fb64dc7c740 10 client.0 _ll_register_callbacks cb 0x5618bb441cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:04.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.867+0000 7fb648e2b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb5216a0 0x5618bb52bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.867+0000 7fb64962c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb531200 0x5618bb5335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.867+0000 7fb64962c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb531200 0x5618bb5335f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45994/0 (socket says 172.21.15.70:45994) 2024-04-01T01:37:04.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.867+0000 7fb64962c640 1 -- 172.21.15.70:0/2018314975 learned_addr learned my addr 172.21.15.70:0/2018314975 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.867+0000 7fb643fff640 1 --2- 172.21.15.70:0/2018314975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb527200 0x5618bb52c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.868+0000 7fb64962c640 1 -- 172.21.15.70:0/2018314975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb527200 msgr2=0x5618bb52c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.868+0000 7fb64962c640 1 --2- 172.21.15.70:0/2018314975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb527200 0x5618bb52c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.868+0000 7fb64962c640 1 -- 172.21.15.70:0/2018314975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb5216a0 msgr2=0x5618bb52bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.868+0000 7fb64962c640 1 --2- 172.21.15.70:0/2018314975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb5216a0 0x5618bb52bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.868+0000 7fb64962c640 1 -- 172.21.15.70:0/2018314975 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618bb19a3f0 con 0x5618bb531200 2024-04-01T01:37:04.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.868+0000 7fb64962c640 1 --2- 172.21.15.70:0/2018314975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb531200 0x5618bb5335f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb63c00ff40 tx=0x7fb63c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f92eedf18b572e0b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.869+0000 7fb623fff640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:37:04.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.869+0000 7fb623fff640 1 -- 172.21.15.70:0/2018314975 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c022070 con 0x5618bb531200 2024-04-01T01:37:04.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.869+0000 7fb623fff640 1 -- 172.21.15.70:0/2018314975 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb63c01e860 con 0x5618bb531200 2024-04-01T01:37:04.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.869+0000 7fb64dc7c740 1 -- 172.21.15.70:0/2018314975 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb62c09b180 con 0x5618bb531200 2024-04-01T01:37:04.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.869+0000 7fb623fff640 1 -- 172.21.15.70:0/2018314975 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb63c02c050 con 0x5618bb531200 2024-04-01T01:37:04.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.869+0000 7fb64dc7c740 10 client.26449 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:04.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.869+0000 7fb64dc7c740 1 -- 172.21.15.70:0/2018314975 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5618bb2647d0 con 0x5618bb531200 2024-04-01T01:37:04.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.870+0000 7fb623fff640 1 -- 172.21.15.70:0/2018314975 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb63c027440 con 0x5618bb531200 2024-04-01T01:37:04.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.870+0000 7fb623fff640 1 -- 172.21.15.70:0/2018314975 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb63c0103d0 con 0x5618bb531200 2024-04-01T01:37:04.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.870+0000 7fb64dc7c740 10 client.26449 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:04.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.870+0000 7fb64dc7c740 10 client.26449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:04.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.870+0000 7fb64dc7c740 20 client.26449 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:04.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.870+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/2018314975 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5618bb537bf0 0x5618bb53a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.870+0000 7fb64dc7c740 4 client.26449 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:04.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.870+0000 7fb64dc7c740 1 -- 172.21.15.70:0/2018314975 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb62c09b180 con 0x5618bb537bf0 2024-04-01T01:37:04.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.870+0000 7fb648e2b640 1 --2- 172.21.15.70:0/2018314975 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5618bb537bf0 0x5618bb53a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.871+0000 7fb623fff640 10 client.26449 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:04.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.871+0000 7fb648e2b640 1 --2- 172.21.15.70:0/2018314975 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5618bb537bf0 0x5618bb53a060 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff86ddaee07079d4 server_cookie=1c66a8ec3e1e38a6 in_seq=0 out_seq=0 2024-04-01T01:37:04.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.872+0000 7fb623fff640 1 -- 172.21.15.70:0/2018314975 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fb62c09b180 con 0x5618bb537bf0 2024-04-01T01:37:04.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.872+0000 7fb623fff640 10 client.26449 handle_command_reply: tid=0 2024-04-01T01:37:04.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.873+0000 7fb64dc7c740 1 client.26449 shutdown 2024-04-01T01:37:04.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.873+0000 7fb64dc7c740 1 -- 172.21.15.70:0/2018314975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb531200 msgr2=0x5618bb5335f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.873+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/2018314975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb531200 0x5618bb5335f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb63c00ff40 tx=0x7fb63c0046c0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.874+0000 7fb64dc7c740 1 -- 172.21.15.70:0/2018314975 shutdown_connections 2024-04-01T01:37:04.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.874+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/2018314975 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5618bb537bf0 0x5618bb53a060 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.874+0000 7fb623fff640 0 client.26449 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:04.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.874+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/2018314975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb531200 0x5618bb5335f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.874+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/2018314975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb527200 0x5618bb52c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.874+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/2018314975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb5216a0 0x5618bb52bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.875+0000 7fb64dc7c740 1 -- 172.21.15.70:0/2018314975 >> 172.21.15.70:0/2018314975 conn(0x5618bb51cc10 msgr2=0x5618bb51bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.877+0000 7fb64dc7c740 1 -- 172.21.15.70:0/2018314975 shutdown_connections 2024-04-01T01:37:04.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.877+0000 7fb64dc7c740 1 -- 172.21.15.70:0/2018314975 wait complete. 2024-04-01T01:37:04.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.878+0000 7fb64dc7c740 20 client.26449 trim_cache size 0 max 16384 2024-04-01T01:37:04.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.906+0000 7fb64dc7c740 1 Processor -- start 2024-04-01T01:37:04.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.906+0000 7fb64dc7c740 1 -- start start 2024-04-01T01:37:04.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.906+0000 7fb64dc7c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb5216a0 0x5618bb51b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.907+0000 7fb64dc7c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb527200 0x5618bb51bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.907+0000 7fb648e2b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb5216a0 0x5618bb51b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.907+0000 7fb648e2b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb5216a0 0x5618bb51b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:50692/0 (socket says 172.21.15.70:50692) 2024-04-01T01:37:04.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.907+0000 7fb643fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb527200 0x5618bb51bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.907+0000 7fb643fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb527200 0x5618bb51bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46004/0 (socket says 172.21.15.70:46004) 2024-04-01T01:37:04.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.907+0000 7fb643fff640 1 -- 172.21.15.70:0/997988893 learned_addr learned my addr 172.21.15.70:0/997988893 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.907+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/997988893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb531200 0x5618bb51c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.907+0000 7fb64dc7c740 1 -- 172.21.15.70:0/997988893 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb6441b0780 con 0x5618bb531200 2024-04-01T01:37:04.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.908+0000 7fb64dc7c740 1 -- 172.21.15.70:0/997988893 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb6441b22c0 con 0x5618bb5216a0 2024-04-01T01:37:04.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.908+0000 7fb64dc7c740 1 -- 172.21.15.70:0/997988893 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb6441b0e50 con 0x5618bb527200 2024-04-01T01:37:04.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.908+0000 7fb64962c640 1 --2- 172.21.15.70:0/997988893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb531200 0x5618bb51c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.909+0000 7fb64962c640 1 -- 172.21.15.70:0/997988893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb527200 msgr2=0x5618bb51bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.909+0000 7fb64962c640 1 --2- 172.21.15.70:0/997988893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb527200 0x5618bb51bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.909+0000 7fb64962c640 1 -- 172.21.15.70:0/997988893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb5216a0 msgr2=0x5618bb51b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.909+0000 7fb64962c640 1 --2- 172.21.15.70:0/997988893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb5216a0 0x5618bb51b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.909+0000 7fb64962c640 1 -- 172.21.15.70:0/997988893 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618bb52ccf0 con 0x5618bb531200 2024-04-01T01:37:04.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.909+0000 7fb64962c640 1 --2- 172.21.15.70:0/997988893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb531200 0x5618bb51c180 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb63c010870 tx=0x7fb63c00fb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9afa5d03e9fa59ab server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.909+0000 7fb63a7fc640 1 -- 172.21.15.70:0/997988893 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c022070 con 0x5618bb531200 2024-04-01T01:37:04.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.910+0000 7fb63a7fc640 1 -- 172.21.15.70:0/997988893 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb63c010c60 con 0x5618bb531200 2024-04-01T01:37:04.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.910+0000 7fb63a7fc640 1 -- 172.21.15.70:0/997988893 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c0054b0 con 0x5618bb531200 2024-04-01T01:37:04.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.911+0000 7fb64dc7c740 1 -- 172.21.15.70:0/997988893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb531200 msgr2=0x5618bb51c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.911+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/997988893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb531200 0x5618bb51c180 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb63c010870 tx=0x7fb63c00fb80 comp rx=0 tx=0).stop 2024-04-01T01:37:04.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.911+0000 7fb64dc7c740 1 -- 172.21.15.70:0/997988893 shutdown_connections 2024-04-01T01:37:04.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.911+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/997988893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb531200 0x5618bb51c180 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.911+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/997988893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb527200 0x5618bb51bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.911+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/997988893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb5216a0 0x5618bb51b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.911+0000 7fb64dc7c740 1 -- 172.21.15.70:0/997988893 >> 172.21.15.70:0/997988893 conn(0x5618bb51cc10 msgr2=0x5618bb52a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.912+0000 7fb64dc7c740 1 -- 172.21.15.70:0/997988893 shutdown_connections 2024-04-01T01:37:04.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.912+0000 7fb64dc7c740 1 -- 172.21.15.70:0/997988893 wait complete. 2024-04-01T01:37:04.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.913+0000 7fb64dc7c740 1 -- start start 2024-04-01T01:37:04.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.913+0000 7fb64dc7c740 1 Processor -- start 2024-04-01T01:37:04.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.914+0000 7fb64dc7c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb5216a0 0x5618bb549010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.914+0000 7fb64dc7c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb527200 0x5618bb549550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.914+0000 7fb64dc7c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb531200 0x5618bb54a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.914+0000 7fb64dc7c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb6441b1bf0 con 0x5618bb531200 2024-04-01T01:37:04.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.914+0000 7fb64dc7c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb6441b1520 con 0x5618bb527200 2024-04-01T01:37:04.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.914+0000 7fb64dc7c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb6441ae760 con 0x5618bb5216a0 2024-04-01T01:37:04.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.914+0000 7fb64dc7c740 10 client.0 _ll_register_callbacks cb 0x7fb63c00fbf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:04.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.914+0000 7fb64962c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb531200 0x5618bb54a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.914+0000 7fb64962c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb531200 0x5618bb54a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56856/0 (socket says 172.21.15.70:56856) 2024-04-01T01:37:04.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.914+0000 7fb64962c640 1 -- 172.21.15.70:0/519436870 learned_addr learned my addr 172.21.15.70:0/519436870 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:04.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.914+0000 7fb643fff640 1 --2- 172.21.15.70:0/519436870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb527200 0x5618bb549550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.914+0000 7fb648e2b640 1 --2- 172.21.15.70:0/519436870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb5216a0 0x5618bb549010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.915+0000 7fb643fff640 1 -- 172.21.15.70:0/519436870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb5216a0 msgr2=0x5618bb549010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.915+0000 7fb643fff640 1 --2- 172.21.15.70:0/519436870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb5216a0 0x5618bb549010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.915+0000 7fb643fff640 1 -- 172.21.15.70:0/519436870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb531200 msgr2=0x5618bb54a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:04.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.915+0000 7fb643fff640 1 --2- 172.21.15.70:0/519436870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb531200 0x5618bb54a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.915+0000 7fb643fff640 1 -- 172.21.15.70:0/519436870 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618bb54ab50 con 0x5618bb527200 2024-04-01T01:37:04.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.915+0000 7fb64962c640 1 --2- 172.21.15.70:0/519436870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb531200 0x5618bb54a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:04.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.915+0000 7fb648e2b640 1 --2- 172.21.15.70:0/519436870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb5216a0 0x5618bb549010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:37:04.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.915+0000 7fb643fff640 1 --2- 172.21.15.70:0/519436870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb527200 0x5618bb549550 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb634000f80 tx=0x7fb63401a8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=530c47aaca3a9437 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:04.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.915+0000 7fb623fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:04.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.915+0000 7fb623fff640 1 -- 172.21.15.70:0/519436870 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63404a040 con 0x5618bb527200 2024-04-01T01:37:04.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.915+0000 7fb623fff640 1 -- 172.21.15.70:0/519436870 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb634019040 con 0x5618bb527200 2024-04-01T01:37:04.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.915+0000 7fb64dc7c740 1 -- 172.21.15.70:0/519436870 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb63c010c60 con 0x5618bb527200 2024-04-01T01:37:04.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.916+0000 7fb623fff640 1 -- 172.21.15.70:0/519436870 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb6340457b0 con 0x5618bb527200 2024-04-01T01:37:04.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.916+0000 7fb64dc7c740 10 client.26558 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:04.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.916+0000 7fb64dc7c740 1 -- 172.21.15.70:0/519436870 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5618bb59e1a0 con 0x5618bb527200 2024-04-01T01:37:04.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.916+0000 7fb623fff640 1 -- 172.21.15.70:0/519436870 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb634004040 con 0x5618bb527200 2024-04-01T01:37:04.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.916+0000 7fb623fff640 1 -- 172.21.15.70:0/519436870 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb63403dce0 con 0x5618bb527200 2024-04-01T01:37:04.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.917+0000 7fb64dc7c740 10 client.26558 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:04.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.917+0000 7fb64dc7c740 10 client.26558 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:04.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.917+0000 7fb64dc7c740 20 client.26558 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:04.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.917+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/519436870 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5618bb59eac0 0x5618bb5a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:04.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.917+0000 7fb64dc7c740 4 client.26558 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:37:04.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.917+0000 7fb64dc7c740 1 -- 172.21.15.70:0/519436870 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fb63c010c60 con 0x5618bb59eac0 2024-04-01T01:37:04.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.917+0000 7fb648e2b640 1 --2- 172.21.15.70:0/519436870 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5618bb59eac0 0x5618bb5a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:04.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.918+0000 7fb648e2b640 1 --2- 172.21.15.70:0/519436870 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5618bb59eac0 0x5618bb5a0ee0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62571993ff45cde6 server_cookie=3652cb6412053be in_seq=0 out_seq=0 2024-04-01T01:37:04.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.918+0000 7fb623fff640 10 client.26558 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:04.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.919+0000 7fb623fff640 1 -- 172.21.15.70:0/519436870 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fb63c010c60 con 0x5618bb59eac0 2024-04-01T01:37:04.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.919+0000 7fb623fff640 10 client.26558 handle_command_reply: tid=0 2024-04-01T01:37:04.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.919+0000 7fb64dc7c740 1 client.26558 shutdown 2024-04-01T01:37:04.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.919+0000 7fb64dc7c740 1 -- 172.21.15.70:0/519436870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb527200 msgr2=0x5618bb549550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.919+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/519436870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb527200 0x5618bb549550 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb634000f80 tx=0x7fb63401a8f0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.919+0000 7fb64dc7c740 1 -- 172.21.15.70:0/519436870 shutdown_connections 2024-04-01T01:37:04.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.919+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/519436870 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5618bb59eac0 0x5618bb5a0ee0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.919+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/519436870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5618bb531200 0x5618bb54a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.919+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/519436870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5618bb527200 0x5618bb549550 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.919+0000 7fb623fff640 0 client.26558 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:04.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.919+0000 7fb64dc7c740 1 --2- 172.21.15.70:0/519436870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618bb5216a0 0x5618bb549010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.919+0000 7fb64dc7c740 1 -- 172.21.15.70:0/519436870 >> 172.21.15.70:0/519436870 conn(0x5618bb51cc10 msgr2=0x5618bb51b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.920+0000 7fb64dc7c740 1 -- 172.21.15.70:0/519436870 shutdown_connections 2024-04-01T01:37:04.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.920+0000 7fb64dc7c740 1 -- 172.21.15.70:0/519436870 wait complete. 2024-04-01T01:37:04.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.920+0000 7fb64dc7c740 20 client.26558 trim_cache size 0 max 16384 2024-04-01T01:37:04.920 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:37:04.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.921+0000 7fb649e2d640 1 -- 172.21.15.70:0/3488679918 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6300a2670 msgr2=0x7fb6300a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.921+0000 7fb649e2d640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6300a2670 0x7fb6300a4b30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb63c005d00 tx=0x7fb63c005ed0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.921+0000 7fb649e2d640 1 -- 172.21.15.70:0/3488679918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb644072b60 msgr2=0x7fb64419de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:04.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.921+0000 7fb649e2d640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb644072b60 0x7fb64419de50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb62c00d770 tx=0x7fb62c00dc30 comp rx=0 tx=0).stop 2024-04-01T01:37:04.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.921+0000 7fb649e2d640 1 -- 172.21.15.70:0/3488679918 shutdown_connections 2024-04-01T01:37:04.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.921+0000 7fb649e2d640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6300a2670 0x7fb6300a4b30 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.921+0000 7fb649e2d640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6441097e0 0x7fb6441a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.921+0000 7fb649e2d640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb644108e20 0x7fb64419e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.921+0000 7fb649e2d640 1 --2- 172.21.15.70:0/3488679918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb644072b60 0x7fb64419de50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:04.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.921+0000 7fb649e2d640 1 -- 172.21.15.70:0/3488679918 >> 172.21.15.70:0/3488679918 conn(0x7fb64406e630 msgr2=0x7fb64410c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:04.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.921+0000 7fb649e2d640 1 -- 172.21.15.70:0/3488679918 shutdown_connections 2024-04-01T01:37:04.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:04.921+0000 7fb649e2d640 1 -- 172.21.15.70:0/3488679918 wait complete. 2024-04-01T01:37:04.939 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:04 smithi070 ceph-mon[32179]: pgmap v466: 129 pgs: 129 active+clean; 1011 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 9.8 MiB/s wr, 1.16k op/s 2024-04-01T01:37:05.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:04 smithi081 ceph-mon[35142]: pgmap v466: 129 pgs: 129 active+clean; 1011 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 9.8 MiB/s wr, 1.16k op/s 2024-04-01T01:37:05.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:04 smithi189 ceph-mon[35753]: pgmap v466: 129 pgs: 129 active+clean; 1011 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 9.8 MiB/s wr, 1.16k op/s 2024-04-01T01:37:05.941 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:37:05.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:37:05.941 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:37:06.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.146+0000 7ff3087c7640 1 -- 172.21.15.70:0/520340626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff300102200 msgr2=0x7ff300108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.146+0000 7ff3087c7640 1 --2- 172.21.15.70:0/520340626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff300102200 0x7ff300108a90 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009900 tx=0x7ff2f002bc40 comp rx=0 tx=0).stop 2024-04-01T01:37:06.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.146+0000 7ff3087c7640 1 -- 172.21.15.70:0/520340626 shutdown_connections 2024-04-01T01:37:06.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.146+0000 7ff3087c7640 1 --2- 172.21.15.70:0/520340626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff300102200 0x7ff300108a90 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.146+0000 7ff3087c7640 1 --2- 172.21.15.70:0/520340626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff300101840 0x7ff300101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.146+0000 7ff3087c7640 1 --2- 172.21.15.70:0/520340626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff300100650 0x7ff300100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.146+0000 7ff3087c7640 1 -- 172.21.15.70:0/520340626 >> 172.21.15.70:0/520340626 conn(0x7ff3000fbbe0 msgr2=0x7ff3000fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:06.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.146+0000 7ff3087c7640 1 -- 172.21.15.70:0/520340626 shutdown_connections 2024-04-01T01:37:06.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.146+0000 7ff3087c7640 1 -- 172.21.15.70:0/520340626 wait complete. 2024-04-01T01:37:06.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.147+0000 7ff3087c7640 1 Processor -- start 2024-04-01T01:37:06.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.148+0000 7ff3087c7640 1 -- start start 2024-04-01T01:37:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.148+0000 7ff3087c7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff300100650 0x7ff30019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.148+0000 7ff3087c7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff300101840 0x7ff30019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.148+0000 7ff3087c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff300102200 0x7ff3001a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.148+0000 7ff3087c7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff300109ef0 con 0x7ff300101840 2024-04-01T01:37:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.148+0000 7ff3087c7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff300109d70 con 0x7ff300102200 2024-04-01T01:37:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.148+0000 7ff3087c7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff30010a070 con 0x7ff300100650 2024-04-01T01:37:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.149+0000 7ff306d64640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff300102200 0x7ff3001a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.149+0000 7ff306563640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff300100650 0x7ff30019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.149+0000 7ff305d62640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff300101840 0x7ff30019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.149+0000 7ff306563640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff300100650 0x7ff30019e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46022/0 (socket says 172.21.15.70:46022) 2024-04-01T01:37:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.149+0000 7ff306563640 1 -- 172.21.15.70:0/55438268 learned_addr learned my addr 172.21.15.70:0/55438268 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.149+0000 7ff306d64640 1 -- 172.21.15.70:0/55438268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff300100650 msgr2=0x7ff30019e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.149+0000 7ff306d64640 1 --2- 172.21.15.70:0/55438268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff300100650 0x7ff30019e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.149+0000 7ff306d64640 1 -- 172.21.15.70:0/55438268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff300101840 msgr2=0x7ff30019e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.149+0000 7ff306d64640 1 --2- 172.21.15.70:0/55438268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff300101840 0x7ff30019e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.149+0000 7ff306d64640 1 -- 172.21.15.70:0/55438268 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3001a5b70 con 0x7ff300102200 2024-04-01T01:37:06.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.149+0000 7ff305d62640 1 --2- 172.21.15.70:0/55438268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff300101840 0x7ff30019e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:06.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.150+0000 7ff306d64640 1 --2- 172.21.15.70:0/55438268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff300102200 0x7ff3001a5500 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0036040 tx=0x7ff2f002c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:06.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.150+0000 7ff2f77fe640 1 -- 172.21.15.70:0/55438268 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f0043020 con 0x7ff300102200 2024-04-01T01:37:06.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.150+0000 7ff2f77fe640 1 -- 172.21.15.70:0/55438268 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2f0004350 con 0x7ff300102200 2024-04-01T01:37:06.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.150+0000 7ff2f77fe640 1 -- 172.21.15.70:0/55438268 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f0003a80 con 0x7ff300102200 2024-04-01T01:37:06.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.150+0000 7ff3087c7640 1 -- 172.21.15.70:0/55438268 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3001a5e00 con 0x7ff300102200 2024-04-01T01:37:06.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.150+0000 7ff3087c7640 1 -- 172.21.15.70:0/55438268 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3001a6270 con 0x7ff300102200 2024-04-01T01:37:06.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.154+0000 7ff2f77fe640 1 -- 172.21.15.70:0/55438268 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7ff2f0046020 con 0x7ff300102200 2024-04-01T01:37:06.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.156+0000 7ff30b3a1740 1 Processor -- start 2024-04-01T01:37:06.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.156+0000 7ff2f77fe640 1 --2- 172.21.15.70:0/55438268 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2ec0a2570 0x7ff2ec0a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.156+0000 7ff30b3a1740 1 -- start start 2024-04-01T01:37:06.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff30b3a1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab404a10 0x561eab404e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff30b3a1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab405400 0x561eab409f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff30b3a1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab40a570 0x561eab40ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff30b3a1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561eab3060b0 con 0x561eab404a10 2024-04-01T01:37:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff30b3a1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561eab368ea0 con 0x561eab405400 2024-04-01T01:37:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff30b3a1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561eab311c00 con 0x561eab40a570 2024-04-01T01:37:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff305d62640 1 --2- 172.21.15.70:0/55438268 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2ec0a2570 0x7ff2ec0a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff2f77fe640 1 -- 172.21.15.70:0/55438268 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff2f00f0c60 con 0x7ff300102200 2024-04-01T01:37:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff305d62640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab405400 0x561eab409f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff305d62640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab405400 0x561eab409f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:50740/0 (socket says 172.21.15.70:50740) 2024-04-01T01:37:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff305d62640 1 -- 172.21.15.70:0/1432232253 learned_addr learned my addr 172.21.15.70:0/1432232253 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff306563640 1 --2- 172.21.15.70:0/1432232253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab404a10 0x561eab404e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff306d64640 1 --2- 172.21.15.70:0/1432232253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab40a570 0x561eab40ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff305d62640 1 -- 172.21.15.70:0/1432232253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab40a570 msgr2=0x561eab40ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff305d62640 1 --2- 172.21.15.70:0/1432232253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab40a570 0x561eab40ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff305d62640 1 -- 172.21.15.70:0/1432232253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab404a10 msgr2=0x561eab404e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff305d62640 1 --2- 172.21.15.70:0/1432232253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab404a10 0x561eab404e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.157+0000 7ff305d62640 1 -- 172.21.15.70:0/1432232253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561eab1457e0 con 0x561eab405400 2024-04-01T01:37:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.158+0000 7ff305d62640 1 --2- 172.21.15.70:0/1432232253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab405400 0x561eab409f00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc013ed0 tx=0x7ff2fc0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5cbbcc50e83df9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.158+0000 7ff305d62640 1 --2- 172.21.15.70:0/55438268 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2ec0a2570 0x7ff2ec0a4a30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc010ca0 tx=0x7ff2fc008000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.158+0000 7ff2f4ff9640 1 -- 172.21.15.70:0/1432232253 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc023070 con 0x561eab405400 2024-04-01T01:37:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.159+0000 7ff2f4ff9640 1 -- 172.21.15.70:0/1432232253 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2fc009e40 con 0x561eab405400 2024-04-01T01:37:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.159+0000 7ff2f4ff9640 1 -- 172.21.15.70:0/1432232253 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc011af0 con 0x561eab405400 2024-04-01T01:37:06.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.160+0000 7ff30b3a1740 1 -- 172.21.15.70:0/1432232253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab405400 msgr2=0x561eab409f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.160+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/1432232253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab405400 0x561eab409f00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc013ed0 tx=0x7ff2fc0107a0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.160+0000 7ff30b3a1740 1 -- 172.21.15.70:0/1432232253 shutdown_connections 2024-04-01T01:37:06.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.160+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/1432232253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab40a570 0x561eab40ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.160+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/1432232253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab405400 0x561eab409f00 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.160+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/1432232253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab404a10 0x561eab404e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.160+0000 7ff30b3a1740 1 -- 172.21.15.70:0/1432232253 >> 172.21.15.70:0/1432232253 conn(0x561eab3fff80 msgr2=0x561eab400380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.160+0000 7ff30b3a1740 1 -- 172.21.15.70:0/1432232253 shutdown_connections 2024-04-01T01:37:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.160+0000 7ff30b3a1740 1 -- 172.21.15.70:0/1432232253 wait complete. 2024-04-01T01:37:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.160+0000 7ff30b3a1740 1 -- start start 2024-04-01T01:37:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.161+0000 7ff30b3a1740 1 Processor -- start 2024-04-01T01:37:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.161+0000 7ff30b3a1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab404a10 0x561eab40ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.161+0000 7ff30b3a1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab405400 0x561eab40f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.161+0000 7ff30b3a1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab40a570 0x561eab416380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.161+0000 7ff30b3a1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561eab3073d0 con 0x561eab40a570 2024-04-01T01:37:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.161+0000 7ff30b3a1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561eab364f40 con 0x561eab405400 2024-04-01T01:37:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.161+0000 7ff30b3a1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561eab40d4c0 con 0x561eab404a10 2024-04-01T01:37:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.161+0000 7ff30b3a1740 10 client.0 _ll_register_callbacks cb 0x561eab324b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.162+0000 7ff305d62640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab405400 0x561eab40f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.162+0000 7ff305d62640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab405400 0x561eab40f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:50750/0 (socket says 172.21.15.70:50750) 2024-04-01T01:37:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.162+0000 7ff305d62640 1 -- 172.21.15.70:0/3071181062 learned_addr learned my addr 172.21.15.70:0/3071181062 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.162+0000 7ff306d64640 1 --2- 172.21.15.70:0/3071181062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab40a570 0x561eab416380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.162+0000 7ff306d64640 1 -- 172.21.15.70:0/3071181062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab404a10 msgr2=0x561eab40ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.162+0000 7ff306d64640 1 --2- 172.21.15.70:0/3071181062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab404a10 0x561eab40ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.162+0000 7ff306d64640 1 -- 172.21.15.70:0/3071181062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab405400 msgr2=0x561eab40f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.162+0000 7ff306d64640 1 --2- 172.21.15.70:0/3071181062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab405400 0x561eab40f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.162+0000 7ff306d64640 1 -- 172.21.15.70:0/3071181062 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561eab1a5f70 con 0x561eab40a570 2024-04-01T01:37:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.163+0000 7ff306d64640 1 --2- 172.21.15.70:0/3071181062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab40a570 0x561eab416380 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009850 tx=0x7ff2f00bcf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe37f49e9e5de9dd server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.163+0000 7ff2e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:37:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.163+0000 7ff2e9ffb640 1 -- 172.21.15.70:0/3071181062 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f00bd350 con 0x561eab40a570 2024-04-01T01:37:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.163+0000 7ff2e9ffb640 1 -- 172.21.15.70:0/3071181062 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff2f0005440 con 0x561eab40a570 2024-04-01T01:37:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.163+0000 7ff30b3a1740 1 -- 172.21.15.70:0/3071181062 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561eab3763e0 con 0x561eab40a570 2024-04-01T01:37:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.164+0000 7ff2e9ffb640 1 -- 172.21.15.70:0/3071181062 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff2f00c14a0 con 0x561eab40a570 2024-04-01T01:37:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.164+0000 7ff30b3a1740 10 client.16716 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.164+0000 7ff30b3a1740 1 -- 172.21.15.70:0/3071181062 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561eab03d170 con 0x561eab40a570 2024-04-01T01:37:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.165+0000 7ff2e9ffb640 1 -- 172.21.15.70:0/3071181062 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x561eab3763e0 con 0x561eab40a570 2024-04-01T01:37:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.165+0000 7ff2e9ffb640 1 -- 172.21.15.70:0/3071181062 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff2f00d7070 con 0x561eab40a570 2024-04-01T01:37:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.165+0000 7ff30b3a1740 10 client.16716 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.165+0000 7ff30b3a1740 10 client.16716 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.165+0000 7ff30b3a1740 20 client.16716 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.165+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/3071181062 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561eab41a870 0x561eab41cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.165+0000 7ff30b3a1740 4 client.16716 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.165+0000 7ff30b3a1740 1 -- 172.21.15.70:0/3071181062 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561eab370730 con 0x561eab41a870 2024-04-01T01:37:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.166+0000 7ff306563640 1 --2- 172.21.15.70:0/3071181062 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561eab41a870 0x561eab41cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.166+0000 7ff306563640 1 --2- 172.21.15.70:0/3071181062 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561eab41a870 0x561eab41cd30 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=460fc2c5d13aac0d server_cookie=c3d3c5707942fd61 in_seq=0 out_seq=0 2024-04-01T01:37:06.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.167+0000 7ff2e9ffb640 10 client.16716 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.167+0000 7ff2e9ffb640 1 -- 172.21.15.70:0/3071181062 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561eab370730 con 0x561eab41a870 2024-04-01T01:37:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.167+0000 7ff2e9ffb640 10 client.16716 handle_command_reply: tid=0 2024-04-01T01:37:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.167+0000 7ff30b3a1740 1 client.16716 shutdown 2024-04-01T01:37:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.168+0000 7ff30b3a1740 1 -- 172.21.15.70:0/3071181062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab40a570 msgr2=0x561eab416380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.169+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/3071181062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab40a570 0x561eab416380 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009850 tx=0x7ff2f00bcf50 comp rx=0 tx=0).stop 2024-04-01T01:37:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.169+0000 7ff30b3a1740 1 -- 172.21.15.70:0/3071181062 shutdown_connections 2024-04-01T01:37:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.169+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/3071181062 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561eab41a870 0x561eab41cd30 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.169+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/3071181062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab40a570 0x561eab416380 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.169+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/3071181062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab405400 0x561eab40f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.169+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/3071181062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab404a10 0x561eab40ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.169+0000 7ff30b3a1740 1 -- 172.21.15.70:0/3071181062 >> 172.21.15.70:0/3071181062 conn(0x561eab3fff80 msgr2=0x561eab3ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.169+0000 7ff2e9ffb640 0 client.16716 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.169+0000 7ff30b3a1740 1 -- 172.21.15.70:0/3071181062 shutdown_connections 2024-04-01T01:37:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.169+0000 7ff30b3a1740 1 -- 172.21.15.70:0/3071181062 wait complete. 2024-04-01T01:37:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.169+0000 7ff30b3a1740 20 client.16716 trim_cache size 0 max 16384 2024-04-01T01:37:06.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.204+0000 7ff30b3a1740 1 Processor -- start 2024-04-01T01:37:06.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.205+0000 7ff30b3a1740 1 -- start start 2024-04-01T01:37:06.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.207+0000 7ff30b3a1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab404a10 0x561eab3fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.207+0000 7ff30b3a1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab405400 0x561eab416340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.207+0000 7ff30b3a1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab40a570 0x561eab3fee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.207+0000 7ff30b3a1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff3001b0a00 con 0x561eab405400 2024-04-01T01:37:06.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.207+0000 7ff30b3a1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff3001b2540 con 0x561eab40a570 2024-04-01T01:37:06.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.207+0000 7ff30b3a1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff3001b10d0 con 0x561eab404a10 2024-04-01T01:37:06.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.208+0000 7ff305d62640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab405400 0x561eab416340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.208+0000 7ff306563640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab404a10 0x561eab3fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.208+0000 7ff305d62640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab405400 0x561eab416340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56902/0 (socket says 172.21.15.70:56902) 2024-04-01T01:37:06.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.208+0000 7ff305d62640 1 -- 172.21.15.70:0/2434972424 learned_addr learned my addr 172.21.15.70:0/2434972424 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:06.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.208+0000 7ff306d64640 1 --2- 172.21.15.70:0/2434972424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab40a570 0x561eab3fee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.209+0000 7ff306563640 1 -- 172.21.15.70:0/2434972424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab40a570 msgr2=0x561eab3fee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:06.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.209+0000 7ff306563640 1 --2- 172.21.15.70:0/2434972424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab40a570 0x561eab3fee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.209+0000 7ff306563640 1 -- 172.21.15.70:0/2434972424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab405400 msgr2=0x561eab416340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:06.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.209+0000 7ff306563640 1 --2- 172.21.15.70:0/2434972424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab405400 0x561eab416340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.209+0000 7ff306563640 1 -- 172.21.15.70:0/2434972424 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561eab409c50 con 0x561eab404a10 2024-04-01T01:37:06.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.209+0000 7ff306563640 1 --2- 172.21.15.70:0/2434972424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab404a10 0x561eab3fe8f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8000f80 tx=0x7ff2f8012420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a3241f44a478ab3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:06.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.209+0000 7ff2ebfff640 1 -- 172.21.15.70:0/2434972424 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8013d90 con 0x561eab404a10 2024-04-01T01:37:06.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.209+0000 7ff2ebfff640 1 -- 172.21.15.70:0/2434972424 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2f801c340 con 0x561eab404a10 2024-04-01T01:37:06.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.209+0000 7ff2ebfff640 1 -- 172.21.15.70:0/2434972424 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8005510 con 0x561eab404a10 2024-04-01T01:37:06.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.210+0000 7ff30b3a1740 1 -- 172.21.15.70:0/2434972424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab404a10 msgr2=0x561eab3fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.210+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/2434972424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab404a10 0x561eab3fe8f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8000f80 tx=0x7ff2f8012420 comp rx=0 tx=0).stop 2024-04-01T01:37:06.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.210+0000 7ff30b3a1740 1 -- 172.21.15.70:0/2434972424 shutdown_connections 2024-04-01T01:37:06.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.210+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/2434972424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab40a570 0x561eab3fee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.210+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/2434972424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab405400 0x561eab416340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.210+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/2434972424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab404a10 0x561eab3fe8f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.210+0000 7ff30b3a1740 1 -- 172.21.15.70:0/2434972424 >> 172.21.15.70:0/2434972424 conn(0x561eab3fff80 msgr2=0x561eab40d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:06.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.210+0000 7ff30b3a1740 1 -- 172.21.15.70:0/2434972424 shutdown_connections 2024-04-01T01:37:06.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.210+0000 7ff30b3a1740 1 -- 172.21.15.70:0/2434972424 wait complete. 2024-04-01T01:37:06.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.211+0000 7ff30b3a1740 1 -- start start 2024-04-01T01:37:06.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.211+0000 7ff30b3a1740 1 Processor -- start 2024-04-01T01:37:06.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.211+0000 7ff30b3a1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab404a10 0x561eab413e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.211+0000 7ff30b3a1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab405400 0x561eab414390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.211+0000 7ff30b3a1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab40a570 0x561eab415240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.211+0000 7ff306563640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab404a10 0x561eab413e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.211+0000 7ff30b3a1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff3001b1e70 con 0x561eab405400 2024-04-01T01:37:06.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.211+0000 7ff30b3a1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff3001b17a0 con 0x561eab404a10 2024-04-01T01:37:06.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.211+0000 7ff30b3a1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff3001ae9e0 con 0x561eab40a570 2024-04-01T01:37:06.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.212+0000 7ff30b3a1740 10 client.0 _ll_register_callbacks cb 0x7ff2f00bd120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:06.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.212+0000 7ff306563640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab404a10 0x561eab413e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:50768/0 (socket says 172.21.15.70:50768) 2024-04-01T01:37:06.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.212+0000 7ff306563640 1 -- 172.21.15.70:0/3518568699 learned_addr learned my addr 172.21.15.70:0/3518568699 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:06.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.212+0000 7ff306d64640 1 --2- 172.21.15.70:0/3518568699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab40a570 0x561eab415240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.213+0000 7ff306563640 1 -- 172.21.15.70:0/3518568699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab40a570 msgr2=0x561eab415240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:06.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.213+0000 7ff306563640 1 --2- 172.21.15.70:0/3518568699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab40a570 0x561eab415240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.213+0000 7ff306563640 1 -- 172.21.15.70:0/3518568699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab405400 msgr2=0x561eab414390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:06.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.213+0000 7ff306563640 1 --2- 172.21.15.70:0/3518568699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab405400 0x561eab414390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.213+0000 7ff306563640 1 -- 172.21.15.70:0/3518568699 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561eab415830 con 0x561eab404a10 2024-04-01T01:37:06.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.213+0000 7ff306563640 1 --2- 172.21.15.70:0/3518568699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab404a10 0x561eab413e50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009f50 tx=0x7ff2f8013c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2220b25b18b8c59c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:06.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.213+0000 7ff2e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.213+0000 7ff2e9ffb640 1 -- 172.21.15.70:0/3518568699 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f801c4f0 con 0x561eab404a10 2024-04-01T01:37:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.213+0000 7ff2e9ffb640 1 -- 172.21.15.70:0/3518568699 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff2f801c690 con 0x561eab404a10 2024-04-01T01:37:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.214+0000 7ff30b3a1740 1 -- 172.21.15.70:0/3518568699 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2f801c340 con 0x561eab404a10 2024-04-01T01:37:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.214+0000 7ff2e9ffb640 1 -- 172.21.15.70:0/3518568699 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff2f8034050 con 0x561eab404a10 2024-04-01T01:37:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.214+0000 7ff30b3a1740 10 client.26576 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.214+0000 7ff30b3a1740 1 -- 172.21.15.70:0/3518568699 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561eab480f40 con 0x561eab404a10 2024-04-01T01:37:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.214+0000 7ff2e9ffb640 1 -- 172.21.15.70:0/3518568699 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2f801c340 con 0x561eab404a10 2024-04-01T01:37:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.214+0000 7ff2e9ffb640 1 -- 172.21.15.70:0/3518568699 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff2f80040d0 con 0x561eab404a10 2024-04-01T01:37:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.215+0000 7ff30b3a1740 10 client.26576 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.215+0000 7ff30b3a1740 10 client.26576 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.215+0000 7ff30b3a1740 20 client.26576 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.215+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/3518568699 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561eab481900 0x561eab483d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.215+0000 7ff30b3a1740 4 client.26576 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:37:06.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.215+0000 7ff30b3a1740 1 -- 172.21.15.70:0/3518568699 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561eab2eebe0 con 0x561eab481900 2024-04-01T01:37:06.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.215+0000 7ff305d62640 1 --2- 172.21.15.70:0/3518568699 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561eab481900 0x561eab483d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.215+0000 7ff305d62640 1 --2- 172.21.15.70:0/3518568699 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561eab481900 0x561eab483d20 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f06a7ac8fba021d8 server_cookie=5223a7e4fd3a120c in_seq=0 out_seq=0 2024-04-01T01:37:06.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.215+0000 7ff2e9ffb640 10 client.26576 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:06.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.217+0000 7ff2e9ffb640 1 -- 172.21.15.70:0/3518568699 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561eab2eebe0 con 0x561eab481900 2024-04-01T01:37:06.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.217+0000 7ff2e9ffb640 10 client.26576 handle_command_reply: tid=0 2024-04-01T01:37:06.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.217+0000 7ff30b3a1740 1 client.26576 shutdown 2024-04-01T01:37:06.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.217+0000 7ff30b3a1740 1 -- 172.21.15.70:0/3518568699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab404a10 msgr2=0x561eab413e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.217+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/3518568699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab404a10 0x561eab413e50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009f50 tx=0x7ff2f8013c10 comp rx=0 tx=0).stop 2024-04-01T01:37:06.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.217+0000 7ff30b3a1740 1 -- 172.21.15.70:0/3518568699 shutdown_connections 2024-04-01T01:37:06.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.217+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/3518568699 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561eab481900 0x561eab483d20 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.217+0000 7ff2e9ffb640 0 client.26576 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:06.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.217+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/3518568699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eab40a570 0x561eab415240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.217+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/3518568699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561eab405400 0x561eab414390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.217+0000 7ff30b3a1740 1 --2- 172.21.15.70:0/3518568699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561eab404a10 0x561eab413e50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.217+0000 7ff30b3a1740 1 -- 172.21.15.70:0/3518568699 >> 172.21.15.70:0/3518568699 conn(0x561eab3fff80 msgr2=0x561eab4031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:06.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.217+0000 7ff30b3a1740 1 -- 172.21.15.70:0/3518568699 shutdown_connections 2024-04-01T01:37:06.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.217+0000 7ff30b3a1740 1 -- 172.21.15.70:0/3518568699 wait complete. 2024-04-01T01:37:06.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.218+0000 7ff30b3a1740 20 client.26576 trim_cache size 0 max 16384 2024-04-01T01:37:06.218 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:37:06.218 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:37:06.218 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "433fd0ab-273a-46b0-8d1a-3eba8a5456ae", 2024-04-01T01:37:06.218 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:37:06.219 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:37:06.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.219+0000 7ff3087c7640 1 -- 172.21.15.70:0/55438268 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2ec0a2570 msgr2=0x7ff2ec0a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.219+0000 7ff3087c7640 1 --2- 172.21.15.70:0/55438268 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2ec0a2570 0x7ff2ec0a4a30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc010ca0 tx=0x7ff2fc008000 comp rx=0 tx=0).stop 2024-04-01T01:37:06.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.219+0000 7ff3087c7640 1 -- 172.21.15.70:0/55438268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff300102200 msgr2=0x7ff3001a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.219+0000 7ff3087c7640 1 --2- 172.21.15.70:0/55438268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff300102200 0x7ff3001a5500 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0036040 tx=0x7ff2f002c640 comp rx=0 tx=0).stop 2024-04-01T01:37:06.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.219+0000 7ff3087c7640 1 -- 172.21.15.70:0/55438268 shutdown_connections 2024-04-01T01:37:06.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.219+0000 7ff3087c7640 1 --2- 172.21.15.70:0/55438268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff300102200 0x7ff3001a5500 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.219+0000 7ff3087c7640 1 --2- 172.21.15.70:0/55438268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff300101840 0x7ff30019e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.219+0000 7ff3087c7640 1 --2- 172.21.15.70:0/55438268 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2ec0a2570 0x7ff2ec0a4a30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.219+0000 7ff3087c7640 1 --2- 172.21.15.70:0/55438268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff300100650 0x7ff30019e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.219+0000 7ff3087c7640 1 -- 172.21.15.70:0/55438268 >> 172.21.15.70:0/55438268 conn(0x7ff3000fbbe0 msgr2=0x7ff3000fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:06.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.219+0000 7ff3087c7640 1 -- 172.21.15.70:0/55438268 shutdown_connections 2024-04-01T01:37:06.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.219+0000 7ff3087c7640 1 -- 172.21.15.70:0/55438268 wait complete. 2024-04-01T01:37:06.235 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:37:06.235 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:37:06.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.441+0000 7f5381ffb640 1 -- 172.21.15.70:0/1586764180 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537802cb90 con 0x7f5384109940 2024-04-01T01:37:06.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.441+0000 7f5389402640 1 -- 172.21.15.70:0/1586764180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5384109940 msgr2=0x7f53841101d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.441+0000 7f5389402640 1 --2- 172.21.15.70:0/1586764180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5384109940 0x7f53841101d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5378009900 tx=0x7f537802bc50 comp rx=0 tx=0).stop 2024-04-01T01:37:06.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.442+0000 7f5389402640 1 -- 172.21.15.70:0/1586764180 shutdown_connections 2024-04-01T01:37:06.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.442+0000 7f5389402640 1 --2- 172.21.15.70:0/1586764180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5384109940 0x7f53841101d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.442+0000 7f5389402640 1 --2- 172.21.15.70:0/1586764180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5384108f80 0x7f5384109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.442+0000 7f5389402640 1 --2- 172.21.15.70:0/1586764180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5384107d90 0x7f53841081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.442+0000 7f5389402640 1 -- 172.21.15.70:0/1586764180 >> 172.21.15.70:0/1586764180 conn(0x7f5384076d20 msgr2=0x7f5384079160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:06.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.442+0000 7f5389402640 1 -- 172.21.15.70:0/1586764180 shutdown_connections 2024-04-01T01:37:06.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.442+0000 7f5389402640 1 -- 172.21.15.70:0/1586764180 wait complete. 2024-04-01T01:37:06.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.443+0000 7f5389402640 1 Processor -- start 2024-04-01T01:37:06.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.444+0000 7f5389402640 1 -- start start 2024-04-01T01:37:06.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.444+0000 7f5389402640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5384107d90 0x7f538407fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.444+0000 7f5389402640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5384108f80 0x7f5384080110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.444+0000 7f5389402640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5384109940 0x7f5384081150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.444+0000 7f5389402640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f538407a470 con 0x7f5384109940 2024-04-01T01:37:06.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.444+0000 7f5389402640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f538407a2f0 con 0x7f5384108f80 2024-04-01T01:37:06.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.444+0000 7f5389402640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f538407a5f0 con 0x7f5384107d90 2024-04-01T01:37:06.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.444+0000 7f5382ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5384107d90 0x7f538407fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.444+0000 7f53827fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5384108f80 0x7f5384080110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.444+0000 7f53827fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5384108f80 0x7f5384080110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:50786/0 (socket says 172.21.15.70:50786) 2024-04-01T01:37:06.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.444+0000 7f53827fc640 1 -- 172.21.15.70:0/450159256 learned_addr learned my addr 172.21.15.70:0/450159256 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:06.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.445+0000 7f53837fe640 1 --2- 172.21.15.70:0/450159256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5384109940 0x7f5384081150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.445+0000 7f53827fc640 1 -- 172.21.15.70:0/450159256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5384107d90 msgr2=0x7f538407fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.445+0000 7f53827fc640 1 --2- 172.21.15.70:0/450159256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5384107d90 0x7f538407fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.445+0000 7f53827fc640 1 -- 172.21.15.70:0/450159256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5384109940 msgr2=0x7f5384081150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.445+0000 7f53827fc640 1 --2- 172.21.15.70:0/450159256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5384109940 0x7f5384081150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.445+0000 7f53827fc640 1 -- 172.21.15.70:0/450159256 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53840817c0 con 0x7f5384108f80 2024-04-01T01:37:06.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.445+0000 7f5382ffd640 1 --2- 172.21.15.70:0/450159256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5384107d90 0x7f538407fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:37:06.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.445+0000 7f53827fc640 1 --2- 172.21.15.70:0/450159256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5384108f80 0x7f5384080110 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f537c00eab0 tx=0x7f537c00ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:06.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.445+0000 7f536bfff640 1 -- 172.21.15.70:0/450159256 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c00ce40 con 0x7f5384108f80 2024-04-01T01:37:06.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.446+0000 7f536bfff640 1 -- 172.21.15.70:0/450159256 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f537c011920 con 0x7f5384108f80 2024-04-01T01:37:06.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.447+0000 7f536bfff640 1 -- 172.21.15.70:0/450159256 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c010a00 con 0x7f5384108f80 2024-04-01T01:37:06.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.447+0000 7f5389402640 1 -- 172.21.15.70:0/450159256 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5384081ab0 con 0x7f5384108f80 2024-04-01T01:37:06.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.447+0000 7f5389402640 1 -- 172.21.15.70:0/450159256 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5384081fd0 con 0x7f5384108f80 2024-04-01T01:37:06.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.450+0000 7f536bfff640 1 -- 172.21.15.70:0/450159256 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f537c011470 con 0x7f5384108f80 2024-04-01T01:37:06.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.450+0000 7f538bfe6740 1 Processor -- start 2024-04-01T01:37:06.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.450+0000 7f538bfe6740 1 -- start start 2024-04-01T01:37:06.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.450+0000 7f538bfe6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033eba2b0 0x559033eba6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.450+0000 7f538bfe6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebaca0 0x559033ebf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f538bfe6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033ebfe10 0x559033ec22a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f538bfe6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559033e1f180 con 0x559033ebfe10 2024-04-01T01:37:06.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f538bfe6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x559033dc7b70 con 0x559033ebaca0 2024-04-01T01:37:06.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f538bfe6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559033e1b220 con 0x559033eba2b0 2024-04-01T01:37:06.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f53837fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033eba2b0 0x559033eba6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f53837fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033eba2b0 0x559033eba6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46102/0 (socket says 172.21.15.70:46102) 2024-04-01T01:37:06.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f53837fe640 1 -- 172.21.15.70:0/3735018617 learned_addr learned my addr 172.21.15.70:0/3735018617 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:06.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f5382ffd640 1 --2- 172.21.15.70:0/3735018617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebaca0 0x559033ebf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f536bfff640 1 --2- 172.21.15.70:0/450159256 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f536c0a2480 0x7f536c0a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f53827fc640 1 --2- 172.21.15.70:0/3735018617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033ebfe10 0x559033ec22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f5382ffd640 1 -- 172.21.15.70:0/3735018617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033eba2b0 msgr2=0x559033eba6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:06.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f5382ffd640 1 --2- 172.21.15.70:0/3735018617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033eba2b0 0x559033eba6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f5382ffd640 1 -- 172.21.15.70:0/3735018617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033ebfe10 msgr2=0x559033ec22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:06.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f5382ffd640 1 --2- 172.21.15.70:0/3735018617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033ebfe10 0x559033ec22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f5382ffd640 1 -- 172.21.15.70:0/3735018617 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559033ddb7c0 con 0x559033ebaca0 2024-04-01T01:37:06.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f53837fe640 1 --2- 172.21.15.70:0/3735018617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033eba2b0 0x559033eba6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:06.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f53837fe640 1 --2- 172.21.15.70:0/450159256 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f536c0a2480 0x7f536c0a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.451+0000 7f536bfff640 1 -- 172.21.15.70:0/450159256 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f537c0cec70 con 0x7f5384108f80 2024-04-01T01:37:06.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.452+0000 7f5382ffd640 1 --2- 172.21.15.70:0/3735018617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebaca0 0x559033ebf7a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5374009c00 tx=0x7f53740117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b87cf95f47d27d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:06.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.452+0000 7f53697fa640 1 -- 172.21.15.70:0/3735018617 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374005020 con 0x559033ebaca0 2024-04-01T01:37:06.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.452+0000 7f53697fa640 1 -- 172.21.15.70:0/3735018617 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53740047a0 con 0x559033ebaca0 2024-04-01T01:37:06.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.452+0000 7f53697fa640 1 -- 172.21.15.70:0/3735018617 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5374012740 con 0x559033ebaca0 2024-04-01T01:37:06.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.453+0000 7f538bfe6740 1 -- 172.21.15.70:0/3735018617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebaca0 msgr2=0x559033ebf7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.453+0000 7f538bfe6740 1 --2- 172.21.15.70:0/3735018617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebaca0 0x559033ebf7a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5374009c00 tx=0x7f53740117b0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.453+0000 7f53837fe640 1 --2- 172.21.15.70:0/450159256 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f536c0a2480 0x7f536c0a4940 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5378036040 tx=0x7f537802bc50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:06.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.453+0000 7f538bfe6740 1 -- 172.21.15.70:0/3735018617 shutdown_connections 2024-04-01T01:37:06.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.453+0000 7f538bfe6740 1 --2- 172.21.15.70:0/3735018617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033ebfe10 0x559033ec22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.453+0000 7f538bfe6740 1 --2- 172.21.15.70:0/3735018617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebaca0 0x559033ebf7a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.453+0000 7f538bfe6740 1 --2- 172.21.15.70:0/3735018617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033eba2b0 0x559033eba6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.453+0000 7f538bfe6740 1 -- 172.21.15.70:0/3735018617 >> 172.21.15.70:0/3735018617 conn(0x559033eb5820 msgr2=0x559033eb5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:06.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.453+0000 7f538bfe6740 1 -- 172.21.15.70:0/3735018617 shutdown_connections 2024-04-01T01:37:06.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.453+0000 7f538bfe6740 1 -- 172.21.15.70:0/3735018617 wait complete. 2024-04-01T01:37:06.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.454+0000 7f538bfe6740 1 -- start start 2024-04-01T01:37:06.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.454+0000 7f538bfe6740 1 Processor -- start 2024-04-01T01:37:06.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.454+0000 7f538bfe6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033eba2b0 0x559033ec4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.454+0000 7f538bfe6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033ebfe10 0x559033ec4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.454+0000 7f538bfe6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ec9ec0 0x559033ecc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.454+0000 7f538bfe6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559033ec2dc0 con 0x559033ebfe10 2024-04-01T01:37:06.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.454+0000 7f538bfe6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x559033dbd4a0 con 0x559033ec9ec0 2024-04-01T01:37:06.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.454+0000 7f538bfe6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559033ec2f40 con 0x559033eba2b0 2024-04-01T01:37:06.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.454+0000 7f538bfe6740 10 client.0 _ll_register_callbacks cb 0x559033dc0d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.455+0000 7f53827fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033ebfe10 0x559033ec4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.455+0000 7f53827fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033ebfe10 0x559033ec4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56946/0 (socket says 172.21.15.70:56946) 2024-04-01T01:37:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.455+0000 7f53827fc640 1 -- 172.21.15.70:0/2654110358 learned_addr learned my addr 172.21.15.70:0/2654110358 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.455+0000 7f53837fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ec9ec0 0x559033ecc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.455+0000 7f5382ffd640 1 --2- 172.21.15.70:0/2654110358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033eba2b0 0x559033ec4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.455+0000 7f5382ffd640 1 -- 172.21.15.70:0/2654110358 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ec9ec0 msgr2=0x559033ecc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.455+0000 7f5382ffd640 1 --2- 172.21.15.70:0/2654110358 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ec9ec0 0x559033ecc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.455+0000 7f5382ffd640 1 -- 172.21.15.70:0/2654110358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033ebfe10 msgr2=0x559033ec4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.455+0000 7f5382ffd640 1 --2- 172.21.15.70:0/2654110358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033ebfe10 0x559033ec4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.455+0000 7f5382ffd640 1 -- 172.21.15.70:0/2654110358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559033bfd7d0 con 0x559033eba2b0 2024-04-01T01:37:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.455+0000 7f5382ffd640 1 --2- 172.21.15.70:0/2654110358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033eba2b0 0x559033ec4950 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x559033ebb160 tx=0x7f53740098b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba7c3af3fa16b302 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.455+0000 7f5366ff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:37:06.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.456+0000 7f5366ff5640 1 -- 172.21.15.70:0/2654110358 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53740051b0 con 0x559033eba2b0 2024-04-01T01:37:06.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.456+0000 7f5366ff5640 1 -- 172.21.15.70:0/2654110358 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5374005350 con 0x559033eba2b0 2024-04-01T01:37:06.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.456+0000 7f538bfe6740 1 -- 172.21.15.70:0/2654110358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53740047a0 con 0x559033eba2b0 2024-04-01T01:37:06.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.456+0000 7f5366ff5640 1 -- 172.21.15.70:0/2654110358 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f53740124c0 con 0x559033eba2b0 2024-04-01T01:37:06.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.456+0000 7f538bfe6740 10 client.26482 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:06.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.456+0000 7f538bfe6740 1 -- 172.21.15.70:0/2654110358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559033b333f0 con 0x559033eba2b0 2024-04-01T01:37:06.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.457+0000 7f5366ff5640 1 -- 172.21.15.70:0/2654110358 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53740047a0 con 0x559033eba2b0 2024-04-01T01:37:06.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.457+0000 7f5366ff5640 1 -- 172.21.15.70:0/2654110358 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f537401a500 con 0x559033eba2b0 2024-04-01T01:37:06.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.457+0000 7f538bfe6740 10 client.26482 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:06.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.457+0000 7f538bfe6740 10 client.26482 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:06.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.457+0000 7f538bfe6740 20 client.26482 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:06.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.457+0000 7f538bfe6740 1 --2- 172.21.15.70:0/2654110358 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559033ee17b0 0x559033ee3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.457+0000 7f538bfe6740 4 client.26482 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:06.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.457+0000 7f538bfe6740 1 -- 172.21.15.70:0/2654110358 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559033e2c6c0 con 0x559033ee17b0 2024-04-01T01:37:06.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.458+0000 7f53827fc640 1 --2- 172.21.15.70:0/2654110358 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559033ee17b0 0x559033ee3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.459+0000 7f53827fc640 1 --2- 172.21.15.70:0/2654110358 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559033ee17b0 0x559033ee3ba0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a54671e4d2bcd92 server_cookie=5e1b3ce6a40b36e7 in_seq=0 out_seq=0 2024-04-01T01:37:06.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.459+0000 7f5366ff5640 10 client.26482 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:06.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.460+0000 7f5366ff5640 1 -- 172.21.15.70:0/2654110358 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559033e2c6c0 con 0x559033ee17b0 2024-04-01T01:37:06.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.460+0000 7f5366ff5640 10 client.26482 handle_command_reply: tid=0 2024-04-01T01:37:06.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.460+0000 7f538bfe6740 1 client.26482 shutdown 2024-04-01T01:37:06.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.460+0000 7f538bfe6740 1 -- 172.21.15.70:0/2654110358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033eba2b0 msgr2=0x559033ec4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.460+0000 7f538bfe6740 1 --2- 172.21.15.70:0/2654110358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033eba2b0 0x559033ec4950 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x559033ebb160 tx=0x7f53740098b0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.460+0000 7f538bfe6740 1 -- 172.21.15.70:0/2654110358 shutdown_connections 2024-04-01T01:37:06.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.460+0000 7f538bfe6740 1 --2- 172.21.15.70:0/2654110358 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559033ee17b0 0x559033ee3ba0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.460+0000 7f538bfe6740 1 --2- 172.21.15.70:0/2654110358 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ec9ec0 0x559033ecc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.460+0000 7f538bfe6740 1 --2- 172.21.15.70:0/2654110358 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033ebfe10 0x559033ec4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.460+0000 7f538bfe6740 1 --2- 172.21.15.70:0/2654110358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033eba2b0 0x559033ec4950 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.460+0000 7f538bfe6740 1 -- 172.21.15.70:0/2654110358 >> 172.21.15.70:0/2654110358 conn(0x559033eb5820 msgr2=0x559033eb4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:06.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.460+0000 7f5366ff5640 0 client.26482 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:06.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.460+0000 7f538bfe6740 1 -- 172.21.15.70:0/2654110358 shutdown_connections 2024-04-01T01:37:06.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.460+0000 7f538bfe6740 1 -- 172.21.15.70:0/2654110358 wait complete. 2024-04-01T01:37:06.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.461+0000 7f538bfe6740 20 client.26482 trim_cache size 0 max 16384 2024-04-01T01:37:06.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.497+0000 7f538bfe6740 1 Processor -- start 2024-04-01T01:37:06.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.497+0000 7f538bfe6740 1 -- start start 2024-04-01T01:37:06.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.498+0000 7f538bfe6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033eba2b0 0x559033ee83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.498+0000 7f538bfe6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebfe10 0x559033ee88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.499+0000 7f538bfe6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033ec9ec0 0x559033ee8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.499+0000 7f538bfe6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f53841b06d0 con 0x559033eba2b0 2024-04-01T01:37:06.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.499+0000 7f538bfe6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f53841b2210 con 0x559033ebfe10 2024-04-01T01:37:06.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.499+0000 7f538bfe6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f53841b0da0 con 0x559033ec9ec0 2024-04-01T01:37:06.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.500+0000 7f5382ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033eba2b0 0x559033ee83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.500+0000 7f5382ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033eba2b0 0x559033ee83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56956/0 (socket says 172.21.15.70:56956) 2024-04-01T01:37:06.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.500+0000 7f5382ffd640 1 -- 172.21.15.70:0/1913431711 learned_addr learned my addr 172.21.15.70:0/1913431711 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:06.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.500+0000 7f53837fe640 1 --2- 172.21.15.70:0/1913431711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033ec9ec0 0x559033ee8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.500+0000 7f53827fc640 1 --2- 172.21.15.70:0/1913431711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebfe10 0x559033ee88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.500+0000 7f53827fc640 1 -- 172.21.15.70:0/1913431711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033ec9ec0 msgr2=0x559033ee8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:06.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.500+0000 7f53827fc640 1 --2- 172.21.15.70:0/1913431711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033ec9ec0 0x559033ee8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.500+0000 7f53827fc640 1 -- 172.21.15.70:0/1913431711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033eba2b0 msgr2=0x559033ee83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:06.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.500+0000 7f53827fc640 1 --2- 172.21.15.70:0/1913431711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033eba2b0 0x559033ee83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.500+0000 7f53827fc640 1 -- 172.21.15.70:0/1913431711 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559033af30f0 con 0x559033ebfe10 2024-04-01T01:37:06.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.501+0000 7f53827fc640 1 --2- 172.21.15.70:0/1913431711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebfe10 0x559033ee88e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f537c09b000 tx=0x7f537c066650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e4c8b4b2d68a987 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:06.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.502+0000 7f5368ff9640 1 -- 172.21.15.70:0/1913431711 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c09ba00 con 0x559033ebfe10 2024-04-01T01:37:06.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.502+0000 7f5368ff9640 1 -- 172.21.15.70:0/1913431711 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f537c09b300 con 0x559033ebfe10 2024-04-01T01:37:06.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.502+0000 7f5368ff9640 1 -- 172.21.15.70:0/1913431711 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c09f120 con 0x559033ebfe10 2024-04-01T01:37:06.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.503+0000 7f538bfe6740 1 -- 172.21.15.70:0/1913431711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebfe10 msgr2=0x559033ee88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.503+0000 7f538bfe6740 1 --2- 172.21.15.70:0/1913431711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebfe10 0x559033ee88e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f537c09b000 tx=0x7f537c066650 comp rx=0 tx=0).stop 2024-04-01T01:37:06.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.503+0000 7f538bfe6740 1 -- 172.21.15.70:0/1913431711 shutdown_connections 2024-04-01T01:37:06.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.503+0000 7f538bfe6740 1 --2- 172.21.15.70:0/1913431711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033ec9ec0 0x559033ee8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.503+0000 7f538bfe6740 1 --2- 172.21.15.70:0/1913431711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebfe10 0x559033ee88e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.503+0000 7f538bfe6740 1 --2- 172.21.15.70:0/1913431711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033eba2b0 0x559033ee83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.503+0000 7f538bfe6740 1 -- 172.21.15.70:0/1913431711 >> 172.21.15.70:0/1913431711 conn(0x559033eb5820 msgr2=0x559033eb4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:06.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.504+0000 7f538bfe6740 1 -- 172.21.15.70:0/1913431711 shutdown_connections 2024-04-01T01:37:06.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.504+0000 7f538bfe6740 1 -- 172.21.15.70:0/1913431711 wait complete. 2024-04-01T01:37:06.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.504+0000 7f538bfe6740 1 -- start start 2024-04-01T01:37:06.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.504+0000 7f538bfe6740 1 Processor -- start 2024-04-01T01:37:06.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.505+0000 7f538bfe6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033eba2b0 0x559033ef4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.505+0000 7f538bfe6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebfe10 0x559033ef4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.505+0000 7f538bfe6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033ec9ec0 0x559033f45a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.505+0000 7f538bfe6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f53841b1b40 con 0x559033eba2b0 2024-04-01T01:37:06.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.505+0000 7f538bfe6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f53841b1470 con 0x559033ebfe10 2024-04-01T01:37:06.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.505+0000 7f538bfe6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5384072eb0 con 0x559033ec9ec0 2024-04-01T01:37:06.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.505+0000 7f538bfe6740 10 client.0 _ll_register_callbacks cb 0x7f5374005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:06.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.505+0000 7f53827fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebfe10 0x559033ef4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.505+0000 7f53827fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebfe10 0x559033ef4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:50830/0 (socket says 172.21.15.70:50830) 2024-04-01T01:37:06.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.505+0000 7f53827fc640 1 -- 172.21.15.70:0/2002089315 learned_addr learned my addr 172.21.15.70:0/2002089315 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:06.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.505+0000 7f5382ffd640 1 --2- 172.21.15.70:0/2002089315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033eba2b0 0x559033ef4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.506+0000 7f5382ffd640 1 -- 172.21.15.70:0/2002089315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033ec9ec0 msgr2=0x559033f45a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:06.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.506+0000 7f5382ffd640 1 --2- 172.21.15.70:0/2002089315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033ec9ec0 0x559033f45a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.506+0000 7f5382ffd640 1 -- 172.21.15.70:0/2002089315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebfe10 msgr2=0x559033ef4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:06.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.506+0000 7f5382ffd640 1 --2- 172.21.15.70:0/2002089315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebfe10 0x559033ef4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.506+0000 7f5382ffd640 1 -- 172.21.15.70:0/2002089315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559033f46090 con 0x559033eba2b0 2024-04-01T01:37:06.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.507+0000 7f5382ffd640 1 --2- 172.21.15.70:0/2002089315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033eba2b0 0x559033ef4330 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5374012490 tx=0x7f5374004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ea6b34eaa0cd1b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:06.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.507+0000 7f5366ff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:37:06.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.507+0000 7f5366ff5640 1 -- 172.21.15.70:0/2002089315 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537403b060 con 0x559033eba2b0 2024-04-01T01:37:06.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.507+0000 7f5366ff5640 1 -- 172.21.15.70:0/2002089315 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f537401a400 con 0x559033eba2b0 2024-04-01T01:37:06.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.508+0000 7f538bfe6740 1 -- 172.21.15.70:0/2002089315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f537c09b300 con 0x559033eba2b0 2024-04-01T01:37:06.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.508+0000 7f5366ff5640 1 -- 172.21.15.70:0/2002089315 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f53740092a0 con 0x559033eba2b0 2024-04-01T01:37:06.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.508+0000 7f538bfe6740 10 client.16734 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:06.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.508+0000 7f538bfe6740 1 -- 172.21.15.70:0/2002089315 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559033f473a0 con 0x559033eba2b0 2024-04-01T01:37:06.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.509+0000 7f5366ff5640 1 -- 172.21.15.70:0/2002089315 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f537c09b300 con 0x559033eba2b0 2024-04-01T01:37:06.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.509+0000 7f5366ff5640 1 -- 172.21.15.70:0/2002089315 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f53740129a0 con 0x559033eba2b0 2024-04-01T01:37:06.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.511+0000 7f538bfe6740 10 client.16734 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:06.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.511+0000 7f538bfe6740 10 client.16734 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:06.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.511+0000 7f538bfe6740 20 client.16734 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:06.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.511+0000 7f538bfe6740 1 --2- 172.21.15.70:0/2002089315 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559033f47cd0 0x559033f4a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:06.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.511+0000 7f538bfe6740 4 client.16734 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:37:06.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.511+0000 7f538bfe6740 1 -- 172.21.15.70:0/2002089315 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559033e26a10 con 0x559033f47cd0 2024-04-01T01:37:06.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.512+0000 7f53827fc640 1 --2- 172.21.15.70:0/2002089315 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559033f47cd0 0x559033f4a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:06.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.517+0000 7f53827fc640 1 --2- 172.21.15.70:0/2002089315 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559033f47cd0 0x559033f4a0f0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5f015a6f27e85ae server_cookie=3ada5356bbc25be in_seq=0 out_seq=0 2024-04-01T01:37:06.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.517+0000 7f5366ff5640 10 client.16734 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:06.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.517+0000 7f5366ff5640 1 -- 172.21.15.70:0/2002089315 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559033e26a10 con 0x559033f47cd0 2024-04-01T01:37:06.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.517+0000 7f5366ff5640 10 client.16734 handle_command_reply: tid=0 2024-04-01T01:37:06.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.517+0000 7f538bfe6740 1 client.16734 shutdown 2024-04-01T01:37:06.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.517+0000 7f538bfe6740 1 -- 172.21.15.70:0/2002089315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033eba2b0 msgr2=0x559033ef4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.518+0000 7f538bfe6740 1 --2- 172.21.15.70:0/2002089315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033eba2b0 0x559033ef4330 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5374012490 tx=0x7f5374004e60 comp rx=0 tx=0).stop 2024-04-01T01:37:06.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.518+0000 7f538bfe6740 1 -- 172.21.15.70:0/2002089315 shutdown_connections 2024-04-01T01:37:06.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.518+0000 7f538bfe6740 1 --2- 172.21.15.70:0/2002089315 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559033f47cd0 0x559033f4a0f0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.518+0000 7f538bfe6740 1 --2- 172.21.15.70:0/2002089315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559033ec9ec0 0x559033f45a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.518+0000 7f538bfe6740 1 --2- 172.21.15.70:0/2002089315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559033ebfe10 0x559033ef4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.518+0000 7f538bfe6740 1 --2- 172.21.15.70:0/2002089315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559033eba2b0 0x559033ef4330 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.518+0000 7f538bfe6740 1 -- 172.21.15.70:0/2002089315 >> 172.21.15.70:0/2002089315 conn(0x559033eb5820 msgr2=0x559033eb8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:06.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.518+0000 7f5366ff5640 0 client.16734 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:06.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.518+0000 7f538bfe6740 1 -- 172.21.15.70:0/2002089315 shutdown_connections 2024-04-01T01:37:06.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.518+0000 7f538bfe6740 1 -- 172.21.15.70:0/2002089315 wait complete. 2024-04-01T01:37:06.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.519+0000 7f538bfe6740 20 client.16734 trim_cache size 0 max 16384 2024-04-01T01:37:06.519 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:37:06.519 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:37:06.519 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:37:06.519 INFO:teuthology.orchestra.run.smithi070.stdout: "433fd0ab-273a-46b0-8d1a-3eba8a5456ae": { 2024-04-01T01:37:06.519 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:37:06.519 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "433fd0ab-273a-46b0-8d1a-3eba8a5456ae", 2024-04-01T01:37:06.520 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:37:06.520 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:37:06.520 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:37:06.520 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:37:06.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.520+0000 7f5389402640 1 -- 172.21.15.70:0/450159256 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f536c0a2480 msgr2=0x7f536c0a4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.520+0000 7f5389402640 1 --2- 172.21.15.70:0/450159256 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f536c0a2480 0x7f536c0a4940 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5378036040 tx=0x7f537802bc50 comp rx=0 tx=0).stop 2024-04-01T01:37:06.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.520+0000 7f5389402640 1 -- 172.21.15.70:0/450159256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5384108f80 msgr2=0x7f5384080110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:06.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.520+0000 7f5389402640 1 --2- 172.21.15.70:0/450159256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5384108f80 0x7f5384080110 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f537c00eab0 tx=0x7f537c00ef70 comp rx=0 tx=0).stop 2024-04-01T01:37:06.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.520+0000 7f5389402640 1 -- 172.21.15.70:0/450159256 shutdown_connections 2024-04-01T01:37:06.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.520+0000 7f5389402640 1 --2- 172.21.15.70:0/450159256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5384109940 0x7f5384081150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.520+0000 7f5389402640 1 --2- 172.21.15.70:0/450159256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5384108f80 0x7f5384080110 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.520+0000 7f5389402640 1 --2- 172.21.15.70:0/450159256 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f536c0a2480 0x7f536c0a4940 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.520+0000 7f5389402640 1 --2- 172.21.15.70:0/450159256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5384107d90 0x7f538407fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:06.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.520+0000 7f5389402640 1 -- 172.21.15.70:0/450159256 >> 172.21.15.70:0/450159256 conn(0x7f5384076d20 msgr2=0x7f538410c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:06.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.521+0000 7f5389402640 1 -- 172.21.15.70:0/450159256 shutdown_connections 2024-04-01T01:37:06.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:06.521+0000 7f5389402640 1 -- 172.21.15.70:0/450159256 wait complete. 2024-04-01T01:37:06.543 INFO:tasks.cephfs.filesystem:scrub status for tag:433fd0ab-273a-46b0-8d1a-3eba8a5456ae - {'path': '/', 'tag': '433fd0ab-273a-46b0-8d1a-3eba8a5456ae', 'options': 'recursive,force'} 2024-04-01T01:37:06.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 140.0 s, 997.2 tps, lat 100.207 ms stddev 32.946 2024-04-01T01:37:07.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:07 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:37:07.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:07 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:37:07.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:07 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:37:07.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:07 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:37:07.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:07 smithi189 ceph-mon[35753]: pgmap v467: 129 pgs: 129 active+clean; 1011 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-04-01T01:37:08.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:07 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:37:08.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:07 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:37:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:07 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:37:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:07 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:37:08.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:07 smithi070 ceph-mon[32179]: pgmap v467: 129 pgs: 129 active+clean; 1011 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-04-01T01:37:08.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:07 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:37:08.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:07 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:37:08.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:07 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:37:08.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:07 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:37:08.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:07 smithi081 ceph-mon[35142]: pgmap v467: 129 pgs: 129 active+clean; 1011 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 11 MiB/s wr, 1.29k op/s 2024-04-01T01:37:09.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:09 smithi189 ceph-mon[35753]: pgmap v468: 129 pgs: 129 active+clean; 1011 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 9.8 MiB/s wr, 1.21k op/s 2024-04-01T01:37:10.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:09 smithi070 ceph-mon[32179]: pgmap v468: 129 pgs: 129 active+clean; 1011 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 9.8 MiB/s wr, 1.21k op/s 2024-04-01T01:37:10.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:09 smithi081 ceph-mon[35142]: pgmap v468: 129 pgs: 129 active+clean; 1011 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 9.8 MiB/s wr, 1.21k op/s 2024-04-01T01:37:11.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 145.0 s, 1077.6 tps, lat 92.825 ms stddev 48.601 2024-04-01T01:37:11.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:11 smithi189 ceph-mon[35753]: pgmap v469: 129 pgs: 129 active+clean; 1011 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 8.9 MiB/s wr, 1.11k op/s 2024-04-01T01:37:11.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:11 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:37:12.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:11 smithi070 ceph-mon[32179]: pgmap v469: 129 pgs: 129 active+clean; 1011 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 8.9 MiB/s wr, 1.11k op/s 2024-04-01T01:37:12.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:11 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:37:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:11 smithi081 ceph-mon[35142]: pgmap v469: 129 pgs: 129 active+clean; 1011 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 8.9 MiB/s wr, 1.11k op/s 2024-04-01T01:37:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:11 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:37:13.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:13 smithi189 ceph-mon[35753]: pgmap v470: 129 pgs: 129 active+clean; 1011 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 10 MiB/s wr, 1.27k op/s 2024-04-01T01:37:14.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:13 smithi070 ceph-mon[32179]: pgmap v470: 129 pgs: 129 active+clean; 1011 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 10 MiB/s wr, 1.27k op/s 2024-04-01T01:37:14.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:13 smithi081 ceph-mon[35142]: pgmap v470: 129 pgs: 129 active+clean; 1011 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 10 MiB/s wr, 1.27k op/s 2024-04-01T01:37:16.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:15 smithi070 ceph-mon[32179]: pgmap v471: 129 pgs: 129 active+clean; 1011 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 8.5 MiB/s wr, 1.09k op/s 2024-04-01T01:37:16.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:15 smithi081 ceph-mon[35142]: pgmap v471: 129 pgs: 129 active+clean; 1011 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 8.5 MiB/s wr, 1.09k op/s 2024-04-01T01:37:16.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:15 smithi189 ceph-mon[35753]: pgmap v471: 129 pgs: 129 active+clean; 1011 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 8.5 MiB/s wr, 1.09k op/s 2024-04-01T01:37:16.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 150.0 s, 1013.6 tps, lat 98.335 ms stddev 97.344 2024-04-01T01:37:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.912+0000 7f24b29e5640 1 -- 172.21.15.70:0/2459869447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24ac072a00 msgr2=0x7f24ac072e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:16.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.912+0000 7f24b29e5640 1 --2- 172.21.15.70:0/2459869447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24ac072a00 0x7f24ac072e20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009900 tx=0x7f24a802bc40 comp rx=0 tx=0).stop 2024-04-01T01:37:16.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.913+0000 7f24b29e5640 1 -- 172.21.15.70:0/2459869447 shutdown_connections 2024-04-01T01:37:16.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.913+0000 7f24b29e5640 1 --2- 172.21.15.70:0/2459869447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ac109440 0x7f24ac10e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.913+0000 7f24b29e5640 1 --2- 172.21.15.70:0/2459869447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24ac073360 0x7f24ac108d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.913+0000 7f24b29e5640 1 --2- 172.21.15.70:0/2459869447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24ac072a00 0x7f24ac072e20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.913+0000 7f24b29e5640 1 -- 172.21.15.70:0/2459869447 >> 172.21.15.70:0/2459869447 conn(0x7f24ac06e5c0 msgr2=0x7f24ac070a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.913+0000 7f24b29e5640 1 -- 172.21.15.70:0/2459869447 shutdown_connections 2024-04-01T01:37:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.913+0000 7f24b29e5640 1 -- 172.21.15.70:0/2459869447 wait complete. 2024-04-01T01:37:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.914+0000 7f24b29e5640 1 Processor -- start 2024-04-01T01:37:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.914+0000 7f24b29e5640 1 -- start start 2024-04-01T01:37:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.914+0000 7f24b29e5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24ac072a00 0x7f24ac1a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.914+0000 7f24b29e5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ac073360 0x7f24ac1a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.914+0000 7f24b29e5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24ac109440 0x7f24ac1ad980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.914+0000 7f24b29e5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac11bf30 con 0x7f24ac072a00 2024-04-01T01:37:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.914+0000 7f24b29e5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac11bdb0 con 0x7f24ac109440 2024-04-01T01:37:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.914+0000 7f24b29e5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac11c0b0 con 0x7f24ac073360 2024-04-01T01:37:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.915+0000 7f24b11e2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ac073360 0x7f24ac1a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.915+0000 7f24b11e2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ac073360 0x7f24ac1a6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52630/0 (socket says 172.21.15.70:52630) 2024-04-01T01:37:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.915+0000 7f24b11e2640 1 -- 172.21.15.70:0/3846867450 learned_addr learned my addr 172.21.15.70:0/3846867450 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.915+0000 7f24b19e3640 1 --2- 172.21.15.70:0/3846867450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24ac072a00 0x7f24ac1a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.915+0000 7f24b21e4640 1 --2- 172.21.15.70:0/3846867450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24ac109440 0x7f24ac1ad980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.915+0000 7f24b21e4640 1 -- 172.21.15.70:0/3846867450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ac073360 msgr2=0x7f24ac1a6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.915+0000 7f24b21e4640 1 --2- 172.21.15.70:0/3846867450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ac073360 0x7f24ac1a6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.915+0000 7f24b21e4640 1 -- 172.21.15.70:0/3846867450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24ac072a00 msgr2=0x7f24ac1a63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.915+0000 7f24b21e4640 1 --2- 172.21.15.70:0/3846867450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24ac072a00 0x7f24ac1a63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.915+0000 7f24b21e4640 1 -- 172.21.15.70:0/3846867450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24ac1adff0 con 0x7f24ac109440 2024-04-01T01:37:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.915+0000 7f24b11e2640 1 --2- 172.21.15.70:0/3846867450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ac073360 0x7f24ac1a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.915+0000 7f24b21e4640 1 --2- 172.21.15.70:0/3846867450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24ac109440 0x7f24ac1ad980 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f24a400e940 tx=0x7f24a400ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.916+0000 7f24a2ffd640 1 -- 172.21.15.70:0/3846867450 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a4004d20 con 0x7f24ac109440 2024-04-01T01:37:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.916+0000 7f24a2ffd640 1 -- 172.21.15.70:0/3846867450 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24a4005020 con 0x7f24ac109440 2024-04-01T01:37:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.916+0000 7f24b29e5640 1 -- 172.21.15.70:0/3846867450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24ac1ae2e0 con 0x7f24ac109440 2024-04-01T01:37:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.916+0000 7f24b29e5640 1 -- 172.21.15.70:0/3846867450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24ac1ae830 con 0x7f24ac109440 2024-04-01T01:37:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.916+0000 7f24a2ffd640 1 -- 172.21.15.70:0/3846867450 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a40116d0 con 0x7f24ac109440 2024-04-01T01:37:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.920+0000 7f24b682f740 1 Processor -- start 2024-04-01T01:37:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.921+0000 7f24b682f740 1 -- start start 2024-04-01T01:37:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.921+0000 7f24a2ffd640 1 -- 172.21.15.70:0/3846867450 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f24a400cb20 con 0x7f24ac109440 2024-04-01T01:37:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.921+0000 7f24b682f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a48536eae0 0x55a48536ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.921+0000 7f24b682f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a48536f4d0 0x55a485373fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.921+0000 7f24b682f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a485374640 0x55a485376ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.921+0000 7f24b682f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a4852d28f0 con 0x55a48536eae0 2024-04-01T01:37:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.921+0000 7f24b682f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a48527b8d0 con 0x55a485374640 2024-04-01T01:37:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.921+0000 7f24b682f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a4852ce990 con 0x55a48536f4d0 2024-04-01T01:37:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.922+0000 7f24b11e2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a485374640 0x55a485376ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.922+0000 7f24b19e3640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a48536f4d0 0x55a485373fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.922+0000 7f24b11e2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a485374640 0x55a485376ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49388/0 (socket says 172.21.15.70:49388) 2024-04-01T01:37:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.922+0000 7f24b11e2640 1 -- 172.21.15.70:0/3133659092 learned_addr learned my addr 172.21.15.70:0/3133659092 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.922+0000 7f24b11e2640 1 --2- 172.21.15.70:0/3133659092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a48536eae0 0x55a48536ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.922+0000 7f24a2ffd640 1 --2- 172.21.15.70:0/3846867450 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24980a2660 0x7f24980a4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.922+0000 7f24b11e2640 1 -- 172.21.15.70:0/3133659092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a48536f4d0 msgr2=0x55a485373fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.922+0000 7f24b11e2640 1 --2- 172.21.15.70:0/3133659092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a48536f4d0 0x55a485373fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.922+0000 7f24b11e2640 1 -- 172.21.15.70:0/3133659092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a48536eae0 msgr2=0x55a48536ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.922+0000 7f24b11e2640 1 --2- 172.21.15.70:0/3133659092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a48536eae0 0x55a48536ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.922+0000 7f24b11e2640 1 -- 172.21.15.70:0/3133659092 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a484fe7210 con 0x55a485374640 2024-04-01T01:37:16.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.923+0000 7f24a2ffd640 1 -- 172.21.15.70:0/3846867450 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f24a40cedc0 con 0x7f24ac109440 2024-04-01T01:37:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.923+0000 7f24b21e4640 1 --2- 172.21.15.70:0/3846867450 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24980a2660 0x7f24980a4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.923+0000 7f24b11e2640 1 --2- 172.21.15.70:0/3133659092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a485374640 0x55a485376ad0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f249c0097f0 tx=0x7f249c009c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8dff3072a1c338f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.924+0000 7f2497fff640 1 -- 172.21.15.70:0/3133659092 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249c00fc50 con 0x55a485374640 2024-04-01T01:37:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.924+0000 7f2497fff640 1 -- 172.21.15.70:0/3133659092 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f249c00fdf0 con 0x55a485374640 2024-04-01T01:37:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.924+0000 7f2497fff640 1 -- 172.21.15.70:0/3133659092 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249c004db0 con 0x55a485374640 2024-04-01T01:37:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.924+0000 7f24b21e4640 1 --2- 172.21.15.70:0/3846867450 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24980a2660 0x7f24980a4b20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f24a40cf0a0 tx=0x7f24a40bb040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.925+0000 7f24b682f740 1 -- 172.21.15.70:0/3133659092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a485374640 msgr2=0x55a485376ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.925+0000 7f24b682f740 1 --2- 172.21.15.70:0/3133659092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a485374640 0x55a485376ad0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f249c0097f0 tx=0x7f249c009c10 comp rx=0 tx=0).stop 2024-04-01T01:37:16.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.925+0000 7f24b682f740 1 -- 172.21.15.70:0/3133659092 shutdown_connections 2024-04-01T01:37:16.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.925+0000 7f24b682f740 1 --2- 172.21.15.70:0/3133659092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a485374640 0x55a485376ad0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.925+0000 7f24b682f740 1 --2- 172.21.15.70:0/3133659092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a48536f4d0 0x55a485373fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.925+0000 7f24b682f740 1 --2- 172.21.15.70:0/3133659092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a48536eae0 0x55a48536ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.925+0000 7f24b682f740 1 -- 172.21.15.70:0/3133659092 >> 172.21.15.70:0/3133659092 conn(0x55a48536a050 msgr2=0x55a48536a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.926+0000 7f24b682f740 1 -- 172.21.15.70:0/3133659092 shutdown_connections 2024-04-01T01:37:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.926+0000 7f24b682f740 1 -- 172.21.15.70:0/3133659092 wait complete. 2024-04-01T01:37:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.926+0000 7f24b682f740 1 -- start start 2024-04-01T01:37:16.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.926+0000 7f24b682f740 1 Processor -- start 2024-04-01T01:37:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b682f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a48536eae0 0x55a485378dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b682f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a48536f4d0 0x55a485379310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b682f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a485374640 0x55a4853802a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b682f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a4853775f0 con 0x55a48536eae0 2024-04-01T01:37:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b682f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a485270f80 con 0x55a485374640 2024-04-01T01:37:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b682f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a485377770 con 0x55a48536f4d0 2024-04-01T01:37:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b682f740 10 client.0 _ll_register_callbacks cb 0x55a4852c9290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b11e2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a485374640 0x55a4853802a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b11e2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a485374640 0x55a4853802a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49390/0 (socket says 172.21.15.70:49390) 2024-04-01T01:37:16.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b11e2640 1 -- 172.21.15.70:0/749138135 learned_addr learned my addr 172.21.15.70:0/749138135 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b19e3640 1 --2- 172.21.15.70:0/749138135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a48536f4d0 0x55a485379310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b11e2640 1 --2- 172.21.15.70:0/749138135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a48536eae0 0x55a485378dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b19e3640 1 -- 172.21.15.70:0/749138135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a485374640 msgr2=0x55a4853802a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b19e3640 1 --2- 172.21.15.70:0/749138135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a485374640 0x55a4853802a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b19e3640 1 -- 172.21.15.70:0/749138135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a48536eae0 msgr2=0x55a485378dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b19e3640 1 --2- 172.21.15.70:0/749138135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a48536eae0 0x55a485378dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.927+0000 7f24b19e3640 1 -- 172.21.15.70:0/749138135 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4850b3620 con 0x55a48536f4d0 2024-04-01T01:37:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.928+0000 7f24b11e2640 1 --2- 172.21.15.70:0/749138135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a485374640 0x55a4853802a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.928+0000 7f24b19e3640 1 --2- 172.21.15.70:0/749138135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a48536f4d0 0x55a485379310 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f24a8036040 tx=0x7f24a80045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d140a7a0127abb61 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.928+0000 7f24957fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:37:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.928+0000 7f24957fa640 1 -- 172.21.15.70:0/749138135 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a8005250 con 0x55a48536f4d0 2024-04-01T01:37:16.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.928+0000 7f24b682f740 1 -- 172.21.15.70:0/749138135 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f249c00fdf0 con 0x55a48536f4d0 2024-04-01T01:37:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.928+0000 7f24957fa640 1 -- 172.21.15.70:0/749138135 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f24a80053f0 con 0x55a48536f4d0 2024-04-01T01:37:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.929+0000 7f24957fa640 1 -- 172.21.15.70:0/749138135 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f24a8052050 con 0x55a48536f4d0 2024-04-01T01:37:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.929+0000 7f24b682f740 10 client.26497 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:16.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.929+0000 7f24b682f740 1 -- 172.21.15.70:0/749138135 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a485383360 con 0x55a48536f4d0 2024-04-01T01:37:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.929+0000 7f24957fa640 1 -- 172.21.15.70:0/749138135 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f249c00fdf0 con 0x55a48536f4d0 2024-04-01T01:37:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.929+0000 7f24957fa640 1 -- 172.21.15.70:0/749138135 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f24a80035f0 con 0x55a48536f4d0 2024-04-01T01:37:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.929+0000 7f24b682f740 10 client.26497 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:16.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.929+0000 7f24b682f740 10 client.26497 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.929+0000 7f24b682f740 20 client.26497 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.929+0000 7f24b682f740 1 --2- 172.21.15.70:0/749138135 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a485384990 0x55a485386dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.929+0000 7f24b682f740 4 client.26497 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.929+0000 7f24b682f740 1 -- 172.21.15.70:0/749138135 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4852e6120 con 0x55a485384990 2024-04-01T01:37:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.930+0000 7f24b11e2640 1 --2- 172.21.15.70:0/749138135 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a485384990 0x55a485386dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.931+0000 7f24b11e2640 1 --2- 172.21.15.70:0/749138135 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a485384990 0x55a485386dc0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27d8798b4ac5ef59 server_cookie=3536ff7a06a20126 in_seq=0 out_seq=0 2024-04-01T01:37:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.931+0000 7f24957fa640 10 client.26497 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.932+0000 7f24957fa640 1 -- 172.21.15.70:0/749138135 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a4852e6120 con 0x55a485384990 2024-04-01T01:37:16.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.932+0000 7f24957fa640 10 client.26497 handle_command_reply: tid=0 2024-04-01T01:37:16.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.934+0000 7f24b682f740 1 client.26497 shutdown 2024-04-01T01:37:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.935+0000 7f24b682f740 1 -- 172.21.15.70:0/749138135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a48536f4d0 msgr2=0x55a485379310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.935+0000 7f24b682f740 1 --2- 172.21.15.70:0/749138135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a48536f4d0 0x55a485379310 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f24a8036040 tx=0x7f24a80045e0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.935+0000 7f24b682f740 1 -- 172.21.15.70:0/749138135 shutdown_connections 2024-04-01T01:37:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.935+0000 7f24b682f740 1 --2- 172.21.15.70:0/749138135 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a485384990 0x55a485386dc0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.935+0000 7f24b682f740 1 --2- 172.21.15.70:0/749138135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a485374640 0x55a4853802a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.935+0000 7f24b682f740 1 --2- 172.21.15.70:0/749138135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a48536f4d0 0x55a485379310 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.935+0000 7f24b682f740 1 --2- 172.21.15.70:0/749138135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a48536eae0 0x55a485378dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.935+0000 7f24957fa640 0 client.26497 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.935+0000 7f24b682f740 1 -- 172.21.15.70:0/749138135 >> 172.21.15.70:0/749138135 conn(0x55a48536a050 msgr2=0x55a485369190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.935+0000 7f24b682f740 1 -- 172.21.15.70:0/749138135 shutdown_connections 2024-04-01T01:37:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.935+0000 7f24b682f740 1 -- 172.21.15.70:0/749138135 wait complete. 2024-04-01T01:37:16.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.935+0000 7f24b682f740 20 client.26497 trim_cache size 0 max 16384 2024-04-01T01:37:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.966+0000 7f24b682f740 1 Processor -- start 2024-04-01T01:37:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.966+0000 7f24b682f740 1 -- start start 2024-04-01T01:37:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.967+0000 7f24b682f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a48536eae0 0x55a485380950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.967+0000 7f24b682f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a48536f4d0 0x55a485380260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.967+0000 7f24b682f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a485374640 0x55a485373de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.967+0000 7f24b682f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac1b0620 con 0x55a485374640 2024-04-01T01:37:16.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.967+0000 7f24b682f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac1b2160 con 0x55a48536f4d0 2024-04-01T01:37:16.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.967+0000 7f24b682f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac1b0cf0 con 0x55a48536eae0 2024-04-01T01:37:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.967+0000 7f24b19e3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a48536f4d0 0x55a485380260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.967+0000 7f24b19e3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a48536f4d0 0x55a485380260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49398/0 (socket says 172.21.15.70:49398) 2024-04-01T01:37:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.967+0000 7f24b19e3640 1 -- 172.21.15.70:0/3384485870 learned_addr learned my addr 172.21.15.70:0/3384485870 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.968+0000 7f24b11e2640 1 --2- 172.21.15.70:0/3384485870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a485374640 0x55a485373de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.968+0000 7f24b19e3640 1 -- 172.21.15.70:0/3384485870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a48536eae0 msgr2=0x55a485380950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.968+0000 7f24b19e3640 1 --2- 172.21.15.70:0/3384485870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a48536eae0 0x55a485380950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.968+0000 7f24b19e3640 1 -- 172.21.15.70:0/3384485870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a485374640 msgr2=0x55a485373de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.968+0000 7f24b19e3640 1 --2- 172.21.15.70:0/3384485870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a485374640 0x55a485373de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.968+0000 7f24b19e3640 1 -- 172.21.15.70:0/3384485870 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a48536f8b0 con 0x55a48536f4d0 2024-04-01T01:37:16.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.969+0000 7f24b19e3640 1 --2- 172.21.15.70:0/3384485870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a48536f4d0 0x55a485380260 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f24a8034460 tx=0x7f24a802bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96b20156a6264e45 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.969+0000 7f24977fe640 1 -- 172.21.15.70:0/3384485870 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a805a030 con 0x55a48536f4d0 2024-04-01T01:37:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.969+0000 7f24977fe640 1 -- 172.21.15.70:0/3384485870 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24a8009520 con 0x55a48536f4d0 2024-04-01T01:37:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.969+0000 7f24977fe640 1 -- 172.21.15.70:0/3384485870 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a8000e60 con 0x55a48536f4d0 2024-04-01T01:37:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.971+0000 7f24b682f740 1 -- 172.21.15.70:0/3384485870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a48536f4d0 msgr2=0x55a485380260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.971+0000 7f24b682f740 1 --2- 172.21.15.70:0/3384485870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a48536f4d0 0x55a485380260 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f24a8034460 tx=0x7f24a802bc40 comp rx=0 tx=0).stop 2024-04-01T01:37:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.971+0000 7f24b682f740 1 -- 172.21.15.70:0/3384485870 shutdown_connections 2024-04-01T01:37:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.971+0000 7f24b682f740 1 --2- 172.21.15.70:0/3384485870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a485374640 0x55a485373de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.971+0000 7f24b682f740 1 --2- 172.21.15.70:0/3384485870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a48536f4d0 0x55a485380260 secure :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f24a8034460 tx=0x7f24a802bc40 comp rx=0 tx=0).stop 2024-04-01T01:37:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.971+0000 7f24b682f740 1 --2- 172.21.15.70:0/3384485870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a48536eae0 0x55a485380950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.971+0000 7f24b682f740 1 -- 172.21.15.70:0/3384485870 >> 172.21.15.70:0/3384485870 conn(0x55a48536a050 msgr2=0x55a4853783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.971+0000 7f24b682f740 1 -- 172.21.15.70:0/3384485870 shutdown_connections 2024-04-01T01:37:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.972+0000 7f24b682f740 1 -- 172.21.15.70:0/3384485870 wait complete. 2024-04-01T01:37:16.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.972+0000 7f24b682f740 1 -- start start 2024-04-01T01:37:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.972+0000 7f24b682f740 1 Processor -- start 2024-04-01T01:37:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.973+0000 7f24b682f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a48536eae0 0x55a48537d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.973+0000 7f24b682f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a485374640 0x55a48537dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.973+0000 7f24b682f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a48537ec10 0x55a48537f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.973+0000 7f24b11e2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a48536eae0 0x55a48537d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.974+0000 7f24b11e2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a48536eae0 0x55a48537d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49410/0 (socket says 172.21.15.70:49410) 2024-04-01T01:37:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.974+0000 7f24b11e2640 1 -- 172.21.15.70:0/343559769 learned_addr learned my addr 172.21.15.70:0/343559769 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.974+0000 7f24b682f740 1 -- 172.21.15.70:0/343559769 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac1b1a90 con 0x55a48537ec10 2024-04-01T01:37:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.974+0000 7f24b682f740 1 -- 172.21.15.70:0/343559769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac1b13c0 con 0x55a48536eae0 2024-04-01T01:37:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.974+0000 7f24b682f740 1 -- 172.21.15.70:0/343559769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f24ac118130 con 0x55a485374640 2024-04-01T01:37:16.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.974+0000 7f24b682f740 10 client.0 _ll_register_callbacks cb 0x7f24a8004270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.975+0000 7f24b11e2640 1 --2- 172.21.15.70:0/343559769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a48537ec10 0x55a48537f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.976+0000 7f24b11e2640 1 -- 172.21.15.70:0/343559769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a485374640 msgr2=0x55a48537dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:16.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.976+0000 7f24b11e2640 1 --2- 172.21.15.70:0/343559769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a485374640 0x55a48537dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.976+0000 7f24b11e2640 1 -- 172.21.15.70:0/343559769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a48537ec10 msgr2=0x55a48537f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.976+0000 7f24b11e2640 1 --2- 172.21.15.70:0/343559769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a48537ec10 0x55a48537f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.976+0000 7f24b11e2640 1 -- 172.21.15.70:0/343559769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a48537f5d0 con 0x55a48536eae0 2024-04-01T01:37:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.976+0000 7f24b11e2640 1 --2- 172.21.15.70:0/343559769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a48536eae0 0x55a48537d730 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f249c009da0 tx=0x7f249c011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aa1aa360cd584a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.976+0000 7f24957fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:16.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.977+0000 7f24957fa640 1 -- 172.21.15.70:0/343559769 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249c036040 con 0x55a48536eae0 2024-04-01T01:37:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.977+0000 7f24957fa640 1 -- 172.21.15.70:0/343559769 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f249c004460 con 0x55a48536eae0 2024-04-01T01:37:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.977+0000 7f24b682f740 1 -- 172.21.15.70:0/343559769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24a8009520 con 0x55a48536eae0 2024-04-01T01:37:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.978+0000 7f24957fa640 1 -- 172.21.15.70:0/343559769 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f249c004940 con 0x55a48536eae0 2024-04-01T01:37:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.978+0000 7f24b682f740 10 client.26612 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:16.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.978+0000 7f24b682f740 1 -- 172.21.15.70:0/343559769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a4853eaef0 con 0x55a48536eae0 2024-04-01T01:37:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.978+0000 7f24957fa640 1 -- 172.21.15.70:0/343559769 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24a8009520 con 0x55a48536eae0 2024-04-01T01:37:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.978+0000 7f24957fa640 1 -- 172.21.15.70:0/343559769 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f249c01d030 con 0x55a48536eae0 2024-04-01T01:37:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.979+0000 7f24b682f740 10 client.26612 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.979+0000 7f24b682f740 10 client.26612 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.979+0000 7f24b682f740 20 client.26612 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.979+0000 7f24b682f740 1 --2- 172.21.15.70:0/343559769 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a4853eb610 0x55a4853eda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.979+0000 7f24b682f740 4 client.26612 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:37:16.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.979+0000 7f24b682f740 1 -- 172.21.15.70:0/343559769 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a4852dfe30 con 0x55a4853eb610 2024-04-01T01:37:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.980+0000 7f24b19e3640 1 --2- 172.21.15.70:0/343559769 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a4853eb610 0x55a4853eda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.981+0000 7f24b19e3640 1 --2- 172.21.15.70:0/343559769 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a4853eb610 0x55a4853eda00 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f5b32b45b50c025 server_cookie=7867cf596d1e801f in_seq=0 out_seq=0 2024-04-01T01:37:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:16.981+0000 7f24957fa640 10 client.26612 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.025+0000 7f24957fa640 1 -- 172.21.15.70:0/343559769 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a4852dfe30 con 0x55a4853eb610 2024-04-01T01:37:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.025+0000 7f24957fa640 10 client.26612 handle_command_reply: tid=0 2024-04-01T01:37:17.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.025+0000 7f24b682f740 1 client.26612 shutdown 2024-04-01T01:37:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.029+0000 7f24b682f740 1 -- 172.21.15.70:0/343559769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a48536eae0 msgr2=0x55a48537d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:17.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.029+0000 7f24b682f740 1 --2- 172.21.15.70:0/343559769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a48536eae0 0x55a48537d730 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f249c009da0 tx=0x7f249c011440 comp rx=0 tx=0).stop 2024-04-01T01:37:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.033+0000 7f24b682f740 1 -- 172.21.15.70:0/343559769 shutdown_connections 2024-04-01T01:37:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.033+0000 7f24b682f740 1 --2- 172.21.15.70:0/343559769 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a4853eb610 0x55a4853eda00 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.033+0000 7f24957fa640 0 client.26612 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.033+0000 7f24b682f740 1 --2- 172.21.15.70:0/343559769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a48537ec10 0x55a48537f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.033+0000 7f24b682f740 1 --2- 172.21.15.70:0/343559769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a485374640 0x55a48537dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.033+0000 7f24b682f740 1 --2- 172.21.15.70:0/343559769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a48536eae0 0x55a48537d730 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.033+0000 7f24b682f740 1 -- 172.21.15.70:0/343559769 >> 172.21.15.70:0/343559769 conn(0x55a48536a050 msgr2=0x55a48536a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.033+0000 7f24b682f740 1 -- 172.21.15.70:0/343559769 shutdown_connections 2024-04-01T01:37:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.034+0000 7f24b682f740 1 -- 172.21.15.70:0/343559769 wait complete. 2024-04-01T01:37:17.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.034+0000 7f24b682f740 20 client.26612 trim_cache size 0 max 16384 2024-04-01T01:37:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:37:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:37:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:37:17.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:37:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.036+0000 7f24b29e5640 1 -- 172.21.15.70:0/3846867450 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24980a2660 msgr2=0x7f24980a4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.036+0000 7f24b29e5640 1 --2- 172.21.15.70:0/3846867450 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24980a2660 0x7f24980a4b20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f24a40cf0a0 tx=0x7f24a40bb040 comp rx=0 tx=0).stop 2024-04-01T01:37:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.036+0000 7f24b29e5640 1 -- 172.21.15.70:0/3846867450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24ac109440 msgr2=0x7f24ac1ad980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.036+0000 7f24b29e5640 1 --2- 172.21.15.70:0/3846867450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24ac109440 0x7f24ac1ad980 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f24a400e940 tx=0x7f24a400ee00 comp rx=0 tx=0).stop 2024-04-01T01:37:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.038+0000 7f24b29e5640 1 -- 172.21.15.70:0/3846867450 shutdown_connections 2024-04-01T01:37:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.038+0000 7f24b29e5640 1 --2- 172.21.15.70:0/3846867450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24ac109440 0x7f24ac1ad980 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.038+0000 7f24b29e5640 1 --2- 172.21.15.70:0/3846867450 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24980a2660 0x7f24980a4b20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.038+0000 7f24b29e5640 1 --2- 172.21.15.70:0/3846867450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ac073360 0x7f24ac1a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.038+0000 7f24b29e5640 1 --2- 172.21.15.70:0/3846867450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24ac072a00 0x7f24ac1a63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.038+0000 7f24b29e5640 1 -- 172.21.15.70:0/3846867450 >> 172.21.15.70:0/3846867450 conn(0x7f24ac06e5c0 msgr2=0x7f24ac10a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.038+0000 7f24b29e5640 1 -- 172.21.15.70:0/3846867450 shutdown_connections 2024-04-01T01:37:17.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:17.038+0000 7f24b29e5640 1 -- 172.21.15.70:0/3846867450 wait complete. 2024-04-01T01:37:18.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:17 smithi081 ceph-mon[35142]: pgmap v472: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-04-01T01:37:18.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:17 smithi189 ceph-mon[35753]: pgmap v472: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-04-01T01:37:18.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:17 smithi070 ceph-mon[32179]: pgmap v472: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 11 MiB/s wr, 1.27k op/s 2024-04-01T01:37:20.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:19 smithi081 ceph-mon[35142]: pgmap v473: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 10 MiB/s wr, 1.13k op/s 2024-04-01T01:37:20.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:19 smithi189 ceph-mon[35753]: pgmap v473: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 10 MiB/s wr, 1.13k op/s 2024-04-01T01:37:20.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:19 smithi070 ceph-mon[32179]: pgmap v473: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 10 MiB/s wr, 1.13k op/s 2024-04-01T01:37:21.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 155.0 s, 969.4 tps, lat 103.705 ms stddev 33.471 2024-04-01T01:37:22.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:21 smithi081 ceph-mon[35142]: pgmap v474: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 10 MiB/s wr, 1.08k op/s 2024-04-01T01:37:22.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:21 smithi189 ceph-mon[35753]: pgmap v474: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 10 MiB/s wr, 1.08k op/s 2024-04-01T01:37:22.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:21 smithi070 ceph-mon[32179]: pgmap v474: 129 pgs: 129 active+clean; 1.0 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 10 MiB/s wr, 1.08k op/s 2024-04-01T01:37:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:22 smithi081 ceph-mon[35142]: pgmap v475: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 11 MiB/s wr, 1.15k op/s 2024-04-01T01:37:23.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:22 smithi189 ceph-mon[35753]: pgmap v475: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 11 MiB/s wr, 1.15k op/s 2024-04-01T01:37:23.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:22 smithi070 ceph-mon[32179]: pgmap v475: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 11 MiB/s wr, 1.15k op/s 2024-04-01T01:37:25.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:25 smithi189 ceph-mon[35753]: pgmap v476: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 9.2 MiB/s wr, 950 op/s 2024-04-01T01:37:26.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:25 smithi070 ceph-mon[32179]: pgmap v476: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 9.2 MiB/s wr, 950 op/s 2024-04-01T01:37:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:25 smithi081 ceph-mon[35142]: pgmap v476: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 9.2 MiB/s wr, 950 op/s 2024-04-01T01:37:26.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 160.0 s, 1031.6 tps, lat 96.648 ms stddev 33.896 2024-04-01T01:37:27.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:27 smithi189 ceph-mon[35753]: pgmap v477: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 11 MiB/s wr, 1.11k op/s 2024-04-01T01:37:28.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:27 smithi070 ceph-mon[32179]: pgmap v477: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 11 MiB/s wr, 1.11k op/s 2024-04-01T01:37:28.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:27 smithi081 ceph-mon[35142]: pgmap v477: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 11 MiB/s wr, 1.11k op/s 2024-04-01T01:37:30.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:29 smithi081 ceph-mon[35142]: pgmap v478: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 8.2 MiB/s wr, 978 op/s 2024-04-01T01:37:30.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.183+0000 7f915626b640 1 -- 172.21.15.70:0/1621733545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9150109960 msgr2=0x7f91501101f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:30.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.183+0000 7f915626b640 1 --2- 172.21.15.70:0/1621733545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9150109960 0x7f91501101f0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f9140009900 tx=0x7f914002bc50 comp rx=0 tx=0).stop 2024-04-01T01:37:30.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.186+0000 7f915626b640 1 -- 172.21.15.70:0/1621733545 shutdown_connections 2024-04-01T01:37:30.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.186+0000 7f915626b640 1 --2- 172.21.15.70:0/1621733545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9150109960 0x7f91501101f0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.186+0000 7f915626b640 1 --2- 172.21.15.70:0/1621733545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9150108fa0 0x7f9150109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.191 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.186+0000 7f915626b640 1 --2- 172.21.15.70:0/1621733545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9150107db0 0x7f91501081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.186+0000 7f915626b640 1 -- 172.21.15.70:0/1621733545 >> 172.21.15.70:0/1621733545 conn(0x7f9150076d60 msgr2=0x7f9150079180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.187+0000 7f915626b640 1 -- 172.21.15.70:0/1621733545 shutdown_connections 2024-04-01T01:37:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.187+0000 7f915626b640 1 -- 172.21.15.70:0/1621733545 wait complete. 2024-04-01T01:37:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.188+0000 7f915626b640 1 Processor -- start 2024-04-01T01:37:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.188+0000 7f915626b640 1 -- start start 2024-04-01T01:37:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.188+0000 7f915626b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9150107db0 0x7f9150070050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.188+0000 7f915626b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9150108fa0 0x7f9150070590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.189+0000 7f915626b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9150109960 0x7f91501ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.189+0000 7f915626b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f91501139d0 con 0x7f9150109960 2024-04-01T01:37:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.189+0000 7f915626b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9150113850 con 0x7f9150108fa0 2024-04-01T01:37:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.189+0000 7f915626b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9150113b50 con 0x7f9150107db0 2024-04-01T01:37:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.189+0000 7f914ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9150107db0 0x7f9150070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.189+0000 7f914ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9150107db0 0x7f9150070050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43844/0 (socket says 172.21.15.70:43844) 2024-04-01T01:37:30.192 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.189+0000 7f914ffff640 1 -- 172.21.15.70:0/441831676 learned_addr learned my addr 172.21.15.70:0/441831676 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.189+0000 7f914f7fe640 1 --2- 172.21.15.70:0/441831676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9150108fa0 0x7f9150070590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.189+0000 7f9154808640 1 --2- 172.21.15.70:0/441831676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9150109960 0x7f91501ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.189+0000 7f914f7fe640 1 -- 172.21.15.70:0/441831676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9150107db0 msgr2=0x7f9150070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.189+0000 7f914f7fe640 1 --2- 172.21.15.70:0/441831676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9150107db0 0x7f9150070050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.189+0000 7f914f7fe640 1 -- 172.21.15.70:0/441831676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9150109960 msgr2=0x7f91501ae0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.189+0000 7f914f7fe640 1 --2- 172.21.15.70:0/441831676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9150109960 0x7f91501ae0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.189+0000 7f914f7fe640 1 -- 172.21.15.70:0/441831676 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9150073710 con 0x7f9150108fa0 2024-04-01T01:37:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.189+0000 7f914ffff640 1 --2- 172.21.15.70:0/441831676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9150107db0 0x7f9150070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:30.193 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.189+0000 7f914f7fe640 1 --2- 172.21.15.70:0/441831676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9150108fa0 0x7f9150070590 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f914800bd50 tx=0x7f914800be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.190+0000 7f914d7fa640 1 -- 172.21.15.70:0/441831676 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148019070 con 0x7f9150108fa0 2024-04-01T01:37:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.190+0000 7f914d7fa640 1 -- 172.21.15.70:0/441831676 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91480148f0 con 0x7f9150108fa0 2024-04-01T01:37:30.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.190+0000 7f914d7fa640 1 -- 172.21.15.70:0/441831676 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148012960 con 0x7f9150108fa0 2024-04-01T01:37:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.190+0000 7f915626b640 1 -- 172.21.15.70:0/441831676 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f915007a000 con 0x7f9150108fa0 2024-04-01T01:37:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.190+0000 7f915626b640 1 -- 172.21.15.70:0/441831676 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f915007a520 con 0x7f9150108fa0 2024-04-01T01:37:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.193+0000 7f914d7fa640 1 -- 172.21.15.70:0/441831676 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f914801c070 con 0x7f9150108fa0 2024-04-01T01:37:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.194+0000 7f914d7fa640 1 --2- 172.21.15.70:0/441831676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f91380a25c0 0x7f91380a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.194+0000 7f914d7fa640 1 -- 172.21.15.70:0/441831676 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f91480ce410 con 0x7f9150108fa0 2024-04-01T01:37:30.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.195+0000 7f9158e46740 1 Processor -- start 2024-04-01T01:37:30.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.195+0000 7f914ffff640 1 --2- 172.21.15.70:0/441831676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f91380a25c0 0x7f91380a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.197+0000 7f9158e46740 1 -- start start 2024-04-01T01:37:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.197+0000 7f914ffff640 1 --2- 172.21.15.70:0/441831676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f91380a25c0 0x7f91380a4a80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f91440058c0 tx=0x7f9144005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f9158e46740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e28d0 0x5610269e2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f9158e46740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269e32c0 0x5610269e7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f9158e46740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e8430 0x5610269ea8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f914ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e28d0 0x5610269e2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f914ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e28d0 0x5610269e2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:40326/0 (socket says 172.21.15.70:40326) 2024-04-01T01:37:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f914f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269e32c0 0x5610269e7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f914f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269e32c0 0x5610269e7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43854/0 (socket says 172.21.15.70:43854) 2024-04-01T01:37:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f9158e46740 1 -- 172.21.15.70:0/3956645150 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5610269468f0 con 0x5610269e8430 2024-04-01T01:37:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f914ffff640 1 -- 172.21.15.70:0/3956645150 learned_addr learned my addr 172.21.15.70:0/3956645150 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f9158e46740 1 -- 172.21.15.70:0/3956645150 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5610268ef8d0 con 0x5610269e28d0 2024-04-01T01:37:30.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f9158e46740 1 -- 172.21.15.70:0/3956645150 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561026942990 con 0x5610269e32c0 2024-04-01T01:37:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f9154808640 1 --2- 172.21.15.70:0/3956645150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e8430 0x5610269ea8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f914ffff640 1 -- 172.21.15.70:0/3956645150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269e32c0 msgr2=0x5610269e7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f914ffff640 1 --2- 172.21.15.70:0/3956645150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269e32c0 0x5610269e7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f914ffff640 1 -- 172.21.15.70:0/3956645150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e8430 msgr2=0x5610269ea8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f914ffff640 1 --2- 172.21.15.70:0/3956645150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e8430 0x5610269ea8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.198+0000 7f914ffff640 1 -- 172.21.15.70:0/3956645150 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56102665b210 con 0x5610269e28d0 2024-04-01T01:37:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.199+0000 7f914ffff640 1 --2- 172.21.15.70:0/3956645150 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e28d0 0x5610269e2cf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f914400f6f0 tx=0x7f9144014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c63640150e7ca0e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:30.200 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.199+0000 7f91367fc640 1 -- 172.21.15.70:0/3956645150 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914402a070 con 0x5610269e28d0 2024-04-01T01:37:30.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.199+0000 7f91367fc640 1 -- 172.21.15.70:0/3956645150 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f914401b8e0 con 0x5610269e28d0 2024-04-01T01:37:30.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.199+0000 7f91367fc640 1 -- 172.21.15.70:0/3956645150 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914401c9e0 con 0x5610269e28d0 2024-04-01T01:37:30.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.200+0000 7f9158e46740 1 -- 172.21.15.70:0/3956645150 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e28d0 msgr2=0x5610269e2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:30.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.200+0000 7f9158e46740 1 --2- 172.21.15.70:0/3956645150 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e28d0 0x5610269e2cf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f914400f6f0 tx=0x7f9144014000 comp rx=0 tx=0).stop 2024-04-01T01:37:30.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.201+0000 7f9158e46740 1 -- 172.21.15.70:0/3956645150 shutdown_connections 2024-04-01T01:37:30.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.201+0000 7f9158e46740 1 --2- 172.21.15.70:0/3956645150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e8430 0x5610269ea8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.201+0000 7f9158e46740 1 --2- 172.21.15.70:0/3956645150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269e32c0 0x5610269e7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.201+0000 7f9158e46740 1 --2- 172.21.15.70:0/3956645150 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e28d0 0x5610269e2cf0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.201+0000 7f9158e46740 1 -- 172.21.15.70:0/3956645150 >> 172.21.15.70:0/3956645150 conn(0x5610269dde40 msgr2=0x5610269de240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:30.203 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.201+0000 7f9158e46740 1 -- 172.21.15.70:0/3956645150 shutdown_connections 2024-04-01T01:37:30.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.201+0000 7f9158e46740 1 -- 172.21.15.70:0/3956645150 wait complete. 2024-04-01T01:37:30.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.201+0000 7f9158e46740 1 -- start start 2024-04-01T01:37:30.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.202+0000 7f9158e46740 1 Processor -- start 2024-04-01T01:37:30.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.202+0000 7f9158e46740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269e28d0 0x5610269ecba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.202+0000 7f9158e46740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e32c0 0x5610269ed0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.202+0000 7f914ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269e28d0 0x5610269ecba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:30.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.202+0000 7f914ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269e28d0 0x5610269ecba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43858/0 (socket says 172.21.15.70:43858) 2024-04-01T01:37:30.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.202+0000 7f914ffff640 1 -- 172.21.15.70:0/832055827 learned_addr learned my addr 172.21.15.70:0/832055827 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:30.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.202+0000 7f9158e46740 1 --2- 172.21.15.70:0/832055827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e8430 0x5610269f4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.203+0000 7f9158e46740 1 -- 172.21.15.70:0/832055827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5610269eb3e0 con 0x5610269e32c0 2024-04-01T01:37:30.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.203+0000 7f9158e46740 1 -- 172.21.15.70:0/832055827 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5610268e4f80 con 0x5610269e8430 2024-04-01T01:37:30.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.203+0000 7f9158e46740 1 -- 172.21.15.70:0/832055827 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5610269eb560 con 0x5610269e28d0 2024-04-01T01:37:30.204 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.203+0000 7f9154808640 1 --2- 172.21.15.70:0/832055827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e8430 0x5610269f4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:30.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.203+0000 7f914ffff640 1 -- 172.21.15.70:0/832055827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e8430 msgr2=0x5610269f4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:30.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.203+0000 7f914ffff640 1 --2- 172.21.15.70:0/832055827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e8430 0x5610269f4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.203+0000 7f914ffff640 1 -- 172.21.15.70:0/832055827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e32c0 msgr2=0x5610269ed0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:30.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.203+0000 7f914ffff640 1 --2- 172.21.15.70:0/832055827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e32c0 0x5610269ed0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.203+0000 7f914ffff640 1 -- 172.21.15.70:0/832055827 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561026727620 con 0x5610269e28d0 2024-04-01T01:37:30.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.203+0000 7f9158e46740 10 client.0 _ll_register_callbacks cb 0x56102693d290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:30.205 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.203+0000 7f9158e46740 1 -- 172.21.15.70:0/832055827 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f914401b8e0 con 0x5610269e28d0 2024-04-01T01:37:30.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.204+0000 7f914ffff640 1 --2- 172.21.15.70:0/832055827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269e28d0 0x5610269ecba0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f914401cfd0 tx=0x7f914401c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d73e1b90001214e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:30.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.204+0000 7f912f7fe640 10 client.26506 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:37:30.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.204+0000 7f912f7fe640 1 -- 172.21.15.70:0/832055827 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144032020 con 0x5610269e28d0 2024-04-01T01:37:30.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.205+0000 7f912f7fe640 1 -- 172.21.15.70:0/832055827 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f914401b430 con 0x5610269e28d0 2024-04-01T01:37:30.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.205+0000 7f912f7fe640 1 -- 172.21.15.70:0/832055827 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f914402a040 con 0x5610269e28d0 2024-04-01T01:37:30.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.205+0000 7f9158e46740 10 client.26506 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:30.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.205+0000 7f9158e46740 1 -- 172.21.15.70:0/832055827 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=29}) -- 0x5610269f6d30 con 0x5610269e28d0 2024-04-01T01:37:30.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.206+0000 7f912f7fe640 1 -- 172.21.15.70:0/832055827 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9144024030 con 0x5610269e28d0 2024-04-01T01:37:30.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.206+0000 7f9158e46740 10 client.26506 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.206+0000 7f9158e46740 10 client.26506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.206+0000 7f9158e46740 20 client.26506 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.206+0000 7f9158e46740 1 --2- 172.21.15.70:0/832055827 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5610269f8080 0x5610269fa470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.206+0000 7f9158e46740 4 client.26506 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.206+0000 7f9158e46740 1 -- 172.21.15.70:0/832055827 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56102695a120 con 0x5610269f8080 2024-04-01T01:37:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.207+0000 7f914f7fe640 1 --2- 172.21.15.70:0/832055827 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5610269f8080 0x5610269fa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.208+0000 7f914f7fe640 1 --2- 172.21.15.70:0/832055827 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5610269f8080 0x5610269fa470 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54247599fab9296c server_cookie=250f37827fce7c89 in_seq=0 out_seq=0 2024-04-01T01:37:30.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.208+0000 7f912f7fe640 10 client.26506 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.209+0000 7f912f7fe640 1 -- 172.21.15.70:0/832055827 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56102695a120 con 0x5610269f8080 2024-04-01T01:37:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.209+0000 7f912f7fe640 10 client.26506 handle_command_reply: tid=0 2024-04-01T01:37:30.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.209+0000 7f9158e46740 1 client.26506 shutdown 2024-04-01T01:37:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.209+0000 7f9158e46740 1 -- 172.21.15.70:0/832055827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269e28d0 msgr2=0x5610269ecba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.209+0000 7f9158e46740 1 --2- 172.21.15.70:0/832055827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269e28d0 0x5610269ecba0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f914401cfd0 tx=0x7f914401c7e0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.209+0000 7f9158e46740 1 -- 172.21.15.70:0/832055827 shutdown_connections 2024-04-01T01:37:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.209+0000 7f9158e46740 1 --2- 172.21.15.70:0/832055827 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5610269f8080 0x5610269fa470 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.209+0000 7f9158e46740 1 --2- 172.21.15.70:0/832055827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e8430 0x5610269f4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.209+0000 7f9158e46740 1 --2- 172.21.15.70:0/832055827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e32c0 0x5610269ed0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.209+0000 7f9158e46740 1 --2- 172.21.15.70:0/832055827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269e28d0 0x5610269ecba0 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f914401cfd0 tx=0x7f914401c7e0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.209+0000 7f9158e46740 1 -- 172.21.15.70:0/832055827 >> 172.21.15.70:0/832055827 conn(0x5610269dde40 msgr2=0x5610269dcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:30.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.210+0000 7f912f7fe640 0 client.26506 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:30.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.211+0000 7f9158e46740 1 -- 172.21.15.70:0/832055827 shutdown_connections 2024-04-01T01:37:30.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.212+0000 7f9158e46740 1 -- 172.21.15.70:0/832055827 wait complete. 2024-04-01T01:37:30.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.212+0000 7f9158e46740 20 client.26506 trim_cache size 0 max 16384 2024-04-01T01:37:30.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:29 smithi189 ceph-mon[35753]: pgmap v478: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 8.2 MiB/s wr, 978 op/s 2024-04-01T01:37:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.248+0000 7f9158e46740 1 Processor -- start 2024-04-01T01:37:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.249+0000 7f9158e46740 1 -- start start 2024-04-01T01:37:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.249+0000 7f9158e46740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e32c0 0x5610269e2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.249+0000 7f9158e46740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e8430 0x5610269f40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.249+0000 7f9158e46740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269de220 0x5610269ffca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.249+0000 7f9158e46740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f91501b0a80 con 0x5610269e8430 2024-04-01T01:37:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.249+0000 7f9158e46740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f91501b25c0 con 0x5610269e32c0 2024-04-01T01:37:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.249+0000 7f9158e46740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f91501b1150 con 0x5610269de220 2024-04-01T01:37:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.250+0000 7f914ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e32c0 0x5610269e2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.250+0000 7f9154808640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269de220 0x5610269ffca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.250+0000 7f914ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e32c0 0x5610269e2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:40338/0 (socket says 172.21.15.70:40338) 2024-04-01T01:37:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.250+0000 7f914ffff640 1 -- 172.21.15.70:0/1001936882 learned_addr learned my addr 172.21.15.70:0/1001936882 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.250+0000 7f914ffff640 1 -- 172.21.15.70:0/1001936882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269de220 msgr2=0x5610269ffca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.250+0000 7f914ffff640 1 --2- 172.21.15.70:0/1001936882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269de220 0x5610269ffca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.250+0000 7f914ffff640 1 -- 172.21.15.70:0/1001936882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e8430 msgr2=0x5610269f40e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:37:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.250+0000 7f914ffff640 1 --2- 172.21.15.70:0/1001936882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e8430 0x5610269f40e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.250+0000 7f914ffff640 1 -- 172.21.15.70:0/1001936882 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610269e2510 con 0x5610269e32c0 2024-04-01T01:37:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.251+0000 7f914ffff640 1 --2- 172.21.15.70:0/1001936882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e32c0 0x5610269e2cd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9144011db0 tx=0x7f9144026000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ca18b3f10f42e4a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.252+0000 7f9135ffb640 1 -- 172.21.15.70:0/1001936882 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144021030 con 0x5610269e32c0 2024-04-01T01:37:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.252+0000 7f9135ffb640 1 -- 172.21.15.70:0/1001936882 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f914401b8e0 con 0x5610269e32c0 2024-04-01T01:37:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.252+0000 7f9135ffb640 1 -- 172.21.15.70:0/1001936882 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144032070 con 0x5610269e32c0 2024-04-01T01:37:30.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:29 smithi070 ceph-mon[32179]: pgmap v478: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 8.2 MiB/s wr, 978 op/s 2024-04-01T01:37:30.254 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.253+0000 7f9158e46740 1 -- 172.21.15.70:0/1001936882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e32c0 msgr2=0x5610269e2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.253+0000 7f9158e46740 1 --2- 172.21.15.70:0/1001936882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e32c0 0x5610269e2cd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9144011db0 tx=0x7f9144026000 comp rx=0 tx=0).stop 2024-04-01T01:37:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.254+0000 7f9158e46740 1 -- 172.21.15.70:0/1001936882 shutdown_connections 2024-04-01T01:37:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.254+0000 7f9158e46740 1 --2- 172.21.15.70:0/1001936882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269de220 0x5610269ffca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.254+0000 7f9158e46740 1 --2- 172.21.15.70:0/1001936882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e8430 0x5610269f40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.254+0000 7f9158e46740 1 --2- 172.21.15.70:0/1001936882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e32c0 0x5610269e2cd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.254+0000 7f9158e46740 1 -- 172.21.15.70:0/1001936882 >> 172.21.15.70:0/1001936882 conn(0x5610269dde40 msgr2=0x5610269d8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.254+0000 7f9158e46740 1 -- 172.21.15.70:0/1001936882 shutdown_connections 2024-04-01T01:37:30.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.254+0000 7f9158e46740 1 -- 172.21.15.70:0/1001936882 wait complete. 2024-04-01T01:37:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.255+0000 7f9158e46740 1 -- start start 2024-04-01T01:37:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.256+0000 7f9158e46740 1 Processor -- start 2024-04-01T01:37:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.256+0000 7f9158e46740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269de220 0x5610269f1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.256+0000 7f9158e46740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e32c0 0x5610269f2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.256+0000 7f9158e46740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e8430 0x5610269f3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.256+0000 7f9158e46740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f91501b1ef0 con 0x5610269e8430 2024-04-01T01:37:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.256+0000 7f9158e46740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f91501b1820 con 0x5610269e32c0 2024-04-01T01:37:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.256+0000 7f9158e46740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f91501aea60 con 0x5610269de220 2024-04-01T01:37:30.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.256+0000 7f9158e46740 10 client.0 _ll_register_callbacks cb 0x7f9144028070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.257+0000 7f914f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e32c0 0x5610269f2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.257+0000 7f914f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e32c0 0x5610269f2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:40346/0 (socket says 172.21.15.70:40346) 2024-04-01T01:37:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.257+0000 7f914f7fe640 1 -- 172.21.15.70:0/3789600791 learned_addr learned my addr 172.21.15.70:0/3789600791 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.257+0000 7f914ffff640 1 --2- 172.21.15.70:0/3789600791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269de220 0x5610269f1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.257+0000 7f9154808640 1 --2- 172.21.15.70:0/3789600791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e8430 0x5610269f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.257+0000 7f914f7fe640 1 -- 172.21.15.70:0/3789600791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269de220 msgr2=0x5610269f1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.257+0000 7f914f7fe640 1 --2- 172.21.15.70:0/3789600791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269de220 0x5610269f1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.257+0000 7f914f7fe640 1 -- 172.21.15.70:0/3789600791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e8430 msgr2=0x5610269f3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.257+0000 7f914f7fe640 1 --2- 172.21.15.70:0/3789600791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e8430 0x5610269f3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.257+0000 7f914f7fe640 1 -- 172.21.15.70:0/3789600791 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610269f3610 con 0x5610269e32c0 2024-04-01T01:37:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.258+0000 7f914f7fe640 1 --2- 172.21.15.70:0/3789600791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e32c0 0x5610269f2010 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f914809a620 tx=0x7f91480cb9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecb7351a2d256c76 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.258+0000 7f912f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:30.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.258+0000 7f912f7fe640 1 -- 172.21.15.70:0/3789600791 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91480ce0b0 con 0x5610269e32c0 2024-04-01T01:37:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.258+0000 7f9158e46740 1 -- 172.21.15.70:0/3789600791 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f914401b8e0 con 0x5610269e32c0 2024-04-01T01:37:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.259+0000 7f912f7fe640 1 -- 172.21.15.70:0/3789600791 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f91480971c0 con 0x5610269e32c0 2024-04-01T01:37:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.259+0000 7f912f7fe640 1 -- 172.21.15.70:0/3789600791 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9148082050 con 0x5610269e32c0 2024-04-01T01:37:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.259+0000 7f9158e46740 10 client.26630 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:30.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.259+0000 7f9158e46740 1 -- 172.21.15.70:0/3789600791 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561026a5e5a0 con 0x5610269e32c0 2024-04-01T01:37:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.260+0000 7f912f7fe640 1 -- 172.21.15.70:0/3789600791 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f914401b8e0 con 0x5610269e32c0 2024-04-01T01:37:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.260+0000 7f912f7fe640 1 -- 172.21.15.70:0/3789600791 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f91480a30d0 con 0x5610269e32c0 2024-04-01T01:37:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.260+0000 7f9158e46740 10 client.26630 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.260+0000 7f9158e46740 10 client.26630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.260+0000 7f9158e46740 20 client.26630 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.260+0000 7f9158e46740 1 --2- 172.21.15.70:0/3789600791 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561026a5ee80 0x561026a612a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.260+0000 7f9158e46740 4 client.26630 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:37:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.260+0000 7f9158e46740 1 -- 172.21.15.70:0/3789600791 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561026953e30 con 0x561026a5ee80 2024-04-01T01:37:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.260+0000 7f914ffff640 1 --2- 172.21.15.70:0/3789600791 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561026a5ee80 0x561026a612a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.261+0000 7f912f7fe640 10 client.26630 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:30.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.261+0000 7f914ffff640 1 --2- 172.21.15.70:0/3789600791 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561026a5ee80 0x561026a612a0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af73ce9afa364e5 server_cookie=d4532bb2ef6aa506 in_seq=0 out_seq=0 2024-04-01T01:37:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.293+0000 7f912f7fe640 1 -- 172.21.15.70:0/3789600791 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561026953e30 con 0x561026a5ee80 2024-04-01T01:37:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.293+0000 7f912f7fe640 10 client.26630 handle_command_reply: tid=0 2024-04-01T01:37:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.293+0000 7f9158e46740 1 client.26630 shutdown 2024-04-01T01:37:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.293+0000 7f9158e46740 1 -- 172.21.15.70:0/3789600791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e32c0 msgr2=0x5610269f2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.293+0000 7f9158e46740 1 --2- 172.21.15.70:0/3789600791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e32c0 0x5610269f2010 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f914809a620 tx=0x7f91480cb9b0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.294+0000 7f9158e46740 1 -- 172.21.15.70:0/3789600791 shutdown_connections 2024-04-01T01:37:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.294+0000 7f9158e46740 1 --2- 172.21.15.70:0/3789600791 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561026a5ee80 0x561026a612a0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.294+0000 7f9158e46740 1 --2- 172.21.15.70:0/3789600791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610269e8430 0x5610269f3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.294+0000 7f9158e46740 1 --2- 172.21.15.70:0/3789600791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5610269e32c0 0x5610269f2010 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.294+0000 7f9158e46740 1 --2- 172.21.15.70:0/3789600791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610269de220 0x5610269f1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.294+0000 7f9158e46740 1 -- 172.21.15.70:0/3789600791 >> 172.21.15.70:0/3789600791 conn(0x5610269dde40 msgr2=0x5610269ec790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.294+0000 7f912f7fe640 0 client.26630 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:30.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.294+0000 7f9158e46740 1 -- 172.21.15.70:0/3789600791 shutdown_connections 2024-04-01T01:37:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.296+0000 7f9158e46740 1 -- 172.21.15.70:0/3789600791 wait complete. 2024-04-01T01:37:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.296+0000 7f9158e46740 20 client.26630 trim_cache size 0 max 16384 2024-04-01T01:37:30.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:37:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:37:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:37:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:37:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.297+0000 7f915626b640 1 -- 172.21.15.70:0/441831676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f91380a25c0 msgr2=0x7f91380a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:30.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.297+0000 7f915626b640 1 --2- 172.21.15.70:0/441831676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f91380a25c0 0x7f91380a4a80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f91440058c0 tx=0x7f9144005f90 comp rx=0 tx=0).stop 2024-04-01T01:37:30.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.298+0000 7f915626b640 1 -- 172.21.15.70:0/441831676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9150108fa0 msgr2=0x7f9150070590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.298+0000 7f915626b640 1 --2- 172.21.15.70:0/441831676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9150108fa0 0x7f9150070590 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f914800bd50 tx=0x7f914800be40 comp rx=0 tx=0).stop 2024-04-01T01:37:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.298+0000 7f915626b640 1 -- 172.21.15.70:0/441831676 shutdown_connections 2024-04-01T01:37:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.298+0000 7f915626b640 1 --2- 172.21.15.70:0/441831676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9150109960 0x7f91501ae0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.298+0000 7f915626b640 1 --2- 172.21.15.70:0/441831676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9150108fa0 0x7f9150070590 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.298+0000 7f915626b640 1 --2- 172.21.15.70:0/441831676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f91380a25c0 0x7f91380a4a80 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.298+0000 7f915626b640 1 --2- 172.21.15.70:0/441831676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9150107db0 0x7f9150070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.298+0000 7f915626b640 1 -- 172.21.15.70:0/441831676 >> 172.21.15.70:0/441831676 conn(0x7f9150076d60 msgr2=0x7f915010c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.298+0000 7f915626b640 1 -- 172.21.15.70:0/441831676 shutdown_connections 2024-04-01T01:37:30.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:30.298+0000 7f915626b640 1 -- 172.21.15.70:0/441831676 wait complete. 2024-04-01T01:37:31.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 165.0 s, 1021.4 tps, lat 97.957 ms stddev 44.223 2024-04-01T01:37:32.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:31 smithi189 ceph-mon[35753]: pgmap v479: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 8.3 MiB/s wr, 978 op/s 2024-04-01T01:37:32.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:31 smithi081 ceph-mon[35142]: pgmap v479: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 8.3 MiB/s wr, 978 op/s 2024-04-01T01:37:32.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:31 smithi070 ceph-mon[32179]: pgmap v479: 129 pgs: 129 active+clean; 1.0 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 8.3 MiB/s wr, 978 op/s 2024-04-01T01:37:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:33 smithi081 ceph-mon[35142]: pgmap v480: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 1.03k op/s 2024-04-01T01:37:34.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:33 smithi189 ceph-mon[35753]: pgmap v480: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 1.03k op/s 2024-04-01T01:37:34.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:33 smithi070 ceph-mon[32179]: pgmap v480: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 1.03k op/s 2024-04-01T01:37:36.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:35 smithi081 ceph-mon[35142]: pgmap v481: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.9 MiB/s wr, 936 op/s 2024-04-01T01:37:36.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:35 smithi189 ceph-mon[35753]: pgmap v481: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.9 MiB/s wr, 936 op/s 2024-04-01T01:37:36.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:35 smithi070 ceph-mon[32179]: pgmap v481: 129 pgs: 129 active+clean; 1.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.9 MiB/s wr, 936 op/s 2024-04-01T01:37:36.544 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:37:36.544 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:37:36.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.743+0000 7f9e82809640 1 -- 172.21.15.70:0/2737612925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e7c077350 msgr2=0x7f9e7c0777b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:36.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.743+0000 7f9e82809640 1 --2- 172.21.15.70:0/2737612925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e7c077350 0x7f9e7c0777b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9e74009ca0 tx=0x7f9e7402c120 comp rx=0 tx=0).stop 2024-04-01T01:37:36.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.744+0000 7f9e82809640 1 -- 172.21.15.70:0/2737612925 shutdown_connections 2024-04-01T01:37:36.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.744+0000 7f9e82809640 1 --2- 172.21.15.70:0/2737612925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e7c077cf0 0x7f9e7c078590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.744+0000 7f9e82809640 1 --2- 172.21.15.70:0/2737612925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e7c077350 0x7f9e7c0777b0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.744+0000 7f9e82809640 1 --2- 172.21.15.70:0/2737612925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7c076110 0x7f9e7c076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.744+0000 7f9e82809640 1 -- 172.21.15.70:0/2737612925 >> 172.21.15.70:0/2737612925 conn(0x7f9e7c0fedc0 msgr2=0x7f9e7c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:36.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.747+0000 7f9e82809640 1 -- 172.21.15.70:0/2737612925 shutdown_connections 2024-04-01T01:37:36.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.747+0000 7f9e82809640 1 -- 172.21.15.70:0/2737612925 wait complete. 2024-04-01T01:37:36.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.748+0000 7f9e82809640 1 Processor -- start 2024-04-01T01:37:36.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.748+0000 7f9e82809640 1 -- start start 2024-04-01T01:37:36.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.748+0000 7f9e82809640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7c076110 0x7f9e7c19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.748+0000 7f9e82809640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e7c077350 0x7f9e7c19e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.748+0000 7f9e82809640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e7c077cf0 0x7f9e7c1a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.748+0000 7f9e82809640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c1137f0 con 0x7f9e7c077cf0 2024-04-01T01:37:36.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.748+0000 7f9e82809640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c113670 con 0x7f9e7c077350 2024-04-01T01:37:36.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.748+0000 7f9e82809640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c113970 con 0x7f9e7c076110 2024-04-01T01:37:36.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.749+0000 7f9e82008640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e7c077cf0 0x7f9e7c1a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.749+0000 7f9e81006640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e7c077350 0x7f9e7c19e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.749+0000 7f9e81006640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e7c077350 0x7f9e7c19e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53458/0 (socket says 172.21.15.70:53458) 2024-04-01T01:37:36.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.749+0000 7f9e81006640 1 -- 172.21.15.70:0/916390009 learned_addr learned my addr 172.21.15.70:0/916390009 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:36.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.749+0000 7f9e81807640 1 --2- 172.21.15.70:0/916390009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7c076110 0x7f9e7c19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.749+0000 7f9e81006640 1 -- 172.21.15.70:0/916390009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7c076110 msgr2=0x7f9e7c19df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:36.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.749+0000 7f9e81006640 1 --2- 172.21.15.70:0/916390009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7c076110 0x7f9e7c19df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.749+0000 7f9e81006640 1 -- 172.21.15.70:0/916390009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e7c077cf0 msgr2=0x7f9e7c1a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:36.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.749+0000 7f9e81006640 1 --2- 172.21.15.70:0/916390009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e7c077cf0 0x7f9e7c1a5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.749+0000 7f9e81006640 1 -- 172.21.15.70:0/916390009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e7c1a5b70 con 0x7f9e7c077350 2024-04-01T01:37:36.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.749+0000 7f9e81006640 1 --2- 172.21.15.70:0/916390009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e7c077350 0x7f9e7c19e490 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9e74009dc0 tx=0x7f9e740347b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:36.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.749+0000 7f9e72ffd640 1 -- 172.21.15.70:0/916390009 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e74039040 con 0x7f9e7c077350 2024-04-01T01:37:36.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.750+0000 7f9e72ffd640 1 -- 172.21.15.70:0/916390009 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e740325f0 con 0x7f9e7c077350 2024-04-01T01:37:36.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.750+0000 7f9e72ffd640 1 -- 172.21.15.70:0/916390009 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e740042a0 con 0x7f9e7c077350 2024-04-01T01:37:36.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.750+0000 7f9e82809640 1 -- 172.21.15.70:0/916390009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e7c1a5e00 con 0x7f9e7c077350 2024-04-01T01:37:36.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.750+0000 7f9e82809640 1 -- 172.21.15.70:0/916390009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e7c1a6210 con 0x7f9e7c077350 2024-04-01T01:37:36.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.754+0000 7f9e72ffd640 1 -- 172.21.15.70:0/916390009 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f9e74046030 con 0x7f9e7c077350 2024-04-01T01:37:36.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.754+0000 7f9e86646740 1 Processor -- start 2024-04-01T01:37:36.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.754+0000 7f9e86646740 1 -- start start 2024-04-01T01:37:36.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.755+0000 7f9e86646740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032490 0x5621620328b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.755+0000 7f9e86646740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162032e80 0x562162037980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.755+0000 7f9e86646740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162037ff0 0x56216203a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.755+0000 7f9e86646740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562161f97180 con 0x562162032e80 2024-04-01T01:37:36.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.755+0000 7f9e86646740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562161f3fb70 con 0x562162037ff0 2024-04-01T01:37:36.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.755+0000 7f9e86646740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562161f93220 con 0x562162032490 2024-04-01T01:37:36.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.755+0000 7f9e81006640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162037ff0 0x56216203a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.755+0000 7f9e82008640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032490 0x5621620328b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.755+0000 7f9e82008640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032490 0x5621620328b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52552/0 (socket says 172.21.15.70:52552) 2024-04-01T01:37:36.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.755+0000 7f9e82008640 1 -- 172.21.15.70:0/327170642 learned_addr learned my addr 172.21.15.70:0/327170642 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:36.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.755+0000 7f9e72ffd640 1 --2- 172.21.15.70:0/916390009 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e680a24a0 0x7f9e680a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.755+0000 7f9e81807640 1 --2- 172.21.15.70:0/327170642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162032e80 0x562162037980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.756+0000 7f9e82008640 1 --2- 172.21.15.70:0/916390009 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e680a24a0 0x7f9e680a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.756+0000 7f9e72ffd640 1 -- 172.21.15.70:0/916390009 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9e740f0410 con 0x7f9e7c077350 2024-04-01T01:37:36.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.756+0000 7f9e82008640 1 -- 172.21.15.70:0/327170642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162037ff0 msgr2=0x56216203a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:36.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.756+0000 7f9e82008640 1 --2- 172.21.15.70:0/327170642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162037ff0 0x56216203a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.756+0000 7f9e82008640 1 -- 172.21.15.70:0/327170642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162032e80 msgr2=0x562162037980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:36.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.756+0000 7f9e82008640 1 --2- 172.21.15.70:0/327170642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162032e80 0x562162037980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.756+0000 7f9e82008640 1 -- 172.21.15.70:0/327170642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562161f537c0 con 0x562162032490 2024-04-01T01:37:36.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.756+0000 7f9e82008640 1 --2- 172.21.15.70:0/327170642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032490 0x5621620328b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9e780138b0 tx=0x7f9e78013d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37ec570c9add4bc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:36.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.757+0000 7f9e82008640 1 --2- 172.21.15.70:0/916390009 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e680a24a0 0x7f9e680a4960 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9e7801e000 tx=0x7f9e78020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:36.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.757+0000 7f9e67fff640 1 -- 172.21.15.70:0/327170642 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e78012280 con 0x562162032490 2024-04-01T01:37:36.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.757+0000 7f9e67fff640 1 -- 172.21.15.70:0/327170642 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e78023070 con 0x562162032490 2024-04-01T01:37:36.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.757+0000 7f9e67fff640 1 -- 172.21.15.70:0/327170642 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e78012280 con 0x562162032490 2024-04-01T01:37:36.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.757+0000 7f9e86646740 1 -- 172.21.15.70:0/327170642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032490 msgr2=0x5621620328b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:36.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.757+0000 7f9e86646740 1 --2- 172.21.15.70:0/327170642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032490 0x5621620328b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9e780138b0 tx=0x7f9e78013d70 comp rx=0 tx=0).stop 2024-04-01T01:37:36.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.758+0000 7f9e86646740 1 -- 172.21.15.70:0/327170642 shutdown_connections 2024-04-01T01:37:36.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.758+0000 7f9e86646740 1 --2- 172.21.15.70:0/327170642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162037ff0 0x56216203a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.758+0000 7f9e86646740 1 --2- 172.21.15.70:0/327170642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162032e80 0x562162037980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.758+0000 7f9e86646740 1 --2- 172.21.15.70:0/327170642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032490 0x5621620328b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.758+0000 7f9e86646740 1 -- 172.21.15.70:0/327170642 >> 172.21.15.70:0/327170642 conn(0x56216202da00 msgr2=0x56216202de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:36.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.758+0000 7f9e86646740 1 -- 172.21.15.70:0/327170642 shutdown_connections 2024-04-01T01:37:36.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.758+0000 7f9e86646740 1 -- 172.21.15.70:0/327170642 wait complete. 2024-04-01T01:37:36.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.758+0000 7f9e86646740 1 -- start start 2024-04-01T01:37:36.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.758+0000 7f9e86646740 1 Processor -- start 2024-04-01T01:37:36.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.759+0000 7f9e86646740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162032490 0x56216203c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.759+0000 7f9e86646740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032e80 0x56216203cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.759+0000 7f9e86646740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162037ff0 0x562162043f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.759+0000 7f9e86646740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56216203af90 con 0x562162032490 2024-04-01T01:37:36.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.759+0000 7f9e86646740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562161f354a0 con 0x562162037ff0 2024-04-01T01:37:36.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.759+0000 7f9e81006640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032e80 0x56216203cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.759+0000 7f9e86646740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56216203b110 con 0x562162032e80 2024-04-01T01:37:36.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.759+0000 7f9e81006640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032e80 0x56216203cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52554/0 (socket says 172.21.15.70:52554) 2024-04-01T01:37:36.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.759+0000 7f9e81006640 1 -- 172.21.15.70:0/1300431377 learned_addr learned my addr 172.21.15.70:0/1300431377 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:36.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.760+0000 7f9e86646740 10 client.0 _ll_register_callbacks cb 0x562161f38d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:36.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.760+0000 7f9e81807640 1 --2- 172.21.15.70:0/1300431377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162032490 0x56216203c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.760+0000 7f9e81006640 1 -- 172.21.15.70:0/1300431377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162037ff0 msgr2=0x562162043f40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:37:36.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.760+0000 7f9e81006640 1 --2- 172.21.15.70:0/1300431377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162037ff0 0x562162043f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.760+0000 7f9e81006640 1 -- 172.21.15.70:0/1300431377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162032490 msgr2=0x56216203c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:36.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.760+0000 7f9e81006640 1 --2- 172.21.15.70:0/1300431377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162032490 0x56216203c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.760+0000 7f9e81006640 1 -- 172.21.15.70:0/1300431377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562161d757d0 con 0x562162032e80 2024-04-01T01:37:36.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.760+0000 7f9e81807640 1 --2- 172.21.15.70:0/1300431377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162032490 0x56216203c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:37:36.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.760+0000 7f9e81006640 1 --2- 172.21.15.70:0/1300431377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032e80 0x56216203cf00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9e740b9020 tx=0x7f9e740bca50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=382d800e85af2c9d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:36.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.760+0000 7f9e657fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:37:36.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.760+0000 7f9e657fa640 1 -- 172.21.15.70:0/1300431377 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e740098b0 con 0x562162032e80 2024-04-01T01:37:36.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.761+0000 7f9e657fa640 1 -- 172.21.15.70:0/1300431377 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9e740b9230 con 0x562162032e80 2024-04-01T01:37:36.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.761+0000 7f9e86646740 1 -- 172.21.15.70:0/1300431377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e78023070 con 0x562162032e80 2024-04-01T01:37:36.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.761+0000 7f9e657fa640 1 -- 172.21.15.70:0/1300431377 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9e740d5050 con 0x562162032e80 2024-04-01T01:37:36.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.761+0000 7f9e86646740 10 client.26521 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:36.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.761+0000 7f9e86646740 1 -- 172.21.15.70:0/1300431377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562161cab3f0 con 0x562162032e80 2024-04-01T01:37:36.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.761+0000 7f9e657fa640 1 -- 172.21.15.70:0/1300431377 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e78023070 con 0x562162032e80 2024-04-01T01:37:36.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.761+0000 7f9e657fa640 1 -- 172.21.15.70:0/1300431377 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9e7403c030 con 0x562162032e80 2024-04-01T01:37:36.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.761+0000 7f9e86646740 10 client.26521 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:36.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.762+0000 7f9e86646740 10 client.26521 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:36.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.762+0000 7f9e86646740 20 client.26521 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:36.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.762+0000 7f9e86646740 1 --2- 172.21.15.70:0/1300431377 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5621620489f0 0x56216204ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.762+0000 7f9e86646740 4 client.26521 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:36.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.762+0000 7f9e86646740 1 -- 172.21.15.70:0/1300431377 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562161fa46c0 con 0x5621620489f0 2024-04-01T01:37:36.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.762+0000 7f9e81807640 1 --2- 172.21.15.70:0/1300431377 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5621620489f0 0x56216204ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.763+0000 7f9e81807640 1 --2- 172.21.15.70:0/1300431377 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5621620489f0 0x56216204ae70 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e71c26abb5f0af45 server_cookie=bcd6bd5df8fca610 in_seq=0 out_seq=0 2024-04-01T01:37:36.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.763+0000 7f9e657fa640 10 client.26521 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:36.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.764+0000 7f9e657fa640 1 -- 172.21.15.70:0/1300431377 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562161fa46c0 con 0x5621620489f0 2024-04-01T01:37:36.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.764+0000 7f9e657fa640 10 client.26521 handle_command_reply: tid=0 2024-04-01T01:37:36.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.765+0000 7f9e86646740 1 client.26521 shutdown 2024-04-01T01:37:36.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.765+0000 7f9e86646740 1 -- 172.21.15.70:0/1300431377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032e80 msgr2=0x56216203cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:36.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.765+0000 7f9e86646740 1 --2- 172.21.15.70:0/1300431377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032e80 0x56216203cf00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9e740b9020 tx=0x7f9e740bca50 comp rx=0 tx=0).stop 2024-04-01T01:37:36.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.765+0000 7f9e86646740 1 -- 172.21.15.70:0/1300431377 shutdown_connections 2024-04-01T01:37:36.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.765+0000 7f9e86646740 1 --2- 172.21.15.70:0/1300431377 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5621620489f0 0x56216204ae70 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.765+0000 7f9e86646740 1 --2- 172.21.15.70:0/1300431377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162037ff0 0x562162043f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.765+0000 7f9e86646740 1 --2- 172.21.15.70:0/1300431377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032e80 0x56216203cf00 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.766+0000 7f9e86646740 1 --2- 172.21.15.70:0/1300431377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162032490 0x56216203c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.766+0000 7f9e657fa640 0 client.26521 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:36.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.766+0000 7f9e86646740 1 -- 172.21.15.70:0/1300431377 >> 172.21.15.70:0/1300431377 conn(0x56216202da00 msgr2=0x56216202cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:36.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.766+0000 7f9e86646740 1 -- 172.21.15.70:0/1300431377 shutdown_connections 2024-04-01T01:37:36.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.766+0000 7f9e86646740 1 -- 172.21.15.70:0/1300431377 wait complete. 2024-04-01T01:37:36.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.766+0000 7f9e86646740 20 client.26521 trim_cache size 0 max 16384 2024-04-01T01:37:36.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.801+0000 7f9e86646740 1 Processor -- start 2024-04-01T01:37:36.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.801+0000 7f9e86646740 1 -- start start 2024-04-01T01:37:36.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.801+0000 7f9e86646740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162032490 0x562162034380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.801+0000 7f9e86646740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162032e80 0x562162043f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.802+0000 7f9e86646740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162037ff0 0x5621620348c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.802+0000 7f9e86646740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c1b07d0 con 0x562162032490 2024-04-01T01:37:36.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.802+0000 7f9e86646740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c1b2310 con 0x562162032e80 2024-04-01T01:37:36.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.802+0000 7f9e86646740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c1b0ea0 con 0x562162037ff0 2024-04-01T01:37:36.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.802+0000 7f9e82008640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162037ff0 0x5621620348c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.802+0000 7f9e82008640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162037ff0 0x5621620348c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52562/0 (socket says 172.21.15.70:52562) 2024-04-01T01:37:36.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.802+0000 7f9e82008640 1 -- 172.21.15.70:0/3371745673 learned_addr learned my addr 172.21.15.70:0/3371745673 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:36.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.802+0000 7f9e81807640 1 --2- 172.21.15.70:0/3371745673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162032490 0x562162034380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.802+0000 7f9e81006640 1 --2- 172.21.15.70:0/3371745673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162032e80 0x562162043f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.802+0000 7f9e82008640 1 -- 172.21.15.70:0/3371745673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162032e80 msgr2=0x562162043f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:36.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.802+0000 7f9e82008640 1 --2- 172.21.15.70:0/3371745673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162032e80 0x562162043f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.802+0000 7f9e82008640 1 -- 172.21.15.70:0/3371745673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162032490 msgr2=0x562162034380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:36.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.802+0000 7f9e82008640 1 --2- 172.21.15.70:0/3371745673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162032490 0x562162034380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.802+0000 7f9e82008640 1 -- 172.21.15.70:0/3371745673 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562161c6b0f0 con 0x562162037ff0 2024-04-01T01:37:36.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.803+0000 7f9e82008640 1 --2- 172.21.15.70:0/3371745673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162037ff0 0x5621620348c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9e78005c20 tx=0x7f9e78005f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59983490c8e6c76a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:36.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.803+0000 7f9e677fe640 1 -- 172.21.15.70:0/3371745673 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e78036070 con 0x562162037ff0 2024-04-01T01:37:36.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.803+0000 7f9e677fe640 1 -- 172.21.15.70:0/3371745673 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e78023070 con 0x562162037ff0 2024-04-01T01:37:36.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.803+0000 7f9e677fe640 1 -- 172.21.15.70:0/3371745673 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7801f850 con 0x562162037ff0 2024-04-01T01:37:36.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.804+0000 7f9e86646740 1 -- 172.21.15.70:0/3371745673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162037ff0 msgr2=0x5621620348c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:36.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.804+0000 7f9e86646740 1 --2- 172.21.15.70:0/3371745673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162037ff0 0x5621620348c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9e78005c20 tx=0x7f9e78005f60 comp rx=0 tx=0).stop 2024-04-01T01:37:36.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.805+0000 7f9e86646740 1 -- 172.21.15.70:0/3371745673 shutdown_connections 2024-04-01T01:37:36.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.805+0000 7f9e86646740 1 --2- 172.21.15.70:0/3371745673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162037ff0 0x5621620348c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.805+0000 7f9e86646740 1 --2- 172.21.15.70:0/3371745673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162032e80 0x562162043f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.805+0000 7f9e86646740 1 --2- 172.21.15.70:0/3371745673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162032490 0x562162034380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.805+0000 7f9e86646740 1 -- 172.21.15.70:0/3371745673 >> 172.21.15.70:0/3371745673 conn(0x56216202da00 msgr2=0x56216203c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:36.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.805+0000 7f9e86646740 1 -- 172.21.15.70:0/3371745673 shutdown_connections 2024-04-01T01:37:36.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.805+0000 7f9e86646740 1 -- 172.21.15.70:0/3371745673 wait complete. 2024-04-01T01:37:36.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.806+0000 7f9e86646740 1 -- start start 2024-04-01T01:37:36.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.806+0000 7f9e86646740 1 Processor -- start 2024-04-01T01:37:36.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.806+0000 7f9e86646740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032490 0x562162041520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.807+0000 7f9e86646740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162032e80 0x562162041a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.807+0000 7f9e81807640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032490 0x562162041520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.807+0000 7f9e81807640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032490 0x562162041520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52566/0 (socket says 172.21.15.70:52566) 2024-04-01T01:37:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.807+0000 7f9e81807640 1 -- 172.21.15.70:0/3258138557 learned_addr learned my addr 172.21.15.70:0/3258138557 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.807+0000 7f9e81006640 1 --2- 172.21.15.70:0/3258138557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162032e80 0x562162041a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.807+0000 7f9e86646740 1 --2- 172.21.15.70:0/3258138557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162037ff0 0x562162042a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.807+0000 7f9e86646740 1 -- 172.21.15.70:0/3258138557 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c1b1c40 con 0x562162037ff0 2024-04-01T01:37:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.807+0000 7f9e86646740 1 -- 172.21.15.70:0/3258138557 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c1b1570 con 0x562162032e80 2024-04-01T01:37:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.807+0000 7f9e86646740 1 -- 172.21.15.70:0/3258138557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c1ae7b0 con 0x562162032490 2024-04-01T01:37:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.807+0000 7f9e86646740 10 client.0 _ll_register_callbacks cb 0x7f9e74005ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.808+0000 7f9e82008640 1 --2- 172.21.15.70:0/3258138557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162037ff0 0x562162042a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.808+0000 7f9e81807640 1 -- 172.21.15.70:0/3258138557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162032e80 msgr2=0x562162041a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.808+0000 7f9e81807640 1 --2- 172.21.15.70:0/3258138557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162032e80 0x562162041a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.808+0000 7f9e81807640 1 -- 172.21.15.70:0/3258138557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162037ff0 msgr2=0x562162042a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.808+0000 7f9e81807640 1 --2- 172.21.15.70:0/3258138557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162037ff0 0x562162042a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.808+0000 7f9e81807640 1 -- 172.21.15.70:0/3258138557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562162043060 con 0x562162032490 2024-04-01T01:37:36.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.808+0000 7f9e82008640 1 --2- 172.21.15.70:0/3258138557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162037ff0 0x562162042a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:37:36.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.808+0000 7f9e81807640 1 --2- 172.21.15.70:0/3258138557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032490 0x562162041520 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x5621620338f0 tx=0x7f9e6c0090e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc87a43d6843c0da server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:36.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.808+0000 7f9e657fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:37:36.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.808+0000 7f9e657fa640 1 -- 172.21.15.70:0/3258138557 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c005b90 con 0x562162032490 2024-04-01T01:37:36.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.809+0000 7f9e657fa640 1 -- 172.21.15.70:0/3258138557 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9e6c005d30 con 0x562162032490 2024-04-01T01:37:36.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.809+0000 7f9e86646740 1 -- 172.21.15.70:0/3258138557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e78023070 con 0x562162032490 2024-04-01T01:37:36.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.809+0000 7f9e657fa640 1 -- 172.21.15.70:0/3258138557 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9e6c02d050 con 0x562162032490 2024-04-01T01:37:36.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.809+0000 7f9e86646740 10 client.26527 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:36.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.809+0000 7f9e86646740 1 -- 172.21.15.70:0/3258138557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5621620aedb0 con 0x562162032490 2024-04-01T01:37:36.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.810+0000 7f9e657fa640 1 -- 172.21.15.70:0/3258138557 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e78023070 con 0x562162032490 2024-04-01T01:37:36.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.811+0000 7f9e657fa640 1 -- 172.21.15.70:0/3258138557 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9e6c024430 con 0x562162032490 2024-04-01T01:37:36.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.811+0000 7f9e86646740 10 client.26527 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:36.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.811+0000 7f9e86646740 10 client.26527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:36.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.811+0000 7f9e86646740 20 client.26527 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:36.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.811+0000 7f9e86646740 1 --2- 172.21.15.70:0/3258138557 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5621620489f0 0x5621620b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.811+0000 7f9e86646740 4 client.26527 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:37:36.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.811+0000 7f9e86646740 1 -- 172.21.15.70:0/3258138557 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562161f9ea10 con 0x5621620489f0 2024-04-01T01:37:36.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.811+0000 7f9e81006640 1 --2- 172.21.15.70:0/3258138557 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5621620489f0 0x5621620b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.813+0000 7f9e81006640 1 --2- 172.21.15.70:0/3258138557 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5621620489f0 0x5621620b17a0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fa632b3f1fd889e server_cookie=dc4599c9c9d444d3 in_seq=0 out_seq=0 2024-04-01T01:37:36.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.813+0000 7f9e657fa640 10 client.26527 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.813+0000 7f9e657fa640 1 -- 172.21.15.70:0/3258138557 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562161f9ea10 con 0x5621620489f0 2024-04-01T01:37:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.814+0000 7f9e657fa640 10 client.26527 handle_command_reply: tid=0 2024-04-01T01:37:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.814+0000 7f9e86646740 1 client.26527 shutdown 2024-04-01T01:37:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.814+0000 7f9e86646740 1 -- 172.21.15.70:0/3258138557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032490 msgr2=0x562162041520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:36.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.814+0000 7f9e86646740 1 --2- 172.21.15.70:0/3258138557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032490 0x562162041520 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x5621620338f0 tx=0x7f9e6c0090e0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.814+0000 7f9e86646740 1 -- 172.21.15.70:0/3258138557 shutdown_connections 2024-04-01T01:37:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.814+0000 7f9e86646740 1 --2- 172.21.15.70:0/3258138557 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5621620489f0 0x5621620b17a0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.814+0000 7f9e86646740 1 --2- 172.21.15.70:0/3258138557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562162037ff0 0x562162042a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.814+0000 7f9e657fa640 0 client.26527 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.814+0000 7f9e86646740 1 --2- 172.21.15.70:0/3258138557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562162032e80 0x562162041a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.814+0000 7f9e86646740 1 --2- 172.21.15.70:0/3258138557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562162032490 0x562162041520 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.814+0000 7f9e86646740 1 -- 172.21.15.70:0/3258138557 >> 172.21.15.70:0/3258138557 conn(0x56216202da00 msgr2=0x56216203c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.815+0000 7f9e86646740 1 -- 172.21.15.70:0/3258138557 shutdown_connections 2024-04-01T01:37:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.815+0000 7f9e86646740 1 -- 172.21.15.70:0/3258138557 wait complete. 2024-04-01T01:37:36.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.816+0000 7f9e86646740 20 client.26527 trim_cache size 0 max 16384 2024-04-01T01:37:36.816 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:37:36.816 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:37:36.817 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:37:36.817 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:37:36.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.818+0000 7f9e82809640 1 -- 172.21.15.70:0/916390009 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e680a24a0 msgr2=0x7f9e680a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:36.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.818+0000 7f9e82809640 1 --2- 172.21.15.70:0/916390009 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e680a24a0 0x7f9e680a4960 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9e7801e000 tx=0x7f9e78020000 comp rx=0 tx=0).stop 2024-04-01T01:37:36.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.818+0000 7f9e82809640 1 -- 172.21.15.70:0/916390009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e7c077350 msgr2=0x7f9e7c19e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:36.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.818+0000 7f9e82809640 1 --2- 172.21.15.70:0/916390009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e7c077350 0x7f9e7c19e490 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9e74009dc0 tx=0x7f9e740347b0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.818+0000 7f9e82809640 1 -- 172.21.15.70:0/916390009 shutdown_connections 2024-04-01T01:37:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.818+0000 7f9e82809640 1 --2- 172.21.15.70:0/916390009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e7c077cf0 0x7f9e7c1a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.818+0000 7f9e82809640 1 --2- 172.21.15.70:0/916390009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e7c077350 0x7f9e7c19e490 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.818+0000 7f9e82809640 1 --2- 172.21.15.70:0/916390009 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e680a24a0 0x7f9e680a4960 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.818+0000 7f9e82809640 1 --2- 172.21.15.70:0/916390009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7c076110 0x7f9e7c19df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.818+0000 7f9e82809640 1 -- 172.21.15.70:0/916390009 >> 172.21.15.70:0/916390009 conn(0x7f9e7c0fedc0 msgr2=0x7f9e7c100600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:36.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.824+0000 7f9e82809640 1 -- 172.21.15.70:0/916390009 shutdown_connections 2024-04-01T01:37:36.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.824+0000 7f9e82809640 1 -- 172.21.15.70:0/916390009 wait complete. 2024-04-01T01:37:36.845 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:37:36.845 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:37:36.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:36 smithi070 ceph-mon[32179]: pgmap v482: 129 pgs: 129 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.7 MiB/s wr, 1.14k op/s 2024-04-01T01:37:36.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 170.0 s, 1053.4 tps, lat 93.490 ms stddev 33.946 2024-04-01T01:37:36.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.992+0000 7f5febfff640 1 -- 172.21.15.70:0/3001049687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fe40a4b10 msgr2=0x7f5fe40a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:36.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.992+0000 7f5febfff640 1 --2- 172.21.15.70:0/3001049687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fe40a4b10 0x7f5fe40a4f30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5fec067420 tx=0x7f5fec08f500 comp rx=0 tx=0).stop 2024-04-01T01:37:36.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.992+0000 7f5febfff640 1 -- 172.21.15.70:0/3001049687 shutdown_connections 2024-04-01T01:37:36.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.992+0000 7f5febfff640 1 --2- 172.21.15.70:0/3001049687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe40aca00 0x7f5fe40aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.992+0000 7f5febfff640 1 --2- 172.21.15.70:0/3001049687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fe40a5470 0x7f5fe40ac470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.992+0000 7f5febfff640 1 --2- 172.21.15.70:0/3001049687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fe40a4b10 0x7f5fe40a4f30 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5fec067420 tx=0x7f5fec08f500 comp rx=0 tx=0).stop 2024-04-01T01:37:36.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.992+0000 7f5febfff640 1 -- 172.21.15.70:0/3001049687 >> 172.21.15.70:0/3001049687 conn(0x7f5fe40a00d0 msgr2=0x7f5fe40a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:36.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.992+0000 7f5febfff640 1 -- 172.21.15.70:0/3001049687 shutdown_connections 2024-04-01T01:37:36.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.992+0000 7f5febfff640 1 -- 172.21.15.70:0/3001049687 wait complete. 2024-04-01T01:37:36.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.993+0000 7f5febfff640 1 Processor -- start 2024-04-01T01:37:36.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.993+0000 7f5febfff640 1 -- start start 2024-04-01T01:37:36.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.994+0000 7f5febfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fe40a5470 0x7f5fe40a7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.994+0000 7f5febfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe40aca00 0x7f5fe40a6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.994+0000 7f5febfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fe4140db0 0x7f5fe40a6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.994+0000 7f5febfff640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe40b2850 con 0x7f5fe4140db0 2024-04-01T01:37:36.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.994+0000 7f5febfff640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe40b26d0 con 0x7f5fe40aca00 2024-04-01T01:37:36.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.994+0000 7f5febfff640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe40b29d0 con 0x7f5fe40a5470 2024-04-01T01:37:36.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.994+0000 7f5fea7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe40aca00 0x7f5fe40a6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.994+0000 7f5fea7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe40aca00 0x7f5fe40a6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53522/0 (socket says 172.21.15.70:53522) 2024-04-01T01:37:36.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.994+0000 7f5fea7fc640 1 -- 172.21.15.70:0/1388385400 learned_addr learned my addr 172.21.15.70:0/1388385400 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:36.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.994+0000 7f5feaffd640 1 --2- 172.21.15.70:0/1388385400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fe40a5470 0x7f5fe40a7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.995+0000 7f5fea7fc640 1 -- 172.21.15.70:0/1388385400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fe40a5470 msgr2=0x7f5fe40a7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:36.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.995+0000 7f5fea7fc640 1 --2- 172.21.15.70:0/1388385400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fe40a5470 0x7f5fe40a7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.995+0000 7f5fea7fc640 1 -- 172.21.15.70:0/1388385400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fe4140db0 msgr2=0x7f5fe40a6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:36.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.995+0000 7f5fea7fc640 1 --2- 172.21.15.70:0/1388385400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fe4140db0 0x7f5fe40a6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:36.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.995+0000 7f5fea7fc640 1 -- 172.21.15.70:0/1388385400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fe40a71d0 con 0x7f5fe40aca00 2024-04-01T01:37:36.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.995+0000 7f5fea7fc640 1 --2- 172.21.15.70:0/1388385400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe40aca00 0x7f5fe40a6400 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00d950 tx=0x7f5fdc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:36.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.995+0000 7f5fd3fff640 1 -- 172.21.15.70:0/1388385400 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc017020 con 0x7f5fe40aca00 2024-04-01T01:37:36.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.995+0000 7f5febfff640 1 -- 172.21.15.70:0/1388385400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fe4141260 con 0x7f5fe40aca00 2024-04-01T01:37:36.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.995+0000 7f5febfff640 1 -- 172.21.15.70:0/1388385400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fe41417b0 con 0x7f5fe40aca00 2024-04-01T01:37:36.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.996+0000 7f5fd3fff640 1 -- 172.21.15.70:0/1388385400 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fdc009c30 con 0x7f5fe40aca00 2024-04-01T01:37:36.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.996+0000 7f5fd3fff640 1 -- 172.21.15.70:0/1388385400 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc00f6b0 con 0x7f5fe40aca00 2024-04-01T01:37:36.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.997+0000 7f5febfff640 1 -- 172.21.15.70:0/1388385400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fd80052c0 con 0x7f5fe40aca00 2024-04-01T01:37:36.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.999+0000 7f5fd3fff640 1 -- 172.21.15.70:0/1388385400 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f5fdc01a020 con 0x7f5fe40aca00 2024-04-01T01:37:36.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:36.999+0000 7f5fd3fff640 1 --2- 172.21.15.70:0/1388385400 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5fd40a2670 0x7f5fd40a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:36.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.000+0000 7f5feaffd640 1 --2- 172.21.15.70:0/1388385400 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5fd40a2670 0x7f5fd40a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:36.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.000+0000 7f5fd3fff640 1 -- 172.21.15.70:0/1388385400 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5fdc0ce4f0 con 0x7f5fe40aca00 2024-04-01T01:37:36.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.000+0000 7f5feaffd640 1 --2- 172.21.15.70:0/1388385400 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5fd40a2670 0x7f5fd40a4b30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5fec000b60 tx=0x7f5fec0675f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:37.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.007+0000 7f5fd3fff640 1 -- 172.21.15.70:0/1388385400 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5fdc021070 con 0x7f5fe40aca00 2024-04-01T01:37:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:36 smithi081 ceph-mon[35142]: pgmap v482: 129 pgs: 129 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.7 MiB/s wr, 1.14k op/s 2024-04-01T01:37:37.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:36 smithi189 ceph-mon[35753]: pgmap v482: 129 pgs: 129 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.7 MiB/s wr, 1.14k op/s 2024-04-01T01:37:37.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.446+0000 7f5febfff640 1 -- 172.21.15.70:0/1388385400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fd80055b0 con 0x7f5fe40aca00 2024-04-01T01:37:37.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.447+0000 7f5fd3fff640 1 -- 172.21.15.70:0/1388385400 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f5fdc009de0 con 0x7f5fe40aca00 2024-04-01T01:37:37.447 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:37:37.447 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:33:36.894838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24418,24404,24379,24451,24464]},"id":1}]} 2024-04-01T01:37:37.447 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-04-01T01:37:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.451+0000 7f5febfff640 1 -- 172.21.15.70:0/1388385400 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5fd40a2670 msgr2=0x7f5fd40a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.451+0000 7f5febfff640 1 --2- 172.21.15.70:0/1388385400 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5fd40a2670 0x7f5fd40a4b30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5fec000b60 tx=0x7f5fec0675f0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.451+0000 7f5febfff640 1 -- 172.21.15.70:0/1388385400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe40aca00 msgr2=0x7f5fe40a6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.451+0000 7f5febfff640 1 --2- 172.21.15.70:0/1388385400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe40aca00 0x7f5fe40a6400 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00d950 tx=0x7f5fdc00de10 comp rx=0 tx=0).stop 2024-04-01T01:37:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.451+0000 7f5febfff640 1 -- 172.21.15.70:0/1388385400 shutdown_connections 2024-04-01T01:37:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.451+0000 7f5febfff640 1 --2- 172.21.15.70:0/1388385400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fe4140db0 0x7f5fe40a6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.451+0000 7f5febfff640 1 --2- 172.21.15.70:0/1388385400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fe40aca00 0x7f5fe40a6400 secure :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00d950 tx=0x7f5fdc00de10 comp rx=0 tx=0).stop 2024-04-01T01:37:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.451+0000 7f5febfff640 1 --2- 172.21.15.70:0/1388385400 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5fd40a2670 0x7f5fd40a4b30 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.452+0000 7f5febfff640 1 --2- 172.21.15.70:0/1388385400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fe40a5470 0x7f5fe40a7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.452+0000 7f5febfff640 1 -- 172.21.15.70:0/1388385400 >> 172.21.15.70:0/1388385400 conn(0x7f5fe40a00d0 msgr2=0x7f5fe40a8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:37.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.452+0000 7f5febfff640 1 -- 172.21.15.70:0/1388385400 shutdown_connections 2024-04-01T01:37:37.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.452+0000 7f5febfff640 1 -- 172.21.15.70:0/1388385400 wait complete. 2024-04-01T01:37:37.471 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:37:37.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.688+0000 7fcc0e20b640 1 -- 172.21.15.70:0/2114428249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc08103850 msgr2=0x7fcc08103c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.688+0000 7fcc0e20b640 1 --2- 172.21.15.70:0/2114428249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc08103850 0x7fcc08103c70 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009900 tx=0x7fcbf802bc30 comp rx=0 tx=0).stop 2024-04-01T01:37:37.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.688+0000 7fcc0e20b640 1 -- 172.21.15.70:0/2114428249 shutdown_connections 2024-04-01T01:37:37.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.688+0000 7fcc0e20b640 1 --2- 172.21.15.70:0/2114428249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc08105400 0x7fcc0810bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.688+0000 7fcc0e20b640 1 --2- 172.21.15.70:0/2114428249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc08104a40 0x7fcc08104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.688+0000 7fcc0e20b640 1 --2- 172.21.15.70:0/2114428249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc08103850 0x7fcc08103c70 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.688+0000 7fcc0e20b640 1 -- 172.21.15.70:0/2114428249 >> 172.21.15.70:0/2114428249 conn(0x7fcc080fedc0 msgr2=0x7fcc08101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:37.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.689+0000 7fcc0e20b640 1 -- 172.21.15.70:0/2114428249 shutdown_connections 2024-04-01T01:37:37.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.689+0000 7fcc0e20b640 1 -- 172.21.15.70:0/2114428249 wait complete. 2024-04-01T01:37:37.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.689+0000 7fcc0e20b640 1 Processor -- start 2024-04-01T01:37:37.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.690+0000 7fcc0e20b640 1 -- start start 2024-04-01T01:37:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.690+0000 7fcc0e20b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc08103850 0x7fcc0819de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.690+0000 7fcc0e20b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc08104a40 0x7fcc0819e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.690+0000 7fcc0e20b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc08105400 0x7fcc081a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.690+0000 7fcc0e20b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0810f3a0 con 0x7fcc08103850 2024-04-01T01:37:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.690+0000 7fcc0e20b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0810f220 con 0x7fcc08104a40 2024-04-01T01:37:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.690+0000 7fcc0e20b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0810f520 con 0x7fcc08105400 2024-04-01T01:37:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.690+0000 7fcc0ca08640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc08104a40 0x7fcc0819e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.690+0000 7fcc0ca08640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc08104a40 0x7fcc0819e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53544/0 (socket says 172.21.15.70:53544) 2024-04-01T01:37:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.690+0000 7fcc0ca08640 1 -- 172.21.15.70:0/117440924 learned_addr learned my addr 172.21.15.70:0/117440924 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.691+0000 7fcc0d209640 1 --2- 172.21.15.70:0/117440924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc08103850 0x7fcc0819de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.691+0000 7fcc0da0a640 1 --2- 172.21.15.70:0/117440924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc08105400 0x7fcc081a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.691+0000 7fcc0ca08640 1 -- 172.21.15.70:0/117440924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc08105400 msgr2=0x7fcc081a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.691+0000 7fcc0ca08640 1 --2- 172.21.15.70:0/117440924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc08105400 0x7fcc081a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.691+0000 7fcc0ca08640 1 -- 172.21.15.70:0/117440924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc08103850 msgr2=0x7fcc0819de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.691+0000 7fcc0ca08640 1 --2- 172.21.15.70:0/117440924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc08103850 0x7fcc0819de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.691+0000 7fcc0ca08640 1 -- 172.21.15.70:0/117440924 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc081a5ab0 con 0x7fcc08104a40 2024-04-01T01:37:37.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.695+0000 7fcc0ca08640 1 --2- 172.21.15.70:0/117440924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc08104a40 0x7fcc0819e3d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400e9f0 tx=0x7fcc0400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:37.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.695+0000 7fcbfe7fc640 1 -- 172.21.15.70:0/117440924 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0400cd80 con 0x7fcc08104a40 2024-04-01T01:37:37.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.695+0000 7fcc0e20b640 1 -- 172.21.15.70:0/117440924 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc081a5da0 con 0x7fcc08104a40 2024-04-01T01:37:37.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.695+0000 7fcc0e20b640 1 -- 172.21.15.70:0/117440924 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc081a6270 con 0x7fcc08104a40 2024-04-01T01:37:37.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.696+0000 7fcbfe7fc640 1 -- 172.21.15.70:0/117440924 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc040108e0 con 0x7fcc08104a40 2024-04-01T01:37:37.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.696+0000 7fcbfe7fc640 1 -- 172.21.15.70:0/117440924 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc04018920 con 0x7fcc08104a40 2024-04-01T01:37:37.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.699+0000 7fcc12045740 1 Processor -- start 2024-04-01T01:37:37.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.701+0000 7fcbfe7fc640 1 -- 172.21.15.70:0/117440924 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fcc04010430 con 0x7fcc08104a40 2024-04-01T01:37:37.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.702+0000 7fcbfe7fc640 1 --2- 172.21.15.70:0/117440924 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcbf40a24a0 0x7fcbf40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.702+0000 7fcbfe7fc640 1 -- 172.21.15.70:0/117440924 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fcc040cee10 con 0x7fcc08104a40 2024-04-01T01:37:37.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.703+0000 7fcc0da0a640 1 --2- 172.21.15.70:0/117440924 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcbf40a24a0 0x7fcbf40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.703+0000 7fcc12045740 1 -- start start 2024-04-01T01:37:37.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.703+0000 7fcc12045740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c26a0 0x564cb96c2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.703+0000 7fcc12045740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 0x564cb96d8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.703+0000 7fcc12045740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96d9210 0x564cb96db6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.703+0000 7fcc12045740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564cb96271b0 con 0x564cb96d9210 2024-04-01T01:37:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.703+0000 7fcc12045740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x564cb95cfc00 con 0x564cb96c26a0 2024-04-01T01:37:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.703+0000 7fcc12045740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564cb9623250 con 0x564cb96c3090 2024-04-01T01:37:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.704+0000 7fcc0da0a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96d9210 0x564cb96db6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.704+0000 7fcc0da0a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96d9210 0x564cb96db6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47360/0 (socket says 172.21.15.70:47360) 2024-04-01T01:37:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.704+0000 7fcc0da0a640 1 -- 172.21.15.70:0/1103987467 learned_addr learned my addr 172.21.15.70:0/1103987467 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.704+0000 7fcc0d209640 1 --2- 172.21.15.70:0/1103987467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c26a0 0x564cb96c2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.704+0000 7fcc0ca08640 1 --2- 172.21.15.70:0/1103987467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 0x564cb96d8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.704+0000 7fcc0d209640 1 -- 172.21.15.70:0/1103987467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 msgr2=0x564cb96d8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.704+0000 7fcc0d209640 1 --2- 172.21.15.70:0/1103987467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 0x564cb96d8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.704+0000 7fcc0d209640 1 -- 172.21.15.70:0/1103987467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96d9210 msgr2=0x564cb96db6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.704+0000 7fcc0d209640 1 --2- 172.21.15.70:0/1103987467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96d9210 0x564cb96db6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.704+0000 7fcc0d209640 1 -- 172.21.15.70:0/1103987467 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cb95ef3d0 con 0x564cb96c26a0 2024-04-01T01:37:37.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.705+0000 7fcc0da0a640 1 --2- 172.21.15.70:0/117440924 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcbf40a24a0 0x7fcbf40a4960 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcc00005fd0 tx=0x7fcc00009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.705+0000 7fcc0d209640 1 --2- 172.21.15.70:0/1103987467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c26a0 0x564cb96c2ac0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8036040 tx=0x7fcbf8009600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e6c36150af6beb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.705+0000 7fcbf37fe640 1 -- 172.21.15.70:0/1103987467 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8044020 con 0x564cb96c26a0 2024-04-01T01:37:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.705+0000 7fcbf37fe640 1 -- 172.21.15.70:0/1103987467 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcbf802cba0 con 0x564cb96c26a0 2024-04-01T01:37:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.706+0000 7fcbf37fe640 1 -- 172.21.15.70:0/1103987467 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf80046c0 con 0x564cb96c26a0 2024-04-01T01:37:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.706+0000 7fcc12045740 1 -- 172.21.15.70:0/1103987467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c26a0 msgr2=0x564cb96c2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.706+0000 7fcc12045740 1 --2- 172.21.15.70:0/1103987467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c26a0 0x564cb96c2ac0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8036040 tx=0x7fcbf8009600 comp rx=0 tx=0).stop 2024-04-01T01:37:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.706+0000 7fcc12045740 1 -- 172.21.15.70:0/1103987467 shutdown_connections 2024-04-01T01:37:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.706+0000 7fcc12045740 1 --2- 172.21.15.70:0/1103987467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96d9210 0x564cb96db6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.706+0000 7fcc12045740 1 --2- 172.21.15.70:0/1103987467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 0x564cb96d8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.706+0000 7fcc12045740 1 --2- 172.21.15.70:0/1103987467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c26a0 0x564cb96c2ac0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.706+0000 7fcc12045740 1 -- 172.21.15.70:0/1103987467 >> 172.21.15.70:0/1103987467 conn(0x564cb96bdc10 msgr2=0x564cb96be010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.707+0000 7fcc12045740 1 -- 172.21.15.70:0/1103987467 shutdown_connections 2024-04-01T01:37:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.707+0000 7fcc12045740 1 -- 172.21.15.70:0/1103987467 wait complete. 2024-04-01T01:37:37.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.707+0000 7fcc12045740 1 -- start start 2024-04-01T01:37:37.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.707+0000 7fcc12045740 1 Processor -- start 2024-04-01T01:37:37.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.708+0000 7fcc12045740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96c26a0 0x564cb96ddbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.708+0000 7fcc0d209640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96c26a0 0x564cb96ddbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.708+0000 7fcc0d209640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96c26a0 0x564cb96ddbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47374/0 (socket says 172.21.15.70:47374) 2024-04-01T01:37:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.708+0000 7fcc12045740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c3090 0x564cb96de0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.708+0000 7fcc0d209640 1 -- 172.21.15.70:0/2460501709 learned_addr learned my addr 172.21.15.70:0/2460501709 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.708+0000 7fcc12045740 1 --2- 172.21.15.70:0/2460501709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96d9210 0x564cb96e5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.708+0000 7fcc12045740 1 -- 172.21.15.70:0/2460501709 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564cb96dc1b0 con 0x564cb96c26a0 2024-04-01T01:37:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.708+0000 7fcc12045740 1 -- 172.21.15.70:0/2460501709 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x564cb95c5470 con 0x564cb96c3090 2024-04-01T01:37:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.708+0000 7fcc12045740 1 -- 172.21.15.70:0/2460501709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564cb96dc330 con 0x564cb96d9210 2024-04-01T01:37:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.708+0000 7fcc12045740 10 client.0 _ll_register_callbacks cb 0x564cb95e2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.708+0000 7fcc0da0a640 1 --2- 172.21.15.70:0/2460501709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96d9210 0x564cb96e5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.709+0000 7fcc0da0a640 1 -- 172.21.15.70:0/2460501709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c3090 msgr2=0x564cb96de0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:37:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.709+0000 7fcc0da0a640 1 --2- 172.21.15.70:0/2460501709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c3090 0x564cb96de0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.709+0000 7fcc0da0a640 1 -- 172.21.15.70:0/2460501709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96c26a0 msgr2=0x564cb96ddbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.709+0000 7fcc0da0a640 1 --2- 172.21.15.70:0/2460501709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96c26a0 0x564cb96ddbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.709+0000 7fcc0da0a640 1 -- 172.21.15.70:0/2460501709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cb933b3f0 con 0x564cb96d9210 2024-04-01T01:37:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.709+0000 7fcc0da0a640 1 --2- 172.21.15.70:0/2460501709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96d9210 0x564cb96e5130 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcc00005f10 tx=0x7fcc00004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d59daaf96e2bda4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.709+0000 7fcbf0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:37:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.709+0000 7fcbf0ff9640 1 -- 172.21.15.70:0/2460501709 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0001ee90 con 0x564cb96d9210 2024-04-01T01:37:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.709+0000 7fcc12045740 1 -- 172.21.15.70:0/2460501709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbf802cba0 con 0x564cb96d9210 2024-04-01T01:37:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.709+0000 7fcbf0ff9640 1 -- 172.21.15.70:0/2460501709 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fcc0000ec50 con 0x564cb96d9210 2024-04-01T01:37:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.709+0000 7fcbf0ff9640 1 -- 172.21.15.70:0/2460501709 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcc00030050 con 0x564cb96d9210 2024-04-01T01:37:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.710+0000 7fcc12045740 10 client.26545 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.710+0000 7fcc12045740 1 -- 172.21.15.70:0/2460501709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564cb94057d0 con 0x564cb96d9210 2024-04-01T01:37:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.710+0000 7fcbf0ff9640 1 -- 172.21.15.70:0/2460501709 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc0000ee50 con 0x564cb96d9210 2024-04-01T01:37:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.710+0000 7fcbf0ff9640 1 -- 172.21.15.70:0/2460501709 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fcc00004340 con 0x564cb96d9210 2024-04-01T01:37:37.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.710+0000 7fcc12045740 10 client.26545 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.710+0000 7fcc12045740 10 client.26545 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:37:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.710+0000 7fcc12045740 20 client.26545 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.710+0000 7fcc12045740 1 --2- 172.21.15.70:0/2460501709 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x564cb96e97c0 0x564cb96ebc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.710+0000 7fcc12045740 4 client.26545 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.710+0000 7fcc12045740 1 -- 172.21.15.70:0/2460501709 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcbf802cba0 con 0x564cb96e97c0 2024-04-01T01:37:37.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.711+0000 7fcc0d209640 1 --2- 172.21.15.70:0/2460501709 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x564cb96e97c0 0x564cb96ebc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.711+0000 7fcc0d209640 1 --2- 172.21.15.70:0/2460501709 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x564cb96e97c0 0x564cb96ebc20 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7c5d6c77418448db server_cookie=8afb78bc5b7bb618 in_seq=0 out_seq=0 2024-04-01T01:37:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.711+0000 7fcbf0ff9640 10 client.26545 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:37:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.712+0000 7fcbf0ff9640 1 -- 172.21.15.70:0/2460501709 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fcbf802cba0 con 0x564cb96e97c0 2024-04-01T01:37:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.712+0000 7fcbf0ff9640 10 client.26545 handle_command_reply: tid=0 2024-04-01T01:37:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.712+0000 7fcc12045740 1 client.26545 shutdown 2024-04-01T01:37:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.713+0000 7fcc12045740 1 -- 172.21.15.70:0/2460501709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96d9210 msgr2=0x564cb96e5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.713+0000 7fcc12045740 1 --2- 172.21.15.70:0/2460501709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96d9210 0x564cb96e5130 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcc00005f10 tx=0x7fcc00004ab0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.714+0000 7fcc12045740 1 -- 172.21.15.70:0/2460501709 shutdown_connections 2024-04-01T01:37:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.714+0000 7fcc12045740 1 --2- 172.21.15.70:0/2460501709 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x564cb96e97c0 0x564cb96ebc20 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.714+0000 7fcbf0ff9640 0 client.26545 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:37:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.714+0000 7fcc12045740 1 --2- 172.21.15.70:0/2460501709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96d9210 0x564cb96e5130 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.714+0000 7fcc12045740 1 --2- 172.21.15.70:0/2460501709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c3090 0x564cb96de0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.714+0000 7fcc12045740 1 --2- 172.21.15.70:0/2460501709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96c26a0 0x564cb96ddbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.714+0000 7fcc12045740 1 -- 172.21.15.70:0/2460501709 >> 172.21.15.70:0/2460501709 conn(0x564cb96bdc10 msgr2=0x564cb96bcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.715+0000 7fcc12045740 1 -- 172.21.15.70:0/2460501709 shutdown_connections 2024-04-01T01:37:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.715+0000 7fcc12045740 1 -- 172.21.15.70:0/2460501709 wait complete. 2024-04-01T01:37:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.715+0000 7fcc12045740 20 client.26545 trim_cache size 0 max 16384 2024-04-01T01:37:37.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.737+0000 7fcc12045740 1 Processor -- start 2024-04-01T01:37:37.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.737+0000 7fcc12045740 1 -- start start 2024-04-01T01:37:37.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.737+0000 7fcc12045740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c26a0 0x564cb96d8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.737+0000 7fcc12045740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 0x564cb96d8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.737+0000 7fcc12045740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96d9210 0x564cb96e5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.737+0000 7fcc12045740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec00fcc0 con 0x564cb96d9210 2024-04-01T01:37:37.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.737+0000 7fcc12045740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec011800 con 0x564cb96c26a0 2024-04-01T01:37:37.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.737+0000 7fcc12045740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec010390 con 0x564cb96c3090 2024-04-01T01:37:37.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.738+0000 7fcc0da0a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96d9210 0x564cb96e5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.738+0000 7fcc0ca08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 0x564cb96d8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.738+0000 7fcc0d209640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c26a0 0x564cb96d8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.738+0000 7fcc0d209640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c26a0 0x564cb96d8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53582/0 (socket says 172.21.15.70:53582) 2024-04-01T01:37:37.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.738+0000 7fcc0d209640 1 -- 172.21.15.70:0/646688149 learned_addr learned my addr 172.21.15.70:0/646688149 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:37.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.738+0000 7fcc0da0a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96d9210 0x564cb96e5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47376/0 (socket says 172.21.15.70:47376) 2024-04-01T01:37:37.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.738+0000 7fcc0ca08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 0x564cb96d8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52622/0 (socket says 172.21.15.70:52622) 2024-04-01T01:37:37.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.738+0000 7fcc0ca08640 1 -- 172.21.15.70:0/646688149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c26a0 msgr2=0x564cb96d8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:37.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.738+0000 7fcc0ca08640 1 --2- 172.21.15.70:0/646688149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c26a0 0x564cb96d8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.738+0000 7fcc0ca08640 1 -- 172.21.15.70:0/646688149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96d9210 msgr2=0x564cb96e5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:37.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.738+0000 7fcc0ca08640 1 --2- 172.21.15.70:0/646688149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96d9210 0x564cb96e5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.738+0000 7fcc0ca08640 1 -- 172.21.15.70:0/646688149 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cb96aee60 con 0x564cb96c3090 2024-04-01T01:37:37.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.738+0000 7fcc0da0a640 1 --2- 172.21.15.70:0/646688149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96d9210 0x564cb96e5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:37.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.738+0000 7fcc0d209640 1 --2- 172.21.15.70:0/646688149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c26a0 0x564cb96d8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:37.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.739+0000 7fcc0ca08640 1 --2- 172.21.15.70:0/646688149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 0x564cb96d8a90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcc040cc8b0 tx=0x7fcc04010720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e784cc8be1124270 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:37.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.739+0000 7fcbf2ffd640 1 -- 172.21.15.70:0/646688149 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0409f650 con 0x564cb96c3090 2024-04-01T01:37:37.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.739+0000 7fcbf2ffd640 1 -- 172.21.15.70:0/646688149 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc040a3f10 con 0x564cb96c3090 2024-04-01T01:37:37.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.739+0000 7fcbf2ffd640 1 -- 172.21.15.70:0/646688149 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc04097250 con 0x564cb96c3090 2024-04-01T01:37:37.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.740+0000 7fcc12045740 1 -- 172.21.15.70:0/646688149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 msgr2=0x564cb96d8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.740+0000 7fcc12045740 1 --2- 172.21.15.70:0/646688149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 0x564cb96d8a90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcc040cc8b0 tx=0x7fcc04010720 comp rx=0 tx=0).stop 2024-04-01T01:37:37.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.740+0000 7fcc12045740 1 -- 172.21.15.70:0/646688149 shutdown_connections 2024-04-01T01:37:37.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.740+0000 7fcc12045740 1 --2- 172.21.15.70:0/646688149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96d9210 0x564cb96e5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.740+0000 7fcc12045740 1 --2- 172.21.15.70:0/646688149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 0x564cb96d8a90 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.740+0000 7fcc12045740 1 --2- 172.21.15.70:0/646688149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96c26a0 0x564cb96d8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.740+0000 7fcc12045740 1 -- 172.21.15.70:0/646688149 >> 172.21.15.70:0/646688149 conn(0x564cb96bdc10 msgr2=0x564cb96bdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:37.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.740+0000 7fcc12045740 1 -- 172.21.15.70:0/646688149 shutdown_connections 2024-04-01T01:37:37.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.740+0000 7fcc12045740 1 -- 172.21.15.70:0/646688149 wait complete. 2024-04-01T01:37:37.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.740+0000 7fcc12045740 1 -- start start 2024-04-01T01:37:37.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.740+0000 7fcc12045740 1 Processor -- start 2024-04-01T01:37:37.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc12045740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96c26a0 0x564cb96e29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc12045740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 0x564cb96e2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc12045740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96d9210 0x564cb96e3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc12045740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec011130 con 0x564cb96c26a0 2024-04-01T01:37:37.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc12045740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec010a60 con 0x564cb96d9210 2024-04-01T01:37:37.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc12045740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcbec00dca0 con 0x564cb96c3090 2024-04-01T01:37:37.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc0d209640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96c26a0 0x564cb96e29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc12045740 10 client.0 _ll_register_callbacks cb 0x7fcc0000e900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:37.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc0d209640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96c26a0 0x564cb96e29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47392/0 (socket says 172.21.15.70:47392) 2024-04-01T01:37:37.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc0d209640 1 -- 172.21.15.70:0/3497824452 learned_addr learned my addr 172.21.15.70:0/3497824452 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:37.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc0ca08640 1 --2- 172.21.15.70:0/3497824452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 0x564cb96e2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc0da0a640 1 --2- 172.21.15.70:0/3497824452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96d9210 0x564cb96e3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc0d209640 1 -- 172.21.15.70:0/3497824452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 msgr2=0x564cb96e2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:37.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc0d209640 1 --2- 172.21.15.70:0/3497824452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 0x564cb96e2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc0d209640 1 -- 172.21.15.70:0/3497824452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96d9210 msgr2=0x564cb96e3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:37.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc0d209640 1 --2- 172.21.15.70:0/3497824452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96d9210 0x564cb96e3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc0d209640 1 -- 172.21.15.70:0/3497824452 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cb96e4530 con 0x564cb96c26a0 2024-04-01T01:37:37.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc0d209640 1 --2- 172.21.15.70:0/3497824452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96c26a0 0x564cb96e29f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcbf802c240 tx=0x7fcbf8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba6d6161dc8267e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:37.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcbf0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:37:37.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcbf0ff9640 1 -- 172.21.15.70:0/3497824452 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8044040 con 0x564cb96c26a0 2024-04-01T01:37:37.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcbf0ff9640 1 -- 172.21.15.70:0/3497824452 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fcbf8004040 con 0x564cb96c26a0 2024-04-01T01:37:37.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcc12045740 1 -- 172.21.15.70:0/3497824452 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc040a3f10 con 0x564cb96c26a0 2024-04-01T01:37:37.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.741+0000 7fcbf0ff9640 1 -- 172.21.15.70:0/3497824452 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcbf805d050 con 0x564cb96c26a0 2024-04-01T01:37:37.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.742+0000 7fcc12045740 10 client.16797 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:37.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.742+0000 7fcc12045740 1 -- 172.21.15.70:0/3497824452 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564cb974fe10 con 0x564cb96c26a0 2024-04-01T01:37:37.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.742+0000 7fcbf0ff9640 1 -- 172.21.15.70:0/3497824452 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc040a3f10 con 0x564cb96c26a0 2024-04-01T01:37:37.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.742+0000 7fcbf0ff9640 1 -- 172.21.15.70:0/3497824452 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fcbf8003b50 con 0x564cb96c26a0 2024-04-01T01:37:37.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.742+0000 7fcc12045740 10 client.16797 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:37.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.742+0000 7fcc12045740 10 client.16797 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:37:37.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.742+0000 7fcc12045740 20 client.16797 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:37.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.742+0000 7fcc12045740 1 --2- 172.21.15.70:0/3497824452 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x564cb96e97c0 0x564cb9752450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.742+0000 7fcc12045740 4 client.16797 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:37:37.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.742+0000 7fcc12045740 1 -- 172.21.15.70:0/3497824452 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564cb96346f0 con 0x564cb96e97c0 2024-04-01T01:37:37.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.742+0000 7fcc0ca08640 1 --2- 172.21.15.70:0/3497824452 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x564cb96e97c0 0x564cb9752450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcbf0ff9640 10 client.16797 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:37:37.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcc0ca08640 1 --2- 172.21.15.70:0/3497824452 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x564cb96e97c0 0x564cb9752450 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f1d48ff22d1e3c25 server_cookie=7e8e648172991bfe in_seq=0 out_seq=0 2024-04-01T01:37:37.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcbf0ff9640 1 -- 172.21.15.70:0/3497824452 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564cb96346f0 con 0x564cb96e97c0 2024-04-01T01:37:37.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcbf0ff9640 10 client.16797 handle_command_reply: tid=0 2024-04-01T01:37:37.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcc12045740 1 client.16797 shutdown 2024-04-01T01:37:37.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcc12045740 1 -- 172.21.15.70:0/3497824452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96c26a0 msgr2=0x564cb96e29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcc12045740 1 --2- 172.21.15.70:0/3497824452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96c26a0 0x564cb96e29f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcbf802c240 tx=0x7fcbf8000bc0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcc12045740 1 -- 172.21.15.70:0/3497824452 shutdown_connections 2024-04-01T01:37:37.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcc12045740 1 --2- 172.21.15.70:0/3497824452 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x564cb96e97c0 0x564cb9752450 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcbf0ff9640 0 client.16797 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:37:37.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcc12045740 1 --2- 172.21.15.70:0/3497824452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb96d9210 0x564cb96e3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcc12045740 1 --2- 172.21.15.70:0/3497824452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb96c3090 0x564cb96e2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcc12045740 1 --2- 172.21.15.70:0/3497824452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb96c26a0 0x564cb96e29f0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcc12045740 1 -- 172.21.15.70:0/3497824452 >> 172.21.15.70:0/3497824452 conn(0x564cb96bdc10 msgr2=0x564cb96ddb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:37.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcc12045740 1 -- 172.21.15.70:0/3497824452 shutdown_connections 2024-04-01T01:37:37.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcc12045740 1 -- 172.21.15.70:0/3497824452 wait complete. 2024-04-01T01:37:37.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.743+0000 7fcc12045740 20 client.16797 trim_cache size 0 max 16384 2024-04-01T01:37:37.749 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:37:37.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.744+0000 7fcc0e20b640 1 -- 172.21.15.70:0/117440924 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcbf40a24a0 msgr2=0x7fcbf40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.744+0000 7fcc0e20b640 1 --2- 172.21.15.70:0/117440924 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcbf40a24a0 0x7fcbf40a4960 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fcc00005fd0 tx=0x7fcc00009500 comp rx=0 tx=0).stop 2024-04-01T01:37:37.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.744+0000 7fcc0e20b640 1 -- 172.21.15.70:0/117440924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc08104a40 msgr2=0x7fcc0819e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.744+0000 7fcc0e20b640 1 --2- 172.21.15.70:0/117440924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc08104a40 0x7fcc0819e3d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400e9f0 tx=0x7fcc0400eeb0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.744+0000 7fcc0e20b640 1 -- 172.21.15.70:0/117440924 shutdown_connections 2024-04-01T01:37:37.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.744+0000 7fcc0e20b640 1 --2- 172.21.15.70:0/117440924 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcbf40a24a0 0x7fcbf40a4960 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.744+0000 7fcc0e20b640 1 --2- 172.21.15.70:0/117440924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc08105400 0x7fcc081a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.744+0000 7fcc0e20b640 1 --2- 172.21.15.70:0/117440924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc08104a40 0x7fcc0819e3d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.744+0000 7fcc0e20b640 1 --2- 172.21.15.70:0/117440924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc08103850 0x7fcc0819de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.744+0000 7fcc0e20b640 1 -- 172.21.15.70:0/117440924 >> 172.21.15.70:0/117440924 conn(0x7fcc080fedc0 msgr2=0x7fcc08077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:37.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.744+0000 7fcc0e20b640 1 -- 172.21.15.70:0/117440924 shutdown_connections 2024-04-01T01:37:37.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.744+0000 7fcc0e20b640 1 -- 172.21.15.70:0/117440924 wait complete. 2024-04-01T01:37:37.754 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:37:37.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.913+0000 7f418fb93640 1 -- 172.21.15.70:0/4169839934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4188100740 msgr2=0x7f4188100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.913+0000 7f418fb93640 1 --2- 172.21.15.70:0/4169839934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4188100740 0x7f4188100ba0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4184009900 tx=0x7f418402bc30 comp rx=0 tx=0).stop 2024-04-01T01:37:37.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.914+0000 7f418fb93640 1 -- 172.21.15.70:0/4169839934 shutdown_connections 2024-04-01T01:37:37.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.914+0000 7f418fb93640 1 --2- 172.21.15.70:0/4169839934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41881010e0 0x7f4188101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.914+0000 7f418fb93640 1 --2- 172.21.15.70:0/4169839934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4188100740 0x7f4188100ba0 secure :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4184009900 tx=0x7f418402bc30 comp rx=0 tx=0).stop 2024-04-01T01:37:37.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.914+0000 7f418fb93640 1 --2- 172.21.15.70:0/4169839934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41880ff610 0x7f41880ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.914+0000 7f418fb93640 1 -- 172.21.15.70:0/4169839934 >> 172.21.15.70:0/4169839934 conn(0x7f41880faf10 msgr2=0x7f41880fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:37.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.914+0000 7f418fb93640 1 -- 172.21.15.70:0/4169839934 shutdown_connections 2024-04-01T01:37:37.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.914+0000 7f418fb93640 1 -- 172.21.15.70:0/4169839934 wait complete. 2024-04-01T01:37:37.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.915+0000 7f418fb93640 1 Processor -- start 2024-04-01T01:37:37.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.915+0000 7f418fb93640 1 -- start start 2024-04-01T01:37:37.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.915+0000 7f418fb93640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41880ff610 0x7f4188106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.915+0000 7f418fb93640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41881010e0 0x7f4188106cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.915+0000 7f418fb93640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4188103250 0x7f41881036e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.915+0000 7f418fb93640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f418810f4c0 con 0x7f41880ff610 2024-04-01T01:37:37.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.915+0000 7f418fb93640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f418810f340 con 0x7f41881010e0 2024-04-01T01:37:37.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.915+0000 7f418fb93640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f418810f640 con 0x7f4188103250 2024-04-01T01:37:37.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.916+0000 7f418d12e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41881010e0 0x7f4188106cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.916+0000 7f418d12e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41881010e0 0x7f4188106cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53600/0 (socket says 172.21.15.70:53600) 2024-04-01T01:37:37.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.916+0000 7f418d12e640 1 -- 172.21.15.70:0/3741133278 learned_addr learned my addr 172.21.15.70:0/3741133278 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:37.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.916+0000 7f418e130640 1 --2- 172.21.15.70:0/3741133278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4188103250 0x7f41881036e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.916+0000 7f418d12e640 1 -- 172.21.15.70:0/3741133278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4188103250 msgr2=0x7f41881036e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.916+0000 7f418d12e640 1 --2- 172.21.15.70:0/3741133278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4188103250 0x7f41881036e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.916+0000 7f418d12e640 1 -- 172.21.15.70:0/3741133278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41880ff610 msgr2=0x7f4188106790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:37:37.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.916+0000 7f418d12e640 1 --2- 172.21.15.70:0/3741133278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41880ff610 0x7f4188106790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.917+0000 7f418d12e640 1 -- 172.21.15.70:0/3741133278 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4188199a00 con 0x7f41881010e0 2024-04-01T01:37:37.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.917+0000 7f418e130640 1 --2- 172.21.15.70:0/3741133278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4188103250 0x7f41881036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:37.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.917+0000 7f418d12e640 1 --2- 172.21.15.70:0/3741133278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41881010e0 0x7f4188106cd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f418402c1c0 tx=0x7f418402c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:37.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.918+0000 7f417effd640 1 -- 172.21.15.70:0/3741133278 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4184043020 con 0x7f41881010e0 2024-04-01T01:37:37.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.918+0000 7f417effd640 1 -- 172.21.15.70:0/3741133278 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41840043d0 con 0x7f41881010e0 2024-04-01T01:37:37.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.918+0000 7f417effd640 1 -- 172.21.15.70:0/3741133278 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4184003ac0 con 0x7f41881010e0 2024-04-01T01:37:37.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.918+0000 7f418fb93640 1 -- 172.21.15.70:0/3741133278 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4188199c90 con 0x7f41881010e0 2024-04-01T01:37:37.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.918+0000 7f418fb93640 1 -- 172.21.15.70:0/3741133278 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f418819a160 con 0x7f41881010e0 2024-04-01T01:37:37.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.922+0000 7f4192779740 1 Processor -- start 2024-04-01T01:37:37.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.923+0000 7f4192779740 1 -- start start 2024-04-01T01:37:37.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.923+0000 7f4192779740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617ac490 0x55e1617ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.923+0000 7f4192779740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ace80 0x55e1617b1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.923+0000 7f4192779740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617b1ff0 0x55e1617b4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.923+0000 7f4192779740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e1617111b0 con 0x55e1617ac490 2024-04-01T01:37:37.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.923+0000 7f4192779740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e1616b9c00 con 0x55e1617b1ff0 2024-04-01T01:37:37.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.923+0000 7f4192779740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e16170d250 con 0x55e1617ace80 2024-04-01T01:37:37.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.923+0000 7f418e130640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617ac490 0x55e1617ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.923+0000 7f417effd640 1 -- 172.21.15.70:0/3741133278 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f4184046020 con 0x7f41881010e0 2024-04-01T01:37:37.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.923+0000 7f418e130640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617ac490 0x55e1617ac8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47414/0 (socket says 172.21.15.70:47414) 2024-04-01T01:37:37.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.923+0000 7f418e130640 1 -- 172.21.15.70:0/1069316697 learned_addr learned my addr 172.21.15.70:0/1069316697 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:37.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.923+0000 7f418d92f640 1 --2- 172.21.15.70:0/1069316697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ace80 0x55e1617b1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.924+0000 7f418d12e640 1 --2- 172.21.15.70:0/1069316697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617b1ff0 0x55e1617b4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.924+0000 7f418d92f640 1 -- 172.21.15.70:0/1069316697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617b1ff0 msgr2=0x55e1617b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:37.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.924+0000 7f418d92f640 1 --2- 172.21.15.70:0/1069316697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617b1ff0 0x55e1617b4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.924+0000 7f418d92f640 1 -- 172.21.15.70:0/1069316697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617ac490 msgr2=0x55e1617ac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:37.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.924+0000 7f418d92f640 1 --2- 172.21.15.70:0/1069316697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617ac490 0x55e1617ac8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.924+0000 7f418d92f640 1 -- 172.21.15.70:0/1069316697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1616d93d0 con 0x55e1617ace80 2024-04-01T01:37:37.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.924+0000 7f418e130640 1 --2- 172.21.15.70:0/1069316697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617ac490 0x55e1617ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:37:37.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.924+0000 7f418d92f640 1 --2- 172.21.15.70:0/1069316697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ace80 0x55e1617b1980 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4180009d70 tx=0x7f418000e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1cdc5c231b13a3e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:37.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.925+0000 7f4173fff640 1 -- 172.21.15.70:0/1069316697 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180004b20 con 0x55e1617ace80 2024-04-01T01:37:37.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.925+0000 7f4173fff640 1 -- 172.21.15.70:0/1069316697 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4180003e20 con 0x55e1617ace80 2024-04-01T01:37:37.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.925+0000 7f4173fff640 1 -- 172.21.15.70:0/1069316697 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180005020 con 0x55e1617ace80 2024-04-01T01:37:37.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.925+0000 7f417effd640 1 --2- 172.21.15.70:0/3741133278 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41740a26a0 0x7f41740a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.926+0000 7f418e130640 1 --2- 172.21.15.70:0/3741133278 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41740a26a0 0x7f41740a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.926+0000 7f417effd640 1 -- 172.21.15.70:0/3741133278 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f41840f0b20 con 0x7f41881010e0 2024-04-01T01:37:37.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.926+0000 7f4192779740 1 -- 172.21.15.70:0/1069316697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ace80 msgr2=0x55e1617b1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.926+0000 7f4192779740 1 --2- 172.21.15.70:0/1069316697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ace80 0x55e1617b1980 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4180009d70 tx=0x7f418000e720 comp rx=0 tx=0).stop 2024-04-01T01:37:37.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.926+0000 7f4192779740 1 -- 172.21.15.70:0/1069316697 shutdown_connections 2024-04-01T01:37:37.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.926+0000 7f4192779740 1 --2- 172.21.15.70:0/1069316697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617b1ff0 0x55e1617b4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.926+0000 7f4192779740 1 --2- 172.21.15.70:0/1069316697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ace80 0x55e1617b1980 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.926+0000 7f4192779740 1 --2- 172.21.15.70:0/1069316697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617ac490 0x55e1617ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.926+0000 7f4192779740 1 -- 172.21.15.70:0/1069316697 >> 172.21.15.70:0/1069316697 conn(0x55e1617a7a00 msgr2=0x55e1617a7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:37.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.926+0000 7f4192779740 1 -- 172.21.15.70:0/1069316697 shutdown_connections 2024-04-01T01:37:37.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.926+0000 7f4192779740 1 -- 172.21.15.70:0/1069316697 wait complete. 2024-04-01T01:37:37.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.926+0000 7f418e130640 1 --2- 172.21.15.70:0/3741133278 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41740a26a0 0x7f41740a4b60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4178009fd0 tx=0x7f4178005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:37.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.926+0000 7f4192779740 1 -- start start 2024-04-01T01:37:37.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.927+0000 7f4192779740 1 Processor -- start 2024-04-01T01:37:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.927+0000 7f4192779740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ac490 0x55e1617b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.927+0000 7f4192779740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ace80 0x55e1617b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.927+0000 7f4192779740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617b1ff0 0x55e1617bdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.927+0000 7f4192779740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e1617b4fa0 con 0x55e1617b1ff0 2024-04-01T01:37:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.927+0000 7f4192779740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e1616af470 con 0x55e1617ace80 2024-04-01T01:37:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.927+0000 7f4192779740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e1617b5120 con 0x55e1617ac490 2024-04-01T01:37:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.927+0000 7f4192779740 10 client.0 _ll_register_callbacks cb 0x55e1616cccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.928+0000 7f418d12e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ace80 0x55e1617b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.928+0000 7f418d92f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ac490 0x55e1617b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.928+0000 7f418d92f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ac490 0x55e1617b6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52682/0 (socket says 172.21.15.70:52682) 2024-04-01T01:37:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.928+0000 7f418d12e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ace80 0x55e1617b6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53620/0 (socket says 172.21.15.70:53620) 2024-04-01T01:37:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.928+0000 7f418d92f640 1 -- 172.21.15.70:0/2214343171 learned_addr learned my addr 172.21.15.70:0/2214343171 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.928+0000 7f418e130640 1 --2- 172.21.15.70:0/2214343171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617b1ff0 0x55e1617bdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.928+0000 7f418d92f640 1 -- 172.21.15.70:0/2214343171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ace80 msgr2=0x55e1617b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.928+0000 7f418d92f640 1 --2- 172.21.15.70:0/2214343171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ace80 0x55e1617b6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.928+0000 7f418d92f640 1 -- 172.21.15.70:0/2214343171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617b1ff0 msgr2=0x55e1617bdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.928+0000 7f418d92f640 1 --2- 172.21.15.70:0/2214343171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617b1ff0 0x55e1617bdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.928+0000 7f418d92f640 1 -- 172.21.15.70:0/2214343171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1614253f0 con 0x55e1617ac490 2024-04-01T01:37:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.928+0000 7f418d12e640 1 --2- 172.21.15.70:0/2214343171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ace80 0x55e1617b6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.929+0000 7f418d92f640 1 --2- 172.21.15.70:0/2214343171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ac490 0x55e1617b6a10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4180006010 tx=0x7f41800047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ce83888700ef2bb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.929+0000 7f41717fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:37:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.929+0000 7f41717fa640 1 -- 172.21.15.70:0/2214343171 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180003b80 con 0x55e1617ac490 2024-04-01T01:37:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.929+0000 7f4192779740 1 -- 172.21.15.70:0/2214343171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4180003e20 con 0x55e1617ac490 2024-04-01T01:37:37.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.929+0000 7f41717fa640 1 -- 172.21.15.70:0/2214343171 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f418001f040 con 0x55e1617ac490 2024-04-01T01:37:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.929+0000 7f41717fa640 1 -- 172.21.15.70:0/2214343171 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f418002f050 con 0x55e1617ac490 2024-04-01T01:37:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.929+0000 7f4192779740 10 client.26563 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.929+0000 7f4192779740 1 -- 172.21.15.70:0/2214343171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e1614ef7d0 con 0x55e1617ac490 2024-04-01T01:37:37.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.930+0000 7f41717fa640 1 -- 172.21.15.70:0/2214343171 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4180003e20 con 0x55e1617ac490 2024-04-01T01:37:37.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.930+0000 7f41717fa640 1 -- 172.21.15.70:0/2214343171 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f41800055b0 con 0x55e1617ac490 2024-04-01T01:37:37.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.930+0000 7f4192779740 10 client.26563 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:37.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.930+0000 7f4192779740 10 client.26563 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:37:37.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.930+0000 7f4192779740 20 client.26563 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:37.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.930+0000 7f4192779740 1 --2- 172.21.15.70:0/2214343171 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55e1617c2160 0x55e1617c45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.930+0000 7f4192779740 4 client.26563 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:37.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.930+0000 7f4192779740 1 -- 172.21.15.70:0/2214343171 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e16171e6f0 con 0x55e1617c2160 2024-04-01T01:37:37.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.931+0000 7f418d12e640 1 --2- 172.21.15.70:0/2214343171 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55e1617c2160 0x55e1617c45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.932+0000 7f418d12e640 1 --2- 172.21.15.70:0/2214343171 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55e1617c2160 0x55e1617c45b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2a777e79ec43ac20 server_cookie=376b595373be48c8 in_seq=0 out_seq=0 2024-04-01T01:37:37.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.932+0000 7f41717fa640 10 client.26563 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:37:37.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.933+0000 7f41717fa640 1 -- 172.21.15.70:0/2214343171 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e16171e6f0 con 0x55e1617c2160 2024-04-01T01:37:37.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.933+0000 7f41717fa640 10 client.26563 handle_command_reply: tid=0 2024-04-01T01:37:37.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.933+0000 7f4192779740 1 client.26563 shutdown 2024-04-01T01:37:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.933+0000 7f4192779740 1 -- 172.21.15.70:0/2214343171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ac490 msgr2=0x55e1617b6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.933+0000 7f4192779740 1 --2- 172.21.15.70:0/2214343171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ac490 0x55e1617b6a10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4180006010 tx=0x7f41800047d0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.933+0000 7f4192779740 1 -- 172.21.15.70:0/2214343171 shutdown_connections 2024-04-01T01:37:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.933+0000 7f4192779740 1 --2- 172.21.15.70:0/2214343171 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55e1617c2160 0x55e1617c45b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.933+0000 7f4192779740 1 --2- 172.21.15.70:0/2214343171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617b1ff0 0x55e1617bdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.933+0000 7f4192779740 1 --2- 172.21.15.70:0/2214343171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ace80 0x55e1617b6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.933+0000 7f4192779740 1 --2- 172.21.15.70:0/2214343171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ac490 0x55e1617b6a10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.933+0000 7f4192779740 1 -- 172.21.15.70:0/2214343171 >> 172.21.15.70:0/2214343171 conn(0x55e1617a7a00 msgr2=0x55e1617a6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:37.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.933+0000 7f41717fa640 0 client.26563 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:37:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.934+0000 7f4192779740 1 -- 172.21.15.70:0/2214343171 shutdown_connections 2024-04-01T01:37:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.935+0000 7f4192779740 1 -- 172.21.15.70:0/2214343171 wait complete. 2024-04-01T01:37:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.935+0000 7f4192779740 20 client.26563 trim_cache size 0 max 16384 2024-04-01T01:37:37.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.960+0000 7f4192779740 1 Processor -- start 2024-04-01T01:37:37.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.960+0000 7f4192779740 1 -- start start 2024-04-01T01:37:37.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.961+0000 7f4192779740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ac490 0x55e1617b6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.961+0000 7f4192779740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ace80 0x55e1617bdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.961+0000 7f418d92f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ac490 0x55e1617b6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.961+0000 7f418d92f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ac490 0x55e1617b6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53632/0 (socket says 172.21.15.70:53632) 2024-04-01T01:37:37.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.961+0000 7f418d92f640 1 -- 172.21.15.70:0/1326658149 learned_addr learned my addr 172.21.15.70:0/1326658149 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:37.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.961+0000 7f4192779740 1 --2- 172.21.15.70:0/1326658149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617b1ff0 0x55e1617b6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.961+0000 7f4192779740 1 -- 172.21.15.70:0/1326658149 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f41881b97c0 con 0x55e1617b1ff0 2024-04-01T01:37:37.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.961+0000 7f4192779740 1 -- 172.21.15.70:0/1326658149 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f41881bb300 con 0x55e1617ac490 2024-04-01T01:37:37.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.961+0000 7f4192779740 1 -- 172.21.15.70:0/1326658149 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f41881b9e90 con 0x55e1617ace80 2024-04-01T01:37:37.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.962+0000 7f418e130640 1 --2- 172.21.15.70:0/1326658149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617b1ff0 0x55e1617b6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.962+0000 7f418d92f640 1 -- 172.21.15.70:0/1326658149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ace80 msgr2=0x55e1617bdf50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:37:37.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.962+0000 7f418d92f640 1 --2- 172.21.15.70:0/1326658149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ace80 0x55e1617bdf50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.962+0000 7f418d92f640 1 -- 172.21.15.70:0/1326658149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617b1ff0 msgr2=0x55e1617b6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:37.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.962+0000 7f418d92f640 1 --2- 172.21.15.70:0/1326658149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617b1ff0 0x55e1617b6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.962+0000 7f418d92f640 1 -- 172.21.15.70:0/1326658149 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1617b4e80 con 0x55e1617ac490 2024-04-01T01:37:37.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.962+0000 7f418d92f640 1 --2- 172.21.15.70:0/1326658149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ac490 0x55e1617b6030 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f418000eca0 tx=0x7f4180010610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c02051c11330ba5a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:37.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.963+0000 7f41737fe640 1 -- 172.21.15.70:0/1326658149 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41800098e0 con 0x55e1617ac490 2024-04-01T01:37:37.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.963+0000 7f41737fe640 1 -- 172.21.15.70:0/1326658149 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4180004310 con 0x55e1617ac490 2024-04-01T01:37:37.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.963+0000 7f41737fe640 1 -- 172.21.15.70:0/1326658149 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180010820 con 0x55e1617ac490 2024-04-01T01:37:37.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.964+0000 7f4192779740 1 -- 172.21.15.70:0/1326658149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ac490 msgr2=0x55e1617b6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.964+0000 7f4192779740 1 --2- 172.21.15.70:0/1326658149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ac490 0x55e1617b6030 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f418000eca0 tx=0x7f4180010610 comp rx=0 tx=0).stop 2024-04-01T01:37:37.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.964+0000 7f4192779740 1 -- 172.21.15.70:0/1326658149 shutdown_connections 2024-04-01T01:37:37.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.964+0000 7f4192779740 1 --2- 172.21.15.70:0/1326658149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617b1ff0 0x55e1617b6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.964+0000 7f4192779740 1 --2- 172.21.15.70:0/1326658149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617ace80 0x55e1617bdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.965+0000 7f4192779740 1 --2- 172.21.15.70:0/1326658149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ac490 0x55e1617b6030 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.965+0000 7f4192779740 1 -- 172.21.15.70:0/1326658149 >> 172.21.15.70:0/1326658149 conn(0x55e1617a7a00 msgr2=0x55e1617ab4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:37.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.965+0000 7f4192779740 1 -- 172.21.15.70:0/1326658149 shutdown_connections 2024-04-01T01:37:37.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.965+0000 7f4192779740 1 -- 172.21.15.70:0/1326658149 wait complete. 2024-04-01T01:37:37.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.965+0000 7f4192779740 1 -- start start 2024-04-01T01:37:37.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.966+0000 7f4192779740 1 Processor -- start 2024-04-01T01:37:37.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.966+0000 7f4192779740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ac490 0x55e1617baf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.967+0000 7f4192779740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617ace80 0x55e1617bb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.967+0000 7f418d92f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ac490 0x55e1617baf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.967+0000 7f418d92f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ac490 0x55e1617baf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53634/0 (socket says 172.21.15.70:53634) 2024-04-01T01:37:37.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.967+0000 7f418d92f640 1 -- 172.21.15.70:0/3830534381 learned_addr learned my addr 172.21.15.70:0/3830534381 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:37.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.967+0000 7f4192779740 1 --2- 172.21.15.70:0/3830534381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617b1ff0 0x55e1618264a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.967+0000 7f4192779740 1 -- 172.21.15.70:0/3830534381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f41881bac30 con 0x55e1617ace80 2024-04-01T01:37:37.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.967+0000 7f4192779740 1 -- 172.21.15.70:0/3830534381 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f41881ba560 con 0x55e1617ac490 2024-04-01T01:37:37.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.967+0000 7f4192779740 1 -- 172.21.15.70:0/3830534381 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f41881a1e60 con 0x55e1617b1ff0 2024-04-01T01:37:37.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.967+0000 7f4192779740 10 client.0 _ll_register_callbacks cb 0x7f418000ee30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:37.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.968+0000 7f418d92f640 1 -- 172.21.15.70:0/3830534381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617b1ff0 msgr2=0x55e1618264a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:37:37.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.968+0000 7f418d92f640 1 --2- 172.21.15.70:0/3830534381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617b1ff0 0x55e1618264a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.968+0000 7f418d92f640 1 -- 172.21.15.70:0/3830534381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617ace80 msgr2=0x55e1617bb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:37.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.968+0000 7f418d92f640 1 --2- 172.21.15.70:0/3830534381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617ace80 0x55e1617bb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.968+0000 7f418d92f640 1 -- 172.21.15.70:0/3830534381 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e161826a90 con 0x55e1617ac490 2024-04-01T01:37:37.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.968+0000 7f418d92f640 1 --2- 172.21.15.70:0/3830534381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ac490 0x55e1617baf10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f418000ef40 tx=0x7f4180011a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b97da77aeb1f33d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:37.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.968+0000 7f41717fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:37.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.969+0000 7f41717fa640 1 -- 172.21.15.70:0/3830534381 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180003e20 con 0x55e1617ac490 2024-04-01T01:37:37.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.969+0000 7f41717fa640 1 -- 172.21.15.70:0/3830534381 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f4180011de0 con 0x55e1617ac490 2024-04-01T01:37:37.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.969+0000 7f4192779740 1 -- 172.21.15.70:0/3830534381 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4180004310 con 0x55e1617ac490 2024-04-01T01:37:37.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.971+0000 7f41717fa640 1 -- 172.21.15.70:0/3830534381 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4180044050 con 0x55e1617ac490 2024-04-01T01:37:37.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.971+0000 7f4192779740 10 client.26681 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:37.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.971+0000 7f4192779740 1 -- 172.21.15.70:0/3830534381 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e161827d10 con 0x55e1617ac490 2024-04-01T01:37:37.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.972+0000 7f41717fa640 1 -- 172.21.15.70:0/3830534381 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4180004c60 con 0x55e1617ac490 2024-04-01T01:37:37.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.973+0000 7f41717fa640 1 -- 172.21.15.70:0/3830534381 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4180033430 con 0x55e1617ac490 2024-04-01T01:37:37.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.973+0000 7f4192779740 10 client.26681 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:37.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.973+0000 7f4192779740 10 client.26681 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:37:37.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.973+0000 7f4192779740 20 client.26681 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:37.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.973+0000 7f4192779740 1 --2- 172.21.15.70:0/3830534381 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55e1618284f0 0x55e16182a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:37.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.973+0000 7f4192779740 4 client.26681 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:37:37.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.973+0000 7f4192779740 1 -- 172.21.15.70:0/3830534381 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f4180004310 con 0x55e1618284f0 2024-04-01T01:37:37.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.973+0000 7f418d12e640 1 --2- 172.21.15.70:0/3830534381 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55e1618284f0 0x55e16182a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:37.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.974+0000 7f418d12e640 1 --2- 172.21.15.70:0/3830534381 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55e1618284f0 0x55e16182a910 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f4432140756f6ad5 server_cookie=a11d26987d31afa in_seq=0 out_seq=0 2024-04-01T01:37:37.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.974+0000 7f41717fa640 10 client.26681 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:37:37.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.974+0000 7f41717fa640 1 -- 172.21.15.70:0/3830534381 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f4180004310 con 0x55e1618284f0 2024-04-01T01:37:37.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.974+0000 7f41717fa640 10 client.26681 handle_command_reply: tid=0 2024-04-01T01:37:37.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.974+0000 7f4192779740 1 client.26681 shutdown 2024-04-01T01:37:37.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.975+0000 7f4192779740 1 -- 172.21.15.70:0/3830534381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ac490 msgr2=0x55e1617baf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.975+0000 7f4192779740 1 --2- 172.21.15.70:0/3830534381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ac490 0x55e1617baf10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f418000ef40 tx=0x7f4180011a60 comp rx=0 tx=0).stop 2024-04-01T01:37:37.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.975+0000 7f4192779740 1 -- 172.21.15.70:0/3830534381 shutdown_connections 2024-04-01T01:37:37.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.975+0000 7f4192779740 1 --2- 172.21.15.70:0/3830534381 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55e1618284f0 0x55e16182a910 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.975+0000 7f4192779740 1 --2- 172.21.15.70:0/3830534381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1617b1ff0 0x55e1618264a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.975+0000 7f4192779740 1 --2- 172.21.15.70:0/3830534381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1617ace80 0x55e1617bb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.975+0000 7f4192779740 1 --2- 172.21.15.70:0/3830534381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1617ac490 0x55e1617baf10 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.975+0000 7f4192779740 1 -- 172.21.15.70:0/3830534381 >> 172.21.15.70:0/3830534381 conn(0x55e1617a7a00 msgr2=0x55e1617a64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:37.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.975+0000 7f41717fa640 0 client.26681 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:37:37.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.978+0000 7f4192779740 1 -- 172.21.15.70:0/3830534381 shutdown_connections 2024-04-01T01:37:37.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.978+0000 7f4192779740 1 -- 172.21.15.70:0/3830534381 wait complete. 2024-04-01T01:37:37.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.978+0000 7f4192779740 20 client.26681 trim_cache size 0 max 16384 2024-04-01T01:37:37.979 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:37:37.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.981+0000 7f418fb93640 1 -- 172.21.15.70:0/3741133278 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41740a26a0 msgr2=0x7f41740a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.981+0000 7f418fb93640 1 --2- 172.21.15.70:0/3741133278 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41740a26a0 0x7f41740a4b60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4178009fd0 tx=0x7f4178005f90 comp rx=0 tx=0).stop 2024-04-01T01:37:37.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.981+0000 7f418fb93640 1 -- 172.21.15.70:0/3741133278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41881010e0 msgr2=0x7f4188106cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:37.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.981+0000 7f418fb93640 1 --2- 172.21.15.70:0/3741133278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41881010e0 0x7f4188106cd0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f418402c1c0 tx=0x7f418402c640 comp rx=0 tx=0).stop 2024-04-01T01:37:37.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.981+0000 7f418fb93640 1 -- 172.21.15.70:0/3741133278 shutdown_connections 2024-04-01T01:37:37.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.981+0000 7f418fb93640 1 --2- 172.21.15.70:0/3741133278 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41740a26a0 0x7f41740a4b60 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.981+0000 7f418fb93640 1 --2- 172.21.15.70:0/3741133278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4188103250 0x7f41881036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.981+0000 7f418fb93640 1 --2- 172.21.15.70:0/3741133278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41881010e0 0x7f4188106cd0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.981+0000 7f418fb93640 1 --2- 172.21.15.70:0/3741133278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41880ff610 0x7f4188106790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:37.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.982+0000 7f418fb93640 1 -- 172.21.15.70:0/3741133278 >> 172.21.15.70:0/3741133278 conn(0x7f41880faf10 msgr2=0x7f41880fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:37.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.983+0000 7f418fb93640 1 -- 172.21.15.70:0/3741133278 shutdown_connections 2024-04-01T01:37:37.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:37.983+0000 7f418fb93640 1 -- 172.21.15.70:0/3741133278 wait complete. 2024-04-01T01:37:38.001 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:37:38.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:37 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1388385400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:37:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:37 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1388385400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:37:38.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.183+0000 7f93bd4c0640 1 -- 172.21.15.70:0/1136365777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b81052e0 msgr2=0x7f93b8105760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.183+0000 7f93bd4c0640 1 --2- 172.21.15.70:0/1136365777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b81052e0 0x7f93b8105760 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009ca0 tx=0x7f93b002c100 comp rx=0 tx=0).stop 2024-04-01T01:37:38.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.183+0000 7f93bd4c0640 1 -- 172.21.15.70:0/1136365777 shutdown_connections 2024-04-01T01:37:38.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.183+0000 7f93bd4c0640 1 --2- 172.21.15.70:0/1136365777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b81052e0 0x7f93b8105760 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.183+0000 7f93bd4c0640 1 --2- 172.21.15.70:0/1136365777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93b8102930 0x7f93b8104da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.183+0000 7f93bd4c0640 1 --2- 172.21.15.70:0/1136365777 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93b80fffc0 0x7f93b81023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.183+0000 7f93bd4c0640 1 -- 172.21.15.70:0/1136365777 >> 172.21.15.70:0/1136365777 conn(0x7f93b80fbbe0 msgr2=0x7f93b80fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.183+0000 7f93bd4c0640 1 -- 172.21.15.70:0/1136365777 shutdown_connections 2024-04-01T01:37:38.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.183+0000 7f93bd4c0640 1 -- 172.21.15.70:0/1136365777 wait complete. 2024-04-01T01:37:38.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.184+0000 7f93bd4c0640 1 Processor -- start 2024-04-01T01:37:38.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.184+0000 7f93bd4c0640 1 -- start start 2024-04-01T01:37:38.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.185+0000 7f93bd4c0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93b80fffc0 0x7f93b819df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.185+0000 7f93bd4c0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b8102930 0x7f93b819e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.185+0000 7f93bd4c0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93b81052e0 0x7f93b81a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.185+0000 7f93bd4c0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f93b8113850 con 0x7f93b80fffc0 2024-04-01T01:37:38.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.185+0000 7f93bd4c0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f93b81136d0 con 0x7f93b81052e0 2024-04-01T01:37:38.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.185+0000 7f93bd4c0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f93b81139d0 con 0x7f93b8102930 2024-04-01T01:37:38.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.185+0000 7f93b6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93b80fffc0 0x7f93b819df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.185+0000 7f93b6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93b80fffc0 0x7f93b819df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47454/0 (socket says 172.21.15.70:47454) 2024-04-01T01:37:38.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.185+0000 7f93b6ffd640 1 -- 172.21.15.70:0/512443480 learned_addr learned my addr 172.21.15.70:0/512443480 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:38.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.185+0000 7f93b77fe640 1 --2- 172.21.15.70:0/512443480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93b81052e0 0x7f93b81a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.185+0000 7f93b67fc640 1 --2- 172.21.15.70:0/512443480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b8102930 0x7f93b819e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.186+0000 7f93b77fe640 1 -- 172.21.15.70:0/512443480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b8102930 msgr2=0x7f93b819e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.186+0000 7f93b77fe640 1 --2- 172.21.15.70:0/512443480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b8102930 0x7f93b819e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.186+0000 7f93b77fe640 1 -- 172.21.15.70:0/512443480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93b80fffc0 msgr2=0x7f93b819df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.186+0000 7f93b77fe640 1 --2- 172.21.15.70:0/512443480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93b80fffc0 0x7f93b819df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.186+0000 7f93b77fe640 1 -- 172.21.15.70:0/512443480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b81a5bb0 con 0x7f93b81052e0 2024-04-01T01:37:38.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.186+0000 7f93b77fe640 1 --2- 172.21.15.70:0/512443480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93b81052e0 0x7f93b81a5540 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f93b0037040 tx=0x7f93b0035ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.188+0000 7f939ffff640 1 -- 172.21.15.70:0/512443480 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b0039040 con 0x7f93b81052e0 2024-04-01T01:37:38.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.188+0000 7f939ffff640 1 -- 172.21.15.70:0/512443480 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93b0044070 con 0x7f93b81052e0 2024-04-01T01:37:38.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.188+0000 7f939ffff640 1 -- 172.21.15.70:0/512443480 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b0004330 con 0x7f93b81052e0 2024-04-01T01:37:38.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.188+0000 7f93bd4c0640 1 -- 172.21.15.70:0/512443480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93b81a5e40 con 0x7f93b81052e0 2024-04-01T01:37:38.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.188+0000 7f93bd4c0640 1 -- 172.21.15.70:0/512443480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93b81a6250 con 0x7f93b81052e0 2024-04-01T01:37:38.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.191+0000 7f939ffff640 1 -- 172.21.15.70:0/512443480 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f93b00044d0 con 0x7f93b81052e0 2024-04-01T01:37:38.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.192+0000 7f939ffff640 1 --2- 172.21.15.70:0/512443480 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93a00a26a0 0x7f93a00a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.192+0000 7f939ffff640 1 -- 172.21.15.70:0/512443480 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f93b00f10f0 con 0x7f93b81052e0 2024-04-01T01:37:38.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.192+0000 7f93c0180740 1 Processor -- start 2024-04-01T01:37:38.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.192+0000 7f93c0180740 1 -- start start 2024-04-01T01:37:38.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.193+0000 7f93c0180740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93490 0x55f451a938b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.193+0000 7f93c0180740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 0x55f451a98980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.193+0000 7f93c0180740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a98ff0 0x55f451a9b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.193+0000 7f93c0180740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f4519f81b0 con 0x55f451a98ff0 2024-04-01T01:37:38.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.193+0000 7f93c0180740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f4519a0c00 con 0x55f451a93490 2024-04-01T01:37:38.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.193+0000 7f93c0180740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f4519f4250 con 0x55f451a93e80 2024-04-01T01:37:38.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.193+0000 7f93b67fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 0x55f451a98980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.193+0000 7f93b77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a98ff0 0x55f451a9b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.193+0000 7f93b77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a98ff0 0x55f451a9b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47462/0 (socket says 172.21.15.70:47462) 2024-04-01T01:37:38.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.193+0000 7f93b77fe640 1 -- 172.21.15.70:0/456195115 learned_addr learned my addr 172.21.15.70:0/456195115 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:38.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.194+0000 7f93b77fe640 1 -- 172.21.15.70:0/456195115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 msgr2=0x55f451a98980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.194+0000 7f93b6ffd640 1 --2- 172.21.15.70:0/512443480 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93a00a26a0 0x7f93a00a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.194+0000 7f93b77fe640 1 --2- 172.21.15.70:0/456195115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 0x55f451a98980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.194+0000 7f93b6ffd640 1 --2- 172.21.15.70:0/456195115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93490 0x55f451a938b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.194+0000 7f93b77fe640 1 -- 172.21.15.70:0/456195115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93490 msgr2=0x55f451a938b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.194+0000 7f93b77fe640 1 --2- 172.21.15.70:0/456195115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93490 0x55f451a938b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.194+0000 7f93b77fe640 1 -- 172.21.15.70:0/456195115 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4519c03d0 con 0x55f451a98ff0 2024-04-01T01:37:38.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.194+0000 7f93b6ffd640 1 --2- 172.21.15.70:0/456195115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93490 0x55f451a938b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:37:38.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.194+0000 7f93b67fc640 1 --2- 172.21.15.70:0/456195115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 0x55f451a98980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:37:38.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.194+0000 7f93b77fe640 1 --2- 172.21.15.70:0/456195115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a98ff0 0x55f451a9b480 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f93b002c660 tx=0x7f93b003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec21503ccdb3eb55 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.195+0000 7f939d7fa640 1 -- 172.21.15.70:0/456195115 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b00bebe0 con 0x55f451a98ff0 2024-04-01T01:37:38.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.195+0000 7f93b6ffd640 1 --2- 172.21.15.70:0/512443480 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93a00a26a0 0x7f93a00a4b60 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f93ac00b770 tx=0x7f93ac00b9c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.195+0000 7f939d7fa640 1 -- 172.21.15.70:0/456195115 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93b00be4e0 con 0x55f451a98ff0 2024-04-01T01:37:38.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.195+0000 7f939d7fa640 1 -- 172.21.15.70:0/456195115 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b0003a90 con 0x55f451a98ff0 2024-04-01T01:37:38.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.196+0000 7f93c0180740 1 -- 172.21.15.70:0/456195115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a98ff0 msgr2=0x55f451a9b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.196+0000 7f93c0180740 1 --2- 172.21.15.70:0/456195115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a98ff0 0x55f451a9b480 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f93b002c660 tx=0x7f93b003d000 comp rx=0 tx=0).stop 2024-04-01T01:37:38.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.196+0000 7f93c0180740 1 -- 172.21.15.70:0/456195115 shutdown_connections 2024-04-01T01:37:38.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.196+0000 7f93c0180740 1 --2- 172.21.15.70:0/456195115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a98ff0 0x55f451a9b480 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.196+0000 7f93c0180740 1 --2- 172.21.15.70:0/456195115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 0x55f451a98980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.196+0000 7f93c0180740 1 --2- 172.21.15.70:0/456195115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93490 0x55f451a938b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.196+0000 7f93c0180740 1 -- 172.21.15.70:0/456195115 >> 172.21.15.70:0/456195115 conn(0x55f451a8ea00 msgr2=0x55f451a8ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.197+0000 7f93c0180740 1 -- 172.21.15.70:0/456195115 shutdown_connections 2024-04-01T01:37:38.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.197+0000 7f93c0180740 1 -- 172.21.15.70:0/456195115 wait complete. 2024-04-01T01:37:38.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.198+0000 7f93c0180740 1 -- start start 2024-04-01T01:37:38.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.198+0000 7f93c0180740 1 Processor -- start 2024-04-01T01:37:38.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.198+0000 7f93c0180740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93490 0x55f451a9d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.199+0000 7f93c0180740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 0x55f451a9da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.199+0000 7f93b6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93490 0x55f451a9d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.199+0000 7f93b6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93490 0x55f451a9d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53664/0 (socket says 172.21.15.70:53664) 2024-04-01T01:37:38.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.199+0000 7f93b6ffd640 1 -- 172.21.15.70:0/209959296 learned_addr learned my addr 172.21.15.70:0/209959296 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:38.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.199+0000 7f93b67fc640 1 --2- 172.21.15.70:0/209959296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 0x55f451a9da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.199+0000 7f93c0180740 1 --2- 172.21.15.70:0/209959296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a98ff0 0x55f451aa4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.199+0000 7f93b77fe640 1 --2- 172.21.15.70:0/209959296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a98ff0 0x55f451aa4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.199+0000 7f93c0180740 1 -- 172.21.15.70:0/209959296 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f451a9bd60 con 0x55f451a98ff0 2024-04-01T01:37:38.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.200+0000 7f93c0180740 1 -- 172.21.15.70:0/209959296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f451996470 con 0x55f451a93490 2024-04-01T01:37:38.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.200+0000 7f93c0180740 1 -- 172.21.15.70:0/209959296 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f451a9bee0 con 0x55f451a93e80 2024-04-01T01:37:38.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.200+0000 7f93b6ffd640 1 -- 172.21.15.70:0/209959296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 msgr2=0x55f451a9da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.200+0000 7f93b6ffd640 1 --2- 172.21.15.70:0/209959296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 0x55f451a9da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.200+0000 7f93b6ffd640 1 -- 172.21.15.70:0/209959296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a98ff0 msgr2=0x55f451aa4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.200+0000 7f93b6ffd640 1 --2- 172.21.15.70:0/209959296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a98ff0 0x55f451aa4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.200+0000 7f93b6ffd640 1 -- 172.21.15.70:0/209959296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f45170c3f0 con 0x55f451a93490 2024-04-01T01:37:38.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.200+0000 7f93c0180740 10 client.0 _ll_register_callbacks cb 0x55f4519b3cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:38.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.200+0000 7f93c0180740 1 -- 172.21.15.70:0/209959296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93b00be4e0 con 0x55f451a93490 2024-04-01T01:37:38.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.200+0000 7f93b6ffd640 1 --2- 172.21.15.70:0/209959296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93490 0x55f451a9d4e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f93ac0118d0 tx=0x7f93ac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b564c150fe0f1bab server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.200+0000 7f939aff5640 10 client.26690 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:38.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.201+0000 7f939aff5640 1 -- 172.21.15.70:0/209959296 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ac004460 con 0x55f451a93490 2024-04-01T01:37:38.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.201+0000 7f939aff5640 1 -- 172.21.15.70:0/209959296 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f93ac01d980 con 0x55f451a93490 2024-04-01T01:37:38.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.201+0000 7f939aff5640 1 -- 172.21.15.70:0/209959296 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f93ac01dba0 con 0x55f451a93490 2024-04-01T01:37:38.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.201+0000 7f93c0180740 10 client.26690 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:38.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.201+0000 7f93c0180740 1 -- 172.21.15.70:0/209959296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55f4517d67d0 con 0x55f451a93490 2024-04-01T01:37:38.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.201+0000 7f939aff5640 1 -- 172.21.15.70:0/209959296 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f93ac024070 con 0x55f451a93490 2024-04-01T01:37:38.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.202+0000 7f93c0180740 10 client.26690 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:38.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.202+0000 7f93c0180740 10 client.26690 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:37:38.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.202+0000 7f93c0180740 20 client.26690 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:38.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.202+0000 7f93c0180740 1 --2- 172.21.15.70:0/209959296 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55f451aa8e10 0x55f451aab220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.202+0000 7f93c0180740 4 client.26690 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:38.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.202+0000 7f93c0180740 1 -- 172.21.15.70:0/209959296 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f451a056f0 con 0x55f451aa8e10 2024-04-01T01:37:38.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.203+0000 7f93b67fc640 1 --2- 172.21.15.70:0/209959296 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55f451aa8e10 0x55f451aab220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.204+0000 7f93b67fc640 1 --2- 172.21.15.70:0/209959296 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55f451aa8e10 0x55f451aab220 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cf2c333881001cd4 server_cookie=ec74fb87b68ad059 in_seq=0 out_seq=0 2024-04-01T01:37:38.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.204+0000 7f939aff5640 10 client.26690 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:37:38.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.205+0000 7f939aff5640 1 -- 172.21.15.70:0/209959296 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f451a056f0 con 0x55f451aa8e10 2024-04-01T01:37:38.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.205+0000 7f939aff5640 10 client.26690 handle_command_reply: tid=0 2024-04-01T01:37:38.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.205+0000 7f93c0180740 1 client.26690 shutdown 2024-04-01T01:37:38.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.205+0000 7f93c0180740 1 -- 172.21.15.70:0/209959296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93490 msgr2=0x55f451a9d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.205+0000 7f93c0180740 1 --2- 172.21.15.70:0/209959296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93490 0x55f451a9d4e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f93ac0118d0 tx=0x7f93ac0046c0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.206+0000 7f93c0180740 1 -- 172.21.15.70:0/209959296 shutdown_connections 2024-04-01T01:37:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.206+0000 7f93c0180740 1 --2- 172.21.15.70:0/209959296 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55f451aa8e10 0x55f451aab220 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.206+0000 7f93c0180740 1 --2- 172.21.15.70:0/209959296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a98ff0 0x55f451aa4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.206+0000 7f93c0180740 1 --2- 172.21.15.70:0/209959296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 0x55f451a9da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.206+0000 7f93c0180740 1 --2- 172.21.15.70:0/209959296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93490 0x55f451a9d4e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.206+0000 7f93c0180740 1 -- 172.21.15.70:0/209959296 >> 172.21.15.70:0/209959296 conn(0x55f451a8ea00 msgr2=0x55f451a8dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.206+0000 7f939aff5640 0 client.26690 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:37:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.207+0000 7f93c0180740 1 -- 172.21.15.70:0/209959296 shutdown_connections 2024-04-01T01:37:38.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.207+0000 7f93c0180740 1 -- 172.21.15.70:0/209959296 wait complete. 2024-04-01T01:37:38.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.207+0000 7f93c0180740 20 client.26690 trim_cache size 0 max 16384 2024-04-01T01:37:38.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.232+0000 7f93c0180740 1 Processor -- start 2024-04-01T01:37:38.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.232+0000 7f93c0180740 1 -- start start 2024-04-01T01:37:38.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.233+0000 7f93c0180740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a8ea00 0x55f451a9d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.233+0000 7f93c0180740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93490 0x55f451aa49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.233+0000 7f93c0180740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93e80 0x55f451a8d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.233+0000 7f93c0180740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f93b81b09e0 con 0x55f451a8ea00 2024-04-01T01:37:38.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.233+0000 7f93b67fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93490 0x55f451aa49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.233+0000 7f93b67fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93490 0x55f451aa49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52736/0 (socket says 172.21.15.70:52736) 2024-04-01T01:37:38.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.233+0000 7f93b67fc640 1 -- 172.21.15.70:0/4226871489 learned_addr learned my addr 172.21.15.70:0/4226871489 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:38.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.233+0000 7f93b6ffd640 1 --2- 172.21.15.70:0/4226871489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a8ea00 0x55f451a9d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.233+0000 7f93b77fe640 1 --2- 172.21.15.70:0/4226871489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93e80 0x55f451a8d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.233+0000 7f93c0180740 1 -- 172.21.15.70:0/4226871489 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f93b81b2520 con 0x55f451a93e80 2024-04-01T01:37:38.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.233+0000 7f93c0180740 1 -- 172.21.15.70:0/4226871489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f93b81b10b0 con 0x55f451a93490 2024-04-01T01:37:38.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.234+0000 7f93b67fc640 1 -- 172.21.15.70:0/4226871489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93e80 msgr2=0x55f451a8d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.234+0000 7f93b67fc640 1 --2- 172.21.15.70:0/4226871489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93e80 0x55f451a8d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.234+0000 7f93b67fc640 1 -- 172.21.15.70:0/4226871489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a8ea00 msgr2=0x55f451a9d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.234+0000 7f93b67fc640 1 --2- 172.21.15.70:0/4226871489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a8ea00 0x55f451a9d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.234+0000 7f93b67fc640 1 -- 172.21.15.70:0/4226871489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f451a9e610 con 0x55f451a93490 2024-04-01T01:37:38.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.234+0000 7f93b67fc640 1 --2- 172.21.15.70:0/4226871489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93490 0x55f451aa49f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f93b819f4a0 tx=0x7f93a8005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98f82a6a35732abd server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.234+0000 7f939cff9640 1 -- 172.21.15.70:0/4226871489 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a8005020 con 0x55f451a93490 2024-04-01T01:37:38.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.235+0000 7f939cff9640 1 -- 172.21.15.70:0/4226871489 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93a8024030 con 0x55f451a93490 2024-04-01T01:37:38.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.235+0000 7f939cff9640 1 -- 172.21.15.70:0/4226871489 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a80200f0 con 0x55f451a93490 2024-04-01T01:37:38.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.237+0000 7f93c0180740 1 -- 172.21.15.70:0/4226871489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93490 msgr2=0x55f451aa49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.237+0000 7f93c0180740 1 --2- 172.21.15.70:0/4226871489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93490 0x55f451aa49f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f93b819f4a0 tx=0x7f93a8005f90 comp rx=0 tx=0).stop 2024-04-01T01:37:38.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.240+0000 7f93c0180740 1 -- 172.21.15.70:0/4226871489 shutdown_connections 2024-04-01T01:37:38.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.240+0000 7f93c0180740 1 --2- 172.21.15.70:0/4226871489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a93e80 0x55f451a8d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.240+0000 7f93c0180740 1 --2- 172.21.15.70:0/4226871489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93490 0x55f451aa49f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.240+0000 7f93c0180740 1 --2- 172.21.15.70:0/4226871489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a8ea00 0x55f451a9d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.240+0000 7f93c0180740 1 -- 172.21.15.70:0/4226871489 >> 172.21.15.70:0/4226871489 conn(0x55f451a98b30 msgr2=0x55f451a9d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.240+0000 7f93c0180740 1 -- 172.21.15.70:0/4226871489 shutdown_connections 2024-04-01T01:37:38.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.241+0000 7f93c0180740 1 -- 172.21.15.70:0/4226871489 wait complete. 2024-04-01T01:37:38.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.241+0000 7f93c0180740 1 -- start start 2024-04-01T01:37:38.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.241+0000 7f93c0180740 1 Processor -- start 2024-04-01T01:37:38.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.242+0000 7f93c0180740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a8ea00 0x55f451aa25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.242+0000 7f93c0180740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a93490 0x55f451aa2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.242+0000 7f93c0180740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 0x55f451aa3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.242+0000 7f93c0180740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f93b81b1e50 con 0x55f451a93490 2024-04-01T01:37:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.242+0000 7f93c0180740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f93b81b1780 con 0x55f451a8ea00 2024-04-01T01:37:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.242+0000 7f93c0180740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f93b81ae9c0 con 0x55f451a93e80 2024-04-01T01:37:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.242+0000 7f93c0180740 10 client.0 _ll_register_callbacks cb 0x7f93ac0114a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.242+0000 7f93b6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a8ea00 0x55f451aa25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.242+0000 7f93b6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a8ea00 0x55f451aa25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53680/0 (socket says 172.21.15.70:53680) 2024-04-01T01:37:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.242+0000 7f93b77fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 0x55f451aa3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.242+0000 7f93b6ffd640 1 -- 172.21.15.70:0/2792638336 learned_addr learned my addr 172.21.15.70:0/2792638336 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.242+0000 7f93b67fc640 1 --2- 172.21.15.70:0/2792638336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a93490 0x55f451aa2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.242+0000 7f93b77fe640 1 -- 172.21.15.70:0/2792638336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a8ea00 msgr2=0x55f451aa25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.242+0000 7f93b77fe640 1 --2- 172.21.15.70:0/2792638336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a8ea00 0x55f451aa25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.243+0000 7f93b77fe640 1 -- 172.21.15.70:0/2792638336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a93490 msgr2=0x55f451aa2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.243+0000 7f93b77fe640 1 --2- 172.21.15.70:0/2792638336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a93490 0x55f451aa2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.243+0000 7f93b77fe640 1 -- 172.21.15.70:0/2792638336 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f451a8a240 con 0x55f451a93e80 2024-04-01T01:37:38.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.243+0000 7f93b77fe640 1 --2- 172.21.15.70:0/2792638336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 0x55f451aa3b20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f93b00bca30 tx=0x7f93b0009650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25811fa06a82efaa server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.243+0000 7f939aff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:37:38.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.243+0000 7f939aff5640 1 -- 172.21.15.70:0/2792638336 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b00bebe0 con 0x55f451a93e80 2024-04-01T01:37:38.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.243+0000 7f939aff5640 1 -- 172.21.15.70:0/2792638336 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f93b00be740 con 0x55f451a93e80 2024-04-01T01:37:38.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.243+0000 7f93c0180740 1 -- 172.21.15.70:0/2792638336 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93a8024030 con 0x55f451a93e80 2024-04-01T01:37:38.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.244+0000 7f939aff5640 1 -- 172.21.15.70:0/2792638336 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f93b00c6740 con 0x55f451a93e80 2024-04-01T01:37:38.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.244+0000 7f93c0180740 10 client.26581 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:38.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.244+0000 7f93c0180740 1 -- 172.21.15.70:0/2792638336 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f451b0f370 con 0x55f451a93e80 2024-04-01T01:37:38.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.244+0000 7f939aff5640 1 -- 172.21.15.70:0/2792638336 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93b00be940 con 0x55f451a93e80 2024-04-01T01:37:38.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.244+0000 7f939aff5640 1 -- 172.21.15.70:0/2792638336 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f93b003c030 con 0x55f451a93e80 2024-04-01T01:37:38.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.245+0000 7f93c0180740 10 client.26581 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:38.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.245+0000 7f93c0180740 10 client.26581 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:37:38.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.245+0000 7f93c0180740 20 client.26581 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:38.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.245+0000 7f93c0180740 1 --2- 172.21.15.70:0/2792638336 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55f451a98ff0 0x55f451b11940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.245+0000 7f93b6ffd640 1 --2- 172.21.15.70:0/2792638336 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55f451a98ff0 0x55f451b11940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.245+0000 7f93c0180740 4 client.26581 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:37:38.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.245+0000 7f93c0180740 1 -- 172.21.15.70:0/2792638336 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f93a8024030 con 0x55f451a98ff0 2024-04-01T01:37:38.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.246+0000 7f93b6ffd640 1 --2- 172.21.15.70:0/2792638336 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55f451a98ff0 0x55f451b11940 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7f0f2d737ea70876 server_cookie=6ca201ba4db8a716 in_seq=0 out_seq=0 2024-04-01T01:37:38.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.246+0000 7f939aff5640 10 client.26581 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:37:38.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.246+0000 7f939aff5640 1 -- 172.21.15.70:0/2792638336 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f93a8024030 con 0x55f451a98ff0 2024-04-01T01:37:38.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.246+0000 7f939aff5640 10 client.26581 handle_command_reply: tid=0 2024-04-01T01:37:38.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.247+0000 7f93c0180740 1 client.26581 shutdown 2024-04-01T01:37:38.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.247+0000 7f93c0180740 1 -- 172.21.15.70:0/2792638336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 msgr2=0x55f451aa3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.247+0000 7f93c0180740 1 --2- 172.21.15.70:0/2792638336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 0x55f451aa3b20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f93b00bca30 tx=0x7f93b0009650 comp rx=0 tx=0).stop 2024-04-01T01:37:38.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.247+0000 7f93c0180740 1 -- 172.21.15.70:0/2792638336 shutdown_connections 2024-04-01T01:37:38.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.247+0000 7f93c0180740 1 --2- 172.21.15.70:0/2792638336 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55f451a98ff0 0x55f451b11940 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.248+0000 7f93c0180740 1 --2- 172.21.15.70:0/2792638336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f451a93e80 0x55f451aa3b20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.248+0000 7f939aff5640 0 client.26581 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:37:38.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.248+0000 7f93c0180740 1 --2- 172.21.15.70:0/2792638336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f451a93490 0x55f451aa2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.248+0000 7f93c0180740 1 --2- 172.21.15.70:0/2792638336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f451a8ea00 0x55f451aa25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.248+0000 7f93c0180740 1 -- 172.21.15.70:0/2792638336 >> 172.21.15.70:0/2792638336 conn(0x55f451a98b30 msgr2=0x55f451a91c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.248+0000 7f93c0180740 1 -- 172.21.15.70:0/2792638336 shutdown_connections 2024-04-01T01:37:38.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.248+0000 7f93c0180740 1 -- 172.21.15.70:0/2792638336 wait complete. 2024-04-01T01:37:38.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.248+0000 7f93c0180740 20 client.26581 trim_cache size 0 max 16384 2024-04-01T01:37:38.249 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:37:38.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:37 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1388385400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:37:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.251+0000 7f93bd4c0640 1 -- 172.21.15.70:0/512443480 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93a00a26a0 msgr2=0x7f93a00a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.251+0000 7f93bd4c0640 1 --2- 172.21.15.70:0/512443480 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93a00a26a0 0x7f93a00a4b60 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f93ac00b770 tx=0x7f93ac00b9c0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.251+0000 7f93bd4c0640 1 -- 172.21.15.70:0/512443480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93b81052e0 msgr2=0x7f93b81a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.251+0000 7f93bd4c0640 1 --2- 172.21.15.70:0/512443480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93b81052e0 0x7f93b81a5540 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f93b0037040 tx=0x7f93b0035ce0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.251+0000 7f93bd4c0640 1 -- 172.21.15.70:0/512443480 shutdown_connections 2024-04-01T01:37:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.251+0000 7f93bd4c0640 1 --2- 172.21.15.70:0/512443480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93b81052e0 0x7f93b81a5540 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.251+0000 7f93bd4c0640 1 --2- 172.21.15.70:0/512443480 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93a00a26a0 0x7f93a00a4b60 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.252+0000 7f93bd4c0640 1 --2- 172.21.15.70:0/512443480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b8102930 0x7f93b819e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.252+0000 7f93bd4c0640 1 --2- 172.21.15.70:0/512443480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93b80fffc0 0x7f93b819df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.252+0000 7f93bd4c0640 1 -- 172.21.15.70:0/512443480 >> 172.21.15.70:0/512443480 conn(0x7f93b80fbbe0 msgr2=0x7f93b8100940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.253+0000 7f93bd4c0640 1 -- 172.21.15.70:0/512443480 shutdown_connections 2024-04-01T01:37:38.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.253+0000 7f93bd4c0640 1 -- 172.21.15.70:0/512443480 wait complete. 2024-04-01T01:37:38.280 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:37:38.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.471+0000 7fafa893f640 1 -- 172.21.15.70:0/3097808082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafa4108e40 msgr2=0x7fafa41092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.471+0000 7fafa893f640 1 --2- 172.21.15.70:0/3097808082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafa4108e40 0x7fafa41092c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7faf8c009860 tx=0x7faf8c02bb70 comp rx=0 tx=0).stop 2024-04-01T01:37:38.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.471+0000 7fafa893f640 1 -- 172.21.15.70:0/3097808082 shutdown_connections 2024-04-01T01:37:38.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.471+0000 7fafa893f640 1 --2- 172.21.15.70:0/3097808082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafa4109800 0x7fafa4110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.471+0000 7fafa893f640 1 --2- 172.21.15.70:0/3097808082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafa4108e40 0x7fafa41092c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.471+0000 7fafa893f640 1 --2- 172.21.15.70:0/3097808082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafa4107c50 0x7fafa4108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.472+0000 7fafa893f640 1 -- 172.21.15.70:0/3097808082 >> 172.21.15.70:0/3097808082 conn(0x7fafa4076bf0 msgr2=0x7fafa4079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.473+0000 7fafa893f640 1 -- 172.21.15.70:0/3097808082 shutdown_connections 2024-04-01T01:37:38.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.475+0000 7fafa893f640 1 -- 172.21.15.70:0/3097808082 wait complete. 2024-04-01T01:37:38.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.476+0000 7fafa893f640 1 Processor -- start 2024-04-01T01:37:38.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.480+0000 7fafa893f640 1 -- start start 2024-04-01T01:37:38.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.481+0000 7fafa893f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafa4107c50 0x7fafa419de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.481+0000 7fafa893f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafa4108e40 0x7fafa419e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.481+0000 7fafa893f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafa4109800 0x7fafa41a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.481+0000 7fafa37fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafa4107c50 0x7fafa419de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.482+0000 7fafa37fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafa4107c50 0x7fafa419de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52760/0 (socket says 172.21.15.70:52760) 2024-04-01T01:37:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.482+0000 7fafa37fe640 1 -- 172.21.15.70:0/1355431443 learned_addr learned my addr 172.21.15.70:0/1355431443 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.482+0000 7fafa893f640 1 -- 172.21.15.70:0/1355431443 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fafa41137b0 con 0x7fafa4108e40 2024-04-01T01:37:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.482+0000 7fafa893f640 1 -- 172.21.15.70:0/1355431443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fafa4113630 con 0x7fafa4109800 2024-04-01T01:37:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.482+0000 7fafa893f640 1 -- 172.21.15.70:0/1355431443 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fafa4113930 con 0x7fafa4107c50 2024-04-01T01:37:38.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.483+0000 7faf9bfff640 1 --2- 172.21.15.70:0/1355431443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafa4108e40 0x7fafa419e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.483+0000 7fafa3fff640 1 --2- 172.21.15.70:0/1355431443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafa4109800 0x7fafa41a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.483+0000 7fafa3fff640 1 -- 172.21.15.70:0/1355431443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafa4107c50 msgr2=0x7fafa419de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.483+0000 7fafa3fff640 1 --2- 172.21.15.70:0/1355431443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafa4107c50 0x7fafa419de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.483+0000 7fafa3fff640 1 -- 172.21.15.70:0/1355431443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafa4108e40 msgr2=0x7fafa419e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.483+0000 7fafa3fff640 1 --2- 172.21.15.70:0/1355431443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafa4108e40 0x7fafa419e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.483+0000 7fafa3fff640 1 -- 172.21.15.70:0/1355431443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa41a5ab0 con 0x7fafa4109800 2024-04-01T01:37:38.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.484+0000 7fafa3fff640 1 --2- 172.21.15.70:0/1355431443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafa4109800 0x7fafa41a5440 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7faf9c00ef00 tx=0x7faf9c00c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.484+0000 7fafa17fa640 1 -- 172.21.15.70:0/1355431443 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c004b20 con 0x7fafa4109800 2024-04-01T01:37:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.484+0000 7fafa17fa640 1 -- 172.21.15.70:0/1355431443 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf9c003e20 con 0x7fafa4109800 2024-04-01T01:37:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.484+0000 7fafa17fa640 1 -- 172.21.15.70:0/1355431443 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c00b570 con 0x7fafa4109800 2024-04-01T01:37:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.484+0000 7fafa893f640 1 -- 172.21.15.70:0/1355431443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafa41a5da0 con 0x7fafa4109800 2024-04-01T01:37:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.484+0000 7fafa893f640 1 -- 172.21.15.70:0/1355431443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafa41a6210 con 0x7fafa4109800 2024-04-01T01:37:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.488+0000 7fafac786740 1 Processor -- start 2024-04-01T01:37:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.488+0000 7fafa17fa640 1 -- 172.21.15.70:0/1355431443 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7faf9c01d020 con 0x7fafa4109800 2024-04-01T01:37:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.488+0000 7fafac786740 1 -- start start 2024-04-01T01:37:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.488+0000 7fafac786740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddbf6a0 0x55710ddbfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.488+0000 7fafac786740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc0090 0x55710ddc4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.488+0000 7fafac786740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddc5200 0x55710ddc7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.488+0000 7fafac786740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55710dd241b0 con 0x55710ddc5200 2024-04-01T01:37:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.488+0000 7fafac786740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55710dcccc00 con 0x55710ddbf6a0 2024-04-01T01:37:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.488+0000 7fafac786740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55710dd20250 con 0x55710ddc0090 2024-04-01T01:37:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.489+0000 7faf9bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddbf6a0 0x55710ddbfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.489+0000 7fafa37fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc0090 0x55710ddc4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.489+0000 7faf9bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddbf6a0 0x55710ddbfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53702/0 (socket says 172.21.15.70:53702) 2024-04-01T01:37:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.489+0000 7faf9bfff640 1 -- 172.21.15.70:0/1653519808 learned_addr learned my addr 172.21.15.70:0/1653519808 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.489+0000 7faf9bfff640 1 --2- 172.21.15.70:0/1653519808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddc5200 0x55710ddc7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.489+0000 7faf9bfff640 1 -- 172.21.15.70:0/1653519808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc0090 msgr2=0x55710ddc4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.489+0000 7faf9bfff640 1 --2- 172.21.15.70:0/1653519808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc0090 0x55710ddc4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.489+0000 7faf9bfff640 1 -- 172.21.15.70:0/1653519808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddc5200 msgr2=0x55710ddc7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.489+0000 7faf9bfff640 1 --2- 172.21.15.70:0/1653519808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddc5200 0x55710ddc7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.489+0000 7faf9bfff640 1 -- 172.21.15.70:0/1653519808 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55710dcec3d0 con 0x55710ddbf6a0 2024-04-01T01:37:38.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.489+0000 7faf9bfff640 1 --2- 172.21.15.70:0/1653519808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddbf6a0 0x55710ddbfac0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7faf8c005b60 tx=0x7faf8c0056f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f2628c2c7e662b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.489+0000 7faf99ffb640 1 -- 172.21.15.70:0/1653519808 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8c049070 con 0x55710ddbf6a0 2024-04-01T01:37:38.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.490+0000 7faf99ffb640 1 -- 172.21.15.70:0/1653519808 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf8c02c510 con 0x55710ddbf6a0 2024-04-01T01:37:38.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.490+0000 7faf99ffb640 1 -- 172.21.15.70:0/1653519808 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8c039040 con 0x55710ddbf6a0 2024-04-01T01:37:38.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.490+0000 7fafa17fa640 1 --2- 172.21.15.70:0/1355431443 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faf900a2620 0x7faf900a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.490+0000 7fafa17fa640 1 -- 172.21.15.70:0/1355431443 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7faf9c0ce810 con 0x7fafa4109800 2024-04-01T01:37:38.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.490+0000 7fafa37fe640 1 --2- 172.21.15.70:0/1355431443 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faf900a2620 0x7faf900a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.490+0000 7fafac786740 1 -- 172.21.15.70:0/1653519808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddbf6a0 msgr2=0x55710ddbfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.490+0000 7fafac786740 1 --2- 172.21.15.70:0/1653519808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddbf6a0 0x55710ddbfac0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7faf8c005b60 tx=0x7faf8c0056f0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.491+0000 7fafac786740 1 -- 172.21.15.70:0/1653519808 shutdown_connections 2024-04-01T01:37:38.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.491+0000 7fafac786740 1 --2- 172.21.15.70:0/1653519808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddc5200 0x55710ddc7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.491+0000 7fafac786740 1 --2- 172.21.15.70:0/1653519808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc0090 0x55710ddc4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.491+0000 7fafac786740 1 --2- 172.21.15.70:0/1653519808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddbf6a0 0x55710ddbfac0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.491+0000 7fafac786740 1 -- 172.21.15.70:0/1653519808 >> 172.21.15.70:0/1653519808 conn(0x55710ddbac10 msgr2=0x55710ddbb010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.491+0000 7fafa37fe640 1 --2- 172.21.15.70:0/1355431443 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faf900a2620 0x7faf900a4ae0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x55710ddc0490 tx=0x7faf94005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.491+0000 7fafac786740 1 -- 172.21.15.70:0/1653519808 shutdown_connections 2024-04-01T01:37:38.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.491+0000 7fafac786740 1 -- 172.21.15.70:0/1653519808 wait complete. 2024-04-01T01:37:38.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.491+0000 7fafac786740 1 -- start start 2024-04-01T01:37:38.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.492+0000 7fafac786740 1 Processor -- start 2024-04-01T01:37:38.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.492+0000 7fafac786740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddbf6a0 0x55710ddc9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.492+0000 7fafac786740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddc0090 0x55710ddca150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.492+0000 7fafac786740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddc5200 0x55710ddd1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.492+0000 7fafac786740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55710ddc8150 con 0x55710ddc5200 2024-04-01T01:37:38.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.492+0000 7fafac786740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55710dcc2470 con 0x55710ddc0090 2024-04-01T01:37:38.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.492+0000 7fafac786740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55710ddc82d0 con 0x55710ddbf6a0 2024-04-01T01:37:38.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.492+0000 7fafac786740 10 client.0 _ll_register_callbacks cb 0x55710dcdfcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:38.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.492+0000 7fafa37fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddbf6a0 0x55710ddc9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.492+0000 7fafa37fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddbf6a0 0x55710ddc9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52770/0 (socket says 172.21.15.70:52770) 2024-04-01T01:37:38.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.492+0000 7fafa37fe640 1 -- 172.21.15.70:0/1650338692 learned_addr learned my addr 172.21.15.70:0/1650338692 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:38.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.492+0000 7fafa3fff640 1 --2- 172.21.15.70:0/1650338692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddc5200 0x55710ddd1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.492+0000 7faf9bfff640 1 --2- 172.21.15.70:0/1650338692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddc0090 0x55710ddca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.493+0000 7fafa37fe640 1 -- 172.21.15.70:0/1650338692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddc0090 msgr2=0x55710ddca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.493+0000 7fafa37fe640 1 --2- 172.21.15.70:0/1650338692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddc0090 0x55710ddca150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.493+0000 7fafa37fe640 1 -- 172.21.15.70:0/1650338692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddc5200 msgr2=0x55710ddd1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.493+0000 7fafa37fe640 1 --2- 172.21.15.70:0/1650338692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddc5200 0x55710ddd1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.493+0000 7fafa37fe640 1 -- 172.21.15.70:0/1650338692 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55710da383f0 con 0x55710ddbf6a0 2024-04-01T01:37:38.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.493+0000 7fafa37fe640 1 --2- 172.21.15.70:0/1650338692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddbf6a0 0x55710ddc9c10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faf94004620 tx=0x7faf94004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c604d2da354ddc9f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.493+0000 7faf82ffd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:37:38.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.494+0000 7faf82ffd640 1 -- 172.21.15.70:0/1650338692 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf94005020 con 0x55710ddbf6a0 2024-04-01T01:37:38.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.494+0000 7faf82ffd640 1 -- 172.21.15.70:0/1650338692 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7faf940051c0 con 0x55710ddbf6a0 2024-04-01T01:37:38.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.494+0000 7fafac786740 1 -- 172.21.15.70:0/1650338692 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf8c02c510 con 0x55710ddbf6a0 2024-04-01T01:37:38.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.494+0000 7faf82ffd640 1 -- 172.21.15.70:0/1650338692 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faf940053c0 con 0x55710ddbf6a0 2024-04-01T01:37:38.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.495+0000 7fafac786740 10 client.26593 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:38.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.495+0000 7fafac786740 1 -- 172.21.15.70:0/1650338692 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55710db027d0 con 0x55710ddbf6a0 2024-04-01T01:37:38.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.495+0000 7faf82ffd640 1 -- 172.21.15.70:0/1650338692 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf94005560 con 0x55710ddbf6a0 2024-04-01T01:37:38.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.496+0000 7faf82ffd640 1 -- 172.21.15.70:0/1650338692 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7faf94027070 con 0x55710ddbf6a0 2024-04-01T01:37:38.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.496+0000 7fafac786740 10 client.26593 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:38.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.496+0000 7fafac786740 10 client.26593 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:37:38.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.496+0000 7fafac786740 20 client.26593 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:38.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.496+0000 7fafac786740 1 --2- 172.21.15.70:0/1650338692 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55710ddd58a0 0x55710ddd7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.496+0000 7fafac786740 4 client.26593 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:38.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.496+0000 7fafac786740 1 -- 172.21.15.70:0/1650338692 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7faf8c02c510 con 0x55710ddd58a0 2024-04-01T01:37:38.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.496+0000 7faf9bfff640 1 --2- 172.21.15.70:0/1650338692 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55710ddd58a0 0x55710ddd7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.497+0000 7faf82ffd640 10 client.26593 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:37:38.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.497+0000 7faf9bfff640 1 --2- 172.21.15.70:0/1650338692 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55710ddd58a0 0x55710ddd7d00 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2ea43c900677e4c4 server_cookie=3358adcdb8935f84 in_seq=0 out_seq=0 2024-04-01T01:37:38.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.498+0000 7faf82ffd640 1 -- 172.21.15.70:0/1650338692 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7faf8c02c510 con 0x55710ddd58a0 2024-04-01T01:37:38.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.498+0000 7faf82ffd640 10 client.26593 handle_command_reply: tid=0 2024-04-01T01:37:38.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.499+0000 7fafac786740 1 client.26593 shutdown 2024-04-01T01:37:38.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.499+0000 7fafac786740 1 -- 172.21.15.70:0/1650338692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddbf6a0 msgr2=0x55710ddc9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.499+0000 7fafac786740 1 --2- 172.21.15.70:0/1650338692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddbf6a0 0x55710ddc9c10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faf94004620 tx=0x7faf94004730 comp rx=0 tx=0).stop 2024-04-01T01:37:38.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.499+0000 7fafac786740 1 -- 172.21.15.70:0/1650338692 shutdown_connections 2024-04-01T01:37:38.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.499+0000 7fafac786740 1 --2- 172.21.15.70:0/1650338692 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55710ddd58a0 0x55710ddd7d00 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.500+0000 7fafac786740 1 --2- 172.21.15.70:0/1650338692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddc5200 0x55710ddd1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.500+0000 7faf82ffd640 0 client.26593 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:37:38.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.500+0000 7fafac786740 1 --2- 172.21.15.70:0/1650338692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddc0090 0x55710ddca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.500+0000 7fafac786740 1 --2- 172.21.15.70:0/1650338692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddbf6a0 0x55710ddc9c10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.500+0000 7fafac786740 1 -- 172.21.15.70:0/1650338692 >> 172.21.15.70:0/1650338692 conn(0x55710ddbac10 msgr2=0x55710ddb9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.500+0000 7fafac786740 1 -- 172.21.15.70:0/1650338692 shutdown_connections 2024-04-01T01:37:38.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.501+0000 7fafac786740 1 -- 172.21.15.70:0/1650338692 wait complete. 2024-04-01T01:37:38.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.501+0000 7fafac786740 20 client.26593 trim_cache size 0 max 16384 2024-04-01T01:37:38.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.529+0000 7fafac786740 1 Processor -- start 2024-04-01T01:37:38.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.529+0000 7fafac786740 1 -- start start 2024-04-01T01:37:38.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.529+0000 7fafac786740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddbf6a0 0x55710ddca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.529+0000 7fafac786740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddc0090 0x55710ddd1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.529+0000 7fafac786740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc5200 0x55710ddc0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.529+0000 7fafac786740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fafa41b07f0 con 0x55710ddc0090 2024-04-01T01:37:38.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.529+0000 7fafac786740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fafa41b2330 con 0x55710ddbf6a0 2024-04-01T01:37:38.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.529+0000 7fafac786740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fafa41b0ec0 con 0x55710ddc5200 2024-04-01T01:37:38.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.530+0000 7fafa37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddbf6a0 0x55710ddca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.530+0000 7fafa3fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc5200 0x55710ddc0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.530+0000 7fafa3fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc5200 0x55710ddc0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52774/0 (socket says 172.21.15.70:52774) 2024-04-01T01:37:38.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.530+0000 7fafa3fff640 1 -- 172.21.15.70:0/642402149 learned_addr learned my addr 172.21.15.70:0/642402149 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:38.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.530+0000 7faf9bfff640 1 --2- 172.21.15.70:0/642402149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddc0090 0x55710ddd1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.530+0000 7fafa3fff640 1 -- 172.21.15.70:0/642402149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddbf6a0 msgr2=0x55710ddca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.530+0000 7fafa3fff640 1 --2- 172.21.15.70:0/642402149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddbf6a0 0x55710ddca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.530+0000 7fafa3fff640 1 -- 172.21.15.70:0/642402149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddc0090 msgr2=0x55710ddd1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.530+0000 7fafa3fff640 1 --2- 172.21.15.70:0/642402149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddc0090 0x55710ddd1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.530+0000 7fafa3fff640 1 -- 172.21.15.70:0/642402149 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55710ddbbb80 con 0x55710ddc5200 2024-04-01T01:37:38.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.531+0000 7fafa3fff640 1 --2- 172.21.15.70:0/642402149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc5200 0x55710ddc0ab0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faf9c0cc2b0 tx=0x7faf9c09b6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60338163aa1afc2f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.531+0000 7faf997fa640 1 -- 172.21.15.70:0/642402149 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c01a040 con 0x55710ddc5200 2024-04-01T01:37:38.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.531+0000 7faf997fa640 1 -- 172.21.15.70:0/642402149 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf9c0a3910 con 0x55710ddc5200 2024-04-01T01:37:38.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.531+0000 7faf997fa640 1 -- 172.21.15.70:0/642402149 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c0a3480 con 0x55710ddc5200 2024-04-01T01:37:38.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.532+0000 7fafac786740 1 -- 172.21.15.70:0/642402149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc5200 msgr2=0x55710ddc0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.532+0000 7fafac786740 1 --2- 172.21.15.70:0/642402149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc5200 0x55710ddc0ab0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faf9c0cc2b0 tx=0x7faf9c09b6c0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.532+0000 7fafac786740 1 -- 172.21.15.70:0/642402149 shutdown_connections 2024-04-01T01:37:38.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.532+0000 7fafac786740 1 --2- 172.21.15.70:0/642402149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc5200 0x55710ddc0ab0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.532+0000 7fafac786740 1 --2- 172.21.15.70:0/642402149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddc0090 0x55710ddd1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.533+0000 7fafac786740 1 --2- 172.21.15.70:0/642402149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddbf6a0 0x55710ddca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.533+0000 7fafac786740 1 -- 172.21.15.70:0/642402149 >> 172.21.15.70:0/642402149 conn(0x55710ddbac10 msgr2=0x55710ddb5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.533+0000 7fafac786740 1 -- 172.21.15.70:0/642402149 shutdown_connections 2024-04-01T01:37:38.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.533+0000 7fafac786740 1 -- 172.21.15.70:0/642402149 wait complete. 2024-04-01T01:37:38.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.534+0000 7fafac786740 1 -- start start 2024-04-01T01:37:38.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.534+0000 7fafac786740 1 Processor -- start 2024-04-01T01:37:38.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.535+0000 7fafac786740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddbf6a0 0x55710ddcea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.535+0000 7fafac786740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc0090 0x55710ddcefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.535+0000 7fafac786740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddc5200 0x55710ddcff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.535+0000 7fafac786740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fafa41b1c60 con 0x55710ddbf6a0 2024-04-01T01:37:38.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.535+0000 7fafac786740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fafa41b1590 con 0x55710ddc5200 2024-04-01T01:37:38.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.535+0000 7fafac786740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fafa41ae7d0 con 0x55710ddc0090 2024-04-01T01:37:38.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.535+0000 7fafac786740 10 client.0 _ll_register_callbacks cb 0x7faf940044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:38.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.535+0000 7fafa37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddbf6a0 0x55710ddcea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.535+0000 7fafa37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddbf6a0 0x55710ddcea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47536/0 (socket says 172.21.15.70:47536) 2024-04-01T01:37:38.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.535+0000 7fafa37fe640 1 -- 172.21.15.70:0/2567214125 learned_addr learned my addr 172.21.15.70:0/2567214125 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:38.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.535+0000 7fafa3fff640 1 --2- 172.21.15.70:0/2567214125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddc5200 0x55710ddcff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.535+0000 7faf9bfff640 1 --2- 172.21.15.70:0/2567214125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc0090 0x55710ddcefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.536+0000 7fafa3fff640 1 -- 172.21.15.70:0/2567214125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc0090 msgr2=0x55710ddcefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.536+0000 7fafa3fff640 1 --2- 172.21.15.70:0/2567214125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc0090 0x55710ddcefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.536+0000 7fafa3fff640 1 -- 172.21.15.70:0/2567214125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddbf6a0 msgr2=0x55710ddcea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.536+0000 7fafa3fff640 1 --2- 172.21.15.70:0/2567214125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddbf6a0 0x55710ddcea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.536+0000 7fafa3fff640 1 -- 172.21.15.70:0/2567214125 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55710ddd0560 con 0x55710ddc5200 2024-04-01T01:37:38.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.536+0000 7fafa37fe640 1 --2- 172.21.15.70:0/2567214125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddbf6a0 0x55710ddcea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:38.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.536+0000 7fafa3fff640 1 --2- 172.21.15.70:0/2567214125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddc5200 0x55710ddcff70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faf9c09b7b0 tx=0x7faf9c09ada0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c833b7c6d60110b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.536+0000 7faf82ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:38.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.536+0000 7faf82ffd640 1 -- 172.21.15.70:0/2567214125 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9c0a8060 con 0x55710ddc5200 2024-04-01T01:37:38.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.536+0000 7faf82ffd640 1 -- 172.21.15.70:0/2567214125 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7faf9c0a3b60 con 0x55710ddc5200 2024-04-01T01:37:38.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.536+0000 7fafac786740 1 -- 172.21.15.70:0/2567214125 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf9c0a3910 con 0x55710ddc5200 2024-04-01T01:37:38.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.537+0000 7faf82ffd640 1 -- 172.21.15.70:0/2567214125 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faf9c01a070 con 0x55710ddc5200 2024-04-01T01:37:38.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.537+0000 7fafac786740 10 client.26714 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:38.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.537+0000 7fafac786740 1 -- 172.21.15.70:0/2567214125 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55710de3be20 con 0x55710ddc5200 2024-04-01T01:37:38.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.537+0000 7faf82ffd640 1 -- 172.21.15.70:0/2567214125 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf9c0a3910 con 0x55710ddc5200 2024-04-01T01:37:38.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.537+0000 7faf82ffd640 1 -- 172.21.15.70:0/2567214125 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7faf9c00b040 con 0x55710ddc5200 2024-04-01T01:37:38.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.537+0000 7fafac786740 10 client.26714 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:38.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.537+0000 7fafac786740 10 client.26714 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:37:38.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.537+0000 7fafac786740 20 client.26714 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:38.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.537+0000 7fafac786740 1 --2- 172.21.15.70:0/2567214125 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55710ddd58a0 0x55710de3e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.537+0000 7fafac786740 4 client.26714 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:37:38.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.537+0000 7fafac786740 1 -- 172.21.15.70:0/2567214125 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55710dd316f0 con 0x55710ddd58a0 2024-04-01T01:37:38.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.538+0000 7fafa37fe640 1 --2- 172.21.15.70:0/2567214125 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55710ddd58a0 0x55710de3e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.539+0000 7fafa37fe640 1 --2- 172.21.15.70:0/2567214125 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55710ddd58a0 0x55710de3e700 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8a2c6a02ab18457f server_cookie=e0b798ea6855aa47 in_seq=0 out_seq=0 2024-04-01T01:37:38.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.540+0000 7faf82ffd640 10 client.26714 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:37:38.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.540+0000 7faf82ffd640 1 -- 172.21.15.70:0/2567214125 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55710dd316f0 con 0x55710ddd58a0 2024-04-01T01:37:38.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.540+0000 7faf82ffd640 10 client.26714 handle_command_reply: tid=0 2024-04-01T01:37:38.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.540+0000 7fafac786740 1 client.26714 shutdown 2024-04-01T01:37:38.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.541+0000 7fafac786740 1 -- 172.21.15.70:0/2567214125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddc5200 msgr2=0x55710ddcff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.541+0000 7fafac786740 1 --2- 172.21.15.70:0/2567214125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddc5200 0x55710ddcff70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faf9c09b7b0 tx=0x7faf9c09ada0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.541+0000 7fafac786740 1 -- 172.21.15.70:0/2567214125 shutdown_connections 2024-04-01T01:37:38.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.541+0000 7fafac786740 1 --2- 172.21.15.70:0/2567214125 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55710ddd58a0 0x55710de3e700 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.541+0000 7fafac786740 1 --2- 172.21.15.70:0/2567214125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55710ddc5200 0x55710ddcff70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.541+0000 7fafac786740 1 --2- 172.21.15.70:0/2567214125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55710ddc0090 0x55710ddcefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.541+0000 7fafac786740 1 --2- 172.21.15.70:0/2567214125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55710ddbf6a0 0x55710ddcea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.541+0000 7fafac786740 1 -- 172.21.15.70:0/2567214125 >> 172.21.15.70:0/2567214125 conn(0x55710ddbac10 msgr2=0x55710ddc0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.541+0000 7faf82ffd640 0 client.26714 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:37:38.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.541+0000 7fafac786740 1 -- 172.21.15.70:0/2567214125 shutdown_connections 2024-04-01T01:37:38.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.541+0000 7fafac786740 1 -- 172.21.15.70:0/2567214125 wait complete. 2024-04-01T01:37:38.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.541+0000 7fafac786740 20 client.26714 trim_cache size 0 max 16384 2024-04-01T01:37:38.542 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:37:38.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.542+0000 7fafa893f640 1 -- 172.21.15.70:0/1355431443 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faf900a2620 msgr2=0x7faf900a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.542+0000 7fafa893f640 1 --2- 172.21.15.70:0/1355431443 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faf900a2620 0x7faf900a4ae0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x55710ddc0490 tx=0x7faf94005f90 comp rx=0 tx=0).stop 2024-04-01T01:37:38.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.542+0000 7fafa893f640 1 -- 172.21.15.70:0/1355431443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafa4109800 msgr2=0x7fafa41a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.542+0000 7fafa893f640 1 --2- 172.21.15.70:0/1355431443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafa4109800 0x7fafa41a5440 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7faf9c00ef00 tx=0x7faf9c00c740 comp rx=0 tx=0).stop 2024-04-01T01:37:38.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.542+0000 7fafa893f640 1 -- 172.21.15.70:0/1355431443 shutdown_connections 2024-04-01T01:37:38.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.542+0000 7fafa893f640 1 --2- 172.21.15.70:0/1355431443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafa4109800 0x7fafa41a5440 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.542+0000 7fafa893f640 1 --2- 172.21.15.70:0/1355431443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fafa4108e40 0x7fafa419e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.542+0000 7fafa893f640 1 --2- 172.21.15.70:0/1355431443 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faf900a2620 0x7faf900a4ae0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.542+0000 7fafa893f640 1 --2- 172.21.15.70:0/1355431443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafa4107c50 0x7fafa419de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.542+0000 7fafa893f640 1 -- 172.21.15.70:0/1355431443 >> 172.21.15.70:0/1355431443 conn(0x7fafa4076bf0 msgr2=0x7fafa410c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.543+0000 7fafa893f640 1 -- 172.21.15.70:0/1355431443 shutdown_connections 2024-04-01T01:37:38.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.543+0000 7fafa893f640 1 -- 172.21.15.70:0/1355431443 wait complete. 2024-04-01T01:37:38.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:37:38.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.757+0000 7f0293e1c640 1 -- 172.21.15.70:0/1228234324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f028c102220 msgr2=0x7f028c108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.757+0000 7f0293e1c640 1 --2- 172.21.15.70:0/1228234324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f028c102220 0x7f028c108ab0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0284009900 tx=0x7f028402bc40 comp rx=0 tx=0).stop 2024-04-01T01:37:38.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.757+0000 7f0293e1c640 1 -- 172.21.15.70:0/1228234324 shutdown_connections 2024-04-01T01:37:38.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.757+0000 7f0293e1c640 1 --2- 172.21.15.70:0/1228234324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f028c102220 0x7f028c108ab0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.757+0000 7f0293e1c640 1 --2- 172.21.15.70:0/1228234324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f028c101860 0x7f028c101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.757+0000 7f0293e1c640 1 --2- 172.21.15.70:0/1228234324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c100670 0x7f028c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.757+0000 7f0293e1c640 1 -- 172.21.15.70:0/1228234324 >> 172.21.15.70:0/1228234324 conn(0x7f028c0fbc00 msgr2=0x7f028c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.757+0000 7f0293e1c640 1 -- 172.21.15.70:0/1228234324 shutdown_connections 2024-04-01T01:37:38.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.758+0000 7f0293e1c640 1 -- 172.21.15.70:0/1228234324 wait complete. 2024-04-01T01:37:38.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.758+0000 7f0293e1c640 1 Processor -- start 2024-04-01T01:37:38.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.758+0000 7f0293e1c640 1 -- start start 2024-04-01T01:37:38.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.759+0000 7f0293e1c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f028c100670 0x7f028c199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.759+0000 7f0293e1c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f028c101860 0x7f028c19a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.759+0000 7f0293e1c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c102220 0x7f028c1a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.759+0000 7f0293e1c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f028c10b0b0 con 0x7f028c102220 2024-04-01T01:37:38.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.759+0000 7f0293e1c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f028c10af30 con 0x7f028c101860 2024-04-01T01:37:38.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.759+0000 7f0293e1c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f028c10b230 con 0x7f028c100670 2024-04-01T01:37:38.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.759+0000 7f02923b9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c102220 0x7f028c1a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.759+0000 7f02913b7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f028c101860 0x7f028c19a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.759+0000 7f02923b9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c102220 0x7f028c1a10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47548/0 (socket says 172.21.15.70:47548) 2024-04-01T01:37:38.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.759+0000 7f02923b9640 1 -- 172.21.15.70:0/727980253 learned_addr learned my addr 172.21.15.70:0/727980253 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:38.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.759+0000 7f0291bb8640 1 --2- 172.21.15.70:0/727980253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f028c100670 0x7f028c199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.760+0000 7f02923b9640 1 -- 172.21.15.70:0/727980253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f028c100670 msgr2=0x7f028c199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.760+0000 7f02923b9640 1 --2- 172.21.15.70:0/727980253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f028c100670 0x7f028c199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.760+0000 7f02923b9640 1 -- 172.21.15.70:0/727980253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f028c101860 msgr2=0x7f028c19a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.760+0000 7f02923b9640 1 --2- 172.21.15.70:0/727980253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f028c101860 0x7f028c19a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.760+0000 7f02923b9640 1 -- 172.21.15.70:0/727980253 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f028c1a1750 con 0x7f028c102220 2024-04-01T01:37:38.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.760+0000 7f02923b9640 1 --2- 172.21.15.70:0/727980253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c102220 0x7f028c1a10e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0284036040 tx=0x7f028402cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.761+0000 7f0282ffd640 1 -- 172.21.15.70:0/727980253 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0284043020 con 0x7f028c102220 2024-04-01T01:37:38.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.762+0000 7f0282ffd640 1 -- 172.21.15.70:0/727980253 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02840044c0 con 0x7f028c102220 2024-04-01T01:37:38.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.762+0000 7f0282ffd640 1 -- 172.21.15.70:0/727980253 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0284003bf0 con 0x7f028c102220 2024-04-01T01:37:38.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.762+0000 7f0293e1c640 1 -- 172.21.15.70:0/727980253 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f028c1a19e0 con 0x7f028c102220 2024-04-01T01:37:38.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.763+0000 7f0293e1c640 1 -- 172.21.15.70:0/727980253 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f028c1a1f30 con 0x7f028c102220 2024-04-01T01:37:38.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.766+0000 7f0296a00740 1 Processor -- start 2024-04-01T01:37:38.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.768+0000 7f0282ffd640 1 -- 172.21.15.70:0/727980253 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f0284046020 con 0x7f028c102220 2024-04-01T01:37:38.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.769+0000 7f0296a00740 1 -- start start 2024-04-01T01:37:38.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.773+0000 7f0296a00740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25e8490 0x562ef25e88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.773+0000 7f0296a00740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25e8e80 0x562ef25ed980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.773+0000 7f0296a00740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25edff0 0x562ef25f0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.773+0000 7f0296a00740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562ef254d1b0 con 0x562ef25edff0 2024-04-01T01:37:38.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.773+0000 7f0296a00740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562ef24f5c00 con 0x562ef25e8e80 2024-04-01T01:37:38.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.773+0000 7f0296a00740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562ef2549250 con 0x562ef25e8490 2024-04-01T01:37:38.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.774+0000 7f02913b7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25e8490 0x562ef25e88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.774+0000 7f02913b7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25e8490 0x562ef25e88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52798/0 (socket says 172.21.15.70:52798) 2024-04-01T01:37:38.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.774+0000 7f02913b7640 1 -- 172.21.15.70:0/506059438 learned_addr learned my addr 172.21.15.70:0/506059438 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:38.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.774+0000 7f02913b7640 1 --2- 172.21.15.70:0/506059438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25edff0 0x562ef25f0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.774+0000 7f02913b7640 1 -- 172.21.15.70:0/506059438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25e8e80 msgr2=0x562ef25ed980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:37:38.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.774+0000 7f02913b7640 1 --2- 172.21.15.70:0/506059438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25e8e80 0x562ef25ed980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.774+0000 7f02913b7640 1 -- 172.21.15.70:0/506059438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25edff0 msgr2=0x562ef25f0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.774+0000 7f02913b7640 1 --2- 172.21.15.70:0/506059438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25edff0 0x562ef25f0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.774+0000 7f0282ffd640 1 --2- 172.21.15.70:0/727980253 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f02780a2670 0x7f02780a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.774+0000 7f02913b7640 1 -- 172.21.15.70:0/506059438 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ef25153d0 con 0x562ef25e8490 2024-04-01T01:37:38.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.774+0000 7f02913b7640 1 --2- 172.21.15.70:0/506059438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25e8490 0x562ef25e88b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f027c00bfc0 tx=0x7f027c005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f63139c5daced3e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.774+0000 7f0282ffd640 1 -- 172.21.15.70:0/727980253 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f02840f0cd0 con 0x7f028c102220 2024-04-01T01:37:38.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.775+0000 7f0277fff640 1 -- 172.21.15.70:0/506059438 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c005020 con 0x562ef25e8490 2024-04-01T01:37:38.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.775+0000 7f0277fff640 1 -- 172.21.15.70:0/506059438 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f027c0047a0 con 0x562ef25e8490 2024-04-01T01:37:38.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.775+0000 7f0277fff640 1 -- 172.21.15.70:0/506059438 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c015720 con 0x562ef25e8490 2024-04-01T01:37:38.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.776+0000 7f0296a00740 1 -- 172.21.15.70:0/506059438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25e8490 msgr2=0x562ef25e88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.776+0000 7f0296a00740 1 --2- 172.21.15.70:0/506059438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25e8490 0x562ef25e88b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f027c00bfc0 tx=0x7f027c005850 comp rx=0 tx=0).stop 2024-04-01T01:37:38.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.776+0000 7f0291bb8640 1 --2- 172.21.15.70:0/727980253 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f02780a2670 0x7f02780a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.776+0000 7f0296a00740 1 -- 172.21.15.70:0/506059438 shutdown_connections 2024-04-01T01:37:38.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.776+0000 7f0296a00740 1 --2- 172.21.15.70:0/506059438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25edff0 0x562ef25f0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.776+0000 7f0296a00740 1 --2- 172.21.15.70:0/506059438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25e8e80 0x562ef25ed980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.776+0000 7f0296a00740 1 --2- 172.21.15.70:0/506059438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25e8490 0x562ef25e88b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.776+0000 7f0296a00740 1 -- 172.21.15.70:0/506059438 >> 172.21.15.70:0/506059438 conn(0x562ef25e3a00 msgr2=0x562ef25e3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.777+0000 7f0291bb8640 1 --2- 172.21.15.70:0/727980253 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f02780a2670 0x7f02780a4b30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x562ef25e4970 tx=0x7f0288009670 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.777+0000 7f0296a00740 1 -- 172.21.15.70:0/506059438 shutdown_connections 2024-04-01T01:37:38.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.777+0000 7f0296a00740 1 -- 172.21.15.70:0/506059438 wait complete. 2024-04-01T01:37:38.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.777+0000 7f0296a00740 1 -- start start 2024-04-01T01:37:38.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.777+0000 7f0296a00740 1 Processor -- start 2024-04-01T01:37:38.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.778+0000 7f0296a00740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25e8490 0x562ef25f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.778+0000 7f0296a00740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25e8e80 0x562ef25f3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.778+0000 7f0296a00740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25edff0 0x562ef25fa020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.778+0000 7f0296a00740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562ef25f0ff0 con 0x562ef25e8490 2024-04-01T01:37:38.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.778+0000 7f0296a00740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562ef24eb470 con 0x562ef25e8e80 2024-04-01T01:37:38.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.778+0000 7f0296a00740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562ef25f1170 con 0x562ef25edff0 2024-04-01T01:37:38.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.778+0000 7f0296a00740 10 client.0 _ll_register_callbacks cb 0x562ef2508cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:38.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.778+0000 7f0291bb8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25e8490 0x562ef25f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.778+0000 7f0291bb8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25e8490 0x562ef25f2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47578/0 (socket says 172.21.15.70:47578) 2024-04-01T01:37:38.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.778+0000 7f0291bb8640 1 -- 172.21.15.70:0/2552535145 learned_addr learned my addr 172.21.15.70:0/2552535145 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:38.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.778+0000 7f02923b9640 1 --2- 172.21.15.70:0/2552535145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25edff0 0x562ef25fa020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.779+0000 7f0291bb8640 1 -- 172.21.15.70:0/2552535145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25edff0 msgr2=0x562ef25fa020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.779+0000 7f0291bb8640 1 --2- 172.21.15.70:0/2552535145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25edff0 0x562ef25fa020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.779+0000 7f0291bb8640 1 -- 172.21.15.70:0/2552535145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25e8e80 msgr2=0x562ef25f3050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:37:38.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.779+0000 7f0291bb8640 1 --2- 172.21.15.70:0/2552535145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25e8e80 0x562ef25f3050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.779+0000 7f0291bb8640 1 -- 172.21.15.70:0/2552535145 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ef22613f0 con 0x562ef25e8490 2024-04-01T01:37:38.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.779+0000 7f0291bb8640 1 --2- 172.21.15.70:0/2552535145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25e8490 0x562ef25f2b10 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f02880127b0 tx=0x7f0288004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c399b2aa30e3a09 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.779+0000 7f02757fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:37:38.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.780+0000 7f02757fa640 1 -- 172.21.15.70:0/2552535145 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0288034070 con 0x562ef25e8490 2024-04-01T01:37:38.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.780+0000 7f0296a00740 1 -- 172.21.15.70:0/2552535145 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f027c0047a0 con 0x562ef25e8490 2024-04-01T01:37:38.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.781+0000 7f02757fa640 1 -- 172.21.15.70:0/2552535145 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0288011ae0 con 0x562ef25e8490 2024-04-01T01:37:38.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.782+0000 7f02757fa640 1 -- 172.21.15.70:0/2552535145 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0288040050 con 0x562ef25e8490 2024-04-01T01:37:38.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.782+0000 7f0296a00740 10 client.16857 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:38.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.782+0000 7f0296a00740 1 -- 172.21.15.70:0/2552535145 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562ef232b7d0 con 0x562ef25e8490 2024-04-01T01:37:38.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.783+0000 7f02757fa640 1 -- 172.21.15.70:0/2552535145 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f027c0047a0 con 0x562ef25e8490 2024-04-01T01:37:38.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.783+0000 7f02757fa640 1 -- 172.21.15.70:0/2552535145 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f028800f040 con 0x562ef25e8490 2024-04-01T01:37:38.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.783+0000 7f0296a00740 10 client.16857 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:38.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.783+0000 7f0296a00740 10 client.16857 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:38.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.783+0000 7f0296a00740 20 client.16857 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:38.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.783+0000 7f0296a00740 1 --2- 172.21.15.70:0/2552535145 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562ef25fbc20 0x562ef25fe0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.783+0000 7f0296a00740 4 client.16857 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:38.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.783+0000 7f0296a00740 1 -- 172.21.15.70:0/2552535145 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ef255a6f0 con 0x562ef25fbc20 2024-04-01T01:37:38.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.784+0000 7f02913b7640 1 --2- 172.21.15.70:0/2552535145 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562ef25fbc20 0x562ef25fe0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.785+0000 7f02757fa640 10 client.16857 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:38.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.785+0000 7f02913b7640 1 --2- 172.21.15.70:0/2552535145 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562ef25fbc20 0x562ef25fe0e0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d0d69aac2f4727a server_cookie=33bfbcc713071ee7 in_seq=0 out_seq=0 2024-04-01T01:37:38.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.786+0000 7f02757fa640 1 -- 172.21.15.70:0/2552535145 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562ef255a6f0 con 0x562ef25fbc20 2024-04-01T01:37:38.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.786+0000 7f02757fa640 10 client.16857 handle_command_reply: tid=0 2024-04-01T01:37:38.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.786+0000 7f0296a00740 1 client.16857 shutdown 2024-04-01T01:37:38.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.786+0000 7f0296a00740 1 -- 172.21.15.70:0/2552535145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25e8490 msgr2=0x562ef25f2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.786+0000 7f0296a00740 1 --2- 172.21.15.70:0/2552535145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25e8490 0x562ef25f2b10 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f02880127b0 tx=0x7f0288004a40 comp rx=0 tx=0).stop 2024-04-01T01:37:38.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.787+0000 7f0296a00740 1 -- 172.21.15.70:0/2552535145 shutdown_connections 2024-04-01T01:37:38.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.787+0000 7f0296a00740 1 --2- 172.21.15.70:0/2552535145 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562ef25fbc20 0x562ef25fe0e0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.787+0000 7f0296a00740 1 --2- 172.21.15.70:0/2552535145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25edff0 0x562ef25fa020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.787+0000 7f0296a00740 1 --2- 172.21.15.70:0/2552535145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25e8e80 0x562ef25f3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.787+0000 7f0296a00740 1 --2- 172.21.15.70:0/2552535145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25e8490 0x562ef25f2b10 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.787+0000 7f0296a00740 1 -- 172.21.15.70:0/2552535145 >> 172.21.15.70:0/2552535145 conn(0x562ef25e3a00 msgr2=0x562ef25e2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.787+0000 7f02757fa640 0 client.16857 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:38.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.787+0000 7f0296a00740 1 -- 172.21.15.70:0/2552535145 shutdown_connections 2024-04-01T01:37:38.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.787+0000 7f0296a00740 1 -- 172.21.15.70:0/2552535145 wait complete. 2024-04-01T01:37:38.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.787+0000 7f0296a00740 20 client.16857 trim_cache size 0 max 16384 2024-04-01T01:37:38.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.812+0000 7f0296a00740 1 Processor -- start 2024-04-01T01:37:38.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.812+0000 7f0296a00740 1 -- start start 2024-04-01T01:37:38.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.813+0000 7f0296a00740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25e8490 0x562ef25f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.813+0000 7f0296a00740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25e8e80 0x562ef25e98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.813+0000 7f0296a00740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25edff0 0x562ef25e3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.813+0000 7f0296a00740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f028c1a8150 con 0x562ef25e8490 2024-04-01T01:37:38.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.813+0000 7f0296a00740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f028c1a9c90 con 0x562ef25edff0 2024-04-01T01:37:38.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.813+0000 7f0296a00740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f028c1a8820 con 0x562ef25e8e80 2024-04-01T01:37:38.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.813+0000 7f02923b9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25edff0 0x562ef25e3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.813+0000 7f02923b9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25edff0 0x562ef25e3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53750/0 (socket says 172.21.15.70:53750) 2024-04-01T01:37:38.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.813+0000 7f02923b9640 1 -- 172.21.15.70:0/3733789402 learned_addr learned my addr 172.21.15.70:0/3733789402 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:38.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.813+0000 7f02923b9640 1 -- 172.21.15.70:0/3733789402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25e8e80 msgr2=0x562ef25e98f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:37:38.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.813+0000 7f02923b9640 1 --2- 172.21.15.70:0/3733789402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25e8e80 0x562ef25e98f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.813+0000 7f02923b9640 1 -- 172.21.15.70:0/3733789402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25e8490 msgr2=0x562ef25f2fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:37:38.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.813+0000 7f02923b9640 1 --2- 172.21.15.70:0/3733789402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25e8490 0x562ef25f2fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.813+0000 7f02923b9640 1 -- 172.21.15.70:0/3733789402 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ef25ed740 con 0x562ef25edff0 2024-04-01T01:37:38.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.814+0000 7f02923b9640 1 --2- 172.21.15.70:0/3733789402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25edff0 0x562ef25e3de0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f02840ee3a0 tx=0x7f0284005a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce8bd5bec2cfa8eb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.814+0000 7f02777fe640 1 -- 172.21.15.70:0/3733789402 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02840d5020 con 0x562ef25edff0 2024-04-01T01:37:38.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.814+0000 7f02777fe640 1 -- 172.21.15.70:0/3733789402 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02840bda60 con 0x562ef25edff0 2024-04-01T01:37:38.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.814+0000 7f02777fe640 1 -- 172.21.15.70:0/3733789402 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02840ee4b0 con 0x562ef25edff0 2024-04-01T01:37:38.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.815+0000 7f0296a00740 1 -- 172.21.15.70:0/3733789402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25edff0 msgr2=0x562ef25e3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.815+0000 7f0296a00740 1 --2- 172.21.15.70:0/3733789402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25edff0 0x562ef25e3de0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f02840ee3a0 tx=0x7f0284005a40 comp rx=0 tx=0).stop 2024-04-01T01:37:38.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.817+0000 7f0296a00740 1 -- 172.21.15.70:0/3733789402 shutdown_connections 2024-04-01T01:37:38.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.817+0000 7f0296a00740 1 --2- 172.21.15.70:0/3733789402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25edff0 0x562ef25e3de0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.817+0000 7f0296a00740 1 --2- 172.21.15.70:0/3733789402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25e8e80 0x562ef25e98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.817+0000 7f0296a00740 1 --2- 172.21.15.70:0/3733789402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25e8490 0x562ef25f2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.817+0000 7f0296a00740 1 -- 172.21.15.70:0/3733789402 >> 172.21.15.70:0/3733789402 conn(0x562ef25e3a00 msgr2=0x562ef25f1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.817+0000 7f0296a00740 1 -- 172.21.15.70:0/3733789402 shutdown_connections 2024-04-01T01:37:38.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.817+0000 7f0296a00740 1 -- 172.21.15.70:0/3733789402 wait complete. 2024-04-01T01:37:38.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.818+0000 7f0296a00740 1 -- start start 2024-04-01T01:37:38.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.818+0000 7f0296a00740 1 Processor -- start 2024-04-01T01:37:38.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.818+0000 7f0296a00740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25e8490 0x562ef260ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.818+0000 7f0296a00740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25e8e80 0x562ef2660e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.818+0000 7f0296a00740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25edff0 0x562ef2663470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.818+0000 7f0296a00740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f028c1a95c0 con 0x562ef25edff0 2024-04-01T01:37:38.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.818+0000 7f0296a00740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f028c1a8ef0 con 0x562ef25e8e80 2024-04-01T01:37:38.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.818+0000 7f0296a00740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f028c1a6130 con 0x562ef25e8490 2024-04-01T01:37:38.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.818+0000 7f0296a00740 10 client.0 _ll_register_callbacks cb 0x7f0288012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:38.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.819+0000 7f02913b7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25e8e80 0x562ef2660e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.819+0000 7f0291bb8640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25e8490 0x562ef260ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.819+0000 7f02913b7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25e8e80 0x562ef2660e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53762/0 (socket says 172.21.15.70:53762) 2024-04-01T01:37:38.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.819+0000 7f02913b7640 1 -- 172.21.15.70:0/2231979778 learned_addr learned my addr 172.21.15.70:0/2231979778 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:38.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.820+0000 7f02913b7640 1 -- 172.21.15.70:0/2231979778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25e8490 msgr2=0x562ef260ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.820+0000 7f02913b7640 1 --2- 172.21.15.70:0/2231979778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25e8490 0x562ef260ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.820+0000 7f02913b7640 1 -- 172.21.15.70:0/2231979778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25edff0 msgr2=0x562ef2663470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:38.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.820+0000 7f02913b7640 1 --2- 172.21.15.70:0/2231979778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25edff0 0x562ef2663470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.820+0000 7f02913b7640 1 -- 172.21.15.70:0/2231979778 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ef2663ca0 con 0x562ef25e8e80 2024-04-01T01:37:38.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.820+0000 7f02913b7640 1 --2- 172.21.15.70:0/2231979778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25e8e80 0x562ef2660e30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f027c00b840 tx=0x7f027c00baf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70f272e636e2a495 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:38.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.820+0000 7f02757fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:38.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.820+0000 7f02757fa640 1 -- 172.21.15.70:0/2231979778 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c02d060 con 0x562ef25e8e80 2024-04-01T01:37:38.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.820+0000 7f02757fa640 1 -- 172.21.15.70:0/2231979778 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f027c004fa0 con 0x562ef25e8e80 2024-04-01T01:37:38.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.821+0000 7f0296a00740 1 -- 172.21.15.70:0/2231979778 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02840bda60 con 0x562ef25e8e80 2024-04-01T01:37:38.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.829+0000 7f02757fa640 1 -- 172.21.15.70:0/2231979778 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f027c034050 con 0x562ef25e8e80 2024-04-01T01:37:38.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.829+0000 7f0296a00740 10 client.26726 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:38.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.829+0000 7f0296a00740 1 -- 172.21.15.70:0/2231979778 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562ef2664fd0 con 0x562ef25e8e80 2024-04-01T01:37:38.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.830+0000 7f02757fa640 1 -- 172.21.15.70:0/2231979778 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02840bda60 con 0x562ef25e8e80 2024-04-01T01:37:38.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.830+0000 7f02757fa640 1 -- 172.21.15.70:0/2231979778 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f027c004560 con 0x562ef25e8e80 2024-04-01T01:37:38.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.830+0000 7f0296a00740 10 client.26726 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:38.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.830+0000 7f0296a00740 10 client.26726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:38.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.830+0000 7f0296a00740 20 client.26726 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:38.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.830+0000 7f0296a00740 1 --2- 172.21.15.70:0/2231979778 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562ef26658c0 0x562ef2667ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:38.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.830+0000 7f0296a00740 4 client.26726 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:37:38.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.830+0000 7f0296a00740 1 -- 172.21.15.70:0/2231979778 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562ef2554a40 con 0x562ef26658c0 2024-04-01T01:37:38.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.832+0000 7f0291bb8640 1 --2- 172.21.15.70:0/2231979778 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562ef26658c0 0x562ef2667ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:38.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.832+0000 7f0291bb8640 1 --2- 172.21.15.70:0/2231979778 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562ef26658c0 0x562ef2667ce0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=465761dc8c576dfd server_cookie=b52acb6bc479d83e in_seq=0 out_seq=0 2024-04-01T01:37:38.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.832+0000 7f02757fa640 10 client.26726 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:38.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.833+0000 7f02757fa640 1 -- 172.21.15.70:0/2231979778 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562ef2554a40 con 0x562ef26658c0 2024-04-01T01:37:38.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.833+0000 7f02757fa640 10 client.26726 handle_command_reply: tid=0 2024-04-01T01:37:38.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.833+0000 7f0296a00740 1 client.26726 shutdown 2024-04-01T01:37:38.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.834+0000 7f0296a00740 1 -- 172.21.15.70:0/2231979778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25e8e80 msgr2=0x562ef2660e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.834+0000 7f0296a00740 1 --2- 172.21.15.70:0/2231979778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25e8e80 0x562ef2660e30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f027c00b840 tx=0x7f027c00baf0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.835+0000 7f0296a00740 1 -- 172.21.15.70:0/2231979778 shutdown_connections 2024-04-01T01:37:38.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.835+0000 7f0296a00740 1 --2- 172.21.15.70:0/2231979778 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562ef26658c0 0x562ef2667ce0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.835+0000 7f0296a00740 1 --2- 172.21.15.70:0/2231979778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ef25edff0 0x562ef2663470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.835+0000 7f0296a00740 1 --2- 172.21.15.70:0/2231979778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ef25e8e80 0x562ef2660e30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.835+0000 7f0296a00740 1 --2- 172.21.15.70:0/2231979778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ef25e8490 0x562ef260ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.835+0000 7f0296a00740 1 -- 172.21.15.70:0/2231979778 >> 172.21.15.70:0/2231979778 conn(0x562ef25e3a00 msgr2=0x562ef25f1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.835+0000 7f02757fa640 0 client.26726 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:38.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.836+0000 7f0296a00740 1 -- 172.21.15.70:0/2231979778 shutdown_connections 2024-04-01T01:37:38.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.837+0000 7f0296a00740 1 -- 172.21.15.70:0/2231979778 wait complete. 2024-04-01T01:37:38.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.837+0000 7f0296a00740 20 client.26726 trim_cache size 0 max 16384 2024-04-01T01:37:38.837 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:37:38.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.838+0000 7f0293e1c640 1 -- 172.21.15.70:0/727980253 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f02780a2670 msgr2=0x7f02780a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.838+0000 7f0293e1c640 1 --2- 172.21.15.70:0/727980253 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f02780a2670 0x7f02780a4b30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x562ef25e4970 tx=0x7f0288009670 comp rx=0 tx=0).stop 2024-04-01T01:37:38.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.838+0000 7f0293e1c640 1 -- 172.21.15.70:0/727980253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c102220 msgr2=0x7f028c1a10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:38.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.839+0000 7f0293e1c640 1 --2- 172.21.15.70:0/727980253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c102220 0x7f028c1a10e0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0284036040 tx=0x7f028402cde0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.839+0000 7f0293e1c640 1 -- 172.21.15.70:0/727980253 shutdown_connections 2024-04-01T01:37:38.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.839+0000 7f0293e1c640 1 --2- 172.21.15.70:0/727980253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f028c102220 0x7f028c1a10e0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.839+0000 7f0293e1c640 1 --2- 172.21.15.70:0/727980253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f028c101860 0x7f028c19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.839+0000 7f0293e1c640 1 --2- 172.21.15.70:0/727980253 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f02780a2670 0x7f02780a4b30 secure :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0x562ef25e4970 tx=0x7f0288009670 comp rx=0 tx=0).stop 2024-04-01T01:37:38.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.839+0000 7f0293e1c640 1 --2- 172.21.15.70:0/727980253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f028c100670 0x7f028c199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:38.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.839+0000 7f0293e1c640 1 -- 172.21.15.70:0/727980253 >> 172.21.15.70:0/727980253 conn(0x7f028c0fbc00 msgr2=0x7f028c104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:38.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.840+0000 7f0293e1c640 1 -- 172.21.15.70:0/727980253 shutdown_connections 2024-04-01T01:37:38.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:38.840+0000 7f0293e1c640 1 -- 172.21.15.70:0/727980253 wait complete. 2024-04-01T01:37:39.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:38 smithi070 ceph-mon[32179]: pgmap v483: 129 pgs: 129 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 1.03k op/s 2024-04-01T01:37:39.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:38 smithi189 ceph-mon[35753]: pgmap v483: 129 pgs: 129 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 1.03k op/s 2024-04-01T01:37:39.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:38 smithi081 ceph-mon[35142]: pgmap v483: 129 pgs: 129 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 1.03k op/s 2024-04-01T01:37:39.862 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:37:39.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:37:39.863 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:37:40.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.052+0000 7f1f01ac9640 1 -- 172.21.15.70:0/4028188807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1efc109110 msgr2=0x7f1efc1123c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.052+0000 7f1f01ac9640 1 --2- 172.21.15.70:0/4028188807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1efc109110 0x7f1efc1123c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009cd0 tx=0x7f1ef402c0e0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.052+0000 7f1f01ac9640 1 -- 172.21.15.70:0/4028188807 shutdown_connections 2024-04-01T01:37:40.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.052+0000 7f1f01ac9640 1 --2- 172.21.15.70:0/4028188807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1efc109110 0x7f1efc1123c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.052+0000 7f1f01ac9640 1 --2- 172.21.15.70:0/4028188807 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1efc108750 0x7f1efc108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.052+0000 7f1f01ac9640 1 --2- 172.21.15.70:0/4028188807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1efc109860 0x7f1efc10bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.052+0000 7f1f01ac9640 1 -- 172.21.15.70:0/4028188807 >> 172.21.15.70:0/4028188807 conn(0x7f1efc06e5b0 msgr2=0x7f1efc0709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:40.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.053+0000 7f1f01ac9640 1 -- 172.21.15.70:0/4028188807 shutdown_connections 2024-04-01T01:37:40.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.053+0000 7f1f01ac9640 1 -- 172.21.15.70:0/4028188807 wait complete. 2024-04-01T01:37:40.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.054+0000 7f1f01ac9640 1 Processor -- start 2024-04-01T01:37:40.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.054+0000 7f1f01ac9640 1 -- start start 2024-04-01T01:37:40.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.054+0000 7f1f01ac9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1efc108750 0x7f1efc119270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.054+0000 7f1f01ac9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1efc109110 0x7f1efc1197b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.054+0000 7f1f01ac9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1efc109860 0x7f1efc1207d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.054+0000 7f1f01ac9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc10da40 con 0x7f1efc109860 2024-04-01T01:37:40.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.054+0000 7f1f01ac9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc10d8c0 con 0x7f1efc108750 2024-04-01T01:37:40.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.054+0000 7f1f01ac9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc10dbc0 con 0x7f1efc109110 2024-04-01T01:37:40.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.055+0000 7f1efbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1efc109860 0x7f1efc1207d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.055+0000 7f1efb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1efc108750 0x7f1efc119270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.055+0000 7f1ef2ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1efc109110 0x7f1efc1197b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.055+0000 7f1ef2ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1efc109110 0x7f1efc1197b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52838/0 (socket says 172.21.15.70:52838) 2024-04-01T01:37:40.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.055+0000 7f1ef2ffd640 1 -- 172.21.15.70:0/710428272 learned_addr learned my addr 172.21.15.70:0/710428272 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:40.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.055+0000 7f1ef2ffd640 1 -- 172.21.15.70:0/710428272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1efc108750 msgr2=0x7f1efc119270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.055+0000 7f1ef2ffd640 1 --2- 172.21.15.70:0/710428272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1efc108750 0x7f1efc119270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.055+0000 7f1ef2ffd640 1 -- 172.21.15.70:0/710428272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1efc109860 msgr2=0x7f1efc1207d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.055+0000 7f1ef2ffd640 1 --2- 172.21.15.70:0/710428272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1efc109860 0x7f1efc1207d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.055+0000 7f1ef2ffd640 1 -- 172.21.15.70:0/710428272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1efc1b6b80 con 0x7f1efc109110 2024-04-01T01:37:40.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.056+0000 7f1ef2ffd640 1 --2- 172.21.15.70:0/710428272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1efc109110 0x7f1efc1197b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1ee400d7a0 tx=0x7f1ee400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:40.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.056+0000 7f1ef97fa640 1 -- 172.21.15.70:0/710428272 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4017070 con 0x7f1efc109110 2024-04-01T01:37:40.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.056+0000 7f1ef97fa640 1 -- 172.21.15.70:0/710428272 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ee4009c30 con 0x7f1efc109110 2024-04-01T01:37:40.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.056+0000 7f1ef97fa640 1 -- 172.21.15.70:0/710428272 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4003cc0 con 0x7f1efc109110 2024-04-01T01:37:40.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.056+0000 7f1f01ac9640 1 -- 172.21.15.70:0/710428272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1efc1b6e70 con 0x7f1efc109110 2024-04-01T01:37:40.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.056+0000 7f1f01ac9640 1 -- 172.21.15.70:0/710428272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1efc1b72e0 con 0x7f1efc109110 2024-04-01T01:37:40.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.060+0000 7f1f04787740 1 Processor -- start 2024-04-01T01:37:40.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.060+0000 7f1f04787740 1 -- start start 2024-04-01T01:37:40.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.060+0000 7f1f04787740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56133683aa10 0x56133683ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.060+0000 7f1f04787740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683b400 0x56133683ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.060+0000 7f1f04787740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 0x561336842a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.060+0000 7f1f04787740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56133673c0b0 con 0x56133683aa10 2024-04-01T01:37:40.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.060+0000 7f1f04787740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56133679eea0 con 0x56133683b400 2024-04-01T01:37:40.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.060+0000 7f1f04787740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561336747c00 con 0x561336840570 2024-04-01T01:37:40.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.061+0000 7f1ef2ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 0x561336842a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.061+0000 7f1ef2ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 0x561336842a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52848/0 (socket says 172.21.15.70:52848) 2024-04-01T01:37:40.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.061+0000 7f1ef2ffd640 1 -- 172.21.15.70:0/55242893 learned_addr learned my addr 172.21.15.70:0/55242893 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:40.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.061+0000 7f1ef97fa640 1 -- 172.21.15.70:0/710428272 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f1ee4003e60 con 0x7f1efc109110 2024-04-01T01:37:40.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.061+0000 7f1efbfff640 1 --2- 172.21.15.70:0/55242893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56133683aa10 0x56133683ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.061+0000 7f1ef2ffd640 1 -- 172.21.15.70:0/55242893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683b400 msgr2=0x56133683ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:40.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.061+0000 7f1ef2ffd640 1 --2- 172.21.15.70:0/55242893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683b400 0x56133683ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.061+0000 7f1ef2ffd640 1 -- 172.21.15.70:0/55242893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56133683aa10 msgr2=0x56133683ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:40.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.061+0000 7f1ef2ffd640 1 --2- 172.21.15.70:0/55242893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56133683aa10 0x56133683ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.061+0000 7f1ef2ffd640 1 -- 172.21.15.70:0/55242893 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56133657b7e0 con 0x561336840570 2024-04-01T01:37:40.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.062+0000 7f1ef2ffd640 1 --2- 172.21.15.70:0/55242893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 0x561336842a00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1ee40d64d0 tx=0x7f1ee40d6990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bfa72f55f3a9278 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:40.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.062+0000 7f1ef97fa640 1 --2- 172.21.15.70:0/710428272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1edc0a2670 0x7f1edc0a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.062+0000 7f1ef97fa640 1 -- 172.21.15.70:0/710428272 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1ee40ce310 con 0x7f1efc109110 2024-04-01T01:37:40.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.063+0000 7f1efbfff640 1 --2- 172.21.15.70:0/710428272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1edc0a2670 0x7f1edc0a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.063+0000 7f1ef1ffb640 1 -- 172.21.15.70:0/55242893 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee40e1070 con 0x561336840570 2024-04-01T01:37:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.063+0000 7f1ef1ffb640 1 -- 172.21.15.70:0/55242893 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ee4009de0 con 0x561336840570 2024-04-01T01:37:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.063+0000 7f1ef1ffb640 1 -- 172.21.15.70:0/55242893 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4096cf0 con 0x561336840570 2024-04-01T01:37:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.064+0000 7f1f04787740 1 -- 172.21.15.70:0/55242893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 msgr2=0x561336842a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.064+0000 7f1f04787740 1 --2- 172.21.15.70:0/55242893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 0x561336842a00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1ee40d64d0 tx=0x7f1ee40d6990 comp rx=0 tx=0).stop 2024-04-01T01:37:40.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.064+0000 7f1efbfff640 1 --2- 172.21.15.70:0/710428272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1edc0a2670 0x7f1edc0a4b30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1ef402c620 tx=0x7f1ef402e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:40.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.064+0000 7f1f04787740 1 -- 172.21.15.70:0/55242893 shutdown_connections 2024-04-01T01:37:40.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.064+0000 7f1f04787740 1 --2- 172.21.15.70:0/55242893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 0x561336842a00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.064+0000 7f1f04787740 1 --2- 172.21.15.70:0/55242893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683b400 0x56133683ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.064+0000 7f1f04787740 1 --2- 172.21.15.70:0/55242893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56133683aa10 0x56133683ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.064+0000 7f1f04787740 1 -- 172.21.15.70:0/55242893 >> 172.21.15.70:0/55242893 conn(0x561336835f80 msgr2=0x561336836380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:40.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.064+0000 7f1f04787740 1 -- 172.21.15.70:0/55242893 shutdown_connections 2024-04-01T01:37:40.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.065+0000 7f1f04787740 1 -- 172.21.15.70:0/55242893 wait complete. 2024-04-01T01:37:40.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.065+0000 7f1f04787740 1 -- start start 2024-04-01T01:37:40.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.065+0000 7f1f04787740 1 Processor -- start 2024-04-01T01:37:40.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.066+0000 7f1f04787740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683aa10 0x561336844e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.066+0000 7f1f04787740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56133683b400 0x5613368453a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.066+0000 7f1f04787740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 0x56133684c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.066+0000 7f1f04787740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56133673d3d0 con 0x56133683b400 2024-04-01T01:37:40.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.066+0000 7f1f04787740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56133679af40 con 0x56133683aa10 2024-04-01T01:37:40.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.066+0000 7f1f04787740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561336843520 con 0x561336840570 2024-04-01T01:37:40.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.066+0000 7f1f04787740 10 client.0 _ll_register_callbacks cb 0x56133675ab70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:40.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.066+0000 7f1efbfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 0x56133684c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.066+0000 7f1efb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683aa10 0x561336844e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.066+0000 7f1efb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683aa10 0x561336844e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53796/0 (socket says 172.21.15.70:53796) 2024-04-01T01:37:40.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.066+0000 7f1efb7fe640 1 -- 172.21.15.70:0/3548914735 learned_addr learned my addr 172.21.15.70:0/3548914735 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:40.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.067+0000 7f1ef2ffd640 1 --2- 172.21.15.70:0/3548914735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56133683b400 0x5613368453a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.067+0000 7f1efb7fe640 1 -- 172.21.15.70:0/3548914735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 msgr2=0x56133684c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:40.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.067+0000 7f1efb7fe640 1 --2- 172.21.15.70:0/3548914735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 0x56133684c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.067+0000 7f1efb7fe640 1 -- 172.21.15.70:0/3548914735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56133683b400 msgr2=0x5613368453a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:40.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.067+0000 7f1efb7fe640 1 --2- 172.21.15.70:0/3548914735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56133683b400 0x5613368453a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.067+0000 7f1efb7fe640 1 -- 172.21.15.70:0/3548914735 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613365dbf70 con 0x56133683aa10 2024-04-01T01:37:40.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.067+0000 7f1efb7fe640 1 --2- 172.21.15.70:0/3548914735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683aa10 0x561336844e60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1eec0116c0 tx=0x7f1eec011b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a72ce06449df2c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:40.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.067+0000 7f1eeaffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:40.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.068+0000 7f1eeaffd640 1 -- 172.21.15.70:0/3548914735 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eec00fcb0 con 0x56133683aa10 2024-04-01T01:37:40.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.068+0000 7f1f04787740 1 -- 172.21.15.70:0/3548914735 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ee4009de0 con 0x56133683aa10 2024-04-01T01:37:40.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.068+0000 7f1eeaffd640 1 -- 172.21.15.70:0/3548914735 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1eec005020 con 0x56133683aa10 2024-04-01T01:37:40.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.068+0000 7f1eeaffd640 1 -- 172.21.15.70:0/3548914735 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1eec024050 con 0x56133683aa10 2024-04-01T01:37:40.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.069+0000 7f1f04787740 10 client.26735 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:40.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.069+0000 7f1f04787740 1 -- 172.21.15.70:0/3548914735 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561336473170 con 0x56133683aa10 2024-04-01T01:37:40.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.069+0000 7f1eeaffd640 1 -- 172.21.15.70:0/3548914735 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ee4009de0 con 0x56133683aa10 2024-04-01T01:37:40.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.069+0000 7f1eeaffd640 1 -- 172.21.15.70:0/3548914735 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1eec019430 con 0x56133683aa10 2024-04-01T01:37:40.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.069+0000 7f1f04787740 10 client.26735 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:40.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.069+0000 7f1f04787740 10 client.26735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.069+0000 7f1f04787740 20 client.26735 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.069+0000 7f1f04787740 1 --2- 172.21.15.70:0/3548914735 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561336850cc0 0x561336853180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.069+0000 7f1f04787740 4 client.26735 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.069+0000 7f1f04787740 1 -- 172.21.15.70:0/3548914735 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613367ac3e0 con 0x561336850cc0 2024-04-01T01:37:40.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.070+0000 7f1ef2ffd640 1 --2- 172.21.15.70:0/3548914735 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561336850cc0 0x561336853180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.071+0000 7f1ef2ffd640 1 --2- 172.21.15.70:0/3548914735 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561336850cc0 0x561336853180 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1ae45c9ea18fbc7 server_cookie=c7e9c251e8057f41 in_seq=0 out_seq=0 2024-04-01T01:37:40.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.071+0000 7f1eeaffd640 10 client.26735 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:40.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.072+0000 7f1eeaffd640 1 -- 172.21.15.70:0/3548914735 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5613367ac3e0 con 0x561336850cc0 2024-04-01T01:37:40.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.072+0000 7f1eeaffd640 10 client.26735 handle_command_reply: tid=0 2024-04-01T01:37:40.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.072+0000 7f1f04787740 1 client.26735 shutdown 2024-04-01T01:37:40.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.073+0000 7f1f04787740 1 -- 172.21.15.70:0/3548914735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683aa10 msgr2=0x561336844e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.073+0000 7f1f04787740 1 --2- 172.21.15.70:0/3548914735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683aa10 0x561336844e60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1eec0116c0 tx=0x7f1eec011b80 comp rx=0 tx=0).stop 2024-04-01T01:37:40.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.073+0000 7f1f04787740 1 -- 172.21.15.70:0/3548914735 shutdown_connections 2024-04-01T01:37:40.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.073+0000 7f1f04787740 1 --2- 172.21.15.70:0/3548914735 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561336850cc0 0x561336853180 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.073+0000 7f1f04787740 1 --2- 172.21.15.70:0/3548914735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 0x56133684c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.073+0000 7f1f04787740 1 --2- 172.21.15.70:0/3548914735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56133683b400 0x5613368453a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.073+0000 7f1f04787740 1 --2- 172.21.15.70:0/3548914735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683aa10 0x561336844e60 secure :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1eec0116c0 tx=0x7f1eec011b80 comp rx=0 tx=0).stop 2024-04-01T01:37:40.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.073+0000 7f1f04787740 1 -- 172.21.15.70:0/3548914735 >> 172.21.15.70:0/3548914735 conn(0x561336835f80 msgr2=0x561336835180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:40.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.073+0000 7f1eeaffd640 0 client.26735 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:40.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.073+0000 7f1f04787740 1 -- 172.21.15.70:0/3548914735 shutdown_connections 2024-04-01T01:37:40.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.073+0000 7f1f04787740 1 -- 172.21.15.70:0/3548914735 wait complete. 2024-04-01T01:37:40.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.074+0000 7f1f04787740 20 client.26735 trim_cache size 0 max 16384 2024-04-01T01:37:40.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.106+0000 7f1f04787740 1 Processor -- start 2024-04-01T01:37:40.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.107+0000 7f1f04787740 1 -- start start 2024-04-01T01:37:40.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.107+0000 7f1f04787740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683b400 0x5613368398f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.107+0000 7f1f04787740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 0x56133684c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.107+0000 7f1f04787740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613368444b0 0x561336844930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.107+0000 7f1f04787740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1c1b00 con 0x5613368444b0 2024-04-01T01:37:40.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.107+0000 7f1f04787740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1c3640 con 0x56133683b400 2024-04-01T01:37:40.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.107+0000 7f1f04787740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1c21d0 con 0x561336840570 2024-04-01T01:37:40.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.107+0000 7f1ef2ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 0x56133684c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.107+0000 7f1ef2ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 0x56133684c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52864/0 (socket says 172.21.15.70:52864) 2024-04-01T01:37:40.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.107+0000 7f1ef2ffd640 1 -- 172.21.15.70:0/1059007322 learned_addr learned my addr 172.21.15.70:0/1059007322 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:40.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.107+0000 7f1efbfff640 1 --2- 172.21.15.70:0/1059007322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613368444b0 0x561336844930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.108+0000 7f1ef2ffd640 1 -- 172.21.15.70:0/1059007322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683b400 msgr2=0x5613368398f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:37:40.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.108+0000 7f1ef2ffd640 1 --2- 172.21.15.70:0/1059007322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683b400 0x5613368398f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.108+0000 7f1ef2ffd640 1 -- 172.21.15.70:0/1059007322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613368444b0 msgr2=0x561336844930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:40.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.108+0000 7f1ef2ffd640 1 --2- 172.21.15.70:0/1059007322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613368444b0 0x561336844930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.108+0000 7f1ef2ffd640 1 -- 172.21.15.70:0/1059007322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561336840230 con 0x561336840570 2024-04-01T01:37:40.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.108+0000 7f1efbfff640 1 --2- 172.21.15.70:0/1059007322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613368444b0 0x561336844930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:40.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.108+0000 7f1ef2ffd640 1 --2- 172.21.15.70:0/1059007322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 0x56133684c3a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1ee40d6260 tx=0x7f1ee4005f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32f360b244d602a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:40.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.108+0000 7f1ef17fa640 1 -- 172.21.15.70:0/1059007322 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4020040 con 0x561336840570 2024-04-01T01:37:40.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.108+0000 7f1ef17fa640 1 -- 172.21.15.70:0/1059007322 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ee4009de0 con 0x561336840570 2024-04-01T01:37:40.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.108+0000 7f1ef17fa640 1 -- 172.21.15.70:0/1059007322 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee409a3b0 con 0x561336840570 2024-04-01T01:37:40.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.109+0000 7f1f04787740 1 -- 172.21.15.70:0/1059007322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 msgr2=0x56133684c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.109+0000 7f1f04787740 1 --2- 172.21.15.70:0/1059007322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 0x56133684c3a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1ee40d6260 tx=0x7f1ee4005f30 comp rx=0 tx=0).stop 2024-04-01T01:37:40.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.109+0000 7f1f04787740 1 -- 172.21.15.70:0/1059007322 shutdown_connections 2024-04-01T01:37:40.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.109+0000 7f1f04787740 1 --2- 172.21.15.70:0/1059007322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5613368444b0 0x561336844930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.109+0000 7f1f04787740 1 --2- 172.21.15.70:0/1059007322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561336840570 0x56133684c3a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.109+0000 7f1f04787740 1 --2- 172.21.15.70:0/1059007322 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683b400 0x5613368398f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.109+0000 7f1f04787740 1 -- 172.21.15.70:0/1059007322 >> 172.21.15.70:0/1059007322 conn(0x561336835f80 msgr2=0x5613368391e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:40.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.111+0000 7f1f04787740 1 -- 172.21.15.70:0/1059007322 shutdown_connections 2024-04-01T01:37:40.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.111+0000 7f1f04787740 1 -- 172.21.15.70:0/1059007322 wait complete. 2024-04-01T01:37:40.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.111+0000 7f1f04787740 1 -- start start 2024-04-01T01:37:40.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.111+0000 7f1f04787740 1 Processor -- start 2024-04-01T01:37:40.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.112+0000 7f1f04787740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683b400 0x561336849ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.112+0000 7f1f04787740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561336840570 0x56133684a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.112+0000 7f1f04787740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613368444b0 0x56133684b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.112+0000 7f1f04787740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1c2f70 con 0x561336840570 2024-04-01T01:37:40.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.112+0000 7f1f04787740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1c28a0 con 0x56133683b400 2024-04-01T01:37:40.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.112+0000 7f1f04787740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1bfae0 con 0x5613368444b0 2024-04-01T01:37:40.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.112+0000 7f1f04787740 10 client.0 _ll_register_callbacks cb 0x7f1eec009c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:40.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.112+0000 7f1efbfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613368444b0 0x56133684b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.112+0000 7f1efbfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613368444b0 0x56133684b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52870/0 (socket says 172.21.15.70:52870) 2024-04-01T01:37:40.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.112+0000 7f1efbfff640 1 -- 172.21.15.70:0/2556914463 learned_addr learned my addr 172.21.15.70:0/2556914463 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:40.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.112+0000 7f1efb7fe640 1 --2- 172.21.15.70:0/2556914463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683b400 0x561336849ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.112+0000 7f1ef2ffd640 1 --2- 172.21.15.70:0/2556914463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561336840570 0x56133684a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.113+0000 7f1efbfff640 1 -- 172.21.15.70:0/2556914463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683b400 msgr2=0x561336849ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:40.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.113+0000 7f1efbfff640 1 --2- 172.21.15.70:0/2556914463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683b400 0x561336849ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.113+0000 7f1efbfff640 1 -- 172.21.15.70:0/2556914463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561336840570 msgr2=0x56133684a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:40.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.113+0000 7f1efbfff640 1 --2- 172.21.15.70:0/2556914463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561336840570 0x56133684a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.113+0000 7f1efbfff640 1 -- 172.21.15.70:0/2556914463 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56133684b610 con 0x5613368444b0 2024-04-01T01:37:40.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.113+0000 7f1efb7fe640 1 --2- 172.21.15.70:0/2556914463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683b400 0x561336849ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:40.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.113+0000 7f1efbfff640 1 --2- 172.21.15.70:0/2556914463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613368444b0 0x56133684b020 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4032b20 tx=0x7f1ef402c0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b56923f83dcf569b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:40.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.114+0000 7f1eeaffd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:37:40.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.114+0000 7f1eeaffd640 1 -- 172.21.15.70:0/2556914463 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4053030 con 0x5613368444b0 2024-04-01T01:37:40.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.114+0000 7f1eeaffd640 1 -- 172.21.15.70:0/2556914463 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1ef4032690 con 0x5613368444b0 2024-04-01T01:37:40.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.115+0000 7f1f04787740 1 -- 172.21.15.70:0/2556914463 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ee4009de0 con 0x5613368444b0 2024-04-01T01:37:40.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.115+0000 7f1eeaffd640 1 -- 172.21.15.70:0/2556914463 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1ef4039040 con 0x5613368444b0 2024-04-01T01:37:40.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.116+0000 7f1f04787740 10 client.26632 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:40.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.116+0000 7f1f04787740 1 -- 172.21.15.70:0/2556914463 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5613368b7130 con 0x5613368444b0 2024-04-01T01:37:40.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.117+0000 7f1eeaffd640 1 -- 172.21.15.70:0/2556914463 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1ee4009de0 con 0x5613368444b0 2024-04-01T01:37:40.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.118+0000 7f1eeaffd640 1 -- 172.21.15.70:0/2556914463 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1ef4040c50 con 0x5613368444b0 2024-04-01T01:37:40.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.118+0000 7f1f04787740 10 client.26632 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:40.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.118+0000 7f1f04787740 10 client.26632 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:40.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.118+0000 7f1f04787740 20 client.26632 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:40.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.118+0000 7f1f04787740 1 --2- 172.21.15.70:0/2556914463 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5613368b7ad0 0x5613368b9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.118+0000 7f1f04787740 4 client.26632 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:37:40.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.118+0000 7f1f04787740 1 -- 172.21.15.70:0/2556914463 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5613367a6730 con 0x5613368b7ad0 2024-04-01T01:37:40.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.119+0000 7f1efb7fe640 1 --2- 172.21.15.70:0/2556914463 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5613368b7ad0 0x5613368b9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.119+0000 7f1efb7fe640 1 --2- 172.21.15.70:0/2556914463 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5613368b7ad0 0x5613368b9ef0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b1a0dfc2ab1a489 server_cookie=e6f75fa35975cd2a in_seq=0 out_seq=0 2024-04-01T01:37:40.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.119+0000 7f1eeaffd640 10 client.26632 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:40.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.121+0000 7f1eeaffd640 1 -- 172.21.15.70:0/2556914463 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5613367a6730 con 0x5613368b7ad0 2024-04-01T01:37:40.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.121+0000 7f1eeaffd640 10 client.26632 handle_command_reply: tid=0 2024-04-01T01:37:40.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.121+0000 7f1f04787740 1 client.26632 shutdown 2024-04-01T01:37:40.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.122+0000 7f1f04787740 1 -- 172.21.15.70:0/2556914463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613368444b0 msgr2=0x56133684b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.122+0000 7f1f04787740 1 --2- 172.21.15.70:0/2556914463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613368444b0 0x56133684b020 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4032b20 tx=0x7f1ef402c0e0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.122+0000 7f1f04787740 1 -- 172.21.15.70:0/2556914463 shutdown_connections 2024-04-01T01:37:40.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.122+0000 7f1f04787740 1 --2- 172.21.15.70:0/2556914463 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5613368b7ad0 0x5613368b9ef0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.122+0000 7f1f04787740 1 --2- 172.21.15.70:0/2556914463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613368444b0 0x56133684b020 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.122+0000 7f1f04787740 1 --2- 172.21.15.70:0/2556914463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561336840570 0x56133684a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.122+0000 7f1f04787740 1 --2- 172.21.15.70:0/2556914463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56133683b400 0x561336849ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.122+0000 7f1f04787740 1 -- 172.21.15.70:0/2556914463 >> 172.21.15.70:0/2556914463 conn(0x561336835f80 msgr2=0x5613368451c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:40.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.122+0000 7f1eeaffd640 0 client.26632 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:40.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.123+0000 7f1f04787740 1 -- 172.21.15.70:0/2556914463 shutdown_connections 2024-04-01T01:37:40.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.123+0000 7f1f04787740 1 -- 172.21.15.70:0/2556914463 wait complete. 2024-04-01T01:37:40.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.123+0000 7f1f04787740 20 client.26632 trim_cache size 0 max 16384 2024-04-01T01:37:40.123 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:37:40.123 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:37:40.123 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "285820d8-38bc-428a-92b0-dd9eda64a516", 2024-04-01T01:37:40.123 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:37:40.124 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:37:40.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.124+0000 7f1f01ac9640 1 -- 172.21.15.70:0/710428272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1edc0a2670 msgr2=0x7f1edc0a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.124+0000 7f1f01ac9640 1 --2- 172.21.15.70:0/710428272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1edc0a2670 0x7f1edc0a4b30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1ef402c620 tx=0x7f1ef402e040 comp rx=0 tx=0).stop 2024-04-01T01:37:40.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.124+0000 7f1f01ac9640 1 -- 172.21.15.70:0/710428272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1efc109110 msgr2=0x7f1efc1197b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.124+0000 7f1f01ac9640 1 --2- 172.21.15.70:0/710428272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1efc109110 0x7f1efc1197b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1ee400d7a0 tx=0x7f1ee400dc60 comp rx=0 tx=0).stop 2024-04-01T01:37:40.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.124+0000 7f1f01ac9640 1 -- 172.21.15.70:0/710428272 shutdown_connections 2024-04-01T01:37:40.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.124+0000 7f1f01ac9640 1 --2- 172.21.15.70:0/710428272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1efc109860 0x7f1efc1207d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.124+0000 7f1f01ac9640 1 --2- 172.21.15.70:0/710428272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1edc0a2670 0x7f1edc0a4b30 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.124+0000 7f1f01ac9640 1 --2- 172.21.15.70:0/710428272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1efc109110 0x7f1efc1197b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.124+0000 7f1f01ac9640 1 --2- 172.21.15.70:0/710428272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1efc108750 0x7f1efc119270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.124+0000 7f1f01ac9640 1 -- 172.21.15.70:0/710428272 >> 172.21.15.70:0/710428272 conn(0x7f1efc06e5b0 msgr2=0x7f1efc06eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:40.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.124+0000 7f1f01ac9640 1 -- 172.21.15.70:0/710428272 shutdown_connections 2024-04-01T01:37:40.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.125+0000 7f1f01ac9640 1 -- 172.21.15.70:0/710428272 wait complete. 2024-04-01T01:37:40.145 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:37:40.145 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:37:40.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.345+0000 7f8a9e638640 1 -- 172.21.15.70:0/2150299653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a981097e0 msgr2=0x7f8a9810a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.345+0000 7f8a9e638640 1 --2- 172.21.15.70:0/2150299653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a981097e0 0x7f8a9810a060 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009900 tx=0x7f8a9402bc50 comp rx=0 tx=0).stop 2024-04-01T01:37:40.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.345+0000 7f8a9e638640 1 -- 172.21.15.70:0/2150299653 shutdown_connections 2024-04-01T01:37:40.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.345+0000 7f8a9e638640 1 --2- 172.21.15.70:0/2150299653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a981097e0 0x7f8a9810a060 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.345+0000 7f8a9e638640 1 --2- 172.21.15.70:0/2150299653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a98108e20 0x7f8a981092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.345+0000 7f8a9e638640 1 --2- 172.21.15.70:0/2150299653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a98072b60 0x7f8a98072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.345+0000 7f8a9e638640 1 -- 172.21.15.70:0/2150299653 >> 172.21.15.70:0/2150299653 conn(0x7f8a9806e630 msgr2=0x7f8a98070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:40.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.346+0000 7f8a9e638640 1 -- 172.21.15.70:0/2150299653 shutdown_connections 2024-04-01T01:37:40.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.347+0000 7f8a9e638640 1 -- 172.21.15.70:0/2150299653 wait complete. 2024-04-01T01:37:40.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.347+0000 7f8a9e638640 1 Processor -- start 2024-04-01T01:37:40.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.348+0000 7f8a9e638640 1 -- start start 2024-04-01T01:37:40.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.348+0000 7f8a9e638640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a98072b60 0x7f8a981a6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.348+0000 7f8a9e638640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a98108e20 0x7f8a981a6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.348+0000 7f8a9e638640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a981097e0 0x7f8a981adc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.348+0000 7f8a9e638640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a98117be0 con 0x7f8a98108e20 2024-04-01T01:37:40.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.348+0000 7f8a9e638640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8a98117a60 con 0x7f8a981097e0 2024-04-01T01:37:40.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.348+0000 7f8a9e638640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8a98117d60 con 0x7f8a98072b60 2024-04-01T01:37:40.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.348+0000 7f8a9d636640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a98072b60 0x7f8a981a6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.348+0000 7f8a9d636640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a98072b60 0x7f8a981a6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52898/0 (socket says 172.21.15.70:52898) 2024-04-01T01:37:40.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.348+0000 7f8a9d636640 1 -- 172.21.15.70:0/3885387629 learned_addr learned my addr 172.21.15.70:0/3885387629 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:40.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.348+0000 7f8a9de37640 1 --2- 172.21.15.70:0/3885387629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a981097e0 0x7f8a981adc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.348+0000 7f8a9ce35640 1 --2- 172.21.15.70:0/3885387629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a98108e20 0x7f8a981a6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.349+0000 7f8a9ce35640 1 -- 172.21.15.70:0/3885387629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a98072b60 msgr2=0x7f8a981a6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.349+0000 7f8a9ce35640 1 --2- 172.21.15.70:0/3885387629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a98072b60 0x7f8a981a6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.349+0000 7f8a9ce35640 1 -- 172.21.15.70:0/3885387629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a981097e0 msgr2=0x7f8a981adc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.349+0000 7f8a9ce35640 1 --2- 172.21.15.70:0/3885387629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a981097e0 0x7f8a981adc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.349+0000 7f8a9ce35640 1 -- 172.21.15.70:0/3885387629 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a981ae2b0 con 0x7f8a98108e20 2024-04-01T01:37:40.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.349+0000 7f8a9d636640 1 --2- 172.21.15.70:0/3885387629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a98072b60 0x7f8a981a6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:40.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.349+0000 7f8a9de37640 1 --2- 172.21.15.70:0/3885387629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a981097e0 0x7f8a981adc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:40.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.350+0000 7f8a9ce35640 1 --2- 172.21.15.70:0/3885387629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a98108e20 0x7f8a981a6bd0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000bd50 tx=0x7f8a9000be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:40.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.350+0000 7f8a8e7fc640 1 -- 172.21.15.70:0/3885387629 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a90019070 con 0x7f8a98108e20 2024-04-01T01:37:40.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.351+0000 7f8a8e7fc640 1 -- 172.21.15.70:0/3885387629 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a900148f0 con 0x7f8a98108e20 2024-04-01T01:37:40.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.351+0000 7f8a8e7fc640 1 -- 172.21.15.70:0/3885387629 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a90012910 con 0x7f8a98108e20 2024-04-01T01:37:40.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.351+0000 7f8a9e638640 1 -- 172.21.15.70:0/3885387629 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a981ae5a0 con 0x7f8a98108e20 2024-04-01T01:37:40.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.351+0000 7f8a9e638640 1 -- 172.21.15.70:0/3885387629 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a981aeb20 con 0x7f8a98108e20 2024-04-01T01:37:40.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.354+0000 7f8aa247e740 1 Processor -- start 2024-04-01T01:37:40.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.354+0000 7f8aa247e740 1 -- start start 2024-04-01T01:37:40.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.355+0000 7f8aa247e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514053490 0x5655140538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.355+0000 7f8aa247e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 0x565514058980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.355+0000 7f8aa247e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514058ff0 0x56551405b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.355+0000 7f8aa247e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x565513fb8180 con 0x565514058ff0 2024-04-01T01:37:40.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.355+0000 7f8aa247e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x565513f60b70 con 0x565514053490 2024-04-01T01:37:40.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.355+0000 7f8aa247e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x565513fb4220 con 0x565514053e80 2024-04-01T01:37:40.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.355+0000 7f8a9ce35640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514058ff0 0x56551405b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.355+0000 7f8a9ce35640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514058ff0 0x56551405b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47666/0 (socket says 172.21.15.70:47666) 2024-04-01T01:37:40.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.355+0000 7f8a9ce35640 1 -- 172.21.15.70:0/2202695588 learned_addr learned my addr 172.21.15.70:0/2202695588 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:40.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.356+0000 7f8a9d636640 1 --2- 172.21.15.70:0/2202695588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 0x565514058980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.357+0000 7f8a9de37640 1 --2- 172.21.15.70:0/2202695588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514053490 0x5655140538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.358+0000 7f8a9ce35640 1 -- 172.21.15.70:0/2202695588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 msgr2=0x565514058980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.358+0000 7f8a9ce35640 1 --2- 172.21.15.70:0/2202695588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 0x565514058980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.358+0000 7f8a9ce35640 1 -- 172.21.15.70:0/2202695588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514053490 msgr2=0x5655140538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.358+0000 7f8a9ce35640 1 --2- 172.21.15.70:0/2202695588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514053490 0x5655140538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.358+0000 7f8a9ce35640 1 -- 172.21.15.70:0/2202695588 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565513f747c0 con 0x565514058ff0 2024-04-01T01:37:40.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.360+0000 7f8a8e7fc640 1 -- 172.21.15.70:0/3885387629 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f8a9001c070 con 0x7f8a98108e20 2024-04-01T01:37:40.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.360+0000 7f8a9ce35640 1 --2- 172.21.15.70:0/2202695588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514058ff0 0x56551405b480 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009910 tx=0x7f8a90014e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de23f2229b5aa40 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:40.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.360+0000 7f8a8e7fc640 1 --2- 172.21.15.70:0/3885387629 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8a840a2570 0x7f8a840a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.361+0000 7f8a8e7fc640 1 -- 172.21.15.70:0/3885387629 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8a900cf480 con 0x7f8a98108e20 2024-04-01T01:37:40.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.361+0000 7f8a9de37640 1 --2- 172.21.15.70:0/3885387629 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8a840a2570 0x7f8a840a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.363+0000 7f8a837fe640 1 -- 172.21.15.70:0/2202695588 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a900d4030 con 0x565514058ff0 2024-04-01T01:37:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.363+0000 7f8a837fe640 1 -- 172.21.15.70:0/2202695588 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a9009bd60 con 0x565514058ff0 2024-04-01T01:37:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.363+0000 7f8a837fe640 1 -- 172.21.15.70:0/2202695588 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a900a4630 con 0x565514058ff0 2024-04-01T01:37:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.364+0000 7f8aa247e740 1 -- 172.21.15.70:0/2202695588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514058ff0 msgr2=0x56551405b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.364+0000 7f8aa247e740 1 --2- 172.21.15.70:0/2202695588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514058ff0 0x56551405b480 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009910 tx=0x7f8a90014e30 comp rx=0 tx=0).stop 2024-04-01T01:37:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.366+0000 7f8aa247e740 1 -- 172.21.15.70:0/2202695588 shutdown_connections 2024-04-01T01:37:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.366+0000 7f8aa247e740 1 --2- 172.21.15.70:0/2202695588 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514058ff0 0x56551405b480 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.366+0000 7f8aa247e740 1 --2- 172.21.15.70:0/2202695588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 0x565514058980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.366+0000 7f8aa247e740 1 --2- 172.21.15.70:0/2202695588 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514053490 0x5655140538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.366+0000 7f8aa247e740 1 -- 172.21.15.70:0/2202695588 >> 172.21.15.70:0/2202695588 conn(0x56551404ea00 msgr2=0x56551404ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.366+0000 7f8aa247e740 1 -- 172.21.15.70:0/2202695588 shutdown_connections 2024-04-01T01:37:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.366+0000 7f8aa247e740 1 -- 172.21.15.70:0/2202695588 wait complete. 2024-04-01T01:37:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.367+0000 7f8aa247e740 1 -- start start 2024-04-01T01:37:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.367+0000 7f8aa247e740 1 Processor -- start 2024-04-01T01:37:40.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.367+0000 7f8a9de37640 1 --2- 172.21.15.70:0/3885387629 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8a840a2570 0x7f8a840a4a30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8a94006010 tx=0x7f8a9402c800 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:40.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.367+0000 7f8aa247e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514053490 0x56551405d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.367+0000 7f8aa247e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053e80 0x56551405dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.367+0000 7f8aa247e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514058ff0 0x565514064f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.367+0000 7f8aa247e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56551405bfa0 con 0x565514053e80 2024-04-01T01:37:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.367+0000 7f8aa247e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x565513f564a0 con 0x565514053490 2024-04-01T01:37:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.367+0000 7f8aa247e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56551405c120 con 0x565514058ff0 2024-04-01T01:37:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.367+0000 7f8aa247e740 10 client.0 _ll_register_callbacks cb 0x565513f59d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.368+0000 7f8a9d636640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514053490 0x56551405d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.368+0000 7f8a9ce35640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053e80 0x56551405dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.368+0000 7f8a9d636640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514053490 0x56551405d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53822/0 (socket says 172.21.15.70:53822) 2024-04-01T01:37:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.368+0000 7f8a9d636640 1 -- 172.21.15.70:0/432722682 learned_addr learned my addr 172.21.15.70:0/432722682 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.368+0000 7f8a9de37640 1 --2- 172.21.15.70:0/432722682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514058ff0 0x565514064f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.368+0000 7f8a9d636640 1 -- 172.21.15.70:0/432722682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514058ff0 msgr2=0x565514064f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.368+0000 7f8a9d636640 1 --2- 172.21.15.70:0/432722682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514058ff0 0x565514064f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.368+0000 7f8a9d636640 1 -- 172.21.15.70:0/432722682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053e80 msgr2=0x56551405dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.368+0000 7f8a9d636640 1 --2- 172.21.15.70:0/432722682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053e80 0x56551405dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.368+0000 7f8a9d636640 1 -- 172.21.15.70:0/432722682 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565513d967d0 con 0x565514053490 2024-04-01T01:37:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.369+0000 7f8a9d636640 1 --2- 172.21.15.70:0/432722682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514053490 0x56551405d9a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8a88011830 tx=0x7f8a88011cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce74088b0decfd70 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:40.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.369+0000 7f8a80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:40.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.369+0000 7f8a80ff9640 1 -- 172.21.15.70:0/432722682 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a88010ba0 con 0x565514053490 2024-04-01T01:37:40.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.369+0000 7f8a80ff9640 1 -- 172.21.15.70:0/432722682 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8a88010d40 con 0x565514053490 2024-04-01T01:37:40.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.369+0000 7f8aa247e740 1 -- 172.21.15.70:0/432722682 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a9009bd60 con 0x565514053490 2024-04-01T01:37:40.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.370+0000 7f8a80ff9640 1 -- 172.21.15.70:0/432722682 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8a88025050 con 0x565514053490 2024-04-01T01:37:40.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.370+0000 7f8aa247e740 10 client.26744 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:40.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.370+0000 7f8aa247e740 1 -- 172.21.15.70:0/432722682 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x565513ccc3f0 con 0x565514053490 2024-04-01T01:37:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.371+0000 7f8a80ff9640 1 -- 172.21.15.70:0/432722682 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a9009bd60 con 0x565514053490 2024-04-01T01:37:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.371+0000 7f8a80ff9640 1 -- 172.21.15.70:0/432722682 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8a8800f4d0 con 0x565514053490 2024-04-01T01:37:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.373+0000 7f8aa247e740 10 client.26744 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.373+0000 7f8aa247e740 10 client.26744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.373+0000 7f8aa247e740 20 client.26744 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.373+0000 7f8aa247e740 1 --2- 172.21.15.70:0/432722682 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56551407a850 0x56551407ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.373+0000 7f8aa247e740 4 client.26744 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:40.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.373+0000 7f8aa247e740 1 -- 172.21.15.70:0/432722682 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565513fc56c0 con 0x56551407a850 2024-04-01T01:37:40.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.374+0000 7f8a9ce35640 1 --2- 172.21.15.70:0/432722682 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56551407a850 0x56551407ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.377+0000 7f8a9ce35640 1 --2- 172.21.15.70:0/432722682 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56551407a850 0x56551407ccb0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ce75ceb5f6e0fef server_cookie=7bedc11f7c8eb8f8 in_seq=0 out_seq=0 2024-04-01T01:37:40.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.378+0000 7f8a80ff9640 10 client.26744 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:40.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.378+0000 7f8a80ff9640 1 -- 172.21.15.70:0/432722682 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x565513fc56c0 con 0x56551407a850 2024-04-01T01:37:40.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.378+0000 7f8a80ff9640 10 client.26744 handle_command_reply: tid=0 2024-04-01T01:37:40.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.379+0000 7f8aa247e740 1 client.26744 shutdown 2024-04-01T01:37:40.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.380+0000 7f8aa247e740 1 -- 172.21.15.70:0/432722682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514053490 msgr2=0x56551405d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.380+0000 7f8aa247e740 1 --2- 172.21.15.70:0/432722682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514053490 0x56551405d9a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8a88011830 tx=0x7f8a88011cf0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.381+0000 7f8aa247e740 1 -- 172.21.15.70:0/432722682 shutdown_connections 2024-04-01T01:37:40.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.381+0000 7f8aa247e740 1 --2- 172.21.15.70:0/432722682 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56551407a850 0x56551407ccb0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.381+0000 7f8aa247e740 1 --2- 172.21.15.70:0/432722682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514058ff0 0x565514064f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.381+0000 7f8aa247e740 1 --2- 172.21.15.70:0/432722682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053e80 0x56551405dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.381+0000 7f8aa247e740 1 --2- 172.21.15.70:0/432722682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514053490 0x56551405d9a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.381+0000 7f8aa247e740 1 -- 172.21.15.70:0/432722682 >> 172.21.15.70:0/432722682 conn(0x56551404ea00 msgr2=0x56551404dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:40.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.382+0000 7f8a80ff9640 0 client.26744 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:40.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.382+0000 7f8aa247e740 1 -- 172.21.15.70:0/432722682 shutdown_connections 2024-04-01T01:37:40.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.384+0000 7f8aa247e740 1 -- 172.21.15.70:0/432722682 wait complete. 2024-04-01T01:37:40.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.384+0000 7f8aa247e740 20 client.26744 trim_cache size 0 max 16384 2024-04-01T01:37:40.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.420+0000 7f8aa247e740 1 Processor -- start 2024-04-01T01:37:40.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.420+0000 7f8aa247e740 1 -- start start 2024-04-01T01:37:40.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.421+0000 7f8aa247e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053490 0x565514052370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.421+0000 7f8aa247e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 0x565514064ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.421+0000 7f8aa247e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514058ff0 0x56551405cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.421+0000 7f8aa247e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a981b0930 con 0x565514053490 2024-04-01T01:37:40.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.421+0000 7f8aa247e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8a981b2470 con 0x565514058ff0 2024-04-01T01:37:40.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.421+0000 7f8aa247e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8a981b1000 con 0x565514053e80 2024-04-01T01:37:40.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.421+0000 7f8a9d636640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053490 0x565514052370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.421+0000 7f8a9d636640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053490 0x565514052370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47692/0 (socket says 172.21.15.70:47692) 2024-04-01T01:37:40.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.421+0000 7f8a9d636640 1 -- 172.21.15.70:0/1139886123 learned_addr learned my addr 172.21.15.70:0/1139886123 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:40.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.421+0000 7f8a9de37640 1 --2- 172.21.15.70:0/1139886123 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514058ff0 0x56551405cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.422+0000 7f8a9ce35640 1 --2- 172.21.15.70:0/1139886123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 0x565514064ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.422+0000 7f8a9de37640 1 -- 172.21.15.70:0/1139886123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 msgr2=0x565514064ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:40.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.422+0000 7f8a9de37640 1 --2- 172.21.15.70:0/1139886123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 0x565514064ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.422+0000 7f8a9de37640 1 -- 172.21.15.70:0/1139886123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053490 msgr2=0x565514052370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:40.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.422+0000 7f8a9de37640 1 --2- 172.21.15.70:0/1139886123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053490 0x565514052370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.422+0000 7f8a9de37640 1 -- 172.21.15.70:0/1139886123 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56551404a280 con 0x565514058ff0 2024-04-01T01:37:40.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.422+0000 7f8a9d636640 1 --2- 172.21.15.70:0/1139886123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053490 0x565514052370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:40.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.422+0000 7f8a9ce35640 1 --2- 172.21.15.70:0/1139886123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 0x565514064ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:37:40.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.422+0000 7f8a9de37640 1 --2- 172.21.15.70:0/1139886123 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514058ff0 0x56551405cfc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8a94036040 tx=0x7f8a9403f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9096833750a9b466 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:40.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.422+0000 7f8a82ffd640 1 -- 172.21.15.70:0/1139886123 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a94051070 con 0x565514058ff0 2024-04-01T01:37:40.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.422+0000 7f8a82ffd640 1 -- 172.21.15.70:0/1139886123 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a940046c0 con 0x565514058ff0 2024-04-01T01:37:40.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.422+0000 7f8a82ffd640 1 -- 172.21.15.70:0/1139886123 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a94003850 con 0x565514058ff0 2024-04-01T01:37:40.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.424+0000 7f8aa247e740 1 -- 172.21.15.70:0/1139886123 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514058ff0 msgr2=0x56551405cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.424+0000 7f8aa247e740 1 --2- 172.21.15.70:0/1139886123 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514058ff0 0x56551405cfc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8a94036040 tx=0x7f8a9403f000 comp rx=0 tx=0).stop 2024-04-01T01:37:40.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.424+0000 7f8aa247e740 1 -- 172.21.15.70:0/1139886123 shutdown_connections 2024-04-01T01:37:40.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.424+0000 7f8aa247e740 1 --2- 172.21.15.70:0/1139886123 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514058ff0 0x56551405cfc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.424+0000 7f8aa247e740 1 --2- 172.21.15.70:0/1139886123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 0x565514064ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.424+0000 7f8aa247e740 1 --2- 172.21.15.70:0/1139886123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053490 0x565514052370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.424+0000 7f8aa247e740 1 -- 172.21.15.70:0/1139886123 >> 172.21.15.70:0/1139886123 conn(0x56551404ea00 msgr2=0x565514051c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:40.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.424+0000 7f8aa247e740 1 -- 172.21.15.70:0/1139886123 shutdown_connections 2024-04-01T01:37:40.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.425+0000 7f8aa247e740 1 -- 172.21.15.70:0/1139886123 wait complete. 2024-04-01T01:37:40.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.425+0000 7f8aa247e740 1 -- start start 2024-04-01T01:37:40.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.425+0000 7f8aa247e740 1 Processor -- start 2024-04-01T01:37:40.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.426+0000 7f8aa247e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053490 0x565514061ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.426+0000 7f8aa247e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 0x565514062530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.426+0000 7f8aa247e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514058ff0 0x565514063540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.427+0000 7f8a9ce35640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 0x565514062530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.427+0000 7f8a9ce35640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 0x565514062530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52920/0 (socket says 172.21.15.70:52920) 2024-04-01T01:37:40.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.427+0000 7f8a9ce35640 1 -- 172.21.15.70:0/3731540644 learned_addr learned my addr 172.21.15.70:0/3731540644 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:40.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.427+0000 7f8a9de37640 1 --2- 172.21.15.70:0/3731540644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514058ff0 0x565514063540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.427+0000 7f8aa247e740 1 -- 172.21.15.70:0/3731540644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a981b1da0 con 0x565514053490 2024-04-01T01:37:40.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.427+0000 7f8aa247e740 1 -- 172.21.15.70:0/3731540644 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8a981b16d0 con 0x565514058ff0 2024-04-01T01:37:40.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.427+0000 7f8aa247e740 1 -- 172.21.15.70:0/3731540644 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8a981a5660 con 0x565514053e80 2024-04-01T01:37:40.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.427+0000 7f8aa247e740 10 client.0 _ll_register_callbacks cb 0x7f8a8800bd50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:40.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.427+0000 7f8a9d636640 1 --2- 172.21.15.70:0/3731540644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053490 0x565514061ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.428+0000 7f8a9de37640 1 -- 172.21.15.70:0/3731540644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 msgr2=0x565514062530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:40.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.428+0000 7f8a9de37640 1 --2- 172.21.15.70:0/3731540644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 0x565514062530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.428+0000 7f8a9de37640 1 -- 172.21.15.70:0/3731540644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053490 msgr2=0x565514061ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:40.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.428+0000 7f8a9de37640 1 --2- 172.21.15.70:0/3731540644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053490 0x565514061ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.428+0000 7f8a9de37640 1 -- 172.21.15.70:0/3731540644 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565513c8c0f0 con 0x565514058ff0 2024-04-01T01:37:40.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.428+0000 7f8a9de37640 1 --2- 172.21.15.70:0/3731540644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514058ff0 0x565514063540 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8a9402c6c0 tx=0x7f8a94004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88616ef947f5d669 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:40.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.428+0000 7f8a80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:40.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.428+0000 7f8a80ff9640 1 -- 172.21.15.70:0/3731540644 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a94051070 con 0x565514058ff0 2024-04-01T01:37:40.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.429+0000 7f8a80ff9640 1 -- 172.21.15.70:0/3731540644 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8a94003a00 con 0x565514058ff0 2024-04-01T01:37:40.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.429+0000 7f8aa247e740 1 -- 172.21.15.70:0/3731540644 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a940046c0 con 0x565514058ff0 2024-04-01T01:37:40.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.429+0000 7f8a80ff9640 1 -- 172.21.15.70:0/3731540644 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8a94003c40 con 0x565514058ff0 2024-04-01T01:37:40.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.429+0000 7f8aa247e740 10 client.26750 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:40.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.429+0000 7f8aa247e740 1 -- 172.21.15.70:0/3731540644 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x565514064be0 con 0x565514058ff0 2024-04-01T01:37:40.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.430+0000 7f8a80ff9640 1 -- 172.21.15.70:0/3731540644 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8a940046c0 con 0x565514058ff0 2024-04-01T01:37:40.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.430+0000 7f8a80ff9640 1 -- 172.21.15.70:0/3731540644 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8a94056450 con 0x565514058ff0 2024-04-01T01:37:40.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.430+0000 7f8aa247e740 10 client.26750 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:40.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.430+0000 7f8aa247e740 10 client.26750 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:40.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.430+0000 7f8aa247e740 20 client.26750 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:40.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.430+0000 7f8aa247e740 1 --2- 172.21.15.70:0/3731540644 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5655140e0c00 0x5655140e3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:40.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.430+0000 7f8aa247e740 4 client.26750 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:37:40.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.430+0000 7f8aa247e740 1 -- 172.21.15.70:0/3731540644 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x565513fbfa10 con 0x5655140e0c00 2024-04-01T01:37:40.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.431+0000 7f8a9d636640 1 --2- 172.21.15.70:0/3731540644 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5655140e0c00 0x5655140e3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:40.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.432+0000 7f8a9d636640 1 --2- 172.21.15.70:0/3731540644 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5655140e0c00 0x5655140e3020 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba85c0aadc43fef2 server_cookie=f73db3e88ae9ca73 in_seq=0 out_seq=0 2024-04-01T01:37:40.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.432+0000 7f8a80ff9640 10 client.26750 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:40.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.432+0000 7f8a80ff9640 1 -- 172.21.15.70:0/3731540644 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x565513fbfa10 con 0x5655140e0c00 2024-04-01T01:37:40.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.432+0000 7f8a80ff9640 10 client.26750 handle_command_reply: tid=0 2024-04-01T01:37:40.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.432+0000 7f8aa247e740 1 client.26750 shutdown 2024-04-01T01:37:40.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.433+0000 7f8aa247e740 1 -- 172.21.15.70:0/3731540644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514058ff0 msgr2=0x565514063540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.433+0000 7f8aa247e740 1 --2- 172.21.15.70:0/3731540644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514058ff0 0x565514063540 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8a9402c6c0 tx=0x7f8a94004c80 comp rx=0 tx=0).stop 2024-04-01T01:37:40.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.433+0000 7f8aa247e740 1 -- 172.21.15.70:0/3731540644 shutdown_connections 2024-04-01T01:37:40.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.433+0000 7f8aa247e740 1 --2- 172.21.15.70:0/3731540644 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5655140e0c00 0x5655140e3020 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.433+0000 7f8aa247e740 1 --2- 172.21.15.70:0/3731540644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x565514058ff0 0x565514063540 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.433+0000 7f8aa247e740 1 --2- 172.21.15.70:0/3731540644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565514053e80 0x565514062530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.433+0000 7f8aa247e740 1 --2- 172.21.15.70:0/3731540644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565514053490 0x565514061ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.433+0000 7f8aa247e740 1 -- 172.21.15.70:0/3731540644 >> 172.21.15.70:0/3731540644 conn(0x56551404ea00 msgr2=0x5655140524f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:40.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.433+0000 7f8a80ff9640 0 client.26750 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:40.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.434+0000 7f8aa247e740 1 -- 172.21.15.70:0/3731540644 shutdown_connections 2024-04-01T01:37:40.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.434+0000 7f8aa247e740 1 -- 172.21.15.70:0/3731540644 wait complete. 2024-04-01T01:37:40.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.435+0000 7f8aa247e740 20 client.26750 trim_cache size 0 max 16384 2024-04-01T01:37:40.436 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:37:40.437 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:37:40.437 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:37:40.437 INFO:teuthology.orchestra.run.smithi070.stdout: "285820d8-38bc-428a-92b0-dd9eda64a516": { 2024-04-01T01:37:40.437 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:37:40.437 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "285820d8-38bc-428a-92b0-dd9eda64a516", 2024-04-01T01:37:40.437 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:37:40.437 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:37:40.437 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:37:40.437 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:37:40.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.438+0000 7f8a9e638640 1 -- 172.21.15.70:0/3885387629 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8a840a2570 msgr2=0x7f8a840a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.438+0000 7f8a9e638640 1 --2- 172.21.15.70:0/3885387629 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8a840a2570 0x7f8a840a4a30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8a94006010 tx=0x7f8a9402c800 comp rx=0 tx=0).stop 2024-04-01T01:37:40.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.438+0000 7f8a9e638640 1 -- 172.21.15.70:0/3885387629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a98108e20 msgr2=0x7f8a981a6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:40.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.438+0000 7f8a9e638640 1 --2- 172.21.15.70:0/3885387629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a98108e20 0x7f8a981a6bd0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000bd50 tx=0x7f8a9000be40 comp rx=0 tx=0).stop 2024-04-01T01:37:40.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.441+0000 7f8a9e638640 1 -- 172.21.15.70:0/3885387629 shutdown_connections 2024-04-01T01:37:40.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.441+0000 7f8a9e638640 1 --2- 172.21.15.70:0/3885387629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a981097e0 0x7f8a981adc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.441+0000 7f8a9e638640 1 --2- 172.21.15.70:0/3885387629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a98108e20 0x7f8a981a6bd0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.441+0000 7f8a9e638640 1 --2- 172.21.15.70:0/3885387629 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8a840a2570 0x7f8a840a4a30 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.441+0000 7f8a9e638640 1 --2- 172.21.15.70:0/3885387629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a98072b60 0x7f8a981a6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:40.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.441+0000 7f8a9e638640 1 -- 172.21.15.70:0/3885387629 >> 172.21.15.70:0/3885387629 conn(0x7f8a9806e630 msgr2=0x7f8a98070d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:40.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.441+0000 7f8a9e638640 1 -- 172.21.15.70:0/3885387629 shutdown_connections 2024-04-01T01:37:40.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:37:40.442+0000 7f8a9e638640 1 -- 172.21.15.70:0/3885387629 wait complete. 2024-04-01T01:37:40.466 INFO:tasks.cephfs.filesystem:scrub status for tag:285820d8-38bc-428a-92b0-dd9eda64a516 - {'path': '/', 'tag': '285820d8-38bc-428a-92b0-dd9eda64a516', 'options': 'recursive,force'} 2024-04-01T01:37:41.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 175.0 s, 980.8 tps, lat 103.164 ms stddev 43.124 2024-04-01T01:37:41.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:41 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:37:41.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:41 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:37:41.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:41 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:37:41.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:41 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:37:41.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:41 smithi189 ceph-mon[35753]: pgmap v484: 129 pgs: 129 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.2 MiB/s wr, 988 op/s 2024-04-01T01:37:41.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:41 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:37:42.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:41 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:37:42.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:41 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:37:42.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:41 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:37:42.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:41 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:37:42.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:41 smithi070 ceph-mon[32179]: pgmap v484: 129 pgs: 129 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.2 MiB/s wr, 988 op/s 2024-04-01T01:37:42.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:41 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:37:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:41 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:37:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:41 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:37:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:41 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:37:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:41 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:37:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:41 smithi081 ceph-mon[35142]: pgmap v484: 129 pgs: 129 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 8.2 MiB/s wr, 988 op/s 2024-04-01T01:37:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:41 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:37:43.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.454+0000 7efe5588f640 1 -- 172.21.15.70:0/785885029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe5010a1a0 msgr2=0x7efe50110a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:43.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.454+0000 7efe5588f640 1 --2- 172.21.15.70:0/785885029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe5010a1a0 0x7efe50110a50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efe44009860 tx=0x7efe4402bb70 comp rx=0 tx=0).stop 2024-04-01T01:37:43.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.454+0000 7efe5588f640 1 -- 172.21.15.70:0/785885029 shutdown_connections 2024-04-01T01:37:43.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.454+0000 7efe5588f640 1 --2- 172.21.15.70:0/785885029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe5010a1a0 0x7efe50110a50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.454+0000 7efe5588f640 1 --2- 172.21.15.70:0/785885029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe50109800 0x7efe50109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.454+0000 7efe5588f640 1 --2- 172.21.15.70:0/785885029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe50108af0 0x7efe50108f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.454+0000 7efe5588f640 1 -- 172.21.15.70:0/785885029 >> 172.21.15.70:0/785885029 conn(0x7efe5006ed90 msgr2=0x7efe500711d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.454+0000 7efe5588f640 1 -- 172.21.15.70:0/785885029 shutdown_connections 2024-04-01T01:37:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.455+0000 7efe5588f640 1 -- 172.21.15.70:0/785885029 wait complete. 2024-04-01T01:37:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.455+0000 7efe5588f640 1 Processor -- start 2024-04-01T01:37:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.456+0000 7efe5588f640 1 -- start start 2024-04-01T01:37:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.456+0000 7efe5588f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe50108af0 0x7efe50117c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.456+0000 7efe5588f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe50109800 0x7efe50118150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.456+0000 7efe5588f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe5010a1a0 0x7efe50119190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.456+0000 7efe5588f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efe50112550 con 0x7efe50108af0 2024-04-01T01:37:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.456+0000 7efe5588f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efe501123d0 con 0x7efe50109800 2024-04-01T01:37:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.456+0000 7efe5588f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efe501126d0 con 0x7efe5010a1a0 2024-04-01T01:37:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.456+0000 7efe4e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe50109800 0x7efe50118150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.456+0000 7efe4e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe50109800 0x7efe50118150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53856/0 (socket says 172.21.15.70:53856) 2024-04-01T01:37:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.456+0000 7efe4e7fc640 1 -- 172.21.15.70:0/3699961519 learned_addr learned my addr 172.21.15.70:0/3699961519 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.456+0000 7efe4f7fe640 1 --2- 172.21.15.70:0/3699961519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe5010a1a0 0x7efe50119190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.457+0000 7efe4e7fc640 1 -- 172.21.15.70:0/3699961519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe5010a1a0 msgr2=0x7efe50119190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.457+0000 7efe4e7fc640 1 --2- 172.21.15.70:0/3699961519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe5010a1a0 0x7efe50119190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.457+0000 7efe4e7fc640 1 -- 172.21.15.70:0/3699961519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe50108af0 msgr2=0x7efe50117c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:43.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.457+0000 7efe4e7fc640 1 --2- 172.21.15.70:0/3699961519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe50108af0 0x7efe50117c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.457+0000 7efe4e7fc640 1 -- 172.21.15.70:0/3699961519 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe50119800 con 0x7efe50109800 2024-04-01T01:37:43.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.457+0000 7efe4e7fc640 1 --2- 172.21.15.70:0/3699961519 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe50109800 0x7efe50118150 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efe4800ed10 tx=0x7efe4800c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:43.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.457+0000 7efe5488d640 1 -- 172.21.15.70:0/3699961519 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4800b570 con 0x7efe50109800 2024-04-01T01:37:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.457+0000 7efe5488d640 1 -- 172.21.15.70:0/3699961519 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe4800bbc0 con 0x7efe50109800 2024-04-01T01:37:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.457+0000 7efe5488d640 1 -- 172.21.15.70:0/3699961519 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe48016be0 con 0x7efe50109800 2024-04-01T01:37:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.457+0000 7efe5588f640 1 -- 172.21.15.70:0/3699961519 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe50119af0 con 0x7efe50109800 2024-04-01T01:37:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.457+0000 7efe5588f640 1 -- 172.21.15.70:0/3699961519 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe50119f90 con 0x7efe50109800 2024-04-01T01:37:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.461+0000 7efe5847b740 1 Processor -- start 2024-04-01T01:37:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.463+0000 7efe5847b740 1 -- start start 2024-04-01T01:37:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.463+0000 7efe5847b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c64a8d0 0x55818c64acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.463+0000 7efe5847b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c64b2c0 0x55818c64fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.463+0000 7efe5847b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c650430 0x55818c6528c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.463+0000 7efe5847b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55818c5ae8f0 con 0x55818c64a8d0 2024-04-01T01:37:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.463+0000 7efe5847b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55818c5578d0 con 0x55818c650430 2024-04-01T01:37:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.463+0000 7efe5847b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55818c5aa990 con 0x55818c64b2c0 2024-04-01T01:37:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.463+0000 7efe5488d640 1 -- 172.21.15.70:0/3699961519 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7efe480163e0 con 0x7efe50109800 2024-04-01T01:37:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.463+0000 7efe4e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c650430 0x55818c6528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.463+0000 7efe4e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c650430 0x55818c6528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53860/0 (socket says 172.21.15.70:53860) 2024-04-01T01:37:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.463+0000 7efe4e7fc640 1 -- 172.21.15.70:0/1810865267 learned_addr learned my addr 172.21.15.70:0/1810865267 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.464+0000 7efe4f7fe640 1 --2- 172.21.15.70:0/1810865267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c64a8d0 0x55818c64acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.464+0000 7efe4effd640 1 --2- 172.21.15.70:0/1810865267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c64b2c0 0x55818c64fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.464+0000 7efe4e7fc640 1 -- 172.21.15.70:0/1810865267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c64b2c0 msgr2=0x55818c64fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.464+0000 7efe4e7fc640 1 --2- 172.21.15.70:0/1810865267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c64b2c0 0x55818c64fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.464+0000 7efe4e7fc640 1 -- 172.21.15.70:0/1810865267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c64a8d0 msgr2=0x55818c64acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.464+0000 7efe4e7fc640 1 --2- 172.21.15.70:0/1810865267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c64a8d0 0x55818c64acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.464+0000 7efe4e7fc640 1 -- 172.21.15.70:0/1810865267 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55818c2c3210 con 0x55818c650430 2024-04-01T01:37:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.464+0000 7efe5488d640 1 --2- 172.21.15.70:0/3699961519 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efe380a2700 0x7efe380a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.464+0000 7efe5488d640 1 -- 172.21.15.70:0/3699961519 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7efe480cef80 con 0x7efe50109800 2024-04-01T01:37:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.465+0000 7efe4f7fe640 1 --2- 172.21.15.70:0/3699961519 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efe380a2700 0x7efe380a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.465+0000 7efe4e7fc640 1 --2- 172.21.15.70:0/1810865267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c650430 0x55818c6528c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7efe480d2f00 tx=0x7efe480d74b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32165ab36ae0c910 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.465+0000 7efe35ffb640 1 -- 172.21.15.70:0/1810865267 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe48097910 con 0x55818c650430 2024-04-01T01:37:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.465+0000 7efe35ffb640 1 -- 172.21.15.70:0/1810865267 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe4809bd70 con 0x55818c650430 2024-04-01T01:37:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.465+0000 7efe35ffb640 1 -- 172.21.15.70:0/1810865267 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4809afd0 con 0x55818c650430 2024-04-01T01:37:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.466+0000 7efe4f7fe640 1 --2- 172.21.15.70:0/3699961519 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efe380a2700 0x7efe380a4bc0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7efe44005d00 tx=0x7efe4402c090 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.466+0000 7efe5847b740 1 -- 172.21.15.70:0/1810865267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c650430 msgr2=0x55818c6528c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.466+0000 7efe5847b740 1 --2- 172.21.15.70:0/1810865267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c650430 0x55818c6528c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7efe480d2f00 tx=0x7efe480d74b0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.467+0000 7efe5847b740 1 -- 172.21.15.70:0/1810865267 shutdown_connections 2024-04-01T01:37:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.467+0000 7efe5847b740 1 --2- 172.21.15.70:0/1810865267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c650430 0x55818c6528c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.467+0000 7efe5847b740 1 --2- 172.21.15.70:0/1810865267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c64b2c0 0x55818c64fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.467+0000 7efe5847b740 1 --2- 172.21.15.70:0/1810865267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c64a8d0 0x55818c64acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.467+0000 7efe5847b740 1 -- 172.21.15.70:0/1810865267 >> 172.21.15.70:0/1810865267 conn(0x55818c645e40 msgr2=0x55818c646240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.467+0000 7efe5847b740 1 -- 172.21.15.70:0/1810865267 shutdown_connections 2024-04-01T01:37:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.467+0000 7efe5847b740 1 -- 172.21.15.70:0/1810865267 wait complete. 2024-04-01T01:37:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.467+0000 7efe5847b740 1 -- start start 2024-04-01T01:37:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.468+0000 7efe5847b740 1 Processor -- start 2024-04-01T01:37:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.468+0000 7efe5847b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c64a8d0 0x55818c654a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.468+0000 7efe5847b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c64b2c0 0x55818c654f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.468+0000 7efe5847b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c650430 0x55818c65bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.468+0000 7efe4effd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c64a8d0 0x55818c654a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.468+0000 7efe4effd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c64a8d0 0x55818c654a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52954/0 (socket says 172.21.15.70:52954) 2024-04-01T01:37:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.469+0000 7efe4e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c64b2c0 0x55818c654f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.469+0000 7efe4effd640 1 -- 172.21.15.70:0/4272843799 learned_addr learned my addr 172.21.15.70:0/4272843799 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.469+0000 7efe5847b740 1 -- 172.21.15.70:0/4272843799 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55818c653260 con 0x55818c64b2c0 2024-04-01T01:37:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.469+0000 7efe5847b740 1 -- 172.21.15.70:0/4272843799 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55818c54cf80 con 0x55818c650430 2024-04-01T01:37:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.469+0000 7efe5847b740 1 -- 172.21.15.70:0/4272843799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55818c6533e0 con 0x55818c64a8d0 2024-04-01T01:37:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.469+0000 7efe5847b740 10 client.0 _ll_register_callbacks cb 0x55818c5a5290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.469+0000 7efe4f7fe640 1 --2- 172.21.15.70:0/4272843799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c650430 0x55818c65bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.469+0000 7efe4f7fe640 1 -- 172.21.15.70:0/4272843799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c64a8d0 msgr2=0x55818c654a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.469+0000 7efe4f7fe640 1 --2- 172.21.15.70:0/4272843799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c64a8d0 0x55818c654a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.469+0000 7efe4f7fe640 1 -- 172.21.15.70:0/4272843799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c64b2c0 msgr2=0x55818c654f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.469+0000 7efe4f7fe640 1 --2- 172.21.15.70:0/4272843799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c64b2c0 0x55818c654f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.469+0000 7efe4f7fe640 1 -- 172.21.15.70:0/4272843799 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55818c38f620 con 0x55818c650430 2024-04-01T01:37:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.470+0000 7efe4f7fe640 1 --2- 172.21.15.70:0/4272843799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c650430 0x55818c65bfb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7efe44009780 tx=0x7efe44005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f80526e2d4673729 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.470+0000 7efe2effd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.470+0000 7efe2effd640 1 -- 172.21.15.70:0/4272843799 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4404e020 con 0x55818c650430 2024-04-01T01:37:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.470+0000 7efe2effd640 1 -- 172.21.15.70:0/4272843799 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7efe440035f0 con 0x55818c650430 2024-04-01T01:37:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.470+0000 7efe5847b740 1 -- 172.21.15.70:0/4272843799 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe4809bd70 con 0x55818c650430 2024-04-01T01:37:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.471+0000 7efe2effd640 1 -- 172.21.15.70:0/4272843799 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efe44039040 con 0x55818c650430 2024-04-01T01:37:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.471+0000 7efe5847b740 10 client.26762 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.471+0000 7efe5847b740 1 -- 172.21.15.70:0/4272843799 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55818c65f000 con 0x55818c650430 2024-04-01T01:37:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.471+0000 7efe2effd640 1 -- 172.21.15.70:0/4272843799 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe44045070 con 0x55818c650430 2024-04-01T01:37:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.471+0000 7efe2effd640 1 -- 172.21.15.70:0/4272843799 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7efe4403f6f0 con 0x55818c650430 2024-04-01T01:37:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.471+0000 7efe5847b740 10 client.26762 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.471+0000 7efe5847b740 10 client.26762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.471+0000 7efe5847b740 20 client.26762 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.471+0000 7efe5847b740 1 --2- 172.21.15.70:0/4272843799 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55818c660700 0x55818c662af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.472+0000 7efe5847b740 4 client.26762 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.472+0000 7efe5847b740 1 -- 172.21.15.70:0/4272843799 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7efe4809bd70 con 0x55818c660700 2024-04-01T01:37:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.472+0000 7efe4effd640 1 --2- 172.21.15.70:0/4272843799 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55818c660700 0x55818c662af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.472+0000 7efe2effd640 10 client.26762 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.473+0000 7efe4effd640 1 --2- 172.21.15.70:0/4272843799 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55818c660700 0x55818c662af0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5828cd0e5cbb12b8 server_cookie=a2247bc209e2bf96 in_seq=0 out_seq=0 2024-04-01T01:37:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.474+0000 7efe2effd640 1 -- 172.21.15.70:0/4272843799 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7efe4809bd70 con 0x55818c660700 2024-04-01T01:37:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.474+0000 7efe2effd640 10 client.26762 handle_command_reply: tid=0 2024-04-01T01:37:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.474+0000 7efe5847b740 1 client.26762 shutdown 2024-04-01T01:37:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.475+0000 7efe5847b740 1 -- 172.21.15.70:0/4272843799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c650430 msgr2=0x55818c65bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.475+0000 7efe5847b740 1 --2- 172.21.15.70:0/4272843799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c650430 0x55818c65bfb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7efe44009780 tx=0x7efe44005c90 comp rx=0 tx=0).stop 2024-04-01T01:37:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.475+0000 7efe5847b740 1 -- 172.21.15.70:0/4272843799 shutdown_connections 2024-04-01T01:37:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.475+0000 7efe5847b740 1 --2- 172.21.15.70:0/4272843799 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55818c660700 0x55818c662af0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.475+0000 7efe2effd640 0 client.26762 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.475+0000 7efe5847b740 1 --2- 172.21.15.70:0/4272843799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c650430 0x55818c65bfb0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.475+0000 7efe5847b740 1 --2- 172.21.15.70:0/4272843799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c64b2c0 0x55818c654f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.475+0000 7efe5847b740 1 --2- 172.21.15.70:0/4272843799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c64a8d0 0x55818c654a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.475+0000 7efe5847b740 1 -- 172.21.15.70:0/4272843799 >> 172.21.15.70:0/4272843799 conn(0x55818c645e40 msgr2=0x55818c644f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.475+0000 7efe5847b740 1 -- 172.21.15.70:0/4272843799 shutdown_connections 2024-04-01T01:37:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.475+0000 7efe5847b740 1 -- 172.21.15.70:0/4272843799 wait complete. 2024-04-01T01:37:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.475+0000 7efe5847b740 20 client.26762 trim_cache size 0 max 16384 2024-04-01T01:37:43.505 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.505+0000 7efe5847b740 1 Processor -- start 2024-04-01T01:37:43.505 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.506+0000 7efe5847b740 1 -- start start 2024-04-01T01:37:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.506+0000 7efe5847b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c64a8d0 0x55818c640e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.506+0000 7efe5847b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c64b2c0 0x55818c65bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.506+0000 7efe5847b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c650430 0x55818c653d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.506+0000 7efe5847b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c00fd20 con 0x55818c64a8d0 2024-04-01T01:37:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.506+0000 7efe4effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c64a8d0 0x55818c640e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.506+0000 7efe4effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c64a8d0 0x55818c640e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47736/0 (socket says 172.21.15.70:47736) 2024-04-01T01:37:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.506+0000 7efe4effd640 1 -- 172.21.15.70:0/2358720799 learned_addr learned my addr 172.21.15.70:0/2358720799 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.507+0000 7efe4e7fc640 1 --2- 172.21.15.70:0/2358720799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c64b2c0 0x55818c65bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.507+0000 7efe4f7fe640 1 --2- 172.21.15.70:0/2358720799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c650430 0x55818c653d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.507+0000 7efe5847b740 1 -- 172.21.15.70:0/2358720799 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c011860 con 0x55818c64b2c0 2024-04-01T01:37:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.507+0000 7efe5847b740 1 -- 172.21.15.70:0/2358720799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c0103f0 con 0x55818c650430 2024-04-01T01:37:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.507+0000 7efe4e7fc640 1 -- 172.21.15.70:0/2358720799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c650430 msgr2=0x55818c653d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.507+0000 7efe4e7fc640 1 --2- 172.21.15.70:0/2358720799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c650430 0x55818c653d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.507+0000 7efe4e7fc640 1 -- 172.21.15.70:0/2358720799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c64a8d0 msgr2=0x55818c640e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.507+0000 7efe4e7fc640 1 --2- 172.21.15.70:0/2358720799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c64a8d0 0x55818c640e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.507+0000 7efe4e7fc640 1 -- 172.21.15.70:0/2358720799 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55818c64c030 con 0x55818c64b2c0 2024-04-01T01:37:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.508+0000 7efe4e7fc640 1 --2- 172.21.15.70:0/2358720799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c64b2c0 0x55818c65bf70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efe48098040 tx=0x7efe4800a6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94dc8e98e4bbe777 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.508+0000 7efe357fa640 1 -- 172.21.15.70:0/2358720799 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4809b1b0 con 0x55818c64b2c0 2024-04-01T01:37:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.508+0000 7efe357fa640 1 -- 172.21.15.70:0/2358720799 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe480d28b0 con 0x55818c64b2c0 2024-04-01T01:37:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.508+0000 7efe357fa640 1 -- 172.21.15.70:0/2358720799 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe4809b440 con 0x55818c64b2c0 2024-04-01T01:37:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.509+0000 7efe5847b740 1 -- 172.21.15.70:0/2358720799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c64b2c0 msgr2=0x55818c65bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.509+0000 7efe5847b740 1 --2- 172.21.15.70:0/2358720799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c64b2c0 0x55818c65bf70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efe48098040 tx=0x7efe4800a6a0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.509+0000 7efe5847b740 1 -- 172.21.15.70:0/2358720799 shutdown_connections 2024-04-01T01:37:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.509+0000 7efe5847b740 1 --2- 172.21.15.70:0/2358720799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c650430 0x55818c653d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.509+0000 7efe5847b740 1 --2- 172.21.15.70:0/2358720799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c64b2c0 0x55818c65bf70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.509+0000 7efe5847b740 1 --2- 172.21.15.70:0/2358720799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c64a8d0 0x55818c640e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.509+0000 7efe5847b740 1 -- 172.21.15.70:0/2358720799 >> 172.21.15.70:0/2358720799 conn(0x55818c645e40 msgr2=0x55818c646220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.510+0000 7efe5847b740 1 -- 172.21.15.70:0/2358720799 shutdown_connections 2024-04-01T01:37:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.510+0000 7efe5847b740 1 -- 172.21.15.70:0/2358720799 wait complete. 2024-04-01T01:37:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.510+0000 7efe5847b740 1 -- start start 2024-04-01T01:37:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.510+0000 7efe5847b740 1 Processor -- start 2024-04-01T01:37:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.511+0000 7efe5847b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c64a8d0 0x55818c659620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.511+0000 7efe5847b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c64b2c0 0x55818c659b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.511+0000 7efe4effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c64a8d0 0x55818c659620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.511+0000 7efe4effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c64a8d0 0x55818c659620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53896/0 (socket says 172.21.15.70:53896) 2024-04-01T01:37:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.511+0000 7efe4effd640 1 -- 172.21.15.70:0/2687959396 learned_addr learned my addr 172.21.15.70:0/2687959396 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.511+0000 7efe5847b740 1 --2- 172.21.15.70:0/2687959396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c650430 0x55818c65ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.511+0000 7efe4e7fc640 1 --2- 172.21.15.70:0/2687959396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c64b2c0 0x55818c659b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.511+0000 7efe5847b740 1 -- 172.21.15.70:0/2687959396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c011190 con 0x55818c650430 2024-04-01T01:37:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.512+0000 7efe5847b740 1 -- 172.21.15.70:0/2687959396 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c010ac0 con 0x55818c64a8d0 2024-04-01T01:37:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.512+0000 7efe5847b740 1 -- 172.21.15.70:0/2687959396 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c00dd00 con 0x55818c64b2c0 2024-04-01T01:37:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.512+0000 7efe4effd640 1 -- 172.21.15.70:0/2687959396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c64b2c0 msgr2=0x55818c659b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.512+0000 7efe4effd640 1 --2- 172.21.15.70:0/2687959396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c64b2c0 0x55818c659b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.512+0000 7efe4effd640 1 -- 172.21.15.70:0/2687959396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c650430 msgr2=0x55818c65ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.512+0000 7efe4effd640 1 --2- 172.21.15.70:0/2687959396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c650430 0x55818c65ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.512+0000 7efe4effd640 1 -- 172.21.15.70:0/2687959396 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55818c65b0f0 con 0x55818c64a8d0 2024-04-01T01:37:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.512+0000 7efe4effd640 1 --2- 172.21.15.70:0/2687959396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c64a8d0 0x55818c659620 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efe40009910 tx=0x7efe40004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bb5bc6741f22cf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.512+0000 7efe2effd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.512+0000 7efe2effd640 1 -- 172.21.15.70:0/2687959396 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe40005020 con 0x55818c64a8d0 2024-04-01T01:37:43.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.513+0000 7efe2effd640 1 -- 172.21.15.70:0/2687959396 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7efe400051c0 con 0x55818c64a8d0 2024-04-01T01:37:43.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.513+0000 7efe2effd640 10 client.0 inactive, discarding osd_map(93..93 src has 1..93) 2024-04-01T01:37:43.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.513+0000 7efe5847b740 10 client.0 _ll_register_callbacks cb 0x7efe40005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:43.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.513+0000 7efe5847b740 1 -- 172.21.15.70:0/2687959396 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe480d28b0 con 0x55818c64a8d0 2024-04-01T01:37:43.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.513+0000 7efe2effd640 1 -- 172.21.15.70:0/2687959396 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efe4002a050 con 0x55818c64a8d0 2024-04-01T01:37:43.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.513+0000 7efe5847b740 10 client.26768 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:43.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.513+0000 7efe5847b740 1 -- 172.21.15.70:0/2687959396 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55818c6c6a30 con 0x55818c64a8d0 2024-04-01T01:37:43.514 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.514+0000 7efe2effd640 1 -- 172.21.15.70:0/2687959396 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7efe400053c0 con 0x55818c64a8d0 2024-04-01T01:37:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.514+0000 7efe5847b740 10 client.26768 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.514+0000 7efe5847b740 10 client.26768 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.514+0000 7efe5847b740 20 client.26768 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.514+0000 7efe5847b740 1 --2- 172.21.15.70:0/2687959396 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55818c660700 0x55818c6c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.514+0000 7efe5847b740 4 client.26768 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:37:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.514+0000 7efe5847b740 1 -- 172.21.15.70:0/2687959396 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7efe480d28b0 con 0x55818c660700 2024-04-01T01:37:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.515+0000 7efe4e7fc640 1 --2- 172.21.15.70:0/2687959396 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55818c660700 0x55818c6c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:43.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.518+0000 7efe4e7fc640 1 --2- 172.21.15.70:0/2687959396 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55818c660700 0x55818c6c92e0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f403e7360875440b server_cookie=6923fac54dbb0009 in_seq=0 out_seq=0 2024-04-01T01:37:43.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.518+0000 7efe2effd640 10 client.26768 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.559+0000 7efe2effd640 1 -- 172.21.15.70:0/2687959396 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7efe480d28b0 con 0x55818c660700 2024-04-01T01:37:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.559+0000 7efe2effd640 10 client.26768 handle_command_reply: tid=0 2024-04-01T01:37:43.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.559+0000 7efe5847b740 1 client.26768 shutdown 2024-04-01T01:37:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.560+0000 7efe5847b740 1 -- 172.21.15.70:0/2687959396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c64a8d0 msgr2=0x55818c659620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.560+0000 7efe5847b740 1 --2- 172.21.15.70:0/2687959396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c64a8d0 0x55818c659620 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efe40009910 tx=0x7efe40004040 comp rx=0 tx=0).stop 2024-04-01T01:37:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.560+0000 7efe5847b740 1 -- 172.21.15.70:0/2687959396 shutdown_connections 2024-04-01T01:37:43.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.560+0000 7efe5847b740 1 --2- 172.21.15.70:0/2687959396 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55818c660700 0x55818c6c92e0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.560+0000 7efe5847b740 1 --2- 172.21.15.70:0/2687959396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55818c650430 0x55818c65ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.560+0000 7efe2effd640 0 client.26768 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:43.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.560+0000 7efe5847b740 1 --2- 172.21.15.70:0/2687959396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55818c64b2c0 0x55818c659b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.560+0000 7efe5847b740 1 --2- 172.21.15.70:0/2687959396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55818c64a8d0 0x55818c659620 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.560+0000 7efe5847b740 1 -- 172.21.15.70:0/2687959396 >> 172.21.15.70:0/2687959396 conn(0x55818c645e40 msgr2=0x55818c653d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:43.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.561+0000 7efe5847b740 1 -- 172.21.15.70:0/2687959396 shutdown_connections 2024-04-01T01:37:43.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.562+0000 7efe5847b740 1 -- 172.21.15.70:0/2687959396 wait complete. 2024-04-01T01:37:43.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.562+0000 7efe5847b740 20 client.26768 trim_cache size 0 max 16384 2024-04-01T01:37:43.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:37:43.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:37:43.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:37:43.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:37:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.564+0000 7efe5588f640 1 -- 172.21.15.70:0/3699961519 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efe380a2700 msgr2=0x7efe380a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.564+0000 7efe5588f640 1 --2- 172.21.15.70:0/3699961519 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efe380a2700 0x7efe380a4bc0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7efe44005d00 tx=0x7efe4402c090 comp rx=0 tx=0).stop 2024-04-01T01:37:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.564+0000 7efe5588f640 1 -- 172.21.15.70:0/3699961519 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe50109800 msgr2=0x7efe50118150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.564+0000 7efe5588f640 1 --2- 172.21.15.70:0/3699961519 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe50109800 0x7efe50118150 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efe4800ed10 tx=0x7efe4800c7a0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.564+0000 7efe5588f640 1 -- 172.21.15.70:0/3699961519 shutdown_connections 2024-04-01T01:37:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.564+0000 7efe5588f640 1 --2- 172.21.15.70:0/3699961519 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efe380a2700 0x7efe380a4bc0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.564+0000 7efe5588f640 1 --2- 172.21.15.70:0/3699961519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe5010a1a0 0x7efe50119190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.564+0000 7efe5588f640 1 --2- 172.21.15.70:0/3699961519 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe50109800 0x7efe50118150 secure :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7efe4800ed10 tx=0x7efe4800c7a0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.564+0000 7efe5588f640 1 --2- 172.21.15.70:0/3699961519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe50108af0 0x7efe50117c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:43.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.564+0000 7efe5588f640 1 -- 172.21.15.70:0/3699961519 >> 172.21.15.70:0/3699961519 conn(0x7efe5006ed90 msgr2=0x7efe5010ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:43.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.566+0000 7efe5588f640 1 -- 172.21.15.70:0/3699961519 shutdown_connections 2024-04-01T01:37:43.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:43.566+0000 7efe5588f640 1 -- 172.21.15.70:0/3699961519 wait complete. 2024-04-01T01:37:44.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:43 smithi070 ceph-mon[32179]: pgmap v485: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 9.0 MiB/s wr, 1.13k op/s 2024-04-01T01:37:44.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:43 smithi081 ceph-mon[35142]: pgmap v485: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 9.0 MiB/s wr, 1.13k op/s 2024-04-01T01:37:44.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:43 smithi189 ceph-mon[35753]: pgmap v485: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 9.0 MiB/s wr, 1.13k op/s 2024-04-01T01:37:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:45 smithi081 ceph-mon[35142]: pgmap v486: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 8.0 MiB/s wr, 1.02k op/s 2024-04-01T01:37:46.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:45 smithi189 ceph-mon[35753]: pgmap v486: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 8.0 MiB/s wr, 1.02k op/s 2024-04-01T01:37:46.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:45 smithi070 ceph-mon[32179]: pgmap v486: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 8.0 MiB/s wr, 1.02k op/s 2024-04-01T01:37:46.897 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 180.0 s, 976.8 tps, lat 102.291 ms stddev 34.039 2024-04-01T01:37:48.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:47 smithi081 ceph-mon[35142]: pgmap v487: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.3 MiB/s wr, 1.18k op/s 2024-04-01T01:37:48.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:47 smithi189 ceph-mon[35753]: pgmap v487: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.3 MiB/s wr, 1.18k op/s 2024-04-01T01:37:48.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:47 smithi070 ceph-mon[32179]: pgmap v487: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.3 MiB/s wr, 1.18k op/s 2024-04-01T01:37:50.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:49 smithi081 ceph-mon[35142]: pgmap v488: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 7.9 MiB/s wr, 1.03k op/s 2024-04-01T01:37:50.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:49 smithi189 ceph-mon[35753]: pgmap v488: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 7.9 MiB/s wr, 1.03k op/s 2024-04-01T01:37:50.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:49 smithi070 ceph-mon[32179]: pgmap v488: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 7.9 MiB/s wr, 1.03k op/s 2024-04-01T01:37:51.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 185.0 s, 968.4 tps, lat 103.328 ms stddev 33.440 2024-04-01T01:37:52.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:51 smithi081 ceph-mon[35142]: pgmap v489: 129 pgs: 129 active+clean; 1.0 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 8.0 MiB/s wr, 1.03k op/s 2024-04-01T01:37:52.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:51 smithi189 ceph-mon[35753]: pgmap v489: 129 pgs: 129 active+clean; 1.0 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 8.0 MiB/s wr, 1.03k op/s 2024-04-01T01:37:52.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:51 smithi070 ceph-mon[32179]: pgmap v489: 129 pgs: 129 active+clean; 1.0 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 8.0 MiB/s wr, 1.03k op/s 2024-04-01T01:37:54.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:53 smithi081 ceph-mon[35142]: pgmap v490: 129 pgs: 129 active+clean; 1.0 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 8.7 MiB/s wr, 1.11k op/s 2024-04-01T01:37:54.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:53 smithi189 ceph-mon[35753]: pgmap v490: 129 pgs: 129 active+clean; 1.0 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 8.7 MiB/s wr, 1.11k op/s 2024-04-01T01:37:54.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:53 smithi070 ceph-mon[32179]: pgmap v490: 129 pgs: 129 active+clean; 1.0 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 8.7 MiB/s wr, 1.11k op/s 2024-04-01T01:37:55.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:54 smithi081 ceph-mon[35142]: pgmap v491: 129 pgs: 129 active+clean; 1.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.6 MiB/s wr, 906 op/s 2024-04-01T01:37:55.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:54 smithi189 ceph-mon[35753]: pgmap v491: 129 pgs: 129 active+clean; 1.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.6 MiB/s wr, 906 op/s 2024-04-01T01:37:55.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:54 smithi070 ceph-mon[32179]: pgmap v491: 129 pgs: 129 active+clean; 1.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.6 MiB/s wr, 906 op/s 2024-04-01T01:37:56.721 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.720+0000 7fa1c8848640 1 -- 172.21.15.70:0/3275854267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c4105150 msgr2=0x7fa1c41055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:56.721 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.720+0000 7fa1c8848640 1 --2- 172.21.15.70:0/3275854267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c4105150 0x7fa1c41055d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8009900 tx=0x7fa1b802bc50 comp rx=0 tx=0).stop 2024-04-01T01:37:56.721 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.720+0000 7fa1c8848640 1 -- 172.21.15.70:0/3275854267 shutdown_connections 2024-04-01T01:37:56.721 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.720+0000 7fa1c8848640 1 --2- 172.21.15.70:0/3275854267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c4105150 0x7fa1c41055d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.721 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.720+0000 7fa1c8848640 1 --2- 172.21.15.70:0/3275854267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1c41027a0 0x7fa1c4104c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.721 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.720+0000 7fa1c8848640 1 --2- 172.21.15.70:0/3275854267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1c40ffe30 0x7fa1c4102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.722 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.720+0000 7fa1c8848640 1 -- 172.21.15.70:0/3275854267 >> 172.21.15.70:0/3275854267 conn(0x7fa1c40fba50 msgr2=0x7fa1c40fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:56.722 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.720+0000 7fa1c8848640 1 -- 172.21.15.70:0/3275854267 shutdown_connections 2024-04-01T01:37:56.722 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.721+0000 7fa1c8848640 1 -- 172.21.15.70:0/3275854267 wait complete. 2024-04-01T01:37:56.722 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.722+0000 7fa1c8848640 1 Processor -- start 2024-04-01T01:37:56.723 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.723+0000 7fa1c8848640 1 -- start start 2024-04-01T01:37:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.724+0000 7fa1c8848640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c40ffe30 0x7fa1c419ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.724+0000 7fa1c8848640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1c41027a0 0x7fa1c419e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.724+0000 7fa1c8848640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1c4105150 0x7fa1c41a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.724+0000 7fa1c8848640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c41137d0 con 0x7fa1c41027a0 2024-04-01T01:37:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.724+0000 7fa1c8848640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c4113650 con 0x7fa1c40ffe30 2024-04-01T01:37:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.724+0000 7fa1c8848640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c4113950 con 0x7fa1c4105150 2024-04-01T01:37:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.724+0000 7fa1c2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c40ffe30 0x7fa1c419ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.724+0000 7fa1c2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c40ffe30 0x7fa1c419ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33630/0 (socket says 172.21.15.70:33630) 2024-04-01T01:37:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.724+0000 7fa1c2d9d640 1 -- 172.21.15.70:0/1842730208 learned_addr learned my addr 172.21.15.70:0/1842730208 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.724+0000 7fa1c359e640 1 --2- 172.21.15.70:0/1842730208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1c4105150 0x7fa1c41a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.724+0000 7fa1c2d9d640 1 -- 172.21.15.70:0/1842730208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1c4105150 msgr2=0x7fa1c41a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.724+0000 7fa1c2d9d640 1 --2- 172.21.15.70:0/1842730208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1c4105150 0x7fa1c41a5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.724+0000 7fa1c2d9d640 1 -- 172.21.15.70:0/1842730208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1c41027a0 msgr2=0x7fa1c419e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.724+0000 7fa1c2d9d640 1 --2- 172.21.15.70:0/1842730208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1c41027a0 0x7fa1c419e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.724+0000 7fa1c2d9d640 1 -- 172.21.15.70:0/1842730208 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c41a59d0 con 0x7fa1c40ffe30 2024-04-01T01:37:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.725+0000 7fa1c2d9d640 1 --2- 172.21.15.70:0/1842730208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c40ffe30 0x7fa1c419ddb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc00ba50 tx=0x7fa1bc00bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.727+0000 7fa1abfff640 1 -- 172.21.15.70:0/1842730208 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc019070 con 0x7fa1c40ffe30 2024-04-01T01:37:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.727+0000 7fa1abfff640 1 -- 172.21.15.70:0/1842730208 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1bc00cc10 con 0x7fa1c40ffe30 2024-04-01T01:37:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.727+0000 7fa1abfff640 1 -- 172.21.15.70:0/1842730208 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc0076a0 con 0x7fa1c40ffe30 2024-04-01T01:37:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.727+0000 7fa1c8848640 1 -- 172.21.15.70:0/1842730208 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1c41a5cc0 con 0x7fa1c40ffe30 2024-04-01T01:37:56.729 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.727+0000 7fa1c8848640 1 -- 172.21.15.70:0/1842730208 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1c41a6160 con 0x7fa1c40ffe30 2024-04-01T01:37:56.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.731+0000 7fa1abfff640 1 -- 172.21.15.70:0/1842730208 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fa1bc01c070 con 0x7fa1c40ffe30 2024-04-01T01:37:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.732+0000 7fa1abfff640 1 --2- 172.21.15.70:0/1842730208 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa1ac0a26a0 0x7fa1ac0a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.732+0000 7fa1abfff640 1 -- 172.21.15.70:0/1842730208 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa1bc0cdfc0 con 0x7fa1c40ffe30 2024-04-01T01:37:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.732+0000 7fa1cbc1f740 1 Processor -- start 2024-04-01T01:37:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.732+0000 7fa1cbc1f740 1 -- start start 2024-04-01T01:37:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.732+0000 7fa1c259c640 1 --2- 172.21.15.70:0/1842730208 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa1ac0a26a0 0x7fa1ac0a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1cbc1f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fd5ae0 0x55bc98fd5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1cbc1f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fd64d0 0x55bc98fdafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1cbc1f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fdb640 0x55bc98fddad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1cbc1f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bc98f398f0 con 0x55bc98fd5ae0 2024-04-01T01:37:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1cbc1f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55bc98ee28d0 con 0x55bc98fd64d0 2024-04-01T01:37:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1cbc1f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bc98f35990 con 0x55bc98fdb640 2024-04-01T01:37:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1c359e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fdb640 0x55bc98fddad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1c359e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fdb640 0x55bc98fddad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41142/0 (socket says 172.21.15.70:41142) 2024-04-01T01:37:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1c359e640 1 -- 172.21.15.70:0/1408341760 learned_addr learned my addr 172.21.15.70:0/1408341760 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1c259c640 1 --2- 172.21.15.70:0/1408341760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fd64d0 0x55bc98fdafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1c2d9d640 1 --2- 172.21.15.70:0/1408341760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fd5ae0 0x55bc98fd5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1c259c640 1 --2- 172.21.15.70:0/1842730208 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa1ac0a26a0 0x7fa1ac0a4b60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa1c419f2c0 tx=0x7fa1b4009b90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1c259c640 1 -- 172.21.15.70:0/1408341760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fdb640 msgr2=0x55bc98fddad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1c259c640 1 --2- 172.21.15.70:0/1408341760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fdb640 0x55bc98fddad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1c259c640 1 -- 172.21.15.70:0/1408341760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fd5ae0 msgr2=0x55bc98fd5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:56.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1c259c640 1 --2- 172.21.15.70:0/1408341760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fd5ae0 0x55bc98fd5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.733+0000 7fa1c259c640 1 -- 172.21.15.70:0/1408341760 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc98c4e210 con 0x55bc98fd64d0 2024-04-01T01:37:56.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.734+0000 7fa1c259c640 1 --2- 172.21.15.70:0/1408341760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fd64d0 0x55bc98fdafd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa1b400ba30 tx=0x7fa1b4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4862a6c96813359 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:56.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.734+0000 7fa1a97fa640 1 -- 172.21.15.70:0/1408341760 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4023070 con 0x55bc98fd64d0 2024-04-01T01:37:56.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.734+0000 7fa1a97fa640 1 -- 172.21.15.70:0/1408341760 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1b4016c70 con 0x55bc98fd64d0 2024-04-01T01:37:56.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.734+0000 7fa1a97fa640 1 -- 172.21.15.70:0/1408341760 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b400f040 con 0x55bc98fd64d0 2024-04-01T01:37:56.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.735+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/1408341760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fd64d0 msgr2=0x55bc98fdafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:56.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.735+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/1408341760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fd64d0 0x55bc98fdafd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa1b400ba30 tx=0x7fa1b4016000 comp rx=0 tx=0).stop 2024-04-01T01:37:56.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.735+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/1408341760 shutdown_connections 2024-04-01T01:37:56.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.736+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/1408341760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fdb640 0x55bc98fddad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.736+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/1408341760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fd64d0 0x55bc98fdafd0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.736+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/1408341760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fd5ae0 0x55bc98fd5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.736+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/1408341760 >> 172.21.15.70:0/1408341760 conn(0x55bc98fd1050 msgr2=0x55bc98fd1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:56.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.736+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/1408341760 shutdown_connections 2024-04-01T01:37:56.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.736+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/1408341760 wait complete. 2024-04-01T01:37:56.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.736+0000 7fa1cbc1f740 1 -- start start 2024-04-01T01:37:56.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.737+0000 7fa1cbc1f740 1 Processor -- start 2024-04-01T01:37:56.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.737+0000 7fa1cbc1f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 0x55bc98fdfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.737+0000 7fa1cbc1f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fd64d0 0x55bc98fe03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.738+0000 7fa1c2d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 0x55bc98fdfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:56.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.738+0000 7fa1c2d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 0x55bc98fdfe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41154/0 (socket says 172.21.15.70:41154) 2024-04-01T01:37:56.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.738+0000 7fa1c2d9d640 1 -- 172.21.15.70:0/126066297 learned_addr learned my addr 172.21.15.70:0/126066297 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:56.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.738+0000 7fa1c259c640 1 --2- 172.21.15.70:0/126066297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fd64d0 0x55bc98fe03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.738+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/126066297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fdb640 0x55bc98fe7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.738+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/126066297 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bc98fde640 con 0x55bc98fd64d0 2024-04-01T01:37:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.738+0000 7fa1c359e640 1 --2- 172.21.15.70:0/126066297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fdb640 0x55bc98fe7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.738+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/126066297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55bc98ed7f80 con 0x55bc98fdb640 2024-04-01T01:37:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.738+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/126066297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bc98fde7c0 con 0x55bc98fd5ae0 2024-04-01T01:37:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.738+0000 7fa1cbc1f740 10 client.0 _ll_register_callbacks cb 0x55bc98f30290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.739+0000 7fa1c359e640 1 -- 172.21.15.70:0/126066297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 msgr2=0x55bc98fdfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.739+0000 7fa1c359e640 1 --2- 172.21.15.70:0/126066297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 0x55bc98fdfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.739+0000 7fa1c359e640 1 -- 172.21.15.70:0/126066297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fd64d0 msgr2=0x55bc98fe03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.739+0000 7fa1c359e640 1 --2- 172.21.15.70:0/126066297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fd64d0 0x55bc98fe03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.739+0000 7fa1c359e640 1 -- 172.21.15.70:0/126066297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc98d1a620 con 0x55bc98fdb640 2024-04-01T01:37:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.739+0000 7fa1c259c640 1 --2- 172.21.15.70:0/126066297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fd64d0 0x55bc98fe03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:37:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.739+0000 7fa1c359e640 1 --2- 172.21.15.70:0/126066297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fdb640 0x55bc98fe7400 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8009230 tx=0x7fa1b8005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cb26aab582b66e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.740+0000 7fa1a6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.740+0000 7fa1a6ff5640 1 -- 172.21.15.70:0/126066297 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b80053c0 con 0x55bc98fdb640 2024-04-01T01:37:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.740+0000 7fa1a6ff5640 1 -- 172.21.15.70:0/126066297 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa1b8005560 con 0x55bc98fdb640 2024-04-01T01:37:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.740+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/126066297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1b4016c70 con 0x55bc98fdb640 2024-04-01T01:37:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.740+0000 7fa1a6ff5640 1 -- 172.21.15.70:0/126066297 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa1b8004720 con 0x55bc98fdb640 2024-04-01T01:37:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.740+0000 7fa1cbc1f740 10 client.26780 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.740+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/126066297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bc98fea450 con 0x55bc98fdb640 2024-04-01T01:37:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.741+0000 7fa1a6ff5640 1 -- 172.21.15.70:0/126066297 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1b8004d70 con 0x55bc98fdb640 2024-04-01T01:37:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.741+0000 7fa1a6ff5640 1 -- 172.21.15.70:0/126066297 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa1b8034440 con 0x55bc98fdb640 2024-04-01T01:37:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.741+0000 7fa1cbc1f740 10 client.26780 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.741+0000 7fa1cbc1f740 10 client.26780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.741+0000 7fa1cbc1f740 20 client.26780 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.741+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/126066297 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bc98febf50 0x55bc98fee3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.741+0000 7fa1cbc1f740 4 client.26780 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:37:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.741+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/126066297 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa1b4016c70 con 0x55bc98febf50 2024-04-01T01:37:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.741+0000 7fa1c2d9d640 1 --2- 172.21.15.70:0/126066297 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bc98febf50 0x55bc98fee3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.742+0000 7fa1a6ff5640 10 client.26780 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.742+0000 7fa1c2d9d640 1 --2- 172.21.15.70:0/126066297 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bc98febf50 0x55bc98fee3b0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7be3ae5452076940 server_cookie=300bf89b83ae2408 in_seq=0 out_seq=0 2024-04-01T01:37:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.743+0000 7fa1a6ff5640 1 -- 172.21.15.70:0/126066297 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fa1b4016c70 con 0x55bc98febf50 2024-04-01T01:37:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.743+0000 7fa1a6ff5640 10 client.26780 handle_command_reply: tid=0 2024-04-01T01:37:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.743+0000 7fa1cbc1f740 1 client.26780 shutdown 2024-04-01T01:37:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.743+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/126066297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fdb640 msgr2=0x55bc98fe7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.743+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/126066297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fdb640 0x55bc98fe7400 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8009230 tx=0x7fa1b8005040 comp rx=0 tx=0).stop 2024-04-01T01:37:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.743+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/126066297 shutdown_connections 2024-04-01T01:37:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.743+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/126066297 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bc98febf50 0x55bc98fee3b0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.743+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/126066297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fdb640 0x55bc98fe7400 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.743+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/126066297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fd64d0 0x55bc98fe03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.743+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/126066297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 0x55bc98fdfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.743+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/126066297 >> 172.21.15.70:0/126066297 conn(0x55bc98fd1050 msgr2=0x55bc98fd0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.744+0000 7fa1a6ff5640 0 client.26780 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.744+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/126066297 shutdown_connections 2024-04-01T01:37:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.744+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/126066297 wait complete. 2024-04-01T01:37:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.744+0000 7fa1cbc1f740 20 client.26780 trim_cache size 0 max 16384 2024-04-01T01:37:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.772+0000 7fa1cbc1f740 1 Processor -- start 2024-04-01T01:37:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.772+0000 7fa1cbc1f740 1 -- start start 2024-04-01T01:37:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.772+0000 7fa1cbc1f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 0x55bc98fd1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.772+0000 7fa1cbc1f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fd64d0 0x55bc98fe73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.772+0000 7fa1cbc1f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fdb640 0x55bc98ff3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.772+0000 7fa1cbc1f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b000fda0 con 0x55bc98fd64d0 2024-04-01T01:37:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.772+0000 7fa1cbc1f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b00118e0 con 0x55bc98fdb640 2024-04-01T01:37:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.772+0000 7fa1cbc1f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b0010470 con 0x55bc98fd5ae0 2024-04-01T01:37:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.772+0000 7fa1c359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fdb640 0x55bc98ff3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.772+0000 7fa1c359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fdb640 0x55bc98ff3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33650/0 (socket says 172.21.15.70:33650) 2024-04-01T01:37:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.772+0000 7fa1c359e640 1 -- 172.21.15.70:0/2404399680 learned_addr learned my addr 172.21.15.70:0/2404399680 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.772+0000 7fa1c2d9d640 1 --2- 172.21.15.70:0/2404399680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 0x55bc98fd1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.773+0000 7fa1c359e640 1 -- 172.21.15.70:0/2404399680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 msgr2=0x55bc98fd1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:56.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.773+0000 7fa1c359e640 1 --2- 172.21.15.70:0/2404399680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 0x55bc98fd1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.773+0000 7fa1c359e640 1 -- 172.21.15.70:0/2404399680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fd64d0 msgr2=0x55bc98fe73c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:37:56.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.773+0000 7fa1c359e640 1 --2- 172.21.15.70:0/2404399680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fd64d0 0x55bc98fe73c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.773+0000 7fa1c359e640 1 -- 172.21.15.70:0/2404399680 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc98fd1dc0 con 0x55bc98fdb640 2024-04-01T01:37:56.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.773+0000 7fa1c359e640 1 --2- 172.21.15.70:0/2404399680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fdb640 0x55bc98ff3cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8009850 tx=0x7fa1b802bc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=335aaee3bfb5f1e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:56.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.773+0000 7fa1a8ff9640 1 -- 172.21.15.70:0/2404399680 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b805b020 con 0x55bc98fdb640 2024-04-01T01:37:56.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.773+0000 7fa1a8ff9640 1 -- 172.21.15.70:0/2404399680 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1b8034440 con 0x55bc98fdb640 2024-04-01T01:37:56.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.774+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/2404399680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fdb640 msgr2=0x55bc98ff3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:56.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.774+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/2404399680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fdb640 0x55bc98ff3cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8009850 tx=0x7fa1b802bc50 comp rx=0 tx=0).stop 2024-04-01T01:37:56.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.774+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/2404399680 shutdown_connections 2024-04-01T01:37:56.775 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.774+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/2404399680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fdb640 0x55bc98ff3cc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.775+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/2404399680 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fd64d0 0x55bc98fe73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.775+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/2404399680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 0x55bc98fd1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.775+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/2404399680 >> 172.21.15.70:0/2404399680 conn(0x55bc98fd1050 msgr2=0x55bc98fcbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:56.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.775+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/2404399680 shutdown_connections 2024-04-01T01:37:56.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.775+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/2404399680 wait complete. 2024-04-01T01:37:56.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.775+0000 7fa1cbc1f740 1 -- start start 2024-04-01T01:37:56.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.775+0000 7fa1cbc1f740 1 Processor -- start 2024-04-01T01:37:56.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.776+0000 7fa1cbc1f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 0x55bc98fe2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.776+0000 7fa1cbc1f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fd64d0 0x55bc98fe2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.776+0000 7fa1cbc1f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fdb640 0x55bc990612f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.776+0000 7fa1cbc1f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b0011210 con 0x55bc98fdb640 2024-04-01T01:37:56.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.776+0000 7fa1cbc1f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b0010b40 con 0x55bc98fd64d0 2024-04-01T01:37:56.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.776+0000 7fa1cbc1f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b000dd80 con 0x55bc98fd5ae0 2024-04-01T01:37:56.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.776+0000 7fa1cbc1f740 10 client.0 _ll_register_callbacks cb 0x7fa1b802c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:37:56.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.776+0000 7fa1c2d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 0x55bc98fe2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:56.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.776+0000 7fa1c2d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 0x55bc98fe2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41174/0 (socket says 172.21.15.70:41174) 2024-04-01T01:37:56.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.776+0000 7fa1c2d9d640 1 -- 172.21.15.70:0/1342647199 learned_addr learned my addr 172.21.15.70:0/1342647199 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:37:56.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.776+0000 7fa1c259c640 1 --2- 172.21.15.70:0/1342647199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fd64d0 0x55bc98fe2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:56.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.776+0000 7fa1c359e640 1 --2- 172.21.15.70:0/1342647199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fdb640 0x55bc990612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:56.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.776+0000 7fa1c259c640 1 -- 172.21.15.70:0/1342647199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 msgr2=0x55bc98fe2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:56.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.777+0000 7fa1c259c640 1 --2- 172.21.15.70:0/1342647199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 0x55bc98fe2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.777+0000 7fa1c259c640 1 -- 172.21.15.70:0/1342647199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fdb640 msgr2=0x55bc990612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:37:56.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.777+0000 7fa1c259c640 1 --2- 172.21.15.70:0/1342647199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fdb640 0x55bc990612f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.777+0000 7fa1c259c640 1 -- 172.21.15.70:0/1342647199 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc99061830 con 0x55bc98fd64d0 2024-04-01T01:37:56.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.777+0000 7fa1c259c640 1 --2- 172.21.15.70:0/1342647199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fd64d0 0x55bc98fe2ad0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa1b400bfe0 tx=0x7fa1b400b600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=330547f2735aea2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:37:56.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.777+0000 7fa1a6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:37:56.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.777+0000 7fa1a6ff5640 1 -- 172.21.15.70:0/1342647199 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b402e070 con 0x55bc98fd64d0 2024-04-01T01:37:56.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.778+0000 7fa1a6ff5640 1 -- 172.21.15.70:0/1342647199 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa1b4016910 con 0x55bc98fd64d0 2024-04-01T01:37:56.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.778+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/1342647199 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bc98f4d120 con 0x55bc98fd64d0 2024-04-01T01:37:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.778+0000 7fa1a6ff5640 1 -- 172.21.15.70:0/1342647199 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa1b4037050 con 0x55bc98fd64d0 2024-04-01T01:37:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.779+0000 7fa1cbc1f740 10 client.26786 fetch_fsmap learned FSMap version 29 2024-04-01T01:37:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.779+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/1342647199 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bc99062c40 con 0x55bc98fd64d0 2024-04-01T01:37:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.779+0000 7fa1a6ff5640 1 -- 172.21.15.70:0/1342647199 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55bc98f4d120 con 0x55bc98fd64d0 2024-04-01T01:37:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.779+0000 7fa1a6ff5640 1 -- 172.21.15.70:0/1342647199 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa1b4029040 con 0x55bc98fd64d0 2024-04-01T01:37:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.780+0000 7fa1cbc1f740 10 client.26786 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:37:56.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.780+0000 7fa1cbc1f740 10 client.26786 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:37:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.780+0000 7fa1cbc1f740 20 client.26786 populate_metadata read hostname 'smithi070' 2024-04-01T01:37:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.780+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/1342647199 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bc99063360 0x55bc99065780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:37:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.780+0000 7fa1cbc1f740 4 client.26786 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:37:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.780+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/1342647199 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bc98f46e30 con 0x55bc99063360 2024-04-01T01:37:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.781+0000 7fa1c2d9d640 1 --2- 172.21.15.70:0/1342647199 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bc99063360 0x55bc99065780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:37:56.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.781+0000 7fa1c2d9d640 1 --2- 172.21.15.70:0/1342647199 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bc99063360 0x55bc99065780 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ef5b1754cd51fef server_cookie=515c4f1a412ac737 in_seq=0 out_seq=0 2024-04-01T01:37:56.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.781+0000 7fa1a6ff5640 10 client.26786 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:56.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.809+0000 7fa1a6ff5640 1 -- 172.21.15.70:0/1342647199 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bc98f46e30 con 0x55bc99063360 2024-04-01T01:37:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.809+0000 7fa1a6ff5640 10 client.26786 handle_command_reply: tid=0 2024-04-01T01:37:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.809+0000 7fa1cbc1f740 1 client.26786 shutdown 2024-04-01T01:37:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.809+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/1342647199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fd64d0 msgr2=0x55bc98fe2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.809+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/1342647199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fd64d0 0x55bc98fe2ad0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa1b400bfe0 tx=0x7fa1b400b600 comp rx=0 tx=0).stop 2024-04-01T01:37:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.810+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/1342647199 shutdown_connections 2024-04-01T01:37:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.810+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/1342647199 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bc99063360 0x55bc99065780 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.810+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/1342647199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bc98fdb640 0x55bc990612f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.810+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/1342647199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc98fd64d0 0x55bc98fe2ad0 secure :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa1b400bfe0 tx=0x7fa1b400b600 comp rx=0 tx=0).stop 2024-04-01T01:37:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.810+0000 7fa1cbc1f740 1 --2- 172.21.15.70:0/1342647199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bc98fd5ae0 0x55bc98fe2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.810+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/1342647199 >> 172.21.15.70:0/1342647199 conn(0x55bc98fd1050 msgr2=0x55bc98fcbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.810+0000 7fa1a6ff5640 0 client.26786 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:37:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.813+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/1342647199 shutdown_connections 2024-04-01T01:37:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.813+0000 7fa1cbc1f740 1 -- 172.21.15.70:0/1342647199 wait complete. 2024-04-01T01:37:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.813+0000 7fa1cbc1f740 20 client.26786 trim_cache size 0 max 16384 2024-04-01T01:37:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:37:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:37:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:37:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:37:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.817+0000 7fa1c8848640 1 -- 172.21.15.70:0/1842730208 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa1ac0a26a0 msgr2=0x7fa1ac0a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.817+0000 7fa1c8848640 1 --2- 172.21.15.70:0/1842730208 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa1ac0a26a0 0x7fa1ac0a4b60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa1c419f2c0 tx=0x7fa1b4009b90 comp rx=0 tx=0).stop 2024-04-01T01:37:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.818+0000 7fa1c8848640 1 -- 172.21.15.70:0/1842730208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c40ffe30 msgr2=0x7fa1c419ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:37:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.818+0000 7fa1c8848640 1 --2- 172.21.15.70:0/1842730208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c40ffe30 0x7fa1c419ddb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc00ba50 tx=0x7fa1bc00bf10 comp rx=0 tx=0).stop 2024-04-01T01:37:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.818+0000 7fa1c8848640 1 -- 172.21.15.70:0/1842730208 shutdown_connections 2024-04-01T01:37:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.818+0000 7fa1c8848640 1 --2- 172.21.15.70:0/1842730208 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa1ac0a26a0 0x7fa1ac0a4b60 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.818+0000 7fa1c8848640 1 --2- 172.21.15.70:0/1842730208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1c4105150 0x7fa1c41a5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.818+0000 7fa1c8848640 1 --2- 172.21.15.70:0/1842730208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1c41027a0 0x7fa1c419e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:37:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.818+0000 7fa1c8848640 1 --2- 172.21.15.70:0/1842730208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c40ffe30 0x7fa1c419ddb0 secure :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc00ba50 tx=0x7fa1bc00bf10 comp rx=0 tx=0).stop 2024-04-01T01:37:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.818+0000 7fa1c8848640 1 -- 172.21.15.70:0/1842730208 >> 172.21.15.70:0/1842730208 conn(0x7fa1c40fba50 msgr2=0x7fa1c4100ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:37:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.819+0000 7fa1c8848640 1 -- 172.21.15.70:0/1842730208 shutdown_connections 2024-04-01T01:37:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:37:56.819+0000 7fa1c8848640 1 -- 172.21.15.70:0/1842730208 wait complete. 2024-04-01T01:37:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 190.0 s, 891.8 tps, lat 111.910 ms stddev 117.797 2024-04-01T01:37:58.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:57 smithi070 ceph-mon[32179]: pgmap v492: 129 pgs: 129 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 10 MiB/s wr, 1.06k op/s 2024-04-01T01:37:58.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:57 smithi081 ceph-mon[35142]: pgmap v492: 129 pgs: 129 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 10 MiB/s wr, 1.06k op/s 2024-04-01T01:37:58.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:57 smithi189 ceph-mon[35753]: pgmap v492: 129 pgs: 129 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 10 MiB/s wr, 1.06k op/s 2024-04-01T01:37:59.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:58 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:37:59.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:58 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:37:59.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:58 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:38:00.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:37:59 smithi081 ceph-mon[35142]: pgmap v493: 129 pgs: 129 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.2 MiB/s wr, 944 op/s 2024-04-01T01:38:00.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:37:59 smithi189 ceph-mon[35753]: pgmap v493: 129 pgs: 129 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.2 MiB/s wr, 944 op/s 2024-04-01T01:38:00.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:37:59 smithi070 ceph-mon[32179]: pgmap v493: 129 pgs: 129 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.2 MiB/s wr, 944 op/s 2024-04-01T01:38:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:38:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:38:01.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:38:01.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:00 smithi081 ceph-mon[35142]: pgmap v494: 129 pgs: 129 active+clean; 1.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.2 MiB/s wr, 927 op/s 2024-04-01T01:38:01.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:38:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:38:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:38:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:00 smithi189 ceph-mon[35753]: pgmap v494: 129 pgs: 129 active+clean; 1.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.2 MiB/s wr, 927 op/s 2024-04-01T01:38:01.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:38:01.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:38:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:38:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:00 smithi070 ceph-mon[32179]: pgmap v494: 129 pgs: 129 active+clean; 1.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.2 MiB/s wr, 927 op/s 2024-04-01T01:38:01.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 195.0 s, 901.4 tps, lat 111.141 ms stddev 43.600 2024-04-01T01:38:04.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:03 smithi070 ceph-mon[32179]: pgmap v495: 129 pgs: 129 active+clean; 1.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 9.2 MiB/s wr, 946 op/s 2024-04-01T01:38:04.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:03 smithi081 ceph-mon[35142]: pgmap v495: 129 pgs: 129 active+clean; 1.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 9.2 MiB/s wr, 946 op/s 2024-04-01T01:38:04.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:03 smithi189 ceph-mon[35753]: pgmap v495: 129 pgs: 129 active+clean; 1.0 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 9.2 MiB/s wr, 946 op/s 2024-04-01T01:38:06.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:05 smithi081 ceph-mon[35142]: pgmap v496: 129 pgs: 129 active+clean; 1.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.4 MiB/s wr, 841 op/s 2024-04-01T01:38:06.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:05 smithi189 ceph-mon[35753]: pgmap v496: 129 pgs: 129 active+clean; 1.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.4 MiB/s wr, 841 op/s 2024-04-01T01:38:06.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:05 smithi070 ceph-mon[32179]: pgmap v496: 129 pgs: 129 active+clean; 1.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 8.4 MiB/s wr, 841 op/s 2024-04-01T01:38:06.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 200.0 s, 994.6 tps, lat 100.892 ms stddev 57.011 2024-04-01T01:38:08.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:07 smithi081 ceph-mon[35142]: pgmap v497: 129 pgs: 129 active+clean; 1.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 9.9 MiB/s wr, 1.02k op/s 2024-04-01T01:38:08.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:07 smithi189 ceph-mon[35753]: pgmap v497: 129 pgs: 129 active+clean; 1.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 9.9 MiB/s wr, 1.02k op/s 2024-04-01T01:38:08.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:07 smithi070 ceph-mon[32179]: pgmap v497: 129 pgs: 129 active+clean; 1.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 9.9 MiB/s wr, 1.02k op/s 2024-04-01T01:38:09.967 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.968+0000 7f7bfffff640 1 -- 172.21.15.70:0/58787970 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8005220 con 0x7f7c08108750 2024-04-01T01:38:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.969+0000 7f7c0f9f2640 1 -- 172.21.15.70:0/58787970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c08108750 msgr2=0x7f7c0810ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:09.968 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.969+0000 7f7c0f9f2640 1 --2- 172.21.15.70:0/58787970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c08108750 0x7f7c0810ab80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8009900 tx=0x7f7bf802bc30 comp rx=0 tx=0).stop 2024-04-01T01:38:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.969+0000 7f7c0f9f2640 1 -- 172.21.15.70:0/58787970 shutdown_connections 2024-04-01T01:38:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.969+0000 7f7c0f9f2640 1 --2- 172.21.15.70:0/58787970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c0810da70 0x7f7c0810def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.969+0000 7f7c0f9f2640 1 --2- 172.21.15.70:0/58787970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c0810b0c0 0x7f7c0810d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.969+0000 7f7c0f9f2640 1 --2- 172.21.15.70:0/58787970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c08108750 0x7f7c0810ab80 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.969+0000 7f7c0f9f2640 1 -- 172.21.15.70:0/58787970 >> 172.21.15.70:0/58787970 conn(0x7f7c0806e790 msgr2=0x7f7c08070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.969+0000 7f7c0f9f2640 1 -- 172.21.15.70:0/58787970 shutdown_connections 2024-04-01T01:38:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.970+0000 7f7c0f9f2640 1 -- 172.21.15.70:0/58787970 wait complete. 2024-04-01T01:38:09.969 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.971+0000 7f7c0f9f2640 1 Processor -- start 2024-04-01T01:38:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.971+0000 7f7c0f9f2640 1 -- start start 2024-04-01T01:38:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.971+0000 7f7c0f9f2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c08108750 0x7f7c08117be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.971+0000 7f7c0f9f2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c0810b0c0 0x7f7c08118120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.971+0000 7f7c0f9f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c0810da70 0x7f7c08119160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.971+0000 7f7c0f9f2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7c08110330 con 0x7f7c0810da70 2024-04-01T01:38:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.971+0000 7f7c0f9f2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081101b0 con 0x7f7c0810b0c0 2024-04-01T01:38:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.971+0000 7f7c0f9f2640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081104b0 con 0x7f7c08108750 2024-04-01T01:38:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.971+0000 7f7c0d78e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c08108750 0x7f7c08117be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.971+0000 7f7c0d78e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c08108750 0x7f7c08117be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36018/0 (socket says 172.21.15.70:36018) 2024-04-01T01:38:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.971+0000 7f7c0d78e640 1 -- 172.21.15.70:0/933226477 learned_addr learned my addr 172.21.15.70:0/933226477 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.972+0000 7f7c0d78e640 1 -- 172.21.15.70:0/933226477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c0810b0c0 msgr2=0x7f7c08118120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.972+0000 7f7c0d78e640 1 --2- 172.21.15.70:0/933226477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c0810b0c0 0x7f7c08118120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.972+0000 7f7c0d78e640 1 -- 172.21.15.70:0/933226477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c0810da70 msgr2=0x7f7c08119160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.972+0000 7f7c0d78e640 1 --2- 172.21.15.70:0/933226477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c0810da70 0x7f7c08119160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.972+0000 7f7c0d78e640 1 -- 172.21.15.70:0/933226477 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c081197d0 con 0x7f7c08108750 2024-04-01T01:38:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.973+0000 7f7c0d78e640 1 --2- 172.21.15.70:0/933226477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c08108750 0x7f7c08117be0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8000c00 tx=0x7f7bf802de80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.973+0000 7f7bfe7fc640 1 -- 172.21.15.70:0/933226477 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8043020 con 0x7f7c08108750 2024-04-01T01:38:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.973+0000 7f7bfe7fc640 1 -- 172.21.15.70:0/933226477 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7bf8004950 con 0x7f7c08108750 2024-04-01T01:38:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.973+0000 7f7c0f9f2640 1 -- 172.21.15.70:0/933226477 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c08119a60 con 0x7f7c08108750 2024-04-01T01:38:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.973+0000 7f7bfe7fc640 1 -- 172.21.15.70:0/933226477 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8005220 con 0x7f7c08108750 2024-04-01T01:38:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.974+0000 7f7c0f9f2640 1 -- 172.21.15.70:0/933226477 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c08119ef0 con 0x7f7c08108750 2024-04-01T01:38:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.977+0000 7f7c125cc740 1 Processor -- start 2024-04-01T01:38:09.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.980+0000 7f7c125cc740 1 -- start start 2024-04-01T01:38:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.980+0000 7f7c125cc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b0678d0 0x55785b067cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.980+0000 7f7c125cc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0682c0 0x55785b06cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.981+0000 7f7c125cc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b06d430 0x55785b06f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.981+0000 7f7bfe7fc640 1 -- 172.21.15.70:0/933226477 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f7bf8046020 con 0x7f7c08108750 2024-04-01T01:38:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.981+0000 7f7c125cc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55785afcb8f0 con 0x55785b0682c0 2024-04-01T01:38:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.981+0000 7f7c125cc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55785af748d0 con 0x55785b0678d0 2024-04-01T01:38:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.981+0000 7f7c125cc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55785afc7990 con 0x55785b06d430 2024-04-01T01:38:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.981+0000 7f7c0cf8d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b06d430 0x55785b06f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.981+0000 7f7c0cf8d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b06d430 0x55785b06f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36022/0 (socket says 172.21.15.70:36022) 2024-04-01T01:38:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.981+0000 7f7c0cf8d640 1 -- 172.21.15.70:0/1961488677 learned_addr learned my addr 172.21.15.70:0/1961488677 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:09.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.982+0000 7f7bfe7fc640 1 --2- 172.21.15.70:0/933226477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7bf40a2730 0x7f7bf40a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:09.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.983+0000 7f7c0cf8d640 1 -- 172.21.15.70:0/1961488677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b0678d0 msgr2=0x55785b067cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:09.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.983+0000 7f7c0cf8d640 1 --2- 172.21.15.70:0/1961488677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b0678d0 0x55785b067cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.983+0000 7f7c0cf8d640 1 -- 172.21.15.70:0/1961488677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0682c0 msgr2=0x55785b06cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.983+0000 7f7c0df8f640 1 --2- 172.21.15.70:0/933226477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7bf40a2730 0x7f7bf40a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.983+0000 7f7bfe7fc640 1 -- 172.21.15.70:0/933226477 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f7bf80f0d90 con 0x7f7c08108750 2024-04-01T01:38:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.983+0000 7f7c0cf8d640 1 --2- 172.21.15.70:0/1961488677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0682c0 0x55785b06cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.983+0000 7f7c0cf8d640 1 -- 172.21.15.70:0/1961488677 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55785ace0210 con 0x55785b06d430 2024-04-01T01:38:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.983+0000 7f7c0cf8d640 1 --2- 172.21.15.70:0/1961488677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b06d430 0x55785b06f8c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009b20 tx=0x7f7c0400e350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=100c72b7824d61a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.984+0000 7f7bf37fe640 1 -- 172.21.15.70:0/1961488677 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400ed80 con 0x55785b06d430 2024-04-01T01:38:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.984+0000 7f7bf37fe640 1 -- 172.21.15.70:0/1961488677 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c040108e0 con 0x55785b06d430 2024-04-01T01:38:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.984+0000 7f7bf37fe640 1 -- 172.21.15.70:0/1961488677 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c04018970 con 0x55785b06d430 2024-04-01T01:38:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.985+0000 7f7c125cc740 1 -- 172.21.15.70:0/1961488677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b06d430 msgr2=0x55785b06f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:09.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.985+0000 7f7c125cc740 1 --2- 172.21.15.70:0/1961488677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b06d430 0x55785b06f8c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009b20 tx=0x7f7c0400e350 comp rx=0 tx=0).stop 2024-04-01T01:38:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.985+0000 7f7c0df8f640 1 --2- 172.21.15.70:0/933226477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7bf40a2730 0x7f7bf40a4bf0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x55785b068b20 tx=0x7f7c00009780 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.986+0000 7f7c125cc740 1 -- 172.21.15.70:0/1961488677 shutdown_connections 2024-04-01T01:38:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.986+0000 7f7c125cc740 1 --2- 172.21.15.70:0/1961488677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b06d430 0x55785b06f8c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.986+0000 7f7c125cc740 1 --2- 172.21.15.70:0/1961488677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0682c0 0x55785b06cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.986+0000 7f7c125cc740 1 --2- 172.21.15.70:0/1961488677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b0678d0 0x55785b067cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.987+0000 7f7c125cc740 1 -- 172.21.15.70:0/1961488677 >> 172.21.15.70:0/1961488677 conn(0x55785b062e40 msgr2=0x55785b063240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.987+0000 7f7c125cc740 1 -- 172.21.15.70:0/1961488677 shutdown_connections 2024-04-01T01:38:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.988+0000 7f7c125cc740 1 -- 172.21.15.70:0/1961488677 wait complete. 2024-04-01T01:38:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.988+0000 7f7c125cc740 1 -- start start 2024-04-01T01:38:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.989+0000 7f7c125cc740 1 Processor -- start 2024-04-01T01:38:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.989+0000 7f7c125cc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0678d0 0x55785b071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.989+0000 7f7c125cc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0682c0 0x55785b0721e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.989+0000 7f7c125cc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 0x55785b079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.989+0000 7f7c125cc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55785b070430 con 0x55785b0678d0 2024-04-01T01:38:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.989+0000 7f7c125cc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55785af69f80 con 0x55785b06d430 2024-04-01T01:38:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.989+0000 7f7c125cc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55785b0705b0 con 0x55785b0682c0 2024-04-01T01:38:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.990+0000 7f7c125cc740 10 client.0 _ll_register_callbacks cb 0x55785afc2290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.990+0000 7f7c0cf8d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0682c0 0x55785b0721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.990+0000 7f7c0cf8d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0682c0 0x55785b0721e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36028/0 (socket says 172.21.15.70:36028) 2024-04-01T01:38:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.990+0000 7f7c0cf8d640 1 -- 172.21.15.70:0/500068472 learned_addr learned my addr 172.21.15.70:0/500068472 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.990+0000 7f7c0df8f640 1 --2- 172.21.15.70:0/500068472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 0x55785b079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.991+0000 7f7c0cf8d640 1 -- 172.21.15.70:0/500068472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 msgr2=0x55785b079210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.991+0000 7f7c0cf8d640 1 --2- 172.21.15.70:0/500068472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 0x55785b079210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.991+0000 7f7c0cf8d640 1 -- 172.21.15.70:0/500068472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0678d0 msgr2=0x55785b071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.991+0000 7f7c0cf8d640 1 --2- 172.21.15.70:0/500068472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0678d0 0x55785b071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.991+0000 7f7c0cf8d640 1 -- 172.21.15.70:0/500068472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55785adac620 con 0x55785b0682c0 2024-04-01T01:38:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.991+0000 7f7c0df8f640 1 --2- 172.21.15.70:0/500068472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 0x55785b079210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:38:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.991+0000 7f7c0cf8d640 1 --2- 172.21.15.70:0/500068472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0682c0 0x55785b0721e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400edb0 tx=0x7f7c0400ede0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f73de4d42400b2ac server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.991+0000 7f7bf0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:38:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.993+0000 7f7bf0ff9640 1 -- 172.21.15.70:0/500068472 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c04004030 con 0x55785b0682c0 2024-04-01T01:38:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.993+0000 7f7bf0ff9640 1 -- 172.21.15.70:0/500068472 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f7c04010ca0 con 0x55785b0682c0 2024-04-01T01:38:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.994+0000 7f7c125cc740 1 -- 172.21.15.70:0/500068472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c040108e0 con 0x55785b0682c0 2024-04-01T01:38:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.995+0000 7f7bf0ff9640 1 -- 172.21.15.70:0/500068472 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7c04004030 con 0x55785b0682c0 2024-04-01T01:38:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.995+0000 7f7c125cc740 10 client.26689 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.995+0000 7f7c125cc740 1 -- 172.21.15.70:0/500068472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55785b07c1f0 con 0x55785b0682c0 2024-04-01T01:38:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.995+0000 7f7bf0ff9640 1 -- 172.21.15.70:0/500068472 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c040108e0 con 0x55785b0682c0 2024-04-01T01:38:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.996+0000 7f7bf0ff9640 1 -- 172.21.15.70:0/500068472 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7c040047a0 con 0x55785b0682c0 2024-04-01T01:38:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.996+0000 7f7c125cc740 10 client.26689 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.996+0000 7f7c125cc740 10 client.26689 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.996+0000 7f7c125cc740 20 client.26689 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.996+0000 7f7c125cc740 1 --2- 172.21.15.70:0/500068472 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55785b07dcc0 0x55785b080160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.996+0000 7f7c125cc740 4 client.26689 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.996+0000 7f7c125cc740 1 -- 172.21.15.70:0/500068472 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55785afdf120 con 0x55785b07dcc0 2024-04-01T01:38:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.997+0000 7f7c0d78e640 1 --2- 172.21.15.70:0/500068472 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55785b07dcc0 0x55785b080160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:09.997 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.998+0000 7f7c0d78e640 1 --2- 172.21.15.70:0/500068472 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55785b07dcc0 0x55785b080160 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d164ddd124ca315 server_cookie=20010864c7987ead in_seq=0 out_seq=0 2024-04-01T01:38:09.997 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:09.998+0000 7f7bf0ff9640 10 client.26689 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.000+0000 7f7bf0ff9640 1 -- 172.21.15.70:0/500068472 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55785afdf120 con 0x55785b07dcc0 2024-04-01T01:38:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.000+0000 7f7bf0ff9640 10 client.26689 handle_command_reply: tid=0 2024-04-01T01:38:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.000+0000 7f7c125cc740 1 client.26689 shutdown 2024-04-01T01:38:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.001+0000 7f7c125cc740 1 -- 172.21.15.70:0/500068472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0682c0 msgr2=0x55785b0721e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.001+0000 7f7c125cc740 1 --2- 172.21.15.70:0/500068472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0682c0 0x55785b0721e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400edb0 tx=0x7f7c0400ede0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.003+0000 7f7c125cc740 1 -- 172.21.15.70:0/500068472 shutdown_connections 2024-04-01T01:38:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.003+0000 7f7c125cc740 1 --2- 172.21.15.70:0/500068472 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55785b07dcc0 0x55785b080160 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.003+0000 7f7c125cc740 1 --2- 172.21.15.70:0/500068472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 0x55785b079210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.003+0000 7f7c125cc740 1 --2- 172.21.15.70:0/500068472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0682c0 0x55785b0721e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.003+0000 7f7c125cc740 1 --2- 172.21.15.70:0/500068472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0678d0 0x55785b071ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.003+0000 7f7c125cc740 1 -- 172.21.15.70:0/500068472 >> 172.21.15.70:0/500068472 conn(0x55785b062e40 msgr2=0x55785b061f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.004+0000 7f7bf0ff9640 0 client.26689 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.006+0000 7f7c125cc740 1 -- 172.21.15.70:0/500068472 shutdown_connections 2024-04-01T01:38:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.006+0000 7f7c125cc740 1 -- 172.21.15.70:0/500068472 wait complete. 2024-04-01T01:38:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.007+0000 7f7c125cc740 20 client.26689 trim_cache size 0 max 16384 2024-04-01T01:38:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.045+0000 7f7c125cc740 1 Processor -- start 2024-04-01T01:38:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.046+0000 7f7c125cc740 1 -- start start 2024-04-01T01:38:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.046+0000 7f7c125cc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0678d0 0x55785b06ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.046+0000 7f7c125cc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0682c0 0x55785b079130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.046+0000 7f7c125cc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 0x55785b0660a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.046+0000 7f7c125cc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081c1970 con 0x55785b0682c0 2024-04-01T01:38:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.046+0000 7f7c125cc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081c34b0 con 0x55785b06d430 2024-04-01T01:38:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.046+0000 7f7c125cc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081c2040 con 0x55785b0678d0 2024-04-01T01:38:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.047+0000 7f7c0d78e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0678d0 0x55785b06ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.047+0000 7f7c0d78e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0678d0 0x55785b06ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36040/0 (socket says 172.21.15.70:36040) 2024-04-01T01:38:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.047+0000 7f7c0d78e640 1 -- 172.21.15.70:0/2993436113 learned_addr learned my addr 172.21.15.70:0/2993436113 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.047+0000 7f7c0df8f640 1 --2- 172.21.15.70:0/2993436113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 0x55785b0660a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.047+0000 7f7c0cf8d640 1 --2- 172.21.15.70:0/2993436113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0682c0 0x55785b079130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.048+0000 7f7c0cf8d640 1 -- 172.21.15.70:0/2993436113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0678d0 msgr2=0x55785b06ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.048+0000 7f7c0cf8d640 1 --2- 172.21.15.70:0/2993436113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0678d0 0x55785b06ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.048+0000 7f7c0cf8d640 1 -- 172.21.15.70:0/2993436113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 msgr2=0x55785b0660a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.048+0000 7f7c0cf8d640 1 --2- 172.21.15.70:0/2993436113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 0x55785b0660a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.048+0000 7f7c0cf8d640 1 -- 172.21.15.70:0/2993436113 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55785b068b50 con 0x55785b0682c0 2024-04-01T01:38:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.049+0000 7f7c0cf8d640 1 --2- 172.21.15.70:0/2993436113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0682c0 0x55785b079130 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7c0402b750 tx=0x7f7c04000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b35de002e7686 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.050+0000 7f7bf2ffd640 1 -- 172.21.15.70:0/2993436113 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c04009370 con 0x55785b0682c0 2024-04-01T01:38:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.051+0000 7f7bf2ffd640 1 -- 172.21.15.70:0/2993436113 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c04027070 con 0x55785b0682c0 2024-04-01T01:38:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.051+0000 7f7bf2ffd640 1 -- 172.21.15.70:0/2993436113 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400fbe0 con 0x55785b0682c0 2024-04-01T01:38:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.053+0000 7f7c125cc740 1 -- 172.21.15.70:0/2993436113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0682c0 msgr2=0x55785b079130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.053+0000 7f7c125cc740 1 --2- 172.21.15.70:0/2993436113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0682c0 0x55785b079130 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7c0402b750 tx=0x7f7c04000bc0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.054+0000 7f7c125cc740 1 -- 172.21.15.70:0/2993436113 shutdown_connections 2024-04-01T01:38:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.054+0000 7f7c125cc740 1 --2- 172.21.15.70:0/2993436113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 0x55785b0660a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.054+0000 7f7c125cc740 1 --2- 172.21.15.70:0/2993436113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0682c0 0x55785b079130 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.054+0000 7f7c125cc740 1 --2- 172.21.15.70:0/2993436113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0678d0 0x55785b06ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.054+0000 7f7c125cc740 1 -- 172.21.15.70:0/2993436113 >> 172.21.15.70:0/2993436113 conn(0x55785b062e40 msgr2=0x55785b063220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:10.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.054+0000 7f7c125cc740 1 -- 172.21.15.70:0/2993436113 shutdown_connections 2024-04-01T01:38:10.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.054+0000 7f7c125cc740 1 -- 172.21.15.70:0/2993436113 wait complete. 2024-04-01T01:38:10.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.055+0000 7f7c125cc740 1 -- start start 2024-04-01T01:38:10.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.055+0000 7f7c125cc740 1 Processor -- start 2024-04-01T01:38:10.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.056+0000 7f7c125cc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0678d0 0x55785b079260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.056+0000 7f7c125cc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0682c0 0x55785b075d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.056+0000 7f7c125cc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 0x55785b076330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.056+0000 7f7c0cf8d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0682c0 0x55785b075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.056+0000 7f7c0cf8d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0682c0 0x55785b075d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36056/0 (socket says 172.21.15.70:36056) 2024-04-01T01:38:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.056+0000 7f7c0cf8d640 1 -- 172.21.15.70:0/2790725998 learned_addr learned my addr 172.21.15.70:0/2790725998 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.057+0000 7f7c125cc740 1 -- 172.21.15.70:0/2790725998 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081c2de0 con 0x55785b0678d0 2024-04-01T01:38:10.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.057+0000 7f7c0df8f640 1 --2- 172.21.15.70:0/2790725998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 0x55785b076330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.057+0000 7f7c125cc740 1 -- 172.21.15.70:0/2790725998 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081c2710 con 0x55785b06d430 2024-04-01T01:38:10.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.057+0000 7f7c125cc740 1 -- 172.21.15.70:0/2790725998 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081bf8f0 con 0x55785b0682c0 2024-04-01T01:38:10.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.057+0000 7f7c125cc740 10 client.0 _ll_register_callbacks cb 0x7f7c04009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.058+0000 7f7c0d78e640 1 --2- 172.21.15.70:0/2790725998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0678d0 0x55785b079260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.058+0000 7f7c0df8f640 1 -- 172.21.15.70:0/2790725998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0682c0 msgr2=0x55785b075d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.058+0000 7f7c0df8f640 1 --2- 172.21.15.70:0/2790725998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0682c0 0x55785b075d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.058+0000 7f7c0df8f640 1 -- 172.21.15.70:0/2790725998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0678d0 msgr2=0x55785b079260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.058+0000 7f7c0df8f640 1 --2- 172.21.15.70:0/2790725998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0678d0 0x55785b079260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.058+0000 7f7c0df8f640 1 -- 172.21.15.70:0/2790725998 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55785b0e2100 con 0x55785b06d430 2024-04-01T01:38:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.058+0000 7f7c0df8f640 1 --2- 172.21.15.70:0/2790725998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 0x55785b076330 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009a10 tx=0x7f7c00004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80889ebf10bbef99 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.058+0000 7f7bf0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:10.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.058+0000 7f7bf0ff9640 1 -- 172.21.15.70:0/2790725998 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c00004b20 con 0x55785b06d430 2024-04-01T01:38:10.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.058+0000 7f7c125cc740 1 -- 172.21.15.70:0/2790725998 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c04027070 con 0x55785b06d430 2024-04-01T01:38:10.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.058+0000 7f7bf0ff9640 1 -- 172.21.15.70:0/2790725998 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f7c00004cc0 con 0x55785b06d430 2024-04-01T01:38:10.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.059+0000 7f7bf0ff9640 1 -- 172.21.15.70:0/2790725998 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7c00030050 con 0x55785b06d430 2024-04-01T01:38:10.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.059+0000 7f7c125cc740 10 client.26795 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:10.059 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.059+0000 7f7c125cc740 1 -- 172.21.15.70:0/2790725998 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55785b0e3480 con 0x55785b06d430 2024-04-01T01:38:10.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.059+0000 7f7bf0ff9640 1 -- 172.21.15.70:0/2790725998 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c04027070 con 0x55785b06d430 2024-04-01T01:38:10.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.061+0000 7f7bf0ff9640 1 -- 172.21.15.70:0/2790725998 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7c0000f040 con 0x55785b06d430 2024-04-01T01:38:10.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.061+0000 7f7c125cc740 10 client.26795 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:10.060 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.061+0000 7f7c125cc740 10 client.26795 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.061+0000 7f7c125cc740 20 client.26795 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.061+0000 7f7c125cc740 1 --2- 172.21.15.70:0/2790725998 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55785b0e3e20 0x55785b0e6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.061+0000 7f7c125cc740 4 client.26795 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:38:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.061+0000 7f7c125cc740 1 -- 172.21.15.70:0/2790725998 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55785afd8e30 con 0x55785b0e3e20 2024-04-01T01:38:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.061+0000 7f7c0d78e640 1 --2- 172.21.15.70:0/2790725998 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55785b0e3e20 0x55785b0e6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.062+0000 7f7c0d78e640 1 --2- 172.21.15.70:0/2790725998 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55785b0e3e20 0x55785b0e6240 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7157f18b4407f155 server_cookie=20014cc76aaaaaa6 in_seq=0 out_seq=0 2024-04-01T01:38:10.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.062+0000 7f7bf0ff9640 10 client.26795 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:10.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:09 smithi081 ceph-mon[35142]: pgmap v498: 129 pgs: 129 active+clean; 1.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 8.0 MiB/s wr, 926 op/s 2024-04-01T01:38:10.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.111+0000 7f7bf0ff9640 1 -- 172.21.15.70:0/2790725998 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55785afd8e30 con 0x55785b0e3e20 2024-04-01T01:38:10.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.111+0000 7f7bf0ff9640 10 client.26795 handle_command_reply: tid=0 2024-04-01T01:38:10.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.112+0000 7f7c125cc740 1 client.26795 shutdown 2024-04-01T01:38:10.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.113+0000 7f7c125cc740 1 -- 172.21.15.70:0/2790725998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 msgr2=0x55785b076330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:10.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.113+0000 7f7c125cc740 1 --2- 172.21.15.70:0/2790725998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 0x55785b076330 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009a10 tx=0x7f7c00004a40 comp rx=0 tx=0).stop 2024-04-01T01:38:10.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.113+0000 7f7c125cc740 1 -- 172.21.15.70:0/2790725998 shutdown_connections 2024-04-01T01:38:10.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.113+0000 7f7c125cc740 1 --2- 172.21.15.70:0/2790725998 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55785b0e3e20 0x55785b0e6240 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.113+0000 7f7c125cc740 1 --2- 172.21.15.70:0/2790725998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55785b06d430 0x55785b076330 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.113+0000 7f7c125cc740 1 --2- 172.21.15.70:0/2790725998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55785b0682c0 0x55785b075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.113+0000 7f7c125cc740 1 --2- 172.21.15.70:0/2790725998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55785b0678d0 0x55785b079260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.113+0000 7f7c125cc740 1 -- 172.21.15.70:0/2790725998 >> 172.21.15.70:0/2790725998 conn(0x55785b062e40 msgr2=0x55785b0660a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:10.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.114+0000 7f7bf0ff9640 0 client.26795 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:10.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.114+0000 7f7c125cc740 1 -- 172.21.15.70:0/2790725998 shutdown_connections 2024-04-01T01:38:10.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.115+0000 7f7c125cc740 1 -- 172.21.15.70:0/2790725998 wait complete. 2024-04-01T01:38:10.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.115+0000 7f7c125cc740 20 client.26795 trim_cache size 0 max 16384 2024-04-01T01:38:10.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:38:10.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:38:10.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:38:10.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:38:10.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.119+0000 7f7c0f9f2640 1 -- 172.21.15.70:0/933226477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7bf40a2730 msgr2=0x7f7bf40a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:10.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.119+0000 7f7c0f9f2640 1 --2- 172.21.15.70:0/933226477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7bf40a2730 0x7f7bf40a4bf0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x55785b068b20 tx=0x7f7c00009780 comp rx=0 tx=0).stop 2024-04-01T01:38:10.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.119+0000 7f7c0f9f2640 1 -- 172.21.15.70:0/933226477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c08108750 msgr2=0x7f7c08117be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:10.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.119+0000 7f7c0f9f2640 1 --2- 172.21.15.70:0/933226477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c08108750 0x7f7c08117be0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8000c00 tx=0x7f7bf802de80 comp rx=0 tx=0).stop 2024-04-01T01:38:10.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.120+0000 7f7c0f9f2640 1 -- 172.21.15.70:0/933226477 shutdown_connections 2024-04-01T01:38:10.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.120+0000 7f7c0f9f2640 1 --2- 172.21.15.70:0/933226477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c0810da70 0x7f7c08119160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.120+0000 7f7c0f9f2640 1 --2- 172.21.15.70:0/933226477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c0810b0c0 0x7f7c08118120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.120+0000 7f7c0f9f2640 1 --2- 172.21.15.70:0/933226477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7bf40a2730 0x7f7bf40a4bf0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.120+0000 7f7c0f9f2640 1 --2- 172.21.15.70:0/933226477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c08108750 0x7f7c08117be0 secure :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8000c00 tx=0x7f7bf802de80 comp rx=0 tx=0).stop 2024-04-01T01:38:10.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.120+0000 7f7c0f9f2640 1 -- 172.21.15.70:0/933226477 >> 172.21.15.70:0/933226477 conn(0x7f7c0806e790 msgr2=0x7f7c08108ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:10.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.120+0000 7f7c0f9f2640 1 -- 172.21.15.70:0/933226477 shutdown_connections 2024-04-01T01:38:10.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:10.121+0000 7f7c0f9f2640 1 -- 172.21.15.70:0/933226477 wait complete. 2024-04-01T01:38:10.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:09 smithi189 ceph-mon[35753]: pgmap v498: 129 pgs: 129 active+clean; 1.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 8.0 MiB/s wr, 926 op/s 2024-04-01T01:38:10.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:09 smithi070 ceph-mon[32179]: pgmap v498: 129 pgs: 129 active+clean; 1.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 8.0 MiB/s wr, 926 op/s 2024-04-01T01:38:10.467 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:38:10.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:38:10.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.650+0000 7fd98bfff640 1 -- 172.21.15.70:0/1479527313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd98c108e20 msgr2=0x7fd98c1092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:10.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.650+0000 7fd98bfff640 1 --2- 172.21.15.70:0/1479527313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd98c108e20 0x7fd98c1092a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd98401acb0 tx=0x7fd98403d130 comp rx=0 tx=0).stop 2024-04-01T01:38:10.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.651+0000 7fd98bfff640 1 -- 172.21.15.70:0/1479527313 shutdown_connections 2024-04-01T01:38:10.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.651+0000 7fd98bfff640 1 --2- 172.21.15.70:0/1479527313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd98c1097e0 0x7fd98c10e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.651+0000 7fd98bfff640 1 --2- 172.21.15.70:0/1479527313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd98c108e20 0x7fd98c1092a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.651+0000 7fd98bfff640 1 --2- 172.21.15.70:0/1479527313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd98c072b60 0x7fd98c072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.651+0000 7fd98bfff640 1 -- 172.21.15.70:0/1479527313 >> 172.21.15.70:0/1479527313 conn(0x7fd98c06e630 msgr2=0x7fd98c070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:10.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.652+0000 7fd98bfff640 1 -- 172.21.15.70:0/1479527313 shutdown_connections 2024-04-01T01:38:10.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.652+0000 7fd98bfff640 1 -- 172.21.15.70:0/1479527313 wait complete. 2024-04-01T01:38:10.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.653+0000 7fd98bfff640 1 Processor -- start 2024-04-01T01:38:10.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.653+0000 7fd98bfff640 1 -- start start 2024-04-01T01:38:10.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.653+0000 7fd98bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd98c072b60 0x7fd98c19df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.653+0000 7fd98bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd98c108e20 0x7fd98c19e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.654+0000 7fd98bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd98c1097e0 0x7fd98c1a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.654+0000 7fd98bfff640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c111900 con 0x7fd98c072b60 2024-04-01T01:38:10.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.654+0000 7fd98bfff640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c111780 con 0x7fd98c1097e0 2024-04-01T01:38:10.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.654+0000 7fd98bfff640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c111a80 con 0x7fd98c108e20 2024-04-01T01:38:10.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.654+0000 7fd98a7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd98c108e20 0x7fd98c19e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.654+0000 7fd98a7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd98c108e20 0x7fd98c19e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36064/0 (socket says 172.21.15.70:36064) 2024-04-01T01:38:10.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.654+0000 7fd98a7fc640 1 -- 172.21.15.70:0/1554903216 learned_addr learned my addr 172.21.15.70:0/1554903216 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:10.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.654+0000 7fd98affd640 1 --2- 172.21.15.70:0/1554903216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd98c072b60 0x7fd98c19df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.654+0000 7fd98a7fc640 1 -- 172.21.15.70:0/1554903216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd98c1097e0 msgr2=0x7fd98c1a54d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:38:10.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.654+0000 7fd98a7fc640 1 --2- 172.21.15.70:0/1554903216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd98c1097e0 0x7fd98c1a54d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.654+0000 7fd98a7fc640 1 -- 172.21.15.70:0/1554903216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd98c072b60 msgr2=0x7fd98c19df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:10.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.654+0000 7fd98a7fc640 1 --2- 172.21.15.70:0/1554903216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd98c072b60 0x7fd98c19df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.654+0000 7fd98a7fc640 1 -- 172.21.15.70:0/1554903216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98c1a5b40 con 0x7fd98c108e20 2024-04-01T01:38:10.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.655+0000 7fd98a7fc640 1 --2- 172.21.15.70:0/1554903216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd98c108e20 0x7fd98c19e460 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd984000f80 tx=0x7fd98401a3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:10.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.655+0000 7fd973fff640 1 -- 172.21.15.70:0/1554903216 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98404a040 con 0x7fd98c108e20 2024-04-01T01:38:10.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.656+0000 7fd98bfff640 1 -- 172.21.15.70:0/1554903216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98c1a5dd0 con 0x7fd98c108e20 2024-04-01T01:38:10.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.656+0000 7fd98bfff640 1 -- 172.21.15.70:0/1554903216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd98c1a61b0 con 0x7fd98c108e20 2024-04-01T01:38:10.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.656+0000 7fd973fff640 1 -- 172.21.15.70:0/1554903216 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd984055070 con 0x7fd98c108e20 2024-04-01T01:38:10.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.656+0000 7fd973fff640 1 -- 172.21.15.70:0/1554903216 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9840456a0 con 0x7fd98c108e20 2024-04-01T01:38:10.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.660+0000 7fd993fa2740 1 Processor -- start 2024-04-01T01:38:10.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.660+0000 7fd993fa2740 1 -- start start 2024-04-01T01:38:10.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.660+0000 7fd993fa2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039daef490 0x56039daef8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.660+0000 7fd993fa2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039daefe80 0x56039daf4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.660+0000 7fd993fa2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daf4ff0 0x56039daf7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.660+0000 7fd993fa2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56039da54180 con 0x56039daef490 2024-04-01T01:38:10.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.660+0000 7fd993fa2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56039d9fcb70 con 0x56039daf4ff0 2024-04-01T01:38:10.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.660+0000 7fd993fa2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56039da50220 con 0x56039daefe80 2024-04-01T01:38:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.660+0000 7fd98b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039daef490 0x56039daef8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.661+0000 7fd98a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daf4ff0 0x56039daf7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.661+0000 7fd98a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daf4ff0 0x56039daf7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47848/0 (socket says 172.21.15.70:47848) 2024-04-01T01:38:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.661+0000 7fd98a7fc640 1 -- 172.21.15.70:0/395286398 learned_addr learned my addr 172.21.15.70:0/395286398 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.661+0000 7fd98a7fc640 1 -- 172.21.15.70:0/395286398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039daefe80 msgr2=0x56039daf4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.661+0000 7fd98a7fc640 1 --2- 172.21.15.70:0/395286398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039daefe80 0x56039daf4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.661+0000 7fd973fff640 1 -- 172.21.15.70:0/1554903216 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fd984045840 con 0x7fd98c108e20 2024-04-01T01:38:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.661+0000 7fd98a7fc640 1 -- 172.21.15.70:0/395286398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039daef490 msgr2=0x56039daef8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.661+0000 7fd98b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039daef490 0x56039daef8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52556/0 (socket says 172.21.15.70:52556) 2024-04-01T01:38:10.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.661+0000 7fd98a7fc640 1 --2- 172.21.15.70:0/395286398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039daef490 0x56039daef8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.661+0000 7fd98a7fc640 1 -- 172.21.15.70:0/395286398 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56039da107c0 con 0x56039daf4ff0 2024-04-01T01:38:10.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.662+0000 7fd98b7fe640 1 --2- 172.21.15.70:0/395286398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039daef490 0x56039daef8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:38:10.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.662+0000 7fd98a7fc640 1 --2- 172.21.15.70:0/395286398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daf4ff0 0x56039daf7480 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd984105d10 tx=0x7fd984105e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de38a898baaa7ba9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:10.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.663+0000 7fd9717fa640 1 -- 172.21.15.70:0/395286398 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984117070 con 0x56039daf4ff0 2024-04-01T01:38:10.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.663+0000 7fd9717fa640 1 -- 172.21.15.70:0/395286398 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9841063d0 con 0x56039daf4ff0 2024-04-01T01:38:10.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.663+0000 7fd9717fa640 1 -- 172.21.15.70:0/395286398 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98410f570 con 0x56039daf4ff0 2024-04-01T01:38:10.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.663+0000 7fd973fff640 1 --2- 172.21.15.70:0/1554903216 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9740a2670 0x7fd9740a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.664+0000 7fd973fff640 1 -- 172.21.15.70:0/1554903216 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd984101db0 con 0x7fd98c108e20 2024-04-01T01:38:10.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.664+0000 7fd993fa2740 1 -- 172.21.15.70:0/395286398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daf4ff0 msgr2=0x56039daf7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:10.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.664+0000 7fd993fa2740 1 --2- 172.21.15.70:0/395286398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daf4ff0 0x56039daf7480 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd984105d10 tx=0x7fd984105e00 comp rx=0 tx=0).stop 2024-04-01T01:38:10.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.665+0000 7fd98b7fe640 1 --2- 172.21.15.70:0/1554903216 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9740a2670 0x7fd9740a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.665+0000 7fd993fa2740 1 -- 172.21.15.70:0/395286398 shutdown_connections 2024-04-01T01:38:10.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.665+0000 7fd993fa2740 1 --2- 172.21.15.70:0/395286398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daf4ff0 0x56039daf7480 secure :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd984105d10 tx=0x7fd984105e00 comp rx=0 tx=0).stop 2024-04-01T01:38:10.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.665+0000 7fd993fa2740 1 --2- 172.21.15.70:0/395286398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039daefe80 0x56039daf4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.665+0000 7fd993fa2740 1 --2- 172.21.15.70:0/395286398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039daef490 0x56039daef8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.665+0000 7fd993fa2740 1 -- 172.21.15.70:0/395286398 >> 172.21.15.70:0/395286398 conn(0x56039daeaa00 msgr2=0x56039daeae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:10.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.666+0000 7fd993fa2740 1 -- 172.21.15.70:0/395286398 shutdown_connections 2024-04-01T01:38:10.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.666+0000 7fd993fa2740 1 -- 172.21.15.70:0/395286398 wait complete. 2024-04-01T01:38:10.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.666+0000 7fd98b7fe640 1 --2- 172.21.15.70:0/1554903216 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9740a2670 0x7fd9740a4b30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x56039daf06e0 tx=0x7fd980008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:10.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.666+0000 7fd993fa2740 1 -- start start 2024-04-01T01:38:10.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.666+0000 7fd993fa2740 1 Processor -- start 2024-04-01T01:38:10.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.670+0000 7fd993fa2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039daef490 0x56039daf9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.670+0000 7fd993fa2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daefe80 0x56039daf9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.670+0000 7fd993fa2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039dafee90 0x56039db01280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.671+0000 7fd993fa2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56039daf7d60 con 0x56039dafee90 2024-04-01T01:38:10.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.671+0000 7fd98affd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039daef490 0x56039daf9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.671+0000 7fd98a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daefe80 0x56039daf9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.671+0000 7fd993fa2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56039d9f24a0 con 0x56039daefe80 2024-04-01T01:38:10.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.671+0000 7fd993fa2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56039daf7ee0 con 0x56039daef490 2024-04-01T01:38:10.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.671+0000 7fd98a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daefe80 0x56039daf9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47860/0 (socket says 172.21.15.70:47860) 2024-04-01T01:38:10.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.671+0000 7fd98a7fc640 1 -- 172.21.15.70:0/1211290128 learned_addr learned my addr 172.21.15.70:0/1211290128 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:10.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.671+0000 7fd993fa2740 10 client.0 _ll_register_callbacks cb 0x56039d9f5d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:10.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.671+0000 7fd98affd640 1 -- 172.21.15.70:0/1211290128 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daefe80 msgr2=0x56039daf9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:10.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.671+0000 7fd98affd640 1 --2- 172.21.15.70:0/1211290128 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daefe80 0x56039daf9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.671+0000 7fd98affd640 1 -- 172.21.15.70:0/1211290128 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039dafee90 msgr2=0x56039db01280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:10.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.671+0000 7fd98affd640 1 --2- 172.21.15.70:0/1211290128 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039dafee90 0x56039db01280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.671+0000 7fd98affd640 1 -- 172.21.15.70:0/1211290128 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56039d8327d0 con 0x56039daef490 2024-04-01T01:38:10.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.672+0000 7fd98affd640 1 --2- 172.21.15.70:0/1211290128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039daef490 0x56039daf9990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x56039daf0280 tx=0x7fd97c00f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a96065e14a1a6bf server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:10.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.672+0000 7fd96eff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:38:10.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.672+0000 7fd96eff5640 1 -- 172.21.15.70:0/1211290128 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c015070 con 0x56039daef490 2024-04-01T01:38:10.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.672+0000 7fd96eff5640 1 -- 172.21.15.70:0/1211290128 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd97c004220 con 0x56039daef490 2024-04-01T01:38:10.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.672+0000 7fd993fa2740 1 -- 172.21.15.70:0/1211290128 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9841063d0 con 0x56039daef490 2024-04-01T01:38:10.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.675+0000 7fd96eff5640 1 -- 172.21.15.70:0/1211290128 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd97c004420 con 0x56039daef490 2024-04-01T01:38:10.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.676+0000 7fd993fa2740 10 client.26704 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:10.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.676+0000 7fd993fa2740 1 -- 172.21.15.70:0/1211290128 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56039d7683f0 con 0x56039daef490 2024-04-01T01:38:10.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.677+0000 7fd96eff5640 1 -- 172.21.15.70:0/1211290128 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9841063d0 con 0x56039daef490 2024-04-01T01:38:10.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.677+0000 7fd96eff5640 1 -- 172.21.15.70:0/1211290128 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd97c011600 con 0x56039daef490 2024-04-01T01:38:10.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.678+0000 7fd993fa2740 10 client.26704 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:10.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.678+0000 7fd993fa2740 10 client.26704 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:10.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.678+0000 7fd993fa2740 20 client.26704 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:10.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.678+0000 7fd993fa2740 1 --2- 172.21.15.70:0/1211290128 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56039db05b00 0x56039db07f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.678+0000 7fd993fa2740 4 client.26704 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:10.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.678+0000 7fd993fa2740 1 -- 172.21.15.70:0/1211290128 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56039da616c0 con 0x56039db05b00 2024-04-01T01:38:10.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.679+0000 7fd98a7fc640 1 --2- 172.21.15.70:0/1211290128 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56039db05b00 0x56039db07f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.683+0000 7fd98a7fc640 1 --2- 172.21.15.70:0/1211290128 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56039db05b00 0x56039db07f90 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a9c59cab340a4 server_cookie=bbd75c2c806e383b in_seq=0 out_seq=0 2024-04-01T01:38:10.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.683+0000 7fd96eff5640 10 client.26704 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:10.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.688+0000 7fd96eff5640 1 -- 172.21.15.70:0/1211290128 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56039da616c0 con 0x56039db05b00 2024-04-01T01:38:10.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.688+0000 7fd96eff5640 10 client.26704 handle_command_reply: tid=0 2024-04-01T01:38:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.688+0000 7fd993fa2740 1 client.26704 shutdown 2024-04-01T01:38:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.689+0000 7fd993fa2740 1 -- 172.21.15.70:0/1211290128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039daef490 msgr2=0x56039daf9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.689+0000 7fd993fa2740 1 --2- 172.21.15.70:0/1211290128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039daef490 0x56039daf9990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x56039daf0280 tx=0x7fd97c00f730 comp rx=0 tx=0).stop 2024-04-01T01:38:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.690+0000 7fd993fa2740 1 -- 172.21.15.70:0/1211290128 shutdown_connections 2024-04-01T01:38:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.690+0000 7fd993fa2740 1 --2- 172.21.15.70:0/1211290128 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56039db05b00 0x56039db07f90 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.690+0000 7fd993fa2740 1 --2- 172.21.15.70:0/1211290128 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039dafee90 0x56039db01280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.690+0000 7fd993fa2740 1 --2- 172.21.15.70:0/1211290128 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daefe80 0x56039daf9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.690+0000 7fd993fa2740 1 --2- 172.21.15.70:0/1211290128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039daef490 0x56039daf9990 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.690+0000 7fd993fa2740 1 -- 172.21.15.70:0/1211290128 >> 172.21.15.70:0/1211290128 conn(0x56039daeaa00 msgr2=0x56039dae9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:10.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.691+0000 7fd96eff5640 0 client.26704 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:10.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.691+0000 7fd993fa2740 1 -- 172.21.15.70:0/1211290128 shutdown_connections 2024-04-01T01:38:10.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.691+0000 7fd993fa2740 1 -- 172.21.15.70:0/1211290128 wait complete. 2024-04-01T01:38:10.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.693+0000 7fd993fa2740 20 client.26704 trim_cache size 0 max 16384 2024-04-01T01:38:10.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.741+0000 7fd993fa2740 1 Processor -- start 2024-04-01T01:38:10.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.741+0000 7fd993fa2740 1 -- start start 2024-04-01T01:38:10.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.741+0000 7fd993fa2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daef490 0x56039dae9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.741+0000 7fd993fa2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039daefe80 0x56039dae98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.742+0000 7fd993fa2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039dafee90 0x56039daf8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.742+0000 7fd993fa2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c1b0890 con 0x56039daefe80 2024-04-01T01:38:10.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.742+0000 7fd993fa2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c1b23d0 con 0x56039daef490 2024-04-01T01:38:10.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.742+0000 7fd993fa2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c1b0f60 con 0x56039dafee90 2024-04-01T01:38:10.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.742+0000 7fd98b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039dafee90 0x56039daf8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.742+0000 7fd98b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039dafee90 0x56039daf8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36086/0 (socket says 172.21.15.70:36086) 2024-04-01T01:38:10.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.742+0000 7fd98b7fe640 1 -- 172.21.15.70:0/1896335048 learned_addr learned my addr 172.21.15.70:0/1896335048 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:10.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.743+0000 7fd98b7fe640 1 -- 172.21.15.70:0/1896335048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daef490 msgr2=0x56039dae9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:10.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.743+0000 7fd98b7fe640 1 --2- 172.21.15.70:0/1896335048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daef490 0x56039dae9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.743+0000 7fd98b7fe640 1 -- 172.21.15.70:0/1896335048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039daefe80 msgr2=0x56039dae98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:10.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.743+0000 7fd98b7fe640 1 --2- 172.21.15.70:0/1896335048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039daefe80 0x56039dae98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.743+0000 7fd98b7fe640 1 -- 172.21.15.70:0/1896335048 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56039d7280f0 con 0x56039dafee90 2024-04-01T01:38:10.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.743+0000 7fd98b7fe640 1 --2- 172.21.15.70:0/1896335048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039dafee90 0x56039daf8fb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd98000ca60 tx=0x7fd9800046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=362b017f2a336838 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:10.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.743+0000 7fd970ff9640 1 -- 172.21.15.70:0/1896335048 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd980023070 con 0x56039dafee90 2024-04-01T01:38:10.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.744+0000 7fd970ff9640 1 -- 172.21.15.70:0/1896335048 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd980013d30 con 0x56039dafee90 2024-04-01T01:38:10.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.744+0000 7fd970ff9640 1 -- 172.21.15.70:0/1896335048 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd980011630 con 0x56039dafee90 2024-04-01T01:38:10.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.745+0000 7fd993fa2740 1 -- 172.21.15.70:0/1896335048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039dafee90 msgr2=0x56039daf8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:10.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.745+0000 7fd993fa2740 1 --2- 172.21.15.70:0/1896335048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039dafee90 0x56039daf8fb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd98000ca60 tx=0x7fd9800046c0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.745+0000 7fd993fa2740 1 -- 172.21.15.70:0/1896335048 shutdown_connections 2024-04-01T01:38:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.745+0000 7fd993fa2740 1 --2- 172.21.15.70:0/1896335048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039dafee90 0x56039daf8fb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.745+0000 7fd993fa2740 1 --2- 172.21.15.70:0/1896335048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039daefe80 0x56039dae98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.745+0000 7fd993fa2740 1 --2- 172.21.15.70:0/1896335048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039daef490 0x56039dae9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.745+0000 7fd993fa2740 1 -- 172.21.15.70:0/1896335048 >> 172.21.15.70:0/1896335048 conn(0x56039daeaa00 msgr2=0x56039daf81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.745+0000 7fd993fa2740 1 -- 172.21.15.70:0/1896335048 shutdown_connections 2024-04-01T01:38:10.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.746+0000 7fd993fa2740 1 -- 172.21.15.70:0/1896335048 wait complete. 2024-04-01T01:38:10.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.746+0000 7fd993fa2740 1 -- start start 2024-04-01T01:38:10.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.746+0000 7fd993fa2740 1 Processor -- start 2024-04-01T01:38:10.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.747+0000 7fd993fa2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039daef490 0x56039db167c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.747+0000 7fd993fa2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039daefe80 0x56039db16d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.747+0000 7fd993fa2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039dafee90 0x56039db17ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.747+0000 7fd993fa2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c1b1d00 con 0x56039daef490 2024-04-01T01:38:10.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.747+0000 7fd993fa2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c1b1630 con 0x56039dafee90 2024-04-01T01:38:10.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.747+0000 7fd993fa2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c1ae870 con 0x56039daefe80 2024-04-01T01:38:10.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.748+0000 7fd98b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039dafee90 0x56039db17ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.748+0000 7fd98b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039dafee90 0x56039db17ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47884/0 (socket says 172.21.15.70:47884) 2024-04-01T01:38:10.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.748+0000 7fd98b7fe640 1 -- 172.21.15.70:0/3141025 learned_addr learned my addr 172.21.15.70:0/3141025 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:10.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.748+0000 7fd993fa2740 10 client.0 _ll_register_callbacks cb 0x7fd97c005f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:10.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.748+0000 7fd98affd640 1 --2- 172.21.15.70:0/3141025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039daef490 0x56039db167c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.749+0000 7fd98a7fc640 1 --2- 172.21.15.70:0/3141025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039daefe80 0x56039db16d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.749+0000 7fd98b7fe640 1 -- 172.21.15.70:0/3141025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039daefe80 msgr2=0x56039db16d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:10.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.749+0000 7fd98b7fe640 1 --2- 172.21.15.70:0/3141025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039daefe80 0x56039db16d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.749+0000 7fd98b7fe640 1 -- 172.21.15.70:0/3141025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039daef490 msgr2=0x56039db167c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:10.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.749+0000 7fd98b7fe640 1 --2- 172.21.15.70:0/3141025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039daef490 0x56039db167c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.749+0000 7fd98b7fe640 1 -- 172.21.15.70:0/3141025 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56039db18290 con 0x56039dafee90 2024-04-01T01:38:10.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.749+0000 7fd98b7fe640 1 --2- 172.21.15.70:0/3141025 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039dafee90 0x56039db17ca0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd980004810 tx=0x7fd98001e6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13bfe851695b31f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:10.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.749+0000 7fd96eff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:10.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.750+0000 7fd96eff5640 1 -- 172.21.15.70:0/3141025 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd980023050 con 0x56039dafee90 2024-04-01T01:38:10.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.750+0000 7fd96eff5640 1 -- 172.21.15.70:0/3141025 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd98002a450 con 0x56039dafee90 2024-04-01T01:38:10.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.750+0000 7fd993fa2740 1 -- 172.21.15.70:0/3141025 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd980013d30 con 0x56039dafee90 2024-04-01T01:38:10.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.750+0000 7fd96eff5640 1 -- 172.21.15.70:0/3141025 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd98002a650 con 0x56039dafee90 2024-04-01T01:38:10.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.750+0000 7fd993fa2740 10 client.26807 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:10.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.751+0000 7fd993fa2740 1 -- 172.21.15.70:0/3141025 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56039db6b820 con 0x56039dafee90 2024-04-01T01:38:10.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.751+0000 7fd96eff5640 1 -- 172.21.15.70:0/3141025 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd980013d30 con 0x56039dafee90 2024-04-01T01:38:10.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.751+0000 7fd96eff5640 1 -- 172.21.15.70:0/3141025 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd980034750 con 0x56039dafee90 2024-04-01T01:38:10.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.751+0000 7fd993fa2740 10 client.26807 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:10.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.751+0000 7fd993fa2740 10 client.26807 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:10.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.751+0000 7fd993fa2740 20 client.26807 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:10.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.751+0000 7fd993fa2740 1 --2- 172.21.15.70:0/3141025 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56039db6c170 0x56039db6e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.751+0000 7fd993fa2740 4 client.26807 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:38:10.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.751+0000 7fd993fa2740 1 -- 172.21.15.70:0/3141025 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56039da5ba10 con 0x56039db6c170 2024-04-01T01:38:10.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.751+0000 7fd98affd640 1 --2- 172.21.15.70:0/3141025 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56039db6c170 0x56039db6e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.752+0000 7fd98affd640 1 --2- 172.21.15.70:0/3141025 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56039db6c170 0x56039db6e590 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71ffd21e5c7a157f server_cookie=50184bcd15d12903 in_seq=0 out_seq=0 2024-04-01T01:38:10.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.752+0000 7fd96eff5640 10 client.26807 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:10.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.752+0000 7fd96eff5640 1 -- 172.21.15.70:0/3141025 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56039da5ba10 con 0x56039db6c170 2024-04-01T01:38:10.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.752+0000 7fd96eff5640 10 client.26807 handle_command_reply: tid=0 2024-04-01T01:38:10.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.752+0000 7fd993fa2740 1 client.26807 shutdown 2024-04-01T01:38:10.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.754+0000 7fd993fa2740 1 -- 172.21.15.70:0/3141025 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039dafee90 msgr2=0x56039db17ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:10.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.754+0000 7fd993fa2740 1 --2- 172.21.15.70:0/3141025 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039dafee90 0x56039db17ca0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd980004810 tx=0x7fd98001e6b0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.754+0000 7fd993fa2740 1 -- 172.21.15.70:0/3141025 shutdown_connections 2024-04-01T01:38:10.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.754+0000 7fd993fa2740 1 --2- 172.21.15.70:0/3141025 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56039db6c170 0x56039db6e590 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.754+0000 7fd993fa2740 1 --2- 172.21.15.70:0/3141025 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56039dafee90 0x56039db17ca0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.754+0000 7fd993fa2740 1 --2- 172.21.15.70:0/3141025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56039daefe80 0x56039db16d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.754+0000 7fd96eff5640 0 client.26807 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:10.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.754+0000 7fd993fa2740 1 --2- 172.21.15.70:0/3141025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56039daef490 0x56039db167c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.754+0000 7fd993fa2740 1 -- 172.21.15.70:0/3141025 >> 172.21.15.70:0/3141025 conn(0x56039daeaa00 msgr2=0x56039daedc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:10.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.754+0000 7fd993fa2740 1 -- 172.21.15.70:0/3141025 shutdown_connections 2024-04-01T01:38:10.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.754+0000 7fd993fa2740 1 -- 172.21.15.70:0/3141025 wait complete. 2024-04-01T01:38:10.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.754+0000 7fd993fa2740 20 client.26807 trim_cache size 0 max 16384 2024-04-01T01:38:10.754 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:38:10.755 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:38:10.755 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:38:10.755 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:38:10.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.755+0000 7fd98bfff640 1 -- 172.21.15.70:0/1554903216 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9740a2670 msgr2=0x7fd9740a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:10.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.755+0000 7fd98bfff640 1 --2- 172.21.15.70:0/1554903216 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9740a2670 0x7fd9740a4b30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x56039daf06e0 tx=0x7fd980008040 comp rx=0 tx=0).stop 2024-04-01T01:38:10.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.755+0000 7fd98bfff640 1 -- 172.21.15.70:0/1554903216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd98c108e20 msgr2=0x7fd98c19e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:10.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.755+0000 7fd98bfff640 1 --2- 172.21.15.70:0/1554903216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd98c108e20 0x7fd98c19e460 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd984000f80 tx=0x7fd98401a3e0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.756+0000 7fd98bfff640 1 -- 172.21.15.70:0/1554903216 shutdown_connections 2024-04-01T01:38:10.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.756+0000 7fd98bfff640 1 --2- 172.21.15.70:0/1554903216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd98c1097e0 0x7fd98c1a54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.756+0000 7fd98bfff640 1 --2- 172.21.15.70:0/1554903216 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9740a2670 0x7fd9740a4b30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.756+0000 7fd98bfff640 1 --2- 172.21.15.70:0/1554903216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd98c108e20 0x7fd98c19e460 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.756+0000 7fd98bfff640 1 --2- 172.21.15.70:0/1554903216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd98c072b60 0x7fd98c19df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.756+0000 7fd98bfff640 1 -- 172.21.15.70:0/1554903216 >> 172.21.15.70:0/1554903216 conn(0x7fd98c06e630 msgr2=0x7fd98c10c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:10.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.756+0000 7fd98bfff640 1 -- 172.21.15.70:0/1554903216 shutdown_connections 2024-04-01T01:38:10.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.756+0000 7fd98bfff640 1 -- 172.21.15.70:0/1554903216 wait complete. 2024-04-01T01:38:10.774 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:38:10.774 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:38:10.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.949+0000 7f8965e6b640 1 -- 172.21.15.70:0/1387546402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8960102220 msgr2=0x7f8960108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:10.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.949+0000 7f8965e6b640 1 --2- 172.21.15.70:0/1387546402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8960102220 0x7f8960108ab0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8958009ca0 tx=0x7f895802c120 comp rx=0 tx=0).stop 2024-04-01T01:38:10.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.949+0000 7f8965e6b640 1 -- 172.21.15.70:0/1387546402 shutdown_connections 2024-04-01T01:38:10.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.949+0000 7f8965e6b640 1 --2- 172.21.15.70:0/1387546402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8960102220 0x7f8960108ab0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.949+0000 7f8965e6b640 1 --2- 172.21.15.70:0/1387546402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8960101860 0x7f8960101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.949+0000 7f8965e6b640 1 --2- 172.21.15.70:0/1387546402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8960100670 0x7f8960100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.949+0000 7f8965e6b640 1 -- 172.21.15.70:0/1387546402 >> 172.21.15.70:0/1387546402 conn(0x7f89600fbbe0 msgr2=0x7f89600fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:10.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.950+0000 7f8965e6b640 1 -- 172.21.15.70:0/1387546402 shutdown_connections 2024-04-01T01:38:10.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.950+0000 7f8965e6b640 1 -- 172.21.15.70:0/1387546402 wait complete. 2024-04-01T01:38:10.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.950+0000 7f8965e6b640 1 Processor -- start 2024-04-01T01:38:10.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.951+0000 7f8965e6b640 1 -- start start 2024-04-01T01:38:10.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.951+0000 7f8965e6b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8960100670 0x7f8960070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.951+0000 7f8965e6b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8960101860 0x7f8960070570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.951+0000 7f8965e6b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8960102220 0x7f89601a58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.951+0000 7f8965e6b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f896006b780 con 0x7f8960101860 2024-04-01T01:38:10.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.951+0000 7f8965e6b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f896006b600 con 0x7f8960100670 2024-04-01T01:38:10.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.951+0000 7f8965e6b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f896006b900 con 0x7f8960102220 2024-04-01T01:38:10.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.951+0000 7f895f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8960100670 0x7f8960070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.951+0000 7f895ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8960102220 0x7f89601a58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.951+0000 7f895f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8960100670 0x7f8960070030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47910/0 (socket says 172.21.15.70:47910) 2024-04-01T01:38:10.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.951+0000 7f895f7fe640 1 -- 172.21.15.70:0/1213960880 learned_addr learned my addr 172.21.15.70:0/1213960880 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:10.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.951+0000 7f895effd640 1 --2- 172.21.15.70:0/1213960880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8960101860 0x7f8960070570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.952+0000 7f895ffff640 1 -- 172.21.15.70:0/1213960880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8960100670 msgr2=0x7f8960070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:10.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.952+0000 7f895ffff640 1 --2- 172.21.15.70:0/1213960880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8960100670 0x7f8960070030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.952+0000 7f895ffff640 1 -- 172.21.15.70:0/1213960880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8960101860 msgr2=0x7f8960070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:10.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.952+0000 7f895ffff640 1 --2- 172.21.15.70:0/1213960880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8960101860 0x7f8960070570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:10.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.952+0000 7f895ffff640 1 -- 172.21.15.70:0/1213960880 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89600736f0 con 0x7f8960102220 2024-04-01T01:38:10.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.952+0000 7f895f7fe640 1 --2- 172.21.15.70:0/1213960880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8960100670 0x7f8960070030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:38:10.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.952+0000 7f895effd640 1 --2- 172.21.15.70:0/1213960880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8960101860 0x7f8960070570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:38:10.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.952+0000 7f895ffff640 1 --2- 172.21.15.70:0/1213960880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8960102220 0x7f89601a58b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8958009bf0 tx=0x7f8958003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:10.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.953+0000 7f895cff9640 1 -- 172.21.15.70:0/1213960880 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8958039040 con 0x7f8960102220 2024-04-01T01:38:10.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.953+0000 7f895cff9640 1 -- 172.21.15.70:0/1213960880 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8958003c90 con 0x7f8960102220 2024-04-01T01:38:10.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.953+0000 7f895cff9640 1 -- 172.21.15.70:0/1213960880 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8958039040 con 0x7f8960102220 2024-04-01T01:38:10.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.953+0000 7f8965e6b640 1 -- 172.21.15.70:0/1213960880 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89601a5df0 con 0x7f8960102220 2024-04-01T01:38:10.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.953+0000 7f8965e6b640 1 -- 172.21.15.70:0/1213960880 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89601a6290 con 0x7f8960102220 2024-04-01T01:38:10.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.955+0000 7f8965e6b640 1 -- 172.21.15.70:0/1213960880 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89601a65b0 con 0x7f8960102220 2024-04-01T01:38:10.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.957+0000 7f895cff9640 1 -- 172.21.15.70:0/1213960880 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f895806c9d0 con 0x7f8960102220 2024-04-01T01:38:10.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.958+0000 7f895cff9640 1 --2- 172.21.15.70:0/1213960880 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f89480a28e0 0x7f89480a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:10.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.958+0000 7f895cff9640 1 -- 172.21.15.70:0/1213960880 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8958044070 con 0x7f8960102220 2024-04-01T01:38:10.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.958+0000 7f895f7fe640 1 --2- 172.21.15.70:0/1213960880 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f89480a28e0 0x7f89480a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:10.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.959+0000 7f895f7fe640 1 --2- 172.21.15.70:0/1213960880 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f89480a28e0 0x7f89480a4da0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f895400f4d0 tx=0x7f895400a3b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:10.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:10.965+0000 7f895cff9640 1 -- 172.21.15.70:0/1213960880 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f89580bcfe0 con 0x7f8960102220 2024-04-01T01:38:11.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.390+0000 7f8965e6b640 1 -- 172.21.15.70:0/1213960880 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89601a67e0 con 0x7f8960102220 2024-04-01T01:38:11.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.392+0000 7f895cff9640 1 -- 172.21.15.70:0/1213960880 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f89580bcfe0 con 0x7f8960102220 2024-04-01T01:38:11.393 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:38:11.394 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:33:36.894838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24418,24404,24379,24451,24464]},"id":1}]} 2024-04-01T01:38:11.394 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-04-01T01:38:11.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.397+0000 7f8965e6b640 1 -- 172.21.15.70:0/1213960880 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f89480a28e0 msgr2=0x7f89480a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.397+0000 7f8965e6b640 1 --2- 172.21.15.70:0/1213960880 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f89480a28e0 0x7f89480a4da0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f895400f4d0 tx=0x7f895400a3b0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.397+0000 7f8965e6b640 1 -- 172.21.15.70:0/1213960880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8960102220 msgr2=0x7f89601a58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.397+0000 7f8965e6b640 1 --2- 172.21.15.70:0/1213960880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8960102220 0x7f89601a58b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8958009bf0 tx=0x7f8958003ba0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.398+0000 7f8965e6b640 1 -- 172.21.15.70:0/1213960880 shutdown_connections 2024-04-01T01:38:11.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.398+0000 7f8965e6b640 1 --2- 172.21.15.70:0/1213960880 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f89480a28e0 0x7f89480a4da0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.398+0000 7f8965e6b640 1 --2- 172.21.15.70:0/1213960880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8960102220 0x7f89601a58b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.398+0000 7f8965e6b640 1 --2- 172.21.15.70:0/1213960880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8960101860 0x7f8960070570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.398+0000 7f8965e6b640 1 --2- 172.21.15.70:0/1213960880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8960100670 0x7f8960070030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.399+0000 7f8965e6b640 1 -- 172.21.15.70:0/1213960880 >> 172.21.15.70:0/1213960880 conn(0x7f89600fbbe0 msgr2=0x7f89600fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:11.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.400+0000 7f8965e6b640 1 -- 172.21.15.70:0/1213960880 shutdown_connections 2024-04-01T01:38:11.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.401+0000 7f8965e6b640 1 -- 172.21.15.70:0/1213960880 wait complete. 2024-04-01T01:38:11.423 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:38:11.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.608+0000 7f5110c16640 1 -- 172.21.15.70:0/2377251611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f510c06a420 msgr2=0x7f510c1105e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.608+0000 7f5110c16640 1 --2- 172.21.15.70:0/2377251611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f510c06a420 0x7f510c1105e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f50fc009900 tx=0x7f50fc02bc60 comp rx=0 tx=0).stop 2024-04-01T01:38:11.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.608+0000 7f5110c16640 1 -- 172.21.15.70:0/2377251611 shutdown_connections 2024-04-01T01:38:11.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.608+0000 7f5110c16640 1 --2- 172.21.15.70:0/2377251611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f510c06a420 0x7f510c1105e0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.608+0000 7f5110c16640 1 --2- 172.21.15.70:0/2377251611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f510c10b260 0x7f510c10d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.608+0000 7f5110c16640 1 --2- 172.21.15.70:0/2377251611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f510c108930 0x7f510c10ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.608+0000 7f5110c16640 1 -- 172.21.15.70:0/2377251611 >> 172.21.15.70:0/2377251611 conn(0x7f510c06e820 msgr2=0x7f510c070c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:11.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.609+0000 7f5110c16640 1 -- 172.21.15.70:0/2377251611 shutdown_connections 2024-04-01T01:38:11.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.613+0000 7f5110c16640 1 -- 172.21.15.70:0/2377251611 wait complete. 2024-04-01T01:38:11.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.614+0000 7f5110c16640 1 Processor -- start 2024-04-01T01:38:11.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.614+0000 7f5110c16640 1 -- start start 2024-04-01T01:38:11.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.614+0000 7f5110c16640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f510c06a420 0x7f510c19de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.614+0000 7f5110c16640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f510c108930 0x7f510c19e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.614+0000 7f5110c16640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f510c10b260 0x7f510c1adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.614+0000 7f5110c16640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f510c1159e0 con 0x7f510c06a420 2024-04-01T01:38:11.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.614+0000 7f5110c16640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f510c115860 con 0x7f510c10b260 2024-04-01T01:38:11.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.614+0000 7f5110c16640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f510c115b60 con 0x7f510c108930 2024-04-01T01:38:11.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.614+0000 7f510b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f510c06a420 0x7f510c19de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.614+0000 7f510b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f510c06a420 0x7f510c19de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52626/0 (socket says 172.21.15.70:52626) 2024-04-01T01:38:11.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.614+0000 7f510b7fe640 1 -- 172.21.15.70:0/4058405413 learned_addr learned my addr 172.21.15.70:0/4058405413 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:11.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.615+0000 7f510bfff640 1 --2- 172.21.15.70:0/4058405413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f510c10b260 0x7f510c1adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.615+0000 7f510b7fe640 1 -- 172.21.15.70:0/4058405413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f510c108930 msgr2=0x7f510c19e380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:38:11.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.615+0000 7f510b7fe640 1 --2- 172.21.15.70:0/4058405413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f510c108930 0x7f510c19e380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.615+0000 7f510b7fe640 1 -- 172.21.15.70:0/4058405413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f510c10b260 msgr2=0x7f510c1adcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.615+0000 7f510b7fe640 1 --2- 172.21.15.70:0/4058405413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f510c10b260 0x7f510c1adcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.615+0000 7f510b7fe640 1 -- 172.21.15.70:0/4058405413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f510c1ae280 con 0x7f510c06a420 2024-04-01T01:38:11.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.615+0000 7f510bfff640 1 --2- 172.21.15.70:0/4058405413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f510c10b260 0x7f510c1adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:38:11.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.616+0000 7f510b7fe640 1 --2- 172.21.15.70:0/4058405413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f510c06a420 0x7f510c19de40 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f510400eab0 tx=0x7f510400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:11.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.616+0000 7f5108ff9640 1 -- 172.21.15.70:0/4058405413 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510400ce40 con 0x7f510c06a420 2024-04-01T01:38:11.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.616+0000 7f5108ff9640 1 -- 172.21.15.70:0/4058405413 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5104011920 con 0x7f510c06a420 2024-04-01T01:38:11.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.616+0000 7f5108ff9640 1 -- 172.21.15.70:0/4058405413 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5104010a00 con 0x7f510c06a420 2024-04-01T01:38:11.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.616+0000 7f5110c16640 1 -- 172.21.15.70:0/4058405413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f510c1ae570 con 0x7f510c06a420 2024-04-01T01:38:11.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.616+0000 7f5110c16640 1 -- 172.21.15.70:0/4058405413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f510c1aea10 con 0x7f510c06a420 2024-04-01T01:38:11.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.620+0000 7f5114a56740 1 Processor -- start 2024-04-01T01:38:11.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.620+0000 7f5114a56740 1 -- start start 2024-04-01T01:38:11.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.620+0000 7f5114a56740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff828680 0x55d3ff828aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.620+0000 7f5114a56740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff829070 0x55d3ff82db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.620+0000 7f5114a56740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 0x55d3ff830670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.620+0000 7f5114a56740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d3ff78d1b0 con 0x55d3ff828680 2024-04-01T01:38:11.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.620+0000 7f5114a56740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d3ff735c00 con 0x55d3ff829070 2024-04-01T01:38:11.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.620+0000 7f5114a56740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d3ff789250 con 0x55d3ff82e1e0 2024-04-01T01:38:11.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.621+0000 7f510affd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 0x55d3ff830670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.621+0000 7f510bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff828680 0x55d3ff828aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.621+0000 7f510bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff828680 0x55d3ff828aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52638/0 (socket says 172.21.15.70:52638) 2024-04-01T01:38:11.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.621+0000 7f510bfff640 1 -- 172.21.15.70:0/417017859 learned_addr learned my addr 172.21.15.70:0/417017859 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:11.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.621+0000 7f510affd640 1 -- 172.21.15.70:0/417017859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff829070 msgr2=0x55d3ff82db70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:38:11.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.621+0000 7f510affd640 1 --2- 172.21.15.70:0/417017859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff829070 0x55d3ff82db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.621+0000 7f510affd640 1 -- 172.21.15.70:0/417017859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff828680 msgr2=0x55d3ff828aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:11.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.621+0000 7f510affd640 1 --2- 172.21.15.70:0/417017859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff828680 0x55d3ff828aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.621+0000 7f510affd640 1 -- 172.21.15.70:0/417017859 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3ff7553d0 con 0x55d3ff82e1e0 2024-04-01T01:38:11.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.621+0000 7f5108ff9640 1 -- 172.21.15.70:0/4058405413 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f5104011ad0 con 0x7f510c06a420 2024-04-01T01:38:11.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.622+0000 7f510affd640 1 --2- 172.21.15.70:0/417017859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 0x55d3ff830670 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f510c073950 tx=0x7f5100009f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=544d71d67fa50893 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:11.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.622+0000 7f5108ff9640 1 --2- 172.21.15.70:0/4058405413 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f50f40a25f0 0x7f50f40a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.623+0000 7f5108ff9640 1 -- 172.21.15.70:0/4058405413 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f51040cf210 con 0x7f510c06a420 2024-04-01T01:38:11.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.623+0000 7f50f1ffb640 1 -- 172.21.15.70:0/417017859 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510000ec80 con 0x55d3ff82e1e0 2024-04-01T01:38:11.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.623+0000 7f50f1ffb640 1 -- 172.21.15.70:0/417017859 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f510000ee20 con 0x55d3ff82e1e0 2024-04-01T01:38:11.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.623+0000 7f50f1ffb640 1 -- 172.21.15.70:0/417017859 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5100004a00 con 0x55d3ff82e1e0 2024-04-01T01:38:11.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.623+0000 7f510bfff640 1 --2- 172.21.15.70:0/4058405413 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f50f40a25f0 0x7f50f40a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.624+0000 7f5114a56740 1 -- 172.21.15.70:0/417017859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 msgr2=0x55d3ff830670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.624+0000 7f5114a56740 1 --2- 172.21.15.70:0/417017859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 0x55d3ff830670 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f510c073950 tx=0x7f5100009f30 comp rx=0 tx=0).stop 2024-04-01T01:38:11.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.624+0000 7f510bfff640 1 --2- 172.21.15.70:0/4058405413 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f50f40a25f0 0x7f50f40a4ab0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f50fc005c40 tx=0x7f50fc005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:11.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.624+0000 7f5114a56740 1 -- 172.21.15.70:0/417017859 shutdown_connections 2024-04-01T01:38:11.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.624+0000 7f5114a56740 1 --2- 172.21.15.70:0/417017859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 0x55d3ff830670 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.624+0000 7f5114a56740 1 --2- 172.21.15.70:0/417017859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff829070 0x55d3ff82db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.624+0000 7f5114a56740 1 --2- 172.21.15.70:0/417017859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff828680 0x55d3ff828aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.624+0000 7f5114a56740 1 -- 172.21.15.70:0/417017859 >> 172.21.15.70:0/417017859 conn(0x55d3ff823bf0 msgr2=0x55d3ff823ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:11.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.625+0000 7f5114a56740 1 -- 172.21.15.70:0/417017859 shutdown_connections 2024-04-01T01:38:11.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.625+0000 7f5114a56740 1 -- 172.21.15.70:0/417017859 wait complete. 2024-04-01T01:38:11.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.626+0000 7f5114a56740 1 -- start start 2024-04-01T01:38:11.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.626+0000 7f5114a56740 1 Processor -- start 2024-04-01T01:38:11.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.626+0000 7f5114a56740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff828680 0x55d3ff832ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.627+0000 7f5114a56740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff829070 0x55d3ff8331e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.627+0000 7f5114a56740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 0x55d3ff83a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.627+0000 7f5114a56740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d3ff831190 con 0x55d3ff829070 2024-04-01T01:38:11.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.627+0000 7f5114a56740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d3ff72b470 con 0x55d3ff828680 2024-04-01T01:38:11.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.627+0000 7f5114a56740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d3ff831310 con 0x55d3ff82e1e0 2024-04-01T01:38:11.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.627+0000 7f5114a56740 10 client.0 _ll_register_callbacks cb 0x55d3ff748cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:11.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.627+0000 7f510b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff828680 0x55d3ff832ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.627+0000 7f510b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff828680 0x55d3ff832ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47930/0 (socket says 172.21.15.70:47930) 2024-04-01T01:38:11.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.627+0000 7f510b7fe640 1 -- 172.21.15.70:0/428567511 learned_addr learned my addr 172.21.15.70:0/428567511 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:11.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.627+0000 7f510bfff640 1 --2- 172.21.15.70:0/428567511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 0x55d3ff83a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.628+0000 7f510affd640 1 --2- 172.21.15.70:0/428567511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff829070 0x55d3ff8331e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.628+0000 7f510b7fe640 1 -- 172.21.15.70:0/428567511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 msgr2=0x55d3ff83a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:11.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.628+0000 7f510b7fe640 1 --2- 172.21.15.70:0/428567511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 0x55d3ff83a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.628+0000 7f510b7fe640 1 -- 172.21.15.70:0/428567511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff829070 msgr2=0x55d3ff8331e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:11.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.628+0000 7f510b7fe640 1 --2- 172.21.15.70:0/428567511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff829070 0x55d3ff8331e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.628+0000 7f510b7fe640 1 -- 172.21.15.70:0/428567511 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3ff4a13f0 con 0x55d3ff828680 2024-04-01T01:38:11.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.628+0000 7f510bfff640 1 --2- 172.21.15.70:0/428567511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 0x55d3ff83a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:38:11.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.629+0000 7f510b7fe640 1 --2- 172.21.15.70:0/428567511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff828680 0x55d3ff832ca0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5104097580 tx=0x7f5104011e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d658415b9ef8234d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:11.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.629+0000 7f50eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:11.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.629+0000 7f50eaffd640 1 -- 172.21.15.70:0/428567511 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51040b4060 con 0x55d3ff828680 2024-04-01T01:38:11.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.629+0000 7f50eaffd640 1 -- 172.21.15.70:0/428567511 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5104086070 con 0x55d3ff828680 2024-04-01T01:38:11.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.629+0000 7f5114a56740 1 -- 172.21.15.70:0/428567511 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f510000ee20 con 0x55d3ff828680 2024-04-01T01:38:11.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.629+0000 7f50eaffd640 1 -- 172.21.15.70:0/428567511 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f51040204b0 con 0x55d3ff828680 2024-04-01T01:38:11.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.629+0000 7f5114a56740 10 client.26816 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:11.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.629+0000 7f5114a56740 1 -- 172.21.15.70:0/428567511 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d3ff56b7d0 con 0x55d3ff828680 2024-04-01T01:38:11.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.630+0000 7f50eaffd640 1 -- 172.21.15.70:0/428567511 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f510000ee20 con 0x55d3ff828680 2024-04-01T01:38:11.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.630+0000 7f50eaffd640 1 -- 172.21.15.70:0/428567511 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f510400f040 con 0x55d3ff828680 2024-04-01T01:38:11.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.630+0000 7f5114a56740 10 client.26816 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:11.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.630+0000 7f5114a56740 10 client.26816 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:38:11.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.630+0000 7f5114a56740 20 client.26816 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:11.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.630+0000 7f5114a56740 1 --2- 172.21.15.70:0/428567511 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55d3ff83ea80 0x55d3ff840ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.630+0000 7f5114a56740 4 client.26816 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:11.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.630+0000 7f5114a56740 1 -- 172.21.15.70:0/428567511 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3ff79a6f0 con 0x55d3ff83ea80 2024-04-01T01:38:11.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.630+0000 7f510affd640 1 --2- 172.21.15.70:0/428567511 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55d3ff83ea80 0x55d3ff840ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.631+0000 7f510affd640 1 --2- 172.21.15.70:0/428567511 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55d3ff83ea80 0x55d3ff840ee0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cf99847091293f74 server_cookie=9eda345c13e0e6ba in_seq=0 out_seq=0 2024-04-01T01:38:11.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.631+0000 7f50eaffd640 10 client.26816 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:38:11.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.632+0000 7f50eaffd640 1 -- 172.21.15.70:0/428567511 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d3ff79a6f0 con 0x55d3ff83ea80 2024-04-01T01:38:11.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.632+0000 7f50eaffd640 10 client.26816 handle_command_reply: tid=0 2024-04-01T01:38:11.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.632+0000 7f5114a56740 1 client.26816 shutdown 2024-04-01T01:38:11.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.633+0000 7f5114a56740 1 -- 172.21.15.70:0/428567511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff828680 msgr2=0x55d3ff832ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.633+0000 7f5114a56740 1 --2- 172.21.15.70:0/428567511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff828680 0x55d3ff832ca0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5104097580 tx=0x7f5104011e00 comp rx=0 tx=0).stop 2024-04-01T01:38:11.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.633+0000 7f5114a56740 1 -- 172.21.15.70:0/428567511 shutdown_connections 2024-04-01T01:38:11.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.633+0000 7f5114a56740 1 --2- 172.21.15.70:0/428567511 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55d3ff83ea80 0x55d3ff840ee0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.633+0000 7f5114a56740 1 --2- 172.21.15.70:0/428567511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 0x55d3ff83a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.633+0000 7f5114a56740 1 --2- 172.21.15.70:0/428567511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff829070 0x55d3ff8331e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.633+0000 7f5114a56740 1 --2- 172.21.15.70:0/428567511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff828680 0x55d3ff832ca0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.633+0000 7f5114a56740 1 -- 172.21.15.70:0/428567511 >> 172.21.15.70:0/428567511 conn(0x55d3ff823bf0 msgr2=0x55d3ff822f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:11.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.633+0000 7f50eaffd640 0 client.26816 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:38:11.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.634+0000 7f5114a56740 1 -- 172.21.15.70:0/428567511 shutdown_connections 2024-04-01T01:38:11.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.634+0000 7f5114a56740 1 -- 172.21.15.70:0/428567511 wait complete. 2024-04-01T01:38:11.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.634+0000 7f5114a56740 20 client.26816 trim_cache size 0 max 16384 2024-04-01T01:38:11.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f5114a56740 1 Processor -- start 2024-04-01T01:38:11.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f5114a56740 1 -- start start 2024-04-01T01:38:11.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f5114a56740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff828680 0x55d3ff822700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f5114a56740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff829070 0x55d3ff83a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f5114a56740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 0x55d3ff822c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f5114a56740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f510c1b8fe0 con 0x55d3ff829070 2024-04-01T01:38:11.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f5114a56740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f510c1bab20 con 0x55d3ff828680 2024-04-01T01:38:11.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f5114a56740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f510c1b96b0 con 0x55d3ff82e1e0 2024-04-01T01:38:11.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f510affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff829070 0x55d3ff83a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f510affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff829070 0x55d3ff83a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52662/0 (socket says 172.21.15.70:52662) 2024-04-01T01:38:11.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f510affd640 1 -- 172.21.15.70:0/1402315277 learned_addr learned my addr 172.21.15.70:0/1402315277 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:11.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f510bfff640 1 --2- 172.21.15.70:0/1402315277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 0x55d3ff822c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f510b7fe640 1 --2- 172.21.15.70:0/1402315277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff828680 0x55d3ff822700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f510affd640 1 -- 172.21.15.70:0/1402315277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 msgr2=0x55d3ff822c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f510affd640 1 --2- 172.21.15.70:0/1402315277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 0x55d3ff822c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f510affd640 1 -- 172.21.15.70:0/1402315277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff828680 msgr2=0x55d3ff822700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f510affd640 1 --2- 172.21.15.70:0/1402315277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff828680 0x55d3ff822700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.649+0000 7f510affd640 1 -- 172.21.15.70:0/1402315277 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3ff81f660 con 0x55d3ff829070 2024-04-01T01:38:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.650+0000 7f510affd640 1 --2- 172.21.15.70:0/1402315277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff829070 0x55d3ff83a1a0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f5100009a70 tx=0x7f51000058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf13dc7b05c54e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.650+0000 7f50f17fa640 1 -- 172.21.15.70:0/1402315277 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5100030070 con 0x55d3ff829070 2024-04-01T01:38:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.650+0000 7f50f17fa640 1 -- 172.21.15.70:0/1402315277 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5100009530 con 0x55d3ff829070 2024-04-01T01:38:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.650+0000 7f50f17fa640 1 -- 172.21.15.70:0/1402315277 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510002a7f0 con 0x55d3ff829070 2024-04-01T01:38:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.650+0000 7f5114a56740 1 -- 172.21.15.70:0/1402315277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff829070 msgr2=0x55d3ff83a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.650+0000 7f5114a56740 1 --2- 172.21.15.70:0/1402315277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff829070 0x55d3ff83a1a0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f5100009a70 tx=0x7f51000058b0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.650+0000 7f5114a56740 1 -- 172.21.15.70:0/1402315277 shutdown_connections 2024-04-01T01:38:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.650+0000 7f5114a56740 1 --2- 172.21.15.70:0/1402315277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff82e1e0 0x55d3ff822c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.650+0000 7f5114a56740 1 --2- 172.21.15.70:0/1402315277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff829070 0x55d3ff83a1a0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.651+0000 7f5114a56740 1 --2- 172.21.15.70:0/1402315277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff828680 0x55d3ff822700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.651+0000 7f5114a56740 1 -- 172.21.15.70:0/1402315277 >> 172.21.15.70:0/1402315277 conn(0x55d3ff823bf0 msgr2=0x55d3ff832c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.651+0000 7f5114a56740 1 -- 172.21.15.70:0/1402315277 shutdown_connections 2024-04-01T01:38:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.651+0000 7f5114a56740 1 -- 172.21.15.70:0/1402315277 wait complete. 2024-04-01T01:38:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.651+0000 7f5114a56740 1 -- start start 2024-04-01T01:38:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.651+0000 7f5114a56740 1 Processor -- start 2024-04-01T01:38:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.651+0000 7f5114a56740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff828680 0x55d3ff837490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.651+0000 7f5114a56740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff829070 0x55d3ff8379d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f5114a56740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff82e1e0 0x55d3ff838990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f5114a56740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f510c1ba450 con 0x55d3ff828680 2024-04-01T01:38:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f5114a56740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f510c1b9d80 con 0x55d3ff82e1e0 2024-04-01T01:38:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f5114a56740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f510c1b6fc0 con 0x55d3ff829070 2024-04-01T01:38:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f510b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff828680 0x55d3ff837490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f5114a56740 10 client.0 _ll_register_callbacks cb 0x7f510409b690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f510b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff828680 0x55d3ff837490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52670/0 (socket says 172.21.15.70:52670) 2024-04-01T01:38:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f510affd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff829070 0x55d3ff8379d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f510bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff82e1e0 0x55d3ff838990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f510b7fe640 1 -- 172.21.15.70:0/3070442704 learned_addr learned my addr 172.21.15.70:0/3070442704 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f510b7fe640 1 -- 172.21.15.70:0/3070442704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff829070 msgr2=0x55d3ff8379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f510b7fe640 1 --2- 172.21.15.70:0/3070442704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff829070 0x55d3ff8379d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f510b7fe640 1 -- 172.21.15.70:0/3070442704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff82e1e0 msgr2=0x55d3ff838990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f510b7fe640 1 --2- 172.21.15.70:0/3070442704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff82e1e0 0x55d3ff838990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f510b7fe640 1 -- 172.21.15.70:0/3070442704 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3ff838f80 con 0x55d3ff828680 2024-04-01T01:38:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f510bfff640 1 --2- 172.21.15.70:0/3070442704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff82e1e0 0x55d3ff838990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:38:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f510b7fe640 1 --2- 172.21.15.70:0/3070442704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff828680 0x55d3ff837490 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f51040a3d80 tx=0x7f51040097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9436bd015a8cca82 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f50eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:38:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f50eaffd640 1 -- 172.21.15.70:0/3070442704 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51040afd60 con 0x55d3ff828680 2024-04-01T01:38:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f50eaffd640 1 -- 172.21.15.70:0/3070442704 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5104011470 con 0x55d3ff828680 2024-04-01T01:38:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f5114a56740 1 -- 172.21.15.70:0/3070442704 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5100009530 con 0x55d3ff828680 2024-04-01T01:38:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.652+0000 7f50eaffd640 1 -- 172.21.15.70:0/3070442704 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f51040862c0 con 0x55d3ff828680 2024-04-01T01:38:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.653+0000 7f5114a56740 10 client.16950 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.653+0000 7f5114a56740 1 -- 172.21.15.70:0/3070442704 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d3ff83a2f0 con 0x55d3ff828680 2024-04-01T01:38:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.653+0000 7f50eaffd640 1 -- 172.21.15.70:0/3070442704 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5100009530 con 0x55d3ff828680 2024-04-01T01:38:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.653+0000 7f50eaffd640 1 -- 172.21.15.70:0/3070442704 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5104010ba0 con 0x55d3ff828680 2024-04-01T01:38:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.653+0000 7f5114a56740 10 client.16950 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.653+0000 7f5114a56740 10 client.16950 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:38:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.653+0000 7f5114a56740 20 client.16950 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.653+0000 7f5114a56740 1 --2- 172.21.15.70:0/3070442704 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55d3ff8a52d0 0x55d3ff8a76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.653+0000 7f5114a56740 4 client.16950 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:38:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.653+0000 7f5114a56740 1 -- 172.21.15.70:0/3070442704 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d3ff794a40 con 0x55d3ff8a52d0 2024-04-01T01:38:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.653+0000 7f510affd640 1 --2- 172.21.15.70:0/3070442704 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55d3ff8a52d0 0x55d3ff8a76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f50eaffd640 10 client.16950 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:38:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f510affd640 1 --2- 172.21.15.70:0/3070442704 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55d3ff8a52d0 0x55d3ff8a76f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=48c32be067dfe1a1 server_cookie=6ace7f1f0abf51c3 in_seq=0 out_seq=0 2024-04-01T01:38:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f50eaffd640 1 -- 172.21.15.70:0/3070442704 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d3ff794a40 con 0x55d3ff8a52d0 2024-04-01T01:38:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f50eaffd640 10 client.16950 handle_command_reply: tid=0 2024-04-01T01:38:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f5114a56740 1 client.16950 shutdown 2024-04-01T01:38:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f5114a56740 1 -- 172.21.15.70:0/3070442704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff828680 msgr2=0x55d3ff837490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f5114a56740 1 --2- 172.21.15.70:0/3070442704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff828680 0x55d3ff837490 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f51040a3d80 tx=0x7f51040097b0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f5114a56740 1 -- 172.21.15.70:0/3070442704 shutdown_connections 2024-04-01T01:38:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f5114a56740 1 --2- 172.21.15.70:0/3070442704 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55d3ff8a52d0 0x55d3ff8a76f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f50eaffd640 0 client.16950 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:38:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f5114a56740 1 --2- 172.21.15.70:0/3070442704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d3ff82e1e0 0x55d3ff838990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f5114a56740 1 --2- 172.21.15.70:0/3070442704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d3ff829070 0x55d3ff8379d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f5114a56740 1 --2- 172.21.15.70:0/3070442704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d3ff828680 0x55d3ff837490 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f5114a56740 1 -- 172.21.15.70:0/3070442704 >> 172.21.15.70:0/3070442704 conn(0x55d3ff823bf0 msgr2=0x55d3ff826e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f5114a56740 1 -- 172.21.15.70:0/3070442704 shutdown_connections 2024-04-01T01:38:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f5114a56740 1 -- 172.21.15.70:0/3070442704 wait complete. 2024-04-01T01:38:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.654+0000 7f5114a56740 20 client.16950 trim_cache size 0 max 16384 2024-04-01T01:38:11.661 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:38:11.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.655+0000 7f5110c16640 1 -- 172.21.15.70:0/4058405413 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f50f40a25f0 msgr2=0x7f50f40a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.655+0000 7f5110c16640 1 --2- 172.21.15.70:0/4058405413 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f50f40a25f0 0x7f50f40a4ab0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f50fc005c40 tx=0x7f50fc005b50 comp rx=0 tx=0).stop 2024-04-01T01:38:11.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.655+0000 7f5110c16640 1 -- 172.21.15.70:0/4058405413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f510c06a420 msgr2=0x7f510c19de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.655+0000 7f5110c16640 1 --2- 172.21.15.70:0/4058405413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f510c06a420 0x7f510c19de40 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f510400eab0 tx=0x7f510400ef70 comp rx=0 tx=0).stop 2024-04-01T01:38:11.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.655+0000 7f5110c16640 1 -- 172.21.15.70:0/4058405413 shutdown_connections 2024-04-01T01:38:11.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.655+0000 7f5110c16640 1 --2- 172.21.15.70:0/4058405413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f510c10b260 0x7f510c1adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.655+0000 7f5110c16640 1 --2- 172.21.15.70:0/4058405413 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f50f40a25f0 0x7f50f40a4ab0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.655+0000 7f5110c16640 1 --2- 172.21.15.70:0/4058405413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f510c108930 0x7f510c19e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.655+0000 7f5110c16640 1 --2- 172.21.15.70:0/4058405413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f510c06a420 0x7f510c19de40 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.655+0000 7f5110c16640 1 -- 172.21.15.70:0/4058405413 >> 172.21.15.70:0/4058405413 conn(0x7f510c06e820 msgr2=0x7f510c113740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:11.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.655+0000 7f5110c16640 1 -- 172.21.15.70:0/4058405413 shutdown_connections 2024-04-01T01:38:11.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.655+0000 7f5110c16640 1 -- 172.21.15.70:0/4058405413 wait complete. 2024-04-01T01:38:11.664 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:38:11.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.837+0000 7f06cce4e640 1 -- 172.21.15.70:0/3596515708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c80ff9a0 msgr2=0x7f06c80ffdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.837+0000 7f06cce4e640 1 --2- 172.21.15.70:0/3596515708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c80ff9a0 0x7f06c80ffdc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009900 tx=0x7f06b802bc30 comp rx=0 tx=0).stop 2024-04-01T01:38:11.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.838+0000 7f06cce4e640 1 -- 172.21.15.70:0/3596515708 shutdown_connections 2024-04-01T01:38:11.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.838+0000 7f06cce4e640 1 --2- 172.21.15.70:0/3596515708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c8101550 0x7f06c8107de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.838+0000 7f06cce4e640 1 --2- 172.21.15.70:0/3596515708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c8100b90 0x7f06c8101010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.838+0000 7f06cce4e640 1 --2- 172.21.15.70:0/3596515708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c80ff9a0 0x7f06c80ffdc0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.838+0000 7f06cce4e640 1 -- 172.21.15.70:0/3596515708 >> 172.21.15.70:0/3596515708 conn(0x7f06c80faf10 msgr2=0x7f06c80fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:11.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.839+0000 7f06cce4e640 1 -- 172.21.15.70:0/3596515708 shutdown_connections 2024-04-01T01:38:11.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.840+0000 7f06cce4e640 1 -- 172.21.15.70:0/3596515708 wait complete. 2024-04-01T01:38:11.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.841+0000 7f06cce4e640 1 Processor -- start 2024-04-01T01:38:11.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.842+0000 7f06cce4e640 1 -- start start 2024-04-01T01:38:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.842+0000 7f06cce4e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c80ff9a0 0x7f06c81957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.842+0000 7f06cce4e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c8100b90 0x7f06c8195d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.842+0000 7f06cce4e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c8101550 0x7f06c819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.842+0000 7f06cce4e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f06c810b0e0 con 0x7f06c8101550 2024-04-01T01:38:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.842+0000 7f06c6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c8101550 0x7f06c819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.842+0000 7f06c6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c8101550 0x7f06c819cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52680/0 (socket says 172.21.15.70:52680) 2024-04-01T01:38:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.842+0000 7f06c6d9d640 1 -- 172.21.15.70:0/596990721 learned_addr learned my addr 172.21.15.70:0/596990721 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.842+0000 7f06cce4e640 1 -- 172.21.15.70:0/596990721 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f06c810af60 con 0x7f06c8100b90 2024-04-01T01:38:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.842+0000 7f06cce4e640 1 -- 172.21.15.70:0/596990721 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f06c810b260 con 0x7f06c80ff9a0 2024-04-01T01:38:11.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.843+0000 7f06c659c640 1 --2- 172.21.15.70:0/596990721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c80ff9a0 0x7f06c81957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.843+0000 7f06c5d9b640 1 --2- 172.21.15.70:0/596990721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c8100b90 0x7f06c8195d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.843+0000 7f06c659c640 1 -- 172.21.15.70:0/596990721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c8100b90 msgr2=0x7f06c8195d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.843+0000 7f06c659c640 1 --2- 172.21.15.70:0/596990721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c8100b90 0x7f06c8195d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.843+0000 7f06c659c640 1 -- 172.21.15.70:0/596990721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c8101550 msgr2=0x7f06c819cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.843+0000 7f06c659c640 1 --2- 172.21.15.70:0/596990721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c8101550 0x7f06c819cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.843+0000 7f06c659c640 1 -- 172.21.15.70:0/596990721 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c819d410 con 0x7f06c80ff9a0 2024-04-01T01:38:11.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.844+0000 7f06c659c640 1 --2- 172.21.15.70:0/596990721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c80ff9a0 0x7f06c81957f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f06b8036040 tx=0x7f06b8004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:11.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.844+0000 7f06b77fe640 1 -- 172.21.15.70:0/596990721 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b8043020 con 0x7f06c80ff9a0 2024-04-01T01:38:11.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.844+0000 7f06b77fe640 1 -- 172.21.15.70:0/596990721 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06b80043c0 con 0x7f06c80ff9a0 2024-04-01T01:38:11.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.844+0000 7f06cce4e640 1 -- 172.21.15.70:0/596990721 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06c819d6a0 con 0x7f06c80ff9a0 2024-04-01T01:38:11.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.845+0000 7f06b77fe640 1 -- 172.21.15.70:0/596990721 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b8003ae0 con 0x7f06c80ff9a0 2024-04-01T01:38:11.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.845+0000 7f06cce4e640 1 -- 172.21.15.70:0/596990721 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06c819db50 con 0x7f06c80ff9a0 2024-04-01T01:38:11.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.849+0000 7f06b77fe640 1 -- 172.21.15.70:0/596990721 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f06b8046020 con 0x7f06c80ff9a0 2024-04-01T01:38:11.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.850+0000 7f06cfa35740 1 Processor -- start 2024-04-01T01:38:11.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.851+0000 7f06cfa35740 1 -- start start 2024-04-01T01:38:11.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.851+0000 7f06cfa35740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb1cca0 0x55845bb1d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.851+0000 7f06cfa35740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb22190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.851+0000 7f06cfa35740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb22800 0x55845bb24c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.851+0000 7f06cfa35740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55845ba791b0 con 0x55845bb22800 2024-04-01T01:38:11.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.851+0000 7f06cfa35740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55845ba21c00 con 0x55845bb1cca0 2024-04-01T01:38:11.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.851+0000 7f06cfa35740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55845ba75250 con 0x55845bb1d690 2024-04-01T01:38:11.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.852+0000 7f06b77fe640 1 --2- 172.21.15.70:0/596990721 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06b00a2600 0x7f06b00a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.852+0000 7f06b77fe640 1 -- 172.21.15.70:0/596990721 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f06b80f0cd0 con 0x7f06c80ff9a0 2024-04-01T01:38:11.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.852+0000 7f06c5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb22800 0x55845bb24c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.852+0000 7f06c5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb22800 0x55845bb24c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52686/0 (socket says 172.21.15.70:52686) 2024-04-01T01:38:11.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.852+0000 7f06c5d9b640 1 -- 172.21.15.70:0/749957792 learned_addr learned my addr 172.21.15.70:0/749957792 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:11.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.852+0000 7f06c6d9d640 1 --2- 172.21.15.70:0/749957792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb1cca0 0x55845bb1d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.852+0000 7f06c659c640 1 --2- 172.21.15.70:0/749957792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb22190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.853+0000 7f06c5d9b640 1 -- 172.21.15.70:0/749957792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 msgr2=0x55845bb22190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:11.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.853+0000 7f06c5d9b640 1 --2- 172.21.15.70:0/749957792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb22190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.853+0000 7f06c5d9b640 1 -- 172.21.15.70:0/749957792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb1cca0 msgr2=0x55845bb1d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:11.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.853+0000 7f06c5d9b640 1 --2- 172.21.15.70:0/749957792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb1cca0 0x55845bb1d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.853+0000 7f06c5d9b640 1 -- 172.21.15.70:0/749957792 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55845ba413d0 con 0x55845bb22800 2024-04-01T01:38:11.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.853+0000 7f06c6d9d640 1 --2- 172.21.15.70:0/596990721 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06b00a2600 0x7f06b00a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.853+0000 7f06c5d9b640 1 --2- 172.21.15.70:0/749957792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb22800 0x55845bb24c90 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f06bc009c10 tx=0x7f06bc0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece6132e3658295 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:11.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.854+0000 7f06b4ff9640 1 -- 172.21.15.70:0/749957792 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc012450 con 0x55845bb22800 2024-04-01T01:38:11.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.854+0000 7f06b4ff9640 1 -- 172.21.15.70:0/749957792 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06bc012aa0 con 0x55845bb22800 2024-04-01T01:38:11.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.854+0000 7f06b4ff9640 1 -- 172.21.15.70:0/749957792 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc00fb10 con 0x55845bb22800 2024-04-01T01:38:11.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.855+0000 7f06cfa35740 1 -- 172.21.15.70:0/749957792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb22800 msgr2=0x55845bb24c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.855+0000 7f06cfa35740 1 --2- 172.21.15.70:0/749957792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb22800 0x55845bb24c90 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f06bc009c10 tx=0x7f06bc0117b0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.855+0000 7f06c6d9d640 1 --2- 172.21.15.70:0/596990721 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06b00a2600 0x7f06b00a4ac0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f06c0009270 tx=0x7f06c0004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:11.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.855+0000 7f06cfa35740 1 -- 172.21.15.70:0/749957792 shutdown_connections 2024-04-01T01:38:11.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.855+0000 7f06cfa35740 1 --2- 172.21.15.70:0/749957792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb22800 0x55845bb24c90 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.855+0000 7f06cfa35740 1 --2- 172.21.15.70:0/749957792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb22190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.855+0000 7f06cfa35740 1 --2- 172.21.15.70:0/749957792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb1cca0 0x55845bb1d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.855+0000 7f06cfa35740 1 -- 172.21.15.70:0/749957792 >> 172.21.15.70:0/749957792 conn(0x55845bb18210 msgr2=0x55845bb18610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:11.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.856+0000 7f06cfa35740 1 -- 172.21.15.70:0/749957792 shutdown_connections 2024-04-01T01:38:11.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.856+0000 7f06cfa35740 1 -- 172.21.15.70:0/749957792 wait complete. 2024-04-01T01:38:11.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.856+0000 7f06cfa35740 1 -- start start 2024-04-01T01:38:11.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.856+0000 7f06cfa35740 1 Processor -- start 2024-04-01T01:38:11.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.856+0000 7f06cfa35740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb1cca0 0x55845bb38080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.856+0000 7f06cfa35740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb385c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.856+0000 7f06cfa35740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 0x55845bb3f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06cfa35740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55845bb367c0 con 0x55845bb1cca0 2024-04-01T01:38:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06cfa35740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55845ba17470 con 0x55845bb22800 2024-04-01T01:38:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06cfa35740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55845bb36940 con 0x55845bb1d690 2024-04-01T01:38:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06c659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb1cca0 0x55845bb38080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06c659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb1cca0 0x55845bb38080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52696/0 (socket says 172.21.15.70:52696) 2024-04-01T01:38:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06c659c640 1 -- 172.21.15.70:0/3271055218 learned_addr learned my addr 172.21.15.70:0/3271055218 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06c6d9d640 1 --2- 172.21.15.70:0/3271055218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 0x55845bb3f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06cfa35740 10 client.0 _ll_register_callbacks cb 0x55845ba34cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:11.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06c5d9b640 1 --2- 172.21.15.70:0/3271055218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb385c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06c659c640 1 -- 172.21.15.70:0/3271055218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 msgr2=0x55845bb385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:11.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06c659c640 1 --2- 172.21.15.70:0/3271055218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb385c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06c659c640 1 -- 172.21.15.70:0/3271055218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 msgr2=0x55845bb3f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:11.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06c659c640 1 --2- 172.21.15.70:0/3271055218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 0x55845bb3f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06c659c640 1 -- 172.21.15.70:0/3271055218 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55845b78d3f0 con 0x55845bb1cca0 2024-04-01T01:38:11.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06c659c640 1 --2- 172.21.15.70:0/3271055218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb1cca0 0x55845bb38080 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009900 tx=0x7f06b80bcd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f64d185a25c2e6cb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:11.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06adffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:38:11.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.857+0000 7f06adffb640 1 -- 172.21.15.70:0/3271055218 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b80bd360 con 0x55845bb1cca0 2024-04-01T01:38:11.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.858+0000 7f06adffb640 1 -- 172.21.15.70:0/3271055218 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f06b80b9940 con 0x55845bb1cca0 2024-04-01T01:38:11.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.858+0000 7f06cfa35740 1 -- 172.21.15.70:0/3271055218 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06bc012aa0 con 0x55845bb1cca0 2024-04-01T01:38:11.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.858+0000 7f06adffb640 1 -- 172.21.15.70:0/3271055218 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f06b80c14b0 con 0x55845bb1cca0 2024-04-01T01:38:11.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.858+0000 7f06cfa35740 10 client.16962 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:11.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.858+0000 7f06cfa35740 1 -- 172.21.15.70:0/3271055218 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55845b8577d0 con 0x55845bb1cca0 2024-04-01T01:38:11.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.858+0000 7f06adffb640 1 -- 172.21.15.70:0/3271055218 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06bc012aa0 con 0x55845bb1cca0 2024-04-01T01:38:11.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.859+0000 7f06adffb640 1 -- 172.21.15.70:0/3271055218 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f06b80c58e0 con 0x55845bb1cca0 2024-04-01T01:38:11.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.859+0000 7f06cfa35740 10 client.16962 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:11.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.859+0000 7f06cfa35740 10 client.16962 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:38:11.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.859+0000 7f06cfa35740 20 client.16962 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:11.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.859+0000 7f06cfa35740 1 --2- 172.21.15.70:0/3271055218 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55845bb43af0 0x55845bb45f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.859+0000 7f06cfa35740 4 client.16962 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:11.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.859+0000 7f06cfa35740 1 -- 172.21.15.70:0/3271055218 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55845ba866f0 con 0x55845bb43af0 2024-04-01T01:38:11.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.862+0000 7f06c5d9b640 1 --2- 172.21.15.70:0/3271055218 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55845bb43af0 0x55845bb45f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.865+0000 7f06c5d9b640 1 --2- 172.21.15.70:0/3271055218 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55845bb43af0 0x55845bb45f90 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bc74bb74b5addd52 server_cookie=33dc64bbc5420ad7 in_seq=0 out_seq=0 2024-04-01T01:38:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.865+0000 7f06adffb640 10 client.16962 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:38:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.866+0000 7f06adffb640 1 -- 172.21.15.70:0/3271055218 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55845ba866f0 con 0x55845bb43af0 2024-04-01T01:38:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.866+0000 7f06adffb640 10 client.16962 handle_command_reply: tid=0 2024-04-01T01:38:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.866+0000 7f06cfa35740 1 client.16962 shutdown 2024-04-01T01:38:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.866+0000 7f06cfa35740 1 -- 172.21.15.70:0/3271055218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb1cca0 msgr2=0x55845bb38080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.866+0000 7f06cfa35740 1 --2- 172.21.15.70:0/3271055218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb1cca0 0x55845bb38080 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009900 tx=0x7f06b80bcd80 comp rx=0 tx=0).stop 2024-04-01T01:38:11.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.866+0000 7f06cfa35740 1 -- 172.21.15.70:0/3271055218 shutdown_connections 2024-04-01T01:38:11.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.866+0000 7f06cfa35740 1 --2- 172.21.15.70:0/3271055218 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55845bb43af0 0x55845bb45f90 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.866+0000 7f06cfa35740 1 --2- 172.21.15.70:0/3271055218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 0x55845bb3f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.866+0000 7f06cfa35740 1 --2- 172.21.15.70:0/3271055218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb385c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.866+0000 7f06cfa35740 1 --2- 172.21.15.70:0/3271055218 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb1cca0 0x55845bb38080 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.866+0000 7f06cfa35740 1 -- 172.21.15.70:0/3271055218 >> 172.21.15.70:0/3271055218 conn(0x55845bb18210 msgr2=0x55845bb0ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:11.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.867+0000 7f06adffb640 0 client.16962 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:38:11.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.868+0000 7f06cfa35740 1 -- 172.21.15.70:0/3271055218 shutdown_connections 2024-04-01T01:38:11.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.868+0000 7f06cfa35740 1 -- 172.21.15.70:0/3271055218 wait complete. 2024-04-01T01:38:11.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.868+0000 7f06cfa35740 20 client.16962 trim_cache size 0 max 16384 2024-04-01T01:38:11.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 205.0 s, 1021.8 tps, lat 97.683 ms stddev 33.240 2024-04-01T01:38:11.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.896+0000 7f06cfa35740 1 Processor -- start 2024-04-01T01:38:11.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.896+0000 7f06cfa35740 1 -- start start 2024-04-01T01:38:11.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.897+0000 7f06cfa35740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb1cca0 0x55845bb383a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.897+0000 7f06cfa35740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb3f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.897+0000 7f06cfa35740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 0x55845bb4b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.897+0000 7f06cfa35740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f06c819f9f0 con 0x55845bb1cca0 2024-04-01T01:38:11.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.897+0000 7f06cfa35740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f06c81a1530 con 0x55845bb22800 2024-04-01T01:38:11.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.897+0000 7f06cfa35740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f06c81a00c0 con 0x55845bb1d690 2024-04-01T01:38:11.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.897+0000 7f06c6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 0x55845bb4b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.897+0000 7f06c6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 0x55845bb4b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47992/0 (socket says 172.21.15.70:47992) 2024-04-01T01:38:11.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.897+0000 7f06c6d9d640 1 -- 172.21.15.70:0/3442364989 learned_addr learned my addr 172.21.15.70:0/3442364989 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:11.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.897+0000 7f06c659c640 1 --2- 172.21.15.70:0/3442364989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb1cca0 0x55845bb383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.897+0000 7f06c5d9b640 1 --2- 172.21.15.70:0/3442364989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb3f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.898+0000 7f06c6d9d640 1 -- 172.21.15.70:0/3442364989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 msgr2=0x55845bb3f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:11.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.898+0000 7f06c6d9d640 1 --2- 172.21.15.70:0/3442364989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb3f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.898+0000 7f06c6d9d640 1 -- 172.21.15.70:0/3442364989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb1cca0 msgr2=0x55845bb383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:11.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.898+0000 7f06c6d9d640 1 --2- 172.21.15.70:0/3442364989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb1cca0 0x55845bb383a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.898+0000 7f06c6d9d640 1 -- 172.21.15.70:0/3442364989 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55845bb1da70 con 0x55845bb22800 2024-04-01T01:38:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.898+0000 7f06c6d9d640 1 --2- 172.21.15.70:0/3442364989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 0x55845bb4b870 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f06c000fbf0 tx=0x7f06c00091e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56ecb44740ae12ee server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.898+0000 7f06affff640 1 -- 172.21.15.70:0/3442364989 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c0019070 con 0x55845bb22800 2024-04-01T01:38:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.898+0000 7f06affff640 1 -- 172.21.15.70:0/3442364989 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06c0020af0 con 0x55845bb22800 2024-04-01T01:38:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.899+0000 7f06cfa35740 1 -- 172.21.15.70:0/3442364989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 msgr2=0x55845bb4b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.899+0000 7f06cfa35740 1 --2- 172.21.15.70:0/3442364989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 0x55845bb4b870 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f06c000fbf0 tx=0x7f06c00091e0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.900+0000 7f06cfa35740 1 -- 172.21.15.70:0/3442364989 shutdown_connections 2024-04-01T01:38:11.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.900+0000 7f06cfa35740 1 --2- 172.21.15.70:0/3442364989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 0x55845bb4b870 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.900+0000 7f06cfa35740 1 --2- 172.21.15.70:0/3442364989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb3f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.900+0000 7f06cfa35740 1 --2- 172.21.15.70:0/3442364989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb1cca0 0x55845bb383a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.900+0000 7f06cfa35740 1 -- 172.21.15.70:0/3442364989 >> 172.21.15.70:0/3442364989 conn(0x55845bb18210 msgr2=0x55845bb37c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:11.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.900+0000 7f06cfa35740 1 -- 172.21.15.70:0/3442364989 shutdown_connections 2024-04-01T01:38:11.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.900+0000 7f06cfa35740 1 -- 172.21.15.70:0/3442364989 wait complete. 2024-04-01T01:38:11.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.900+0000 7f06cfa35740 1 -- start start 2024-04-01T01:38:11.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.900+0000 7f06cfa35740 1 Processor -- start 2024-04-01T01:38:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.901+0000 7f06cfa35740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb1cca0 0x55845bb3dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.901+0000 7f06cfa35740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb3e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.901+0000 7f06cfa35740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 0x55845bb3e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.901+0000 7f06cfa35740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f06c81a0e60 con 0x55845bb1cca0 2024-04-01T01:38:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.901+0000 7f06cfa35740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f06c81a0790 con 0x55845bb22800 2024-04-01T01:38:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.901+0000 7f06cfa35740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f06c8073590 con 0x55845bb1d690 2024-04-01T01:38:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.901+0000 7f06cfa35740 10 client.0 _ll_register_callbacks cb 0x7f06b80bcea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.901+0000 7f06c5d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb3e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.901+0000 7f06c5d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb3e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36238/0 (socket says 172.21.15.70:36238) 2024-04-01T01:38:11.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.901+0000 7f06c5d9b640 1 -- 172.21.15.70:0/4210561394 learned_addr learned my addr 172.21.15.70:0/4210561394 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:11.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.902+0000 7f06c6d9d640 1 --2- 172.21.15.70:0/4210561394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 0x55845bb3e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.902+0000 7f06c5d9b640 1 -- 172.21.15.70:0/4210561394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 msgr2=0x55845bb3e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:11.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.902+0000 7f06c5d9b640 1 --2- 172.21.15.70:0/4210561394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 0x55845bb3e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.902+0000 7f06c5d9b640 1 -- 172.21.15.70:0/4210561394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb1cca0 msgr2=0x55845bb3dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:11.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.902+0000 7f06c5d9b640 1 --2- 172.21.15.70:0/4210561394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb1cca0 0x55845bb3dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.902+0000 7f06c5d9b640 1 -- 172.21.15.70:0/4210561394 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55845bb14640 con 0x55845bb1d690 2024-04-01T01:38:11.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.903+0000 7f06c5d9b640 1 --2- 172.21.15.70:0/4210561394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb3e400 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x55845bb19180 tx=0x7f06bc00f630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24b1c571936fa27 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:11.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.903+0000 7f06adffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:38:11.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.903+0000 7f06adffb640 1 -- 172.21.15.70:0/4210561394 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc02dcb0 con 0x55845bb1d690 2024-04-01T01:38:11.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.903+0000 7f06adffb640 1 -- 172.21.15.70:0/4210561394 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f06bc004020 con 0x55845bb1d690 2024-04-01T01:38:11.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.904+0000 7f06cfa35740 1 -- 172.21.15.70:0/4210561394 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06c0020af0 con 0x55845bb1d690 2024-04-01T01:38:11.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.906+0000 7f06adffb640 1 -- 172.21.15.70:0/4210561394 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f06bc02de50 con 0x55845bb1d690 2024-04-01T01:38:11.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.906+0000 7f06cfa35740 10 client.26749 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:11.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.906+0000 7f06cfa35740 1 -- 172.21.15.70:0/4210561394 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55845bb159c0 con 0x55845bb1d690 2024-04-01T01:38:11.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.907+0000 7f06adffb640 1 -- 172.21.15.70:0/4210561394 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06c0020af0 con 0x55845bb1d690 2024-04-01T01:38:11.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.907+0000 7f06adffb640 1 -- 172.21.15.70:0/4210561394 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f06bc005040 con 0x55845bb1d690 2024-04-01T01:38:11.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.907+0000 7f06cfa35740 10 client.26749 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:11.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.907+0000 7f06cfa35740 10 client.26749 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:38:11.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.907+0000 7f06cfa35740 20 client.26749 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:11.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.907+0000 7f06cfa35740 1 --2- 172.21.15.70:0/4210561394 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55845bb16090 0x55845bb164a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:11.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.907+0000 7f06cfa35740 4 client.26749 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:38:11.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.907+0000 7f06cfa35740 1 -- 172.21.15.70:0/4210561394 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55845ba80a40 con 0x55845bb16090 2024-04-01T01:38:11.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.907+0000 7f06c659c640 1 --2- 172.21.15.70:0/4210561394 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55845bb16090 0x55845bb164a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:11.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.908+0000 7f06c659c640 1 --2- 172.21.15.70:0/4210561394 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55845bb16090 0x55845bb164a0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5ffb8fc7794c0f38 server_cookie=53cb16e0a56adaf1 in_seq=0 out_seq=0 2024-04-01T01:38:11.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.908+0000 7f06adffb640 10 client.26749 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:38:11.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.908+0000 7f06adffb640 1 -- 172.21.15.70:0/4210561394 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55845ba80a40 con 0x55845bb16090 2024-04-01T01:38:11.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.908+0000 7f06adffb640 10 client.26749 handle_command_reply: tid=0 2024-04-01T01:38:11.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.908+0000 7f06cfa35740 1 client.26749 shutdown 2024-04-01T01:38:11.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.908+0000 7f06cfa35740 1 -- 172.21.15.70:0/4210561394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 msgr2=0x55845bb3e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.908+0000 7f06cfa35740 1 --2- 172.21.15.70:0/4210561394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb3e400 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x55845bb19180 tx=0x7f06bc00f630 comp rx=0 tx=0).stop 2024-04-01T01:38:11.909 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:38:11.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.909+0000 7f06cfa35740 1 -- 172.21.15.70:0/4210561394 shutdown_connections 2024-04-01T01:38:11.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.909+0000 7f06cfa35740 1 --2- 172.21.15.70:0/4210561394 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55845bb16090 0x55845bb164a0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.909+0000 7f06cfa35740 1 --2- 172.21.15.70:0/4210561394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55845bb22800 0x55845bb3e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.909+0000 7f06cfa35740 1 --2- 172.21.15.70:0/4210561394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55845bb1d690 0x55845bb3e400 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.909+0000 7f06cfa35740 1 --2- 172.21.15.70:0/4210561394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55845bb1cca0 0x55845bb3dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.909+0000 7f06cfa35740 1 -- 172.21.15.70:0/4210561394 >> 172.21.15.70:0/4210561394 conn(0x55845bb18210 msgr2=0x55845bb4a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:11.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.909+0000 7f06adffb640 0 client.26749 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:38:11.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.909+0000 7f06cfa35740 1 -- 172.21.15.70:0/4210561394 shutdown_connections 2024-04-01T01:38:11.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.909+0000 7f06cfa35740 1 -- 172.21.15.70:0/4210561394 wait complete. 2024-04-01T01:38:11.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.910+0000 7f06cfa35740 20 client.26749 trim_cache size 0 max 16384 2024-04-01T01:38:11.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.910+0000 7f06cce4e640 1 -- 172.21.15.70:0/596990721 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06b00a2600 msgr2=0x7f06b00a4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.910+0000 7f06cce4e640 1 --2- 172.21.15.70:0/596990721 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06b00a2600 0x7f06b00a4ac0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f06c0009270 tx=0x7f06c0004ab0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.910+0000 7f06cce4e640 1 -- 172.21.15.70:0/596990721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c80ff9a0 msgr2=0x7f06c81957f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:11.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.910+0000 7f06cce4e640 1 --2- 172.21.15.70:0/596990721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c80ff9a0 0x7f06c81957f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f06b8036040 tx=0x7f06b8004040 comp rx=0 tx=0).stop 2024-04-01T01:38:11.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.911+0000 7f06cce4e640 1 -- 172.21.15.70:0/596990721 shutdown_connections 2024-04-01T01:38:11.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.911+0000 7f06cce4e640 1 --2- 172.21.15.70:0/596990721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06c8101550 0x7f06c819cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.911+0000 7f06cce4e640 1 --2- 172.21.15.70:0/596990721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06c8100b90 0x7f06c8195d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.911+0000 7f06cce4e640 1 --2- 172.21.15.70:0/596990721 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06b00a2600 0x7f06b00a4ac0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.911+0000 7f06cce4e640 1 --2- 172.21.15.70:0/596990721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06c80ff9a0 0x7f06c81957f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:11.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.911+0000 7f06cce4e640 1 -- 172.21.15.70:0/596990721 >> 172.21.15.70:0/596990721 conn(0x7f06c80faf10 msgr2=0x7f06c8103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:11.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.912+0000 7f06cce4e640 1 -- 172.21.15.70:0/596990721 shutdown_connections 2024-04-01T01:38:11.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:11.913+0000 7f06cce4e640 1 -- 172.21.15.70:0/596990721 wait complete. 2024-04-01T01:38:11.932 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:38:11.977 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:11 smithi070 ceph-mon[32179]: pgmap v499: 129 pgs: 129 active+clean; 1.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 7.7 MiB/s wr, 897 op/s 2024-04-01T01:38:11.977 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:11 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1213960880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:38:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:11 smithi081 ceph-mon[35142]: pgmap v499: 129 pgs: 129 active+clean; 1.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 7.7 MiB/s wr, 897 op/s 2024-04-01T01:38:12.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:11 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1213960880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:38:12.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.125+0000 7f4e6759e640 1 -- 172.21.15.70:0/2367213794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e680ffea0 msgr2=0x7f4e681022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.125+0000 7f4e6759e640 1 --2- 172.21.15.70:0/2367213794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e680ffea0 0x7f4e681022d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4e58009900 tx=0x7f4e5802bc50 comp rx=0 tx=0).stop 2024-04-01T01:38:12.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.125+0000 7f4e6759e640 1 -- 172.21.15.70:0/2367213794 shutdown_connections 2024-04-01T01:38:12.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.125+0000 7f4e6759e640 1 --2- 172.21.15.70:0/2367213794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e681051c0 0x7f4e68105640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.125+0000 7f4e6759e640 1 --2- 172.21.15.70:0/2367213794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e68102810 0x7f4e68104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.125+0000 7f4e6759e640 1 --2- 172.21.15.70:0/2367213794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e680ffea0 0x7f4e681022d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.125+0000 7f4e6759e640 1 -- 172.21.15.70:0/2367213794 >> 172.21.15.70:0/2367213794 conn(0x7f4e680fbac0 msgr2=0x7f4e680fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.130+0000 7f4e6759e640 1 -- 172.21.15.70:0/2367213794 shutdown_connections 2024-04-01T01:38:12.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.130+0000 7f4e6759e640 1 -- 172.21.15.70:0/2367213794 wait complete. 2024-04-01T01:38:12.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.132+0000 7f4e6759e640 1 Processor -- start 2024-04-01T01:38:12.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.132+0000 7f4e6759e640 1 -- start start 2024-04-01T01:38:12.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.132+0000 7f4e6759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e680ffea0 0x7f4e6819dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.132+0000 7f4e6759e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e68102810 0x7f4e6819e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.133+0000 7f4e6659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e680ffea0 0x7f4e6819dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.133+0000 7f4e6659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e680ffea0 0x7f4e6819dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52724/0 (socket says 172.21.15.70:52724) 2024-04-01T01:38:12.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.133+0000 7f4e65d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e68102810 0x7f4e6819e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.133+0000 7f4e6759e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e681051c0 0x7f4e681a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.133+0000 7f4e65d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e68102810 0x7f4e6819e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48024/0 (socket says 172.21.15.70:48024) 2024-04-01T01:38:12.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.133+0000 7f4e6759e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e68113820 con 0x7f4e680ffea0 2024-04-01T01:38:12.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.133+0000 7f4e6659c640 1 -- 172.21.15.70:0/2021689431 learned_addr learned my addr 172.21.15.70:0/2021689431 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:12.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.133+0000 7f4e6759e640 1 -- 172.21.15.70:0/2021689431 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4e681136a0 con 0x7f4e68102810 2024-04-01T01:38:12.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.133+0000 7f4e66d9d640 1 --2- 172.21.15.70:0/2021689431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e681051c0 0x7f4e681a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.133+0000 7f4e6759e640 1 -- 172.21.15.70:0/2021689431 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4e681139a0 con 0x7f4e681051c0 2024-04-01T01:38:12.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.136+0000 7f4e6659c640 1 -- 172.21.15.70:0/2021689431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e681051c0 msgr2=0x7f4e681a5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.136+0000 7f4e6659c640 1 --2- 172.21.15.70:0/2021689431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e681051c0 0x7f4e681a5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.136+0000 7f4e6659c640 1 -- 172.21.15.70:0/2021689431 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e68102810 msgr2=0x7f4e6819e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.136+0000 7f4e6659c640 1 --2- 172.21.15.70:0/2021689431 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e68102810 0x7f4e6819e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.136+0000 7f4e6659c640 1 -- 172.21.15.70:0/2021689431 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e681a5970 con 0x7f4e680ffea0 2024-04-01T01:38:12.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.136+0000 7f4e6659c640 1 --2- 172.21.15.70:0/2021689431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e680ffea0 0x7f4e6819dda0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f4e58037040 tx=0x7f4e58004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.136+0000 7f4e65d9b640 1 --2- 172.21.15.70:0/2021689431 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e68102810 0x7f4e6819e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:38:12.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.136+0000 7f4e66d9d640 1 --2- 172.21.15.70:0/2021689431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e681051c0 0x7f4e681a5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:38:12.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.136+0000 7f4e577fe640 1 -- 172.21.15.70:0/2021689431 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e58039040 con 0x7f4e680ffea0 2024-04-01T01:38:12.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.137+0000 7f4e6759e640 1 -- 172.21.15.70:0/2021689431 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e681a5c00 con 0x7f4e680ffea0 2024-04-01T01:38:12.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.137+0000 7f4e6759e640 1 -- 172.21.15.70:0/2021689431 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e681a60f0 con 0x7f4e680ffea0 2024-04-01T01:38:12.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.137+0000 7f4e577fe640 1 -- 172.21.15.70:0/2021689431 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e58044070 con 0x7f4e680ffea0 2024-04-01T01:38:12.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.137+0000 7f4e577fe640 1 -- 172.21.15.70:0/2021689431 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e580035f0 con 0x7f4e680ffea0 2024-04-01T01:38:12.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.140+0000 7f4e6f9e6740 1 Processor -- start 2024-04-01T01:38:12.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.140+0000 7f4e6f9e6740 1 -- start start 2024-04-01T01:38:12.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.141+0000 7f4e6f9e6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c26a0 0x5592759c2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.141+0000 7f4e6f9e6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c3090 0x5592759c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.141+0000 7f4e6f9e6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c8200 0x5592759ca690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.141+0000 7f4e6f9e6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5592759271b0 con 0x5592759c8200 2024-04-01T01:38:12.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.141+0000 7f4e6f9e6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5592758cfc00 con 0x5592759c26a0 2024-04-01T01:38:12.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.141+0000 7f4e6f9e6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559275923250 con 0x5592759c3090 2024-04-01T01:38:12.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.141+0000 7f4e66d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c26a0 0x5592759c2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.141+0000 7f4e66d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c26a0 0x5592759c2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48038/0 (socket says 172.21.15.70:48038) 2024-04-01T01:38:12.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.141+0000 7f4e66d9d640 1 -- 172.21.15.70:0/2798056316 learned_addr learned my addr 172.21.15.70:0/2798056316 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:12.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.142+0000 7f4e65d9b640 1 --2- 172.21.15.70:0/2798056316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c8200 0x5592759ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.142+0000 7f4e577fe640 1 -- 172.21.15.70:0/2021689431 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f4e58003790 con 0x7f4e680ffea0 2024-04-01T01:38:12.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.142+0000 7f4e66d9d640 1 -- 172.21.15.70:0/2798056316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c3090 msgr2=0x5592759c7b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:38:12.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.142+0000 7f4e66d9d640 1 --2- 172.21.15.70:0/2798056316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c3090 0x5592759c7b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.142+0000 7f4e66d9d640 1 -- 172.21.15.70:0/2798056316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c8200 msgr2=0x5592759ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.142+0000 7f4e66d9d640 1 --2- 172.21.15.70:0/2798056316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c8200 0x5592759ca690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.142+0000 7f4e66d9d640 1 -- 172.21.15.70:0/2798056316 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592758ef3d0 con 0x5592759c26a0 2024-04-01T01:38:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.142+0000 7f4e66d9d640 1 --2- 172.21.15.70:0/2798056316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c26a0 0x5592759c2ac0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4e6000c450 tx=0x7f4e6000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=132249181354970 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.143+0000 7f4e577fe640 1 --2- 172.21.15.70:0/2021689431 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e500a2570 0x7f4e500a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.143+0000 7f4e577fe640 1 -- 172.21.15.70:0/2021689431 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f4e58047e00 con 0x7f4e680ffea0 2024-04-01T01:38:12.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.144+0000 7f4e6659c640 1 --2- 172.21.15.70:0/2021689431 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e500a2570 0x7f4e500a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.144+0000 7f4e54ff9640 1 -- 172.21.15.70:0/2798056316 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e60007430 con 0x5592759c26a0 2024-04-01T01:38:12.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.144+0000 7f4e54ff9640 1 -- 172.21.15.70:0/2798056316 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e60007a80 con 0x5592759c26a0 2024-04-01T01:38:12.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.144+0000 7f4e54ff9640 1 -- 172.21.15.70:0/2798056316 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6000ab40 con 0x5592759c26a0 2024-04-01T01:38:12.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.145+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/2798056316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c26a0 msgr2=0x5592759c2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.145+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/2798056316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c26a0 0x5592759c2ac0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4e6000c450 tx=0x7f4e6000c910 comp rx=0 tx=0).stop 2024-04-01T01:38:12.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.145+0000 7f4e6659c640 1 --2- 172.21.15.70:0/2021689431 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e500a2570 0x7f4e500a4a30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4e580ba7c0 tx=0x7f4e580dd040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.146+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/2798056316 shutdown_connections 2024-04-01T01:38:12.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.146+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/2798056316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c8200 0x5592759ca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.146+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/2798056316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c3090 0x5592759c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.146+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/2798056316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c26a0 0x5592759c2ac0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.146+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/2798056316 >> 172.21.15.70:0/2798056316 conn(0x5592759bdc10 msgr2=0x5592759be010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.147+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/2798056316 shutdown_connections 2024-04-01T01:38:12.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.147+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/2798056316 wait complete. 2024-04-01T01:38:12.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.148+0000 7f4e6f9e6740 1 -- start start 2024-04-01T01:38:12.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.148+0000 7f4e6f9e6740 1 Processor -- start 2024-04-01T01:38:12.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.149+0000 7f4e6f9e6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c26a0 0x5592759ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.149+0000 7f4e6f9e6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 0x5592759cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.149+0000 7f4e66d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c26a0 0x5592759ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.149+0000 7f4e66d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c26a0 0x5592759ccc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52756/0 (socket says 172.21.15.70:52756) 2024-04-01T01:38:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.149+0000 7f4e6f9e6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c8200 0x5592759d4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.149+0000 7f4e66d9d640 1 -- 172.21.15.70:0/1320647656 learned_addr learned my addr 172.21.15.70:0/1320647656 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.149+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/1320647656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5592759cb1b0 con 0x5592759c26a0 2024-04-01T01:38:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.149+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/1320647656 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5592758c5470 con 0x5592759c3090 2024-04-01T01:38:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.149+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/1320647656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5592759cb330 con 0x5592759c8200 2024-04-01T01:38:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.149+0000 7f4e65d9b640 1 --2- 172.21.15.70:0/1320647656 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 0x5592759cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.149+0000 7f4e6f9e6740 10 client.0 _ll_register_callbacks cb 0x5592758e2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:12.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.150+0000 7f4e66d9d640 1 --2- 172.21.15.70:0/1320647656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c8200 0x5592759d4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.150+0000 7f4e65d9b640 1 -- 172.21.15.70:0/1320647656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c8200 msgr2=0x5592759d4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.150+0000 7f4e65d9b640 1 --2- 172.21.15.70:0/1320647656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c8200 0x5592759d4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.150+0000 7f4e65d9b640 1 -- 172.21.15.70:0/1320647656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c26a0 msgr2=0x5592759ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.150+0000 7f4e65d9b640 1 --2- 172.21.15.70:0/1320647656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c26a0 0x5592759ccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.150+0000 7f4e65d9b640 1 -- 172.21.15.70:0/1320647656 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55927563b3f0 con 0x5592759c3090 2024-04-01T01:38:12.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.151+0000 7f4e65d9b640 1 --2- 172.21.15.70:0/1320647656 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 0x5592759cd150 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4e6819f260 tx=0x7f4e5c011cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cc826123d58e536 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.151+0000 7f4e4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:12.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.151+0000 7f4e4dffb640 1 -- 172.21.15.70:0/1320647656 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c00fe40 con 0x5592759c3090 2024-04-01T01:38:12.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.151+0000 7f4e4dffb640 1 -- 172.21.15.70:0/1320647656 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f4e5c004020 con 0x5592759c3090 2024-04-01T01:38:12.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.151+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/1320647656 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e60007a80 con 0x5592759c3090 2024-04-01T01:38:12.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.152+0000 7f4e4dffb640 1 -- 172.21.15.70:0/1320647656 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4e5c004220 con 0x5592759c3090 2024-04-01T01:38:12.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.152+0000 7f4e6f9e6740 10 client.26852 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:12.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.152+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/1320647656 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5592757057d0 con 0x5592759c3090 2024-04-01T01:38:12.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.153+0000 7f4e4dffb640 1 -- 172.21.15.70:0/1320647656 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e5c0043c0 con 0x5592759c3090 2024-04-01T01:38:12.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.153+0000 7f4e4dffb640 1 -- 172.21.15.70:0/1320647656 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4e5c018030 con 0x5592759c3090 2024-04-01T01:38:12.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.153+0000 7f4e6f9e6740 10 client.26852 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:12.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.153+0000 7f4e6f9e6740 10 client.26852 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:38:12.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.153+0000 7f4e6f9e6740 20 client.26852 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:12.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.153+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/1320647656 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5592759d86f0 0x5592759dab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.153+0000 7f4e6f9e6740 4 client.26852 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:12.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.153+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/1320647656 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4e60007a80 con 0x5592759d86f0 2024-04-01T01:38:12.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.153+0000 7f4e66d9d640 1 --2- 172.21.15.70:0/1320647656 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5592759d86f0 0x5592759dab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.154+0000 7f4e4dffb640 10 client.26852 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:38:12.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.154+0000 7f4e66d9d640 1 --2- 172.21.15.70:0/1320647656 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5592759d86f0 0x5592759dab90 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=314d6e315fc5e4ae server_cookie=1eaed8d2d77f071e in_seq=0 out_seq=0 2024-04-01T01:38:12.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.156+0000 7f4e4dffb640 1 -- 172.21.15.70:0/1320647656 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f4e60007a80 con 0x5592759d86f0 2024-04-01T01:38:12.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.156+0000 7f4e4dffb640 10 client.26852 handle_command_reply: tid=0 2024-04-01T01:38:12.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.156+0000 7f4e6f9e6740 1 client.26852 shutdown 2024-04-01T01:38:12.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.160+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/1320647656 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 msgr2=0x5592759cd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.160+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/1320647656 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 0x5592759cd150 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4e6819f260 tx=0x7f4e5c011cb0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.162+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/1320647656 shutdown_connections 2024-04-01T01:38:12.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.162+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/1320647656 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5592759d86f0 0x5592759dab90 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.162+0000 7f4e4dffb640 0 client.26852 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:38:12.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.162+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/1320647656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c8200 0x5592759d4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.162+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/1320647656 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 0x5592759cd150 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.162+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/1320647656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c26a0 0x5592759ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.162+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/1320647656 >> 172.21.15.70:0/1320647656 conn(0x5592759bdc10 msgr2=0x5592759bcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.163+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/1320647656 shutdown_connections 2024-04-01T01:38:12.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.163+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/1320647656 wait complete. 2024-04-01T01:38:12.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.163+0000 7f4e6f9e6740 20 client.26852 trim_cache size 0 max 16384 2024-04-01T01:38:12.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.192+0000 7f4e6f9e6740 1 Processor -- start 2024-04-01T01:38:12.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.192+0000 7f4e6f9e6740 1 -- start start 2024-04-01T01:38:12.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.192+0000 7f4e6f9e6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c26a0 0x5592759c4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.192+0000 7f4e6f9e6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 0x5592759c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.192+0000 7f4e6f9e6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c8200 0x5592759d4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.192+0000 7f4e66d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c26a0 0x5592759c4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.192+0000 7f4e66d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c26a0 0x5592759c4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36288/0 (socket says 172.21.15.70:36288) 2024-04-01T01:38:12.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.192+0000 7f4e66d9d640 1 -- 172.21.15.70:0/133019830 learned_addr learned my addr 172.21.15.70:0/133019830 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:12.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.192+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/133019830 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4800fda0 con 0x5592759c8200 2024-04-01T01:38:12.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.193+0000 7f4e65d9b640 1 --2- 172.21.15.70:0/133019830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 0x5592759c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.193+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/133019830 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4e480118e0 con 0x5592759c3090 2024-04-01T01:38:12.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.193+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/133019830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4e48010470 con 0x5592759c26a0 2024-04-01T01:38:12.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.193+0000 7f4e66d9d640 1 --2- 172.21.15.70:0/133019830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c8200 0x5592759d4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.193+0000 7f4e66d9d640 1 -- 172.21.15.70:0/133019830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 msgr2=0x5592759c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.193+0000 7f4e66d9d640 1 --2- 172.21.15.70:0/133019830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 0x5592759c4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.193+0000 7f4e66d9d640 1 -- 172.21.15.70:0/133019830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c8200 msgr2=0x5592759d4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.193+0000 7f4e66d9d640 1 --2- 172.21.15.70:0/133019830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c8200 0x5592759d4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.193+0000 7f4e66d9d640 1 -- 172.21.15.70:0/133019830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592759cdc00 con 0x5592759c26a0 2024-04-01T01:38:12.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.193+0000 7f4e65d9b640 1 --2- 172.21.15.70:0/133019830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 0x5592759c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:38:12.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.194+0000 7f4e66d9d640 1 --2- 172.21.15.70:0/133019830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c26a0 0x5592759c4590 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4e60009790 tx=0x7f4e600096e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3526c23f329b5ead server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.195+0000 7f4e4ffff640 1 -- 172.21.15.70:0/133019830 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6001b040 con 0x5592759c26a0 2024-04-01T01:38:12.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.195+0000 7f4e4ffff640 1 -- 172.21.15.70:0/133019830 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e600260d0 con 0x5592759c26a0 2024-04-01T01:38:12.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.195+0000 7f4e4ffff640 1 -- 172.21.15.70:0/133019830 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6000ae20 con 0x5592759c26a0 2024-04-01T01:38:12.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.196+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/133019830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c26a0 msgr2=0x5592759c4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.196+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/133019830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c26a0 0x5592759c4590 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4e60009790 tx=0x7f4e600096e0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.197+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/133019830 shutdown_connections 2024-04-01T01:38:12.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.197+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/133019830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c8200 0x5592759d4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.197+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/133019830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 0x5592759c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.197+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/133019830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c26a0 0x5592759c4590 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.197+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/133019830 >> 172.21.15.70:0/133019830 conn(0x5592759bdc10 msgr2=0x5592759cb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.198+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/133019830 shutdown_connections 2024-04-01T01:38:12.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.198+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/133019830 wait complete. 2024-04-01T01:38:12.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.198+0000 7f4e6f9e6740 1 -- start start 2024-04-01T01:38:12.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.199+0000 7f4e6f9e6740 1 Processor -- start 2024-04-01T01:38:12.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.199+0000 7f4e6f9e6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c26a0 0x5592759d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.199+0000 7f4e6f9e6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 0x5592759d1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.199+0000 7f4e6f9e6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c8200 0x5592759d2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.199+0000 7f4e6f9e6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e48011210 con 0x5592759c8200 2024-04-01T01:38:12.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.199+0000 7f4e6f9e6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4e48010b40 con 0x5592759c3090 2024-04-01T01:38:12.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.199+0000 7f4e6f9e6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4800dd80 con 0x5592759c26a0 2024-04-01T01:38:12.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.199+0000 7f4e6f9e6740 10 client.0 _ll_register_callbacks cb 0x7f4e5c011460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:12.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.200+0000 7f4e66d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c26a0 0x5592759d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.200+0000 7f4e65d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 0x5592759d1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.200+0000 7f4e66d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c26a0 0x5592759d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36302/0 (socket says 172.21.15.70:36302) 2024-04-01T01:38:12.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.200+0000 7f4e66d9d640 1 -- 172.21.15.70:0/4282657723 learned_addr learned my addr 172.21.15.70:0/4282657723 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:12.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.200+0000 7f4e66d9d640 1 --2- 172.21.15.70:0/4282657723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c8200 0x5592759d2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.200+0000 7f4e66d9d640 1 -- 172.21.15.70:0/4282657723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 msgr2=0x5592759d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.200+0000 7f4e66d9d640 1 --2- 172.21.15.70:0/4282657723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 0x5592759d1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.201+0000 7f4e66d9d640 1 -- 172.21.15.70:0/4282657723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c8200 msgr2=0x5592759d2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.201+0000 7f4e66d9d640 1 --2- 172.21.15.70:0/4282657723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c8200 0x5592759d2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.201+0000 7f4e66d9d640 1 -- 172.21.15.70:0/4282657723 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592759d3440 con 0x5592759c26a0 2024-04-01T01:38:12.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.201+0000 7f4e66d9d640 1 --2- 172.21.15.70:0/4282657723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c26a0 0x5592759d1900 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4e60000f80 tx=0x7f4e60009dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aed9028717089090 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.201+0000 7f4e4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:38:12.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.201+0000 7f4e4dffb640 1 -- 172.21.15.70:0/4282657723 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6001b040 con 0x5592759c26a0 2024-04-01T01:38:12.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.202+0000 7f4e4dffb640 1 -- 172.21.15.70:0/4282657723 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f4e6000adb0 con 0x5592759c26a0 2024-04-01T01:38:12.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.202+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/4282657723 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e600260d0 con 0x5592759c26a0 2024-04-01T01:38:12.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.202+0000 7f4e4dffb640 1 -- 172.21.15.70:0/4282657723 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4e60026510 con 0x5592759c26a0 2024-04-01T01:38:12.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.202+0000 7f4e6f9e6740 10 client.26761 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:12.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.202+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/4282657723 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559275a4fae0 con 0x5592759c26a0 2024-04-01T01:38:12.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.204+0000 7f4e4dffb640 1 -- 172.21.15.70:0/4282657723 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e600266b0 con 0x5592759c26a0 2024-04-01T01:38:12.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.205+0000 7f4e4dffb640 1 -- 172.21.15.70:0/4282657723 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4e60030040 con 0x5592759c26a0 2024-04-01T01:38:12.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.205+0000 7f4e6f9e6740 10 client.26761 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:12.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.205+0000 7f4e6f9e6740 10 client.26761 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:38:12.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.205+0000 7f4e6f9e6740 20 client.26761 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:12.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.205+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/4282657723 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5592759d86f0 0x559275a52230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.206+0000 7f4e6f9e6740 4 client.26761 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:38:12.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.206+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/4282657723 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f4e600260d0 con 0x5592759d86f0 2024-04-01T01:38:12.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.206+0000 7f4e65d9b640 1 --2- 172.21.15.70:0/4282657723 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5592759d86f0 0x559275a52230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.209+0000 7f4e65d9b640 1 --2- 172.21.15.70:0/4282657723 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5592759d86f0 0x559275a52230 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c4681fb00e5392d4 server_cookie=109d9a02f35e6090 in_seq=0 out_seq=0 2024-04-01T01:38:12.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.211+0000 7f4e4dffb640 10 client.26761 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:38:12.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.211+0000 7f4e4dffb640 1 -- 172.21.15.70:0/4282657723 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f4e600260d0 con 0x5592759d86f0 2024-04-01T01:38:12.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.211+0000 7f4e4dffb640 10 client.26761 handle_command_reply: tid=0 2024-04-01T01:38:12.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.211+0000 7f4e6f9e6740 1 client.26761 shutdown 2024-04-01T01:38:12.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.212+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/4282657723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c26a0 msgr2=0x5592759d1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.212+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/4282657723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c26a0 0x5592759d1900 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4e60000f80 tx=0x7f4e60009dc0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.212+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/4282657723 shutdown_connections 2024-04-01T01:38:12.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.212+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/4282657723 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5592759d86f0 0x559275a52230 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.212+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/4282657723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592759c8200 0x5592759d2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.212+0000 7f4e4dffb640 0 client.26761 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:38:12.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.212+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/4282657723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592759c3090 0x5592759d1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.212+0000 7f4e6f9e6740 1 --2- 172.21.15.70:0/4282657723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592759c26a0 0x5592759d1900 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.212+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/4282657723 >> 172.21.15.70:0/4282657723 conn(0x5592759bdc10 msgr2=0x5592759ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.213+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/4282657723 shutdown_connections 2024-04-01T01:38:12.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.213+0000 7f4e6f9e6740 1 -- 172.21.15.70:0/4282657723 wait complete. 2024-04-01T01:38:12.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.215+0000 7f4e6f9e6740 20 client.26761 trim_cache size 0 max 16384 2024-04-01T01:38:12.217 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:38:12.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.216+0000 7f4e6759e640 1 -- 172.21.15.70:0/2021689431 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e500a2570 msgr2=0x7f4e500a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.216+0000 7f4e6759e640 1 --2- 172.21.15.70:0/2021689431 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e500a2570 0x7f4e500a4a30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4e580ba7c0 tx=0x7f4e580dd040 comp rx=0 tx=0).stop 2024-04-01T01:38:12.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.216+0000 7f4e6759e640 1 -- 172.21.15.70:0/2021689431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e680ffea0 msgr2=0x7f4e6819dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.216+0000 7f4e6759e640 1 --2- 172.21.15.70:0/2021689431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e680ffea0 0x7f4e6819dda0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f4e58037040 tx=0x7f4e58004270 comp rx=0 tx=0).stop 2024-04-01T01:38:12.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.216+0000 7f4e6759e640 1 -- 172.21.15.70:0/2021689431 shutdown_connections 2024-04-01T01:38:12.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.216+0000 7f4e6759e640 1 --2- 172.21.15.70:0/2021689431 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e500a2570 0x7f4e500a4a30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.216+0000 7f4e6759e640 1 --2- 172.21.15.70:0/2021689431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e681051c0 0x7f4e681a5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.216+0000 7f4e6759e640 1 --2- 172.21.15.70:0/2021689431 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e68102810 0x7f4e6819e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.216+0000 7f4e6759e640 1 --2- 172.21.15.70:0/2021689431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e680ffea0 0x7f4e6819dda0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.216+0000 7f4e6759e640 1 -- 172.21.15.70:0/2021689431 >> 172.21.15.70:0/2021689431 conn(0x7f4e680fbac0 msgr2=0x7f4e681031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.217+0000 7f4e6759e640 1 -- 172.21.15.70:0/2021689431 shutdown_connections 2024-04-01T01:38:12.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.217+0000 7f4e6759e640 1 -- 172.21.15.70:0/2021689431 wait complete. 2024-04-01T01:38:12.237 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:38:12.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:11 smithi189 ceph-mon[35753]: pgmap v499: 129 pgs: 129 active+clean; 1.0 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 7.7 MiB/s wr, 897 op/s 2024-04-01T01:38:12.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:11 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1213960880' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:38:12.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.457+0000 7f61bbc6b640 1 -- 172.21.15.70:0/3205588736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b4108750 msgr2=0x7f61b4108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.457+0000 7f61bbc6b640 1 --2- 172.21.15.70:0/3205588736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b4108750 0x7f61b4108b70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009900 tx=0x7f61b002bc30 comp rx=0 tx=0).stop 2024-04-01T01:38:12.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.460+0000 7f61bbc6b640 1 -- 172.21.15.70:0/3205588736 shutdown_connections 2024-04-01T01:38:12.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.460+0000 7f61bbc6b640 1 --2- 172.21.15.70:0/3205588736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61b410ffd0 0x7f61b41123c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.460+0000 7f61bbc6b640 1 --2- 172.21.15.70:0/3205588736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61b41090b0 0x7f61b410fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.460+0000 7f61bbc6b640 1 --2- 172.21.15.70:0/3205588736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b4108750 0x7f61b4108b70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.460+0000 7f61bbc6b640 1 -- 172.21.15.70:0/3205588736 >> 172.21.15.70:0/3205588736 conn(0x7f61b406e950 msgr2=0x7f61b4070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.462+0000 7f61bbc6b640 1 -- 172.21.15.70:0/3205588736 shutdown_connections 2024-04-01T01:38:12.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.464+0000 7f61bbc6b640 1 -- 172.21.15.70:0/3205588736 wait complete. 2024-04-01T01:38:12.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.465+0000 7f61bbc6b640 1 Processor -- start 2024-04-01T01:38:12.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.465+0000 7f61bbc6b640 1 -- start start 2024-04-01T01:38:12.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.466+0000 7f61bbc6b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61b4108750 0x7f61b419df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.466+0000 7f61bbc6b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61b41090b0 0x7f61b419e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.466+0000 7f61bbc6b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b410ffd0 0x7f61b41a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.466+0000 7f61bbc6b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61b41138d0 con 0x7f61b41090b0 2024-04-01T01:38:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.466+0000 7f61bbc6b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f61b4113750 con 0x7f61b4108750 2024-04-01T01:38:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.466+0000 7f61ba208640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b410ffd0 0x7f61b41a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.466+0000 7f61ba208640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b410ffd0 0x7f61b41a5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36314/0 (socket says 172.21.15.70:36314) 2024-04-01T01:38:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.467+0000 7f61ba208640 1 -- 172.21.15.70:0/358154931 learned_addr learned my addr 172.21.15.70:0/358154931 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:12.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.467+0000 7f61bbc6b640 1 -- 172.21.15.70:0/358154931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f61b4113a50 con 0x7f61b410ffd0 2024-04-01T01:38:12.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.467+0000 7f61ba208640 1 -- 172.21.15.70:0/358154931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61b4108750 msgr2=0x7f61b419df60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:38:12.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.467+0000 7f61ba208640 1 --2- 172.21.15.70:0/358154931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61b4108750 0x7f61b419df60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.467+0000 7f61ba208640 1 -- 172.21.15.70:0/358154931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61b41090b0 msgr2=0x7f61b419e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.467+0000 7f61ba208640 1 --2- 172.21.15.70:0/358154931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61b41090b0 0x7f61b419e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.467+0000 7f61ba208640 1 -- 172.21.15.70:0/358154931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b41a5b80 con 0x7f61b410ffd0 2024-04-01T01:38:12.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.468+0000 7f61ba208640 1 --2- 172.21.15.70:0/358154931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b410ffd0 0x7f61b41a5510 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00edb0 tx=0x7f61ac012490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.468+0000 7f61aaffd640 1 -- 172.21.15.70:0/358154931 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac01c030 con 0x7f61b410ffd0 2024-04-01T01:38:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.468+0000 7f61aaffd640 1 -- 172.21.15.70:0/358154931 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61ac012db0 con 0x7f61b410ffd0 2024-04-01T01:38:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.468+0000 7f61bbc6b640 1 -- 172.21.15.70:0/358154931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b41a5e70 con 0x7f61b410ffd0 2024-04-01T01:38:12.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.468+0000 7f61bbc6b640 1 -- 172.21.15.70:0/358154931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b41a6330 con 0x7f61b410ffd0 2024-04-01T01:38:12.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.470+0000 7f61aaffd640 1 -- 172.21.15.70:0/358154931 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac018400 con 0x7f61b410ffd0 2024-04-01T01:38:12.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.475+0000 7f61aaffd640 1 -- 172.21.15.70:0/358154931 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f61ac01f070 con 0x7f61b410ffd0 2024-04-01T01:38:12.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.475+0000 7f61be846740 1 Processor -- start 2024-04-01T01:38:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.475+0000 7f61be846740 1 -- start start 2024-04-01T01:38:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.476+0000 7f61be846740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91490 0x561504c918b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.476+0000 7f61be846740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c91e80 0x561504c96980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.476+0000 7f61be846740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c96ff0 0x561504c99480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.476+0000 7f61be846740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561504bf61b0 con 0x561504c96ff0 2024-04-01T01:38:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.476+0000 7f61be846740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561504b9ec00 con 0x561504c91e80 2024-04-01T01:38:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.476+0000 7f61be846740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561504bf2250 con 0x561504c91490 2024-04-01T01:38:12.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.476+0000 7f61b9a07640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c91e80 0x561504c96980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.476+0000 7f61b9a07640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c91e80 0x561504c96980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48084/0 (socket says 172.21.15.70:48084) 2024-04-01T01:38:12.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.476+0000 7f61b9a07640 1 -- 172.21.15.70:0/3388134864 learned_addr learned my addr 172.21.15.70:0/3388134864 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:12.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.476+0000 7f61aaffd640 1 --2- 172.21.15.70:0/358154931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61a00a25f0 0x7f61a00a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.476+0000 7f61b9a07640 1 -- 172.21.15.70:0/3388134864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91490 msgr2=0x561504c918b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:38:12.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.476+0000 7f61b9a07640 1 --2- 172.21.15.70:0/3388134864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91490 0x561504c918b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.476+0000 7f61b9a07640 1 -- 172.21.15.70:0/3388134864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c96ff0 msgr2=0x561504c99480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:38:12.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.477+0000 7f61b9206640 1 --2- 172.21.15.70:0/3388134864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c96ff0 0x561504c99480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.477+0000 7f61b9a07640 1 --2- 172.21.15.70:0/3388134864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c96ff0 0x561504c99480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.477+0000 7f61b9a07640 1 -- 172.21.15.70:0/3388134864 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561504bbe3d0 con 0x561504c91e80 2024-04-01T01:38:12.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.477+0000 7f61aaffd640 1 -- 172.21.15.70:0/358154931 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f61ac0d2ab0 con 0x7f61b410ffd0 2024-04-01T01:38:12.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.478+0000 7f61b9a07640 1 --2- 172.21.15.70:0/3388134864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c91e80 0x561504c96980 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f61b002c170 tx=0x7f61b0005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68d7e244c74f66 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.478+0000 7f619ffff640 1 -- 172.21.15.70:0/3388134864 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b0042020 con 0x561504c91e80 2024-04-01T01:38:12.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.479+0000 7f619ffff640 1 -- 172.21.15.70:0/3388134864 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61b00053c0 con 0x561504c91e80 2024-04-01T01:38:12.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.479+0000 7f61ba208640 1 --2- 172.21.15.70:0/358154931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61a00a25f0 0x7f61a00a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.479+0000 7f61b9206640 1 --2- 172.21.15.70:0/3388134864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c96ff0 0x561504c99480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:38:12.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.480+0000 7f619ffff640 1 -- 172.21.15.70:0/3388134864 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b0005220 con 0x561504c91e80 2024-04-01T01:38:12.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.480+0000 7f61be846740 1 -- 172.21.15.70:0/3388134864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c91e80 msgr2=0x561504c96980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.480+0000 7f61ba208640 1 --2- 172.21.15.70:0/358154931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61a00a25f0 0x7f61a00a4ab0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f61ac09ee40 tx=0x7f61ac005b30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.480+0000 7f61be846740 1 --2- 172.21.15.70:0/3388134864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c91e80 0x561504c96980 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f61b002c170 tx=0x7f61b0005b50 comp rx=0 tx=0).stop 2024-04-01T01:38:12.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.480+0000 7f61be846740 1 -- 172.21.15.70:0/3388134864 shutdown_connections 2024-04-01T01:38:12.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.480+0000 7f61be846740 1 --2- 172.21.15.70:0/3388134864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c96ff0 0x561504c99480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.480+0000 7f61be846740 1 --2- 172.21.15.70:0/3388134864 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c91e80 0x561504c96980 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.480+0000 7f61be846740 1 --2- 172.21.15.70:0/3388134864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91490 0x561504c918b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.480+0000 7f61be846740 1 -- 172.21.15.70:0/3388134864 >> 172.21.15.70:0/3388134864 conn(0x561504c8ca00 msgr2=0x561504c8ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.481+0000 7f61be846740 1 -- 172.21.15.70:0/3388134864 shutdown_connections 2024-04-01T01:38:12.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.481+0000 7f61be846740 1 -- 172.21.15.70:0/3388134864 wait complete. 2024-04-01T01:38:12.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.482+0000 7f61be846740 1 -- start start 2024-04-01T01:38:12.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.482+0000 7f61be846740 1 Processor -- start 2024-04-01T01:38:12.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.483+0000 7f61be846740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c91490 0x561504c9ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.483+0000 7f61be846740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91e80 0x561504c9bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.483+0000 7f61be846740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c96ff0 0x561504ca2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.483+0000 7f61be846740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561504c99f90 con 0x561504c96ff0 2024-04-01T01:38:12.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.483+0000 7f61be846740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561504b94470 con 0x561504c91490 2024-04-01T01:38:12.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.483+0000 7f61be846740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561504c9a110 con 0x561504c91e80 2024-04-01T01:38:12.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.483+0000 7f61be846740 10 client.0 _ll_register_callbacks cb 0x561504bb1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:12.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.484+0000 7f61b9206640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c91490 0x561504c9ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.484+0000 7f61b9206640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c91490 0x561504c9ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48090/0 (socket says 172.21.15.70:48090) 2024-04-01T01:38:12.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.484+0000 7f61b9206640 1 -- 172.21.15.70:0/2617493206 learned_addr learned my addr 172.21.15.70:0/2617493206 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:12.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.485+0000 7f61b9206640 1 -- 172.21.15.70:0/2617493206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91e80 msgr2=0x561504c9bf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:38:12.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.485+0000 7f61b9206640 1 --2- 172.21.15.70:0/2617493206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91e80 0x561504c9bf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.485+0000 7f61b9206640 1 -- 172.21.15.70:0/2617493206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c96ff0 msgr2=0x561504ca2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.485+0000 7f61b9206640 1 --2- 172.21.15.70:0/2617493206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c96ff0 0x561504ca2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.485+0000 7f61b9206640 1 -- 172.21.15.70:0/2617493206 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56150490a3f0 con 0x561504c91490 2024-04-01T01:38:12.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.486+0000 7f61b9206640 1 --2- 172.21.15.70:0/2617493206 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c91490 0x561504c9ba20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f61a400be00 tx=0x7f61a40109c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83bb48e2ea714ba3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.486+0000 7f619d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:12.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.486+0000 7f619d7fa640 1 -- 172.21.15.70:0/2617493206 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a4005220 con 0x561504c91490 2024-04-01T01:38:12.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.486+0000 7f61be846740 1 -- 172.21.15.70:0/2617493206 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61b00053c0 con 0x561504c91490 2024-04-01T01:38:12.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.486+0000 7f619d7fa640 1 -- 172.21.15.70:0/2617493206 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f61a40053c0 con 0x561504c91490 2024-04-01T01:38:12.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.487+0000 7f619d7fa640 1 -- 172.21.15.70:0/2617493206 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f61a4022050 con 0x561504c91490 2024-04-01T01:38:12.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.487+0000 7f61be846740 10 client.26867 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:12.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.487+0000 7f61be846740 1 -- 172.21.15.70:0/2617493206 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5615049d47d0 con 0x561504c91490 2024-04-01T01:38:12.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.487+0000 7f619d7fa640 1 -- 172.21.15.70:0/2617493206 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61b00053c0 con 0x561504c91490 2024-04-01T01:38:12.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.488+0000 7f619d7fa640 1 -- 172.21.15.70:0/2617493206 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f61a400f040 con 0x561504c91490 2024-04-01T01:38:12.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.488+0000 7f61be846740 10 client.26867 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:12.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.488+0000 7f61be846740 10 client.26867 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:38:12.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.488+0000 7f61be846740 20 client.26867 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:12.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.488+0000 7f61be846740 1 --2- 172.21.15.70:0/2617493206 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x561504ca73b0 0x561504ca9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.488+0000 7f61be846740 4 client.26867 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:12.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.488+0000 7f61be846740 1 -- 172.21.15.70:0/2617493206 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561504c036f0 con 0x561504ca73b0 2024-04-01T01:38:12.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.488+0000 7f61b9a07640 1 --2- 172.21.15.70:0/2617493206 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x561504ca73b0 0x561504ca9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.491+0000 7f61b9a07640 1 --2- 172.21.15.70:0/2617493206 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x561504ca73b0 0x561504ca9830 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d23e55a2abda925 server_cookie=636235a57bd8e087 in_seq=0 out_seq=0 2024-04-01T01:38:12.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.491+0000 7f619d7fa640 10 client.26867 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:38:12.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.492+0000 7f619d7fa640 1 -- 172.21.15.70:0/2617493206 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561504c036f0 con 0x561504ca73b0 2024-04-01T01:38:12.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.492+0000 7f619d7fa640 10 client.26867 handle_command_reply: tid=0 2024-04-01T01:38:12.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.492+0000 7f61be846740 1 client.26867 shutdown 2024-04-01T01:38:12.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.493+0000 7f61be846740 1 -- 172.21.15.70:0/2617493206 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c91490 msgr2=0x561504c9ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.493+0000 7f61be846740 1 --2- 172.21.15.70:0/2617493206 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c91490 0x561504c9ba20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f61a400be00 tx=0x7f61a40109c0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.493+0000 7f61be846740 1 -- 172.21.15.70:0/2617493206 shutdown_connections 2024-04-01T01:38:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.493+0000 7f61be846740 1 --2- 172.21.15.70:0/2617493206 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x561504ca73b0 0x561504ca9830 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.493+0000 7f61be846740 1 --2- 172.21.15.70:0/2617493206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c96ff0 0x561504ca2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.493+0000 7f61be846740 1 --2- 172.21.15.70:0/2617493206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91e80 0x561504c9bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.493+0000 7f61be846740 1 --2- 172.21.15.70:0/2617493206 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c91490 0x561504c9ba20 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.493+0000 7f61be846740 1 -- 172.21.15.70:0/2617493206 >> 172.21.15.70:0/2617493206 conn(0x561504c8ca00 msgr2=0x561504c8bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.493+0000 7f619d7fa640 0 client.26867 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:38:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.493+0000 7f61be846740 1 -- 172.21.15.70:0/2617493206 shutdown_connections 2024-04-01T01:38:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.493+0000 7f61be846740 1 -- 172.21.15.70:0/2617493206 wait complete. 2024-04-01T01:38:12.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.493+0000 7f61be846740 20 client.26867 trim_cache size 0 max 16384 2024-04-01T01:38:12.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.520+0000 7f61be846740 1 Processor -- start 2024-04-01T01:38:12.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.520+0000 7f61be846740 1 -- start start 2024-04-01T01:38:12.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.521+0000 7f61be846740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c91490 0x561504c9bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.521+0000 7f61be846740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91e80 0x561504ca2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.522+0000 7f61b9a07640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91e80 0x561504ca2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.522+0000 7f61b9206640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c91490 0x561504c9bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.522+0000 7f61b9a07640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91e80 0x561504ca2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36320/0 (socket says 172.21.15.70:36320) 2024-04-01T01:38:12.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.522+0000 7f61b9a07640 1 -- 172.21.15.70:0/1518028832 learned_addr learned my addr 172.21.15.70:0/1518028832 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:12.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.522+0000 7f61be846740 1 --2- 172.21.15.70:0/1518028832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c96ff0 0x561504c8b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.522+0000 7f61be846740 1 -- 172.21.15.70:0/1518028832 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61b41b0b30 con 0x561504c91490 2024-04-01T01:38:12.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.522+0000 7f61be846740 1 -- 172.21.15.70:0/1518028832 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f61b41b2670 con 0x561504c96ff0 2024-04-01T01:38:12.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.522+0000 7f61be846740 1 -- 172.21.15.70:0/1518028832 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f61b41b1200 con 0x561504c91e80 2024-04-01T01:38:12.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.523+0000 7f61b9a07640 1 -- 172.21.15.70:0/1518028832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c96ff0 msgr2=0x561504c8b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.523+0000 7f61b9a07640 1 --2- 172.21.15.70:0/1518028832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c96ff0 0x561504c8b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.523+0000 7f61b9a07640 1 -- 172.21.15.70:0/1518028832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c91490 msgr2=0x561504c9bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.523+0000 7f61b9a07640 1 --2- 172.21.15.70:0/1518028832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c91490 0x561504c9bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.523+0000 7f61b9a07640 1 -- 172.21.15.70:0/1518028832 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561504c910d0 con 0x561504c91e80 2024-04-01T01:38:12.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.523+0000 7f61b9a07640 1 --2- 172.21.15.70:0/1518028832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91e80 0x561504ca2f60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009230 tx=0x7f61b0005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e59ce033a61b84c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.524+0000 7f619f7fe640 1 -- 172.21.15.70:0/1518028832 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b0042040 con 0x561504c91e80 2024-04-01T01:38:12.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.524+0000 7f619f7fe640 1 -- 172.21.15.70:0/1518028832 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61b0005850 con 0x561504c91e80 2024-04-01T01:38:12.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.524+0000 7f619f7fe640 1 -- 172.21.15.70:0/1518028832 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b0004ae0 con 0x561504c91e80 2024-04-01T01:38:12.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.525+0000 7f61be846740 1 -- 172.21.15.70:0/1518028832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91e80 msgr2=0x561504ca2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.525+0000 7f61be846740 1 --2- 172.21.15.70:0/1518028832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91e80 0x561504ca2f60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009230 tx=0x7f61b0005e50 comp rx=0 tx=0).stop 2024-04-01T01:38:12.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.525+0000 7f61be846740 1 -- 172.21.15.70:0/1518028832 shutdown_connections 2024-04-01T01:38:12.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.525+0000 7f61be846740 1 --2- 172.21.15.70:0/1518028832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c96ff0 0x561504c8b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.525+0000 7f61be846740 1 --2- 172.21.15.70:0/1518028832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91e80 0x561504ca2f60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.526+0000 7f61be846740 1 --2- 172.21.15.70:0/1518028832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c91490 0x561504c9bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.526+0000 7f61be846740 1 -- 172.21.15.70:0/1518028832 >> 172.21.15.70:0/1518028832 conn(0x561504c8ca00 msgr2=0x561504c9b630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.526+0000 7f61be846740 1 -- 172.21.15.70:0/1518028832 shutdown_connections 2024-04-01T01:38:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.526+0000 7f61be846740 1 -- 172.21.15.70:0/1518028832 wait complete. 2024-04-01T01:38:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.526+0000 7f61be846740 1 -- start start 2024-04-01T01:38:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.526+0000 7f61be846740 1 Processor -- start 2024-04-01T01:38:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.527+0000 7f61be846740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91490 0x561504ca0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.527+0000 7f61be846740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c91e80 0x561504ca11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.527+0000 7f61be846740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c96ff0 0x561504ca2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.527+0000 7f61be846740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61b41b1fa0 con 0x561504c91e80 2024-04-01T01:38:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.527+0000 7f61be846740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f61b41b18d0 con 0x561504c96ff0 2024-04-01T01:38:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.527+0000 7f61be846740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f61b41aeb10 con 0x561504c91490 2024-04-01T01:38:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.527+0000 7f61be846740 10 client.0 _ll_register_callbacks cb 0x7f61a400bcc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.527+0000 7f61b9a07640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c91e80 0x561504ca11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.527+0000 7f61b9206640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91490 0x561504ca0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.527+0000 7f61b9206640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91490 0x561504ca0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36334/0 (socket says 172.21.15.70:36334) 2024-04-01T01:38:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.527+0000 7f61b9a07640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c91e80 0x561504ca11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52826/0 (socket says 172.21.15.70:52826) 2024-04-01T01:38:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.527+0000 7f61b9206640 1 -- 172.21.15.70:0/3022135023 learned_addr learned my addr 172.21.15.70:0/3022135023 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.527+0000 7f61b9206640 1 --2- 172.21.15.70:0/3022135023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c96ff0 0x561504ca2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.528+0000 7f61b9206640 1 -- 172.21.15.70:0/3022135023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91490 msgr2=0x561504ca0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.528+0000 7f61b9206640 1 --2- 172.21.15.70:0/3022135023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91490 0x561504ca0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.528+0000 7f61b9206640 1 -- 172.21.15.70:0/3022135023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c91e80 msgr2=0x561504ca11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.528+0000 7f61b9206640 1 --2- 172.21.15.70:0/3022135023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c91e80 0x561504ca11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.528+0000 7f61b9206640 1 -- 172.21.15.70:0/3022135023 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561504ca2760 con 0x561504c96ff0 2024-04-01T01:38:12.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.530+0000 7f61b9206640 1 --2- 172.21.15.70:0/3022135023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c96ff0 0x561504ca2170 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f61a4004700 tx=0x7f61a400b5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2198974acbbb0c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.530+0000 7f619d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:12.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.530+0000 7f619d7fa640 1 -- 172.21.15.70:0/3022135023 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a402b070 con 0x561504c96ff0 2024-04-01T01:38:12.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.530+0000 7f61be846740 1 -- 172.21.15.70:0/3022135023 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61b0005850 con 0x561504c96ff0 2024-04-01T01:38:12.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.530+0000 7f619d7fa640 1 -- 172.21.15.70:0/3022135023 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f61a401a900 con 0x561504c96ff0 2024-04-01T01:38:12.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.530+0000 7f619d7fa640 1 -- 172.21.15.70:0/3022135023 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f61a4039050 con 0x561504c96ff0 2024-04-01T01:38:12.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.530+0000 7f61be846740 10 client.26870 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:12.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.530+0000 7f61be846740 1 -- 172.21.15.70:0/3022135023 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561504d0dcf0 con 0x561504c96ff0 2024-04-01T01:38:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.531+0000 7f619d7fa640 1 -- 172.21.15.70:0/3022135023 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61b0005850 con 0x561504c96ff0 2024-04-01T01:38:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.531+0000 7f619d7fa640 1 -- 172.21.15.70:0/3022135023 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f61a4027990 con 0x561504c96ff0 2024-04-01T01:38:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.531+0000 7f61be846740 10 client.26870 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.531+0000 7f61be846740 10 client.26870 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:38:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.531+0000 7f61be846740 20 client.26870 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.531+0000 7f61be846740 1 --2- 172.21.15.70:0/3022135023 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x561504d0e630 0x561504d10a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.531+0000 7f61be846740 4 client.26870 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:38:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.531+0000 7f61be846740 1 -- 172.21.15.70:0/3022135023 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561504bfda40 con 0x561504d0e630 2024-04-01T01:38:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.531+0000 7f61b9a07640 1 --2- 172.21.15.70:0/3022135023 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x561504d0e630 0x561504d10a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.532+0000 7f61b9a07640 1 --2- 172.21.15.70:0/3022135023 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x561504d0e630 0x561504d10a50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa55a03f2dc4a3c6 server_cookie=46551de362e71b5b in_seq=0 out_seq=0 2024-04-01T01:38:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.532+0000 7f619d7fa640 10 client.26870 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:38:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.532+0000 7f619d7fa640 1 -- 172.21.15.70:0/3022135023 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561504bfda40 con 0x561504d0e630 2024-04-01T01:38:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.532+0000 7f619d7fa640 10 client.26870 handle_command_reply: tid=0 2024-04-01T01:38:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.532+0000 7f61be846740 1 client.26870 shutdown 2024-04-01T01:38:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.533+0000 7f61be846740 1 -- 172.21.15.70:0/3022135023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c96ff0 msgr2=0x561504ca2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.533+0000 7f61be846740 1 --2- 172.21.15.70:0/3022135023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c96ff0 0x561504ca2170 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f61a4004700 tx=0x7f61a400b5f0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.533+0000 7f61be846740 1 -- 172.21.15.70:0/3022135023 shutdown_connections 2024-04-01T01:38:12.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.533+0000 7f61be846740 1 --2- 172.21.15.70:0/3022135023 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x561504d0e630 0x561504d10a50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.533+0000 7f61be846740 1 --2- 172.21.15.70:0/3022135023 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561504c96ff0 0x561504ca2170 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.533+0000 7f61be846740 1 --2- 172.21.15.70:0/3022135023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561504c91e80 0x561504ca11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.533+0000 7f61be846740 1 --2- 172.21.15.70:0/3022135023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561504c91490 0x561504ca0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.533+0000 7f61be846740 1 -- 172.21.15.70:0/3022135023 >> 172.21.15.70:0/3022135023 conn(0x561504c8ca00 msgr2=0x561504c8fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.533+0000 7f619d7fa640 0 client.26870 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:38:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.533+0000 7f61be846740 1 -- 172.21.15.70:0/3022135023 shutdown_connections 2024-04-01T01:38:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.534+0000 7f61be846740 1 -- 172.21.15.70:0/3022135023 wait complete. 2024-04-01T01:38:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.534+0000 7f61be846740 20 client.26870 trim_cache size 0 max 16384 2024-04-01T01:38:12.534 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:38:12.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.535+0000 7f61bbc6b640 1 -- 172.21.15.70:0/358154931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61a00a25f0 msgr2=0x7f61a00a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.535+0000 7f61bbc6b640 1 --2- 172.21.15.70:0/358154931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61a00a25f0 0x7f61a00a4ab0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f61ac09ee40 tx=0x7f61ac005b30 comp rx=0 tx=0).stop 2024-04-01T01:38:12.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.535+0000 7f61bbc6b640 1 -- 172.21.15.70:0/358154931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b410ffd0 msgr2=0x7f61b41a5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.535+0000 7f61bbc6b640 1 --2- 172.21.15.70:0/358154931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b410ffd0 0x7f61b41a5510 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00edb0 tx=0x7f61ac012490 comp rx=0 tx=0).stop 2024-04-01T01:38:12.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.535+0000 7f61bbc6b640 1 -- 172.21.15.70:0/358154931 shutdown_connections 2024-04-01T01:38:12.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.535+0000 7f61bbc6b640 1 --2- 172.21.15.70:0/358154931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61a00a25f0 0x7f61a00a4ab0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.535+0000 7f61bbc6b640 1 --2- 172.21.15.70:0/358154931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61b410ffd0 0x7f61b41a5510 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.535+0000 7f61bbc6b640 1 --2- 172.21.15.70:0/358154931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61b41090b0 0x7f61b419e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.535+0000 7f61bbc6b640 1 --2- 172.21.15.70:0/358154931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61b4108750 0x7f61b419df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.535+0000 7f61bbc6b640 1 -- 172.21.15.70:0/358154931 >> 172.21.15.70:0/358154931 conn(0x7f61b406e950 msgr2=0x7f61b410ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.535+0000 7f61bbc6b640 1 -- 172.21.15.70:0/358154931 shutdown_connections 2024-04-01T01:38:12.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.535+0000 7f61bbc6b640 1 -- 172.21.15.70:0/358154931 wait complete. 2024-04-01T01:38:12.551 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:38:12.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.735+0000 7f0edd815640 1 -- 172.21.15.70:0/3521784738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ed8108f80 msgr2=0x7f0ed8109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.735+0000 7f0edd815640 1 --2- 172.21.15.70:0/3521784738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ed8108f80 0x7f0ed8109400 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc009900 tx=0x7f0ecc02bc50 comp rx=0 tx=0).stop 2024-04-01T01:38:12.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.736+0000 7f0edd815640 1 -- 172.21.15.70:0/3521784738 shutdown_connections 2024-04-01T01:38:12.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.736+0000 7f0edd815640 1 --2- 172.21.15.70:0/3521784738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ed8109940 0x7f0ed81101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.736+0000 7f0edd815640 1 --2- 172.21.15.70:0/3521784738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ed8108f80 0x7f0ed8109400 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.736+0000 7f0edd815640 1 --2- 172.21.15.70:0/3521784738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ed8107d90 0x7f0ed81081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.736+0000 7f0edd815640 1 -- 172.21.15.70:0/3521784738 >> 172.21.15.70:0/3521784738 conn(0x7f0ed8076d20 msgr2=0x7f0ed8079160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.736+0000 7f0edd815640 1 -- 172.21.15.70:0/3521784738 shutdown_connections 2024-04-01T01:38:12.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.736+0000 7f0edd815640 1 -- 172.21.15.70:0/3521784738 wait complete. 2024-04-01T01:38:12.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.737+0000 7f0edd815640 1 Processor -- start 2024-04-01T01:38:12.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.737+0000 7f0edd815640 1 -- start start 2024-04-01T01:38:12.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.737+0000 7f0edd815640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ed8107d90 0x7f0ed819e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.737+0000 7f0edd815640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ed8108f80 0x7f0ed819e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.737+0000 7f0edd815640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ed8109940 0x7f0ed81a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.737+0000 7f0edd815640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed8113990 con 0x7f0ed8109940 2024-04-01T01:38:12.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.737+0000 7f0edd815640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed8113810 con 0x7f0ed8108f80 2024-04-01T01:38:12.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.737+0000 7f0edd815640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed8113b10 con 0x7f0ed8107d90 2024-04-01T01:38:12.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.738+0000 7f0ed67fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ed8108f80 0x7f0ed819e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.738+0000 7f0ed6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ed8107d90 0x7f0ed819e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.738+0000 7f0ed67fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ed8108f80 0x7f0ed819e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48120/0 (socket says 172.21.15.70:48120) 2024-04-01T01:38:12.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.738+0000 7f0ed77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ed8109940 0x7f0ed81a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.738+0000 7f0ed67fc640 1 -- 172.21.15.70:0/2891333632 learned_addr learned my addr 172.21.15.70:0/2891333632 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:12.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.738+0000 7f0ed77fe640 1 -- 172.21.15.70:0/2891333632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ed8107d90 msgr2=0x7f0ed819e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.738+0000 7f0ed77fe640 1 --2- 172.21.15.70:0/2891333632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ed8107d90 0x7f0ed819e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.738+0000 7f0ed77fe640 1 -- 172.21.15.70:0/2891333632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ed8108f80 msgr2=0x7f0ed819e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.738+0000 7f0ed77fe640 1 --2- 172.21.15.70:0/2891333632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ed8108f80 0x7f0ed819e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.738+0000 7f0ed77fe640 1 -- 172.21.15.70:0/2891333632 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed81a5c90 con 0x7f0ed8109940 2024-04-01T01:38:12.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.738+0000 7f0ed6ffd640 1 --2- 172.21.15.70:0/2891333632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ed8107d90 0x7f0ed819e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:38:12.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.738+0000 7f0ed77fe640 1 --2- 172.21.15.70:0/2891333632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ed8109940 0x7f0ed81a5620 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800b3e0 tx=0x7f0ec800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.739+0000 7f0edc813640 1 -- 172.21.15.70:0/2891333632 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8015020 con 0x7f0ed8109940 2024-04-01T01:38:12.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.739+0000 7f0edc813640 1 -- 172.21.15.70:0/2891333632 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ec8011ce0 con 0x7f0ed8109940 2024-04-01T01:38:12.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.739+0000 7f0edc813640 1 -- 172.21.15.70:0/2891333632 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8010e00 con 0x7f0ed8109940 2024-04-01T01:38:12.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.739+0000 7f0edd815640 1 -- 172.21.15.70:0/2891333632 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed81a5f80 con 0x7f0ed8109940 2024-04-01T01:38:12.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.739+0000 7f0edd815640 1 -- 172.21.15.70:0/2891333632 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ed81a63b0 con 0x7f0ed8109940 2024-04-01T01:38:12.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.743+0000 7f0edc813640 1 -- 172.21.15.70:0/2891333632 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f0ec8018020 con 0x7f0ed8109940 2024-04-01T01:38:12.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.744+0000 7f0edc813640 1 --2- 172.21.15.70:0/2891333632 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ec00a26a0 0x7f0ec00a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.744+0000 7f0edc813640 1 -- 172.21.15.70:0/2891333632 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0ec80ce190 con 0x7f0ed8109940 2024-04-01T01:38:12.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.744+0000 7f0ed6ffd640 1 --2- 172.21.15.70:0/2891333632 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ec00a26a0 0x7f0ec00a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.745+0000 7f0ed6ffd640 1 --2- 172.21.15.70:0/2891333632 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ec00a26a0 0x7f0ec00a4b60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00099a0 tx=0x7f0ed0009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.745+0000 7f0ee03f6740 1 Processor -- start 2024-04-01T01:38:12.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.747+0000 7f0ee03f6740 1 -- start start 2024-04-01T01:38:12.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.748+0000 7f0ee03f6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5490 0x556b2e6b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.748+0000 7f0ee03f6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6b5e80 0x556b2e6ba980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.748+0000 7f0ee03f6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6baff0 0x556b2e6bd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.748+0000 7f0ee03f6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556b2e61a1b0 con 0x556b2e6b5490 2024-04-01T01:38:12.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.748+0000 7f0ed67fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6b5e80 0x556b2e6ba980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.748+0000 7f0ed67fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6b5e80 0x556b2e6ba980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36366/0 (socket says 172.21.15.70:36366) 2024-04-01T01:38:12.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.748+0000 7f0ed67fc640 1 -- 172.21.15.70:0/4283700006 learned_addr learned my addr 172.21.15.70:0/4283700006 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:12.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.748+0000 7f0ee03f6740 1 -- 172.21.15.70:0/4283700006 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x556b2e5c2c00 con 0x556b2e6baff0 2024-04-01T01:38:12.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.748+0000 7f0ee03f6740 1 -- 172.21.15.70:0/4283700006 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556b2e616250 con 0x556b2e6b5e80 2024-04-01T01:38:12.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.748+0000 7f0ed6ffd640 1 --2- 172.21.15.70:0/4283700006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5490 0x556b2e6b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.749+0000 7f0ed6ffd640 1 -- 172.21.15.70:0/4283700006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6b5e80 msgr2=0x556b2e6ba980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.749+0000 7f0ed6ffd640 1 --2- 172.21.15.70:0/4283700006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6b5e80 0x556b2e6ba980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.749+0000 7f0ed6ffd640 1 -- 172.21.15.70:0/4283700006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6baff0 msgr2=0x556b2e6bd480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:38:12.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.749+0000 7f0ed6ffd640 1 --2- 172.21.15.70:0/4283700006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6baff0 0x556b2e6bd480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.749+0000 7f0ed6ffd640 1 -- 172.21.15.70:0/4283700006 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b2e5e23d0 con 0x556b2e6b5490 2024-04-01T01:38:12.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.750+0000 7f0ed6ffd640 1 --2- 172.21.15.70:0/4283700006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5490 0x556b2e6b58b0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8108de0 tx=0x7f0ed0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c95acb1a2a30628b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.750+0000 7f0ebdffb640 1 -- 172.21.15.70:0/4283700006 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed000fc80 con 0x556b2e6b5490 2024-04-01T01:38:12.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.750+0000 7f0ebdffb640 1 -- 172.21.15.70:0/4283700006 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ed0003e20 con 0x556b2e6b5490 2024-04-01T01:38:12.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.750+0000 7f0ebdffb640 1 -- 172.21.15.70:0/4283700006 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed00047a0 con 0x556b2e6b5490 2024-04-01T01:38:12.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.751+0000 7f0ee03f6740 1 -- 172.21.15.70:0/4283700006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5490 msgr2=0x556b2e6b58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.751+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/4283700006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5490 0x556b2e6b58b0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8108de0 tx=0x7f0ed0014000 comp rx=0 tx=0).stop 2024-04-01T01:38:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.752+0000 7f0ee03f6740 1 -- 172.21.15.70:0/4283700006 shutdown_connections 2024-04-01T01:38:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.752+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/4283700006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6baff0 0x556b2e6bd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.752+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/4283700006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6b5e80 0x556b2e6ba980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.752+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/4283700006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5490 0x556b2e6b58b0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.752+0000 7f0ee03f6740 1 -- 172.21.15.70:0/4283700006 >> 172.21.15.70:0/4283700006 conn(0x556b2e6b0a00 msgr2=0x556b2e6b0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.752+0000 7f0ee03f6740 1 -- 172.21.15.70:0/4283700006 shutdown_connections 2024-04-01T01:38:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.752+0000 7f0ee03f6740 1 -- 172.21.15.70:0/4283700006 wait complete. 2024-04-01T01:38:12.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.753+0000 7f0ee03f6740 1 -- start start 2024-04-01T01:38:12.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.753+0000 7f0ee03f6740 1 Processor -- start 2024-04-01T01:38:12.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.753+0000 7f0ee03f6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5490 0x556b2e6bf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.753+0000 7f0ee03f6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6b5e80 0x556b2e6bfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.753+0000 7f0ee03f6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6baff0 0x556b2e6c6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.753+0000 7f0ee03f6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556b2e6bdfa0 con 0x556b2e6baff0 2024-04-01T01:38:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.753+0000 7f0ee03f6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x556b2e5b8470 con 0x556b2e6b5490 2024-04-01T01:38:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.753+0000 7f0ee03f6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556b2e6be120 con 0x556b2e6b5e80 2024-04-01T01:38:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.754+0000 7f0ee03f6740 10 client.0 _ll_register_callbacks cb 0x556b2e5d5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.754+0000 7f0ed6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5490 0x556b2e6bf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.754+0000 7f0ed6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5490 0x556b2e6bf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48136/0 (socket says 172.21.15.70:48136) 2024-04-01T01:38:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.754+0000 7f0ed6ffd640 1 -- 172.21.15.70:0/4147346083 learned_addr learned my addr 172.21.15.70:0/4147346083 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.755+0000 7f0ed67fc640 1 --2- 172.21.15.70:0/4147346083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6b5e80 0x556b2e6bfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.755+0000 7f0ed6ffd640 1 -- 172.21.15.70:0/4147346083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6b5e80 msgr2=0x556b2e6bfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.755+0000 7f0ed6ffd640 1 --2- 172.21.15.70:0/4147346083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6b5e80 0x556b2e6bfed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.755+0000 7f0ed6ffd640 1 -- 172.21.15.70:0/4147346083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6baff0 msgr2=0x556b2e6c6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.755+0000 7f0ed6ffd640 1 --2- 172.21.15.70:0/4147346083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6baff0 0x556b2e6c6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.755+0000 7f0ed6ffd640 1 -- 172.21.15.70:0/4147346083 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b2e32e3f0 con 0x556b2e6b5490 2024-04-01T01:38:12.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.756+0000 7f0ed6ffd640 1 --2- 172.21.15.70:0/4147346083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5490 0x556b2e6bf990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000a010 tx=0x7f0ed0003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=808c003f4c9a9e9c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.756+0000 7f0eb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:12.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.756+0000 7f0eb6ffd640 1 -- 172.21.15.70:0/4147346083 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0046060 con 0x556b2e6b5490 2024-04-01T01:38:12.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.756+0000 7f0eb6ffd640 1 -- 172.21.15.70:0/4147346083 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0ed00046b0 con 0x556b2e6b5490 2024-04-01T01:38:12.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.756+0000 7f0ee03f6740 1 -- 172.21.15.70:0/4147346083 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ed0003e20 con 0x556b2e6b5490 2024-04-01T01:38:12.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.756+0000 7f0eb6ffd640 1 -- 172.21.15.70:0/4147346083 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0ed004d050 con 0x556b2e6b5490 2024-04-01T01:38:12.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.756+0000 7f0ee03f6740 10 client.26879 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:12.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.756+0000 7f0ee03f6740 1 -- 172.21.15.70:0/4147346083 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556b2e3f87d0 con 0x556b2e6b5490 2024-04-01T01:38:12.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.757+0000 7f0eb6ffd640 1 -- 172.21.15.70:0/4147346083 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ed0003e20 con 0x556b2e6b5490 2024-04-01T01:38:12.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.757+0000 7f0eb6ffd640 1 -- 172.21.15.70:0/4147346083 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0ed0005640 con 0x556b2e6b5490 2024-04-01T01:38:12.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.757+0000 7f0ee03f6740 10 client.26879 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:12.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.757+0000 7f0ee03f6740 10 client.26879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:12.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.757+0000 7f0ee03f6740 20 client.26879 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.757+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/4147346083 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556b2e6cb770 0x556b2e6cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.757+0000 7f0ee03f6740 4 client.26879 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.757+0000 7f0ee03f6740 1 -- 172.21.15.70:0/4147346083 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b2e6276f0 con 0x556b2e6cb770 2024-04-01T01:38:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.758+0000 7f0ed67fc640 1 --2- 172.21.15.70:0/4147346083 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556b2e6cb770 0x556b2e6cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.758+0000 7f0ed67fc640 1 --2- 172.21.15.70:0/4147346083 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556b2e6cb770 0x556b2e6cdc10 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d13024d29ac5c369 server_cookie=5057e37432ce7b3 in_seq=0 out_seq=0 2024-04-01T01:38:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.759+0000 7f0eb6ffd640 10 client.26879 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.759+0000 7f0eb6ffd640 1 -- 172.21.15.70:0/4147346083 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556b2e6276f0 con 0x556b2e6cb770 2024-04-01T01:38:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.759+0000 7f0eb6ffd640 10 client.26879 handle_command_reply: tid=0 2024-04-01T01:38:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.759+0000 7f0ee03f6740 1 client.26879 shutdown 2024-04-01T01:38:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.760+0000 7f0ee03f6740 1 -- 172.21.15.70:0/4147346083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5490 msgr2=0x556b2e6bf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.760+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/4147346083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5490 0x556b2e6bf990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000a010 tx=0x7f0ed0003ba0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.760+0000 7f0ee03f6740 1 -- 172.21.15.70:0/4147346083 shutdown_connections 2024-04-01T01:38:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.760+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/4147346083 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556b2e6cb770 0x556b2e6cdc10 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.760+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/4147346083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6baff0 0x556b2e6c6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.760+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/4147346083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6b5e80 0x556b2e6bfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.760+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/4147346083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5490 0x556b2e6bf990 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.760+0000 7f0ee03f6740 1 -- 172.21.15.70:0/4147346083 >> 172.21.15.70:0/4147346083 conn(0x556b2e6b0a00 msgr2=0x556b2e6afd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.760+0000 7f0eb6ffd640 0 client.26879 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:12.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.760+0000 7f0ee03f6740 1 -- 172.21.15.70:0/4147346083 shutdown_connections 2024-04-01T01:38:12.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.760+0000 7f0ee03f6740 1 -- 172.21.15.70:0/4147346083 wait complete. 2024-04-01T01:38:12.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.760+0000 7f0ee03f6740 20 client.26879 trim_cache size 0 max 16384 2024-04-01T01:38:12.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.787+0000 7f0ee03f6740 1 Processor -- start 2024-04-01T01:38:12.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.788+0000 7f0ee03f6740 1 -- start start 2024-04-01T01:38:12.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.788+0000 7f0ee03f6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5490 0x556b2e6bfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.789+0000 7f0ee03f6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5e80 0x556b2e6c6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.789+0000 7f0ed6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5490 0x556b2e6bfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.789+0000 7f0ed6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5490 0x556b2e6bfe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52874/0 (socket says 172.21.15.70:52874) 2024-04-01T01:38:12.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.789+0000 7f0ed6ffd640 1 -- 172.21.15.70:0/546924232 learned_addr learned my addr 172.21.15.70:0/546924232 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:12.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.789+0000 7f0ed67fc640 1 --2- 172.21.15.70:0/546924232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5e80 0x556b2e6c6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.789+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/546924232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6baff0 0x556b2e6abbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.789+0000 7f0ee03f6740 1 -- 172.21.15.70:0/546924232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed81a8300 con 0x556b2e6b5490 2024-04-01T01:38:12.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.789+0000 7f0ee03f6740 1 -- 172.21.15.70:0/546924232 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed81a9e40 con 0x556b2e6b5e80 2024-04-01T01:38:12.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.789+0000 7f0ee03f6740 1 -- 172.21.15.70:0/546924232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed81a89d0 con 0x556b2e6baff0 2024-04-01T01:38:12.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.790+0000 7f0ed6ffd640 1 -- 172.21.15.70:0/546924232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6baff0 msgr2=0x556b2e6abbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.790+0000 7f0ed77fe640 1 --2- 172.21.15.70:0/546924232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6baff0 0x556b2e6abbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.790+0000 7f0ed6ffd640 1 --2- 172.21.15.70:0/546924232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6baff0 0x556b2e6abbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.790+0000 7f0ed6ffd640 1 -- 172.21.15.70:0/546924232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5e80 msgr2=0x556b2e6c6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.790+0000 7f0ed6ffd640 1 --2- 172.21.15.70:0/546924232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5e80 0x556b2e6c6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.790+0000 7f0ed6ffd640 1 -- 172.21.15.70:0/546924232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b2e6b66e0 con 0x556b2e6b5490 2024-04-01T01:38:12.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.790+0000 7f0ed77fe640 1 --2- 172.21.15.70:0/546924232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6baff0 0x556b2e6abbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:38:12.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.790+0000 7f0ed67fc640 1 --2- 172.21.15.70:0/546924232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5e80 0x556b2e6c6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:38:12.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.790+0000 7f0ed6ffd640 1 --2- 172.21.15.70:0/546924232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5490 0x556b2e6bfe30 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000ed10 tx=0x7f0ed0004200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=494d5747c40622f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.791+0000 7f0ebd7fa640 1 -- 172.21.15.70:0/546924232 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0046040 con 0x556b2e6b5490 2024-04-01T01:38:12.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.791+0000 7f0ebd7fa640 1 -- 172.21.15.70:0/546924232 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ed001edc0 con 0x556b2e6b5490 2024-04-01T01:38:12.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.791+0000 7f0ebd7fa640 1 -- 172.21.15.70:0/546924232 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed00046b0 con 0x556b2e6b5490 2024-04-01T01:38:12.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.792+0000 7f0ee03f6740 1 -- 172.21.15.70:0/546924232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5490 msgr2=0x556b2e6bfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.792+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/546924232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5490 0x556b2e6bfe30 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000ed10 tx=0x7f0ed0004200 comp rx=0 tx=0).stop 2024-04-01T01:38:12.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.792+0000 7f0ee03f6740 1 -- 172.21.15.70:0/546924232 shutdown_connections 2024-04-01T01:38:12.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.792+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/546924232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6baff0 0x556b2e6abbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.792+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/546924232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5e80 0x556b2e6c6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.792+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/546924232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5490 0x556b2e6bfe30 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.792+0000 7f0ee03f6740 1 -- 172.21.15.70:0/546924232 >> 172.21.15.70:0/546924232 conn(0x556b2e6b0a00 msgr2=0x556b2e6b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.793+0000 7f0ee03f6740 1 -- 172.21.15.70:0/546924232 shutdown_connections 2024-04-01T01:38:12.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.793+0000 7f0ee03f6740 1 -- 172.21.15.70:0/546924232 wait complete. 2024-04-01T01:38:12.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.793+0000 7f0ee03f6740 1 -- start start 2024-04-01T01:38:12.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.794+0000 7f0ee03f6740 1 Processor -- start 2024-04-01T01:38:12.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.794+0000 7f0ee03f6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5490 0x556b2e6c4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.794+0000 7f0ee03f6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5e80 0x556b2e6c4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.794+0000 7f0ee03f6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6baff0 0x556b2e6c5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.794+0000 7f0ee03f6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed81a9770 con 0x556b2e6b5e80 2024-04-01T01:38:12.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.794+0000 7f0ee03f6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed81a90a0 con 0x556b2e6b5490 2024-04-01T01:38:12.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.794+0000 7f0ee03f6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed80823f0 con 0x556b2e6baff0 2024-04-01T01:38:12.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.795+0000 7f0ed6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5490 0x556b2e6c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.795+0000 7f0ed6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5490 0x556b2e6c4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48150/0 (socket says 172.21.15.70:48150) 2024-04-01T01:38:12.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.795+0000 7f0ed6ffd640 1 -- 172.21.15.70:0/2153924320 learned_addr learned my addr 172.21.15.70:0/2153924320 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:12.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.795+0000 7f0ed77fe640 1 --2- 172.21.15.70:0/2153924320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6baff0 0x556b2e6c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.795+0000 7f0ee03f6740 10 client.0 _ll_register_callbacks cb 0x7f0ed000fb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:12.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.795+0000 7f0ed67fc640 1 --2- 172.21.15.70:0/2153924320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5e80 0x556b2e6c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.795+0000 7f0ed6ffd640 1 -- 172.21.15.70:0/2153924320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6baff0 msgr2=0x556b2e6c5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.795+0000 7f0ed6ffd640 1 --2- 172.21.15.70:0/2153924320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6baff0 0x556b2e6c5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.795+0000 7f0ed6ffd640 1 -- 172.21.15.70:0/2153924320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5e80 msgr2=0x556b2e6c4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:12.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.795+0000 7f0ed6ffd640 1 --2- 172.21.15.70:0/2153924320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5e80 0x556b2e6c4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.795+0000 7f0ed6ffd640 1 -- 172.21.15.70:0/2153924320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b2e6c5f40 con 0x556b2e6b5490 2024-04-01T01:38:12.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.795+0000 7f0ed77fe640 1 --2- 172.21.15.70:0/2153924320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6baff0 0x556b2e6c5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:38:12.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.796+0000 7f0ed6ffd640 1 --2- 172.21.15.70:0/2153924320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5490 0x556b2e6c4400 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000a010 tx=0x7f0ed0005980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8686a06a4702f22c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:12.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.796+0000 7f0eb6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:12.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.796+0000 7f0eb6ffd640 1 -- 172.21.15.70:0/2153924320 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0046040 con 0x556b2e6b5490 2024-04-01T01:38:12.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.796+0000 7f0eb6ffd640 1 -- 172.21.15.70:0/2153924320 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0ed0009040 con 0x556b2e6b5490 2024-04-01T01:38:12.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.797+0000 7f0ee03f6740 1 -- 172.21.15.70:0/2153924320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ed001edc0 con 0x556b2e6b5490 2024-04-01T01:38:12.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.797+0000 7f0eb6ffd640 1 -- 172.21.15.70:0/2153924320 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0ed0003b80 con 0x556b2e6b5490 2024-04-01T01:38:12.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.798+0000 7f0ee03f6740 10 client.26885 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:12.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.798+0000 7f0ee03f6740 1 -- 172.21.15.70:0/2153924320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556b2e6c71b0 con 0x556b2e6b5490 2024-04-01T01:38:12.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.798+0000 7f0eb6ffd640 1 -- 172.21.15.70:0/2153924320 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ed0003d20 con 0x556b2e6b5490 2024-04-01T01:38:12.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.799+0000 7f0eb6ffd640 1 -- 172.21.15.70:0/2153924320 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0ed005cbf0 con 0x556b2e6b5490 2024-04-01T01:38:12.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.800+0000 7f0ee03f6740 10 client.26885 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:12.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.800+0000 7f0ee03f6740 10 client.26885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:12.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.800+0000 7f0ee03f6740 20 client.26885 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:12.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.800+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/2153924320 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556b2e7320d0 0x556b2e7344f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:12.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.800+0000 7f0ee03f6740 4 client.26885 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:38:12.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.800+0000 7f0ee03f6740 1 -- 172.21.15.70:0/2153924320 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f0ed001edc0 con 0x556b2e7320d0 2024-04-01T01:38:12.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.800+0000 7f0ed67fc640 1 --2- 172.21.15.70:0/2153924320 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556b2e7320d0 0x556b2e7344f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:12.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.801+0000 7f0ed67fc640 1 --2- 172.21.15.70:0/2153924320 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556b2e7320d0 0x556b2e7344f0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26d84a87e8a9ce75 server_cookie=e3502302b7182186 in_seq=0 out_seq=0 2024-04-01T01:38:12.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.801+0000 7f0eb6ffd640 10 client.26885 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:12.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.801+0000 7f0eb6ffd640 1 -- 172.21.15.70:0/2153924320 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f0ed001edc0 con 0x556b2e7320d0 2024-04-01T01:38:12.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.801+0000 7f0eb6ffd640 10 client.26885 handle_command_reply: tid=0 2024-04-01T01:38:12.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.801+0000 7f0ee03f6740 1 client.26885 shutdown 2024-04-01T01:38:12.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.802+0000 7f0ee03f6740 1 -- 172.21.15.70:0/2153924320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5490 msgr2=0x556b2e6c4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.802+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/2153924320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5490 0x556b2e6c4400 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000a010 tx=0x7f0ed0005980 comp rx=0 tx=0).stop 2024-04-01T01:38:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.803+0000 7f0ee03f6740 1 -- 172.21.15.70:0/2153924320 shutdown_connections 2024-04-01T01:38:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.803+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/2153924320 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556b2e7320d0 0x556b2e7344f0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.803+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/2153924320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b2e6baff0 0x556b2e6c5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.803+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/2153924320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556b2e6b5e80 0x556b2e6c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.803+0000 7f0ee03f6740 1 --2- 172.21.15.70:0/2153924320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b2e6b5490 0x556b2e6c4400 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.803+0000 7f0ee03f6740 1 -- 172.21.15.70:0/2153924320 >> 172.21.15.70:0/2153924320 conn(0x556b2e6b0a00 msgr2=0x556b2e6abbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.803+0000 7f0eb6ffd640 0 client.26885 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.803+0000 7f0ee03f6740 1 -- 172.21.15.70:0/2153924320 shutdown_connections 2024-04-01T01:38:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.803+0000 7f0ee03f6740 1 -- 172.21.15.70:0/2153924320 wait complete. 2024-04-01T01:38:12.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.803+0000 7f0ee03f6740 20 client.26885 trim_cache size 0 max 16384 2024-04-01T01:38:12.806 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:38:12.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.808+0000 7f0edd815640 1 -- 172.21.15.70:0/2891333632 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ec00a26a0 msgr2=0x7f0ec00a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.808+0000 7f0edd815640 1 --2- 172.21.15.70:0/2891333632 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ec00a26a0 0x7f0ec00a4b60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00099a0 tx=0x7f0ed0009500 comp rx=0 tx=0).stop 2024-04-01T01:38:12.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.808+0000 7f0edd815640 1 -- 172.21.15.70:0/2891333632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ed8109940 msgr2=0x7f0ed81a5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:12.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.808+0000 7f0edd815640 1 --2- 172.21.15.70:0/2891333632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ed8109940 0x7f0ed81a5620 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800b3e0 tx=0x7f0ec800b8a0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.809+0000 7f0edd815640 1 -- 172.21.15.70:0/2891333632 shutdown_connections 2024-04-01T01:38:12.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.809+0000 7f0edd815640 1 --2- 172.21.15.70:0/2891333632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ed8109940 0x7f0ed81a5620 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.809+0000 7f0edd815640 1 --2- 172.21.15.70:0/2891333632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ed8108f80 0x7f0ed819e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.809+0000 7f0edd815640 1 --2- 172.21.15.70:0/2891333632 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0ec00a26a0 0x7f0ec00a4b60 secure :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00099a0 tx=0x7f0ed0009500 comp rx=0 tx=0).stop 2024-04-01T01:38:12.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.809+0000 7f0edd815640 1 --2- 172.21.15.70:0/2891333632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ed8107d90 0x7f0ed819e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:12.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.809+0000 7f0edd815640 1 -- 172.21.15.70:0/2891333632 >> 172.21.15.70:0/2891333632 conn(0x7f0ed8076d20 msgr2=0x7f0ed810c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:12.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.810+0000 7f0edd815640 1 -- 172.21.15.70:0/2891333632 shutdown_connections 2024-04-01T01:38:12.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:12.811+0000 7f0edd815640 1 -- 172.21.15.70:0/2891333632 wait complete. 2024-04-01T01:38:13.835 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:38:13.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:38:13.836 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:38:14.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.021+0000 7f2a226d4640 1 -- 172.21.15.70:0/2429031472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c101840 msgr2=0x7f2a1c101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.021+0000 7f2a226d4640 1 --2- 172.21.15.70:0/2429031472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c101840 0x7f2a1c101cc0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009860 tx=0x7f2a1002bbb0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.021+0000 7f2a226d4640 1 -- 172.21.15.70:0/2429031472 shutdown_connections 2024-04-01T01:38:14.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.021+0000 7f2a226d4640 1 --2- 172.21.15.70:0/2429031472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1c102200 0x7f2a1c108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.021+0000 7f2a226d4640 1 --2- 172.21.15.70:0/2429031472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c101840 0x7f2a1c101cc0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.021+0000 7f2a226d4640 1 --2- 172.21.15.70:0/2429031472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a1c100650 0x7f2a1c100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.021+0000 7f2a226d4640 1 -- 172.21.15.70:0/2429031472 >> 172.21.15.70:0/2429031472 conn(0x7f2a1c0fbbe0 msgr2=0x7f2a1c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:14.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.021+0000 7f2a226d4640 1 -- 172.21.15.70:0/2429031472 shutdown_connections 2024-04-01T01:38:14.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.021+0000 7f2a226d4640 1 -- 172.21.15.70:0/2429031472 wait complete. 2024-04-01T01:38:14.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.023+0000 7f2a226d4640 1 Processor -- start 2024-04-01T01:38:14.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.023+0000 7f2a226d4640 1 -- start start 2024-04-01T01:38:14.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.023+0000 7f2a226d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a1c100650 0x7f2a1c195770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.023+0000 7f2a226d4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c101840 0x7f2a1c195cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.024+0000 7f2a226d4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1c102200 0x7f2a1c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.024+0000 7f2a226d4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c10b030 con 0x7f2a1c101840 2024-04-01T01:38:14.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.024+0000 7f2a226d4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c10aeb0 con 0x7f2a1c100650 2024-04-01T01:38:14.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.024+0000 7f2a226d4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c10b1b0 con 0x7f2a1c102200 2024-04-01T01:38:14.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.024+0000 7f2a1b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c101840 0x7f2a1c195cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.024+0000 7f2a1b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c101840 0x7f2a1c195cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52892/0 (socket says 172.21.15.70:52892) 2024-04-01T01:38:14.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.024+0000 7f2a1b7fe640 1 -- 172.21.15.70:0/1049698382 learned_addr learned my addr 172.21.15.70:0/1049698382 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:14.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.024+0000 7f2a20c71640 1 --2- 172.21.15.70:0/1049698382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1c102200 0x7f2a1c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.024+0000 7f2a1bfff640 1 --2- 172.21.15.70:0/1049698382 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a1c100650 0x7f2a1c195770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.024+0000 7f2a1b7fe640 1 -- 172.21.15.70:0/1049698382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1c102200 msgr2=0x7f2a1c19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.024+0000 7f2a1b7fe640 1 --2- 172.21.15.70:0/1049698382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1c102200 0x7f2a1c19cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.024+0000 7f2a1b7fe640 1 -- 172.21.15.70:0/1049698382 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a1c100650 msgr2=0x7f2a1c195770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.024+0000 7f2a1b7fe640 1 --2- 172.21.15.70:0/1049698382 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a1c100650 0x7f2a1c195770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.024+0000 7f2a1b7fe640 1 -- 172.21.15.70:0/1049698382 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a1c19d390 con 0x7f2a1c101840 2024-04-01T01:38:14.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.025+0000 7f2a1b7fe640 1 --2- 172.21.15.70:0/1049698382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c101840 0x7f2a1c195cb0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009980 tx=0x7f2a100045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:14.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.025+0000 7f2a197fa640 1 -- 172.21.15.70:0/1049698382 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a10039040 con 0x7f2a1c101840 2024-04-01T01:38:14.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.025+0000 7f2a197fa640 1 -- 172.21.15.70:0/1049698382 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2a10043070 con 0x7f2a1c101840 2024-04-01T01:38:14.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.025+0000 7f2a197fa640 1 -- 172.21.15.70:0/1049698382 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a10004920 con 0x7f2a1c101840 2024-04-01T01:38:14.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.025+0000 7f2a226d4640 1 -- 172.21.15.70:0/1049698382 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a1c19d620 con 0x7f2a1c101840 2024-04-01T01:38:14.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.026+0000 7f2a226d4640 1 -- 172.21.15.70:0/1049698382 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a1c19da60 con 0x7f2a1c101840 2024-04-01T01:38:14.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.029+0000 7f2a25399740 1 Processor -- start 2024-04-01T01:38:14.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.029+0000 7f2a25399740 1 -- start start 2024-04-01T01:38:14.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.030+0000 7f2a25399740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e1a10 0x561b851e1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.030+0000 7f2a25399740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e2400 0x561b851e6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.030+0000 7f2a25399740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e7570 0x561b851e9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.030+0000 7f2a25399740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561b850e30b0 con 0x561b851e1a10 2024-04-01T01:38:14.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.030+0000 7f2a25399740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561b85145ea0 con 0x561b851e7570 2024-04-01T01:38:14.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.030+0000 7f2a25399740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561b850eec00 con 0x561b851e2400 2024-04-01T01:38:14.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.030+0000 7f2a1bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e2400 0x561b851e6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.030+0000 7f2a1bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e2400 0x561b851e6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36424/0 (socket says 172.21.15.70:36424) 2024-04-01T01:38:14.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.030+0000 7f2a1bfff640 1 -- 172.21.15.70:0/833465268 learned_addr learned my addr 172.21.15.70:0/833465268 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:14.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.030+0000 7f2a1b7fe640 1 --2- 172.21.15.70:0/833465268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e7570 0x561b851e9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.030+0000 7f2a197fa640 1 -- 172.21.15.70:0/1049698382 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f2a100051e0 con 0x7f2a1c101840 2024-04-01T01:38:14.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.031+0000 7f2a1bfff640 1 -- 172.21.15.70:0/833465268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e7570 msgr2=0x561b851e9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:14.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.031+0000 7f2a1bfff640 1 --2- 172.21.15.70:0/833465268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e7570 0x561b851e9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.031+0000 7f2a1bfff640 1 -- 172.21.15.70:0/833465268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e1a10 msgr2=0x561b851e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:14.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.031+0000 7f2a1bfff640 1 --2- 172.21.15.70:0/833465268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e1a10 0x561b851e1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.031+0000 7f2a1bfff640 1 -- 172.21.15.70:0/833465268 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b84f227e0 con 0x561b851e2400 2024-04-01T01:38:14.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.031+0000 7f2a1bfff640 1 --2- 172.21.15.70:0/833465268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e2400 0x561b851e6f00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00bcb0 tx=0x7f2a0c0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8450c60894b1eb39 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:14.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.031+0000 7f2a197fa640 1 --2- 172.21.15.70:0/1049698382 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2a040a2670 0x7f2a040a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.031+0000 7f2a197fa640 1 -- 172.21.15.70:0/1049698382 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f2a100f0500 con 0x7f2a1c101840 2024-04-01T01:38:14.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.031+0000 7f2a027fc640 1 -- 172.21.15.70:0/833465268 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c00be60 con 0x561b851e2400 2024-04-01T01:38:14.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.032+0000 7f2a027fc640 1 -- 172.21.15.70:0/833465268 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2a0c0047a0 con 0x561b851e2400 2024-04-01T01:38:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.032+0000 7f2a20c71640 1 --2- 172.21.15.70:0/1049698382 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2a040a2670 0x7f2a040a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.033+0000 7f2a25399740 1 -- 172.21.15.70:0/833465268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e2400 msgr2=0x561b851e6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.033+0000 7f2a25399740 1 --2- 172.21.15.70:0/833465268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e2400 0x561b851e6f00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00bcb0 tx=0x7f2a0c0107a0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.033+0000 7f2a20c71640 1 --2- 172.21.15.70:0/1049698382 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2a040a2670 0x7f2a040a4b30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x561b851e2c60 tx=0x7f2a14009ae0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.033+0000 7f2a25399740 1 -- 172.21.15.70:0/833465268 shutdown_connections 2024-04-01T01:38:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.033+0000 7f2a25399740 1 --2- 172.21.15.70:0/833465268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e7570 0x561b851e9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.033+0000 7f2a25399740 1 --2- 172.21.15.70:0/833465268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e2400 0x561b851e6f00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.033+0000 7f2a25399740 1 --2- 172.21.15.70:0/833465268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e1a10 0x561b851e1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.033+0000 7f2a25399740 1 -- 172.21.15.70:0/833465268 >> 172.21.15.70:0/833465268 conn(0x561b851dcf80 msgr2=0x561b851dd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:14.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.036+0000 7f2a25399740 1 -- 172.21.15.70:0/833465268 shutdown_connections 2024-04-01T01:38:14.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.036+0000 7f2a25399740 1 -- 172.21.15.70:0/833465268 wait complete. 2024-04-01T01:38:14.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.036+0000 7f2a25399740 1 -- start start 2024-04-01T01:38:14.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.036+0000 7f2a25399740 1 Processor -- start 2024-04-01T01:38:14.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a25399740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e1a10 0x561b851f4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a25399740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e2400 0x561b851f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a25399740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e7570 0x561b851fbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a25399740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561b850e43d0 con 0x561b851e2400 2024-04-01T01:38:14.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a25399740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561b85141f40 con 0x561b851e7570 2024-04-01T01:38:14.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a25399740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561b851f2d80 con 0x561b851e1a10 2024-04-01T01:38:14.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a25399740 10 client.0 _ll_register_callbacks cb 0x561b85101b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:14.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a1bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e1a10 0x561b851f4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a1bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e1a10 0x561b851f4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36438/0 (socket says 172.21.15.70:36438) 2024-04-01T01:38:14.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a1bfff640 1 -- 172.21.15.70:0/3466294618 learned_addr learned my addr 172.21.15.70:0/3466294618 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:14.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a1b7fe640 1 --2- 172.21.15.70:0/3466294618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e2400 0x561b851f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a20c71640 1 --2- 172.21.15.70:0/3466294618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e7570 0x561b851fbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a1bfff640 1 -- 172.21.15.70:0/3466294618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e7570 msgr2=0x561b851fbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:14.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a1bfff640 1 --2- 172.21.15.70:0/3466294618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e7570 0x561b851fbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a1bfff640 1 -- 172.21.15.70:0/3466294618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e2400 msgr2=0x561b851f4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:14.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a1bfff640 1 --2- 172.21.15.70:0/3466294618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e2400 0x561b851f4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.037+0000 7f2a1bfff640 1 -- 172.21.15.70:0/3466294618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b84f82f70 con 0x561b851e1a10 2024-04-01T01:38:14.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.038+0000 7f2a1bfff640 1 --2- 172.21.15.70:0/3466294618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e1a10 0x561b851f4650 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00bc80 tx=0x7f2a0c00ba00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27c6a4b725b5edff server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:14.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.038+0000 7f29ffff7640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:38:14.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.038+0000 7f29ffff7640 1 -- 172.21.15.70:0/3466294618 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c00be60 con 0x561b851e1a10 2024-04-01T01:38:14.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.038+0000 7f29ffff7640 1 -- 172.21.15.70:0/3466294618 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f2a0c011b30 con 0x561b851e1a10 2024-04-01T01:38:14.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.038+0000 7f2a25399740 1 -- 172.21.15.70:0/3466294618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a0c0047a0 con 0x561b851e1a10 2024-04-01T01:38:14.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.039+0000 7f29ffff7640 1 -- 172.21.15.70:0/3466294618 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2a0c004c00 con 0x561b851e1a10 2024-04-01T01:38:14.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.040+0000 7f2a25399740 10 client.26794 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:14.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.040+0000 7f2a25399740 1 -- 172.21.15.70:0/3466294618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561b84e1a170 con 0x561b851e1a10 2024-04-01T01:38:14.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.040+0000 7f29ffff7640 1 -- 172.21.15.70:0/3466294618 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2a0c0047a0 con 0x561b851e1a10 2024-04-01T01:38:14.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.040+0000 7f29ffff7640 1 -- 172.21.15.70:0/3466294618 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2a0c0307a0 con 0x561b851e1a10 2024-04-01T01:38:14.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.040+0000 7f2a25399740 10 client.26794 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:14.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.040+0000 7f2a25399740 10 client.26794 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:14.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.040+0000 7f2a25399740 20 client.26794 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:14.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.040+0000 7f2a25399740 1 --2- 172.21.15.70:0/3466294618 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561b851ffe80 0x561b852022b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.040+0000 7f2a25399740 4 client.26794 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:14.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.040+0000 7f2a25399740 1 -- 172.21.15.70:0/3466294618 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b851533e0 con 0x561b851ffe80 2024-04-01T01:38:14.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.042+0000 7f2a1b7fe640 1 --2- 172.21.15.70:0/3466294618 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561b851ffe80 0x561b852022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.042+0000 7f2a1b7fe640 1 --2- 172.21.15.70:0/3466294618 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561b851ffe80 0x561b852022b0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3835af95b85f4051 server_cookie=7cf15b929fa279e6 in_seq=0 out_seq=0 2024-04-01T01:38:14.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.042+0000 7f29ffff7640 10 client.26794 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:14.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.043+0000 7f29ffff7640 1 -- 172.21.15.70:0/3466294618 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561b851533e0 con 0x561b851ffe80 2024-04-01T01:38:14.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.043+0000 7f29ffff7640 10 client.26794 handle_command_reply: tid=0 2024-04-01T01:38:14.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.043+0000 7f2a25399740 1 client.26794 shutdown 2024-04-01T01:38:14.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.043+0000 7f2a25399740 1 -- 172.21.15.70:0/3466294618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e1a10 msgr2=0x561b851f4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.043+0000 7f2a25399740 1 --2- 172.21.15.70:0/3466294618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e1a10 0x561b851f4650 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00bc80 tx=0x7f2a0c00ba00 comp rx=0 tx=0).stop 2024-04-01T01:38:14.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.043+0000 7f2a25399740 1 -- 172.21.15.70:0/3466294618 shutdown_connections 2024-04-01T01:38:14.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.043+0000 7f2a25399740 1 --2- 172.21.15.70:0/3466294618 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561b851ffe80 0x561b852022b0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.044+0000 7f2a25399740 1 --2- 172.21.15.70:0/3466294618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e7570 0x561b851fbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.044+0000 7f2a25399740 1 --2- 172.21.15.70:0/3466294618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e2400 0x561b851f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.044+0000 7f2a25399740 1 --2- 172.21.15.70:0/3466294618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e1a10 0x561b851f4650 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.044+0000 7f2a25399740 1 -- 172.21.15.70:0/3466294618 >> 172.21.15.70:0/3466294618 conn(0x561b851dcf80 msgr2=0x561b851dc180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:14.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.044+0000 7f29ffff7640 0 client.26794 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:14.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.044+0000 7f2a25399740 1 -- 172.21.15.70:0/3466294618 shutdown_connections 2024-04-01T01:38:14.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.044+0000 7f2a25399740 1 -- 172.21.15.70:0/3466294618 wait complete. 2024-04-01T01:38:14.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.044+0000 7f2a25399740 20 client.26794 trim_cache size 0 max 16384 2024-04-01T01:38:14.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.075+0000 7f2a25399740 1 Processor -- start 2024-04-01T01:38:14.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.075+0000 7f2a25399740 1 -- start start 2024-04-01T01:38:14.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.075+0000 7f2a25399740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e1a10 0x561b851e08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.075+0000 7f2a25399740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e2400 0x561b851fbb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.075+0000 7f2a25399740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e7570 0x561b851f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.075+0000 7f2a25399740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c1a8230 con 0x561b851e1a10 2024-04-01T01:38:14.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.075+0000 7f2a25399740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c1a9d70 con 0x561b851e7570 2024-04-01T01:38:14.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.075+0000 7f2a25399740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c1a8900 con 0x561b851e2400 2024-04-01T01:38:14.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.076+0000 7f2a1b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e2400 0x561b851fbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.076+0000 7f2a1b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e2400 0x561b851fbb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36440/0 (socket says 172.21.15.70:36440) 2024-04-01T01:38:14.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.076+0000 7f2a1b7fe640 1 -- 172.21.15.70:0/2315685545 learned_addr learned my addr 172.21.15.70:0/2315685545 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:14.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.076+0000 7f2a20c71640 1 --2- 172.21.15.70:0/2315685545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e7570 0x561b851f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.076+0000 7f2a20c71640 1 -- 172.21.15.70:0/2315685545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e2400 msgr2=0x561b851fbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:14.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.076+0000 7f2a20c71640 1 --2- 172.21.15.70:0/2315685545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e2400 0x561b851fbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.076+0000 7f2a20c71640 1 -- 172.21.15.70:0/2315685545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e1a10 msgr2=0x561b851e08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:14.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.076+0000 7f2a20c71640 1 --2- 172.21.15.70:0/2315685545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e1a10 0x561b851e08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.076+0000 7f2a20c71640 1 -- 172.21.15.70:0/2315685545 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b851e7270 con 0x561b851e7570 2024-04-01T01:38:14.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.077+0000 7f2a20c71640 1 --2- 172.21.15.70:0/2315685545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e7570 0x561b851f3c70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2a14004a80 tx=0x7f2a14013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6970ea8942a99b00 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:14.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.077+0000 7f2a01ffb640 1 -- 172.21.15.70:0/2315685545 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a140206f0 con 0x561b851e7570 2024-04-01T01:38:14.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.077+0000 7f2a01ffb640 1 -- 172.21.15.70:0/2315685545 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2a14020d40 con 0x561b851e7570 2024-04-01T01:38:14.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.077+0000 7f2a01ffb640 1 -- 172.21.15.70:0/2315685545 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a14011dd0 con 0x561b851e7570 2024-04-01T01:38:14.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:13 smithi081 ceph-mon[35142]: pgmap v500: 129 pgs: 129 active+clean; 1.0 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.8 MiB/s wr, 1.03k op/s 2024-04-01T01:38:14.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.078+0000 7f2a25399740 1 -- 172.21.15.70:0/2315685545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e7570 msgr2=0x561b851f3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.078+0000 7f2a25399740 1 --2- 172.21.15.70:0/2315685545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e7570 0x561b851f3c70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2a14004a80 tx=0x7f2a14013000 comp rx=0 tx=0).stop 2024-04-01T01:38:14.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.079+0000 7f2a25399740 1 -- 172.21.15.70:0/2315685545 shutdown_connections 2024-04-01T01:38:14.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.079+0000 7f2a25399740 1 --2- 172.21.15.70:0/2315685545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e7570 0x561b851f3c70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.079+0000 7f2a25399740 1 --2- 172.21.15.70:0/2315685545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e2400 0x561b851fbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.079+0000 7f2a25399740 1 --2- 172.21.15.70:0/2315685545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e1a10 0x561b851e08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.079+0000 7f2a25399740 1 -- 172.21.15.70:0/2315685545 >> 172.21.15.70:0/2315685545 conn(0x561b851dcf80 msgr2=0x561b851e01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:14.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.079+0000 7f2a25399740 1 -- 172.21.15.70:0/2315685545 shutdown_connections 2024-04-01T01:38:14.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.079+0000 7f2a25399740 1 -- 172.21.15.70:0/2315685545 wait complete. 2024-04-01T01:38:14.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.079+0000 7f2a25399740 1 -- start start 2024-04-01T01:38:14.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.080+0000 7f2a25399740 1 Processor -- start 2024-04-01T01:38:14.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.080+0000 7f2a25399740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e1a10 0x561b851f9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.080+0000 7f2a25399740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e2400 0x561b851f9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.080+0000 7f2a1bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e1a10 0x561b851f9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.080+0000 7f2a1bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e1a10 0x561b851f9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52922/0 (socket says 172.21.15.70:52922) 2024-04-01T01:38:14.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.080+0000 7f2a1bfff640 1 -- 172.21.15.70:0/2287693175 learned_addr learned my addr 172.21.15.70:0/2287693175 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:14.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.080+0000 7f2a25399740 1 --2- 172.21.15.70:0/2287693175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e7570 0x561b851fa610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.080+0000 7f2a25399740 1 -- 172.21.15.70:0/2287693175 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c1a96a0 con 0x561b851e1a10 2024-04-01T01:38:14.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.080+0000 7f2a25399740 1 -- 172.21.15.70:0/2287693175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c1a8fd0 con 0x561b851e2400 2024-04-01T01:38:14.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.080+0000 7f2a25399740 1 -- 172.21.15.70:0/2287693175 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2a1c1a6210 con 0x561b851e7570 2024-04-01T01:38:14.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.080+0000 7f2a25399740 10 client.0 _ll_register_callbacks cb 0x7f2a0c00b8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:14.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.081+0000 7f2a1b7fe640 1 --2- 172.21.15.70:0/2287693175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e2400 0x561b851f9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.081+0000 7f2a20c71640 1 --2- 172.21.15.70:0/2287693175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e7570 0x561b851fa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.081+0000 7f2a1bfff640 1 -- 172.21.15.70:0/2287693175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e7570 msgr2=0x561b851fa610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:14.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.081+0000 7f2a1bfff640 1 --2- 172.21.15.70:0/2287693175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e7570 0x561b851fa610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.081+0000 7f2a1bfff640 1 -- 172.21.15.70:0/2287693175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e2400 msgr2=0x561b851f9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:14.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.081+0000 7f2a1bfff640 1 --2- 172.21.15.70:0/2287693175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e2400 0x561b851f9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.081+0000 7f2a1bfff640 1 -- 172.21.15.70:0/2287693175 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b851fac00 con 0x561b851e1a10 2024-04-01T01:38:14.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.081+0000 7f2a1bfff640 1 --2- 172.21.15.70:0/2287693175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e1a10 0x561b851f9130 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c01b450 tx=0x7f2a0c004660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07af50e0ef455f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:14.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.081+0000 7f29ffff7640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:38:14.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.082+0000 7f29ffff7640 1 -- 172.21.15.70:0/2287693175 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c01f070 con 0x561b851e1a10 2024-04-01T01:38:14.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.082+0000 7f2a25399740 1 -- 172.21.15.70:0/2287693175 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a14020d40 con 0x561b851e1a10 2024-04-01T01:38:14.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.082+0000 7f29ffff7640 1 -- 172.21.15.70:0/2287693175 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f2a0c030cf0 con 0x561b851e1a10 2024-04-01T01:38:14.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.082+0000 7f29ffff7640 1 -- 172.21.15.70:0/2287693175 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2a0c03f050 con 0x561b851e1a10 2024-04-01T01:38:14.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.082+0000 7f2a25399740 10 client.17016 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:14.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.082+0000 7f2a25399740 1 -- 172.21.15.70:0/2287693175 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561b851fbf20 con 0x561b851e1a10 2024-04-01T01:38:14.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.082+0000 7f29ffff7640 1 -- 172.21.15.70:0/2287693175 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2a14020d40 con 0x561b851e1a10 2024-04-01T01:38:14.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.083+0000 7f29ffff7640 1 -- 172.21.15.70:0/2287693175 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2a0c004ff0 con 0x561b851e1a10 2024-04-01T01:38:14.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.083+0000 7f2a25399740 10 client.17016 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:14.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.083+0000 7f2a25399740 10 client.17016 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:14.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.083+0000 7f2a25399740 20 client.17016 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:14.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.083+0000 7f2a25399740 1 --2- 172.21.15.70:0/2287693175 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561b85266900 0x561b85268d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.083+0000 7f2a25399740 4 client.17016 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:38:14.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.083+0000 7f2a25399740 1 -- 172.21.15.70:0/2287693175 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561b8514d730 con 0x561b85266900 2024-04-01T01:38:14.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.083+0000 7f2a1b7fe640 1 --2- 172.21.15.70:0/2287693175 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561b85266900 0x561b85268d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.085+0000 7f2a1b7fe640 1 --2- 172.21.15.70:0/2287693175 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561b85266900 0x561b85268d20 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5084f82151a90c server_cookie=3865e231f033b59a in_seq=0 out_seq=0 2024-04-01T01:38:14.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.085+0000 7f29ffff7640 10 client.17016 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:14.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.088+0000 7f29ffff7640 1 -- 172.21.15.70:0/2287693175 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561b8514d730 con 0x561b85266900 2024-04-01T01:38:14.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.088+0000 7f29ffff7640 10 client.17016 handle_command_reply: tid=0 2024-04-01T01:38:14.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.088+0000 7f2a25399740 1 client.17016 shutdown 2024-04-01T01:38:14.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.089+0000 7f2a25399740 1 -- 172.21.15.70:0/2287693175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e1a10 msgr2=0x561b851f9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.089+0000 7f2a25399740 1 --2- 172.21.15.70:0/2287693175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e1a10 0x561b851f9130 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c01b450 tx=0x7f2a0c004660 comp rx=0 tx=0).stop 2024-04-01T01:38:14.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.089+0000 7f2a25399740 1 -- 172.21.15.70:0/2287693175 shutdown_connections 2024-04-01T01:38:14.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.089+0000 7f2a25399740 1 --2- 172.21.15.70:0/2287693175 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561b85266900 0x561b85268d20 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.089+0000 7f2a25399740 1 --2- 172.21.15.70:0/2287693175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b851e7570 0x561b851fa610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.090+0000 7f29ffff7640 0 client.17016 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:14.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.090+0000 7f2a25399740 1 --2- 172.21.15.70:0/2287693175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b851e2400 0x561b851f9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.090+0000 7f2a25399740 1 --2- 172.21.15.70:0/2287693175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b851e1a10 0x561b851f9130 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.090+0000 7f2a25399740 1 -- 172.21.15.70:0/2287693175 >> 172.21.15.70:0/2287693175 conn(0x561b851dcf80 msgr2=0x561b851db8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:14.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.090+0000 7f2a25399740 1 -- 172.21.15.70:0/2287693175 shutdown_connections 2024-04-01T01:38:14.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.090+0000 7f2a25399740 1 -- 172.21.15.70:0/2287693175 wait complete. 2024-04-01T01:38:14.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.090+0000 7f2a25399740 20 client.17016 trim_cache size 0 max 16384 2024-04-01T01:38:14.091 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:38:14.091 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:38:14.091 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "f0f48482-ddda-43c9-a8b7-fae1945d7053", 2024-04-01T01:38:14.092 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:38:14.092 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:38:14.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.091+0000 7f2a226d4640 1 -- 172.21.15.70:0/1049698382 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2a040a2670 msgr2=0x7f2a040a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.091+0000 7f2a226d4640 1 --2- 172.21.15.70:0/1049698382 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2a040a2670 0x7f2a040a4b30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x561b851e2c60 tx=0x7f2a14009ae0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.091+0000 7f2a226d4640 1 -- 172.21.15.70:0/1049698382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c101840 msgr2=0x7f2a1c195cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.091+0000 7f2a226d4640 1 --2- 172.21.15.70:0/1049698382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c101840 0x7f2a1c195cb0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009980 tx=0x7f2a100045e0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.091+0000 7f2a226d4640 1 -- 172.21.15.70:0/1049698382 shutdown_connections 2024-04-01T01:38:14.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.091+0000 7f2a226d4640 1 --2- 172.21.15.70:0/1049698382 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2a040a2670 0x7f2a040a4b30 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.091+0000 7f2a226d4640 1 --2- 172.21.15.70:0/1049698382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a1c102200 0x7f2a1c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.091+0000 7f2a226d4640 1 --2- 172.21.15.70:0/1049698382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a1c101840 0x7f2a1c195cb0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.091+0000 7f2a226d4640 1 --2- 172.21.15.70:0/1049698382 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a1c100650 0x7f2a1c195770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.091+0000 7f2a226d4640 1 -- 172.21.15.70:0/1049698382 >> 172.21.15.70:0/1049698382 conn(0x7f2a1c0fbbe0 msgr2=0x7f2a1c104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:14.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.092+0000 7f2a226d4640 1 -- 172.21.15.70:0/1049698382 shutdown_connections 2024-04-01T01:38:14.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.092+0000 7f2a226d4640 1 -- 172.21.15.70:0/1049698382 wait complete. 2024-04-01T01:38:14.114 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:13 smithi070 ceph-mon[32179]: pgmap v500: 129 pgs: 129 active+clean; 1.0 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.8 MiB/s wr, 1.03k op/s 2024-04-01T01:38:14.116 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:38:14.116 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:38:14.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:13 smithi189 ceph-mon[35753]: pgmap v500: 129 pgs: 129 active+clean; 1.0 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.8 MiB/s wr, 1.03k op/s 2024-04-01T01:38:14.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.304+0000 7f2cc305c640 1 -- 172.21.15.70:0/694040899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb409df20 msgr2=0x7f2cb40a47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.304+0000 7f2cc305c640 1 --2- 172.21.15.70:0/694040899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb409df20 0x7f2cb40a47b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2cac009860 tx=0x7f2cac02bb90 comp rx=0 tx=0).stop 2024-04-01T01:38:14.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.304+0000 7f2cc305c640 1 -- 172.21.15.70:0/694040899 shutdown_connections 2024-04-01T01:38:14.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.304+0000 7f2cc305c640 1 --2- 172.21.15.70:0/694040899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb409df20 0x7f2cb40a47b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.304+0000 7f2cc305c640 1 --2- 172.21.15.70:0/694040899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cb409d560 0x7f2cb409d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.304+0000 7f2cc305c640 1 --2- 172.21.15.70:0/694040899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cb409c370 0x7f2cb409c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.304+0000 7f2cc305c640 1 -- 172.21.15.70:0/694040899 >> 172.21.15.70:0/694040899 conn(0x7f2cb400b310 msgr2=0x7f2cb400b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:14.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.305+0000 7f2cc305c640 1 -- 172.21.15.70:0/694040899 shutdown_connections 2024-04-01T01:38:14.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.305+0000 7f2cc305c640 1 -- 172.21.15.70:0/694040899 wait complete. 2024-04-01T01:38:14.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.306+0000 7f2cc305c640 1 Processor -- start 2024-04-01T01:38:14.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.306+0000 7f2cc305c640 1 -- start start 2024-04-01T01:38:14.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.306+0000 7f2cc305c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cb409c370 0x7f2cb4132570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.306+0000 7f2cc305c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cb409d560 0x7f2cb4132ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.306+0000 7f2cc305c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb409df20 0x7f2cb4139b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.306+0000 7f2cc305c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb40a7f10 con 0x7f2cb409d560 2024-04-01T01:38:14.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.306+0000 7f2cc305c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb40a7d90 con 0x7f2cb409c370 2024-04-01T01:38:14.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.306+0000 7f2cc305c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb40a8090 con 0x7f2cb409df20 2024-04-01T01:38:14.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.306+0000 7f2cc0df8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cb409c370 0x7f2cb4132570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.306+0000 7f2cc15f9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb409df20 0x7f2cb4139b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.307+0000 7f2cc15f9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb409df20 0x7f2cb4139b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36460/0 (socket says 172.21.15.70:36460) 2024-04-01T01:38:14.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.307+0000 7f2cc15f9640 1 -- 172.21.15.70:0/3375401015 learned_addr learned my addr 172.21.15.70:0/3375401015 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:14.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.307+0000 7f2cbbfff640 1 --2- 172.21.15.70:0/3375401015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cb409d560 0x7f2cb4132ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.307+0000 7f2cc15f9640 1 -- 172.21.15.70:0/3375401015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cb409c370 msgr2=0x7f2cb4132570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.307+0000 7f2cc15f9640 1 --2- 172.21.15.70:0/3375401015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cb409c370 0x7f2cb4132570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.307+0000 7f2cc15f9640 1 -- 172.21.15.70:0/3375401015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cb409d560 msgr2=0x7f2cb4132ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.307+0000 7f2cc15f9640 1 --2- 172.21.15.70:0/3375401015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cb409d560 0x7f2cb4132ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.307+0000 7f2cc15f9640 1 -- 172.21.15.70:0/3375401015 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb413a190 con 0x7f2cb409df20 2024-04-01T01:38:14.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.307+0000 7f2cc0df8640 1 --2- 172.21.15.70:0/3375401015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cb409c370 0x7f2cb4132570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:38:14.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.308+0000 7f2cc15f9640 1 --2- 172.21.15.70:0/3375401015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb409df20 0x7f2cb4139b20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2cac034a10 tx=0x7f2cac004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:14.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.308+0000 7f2cb9ffb640 1 -- 172.21.15.70:0/3375401015 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac041020 con 0x7f2cb409df20 2024-04-01T01:38:14.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.308+0000 7f2cb9ffb640 1 -- 172.21.15.70:0/3375401015 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2cac0035f0 con 0x7f2cb409df20 2024-04-01T01:38:14.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.308+0000 7f2cc305c640 1 -- 172.21.15.70:0/3375401015 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cb413a420 con 0x7f2cb409df20 2024-04-01T01:38:14.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.308+0000 7f2cb9ffb640 1 -- 172.21.15.70:0/3375401015 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cac039040 con 0x7f2cb409df20 2024-04-01T01:38:14.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.308+0000 7f2cc305c640 1 -- 172.21.15.70:0/3375401015 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cb413a9e0 con 0x7f2cb409df20 2024-04-01T01:38:14.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.312+0000 7f2cc5c3f740 1 Processor -- start 2024-04-01T01:38:14.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.312+0000 7f2cb9ffb640 1 -- 172.21.15.70:0/3375401015 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f2cac044020 con 0x7f2cb409df20 2024-04-01T01:38:14.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.312+0000 7f2cc5c3f740 1 -- start start 2024-04-01T01:38:14.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.312+0000 7f2cc5c3f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba21490 0x55c34ba218b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.312+0000 7f2cc5c3f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21e80 0x55c34ba26980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.312+0000 7f2cc5c3f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba26ff0 0x55c34ba29480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.312+0000 7f2cc5c3f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c34b986180 con 0x55c34ba26ff0 2024-04-01T01:38:14.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.312+0000 7f2cc5c3f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c34b92eb70 con 0x55c34ba21e80 2024-04-01T01:38:14.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.312+0000 7f2cc5c3f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c34b982220 con 0x55c34ba21490 2024-04-01T01:38:14.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.313+0000 7f2cbbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba26ff0 0x55c34ba29480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.313+0000 7f2cbbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba26ff0 0x55c34ba29480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52950/0 (socket says 172.21.15.70:52950) 2024-04-01T01:38:14.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.313+0000 7f2cbbfff640 1 -- 172.21.15.70:0/429167235 learned_addr learned my addr 172.21.15.70:0/429167235 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:14.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.313+0000 7f2cbbfff640 1 --2- 172.21.15.70:0/429167235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba21490 0x55c34ba218b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.313+0000 7f2cc0df8640 1 --2- 172.21.15.70:0/429167235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21e80 0x55c34ba26980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.313+0000 7f2cb9ffb640 1 --2- 172.21.15.70:0/3375401015 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2ca40a2520 0x7f2ca40a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.314+0000 7f2cb9ffb640 1 -- 172.21.15.70:0/3375401015 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f2cac0f0640 con 0x7f2cb409df20 2024-04-01T01:38:14.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.314+0000 7f2cc0df8640 1 -- 172.21.15.70:0/429167235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba21490 msgr2=0x55c34ba218b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:14.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.314+0000 7f2cc0df8640 1 --2- 172.21.15.70:0/429167235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba21490 0x55c34ba218b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.314+0000 7f2cc0df8640 1 -- 172.21.15.70:0/429167235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba26ff0 msgr2=0x55c34ba29480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:14.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.314+0000 7f2cc0df8640 1 --2- 172.21.15.70:0/429167235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba26ff0 0x55c34ba29480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.314+0000 7f2cc0df8640 1 -- 172.21.15.70:0/429167235 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c34b9427c0 con 0x55c34ba21e80 2024-04-01T01:38:14.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.314+0000 7f2cc15f9640 1 --2- 172.21.15.70:0/3375401015 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2ca40a2520 0x7f2ca40a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.315+0000 7f2cc15f9640 1 --2- 172.21.15.70:0/3375401015 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2ca40a2520 0x7f2ca40a49e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2cac0b9130 tx=0x7f2cac049ff0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:14.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.316+0000 7f2cc0df8640 1 --2- 172.21.15.70:0/429167235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21e80 0x55c34ba26980 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc050db0 tx=0x7f2cbc0504d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14cdb9f1499eab4d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:14.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.316+0000 7f2ca2ffd640 1 -- 172.21.15.70:0/429167235 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc073e10 con 0x55c34ba21e80 2024-04-01T01:38:14.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.316+0000 7f2ca2ffd640 1 -- 172.21.15.70:0/429167235 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2cbc04fda0 con 0x55c34ba21e80 2024-04-01T01:38:14.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.316+0000 7f2ca2ffd640 1 -- 172.21.15.70:0/429167235 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc080b00 con 0x55c34ba21e80 2024-04-01T01:38:14.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.317+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/429167235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21e80 msgr2=0x55c34ba26980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.317+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/429167235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21e80 0x55c34ba26980 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc050db0 tx=0x7f2cbc0504d0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.318+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/429167235 shutdown_connections 2024-04-01T01:38:14.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.318+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/429167235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba26ff0 0x55c34ba29480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.318+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/429167235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21e80 0x55c34ba26980 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.318+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/429167235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba21490 0x55c34ba218b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.318+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/429167235 >> 172.21.15.70:0/429167235 conn(0x55c34ba1ca00 msgr2=0x55c34ba1ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:14.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.319+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/429167235 shutdown_connections 2024-04-01T01:38:14.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.320+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/429167235 wait complete. 2024-04-01T01:38:14.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.320+0000 7f2cc5c3f740 1 -- start start 2024-04-01T01:38:14.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.321+0000 7f2cc5c3f740 1 Processor -- start 2024-04-01T01:38:14.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.321+0000 7f2cc5c3f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba21490 0x55c34ba3c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.321+0000 7f2cc5c3f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21e80 0x55c34ba3cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.321+0000 7f2cc5c3f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba26ff0 0x55c34ba43f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.321+0000 7f2cc5c3f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c34ba3afb0 con 0x55c34ba26ff0 2024-04-01T01:38:14.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.321+0000 7f2cc5c3f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c34b9244a0 con 0x55c34ba21e80 2024-04-01T01:38:14.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.321+0000 7f2cc5c3f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c34ba3b130 con 0x55c34ba21490 2024-04-01T01:38:14.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.321+0000 7f2cc5c3f740 10 client.0 _ll_register_callbacks cb 0x55c34b927d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:14.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.321+0000 7f2cbbfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba21490 0x55c34ba3c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.321+0000 7f2cbbfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba21490 0x55c34ba3c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36478/0 (socket says 172.21.15.70:36478) 2024-04-01T01:38:14.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.321+0000 7f2cbbfff640 1 -- 172.21.15.70:0/2265288921 learned_addr learned my addr 172.21.15.70:0/2265288921 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:14.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.322+0000 7f2cc0df8640 1 --2- 172.21.15.70:0/2265288921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21e80 0x55c34ba3cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.322+0000 7f2cbbfff640 1 --2- 172.21.15.70:0/2265288921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba26ff0 0x55c34ba43f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.322+0000 7f2cbbfff640 1 -- 172.21.15.70:0/2265288921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21e80 msgr2=0x55c34ba3cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:14.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.322+0000 7f2cbbfff640 1 --2- 172.21.15.70:0/2265288921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21e80 0x55c34ba3cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.322+0000 7f2cbbfff640 1 -- 172.21.15.70:0/2265288921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba26ff0 msgr2=0x55c34ba43f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.322+0000 7f2cbbfff640 1 --2- 172.21.15.70:0/2265288921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba26ff0 0x55c34ba43f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.322+0000 7f2cbbfff640 1 -- 172.21.15.70:0/2265288921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c34b7647d0 con 0x55c34ba21490 2024-04-01T01:38:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.322+0000 7f2cbbfff640 1 --2- 172.21.15.70:0/2265288921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba21490 0x55c34ba3c9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0014440 tx=0x7f2cb000cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63285a2c4fd042ac server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.323+0000 7f2c9bfff640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:38:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.323+0000 7f2c9bfff640 1 -- 172.21.15.70:0/2265288921 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0011470 con 0x55c34ba21490 2024-04-01T01:38:14.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.323+0000 7f2c9bfff640 1 -- 172.21.15.70:0/2265288921 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f2cb0011610 con 0x55c34ba21490 2024-04-01T01:38:14.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.325+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/2265288921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c34b9936c0 con 0x55c34ba21490 2024-04-01T01:38:14.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.327+0000 7f2c9bfff640 1 -- 172.21.15.70:0/2265288921 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2cb0011810 con 0x55c34ba21490 2024-04-01T01:38:14.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.327+0000 7f2cc5c3f740 10 client.26812 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:14.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.327+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/2265288921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c34b69a3f0 con 0x55c34ba21490 2024-04-01T01:38:14.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.328+0000 7f2c9bfff640 1 -- 172.21.15.70:0/2265288921 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55c34b9936c0 con 0x55c34ba21490 2024-04-01T01:38:14.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.328+0000 7f2c9bfff640 1 -- 172.21.15.70:0/2265288921 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2cb000fa00 con 0x55c34ba21490 2024-04-01T01:38:14.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.328+0000 7f2cc5c3f740 10 client.26812 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:14.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.328+0000 7f2cc5c3f740 10 client.26812 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:14.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.328+0000 7f2cc5c3f740 20 client.26812 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:14.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.328+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/2265288921 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c34ba48700 0x55c34ba4aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.328+0000 7f2cc5c3f740 4 client.26812 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:14.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.328+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/2265288921 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c34b98da10 con 0x55c34ba48700 2024-04-01T01:38:14.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.330+0000 7f2cc0df8640 1 --2- 172.21.15.70:0/2265288921 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c34ba48700 0x55c34ba4aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.330+0000 7f2cc0df8640 1 --2- 172.21.15.70:0/2265288921 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c34ba48700 0x55c34ba4aba0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94b76f04dad7c8f server_cookie=ae3e5ae7c68d7d56 in_seq=0 out_seq=0 2024-04-01T01:38:14.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.330+0000 7f2c9bfff640 10 client.26812 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:14.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.332+0000 7f2c9bfff640 1 -- 172.21.15.70:0/2265288921 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c34b98da10 con 0x55c34ba48700 2024-04-01T01:38:14.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.332+0000 7f2c9bfff640 10 client.26812 handle_command_reply: tid=0 2024-04-01T01:38:14.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.332+0000 7f2cc5c3f740 1 client.26812 shutdown 2024-04-01T01:38:14.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.333+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/2265288921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba21490 msgr2=0x55c34ba3c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.333+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/2265288921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba21490 0x55c34ba3c9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0014440 tx=0x7f2cb000cc10 comp rx=0 tx=0).stop 2024-04-01T01:38:14.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.334+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/2265288921 shutdown_connections 2024-04-01T01:38:14.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.334+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/2265288921 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c34ba48700 0x55c34ba4aba0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.334+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/2265288921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba26ff0 0x55c34ba43f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.334+0000 7f2c9bfff640 0 client.26812 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:14.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.334+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/2265288921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21e80 0x55c34ba3cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.334+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/2265288921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba21490 0x55c34ba3c9c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.334+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/2265288921 >> 172.21.15.70:0/2265288921 conn(0x55c34ba1ca00 msgr2=0x55c34ba1bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:14.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.334+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/2265288921 shutdown_connections 2024-04-01T01:38:14.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.335+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/2265288921 wait complete. 2024-04-01T01:38:14.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.335+0000 7f2cc5c3f740 20 client.26812 trim_cache size 0 max 16384 2024-04-01T01:38:14.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.373+0000 7f2cc5c3f740 1 Processor -- start 2024-04-01T01:38:14.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.373+0000 7f2cc5c3f740 1 -- start start 2024-04-01T01:38:14.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.374+0000 7f2cc5c3f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21490 0x55c34ba3ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.374+0000 7f2cc5c3f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba21e80 0x55c34ba43f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.374+0000 7f2cc5c3f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba26ff0 0x55c34ba20370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.374+0000 7f2cbbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21490 0x55c34ba3ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.374+0000 7f2cbbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21490 0x55c34ba3ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48262/0 (socket says 172.21.15.70:48262) 2024-04-01T01:38:14.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.374+0000 7f2cc0df8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba21e80 0x55c34ba43f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.374+0000 7f2cc0df8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba21e80 0x55c34ba43f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52970/0 (socket says 172.21.15.70:52970) 2024-04-01T01:38:14.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.374+0000 7f2cbbfff640 1 -- 172.21.15.70:0/3710437112 learned_addr learned my addr 172.21.15.70:0/3710437112 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:14.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.374+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/3710437112 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb413c910 con 0x55c34ba21e80 2024-04-01T01:38:14.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.374+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/3710437112 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb413e450 con 0x55c34ba21490 2024-04-01T01:38:14.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.374+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/3710437112 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb413cfe0 con 0x55c34ba26ff0 2024-04-01T01:38:14.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.375+0000 7f2cc0df8640 1 -- 172.21.15.70:0/3710437112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba26ff0 msgr2=0x55c34ba20370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:14.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.375+0000 7f2cc0df8640 1 --2- 172.21.15.70:0/3710437112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba26ff0 0x55c34ba20370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.375+0000 7f2cc0df8640 1 -- 172.21.15.70:0/3710437112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21490 msgr2=0x55c34ba3ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:14.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.375+0000 7f2cc0df8640 1 --2- 172.21.15.70:0/3710437112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21490 0x55c34ba3ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.375+0000 7f2cc0df8640 1 -- 172.21.15.70:0/3710437112 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c34ba18280 con 0x55c34ba21e80 2024-04-01T01:38:14.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.375+0000 7f2cc0df8640 1 --2- 172.21.15.70:0/3710437112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba21e80 0x55c34ba43f00 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x55c34ba1d970 tx=0x7f2cbc0675f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe9ba5253e97c54 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:14.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.376+0000 7f2ca27fc640 1 -- 172.21.15.70:0/3710437112 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc094020 con 0x55c34ba21e80 2024-04-01T01:38:14.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.376+0000 7f2ca27fc640 1 -- 172.21.15.70:0/3710437112 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2cbc0807d0 con 0x55c34ba21e80 2024-04-01T01:38:14.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.376+0000 7f2ca27fc640 1 -- 172.21.15.70:0/3710437112 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc08f070 con 0x55c34ba21e80 2024-04-01T01:38:14.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.377+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/3710437112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba21e80 msgr2=0x55c34ba43f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.377+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/3710437112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba21e80 0x55c34ba43f00 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x55c34ba1d970 tx=0x7f2cbc0675f0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.377+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/3710437112 shutdown_connections 2024-04-01T01:38:14.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.377+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/3710437112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba26ff0 0x55c34ba20370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.377+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/3710437112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba21e80 0x55c34ba43f00 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.377+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/3710437112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21490 0x55c34ba3ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.377+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/3710437112 >> 172.21.15.70:0/3710437112 conn(0x55c34ba1ca00 msgr2=0x55c34ba1fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:14.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.377+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/3710437112 shutdown_connections 2024-04-01T01:38:14.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.377+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/3710437112 wait complete. 2024-04-01T01:38:14.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.378+0000 7f2cc5c3f740 1 -- start start 2024-04-01T01:38:14.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.378+0000 7f2cc5c3f740 1 Processor -- start 2024-04-01T01:38:14.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.379+0000 7f2cc5c3f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21490 0x55c34ba41690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.379+0000 7f2cc5c3f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba21e80 0x55c34ba41bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.379+0000 7f2cc5c3f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba26ff0 0x55c34ba42be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.379+0000 7f2cc5c3f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb413dd80 con 0x55c34ba21e80 2024-04-01T01:38:14.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.379+0000 7f2cc5c3f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb413d6b0 con 0x55c34ba21490 2024-04-01T01:38:14.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.379+0000 7f2cc5c3f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb40156f0 con 0x55c34ba26ff0 2024-04-01T01:38:14.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.379+0000 7f2cc5c3f740 10 client.0 _ll_register_callbacks cb 0x7f2cb0014810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:14.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.379+0000 7f2cbbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21490 0x55c34ba41690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.379+0000 7f2cc0df8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba21e80 0x55c34ba41bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.379+0000 7f2cc0df8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba21e80 0x55c34ba41bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52982/0 (socket says 172.21.15.70:52982) 2024-04-01T01:38:14.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.379+0000 7f2cc0df8640 1 -- 172.21.15.70:0/3022547667 learned_addr learned my addr 172.21.15.70:0/3022547667 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:14.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.379+0000 7f2cbbfff640 1 --2- 172.21.15.70:0/3022547667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba26ff0 0x55c34ba42be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.380+0000 7f2cc0df8640 1 -- 172.21.15.70:0/3022547667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba26ff0 msgr2=0x55c34ba42be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:14.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.380+0000 7f2cc0df8640 1 --2- 172.21.15.70:0/3022547667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba26ff0 0x55c34ba42be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.380+0000 7f2cc0df8640 1 -- 172.21.15.70:0/3022547667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21490 msgr2=0x55c34ba41690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:14.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.380+0000 7f2cc0df8640 1 --2- 172.21.15.70:0/3022547667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21490 0x55c34ba41690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.380+0000 7f2cc0df8640 1 -- 172.21.15.70:0/3022547667 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c34b65a0f0 con 0x55c34ba21e80 2024-04-01T01:38:14.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.380+0000 7f2cc0df8640 1 --2- 172.21.15.70:0/3022547667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba21e80 0x55c34ba41bd0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc050260 tx=0x7f2cbc080480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6840caa84a703fbe server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:14.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.380+0000 7f2c9bfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:38:14.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.380+0000 7f2c9bfff640 1 -- 172.21.15.70:0/3022547667 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc08f070 con 0x55c34ba21e80 2024-04-01T01:38:14.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.380+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/3022547667 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cbc0807d0 con 0x55c34ba21e80 2024-04-01T01:38:14.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.381+0000 7f2c9bfff640 1 -- 172.21.15.70:0/3022547667 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f2cbc080b00 con 0x55c34ba21e80 2024-04-01T01:38:14.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.381+0000 7f2c9bfff640 1 -- 172.21.15.70:0/3022547667 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2cbc094050 con 0x55c34ba21e80 2024-04-01T01:38:14.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.381+0000 7f2cc5c3f740 10 client.17031 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:14.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.381+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/3022547667 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c34baaeaa0 con 0x55c34ba21e80 2024-04-01T01:38:14.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.381+0000 7f2c9bfff640 1 -- 172.21.15.70:0/3022547667 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2cbc091070 con 0x55c34ba21e80 2024-04-01T01:38:14.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.381+0000 7f2c9bfff640 1 -- 172.21.15.70:0/3022547667 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2cbc089bb0 con 0x55c34ba21e80 2024-04-01T01:38:14.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.382+0000 7f2cc5c3f740 10 client.17031 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:14.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.382+0000 7f2cc5c3f740 10 client.17031 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:14.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.382+0000 7f2cc5c3f740 20 client.17031 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:14.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.382+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/3022547667 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c34baaf1c0 0x55c34bab15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:14.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.382+0000 7f2cc5c3f740 4 client.17031 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:38:14.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.382+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/3022547667 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f2cbc0807d0 con 0x55c34baaf1c0 2024-04-01T01:38:14.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.382+0000 7f2cbbfff640 1 --2- 172.21.15.70:0/3022547667 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c34baaf1c0 0x55c34bab15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:14.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.383+0000 7f2cbbfff640 1 --2- 172.21.15.70:0/3022547667 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c34baaf1c0 0x55c34bab15e0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=808fe3c0cb25a68f server_cookie=2ed754d8baaa036d in_seq=0 out_seq=0 2024-04-01T01:38:14.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.383+0000 7f2c9bfff640 10 client.17031 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:14.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.384+0000 7f2c9bfff640 1 -- 172.21.15.70:0/3022547667 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x7f2cbc0807d0 con 0x55c34baaf1c0 2024-04-01T01:38:14.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.384+0000 7f2c9bfff640 10 client.17031 handle_command_reply: tid=0 2024-04-01T01:38:14.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.384+0000 7f2cc5c3f740 1 client.17031 shutdown 2024-04-01T01:38:14.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.384+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/3022547667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba21e80 msgr2=0x55c34ba41bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.384+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/3022547667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba21e80 0x55c34ba41bd0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc050260 tx=0x7f2cbc080480 comp rx=0 tx=0).stop 2024-04-01T01:38:14.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.385+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/3022547667 shutdown_connections 2024-04-01T01:38:14.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.385+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/3022547667 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c34baaf1c0 0x55c34bab15e0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.385+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/3022547667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c34ba26ff0 0x55c34ba42be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.385+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/3022547667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c34ba21e80 0x55c34ba41bd0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.385+0000 7f2cc5c3f740 1 --2- 172.21.15.70:0/3022547667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c34ba21490 0x55c34ba41690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.385+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/3022547667 >> 172.21.15.70:0/3022547667 conn(0x55c34ba1ca00 msgr2=0x55c34ba204f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:14.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.385+0000 7f2c9bfff640 0 client.17031 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:14.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.386+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/3022547667 shutdown_connections 2024-04-01T01:38:14.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.386+0000 7f2cc5c3f740 1 -- 172.21.15.70:0/3022547667 wait complete. 2024-04-01T01:38:14.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.386+0000 7f2cc5c3f740 20 client.17031 trim_cache size 0 max 16384 2024-04-01T01:38:14.387 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:38:14.387 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "scrub active (14 inodes in the stack)", 2024-04-01T01:38:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:38:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: "f0f48482-ddda-43c9-a8b7-fae1945d7053": { 2024-04-01T01:38:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:38:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "f0f48482-ddda-43c9-a8b7-fae1945d7053", 2024-04-01T01:38:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:38:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:38:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:38:14.388 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:38:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.388+0000 7f2cc305c640 1 -- 172.21.15.70:0/3375401015 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2ca40a2520 msgr2=0x7f2ca40a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.389+0000 7f2cc305c640 1 --2- 172.21.15.70:0/3375401015 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2ca40a2520 0x7f2ca40a49e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2cac0b9130 tx=0x7f2cac049ff0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.389+0000 7f2cc305c640 1 -- 172.21.15.70:0/3375401015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb409df20 msgr2=0x7f2cb4139b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.389+0000 7f2cc305c640 1 --2- 172.21.15.70:0/3375401015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb409df20 0x7f2cb4139b20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2cac034a10 tx=0x7f2cac004270 comp rx=0 tx=0).stop 2024-04-01T01:38:14.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.390+0000 7f2cc305c640 1 -- 172.21.15.70:0/3375401015 shutdown_connections 2024-04-01T01:38:14.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.390+0000 7f2cc305c640 1 --2- 172.21.15.70:0/3375401015 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2ca40a2520 0x7f2ca40a49e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.390+0000 7f2cc305c640 1 --2- 172.21.15.70:0/3375401015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cb409df20 0x7f2cb4139b20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.390+0000 7f2cc305c640 1 --2- 172.21.15.70:0/3375401015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cb409d560 0x7f2cb4132ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.391+0000 7f2cc305c640 1 --2- 172.21.15.70:0/3375401015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cb409c370 0x7f2cb4132570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:14.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.391+0000 7f2cc305c640 1 -- 172.21.15.70:0/3375401015 >> 172.21.15.70:0/3375401015 conn(0x7f2cb400b310 msgr2=0x7f2cb40a0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:14.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.391+0000 7f2cc305c640 1 -- 172.21.15.70:0/3375401015 shutdown_connections 2024-04-01T01:38:14.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:14.391+0000 7f2cc305c640 1 -- 172.21.15.70:0/3375401015 wait complete. 2024-04-01T01:38:14.415 INFO:tasks.cephfs.filesystem:scrub status for tag:f0f48482-ddda-43c9-a8b7-fae1945d7053 - {'path': '/', 'tag': 'f0f48482-ddda-43c9-a8b7-fae1945d7053', 'options': 'recursive,force'} 2024-04-01T01:38:15.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:14 smithi081 ceph-mon[35142]: pgmap v501: 129 pgs: 129 active+clean; 1.0 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.1 MiB/s wr, 947 op/s 2024-04-01T01:38:15.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:14 smithi189 ceph-mon[35753]: pgmap v501: 129 pgs: 129 active+clean; 1.0 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.1 MiB/s wr, 947 op/s 2024-04-01T01:38:15.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:14 smithi070 ceph-mon[32179]: pgmap v501: 129 pgs: 129 active+clean; 1.0 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.1 MiB/s wr, 947 op/s 2024-04-01T01:38:16.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:15 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:38:16.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:15 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:38:16.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:15 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:38:16.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:15 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:38:16.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:15 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:38:16.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:15 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:38:16.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:15 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:38:16.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:15 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:38:16.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:15 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:38:16.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:15 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:38:16.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:15 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:38:16.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:15 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:38:16.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 210.0 s, 1054.2 tps, lat 93.296 ms stddev 38.181 2024-04-01T01:38:17.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:16 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:38:17.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:16 smithi189 ceph-mon[35753]: pgmap v502: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 9.8 MiB/s wr, 1.18k op/s 2024-04-01T01:38:17.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:16 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:38:17.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:16 smithi070 ceph-mon[32179]: pgmap v502: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 9.8 MiB/s wr, 1.18k op/s 2024-04-01T01:38:17.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:16 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:38:17.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:16 smithi081 ceph-mon[35142]: pgmap v502: 129 pgs: 129 active+clean; 1.0 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 9.8 MiB/s wr, 1.18k op/s 2024-04-01T01:38:20.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:19 smithi081 ceph-mon[35142]: pgmap v503: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 8.5 MiB/s wr, 1.10k op/s 2024-04-01T01:38:20.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:19 smithi189 ceph-mon[35753]: pgmap v503: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 8.5 MiB/s wr, 1.10k op/s 2024-04-01T01:38:20.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:19 smithi070 ceph-mon[32179]: pgmap v503: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 8.5 MiB/s wr, 1.10k op/s 2024-04-01T01:38:21.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 215.0 s, 1140.0 tps, lat 88.979 ms stddev 56.117 2024-04-01T01:38:22.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:21 smithi081 ceph-mon[35142]: pgmap v504: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 8.1 MiB/s wr, 1.05k op/s 2024-04-01T01:38:22.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:21 smithi189 ceph-mon[35753]: pgmap v504: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 8.1 MiB/s wr, 1.05k op/s 2024-04-01T01:38:22.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:21 smithi070 ceph-mon[32179]: pgmap v504: 129 pgs: 129 active+clean; 1.0 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 8.1 MiB/s wr, 1.05k op/s 2024-04-01T01:38:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.268+0000 7f9067128640 1 -- 172.21.15.70:0/4029455109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90601017d0 msgr2=0x7f9060108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.268+0000 7f9067128640 1 --2- 172.21.15.70:0/4029455109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90601017d0 0x7f9060108060 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f90580081d0 tx=0x7f9058034490 comp rx=0 tx=0).stop 2024-04-01T01:38:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.269+0000 7f9067128640 1 -- 172.21.15.70:0/4029455109 shutdown_connections 2024-04-01T01:38:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.269+0000 7f9067128640 1 --2- 172.21.15.70:0/4029455109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90601017d0 0x7f9060108060 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.269+0000 7f9067128640 1 --2- 172.21.15.70:0/4029455109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9060100e10 0x7f9060101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.269+0000 7f9067128640 1 --2- 172.21.15.70:0/4029455109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90600ffc20 0x7f9060100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.269+0000 7f9067128640 1 -- 172.21.15.70:0/4029455109 >> 172.21.15.70:0/4029455109 conn(0x7f90600fb190 msgr2=0x7f90600fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.269+0000 7f9067128640 1 -- 172.21.15.70:0/4029455109 shutdown_connections 2024-04-01T01:38:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.269+0000 7f9067128640 1 -- 172.21.15.70:0/4029455109 wait complete. 2024-04-01T01:38:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.270+0000 7f9067128640 1 Processor -- start 2024-04-01T01:38:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.270+0000 7f9067128640 1 -- start start 2024-04-01T01:38:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.270+0000 7f9067128640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90600ffc20 0x7f9060195570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.270+0000 7f9067128640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9060100e10 0x7f9060195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.270+0000 7f9067128640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90601017d0 0x7f906019cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.271+0000 7f9067128640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f906010aee0 con 0x7f9060100e10 2024-04-01T01:38:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.271+0000 7f9067128640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f906010ad60 con 0x7f90600ffc20 2024-04-01T01:38:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.271+0000 7f9067128640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f906010b060 con 0x7f90601017d0 2024-04-01T01:38:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.271+0000 7f9066927640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90601017d0 0x7f906019cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.271+0000 7f9065925640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9060100e10 0x7f9060195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.271+0000 7f9066927640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90601017d0 0x7f906019cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57570/0 (socket says 172.21.15.70:57570) 2024-04-01T01:38:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.271+0000 7f9066927640 1 -- 172.21.15.70:0/214704331 learned_addr learned my addr 172.21.15.70:0/214704331 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.271+0000 7f9066126640 1 --2- 172.21.15.70:0/214704331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90600ffc20 0x7f9060195570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.272+0000 7f9066927640 1 -- 172.21.15.70:0/214704331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90600ffc20 msgr2=0x7f9060195570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.272+0000 7f9066927640 1 --2- 172.21.15.70:0/214704331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90600ffc20 0x7f9060195570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.272+0000 7f9066927640 1 -- 172.21.15.70:0/214704331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9060100e10 msgr2=0x7f9060195ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.272+0000 7f9066927640 1 --2- 172.21.15.70:0/214704331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9060100e10 0x7f9060195ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.272+0000 7f9066927640 1 -- 172.21.15.70:0/214704331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906019d190 con 0x7f90601017d0 2024-04-01T01:38:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.272+0000 7f9065925640 1 --2- 172.21.15.70:0/214704331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9060100e10 0x7f9060195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:38:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.272+0000 7f9066927640 1 --2- 172.21.15.70:0/214704331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90601017d0 0x7f906019cb20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9058006010 tx=0x7f9058003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.273+0000 7f90577fe640 1 -- 172.21.15.70:0/214704331 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905804a020 con 0x7f90601017d0 2024-04-01T01:38:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.274+0000 7f9067128640 1 -- 172.21.15.70:0/214704331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f906019d420 con 0x7f90601017d0 2024-04-01T01:38:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.274+0000 7f9067128640 1 -- 172.21.15.70:0/214704331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f906019d970 con 0x7f90601017d0 2024-04-01T01:38:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.274+0000 7f90577fe640 1 -- 172.21.15.70:0/214704331 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f905800ba80 con 0x7f90601017d0 2024-04-01T01:38:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.274+0000 7f90577fe640 1 -- 172.21.15.70:0/214704331 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058040040 con 0x7f90601017d0 2024-04-01T01:38:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.277+0000 7f906af6e740 1 Processor -- start 2024-04-01T01:38:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.278+0000 7f90577fe640 1 -- 172.21.15.70:0/214704331 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f905804d020 con 0x7f90601017d0 2024-04-01T01:38:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.278+0000 7f906af6e740 1 -- start start 2024-04-01T01:38:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.278+0000 7f906af6e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a59ae0 0x561218a59f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.278+0000 7f906af6e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a5a4d0 0x561218a5efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.278+0000 7f906af6e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5f640 0x561218a61ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.278+0000 7f906af6e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5612189bd8f0 con 0x561218a59ae0 2024-04-01T01:38:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.278+0000 7f906af6e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5612189668d0 con 0x561218a5f640 2024-04-01T01:38:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.278+0000 7f906af6e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5612189b9990 con 0x561218a5a4d0 2024-04-01T01:38:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.279+0000 7f9065925640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a59ae0 0x561218a59f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.279+0000 7f9065925640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a59ae0 0x561218a59f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38094/0 (socket says 172.21.15.70:38094) 2024-04-01T01:38:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.279+0000 7f9065925640 1 -- 172.21.15.70:0/1887630280 learned_addr learned my addr 172.21.15.70:0/1887630280 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:23.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.279+0000 7f9065925640 1 --2- 172.21.15.70:0/1887630280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5f640 0x561218a61ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.279+0000 7f9066126640 1 --2- 172.21.15.70:0/1887630280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a5a4d0 0x561218a5efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.279+0000 7f9066126640 1 -- 172.21.15.70:0/1887630280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5f640 msgr2=0x561218a61ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:23.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.279+0000 7f9066126640 1 --2- 172.21.15.70:0/1887630280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5f640 0x561218a61ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.279+0000 7f9066126640 1 -- 172.21.15.70:0/1887630280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a59ae0 msgr2=0x561218a59f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:23.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.279+0000 7f9066126640 1 --2- 172.21.15.70:0/1887630280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a59ae0 0x561218a59f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.279+0000 7f9066126640 1 -- 172.21.15.70:0/1887630280 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612186d2210 con 0x561218a5a4d0 2024-04-01T01:38:23.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.280+0000 7f9066126640 1 --2- 172.21.15.70:0/1887630280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a5a4d0 0x561218a5efd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f905c005fd0 tx=0x7f905c00abb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46ab4b5f1516b58c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:23.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.280+0000 7f90577fe640 1 --2- 172.21.15.70:0/214704331 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f904c0a25e0 0x7f904c0a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.280+0000 7f90577fe640 1 -- 172.21.15.70:0/214704331 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f90580f91e0 con 0x7f90601017d0 2024-04-01T01:38:23.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.280+0000 7f9065925640 1 --2- 172.21.15.70:0/214704331 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f904c0a25e0 0x7f904c0a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.280+0000 7f9054bfe640 1 -- 172.21.15.70:0/1887630280 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c00ae00 con 0x561218a5a4d0 2024-04-01T01:38:23.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.280+0000 7f9054bfe640 1 -- 172.21.15.70:0/1887630280 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f905c003ce0 con 0x561218a5a4d0 2024-04-01T01:38:23.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.280+0000 7f9054bfe640 1 -- 172.21.15.70:0/1887630280 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c005640 con 0x561218a5a4d0 2024-04-01T01:38:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.281+0000 7f906af6e740 1 -- 172.21.15.70:0/1887630280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a5a4d0 msgr2=0x561218a5efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.281+0000 7f906af6e740 1 --2- 172.21.15.70:0/1887630280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a5a4d0 0x561218a5efd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f905c005fd0 tx=0x7f905c00abb0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.281+0000 7f9065925640 1 --2- 172.21.15.70:0/214704331 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f904c0a25e0 0x7f904c0a4aa0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x561218a5ad30 tx=0x7f905000bbb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.281+0000 7f906af6e740 1 -- 172.21.15.70:0/1887630280 shutdown_connections 2024-04-01T01:38:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.281+0000 7f906af6e740 1 --2- 172.21.15.70:0/1887630280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5f640 0x561218a61ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.281+0000 7f906af6e740 1 --2- 172.21.15.70:0/1887630280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a5a4d0 0x561218a5efd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.281+0000 7f906af6e740 1 --2- 172.21.15.70:0/1887630280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a59ae0 0x561218a59f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.281+0000 7f906af6e740 1 -- 172.21.15.70:0/1887630280 >> 172.21.15.70:0/1887630280 conn(0x561218a55050 msgr2=0x561218a55450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.281+0000 7f906af6e740 1 -- 172.21.15.70:0/1887630280 shutdown_connections 2024-04-01T01:38:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.281+0000 7f906af6e740 1 -- 172.21.15.70:0/1887630280 wait complete. 2024-04-01T01:38:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.282+0000 7f906af6e740 1 -- start start 2024-04-01T01:38:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.282+0000 7f906af6e740 1 Processor -- start 2024-04-01T01:38:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.282+0000 7f906af6e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a59ae0 0x561218a74e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.282+0000 7f906af6e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a5a4d0 0x561218a75380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.282+0000 7f906af6e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5f640 0x561218a7c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.282+0000 7f9066126640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a59ae0 0x561218a74e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.282+0000 7f9066126640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a59ae0 0x561218a74e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38106/0 (socket says 172.21.15.70:38106) 2024-04-01T01:38:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.282+0000 7f906af6e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561218a73650 con 0x561218a59ae0 2024-04-01T01:38:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.282+0000 7f9066126640 1 -- 172.21.15.70:0/700136099 learned_addr learned my addr 172.21.15.70:0/700136099 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.282+0000 7f906af6e740 1 -- 172.21.15.70:0/700136099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56121895bf80 con 0x561218a5f640 2024-04-01T01:38:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.282+0000 7f906af6e740 1 -- 172.21.15.70:0/700136099 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561218a737d0 con 0x561218a5a4d0 2024-04-01T01:38:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.282+0000 7f9066927640 1 --2- 172.21.15.70:0/700136099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5f640 0x561218a7c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.282+0000 7f9065925640 1 --2- 172.21.15.70:0/700136099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a5a4d0 0x561218a75380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.283+0000 7f906af6e740 10 client.0 _ll_register_callbacks cb 0x5612189b4290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.283+0000 7f9066927640 1 -- 172.21.15.70:0/700136099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a5a4d0 msgr2=0x561218a75380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.283+0000 7f9066927640 1 --2- 172.21.15.70:0/700136099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a5a4d0 0x561218a75380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.283+0000 7f9066927640 1 -- 172.21.15.70:0/700136099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a59ae0 msgr2=0x561218a74e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.283+0000 7f9066927640 1 --2- 172.21.15.70:0/700136099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a59ae0 0x561218a74e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.283+0000 7f9066927640 1 -- 172.21.15.70:0/700136099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56121879e620 con 0x561218a5f640 2024-04-01T01:38:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.283+0000 7f9066927640 1 --2- 172.21.15.70:0/700136099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5f640 0x561218a7c350 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f90580f6960 tx=0x7f90580c54f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f50ecf946a84cbee server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.283+0000 7f9049ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.283+0000 7f9049ffb640 1 -- 172.21.15.70:0/700136099 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90580de060 con 0x561218a5f640 2024-04-01T01:38:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.283+0000 7f906af6e740 1 -- 172.21.15.70:0/700136099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f905c003ce0 con 0x561218a5f640 2024-04-01T01:38:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.284+0000 7f9049ffb640 1 -- 172.21.15.70:0/700136099 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f90580cde50 con 0x561218a5f640 2024-04-01T01:38:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.286+0000 7f9049ffb640 1 -- 172.21.15.70:0/700136099 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f90580e3050 con 0x561218a5f640 2024-04-01T01:38:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.287+0000 7f906af6e740 10 client.26933 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.287+0000 7f906af6e740 1 -- 172.21.15.70:0/700136099 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561218a7f100 con 0x561218a5f640 2024-04-01T01:38:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.287+0000 7f9049ffb640 1 -- 172.21.15.70:0/700136099 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f905c003ce0 con 0x561218a5f640 2024-04-01T01:38:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.288+0000 7f9049ffb640 1 -- 172.21.15.70:0/700136099 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9058043030 con 0x561218a5f640 2024-04-01T01:38:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.288+0000 7f906af6e740 10 client.26933 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.288+0000 7f906af6e740 10 client.26933 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.288+0000 7f906af6e740 20 client.26933 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.288+0000 7f906af6e740 1 --2- 172.21.15.70:0/700136099 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561218a80a10 0x561218a82ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.288+0000 7f906af6e740 4 client.26933 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.288+0000 7f906af6e740 1 -- 172.21.15.70:0/700136099 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612189d1120 con 0x561218a80a10 2024-04-01T01:38:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.288+0000 7f9066126640 1 --2- 172.21.15.70:0/700136099 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561218a80a10 0x561218a82ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.289+0000 7f9066126640 1 --2- 172.21.15.70:0/700136099 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561218a80a10 0x561218a82ea0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3c2bf10be14058f server_cookie=fc6845155181cdc2 in_seq=0 out_seq=0 2024-04-01T01:38:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.289+0000 7f9049ffb640 10 client.26933 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:23.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.290+0000 7f9049ffb640 1 -- 172.21.15.70:0/700136099 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5612189d1120 con 0x561218a80a10 2024-04-01T01:38:23.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.290+0000 7f9049ffb640 10 client.26933 handle_command_reply: tid=0 2024-04-01T01:38:23.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.290+0000 7f906af6e740 1 client.26933 shutdown 2024-04-01T01:38:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.291+0000 7f906af6e740 1 -- 172.21.15.70:0/700136099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5f640 msgr2=0x561218a7c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.291+0000 7f906af6e740 1 --2- 172.21.15.70:0/700136099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5f640 0x561218a7c350 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f90580f6960 tx=0x7f90580c54f0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.291+0000 7f906af6e740 1 -- 172.21.15.70:0/700136099 shutdown_connections 2024-04-01T01:38:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.291+0000 7f906af6e740 1 --2- 172.21.15.70:0/700136099 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561218a80a10 0x561218a82ea0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.291+0000 7f906af6e740 1 --2- 172.21.15.70:0/700136099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5f640 0x561218a7c350 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.291+0000 7f9049ffb640 0 client.26933 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.292+0000 7f906af6e740 1 --2- 172.21.15.70:0/700136099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a5a4d0 0x561218a75380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.292+0000 7f906af6e740 1 --2- 172.21.15.70:0/700136099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a59ae0 0x561218a74e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.292+0000 7f906af6e740 1 -- 172.21.15.70:0/700136099 >> 172.21.15.70:0/700136099 conn(0x561218a55050 msgr2=0x561218a54190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.292+0000 7f906af6e740 1 -- 172.21.15.70:0/700136099 shutdown_connections 2024-04-01T01:38:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.292+0000 7f906af6e740 1 -- 172.21.15.70:0/700136099 wait complete. 2024-04-01T01:38:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.293+0000 7f906af6e740 20 client.26933 trim_cache size 0 max 16384 2024-04-01T01:38:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.323+0000 7f906af6e740 1 Processor -- start 2024-04-01T01:38:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.324+0000 7f906af6e740 1 -- start start 2024-04-01T01:38:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.324+0000 7f906af6e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a55050 0x561218a55430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.324+0000 7f906af6e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a59ae0 0x561218a7c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.325+0000 7f906af6e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a5a4d0 0x561218a88780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.325+0000 7f906af6e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f90601a8070 con 0x561218a5a4d0 2024-04-01T01:38:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.325+0000 7f906af6e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f90601a9bb0 con 0x561218a59ae0 2024-04-01T01:38:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.325+0000 7f906af6e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f90601a8740 con 0x561218a55050 2024-04-01T01:38:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.325+0000 7f9065925640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a59ae0 0x561218a7c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.325+0000 7f9065925640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a59ae0 0x561218a7c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35278/0 (socket says 172.21.15.70:35278) 2024-04-01T01:38:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.325+0000 7f9065925640 1 -- 172.21.15.70:0/2319625056 learned_addr learned my addr 172.21.15.70:0/2319625056 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.325+0000 7f9066927640 1 --2- 172.21.15.70:0/2319625056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a5a4d0 0x561218a88780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.325+0000 7f9065925640 1 -- 172.21.15.70:0/2319625056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a55050 msgr2=0x561218a55430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:38:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.325+0000 7f9065925640 1 --2- 172.21.15.70:0/2319625056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a55050 0x561218a55430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.325+0000 7f9065925640 1 -- 172.21.15.70:0/2319625056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a5a4d0 msgr2=0x561218a88780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.325+0000 7f9065925640 1 --2- 172.21.15.70:0/2319625056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a5a4d0 0x561218a88780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.325+0000 7f9065925640 1 -- 172.21.15.70:0/2319625056 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561218a5ea50 con 0x561218a59ae0 2024-04-01T01:38:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.326+0000 7f9065925640 1 --2- 172.21.15.70:0/2319625056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a59ae0 0x561218a7c310 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f90500058c0 tx=0x7f90500046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9478b2b52e58ecce server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.326+0000 7f904bfff640 1 -- 172.21.15.70:0/2319625056 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905001ccf0 con 0x561218a59ae0 2024-04-01T01:38:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.326+0000 7f904bfff640 1 -- 172.21.15.70:0/2319625056 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9050005020 con 0x561218a59ae0 2024-04-01T01:38:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.326+0000 7f904bfff640 1 -- 172.21.15.70:0/2319625056 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90500047a0 con 0x561218a59ae0 2024-04-01T01:38:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.328+0000 7f906af6e740 1 -- 172.21.15.70:0/2319625056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a59ae0 msgr2=0x561218a7c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.328+0000 7f906af6e740 1 --2- 172.21.15.70:0/2319625056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a59ae0 0x561218a7c310 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f90500058c0 tx=0x7f90500046c0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.328+0000 7f906af6e740 1 -- 172.21.15.70:0/2319625056 shutdown_connections 2024-04-01T01:38:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.328+0000 7f906af6e740 1 --2- 172.21.15.70:0/2319625056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a5a4d0 0x561218a88780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.328+0000 7f906af6e740 1 --2- 172.21.15.70:0/2319625056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a59ae0 0x561218a7c310 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.328+0000 7f906af6e740 1 --2- 172.21.15.70:0/2319625056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a55050 0x561218a55430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.328+0000 7f906af6e740 1 -- 172.21.15.70:0/2319625056 >> 172.21.15.70:0/2319625056 conn(0x561218a5f140 msgr2=0x561218a4fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.329+0000 7f906af6e740 1 -- 172.21.15.70:0/2319625056 shutdown_connections 2024-04-01T01:38:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.330+0000 7f906af6e740 1 -- 172.21.15.70:0/2319625056 wait complete. 2024-04-01T01:38:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.330+0000 7f906af6e740 1 -- start start 2024-04-01T01:38:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.330+0000 7f906af6e740 1 Processor -- start 2024-04-01T01:38:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.331+0000 7f906af6e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a55050 0x561218a77720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.331+0000 7f906af6e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a59ae0 0x561218a77c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.331+0000 7f906af6e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5a4d0 0x561218a7ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.332+0000 7f9065925640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a59ae0 0x561218a77c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.332+0000 7f9065925640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a59ae0 0x561218a77c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57618/0 (socket says 172.21.15.70:57618) 2024-04-01T01:38:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.332+0000 7f9065925640 1 -- 172.21.15.70:0/1167664230 learned_addr learned my addr 172.21.15.70:0/1167664230 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.332+0000 7f9066126640 1 --2- 172.21.15.70:0/1167664230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a55050 0x561218a77720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.331+0000 7f906af6e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f90601a94e0 con 0x561218a55050 2024-04-01T01:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.332+0000 7f9066927640 1 --2- 172.21.15.70:0/1167664230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5a4d0 0x561218a7ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.332+0000 7f906af6e740 1 -- 172.21.15.70:0/1167664230 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f90601a8e10 con 0x561218a5a4d0 2024-04-01T01:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.332+0000 7f906af6e740 1 -- 172.21.15.70:0/1167664230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f90601a6050 con 0x561218a59ae0 2024-04-01T01:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.332+0000 7f906af6e740 10 client.0 _ll_register_callbacks cb 0x7f9058005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.332+0000 7f9066927640 1 -- 172.21.15.70:0/1167664230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a59ae0 msgr2=0x561218a77c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.332+0000 7f9066927640 1 --2- 172.21.15.70:0/1167664230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a59ae0 0x561218a77c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.332+0000 7f9066927640 1 -- 172.21.15.70:0/1167664230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a55050 msgr2=0x561218a77720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.332+0000 7f9066927640 1 --2- 172.21.15.70:0/1167664230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a55050 0x561218a77720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.332+0000 7f9066927640 1 -- 172.21.15.70:0/1167664230 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561218a7b270 con 0x561218a5a4d0 2024-04-01T01:38:23.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.333+0000 7f9066927640 1 --2- 172.21.15.70:0/1167664230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5a4d0 0x561218a7ac80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9058004af0 tx=0x7f90580c5a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca323642244b585 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:23.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.333+0000 7f9049ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:23.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.333+0000 7f9049ffb640 1 -- 172.21.15.70:0/1167664230 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9058013080 con 0x561218a5a4d0 2024-04-01T01:38:23.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.333+0000 7f9049ffb640 1 -- 172.21.15.70:0/1167664230 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f90580165a0 con 0x561218a5a4d0 2024-04-01T01:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.334+0000 7f906af6e740 1 -- 172.21.15.70:0/1167664230 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9050005020 con 0x561218a5a4d0 2024-04-01T01:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.334+0000 7f9049ffb640 1 -- 172.21.15.70:0/1167664230 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f90580c1e60 con 0x561218a5a4d0 2024-04-01T01:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.334+0000 7f906af6e740 10 client.26939 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.334+0000 7f906af6e740 1 -- 172.21.15.70:0/1167664230 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561218a7c5f0 con 0x561218a5a4d0 2024-04-01T01:38:23.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.334+0000 7f9049ffb640 1 -- 172.21.15.70:0/1167664230 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f90580c9a20 con 0x561218a5a4d0 2024-04-01T01:38:23.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.335+0000 7f9049ffb640 1 -- 172.21.15.70:0/1167664230 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9058046bd0 con 0x561218a5a4d0 2024-04-01T01:38:23.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.336+0000 7f906af6e740 10 client.26939 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:23.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.336+0000 7f906af6e740 10 client.26939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:23.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.336+0000 7f906af6e740 20 client.26939 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:23.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.336+0000 7f906af6e740 1 --2- 172.21.15.70:0/1167664230 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561218a5f640 0x561218afa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:23.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.336+0000 7f906af6e740 4 client.26939 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:38:23.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.336+0000 7f906af6e740 1 -- 172.21.15.70:0/1167664230 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f9050005020 con 0x561218a5f640 2024-04-01T01:38:23.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.336+0000 7f9066126640 1 --2- 172.21.15.70:0/1167664230 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561218a5f640 0x561218afa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:23.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.337+0000 7f9066126640 1 --2- 172.21.15.70:0/1167664230 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561218a5f640 0x561218afa240 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44bfe34f10ca6ef4 server_cookie=47d61da1550f43d2 in_seq=0 out_seq=0 2024-04-01T01:38:23.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.337+0000 7f9049ffb640 10 client.26939 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.384+0000 7f9049ffb640 1 -- 172.21.15.70:0/1167664230 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f9050005020 con 0x561218a5f640 2024-04-01T01:38:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.384+0000 7f9049ffb640 10 client.26939 handle_command_reply: tid=0 2024-04-01T01:38:23.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.385+0000 7f906af6e740 1 client.26939 shutdown 2024-04-01T01:38:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.387+0000 7f906af6e740 1 -- 172.21.15.70:0/1167664230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5a4d0 msgr2=0x561218a7ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.387+0000 7f906af6e740 1 --2- 172.21.15.70:0/1167664230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5a4d0 0x561218a7ac80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9058004af0 tx=0x7f90580c5a50 comp rx=0 tx=0).stop 2024-04-01T01:38:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.387+0000 7f906af6e740 1 -- 172.21.15.70:0/1167664230 shutdown_connections 2024-04-01T01:38:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.387+0000 7f906af6e740 1 --2- 172.21.15.70:0/1167664230 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561218a5f640 0x561218afa240 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.387+0000 7f906af6e740 1 --2- 172.21.15.70:0/1167664230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561218a5a4d0 0x561218a7ac80 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.387+0000 7f9049ffb640 0 client.26939 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.387+0000 7f906af6e740 1 --2- 172.21.15.70:0/1167664230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561218a59ae0 0x561218a77c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.387+0000 7f906af6e740 1 --2- 172.21.15.70:0/1167664230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561218a55050 0x561218a77720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.387+0000 7f906af6e740 1 -- 172.21.15.70:0/1167664230 >> 172.21.15.70:0/1167664230 conn(0x561218a5f140 msgr2=0x561218a55430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.387+0000 7f906af6e740 1 -- 172.21.15.70:0/1167664230 shutdown_connections 2024-04-01T01:38:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.387+0000 7f906af6e740 1 -- 172.21.15.70:0/1167664230 wait complete. 2024-04-01T01:38:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.388+0000 7f906af6e740 20 client.26939 trim_cache size 0 max 16384 2024-04-01T01:38:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:38:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:38:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:38:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:38:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.389+0000 7f9067128640 1 -- 172.21.15.70:0/214704331 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f904c0a25e0 msgr2=0x7f904c0a4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.389+0000 7f9067128640 1 --2- 172.21.15.70:0/214704331 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f904c0a25e0 0x7f904c0a4aa0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x561218a5ad30 tx=0x7f905000bbb0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.389+0000 7f9067128640 1 -- 172.21.15.70:0/214704331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90601017d0 msgr2=0x7f906019cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.389+0000 7f9067128640 1 --2- 172.21.15.70:0/214704331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90601017d0 0x7f906019cb20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9058006010 tx=0x7f9058003ba0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.390+0000 7f9067128640 1 -- 172.21.15.70:0/214704331 shutdown_connections 2024-04-01T01:38:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.390+0000 7f9067128640 1 --2- 172.21.15.70:0/214704331 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f904c0a25e0 0x7f904c0a4aa0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.390+0000 7f9067128640 1 --2- 172.21.15.70:0/214704331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90601017d0 0x7f906019cb20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.390+0000 7f9067128640 1 --2- 172.21.15.70:0/214704331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9060100e10 0x7f9060195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.390+0000 7f9067128640 1 --2- 172.21.15.70:0/214704331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90600ffc20 0x7f9060195570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.390+0000 7f9067128640 1 -- 172.21.15.70:0/214704331 >> 172.21.15.70:0/214704331 conn(0x7f90600fb190 msgr2=0x7f9060104180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.392+0000 7f9067128640 1 -- 172.21.15.70:0/214704331 shutdown_connections 2024-04-01T01:38:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:23.392+0000 7f9067128640 1 -- 172.21.15.70:0/214704331 wait complete. 2024-04-01T01:38:24.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:23 smithi081 ceph-mon[35142]: pgmap v505: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 9.4 MiB/s wr, 1.19k op/s 2024-04-01T01:38:24.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:23 smithi189 ceph-mon[35753]: pgmap v505: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 9.4 MiB/s wr, 1.19k op/s 2024-04-01T01:38:24.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:23 smithi070 ceph-mon[32179]: pgmap v505: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 9.4 MiB/s wr, 1.19k op/s 2024-04-01T01:38:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:25 smithi081 ceph-mon[35142]: pgmap v506: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 8.3 MiB/s wr, 1.06k op/s 2024-04-01T01:38:26.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:25 smithi189 ceph-mon[35753]: pgmap v506: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 8.3 MiB/s wr, 1.06k op/s 2024-04-01T01:38:26.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:25 smithi070 ceph-mon[32179]: pgmap v506: 129 pgs: 129 active+clean; 1.0 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 8.3 MiB/s wr, 1.06k op/s 2024-04-01T01:38:26.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 220.0 s, 1161.4 tps, lat 85.991 ms stddev 57.403 2024-04-01T01:38:28.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:27 smithi081 ceph-mon[35142]: pgmap v507: 129 pgs: 129 active+clean; 1.0 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 9.6 MiB/s wr, 1.22k op/s 2024-04-01T01:38:28.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:27 smithi189 ceph-mon[35753]: pgmap v507: 129 pgs: 129 active+clean; 1.0 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 9.6 MiB/s wr, 1.22k op/s 2024-04-01T01:38:28.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:27 smithi070 ceph-mon[32179]: pgmap v507: 129 pgs: 129 active+clean; 1.0 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 9.6 MiB/s wr, 1.22k op/s 2024-04-01T01:38:30.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:29 smithi081 ceph-mon[35142]: pgmap v508: 129 pgs: 129 active+clean; 1.0 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 9.2 MiB/s wr, 1.12k op/s 2024-04-01T01:38:30.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:29 smithi189 ceph-mon[35753]: pgmap v508: 129 pgs: 129 active+clean; 1.0 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 9.2 MiB/s wr, 1.12k op/s 2024-04-01T01:38:30.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:29 smithi070 ceph-mon[32179]: pgmap v508: 129 pgs: 129 active+clean; 1.0 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 9.2 MiB/s wr, 1.12k op/s 2024-04-01T01:38:31.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:30 smithi081 ceph-mon[35142]: pgmap v509: 129 pgs: 129 active+clean; 1.0 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.8 MiB/s wr, 1.04k op/s 2024-04-01T01:38:31.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:30 smithi189 ceph-mon[35753]: pgmap v509: 129 pgs: 129 active+clean; 1.0 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.8 MiB/s wr, 1.04k op/s 2024-04-01T01:38:31.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:30 smithi070 ceph-mon[32179]: pgmap v509: 129 pgs: 129 active+clean; 1.0 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.8 MiB/s wr, 1.04k op/s 2024-04-01T01:38:31.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 225.0 s, 1115.6 tps, lat 89.971 ms stddev 124.228 2024-04-01T01:38:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:33 smithi081 ceph-mon[35142]: pgmap v510: 129 pgs: 129 active+clean; 1.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 MiB/s wr, 1.14k op/s 2024-04-01T01:38:34.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:33 smithi189 ceph-mon[35753]: pgmap v510: 129 pgs: 129 active+clean; 1.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 MiB/s wr, 1.14k op/s 2024-04-01T01:38:34.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:33 smithi070 ceph-mon[32179]: pgmap v510: 129 pgs: 129 active+clean; 1.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 10 MiB/s wr, 1.14k op/s 2024-04-01T01:38:36.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:35 smithi081 ceph-mon[35142]: pgmap v511: 129 pgs: 129 active+clean; 1.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 MiB/s wr, 1.03k op/s 2024-04-01T01:38:36.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:35 smithi189 ceph-mon[35753]: pgmap v511: 129 pgs: 129 active+clean; 1.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 MiB/s wr, 1.03k op/s 2024-04-01T01:38:36.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:35 smithi070 ceph-mon[32179]: pgmap v511: 129 pgs: 129 active+clean; 1.0 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 MiB/s wr, 1.03k op/s 2024-04-01T01:38:36.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.539+0000 7fcc3d48f640 1 -- 172.21.15.70:0/1091118409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc3806f3d0 msgr2=0x7fcc3806fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.539+0000 7fcc3d48f640 1 --2- 172.21.15.70:0/1091118409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc3806f3d0 0x7fcc3806fc70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c009900 tx=0x7fcc2c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:38:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.539+0000 7fcc3d48f640 1 -- 172.21.15.70:0/1091118409 shutdown_connections 2024-04-01T01:38:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.539+0000 7fcc3d48f640 1 --2- 172.21.15.70:0/1091118409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc3806f3d0 0x7fcc3806fc70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.539+0000 7fcc3d48f640 1 --2- 172.21.15.70:0/1091118409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc3806ea30 0x7fcc3806ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.539+0000 7fcc3d48f640 1 --2- 172.21.15.70:0/1091118409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc3806d7f0 0x7fcc3806dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.539+0000 7fcc3d48f640 1 -- 172.21.15.70:0/1091118409 >> 172.21.15.70:0/1091118409 conn(0x7fcc3806c330 msgr2=0x7fcc3806c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.539+0000 7fcc3d48f640 1 -- 172.21.15.70:0/1091118409 shutdown_connections 2024-04-01T01:38:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.539+0000 7fcc3d48f640 1 -- 172.21.15.70:0/1091118409 wait complete. 2024-04-01T01:38:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.540+0000 7fcc3d48f640 1 Processor -- start 2024-04-01T01:38:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.540+0000 7fcc3d48f640 1 -- start start 2024-04-01T01:38:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.541+0000 7fcc3d48f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc3806d7f0 0x7fcc381a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.541+0000 7fcc3d48f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc3806ea30 0x7fcc381a6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.541+0000 7fcc3d48f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc3806f3d0 0x7fcc381adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.541+0000 7fcc3d48f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3811c140 con 0x7fcc3806ea30 2024-04-01T01:38:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.541+0000 7fcc3d48f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3811bfc0 con 0x7fcc3806d7f0 2024-04-01T01:38:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.541+0000 7fcc3d48f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3811c2c0 con 0x7fcc3806f3d0 2024-04-01T01:38:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.542+0000 7fcc367fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc3806ea30 0x7fcc381a6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.542+0000 7fcc36ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc3806d7f0 0x7fcc381a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.542+0000 7fcc36ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc3806d7f0 0x7fcc381a6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:43044/0 (socket says 172.21.15.70:43044) 2024-04-01T01:38:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.542+0000 7fcc36ffd640 1 -- 172.21.15.70:0/921965897 learned_addr learned my addr 172.21.15.70:0/921965897 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.542+0000 7fcc377fe640 1 --2- 172.21.15.70:0/921965897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc3806f3d0 0x7fcc381adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.542+0000 7fcc36ffd640 1 -- 172.21.15.70:0/921965897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc3806f3d0 msgr2=0x7fcc381adcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.542+0000 7fcc36ffd640 1 --2- 172.21.15.70:0/921965897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc3806f3d0 0x7fcc381adcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.542+0000 7fcc36ffd640 1 -- 172.21.15.70:0/921965897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc3806ea30 msgr2=0x7fcc381a6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.542+0000 7fcc36ffd640 1 --2- 172.21.15.70:0/921965897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc3806ea30 0x7fcc381a6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.542+0000 7fcc36ffd640 1 -- 172.21.15.70:0/921965897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc381ae320 con 0x7fcc3806d7f0 2024-04-01T01:38:36.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.542+0000 7fcc377fe640 1 --2- 172.21.15.70:0/921965897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc3806f3d0 0x7fcc381adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:38:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.543+0000 7fcc36ffd640 1 --2- 172.21.15.70:0/921965897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc3806d7f0 0x7fcc381a6750 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000ebc0 tx=0x7fcc3000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.543+0000 7fcc1ffff640 1 -- 172.21.15.70:0/921965897 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc30010430 con 0x7fcc3806d7f0 2024-04-01T01:38:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.543+0000 7fcc3d48f640 1 -- 172.21.15.70:0/921965897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc381ae610 con 0x7fcc3806d7f0 2024-04-01T01:38:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.543+0000 7fcc1ffff640 1 -- 172.21.15.70:0/921965897 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc30010a80 con 0x7fcc3806d7f0 2024-04-01T01:38:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.543+0000 7fcc3d48f640 1 -- 172.21.15.70:0/921965897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc381aebe0 con 0x7fcc3806d7f0 2024-04-01T01:38:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.544+0000 7fcc1ffff640 1 -- 172.21.15.70:0/921965897 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc30010430 con 0x7fcc3806d7f0 2024-04-01T01:38:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.547+0000 7fcc4006c740 1 Processor -- start 2024-04-01T01:38:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.547+0000 7fcc4006c740 1 -- start start 2024-04-01T01:38:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.547+0000 7fcc4006c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb073c8d0 0x564cb073ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.547+0000 7fcc4006c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073d2c0 0x564cb0741dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.547+0000 7fcc4006c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb07448c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.547+0000 7fcc4006c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564cb06a08f0 con 0x564cb073d2c0 2024-04-01T01:38:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.547+0000 7fcc4006c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x564cb06498d0 con 0x564cb073c8d0 2024-04-01T01:38:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.547+0000 7fcc4006c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564cb069c990 con 0x564cb0742430 2024-04-01T01:38:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.547+0000 7fcc367fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb07448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.547+0000 7fcc367fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb07448c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52844/0 (socket says 172.21.15.70:52844) 2024-04-01T01:38:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.547+0000 7fcc367fc640 1 -- 172.21.15.70:0/2958363355 learned_addr learned my addr 172.21.15.70:0/2958363355 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.548+0000 7fcc367fc640 1 -- 172.21.15.70:0/2958363355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb073c8d0 msgr2=0x564cb073ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.547+0000 7fcc377fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb073c8d0 0x564cb073ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.549+0000 7fcc36ffd640 1 --2- 172.21.15.70:0/2958363355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073d2c0 0x564cb0741dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.549+0000 7fcc367fc640 1 --2- 172.21.15.70:0/2958363355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb073c8d0 0x564cb073ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.549+0000 7fcc367fc640 1 -- 172.21.15.70:0/2958363355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073d2c0 msgr2=0x564cb0741dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.549+0000 7fcc367fc640 1 --2- 172.21.15.70:0/2958363355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073d2c0 0x564cb0741dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.549+0000 7fcc367fc640 1 -- 172.21.15.70:0/2958363355 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cb03b5210 con 0x564cb0742430 2024-04-01T01:38:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.549+0000 7fcc377fe640 1 --2- 172.21.15.70:0/2958363355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb073c8d0 0x564cb073ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:38:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.549+0000 7fcc1ffff640 1 -- 172.21.15.70:0/921965897 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fcc30010c30 con 0x7fcc3806d7f0 2024-04-01T01:38:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.550+0000 7fcc367fc640 1 --2- 172.21.15.70:0/2958363355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb07448c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009a40 tx=0x7fcc28009f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=966679076990387c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.550+0000 7fcc1d7fa640 1 -- 172.21.15.70:0/2958363355 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc28017070 con 0x564cb0742430 2024-04-01T01:38:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.550+0000 7fcc1ffff640 1 --2- 172.21.15.70:0/921965897 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcc200a29c0 0x7fcc200a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.550+0000 7fcc1d7fa640 1 -- 172.21.15.70:0/2958363355 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc28013470 con 0x564cb0742430 2024-04-01T01:38:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.550+0000 7fcc1d7fa640 1 -- 172.21.15.70:0/2958363355 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc28017070 con 0x564cb0742430 2024-04-01T01:38:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.551+0000 7fcc1ffff640 1 -- 172.21.15.70:0/921965897 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fcc3000cd10 con 0x7fcc3806d7f0 2024-04-01T01:38:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.551+0000 7fcc377fe640 1 --2- 172.21.15.70:0/921965897 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcc200a29c0 0x7fcc200a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.552+0000 7fcc4006c740 1 -- 172.21.15.70:0/2958363355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 msgr2=0x564cb07448c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.552+0000 7fcc4006c740 1 --2- 172.21.15.70:0/2958363355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb07448c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009a40 tx=0x7fcc28009f00 comp rx=0 tx=0).stop 2024-04-01T01:38:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.552+0000 7fcc377fe640 1 --2- 172.21.15.70:0/921965897 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcc200a29c0 0x7fcc200a4e80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c006010 tx=0x7fcc2c02c5d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.552+0000 7fcc36ffd640 1 --2- 172.21.15.70:0/2958363355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073d2c0 0x564cb0741dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:38:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.552+0000 7fcc4006c740 1 -- 172.21.15.70:0/2958363355 shutdown_connections 2024-04-01T01:38:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.552+0000 7fcc4006c740 1 --2- 172.21.15.70:0/2958363355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb07448c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.552+0000 7fcc4006c740 1 --2- 172.21.15.70:0/2958363355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073d2c0 0x564cb0741dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.552+0000 7fcc4006c740 1 --2- 172.21.15.70:0/2958363355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb073c8d0 0x564cb073ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.552+0000 7fcc4006c740 1 -- 172.21.15.70:0/2958363355 >> 172.21.15.70:0/2958363355 conn(0x564cb0737e40 msgr2=0x564cb0738240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.552+0000 7fcc4006c740 1 -- 172.21.15.70:0/2958363355 shutdown_connections 2024-04-01T01:38:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.553+0000 7fcc4006c740 1 -- 172.21.15.70:0/2958363355 wait complete. 2024-04-01T01:38:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.553+0000 7fcc4006c740 1 -- start start 2024-04-01T01:38:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.554+0000 7fcc4006c740 1 Processor -- start 2024-04-01T01:38:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.554+0000 7fcc4006c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb073c8d0 0x564cb0757a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.554+0000 7fcc4006c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073d2c0 0x564cb0757fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.554+0000 7fcc4006c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb0742430 0x564cb075efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.554+0000 7fcc4006c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564cb07563f0 con 0x564cb073d2c0 2024-04-01T01:38:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.554+0000 7fcc4006c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x564cb063ef80 con 0x564cb0742430 2024-04-01T01:38:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.554+0000 7fcc4006c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564cb0756570 con 0x564cb073c8d0 2024-04-01T01:38:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.554+0000 7fcc4006c740 10 client.0 _ll_register_callbacks cb 0x564cb0697290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.554+0000 7fcc367fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073d2c0 0x564cb0757fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.554+0000 7fcc367fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073d2c0 0x564cb0757fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36522/0 (socket says 172.21.15.70:36522) 2024-04-01T01:38:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.554+0000 7fcc367fc640 1 -- 172.21.15.70:0/1054677615 learned_addr learned my addr 172.21.15.70:0/1054677615 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.554+0000 7fcc36ffd640 1 --2- 172.21.15.70:0/1054677615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb073c8d0 0x564cb0757a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.555+0000 7fcc377fe640 1 --2- 172.21.15.70:0/1054677615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb0742430 0x564cb075efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.555+0000 7fcc367fc640 1 -- 172.21.15.70:0/1054677615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb073c8d0 msgr2=0x564cb0757a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.555+0000 7fcc367fc640 1 --2- 172.21.15.70:0/1054677615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb073c8d0 0x564cb0757a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.555+0000 7fcc367fc640 1 -- 172.21.15.70:0/1054677615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb0742430 msgr2=0x564cb075efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.555+0000 7fcc367fc640 1 --2- 172.21.15.70:0/1054677615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb0742430 0x564cb075efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.555+0000 7fcc367fc640 1 -- 172.21.15.70:0/1054677615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cb0481620 con 0x564cb073d2c0 2024-04-01T01:38:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.555+0000 7fcc1aff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:38:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.555+0000 7fcc367fc640 1 --2- 172.21.15.70:0/1054677615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073d2c0 0x564cb0757fa0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fcc28013650 tx=0x7fcc28013680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e5266f08e7f85d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.556+0000 7fcc1aff5640 1 -- 172.21.15.70:0/1054677615 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc28017070 con 0x564cb073d2c0 2024-04-01T01:38:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.556+0000 7fcc4006c740 1 -- 172.21.15.70:0/1054677615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc28013470 con 0x564cb073d2c0 2024-04-01T01:38:36.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.556+0000 7fcc1aff5640 1 -- 172.21.15.70:0/1054677615 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fcc2800f040 con 0x564cb073d2c0 2024-04-01T01:38:36.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.556+0000 7fcc1aff5640 1 -- 172.21.15.70:0/1054677615 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcc28017070 con 0x564cb073d2c0 2024-04-01T01:38:36.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.558+0000 7fcc4006c740 10 client.17055 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:36.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.558+0000 7fcc4006c740 1 -- 172.21.15.70:0/1054677615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564cb0761f30 con 0x564cb073d2c0 2024-04-01T01:38:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.560+0000 7fcc1aff5640 1 -- 172.21.15.70:0/1054677615 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc28013470 con 0x564cb073d2c0 2024-04-01T01:38:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.560+0000 7fcc1aff5640 1 -- 172.21.15.70:0/1054677615 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fcc28023070 con 0x564cb073d2c0 2024-04-01T01:38:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.560+0000 7fcc4006c740 10 client.17055 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.560+0000 7fcc4006c740 10 client.17055 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.560+0000 7fcc4006c740 20 client.17055 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.560+0000 7fcc4006c740 1 --2- 172.21.15.70:0/1054677615 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564cb0763360 0x564cb0765750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.560+0000 7fcc4006c740 4 client.17055 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.560+0000 7fcc4006c740 1 -- 172.21.15.70:0/1054677615 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cb06b4120 con 0x564cb0763360 2024-04-01T01:38:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.561+0000 7fcc36ffd640 1 --2- 172.21.15.70:0/1054677615 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564cb0763360 0x564cb0765750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.561+0000 7fcc1aff5640 10 client.17055 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.561+0000 7fcc36ffd640 1 --2- 172.21.15.70:0/1054677615 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564cb0763360 0x564cb0765750 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0f21e086b5e7f2f server_cookie=86354e811297097f in_seq=0 out_seq=0 2024-04-01T01:38:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.562+0000 7fcc1aff5640 1 -- 172.21.15.70:0/1054677615 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564cb06b4120 con 0x564cb0763360 2024-04-01T01:38:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.562+0000 7fcc1aff5640 10 client.17055 handle_command_reply: tid=0 2024-04-01T01:38:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.562+0000 7fcc4006c740 1 client.17055 shutdown 2024-04-01T01:38:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.563+0000 7fcc4006c740 1 -- 172.21.15.70:0/1054677615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073d2c0 msgr2=0x564cb0757fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.563+0000 7fcc4006c740 1 --2- 172.21.15.70:0/1054677615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073d2c0 0x564cb0757fa0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fcc28013650 tx=0x7fcc28013680 comp rx=0 tx=0).stop 2024-04-01T01:38:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.564+0000 7fcc4006c740 1 -- 172.21.15.70:0/1054677615 shutdown_connections 2024-04-01T01:38:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.564+0000 7fcc4006c740 1 --2- 172.21.15.70:0/1054677615 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564cb0763360 0x564cb0765750 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.564+0000 7fcc4006c740 1 --2- 172.21.15.70:0/1054677615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb0742430 0x564cb075efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.564+0000 7fcc4006c740 1 --2- 172.21.15.70:0/1054677615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073d2c0 0x564cb0757fa0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.564+0000 7fcc1aff5640 0 client.17055 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.564+0000 7fcc4006c740 1 --2- 172.21.15.70:0/1054677615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb073c8d0 0x564cb0757a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.564+0000 7fcc4006c740 1 -- 172.21.15.70:0/1054677615 >> 172.21.15.70:0/1054677615 conn(0x564cb0737e40 msgr2=0x564cb0736e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.564+0000 7fcc4006c740 1 -- 172.21.15.70:0/1054677615 shutdown_connections 2024-04-01T01:38:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.564+0000 7fcc4006c740 1 -- 172.21.15.70:0/1054677615 wait complete. 2024-04-01T01:38:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.564+0000 7fcc4006c740 20 client.17055 trim_cache size 0 max 16384 2024-04-01T01:38:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.592+0000 7fcc4006c740 1 Processor -- start 2024-04-01T01:38:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.592+0000 7fcc4006c740 1 -- start start 2024-04-01T01:38:36.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.592+0000 7fcc4006c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073c8d0 0x564cb0738450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.592+0000 7fcc4006c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb073d2c0 0x564cb075efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.593+0000 7fcc4006c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb0769fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.593+0000 7fcc4006c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcc381b0d30 con 0x564cb073c8d0 2024-04-01T01:38:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.593+0000 7fcc4006c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcc381b2870 con 0x564cb073d2c0 2024-04-01T01:38:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.593+0000 7fcc4006c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcc381b1400 con 0x564cb0742430 2024-04-01T01:38:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.593+0000 7fcc377fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb0769fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.593+0000 7fcc377fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb0769fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52850/0 (socket says 172.21.15.70:52850) 2024-04-01T01:38:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.593+0000 7fcc377fe640 1 -- 172.21.15.70:0/3779322922 learned_addr learned my addr 172.21.15.70:0/3779322922 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.593+0000 7fcc367fc640 1 --2- 172.21.15.70:0/3779322922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb073d2c0 0x564cb075efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.593+0000 7fcc36ffd640 1 --2- 172.21.15.70:0/3779322922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073c8d0 0x564cb0738450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.594+0000 7fcc377fe640 1 -- 172.21.15.70:0/3779322922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb073d2c0 msgr2=0x564cb075efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.594+0000 7fcc377fe640 1 --2- 172.21.15.70:0/3779322922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb073d2c0 0x564cb075efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.594+0000 7fcc377fe640 1 -- 172.21.15.70:0/3779322922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073c8d0 msgr2=0x564cb0738450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.594+0000 7fcc377fe640 1 --2- 172.21.15.70:0/3779322922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073c8d0 0x564cb0738450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.594+0000 7fcc377fe640 1 -- 172.21.15.70:0/3779322922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cb0765d80 con 0x564cb0742430 2024-04-01T01:38:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.594+0000 7fcc36ffd640 1 --2- 172.21.15.70:0/3779322922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073c8d0 0x564cb0738450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:38:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.594+0000 7fcc377fe640 1 --2- 172.21.15.70:0/3779322922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb0769fb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c009900 tx=0x7fcc2c03e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0f8a49c841f012b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.594+0000 7fcc1cff9640 1 -- 172.21.15.70:0/3779322922 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c050070 con 0x564cb0742430 2024-04-01T01:38:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.595+0000 7fcc1cff9640 1 -- 172.21.15.70:0/3779322922 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc2c0046c0 con 0x564cb0742430 2024-04-01T01:38:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.595+0000 7fcc1cff9640 1 -- 172.21.15.70:0/3779322922 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c003850 con 0x564cb0742430 2024-04-01T01:38:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.596+0000 7fcc4006c740 1 -- 172.21.15.70:0/3779322922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 msgr2=0x564cb0769fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.596+0000 7fcc4006c740 1 --2- 172.21.15.70:0/3779322922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb0769fb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c009900 tx=0x7fcc2c03e000 comp rx=0 tx=0).stop 2024-04-01T01:38:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.598+0000 7fcc4006c740 1 -- 172.21.15.70:0/3779322922 shutdown_connections 2024-04-01T01:38:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.598+0000 7fcc4006c740 1 --2- 172.21.15.70:0/3779322922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb0769fb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.598+0000 7fcc4006c740 1 --2- 172.21.15.70:0/3779322922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb073d2c0 0x564cb075efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.598+0000 7fcc4006c740 1 --2- 172.21.15.70:0/3779322922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073c8d0 0x564cb0738450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.598+0000 7fcc4006c740 1 -- 172.21.15.70:0/3779322922 >> 172.21.15.70:0/3779322922 conn(0x564cb0737e40 msgr2=0x564cb073e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.598+0000 7fcc4006c740 1 -- 172.21.15.70:0/3779322922 shutdown_connections 2024-04-01T01:38:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.598+0000 7fcc4006c740 1 -- 172.21.15.70:0/3779322922 wait complete. 2024-04-01T01:38:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.598+0000 7fcc4006c740 1 -- start start 2024-04-01T01:38:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.598+0000 7fcc4006c740 1 Processor -- start 2024-04-01T01:38:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.599+0000 7fcc4006c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073c8d0 0x564cb075c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.599+0000 7fcc4006c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb073d2c0 0x564cb075cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.599+0000 7fcc36ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073c8d0 0x564cb075c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.599+0000 7fcc36ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073c8d0 0x564cb075c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36540/0 (socket says 172.21.15.70:36540) 2024-04-01T01:38:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.599+0000 7fcc4006c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb075dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.599+0000 7fcc36ffd640 1 -- 172.21.15.70:0/2436241050 learned_addr learned my addr 172.21.15.70:0/2436241050 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.599+0000 7fcc4006c740 1 -- 172.21.15.70:0/2436241050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcc381b21a0 con 0x564cb073c8d0 2024-04-01T01:38:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.599+0000 7fcc377fe640 1 --2- 172.21.15.70:0/2436241050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb075dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.600+0000 7fcc4006c740 1 -- 172.21.15.70:0/2436241050 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcc381b1ad0 con 0x564cb073d2c0 2024-04-01T01:38:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.600+0000 7fcc4006c740 1 -- 172.21.15.70:0/2436241050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcc38114160 con 0x564cb0742430 2024-04-01T01:38:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.600+0000 7fcc367fc640 1 --2- 172.21.15.70:0/2436241050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb073d2c0 0x564cb075cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.600+0000 7fcc377fe640 1 -- 172.21.15.70:0/2436241050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb073d2c0 msgr2=0x564cb075cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.600+0000 7fcc377fe640 1 --2- 172.21.15.70:0/2436241050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb073d2c0 0x564cb075cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.600+0000 7fcc377fe640 1 -- 172.21.15.70:0/2436241050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073c8d0 msgr2=0x564cb075c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.600+0000 7fcc377fe640 1 --2- 172.21.15.70:0/2436241050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073c8d0 0x564cb075c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.600+0000 7fcc377fe640 1 -- 172.21.15.70:0/2436241050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cb075e390 con 0x564cb0742430 2024-04-01T01:38:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.600+0000 7fcc4006c740 10 client.0 _ll_register_callbacks cb 0x7fcc28009890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.600+0000 7fcc4006c740 1 -- 172.21.15.70:0/2436241050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc2c0046c0 con 0x564cb0742430 2024-04-01T01:38:36.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.601+0000 7fcc377fe640 1 --2- 172.21.15.70:0/2436241050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb075dda0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c02c240 tx=0x7fcc2c005370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e7bdeaad2c09169 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.601+0000 7fcc1aff5640 10 client.26848 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:38:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.601+0000 7fcc1aff5640 1 -- 172.21.15.70:0/2436241050 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc2c059070 con 0x564cb0742430 2024-04-01T01:38:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.601+0000 7fcc1aff5640 1 -- 172.21.15.70:0/2436241050 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fcc2c0037f0 con 0x564cb0742430 2024-04-01T01:38:36.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.601+0000 7fcc1aff5640 1 -- 172.21.15.70:0/2436241050 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcc2c050040 con 0x564cb0742430 2024-04-01T01:38:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.604+0000 7fcc4006c740 10 client.26848 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.605+0000 7fcc4006c740 1 -- 172.21.15.70:0/2436241050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=29}) -- 0x564cb07c9690 con 0x564cb0742430 2024-04-01T01:38:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.607+0000 7fcc1aff5640 1 -- 172.21.15.70:0/2436241050 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fcc2c0039f0 con 0x564cb0742430 2024-04-01T01:38:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.607+0000 7fcc4006c740 10 client.26848 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.607+0000 7fcc4006c740 10 client.26848 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.607+0000 7fcc4006c740 20 client.26848 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.607+0000 7fcc4006c740 1 --2- 172.21.15.70:0/2436241050 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564cb07c9e20 0x564cb07cc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.607+0000 7fcc4006c740 4 client.26848 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:38:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.608+0000 7fcc4006c740 1 -- 172.21.15.70:0/2436241050 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564cb06ade30 con 0x564cb07c9e20 2024-04-01T01:38:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.611+0000 7fcc36ffd640 1 --2- 172.21.15.70:0/2436241050 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564cb07c9e20 0x564cb07cc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.612+0000 7fcc36ffd640 1 --2- 172.21.15.70:0/2436241050 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564cb07c9e20 0x564cb07cc210 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dd0508bd770a55b server_cookie=ca10a49f6d10945d in_seq=0 out_seq=0 2024-04-01T01:38:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.612+0000 7fcc1aff5640 10 client.26848 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.637+0000 7fcc1aff5640 1 -- 172.21.15.70:0/2436241050 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564cb06ade30 con 0x564cb07c9e20 2024-04-01T01:38:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.637+0000 7fcc1aff5640 10 client.26848 handle_command_reply: tid=0 2024-04-01T01:38:36.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.638+0000 7fcc4006c740 1 client.26848 shutdown 2024-04-01T01:38:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.638+0000 7fcc4006c740 1 -- 172.21.15.70:0/2436241050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 msgr2=0x564cb075dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.638+0000 7fcc4006c740 1 --2- 172.21.15.70:0/2436241050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb075dda0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c02c240 tx=0x7fcc2c005370 comp rx=0 tx=0).stop 2024-04-01T01:38:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.639+0000 7fcc4006c740 1 -- 172.21.15.70:0/2436241050 shutdown_connections 2024-04-01T01:38:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.639+0000 7fcc4006c740 1 --2- 172.21.15.70:0/2436241050 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564cb07c9e20 0x564cb07cc210 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.639+0000 7fcc4006c740 1 --2- 172.21.15.70:0/2436241050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cb0742430 0x564cb075dda0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.639+0000 7fcc4006c740 1 --2- 172.21.15.70:0/2436241050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564cb073d2c0 0x564cb075cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.639+0000 7fcc4006c740 1 --2- 172.21.15.70:0/2436241050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564cb073c8d0 0x564cb075c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.639+0000 7fcc1aff5640 0 client.26848 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.639+0000 7fcc4006c740 1 -- 172.21.15.70:0/2436241050 >> 172.21.15.70:0/2436241050 conn(0x564cb0737e40 msgr2=0x564cb073b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.639+0000 7fcc4006c740 1 -- 172.21.15.70:0/2436241050 shutdown_connections 2024-04-01T01:38:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.639+0000 7fcc4006c740 1 -- 172.21.15.70:0/2436241050 wait complete. 2024-04-01T01:38:36.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.640+0000 7fcc4006c740 20 client.26848 trim_cache size 0 max 16384 2024-04-01T01:38:36.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:38:36.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:38:36.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:38:36.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:38:36.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.645+0000 7fcc3d48f640 1 -- 172.21.15.70:0/921965897 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcc200a29c0 msgr2=0x7fcc200a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:36.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.646+0000 7fcc3d48f640 1 --2- 172.21.15.70:0/921965897 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcc200a29c0 0x7fcc200a4e80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c006010 tx=0x7fcc2c02c5d0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.646+0000 7fcc3d48f640 1 -- 172.21.15.70:0/921965897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc3806d7f0 msgr2=0x7fcc381a6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:36.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.646+0000 7fcc3d48f640 1 --2- 172.21.15.70:0/921965897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc3806d7f0 0x7fcc381a6750 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000ebc0 tx=0x7fcc3000c2d0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.647+0000 7fcc3d48f640 1 -- 172.21.15.70:0/921965897 shutdown_connections 2024-04-01T01:38:36.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.647+0000 7fcc3d48f640 1 --2- 172.21.15.70:0/921965897 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcc200a29c0 0x7fcc200a4e80 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.647+0000 7fcc3d48f640 1 --2- 172.21.15.70:0/921965897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc3806f3d0 0x7fcc381adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.647+0000 7fcc3d48f640 1 --2- 172.21.15.70:0/921965897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc3806ea30 0x7fcc381a6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.647+0000 7fcc3d48f640 1 --2- 172.21.15.70:0/921965897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc3806d7f0 0x7fcc381a6750 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:36.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.647+0000 7fcc3d48f640 1 -- 172.21.15.70:0/921965897 >> 172.21.15.70:0/921965897 conn(0x7fcc3806c330 msgr2=0x7fcc3810a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:36.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.648+0000 7fcc3d48f640 1 -- 172.21.15.70:0/921965897 shutdown_connections 2024-04-01T01:38:36.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:36.648+0000 7fcc3d48f640 1 -- 172.21.15.70:0/921965897 wait complete. 2024-04-01T01:38:36.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 230.0 s, 1116.8 tps, lat 89.361 ms stddev 68.086 2024-04-01T01:38:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:37 smithi081 ceph-mon[35142]: pgmap v512: 129 pgs: 129 active+clean; 1.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 11 MiB/s wr, 1.14k op/s 2024-04-01T01:38:38.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:37 smithi189 ceph-mon[35753]: pgmap v512: 129 pgs: 129 active+clean; 1.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 11 MiB/s wr, 1.14k op/s 2024-04-01T01:38:38.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:37 smithi070 ceph-mon[32179]: pgmap v512: 129 pgs: 129 active+clean; 1.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 11 MiB/s wr, 1.14k op/s 2024-04-01T01:38:40.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:39 smithi081 ceph-mon[35142]: pgmap v513: 129 pgs: 129 active+clean; 1.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 1.09k op/s 2024-04-01T01:38:40.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:39 smithi189 ceph-mon[35753]: pgmap v513: 129 pgs: 129 active+clean; 1.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 1.09k op/s 2024-04-01T01:38:40.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:39 smithi070 ceph-mon[32179]: pgmap v513: 129 pgs: 129 active+clean; 1.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 11 MiB/s wr, 1.09k op/s 2024-04-01T01:38:41.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:40 smithi189 ceph-mon[35753]: pgmap v514: 129 pgs: 129 active+clean; 1.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 9.6 MiB/s wr, 972 op/s 2024-04-01T01:38:41.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:40 smithi070 ceph-mon[32179]: pgmap v514: 129 pgs: 129 active+clean; 1.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 9.6 MiB/s wr, 972 op/s 2024-04-01T01:38:41.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:40 smithi081 ceph-mon[35142]: pgmap v514: 129 pgs: 129 active+clean; 1.0 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 9.6 MiB/s wr, 972 op/s 2024-04-01T01:38:41.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 235.0 s, 1129.7 tps, lat 88.570 ms stddev 36.916 2024-04-01T01:38:44.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:43 smithi081 ceph-mon[35142]: pgmap v515: 129 pgs: 129 active+clean; 1.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 1.12k op/s 2024-04-01T01:38:44.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:43 smithi189 ceph-mon[35753]: pgmap v515: 129 pgs: 129 active+clean; 1.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 1.12k op/s 2024-04-01T01:38:44.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:43 smithi070 ceph-mon[32179]: pgmap v515: 129 pgs: 129 active+clean; 1.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 1.12k op/s 2024-04-01T01:38:44.417 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:38:44.417 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:38:44.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.609+0000 7fe646610640 1 -- 172.21.15.70:0/264145860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6401020e0 msgr2=0x7fe640108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:44.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.609+0000 7fe646610640 1 --2- 172.21.15.70:0/264145860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6401020e0 0x7fe640108970 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe630009ca0 tx=0x7fe63002e310 comp rx=0 tx=0).stop 2024-04-01T01:38:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.611+0000 7fe646610640 1 -- 172.21.15.70:0/264145860 shutdown_connections 2024-04-01T01:38:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.611+0000 7fe646610640 1 --2- 172.21.15.70:0/264145860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6401020e0 0x7fe640108970 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.611+0000 7fe646610640 1 --2- 172.21.15.70:0/264145860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe640101720 0x7fe640101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.611+0000 7fe646610640 1 --2- 172.21.15.70:0/264145860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe640100530 0x7fe640100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.611+0000 7fe646610640 1 -- 172.21.15.70:0/264145860 >> 172.21.15.70:0/264145860 conn(0x7fe6400fbac0 msgr2=0x7fe6400fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.611+0000 7fe646610640 1 -- 172.21.15.70:0/264145860 shutdown_connections 2024-04-01T01:38:44.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.611+0000 7fe646610640 1 -- 172.21.15.70:0/264145860 wait complete. 2024-04-01T01:38:44.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.612+0000 7fe646610640 1 Processor -- start 2024-04-01T01:38:44.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.612+0000 7fe646610640 1 -- start start 2024-04-01T01:38:44.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.613+0000 7fe646610640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe640100530 0x7fe64019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.613+0000 7fe646610640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe640101720 0x7fe64019e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.613+0000 7fe646610640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6401020e0 0x7fe6401a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.613+0000 7fe646610640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe64010a730 con 0x7fe640101720 2024-04-01T01:38:44.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.613+0000 7fe646610640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe64010a5b0 con 0x7fe640100530 2024-04-01T01:38:44.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.613+0000 7fe64560e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe640100530 0x7fe64019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.613+0000 7fe644e0d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe640101720 0x7fe64019e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.613+0000 7fe64560e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe640100530 0x7fe64019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48680/0 (socket says 172.21.15.70:48680) 2024-04-01T01:38:44.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.613+0000 7fe64560e640 1 -- 172.21.15.70:0/778563514 learned_addr learned my addr 172.21.15.70:0/778563514 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:44.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.613+0000 7fe646610640 1 -- 172.21.15.70:0/778563514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe64010a8b0 con 0x7fe6401020e0 2024-04-01T01:38:44.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.614+0000 7fe645e0f640 1 --2- 172.21.15.70:0/778563514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6401020e0 0x7fe6401a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.614+0000 7fe645e0f640 1 -- 172.21.15.70:0/778563514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe640100530 msgr2=0x7fe64019de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:44.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.614+0000 7fe645e0f640 1 --2- 172.21.15.70:0/778563514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe640100530 0x7fe64019de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.614+0000 7fe645e0f640 1 -- 172.21.15.70:0/778563514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe640101720 msgr2=0x7fe64019e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:44.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.614+0000 7fe645e0f640 1 --2- 172.21.15.70:0/778563514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe640101720 0x7fe64019e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.614+0000 7fe645e0f640 1 -- 172.21.15.70:0/778563514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6401a5a40 con 0x7fe6401020e0 2024-04-01T01:38:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.615+0000 7fe645e0f640 1 --2- 172.21.15.70:0/778563514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6401020e0 0x7fe6401a53d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe630039040 tx=0x7fe630037cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.615+0000 7fe63e7fc640 1 -- 172.21.15.70:0/778563514 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe630046070 con 0x7fe6401020e0 2024-04-01T01:38:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.615+0000 7fe646610640 1 -- 172.21.15.70:0/778563514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6401a5cd0 con 0x7fe6401020e0 2024-04-01T01:38:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.615+0000 7fe63e7fc640 1 -- 172.21.15.70:0/778563514 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6300368f0 con 0x7fe6401020e0 2024-04-01T01:38:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.615+0000 7fe63e7fc640 1 -- 172.21.15.70:0/778563514 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63003b070 con 0x7fe6401020e0 2024-04-01T01:38:44.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.615+0000 7fe646610640 1 -- 172.21.15.70:0/778563514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6401a6190 con 0x7fe6401020e0 2024-04-01T01:38:44.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.619+0000 7fe64a44f740 1 Processor -- start 2024-04-01T01:38:44.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.620+0000 7fe64a44f740 1 -- start start 2024-04-01T01:38:44.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.620+0000 7fe64a44f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665ffec490 0x55665ffec8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.620+0000 7fe64a44f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffece80 0x55665fff1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.620+0000 7fe64a44f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665fff1ff0 0x55665fff4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.620+0000 7fe64a44f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55665ff51180 con 0x55665fff1ff0 2024-04-01T01:38:44.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.620+0000 7fe64a44f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55665fef9b70 con 0x55665ffece80 2024-04-01T01:38:44.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.620+0000 7fe64a44f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55665ff4d220 con 0x55665ffec490 2024-04-01T01:38:44.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.620+0000 7fe644e0d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665fff1ff0 0x55665fff4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.620+0000 7fe64560e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffece80 0x55665fff1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.620+0000 7fe644e0d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665fff1ff0 0x55665fff4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44862/0 (socket says 172.21.15.70:44862) 2024-04-01T01:38:44.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.620+0000 7fe644e0d640 1 -- 172.21.15.70:0/2747579343 learned_addr learned my addr 172.21.15.70:0/2747579343 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:44.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.620+0000 7fe63e7fc640 1 -- 172.21.15.70:0/778563514 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fe630049070 con 0x7fe6401020e0 2024-04-01T01:38:44.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.620+0000 7fe644e0d640 1 --2- 172.21.15.70:0/2747579343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665ffec490 0x55665ffec8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.621+0000 7fe64560e640 1 -- 172.21.15.70:0/2747579343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665ffec490 msgr2=0x55665ffec8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:44.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.621+0000 7fe64560e640 1 --2- 172.21.15.70:0/2747579343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665ffec490 0x55665ffec8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.621+0000 7fe64560e640 1 -- 172.21.15.70:0/2747579343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665fff1ff0 msgr2=0x55665fff4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:44.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.621+0000 7fe64560e640 1 --2- 172.21.15.70:0/2747579343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665fff1ff0 0x55665fff4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.621+0000 7fe64560e640 1 -- 172.21.15.70:0/2747579343 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55665ff0d7c0 con 0x55665ffece80 2024-04-01T01:38:44.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.621+0000 7fe64560e640 1 --2- 172.21.15.70:0/2747579343 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffece80 0x55665fff1980 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe634009c00 tx=0x7fe6340117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc8e58dad8a08ee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:44.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.621+0000 7fe63e7fc640 1 --2- 172.21.15.70:0/778563514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6280a25f0 0x7fe6280a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.622+0000 7fe6277fe640 1 -- 172.21.15.70:0/2747579343 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe634005020 con 0x55665ffece80 2024-04-01T01:38:44.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.622+0000 7fe6277fe640 1 -- 172.21.15.70:0/2747579343 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6340047a0 con 0x55665ffece80 2024-04-01T01:38:44.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.622+0000 7fe644e0d640 1 --2- 172.21.15.70:0/778563514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6280a25f0 0x7fe6280a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.622+0000 7fe63e7fc640 1 -- 172.21.15.70:0/778563514 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe6300f0e40 con 0x7fe6401020e0 2024-04-01T01:38:44.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.622+0000 7fe6277fe640 1 -- 172.21.15.70:0/2747579343 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe634012690 con 0x55665ffece80 2024-04-01T01:38:44.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.623+0000 7fe64a44f740 1 -- 172.21.15.70:0/2747579343 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffece80 msgr2=0x55665fff1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:44.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.623+0000 7fe644e0d640 1 --2- 172.21.15.70:0/778563514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6280a25f0 0x7fe6280a4ab0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x55665ffed6e0 tx=0x7fe638014040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:44.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.623+0000 7fe64a44f740 1 --2- 172.21.15.70:0/2747579343 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffece80 0x55665fff1980 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe634009c00 tx=0x7fe6340117b0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.623+0000 7fe64a44f740 1 -- 172.21.15.70:0/2747579343 shutdown_connections 2024-04-01T01:38:44.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.623+0000 7fe64a44f740 1 --2- 172.21.15.70:0/2747579343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665fff1ff0 0x55665fff4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.623+0000 7fe64a44f740 1 --2- 172.21.15.70:0/2747579343 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffece80 0x55665fff1980 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.623+0000 7fe64a44f740 1 --2- 172.21.15.70:0/2747579343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665ffec490 0x55665ffec8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.623+0000 7fe64a44f740 1 -- 172.21.15.70:0/2747579343 >> 172.21.15.70:0/2747579343 conn(0x55665ffe7a00 msgr2=0x55665ffe7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:44.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.623+0000 7fe64a44f740 1 -- 172.21.15.70:0/2747579343 shutdown_connections 2024-04-01T01:38:44.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.623+0000 7fe64a44f740 1 -- 172.21.15.70:0/2747579343 wait complete. 2024-04-01T01:38:44.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.624+0000 7fe64a44f740 1 -- start start 2024-04-01T01:38:44.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.624+0000 7fe64a44f740 1 Processor -- start 2024-04-01T01:38:44.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.624+0000 7fe64a44f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665ffec490 0x55665fff6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.625+0000 7fe64a44f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665ffece80 0x55665fff6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.625+0000 7fe64560e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665ffec490 0x55665fff6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.625+0000 7fe64560e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665ffec490 0x55665fff6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:54606/0 (socket says 172.21.15.70:54606) 2024-04-01T01:38:44.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.625+0000 7fe64560e640 1 -- 172.21.15.70:0/933501400 learned_addr learned my addr 172.21.15.70:0/933501400 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:44.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.625+0000 7fe64a44f740 1 --2- 172.21.15.70:0/933501400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665fff1ff0 0x55665fffdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.625+0000 7fe644e0d640 1 --2- 172.21.15.70:0/933501400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665ffece80 0x55665fff6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.625+0000 7fe64a44f740 1 -- 172.21.15.70:0/933501400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55665fff4ff0 con 0x55665ffece80 2024-04-01T01:38:44.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.626+0000 7fe64a44f740 1 -- 172.21.15.70:0/933501400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55665feef4a0 con 0x55665fff1ff0 2024-04-01T01:38:44.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.626+0000 7fe645e0f640 1 --2- 172.21.15.70:0/933501400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665fff1ff0 0x55665fffdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.626+0000 7fe64a44f740 1 -- 172.21.15.70:0/933501400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55665fff5170 con 0x55665ffec490 2024-04-01T01:38:44.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.626+0000 7fe64a44f740 10 client.0 _ll_register_callbacks cb 0x55665fef2d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:44.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.626+0000 7fe645e0f640 1 -- 172.21.15.70:0/933501400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665ffec490 msgr2=0x55665fff6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:44.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.626+0000 7fe645e0f640 1 --2- 172.21.15.70:0/933501400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665ffec490 0x55665fff6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.626+0000 7fe645e0f640 1 -- 172.21.15.70:0/933501400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665ffece80 msgr2=0x55665fff6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:44.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.626+0000 7fe645e0f640 1 --2- 172.21.15.70:0/933501400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665ffece80 0x55665fff6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.626+0000 7fe645e0f640 1 -- 172.21.15.70:0/933501400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55665fd2f7d0 con 0x55665fff1ff0 2024-04-01T01:38:44.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.626+0000 7fe644e0d640 1 --2- 172.21.15.70:0/933501400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665ffece80 0x55665fff6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:38:44.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.627+0000 7fe645e0f640 1 --2- 172.21.15.70:0/933501400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665fff1ff0 0x55665fffdfb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe630034730 tx=0x7fe6300bd0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=500fd0473c2b796 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:44.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.627+0000 7fe624ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:44.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.627+0000 7fe624ff9640 1 -- 172.21.15.70:0/933501400 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6300bdbd0 con 0x55665fff1ff0 2024-04-01T01:38:44.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.627+0000 7fe624ff9640 1 -- 172.21.15.70:0/933501400 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe6300dd0e0 con 0x55665fff1ff0 2024-04-01T01:38:44.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.627+0000 7fe64a44f740 1 -- 172.21.15.70:0/933501400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6340047a0 con 0x55665fff1ff0 2024-04-01T01:38:44.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.627+0000 7fe624ff9640 1 -- 172.21.15.70:0/933501400 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe6300bd4d0 con 0x55665fff1ff0 2024-04-01T01:38:44.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.627+0000 7fe64a44f740 10 client.26966 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:44.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.627+0000 7fe64a44f740 1 -- 172.21.15.70:0/933501400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55665fc653f0 con 0x55665fff1ff0 2024-04-01T01:38:44.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.627+0000 7fe624ff9640 1 -- 172.21.15.70:0/933501400 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6300c1620 con 0x55665fff1ff0 2024-04-01T01:38:44.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.628+0000 7fe624ff9640 1 -- 172.21.15.70:0/933501400 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe6300c5a50 con 0x55665fff1ff0 2024-04-01T01:38:44.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.628+0000 7fe64a44f740 10 client.26966 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:44.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.628+0000 7fe64a44f740 10 client.26966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:44.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.628+0000 7fe64a44f740 20 client.26966 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:44.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.628+0000 7fe64a44f740 1 --2- 172.21.15.70:0/933501400 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556660002a10 0x556660004e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.628+0000 7fe64a44f740 4 client.26966 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:44.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.628+0000 7fe64a44f740 1 -- 172.21.15.70:0/933501400 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe6340047a0 con 0x556660002a10 2024-04-01T01:38:44.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.628+0000 7fe64560e640 1 --2- 172.21.15.70:0/933501400 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556660002a10 0x556660004e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.629+0000 7fe624ff9640 10 client.26966 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:44.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.629+0000 7fe64560e640 1 --2- 172.21.15.70:0/933501400 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556660002a10 0x556660004e30 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9669098f3a6d22e server_cookie=310005942cdd0e57 in_seq=0 out_seq=0 2024-04-01T01:38:44.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.630+0000 7fe624ff9640 1 -- 172.21.15.70:0/933501400 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fe6340047a0 con 0x556660002a10 2024-04-01T01:38:44.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.630+0000 7fe624ff9640 10 client.26966 handle_command_reply: tid=0 2024-04-01T01:38:44.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.630+0000 7fe64a44f740 1 client.26966 shutdown 2024-04-01T01:38:44.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.631+0000 7fe64a44f740 1 -- 172.21.15.70:0/933501400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665fff1ff0 msgr2=0x55665fffdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:44.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.631+0000 7fe64a44f740 1 --2- 172.21.15.70:0/933501400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665fff1ff0 0x55665fffdfb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe630034730 tx=0x7fe6300bd0d0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.631+0000 7fe64a44f740 1 -- 172.21.15.70:0/933501400 shutdown_connections 2024-04-01T01:38:44.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.631+0000 7fe64a44f740 1 --2- 172.21.15.70:0/933501400 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556660002a10 0x556660004e30 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.631+0000 7fe64a44f740 1 --2- 172.21.15.70:0/933501400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665fff1ff0 0x55665fffdfb0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.631+0000 7fe624ff9640 0 client.26966 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:44.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.631+0000 7fe64a44f740 1 --2- 172.21.15.70:0/933501400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665ffece80 0x55665fff6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.631+0000 7fe64a44f740 1 --2- 172.21.15.70:0/933501400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665ffec490 0x55665fff6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.631+0000 7fe64a44f740 1 -- 172.21.15.70:0/933501400 >> 172.21.15.70:0/933501400 conn(0x55665ffe7a00 msgr2=0x55665ffe6c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:44.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.632+0000 7fe64a44f740 1 -- 172.21.15.70:0/933501400 shutdown_connections 2024-04-01T01:38:44.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.632+0000 7fe64a44f740 1 -- 172.21.15.70:0/933501400 wait complete. 2024-04-01T01:38:44.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.632+0000 7fe64a44f740 20 client.26966 trim_cache size 0 max 16384 2024-04-01T01:38:44.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.664+0000 7fe64a44f740 1 Processor -- start 2024-04-01T01:38:44.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.664+0000 7fe64a44f740 1 -- start start 2024-04-01T01:38:44.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.664+0000 7fe64a44f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffec490 0x55665fff6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.665+0000 7fe64a44f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665ffece80 0x55665fff6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.665+0000 7fe64a44f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665fff1ff0 0x55665ffe6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.665+0000 7fe64a44f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe6401b06f0 con 0x55665ffece80 2024-04-01T01:38:44.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.665+0000 7fe64a44f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe6401b2230 con 0x55665ffec490 2024-04-01T01:38:44.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.665+0000 7fe64a44f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe6401b0dc0 con 0x55665fff1ff0 2024-04-01T01:38:44.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.665+0000 7fe645e0f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665fff1ff0 0x55665ffe6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.665+0000 7fe645e0f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665fff1ff0 0x55665ffe6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:54614/0 (socket says 172.21.15.70:54614) 2024-04-01T01:38:44.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.665+0000 7fe645e0f640 1 -- 172.21.15.70:0/1680265849 learned_addr learned my addr 172.21.15.70:0/1680265849 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:44.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.665+0000 7fe645e0f640 1 -- 172.21.15.70:0/1680265849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffec490 msgr2=0x55665fff6a30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:38:44.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.665+0000 7fe645e0f640 1 --2- 172.21.15.70:0/1680265849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffec490 0x55665fff6a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.665+0000 7fe645e0f640 1 -- 172.21.15.70:0/1680265849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665ffece80 msgr2=0x55665fff6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:44.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.665+0000 7fe645e0f640 1 --2- 172.21.15.70:0/1680265849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665ffece80 0x55665fff6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.666+0000 7fe645e0f640 1 -- 172.21.15.70:0/1680265849 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55665fc250f0 con 0x55665fff1ff0 2024-04-01T01:38:44.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.666+0000 7fe645e0f640 1 --2- 172.21.15.70:0/1680265849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665fff1ff0 0x55665ffe6390 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe6300b9930 tx=0x7fe6300a8050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b7269d8bb3d6401 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:44.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.666+0000 7fe626ffd640 1 -- 172.21.15.70:0/1680265849 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6300c1620 con 0x55665fff1ff0 2024-04-01T01:38:44.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.670+0000 7fe626ffd640 1 -- 172.21.15.70:0/1680265849 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6300c5ee0 con 0x55665fff1ff0 2024-04-01T01:38:44.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.670+0000 7fe626ffd640 1 -- 172.21.15.70:0/1680265849 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6300dd0d0 con 0x55665fff1ff0 2024-04-01T01:38:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.671+0000 7fe64a44f740 1 -- 172.21.15.70:0/1680265849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665fff1ff0 msgr2=0x55665ffe6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.671+0000 7fe64a44f740 1 --2- 172.21.15.70:0/1680265849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665fff1ff0 0x55665ffe6390 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe6300b9930 tx=0x7fe6300a8050 comp rx=0 tx=0).stop 2024-04-01T01:38:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.672+0000 7fe64a44f740 1 -- 172.21.15.70:0/1680265849 shutdown_connections 2024-04-01T01:38:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.672+0000 7fe64a44f740 1 --2- 172.21.15.70:0/1680265849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665fff1ff0 0x55665ffe6390 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.672+0000 7fe64a44f740 1 --2- 172.21.15.70:0/1680265849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665ffece80 0x55665fff6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.672+0000 7fe64a44f740 1 --2- 172.21.15.70:0/1680265849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffec490 0x55665fff6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.672+0000 7fe64a44f740 1 -- 172.21.15.70:0/1680265849 >> 172.21.15.70:0/1680265849 conn(0x55665ffe7a00 msgr2=0x55665ffe7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:44.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.672+0000 7fe64a44f740 1 -- 172.21.15.70:0/1680265849 shutdown_connections 2024-04-01T01:38:44.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.675+0000 7fe64a44f740 1 -- 172.21.15.70:0/1680265849 wait complete. 2024-04-01T01:38:44.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.675+0000 7fe64a44f740 1 -- start start 2024-04-01T01:38:44.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.675+0000 7fe64a44f740 1 Processor -- start 2024-04-01T01:38:44.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.676+0000 7fe64a44f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffec490 0x55666000a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.676+0000 7fe64a44f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665ffece80 0x55666000a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.676+0000 7fe64a44f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665fff1ff0 0x556660006d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.676+0000 7fe64a44f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe6401b1b60 con 0x55665fff1ff0 2024-04-01T01:38:44.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.676+0000 7fe64a44f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe6401b1490 con 0x55665ffec490 2024-04-01T01:38:44.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.676+0000 7fe64a44f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe6401ae6d0 con 0x55665ffece80 2024-04-01T01:38:44.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.676+0000 7fe64a44f740 10 client.0 _ll_register_callbacks cb 0x7fe6300b97e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:44.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.676+0000 7fe645e0f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665fff1ff0 0x556660006d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.676+0000 7fe64560e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffec490 0x55666000a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.676+0000 7fe64560e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffec490 0x55666000a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48726/0 (socket says 172.21.15.70:48726) 2024-04-01T01:38:44.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.676+0000 7fe64560e640 1 -- 172.21.15.70:0/3641585534 learned_addr learned my addr 172.21.15.70:0/3641585534 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:44.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.677+0000 7fe644e0d640 1 --2- 172.21.15.70:0/3641585534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665ffece80 0x55666000a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.677+0000 7fe64560e640 1 -- 172.21.15.70:0/3641585534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665ffece80 msgr2=0x55666000a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:44.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.677+0000 7fe64560e640 1 --2- 172.21.15.70:0/3641585534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665ffece80 0x55666000a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.677+0000 7fe64560e640 1 -- 172.21.15.70:0/3641585534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665fff1ff0 msgr2=0x556660006d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:44.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.677+0000 7fe64560e640 1 --2- 172.21.15.70:0/3641585534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665fff1ff0 0x556660006d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.677+0000 7fe64560e640 1 -- 172.21.15.70:0/3641585534 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556660007380 con 0x55665ffec490 2024-04-01T01:38:44.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.679+0000 7fe64560e640 1 --2- 172.21.15.70:0/3641585534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffec490 0x55666000a2a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe634009b50 tx=0x7fe6340057d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c065ec23c570540 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:44.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.679+0000 7fe624ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:44.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.680+0000 7fe624ff9640 1 -- 172.21.15.70:0/3641585534 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe634032060 con 0x55665ffec490 2024-04-01T01:38:44.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.680+0000 7fe624ff9640 1 -- 172.21.15.70:0/3641585534 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe634005230 con 0x55665ffec490 2024-04-01T01:38:44.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.680+0000 7fe64a44f740 1 -- 172.21.15.70:0/3641585534 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6300c5ee0 con 0x55665ffec490 2024-04-01T01:38:44.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.681+0000 7fe624ff9640 1 -- 172.21.15.70:0/3641585534 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe63401f040 con 0x55665ffec490 2024-04-01T01:38:44.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.682+0000 7fe64a44f740 10 client.26969 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:44.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.682+0000 7fe64a44f740 1 -- 172.21.15.70:0/3641585534 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55666007a8d0 con 0x55665ffec490 2024-04-01T01:38:44.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.683+0000 7fe624ff9640 1 -- 172.21.15.70:0/3641585534 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe6300c5ee0 con 0x55665ffec490 2024-04-01T01:38:44.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.684+0000 7fe624ff9640 1 -- 172.21.15.70:0/3641585534 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe634012470 con 0x55665ffec490 2024-04-01T01:38:44.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.685+0000 7fe64a44f740 10 client.26969 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:44.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.685+0000 7fe64a44f740 10 client.26969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:44.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.685+0000 7fe64a44f740 20 client.26969 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:44.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.685+0000 7fe64a44f740 1 --2- 172.21.15.70:0/3641585534 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55666007af50 0x55666007d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.685+0000 7fe64a44f740 4 client.26969 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:38:44.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.685+0000 7fe64a44f740 1 -- 172.21.15.70:0/3641585534 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55665ff5e6c0 con 0x55666007af50 2024-04-01T01:38:44.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.686+0000 7fe644e0d640 1 --2- 172.21.15.70:0/3641585534 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55666007af50 0x55666007d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.687+0000 7fe644e0d640 1 --2- 172.21.15.70:0/3641585534 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55666007af50 0x55666007d340 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee61f91f0ad0a69e server_cookie=8f4e2bab6cf3c09d in_seq=0 out_seq=0 2024-04-01T01:38:44.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.687+0000 7fe624ff9640 10 client.26969 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:44.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.687+0000 7fe624ff9640 1 -- 172.21.15.70:0/3641585534 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55665ff5e6c0 con 0x55666007af50 2024-04-01T01:38:44.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.687+0000 7fe624ff9640 10 client.26969 handle_command_reply: tid=0 2024-04-01T01:38:44.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.687+0000 7fe64a44f740 1 client.26969 shutdown 2024-04-01T01:38:44.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.688+0000 7fe64a44f740 1 -- 172.21.15.70:0/3641585534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffec490 msgr2=0x55666000a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:44.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.688+0000 7fe64a44f740 1 --2- 172.21.15.70:0/3641585534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffec490 0x55666000a2a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe634009b50 tx=0x7fe6340057d0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.689+0000 7fe64a44f740 1 -- 172.21.15.70:0/3641585534 shutdown_connections 2024-04-01T01:38:44.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.689+0000 7fe64a44f740 1 --2- 172.21.15.70:0/3641585534 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55666007af50 0x55666007d340 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.689+0000 7fe64a44f740 1 --2- 172.21.15.70:0/3641585534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55665fff1ff0 0x556660006d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.689+0000 7fe64a44f740 1 --2- 172.21.15.70:0/3641585534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55665ffece80 0x55666000a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.689+0000 7fe64a44f740 1 --2- 172.21.15.70:0/3641585534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55665ffec490 0x55666000a2a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.689+0000 7fe64a44f740 1 -- 172.21.15.70:0/3641585534 >> 172.21.15.70:0/3641585534 conn(0x55665ffe7a00 msgr2=0x55665fff6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:44.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.689+0000 7fe624ff9640 0 client.26969 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:44.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.689+0000 7fe64a44f740 1 -- 172.21.15.70:0/3641585534 shutdown_connections 2024-04-01T01:38:44.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.689+0000 7fe64a44f740 1 -- 172.21.15.70:0/3641585534 wait complete. 2024-04-01T01:38:44.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.689+0000 7fe64a44f740 20 client.26969 trim_cache size 0 max 16384 2024-04-01T01:38:44.690 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:38:44.690 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:38:44.690 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:38:44.690 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:38:44.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.692+0000 7fe646610640 1 -- 172.21.15.70:0/778563514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6280a25f0 msgr2=0x7fe6280a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:44.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.692+0000 7fe646610640 1 --2- 172.21.15.70:0/778563514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6280a25f0 0x7fe6280a4ab0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x55665ffed6e0 tx=0x7fe638014040 comp rx=0 tx=0).stop 2024-04-01T01:38:44.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.692+0000 7fe646610640 1 -- 172.21.15.70:0/778563514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6401020e0 msgr2=0x7fe6401a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:44.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.692+0000 7fe646610640 1 --2- 172.21.15.70:0/778563514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6401020e0 0x7fe6401a53d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe630039040 tx=0x7fe630037cb0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.693+0000 7fe646610640 1 -- 172.21.15.70:0/778563514 shutdown_connections 2024-04-01T01:38:44.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.693+0000 7fe646610640 1 --2- 172.21.15.70:0/778563514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6280a25f0 0x7fe6280a4ab0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.693+0000 7fe646610640 1 --2- 172.21.15.70:0/778563514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6401020e0 0x7fe6401a53d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.693+0000 7fe646610640 1 --2- 172.21.15.70:0/778563514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe640101720 0x7fe64019e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.693+0000 7fe646610640 1 --2- 172.21.15.70:0/778563514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe640100530 0x7fe64019de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.693+0000 7fe646610640 1 -- 172.21.15.70:0/778563514 >> 172.21.15.70:0/778563514 conn(0x7fe6400fbac0 msgr2=0x7fe64010d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:44.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.694+0000 7fe646610640 1 -- 172.21.15.70:0/778563514 shutdown_connections 2024-04-01T01:38:44.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.694+0000 7fe646610640 1 -- 172.21.15.70:0/778563514 wait complete. 2024-04-01T01:38:44.716 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:38:44.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:38:44.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.897+0000 7f33d2e14640 1 -- 172.21.15.70:0/482954526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33cc107d90 msgr2=0x7f33cc1081b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:44.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.897+0000 7f33d2e14640 1 --2- 172.21.15.70:0/482954526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33cc107d90 0x7f33cc1081b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f33bc009900 tx=0x7f33bc02bc40 comp rx=0 tx=0).stop 2024-04-01T01:38:44.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.898+0000 7f33d2e14640 1 -- 172.21.15.70:0/482954526 shutdown_connections 2024-04-01T01:38:44.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.898+0000 7f33d2e14640 1 --2- 172.21.15.70:0/482954526 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc109940 0x7f33cc1101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.898+0000 7f33d2e14640 1 --2- 172.21.15.70:0/482954526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33cc108f80 0x7f33cc109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.898+0000 7f33d2e14640 1 --2- 172.21.15.70:0/482954526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33cc107d90 0x7f33cc1081b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.898+0000 7f33d2e14640 1 -- 172.21.15.70:0/482954526 >> 172.21.15.70:0/482954526 conn(0x7f33cc076d60 msgr2=0x7f33cc079180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.898+0000 7f33d2e14640 1 -- 172.21.15.70:0/482954526 shutdown_connections 2024-04-01T01:38:44.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.898+0000 7f33d2e14640 1 -- 172.21.15.70:0/482954526 wait complete. 2024-04-01T01:38:44.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.899+0000 7f33d2e14640 1 Processor -- start 2024-04-01T01:38:44.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.899+0000 7f33d2e14640 1 -- start start 2024-04-01T01:38:44.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.899+0000 7f33d2e14640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc107d90 0x7f33cc080060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.900+0000 7f33d2e14640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33cc108f80 0x7f33cc0805a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.900+0000 7f33d2e14640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33cc109940 0x7f33cc081620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.900+0000 7f33d2e14640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc07a8a0 con 0x7f33cc109940 2024-04-01T01:38:44.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.900+0000 7f33d2e14640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc07a720 con 0x7f33cc107d90 2024-04-01T01:38:44.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.900+0000 7f33d2e14640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc07aa20 con 0x7f33cc108f80 2024-04-01T01:38:44.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.900+0000 7f33d0bb0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc107d90 0x7f33cc080060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.900+0000 7f33d0bb0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc107d90 0x7f33cc080060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48740/0 (socket says 172.21.15.70:48740) 2024-04-01T01:38:44.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.900+0000 7f33d0bb0640 1 -- 172.21.15.70:0/3629564378 learned_addr learned my addr 172.21.15.70:0/3629564378 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:44.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.901+0000 7f33d0bb0640 1 -- 172.21.15.70:0/3629564378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33cc108f80 msgr2=0x7f33cc0805a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:38:44.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.901+0000 7f33d0bb0640 1 --2- 172.21.15.70:0/3629564378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33cc108f80 0x7f33cc0805a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.901+0000 7f33d0bb0640 1 -- 172.21.15.70:0/3629564378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33cc109940 msgr2=0x7f33cc081620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:44.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.901+0000 7f33d0bb0640 1 --2- 172.21.15.70:0/3629564378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33cc109940 0x7f33cc081620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.901+0000 7f33d0bb0640 1 -- 172.21.15.70:0/3629564378 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33cc081cc0 con 0x7f33cc107d90 2024-04-01T01:38:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.902+0000 7f33d0bb0640 1 --2- 172.21.15.70:0/3629564378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc107d90 0x7f33cc080060 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f33bc009850 tx=0x7f33bc02cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.902+0000 7f33c9ffb640 1 -- 172.21.15.70:0/3629564378 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33bc043020 con 0x7f33cc107d90 2024-04-01T01:38:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.902+0000 7f33c9ffb640 1 -- 172.21.15.70:0/3629564378 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f33bc0052b0 con 0x7f33cc107d90 2024-04-01T01:38:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.902+0000 7f33c9ffb640 1 -- 172.21.15.70:0/3629564378 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33bc003d80 con 0x7f33cc107d90 2024-04-01T01:38:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.903+0000 7f33d2e14640 1 -- 172.21.15.70:0/3629564378 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33cc081f50 con 0x7f33cc107d90 2024-04-01T01:38:44.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.903+0000 7f33d2e14640 1 -- 172.21.15.70:0/3629564378 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33cc082450 con 0x7f33cc107d90 2024-04-01T01:38:44.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.904+0000 7f33d2e14640 1 -- 172.21.15.70:0/3629564378 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33cc068990 con 0x7f33cc107d90 2024-04-01T01:38:44.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.907+0000 7f33c9ffb640 1 -- 172.21.15.70:0/3629564378 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f33bc046020 con 0x7f33cc107d90 2024-04-01T01:38:44.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.908+0000 7f33c9ffb640 1 --2- 172.21.15.70:0/3629564378 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f33b40a2620 0x7f33b40a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:44.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.908+0000 7f33c9ffb640 1 -- 172.21.15.70:0/3629564378 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f33bc0f14c0 con 0x7f33cc107d90 2024-04-01T01:38:44.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.909+0000 7f33cbfff640 1 --2- 172.21.15.70:0/3629564378 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f33b40a2620 0x7f33b40a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:44.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.909+0000 7f33cbfff640 1 --2- 172.21.15.70:0/3629564378 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f33b40a2620 0x7f33b40a4ae0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f33cc081180 tx=0x7f33c0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:44.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:44.913+0000 7f33c9ffb640 1 -- 172.21.15.70:0/3629564378 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f33bc04d020 con 0x7f33cc107d90 2024-04-01T01:38:45.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.337+0000 7f33d2e14640 1 -- 172.21.15.70:0/3629564378 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33cc10efe0 con 0x7f33cc107d90 2024-04-01T01:38:45.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.338+0000 7f33c9ffb640 1 -- 172.21.15.70:0/3629564378 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f33bc0be450 con 0x7f33cc107d90 2024-04-01T01:38:45.338 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:38:45.338 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:33:36.894838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24418,24404,24379,24451,24464]},"id":1}]} 2024-04-01T01:38:45.339 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-04-01T01:38:45.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.342+0000 7f33d2e14640 1 -- 172.21.15.70:0/3629564378 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f33b40a2620 msgr2=0x7f33b40a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.342+0000 7f33d2e14640 1 --2- 172.21.15.70:0/3629564378 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f33b40a2620 0x7f33b40a4ae0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f33cc081180 tx=0x7f33c0005f90 comp rx=0 tx=0).stop 2024-04-01T01:38:45.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.342+0000 7f33d2e14640 1 -- 172.21.15.70:0/3629564378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc107d90 msgr2=0x7f33cc080060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.342+0000 7f33d2e14640 1 --2- 172.21.15.70:0/3629564378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc107d90 0x7f33cc080060 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f33bc009850 tx=0x7f33bc02cde0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.342+0000 7f33d2e14640 1 -- 172.21.15.70:0/3629564378 shutdown_connections 2024-04-01T01:38:45.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.342+0000 7f33d2e14640 1 --2- 172.21.15.70:0/3629564378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33cc109940 0x7f33cc081620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.342+0000 7f33d2e14640 1 --2- 172.21.15.70:0/3629564378 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f33b40a2620 0x7f33b40a4ae0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.342+0000 7f33d2e14640 1 --2- 172.21.15.70:0/3629564378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33cc108f80 0x7f33cc0805a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.342+0000 7f33d2e14640 1 --2- 172.21.15.70:0/3629564378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33cc107d90 0x7f33cc080060 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.342+0000 7f33d2e14640 1 -- 172.21.15.70:0/3629564378 >> 172.21.15.70:0/3629564378 conn(0x7f33cc076d60 msgr2=0x7f33cc10c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:45.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.343+0000 7f33d2e14640 1 -- 172.21.15.70:0/3629564378 shutdown_connections 2024-04-01T01:38:45.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.343+0000 7f33d2e14640 1 -- 172.21.15.70:0/3629564378 wait complete. 2024-04-01T01:38:45.360 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:38:45.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.539+0000 7f9f12a0d640 1 -- 172.21.15.70:0/2595763973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c100530 msgr2=0x7f9f0c100950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.539+0000 7f9f12a0d640 1 --2- 172.21.15.70:0/2595763973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c100530 0x7f9f0c100950 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009900 tx=0x7f9f0802bc50 comp rx=0 tx=0).stop 2024-04-01T01:38:45.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.540+0000 7f9f12a0d640 1 -- 172.21.15.70:0/2595763973 shutdown_connections 2024-04-01T01:38:45.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.540+0000 7f9f12a0d640 1 --2- 172.21.15.70:0/2595763973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f0c1020e0 0x7f9f0c108970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.540+0000 7f9f12a0d640 1 --2- 172.21.15.70:0/2595763973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f0c101720 0x7f9f0c101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.540+0000 7f9f12a0d640 1 --2- 172.21.15.70:0/2595763973 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c100530 0x7f9f0c100950 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.540+0000 7f9f12a0d640 1 -- 172.21.15.70:0/2595763973 >> 172.21.15.70:0/2595763973 conn(0x7f9f0c0fbac0 msgr2=0x7f9f0c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:45.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.540+0000 7f9f12a0d640 1 -- 172.21.15.70:0/2595763973 shutdown_connections 2024-04-01T01:38:45.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.540+0000 7f9f12a0d640 1 -- 172.21.15.70:0/2595763973 wait complete. 2024-04-01T01:38:45.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.541+0000 7f9f12a0d640 1 Processor -- start 2024-04-01T01:38:45.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.541+0000 7f9f12a0d640 1 -- start start 2024-04-01T01:38:45.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.542+0000 7f9f12a0d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f0c100530 0x7f9f0c19e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.542+0000 7f9f12a0d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c101720 0x7f9f0c19e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.542+0000 7f9f12a0d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f0c1020e0 0x7f9f0c1a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.542+0000 7f9f12a0d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c10b090 con 0x7f9f0c100530 2024-04-01T01:38:45.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.542+0000 7f9f12a0d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c10af10 con 0x7f9f0c101720 2024-04-01T01:38:45.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.542+0000 7f9f12a0d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c10b210 con 0x7f9f0c1020e0 2024-04-01T01:38:45.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.542+0000 7f9f11a0b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f0c100530 0x7f9f0c19e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.542+0000 7f9f1220c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f0c1020e0 0x7f9f0c1a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.542+0000 7f9f11a0b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f0c100530 0x7f9f0c19e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44914/0 (socket says 172.21.15.70:44914) 2024-04-01T01:38:45.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.542+0000 7f9f11a0b640 1 -- 172.21.15.70:0/1775635474 learned_addr learned my addr 172.21.15.70:0/1775635474 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:45.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.542+0000 7f9f1120a640 1 --2- 172.21.15.70:0/1775635474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c101720 0x7f9f0c19e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.543+0000 7f9f1120a640 1 -- 172.21.15.70:0/1775635474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f0c1020e0 msgr2=0x7f9f0c1a55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.543+0000 7f9f1120a640 1 --2- 172.21.15.70:0/1775635474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f0c1020e0 0x7f9f0c1a55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.543+0000 7f9f1120a640 1 -- 172.21.15.70:0/1775635474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f0c100530 msgr2=0x7f9f0c19e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.543+0000 7f9f1120a640 1 --2- 172.21.15.70:0/1775635474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f0c100530 0x7f9f0c19e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.543+0000 7f9f1120a640 1 -- 172.21.15.70:0/1775635474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f0c1a5c50 con 0x7f9f0c101720 2024-04-01T01:38:45.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.543+0000 7f9f1120a640 1 --2- 172.21.15.70:0/1775635474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c101720 0x7f9f0c19e570 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00b3f0 tx=0x7f9efc00b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:45.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.544+0000 7f9f02ffd640 1 -- 172.21.15.70:0/1775635474 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc015020 con 0x7f9f0c101720 2024-04-01T01:38:45.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.544+0000 7f9f02ffd640 1 -- 172.21.15.70:0/1775635474 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9efc011ce0 con 0x7f9f0c101720 2024-04-01T01:38:45.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.544+0000 7f9f02ffd640 1 -- 172.21.15.70:0/1775635474 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc010de0 con 0x7f9f0c101720 2024-04-01T01:38:45.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.544+0000 7f9f12a0d640 1 -- 172.21.15.70:0/1775635474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f0c1a5f40 con 0x7f9f0c101720 2024-04-01T01:38:45.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.544+0000 7f9f12a0d640 1 -- 172.21.15.70:0/1775635474 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f0c1a63d0 con 0x7f9f0c101720 2024-04-01T01:38:45.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.547+0000 7f9f02ffd640 1 -- 172.21.15.70:0/1775635474 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f9efc018020 con 0x7f9f0c101720 2024-04-01T01:38:45.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.548+0000 7f9f16851740 1 Processor -- start 2024-04-01T01:38:45.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.548+0000 7f9f16851740 1 -- start start 2024-04-01T01:38:45.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.548+0000 7f9f16851740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e246a0 0x55f6f9e24ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.549+0000 7f9f16851740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e25090 0x55f6f9e29b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.549+0000 7f9f16851740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e2a200 0x55f6f9e2c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.549+0000 7f9f16851740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f6f9d891b0 con 0x55f6f9e2a200 2024-04-01T01:38:45.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.549+0000 7f9f16851740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f6f9d31c00 con 0x55f6f9e25090 2024-04-01T01:38:45.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.549+0000 7f9f16851740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f6f9d85250 con 0x55f6f9e246a0 2024-04-01T01:38:45.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.549+0000 7f9f1220c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e246a0 0x55f6f9e24ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.549+0000 7f9f1220c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e246a0 0x55f6f9e24ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:54666/0 (socket says 172.21.15.70:54666) 2024-04-01T01:38:45.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.549+0000 7f9f1220c640 1 -- 172.21.15.70:0/891763991 learned_addr learned my addr 172.21.15.70:0/891763991 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:45.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.549+0000 7f9f1220c640 1 -- 172.21.15.70:0/891763991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e25090 msgr2=0x55f6f9e29b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:38:45.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.549+0000 7f9f1220c640 1 --2- 172.21.15.70:0/891763991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e25090 0x55f6f9e29b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.549+0000 7f9f1220c640 1 -- 172.21.15.70:0/891763991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e2a200 msgr2=0x55f6f9e2c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:45.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.549+0000 7f9f1220c640 1 --2- 172.21.15.70:0/891763991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e2a200 0x55f6f9e2c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.549+0000 7f9f1220c640 1 -- 172.21.15.70:0/891763991 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6f9d513d0 con 0x55f6f9e246a0 2024-04-01T01:38:45.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.550+0000 7f9f02ffd640 1 --2- 172.21.15.70:0/1775635474 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9ef80a2650 0x7f9ef80a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.550+0000 7f9f1220c640 1 --2- 172.21.15.70:0/891763991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e246a0 0x55f6f9e24ac0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9f04009f40 tx=0x7f9f0400c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89975bb7b9b1c238 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:45.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.550+0000 7f9f11a0b640 1 --2- 172.21.15.70:0/1775635474 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9ef80a2650 0x7f9ef80a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.550+0000 7f9ef7fff640 1 -- 172.21.15.70:0/891763991 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0401a070 con 0x55f6f9e246a0 2024-04-01T01:38:45.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.550+0000 7f9f02ffd640 1 -- 172.21.15.70:0/1775635474 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9efc0ce2a0 con 0x7f9f0c101720 2024-04-01T01:38:45.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.550+0000 7f9ef7fff640 1 -- 172.21.15.70:0/891763991 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f0400aaa0 con 0x55f6f9e246a0 2024-04-01T01:38:45.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.550+0000 7f9ef7fff640 1 -- 172.21.15.70:0/891763991 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f04015c00 con 0x55f6f9e246a0 2024-04-01T01:38:45.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.551+0000 7f9f16851740 1 -- 172.21.15.70:0/891763991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e246a0 msgr2=0x55f6f9e24ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.551+0000 7f9f16851740 1 --2- 172.21.15.70:0/891763991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e246a0 0x55f6f9e24ac0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9f04009f40 tx=0x7f9f0400c890 comp rx=0 tx=0).stop 2024-04-01T01:38:45.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.551+0000 7f9f11a0b640 1 --2- 172.21.15.70:0/1775635474 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9ef80a2650 0x7f9ef80a4b10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x55f6f9e25b00 tx=0x7f9f0802c800 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:45.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.551+0000 7f9f16851740 1 -- 172.21.15.70:0/891763991 shutdown_connections 2024-04-01T01:38:45.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.551+0000 7f9f16851740 1 --2- 172.21.15.70:0/891763991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e2a200 0x55f6f9e2c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.551+0000 7f9f16851740 1 --2- 172.21.15.70:0/891763991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e25090 0x55f6f9e29b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.551+0000 7f9f16851740 1 --2- 172.21.15.70:0/891763991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e246a0 0x55f6f9e24ac0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.551+0000 7f9f16851740 1 -- 172.21.15.70:0/891763991 >> 172.21.15.70:0/891763991 conn(0x55f6f9e1fc10 msgr2=0x55f6f9e20010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:45.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.552+0000 7f9f16851740 1 -- 172.21.15.70:0/891763991 shutdown_connections 2024-04-01T01:38:45.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.552+0000 7f9f16851740 1 -- 172.21.15.70:0/891763991 wait complete. 2024-04-01T01:38:45.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.552+0000 7f9f16851740 1 -- start start 2024-04-01T01:38:45.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.552+0000 7f9f16851740 1 Processor -- start 2024-04-01T01:38:45.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.553+0000 7f9f16851740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e246a0 0x55f6f9e2ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.553+0000 7f9f16851740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e25090 0x55f6f9e2efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.553+0000 7f9f16851740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e2a200 0x55f6f9e35ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.553+0000 7f9f16851740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f6f9e2d1b0 con 0x55f6f9e25090 2024-04-01T01:38:45.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.553+0000 7f9f16851740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f6f9d27470 con 0x55f6f9e246a0 2024-04-01T01:38:45.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.553+0000 7f9f16851740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f6f9e2d330 con 0x55f6f9e2a200 2024-04-01T01:38:45.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.553+0000 7f9f16851740 10 client.0 _ll_register_callbacks cb 0x55f6f9d44cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:45.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.553+0000 7f9f1220c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e2a200 0x55f6f9e35ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.553+0000 7f9f1220c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e2a200 0x55f6f9e35ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:54682/0 (socket says 172.21.15.70:54682) 2024-04-01T01:38:45.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.553+0000 7f9f1220c640 1 -- 172.21.15.70:0/1972709675 learned_addr learned my addr 172.21.15.70:0/1972709675 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:45.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.553+0000 7f9f11a0b640 1 --2- 172.21.15.70:0/1972709675 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e246a0 0x55f6f9e2ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.554+0000 7f9f1220c640 1 -- 172.21.15.70:0/1972709675 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e246a0 msgr2=0x55f6f9e2ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:45.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.554+0000 7f9f1220c640 1 --2- 172.21.15.70:0/1972709675 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e246a0 0x55f6f9e2ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.554+0000 7f9f1220c640 1 -- 172.21.15.70:0/1972709675 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e25090 msgr2=0x55f6f9e2efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:45.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.554+0000 7f9f1220c640 1 --2- 172.21.15.70:0/1972709675 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e25090 0x55f6f9e2efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.554+0000 7f9f1220c640 1 -- 172.21.15.70:0/1972709675 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6f9a9d3f0 con 0x55f6f9e2a200 2024-04-01T01:38:45.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.554+0000 7f9f1220c640 1 --2- 172.21.15.70:0/1972709675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e2a200 0x55f6f9e35ff0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9f04015bb0 tx=0x7f9f04015be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5adf626b93f0d962 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:45.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.554+0000 7f9ef57fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:38:45.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.554+0000 7f9ef57fa640 1 -- 172.21.15.70:0/1972709675 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f04024020 con 0x55f6f9e2a200 2024-04-01T01:38:45.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.554+0000 7f9ef57fa640 1 -- 172.21.15.70:0/1972709675 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9f04015e00 con 0x55f6f9e2a200 2024-04-01T01:38:45.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.554+0000 7f9f16851740 1 -- 172.21.15.70:0/1972709675 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f0400aaa0 con 0x55f6f9e2a200 2024-04-01T01:38:45.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.555+0000 7f9ef57fa640 1 -- 172.21.15.70:0/1972709675 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9f0402f050 con 0x55f6f9e2a200 2024-04-01T01:38:45.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.555+0000 7f9f16851740 10 client.26878 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:45.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.555+0000 7f9f16851740 1 -- 172.21.15.70:0/1972709675 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f6f9b677d0 con 0x55f6f9e2a200 2024-04-01T01:38:45.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.556+0000 7f9ef57fa640 1 -- 172.21.15.70:0/1972709675 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f0400aaa0 con 0x55f6f9e2a200 2024-04-01T01:38:45.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.556+0000 7f9ef57fa640 1 -- 172.21.15.70:0/1972709675 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9f0401f3d0 con 0x55f6f9e2a200 2024-04-01T01:38:45.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.556+0000 7f9f16851740 10 client.26878 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:45.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.556+0000 7f9f16851740 10 client.26878 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:38:45.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.556+0000 7f9f16851740 20 client.26878 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:45.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.556+0000 7f9f16851740 1 --2- 172.21.15.70:0/1972709675 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55f6f9e3a200 0x55f6f9e3c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.556+0000 7f9f16851740 4 client.26878 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:45.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.556+0000 7f9f16851740 1 -- 172.21.15.70:0/1972709675 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6f9d966f0 con 0x55f6f9e3a200 2024-04-01T01:38:45.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.556+0000 7f9f11a0b640 1 --2- 172.21.15.70:0/1972709675 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55f6f9e3a200 0x55f6f9e3c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.557+0000 7f9ef57fa640 10 client.26878 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:38:45.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.557+0000 7f9f11a0b640 1 --2- 172.21.15.70:0/1972709675 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55f6f9e3a200 0x55f6f9e3c5f0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=58077a6845ce3c9a server_cookie=81ae8331c9981b5 in_seq=0 out_seq=0 2024-04-01T01:38:45.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.558+0000 7f9ef57fa640 1 -- 172.21.15.70:0/1972709675 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f6f9d966f0 con 0x55f6f9e3a200 2024-04-01T01:38:45.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.558+0000 7f9ef57fa640 10 client.26878 handle_command_reply: tid=0 2024-04-01T01:38:45.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.558+0000 7f9f16851740 1 client.26878 shutdown 2024-04-01T01:38:45.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.559+0000 7f9f16851740 1 -- 172.21.15.70:0/1972709675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e2a200 msgr2=0x55f6f9e35ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.559+0000 7f9f16851740 1 --2- 172.21.15.70:0/1972709675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e2a200 0x55f6f9e35ff0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9f04015bb0 tx=0x7f9f04015be0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.559+0000 7f9f16851740 1 -- 172.21.15.70:0/1972709675 shutdown_connections 2024-04-01T01:38:45.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.559+0000 7f9f16851740 1 --2- 172.21.15.70:0/1972709675 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55f6f9e3a200 0x55f6f9e3c5f0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.559+0000 7f9f16851740 1 --2- 172.21.15.70:0/1972709675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e2a200 0x55f6f9e35ff0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.559+0000 7f9f16851740 1 --2- 172.21.15.70:0/1972709675 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e25090 0x55f6f9e2efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.559+0000 7f9f16851740 1 --2- 172.21.15.70:0/1972709675 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e246a0 0x55f6f9e2ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.559+0000 7f9f16851740 1 -- 172.21.15.70:0/1972709675 >> 172.21.15.70:0/1972709675 conn(0x55f6f9e1fc10 msgr2=0x55f6f9e1ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:45.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.561+0000 7f9ef57fa640 0 client.26878 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:38:45.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.561+0000 7f9f16851740 1 -- 172.21.15.70:0/1972709675 shutdown_connections 2024-04-01T01:38:45.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.561+0000 7f9f16851740 1 -- 172.21.15.70:0/1972709675 wait complete. 2024-04-01T01:38:45.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.563+0000 7f9f16851740 20 client.26878 trim_cache size 0 max 16384 2024-04-01T01:38:45.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.589+0000 7f9f16851740 1 Processor -- start 2024-04-01T01:38:45.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.589+0000 7f9f16851740 1 -- start start 2024-04-01T01:38:45.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.589+0000 7f9f16851740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e246a0 0x55f6f9e2ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.589+0000 7f9f16851740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e25090 0x55f6f9e35fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.589+0000 7f9f16851740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e2a200 0x55f6f9e2efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.589+0000 7f9f16851740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c1b09d0 con 0x55f6f9e2a200 2024-04-01T01:38:45.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.589+0000 7f9f16851740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c1b2510 con 0x55f6f9e246a0 2024-04-01T01:38:45.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.589+0000 7f9f16851740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c1b10a0 con 0x55f6f9e25090 2024-04-01T01:38:45.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.589+0000 7f9f1220c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e2a200 0x55f6f9e2efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.589+0000 7f9f1220c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e2a200 0x55f6f9e2efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44940/0 (socket says 172.21.15.70:44940) 2024-04-01T01:38:45.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.589+0000 7f9f1220c640 1 -- 172.21.15.70:0/2189533672 learned_addr learned my addr 172.21.15.70:0/2189533672 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:45.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.590+0000 7f9f1220c640 1 -- 172.21.15.70:0/2189533672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e25090 msgr2=0x55f6f9e35fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:38:45.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.590+0000 7f9f1220c640 1 --2- 172.21.15.70:0/2189533672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e25090 0x55f6f9e35fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.590+0000 7f9f1220c640 1 -- 172.21.15.70:0/2189533672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e246a0 msgr2=0x55f6f9e2ea70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:38:45.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.590+0000 7f9f1220c640 1 --2- 172.21.15.70:0/2189533672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e246a0 0x55f6f9e2ea70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.590+0000 7f9f1220c640 1 -- 172.21.15.70:0/2189533672 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6f9e3cbd0 con 0x55f6f9e2a200 2024-04-01T01:38:45.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.590+0000 7f9f1220c640 1 --2- 172.21.15.70:0/2189533672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e2a200 0x55f6f9e2efb0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9f040022f0 tx=0x7f9f04015680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ccdfd859c7e8da server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:45.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.591+0000 7f9ef77fe640 1 -- 172.21.15.70:0/2189533672 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f04024030 con 0x55f6f9e2a200 2024-04-01T01:38:45.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.591+0000 7f9ef77fe640 1 -- 172.21.15.70:0/2189533672 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f0401feb0 con 0x55f6f9e2a200 2024-04-01T01:38:45.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.591+0000 7f9ef77fe640 1 -- 172.21.15.70:0/2189533672 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0401a040 con 0x55f6f9e2a200 2024-04-01T01:38:45.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.592+0000 7f9f16851740 1 -- 172.21.15.70:0/2189533672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e2a200 msgr2=0x55f6f9e2efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.592+0000 7f9f16851740 1 --2- 172.21.15.70:0/2189533672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e2a200 0x55f6f9e2efb0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9f040022f0 tx=0x7f9f04015680 comp rx=0 tx=0).stop 2024-04-01T01:38:45.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.593+0000 7f9f16851740 1 -- 172.21.15.70:0/2189533672 shutdown_connections 2024-04-01T01:38:45.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.593+0000 7f9f16851740 1 --2- 172.21.15.70:0/2189533672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e2a200 0x55f6f9e2efb0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.593+0000 7f9f16851740 1 --2- 172.21.15.70:0/2189533672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e25090 0x55f6f9e35fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.593+0000 7f9f16851740 1 --2- 172.21.15.70:0/2189533672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e246a0 0x55f6f9e2ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.593+0000 7f9f16851740 1 -- 172.21.15.70:0/2189533672 >> 172.21.15.70:0/2189533672 conn(0x55f6f9e1fc10 msgr2=0x55f6f9e26450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:45.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.593+0000 7f9f16851740 1 -- 172.21.15.70:0/2189533672 shutdown_connections 2024-04-01T01:38:45.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.594+0000 7f9f16851740 1 -- 172.21.15.70:0/2189533672 wait complete. 2024-04-01T01:38:45.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.595+0000 7f9f16851740 1 -- start start 2024-04-01T01:38:45.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.595+0000 7f9f16851740 1 Processor -- start 2024-04-01T01:38:45.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.596+0000 7f9f16851740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e246a0 0x55f6f9e336b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.596+0000 7f9f16851740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e25090 0x55f6f9e33bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.596+0000 7f9f16851740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e2a200 0x55f6f9e34b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.596+0000 7f9f1120a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e25090 0x55f6f9e33bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.596+0000 7f9f1120a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e25090 0x55f6f9e33bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44956/0 (socket says 172.21.15.70:44956) 2024-04-01T01:38:45.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.596+0000 7f9f1120a640 1 -- 172.21.15.70:0/236136648 learned_addr learned my addr 172.21.15.70:0/236136648 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:45.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.597+0000 7f9f1220c640 1 --2- 172.21.15.70:0/236136648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e2a200 0x55f6f9e34b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.597+0000 7f9f16851740 1 -- 172.21.15.70:0/236136648 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c1b1e40 con 0x55f6f9e25090 2024-04-01T01:38:45.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.597+0000 7f9f16851740 1 -- 172.21.15.70:0/236136648 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c1b1770 con 0x55f6f9e2a200 2024-04-01T01:38:45.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.597+0000 7f9f16851740 1 -- 172.21.15.70:0/236136648 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c1ae9b0 con 0x55f6f9e246a0 2024-04-01T01:38:45.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.598+0000 7f9f1220c640 1 -- 172.21.15.70:0/236136648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e246a0 msgr2=0x55f6f9e336b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:38:45.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.598+0000 7f9f1220c640 1 --2- 172.21.15.70:0/236136648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e246a0 0x55f6f9e336b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.598+0000 7f9f1220c640 1 -- 172.21.15.70:0/236136648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e25090 msgr2=0x55f6f9e33bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:45.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.598+0000 7f9f1220c640 1 --2- 172.21.15.70:0/236136648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e25090 0x55f6f9e33bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.598+0000 7f9f1220c640 1 -- 172.21.15.70:0/236136648 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6f9e35180 con 0x55f6f9e2a200 2024-04-01T01:38:45.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.598+0000 7f9f16851740 10 client.0 _ll_register_callbacks cb 0x7f9f0400a820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:45.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.598+0000 7f9f16851740 1 -- 172.21.15.70:0/236136648 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f6f9d90a40 con 0x55f6f9e2a200 2024-04-01T01:38:45.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.598+0000 7f9f1220c640 1 --2- 172.21.15.70:0/236136648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e2a200 0x55f6f9e34b90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9f04009d50 tx=0x7f9f04015410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e7aba539cd54663 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:45.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.598+0000 7f9ef57fa640 10 client.26987 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:45.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.598+0000 7f9ef57fa640 1 -- 172.21.15.70:0/236136648 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f04024070 con 0x55f6f9e2a200 2024-04-01T01:38:45.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.598+0000 7f9ef57fa640 1 -- 172.21.15.70:0/236136648 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9f04003a30 con 0x55f6f9e2a200 2024-04-01T01:38:45.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.599+0000 7f9ef57fa640 1 -- 172.21.15.70:0/236136648 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9f0401a040 con 0x55f6f9e2a200 2024-04-01T01:38:45.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.600+0000 7f9f16851740 10 client.26987 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:45.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.600+0000 7f9f16851740 1 -- 172.21.15.70:0/236136648 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55f6f9e362d0 con 0x55f6f9e2a200 2024-04-01T01:38:45.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.601+0000 7f9ef57fa640 1 -- 172.21.15.70:0/236136648 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9f04003c90 con 0x55f6f9e2a200 2024-04-01T01:38:45.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.601+0000 7f9f16851740 10 client.26987 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:45.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.601+0000 7f9f16851740 10 client.26987 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:38:45.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.601+0000 7f9f16851740 20 client.26987 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:45.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.601+0000 7f9f16851740 1 --2- 172.21.15.70:0/236136648 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55f6f9ea09a0 0x55f6f9ea2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.601+0000 7f9f16851740 4 client.26987 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:38:45.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.601+0000 7f9f16851740 1 -- 172.21.15.70:0/236136648 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f6f9d0e6a0 con 0x55f6f9ea09a0 2024-04-01T01:38:45.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.602+0000 7f9f11a0b640 1 --2- 172.21.15.70:0/236136648 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55f6f9ea09a0 0x55f6f9ea2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.602+0000 7f9f11a0b640 1 --2- 172.21.15.70:0/236136648 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55f6f9ea09a0 0x55f6f9ea2d90 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=593bbfa45e57f52e server_cookie=45c1bd35cdd4601d in_seq=0 out_seq=0 2024-04-01T01:38:45.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.602+0000 7f9ef57fa640 10 client.26987 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:38:45.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.603+0000 7f9ef57fa640 1 -- 172.21.15.70:0/236136648 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f6f9d0e6a0 con 0x55f6f9ea09a0 2024-04-01T01:38:45.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.603+0000 7f9ef57fa640 10 client.26987 handle_command_reply: tid=0 2024-04-01T01:38:45.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.605+0000 7f9f16851740 1 client.26987 shutdown 2024-04-01T01:38:45.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.606+0000 7f9f16851740 1 -- 172.21.15.70:0/236136648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e2a200 msgr2=0x55f6f9e34b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.606+0000 7f9f16851740 1 --2- 172.21.15.70:0/236136648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e2a200 0x55f6f9e34b90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9f04009d50 tx=0x7f9f04015410 comp rx=0 tx=0).stop 2024-04-01T01:38:45.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.606+0000 7f9f16851740 1 -- 172.21.15.70:0/236136648 shutdown_connections 2024-04-01T01:38:45.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.606+0000 7f9f16851740 1 --2- 172.21.15.70:0/236136648 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55f6f9ea09a0 0x55f6f9ea2d90 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.606+0000 7f9f16851740 1 --2- 172.21.15.70:0/236136648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6f9e2a200 0x55f6f9e34b90 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.606+0000 7f9f16851740 1 --2- 172.21.15.70:0/236136648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6f9e25090 0x55f6f9e33bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.606+0000 7f9f16851740 1 --2- 172.21.15.70:0/236136648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6f9e246a0 0x55f6f9e336b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.606+0000 7f9f16851740 1 -- 172.21.15.70:0/236136648 >> 172.21.15.70:0/236136648 conn(0x55f6f9e1fc10 msgr2=0x55f6f9e2ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:45.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.606+0000 7f9ef57fa640 0 client.26987 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:38:45.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.610+0000 7f9f16851740 1 -- 172.21.15.70:0/236136648 shutdown_connections 2024-04-01T01:38:45.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.610+0000 7f9f16851740 1 -- 172.21.15.70:0/236136648 wait complete. 2024-04-01T01:38:45.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.610+0000 7f9f16851740 20 client.26987 trim_cache size 0 max 16384 2024-04-01T01:38:45.612 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:38:45.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.614+0000 7f9f12a0d640 1 -- 172.21.15.70:0/1775635474 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9ef80a2650 msgr2=0x7f9ef80a4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.614+0000 7f9f12a0d640 1 --2- 172.21.15.70:0/1775635474 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9ef80a2650 0x7f9ef80a4b10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x55f6f9e25b00 tx=0x7f9f0802c800 comp rx=0 tx=0).stop 2024-04-01T01:38:45.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.614+0000 7f9f12a0d640 1 -- 172.21.15.70:0/1775635474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c101720 msgr2=0x7f9f0c19e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.614+0000 7f9f12a0d640 1 --2- 172.21.15.70:0/1775635474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c101720 0x7f9f0c19e570 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9efc00b3f0 tx=0x7f9efc00b8b0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.615+0000 7f9f12a0d640 1 -- 172.21.15.70:0/1775635474 shutdown_connections 2024-04-01T01:38:45.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.615+0000 7f9f12a0d640 1 --2- 172.21.15.70:0/1775635474 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9ef80a2650 0x7f9ef80a4b10 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.615+0000 7f9f12a0d640 1 --2- 172.21.15.70:0/1775635474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f0c1020e0 0x7f9f0c1a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.615+0000 7f9f12a0d640 1 --2- 172.21.15.70:0/1775635474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c101720 0x7f9f0c19e570 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.615+0000 7f9f12a0d640 1 --2- 172.21.15.70:0/1775635474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f0c100530 0x7f9f0c19e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.615+0000 7f9f12a0d640 1 -- 172.21.15.70:0/1775635474 >> 172.21.15.70:0/1775635474 conn(0x7f9f0c0fbac0 msgr2=0x7f9f0c104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:45.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.617+0000 7f9f12a0d640 1 -- 172.21.15.70:0/1775635474 shutdown_connections 2024-04-01T01:38:45.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.618+0000 7f9f12a0d640 1 -- 172.21.15.70:0/1775635474 wait complete. 2024-04-01T01:38:45.638 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:38:45.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.820+0000 7febad309640 1 -- 172.21.15.70:0/1410407516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feba8108e20 msgr2=0x7feba81092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.820+0000 7febad309640 1 --2- 172.21.15.70:0/1410407516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feba8108e20 0x7feba81092a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009900 tx=0x7feb9c02bc60 comp rx=0 tx=0).stop 2024-04-01T01:38:45.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.822+0000 7febad309640 1 -- 172.21.15.70:0/1410407516 shutdown_connections 2024-04-01T01:38:45.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.822+0000 7febad309640 1 --2- 172.21.15.70:0/1410407516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feba81097e0 0x7feba810e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.822+0000 7febad309640 1 --2- 172.21.15.70:0/1410407516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feba8108e20 0x7feba81092a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.822+0000 7febad309640 1 --2- 172.21.15.70:0/1410407516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feba8072b60 0x7feba8072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.822+0000 7febad309640 1 -- 172.21.15.70:0/1410407516 >> 172.21.15.70:0/1410407516 conn(0x7feba806e630 msgr2=0x7feba8070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:45.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.823+0000 7febad309640 1 -- 172.21.15.70:0/1410407516 shutdown_connections 2024-04-01T01:38:45.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.823+0000 7febad309640 1 -- 172.21.15.70:0/1410407516 wait complete. 2024-04-01T01:38:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.824+0000 7febad309640 1 Processor -- start 2024-04-01T01:38:45.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.824+0000 7febad309640 1 -- start start 2024-04-01T01:38:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.825+0000 7febad309640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feba8072b60 0x7feba81a6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.825+0000 7febad309640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feba8108e20 0x7feba81a6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.825+0000 7febad309640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feba81097e0 0x7feba81adbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.825+0000 7febad309640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7feba811bfa0 con 0x7feba8072b60 2024-04-01T01:38:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.825+0000 7febad309640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7feba811be20 con 0x7feba81097e0 2024-04-01T01:38:45.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.825+0000 7febad309640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7feba811c120 con 0x7feba8108e20 2024-04-01T01:38:45.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.825+0000 7febacb08640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feba81097e0 0x7feba81adbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.825+0000 7febacb08640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feba81097e0 0x7feba81adbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48798/0 (socket says 172.21.15.70:48798) 2024-04-01T01:38:45.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.825+0000 7febacb08640 1 -- 172.21.15.70:0/2426172750 learned_addr learned my addr 172.21.15.70:0/2426172750 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:45.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.826+0000 7febacb08640 1 -- 172.21.15.70:0/2426172750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feba8108e20 msgr2=0x7feba81a6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.826+0000 7feba77fe640 1 --2- 172.21.15.70:0/2426172750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feba8108e20 0x7feba81a6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.826+0000 7feba7fff640 1 --2- 172.21.15.70:0/2426172750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feba8072b60 0x7feba81a6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.826+0000 7febacb08640 1 --2- 172.21.15.70:0/2426172750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feba8108e20 0x7feba81a6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.826+0000 7febacb08640 1 -- 172.21.15.70:0/2426172750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feba8072b60 msgr2=0x7feba81a6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.826+0000 7febacb08640 1 --2- 172.21.15.70:0/2426172750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feba8072b60 0x7feba81a6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.826+0000 7febacb08640 1 -- 172.21.15.70:0/2426172750 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba81ae240 con 0x7feba81097e0 2024-04-01T01:38:45.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.826+0000 7feba7fff640 1 --2- 172.21.15.70:0/2426172750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feba8072b60 0x7feba81a6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:38:45.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.826+0000 7febacb08640 1 --2- 172.21.15.70:0/2426172750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feba81097e0 0x7feba81adbd0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7feba000eb10 tx=0x7feba000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:45.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.826+0000 7feba57fa640 1 -- 172.21.15.70:0/2426172750 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba0004b20 con 0x7feba81097e0 2024-04-01T01:38:45.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.827+0000 7febad309640 1 -- 172.21.15.70:0/2426172750 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feba81ae530 con 0x7feba81097e0 2024-04-01T01:38:45.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.827+0000 7febad309640 1 -- 172.21.15.70:0/2426172750 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feba81aea80 con 0x7feba81097e0 2024-04-01T01:38:45.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.827+0000 7feba57fa640 1 -- 172.21.15.70:0/2426172750 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feba0004cc0 con 0x7feba81097e0 2024-04-01T01:38:45.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.827+0000 7feba57fa640 1 -- 172.21.15.70:0/2426172750 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba0010690 con 0x7feba81097e0 2024-04-01T01:38:45.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.830+0000 7febb1220740 1 Processor -- start 2024-04-01T01:38:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.830+0000 7feba57fa640 1 -- 172.21.15.70:0/2426172750 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7feba0010830 con 0x7feba81097e0 2024-04-01T01:38:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.830+0000 7febb1220740 1 -- start start 2024-04-01T01:38:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.831+0000 7febb1220740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 0x55dae6a7eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.831+0000 7febb1220740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a7f090 0x55dae6a83b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.831+0000 7febb1220740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a84200 0x55dae6a86690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.831+0000 7febb1220740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dae69e31b0 con 0x55dae6a7f090 2024-04-01T01:38:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.831+0000 7febb1220740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55dae698bc00 con 0x55dae6a84200 2024-04-01T01:38:45.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.831+0000 7febb1220740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dae69df250 con 0x55dae6a7e6a0 2024-04-01T01:38:45.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.831+0000 7feba7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a7f090 0x55dae6a83b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.831+0000 7feba7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a7f090 0x55dae6a83b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44982/0 (socket says 172.21.15.70:44982) 2024-04-01T01:38:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.831+0000 7feba7fff640 1 -- 172.21.15.70:0/1130934679 learned_addr learned my addr 172.21.15.70:0/1130934679 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.831+0000 7feba77fe640 1 --2- 172.21.15.70:0/1130934679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 0x55dae6a7eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.831+0000 7feba77fe640 1 --2- 172.21.15.70:0/1130934679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a84200 0x55dae6a86690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.831+0000 7feba7fff640 1 -- 172.21.15.70:0/1130934679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 msgr2=0x55dae6a7eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.831+0000 7feba7fff640 1 --2- 172.21.15.70:0/1130934679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 0x55dae6a7eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.831+0000 7feba7fff640 1 -- 172.21.15.70:0/1130934679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a84200 msgr2=0x55dae6a86690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.831+0000 7feba7fff640 1 --2- 172.21.15.70:0/1130934679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a84200 0x55dae6a86690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.831+0000 7feba7fff640 1 -- 172.21.15.70:0/1130934679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dae69ab3d0 con 0x55dae6a7f090 2024-04-01T01:38:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.832+0000 7feba57fa640 1 --2- 172.21.15.70:0/2426172750 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7feb900a25f0 0x7feb900a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.832+0000 7feba7fff640 1 --2- 172.21.15.70:0/1130934679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a7f090 0x55dae6a83b90 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7feb98009c30 tx=0x7feb9800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e776dc12bc3bd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.832+0000 7feba57fa640 1 -- 172.21.15.70:0/2426172750 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7feba00cf120 con 0x7feba81097e0 2024-04-01T01:38:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.832+0000 7feba77fe640 1 --2- 172.21.15.70:0/2426172750 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7feb900a25f0 0x7feb900a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.832+0000 7feb8e7fc640 1 -- 172.21.15.70:0/1130934679 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb98016070 con 0x55dae6a7f090 2024-04-01T01:38:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.832+0000 7feb8e7fc640 1 -- 172.21.15.70:0/1130934679 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb980054d0 con 0x55dae6a7f090 2024-04-01T01:38:45.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.832+0000 7feb8e7fc640 1 -- 172.21.15.70:0/1130934679 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb98011470 con 0x55dae6a7f090 2024-04-01T01:38:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.833+0000 7febb1220740 1 -- 172.21.15.70:0/1130934679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a7f090 msgr2=0x55dae6a83b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.833+0000 7febb1220740 1 --2- 172.21.15.70:0/1130934679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a7f090 0x55dae6a83b90 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7feb98009c30 tx=0x7feb9800e6c0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.833+0000 7feba77fe640 1 --2- 172.21.15.70:0/2426172750 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7feb900a25f0 0x7feb900a4ab0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7feb9c02c270 tx=0x7feb9c02bc60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.834+0000 7febb1220740 1 -- 172.21.15.70:0/1130934679 shutdown_connections 2024-04-01T01:38:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.834+0000 7febb1220740 1 --2- 172.21.15.70:0/1130934679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a84200 0x55dae6a86690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.834+0000 7febb1220740 1 --2- 172.21.15.70:0/1130934679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a7f090 0x55dae6a83b90 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.834+0000 7febb1220740 1 --2- 172.21.15.70:0/1130934679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 0x55dae6a7eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.834+0000 7febb1220740 1 -- 172.21.15.70:0/1130934679 >> 172.21.15.70:0/1130934679 conn(0x55dae6a79c10 msgr2=0x55dae6a7a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.834+0000 7febb1220740 1 -- 172.21.15.70:0/1130934679 shutdown_connections 2024-04-01T01:38:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.834+0000 7febb1220740 1 -- 172.21.15.70:0/1130934679 wait complete. 2024-04-01T01:38:45.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.834+0000 7febb1220740 1 -- start start 2024-04-01T01:38:45.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.835+0000 7febb1220740 1 Processor -- start 2024-04-01T01:38:45.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.836+0000 7febb1220740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a7e6a0 0x55dae6a88bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.836+0000 7febb1220740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a7f090 0x55dae6a89130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.836+0000 7febb1220740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a84200 0x55dae6a90170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.836+0000 7febb1220740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dae6a871b0 con 0x55dae6a7e6a0 2024-04-01T01:38:45.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.836+0000 7febb1220740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55dae6981470 con 0x55dae6a7f090 2024-04-01T01:38:45.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.836+0000 7febb1220740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dae6a87330 con 0x55dae6a84200 2024-04-01T01:38:45.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.837+0000 7febacb08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a84200 0x55dae6a90170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.837+0000 7feba77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a7f090 0x55dae6a89130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.837+0000 7febacb08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a84200 0x55dae6a90170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:54716/0 (socket says 172.21.15.70:54716) 2024-04-01T01:38:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.837+0000 7febacb08640 1 -- 172.21.15.70:0/3713653939 learned_addr learned my addr 172.21.15.70:0/3713653939 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.837+0000 7febb1220740 10 client.0 _ll_register_callbacks cb 0x55dae699ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.837+0000 7febacb08640 1 -- 172.21.15.70:0/3713653939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a7f090 msgr2=0x55dae6a89130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.837+0000 7febacb08640 1 --2- 172.21.15.70:0/3713653939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a7f090 0x55dae6a89130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.837+0000 7febacb08640 1 -- 172.21.15.70:0/3713653939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a7e6a0 msgr2=0x55dae6a88bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.837+0000 7febacb08640 1 --2- 172.21.15.70:0/3713653939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a7e6a0 0x55dae6a88bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.837+0000 7febacb08640 1 -- 172.21.15.70:0/3713653939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dae66f73f0 con 0x55dae6a84200 2024-04-01T01:38:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.837+0000 7febacb08640 1 --2- 172.21.15.70:0/3713653939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a84200 0x55dae6a90170 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7feba009b740 tx=0x7feba009b1d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80c5488dad29b108 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.837+0000 7feb877fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:38:45.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.838+0000 7feb877fe640 1 -- 172.21.15.70:0/3713653939 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba00b4060 con 0x55dae6a84200 2024-04-01T01:38:45.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.838+0000 7feb877fe640 1 -- 172.21.15.70:0/3713653939 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7feba00a3d30 con 0x55dae6a84200 2024-04-01T01:38:45.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.838+0000 7febb1220740 1 -- 172.21.15.70:0/3713653939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb980054d0 con 0x55dae6a84200 2024-04-01T01:38:45.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.838+0000 7feb877fe640 1 -- 172.21.15.70:0/3713653939 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feba00b9050 con 0x55dae6a84200 2024-04-01T01:38:45.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.839+0000 7febb1220740 10 client.26887 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:45.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.839+0000 7febb1220740 1 -- 172.21.15.70:0/3713653939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dae67c17d0 con 0x55dae6a84200 2024-04-01T01:38:45.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.840+0000 7feb877fe640 1 -- 172.21.15.70:0/3713653939 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feba009b7b0 con 0x55dae6a84200 2024-04-01T01:38:45.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.840+0000 7feb877fe640 1 -- 172.21.15.70:0/3713653939 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7feba0010cc0 con 0x55dae6a84200 2024-04-01T01:38:45.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.840+0000 7febb1220740 10 client.26887 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:45.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.840+0000 7febb1220740 10 client.26887 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:38:45.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.840+0000 7febb1220740 20 client.26887 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:45.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.840+0000 7febb1220740 1 --2- 172.21.15.70:0/3713653939 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55dae6a94710 0x55dae6a96ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.840+0000 7febb1220740 4 client.26887 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:45.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.840+0000 7febb1220740 1 -- 172.21.15.70:0/3713653939 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7feb980054d0 con 0x55dae6a94710 2024-04-01T01:38:45.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.840+0000 7feba7fff640 1 --2- 172.21.15.70:0/3713653939 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55dae6a94710 0x55dae6a96ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.842+0000 7feba7fff640 1 --2- 172.21.15.70:0/3713653939 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55dae6a94710 0x55dae6a96ba0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bf11f3ed6b05dc55 server_cookie=f27c788649ab753f in_seq=0 out_seq=0 2024-04-01T01:38:45.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.842+0000 7feb877fe640 10 client.26887 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:38:45.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.843+0000 7feb877fe640 1 -- 172.21.15.70:0/3713653939 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7feb980054d0 con 0x55dae6a94710 2024-04-01T01:38:45.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.843+0000 7feb877fe640 10 client.26887 handle_command_reply: tid=0 2024-04-01T01:38:45.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.844+0000 7febb1220740 1 client.26887 shutdown 2024-04-01T01:38:45.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.846+0000 7febb1220740 1 -- 172.21.15.70:0/3713653939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a84200 msgr2=0x55dae6a90170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.846+0000 7febb1220740 1 --2- 172.21.15.70:0/3713653939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a84200 0x55dae6a90170 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7feba009b740 tx=0x7feba009b1d0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.848+0000 7febb1220740 1 -- 172.21.15.70:0/3713653939 shutdown_connections 2024-04-01T01:38:45.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.848+0000 7febb1220740 1 --2- 172.21.15.70:0/3713653939 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55dae6a94710 0x55dae6a96ba0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.848+0000 7febb1220740 1 --2- 172.21.15.70:0/3713653939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a84200 0x55dae6a90170 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.848+0000 7febb1220740 1 --2- 172.21.15.70:0/3713653939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a7f090 0x55dae6a89130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.848+0000 7febb1220740 1 --2- 172.21.15.70:0/3713653939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a7e6a0 0x55dae6a88bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.848+0000 7febb1220740 1 -- 172.21.15.70:0/3713653939 >> 172.21.15.70:0/3713653939 conn(0x55dae6a79c10 msgr2=0x55dae6a78f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:45.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.848+0000 7feb877fe640 0 client.26887 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:38:45.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.848+0000 7febb1220740 1 -- 172.21.15.70:0/3713653939 shutdown_connections 2024-04-01T01:38:45.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.848+0000 7febb1220740 1 -- 172.21.15.70:0/3713653939 wait complete. 2024-04-01T01:38:45.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.849+0000 7febb1220740 20 client.26887 trim_cache size 0 max 16384 2024-04-01T01:38:45.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.873+0000 7febb1220740 1 Processor -- start 2024-04-01T01:38:45.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.873+0000 7febb1220740 1 -- start start 2024-04-01T01:38:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.874+0000 7febb1220740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 0x55dae6a96720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.874+0000 7febb1220740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a7f090 0x55dae6a90130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.874+0000 7febb1220740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a84200 0x55dae6a78700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.874+0000 7feba7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 0x55dae6a96720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.874+0000 7feba7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 0x55dae6a96720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:54722/0 (socket says 172.21.15.70:54722) 2024-04-01T01:38:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.874+0000 7feba7fff640 1 -- 172.21.15.70:0/2167378189 learned_addr learned my addr 172.21.15.70:0/2167378189 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.875+0000 7febacb08640 1 --2- 172.21.15.70:0/2167378189 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a84200 0x55dae6a78700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.875+0000 7febb1220740 1 -- 172.21.15.70:0/2167378189 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7feb9400fd20 con 0x55dae6a7f090 2024-04-01T01:38:45.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.875+0000 7febb1220740 1 -- 172.21.15.70:0/2167378189 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7feb94011860 con 0x55dae6a84200 2024-04-01T01:38:45.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.875+0000 7febb1220740 1 -- 172.21.15.70:0/2167378189 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7feb940103f0 con 0x55dae6a7e6a0 2024-04-01T01:38:45.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.875+0000 7febacb08640 1 -- 172.21.15.70:0/2167378189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 msgr2=0x55dae6a96720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:45.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.875+0000 7febacb08640 1 --2- 172.21.15.70:0/2167378189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 0x55dae6a96720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.875+0000 7febacb08640 1 -- 172.21.15.70:0/2167378189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a7f090 msgr2=0x55dae6a90130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:38:45.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.875+0000 7febacb08640 1 --2- 172.21.15.70:0/2167378189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a7f090 0x55dae6a90130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.875+0000 7febacb08640 1 -- 172.21.15.70:0/2167378189 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dae6a7f920 con 0x55dae6a84200 2024-04-01T01:38:45.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.876+0000 7febacb08640 1 --2- 172.21.15.70:0/2167378189 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a84200 0x55dae6a78700 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7feba0098180 tx=0x7feba009b240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea6b109a0c755d1b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:45.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.876+0000 7feb8dffb640 1 -- 172.21.15.70:0/2167378189 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba007f080 con 0x55dae6a84200 2024-04-01T01:38:45.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.876+0000 7feb8dffb640 1 -- 172.21.15.70:0/2167378189 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feba009b7b0 con 0x55dae6a84200 2024-04-01T01:38:45.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.876+0000 7feb8dffb640 1 -- 172.21.15.70:0/2167378189 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba00a3d30 con 0x55dae6a84200 2024-04-01T01:38:45.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.877+0000 7febb1220740 1 -- 172.21.15.70:0/2167378189 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a84200 msgr2=0x55dae6a78700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.877+0000 7febb1220740 1 --2- 172.21.15.70:0/2167378189 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a84200 0x55dae6a78700 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7feba0098180 tx=0x7feba009b240 comp rx=0 tx=0).stop 2024-04-01T01:38:45.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.877+0000 7febb1220740 1 -- 172.21.15.70:0/2167378189 shutdown_connections 2024-04-01T01:38:45.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.877+0000 7febb1220740 1 --2- 172.21.15.70:0/2167378189 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a84200 0x55dae6a78700 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.877+0000 7febb1220740 1 --2- 172.21.15.70:0/2167378189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a7f090 0x55dae6a90130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.877+0000 7febb1220740 1 --2- 172.21.15.70:0/2167378189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 0x55dae6a96720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.877+0000 7febb1220740 1 -- 172.21.15.70:0/2167378189 >> 172.21.15.70:0/2167378189 conn(0x55dae6a79c10 msgr2=0x55dae6a80590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:45.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.879+0000 7febb1220740 1 -- 172.21.15.70:0/2167378189 shutdown_connections 2024-04-01T01:38:45.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.879+0000 7febb1220740 1 -- 172.21.15.70:0/2167378189 wait complete. 2024-04-01T01:38:45.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.880+0000 7febb1220740 1 -- start start 2024-04-01T01:38:45.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.880+0000 7febb1220740 1 Processor -- start 2024-04-01T01:38:45.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.881+0000 7febb1220740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 0x55dae6a8d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.881+0000 7febb1220740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a7f090 0x55dae6a8de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.881+0000 7feba7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 0x55dae6a8d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.881+0000 7feba7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 0x55dae6a8d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:54732/0 (socket says 172.21.15.70:54732) 2024-04-01T01:38:45.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.881+0000 7feba77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a7f090 0x55dae6a8de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.881+0000 7feba77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a7f090 0x55dae6a8de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48818/0 (socket says 172.21.15.70:48818) 2024-04-01T01:38:45.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.881+0000 7feba7fff640 1 -- 172.21.15.70:0/610316339 learned_addr learned my addr 172.21.15.70:0/610316339 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:45.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.881+0000 7febb1220740 1 --2- 172.21.15.70:0/610316339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a84200 0x55dae6a8ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.882+0000 7febb1220740 1 -- 172.21.15.70:0/610316339 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7feb94011190 con 0x55dae6a84200 2024-04-01T01:38:45.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.882+0000 7febb1220740 1 -- 172.21.15.70:0/610316339 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7feb94010ac0 con 0x55dae6a7f090 2024-04-01T01:38:45.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.882+0000 7febb1220740 1 -- 172.21.15.70:0/610316339 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7feb9400dd00 con 0x55dae6a7e6a0 2024-04-01T01:38:45.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.882+0000 7febacb08640 1 --2- 172.21.15.70:0/610316339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a84200 0x55dae6a8ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.882+0000 7febb1220740 10 client.0 _ll_register_callbacks cb 0x7feba0097b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:45.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.883+0000 7feba77fe640 1 -- 172.21.15.70:0/610316339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 msgr2=0x55dae6a8d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:45.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.883+0000 7feba77fe640 1 --2- 172.21.15.70:0/610316339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 0x55dae6a8d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.883+0000 7feba77fe640 1 -- 172.21.15.70:0/610316339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a84200 msgr2=0x55dae6a8ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:45.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.883+0000 7feba77fe640 1 --2- 172.21.15.70:0/610316339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a84200 0x55dae6a8ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.883+0000 7feba77fe640 1 -- 172.21.15.70:0/610316339 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dae6a8f440 con 0x55dae6a7f090 2024-04-01T01:38:45.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.883+0000 7febacb08640 1 --2- 172.21.15.70:0/610316339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a84200 0x55dae6a8ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:38:45.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.883+0000 7feba77fe640 1 --2- 172.21.15.70:0/610316339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a7f090 0x55dae6a8de40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7feb9c005c20 tx=0x7feb9c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd446a02b808fd33 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:45.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.883+0000 7feb877fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:45.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.884+0000 7feb877fe640 1 -- 172.21.15.70:0/610316339 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c0048c0 con 0x55dae6a7f090 2024-04-01T01:38:45.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.884+0000 7feb877fe640 1 -- 172.21.15.70:0/610316339 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7feb9c004a60 con 0x55dae6a7f090 2024-04-01T01:38:45.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.884+0000 7febb1220740 1 -- 172.21.15.70:0/610316339 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feba009b7b0 con 0x55dae6a7f090 2024-04-01T01:38:45.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.884+0000 7feb877fe640 1 -- 172.21.15.70:0/610316339 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feb9c05d050 con 0x55dae6a7f090 2024-04-01T01:38:45.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.884+0000 7febb1220740 10 client.27005 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:45.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.884+0000 7febb1220740 1 -- 172.21.15.70:0/610316339 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dae6afac10 con 0x55dae6a7f090 2024-04-01T01:38:45.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.884+0000 7feb877fe640 1 -- 172.21.15.70:0/610316339 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb9c054070 con 0x55dae6a7f090 2024-04-01T01:38:45.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.885+0000 7feb877fe640 1 -- 172.21.15.70:0/610316339 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7feb9c043480 con 0x55dae6a7f090 2024-04-01T01:38:45.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.885+0000 7febb1220740 10 client.27005 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:45.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.885+0000 7febb1220740 10 client.27005 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:38:45.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.885+0000 7febb1220740 20 client.27005 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:45.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.885+0000 7febb1220740 1 --2- 172.21.15.70:0/610316339 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55dae6afb580 0x55dae6afd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:45.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.885+0000 7febb1220740 4 client.27005 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:38:45.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.885+0000 7febb1220740 1 -- 172.21.15.70:0/610316339 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7feba009b7b0 con 0x55dae6afb580 2024-04-01T01:38:45.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.886+0000 7feba7fff640 1 --2- 172.21.15.70:0/610316339 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55dae6afb580 0x55dae6afd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:45.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.886+0000 7feba7fff640 1 --2- 172.21.15.70:0/610316339 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55dae6afb580 0x55dae6afd9a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5b416472d74f4672 server_cookie=8fcd8e7841553762 in_seq=0 out_seq=0 2024-04-01T01:38:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.886+0000 7feb877fe640 10 client.27005 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:38:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.887+0000 7feb877fe640 1 -- 172.21.15.70:0/610316339 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7feba009b7b0 con 0x55dae6afb580 2024-04-01T01:38:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.887+0000 7feb877fe640 10 client.27005 handle_command_reply: tid=0 2024-04-01T01:38:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.887+0000 7febb1220740 1 client.27005 shutdown 2024-04-01T01:38:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.887+0000 7febb1220740 1 -- 172.21.15.70:0/610316339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a7f090 msgr2=0x55dae6a8de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.887+0000 7febb1220740 1 --2- 172.21.15.70:0/610316339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a7f090 0x55dae6a8de40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7feb9c005c20 tx=0x7feb9c009260 comp rx=0 tx=0).stop 2024-04-01T01:38:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.887+0000 7febb1220740 1 -- 172.21.15.70:0/610316339 shutdown_connections 2024-04-01T01:38:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.887+0000 7febb1220740 1 --2- 172.21.15.70:0/610316339 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55dae6afb580 0x55dae6afd9a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.887+0000 7febb1220740 1 --2- 172.21.15.70:0/610316339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dae6a84200 0x55dae6a8ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.887+0000 7feb877fe640 0 client.27005 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:38:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.887+0000 7febb1220740 1 --2- 172.21.15.70:0/610316339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dae6a7f090 0x55dae6a8de40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.888+0000 7febb1220740 1 --2- 172.21.15.70:0/610316339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dae6a7e6a0 0x55dae6a8d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.888+0000 7febb1220740 1 -- 172.21.15.70:0/610316339 >> 172.21.15.70:0/610316339 conn(0x55dae6a79c10 msgr2=0x55dae6a7ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:45.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.888+0000 7febb1220740 1 -- 172.21.15.70:0/610316339 shutdown_connections 2024-04-01T01:38:45.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.888+0000 7febb1220740 1 -- 172.21.15.70:0/610316339 wait complete. 2024-04-01T01:38:45.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.888+0000 7febb1220740 20 client.27005 trim_cache size 0 max 16384 2024-04-01T01:38:45.889 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:38:45.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.889+0000 7febad309640 1 -- 172.21.15.70:0/2426172750 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7feb900a25f0 msgr2=0x7feb900a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.889+0000 7febad309640 1 --2- 172.21.15.70:0/2426172750 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7feb900a25f0 0x7feb900a4ab0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7feb9c02c270 tx=0x7feb9c02bc60 comp rx=0 tx=0).stop 2024-04-01T01:38:45.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.889+0000 7febad309640 1 -- 172.21.15.70:0/2426172750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feba81097e0 msgr2=0x7feba81adbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.889+0000 7febad309640 1 --2- 172.21.15.70:0/2426172750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feba81097e0 0x7feba81adbd0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7feba000eb10 tx=0x7feba000c2d0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.889+0000 7febad309640 1 -- 172.21.15.70:0/2426172750 shutdown_connections 2024-04-01T01:38:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.889+0000 7febad309640 1 --2- 172.21.15.70:0/2426172750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feba81097e0 0x7feba81adbd0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.889+0000 7febad309640 1 --2- 172.21.15.70:0/2426172750 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7feb900a25f0 0x7feb900a4ab0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.889+0000 7febad309640 1 --2- 172.21.15.70:0/2426172750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feba8108e20 0x7feba81a6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.889+0000 7febad309640 1 --2- 172.21.15.70:0/2426172750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feba8072b60 0x7feba81a6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:45.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.889+0000 7febad309640 1 -- 172.21.15.70:0/2426172750 >> 172.21.15.70:0/2426172750 conn(0x7feba806e630 msgr2=0x7feba8070290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:45.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.891+0000 7febad309640 1 -- 172.21.15.70:0/2426172750 shutdown_connections 2024-04-01T01:38:45.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:45.891+0000 7febad309640 1 -- 172.21.15.70:0/2426172750 wait complete. 2024-04-01T01:38:45.907 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:45 smithi070 ceph-mon[32179]: pgmap v516: 129 pgs: 129 active+clean; 1.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 9.3 MiB/s wr, 1.00k op/s 2024-04-01T01:38:45.907 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:45 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3629564378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:38:45.908 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:38:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:45 smithi081 ceph-mon[35142]: pgmap v516: 129 pgs: 129 active+clean; 1.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 9.3 MiB/s wr, 1.00k op/s 2024-04-01T01:38:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:45 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3629564378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:38:46.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.090+0000 7ff692e92640 1 -- 172.21.15.70:0/2678352694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68c104a40 msgr2=0x7ff68c104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.090+0000 7ff692e92640 1 --2- 172.21.15.70:0/2678352694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68c104a40 0x7ff68c104ec0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009900 tx=0x7ff67c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:38:46.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.090+0000 7ff692e92640 1 -- 172.21.15.70:0/2678352694 shutdown_connections 2024-04-01T01:38:46.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.090+0000 7ff692e92640 1 --2- 172.21.15.70:0/2678352694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff68c105400 0x7ff68c10bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.090+0000 7ff692e92640 1 --2- 172.21.15.70:0/2678352694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68c104a40 0x7ff68c104ec0 secure :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009900 tx=0x7ff67c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:38:46.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.090+0000 7ff692e92640 1 --2- 172.21.15.70:0/2678352694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff68c103850 0x7ff68c103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.090+0000 7ff692e92640 1 -- 172.21.15.70:0/2678352694 >> 172.21.15.70:0/2678352694 conn(0x7ff68c0fedc0 msgr2=0x7ff68c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.091+0000 7ff692e92640 1 -- 172.21.15.70:0/2678352694 shutdown_connections 2024-04-01T01:38:46.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.091+0000 7ff692e92640 1 -- 172.21.15.70:0/2678352694 wait complete. 2024-04-01T01:38:46.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.091+0000 7ff692e92640 1 Processor -- start 2024-04-01T01:38:46.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff692e92640 1 -- start start 2024-04-01T01:38:46.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff692e92640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff68c103850 0x7ff68c19dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff692e92640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff68c105400 0x7ff68c19e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff692e92640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68c1a3570 0x7ff68c1a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff692e92640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff68c10f3b0 con 0x7ff68c105400 2024-04-01T01:38:46.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff692e92640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff68c10f230 con 0x7ff68c103850 2024-04-01T01:38:46.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff692e92640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff68c10f530 con 0x7ff68c1a3570 2024-04-01T01:38:46.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff691e90640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff68c103850 0x7ff68c19dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff691e90640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff68c103850 0x7ff68c19dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48838/0 (socket says 172.21.15.70:48838) 2024-04-01T01:38:46.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff691e90640 1 -- 172.21.15.70:0/1697788970 learned_addr learned my addr 172.21.15.70:0/1697788970 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:46.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff69168f640 1 --2- 172.21.15.70:0/1697788970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff68c105400 0x7ff68c19e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff692691640 1 --2- 172.21.15.70:0/1697788970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68c1a3570 0x7ff68c1a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff691e90640 1 -- 172.21.15.70:0/1697788970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68c1a3570 msgr2=0x7ff68c1a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff691e90640 1 --2- 172.21.15.70:0/1697788970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68c1a3570 0x7ff68c1a5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff691e90640 1 -- 172.21.15.70:0/1697788970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff68c105400 msgr2=0x7ff68c19e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff691e90640 1 --2- 172.21.15.70:0/1697788970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff68c105400 0x7ff68c19e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.092+0000 7ff691e90640 1 -- 172.21.15.70:0/1697788970 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff68c194bc0 con 0x7ff68c103850 2024-04-01T01:38:46.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.093+0000 7ff691e90640 1 --2- 172.21.15.70:0/1697788970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff68c103850 0x7ff68c19dfd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff68800e9c0 tx=0x7ff68800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.093+0000 7ff682ffd640 1 -- 172.21.15.70:0/1697788970 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68800cd80 con 0x7ff68c103850 2024-04-01T01:38:46.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.093+0000 7ff682ffd640 1 -- 172.21.15.70:0/1697788970 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff6880035f0 con 0x7ff68c103850 2024-04-01T01:38:46.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.093+0000 7ff682ffd640 1 -- 172.21.15.70:0/1697788970 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff688010690 con 0x7ff68c103850 2024-04-01T01:38:46.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.093+0000 7ff692e92640 1 -- 172.21.15.70:0/1697788970 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff68c194eb0 con 0x7ff68c103850 2024-04-01T01:38:46.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.093+0000 7ff692e92640 1 -- 172.21.15.70:0/1697788970 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff68c195420 con 0x7ff68c103850 2024-04-01T01:38:46.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.097+0000 7ff696da8740 1 Processor -- start 2024-04-01T01:38:46.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.097+0000 7ff682ffd640 1 -- 172.21.15.70:0/1697788970 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7ff6880037a0 con 0x7ff68c103850 2024-04-01T01:38:46.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.098+0000 7ff696da8740 1 -- start start 2024-04-01T01:38:46.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.098+0000 7ff696da8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae676a0 0x5563aae67ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.098+0000 7ff696da8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae68090 0x5563aae6cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.098+0000 7ff696da8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae6d200 0x5563aae6f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.099+0000 7ff696da8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5563aadcc1b0 con 0x5563aae68090 2024-04-01T01:38:46.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.099+0000 7ff696da8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5563aad74c00 con 0x5563aae676a0 2024-04-01T01:38:46.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.099+0000 7ff696da8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5563aadc8250 con 0x5563aae6d200 2024-04-01T01:38:46.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.099+0000 7ff692691640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae676a0 0x5563aae67ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.099+0000 7ff692691640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae676a0 0x5563aae67ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48844/0 (socket says 172.21.15.70:48844) 2024-04-01T01:38:46.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.099+0000 7ff692691640 1 -- 172.21.15.70:0/347929586 learned_addr learned my addr 172.21.15.70:0/347929586 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:46.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.099+0000 7ff69168f640 1 --2- 172.21.15.70:0/347929586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae6d200 0x5563aae6f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.099+0000 7ff691e90640 1 --2- 172.21.15.70:0/347929586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae68090 0x5563aae6cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.099+0000 7ff682ffd640 1 --2- 172.21.15.70:0/1697788970 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff6780a2560 0x7ff6780a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.099+0000 7ff692691640 1 -- 172.21.15.70:0/347929586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae6d200 msgr2=0x5563aae6f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.099+0000 7ff692691640 1 --2- 172.21.15.70:0/347929586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae6d200 0x5563aae6f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.099+0000 7ff692691640 1 -- 172.21.15.70:0/347929586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae68090 msgr2=0x5563aae6cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.099+0000 7ff692691640 1 --2- 172.21.15.70:0/347929586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae68090 0x5563aae6cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.099+0000 7ff692691640 1 -- 172.21.15.70:0/347929586 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563aad943d0 con 0x5563aae676a0 2024-04-01T01:38:46.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.100+0000 7ff692691640 1 --2- 172.21.15.70:0/1697788970 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff6780a2560 0x7ff6780a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.100+0000 7ff682ffd640 1 -- 172.21.15.70:0/1697788970 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff6880cee40 con 0x7ff68c103850 2024-04-01T01:38:46.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.100+0000 7ff692691640 1 --2- 172.21.15.70:0/347929586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae676a0 0x5563aae67ac0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff684011590 tx=0x7ff684011a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=413910b68e0d81c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.100+0000 7ff677fff640 1 -- 172.21.15.70:0/347929586 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff684017050 con 0x5563aae676a0 2024-04-01T01:38:46.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.100+0000 7ff677fff640 1 -- 172.21.15.70:0/347929586 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff68400fcf0 con 0x5563aae676a0 2024-04-01T01:38:46.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.100+0000 7ff677fff640 1 -- 172.21.15.70:0/347929586 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68401ba50 con 0x5563aae676a0 2024-04-01T01:38:46.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.100+0000 7ff692691640 1 --2- 172.21.15.70:0/1697788970 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff6780a2560 0x7ff6780a4a20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff684022000 tx=0x7ff684023000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.101+0000 7ff696da8740 1 -- 172.21.15.70:0/347929586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae676a0 msgr2=0x5563aae67ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.101+0000 7ff696da8740 1 --2- 172.21.15.70:0/347929586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae676a0 0x5563aae67ac0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff684011590 tx=0x7ff684011a50 comp rx=0 tx=0).stop 2024-04-01T01:38:46.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.101+0000 7ff696da8740 1 -- 172.21.15.70:0/347929586 shutdown_connections 2024-04-01T01:38:46.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.101+0000 7ff696da8740 1 --2- 172.21.15.70:0/347929586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae6d200 0x5563aae6f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.101+0000 7ff696da8740 1 --2- 172.21.15.70:0/347929586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae68090 0x5563aae6cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.101+0000 7ff696da8740 1 --2- 172.21.15.70:0/347929586 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae676a0 0x5563aae67ac0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.101+0000 7ff696da8740 1 -- 172.21.15.70:0/347929586 >> 172.21.15.70:0/347929586 conn(0x5563aae62c10 msgr2=0x5563aae63010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.102+0000 7ff696da8740 1 -- 172.21.15.70:0/347929586 shutdown_connections 2024-04-01T01:38:46.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.102+0000 7ff696da8740 1 -- 172.21.15.70:0/347929586 wait complete. 2024-04-01T01:38:46.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.102+0000 7ff696da8740 1 -- start start 2024-04-01T01:38:46.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.102+0000 7ff696da8740 1 Processor -- start 2024-04-01T01:38:46.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.102+0000 7ff696da8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae676a0 0x5563aae71be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.102+0000 7ff696da8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae68090 0x5563aae72120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.102+0000 7ff696da8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae6d200 0x5563aae79160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.102+0000 7ff696da8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5563aae701b0 con 0x5563aae6d200 2024-04-01T01:38:46.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.102+0000 7ff696da8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5563aad6a470 con 0x5563aae676a0 2024-04-01T01:38:46.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.102+0000 7ff696da8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5563aae70330 con 0x5563aae68090 2024-04-01T01:38:46.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.102+0000 7ff696da8740 10 client.0 _ll_register_callbacks cb 0x5563aad87cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:46.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.103+0000 7ff691e90640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae676a0 0x5563aae71be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.103+0000 7ff691e90640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae676a0 0x5563aae71be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48860/0 (socket says 172.21.15.70:48860) 2024-04-01T01:38:46.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.103+0000 7ff691e90640 1 -- 172.21.15.70:0/3000212019 learned_addr learned my addr 172.21.15.70:0/3000212019 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:46.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.103+0000 7ff692691640 1 --2- 172.21.15.70:0/3000212019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae6d200 0x5563aae79160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.103+0000 7ff69168f640 1 --2- 172.21.15.70:0/3000212019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae68090 0x5563aae72120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.103+0000 7ff691e90640 1 -- 172.21.15.70:0/3000212019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae68090 msgr2=0x5563aae72120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.103+0000 7ff691e90640 1 --2- 172.21.15.70:0/3000212019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae68090 0x5563aae72120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.103+0000 7ff691e90640 1 -- 172.21.15.70:0/3000212019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae6d200 msgr2=0x5563aae79160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.103+0000 7ff691e90640 1 --2- 172.21.15.70:0/3000212019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae6d200 0x5563aae79160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.103+0000 7ff691e90640 1 -- 172.21.15.70:0/3000212019 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563aaae03f0 con 0x5563aae676a0 2024-04-01T01:38:46.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.103+0000 7ff69168f640 1 --2- 172.21.15.70:0/3000212019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae68090 0x5563aae72120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:38:46.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.103+0000 7ff691e90640 1 --2- 172.21.15.70:0/3000212019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae676a0 0x5563aae71be0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff68800c6a0 tx=0x7ff6880097c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d673d0e1416d3fa server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.103+0000 7ff6757fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:46.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.104+0000 7ff6757fa640 1 -- 172.21.15.70:0/3000212019 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6880a3c50 con 0x5563aae676a0 2024-04-01T01:38:46.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.104+0000 7ff6757fa640 1 -- 172.21.15.70:0/3000212019 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff6880bb0b0 con 0x5563aae676a0 2024-04-01T01:38:46.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.104+0000 7ff696da8740 1 -- 172.21.15.70:0/3000212019 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff68400fcf0 con 0x5563aae676a0 2024-04-01T01:38:46.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.104+0000 7ff6757fa640 1 -- 172.21.15.70:0/3000212019 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff688003b30 con 0x5563aae676a0 2024-04-01T01:38:46.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.105+0000 7ff696da8740 10 client.27017 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:46.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.105+0000 7ff696da8740 1 -- 172.21.15.70:0/3000212019 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5563aabaa7d0 con 0x5563aae676a0 2024-04-01T01:38:46.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.105+0000 7ff6757fa640 1 -- 172.21.15.70:0/3000212019 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff68400fcf0 con 0x5563aae676a0 2024-04-01T01:38:46.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.105+0000 7ff6757fa640 1 -- 172.21.15.70:0/3000212019 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff6880b40d0 con 0x5563aae676a0 2024-04-01T01:38:46.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.105+0000 7ff696da8740 10 client.27017 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:46.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.105+0000 7ff696da8740 10 client.27017 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:38:46.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.105+0000 7ff696da8740 20 client.27017 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:46.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.105+0000 7ff696da8740 1 --2- 172.21.15.70:0/3000212019 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5563aae7d5f0 0x5563aae7fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.105+0000 7ff696da8740 4 client.27017 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:46.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.105+0000 7ff696da8740 1 -- 172.21.15.70:0/3000212019 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563aadd96f0 con 0x5563aae7d5f0 2024-04-01T01:38:46.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.105+0000 7ff69168f640 1 --2- 172.21.15.70:0/3000212019 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5563aae7d5f0 0x5563aae7fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.106+0000 7ff6757fa640 10 client.27017 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:38:46.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.106+0000 7ff69168f640 1 --2- 172.21.15.70:0/3000212019 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5563aae7d5f0 0x5563aae7fa40 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c577e7c5f046ac47 server_cookie=6e207238efd5e8af in_seq=0 out_seq=0 2024-04-01T01:38:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.107+0000 7ff6757fa640 1 -- 172.21.15.70:0/3000212019 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5563aadd96f0 con 0x5563aae7d5f0 2024-04-01T01:38:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.107+0000 7ff6757fa640 10 client.27017 handle_command_reply: tid=0 2024-04-01T01:38:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.107+0000 7ff696da8740 1 client.27017 shutdown 2024-04-01T01:38:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.108+0000 7ff696da8740 1 -- 172.21.15.70:0/3000212019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae676a0 msgr2=0x5563aae71be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.108+0000 7ff696da8740 1 --2- 172.21.15.70:0/3000212019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae676a0 0x5563aae71be0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff68800c6a0 tx=0x7ff6880097c0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.108+0000 7ff696da8740 1 -- 172.21.15.70:0/3000212019 shutdown_connections 2024-04-01T01:38:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.108+0000 7ff696da8740 1 --2- 172.21.15.70:0/3000212019 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5563aae7d5f0 0x5563aae7fa40 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.108+0000 7ff696da8740 1 --2- 172.21.15.70:0/3000212019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae6d200 0x5563aae79160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.108+0000 7ff696da8740 1 --2- 172.21.15.70:0/3000212019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae68090 0x5563aae72120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.108+0000 7ff696da8740 1 --2- 172.21.15.70:0/3000212019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae676a0 0x5563aae71be0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.108+0000 7ff696da8740 1 -- 172.21.15.70:0/3000212019 >> 172.21.15.70:0/3000212019 conn(0x5563aae62c10 msgr2=0x5563aae61f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.108+0000 7ff6757fa640 0 client.27017 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:38:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.108+0000 7ff696da8740 1 -- 172.21.15.70:0/3000212019 shutdown_connections 2024-04-01T01:38:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.108+0000 7ff696da8740 1 -- 172.21.15.70:0/3000212019 wait complete. 2024-04-01T01:38:46.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.108+0000 7ff696da8740 20 client.27017 trim_cache size 0 max 16384 2024-04-01T01:38:46.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.131+0000 7ff696da8740 1 Processor -- start 2024-04-01T01:38:46.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.131+0000 7ff696da8740 1 -- start start 2024-04-01T01:38:46.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.132+0000 7ff696da8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae676a0 0x5563aae71f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.132+0000 7ff696da8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae68090 0x5563aae79120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.132+0000 7ff696da8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae6d200 0x5563aae61700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.132+0000 7ff696da8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff68c1b9470 con 0x5563aae676a0 2024-04-01T01:38:46.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.132+0000 7ff696da8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff68c1bafb0 con 0x5563aae6d200 2024-04-01T01:38:46.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.132+0000 7ff696da8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff68c1b9b40 con 0x5563aae68090 2024-04-01T01:38:46.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.132+0000 7ff691e90640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae676a0 0x5563aae71f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.132+0000 7ff69168f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae68090 0x5563aae79120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.132+0000 7ff691e90640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae676a0 0x5563aae71f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45054/0 (socket says 172.21.15.70:45054) 2024-04-01T01:38:46.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.132+0000 7ff691e90640 1 -- 172.21.15.70:0/1754092848 learned_addr learned my addr 172.21.15.70:0/1754092848 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:46.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.132+0000 7ff692691640 1 --2- 172.21.15.70:0/1754092848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae6d200 0x5563aae61700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.133+0000 7ff692691640 1 -- 172.21.15.70:0/1754092848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae68090 msgr2=0x5563aae79120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.133+0000 7ff692691640 1 --2- 172.21.15.70:0/1754092848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae68090 0x5563aae79120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.133+0000 7ff692691640 1 -- 172.21.15.70:0/1754092848 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae676a0 msgr2=0x5563aae71f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.133+0000 7ff692691640 1 --2- 172.21.15.70:0/1754092848 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae676a0 0x5563aae71f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.133+0000 7ff692691640 1 -- 172.21.15.70:0/1754092848 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563aae672e0 con 0x5563aae6d200 2024-04-01T01:38:46.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.133+0000 7ff692691640 1 --2- 172.21.15.70:0/1754092848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae6d200 0x5563aae61700 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff684005fd0 tx=0x7ff684009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5271016e28670ef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.133+0000 7ff6777fe640 1 -- 172.21.15.70:0/1754092848 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68401bd70 con 0x5563aae6d200 2024-04-01T01:38:46.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.133+0000 7ff6777fe640 1 -- 172.21.15.70:0/1754092848 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff68401b4a0 con 0x5563aae6d200 2024-04-01T01:38:46.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.133+0000 7ff6777fe640 1 -- 172.21.15.70:0/1754092848 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68400f8a0 con 0x5563aae6d200 2024-04-01T01:38:46.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.134+0000 7ff696da8740 1 -- 172.21.15.70:0/1754092848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae6d200 msgr2=0x5563aae61700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.134+0000 7ff696da8740 1 --2- 172.21.15.70:0/1754092848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae6d200 0x5563aae61700 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff684005fd0 tx=0x7ff684009500 comp rx=0 tx=0).stop 2024-04-01T01:38:46.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.134+0000 7ff696da8740 1 -- 172.21.15.70:0/1754092848 shutdown_connections 2024-04-01T01:38:46.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.134+0000 7ff696da8740 1 --2- 172.21.15.70:0/1754092848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae6d200 0x5563aae61700 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.134+0000 7ff696da8740 1 --2- 172.21.15.70:0/1754092848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae68090 0x5563aae79120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.134+0000 7ff696da8740 1 --2- 172.21.15.70:0/1754092848 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae676a0 0x5563aae71f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.134+0000 7ff696da8740 1 -- 172.21.15.70:0/1754092848 >> 172.21.15.70:0/1754092848 conn(0x5563aae62c10 msgr2=0x5563aae717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.135+0000 7ff696da8740 1 -- 172.21.15.70:0/1754092848 shutdown_connections 2024-04-01T01:38:46.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.135+0000 7ff696da8740 1 -- 172.21.15.70:0/1754092848 wait complete. 2024-04-01T01:38:46.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.135+0000 7ff696da8740 1 -- start start 2024-04-01T01:38:46.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.135+0000 7ff696da8740 1 Processor -- start 2024-04-01T01:38:46.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff696da8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae676a0 0x5563aae75e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff696da8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae68090 0x5563aae763b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff696da8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae6d200 0x5563aaee1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff696da8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff68c1ba8e0 con 0x5563aae676a0 2024-04-01T01:38:46.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff696da8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff68c1ba210 con 0x5563aae68090 2024-04-01T01:38:46.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff696da8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff68c19d000 con 0x5563aae6d200 2024-04-01T01:38:46.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff696da8740 10 client.0 _ll_register_callbacks cb 0x7ff688003d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:46.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff69168f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae68090 0x5563aae763b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff69168f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae68090 0x5563aae763b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48872/0 (socket says 172.21.15.70:48872) 2024-04-01T01:38:46.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff69168f640 1 -- 172.21.15.70:0/4010275337 learned_addr learned my addr 172.21.15.70:0/4010275337 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:46.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff692691640 1 --2- 172.21.15.70:0/4010275337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae6d200 0x5563aaee1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff691e90640 1 --2- 172.21.15.70:0/4010275337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae676a0 0x5563aae75e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff69168f640 1 -- 172.21.15.70:0/4010275337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae6d200 msgr2=0x5563aaee1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff69168f640 1 --2- 172.21.15.70:0/4010275337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae6d200 0x5563aaee1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff69168f640 1 -- 172.21.15.70:0/4010275337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae676a0 msgr2=0x5563aae75e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff69168f640 1 --2- 172.21.15.70:0/4010275337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae676a0 0x5563aae75e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff69168f640 1 -- 172.21.15.70:0/4010275337 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563aae769f0 con 0x5563aae68090 2024-04-01T01:38:46.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff692691640 1 --2- 172.21.15.70:0/4010275337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae6d200 0x5563aaee1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:38:46.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff69168f640 1 --2- 172.21.15.70:0/4010275337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae68090 0x5563aae763b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff67c0096c0 tx=0x7ff67c02c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42feeb46d0166132 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.136+0000 7ff6757fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:46.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.137+0000 7ff6757fa640 1 -- 172.21.15.70:0/4010275337 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67c04ae30 con 0x5563aae68090 2024-04-01T01:38:46.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.137+0000 7ff6757fa640 1 -- 172.21.15.70:0/4010275337 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff67c0042d0 con 0x5563aae68090 2024-04-01T01:38:46.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.137+0000 7ff696da8740 1 -- 172.21.15.70:0/4010275337 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff68401b4a0 con 0x5563aae68090 2024-04-01T01:38:46.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.137+0000 7ff6757fa640 1 -- 172.21.15.70:0/4010275337 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff67c058050 con 0x5563aae68090 2024-04-01T01:38:46.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.137+0000 7ff696da8740 10 client.27023 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:46.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.138+0000 7ff696da8740 1 -- 172.21.15.70:0/4010275337 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5563aaee2fc0 con 0x5563aae68090 2024-04-01T01:38:46.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.138+0000 7ff6757fa640 1 -- 172.21.15.70:0/4010275337 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff68401b4a0 con 0x5563aae68090 2024-04-01T01:38:46.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.138+0000 7ff6757fa640 1 -- 172.21.15.70:0/4010275337 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff67c0047a0 con 0x5563aae68090 2024-04-01T01:38:46.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.138+0000 7ff696da8740 10 client.27023 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:46.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.138+0000 7ff696da8740 10 client.27023 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:38:46.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.138+0000 7ff696da8740 20 client.27023 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:46.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.138+0000 7ff696da8740 1 --2- 172.21.15.70:0/4010275337 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5563aaee39a0 0x5563aaee5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.138+0000 7ff696da8740 4 client.27023 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:38:46.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.138+0000 7ff696da8740 1 -- 172.21.15.70:0/4010275337 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5563aae7fdd0 con 0x5563aaee39a0 2024-04-01T01:38:46.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.139+0000 7ff691e90640 1 --2- 172.21.15.70:0/4010275337 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5563aaee39a0 0x5563aaee5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.140+0000 7ff6757fa640 10 client.27023 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:38:46.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.140+0000 7ff691e90640 1 --2- 172.21.15.70:0/4010275337 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5563aaee39a0 0x5563aaee5dc0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a0d444bf9c8c9004 server_cookie=31ffab34c88452e8 in_seq=0 out_seq=0 2024-04-01T01:38:46.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.140+0000 7ff6757fa640 1 -- 172.21.15.70:0/4010275337 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5563aae7fdd0 con 0x5563aaee39a0 2024-04-01T01:38:46.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.140+0000 7ff6757fa640 10 client.27023 handle_command_reply: tid=0 2024-04-01T01:38:46.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.140+0000 7ff696da8740 1 client.27023 shutdown 2024-04-01T01:38:46.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.141+0000 7ff696da8740 1 -- 172.21.15.70:0/4010275337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae68090 msgr2=0x5563aae763b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.141+0000 7ff696da8740 1 --2- 172.21.15.70:0/4010275337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae68090 0x5563aae763b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff67c0096c0 tx=0x7ff67c02c750 comp rx=0 tx=0).stop 2024-04-01T01:38:46.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.141+0000 7ff696da8740 1 -- 172.21.15.70:0/4010275337 shutdown_connections 2024-04-01T01:38:46.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.141+0000 7ff696da8740 1 --2- 172.21.15.70:0/4010275337 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5563aaee39a0 0x5563aaee5dc0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.141+0000 7ff696da8740 1 --2- 172.21.15.70:0/4010275337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5563aae6d200 0x5563aaee1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.141+0000 7ff696da8740 1 --2- 172.21.15.70:0/4010275337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5563aae68090 0x5563aae763b0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.141+0000 7ff6757fa640 0 client.27023 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:38:46.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.141+0000 7ff696da8740 1 --2- 172.21.15.70:0/4010275337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563aae676a0 0x5563aae75e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.141+0000 7ff696da8740 1 -- 172.21.15.70:0/4010275337 >> 172.21.15.70:0/4010275337 conn(0x5563aae62c10 msgr2=0x5563aae65e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.141+0000 7ff696da8740 1 -- 172.21.15.70:0/4010275337 shutdown_connections 2024-04-01T01:38:46.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.141+0000 7ff696da8740 1 -- 172.21.15.70:0/4010275337 wait complete. 2024-04-01T01:38:46.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.141+0000 7ff696da8740 20 client.27023 trim_cache size 0 max 16384 2024-04-01T01:38:46.142 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:38:46.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.142+0000 7ff692e92640 1 -- 172.21.15.70:0/1697788970 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff6780a2560 msgr2=0x7ff6780a4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.142+0000 7ff692e92640 1 --2- 172.21.15.70:0/1697788970 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff6780a2560 0x7ff6780a4a20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7ff684022000 tx=0x7ff684023000 comp rx=0 tx=0).stop 2024-04-01T01:38:46.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.142+0000 7ff692e92640 1 -- 172.21.15.70:0/1697788970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff68c103850 msgr2=0x7ff68c19dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.142+0000 7ff692e92640 1 --2- 172.21.15.70:0/1697788970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff68c103850 0x7ff68c19dfd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff68800e9c0 tx=0x7ff68800ee80 comp rx=0 tx=0).stop 2024-04-01T01:38:46.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.142+0000 7ff692e92640 1 -- 172.21.15.70:0/1697788970 shutdown_connections 2024-04-01T01:38:46.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.142+0000 7ff692e92640 1 --2- 172.21.15.70:0/1697788970 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff6780a2560 0x7ff6780a4a20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.142+0000 7ff692e92640 1 --2- 172.21.15.70:0/1697788970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68c1a3570 0x7ff68c1a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.142+0000 7ff692e92640 1 --2- 172.21.15.70:0/1697788970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff68c105400 0x7ff68c19e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.142+0000 7ff692e92640 1 --2- 172.21.15.70:0/1697788970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff68c103850 0x7ff68c19dfd0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.142+0000 7ff692e92640 1 -- 172.21.15.70:0/1697788970 >> 172.21.15.70:0/1697788970 conn(0x7ff68c0fedc0 msgr2=0x7ff68c077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.143+0000 7ff692e92640 1 -- 172.21.15.70:0/1697788970 shutdown_connections 2024-04-01T01:38:46.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.143+0000 7ff692e92640 1 -- 172.21.15.70:0/1697788970 wait complete. 2024-04-01T01:38:46.165 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:38:46.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:45 smithi189 ceph-mon[35753]: pgmap v516: 129 pgs: 129 active+clean; 1.0 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 9.3 MiB/s wr, 1.00k op/s 2024-04-01T01:38:46.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:45 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3629564378' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:38:46.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.345+0000 7f3e397c7640 1 -- 172.21.15.70:0/636383280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e34109d40 msgr2=0x7f3e3410a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.345+0000 7f3e397c7640 1 --2- 172.21.15.70:0/636383280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e34109d40 0x7f3e3410a1c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c009be0 tx=0x7f3e2c02bef0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.345+0000 7f3e397c7640 1 -- 172.21.15.70:0/636383280 shutdown_connections 2024-04-01T01:38:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.345+0000 7f3e397c7640 1 --2- 172.21.15.70:0/636383280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e34109d40 0x7f3e3410a1c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.345+0000 7f3e397c7640 1 --2- 172.21.15.70:0/636383280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e34108ff0 0x7f3e34109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.345+0000 7f3e397c7640 1 --2- 172.21.15.70:0/636383280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e34072dc0 0x7f3e34108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.345+0000 7f3e397c7640 1 -- 172.21.15.70:0/636383280 >> 172.21.15.70:0/636383280 conn(0x7f3e3406e950 msgr2=0x7f3e34070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.345+0000 7f3e397c7640 1 -- 172.21.15.70:0/636383280 shutdown_connections 2024-04-01T01:38:46.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.346+0000 7f3e397c7640 1 -- 172.21.15.70:0/636383280 wait complete. 2024-04-01T01:38:46.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.346+0000 7f3e397c7640 1 Processor -- start 2024-04-01T01:38:46.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.346+0000 7f3e397c7640 1 -- start start 2024-04-01T01:38:46.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.347+0000 7f3e397c7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e34072dc0 0x7f3e341a6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.347+0000 7f3e397c7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e34108ff0 0x7f3e341a6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.347+0000 7f3e397c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e34109d40 0x7f3e341adb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.347+0000 7f3e397c7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3410d8e0 con 0x7f3e34072dc0 2024-04-01T01:38:46.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.347+0000 7f3e397c7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3410d760 con 0x7f3e34109d40 2024-04-01T01:38:46.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.347+0000 7f3e397c7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3410da60 con 0x7f3e34108ff0 2024-04-01T01:38:46.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.347+0000 7f3e337fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e34109d40 0x7f3e341adb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.347+0000 7f3e32ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e34072dc0 0x7f3e341a6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.347+0000 7f3e337fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e34109d40 0x7f3e341adb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48886/0 (socket says 172.21.15.70:48886) 2024-04-01T01:38:46.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.347+0000 7f3e337fe640 1 -- 172.21.15.70:0/1745305931 learned_addr learned my addr 172.21.15.70:0/1745305931 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:46.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.347+0000 7f3e2bfff640 1 --2- 172.21.15.70:0/1745305931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e34108ff0 0x7f3e341a6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.347+0000 7f3e337fe640 1 -- 172.21.15.70:0/1745305931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e34108ff0 msgr2=0x7f3e341a6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.348+0000 7f3e337fe640 1 --2- 172.21.15.70:0/1745305931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e34108ff0 0x7f3e341a6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.348+0000 7f3e337fe640 1 -- 172.21.15.70:0/1745305931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e34072dc0 msgr2=0x7f3e341a6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.348+0000 7f3e337fe640 1 --2- 172.21.15.70:0/1745305931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e34072dc0 0x7f3e341a6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.348+0000 7f3e337fe640 1 -- 172.21.15.70:0/1745305931 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e341ae180 con 0x7f3e34109d40 2024-04-01T01:38:46.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.348+0000 7f3e2bfff640 1 --2- 172.21.15.70:0/1745305931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e34108ff0 0x7f3e341a6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:38:46.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.348+0000 7f3e337fe640 1 --2- 172.21.15.70:0/1745305931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e34109d40 0x7f3e341adb10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c033940 tx=0x7f3e2c033cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.348+0000 7f3e30ff9640 1 -- 172.21.15.70:0/1745305931 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c043020 con 0x7f3e34109d40 2024-04-01T01:38:46.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.348+0000 7f3e30ff9640 1 -- 172.21.15.70:0/1745305931 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e2c033e20 con 0x7f3e34109d40 2024-04-01T01:38:46.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.348+0000 7f3e30ff9640 1 -- 172.21.15.70:0/1745305931 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c034690 con 0x7f3e34109d40 2024-04-01T01:38:46.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.349+0000 7f3e397c7640 1 -- 172.21.15.70:0/1745305931 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e341ae410 con 0x7f3e34109d40 2024-04-01T01:38:46.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.349+0000 7f3e397c7640 1 -- 172.21.15.70:0/1745305931 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e341ae850 con 0x7f3e34109d40 2024-04-01T01:38:46.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.351+0000 7f3e30ff9640 1 -- 172.21.15.70:0/1745305931 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f3e2c046020 con 0x7f3e34109d40 2024-04-01T01:38:46.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.352+0000 7f3e30ff9640 1 --2- 172.21.15.70:0/1745305931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3e200a2510 0x7f3e200a49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.352+0000 7f3e30ff9640 1 -- 172.21.15.70:0/1745305931 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3e2c0f13c0 con 0x7f3e34109d40 2024-04-01T01:38:46.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.352+0000 7f3e3c3a5740 1 Processor -- start 2024-04-01T01:38:46.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.352+0000 7f3e3c3a5740 1 -- start start 2024-04-01T01:38:46.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.352+0000 7f3e32ffd640 1 --2- 172.21.15.70:0/1745305931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3e200a2510 0x7f3e200a49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e3c3a5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01490 0x556400f018b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e3c3a5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01e80 0x556400f06980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e3c3a5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f06ff0 0x556400f09480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e3c3a5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556400e661b0 con 0x556400f06ff0 2024-04-01T01:38:46.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e3c3a5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x556400e0ec00 con 0x556400f01e80 2024-04-01T01:38:46.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e3c3a5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556400e62250 con 0x556400f01490 2024-04-01T01:38:46.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e32ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01490 0x556400f018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e337fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f06ff0 0x556400f09480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e337fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f06ff0 0x556400f09480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45092/0 (socket says 172.21.15.70:45092) 2024-04-01T01:38:46.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e337fe640 1 -- 172.21.15.70:0/935956530 learned_addr learned my addr 172.21.15.70:0/935956530 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:46.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e2bfff640 1 --2- 172.21.15.70:0/935956530 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01e80 0x556400f06980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e32ffd640 1 --2- 172.21.15.70:0/1745305931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3e200a2510 0x7f3e200a49d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3e24005fd0 tx=0x7f3e24009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e32ffd640 1 -- 172.21.15.70:0/935956530 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01e80 msgr2=0x556400f06980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e32ffd640 1 --2- 172.21.15.70:0/935956530 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01e80 0x556400f06980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e32ffd640 1 -- 172.21.15.70:0/935956530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f06ff0 msgr2=0x556400f09480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e32ffd640 1 --2- 172.21.15.70:0/935956530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f06ff0 0x556400f09480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.353+0000 7f3e32ffd640 1 -- 172.21.15.70:0/935956530 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556400e2e3d0 con 0x556400f01490 2024-04-01T01:38:46.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.354+0000 7f3e337fe640 1 --2- 172.21.15.70:0/935956530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f06ff0 0x556400f09480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:38:46.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.354+0000 7f3e32ffd640 1 --2- 172.21.15.70:0/935956530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01490 0x556400f018b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3e240099f0 tx=0x7f3e2401c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9271f475a50b2c39 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.354+0000 7f3e297fa640 1 -- 172.21.15.70:0/935956530 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e24004230 con 0x556400f01490 2024-04-01T01:38:46.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.354+0000 7f3e297fa640 1 -- 172.21.15.70:0/935956530 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e2401ce10 con 0x556400f01490 2024-04-01T01:38:46.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.354+0000 7f3e297fa640 1 -- 172.21.15.70:0/935956530 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e24004c60 con 0x556400f01490 2024-04-01T01:38:46.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.355+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/935956530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01490 msgr2=0x556400f018b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.355+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/935956530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01490 0x556400f018b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3e240099f0 tx=0x7f3e2401c000 comp rx=0 tx=0).stop 2024-04-01T01:38:46.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.355+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/935956530 shutdown_connections 2024-04-01T01:38:46.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.355+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/935956530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f06ff0 0x556400f09480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.355+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/935956530 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01e80 0x556400f06980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.355+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/935956530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01490 0x556400f018b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.355+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/935956530 >> 172.21.15.70:0/935956530 conn(0x556400efca00 msgr2=0x556400efce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.355+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/935956530 shutdown_connections 2024-04-01T01:38:46.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.356+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/935956530 wait complete. 2024-04-01T01:38:46.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.356+0000 7f3e3c3a5740 1 -- start start 2024-04-01T01:38:46.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.356+0000 7f3e3c3a5740 1 Processor -- start 2024-04-01T01:38:46.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e3c3a5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f01490 0x556400f0b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e3c3a5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01e80 0x556400f0bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e3c3a5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f06ff0 0x556400f12f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e3c3a5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556400f09fa0 con 0x556400f01490 2024-04-01T01:38:46.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e3c3a5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x556400e04470 con 0x556400f06ff0 2024-04-01T01:38:46.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e3c3a5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556400f0a120 con 0x556400f01e80 2024-04-01T01:38:46.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e3c3a5740 10 client.0 _ll_register_callbacks cb 0x556400e21cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:46.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e337fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f06ff0 0x556400f12f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e2bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01e80 0x556400f0bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e337fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f06ff0 0x556400f12f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48902/0 (socket says 172.21.15.70:48902) 2024-04-01T01:38:46.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e337fe640 1 -- 172.21.15.70:0/261902879 learned_addr learned my addr 172.21.15.70:0/261902879 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:46.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e32ffd640 1 --2- 172.21.15.70:0/261902879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f01490 0x556400f0b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e337fe640 1 -- 172.21.15.70:0/261902879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01e80 msgr2=0x556400f0bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e337fe640 1 --2- 172.21.15.70:0/261902879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01e80 0x556400f0bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e337fe640 1 -- 172.21.15.70:0/261902879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f01490 msgr2=0x556400f0b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e337fe640 1 --2- 172.21.15.70:0/261902879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f01490 0x556400f0b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.357+0000 7f3e337fe640 1 -- 172.21.15.70:0/261902879 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556400b7a3f0 con 0x556400f06ff0 2024-04-01T01:38:46.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.358+0000 7f3e337fe640 1 --2- 172.21.15.70:0/261902879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f06ff0 0x556400f12f30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0eea90 tx=0x7f3e2c0bd590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69ceb7aaddbf4d35 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.358+0000 7f3e1a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:46.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.358+0000 7f3e1a7fc640 1 -- 172.21.15.70:0/261902879 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c0d6060 con 0x556400f06ff0 2024-04-01T01:38:46.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.358+0000 7f3e1a7fc640 1 -- 172.21.15.70:0/261902879 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3e2c0f1160 con 0x556400f06ff0 2024-04-01T01:38:46.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.358+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/261902879 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e2401ce10 con 0x556400f06ff0 2024-04-01T01:38:46.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.359+0000 7f3e1a7fc640 1 -- 172.21.15.70:0/261902879 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3e2c0be150 con 0x556400f06ff0 2024-04-01T01:38:46.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.359+0000 7f3e3c3a5740 10 client.27035 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:46.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.359+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/261902879 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556400c447d0 con 0x556400f06ff0 2024-04-01T01:38:46.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.359+0000 7f3e1a7fc640 1 -- 172.21.15.70:0/261902879 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e2401ce10 con 0x556400f06ff0 2024-04-01T01:38:46.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.359+0000 7f3e1a7fc640 1 -- 172.21.15.70:0/261902879 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3e2c0bcea0 con 0x556400f06ff0 2024-04-01T01:38:46.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.359+0000 7f3e3c3a5740 10 client.27035 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:46.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.359+0000 7f3e3c3a5740 10 client.27035 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:38:46.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.359+0000 7f3e3c3a5740 20 client.27035 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:46.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.359+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/261902879 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x556400f17330 0x556400f197b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.359+0000 7f3e3c3a5740 4 client.27035 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:46.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.359+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/261902879 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556400e736f0 con 0x556400f17330 2024-04-01T01:38:46.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.360+0000 7f3e32ffd640 1 --2- 172.21.15.70:0/261902879 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x556400f17330 0x556400f197b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.360+0000 7f3e1a7fc640 10 client.27035 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:38:46.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.360+0000 7f3e32ffd640 1 --2- 172.21.15.70:0/261902879 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x556400f17330 0x556400f197b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4a1ccda6a990f83f server_cookie=4213f63150c1878a in_seq=0 out_seq=0 2024-04-01T01:38:46.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.361+0000 7f3e1a7fc640 1 -- 172.21.15.70:0/261902879 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556400e736f0 con 0x556400f17330 2024-04-01T01:38:46.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.361+0000 7f3e1a7fc640 10 client.27035 handle_command_reply: tid=0 2024-04-01T01:38:46.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.361+0000 7f3e3c3a5740 1 client.27035 shutdown 2024-04-01T01:38:46.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.361+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/261902879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f06ff0 msgr2=0x556400f12f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.361+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/261902879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f06ff0 0x556400f12f30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0eea90 tx=0x7f3e2c0bd590 comp rx=0 tx=0).stop 2024-04-01T01:38:46.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.361+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/261902879 shutdown_connections 2024-04-01T01:38:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.361+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/261902879 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x556400f17330 0x556400f197b0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.362+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/261902879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f06ff0 0x556400f12f30 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.362+0000 7f3e1a7fc640 0 client.27035 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:38:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.362+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/261902879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01e80 0x556400f0bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.362+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/261902879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f01490 0x556400f0b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.362+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/261902879 >> 172.21.15.70:0/261902879 conn(0x556400efca00 msgr2=0x556400efbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.362+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/261902879 shutdown_connections 2024-04-01T01:38:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.362+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/261902879 wait complete. 2024-04-01T01:38:46.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.362+0000 7f3e3c3a5740 20 client.27035 trim_cache size 0 max 16384 2024-04-01T01:38:46.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.386+0000 7f3e3c3a5740 1 Processor -- start 2024-04-01T01:38:46.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.386+0000 7f3e3c3a5740 1 -- start start 2024-04-01T01:38:46.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.387+0000 7f3e3c3a5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01490 0x556400f00370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.387+0000 7f3e3c3a5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01e80 0x556400f12ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.387+0000 7f3e3c3a5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f06ff0 0x556400f1e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.387+0000 7f3e337fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f06ff0 0x556400f1e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.387+0000 7f3e2bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01e80 0x556400f12ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.387+0000 7f3e32ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01490 0x556400f00370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.387+0000 7f3e2bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01e80 0x556400f12ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48908/0 (socket says 172.21.15.70:48908) 2024-04-01T01:38:46.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.387+0000 7f3e32ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01490 0x556400f00370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:54820/0 (socket says 172.21.15.70:54820) 2024-04-01T01:38:46.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.387+0000 7f3e337fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f06ff0 0x556400f1e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45106/0 (socket says 172.21.15.70:45106) 2024-04-01T01:38:46.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.387+0000 7f3e2bfff640 1 -- 172.21.15.70:0/1534481404 learned_addr learned my addr 172.21.15.70:0/1534481404 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:46.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.387+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/1534481404 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341b9020 con 0x556400f06ff0 2024-04-01T01:38:46.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.387+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/1534481404 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341bab60 con 0x556400f01e80 2024-04-01T01:38:46.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.387+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/1534481404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341b96f0 con 0x556400f01490 2024-04-01T01:38:46.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.388+0000 7f3e2bfff640 1 -- 172.21.15.70:0/1534481404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01490 msgr2=0x556400f00370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.388+0000 7f3e2bfff640 1 --2- 172.21.15.70:0/1534481404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01490 0x556400f00370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.388+0000 7f3e2bfff640 1 -- 172.21.15.70:0/1534481404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f06ff0 msgr2=0x556400f1e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.388+0000 7f3e2bfff640 1 --2- 172.21.15.70:0/1534481404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f06ff0 0x556400f1e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.388+0000 7f3e2bfff640 1 -- 172.21.15.70:0/1534481404 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556400f09e80 con 0x556400f01e80 2024-04-01T01:38:46.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.388+0000 7f3e2bfff640 1 --2- 172.21.15.70:0/1534481404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01e80 0x556400f12ef0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009c30 tx=0x7f3e1c013830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb77f57af34eaa39 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.388+0000 7f3e28ff9640 1 -- 172.21.15.70:0/1534481404 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c009d20 con 0x556400f01e80 2024-04-01T01:38:46.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.388+0000 7f3e28ff9640 1 -- 172.21.15.70:0/1534481404 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e1c0035f0 con 0x556400f01e80 2024-04-01T01:38:46.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.388+0000 7f3e28ff9640 1 -- 172.21.15.70:0/1534481404 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c0116a0 con 0x556400f01e80 2024-04-01T01:38:46.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.389+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/1534481404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01e80 msgr2=0x556400f12ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.390+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/1534481404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01e80 0x556400f12ef0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009c30 tx=0x7f3e1c013830 comp rx=0 tx=0).stop 2024-04-01T01:38:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.390+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/1534481404 shutdown_connections 2024-04-01T01:38:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.390+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/1534481404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f06ff0 0x556400f1e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.390+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/1534481404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01e80 0x556400f12ef0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.390+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/1534481404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f01490 0x556400f00370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.390+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/1534481404 >> 172.21.15.70:0/1534481404 conn(0x556400efca00 msgr2=0x556400effc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.390+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/1534481404 shutdown_connections 2024-04-01T01:38:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.390+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/1534481404 wait complete. 2024-04-01T01:38:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.391+0000 7f3e3c3a5740 1 -- start start 2024-04-01T01:38:46.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.391+0000 7f3e3c3a5740 1 Processor -- start 2024-04-01T01:38:46.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.391+0000 7f3e3c3a5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01490 0x556400f10e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.391+0000 7f3e3c3a5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f01e80 0x556400f113b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.391+0000 7f3e3c3a5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f06ff0 0x556400f123c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e3c3a5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341ba490 con 0x556400f01e80 2024-04-01T01:38:46.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e3c3a5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341b9dc0 con 0x556400f01490 2024-04-01T01:38:46.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e3c3a5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341b7000 con 0x556400f06ff0 2024-04-01T01:38:46.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e3c3a5740 10 client.0 _ll_register_callbacks cb 0x7f3e2c0b9de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:46.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e32ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01490 0x556400f10e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e32ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01490 0x556400f10e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48922/0 (socket says 172.21.15.70:48922) 2024-04-01T01:38:46.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e32ffd640 1 -- 172.21.15.70:0/200215141 learned_addr learned my addr 172.21.15.70:0/200215141 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:46.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e2bfff640 1 --2- 172.21.15.70:0/200215141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f01e80 0x556400f113b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e337fe640 1 --2- 172.21.15.70:0/200215141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f06ff0 0x556400f123c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e32ffd640 1 -- 172.21.15.70:0/200215141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f06ff0 msgr2=0x556400f123c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e32ffd640 1 --2- 172.21.15.70:0/200215141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f06ff0 0x556400f123c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e32ffd640 1 -- 172.21.15.70:0/200215141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f01e80 msgr2=0x556400f113b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e32ffd640 1 --2- 172.21.15.70:0/200215141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f01e80 0x556400f113b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e32ffd640 1 -- 172.21.15.70:0/200215141 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556400f129b0 con 0x556400f01490 2024-04-01T01:38:46.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e2bfff640 1 --2- 172.21.15.70:0/200215141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f01e80 0x556400f113b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:38:46.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e32ffd640 1 --2- 172.21.15.70:0/200215141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01490 0x556400f10e70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3e2401e010 tx=0x7f3e24005e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73956acf5edacd49 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e1a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:46.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.392+0000 7f3e1a7fc640 1 -- 172.21.15.70:0/200215141 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2401ce10 con 0x556400f01490 2024-04-01T01:38:46.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.393+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/200215141 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e1c0035f0 con 0x556400f01490 2024-04-01T01:38:46.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.393+0000 7f3e1a7fc640 1 -- 172.21.15.70:0/200215141 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3e24004040 con 0x556400f01490 2024-04-01T01:38:46.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.393+0000 7f3e1a7fc640 1 -- 172.21.15.70:0/200215141 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3e24049050 con 0x556400f01490 2024-04-01T01:38:46.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.393+0000 7f3e3c3a5740 10 client.27041 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:46.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.393+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/200215141 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556400f7df50 con 0x556400f01490 2024-04-01T01:38:46.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.393+0000 7f3e1a7fc640 1 -- 172.21.15.70:0/200215141 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e1c0035f0 con 0x556400f01490 2024-04-01T01:38:46.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.393+0000 7f3e1a7fc640 1 -- 172.21.15.70:0/200215141 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3e24023070 con 0x556400f01490 2024-04-01T01:38:46.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.393+0000 7f3e3c3a5740 10 client.27041 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:46.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.393+0000 7f3e3c3a5740 10 client.27041 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:38:46.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.393+0000 7f3e3c3a5740 20 client.27041 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:46.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.394+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/200215141 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x556400f17330 0x556400f806a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.394+0000 7f3e3c3a5740 4 client.27041 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:38:46.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.394+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/200215141 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556400e6da40 con 0x556400f17330 2024-04-01T01:38:46.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.394+0000 7f3e2bfff640 1 --2- 172.21.15.70:0/200215141 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x556400f17330 0x556400f806a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.394+0000 7f3e1a7fc640 10 client.27041 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:38:46.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.394+0000 7f3e2bfff640 1 --2- 172.21.15.70:0/200215141 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x556400f17330 0x556400f806a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b07536a97cdd4bb0 server_cookie=3cb8bfa497799648 in_seq=0 out_seq=0 2024-04-01T01:38:46.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.395+0000 7f3e1a7fc640 1 -- 172.21.15.70:0/200215141 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556400e6da40 con 0x556400f17330 2024-04-01T01:38:46.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.395+0000 7f3e1a7fc640 10 client.27041 handle_command_reply: tid=0 2024-04-01T01:38:46.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.395+0000 7f3e3c3a5740 1 client.27041 shutdown 2024-04-01T01:38:46.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.395+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/200215141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01490 msgr2=0x556400f10e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.395+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/200215141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01490 0x556400f10e70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3e2401e010 tx=0x7f3e24005e60 comp rx=0 tx=0).stop 2024-04-01T01:38:46.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.395+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/200215141 shutdown_connections 2024-04-01T01:38:46.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.395+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/200215141 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x556400f17330 0x556400f806a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.395+0000 7f3e1a7fc640 0 client.27041 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:38:46.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.395+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/200215141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556400f06ff0 0x556400f123c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.395+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/200215141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556400f01e80 0x556400f113b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.395+0000 7f3e3c3a5740 1 --2- 172.21.15.70:0/200215141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556400f01490 0x556400f10e70 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.395+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/200215141 >> 172.21.15.70:0/200215141 conn(0x556400efca00 msgr2=0x556400f004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.395+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/200215141 shutdown_connections 2024-04-01T01:38:46.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.396+0000 7f3e3c3a5740 1 -- 172.21.15.70:0/200215141 wait complete. 2024-04-01T01:38:46.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.396+0000 7f3e3c3a5740 20 client.27041 trim_cache size 0 max 16384 2024-04-01T01:38:46.404 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:38:46.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.397+0000 7f3e397c7640 1 -- 172.21.15.70:0/1745305931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3e200a2510 msgr2=0x7f3e200a49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.397+0000 7f3e397c7640 1 --2- 172.21.15.70:0/1745305931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3e200a2510 0x7f3e200a49d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3e24005fd0 tx=0x7f3e24009260 comp rx=0 tx=0).stop 2024-04-01T01:38:46.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.397+0000 7f3e397c7640 1 -- 172.21.15.70:0/1745305931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e34109d40 msgr2=0x7f3e341adb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.397+0000 7f3e397c7640 1 --2- 172.21.15.70:0/1745305931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e34109d40 0x7f3e341adb10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c033940 tx=0x7f3e2c033cb0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.397+0000 7f3e397c7640 1 -- 172.21.15.70:0/1745305931 shutdown_connections 2024-04-01T01:38:46.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.397+0000 7f3e397c7640 1 --2- 172.21.15.70:0/1745305931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e34109d40 0x7f3e341adb10 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.397+0000 7f3e397c7640 1 --2- 172.21.15.70:0/1745305931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3e200a2510 0x7f3e200a49d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.397+0000 7f3e397c7640 1 --2- 172.21.15.70:0/1745305931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e34108ff0 0x7f3e341a6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.397+0000 7f3e397c7640 1 --2- 172.21.15.70:0/1745305931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e34072dc0 0x7f3e341a6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.397+0000 7f3e397c7640 1 -- 172.21.15.70:0/1745305931 >> 172.21.15.70:0/1745305931 conn(0x7f3e3406e950 msgr2=0x7f3e3406fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.397+0000 7f3e397c7640 1 -- 172.21.15.70:0/1745305931 shutdown_connections 2024-04-01T01:38:46.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.397+0000 7f3e397c7640 1 -- 172.21.15.70:0/1745305931 wait complete. 2024-04-01T01:38:46.416 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:38:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.599+0000 7fe8c8455640 1 -- 172.21.15.70:0/1501930616 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8c0101760 msgr2=0x7fe8c0101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.599+0000 7fe8c8455640 1 --2- 172.21.15.70:0/1501930616 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8c0101760 0x7fe8c0101b80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc009900 tx=0x7fe8bc02bc40 comp rx=0 tx=0).stop 2024-04-01T01:38:46.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.600+0000 7fe8c8455640 1 -- 172.21.15.70:0/1501930616 shutdown_connections 2024-04-01T01:38:46.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.600+0000 7fe8c8455640 1 --2- 172.21.15.70:0/1501930616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c0103310 0x7fe8c0109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.600+0000 7fe8c8455640 1 --2- 172.21.15.70:0/1501930616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c0102950 0x7fe8c0102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.600+0000 7fe8c8455640 1 --2- 172.21.15.70:0/1501930616 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8c0101760 0x7fe8c0101b80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.600+0000 7fe8c8455640 1 -- 172.21.15.70:0/1501930616 >> 172.21.15.70:0/1501930616 conn(0x7fe8c00fccf0 msgr2=0x7fe8c00ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.600+0000 7fe8c8455640 1 -- 172.21.15.70:0/1501930616 shutdown_connections 2024-04-01T01:38:46.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.600+0000 7fe8c8455640 1 -- 172.21.15.70:0/1501930616 wait complete. 2024-04-01T01:38:46.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.601+0000 7fe8c8455640 1 Processor -- start 2024-04-01T01:38:46.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.601+0000 7fe8c8455640 1 -- start start 2024-04-01T01:38:46.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.601+0000 7fe8c8455640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8c0101760 0x7fe8c00700e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.602+0000 7fe8c8455640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c0102950 0x7fe8c0070620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.602+0000 7fe8c8455640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c0103310 0x7fe8c01a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.602+0000 7fe8c8455640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c00750b0 con 0x7fe8c0103310 2024-04-01T01:38:46.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.602+0000 7fe8c8455640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c0074f30 con 0x7fe8c0101760 2024-04-01T01:38:46.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.602+0000 7fe8c8455640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c0075230 con 0x7fe8c0102950 2024-04-01T01:38:46.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.602+0000 7fe8c59f0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c0102950 0x7fe8c0070620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.602+0000 7fe8c59f0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c0102950 0x7fe8c0070620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:54854/0 (socket says 172.21.15.70:54854) 2024-04-01T01:38:46.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.602+0000 7fe8c59f0640 1 -- 172.21.15.70:0/3614612046 learned_addr learned my addr 172.21.15.70:0/3614612046 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:46.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.602+0000 7fe8c69f2640 1 --2- 172.21.15.70:0/3614612046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c0103310 0x7fe8c01a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.602+0000 7fe8c59f0640 1 -- 172.21.15.70:0/3614612046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8c0101760 msgr2=0x7fe8c00700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.603+0000 7fe8c59f0640 1 --2- 172.21.15.70:0/3614612046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8c0101760 0x7fe8c00700e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.603+0000 7fe8c59f0640 1 -- 172.21.15.70:0/3614612046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c0103310 msgr2=0x7fe8c01a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.603+0000 7fe8c59f0640 1 --2- 172.21.15.70:0/3614612046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c0103310 0x7fe8c01a9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.603+0000 7fe8c59f0640 1 -- 172.21.15.70:0/3614612046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c0070cc0 con 0x7fe8c0102950 2024-04-01T01:38:46.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.603+0000 7fe8c59f0640 1 --2- 172.21.15.70:0/3614612046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c0102950 0x7fe8c0070620 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000e9f0 tx=0x7fe8b000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.603+0000 7fe8b77fe640 1 -- 172.21.15.70:0/3614612046 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b000cd80 con 0x7fe8c0102950 2024-04-01T01:38:46.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.603+0000 7fe8c8455640 1 -- 172.21.15.70:0/3614612046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8c01aa3b0 con 0x7fe8c0102950 2024-04-01T01:38:46.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.603+0000 7fe8c8455640 1 -- 172.21.15.70:0/3614612046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8c01aa950 con 0x7fe8c0102950 2024-04-01T01:38:46.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.604+0000 7fe8b77fe640 1 -- 172.21.15.70:0/3614612046 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8b00108e0 con 0x7fe8c0102950 2024-04-01T01:38:46.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.605+0000 7fe8b77fe640 1 -- 172.21.15.70:0/3614612046 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0018880 con 0x7fe8c0102950 2024-04-01T01:38:46.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.608+0000 7fe8b77fe640 1 -- 172.21.15.70:0/3614612046 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fe8b0018a20 con 0x7fe8c0102950 2024-04-01T01:38:46.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.610+0000 7fe8cb036740 1 Processor -- start 2024-04-01T01:38:46.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.610+0000 7fe8cb036740 1 -- start start 2024-04-01T01:38:46.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.610+0000 7fe8cb036740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3b1ca0 0x55745f3b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.610+0000 7fe8cb036740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b2690 0x55745f3b7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.610+0000 7fe8cb036740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b7800 0x55745f3b9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.610+0000 7fe8c69f2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3b1ca0 0x55745f3b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.610+0000 7fe8c69f2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3b1ca0 0x55745f3b20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:54864/0 (socket says 172.21.15.70:54864) 2024-04-01T01:38:46.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.610+0000 7fe8c69f2640 1 -- 172.21.15.70:0/2988019440 learned_addr learned my addr 172.21.15.70:0/2988019440 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:46.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.611+0000 7fe8c61f1640 1 --2- 172.21.15.70:0/2988019440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b2690 0x55745f3b7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.610+0000 7fe8cb036740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55745f30e1b0 con 0x55745f3b2690 2024-04-01T01:38:46.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.611+0000 7fe8cb036740 1 -- 172.21.15.70:0/2988019440 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55745f2b6c00 con 0x55745f3b7800 2024-04-01T01:38:46.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.611+0000 7fe8c59f0640 1 --2- 172.21.15.70:0/2988019440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b7800 0x55745f3b9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.611+0000 7fe8cb036740 1 -- 172.21.15.70:0/2988019440 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55745f30a250 con 0x55745f3b1ca0 2024-04-01T01:38:46.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.611+0000 7fe8b77fe640 1 --2- 172.21.15.70:0/3614612046 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8ac0a2670 0x7fe8ac0a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.611+0000 7fe8c69f2640 1 --2- 172.21.15.70:0/3614612046 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8ac0a2670 0x7fe8ac0a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.611+0000 7fe8c59f0640 1 -- 172.21.15.70:0/2988019440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3b1ca0 msgr2=0x55745f3b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.611+0000 7fe8c59f0640 1 --2- 172.21.15.70:0/2988019440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3b1ca0 0x55745f3b20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.611+0000 7fe8c59f0640 1 -- 172.21.15.70:0/2988019440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b2690 msgr2=0x55745f3b7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.611+0000 7fe8c59f0640 1 --2- 172.21.15.70:0/2988019440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b2690 0x55745f3b7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.611+0000 7fe8c59f0640 1 -- 172.21.15.70:0/2988019440 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55745f2d63d0 con 0x55745f3b7800 2024-04-01T01:38:46.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.611+0000 7fe8b77fe640 1 -- 172.21.15.70:0/3614612046 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe8b00cfe60 con 0x7fe8c0102950 2024-04-01T01:38:46.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.611+0000 7fe8c59f0640 1 --2- 172.21.15.70:0/2988019440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b7800 0x55745f3b9c90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000c6a0 tx=0x7fe8b0013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15d54b0a7510f92 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.612+0000 7fe8b4bfe640 1 -- 172.21.15.70:0/2988019440 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0013190 con 0x55745f3b7800 2024-04-01T01:38:46.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.612+0000 7fe8b4bfe640 1 -- 172.21.15.70:0/2988019440 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8b009cbf0 con 0x55745f3b7800 2024-04-01T01:38:46.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.612+0000 7fe8b4bfe640 1 -- 172.21.15.70:0/2988019440 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b001ce80 con 0x55745f3b7800 2024-04-01T01:38:46.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.612+0000 7fe8c69f2640 1 --2- 172.21.15.70:0/3614612046 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8ac0a2670 0x7fe8ac0a4b30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x55745f3b2ef0 tx=0x7fe8b8004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.613+0000 7fe8cb036740 1 -- 172.21.15.70:0/2988019440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b7800 msgr2=0x55745f3b9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.613+0000 7fe8cb036740 1 --2- 172.21.15.70:0/2988019440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b7800 0x55745f3b9c90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000c6a0 tx=0x7fe8b0013000 comp rx=0 tx=0).stop 2024-04-01T01:38:46.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.613+0000 7fe8cb036740 1 -- 172.21.15.70:0/2988019440 shutdown_connections 2024-04-01T01:38:46.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.613+0000 7fe8cb036740 1 --2- 172.21.15.70:0/2988019440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b7800 0x55745f3b9c90 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.613+0000 7fe8cb036740 1 --2- 172.21.15.70:0/2988019440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b2690 0x55745f3b7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.613+0000 7fe8cb036740 1 --2- 172.21.15.70:0/2988019440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3b1ca0 0x55745f3b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.613+0000 7fe8cb036740 1 -- 172.21.15.70:0/2988019440 >> 172.21.15.70:0/2988019440 conn(0x55745f3ad210 msgr2=0x55745f3ad610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.613+0000 7fe8cb036740 1 -- 172.21.15.70:0/2988019440 shutdown_connections 2024-04-01T01:38:46.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.614+0000 7fe8cb036740 1 -- 172.21.15.70:0/2988019440 wait complete. 2024-04-01T01:38:46.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.614+0000 7fe8cb036740 1 -- start start 2024-04-01T01:38:46.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.614+0000 7fe8cb036740 1 Processor -- start 2024-04-01T01:38:46.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.614+0000 7fe8cb036740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3b1ca0 0x55745f3bbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.615+0000 7fe8cb036740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b2690 0x55745f3bc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.615+0000 7fe8cb036740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b7800 0x55745f3c3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.615+0000 7fe8c61f1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3b1ca0 0x55745f3bbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.615+0000 7fe8c61f1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3b1ca0 0x55745f3bbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46618/0 (socket says 172.21.15.70:46618) 2024-04-01T01:38:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.615+0000 7fe8c61f1640 1 -- 172.21.15.70:0/399546860 learned_addr learned my addr 172.21.15.70:0/399546860 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.615+0000 7fe8cb036740 1 -- 172.21.15.70:0/399546860 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55745f3ba750 con 0x55745f3b2690 2024-04-01T01:38:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.615+0000 7fe8cb036740 1 -- 172.21.15.70:0/399546860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55745f2ac470 con 0x55745f3b7800 2024-04-01T01:38:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.615+0000 7fe8cb036740 1 -- 172.21.15.70:0/399546860 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55745f3ba8d0 con 0x55745f3b1ca0 2024-04-01T01:38:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.615+0000 7fe8c59f0640 1 --2- 172.21.15.70:0/399546860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b2690 0x55745f3bc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.615+0000 7fe8c69f2640 1 --2- 172.21.15.70:0/399546860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b7800 0x55745f3c3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.615+0000 7fe8cb036740 10 client.0 _ll_register_callbacks cb 0x55745f2c9cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.615+0000 7fe8c69f2640 1 -- 172.21.15.70:0/399546860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3b1ca0 msgr2=0x55745f3bbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.615+0000 7fe8c69f2640 1 --2- 172.21.15.70:0/399546860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3b1ca0 0x55745f3bbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.615+0000 7fe8c69f2640 1 -- 172.21.15.70:0/399546860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b2690 msgr2=0x55745f3bc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.615+0000 7fe8c69f2640 1 --2- 172.21.15.70:0/399546860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b2690 0x55745f3bc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.615+0000 7fe8c69f2640 1 -- 172.21.15.70:0/399546860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55745f0223f0 con 0x55745f3b7800 2024-04-01T01:38:46.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.616+0000 7fe8c69f2640 1 --2- 172.21.15.70:0/399546860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b7800 0x55745f3c3480 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe8b8009fd0 tx=0x7fe8b8004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd7145113919c1aa server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.616+0000 7fe8a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:46.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.617+0000 7fe8a9ffb640 1 -- 172.21.15.70:0/399546860 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b8015070 con 0x55745f3b7800 2024-04-01T01:38:46.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.617+0000 7fe8a9ffb640 1 -- 172.21.15.70:0/399546860 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe8b800ad60 con 0x55745f3b7800 2024-04-01T01:38:46.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.617+0000 7fe8cb036740 1 -- 172.21.15.70:0/399546860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8b009cbf0 con 0x55745f3b7800 2024-04-01T01:38:46.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.617+0000 7fe8a9ffb640 1 -- 172.21.15.70:0/399546860 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe8b8007e30 con 0x55745f3b7800 2024-04-01T01:38:46.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.617+0000 7fe8cb036740 10 client.27050 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:46.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.617+0000 7fe8cb036740 1 -- 172.21.15.70:0/399546860 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55745f0ec7d0 con 0x55745f3b7800 2024-04-01T01:38:46.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.618+0000 7fe8a9ffb640 1 -- 172.21.15.70:0/399546860 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8b8016040 con 0x55745f3b7800 2024-04-01T01:38:46.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.618+0000 7fe8a9ffb640 1 -- 172.21.15.70:0/399546860 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe8b80314c0 con 0x55745f3b7800 2024-04-01T01:38:46.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.619+0000 7fe8cb036740 10 client.27050 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:46.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.619+0000 7fe8cb036740 10 client.27050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:46.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.619+0000 7fe8cb036740 20 client.27050 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:46.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.619+0000 7fe8cb036740 1 --2- 172.21.15.70:0/399546860 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55745f3c7970 0x55745f3c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.619+0000 7fe8cb036740 4 client.27050 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:46.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.619+0000 7fe8cb036740 1 -- 172.21.15.70:0/399546860 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe8b009cbf0 con 0x55745f3c7970 2024-04-01T01:38:46.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.620+0000 7fe8c61f1640 1 --2- 172.21.15.70:0/399546860 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55745f3c7970 0x55745f3c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.621+0000 7fe8c61f1640 1 --2- 172.21.15.70:0/399546860 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55745f3c7970 0x55745f3c9db0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d36bcaf1460c943 server_cookie=84fddf5d5ab17bd7 in_seq=0 out_seq=0 2024-04-01T01:38:46.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.621+0000 7fe8a9ffb640 10 client.27050 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:46.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.625+0000 7fe8a9ffb640 1 -- 172.21.15.70:0/399546860 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fe8b009cbf0 con 0x55745f3c7970 2024-04-01T01:38:46.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.625+0000 7fe8a9ffb640 10 client.27050 handle_command_reply: tid=0 2024-04-01T01:38:46.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.625+0000 7fe8cb036740 1 client.27050 shutdown 2024-04-01T01:38:46.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.626+0000 7fe8cb036740 1 -- 172.21.15.70:0/399546860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b7800 msgr2=0x55745f3c3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.626+0000 7fe8cb036740 1 --2- 172.21.15.70:0/399546860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b7800 0x55745f3c3480 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe8b8009fd0 tx=0x7fe8b8004b50 comp rx=0 tx=0).stop 2024-04-01T01:38:46.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.628+0000 7fe8cb036740 1 -- 172.21.15.70:0/399546860 shutdown_connections 2024-04-01T01:38:46.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.628+0000 7fe8cb036740 1 --2- 172.21.15.70:0/399546860 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55745f3c7970 0x55745f3c9db0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.628+0000 7fe8cb036740 1 --2- 172.21.15.70:0/399546860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b7800 0x55745f3c3480 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.628+0000 7fe8cb036740 1 --2- 172.21.15.70:0/399546860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b2690 0x55745f3bc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.628+0000 7fe8cb036740 1 --2- 172.21.15.70:0/399546860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3b1ca0 0x55745f3bbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.628+0000 7fe8a9ffb640 0 client.27050 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:46.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.628+0000 7fe8cb036740 1 -- 172.21.15.70:0/399546860 >> 172.21.15.70:0/399546860 conn(0x55745f3ad210 msgr2=0x55745f3a3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.628+0000 7fe8cb036740 1 -- 172.21.15.70:0/399546860 shutdown_connections 2024-04-01T01:38:46.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.628+0000 7fe8cb036740 1 -- 172.21.15.70:0/399546860 wait complete. 2024-04-01T01:38:46.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.629+0000 7fe8cb036740 20 client.27050 trim_cache size 0 max 16384 2024-04-01T01:38:46.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.655+0000 7fe8cb036740 1 Processor -- start 2024-04-01T01:38:46.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.655+0000 7fe8cb036740 1 -- start start 2024-04-01T01:38:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.656+0000 7fe8cb036740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3ad210 0x55745f3a34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.656+0000 7fe8cb036740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b1ca0 0x55745f3c33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.656+0000 7fe8cb036740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b2690 0x55745f3a39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.656+0000 7fe8cb036740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a400fdd0 con 0x55745f3b1ca0 2024-04-01T01:38:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.656+0000 7fe8cb036740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a4011910 con 0x55745f3b2690 2024-04-01T01:38:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.656+0000 7fe8cb036740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a40104a0 con 0x55745f3ad210 2024-04-01T01:38:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.656+0000 7fe8c59f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b1ca0 0x55745f3c33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.656+0000 7fe8c59f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b1ca0 0x55745f3c33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45306/0 (socket says 172.21.15.70:45306) 2024-04-01T01:38:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.656+0000 7fe8c59f0640 1 -- 172.21.15.70:0/2275140056 learned_addr learned my addr 172.21.15.70:0/2275140056 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.656+0000 7fe8c69f2640 1 --2- 172.21.15.70:0/2275140056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b2690 0x55745f3a39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.656+0000 7fe8c61f1640 1 --2- 172.21.15.70:0/2275140056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3ad210 0x55745f3a34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.656+0000 7fe8c59f0640 1 -- 172.21.15.70:0/2275140056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3ad210 msgr2=0x55745f3a34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.656+0000 7fe8c59f0640 1 --2- 172.21.15.70:0/2275140056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3ad210 0x55745f3a34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.656+0000 7fe8c59f0640 1 -- 172.21.15.70:0/2275140056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b2690 msgr2=0x55745f3a39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.656+0000 7fe8c59f0640 1 --2- 172.21.15.70:0/2275140056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b2690 0x55745f3a39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.656+0000 7fe8c59f0640 1 -- 172.21.15.70:0/2275140056 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55745f3b9c50 con 0x55745f3b1ca0 2024-04-01T01:38:46.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.660+0000 7fe8c59f0640 1 --2- 172.21.15.70:0/2275140056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b1ca0 0x55745f3c33a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe8b009c6b0 tx=0x7fe8b0098160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e9f5e4ec77ab97 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.661+0000 7fe8abfff640 1 -- 172.21.15.70:0/2275140056 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b001ce80 con 0x55745f3b1ca0 2024-04-01T01:38:46.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.661+0000 7fe8abfff640 1 -- 172.21.15.70:0/2275140056 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8b009cbf0 con 0x55745f3b1ca0 2024-04-01T01:38:46.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.661+0000 7fe8abfff640 1 -- 172.21.15.70:0/2275140056 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0098b60 con 0x55745f3b1ca0 2024-04-01T01:38:46.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.662+0000 7fe8cb036740 1 -- 172.21.15.70:0/2275140056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b1ca0 msgr2=0x55745f3c33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.662+0000 7fe8cb036740 1 --2- 172.21.15.70:0/2275140056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b1ca0 0x55745f3c33a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe8b009c6b0 tx=0x7fe8b0098160 comp rx=0 tx=0).stop 2024-04-01T01:38:46.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.666+0000 7fe8cb036740 1 -- 172.21.15.70:0/2275140056 shutdown_connections 2024-04-01T01:38:46.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.666+0000 7fe8cb036740 1 --2- 172.21.15.70:0/2275140056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3b2690 0x55745f3a39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.666+0000 7fe8cb036740 1 --2- 172.21.15.70:0/2275140056 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b1ca0 0x55745f3c33a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.666+0000 7fe8cb036740 1 --2- 172.21.15.70:0/2275140056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3ad210 0x55745f3a34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.666+0000 7fe8cb036740 1 -- 172.21.15.70:0/2275140056 >> 172.21.15.70:0/2275140056 conn(0x55745f3b3290 msgr2=0x55745f3b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.668+0000 7fe8cb036740 1 -- 172.21.15.70:0/2275140056 shutdown_connections 2024-04-01T01:38:46.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.668+0000 7fe8cb036740 1 -- 172.21.15.70:0/2275140056 wait complete. 2024-04-01T01:38:46.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.669+0000 7fe8cb036740 1 -- start start 2024-04-01T01:38:46.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.669+0000 7fe8cb036740 1 Processor -- start 2024-04-01T01:38:46.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.670+0000 7fe8cb036740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3ad210 0x55745f3c1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.670+0000 7fe8cb036740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3b1ca0 0x55745f3c1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.670+0000 7fe8cb036740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b2690 0x55745f3c2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.670+0000 7fe8cb036740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a4011240 con 0x55745f3b2690 2024-04-01T01:38:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.670+0000 7fe8c61f1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3ad210 0x55745f3c1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.670+0000 7fe8c61f1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3ad210 0x55745f3c1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49328/0 (socket says 172.21.15.70:49328) 2024-04-01T01:38:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.670+0000 7fe8c61f1640 1 -- 172.21.15.70:0/1114702950 learned_addr learned my addr 172.21.15.70:0/1114702950 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.670+0000 7fe8cb036740 1 -- 172.21.15.70:0/1114702950 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a4010b70 con 0x55745f3ad210 2024-04-01T01:38:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.670+0000 7fe8cb036740 1 -- 172.21.15.70:0/1114702950 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a400ddb0 con 0x55745f3b1ca0 2024-04-01T01:38:46.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.671+0000 7fe8cb036740 10 client.0 _ll_register_callbacks cb 0x7fe8b800c320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:46.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.671+0000 7fe8c69f2640 1 --2- 172.21.15.70:0/1114702950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b2690 0x55745f3c2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.671+0000 7fe8c61f1640 1 -- 172.21.15.70:0/1114702950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3b1ca0 msgr2=0x55745f3c1640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:38:46.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.671+0000 7fe8c61f1640 1 --2- 172.21.15.70:0/1114702950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3b1ca0 0x55745f3c1640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.671+0000 7fe8c61f1640 1 -- 172.21.15.70:0/1114702950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b2690 msgr2=0x55745f3c2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:46.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.671+0000 7fe8c61f1640 1 --2- 172.21.15.70:0/1114702950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b2690 0x55745f3c2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.671+0000 7fe8c61f1640 1 -- 172.21.15.70:0/1114702950 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55745f3c2c40 con 0x55745f3ad210 2024-04-01T01:38:46.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.671+0000 7fe8c69f2640 1 --2- 172.21.15.70:0/1114702950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b2690 0x55745f3c2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:38:46.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.671+0000 7fe8c61f1640 1 --2- 172.21.15.70:0/1114702950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3ad210 0x55745f3c1100 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc036040 tx=0x7fe8bc000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50ff7808244623fe server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:46.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.671+0000 7fe8a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:46.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.672+0000 7fe8a9ffb640 1 -- 172.21.15.70:0/1114702950 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8bc046c40 con 0x55745f3ad210 2024-04-01T01:38:46.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.672+0000 7fe8a9ffb640 1 -- 172.21.15.70:0/1114702950 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe8bc046de0 con 0x55745f3ad210 2024-04-01T01:38:46.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.672+0000 7fe8cb036740 1 -- 172.21.15.70:0/1114702950 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8b009cbf0 con 0x55745f3ad210 2024-04-01T01:38:46.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.672+0000 7fe8a9ffb640 1 -- 172.21.15.70:0/1114702950 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe8bc054050 con 0x55745f3ad210 2024-04-01T01:38:46.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.673+0000 7fe8cb036740 10 client.27056 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:46.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.673+0000 7fe8cb036740 1 -- 172.21.15.70:0/1114702950 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55745f42e360 con 0x55745f3ad210 2024-04-01T01:38:46.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.674+0000 7fe8a9ffb640 1 -- 172.21.15.70:0/1114702950 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8bc0462d0 con 0x55745f3ad210 2024-04-01T01:38:46.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.674+0000 7fe8a9ffb640 1 -- 172.21.15.70:0/1114702950 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fe8bc0051e0 con 0x55745f3ad210 2024-04-01T01:38:46.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.675+0000 7fe8cb036740 10 client.27056 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:46.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.675+0000 7fe8cb036740 10 client.27056 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:46.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.675+0000 7fe8cb036740 20 client.27056 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:46.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.675+0000 7fe8cb036740 1 --2- 172.21.15.70:0/1114702950 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55745f3b7800 0x55745f430d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:46.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.675+0000 7fe8cb036740 4 client.27056 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:38:46.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.675+0000 7fe8cb036740 1 -- 172.21.15.70:0/1114702950 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fe8b009cbf0 con 0x55745f3b7800 2024-04-01T01:38:46.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.677+0000 7fe8c59f0640 1 --2- 172.21.15.70:0/1114702950 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55745f3b7800 0x55745f430d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:46.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.680+0000 7fe8c59f0640 1 --2- 172.21.15.70:0/1114702950 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55745f3b7800 0x55745f430d70 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc480d794bbbdff1 server_cookie=fbfece0c3b1edd9 in_seq=0 out_seq=0 2024-04-01T01:38:46.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.681+0000 7fe8a9ffb640 10 client.27056 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:46.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.681+0000 7fe8a9ffb640 1 -- 172.21.15.70:0/1114702950 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fe8b009cbf0 con 0x55745f3b7800 2024-04-01T01:38:46.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.681+0000 7fe8a9ffb640 10 client.27056 handle_command_reply: tid=0 2024-04-01T01:38:46.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.682+0000 7fe8cb036740 1 client.27056 shutdown 2024-04-01T01:38:46.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.684+0000 7fe8cb036740 1 -- 172.21.15.70:0/1114702950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3ad210 msgr2=0x55745f3c1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.685+0000 7fe8cb036740 1 --2- 172.21.15.70:0/1114702950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3ad210 0x55745f3c1100 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc036040 tx=0x7fe8bc000bc0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.685+0000 7fe8cb036740 1 -- 172.21.15.70:0/1114702950 shutdown_connections 2024-04-01T01:38:46.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.685+0000 7fe8cb036740 1 --2- 172.21.15.70:0/1114702950 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55745f3b7800 0x55745f430d70 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.685+0000 7fe8cb036740 1 --2- 172.21.15.70:0/1114702950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55745f3b2690 0x55745f3c2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.685+0000 7fe8cb036740 1 --2- 172.21.15.70:0/1114702950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745f3b1ca0 0x55745f3c1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.685+0000 7fe8cb036740 1 --2- 172.21.15.70:0/1114702950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55745f3ad210 0x55745f3c1100 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.685+0000 7fe8cb036740 1 -- 172.21.15.70:0/1114702950 >> 172.21.15.70:0/1114702950 conn(0x55745f3b3290 msgr2=0x55745f3a34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.685+0000 7fe8a9ffb640 0 client.27056 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:46.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.685+0000 7fe8cb036740 1 -- 172.21.15.70:0/1114702950 shutdown_connections 2024-04-01T01:38:46.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.685+0000 7fe8cb036740 1 -- 172.21.15.70:0/1114702950 wait complete. 2024-04-01T01:38:46.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.685+0000 7fe8cb036740 20 client.27056 trim_cache size 0 max 16384 2024-04-01T01:38:46.686 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:38:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.686+0000 7fe8c8455640 1 -- 172.21.15.70:0/3614612046 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8ac0a2670 msgr2=0x7fe8ac0a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.686+0000 7fe8c8455640 1 --2- 172.21.15.70:0/3614612046 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8ac0a2670 0x7fe8ac0a4b30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x55745f3b2ef0 tx=0x7fe8b8004a40 comp rx=0 tx=0).stop 2024-04-01T01:38:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.686+0000 7fe8c8455640 1 -- 172.21.15.70:0/3614612046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c0102950 msgr2=0x7fe8c0070620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.687+0000 7fe8c8455640 1 --2- 172.21.15.70:0/3614612046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c0102950 0x7fe8c0070620 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000e9f0 tx=0x7fe8b000eeb0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.687+0000 7fe8c8455640 1 -- 172.21.15.70:0/3614612046 shutdown_connections 2024-04-01T01:38:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.687+0000 7fe8c8455640 1 --2- 172.21.15.70:0/3614612046 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8c0103310 0x7fe8c01a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.687+0000 7fe8c8455640 1 --2- 172.21.15.70:0/3614612046 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8ac0a2670 0x7fe8ac0a4b30 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.687+0000 7fe8c8455640 1 --2- 172.21.15.70:0/3614612046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8c0102950 0x7fe8c0070620 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.687+0000 7fe8c8455640 1 --2- 172.21.15.70:0/3614612046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8c0101760 0x7fe8c00700e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:46.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.687+0000 7fe8c8455640 1 -- 172.21.15.70:0/3614612046 >> 172.21.15.70:0/3614612046 conn(0x7fe8c00fccf0 msgr2=0x7fe8c0105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:46.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.687+0000 7fe8c8455640 1 -- 172.21.15.70:0/3614612046 shutdown_connections 2024-04-01T01:38:46.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:46.687+0000 7fe8c8455640 1 -- 172.21.15.70:0/3614612046 wait complete. 2024-04-01T01:38:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 240.0 s, 1035.9 tps, lat 96.692 ms stddev 45.063 2024-04-01T01:38:47.710 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:38:47.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:38:47.710 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:38:47.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.892+0000 7f626f202640 1 -- 172.21.15.70:0/2488971437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6268108af0 msgr2=0x7f6268108f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:47.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.892+0000 7f626f202640 1 --2- 172.21.15.70:0/2488971437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6268108af0 0x7f6268108f10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f6258009ca0 tx=0x7f625802c100 comp rx=0 tx=0).stop 2024-04-01T01:38:47.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.892+0000 7f626f202640 1 -- 172.21.15.70:0/2488971437 shutdown_connections 2024-04-01T01:38:47.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.892+0000 7f626f202640 1 --2- 172.21.15.70:0/2488971437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f626810a1a0 0x7f6268110a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.892+0000 7f626f202640 1 --2- 172.21.15.70:0/2488971437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6268109800 0x7f6268109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.892+0000 7f626f202640 1 --2- 172.21.15.70:0/2488971437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6268108af0 0x7f6268108f10 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.892+0000 7f626f202640 1 -- 172.21.15.70:0/2488971437 >> 172.21.15.70:0/2488971437 conn(0x7f626806e5b0 msgr2=0x7f62680709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:47.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.893+0000 7f626f202640 1 -- 172.21.15.70:0/2488971437 shutdown_connections 2024-04-01T01:38:47.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.893+0000 7f626f202640 1 -- 172.21.15.70:0/2488971437 wait complete. 2024-04-01T01:38:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.893+0000 7f626f202640 1 Processor -- start 2024-04-01T01:38:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f626f202640 1 -- start start 2024-04-01T01:38:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f626f202640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6268108af0 0x7f62681a6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f626f202640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6268109800 0x7f62681a6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f626f202640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f626810a1a0 0x7f62681adf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f626f202640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62681138c0 con 0x7f626810a1a0 2024-04-01T01:38:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f626f202640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6268113740 con 0x7f6268108af0 2024-04-01T01:38:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f626f202640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6268113a40 con 0x7f6268109800 2024-04-01T01:38:47.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f626cf9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6268108af0 0x7f62681a6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f626cf9e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6268108af0 0x7f62681a6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49336/0 (socket says 172.21.15.70:49336) 2024-04-01T01:38:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f626cf9e640 1 -- 172.21.15.70:0/2169270599 learned_addr learned my addr 172.21.15.70:0/2169270599 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f626d79f640 1 --2- 172.21.15.70:0/2169270599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f626810a1a0 0x7f62681adf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f6267fff640 1 --2- 172.21.15.70:0/2169270599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6268109800 0x7f62681a6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f626cf9e640 1 -- 172.21.15.70:0/2169270599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6268109800 msgr2=0x7f62681a6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f626cf9e640 1 --2- 172.21.15.70:0/2169270599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6268109800 0x7f62681a6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f626cf9e640 1 -- 172.21.15.70:0/2169270599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f626810a1a0 msgr2=0x7f62681adf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f626cf9e640 1 --2- 172.21.15.70:0/2169270599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f626810a1a0 0x7f62681adf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.894+0000 7f626cf9e640 1 -- 172.21.15.70:0/2169270599 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62681ae5a0 con 0x7f6268108af0 2024-04-01T01:38:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.895+0000 7f626cf9e640 1 --2- 172.21.15.70:0/2169270599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6268108af0 0x7f62681a6980 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6258006010 tx=0x7f6258003ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:47.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.895+0000 7f6265ffb640 1 -- 172.21.15.70:0/2169270599 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258041020 con 0x7f6268108af0 2024-04-01T01:38:47.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.895+0000 7f626f202640 1 -- 172.21.15.70:0/2169270599 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62681ae830 con 0x7f6268108af0 2024-04-01T01:38:47.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.895+0000 7f6265ffb640 1 -- 172.21.15.70:0/2169270599 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6258003e50 con 0x7f6268108af0 2024-04-01T01:38:47.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.895+0000 7f6265ffb640 1 -- 172.21.15.70:0/2169270599 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258039040 con 0x7f6268108af0 2024-04-01T01:38:47.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.896+0000 7f626f202640 1 -- 172.21.15.70:0/2169270599 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62681aed70 con 0x7f6268108af0 2024-04-01T01:38:47.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.899+0000 7f6271de7740 1 Processor -- start 2024-04-01T01:38:47.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.899+0000 7f6271de7740 1 -- start start 2024-04-01T01:38:47.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.899+0000 7f6271de7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da23005a10 0x55da23005e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.899+0000 7f6271de7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23006400 0x55da2300af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.899+0000 7f6271de7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da2300b570 0x55da2300da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.899+0000 7f6271de7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55da22f070b0 con 0x55da2300b570 2024-04-01T01:38:47.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.899+0000 7f6271de7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55da22f69ea0 con 0x55da23005a10 2024-04-01T01:38:47.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.899+0000 7f6271de7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55da22f12c00 con 0x55da23006400 2024-04-01T01:38:47.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.900+0000 7f626d79f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da23005a10 0x55da23005e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.900+0000 7f6267fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da2300b570 0x55da2300da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.900+0000 7f6267fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da2300b570 0x55da2300da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45330/0 (socket says 172.21.15.70:45330) 2024-04-01T01:38:47.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.900+0000 7f6267fff640 1 -- 172.21.15.70:0/2814209013 learned_addr learned my addr 172.21.15.70:0/2814209013 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:47.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.900+0000 7f6265ffb640 1 -- 172.21.15.70:0/2169270599 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f6258044020 con 0x7f6268108af0 2024-04-01T01:38:47.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.900+0000 7f626d79f640 1 -- 172.21.15.70:0/2814209013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23006400 msgr2=0x55da2300af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:47.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.900+0000 7f626cf9e640 1 --2- 172.21.15.70:0/2814209013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23006400 0x55da2300af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.900+0000 7f626d79f640 1 --2- 172.21.15.70:0/2814209013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23006400 0x55da2300af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.901+0000 7f626d79f640 1 -- 172.21.15.70:0/2814209013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da2300b570 msgr2=0x55da2300da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:47.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.901+0000 7f626d79f640 1 --2- 172.21.15.70:0/2814209013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da2300b570 0x55da2300da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.901+0000 7f626d79f640 1 -- 172.21.15.70:0/2814209013 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da22d467e0 con 0x55da23005a10 2024-04-01T01:38:47.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.901+0000 7f6267fff640 1 --2- 172.21.15.70:0/2814209013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da2300b570 0x55da2300da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:38:47.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.901+0000 7f626d79f640 1 --2- 172.21.15.70:0/2814209013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da23005a10 0x55da23005e30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f626806b440 tx=0x7f626000f360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3049d9b7bffdc73 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:47.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.901+0000 7f6265ffb640 1 --2- 172.21.15.70:0/2169270599 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62540a2560 0x7f62540a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.902+0000 7f6265ffb640 1 -- 172.21.15.70:0/2169270599 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f62580f0c40 con 0x7f6268108af0 2024-04-01T01:38:47.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.902+0000 7f625effd640 1 -- 172.21.15.70:0/2814209013 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626000fd90 con 0x55da23005a10 2024-04-01T01:38:47.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.902+0000 7f625effd640 1 -- 172.21.15.70:0/2814209013 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6260005020 con 0x55da23005a10 2024-04-01T01:38:47.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.902+0000 7f625effd640 1 -- 172.21.15.70:0/2814209013 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6260004a90 con 0x55da23005a10 2024-04-01T01:38:47.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.903+0000 7f626cf9e640 1 --2- 172.21.15.70:0/2169270599 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62540a2560 0x7f62540a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.904+0000 7f6271de7740 1 -- 172.21.15.70:0/2814209013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da23005a10 msgr2=0x55da23005e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:47.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.904+0000 7f6271de7740 1 --2- 172.21.15.70:0/2814209013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da23005a10 0x55da23005e30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f626806b440 tx=0x7f626000f360 comp rx=0 tx=0).stop 2024-04-01T01:38:47.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.905+0000 7f6271de7740 1 -- 172.21.15.70:0/2814209013 shutdown_connections 2024-04-01T01:38:47.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.905+0000 7f6271de7740 1 --2- 172.21.15.70:0/2814209013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da2300b570 0x55da2300da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.905+0000 7f6271de7740 1 --2- 172.21.15.70:0/2814209013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23006400 0x55da2300af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.905+0000 7f6271de7740 1 --2- 172.21.15.70:0/2814209013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da23005a10 0x55da23005e30 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.905+0000 7f6271de7740 1 -- 172.21.15.70:0/2814209013 >> 172.21.15.70:0/2814209013 conn(0x55da23000f80 msgr2=0x55da23001380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:47.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.906+0000 7f6271de7740 1 -- 172.21.15.70:0/2814209013 shutdown_connections 2024-04-01T01:38:47.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.906+0000 7f626cf9e640 1 --2- 172.21.15.70:0/2169270599 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62540a2560 0x7f62540a4a20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f62580ee6e0 tx=0x7f6258000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:47.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.906+0000 7f6271de7740 1 -- 172.21.15.70:0/2814209013 wait complete. 2024-04-01T01:38:47.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.906+0000 7f6271de7740 1 -- start start 2024-04-01T01:38:47.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.906+0000 7f6271de7740 1 Processor -- start 2024-04-01T01:38:47.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.907+0000 7f6271de7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23005a10 0x55da2300fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.907+0000 7f6271de7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23006400 0x55da230103a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.907+0000 7f6271de7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 0x55da230173e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.907+0000 7f6271de7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55da22f083d0 con 0x55da23006400 2024-04-01T01:38:47.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.907+0000 7f6271de7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55da22f65f40 con 0x55da2300b570 2024-04-01T01:38:47.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.907+0000 7f6271de7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55da2300e520 con 0x55da23005a10 2024-04-01T01:38:47.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.907+0000 7f6271de7740 10 client.0 _ll_register_callbacks cb 0x55da22f25b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:47.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.907+0000 7f6267fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23006400 0x55da230103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.907+0000 7f626d79f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23005a10 0x55da2300fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.907+0000 7f626d79f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23005a10 0x55da2300fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46668/0 (socket says 172.21.15.70:46668) 2024-04-01T01:38:47.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.907+0000 7f6267fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23006400 0x55da230103a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45340/0 (socket says 172.21.15.70:45340) 2024-04-01T01:38:47.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.907+0000 7f626d79f640 1 -- 172.21.15.70:0/1557388410 learned_addr learned my addr 172.21.15.70:0/1557388410 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:47.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.907+0000 7f626d79f640 1 --2- 172.21.15.70:0/1557388410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 0x55da230173e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.908+0000 7f626d79f640 1 -- 172.21.15.70:0/1557388410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23005a10 msgr2=0x55da2300fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:47.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.908+0000 7f626d79f640 1 --2- 172.21.15.70:0/1557388410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23005a10 0x55da2300fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.908+0000 7f626d79f640 1 -- 172.21.15.70:0/1557388410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23006400 msgr2=0x55da230103a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:47.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.908+0000 7f626d79f640 1 --2- 172.21.15.70:0/1557388410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23006400 0x55da230103a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.908+0000 7f626d79f640 1 -- 172.21.15.70:0/1557388410 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da22da6f70 con 0x55da2300b570 2024-04-01T01:38:47.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.908+0000 7f626d79f640 1 --2- 172.21.15.70:0/1557388410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 0x55da230173e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f626000f860 tx=0x7f6260000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c3b075c9dd578f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:47.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.908+0000 7f6247fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:47.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.908+0000 7f6247fff640 1 -- 172.21.15.70:0/1557388410 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6260025040 con 0x55da2300b570 2024-04-01T01:38:47.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.909+0000 7f6247fff640 1 -- 172.21.15.70:0/1557388410 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f626000fdc0 con 0x55da2300b570 2024-04-01T01:38:47.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.909+0000 7f6271de7740 1 -- 172.21.15.70:0/1557388410 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6260005020 con 0x55da2300b570 2024-04-01T01:38:47.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.909+0000 7f6247fff640 1 -- 172.21.15.70:0/1557388410 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6260011440 con 0x55da2300b570 2024-04-01T01:38:47.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.909+0000 7f6271de7740 10 client.27068 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:47.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.909+0000 7f6271de7740 1 -- 172.21.15.70:0/1557388410 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55da22c3e170 con 0x55da2300b570 2024-04-01T01:38:47.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.910+0000 7f6247fff640 1 -- 172.21.15.70:0/1557388410 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6260005020 con 0x55da2300b570 2024-04-01T01:38:47.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.910+0000 7f6247fff640 1 -- 172.21.15.70:0/1557388410 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f626002c600 con 0x55da2300b570 2024-04-01T01:38:47.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.910+0000 7f6271de7740 10 client.27068 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:47.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.910+0000 7f6271de7740 10 client.27068 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:47.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.910+0000 7f6271de7740 20 client.27068 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:47.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.910+0000 7f6271de7740 1 --2- 172.21.15.70:0/1557388410 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55da2301b740 0x55da2301db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.910+0000 7f6271de7740 4 client.27068 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:47.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.910+0000 7f6271de7740 1 -- 172.21.15.70:0/1557388410 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da22f773e0 con 0x55da2301b740 2024-04-01T01:38:47.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.911+0000 7f6267fff640 1 --2- 172.21.15.70:0/1557388410 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55da2301b740 0x55da2301db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.912+0000 7f6267fff640 1 --2- 172.21.15.70:0/1557388410 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55da2301b740 0x55da2301db30 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b39a8379fc0083d server_cookie=fdf0587064bef8a2 in_seq=0 out_seq=0 2024-04-01T01:38:47.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.912+0000 7f6247fff640 10 client.27068 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:47.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.913+0000 7f6247fff640 1 -- 172.21.15.70:0/1557388410 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55da22f773e0 con 0x55da2301b740 2024-04-01T01:38:47.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.913+0000 7f6247fff640 10 client.27068 handle_command_reply: tid=0 2024-04-01T01:38:47.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.914+0000 7f6271de7740 1 client.27068 shutdown 2024-04-01T01:38:47.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.914+0000 7f6271de7740 1 -- 172.21.15.70:0/1557388410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 msgr2=0x55da230173e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:47.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.914+0000 7f6271de7740 1 --2- 172.21.15.70:0/1557388410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 0x55da230173e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f626000f860 tx=0x7f6260000bc0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.915+0000 7f6271de7740 1 -- 172.21.15.70:0/1557388410 shutdown_connections 2024-04-01T01:38:47.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.915+0000 7f6271de7740 1 --2- 172.21.15.70:0/1557388410 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55da2301b740 0x55da2301db30 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.915+0000 7f6271de7740 1 --2- 172.21.15.70:0/1557388410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 0x55da230173e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.915+0000 7f6271de7740 1 --2- 172.21.15.70:0/1557388410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23006400 0x55da230103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.915+0000 7f6271de7740 1 --2- 172.21.15.70:0/1557388410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23005a10 0x55da2300fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.915+0000 7f6271de7740 1 -- 172.21.15.70:0/1557388410 >> 172.21.15.70:0/1557388410 conn(0x55da23000f80 msgr2=0x55da23000180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:47.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.915+0000 7f6247fff640 0 client.27068 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:47.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.915+0000 7f6271de7740 1 -- 172.21.15.70:0/1557388410 shutdown_connections 2024-04-01T01:38:47.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.916+0000 7f6271de7740 1 -- 172.21.15.70:0/1557388410 wait complete. 2024-04-01T01:38:47.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.916+0000 7f6271de7740 20 client.27068 trim_cache size 0 max 16384 2024-04-01T01:38:47.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.951+0000 7f6271de7740 1 Processor -- start 2024-04-01T01:38:47.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.952+0000 7f6271de7740 1 -- start start 2024-04-01T01:38:47.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.952+0000 7f6271de7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23005a10 0x55da23000900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.952+0000 7f6271de7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23006400 0x55da23017300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.952+0000 7f6271de7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 0x55da230048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.953+0000 7f626d79f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 0x55da230048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.953+0000 7f626d79f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23005a10 0x55da23000900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.953+0000 7f626d79f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23005a10 0x55da23000900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45346/0 (socket says 172.21.15.70:45346) 2024-04-01T01:38:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.953+0000 7f626d79f640 1 -- 172.21.15.70:0/2119095364 learned_addr learned my addr 172.21.15.70:0/2119095364 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.953+0000 7f6271de7740 1 -- 172.21.15.70:0/2119095364 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f626807dde0 con 0x55da23005a10 2024-04-01T01:38:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.953+0000 7f6271de7740 1 -- 172.21.15.70:0/2119095364 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f626807f920 con 0x55da2300b570 2024-04-01T01:38:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.953+0000 7f6271de7740 1 -- 172.21.15.70:0/2119095364 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f626807e4b0 con 0x55da23006400 2024-04-01T01:38:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.955+0000 7f6267fff640 1 --2- 172.21.15.70:0/2119095364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23006400 0x55da23017300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.955+0000 7f626d79f640 1 -- 172.21.15.70:0/2119095364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23006400 msgr2=0x55da23017300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.955+0000 7f626d79f640 1 --2- 172.21.15.70:0/2119095364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23006400 0x55da23017300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.955+0000 7f626d79f640 1 -- 172.21.15.70:0/2119095364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23005a10 msgr2=0x55da23000900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.955+0000 7f626d79f640 1 --2- 172.21.15.70:0/2119095364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23005a10 0x55da23000900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.955+0000 7f626d79f640 1 -- 172.21.15.70:0/2119095364 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da23010fe0 con 0x55da2300b570 2024-04-01T01:38:47.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.956+0000 7f626d79f640 1 --2- 172.21.15.70:0/2119095364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 0x55da230048f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6260004c80 tx=0x7f626000f360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4f146d405006032 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:47.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.956+0000 7f625e7fc640 1 -- 172.21.15.70:0/2119095364 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626000fe50 con 0x55da2300b570 2024-04-01T01:38:47.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.957+0000 7f625e7fc640 1 -- 172.21.15.70:0/2119095364 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6260004e70 con 0x55da2300b570 2024-04-01T01:38:47.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.957+0000 7f625e7fc640 1 -- 172.21.15.70:0/2119095364 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6260011840 con 0x55da2300b570 2024-04-01T01:38:47.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.958+0000 7f6271de7740 1 -- 172.21.15.70:0/2119095364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 msgr2=0x55da230048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:47.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.958+0000 7f6271de7740 1 --2- 172.21.15.70:0/2119095364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 0x55da230048f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6260004c80 tx=0x7f626000f360 comp rx=0 tx=0).stop 2024-04-01T01:38:47.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.958+0000 7f6271de7740 1 -- 172.21.15.70:0/2119095364 shutdown_connections 2024-04-01T01:38:47.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.958+0000 7f6271de7740 1 --2- 172.21.15.70:0/2119095364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 0x55da230048f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.958+0000 7f6271de7740 1 --2- 172.21.15.70:0/2119095364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23006400 0x55da23017300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.958+0000 7f6271de7740 1 --2- 172.21.15.70:0/2119095364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23005a10 0x55da23000900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.958+0000 7f6271de7740 1 -- 172.21.15.70:0/2119095364 >> 172.21.15.70:0/2119095364 conn(0x55da23000f80 msgr2=0x55da230041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:47.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.958+0000 7f6271de7740 1 -- 172.21.15.70:0/2119095364 shutdown_connections 2024-04-01T01:38:47.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.958+0000 7f6271de7740 1 -- 172.21.15.70:0/2119095364 wait complete. 2024-04-01T01:38:47.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.958+0000 7f6271de7740 1 -- start start 2024-04-01T01:38:47.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.959+0000 7f6271de7740 1 Processor -- start 2024-04-01T01:38:47.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.959+0000 7f6271de7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23005a10 0x55da230147d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.959+0000 7f6271de7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23006400 0x55da23014d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.959+0000 7f626d79f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23005a10 0x55da230147d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.959+0000 7f626d79f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23005a10 0x55da230147d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45350/0 (socket says 172.21.15.70:45350) 2024-04-01T01:38:47.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.959+0000 7f6271de7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 0x55da23015cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.959+0000 7f626d79f640 1 -- 172.21.15.70:0/3792478829 learned_addr learned my addr 172.21.15.70:0/3792478829 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:47.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.960+0000 7f6271de7740 1 -- 172.21.15.70:0/3792478829 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f626807f250 con 0x55da23005a10 2024-04-01T01:38:47.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.960+0000 7f6271de7740 1 -- 172.21.15.70:0/3792478829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f626807eb80 con 0x55da2300b570 2024-04-01T01:38:47.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.960+0000 7f6271de7740 1 -- 172.21.15.70:0/3792478829 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f626807be10 con 0x55da23006400 2024-04-01T01:38:47.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.960+0000 7f6271de7740 10 client.0 _ll_register_callbacks cb 0x7f6260004960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:47.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.962+0000 7f6267fff640 1 --2- 172.21.15.70:0/3792478829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23006400 0x55da23014d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.963+0000 7f626d79f640 1 --2- 172.21.15.70:0/3792478829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 0x55da23015cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.963+0000 7f626d79f640 1 -- 172.21.15.70:0/3792478829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23006400 msgr2=0x55da23014d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:47.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.963+0000 7f626d79f640 1 --2- 172.21.15.70:0/3792478829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23006400 0x55da23014d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.963+0000 7f626d79f640 1 -- 172.21.15.70:0/3792478829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23005a10 msgr2=0x55da230147d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:47.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.963+0000 7f626d79f640 1 --2- 172.21.15.70:0/3792478829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23005a10 0x55da230147d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.963+0000 7f626d79f640 1 -- 172.21.15.70:0/3792478829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da230162c0 con 0x55da2300b570 2024-04-01T01:38:47.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.964+0000 7f626d79f640 1 --2- 172.21.15.70:0/3792478829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 0x55da23015cd0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6260005c20 tx=0x7f6260000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dc90e0ebe51869b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:47.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.964+0000 7f6247fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:47.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.964+0000 7f6247fff640 1 -- 172.21.15.70:0/3792478829 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626000fe50 con 0x55da2300b570 2024-04-01T01:38:47.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.964+0000 7f6247fff640 1 -- 172.21.15.70:0/3792478829 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f62600115e0 con 0x55da2300b570 2024-04-01T01:38:47.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.964+0000 7f6271de7740 1 -- 172.21.15.70:0/3792478829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6260004e70 con 0x55da2300b570 2024-04-01T01:38:47.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.964+0000 7f6247fff640 1 -- 172.21.15.70:0/3792478829 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f62600117e0 con 0x55da2300b570 2024-04-01T01:38:47.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.964+0000 7f6271de7740 10 client.27074 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:47.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.964+0000 7f6271de7740 1 -- 172.21.15.70:0/3792478829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55da230175e0 con 0x55da2300b570 2024-04-01T01:38:47.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.965+0000 7f6247fff640 1 -- 172.21.15.70:0/3792478829 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6260004e70 con 0x55da2300b570 2024-04-01T01:38:47.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.965+0000 7f6247fff640 1 -- 172.21.15.70:0/3792478829 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6260045760 con 0x55da2300b570 2024-04-01T01:38:47.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.965+0000 7f6271de7740 10 client.27074 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:47.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.965+0000 7f6271de7740 10 client.27074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:47.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.965+0000 7f6271de7740 20 client.27074 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:47.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.965+0000 7f6271de7740 1 --2- 172.21.15.70:0/3792478829 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55da23082100 0x55da23084520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:47.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.965+0000 7f6271de7740 4 client.27074 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:38:47.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.965+0000 7f6271de7740 1 -- 172.21.15.70:0/3792478829 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55da22f71730 con 0x55da23082100 2024-04-01T01:38:47.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.965+0000 7f6267fff640 1 --2- 172.21.15.70:0/3792478829 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55da23082100 0x55da23084520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:47.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.966+0000 7f6247fff640 10 client.27074 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:47.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.966+0000 7f6267fff640 1 --2- 172.21.15.70:0/3792478829 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55da23082100 0x55da23084520 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e651ee2c809d9eb server_cookie=d12996f63e492dcb in_seq=0 out_seq=0 2024-04-01T01:38:47.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.967+0000 7f6247fff640 1 -- 172.21.15.70:0/3792478829 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55da22f71730 con 0x55da23082100 2024-04-01T01:38:47.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.967+0000 7f6247fff640 10 client.27074 handle_command_reply: tid=0 2024-04-01T01:38:47.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.967+0000 7f6271de7740 1 client.27074 shutdown 2024-04-01T01:38:47.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.968+0000 7f6271de7740 1 -- 172.21.15.70:0/3792478829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 msgr2=0x55da23015cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:47.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.968+0000 7f6271de7740 1 --2- 172.21.15.70:0/3792478829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 0x55da23015cd0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6260005c20 tx=0x7f6260000bc0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.968+0000 7f6271de7740 1 -- 172.21.15.70:0/3792478829 shutdown_connections 2024-04-01T01:38:47.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.968+0000 7f6271de7740 1 --2- 172.21.15.70:0/3792478829 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55da23082100 0x55da23084520 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.969+0000 7f6271de7740 1 --2- 172.21.15.70:0/3792478829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55da2300b570 0x55da23015cd0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.969+0000 7f6247fff640 0 client.27074 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:47.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.969+0000 7f6271de7740 1 --2- 172.21.15.70:0/3792478829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da23006400 0x55da23014d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.969+0000 7f6271de7740 1 --2- 172.21.15.70:0/3792478829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55da23005a10 0x55da230147d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.969+0000 7f6271de7740 1 -- 172.21.15.70:0/3792478829 >> 172.21.15.70:0/3792478829 conn(0x55da23000f80 msgr2=0x55da23004a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:47.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.969+0000 7f6271de7740 1 -- 172.21.15.70:0/3792478829 shutdown_connections 2024-04-01T01:38:47.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.969+0000 7f6271de7740 1 -- 172.21.15.70:0/3792478829 wait complete. 2024-04-01T01:38:47.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.969+0000 7f6271de7740 20 client.27074 trim_cache size 0 max 16384 2024-04-01T01:38:47.972 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:38:47.972 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:38:47.972 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "a46c1282-3bf5-46bc-84cc-19c3d475f00a", 2024-04-01T01:38:47.972 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:38:47.972 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:38:47.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.974+0000 7f626f202640 1 -- 172.21.15.70:0/2169270599 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62540a2560 msgr2=0x7f62540a4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:47.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.974+0000 7f626f202640 1 --2- 172.21.15.70:0/2169270599 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62540a2560 0x7f62540a4a20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f62580ee6e0 tx=0x7f6258000f40 comp rx=0 tx=0).stop 2024-04-01T01:38:47.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.974+0000 7f626f202640 1 -- 172.21.15.70:0/2169270599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6268108af0 msgr2=0x7f62681a6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:47.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.974+0000 7f626f202640 1 --2- 172.21.15.70:0/2169270599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6268108af0 0x7f62681a6980 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6258006010 tx=0x7f6258003ce0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.975+0000 7f626f202640 1 -- 172.21.15.70:0/2169270599 shutdown_connections 2024-04-01T01:38:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.975+0000 7f626f202640 1 --2- 172.21.15.70:0/2169270599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f626810a1a0 0x7f62681adf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.975+0000 7f626f202640 1 --2- 172.21.15.70:0/2169270599 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62540a2560 0x7f62540a4a20 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.975+0000 7f626f202640 1 --2- 172.21.15.70:0/2169270599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6268109800 0x7f62681a6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.975+0000 7f626f202640 1 --2- 172.21.15.70:0/2169270599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6268108af0 0x7f62681a6980 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.975+0000 7f626f202640 1 -- 172.21.15.70:0/2169270599 >> 172.21.15.70:0/2169270599 conn(0x7f626806e5b0 msgr2=0x7f62681165f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.975+0000 7f626f202640 1 -- 172.21.15.70:0/2169270599 shutdown_connections 2024-04-01T01:38:47.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:47.976+0000 7f626f202640 1 -- 172.21.15.70:0/2169270599 wait complete. 2024-04-01T01:38:47.997 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:47 smithi070 ceph-mon[32179]: pgmap v517: 129 pgs: 129 active+clean; 1.0 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.3 MiB/s wr, 1.10k op/s 2024-04-01T01:38:47.998 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:38:47.998 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:38:48.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:47 smithi081 ceph-mon[35142]: pgmap v517: 129 pgs: 129 active+clean; 1.0 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.3 MiB/s wr, 1.10k op/s 2024-04-01T01:38:48.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.194+0000 7fd637d43640 1 -- 172.21.15.70:0/60427207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd630104b80 msgr2=0x7fd630105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:48.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.194+0000 7fd637d43640 1 --2- 172.21.15.70:0/60427207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd630104b80 0x7fd630105000 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd620009900 tx=0x7fd62002bc30 comp rx=0 tx=0).stop 2024-04-01T01:38:48.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.194+0000 7fd637d43640 1 -- 172.21.15.70:0/60427207 shutdown_connections 2024-04-01T01:38:48.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.194+0000 7fd637d43640 1 --2- 172.21.15.70:0/60427207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd630105540 0x7fd63010bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.194+0000 7fd637d43640 1 --2- 172.21.15.70:0/60427207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd630104b80 0x7fd630105000 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.194+0000 7fd637d43640 1 --2- 172.21.15.70:0/60427207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd630103990 0x7fd630103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.194+0000 7fd637d43640 1 -- 172.21.15.70:0/60427207 >> 172.21.15.70:0/60427207 conn(0x7fd6300fef00 msgr2=0x7fd630101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:48.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.195+0000 7fd637d43640 1 -- 172.21.15.70:0/60427207 shutdown_connections 2024-04-01T01:38:48.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.195+0000 7fd637d43640 1 -- 172.21.15.70:0/60427207 wait complete. 2024-04-01T01:38:48.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.196+0000 7fd637d43640 1 Processor -- start 2024-04-01T01:38:48.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.196+0000 7fd637d43640 1 -- start start 2024-04-01T01:38:48.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.196+0000 7fd637d43640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd630103990 0x7fd630195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.196+0000 7fd637d43640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd630104b80 0x7fd630195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.196+0000 7fd637d43640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd630105540 0x7fd63019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.196+0000 7fd637d43640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd63010f510 con 0x7fd630105540 2024-04-01T01:38:48.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.196+0000 7fd637d43640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd63010f390 con 0x7fd630103990 2024-04-01T01:38:48.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.196+0000 7fd637d43640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd63010f690 con 0x7fd630104b80 2024-04-01T01:38:48.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.196+0000 7fd6362e0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd630105540 0x7fd63019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:48.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.196+0000 7fd6362e0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd630105540 0x7fd63019cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45366/0 (socket says 172.21.15.70:45366) 2024-04-01T01:38:48.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.196+0000 7fd6362e0640 1 -- 172.21.15.70:0/985642703 learned_addr learned my addr 172.21.15.70:0/985642703 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:48.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.196+0000 7fd635adf640 1 --2- 172.21.15.70:0/985642703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd630103990 0x7fd630195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:48.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.196+0000 7fd6352de640 1 --2- 172.21.15.70:0/985642703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd630104b80 0x7fd630195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:48.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.197+0000 7fd6362e0640 1 -- 172.21.15.70:0/985642703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd630104b80 msgr2=0x7fd630195d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:48.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.197+0000 7fd6362e0640 1 --2- 172.21.15.70:0/985642703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd630104b80 0x7fd630195d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.197+0000 7fd6362e0640 1 -- 172.21.15.70:0/985642703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd630103990 msgr2=0x7fd630195800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:48.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.197+0000 7fd6362e0640 1 --2- 172.21.15.70:0/985642703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd630103990 0x7fd630195800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.197+0000 7fd6362e0640 1 -- 172.21.15.70:0/985642703 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6301ae410 con 0x7fd630105540 2024-04-01T01:38:48.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.197+0000 7fd6352de640 1 --2- 172.21.15.70:0/985642703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd630104b80 0x7fd630195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:38:48.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.197+0000 7fd6362e0640 1 --2- 172.21.15.70:0/985642703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd630105540 0x7fd63019cd60 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd62800eb40 tx=0x7fd62800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:48.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.197+0000 7fd626ffd640 1 -- 172.21.15.70:0/985642703 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd628004b20 con 0x7fd630105540 2024-04-01T01:38:48.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.198+0000 7fd626ffd640 1 -- 172.21.15.70:0/985642703 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd628004cc0 con 0x7fd630105540 2024-04-01T01:38:48.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.198+0000 7fd626ffd640 1 -- 172.21.15.70:0/985642703 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd628010690 con 0x7fd630105540 2024-04-01T01:38:48.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.198+0000 7fd637d43640 1 -- 172.21.15.70:0/985642703 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6301ae700 con 0x7fd630105540 2024-04-01T01:38:48.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.198+0000 7fd637d43640 1 -- 172.21.15.70:0/985642703 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6301aebd0 con 0x7fd630105540 2024-04-01T01:38:48.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.202+0000 7fd63a9f5740 1 Processor -- start 2024-04-01T01:38:48.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.203+0000 7fd626ffd640 1 -- 172.21.15.70:0/985642703 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fd628005020 con 0x7fd630105540 2024-04-01T01:38:48.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.203+0000 7fd63a9f5740 1 -- start start 2024-04-01T01:38:48.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.203+0000 7fd63a9f5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdb022b0 0x5625fdb026d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.203+0000 7fd63a9f5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 0x5625fdb077a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.203+0000 7fd63a9f5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb07e10 0x5625fdb0a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.203+0000 7fd63a9f5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5625fda67180 con 0x5625fdb02ca0 2024-04-01T01:38:48.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.203+0000 7fd63a9f5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5625fda0fb70 con 0x5625fdb022b0 2024-04-01T01:38:48.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.203+0000 7fd63a9f5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5625fda63220 con 0x5625fdb07e10 2024-04-01T01:38:48.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.204+0000 7fd626ffd640 1 --2- 172.21.15.70:0/985642703 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd61c0a25f0 0x7fd61c0a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.204+0000 7fd635adf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 0x5625fdb077a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:48.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.204+0000 7fd635adf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 0x5625fdb077a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45368/0 (socket says 172.21.15.70:45368) 2024-04-01T01:38:48.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.204+0000 7fd635adf640 1 -- 172.21.15.70:0/478596532 learned_addr learned my addr 172.21.15.70:0/478596532 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:48.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.204+0000 7fd626ffd640 1 -- 172.21.15.70:0/985642703 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd6280cf110 con 0x7fd630105540 2024-04-01T01:38:48.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.204+0000 7fd6352de640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb07e10 0x5625fdb0a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:48.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.204+0000 7fd6352de640 1 --2- 172.21.15.70:0/478596532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdb022b0 0x5625fdb026d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:48.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.204+0000 7fd6362e0640 1 --2- 172.21.15.70:0/985642703 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd61c0a25f0 0x7fd61c0a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:48.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.204+0000 7fd635adf640 1 -- 172.21.15.70:0/478596532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb07e10 msgr2=0x5625fdb0a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:48.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.204+0000 7fd635adf640 1 --2- 172.21.15.70:0/478596532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb07e10 0x5625fdb0a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.204+0000 7fd635adf640 1 -- 172.21.15.70:0/478596532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdb022b0 msgr2=0x5625fdb026d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:48.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.204+0000 7fd635adf640 1 --2- 172.21.15.70:0/478596532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdb022b0 0x5625fdb026d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.205+0000 7fd635adf640 1 -- 172.21.15.70:0/478596532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625fda237c0 con 0x5625fdb02ca0 2024-04-01T01:38:48.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.205+0000 7fd6362e0640 1 --2- 172.21.15.70:0/985642703 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd61c0a25f0 0x7fd61c0a4ab0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd628098170 tx=0x7fd62800cfa0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:48.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.205+0000 7fd635adf640 1 --2- 172.21.15.70:0/478596532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 0x5625fdb077a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd62c009ca0 tx=0x7fd62c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b779fdfa763d634d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:48.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.205+0000 7fd61bfff640 1 -- 172.21.15.70:0/478596532 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c004220 con 0x5625fdb02ca0 2024-04-01T01:38:48.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.205+0000 7fd61bfff640 1 -- 172.21.15.70:0/478596532 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd62c0043c0 con 0x5625fdb02ca0 2024-04-01T01:38:48.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.205+0000 7fd61bfff640 1 -- 172.21.15.70:0/478596532 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c0198a0 con 0x5625fdb02ca0 2024-04-01T01:38:48.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.206+0000 7fd63a9f5740 1 -- 172.21.15.70:0/478596532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 msgr2=0x5625fdb077a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:48.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.206+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/478596532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 0x5625fdb077a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd62c009ca0 tx=0x7fd62c011810 comp rx=0 tx=0).stop 2024-04-01T01:38:48.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.206+0000 7fd63a9f5740 1 -- 172.21.15.70:0/478596532 shutdown_connections 2024-04-01T01:38:48.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.206+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/478596532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb07e10 0x5625fdb0a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.206+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/478596532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 0x5625fdb077a0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.206+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/478596532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdb022b0 0x5625fdb026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.206+0000 7fd63a9f5740 1 -- 172.21.15.70:0/478596532 >> 172.21.15.70:0/478596532 conn(0x5625fdafd820 msgr2=0x5625fdafdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.207+0000 7fd63a9f5740 1 -- 172.21.15.70:0/478596532 shutdown_connections 2024-04-01T01:38:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.207+0000 7fd63a9f5740 1 -- 172.21.15.70:0/478596532 wait complete. 2024-04-01T01:38:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.207+0000 7fd63a9f5740 1 -- start start 2024-04-01T01:38:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.207+0000 7fd63a9f5740 1 Processor -- start 2024-04-01T01:38:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.208+0000 7fd63a9f5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdb022b0 0x5625fdb0c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.208+0000 7fd63a9f5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb02ca0 0x5625fdb0cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.208+0000 7fd63a9f5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb07e10 0x5625fdb13d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.208+0000 7fd63a9f5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5625fdb0adc0 con 0x5625fdb07e10 2024-04-01T01:38:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.208+0000 7fd63a9f5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5625fda054a0 con 0x5625fdb022b0 2024-04-01T01:38:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.208+0000 7fd63a9f5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5625fdb0af40 con 0x5625fdb02ca0 2024-04-01T01:38:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.208+0000 7fd63a9f5740 10 client.0 _ll_register_callbacks cb 0x5625fda08d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.208+0000 7fd635adf640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb02ca0 0x5625fdb0cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.209+0000 7fd6352de640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdb022b0 0x5625fdb0c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.209+0000 7fd6352de640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdb022b0 0x5625fdb0c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49404/0 (socket says 172.21.15.70:49404) 2024-04-01T01:38:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.209+0000 7fd6352de640 1 -- 172.21.15.70:0/445027999 learned_addr learned my addr 172.21.15.70:0/445027999 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:48.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.209+0000 7fd6352de640 1 --2- 172.21.15.70:0/445027999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb07e10 0x5625fdb13d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:48.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.209+0000 7fd6352de640 1 -- 172.21.15.70:0/445027999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb02ca0 msgr2=0x5625fdb0cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:48.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.209+0000 7fd6352de640 1 --2- 172.21.15.70:0/445027999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb02ca0 0x5625fdb0cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.209+0000 7fd6352de640 1 -- 172.21.15.70:0/445027999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb07e10 msgr2=0x5625fdb13d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:48.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.209+0000 7fd6352de640 1 --2- 172.21.15.70:0/445027999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb07e10 0x5625fdb13d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.209+0000 7fd6352de640 1 -- 172.21.15.70:0/445027999 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625fd8457d0 con 0x5625fdb022b0 2024-04-01T01:38:48.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.209+0000 7fd635adf640 1 --2- 172.21.15.70:0/445027999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb02ca0 0x5625fdb0cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:38:48.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.209+0000 7fd6352de640 1 --2- 172.21.15.70:0/445027999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdb022b0 0x5625fdb0c7a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd620009900 tx=0x7fd62002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e7555a7f8c06d84 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:48.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.210+0000 7fd6197fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:48.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.210+0000 7fd6197fa640 1 -- 172.21.15.70:0/445027999 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd620005030 con 0x5625fdb022b0 2024-04-01T01:38:48.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.210+0000 7fd6197fa640 1 -- 172.21.15.70:0/445027999 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd6200051d0 con 0x5625fdb022b0 2024-04-01T01:38:48.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.210+0000 7fd63a9f5740 1 -- 172.21.15.70:0/445027999 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd62c0043c0 con 0x5625fdb022b0 2024-04-01T01:38:48.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.210+0000 7fd6197fa640 1 -- 172.21.15.70:0/445027999 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd620055050 con 0x5625fdb022b0 2024-04-01T01:38:48.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.210+0000 7fd63a9f5740 10 client.27086 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:48.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.210+0000 7fd63a9f5740 1 -- 172.21.15.70:0/445027999 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625fd77b3f0 con 0x5625fdb022b0 2024-04-01T01:38:48.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.210+0000 7fd6197fa640 1 -- 172.21.15.70:0/445027999 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd62c0043c0 con 0x5625fdb022b0 2024-04-01T01:38:48.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.211+0000 7fd6197fa640 1 -- 172.21.15.70:0/445027999 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd62004e060 con 0x5625fdb022b0 2024-04-01T01:38:48.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.211+0000 7fd63a9f5740 10 client.27086 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:48.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.211+0000 7fd63a9f5740 10 client.27086 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:48.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.211+0000 7fd63a9f5740 20 client.27086 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:48.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.211+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/445027999 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5625fdb185f0 0x5625fdb1aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.211+0000 7fd63a9f5740 4 client.27086 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:48.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.211+0000 7fd63a9f5740 1 -- 172.21.15.70:0/445027999 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625fda746c0 con 0x5625fdb185f0 2024-04-01T01:38:48.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.211+0000 7fd635adf640 1 --2- 172.21.15.70:0/445027999 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5625fdb185f0 0x5625fdb1aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:48.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.212+0000 7fd6197fa640 10 client.27086 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:48.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.212+0000 7fd635adf640 1 --2- 172.21.15.70:0/445027999 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5625fdb185f0 0x5625fdb1aa90 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c52eba37e8cf954a server_cookie=9985ca50d736576f in_seq=0 out_seq=0 2024-04-01T01:38:48.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.213+0000 7fd6197fa640 1 -- 172.21.15.70:0/445027999 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5625fda746c0 con 0x5625fdb185f0 2024-04-01T01:38:48.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.213+0000 7fd6197fa640 10 client.27086 handle_command_reply: tid=0 2024-04-01T01:38:48.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.213+0000 7fd63a9f5740 1 client.27086 shutdown 2024-04-01T01:38:48.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.213+0000 7fd63a9f5740 1 -- 172.21.15.70:0/445027999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdb022b0 msgr2=0x5625fdb0c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:48.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.213+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/445027999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdb022b0 0x5625fdb0c7a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd620009900 tx=0x7fd62002bc30 comp rx=0 tx=0).stop 2024-04-01T01:38:48.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.214+0000 7fd63a9f5740 1 -- 172.21.15.70:0/445027999 shutdown_connections 2024-04-01T01:38:48.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.214+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/445027999 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5625fdb185f0 0x5625fdb1aa90 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.214+0000 7fd6197fa640 0 client.27086 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:48.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.214+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/445027999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb07e10 0x5625fdb13d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.214+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/445027999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb02ca0 0x5625fdb0cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.214+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/445027999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdb022b0 0x5625fdb0c7a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.214+0000 7fd63a9f5740 1 -- 172.21.15.70:0/445027999 >> 172.21.15.70:0/445027999 conn(0x5625fdafd820 msgr2=0x5625fdafcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:48.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.215+0000 7fd63a9f5740 1 -- 172.21.15.70:0/445027999 shutdown_connections 2024-04-01T01:38:48.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.215+0000 7fd63a9f5740 1 -- 172.21.15.70:0/445027999 wait complete. 2024-04-01T01:38:48.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.215+0000 7fd63a9f5740 20 client.27086 trim_cache size 0 max 16384 2024-04-01T01:38:48.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:47 smithi189 ceph-mon[35753]: pgmap v517: 129 pgs: 129 active+clean; 1.0 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.3 MiB/s wr, 1.10k op/s 2024-04-01T01:38:48.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.257+0000 7fd63a9f5740 1 Processor -- start 2024-04-01T01:38:48.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.258+0000 7fd63a9f5740 1 -- start start 2024-04-01T01:38:48.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.259+0000 7fd63a9f5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdafd820 0x5625fdafcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.259+0000 7fd63a9f5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb022b0 0x5625fdb13ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.259+0000 7fd63a9f5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 0x5625fdb1f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.259+0000 7fd63a9f5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6301b9200 con 0x5625fdb02ca0 2024-04-01T01:38:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.259+0000 7fd63a9f5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd6301bad40 con 0x5625fdafd820 2024-04-01T01:38:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.259+0000 7fd63a9f5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd6301b98d0 con 0x5625fdb022b0 2024-04-01T01:38:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.260+0000 7fd635adf640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb022b0 0x5625fdb13ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:48.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.260+0000 7fd635adf640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb022b0 0x5625fdb13ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46736/0 (socket says 172.21.15.70:46736) 2024-04-01T01:38:48.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.261+0000 7fd635adf640 1 -- 172.21.15.70:0/2226025707 learned_addr learned my addr 172.21.15.70:0/2226025707 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:48.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.262+0000 7fd635adf640 1 -- 172.21.15.70:0/2226025707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdafd820 msgr2=0x5625fdafcaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:48.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.262+0000 7fd635adf640 1 --2- 172.21.15.70:0/2226025707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdafd820 0x5625fdafcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.262+0000 7fd635adf640 1 -- 172.21.15.70:0/2226025707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 msgr2=0x5625fdb1f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:48.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.262+0000 7fd635adf640 1 --2- 172.21.15.70:0/2226025707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 0x5625fdb1f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.262+0000 7fd635adf640 1 -- 172.21.15.70:0/2226025707 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625fd73b0f0 con 0x5625fdb022b0 2024-04-01T01:38:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.262+0000 7fd635adf640 1 --2- 172.21.15.70:0/2226025707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb022b0 0x5625fdb13ce0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd62c011d10 tx=0x7fd62c009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdb0cc324d9cb475 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.263+0000 7fd61b7fe640 1 -- 172.21.15.70:0/2226025707 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c02ca30 con 0x5625fdb022b0 2024-04-01T01:38:48.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.264+0000 7fd61b7fe640 1 -- 172.21.15.70:0/2226025707 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd62c032070 con 0x5625fdb022b0 2024-04-01T01:38:48.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.264+0000 7fd61b7fe640 1 -- 172.21.15.70:0/2226025707 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c005040 con 0x5625fdb022b0 2024-04-01T01:38:48.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.265+0000 7fd63a9f5740 1 -- 172.21.15.70:0/2226025707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb022b0 msgr2=0x5625fdb13ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:48.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.265+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/2226025707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb022b0 0x5625fdb13ce0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd62c011d10 tx=0x7fd62c009680 comp rx=0 tx=0).stop 2024-04-01T01:38:48.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.265+0000 7fd63a9f5740 1 -- 172.21.15.70:0/2226025707 shutdown_connections 2024-04-01T01:38:48.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.265+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/2226025707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 0x5625fdb1f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.265+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/2226025707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb022b0 0x5625fdb13ce0 secure :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd62c011d10 tx=0x7fd62c009680 comp rx=0 tx=0).stop 2024-04-01T01:38:48.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.265+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/2226025707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdafd820 0x5625fdafcaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.265+0000 7fd63a9f5740 1 -- 172.21.15.70:0/2226025707 >> 172.21.15.70:0/2226025707 conn(0x5625fdafdf70 msgr2=0x5625fdafc390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:48.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.269+0000 7fd63a9f5740 1 -- 172.21.15.70:0/2226025707 shutdown_connections 2024-04-01T01:38:48.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.270+0000 7fd63a9f5740 1 -- 172.21.15.70:0/2226025707 wait complete. 2024-04-01T01:38:48.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.270+0000 7fd63a9f5740 1 -- start start 2024-04-01T01:38:48.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.270+0000 7fd63a9f5740 1 Processor -- start 2024-04-01T01:38:48.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.270+0000 7fd63a9f5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdafd820 0x5625fdb13a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.271+0000 7fd63a9f5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 0x5625fdb12090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.271+0000 7fd63a9f5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb07e10 0x5625fdb125d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.271+0000 7fd63a9f5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6301ba670 con 0x5625fdb02ca0 2024-04-01T01:38:48.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.271+0000 7fd63a9f5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd6301b9fa0 con 0x5625fdafd820 2024-04-01T01:38:48.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.271+0000 7fd63a9f5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd6301b71e0 con 0x5625fdb07e10 2024-04-01T01:38:48.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.271+0000 7fd63a9f5740 10 client.0 _ll_register_callbacks cb 0x7fd62c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:48.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.271+0000 7fd635adf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 0x5625fdb12090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:48.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.271+0000 7fd635adf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 0x5625fdb12090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45384/0 (socket says 172.21.15.70:45384) 2024-04-01T01:38:48.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.271+0000 7fd635adf640 1 -- 172.21.15.70:0/516476452 learned_addr learned my addr 172.21.15.70:0/516476452 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:48.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.271+0000 7fd6352de640 1 --2- 172.21.15.70:0/516476452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdafd820 0x5625fdb13a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:48.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.271+0000 7fd6352de640 1 --2- 172.21.15.70:0/516476452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb07e10 0x5625fdb125d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:48.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.272+0000 7fd635adf640 1 -- 172.21.15.70:0/516476452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb07e10 msgr2=0x5625fdb125d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:48.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.272+0000 7fd635adf640 1 --2- 172.21.15.70:0/516476452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb07e10 0x5625fdb125d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.272+0000 7fd635adf640 1 -- 172.21.15.70:0/516476452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdafd820 msgr2=0x5625fdb13a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:48.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.272+0000 7fd635adf640 1 --2- 172.21.15.70:0/516476452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdafd820 0x5625fdb13a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.272+0000 7fd635adf640 1 -- 172.21.15.70:0/516476452 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625fdb12c90 con 0x5625fdb02ca0 2024-04-01T01:38:48.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.272+0000 7fd635adf640 1 --2- 172.21.15.70:0/516476452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 0x5625fdb12090 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd62c009710 tx=0x7fd62c0052e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5dbdba9732f989 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:48.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.272+0000 7fd6352de640 1 --2- 172.21.15.70:0/516476452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb07e10 0x5625fdb125d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:38:48.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.272+0000 7fd6197fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:38:48.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.273+0000 7fd6197fa640 1 -- 172.21.15.70:0/516476452 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c0035f0 con 0x5625fdb02ca0 2024-04-01T01:38:48.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.273+0000 7fd6197fa640 1 -- 172.21.15.70:0/516476452 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fd62c003790 con 0x5625fdb02ca0 2024-04-01T01:38:48.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.273+0000 7fd63a9f5740 1 -- 172.21.15.70:0/516476452 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd62c032070 con 0x5625fdb02ca0 2024-04-01T01:38:48.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.273+0000 7fd6197fa640 1 -- 172.21.15.70:0/516476452 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd62c003990 con 0x5625fdb02ca0 2024-04-01T01:38:48.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.273+0000 7fd63a9f5740 10 client.17178 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:48.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.273+0000 7fd63a9f5740 1 -- 172.21.15.70:0/516476452 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625fdb908a0 con 0x5625fdb02ca0 2024-04-01T01:38:48.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.274+0000 7fd6197fa640 1 -- 172.21.15.70:0/516476452 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd62c032070 con 0x5625fdb02ca0 2024-04-01T01:38:48.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.275+0000 7fd6197fa640 1 -- 172.21.15.70:0/516476452 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd62c038460 con 0x5625fdb02ca0 2024-04-01T01:38:48.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.275+0000 7fd63a9f5740 10 client.17178 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:48.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.275+0000 7fd63a9f5740 10 client.17178 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:48.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.275+0000 7fd63a9f5740 20 client.17178 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:48.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.275+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/516476452 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5625fdb185f0 0x5625fdb93170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:48.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.275+0000 7fd63a9f5740 4 client.17178 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:38:48.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.275+0000 7fd63a9f5740 1 -- 172.21.15.70:0/516476452 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5625fda6ea10 con 0x5625fdb185f0 2024-04-01T01:38:48.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.276+0000 7fd6352de640 1 --2- 172.21.15.70:0/516476452 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5625fdb185f0 0x5625fdb93170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:48.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.277+0000 7fd6352de640 1 --2- 172.21.15.70:0/516476452 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5625fdb185f0 0x5625fdb93170 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bebefda7df051b69 server_cookie=4cc16bf5aa292acb in_seq=0 out_seq=0 2024-04-01T01:38:48.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.277+0000 7fd6197fa640 10 client.17178 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:48.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.277+0000 7fd6197fa640 1 -- 172.21.15.70:0/516476452 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5625fda6ea10 con 0x5625fdb185f0 2024-04-01T01:38:48.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.277+0000 7fd6197fa640 10 client.17178 handle_command_reply: tid=0 2024-04-01T01:38:48.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.277+0000 7fd63a9f5740 1 client.17178 shutdown 2024-04-01T01:38:48.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.278+0000 7fd63a9f5740 1 -- 172.21.15.70:0/516476452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 msgr2=0x5625fdb12090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:48.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.278+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/516476452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 0x5625fdb12090 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd62c009710 tx=0x7fd62c0052e0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.278+0000 7fd63a9f5740 1 -- 172.21.15.70:0/516476452 shutdown_connections 2024-04-01T01:38:48.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.278+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/516476452 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5625fdb185f0 0x5625fdb93170 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.278+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/516476452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5625fdb07e10 0x5625fdb125d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.278+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/516476452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625fdb02ca0 0x5625fdb12090 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.278+0000 7fd63a9f5740 1 --2- 172.21.15.70:0/516476452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5625fdafd820 0x5625fdb13a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.278+0000 7fd63a9f5740 1 -- 172.21.15.70:0/516476452 >> 172.21.15.70:0/516476452 conn(0x5625fdafdf70 msgr2=0x5625fdafcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:48.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.278+0000 7fd6197fa640 0 client.17178 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:48.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.278+0000 7fd63a9f5740 1 -- 172.21.15.70:0/516476452 shutdown_connections 2024-04-01T01:38:48.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.278+0000 7fd63a9f5740 1 -- 172.21.15.70:0/516476452 wait complete. 2024-04-01T01:38:48.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.279+0000 7fd63a9f5740 20 client.17178 trim_cache size 0 max 16384 2024-04-01T01:38:48.279 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:38:48.279 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:38:48.279 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:38:48.279 INFO:teuthology.orchestra.run.smithi070.stdout: "a46c1282-3bf5-46bc-84cc-19c3d475f00a": { 2024-04-01T01:38:48.279 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:38:48.279 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "a46c1282-3bf5-46bc-84cc-19c3d475f00a", 2024-04-01T01:38:48.279 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:38:48.279 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:38:48.279 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:38:48.279 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:38:48.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.280+0000 7fd637d43640 1 -- 172.21.15.70:0/985642703 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd61c0a25f0 msgr2=0x7fd61c0a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:48.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.280+0000 7fd637d43640 1 --2- 172.21.15.70:0/985642703 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd61c0a25f0 0x7fd61c0a4ab0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd628098170 tx=0x7fd62800cfa0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.280+0000 7fd637d43640 1 -- 172.21.15.70:0/985642703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd630105540 msgr2=0x7fd63019cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:48.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.280+0000 7fd637d43640 1 --2- 172.21.15.70:0/985642703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd630105540 0x7fd63019cd60 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd62800eb40 tx=0x7fd62800c2d0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.280+0000 7fd637d43640 1 -- 172.21.15.70:0/985642703 shutdown_connections 2024-04-01T01:38:48.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.280+0000 7fd637d43640 1 --2- 172.21.15.70:0/985642703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd630105540 0x7fd63019cd60 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.280+0000 7fd637d43640 1 --2- 172.21.15.70:0/985642703 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd61c0a25f0 0x7fd61c0a4ab0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.280+0000 7fd637d43640 1 --2- 172.21.15.70:0/985642703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd630104b80 0x7fd630195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.280+0000 7fd637d43640 1 --2- 172.21.15.70:0/985642703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd630103990 0x7fd630195800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:48.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.281+0000 7fd637d43640 1 -- 172.21.15.70:0/985642703 >> 172.21.15.70:0/985642703 conn(0x7fd6300fef00 msgr2=0x7fd630077cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:48.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.281+0000 7fd637d43640 1 -- 172.21.15.70:0/985642703 shutdown_connections 2024-04-01T01:38:48.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:38:48.281+0000 7fd637d43640 1 -- 172.21.15.70:0/985642703 wait complete. 2024-04-01T01:38:48.298 INFO:tasks.cephfs.filesystem:scrub status for tag:a46c1282-3bf5-46bc-84cc-19c3d475f00a - {'path': '/', 'tag': 'a46c1282-3bf5-46bc-84cc-19c3d475f00a', 'options': 'recursive,force'} 2024-04-01T01:38:49.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:48 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:38:49.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:48 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:38:49.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:48 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:38:49.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:48 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:38:49.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:48 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:38:49.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:48 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:38:49.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:48 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:38:49.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:48 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:38:49.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:48 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:38:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.794+0000 7f21b6884640 1 -- 172.21.15.70:0/3495266886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21b010eb10 msgr2=0x7f21b0110fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.794+0000 7f21b6884640 1 --2- 172.21.15.70:0/3495266886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21b010eb10 0x7f21b0110fa0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f21a8009be0 tx=0x7f21a802bef0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.794+0000 7f21b6884640 1 -- 172.21.15.70:0/3495266886 shutdown_connections 2024-04-01T01:38:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.794+0000 7f21b6884640 1 --2- 172.21.15.70:0/3495266886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21b010eb10 0x7f21b0110fa0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.794+0000 7f21b6884640 1 --2- 172.21.15.70:0/3495266886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21b010c190 0x7f21b010e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.794+0000 7f21b6884640 1 --2- 172.21.15.70:0/3495266886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21b0109860 0x7f21b010bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.794+0000 7f21b6884640 1 -- 172.21.15.70:0/3495266886 >> 172.21.15.70:0/3495266886 conn(0x7f21b006e950 msgr2=0x7f21b0070d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.795+0000 7f21b6884640 1 -- 172.21.15.70:0/3495266886 shutdown_connections 2024-04-01T01:38:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.796+0000 7f21b6884640 1 -- 172.21.15.70:0/3495266886 wait complete. 2024-04-01T01:38:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.796+0000 7f21b6884640 1 Processor -- start 2024-04-01T01:38:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.796+0000 7f21b6884640 1 -- start start 2024-04-01T01:38:49.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.797+0000 7f21b6884640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21b0109860 0x7f21b01a8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.797+0000 7f21b6884640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21b010c190 0x7f21b01ab3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.797+0000 7f21b6884640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21b010eb10 0x7f21b01ada20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.797+0000 7f21b6884640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f21b0113890 con 0x7f21b0109860 2024-04-01T01:38:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.797+0000 7f21b6884640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f21b0113710 con 0x7f21b010eb10 2024-04-01T01:38:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.797+0000 7f21b6884640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f21b0113a10 con 0x7f21b010c190 2024-04-01T01:38:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.797+0000 7f21affff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21b0109860 0x7f21b01a8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.798+0000 7f21affff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21b0109860 0x7f21b01a8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45390/0 (socket says 172.21.15.70:45390) 2024-04-01T01:38:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.798+0000 7f21affff640 1 -- 172.21.15.70:0/91160313 learned_addr learned my addr 172.21.15.70:0/91160313 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.798+0000 7f21b4e21640 1 --2- 172.21.15.70:0/91160313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21b010eb10 0x7f21b01ada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:49.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.798+0000 7f21affff640 1 -- 172.21.15.70:0/91160313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21b010c190 msgr2=0x7f21b01ab3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:38:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.798+0000 7f21affff640 1 --2- 172.21.15.70:0/91160313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21b010c190 0x7f21b01ab3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.798+0000 7f21affff640 1 -- 172.21.15.70:0/91160313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21b010eb10 msgr2=0x7f21b01ada20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.798+0000 7f21affff640 1 --2- 172.21.15.70:0/91160313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21b010eb10 0x7f21b01ada20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.798+0000 7f21affff640 1 -- 172.21.15.70:0/91160313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b01ae2e0 con 0x7f21b0109860 2024-04-01T01:38:49.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.799+0000 7f21affff640 1 --2- 172.21.15.70:0/91160313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21b0109860 0x7f21b01a8e80 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f21a000e9f0 tx=0x7f21a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.799+0000 7f21ad7fa640 1 -- 172.21.15.70:0/91160313 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a000cd80 con 0x7f21b0109860 2024-04-01T01:38:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.799+0000 7f21ad7fa640 1 -- 172.21.15.70:0/91160313 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21a00108e0 con 0x7f21b0109860 2024-04-01T01:38:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.799+0000 7f21ad7fa640 1 -- 172.21.15.70:0/91160313 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a0018920 con 0x7f21b0109860 2024-04-01T01:38:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.799+0000 7f21b6884640 1 -- 172.21.15.70:0/91160313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b01ae5d0 con 0x7f21b0109860 2024-04-01T01:38:49.800 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.800+0000 7f21b6884640 1 -- 172.21.15.70:0/91160313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b01aead0 con 0x7f21b0109860 2024-04-01T01:38:49.803 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.804+0000 7f21b9465740 1 Processor -- start 2024-04-01T01:38:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.804+0000 7f21b9465740 1 -- start start 2024-04-01T01:38:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.805+0000 7f21b9465740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4fb8d0 0x5642df4fbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.805+0000 7f21ad7fa640 1 -- 172.21.15.70:0/91160313 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f21a0010430 con 0x7f21b0109860 2024-04-01T01:38:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.805+0000 7f21b9465740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fc2c0 0x5642df500dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.805+0000 7f21b9465740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df501430 0x5642df5038c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.805+0000 7f21b9465740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5642df45f8f0 con 0x5642df4fc2c0 2024-04-01T01:38:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.805+0000 7f21b9465740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5642df4088d0 con 0x5642df4fb8d0 2024-04-01T01:38:49.804 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.805+0000 7f21b9465740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5642df45b990 con 0x5642df501430 2024-04-01T01:38:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.805+0000 7f21b4e21640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4fb8d0 0x5642df4fbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.805+0000 7f21affff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fc2c0 0x5642df500dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.805+0000 7f21b4e21640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4fb8d0 0x5642df4fbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49458/0 (socket says 172.21.15.70:49458) 2024-04-01T01:38:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.805+0000 7f21b4e21640 1 -- 172.21.15.70:0/3104498946 learned_addr learned my addr 172.21.15.70:0/3104498946 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.805+0000 7f21ad7fa640 1 --2- 172.21.15.70:0/91160313 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21980a26b0 0x7f21980a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.806+0000 7f21af7fe640 1 --2- 172.21.15.70:0/3104498946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df501430 0x5642df5038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.806+0000 7f21b4e21640 1 -- 172.21.15.70:0/3104498946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df501430 msgr2=0x5642df5038c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:49.806 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.806+0000 7f21b4e21640 1 --2- 172.21.15.70:0/3104498946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df501430 0x5642df5038c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.806+0000 7f21b4e21640 1 -- 172.21.15.70:0/3104498946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fc2c0 msgr2=0x5642df500dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.806+0000 7f21b4e21640 1 --2- 172.21.15.70:0/3104498946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fc2c0 0x5642df500dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.806+0000 7f21b4e21640 1 -- 172.21.15.70:0/3104498946 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642df174210 con 0x5642df4fb8d0 2024-04-01T01:38:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.806+0000 7f21b4e21640 1 --2- 172.21.15.70:0/91160313 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21980a26b0 0x7f21980a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.806+0000 7f21ad7fa640 1 -- 172.21.15.70:0/91160313 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f21a00993d0 con 0x7f21b0109860 2024-04-01T01:38:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.806+0000 7f21b4e21640 1 --2- 172.21.15.70:0/3104498946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4fb8d0 0x5642df4fbcf0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f21a8009be0 tx=0x7f21a8005c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd7c2651ed826cf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.806+0000 7f21967fc640 1 -- 172.21.15.70:0/3104498946 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a8033a60 con 0x5642df4fb8d0 2024-04-01T01:38:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.806+0000 7f21967fc640 1 -- 172.21.15.70:0/3104498946 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21a8003e20 con 0x5642df4fb8d0 2024-04-01T01:38:49.807 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.806+0000 7f21967fc640 1 -- 172.21.15.70:0/3104498946 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a80042a0 con 0x5642df4fb8d0 2024-04-01T01:38:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.807+0000 7f21b9465740 1 -- 172.21.15.70:0/3104498946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4fb8d0 msgr2=0x5642df4fbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.807+0000 7f21b4e21640 1 --2- 172.21.15.70:0/91160313 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21980a26b0 0x7f21980a4b70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f21a804d000 tx=0x7f21a8004950 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:49.808 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.807+0000 7f21b9465740 1 --2- 172.21.15.70:0/3104498946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4fb8d0 0x5642df4fbcf0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f21a8009be0 tx=0x7f21a8005c80 comp rx=0 tx=0).stop 2024-04-01T01:38:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.808+0000 7f21b9465740 1 -- 172.21.15.70:0/3104498946 shutdown_connections 2024-04-01T01:38:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.808+0000 7f21b9465740 1 --2- 172.21.15.70:0/3104498946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df501430 0x5642df5038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.808+0000 7f21b9465740 1 --2- 172.21.15.70:0/3104498946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fc2c0 0x5642df500dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.808+0000 7f21b9465740 1 --2- 172.21.15.70:0/3104498946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4fb8d0 0x5642df4fbcf0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.808+0000 7f21b9465740 1 -- 172.21.15.70:0/3104498946 >> 172.21.15.70:0/3104498946 conn(0x5642df4f6e40 msgr2=0x5642df4f7240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.808+0000 7f21b9465740 1 -- 172.21.15.70:0/3104498946 shutdown_connections 2024-04-01T01:38:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.808+0000 7f21b9465740 1 -- 172.21.15.70:0/3104498946 wait complete. 2024-04-01T01:38:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.808+0000 7f21b9465740 1 -- start start 2024-04-01T01:38:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.809+0000 7f21b9465740 1 Processor -- start 2024-04-01T01:38:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.809+0000 7f21b9465740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4fb8d0 0x5642df505c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.809+0000 7f21b9465740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df4fc2c0 0x5642df506140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.809+0000 7f21b9465740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df501430 0x5642df50d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.809 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.809+0000 7f21b9465740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5642df5043e0 con 0x5642df501430 2024-04-01T01:38:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.809+0000 7f21b9465740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5642df3fdf80 con 0x5642df4fb8d0 2024-04-01T01:38:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.809+0000 7f21b9465740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5642df504560 con 0x5642df4fc2c0 2024-04-01T01:38:49.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.809+0000 7f21b9465740 10 client.0 _ll_register_callbacks cb 0x5642df456290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.810+0000 7f21af7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df4fc2c0 0x5642df506140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.810+0000 7f21af7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df4fc2c0 0x5642df506140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46768/0 (socket says 172.21.15.70:46768) 2024-04-01T01:38:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.810+0000 7f21af7fe640 1 -- 172.21.15.70:0/2318541665 learned_addr learned my addr 172.21.15.70:0/2318541665 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.810+0000 7f21affff640 1 --2- 172.21.15.70:0/2318541665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4fb8d0 0x5642df505c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.810+0000 7f21af7fe640 1 -- 172.21.15.70:0/2318541665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4fb8d0 msgr2=0x5642df505c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.810+0000 7f21af7fe640 1 --2- 172.21.15.70:0/2318541665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4fb8d0 0x5642df505c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.810+0000 7f21af7fe640 1 -- 172.21.15.70:0/2318541665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df501430 msgr2=0x5642df50d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:49.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.810+0000 7f21af7fe640 1 --2- 172.21.15.70:0/2318541665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df501430 0x5642df50d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.810+0000 7f21af7fe640 1 -- 172.21.15.70:0/2318541665 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642df240620 con 0x5642df4fc2c0 2024-04-01T01:38:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.811+0000 7f21af7fe640 1 --2- 172.21.15.70:0/2318541665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df4fc2c0 0x5642df506140 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f21a4010410 tx=0x7f21a40108d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7584b81ce295d02c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.811+0000 7f218f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:38:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.811+0000 7f218f7fe640 1 -- 172.21.15.70:0/2318541665 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a4005220 con 0x5642df4fc2c0 2024-04-01T01:38:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.811+0000 7f218f7fe640 1 -- 172.21.15.70:0/2318541665 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f21a40053c0 con 0x5642df4fc2c0 2024-04-01T01:38:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.811+0000 7f21b9465740 1 -- 172.21.15.70:0/2318541665 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21a8003e20 con 0x5642df4fc2c0 2024-04-01T01:38:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.811+0000 7f218f7fe640 1 -- 172.21.15.70:0/2318541665 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f21a4023050 con 0x5642df4fc2c0 2024-04-01T01:38:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.812+0000 7f21b9465740 10 client.26977 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.812+0000 7f21b9465740 1 -- 172.21.15.70:0/2318541665 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5642df510260 con 0x5642df4fc2c0 2024-04-01T01:38:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.812+0000 7f218f7fe640 1 -- 172.21.15.70:0/2318541665 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21a8003e20 con 0x5642df4fc2c0 2024-04-01T01:38:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.812+0000 7f218f7fe640 1 -- 172.21.15.70:0/2318541665 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f21a400f040 con 0x5642df4fc2c0 2024-04-01T01:38:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.812+0000 7f21b9465740 10 client.26977 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.812+0000 7f21b9465740 10 client.26977 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.812+0000 7f21b9465740 20 client.26977 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:49.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.813+0000 7f21b9465740 1 --2- 172.21.15.70:0/2318541665 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5642df511dc0 0x5642df514260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.813+0000 7f21b9465740 4 client.26977 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:38:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.813+0000 7f21b9465740 1 -- 172.21.15.70:0/2318541665 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642df473120 con 0x5642df511dc0 2024-04-01T01:38:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.815+0000 7f21affff640 1 --2- 172.21.15.70:0/2318541665 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5642df511dc0 0x5642df514260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.816+0000 7f21affff640 1 --2- 172.21.15.70:0/2318541665 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5642df511dc0 0x5642df514260 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ff29102dc54e37a server_cookie=7c6a83f0784ffce2 in_seq=0 out_seq=0 2024-04-01T01:38:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.816+0000 7f218f7fe640 10 client.26977 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.817+0000 7f218f7fe640 1 -- 172.21.15.70:0/2318541665 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5642df473120 con 0x5642df511dc0 2024-04-01T01:38:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.817+0000 7f218f7fe640 10 client.26977 handle_command_reply: tid=0 2024-04-01T01:38:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.817+0000 7f21b9465740 1 client.26977 shutdown 2024-04-01T01:38:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.817+0000 7f21b9465740 1 -- 172.21.15.70:0/2318541665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df4fc2c0 msgr2=0x5642df506140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.817+0000 7f21b9465740 1 --2- 172.21.15.70:0/2318541665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df4fc2c0 0x5642df506140 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f21a4010410 tx=0x7f21a40108d0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.821+0000 7f21b9465740 1 -- 172.21.15.70:0/2318541665 shutdown_connections 2024-04-01T01:38:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.821+0000 7f21b9465740 1 --2- 172.21.15.70:0/2318541665 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5642df511dc0 0x5642df514260 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.821+0000 7f21b9465740 1 --2- 172.21.15.70:0/2318541665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df501430 0x5642df50d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.821+0000 7f21b9465740 1 --2- 172.21.15.70:0/2318541665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df4fc2c0 0x5642df506140 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.821+0000 7f21b9465740 1 --2- 172.21.15.70:0/2318541665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4fb8d0 0x5642df505c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.822 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.821+0000 7f21b9465740 1 -- 172.21.15.70:0/2318541665 >> 172.21.15.70:0/2318541665 conn(0x5642df4f6e40 msgr2=0x5642df4f5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:49.822 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.822+0000 7f218f7fe640 0 client.26977 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.823+0000 7f21b9465740 1 -- 172.21.15.70:0/2318541665 shutdown_connections 2024-04-01T01:38:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.823+0000 7f21b9465740 1 -- 172.21.15.70:0/2318541665 wait complete. 2024-04-01T01:38:49.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.824+0000 7f21b9465740 20 client.26977 trim_cache size 0 max 16384 2024-04-01T01:38:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.856+0000 7f21b9465740 1 Processor -- start 2024-04-01T01:38:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.856+0000 7f21b9465740 1 -- start start 2024-04-01T01:38:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.856+0000 7f21b9465740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df4f6e40 0x5642df4fd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.856+0000 7f21b9465740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4fb8d0 0x5642df4fdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.857+0000 7f21b9465740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fc2c0 0x5642df50d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.857+0000 7f21b9465740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f219c00fd90 con 0x5642df4fc2c0 2024-04-01T01:38:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.857+0000 7f21b9465740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f219c0118d0 con 0x5642df4fb8d0 2024-04-01T01:38:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.857+0000 7f21b9465740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f219c010460 con 0x5642df4f6e40 2024-04-01T01:38:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.857+0000 7f21b4e21640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fc2c0 0x5642df50d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.857+0000 7f21b4e21640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fc2c0 0x5642df50d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45414/0 (socket says 172.21.15.70:45414) 2024-04-01T01:38:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.857+0000 7f21b4e21640 1 -- 172.21.15.70:0/1107449892 learned_addr learned my addr 172.21.15.70:0/1107449892 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.858+0000 7f21b4e21640 1 -- 172.21.15.70:0/1107449892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df4f6e40 msgr2=0x5642df4fd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.858+0000 7f21b4e21640 1 --2- 172.21.15.70:0/1107449892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df4f6e40 0x5642df4fd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.858+0000 7f21b4e21640 1 -- 172.21.15.70:0/1107449892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4fb8d0 msgr2=0x5642df4fdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.858+0000 7f21b4e21640 1 --2- 172.21.15.70:0/1107449892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4fb8d0 0x5642df4fdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.858+0000 7f21b4e21640 1 -- 172.21.15.70:0/1107449892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642df4fbcb0 con 0x5642df4fc2c0 2024-04-01T01:38:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.858+0000 7f21b4e21640 1 --2- 172.21.15.70:0/1107449892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fc2c0 0x5642df50d140 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f21a804e040 tx=0x7f21a8048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fd9333a3a00d544 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.859+0000 7f2195ffb640 1 -- 172.21.15.70:0/1107449892 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a805d070 con 0x5642df4fc2c0 2024-04-01T01:38:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.859+0000 7f2195ffb640 1 -- 172.21.15.70:0/1107449892 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21a8034a90 con 0x5642df4fc2c0 2024-04-01T01:38:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.859+0000 7f2195ffb640 1 -- 172.21.15.70:0/1107449892 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a802c890 con 0x5642df4fc2c0 2024-04-01T01:38:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.860+0000 7f21b9465740 1 -- 172.21.15.70:0/1107449892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fc2c0 msgr2=0x5642df50d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.860+0000 7f21b9465740 1 --2- 172.21.15.70:0/1107449892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fc2c0 0x5642df50d140 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f21a804e040 tx=0x7f21a8048000 comp rx=0 tx=0).stop 2024-04-01T01:38:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.860+0000 7f21b9465740 1 -- 172.21.15.70:0/1107449892 shutdown_connections 2024-04-01T01:38:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.860+0000 7f21b9465740 1 --2- 172.21.15.70:0/1107449892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fc2c0 0x5642df50d140 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.860+0000 7f21b9465740 1 --2- 172.21.15.70:0/1107449892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4fb8d0 0x5642df4fdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.860+0000 7f21b9465740 1 --2- 172.21.15.70:0/1107449892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df4f6e40 0x5642df4fd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.860+0000 7f21b9465740 1 -- 172.21.15.70:0/1107449892 >> 172.21.15.70:0/1107449892 conn(0x5642df500f70 msgr2=0x5642df4f7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.861+0000 7f21b9465740 1 -- 172.21.15.70:0/1107449892 shutdown_connections 2024-04-01T01:38:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.861+0000 7f21b9465740 1 -- 172.21.15.70:0/1107449892 wait complete. 2024-04-01T01:38:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.862+0000 7f21b9465740 1 -- start start 2024-04-01T01:38:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.862+0000 7f21b9465740 1 Processor -- start 2024-04-01T01:38:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.863+0000 7f21b9465740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4f6e40 0x5642df509fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.863+0000 7f21b9465740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fb8d0 0x5642df50a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.863+0000 7f21affff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4f6e40 0x5642df509fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.863+0000 7f21affff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4f6e40 0x5642df509fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49484/0 (socket says 172.21.15.70:49484) 2024-04-01T01:38:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.863+0000 7f21affff640 1 -- 172.21.15.70:0/4087652517 learned_addr learned my addr 172.21.15.70:0/4087652517 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:38:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.863+0000 7f21b9465740 1 --2- 172.21.15.70:0/4087652517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df4fc2c0 0x5642df576190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.864+0000 7f21af7fe640 1 --2- 172.21.15.70:0/4087652517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fb8d0 0x5642df50a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.864+0000 7f21b9465740 1 -- 172.21.15.70:0/4087652517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f219c011200 con 0x5642df4fb8d0 2024-04-01T01:38:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.864+0000 7f21b9465740 1 -- 172.21.15.70:0/4087652517 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f219c010b30 con 0x5642df4f6e40 2024-04-01T01:38:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.864+0000 7f21b9465740 1 -- 172.21.15.70:0/4087652517 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f219c00dd70 con 0x5642df4fc2c0 2024-04-01T01:38:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.864+0000 7f21b9465740 10 client.0 _ll_register_callbacks cb 0x7f21a4009600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:38:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.864+0000 7f21affff640 1 -- 172.21.15.70:0/4087652517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df4fc2c0 msgr2=0x5642df576190 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:38:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.864+0000 7f21affff640 1 --2- 172.21.15.70:0/4087652517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df4fc2c0 0x5642df576190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.864+0000 7f21affff640 1 -- 172.21.15.70:0/4087652517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fb8d0 msgr2=0x5642df50a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:38:49.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.864+0000 7f21affff640 1 --2- 172.21.15.70:0/4087652517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fb8d0 0x5642df50a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.864+0000 7f21affff640 1 -- 172.21.15.70:0/4087652517 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642df5766d0 con 0x5642df4f6e40 2024-04-01T01:38:49.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.864+0000 7f21af7fe640 1 --2- 172.21.15.70:0/4087652517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fb8d0 0x5642df50a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:38:49.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.865+0000 7f21affff640 1 --2- 172.21.15.70:0/4087652517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4f6e40 0x5642df509fc0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f21a009b030 tx=0x7f21a000cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c42d7688f8cfb852 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:38:49.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.865+0000 7f218f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:38:49.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.866+0000 7f218f7fe640 1 -- 172.21.15.70:0/4087652517 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a00d3060 con 0x5642df4f6e40 2024-04-01T01:38:49.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.866+0000 7f218f7fe640 1 -- 172.21.15.70:0/4087652517 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f21a009f150 con 0x5642df4f6e40 2024-04-01T01:38:49.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.866+0000 7f21b9465740 1 -- 172.21.15.70:0/4087652517 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21a8034a90 con 0x5642df4f6e40 2024-04-01T01:38:49.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.866+0000 7f218f7fe640 1 -- 172.21.15.70:0/4087652517 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f21a009f480 con 0x5642df4f6e40 2024-04-01T01:38:49.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.866+0000 7f21b9465740 10 client.27104 fetch_fsmap learned FSMap version 29 2024-04-01T01:38:49.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.866+0000 7f21b9465740 1 -- 172.21.15.70:0/4087652517 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5642df5778a0 con 0x5642df4f6e40 2024-04-01T01:38:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.867+0000 7f218f7fe640 1 -- 172.21.15.70:0/4087652517 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21a8034a90 con 0x5642df4f6e40 2024-04-01T01:38:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.867+0000 7f218f7fe640 1 -- 172.21.15.70:0/4087652517 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f21a0010bf0 con 0x5642df4f6e40 2024-04-01T01:38:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.867+0000 7f21b9465740 10 client.27104 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:38:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.867+0000 7f21b9465740 10 client.27104 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:38:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.867+0000 7f21b9465740 20 client.27104 populate_metadata read hostname 'smithi070' 2024-04-01T01:38:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.867+0000 7f21b9465740 1 --2- 172.21.15.70:0/4087652517 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5642df501430 0x5642df57a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:38:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.867+0000 7f21b9465740 4 client.27104 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:38:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.867+0000 7f21b9465740 1 -- 172.21.15.70:0/4087652517 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5642df46ce30 con 0x5642df501430 2024-04-01T01:38:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.868+0000 7f21af7fe640 1 --2- 172.21.15.70:0/4087652517 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5642df501430 0x5642df57a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:38:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.868+0000 7f218f7fe640 10 client.27104 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.868+0000 7f21af7fe640 1 --2- 172.21.15.70:0/4087652517 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5642df501430 0x5642df57a260 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9085e937cd0f71fa server_cookie=57a8883a146e6f47 in_seq=0 out_seq=0 2024-04-01T01:38:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.899+0000 7f218f7fe640 1 -- 172.21.15.70:0/4087652517 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5642df46ce30 con 0x5642df501430 2024-04-01T01:38:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.899+0000 7f218f7fe640 10 client.27104 handle_command_reply: tid=0 2024-04-01T01:38:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.899+0000 7f21b9465740 1 client.27104 shutdown 2024-04-01T01:38:49.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.901+0000 7f21b9465740 1 -- 172.21.15.70:0/4087652517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4f6e40 msgr2=0x5642df509fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:49.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.901+0000 7f21b9465740 1 --2- 172.21.15.70:0/4087652517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4f6e40 0x5642df509fc0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f21a009b030 tx=0x7f21a000cf20 comp rx=0 tx=0).stop 2024-04-01T01:38:49.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.903+0000 7f21b9465740 1 -- 172.21.15.70:0/4087652517 shutdown_connections 2024-04-01T01:38:49.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.903+0000 7f21b9465740 1 --2- 172.21.15.70:0/4087652517 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5642df501430 0x5642df57a260 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.903+0000 7f21b9465740 1 --2- 172.21.15.70:0/4087652517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642df4fc2c0 0x5642df576190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.903+0000 7f21b9465740 1 --2- 172.21.15.70:0/4087652517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642df4fb8d0 0x5642df50a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.903+0000 7f21b9465740 1 --2- 172.21.15.70:0/4087652517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642df4f6e40 0x5642df509fc0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.903+0000 7f218f7fe640 0 client.27104 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:38:49.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.903+0000 7f21b9465740 1 -- 172.21.15.70:0/4087652517 >> 172.21.15.70:0/4087652517 conn(0x5642df500f70 msgr2=0x5642df4f56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:49.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.903+0000 7f21b9465740 1 -- 172.21.15.70:0/4087652517 shutdown_connections 2024-04-01T01:38:49.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.904+0000 7f21b9465740 1 -- 172.21.15.70:0/4087652517 wait complete. 2024-04-01T01:38:49.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.905+0000 7f21b9465740 20 client.27104 trim_cache size 0 max 16384 2024-04-01T01:38:49.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:38:49.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:38:49.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:38:49.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:38:49.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.906+0000 7f21b6884640 1 -- 172.21.15.70:0/91160313 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21980a26b0 msgr2=0x7f21980a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.906+0000 7f21b6884640 1 --2- 172.21.15.70:0/91160313 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21980a26b0 0x7f21980a4b70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f21a804d000 tx=0x7f21a8004950 comp rx=0 tx=0).stop 2024-04-01T01:38:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.906+0000 7f21b6884640 1 -- 172.21.15.70:0/91160313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21b0109860 msgr2=0x7f21b01a8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:38:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.907+0000 7f21b6884640 1 --2- 172.21.15.70:0/91160313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21b0109860 0x7f21b01a8e80 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f21a000e9f0 tx=0x7f21a000eeb0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.907+0000 7f21b6884640 1 -- 172.21.15.70:0/91160313 shutdown_connections 2024-04-01T01:38:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.907+0000 7f21b6884640 1 --2- 172.21.15.70:0/91160313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21b010eb10 0x7f21b01ada20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.907+0000 7f21b6884640 1 --2- 172.21.15.70:0/91160313 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21980a26b0 0x7f21980a4b70 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.907+0000 7f21b6884640 1 --2- 172.21.15.70:0/91160313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21b010c190 0x7f21b01ab3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.907+0000 7f21b6884640 1 --2- 172.21.15.70:0/91160313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f21b0109860 0x7f21b01a8e80 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:38:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.907+0000 7f21b6884640 1 -- 172.21.15.70:0/91160313 >> 172.21.15.70:0/91160313 conn(0x7f21b006e950 msgr2=0x7f21b010a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:38:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.907+0000 7f21b6884640 1 -- 172.21.15.70:0/91160313 shutdown_connections 2024-04-01T01:38:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:38:49.907+0000 7f21b6884640 1 -- 172.21.15.70:0/91160313 wait complete. 2024-04-01T01:38:50.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:49 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:38:50.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:49 smithi081 ceph-mon[35142]: pgmap v518: 129 pgs: 129 active+clean; 1.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.8 MiB/s wr, 1.05k op/s 2024-04-01T01:38:50.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:49 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:38:50.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:49 smithi189 ceph-mon[35753]: pgmap v518: 129 pgs: 129 active+clean; 1.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.8 MiB/s wr, 1.05k op/s 2024-04-01T01:38:50.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:49 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:38:50.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:49 smithi070 ceph-mon[32179]: pgmap v518: 129 pgs: 129 active+clean; 1.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.8 MiB/s wr, 1.05k op/s 2024-04-01T01:38:51.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:50 smithi189 ceph-mon[35753]: pgmap v519: 129 pgs: 129 active+clean; 1.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 7.9 MiB/s wr, 975 op/s 2024-04-01T01:38:51.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:50 smithi070 ceph-mon[32179]: pgmap v519: 129 pgs: 129 active+clean; 1.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 7.9 MiB/s wr, 975 op/s 2024-04-01T01:38:51.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:50 smithi081 ceph-mon[35142]: pgmap v519: 129 pgs: 129 active+clean; 1.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 7.9 MiB/s wr, 975 op/s 2024-04-01T01:38:51.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 245.0 s, 1054.0 tps, lat 94.605 ms stddev 39.537 2024-04-01T01:38:52.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:51 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:38:52.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:51 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:38:52.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:51 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:38:53.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:52 smithi189 ceph-mon[35753]: pgmap v520: 129 pgs: 129 active+clean; 1.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.3 MiB/s wr, 1.18k op/s 2024-04-01T01:38:53.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:52 smithi070 ceph-mon[32179]: pgmap v520: 129 pgs: 129 active+clean; 1.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.3 MiB/s wr, 1.18k op/s 2024-04-01T01:38:53.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:52 smithi081 ceph-mon[35142]: pgmap v520: 129 pgs: 129 active+clean; 1.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.3 MiB/s wr, 1.18k op/s 2024-04-01T01:38:56.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:55 smithi081 ceph-mon[35142]: pgmap v521: 129 pgs: 129 active+clean; 1.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 7.8 MiB/s wr, 1.02k op/s 2024-04-01T01:38:56.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:55 smithi189 ceph-mon[35753]: pgmap v521: 129 pgs: 129 active+clean; 1.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 7.8 MiB/s wr, 1.02k op/s 2024-04-01T01:38:56.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:55 smithi070 ceph-mon[32179]: pgmap v521: 129 pgs: 129 active+clean; 1.0 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 7.8 MiB/s wr, 1.02k op/s 2024-04-01T01:38:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 250.0 s, 1075.8 tps, lat 93.251 ms stddev 42.608 2024-04-01T01:38:58.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:57 smithi081 ceph-mon[35142]: pgmap v522: 129 pgs: 129 active+clean; 1.0 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 9.1 MiB/s wr, 1.17k op/s 2024-04-01T01:38:58.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:57 smithi189 ceph-mon[35753]: pgmap v522: 129 pgs: 129 active+clean; 1.0 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 9.1 MiB/s wr, 1.17k op/s 2024-04-01T01:38:58.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:57 smithi070 ceph-mon[32179]: pgmap v522: 129 pgs: 129 active+clean; 1.0 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 9.1 MiB/s wr, 1.17k op/s 2024-04-01T01:39:00.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:38:59 smithi081 ceph-mon[35142]: pgmap v523: 129 pgs: 129 active+clean; 1.0 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 9.0 MiB/s wr, 1.14k op/s 2024-04-01T01:39:00.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:38:59 smithi189 ceph-mon[35753]: pgmap v523: 129 pgs: 129 active+clean; 1.0 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 9.0 MiB/s wr, 1.14k op/s 2024-04-01T01:39:00.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:38:59 smithi070 ceph-mon[32179]: pgmap v523: 129 pgs: 129 active+clean; 1.0 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 9.0 MiB/s wr, 1.14k op/s 2024-04-01T01:39:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:00 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:39:01.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:00 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:39:01.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:00 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:39:01.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 255.0 s, 964.0 tps, lat 103.640 ms stddev 46.120 2024-04-01T01:39:02.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:01 smithi081 ceph-mon[35142]: pgmap v524: 129 pgs: 129 active+clean; 1.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 8.3 MiB/s wr, 1.05k op/s 2024-04-01T01:39:02.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:01 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:39:02.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:01 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:39:02.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:01 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:39:02.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:01 smithi189 ceph-mon[35753]: pgmap v524: 129 pgs: 129 active+clean; 1.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 8.3 MiB/s wr, 1.05k op/s 2024-04-01T01:39:02.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:01 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:39:02.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:01 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:39:02.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:01 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:39:02.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:01 smithi070 ceph-mon[32179]: pgmap v524: 129 pgs: 129 active+clean; 1.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 8.3 MiB/s wr, 1.05k op/s 2024-04-01T01:39:02.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:01 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:39:02.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:01 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:39:02.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:01 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:39:03.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:02 smithi081 ceph-mon[35142]: pgmap v525: 129 pgs: 129 active+clean; 1.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 9.0 MiB/s wr, 1.12k op/s 2024-04-01T01:39:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.138+0000 7f094d53e640 1 -- 172.21.15.70:0/202601666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0948107db0 msgr2=0x7f09481081d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.138+0000 7f094d53e640 1 --2- 172.21.15.70:0/202601666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0948107db0 0x7f09481081d0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0938009900 tx=0x7f093802bc50 comp rx=0 tx=0).stop 2024-04-01T01:39:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.138+0000 7f094d53e640 1 -- 172.21.15.70:0/202601666 shutdown_connections 2024-04-01T01:39:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.138+0000 7f094d53e640 1 --2- 172.21.15.70:0/202601666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0948109960 0x7f09481101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.138+0000 7f094d53e640 1 --2- 172.21.15.70:0/202601666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0948108fa0 0x7f0948109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.138+0000 7f094d53e640 1 --2- 172.21.15.70:0/202601666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0948107db0 0x7f09481081d0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.138+0000 7f094d53e640 1 -- 172.21.15.70:0/202601666 >> 172.21.15.70:0/202601666 conn(0x7f0948076d50 msgr2=0x7f0948079190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.138+0000 7f094d53e640 1 -- 172.21.15.70:0/202601666 shutdown_connections 2024-04-01T01:39:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.138+0000 7f094d53e640 1 -- 172.21.15.70:0/202601666 wait complete. 2024-04-01T01:39:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.139+0000 7f094d53e640 1 Processor -- start 2024-04-01T01:39:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.139+0000 7f094d53e640 1 -- start start 2024-04-01T01:39:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.139+0000 7f094d53e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0948107db0 0x7f094819dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.139+0000 7f094d53e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0948108fa0 0x7f094819e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.139+0000 7f094d53e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0948109960 0x7f09481a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.140+0000 7f094d53e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0948113980 con 0x7f0948107db0 2024-04-01T01:39:03.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.140+0000 7f094d53e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0948113800 con 0x7f0948108fa0 2024-04-01T01:39:03.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.140+0000 7f094d53e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0948113b00 con 0x7f0948109960 2024-04-01T01:39:03.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.140+0000 7f0946ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0948107db0 0x7f094819dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.140+0000 7f09467fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0948108fa0 0x7f094819e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.140+0000 7f0946ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0948107db0 0x7f094819dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49308/0 (socket says 172.21.15.70:49308) 2024-04-01T01:39:03.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.140+0000 7f0946ffd640 1 -- 172.21.15.70:0/2775217438 learned_addr learned my addr 172.21.15.70:0/2775217438 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:03.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.140+0000 7f09477fe640 1 --2- 172.21.15.70:0/2775217438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0948109960 0x7f09481a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.140+0000 7f0946ffd640 1 -- 172.21.15.70:0/2775217438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0948109960 msgr2=0x7f09481a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:03.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.140+0000 7f0946ffd640 1 --2- 172.21.15.70:0/2775217438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0948109960 0x7f09481a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.140+0000 7f0946ffd640 1 -- 172.21.15.70:0/2775217438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0948108fa0 msgr2=0x7f094819e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:03.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.140+0000 7f0946ffd640 1 --2- 172.21.15.70:0/2775217438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0948108fa0 0x7f094819e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.140+0000 7f0946ffd640 1 -- 172.21.15.70:0/2775217438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09481a5bf0 con 0x7f0948107db0 2024-04-01T01:39:03.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.141+0000 7f0946ffd640 1 --2- 172.21.15.70:0/2775217438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0948107db0 0x7f094819dfd0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0938004a70 tx=0x7f093802ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:03.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.141+0000 7f092ffff640 1 -- 172.21.15.70:0/2775217438 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938041020 con 0x7f0948107db0 2024-04-01T01:39:03.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.141+0000 7f092ffff640 1 -- 172.21.15.70:0/2775217438 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f093802ee00 con 0x7f0948107db0 2024-04-01T01:39:03.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.141+0000 7f092ffff640 1 -- 172.21.15.70:0/2775217438 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938004fa0 con 0x7f0948107db0 2024-04-01T01:39:03.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.141+0000 7f094d53e640 1 -- 172.21.15.70:0/2775217438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09481a5e80 con 0x7f0948107db0 2024-04-01T01:39:03.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.141+0000 7f094d53e640 1 -- 172.21.15.70:0/2775217438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09481a62b0 con 0x7f0948107db0 2024-04-01T01:39:03.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.144+0000 7f092ffff640 1 -- 172.21.15.70:0/2775217438 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f0938044020 con 0x7f0948107db0 2024-04-01T01:39:03.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.145+0000 7f092ffff640 1 --2- 172.21.15.70:0/2775217438 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09300a26b0 0x7f09300a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.145+0000 7f092ffff640 1 -- 172.21.15.70:0/2775217438 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f09380f0300 con 0x7f0948107db0 2024-04-01T01:39:03.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.146+0000 7f09467fc640 1 --2- 172.21.15.70:0/2775217438 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09300a26b0 0x7f09300a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.146 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.146+0000 7f09467fc640 1 --2- 172.21.15.70:0/2775217438 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09300a26b0 0x7f09300a4b70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f094819f4e0 tx=0x7f093c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:03.148 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:02 smithi189 ceph-mon[35753]: pgmap v525: 129 pgs: 129 active+clean; 1.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 9.0 MiB/s wr, 1.12k op/s 2024-04-01T01:39:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.164+0000 7f09501ff740 1 Processor -- start 2024-04-01T01:39:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.164+0000 7f09501ff740 1 -- start start 2024-04-01T01:39:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.164+0000 7f09501ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c28d0 0x55727d2c2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.164+0000 7f09501ff740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c32c0 0x55727d2c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.164+0000 7f09501ff740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d2ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.164+0000 7f09501ff740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55727d2268f0 con 0x55727d2c8430 2024-04-01T01:39:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.164+0000 7f09501ff740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55727d1cf8d0 con 0x55727d2c32c0 2024-04-01T01:39:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.164+0000 7f09501ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55727d222990 con 0x55727d2c28d0 2024-04-01T01:39:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.165+0000 7f09477fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d2ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.165+0000 7f09467fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c32c0 0x55727d2c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.165+0000 7f09477fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d2ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49320/0 (socket says 172.21.15.70:49320) 2024-04-01T01:39:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.165+0000 7f09477fe640 1 -- 172.21.15.70:0/1619204708 learned_addr learned my addr 172.21.15.70:0/1619204708 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.165+0000 7f0946ffd640 1 --2- 172.21.15.70:0/1619204708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c28d0 0x55727d2c2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.165+0000 7f09477fe640 1 -- 172.21.15.70:0/1619204708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c28d0 msgr2=0x55727d2c2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.165+0000 7f09477fe640 1 --2- 172.21.15.70:0/1619204708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c28d0 0x55727d2c2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.165+0000 7f09477fe640 1 -- 172.21.15.70:0/1619204708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c32c0 msgr2=0x55727d2c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.165+0000 7f09477fe640 1 --2- 172.21.15.70:0/1619204708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c32c0 0x55727d2c7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.165+0000 7f09477fe640 1 -- 172.21.15.70:0/1619204708 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55727cf3b210 con 0x55727d2c8430 2024-04-01T01:39:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.165+0000 7f09467fc640 1 --2- 172.21.15.70:0/1619204708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c32c0 0x55727d2c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.165+0000 7f09477fe640 1 --2- 172.21.15.70:0/1619204708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d2ca8c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f094000a430 tx=0x7f0940009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7723e065d00723 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.165+0000 7f092d7fa640 1 -- 172.21.15.70:0/1619204708 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940013790 con 0x55727d2c8430 2024-04-01T01:39:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.166+0000 7f092d7fa640 1 -- 172.21.15.70:0/1619204708 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0940019070 con 0x55727d2c8430 2024-04-01T01:39:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.166+0000 7f092d7fa640 1 -- 172.21.15.70:0/1619204708 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940014d00 con 0x55727d2c8430 2024-04-01T01:39:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.167+0000 7f09501ff740 1 -- 172.21.15.70:0/1619204708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 msgr2=0x55727d2ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.167+0000 7f09501ff740 1 --2- 172.21.15.70:0/1619204708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d2ca8c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f094000a430 tx=0x7f0940009650 comp rx=0 tx=0).stop 2024-04-01T01:39:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.167+0000 7f09501ff740 1 -- 172.21.15.70:0/1619204708 shutdown_connections 2024-04-01T01:39:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.167+0000 7f09501ff740 1 --2- 172.21.15.70:0/1619204708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d2ca8c0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.167+0000 7f09501ff740 1 --2- 172.21.15.70:0/1619204708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c32c0 0x55727d2c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.167+0000 7f09501ff740 1 --2- 172.21.15.70:0/1619204708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c28d0 0x55727d2c2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.167+0000 7f09501ff740 1 -- 172.21.15.70:0/1619204708 >> 172.21.15.70:0/1619204708 conn(0x55727d2bde40 msgr2=0x55727d2be240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.167+0000 7f09501ff740 1 -- 172.21.15.70:0/1619204708 shutdown_connections 2024-04-01T01:39:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.167+0000 7f09501ff740 1 -- 172.21.15.70:0/1619204708 wait complete. 2024-04-01T01:39:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.167+0000 7f09501ff740 1 -- start start 2024-04-01T01:39:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.167+0000 7f09501ff740 1 Processor -- start 2024-04-01T01:39:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f09501ff740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c28d0 0x55727d2ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f09501ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c32c0 0x55727d2cd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f09501ff740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d2d4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f09501ff740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55727d2cb3e0 con 0x55727d2c8430 2024-04-01T01:39:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f09501ff740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55727d1c4f80 con 0x55727d2c28d0 2024-04-01T01:39:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f09501ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55727d2cb560 con 0x55727d2c32c0 2024-04-01T01:39:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f09501ff740 10 client.0 _ll_register_callbacks cb 0x55727d21d290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f0946ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c28d0 0x55727d2ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f09477fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d2d4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f0946ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c28d0 0x55727d2ccbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49228/0 (socket says 172.21.15.70:49228) 2024-04-01T01:39:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f0946ffd640 1 -- 172.21.15.70:0/2631738048 learned_addr learned my addr 172.21.15.70:0/2631738048 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f09467fc640 1 --2- 172.21.15.70:0/2631738048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c32c0 0x55727d2cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f0946ffd640 1 -- 172.21.15.70:0/2631738048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c32c0 msgr2=0x55727d2cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f0946ffd640 1 --2- 172.21.15.70:0/2631738048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c32c0 0x55727d2cd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f0946ffd640 1 -- 172.21.15.70:0/2631738048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 msgr2=0x55727d2d4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f0946ffd640 1 --2- 172.21.15.70:0/2631738048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d2d4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f0946ffd640 1 -- 172.21.15.70:0/2631738048 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55727d007620 con 0x55727d2c28d0 2024-04-01T01:39:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f09477fe640 1 --2- 172.21.15.70:0/2631738048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d2d4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.168+0000 7f0946ffd640 1 --2- 172.21.15.70:0/2631738048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c28d0 0x55727d2ccbd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f09380096c0 tx=0x7f09380bd090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22097d1a31c31dba server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.169+0000 7f092aff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.169+0000 7f092aff5640 1 -- 172.21.15.70:0/2631738048 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09380d4070 con 0x55727d2c28d0 2024-04-01T01:39:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.169+0000 7f092aff5640 1 -- 172.21.15.70:0/2631738048 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f09380bbde0 con 0x55727d2c28d0 2024-04-01T01:39:03.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.169+0000 7f09501ff740 1 -- 172.21.15.70:0/2631738048 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0940019070 con 0x55727d2c28d0 2024-04-01T01:39:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.169+0000 7f092aff5640 1 -- 172.21.15.70:0/2631738048 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f09380c0ae0 con 0x55727d2c28d0 2024-04-01T01:39:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.169+0000 7f09501ff740 10 client.27116 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.169+0000 7f09501ff740 1 -- 172.21.15.70:0/2631738048 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55727d2d7130 con 0x55727d2c28d0 2024-04-01T01:39:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.169+0000 7f092aff5640 1 -- 172.21.15.70:0/2631738048 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0940019070 con 0x55727d2c28d0 2024-04-01T01:39:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.169+0000 7f092aff5640 1 -- 172.21.15.70:0/2631738048 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f09380bb8c0 con 0x55727d2c28d0 2024-04-01T01:39:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.170+0000 7f09501ff740 10 client.27116 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.170+0000 7f09501ff740 10 client.27116 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.170+0000 7f09501ff740 20 client.27116 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.170+0000 7f09501ff740 1 --2- 172.21.15.70:0/2631738048 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55727d2d88b0 0x55727d2dad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.170+0000 7f09501ff740 4 client.27116 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.170+0000 7f09501ff740 1 -- 172.21.15.70:0/2631738048 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55727d23a120 con 0x55727d2d88b0 2024-04-01T01:39:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.170+0000 7f09467fc640 1 --2- 172.21.15.70:0/2631738048 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55727d2d88b0 0x55727d2dad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.170+0000 7f092aff5640 10 client.27116 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:03.175 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.170+0000 7f09467fc640 1 --2- 172.21.15.70:0/2631738048 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55727d2d88b0 0x55727d2dad00 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be48f082542b1d99 server_cookie=ae83fa55fb431f0f in_seq=0 out_seq=0 2024-04-01T01:39:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.171+0000 7f092aff5640 1 -- 172.21.15.70:0/2631738048 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55727d23a120 con 0x55727d2d88b0 2024-04-01T01:39:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.171+0000 7f092aff5640 10 client.27116 handle_command_reply: tid=0 2024-04-01T01:39:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.171+0000 7f09501ff740 1 client.27116 shutdown 2024-04-01T01:39:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.171+0000 7f09501ff740 1 -- 172.21.15.70:0/2631738048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c28d0 msgr2=0x55727d2ccbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.171+0000 7f09501ff740 1 --2- 172.21.15.70:0/2631738048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c28d0 0x55727d2ccbd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f09380096c0 tx=0x7f09380bd090 comp rx=0 tx=0).stop 2024-04-01T01:39:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.171+0000 7f09501ff740 1 -- 172.21.15.70:0/2631738048 shutdown_connections 2024-04-01T01:39:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.171+0000 7f09501ff740 1 --2- 172.21.15.70:0/2631738048 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55727d2d88b0 0x55727d2dad00 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.171+0000 7f09501ff740 1 --2- 172.21.15.70:0/2631738048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d2d4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.171+0000 7f09501ff740 1 --2- 172.21.15.70:0/2631738048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c32c0 0x55727d2cd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.171+0000 7f09501ff740 1 --2- 172.21.15.70:0/2631738048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c28d0 0x55727d2ccbd0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.171+0000 7f09501ff740 1 -- 172.21.15.70:0/2631738048 >> 172.21.15.70:0/2631738048 conn(0x55727d2bde40 msgr2=0x55727d2bcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.171+0000 7f092aff5640 0 client.27116 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.171+0000 7f09501ff740 1 -- 172.21.15.70:0/2631738048 shutdown_connections 2024-04-01T01:39:03.176 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.171+0000 7f09501ff740 1 -- 172.21.15.70:0/2631738048 wait complete. 2024-04-01T01:39:03.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.171+0000 7f09501ff740 20 client.27116 trim_cache size 0 max 16384 2024-04-01T01:39:03.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.193+0000 7f09501ff740 1 Processor -- start 2024-04-01T01:39:03.194 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.193+0000 7f09501ff740 1 -- start start 2024-04-01T01:39:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.193+0000 7f09501ff740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c28d0 0x55727d2e0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.193+0000 7f09501ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c32c0 0x55727d2d40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.193+0000 7f09501ff740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d2c4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.193+0000 7f09501ff740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f09481b0a60 con 0x55727d2c8430 2024-04-01T01:39:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.193+0000 7f09501ff740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f09481b25a0 con 0x55727d2c28d0 2024-04-01T01:39:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.193+0000 7f09501ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f09481b1130 con 0x55727d2c32c0 2024-04-01T01:39:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.193+0000 7f0946ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c28d0 0x55727d2e0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.193+0000 7f09477fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d2c4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.193+0000 7f0946ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c28d0 0x55727d2e0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49230/0 (socket says 172.21.15.70:49230) 2024-04-01T01:39:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.193+0000 7f0946ffd640 1 -- 172.21.15.70:0/772923013 learned_addr learned my addr 172.21.15.70:0/772923013 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.193+0000 7f09467fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c32c0 0x55727d2d40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.194+0000 7f09477fe640 1 -- 172.21.15.70:0/772923013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c32c0 msgr2=0x55727d2d40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.194+0000 7f09477fe640 1 --2- 172.21.15.70:0/772923013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c32c0 0x55727d2d40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.195 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.194+0000 7f09477fe640 1 -- 172.21.15.70:0/772923013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c28d0 msgr2=0x55727d2e0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:03.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.194+0000 7f09477fe640 1 --2- 172.21.15.70:0/772923013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c28d0 0x55727d2e0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.194+0000 7f09477fe640 1 -- 172.21.15.70:0/772923013 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55727d2cb6a0 con 0x55727d2c8430 2024-04-01T01:39:03.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.194+0000 7f0946ffd640 1 --2- 172.21.15.70:0/772923013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c28d0 0x55727d2e0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:03.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.194+0000 7f09477fe640 1 --2- 172.21.15.70:0/772923013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d2c4580 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0940004b90 tx=0x7f0940009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b45950cdd651729 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:03.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.194+0000 7f092cff9640 1 -- 172.21.15.70:0/772923013 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940013790 con 0x55727d2c8430 2024-04-01T01:39:03.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.194+0000 7f092cff9640 1 -- 172.21.15.70:0/772923013 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0940023070 con 0x55727d2c8430 2024-04-01T01:39:03.196 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.194+0000 7f092cff9640 1 -- 172.21.15.70:0/772923013 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940014ab0 con 0x55727d2c8430 2024-04-01T01:39:03.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.195+0000 7f09501ff740 1 -- 172.21.15.70:0/772923013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 msgr2=0x55727d2c4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:03.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.195+0000 7f09501ff740 1 --2- 172.21.15.70:0/772923013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d2c4580 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0940004b90 tx=0x7f0940009bd0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.197 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.195+0000 7f09501ff740 1 -- 172.21.15.70:0/772923013 shutdown_connections 2024-04-01T01:39:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.195+0000 7f09501ff740 1 --2- 172.21.15.70:0/772923013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d2c4580 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.195+0000 7f09501ff740 1 --2- 172.21.15.70:0/772923013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c32c0 0x55727d2d40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.195+0000 7f09501ff740 1 --2- 172.21.15.70:0/772923013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c28d0 0x55727d2e0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.195+0000 7f09501ff740 1 -- 172.21.15.70:0/772923013 >> 172.21.15.70:0/772923013 conn(0x55727d2bde40 msgr2=0x55727d2be220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.195+0000 7f09501ff740 1 -- 172.21.15.70:0/772923013 shutdown_connections 2024-04-01T01:39:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.195+0000 7f09501ff740 1 -- 172.21.15.70:0/772923013 wait complete. 2024-04-01T01:39:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.196+0000 7f09501ff740 1 -- start start 2024-04-01T01:39:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.196+0000 7f09501ff740 1 Processor -- start 2024-04-01T01:39:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.196+0000 7f09501ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c28d0 0x55727d2d0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.196+0000 7f09501ff740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c32c0 0x55727d2d1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.196+0000 7f09501ff740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d33cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.196+0000 7f09501ff740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f09481b1ed0 con 0x55727d2c8430 2024-04-01T01:39:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.196+0000 7f09501ff740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f09481b1800 con 0x55727d2c32c0 2024-04-01T01:39:03.198 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.196+0000 7f09501ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f09481aea40 con 0x55727d2c28d0 2024-04-01T01:39:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.197+0000 7f09501ff740 10 client.0 _ll_register_callbacks cb 0x7f09380b8cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.197+0000 7f09477fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d33cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.197+0000 7f09477fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d33cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49346/0 (socket says 172.21.15.70:49346) 2024-04-01T01:39:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.197+0000 7f0946ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c28d0 0x55727d2d0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.197+0000 7f09477fe640 1 -- 172.21.15.70:0/3155454402 learned_addr learned my addr 172.21.15.70:0/3155454402 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.197+0000 7f09467fc640 1 --2- 172.21.15.70:0/3155454402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c32c0 0x55727d2d1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.197+0000 7f09477fe640 1 -- 172.21.15.70:0/3155454402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c28d0 msgr2=0x55727d2d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.197+0000 7f09477fe640 1 --2- 172.21.15.70:0/3155454402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c28d0 0x55727d2d0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.197+0000 7f09477fe640 1 -- 172.21.15.70:0/3155454402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c32c0 msgr2=0x55727d2d1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.197+0000 7f09477fe640 1 --2- 172.21.15.70:0/3155454402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c32c0 0x55727d2d1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.197+0000 7f09477fe640 1 -- 172.21.15.70:0/3155454402 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55727d2d1970 con 0x55727d2c8430 2024-04-01T01:39:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.197+0000 7f09477fe640 1 --2- 172.21.15.70:0/3155454402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d33cc30 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f094000a430 tx=0x7f09400099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa2e2fc773d0b8ed server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.198+0000 7f092aff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:39:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.198+0000 7f092aff5640 1 -- 172.21.15.70:0/3155454402 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940032030 con 0x55727d2c8430 2024-04-01T01:39:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.198+0000 7f09501ff740 1 -- 172.21.15.70:0/3155454402 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0940023070 con 0x55727d2c8430 2024-04-01T01:39:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.198+0000 7f092aff5640 1 -- 172.21.15.70:0/3155454402 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0940015b50 con 0x55727d2c8430 2024-04-01T01:39:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.198+0000 7f092aff5640 1 -- 172.21.15.70:0/3155454402 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f094003f050 con 0x55727d2c8430 2024-04-01T01:39:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.198+0000 7f09501ff740 10 client.17211 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.198+0000 7f09501ff740 1 -- 172.21.15.70:0/3155454402 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55727d33e1f0 con 0x55727d2c8430 2024-04-01T01:39:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.199+0000 7f092aff5640 1 -- 172.21.15.70:0/3155454402 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0940023070 con 0x55727d2c8430 2024-04-01T01:39:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.199+0000 7f092aff5640 1 -- 172.21.15.70:0/3155454402 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f094002a4c0 con 0x55727d2c8430 2024-04-01T01:39:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.199+0000 7f09501ff740 10 client.17211 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.199+0000 7f09501ff740 10 client.17211 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.199+0000 7f09501ff740 20 client.17211 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.199+0000 7f09501ff740 1 --2- 172.21.15.70:0/3155454402 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55727d33e940 0x55727d340d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.199+0000 7f09501ff740 4 client.17211 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:39:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.199+0000 7f09501ff740 1 -- 172.21.15.70:0/3155454402 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55727d233e30 con 0x55727d33e940 2024-04-01T01:39:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.199+0000 7f0946ffd640 1 --2- 172.21.15.70:0/3155454402 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55727d33e940 0x55727d340d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.200+0000 7f092aff5640 10 client.17211 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.200+0000 7f0946ffd640 1 --2- 172.21.15.70:0/3155454402 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55727d33e940 0x55727d340d60 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c513ec2ba99fe1b7 server_cookie=d4471660bd038546 in_seq=0 out_seq=0 2024-04-01T01:39:03.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:02 smithi070 ceph-mon[32179]: pgmap v525: 129 pgs: 129 active+clean; 1.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 9.0 MiB/s wr, 1.12k op/s 2024-04-01T01:39:03.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.418+0000 7f092aff5640 1 -- 172.21.15.70:0/3155454402 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55727d233e30 con 0x55727d33e940 2024-04-01T01:39:03.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.418+0000 7f092aff5640 10 client.17211 handle_command_reply: tid=0 2024-04-01T01:39:03.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.418+0000 7f09501ff740 1 client.17211 shutdown 2024-04-01T01:39:03.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.418+0000 7f09501ff740 1 -- 172.21.15.70:0/3155454402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 msgr2=0x55727d33cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:03.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.418+0000 7f09501ff740 1 --2- 172.21.15.70:0/3155454402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d33cc30 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f094000a430 tx=0x7f09400099e0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.418+0000 7f09501ff740 1 -- 172.21.15.70:0/3155454402 shutdown_connections 2024-04-01T01:39:03.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.418+0000 7f09501ff740 1 --2- 172.21.15.70:0/3155454402 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55727d33e940 0x55727d340d60 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.418+0000 7f09501ff740 1 --2- 172.21.15.70:0/3155454402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55727d2c8430 0x55727d33cc30 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.418+0000 7f09501ff740 1 --2- 172.21.15.70:0/3155454402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55727d2c32c0 0x55727d2d1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.418+0000 7f09501ff740 1 --2- 172.21.15.70:0/3155454402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55727d2c28d0 0x55727d2d0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.418+0000 7f09501ff740 1 -- 172.21.15.70:0/3155454402 >> 172.21.15.70:0/3155454402 conn(0x55727d2bde40 msgr2=0x55727d2c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:03.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.418+0000 7f092aff5640 0 client.17211 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:03.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.418+0000 7f09501ff740 1 -- 172.21.15.70:0/3155454402 shutdown_connections 2024-04-01T01:39:03.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.418+0000 7f09501ff740 1 -- 172.21.15.70:0/3155454402 wait complete. 2024-04-01T01:39:03.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.419+0000 7f09501ff740 20 client.17211 trim_cache size 0 max 16384 2024-04-01T01:39:03.420 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:39:03.421 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:39:03.421 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:39:03.421 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:39:03.421 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.419+0000 7f094d53e640 1 -- 172.21.15.70:0/2775217438 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09300a26b0 msgr2=0x7f09300a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:03.421 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.419+0000 7f094d53e640 1 --2- 172.21.15.70:0/2775217438 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09300a26b0 0x7f09300a4b70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f094819f4e0 tx=0x7f093c005f90 comp rx=0 tx=0).stop 2024-04-01T01:39:03.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.419+0000 7f094d53e640 1 -- 172.21.15.70:0/2775217438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0948107db0 msgr2=0x7f094819dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:03.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.419+0000 7f094d53e640 1 --2- 172.21.15.70:0/2775217438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0948107db0 0x7f094819dfd0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0938004a70 tx=0x7f093802ec10 comp rx=0 tx=0).stop 2024-04-01T01:39:03.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.419+0000 7f094d53e640 1 -- 172.21.15.70:0/2775217438 shutdown_connections 2024-04-01T01:39:03.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.420+0000 7f094d53e640 1 --2- 172.21.15.70:0/2775217438 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09300a26b0 0x7f09300a4b70 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.420+0000 7f094d53e640 1 --2- 172.21.15.70:0/2775217438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0948109960 0x7f09481a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.420+0000 7f094d53e640 1 --2- 172.21.15.70:0/2775217438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0948108fa0 0x7f094819e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.420+0000 7f094d53e640 1 --2- 172.21.15.70:0/2775217438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0948107db0 0x7f094819dfd0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:03.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.420+0000 7f094d53e640 1 -- 172.21.15.70:0/2775217438 >> 172.21.15.70:0/2775217438 conn(0x7f0948076d50 msgr2=0x7f094810c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:03.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.420+0000 7f094d53e640 1 -- 172.21.15.70:0/2775217438 shutdown_connections 2024-04-01T01:39:03.422 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:03.420+0000 7f094d53e640 1 -- 172.21.15.70:0/2775217438 wait complete. 2024-04-01T01:39:06.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:05 smithi081 ceph-mon[35142]: pgmap v526: 129 pgs: 129 active+clean; 1.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.6 MiB/s wr, 944 op/s 2024-04-01T01:39:06.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:05 smithi189 ceph-mon[35753]: pgmap v526: 129 pgs: 129 active+clean; 1.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.6 MiB/s wr, 944 op/s 2024-04-01T01:39:06.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:05 smithi070 ceph-mon[32179]: pgmap v526: 129 pgs: 129 active+clean; 1.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 8.6 MiB/s wr, 944 op/s 2024-04-01T01:39:06.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 260.0 s, 836.2 tps, lat 113.027 ms stddev 185.674 2024-04-01T01:39:08.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:07 smithi081 ceph-mon[35142]: pgmap v527: 129 pgs: 129 active+clean; 1.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 11 MiB/s wr, 1.08k op/s 2024-04-01T01:39:08.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:07 smithi189 ceph-mon[35753]: pgmap v527: 129 pgs: 129 active+clean; 1.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 11 MiB/s wr, 1.08k op/s 2024-04-01T01:39:08.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:07 smithi070 ceph-mon[32179]: pgmap v527: 129 pgs: 129 active+clean; 1.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 11 MiB/s wr, 1.08k op/s 2024-04-01T01:39:10.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:09 smithi081 ceph-mon[35142]: pgmap v528: 129 pgs: 129 active+clean; 1.1 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 13 MiB/s wr, 1.01k op/s 2024-04-01T01:39:10.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:09 smithi189 ceph-mon[35753]: pgmap v528: 129 pgs: 129 active+clean; 1.1 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 13 MiB/s wr, 1.01k op/s 2024-04-01T01:39:10.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:09 smithi070 ceph-mon[32179]: pgmap v528: 129 pgs: 129 active+clean; 1.1 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 13 MiB/s wr, 1.01k op/s 2024-04-01T01:39:11.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:10 smithi189 ceph-mon[35753]: pgmap v529: 129 pgs: 129 active+clean; 1.1 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 15 MiB/s wr, 917 op/s 2024-04-01T01:39:11.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:10 smithi070 ceph-mon[32179]: pgmap v529: 129 pgs: 129 active+clean; 1.1 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 15 MiB/s wr, 917 op/s 2024-04-01T01:39:11.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:10 smithi081 ceph-mon[35142]: pgmap v529: 129 pgs: 129 active+clean; 1.1 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 15 MiB/s wr, 917 op/s 2024-04-01T01:39:11.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 265.0 s, 903.1 tps, lat 115.940 ms stddev 192.040 2024-04-01T01:39:14.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:13 smithi081 ceph-mon[35142]: pgmap v530: 129 pgs: 129 active+clean; 1.1 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 20 MiB/s wr, 1.05k op/s 2024-04-01T01:39:14.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:13 smithi189 ceph-mon[35753]: pgmap v530: 129 pgs: 129 active+clean; 1.1 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 20 MiB/s wr, 1.05k op/s 2024-04-01T01:39:14.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:13 smithi070 ceph-mon[32179]: pgmap v530: 129 pgs: 129 active+clean; 1.1 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 20 MiB/s wr, 1.05k op/s 2024-04-01T01:39:16.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:15 smithi081 ceph-mon[35142]: pgmap v531: 129 pgs: 129 active+clean; 1.1 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 19 MiB/s wr, 940 op/s 2024-04-01T01:39:16.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:15 smithi189 ceph-mon[35753]: pgmap v531: 129 pgs: 129 active+clean; 1.1 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 19 MiB/s wr, 940 op/s 2024-04-01T01:39:16.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:15 smithi070 ceph-mon[32179]: pgmap v531: 129 pgs: 129 active+clean; 1.1 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 19 MiB/s wr, 940 op/s 2024-04-01T01:39:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.575+0000 7f87e0377640 1 -- 172.21.15.70:0/1407700550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87d8100670 msgr2=0x7f87d8100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.575+0000 7f87e0377640 1 --2- 172.21.15.70:0/1407700550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87d8100670 0x7f87d8100a90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f87d400d910 tx=0x7f87d402dbb0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.577+0000 7f87e0377640 1 -- 172.21.15.70:0/1407700550 shutdown_connections 2024-04-01T01:39:16.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.577+0000 7f87e0377640 1 --2- 172.21.15.70:0/1407700550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87d8102220 0x7f87d8108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.577+0000 7f87e0377640 1 --2- 172.21.15.70:0/1407700550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87d8101860 0x7f87d8101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.577+0000 7f87e0377640 1 --2- 172.21.15.70:0/1407700550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87d8100670 0x7f87d8100a90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.577+0000 7f87e0377640 1 -- 172.21.15.70:0/1407700550 >> 172.21.15.70:0/1407700550 conn(0x7f87d80fbbe0 msgr2=0x7f87d80fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:16.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.577+0000 7f87e0377640 1 -- 172.21.15.70:0/1407700550 shutdown_connections 2024-04-01T01:39:16.576 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.577+0000 7f87e0377640 1 -- 172.21.15.70:0/1407700550 wait complete. 2024-04-01T01:39:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.578+0000 7f87e0377640 1 Processor -- start 2024-04-01T01:39:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.578+0000 7f87e0377640 1 -- start start 2024-04-01T01:39:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.578+0000 7f87e0377640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87d8100670 0x7f87d8195720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.579+0000 7f87e0377640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87d8101860 0x7f87d8195c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.579+0000 7f87e0377640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87d8102220 0x7f87d819ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.579+0000 7f87dd912640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87d8101860 0x7f87d8195c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.579+0000 7f87dd912640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87d8101860 0x7f87d8195c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:56484/0 (socket says 172.21.15.70:56484) 2024-04-01T01:39:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.579+0000 7f87de113640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87d8100670 0x7f87d8195720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.579+0000 7f87dd912640 1 -- 172.21.15.70:0/2417346844 learned_addr learned my addr 172.21.15.70:0/2417346844 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.579+0000 7f87e0377640 1 -- 172.21.15.70:0/2417346844 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f87d810b0b0 con 0x7f87d8100670 2024-04-01T01:39:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.579+0000 7f87e0377640 1 -- 172.21.15.70:0/2417346844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f87d810af30 con 0x7f87d8101860 2024-04-01T01:39:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.579+0000 7f87e0377640 1 -- 172.21.15.70:0/2417346844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f87d810b230 con 0x7f87d8102220 2024-04-01T01:39:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.579+0000 7f87de914640 1 --2- 172.21.15.70:0/2417346844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87d8102220 0x7f87d819ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.579+0000 7f87dd912640 1 -- 172.21.15.70:0/2417346844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87d8102220 msgr2=0x7f87d819ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.579+0000 7f87dd912640 1 --2- 172.21.15.70:0/2417346844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87d8102220 0x7f87d819ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.579+0000 7f87dd912640 1 -- 172.21.15.70:0/2417346844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87d8100670 msgr2=0x7f87d8195720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.579+0000 7f87dd912640 1 --2- 172.21.15.70:0/2417346844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87d8100670 0x7f87d8195720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.579+0000 7f87dd912640 1 -- 172.21.15.70:0/2417346844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d819d340 con 0x7f87d8101860 2024-04-01T01:39:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.580+0000 7f87dd912640 1 --2- 172.21.15.70:0/2417346844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87d8101860 0x7f87d8195c60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f87c800e9f0 tx=0x7f87c800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:16.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.580+0000 7f87cf7fe640 1 -- 172.21.15.70:0/2417346844 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c800cd80 con 0x7f87d8101860 2024-04-01T01:39:16.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.580+0000 7f87e0377640 1 -- 172.21.15.70:0/2417346844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87d819d630 con 0x7f87d8101860 2024-04-01T01:39:16.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.580+0000 7f87e0377640 1 -- 172.21.15.70:0/2417346844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87d819dbd0 con 0x7f87d8101860 2024-04-01T01:39:16.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.580+0000 7f87cf7fe640 1 -- 172.21.15.70:0/2417346844 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87c80108e0 con 0x7f87d8101860 2024-04-01T01:39:16.581 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.580+0000 7f87cf7fe640 1 -- 172.21.15.70:0/2417346844 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8018920 con 0x7f87d8101860 2024-04-01T01:39:16.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.583+0000 7f87cf7fe640 1 -- 172.21.15.70:0/2417346844 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f87c8018ac0 con 0x7f87d8101860 2024-04-01T01:39:16.583 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.584+0000 7f87e3032740 1 Processor -- start 2024-04-01T01:39:16.584 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.584+0000 7f87e3032740 1 -- start start 2024-04-01T01:39:16.584 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.584+0000 7f87e3032740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 0x55bfaac45cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.584+0000 7f87e3032740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 0x55bfaac4adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.584+0000 7f87e3032740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac4b430 0x55bfaac4d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.584+0000 7f87e3032740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bfaaba98f0 con 0x55bfaac4b430 2024-04-01T01:39:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.584+0000 7f87e3032740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55bfaab528d0 con 0x55bfaac462c0 2024-04-01T01:39:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.584+0000 7f87e3032740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bfaaba5990 con 0x55bfaac458d0 2024-04-01T01:39:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.584+0000 7f87de113640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 0x55bfaac4adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.584+0000 7f87de914640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 0x55bfaac45cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.584+0000 7f87de914640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 0x55bfaac45cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37610/0 (socket says 172.21.15.70:37610) 2024-04-01T01:39:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.584+0000 7f87de113640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 0x55bfaac4adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:56500/0 (socket says 172.21.15.70:56500) 2024-04-01T01:39:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.584+0000 7f87de914640 1 -- 172.21.15.70:0/384990711 learned_addr learned my addr 172.21.15.70:0/384990711 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.585+0000 7f87cf7fe640 1 --2- 172.21.15.70:0/2417346844 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f87c40a6a50 0x7f87c40a8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.585+0000 7f87dd912640 1 --2- 172.21.15.70:0/384990711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac4b430 0x55bfaac4d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.585+0000 7f87de113640 1 -- 172.21.15.70:0/384990711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 msgr2=0x55bfaac45cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:16.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.585+0000 7f87de113640 1 --2- 172.21.15.70:0/384990711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 0x55bfaac45cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.585+0000 7f87de113640 1 -- 172.21.15.70:0/384990711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac4b430 msgr2=0x55bfaac4d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:16.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.585+0000 7f87de113640 1 --2- 172.21.15.70:0/384990711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac4b430 0x55bfaac4d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.585+0000 7f87de113640 1 -- 172.21.15.70:0/384990711 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfaa8be210 con 0x55bfaac462c0 2024-04-01T01:39:16.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.585+0000 7f87de914640 1 --2- 172.21.15.70:0/2417346844 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f87c40a6a50 0x7f87c40a8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.585+0000 7f87cf7fe640 1 -- 172.21.15.70:0/2417346844 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f87c80ceda0 con 0x7f87d8101860 2024-04-01T01:39:16.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.585+0000 7f87de113640 1 --2- 172.21.15.70:0/384990711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 0x55bfaac4adc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009230 tx=0x7f87d4005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1760c08944c940 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.586+0000 7f87de914640 1 --2- 172.21.15.70:0/2417346844 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f87c40a6a50 0x7f87c40a8f10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x55bfaac46b20 tx=0x7f87d0004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.586+0000 7f87ccbfe640 1 -- 172.21.15.70:0/384990711 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d400d110 con 0x55bfaac462c0 2024-04-01T01:39:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.586+0000 7f87ccbfe640 1 -- 172.21.15.70:0/384990711 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87d4043070 con 0x55bfaac462c0 2024-04-01T01:39:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.587+0000 7f87ccbfe640 1 -- 172.21.15.70:0/384990711 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d40039f0 con 0x55bfaac462c0 2024-04-01T01:39:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.588+0000 7f87e3032740 1 -- 172.21.15.70:0/384990711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 msgr2=0x55bfaac4adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.588+0000 7f87e3032740 1 --2- 172.21.15.70:0/384990711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 0x55bfaac4adc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f87d4009230 tx=0x7f87d4005c00 comp rx=0 tx=0).stop 2024-04-01T01:39:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.588+0000 7f87e3032740 1 -- 172.21.15.70:0/384990711 shutdown_connections 2024-04-01T01:39:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.588+0000 7f87e3032740 1 --2- 172.21.15.70:0/384990711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac4b430 0x55bfaac4d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.588+0000 7f87e3032740 1 --2- 172.21.15.70:0/384990711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 0x55bfaac4adc0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.588+0000 7f87e3032740 1 --2- 172.21.15.70:0/384990711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 0x55bfaac45cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.588+0000 7f87e3032740 1 -- 172.21.15.70:0/384990711 >> 172.21.15.70:0/384990711 conn(0x55bfaac40e40 msgr2=0x55bfaac41240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.588+0000 7f87e3032740 1 -- 172.21.15.70:0/384990711 shutdown_connections 2024-04-01T01:39:16.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.588+0000 7f87e3032740 1 -- 172.21.15.70:0/384990711 wait complete. 2024-04-01T01:39:16.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.588+0000 7f87e3032740 1 -- start start 2024-04-01T01:39:16.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.589+0000 7f87e3032740 1 Processor -- start 2024-04-01T01:39:16.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.589+0000 7f87e3032740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac458d0 0x55bfaac69450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.589+0000 7f87e3032740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 0x55bfaac69990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.589+0000 7f87e3032740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac4b430 0x55bfaac709d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.589+0000 7f87e3032740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bfaac5f390 con 0x55bfaac458d0 2024-04-01T01:39:16.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.589+0000 7f87e3032740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55bfaab47f80 con 0x55bfaac462c0 2024-04-01T01:39:16.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.589+0000 7f87e3032740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bfaac5f510 con 0x55bfaac4b430 2024-04-01T01:39:16.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.589+0000 7f87e3032740 10 client.0 _ll_register_callbacks cb 0x55bfaaba0290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:16.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.589+0000 7f87de914640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac4b430 0x55bfaac709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.589+0000 7f87de113640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac458d0 0x55bfaac69450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.589+0000 7f87de113640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac458d0 0x55bfaac69450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46458/0 (socket says 172.21.15.70:46458) 2024-04-01T01:39:16.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.589+0000 7f87de113640 1 -- 172.21.15.70:0/1393234591 learned_addr learned my addr 172.21.15.70:0/1393234591 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:16.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.590+0000 7f87dd912640 1 --2- 172.21.15.70:0/1393234591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 0x55bfaac69990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.590+0000 7f87de113640 1 -- 172.21.15.70:0/1393234591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac4b430 msgr2=0x55bfaac709d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:16.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.590+0000 7f87de113640 1 --2- 172.21.15.70:0/1393234591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac4b430 0x55bfaac709d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.590+0000 7f87de113640 1 -- 172.21.15.70:0/1393234591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 msgr2=0x55bfaac69990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:16.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.590+0000 7f87de113640 1 --2- 172.21.15.70:0/1393234591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 0x55bfaac69990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.590+0000 7f87de113640 1 -- 172.21.15.70:0/1393234591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfaa98a620 con 0x55bfaac458d0 2024-04-01T01:39:16.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.590+0000 7f87de113640 1 --2- 172.21.15.70:0/1393234591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac458d0 0x55bfaac69450 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f87d4006010 tx=0x7f87d402e780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce43cf19d2978f95 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:16.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.590+0000 7f87c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:39:16.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.590+0000 7f87c1ffb640 1 -- 172.21.15.70:0/1393234591 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d402ed00 con 0x55bfaac458d0 2024-04-01T01:39:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.590+0000 7f87e3032740 1 -- 172.21.15.70:0/1393234591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87d4043070 con 0x55bfaac458d0 2024-04-01T01:39:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.591+0000 7f87c1ffb640 1 -- 172.21.15.70:0/1393234591 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f87d4004d20 con 0x55bfaac458d0 2024-04-01T01:39:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.591+0000 7f87c1ffb640 1 -- 172.21.15.70:0/1393234591 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f87d4056050 con 0x55bfaac458d0 2024-04-01T01:39:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.591+0000 7f87e3032740 10 client.17223 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.591+0000 7f87e3032740 1 -- 172.21.15.70:0/1393234591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bfaac739b0 con 0x55bfaac458d0 2024-04-01T01:39:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.592+0000 7f87c1ffb640 1 -- 172.21.15.70:0/1393234591 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87d4032050 con 0x55bfaac458d0 2024-04-01T01:39:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.592+0000 7f87c1ffb640 1 -- 172.21.15.70:0/1393234591 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f87d40367d0 con 0x55bfaac458d0 2024-04-01T01:39:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.592+0000 7f87e3032740 10 client.17223 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.592+0000 7f87e3032740 10 client.17223 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.592+0000 7f87e3032740 20 client.17223 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.592+0000 7f87e3032740 1 --2- 172.21.15.70:0/1393234591 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bfaac75240 0x55bfaac77660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.592+0000 7f87e3032740 4 client.17223 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.592+0000 7f87e3032740 1 -- 172.21.15.70:0/1393234591 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f87d4043070 con 0x55bfaac75240 2024-04-01T01:39:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.592+0000 7f87dd912640 1 --2- 172.21.15.70:0/1393234591 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bfaac75240 0x55bfaac77660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.594+0000 7f87dd912640 1 --2- 172.21.15.70:0/1393234591 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bfaac75240 0x55bfaac77660 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45c57c0d3991d0ec server_cookie=d69e71869de81103 in_seq=0 out_seq=0 2024-04-01T01:39:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.594+0000 7f87c1ffb640 10 client.17223 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.594+0000 7f87c1ffb640 1 -- 172.21.15.70:0/1393234591 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f87d4043070 con 0x55bfaac75240 2024-04-01T01:39:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.594+0000 7f87c1ffb640 10 client.17223 handle_command_reply: tid=0 2024-04-01T01:39:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.594+0000 7f87e3032740 1 client.17223 shutdown 2024-04-01T01:39:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.594+0000 7f87e3032740 1 -- 172.21.15.70:0/1393234591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac458d0 msgr2=0x55bfaac69450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.594+0000 7f87e3032740 1 --2- 172.21.15.70:0/1393234591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac458d0 0x55bfaac69450 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f87d4006010 tx=0x7f87d402e780 comp rx=0 tx=0).stop 2024-04-01T01:39:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.595+0000 7f87e3032740 1 -- 172.21.15.70:0/1393234591 shutdown_connections 2024-04-01T01:39:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.595+0000 7f87e3032740 1 --2- 172.21.15.70:0/1393234591 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bfaac75240 0x55bfaac77660 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.595+0000 7f87e3032740 1 --2- 172.21.15.70:0/1393234591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac4b430 0x55bfaac709d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.595+0000 7f87e3032740 1 --2- 172.21.15.70:0/1393234591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 0x55bfaac69990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.595+0000 7f87e3032740 1 --2- 172.21.15.70:0/1393234591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac458d0 0x55bfaac69450 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.595+0000 7f87e3032740 1 -- 172.21.15.70:0/1393234591 >> 172.21.15.70:0/1393234591 conn(0x55bfaac40e40 msgr2=0x55bfaac3ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.595+0000 7f87c1ffb640 0 client.17223 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.595+0000 7f87e3032740 1 -- 172.21.15.70:0/1393234591 shutdown_connections 2024-04-01T01:39:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.595+0000 7f87e3032740 1 -- 172.21.15.70:0/1393234591 wait complete. 2024-04-01T01:39:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.595+0000 7f87e3032740 20 client.17223 trim_cache size 0 max 16384 2024-04-01T01:39:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.624+0000 7f87e3032740 1 Processor -- start 2024-04-01T01:39:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.624+0000 7f87e3032740 1 -- start start 2024-04-01T01:39:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.624+0000 7f87e3032740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 0x55bfaac41220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.624+0000 7f87e3032740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac462c0 0x55bfaac70990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.625 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.624+0000 7f87e3032740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac4b430 0x55bfaac475e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.624+0000 7f87e3032740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc00fcc0 con 0x55bfaac462c0 2024-04-01T01:39:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.624+0000 7f87e3032740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc011800 con 0x55bfaac4b430 2024-04-01T01:39:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.624+0000 7f87e3032740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc010390 con 0x55bfaac458d0 2024-04-01T01:39:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.625+0000 7f87de914640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac4b430 0x55bfaac475e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.625+0000 7f87de914640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac4b430 0x55bfaac475e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54280/0 (socket says 172.21.15.70:54280) 2024-04-01T01:39:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.625+0000 7f87de914640 1 -- 172.21.15.70:0/3777433136 learned_addr learned my addr 172.21.15.70:0/3777433136 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.625+0000 7f87dd912640 1 --2- 172.21.15.70:0/3777433136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac462c0 0x55bfaac70990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.625+0000 7f87de113640 1 --2- 172.21.15.70:0/3777433136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 0x55bfaac41220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.625+0000 7f87de113640 1 -- 172.21.15.70:0/3777433136 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac4b430 msgr2=0x55bfaac475e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.625+0000 7f87de113640 1 --2- 172.21.15.70:0/3777433136 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac4b430 0x55bfaac475e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.625+0000 7f87de113640 1 -- 172.21.15.70:0/3777433136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac462c0 msgr2=0x55bfaac70990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.625+0000 7f87de113640 1 --2- 172.21.15.70:0/3777433136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac462c0 0x55bfaac70990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.625+0000 7f87de113640 1 -- 172.21.15.70:0/3777433136 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfaac466a0 con 0x55bfaac458d0 2024-04-01T01:39:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.625+0000 7f87dd912640 1 --2- 172.21.15.70:0/3777433136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac462c0 0x55bfaac70990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.625+0000 7f87de914640 1 --2- 172.21.15.70:0/3777433136 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac4b430 0x55bfaac475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.626+0000 7f87de113640 1 --2- 172.21.15.70:0/3777433136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 0x55bfaac41220 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f87d400d910 tx=0x7f87d4009230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d3633002dc45f43 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.626+0000 7f87c3fff640 1 -- 172.21.15.70:0/3777433136 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d405f070 con 0x55bfaac458d0 2024-04-01T01:39:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.626+0000 7f87c3fff640 1 -- 172.21.15.70:0/3777433136 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87d4032040 con 0x55bfaac458d0 2024-04-01T01:39:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.626+0000 7f87c3fff640 1 -- 172.21.15.70:0/3777433136 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4048720 con 0x55bfaac458d0 2024-04-01T01:39:16.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.627+0000 7f87e3032740 1 -- 172.21.15.70:0/3777433136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 msgr2=0x55bfaac41220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.627+0000 7f87e3032740 1 --2- 172.21.15.70:0/3777433136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 0x55bfaac41220 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f87d400d910 tx=0x7f87d4009230 comp rx=0 tx=0).stop 2024-04-01T01:39:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.627+0000 7f87e3032740 1 -- 172.21.15.70:0/3777433136 shutdown_connections 2024-04-01T01:39:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.627+0000 7f87e3032740 1 --2- 172.21.15.70:0/3777433136 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac4b430 0x55bfaac475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.627+0000 7f87e3032740 1 --2- 172.21.15.70:0/3777433136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac462c0 0x55bfaac70990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.627+0000 7f87e3032740 1 --2- 172.21.15.70:0/3777433136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 0x55bfaac41220 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.627+0000 7f87e3032740 1 -- 172.21.15.70:0/3777433136 >> 172.21.15.70:0/3777433136 conn(0x55bfaac40e40 msgr2=0x55bfaac3bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.627+0000 7f87e3032740 1 -- 172.21.15.70:0/3777433136 shutdown_connections 2024-04-01T01:39:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.627+0000 7f87e3032740 1 -- 172.21.15.70:0/3777433136 wait complete. 2024-04-01T01:39:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.628+0000 7f87e3032740 1 -- start start 2024-04-01T01:39:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.628+0000 7f87e3032740 1 Processor -- start 2024-04-01T01:39:16.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.628+0000 7f87e3032740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 0x55bfaac6deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.628+0000 7f87e3032740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 0x55bfaac6e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.628+0000 7f87e3032740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac4b430 0x55bfaac6f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.628+0000 7f87e3032740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc011130 con 0x55bfaac4b430 2024-04-01T01:39:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.628+0000 7f87e3032740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc010a60 con 0x55bfaac462c0 2024-04-01T01:39:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.628+0000 7f87e3032740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc00dca0 con 0x55bfaac458d0 2024-04-01T01:39:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.628+0000 7f87e3032740 10 client.0 _ll_register_callbacks cb 0x7f87d0009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.628+0000 7f87de113640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 0x55bfaac6deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.628+0000 7f87de113640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 0x55bfaac6deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42582/0 (socket says 172.21.15.70:42582) 2024-04-01T01:39:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.628+0000 7f87de113640 1 -- 172.21.15.70:0/2391800738 learned_addr learned my addr 172.21.15.70:0/2391800738 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.629+0000 7f87dd912640 1 --2- 172.21.15.70:0/2391800738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 0x55bfaac6e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.629+0000 7f87de914640 1 --2- 172.21.15.70:0/2391800738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac4b430 0x55bfaac6f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.629+0000 7f87de113640 1 -- 172.21.15.70:0/2391800738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 msgr2=0x55bfaac6e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:16.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.629+0000 7f87de113640 1 --2- 172.21.15.70:0/2391800738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 0x55bfaac6e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.629+0000 7f87de113640 1 -- 172.21.15.70:0/2391800738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac4b430 msgr2=0x55bfaac6f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.629+0000 7f87de113640 1 --2- 172.21.15.70:0/2391800738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac4b430 0x55bfaac6f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.629+0000 7f87de113640 1 -- 172.21.15.70:0/2391800738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfaac6f9f0 con 0x55bfaac458d0 2024-04-01T01:39:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.629+0000 7f87de914640 1 --2- 172.21.15.70:0/2391800738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac4b430 0x55bfaac6f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.629+0000 7f87de113640 1 --2- 172.21.15.70:0/2391800738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 0x55bfaac6deb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f87d405d040 tx=0x7f87d4005e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e10b2d1dc554dc00 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.629+0000 7f87c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:39:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.630+0000 7f87c1ffb640 1 -- 172.21.15.70:0/2391800738 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d405f070 con 0x55bfaac458d0 2024-04-01T01:39:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.630+0000 7f87c1ffb640 1 -- 172.21.15.70:0/2391800738 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f87d4036630 con 0x55bfaac458d0 2024-04-01T01:39:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.630+0000 7f87e3032740 1 -- 172.21.15.70:0/2391800738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87d4032040 con 0x55bfaac458d0 2024-04-01T01:39:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.630+0000 7f87c1ffb640 1 -- 172.21.15.70:0/2391800738 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f87d402eb00 con 0x55bfaac458d0 2024-04-01T01:39:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.630+0000 7f87e3032740 10 client.27010 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.630+0000 7f87e3032740 1 -- 172.21.15.70:0/2391800738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bfaac70d00 con 0x55bfaac458d0 2024-04-01T01:39:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.631+0000 7f87c1ffb640 1 -- 172.21.15.70:0/2391800738 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87d4043040 con 0x55bfaac458d0 2024-04-01T01:39:16.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.631+0000 7f87c1ffb640 1 -- 172.21.15.70:0/2391800738 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f87d4063450 con 0x55bfaac458d0 2024-04-01T01:39:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.631+0000 7f87e3032740 10 client.27010 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.631+0000 7f87e3032740 10 client.27010 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.631+0000 7f87e3032740 20 client.27010 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.631+0000 7f87e3032740 1 --2- 172.21.15.70:0/2391800738 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bfaac67090 0x55bfaac694b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.631+0000 7f87e3032740 4 client.27010 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:39:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.631+0000 7f87e3032740 1 -- 172.21.15.70:0/2391800738 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f87d4032040 con 0x55bfaac67090 2024-04-01T01:39:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.631+0000 7f87dd912640 1 --2- 172.21.15.70:0/2391800738 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bfaac67090 0x55bfaac694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.632+0000 7f87c1ffb640 10 client.27010 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:16.634 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.632+0000 7f87dd912640 1 --2- 172.21.15.70:0/2391800738 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bfaac67090 0x55bfaac694b0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11f67b83d274943c server_cookie=fd7b46b8dad6f29f in_seq=0 out_seq=0 2024-04-01T01:39:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.667+0000 7f87c1ffb640 1 -- 172.21.15.70:0/2391800738 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f87d4032040 con 0x55bfaac67090 2024-04-01T01:39:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.667+0000 7f87c1ffb640 10 client.27010 handle_command_reply: tid=0 2024-04-01T01:39:16.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.668+0000 7f87e3032740 1 client.27010 shutdown 2024-04-01T01:39:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.668+0000 7f87e3032740 1 -- 172.21.15.70:0/2391800738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 msgr2=0x55bfaac6deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.668+0000 7f87e3032740 1 --2- 172.21.15.70:0/2391800738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 0x55bfaac6deb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f87d405d040 tx=0x7f87d4005e30 comp rx=0 tx=0).stop 2024-04-01T01:39:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.668+0000 7f87e3032740 1 -- 172.21.15.70:0/2391800738 shutdown_connections 2024-04-01T01:39:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.668+0000 7f87e3032740 1 --2- 172.21.15.70:0/2391800738 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bfaac67090 0x55bfaac694b0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.668+0000 7f87e3032740 1 --2- 172.21.15.70:0/2391800738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfaac4b430 0x55bfaac6f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.668+0000 7f87e3032740 1 --2- 172.21.15.70:0/2391800738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bfaac462c0 0x55bfaac6e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.668+0000 7f87e3032740 1 --2- 172.21.15.70:0/2391800738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bfaac458d0 0x55bfaac6deb0 secure :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f87d405d040 tx=0x7f87d4005e30 comp rx=0 tx=0).stop 2024-04-01T01:39:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.668+0000 7f87e3032740 1 -- 172.21.15.70:0/2391800738 >> 172.21.15.70:0/2391800738 conn(0x55bfaac40e40 msgr2=0x55bfaac4a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.668+0000 7f87c1ffb640 0 client.27010 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.669+0000 7f87e3032740 1 -- 172.21.15.70:0/2391800738 shutdown_connections 2024-04-01T01:39:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.669+0000 7f87e3032740 1 -- 172.21.15.70:0/2391800738 wait complete. 2024-04-01T01:39:16.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.669+0000 7f87e3032740 20 client.27010 trim_cache size 0 max 16384 2024-04-01T01:39:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:39:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:39:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:39:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:39:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.670+0000 7f87e0377640 1 -- 172.21.15.70:0/2417346844 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f87c40a6a50 msgr2=0x7f87c40a8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.670+0000 7f87e0377640 1 --2- 172.21.15.70:0/2417346844 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f87c40a6a50 0x7f87c40a8f10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x55bfaac46b20 tx=0x7f87d0004ab0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.670+0000 7f87e0377640 1 -- 172.21.15.70:0/2417346844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87d8101860 msgr2=0x7f87d8195c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.670+0000 7f87e0377640 1 --2- 172.21.15.70:0/2417346844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87d8101860 0x7f87d8195c60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f87c800e9f0 tx=0x7f87c800eeb0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.670+0000 7f87e0377640 1 -- 172.21.15.70:0/2417346844 shutdown_connections 2024-04-01T01:39:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.670+0000 7f87e0377640 1 --2- 172.21.15.70:0/2417346844 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f87c40a6a50 0x7f87c40a8f10 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.670+0000 7f87e0377640 1 --2- 172.21.15.70:0/2417346844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87d8102220 0x7f87d819ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.670+0000 7f87e0377640 1 --2- 172.21.15.70:0/2417346844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87d8101860 0x7f87d8195c60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.670+0000 7f87e0377640 1 --2- 172.21.15.70:0/2417346844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87d8100670 0x7f87d8195720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.670+0000 7f87e0377640 1 -- 172.21.15.70:0/2417346844 >> 172.21.15.70:0/2417346844 conn(0x7f87d80fbbe0 msgr2=0x7f87d8104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.671+0000 7f87e0377640 1 -- 172.21.15.70:0/2417346844 shutdown_connections 2024-04-01T01:39:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:16.671+0000 7f87e0377640 1 -- 172.21.15.70:0/2417346844 wait complete. 2024-04-01T01:39:16.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 270.0 s, 1050.0 tps, lat 95.952 ms stddev 161.167 2024-04-01T01:39:17.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:16 smithi081 ceph-mon[35142]: pgmap v532: 129 pgs: 129 active+clean; 1.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 23 MiB/s wr, 1.03k op/s 2024-04-01T01:39:17.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:16 smithi189 ceph-mon[35753]: pgmap v532: 129 pgs: 129 active+clean; 1.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 23 MiB/s wr, 1.03k op/s 2024-04-01T01:39:17.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:16 smithi070 ceph-mon[32179]: pgmap v532: 129 pgs: 129 active+clean; 1.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 23 MiB/s wr, 1.03k op/s 2024-04-01T01:39:18.300 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:39:18.300 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:39:18.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.571+0000 7fa0dab81640 1 -- 172.21.15.70:0/3840351845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d410b0c0 msgr2=0x7fa0d410d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:18.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.571+0000 7fa0dab81640 1 --2- 172.21.15.70:0/3840351845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d410b0c0 0x7fa0d410d530 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fa0c801a910 tx=0x7fa0c803cca0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.575+0000 7fa0dab81640 1 -- 172.21.15.70:0/3840351845 shutdown_connections 2024-04-01T01:39:18.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.575+0000 7fa0dab81640 1 --2- 172.21.15.70:0/3840351845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d410da70 0x7fa0d410def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.576+0000 7fa0dab81640 1 --2- 172.21.15.70:0/3840351845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d410b0c0 0x7fa0d410d530 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.576+0000 7fa0dab81640 1 --2- 172.21.15.70:0/3840351845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0d4108750 0x7fa0d410ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.576+0000 7fa0dab81640 1 -- 172.21.15.70:0/3840351845 >> 172.21.15.70:0/3840351845 conn(0x7fa0d406e5b0 msgr2=0x7fa0d40709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:18.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.579+0000 7fa0dab81640 1 -- 172.21.15.70:0/3840351845 shutdown_connections 2024-04-01T01:39:18.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.583+0000 7fa0dab81640 1 -- 172.21.15.70:0/3840351845 wait complete. 2024-04-01T01:39:18.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.583+0000 7fa0dab81640 1 Processor -- start 2024-04-01T01:39:18.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.583+0000 7fa0dab81640 1 -- start start 2024-04-01T01:39:18.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.584+0000 7fa0dab81640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0d4108750 0x7fa0d419e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.584+0000 7fa0dab81640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d410b0c0 0x7fa0d419e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.584+0000 7fa0dab81640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d410da70 0x7fa0d41a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.584+0000 7fa0dab81640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d41119c0 con 0x7fa0d410da70 2024-04-01T01:39:18.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.584+0000 7fa0dab81640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d4111840 con 0x7fa0d410b0c0 2024-04-01T01:39:18.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.584+0000 7fa0dab81640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d4111b40 con 0x7fa0d4108750 2024-04-01T01:39:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.587+0000 7fa0d911e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d410da70 0x7fa0d41a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.587+0000 7fa0d911e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d410da70 0x7fa0d41a5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40398/0 (socket says 172.21.15.70:40398) 2024-04-01T01:39:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.587+0000 7fa0d911e640 1 -- 172.21.15.70:0/3494173426 learned_addr learned my addr 172.21.15.70:0/3494173426 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.587+0000 7fa0d911e640 1 -- 172.21.15.70:0/3494173426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0d4108750 msgr2=0x7fa0d419e070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:39:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.587+0000 7fa0d911e640 1 --2- 172.21.15.70:0/3494173426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0d4108750 0x7fa0d419e070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.587+0000 7fa0d911e640 1 -- 172.21.15.70:0/3494173426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d410b0c0 msgr2=0x7fa0d419e5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:39:18.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.587+0000 7fa0d911e640 1 --2- 172.21.15.70:0/3494173426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d410b0c0 0x7fa0d419e5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.587+0000 7fa0d911e640 1 -- 172.21.15.70:0/3494173426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d41a5c90 con 0x7fa0d410da70 2024-04-01T01:39:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.588+0000 7fa0d911e640 1 --2- 172.21.15.70:0/3494173426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d410da70 0x7fa0d41a5620 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa0c400b3c0 tx=0x7fa0c400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.589+0000 7fa0d1ffb640 1 -- 172.21.15.70:0/3494173426 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4011830 con 0x7fa0d410da70 2024-04-01T01:39:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.590+0000 7fa0d1ffb640 1 -- 172.21.15.70:0/3494173426 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0c4016070 con 0x7fa0d410da70 2024-04-01T01:39:18.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.590+0000 7fa0dab81640 1 -- 172.21.15.70:0/3494173426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d41a5f80 con 0x7fa0d410da70 2024-04-01T01:39:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.590+0000 7fa0dab81640 1 -- 172.21.15.70:0/3494173426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d41a64f0 con 0x7fa0d410da70 2024-04-01T01:39:18.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.592+0000 7fa0d1ffb640 1 -- 172.21.15.70:0/3494173426 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4010e50 con 0x7fa0d410da70 2024-04-01T01:39:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.597+0000 7fa0dd763740 1 Processor -- start 2024-04-01T01:39:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.599+0000 7fa0dd763740 1 -- start start 2024-04-01T01:39:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.599+0000 7fa0dd763740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a0232b0 0x56523a0236d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.599+0000 7fa0dd763740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a023ca0 0x56523a0287a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.599+0000 7fa0dd763740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a028e10 0x56523a02b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.599+0000 7fa0dd763740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x565239f88180 con 0x56523a023ca0 2024-04-01T01:39:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.599+0000 7fa0dd763740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x565239f30b70 con 0x56523a0232b0 2024-04-01T01:39:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.599+0000 7fa0dd763740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x565239f84220 con 0x56523a028e10 2024-04-01T01:39:18.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.599+0000 7fa0d1ffb640 1 -- 172.21.15.70:0/3494173426 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fa0c4010380 con 0x7fa0d410da70 2024-04-01T01:39:18.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.600+0000 7fa0d1ffb640 1 --2- 172.21.15.70:0/3494173426 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa0bc0a2850 0x7fa0bc0a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.601+0000 7fa0d1ffb640 1 -- 172.21.15.70:0/3494173426 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa0c40ce0c0 con 0x7fa0d410da70 2024-04-01T01:39:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.605+0000 7fa0d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a0232b0 0x56523a0236d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.605+0000 7fa0d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a0232b0 0x56523a0236d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54318/0 (socket says 172.21.15.70:54318) 2024-04-01T01:39:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.605+0000 7fa0d3fff640 1 -- 172.21.15.70:0/704289492 learned_addr learned my addr 172.21.15.70:0/704289492 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:18.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.605+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/704289492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a028e10 0x56523a02b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.606+0000 7fa0d911e640 1 --2- 172.21.15.70:0/3494173426 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa0bc0a2850 0x7fa0bc0a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.608+0000 7fa0d3fff640 1 -- 172.21.15.70:0/704289492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a028e10 msgr2=0x56523a02b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.608+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/704289492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a028e10 0x56523a02b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.608+0000 7fa0d3fff640 1 -- 172.21.15.70:0/704289492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a023ca0 msgr2=0x56523a0287a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.608+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/704289492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a023ca0 0x56523a0287a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.609+0000 7fa0d3fff640 1 -- 172.21.15.70:0/704289492 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565239f447c0 con 0x56523a0232b0 2024-04-01T01:39:18.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.609+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/704289492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a0232b0 0x56523a0236d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8004910 tx=0x7fa0c8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5964d4fbf023e723 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.610+0000 7fa0baffd640 1 -- 172.21.15.70:0/704289492 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8059070 con 0x56523a0232b0 2024-04-01T01:39:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.610+0000 7fa0baffd640 1 -- 172.21.15.70:0/704289492 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0c8004030 con 0x56523a0232b0 2024-04-01T01:39:18.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.610+0000 7fa0baffd640 1 -- 172.21.15.70:0/704289492 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c80052f0 con 0x56523a0232b0 2024-04-01T01:39:18.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.613+0000 7fa0dd763740 1 -- 172.21.15.70:0/704289492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a0232b0 msgr2=0x56523a0236d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:18.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.613+0000 7fa0dd763740 1 --2- 172.21.15.70:0/704289492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a0232b0 0x56523a0236d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8004910 tx=0x7fa0c8000bc0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.613+0000 7fa0d911e640 1 --2- 172.21.15.70:0/3494173426 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa0bc0a2850 0x7fa0bc0a4d10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa0c400bfd0 tx=0x7fa0c4011000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:18.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.616+0000 7fa0dd763740 1 -- 172.21.15.70:0/704289492 shutdown_connections 2024-04-01T01:39:18.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.616+0000 7fa0dd763740 1 --2- 172.21.15.70:0/704289492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a028e10 0x56523a02b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.616+0000 7fa0dd763740 1 --2- 172.21.15.70:0/704289492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a023ca0 0x56523a0287a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.616+0000 7fa0dd763740 1 --2- 172.21.15.70:0/704289492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a0232b0 0x56523a0236d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.616+0000 7fa0dd763740 1 -- 172.21.15.70:0/704289492 >> 172.21.15.70:0/704289492 conn(0x56523a01e820 msgr2=0x56523a01ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:18.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.621+0000 7fa0dd763740 1 -- 172.21.15.70:0/704289492 shutdown_connections 2024-04-01T01:39:18.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.627+0000 7fa0dd763740 1 -- 172.21.15.70:0/704289492 wait complete. 2024-04-01T01:39:18.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.628+0000 7fa0dd763740 1 -- start start 2024-04-01T01:39:18.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.628+0000 7fa0dd763740 1 Processor -- start 2024-04-01T01:39:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.628+0000 7fa0dd763740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a0232b0 0x56523a02d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.628+0000 7fa0dd763740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a023ca0 0x56523a02de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.628+0000 7fa0dd763740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a028e10 0x56523a034e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.628+0000 7fa0dd763740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56523a02bd60 con 0x56523a0232b0 2024-04-01T01:39:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.628+0000 7fa0dd763740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x565239f264a0 con 0x56523a028e10 2024-04-01T01:39:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.628+0000 7fa0dd763740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56523a02bee0 con 0x56523a023ca0 2024-04-01T01:39:18.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.629+0000 7fa0dd763740 10 client.0 _ll_register_callbacks cb 0x565239f29d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.631+0000 7fa0d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a028e10 0x56523a034e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.631+0000 7fa0d3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a0232b0 0x56523a02d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.631+0000 7fa0d3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a0232b0 0x56523a02d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40404/0 (socket says 172.21.15.70:40404) 2024-04-01T01:39:18.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.631+0000 7fa0d3fff640 1 -- 172.21.15.70:0/2120864184 learned_addr learned my addr 172.21.15.70:0/2120864184 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:18.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.632+0000 7fa0d3fff640 1 -- 172.21.15.70:0/2120864184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a023ca0 msgr2=0x56523a02de00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:39:18.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.632+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/2120864184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a023ca0 0x56523a02de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.632+0000 7fa0d3fff640 1 -- 172.21.15.70:0/2120864184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a0232b0 msgr2=0x56523a02d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:18.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.632+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/2120864184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a0232b0 0x56523a02d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.632+0000 7fa0d3fff640 1 -- 172.21.15.70:0/2120864184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565239d667d0 con 0x56523a028e10 2024-04-01T01:39:18.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.632+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/2120864184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a028e10 0x56523a034e40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8004a50 tx=0x7fa0c803f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90acd77cc43da114 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:18.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.633+0000 7fa0b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:18.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.634+0000 7fa0b3fff640 1 -- 172.21.15.70:0/2120864184 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8059040 con 0x56523a028e10 2024-04-01T01:39:18.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.634+0000 7fa0b3fff640 1 -- 172.21.15.70:0/2120864184 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa0c8005070 con 0x56523a028e10 2024-04-01T01:39:18.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.635+0000 7fa0dd763740 1 -- 172.21.15.70:0/2120864184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0c8004030 con 0x56523a028e10 2024-04-01T01:39:18.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.635+0000 7fa0b3fff640 1 -- 172.21.15.70:0/2120864184 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa0c8005270 con 0x56523a028e10 2024-04-01T01:39:18.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.636+0000 7fa0dd763740 10 client.27146 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:18.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.636+0000 7fa0dd763740 1 -- 172.21.15.70:0/2120864184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x565239c9c3f0 con 0x56523a028e10 2024-04-01T01:39:18.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.638+0000 7fa0b3fff640 1 -- 172.21.15.70:0/2120864184 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0c8004030 con 0x56523a028e10 2024-04-01T01:39:18.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.639+0000 7fa0b3fff640 1 -- 172.21.15.70:0/2120864184 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa0c805f040 con 0x56523a028e10 2024-04-01T01:39:18.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.639+0000 7fa0dd763740 10 client.27146 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:18.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.639+0000 7fa0dd763740 10 client.27146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:18.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.639+0000 7fa0dd763740 20 client.27146 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:18.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.640+0000 7fa0dd763740 1 --2- 172.21.15.70:0/2120864184 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56523a0397d0 0x56523a03bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.640+0000 7fa0dd763740 4 client.27146 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:18.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.640+0000 7fa0dd763740 1 -- 172.21.15.70:0/2120864184 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565239f956c0 con 0x56523a0397d0 2024-04-01T01:39:18.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.643+0000 7fa0d891d640 1 --2- 172.21.15.70:0/2120864184 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56523a0397d0 0x56523a03bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:18.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.646+0000 7fa0d891d640 1 --2- 172.21.15.70:0/2120864184 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56523a0397d0 0x56523a03bc10 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7660ed8ed1969a02 server_cookie=b446684c23bbd02 in_seq=0 out_seq=0 2024-04-01T01:39:18.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.646+0000 7fa0b3fff640 10 client.27146 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.651+0000 7fa0b3fff640 1 -- 172.21.15.70:0/2120864184 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x565239f956c0 con 0x56523a0397d0 2024-04-01T01:39:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.651+0000 7fa0b3fff640 10 client.27146 handle_command_reply: tid=0 2024-04-01T01:39:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.651+0000 7fa0dd763740 1 client.27146 shutdown 2024-04-01T01:39:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.651+0000 7fa0dd763740 1 -- 172.21.15.70:0/2120864184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a028e10 msgr2=0x56523a034e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:18.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.651+0000 7fa0dd763740 1 --2- 172.21.15.70:0/2120864184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a028e10 0x56523a034e40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8004a50 tx=0x7fa0c803f040 comp rx=0 tx=0).stop 2024-04-01T01:39:18.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.652+0000 7fa0dd763740 1 -- 172.21.15.70:0/2120864184 shutdown_connections 2024-04-01T01:39:18.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.652+0000 7fa0dd763740 1 --2- 172.21.15.70:0/2120864184 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56523a0397d0 0x56523a03bc10 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.652+0000 7fa0dd763740 1 --2- 172.21.15.70:0/2120864184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a028e10 0x56523a034e40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.652+0000 7fa0dd763740 1 --2- 172.21.15.70:0/2120864184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a023ca0 0x56523a02de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.652+0000 7fa0dd763740 1 --2- 172.21.15.70:0/2120864184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a0232b0 0x56523a02d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.653+0000 7fa0dd763740 1 -- 172.21.15.70:0/2120864184 >> 172.21.15.70:0/2120864184 conn(0x56523a01e820 msgr2=0x56523a01dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:18.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.653+0000 7fa0b3fff640 0 client.27146 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:18.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.654+0000 7fa0dd763740 1 -- 172.21.15.70:0/2120864184 shutdown_connections 2024-04-01T01:39:18.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.655+0000 7fa0dd763740 1 -- 172.21.15.70:0/2120864184 wait complete. 2024-04-01T01:39:18.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.655+0000 7fa0dd763740 20 client.27146 trim_cache size 0 max 16384 2024-04-01T01:39:18.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.689+0000 7fa0dd763740 1 Processor -- start 2024-04-01T01:39:18.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.689+0000 7fa0dd763740 1 -- start start 2024-04-01T01:39:18.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.689+0000 7fa0dd763740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a0232b0 0x56523a025270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.689+0000 7fa0dd763740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a023ca0 0x56523a0257b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.689+0000 7fa0dd763740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a028e10 0x56523a034dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.689+0000 7fa0dd763740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c000fc10 con 0x56523a023ca0 2024-04-01T01:39:18.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.689+0000 7fa0dd763740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c0011750 con 0x56523a0232b0 2024-04-01T01:39:18.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.689+0000 7fa0dd763740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c00102e0 con 0x56523a028e10 2024-04-01T01:39:18.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.689+0000 7fa0d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a0232b0 0x56523a025270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:18.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.689+0000 7fa0d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a0232b0 0x56523a025270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54330/0 (socket says 172.21.15.70:54330) 2024-04-01T01:39:18.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.690+0000 7fa0d3fff640 1 -- 172.21.15.70:0/1705078354 learned_addr learned my addr 172.21.15.70:0/1705078354 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:18.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.690+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/1705078354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a028e10 0x56523a034dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:18.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.690+0000 7fa0d891d640 1 --2- 172.21.15.70:0/1705078354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a023ca0 0x56523a0257b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:18.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.691+0000 7fa0d3fff640 1 -- 172.21.15.70:0/1705078354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a028e10 msgr2=0x56523a034dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:18.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.691+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/1705078354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a028e10 0x56523a034dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.691+0000 7fa0d3fff640 1 -- 172.21.15.70:0/1705078354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a023ca0 msgr2=0x56523a0257b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:18.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.691+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/1705078354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a023ca0 0x56523a0257b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.691+0000 7fa0d3fff640 1 -- 172.21.15.70:0/1705078354 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565239c5c0f0 con 0x56523a0232b0 2024-04-01T01:39:18.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.691+0000 7fa0d891d640 1 --2- 172.21.15.70:0/1705078354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a023ca0 0x56523a0257b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:18.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.693+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/1705078354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a0232b0 0x56523a025270 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa0c803d9d0 tx=0x7fa0c8001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b9af4f8c62698f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:18.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.697+0000 7fa0ba7fc640 1 -- 172.21.15.70:0/1705078354 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8003ff0 con 0x56523a0232b0 2024-04-01T01:39:18.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.697+0000 7fa0ba7fc640 1 -- 172.21.15.70:0/1705078354 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0c8054d70 con 0x56523a0232b0 2024-04-01T01:39:18.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.697+0000 7fa0ba7fc640 1 -- 172.21.15.70:0/1705078354 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c80055b0 con 0x56523a0232b0 2024-04-01T01:39:18.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.698+0000 7fa0dd763740 1 -- 172.21.15.70:0/1705078354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a0232b0 msgr2=0x56523a025270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:18.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.698+0000 7fa0dd763740 1 --2- 172.21.15.70:0/1705078354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a0232b0 0x56523a025270 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa0c803d9d0 tx=0x7fa0c8001e70 comp rx=0 tx=0).stop 2024-04-01T01:39:18.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.699+0000 7fa0dd763740 1 -- 172.21.15.70:0/1705078354 shutdown_connections 2024-04-01T01:39:18.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.699+0000 7fa0dd763740 1 --2- 172.21.15.70:0/1705078354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a028e10 0x56523a034dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.699+0000 7fa0dd763740 1 --2- 172.21.15.70:0/1705078354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a023ca0 0x56523a0257b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.699+0000 7fa0dd763740 1 --2- 172.21.15.70:0/1705078354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a0232b0 0x56523a025270 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.699+0000 7fa0dd763740 1 -- 172.21.15.70:0/1705078354 >> 172.21.15.70:0/1705078354 conn(0x56523a01e820 msgr2=0x56523a01ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:18.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.702+0000 7fa0dd763740 1 -- 172.21.15.70:0/1705078354 shutdown_connections 2024-04-01T01:39:18.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.704+0000 7fa0dd763740 1 -- 172.21.15.70:0/1705078354 wait complete. 2024-04-01T01:39:18.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.704+0000 7fa0dd763740 1 -- start start 2024-04-01T01:39:18.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.704+0000 7fa0dd763740 1 Processor -- start 2024-04-01T01:39:18.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.708+0000 7fa0dd763740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a0232b0 0x56523a031b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.708+0000 7fa0dd763740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a023ca0 0x56523a0320d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.708+0000 7fa0dd763740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a028e10 0x56523a09db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.708+0000 7fa0dd763740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c0011080 con 0x56523a028e10 2024-04-01T01:39:18.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.708+0000 7fa0dd763740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c00109b0 con 0x56523a023ca0 2024-04-01T01:39:18.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.708+0000 7fa0dd763740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c000dca0 con 0x56523a0232b0 2024-04-01T01:39:18.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.708+0000 7fa0dd763740 10 client.0 _ll_register_callbacks cb 0x7fa0c8004ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:18.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.709+0000 7fa0d891d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a023ca0 0x56523a0320d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:18.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.709+0000 7fa0d891d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a023ca0 0x56523a0320d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54344/0 (socket says 172.21.15.70:54344) 2024-04-01T01:39:18.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.709+0000 7fa0d891d640 1 -- 172.21.15.70:0/1091598141 learned_addr learned my addr 172.21.15.70:0/1091598141 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:18.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.711+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/1091598141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a028e10 0x56523a09db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:18.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.712+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/1091598141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a0232b0 0x56523a031b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:18.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.713+0000 7fa0d891d640 1 -- 172.21.15.70:0/1091598141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a0232b0 msgr2=0x56523a031b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:18.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.713+0000 7fa0d891d640 1 --2- 172.21.15.70:0/1091598141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a0232b0 0x56523a031b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.713+0000 7fa0d891d640 1 -- 172.21.15.70:0/1091598141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a028e10 msgr2=0x56523a09db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:18.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.713+0000 7fa0d891d640 1 --2- 172.21.15.70:0/1091598141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a028e10 0x56523a09db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.713+0000 7fa0d891d640 1 -- 172.21.15.70:0/1091598141 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56523a09e040 con 0x56523a023ca0 2024-04-01T01:39:18.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.713+0000 7fa0d891d640 1 --2- 172.21.15.70:0/1091598141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a023ca0 0x56523a0320d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x56523a01f790 tx=0x7fa0cc00afd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79826ae7dc6963f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:18.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.714+0000 7fa0b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:18.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.715+0000 7fa0b3fff640 1 -- 172.21.15.70:0/1091598141 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc017370 con 0x56523a023ca0 2024-04-01T01:39:18.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.715+0000 7fa0dd763740 1 -- 172.21.15.70:0/1091598141 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0c8054d70 con 0x56523a023ca0 2024-04-01T01:39:18.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.715+0000 7fa0b3fff640 1 -- 172.21.15.70:0/1091598141 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa0cc017510 con 0x56523a023ca0 2024-04-01T01:39:18.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.716+0000 7fa0b3fff640 1 -- 172.21.15.70:0/1091598141 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa0cc028050 con 0x56523a023ca0 2024-04-01T01:39:18.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.719+0000 7fa0dd763740 10 client.27152 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:18.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.719+0000 7fa0dd763740 1 -- 172.21.15.70:0/1091598141 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56523a09f280 con 0x56523a023ca0 2024-04-01T01:39:18.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.720+0000 7fa0b3fff640 1 -- 172.21.15.70:0/1091598141 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0c8054d70 con 0x56523a023ca0 2024-04-01T01:39:18.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.720+0000 7fa0b3fff640 1 -- 172.21.15.70:0/1091598141 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa0cc00c3f0 con 0x56523a023ca0 2024-04-01T01:39:18.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.720+0000 7fa0dd763740 10 client.27152 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:18.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.720+0000 7fa0dd763740 10 client.27152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:18.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.720+0000 7fa0dd763740 20 client.27152 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:18.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.720+0000 7fa0dd763740 1 --2- 172.21.15.70:0/1091598141 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56523a09fc90 0x56523a0a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.721+0000 7fa0dd763740 4 client.27152 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:39:18.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.721+0000 7fa0dd763740 1 -- 172.21.15.70:0/1091598141 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x565239f8fa10 con 0x56523a09fc90 2024-04-01T01:39:18.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.721+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/1091598141 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56523a09fc90 0x56523a0a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:18.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.723+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/1091598141 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56523a09fc90 0x56523a0a20b0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=228b68e5d2fa6992 server_cookie=b372ba9d14ab0da4 in_seq=0 out_seq=0 2024-04-01T01:39:18.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.723+0000 7fa0b3fff640 10 client.27152 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:18.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.723+0000 7fa0b3fff640 1 -- 172.21.15.70:0/1091598141 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x565239f8fa10 con 0x56523a09fc90 2024-04-01T01:39:18.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.723+0000 7fa0b3fff640 10 client.27152 handle_command_reply: tid=0 2024-04-01T01:39:18.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.723+0000 7fa0dd763740 1 client.27152 shutdown 2024-04-01T01:39:18.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.724+0000 7fa0dd763740 1 -- 172.21.15.70:0/1091598141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a023ca0 msgr2=0x56523a0320d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:18.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.724+0000 7fa0dd763740 1 --2- 172.21.15.70:0/1091598141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a023ca0 0x56523a0320d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x56523a01f790 tx=0x7fa0cc00afd0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.724+0000 7fa0dd763740 1 -- 172.21.15.70:0/1091598141 shutdown_connections 2024-04-01T01:39:18.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.724+0000 7fa0dd763740 1 --2- 172.21.15.70:0/1091598141 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56523a09fc90 0x56523a0a20b0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.724+0000 7fa0dd763740 1 --2- 172.21.15.70:0/1091598141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56523a028e10 0x56523a09db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.724+0000 7fa0dd763740 1 --2- 172.21.15.70:0/1091598141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56523a023ca0 0x56523a0320d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.724+0000 7fa0dd763740 1 --2- 172.21.15.70:0/1091598141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56523a0232b0 0x56523a031b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.724+0000 7fa0dd763740 1 -- 172.21.15.70:0/1091598141 >> 172.21.15.70:0/1091598141 conn(0x56523a01e820 msgr2=0x56523a02d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:18.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.724+0000 7fa0b3fff640 0 client.27152 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:18.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.724+0000 7fa0dd763740 1 -- 172.21.15.70:0/1091598141 shutdown_connections 2024-04-01T01:39:18.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.725+0000 7fa0dd763740 1 -- 172.21.15.70:0/1091598141 wait complete. 2024-04-01T01:39:18.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.725+0000 7fa0dd763740 20 client.27152 trim_cache size 0 max 16384 2024-04-01T01:39:18.726 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:39:18.726 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:39:18.726 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:39:18.726 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:39:18.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.726+0000 7fa0dab81640 1 -- 172.21.15.70:0/3494173426 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa0bc0a2850 msgr2=0x7fa0bc0a4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:18.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.726+0000 7fa0dab81640 1 --2- 172.21.15.70:0/3494173426 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa0bc0a2850 0x7fa0bc0a4d10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa0c400bfd0 tx=0x7fa0c4011000 comp rx=0 tx=0).stop 2024-04-01T01:39:18.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.726+0000 7fa0dab81640 1 -- 172.21.15.70:0/3494173426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d410da70 msgr2=0x7fa0d41a5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:18.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.726+0000 7fa0dab81640 1 --2- 172.21.15.70:0/3494173426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d410da70 0x7fa0d41a5620 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa0c400b3c0 tx=0x7fa0c400b880 comp rx=0 tx=0).stop 2024-04-01T01:39:18.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.727+0000 7fa0dab81640 1 -- 172.21.15.70:0/3494173426 shutdown_connections 2024-04-01T01:39:18.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.727+0000 7fa0dab81640 1 --2- 172.21.15.70:0/3494173426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d410da70 0x7fa0d41a5620 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.727+0000 7fa0dab81640 1 --2- 172.21.15.70:0/3494173426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d410b0c0 0x7fa0d419e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.727+0000 7fa0dab81640 1 --2- 172.21.15.70:0/3494173426 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa0bc0a2850 0x7fa0bc0a4d10 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.727+0000 7fa0dab81640 1 --2- 172.21.15.70:0/3494173426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0d4108750 0x7fa0d419e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.727+0000 7fa0dab81640 1 -- 172.21.15.70:0/3494173426 >> 172.21.15.70:0/3494173426 conn(0x7fa0d406e5b0 msgr2=0x7fa0d410b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:18.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.727+0000 7fa0dab81640 1 -- 172.21.15.70:0/3494173426 shutdown_connections 2024-04-01T01:39:18.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.727+0000 7fa0dab81640 1 -- 172.21.15.70:0/3494173426 wait complete. 2024-04-01T01:39:18.747 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:39:18.747 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:39:18.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.924+0000 7f9cef03b640 1 -- 172.21.15.70:0/3014544658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce810d880 msgr2=0x7f9ce810dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:18.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.924+0000 7f9cef03b640 1 --2- 172.21.15.70:0/3014544658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce810d880 0x7f9ce810dca0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc009900 tx=0x7f9cdc02bc40 comp rx=0 tx=0).stop 2024-04-01T01:39:18.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.924+0000 7f9cef03b640 1 -- 172.21.15.70:0/3014544658 shutdown_connections 2024-04-01T01:39:18.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.924+0000 7f9cef03b640 1 --2- 172.21.15.70:0/3014544658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ce810c770 0x7f9ce81145e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.924+0000 7f9cef03b640 1 --2- 172.21.15.70:0/3014544658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ce810bdb0 0x7f9ce810c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.924+0000 7f9cef03b640 1 --2- 172.21.15.70:0/3014544658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce810d880 0x7f9ce810dca0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.924+0000 7f9cef03b640 1 -- 172.21.15.70:0/3014544658 >> 172.21.15.70:0/3014544658 conn(0x7f9ce806c190 msgr2=0x7f9ce806c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:18.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.925+0000 7f9cef03b640 1 -- 172.21.15.70:0/3014544658 shutdown_connections 2024-04-01T01:39:18.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.925+0000 7f9cef03b640 1 -- 172.21.15.70:0/3014544658 wait complete. 2024-04-01T01:39:18.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.926+0000 7f9cef03b640 1 Processor -- start 2024-04-01T01:39:18.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.927+0000 7f9cef03b640 1 -- start start 2024-04-01T01:39:18.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.927+0000 7f9cef03b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce810bdb0 0x7f9ce81a24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.927+0000 7f9cef03b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ce810c770 0x7f9ce81a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.927+0000 7f9cef03b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ce810d880 0x7f9ce81a9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.927+0000 7f9cef03b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce8117da0 con 0x7f9ce810d880 2024-04-01T01:39:18.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.927+0000 7f9cef03b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce8117c20 con 0x7f9ce810c770 2024-04-01T01:39:18.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.927+0000 7f9cef03b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce8117f20 con 0x7f9ce810bdb0 2024-04-01T01:39:18.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.928+0000 7f9ced5d8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ce810d880 0x7f9ce81a9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:18.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.928+0000 7f9ced5d8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ce810d880 0x7f9ce81a9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40444/0 (socket says 172.21.15.70:40444) 2024-04-01T01:39:18.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.928+0000 7f9ced5d8640 1 -- 172.21.15.70:0/1699454336 learned_addr learned my addr 172.21.15.70:0/1699454336 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:18.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.928+0000 7f9ced5d8640 1 -- 172.21.15.70:0/1699454336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce810bdb0 msgr2=0x7f9ce81a24f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:39:18.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.928+0000 7f9ced5d8640 1 --2- 172.21.15.70:0/1699454336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce810bdb0 0x7f9ce81a24f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.928+0000 7f9ced5d8640 1 -- 172.21.15.70:0/1699454336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ce810c770 msgr2=0x7f9ce81a2a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:39:18.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.928+0000 7f9ced5d8640 1 --2- 172.21.15.70:0/1699454336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ce810c770 0x7f9ce81a2a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:18.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.928+0000 7f9ced5d8640 1 -- 172.21.15.70:0/1699454336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ce81aa110 con 0x7f9ce810d880 2024-04-01T01:39:18.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.929+0000 7f9ced5d8640 1 --2- 172.21.15.70:0/1699454336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ce810d880 0x7f9ce81a9aa0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9ce000eb10 tx=0x7f9ce000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:18.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.929+0000 7f9ce5ffb640 1 -- 172.21.15.70:0/1699454336 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0004b20 con 0x7f9ce810d880 2024-04-01T01:39:18.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.929+0000 7f9cef03b640 1 -- 172.21.15.70:0/1699454336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ce81aa400 con 0x7f9ce810d880 2024-04-01T01:39:18.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.930+0000 7f9ce5ffb640 1 -- 172.21.15.70:0/1699454336 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9ce0004cc0 con 0x7f9ce810d880 2024-04-01T01:39:18.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.930+0000 7f9ce5ffb640 1 -- 172.21.15.70:0/1699454336 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0010690 con 0x7f9ce810d880 2024-04-01T01:39:18.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.930+0000 7f9cef03b640 1 -- 172.21.15.70:0/1699454336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ce81aa8f0 con 0x7f9ce810d880 2024-04-01T01:39:18.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.932+0000 7f9cef03b640 1 -- 172.21.15.70:0/1699454336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cd40052c0 con 0x7f9ce810d880 2024-04-01T01:39:18.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.933+0000 7f9ce5ffb640 1 -- 172.21.15.70:0/1699454336 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f9ce0010830 con 0x7f9ce810d880 2024-04-01T01:39:18.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.933+0000 7f9ce5ffb640 1 --2- 172.21.15.70:0/1699454336 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9cd00a2520 0x7f9cd00a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:18.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.934+0000 7f9ce5ffb640 1 -- 172.21.15.70:0/1699454336 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9ce00cf1d0 con 0x7f9ce810d880 2024-04-01T01:39:18.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.934+0000 7f9cecdd7640 1 --2- 172.21.15.70:0/1699454336 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9cd00a2520 0x7f9cd00a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:18.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.934+0000 7f9cecdd7640 1 --2- 172.21.15.70:0/1699454336 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9cd00a2520 0x7f9cd00a49e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc036040 tx=0x7f9cdc005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:18.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:18.941+0000 7f9ce5ffb640 1 -- 172.21.15.70:0/1699454336 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9ce009bf60 con 0x7f9ce810d880 2024-04-01T01:39:19.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.293+0000 7f9cef03b640 1 -- 172.21.15.70:0/1699454336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cd40055b0 con 0x7f9ce810d880 2024-04-01T01:39:19.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.294+0000 7f9ce5ffb640 1 -- 172.21.15.70:0/1699454336 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11929 (secure 0 0 0) 0x7f9ce009b860 con 0x7f9ce810d880 2024-04-01T01:39:19.294 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:39:19.294 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:33:36.894838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/2296387110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2296387110},{"type":"v1","addr":"172.21.15.189:6837","nonce":2296387110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24418,24404,24379,24451,24464]},"id":1}]} 2024-04-01T01:39:19.294 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 29 2024-04-01T01:39:19.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.297+0000 7f9cef03b640 1 -- 172.21.15.70:0/1699454336 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9cd00a2520 msgr2=0x7f9cd00a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.297+0000 7f9cef03b640 1 --2- 172.21.15.70:0/1699454336 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9cd00a2520 0x7f9cd00a49e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc036040 tx=0x7f9cdc005fd0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.297+0000 7f9cef03b640 1 -- 172.21.15.70:0/1699454336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ce810d880 msgr2=0x7f9ce81a9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.297+0000 7f9cef03b640 1 --2- 172.21.15.70:0/1699454336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ce810d880 0x7f9ce81a9aa0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9ce000eb10 tx=0x7f9ce000c2d0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.297+0000 7f9cef03b640 1 -- 172.21.15.70:0/1699454336 shutdown_connections 2024-04-01T01:39:19.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.297+0000 7f9cef03b640 1 --2- 172.21.15.70:0/1699454336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ce810d880 0x7f9ce81a9aa0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.297+0000 7f9cef03b640 1 --2- 172.21.15.70:0/1699454336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ce810c770 0x7f9ce81a2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.297+0000 7f9cef03b640 1 --2- 172.21.15.70:0/1699454336 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9cd00a2520 0x7f9cd00a49e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.297+0000 7f9cef03b640 1 --2- 172.21.15.70:0/1699454336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ce810bdb0 0x7f9ce81a24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.297+0000 7f9cef03b640 1 -- 172.21.15.70:0/1699454336 >> 172.21.15.70:0/1699454336 conn(0x7f9ce806c190 msgr2=0x7f9ce8110760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.297+0000 7f9cef03b640 1 -- 172.21.15.70:0/1699454336 shutdown_connections 2024-04-01T01:39:19.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.297+0000 7f9cef03b640 1 -- 172.21.15.70:0/1699454336 wait complete. 2024-04-01T01:39:19.313 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:39:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.481+0000 7fb0ef015640 1 -- 172.21.15.70:0/178244495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0e8108750 msgr2=0x7fb0e810ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.481+0000 7fb0ef015640 1 --2- 172.21.15.70:0/178244495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0e8108750 0x7fb0e810ab80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0009900 tx=0x7fb0e002bc30 comp rx=0 tx=0).stop 2024-04-01T01:39:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.481+0000 7fb0ef015640 1 -- 172.21.15.70:0/178244495 shutdown_connections 2024-04-01T01:39:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.481+0000 7fb0ef015640 1 --2- 172.21.15.70:0/178244495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e810da70 0x7fb0e810def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.481+0000 7fb0ef015640 1 --2- 172.21.15.70:0/178244495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0e810b0c0 0x7fb0e810d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.481+0000 7fb0ef015640 1 --2- 172.21.15.70:0/178244495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0e8108750 0x7fb0e810ab80 secure :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0009900 tx=0x7fb0e002bc30 comp rx=0 tx=0).stop 2024-04-01T01:39:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.481+0000 7fb0ef015640 1 -- 172.21.15.70:0/178244495 >> 172.21.15.70:0/178244495 conn(0x7fb0e806e5b0 msgr2=0x7fb0e80709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.481+0000 7fb0ef015640 1 -- 172.21.15.70:0/178244495 shutdown_connections 2024-04-01T01:39:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.481+0000 7fb0ef015640 1 -- 172.21.15.70:0/178244495 wait complete. 2024-04-01T01:39:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.482+0000 7fb0ef015640 1 Processor -- start 2024-04-01T01:39:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.482+0000 7fb0ef015640 1 -- start start 2024-04-01T01:39:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.482+0000 7fb0ef015640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0e810b0c0 0x7fb0e8115190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.482+0000 7fb0ef015640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0e810da70 0x7fb0e81156d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.482+0000 7fb0ef015640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e8116710 0x7fb0e8116bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.482+0000 7fb0ef015640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e811c100 con 0x7fb0e8116710 2024-04-01T01:39:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.482+0000 7fb0ef015640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e811bf80 con 0x7fb0e810da70 2024-04-01T01:39:19.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.482+0000 7fb0ef015640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e811c280 con 0x7fb0e810b0c0 2024-04-01T01:39:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.483+0000 7fb0ecdb1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0e810b0c0 0x7fb0e8115190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.483+0000 7fb0e7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0e810da70 0x7fb0e81156d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.483+0000 7fb0e7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0e810da70 0x7fb0e81156d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54376/0 (socket says 172.21.15.70:54376) 2024-04-01T01:39:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.483+0000 7fb0e7fff640 1 -- 172.21.15.70:0/1064059857 learned_addr learned my addr 172.21.15.70:0/1064059857 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.483+0000 7fb0ed5b2640 1 --2- 172.21.15.70:0/1064059857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e8116710 0x7fb0e8116bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.483+0000 7fb0e7fff640 1 -- 172.21.15.70:0/1064059857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0e810b0c0 msgr2=0x7fb0e8115190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.483+0000 7fb0e7fff640 1 --2- 172.21.15.70:0/1064059857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0e810b0c0 0x7fb0e8115190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.483+0000 7fb0e7fff640 1 -- 172.21.15.70:0/1064059857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e8116710 msgr2=0x7fb0e8116bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.483+0000 7fb0e7fff640 1 --2- 172.21.15.70:0/1064059857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e8116710 0x7fb0e8116bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.483+0000 7fb0e7fff640 1 -- 172.21.15.70:0/1064059857 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e8117180 con 0x7fb0e810da70 2024-04-01T01:39:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.484+0000 7fb0ecdb1640 1 --2- 172.21.15.70:0/1064059857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0e810b0c0 0x7fb0e8115190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.484+0000 7fb0e7fff640 1 --2- 172.21.15.70:0/1064059857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0e810da70 0x7fb0e81156d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb0d800d950 tx=0x7fb0d800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.484+0000 7fb0ed5b2640 1 --2- 172.21.15.70:0/1064059857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e8116710 0x7fb0e8116bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.484+0000 7fb0e5ffb640 1 -- 172.21.15.70:0/1064059857 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8017070 con 0x7fb0e810da70 2024-04-01T01:39:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.484+0000 7fb0e5ffb640 1 -- 172.21.15.70:0/1064059857 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0d8009c30 con 0x7fb0e810da70 2024-04-01T01:39:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.484+0000 7fb0e5ffb640 1 -- 172.21.15.70:0/1064059857 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8017070 con 0x7fb0e810da70 2024-04-01T01:39:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.484+0000 7fb0ef015640 1 -- 172.21.15.70:0/1064059857 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e8117470 con 0x7fb0e810da70 2024-04-01T01:39:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.484+0000 7fb0ef015640 1 -- 172.21.15.70:0/1064059857 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e8117970 con 0x7fb0e810da70 2024-04-01T01:39:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.488+0000 7fb0e5ffb640 1 -- 172.21.15.70:0/1064059857 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb0d8010460 con 0x7fb0e810da70 2024-04-01T01:39:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.488+0000 7fb0f1bf6740 1 Processor -- start 2024-04-01T01:39:19.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.488+0000 7fb0f1bf6740 1 -- start start 2024-04-01T01:39:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.488+0000 7fb0f1bf6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd5e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.488+0000 7fb0f1bf6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd5ee80 0x561f6dd63980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.488+0000 7fb0f1bf6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd63ff0 0x561f6dd66480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.488+0000 7fb0f1bf6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561f6dcc31b0 con 0x561f6dd5ee80 2024-04-01T01:39:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.488+0000 7fb0f1bf6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561f6dc6bc00 con 0x561f6dd5e490 2024-04-01T01:39:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.488+0000 7fb0f1bf6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561f6dcbf250 con 0x561f6dd63ff0 2024-04-01T01:39:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.488+0000 7fb0ed5b2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd5e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.489+0000 7fb0ed5b2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd5e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54390/0 (socket says 172.21.15.70:54390) 2024-04-01T01:39:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.489+0000 7fb0ed5b2640 1 -- 172.21.15.70:0/2282984716 learned_addr learned my addr 172.21.15.70:0/2282984716 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.489+0000 7fb0e7fff640 1 --2- 172.21.15.70:0/2282984716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd63ff0 0x561f6dd66480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.489+0000 7fb0ecdb1640 1 --2- 172.21.15.70:0/2282984716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd5ee80 0x561f6dd63980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.489+0000 7fb0ed5b2640 1 -- 172.21.15.70:0/2282984716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd63ff0 msgr2=0x561f6dd66480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.489+0000 7fb0ed5b2640 1 --2- 172.21.15.70:0/2282984716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd63ff0 0x561f6dd66480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.489+0000 7fb0ed5b2640 1 -- 172.21.15.70:0/2282984716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd5ee80 msgr2=0x561f6dd63980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.489+0000 7fb0ed5b2640 1 --2- 172.21.15.70:0/2282984716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd5ee80 0x561f6dd63980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.489+0000 7fb0ed5b2640 1 -- 172.21.15.70:0/2282984716 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f6dc8b3d0 con 0x561f6dd5e490 2024-04-01T01:39:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.489+0000 7fb0ed5b2640 1 --2- 172.21.15.70:0/2282984716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd5e8b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0009ef0 tx=0x7fb0d0010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bad06cf628046f92 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.490+0000 7fb0e5ffb640 1 --2- 172.21.15.70:0/1064059857 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb0d40a2720 0x7fb0d40a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.490+0000 7fb0deffd640 1 -- 172.21.15.70:0/2282984716 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0016070 con 0x561f6dd5e490 2024-04-01T01:39:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.490+0000 7fb0deffd640 1 -- 172.21.15.70:0/2282984716 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0d001bb10 con 0x561f6dd5e490 2024-04-01T01:39:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.490+0000 7fb0deffd640 1 -- 172.21.15.70:0/2282984716 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d0012ac0 con 0x561f6dd5e490 2024-04-01T01:39:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.490+0000 7fb0e5ffb640 1 -- 172.21.15.70:0/1064059857 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb0d80ce140 con 0x7fb0e810da70 2024-04-01T01:39:19.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.490+0000 7fb0ecdb1640 1 --2- 172.21.15.70:0/1064059857 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb0d40a2720 0x7fb0d40a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.491+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/2282984716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 msgr2=0x561f6dd5e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.491+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/2282984716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd5e8b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0009ef0 tx=0x7fb0d0010740 comp rx=0 tx=0).stop 2024-04-01T01:39:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.491+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/2282984716 shutdown_connections 2024-04-01T01:39:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.491+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/2282984716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd63ff0 0x561f6dd66480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.491+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/2282984716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd5ee80 0x561f6dd63980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.491+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/2282984716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd5e8b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.491+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/2282984716 >> 172.21.15.70:0/2282984716 conn(0x561f6dd59a00 msgr2=0x561f6dd59e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.491+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/2282984716 shutdown_connections 2024-04-01T01:39:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.491+0000 7fb0ecdb1640 1 --2- 172.21.15.70:0/1064059857 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb0d40a2720 0x7fb0d40a4be0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0036040 tx=0x7fb0e002c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.491+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/2282984716 wait complete. 2024-04-01T01:39:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.492+0000 7fb0f1bf6740 1 -- start start 2024-04-01T01:39:19.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.492+0000 7fb0f1bf6740 1 Processor -- start 2024-04-01T01:39:19.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.493+0000 7fb0f1bf6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd68960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.493+0000 7fb0f1bf6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 0x561f6dd68ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.493+0000 7fb0f1bf6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 0x561f6dd6fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.493+0000 7fb0f1bf6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561f6dd66fa0 con 0x561f6dd63ff0 2024-04-01T01:39:19.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.493+0000 7fb0e7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 0x561f6dd68ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.493+0000 7fb0f1bf6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561f6dc61470 con 0x561f6dd5e490 2024-04-01T01:39:19.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.493+0000 7fb0f1bf6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561f6dd67120 con 0x561f6dd5ee80 2024-04-01T01:39:19.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.493+0000 7fb0e7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 0x561f6dd68ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42672/0 (socket says 172.21.15.70:42672) 2024-04-01T01:39:19.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.493+0000 7fb0e7fff640 1 -- 172.21.15.70:0/1005355344 learned_addr learned my addr 172.21.15.70:0/1005355344 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:19.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.493+0000 7fb0ecdb1640 1 --2- 172.21.15.70:0/1005355344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd68960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.494+0000 7fb0f1bf6740 10 client.0 _ll_register_callbacks cb 0x561f6dc7ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:19.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.494+0000 7fb0ed5b2640 1 --2- 172.21.15.70:0/1005355344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 0x561f6dd6fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.494+0000 7fb0e7fff640 1 -- 172.21.15.70:0/1005355344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 msgr2=0x561f6dd68960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.494+0000 7fb0e7fff640 1 --2- 172.21.15.70:0/1005355344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd68960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.494+0000 7fb0e7fff640 1 -- 172.21.15.70:0/1005355344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 msgr2=0x561f6dd6fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.494+0000 7fb0e7fff640 1 --2- 172.21.15.70:0/1005355344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 0x561f6dd6fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.494+0000 7fb0e7fff640 1 -- 172.21.15.70:0/1005355344 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f6d9d73f0 con 0x561f6dd5ee80 2024-04-01T01:39:19.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.494+0000 7fb0ed5b2640 1 --2- 172.21.15.70:0/1005355344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 0x561f6dd6fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:19.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.494+0000 7fb0e7fff640 1 --2- 172.21.15.70:0/1005355344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 0x561f6dd68ea0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80058c0 tx=0x7fb0d800df40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d50c3cd3d8ad7c4e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.494+0000 7fb0c7fff640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:39:19.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.495+0000 7fb0c7fff640 1 -- 172.21.15.70:0/1005355344 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8020050 con 0x561f6dd5ee80 2024-04-01T01:39:19.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.495+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/1005355344 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0d001bb10 con 0x561f6dd5ee80 2024-04-01T01:39:19.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.495+0000 7fb0c7fff640 1 -- 172.21.15.70:0/1005355344 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb0d80850e0 con 0x561f6dd5ee80 2024-04-01T01:39:19.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.496+0000 7fb0c7fff640 1 -- 172.21.15.70:0/1005355344 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb0d809a310 con 0x561f6dd5ee80 2024-04-01T01:39:19.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.496+0000 7fb0f1bf6740 10 client.27034 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:19.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.496+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/1005355344 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561f6daa17d0 con 0x561f6dd5ee80 2024-04-01T01:39:19.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.496+0000 7fb0c7fff640 1 -- 172.21.15.70:0/1005355344 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0d001bb10 con 0x561f6dd5ee80 2024-04-01T01:39:19.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.497+0000 7fb0c7fff640 1 -- 172.21.15.70:0/1005355344 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb0d8023060 con 0x561f6dd5ee80 2024-04-01T01:39:19.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.497+0000 7fb0f1bf6740 10 client.27034 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:19.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.497+0000 7fb0f1bf6740 10 client.27034 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:39:19.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.497+0000 7fb0f1bf6740 20 client.27034 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:19.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.497+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/1005355344 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x561f6dd74470 0x561f6dd768d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.497+0000 7fb0f1bf6740 4 client.27034 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:19.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.497+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/1005355344 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f6dcd06f0 con 0x561f6dd74470 2024-04-01T01:39:19.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.498+0000 7fb0ecdb1640 1 --2- 172.21.15.70:0/1005355344 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x561f6dd74470 0x561f6dd768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.498+0000 7fb0ecdb1640 1 --2- 172.21.15.70:0/1005355344 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x561f6dd74470 0x561f6dd768d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1864e537ae7644dd server_cookie=56e476080674a9e in_seq=0 out_seq=0 2024-04-01T01:39:19.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.498+0000 7fb0c7fff640 10 client.27034 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:39:19.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.500+0000 7fb0c7fff640 1 -- 172.21.15.70:0/1005355344 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561f6dcd06f0 con 0x561f6dd74470 2024-04-01T01:39:19.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.500+0000 7fb0c7fff640 10 client.27034 handle_command_reply: tid=0 2024-04-01T01:39:19.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.500+0000 7fb0f1bf6740 1 client.27034 shutdown 2024-04-01T01:39:19.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.500+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/1005355344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 msgr2=0x561f6dd68ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.500+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/1005355344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 0x561f6dd68ea0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80058c0 tx=0x7fb0d800df40 comp rx=0 tx=0).stop 2024-04-01T01:39:19.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.500+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/1005355344 shutdown_connections 2024-04-01T01:39:19.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.500+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/1005355344 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x561f6dd74470 0x561f6dd768d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.500+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/1005355344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 0x561f6dd6fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.500+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/1005355344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 0x561f6dd68ea0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.500+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/1005355344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd68960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.500+0000 7fb0c7fff640 0 client.27034 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:39:19.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.500+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/1005355344 >> 172.21.15.70:0/1005355344 conn(0x561f6dd59a00 msgr2=0x561f6dd58d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.500+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/1005355344 shutdown_connections 2024-04-01T01:39:19.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.500+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/1005355344 wait complete. 2024-04-01T01:39:19.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.500+0000 7fb0f1bf6740 20 client.27034 trim_cache size 0 max 16384 2024-04-01T01:39:19.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.522+0000 7fb0f1bf6740 1 Processor -- start 2024-04-01T01:39:19.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.522+0000 7fb0f1bf6740 1 -- start start 2024-04-01T01:39:19.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.522+0000 7fb0f1bf6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd68960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.522+0000 7fb0f1bf6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 0x561f6dd68ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.522+0000 7fb0f1bf6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 0x561f6dd6fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.522+0000 7fb0e7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 0x561f6dd68ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.522+0000 7fb0ecdb1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd68960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.522+0000 7fb0e7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 0x561f6dd68ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42676/0 (socket says 172.21.15.70:42676) 2024-04-01T01:39:19.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.522+0000 7fb0e7fff640 1 -- 172.21.15.70:0/193244620 learned_addr learned my addr 172.21.15.70:0/193244620 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:19.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.522+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/193244620 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e81c2170 con 0x561f6dd63ff0 2024-04-01T01:39:19.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.522+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/193244620 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e81c3cb0 con 0x561f6dd5e490 2024-04-01T01:39:19.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.522+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/193244620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e81c2840 con 0x561f6dd5ee80 2024-04-01T01:39:19.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.522+0000 7fb0ed5b2640 1 --2- 172.21.15.70:0/193244620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 0x561f6dd6fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.523+0000 7fb0e7fff640 1 -- 172.21.15.70:0/193244620 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 msgr2=0x561f6dd68960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.523+0000 7fb0e7fff640 1 --2- 172.21.15.70:0/193244620 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd68960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.523+0000 7fb0e7fff640 1 -- 172.21.15.70:0/193244620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 msgr2=0x561f6dd6fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.523+0000 7fb0e7fff640 1 --2- 172.21.15.70:0/193244620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 0x561f6dd6fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.523+0000 7fb0e7fff640 1 -- 172.21.15.70:0/193244620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f6dd74250 con 0x561f6dd5ee80 2024-04-01T01:39:19.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.523+0000 7fb0ed5b2640 1 --2- 172.21.15.70:0/193244620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 0x561f6dd6fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:19.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.523+0000 7fb0e7fff640 1 --2- 172.21.15.70:0/193244620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 0x561f6dd68ea0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80971a0 tx=0x7fb0d8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f51c6988e43b6a29 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.523+0000 7fb0de7fc640 1 -- 172.21.15.70:0/193244620 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8020070 con 0x561f6dd5ee80 2024-04-01T01:39:19.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.523+0000 7fb0de7fc640 1 -- 172.21.15.70:0/193244620 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0d809a7d0 con 0x561f6dd5ee80 2024-04-01T01:39:19.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.523+0000 7fb0de7fc640 1 -- 172.21.15.70:0/193244620 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d809a130 con 0x561f6dd5ee80 2024-04-01T01:39:19.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.524+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/193244620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 msgr2=0x561f6dd68ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.524+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/193244620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 0x561f6dd68ea0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb0d80971a0 tx=0x7fb0d8000bc0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.524+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/193244620 shutdown_connections 2024-04-01T01:39:19.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.525+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/193244620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 0x561f6dd6fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.525+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/193244620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 0x561f6dd68ea0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.525+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/193244620 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd68960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.525+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/193244620 >> 172.21.15.70:0/193244620 conn(0x561f6dd59a00 msgr2=0x561f6dd60410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.525+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/193244620 shutdown_connections 2024-04-01T01:39:19.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.525+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/193244620 wait complete. 2024-04-01T01:39:19.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.525+0000 7fb0f1bf6740 1 -- start start 2024-04-01T01:39:19.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.526+0000 7fb0f1bf6740 1 Processor -- start 2024-04-01T01:39:19.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.526+0000 7fb0f1bf6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd6d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.526+0000 7fb0f1bf6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 0x561f6dd6de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.526+0000 7fb0f1bf6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 0x561f6dd6edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.527+0000 7fb0f1bf6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e81c35e0 con 0x561f6dd63ff0 2024-04-01T01:39:19.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.527+0000 7fb0f1bf6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e81c2f10 con 0x561f6dd5e490 2024-04-01T01:39:19.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.527+0000 7fb0f1bf6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e81c0150 con 0x561f6dd5ee80 2024-04-01T01:39:19.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.527+0000 7fb0ed5b2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 0x561f6dd6edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.527+0000 7fb0ed5b2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 0x561f6dd6edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40518/0 (socket says 172.21.15.70:40518) 2024-04-01T01:39:19.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.527+0000 7fb0ed5b2640 1 -- 172.21.15.70:0/3820550049 learned_addr learned my addr 172.21.15.70:0/3820550049 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:19.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.527+0000 7fb0ecdb1640 1 --2- 172.21.15.70:0/3820550049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd6d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.527+0000 7fb0f1bf6740 10 client.0 _ll_register_callbacks cb 0x7fb0d80d17f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:19.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.527+0000 7fb0e7fff640 1 --2- 172.21.15.70:0/3820550049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 0x561f6dd6de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.527+0000 7fb0ed5b2640 1 -- 172.21.15.70:0/3820550049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 msgr2=0x561f6dd6de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.527+0000 7fb0ed5b2640 1 --2- 172.21.15.70:0/3820550049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 0x561f6dd6de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.527+0000 7fb0ed5b2640 1 -- 172.21.15.70:0/3820550049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 msgr2=0x561f6dd6d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.527+0000 7fb0ed5b2640 1 --2- 172.21.15.70:0/3820550049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd6d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.527+0000 7fb0ed5b2640 1 -- 172.21.15.70:0/3820550049 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f6dd6f3c0 con 0x561f6dd63ff0 2024-04-01T01:39:19.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.527+0000 7fb0ecdb1640 1 --2- 172.21.15.70:0/3820550049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd6d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:19.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.527+0000 7fb0ed5b2640 1 --2- 172.21.15.70:0/3820550049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 0x561f6dd6edd0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0009700 tx=0x7fb0d0004a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c690d9a585659bb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.528+0000 7fb0c7fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:39:19.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.528+0000 7fb0c7fff640 1 -- 172.21.15.70:0/3820550049 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d002c060 con 0x561f6dd63ff0 2024-04-01T01:39:19.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.528+0000 7fb0c7fff640 1 -- 172.21.15.70:0/3820550049 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb0d0005240 con 0x561f6dd63ff0 2024-04-01T01:39:19.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.528+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/3820550049 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0d809a7d0 con 0x561f6dd63ff0 2024-04-01T01:39:19.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.528+0000 7fb0c7fff640 1 -- 172.21.15.70:0/3820550049 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb0d0033050 con 0x561f6dd63ff0 2024-04-01T01:39:19.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.528+0000 7fb0f1bf6740 10 client.17268 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:19.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.528+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/3820550049 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561f6dddaa80 con 0x561f6dd63ff0 2024-04-01T01:39:19.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.529+0000 7fb0c7fff640 1 -- 172.21.15.70:0/3820550049 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb0d00058f0 con 0x561f6dd63ff0 2024-04-01T01:39:19.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.529+0000 7fb0c7fff640 1 -- 172.21.15.70:0/3820550049 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb0d0026070 con 0x561f6dd63ff0 2024-04-01T01:39:19.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.529+0000 7fb0f1bf6740 10 client.17268 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:19.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.529+0000 7fb0f1bf6740 10 client.17268 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:39:19.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.529+0000 7fb0f1bf6740 20 client.17268 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:19.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.529+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/3820550049 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x561f6dddb0a0 0x561f6dddd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.529+0000 7fb0f1bf6740 4 client.17268 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:39:19.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.529+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/3820550049 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fb0d809a7d0 con 0x561f6dddb0a0 2024-04-01T01:39:19.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.529+0000 7fb0ecdb1640 1 --2- 172.21.15.70:0/3820550049 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x561f6dddb0a0 0x561f6dddd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.530+0000 7fb0ecdb1640 1 --2- 172.21.15.70:0/3820550049 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x561f6dddb0a0 0x561f6dddd4c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=97445af5b45093d1 server_cookie=5ce12dbe13811a70 in_seq=0 out_seq=0 2024-04-01T01:39:19.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.530+0000 7fb0c7fff640 10 client.17268 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:39:19.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.530+0000 7fb0c7fff640 1 -- 172.21.15.70:0/3820550049 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fb0d809a7d0 con 0x561f6dddb0a0 2024-04-01T01:39:19.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.530+0000 7fb0c7fff640 10 client.17268 handle_command_reply: tid=0 2024-04-01T01:39:19.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.530+0000 7fb0f1bf6740 1 client.17268 shutdown 2024-04-01T01:39:19.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.530+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/3820550049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 msgr2=0x561f6dd6edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.530+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/3820550049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 0x561f6dd6edd0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb0d0009700 tx=0x7fb0d0004a00 comp rx=0 tx=0).stop 2024-04-01T01:39:19.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.531+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/3820550049 shutdown_connections 2024-04-01T01:39:19.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.531+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/3820550049 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x561f6dddb0a0 0x561f6dddd4c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.531+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/3820550049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f6dd63ff0 0x561f6dd6edd0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.531+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/3820550049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f6dd5ee80 0x561f6dd6de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.531+0000 7fb0f1bf6740 1 --2- 172.21.15.70:0/3820550049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f6dd5e490 0x561f6dd6d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.531+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/3820550049 >> 172.21.15.70:0/3820550049 conn(0x561f6dd59a00 msgr2=0x561f6dd68900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.531+0000 7fb0c7fff640 0 client.17268 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:39:19.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.532+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/3820550049 shutdown_connections 2024-04-01T01:39:19.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.532+0000 7fb0f1bf6740 1 -- 172.21.15.70:0/3820550049 wait complete. 2024-04-01T01:39:19.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.532+0000 7fb0f1bf6740 20 client.17268 trim_cache size 0 max 16384 2024-04-01T01:39:19.536 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:39:19.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.533+0000 7fb0ef015640 1 -- 172.21.15.70:0/1064059857 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb0d40a2720 msgr2=0x7fb0d40a4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.533+0000 7fb0ef015640 1 --2- 172.21.15.70:0/1064059857 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb0d40a2720 0x7fb0d40a4be0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb0e0036040 tx=0x7fb0e002c7e0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.533+0000 7fb0ef015640 1 -- 172.21.15.70:0/1064059857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0e810da70 msgr2=0x7fb0e81156d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.533+0000 7fb0ef015640 1 --2- 172.21.15.70:0/1064059857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0e810da70 0x7fb0e81156d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb0d800d950 tx=0x7fb0d800de10 comp rx=0 tx=0).stop 2024-04-01T01:39:19.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.534+0000 7fb0ef015640 1 -- 172.21.15.70:0/1064059857 shutdown_connections 2024-04-01T01:39:19.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.534+0000 7fb0ef015640 1 --2- 172.21.15.70:0/1064059857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb0e8116710 0x7fb0e8116bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.534+0000 7fb0ef015640 1 --2- 172.21.15.70:0/1064059857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0e810da70 0x7fb0e81156d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.534+0000 7fb0ef015640 1 --2- 172.21.15.70:0/1064059857 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb0d40a2720 0x7fb0d40a4be0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.534+0000 7fb0ef015640 1 --2- 172.21.15.70:0/1064059857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0e810b0c0 0x7fb0e8115190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.534+0000 7fb0ef015640 1 -- 172.21.15.70:0/1064059857 >> 172.21.15.70:0/1064059857 conn(0x7fb0e806e5b0 msgr2=0x7fb0e810b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.534+0000 7fb0ef015640 1 -- 172.21.15.70:0/1064059857 shutdown_connections 2024-04-01T01:39:19.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.534+0000 7fb0ef015640 1 -- 172.21.15.70:0/1064059857 wait complete. 2024-04-01T01:39:19.549 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:39:19.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.716+0000 7f85c888f640 1 -- 172.21.15.70:0/1915975742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c4108f60 msgr2=0x7f85c41093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.716+0000 7f85c888f640 1 --2- 172.21.15.70:0/1915975742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c4108f60 0x7f85c41093e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009900 tx=0x7f85b802bc40 comp rx=0 tx=0).stop 2024-04-01T01:39:19.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.716+0000 7f85c888f640 1 -- 172.21.15.70:0/1915975742 shutdown_connections 2024-04-01T01:39:19.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.716+0000 7f85c888f640 1 --2- 172.21.15.70:0/1915975742 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85c4109920 0x7f85c41101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.716+0000 7f85c888f640 1 --2- 172.21.15.70:0/1915975742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c4108f60 0x7f85c41093e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.716+0000 7f85c888f640 1 --2- 172.21.15.70:0/1915975742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85c4107d70 0x7f85c4108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.716+0000 7f85c888f640 1 -- 172.21.15.70:0/1915975742 >> 172.21.15.70:0/1915975742 conn(0x7f85c4076d10 msgr2=0x7f85c4079150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.716+0000 7f85c888f640 1 -- 172.21.15.70:0/1915975742 shutdown_connections 2024-04-01T01:39:19.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.716+0000 7f85c888f640 1 -- 172.21.15.70:0/1915975742 wait complete. 2024-04-01T01:39:19.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.718+0000 7f85c888f640 1 Processor -- start 2024-04-01T01:39:19.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.718+0000 7f85c888f640 1 -- start start 2024-04-01T01:39:19.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.719+0000 7f85c888f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c4107d70 0x7f85c419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.719+0000 7f85c888f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85c4108f60 0x7f85c419e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.719+0000 7f85c888f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85c4109920 0x7f85c41a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.719+0000 7f85c888f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f85c4113950 con 0x7f85c4108f60 2024-04-01T01:39:19.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.719+0000 7f85c1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85c4108f60 0x7f85c419e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.719+0000 7f85c259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c4107d70 0x7f85c419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.719+0000 7f85c1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85c4108f60 0x7f85c419e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40538/0 (socket says 172.21.15.70:40538) 2024-04-01T01:39:19.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.719+0000 7f85c1d9b640 1 -- 172.21.15.70:0/2305863243 learned_addr learned my addr 172.21.15.70:0/2305863243 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:19.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.719+0000 7f85c888f640 1 -- 172.21.15.70:0/2305863243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f85c41137d0 con 0x7f85c4109920 2024-04-01T01:39:19.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.719+0000 7f85c888f640 1 -- 172.21.15.70:0/2305863243 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f85c4113ad0 con 0x7f85c4107d70 2024-04-01T01:39:19.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.719+0000 7f85c2d9d640 1 --2- 172.21.15.70:0/2305863243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85c4109920 0x7f85c41a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.720+0000 7f85c2d9d640 1 -- 172.21.15.70:0/2305863243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c4107d70 msgr2=0x7f85c419de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.720+0000 7f85c2d9d640 1 --2- 172.21.15.70:0/2305863243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c4107d70 0x7f85c419de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.720+0000 7f85c2d9d640 1 -- 172.21.15.70:0/2305863243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85c4108f60 msgr2=0x7f85c419e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.720+0000 7f85c2d9d640 1 --2- 172.21.15.70:0/2305863243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85c4108f60 0x7f85c419e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.720+0000 7f85c2d9d640 1 -- 172.21.15.70:0/2305863243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c41a5a60 con 0x7f85c4109920 2024-04-01T01:39:19.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.720+0000 7f85c259c640 1 --2- 172.21.15.70:0/2305863243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c4107d70 0x7f85c419de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:19.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.720+0000 7f85c2d9d640 1 --2- 172.21.15.70:0/2305863243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85c4109920 0x7f85c41a53f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f85bc00be20 tx=0x7f85bc00bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.720+0000 7f85b37fe640 1 -- 172.21.15.70:0/2305863243 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc019070 con 0x7f85c4109920 2024-04-01T01:39:19.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.720+0000 7f85b37fe640 1 -- 172.21.15.70:0/2305863243 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85bc0148f0 con 0x7f85c4109920 2024-04-01T01:39:19.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.720+0000 7f85b37fe640 1 -- 172.21.15.70:0/2305863243 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc012850 con 0x7f85c4109920 2024-04-01T01:39:19.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.721+0000 7f85c888f640 1 -- 172.21.15.70:0/2305863243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85c41a5d50 con 0x7f85c4109920 2024-04-01T01:39:19.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.721+0000 7f85c888f640 1 -- 172.21.15.70:0/2305863243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85c41a6210 con 0x7f85c4109920 2024-04-01T01:39:19.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.724+0000 7f85b37fe640 1 -- 172.21.15.70:0/2305863243 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f85bc01c070 con 0x7f85c4109920 2024-04-01T01:39:19.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.724+0000 7f85cb478740 1 Processor -- start 2024-04-01T01:39:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.725+0000 7f85cb478740 1 -- start start 2024-04-01T01:39:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.725+0000 7f85cb478740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dc490 0x55b2549dc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.725+0000 7f85cb478740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 0x55b2549e1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.725+0000 7f85cb478740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549e1ff0 0x55b2549e4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.725+0000 7f85cb478740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b2549411b0 con 0x55b2549dce80 2024-04-01T01:39:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.725+0000 7f85cb478740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b2548e9c00 con 0x55b2549e1ff0 2024-04-01T01:39:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.725+0000 7f85cb478740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b25493d250 con 0x55b2549dc490 2024-04-01T01:39:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.725+0000 7f85c1d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dc490 0x55b2549dc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.725+0000 7f85c1d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549e1ff0 0x55b2549e4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.725+0000 7f85c1d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549e1ff0 0x55b2549e4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54472/0 (socket says 172.21.15.70:54472) 2024-04-01T01:39:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.725+0000 7f85c1d9b640 1 -- 172.21.15.70:0/2923259777 learned_addr learned my addr 172.21.15.70:0/2923259777 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.726+0000 7f85c259c640 1 --2- 172.21.15.70:0/2923259777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 0x55b2549e1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.726+0000 7f85b37fe640 1 --2- 172.21.15.70:0/2305863243 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85ac0a2520 0x7f85ac0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.726+0000 7f85c1d9b640 1 -- 172.21.15.70:0/2923259777 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549e1ff0 msgr2=0x55b2549e4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.726+0000 7f85c1d9b640 1 --2- 172.21.15.70:0/2923259777 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549e1ff0 0x55b2549e4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.726+0000 7f85c1d9b640 1 -- 172.21.15.70:0/2923259777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 msgr2=0x55b2549e1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.726+0000 7f85b37fe640 1 -- 172.21.15.70:0/2305863243 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f85bc0ce320 con 0x7f85c4109920 2024-04-01T01:39:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.726+0000 7f85c1d9b640 1 --2- 172.21.15.70:0/2923259777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 0x55b2549e1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.726+0000 7f85c2d9d640 1 --2- 172.21.15.70:0/2305863243 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85ac0a2520 0x7f85ac0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.726+0000 7f85c1d9b640 1 -- 172.21.15.70:0/2923259777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2549093d0 con 0x55b2549dc490 2024-04-01T01:39:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.726+0000 7f85c259c640 1 --2- 172.21.15.70:0/2923259777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 0x55b2549e1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:39:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.726+0000 7f85c1d9b640 1 --2- 172.21.15.70:0/2923259777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dc490 0x55b2549dc8b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f85b802cdd0 tx=0x7f85b802c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=556fd35feb7e97fc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.727+0000 7f85b0ff9640 1 -- 172.21.15.70:0/2923259777 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b80043e0 con 0x55b2549dc490 2024-04-01T01:39:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.727+0000 7f85c2d9d640 1 --2- 172.21.15.70:0/2305863243 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85ac0a2520 0x7f85ac0a49e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f85bc097490 tx=0x7f85bc00e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.727+0000 7f85b0ff9640 1 -- 172.21.15.70:0/2923259777 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85b8048070 con 0x55b2549dc490 2024-04-01T01:39:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.727+0000 7f85b0ff9640 1 -- 172.21.15.70:0/2923259777 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b8004c80 con 0x55b2549dc490 2024-04-01T01:39:19.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.728+0000 7f85cb478740 1 -- 172.21.15.70:0/2923259777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dc490 msgr2=0x55b2549dc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.728+0000 7f85cb478740 1 --2- 172.21.15.70:0/2923259777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dc490 0x55b2549dc8b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f85b802cdd0 tx=0x7f85b802c670 comp rx=0 tx=0).stop 2024-04-01T01:39:19.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.728+0000 7f85cb478740 1 -- 172.21.15.70:0/2923259777 shutdown_connections 2024-04-01T01:39:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.728+0000 7f85cb478740 1 --2- 172.21.15.70:0/2923259777 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549e1ff0 0x55b2549e4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.728+0000 7f85cb478740 1 --2- 172.21.15.70:0/2923259777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 0x55b2549e1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.728+0000 7f85cb478740 1 --2- 172.21.15.70:0/2923259777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dc490 0x55b2549dc8b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.728+0000 7f85cb478740 1 -- 172.21.15.70:0/2923259777 >> 172.21.15.70:0/2923259777 conn(0x55b2549d7a00 msgr2=0x55b2549d7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.728+0000 7f85cb478740 1 -- 172.21.15.70:0/2923259777 shutdown_connections 2024-04-01T01:39:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.728+0000 7f85cb478740 1 -- 172.21.15.70:0/2923259777 wait complete. 2024-04-01T01:39:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.729+0000 7f85cb478740 1 -- start start 2024-04-01T01:39:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.729+0000 7f85cb478740 1 Processor -- start 2024-04-01T01:39:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.729+0000 7f85cb478740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dc490 0x55b2549e69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.729+0000 7f85cb478740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dce80 0x55b2549e6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.729+0000 7f85cb478740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549e1ff0 0x55b2549edf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.729+0000 7f85cb478740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b2549e4f40 con 0x55b2549dc490 2024-04-01T01:39:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.729+0000 7f85cb478740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b2548df470 con 0x55b2549e1ff0 2024-04-01T01:39:19.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.729+0000 7f85cb478740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b2549e50c0 con 0x55b2549dce80 2024-04-01T01:39:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.729+0000 7f85cb478740 10 client.0 _ll_register_callbacks cb 0x55b2548fccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.729+0000 7f85c259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dce80 0x55b2549e6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.729+0000 7f85c259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dce80 0x55b2549e6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42712/0 (socket says 172.21.15.70:42712) 2024-04-01T01:39:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.729+0000 7f85c259c640 1 -- 172.21.15.70:0/174786856 learned_addr learned my addr 172.21.15.70:0/174786856 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.729+0000 7f85c1d9b640 1 --2- 172.21.15.70:0/174786856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549e1ff0 0x55b2549edf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.729+0000 7f85c1d9b640 1 --2- 172.21.15.70:0/174786856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dc490 0x55b2549e69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.730+0000 7f85c259c640 1 -- 172.21.15.70:0/174786856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549e1ff0 msgr2=0x55b2549edf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.730+0000 7f85c259c640 1 --2- 172.21.15.70:0/174786856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549e1ff0 0x55b2549edf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.730+0000 7f85c259c640 1 -- 172.21.15.70:0/174786856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dc490 msgr2=0x55b2549e69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.730+0000 7f85c259c640 1 --2- 172.21.15.70:0/174786856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dc490 0x55b2549e69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.730+0000 7f85c259c640 1 -- 172.21.15.70:0/174786856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2546553f0 con 0x55b2549dce80 2024-04-01T01:39:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.730+0000 7f85c1d9b640 1 --2- 172.21.15.70:0/174786856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dc490 0x55b2549e69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:39:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.730+0000 7f85c259c640 1 --2- 172.21.15.70:0/174786856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dce80 0x55b2549e6f00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f85b4013d30 tx=0x7f85b40117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d66d6388f5914d0a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.730+0000 7f85a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:39:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.730+0000 7f85a9ffb640 1 -- 172.21.15.70:0/174786856 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b4004250 con 0x55b2549dce80 2024-04-01T01:39:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.730+0000 7f85cb478740 1 -- 172.21.15.70:0/174786856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85b8048070 con 0x55b2549dce80 2024-04-01T01:39:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.730+0000 7f85a9ffb640 1 -- 172.21.15.70:0/174786856 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f85b40043f0 con 0x55b2549dce80 2024-04-01T01:39:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.730+0000 7f85a9ffb640 1 -- 172.21.15.70:0/174786856 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f85b4009e10 con 0x55b2549dce80 2024-04-01T01:39:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.730+0000 7f85cb478740 10 client.27049 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.730+0000 7f85cb478740 1 -- 172.21.15.70:0/174786856 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b25471f7d0 con 0x55b2549dce80 2024-04-01T01:39:19.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.731+0000 7f85a9ffb640 1 -- 172.21.15.70:0/174786856 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85b8048070 con 0x55b2549dce80 2024-04-01T01:39:19.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.731+0000 7f85a9ffb640 1 -- 172.21.15.70:0/174786856 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f85b401a070 con 0x55b2549dce80 2024-04-01T01:39:19.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.731+0000 7f85cb478740 10 client.27049 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:19.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.731+0000 7f85cb478740 10 client.27049 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:39:19.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.731+0000 7f85cb478740 20 client.27049 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:19.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.731+0000 7f85cb478740 1 --2- 172.21.15.70:0/174786856 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55b2549f2450 0x55b2549f48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.731+0000 7f85cb478740 4 client.27049 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.731+0000 7f85cb478740 1 -- 172.21.15.70:0/174786856 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b25494e6f0 con 0x55b2549f2450 2024-04-01T01:39:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.731+0000 7f85c1d9b640 1 --2- 172.21.15.70:0/174786856 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55b2549f2450 0x55b2549f48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.732+0000 7f85a9ffb640 10 client.27049 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:39:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.732+0000 7f85c1d9b640 1 --2- 172.21.15.70:0/174786856 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55b2549f2450 0x55b2549f48f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a96c7dd5fdc39cd6 server_cookie=88459c0702f90659 in_seq=0 out_seq=0 2024-04-01T01:39:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.732+0000 7f85a9ffb640 1 -- 172.21.15.70:0/174786856 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b25494e6f0 con 0x55b2549f2450 2024-04-01T01:39:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.732+0000 7f85a9ffb640 10 client.27049 handle_command_reply: tid=0 2024-04-01T01:39:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.733+0000 7f85cb478740 1 client.27049 shutdown 2024-04-01T01:39:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.733+0000 7f85cb478740 1 -- 172.21.15.70:0/174786856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dce80 msgr2=0x55b2549e6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.733+0000 7f85cb478740 1 --2- 172.21.15.70:0/174786856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dce80 0x55b2549e6f00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f85b4013d30 tx=0x7f85b40117b0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.733+0000 7f85cb478740 1 -- 172.21.15.70:0/174786856 shutdown_connections 2024-04-01T01:39:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.733+0000 7f85cb478740 1 --2- 172.21.15.70:0/174786856 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55b2549f2450 0x55b2549f48f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.733+0000 7f85cb478740 1 --2- 172.21.15.70:0/174786856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549e1ff0 0x55b2549edf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.733+0000 7f85cb478740 1 --2- 172.21.15.70:0/174786856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dce80 0x55b2549e6f00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.733+0000 7f85a9ffb640 0 client.27049 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:39:19.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.733+0000 7f85cb478740 1 --2- 172.21.15.70:0/174786856 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dc490 0x55b2549e69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.733+0000 7f85cb478740 1 -- 172.21.15.70:0/174786856 >> 172.21.15.70:0/174786856 conn(0x55b2549d7a00 msgr2=0x55b2549d6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.733+0000 7f85cb478740 1 -- 172.21.15.70:0/174786856 shutdown_connections 2024-04-01T01:39:19.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.733+0000 7f85cb478740 1 -- 172.21.15.70:0/174786856 wait complete. 2024-04-01T01:39:19.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.733+0000 7f85cb478740 20 client.27049 trim_cache size 0 max 16384 2024-04-01T01:39:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.753+0000 7f85cb478740 1 Processor -- start 2024-04-01T01:39:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.753+0000 7f85cb478740 1 -- start start 2024-04-01T01:39:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.753+0000 7f85cb478740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549dc490 0x55b2549e69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.754+0000 7f85cb478740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 0x55b2549e6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.754+0000 7f85cb478740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549e1ff0 0x55b2549edf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.754+0000 7f85cb478740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f85c41b07a0 con 0x55b2549dce80 2024-04-01T01:39:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.754+0000 7f85c1d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549dc490 0x55b2549e69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.754+0000 7f85c1d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549dc490 0x55b2549e69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54488/0 (socket says 172.21.15.70:54488) 2024-04-01T01:39:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.754+0000 7f85c1d9b640 1 -- 172.21.15.70:0/283738929 learned_addr learned my addr 172.21.15.70:0/283738929 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.754+0000 7f85c259c640 1 --2- 172.21.15.70:0/283738929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 0x55b2549e6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.754+0000 7f85cb478740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f85c41b22e0 con 0x55b2549dc490 2024-04-01T01:39:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.754+0000 7f85cb478740 1 -- 172.21.15.70:0/283738929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f85c41b0e70 con 0x55b2549e1ff0 2024-04-01T01:39:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.754+0000 7f85c1d9b640 1 --2- 172.21.15.70:0/283738929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549e1ff0 0x55b2549edf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.755+0000 7f85c259c640 1 -- 172.21.15.70:0/283738929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549e1ff0 msgr2=0x55b2549edf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.755+0000 7f85c259c640 1 --2- 172.21.15.70:0/283738929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549e1ff0 0x55b2549edf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.755+0000 7f85c259c640 1 -- 172.21.15.70:0/283738929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549dc490 msgr2=0x55b2549e69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.755+0000 7f85c259c640 1 --2- 172.21.15.70:0/283738929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549dc490 0x55b2549e69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.755+0000 7f85c259c640 1 -- 172.21.15.70:0/283738929 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2549f0dd0 con 0x55b2549dce80 2024-04-01T01:39:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.755+0000 7f85c1d9b640 1 --2- 172.21.15.70:0/283738929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549dc490 0x55b2549e69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.755+0000 7f85c259c640 1 --2- 172.21.15.70:0/283738929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 0x55b2549e6f00 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f85b4013d00 tx=0x7f85b4004d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78841c55f0b1f6b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.755+0000 7f85abfff640 1 -- 172.21.15.70:0/283738929 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b4003a60 con 0x55b2549dce80 2024-04-01T01:39:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.756+0000 7f85abfff640 1 -- 172.21.15.70:0/283738929 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85b4005490 con 0x55b2549dce80 2024-04-01T01:39:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.756+0000 7f85abfff640 1 -- 172.21.15.70:0/283738929 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b4029990 con 0x55b2549dce80 2024-04-01T01:39:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.757+0000 7f85cb478740 1 -- 172.21.15.70:0/283738929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 msgr2=0x55b2549e6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.757+0000 7f85cb478740 1 --2- 172.21.15.70:0/283738929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 0x55b2549e6f00 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f85b4013d00 tx=0x7f85b4004d20 comp rx=0 tx=0).stop 2024-04-01T01:39:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.757+0000 7f85cb478740 1 -- 172.21.15.70:0/283738929 shutdown_connections 2024-04-01T01:39:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.757+0000 7f85cb478740 1 --2- 172.21.15.70:0/283738929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549e1ff0 0x55b2549edf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.757+0000 7f85cb478740 1 --2- 172.21.15.70:0/283738929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 0x55b2549e6f00 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.757+0000 7f85cb478740 1 --2- 172.21.15.70:0/283738929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549dc490 0x55b2549e69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.757+0000 7f85cb478740 1 -- 172.21.15.70:0/283738929 >> 172.21.15.70:0/283738929 conn(0x55b2549d7a00 msgr2=0x55b2549d7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.757+0000 7f85cb478740 1 -- 172.21.15.70:0/283738929 shutdown_connections 2024-04-01T01:39:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.757+0000 7f85cb478740 1 -- 172.21.15.70:0/283738929 wait complete. 2024-04-01T01:39:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.758+0000 7f85cb478740 1 -- start start 2024-04-01T01:39:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.758+0000 7f85cb478740 1 Processor -- start 2024-04-01T01:39:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.758+0000 7f85cb478740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dc490 0x55b2549eb520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.758+0000 7f85cb478740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 0x55b2549eba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.758+0000 7f85cb478740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549e1ff0 0x55b2549eca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.758+0000 7f85cb478740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f85c41b1c10 con 0x55b2549dce80 2024-04-01T01:39:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.758+0000 7f85cb478740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f85c41b1540 con 0x55b2549e1ff0 2024-04-01T01:39:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.758+0000 7f85cb478740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f85c41ae780 con 0x55b2549dc490 2024-04-01T01:39:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.758+0000 7f85cb478740 10 client.0 _ll_register_callbacks cb 0x7f85b4013940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.758+0000 7f85c259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 0x55b2549eba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.758+0000 7f85c259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 0x55b2549eba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40570/0 (socket says 172.21.15.70:40570) 2024-04-01T01:39:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.758+0000 7f85c259c640 1 -- 172.21.15.70:0/2861922952 learned_addr learned my addr 172.21.15.70:0/2861922952 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.758+0000 7f85c1d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dc490 0x55b2549eb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.758+0000 7f85c1d9b640 1 --2- 172.21.15.70:0/2861922952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549e1ff0 0x55b2549eca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.759+0000 7f85c1d9b640 1 -- 172.21.15.70:0/2861922952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549e1ff0 msgr2=0x55b2549eca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.759+0000 7f85c1d9b640 1 --2- 172.21.15.70:0/2861922952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549e1ff0 0x55b2549eca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.759+0000 7f85c1d9b640 1 -- 172.21.15.70:0/2861922952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 msgr2=0x55b2549eba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.759+0000 7f85c1d9b640 1 --2- 172.21.15.70:0/2861922952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 0x55b2549eba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.759+0000 7f85c1d9b640 1 -- 172.21.15.70:0/2861922952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2549ed060 con 0x55b2549dc490 2024-04-01T01:39:19.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.759+0000 7f85c259c640 1 --2- 172.21.15.70:0/2861922952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 0x55b2549eba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:19.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.759+0000 7f85c1d9b640 1 --2- 172.21.15.70:0/2861922952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dc490 0x55b2549eb520 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f85b802ce20 tx=0x7f85b8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb71283d164fe7bd server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.759+0000 7f85a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:39:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.759+0000 7f85a9ffb640 1 -- 172.21.15.70:0/2861922952 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b8005240 con 0x55b2549dc490 2024-04-01T01:39:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.759+0000 7f85a9ffb640 1 -- 172.21.15.70:0/2861922952 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f85b80053e0 con 0x55b2549dc490 2024-04-01T01:39:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.759+0000 7f85cb478740 1 -- 172.21.15.70:0/2861922952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85b4005490 con 0x55b2549dc490 2024-04-01T01:39:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.760+0000 7f85a9ffb640 1 -- 172.21.15.70:0/2861922952 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f85b80055e0 con 0x55b2549dc490 2024-04-01T01:39:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.760+0000 7f85cb478740 10 client.27055 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.760+0000 7f85cb478740 1 -- 172.21.15.70:0/2861922952 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b254a69800 con 0x55b2549dc490 2024-04-01T01:39:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.760+0000 7f85a9ffb640 1 -- 172.21.15.70:0/2861922952 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f85b4005490 con 0x55b2549dc490 2024-04-01T01:39:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.760+0000 7f85a9ffb640 1 -- 172.21.15.70:0/2861922952 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f85b8003870 con 0x55b2549dc490 2024-04-01T01:39:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.760+0000 7f85cb478740 10 client.27055 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.760+0000 7f85cb478740 10 client.27055 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:39:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.760+0000 7f85cb478740 20 client.27055 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.760+0000 7f85cb478740 1 --2- 172.21.15.70:0/2861922952 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55b254a69ca0 0x55b254a6c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.760+0000 7f85cb478740 4 client.27055 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:39:19.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.760+0000 7f85cb478740 1 -- 172.21.15.70:0/2861922952 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b254948a40 con 0x55b254a69ca0 2024-04-01T01:39:19.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.761+0000 7f85c259c640 1 --2- 172.21.15.70:0/2861922952 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55b254a69ca0 0x55b254a6c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.761+0000 7f85a9ffb640 10 client.27055 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:39:19.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.762+0000 7f85c259c640 1 --2- 172.21.15.70:0/2861922952 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55b254a69ca0 0x55b254a6c090 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=59cfdc8353ee6b3c server_cookie=668127bbb8ac2016 in_seq=0 out_seq=0 2024-04-01T01:39:19.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.762+0000 7f85a9ffb640 1 -- 172.21.15.70:0/2861922952 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b254948a40 con 0x55b254a69ca0 2024-04-01T01:39:19.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.762+0000 7f85a9ffb640 10 client.27055 handle_command_reply: tid=0 2024-04-01T01:39:19.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.762+0000 7f85cb478740 1 client.27055 shutdown 2024-04-01T01:39:19.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.762+0000 7f85cb478740 1 -- 172.21.15.70:0/2861922952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dc490 msgr2=0x55b2549eb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.762+0000 7f85cb478740 1 --2- 172.21.15.70:0/2861922952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dc490 0x55b2549eb520 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f85b802ce20 tx=0x7f85b8000bc0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.762+0000 7f85cb478740 1 -- 172.21.15.70:0/2861922952 shutdown_connections 2024-04-01T01:39:19.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.762+0000 7f85cb478740 1 --2- 172.21.15.70:0/2861922952 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55b254a69ca0 0x55b254a6c090 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.762+0000 7f85cb478740 1 --2- 172.21.15.70:0/2861922952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b2549e1ff0 0x55b2549eca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.762+0000 7f85a9ffb640 0 client.27055 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:39:19.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.762+0000 7f85cb478740 1 --2- 172.21.15.70:0/2861922952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b2549dce80 0x55b2549eba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.762+0000 7f85cb478740 1 --2- 172.21.15.70:0/2861922952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b2549dc490 0x55b2549eb520 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.762+0000 7f85cb478740 1 -- 172.21.15.70:0/2861922952 >> 172.21.15.70:0/2861922952 conn(0x55b2549d7a00 msgr2=0x55b2549e6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.762+0000 7f85cb478740 1 -- 172.21.15.70:0/2861922952 shutdown_connections 2024-04-01T01:39:19.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.762+0000 7f85cb478740 1 -- 172.21.15.70:0/2861922952 wait complete. 2024-04-01T01:39:19.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.762+0000 7f85cb478740 20 client.27055 trim_cache size 0 max 16384 2024-04-01T01:39:19.765 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:39:19.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.763+0000 7f85c888f640 1 -- 172.21.15.70:0/2305863243 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85ac0a2520 msgr2=0x7f85ac0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.763+0000 7f85c888f640 1 --2- 172.21.15.70:0/2305863243 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85ac0a2520 0x7f85ac0a49e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f85bc097490 tx=0x7f85bc00e040 comp rx=0 tx=0).stop 2024-04-01T01:39:19.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.764+0000 7f85c888f640 1 -- 172.21.15.70:0/2305863243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85c4109920 msgr2=0x7f85c41a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.764+0000 7f85c888f640 1 --2- 172.21.15.70:0/2305863243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85c4109920 0x7f85c41a53f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f85bc00be20 tx=0x7f85bc00bf10 comp rx=0 tx=0).stop 2024-04-01T01:39:19.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.764+0000 7f85c888f640 1 -- 172.21.15.70:0/2305863243 shutdown_connections 2024-04-01T01:39:19.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.764+0000 7f85c888f640 1 --2- 172.21.15.70:0/2305863243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85c4109920 0x7f85c41a53f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.764+0000 7f85c888f640 1 --2- 172.21.15.70:0/2305863243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f85c4108f60 0x7f85c419e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.764+0000 7f85c888f640 1 --2- 172.21.15.70:0/2305863243 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85ac0a2520 0x7f85ac0a49e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.764+0000 7f85c888f640 1 --2- 172.21.15.70:0/2305863243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85c4107d70 0x7f85c419de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.764+0000 7f85c888f640 1 -- 172.21.15.70:0/2305863243 >> 172.21.15.70:0/2305863243 conn(0x7f85c4076d10 msgr2=0x7f85c410c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.764+0000 7f85c888f640 1 -- 172.21.15.70:0/2305863243 shutdown_connections 2024-04-01T01:39:19.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.764+0000 7f85c888f640 1 -- 172.21.15.70:0/2305863243 wait complete. 2024-04-01T01:39:19.780 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:39:19.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.943+0000 7fa4268df640 1 -- 172.21.15.70:0/3525082255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa420103970 msgr2=0x7fa420103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.943+0000 7fa4268df640 1 --2- 172.21.15.70:0/3525082255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa420103970 0x7fa420103d90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa410009860 tx=0x7fa41002bb70 comp rx=0 tx=0).stop 2024-04-01T01:39:19.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.943+0000 7fa4268df640 1 -- 172.21.15.70:0/3525082255 shutdown_connections 2024-04-01T01:39:19.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.943+0000 7fa4268df640 1 --2- 172.21.15.70:0/3525082255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa420105520 0x7fa42010bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.943+0000 7fa4268df640 1 --2- 172.21.15.70:0/3525082255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa420104b60 0x7fa420104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.943+0000 7fa4268df640 1 --2- 172.21.15.70:0/3525082255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa420103970 0x7fa420103d90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.943+0000 7fa4268df640 1 -- 172.21.15.70:0/3525082255 >> 172.21.15.70:0/3525082255 conn(0x7fa4200feec0 msgr2=0x7fa420101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.944+0000 7fa4268df640 1 -- 172.21.15.70:0/3525082255 shutdown_connections 2024-04-01T01:39:19.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.944+0000 7fa4268df640 1 -- 172.21.15.70:0/3525082255 wait complete. 2024-04-01T01:39:19.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.944+0000 7fa4268df640 1 Processor -- start 2024-04-01T01:39:19.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.945+0000 7fa4268df640 1 -- start start 2024-04-01T01:39:19.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.945+0000 7fa4268df640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa420103970 0x7fa42019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.945+0000 7fa4268df640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa420104b60 0x7fa42019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.945+0000 7fa4268df640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa420105520 0x7fa4201a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.945+0000 7fa4268df640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa42010f470 con 0x7fa420103970 2024-04-01T01:39:19.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.945+0000 7fa4268df640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa42010f2f0 con 0x7fa420104b60 2024-04-01T01:39:19.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.945+0000 7fa4268df640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa42010f5f0 con 0x7fa420105520 2024-04-01T01:39:19.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.945+0000 7fa424e7c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa420105520 0x7fa4201a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.945+0000 7fa41f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa420104b60 0x7fa42019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.945+0000 7fa41f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa420104b60 0x7fa42019e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54516/0 (socket says 172.21.15.70:54516) 2024-04-01T01:39:19.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.945+0000 7fa41f7fe640 1 -- 172.21.15.70:0/4216764806 learned_addr learned my addr 172.21.15.70:0/4216764806 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:19.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.946+0000 7fa41ffff640 1 --2- 172.21.15.70:0/4216764806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa420103970 0x7fa42019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.946+0000 7fa41f7fe640 1 -- 172.21.15.70:0/4216764806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa420105520 msgr2=0x7fa4201a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.946+0000 7fa41f7fe640 1 --2- 172.21.15.70:0/4216764806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa420105520 0x7fa4201a55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.946+0000 7fa41f7fe640 1 -- 172.21.15.70:0/4216764806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa420103970 msgr2=0x7fa42019e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.946+0000 7fa41f7fe640 1 --2- 172.21.15.70:0/4216764806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa420103970 0x7fa42019e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.946+0000 7fa41f7fe640 1 -- 172.21.15.70:0/4216764806 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4201a5c20 con 0x7fa420104b60 2024-04-01T01:39:19.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.946+0000 7fa41ffff640 1 --2- 172.21.15.70:0/4216764806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa420103970 0x7fa42019e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:19.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.946+0000 7fa41f7fe640 1 --2- 172.21.15.70:0/4216764806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa420104b60 0x7fa42019e540 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa41400ea60 tx=0x7fa41400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.947+0000 7fa41d7fa640 1 -- 172.21.15.70:0/4216764806 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41400cde0 con 0x7fa420104b60 2024-04-01T01:39:19.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.947+0000 7fa41d7fa640 1 -- 172.21.15.70:0/4216764806 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4140035f0 con 0x7fa420104b60 2024-04-01T01:39:19.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.947+0000 7fa41d7fa640 1 -- 172.21.15.70:0/4216764806 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa414010690 con 0x7fa420104b60 2024-04-01T01:39:19.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.947+0000 7fa4268df640 1 -- 172.21.15.70:0/4216764806 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4201a5f10 con 0x7fa420104b60 2024-04-01T01:39:19.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.947+0000 7fa4268df640 1 -- 172.21.15.70:0/4216764806 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4201a6320 con 0x7fa420104b60 2024-04-01T01:39:19.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.950+0000 7fa41d7fa640 1 -- 172.21.15.70:0/4216764806 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fa4140037a0 con 0x7fa420104b60 2024-04-01T01:39:19.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.950+0000 7fa41d7fa640 1 --2- 172.21.15.70:0/4216764806 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4080a2520 0x7fa4080a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.950+0000 7fa41d7fa640 1 -- 172.21.15.70:0/4216764806 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa4140ceda0 con 0x7fa420104b60 2024-04-01T01:39:19.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.951+0000 7fa41ffff640 1 --2- 172.21.15.70:0/4216764806 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4080a2520 0x7fa4080a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.951+0000 7fa42959f740 1 Processor -- start 2024-04-01T01:39:19.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.951+0000 7fa41ffff640 1 --2- 172.21.15.70:0/4216764806 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4080a2520 0x7fa4080a49e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa410009830 tx=0x7fa410005d20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.951+0000 7fa42959f740 1 -- start start 2024-04-01T01:39:19.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.951+0000 7fa42959f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c659477490 0x55c6594778b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.951+0000 7fa42959f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477e80 0x55c65947c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.951+0000 7fa42959f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c65947cff0 0x55c65947f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.952+0000 7fa42959f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c6593dc1b0 con 0x55c659477490 2024-04-01T01:39:19.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.952+0000 7fa42959f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c659384c00 con 0x55c65947cff0 2024-04-01T01:39:19.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.952+0000 7fa42959f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c6593d8250 con 0x55c659477e80 2024-04-01T01:39:19.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.952+0000 7fa41ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c659477490 0x55c6594778b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.952+0000 7fa41ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c659477490 0x55c6594778b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40592/0 (socket says 172.21.15.70:40592) 2024-04-01T01:39:19.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.952+0000 7fa41ffff640 1 -- 172.21.15.70:0/140025239 learned_addr learned my addr 172.21.15.70:0/140025239 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:19.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.952+0000 7fa41f7fe640 1 --2- 172.21.15.70:0/140025239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477e80 0x55c65947c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.952+0000 7fa424e7c640 1 --2- 172.21.15.70:0/140025239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c65947cff0 0x55c65947f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.952+0000 7fa41ffff640 1 -- 172.21.15.70:0/140025239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477e80 msgr2=0x55c65947c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.952+0000 7fa41ffff640 1 --2- 172.21.15.70:0/140025239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477e80 0x55c65947c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.952+0000 7fa41ffff640 1 -- 172.21.15.70:0/140025239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c65947cff0 msgr2=0x55c65947f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.952+0000 7fa41ffff640 1 --2- 172.21.15.70:0/140025239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c65947cff0 0x55c65947f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.952+0000 7fa41ffff640 1 -- 172.21.15.70:0/140025239 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6593a43d0 con 0x55c659477490 2024-04-01T01:39:19.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.953+0000 7fa41ffff640 1 --2- 172.21.15.70:0/140025239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c659477490 0x55c6594778b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa410036000 tx=0x7fa41003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17c72df5cc50ae3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.953+0000 7fa4067fc640 1 -- 172.21.15.70:0/140025239 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41004f070 con 0x55c659477490 2024-04-01T01:39:19.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.953+0000 7fa4067fc640 1 -- 172.21.15.70:0/140025239 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa410046040 con 0x55c659477490 2024-04-01T01:39:19.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.953+0000 7fa4067fc640 1 -- 172.21.15.70:0/140025239 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa410039040 con 0x55c659477490 2024-04-01T01:39:19.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.954+0000 7fa42959f740 1 -- 172.21.15.70:0/140025239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c659477490 msgr2=0x55c6594778b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.954+0000 7fa42959f740 1 --2- 172.21.15.70:0/140025239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c659477490 0x55c6594778b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa410036000 tx=0x7fa41003d000 comp rx=0 tx=0).stop 2024-04-01T01:39:19.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.954+0000 7fa42959f740 1 -- 172.21.15.70:0/140025239 shutdown_connections 2024-04-01T01:39:19.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.954+0000 7fa42959f740 1 --2- 172.21.15.70:0/140025239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c65947cff0 0x55c65947f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.954+0000 7fa42959f740 1 --2- 172.21.15.70:0/140025239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477e80 0x55c65947c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.954+0000 7fa42959f740 1 --2- 172.21.15.70:0/140025239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c659477490 0x55c6594778b0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.954+0000 7fa42959f740 1 -- 172.21.15.70:0/140025239 >> 172.21.15.70:0/140025239 conn(0x55c659472a00 msgr2=0x55c659472e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.954+0000 7fa42959f740 1 -- 172.21.15.70:0/140025239 shutdown_connections 2024-04-01T01:39:19.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.954+0000 7fa42959f740 1 -- 172.21.15.70:0/140025239 wait complete. 2024-04-01T01:39:19.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.954+0000 7fa42959f740 1 -- start start 2024-04-01T01:39:19.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.955+0000 7fa42959f740 1 Processor -- start 2024-04-01T01:39:19.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.955+0000 7fa42959f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477490 0x55c6594819d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.955+0000 7fa42959f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c659477e80 0x55c659481f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.955+0000 7fa42959f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65947cff0 0x55c659488ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.955+0000 7fa42959f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c65947ff90 con 0x55c65947cff0 2024-04-01T01:39:19.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.955+0000 7fa42959f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c65937a470 con 0x55c659477e80 2024-04-01T01:39:19.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.955+0000 7fa42959f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c659480110 con 0x55c659477490 2024-04-01T01:39:19.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.955+0000 7fa42959f740 10 client.0 _ll_register_callbacks cb 0x55c659397cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:19.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.955+0000 7fa41ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477490 0x55c6594819d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.955+0000 7fa424e7c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65947cff0 0x55c659488ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.955+0000 7fa41ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477490 0x55c6594819d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42762/0 (socket says 172.21.15.70:42762) 2024-04-01T01:39:19.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.955+0000 7fa41ffff640 1 -- 172.21.15.70:0/2832146300 learned_addr learned my addr 172.21.15.70:0/2832146300 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:19.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.955+0000 7fa41f7fe640 1 --2- 172.21.15.70:0/2832146300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c659477e80 0x55c659481f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.956+0000 7fa41f7fe640 1 -- 172.21.15.70:0/2832146300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477490 msgr2=0x55c6594819d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.956+0000 7fa41f7fe640 1 --2- 172.21.15.70:0/2832146300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477490 0x55c6594819d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.956+0000 7fa41f7fe640 1 -- 172.21.15.70:0/2832146300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65947cff0 msgr2=0x55c659488ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.956+0000 7fa41f7fe640 1 --2- 172.21.15.70:0/2832146300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65947cff0 0x55c659488ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.956+0000 7fa41f7fe640 1 -- 172.21.15.70:0/2832146300 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6590f03f0 con 0x55c659477e80 2024-04-01T01:39:19.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.956+0000 7fa424e7c640 1 --2- 172.21.15.70:0/2832146300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65947cff0 0x55c659488ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:19.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.956+0000 7fa41f7fe640 1 --2- 172.21.15.70:0/2832146300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c659477e80 0x55c659481f10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa414020000 tx=0x7fa414003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56f9665ac37b111 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.956+0000 7fa3ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:19.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.956+0000 7fa3ff7fe640 1 -- 172.21.15.70:0/2832146300 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41409f370 con 0x55c659477e80 2024-04-01T01:39:19.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.956+0000 7fa3ff7fe640 1 -- 172.21.15.70:0/2832146300 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa4140bb0c0 con 0x55c659477e80 2024-04-01T01:39:19.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.957+0000 7fa42959f740 1 -- 172.21.15.70:0/2832146300 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa410046040 con 0x55c659477e80 2024-04-01T01:39:19.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.957+0000 7fa3ff7fe640 1 -- 172.21.15.70:0/2832146300 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa41409f5e0 con 0x55c659477e80 2024-04-01T01:39:19.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.957+0000 7fa42959f740 10 client.27203 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:19.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.957+0000 7fa42959f740 1 -- 172.21.15.70:0/2832146300 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c6591ba7d0 con 0x55c659477e80 2024-04-01T01:39:19.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.957+0000 7fa3ff7fe640 1 -- 172.21.15.70:0/2832146300 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa410046040 con 0x55c659477e80 2024-04-01T01:39:19.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.957+0000 7fa3ff7fe640 1 -- 172.21.15.70:0/2832146300 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa4140971e0 con 0x55c659477e80 2024-04-01T01:39:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.957+0000 7fa42959f740 10 client.27203 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.957+0000 7fa42959f740 10 client.27203 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:39:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.957+0000 7fa42959f740 20 client.27203 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.957+0000 7fa42959f740 1 --2- 172.21.15.70:0/2832146300 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55c65948d470 0x55c65948f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.957+0000 7fa42959f740 4 client.27203 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.957+0000 7fa42959f740 1 -- 172.21.15.70:0/2832146300 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6593e96f0 con 0x55c65948d470 2024-04-01T01:39:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.958+0000 7fa41ffff640 1 --2- 172.21.15.70:0/2832146300 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55c65948d470 0x55c65948f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.958+0000 7fa3ff7fe640 10 client.27203 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:39:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.958+0000 7fa41ffff640 1 --2- 172.21.15.70:0/2832146300 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55c65948d470 0x55c65948f8a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=83caa3dfca710fd5 server_cookie=3221e83857a2c08e in_seq=0 out_seq=0 2024-04-01T01:39:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.959+0000 7fa3ff7fe640 1 -- 172.21.15.70:0/2832146300 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c6593e96f0 con 0x55c65948d470 2024-04-01T01:39:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.959+0000 7fa3ff7fe640 10 client.27203 handle_command_reply: tid=0 2024-04-01T01:39:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.959+0000 7fa42959f740 1 client.27203 shutdown 2024-04-01T01:39:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.959+0000 7fa42959f740 1 -- 172.21.15.70:0/2832146300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c659477e80 msgr2=0x55c659481f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.959+0000 7fa42959f740 1 --2- 172.21.15.70:0/2832146300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c659477e80 0x55c659481f10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa414020000 tx=0x7fa414003b90 comp rx=0 tx=0).stop 2024-04-01T01:39:19.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.959+0000 7fa42959f740 1 -- 172.21.15.70:0/2832146300 shutdown_connections 2024-04-01T01:39:19.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.959+0000 7fa42959f740 1 --2- 172.21.15.70:0/2832146300 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55c65948d470 0x55c65948f8a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.959+0000 7fa42959f740 1 --2- 172.21.15.70:0/2832146300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65947cff0 0x55c659488ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.959+0000 7fa3ff7fe640 0 client.27203 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:39:19.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.959+0000 7fa42959f740 1 --2- 172.21.15.70:0/2832146300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c659477e80 0x55c659481f10 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.959+0000 7fa42959f740 1 --2- 172.21.15.70:0/2832146300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477490 0x55c6594819d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.959+0000 7fa42959f740 1 -- 172.21.15.70:0/2832146300 >> 172.21.15.70:0/2832146300 conn(0x55c659472a00 msgr2=0x55c659471d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.959+0000 7fa42959f740 1 -- 172.21.15.70:0/2832146300 shutdown_connections 2024-04-01T01:39:19.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.959+0000 7fa42959f740 1 -- 172.21.15.70:0/2832146300 wait complete. 2024-04-01T01:39:19.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.960+0000 7fa42959f740 20 client.27203 trim_cache size 0 max 16384 2024-04-01T01:39:19.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.982+0000 7fa42959f740 1 Processor -- start 2024-04-01T01:39:19.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.982+0000 7fa42959f740 1 -- start start 2024-04-01T01:39:19.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa42959f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c659477490 0x55c6594714b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa42959f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477e80 0x55c659488ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa42959f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c65947cff0 0x55c6594719f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa42959f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa4201b0aa0 con 0x55c659477490 2024-04-01T01:39:19.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa42959f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa4201b25e0 con 0x55c65947cff0 2024-04-01T01:39:19.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa42959f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa4201b1170 con 0x55c659477e80 2024-04-01T01:39:19.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa424e7c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c65947cff0 0x55c6594719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa424e7c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c65947cff0 0x55c6594719f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54536/0 (socket says 172.21.15.70:54536) 2024-04-01T01:39:19.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa41ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c659477490 0x55c6594714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa424e7c640 1 -- 172.21.15.70:0/4206744401 learned_addr learned my addr 172.21.15.70:0/4206744401 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:19.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa41f7fe640 1 --2- 172.21.15.70:0/4206744401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477e80 0x55c659488ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa424e7c640 1 -- 172.21.15.70:0/4206744401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477e80 msgr2=0x55c659488ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa424e7c640 1 --2- 172.21.15.70:0/4206744401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477e80 0x55c659488ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa424e7c640 1 -- 172.21.15.70:0/4206744401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c659477490 msgr2=0x55c6594714b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa424e7c640 1 --2- 172.21.15.70:0/4206744401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c659477490 0x55c6594714b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa424e7c640 1 -- 172.21.15.70:0/4206744401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c659473770 con 0x55c65947cff0 2024-04-01T01:39:19.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.983+0000 7fa424e7c640 1 --2- 172.21.15.70:0/4206744401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c65947cff0 0x55c6594719f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa41800e860 tx=0x7fa4180095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efa4f5412f32efbd server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.984+0000 7fa405ffb640 1 -- 172.21.15.70:0/4206744401 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418003d80 con 0x55c65947cff0 2024-04-01T01:39:19.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.984+0000 7fa405ffb640 1 -- 172.21.15.70:0/4206744401 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa41800ddf0 con 0x55c65947cff0 2024-04-01T01:39:19.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.984+0000 7fa405ffb640 1 -- 172.21.15.70:0/4206744401 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418019470 con 0x55c65947cff0 2024-04-01T01:39:19.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.985+0000 7fa42959f740 1 -- 172.21.15.70:0/4206744401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c65947cff0 msgr2=0x55c6594719f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.985+0000 7fa42959f740 1 --2- 172.21.15.70:0/4206744401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c65947cff0 0x55c6594719f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa41800e860 tx=0x7fa4180095c0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.985+0000 7fa42959f740 1 -- 172.21.15.70:0/4206744401 shutdown_connections 2024-04-01T01:39:19.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.985+0000 7fa42959f740 1 --2- 172.21.15.70:0/4206744401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c65947cff0 0x55c6594719f0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.985+0000 7fa42959f740 1 --2- 172.21.15.70:0/4206744401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477e80 0x55c659488ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.985+0000 7fa42959f740 1 --2- 172.21.15.70:0/4206744401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c659477490 0x55c6594714b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.985+0000 7fa42959f740 1 -- 172.21.15.70:0/4206744401 >> 172.21.15.70:0/4206744401 conn(0x55c659472a00 msgr2=0x55c659479410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.986+0000 7fa42959f740 1 -- 172.21.15.70:0/4206744401 shutdown_connections 2024-04-01T01:39:19.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.986+0000 7fa42959f740 1 -- 172.21.15.70:0/4206744401 wait complete. 2024-04-01T01:39:19.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.986+0000 7fa42959f740 1 -- start start 2024-04-01T01:39:19.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.986+0000 7fa42959f740 1 Processor -- start 2024-04-01T01:39:19.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.987+0000 7fa42959f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477490 0x55c659486c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.987+0000 7fa42959f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c659477e80 0x55c659487180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.987+0000 7fa42959f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65947cff0 0x55c659488140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.987+0000 7fa42959f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa4201b1f10 con 0x55c65947cff0 2024-04-01T01:39:19.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.987+0000 7fa42959f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa4201b1840 con 0x55c659477e80 2024-04-01T01:39:19.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.987+0000 7fa42959f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa4201aea80 con 0x55c659477490 2024-04-01T01:39:19.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.987+0000 7fa42959f740 10 client.0 _ll_register_callbacks cb 0x7fa41409adf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:19.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.987+0000 7fa41ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477490 0x55c659486c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.987+0000 7fa41ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477490 0x55c659486c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42784/0 (socket says 172.21.15.70:42784) 2024-04-01T01:39:19.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.987+0000 7fa424e7c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65947cff0 0x55c659488140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.987+0000 7fa424e7c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65947cff0 0x55c659488140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40626/0 (socket says 172.21.15.70:40626) 2024-04-01T01:39:19.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.987+0000 7fa424e7c640 1 -- 172.21.15.70:0/2621046325 learned_addr learned my addr 172.21.15.70:0/2621046325 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:19.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.987+0000 7fa41f7fe640 1 --2- 172.21.15.70:0/2621046325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c659477e80 0x55c659487180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.988+0000 7fa424e7c640 1 -- 172.21.15.70:0/2621046325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477490 msgr2=0x55c659486c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.988+0000 7fa424e7c640 1 --2- 172.21.15.70:0/2621046325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477490 0x55c659486c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.988+0000 7fa424e7c640 1 -- 172.21.15.70:0/2621046325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c659477e80 msgr2=0x55c659487180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:19.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.988+0000 7fa424e7c640 1 --2- 172.21.15.70:0/2621046325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c659477e80 0x55c659487180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.988+0000 7fa424e7c640 1 -- 172.21.15.70:0/2621046325 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c659488730 con 0x55c65947cff0 2024-04-01T01:39:19.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.988+0000 7fa41f7fe640 1 --2- 172.21.15.70:0/2621046325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c659477e80 0x55c659487180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.988+0000 7fa424e7c640 1 --2- 172.21.15.70:0/2621046325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65947cff0 0x55c659488140 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa418010010 tx=0x7fa418004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bdd9311f04f1770 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.988+0000 7fa3ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:39:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.988+0000 7fa3ff7fe640 1 -- 172.21.15.70:0/2621046325 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41802f060 con 0x55c65947cff0 2024-04-01T01:39:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.988+0000 7fa42959f740 1 -- 172.21.15.70:0/2621046325 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa41800ddf0 con 0x55c65947cff0 2024-04-01T01:39:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.989+0000 7fa3ff7fe640 1 -- 172.21.15.70:0/2621046325 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa41800d940 con 0x55c65947cff0 2024-04-01T01:39:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.989+0000 7fa3ff7fe640 1 -- 172.21.15.70:0/2621046325 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa418036050 con 0x55c65947cff0 2024-04-01T01:39:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.989+0000 7fa42959f740 10 client.17295 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.989+0000 7fa42959f740 1 -- 172.21.15.70:0/2621046325 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c6594f3e40 con 0x55c65947cff0 2024-04-01T01:39:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.989+0000 7fa3ff7fe640 1 -- 172.21.15.70:0/2621046325 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa41800ddf0 con 0x55c65947cff0 2024-04-01T01:39:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.990+0000 7fa3ff7fe640 1 -- 172.21.15.70:0/2621046325 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa41800b040 con 0x55c65947cff0 2024-04-01T01:39:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.990+0000 7fa42959f740 10 client.17295 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.990+0000 7fa42959f740 10 client.17295 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:39:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.990+0000 7fa42959f740 20 client.17295 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.990+0000 7fa42959f740 1 --2- 172.21.15.70:0/2621046325 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55c6594f4710 0x55c6594f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.990+0000 7fa42959f740 4 client.17295 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:39:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.990+0000 7fa42959f740 1 -- 172.21.15.70:0/2621046325 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c6593e3a40 con 0x55c6594f4710 2024-04-01T01:39:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.990+0000 7fa41ffff640 1 --2- 172.21.15.70:0/2621046325 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55c6594f4710 0x55c6594f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.991+0000 7fa3ff7fe640 10 client.17295 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:39:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.991+0000 7fa41ffff640 1 --2- 172.21.15.70:0/2621046325 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55c6594f4710 0x55c6594f6b30 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=caced76278054c3 server_cookie=bdc0590fc72fbe74 in_seq=0 out_seq=0 2024-04-01T01:39:19.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.991+0000 7fa3ff7fe640 1 -- 172.21.15.70:0/2621046325 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c6593e3a40 con 0x55c6594f4710 2024-04-01T01:39:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.991+0000 7fa3ff7fe640 10 client.17295 handle_command_reply: tid=0 2024-04-01T01:39:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.991+0000 7fa42959f740 1 client.17295 shutdown 2024-04-01T01:39:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.991+0000 7fa42959f740 1 -- 172.21.15.70:0/2621046325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65947cff0 msgr2=0x55c659488140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.991+0000 7fa42959f740 1 --2- 172.21.15.70:0/2621046325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65947cff0 0x55c659488140 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa418010010 tx=0x7fa418004960 comp rx=0 tx=0).stop 2024-04-01T01:39:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.991+0000 7fa42959f740 1 -- 172.21.15.70:0/2621046325 shutdown_connections 2024-04-01T01:39:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.991+0000 7fa42959f740 1 --2- 172.21.15.70:0/2621046325 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55c6594f4710 0x55c6594f6b30 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.991+0000 7fa42959f740 1 --2- 172.21.15.70:0/2621046325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c65947cff0 0x55c659488140 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.991+0000 7fa42959f740 1 --2- 172.21.15.70:0/2621046325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c659477e80 0x55c659487180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.991+0000 7fa42959f740 1 --2- 172.21.15.70:0/2621046325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c659477490 0x55c659486c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.991+0000 7fa42959f740 1 -- 172.21.15.70:0/2621046325 >> 172.21.15.70:0/2621046325 conn(0x55c659472a00 msgr2=0x55c6594714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.991+0000 7fa3ff7fe640 0 client.17295 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:39:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.992+0000 7fa42959f740 1 -- 172.21.15.70:0/2621046325 shutdown_connections 2024-04-01T01:39:19.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.992+0000 7fa42959f740 1 -- 172.21.15.70:0/2621046325 wait complete. 2024-04-01T01:39:19.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.992+0000 7fa42959f740 20 client.17295 trim_cache size 0 max 16384 2024-04-01T01:39:19.994 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:39:19.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.992+0000 7fa4268df640 1 -- 172.21.15.70:0/4216764806 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4080a2520 msgr2=0x7fa4080a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.992+0000 7fa4268df640 1 --2- 172.21.15.70:0/4216764806 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4080a2520 0x7fa4080a49e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa410009830 tx=0x7fa410005d20 comp rx=0 tx=0).stop 2024-04-01T01:39:19.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.992+0000 7fa4268df640 1 -- 172.21.15.70:0/4216764806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa420104b60 msgr2=0x7fa42019e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:19.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.992+0000 7fa4268df640 1 --2- 172.21.15.70:0/4216764806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa420104b60 0x7fa42019e540 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa41400ea60 tx=0x7fa41400ef20 comp rx=0 tx=0).stop 2024-04-01T01:39:19.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.993+0000 7fa4268df640 1 -- 172.21.15.70:0/4216764806 shutdown_connections 2024-04-01T01:39:19.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.993+0000 7fa4268df640 1 --2- 172.21.15.70:0/4216764806 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa4080a2520 0x7fa4080a49e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.993+0000 7fa4268df640 1 --2- 172.21.15.70:0/4216764806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa420105520 0x7fa4201a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.993+0000 7fa4268df640 1 --2- 172.21.15.70:0/4216764806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa420104b60 0x7fa42019e540 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.993+0000 7fa4268df640 1 --2- 172.21.15.70:0/4216764806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa420103970 0x7fa42019e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:19.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.993+0000 7fa4268df640 1 -- 172.21.15.70:0/4216764806 >> 172.21.15.70:0/4216764806 conn(0x7fa4200feec0 msgr2=0x7fa420077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:19.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.993+0000 7fa4268df640 1 -- 172.21.15.70:0/4216764806 shutdown_connections 2024-04-01T01:39:19.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:19.993+0000 7fa4268df640 1 -- 172.21.15.70:0/4216764806 wait complete. 2024-04-01T01:39:20.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:19 smithi070 ceph-mon[32179]: pgmap v533: 129 pgs: 129 active+clean; 1.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 21 MiB/s wr, 961 op/s 2024-04-01T01:39:20.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:19 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1699454336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:39:20.009 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:39:20.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:19 smithi081 ceph-mon[35142]: pgmap v533: 129 pgs: 129 active+clean; 1.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 21 MiB/s wr, 961 op/s 2024-04-01T01:39:20.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:19 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1699454336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:39:20.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.178+0000 7f39aff65640 1 -- 172.21.15.70:0/2350916862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39a806a420 msgr2=0x7f39a8110620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.178+0000 7f39aff65640 1 --2- 172.21.15.70:0/2350916862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39a806a420 0x7f39a8110620 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f39a0009ca0 tx=0x7f39a002c120 comp rx=0 tx=0).stop 2024-04-01T01:39:20.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.179+0000 7f39aff65640 1 -- 172.21.15.70:0/2350916862 shutdown_connections 2024-04-01T01:39:20.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.179+0000 7f39aff65640 1 --2- 172.21.15.70:0/2350916862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39a806a420 0x7f39a8110620 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.179+0000 7f39aff65640 1 --2- 172.21.15.70:0/2350916862 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a810b260 0x7f39a810d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.179+0000 7f39aff65640 1 --2- 172.21.15.70:0/2350916862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39a8108930 0x7f39a810ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.179+0000 7f39aff65640 1 -- 172.21.15.70:0/2350916862 >> 172.21.15.70:0/2350916862 conn(0x7f39a806e850 msgr2=0x7f39a8070c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:20.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.179+0000 7f39aff65640 1 -- 172.21.15.70:0/2350916862 shutdown_connections 2024-04-01T01:39:20.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.179+0000 7f39aff65640 1 -- 172.21.15.70:0/2350916862 wait complete. 2024-04-01T01:39:20.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.180+0000 7f39aff65640 1 Processor -- start 2024-04-01T01:39:20.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.180+0000 7f39aff65640 1 -- start start 2024-04-01T01:39:20.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.180+0000 7f39aff65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39a806a420 0x7f39a81181e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.180+0000 7f39aff65640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a8108930 0x7f39a8118720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.181+0000 7f39ad500640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a8108930 0x7f39a8118720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.181+0000 7f39aff65640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39a810b260 0x7f39a81b66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.181+0000 7f39aff65640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39a811bfd0 con 0x7f39a806a420 2024-04-01T01:39:20.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.181+0000 7f39aff65640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f39a811be50 con 0x7f39a8108930 2024-04-01T01:39:20.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.181+0000 7f39aff65640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f39a811c150 con 0x7f39a810b260 2024-04-01T01:39:20.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.181+0000 7f39add01640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39a806a420 0x7f39a81181e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.181+0000 7f39add01640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39a806a420 0x7f39a81181e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40630/0 (socket says 172.21.15.70:40630) 2024-04-01T01:39:20.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.181+0000 7f39add01640 1 -- 172.21.15.70:0/1625263816 learned_addr learned my addr 172.21.15.70:0/1625263816 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:20.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.181+0000 7f39ae502640 1 --2- 172.21.15.70:0/1625263816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39a810b260 0x7f39a81b66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.181+0000 7f39add01640 1 -- 172.21.15.70:0/1625263816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39a810b260 msgr2=0x7f39a81b66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.181+0000 7f39add01640 1 --2- 172.21.15.70:0/1625263816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39a810b260 0x7f39a81b66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.181+0000 7f39add01640 1 -- 172.21.15.70:0/1625263816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a8108930 msgr2=0x7f39a8118720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.181+0000 7f39add01640 1 --2- 172.21.15.70:0/1625263816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a8108930 0x7f39a8118720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.181+0000 7f39add01640 1 -- 172.21.15.70:0/1625263816 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a811b850 con 0x7f39a806a420 2024-04-01T01:39:20.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.182+0000 7f39add01640 1 --2- 172.21.15.70:0/1625263816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39a806a420 0x7f39a81181e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f39a400e9f0 tx=0x7f39a400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:20.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.182+0000 7f399effd640 1 -- 172.21.15.70:0/1625263816 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a400ccc0 con 0x7f39a806a420 2024-04-01T01:39:20.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.182+0000 7f399effd640 1 -- 172.21.15.70:0/1625263816 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39a40108e0 con 0x7f39a806a420 2024-04-01T01:39:20.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.182+0000 7f399effd640 1 -- 172.21.15.70:0/1625263816 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a40187c0 con 0x7f39a806a420 2024-04-01T01:39:20.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.182+0000 7f39aff65640 1 -- 172.21.15.70:0/1625263816 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39a81b6be0 con 0x7f39a806a420 2024-04-01T01:39:20.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.182+0000 7f39aff65640 1 -- 172.21.15.70:0/1625263816 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39a81b6f90 con 0x7f39a806a420 2024-04-01T01:39:20.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.186+0000 7f399effd640 1 -- 172.21.15.70:0/1625263816 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f39a4010a90 con 0x7f39a806a420 2024-04-01T01:39:20.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.186+0000 7f39b2c28740 1 Processor -- start 2024-04-01T01:39:20.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.187+0000 7f399effd640 1 --2- 172.21.15.70:0/1625263816 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39940a2590 0x7f39940a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.187+0000 7f399effd640 1 -- 172.21.15.70:0/1625263816 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f39a40ceec0 con 0x7f39a806a420 2024-04-01T01:39:20.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.188+0000 7f39b2c28740 1 -- start start 2024-04-01T01:39:20.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.188+0000 7f39b2c28740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b503696a0 0x560b50369ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.188+0000 7f39b2c28740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b5036a090 0x560b5036eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.188+0000 7f39b2c28740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b5036f200 0x560b50371690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.188+0000 7f39b2c28740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560b502ce1b0 con 0x560b5036a090 2024-04-01T01:39:20.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.188+0000 7f39b2c28740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560b50276c00 con 0x560b5036f200 2024-04-01T01:39:20.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.188+0000 7f39b2c28740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560b502ca250 con 0x560b503696a0 2024-04-01T01:39:20.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.188+0000 7f39ae502640 1 --2- 172.21.15.70:0/1625263816 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39940a2590 0x7f39940a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.189+0000 7f39add01640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b503696a0 0x560b50369ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.189+0000 7f39add01640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b503696a0 0x560b50369ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42812/0 (socket says 172.21.15.70:42812) 2024-04-01T01:39:20.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.189+0000 7f39add01640 1 -- 172.21.15.70:0/279601657 learned_addr learned my addr 172.21.15.70:0/279601657 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:20.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.189+0000 7f39ad500640 1 --2- 172.21.15.70:0/279601657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b5036a090 0x560b5036eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.189+0000 7f39add01640 1 -- 172.21.15.70:0/279601657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b5036f200 msgr2=0x560b50371690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:39:20.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.189+0000 7f39add01640 1 --2- 172.21.15.70:0/279601657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b5036f200 0x560b50371690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.189+0000 7f39add01640 1 -- 172.21.15.70:0/279601657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b5036a090 msgr2=0x560b5036eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:20.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.189+0000 7f39add01640 1 --2- 172.21.15.70:0/279601657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b5036a090 0x560b5036eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.189+0000 7f39add01640 1 -- 172.21.15.70:0/279601657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b502963d0 con 0x560b503696a0 2024-04-01T01:39:20.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.190+0000 7f39add01640 1 --2- 172.21.15.70:0/279601657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b503696a0 0x560b50369ac0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f39a4097f20 tx=0x7f39a4009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84684e2fa62b3b97 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:20.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.190+0000 7f3993fff640 1 -- 172.21.15.70:0/279601657 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a40a3f60 con 0x560b503696a0 2024-04-01T01:39:20.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.190+0000 7f3993fff640 1 -- 172.21.15.70:0/279601657 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39a40cc6a0 con 0x560b503696a0 2024-04-01T01:39:20.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.190+0000 7f3993fff640 1 -- 172.21.15.70:0/279601657 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a409f3d0 con 0x560b503696a0 2024-04-01T01:39:20.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.191+0000 7f39b2c28740 1 -- 172.21.15.70:0/279601657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b503696a0 msgr2=0x560b50369ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.191+0000 7f39b2c28740 1 --2- 172.21.15.70:0/279601657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b503696a0 0x560b50369ac0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f39a4097f20 tx=0x7f39a4009700 comp rx=0 tx=0).stop 2024-04-01T01:39:20.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.192+0000 7f39ae502640 1 --2- 172.21.15.70:0/1625263816 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39940a2590 0x7f39940a4a50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f39a0004af0 tx=0x7f39a002e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:20.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.192+0000 7f39b2c28740 1 -- 172.21.15.70:0/279601657 shutdown_connections 2024-04-01T01:39:20.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.192+0000 7f39b2c28740 1 --2- 172.21.15.70:0/279601657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b5036f200 0x560b50371690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.192+0000 7f39b2c28740 1 --2- 172.21.15.70:0/279601657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b5036a090 0x560b5036eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.192+0000 7f39b2c28740 1 --2- 172.21.15.70:0/279601657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b503696a0 0x560b50369ac0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.192+0000 7f39b2c28740 1 -- 172.21.15.70:0/279601657 >> 172.21.15.70:0/279601657 conn(0x560b50364c10 msgr2=0x560b50365010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:20.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.192+0000 7f39b2c28740 1 -- 172.21.15.70:0/279601657 shutdown_connections 2024-04-01T01:39:20.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.192+0000 7f39b2c28740 1 -- 172.21.15.70:0/279601657 wait complete. 2024-04-01T01:39:20.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.193+0000 7f39b2c28740 1 -- start start 2024-04-01T01:39:20.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.193+0000 7f39b2c28740 1 Processor -- start 2024-04-01T01:39:20.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.194+0000 7f39b2c28740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b503696a0 0x560b503739a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.194+0000 7f39b2c28740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b5036a090 0x560b50373ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.194+0000 7f39b2c28740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b5036f200 0x560b5037af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.194+0000 7f39add01640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b503696a0 0x560b503739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.194+0000 7f39ad500640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b5036a090 0x560b50373ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.194+0000 7f39ad500640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b5036a090 0x560b50373ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54582/0 (socket says 172.21.15.70:54582) 2024-04-01T01:39:20.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.194+0000 7f39ad500640 1 -- 172.21.15.70:0/790585049 learned_addr learned my addr 172.21.15.70:0/790585049 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:20.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.194+0000 7f39b2c28740 1 -- 172.21.15.70:0/790585049 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560b503720f0 con 0x560b5036f200 2024-04-01T01:39:20.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.194+0000 7f39b2c28740 1 -- 172.21.15.70:0/790585049 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560b5026c470 con 0x560b5036a090 2024-04-01T01:39:20.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.194+0000 7f39b2c28740 1 -- 172.21.15.70:0/790585049 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560b50372270 con 0x560b503696a0 2024-04-01T01:39:20.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.194+0000 7f39b2c28740 10 client.0 _ll_register_callbacks cb 0x560b50289cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:20.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.195+0000 7f39ae502640 1 --2- 172.21.15.70:0/790585049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b5036f200 0x560b5037af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.195+0000 7f39ad500640 1 -- 172.21.15.70:0/790585049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b503696a0 msgr2=0x560b503739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:20.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.195+0000 7f39ad500640 1 --2- 172.21.15.70:0/790585049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b503696a0 0x560b503739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.195+0000 7f39ad500640 1 -- 172.21.15.70:0/790585049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b5036f200 msgr2=0x560b5037af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:20.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.195+0000 7f39ad500640 1 --2- 172.21.15.70:0/790585049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b5036f200 0x560b5037af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.195+0000 7f39ad500640 1 -- 172.21.15.70:0/790585049 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b4ffe23f0 con 0x560b5036a090 2024-04-01T01:39:20.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.196+0000 7f39ad500640 1 --2- 172.21.15.70:0/790585049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b5036a090 0x560b50373ee0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3998010810 tx=0x7f3998010cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a56e354d372f4b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:20.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.196+0000 7f39917fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:20.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.196+0000 7f39917fa640 1 -- 172.21.15.70:0/790585049 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399801e060 con 0x560b5036a090 2024-04-01T01:39:20.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.197+0000 7f39917fa640 1 -- 172.21.15.70:0/790585049 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f399800fb90 con 0x560b5036a090 2024-04-01T01:39:20.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.197+0000 7f39b2c28740 1 -- 172.21.15.70:0/790585049 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39a40cc6a0 con 0x560b5036a090 2024-04-01T01:39:20.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.197+0000 7f39917fa640 1 -- 172.21.15.70:0/790585049 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3998025050 con 0x560b5036a090 2024-04-01T01:39:20.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.197+0000 7f39b2c28740 10 client.27215 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:20.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.197+0000 7f39b2c28740 1 -- 172.21.15.70:0/790585049 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560b500ac7d0 con 0x560b5036a090 2024-04-01T01:39:20.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.198+0000 7f39917fa640 1 -- 172.21.15.70:0/790585049 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f399800fd90 con 0x560b5036a090 2024-04-01T01:39:20.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.198+0000 7f39917fa640 1 -- 172.21.15.70:0/790585049 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f399800e550 con 0x560b5036a090 2024-04-01T01:39:20.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.198+0000 7f39b2c28740 10 client.27215 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:20.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.198+0000 7f39b2c28740 10 client.27215 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:39:20.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.198+0000 7f39b2c28740 20 client.27215 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:20.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.198+0000 7f39b2c28740 1 --2- 172.21.15.70:0/790585049 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560b5037f4c0 0x560b503818e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.198+0000 7f39b2c28740 4 client.27215 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:20.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.198+0000 7f39b2c28740 1 -- 172.21.15.70:0/790585049 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f39a40cc6a0 con 0x560b5037f4c0 2024-04-01T01:39:20.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.199+0000 7f39add01640 1 --2- 172.21.15.70:0/790585049 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560b5037f4c0 0x560b503818e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.199+0000 7f39add01640 1 --2- 172.21.15.70:0/790585049 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560b5037f4c0 0x560b503818e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4a446e24fc261879 server_cookie=86e9ad8f935eefe9 in_seq=0 out_seq=0 2024-04-01T01:39:20.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.199+0000 7f39917fa640 10 client.27215 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:39:20.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.200+0000 7f39917fa640 1 -- 172.21.15.70:0/790585049 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f39a40cc6a0 con 0x560b5037f4c0 2024-04-01T01:39:20.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.200+0000 7f39917fa640 10 client.27215 handle_command_reply: tid=0 2024-04-01T01:39:20.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.200+0000 7f39b2c28740 1 client.27215 shutdown 2024-04-01T01:39:20.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.200+0000 7f39b2c28740 1 -- 172.21.15.70:0/790585049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b5036a090 msgr2=0x560b50373ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.200+0000 7f39b2c28740 1 --2- 172.21.15.70:0/790585049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b5036a090 0x560b50373ee0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3998010810 tx=0x7f3998010cd0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.201+0000 7f39b2c28740 1 -- 172.21.15.70:0/790585049 shutdown_connections 2024-04-01T01:39:20.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.201+0000 7f39b2c28740 1 --2- 172.21.15.70:0/790585049 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560b5037f4c0 0x560b503818e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.202+0000 7f39b2c28740 1 --2- 172.21.15.70:0/790585049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b5036f200 0x560b5037af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.202+0000 7f39b2c28740 1 --2- 172.21.15.70:0/790585049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b5036a090 0x560b50373ee0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.202+0000 7f39b2c28740 1 --2- 172.21.15.70:0/790585049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b503696a0 0x560b503739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.202+0000 7f39b2c28740 1 -- 172.21.15.70:0/790585049 >> 172.21.15.70:0/790585049 conn(0x560b50364c10 msgr2=0x560b50363f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:20.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.202+0000 7f39917fa640 0 client.27215 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:39:20.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.202+0000 7f39b2c28740 1 -- 172.21.15.70:0/790585049 shutdown_connections 2024-04-01T01:39:20.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.202+0000 7f39b2c28740 1 -- 172.21.15.70:0/790585049 wait complete. 2024-04-01T01:39:20.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.202+0000 7f39b2c28740 20 client.27215 trim_cache size 0 max 16384 2024-04-01T01:39:20.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.232+0000 7f39b2c28740 1 Processor -- start 2024-04-01T01:39:20.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.232+0000 7f39b2c28740 1 -- start start 2024-04-01T01:39:20.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.233+0000 7f39b2c28740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b503696a0 0x560b50367e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.233+0000 7f39b2c28740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b5036a090 0x560b5037aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.233+0000 7f39b2c28740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b5036f200 0x560b503683b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.233+0000 7f39add01640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b503696a0 0x560b50367e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.233+0000 7f39ad500640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b5036a090 0x560b5037aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.233+0000 7f39add01640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b503696a0 0x560b50367e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40670/0 (socket says 172.21.15.70:40670) 2024-04-01T01:39:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.233+0000 7f39ad500640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b5036a090 0x560b5037aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42822/0 (socket says 172.21.15.70:42822) 2024-04-01T01:39:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.233+0000 7f39add01640 1 -- 172.21.15.70:0/201319348 learned_addr learned my addr 172.21.15.70:0/201319348 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:20.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.233+0000 7f39b2c28740 1 -- 172.21.15.70:0/201319348 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39a81c16a0 con 0x560b503696a0 2024-04-01T01:39:20.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.233+0000 7f39b2c28740 1 -- 172.21.15.70:0/201319348 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f39a81c31e0 con 0x560b5036f200 2024-04-01T01:39:20.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.233+0000 7f39b2c28740 1 -- 172.21.15.70:0/201319348 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f39a81c1d70 con 0x560b5036a090 2024-04-01T01:39:20.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.233+0000 7f39ae502640 1 --2- 172.21.15.70:0/201319348 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b5036f200 0x560b503683b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.234+0000 7f39ad500640 1 -- 172.21.15.70:0/201319348 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b5036f200 msgr2=0x560b503683b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.234+0000 7f39ad500640 1 --2- 172.21.15.70:0/201319348 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b5036f200 0x560b503683b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.234+0000 7f39ad500640 1 -- 172.21.15.70:0/201319348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b503696a0 msgr2=0x560b50367e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.234+0000 7f39ad500640 1 --2- 172.21.15.70:0/201319348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b503696a0 0x560b50367e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.234+0000 7f39ad500640 1 -- 172.21.15.70:0/201319348 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b5036a470 con 0x560b5036a090 2024-04-01T01:39:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.234+0000 7f39ad500640 1 --2- 172.21.15.70:0/201319348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b5036a090 0x560b5037aee0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f399800e970 tx=0x7f399800ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97bc5a0dbe89993c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.235+0000 7f39937fe640 1 -- 172.21.15.70:0/201319348 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399802e070 con 0x560b5036a090 2024-04-01T01:39:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.235+0000 7f39937fe640 1 -- 172.21.15.70:0/201319348 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f399800e3b0 con 0x560b5036a090 2024-04-01T01:39:20.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.235+0000 7f39937fe640 1 -- 172.21.15.70:0/201319348 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3998004660 con 0x560b5036a090 2024-04-01T01:39:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.236+0000 7f39b2c28740 1 -- 172.21.15.70:0/201319348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b5036a090 msgr2=0x560b5037aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.236+0000 7f39b2c28740 1 --2- 172.21.15.70:0/201319348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b5036a090 0x560b5037aee0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f399800e970 tx=0x7f399800ed70 comp rx=0 tx=0).stop 2024-04-01T01:39:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.237+0000 7f39b2c28740 1 -- 172.21.15.70:0/201319348 shutdown_connections 2024-04-01T01:39:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.237+0000 7f39b2c28740 1 --2- 172.21.15.70:0/201319348 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b5036f200 0x560b503683b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.237+0000 7f39b2c28740 1 --2- 172.21.15.70:0/201319348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b5036a090 0x560b5037aee0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.237+0000 7f39b2c28740 1 --2- 172.21.15.70:0/201319348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b503696a0 0x560b50367e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.237+0000 7f39b2c28740 1 -- 172.21.15.70:0/201319348 >> 172.21.15.70:0/201319348 conn(0x560b50364c10 msgr2=0x560b5036b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:20.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.237+0000 7f39b2c28740 1 -- 172.21.15.70:0/201319348 shutdown_connections 2024-04-01T01:39:20.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.237+0000 7f39b2c28740 1 -- 172.21.15.70:0/201319348 wait complete. 2024-04-01T01:39:20.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.237+0000 7f39b2c28740 1 -- start start 2024-04-01T01:39:20.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.238+0000 7f39b2c28740 1 Processor -- start 2024-04-01T01:39:20.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.238+0000 7f39b2c28740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b503696a0 0x560b50378b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.239+0000 7f39b2c28740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b5036a090 0x560b503790a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.239+0000 7f39add01640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b503696a0 0x560b50378b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.239+0000 7f39add01640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b503696a0 0x560b50378b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54606/0 (socket says 172.21.15.70:54606) 2024-04-01T01:39:20.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.239+0000 7f39add01640 1 -- 172.21.15.70:0/2119061345 learned_addr learned my addr 172.21.15.70:0/2119061345 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:20.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.239+0000 7f39b2c28740 1 --2- 172.21.15.70:0/2119061345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b5036f200 0x560b5037a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.239+0000 7f39ad500640 1 --2- 172.21.15.70:0/2119061345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b5036a090 0x560b503790a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.239+0000 7f39b2c28740 1 -- 172.21.15.70:0/2119061345 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39a81c2b10 con 0x560b5036f200 2024-04-01T01:39:20.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.239+0000 7f39b2c28740 1 -- 172.21.15.70:0/2119061345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f39a81c2440 con 0x560b503696a0 2024-04-01T01:39:20.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.239+0000 7f39b2c28740 1 -- 172.21.15.70:0/2119061345 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f39a81bf680 con 0x560b5036a090 2024-04-01T01:39:20.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.239+0000 7f39b2c28740 10 client.0 _ll_register_callbacks cb 0x7f399800acc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:20.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.240+0000 7f39ae502640 1 --2- 172.21.15.70:0/2119061345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b5036f200 0x560b5037a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.240+0000 7f39add01640 1 -- 172.21.15.70:0/2119061345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b5036a090 msgr2=0x560b503790a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:20.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.240+0000 7f39add01640 1 --2- 172.21.15.70:0/2119061345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b5036a090 0x560b503790a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.240+0000 7f39add01640 1 -- 172.21.15.70:0/2119061345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b5036f200 msgr2=0x560b5037a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:20.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.240+0000 7f39add01640 1 --2- 172.21.15.70:0/2119061345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b5036f200 0x560b5037a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.240+0000 7f39add01640 1 -- 172.21.15.70:0/2119061345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b5037a630 con 0x560b503696a0 2024-04-01T01:39:20.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.240+0000 7f39ad500640 1 --2- 172.21.15.70:0/2119061345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b5036a090 0x560b503790a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:20.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.240+0000 7f39add01640 1 --2- 172.21.15.70:0/2119061345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b503696a0 0x560b50378b60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f39a40979b0 tx=0x7f39a409b6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9cac0c295429f33 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:20.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.240+0000 7f39917fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:20.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.240+0000 7f39917fa640 1 -- 172.21.15.70:0/2119061345 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a40b4060 con 0x560b503696a0 2024-04-01T01:39:20.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.241+0000 7f39917fa640 1 -- 172.21.15.70:0/2119061345 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f39a40b4200 con 0x560b503696a0 2024-04-01T01:39:20.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.241+0000 7f39b2c28740 1 -- 172.21.15.70:0/2119061345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f399800e3b0 con 0x560b503696a0 2024-04-01T01:39:20.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.241+0000 7f39917fa640 1 -- 172.21.15.70:0/2119061345 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f39a40977f0 con 0x560b503696a0 2024-04-01T01:39:20.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.241+0000 7f39b2c28740 10 client.27221 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:20.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.242+0000 7f39b2c28740 1 -- 172.21.15.70:0/2119061345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560b503e5d10 con 0x560b503696a0 2024-04-01T01:39:20.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.243+0000 7f39917fa640 1 -- 172.21.15.70:0/2119061345 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39a409a9a0 con 0x560b503696a0 2024-04-01T01:39:20.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.243+0000 7f39917fa640 1 -- 172.21.15.70:0/2119061345 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f39a40bb210 con 0x560b503696a0 2024-04-01T01:39:20.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.244+0000 7f39b2c28740 10 client.27221 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:20.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.244+0000 7f39b2c28740 10 client.27221 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:39:20.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.244+0000 7f39b2c28740 20 client.27221 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:20.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.244+0000 7f39b2c28740 1 --2- 172.21.15.70:0/2119061345 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560b503e6190 0x560b503e8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.244+0000 7f39b2c28740 4 client.27221 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:39:20.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.244+0000 7f39b2c28740 1 -- 172.21.15.70:0/2119061345 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f399800e3b0 con 0x560b503e6190 2024-04-01T01:39:20.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.245+0000 7f39ad500640 1 --2- 172.21.15.70:0/2119061345 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560b503e6190 0x560b503e8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.246+0000 7f39ad500640 1 --2- 172.21.15.70:0/2119061345 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560b503e6190 0x560b503e8580 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7898e76a1a9fe696 server_cookie=76b8c00b5baf27d9 in_seq=0 out_seq=0 2024-04-01T01:39:20.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.246+0000 7f39917fa640 10 client.27221 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:39:20.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.246+0000 7f39917fa640 1 -- 172.21.15.70:0/2119061345 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f399800e3b0 con 0x560b503e6190 2024-04-01T01:39:20.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.246+0000 7f39917fa640 10 client.27221 handle_command_reply: tid=0 2024-04-01T01:39:20.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.247+0000 7f39b2c28740 1 client.27221 shutdown 2024-04-01T01:39:20.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.247+0000 7f39b2c28740 1 -- 172.21.15.70:0/2119061345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b503696a0 msgr2=0x560b50378b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.248+0000 7f39b2c28740 1 --2- 172.21.15.70:0/2119061345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b503696a0 0x560b50378b60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f39a40979b0 tx=0x7f39a409b6d0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.248+0000 7f39b2c28740 1 -- 172.21.15.70:0/2119061345 shutdown_connections 2024-04-01T01:39:20.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.248+0000 7f39b2c28740 1 --2- 172.21.15.70:0/2119061345 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560b503e6190 0x560b503e8580 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.248+0000 7f39b2c28740 1 --2- 172.21.15.70:0/2119061345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560b5036f200 0x560b5037a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.248+0000 7f39b2c28740 1 --2- 172.21.15.70:0/2119061345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b5036a090 0x560b503790a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.248+0000 7f39b2c28740 1 --2- 172.21.15.70:0/2119061345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560b503696a0 0x560b50378b60 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.248+0000 7f39b2c28740 1 -- 172.21.15.70:0/2119061345 >> 172.21.15.70:0/2119061345 conn(0x560b50364c10 msgr2=0x560b50373630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:20.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.248+0000 7f39917fa640 0 client.27221 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:39:20.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.248+0000 7f39b2c28740 1 -- 172.21.15.70:0/2119061345 shutdown_connections 2024-04-01T01:39:20.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.248+0000 7f39b2c28740 1 -- 172.21.15.70:0/2119061345 wait complete. 2024-04-01T01:39:20.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.248+0000 7f39b2c28740 20 client.27221 trim_cache size 0 max 16384 2024-04-01T01:39:20.248 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:39:20.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.249+0000 7f39aff65640 1 -- 172.21.15.70:0/1625263816 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39940a2590 msgr2=0x7f39940a4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.249+0000 7f39aff65640 1 --2- 172.21.15.70:0/1625263816 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39940a2590 0x7f39940a4a50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f39a0004af0 tx=0x7f39a002e040 comp rx=0 tx=0).stop 2024-04-01T01:39:20.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.249+0000 7f39aff65640 1 -- 172.21.15.70:0/1625263816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39a806a420 msgr2=0x7f39a81181e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.249+0000 7f39aff65640 1 --2- 172.21.15.70:0/1625263816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39a806a420 0x7f39a81181e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f39a400e9f0 tx=0x7f39a400edf0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.249+0000 7f39aff65640 1 -- 172.21.15.70:0/1625263816 shutdown_connections 2024-04-01T01:39:20.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.249+0000 7f39aff65640 1 --2- 172.21.15.70:0/1625263816 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39940a2590 0x7f39940a4a50 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.249+0000 7f39aff65640 1 --2- 172.21.15.70:0/1625263816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39a810b260 0x7f39a81b66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.249+0000 7f39aff65640 1 --2- 172.21.15.70:0/1625263816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a8108930 0x7f39a8118720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.249+0000 7f39aff65640 1 --2- 172.21.15.70:0/1625263816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39a806a420 0x7f39a81181e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.249+0000 7f39aff65640 1 -- 172.21.15.70:0/1625263816 >> 172.21.15.70:0/1625263816 conn(0x7f39a806e850 msgr2=0x7f39a810bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:20.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.249+0000 7f39aff65640 1 -- 172.21.15.70:0/1625263816 shutdown_connections 2024-04-01T01:39:20.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.249+0000 7f39aff65640 1 -- 172.21.15.70:0/1625263816 wait complete. 2024-04-01T01:39:20.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:19 smithi189 ceph-mon[35753]: pgmap v533: 129 pgs: 129 active+clean; 1.1 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 21 MiB/s wr, 961 op/s 2024-04-01T01:39:20.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:19 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1699454336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:39:20.263 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:39:20.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.444+0000 7f05b895e640 1 -- 172.21.15.70:0/1353378947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05b40ff610 msgr2=0x7f05b40ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.444+0000 7f05b895e640 1 --2- 172.21.15.70:0/1353378947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05b40ff610 0x7f05b40ffa30 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f05a4009900 tx=0x7f05a402bc40 comp rx=0 tx=0).stop 2024-04-01T01:39:20.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.444+0000 7f05b895e640 1 -- 172.21.15.70:0/1353378947 shutdown_connections 2024-04-01T01:39:20.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.444+0000 7f05b895e640 1 --2- 172.21.15.70:0/1353378947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05b41010e0 0x7f05b41059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.444+0000 7f05b895e640 1 --2- 172.21.15.70:0/1353378947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b4100740 0x7f05b4100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.444+0000 7f05b895e640 1 --2- 172.21.15.70:0/1353378947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05b40ff610 0x7f05b40ffa30 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.444+0000 7f05b895e640 1 -- 172.21.15.70:0/1353378947 >> 172.21.15.70:0/1353378947 conn(0x7f05b40faf10 msgr2=0x7f05b40fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:20.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.445+0000 7f05b895e640 1 -- 172.21.15.70:0/1353378947 shutdown_connections 2024-04-01T01:39:20.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.445+0000 7f05b895e640 1 -- 172.21.15.70:0/1353378947 wait complete. 2024-04-01T01:39:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.446+0000 7f05b895e640 1 Processor -- start 2024-04-01T01:39:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.446+0000 7f05b895e640 1 -- start start 2024-04-01T01:39:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.446+0000 7f05b895e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b40ff610 0x7f05b419e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.446+0000 7f05b895e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05b4100740 0x7f05b419e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.446+0000 7f05b895e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05b41010e0 0x7f05b41a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.446+0000 7f05b895e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f05b4113920 con 0x7f05b4100740 2024-04-01T01:39:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.446+0000 7f05b895e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f05b41137a0 con 0x7f05b41010e0 2024-04-01T01:39:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.446+0000 7f05b895e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f05b4113aa0 con 0x7f05b40ff610 2024-04-01T01:39:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.446+0000 7f05b259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b40ff610 0x7f05b419e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.447+0000 7f05b1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05b4100740 0x7f05b419e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.447+0000 7f05b259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b40ff610 0x7f05b419e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42838/0 (socket says 172.21.15.70:42838) 2024-04-01T01:39:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.447+0000 7f05b259c640 1 -- 172.21.15.70:0/939112237 learned_addr learned my addr 172.21.15.70:0/939112237 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.447+0000 7f05b2d9d640 1 --2- 172.21.15.70:0/939112237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05b41010e0 0x7f05b41a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.447+0000 7f05b1d9b640 1 -- 172.21.15.70:0/939112237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b40ff610 msgr2=0x7f05b419e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.447+0000 7f05b1d9b640 1 --2- 172.21.15.70:0/939112237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b40ff610 0x7f05b419e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.447+0000 7f05b1d9b640 1 -- 172.21.15.70:0/939112237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05b41010e0 msgr2=0x7f05b41a55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.447+0000 7f05b1d9b640 1 --2- 172.21.15.70:0/939112237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05b41010e0 0x7f05b41a55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.447+0000 7f05b1d9b640 1 -- 172.21.15.70:0/939112237 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b41a5c40 con 0x7f05b4100740 2024-04-01T01:39:20.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.447+0000 7f05b1d9b640 1 --2- 172.21.15.70:0/939112237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05b4100740 0x7f05b419e560 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f05a800b3f0 tx=0x7f05a800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:20.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.448+0000 7f05a37fe640 1 -- 172.21.15.70:0/939112237 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a8015020 con 0x7f05b4100740 2024-04-01T01:39:20.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.448+0000 7f05a37fe640 1 -- 172.21.15.70:0/939112237 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a8011ce0 con 0x7f05b4100740 2024-04-01T01:39:20.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.448+0000 7f05a37fe640 1 -- 172.21.15.70:0/939112237 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a8010de0 con 0x7f05b4100740 2024-04-01T01:39:20.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.449+0000 7f05b895e640 1 -- 172.21.15.70:0/939112237 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05b41a5f30 con 0x7f05b4100740 2024-04-01T01:39:20.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.450+0000 7f05b895e640 1 -- 172.21.15.70:0/939112237 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05b41a63c0 con 0x7f05b4100740 2024-04-01T01:39:20.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.453+0000 7f05bb61d740 1 Processor -- start 2024-04-01T01:39:20.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.453+0000 7f05a37fe640 1 -- 172.21.15.70:0/939112237 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f05a8018020 con 0x7f05b4100740 2024-04-01T01:39:20.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.453+0000 7f05bb61d740 1 -- start start 2024-04-01T01:39:20.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.454+0000 7f05bb61d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949951490 0x55d9499518b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.454+0000 7f05bb61d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 0x55d949956980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.454+0000 7f05bb61d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949956ff0 0x55d949959480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.454+0000 7f05bb61d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d9498b61b0 con 0x55d949951e80 2024-04-01T01:39:20.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.454+0000 7f05bb61d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d94985ec00 con 0x55d949951490 2024-04-01T01:39:20.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.454+0000 7f05bb61d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d9498b2250 con 0x55d949956ff0 2024-04-01T01:39:20.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.454+0000 7f05b259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 0x55d949956980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.454+0000 7f05b259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 0x55d949956980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40710/0 (socket says 172.21.15.70:40710) 2024-04-01T01:39:20.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.454+0000 7f05b259c640 1 -- 172.21.15.70:0/358625374 learned_addr learned my addr 172.21.15.70:0/358625374 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:20.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.454+0000 7f05b1d9b640 1 --2- 172.21.15.70:0/358625374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949956ff0 0x55d949959480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.454+0000 7f05b2d9d640 1 --2- 172.21.15.70:0/358625374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949951490 0x55d9499518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.455+0000 7f05b1d9b640 1 -- 172.21.15.70:0/358625374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949951490 msgr2=0x55d9499518b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:20.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.455+0000 7f05b1d9b640 1 --2- 172.21.15.70:0/358625374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949951490 0x55d9499518b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.455+0000 7f05b1d9b640 1 -- 172.21.15.70:0/358625374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 msgr2=0x55d949956980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:20.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.455+0000 7f05b1d9b640 1 --2- 172.21.15.70:0/358625374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 0x55d949956980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.455+0000 7f05b1d9b640 1 -- 172.21.15.70:0/358625374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d94987e3d0 con 0x55d949956ff0 2024-04-01T01:39:20.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.455+0000 7f05b259c640 1 --2- 172.21.15.70:0/358625374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 0x55d949956980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:20.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.455+0000 7f05b1d9b640 1 --2- 172.21.15.70:0/358625374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949956ff0 0x55d949959480 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f05a80cb720 tx=0x7f05a80d66f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a66f8bb330be360 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:20.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.455+0000 7f05a37fe640 1 --2- 172.21.15.70:0/939112237 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f059c0a2620 0x7f059c0a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.456+0000 7f05b2d9d640 1 --2- 172.21.15.70:0/939112237 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f059c0a2620 0x7f059c0a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.456+0000 7f05a37fe640 1 -- 172.21.15.70:0/939112237 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f05a80ce0b0 con 0x7f05b4100740 2024-04-01T01:39:20.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.457+0000 7f05a0ff9640 1 -- 172.21.15.70:0/358625374 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a80969e0 con 0x55d949956ff0 2024-04-01T01:39:20.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.457+0000 7f05a0ff9640 1 -- 172.21.15.70:0/358625374 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a80db030 con 0x55d949956ff0 2024-04-01T01:39:20.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.457+0000 7f05a0ff9640 1 -- 172.21.15.70:0/358625374 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a8099b90 con 0x55d949956ff0 2024-04-01T01:39:20.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.458+0000 7f05bb61d740 1 -- 172.21.15.70:0/358625374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949956ff0 msgr2=0x55d949959480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.458+0000 7f05bb61d740 1 --2- 172.21.15.70:0/358625374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949956ff0 0x55d949959480 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f05a80cb720 tx=0x7f05a80d66f0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.458+0000 7f05b2d9d640 1 --2- 172.21.15.70:0/939112237 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f059c0a2620 0x7f059c0a4ae0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x55d9499526e0 tx=0x7f05ac009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:20.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.458+0000 7f05bb61d740 1 -- 172.21.15.70:0/358625374 shutdown_connections 2024-04-01T01:39:20.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.458+0000 7f05bb61d740 1 --2- 172.21.15.70:0/358625374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949956ff0 0x55d949959480 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.458+0000 7f05bb61d740 1 --2- 172.21.15.70:0/358625374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 0x55d949956980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.458+0000 7f05bb61d740 1 --2- 172.21.15.70:0/358625374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949951490 0x55d9499518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.458+0000 7f05bb61d740 1 -- 172.21.15.70:0/358625374 >> 172.21.15.70:0/358625374 conn(0x55d94994ca00 msgr2=0x55d94994ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:20.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.458+0000 7f05bb61d740 1 -- 172.21.15.70:0/358625374 shutdown_connections 2024-04-01T01:39:20.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.458+0000 7f05bb61d740 1 -- 172.21.15.70:0/358625374 wait complete. 2024-04-01T01:39:20.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.458+0000 7f05bb61d740 1 -- start start 2024-04-01T01:39:20.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.458+0000 7f05bb61d740 1 Processor -- start 2024-04-01T01:39:20.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05bb61d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949951490 0x55d94995b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05bb61d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 0x55d94995be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05bb61d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949956ff0 0x55d949962e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05bb61d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d949959e20 con 0x55d949951e80 2024-04-01T01:39:20.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05bb61d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d949854470 con 0x55d949956ff0 2024-04-01T01:39:20.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05bb61d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d949959fa0 con 0x55d949951490 2024-04-01T01:39:20.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05bb61d740 10 client.0 _ll_register_callbacks cb 0x55d949871cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:20.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05b2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949956ff0 0x55d949962e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05b2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949956ff0 0x55d949962e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54650/0 (socket says 172.21.15.70:54650) 2024-04-01T01:39:20.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05b2d9d640 1 -- 172.21.15.70:0/1626447525 learned_addr learned my addr 172.21.15.70:0/1626447525 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:20.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05b259c640 1 --2- 172.21.15.70:0/1626447525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949951490 0x55d94995b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05b1d9b640 1 --2- 172.21.15.70:0/1626447525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 0x55d94995be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05b259c640 1 -- 172.21.15.70:0/1626447525 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949956ff0 msgr2=0x55d949962e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:20.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05b259c640 1 --2- 172.21.15.70:0/1626447525 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949956ff0 0x55d949962e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05b259c640 1 -- 172.21.15.70:0/1626447525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 msgr2=0x55d94995be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:20.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05b259c640 1 --2- 172.21.15.70:0/1626447525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 0x55d94995be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.459+0000 7f05b259c640 1 -- 172.21.15.70:0/1626447525 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9495ca3f0 con 0x55d949951490 2024-04-01T01:39:20.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.460+0000 7f05b259c640 1 --2- 172.21.15.70:0/1626447525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949951490 0x55d94995b8c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55d949952280 tx=0x7f05a40043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4efc7f400a1be01e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:20.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.460+0000 7f0599ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:39:20.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.460+0000 7f0599ffb640 1 -- 172.21.15.70:0/1626447525 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a4046070 con 0x55d949951490 2024-04-01T01:39:20.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.460+0000 7f0599ffb640 1 -- 172.21.15.70:0/1626447525 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f05a4005250 con 0x55d949951490 2024-04-01T01:39:20.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.460+0000 7f05bb61d740 1 -- 172.21.15.70:0/1626447525 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05a80db030 con 0x55d949951490 2024-04-01T01:39:20.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.461+0000 7f0599ffb640 1 -- 172.21.15.70:0/1626447525 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f05a4052050 con 0x55d949951490 2024-04-01T01:39:20.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.461+0000 7f05bb61d740 10 client.27100 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:20.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.461+0000 7f05bb61d740 1 -- 172.21.15.70:0/1626447525 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d9496947d0 con 0x55d949951490 2024-04-01T01:39:20.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.461+0000 7f0599ffb640 1 -- 172.21.15.70:0/1626447525 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a80db030 con 0x55d949951490 2024-04-01T01:39:20.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.461+0000 7f0599ffb640 1 -- 172.21.15.70:0/1626447525 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f05a4003710 con 0x55d949951490 2024-04-01T01:39:20.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.461+0000 7f05bb61d740 10 client.27100 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:20.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.461+0000 7f05bb61d740 10 client.27100 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:20.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.461+0000 7f05bb61d740 20 client.27100 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:20.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.461+0000 7f05bb61d740 1 --2- 172.21.15.70:0/1626447525 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d9499677c0 0x55d949969c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.461+0000 7f05bb61d740 4 client.27100 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:20.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.461+0000 7f05bb61d740 1 -- 172.21.15.70:0/1626447525 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9498c36f0 con 0x55d9499677c0 2024-04-01T01:39:20.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.462+0000 7f05b1d9b640 1 --2- 172.21.15.70:0/1626447525 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d9499677c0 0x55d949969c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.462+0000 7f05b1d9b640 1 --2- 172.21.15.70:0/1626447525 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d9499677c0 0x55d949969c10 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=264b78beb09b0d25 server_cookie=4252eac99a1e4560 in_seq=0 out_seq=0 2024-04-01T01:39:20.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.462+0000 7f0599ffb640 10 client.27100 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:20.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.464+0000 7f0599ffb640 1 -- 172.21.15.70:0/1626447525 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d9498c36f0 con 0x55d9499677c0 2024-04-01T01:39:20.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.464+0000 7f0599ffb640 10 client.27100 handle_command_reply: tid=0 2024-04-01T01:39:20.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.464+0000 7f05bb61d740 1 client.27100 shutdown 2024-04-01T01:39:20.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.465+0000 7f05bb61d740 1 -- 172.21.15.70:0/1626447525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949951490 msgr2=0x55d94995b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.465+0000 7f05bb61d740 1 --2- 172.21.15.70:0/1626447525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949951490 0x55d94995b8c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55d949952280 tx=0x7f05a40043b0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.466+0000 7f05bb61d740 1 -- 172.21.15.70:0/1626447525 shutdown_connections 2024-04-01T01:39:20.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.466+0000 7f05bb61d740 1 --2- 172.21.15.70:0/1626447525 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d9499677c0 0x55d949969c10 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.466+0000 7f05bb61d740 1 --2- 172.21.15.70:0/1626447525 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949956ff0 0x55d949962e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.466+0000 7f05bb61d740 1 --2- 172.21.15.70:0/1626447525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 0x55d94995be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.466+0000 7f05bb61d740 1 --2- 172.21.15.70:0/1626447525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949951490 0x55d94995b8c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.466+0000 7f05bb61d740 1 -- 172.21.15.70:0/1626447525 >> 172.21.15.70:0/1626447525 conn(0x55d94994ca00 msgr2=0x55d94994bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:20.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.466+0000 7f0599ffb640 0 client.27100 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:20.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.466+0000 7f05bb61d740 1 -- 172.21.15.70:0/1626447525 shutdown_connections 2024-04-01T01:39:20.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.466+0000 7f05bb61d740 1 -- 172.21.15.70:0/1626447525 wait complete. 2024-04-01T01:39:20.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.467+0000 7f05bb61d740 20 client.27100 trim_cache size 0 max 16384 2024-04-01T01:39:20.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.498+0000 7f05bb61d740 1 Processor -- start 2024-04-01T01:39:20.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.498+0000 7f05bb61d740 1 -- start start 2024-04-01T01:39:20.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.498+0000 7f05bb61d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949951490 0x55d94995aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.499+0000 7f05bb61d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 0x55d949962dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.499+0000 7f05bb61d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949956ff0 0x55d94995b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.499+0000 7f05bb61d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f05b41a8240 con 0x55d949951e80 2024-04-01T01:39:20.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.499+0000 7f05bb61d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f05b41a9d80 con 0x55d949956ff0 2024-04-01T01:39:20.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.499+0000 7f05bb61d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f05b41a8910 con 0x55d949951490 2024-04-01T01:39:20.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.499+0000 7f05b259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949951490 0x55d94995aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.499+0000 7f05b2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949956ff0 0x55d94995b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.499+0000 7f05b259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949951490 0x55d94995aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42868/0 (socket says 172.21.15.70:42868) 2024-04-01T01:39:20.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.499+0000 7f05b2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949956ff0 0x55d94995b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54654/0 (socket says 172.21.15.70:54654) 2024-04-01T01:39:20.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.499+0000 7f05b259c640 1 -- 172.21.15.70:0/3542765442 learned_addr learned my addr 172.21.15.70:0/3542765442 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:20.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.499+0000 7f05b1d9b640 1 --2- 172.21.15.70:0/3542765442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 0x55d949962dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.499+0000 7f05b259c640 1 -- 172.21.15.70:0/3542765442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949956ff0 msgr2=0x55d94995b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:20.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.499+0000 7f05b259c640 1 --2- 172.21.15.70:0/3542765442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949956ff0 0x55d94995b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.499+0000 7f05b259c640 1 -- 172.21.15.70:0/3542765442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 msgr2=0x55d949962dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:20.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.499+0000 7f05b259c640 1 --2- 172.21.15.70:0/3542765442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 0x55d949962dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.499+0000 7f05b259c640 1 -- 172.21.15.70:0/3542765442 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d949948460 con 0x55d949951490 2024-04-01T01:39:20.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.500+0000 7f05b259c640 1 --2- 172.21.15.70:0/3542765442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949951490 0x55d94995aee0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f05a4003b80 tx=0x7f05a402bc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe7916368562266a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:20.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.500+0000 7f059bfff640 1 -- 172.21.15.70:0/3542765442 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a404f070 con 0x55d949951490 2024-04-01T01:39:20.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.500+0000 7f059bfff640 1 -- 172.21.15.70:0/3542765442 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a4005250 con 0x55d949951490 2024-04-01T01:39:20.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.500+0000 7f059bfff640 1 -- 172.21.15.70:0/3542765442 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a4046070 con 0x55d949951490 2024-04-01T01:39:20.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.501+0000 7f05bb61d740 1 -- 172.21.15.70:0/3542765442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949951490 msgr2=0x55d94995aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.501+0000 7f05bb61d740 1 --2- 172.21.15.70:0/3542765442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949951490 0x55d94995aee0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f05a4003b80 tx=0x7f05a402bc40 comp rx=0 tx=0).stop 2024-04-01T01:39:20.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.501+0000 7f05bb61d740 1 -- 172.21.15.70:0/3542765442 shutdown_connections 2024-04-01T01:39:20.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.501+0000 7f05bb61d740 1 --2- 172.21.15.70:0/3542765442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949956ff0 0x55d94995b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.501+0000 7f05bb61d740 1 --2- 172.21.15.70:0/3542765442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949951e80 0x55d949962dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.501+0000 7f05bb61d740 1 --2- 172.21.15.70:0/3542765442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949951490 0x55d94995aee0 secure :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f05a4003b80 tx=0x7f05a402bc40 comp rx=0 tx=0).stop 2024-04-01T01:39:20.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.501+0000 7f05bb61d740 1 -- 172.21.15.70:0/3542765442 >> 172.21.15.70:0/3542765442 conn(0x55d94994ca00 msgr2=0x55d9499504f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:20.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.502+0000 7f05bb61d740 1 -- 172.21.15.70:0/3542765442 shutdown_connections 2024-04-01T01:39:20.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.502+0000 7f05bb61d740 1 -- 172.21.15.70:0/3542765442 wait complete. 2024-04-01T01:39:20.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.502+0000 7f05bb61d740 1 -- start start 2024-04-01T01:39:20.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.502+0000 7f05bb61d740 1 Processor -- start 2024-04-01T01:39:20.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.502+0000 7f05bb61d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949951e80 0x55d9499607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.502+0000 7f05bb61d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949956ff0 0x55d949960d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.503+0000 7f05b259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949951e80 0x55d9499607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.503+0000 7f05b259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949951e80 0x55d9499607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54670/0 (socket says 172.21.15.70:54670) 2024-04-01T01:39:20.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.503+0000 7f05bb61d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949961c50 0x55d9499620d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.503+0000 7f05bb61d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f05b41a96b0 con 0x55d949956ff0 2024-04-01T01:39:20.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.503+0000 7f05bb61d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f05b41a8fe0 con 0x55d949951e80 2024-04-01T01:39:20.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.503+0000 7f05bb61d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f05b410f600 con 0x55d949961c50 2024-04-01T01:39:20.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.503+0000 7f05bb61d740 10 client.0 _ll_register_callbacks cb 0x7f05a4004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:20.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.503+0000 7f05b259c640 1 -- 172.21.15.70:0/2665924884 learned_addr learned my addr 172.21.15.70:0/2665924884 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:20.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.503+0000 7f05b1d9b640 1 --2- 172.21.15.70:0/2665924884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949956ff0 0x55d949960d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.503+0000 7f05b2d9d640 1 --2- 172.21.15.70:0/2665924884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949961c50 0x55d9499620d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.503+0000 7f05b259c640 1 -- 172.21.15.70:0/2665924884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949961c50 msgr2=0x55d9499620d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:20.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.503+0000 7f05b259c640 1 --2- 172.21.15.70:0/2665924884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949961c50 0x55d9499620d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.503+0000 7f05b259c640 1 -- 172.21.15.70:0/2665924884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949956ff0 msgr2=0x55d949960d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:20.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.503+0000 7f05b259c640 1 --2- 172.21.15.70:0/2665924884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949956ff0 0x55d949960d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.503+0000 7f05b259c640 1 -- 172.21.15.70:0/2665924884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d949962610 con 0x55d949951e80 2024-04-01T01:39:20.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.504+0000 7f05b259c640 1 --2- 172.21.15.70:0/2665924884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949951e80 0x55d9499607c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f05a402c850 tx=0x7f05a4003f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3658def72fc9d517 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:20.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.504+0000 7f0599ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:20.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.504+0000 7f0599ffb640 1 -- 172.21.15.70:0/2665924884 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a404f040 con 0x55d949951e80 2024-04-01T01:39:20.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.504+0000 7f0599ffb640 1 -- 172.21.15.70:0/2665924884 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f05a4005600 con 0x55d949951e80 2024-04-01T01:39:20.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.504+0000 7f05bb61d740 1 -- 172.21.15.70:0/2665924884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05a4005250 con 0x55d949951e80 2024-04-01T01:39:20.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.504+0000 7f0599ffb640 1 -- 172.21.15.70:0/2665924884 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f05a4046040 con 0x55d949951e80 2024-04-01T01:39:20.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.504+0000 7f05bb61d740 10 client.27239 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:20.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.505+0000 7f05bb61d740 1 -- 172.21.15.70:0/2665924884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d9499ce030 con 0x55d949951e80 2024-04-01T01:39:20.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.505+0000 7f0599ffb640 1 -- 172.21.15.70:0/2665924884 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f05a4005250 con 0x55d949951e80 2024-04-01T01:39:20.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.505+0000 7f0599ffb640 1 -- 172.21.15.70:0/2665924884 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f05a406a480 con 0x55d949951e80 2024-04-01T01:39:20.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.505+0000 7f05bb61d740 10 client.27239 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:20.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.505+0000 7f05bb61d740 10 client.27239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:20.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.505+0000 7f05bb61d740 20 client.27239 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:20.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.505+0000 7f05bb61d740 1 --2- 172.21.15.70:0/2665924884 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d9499ce8c0 0x55d9499d0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:20.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.505+0000 7f05bb61d740 4 client.27239 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:39:20.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.505+0000 7f05bb61d740 1 -- 172.21.15.70:0/2665924884 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d949969fa0 con 0x55d9499ce8c0 2024-04-01T01:39:20.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.506+0000 7f05b1d9b640 1 --2- 172.21.15.70:0/2665924884 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d9499ce8c0 0x55d9499d0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:20.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.508+0000 7f05b1d9b640 1 --2- 172.21.15.70:0/2665924884 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d9499ce8c0 0x55d9499d0ce0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=482758187d755131 server_cookie=79f9cb3e1536fdb in_seq=0 out_seq=0 2024-04-01T01:39:20.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.508+0000 7f0599ffb640 10 client.27239 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:20.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.508+0000 7f0599ffb640 1 -- 172.21.15.70:0/2665924884 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d949969fa0 con 0x55d9499ce8c0 2024-04-01T01:39:20.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.508+0000 7f0599ffb640 10 client.27239 handle_command_reply: tid=0 2024-04-01T01:39:20.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.508+0000 7f05bb61d740 1 client.27239 shutdown 2024-04-01T01:39:20.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.511+0000 7f05bb61d740 1 -- 172.21.15.70:0/2665924884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949951e80 msgr2=0x55d9499607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.511+0000 7f05bb61d740 1 --2- 172.21.15.70:0/2665924884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949951e80 0x55d9499607c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f05a402c850 tx=0x7f05a4003f80 comp rx=0 tx=0).stop 2024-04-01T01:39:20.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.511+0000 7f05bb61d740 1 -- 172.21.15.70:0/2665924884 shutdown_connections 2024-04-01T01:39:20.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.511+0000 7f05bb61d740 1 --2- 172.21.15.70:0/2665924884 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d9499ce8c0 0x55d9499d0ce0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.511+0000 7f05bb61d740 1 --2- 172.21.15.70:0/2665924884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d949961c50 0x55d9499620d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.511+0000 7f05bb61d740 1 --2- 172.21.15.70:0/2665924884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d949956ff0 0x55d949960d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.511+0000 7f0599ffb640 0 client.27239 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:20.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.512+0000 7f05bb61d740 1 --2- 172.21.15.70:0/2665924884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d949951e80 0x55d9499607c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.512+0000 7f05bb61d740 1 -- 172.21.15.70:0/2665924884 >> 172.21.15.70:0/2665924884 conn(0x55d94994ca00 msgr2=0x55d94994b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:20.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.512+0000 7f05bb61d740 1 -- 172.21.15.70:0/2665924884 shutdown_connections 2024-04-01T01:39:20.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.512+0000 7f05bb61d740 1 -- 172.21.15.70:0/2665924884 wait complete. 2024-04-01T01:39:20.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.512+0000 7f05bb61d740 20 client.27239 trim_cache size 0 max 16384 2024-04-01T01:39:20.512 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:39:20.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.513+0000 7f05b895e640 1 -- 172.21.15.70:0/939112237 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f059c0a2620 msgr2=0x7f059c0a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.513+0000 7f05b895e640 1 --2- 172.21.15.70:0/939112237 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f059c0a2620 0x7f059c0a4ae0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x55d9499526e0 tx=0x7f05ac009500 comp rx=0 tx=0).stop 2024-04-01T01:39:20.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.513+0000 7f05b895e640 1 -- 172.21.15.70:0/939112237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05b4100740 msgr2=0x7f05b419e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:20.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.513+0000 7f05b895e640 1 --2- 172.21.15.70:0/939112237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05b4100740 0x7f05b419e560 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f05a800b3f0 tx=0x7f05a800b8b0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.513+0000 7f05b895e640 1 -- 172.21.15.70:0/939112237 shutdown_connections 2024-04-01T01:39:20.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.513+0000 7f05b895e640 1 --2- 172.21.15.70:0/939112237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05b41010e0 0x7f05b41a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.513+0000 7f05b895e640 1 --2- 172.21.15.70:0/939112237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05b4100740 0x7f05b419e560 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.513+0000 7f05b895e640 1 --2- 172.21.15.70:0/939112237 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f059c0a2620 0x7f059c0a4ae0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.513+0000 7f05b895e640 1 --2- 172.21.15.70:0/939112237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b40ff610 0x7f05b419e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:20.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.513+0000 7f05b895e640 1 -- 172.21.15.70:0/939112237 >> 172.21.15.70:0/939112237 conn(0x7f05b40faf10 msgr2=0x7f05b4101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:20.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.513+0000 7f05b895e640 1 -- 172.21.15.70:0/939112237 shutdown_connections 2024-04-01T01:39:20.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:20.513+0000 7f05b895e640 1 -- 172.21.15.70:0/939112237 wait complete. 2024-04-01T01:39:21.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:20 smithi189 ceph-mon[35753]: pgmap v534: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 22 MiB/s wr, 910 op/s 2024-04-01T01:39:21.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:20 smithi070 ceph-mon[32179]: pgmap v534: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 22 MiB/s wr, 910 op/s 2024-04-01T01:39:21.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:20 smithi081 ceph-mon[35142]: pgmap v534: 129 pgs: 129 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 22 MiB/s wr, 910 op/s 2024-04-01T01:39:21.535 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:39:21.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:39:21.535 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:39:21.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.735+0000 7f972f68f640 1 -- 172.21.15.70:0/1140398456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f972810b0c0 msgr2=0x7f972810d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.735+0000 7f972f68f640 1 --2- 172.21.15.70:0/1140398456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f972810b0c0 0x7f972810d530 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9718009900 tx=0x7f971802bc40 comp rx=0 tx=0).stop 2024-04-01T01:39:21.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.735+0000 7f972f68f640 1 -- 172.21.15.70:0/1140398456 shutdown_connections 2024-04-01T01:39:21.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.735+0000 7f972f68f640 1 --2- 172.21.15.70:0/1140398456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f972810da70 0x7f972810def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.735+0000 7f972f68f640 1 --2- 172.21.15.70:0/1140398456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f972810b0c0 0x7f972810d530 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.735+0000 7f972f68f640 1 --2- 172.21.15.70:0/1140398456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9728108750 0x7f972810ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.735+0000 7f972f68f640 1 -- 172.21.15.70:0/1140398456 >> 172.21.15.70:0/1140398456 conn(0x7f972806e5b0 msgr2=0x7f97280709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:21.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.736+0000 7f972f68f640 1 -- 172.21.15.70:0/1140398456 shutdown_connections 2024-04-01T01:39:21.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.737+0000 7f972f68f640 1 -- 172.21.15.70:0/1140398456 wait complete. 2024-04-01T01:39:21.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.737+0000 7f972f68f640 1 Processor -- start 2024-04-01T01:39:21.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.737+0000 7f972f68f640 1 -- start start 2024-04-01T01:39:21.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.737+0000 7f972f68f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9728108750 0x7f97281a67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.738+0000 7f972f68f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f972810b0c0 0x7f97281a6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.738+0000 7f972f68f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f972810da70 0x7f97281add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.738+0000 7f972f68f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f972811c120 con 0x7f9728108750 2024-04-01T01:39:21.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.738+0000 7f972f68f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f972811bfa0 con 0x7f972810da70 2024-04-01T01:39:21.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.738+0000 7f972f68f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f972811c2a0 con 0x7f972810b0c0 2024-04-01T01:39:21.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.738+0000 7f972d42b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9728108750 0x7f97281a67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.738+0000 7f972d42b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9728108750 0x7f97281a67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40754/0 (socket says 172.21.15.70:40754) 2024-04-01T01:39:21.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.738+0000 7f972d42b640 1 -- 172.21.15.70:0/2621100867 learned_addr learned my addr 172.21.15.70:0/2621100867 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:21.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.738+0000 7f972dc2c640 1 --2- 172.21.15.70:0/2621100867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f972810da70 0x7f97281add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.738+0000 7f972d42b640 1 -- 172.21.15.70:0/2621100867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f972810b0c0 msgr2=0x7f97281a6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.738+0000 7f972d42b640 1 --2- 172.21.15.70:0/2621100867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f972810b0c0 0x7f97281a6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.738+0000 7f972d42b640 1 -- 172.21.15.70:0/2621100867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f972810da70 msgr2=0x7f97281add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.738+0000 7f972d42b640 1 --2- 172.21.15.70:0/2621100867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f972810da70 0x7f97281add50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.738+0000 7f972d42b640 1 -- 172.21.15.70:0/2621100867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97281ae3c0 con 0x7f9728108750 2024-04-01T01:39:21.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.739+0000 7f972d42b640 1 --2- 172.21.15.70:0/2621100867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9728108750 0x7f97281a67f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f972400e9f0 tx=0x7f972400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:21.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.739+0000 7f971e7fc640 1 -- 172.21.15.70:0/2621100867 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f972400cd80 con 0x7f9728108750 2024-04-01T01:39:21.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.739+0000 7f972f68f640 1 -- 172.21.15.70:0/2621100867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97281ae6b0 con 0x7f9728108750 2024-04-01T01:39:21.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.739+0000 7f972f68f640 1 -- 172.21.15.70:0/2621100867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97281aec50 con 0x7f9728108750 2024-04-01T01:39:21.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.739+0000 7f971e7fc640 1 -- 172.21.15.70:0/2621100867 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f97240108e0 con 0x7f9728108750 2024-04-01T01:39:21.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.740+0000 7f971e7fc640 1 -- 172.21.15.70:0/2621100867 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9724026060 con 0x7f9728108750 2024-04-01T01:39:21.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.742+0000 7f971e7fc640 1 -- 172.21.15.70:0/2621100867 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f9724010430 con 0x7f9728108750 2024-04-01T01:39:21.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.743+0000 7f973226d740 1 Processor -- start 2024-04-01T01:39:21.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.743+0000 7f973226d740 1 -- start start 2024-04-01T01:39:21.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.743+0000 7f973226d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df3da10 0x55fa3df3de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.743+0000 7f973226d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3e400 0x55fa3df42f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.743+0000 7f973226d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df43570 0x55fa3df45a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.743+0000 7f973226d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fa3de3f0b0 con 0x55fa3df3e400 2024-04-01T01:39:21.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.743+0000 7f973226d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55fa3dea1ea0 con 0x55fa3df43570 2024-04-01T01:39:21.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.743+0000 7f973226d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fa3de4ac00 con 0x55fa3df3da10 2024-04-01T01:39:21.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.743+0000 7f971e7fc640 1 --2- 172.21.15.70:0/2621100867 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f97140a2560 0x7f97140a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.743+0000 7f972cc2a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df43570 0x55fa3df45a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.743+0000 7f971e7fc640 1 -- 172.21.15.70:0/2621100867 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f97240ceea0 con 0x7f9728108750 2024-04-01T01:39:21.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.743+0000 7f972dc2c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df3da10 0x55fa3df3de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.743+0000 7f972dc2c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df3da10 0x55fa3df3de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42916/0 (socket says 172.21.15.70:42916) 2024-04-01T01:39:21.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.743+0000 7f972dc2c640 1 -- 172.21.15.70:0/384505612 learned_addr learned my addr 172.21.15.70:0/384505612 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:21.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.743+0000 7f972d42b640 1 --2- 172.21.15.70:0/384505612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3e400 0x55fa3df42f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.743+0000 7f972dc2c640 1 --2- 172.21.15.70:0/2621100867 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f97140a2560 0x7f97140a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.744+0000 7f972cc2a640 1 -- 172.21.15.70:0/384505612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df3da10 msgr2=0x55fa3df3de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:21.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.744+0000 7f972cc2a640 1 --2- 172.21.15.70:0/384505612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df3da10 0x55fa3df3de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.744+0000 7f972cc2a640 1 -- 172.21.15.70:0/384505612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3e400 msgr2=0x55fa3df42f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:21.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.744+0000 7f972cc2a640 1 --2- 172.21.15.70:0/384505612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3e400 0x55fa3df42f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.744+0000 7f972cc2a640 1 -- 172.21.15.70:0/384505612 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa3dc7e7e0 con 0x55fa3df43570 2024-04-01T01:39:21.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.744+0000 7f972cc2a640 1 --2- 172.21.15.70:0/384505612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df43570 0x55fa3df45a00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9718036040 tx=0x7f9718005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9a9153150bb9281 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:21.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.744+0000 7f972dc2c640 1 --2- 172.21.15.70:0/2621100867 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f97140a2560 0x7f97140a4a20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f972000bdc0 tx=0x7f9720009c80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:21.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.744+0000 7f97137fe640 1 -- 172.21.15.70:0/384505612 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9718043020 con 0x55fa3df43570 2024-04-01T01:39:21.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.744+0000 7f97137fe640 1 -- 172.21.15.70:0/384505612 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f971802c8f0 con 0x55fa3df43570 2024-04-01T01:39:21.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.744+0000 7f97137fe640 1 -- 172.21.15.70:0/384505612 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9718003a00 con 0x55fa3df43570 2024-04-01T01:39:21.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.745+0000 7f973226d740 1 -- 172.21.15.70:0/384505612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df43570 msgr2=0x55fa3df45a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.745+0000 7f973226d740 1 --2- 172.21.15.70:0/384505612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df43570 0x55fa3df45a00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9718036040 tx=0x7f9718005be0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.745+0000 7f973226d740 1 -- 172.21.15.70:0/384505612 shutdown_connections 2024-04-01T01:39:21.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.745+0000 7f973226d740 1 --2- 172.21.15.70:0/384505612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df43570 0x55fa3df45a00 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.745+0000 7f973226d740 1 --2- 172.21.15.70:0/384505612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3e400 0x55fa3df42f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.745+0000 7f973226d740 1 --2- 172.21.15.70:0/384505612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df3da10 0x55fa3df3de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.745+0000 7f973226d740 1 -- 172.21.15.70:0/384505612 >> 172.21.15.70:0/384505612 conn(0x55fa3df38f80 msgr2=0x55fa3df39380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:21.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.745+0000 7f973226d740 1 -- 172.21.15.70:0/384505612 shutdown_connections 2024-04-01T01:39:21.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.745+0000 7f973226d740 1 -- 172.21.15.70:0/384505612 wait complete. 2024-04-01T01:39:21.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.746+0000 7f973226d740 1 -- start start 2024-04-01T01:39:21.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.746+0000 7f973226d740 1 Processor -- start 2024-04-01T01:39:21.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.746+0000 7f973226d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3da10 0x55fa3df47d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.746+0000 7f973226d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3e400 0x55fa3df48270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.746+0000 7f973226d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df43570 0x55fa3df4f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.746+0000 7f973226d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fa3de403d0 con 0x55fa3df3da10 2024-04-01T01:39:21.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f973226d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55fa3de9df40 con 0x55fa3df3e400 2024-04-01T01:39:21.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f972cc2a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3e400 0x55fa3df48270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f972cc2a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3e400 0x55fa3df48270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54710/0 (socket says 172.21.15.70:54710) 2024-04-01T01:39:21.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f972cc2a640 1 -- 172.21.15.70:0/3455689784 learned_addr learned my addr 172.21.15.70:0/3455689784 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f972d42b640 1 --2- 172.21.15.70:0/3455689784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3da10 0x55fa3df47d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f973226d740 1 -- 172.21.15.70:0/3455689784 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fa3df464c0 con 0x55fa3df43570 2024-04-01T01:39:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f972dc2c640 1 --2- 172.21.15.70:0/3455689784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df43570 0x55fa3df4f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f973226d740 10 client.0 _ll_register_callbacks cb 0x55fa3de5db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f972cc2a640 1 -- 172.21.15.70:0/3455689784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df43570 msgr2=0x55fa3df4f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f972cc2a640 1 --2- 172.21.15.70:0/3455689784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df43570 0x55fa3df4f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f972cc2a640 1 -- 172.21.15.70:0/3455689784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3da10 msgr2=0x55fa3df47d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f972cc2a640 1 --2- 172.21.15.70:0/3455689784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3da10 0x55fa3df47d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f972cc2a640 1 -- 172.21.15.70:0/3455689784 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa3dcdef70 con 0x55fa3df3e400 2024-04-01T01:39:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f972d42b640 1 --2- 172.21.15.70:0/3455689784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3da10 0x55fa3df47d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f972cc2a640 1 --2- 172.21.15.70:0/3455689784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3e400 0x55fa3df48270 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f971802c250 tx=0x7f9718034b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b94576d619fb2b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f9710ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.747+0000 7f9710ff9640 1 -- 172.21.15.70:0/3455689784 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9718043030 con 0x55fa3df3e400 2024-04-01T01:39:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.748+0000 7f973226d740 1 -- 172.21.15.70:0/3455689784 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f971802c8f0 con 0x55fa3df3e400 2024-04-01T01:39:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.748+0000 7f9710ff9640 1 -- 172.21.15.70:0/3455689784 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9718048420 con 0x55fa3df3e400 2024-04-01T01:39:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.748+0000 7f9710ff9640 1 -- 172.21.15.70:0/3455689784 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9718059050 con 0x55fa3df3e400 2024-04-01T01:39:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.748+0000 7f973226d740 10 client.27251 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.748+0000 7f973226d740 1 -- 172.21.15.70:0/3455689784 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fa3db76170 con 0x55fa3df3e400 2024-04-01T01:39:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.748+0000 7f9710ff9640 1 -- 172.21.15.70:0/3455689784 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f971802c8f0 con 0x55fa3df3e400 2024-04-01T01:39:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.748+0000 7f9710ff9640 1 -- 172.21.15.70:0/3455689784 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f971803f490 con 0x55fa3df3e400 2024-04-01T01:39:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.748+0000 7f973226d740 10 client.27251 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.748+0000 7f973226d740 10 client.27251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.748+0000 7f973226d740 20 client.27251 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.748+0000 7f973226d740 1 --2- 172.21.15.70:0/3455689784 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55fa3df533d0 0x55fa3df557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.748+0000 7f973226d740 4 client.27251 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.748+0000 7f973226d740 1 -- 172.21.15.70:0/3455689784 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa3deaf3e0 con 0x55fa3df533d0 2024-04-01T01:39:21.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.748+0000 7f972d42b640 1 --2- 172.21.15.70:0/3455689784 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55fa3df533d0 0x55fa3df557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.749+0000 7f9710ff9640 10 client.27251 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:21.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.749+0000 7f972d42b640 1 --2- 172.21.15.70:0/3455689784 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55fa3df533d0 0x55fa3df557c0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=476bce8fcfde73d6 server_cookie=7e036a94ac88d793 in_seq=0 out_seq=0 2024-04-01T01:39:21.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.750+0000 7f9710ff9640 1 -- 172.21.15.70:0/3455689784 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fa3deaf3e0 con 0x55fa3df533d0 2024-04-01T01:39:21.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.750+0000 7f9710ff9640 10 client.27251 handle_command_reply: tid=0 2024-04-01T01:39:21.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.750+0000 7f973226d740 1 client.27251 shutdown 2024-04-01T01:39:21.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.750+0000 7f973226d740 1 -- 172.21.15.70:0/3455689784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3e400 msgr2=0x55fa3df48270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.750+0000 7f973226d740 1 --2- 172.21.15.70:0/3455689784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3e400 0x55fa3df48270 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f971802c250 tx=0x7f9718034b10 comp rx=0 tx=0).stop 2024-04-01T01:39:21.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.750+0000 7f973226d740 1 -- 172.21.15.70:0/3455689784 shutdown_connections 2024-04-01T01:39:21.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.750+0000 7f973226d740 1 --2- 172.21.15.70:0/3455689784 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55fa3df533d0 0x55fa3df557c0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.750+0000 7f973226d740 1 --2- 172.21.15.70:0/3455689784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df43570 0x55fa3df4f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.750+0000 7f9710ff9640 0 client.27251 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:21.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.750+0000 7f973226d740 1 --2- 172.21.15.70:0/3455689784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3e400 0x55fa3df48270 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.750+0000 7f973226d740 1 --2- 172.21.15.70:0/3455689784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3da10 0x55fa3df47d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.750+0000 7f973226d740 1 -- 172.21.15.70:0/3455689784 >> 172.21.15.70:0/3455689784 conn(0x55fa3df38f80 msgr2=0x55fa3df38180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:21.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.750+0000 7f973226d740 1 -- 172.21.15.70:0/3455689784 shutdown_connections 2024-04-01T01:39:21.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.750+0000 7f973226d740 1 -- 172.21.15.70:0/3455689784 wait complete. 2024-04-01T01:39:21.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.750+0000 7f973226d740 20 client.27251 trim_cache size 0 max 16384 2024-04-01T01:39:21.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.769+0000 7f973226d740 1 Processor -- start 2024-04-01T01:39:21.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.769+0000 7f973226d740 1 -- start start 2024-04-01T01:39:21.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.769+0000 7f973226d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3da10 0x55fa3df378f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.769+0000 7f973226d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3e400 0x55fa3df4f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.769+0000 7f973226d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df43570 0x55fa3df37e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.769+0000 7f973226d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f970c00fcd0 con 0x55fa3df3da10 2024-04-01T01:39:21.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.769+0000 7f973226d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f970c011810 con 0x55fa3df3e400 2024-04-01T01:39:21.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.769+0000 7f973226d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f970c0103a0 con 0x55fa3df43570 2024-04-01T01:39:21.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.770+0000 7f972d42b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3da10 0x55fa3df378f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.770+0000 7f972cc2a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3e400 0x55fa3df4f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.770+0000 7f972d42b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3da10 0x55fa3df378f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40782/0 (socket says 172.21.15.70:40782) 2024-04-01T01:39:21.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.770+0000 7f972d42b640 1 -- 172.21.15.70:0/249955177 learned_addr learned my addr 172.21.15.70:0/249955177 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:21.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.770+0000 7f972dc2c640 1 --2- 172.21.15.70:0/249955177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df43570 0x55fa3df37e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.770+0000 7f972d42b640 1 -- 172.21.15.70:0/249955177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df43570 msgr2=0x55fa3df37e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:21.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.770+0000 7f972d42b640 1 --2- 172.21.15.70:0/249955177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df43570 0x55fa3df37e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.770+0000 7f972d42b640 1 -- 172.21.15.70:0/249955177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3e400 msgr2=0x55fa3df4f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:21.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.770+0000 7f972d42b640 1 --2- 172.21.15.70:0/249955177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3e400 0x55fa3df4f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.770+0000 7f972d42b640 1 -- 172.21.15.70:0/249955177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa3df46400 con 0x55fa3df3da10 2024-04-01T01:39:21.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.770+0000 7f972cc2a640 1 --2- 172.21.15.70:0/249955177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3e400 0x55fa3df4f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:21.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.770+0000 7f972d42b640 1 --2- 172.21.15.70:0/249955177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3da10 0x55fa3df378f0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9724018490 tx=0x7f9724086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddcd19c0119869 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:21.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.770+0000 7f9712ffd640 1 -- 172.21.15.70:0/249955177 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97240a3f40 con 0x55fa3df3da10 2024-04-01T01:39:21.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.770+0000 7f9712ffd640 1 -- 172.21.15.70:0/249955177 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f97240995d0 con 0x55fa3df3da10 2024-04-01T01:39:21.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.770+0000 7f9712ffd640 1 -- 172.21.15.70:0/249955177 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f972409f370 con 0x55fa3df3da10 2024-04-01T01:39:21.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.771+0000 7f973226d740 1 -- 172.21.15.70:0/249955177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3da10 msgr2=0x55fa3df378f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.771+0000 7f973226d740 1 --2- 172.21.15.70:0/249955177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3da10 0x55fa3df378f0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9724018490 tx=0x7f9724086040 comp rx=0 tx=0).stop 2024-04-01T01:39:21.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.771+0000 7f973226d740 1 -- 172.21.15.70:0/249955177 shutdown_connections 2024-04-01T01:39:21.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.771+0000 7f973226d740 1 --2- 172.21.15.70:0/249955177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df43570 0x55fa3df37e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.771+0000 7f973226d740 1 --2- 172.21.15.70:0/249955177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3e400 0x55fa3df4f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.771+0000 7f973226d740 1 --2- 172.21.15.70:0/249955177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df3da10 0x55fa3df378f0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.771+0000 7f973226d740 1 -- 172.21.15.70:0/249955177 >> 172.21.15.70:0/249955177 conn(0x55fa3df38f80 msgr2=0x55fa3df3f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:21.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.771+0000 7f973226d740 1 -- 172.21.15.70:0/249955177 shutdown_connections 2024-04-01T01:39:21.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.771+0000 7f973226d740 1 -- 172.21.15.70:0/249955177 wait complete. 2024-04-01T01:39:21.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.772+0000 7f973226d740 1 -- start start 2024-04-01T01:39:21.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.772+0000 7f973226d740 1 Processor -- start 2024-04-01T01:39:21.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.772+0000 7f973226d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3da10 0x55fa3df4cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.772+0000 7f973226d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df3e400 0x55fa3df4d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.772+0000 7f972d42b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3da10 0x55fa3df4cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.772+0000 7f972d42b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3da10 0x55fa3df4cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54738/0 (socket says 172.21.15.70:54738) 2024-04-01T01:39:21.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.772+0000 7f973226d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df43570 0x55fa3df4e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.772+0000 7f972d42b640 1 -- 172.21.15.70:0/635217595 learned_addr learned my addr 172.21.15.70:0/635217595 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:21.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f973226d740 1 -- 172.21.15.70:0/635217595 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f970c011140 con 0x55fa3df43570 2024-04-01T01:39:21.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f973226d740 1 -- 172.21.15.70:0/635217595 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f970c010a70 con 0x55fa3df3da10 2024-04-01T01:39:21.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f973226d740 1 -- 172.21.15.70:0/635217595 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f970c00dcb0 con 0x55fa3df3e400 2024-04-01T01:39:21.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f972cc2a640 1 --2- 172.21.15.70:0/635217595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df3e400 0x55fa3df4d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f972dc2c640 1 --2- 172.21.15.70:0/635217595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df43570 0x55fa3df4e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f973226d740 10 client.0 _ll_register_callbacks cb 0x7f9718009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:21.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f972d42b640 1 -- 172.21.15.70:0/635217595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df3e400 msgr2=0x55fa3df4d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:21.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f972d42b640 1 --2- 172.21.15.70:0/635217595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df3e400 0x55fa3df4d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f972d42b640 1 -- 172.21.15.70:0/635217595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df43570 msgr2=0x55fa3df4e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:21.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f972d42b640 1 --2- 172.21.15.70:0/635217595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df43570 0x55fa3df4e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f972d42b640 1 -- 172.21.15.70:0/635217595 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa3df4e740 con 0x55fa3df3da10 2024-04-01T01:39:21.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f973226d740 1 -- 172.21.15.70:0/635217595 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97240995d0 con 0x55fa3df3da10 2024-04-01T01:39:21.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f972d42b640 1 --2- 172.21.15.70:0/635217595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3da10 0x55fa3df4cc00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f972400c6a0 tx=0x7f9724010720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4bcf106edf80e33 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:21.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f9710ff9640 10 client.27257 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:21.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f9710ff9640 1 -- 172.21.15.70:0/635217595 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97240ae020 con 0x55fa3df3da10 2024-04-01T01:39:21.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f9710ff9640 1 -- 172.21.15.70:0/635217595 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f97240bb0d0 con 0x55fa3df3da10 2024-04-01T01:39:21.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f9710ff9640 1 -- 172.21.15.70:0/635217595 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f97240bb310 con 0x55fa3df3da10 2024-04-01T01:39:21.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f973226d740 10 client.27257 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:21.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.773+0000 7f973226d740 1 -- 172.21.15.70:0/635217595 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55fa3dfb9880 con 0x55fa3df3da10 2024-04-01T01:39:21.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.774+0000 7f9710ff9640 1 -- 172.21.15.70:0/635217595 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f97240af250 con 0x55fa3df3da10 2024-04-01T01:39:21.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.774+0000 7f973226d740 10 client.27257 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:21.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.774+0000 7f973226d740 10 client.27257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:21.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.774+0000 7f973226d740 20 client.27257 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:21.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.774+0000 7f973226d740 1 --2- 172.21.15.70:0/635217595 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55fa3df533d0 0x55fa3dfbc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.774+0000 7f973226d740 4 client.27257 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:39:21.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.774+0000 7f973226d740 1 -- 172.21.15.70:0/635217595 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fa3dea9730 con 0x55fa3df533d0 2024-04-01T01:39:21.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.774+0000 7f972cc2a640 1 --2- 172.21.15.70:0/635217595 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55fa3df533d0 0x55fa3dfbc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.774+0000 7f9710ff9640 10 client.27257 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:21.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.774+0000 7f972cc2a640 1 --2- 172.21.15.70:0/635217595 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55fa3df533d0 0x55fa3dfbc120 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32321257a1dcca94 server_cookie=3e07076906d3a246 in_seq=0 out_seq=0 2024-04-01T01:39:21.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.775+0000 7f9710ff9640 1 -- 172.21.15.70:0/635217595 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fa3dea9730 con 0x55fa3df533d0 2024-04-01T01:39:21.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.775+0000 7f9710ff9640 10 client.27257 handle_command_reply: tid=0 2024-04-01T01:39:21.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.775+0000 7f973226d740 1 client.27257 shutdown 2024-04-01T01:39:21.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.776+0000 7f973226d740 1 -- 172.21.15.70:0/635217595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3da10 msgr2=0x55fa3df4cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.776+0000 7f973226d740 1 --2- 172.21.15.70:0/635217595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3da10 0x55fa3df4cc00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f972400c6a0 tx=0x7f9724010720 comp rx=0 tx=0).stop 2024-04-01T01:39:21.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.776+0000 7f973226d740 1 -- 172.21.15.70:0/635217595 shutdown_connections 2024-04-01T01:39:21.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.776+0000 7f973226d740 1 --2- 172.21.15.70:0/635217595 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55fa3df533d0 0x55fa3dfbc120 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.776+0000 7f9710ff9640 0 client.27257 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:21.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.776+0000 7f973226d740 1 --2- 172.21.15.70:0/635217595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fa3df43570 0x55fa3df4e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.776+0000 7f973226d740 1 --2- 172.21.15.70:0/635217595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa3df3e400 0x55fa3df4d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.776+0000 7f973226d740 1 --2- 172.21.15.70:0/635217595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fa3df3da10 0x55fa3df4cc00 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.776+0000 7f973226d740 1 -- 172.21.15.70:0/635217595 >> 172.21.15.70:0/635217595 conn(0x55fa3df38f80 msgr2=0x55fa3df3c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:21.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.776+0000 7f973226d740 1 -- 172.21.15.70:0/635217595 shutdown_connections 2024-04-01T01:39:21.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.776+0000 7f973226d740 1 -- 172.21.15.70:0/635217595 wait complete. 2024-04-01T01:39:21.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.776+0000 7f973226d740 20 client.27257 trim_cache size 0 max 16384 2024-04-01T01:39:21.781 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:39:21.781 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:39:21.781 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "938dc8a0-6cca-4e61-8f31-d954f1302836", 2024-04-01T01:39:21.781 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:39:21.781 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:39:21.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.776+0000 7f972f68f640 1 -- 172.21.15.70:0/2621100867 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f97140a2560 msgr2=0x7f97140a4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.776+0000 7f972f68f640 1 --2- 172.21.15.70:0/2621100867 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f97140a2560 0x7f97140a4a20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f972000bdc0 tx=0x7f9720009c80 comp rx=0 tx=0).stop 2024-04-01T01:39:21.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.776+0000 7f972f68f640 1 -- 172.21.15.70:0/2621100867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9728108750 msgr2=0x7f97281a67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.776+0000 7f972f68f640 1 --2- 172.21.15.70:0/2621100867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9728108750 0x7f97281a67f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f972400e9f0 tx=0x7f972400eeb0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.777+0000 7f972f68f640 1 -- 172.21.15.70:0/2621100867 shutdown_connections 2024-04-01T01:39:21.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.777+0000 7f972f68f640 1 --2- 172.21.15.70:0/2621100867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f972810da70 0x7f97281add50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.777+0000 7f972f68f640 1 --2- 172.21.15.70:0/2621100867 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f97140a2560 0x7f97140a4a20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.777+0000 7f972f68f640 1 --2- 172.21.15.70:0/2621100867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f972810b0c0 0x7f97281a6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.777+0000 7f972f68f640 1 --2- 172.21.15.70:0/2621100867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9728108750 0x7f97281a67f0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.777+0000 7f972f68f640 1 -- 172.21.15.70:0/2621100867 >> 172.21.15.70:0/2621100867 conn(0x7f972806e5b0 msgr2=0x7f9728109d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:21.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.777+0000 7f972f68f640 1 -- 172.21.15.70:0/2621100867 shutdown_connections 2024-04-01T01:39:21.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.777+0000 7f972f68f640 1 -- 172.21.15.70:0/2621100867 wait complete. 2024-04-01T01:39:21.789 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:39:21.790 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:39:21.898 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 275.0 s, 873.2 tps, lat 96.160 ms stddev 154.493 2024-04-01T01:39:21.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.949+0000 7f8301aed640 1 -- 172.21.15.70:0/2761897086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82fc105540 msgr2=0x7f82fc10bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.949+0000 7f8301aed640 1 --2- 172.21.15.70:0/2761897086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82fc105540 0x7f82fc10bdd0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f82f4009ca0 tx=0x7f82f402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.949+0000 7f8301aed640 1 -- 172.21.15.70:0/2761897086 shutdown_connections 2024-04-01T01:39:21.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.949+0000 7f8301aed640 1 --2- 172.21.15.70:0/2761897086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82fc105540 0x7f82fc10bdd0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.949+0000 7f8301aed640 1 --2- 172.21.15.70:0/2761897086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82fc104b80 0x7f82fc105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.949+0000 7f8301aed640 1 --2- 172.21.15.70:0/2761897086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82fc103990 0x7f82fc103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.949+0000 7f8301aed640 1 -- 172.21.15.70:0/2761897086 >> 172.21.15.70:0/2761897086 conn(0x7f82fc0fef00 msgr2=0x7f82fc101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:21.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.949+0000 7f8301aed640 1 -- 172.21.15.70:0/2761897086 shutdown_connections 2024-04-01T01:39:21.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.949+0000 7f8301aed640 1 -- 172.21.15.70:0/2761897086 wait complete. 2024-04-01T01:39:21.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.950+0000 7f8301aed640 1 Processor -- start 2024-04-01T01:39:21.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.950+0000 7f8301aed640 1 -- start start 2024-04-01T01:39:21.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.950+0000 7f8301aed640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82fc103990 0x7f82fc199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.950+0000 7f8301aed640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82fc104b80 0x7f82fc19a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.951+0000 7f8301aed640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82fc105540 0x7f82fc1a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.951+0000 7f8301aed640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc10f4c0 con 0x7f82fc104b80 2024-04-01T01:39:21.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.951+0000 7f82faffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82fc104b80 0x7f82fc19a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.951+0000 7f8301aed640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc10f340 con 0x7f82fc105540 2024-04-01T01:39:21.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.951+0000 7f8301aed640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc10f640 con 0x7f82fc103990 2024-04-01T01:39:21.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.951+0000 7f82faffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82fc104b80 0x7f82fc19a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40816/0 (socket says 172.21.15.70:40816) 2024-04-01T01:39:21.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.951+0000 7f82faffd640 1 -- 172.21.15.70:0/2630441662 learned_addr learned my addr 172.21.15.70:0/2630441662 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:21.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.951+0000 7f82fb7fe640 1 --2- 172.21.15.70:0/2630441662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82fc103990 0x7f82fc199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.951+0000 7f82faffd640 1 -- 172.21.15.70:0/2630441662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82fc103990 msgr2=0x7f82fc199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.951+0000 7f82faffd640 1 --2- 172.21.15.70:0/2630441662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82fc103990 0x7f82fc199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.951+0000 7f82faffd640 1 -- 172.21.15.70:0/2630441662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82fc105540 msgr2=0x7f82fc1a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.951+0000 7f82faffd640 1 --2- 172.21.15.70:0/2630441662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82fc105540 0x7f82fc1a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.951+0000 7f82faffd640 1 -- 172.21.15.70:0/2630441662 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82fc1a1780 con 0x7f82fc104b80 2024-04-01T01:39:21.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.951+0000 7f82faffd640 1 --2- 172.21.15.70:0/2630441662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82fc104b80 0x7f82fc19a0a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f82f000e9f0 tx=0x7f82f000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:21.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.951+0000 7f82f8ff9640 1 -- 172.21.15.70:0/2630441662 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f000cd80 con 0x7f82fc104b80 2024-04-01T01:39:21.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.952+0000 7f8301aed640 1 -- 172.21.15.70:0/2630441662 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82fc1a1a70 con 0x7f82fc104b80 2024-04-01T01:39:21.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.952+0000 7f82f8ff9640 1 -- 172.21.15.70:0/2630441662 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82f00054d0 con 0x7f82fc104b80 2024-04-01T01:39:21.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.952+0000 7f82f8ff9640 1 -- 172.21.15.70:0/2630441662 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f0010490 con 0x7f82fc104b80 2024-04-01T01:39:21.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.952+0000 7f8301aed640 1 -- 172.21.15.70:0/2630441662 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82fc1a1fc0 con 0x7f82fc104b80 2024-04-01T01:39:21.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.954+0000 7f82f8ff9640 1 -- 172.21.15.70:0/2630441662 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f82f0005020 con 0x7f82fc104b80 2024-04-01T01:39:21.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.954+0000 7f83047ad740 1 Processor -- start 2024-04-01T01:39:21.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.954+0000 7f83047ad740 1 -- start start 2024-04-01T01:39:21.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f83047ad740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc922b0 0x56168bc926d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f83047ad740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc92ca0 0x56168bc977a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f83047ad740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc97e10 0x56168bc9a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f83047ad740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56168bbf7180 con 0x56168bc922b0 2024-04-01T01:39:21.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f83047ad740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56168bb9fb70 con 0x56168bc97e10 2024-04-01T01:39:21.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f83047ad740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56168bbf3220 con 0x56168bc92ca0 2024-04-01T01:39:21.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f82fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc922b0 0x56168bc926d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f82fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc922b0 0x56168bc926d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40826/0 (socket says 172.21.15.70:40826) 2024-04-01T01:39:21.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f82fbfff640 1 -- 172.21.15.70:0/62725744 learned_addr learned my addr 172.21.15.70:0/62725744 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:21.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f82faffd640 1 --2- 172.21.15.70:0/62725744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc97e10 0x56168bc9a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f82fb7fe640 1 --2- 172.21.15.70:0/62725744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc92ca0 0x56168bc977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f82f8ff9640 1 --2- 172.21.15.70:0/2630441662 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f82e40a25a0 0x7f82e40a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f82faffd640 1 -- 172.21.15.70:0/62725744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc92ca0 msgr2=0x56168bc977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:21.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f82faffd640 1 --2- 172.21.15.70:0/62725744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc92ca0 0x56168bc977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f82faffd640 1 -- 172.21.15.70:0/62725744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc922b0 msgr2=0x56168bc926d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:21.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f82faffd640 1 --2- 172.21.15.70:0/62725744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc922b0 0x56168bc926d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f82faffd640 1 -- 172.21.15.70:0/62725744 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56168bbb37c0 con 0x56168bc97e10 2024-04-01T01:39:21.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f82f8ff9640 1 -- 172.21.15.70:0/2630441662 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f82f00ceef0 con 0x7f82fc104b80 2024-04-01T01:39:21.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.955+0000 7f82fbfff640 1 --2- 172.21.15.70:0/2630441662 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f82e40a25a0 0x7f82e40a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.956+0000 7f82fb7fe640 1 --2- 172.21.15.70:0/62725744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc92ca0 0x56168bc977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:21.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.956+0000 7f82fbfff640 1 --2- 172.21.15.70:0/2630441662 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f82e40a25a0 0x7f82e40a4a60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x56168bc93500 tx=0x7f82f402e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:21.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.956+0000 7f82faffd640 1 --2- 172.21.15.70:0/62725744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc97e10 0x56168bc9a2a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f82f00cc560 tx=0x7f82f0009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b14c9a95f342346a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:21.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.956+0000 7f82e1ffb640 1 -- 172.21.15.70:0/62725744 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f009b350 con 0x56168bc97e10 2024-04-01T01:39:21.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.956+0000 7f82e1ffb640 1 -- 172.21.15.70:0/62725744 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82f0005310 con 0x56168bc97e10 2024-04-01T01:39:21.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.956+0000 7f82e1ffb640 1 -- 172.21.15.70:0/62725744 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f009aee0 con 0x56168bc97e10 2024-04-01T01:39:21.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.957+0000 7f83047ad740 1 -- 172.21.15.70:0/62725744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc97e10 msgr2=0x56168bc9a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.957+0000 7f83047ad740 1 --2- 172.21.15.70:0/62725744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc97e10 0x56168bc9a2a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f82f00cc560 tx=0x7f82f0009310 comp rx=0 tx=0).stop 2024-04-01T01:39:21.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.957+0000 7f83047ad740 1 -- 172.21.15.70:0/62725744 shutdown_connections 2024-04-01T01:39:21.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.957+0000 7f83047ad740 1 --2- 172.21.15.70:0/62725744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc97e10 0x56168bc9a2a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.957+0000 7f83047ad740 1 --2- 172.21.15.70:0/62725744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc92ca0 0x56168bc977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.957+0000 7f83047ad740 1 --2- 172.21.15.70:0/62725744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc922b0 0x56168bc926d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.957+0000 7f83047ad740 1 -- 172.21.15.70:0/62725744 >> 172.21.15.70:0/62725744 conn(0x56168bc8d820 msgr2=0x56168bc8dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:21.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.957+0000 7f83047ad740 1 -- 172.21.15.70:0/62725744 shutdown_connections 2024-04-01T01:39:21.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.957+0000 7f83047ad740 1 -- 172.21.15.70:0/62725744 wait complete. 2024-04-01T01:39:21.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.957+0000 7f83047ad740 1 -- start start 2024-04-01T01:39:21.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f83047ad740 1 Processor -- start 2024-04-01T01:39:21.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f83047ad740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc922b0 0x56168bc9c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f83047ad740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc92ca0 0x56168bc9ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f83047ad740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc97e10 0x56168bca3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f83047ad740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56168bc9ad60 con 0x56168bc97e10 2024-04-01T01:39:21.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f83047ad740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56168bb954a0 con 0x56168bc922b0 2024-04-01T01:39:21.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f83047ad740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56168bc9aee0 con 0x56168bc92ca0 2024-04-01T01:39:21.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f83047ad740 10 client.0 _ll_register_callbacks cb 0x56168bb98d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:21.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f82faffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc92ca0 0x56168bc9ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f82fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc97e10 0x56168bca3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f82faffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc92ca0 0x56168bc9ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42986/0 (socket says 172.21.15.70:42986) 2024-04-01T01:39:21.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f82faffd640 1 -- 172.21.15.70:0/439365012 learned_addr learned my addr 172.21.15.70:0/439365012 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:21.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f82fb7fe640 1 --2- 172.21.15.70:0/439365012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc922b0 0x56168bc9c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f82fb7fe640 1 -- 172.21.15.70:0/439365012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc92ca0 msgr2=0x56168bc9ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:21.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f82fb7fe640 1 --2- 172.21.15.70:0/439365012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc92ca0 0x56168bc9ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f82fb7fe640 1 -- 172.21.15.70:0/439365012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc97e10 msgr2=0x56168bca3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:21.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f82fb7fe640 1 --2- 172.21.15.70:0/439365012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc97e10 0x56168bca3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.958+0000 7f82fb7fe640 1 -- 172.21.15.70:0/439365012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56168b9d57d0 con 0x56168bc922b0 2024-04-01T01:39:21.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.959+0000 7f82fb7fe640 1 --2- 172.21.15.70:0/439365012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc922b0 0x56168bc9c790 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x56168bc930a0 tx=0x7f82ec012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f226ee5feb51841 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:21.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.959+0000 7f82daffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:21.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.959+0000 7f82daffd640 1 -- 172.21.15.70:0/439365012 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec004030 con 0x56168bc922b0 2024-04-01T01:39:21.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.959+0000 7f82daffd640 1 -- 172.21.15.70:0/439365012 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f82ec0041d0 con 0x56168bc922b0 2024-04-01T01:39:21.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.959+0000 7f83047ad740 1 -- 172.21.15.70:0/439365012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56168bc046c0 con 0x56168bc922b0 2024-04-01T01:39:21.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.960+0000 7f82daffd640 1 -- 172.21.15.70:0/439365012 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f82ec025050 con 0x56168bc922b0 2024-04-01T01:39:21.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.960+0000 7f83047ad740 10 client.27266 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:21.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.960+0000 7f83047ad740 1 -- 172.21.15.70:0/439365012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56168b90b3f0 con 0x56168bc922b0 2024-04-01T01:39:21.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.960+0000 7f82daffd640 1 -- 172.21.15.70:0/439365012 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56168bc046c0 con 0x56168bc922b0 2024-04-01T01:39:21.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.960+0000 7f82daffd640 1 -- 172.21.15.70:0/439365012 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f82ec003990 con 0x56168bc922b0 2024-04-01T01:39:21.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.960+0000 7f83047ad740 10 client.27266 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:21.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.960+0000 7f83047ad740 10 client.27266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:21.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.960+0000 7f83047ad740 20 client.27266 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:21.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.960+0000 7f83047ad740 1 --2- 172.21.15.70:0/439365012 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56168bca8590 0x56168bcaaa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.960+0000 7f83047ad740 4 client.27266 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:21.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.960+0000 7f83047ad740 1 -- 172.21.15.70:0/439365012 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56168bbfea10 con 0x56168bca8590 2024-04-01T01:39:21.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.960+0000 7f82faffd640 1 --2- 172.21.15.70:0/439365012 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56168bca8590 0x56168bcaaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.961+0000 7f82faffd640 1 --2- 172.21.15.70:0/439365012 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56168bca8590 0x56168bcaaa00 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccd55022ddae75b7 server_cookie=4894c8369d77144d in_seq=0 out_seq=0 2024-04-01T01:39:21.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.961+0000 7f82daffd640 10 client.27266 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:21.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.962+0000 7f82daffd640 1 -- 172.21.15.70:0/439365012 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56168bbfea10 con 0x56168bca8590 2024-04-01T01:39:21.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.962+0000 7f82daffd640 10 client.27266 handle_command_reply: tid=0 2024-04-01T01:39:21.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.962+0000 7f83047ad740 1 client.27266 shutdown 2024-04-01T01:39:21.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.962+0000 7f83047ad740 1 -- 172.21.15.70:0/439365012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc922b0 msgr2=0x56168bc9c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.962+0000 7f83047ad740 1 --2- 172.21.15.70:0/439365012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc922b0 0x56168bc9c790 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x56168bc930a0 tx=0x7f82ec012b60 comp rx=0 tx=0).stop 2024-04-01T01:39:21.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.962+0000 7f83047ad740 1 -- 172.21.15.70:0/439365012 shutdown_connections 2024-04-01T01:39:21.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.962+0000 7f83047ad740 1 --2- 172.21.15.70:0/439365012 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56168bca8590 0x56168bcaaa00 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.962+0000 7f83047ad740 1 --2- 172.21.15.70:0/439365012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc97e10 0x56168bca3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.962+0000 7f83047ad740 1 --2- 172.21.15.70:0/439365012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc92ca0 0x56168bc9ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.962+0000 7f83047ad740 1 --2- 172.21.15.70:0/439365012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc922b0 0x56168bc9c790 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.962+0000 7f82daffd640 0 client.27266 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:21.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.962+0000 7f83047ad740 1 -- 172.21.15.70:0/439365012 >> 172.21.15.70:0/439365012 conn(0x56168bc8d820 msgr2=0x56168bc8cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:21.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.962+0000 7f83047ad740 1 -- 172.21.15.70:0/439365012 shutdown_connections 2024-04-01T01:39:21.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.962+0000 7f83047ad740 1 -- 172.21.15.70:0/439365012 wait complete. 2024-04-01T01:39:21.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.963+0000 7f83047ad740 20 client.27266 trim_cache size 0 max 16384 2024-04-01T01:39:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.982+0000 7f83047ad740 1 Processor -- start 2024-04-01T01:39:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.982+0000 7f83047ad740 1 -- start start 2024-04-01T01:39:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.983+0000 7f83047ad740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc922b0 0x56168bc9bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.983+0000 7f83047ad740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc92ca0 0x56168bca3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.983+0000 7f83047ad740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc97e10 0x56168bc9c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.983+0000 7f83047ad740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc1ac690 con 0x56168bc922b0 2024-04-01T01:39:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.983+0000 7f82fbfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc97e10 0x56168bc9c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.983+0000 7f82fb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc922b0 0x56168bc9bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.983+0000 7f82faffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc92ca0 0x56168bca3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.983+0000 7f83047ad740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc1ae1d0 con 0x56168bc92ca0 2024-04-01T01:39:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.983+0000 7f82fbfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc97e10 0x56168bc9c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42998/0 (socket says 172.21.15.70:42998) 2024-04-01T01:39:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.983+0000 7f82fbfff640 1 -- 172.21.15.70:0/711101003 learned_addr learned my addr 172.21.15.70:0/711101003 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.983+0000 7f83047ad740 1 -- 172.21.15.70:0/711101003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc1acd60 con 0x56168bc97e10 2024-04-01T01:39:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.983+0000 7f82fbfff640 1 -- 172.21.15.70:0/711101003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc92ca0 msgr2=0x56168bca3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.983+0000 7f82fbfff640 1 --2- 172.21.15.70:0/711101003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc92ca0 0x56168bca3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.983+0000 7f82fbfff640 1 -- 172.21.15.70:0/711101003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc922b0 msgr2=0x56168bc9bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:21.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.983+0000 7f82fbfff640 1 --2- 172.21.15.70:0/711101003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc922b0 0x56168bc9bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.983+0000 7f82fbfff640 1 -- 172.21.15.70:0/711101003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56168b8cb0f0 con 0x56168bc97e10 2024-04-01T01:39:21.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.984+0000 7f82fbfff640 1 --2- 172.21.15.70:0/711101003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc97e10 0x56168bc9c2f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f82f402c700 tx=0x7f82f403e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aef036517d3e28b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:21.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.984+0000 7f82e17fa640 1 -- 172.21.15.70:0/711101003 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f4039040 con 0x56168bc97e10 2024-04-01T01:39:21.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.984+0000 7f82e17fa640 1 -- 172.21.15.70:0/711101003 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82f4051070 con 0x56168bc97e10 2024-04-01T01:39:21.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.984+0000 7f82e17fa640 1 -- 172.21.15.70:0/711101003 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f40346a0 con 0x56168bc97e10 2024-04-01T01:39:21.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.985+0000 7f83047ad740 1 -- 172.21.15.70:0/711101003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc97e10 msgr2=0x56168bc9c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.985+0000 7f83047ad740 1 --2- 172.21.15.70:0/711101003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc97e10 0x56168bc9c2f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f82f402c700 tx=0x7f82f403e000 comp rx=0 tx=0).stop 2024-04-01T01:39:21.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.985+0000 7f83047ad740 1 -- 172.21.15.70:0/711101003 shutdown_connections 2024-04-01T01:39:21.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.985+0000 7f83047ad740 1 --2- 172.21.15.70:0/711101003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc97e10 0x56168bc9c2f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.985+0000 7f83047ad740 1 --2- 172.21.15.70:0/711101003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc92ca0 0x56168bca3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.985+0000 7f83047ad740 1 --2- 172.21.15.70:0/711101003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc922b0 0x56168bc9bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.985+0000 7f83047ad740 1 -- 172.21.15.70:0/711101003 >> 172.21.15.70:0/711101003 conn(0x56168bc8d820 msgr2=0x56168bc8cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:21.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.985+0000 7f83047ad740 1 -- 172.21.15.70:0/711101003 shutdown_connections 2024-04-01T01:39:21.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.985+0000 7f83047ad740 1 -- 172.21.15.70:0/711101003 wait complete. 2024-04-01T01:39:21.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.986+0000 7f83047ad740 1 -- start start 2024-04-01T01:39:21.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.986+0000 7f83047ad740 1 Processor -- start 2024-04-01T01:39:21.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.986+0000 7f83047ad740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc922b0 0x56168bca16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.986+0000 7f83047ad740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc92ca0 0x56168bca1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.986+0000 7f83047ad740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc97e10 0x56168bca2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.986+0000 7f83047ad740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc1adb00 con 0x56168bc97e10 2024-04-01T01:39:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.986+0000 7f83047ad740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc1ad430 con 0x56168bc92ca0 2024-04-01T01:39:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.986+0000 7f83047ad740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f82fc1aa670 con 0x56168bc922b0 2024-04-01T01:39:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.986+0000 7f83047ad740 10 client.0 _ll_register_callbacks cb 0x7f82ec0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.986+0000 7f82fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc97e10 0x56168bca2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.986+0000 7f82faffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc92ca0 0x56168bca1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.987+0000 7f82faffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc92ca0 0x56168bca1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54796/0 (socket says 172.21.15.70:54796) 2024-04-01T01:39:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.987+0000 7f82fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc97e10 0x56168bca2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40864/0 (socket says 172.21.15.70:40864) 2024-04-01T01:39:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.987+0000 7f82fbfff640 1 -- 172.21.15.70:0/1470120910 learned_addr learned my addr 172.21.15.70:0/1470120910 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.987+0000 7f82fb7fe640 1 --2- 172.21.15.70:0/1470120910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc922b0 0x56168bca16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.987+0000 7f82faffd640 1 -- 172.21.15.70:0/1470120910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc922b0 msgr2=0x56168bca16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.987+0000 7f82faffd640 1 --2- 172.21.15.70:0/1470120910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc922b0 0x56168bca16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.987+0000 7f82faffd640 1 -- 172.21.15.70:0/1470120910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc97e10 msgr2=0x56168bca2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.987+0000 7f82faffd640 1 --2- 172.21.15.70:0/1470120910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc97e10 0x56168bca2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.987+0000 7f82faffd640 1 -- 172.21.15.70:0/1470120910 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56168bca3230 con 0x56168bc92ca0 2024-04-01T01:39:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.987+0000 7f82fbfff640 1 --2- 172.21.15.70:0/1470120910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc97e10 0x56168bca2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.987+0000 7f82faffd640 1 --2- 172.21.15.70:0/1470120910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc92ca0 0x56168bca1c30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f82f00a3a60 tx=0x7f82f00d28b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6a7cce4ea5dd5de server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.987+0000 7f82daffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.987+0000 7f82daffd640 1 -- 172.21.15.70:0/1470120910 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f00b3070 con 0x56168bc92ca0 2024-04-01T01:39:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.987+0000 7f82daffd640 1 -- 172.21.15.70:0/1470120910 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f82f00b4260 con 0x56168bc92ca0 2024-04-01T01:39:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.987+0000 7f83047ad740 1 -- 172.21.15.70:0/1470120910 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82f4051070 con 0x56168bc92ca0 2024-04-01T01:39:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.988+0000 7f82daffd640 1 -- 172.21.15.70:0/1470120910 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f82f009b0c0 con 0x56168bc92ca0 2024-04-01T01:39:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.988+0000 7f83047ad740 10 client.27272 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.988+0000 7f83047ad740 1 -- 172.21.15.70:0/1470120910 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56168bd0ed00 con 0x56168bc92ca0 2024-04-01T01:39:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.988+0000 7f82daffd640 1 -- 172.21.15.70:0/1470120910 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82f4051070 con 0x56168bc92ca0 2024-04-01T01:39:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.988+0000 7f82daffd640 1 -- 172.21.15.70:0/1470120910 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f82f0097bf0 con 0x56168bc92ca0 2024-04-01T01:39:21.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.988+0000 7f83047ad740 10 client.27272 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.988+0000 7f83047ad740 10 client.27272 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.988+0000 7f83047ad740 20 client.27272 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.988+0000 7f83047ad740 1 --2- 172.21.15.70:0/1470120910 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56168bd0f230 0x56168bd11650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.988+0000 7f83047ad740 4 client.27272 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:39:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.989+0000 7f83047ad740 1 -- 172.21.15.70:0/1470120910 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56168bb7c540 con 0x56168bd0f230 2024-04-01T01:39:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.989+0000 7f82fb7fe640 1 --2- 172.21.15.70:0/1470120910 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56168bd0f230 0x56168bd11650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.989+0000 7f82daffd640 10 client.27272 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.989+0000 7f82fb7fe640 1 --2- 172.21.15.70:0/1470120910 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56168bd0f230 0x56168bd11650 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=745919cf39f86412 server_cookie=b229cb8ce340c1b3 in_seq=0 out_seq=0 2024-04-01T01:39:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.990+0000 7f82daffd640 1 -- 172.21.15.70:0/1470120910 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x56168bb7c540 con 0x56168bd0f230 2024-04-01T01:39:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.990+0000 7f82daffd640 10 client.27272 handle_command_reply: tid=0 2024-04-01T01:39:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.990+0000 7f83047ad740 1 client.27272 shutdown 2024-04-01T01:39:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.990+0000 7f83047ad740 1 -- 172.21.15.70:0/1470120910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc92ca0 msgr2=0x56168bca1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.990+0000 7f83047ad740 1 --2- 172.21.15.70:0/1470120910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc92ca0 0x56168bca1c30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f82f00a3a60 tx=0x7f82f00d28b0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.990+0000 7f83047ad740 1 -- 172.21.15.70:0/1470120910 shutdown_connections 2024-04-01T01:39:21.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.990+0000 7f83047ad740 1 --2- 172.21.15.70:0/1470120910 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56168bd0f230 0x56168bd11650 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.990+0000 7f83047ad740 1 --2- 172.21.15.70:0/1470120910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56168bc97e10 0x56168bca2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.990+0000 7f83047ad740 1 --2- 172.21.15.70:0/1470120910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56168bc92ca0 0x56168bca1c30 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.990+0000 7f83047ad740 1 --2- 172.21.15.70:0/1470120910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56168bc922b0 0x56168bca16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.990+0000 7f83047ad740 1 -- 172.21.15.70:0/1470120910 >> 172.21.15.70:0/1470120910 conn(0x56168bc8d820 msgr2=0x56168bc90a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:21.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.990+0000 7f82daffd640 0 client.27272 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:21.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.990+0000 7f83047ad740 1 -- 172.21.15.70:0/1470120910 shutdown_connections 2024-04-01T01:39:21.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.990+0000 7f83047ad740 1 -- 172.21.15.70:0/1470120910 wait complete. 2024-04-01T01:39:21.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.990+0000 7f83047ad740 20 client.27272 trim_cache size 0 max 16384 2024-04-01T01:39:21.994 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:39:21.995 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "scrub active (178 inodes in the stack)", 2024-04-01T01:39:21.995 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:39:21.995 INFO:teuthology.orchestra.run.smithi070.stdout: "938dc8a0-6cca-4e61-8f31-d954f1302836": { 2024-04-01T01:39:21.995 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:39:21.995 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "938dc8a0-6cca-4e61-8f31-d954f1302836", 2024-04-01T01:39:21.995 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:39:21.995 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:39:21.995 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:39:21.995 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:39:21.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.992+0000 7f8301aed640 1 -- 172.21.15.70:0/2630441662 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f82e40a25a0 msgr2=0x7f82e40a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.992+0000 7f8301aed640 1 --2- 172.21.15.70:0/2630441662 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f82e40a25a0 0x7f82e40a4a60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x56168bc93500 tx=0x7f82f402e040 comp rx=0 tx=0).stop 2024-04-01T01:39:21.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.992+0000 7f8301aed640 1 -- 172.21.15.70:0/2630441662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82fc104b80 msgr2=0x7f82fc19a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:21.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.992+0000 7f8301aed640 1 --2- 172.21.15.70:0/2630441662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82fc104b80 0x7f82fc19a0a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f82f000e9f0 tx=0x7f82f000edf0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.992+0000 7f8301aed640 1 -- 172.21.15.70:0/2630441662 shutdown_connections 2024-04-01T01:39:21.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.992+0000 7f8301aed640 1 --2- 172.21.15.70:0/2630441662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82fc105540 0x7f82fc1a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.993+0000 7f8301aed640 1 --2- 172.21.15.70:0/2630441662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82fc104b80 0x7f82fc19a0a0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.993+0000 7f8301aed640 1 --2- 172.21.15.70:0/2630441662 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f82e40a25a0 0x7f82e40a4a60 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.993+0000 7f8301aed640 1 --2- 172.21.15.70:0/2630441662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82fc103990 0x7f82fc199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:21.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.993+0000 7f8301aed640 1 -- 172.21.15.70:0/2630441662 >> 172.21.15.70:0/2630441662 conn(0x7f82fc0fef00 msgr2=0x7f82fc107f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:21.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.993+0000 7f8301aed640 1 -- 172.21.15.70:0/2630441662 shutdown_connections 2024-04-01T01:39:21.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:21.993+0000 7f8301aed640 1 -- 172.21.15.70:0/2630441662 wait complete. 2024-04-01T01:39:22.010 INFO:tasks.cephfs.filesystem:scrub status for tag:938dc8a0-6cca-4e61-8f31-d954f1302836 - {'path': '/', 'tag': '938dc8a0-6cca-4e61-8f31-d954f1302836', 'options': 'recursive,force'} 2024-04-01T01:39:22.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:22 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:39:22.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:22 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:39:22.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:22 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:39:22.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:22 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:39:22.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:22 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:39:22.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:22 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:39:22.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:22 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:39:22.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:22 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:39:22.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:22 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:39:22.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:22 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:39:22.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:22 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:39:22.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:22 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:39:23.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:23 smithi189 ceph-mon[35753]: pgmap v535: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 23 MiB/s wr, 1.01k op/s 2024-04-01T01:39:23.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:23 smithi070 ceph-mon[32179]: pgmap v535: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 23 MiB/s wr, 1.01k op/s 2024-04-01T01:39:23.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:23 smithi081 ceph-mon[35142]: pgmap v535: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 23 MiB/s wr, 1.01k op/s 2024-04-01T01:39:26.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:25 smithi081 ceph-mon[35142]: pgmap v536: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 18 MiB/s wr, 888 op/s 2024-04-01T01:39:26.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:25 smithi189 ceph-mon[35753]: pgmap v536: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 18 MiB/s wr, 888 op/s 2024-04-01T01:39:26.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:25 smithi070 ceph-mon[32179]: pgmap v536: 129 pgs: 129 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 18 MiB/s wr, 888 op/s 2024-04-01T01:39:26.897 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 280.0 s, 1106.7 tps, lat 103.519 ms stddev 157.696 2024-04-01T01:39:27.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:26 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:39:27.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:26 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:39:27.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:26 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:39:28.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:27 smithi081 ceph-mon[35142]: pgmap v537: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 21 MiB/s wr, 1.05k op/s 2024-04-01T01:39:28.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:27 smithi189 ceph-mon[35753]: pgmap v537: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 21 MiB/s wr, 1.05k op/s 2024-04-01T01:39:28.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:27 smithi070 ceph-mon[32179]: pgmap v537: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 21 MiB/s wr, 1.05k op/s 2024-04-01T01:39:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:28 smithi081 ceph-mon[35142]: pgmap v538: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 19 MiB/s wr, 1.02k op/s 2024-04-01T01:39:29.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:28 smithi189 ceph-mon[35753]: pgmap v538: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 19 MiB/s wr, 1.02k op/s 2024-04-01T01:39:29.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:28 smithi070 ceph-mon[32179]: pgmap v538: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 19 MiB/s wr, 1.02k op/s 2024-04-01T01:39:29.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.832+0000 7f83328c0640 1 -- 172.21.15.70:0/1483367641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f832c108750 msgr2=0x7f832c10ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:29.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.832+0000 7f83328c0640 1 --2- 172.21.15.70:0/1483367641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f832c108750 0x7f832c10ab80 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f831c01a910 tx=0x7f831c03cca0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.832+0000 7f83328c0640 1 -- 172.21.15.70:0/1483367641 shutdown_connections 2024-04-01T01:39:29.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.832+0000 7f83328c0640 1 --2- 172.21.15.70:0/1483367641 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f832c10da70 0x7f832c10def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.832+0000 7f83328c0640 1 --2- 172.21.15.70:0/1483367641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f832c10b0c0 0x7f832c10d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.832+0000 7f83328c0640 1 --2- 172.21.15.70:0/1483367641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f832c108750 0x7f832c10ab80 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.832+0000 7f83328c0640 1 -- 172.21.15.70:0/1483367641 >> 172.21.15.70:0/1483367641 conn(0x7f832c06e790 msgr2=0x7f832c070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:29.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.832+0000 7f83328c0640 1 -- 172.21.15.70:0/1483367641 shutdown_connections 2024-04-01T01:39:29.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.832+0000 7f83328c0640 1 -- 172.21.15.70:0/1483367641 wait complete. 2024-04-01T01:39:29.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.833+0000 7f83328c0640 1 Processor -- start 2024-04-01T01:39:29.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.833+0000 7f83328c0640 1 -- start start 2024-04-01T01:39:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.833+0000 7f83328c0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f832c108750 0x7f832c19e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.833+0000 7f83328c0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f832c10b0c0 0x7f832c19e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.833+0000 7f83328c0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f832c10da70 0x7f832c1a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.833+0000 7f83328c0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f832c1119d0 con 0x7f832c108750 2024-04-01T01:39:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.833+0000 7f83328c0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f832c111850 con 0x7f832c10da70 2024-04-01T01:39:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.833+0000 7f83328c0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f832c111b50 con 0x7f832c10b0c0 2024-04-01T01:39:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.834+0000 7f832bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f832c108750 0x7f832c19e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.834+0000 7f832bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f832c108750 0x7f832c19e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50026/0 (socket says 172.21.15.70:50026) 2024-04-01T01:39:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.834+0000 7f832bfff640 1 -- 172.21.15.70:0/1718589807 learned_addr learned my addr 172.21.15.70:0/1718589807 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.834+0000 7f832b7fe640 1 --2- 172.21.15.70:0/1718589807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f832c10b0c0 0x7f832c19e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.834+0000 7f8330e5d640 1 --2- 172.21.15.70:0/1718589807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f832c10da70 0x7f832c1a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.834+0000 7f832bfff640 1 -- 172.21.15.70:0/1718589807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f832c10b0c0 msgr2=0x7f832c19e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.834+0000 7f832bfff640 1 --2- 172.21.15.70:0/1718589807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f832c10b0c0 0x7f832c19e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.834+0000 7f832bfff640 1 -- 172.21.15.70:0/1718589807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f832c10da70 msgr2=0x7f832c1a55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.834+0000 7f832bfff640 1 --2- 172.21.15.70:0/1718589807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f832c10da70 0x7f832c1a55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.834+0000 7f832bfff640 1 -- 172.21.15.70:0/1718589807 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f832c1a5c50 con 0x7f832c108750 2024-04-01T01:39:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.834+0000 7f832bfff640 1 --2- 172.21.15.70:0/1718589807 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f832c108750 0x7f832c19e030 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f831c01a440 tx=0x7f831c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.835+0000 7f83297fa640 1 -- 172.21.15.70:0/1718589807 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c052020 con 0x7f832c108750 2024-04-01T01:39:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.835+0000 7f83297fa640 1 -- 172.21.15.70:0/1718589807 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f831c03dd00 con 0x7f832c108750 2024-04-01T01:39:29.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.835+0000 7f83297fa640 1 -- 172.21.15.70:0/1718589807 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c04e6e0 con 0x7f832c108750 2024-04-01T01:39:29.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.835+0000 7f83328c0640 1 -- 172.21.15.70:0/1718589807 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f832c1a5ee0 con 0x7f832c108750 2024-04-01T01:39:29.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.835+0000 7f83328c0640 1 -- 172.21.15.70:0/1718589807 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f832c1a6310 con 0x7f832c108750 2024-04-01T01:39:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.839+0000 7f83297fa640 1 -- 172.21.15.70:0/1718589807 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f831c055020 con 0x7f832c108750 2024-04-01T01:39:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.840+0000 7f83297fa640 1 --2- 172.21.15.70:0/1718589807 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f83140a2690 0x7f83140a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.841+0000 7f83297fa640 1 -- 172.21.15.70:0/1718589807 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f831c101b40 con 0x7f832c108750 2024-04-01T01:39:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.841+0000 7f832b7fe640 1 --2- 172.21.15.70:0/1718589807 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f83140a2690 0x7f83140a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.842+0000 7f832b7fe640 1 --2- 172.21.15.70:0/1718589807 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f83140a2690 0x7f83140a4b50 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f832c19f540 tx=0x7f83200090e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:29.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.843+0000 7f833557d740 1 Processor -- start 2024-04-01T01:39:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.845+0000 7f833557d740 1 -- start start 2024-04-01T01:39:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.845+0000 7f833557d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9a8d0 0x56496cd9acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.845+0000 7f833557d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cd9b2c0 0x56496cd9fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.845+0000 7f833557d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 0x56496cda28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.845+0000 7f833557d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56496ccfe8f0 con 0x56496cda0430 2024-04-01T01:39:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.845+0000 7f833557d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56496cca78d0 con 0x56496cd9b2c0 2024-04-01T01:39:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.845+0000 7f833557d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56496ccfa990 con 0x56496cd9a8d0 2024-04-01T01:39:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.846+0000 7f8330e5d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 0x56496cda28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.846+0000 7f8330e5d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 0x56496cda28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50032/0 (socket says 172.21.15.70:50032) 2024-04-01T01:39:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.846+0000 7f8330e5d640 1 -- 172.21.15.70:0/1310314989 learned_addr learned my addr 172.21.15.70:0/1310314989 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.847+0000 7f8330e5d640 1 -- 172.21.15.70:0/1310314989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9a8d0 msgr2=0x56496cd9acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.847+0000 7f8330e5d640 1 --2- 172.21.15.70:0/1310314989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9a8d0 0x56496cd9acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.847+0000 7f8330e5d640 1 -- 172.21.15.70:0/1310314989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cd9b2c0 msgr2=0x56496cd9fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.847+0000 7f8330e5d640 1 --2- 172.21.15.70:0/1310314989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cd9b2c0 0x56496cd9fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.847+0000 7f8330e5d640 1 -- 172.21.15.70:0/1310314989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56496ca13210 con 0x56496cda0430 2024-04-01T01:39:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.848+0000 7f8330e5d640 1 --2- 172.21.15.70:0/1310314989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 0x56496cda28c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f832400a4d0 tx=0x7f8324009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56678000f06ce284 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.848+0000 7f83127fc640 1 -- 172.21.15.70:0/1310314989 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8324018020 con 0x56496cda0430 2024-04-01T01:39:29.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.851+0000 7f83127fc640 1 -- 172.21.15.70:0/1310314989 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8324014cc0 con 0x56496cda0430 2024-04-01T01:39:29.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.851+0000 7f83127fc640 1 -- 172.21.15.70:0/1310314989 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8324013d70 con 0x56496cda0430 2024-04-01T01:39:29.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.852+0000 7f833557d740 1 -- 172.21.15.70:0/1310314989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 msgr2=0x56496cda28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:29.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.852+0000 7f833557d740 1 --2- 172.21.15.70:0/1310314989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 0x56496cda28c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f832400a4d0 tx=0x7f8324009650 comp rx=0 tx=0).stop 2024-04-01T01:39:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.855+0000 7f833557d740 1 -- 172.21.15.70:0/1310314989 shutdown_connections 2024-04-01T01:39:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.855+0000 7f833557d740 1 --2- 172.21.15.70:0/1310314989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 0x56496cda28c0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.855+0000 7f833557d740 1 --2- 172.21.15.70:0/1310314989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cd9b2c0 0x56496cd9fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.855+0000 7f833557d740 1 --2- 172.21.15.70:0/1310314989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9a8d0 0x56496cd9acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.855+0000 7f833557d740 1 -- 172.21.15.70:0/1310314989 >> 172.21.15.70:0/1310314989 conn(0x56496cd95e40 msgr2=0x56496cd96240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.856+0000 7f833557d740 1 -- 172.21.15.70:0/1310314989 shutdown_connections 2024-04-01T01:39:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.856+0000 7f833557d740 1 -- 172.21.15.70:0/1310314989 wait complete. 2024-04-01T01:39:29.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.856+0000 7f833557d740 1 -- start start 2024-04-01T01:39:29.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.857+0000 7f833557d740 1 Processor -- start 2024-04-01T01:39:29.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.857+0000 7f833557d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cd9a8d0 0x56496cda4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.857+0000 7f833557d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9b2c0 0x56496cda52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.857+0000 7f833557d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cda0430 0x56496cdac2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.857+0000 7f833557d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56496cda33e0 con 0x56496cd9a8d0 2024-04-01T01:39:29.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.857+0000 7f833557d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56496cc9cf80 con 0x56496cda0430 2024-04-01T01:39:29.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.857+0000 7f833557d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56496cda3560 con 0x56496cd9b2c0 2024-04-01T01:39:29.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.857+0000 7f833557d740 10 client.0 _ll_register_callbacks cb 0x56496ccf5290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:29.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.858+0000 7f8330e5d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cda0430 0x56496cdac2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:29.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.858+0000 7f8330e5d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cda0430 0x56496cdac2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49952/0 (socket says 172.21.15.70:49952) 2024-04-01T01:39:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.858+0000 7f8330e5d640 1 -- 172.21.15.70:0/1898962240 learned_addr learned my addr 172.21.15.70:0/1898962240 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.858+0000 7f832bfff640 1 --2- 172.21.15.70:0/1898962240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cd9a8d0 0x56496cda4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.858+0000 7f8330e5d640 1 -- 172.21.15.70:0/1898962240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9b2c0 msgr2=0x56496cda52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.858+0000 7f8330e5d640 1 --2- 172.21.15.70:0/1898962240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9b2c0 0x56496cda52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.858+0000 7f8330e5d640 1 -- 172.21.15.70:0/1898962240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cd9a8d0 msgr2=0x56496cda4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.858+0000 7f8330e5d640 1 --2- 172.21.15.70:0/1898962240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cd9a8d0 0x56496cda4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.858+0000 7f8330e5d640 1 -- 172.21.15.70:0/1898962240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56496cadf620 con 0x56496cda0430 2024-04-01T01:39:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.859+0000 7f8330e5d640 1 --2- 172.21.15.70:0/1898962240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cda0430 0x56496cdac2b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8324000f80 tx=0x7f8324013d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4adbd4dfa17b3a7d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.859+0000 7f830b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.859+0000 7f830b7fe640 1 -- 172.21.15.70:0/1898962240 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8324014a20 con 0x56496cda0430 2024-04-01T01:39:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.859+0000 7f833557d740 1 -- 172.21.15.70:0/1898962240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8324014cc0 con 0x56496cda0430 2024-04-01T01:39:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.859+0000 7f830b7fe640 1 -- 172.21.15.70:0/1898962240 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f832401d420 con 0x56496cda0430 2024-04-01T01:39:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.859+0000 7f830b7fe640 1 -- 172.21.15.70:0/1898962240 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f832401d620 con 0x56496cda0430 2024-04-01T01:39:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.860+0000 7f833557d740 10 client.27278 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.860+0000 7f833557d740 1 -- 172.21.15.70:0/1898962240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56496cdaf2b0 con 0x56496cda0430 2024-04-01T01:39:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.862+0000 7f830b7fe640 1 -- 172.21.15.70:0/1898962240 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8324014cc0 con 0x56496cda0430 2024-04-01T01:39:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.862+0000 7f830b7fe640 1 -- 172.21.15.70:0/1898962240 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f832401c7b0 con 0x56496cda0430 2024-04-01T01:39:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.863+0000 7f833557d740 10 client.27278 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.863+0000 7f833557d740 10 client.27278 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.863+0000 7f833557d740 20 client.27278 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.863+0000 7f833557d740 1 --2- 172.21.15.70:0/1898962240 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56496cdb07b0 0x56496cdb2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.863+0000 7f833557d740 4 client.27278 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.863+0000 7f833557d740 1 -- 172.21.15.70:0/1898962240 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56496cd12120 con 0x56496cdb07b0 2024-04-01T01:39:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.863+0000 7f832bfff640 1 --2- 172.21.15.70:0/1898962240 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56496cdb07b0 0x56496cdb2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.864+0000 7f832bfff640 1 --2- 172.21.15.70:0/1898962240 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56496cdb07b0 0x56496cdb2ba0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb69b88b66021bf7 server_cookie=578faed7d60586b in_seq=0 out_seq=0 2024-04-01T01:39:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.864+0000 7f830b7fe640 10 client.27278 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.865+0000 7f830b7fe640 1 -- 172.21.15.70:0/1898962240 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56496cd12120 con 0x56496cdb07b0 2024-04-01T01:39:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.865+0000 7f830b7fe640 10 client.27278 handle_command_reply: tid=0 2024-04-01T01:39:29.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.865+0000 7f833557d740 1 client.27278 shutdown 2024-04-01T01:39:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.866+0000 7f833557d740 1 -- 172.21.15.70:0/1898962240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cda0430 msgr2=0x56496cdac2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.866+0000 7f833557d740 1 --2- 172.21.15.70:0/1898962240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cda0430 0x56496cdac2b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8324000f80 tx=0x7f8324013d70 comp rx=0 tx=0).stop 2024-04-01T01:39:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.866+0000 7f833557d740 1 -- 172.21.15.70:0/1898962240 shutdown_connections 2024-04-01T01:39:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.866+0000 7f833557d740 1 --2- 172.21.15.70:0/1898962240 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56496cdb07b0 0x56496cdb2ba0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.866+0000 7f833557d740 1 --2- 172.21.15.70:0/1898962240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cda0430 0x56496cdac2b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.866+0000 7f833557d740 1 --2- 172.21.15.70:0/1898962240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9b2c0 0x56496cda52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.866+0000 7f833557d740 1 --2- 172.21.15.70:0/1898962240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cd9a8d0 0x56496cda4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.866+0000 7f833557d740 1 -- 172.21.15.70:0/1898962240 >> 172.21.15.70:0/1898962240 conn(0x56496cd95e40 msgr2=0x56496cd94f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.866+0000 7f830b7fe640 0 client.27278 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.867+0000 7f833557d740 1 -- 172.21.15.70:0/1898962240 shutdown_connections 2024-04-01T01:39:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.867+0000 7f833557d740 1 -- 172.21.15.70:0/1898962240 wait complete. 2024-04-01T01:39:29.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.867+0000 7f833557d740 20 client.27278 trim_cache size 0 max 16384 2024-04-01T01:39:29.895 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.895+0000 7f833557d740 1 Processor -- start 2024-04-01T01:39:29.895 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.895+0000 7f833557d740 1 -- start start 2024-04-01T01:39:29.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.895+0000 7f833557d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cd9a8d0 0x56496cd997b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.895+0000 7f833557d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9b2c0 0x56496cdac270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.895+0000 7f833557d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 0x56496cdb8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.895+0000 7f833557d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f832c1b0ac0 con 0x56496cda0430 2024-04-01T01:39:29.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.895+0000 7f833557d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f832c1b2600 con 0x56496cd9a8d0 2024-04-01T01:39:29.896 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.895+0000 7f833557d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f832c1b1190 con 0x56496cd9b2c0 2024-04-01T01:39:29.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.897+0000 7f832bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cd9a8d0 0x56496cd997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:29.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.897+0000 7f832bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cd9a8d0 0x56496cd997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49956/0 (socket says 172.21.15.70:49956) 2024-04-01T01:39:29.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.897+0000 7f832bfff640 1 -- 172.21.15.70:0/3511884794 learned_addr learned my addr 172.21.15.70:0/3511884794 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:29.898 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.897+0000 7f8330e5d640 1 --2- 172.21.15.70:0/3511884794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 0x56496cdb8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.897+0000 7f832bfff640 1 -- 172.21.15.70:0/3511884794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9b2c0 msgr2=0x56496cdac270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:39:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.897+0000 7f832bfff640 1 --2- 172.21.15.70:0/3511884794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9b2c0 0x56496cdac270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.897+0000 7f832bfff640 1 -- 172.21.15.70:0/3511884794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 msgr2=0x56496cdb8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.897+0000 7f832bfff640 1 --2- 172.21.15.70:0/3511884794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 0x56496cdb8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.897+0000 7f832bfff640 1 -- 172.21.15.70:0/3511884794 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56496cd9bb50 con 0x56496cd9a8d0 2024-04-01T01:39:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.898+0000 7f832bfff640 1 --2- 172.21.15.70:0/3511884794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cd9a8d0 0x56496cd997b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f831c006010 tx=0x7f831c005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fb042e49b2f49d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.898+0000 7f8311ffb640 1 -- 172.21.15.70:0/3511884794 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c0ce8d0 con 0x56496cd9a8d0 2024-04-01T01:39:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.898+0000 7f8311ffb640 1 -- 172.21.15.70:0/3511884794 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f831c0ce1d0 con 0x56496cd9a8d0 2024-04-01T01:39:29.899 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.898+0000 7f8311ffb640 1 -- 172.21.15.70:0/3511884794 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c0cc4d0 con 0x56496cd9a8d0 2024-04-01T01:39:29.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.899+0000 7f833557d740 1 -- 172.21.15.70:0/3511884794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cd9a8d0 msgr2=0x56496cd997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:29.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.899+0000 7f833557d740 1 --2- 172.21.15.70:0/3511884794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cd9a8d0 0x56496cd997b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f831c006010 tx=0x7f831c005c00 comp rx=0 tx=0).stop 2024-04-01T01:39:29.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.899+0000 7f833557d740 1 -- 172.21.15.70:0/3511884794 shutdown_connections 2024-04-01T01:39:29.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.899+0000 7f833557d740 1 --2- 172.21.15.70:0/3511884794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 0x56496cdb8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.899+0000 7f833557d740 1 --2- 172.21.15.70:0/3511884794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9b2c0 0x56496cdac270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.899+0000 7f833557d740 1 --2- 172.21.15.70:0/3511884794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cd9a8d0 0x56496cd997b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.899+0000 7f833557d740 1 -- 172.21.15.70:0/3511884794 >> 172.21.15.70:0/3511884794 conn(0x56496cd95e40 msgr2=0x56496cd990a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.899+0000 7f833557d740 1 -- 172.21.15.70:0/3511884794 shutdown_connections 2024-04-01T01:39:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.900+0000 7f833557d740 1 -- 172.21.15.70:0/3511884794 wait complete. 2024-04-01T01:39:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.900+0000 7f833557d740 1 -- start start 2024-04-01T01:39:29.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.900+0000 7f833557d740 1 Processor -- start 2024-04-01T01:39:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f833557d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cd9a8d0 0x56496cdaa010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f833557d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9b2c0 0x56496cdaa550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f833557d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 0x56496cdab560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f833557d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f832c1b1f30 con 0x56496cda0430 2024-04-01T01:39:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f833557d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f832c1b1860 con 0x56496cd9a8d0 2024-04-01T01:39:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f833557d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f832c1aeaa0 con 0x56496cd9b2c0 2024-04-01T01:39:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f833557d740 10 client.0 _ll_register_callbacks cb 0x7f8324009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f8330e5d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 0x56496cdab560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f8330e5d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 0x56496cdab560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50074/0 (socket says 172.21.15.70:50074) 2024-04-01T01:39:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f8330e5d640 1 -- 172.21.15.70:0/3755038295 learned_addr learned my addr 172.21.15.70:0/3755038295 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f832b7fe640 1 --2- 172.21.15.70:0/3755038295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9b2c0 0x56496cdaa550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f832bfff640 1 --2- 172.21.15.70:0/3755038295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cd9a8d0 0x56496cdaa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f8330e5d640 1 -- 172.21.15.70:0/3755038295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9b2c0 msgr2=0x56496cdaa550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f8330e5d640 1 --2- 172.21.15.70:0/3755038295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9b2c0 0x56496cdaa550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f8330e5d640 1 -- 172.21.15.70:0/3755038295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cd9a8d0 msgr2=0x56496cdaa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f8330e5d640 1 --2- 172.21.15.70:0/3755038295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cd9a8d0 0x56496cdaa010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f8330e5d640 1 -- 172.21.15.70:0/3755038295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56496cdabb50 con 0x56496cda0430 2024-04-01T01:39:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.901+0000 7f832b7fe640 1 --2- 172.21.15.70:0/3755038295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9b2c0 0x56496cdaa550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.902+0000 7f8330e5d640 1 --2- 172.21.15.70:0/3755038295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 0x56496cdab560 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f832400a010 tx=0x7f832400aea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62224b64ea7fe574 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.902+0000 7f830b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:39:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.902+0000 7f830b7fe640 1 -- 172.21.15.70:0/3755038295 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8324018050 con 0x56496cda0430 2024-04-01T01:39:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.902+0000 7f833557d740 1 -- 172.21.15.70:0/3755038295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f831c0ce1d0 con 0x56496cda0430 2024-04-01T01:39:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.902+0000 7f830b7fe640 1 -- 172.21.15.70:0/3755038295 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8324013670 con 0x56496cda0430 2024-04-01T01:39:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.903+0000 7f830b7fe640 1 -- 172.21.15.70:0/3755038295 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f83240138b0 con 0x56496cda0430 2024-04-01T01:39:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.903+0000 7f833557d740 10 client.17385 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.903+0000 7f833557d740 1 -- 172.21.15.70:0/3755038295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56496ce17120 con 0x56496cda0430 2024-04-01T01:39:29.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.903+0000 7f830b7fe640 1 -- 172.21.15.70:0/3755038295 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f831c0ce1d0 con 0x56496cda0430 2024-04-01T01:39:29.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.903+0000 7f830b7fe640 1 -- 172.21.15.70:0/3755038295 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f832401c4d0 con 0x56496cda0430 2024-04-01T01:39:29.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.903+0000 7f833557d740 10 client.17385 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:29.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.904+0000 7f833557d740 10 client.17385 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:29.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.904+0000 7f833557d740 20 client.17385 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:29.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.904+0000 7f833557d740 1 --2- 172.21.15.70:0/3755038295 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56496ce177b0 0x56496ce19bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:29.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.904+0000 7f833557d740 4 client.17385 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:39:29.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.904+0000 7f833557d740 1 -- 172.21.15.70:0/3755038295 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56496cd0be30 con 0x56496ce177b0 2024-04-01T01:39:29.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.904+0000 7f832bfff640 1 --2- 172.21.15.70:0/3755038295 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56496ce177b0 0x56496ce19bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:29.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.905+0000 7f832bfff640 1 --2- 172.21.15.70:0/3755038295 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56496ce177b0 0x56496ce19bd0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7510d22f7297e58 server_cookie=c547f44cd8d31e76 in_seq=0 out_seq=0 2024-04-01T01:39:29.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:29.905+0000 7f830b7fe640 10 client.17385 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:30.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.028+0000 7f830b7fe640 1 -- 172.21.15.70:0/3755038295 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56496cd0be30 con 0x56496ce177b0 2024-04-01T01:39:30.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.028+0000 7f830b7fe640 10 client.17385 handle_command_reply: tid=0 2024-04-01T01:39:30.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.028+0000 7f833557d740 1 client.17385 shutdown 2024-04-01T01:39:30.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.028+0000 7f833557d740 1 -- 172.21.15.70:0/3755038295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 msgr2=0x56496cdab560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:30.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.028+0000 7f833557d740 1 --2- 172.21.15.70:0/3755038295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 0x56496cdab560 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f832400a010 tx=0x7f832400aea0 comp rx=0 tx=0).stop 2024-04-01T01:39:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.029+0000 7f833557d740 1 -- 172.21.15.70:0/3755038295 shutdown_connections 2024-04-01T01:39:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.029+0000 7f833557d740 1 --2- 172.21.15.70:0/3755038295 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56496ce177b0 0x56496ce19bd0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.029+0000 7f833557d740 1 --2- 172.21.15.70:0/3755038295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56496cda0430 0x56496cdab560 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.029+0000 7f833557d740 1 --2- 172.21.15.70:0/3755038295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56496cd9b2c0 0x56496cdaa550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.029+0000 7f833557d740 1 --2- 172.21.15.70:0/3755038295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56496cd9a8d0 0x56496cdaa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.029+0000 7f833557d740 1 -- 172.21.15.70:0/3755038295 >> 172.21.15.70:0/3755038295 conn(0x56496cd95e40 msgr2=0x56496cd946f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.029+0000 7f830b7fe640 0 client.17385 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:30.031 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.029+0000 7f833557d740 1 -- 172.21.15.70:0/3755038295 shutdown_connections 2024-04-01T01:39:30.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.029+0000 7f833557d740 1 -- 172.21.15.70:0/3755038295 wait complete. 2024-04-01T01:39:30.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.030+0000 7f833557d740 20 client.17385 trim_cache size 0 max 16384 2024-04-01T01:39:30.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:39:30.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:39:30.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:39:30.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:39:30.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.033+0000 7f83328c0640 1 -- 172.21.15.70:0/1718589807 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f83140a2690 msgr2=0x7f83140a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:30.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.033+0000 7f83328c0640 1 --2- 172.21.15.70:0/1718589807 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f83140a2690 0x7f83140a4b50 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f832c19f540 tx=0x7f83200090e0 comp rx=0 tx=0).stop 2024-04-01T01:39:30.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.033+0000 7f83328c0640 1 -- 172.21.15.70:0/1718589807 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f832c108750 msgr2=0x7f832c19e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:30.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.033+0000 7f83328c0640 1 --2- 172.21.15.70:0/1718589807 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f832c108750 0x7f832c19e030 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f831c01a440 tx=0x7f831c004270 comp rx=0 tx=0).stop 2024-04-01T01:39:30.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.033+0000 7f83328c0640 1 -- 172.21.15.70:0/1718589807 shutdown_connections 2024-04-01T01:39:30.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.033+0000 7f83328c0640 1 --2- 172.21.15.70:0/1718589807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f832c10da70 0x7f832c1a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.033+0000 7f83328c0640 1 --2- 172.21.15.70:0/1718589807 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f83140a2690 0x7f83140a4b50 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.033+0000 7f83328c0640 1 --2- 172.21.15.70:0/1718589807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f832c10b0c0 0x7f832c19e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.033+0000 7f83328c0640 1 --2- 172.21.15.70:0/1718589807 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f832c108750 0x7f832c19e030 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.033+0000 7f83328c0640 1 -- 172.21.15.70:0/1718589807 >> 172.21.15.70:0/1718589807 conn(0x7f832c06e790 msgr2=0x7f832c108ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.034+0000 7f83328c0640 1 -- 172.21.15.70:0/1718589807 shutdown_connections 2024-04-01T01:39:30.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:30.034+0000 7f83328c0640 1 -- 172.21.15.70:0/1718589807 wait complete. 2024-04-01T01:39:31.898 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 285.0 s, 1008.0 tps, lat 100.068 ms stddev 160.143 2024-04-01T01:39:32.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:31 smithi081 ceph-mon[35142]: pgmap v539: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 19 MiB/s wr, 969 op/s 2024-04-01T01:39:32.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:31 smithi189 ceph-mon[35753]: pgmap v539: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 19 MiB/s wr, 969 op/s 2024-04-01T01:39:32.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:31 smithi070 ceph-mon[32179]: pgmap v539: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 19 MiB/s wr, 969 op/s 2024-04-01T01:39:34.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:33 smithi189 ceph-mon[35753]: pgmap v540: 129 pgs: 129 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 18 MiB/s wr, 1.06k op/s 2024-04-01T01:39:34.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:33 smithi081 ceph-mon[35142]: pgmap v540: 129 pgs: 129 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 18 MiB/s wr, 1.06k op/s 2024-04-01T01:39:34.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:33 smithi070 ceph-mon[32179]: pgmap v540: 129 pgs: 129 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 18 MiB/s wr, 1.06k op/s 2024-04-01T01:39:35.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:34 smithi189 ceph-mon[35753]: pgmap v541: 129 pgs: 129 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 16 MiB/s wr, 958 op/s 2024-04-01T01:39:35.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:34 smithi070 ceph-mon[32179]: pgmap v541: 129 pgs: 129 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 16 MiB/s wr, 958 op/s 2024-04-01T01:39:35.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:34 smithi081 ceph-mon[35142]: pgmap v541: 129 pgs: 129 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 16 MiB/s wr, 958 op/s 2024-04-01T01:39:36.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 290.0 s, 904.8 tps, lat 96.478 ms stddev 114.344 2024-04-01T01:39:38.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:37 smithi081 ceph-mon[35142]: pgmap v542: 129 pgs: 129 active+clean; 1.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 19 MiB/s wr, 1.08k op/s 2024-04-01T01:39:38.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:37 smithi189 ceph-mon[35753]: pgmap v542: 129 pgs: 129 active+clean; 1.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 19 MiB/s wr, 1.08k op/s 2024-04-01T01:39:38.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:37 smithi070 ceph-mon[32179]: pgmap v542: 129 pgs: 129 active+clean; 1.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 19 MiB/s wr, 1.08k op/s 2024-04-01T01:39:39.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:38 smithi189 ceph-mon[35753]: pgmap v543: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 19 MiB/s wr, 1.01k op/s 2024-04-01T01:39:39.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:38 smithi070 ceph-mon[32179]: pgmap v543: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 19 MiB/s wr, 1.01k op/s 2024-04-01T01:39:39.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:38 smithi081 ceph-mon[35142]: pgmap v543: 129 pgs: 129 active+clean; 1.3 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 19 MiB/s wr, 1.01k op/s 2024-04-01T01:39:41.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 295.0 s, 1046.7 tps, lat 107.731 ms stddev 155.124 2024-04-01T01:39:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:41 smithi081 ceph-mon[35142]: pgmap v544: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 17 MiB/s wr, 938 op/s 2024-04-01T01:39:42.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:41 smithi189 ceph-mon[35753]: pgmap v544: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 17 MiB/s wr, 938 op/s 2024-04-01T01:39:42.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:41 smithi070 ceph-mon[32179]: pgmap v544: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 17 MiB/s wr, 938 op/s 2024-04-01T01:39:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.215+0000 7f96ac5bc640 1 -- 172.21.15.70:0/186648196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96a410da70 msgr2=0x7f96a410def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.215+0000 7f96ac5bc640 1 --2- 172.21.15.70:0/186648196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96a410da70 0x7f96a410def0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9694009900 tx=0x7f969402bc80 comp rx=0 tx=0).stop 2024-04-01T01:39:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.215+0000 7f96ac5bc640 1 -- 172.21.15.70:0/186648196 shutdown_connections 2024-04-01T01:39:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.215+0000 7f96ac5bc640 1 --2- 172.21.15.70:0/186648196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96a410da70 0x7f96a410def0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.215+0000 7f96ac5bc640 1 --2- 172.21.15.70:0/186648196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96a410b0c0 0x7f96a410d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.215+0000 7f96ac5bc640 1 --2- 172.21.15.70:0/186648196 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96a4108750 0x7f96a410ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.216+0000 7f96ac5bc640 1 -- 172.21.15.70:0/186648196 >> 172.21.15.70:0/186648196 conn(0x7f96a406e790 msgr2=0x7f96a4070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:43.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.223+0000 7f96ac5bc640 1 -- 172.21.15.70:0/186648196 shutdown_connections 2024-04-01T01:39:43.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.224+0000 7f96ac5bc640 1 -- 172.21.15.70:0/186648196 wait complete. 2024-04-01T01:39:43.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.224+0000 7f96ac5bc640 1 Processor -- start 2024-04-01T01:39:43.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.224+0000 7f96ac5bc640 1 -- start start 2024-04-01T01:39:43.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.225+0000 7f96ac5bc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96a4108750 0x7f96a4114e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.225+0000 7f96ac5bc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96a410b0c0 0x7f96a4115340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.225+0000 7f96ac5bc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96a410da70 0x7f96a41a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.225+0000 7f96ac5bc640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f96a411c1f0 con 0x7f96a410da70 2024-04-01T01:39:43.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.225+0000 7f96ac5bc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f96a411c070 con 0x7f96a4108750 2024-04-01T01:39:43.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.225+0000 7f96ac5bc640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f96a411c370 con 0x7f96a410b0c0 2024-04-01T01:39:43.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.225+0000 7f96aa358640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96a4108750 0x7f96a4114e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.225+0000 7f96a9b57640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96a410b0c0 0x7f96a4115340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.225+0000 7f96aa358640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96a4108750 0x7f96a4114e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49334/0 (socket says 172.21.15.70:49334) 2024-04-01T01:39:43.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.225+0000 7f96aa358640 1 -- 172.21.15.70:0/3644767663 learned_addr learned my addr 172.21.15.70:0/3644767663 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.227+0000 7f96aab59640 1 --2- 172.21.15.70:0/3644767663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96a410da70 0x7f96a41a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.227+0000 7f96a9b57640 1 -- 172.21.15.70:0/3644767663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96a4108750 msgr2=0x7f96a4114e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.227+0000 7f96a9b57640 1 --2- 172.21.15.70:0/3644767663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96a4108750 0x7f96a4114e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.227+0000 7f96a9b57640 1 -- 172.21.15.70:0/3644767663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96a410da70 msgr2=0x7f96a41a58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.227+0000 7f96a9b57640 1 --2- 172.21.15.70:0/3644767663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96a410da70 0x7f96a41a58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.227+0000 7f96a9b57640 1 -- 172.21.15.70:0/3644767663 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a41159e0 con 0x7f96a410b0c0 2024-04-01T01:39:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.228+0000 7f96a9b57640 1 --2- 172.21.15.70:0/3644767663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96a410b0c0 0x7f96a4115340 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f969c009fd0 tx=0x7f969c00b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.228+0000 7f969b7fe640 1 -- 172.21.15.70:0/3644767663 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c015020 con 0x7f96a410b0c0 2024-04-01T01:39:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.228+0000 7f96ac5bc640 1 -- 172.21.15.70:0/3644767663 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96a41a5f50 con 0x7f96a410b0c0 2024-04-01T01:39:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.229+0000 7f96ac5bc640 1 -- 172.21.15.70:0/3644767663 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96a41a6520 con 0x7f96a410b0c0 2024-04-01T01:39:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.229+0000 7f969b7fe640 1 -- 172.21.15.70:0/3644767663 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f969c011ce0 con 0x7f96a410b0c0 2024-04-01T01:39:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.229+0000 7f969b7fe640 1 -- 172.21.15.70:0/3644767663 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c00f2e0 con 0x7f96a410b0c0 2024-04-01T01:39:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.233+0000 7f96af1a0740 1 Processor -- start 2024-04-01T01:39:43.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.234+0000 7f96af1a0740 1 -- start start 2024-04-01T01:39:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.234+0000 7f96af1a0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a075e8d0 0x5633a075ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.234+0000 7f96af1a0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 0x5633a0763dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.234+0000 7f96af1a0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a0764430 0x5633a07668c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.234+0000 7f96af1a0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5633a06c28f0 con 0x5633a075e8d0 2024-04-01T01:39:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.234+0000 7f96af1a0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5633a066b8d0 con 0x5633a0764430 2024-04-01T01:39:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.234+0000 7f96af1a0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5633a06be990 con 0x5633a075f2c0 2024-04-01T01:39:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.235+0000 7f969b7fe640 1 -- 172.21.15.70:0/3644767663 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f969c018020 con 0x7f96a410b0c0 2024-04-01T01:39:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.236+0000 7f96aa358640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 0x5633a0763dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.236+0000 7f96aa358640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 0x5633a0763dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58300/0 (socket says 172.21.15.70:58300) 2024-04-01T01:39:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.236+0000 7f96aa358640 1 -- 172.21.15.70:0/645983362 learned_addr learned my addr 172.21.15.70:0/645983362 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.236+0000 7f96aab59640 1 --2- 172.21.15.70:0/645983362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a075e8d0 0x5633a075ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.241+0000 7f969b7fe640 1 --2- 172.21.15.70:0/3644767663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96900a26b0 0x7f96900a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.242+0000 7f96aab59640 1 --2- 172.21.15.70:0/3644767663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96900a26b0 0x7f96900a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.242+0000 7f969b7fe640 1 -- 172.21.15.70:0/3644767663 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f969c0ce260 con 0x7f96a410b0c0 2024-04-01T01:39:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.242+0000 7f96a9b57640 1 --2- 172.21.15.70:0/645983362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a0764430 0x5633a07668c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.242+0000 7f96aab59640 1 -- 172.21.15.70:0/645983362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 msgr2=0x5633a0763dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.242+0000 7f96aab59640 1 --2- 172.21.15.70:0/645983362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 0x5633a0763dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.242+0000 7f96aab59640 1 -- 172.21.15.70:0/645983362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a0764430 msgr2=0x5633a07668c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.242+0000 7f96aab59640 1 --2- 172.21.15.70:0/645983362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a0764430 0x5633a07668c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.242+0000 7f96aab59640 1 -- 172.21.15.70:0/645983362 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633a03d7210 con 0x5633a075e8d0 2024-04-01T01:39:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.243+0000 7f96aab59640 1 --2- 172.21.15.70:0/645983362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a075e8d0 0x5633a075ecf0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f969402c1c0 tx=0x7f96940040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce57badf09ff859b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.243+0000 7f96aab59640 1 --2- 172.21.15.70:0/3644767663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96900a26b0 0x7f96900a4b70 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9694046000 tx=0x7f9694049000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.244+0000 7f9698bfe640 1 -- 172.21.15.70:0/645983362 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9694005610 con 0x5633a075e8d0 2024-04-01T01:39:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.244+0000 7f9698bfe640 1 -- 172.21.15.70:0/645983362 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96940044d0 con 0x5633a075e8d0 2024-04-01T01:39:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.245+0000 7f96af1a0740 1 -- 172.21.15.70:0/645983362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a075e8d0 msgr2=0x5633a075ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.245+0000 7f96af1a0740 1 --2- 172.21.15.70:0/645983362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a075e8d0 0x5633a075ecf0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f969402c1c0 tx=0x7f96940040c0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.245+0000 7f96af1a0740 1 -- 172.21.15.70:0/645983362 shutdown_connections 2024-04-01T01:39:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.245+0000 7f96af1a0740 1 --2- 172.21.15.70:0/645983362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a0764430 0x5633a07668c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.245+0000 7f96af1a0740 1 --2- 172.21.15.70:0/645983362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 0x5633a0763dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.245+0000 7f96af1a0740 1 --2- 172.21.15.70:0/645983362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a075e8d0 0x5633a075ecf0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.245+0000 7f96af1a0740 1 -- 172.21.15.70:0/645983362 >> 172.21.15.70:0/645983362 conn(0x5633a0759e40 msgr2=0x5633a075a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.245+0000 7f96af1a0740 1 -- 172.21.15.70:0/645983362 shutdown_connections 2024-04-01T01:39:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.250+0000 7f96af1a0740 1 -- 172.21.15.70:0/645983362 wait complete. 2024-04-01T01:39:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.250+0000 7f96af1a0740 1 -- start start 2024-04-01T01:39:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.250+0000 7f96af1a0740 1 Processor -- start 2024-04-01T01:39:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.250+0000 7f96af1a0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a075e8d0 0x5633a0768c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.250+0000 7f96af1a0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 0x5633a07691b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.250+0000 7f96af1a0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a0764430 0x5633a07701b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.250+0000 7f96af1a0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5633a07673d0 con 0x5633a0764430 2024-04-01T01:39:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.250+0000 7f96af1a0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5633a0660f80 con 0x5633a075e8d0 2024-04-01T01:39:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.250+0000 7f96af1a0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5633a0767550 con 0x5633a075f2c0 2024-04-01T01:39:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.251+0000 7f96af1a0740 10 client.0 _ll_register_callbacks cb 0x5633a06b9290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.251+0000 7f96aa358640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a075e8d0 0x5633a0768c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.251+0000 7f96aa358640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a075e8d0 0x5633a0768c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49354/0 (socket says 172.21.15.70:49354) 2024-04-01T01:39:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.251+0000 7f96aa358640 1 -- 172.21.15.70:0/187697653 learned_addr learned my addr 172.21.15.70:0/187697653 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.251+0000 7f96a9b57640 1 --2- 172.21.15.70:0/187697653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 0x5633a07691b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.251+0000 7f96aab59640 1 --2- 172.21.15.70:0/187697653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a0764430 0x5633a07701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.251+0000 7f96aa358640 1 -- 172.21.15.70:0/187697653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 msgr2=0x5633a07691b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.251+0000 7f96aa358640 1 --2- 172.21.15.70:0/187697653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 0x5633a07691b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.251+0000 7f96aa358640 1 -- 172.21.15.70:0/187697653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a0764430 msgr2=0x5633a07701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.251+0000 7f96aa358640 1 --2- 172.21.15.70:0/187697653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a0764430 0x5633a07701b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.251+0000 7f96aa358640 1 -- 172.21.15.70:0/187697653 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633a04a3620 con 0x5633a075e8d0 2024-04-01T01:39:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.252+0000 7f96aa358640 1 --2- 172.21.15.70:0/187697653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a075e8d0 0x5633a0768c70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f96a0011ca0 tx=0x7f96a000f580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76dc30d608a56206 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.252+0000 7f968dffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.253+0000 7f968dffb640 1 -- 172.21.15.70:0/187697653 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a000fe10 con 0x5633a075e8d0 2024-04-01T01:39:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.253+0000 7f96af1a0740 1 -- 172.21.15.70:0/187697653 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96940044d0 con 0x5633a075e8d0 2024-04-01T01:39:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.253+0000 7f968dffb640 1 -- 172.21.15.70:0/187697653 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f96a0015430 con 0x5633a075e8d0 2024-04-01T01:39:43.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.253+0000 7f968dffb640 1 -- 172.21.15.70:0/187697653 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f96a0026050 con 0x5633a075e8d0 2024-04-01T01:39:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.253+0000 7f96af1a0740 10 client.27290 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.253+0000 7f96af1a0740 1 -- 172.21.15.70:0/187697653 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5633a07732b0 con 0x5633a075e8d0 2024-04-01T01:39:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.254+0000 7f968dffb640 1 -- 172.21.15.70:0/187697653 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96940044d0 con 0x5633a075e8d0 2024-04-01T01:39:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.254+0000 7f968dffb640 1 -- 172.21.15.70:0/187697653 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f96a001d7a0 con 0x5633a075e8d0 2024-04-01T01:39:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.254+0000 7f96af1a0740 10 client.27290 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.254+0000 7f96af1a0740 10 client.27290 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.254+0000 7f96af1a0740 20 client.27290 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.254+0000 7f96af1a0740 1 --2- 172.21.15.70:0/187697653 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5633a0774e20 0x5633a0777280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.254+0000 7f96af1a0740 4 client.27290 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:43.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.254+0000 7f96af1a0740 1 -- 172.21.15.70:0/187697653 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633a06d6120 con 0x5633a0774e20 2024-04-01T01:39:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.257+0000 7f96a9b57640 1 --2- 172.21.15.70:0/187697653 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5633a0774e20 0x5633a0777280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.258+0000 7f968dffb640 10 client.27290 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.258+0000 7f96a9b57640 1 --2- 172.21.15.70:0/187697653 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5633a0774e20 0x5633a0777280 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17bcae1f69a4cd9b server_cookie=30e4ae0ea72c7124 in_seq=0 out_seq=0 2024-04-01T01:39:43.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.259+0000 7f968dffb640 1 -- 172.21.15.70:0/187697653 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5633a06d6120 con 0x5633a0774e20 2024-04-01T01:39:43.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.260+0000 7f968dffb640 10 client.27290 handle_command_reply: tid=0 2024-04-01T01:39:43.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.260+0000 7f96af1a0740 1 client.27290 shutdown 2024-04-01T01:39:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.265+0000 7f96af1a0740 1 -- 172.21.15.70:0/187697653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a075e8d0 msgr2=0x5633a0768c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:43.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.265+0000 7f96af1a0740 1 --2- 172.21.15.70:0/187697653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a075e8d0 0x5633a0768c70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f96a0011ca0 tx=0x7f96a000f580 comp rx=0 tx=0).stop 2024-04-01T01:39:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.266+0000 7f96af1a0740 1 -- 172.21.15.70:0/187697653 shutdown_connections 2024-04-01T01:39:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.266+0000 7f96af1a0740 1 --2- 172.21.15.70:0/187697653 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5633a0774e20 0x5633a0777280 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.266+0000 7f96af1a0740 1 --2- 172.21.15.70:0/187697653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a0764430 0x5633a07701b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.266+0000 7f96af1a0740 1 --2- 172.21.15.70:0/187697653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 0x5633a07691b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.266+0000 7f96af1a0740 1 --2- 172.21.15.70:0/187697653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a075e8d0 0x5633a0768c70 secure :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f96a0011ca0 tx=0x7f96a000f580 comp rx=0 tx=0).stop 2024-04-01T01:39:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.266+0000 7f96af1a0740 1 -- 172.21.15.70:0/187697653 >> 172.21.15.70:0/187697653 conn(0x5633a0759e40 msgr2=0x5633a0758f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.266+0000 7f968dffb640 0 client.27290 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:43.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.268+0000 7f96af1a0740 1 -- 172.21.15.70:0/187697653 shutdown_connections 2024-04-01T01:39:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.269+0000 7f96af1a0740 1 -- 172.21.15.70:0/187697653 wait complete. 2024-04-01T01:39:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.269+0000 7f96af1a0740 20 client.27290 trim_cache size 0 max 16384 2024-04-01T01:39:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.301+0000 7f96af1a0740 1 Processor -- start 2024-04-01T01:39:43.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.301+0000 7f96af1a0740 1 -- start start 2024-04-01T01:39:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.301+0000 7f96af1a0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 0x5633a0768a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.301+0000 7f96af1a0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a0764430 0x5633a0770170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.302+0000 7f96aa358640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 0x5633a0768a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.302+0000 7f96aa358640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 0x5633a0768a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58320/0 (socket says 172.21.15.70:58320) 2024-04-01T01:39:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.302+0000 7f96aa358640 1 -- 172.21.15.70:0/845511068 learned_addr learned my addr 172.21.15.70:0/845511068 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.302+0000 7f96af1a0740 1 --2- 172.21.15.70:0/845511068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a07684d0 0x5633a0768f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.302+0000 7f96af1a0740 1 -- 172.21.15.70:0/845511068 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f968800fe30 con 0x5633a0764430 2024-04-01T01:39:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.302+0000 7f96af1a0740 1 -- 172.21.15.70:0/845511068 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9688011970 con 0x5633a07684d0 2024-04-01T01:39:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.302+0000 7f96af1a0740 1 -- 172.21.15.70:0/845511068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9688010500 con 0x5633a075f2c0 2024-04-01T01:39:43.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.302+0000 7f96aab59640 1 --2- 172.21.15.70:0/845511068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a07684d0 0x5633a0768f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.303+0000 7f96a9b57640 1 --2- 172.21.15.70:0/845511068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a0764430 0x5633a0770170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.303+0000 7f96aa358640 1 -- 172.21.15.70:0/845511068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a07684d0 msgr2=0x5633a0768f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.303+0000 7f96aa358640 1 --2- 172.21.15.70:0/845511068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a07684d0 0x5633a0768f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.303+0000 7f96aa358640 1 -- 172.21.15.70:0/845511068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a0764430 msgr2=0x5633a0770170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.303+0000 7f96aa358640 1 --2- 172.21.15.70:0/845511068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a0764430 0x5633a0770170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.303+0000 7f96aa358640 1 -- 172.21.15.70:0/845511068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633a075ecb0 con 0x5633a075f2c0 2024-04-01T01:39:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.303+0000 7f96a9b57640 1 --2- 172.21.15.70:0/845511068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a0764430 0x5633a0770170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.303+0000 7f96aa358640 1 --2- 172.21.15.70:0/845511068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 0x5633a0768a00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f96a00022f0 tx=0x7f96a0011890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce49169f07b74b0c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.304+0000 7f968ffff640 1 -- 172.21.15.70:0/845511068 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a0030020 con 0x5633a075f2c0 2024-04-01T01:39:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.304+0000 7f968ffff640 1 -- 172.21.15.70:0/845511068 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96a0009880 con 0x5633a075f2c0 2024-04-01T01:39:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.304+0000 7f968ffff640 1 -- 172.21.15.70:0/845511068 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a000fea0 con 0x5633a075f2c0 2024-04-01T01:39:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.305+0000 7f96af1a0740 1 -- 172.21.15.70:0/845511068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 msgr2=0x5633a0768a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.305+0000 7f96af1a0740 1 --2- 172.21.15.70:0/845511068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 0x5633a0768a00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f96a00022f0 tx=0x7f96a0011890 comp rx=0 tx=0).stop 2024-04-01T01:39:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.305+0000 7f96af1a0740 1 -- 172.21.15.70:0/845511068 shutdown_connections 2024-04-01T01:39:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.305+0000 7f96af1a0740 1 --2- 172.21.15.70:0/845511068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a07684d0 0x5633a0768f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.305+0000 7f96af1a0740 1 --2- 172.21.15.70:0/845511068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a0764430 0x5633a0770170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.305+0000 7f96af1a0740 1 --2- 172.21.15.70:0/845511068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a075f2c0 0x5633a0768a00 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.305+0000 7f96af1a0740 1 -- 172.21.15.70:0/845511068 >> 172.21.15.70:0/845511068 conn(0x5633a0759e40 msgr2=0x5633a075a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:43.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.305+0000 7f96af1a0740 1 -- 172.21.15.70:0/845511068 shutdown_connections 2024-04-01T01:39:43.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.305+0000 7f96af1a0740 1 -- 172.21.15.70:0/845511068 wait complete. 2024-04-01T01:39:43.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.306+0000 7f96af1a0740 1 -- start start 2024-04-01T01:39:43.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.306+0000 7f96af1a0740 1 Processor -- start 2024-04-01T01:39:43.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.306+0000 7f96af1a0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a075f2c0 0x5633a076d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.306+0000 7f96af1a0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a0764430 0x5633a076dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.306+0000 7f96af1a0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a07684d0 0x5633a076edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.306+0000 7f96af1a0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f96880112a0 con 0x5633a075f2c0 2024-04-01T01:39:43.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.306+0000 7f96af1a0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9688010bd0 con 0x5633a07684d0 2024-04-01T01:39:43.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.306+0000 7f96af1a0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f968800de10 con 0x5633a0764430 2024-04-01T01:39:43.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.306+0000 7f96af1a0740 10 client.0 _ll_register_callbacks cb 0x7f96a0009430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:43.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.306+0000 7f96aa358640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a075f2c0 0x5633a076d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.307+0000 7f96aa358640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a075f2c0 0x5633a076d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48400/0 (socket says 172.21.15.70:48400) 2024-04-01T01:39:43.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.307+0000 7f96aa358640 1 -- 172.21.15.70:0/1431017380 learned_addr learned my addr 172.21.15.70:0/1431017380 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:43.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.307+0000 7f96a9b57640 1 --2- 172.21.15.70:0/1431017380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a0764430 0x5633a076dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.307+0000 7f96aab59640 1 --2- 172.21.15.70:0/1431017380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a07684d0 0x5633a076edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.307+0000 7f96a9b57640 1 -- 172.21.15.70:0/1431017380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a07684d0 msgr2=0x5633a076edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:43.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.307+0000 7f96a9b57640 1 --2- 172.21.15.70:0/1431017380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a07684d0 0x5633a076edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.307+0000 7f96a9b57640 1 -- 172.21.15.70:0/1431017380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a075f2c0 msgr2=0x5633a076d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:43.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.307+0000 7f96a9b57640 1 --2- 172.21.15.70:0/1431017380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a075f2c0 0x5633a076d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.307+0000 7f96a9b57640 1 -- 172.21.15.70:0/1431017380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633a076f3e0 con 0x5633a0764430 2024-04-01T01:39:43.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.307+0000 7f96a9b57640 1 --2- 172.21.15.70:0/1431017380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a0764430 0x5633a076dde0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f969c0cbd00 tx=0x7f969c01c040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51a1d47ee885c40b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:43.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.307+0000 7f968dffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:39:43.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.307+0000 7f96aab59640 1 --2- 172.21.15.70:0/1431017380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a07684d0 0x5633a076edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:43.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.307+0000 7f96aa358640 1 --2- 172.21.15.70:0/1431017380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a075f2c0 0x5633a076d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:43.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.308+0000 7f968dffb640 1 -- 172.21.15.70:0/1431017380 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c0b3060 con 0x5633a0764430 2024-04-01T01:39:43.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.308+0000 7f96af1a0740 1 -- 172.21.15.70:0/1431017380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96a0009880 con 0x5633a0764430 2024-04-01T01:39:43.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.308+0000 7f968dffb640 1 -- 172.21.15.70:0/1431017380 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f969c097f60 con 0x5633a0764430 2024-04-01T01:39:43.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.308+0000 7f968dffb640 1 -- 172.21.15.70:0/1431017380 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f969c0cce60 con 0x5633a0764430 2024-04-01T01:39:43.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.310+0000 7f96af1a0740 10 client.27160 fetch_fsmap learned FSMap version 29 2024-04-01T01:39:43.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.310+0000 7f96af1a0740 1 -- 172.21.15.70:0/1431017380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5633a07db220 con 0x5633a0764430 2024-04-01T01:39:43.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.310+0000 7f968dffb640 1 -- 172.21.15.70:0/1431017380 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96a0009880 con 0x5633a0764430 2024-04-01T01:39:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.311+0000 7f968dffb640 1 -- 172.21.15.70:0/1431017380 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f969c0a3f90 con 0x5633a0764430 2024-04-01T01:39:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.311+0000 7f96af1a0740 10 client.27160 fetch_fsmap finished waiting for FSMap version 29 2024-04-01T01:39:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.311+0000 7f96af1a0740 10 client.27160 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.311+0000 7f96af1a0740 20 client.27160 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.311+0000 7f96af1a0740 1 --2- 172.21.15.70:0/1431017380 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5633a07dbbd0 0x5633a07ddff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.311+0000 7f96af1a0740 4 client.27160 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:39:43.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.311+0000 7f96af1a0740 1 -- 172.21.15.70:0/1431017380 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5633a06cfe30 con 0x5633a07dbbd0 2024-04-01T01:39:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.313+0000 7f96aa358640 1 --2- 172.21.15.70:0/1431017380 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5633a07dbbd0 0x5633a07ddff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.315+0000 7f96aa358640 1 --2- 172.21.15.70:0/1431017380 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5633a07dbbd0 0x5633a07ddff0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b646115a8b364a5 server_cookie=78d020a2ecde9c49 in_seq=0 out_seq=0 2024-04-01T01:39:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.315+0000 7f968dffb640 10 client.27160 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.359+0000 7f968dffb640 1 -- 172.21.15.70:0/1431017380 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5633a06cfe30 con 0x5633a07dbbd0 2024-04-01T01:39:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.359+0000 7f968dffb640 10 client.27160 handle_command_reply: tid=0 2024-04-01T01:39:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.359+0000 7f96af1a0740 1 client.27160 shutdown 2024-04-01T01:39:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.359+0000 7f96af1a0740 1 -- 172.21.15.70:0/1431017380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a0764430 msgr2=0x5633a076dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.359+0000 7f96af1a0740 1 --2- 172.21.15.70:0/1431017380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a0764430 0x5633a076dde0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f969c0cbd00 tx=0x7f969c01c040 comp rx=0 tx=0).stop 2024-04-01T01:39:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.360+0000 7f96af1a0740 1 -- 172.21.15.70:0/1431017380 shutdown_connections 2024-04-01T01:39:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.360+0000 7f96af1a0740 1 --2- 172.21.15.70:0/1431017380 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5633a07dbbd0 0x5633a07ddff0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.360+0000 7f96af1a0740 1 --2- 172.21.15.70:0/1431017380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5633a07684d0 0x5633a076edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.360+0000 7f96af1a0740 1 --2- 172.21.15.70:0/1431017380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5633a0764430 0x5633a076dde0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.360+0000 7f96af1a0740 1 --2- 172.21.15.70:0/1431017380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633a075f2c0 0x5633a076d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.360+0000 7f96af1a0740 1 -- 172.21.15.70:0/1431017380 >> 172.21.15.70:0/1431017380 conn(0x5633a0759e40 msgr2=0x5633a07637b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.361+0000 7f968dffb640 0 client.27160 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.361+0000 7f96af1a0740 1 -- 172.21.15.70:0/1431017380 shutdown_connections 2024-04-01T01:39:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.361+0000 7f96af1a0740 1 -- 172.21.15.70:0/1431017380 wait complete. 2024-04-01T01:39:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.361+0000 7f96af1a0740 20 client.27160 trim_cache size 0 max 16384 2024-04-01T01:39:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:39:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:39:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:39:43.362 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:39:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.362+0000 7f96ac5bc640 1 -- 172.21.15.70:0/3644767663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96900a26b0 msgr2=0x7f96900a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.362+0000 7f96ac5bc640 1 --2- 172.21.15.70:0/3644767663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96900a26b0 0x7f96900a4b70 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9694046000 tx=0x7f9694049000 comp rx=0 tx=0).stop 2024-04-01T01:39:43.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.362+0000 7f96ac5bc640 1 -- 172.21.15.70:0/3644767663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96a410b0c0 msgr2=0x7f96a4115340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.362+0000 7f96ac5bc640 1 --2- 172.21.15.70:0/3644767663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96a410b0c0 0x7f96a4115340 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f969c009fd0 tx=0x7f969c00b870 comp rx=0 tx=0).stop 2024-04-01T01:39:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.362+0000 7f96ac5bc640 1 -- 172.21.15.70:0/3644767663 shutdown_connections 2024-04-01T01:39:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.362+0000 7f96ac5bc640 1 --2- 172.21.15.70:0/3644767663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96a410da70 0x7f96a41a58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.362+0000 7f96ac5bc640 1 --2- 172.21.15.70:0/3644767663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96900a26b0 0x7f96900a4b70 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.362+0000 7f96ac5bc640 1 --2- 172.21.15.70:0/3644767663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96a410b0c0 0x7f96a4115340 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.362+0000 7f96ac5bc640 1 --2- 172.21.15.70:0/3644767663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96a4108750 0x7f96a4114e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.362+0000 7f96ac5bc640 1 -- 172.21.15.70:0/3644767663 >> 172.21.15.70:0/3644767663 conn(0x7f96a406e790 msgr2=0x7f96a410a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.363+0000 7f96ac5bc640 1 -- 172.21.15.70:0/3644767663 shutdown_connections 2024-04-01T01:39:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:43.363+0000 7f96ac5bc640 1 -- 172.21.15.70:0/3644767663 wait complete. 2024-04-01T01:39:44.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:43 smithi070 ceph-mon[32179]: pgmap v545: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 19 MiB/s wr, 1.03k op/s 2024-04-01T01:39:44.077 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:43 smithi081 ceph-mon[35142]: pgmap v545: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 19 MiB/s wr, 1.03k op/s 2024-04-01T01:39:44.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:43 smithi189 ceph-mon[35753]: pgmap v545: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 19 MiB/s wr, 1.03k op/s 2024-04-01T01:39:44.249 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:39:43 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-c[66342]: ignoring --setuser ceph since I am not root 2024-04-01T01:39:44.249 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:39:43 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-c[66342]: ignoring --setgroup ceph since I am not root 2024-04-01T01:39:44.249 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:39:43 smithi189 ceph-mds[66346]: -- 172.21.15.189:0/1710022614 <== mon.2 v2:172.21.15.81:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56019e32fa40 con 0x56019e302800 2024-04-01T01:39:44.249 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:39:43 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-c[66342]: starting mds.c at 2024-04-01T01:39:45.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:44 smithi081 ceph-mon[35142]: daemon mds.c restarted 2024-04-01T01:39:45.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:44 smithi189 ceph-mon[35753]: daemon mds.c restarted 2024-04-01T01:39:45.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:44 smithi070 ceph-mon[32179]: daemon mds.c restarted 2024-04-01T01:39:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:45 smithi081 ceph-mon[35142]: pgmap v546: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 16 MiB/s wr, 915 op/s 2024-04-01T01:39:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:45 smithi081 ceph-mon[35142]: osdmap e94: 12 total, 12 up, 12 in 2024-04-01T01:39:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:45 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.189:6836/980664761,v1:172.21.15.189:6837/980664761] up:boot 2024-04-01T01:39:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:45 smithi081 ceph-mon[35142]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-01T01:39:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:45 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-01T01:39:46.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:45 smithi081 ceph-mon[35142]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-01T01:39:46.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:45 smithi189 ceph-mon[35753]: pgmap v546: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 16 MiB/s wr, 915 op/s 2024-04-01T01:39:46.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:45 smithi189 ceph-mon[35753]: osdmap e94: 12 total, 12 up, 12 in 2024-04-01T01:39:46.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:45 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.189:6836/980664761,v1:172.21.15.189:6837/980664761] up:boot 2024-04-01T01:39:46.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:45 smithi189 ceph-mon[35753]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-01T01:39:46.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:45 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-01T01:39:46.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:45 smithi189 ceph-mon[35753]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-01T01:39:46.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:45 smithi070 ceph-mon[32179]: pgmap v546: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 16 MiB/s wr, 915 op/s 2024-04-01T01:39:46.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:45 smithi070 ceph-mon[32179]: osdmap e94: 12 total, 12 up, 12 in 2024-04-01T01:39:46.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:45 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.189:6836/980664761,v1:172.21.15.189:6837/980664761] up:boot 2024-04-01T01:39:46.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:45 smithi070 ceph-mon[32179]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-01T01:39:46.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:45 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-01T01:39:46.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:45 smithi070 ceph-mon[32179]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-01T01:39:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 300.0 s, 861.2 tps, lat 116.272 ms stddev 179.087 2024-04-01T01:39:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:46 smithi081 ceph-mon[35142]: pgmap v548: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 16 MiB/s wr, 948 op/s 2024-04-01T01:39:47.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:46 smithi189 ceph-mon[35753]: pgmap v548: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 16 MiB/s wr, 948 op/s 2024-04-01T01:39:47.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:46 smithi070 ceph-mon[32179]: pgmap v548: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 16 MiB/s wr, 948 op/s 2024-04-01T01:39:49.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:48 smithi189 ceph-mon[35753]: pgmap v549: 129 pgs: 129 active+clean; 1.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 15 MiB/s wr, 924 op/s 2024-04-01T01:39:49.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:48 smithi070 ceph-mon[32179]: pgmap v549: 129 pgs: 129 active+clean; 1.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 15 MiB/s wr, 924 op/s 2024-04-01T01:39:49.327 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:48 smithi081 ceph-mon[35142]: pgmap v549: 129 pgs: 129 active+clean; 1.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 15 MiB/s wr, 924 op/s 2024-04-01T01:39:51.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 305.0 s, 1021.0 tps, lat 97.938 ms stddev 126.563 2024-04-01T01:39:52.012 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:39:52.012 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:39:52.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:51 smithi070 ceph-mon[32179]: pgmap v550: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 16 MiB/s wr, 912 op/s 2024-04-01T01:39:52.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:51 smithi081 ceph-mon[35142]: pgmap v550: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 16 MiB/s wr, 912 op/s 2024-04-01T01:39:52.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.196+0000 7f3f9d556640 1 -- 172.21.15.70:0/1641297017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f9810b0c0 msgr2=0x7f3f9810d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:52.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.196+0000 7f3f9d556640 1 --2- 172.21.15.70:0/1641297017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f9810b0c0 0x7f3f9810d530 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3f88009900 tx=0x7f3f8802bc60 comp rx=0 tx=0).stop 2024-04-01T01:39:52.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.196+0000 7f3f9d556640 1 -- 172.21.15.70:0/1641297017 shutdown_connections 2024-04-01T01:39:52.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.196+0000 7f3f9d556640 1 --2- 172.21.15.70:0/1641297017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9810da70 0x7f3f9810def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.196+0000 7f3f9d556640 1 --2- 172.21.15.70:0/1641297017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f9810b0c0 0x7f3f9810d530 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.196+0000 7f3f9d556640 1 --2- 172.21.15.70:0/1641297017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f98108750 0x7f3f9810ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.196+0000 7f3f9d556640 1 -- 172.21.15.70:0/1641297017 >> 172.21.15.70:0/1641297017 conn(0x7f3f9806e5b0 msgr2=0x7f3f980709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:52.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.196+0000 7f3f9d556640 1 -- 172.21.15.70:0/1641297017 shutdown_connections 2024-04-01T01:39:52.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.197+0000 7f3f9d556640 1 -- 172.21.15.70:0/1641297017 wait complete. 2024-04-01T01:39:52.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.197+0000 7f3f9d556640 1 Processor -- start 2024-04-01T01:39:52.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.198+0000 7f3f9d556640 1 -- start start 2024-04-01T01:39:52.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.198+0000 7f3f9d556640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f98108750 0x7f3f981a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.198+0000 7f3f9d556640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f9810b0c0 0x7f3f981a6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.198+0000 7f3f9d556640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9810da70 0x7f3f981ade40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.198+0000 7f3f9d556640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9811c140 con 0x7f3f98108750 2024-04-01T01:39:52.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.198+0000 7f3f9d556640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9811bfc0 con 0x7f3f9810b0c0 2024-04-01T01:39:52.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.198+0000 7f3f9d556640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9811c2c0 con 0x7f3f9810da70 2024-04-01T01:39:52.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.198+0000 7f3f967fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f9810b0c0 0x7f3f981a6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.198+0000 7f3f967fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f9810b0c0 0x7f3f981a6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37366/0 (socket says 172.21.15.70:37366) 2024-04-01T01:39:52.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.198+0000 7f3f967fc640 1 -- 172.21.15.70:0/1224077102 learned_addr learned my addr 172.21.15.70:0/1224077102 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:52.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.198+0000 7f3f977fe640 1 --2- 172.21.15.70:0/1224077102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9810da70 0x7f3f981ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.199+0000 7f3f96ffd640 1 --2- 172.21.15.70:0/1224077102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f98108750 0x7f3f981a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.199+0000 7f3f977fe640 1 -- 172.21.15.70:0/1224077102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f9810b0c0 msgr2=0x7f3f981a6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:52.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.199+0000 7f3f977fe640 1 --2- 172.21.15.70:0/1224077102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f9810b0c0 0x7f3f981a6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.199+0000 7f3f977fe640 1 -- 172.21.15.70:0/1224077102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f98108750 msgr2=0x7f3f981a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:52.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.199+0000 7f3f977fe640 1 --2- 172.21.15.70:0/1224077102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f98108750 0x7f3f981a68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.199+0000 7f3f977fe640 1 -- 172.21.15.70:0/1224077102 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f981ae4b0 con 0x7f3f9810da70 2024-04-01T01:39:52.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.199+0000 7f3f967fc640 1 --2- 172.21.15.70:0/1224077102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f9810b0c0 0x7f3f981a6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:52.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.199+0000 7f3f977fe640 1 --2- 172.21.15.70:0/1224077102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9810da70 0x7f3f981ade40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3f9806b440 tx=0x7f3f8c00cba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:52.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.199+0000 7f3f7ffff640 1 -- 172.21.15.70:0/1224077102 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c004220 con 0x7f3f9810da70 2024-04-01T01:39:52.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.200+0000 7f3f9d556640 1 -- 172.21.15.70:0/1224077102 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f981ae7a0 con 0x7f3f9810da70 2024-04-01T01:39:52.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.200+0000 7f3f7ffff640 1 -- 172.21.15.70:0/1224077102 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f8c0043c0 con 0x7f3f9810da70 2024-04-01T01:39:52.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.200+0000 7f3f7ffff640 1 -- 172.21.15.70:0/1224077102 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c003de0 con 0x7f3f9810da70 2024-04-01T01:39:52.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.200+0000 7f3f9d556640 1 -- 172.21.15.70:0/1224077102 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f981aed70 con 0x7f3f9810da70 2024-04-01T01:39:52.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.203+0000 7f3fa0214740 1 Processor -- start 2024-04-01T01:39:52.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.204+0000 7f3f7ffff640 1 -- 172.21.15.70:0/1224077102 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f3f8c0077a0 con 0x7f3f9810da70 2024-04-01T01:39:52.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.205+0000 7f3fa0214740 1 -- start start 2024-04-01T01:39:52.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.205+0000 7f3fa0214740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a57f2b0 0x56321a57f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.205+0000 7f3fa0214740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 0x56321a5847a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.205+0000 7f3fa0214740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a584e10 0x56321a5872a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.205+0000 7f3fa0214740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56321a4e4180 con 0x56321a57fca0 2024-04-01T01:39:52.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.205+0000 7f3fa0214740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56321a48cb70 con 0x56321a57f2b0 2024-04-01T01:39:52.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.205+0000 7f3fa0214740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56321a4e0220 con 0x56321a584e10 2024-04-01T01:39:52.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.205+0000 7f3f7ffff640 1 --2- 172.21.15.70:0/1224077102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f800a2520 0x7f3f800a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.205+0000 7f3f967fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a57f2b0 0x56321a57f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.205+0000 7f3f967fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a57f2b0 0x56321a57f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37374/0 (socket says 172.21.15.70:37374) 2024-04-01T01:39:52.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.205+0000 7f3f967fc640 1 -- 172.21.15.70:0/2446839014 learned_addr learned my addr 172.21.15.70:0/2446839014 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:52.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.205+0000 7f3f96ffd640 1 --2- 172.21.15.70:0/2446839014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 0x56321a5847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.205+0000 7f3f967fc640 1 --2- 172.21.15.70:0/2446839014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a584e10 0x56321a5872a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.205+0000 7f3f977fe640 1 --2- 172.21.15.70:0/1224077102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f800a2520 0x7f3f800a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.205+0000 7f3f7ffff640 1 -- 172.21.15.70:0/1224077102 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3f8c0ce630 con 0x7f3f9810da70 2024-04-01T01:39:52.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.206+0000 7f3f96ffd640 1 -- 172.21.15.70:0/2446839014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a584e10 msgr2=0x56321a5872a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:52.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.206+0000 7f3f96ffd640 1 --2- 172.21.15.70:0/2446839014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a584e10 0x56321a5872a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.206+0000 7f3f96ffd640 1 -- 172.21.15.70:0/2446839014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a57f2b0 msgr2=0x56321a57f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:52.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.206+0000 7f3f96ffd640 1 --2- 172.21.15.70:0/2446839014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a57f2b0 0x56321a57f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.206+0000 7f3f96ffd640 1 -- 172.21.15.70:0/2446839014 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56321a4a07c0 con 0x56321a57fca0 2024-04-01T01:39:52.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.206+0000 7f3f96ffd640 1 --2- 172.21.15.70:0/2446839014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 0x56321a5847a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f3f90009f20 tx=0x7f3f9000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3924153ffb4ea269 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:52.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.206+0000 7f3f7d7fa640 1 -- 172.21.15.70:0/2446839014 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9001a070 con 0x56321a57fca0 2024-04-01T01:39:52.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.206+0000 7f3f7d7fa640 1 -- 172.21.15.70:0/2446839014 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f900158c0 con 0x56321a57fca0 2024-04-01T01:39:52.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.206+0000 7f3f7d7fa640 1 -- 172.21.15.70:0/2446839014 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9000a8a0 con 0x56321a57fca0 2024-04-01T01:39:52.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.207+0000 7f3fa0214740 1 -- 172.21.15.70:0/2446839014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 msgr2=0x56321a5847a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:52.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.207+0000 7f3fa0214740 1 --2- 172.21.15.70:0/2446839014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 0x56321a5847a0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f3f90009f20 tx=0x7f3f9000c8b0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.207+0000 7f3f977fe640 1 --2- 172.21.15.70:0/1224077102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f800a2520 0x7f3f800a49e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c005fd0 tx=0x7f3f8c065c00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:52.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.207+0000 7f3fa0214740 1 -- 172.21.15.70:0/2446839014 shutdown_connections 2024-04-01T01:39:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.207+0000 7f3fa0214740 1 --2- 172.21.15.70:0/2446839014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a584e10 0x56321a5872a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.207+0000 7f3fa0214740 1 --2- 172.21.15.70:0/2446839014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 0x56321a5847a0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.207+0000 7f3fa0214740 1 --2- 172.21.15.70:0/2446839014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a57f2b0 0x56321a57f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.207+0000 7f3fa0214740 1 -- 172.21.15.70:0/2446839014 >> 172.21.15.70:0/2446839014 conn(0x56321a57a820 msgr2=0x56321a57ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.208+0000 7f3fa0214740 1 -- 172.21.15.70:0/2446839014 shutdown_connections 2024-04-01T01:39:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.208+0000 7f3fa0214740 1 -- 172.21.15.70:0/2446839014 wait complete. 2024-04-01T01:39:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.208+0000 7f3fa0214740 1 -- start start 2024-04-01T01:39:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.208+0000 7f3fa0214740 1 Processor -- start 2024-04-01T01:39:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.209+0000 7f3fa0214740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 0x56321a5897d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.209+0000 7f3fa0214740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a57fca0 0x56321a589d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.209+0000 7f3fa0214740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a584e10 0x56321a590d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.209+0000 7f3fa0214740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56321a587dc0 con 0x56321a584e10 2024-04-01T01:39:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.209+0000 7f3fa0214740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56321a4824a0 con 0x56321a57fca0 2024-04-01T01:39:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.209+0000 7f3fa0214740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56321a587f40 con 0x56321a57f2b0 2024-04-01T01:39:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.209+0000 7f3fa0214740 10 client.0 _ll_register_callbacks cb 0x56321a485d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.209+0000 7f3f967fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 0x56321a5897d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.209+0000 7f3f967fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 0x56321a5897d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49618/0 (socket says 172.21.15.70:49618) 2024-04-01T01:39:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.209+0000 7f3f967fc640 1 -- 172.21.15.70:0/2622483145 learned_addr learned my addr 172.21.15.70:0/2622483145 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.209+0000 7f3f967fc640 1 --2- 172.21.15.70:0/2622483145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a584e10 0x56321a590d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.210+0000 7f3f96ffd640 1 --2- 172.21.15.70:0/2622483145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a57fca0 0x56321a589d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.210+0000 7f3f967fc640 1 -- 172.21.15.70:0/2622483145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a57fca0 msgr2=0x56321a589d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.210+0000 7f3f967fc640 1 --2- 172.21.15.70:0/2622483145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a57fca0 0x56321a589d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.210+0000 7f3f967fc640 1 -- 172.21.15.70:0/2622483145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a584e10 msgr2=0x56321a590d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.210+0000 7f3f967fc640 1 --2- 172.21.15.70:0/2622483145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a584e10 0x56321a590d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.210+0000 7f3f967fc640 1 -- 172.21.15.70:0/2622483145 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56321a2c27d0 con 0x56321a57f2b0 2024-04-01T01:39:52.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.210+0000 7f3f967fc640 1 --2- 172.21.15.70:0/2622483145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 0x56321a5897d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3f88005730 tx=0x7f3f8802bc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9276d0f3239d0a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:52.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.210+0000 7f3f7aff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:39:52.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.211+0000 7f3f7aff5640 1 -- 172.21.15.70:0/2622483145 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f88004b00 con 0x56321a57f2b0 2024-04-01T01:39:52.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.211+0000 7f3f7aff5640 1 -- 172.21.15.70:0/2622483145 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3f88004ca0 con 0x56321a57f2b0 2024-04-01T01:39:52.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.211+0000 7f3fa0214740 1 -- 172.21.15.70:0/2622483145 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f900158c0 con 0x56321a57f2b0 2024-04-01T01:39:52.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.211+0000 7f3f7aff5640 1 -- 172.21.15.70:0/2622483145 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3f88057050 con 0x56321a57f2b0 2024-04-01T01:39:52.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.211+0000 7f3fa0214740 10 client.27175 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:52.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.211+0000 7f3fa0214740 1 -- 172.21.15.70:0/2622483145 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56321a1f83f0 con 0x56321a57f2b0 2024-04-01T01:39:52.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.211+0000 7f3f7aff5640 1 -- 172.21.15.70:0/2622483145 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f900158c0 con 0x56321a57f2b0 2024-04-01T01:39:52.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.212+0000 7f3f7aff5640 1 -- 172.21.15.70:0/2622483145 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f3f880514b0 con 0x56321a57f2b0 2024-04-01T01:39:52.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.212+0000 7f3fa0214740 10 client.27175 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:52.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.212+0000 7f3fa0214740 10 client.27175 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:52.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.212+0000 7f3fa0214740 20 client.27175 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:52.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.212+0000 7f3fa0214740 1 --2- 172.21.15.70:0/2622483145 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56321a5955d0 0x56321a597a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.212+0000 7f3fa0214740 4 client.27175 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:52.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.212+0000 7f3fa0214740 1 -- 172.21.15.70:0/2622483145 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56321a4f16c0 con 0x56321a5955d0 2024-04-01T01:39:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.212+0000 7f3f96ffd640 1 --2- 172.21.15.70:0/2622483145 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56321a5955d0 0x56321a597a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.213+0000 7f3f96ffd640 1 --2- 172.21.15.70:0/2622483145 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56321a5955d0 0x56321a597a50 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a81bec6ae226e5f server_cookie=7ff2d375be684d99 in_seq=0 out_seq=0 2024-04-01T01:39:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.213+0000 7f3f7aff5640 10 client.27175 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.214+0000 7f3f7aff5640 1 -- 172.21.15.70:0/2622483145 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56321a4f16c0 con 0x56321a5955d0 2024-04-01T01:39:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.214+0000 7f3f7aff5640 10 client.27175 handle_command_reply: tid=0 2024-04-01T01:39:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.214+0000 7f3fa0214740 1 client.27175 shutdown 2024-04-01T01:39:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.215+0000 7f3fa0214740 1 -- 172.21.15.70:0/2622483145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 msgr2=0x56321a5897d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.215+0000 7f3fa0214740 1 --2- 172.21.15.70:0/2622483145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 0x56321a5897d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3f88005730 tx=0x7f3f8802bc60 comp rx=0 tx=0).stop 2024-04-01T01:39:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.216+0000 7f3fa0214740 1 -- 172.21.15.70:0/2622483145 shutdown_connections 2024-04-01T01:39:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.216+0000 7f3fa0214740 1 --2- 172.21.15.70:0/2622483145 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56321a5955d0 0x56321a597a50 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.216+0000 7f3fa0214740 1 --2- 172.21.15.70:0/2622483145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a584e10 0x56321a590d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.216+0000 7f3fa0214740 1 --2- 172.21.15.70:0/2622483145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a57fca0 0x56321a589d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.216+0000 7f3fa0214740 1 --2- 172.21.15.70:0/2622483145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 0x56321a5897d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.216+0000 7f3fa0214740 1 -- 172.21.15.70:0/2622483145 >> 172.21.15.70:0/2622483145 conn(0x56321a57a820 msgr2=0x56321a579c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.216+0000 7f3f7aff5640 0 client.27175 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:52.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.218+0000 7f3fa0214740 1 -- 172.21.15.70:0/2622483145 shutdown_connections 2024-04-01T01:39:52.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.218+0000 7f3fa0214740 1 -- 172.21.15.70:0/2622483145 wait complete. 2024-04-01T01:39:52.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.218+0000 7f3fa0214740 20 client.27175 trim_cache size 0 max 16384 2024-04-01T01:39:52.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:51 smithi189 ceph-mon[35753]: pgmap v550: 129 pgs: 129 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 16 MiB/s wr, 912 op/s 2024-04-01T01:39:52.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.254+0000 7f3fa0214740 1 Processor -- start 2024-04-01T01:39:52.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.254+0000 7f3fa0214740 1 -- start start 2024-04-01T01:39:52.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.254+0000 7f3fa0214740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 0x56321a589c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.254+0000 7f3fa0214740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 0x56321a590d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.254+0000 7f3fa0214740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a584e10 0x56321a579390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.254+0000 7f3fa0214740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3f981b9300 con 0x56321a57fca0 2024-04-01T01:39:52.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.254+0000 7f3fa0214740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3f981bae40 con 0x56321a584e10 2024-04-01T01:39:52.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.255+0000 7f3fa0214740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f981b99d0 con 0x56321a57f2b0 2024-04-01T01:39:52.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.256+0000 7f3f96ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 0x56321a590d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.256+0000 7f3f96ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 0x56321a590d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44426/0 (socket says 172.21.15.70:44426) 2024-04-01T01:39:52.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.256+0000 7f3f96ffd640 1 -- 172.21.15.70:0/4281985394 learned_addr learned my addr 172.21.15.70:0/4281985394 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:52.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.256+0000 7f3f967fc640 1 --2- 172.21.15.70:0/4281985394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a584e10 0x56321a579390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.256+0000 7f3f967fc640 1 --2- 172.21.15.70:0/4281985394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 0x56321a589c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.256+0000 7f3f967fc640 1 -- 172.21.15.70:0/4281985394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 msgr2=0x56321a589c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.256+0000 7f3f967fc640 1 --2- 172.21.15.70:0/4281985394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 0x56321a589c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.256+0000 7f3f967fc640 1 -- 172.21.15.70:0/4281985394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 msgr2=0x56321a590d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.256+0000 7f3f967fc640 1 --2- 172.21.15.70:0/4281985394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 0x56321a590d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.256+0000 7f3f967fc640 1 -- 172.21.15.70:0/4281985394 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56321a1b80f0 con 0x56321a584e10 2024-04-01T01:39:52.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.257+0000 7f3f967fc640 1 --2- 172.21.15.70:0/4281985394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a584e10 0x56321a579390 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3f88006010 tx=0x7f3f8802c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b960cf362020f7d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.258+0000 7f3f7cff9640 1 -- 172.21.15.70:0/4281985394 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f88004b00 con 0x56321a584e10 2024-04-01T01:39:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.258+0000 7f3f7cff9640 1 -- 172.21.15.70:0/4281985394 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f880510d0 con 0x56321a584e10 2024-04-01T01:39:52.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.258+0000 7f3f7cff9640 1 -- 172.21.15.70:0/4281985394 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f88003970 con 0x56321a584e10 2024-04-01T01:39:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.259+0000 7f3fa0214740 1 -- 172.21.15.70:0/4281985394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a584e10 msgr2=0x56321a579390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.259+0000 7f3fa0214740 1 --2- 172.21.15.70:0/4281985394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a584e10 0x56321a579390 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3f88006010 tx=0x7f3f8802c810 comp rx=0 tx=0).stop 2024-04-01T01:39:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.259+0000 7f3fa0214740 1 -- 172.21.15.70:0/4281985394 shutdown_connections 2024-04-01T01:39:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.259+0000 7f3fa0214740 1 --2- 172.21.15.70:0/4281985394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a584e10 0x56321a579390 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.259+0000 7f3fa0214740 1 --2- 172.21.15.70:0/4281985394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 0x56321a590d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.259+0000 7f3fa0214740 1 --2- 172.21.15.70:0/4281985394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 0x56321a589c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.259+0000 7f3fa0214740 1 -- 172.21.15.70:0/4281985394 >> 172.21.15.70:0/4281985394 conn(0x56321a57a820 msgr2=0x56321a57ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.259+0000 7f3fa0214740 1 -- 172.21.15.70:0/4281985394 shutdown_connections 2024-04-01T01:39:52.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.260+0000 7f3fa0214740 1 -- 172.21.15.70:0/4281985394 wait complete. 2024-04-01T01:39:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.260+0000 7f3fa0214740 1 -- start start 2024-04-01T01:39:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.260+0000 7f3fa0214740 1 Processor -- start 2024-04-01T01:39:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.260+0000 7f3fa0214740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 0x56321a58e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.260+0000 7f3fa0214740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 0x56321a58eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.260+0000 7f3fa0214740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a584e10 0x56321a58faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.260+0000 7f3fa0214740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3f981ba770 con 0x56321a57fca0 2024-04-01T01:39:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.260+0000 7f3fa0214740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3f981ba0a0 con 0x56321a584e10 2024-04-01T01:39:52.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.260+0000 7f3fa0214740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f981b72e0 con 0x56321a57f2b0 2024-04-01T01:39:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.261+0000 7f3fa0214740 10 client.0 _ll_register_callbacks cb 0x7f3f880055f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.261+0000 7f3f967fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 0x56321a58e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.261+0000 7f3f967fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 0x56321a58e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49642/0 (socket says 172.21.15.70:49642) 2024-04-01T01:39:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.261+0000 7f3f967fc640 1 -- 172.21.15.70:0/2005140851 learned_addr learned my addr 172.21.15.70:0/2005140851 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.261+0000 7f3f967fc640 1 --2- 172.21.15.70:0/2005140851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a584e10 0x56321a58faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.261+0000 7f3f96ffd640 1 --2- 172.21.15.70:0/2005140851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 0x56321a58eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.261+0000 7f3f967fc640 1 -- 172.21.15.70:0/2005140851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a584e10 msgr2=0x56321a58faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.261+0000 7f3f967fc640 1 --2- 172.21.15.70:0/2005140851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a584e10 0x56321a58faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.261+0000 7f3f967fc640 1 -- 172.21.15.70:0/2005140851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 msgr2=0x56321a58eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.261+0000 7f3f967fc640 1 --2- 172.21.15.70:0/2005140851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 0x56321a58eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.261+0000 7f3f967fc640 1 -- 172.21.15.70:0/2005140851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56321a5900e0 con 0x56321a57f2b0 2024-04-01T01:39:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.261+0000 7f3f967fc640 1 --2- 172.21.15.70:0/2005140851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 0x56321a58e610 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3f88051e30 tx=0x7f3f88005d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f3dc5ac44031d1f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:52.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.261+0000 7f3f7aff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:39:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.262+0000 7f3f7aff5640 1 -- 172.21.15.70:0/2005140851 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8805f040 con 0x56321a57f2b0 2024-04-01T01:39:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.262+0000 7f3f7aff5640 1 -- 172.21.15.70:0/2005140851 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3f88004b00 con 0x56321a57f2b0 2024-04-01T01:39:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.262+0000 7f3fa0214740 1 -- 172.21.15.70:0/2005140851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f880510d0 con 0x56321a57f2b0 2024-04-01T01:39:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.263+0000 7f3f7aff5640 1 -- 172.21.15.70:0/2005140851 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3f88004d00 con 0x56321a57f2b0 2024-04-01T01:39:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.263+0000 7f3fa0214740 10 client.27181 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.263+0000 7f3fa0214740 1 -- 172.21.15.70:0/2005140851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56321a5fbb40 con 0x56321a57f2b0 2024-04-01T01:39:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.263+0000 7f3f7aff5640 1 -- 172.21.15.70:0/2005140851 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f880510d0 con 0x56321a57f2b0 2024-04-01T01:39:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.263+0000 7f3f7aff5640 1 -- 172.21.15.70:0/2005140851 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f3f88034440 con 0x56321a57f2b0 2024-04-01T01:39:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.264+0000 7f3fa0214740 10 client.27181 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.264+0000 7f3fa0214740 10 client.27181 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.264+0000 7f3fa0214740 20 client.27181 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.264+0000 7f3fa0214740 1 --2- 172.21.15.70:0/2005140851 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56321a5fc440 0x56321a5fe860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.264+0000 7f3fa0214740 4 client.27181 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:39:52.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.264+0000 7f3fa0214740 1 -- 172.21.15.70:0/2005140851 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56321a4eba10 con 0x56321a5fc440 2024-04-01T01:39:52.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.265+0000 7f3f96ffd640 1 --2- 172.21.15.70:0/2005140851 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56321a5fc440 0x56321a5fe860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.266+0000 7f3f96ffd640 1 --2- 172.21.15.70:0/2005140851 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56321a5fc440 0x56321a5fe860 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=348b1d0be01442aa server_cookie=56bb6fae7c55f381 in_seq=0 out_seq=0 2024-04-01T01:39:52.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.266+0000 7f3f7aff5640 10 client.27181 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.269+0000 7f3f7aff5640 1 -- 172.21.15.70:0/2005140851 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56321a4eba10 con 0x56321a5fc440 2024-04-01T01:39:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.269+0000 7f3f7aff5640 10 client.27181 handle_command_reply: tid=0 2024-04-01T01:39:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.269+0000 7f3fa0214740 1 client.27181 shutdown 2024-04-01T01:39:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.269+0000 7f3fa0214740 1 -- 172.21.15.70:0/2005140851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 msgr2=0x56321a58e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.269+0000 7f3fa0214740 1 --2- 172.21.15.70:0/2005140851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 0x56321a58e610 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3f88051e30 tx=0x7f3f88005d90 comp rx=0 tx=0).stop 2024-04-01T01:39:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.270+0000 7f3fa0214740 1 -- 172.21.15.70:0/2005140851 shutdown_connections 2024-04-01T01:39:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.270+0000 7f3fa0214740 1 --2- 172.21.15.70:0/2005140851 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56321a5fc440 0x56321a5fe860 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.270+0000 7f3fa0214740 1 --2- 172.21.15.70:0/2005140851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56321a584e10 0x56321a58faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.270+0000 7f3fa0214740 1 --2- 172.21.15.70:0/2005140851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56321a57fca0 0x56321a58eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.270+0000 7f3fa0214740 1 --2- 172.21.15.70:0/2005140851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56321a57f2b0 0x56321a58e610 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.270+0000 7f3fa0214740 1 -- 172.21.15.70:0/2005140851 >> 172.21.15.70:0/2005140851 conn(0x56321a57a820 msgr2=0x56321a579390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.270+0000 7f3f7aff5640 0 client.27181 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:52.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.270+0000 7f3fa0214740 1 -- 172.21.15.70:0/2005140851 shutdown_connections 2024-04-01T01:39:52.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.270+0000 7f3fa0214740 1 -- 172.21.15.70:0/2005140851 wait complete. 2024-04-01T01:39:52.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.271+0000 7f3fa0214740 20 client.27181 trim_cache size 0 max 16384 2024-04-01T01:39:52.272 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:39:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:39:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:39:52.272 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:39:52.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.277+0000 7f3f9d556640 1 -- 172.21.15.70:0/1224077102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f800a2520 msgr2=0x7f3f800a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:52.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.277+0000 7f3f9d556640 1 --2- 172.21.15.70:0/1224077102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f800a2520 0x7f3f800a49e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c005fd0 tx=0x7f3f8c065c00 comp rx=0 tx=0).stop 2024-04-01T01:39:52.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.277+0000 7f3f9d556640 1 -- 172.21.15.70:0/1224077102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9810da70 msgr2=0x7f3f981ade40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:52.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.277+0000 7f3f9d556640 1 --2- 172.21.15.70:0/1224077102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9810da70 0x7f3f981ade40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3f9806b440 tx=0x7f3f8c00cba0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.278+0000 7f3f9d556640 1 -- 172.21.15.70:0/1224077102 shutdown_connections 2024-04-01T01:39:52.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.278+0000 7f3f9d556640 1 --2- 172.21.15.70:0/1224077102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3f800a2520 0x7f3f800a49e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.278+0000 7f3f9d556640 1 --2- 172.21.15.70:0/1224077102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f9810da70 0x7f3f981ade40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.278+0000 7f3f9d556640 1 --2- 172.21.15.70:0/1224077102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f9810b0c0 0x7f3f981a6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.278+0000 7f3f9d556640 1 --2- 172.21.15.70:0/1224077102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f98108750 0x7f3f981a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.278+0000 7f3f9d556640 1 -- 172.21.15.70:0/1224077102 >> 172.21.15.70:0/1224077102 conn(0x7f3f9806e5b0 msgr2=0x7f3f98108b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:52.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.278+0000 7f3f9d556640 1 -- 172.21.15.70:0/1224077102 shutdown_connections 2024-04-01T01:39:52.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.279+0000 7f3f9d556640 1 -- 172.21.15.70:0/1224077102 wait complete. 2024-04-01T01:39:52.297 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:39:52.298 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:39:52.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.514+0000 7f63d300e640 1 -- 172.21.15.70:0/1391411887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63cc076110 msgr2=0x7f63cc076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:52.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.514+0000 7f63d300e640 1 --2- 172.21.15.70:0/1391411887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63cc076110 0x7f63cc076530 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f63c4009ca0 tx=0x7f63c402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.514+0000 7f63d300e640 1 -- 172.21.15.70:0/1391411887 shutdown_connections 2024-04-01T01:39:52.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.514+0000 7f63d300e640 1 --2- 172.21.15.70:0/1391411887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63cc077cf0 0x7f63cc078590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.514+0000 7f63d300e640 1 --2- 172.21.15.70:0/1391411887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63cc077350 0x7f63cc0777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.514+0000 7f63d300e640 1 --2- 172.21.15.70:0/1391411887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63cc076110 0x7f63cc076530 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.514+0000 7f63d300e640 1 -- 172.21.15.70:0/1391411887 >> 172.21.15.70:0/1391411887 conn(0x7f63cc0fef00 msgr2=0x7f63cc101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:52.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.514+0000 7f63d300e640 1 -- 172.21.15.70:0/1391411887 shutdown_connections 2024-04-01T01:39:52.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.514+0000 7f63d300e640 1 -- 172.21.15.70:0/1391411887 wait complete. 2024-04-01T01:39:52.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.515+0000 7f63d300e640 1 Processor -- start 2024-04-01T01:39:52.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.515+0000 7f63d300e640 1 -- start start 2024-04-01T01:39:52.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.515+0000 7f63d300e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63cc076110 0x7f63cc19e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.515+0000 7f63d300e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63cc077350 0x7f63cc19e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.515+0000 7f63d300e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63cc077cf0 0x7f63cc1a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.515+0000 7f63d300e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc113950 con 0x7f63cc077cf0 2024-04-01T01:39:52.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.515+0000 7f63d300e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc1137d0 con 0x7f63cc076110 2024-04-01T01:39:52.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.515+0000 7f63d300e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc113ad0 con 0x7f63cc077350 2024-04-01T01:39:52.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.516+0000 7f63d0daa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63cc076110 0x7f63cc19e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.516+0000 7f63d0daa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63cc076110 0x7f63cc19e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37424/0 (socket says 172.21.15.70:37424) 2024-04-01T01:39:52.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.516+0000 7f63d0daa640 1 -- 172.21.15.70:0/317687592 learned_addr learned my addr 172.21.15.70:0/317687592 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:52.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.516+0000 7f63d15ab640 1 --2- 172.21.15.70:0/317687592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63cc077cf0 0x7f63cc1a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.516+0000 7f63cbfff640 1 --2- 172.21.15.70:0/317687592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63cc077350 0x7f63cc19e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.516+0000 7f63d0daa640 1 -- 172.21.15.70:0/317687592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63cc077350 msgr2=0x7f63cc19e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:52.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.516+0000 7f63d0daa640 1 --2- 172.21.15.70:0/317687592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63cc077350 0x7f63cc19e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.516+0000 7f63d0daa640 1 -- 172.21.15.70:0/317687592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63cc077cf0 msgr2=0x7f63cc1a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:52.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.516+0000 7f63d0daa640 1 --2- 172.21.15.70:0/317687592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63cc077cf0 0x7f63cc1a5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.516+0000 7f63d0daa640 1 -- 172.21.15.70:0/317687592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63cc1a5c80 con 0x7f63cc076110 2024-04-01T01:39:52.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.516+0000 7f63d0daa640 1 --2- 172.21.15.70:0/317687592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63cc076110 0x7f63cc19e060 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f63c402c700 tx=0x7f63c4003ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:52.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.517+0000 7f63c9ffb640 1 -- 172.21.15.70:0/317687592 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c4039040 con 0x7f63cc076110 2024-04-01T01:39:52.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.517+0000 7f63c9ffb640 1 -- 172.21.15.70:0/317687592 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63c4044070 con 0x7f63cc076110 2024-04-01T01:39:52.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.517+0000 7f63d300e640 1 -- 172.21.15.70:0/317687592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63cc1a5f10 con 0x7f63cc076110 2024-04-01T01:39:52.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.517+0000 7f63d300e640 1 -- 172.21.15.70:0/317687592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63cc1a63f0 con 0x7f63cc076110 2024-04-01T01:39:52.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.517+0000 7f63c9ffb640 1 -- 172.21.15.70:0/317687592 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c4032ca0 con 0x7f63cc076110 2024-04-01T01:39:52.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.518+0000 7f63d300e640 1 -- 172.21.15.70:0/317687592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63cc110300 con 0x7f63cc076110 2024-04-01T01:39:52.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.520+0000 7f63c9ffb640 1 -- 172.21.15.70:0/317687592 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f63c406fb10 con 0x7f63cc076110 2024-04-01T01:39:52.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.521+0000 7f63c9ffb640 1 --2- 172.21.15.70:0/317687592 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f63b00a25f0 0x7f63b00a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:52.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.521+0000 7f63c9ffb640 1 -- 172.21.15.70:0/317687592 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f63c40f0c80 con 0x7f63cc076110 2024-04-01T01:39:52.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.522+0000 7f63cbfff640 1 --2- 172.21.15.70:0/317687592 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f63b00a25f0 0x7f63b00a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:52.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.523+0000 7f63cbfff640 1 --2- 172.21.15.70:0/317687592 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f63b00a25f0 0x7f63b00a4ab0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f63cc19f570 tx=0x7f63bc009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:52.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.528+0000 7f63c9ffb640 1 -- 172.21.15.70:0/317687592 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f63c404d020 con 0x7f63cc076110 2024-04-01T01:39:52.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.957+0000 7f63d300e640 1 -- 172.21.15.70:0/317687592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63cc02d170 con 0x7f63cc076110 2024-04-01T01:39:52.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.958+0000 7f63c9ffb640 1 -- 172.21.15.70:0/317687592 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7f63c40bda10 con 0x7f63cc076110 2024-04-01T01:39:52.958 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:39:52.958 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:39:52.958 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:39:52.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.962+0000 7f63d300e640 1 -- 172.21.15.70:0/317687592 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f63b00a25f0 msgr2=0x7f63b00a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:52.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.963+0000 7f63d300e640 1 --2- 172.21.15.70:0/317687592 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f63b00a25f0 0x7f63b00a4ab0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f63cc19f570 tx=0x7f63bc009110 comp rx=0 tx=0).stop 2024-04-01T01:39:52.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.963+0000 7f63d300e640 1 -- 172.21.15.70:0/317687592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63cc076110 msgr2=0x7f63cc19e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:52.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.963+0000 7f63d300e640 1 --2- 172.21.15.70:0/317687592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63cc076110 0x7f63cc19e060 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f63c402c700 tx=0x7f63c4003ce0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.963+0000 7f63d300e640 1 -- 172.21.15.70:0/317687592 shutdown_connections 2024-04-01T01:39:52.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.963+0000 7f63d300e640 1 --2- 172.21.15.70:0/317687592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63cc077cf0 0x7f63cc1a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.963+0000 7f63d300e640 1 --2- 172.21.15.70:0/317687592 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f63b00a25f0 0x7f63b00a4ab0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.963+0000 7f63d300e640 1 --2- 172.21.15.70:0/317687592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63cc077350 0x7f63cc19e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.963+0000 7f63d300e640 1 --2- 172.21.15.70:0/317687592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63cc076110 0x7f63cc19e060 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:52.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.963+0000 7f63d300e640 1 -- 172.21.15.70:0/317687592 >> 172.21.15.70:0/317687592 conn(0x7f63cc0fef00 msgr2=0x7f63cc102260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:52.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.963+0000 7f63d300e640 1 -- 172.21.15.70:0/317687592 shutdown_connections 2024-04-01T01:39:52.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:52.964+0000 7f63d300e640 1 -- 172.21.15.70:0/317687592 wait complete. 2024-04-01T01:39:52.985 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:39:53.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.168+0000 7f4f65ffb640 1 -- 172.21.15.70:0/1844848435 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c034e10 con 0x7f4f68109800 2024-04-01T01:39:53.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.169+0000 7f4f6d981640 1 -- 172.21.15.70:0/1844848435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f68109800 msgr2=0x7f4f6810e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.169+0000 7f4f6d981640 1 --2- 172.21.15.70:0/1844848435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f68109800 0x7f4f6810e0c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c009860 tx=0x7f4f5c02bc00 comp rx=0 tx=0).stop 2024-04-01T01:39:53.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.169+0000 7f4f6d981640 1 -- 172.21.15.70:0/1844848435 shutdown_connections 2024-04-01T01:39:53.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.169+0000 7f4f6d981640 1 --2- 172.21.15.70:0/1844848435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f68109800 0x7f4f6810e0c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.169+0000 7f4f6d981640 1 --2- 172.21.15.70:0/1844848435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f68108e60 0x7f4f681092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.169+0000 7f4f6d981640 1 --2- 172.21.15.70:0/1844848435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f68072b30 0x7f4f68072f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.169+0000 7f4f6d981640 1 -- 172.21.15.70:0/1844848435 >> 172.21.15.70:0/1844848435 conn(0x7f4f6806e630 msgr2=0x7f4f68070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.170+0000 7f4f6d981640 1 -- 172.21.15.70:0/1844848435 shutdown_connections 2024-04-01T01:39:53.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.170+0000 7f4f6d981640 1 -- 172.21.15.70:0/1844848435 wait complete. 2024-04-01T01:39:53.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.171+0000 7f4f6d981640 1 Processor -- start 2024-04-01T01:39:53.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.171+0000 7f4f6d981640 1 -- start start 2024-04-01T01:39:53.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.171+0000 7f4f6d981640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f68072b30 0x7f4f681a6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.171+0000 7f4f6d981640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f68108e60 0x7f4f681a6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.171+0000 7f4f6d981640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f68109800 0x7f4f681adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.171+0000 7f4f6d981640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6811c020 con 0x7f4f68109800 2024-04-01T01:39:53.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.171+0000 7f4f6d981640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6811bea0 con 0x7f4f68072b30 2024-04-01T01:39:53.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.171+0000 7f4f6d981640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6811c1a0 con 0x7f4f68108e60 2024-04-01T01:39:53.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.172+0000 7f4f677fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f68109800 0x7f4f681adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.172+0000 7f4f677fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f68109800 0x7f4f681adbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44464/0 (socket says 172.21.15.70:44464) 2024-04-01T01:39:53.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.172+0000 7f4f677fe640 1 -- 172.21.15.70:0/1083150223 learned_addr learned my addr 172.21.15.70:0/1083150223 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:53.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.173+0000 7f4f677fe640 1 -- 172.21.15.70:0/1083150223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f68108e60 msgr2=0x7f4f681a6b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:39:53.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.173+0000 7f4f677fe640 1 --2- 172.21.15.70:0/1083150223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f68108e60 0x7f4f681a6b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.173+0000 7f4f677fe640 1 -- 172.21.15.70:0/1083150223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f68072b30 msgr2=0x7f4f681a6640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:39:53.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.173+0000 7f4f677fe640 1 --2- 172.21.15.70:0/1083150223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f68072b30 0x7f4f681a6640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.173+0000 7f4f677fe640 1 -- 172.21.15.70:0/1083150223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f681ae260 con 0x7f4f68109800 2024-04-01T01:39:53.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.173+0000 7f4f677fe640 1 --2- 172.21.15.70:0/1083150223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f68109800 0x7f4f681adbf0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0057c0 tx=0x7f4f5c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.174+0000 7f4f6c97f640 1 -- 172.21.15.70:0/1083150223 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c039050 con 0x7f4f68109800 2024-04-01T01:39:53.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.174+0000 7f4f6c97f640 1 -- 172.21.15.70:0/1083150223 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f5c045070 con 0x7f4f68109800 2024-04-01T01:39:53.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.174+0000 7f4f6c97f640 1 -- 172.21.15.70:0/1083150223 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c040450 con 0x7f4f68109800 2024-04-01T01:39:53.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.174+0000 7f4f6d981640 1 -- 172.21.15.70:0/1083150223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f681ae4f0 con 0x7f4f68109800 2024-04-01T01:39:53.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.174+0000 7f4f6d981640 1 -- 172.21.15.70:0/1083150223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f681ae990 con 0x7f4f68109800 2024-04-01T01:39:53.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.178+0000 7f4f70563740 1 Processor -- start 2024-04-01T01:39:53.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.179+0000 7f4f6c97f640 1 -- 172.21.15.70:0/1083150223 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f4f5c0035f0 con 0x7f4f68109800 2024-04-01T01:39:53.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.179+0000 7f4f6c97f640 1 --2- 172.21.15.70:0/1083150223 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4f500a25f0 0x7f4f500a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.180+0000 7f4f70563740 1 -- start start 2024-04-01T01:39:53.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.180+0000 7f4f70563740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f5456a0 0x561f4f545ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.180+0000 7f4f70563740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f546090 0x561f4f54ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.180+0000 7f4f70563740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f54d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.180+0000 7f4f70563740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561f4f4aa1b0 con 0x561f4f5456a0 2024-04-01T01:39:53.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.180+0000 7f4f70563740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561f4f452c00 con 0x561f4f546090 2024-04-01T01:39:53.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.180+0000 7f4f70563740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561f4f4a6250 con 0x561f4f54b200 2024-04-01T01:39:53.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.183+0000 7f4f6c97f640 1 -- 172.21.15.70:0/1083150223 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f4f5c0ed800 con 0x7f4f68109800 2024-04-01T01:39:53.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.183+0000 7f4f677fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f54d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.183+0000 7f4f677fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f54d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49684/0 (socket says 172.21.15.70:49684) 2024-04-01T01:39:53.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.183+0000 7f4f667fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f546090 0x561f4f54ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.183+0000 7f4f667fc640 1 --2- 172.21.15.70:0/1083150223 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4f500a25f0 0x7f4f500a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.183+0000 7f4f677fe640 1 -- 172.21.15.70:0/1868766619 learned_addr learned my addr 172.21.15.70:0/1868766619 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:53.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.183+0000 7f4f667fc640 1 --2- 172.21.15.70:0/1868766619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f546090 0x561f4f54ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37436/0 (socket says 172.21.15.70:37436) 2024-04-01T01:39:53.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.184+0000 7f4f667fc640 1 -- 172.21.15.70:0/1868766619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 msgr2=0x561f4f54d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.184+0000 7f4f667fc640 1 --2- 172.21.15.70:0/1868766619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f54d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.184+0000 7f4f667fc640 1 -- 172.21.15.70:0/1868766619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f5456a0 msgr2=0x561f4f545ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.184+0000 7f4f667fc640 1 --2- 172.21.15.70:0/1868766619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f5456a0 0x561f4f545ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.184+0000 7f4f667fc640 1 -- 172.21.15.70:0/1868766619 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f4f4723d0 con 0x561f4f546090 2024-04-01T01:39:53.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.184+0000 7f4f667fc640 1 --2- 172.21.15.70:0/1868766619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f546090 0x561f4f54ab90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009330 tx=0x7f4f6000d7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30742bb773f43e8b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.185+0000 7f4f667fc640 1 --2- 172.21.15.70:0/1083150223 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4f500a25f0 0x7f4f500a4ab0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4f681a7b50 tx=0x7f4f60017000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.185+0000 7f4f4dffb640 1 -- 172.21.15.70:0/1868766619 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6000bde0 con 0x561f4f546090 2024-04-01T01:39:53.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.185+0000 7f4f4dffb640 1 -- 172.21.15.70:0/1868766619 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f60003970 con 0x561f4f546090 2024-04-01T01:39:53.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.185+0000 7f4f4dffb640 1 -- 172.21.15.70:0/1868766619 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f60004250 con 0x561f4f546090 2024-04-01T01:39:53.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.186+0000 7f4f70563740 1 -- 172.21.15.70:0/1868766619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f546090 msgr2=0x561f4f54ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.186+0000 7f4f70563740 1 --2- 172.21.15.70:0/1868766619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f546090 0x561f4f54ab90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009330 tx=0x7f4f6000d7b0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.187+0000 7f4f70563740 1 -- 172.21.15.70:0/1868766619 shutdown_connections 2024-04-01T01:39:53.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.187+0000 7f4f70563740 1 --2- 172.21.15.70:0/1868766619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f54d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.187+0000 7f4f70563740 1 --2- 172.21.15.70:0/1868766619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f546090 0x561f4f54ab90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.187+0000 7f4f70563740 1 --2- 172.21.15.70:0/1868766619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f5456a0 0x561f4f545ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.187+0000 7f4f70563740 1 -- 172.21.15.70:0/1868766619 >> 172.21.15.70:0/1868766619 conn(0x561f4f540c10 msgr2=0x561f4f541010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.194+0000 7f4f70563740 1 -- 172.21.15.70:0/1868766619 shutdown_connections 2024-04-01T01:39:53.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.198+0000 7f4f70563740 1 -- 172.21.15.70:0/1868766619 wait complete. 2024-04-01T01:39:53.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.198+0000 7f4f70563740 1 -- start start 2024-04-01T01:39:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.198+0000 7f4f70563740 1 Processor -- start 2024-04-01T01:39:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.199+0000 7f4f70563740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f5456a0 0x561f4f560ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.199+0000 7f4f70563740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f546090 0x561f4f5611e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.199+0000 7f4f70563740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f568220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.199+0000 7f4f70563740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561f4f54e150 con 0x561f4f546090 2024-04-01T01:39:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.199+0000 7f4f70563740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561f4f448470 con 0x561f4f5456a0 2024-04-01T01:39:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.199+0000 7f4f70563740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561f4f54e2d0 con 0x561f4f54b200 2024-04-01T01:39:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.199+0000 7f4f70563740 10 client.0 _ll_register_callbacks cb 0x561f4f465cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.199+0000 7f4f66ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f5456a0 0x561f4f560ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.199+0000 7f4f66ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f5456a0 0x561f4f560ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37452/0 (socket says 172.21.15.70:37452) 2024-04-01T01:39:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.199+0000 7f4f66ffd640 1 -- 172.21.15.70:0/1601179409 learned_addr learned my addr 172.21.15.70:0/1601179409 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:53.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.199+0000 7f4f677fe640 1 --2- 172.21.15.70:0/1601179409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f568220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.200+0000 7f4f66ffd640 1 -- 172.21.15.70:0/1601179409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 msgr2=0x561f4f568220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.200+0000 7f4f66ffd640 1 --2- 172.21.15.70:0/1601179409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f568220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.200+0000 7f4f66ffd640 1 -- 172.21.15.70:0/1601179409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f546090 msgr2=0x561f4f5611e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.200+0000 7f4f66ffd640 1 --2- 172.21.15.70:0/1601179409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f546090 0x561f4f5611e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.200+0000 7f4f66ffd640 1 -- 172.21.15.70:0/1601179409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f4f1be3f0 con 0x561f4f5456a0 2024-04-01T01:39:53.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.200+0000 7f4f66ffd640 1 --2- 172.21.15.70:0/1601179409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f5456a0 0x561f4f560ca0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4f58009fd0 tx=0x7f4f5800e7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f683da74bafe0f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.200+0000 7f4f46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:53.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.201+0000 7f4f46ffd640 1 -- 172.21.15.70:0/1601179409 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f58010840 con 0x561f4f5456a0 2024-04-01T01:39:53.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.201+0000 7f4f70563740 1 -- 172.21.15.70:0/1601179409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f60003970 con 0x561f4f5456a0 2024-04-01T01:39:53.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.201+0000 7f4f46ffd640 1 -- 172.21.15.70:0/1601179409 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f4f580109e0 con 0x561f4f5456a0 2024-04-01T01:39:53.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.201+0000 7f4f46ffd640 1 -- 172.21.15.70:0/1601179409 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4f58023050 con 0x561f4f5456a0 2024-04-01T01:39:53.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.201+0000 7f4f70563740 10 client.27332 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:53.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.201+0000 7f4f70563740 1 -- 172.21.15.70:0/1601179409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561f4f2887d0 con 0x561f4f5456a0 2024-04-01T01:39:53.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.202+0000 7f4f46ffd640 1 -- 172.21.15.70:0/1601179409 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f60003970 con 0x561f4f5456a0 2024-04-01T01:39:53.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.202+0000 7f4f46ffd640 1 -- 172.21.15.70:0/1601179409 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f4f580173d0 con 0x561f4f5456a0 2024-04-01T01:39:53.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.202+0000 7f4f70563740 10 client.27332 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:53.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.202+0000 7f4f70563740 10 client.27332 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:39:53.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.202+0000 7f4f70563740 20 client.27332 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:53.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.202+0000 7f4f70563740 1 --2- 172.21.15.70:0/1601179409 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x561f4f552b40 0x561f4f554fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.202+0000 7f4f70563740 4 client.27332 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:53.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.202+0000 7f4f70563740 1 -- 172.21.15.70:0/1601179409 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f4f4b76f0 con 0x561f4f552b40 2024-04-01T01:39:53.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.203+0000 7f4f667fc640 1 --2- 172.21.15.70:0/1601179409 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x561f4f552b40 0x561f4f554fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.204+0000 7f4f667fc640 1 --2- 172.21.15.70:0/1601179409 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x561f4f552b40 0x561f4f554fa0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=35592db28e2a471e server_cookie=3066c7d795f9bfbd in_seq=0 out_seq=0 2024-04-01T01:39:53.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.204+0000 7f4f46ffd640 10 client.27332 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:39:53.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.208+0000 7f4f46ffd640 1 -- 172.21.15.70:0/1601179409 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561f4f4b76f0 con 0x561f4f552b40 2024-04-01T01:39:53.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.208+0000 7f4f46ffd640 10 client.27332 handle_command_reply: tid=0 2024-04-01T01:39:53.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.209+0000 7f4f70563740 1 client.27332 shutdown 2024-04-01T01:39:53.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.211+0000 7f4f70563740 1 -- 172.21.15.70:0/1601179409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f5456a0 msgr2=0x561f4f560ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.211+0000 7f4f70563740 1 --2- 172.21.15.70:0/1601179409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f5456a0 0x561f4f560ca0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4f58009fd0 tx=0x7f4f5800e7e0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.212+0000 7f4f70563740 1 -- 172.21.15.70:0/1601179409 shutdown_connections 2024-04-01T01:39:53.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.212+0000 7f4f70563740 1 --2- 172.21.15.70:0/1601179409 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x561f4f552b40 0x561f4f554fa0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.212+0000 7f4f70563740 1 --2- 172.21.15.70:0/1601179409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f568220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.212+0000 7f4f70563740 1 --2- 172.21.15.70:0/1601179409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f546090 0x561f4f5611e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.212+0000 7f4f70563740 1 --2- 172.21.15.70:0/1601179409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f5456a0 0x561f4f560ca0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.212+0000 7f4f70563740 1 -- 172.21.15.70:0/1601179409 >> 172.21.15.70:0/1601179409 conn(0x561f4f540c10 msgr2=0x561f4f53ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.212+0000 7f4f46ffd640 0 client.27332 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:39:53.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.213+0000 7f4f70563740 1 -- 172.21.15.70:0/1601179409 shutdown_connections 2024-04-01T01:39:53.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.214+0000 7f4f70563740 1 -- 172.21.15.70:0/1601179409 wait complete. 2024-04-01T01:39:53.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.215+0000 7f4f70563740 20 client.27332 trim_cache size 0 max 16384 2024-04-01T01:39:53.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.248+0000 7f4f70563740 1 Processor -- start 2024-04-01T01:39:53.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.248+0000 7f4f70563740 1 -- start start 2024-04-01T01:39:53.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:52 smithi189 ceph-mon[35753]: pgmap v551: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 15 MiB/s wr, 916 op/s 2024-04-01T01:39:53.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.249+0000 7f4f70563740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f5456a0 0x561f4f544580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.249+0000 7f4f70563740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f546090 0x561f4f5681e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.249+0000 7f4f70563740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f55f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.249+0000 7f4f70563740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4f681b08d0 con 0x561f4f5456a0 2024-04-01T01:39:53.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.249+0000 7f4f70563740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4f681b2410 con 0x561f4f546090 2024-04-01T01:39:53.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.249+0000 7f4f70563740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4f681b0fa0 con 0x561f4f54b200 2024-04-01T01:39:53.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.249+0000 7f4f677fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f55f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.249+0000 7f4f677fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f55f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49708/0 (socket says 172.21.15.70:49708) 2024-04-01T01:39:53.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.250+0000 7f4f677fe640 1 -- 172.21.15.70:0/4070379283 learned_addr learned my addr 172.21.15.70:0/4070379283 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:53.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.252+0000 7f4f66ffd640 1 --2- 172.21.15.70:0/4070379283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f5456a0 0x561f4f544580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.252+0000 7f4f667fc640 1 --2- 172.21.15.70:0/4070379283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f546090 0x561f4f5681e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.252+0000 7f4f677fe640 1 -- 172.21.15.70:0/4070379283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f546090 msgr2=0x561f4f5681e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.252+0000 7f4f677fe640 1 --2- 172.21.15.70:0/4070379283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f546090 0x561f4f5681e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.252+0000 7f4f677fe640 1 -- 172.21.15.70:0/4070379283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f5456a0 msgr2=0x561f4f544580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.252+0000 7f4f677fe640 1 --2- 172.21.15.70:0/4070379283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f5456a0 0x561f4f544580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.252+0000 7f4f677fe640 1 -- 172.21.15.70:0/4070379283 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f4f531e60 con 0x561f4f54b200 2024-04-01T01:39:53.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.253+0000 7f4f677fe640 1 --2- 172.21.15.70:0/4070379283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f55f8d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c005940 tx=0x7f4f5c0c1920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=349474a2aa9997b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.255+0000 7f4f4d7fa640 1 -- 172.21.15.70:0/4070379283 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c0b90e0 con 0x561f4f54b200 2024-04-01T01:39:53.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:52 smithi070 ceph-mon[32179]: pgmap v551: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 15 MiB/s wr, 916 op/s 2024-04-01T01:39:53.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.255+0000 7f4f4d7fa640 1 -- 172.21.15.70:0/4070379283 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f5c03a040 con 0x561f4f54b200 2024-04-01T01:39:53.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.255+0000 7f4f4d7fa640 1 -- 172.21.15.70:0/4070379283 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c0b9280 con 0x561f4f54b200 2024-04-01T01:39:53.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.257+0000 7f4f70563740 1 -- 172.21.15.70:0/4070379283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 msgr2=0x561f4f55f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.257+0000 7f4f70563740 1 --2- 172.21.15.70:0/4070379283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f55f8d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c005940 tx=0x7f4f5c0c1920 comp rx=0 tx=0).stop 2024-04-01T01:39:53.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.262+0000 7f4f70563740 1 -- 172.21.15.70:0/4070379283 shutdown_connections 2024-04-01T01:39:53.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.262+0000 7f4f70563740 1 --2- 172.21.15.70:0/4070379283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f55f8d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.262+0000 7f4f70563740 1 --2- 172.21.15.70:0/4070379283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f546090 0x561f4f5681e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.262+0000 7f4f70563740 1 --2- 172.21.15.70:0/4070379283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f5456a0 0x561f4f544580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.262+0000 7f4f70563740 1 -- 172.21.15.70:0/4070379283 >> 172.21.15.70:0/4070379283 conn(0x561f4f540c10 msgr2=0x561f4f543e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.264+0000 7f4f70563740 1 -- 172.21.15.70:0/4070379283 shutdown_connections 2024-04-01T01:39:53.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.264+0000 7f4f70563740 1 -- 172.21.15.70:0/4070379283 wait complete. 2024-04-01T01:39:53.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.265+0000 7f4f70563740 1 -- start start 2024-04-01T01:39:53.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.265+0000 7f4f70563740 1 Processor -- start 2024-04-01T01:39:53.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.266+0000 7f4f70563740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f5456a0 0x561f4f565e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.266+0000 7f4f70563740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f546090 0x561f4f5663c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.266+0000 7f4f66ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f5456a0 0x561f4f565e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.266+0000 7f4f66ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f5456a0 0x561f4f565e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37478/0 (socket says 172.21.15.70:37478) 2024-04-01T01:39:53.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.266+0000 7f4f667fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f546090 0x561f4f5663c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.266+0000 7f4f667fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f546090 0x561f4f5663c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44496/0 (socket says 172.21.15.70:44496) 2024-04-01T01:39:53.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.266+0000 7f4f66ffd640 1 -- 172.21.15.70:0/591727938 learned_addr learned my addr 172.21.15.70:0/591727938 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:53.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.266+0000 7f4f70563740 1 --2- 172.21.15.70:0/591727938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f5673d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.267+0000 7f4f70563740 1 -- 172.21.15.70:0/591727938 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4f681b1d40 con 0x561f4f546090 2024-04-01T01:39:53.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.267+0000 7f4f70563740 1 -- 172.21.15.70:0/591727938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4f681b1670 con 0x561f4f5456a0 2024-04-01T01:39:53.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.267+0000 7f4f677fe640 1 --2- 172.21.15.70:0/591727938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f5673d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.267+0000 7f4f70563740 1 -- 172.21.15.70:0/591727938 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4f68118340 con 0x561f4f54b200 2024-04-01T01:39:53.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.267+0000 7f4f66ffd640 1 -- 172.21.15.70:0/591727938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 msgr2=0x561f4f5673d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.267+0000 7f4f66ffd640 1 --2- 172.21.15.70:0/591727938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f5673d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.267+0000 7f4f66ffd640 1 -- 172.21.15.70:0/591727938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f546090 msgr2=0x561f4f5663c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.267+0000 7f4f66ffd640 1 --2- 172.21.15.70:0/591727938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f546090 0x561f4f5663c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.267+0000 7f4f66ffd640 1 -- 172.21.15.70:0/591727938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f4f5679c0 con 0x561f4f5456a0 2024-04-01T01:39:53.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.268+0000 7f4f70563740 10 client.0 _ll_register_callbacks cb 0x7f4f58009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:53.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.268+0000 7f4f66ffd640 1 --2- 172.21.15.70:0/591727938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f5456a0 0x561f4f565e80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4f580057c0 tx=0x7f4f58017690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a6fa880c781150f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.268+0000 7f4f70563740 1 -- 172.21.15.70:0/591727938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f5c03a040 con 0x561f4f5456a0 2024-04-01T01:39:53.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.268+0000 7f4f46ffd640 10 client.27338 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:53.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.268+0000 7f4f46ffd640 1 -- 172.21.15.70:0/591727938 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f58037050 con 0x561f4f5456a0 2024-04-01T01:39:53.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.269+0000 7f4f46ffd640 1 -- 172.21.15.70:0/591727938 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f4f58005040 con 0x561f4f5456a0 2024-04-01T01:39:53.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.269+0000 7f4f46ffd640 1 -- 172.21.15.70:0/591727938 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4f58005260 con 0x561f4f5456a0 2024-04-01T01:39:53.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.275+0000 7f4f70563740 10 client.27338 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:53.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.275+0000 7f4f70563740 1 -- 172.21.15.70:0/591727938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=31}) -- 0x561f4f5e3ac0 con 0x561f4f5456a0 2024-04-01T01:39:53.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.277+0000 7f4f46ffd640 1 -- 172.21.15.70:0/591727938 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f4f58005400 con 0x561f4f5456a0 2024-04-01T01:39:53.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.277+0000 7f4f70563740 10 client.27338 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:53.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.277+0000 7f4f70563740 10 client.27338 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:39:53.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.277+0000 7f4f70563740 20 client.27338 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:53.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.277+0000 7f4f70563740 1 --2- 172.21.15.70:0/591727938 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x561f4f5e4330 0x561f4f5e6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.277+0000 7f4f70563740 4 client.27338 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:39:53.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.277+0000 7f4f70563740 1 -- 172.21.15.70:0/591727938 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f4f5c03a040 con 0x561f4f5e4330 2024-04-01T01:39:53.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.278+0000 7f4f667fc640 1 --2- 172.21.15.70:0/591727938 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x561f4f5e4330 0x561f4f5e6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.280+0000 7f4f667fc640 1 --2- 172.21.15.70:0/591727938 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x561f4f5e4330 0x561f4f5e6750 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9e79ec93b3508ce2 server_cookie=e9472651d730897a in_seq=0 out_seq=0 2024-04-01T01:39:53.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.280+0000 7f4f46ffd640 10 client.27338 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:39:53.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.280+0000 7f4f46ffd640 1 -- 172.21.15.70:0/591727938 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f4f5c03a040 con 0x561f4f5e4330 2024-04-01T01:39:53.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.280+0000 7f4f46ffd640 10 client.27338 handle_command_reply: tid=0 2024-04-01T01:39:53.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.281+0000 7f4f70563740 1 client.27338 shutdown 2024-04-01T01:39:53.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.281+0000 7f4f70563740 1 -- 172.21.15.70:0/591727938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f5456a0 msgr2=0x561f4f565e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.281+0000 7f4f70563740 1 --2- 172.21.15.70:0/591727938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f5456a0 0x561f4f565e80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4f580057c0 tx=0x7f4f58017690 comp rx=0 tx=0).stop 2024-04-01T01:39:53.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.282+0000 7f4f70563740 1 -- 172.21.15.70:0/591727938 shutdown_connections 2024-04-01T01:39:53.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.282+0000 7f4f70563740 1 --2- 172.21.15.70:0/591727938 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x561f4f5e4330 0x561f4f5e6750 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.282+0000 7f4f70563740 1 --2- 172.21.15.70:0/591727938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561f4f54b200 0x561f4f5673d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.282+0000 7f4f70563740 1 --2- 172.21.15.70:0/591727938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f4f546090 0x561f4f5663c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.282+0000 7f4f70563740 1 --2- 172.21.15.70:0/591727938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561f4f5456a0 0x561f4f565e80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.282+0000 7f4f70563740 1 -- 172.21.15.70:0/591727938 >> 172.21.15.70:0/591727938 conn(0x561f4f540c10 msgr2=0x561f4f544700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.282+0000 7f4f46ffd640 0 client.27338 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:39:53.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.282+0000 7f4f70563740 1 -- 172.21.15.70:0/591727938 shutdown_connections 2024-04-01T01:39:53.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.282+0000 7f4f70563740 1 -- 172.21.15.70:0/591727938 wait complete. 2024-04-01T01:39:53.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.282+0000 7f4f70563740 20 client.27338 trim_cache size 0 max 16384 2024-04-01T01:39:53.282 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:39:53.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.287+0000 7f4f6d981640 1 -- 172.21.15.70:0/1083150223 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4f500a25f0 msgr2=0x7f4f500a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.287+0000 7f4f6d981640 1 --2- 172.21.15.70:0/1083150223 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4f500a25f0 0x7f4f500a4ab0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4f681a7b50 tx=0x7f4f60017000 comp rx=0 tx=0).stop 2024-04-01T01:39:53.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.287+0000 7f4f6d981640 1 -- 172.21.15.70:0/1083150223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f68109800 msgr2=0x7f4f681adbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.287+0000 7f4f6d981640 1 --2- 172.21.15.70:0/1083150223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f68109800 0x7f4f681adbf0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0057c0 tx=0x7f4f5c0042f0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.287+0000 7f4f6d981640 1 -- 172.21.15.70:0/1083150223 shutdown_connections 2024-04-01T01:39:53.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.287+0000 7f4f6d981640 1 --2- 172.21.15.70:0/1083150223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f68109800 0x7f4f681adbf0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.287+0000 7f4f6d981640 1 --2- 172.21.15.70:0/1083150223 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4f500a25f0 0x7f4f500a4ab0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.287+0000 7f4f6d981640 1 --2- 172.21.15.70:0/1083150223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f68108e60 0x7f4f681a6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.287+0000 7f4f6d981640 1 --2- 172.21.15.70:0/1083150223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f68072b30 0x7f4f681a6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.288+0000 7f4f6d981640 1 -- 172.21.15.70:0/1083150223 >> 172.21.15.70:0/1083150223 conn(0x7f4f6806e630 msgr2=0x7f4f68070240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.288+0000 7f4f6d981640 1 -- 172.21.15.70:0/1083150223 shutdown_connections 2024-04-01T01:39:53.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.288+0000 7f4f6d981640 1 -- 172.21.15.70:0/1083150223 wait complete. 2024-04-01T01:39:53.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:39:53.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:52 smithi081 ceph-mon[35142]: pgmap v551: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 15 MiB/s wr, 916 op/s 2024-04-01T01:39:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.509+0000 7f93f584e640 1 -- 172.21.15.70:0/2018597557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93f0107d70 msgr2=0x7f93f0108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.509+0000 7f93f584e640 1 --2- 172.21.15.70:0/2018597557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93f0107d70 0x7f93f0108190 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f93e8009ca0 tx=0x7f93e802c140 comp rx=0 tx=0).stop 2024-04-01T01:39:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.509+0000 7f93f584e640 1 -- 172.21.15.70:0/2018597557 shutdown_connections 2024-04-01T01:39:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.509+0000 7f93f584e640 1 --2- 172.21.15.70:0/2018597557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93f0109920 0x7f93f01101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.509+0000 7f93f584e640 1 --2- 172.21.15.70:0/2018597557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93f0108f60 0x7f93f01093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.509+0000 7f93f584e640 1 --2- 172.21.15.70:0/2018597557 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93f0107d70 0x7f93f0108190 secure :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f93e8009ca0 tx=0x7f93e802c140 comp rx=0 tx=0).stop 2024-04-01T01:39:53.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.509+0000 7f93f584e640 1 -- 172.21.15.70:0/2018597557 >> 172.21.15.70:0/2018597557 conn(0x7f93f0076d10 msgr2=0x7f93f0079150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.512+0000 7f93f584e640 1 -- 172.21.15.70:0/2018597557 shutdown_connections 2024-04-01T01:39:53.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.514+0000 7f93f584e640 1 -- 172.21.15.70:0/2018597557 wait complete. 2024-04-01T01:39:53.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.515+0000 7f93f584e640 1 Processor -- start 2024-04-01T01:39:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.515+0000 7f93f584e640 1 -- start start 2024-04-01T01:39:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.515+0000 7f93f584e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93f0108f60 0x7f93f00703c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.515+0000 7f93f584e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93f0109920 0x7f93f0070900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.515+0000 7f93f584e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93f01abf70 0x7f93f01ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.515+0000 7f93f584e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f93f0113870 con 0x7f93f0109920 2024-04-01T01:39:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.515+0000 7f93f584e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f93f01136f0 con 0x7f93f01abf70 2024-04-01T01:39:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.515+0000 7f93f584e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f93f01139f0 con 0x7f93f0108f60 2024-04-01T01:39:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.516+0000 7f93eeffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93f0108f60 0x7f93f00703c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.516+0000 7f93eeffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93f0108f60 0x7f93f00703c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49738/0 (socket says 172.21.15.70:49738) 2024-04-01T01:39:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.516+0000 7f93eeffd640 1 -- 172.21.15.70:0/2469294167 learned_addr learned my addr 172.21.15.70:0/2469294167 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:53.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.516+0000 7f93ef7fe640 1 --2- 172.21.15.70:0/2469294167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93f01abf70 0x7f93f01ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.516+0000 7f93ee7fc640 1 --2- 172.21.15.70:0/2469294167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93f0109920 0x7f93f0070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.517+0000 7f93eeffd640 1 -- 172.21.15.70:0/2469294167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93f01abf70 msgr2=0x7f93f01ae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.517+0000 7f93eeffd640 1 --2- 172.21.15.70:0/2469294167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93f01abf70 0x7f93f01ae360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.517+0000 7f93eeffd640 1 -- 172.21.15.70:0/2469294167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93f0109920 msgr2=0x7f93f0070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.517+0000 7f93eeffd640 1 --2- 172.21.15.70:0/2469294167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93f0109920 0x7f93f0070900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.517+0000 7f93eeffd640 1 -- 172.21.15.70:0/2469294167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f007a0d0 con 0x7f93f0108f60 2024-04-01T01:39:53.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.518+0000 7f93eeffd640 1 --2- 172.21.15.70:0/2469294167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93f0108f60 0x7f93f00703c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f93e8032440 tx=0x7f93e8003c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.518+0000 7f93ef7fe640 1 --2- 172.21.15.70:0/2469294167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93f01abf70 0x7f93f01ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:53.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.518+0000 7f93ee7fc640 1 --2- 172.21.15.70:0/2469294167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93f0109920 0x7f93f0070900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:53.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.521+0000 7f93f484c640 1 -- 172.21.15.70:0/2469294167 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e8042070 con 0x7f93f0108f60 2024-04-01T01:39:53.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.521+0000 7f93f484c640 1 -- 172.21.15.70:0/2469294167 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93e8003e10 con 0x7f93f0108f60 2024-04-01T01:39:53.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.521+0000 7f93f484c640 1 -- 172.21.15.70:0/2469294167 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e8039070 con 0x7f93f0108f60 2024-04-01T01:39:53.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.522+0000 7f93f584e640 1 -- 172.21.15.70:0/2469294167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93f007a360 con 0x7f93f0108f60 2024-04-01T01:39:53.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.522+0000 7f93f584e640 1 -- 172.21.15.70:0/2469294167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93f007a800 con 0x7f93f0108f60 2024-04-01T01:39:53.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.526+0000 7f93f8432740 1 Processor -- start 2024-04-01T01:39:53.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.527+0000 7f93f8432740 1 -- start start 2024-04-01T01:39:53.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.527+0000 7f93f8432740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5da490 0x55934f5da8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.527+0000 7f93f8432740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5dae80 0x55934f5df980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.527+0000 7f93f8432740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dfff0 0x55934f5e2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.527+0000 7f93f8432740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55934f53f1b0 con 0x55934f5dae80 2024-04-01T01:39:53.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.527+0000 7f93f8432740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55934f4e7c00 con 0x55934f5dfff0 2024-04-01T01:39:53.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.527+0000 7f93f8432740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55934f53b250 con 0x55934f5da490 2024-04-01T01:39:53.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.528+0000 7f93f484c640 1 -- 172.21.15.70:0/2469294167 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f93e8045070 con 0x7f93f0108f60 2024-04-01T01:39:53.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.529+0000 7f93f484c640 1 --2- 172.21.15.70:0/2469294167 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93d80a25f0 0x7f93d80a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.529+0000 7f93f484c640 1 -- 172.21.15.70:0/2469294167 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f93e8033e60 con 0x7f93f0108f60 2024-04-01T01:39:53.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.530+0000 7f93eeffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5dae80 0x55934f5df980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.530+0000 7f93eeffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5dae80 0x55934f5df980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44520/0 (socket says 172.21.15.70:44520) 2024-04-01T01:39:53.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.530+0000 7f93eeffd640 1 -- 172.21.15.70:0/1498172200 learned_addr learned my addr 172.21.15.70:0/1498172200 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:53.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.531+0000 7f93ee7fc640 1 --2- 172.21.15.70:0/1498172200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dfff0 0x55934f5e2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.531+0000 7f93ef7fe640 1 --2- 172.21.15.70:0/1498172200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5da490 0x55934f5da8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.532+0000 7f93ef7fe640 1 --2- 172.21.15.70:0/2469294167 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93d80a25f0 0x7f93d80a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.532+0000 7f93ee7fc640 1 -- 172.21.15.70:0/1498172200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5da490 msgr2=0x55934f5da8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.532+0000 7f93ee7fc640 1 --2- 172.21.15.70:0/1498172200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5da490 0x55934f5da8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.532+0000 7f93ee7fc640 1 -- 172.21.15.70:0/1498172200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5dae80 msgr2=0x55934f5df980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.532+0000 7f93ee7fc640 1 --2- 172.21.15.70:0/1498172200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5dae80 0x55934f5df980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.532+0000 7f93ee7fc640 1 -- 172.21.15.70:0/1498172200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55934f5073d0 con 0x55934f5dfff0 2024-04-01T01:39:53.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.535+0000 7f93ee7fc640 1 --2- 172.21.15.70:0/1498172200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dfff0 0x55934f5e2480 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f93e4005fd0 tx=0x7f93e400f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ec2f659286b6394 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.535+0000 7f93ef7fe640 1 --2- 172.21.15.70:0/2469294167 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93d80a25f0 0x7f93d80a4ab0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f93e0005fd0 tx=0x7f93e000d720 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.536+0000 7f93d5ffb640 1 -- 172.21.15.70:0/1498172200 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e4005020 con 0x55934f5dfff0 2024-04-01T01:39:53.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.536+0000 7f93d5ffb640 1 -- 172.21.15.70:0/1498172200 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93e40047a0 con 0x55934f5dfff0 2024-04-01T01:39:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.537+0000 7f93f8432740 1 -- 172.21.15.70:0/1498172200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dfff0 msgr2=0x55934f5e2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.537+0000 7f93f8432740 1 --2- 172.21.15.70:0/1498172200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dfff0 0x55934f5e2480 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f93e4005fd0 tx=0x7f93e400f730 comp rx=0 tx=0).stop 2024-04-01T01:39:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.538+0000 7f93f8432740 1 -- 172.21.15.70:0/1498172200 shutdown_connections 2024-04-01T01:39:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.538+0000 7f93f8432740 1 --2- 172.21.15.70:0/1498172200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dfff0 0x55934f5e2480 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.538+0000 7f93f8432740 1 --2- 172.21.15.70:0/1498172200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5dae80 0x55934f5df980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.538+0000 7f93f8432740 1 --2- 172.21.15.70:0/1498172200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5da490 0x55934f5da8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.538+0000 7f93f8432740 1 -- 172.21.15.70:0/1498172200 >> 172.21.15.70:0/1498172200 conn(0x55934f5d5a00 msgr2=0x55934f5d5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.541+0000 7f93f8432740 1 -- 172.21.15.70:0/1498172200 shutdown_connections 2024-04-01T01:39:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.542+0000 7f93f8432740 1 -- 172.21.15.70:0/1498172200 wait complete. 2024-04-01T01:39:53.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.542+0000 7f93f8432740 1 -- start start 2024-04-01T01:39:53.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.543+0000 7f93f8432740 1 Processor -- start 2024-04-01T01:39:53.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.545+0000 7f93f8432740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5da490 0x55934f5e49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.545+0000 7f93f8432740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 0x55934f5e4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.545+0000 7f93f8432740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5dfff0 0x55934f5ebed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.545+0000 7f93f8432740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55934f5e2ff0 con 0x55934f5da490 2024-04-01T01:39:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.545+0000 7f93f8432740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55934f4dd470 con 0x55934f5dae80 2024-04-01T01:39:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.545+0000 7f93f8432740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55934f5e3170 con 0x55934f5dfff0 2024-04-01T01:39:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.545+0000 7f93f8432740 10 client.0 _ll_register_callbacks cb 0x55934f4facb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.546+0000 7f93ef7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5dfff0 0x55934f5ebed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.546+0000 7f93ef7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5dfff0 0x55934f5ebed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49758/0 (socket says 172.21.15.70:49758) 2024-04-01T01:39:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.546+0000 7f93ee7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 0x55934f5e4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.546+0000 7f93ee7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 0x55934f5e4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37514/0 (socket says 172.21.15.70:37514) 2024-04-01T01:39:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.546+0000 7f93ef7fe640 1 -- 172.21.15.70:0/2804972809 learned_addr learned my addr 172.21.15.70:0/2804972809 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.546+0000 7f93eeffd640 1 --2- 172.21.15.70:0/2804972809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5da490 0x55934f5e49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.547+0000 7f93ee7fc640 1 -- 172.21.15.70:0/2804972809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5dfff0 msgr2=0x55934f5ebed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.547+0000 7f93ee7fc640 1 --2- 172.21.15.70:0/2804972809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5dfff0 0x55934f5ebed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.547+0000 7f93ee7fc640 1 -- 172.21.15.70:0/2804972809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5da490 msgr2=0x55934f5e49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.547+0000 7f93ee7fc640 1 --2- 172.21.15.70:0/2804972809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5da490 0x55934f5e49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.547+0000 7f93ee7fc640 1 -- 172.21.15.70:0/2804972809 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55934f2533f0 con 0x55934f5dae80 2024-04-01T01:39:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.547+0000 7f93ef7fe640 1 --2- 172.21.15.70:0/2804972809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5dfff0 0x55934f5ebed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.548+0000 7f93ee7fc640 1 --2- 172.21.15.70:0/2804972809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 0x55934f5e4f00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x55934f5d6970 tx=0x7f93e400b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7da00343325d0e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.549+0000 7f93ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.549+0000 7f93ceffd640 1 -- 172.21.15.70:0/2804972809 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e402a060 con 0x55934f5dae80 2024-04-01T01:39:53.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.549+0000 7f93ceffd640 1 -- 172.21.15.70:0/2804972809 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f93e4004b60 con 0x55934f5dae80 2024-04-01T01:39:53.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.550+0000 7f93f8432740 1 -- 172.21.15.70:0/2804972809 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93e40047a0 con 0x55934f5dae80 2024-04-01T01:39:53.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.553+0000 7f93ceffd640 1 -- 172.21.15.70:0/2804972809 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f93e4011900 con 0x55934f5dae80 2024-04-01T01:39:53.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.553+0000 7f93f8432740 10 client.27350 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:53.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.553+0000 7f93f8432740 1 -- 172.21.15.70:0/2804972809 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55934f31d7d0 con 0x55934f5dae80 2024-04-01T01:39:53.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.554+0000 7f93ceffd640 1 -- 172.21.15.70:0/2804972809 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93e40047a0 con 0x55934f5dae80 2024-04-01T01:39:53.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.554+0000 7f93ceffd640 1 -- 172.21.15.70:0/2804972809 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f93e4025410 con 0x55934f5dae80 2024-04-01T01:39:53.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.554+0000 7f93f8432740 10 client.27350 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:53.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.554+0000 7f93f8432740 10 client.27350 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:39:53.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.554+0000 7f93f8432740 20 client.27350 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:53.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.554+0000 7f93f8432740 1 --2- 172.21.15.70:0/2804972809 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55934f5f00e0 0x55934f5f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.554+0000 7f93f8432740 4 client.27350 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:53.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.554+0000 7f93f8432740 1 -- 172.21.15.70:0/2804972809 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55934f54c6f0 con 0x55934f5f00e0 2024-04-01T01:39:53.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.554+0000 7f93eeffd640 1 --2- 172.21.15.70:0/2804972809 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55934f5f00e0 0x55934f5f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.555+0000 7f93eeffd640 1 --2- 172.21.15.70:0/2804972809 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55934f5f00e0 0x55934f5f24d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7d9f7b6cec4cb5 server_cookie=e0744ce864fcab6e in_seq=0 out_seq=0 2024-04-01T01:39:53.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.555+0000 7f93ceffd640 10 client.27350 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:39:53.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.556+0000 7f93ceffd640 1 -- 172.21.15.70:0/2804972809 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55934f54c6f0 con 0x55934f5f00e0 2024-04-01T01:39:53.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.556+0000 7f93ceffd640 10 client.27350 handle_command_reply: tid=0 2024-04-01T01:39:53.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.556+0000 7f93f8432740 1 client.27350 shutdown 2024-04-01T01:39:53.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.557+0000 7f93f8432740 1 -- 172.21.15.70:0/2804972809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 msgr2=0x55934f5e4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.557+0000 7f93f8432740 1 --2- 172.21.15.70:0/2804972809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 0x55934f5e4f00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x55934f5d6970 tx=0x7f93e400b8c0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.557+0000 7f93f8432740 1 -- 172.21.15.70:0/2804972809 shutdown_connections 2024-04-01T01:39:53.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.557+0000 7f93f8432740 1 --2- 172.21.15.70:0/2804972809 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55934f5f00e0 0x55934f5f24d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.557+0000 7f93f8432740 1 --2- 172.21.15.70:0/2804972809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5dfff0 0x55934f5ebed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.557+0000 7f93f8432740 1 --2- 172.21.15.70:0/2804972809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 0x55934f5e4f00 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.557+0000 7f93ceffd640 0 client.27350 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:39:53.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.557+0000 7f93f8432740 1 --2- 172.21.15.70:0/2804972809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5da490 0x55934f5e49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.557+0000 7f93f8432740 1 -- 172.21.15.70:0/2804972809 >> 172.21.15.70:0/2804972809 conn(0x55934f5d5a00 msgr2=0x55934f5d4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.557+0000 7f93f8432740 1 -- 172.21.15.70:0/2804972809 shutdown_connections 2024-04-01T01:39:53.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.557+0000 7f93f8432740 1 -- 172.21.15.70:0/2804972809 wait complete. 2024-04-01T01:39:53.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.557+0000 7f93f8432740 20 client.27350 trim_cache size 0 max 16384 2024-04-01T01:39:53.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.573+0000 7f93f8432740 1 Processor -- start 2024-04-01T01:39:53.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.573+0000 7f93f8432740 1 -- start start 2024-04-01T01:39:53.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.573+0000 7f93f8432740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5da490 0x55934f5db670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93f8432740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 0x55934f5ebe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93f8432740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5dfff0 0x55934f5d9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93f8432740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f93f01c20c0 con 0x55934f5da490 2024-04-01T01:39:53.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93f8432740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f93f01c3c00 con 0x55934f5dae80 2024-04-01T01:39:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93f8432740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f93f01c2790 con 0x55934f5dfff0 2024-04-01T01:39:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93eeffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5da490 0x55934f5db670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93eeffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5da490 0x55934f5db670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44536/0 (socket says 172.21.15.70:44536) 2024-04-01T01:39:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93eeffd640 1 -- 172.21.15.70:0/1773684989 learned_addr learned my addr 172.21.15.70:0/1773684989 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93ee7fc640 1 --2- 172.21.15.70:0/1773684989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 0x55934f5ebe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93ef7fe640 1 --2- 172.21.15.70:0/1773684989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5dfff0 0x55934f5d9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93eeffd640 1 -- 172.21.15.70:0/1773684989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5dfff0 msgr2=0x55934f5d9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93eeffd640 1 --2- 172.21.15.70:0/1773684989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5dfff0 0x55934f5d9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93eeffd640 1 -- 172.21.15.70:0/1773684989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 msgr2=0x55934f5ebe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93eeffd640 1 --2- 172.21.15.70:0/1773684989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 0x55934f5ebe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93eeffd640 1 -- 172.21.15.70:0/1773684989 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55934f5db260 con 0x55934f5da490 2024-04-01T01:39:53.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93ef7fe640 1 --2- 172.21.15.70:0/1773684989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5dfff0 0x55934f5d9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:53.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.574+0000 7f93eeffd640 1 --2- 172.21.15.70:0/1773684989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5da490 0x55934f5db670 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f93e8004af0 tx=0x7f93e80c4d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705a515fb1c11f98 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.575+0000 7f93d57fa640 1 -- 172.21.15.70:0/1773684989 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e80ba340 con 0x55934f5da490 2024-04-01T01:39:53.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.575+0000 7f93d57fa640 1 -- 172.21.15.70:0/1773684989 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93e802ed60 con 0x55934f5da490 2024-04-01T01:39:53.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.575+0000 7f93d57fa640 1 -- 172.21.15.70:0/1773684989 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e80ba4e0 con 0x55934f5da490 2024-04-01T01:39:53.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.576+0000 7f93f8432740 1 -- 172.21.15.70:0/1773684989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5da490 msgr2=0x55934f5db670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.576+0000 7f93f8432740 1 --2- 172.21.15.70:0/1773684989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5da490 0x55934f5db670 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f93e8004af0 tx=0x7f93e80c4d70 comp rx=0 tx=0).stop 2024-04-01T01:39:53.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.576+0000 7f93f8432740 1 -- 172.21.15.70:0/1773684989 shutdown_connections 2024-04-01T01:39:53.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.576+0000 7f93f8432740 1 --2- 172.21.15.70:0/1773684989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5dfff0 0x55934f5d9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.576+0000 7f93f8432740 1 --2- 172.21.15.70:0/1773684989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 0x55934f5ebe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.576+0000 7f93f8432740 1 --2- 172.21.15.70:0/1773684989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5da490 0x55934f5db670 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.576+0000 7f93f8432740 1 -- 172.21.15.70:0/1773684989 >> 172.21.15.70:0/1773684989 conn(0x55934f5d5a00 msgr2=0x55934f5d8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.576+0000 7f93f8432740 1 -- 172.21.15.70:0/1773684989 shutdown_connections 2024-04-01T01:39:53.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.576+0000 7f93f8432740 1 -- 172.21.15.70:0/1773684989 wait complete. 2024-04-01T01:39:53.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.576+0000 7f93f8432740 1 -- start start 2024-04-01T01:39:53.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.576+0000 7f93f8432740 1 Processor -- start 2024-04-01T01:39:53.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93f8432740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5da490 0x55934f5ebc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93f8432740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 0x55934f5ea3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93f8432740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5dfff0 0x55934f5ea900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93eeffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5da490 0x55934f5ebc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93f8432740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f93f01c3530 con 0x55934f5dfff0 2024-04-01T01:39:53.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93f8432740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f93f01c2e60 con 0x55934f5dae80 2024-04-01T01:39:53.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93f8432740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f93f01c01c0 con 0x55934f5da490 2024-04-01T01:39:53.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93eeffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5da490 0x55934f5ebc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49780/0 (socket says 172.21.15.70:49780) 2024-04-01T01:39:53.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93ee7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 0x55934f5ea3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93eeffd640 1 -- 172.21.15.70:0/2736281632 learned_addr learned my addr 172.21.15.70:0/2736281632 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:53.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93ef7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5dfff0 0x55934f5ea900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93f8432740 10 client.0 _ll_register_callbacks cb 0x7f93e000dbb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:53.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93ee7fc640 1 -- 172.21.15.70:0/2736281632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5da490 msgr2=0x55934f5ebc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93ee7fc640 1 --2- 172.21.15.70:0/2736281632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5da490 0x55934f5ebc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93ee7fc640 1 -- 172.21.15.70:0/2736281632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5dfff0 msgr2=0x55934f5ea900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93ee7fc640 1 --2- 172.21.15.70:0/2736281632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5dfff0 0x55934f5ea900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93ee7fc640 1 -- 172.21.15.70:0/2736281632 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55934f5eb010 con 0x55934f5dae80 2024-04-01T01:39:53.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.577+0000 7f93eeffd640 1 --2- 172.21.15.70:0/2736281632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5da490 0x55934f5ebc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:53.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.578+0000 7f93ee7fc640 1 --2- 172.21.15.70:0/2736281632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 0x55934f5ea3c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f93e4006010 tx=0x7f93e400ff60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=649372c18659dfd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.578+0000 7f93ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:53.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.578+0000 7f93ceffd640 1 -- 172.21.15.70:0/2736281632 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e402a040 con 0x55934f5dae80 2024-04-01T01:39:53.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.578+0000 7f93f8432740 1 -- 172.21.15.70:0/2736281632 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93e802ed60 con 0x55934f5dae80 2024-04-01T01:39:53.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.578+0000 7f93ceffd640 1 -- 172.21.15.70:0/2736281632 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f93e4022040 con 0x55934f5dae80 2024-04-01T01:39:53.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.578+0000 7f93ceffd640 1 -- 172.21.15.70:0/2736281632 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f93e40255f0 con 0x55934f5dae80 2024-04-01T01:39:53.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.578+0000 7f93f8432740 10 client.27356 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:53.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.578+0000 7f93f8432740 1 -- 172.21.15.70:0/2736281632 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55934f657250 con 0x55934f5dae80 2024-04-01T01:39:53.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.578+0000 7f93ceffd640 1 -- 172.21.15.70:0/2736281632 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93e802ed60 con 0x55934f5dae80 2024-04-01T01:39:53.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.579+0000 7f93ceffd640 1 -- 172.21.15.70:0/2736281632 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f93e4025b70 con 0x55934f5dae80 2024-04-01T01:39:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.579+0000 7f93f8432740 10 client.27356 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.579+0000 7f93f8432740 10 client.27356 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:39:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.579+0000 7f93f8432740 20 client.27356 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.579+0000 7f93f8432740 1 --2- 172.21.15.70:0/2736281632 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55934f657bb0 0x55934f659fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.579+0000 7f93f8432740 4 client.27356 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:39:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.579+0000 7f93f8432740 1 -- 172.21.15.70:0/2736281632 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55934f546a40 con 0x55934f657bb0 2024-04-01T01:39:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.579+0000 7f93eeffd640 1 --2- 172.21.15.70:0/2736281632 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55934f657bb0 0x55934f659fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.579+0000 7f93eeffd640 1 --2- 172.21.15.70:0/2736281632 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55934f657bb0 0x55934f659fd0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6b68ea423c35337f server_cookie=dda4965cca8881b9 in_seq=0 out_seq=0 2024-04-01T01:39:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.579+0000 7f93ceffd640 10 client.27356 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:39:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.579+0000 7f93ceffd640 1 -- 172.21.15.70:0/2736281632 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55934f546a40 con 0x55934f657bb0 2024-04-01T01:39:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.579+0000 7f93ceffd640 10 client.27356 handle_command_reply: tid=0 2024-04-01T01:39:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.579+0000 7f93f8432740 1 client.27356 shutdown 2024-04-01T01:39:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.579+0000 7f93f8432740 1 -- 172.21.15.70:0/2736281632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 msgr2=0x55934f5ea3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.580+0000 7f93f8432740 1 --2- 172.21.15.70:0/2736281632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 0x55934f5ea3c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f93e4006010 tx=0x7f93e400ff60 comp rx=0 tx=0).stop 2024-04-01T01:39:53.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.580+0000 7f93f8432740 1 -- 172.21.15.70:0/2736281632 shutdown_connections 2024-04-01T01:39:53.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.580+0000 7f93f8432740 1 --2- 172.21.15.70:0/2736281632 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55934f657bb0 0x55934f659fd0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.580+0000 7f93f8432740 1 --2- 172.21.15.70:0/2736281632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55934f5dfff0 0x55934f5ea900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.580+0000 7f93f8432740 1 --2- 172.21.15.70:0/2736281632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55934f5dae80 0x55934f5ea3c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.580+0000 7f93f8432740 1 --2- 172.21.15.70:0/2736281632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55934f5da490 0x55934f5ebc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.580+0000 7f93f8432740 1 -- 172.21.15.70:0/2736281632 >> 172.21.15.70:0/2736281632 conn(0x55934f5d5a00 msgr2=0x55934f5d8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.580+0000 7f93ceffd640 0 client.27356 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:39:53.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.580+0000 7f93f8432740 1 -- 172.21.15.70:0/2736281632 shutdown_connections 2024-04-01T01:39:53.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.580+0000 7f93f8432740 1 -- 172.21.15.70:0/2736281632 wait complete. 2024-04-01T01:39:53.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.580+0000 7f93f8432740 20 client.27356 trim_cache size 0 max 16384 2024-04-01T01:39:53.585 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:39:53.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.580+0000 7f93f584e640 1 -- 172.21.15.70:0/2469294167 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93d80a25f0 msgr2=0x7f93d80a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.580+0000 7f93f584e640 1 --2- 172.21.15.70:0/2469294167 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93d80a25f0 0x7f93d80a4ab0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f93e0005fd0 tx=0x7f93e000d720 comp rx=0 tx=0).stop 2024-04-01T01:39:53.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.580+0000 7f93f584e640 1 -- 172.21.15.70:0/2469294167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93f0108f60 msgr2=0x7f93f00703c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.580+0000 7f93f584e640 1 --2- 172.21.15.70:0/2469294167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93f0108f60 0x7f93f00703c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f93e8032440 tx=0x7f93e8003c20 comp rx=0 tx=0).stop 2024-04-01T01:39:53.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.581+0000 7f93f584e640 1 -- 172.21.15.70:0/2469294167 shutdown_connections 2024-04-01T01:39:53.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.581+0000 7f93f584e640 1 --2- 172.21.15.70:0/2469294167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93f01abf70 0x7f93f01ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.581+0000 7f93f584e640 1 --2- 172.21.15.70:0/2469294167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93f0109920 0x7f93f0070900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.581+0000 7f93f584e640 1 --2- 172.21.15.70:0/2469294167 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93d80a25f0 0x7f93d80a4ab0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.581+0000 7f93f584e640 1 --2- 172.21.15.70:0/2469294167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93f0108f60 0x7f93f00703c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.581+0000 7f93f584e640 1 -- 172.21.15.70:0/2469294167 >> 172.21.15.70:0/2469294167 conn(0x7f93f0076d10 msgr2=0x7f93f010c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.581+0000 7f93f584e640 1 -- 172.21.15.70:0/2469294167 shutdown_connections 2024-04-01T01:39:53.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.581+0000 7f93f584e640 1 -- 172.21.15.70:0/2469294167 wait complete. 2024-04-01T01:39:53.592 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:39:53.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.729+0000 7ff64dd01640 1 -- 172.21.15.70:0/3871372010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff648100650 msgr2=0x7ff648100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.729+0000 7ff64dd01640 1 --2- 172.21.15.70:0/3871372010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff648100650 0x7ff648100a70 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff638009900 tx=0x7ff63802bc30 comp rx=0 tx=0).stop 2024-04-01T01:39:53.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.729+0000 7ff64dd01640 1 -- 172.21.15.70:0/3871372010 shutdown_connections 2024-04-01T01:39:53.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.729+0000 7ff64dd01640 1 --2- 172.21.15.70:0/3871372010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff648102200 0x7ff648108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.729+0000 7ff64dd01640 1 --2- 172.21.15.70:0/3871372010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff648101840 0x7ff648101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.729+0000 7ff64dd01640 1 --2- 172.21.15.70:0/3871372010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff648100650 0x7ff648100a70 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.729+0000 7ff64dd01640 1 -- 172.21.15.70:0/3871372010 >> 172.21.15.70:0/3871372010 conn(0x7ff6480fbbe0 msgr2=0x7ff6480fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.729+0000 7ff64dd01640 1 -- 172.21.15.70:0/3871372010 shutdown_connections 2024-04-01T01:39:53.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.729+0000 7ff64dd01640 1 -- 172.21.15.70:0/3871372010 wait complete. 2024-04-01T01:39:53.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.730+0000 7ff64dd01640 1 Processor -- start 2024-04-01T01:39:53.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.730+0000 7ff64dd01640 1 -- start start 2024-04-01T01:39:53.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.730+0000 7ff64dd01640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff648100650 0x7ff648196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.730+0000 7ff64dd01640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff648101840 0x7ff648196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.730+0000 7ff64dd01640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff648102200 0x7ff64819dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.730+0000 7ff64dd01640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff64810c180 con 0x7ff648102200 2024-04-01T01:39:53.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.730+0000 7ff64dd01640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff64810c000 con 0x7ff648100650 2024-04-01T01:39:53.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.730+0000 7ff64dd01640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff64810c300 con 0x7ff648101840 2024-04-01T01:39:53.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.730+0000 7ff647fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff648102200 0x7ff64819dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.730+0000 7ff6477fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff648100650 0x7ff648196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.730+0000 7ff646ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff648101840 0x7ff648196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.730+0000 7ff646ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff648101840 0x7ff648196e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49798/0 (socket says 172.21.15.70:49798) 2024-04-01T01:39:53.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.730+0000 7ff6477fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff648100650 0x7ff648196910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37550/0 (socket says 172.21.15.70:37550) 2024-04-01T01:39:53.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.730+0000 7ff646ffd640 1 -- 172.21.15.70:0/852186317 learned_addr learned my addr 172.21.15.70:0/852186317 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:53.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.730+0000 7ff647fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff648102200 0x7ff64819dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44576/0 (socket says 172.21.15.70:44576) 2024-04-01T01:39:53.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.731+0000 7ff6477fe640 1 -- 172.21.15.70:0/852186317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff648101840 msgr2=0x7ff648196e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.731+0000 7ff6477fe640 1 --2- 172.21.15.70:0/852186317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff648101840 0x7ff648196e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.731+0000 7ff6477fe640 1 -- 172.21.15.70:0/852186317 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff648102200 msgr2=0x7ff64819dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.731+0000 7ff6477fe640 1 --2- 172.21.15.70:0/852186317 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff648102200 0x7ff64819dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.731+0000 7ff6477fe640 1 -- 172.21.15.70:0/852186317 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64819e530 con 0x7ff648100650 2024-04-01T01:39:53.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.731+0000 7ff646ffd640 1 --2- 172.21.15.70:0/852186317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff648101840 0x7ff648196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:53.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.731+0000 7ff6477fe640 1 --2- 172.21.15.70:0/852186317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff648100650 0x7ff648196910 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff638036040 tx=0x7ff638005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.731+0000 7ff644ff9640 1 -- 172.21.15.70:0/852186317 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff638043020 con 0x7ff648100650 2024-04-01T01:39:53.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.731+0000 7ff644ff9640 1 -- 172.21.15.70:0/852186317 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff638004350 con 0x7ff648100650 2024-04-01T01:39:53.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.731+0000 7ff64dd01640 1 -- 172.21.15.70:0/852186317 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff64819e7c0 con 0x7ff648100650 2024-04-01T01:39:53.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.731+0000 7ff644ff9640 1 -- 172.21.15.70:0/852186317 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff638003ab0 con 0x7ff648100650 2024-04-01T01:39:53.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.731+0000 7ff64dd01640 1 -- 172.21.15.70:0/852186317 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff64819ece0 con 0x7ff648100650 2024-04-01T01:39:53.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.733+0000 7ff644ff9640 1 -- 172.21.15.70:0/852186317 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7ff638046020 con 0x7ff648100650 2024-04-01T01:39:53.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.734+0000 7ff644ff9640 1 --2- 172.21.15.70:0/852186317 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff6300a24a0 0x7ff6300a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.734+0000 7ff644ff9640 1 -- 172.21.15.70:0/852186317 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff6380f0bd0 con 0x7ff648100650 2024-04-01T01:39:53.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.734+0000 7ff646ffd640 1 --2- 172.21.15.70:0/852186317 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff6300a24a0 0x7ff6300a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.734+0000 7ff6509b7740 1 Processor -- start 2024-04-01T01:39:53.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.734+0000 7ff6509b7740 1 -- start start 2024-04-01T01:39:53.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.734+0000 7ff6509b7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf06a0 0x561ecdbf0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.734+0000 7ff6509b7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdbf5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.734+0000 7ff6509b7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf6200 0x561ecdbf8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.734+0000 7ff6509b7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561ecdb531b0 con 0x561ecdbf1090 2024-04-01T01:39:53.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.734+0000 7ff6509b7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561ecdafbc00 con 0x561ecdbf6200 2024-04-01T01:39:53.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.734+0000 7ff6509b7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561ecdb4f250 con 0x561ecdbf06a0 2024-04-01T01:39:53.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.734+0000 7ff646ffd640 1 --2- 172.21.15.70:0/852186317 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff6300a24a0 0x7ff6300a4960 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff648197e20 tx=0x7ff63c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.735+0000 7ff646ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdbf5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.735+0000 7ff646ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdbf5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44592/0 (socket says 172.21.15.70:44592) 2024-04-01T01:39:53.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.735+0000 7ff6477fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf06a0 0x561ecdbf0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.735+0000 7ff646ffd640 1 -- 172.21.15.70:0/2235298170 learned_addr learned my addr 172.21.15.70:0/2235298170 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:53.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.735+0000 7ff647fff640 1 --2- 172.21.15.70:0/2235298170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf6200 0x561ecdbf8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.735+0000 7ff646ffd640 1 -- 172.21.15.70:0/2235298170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf06a0 msgr2=0x561ecdbf0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.735+0000 7ff646ffd640 1 --2- 172.21.15.70:0/2235298170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf06a0 0x561ecdbf0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.735+0000 7ff646ffd640 1 -- 172.21.15.70:0/2235298170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf6200 msgr2=0x561ecdbf8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.735+0000 7ff646ffd640 1 --2- 172.21.15.70:0/2235298170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf6200 0x561ecdbf8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.735+0000 7ff646ffd640 1 -- 172.21.15.70:0/2235298170 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ecdb1b3d0 con 0x561ecdbf1090 2024-04-01T01:39:53.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.735+0000 7ff646ffd640 1 --2- 172.21.15.70:0/2235298170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdbf5b90 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00f750 tx=0x7ff63c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32dc53e572a02d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.735+0000 7ff62dffb640 1 -- 172.21.15.70:0/2235298170 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c01c850 con 0x561ecdbf1090 2024-04-01T01:39:53.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.735+0000 7ff62dffb640 1 -- 172.21.15.70:0/2235298170 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff63c025390 con 0x561ecdbf1090 2024-04-01T01:39:53.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.735+0000 7ff62dffb640 1 -- 172.21.15.70:0/2235298170 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c01a5f0 con 0x561ecdbf1090 2024-04-01T01:39:53.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.736+0000 7ff6509b7740 1 -- 172.21.15.70:0/2235298170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 msgr2=0x561ecdbf5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.736+0000 7ff6509b7740 1 --2- 172.21.15.70:0/2235298170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdbf5b90 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00f750 tx=0x7ff63c014000 comp rx=0 tx=0).stop 2024-04-01T01:39:53.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.736+0000 7ff6509b7740 1 -- 172.21.15.70:0/2235298170 shutdown_connections 2024-04-01T01:39:53.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.736+0000 7ff6509b7740 1 --2- 172.21.15.70:0/2235298170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf6200 0x561ecdbf8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.736+0000 7ff6509b7740 1 --2- 172.21.15.70:0/2235298170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdbf5b90 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.736+0000 7ff6509b7740 1 --2- 172.21.15.70:0/2235298170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf06a0 0x561ecdbf0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.736+0000 7ff6509b7740 1 -- 172.21.15.70:0/2235298170 >> 172.21.15.70:0/2235298170 conn(0x561ecdbebc10 msgr2=0x561ecdbec010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.736+0000 7ff6509b7740 1 -- 172.21.15.70:0/2235298170 shutdown_connections 2024-04-01T01:39:53.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.736+0000 7ff6509b7740 1 -- 172.21.15.70:0/2235298170 wait complete. 2024-04-01T01:39:53.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.736+0000 7ff6509b7740 1 -- start start 2024-04-01T01:39:53.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.737+0000 7ff6509b7740 1 Processor -- start 2024-04-01T01:39:53.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.737+0000 7ff6509b7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf06a0 0x561ecdbeaf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.737+0000 7ff6509b7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdbe9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.737+0000 7ff6509b7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf6200 0x561ecdbe9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.737+0000 7ff6509b7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561ecdbf91b0 con 0x561ecdbf1090 2024-04-01T01:39:53.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.737+0000 7ff6509b7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561ecdaf1470 con 0x561ecdbf6200 2024-04-01T01:39:53.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.737+0000 7ff6509b7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561ecdbf9330 con 0x561ecdbf06a0 2024-04-01T01:39:53.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.737+0000 7ff6509b7740 10 client.0 _ll_register_callbacks cb 0x561ecdb0ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:53.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.737+0000 7ff646ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdbe9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.738+0000 7ff646ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdbe9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44596/0 (socket says 172.21.15.70:44596) 2024-04-01T01:39:53.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.738+0000 7ff646ffd640 1 -- 172.21.15.70:0/1359699923 learned_addr learned my addr 172.21.15.70:0/1359699923 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:53.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.738+0000 7ff647fff640 1 --2- 172.21.15.70:0/1359699923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf6200 0x561ecdbe9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.738+0000 7ff6477fe640 1 --2- 172.21.15.70:0/1359699923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf06a0 0x561ecdbeaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.738+0000 7ff646ffd640 1 -- 172.21.15.70:0/1359699923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf06a0 msgr2=0x561ecdbeaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.738+0000 7ff646ffd640 1 --2- 172.21.15.70:0/1359699923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf06a0 0x561ecdbeaf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.738+0000 7ff646ffd640 1 -- 172.21.15.70:0/1359699923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf6200 msgr2=0x561ecdbe9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.738+0000 7ff646ffd640 1 --2- 172.21.15.70:0/1359699923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf6200 0x561ecdbe9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.738+0000 7ff646ffd640 1 -- 172.21.15.70:0/1359699923 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ecd8673f0 con 0x561ecdbf1090 2024-04-01T01:39:53.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.738+0000 7ff647fff640 1 --2- 172.21.15.70:0/1359699923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf6200 0x561ecdbe9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:53.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.738+0000 7ff646ffd640 1 --2- 172.21.15.70:0/1359699923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdbe9590 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff63c025a00 tx=0x7ff63c005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe280869741be79 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.738+0000 7ff626ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:39:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.738+0000 7ff626ffd640 1 -- 172.21.15.70:0/1359699923 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c033060 con 0x561ecdbf1090 2024-04-01T01:39:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.739+0000 7ff626ffd640 1 -- 172.21.15.70:0/1359699923 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff63c005260 con 0x561ecdbf1090 2024-04-01T01:39:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.739+0000 7ff6509b7740 1 -- 172.21.15.70:0/1359699923 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff63c025390 con 0x561ecdbf1090 2024-04-01T01:39:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.739+0000 7ff626ffd640 1 -- 172.21.15.70:0/1359699923 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff63c005460 con 0x561ecdbf1090 2024-04-01T01:39:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.739+0000 7ff6509b7740 10 client.17469 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.739+0000 7ff6509b7740 1 -- 172.21.15.70:0/1359699923 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561ecd9317d0 con 0x561ecdbf1090 2024-04-01T01:39:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.739+0000 7ff626ffd640 1 -- 172.21.15.70:0/1359699923 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff63c025390 con 0x561ecdbf1090 2024-04-01T01:39:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.739+0000 7ff626ffd640 1 -- 172.21.15.70:0/1359699923 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff63c01c910 con 0x561ecdbf1090 2024-04-01T01:39:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.739+0000 7ff6509b7740 10 client.17469 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.739+0000 7ff6509b7740 10 client.17469 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:39:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.739+0000 7ff6509b7740 20 client.17469 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.739+0000 7ff6509b7740 1 --2- 172.21.15.70:0/1359699923 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x561ecdc0c8c0 0x561ecdc0ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.739+0000 7ff6509b7740 4 client.17469 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.739+0000 7ff6509b7740 1 -- 172.21.15.70:0/1359699923 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ecdb606f0 con 0x561ecdc0c8c0 2024-04-01T01:39:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.739+0000 7ff6477fe640 1 --2- 172.21.15.70:0/1359699923 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x561ecdc0c8c0 0x561ecdc0ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.740+0000 7ff626ffd640 10 client.17469 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:39:53.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.740+0000 7ff6477fe640 1 --2- 172.21.15.70:0/1359699923 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x561ecdc0c8c0 0x561ecdc0ecb0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=52352aa27830d565 server_cookie=10917dff3d883b5b in_seq=0 out_seq=0 2024-04-01T01:39:53.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.741+0000 7ff626ffd640 1 -- 172.21.15.70:0/1359699923 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561ecdb606f0 con 0x561ecdc0c8c0 2024-04-01T01:39:53.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.741+0000 7ff626ffd640 10 client.17469 handle_command_reply: tid=0 2024-04-01T01:39:53.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.741+0000 7ff6509b7740 1 client.17469 shutdown 2024-04-01T01:39:53.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.741+0000 7ff6509b7740 1 -- 172.21.15.70:0/1359699923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 msgr2=0x561ecdbe9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.741+0000 7ff6509b7740 1 --2- 172.21.15.70:0/1359699923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdbe9590 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff63c025a00 tx=0x7ff63c005750 comp rx=0 tx=0).stop 2024-04-01T01:39:53.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.741+0000 7ff6509b7740 1 -- 172.21.15.70:0/1359699923 shutdown_connections 2024-04-01T01:39:53.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.741+0000 7ff6509b7740 1 --2- 172.21.15.70:0/1359699923 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x561ecdc0c8c0 0x561ecdc0ecb0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.741+0000 7ff6509b7740 1 --2- 172.21.15.70:0/1359699923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf6200 0x561ecdbe9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.741+0000 7ff6509b7740 1 --2- 172.21.15.70:0/1359699923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdbe9590 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.741+0000 7ff6509b7740 1 --2- 172.21.15.70:0/1359699923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf06a0 0x561ecdbeaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.741+0000 7ff626ffd640 0 client.17469 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:39:53.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.741+0000 7ff6509b7740 1 -- 172.21.15.70:0/1359699923 >> 172.21.15.70:0/1359699923 conn(0x561ecdbebc10 msgr2=0x561ecdbece60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.741+0000 7ff6509b7740 1 -- 172.21.15.70:0/1359699923 shutdown_connections 2024-04-01T01:39:53.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.741+0000 7ff6509b7740 1 -- 172.21.15.70:0/1359699923 wait complete. 2024-04-01T01:39:53.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.741+0000 7ff6509b7740 20 client.17469 trim_cache size 0 max 16384 2024-04-01T01:39:53.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.777+0000 7ff6509b7740 1 Processor -- start 2024-04-01T01:39:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.784+0000 7ff6509b7740 1 -- start start 2024-04-01T01:39:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.784+0000 7ff6509b7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf06a0 0x561ecdc145a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.784+0000 7ff6509b7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdc1e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.784+0000 7ff6509b7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf6200 0x561ecdc1ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.784+0000 7ff6509b7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff6481a81f0 con 0x561ecdbf1090 2024-04-01T01:39:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.784+0000 7ff6509b7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff6481a9d30 con 0x561ecdbf06a0 2024-04-01T01:39:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.784+0000 7ff6509b7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff6481a88c0 con 0x561ecdbf6200 2024-04-01T01:39:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.784+0000 7ff646ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdc1e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.784+0000 7ff646ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdc1e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44600/0 (socket says 172.21.15.70:44600) 2024-04-01T01:39:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.784+0000 7ff646ffd640 1 -- 172.21.15.70:0/204694547 learned_addr learned my addr 172.21.15.70:0/204694547 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.784+0000 7ff647fff640 1 --2- 172.21.15.70:0/204694547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf6200 0x561ecdc1ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.785+0000 7ff646ffd640 1 -- 172.21.15.70:0/204694547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf6200 msgr2=0x561ecdc1ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.785+0000 7ff646ffd640 1 --2- 172.21.15.70:0/204694547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf6200 0x561ecdc1ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.785+0000 7ff646ffd640 1 -- 172.21.15.70:0/204694547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf06a0 msgr2=0x561ecdc145a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:39:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.785+0000 7ff646ffd640 1 --2- 172.21.15.70:0/204694547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf06a0 0x561ecdc145a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.785+0000 7ff646ffd640 1 -- 172.21.15.70:0/204694547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ecdbeb5b0 con 0x561ecdbf1090 2024-04-01T01:39:53.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.790+0000 7ff646ffd640 1 --2- 172.21.15.70:0/204694547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdc1e930 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff63c01cea0 tx=0x7ff63c025de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269ebaea6388708d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.795+0000 7ff62d7fa640 1 -- 172.21.15.70:0/204694547 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c033070 con 0x561ecdbf1090 2024-04-01T01:39:53.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.795+0000 7ff62d7fa640 1 -- 172.21.15.70:0/204694547 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff63c025390 con 0x561ecdbf1090 2024-04-01T01:39:53.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.795+0000 7ff62d7fa640 1 -- 172.21.15.70:0/204694547 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c01c650 con 0x561ecdbf1090 2024-04-01T01:39:53.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.798+0000 7ff6509b7740 1 -- 172.21.15.70:0/204694547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 msgr2=0x561ecdc1e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.798+0000 7ff6509b7740 1 --2- 172.21.15.70:0/204694547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdc1e930 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff63c01cea0 tx=0x7ff63c025de0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.801+0000 7ff6509b7740 1 -- 172.21.15.70:0/204694547 shutdown_connections 2024-04-01T01:39:53.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.801+0000 7ff6509b7740 1 --2- 172.21.15.70:0/204694547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf6200 0x561ecdc1ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.801+0000 7ff6509b7740 1 --2- 172.21.15.70:0/204694547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf1090 0x561ecdc1e930 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.801+0000 7ff6509b7740 1 --2- 172.21.15.70:0/204694547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf06a0 0x561ecdc145a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.801+0000 7ff6509b7740 1 -- 172.21.15.70:0/204694547 >> 172.21.15.70:0/204694547 conn(0x561ecdbebc10 msgr2=0x561ecdbef700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.801+0000 7ff6509b7740 1 -- 172.21.15.70:0/204694547 shutdown_connections 2024-04-01T01:39:53.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.801+0000 7ff6509b7740 1 -- 172.21.15.70:0/204694547 wait complete. 2024-04-01T01:39:53.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.802+0000 7ff6509b7740 1 -- start start 2024-04-01T01:39:53.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.803+0000 7ff6509b7740 1 Processor -- start 2024-04-01T01:39:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.803+0000 7ff6509b7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf06a0 0x561ecdc1f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.803+0000 7ff6509b7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf1090 0x561ecdc1fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.803+0000 7ff6509b7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf6200 0x561ecdc1c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.803+0000 7ff6477fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf06a0 0x561ecdc1f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.803+0000 7ff6477fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf06a0 0x561ecdc1f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37592/0 (socket says 172.21.15.70:37592) 2024-04-01T01:39:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.803+0000 7ff6477fe640 1 -- 172.21.15.70:0/820115532 learned_addr learned my addr 172.21.15.70:0/820115532 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.804+0000 7ff6509b7740 1 -- 172.21.15.70:0/820115532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff6481a9660 con 0x561ecdbf6200 2024-04-01T01:39:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.804+0000 7ff6509b7740 1 -- 172.21.15.70:0/820115532 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff6481a8f90 con 0x561ecdbf06a0 2024-04-01T01:39:53.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.804+0000 7ff6509b7740 1 -- 172.21.15.70:0/820115532 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff6481a61d0 con 0x561ecdbf1090 2024-04-01T01:39:53.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.804+0000 7ff647fff640 1 --2- 172.21.15.70:0/820115532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf6200 0x561ecdc1c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.804+0000 7ff6509b7740 10 client.0 _ll_register_callbacks cb 0x7ff63c00f650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:53.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.804+0000 7ff646ffd640 1 --2- 172.21.15.70:0/820115532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf1090 0x561ecdc1fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.804+0000 7ff647fff640 1 -- 172.21.15.70:0/820115532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf1090 msgr2=0x561ecdc1fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.804+0000 7ff647fff640 1 --2- 172.21.15.70:0/820115532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf1090 0x561ecdc1fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.804+0000 7ff647fff640 1 -- 172.21.15.70:0/820115532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf06a0 msgr2=0x561ecdc1f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:53.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.804+0000 7ff647fff640 1 --2- 172.21.15.70:0/820115532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf06a0 0x561ecdc1f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.804+0000 7ff647fff640 1 -- 172.21.15.70:0/820115532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ecdc1c8e0 con 0x561ecdbf6200 2024-04-01T01:39:53.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.804+0000 7ff6477fe640 1 --2- 172.21.15.70:0/820115532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf06a0 0x561ecdc1f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:53.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.805+0000 7ff647fff640 1 --2- 172.21.15.70:0/820115532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf6200 0x561ecdc1c1a0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x561ecdbeb4e0 tx=0x7ff64000f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d22add552fa9e1a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:53.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.806+0000 7ff626ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:39:53.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.806+0000 7ff626ffd640 1 -- 172.21.15.70:0/820115532 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff640003c40 con 0x561ecdbf6200 2024-04-01T01:39:53.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.806+0000 7ff626ffd640 1 -- 172.21.15.70:0/820115532 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff640003de0 con 0x561ecdbf6200 2024-04-01T01:39:53.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.807+0000 7ff6509b7740 1 -- 172.21.15.70:0/820115532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff63c025390 con 0x561ecdbf6200 2024-04-01T01:39:53.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.811+0000 7ff626ffd640 1 -- 172.21.15.70:0/820115532 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff640004b20 con 0x561ecdbf6200 2024-04-01T01:39:53.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.813+0000 7ff6509b7740 10 client.17475 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:53.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.813+0000 7ff6509b7740 1 -- 172.21.15.70:0/820115532 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561ecdc84300 con 0x561ecdbf6200 2024-04-01T01:39:53.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.815+0000 7ff626ffd640 1 -- 172.21.15.70:0/820115532 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff63c025390 con 0x561ecdbf6200 2024-04-01T01:39:53.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.815+0000 7ff626ffd640 1 -- 172.21.15.70:0/820115532 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff64001d830 con 0x561ecdbf6200 2024-04-01T01:39:53.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.818+0000 7ff6509b7740 10 client.17475 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:53.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.818+0000 7ff6509b7740 10 client.17475 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:39:53.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.819+0000 7ff6509b7740 20 client.17475 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:53.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.819+0000 7ff6509b7740 1 --2- 172.21.15.70:0/820115532 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x561ecdc84ce0 0x561ecdc870d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:53.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.819+0000 7ff6509b7740 4 client.17475 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:39:53.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.819+0000 7ff6509b7740 1 -- 172.21.15.70:0/820115532 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561ecdb5aa40 con 0x561ecdc84ce0 2024-04-01T01:39:53.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.819+0000 7ff6477fe640 1 --2- 172.21.15.70:0/820115532 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x561ecdc84ce0 0x561ecdc870d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:53.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.820+0000 7ff6477fe640 1 --2- 172.21.15.70:0/820115532 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x561ecdc84ce0 0x561ecdc870d0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fb46dd85e5ddbd8 server_cookie=ba24dc6f2b7792dc in_seq=0 out_seq=0 2024-04-01T01:39:53.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.821+0000 7ff626ffd640 10 client.17475 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:39:53.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.823+0000 7ff626ffd640 1 -- 172.21.15.70:0/820115532 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561ecdb5aa40 con 0x561ecdc84ce0 2024-04-01T01:39:53.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.823+0000 7ff626ffd640 10 client.17475 handle_command_reply: tid=0 2024-04-01T01:39:53.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.823+0000 7ff6509b7740 1 client.17475 shutdown 2024-04-01T01:39:53.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.824+0000 7ff6509b7740 1 -- 172.21.15.70:0/820115532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf6200 msgr2=0x561ecdc1c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.824+0000 7ff6509b7740 1 --2- 172.21.15.70:0/820115532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf6200 0x561ecdc1c1a0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x561ecdbeb4e0 tx=0x7ff64000f850 comp rx=0 tx=0).stop 2024-04-01T01:39:53.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.824+0000 7ff6509b7740 1 -- 172.21.15.70:0/820115532 shutdown_connections 2024-04-01T01:39:53.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.824+0000 7ff6509b7740 1 --2- 172.21.15.70:0/820115532 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x561ecdc84ce0 0x561ecdc870d0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.824+0000 7ff6509b7740 1 --2- 172.21.15.70:0/820115532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561ecdbf6200 0x561ecdc1c1a0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.824+0000 7ff626ffd640 0 client.17475 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:39:53.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.824+0000 7ff6509b7740 1 --2- 172.21.15.70:0/820115532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561ecdbf1090 0x561ecdc1fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.824+0000 7ff6509b7740 1 --2- 172.21.15.70:0/820115532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561ecdbf06a0 0x561ecdc1f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.824+0000 7ff6509b7740 1 -- 172.21.15.70:0/820115532 >> 172.21.15.70:0/820115532 conn(0x561ecdbebc10 msgr2=0x561ecdc13590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.825+0000 7ff6509b7740 1 -- 172.21.15.70:0/820115532 shutdown_connections 2024-04-01T01:39:53.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.825+0000 7ff6509b7740 1 -- 172.21.15.70:0/820115532 wait complete. 2024-04-01T01:39:53.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.828+0000 7ff6509b7740 20 client.17475 trim_cache size 0 max 16384 2024-04-01T01:39:53.830 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:39:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.832+0000 7ff64dd01640 1 -- 172.21.15.70:0/852186317 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff6300a24a0 msgr2=0x7ff6300a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.832+0000 7ff64dd01640 1 --2- 172.21.15.70:0/852186317 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff6300a24a0 0x7ff6300a4960 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff648197e20 tx=0x7ff63c005f90 comp rx=0 tx=0).stop 2024-04-01T01:39:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.832+0000 7ff64dd01640 1 -- 172.21.15.70:0/852186317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff648100650 msgr2=0x7ff648196910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.832+0000 7ff64dd01640 1 --2- 172.21.15.70:0/852186317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff648100650 0x7ff648196910 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff638036040 tx=0x7ff638005dc0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.833+0000 7ff64dd01640 1 -- 172.21.15.70:0/852186317 shutdown_connections 2024-04-01T01:39:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.833+0000 7ff64dd01640 1 --2- 172.21.15.70:0/852186317 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff648102200 0x7ff64819dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.833+0000 7ff64dd01640 1 --2- 172.21.15.70:0/852186317 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff6300a24a0 0x7ff6300a4960 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.833+0000 7ff64dd01640 1 --2- 172.21.15.70:0/852186317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff648101840 0x7ff648196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.833+0000 7ff64dd01640 1 --2- 172.21.15.70:0/852186317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff648100650 0x7ff648196910 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.833+0000 7ff64dd01640 1 -- 172.21.15.70:0/852186317 >> 172.21.15.70:0/852186317 conn(0x7ff6480fbbe0 msgr2=0x7ff648104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.834+0000 7ff64dd01640 1 -- 172.21.15.70:0/852186317 shutdown_connections 2024-04-01T01:39:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:53.834+0000 7ff64dd01640 1 -- 172.21.15.70:0/852186317 wait complete. 2024-04-01T01:39:53.854 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:39:54.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.009+0000 7f3608555640 1 -- 172.21.15.70:0/3640380437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3600101170 msgr2=0x7f3600105a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.009+0000 7f3608555640 1 --2- 172.21.15.70:0/3640380437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3600101170 0x7f3600105a10 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f35f8009ca0 tx=0x7f35f802c0f0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.010+0000 7f3608555640 1 -- 172.21.15.70:0/3640380437 shutdown_connections 2024-04-01T01:39:54.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.010+0000 7f3608555640 1 --2- 172.21.15.70:0/3640380437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3600101170 0x7f3600105a10 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.010+0000 7f3608555640 1 --2- 172.21.15.70:0/3640380437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36001007b0 0x7f3600100c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.010+0000 7f3608555640 1 --2- 172.21.15.70:0/3640380437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36000ffa50 0x7f36000ffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.010+0000 7f3608555640 1 -- 172.21.15.70:0/3640380437 >> 172.21.15.70:0/3640380437 conn(0x7f36000fb380 msgr2=0x7f36000fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:54.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.010+0000 7f3608555640 1 -- 172.21.15.70:0/3640380437 shutdown_connections 2024-04-01T01:39:54.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.010+0000 7f3608555640 1 -- 172.21.15.70:0/3640380437 wait complete. 2024-04-01T01:39:54.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.011+0000 7f3608555640 1 Processor -- start 2024-04-01T01:39:54.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.011+0000 7f3608555640 1 -- start start 2024-04-01T01:39:54.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.011+0000 7f3608555640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36000ffa50 0x7f360019de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.011+0000 7f3608555640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36001007b0 0x7f360019e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.011+0000 7f3608555640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3600101170 0x7f36001a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.011+0000 7f3608555640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3600113930 con 0x7f3600101170 2024-04-01T01:39:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.012+0000 7f36062f1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36000ffa50 0x7f360019de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.012+0000 7f3605af0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36001007b0 0x7f360019e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.012+0000 7f3608555640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f36001137b0 con 0x7f36000ffa50 2024-04-01T01:39:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.012+0000 7f3608555640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3600113ab0 con 0x7f36001007b0 2024-04-01T01:39:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.012+0000 7f3605af0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36001007b0 0x7f360019e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49870/0 (socket says 172.21.15.70:49870) 2024-04-01T01:39:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.012+0000 7f3605af0640 1 -- 172.21.15.70:0/1600014674 learned_addr learned my addr 172.21.15.70:0/1600014674 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.012+0000 7f3606af2640 1 --2- 172.21.15.70:0/1600014674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3600101170 0x7f36001a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.012+0000 7f3605af0640 1 -- 172.21.15.70:0/1600014674 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36000ffa50 msgr2=0x7f360019de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.012+0000 7f3605af0640 1 --2- 172.21.15.70:0/1600014674 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36000ffa50 0x7f360019de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.012+0000 7f3605af0640 1 -- 172.21.15.70:0/1600014674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3600101170 msgr2=0x7f36001a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.012+0000 7f3605af0640 1 --2- 172.21.15.70:0/1600014674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3600101170 0x7f36001a5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.012+0000 7f3605af0640 1 -- 172.21.15.70:0/1600014674 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36001a59d0 con 0x7f36001007b0 2024-04-01T01:39:54.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.012+0000 7f36062f1640 1 --2- 172.21.15.70:0/1600014674 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36000ffa50 0x7f360019de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:54.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.012+0000 7f3606af2640 1 --2- 172.21.15.70:0/1600014674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3600101170 0x7f36001a5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:54.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.012+0000 7f3605af0640 1 --2- 172.21.15.70:0/1600014674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36001007b0 0x7f360019e340 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f35f000e9c0 tx=0x7f35f000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:54.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.013+0000 7f35f77fe640 1 -- 172.21.15.70:0/1600014674 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f000cde0 con 0x7f36001007b0 2024-04-01T01:39:54.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.013+0000 7f35f77fe640 1 -- 172.21.15.70:0/1600014674 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35f00108e0 con 0x7f36001007b0 2024-04-01T01:39:54.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.013+0000 7f35f77fe640 1 -- 172.21.15.70:0/1600014674 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f0018990 con 0x7f36001007b0 2024-04-01T01:39:54.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.013+0000 7f3608555640 1 -- 172.21.15.70:0/1600014674 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36001a5cc0 con 0x7f36001007b0 2024-04-01T01:39:54.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.013+0000 7f3608555640 1 -- 172.21.15.70:0/1600014674 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36001a61c0 con 0x7f36001007b0 2024-04-01T01:39:54.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.016+0000 7f35f77fe640 1 -- 172.21.15.70:0/1600014674 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f35f001f020 con 0x7f36001007b0 2024-04-01T01:39:54.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.017+0000 7f360b20c740 1 Processor -- start 2024-04-01T01:39:54.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.017+0000 7f360b20c740 1 -- start start 2024-04-01T01:39:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f360b20c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac7d490 0x5617bac7d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f360b20c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7de80 0x5617bac82980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f360b20c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac82ff0 0x5617bac85480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f360b20c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5617babe21b0 con 0x5617bac82ff0 2024-04-01T01:39:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f360b20c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5617bab8ac00 con 0x5617bac7d490 2024-04-01T01:39:54.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f360b20c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5617babde250 con 0x5617bac7de80 2024-04-01T01:39:54.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f35f77fe640 1 --2- 172.21.15.70:0/1600014674 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f35ec0a24f0 0x7f35ec0a49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f3606af2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac7d490 0x5617bac7d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f3606af2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac7d490 0x5617bac7d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37632/0 (socket says 172.21.15.70:37632) 2024-04-01T01:39:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f3606af2640 1 -- 172.21.15.70:0/2771271057 learned_addr learned my addr 172.21.15.70:0/2771271057 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f36062f1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7de80 0x5617bac82980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f3605af0640 1 --2- 172.21.15.70:0/2771271057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac82ff0 0x5617bac85480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f3606af2640 1 --2- 172.21.15.70:0/1600014674 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f35ec0a24f0 0x7f35ec0a49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f35f77fe640 1 -- 172.21.15.70:0/1600014674 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f35f00cecb0 con 0x7f36001007b0 2024-04-01T01:39:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f36062f1640 1 -- 172.21.15.70:0/2771271057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac7d490 msgr2=0x5617bac7d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f36062f1640 1 --2- 172.21.15.70:0/2771271057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac7d490 0x5617bac7d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f36062f1640 1 -- 172.21.15.70:0/2771271057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac82ff0 msgr2=0x5617bac85480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f36062f1640 1 --2- 172.21.15.70:0/2771271057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac82ff0 0x5617bac85480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f36062f1640 1 -- 172.21.15.70:0/2771271057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617babaa3d0 con 0x5617bac7de80 2024-04-01T01:39:54.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.018+0000 7f3605af0640 1 --2- 172.21.15.70:0/2771271057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac82ff0 0x5617bac85480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:39:54.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.019+0000 7f36062f1640 1 --2- 172.21.15.70:0/2771271057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7de80 0x5617bac82980 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009fd0 tx=0x7f35fc0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffce21246c8f1a94 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:54.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.019+0000 7f3606af2640 1 --2- 172.21.15.70:0/1600014674 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f35ec0a24f0 0x7f35ec0a49b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x5617bac7e6e0 tx=0x7f35f802c0f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:54.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.019+0000 7f35f4bfe640 1 -- 172.21.15.70:0/2771271057 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc004030 con 0x5617bac7de80 2024-04-01T01:39:54.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.019+0000 7f35f4bfe640 1 -- 172.21.15.70:0/2771271057 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35fc0054d0 con 0x5617bac7de80 2024-04-01T01:39:54.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.019+0000 7f35f4bfe640 1 -- 172.21.15.70:0/2771271057 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc004c10 con 0x5617bac7de80 2024-04-01T01:39:54.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.020+0000 7f360b20c740 1 -- 172.21.15.70:0/2771271057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7de80 msgr2=0x5617bac82980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.020+0000 7f360b20c740 1 --2- 172.21.15.70:0/2771271057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7de80 0x5617bac82980 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009fd0 tx=0x7f35fc0118d0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.020+0000 7f360b20c740 1 -- 172.21.15.70:0/2771271057 shutdown_connections 2024-04-01T01:39:54.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.020+0000 7f360b20c740 1 --2- 172.21.15.70:0/2771271057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac82ff0 0x5617bac85480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.020+0000 7f360b20c740 1 --2- 172.21.15.70:0/2771271057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7de80 0x5617bac82980 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.020+0000 7f360b20c740 1 --2- 172.21.15.70:0/2771271057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac7d490 0x5617bac7d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.020+0000 7f360b20c740 1 -- 172.21.15.70:0/2771271057 >> 172.21.15.70:0/2771271057 conn(0x5617bac78a00 msgr2=0x5617bac78e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:54.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.020+0000 7f360b20c740 1 -- 172.21.15.70:0/2771271057 shutdown_connections 2024-04-01T01:39:54.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.020+0000 7f360b20c740 1 -- 172.21.15.70:0/2771271057 wait complete. 2024-04-01T01:39:54.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.021+0000 7f360b20c740 1 -- start start 2024-04-01T01:39:54.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.021+0000 7f360b20c740 1 Processor -- start 2024-04-01T01:39:54.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.021+0000 7f360b20c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7d490 0x5617bac87a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.021+0000 7f360b20c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac7de80 0x5617bac87f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.021+0000 7f360b20c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac82ff0 0x5617bac8ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.021+0000 7f360b20c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5617bac85ff0 con 0x5617bac82ff0 2024-04-01T01:39:54.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.021+0000 7f360b20c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5617bab80470 con 0x5617bac7de80 2024-04-01T01:39:54.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.021+0000 7f360b20c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5617bac86170 con 0x5617bac7d490 2024-04-01T01:39:54.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.021+0000 7f360b20c740 10 client.0 _ll_register_callbacks cb 0x5617bab9dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:54.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.021+0000 7f36062f1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7d490 0x5617bac87a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.021+0000 7f36062f1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7d490 0x5617bac87a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49890/0 (socket says 172.21.15.70:49890) 2024-04-01T01:39:54.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.021+0000 7f36062f1640 1 -- 172.21.15.70:0/1245917320 learned_addr learned my addr 172.21.15.70:0/1245917320 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:54.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.021+0000 7f3606af2640 1 --2- 172.21.15.70:0/1245917320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac82ff0 0x5617bac8ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.021+0000 7f3605af0640 1 --2- 172.21.15.70:0/1245917320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac7de80 0x5617bac87f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.022+0000 7f36062f1640 1 -- 172.21.15.70:0/1245917320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac7de80 msgr2=0x5617bac87f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:54.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.022+0000 7f36062f1640 1 --2- 172.21.15.70:0/1245917320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac7de80 0x5617bac87f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.022+0000 7f36062f1640 1 -- 172.21.15.70:0/1245917320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac82ff0 msgr2=0x5617bac8ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:54.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.022+0000 7f36062f1640 1 --2- 172.21.15.70:0/1245917320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac82ff0 0x5617bac8ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.022+0000 7f36062f1640 1 -- 172.21.15.70:0/1245917320 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617ba8f63f0 con 0x5617bac7d490 2024-04-01T01:39:54.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.022+0000 7f3605af0640 1 --2- 172.21.15.70:0/1245917320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac7de80 0x5617bac87f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:54.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.022+0000 7f3606af2640 1 --2- 172.21.15.70:0/1245917320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac82ff0 0x5617bac8ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:54.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.022+0000 7f36062f1640 1 --2- 172.21.15.70:0/1245917320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7d490 0x5617bac87a00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009fa0 tx=0x7f35fc00fcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81e99ab097b15952 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:54.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.022+0000 7f35e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:39:54.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.022+0000 7f35e9ffb640 1 -- 172.21.15.70:0/1245917320 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc029020 con 0x5617bac7d490 2024-04-01T01:39:54.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.022+0000 7f360b20c740 1 -- 172.21.15.70:0/1245917320 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35fc0054d0 con 0x5617bac7d490 2024-04-01T01:39:54.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.022+0000 7f35e9ffb640 1 -- 172.21.15.70:0/1245917320 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f35fc025680 con 0x5617bac7d490 2024-04-01T01:39:54.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.022+0000 7f35e9ffb640 1 -- 172.21.15.70:0/1245917320 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f35fc034050 con 0x5617bac7d490 2024-04-01T01:39:54.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.023+0000 7f360b20c740 10 client.27241 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:54.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.023+0000 7f360b20c740 1 -- 172.21.15.70:0/1245917320 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5617ba9c07d0 con 0x5617bac7d490 2024-04-01T01:39:54.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.023+0000 7f35e9ffb640 1 -- 172.21.15.70:0/1245917320 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35fc0054d0 con 0x5617bac7d490 2024-04-01T01:39:54.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.023+0000 7f35e9ffb640 1 -- 172.21.15.70:0/1245917320 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f35fc019700 con 0x5617bac7d490 2024-04-01T01:39:54.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.024+0000 7f360b20c740 10 client.27241 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:54.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.024+0000 7f360b20c740 10 client.27241 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:39:54.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.024+0000 7f360b20c740 20 client.27241 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:54.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.024+0000 7f360b20c740 1 --2- 172.21.15.70:0/1245917320 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5617bac93730 0x5617bac95ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.024+0000 7f360b20c740 4 client.27241 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:54.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.024+0000 7f360b20c740 1 -- 172.21.15.70:0/1245917320 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617babef6f0 con 0x5617bac93730 2024-04-01T01:39:54.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.024+0000 7f3605af0640 1 --2- 172.21.15.70:0/1245917320 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5617bac93730 0x5617bac95ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.024+0000 7f3605af0640 1 --2- 172.21.15.70:0/1245917320 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5617bac93730 0x5617bac95ba0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3f1e2a8b2e7e72de server_cookie=a75b205801e71a4b in_seq=0 out_seq=0 2024-04-01T01:39:54.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.024+0000 7f35e9ffb640 10 client.27241 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:39:54.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.025+0000 7f35e9ffb640 1 -- 172.21.15.70:0/1245917320 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5617babef6f0 con 0x5617bac93730 2024-04-01T01:39:54.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.025+0000 7f35e9ffb640 10 client.27241 handle_command_reply: tid=0 2024-04-01T01:39:54.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.025+0000 7f360b20c740 1 client.27241 shutdown 2024-04-01T01:39:54.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.025+0000 7f360b20c740 1 -- 172.21.15.70:0/1245917320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7d490 msgr2=0x5617bac87a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.025+0000 7f360b20c740 1 --2- 172.21.15.70:0/1245917320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7d490 0x5617bac87a00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009fa0 tx=0x7f35fc00fcc0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.025+0000 7f360b20c740 1 -- 172.21.15.70:0/1245917320 shutdown_connections 2024-04-01T01:39:54.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.025+0000 7f360b20c740 1 --2- 172.21.15.70:0/1245917320 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5617bac93730 0x5617bac95ba0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.025+0000 7f360b20c740 1 --2- 172.21.15.70:0/1245917320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac82ff0 0x5617bac8ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.025+0000 7f360b20c740 1 --2- 172.21.15.70:0/1245917320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac7de80 0x5617bac87f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.026+0000 7f360b20c740 1 --2- 172.21.15.70:0/1245917320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7d490 0x5617bac87a00 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.026+0000 7f360b20c740 1 -- 172.21.15.70:0/1245917320 >> 172.21.15.70:0/1245917320 conn(0x5617bac78a00 msgr2=0x5617bac77d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:54.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.026+0000 7f35e9ffb640 0 client.27241 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:39:54.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.026+0000 7f360b20c740 1 -- 172.21.15.70:0/1245917320 shutdown_connections 2024-04-01T01:39:54.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.026+0000 7f360b20c740 1 -- 172.21.15.70:0/1245917320 wait complete. 2024-04-01T01:39:54.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.026+0000 7f360b20c740 20 client.27241 trim_cache size 0 max 16384 2024-04-01T01:39:54.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.049+0000 7f360b20c740 1 Processor -- start 2024-04-01T01:39:54.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.049+0000 7f360b20c740 1 -- start start 2024-04-01T01:39:54.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f360b20c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac7d490 0x5617bac774b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f360b20c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7de80 0x5617bac8eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f360b20c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac82ff0 0x5617bac779f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f360b20c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f36001b09a0 con 0x5617bac7d490 2024-04-01T01:39:54.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f360b20c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f36001b24e0 con 0x5617bac82ff0 2024-04-01T01:39:54.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f360b20c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f36001b1070 con 0x5617bac7de80 2024-04-01T01:39:54.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f3605af0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7de80 0x5617bac8eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f36062f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac7d490 0x5617bac774b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f3605af0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7de80 0x5617bac8eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49892/0 (socket says 172.21.15.70:49892) 2024-04-01T01:39:54.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f3605af0640 1 -- 172.21.15.70:0/968341394 learned_addr learned my addr 172.21.15.70:0/968341394 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f3606af2640 1 --2- 172.21.15.70:0/968341394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac82ff0 0x5617bac779f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f3605af0640 1 -- 172.21.15.70:0/968341394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac82ff0 msgr2=0x5617bac779f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f3605af0640 1 --2- 172.21.15.70:0/968341394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac82ff0 0x5617bac779f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f3605af0640 1 -- 172.21.15.70:0/968341394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac7d490 msgr2=0x5617bac774b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f3605af0640 1 --2- 172.21.15.70:0/968341394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac7d490 0x5617bac774b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f3605af0640 1 -- 172.21.15.70:0/968341394 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617bac7fca0 con 0x5617bac7de80 2024-04-01T01:39:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f3606af2640 1 --2- 172.21.15.70:0/968341394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac82ff0 0x5617bac779f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:39:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.050+0000 7f36062f1640 1 --2- 172.21.15.70:0/968341394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac7d490 0x5617bac774b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:39:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.051+0000 7f3605af0640 1 --2- 172.21.15.70:0/968341394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7de80 0x5617bac8eed0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f35f0000c00 tx=0x7f35f0097040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cf4dd5e2c11c55c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.051+0000 7f35ebfff640 1 -- 172.21.15.70:0/968341394 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f0069e60 con 0x5617bac7de80 2024-04-01T01:39:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.051+0000 7f35ebfff640 1 -- 172.21.15.70:0/968341394 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35f009b3a0 con 0x5617bac7de80 2024-04-01T01:39:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.051+0000 7f35ebfff640 1 -- 172.21.15.70:0/968341394 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f0097a10 con 0x5617bac7de80 2024-04-01T01:39:54.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.052+0000 7f360b20c740 1 -- 172.21.15.70:0/968341394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7de80 msgr2=0x5617bac8eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.052+0000 7f360b20c740 1 --2- 172.21.15.70:0/968341394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7de80 0x5617bac8eed0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f35f0000c00 tx=0x7f35f0097040 comp rx=0 tx=0).stop 2024-04-01T01:39:54.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.052+0000 7f360b20c740 1 -- 172.21.15.70:0/968341394 shutdown_connections 2024-04-01T01:39:54.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.052+0000 7f360b20c740 1 --2- 172.21.15.70:0/968341394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac82ff0 0x5617bac779f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.052+0000 7f360b20c740 1 --2- 172.21.15.70:0/968341394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7de80 0x5617bac8eed0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.052+0000 7f360b20c740 1 --2- 172.21.15.70:0/968341394 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac7d490 0x5617bac774b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.052+0000 7f360b20c740 1 -- 172.21.15.70:0/968341394 >> 172.21.15.70:0/968341394 conn(0x5617bac78a00 msgr2=0x5617bac78de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:54.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.053+0000 7f360b20c740 1 -- 172.21.15.70:0/968341394 shutdown_connections 2024-04-01T01:39:54.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.053+0000 7f360b20c740 1 -- 172.21.15.70:0/968341394 wait complete. 2024-04-01T01:39:54.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.053+0000 7f360b20c740 1 -- start start 2024-04-01T01:39:54.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.053+0000 7f360b20c740 1 Processor -- start 2024-04-01T01:39:54.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.054+0000 7f360b20c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7d490 0x5617bac8bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.054+0000 7f360b20c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac7de80 0x5617bac8c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.054+0000 7f36062f1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7d490 0x5617bac8bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.054+0000 7f36062f1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7d490 0x5617bac8bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49898/0 (socket says 172.21.15.70:49898) 2024-04-01T01:39:54.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.054+0000 7f36062f1640 1 -- 172.21.15.70:0/3070221097 learned_addr learned my addr 172.21.15.70:0/3070221097 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:54.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.054+0000 7f360b20c740 1 --2- 172.21.15.70:0/3070221097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac82ff0 0x5617bacf7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.054+0000 7f360b20c740 1 -- 172.21.15.70:0/3070221097 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f36001b1e10 con 0x5617bac7de80 2024-04-01T01:39:54.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.054+0000 7f360b20c740 1 -- 172.21.15.70:0/3070221097 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f36001b1740 con 0x5617bac82ff0 2024-04-01T01:39:54.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.054+0000 7f3605af0640 1 --2- 172.21.15.70:0/3070221097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac7de80 0x5617bac8c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.055+0000 7f360b20c740 1 -- 172.21.15.70:0/3070221097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f36001ae980 con 0x5617bac7d490 2024-04-01T01:39:54.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.055+0000 7f360b20c740 10 client.0 _ll_register_callbacks cb 0x7f35fc00f8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:54.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.055+0000 7f3606af2640 1 --2- 172.21.15.70:0/3070221097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac82ff0 0x5617bacf7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.055+0000 7f36062f1640 1 -- 172.21.15.70:0/3070221097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac82ff0 msgr2=0x5617bacf7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:54.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.055+0000 7f36062f1640 1 --2- 172.21.15.70:0/3070221097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac82ff0 0x5617bacf7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.055+0000 7f36062f1640 1 -- 172.21.15.70:0/3070221097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac7de80 msgr2=0x5617bac8c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:54.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.055+0000 7f36062f1640 1 --2- 172.21.15.70:0/3070221097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac7de80 0x5617bac8c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.055+0000 7f36062f1640 1 -- 172.21.15.70:0/3070221097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617bacf8010 con 0x5617bac7d490 2024-04-01T01:39:54.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.055+0000 7f3605af0640 1 --2- 172.21.15.70:0/3070221097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac7de80 0x5617bac8c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:54.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.055+0000 7f3606af2640 1 --2- 172.21.15.70:0/3070221097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac82ff0 0x5617bacf7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:54.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.056+0000 7f36062f1640 1 --2- 172.21.15.70:0/3070221097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7d490 0x5617bac8bdf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00a010 tx=0x7f35fc009f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8e83371057520a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:54.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.056+0000 7f35e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:39:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.056+0000 7f35e9ffb640 1 -- 172.21.15.70:0/3070221097 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc029040 con 0x5617bac7d490 2024-04-01T01:39:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.056+0000 7f35e9ffb640 1 -- 172.21.15.70:0/3070221097 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f35fc019650 con 0x5617bac7d490 2024-04-01T01:39:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.056+0000 7f360b20c740 1 -- 172.21.15.70:0/3070221097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35f009b3a0 con 0x5617bac7d490 2024-04-01T01:39:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.056+0000 7f35e9ffb640 1 -- 172.21.15.70:0/3070221097 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f35fc019850 con 0x5617bac7d490 2024-04-01T01:39:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.056+0000 7f360b20c740 10 client.27247 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.056+0000 7f360b20c740 1 -- 172.21.15.70:0/3070221097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5617bacf9290 con 0x5617bac7d490 2024-04-01T01:39:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.056+0000 7f35e9ffb640 1 -- 172.21.15.70:0/3070221097 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35fc025b30 con 0x5617bac7d490 2024-04-01T01:39:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.057+0000 7f35e9ffb640 1 -- 172.21.15.70:0/3070221097 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f35fc0038f0 con 0x5617bac7d490 2024-04-01T01:39:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.057+0000 7f360b20c740 10 client.27247 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.057+0000 7f360b20c740 10 client.27247 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:39:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.057+0000 7f360b20c740 20 client.27247 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.057+0000 7f360b20c740 1 --2- 172.21.15.70:0/3070221097 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5617bacf9cc0 0x5617bacfc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.057+0000 7f360b20c740 4 client.27247 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:39:54.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.057+0000 7f360b20c740 1 -- 172.21.15.70:0/3070221097 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f35f009b3a0 con 0x5617bacf9cc0 2024-04-01T01:39:54.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.057+0000 7f3605af0640 1 --2- 172.21.15.70:0/3070221097 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5617bacf9cc0 0x5617bacfc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.057+0000 7f3605af0640 1 --2- 172.21.15.70:0/3070221097 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5617bacf9cc0 0x5617bacfc0e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f5f37847aef16b3c server_cookie=7634bc84a7b84ef1 in_seq=0 out_seq=0 2024-04-01T01:39:54.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.057+0000 7f35e9ffb640 10 client.27247 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:39:54.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.058+0000 7f35e9ffb640 1 -- 172.21.15.70:0/3070221097 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f35f009b3a0 con 0x5617bacf9cc0 2024-04-01T01:39:54.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.058+0000 7f35e9ffb640 10 client.27247 handle_command_reply: tid=0 2024-04-01T01:39:54.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.058+0000 7f360b20c740 1 client.27247 shutdown 2024-04-01T01:39:54.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.058+0000 7f360b20c740 1 -- 172.21.15.70:0/3070221097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7d490 msgr2=0x5617bac8bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.058+0000 7f360b20c740 1 --2- 172.21.15.70:0/3070221097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7d490 0x5617bac8bdf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00a010 tx=0x7f35fc009f10 comp rx=0 tx=0).stop 2024-04-01T01:39:54.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.058+0000 7f360b20c740 1 -- 172.21.15.70:0/3070221097 shutdown_connections 2024-04-01T01:39:54.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.058+0000 7f360b20c740 1 --2- 172.21.15.70:0/3070221097 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5617bacf9cc0 0x5617bacfc0e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.058+0000 7f360b20c740 1 --2- 172.21.15.70:0/3070221097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5617bac82ff0 0x5617bacf7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.058+0000 7f360b20c740 1 --2- 172.21.15.70:0/3070221097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617bac7de80 0x5617bac8c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.058+0000 7f360b20c740 1 --2- 172.21.15.70:0/3070221097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5617bac7d490 0x5617bac8bdf0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.058+0000 7f360b20c740 1 -- 172.21.15.70:0/3070221097 >> 172.21.15.70:0/3070221097 conn(0x5617bac78a00 msgr2=0x5617bac7bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:54.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.058+0000 7f35e9ffb640 0 client.27247 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:39:54.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.058+0000 7f360b20c740 1 -- 172.21.15.70:0/3070221097 shutdown_connections 2024-04-01T01:39:54.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.058+0000 7f360b20c740 1 -- 172.21.15.70:0/3070221097 wait complete. 2024-04-01T01:39:54.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.058+0000 7f360b20c740 20 client.27247 trim_cache size 0 max 16384 2024-04-01T01:39:54.061 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:39:54.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.059+0000 7f3608555640 1 -- 172.21.15.70:0/1600014674 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f35ec0a24f0 msgr2=0x7f35ec0a49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.059+0000 7f3608555640 1 --2- 172.21.15.70:0/1600014674 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f35ec0a24f0 0x7f35ec0a49b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x5617bac7e6e0 tx=0x7f35f802c0f0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.059+0000 7f3608555640 1 -- 172.21.15.70:0/1600014674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36001007b0 msgr2=0x7f360019e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.059+0000 7f3608555640 1 --2- 172.21.15.70:0/1600014674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36001007b0 0x7f360019e340 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f35f000e9c0 tx=0x7f35f000ee80 comp rx=0 tx=0).stop 2024-04-01T01:39:54.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.059+0000 7f3608555640 1 -- 172.21.15.70:0/1600014674 shutdown_connections 2024-04-01T01:39:54.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.059+0000 7f3608555640 1 --2- 172.21.15.70:0/1600014674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3600101170 0x7f36001a5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.059+0000 7f3608555640 1 --2- 172.21.15.70:0/1600014674 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f35ec0a24f0 0x7f35ec0a49b0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.059+0000 7f3608555640 1 --2- 172.21.15.70:0/1600014674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36001007b0 0x7f360019e340 secure :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f35f000e9c0 tx=0x7f35f000ee80 comp rx=0 tx=0).stop 2024-04-01T01:39:54.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.059+0000 7f3608555640 1 --2- 172.21.15.70:0/1600014674 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36000ffa50 0x7f360019de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.059+0000 7f3608555640 1 -- 172.21.15.70:0/1600014674 >> 172.21.15.70:0/1600014674 conn(0x7f36000fb380 msgr2=0x7f36001019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:54.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.059+0000 7f3608555640 1 -- 172.21.15.70:0/1600014674 shutdown_connections 2024-04-01T01:39:54.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.059+0000 7f3608555640 1 -- 172.21.15.70:0/1600014674 wait complete. 2024-04-01T01:39:54.074 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:39:54.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:53 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/317687592' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:39:54.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.237+0000 7f2185e54640 1 -- 172.21.15.70:0/3596341106 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2180103330 msgr2=0x7f2180109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.237+0000 7f2185e54640 1 --2- 172.21.15.70:0/3596341106 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2180103330 0x7f2180109bc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2178009ca0 tx=0x7f217802c100 comp rx=0 tx=0).stop 2024-04-01T01:39:54.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.238+0000 7f2185e54640 1 -- 172.21.15.70:0/3596341106 shutdown_connections 2024-04-01T01:39:54.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.238+0000 7f2185e54640 1 --2- 172.21.15.70:0/3596341106 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2180103330 0x7f2180109bc0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.238+0000 7f2185e54640 1 --2- 172.21.15.70:0/3596341106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2180102970 0x7f2180102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.238+0000 7f2185e54640 1 --2- 172.21.15.70:0/3596341106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2180101780 0x7f2180101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.238+0000 7f2185e54640 1 -- 172.21.15.70:0/3596341106 >> 172.21.15.70:0/3596341106 conn(0x7f21800fccf0 msgr2=0x7f21800ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:54.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.238+0000 7f2185e54640 1 -- 172.21.15.70:0/3596341106 shutdown_connections 2024-04-01T01:39:54.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.238+0000 7f2185e54640 1 -- 172.21.15.70:0/3596341106 wait complete. 2024-04-01T01:39:54.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.239+0000 7f2185e54640 1 Processor -- start 2024-04-01T01:39:54.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.239+0000 7f2185e54640 1 -- start start 2024-04-01T01:39:54.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.240+0000 7f2185e54640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2180101780 0x7f2180197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.240+0000 7f2185e54640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2180102970 0x7f2180197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.240+0000 7f217f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2180101780 0x7f2180197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.240+0000 7f217f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2180101780 0x7f2180197780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49920/0 (socket says 172.21.15.70:49920) 2024-04-01T01:39:54.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.240+0000 7f217f7fe640 1 -- 172.21.15.70:0/3840328510 learned_addr learned my addr 172.21.15.70:0/3840328510 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:54.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.240+0000 7f2185e54640 1 --2- 172.21.15.70:0/3840328510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2180103330 0x7f218019ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.240+0000 7f2185e54640 1 -- 172.21.15.70:0/3840328510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f218010d290 con 0x7f2180102970 2024-04-01T01:39:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.241+0000 7f2185e54640 1 -- 172.21.15.70:0/3840328510 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f218010d110 con 0x7f2180103330 2024-04-01T01:39:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.241+0000 7f2185e54640 1 -- 172.21.15.70:0/3840328510 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f218010d410 con 0x7f2180101780 2024-04-01T01:39:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.241+0000 7f217effd640 1 --2- 172.21.15.70:0/3840328510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2180102970 0x7f2180197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.241+0000 7f217ffff640 1 --2- 172.21.15.70:0/3840328510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2180103330 0x7f218019ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.241+0000 7f217effd640 1 -- 172.21.15.70:0/3840328510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2180101780 msgr2=0x7f2180197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.241+0000 7f217effd640 1 --2- 172.21.15.70:0/3840328510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2180101780 0x7f2180197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.241+0000 7f217effd640 1 -- 172.21.15.70:0/3840328510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2180103330 msgr2=0x7f218019ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.241+0000 7f217effd640 1 --2- 172.21.15.70:0/3840328510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2180103330 0x7f218019ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.241+0000 7f217effd640 1 -- 172.21.15.70:0/3840328510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f218019f3a0 con 0x7f2180102970 2024-04-01T01:39:54.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.241+0000 7f217f7fe640 1 --2- 172.21.15.70:0/3840328510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2180101780 0x7f2180197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:54.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.242+0000 7f217effd640 1 --2- 172.21.15.70:0/3840328510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2180102970 0x7f2180197cc0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f217400d6d0 tx=0x7f217400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:54.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.242+0000 7f217cff9640 1 -- 172.21.15.70:0/3840328510 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2174017070 con 0x7f2180102970 2024-04-01T01:39:54.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.242+0000 7f217cff9640 1 -- 172.21.15.70:0/3840328510 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2174009820 con 0x7f2180102970 2024-04-01T01:39:54.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.242+0000 7f217cff9640 1 -- 172.21.15.70:0/3840328510 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2174004c40 con 0x7f2180102970 2024-04-01T01:39:54.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.242+0000 7f2185e54640 1 -- 172.21.15.70:0/3840328510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f218019f690 con 0x7f2180102970 2024-04-01T01:39:54.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.242+0000 7f2185e54640 1 -- 172.21.15.70:0/3840328510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f218019fbe0 con 0x7f2180102970 2024-04-01T01:39:54.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.246+0000 7f217cff9640 1 -- 172.21.15.70:0/3840328510 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f21740099d0 con 0x7f2180102970 2024-04-01T01:39:54.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.246+0000 7f217cff9640 1 --2- 172.21.15.70:0/3840328510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21680a25f0 0x7f21680a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.246+0000 7f2188a31740 1 Processor -- start 2024-04-01T01:39:54.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f2188a31740 1 -- start start 2024-04-01T01:39:54.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f217cff9640 1 -- 172.21.15.70:0/3840328510 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f21740cde60 con 0x7f2180102970 2024-04-01T01:39:54.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f2188a31740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164b56b0 0x558b164b5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f2188a31740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b60a0 0x558b164b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f2188a31740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b9200 0x558b164cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f217f7fe640 1 --2- 172.21.15.70:0/3840328510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21680a25f0 0x7f21680a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f2188a31740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558b1641f1b0 con 0x558b164b60a0 2024-04-01T01:39:54.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f2188a31740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x558b163c7c00 con 0x558b164b9200 2024-04-01T01:39:54.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f2188a31740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558b1641b250 con 0x558b164b56b0 2024-04-01T01:39:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f217effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b60a0 0x558b164b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f217effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b60a0 0x558b164b8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44702/0 (socket says 172.21.15.70:44702) 2024-04-01T01:39:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f217effd640 1 -- 172.21.15.70:0/2553813080 learned_addr learned my addr 172.21.15.70:0/2553813080 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f217ffff640 1 --2- 172.21.15.70:0/2553813080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b9200 0x558b164cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f217f7fe640 1 --2- 172.21.15.70:0/2553813080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164b56b0 0x558b164b5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f217effd640 1 -- 172.21.15.70:0/2553813080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164b56b0 msgr2=0x558b164b5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f217effd640 1 --2- 172.21.15.70:0/2553813080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164b56b0 0x558b164b5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f217effd640 1 -- 172.21.15.70:0/2553813080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b9200 msgr2=0x558b164cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f217effd640 1 --2- 172.21.15.70:0/2553813080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b9200 0x558b164cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f217effd640 1 -- 172.21.15.70:0/2553813080 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b163e73d0 con 0x558b164b60a0 2024-04-01T01:39:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.247+0000 7f217ffff640 1 --2- 172.21.15.70:0/2553813080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b9200 0x558b164cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.248+0000 7f217f7fe640 1 --2- 172.21.15.70:0/3840328510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21680a25f0 0x7f21680a4ab0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2170005fd0 tx=0x7f2170012d80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.248+0000 7f217effd640 1 --2- 172.21.15.70:0/2553813080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b60a0 0x558b164b8b90 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f217409a610 tx=0x7f2174009d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cb877bb72985048 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:54.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.248+0000 7f2165ffb640 1 -- 172.21.15.70:0/2553813080 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21740b3070 con 0x558b164b60a0 2024-04-01T01:39:54.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:53 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/317687592' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:39:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.248+0000 7f2165ffb640 1 -- 172.21.15.70:0/2553813080 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f217409e720 con 0x558b164b60a0 2024-04-01T01:39:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.248+0000 7f2165ffb640 1 -- 172.21.15.70:0/2553813080 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2174020040 con 0x558b164b60a0 2024-04-01T01:39:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.249+0000 7f2188a31740 1 -- 172.21.15.70:0/2553813080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b60a0 msgr2=0x558b164b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.249+0000 7f2188a31740 1 --2- 172.21.15.70:0/2553813080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b60a0 0x558b164b8b90 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f217409a610 tx=0x7f2174009d40 comp rx=0 tx=0).stop 2024-04-01T01:39:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.249+0000 7f2188a31740 1 -- 172.21.15.70:0/2553813080 shutdown_connections 2024-04-01T01:39:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.249+0000 7f2188a31740 1 --2- 172.21.15.70:0/2553813080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b9200 0x558b164cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.249+0000 7f2188a31740 1 --2- 172.21.15.70:0/2553813080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b60a0 0x558b164b8b90 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.249+0000 7f2188a31740 1 --2- 172.21.15.70:0/2553813080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164b56b0 0x558b164b5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.249+0000 7f2188a31740 1 -- 172.21.15.70:0/2553813080 >> 172.21.15.70:0/2553813080 conn(0x558b164b9e10 msgr2=0x558b164ba210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.250+0000 7f2188a31740 1 -- 172.21.15.70:0/2553813080 shutdown_connections 2024-04-01T01:39:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.250+0000 7f2188a31740 1 -- 172.21.15.70:0/2553813080 wait complete. 2024-04-01T01:39:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.250+0000 7f2188a31740 1 -- start start 2024-04-01T01:39:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.251+0000 7f2188a31740 1 Processor -- start 2024-04-01T01:39:54.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.251+0000 7f2188a31740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b56b0 0x558b164cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.251+0000 7f2188a31740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164b9200 0x558b164cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.251+0000 7f2188a31740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164d28f0 0x558b164d4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.251+0000 7f2188a31740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558b164cb7c0 con 0x558b164d28f0 2024-04-01T01:39:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.251+0000 7f2188a31740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x558b163bd470 con 0x558b164b56b0 2024-04-01T01:39:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.251+0000 7f2188a31740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558b164cb940 con 0x558b164b9200 2024-04-01T01:39:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.251+0000 7f2188a31740 10 client.0 _ll_register_callbacks cb 0x558b163dacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.251+0000 7f217f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b56b0 0x558b164cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.251+0000 7f217f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b56b0 0x558b164cd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37700/0 (socket says 172.21.15.70:37700) 2024-04-01T01:39:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.251+0000 7f217f7fe640 1 -- 172.21.15.70:0/3448124806 learned_addr learned my addr 172.21.15.70:0/3448124806 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.251+0000 7f217effd640 1 --2- 172.21.15.70:0/3448124806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164b9200 0x558b164cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.251+0000 7f217ffff640 1 --2- 172.21.15.70:0/3448124806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164d28f0 0x558b164d4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.252+0000 7f217f7fe640 1 -- 172.21.15.70:0/3448124806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164b9200 msgr2=0x558b164cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.252+0000 7f217f7fe640 1 --2- 172.21.15.70:0/3448124806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164b9200 0x558b164cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.252+0000 7f217f7fe640 1 -- 172.21.15.70:0/3448124806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164d28f0 msgr2=0x558b164d4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:54.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.252+0000 7f217f7fe640 1 --2- 172.21.15.70:0/3448124806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164d28f0 0x558b164d4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.252+0000 7f217f7fe640 1 -- 172.21.15.70:0/3448124806 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b161333f0 con 0x558b164b56b0 2024-04-01T01:39:54.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.252+0000 7f217f7fe640 1 --2- 172.21.15.70:0/3448124806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b56b0 0x558b164cd380 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f217000edc0 tx=0x7f2170004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d931dba34be1bef server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:54.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.252+0000 7f215effd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:54.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.252+0000 7f215effd640 1 -- 172.21.15.70:0/3448124806 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2170005220 con 0x558b164b56b0 2024-04-01T01:39:54.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.252+0000 7f215effd640 1 -- 172.21.15.70:0/3448124806 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f21700053c0 con 0x558b164b56b0 2024-04-01T01:39:54.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.252+0000 7f2188a31740 1 -- 172.21.15.70:0/3448124806 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f217409e720 con 0x558b164b56b0 2024-04-01T01:39:54.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.253+0000 7f215effd640 1 -- 172.21.15.70:0/3448124806 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f217002f050 con 0x558b164b56b0 2024-04-01T01:39:54.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.253+0000 7f2188a31740 10 client.27398 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:54.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.253+0000 7f2188a31740 1 -- 172.21.15.70:0/3448124806 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558b161fd7d0 con 0x558b164b56b0 2024-04-01T01:39:54.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.253+0000 7f215effd640 1 -- 172.21.15.70:0/3448124806 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f217409e720 con 0x558b164b56b0 2024-04-01T01:39:54.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.253+0000 7f215effd640 1 -- 172.21.15.70:0/3448124806 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f217001f510 con 0x558b164b56b0 2024-04-01T01:39:54.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.253+0000 7f2188a31740 10 client.27398 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:54.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.253+0000 7f2188a31740 10 client.27398 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:54.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.253+0000 7f2188a31740 20 client.27398 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:54.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.253+0000 7f2188a31740 1 --2- 172.21.15.70:0/3448124806 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558b164d9190 0x558b164db630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.253+0000 7f2188a31740 4 client.27398 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:54.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.253+0000 7f2188a31740 1 -- 172.21.15.70:0/3448124806 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b1642c6f0 con 0x558b164d9190 2024-04-01T01:39:54.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.254+0000 7f217effd640 1 --2- 172.21.15.70:0/3448124806 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558b164d9190 0x558b164db630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.254+0000 7f215effd640 10 client.27398 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:54.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.254+0000 7f217effd640 1 --2- 172.21.15.70:0/3448124806 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558b164d9190 0x558b164db630 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fcaaae0260811b9 server_cookie=7ca0cb764e9a3bd6 in_seq=0 out_seq=0 2024-04-01T01:39:54.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.255+0000 7f215effd640 1 -- 172.21.15.70:0/3448124806 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558b1642c6f0 con 0x558b164d9190 2024-04-01T01:39:54.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.255+0000 7f215effd640 10 client.27398 handle_command_reply: tid=0 2024-04-01T01:39:54.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.255+0000 7f2188a31740 1 client.27398 shutdown 2024-04-01T01:39:54.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.255+0000 7f2188a31740 1 -- 172.21.15.70:0/3448124806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b56b0 msgr2=0x558b164cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.255+0000 7f2188a31740 1 --2- 172.21.15.70:0/3448124806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b56b0 0x558b164cd380 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f217000edc0 tx=0x7f2170004730 comp rx=0 tx=0).stop 2024-04-01T01:39:54.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.255+0000 7f2188a31740 1 -- 172.21.15.70:0/3448124806 shutdown_connections 2024-04-01T01:39:54.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.255+0000 7f2188a31740 1 --2- 172.21.15.70:0/3448124806 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558b164d9190 0x558b164db630 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.255+0000 7f2188a31740 1 --2- 172.21.15.70:0/3448124806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164d28f0 0x558b164d4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.255+0000 7f2188a31740 1 --2- 172.21.15.70:0/3448124806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164b9200 0x558b164cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.255+0000 7f2188a31740 1 --2- 172.21.15.70:0/3448124806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b56b0 0x558b164cd380 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.256+0000 7f2188a31740 1 -- 172.21.15.70:0/3448124806 >> 172.21.15.70:0/3448124806 conn(0x558b164b9e10 msgr2=0x558b164bb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:54.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.256+0000 7f215effd640 0 client.27398 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:54.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.256+0000 7f2188a31740 1 -- 172.21.15.70:0/3448124806 shutdown_connections 2024-04-01T01:39:54.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.256+0000 7f2188a31740 1 -- 172.21.15.70:0/3448124806 wait complete. 2024-04-01T01:39:54.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.256+0000 7f2188a31740 20 client.27398 trim_cache size 0 max 16384 2024-04-01T01:39:54.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.282+0000 7f2188a31740 1 Processor -- start 2024-04-01T01:39:54.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.282+0000 7f2188a31740 1 -- start start 2024-04-01T01:39:54.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.282+0000 7f2188a31740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b56b0 0x558b164cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.283+0000 7f217f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b56b0 0x558b164cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.283+0000 7f217f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b56b0 0x558b164cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44710/0 (socket says 172.21.15.70:44710) 2024-04-01T01:39:54.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.283+0000 7f217f7fe640 1 -- 172.21.15.70:0/1539915826 learned_addr learned my addr 172.21.15.70:0/1539915826 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:54.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.283+0000 7f2188a31740 1 --2- 172.21.15.70:0/1539915826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b9200 0x558b164b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.283+0000 7f2188a31740 1 --2- 172.21.15.70:0/1539915826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164d28f0 0x558b164b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.283+0000 7f2188a31740 1 -- 172.21.15.70:0/1539915826 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f21801a8040 con 0x558b164b56b0 2024-04-01T01:39:54.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.283+0000 7f2188a31740 1 -- 172.21.15.70:0/1539915826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f21801a9b80 con 0x558b164b9200 2024-04-01T01:39:54.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.283+0000 7f2188a31740 1 -- 172.21.15.70:0/1539915826 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f21801a8710 con 0x558b164d28f0 2024-04-01T01:39:54.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.283+0000 7f217effd640 1 --2- 172.21.15.70:0/1539915826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b9200 0x558b164b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.283+0000 7f217ffff640 1 --2- 172.21.15.70:0/1539915826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164d28f0 0x558b164b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.283+0000 7f217effd640 1 -- 172.21.15.70:0/1539915826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164d28f0 msgr2=0x558b164b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:54.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.283+0000 7f217effd640 1 --2- 172.21.15.70:0/1539915826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164d28f0 0x558b164b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.283+0000 7f217effd640 1 -- 172.21.15.70:0/1539915826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b56b0 msgr2=0x558b164cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:54.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.283+0000 7f217effd640 1 --2- 172.21.15.70:0/1539915826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b56b0 0x558b164cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.283+0000 7f217effd640 1 -- 172.21.15.70:0/1539915826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b164c8170 con 0x558b164b9200 2024-04-01T01:39:54.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.283+0000 7f217f7fe640 1 --2- 172.21.15.70:0/1539915826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b56b0 0x558b164cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:54.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.284+0000 7f217effd640 1 --2- 172.21.15.70:0/1539915826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b9200 0x558b164b44b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f217409a610 tx=0x7f2174096260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d27fca5e3d8598b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:54.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.284+0000 7f21657fa640 1 -- 172.21.15.70:0/1539915826 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21740b9070 con 0x558b164b9200 2024-04-01T01:39:54.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.284+0000 7f21657fa640 1 -- 172.21.15.70:0/1539915826 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21740b7070 con 0x558b164b9200 2024-04-01T01:39:54.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.284+0000 7f21657fa640 1 -- 172.21.15.70:0/1539915826 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2174020040 con 0x558b164b9200 2024-04-01T01:39:54.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.285+0000 7f2188a31740 1 -- 172.21.15.70:0/1539915826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b9200 msgr2=0x558b164b44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.285+0000 7f2188a31740 1 --2- 172.21.15.70:0/1539915826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b9200 0x558b164b44b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f217409a610 tx=0x7f2174096260 comp rx=0 tx=0).stop 2024-04-01T01:39:54.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.285+0000 7f2188a31740 1 -- 172.21.15.70:0/1539915826 shutdown_connections 2024-04-01T01:39:54.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.285+0000 7f2188a31740 1 --2- 172.21.15.70:0/1539915826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164d28f0 0x558b164b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.285+0000 7f2188a31740 1 --2- 172.21.15.70:0/1539915826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164b9200 0x558b164b44b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.285+0000 7f2188a31740 1 --2- 172.21.15.70:0/1539915826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b56b0 0x558b164cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.285+0000 7f2188a31740 1 -- 172.21.15.70:0/1539915826 >> 172.21.15.70:0/1539915826 conn(0x558b164b9e10 msgr2=0x558b164ccf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:54.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.285+0000 7f2188a31740 1 -- 172.21.15.70:0/1539915826 shutdown_connections 2024-04-01T01:39:54.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.285+0000 7f2188a31740 1 -- 172.21.15.70:0/1539915826 wait complete. 2024-04-01T01:39:54.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.286+0000 7f2188a31740 1 -- start start 2024-04-01T01:39:54.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.286+0000 7f2188a31740 1 Processor -- start 2024-04-01T01:39:54.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.286+0000 7f2188a31740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b56b0 0x558b164ea1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.286+0000 7f217f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b56b0 0x558b164ea1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.286+0000 7f217f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b56b0 0x558b164ea1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44716/0 (socket says 172.21.15.70:44716) 2024-04-01T01:39:54.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.286+0000 7f217f7fe640 1 -- 172.21.15.70:0/744640135 learned_addr learned my addr 172.21.15.70:0/744640135 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:54.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.286+0000 7f2188a31740 1 --2- 172.21.15.70:0/744640135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164b9200 0x558b164ea710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.286+0000 7f2188a31740 1 --2- 172.21.15.70:0/744640135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164d28f0 0x558b164eb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.286+0000 7f2188a31740 1 -- 172.21.15.70:0/744640135 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f21801a94b0 con 0x558b164b56b0 2024-04-01T01:39:54.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.286+0000 7f2188a31740 1 -- 172.21.15.70:0/744640135 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f21801a8de0 con 0x558b164d28f0 2024-04-01T01:39:54.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.286+0000 7f2188a31740 1 -- 172.21.15.70:0/744640135 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f21801a6020 con 0x558b164b9200 2024-04-01T01:39:54.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.287+0000 7f2188a31740 10 client.0 _ll_register_callbacks cb 0x7f21700126a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:54.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.287+0000 7f217effd640 1 --2- 172.21.15.70:0/744640135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164b9200 0x558b164ea710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.287+0000 7f217ffff640 1 --2- 172.21.15.70:0/744640135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164d28f0 0x558b164eb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.287+0000 7f217f7fe640 1 -- 172.21.15.70:0/744640135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164b9200 msgr2=0x558b164ea710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:54.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.287+0000 7f217f7fe640 1 --2- 172.21.15.70:0/744640135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164b9200 0x558b164ea710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.287+0000 7f217f7fe640 1 -- 172.21.15.70:0/744640135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164d28f0 msgr2=0x558b164eb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:54.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.287+0000 7f217f7fe640 1 --2- 172.21.15.70:0/744640135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164d28f0 0x558b164eb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.287+0000 7f217f7fe640 1 -- 172.21.15.70:0/744640135 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b164ebd10 con 0x558b164b56b0 2024-04-01T01:39:54.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.287+0000 7f217ffff640 1 --2- 172.21.15.70:0/744640135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164d28f0 0x558b164eb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:54.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.287+0000 7f217f7fe640 1 --2- 172.21.15.70:0/744640135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b56b0 0x558b164ea1d0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f217001fad0 tx=0x7f217001fd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a61c86189ce00429 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:54.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.287+0000 7f215effd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:39:54.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.287+0000 7f215effd640 1 -- 172.21.15.70:0/744640135 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217000fb60 con 0x558b164b56b0 2024-04-01T01:39:54.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.288+0000 7f215effd640 1 -- 172.21.15.70:0/744640135 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f2170035440 con 0x558b164b56b0 2024-04-01T01:39:54.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.288+0000 7f2188a31740 1 -- 172.21.15.70:0/744640135 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21740b7070 con 0x558b164b56b0 2024-04-01T01:39:54.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.288+0000 7f215effd640 1 -- 172.21.15.70:0/744640135 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2170043050 con 0x558b164b56b0 2024-04-01T01:39:54.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.288+0000 7f2188a31740 10 client.17505 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:54.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.288+0000 7f2188a31740 1 -- 172.21.15.70:0/744640135 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558b1653f2a0 con 0x558b164b56b0 2024-04-01T01:39:54.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.289+0000 7f215effd640 1 -- 172.21.15.70:0/744640135 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f21700124d0 con 0x558b164b56b0 2024-04-01T01:39:54.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.289+0000 7f215effd640 1 -- 172.21.15.70:0/744640135 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f2170027400 con 0x558b164b56b0 2024-04-01T01:39:54.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.289+0000 7f2188a31740 10 client.17505 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:54.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.289+0000 7f2188a31740 10 client.17505 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:54.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.289+0000 7f2188a31740 20 client.17505 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:54.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.289+0000 7f2188a31740 1 --2- 172.21.15.70:0/744640135 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558b1653fc70 0x558b16542090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:54.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.289+0000 7f2188a31740 4 client.17505 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:39:54.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.289+0000 7f2188a31740 1 -- 172.21.15.70:0/744640135 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f21740b7070 con 0x558b1653fc70 2024-04-01T01:39:54.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.289+0000 7f217effd640 1 --2- 172.21.15.70:0/744640135 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558b1653fc70 0x558b16542090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:54.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.290+0000 7f215effd640 10 client.17505 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:54.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.290+0000 7f217effd640 1 --2- 172.21.15.70:0/744640135 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558b1653fc70 0x558b16542090 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cac7a0c4c506ea6 server_cookie=edc2bef6e2075e6b in_seq=0 out_seq=0 2024-04-01T01:39:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.291+0000 7f215effd640 1 -- 172.21.15.70:0/744640135 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f21740b7070 con 0x558b1653fc70 2024-04-01T01:39:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.291+0000 7f215effd640 10 client.17505 handle_command_reply: tid=0 2024-04-01T01:39:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.291+0000 7f2188a31740 1 client.17505 shutdown 2024-04-01T01:39:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.291+0000 7f2188a31740 1 -- 172.21.15.70:0/744640135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b56b0 msgr2=0x558b164ea1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.291+0000 7f2188a31740 1 --2- 172.21.15.70:0/744640135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b56b0 0x558b164ea1d0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f217001fad0 tx=0x7f217001fd80 comp rx=0 tx=0).stop 2024-04-01T01:39:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.291+0000 7f2188a31740 1 -- 172.21.15.70:0/744640135 shutdown_connections 2024-04-01T01:39:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.291+0000 7f2188a31740 1 --2- 172.21.15.70:0/744640135 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558b1653fc70 0x558b16542090 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.291+0000 7f2188a31740 1 --2- 172.21.15.70:0/744640135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558b164d28f0 0x558b164eb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.291+0000 7f2188a31740 1 --2- 172.21.15.70:0/744640135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558b164b9200 0x558b164ea710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.291+0000 7f215effd640 0 client.17505 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.291+0000 7f2188a31740 1 --2- 172.21.15.70:0/744640135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558b164b56b0 0x558b164ea1d0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.291+0000 7f2188a31740 1 -- 172.21.15.70:0/744640135 >> 172.21.15.70:0/744640135 conn(0x558b164b9e10 msgr2=0x558b164bd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.292+0000 7f2188a31740 1 -- 172.21.15.70:0/744640135 shutdown_connections 2024-04-01T01:39:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.292+0000 7f2188a31740 1 -- 172.21.15.70:0/744640135 wait complete. 2024-04-01T01:39:54.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.292+0000 7f2188a31740 20 client.17505 trim_cache size 0 max 16384 2024-04-01T01:39:54.294 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:39:54.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.293+0000 7f2185e54640 1 -- 172.21.15.70:0/3840328510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21680a25f0 msgr2=0x7f21680a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.293+0000 7f2185e54640 1 --2- 172.21.15.70:0/3840328510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21680a25f0 0x7f21680a4ab0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2170005fd0 tx=0x7f2170012d80 comp rx=0 tx=0).stop 2024-04-01T01:39:54.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.293+0000 7f2185e54640 1 -- 172.21.15.70:0/3840328510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2180102970 msgr2=0x7f2180197cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:54.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.293+0000 7f2185e54640 1 --2- 172.21.15.70:0/3840328510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2180102970 0x7f2180197cc0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f217400d6d0 tx=0x7f217400db90 comp rx=0 tx=0).stop 2024-04-01T01:39:54.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.293+0000 7f2185e54640 1 -- 172.21.15.70:0/3840328510 shutdown_connections 2024-04-01T01:39:54.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.293+0000 7f2185e54640 1 --2- 172.21.15.70:0/3840328510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2180103330 0x7f218019ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.293+0000 7f2185e54640 1 --2- 172.21.15.70:0/3840328510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2180102970 0x7f2180197cc0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.293+0000 7f2185e54640 1 --2- 172.21.15.70:0/3840328510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f21680a25f0 0x7f21680a4ab0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.293+0000 7f2185e54640 1 --2- 172.21.15.70:0/3840328510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2180101780 0x7f2180197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:54.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.293+0000 7f2185e54640 1 -- 172.21.15.70:0/3840328510 >> 172.21.15.70:0/3840328510 conn(0x7f21800fccf0 msgr2=0x7f2180105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:54.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.293+0000 7f2185e54640 1 -- 172.21.15.70:0/3840328510 shutdown_connections 2024-04-01T01:39:54.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:54.293+0000 7f2185e54640 1 -- 172.21.15.70:0/3840328510 wait complete. 2024-04-01T01:39:54.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:53 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/317687592' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:39:55.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:54 smithi189 ceph-mon[35753]: pgmap v552: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 15 MiB/s wr, 898 op/s 2024-04-01T01:39:55.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:54 smithi070 ceph-mon[32179]: pgmap v552: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 15 MiB/s wr, 898 op/s 2024-04-01T01:39:55.310 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:39:55.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:39:55.310 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:39:55.327 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:54 smithi081 ceph-mon[35142]: pgmap v552: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 15 MiB/s wr, 898 op/s 2024-04-01T01:39:55.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.492+0000 7f46accdc640 1 -- 172.21.15.70:0/1497684411 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a8102110 msgr2=0x7f46a81089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.492+0000 7f46accdc640 1 --2- 172.21.15.70:0/1497684411 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a8102110 0x7f46a81089a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009ca0 tx=0x7f46a002c120 comp rx=0 tx=0).stop 2024-04-01T01:39:55.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.493+0000 7f46accdc640 1 -- 172.21.15.70:0/1497684411 shutdown_connections 2024-04-01T01:39:55.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.493+0000 7f46accdc640 1 --2- 172.21.15.70:0/1497684411 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a8102110 0x7f46a81089a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.493+0000 7f46accdc640 1 --2- 172.21.15.70:0/1497684411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46a8101750 0x7f46a8101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.493+0000 7f46accdc640 1 --2- 172.21.15.70:0/1497684411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46a8100560 0x7f46a8100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.493+0000 7f46accdc640 1 -- 172.21.15.70:0/1497684411 >> 172.21.15.70:0/1497684411 conn(0x7f46a80fbad0 msgr2=0x7f46a80fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:55.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.493+0000 7f46accdc640 1 -- 172.21.15.70:0/1497684411 shutdown_connections 2024-04-01T01:39:55.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.493+0000 7f46accdc640 1 -- 172.21.15.70:0/1497684411 wait complete. 2024-04-01T01:39:55.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.494+0000 7f46accdc640 1 Processor -- start 2024-04-01T01:39:55.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.494+0000 7f46accdc640 1 -- start start 2024-04-01T01:39:55.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.494+0000 7f46accdc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46a8100560 0x7f46a8198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.494+0000 7f46accdc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46a8101750 0x7f46a8199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.494+0000 7f46accdc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a8102110 0x7f46a8195760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.494+0000 7f46accdc640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f46a810aff0 con 0x7f46a8100560 2024-04-01T01:39:55.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.494+0000 7f46accdc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46a810ae70 con 0x7f46a8102110 2024-04-01T01:39:55.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.494+0000 7f46accdc640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f46a810b170 con 0x7f46a8101750 2024-04-01T01:39:55.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.494+0000 7f46a6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46a8101750 0x7f46a8199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.494+0000 7f46a6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46a8101750 0x7f46a8199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49974/0 (socket says 172.21.15.70:49974) 2024-04-01T01:39:55.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.494+0000 7f46a6ffd640 1 -- 172.21.15.70:0/733707237 learned_addr learned my addr 172.21.15.70:0/733707237 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:55.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.494+0000 7f46a7fff640 1 --2- 172.21.15.70:0/733707237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a8102110 0x7f46a8195760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.495+0000 7f46a7fff640 1 -- 172.21.15.70:0/733707237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46a8101750 msgr2=0x7f46a8199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.495+0000 7f46a7fff640 1 --2- 172.21.15.70:0/733707237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46a8101750 0x7f46a8199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.495+0000 7f46a7fff640 1 -- 172.21.15.70:0/733707237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46a8100560 msgr2=0x7f46a8198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.495+0000 7f46a7fff640 1 --2- 172.21.15.70:0/733707237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46a8100560 0x7f46a8198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.495+0000 7f46a7fff640 1 -- 172.21.15.70:0/733707237 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a8196020 con 0x7f46a8102110 2024-04-01T01:39:55.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.495+0000 7f46a7fff640 1 --2- 172.21.15.70:0/733707237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a8102110 0x7f46a8195760 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f46a0037040 tx=0x7f46a0032810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:55.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.496+0000 7f46a4ff9640 1 -- 172.21.15.70:0/733707237 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a0041020 con 0x7f46a8102110 2024-04-01T01:39:55.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.496+0000 7f46accdc640 1 -- 172.21.15.70:0/733707237 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a8199690 con 0x7f46a8102110 2024-04-01T01:39:55.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.496+0000 7f46accdc640 1 -- 172.21.15.70:0/733707237 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a8199c20 con 0x7f46a8102110 2024-04-01T01:39:55.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.496+0000 7f46a4ff9640 1 -- 172.21.15.70:0/733707237 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46a0034e10 con 0x7f46a8102110 2024-04-01T01:39:55.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.496+0000 7f46a4ff9640 1 -- 172.21.15.70:0/733707237 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a0039040 con 0x7f46a8102110 2024-04-01T01:39:55.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.499+0000 7f46b0bf7740 1 Processor -- start 2024-04-01T01:39:55.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.500+0000 7f46b0bf7740 1 -- start start 2024-04-01T01:39:55.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.500+0000 7f46a4ff9640 1 -- 172.21.15.70:0/733707237 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f46a0044020 con 0x7f46a8102110 2024-04-01T01:39:55.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.500+0000 7f46b0bf7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e132bbf0 0x5642e132c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.500+0000 7f46b0bf7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132c5e0 0x5642e13310e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.500+0000 7f46b0bf7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e1331750 0x5642e1333be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.500+0000 7f46b0bf7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5642e122d0b0 con 0x5642e1331750 2024-04-01T01:39:55.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.500+0000 7f46b0bf7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5642e128fea0 con 0x5642e132bbf0 2024-04-01T01:39:55.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.500+0000 7f46b0bf7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5642e1238c00 con 0x5642e132c5e0 2024-04-01T01:39:55.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.500+0000 7f46a77fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132c5e0 0x5642e13310e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.500+0000 7f46a77fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132c5e0 0x5642e13310e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49976/0 (socket says 172.21.15.70:49976) 2024-04-01T01:39:55.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.500+0000 7f46a77fe640 1 -- 172.21.15.70:0/3856800422 learned_addr learned my addr 172.21.15.70:0/3856800422 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:55.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.500+0000 7f46a6ffd640 1 --2- 172.21.15.70:0/3856800422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e132bbf0 0x5642e132c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.500+0000 7f46a6ffd640 1 --2- 172.21.15.70:0/3856800422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e1331750 0x5642e1333be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.501+0000 7f46a77fe640 1 -- 172.21.15.70:0/3856800422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e132bbf0 msgr2=0x5642e132c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:55.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.501+0000 7f46a77fe640 1 --2- 172.21.15.70:0/3856800422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e132bbf0 0x5642e132c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.501+0000 7f46a77fe640 1 -- 172.21.15.70:0/3856800422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e1331750 msgr2=0x5642e1333be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:55.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.501+0000 7f46a77fe640 1 --2- 172.21.15.70:0/3856800422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e1331750 0x5642e1333be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.501+0000 7f46a77fe640 1 -- 172.21.15.70:0/3856800422 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642e106c7e0 con 0x5642e132c5e0 2024-04-01T01:39:55.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.501+0000 7f46a4ff9640 1 --2- 172.21.15.70:0/733707237 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46900a2580 0x7f46900a4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.502+0000 7f46a4ff9640 1 -- 172.21.15.70:0/733707237 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f46a00f0c30 con 0x7f46a8102110 2024-04-01T01:39:55.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.502+0000 7f46a77fe640 1 --2- 172.21.15.70:0/3856800422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132c5e0 0x5642e13310e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f46980097f0 tx=0x7f4698009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30b9d064ab5a5fd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:55.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.502+0000 7f46a6ffd640 1 --2- 172.21.15.70:0/733707237 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46900a2580 0x7f46900a4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.502+0000 7f468dffb640 1 -- 172.21.15.70:0/3856800422 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4698015020 con 0x5642e132c5e0 2024-04-01T01:39:55.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.502+0000 7f468dffb640 1 -- 172.21.15.70:0/3856800422 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4698011420 con 0x5642e132c5e0 2024-04-01T01:39:55.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.503+0000 7f468dffb640 1 -- 172.21.15.70:0/3856800422 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46980105c0 con 0x5642e132c5e0 2024-04-01T01:39:55.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.503+0000 7f46b0bf7740 1 -- 172.21.15.70:0/3856800422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132c5e0 msgr2=0x5642e13310e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.503+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/3856800422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132c5e0 0x5642e13310e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f46980097f0 tx=0x7f4698009cb0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.503+0000 7f46a6ffd640 1 --2- 172.21.15.70:0/733707237 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46900a2580 0x7f46900a4a40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x5642e132ce40 tx=0x7f469c013040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:55.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.504+0000 7f46b0bf7740 1 -- 172.21.15.70:0/3856800422 shutdown_connections 2024-04-01T01:39:55.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.504+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/3856800422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e1331750 0x5642e1333be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.504+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/3856800422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132c5e0 0x5642e13310e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.504+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/3856800422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e132bbf0 0x5642e132c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.504+0000 7f46b0bf7740 1 -- 172.21.15.70:0/3856800422 >> 172.21.15.70:0/3856800422 conn(0x5642e1327160 msgr2=0x5642e1327560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:55.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.505+0000 7f46b0bf7740 1 -- 172.21.15.70:0/3856800422 shutdown_connections 2024-04-01T01:39:55.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.505+0000 7f46b0bf7740 1 -- 172.21.15.70:0/3856800422 wait complete. 2024-04-01T01:39:55.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.505+0000 7f46b0bf7740 1 -- start start 2024-04-01T01:39:55.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.505+0000 7f46b0bf7740 1 Processor -- start 2024-04-01T01:39:55.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.506+0000 7f46b0bf7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132bbf0 0x5642e1347030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.506+0000 7f46b0bf7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e132c5e0 0x5642e1347570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.506+0000 7f46b0bf7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e1331750 0x5642e134e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.506+0000 7f46b0bf7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5642e122e3d0 con 0x5642e1331750 2024-04-01T01:39:55.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.506+0000 7f46b0bf7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5642e128bf40 con 0x5642e132c5e0 2024-04-01T01:39:55.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.506+0000 7f46b0bf7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5642e1345760 con 0x5642e132bbf0 2024-04-01T01:39:55.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.506+0000 7f46b0bf7740 10 client.0 _ll_register_callbacks cb 0x5642e124bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:55.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.506+0000 7f46a77fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132bbf0 0x5642e1347030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.506+0000 7f46a77fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132bbf0 0x5642e1347030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49978/0 (socket says 172.21.15.70:49978) 2024-04-01T01:39:55.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.506+0000 7f46a77fe640 1 -- 172.21.15.70:0/944863292 learned_addr learned my addr 172.21.15.70:0/944863292 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:55.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.506+0000 7f46a6ffd640 1 --2- 172.21.15.70:0/944863292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e132c5e0 0x5642e1347570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.507+0000 7f46a77fe640 1 -- 172.21.15.70:0/944863292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e132c5e0 msgr2=0x5642e1347570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:55.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.507+0000 7f46a77fe640 1 --2- 172.21.15.70:0/944863292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e132c5e0 0x5642e1347570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.507+0000 7f46a77fe640 1 -- 172.21.15.70:0/944863292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e1331750 msgr2=0x5642e134e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:55.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.507+0000 7f46a77fe640 1 --2- 172.21.15.70:0/944863292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e1331750 0x5642e134e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.507+0000 7f46a77fe640 1 -- 172.21.15.70:0/944863292 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642e10ccf70 con 0x5642e132bbf0 2024-04-01T01:39:55.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.507+0000 7f46a77fe640 1 --2- 172.21.15.70:0/944863292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132bbf0 0x5642e1347030 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4698016000 tx=0x7f4698009530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13ad7fdff896ec96 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:55.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.507+0000 7f4686ffd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:39:55.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.507+0000 7f4686ffd640 1 -- 172.21.15.70:0/944863292 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4698027060 con 0x5642e132bbf0 2024-04-01T01:39:55.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.507+0000 7f46b0bf7740 1 -- 172.21.15.70:0/944863292 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4698011420 con 0x5642e132bbf0 2024-04-01T01:39:55.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.507+0000 7f4686ffd640 1 -- 172.21.15.70:0/944863292 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f46980118a0 con 0x5642e132bbf0 2024-04-01T01:39:55.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.508+0000 7f4686ffd640 1 -- 172.21.15.70:0/944863292 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4698015020 con 0x5642e132bbf0 2024-04-01T01:39:55.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.508+0000 7f46b0bf7740 10 client.27277 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:55.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.508+0000 7f46b0bf7740 1 -- 172.21.15.70:0/944863292 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5642e0f64170 con 0x5642e132bbf0 2024-04-01T01:39:55.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.511+0000 7f4686ffd640 1 -- 172.21.15.70:0/944863292 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4698011420 con 0x5642e132bbf0 2024-04-01T01:39:55.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.512+0000 7f4686ffd640 1 -- 172.21.15.70:0/944863292 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f469801aac0 con 0x5642e132bbf0 2024-04-01T01:39:55.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.512+0000 7f46b0bf7740 10 client.27277 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:55.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.512+0000 7f46b0bf7740 10 client.27277 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:55.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.512+0000 7f46b0bf7740 20 client.27277 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:55.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.512+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/944863292 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5642e1352890 0x5642e1354ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.512+0000 7f46b0bf7740 4 client.27277 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:55.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.512+0000 7f46b0bf7740 1 -- 172.21.15.70:0/944863292 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642e129d3e0 con 0x5642e1352890 2024-04-01T01:39:55.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.513+0000 7f46a6ffd640 1 --2- 172.21.15.70:0/944863292 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5642e1352890 0x5642e1354ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.515+0000 7f46a6ffd640 1 --2- 172.21.15.70:0/944863292 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5642e1352890 0x5642e1354ce0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59f2389272f8b4b5 server_cookie=c8c46c468c350d3 in_seq=0 out_seq=0 2024-04-01T01:39:55.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.515+0000 7f4686ffd640 10 client.27277 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:55.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.515+0000 7f4686ffd640 1 -- 172.21.15.70:0/944863292 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5642e129d3e0 con 0x5642e1352890 2024-04-01T01:39:55.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.515+0000 7f4686ffd640 10 client.27277 handle_command_reply: tid=0 2024-04-01T01:39:55.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.515+0000 7f46b0bf7740 1 client.27277 shutdown 2024-04-01T01:39:55.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.516+0000 7f46b0bf7740 1 -- 172.21.15.70:0/944863292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132bbf0 msgr2=0x5642e1347030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.516+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/944863292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132bbf0 0x5642e1347030 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4698016000 tx=0x7f4698009530 comp rx=0 tx=0).stop 2024-04-01T01:39:55.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.516+0000 7f46b0bf7740 1 -- 172.21.15.70:0/944863292 shutdown_connections 2024-04-01T01:39:55.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.516+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/944863292 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5642e1352890 0x5642e1354ce0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.516+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/944863292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e1331750 0x5642e134e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.516+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/944863292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e132c5e0 0x5642e1347570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.516+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/944863292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132bbf0 0x5642e1347030 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.516+0000 7f46b0bf7740 1 -- 172.21.15.70:0/944863292 >> 172.21.15.70:0/944863292 conn(0x5642e1327160 msgr2=0x5642e13262b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:55.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.516+0000 7f4686ffd640 0 client.27277 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:55.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.516+0000 7f46b0bf7740 1 -- 172.21.15.70:0/944863292 shutdown_connections 2024-04-01T01:39:55.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.516+0000 7f46b0bf7740 1 -- 172.21.15.70:0/944863292 wait complete. 2024-04-01T01:39:55.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.516+0000 7f46b0bf7740 20 client.27277 trim_cache size 0 max 16384 2024-04-01T01:39:55.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.556+0000 7f46b0bf7740 1 Processor -- start 2024-04-01T01:39:55.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.556+0000 7f46b0bf7740 1 -- start start 2024-04-01T01:39:55.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.557+0000 7f46b0bf7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132bbf0 0x5642e1326130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.557+0000 7f46b0bf7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e132c5e0 0x5642e134e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.557+0000 7f46b0bf7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e1331750 0x5642e1346670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.557+0000 7f46b0bf7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f46a806d390 con 0x5642e132c5e0 2024-04-01T01:39:55.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.557+0000 7f46b0bf7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46a806eed0 con 0x5642e1331750 2024-04-01T01:39:55.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.557+0000 7f46b0bf7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f46a806da60 con 0x5642e132bbf0 2024-04-01T01:39:55.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.557+0000 7f46a77fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132bbf0 0x5642e1326130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.557+0000 7f46a77fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132bbf0 0x5642e1326130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:49988/0 (socket says 172.21.15.70:49988) 2024-04-01T01:39:55.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.557+0000 7f46a77fe640 1 -- 172.21.15.70:0/1295888820 learned_addr learned my addr 172.21.15.70:0/1295888820 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:55.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.557+0000 7f46a7fff640 1 --2- 172.21.15.70:0/1295888820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e1331750 0x5642e1346670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.557+0000 7f46a6ffd640 1 --2- 172.21.15.70:0/1295888820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e132c5e0 0x5642e134e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.558+0000 7f46a77fe640 1 -- 172.21.15.70:0/1295888820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e1331750 msgr2=0x5642e1346670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:55.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.558+0000 7f46a77fe640 1 --2- 172.21.15.70:0/1295888820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e1331750 0x5642e1346670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.558+0000 7f46a77fe640 1 -- 172.21.15.70:0/1295888820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e132c5e0 msgr2=0x5642e134e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:55.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.558+0000 7f46a77fe640 1 --2- 172.21.15.70:0/1295888820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e132c5e0 0x5642e134e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.558+0000 7f46a77fe640 1 -- 172.21.15.70:0/1295888820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642e13458e0 con 0x5642e132bbf0 2024-04-01T01:39:55.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.558+0000 7f46a77fe640 1 --2- 172.21.15.70:0/1295888820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132bbf0 0x5642e1326130 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f469801ae90 tx=0x7f4698009640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0031619f53779d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:55.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.558+0000 7f468d7fa640 1 -- 172.21.15.70:0/1295888820 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4698027050 con 0x5642e132bbf0 2024-04-01T01:39:55.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.559+0000 7f468d7fa640 1 -- 172.21.15.70:0/1295888820 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4698022e10 con 0x5642e132bbf0 2024-04-01T01:39:55.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.559+0000 7f468d7fa640 1 -- 172.21.15.70:0/1295888820 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469800bb30 con 0x5642e132bbf0 2024-04-01T01:39:55.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.560+0000 7f46b0bf7740 1 -- 172.21.15.70:0/1295888820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132bbf0 msgr2=0x5642e1326130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.560+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/1295888820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132bbf0 0x5642e1326130 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f469801ae90 tx=0x7f4698009640 comp rx=0 tx=0).stop 2024-04-01T01:39:55.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.560+0000 7f46b0bf7740 1 -- 172.21.15.70:0/1295888820 shutdown_connections 2024-04-01T01:39:55.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.560+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/1295888820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e1331750 0x5642e1346670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.560+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/1295888820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e132c5e0 0x5642e134e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.560+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/1295888820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e132bbf0 0x5642e1326130 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.561+0000 7f46b0bf7740 1 -- 172.21.15.70:0/1295888820 >> 172.21.15.70:0/1295888820 conn(0x5642e1327160 msgr2=0x5642e1325a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:55.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.561+0000 7f46b0bf7740 1 -- 172.21.15.70:0/1295888820 shutdown_connections 2024-04-01T01:39:55.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.561+0000 7f46b0bf7740 1 -- 172.21.15.70:0/1295888820 wait complete. 2024-04-01T01:39:55.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.562+0000 7f46b0bf7740 1 -- start start 2024-04-01T01:39:55.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.563+0000 7f46b0bf7740 1 Processor -- start 2024-04-01T01:39:55.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.563+0000 7f46b0bf7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e132bbf0 0x5642e134b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46b0bf7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e132c5e0 0x5642e134bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46b0bf7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e1331750 0x5642e134ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46b0bf7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f46a806e800 con 0x5642e132bbf0 2024-04-01T01:39:55.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46b0bf7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46a806e130 con 0x5642e132c5e0 2024-04-01T01:39:55.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46b0bf7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f46a806b390 con 0x5642e1331750 2024-04-01T01:39:55.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46b0bf7740 10 client.0 _ll_register_callbacks cb 0x7f4698011c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:55.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46a77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e132bbf0 0x5642e134b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46a77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e132bbf0 0x5642e134b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44764/0 (socket says 172.21.15.70:44764) 2024-04-01T01:39:55.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46a77fe640 1 -- 172.21.15.70:0/1164306000 learned_addr learned my addr 172.21.15.70:0/1164306000 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:55.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46a6ffd640 1 --2- 172.21.15.70:0/1164306000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e132c5e0 0x5642e134bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46a7fff640 1 --2- 172.21.15.70:0/1164306000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e1331750 0x5642e134ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46a6ffd640 1 -- 172.21.15.70:0/1164306000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e1331750 msgr2=0x5642e134ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:55.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46a6ffd640 1 --2- 172.21.15.70:0/1164306000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e1331750 0x5642e134ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46a6ffd640 1 -- 172.21.15.70:0/1164306000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e132bbf0 msgr2=0x5642e134b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:55.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46a6ffd640 1 --2- 172.21.15.70:0/1164306000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e132bbf0 0x5642e134b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46a6ffd640 1 -- 172.21.15.70:0/1164306000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642e134d2a0 con 0x5642e132c5e0 2024-04-01T01:39:55.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.564+0000 7f46a77fe640 1 --2- 172.21.15.70:0/1164306000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e132bbf0 0x5642e134b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:55.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.565+0000 7f46a6ffd640 1 --2- 172.21.15.70:0/1164306000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e132c5e0 0x5642e134bca0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x5642e1355520 tx=0x7f469c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e52b5ea9effe1859 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:55.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.565+0000 7f4686ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:55.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.565+0000 7f4686ffd640 1 -- 172.21.15.70:0/1164306000 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c02d7a0 con 0x5642e132c5e0 2024-04-01T01:39:55.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.565+0000 7f46b0bf7740 1 -- 172.21.15.70:0/1164306000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4698022e10 con 0x5642e132c5e0 2024-04-01T01:39:55.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.565+0000 7f4686ffd640 1 -- 172.21.15.70:0/1164306000 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f469c01a070 con 0x5642e132c5e0 2024-04-01T01:39:55.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.565+0000 7f4686ffd640 1 -- 172.21.15.70:0/1164306000 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f469c03b050 con 0x5642e132c5e0 2024-04-01T01:39:55.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.565+0000 7f46b0bf7740 10 client.27422 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:55.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.565+0000 7f46b0bf7740 1 -- 172.21.15.70:0/1164306000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5642e134e650 con 0x5642e132c5e0 2024-04-01T01:39:55.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.566+0000 7f4686ffd640 1 -- 172.21.15.70:0/1164306000 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4698022e10 con 0x5642e132c5e0 2024-04-01T01:39:55.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.566+0000 7f4686ffd640 1 -- 172.21.15.70:0/1164306000 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f469c00fd00 con 0x5642e132c5e0 2024-04-01T01:39:55.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.566+0000 7f46b0bf7740 10 client.27422 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:55.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.566+0000 7f46b0bf7740 10 client.27422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:55.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.566+0000 7f46b0bf7740 20 client.27422 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:55.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.566+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/1164306000 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5642e13b91b0 0x5642e13bb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.566+0000 7f46b0bf7740 4 client.27422 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:39:55.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.566+0000 7f46b0bf7740 1 -- 172.21.15.70:0/1164306000 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5642e1297730 con 0x5642e13b91b0 2024-04-01T01:39:55.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.566+0000 7f46a77fe640 1 --2- 172.21.15.70:0/1164306000 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5642e13b91b0 0x5642e13bb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.567+0000 7f46a77fe640 1 --2- 172.21.15.70:0/1164306000 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5642e13b91b0 0x5642e13bb5d0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae4a900c09468770 server_cookie=62c28f8c9bb411a9 in_seq=0 out_seq=0 2024-04-01T01:39:55.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.567+0000 7f4686ffd640 10 client.27422 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:55.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.568+0000 7f4686ffd640 1 -- 172.21.15.70:0/1164306000 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5642e1297730 con 0x5642e13b91b0 2024-04-01T01:39:55.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.568+0000 7f4686ffd640 10 client.27422 handle_command_reply: tid=0 2024-04-01T01:39:55.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.568+0000 7f46b0bf7740 1 client.27422 shutdown 2024-04-01T01:39:55.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.568+0000 7f46b0bf7740 1 -- 172.21.15.70:0/1164306000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e132c5e0 msgr2=0x5642e134bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.569+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/1164306000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e132c5e0 0x5642e134bca0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x5642e1355520 tx=0x7f469c004730 comp rx=0 tx=0).stop 2024-04-01T01:39:55.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.569+0000 7f46b0bf7740 1 -- 172.21.15.70:0/1164306000 shutdown_connections 2024-04-01T01:39:55.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.569+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/1164306000 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5642e13b91b0 0x5642e13bb5d0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.569+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/1164306000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642e1331750 0x5642e134ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.570+0000 7f4686ffd640 0 client.27422 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:55.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.570+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/1164306000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5642e132c5e0 0x5642e134bca0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.570+0000 7f46b0bf7740 1 --2- 172.21.15.70:0/1164306000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5642e132bbf0 0x5642e134b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.570+0000 7f46b0bf7740 1 -- 172.21.15.70:0/1164306000 >> 172.21.15.70:0/1164306000 conn(0x5642e1327160 msgr2=0x5642e13262b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:55.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.570+0000 7f46b0bf7740 1 -- 172.21.15.70:0/1164306000 shutdown_connections 2024-04-01T01:39:55.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.570+0000 7f46b0bf7740 1 -- 172.21.15.70:0/1164306000 wait complete. 2024-04-01T01:39:55.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.570+0000 7f46b0bf7740 20 client.27422 trim_cache size 0 max 16384 2024-04-01T01:39:55.571 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:39:55.571 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:39:55.571 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "86d68ac6-7c67-4888-86a0-3cdbd651a964", 2024-04-01T01:39:55.571 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:39:55.571 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:39:55.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.571+0000 7f46accdc640 1 -- 172.21.15.70:0/733707237 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46900a2580 msgr2=0x7f46900a4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.571+0000 7f46accdc640 1 --2- 172.21.15.70:0/733707237 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46900a2580 0x7f46900a4a40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x5642e132ce40 tx=0x7f469c013040 comp rx=0 tx=0).stop 2024-04-01T01:39:55.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.572+0000 7f46accdc640 1 -- 172.21.15.70:0/733707237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a8102110 msgr2=0x7f46a8195760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.572+0000 7f46accdc640 1 --2- 172.21.15.70:0/733707237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a8102110 0x7f46a8195760 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f46a0037040 tx=0x7f46a0032810 comp rx=0 tx=0).stop 2024-04-01T01:39:55.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.572+0000 7f46accdc640 1 -- 172.21.15.70:0/733707237 shutdown_connections 2024-04-01T01:39:55.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.572+0000 7f46accdc640 1 --2- 172.21.15.70:0/733707237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a8102110 0x7f46a8195760 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.572+0000 7f46accdc640 1 --2- 172.21.15.70:0/733707237 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46900a2580 0x7f46900a4a40 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.572+0000 7f46accdc640 1 --2- 172.21.15.70:0/733707237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46a8101750 0x7f46a8199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.572+0000 7f46accdc640 1 --2- 172.21.15.70:0/733707237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46a8100560 0x7f46a8198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.572+0000 7f46accdc640 1 -- 172.21.15.70:0/733707237 >> 172.21.15.70:0/733707237 conn(0x7f46a80fbad0 msgr2=0x7f46a8104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:55.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.572+0000 7f46accdc640 1 -- 172.21.15.70:0/733707237 shutdown_connections 2024-04-01T01:39:55.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.572+0000 7f46accdc640 1 -- 172.21.15.70:0/733707237 wait complete. 2024-04-01T01:39:55.594 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:39:55.594 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:39:55.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.782+0000 7f249fa6b640 1 -- 172.21.15.70:0/830483292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2498104b60 msgr2=0x7f2498104fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.782+0000 7f249fa6b640 1 --2- 172.21.15.70:0/830483292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2498104b60 0x7f2498104fe0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2494009860 tx=0x7f249402bb70 comp rx=0 tx=0).stop 2024-04-01T01:39:55.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.782+0000 7f249fa6b640 1 -- 172.21.15.70:0/830483292 shutdown_connections 2024-04-01T01:39:55.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.782+0000 7f249fa6b640 1 --2- 172.21.15.70:0/830483292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2498105520 0x7f249810bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.782+0000 7f249fa6b640 1 --2- 172.21.15.70:0/830483292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2498104b60 0x7f2498104fe0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.783+0000 7f249fa6b640 1 --2- 172.21.15.70:0/830483292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2498103970 0x7f2498103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.783+0000 7f249fa6b640 1 -- 172.21.15.70:0/830483292 >> 172.21.15.70:0/830483292 conn(0x7f24980fef00 msgr2=0x7f2498101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:55.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.783+0000 7f249fa6b640 1 -- 172.21.15.70:0/830483292 shutdown_connections 2024-04-01T01:39:55.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.783+0000 7f249fa6b640 1 -- 172.21.15.70:0/830483292 wait complete. 2024-04-01T01:39:55.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.784+0000 7f249fa6b640 1 Processor -- start 2024-04-01T01:39:55.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.784+0000 7f249fa6b640 1 -- start start 2024-04-01T01:39:55.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.784+0000 7f249fa6b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2498103970 0x7f249819ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.784+0000 7f249fa6b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2498104b60 0x7f249819e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.784+0000 7f249fa6b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2498105520 0x7f24981a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.784+0000 7f249fa6b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f249810f4a0 con 0x7f2498105520 2024-04-01T01:39:55.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.784+0000 7f249fa6b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f249810f320 con 0x7f2498104b60 2024-04-01T01:39:55.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.784+0000 7f249fa6b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f249810f620 con 0x7f2498103970 2024-04-01T01:39:55.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.785+0000 7f249d006640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2498104b60 0x7f249819e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.785+0000 7f249d006640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2498104b60 0x7f249819e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37800/0 (socket says 172.21.15.70:37800) 2024-04-01T01:39:55.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.785+0000 7f249d006640 1 -- 172.21.15.70:0/3974514109 learned_addr learned my addr 172.21.15.70:0/3974514109 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:55.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.785+0000 7f249e008640 1 --2- 172.21.15.70:0/3974514109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2498105520 0x7f24981a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.785+0000 7f249d807640 1 --2- 172.21.15.70:0/3974514109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2498103970 0x7f249819ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.785+0000 7f249d006640 1 -- 172.21.15.70:0/3974514109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2498103970 msgr2=0x7f249819ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.785+0000 7f249d006640 1 --2- 172.21.15.70:0/3974514109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2498103970 0x7f249819ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.785+0000 7f249d006640 1 -- 172.21.15.70:0/3974514109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2498105520 msgr2=0x7f24981a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.785+0000 7f249d006640 1 --2- 172.21.15.70:0/3974514109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2498105520 0x7f24981a5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.785+0000 7f249d006640 1 -- 172.21.15.70:0/3974514109 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24981a59d0 con 0x7f2498104b60 2024-04-01T01:39:55.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.785+0000 7f249d807640 1 --2- 172.21.15.70:0/3974514109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2498103970 0x7f249819ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:39:55.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.785+0000 7f249d006640 1 --2- 172.21.15.70:0/3974514109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2498104b60 0x7f249819e2f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f249402c180 tx=0x7f2494004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:55.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.786+0000 7f248effd640 1 -- 172.21.15.70:0/3974514109 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494039040 con 0x7f2498104b60 2024-04-01T01:39:55.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.786+0000 7f249fa6b640 1 -- 172.21.15.70:0/3974514109 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24981a5c60 con 0x7f2498104b60 2024-04-01T01:39:55.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.786+0000 7f249fa6b640 1 -- 172.21.15.70:0/3974514109 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24981a6150 con 0x7f2498104b60 2024-04-01T01:39:55.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.787+0000 7f248effd640 1 -- 172.21.15.70:0/3974514109 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2494042070 con 0x7f2498104b60 2024-04-01T01:39:55.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.787+0000 7f248effd640 1 -- 172.21.15.70:0/3974514109 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494004b60 con 0x7f2498104b60 2024-04-01T01:39:55.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.790+0000 7f24a2648740 1 Processor -- start 2024-04-01T01:39:55.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.791+0000 7f248effd640 1 -- 172.21.15.70:0/3974514109 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f2494004d00 con 0x7f2498104b60 2024-04-01T01:39:55.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.791+0000 7f24a2648740 1 -- start start 2024-04-01T01:39:55.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.792+0000 7f24a2648740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e2625142b0 0x55e2625146d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.792+0000 7f24a2648740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e262514ca0 0x55e2625197a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.792+0000 7f24a2648740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e262519e10 0x55e26251c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.792+0000 7f24a2648740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e262479180 con 0x55e262514ca0 2024-04-01T01:39:55.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.792+0000 7f24a2648740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e262421b70 con 0x55e2625142b0 2024-04-01T01:39:55.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.792+0000 7f24a2648740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e262475220 con 0x55e262519e10 2024-04-01T01:39:55.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.792+0000 7f249e008640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e2625142b0 0x55e2625146d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.792+0000 7f249e008640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e2625142b0 0x55e2625146d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37808/0 (socket says 172.21.15.70:37808) 2024-04-01T01:39:55.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.792+0000 7f249e008640 1 -- 172.21.15.70:0/2917068423 learned_addr learned my addr 172.21.15.70:0/2917068423 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:55.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.792+0000 7f249d807640 1 --2- 172.21.15.70:0/2917068423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e262514ca0 0x55e2625197a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.792+0000 7f249d006640 1 --2- 172.21.15.70:0/2917068423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e262519e10 0x55e26251c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.792+0000 7f249e008640 1 -- 172.21.15.70:0/2917068423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e262519e10 msgr2=0x55e26251c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:55.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.792+0000 7f249e008640 1 --2- 172.21.15.70:0/2917068423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e262519e10 0x55e26251c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.792+0000 7f249e008640 1 -- 172.21.15.70:0/2917068423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e262514ca0 msgr2=0x55e2625197a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:55.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.792+0000 7f249e008640 1 --2- 172.21.15.70:0/2917068423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e262514ca0 0x55e2625197a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.792+0000 7f249e008640 1 -- 172.21.15.70:0/2917068423 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2624357c0 con 0x55e2625142b0 2024-04-01T01:39:55.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.793+0000 7f248effd640 1 --2- 172.21.15.70:0/3974514109 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24840a2670 0x7f24840a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.793+0000 7f249e008640 1 --2- 172.21.15.70:0/2917068423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e2625142b0 0x55e2625146d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2490010370 tx=0x7f2490010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97f3567a719a1651 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:55.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.793+0000 7f248effd640 1 -- 172.21.15.70:0/3974514109 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f24940f0460 con 0x7f2498104b60 2024-04-01T01:39:55.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.793+0000 7f2483fff640 1 -- 172.21.15.70:0/2917068423 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2490016070 con 0x55e2625142b0 2024-04-01T01:39:55.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.793+0000 7f2483fff640 1 -- 172.21.15.70:0/2917068423 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2490004d20 con 0x55e2625142b0 2024-04-01T01:39:55.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.793+0000 7f2483fff640 1 -- 172.21.15.70:0/2917068423 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2490005440 con 0x55e2625142b0 2024-04-01T01:39:55.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.793+0000 7f249d807640 1 --2- 172.21.15.70:0/3974514109 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24840a2670 0x7f24840a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.794+0000 7f24a2648740 1 -- 172.21.15.70:0/2917068423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e2625142b0 msgr2=0x55e2625146d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.794+0000 7f249d807640 1 --2- 172.21.15.70:0/3974514109 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24840a2670 0x7f24840a4b30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x55e2625150a0 tx=0x7f2488009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:55.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.794+0000 7f24a2648740 1 --2- 172.21.15.70:0/2917068423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e2625142b0 0x55e2625146d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2490010370 tx=0x7f2490010830 comp rx=0 tx=0).stop 2024-04-01T01:39:55.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.794+0000 7f24a2648740 1 -- 172.21.15.70:0/2917068423 shutdown_connections 2024-04-01T01:39:55.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.794+0000 7f24a2648740 1 --2- 172.21.15.70:0/2917068423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e262519e10 0x55e26251c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.794+0000 7f24a2648740 1 --2- 172.21.15.70:0/2917068423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e262514ca0 0x55e2625197a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.794+0000 7f24a2648740 1 --2- 172.21.15.70:0/2917068423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e2625142b0 0x55e2625146d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.794+0000 7f24a2648740 1 -- 172.21.15.70:0/2917068423 >> 172.21.15.70:0/2917068423 conn(0x55e26250f820 msgr2=0x55e26250fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:55.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.795+0000 7f24a2648740 1 -- 172.21.15.70:0/2917068423 shutdown_connections 2024-04-01T01:39:55.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.795+0000 7f24a2648740 1 -- 172.21.15.70:0/2917068423 wait complete. 2024-04-01T01:39:55.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.795+0000 7f24a2648740 1 -- start start 2024-04-01T01:39:55.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.795+0000 7f24a2648740 1 Processor -- start 2024-04-01T01:39:55.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.796+0000 7f24a2648740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2625142b0 0x55e26251e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.796+0000 7f24a2648740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e262514ca0 0x55e26251eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.796+0000 7f24a2648740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e262519e10 0x55e262525bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.796+0000 7f24a2648740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e26251cdc0 con 0x55e262519e10 2024-04-01T01:39:55.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.796+0000 7f24a2648740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e2624174a0 con 0x55e262514ca0 2024-04-01T01:39:55.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.796+0000 7f24a2648740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e26251cf40 con 0x55e2625142b0 2024-04-01T01:39:55.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.796+0000 7f249d807640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2625142b0 0x55e26251e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.796+0000 7f249d807640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2625142b0 0x55e26251e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:50050/0 (socket says 172.21.15.70:50050) 2024-04-01T01:39:55.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.796+0000 7f249d807640 1 -- 172.21.15.70:0/246234419 learned_addr learned my addr 172.21.15.70:0/246234419 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:55.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.796+0000 7f24a2648740 10 client.0 _ll_register_callbacks cb 0x55e26241ad90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:55.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.797+0000 7f249d006640 1 --2- 172.21.15.70:0/246234419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e262514ca0 0x55e26251eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.797+0000 7f249d807640 1 -- 172.21.15.70:0/246234419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e262514ca0 msgr2=0x55e26251eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:55.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.797+0000 7f249d807640 1 --2- 172.21.15.70:0/246234419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e262514ca0 0x55e26251eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.797+0000 7f249d807640 1 -- 172.21.15.70:0/246234419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e262519e10 msgr2=0x55e262525bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:55.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.797+0000 7f249d807640 1 --2- 172.21.15.70:0/246234419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e262519e10 0x55e262525bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.797+0000 7f249d807640 1 -- 172.21.15.70:0/246234419 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2622577d0 con 0x55e2625142b0 2024-04-01T01:39:55.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.797+0000 7f249d807640 1 --2- 172.21.15.70:0/246234419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2625142b0 0x55e26251e640 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2488011730 tx=0x7f24880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d44efbebc9c2007c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:55.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.797+0000 7f24817fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:39:55.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.797+0000 7f24817fa640 1 -- 172.21.15.70:0/246234419 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2488024030 con 0x55e2625142b0 2024-04-01T01:39:55.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.797+0000 7f24817fa640 1 -- 172.21.15.70:0/246234419 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f24880107a0 con 0x55e2625142b0 2024-04-01T01:39:55.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.798+0000 7f24a2648740 1 -- 172.21.15.70:0/246234419 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2490004d20 con 0x55e2625142b0 2024-04-01T01:39:55.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.798+0000 7f24817fa640 1 -- 172.21.15.70:0/246234419 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f24880109a0 con 0x55e2625142b0 2024-04-01T01:39:55.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.798+0000 7f24a2648740 10 client.27292 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:55.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.798+0000 7f24a2648740 1 -- 172.21.15.70:0/246234419 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e26218d3f0 con 0x55e2625142b0 2024-04-01T01:39:55.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.799+0000 7f24817fa640 1 -- 172.21.15.70:0/246234419 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2490004d20 con 0x55e2625142b0 2024-04-01T01:39:55.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.799+0000 7f24817fa640 1 -- 172.21.15.70:0/246234419 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f248801d050 con 0x55e2625142b0 2024-04-01T01:39:55.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.799+0000 7f24a2648740 10 client.27292 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:55.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.799+0000 7f24a2648740 10 client.27292 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:55.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.799+0000 7f24a2648740 20 client.27292 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:55.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.799+0000 7f24a2648740 1 --2- 172.21.15.70:0/246234419 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e26252a5e0 0x55e26252ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.799+0000 7f24a2648740 4 client.27292 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:55.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.799+0000 7f24a2648740 1 -- 172.21.15.70:0/246234419 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2624866c0 con 0x55e26252a5e0 2024-04-01T01:39:55.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.799+0000 7f249d006640 1 --2- 172.21.15.70:0/246234419 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e26252a5e0 0x55e26252ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.800+0000 7f24817fa640 10 client.27292 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:55.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.800+0000 7f249d006640 1 --2- 172.21.15.70:0/246234419 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e26252a5e0 0x55e26252ca80 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7be7747ec186710d server_cookie=3938676767389869 in_seq=0 out_seq=0 2024-04-01T01:39:55.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.801+0000 7f24817fa640 1 -- 172.21.15.70:0/246234419 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e2624866c0 con 0x55e26252a5e0 2024-04-01T01:39:55.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.801+0000 7f24817fa640 10 client.27292 handle_command_reply: tid=0 2024-04-01T01:39:55.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.801+0000 7f24a2648740 1 client.27292 shutdown 2024-04-01T01:39:55.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.802+0000 7f24a2648740 1 -- 172.21.15.70:0/246234419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2625142b0 msgr2=0x55e26251e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.802+0000 7f24a2648740 1 --2- 172.21.15.70:0/246234419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2625142b0 0x55e26251e640 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2488011730 tx=0x7f24880046c0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.802+0000 7f24a2648740 1 -- 172.21.15.70:0/246234419 shutdown_connections 2024-04-01T01:39:55.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.802+0000 7f24a2648740 1 --2- 172.21.15.70:0/246234419 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e26252a5e0 0x55e26252ca80 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.802+0000 7f24a2648740 1 --2- 172.21.15.70:0/246234419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e262519e10 0x55e262525bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.802+0000 7f24a2648740 1 --2- 172.21.15.70:0/246234419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e262514ca0 0x55e26251eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.803+0000 7f24a2648740 1 --2- 172.21.15.70:0/246234419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2625142b0 0x55e26251e640 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.803+0000 7f24a2648740 1 -- 172.21.15.70:0/246234419 >> 172.21.15.70:0/246234419 conn(0x55e26250f820 msgr2=0x55e26250ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:55.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.802+0000 7f24817fa640 0 client.27292 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:55.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.803+0000 7f24a2648740 1 -- 172.21.15.70:0/246234419 shutdown_connections 2024-04-01T01:39:55.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.803+0000 7f24a2648740 1 -- 172.21.15.70:0/246234419 wait complete. 2024-04-01T01:39:55.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.803+0000 7f24a2648740 20 client.27292 trim_cache size 0 max 16384 2024-04-01T01:39:55.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.836+0000 7f24a2648740 1 Processor -- start 2024-04-01T01:39:55.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.836+0000 7f24a2648740 1 -- start start 2024-04-01T01:39:55.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.836+0000 7f24a2648740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e26250f820 0x55e26251da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f24a2648740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2625142b0 0x55e262525b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f24a2648740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e262514ca0 0x55e26251e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f24a2648740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f247c00fcd0 con 0x55e26250f820 2024-04-01T01:39:55.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f24a2648740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f247c011810 con 0x55e262514ca0 2024-04-01T01:39:55.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f24a2648740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f247c0103a0 con 0x55e2625142b0 2024-04-01T01:39:55.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f249d807640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e26250f820 0x55e26251da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f249d807640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e26250f820 0x55e26251da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44798/0 (socket says 172.21.15.70:44798) 2024-04-01T01:39:55.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f249d807640 1 -- 172.21.15.70:0/1987554749 learned_addr learned my addr 172.21.15.70:0/1987554749 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:55.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f249d006640 1 --2- 172.21.15.70:0/1987554749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2625142b0 0x55e262525b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f249e008640 1 --2- 172.21.15.70:0/1987554749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e262514ca0 0x55e26251e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f249d006640 1 -- 172.21.15.70:0/1987554749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e262514ca0 msgr2=0x55e26251e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:55.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f249d006640 1 --2- 172.21.15.70:0/1987554749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e262514ca0 0x55e26251e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f249d006640 1 -- 172.21.15.70:0/1987554749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e26250f820 msgr2=0x55e26251da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:55.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f249d006640 1 --2- 172.21.15.70:0/1987554749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e26250f820 0x55e26251da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f249d006640 1 -- 172.21.15.70:0/1987554749 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e26214d0f0 con 0x55e2625142b0 2024-04-01T01:39:55.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f249d807640 1 --2- 172.21.15.70:0/1987554749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e26250f820 0x55e26251da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:55.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.837+0000 7f249e008640 1 --2- 172.21.15.70:0/1987554749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e262514ca0 0x55e26251e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:55.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.838+0000 7f249d006640 1 --2- 172.21.15.70:0/1987554749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2625142b0 0x55e262525b80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55e26251fb20 tx=0x7f2494000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=663390466dbd78f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:55.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.838+0000 7f24837fe640 1 -- 172.21.15.70:0/1987554749 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24940bc630 con 0x55e2625142b0 2024-04-01T01:39:55.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.838+0000 7f24837fe640 1 -- 172.21.15.70:0/1987554749 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24940bd1f0 con 0x55e2625142b0 2024-04-01T01:39:55.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.838+0000 7f24837fe640 1 -- 172.21.15.70:0/1987554749 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24940bc450 con 0x55e2625142b0 2024-04-01T01:39:55.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.839+0000 7f24a2648740 1 -- 172.21.15.70:0/1987554749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2625142b0 msgr2=0x55e262525b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.839+0000 7f24a2648740 1 --2- 172.21.15.70:0/1987554749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2625142b0 0x55e262525b80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55e26251fb20 tx=0x7f2494000bc0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.839+0000 7f24a2648740 1 -- 172.21.15.70:0/1987554749 shutdown_connections 2024-04-01T01:39:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.839+0000 7f24a2648740 1 --2- 172.21.15.70:0/1987554749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e262514ca0 0x55e26251e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.839+0000 7f24a2648740 1 --2- 172.21.15.70:0/1987554749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2625142b0 0x55e262525b80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.839+0000 7f24a2648740 1 --2- 172.21.15.70:0/1987554749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e26250f820 0x55e26251da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.839+0000 7f24a2648740 1 -- 172.21.15.70:0/1987554749 >> 172.21.15.70:0/1987554749 conn(0x55e2625158a0 msgr2=0x55e26251d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.839+0000 7f24a2648740 1 -- 172.21.15.70:0/1987554749 shutdown_connections 2024-04-01T01:39:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.839+0000 7f24a2648740 1 -- 172.21.15.70:0/1987554749 wait complete. 2024-04-01T01:39:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.840+0000 7f24a2648740 1 -- start start 2024-04-01T01:39:55.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.840+0000 7f24a2648740 1 Processor -- start 2024-04-01T01:39:55.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.840+0000 7f24a2648740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e26250f820 0x55e262522af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.840+0000 7f24a2648740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2625142b0 0x55e262523030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.840+0000 7f24a2648740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e262514ca0 0x55e26258e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.840+0000 7f24a2648740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f247c011140 con 0x55e2625142b0 2024-04-01T01:39:55.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.840+0000 7f24a2648740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f247c010a70 con 0x55e26250f820 2024-04-01T01:39:55.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.840+0000 7f24a2648740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f247c00dcb0 con 0x55e262514ca0 2024-04-01T01:39:55.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.840+0000 7f24a2648740 10 client.0 _ll_register_callbacks cb 0x7f24880113e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:55.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.841+0000 7f249e008640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e262514ca0 0x55e26258e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.841+0000 7f249d006640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2625142b0 0x55e262523030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.841+0000 7f249e008640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e262514ca0 0x55e26258e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:50068/0 (socket says 172.21.15.70:50068) 2024-04-01T01:39:55.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.841+0000 7f249e008640 1 -- 172.21.15.70:0/2071852415 learned_addr learned my addr 172.21.15.70:0/2071852415 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:55.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.841+0000 7f249d807640 1 --2- 172.21.15.70:0/2071852415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e26250f820 0x55e262522af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.841+0000 7f249e008640 1 -- 172.21.15.70:0/2071852415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e26250f820 msgr2=0x55e262522af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:55.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.841+0000 7f249e008640 1 --2- 172.21.15.70:0/2071852415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e26250f820 0x55e262522af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.841+0000 7f249e008640 1 -- 172.21.15.70:0/2071852415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2625142b0 msgr2=0x55e262523030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:55.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.841+0000 7f249e008640 1 --2- 172.21.15.70:0/2071852415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2625142b0 0x55e262523030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.841+0000 7f249e008640 1 -- 172.21.15.70:0/2071852415 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e26258efa0 con 0x55e262514ca0 2024-04-01T01:39:55.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.841+0000 7f249e008640 1 --2- 172.21.15.70:0/2071852415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e262514ca0 0x55e26258e9b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2490019000 tx=0x7f2490013750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3af024e36c4bd404 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:55.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.841+0000 7f24817fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:39:55.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.842+0000 7f24817fa640 1 -- 172.21.15.70:0/2071852415 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249002a060 con 0x55e262514ca0 2024-04-01T01:39:55.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.842+0000 7f24817fa640 1 -- 172.21.15.70:0/2071852415 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f2490003d80 con 0x55e262514ca0 2024-04-01T01:39:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.842+0000 7f24a2648740 1 -- 172.21.15.70:0/2071852415 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24940bd1f0 con 0x55e262514ca0 2024-04-01T01:39:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.842+0000 7f24817fa640 1 -- 172.21.15.70:0/2071852415 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2490031050 con 0x55e262514ca0 2024-04-01T01:39:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.842+0000 7f24a2648740 10 client.27298 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.842+0000 7f24a2648740 1 -- 172.21.15.70:0/2071852415 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e2625902c0 con 0x55e262514ca0 2024-04-01T01:39:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.842+0000 7f24817fa640 1 -- 172.21.15.70:0/2071852415 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24940bd1f0 con 0x55e262514ca0 2024-04-01T01:39:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.842+0000 7f24817fa640 1 -- 172.21.15.70:0/2071852415 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f249000f040 con 0x55e262514ca0 2024-04-01T01:39:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.843+0000 7f24a2648740 10 client.27298 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.843+0000 7f24a2648740 10 client.27298 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.843+0000 7f24a2648740 20 client.27298 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.843+0000 7f24a2648740 1 --2- 172.21.15.70:0/2071852415 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e262519e10 0x55e262592c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.843+0000 7f24a2648740 4 client.27298 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:39:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.843+0000 7f24a2648740 1 -- 172.21.15.70:0/2071852415 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e262480a10 con 0x55e262519e10 2024-04-01T01:39:55.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.843+0000 7f249d807640 1 --2- 172.21.15.70:0/2071852415 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e262519e10 0x55e262592c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:55.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.843+0000 7f249d807640 1 --2- 172.21.15.70:0/2071852415 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e262519e10 0x55e262592c30 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9e66377e18422a server_cookie=ed39c704a38e9a45 in_seq=0 out_seq=0 2024-04-01T01:39:55.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.844+0000 7f24817fa640 10 client.27298 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:55.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.844+0000 7f24817fa640 1 -- 172.21.15.70:0/2071852415 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e262480a10 con 0x55e262519e10 2024-04-01T01:39:55.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.844+0000 7f24817fa640 10 client.27298 handle_command_reply: tid=0 2024-04-01T01:39:55.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.844+0000 7f24a2648740 1 client.27298 shutdown 2024-04-01T01:39:55.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.844+0000 7f24a2648740 1 -- 172.21.15.70:0/2071852415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e262514ca0 msgr2=0x55e26258e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.844+0000 7f24a2648740 1 --2- 172.21.15.70:0/2071852415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e262514ca0 0x55e26258e9b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2490019000 tx=0x7f2490013750 comp rx=0 tx=0).stop 2024-04-01T01:39:55.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.845+0000 7f24a2648740 1 -- 172.21.15.70:0/2071852415 shutdown_connections 2024-04-01T01:39:55.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.845+0000 7f24a2648740 1 --2- 172.21.15.70:0/2071852415 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e262519e10 0x55e262592c30 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.845+0000 7f24a2648740 1 --2- 172.21.15.70:0/2071852415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e262514ca0 0x55e26258e9b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.845+0000 7f24a2648740 1 --2- 172.21.15.70:0/2071852415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e2625142b0 0x55e262523030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.845+0000 7f24a2648740 1 --2- 172.21.15.70:0/2071852415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e26250f820 0x55e262522af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.845+0000 7f24a2648740 1 -- 172.21.15.70:0/2071852415 >> 172.21.15.70:0/2071852415 conn(0x55e2625158a0 msgr2=0x55e26250ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:55.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.845+0000 7f24817fa640 0 client.27298 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:55.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.845+0000 7f24a2648740 1 -- 172.21.15.70:0/2071852415 shutdown_connections 2024-04-01T01:39:55.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.845+0000 7f24a2648740 1 -- 172.21.15.70:0/2071852415 wait complete. 2024-04-01T01:39:55.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.845+0000 7f24a2648740 20 client.27298 trim_cache size 0 max 16384 2024-04-01T01:39:55.847 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:39:55.847 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:39:55.848 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:39:55.848 INFO:teuthology.orchestra.run.smithi070.stdout: "86d68ac6-7c67-4888-86a0-3cdbd651a964": { 2024-04-01T01:39:55.848 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:39:55.848 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "86d68ac6-7c67-4888-86a0-3cdbd651a964", 2024-04-01T01:39:55.848 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:39:55.848 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:39:55.848 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:39:55.848 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:39:55.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.846+0000 7f249fa6b640 1 -- 172.21.15.70:0/3974514109 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24840a2670 msgr2=0x7f24840a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.846+0000 7f249fa6b640 1 --2- 172.21.15.70:0/3974514109 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24840a2670 0x7f24840a4b30 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x55e2625150a0 tx=0x7f2488009260 comp rx=0 tx=0).stop 2024-04-01T01:39:55.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.847+0000 7f249fa6b640 1 -- 172.21.15.70:0/3974514109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2498104b60 msgr2=0x7f249819e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:55.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.847+0000 7f249fa6b640 1 --2- 172.21.15.70:0/3974514109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2498104b60 0x7f249819e2f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f249402c180 tx=0x7f2494004430 comp rx=0 tx=0).stop 2024-04-01T01:39:55.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.847+0000 7f249fa6b640 1 -- 172.21.15.70:0/3974514109 shutdown_connections 2024-04-01T01:39:55.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.847+0000 7f249fa6b640 1 --2- 172.21.15.70:0/3974514109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2498105520 0x7f24981a5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.847+0000 7f249fa6b640 1 --2- 172.21.15.70:0/3974514109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2498104b60 0x7f249819e2f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.847+0000 7f249fa6b640 1 --2- 172.21.15.70:0/3974514109 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24840a2670 0x7f24840a4b30 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.847+0000 7f249fa6b640 1 --2- 172.21.15.70:0/3974514109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2498103970 0x7f249819ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:55.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.847+0000 7f249fa6b640 1 -- 172.21.15.70:0/3974514109 >> 172.21.15.70:0/3974514109 conn(0x7f24980fef00 msgr2=0x7f2498077ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:55.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.847+0000 7f249fa6b640 1 -- 172.21.15.70:0/3974514109 shutdown_connections 2024-04-01T01:39:55.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:39:55.848+0000 7f249fa6b640 1 -- 172.21.15.70:0/3974514109 wait complete. 2024-04-01T01:39:55.864 INFO:tasks.cephfs.filesystem:scrub status for tag:86d68ac6-7c67-4888-86a0-3cdbd651a964 - {'path': '/', 'tag': '86d68ac6-7c67-4888-86a0-3cdbd651a964', 'options': 'recursive,force'} 2024-04-01T01:39:56.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:56 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:39:56.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:56 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:39:56.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:56 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:39:56.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:56 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:39:56.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:56 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:39:56.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:56 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:39:56.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:56 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:39:56.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:56 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:39:56.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:56 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:39:56.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:56 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:39:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.512+0000 7f59b148d640 1 -- 172.21.15.70:0/692851750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a409c3a0 msgr2=0x7f59a409c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:56.511 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.512+0000 7f59b148d640 1 --2- 172.21.15.70:0/692851750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a409c3a0 0x7f59a409c7c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f599c009860 tx=0x7f599c02bbb0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.513+0000 7f59b148d640 1 -- 172.21.15.70:0/692851750 shutdown_connections 2024-04-01T01:39:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.513+0000 7f59b148d640 1 --2- 172.21.15.70:0/692851750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59a409df50 0x7f59a40a47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.513+0000 7f59b148d640 1 --2- 172.21.15.70:0/692851750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a409d590 0x7f59a409da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.513+0000 7f59b148d640 1 --2- 172.21.15.70:0/692851750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a409c3a0 0x7f59a409c7c0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.512 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.513+0000 7f59b148d640 1 -- 172.21.15.70:0/692851750 >> 172.21.15.70:0/692851750 conn(0x7f59a400b480 msgr2=0x7f59a400d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.515+0000 7f59b148d640 1 -- 172.21.15.70:0/692851750 shutdown_connections 2024-04-01T01:39:56.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.516+0000 7f59b148d640 1 -- 172.21.15.70:0/692851750 wait complete. 2024-04-01T01:39:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.517+0000 7f59b148d640 1 Processor -- start 2024-04-01T01:39:56.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.517+0000 7f59b148d640 1 -- start start 2024-04-01T01:39:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.518+0000 7f59b148d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59a409c3a0 0x7f59a41325d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.518+0000 7f59b148d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a409d590 0x7f59a4132b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.518+0000 7f59b148d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a409df50 0x7f59a4139b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.518+0000 7f59b148d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f59a40a7eb0 con 0x7f59a409d590 2024-04-01T01:39:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.518+0000 7f59aaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59a409c3a0 0x7f59a41325d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.518+0000 7f59b148d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f59a40a7d30 con 0x7f59a409c3a0 2024-04-01T01:39:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.518+0000 7f59b148d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f59a40a8030 con 0x7f59a409df50 2024-04-01T01:39:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.519+0000 7f59ab7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a409df50 0x7f59a4139b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.519+0000 7f59ab7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a409df50 0x7f59a4139b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:50080/0 (socket says 172.21.15.70:50080) 2024-04-01T01:39:56.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.519+0000 7f59ab7fe640 1 -- 172.21.15.70:0/3131165988 learned_addr learned my addr 172.21.15.70:0/3131165988 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:56.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.519+0000 7f59ab7fe640 1 -- 172.21.15.70:0/3131165988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59a409c3a0 msgr2=0x7f59a41325d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:56.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.519+0000 7f59ab7fe640 1 --2- 172.21.15.70:0/3131165988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59a409c3a0 0x7f59a41325d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.519+0000 7f59ab7fe640 1 -- 172.21.15.70:0/3131165988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a409d590 msgr2=0x7f59a4132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:56.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.519+0000 7f59ab7fe640 1 --2- 172.21.15.70:0/3131165988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a409d590 0x7f59a4132b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.519+0000 7f59ab7fe640 1 -- 172.21.15.70:0/3131165988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a413a1f0 con 0x7f59a409df50 2024-04-01T01:39:56.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.519+0000 7f59aaffd640 1 --2- 172.21.15.70:0/3131165988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59a409c3a0 0x7f59a41325d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:39:56.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.520+0000 7f59ab7fe640 1 --2- 172.21.15.70:0/3131165988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a409df50 0x7f59a4139b80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f59ac050db0 tx=0x7f59ac07bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:56.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.520+0000 7f5993fff640 1 -- 172.21.15.70:0/3131165988 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac082070 con 0x7f59a409df50 2024-04-01T01:39:56.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.520+0000 7f5993fff640 1 -- 172.21.15.70:0/3131165988 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59ac07e460 con 0x7f59a409df50 2024-04-01T01:39:56.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.520+0000 7f59b148d640 1 -- 172.21.15.70:0/3131165988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59a413a4e0 con 0x7f59a409df50 2024-04-01T01:39:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.520+0000 7f5993fff640 1 -- 172.21.15.70:0/3131165988 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ac07d620 con 0x7f59a409df50 2024-04-01T01:39:56.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.520+0000 7f59b148d640 1 -- 172.21.15.70:0/3131165988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59a413a9b0 con 0x7f59a409df50 2024-04-01T01:39:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.524+0000 7f59b406a740 1 Processor -- start 2024-04-01T01:39:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.524+0000 7f59b406a740 1 -- start start 2024-04-01T01:39:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.524+0000 7f59b406a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db92af0 0x56069db92f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.525+0000 7f59b406a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db934e0 0x56069db97fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.525+0000 7f59b406a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db98650 0x56069db9aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.525+0000 7f59b406a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56069dae58f0 con 0x56069db934e0 2024-04-01T01:39:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.525+0000 7f59b406a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56069da8e8d0 con 0x56069db92af0 2024-04-01T01:39:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.525+0000 7f59b406a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56069dae1990 con 0x56069db98650 2024-04-01T01:39:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.525+0000 7f5993fff640 1 -- 172.21.15.70:0/3131165988 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f59ac085070 con 0x7f59a409df50 2024-04-01T01:39:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.526+0000 7f5993fff640 1 --2- 172.21.15.70:0/3131165988 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59940a25a0 0x7f59940a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.526+0000 7f59ab7fe640 1 --2- 172.21.15.70:0/3131165988 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59940a25a0 0x7f59940a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.526+0000 7f5993fff640 1 -- 172.21.15.70:0/3131165988 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f59ac13b4c0 con 0x7f59a409df50 2024-04-01T01:39:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.527+0000 7f59aaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db934e0 0x56069db97fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.527+0000 7f59aa7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db98650 0x56069db9aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.527+0000 7f59aa7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db98650 0x56069db9aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:50084/0 (socket says 172.21.15.70:50084) 2024-04-01T01:39:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.527+0000 7f59aa7fc640 1 -- 172.21.15.70:0/1364455664 learned_addr learned my addr 172.21.15.70:0/1364455664 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:56.526 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.527+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/1364455664 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db92af0 0x56069db92f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.527+0000 7f59ab7fe640 1 --2- 172.21.15.70:0/3131165988 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59940a25a0 0x7f59940a4a60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f59ac066e90 tx=0x7f59ac067690 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.527+0000 7f59aa7fc640 1 -- 172.21.15.70:0/1364455664 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db92af0 msgr2=0x56069db92f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.527+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/1364455664 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db92af0 0x56069db92f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.527+0000 7f59aa7fc640 1 -- 172.21.15.70:0/1364455664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db934e0 msgr2=0x56069db97fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:56.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.527+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/1364455664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db934e0 0x56069db97fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.527+0000 7f59aa7fc640 1 -- 172.21.15.70:0/1364455664 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56069d7fa210 con 0x56069db98650 2024-04-01T01:39:56.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.529+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/1364455664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db98650 0x56069db9aae0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f59a0012790 tx=0x7f59a000d3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e21e9a7b05ffa21 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.530+0000 7f59917fa640 1 -- 172.21.15.70:0/1364455664 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a000a6b0 con 0x56069db98650 2024-04-01T01:39:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.530+0000 7f59917fa640 1 -- 172.21.15.70:0/1364455664 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59a000ad00 con 0x56069db98650 2024-04-01T01:39:56.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.530+0000 7f59917fa640 1 -- 172.21.15.70:0/1364455664 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a0010cc0 con 0x56069db98650 2024-04-01T01:39:56.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.531+0000 7f59b406a740 1 -- 172.21.15.70:0/1364455664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db98650 msgr2=0x56069db9aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:56.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.531+0000 7f59b406a740 1 --2- 172.21.15.70:0/1364455664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db98650 0x56069db9aae0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f59a0012790 tx=0x7f59a000d3c0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.531+0000 7f59b406a740 1 -- 172.21.15.70:0/1364455664 shutdown_connections 2024-04-01T01:39:56.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.531+0000 7f59b406a740 1 --2- 172.21.15.70:0/1364455664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db98650 0x56069db9aae0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.531+0000 7f59b406a740 1 --2- 172.21.15.70:0/1364455664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db934e0 0x56069db97fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.531+0000 7f59b406a740 1 --2- 172.21.15.70:0/1364455664 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db92af0 0x56069db92f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.531+0000 7f59b406a740 1 -- 172.21.15.70:0/1364455664 >> 172.21.15.70:0/1364455664 conn(0x56069db7d050 msgr2=0x56069db7d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:56.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.531+0000 7f59b406a740 1 -- 172.21.15.70:0/1364455664 shutdown_connections 2024-04-01T01:39:56.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.531+0000 7f59b406a740 1 -- 172.21.15.70:0/1364455664 wait complete. 2024-04-01T01:39:56.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.531+0000 7f59b406a740 1 -- start start 2024-04-01T01:39:56.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.531+0000 7f59b406a740 1 Processor -- start 2024-04-01T01:39:56.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.532+0000 7f59b406a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db92af0 0x56069db9ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.532+0000 7f59b406a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db934e0 0x56069db9d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.532+0000 7f59b406a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db98650 0x56069dba43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.532+0000 7f59b406a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56069db9b600 con 0x56069db92af0 2024-04-01T01:39:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.532+0000 7f59b406a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56069da83f80 con 0x56069db98650 2024-04-01T01:39:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.532+0000 7f59b406a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56069db9b780 con 0x56069db934e0 2024-04-01T01:39:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.532+0000 7f59b406a740 10 client.0 _ll_register_callbacks cb 0x56069dadc290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.532+0000 7f59aa7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db92af0 0x56069db9ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.532+0000 7f59aa7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db92af0 0x56069db9ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44836/0 (socket says 172.21.15.70:44836) 2024-04-01T01:39:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.532+0000 7f59aa7fc640 1 -- 172.21.15.70:0/753212781 learned_addr learned my addr 172.21.15.70:0/753212781 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.532+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/753212781 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db98650 0x56069dba43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.533+0000 7f59aa7fc640 1 -- 172.21.15.70:0/753212781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db934e0 msgr2=0x56069db9d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.533+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/753212781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db934e0 0x56069db9d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.533+0000 7f59aa7fc640 1 -- 172.21.15.70:0/753212781 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db98650 msgr2=0x56069dba43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.533+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/753212781 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db98650 0x56069dba43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.533+0000 7f59aa7fc640 1 -- 172.21.15.70:0/753212781 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56069d8c6620 con 0x56069db92af0 2024-04-01T01:39:56.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.533+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/753212781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db92af0 0x56069db9ce30 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f59a000d670 tx=0x7f59a000dfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff463828b939ef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.533+0000 7f598eff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:39:56.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.533+0000 7f598eff5640 1 -- 172.21.15.70:0/753212781 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a00103d0 con 0x56069db92af0 2024-04-01T01:39:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.534+0000 7f59b406a740 1 -- 172.21.15.70:0/753212781 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59a000ad00 con 0x56069db92af0 2024-04-01T01:39:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.534+0000 7f598eff5640 1 -- 172.21.15.70:0/753212781 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f59a0010570 con 0x56069db92af0 2024-04-01T01:39:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.534+0000 7f598eff5640 1 -- 172.21.15.70:0/753212781 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f59a0012040 con 0x56069db92af0 2024-04-01T01:39:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.534+0000 7f59b406a740 10 client.17538 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.534+0000 7f59b406a740 1 -- 172.21.15.70:0/753212781 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56069dba73e0 con 0x56069db92af0 2024-04-01T01:39:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.535+0000 7f598eff5640 1 -- 172.21.15.70:0/753212781 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59a000ad00 con 0x56069db92af0 2024-04-01T01:39:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.535+0000 7f598eff5640 1 -- 172.21.15.70:0/753212781 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f59a0034020 con 0x56069db92af0 2024-04-01T01:39:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.536+0000 7f59b406a740 10 client.17538 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.536+0000 7f59b406a740 10 client.17538 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:56.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.536+0000 7f59b406a740 20 client.17538 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.536+0000 7f59b406a740 1 --2- 172.21.15.70:0/753212781 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56069dba8920 0x56069dbaad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.536+0000 7f59b406a740 4 client.17538 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:39:56.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.536+0000 7f59b406a740 1 -- 172.21.15.70:0/753212781 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56069daf9120 con 0x56069dba8920 2024-04-01T01:39:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.536+0000 7f59aaffd640 1 --2- 172.21.15.70:0/753212781 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56069dba8920 0x56069dbaad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.536+0000 7f598eff5640 10 client.17538 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.537+0000 7f59aaffd640 1 --2- 172.21.15.70:0/753212781 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56069dba8920 0x56069dbaad60 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf8bed4878e18220 server_cookie=231a2db40af4f956 in_seq=0 out_seq=0 2024-04-01T01:39:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.537+0000 7f598eff5640 1 -- 172.21.15.70:0/753212781 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56069daf9120 con 0x56069dba8920 2024-04-01T01:39:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.537+0000 7f598eff5640 10 client.17538 handle_command_reply: tid=0 2024-04-01T01:39:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.537+0000 7f59b406a740 1 client.17538 shutdown 2024-04-01T01:39:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.538+0000 7f59b406a740 1 -- 172.21.15.70:0/753212781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db92af0 msgr2=0x56069db9ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:56.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.538+0000 7f59b406a740 1 --2- 172.21.15.70:0/753212781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db92af0 0x56069db9ce30 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f59a000d670 tx=0x7f59a000dfa0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.538+0000 7f59b406a740 1 -- 172.21.15.70:0/753212781 shutdown_connections 2024-04-01T01:39:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.538+0000 7f59b406a740 1 --2- 172.21.15.70:0/753212781 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56069dba8920 0x56069dbaad60 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.538+0000 7f59b406a740 1 --2- 172.21.15.70:0/753212781 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db98650 0x56069dba43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.538+0000 7f598eff5640 0 client.17538 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.538+0000 7f59b406a740 1 --2- 172.21.15.70:0/753212781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db934e0 0x56069db9d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.538+0000 7f59b406a740 1 --2- 172.21.15.70:0/753212781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db92af0 0x56069db9ce30 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.538+0000 7f59b406a740 1 -- 172.21.15.70:0/753212781 >> 172.21.15.70:0/753212781 conn(0x56069db7d050 msgr2=0x56069db7c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:56.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.539+0000 7f59b406a740 1 -- 172.21.15.70:0/753212781 shutdown_connections 2024-04-01T01:39:56.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.539+0000 7f59b406a740 1 -- 172.21.15.70:0/753212781 wait complete. 2024-04-01T01:39:56.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.539+0000 7f59b406a740 20 client.17538 trim_cache size 0 max 16384 2024-04-01T01:39:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.566+0000 7f59b406a740 1 Processor -- start 2024-04-01T01:39:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.566+0000 7f59b406a740 1 -- start start 2024-04-01T01:39:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.566+0000 7f59b406a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db92af0 0x56069db7d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.566+0000 7f59b406a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db934e0 0x56069dba4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.566+0000 7f59b406a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db98650 0x56069dbb0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.566+0000 7f59b406a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f599800fd10 con 0x56069db98650 2024-04-01T01:39:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.566+0000 7f59b406a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5998011850 con 0x56069db92af0 2024-04-01T01:39:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.566+0000 7f59b406a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f59980103e0 con 0x56069db934e0 2024-04-01T01:39:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.566+0000 7f59aaffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db934e0 0x56069dba4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.567+0000 7f59aaffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db934e0 0x56069dba4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:50092/0 (socket says 172.21.15.70:50092) 2024-04-01T01:39:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.567+0000 7f59aaffd640 1 -- 172.21.15.70:0/672842459 learned_addr learned my addr 172.21.15.70:0/672842459 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.567+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/672842459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db92af0 0x56069db7d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.567+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/672842459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db98650 0x56069dbb0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.567+0000 7f59aa7fc640 1 -- 172.21.15.70:0/672842459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db934e0 msgr2=0x56069dba4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.567+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/672842459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db934e0 0x56069dba4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.567+0000 7f59aa7fc640 1 -- 172.21.15.70:0/672842459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db98650 msgr2=0x56069dbb0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.567+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/672842459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db98650 0x56069dbb0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.567+0000 7f59aa7fc640 1 -- 172.21.15.70:0/672842459 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56069db938c0 con 0x56069db92af0 2024-04-01T01:39:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.568+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/672842459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db92af0 0x56069db7d430 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f59a0012e70 tx=0x7f59a0010fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26ab1194b77405b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.568+0000 7f5990ff9640 1 -- 172.21.15.70:0/672842459 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a000d9c0 con 0x56069db92af0 2024-04-01T01:39:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.568+0000 7f5990ff9640 1 -- 172.21.15.70:0/672842459 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59a000a680 con 0x56069db92af0 2024-04-01T01:39:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.568+0000 7f5990ff9640 1 -- 172.21.15.70:0/672842459 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a0005480 con 0x56069db92af0 2024-04-01T01:39:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.569+0000 7f59b406a740 1 -- 172.21.15.70:0/672842459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db92af0 msgr2=0x56069db7d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.569+0000 7f59b406a740 1 --2- 172.21.15.70:0/672842459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db92af0 0x56069db7d430 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f59a0012e70 tx=0x7f59a0010fa0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.569+0000 7f59b406a740 1 -- 172.21.15.70:0/672842459 shutdown_connections 2024-04-01T01:39:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.569+0000 7f59b406a740 1 --2- 172.21.15.70:0/672842459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db98650 0x56069dbb0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.569+0000 7f59b406a740 1 --2- 172.21.15.70:0/672842459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db934e0 0x56069dba4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.569+0000 7f59b406a740 1 --2- 172.21.15.70:0/672842459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db92af0 0x56069db7d430 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.569+0000 7f59b406a740 1 -- 172.21.15.70:0/672842459 >> 172.21.15.70:0/672842459 conn(0x56069db7d050 msgr2=0x56069db97e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.569+0000 7f59b406a740 1 -- 172.21.15.70:0/672842459 shutdown_connections 2024-04-01T01:39:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.569+0000 7f59b406a740 1 -- 172.21.15.70:0/672842459 wait complete. 2024-04-01T01:39:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.570+0000 7f59b406a740 1 -- start start 2024-04-01T01:39:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.570+0000 7f59b406a740 1 Processor -- start 2024-04-01T01:39:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.570+0000 7f59b406a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db92af0 0x56069dba15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.570+0000 7f59b406a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db934e0 0x56069dba1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.570+0000 7f59b406a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db98650 0x56069dc0cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.570+0000 7f59b406a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5998011180 con 0x56069db934e0 2024-04-01T01:39:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.570+0000 7f59b406a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5998010ab0 con 0x56069db98650 2024-04-01T01:39:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.570+0000 7f59b406a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f599800dcf0 con 0x56069db92af0 2024-04-01T01:39:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.570+0000 7f59b406a740 10 client.0 _ll_register_callbacks cb 0x7f599c0096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:39:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.571+0000 7f59aa7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db92af0 0x56069dba15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.571+0000 7f59aa7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db92af0 0x56069dba15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:50100/0 (socket says 172.21.15.70:50100) 2024-04-01T01:39:56.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.571+0000 7f59aa7fc640 1 -- 172.21.15.70:0/946436694 learned_addr learned my addr 172.21.15.70:0/946436694 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:39:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.571+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/946436694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db98650 0x56069dc0cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.571+0000 7f59aaffd640 1 --2- 172.21.15.70:0/946436694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db934e0 0x56069dba1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.571+0000 7f59aa7fc640 1 -- 172.21.15.70:0/946436694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db92af0 msgr2=0x56069dba15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.571+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/946436694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db92af0 0x56069dba15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.571+0000 7f59aa7fc640 1 -- 172.21.15.70:0/946436694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db934e0 msgr2=0x56069dba1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:39:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.571+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/946436694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db934e0 0x56069dba1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.571+0000 7f59aa7fc640 1 -- 172.21.15.70:0/946436694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56069dba2180 con 0x56069db98650 2024-04-01T01:39:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.571+0000 7f59aaffd640 1 --2- 172.21.15.70:0/946436694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db934e0 0x56069dba1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:39:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.571+0000 7f59aa7fc640 1 --2- 172.21.15.70:0/946436694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db98650 0x56069dc0cc50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x56069dba25a0 tx=0x7f59a0012590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a648ce9552c4add3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:39:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.571+0000 7f598eff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:39:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.572+0000 7f598eff5640 1 -- 172.21.15.70:0/946436694 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a0011500 con 0x56069db98650 2024-04-01T01:39:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.572+0000 7f598eff5640 1 -- 172.21.15.70:0/946436694 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f59a000ab10 con 0x56069db98650 2024-04-01T01:39:56.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.572+0000 7f59b406a740 1 -- 172.21.15.70:0/946436694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59a000a680 con 0x56069db98650 2024-04-01T01:39:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.572+0000 7f598eff5640 1 -- 172.21.15.70:0/946436694 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f59a000ad10 con 0x56069db98650 2024-04-01T01:39:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.572+0000 7f59b406a740 10 client.27455 fetch_fsmap learned FSMap version 31 2024-04-01T01:39:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.572+0000 7f59b406a740 1 -- 172.21.15.70:0/946436694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56069dc0e2d0 con 0x56069db98650 2024-04-01T01:39:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.572+0000 7f598eff5640 1 -- 172.21.15.70:0/946436694 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f59a000a680 con 0x56069db98650 2024-04-01T01:39:56.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.573+0000 7f598eff5640 1 -- 172.21.15.70:0/946436694 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f59a0034070 con 0x56069db98650 2024-04-01T01:39:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.573+0000 7f59b406a740 10 client.27455 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:39:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.573+0000 7f59b406a740 10 client.27455 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:39:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.573+0000 7f59b406a740 20 client.27455 populate_metadata read hostname 'smithi070' 2024-04-01T01:39:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.573+0000 7f59b406a740 1 --2- 172.21.15.70:0/946436694 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56069dc0ec60 0x56069dc11080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:39:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.573+0000 7f59b406a740 4 client.27455 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:39:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.573+0000 7f59b406a740 1 -- 172.21.15.70:0/946436694 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56069daf2e30 con 0x56069dc0ec60 2024-04-01T01:39:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.574+0000 7f59aaffd640 1 --2- 172.21.15.70:0/946436694 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56069dc0ec60 0x56069dc11080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:39:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.575+0000 7f59aaffd640 1 --2- 172.21.15.70:0/946436694 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56069dc0ec60 0x56069dc11080 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e520669acfe27552 server_cookie=3ecf5b8743c78f65 in_seq=0 out_seq=0 2024-04-01T01:39:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.575+0000 7f598eff5640 10 client.27455 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:56.577 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:56 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:39:56.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:56 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:39:56.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:56 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:39:56.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:56 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:39:56.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:56 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:39:56.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.607+0000 7f598eff5640 1 -- 172.21.15.70:0/946436694 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56069daf2e30 con 0x56069dc0ec60 2024-04-01T01:39:56.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.607+0000 7f598eff5640 10 client.27455 handle_command_reply: tid=0 2024-04-01T01:39:56.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.607+0000 7f59b406a740 1 client.27455 shutdown 2024-04-01T01:39:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.609+0000 7f59b406a740 1 -- 172.21.15.70:0/946436694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db98650 msgr2=0x56069dc0cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.609+0000 7f59b406a740 1 --2- 172.21.15.70:0/946436694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db98650 0x56069dc0cc50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x56069dba25a0 tx=0x7f59a0012590 comp rx=0 tx=0).stop 2024-04-01T01:39:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.609+0000 7f59b406a740 1 -- 172.21.15.70:0/946436694 shutdown_connections 2024-04-01T01:39:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.609+0000 7f59b406a740 1 --2- 172.21.15.70:0/946436694 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56069dc0ec60 0x56069dc11080 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.609+0000 7f59b406a740 1 --2- 172.21.15.70:0/946436694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56069db98650 0x56069dc0cc50 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.609+0000 7f598eff5640 0 client.27455 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:39:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.609+0000 7f59b406a740 1 --2- 172.21.15.70:0/946436694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56069db934e0 0x56069dba1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.609+0000 7f59b406a740 1 --2- 172.21.15.70:0/946436694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56069db92af0 0x56069dba15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.609+0000 7f59b406a740 1 -- 172.21.15.70:0/946436694 >> 172.21.15.70:0/946436694 conn(0x56069db7d050 msgr2=0x56069db7d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.609+0000 7f59b406a740 1 -- 172.21.15.70:0/946436694 shutdown_connections 2024-04-01T01:39:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.610+0000 7f59b406a740 1 -- 172.21.15.70:0/946436694 wait complete. 2024-04-01T01:39:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.610+0000 7f59b406a740 20 client.27455 trim_cache size 0 max 16384 2024-04-01T01:39:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:39:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:39:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:39:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:39:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.611+0000 7f59b148d640 1 -- 172.21.15.70:0/3131165988 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59940a25a0 msgr2=0x7f59940a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.611+0000 7f59b148d640 1 --2- 172.21.15.70:0/3131165988 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59940a25a0 0x7f59940a4a60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f59ac066e90 tx=0x7f59ac067690 comp rx=0 tx=0).stop 2024-04-01T01:39:56.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.611+0000 7f59b148d640 1 -- 172.21.15.70:0/3131165988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a409df50 msgr2=0x7f59a4139b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:39:56.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.611+0000 7f59b148d640 1 --2- 172.21.15.70:0/3131165988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a409df50 0x7f59a4139b80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f59ac050db0 tx=0x7f59ac07bd90 comp rx=0 tx=0).stop 2024-04-01T01:39:56.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.611+0000 7f59b148d640 1 -- 172.21.15.70:0/3131165988 shutdown_connections 2024-04-01T01:39:56.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.611+0000 7f59b148d640 1 --2- 172.21.15.70:0/3131165988 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f59940a25a0 0x7f59940a4a60 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.611+0000 7f59b148d640 1 --2- 172.21.15.70:0/3131165988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a409df50 0x7f59a4139b80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.612+0000 7f59b148d640 1 --2- 172.21.15.70:0/3131165988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f59a409d590 0x7f59a4132b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.612+0000 7f59b148d640 1 --2- 172.21.15.70:0/3131165988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59a409c3a0 0x7f59a41325d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:39:56.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.612+0000 7f59b148d640 1 -- 172.21.15.70:0/3131165988 >> 172.21.15.70:0/3131165988 conn(0x7f59a400b480 msgr2=0x7f59a40a0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:39:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.614+0000 7f59b148d640 1 -- 172.21.15.70:0/3131165988 shutdown_connections 2024-04-01T01:39:56.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:39:56.614+0000 7f59b148d640 1 -- 172.21.15.70:0/3131165988 wait complete. 2024-04-01T01:39:56.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 310.0 s, 926.4 tps, lat 108.156 ms stddev 166.501 2024-04-01T01:39:57.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:57 smithi189 ceph-mon[35753]: pgmap v553: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 16 MiB/s wr, 969 op/s 2024-04-01T01:39:57.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:57 smithi070 ceph-mon[32179]: pgmap v553: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 16 MiB/s wr, 969 op/s 2024-04-01T01:39:57.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:57 smithi081 ceph-mon[35142]: pgmap v553: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 16 MiB/s wr, 969 op/s 2024-04-01T01:39:59.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:39:58 smithi189 ceph-mon[35753]: pgmap v554: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 14 MiB/s wr, 957 op/s 2024-04-01T01:39:59.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:39:58 smithi070 ceph-mon[32179]: pgmap v554: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 14 MiB/s wr, 957 op/s 2024-04-01T01:39:59.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:39:58 smithi081 ceph-mon[35142]: pgmap v554: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 14 MiB/s wr, 957 op/s 2024-04-01T01:40:00.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:00 smithi081 ceph-mon[35142]: overall HEALTH_OK 2024-04-01T01:40:00.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:00 smithi189 ceph-mon[35753]: overall HEALTH_OK 2024-04-01T01:40:00.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:00 smithi070 ceph-mon[32179]: overall HEALTH_OK 2024-04-01T01:40:01.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:01 smithi081 ceph-mon[35142]: pgmap v555: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 13 MiB/s wr, 917 op/s 2024-04-01T01:40:01.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:01 smithi189 ceph-mon[35753]: pgmap v555: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 13 MiB/s wr, 917 op/s 2024-04-01T01:40:01.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:01 smithi070 ceph-mon[32179]: pgmap v555: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 13 MiB/s wr, 917 op/s 2024-04-01T01:40:01.897 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 315.0 s, 931.2 tps, lat 107.387 ms stddev 122.235 2024-04-01T01:40:02.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:02 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:40:02.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:02 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:40:02.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:02 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:40:03.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:03 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi081 to 176.2M 2024-04-01T01:40:03.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:03 smithi081 ceph-mon[35142]: Unable to set osd_memory_target on smithi081 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-01T01:40:03.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:03 smithi081 ceph-mon[35142]: pgmap v556: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 14 MiB/s wr, 1.04k op/s 2024-04-01T01:40:03.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:40:03.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:40:03.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:40:03.404 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.405 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.405 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.405 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.405 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:03 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi081 to 176.2M 2024-04-01T01:40:03.405 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:03 smithi189 ceph-mon[35753]: Unable to set osd_memory_target on smithi081 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-01T01:40:03.405 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:03 smithi189 ceph-mon[35753]: pgmap v556: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 14 MiB/s wr, 1.04k op/s 2024-04-01T01:40:03.405 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.405 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.405 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.405 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.405 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:40:03.405 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:40:03.405 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:40:03.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:03 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi081 to 176.2M 2024-04-01T01:40:03.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:03 smithi070 ceph-mon[32179]: Unable to set osd_memory_target on smithi081 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-01T01:40:03.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:03 smithi070 ceph-mon[32179]: pgmap v556: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 14 MiB/s wr, 1.04k op/s 2024-04-01T01:40:03.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:40:03.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:40:03.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:40:03.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:40:04.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:04 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-04-01T01:40:04.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:04 smithi081 ceph-mon[35142]: Unable to set osd_memory_target on smithi070 to 184772403: error parsing value: Value '184772403' is below minimum 939524096 2024-04-01T01:40:04.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:04 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-04-01T01:40:04.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:04 smithi189 ceph-mon[35753]: Unable to set osd_memory_target on smithi070 to 184772403: error parsing value: Value '184772403' is below minimum 939524096 2024-04-01T01:40:04.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:04 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-04-01T01:40:04.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:04 smithi070 ceph-mon[32179]: Unable to set osd_memory_target on smithi070 to 184772403: error parsing value: Value '184772403' is below minimum 939524096 2024-04-01T01:40:05.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:05 smithi189 ceph-mon[35753]: pgmap v557: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 12 MiB/s wr, 907 op/s 2024-04-01T01:40:05.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:05 smithi070 ceph-mon[32179]: pgmap v557: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 12 MiB/s wr, 907 op/s 2024-04-01T01:40:05.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:05 smithi081 ceph-mon[35142]: pgmap v557: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 12 MiB/s wr, 907 op/s 2024-04-01T01:40:06.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 320.0 s, 789.4 tps, lat 126.941 ms stddev 163.635 2024-04-01T01:40:07.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:06 smithi189 ceph-mon[35753]: pgmap v558: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 14 MiB/s wr, 1.00k op/s 2024-04-01T01:40:07.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:06 smithi070 ceph-mon[32179]: pgmap v558: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 14 MiB/s wr, 1.00k op/s 2024-04-01T01:40:07.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:06 smithi081 ceph-mon[35142]: pgmap v558: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 14 MiB/s wr, 1.00k op/s 2024-04-01T01:40:09.129 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:08 smithi081 ceph-mon[35142]: pgmap v559: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 13 MiB/s wr, 965 op/s 2024-04-01T01:40:09.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:08 smithi189 ceph-mon[35753]: pgmap v559: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 13 MiB/s wr, 965 op/s 2024-04-01T01:40:09.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:08 smithi070 ceph-mon[32179]: pgmap v559: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 13 MiB/s wr, 965 op/s 2024-04-01T01:40:09.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.771+0000 7fdadcf7a640 1 -- 172.21.15.70:0/2627742709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdad81019a0 msgr2=0x7fdad8108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:09.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.771+0000 7fdadcf7a640 1 --2- 172.21.15.70:0/2627742709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdad81019a0 0x7fdad8108230 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fdad0005fd0 tx=0x7fdad002c110 comp rx=0 tx=0).stop 2024-04-01T01:40:09.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.771+0000 7fdadcf7a640 1 -- 172.21.15.70:0/2627742709 shutdown_connections 2024-04-01T01:40:09.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.771+0000 7fdadcf7a640 1 --2- 172.21.15.70:0/2627742709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdad81019a0 0x7fdad8108230 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.771+0000 7fdadcf7a640 1 --2- 172.21.15.70:0/2627742709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdad8100fe0 0x7fdad8101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.771+0000 7fdadcf7a640 1 --2- 172.21.15.70:0/2627742709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdad80ffdf0 0x7fdad8100210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.771+0000 7fdadcf7a640 1 -- 172.21.15.70:0/2627742709 >> 172.21.15.70:0/2627742709 conn(0x7fdad80fb3a0 msgr2=0x7fdad80fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.773+0000 7fdadcf7a640 1 -- 172.21.15.70:0/2627742709 shutdown_connections 2024-04-01T01:40:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.773+0000 7fdadcf7a640 1 -- 172.21.15.70:0/2627742709 wait complete. 2024-04-01T01:40:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.774+0000 7fdadcf7a640 1 Processor -- start 2024-04-01T01:40:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.774+0000 7fdadcf7a640 1 -- start start 2024-04-01T01:40:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.774+0000 7fdadcf7a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdad80ffdf0 0x7fdad8195680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.774+0000 7fdadcf7a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdad8100fe0 0x7fdad8195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.775+0000 7fdadcf7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdad81019a0 0x7fdad819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.775+0000 7fdadcf7a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdad810b0b0 con 0x7fdad80ffdf0 2024-04-01T01:40:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.775+0000 7fdad5d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdad8100fe0 0x7fdad8195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.775+0000 7fdad5d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdad8100fe0 0x7fdad8195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:44236/0 (socket says 172.21.15.70:44236) 2024-04-01T01:40:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.775+0000 7fdad5d9b640 1 -- 172.21.15.70:0/1811316751 learned_addr learned my addr 172.21.15.70:0/1811316751 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.775+0000 7fdadcf7a640 1 -- 172.21.15.70:0/1811316751 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdad810af30 con 0x7fdad81019a0 2024-04-01T01:40:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.775+0000 7fdadcf7a640 1 -- 172.21.15.70:0/1811316751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdad810b230 con 0x7fdad8100fe0 2024-04-01T01:40:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.775+0000 7fdad6d9d640 1 --2- 172.21.15.70:0/1811316751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdad81019a0 0x7fdad819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.775+0000 7fdad659c640 1 --2- 172.21.15.70:0/1811316751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdad80ffdf0 0x7fdad8195680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.775+0000 7fdad5d9b640 1 -- 172.21.15.70:0/1811316751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdad81019a0 msgr2=0x7fdad819cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.775+0000 7fdad5d9b640 1 --2- 172.21.15.70:0/1811316751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdad81019a0 0x7fdad819cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.775+0000 7fdad5d9b640 1 -- 172.21.15.70:0/1811316751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdad80ffdf0 msgr2=0x7fdad8195680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.775+0000 7fdad5d9b640 1 --2- 172.21.15.70:0/1811316751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdad80ffdf0 0x7fdad8195680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.775+0000 7fdad5d9b640 1 -- 172.21.15.70:0/1811316751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad819d2a0 con 0x7fdad8100fe0 2024-04-01T01:40:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.776+0000 7fdad5d9b640 1 --2- 172.21.15.70:0/1811316751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdad8100fe0 0x7fdad8195bc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdacc00e9f0 tx=0x7fdacc00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.776+0000 7fdac77fe640 1 -- 172.21.15.70:0/1811316751 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdacc00ccc0 con 0x7fdad8100fe0 2024-04-01T01:40:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.776+0000 7fdac77fe640 1 -- 172.21.15.70:0/1811316751 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdacc0108e0 con 0x7fdad8100fe0 2024-04-01T01:40:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.776+0000 7fdac77fe640 1 -- 172.21.15.70:0/1811316751 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdacc0187c0 con 0x7fdad8100fe0 2024-04-01T01:40:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.777+0000 7fdadcf7a640 1 -- 172.21.15.70:0/1811316751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdad819d590 con 0x7fdad8100fe0 2024-04-01T01:40:09.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.777+0000 7fdadcf7a640 1 -- 172.21.15.70:0/1811316751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdad81a1e60 con 0x7fdad8100fe0 2024-04-01T01:40:09.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.780+0000 7fdadfc38740 1 Processor -- start 2024-04-01T01:40:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.780+0000 7fdadfc38740 1 -- start start 2024-04-01T01:40:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.781+0000 7fdadfc38740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d60e0 0x5601128d6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.781+0000 7fdadfc38740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128d6ad0 0x5601128db5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.781+0000 7fdadfc38740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128dbc40 0x5601128de0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.781+0000 7fdadfc38740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5601128318f0 con 0x5601128d6ad0 2024-04-01T01:40:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.781+0000 7fdadfc38740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5601127da8d0 con 0x5601128dbc40 2024-04-01T01:40:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.781+0000 7fdadfc38740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56011282d990 con 0x5601128d60e0 2024-04-01T01:40:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.781+0000 7fdac77fe640 1 -- 172.21.15.70:0/1811316751 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fdacc018960 con 0x7fdad8100fe0 2024-04-01T01:40:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.781+0000 7fdad6d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d60e0 0x5601128d6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.781+0000 7fdad6d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d60e0 0x5601128d6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:44242/0 (socket says 172.21.15.70:44242) 2024-04-01T01:40:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.781+0000 7fdad6d9d640 1 -- 172.21.15.70:0/3794429692 learned_addr learned my addr 172.21.15.70:0/3794429692 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:09.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.781+0000 7fdad5d9b640 1 --2- 172.21.15.70:0/3794429692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128dbc40 0x5601128de0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.781+0000 7fdad6d9d640 1 -- 172.21.15.70:0/3794429692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128dbc40 msgr2=0x5601128de0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.781+0000 7fdad6d9d640 1 --2- 172.21.15.70:0/3794429692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128dbc40 0x5601128de0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.781+0000 7fdad6d9d640 1 -- 172.21.15.70:0/3794429692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128d6ad0 msgr2=0x5601128db5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.781+0000 7fdad6d9d640 1 --2- 172.21.15.70:0/3794429692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128d6ad0 0x5601128db5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.781+0000 7fdad6d9d640 1 -- 172.21.15.70:0/3794429692 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560112546210 con 0x5601128d60e0 2024-04-01T01:40:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.782+0000 7fdac77fe640 1 --2- 172.21.15.70:0/1811316751 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdac00a2780 0x7fdac00a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.782+0000 7fdac77fe640 1 -- 172.21.15.70:0/1811316751 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fdacc0ced60 con 0x7fdad8100fe0 2024-04-01T01:40:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.782+0000 7fdad6d9d640 1 --2- 172.21.15.70:0/3794429692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d60e0 0x5601128d6500 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdad0035000 tx=0x7fdad0005e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fed3cb840a7b1e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.782+0000 7fdac4ff9640 1 -- 172.21.15.70:0/3794429692 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad0045070 con 0x5601128d60e0 2024-04-01T01:40:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.783+0000 7fdad659c640 1 --2- 172.21.15.70:0/1811316751 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdac00a2780 0x7fdac00a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.783+0000 7fdac4ff9640 1 -- 172.21.15.70:0/3794429692 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdad000a840 con 0x5601128d60e0 2024-04-01T01:40:09.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.783+0000 7fdac4ff9640 1 -- 172.21.15.70:0/3794429692 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad0004280 con 0x5601128d60e0 2024-04-01T01:40:09.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.784+0000 7fdadfc38740 1 -- 172.21.15.70:0/3794429692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d60e0 msgr2=0x5601128d6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:09.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.784+0000 7fdad659c640 1 --2- 172.21.15.70:0/1811316751 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdac00a2780 0x7fdac00a4c40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fdac800d6a0 tx=0x7fdac800d630 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:09.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.784+0000 7fdadfc38740 1 --2- 172.21.15.70:0/3794429692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d60e0 0x5601128d6500 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdad0035000 tx=0x7fdad0005e30 comp rx=0 tx=0).stop 2024-04-01T01:40:09.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.784+0000 7fdadfc38740 1 -- 172.21.15.70:0/3794429692 shutdown_connections 2024-04-01T01:40:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.784+0000 7fdadfc38740 1 --2- 172.21.15.70:0/3794429692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128dbc40 0x5601128de0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.784+0000 7fdadfc38740 1 --2- 172.21.15.70:0/3794429692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128d6ad0 0x5601128db5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.784+0000 7fdadfc38740 1 --2- 172.21.15.70:0/3794429692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d60e0 0x5601128d6500 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.784+0000 7fdadfc38740 1 -- 172.21.15.70:0/3794429692 >> 172.21.15.70:0/3794429692 conn(0x5601128d1650 msgr2=0x5601128d1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.784+0000 7fdadfc38740 1 -- 172.21.15.70:0/3794429692 shutdown_connections 2024-04-01T01:40:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.784+0000 7fdadfc38740 1 -- 172.21.15.70:0/3794429692 wait complete. 2024-04-01T01:40:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.784+0000 7fdadfc38740 1 -- start start 2024-04-01T01:40:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.785+0000 7fdadfc38740 1 Processor -- start 2024-04-01T01:40:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.785+0000 7fdadfc38740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128d60e0 0x5601128e03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.785+0000 7fdadfc38740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d6ad0 0x5601128e0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.785+0000 7fdadfc38740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128dbc40 0x5601128e7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.785+0000 7fdadfc38740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5601128deb90 con 0x5601128d60e0 2024-04-01T01:40:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.785+0000 7fdadfc38740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5601127cff80 con 0x5601128dbc40 2024-04-01T01:40:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.785+0000 7fdadfc38740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5601128ded10 con 0x5601128d6ad0 2024-04-01T01:40:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.785+0000 7fdad6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128dbc40 0x5601128e7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.785+0000 7fdad6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128dbc40 0x5601128e7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38664/0 (socket says 172.21.15.70:38664) 2024-04-01T01:40:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.785+0000 7fdad6d9d640 1 -- 172.21.15.70:0/1510206554 learned_addr learned my addr 172.21.15.70:0/1510206554 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.785+0000 7fdad659c640 1 --2- 172.21.15.70:0/1510206554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128d60e0 0x5601128e03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.785+0000 7fdad5d9b640 1 --2- 172.21.15.70:0/1510206554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d6ad0 0x5601128e0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.785+0000 7fdadfc38740 10 client.0 _ll_register_callbacks cb 0x560112828290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.786+0000 7fdad6d9d640 1 -- 172.21.15.70:0/1510206554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d6ad0 msgr2=0x5601128e0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:09.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.786+0000 7fdad6d9d640 1 --2- 172.21.15.70:0/1510206554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d6ad0 0x5601128e0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.786+0000 7fdad6d9d640 1 -- 172.21.15.70:0/1510206554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128d60e0 msgr2=0x5601128e03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.786+0000 7fdad6d9d640 1 --2- 172.21.15.70:0/1510206554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128d60e0 0x5601128e03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.786+0000 7fdad6d9d640 1 -- 172.21.15.70:0/1510206554 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560112612620 con 0x5601128dbc40 2024-04-01T01:40:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.786+0000 7fdad6d9d640 1 --2- 172.21.15.70:0/1510206554 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128dbc40 0x5601128e7940 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdad000ab50 tx=0x7fdad002c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95c7807ee26e86e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.786+0000 7fdabdffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:40:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.786+0000 7fdabdffb640 1 -- 172.21.15.70:0/1510206554 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad0045040 con 0x5601128dbc40 2024-04-01T01:40:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.787+0000 7fdadfc38740 1 -- 172.21.15.70:0/1510206554 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdad000a840 con 0x5601128dbc40 2024-04-01T01:40:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.787+0000 7fdabdffb640 1 -- 172.21.15.70:0/1510206554 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fdad00041d0 con 0x5601128dbc40 2024-04-01T01:40:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.787+0000 7fdabdffb640 1 -- 172.21.15.70:0/1510206554 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdad00043d0 con 0x5601128dbc40 2024-04-01T01:40:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.787+0000 7fdadfc38740 10 client.27467 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.787+0000 7fdadfc38740 1 -- 172.21.15.70:0/1510206554 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5601128ea8e0 con 0x5601128dbc40 2024-04-01T01:40:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.787+0000 7fdabdffb640 1 -- 172.21.15.70:0/1510206554 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdad000a840 con 0x5601128dbc40 2024-04-01T01:40:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.788+0000 7fdabdffb640 1 -- 172.21.15.70:0/1510206554 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fdad000b040 con 0x5601128dbc40 2024-04-01T01:40:09.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.788+0000 7fdadfc38740 10 client.27467 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:09.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.788+0000 7fdadfc38740 10 client.27467 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:40:09.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.788+0000 7fdadfc38740 20 client.27467 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:09.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.788+0000 7fdadfc38740 1 --2- 172.21.15.70:0/1510206554 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5601128ec170 0x5601128ee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.788+0000 7fdadfc38740 4 client.27467 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:40:09.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.788+0000 7fdadfc38740 1 -- 172.21.15.70:0/1510206554 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560112845120 con 0x5601128ec170 2024-04-01T01:40:09.790 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.788+0000 7fdad659c640 1 --2- 172.21.15.70:0/1510206554 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5601128ec170 0x5601128ee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.789+0000 7fdad659c640 1 --2- 172.21.15.70:0/1510206554 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5601128ec170 0x5601128ee5b0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30121e392ab49c15 server_cookie=dd4d729a1a0ba3a7 in_seq=0 out_seq=0 2024-04-01T01:40:09.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.789+0000 7fdabdffb640 10 client.27467 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:09.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.789+0000 7fdabdffb640 1 -- 172.21.15.70:0/1510206554 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560112845120 con 0x5601128ec170 2024-04-01T01:40:09.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.789+0000 7fdabdffb640 10 client.27467 handle_command_reply: tid=0 2024-04-01T01:40:09.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.790+0000 7fdadfc38740 1 client.27467 shutdown 2024-04-01T01:40:09.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.791+0000 7fdadfc38740 1 -- 172.21.15.70:0/1510206554 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128dbc40 msgr2=0x5601128e7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:09.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.791+0000 7fdadfc38740 1 --2- 172.21.15.70:0/1510206554 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128dbc40 0x5601128e7940 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdad000ab50 tx=0x7fdad002c670 comp rx=0 tx=0).stop 2024-04-01T01:40:09.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.791+0000 7fdadfc38740 1 -- 172.21.15.70:0/1510206554 shutdown_connections 2024-04-01T01:40:09.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.791+0000 7fdadfc38740 1 --2- 172.21.15.70:0/1510206554 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5601128ec170 0x5601128ee5b0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.791+0000 7fdadfc38740 1 --2- 172.21.15.70:0/1510206554 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128dbc40 0x5601128e7940 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.791+0000 7fdabdffb640 0 client.27467 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:09.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.791+0000 7fdadfc38740 1 --2- 172.21.15.70:0/1510206554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d6ad0 0x5601128e0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.791+0000 7fdadfc38740 1 --2- 172.21.15.70:0/1510206554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128d60e0 0x5601128e03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.791+0000 7fdadfc38740 1 -- 172.21.15.70:0/1510206554 >> 172.21.15.70:0/1510206554 conn(0x5601128d1650 msgr2=0x5601128c7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:09.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.792+0000 7fdadfc38740 1 -- 172.21.15.70:0/1510206554 shutdown_connections 2024-04-01T01:40:09.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.792+0000 7fdadfc38740 1 -- 172.21.15.70:0/1510206554 wait complete. 2024-04-01T01:40:09.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.792+0000 7fdadfc38740 20 client.27467 trim_cache size 0 max 16384 2024-04-01T01:40:09.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.821+0000 7fdadfc38740 1 Processor -- start 2024-04-01T01:40:09.822 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.821+0000 7fdadfc38740 1 -- start start 2024-04-01T01:40:09.822 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.821+0000 7fdadfc38740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d60e0 0x5601128c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.822+0000 7fdadfc38740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128d6ad0 0x5601128e7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.822+0000 7fdadfc38740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128dbc40 0x5601128df140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.822+0000 7fdad659c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d60e0 0x5601128c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.822+0000 7fdad659c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d60e0 0x5601128c3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:44272/0 (socket says 172.21.15.70:44272) 2024-04-01T01:40:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.822+0000 7fdad5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128d6ad0 0x5601128e7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.822+0000 7fdad5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128d6ad0 0x5601128e7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38676/0 (socket says 172.21.15.70:38676) 2024-04-01T01:40:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.822+0000 7fdad659c640 1 -- 172.21.15.70:0/2754338730 learned_addr learned my addr 172.21.15.70:0/2754338730 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.822+0000 7fdadfc38740 1 -- 172.21.15.70:0/2754338730 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdab800fcd0 con 0x5601128dbc40 2024-04-01T01:40:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.822+0000 7fdadfc38740 1 -- 172.21.15.70:0/2754338730 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdab8011810 con 0x5601128d6ad0 2024-04-01T01:40:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.822+0000 7fdadfc38740 1 -- 172.21.15.70:0/2754338730 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdab80103a0 con 0x5601128d60e0 2024-04-01T01:40:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.822+0000 7fdad6d9d640 1 --2- 172.21.15.70:0/2754338730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128dbc40 0x5601128df140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.824+0000 7fdad6d9d640 1 -- 172.21.15.70:0/2754338730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d60e0 msgr2=0x5601128c3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.824+0000 7fdad6d9d640 1 --2- 172.21.15.70:0/2754338730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d60e0 0x5601128c3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.824+0000 7fdad6d9d640 1 -- 172.21.15.70:0/2754338730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128d6ad0 msgr2=0x5601128e7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.824+0000 7fdad6d9d640 1 --2- 172.21.15.70:0/2754338730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128d6ad0 0x5601128e7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.824+0000 7fdad6d9d640 1 -- 172.21.15.70:0/2754338730 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601128d23c0 con 0x5601128dbc40 2024-04-01T01:40:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.824+0000 7fdad6d9d640 1 --2- 172.21.15.70:0/2754338730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128dbc40 0x5601128df140 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fdad00336e0 tx=0x7fdad0033eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e0a4ebd556d05f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.825+0000 7fdabffff640 1 -- 172.21.15.70:0/2754338730 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad0045040 con 0x5601128dbc40 2024-04-01T01:40:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.825+0000 7fdabffff640 1 -- 172.21.15.70:0/2754338730 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdad0033bc0 con 0x5601128dbc40 2024-04-01T01:40:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.825+0000 7fdabffff640 1 -- 172.21.15.70:0/2754338730 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad0003b80 con 0x5601128dbc40 2024-04-01T01:40:09.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.826+0000 7fdadfc38740 1 -- 172.21.15.70:0/2754338730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128dbc40 msgr2=0x5601128df140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:09.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.826+0000 7fdadfc38740 1 --2- 172.21.15.70:0/2754338730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128dbc40 0x5601128df140 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fdad00336e0 tx=0x7fdad0033eb0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.827+0000 7fdadfc38740 1 -- 172.21.15.70:0/2754338730 shutdown_connections 2024-04-01T01:40:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.827+0000 7fdadfc38740 1 --2- 172.21.15.70:0/2754338730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128dbc40 0x5601128df140 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.827+0000 7fdadfc38740 1 --2- 172.21.15.70:0/2754338730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128d6ad0 0x5601128e7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.827+0000 7fdadfc38740 1 --2- 172.21.15.70:0/2754338730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128d60e0 0x5601128c3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.827+0000 7fdadfc38740 1 -- 172.21.15.70:0/2754338730 >> 172.21.15.70:0/2754338730 conn(0x5601128d1650 msgr2=0x5601128db3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.827+0000 7fdadfc38740 1 -- 172.21.15.70:0/2754338730 shutdown_connections 2024-04-01T01:40:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.827+0000 7fdadfc38740 1 -- 172.21.15.70:0/2754338730 wait complete. 2024-04-01T01:40:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.828+0000 7fdadfc38740 1 -- start start 2024-04-01T01:40:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.828+0000 7fdadfc38740 1 Processor -- start 2024-04-01T01:40:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.828+0000 7fdadfc38740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128d60e0 0x5601128e77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.828+0000 7fdadfc38740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128d6ad0 0x5601128e4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.828+0000 7fdadfc38740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128dbc40 0x5601128e4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.829+0000 7fdad5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128d6ad0 0x5601128e4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.829+0000 7fdad5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128d6ad0 0x5601128e4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38684/0 (socket says 172.21.15.70:38684) 2024-04-01T01:40:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.829+0000 7fdad5d9b640 1 -- 172.21.15.70:0/3480429617 learned_addr learned my addr 172.21.15.70:0/3480429617 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.829+0000 7fdadfc38740 1 -- 172.21.15.70:0/3480429617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdab8011140 con 0x5601128d60e0 2024-04-01T01:40:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.829+0000 7fdad6d9d640 1 --2- 172.21.15.70:0/3480429617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128dbc40 0x5601128e4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.829+0000 7fdadfc38740 1 -- 172.21.15.70:0/3480429617 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdab8010a70 con 0x5601128d6ad0 2024-04-01T01:40:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.829+0000 7fdadfc38740 1 -- 172.21.15.70:0/3480429617 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdab800dcb0 con 0x5601128dbc40 2024-04-01T01:40:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.829+0000 7fdad659c640 1 --2- 172.21.15.70:0/3480429617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128d60e0 0x5601128e77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.829+0000 7fdadfc38740 10 client.0 _ll_register_callbacks cb 0x7fdad000aa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.829+0000 7fdad5d9b640 1 -- 172.21.15.70:0/3480429617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128dbc40 msgr2=0x5601128e4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.829+0000 7fdad5d9b640 1 --2- 172.21.15.70:0/3480429617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128dbc40 0x5601128e4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.829+0000 7fdad5d9b640 1 -- 172.21.15.70:0/3480429617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128d60e0 msgr2=0x5601128e77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.829+0000 7fdad5d9b640 1 --2- 172.21.15.70:0/3480429617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128d60e0 0x5601128e77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.829+0000 7fdad5d9b640 1 -- 172.21.15.70:0/3480429617 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601129504a0 con 0x5601128d6ad0 2024-04-01T01:40:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.829+0000 7fdad5d9b640 1 --2- 172.21.15.70:0/3480429617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128d6ad0 0x5601128e4330 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0a3da0 tx=0x7fdacc09b430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e1c9509548751c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.829+0000 7fdabdffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:40:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.830+0000 7fdabdffb640 1 -- 172.21.15.70:0/3480429617 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdacc098080 con 0x5601128d6ad0 2024-04-01T01:40:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.830+0000 7fdabdffb640 1 -- 172.21.15.70:0/3480429617 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fdacc0bb0b0 con 0x5601128d6ad0 2024-04-01T01:40:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.830+0000 7fdadfc38740 1 -- 172.21.15.70:0/3480429617 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56011283ee30 con 0x5601128d6ad0 2024-04-01T01:40:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.830+0000 7fdabdffb640 1 -- 172.21.15.70:0/3480429617 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdacc098220 con 0x5601128d6ad0 2024-04-01T01:40:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.830+0000 7fdadfc38740 10 client.27473 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.830+0000 7fdadfc38740 1 -- 172.21.15.70:0/3480429617 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560112951720 con 0x5601128d6ad0 2024-04-01T01:40:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.830+0000 7fdabdffb640 1 -- 172.21.15.70:0/3480429617 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56011283ee30 con 0x5601128d6ad0 2024-04-01T01:40:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.831+0000 7fdabdffb640 1 -- 172.21.15.70:0/3480429617 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fdacc09a8d0 con 0x5601128d6ad0 2024-04-01T01:40:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.831+0000 7fdadfc38740 10 client.27473 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.831+0000 7fdadfc38740 10 client.27473 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:40:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.831+0000 7fdadfc38740 20 client.27473 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.831+0000 7fdadfc38740 1 --2- 172.21.15.70:0/3480429617 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5601128ec170 0x560112954140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.831+0000 7fdadfc38740 4 client.27473 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:40:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.831+0000 7fdadfc38740 1 -- 172.21.15.70:0/3480429617 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560112839180 con 0x5601128ec170 2024-04-01T01:40:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.831+0000 7fdad659c640 1 --2- 172.21.15.70:0/3480429617 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5601128ec170 0x560112954140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.832+0000 7fdad659c640 1 --2- 172.21.15.70:0/3480429617 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5601128ec170 0x560112954140 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b7328a57d172035 server_cookie=6e578bceb1de65c in_seq=0 out_seq=0 2024-04-01T01:40:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.832+0000 7fdabdffb640 10 client.27473 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:09.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.871+0000 7fdabdffb640 1 -- 172.21.15.70:0/3480429617 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560112839180 con 0x5601128ec170 2024-04-01T01:40:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.871+0000 7fdabdffb640 10 client.27473 handle_command_reply: tid=0 2024-04-01T01:40:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.871+0000 7fdadfc38740 1 client.27473 shutdown 2024-04-01T01:40:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.872+0000 7fdadfc38740 1 -- 172.21.15.70:0/3480429617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128d6ad0 msgr2=0x5601128e4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.872+0000 7fdadfc38740 1 --2- 172.21.15.70:0/3480429617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128d6ad0 0x5601128e4330 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0a3da0 tx=0x7fdacc09b430 comp rx=0 tx=0).stop 2024-04-01T01:40:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.874+0000 7fdadfc38740 1 -- 172.21.15.70:0/3480429617 shutdown_connections 2024-04-01T01:40:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.874+0000 7fdadfc38740 1 --2- 172.21.15.70:0/3480429617 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5601128ec170 0x560112954140 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.874+0000 7fdadfc38740 1 --2- 172.21.15.70:0/3480429617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601128dbc40 0x5601128e4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.874+0000 7fdadfc38740 1 --2- 172.21.15.70:0/3480429617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5601128d6ad0 0x5601128e4330 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.874+0000 7fdadfc38740 1 --2- 172.21.15.70:0/3480429617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5601128d60e0 0x5601128e77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.874+0000 7fdadfc38740 1 -- 172.21.15.70:0/3480429617 >> 172.21.15.70:0/3480429617 conn(0x5601128d1650 msgr2=0x5601128db3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.875+0000 7fdabdffb640 0 client.27473 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.875+0000 7fdadfc38740 1 -- 172.21.15.70:0/3480429617 shutdown_connections 2024-04-01T01:40:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.875+0000 7fdadfc38740 1 -- 172.21.15.70:0/3480429617 wait complete. 2024-04-01T01:40:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.876+0000 7fdadfc38740 20 client.27473 trim_cache size 0 max 16384 2024-04-01T01:40:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:40:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:40:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:40:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:40:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.877+0000 7fdadcf7a640 1 -- 172.21.15.70:0/1811316751 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdac00a2780 msgr2=0x7fdac00a4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.877+0000 7fdadcf7a640 1 --2- 172.21.15.70:0/1811316751 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdac00a2780 0x7fdac00a4c40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fdac800d6a0 tx=0x7fdac800d630 comp rx=0 tx=0).stop 2024-04-01T01:40:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.877+0000 7fdadcf7a640 1 -- 172.21.15.70:0/1811316751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdad8100fe0 msgr2=0x7fdad8195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.877+0000 7fdadcf7a640 1 --2- 172.21.15.70:0/1811316751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdad8100fe0 0x7fdad8195bc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdacc00e9f0 tx=0x7fdacc00edf0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.877+0000 7fdadcf7a640 1 -- 172.21.15.70:0/1811316751 shutdown_connections 2024-04-01T01:40:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.877+0000 7fdadcf7a640 1 --2- 172.21.15.70:0/1811316751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdad81019a0 0x7fdad819cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.877+0000 7fdadcf7a640 1 --2- 172.21.15.70:0/1811316751 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdac00a2780 0x7fdac00a4c40 secure :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fdac800d6a0 tx=0x7fdac800d630 comp rx=0 tx=0).stop 2024-04-01T01:40:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.877+0000 7fdadcf7a640 1 --2- 172.21.15.70:0/1811316751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdad8100fe0 0x7fdad8195bc0 secure :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdacc00e9f0 tx=0x7fdacc00edf0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.877+0000 7fdadcf7a640 1 --2- 172.21.15.70:0/1811316751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdad80ffdf0 0x7fdad8195680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.877+0000 7fdadcf7a640 1 -- 172.21.15.70:0/1811316751 >> 172.21.15.70:0/1811316751 conn(0x7fdad80fb3a0 msgr2=0x7fdad81042e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.877+0000 7fdadcf7a640 1 -- 172.21.15.70:0/1811316751 shutdown_connections 2024-04-01T01:40:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:09.877+0000 7fdadcf7a640 1 -- 172.21.15.70:0/1811316751 wait complete. 2024-04-01T01:40:11.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:10 smithi070 ceph-mon[32179]: pgmap v560: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 13 MiB/s wr, 911 op/s 2024-04-01T01:40:11.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:10 smithi189 ceph-mon[35753]: pgmap v560: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 13 MiB/s wr, 911 op/s 2024-04-01T01:40:11.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:10 smithi081 ceph-mon[35142]: pgmap v560: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 13 MiB/s wr, 911 op/s 2024-04-01T01:40:11.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 325.0 s, 782.6 tps, lat 127.414 ms stddev 166.182 2024-04-01T01:40:13.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:12 smithi189 ceph-mon[35753]: pgmap v561: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 15 MiB/s wr, 1.01k op/s 2024-04-01T01:40:13.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:12 smithi070 ceph-mon[32179]: pgmap v561: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 15 MiB/s wr, 1.01k op/s 2024-04-01T01:40:13.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:12 smithi081 ceph-mon[35142]: pgmap v561: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 15 MiB/s wr, 1.01k op/s 2024-04-01T01:40:15.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:14 smithi189 ceph-mon[35753]: pgmap v562: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 12 MiB/s wr, 875 op/s 2024-04-01T01:40:15.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:14 smithi070 ceph-mon[32179]: pgmap v562: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 12 MiB/s wr, 875 op/s 2024-04-01T01:40:15.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:14 smithi081 ceph-mon[35142]: pgmap v562: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 12 MiB/s wr, 875 op/s 2024-04-01T01:40:16.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 330.0 s, 849.4 tps, lat 117.241 ms stddev 120.147 2024-04-01T01:40:17.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:16 smithi189 ceph-mon[35753]: pgmap v563: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 15 MiB/s wr, 1.05k op/s 2024-04-01T01:40:17.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:16 smithi070 ceph-mon[32179]: pgmap v563: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 15 MiB/s wr, 1.05k op/s 2024-04-01T01:40:17.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:16 smithi081 ceph-mon[35142]: pgmap v563: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 15 MiB/s wr, 1.05k op/s 2024-04-01T01:40:19.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:18 smithi189 ceph-mon[35753]: pgmap v564: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 15 MiB/s wr, 1.04k op/s 2024-04-01T01:40:19.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:18 smithi070 ceph-mon[32179]: pgmap v564: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 15 MiB/s wr, 1.04k op/s 2024-04-01T01:40:19.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:18 smithi081 ceph-mon[35142]: pgmap v564: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 15 MiB/s wr, 1.04k op/s 2024-04-01T01:40:21.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:20 smithi189 ceph-mon[35753]: pgmap v565: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 13 MiB/s wr, 960 op/s 2024-04-01T01:40:21.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:20 smithi070 ceph-mon[32179]: pgmap v565: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 13 MiB/s wr, 960 op/s 2024-04-01T01:40:21.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:20 smithi081 ceph-mon[35142]: pgmap v565: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 13 MiB/s wr, 960 op/s 2024-04-01T01:40:21.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 335.0 s, 392.6 tps, lat 144.761 ms stddev 172.007 2024-04-01T01:40:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.001+0000 7f359653e640 1 -- 172.21.15.70:0/151845284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35900ffa30 msgr2=0x7f35900ffe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.001+0000 7f359653e640 1 --2- 172.21.15.70:0/151845284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35900ffa30 0x7f35900ffe50 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3580009930 tx=0x7f358002bc30 comp rx=0 tx=0).stop 2024-04-01T01:40:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.001+0000 7f359653e640 1 -- 172.21.15.70:0/151845284 shutdown_connections 2024-04-01T01:40:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.001+0000 7f359653e640 1 --2- 172.21.15.70:0/151845284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35901033c0 0x7f3590109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.001+0000 7f359653e640 1 --2- 172.21.15.70:0/151845284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3590102a00 0x7f3590102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.001+0000 7f359653e640 1 --2- 172.21.15.70:0/151845284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35900ffa30 0x7f35900ffe50 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.001+0000 7f359653e640 1 -- 172.21.15.70:0/151845284 >> 172.21.15.70:0/151845284 conn(0x7f35900fb3a0 msgr2=0x7f35900fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.001+0000 7f359653e640 1 -- 172.21.15.70:0/151845284 shutdown_connections 2024-04-01T01:40:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.001+0000 7f359653e640 1 -- 172.21.15.70:0/151845284 wait complete. 2024-04-01T01:40:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.002+0000 7f359653e640 1 Processor -- start 2024-04-01T01:40:23.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.002+0000 7f359653e640 1 -- start start 2024-04-01T01:40:23.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.002+0000 7f359653e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35900ffa30 0x7f3590199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.002+0000 7f359653e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3590102a00 0x7f359019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.002+0000 7f359653e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35901033c0 0x7f35901a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.002+0000 7f359653e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3590102730 con 0x7f35901033c0 2024-04-01T01:40:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.002+0000 7f359653e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f35901025b0 con 0x7f35900ffa30 2024-04-01T01:40:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.002+0000 7f359653e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f359010f780 con 0x7f3590102a00 2024-04-01T01:40:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.003+0000 7f358f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3590102a00 0x7f359019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.003+0000 7f358f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3590102a00 0x7f359019a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:50368/0 (socket says 172.21.15.70:50368) 2024-04-01T01:40:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.003+0000 7f358f7fe640 1 -- 172.21.15.70:0/2181135621 learned_addr learned my addr 172.21.15.70:0/2181135621 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.003+0000 7f3594adb640 1 --2- 172.21.15.70:0/2181135621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35901033c0 0x7f35901a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.003+0000 7f358ffff640 1 --2- 172.21.15.70:0/2181135621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35900ffa30 0x7f3590199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.003+0000 7f3594adb640 1 -- 172.21.15.70:0/2181135621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3590102a00 msgr2=0x7f359019a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.003+0000 7f3594adb640 1 --2- 172.21.15.70:0/2181135621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3590102a00 0x7f359019a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.003+0000 7f3594adb640 1 -- 172.21.15.70:0/2181135621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35900ffa30 msgr2=0x7f3590199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:23.004 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.003+0000 7f3594adb640 1 --2- 172.21.15.70:0/2181135621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35900ffa30 0x7f3590199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.003+0000 7f3594adb640 1 -- 172.21.15.70:0/2181135621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35901a1790 con 0x7f35901033c0 2024-04-01T01:40:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.003+0000 7f3594adb640 1 --2- 172.21.15.70:0/2181135621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35901033c0 0x7f35901a1120 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3588009fd0 tx=0x7f358800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.004+0000 7f358d7fa640 1 -- 172.21.15.70:0/2181135621 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358800ce20 con 0x7f35901033c0 2024-04-01T01:40:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.004+0000 7f359653e640 1 -- 172.21.15.70:0/2181135621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35901a1a80 con 0x7f35901033c0 2024-04-01T01:40:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.004+0000 7f359653e640 1 -- 172.21.15.70:0/2181135621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35901a2020 con 0x7f35901033c0 2024-04-01T01:40:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.004+0000 7f358d7fa640 1 -- 172.21.15.70:0/2181135621 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3588005020 con 0x7f35901033c0 2024-04-01T01:40:23.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.004+0000 7f358d7fa640 1 -- 172.21.15.70:0/2181135621 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3588010690 con 0x7f35901033c0 2024-04-01T01:40:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.008+0000 7f358d7fa640 1 -- 172.21.15.70:0/2181135621 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f35880051d0 con 0x7f35901033c0 2024-04-01T01:40:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.008+0000 7f35991fe740 1 Processor -- start 2024-04-01T01:40:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.008+0000 7f35991fe740 1 -- start start 2024-04-01T01:40:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.008+0000 7f35991fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee270e0 0x561cfee27500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.008+0000 7f35991fe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee27ad0 0x561cfee2c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.008+0000 7f35991fe740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee2cc40 0x561cfee2f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.008+0000 7f35991fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561cfed828f0 con 0x561cfee270e0 2024-04-01T01:40:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.008+0000 7f35991fe740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561cfed2b8d0 con 0x561cfee2cc40 2024-04-01T01:40:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.008+0000 7f35991fe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561cfed7e990 con 0x561cfee27ad0 2024-04-01T01:40:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.008+0000 7f358f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee270e0 0x561cfee27500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.008+0000 7f358f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee270e0 0x561cfee27500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41338/0 (socket says 172.21.15.70:41338) 2024-04-01T01:40:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.008+0000 7f358f7fe640 1 -- 172.21.15.70:0/1437322341 learned_addr learned my addr 172.21.15.70:0/1437322341 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.009+0000 7f358ffff640 1 --2- 172.21.15.70:0/1437322341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee27ad0 0x561cfee2c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.009+0000 7f358f7fe640 1 --2- 172.21.15.70:0/1437322341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee2cc40 0x561cfee2f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.009+0000 7f358ffff640 1 -- 172.21.15.70:0/1437322341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee2cc40 msgr2=0x561cfee2f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.009+0000 7f358ffff640 1 --2- 172.21.15.70:0/1437322341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee2cc40 0x561cfee2f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.009+0000 7f358ffff640 1 -- 172.21.15.70:0/1437322341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee270e0 msgr2=0x561cfee27500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.009+0000 7f358ffff640 1 --2- 172.21.15.70:0/1437322341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee270e0 0x561cfee27500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.009+0000 7f358ffff640 1 -- 172.21.15.70:0/1437322341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cfea97210 con 0x561cfee27ad0 2024-04-01T01:40:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.009+0000 7f358f7fe640 1 --2- 172.21.15.70:0/1437322341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee270e0 0x561cfee27500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:40:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.009+0000 7f358ffff640 1 --2- 172.21.15.70:0/1437322341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee27ad0 0x561cfee2c5d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f358002c170 tx=0x7f3580005900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4bb4628730bb75f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.010+0000 7f35767fc640 1 -- 172.21.15.70:0/1437322341 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3580005220 con 0x561cfee27ad0 2024-04-01T01:40:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.010+0000 7f358d7fa640 1 --2- 172.21.15.70:0/2181135621 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f35780a2640 0x7f35780a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.010+0000 7f358f7fe640 1 --2- 172.21.15.70:0/2181135621 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f35780a2640 0x7f35780a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.010+0000 7f35767fc640 1 -- 172.21.15.70:0/1437322341 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35800053c0 con 0x561cfee27ad0 2024-04-01T01:40:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.010+0000 7f35767fc640 1 -- 172.21.15.70:0/1437322341 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35800043c0 con 0x561cfee27ad0 2024-04-01T01:40:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.010+0000 7f358d7fa640 1 -- 172.21.15.70:0/2181135621 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f35880cf240 con 0x7f35901033c0 2024-04-01T01:40:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.011+0000 7f35991fe740 1 -- 172.21.15.70:0/1437322341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee27ad0 msgr2=0x561cfee2c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.011+0000 7f358f7fe640 1 --2- 172.21.15.70:0/2181135621 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f35780a2640 0x7f35780a4b00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x561cfee28330 tx=0x7f3584015040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.011+0000 7f35991fe740 1 --2- 172.21.15.70:0/1437322341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee27ad0 0x561cfee2c5d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f358002c170 tx=0x7f3580005900 comp rx=0 tx=0).stop 2024-04-01T01:40:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.011+0000 7f35991fe740 1 -- 172.21.15.70:0/1437322341 shutdown_connections 2024-04-01T01:40:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.011+0000 7f35991fe740 1 --2- 172.21.15.70:0/1437322341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee2cc40 0x561cfee2f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.011+0000 7f35991fe740 1 --2- 172.21.15.70:0/1437322341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee27ad0 0x561cfee2c5d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.011+0000 7f35991fe740 1 --2- 172.21.15.70:0/1437322341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee270e0 0x561cfee27500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.011+0000 7f35991fe740 1 -- 172.21.15.70:0/1437322341 >> 172.21.15.70:0/1437322341 conn(0x561cfee22650 msgr2=0x561cfee22a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.011+0000 7f35991fe740 1 -- 172.21.15.70:0/1437322341 shutdown_connections 2024-04-01T01:40:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.012+0000 7f35991fe740 1 -- 172.21.15.70:0/1437322341 wait complete. 2024-04-01T01:40:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.012+0000 7f35991fe740 1 -- start start 2024-04-01T01:40:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.012+0000 7f35991fe740 1 Processor -- start 2024-04-01T01:40:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f35991fe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee270e0 0x561cfee31410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f35991fe740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee27ad0 0x561cfee31950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f35991fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee2cc40 0x561cfee38990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f35991fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561cfee2fb90 con 0x561cfee2cc40 2024-04-01T01:40:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f35991fe740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561cfed20f80 con 0x561cfee27ad0 2024-04-01T01:40:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f35991fe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561cfee2fd10 con 0x561cfee270e0 2024-04-01T01:40:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f35991fe740 10 client.0 _ll_register_callbacks cb 0x561cfed79290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f358ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee270e0 0x561cfee31410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f358ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee270e0 0x561cfee31410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:50394/0 (socket says 172.21.15.70:50394) 2024-04-01T01:40:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f358ffff640 1 -- 172.21.15.70:0/1345579035 learned_addr learned my addr 172.21.15.70:0/1345579035 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f3594adb640 1 --2- 172.21.15.70:0/1345579035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee2cc40 0x561cfee38990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f358f7fe640 1 --2- 172.21.15.70:0/1345579035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee27ad0 0x561cfee31950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f3594adb640 1 -- 172.21.15.70:0/1345579035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee270e0 msgr2=0x561cfee31410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f3594adb640 1 --2- 172.21.15.70:0/1345579035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee270e0 0x561cfee31410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f3594adb640 1 -- 172.21.15.70:0/1345579035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee27ad0 msgr2=0x561cfee31950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f3594adb640 1 --2- 172.21.15.70:0/1345579035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee27ad0 0x561cfee31950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.013+0000 7f3594adb640 1 -- 172.21.15.70:0/1345579035 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cfeb63620 con 0x561cfee2cc40 2024-04-01T01:40:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.014+0000 7f3594adb640 1 --2- 172.21.15.70:0/1345579035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee2cc40 0x561cfee38990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f358809b860 tx=0x7f358809c110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e5d4ddc05d17cfc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.014+0000 7f356f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:40:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.014+0000 7f356f7fe640 1 -- 172.21.15.70:0/1345579035 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35880b4060 con 0x561cfee2cc40 2024-04-01T01:40:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.014+0000 7f356f7fe640 1 -- 172.21.15.70:0/1345579035 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f35880860c0 con 0x561cfee2cc40 2024-04-01T01:40:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.014+0000 7f35991fe740 1 -- 172.21.15.70:0/1345579035 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35800053c0 con 0x561cfee2cc40 2024-04-01T01:40:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.014+0000 7f356f7fe640 1 -- 172.21.15.70:0/1345579035 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3588084e60 con 0x561cfee2cc40 2024-04-01T01:40:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.014+0000 7f35991fe740 10 client.17571 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.014+0000 7f35991fe740 1 -- 172.21.15.70:0/1345579035 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561cfee3ba90 con 0x561cfee2cc40 2024-04-01T01:40:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.014+0000 7f356f7fe640 1 -- 172.21.15.70:0/1345579035 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35800053c0 con 0x561cfee2cc40 2024-04-01T01:40:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.015+0000 7f356f7fe640 1 -- 172.21.15.70:0/1345579035 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f35880afb90 con 0x561cfee2cc40 2024-04-01T01:40:23.017 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.015+0000 7f35991fe740 10 client.17571 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.015+0000 7f35991fe740 10 client.17571 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:40:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.015+0000 7f35991fe740 20 client.17571 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.015+0000 7f35991fe740 1 --2- 172.21.15.70:0/1345579035 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561cfee3d5e0 0x561cfee3fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.015+0000 7f35991fe740 4 client.17571 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:40:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.015+0000 7f35991fe740 1 -- 172.21.15.70:0/1345579035 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cfed96120 con 0x561cfee3d5e0 2024-04-01T01:40:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.015+0000 7f358ffff640 1 --2- 172.21.15.70:0/1345579035 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561cfee3d5e0 0x561cfee3fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.016+0000 7f358ffff640 1 --2- 172.21.15.70:0/1345579035 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561cfee3d5e0 0x561cfee3fa40 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=932d8170d466a02d server_cookie=2a4f2a913dab2f9c in_seq=0 out_seq=0 2024-04-01T01:40:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.016+0000 7f356f7fe640 10 client.17571 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.017+0000 7f356f7fe640 1 -- 172.21.15.70:0/1345579035 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561cfed96120 con 0x561cfee3d5e0 2024-04-01T01:40:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.017+0000 7f356f7fe640 10 client.17571 handle_command_reply: tid=0 2024-04-01T01:40:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.017+0000 7f35991fe740 1 client.17571 shutdown 2024-04-01T01:40:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.017+0000 7f35991fe740 1 -- 172.21.15.70:0/1345579035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee2cc40 msgr2=0x561cfee38990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.017+0000 7f35991fe740 1 --2- 172.21.15.70:0/1345579035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee2cc40 0x561cfee38990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f358809b860 tx=0x7f358809c110 comp rx=0 tx=0).stop 2024-04-01T01:40:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.017+0000 7f35991fe740 1 -- 172.21.15.70:0/1345579035 shutdown_connections 2024-04-01T01:40:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.017+0000 7f35991fe740 1 --2- 172.21.15.70:0/1345579035 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561cfee3d5e0 0x561cfee3fa40 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.017+0000 7f35991fe740 1 --2- 172.21.15.70:0/1345579035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee2cc40 0x561cfee38990 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.017+0000 7f35991fe740 1 --2- 172.21.15.70:0/1345579035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee27ad0 0x561cfee31950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.017+0000 7f35991fe740 1 --2- 172.21.15.70:0/1345579035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee270e0 0x561cfee31410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.017+0000 7f35991fe740 1 -- 172.21.15.70:0/1345579035 >> 172.21.15.70:0/1345579035 conn(0x561cfee22650 msgr2=0x561cfee18f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.017+0000 7f356f7fe640 0 client.17571 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.018+0000 7f35991fe740 1 -- 172.21.15.70:0/1345579035 shutdown_connections 2024-04-01T01:40:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.018+0000 7f35991fe740 1 -- 172.21.15.70:0/1345579035 wait complete. 2024-04-01T01:40:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.018+0000 7f35991fe740 20 client.17571 trim_cache size 0 max 16384 2024-04-01T01:40:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.040+0000 7f35991fe740 1 Processor -- start 2024-04-01T01:40:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.040+0000 7f35991fe740 1 -- start start 2024-04-01T01:40:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.040+0000 7f35991fe740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee270e0 0x561cfee45350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.040+0000 7f35991fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee27ad0 0x561cfee38950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.041+0000 7f35991fe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee2cc40 0x561cfee31410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.041+0000 7f35991fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35901a81b0 con 0x561cfee27ad0 2024-04-01T01:40:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.041+0000 7f358ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee270e0 0x561cfee45350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.041+0000 7f358ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee270e0 0x561cfee45350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47354/0 (socket says 172.21.15.70:47354) 2024-04-01T01:40:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.041+0000 7f358ffff640 1 -- 172.21.15.70:0/1506636700 learned_addr learned my addr 172.21.15.70:0/1506636700 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.041+0000 7f35991fe740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f35901a9cf0 con 0x561cfee270e0 2024-04-01T01:40:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.041+0000 7f35991fe740 1 -- 172.21.15.70:0/1506636700 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f35901a8880 con 0x561cfee2cc40 2024-04-01T01:40:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.041+0000 7f358ffff640 1 -- 172.21.15.70:0/1506636700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee2cc40 msgr2=0x561cfee31410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.041+0000 7f358ffff640 1 --2- 172.21.15.70:0/1506636700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee2cc40 0x561cfee31410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.041+0000 7f358ffff640 1 -- 172.21.15.70:0/1506636700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee27ad0 msgr2=0x561cfee38950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.041+0000 7f358ffff640 1 --2- 172.21.15.70:0/1506636700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee27ad0 0x561cfee38950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.041+0000 7f358ffff640 1 -- 172.21.15.70:0/1506636700 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cfee400c0 con 0x561cfee270e0 2024-04-01T01:40:23.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.042+0000 7f358ffff640 1 --2- 172.21.15.70:0/1506636700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee270e0 0x561cfee45350 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3580036040 tx=0x7f358002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22bf2e065fd106ac server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:23.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.044+0000 7f3575ffb640 1 -- 172.21.15.70:0/1506636700 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3580005d50 con 0x561cfee270e0 2024-04-01T01:40:23.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.044+0000 7f3575ffb640 1 -- 172.21.15.70:0/1506636700 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f358002cac0 con 0x561cfee270e0 2024-04-01T01:40:23.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.044+0000 7f3575ffb640 1 -- 172.21.15.70:0/1506636700 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3580034440 con 0x561cfee270e0 2024-04-01T01:40:23.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.045+0000 7f35991fe740 1 -- 172.21.15.70:0/1506636700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee270e0 msgr2=0x561cfee45350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:23.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.045+0000 7f35991fe740 1 --2- 172.21.15.70:0/1506636700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee270e0 0x561cfee45350 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3580036040 tx=0x7f358002bc30 comp rx=0 tx=0).stop 2024-04-01T01:40:23.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.045+0000 7f35991fe740 1 -- 172.21.15.70:0/1506636700 shutdown_connections 2024-04-01T01:40:23.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.045+0000 7f35991fe740 1 --2- 172.21.15.70:0/1506636700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee2cc40 0x561cfee31410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.045+0000 7f35991fe740 1 --2- 172.21.15.70:0/1506636700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee27ad0 0x561cfee38950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.045+0000 7f35991fe740 1 --2- 172.21.15.70:0/1506636700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee270e0 0x561cfee45350 secure :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3580036040 tx=0x7f358002bc30 comp rx=0 tx=0).stop 2024-04-01T01:40:23.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.045+0000 7f35991fe740 1 -- 172.21.15.70:0/1506636700 >> 172.21.15.70:0/1506636700 conn(0x561cfee22650 msgr2=0x561cfee22a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:23.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.046+0000 7f35991fe740 1 -- 172.21.15.70:0/1506636700 shutdown_connections 2024-04-01T01:40:23.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.046+0000 7f35991fe740 1 -- 172.21.15.70:0/1506636700 wait complete. 2024-04-01T01:40:23.046 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.047+0000 7f35991fe740 1 -- start start 2024-04-01T01:40:23.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.047+0000 7f35991fe740 1 Processor -- start 2024-04-01T01:40:23.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.048+0000 7f35991fe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee27ad0 0x561cfee35bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.048+0000 7f35991fe740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee2cc40 0x561cfee36110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.048+0000 7f35991fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee9f960 0x561cfeea1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.048+0000 7f358f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee2cc40 0x561cfee36110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:23.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.048+0000 7f358f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee2cc40 0x561cfee36110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47370/0 (socket says 172.21.15.70:47370) 2024-04-01T01:40:23.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.048+0000 7f358f7fe640 1 -- 172.21.15.70:0/1241907833 learned_addr learned my addr 172.21.15.70:0/1241907833 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:23.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.048+0000 7f35991fe740 1 -- 172.21.15.70:0/1241907833 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35901a9620 con 0x561cfee9f960 2024-04-01T01:40:23.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.048+0000 7f35991fe740 1 -- 172.21.15.70:0/1241907833 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f35901a8f50 con 0x561cfee2cc40 2024-04-01T01:40:23.048 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.048+0000 7f35991fe740 1 -- 172.21.15.70:0/1241907833 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f35901a6190 con 0x561cfee27ad0 2024-04-01T01:40:23.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.049+0000 7f35991fe740 10 client.0 _ll_register_callbacks cb 0x7f3588097b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.049+0000 7f358ffff640 1 --2- 172.21.15.70:0/1241907833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee27ad0 0x561cfee35bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.050+0000 7f358f7fe640 1 -- 172.21.15.70:0/1241907833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee27ad0 msgr2=0x561cfee35bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.050+0000 7f358f7fe640 1 --2- 172.21.15.70:0/1241907833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee27ad0 0x561cfee35bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.050+0000 7f358f7fe640 1 -- 172.21.15.70:0/1241907833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee9f960 msgr2=0x561cfeea1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.050+0000 7f358f7fe640 1 --2- 172.21.15.70:0/1241907833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee9f960 0x561cfeea1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.050+0000 7f358f7fe640 1 -- 172.21.15.70:0/1241907833 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cfeea2290 con 0x561cfee2cc40 2024-04-01T01:40:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.050+0000 7f358f7fe640 1 --2- 172.21.15.70:0/1241907833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee2cc40 0x561cfee36110 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f358400fd80 tx=0x7f3584009cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61d7b93d2c276f3d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.050+0000 7f356f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:40:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.050+0000 7f356f7fe640 1 -- 172.21.15.70:0/1241907833 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3584010930 con 0x561cfee2cc40 2024-04-01T01:40:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.050+0000 7f35991fe740 1 -- 172.21.15.70:0/1241907833 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f358002cac0 con 0x561cfee2cc40 2024-04-01T01:40:23.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.050+0000 7f356f7fe640 1 -- 172.21.15.70:0/1241907833 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3584010ad0 con 0x561cfee2cc40 2024-04-01T01:40:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.051+0000 7f356f7fe640 1 -- 172.21.15.70:0/1241907833 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3584036050 con 0x561cfee2cc40 2024-04-01T01:40:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.051+0000 7f35991fe740 10 client.27491 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:23.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.051+0000 7f35991fe740 1 -- 172.21.15.70:0/1241907833 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561cfeea35c0 con 0x561cfee2cc40 2024-04-01T01:40:23.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.052+0000 7f356f7fe640 1 -- 172.21.15.70:0/1241907833 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f358002cac0 con 0x561cfee2cc40 2024-04-01T01:40:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.053+0000 7f356f7fe640 1 -- 172.21.15.70:0/1241907833 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f358402a030 con 0x561cfee2cc40 2024-04-01T01:40:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.054+0000 7f35991fe740 10 client.27491 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.054+0000 7f35991fe740 10 client.27491 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:40:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.054+0000 7f35991fe740 20 client.27491 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.054+0000 7f35991fe740 1 --2- 172.21.15.70:0/1241907833 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561cfeea3fd0 0x561cfeea63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.054+0000 7f35991fe740 4 client.27491 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:40:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.054+0000 7f35991fe740 1 -- 172.21.15.70:0/1241907833 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561cfed8fe30 con 0x561cfeea3fd0 2024-04-01T01:40:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.055+0000 7f358ffff640 1 --2- 172.21.15.70:0/1241907833 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561cfeea3fd0 0x561cfeea63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:23.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.056+0000 7f358ffff640 1 --2- 172.21.15.70:0/1241907833 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561cfeea3fd0 0x561cfeea63f0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0278996e6d970a7 server_cookie=d81e79b191186ffb in_seq=0 out_seq=0 2024-04-01T01:40:23.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.056+0000 7f356f7fe640 10 client.27491 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:23.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:22 smithi189 ceph-mon[35753]: pgmap v566: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 15 MiB/s wr, 1.08k op/s 2024-04-01T01:40:23.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:22 smithi070 ceph-mon[32179]: pgmap v566: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 15 MiB/s wr, 1.08k op/s 2024-04-01T01:40:23.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.307+0000 7f356f7fe640 1 -- 172.21.15.70:0/1241907833 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561cfed8fe30 con 0x561cfeea3fd0 2024-04-01T01:40:23.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.307+0000 7f356f7fe640 10 client.27491 handle_command_reply: tid=0 2024-04-01T01:40:23.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.307+0000 7f35991fe740 1 client.27491 shutdown 2024-04-01T01:40:23.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.307+0000 7f35991fe740 1 -- 172.21.15.70:0/1241907833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee2cc40 msgr2=0x561cfee36110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:23.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.307+0000 7f35991fe740 1 --2- 172.21.15.70:0/1241907833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee2cc40 0x561cfee36110 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f358400fd80 tx=0x7f3584009cf0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.307+0000 7f35991fe740 1 -- 172.21.15.70:0/1241907833 shutdown_connections 2024-04-01T01:40:23.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.307+0000 7f35991fe740 1 --2- 172.21.15.70:0/1241907833 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561cfeea3fd0 0x561cfeea63f0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.307+0000 7f35991fe740 1 --2- 172.21.15.70:0/1241907833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561cfee9f960 0x561cfeea1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.307+0000 7f35991fe740 1 --2- 172.21.15.70:0/1241907833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561cfee2cc40 0x561cfee36110 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.307+0000 7f35991fe740 1 --2- 172.21.15.70:0/1241907833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561cfee27ad0 0x561cfee35bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.307+0000 7f35991fe740 1 -- 172.21.15.70:0/1241907833 >> 172.21.15.70:0/1241907833 conn(0x561cfee22650 msgr2=0x561cfee31410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.307+0000 7f356f7fe640 0 client.27491 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.307+0000 7f35991fe740 1 -- 172.21.15.70:0/1241907833 shutdown_connections 2024-04-01T01:40:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.307+0000 7f35991fe740 1 -- 172.21.15.70:0/1241907833 wait complete. 2024-04-01T01:40:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.308+0000 7f35991fe740 20 client.27491 trim_cache size 0 max 16384 2024-04-01T01:40:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:40:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:40:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:40:23.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:40:23.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.308+0000 7f359653e640 1 -- 172.21.15.70:0/2181135621 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f35780a2640 msgr2=0x7f35780a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:23.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.308+0000 7f359653e640 1 --2- 172.21.15.70:0/2181135621 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f35780a2640 0x7f35780a4b00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x561cfee28330 tx=0x7f3584015040 comp rx=0 tx=0).stop 2024-04-01T01:40:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.308+0000 7f359653e640 1 -- 172.21.15.70:0/2181135621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35901033c0 msgr2=0x7f35901a1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.308+0000 7f359653e640 1 --2- 172.21.15.70:0/2181135621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35901033c0 0x7f35901a1120 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3588009fd0 tx=0x7f358800ef10 comp rx=0 tx=0).stop 2024-04-01T01:40:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.308+0000 7f359653e640 1 -- 172.21.15.70:0/2181135621 shutdown_connections 2024-04-01T01:40:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.308+0000 7f359653e640 1 --2- 172.21.15.70:0/2181135621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35901033c0 0x7f35901a1120 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.308+0000 7f359653e640 1 --2- 172.21.15.70:0/2181135621 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f35780a2640 0x7f35780a4b00 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.308+0000 7f359653e640 1 --2- 172.21.15.70:0/2181135621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3590102a00 0x7f359019a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.308+0000 7f359653e640 1 --2- 172.21.15.70:0/2181135621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35900ffa30 0x7f3590199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.308+0000 7f359653e640 1 -- 172.21.15.70:0/2181135621 >> 172.21.15.70:0/2181135621 conn(0x7f35900fb3a0 msgr2=0x7f3590105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.309+0000 7f359653e640 1 -- 172.21.15.70:0/2181135621 shutdown_connections 2024-04-01T01:40:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:23.309+0000 7f359653e640 1 -- 172.21.15.70:0/2181135621 wait complete. 2024-04-01T01:40:23.327 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:22 smithi081 ceph-mon[35142]: pgmap v566: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 15 MiB/s wr, 1.08k op/s 2024-04-01T01:40:25.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:24 smithi189 ceph-mon[35753]: pgmap v567: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 13 MiB/s wr, 986 op/s 2024-04-01T01:40:25.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:24 smithi070 ceph-mon[32179]: pgmap v567: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 13 MiB/s wr, 986 op/s 2024-04-01T01:40:25.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:24 smithi081 ceph-mon[35142]: pgmap v567: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 13 MiB/s wr, 986 op/s 2024-04-01T01:40:25.866 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:40:25.866 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:40:26.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.073+0000 7f742858a640 1 -- 172.21.15.70:0/3913595008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f742010c160 msgr2=0x7f742010c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.073+0000 7f742858a640 1 --2- 172.21.15.70:0/3913595008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f742010c160 0x7f742010c580 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f741c009900 tx=0x7f741c02bc40 comp rx=0 tx=0).stop 2024-04-01T01:40:26.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.074+0000 7f742858a640 1 -- 172.21.15.70:0/3913595008 shutdown_connections 2024-04-01T01:40:26.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.074+0000 7f742858a640 1 --2- 172.21.15.70:0/3913595008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f742010dd10 0x7f74201145a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.074+0000 7f742858a640 1 --2- 172.21.15.70:0/3913595008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742010d350 0x7f742010d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.074+0000 7f742858a640 1 --2- 172.21.15.70:0/3913595008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f742010c160 0x7f742010c580 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.074+0000 7f742858a640 1 -- 172.21.15.70:0/3913595008 >> 172.21.15.70:0/3913595008 conn(0x7f742006c110 msgr2=0x7f742006c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:26.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.075+0000 7f742858a640 1 -- 172.21.15.70:0/3913595008 shutdown_connections 2024-04-01T01:40:26.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.076+0000 7f742858a640 1 -- 172.21.15.70:0/3913595008 wait complete. 2024-04-01T01:40:26.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.077+0000 7f742858a640 1 Processor -- start 2024-04-01T01:40:26.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.077+0000 7f742858a640 1 -- start start 2024-04-01T01:40:26.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.077+0000 7f742858a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742010c160 0x7f74201a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.077+0000 7f742858a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f742010d350 0x7f74201a2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.077+0000 7f742858a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f742010dd10 0x7f74201a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.077+0000 7f742858a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7420117cc0 con 0x7f742010d350 2024-04-01T01:40:26.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.077+0000 7f742858a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7420117b40 con 0x7f742010dd10 2024-04-01T01:40:26.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.077+0000 7f742858a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7420117e40 con 0x7f742010c160 2024-04-01T01:40:26.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.077+0000 7f7426b27640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f742010dd10 0x7f74201a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.077+0000 7f7426b27640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f742010dd10 0x7f74201a97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47382/0 (socket says 172.21.15.70:47382) 2024-04-01T01:40:26.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.077+0000 7f7426b27640 1 -- 172.21.15.70:0/4114301788 learned_addr learned my addr 172.21.15.70:0/4114301788 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:26.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.077+0000 7f7426326640 1 --2- 172.21.15.70:0/4114301788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742010c160 0x7f74201a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.078+0000 7f7425b25640 1 --2- 172.21.15.70:0/4114301788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f742010d350 0x7f74201a2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.078+0000 7f7426b27640 1 -- 172.21.15.70:0/4114301788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742010c160 msgr2=0x7f74201a2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.078+0000 7f7426b27640 1 --2- 172.21.15.70:0/4114301788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742010c160 0x7f74201a2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.078+0000 7f7426b27640 1 -- 172.21.15.70:0/4114301788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f742010d350 msgr2=0x7f74201a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.078+0000 7f7426b27640 1 --2- 172.21.15.70:0/4114301788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f742010d350 0x7f74201a2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.078+0000 7f7426b27640 1 -- 172.21.15.70:0/4114301788 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74201a9e40 con 0x7f742010dd10 2024-04-01T01:40:26.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.078+0000 7f7426b27640 1 --2- 172.21.15.70:0/4114301788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f742010dd10 0x7f74201a97d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f741800eb10 tx=0x7f741800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:26.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.078+0000 7f74177fe640 1 -- 172.21.15.70:0/4114301788 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7418004b20 con 0x7f742010dd10 2024-04-01T01:40:26.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.079+0000 7f74177fe640 1 -- 172.21.15.70:0/4114301788 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7418004cc0 con 0x7f742010dd10 2024-04-01T01:40:26.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.079+0000 7f74177fe640 1 -- 172.21.15.70:0/4114301788 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7418010690 con 0x7f742010dd10 2024-04-01T01:40:26.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.079+0000 7f742858a640 1 -- 172.21.15.70:0/4114301788 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74201aa130 con 0x7f742010dd10 2024-04-01T01:40:26.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.079+0000 7f742858a640 1 -- 172.21.15.70:0/4114301788 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74201aa5d0 con 0x7f742010dd10 2024-04-01T01:40:26.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.082+0000 7f74177fe640 1 -- 172.21.15.70:0/4114301788 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f7418005020 con 0x7f742010dd10 2024-04-01T01:40:26.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.083+0000 7f742b16b740 1 Processor -- start 2024-04-01T01:40:26.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.083+0000 7f74177fe640 1 --2- 172.21.15.70:0/4114301788 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f740c0a25a0 0x7f740c0a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.084+0000 7f74177fe640 1 -- 172.21.15.70:0/4114301788 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f74180cf110 con 0x7f742010dd10 2024-04-01T01:40:26.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.085+0000 7f742b16b740 1 -- start start 2024-04-01T01:40:26.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.085+0000 7f742b16b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbba6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.085+0000 7f742b16b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 0x5578bbbabfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.085+0000 7f742b16b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 0x5578bbbaeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.085+0000 7f742b16b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5578bbb03180 con 0x5578bbbac620 2024-04-01T01:40:26.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.085+0000 7f742b16b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5578bbaabb70 con 0x5578bbba74b0 2024-04-01T01:40:26.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.085+0000 7f742b16b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5578bbaff220 con 0x5578bbba6ac0 2024-04-01T01:40:26.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.085+0000 7f7426326640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbba6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.085+0000 7f7426b27640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 0x5578bbbaeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.085+0000 7f7425b25640 1 --2- 172.21.15.70:0/4114301788 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f740c0a25a0 0x7f740c0a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.085+0000 7f7426b27640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 0x5578bbbaeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41388/0 (socket says 172.21.15.70:41388) 2024-04-01T01:40:26.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.085+0000 7f7426b27640 1 -- 172.21.15.70:0/3522763172 learned_addr learned my addr 172.21.15.70:0/3522763172 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:26.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.085+0000 7f7425b25640 1 --2- 172.21.15.70:0/3522763172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 0x5578bbbabfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.086+0000 7f7425b25640 1 -- 172.21.15.70:0/3522763172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 msgr2=0x5578bbba6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:26.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.086+0000 7f7425b25640 1 --2- 172.21.15.70:0/3522763172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbba6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.086+0000 7f7425b25640 1 -- 172.21.15.70:0/3522763172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 msgr2=0x5578bbbaeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:26.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.086+0000 7f7425b25640 1 --2- 172.21.15.70:0/3522763172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 0x5578bbbaeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.086+0000 7f7425b25640 1 -- 172.21.15.70:0/3522763172 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578bbabf7c0 con 0x5578bbba74b0 2024-04-01T01:40:26.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.086+0000 7f7426326640 1 --2- 172.21.15.70:0/3522763172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbba6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:40:26.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.086+0000 7f7426b27640 1 --2- 172.21.15.70:0/3522763172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 0x5578bbbaeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:40:26.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.086+0000 7f7425b25640 1 --2- 172.21.15.70:0/3522763172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 0x5578bbbabfb0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f74100058c0 tx=0x7f74100117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=339c94570b2c1e63 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:26.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.086+0000 7f7425b25640 1 --2- 172.21.15.70:0/4114301788 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f740c0a25a0 0x7f740c0a4a60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f74201a3730 tx=0x7f7410016000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:26.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.087+0000 7f7414bfe640 1 -- 172.21.15.70:0/3522763172 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741000c4c0 con 0x5578bbba74b0 2024-04-01T01:40:26.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.087+0000 7f7414bfe640 1 -- 172.21.15.70:0/3522763172 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7410014cd0 con 0x5578bbba74b0 2024-04-01T01:40:26.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.087+0000 7f7414bfe640 1 -- 172.21.15.70:0/3522763172 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74100035f0 con 0x5578bbba74b0 2024-04-01T01:40:26.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.088+0000 7f742b16b740 1 -- 172.21.15.70:0/3522763172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 msgr2=0x5578bbbabfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.088+0000 7f742b16b740 1 --2- 172.21.15.70:0/3522763172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 0x5578bbbabfb0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f74100058c0 tx=0x7f74100117b0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.089+0000 7f742b16b740 1 -- 172.21.15.70:0/3522763172 shutdown_connections 2024-04-01T01:40:26.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.089+0000 7f742b16b740 1 --2- 172.21.15.70:0/3522763172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 0x5578bbbaeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.089+0000 7f742b16b740 1 --2- 172.21.15.70:0/3522763172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 0x5578bbbabfb0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.089+0000 7f742b16b740 1 --2- 172.21.15.70:0/3522763172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbba6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.089+0000 7f742b16b740 1 -- 172.21.15.70:0/3522763172 >> 172.21.15.70:0/3522763172 conn(0x5578bbba2030 msgr2=0x5578bbba2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:26.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.090+0000 7f742b16b740 1 -- 172.21.15.70:0/3522763172 shutdown_connections 2024-04-01T01:40:26.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.090+0000 7f742b16b740 1 -- 172.21.15.70:0/3522763172 wait complete. 2024-04-01T01:40:26.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.090+0000 7f742b16b740 1 -- start start 2024-04-01T01:40:26.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.091+0000 7f742b16b740 1 Processor -- start 2024-04-01T01:40:26.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.091+0000 7f742b16b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbb9b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.091+0000 7f742b16b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 0x5578bbb9b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.092+0000 7f742b16b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 0x5578bbba0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.092+0000 7f742b16b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5578bbb99630 con 0x5578bbbac620 2024-04-01T01:40:26.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.092+0000 7f7425b25640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 0x5578bbb9b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.092+0000 7f7425b25640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 0x5578bbb9b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47398/0 (socket says 172.21.15.70:47398) 2024-04-01T01:40:26.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.092+0000 7f7425b25640 1 -- 172.21.15.70:0/2709646385 learned_addr learned my addr 172.21.15.70:0/2709646385 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:26.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.092+0000 7f742b16b740 1 -- 172.21.15.70:0/2709646385 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5578bbaa14a0 con 0x5578bbba74b0 2024-04-01T01:40:26.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.092+0000 7f742b16b740 1 -- 172.21.15.70:0/2709646385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5578bbb997b0 con 0x5578bbba6ac0 2024-04-01T01:40:26.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.092+0000 7f7426326640 1 --2- 172.21.15.70:0/2709646385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbb9b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.092+0000 7f742b16b740 10 client.0 _ll_register_callbacks cb 0x5578bbaa4d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:26.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.093+0000 7f7426b27640 1 --2- 172.21.15.70:0/2709646385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 0x5578bbba0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.093+0000 7f7426326640 1 -- 172.21.15.70:0/2709646385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 msgr2=0x5578bbb9b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:26.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.093+0000 7f7426326640 1 --2- 172.21.15.70:0/2709646385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 0x5578bbb9b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.093+0000 7f7426326640 1 -- 172.21.15.70:0/2709646385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 msgr2=0x5578bbba0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:26.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.093+0000 7f7426326640 1 --2- 172.21.15.70:0/2709646385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 0x5578bbba0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.093+0000 7f7426326640 1 -- 172.21.15.70:0/2709646385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578bb8e17d0 con 0x5578bbba6ac0 2024-04-01T01:40:26.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.093+0000 7f7426b27640 1 --2- 172.21.15.70:0/2709646385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 0x5578bbba0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:40:26.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.093+0000 7f7426326640 1 --2- 172.21.15.70:0/2709646385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbb9b190 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f741c009230 tx=0x7f741c0045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f877933702ec561 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:26.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.093+0000 7f7409ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:40:26.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.094+0000 7f7409ffb640 1 -- 172.21.15.70:0/2709646385 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c046070 con 0x5578bbba6ac0 2024-04-01T01:40:26.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.094+0000 7f742b16b740 1 -- 172.21.15.70:0/2709646385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7410014cd0 con 0x5578bbba6ac0 2024-04-01T01:40:26.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.094+0000 7f7409ffb640 1 -- 172.21.15.70:0/2709646385 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f741c005250 con 0x5578bbba6ac0 2024-04-01T01:40:26.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.094+0000 7f7409ffb640 1 -- 172.21.15.70:0/2709646385 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f741c052050 con 0x5578bbba6ac0 2024-04-01T01:40:26.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.095+0000 7f742b16b740 10 client.27358 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:26.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.095+0000 7f742b16b740 1 -- 172.21.15.70:0/2709646385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5578bb8173f0 con 0x5578bbba6ac0 2024-04-01T01:40:26.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.095+0000 7f7409ffb640 1 -- 172.21.15.70:0/2709646385 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7410014cd0 con 0x5578bbba6ac0 2024-04-01T01:40:26.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.095+0000 7f7409ffb640 1 -- 172.21.15.70:0/2709646385 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f741c0035f0 con 0x5578bbba6ac0 2024-04-01T01:40:26.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.096+0000 7f742b16b740 10 client.27358 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:26.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.096+0000 7f742b16b740 10 client.27358 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:40:26.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.096+0000 7f742b16b740 20 client.27358 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:26.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.096+0000 7f742b16b740 1 --2- 172.21.15.70:0/2709646385 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5578bbbc5620 0x5578bbbc7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.096+0000 7f742b16b740 4 client.27358 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:40:26.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.096+0000 7f742b16b740 1 -- 172.21.15.70:0/2709646385 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578bbb106c0 con 0x5578bbbc5620 2024-04-01T01:40:26.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.096+0000 7f7425b25640 1 --2- 172.21.15.70:0/2709646385 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5578bbbc5620 0x5578bbbc7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.098+0000 7f7425b25640 1 --2- 172.21.15.70:0/2709646385 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5578bbbc5620 0x5578bbbc7a90 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e03d8c10507bad1 server_cookie=805e4461d83ff7e6 in_seq=0 out_seq=0 2024-04-01T01:40:26.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.098+0000 7f7409ffb640 10 client.27358 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:26.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.099+0000 7f7409ffb640 1 -- 172.21.15.70:0/2709646385 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5578bbb106c0 con 0x5578bbbc5620 2024-04-01T01:40:26.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.099+0000 7f7409ffb640 10 client.27358 handle_command_reply: tid=0 2024-04-01T01:40:26.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.099+0000 7f742b16b740 1 client.27358 shutdown 2024-04-01T01:40:26.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.099+0000 7f742b16b740 1 -- 172.21.15.70:0/2709646385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 msgr2=0x5578bbb9b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.099+0000 7f742b16b740 1 --2- 172.21.15.70:0/2709646385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbb9b190 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f741c009230 tx=0x7f741c0045e0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.099+0000 7f742b16b740 1 -- 172.21.15.70:0/2709646385 shutdown_connections 2024-04-01T01:40:26.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.099+0000 7f742b16b740 1 --2- 172.21.15.70:0/2709646385 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5578bbbc5620 0x5578bbbc7a90 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.099+0000 7f742b16b740 1 --2- 172.21.15.70:0/2709646385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 0x5578bbba0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.099+0000 7f742b16b740 1 --2- 172.21.15.70:0/2709646385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 0x5578bbb9b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.099+0000 7f742b16b740 1 --2- 172.21.15.70:0/2709646385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbb9b190 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.099+0000 7f742b16b740 1 -- 172.21.15.70:0/2709646385 >> 172.21.15.70:0/2709646385 conn(0x5578bbba2030 msgr2=0x5578bbba3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:26.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.099+0000 7f7409ffb640 0 client.27358 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:26.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.099+0000 7f742b16b740 1 -- 172.21.15.70:0/2709646385 shutdown_connections 2024-04-01T01:40:26.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.099+0000 7f742b16b740 1 -- 172.21.15.70:0/2709646385 wait complete. 2024-04-01T01:40:26.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.099+0000 7f742b16b740 20 client.27358 trim_cache size 0 max 16384 2024-04-01T01:40:26.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.132+0000 7f742b16b740 1 Processor -- start 2024-04-01T01:40:26.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.132+0000 7f742b16b740 1 -- start start 2024-04-01T01:40:26.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.133+0000 7f742b16b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbb98aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.133+0000 7f742b16b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 0x5578bbb9a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.133+0000 7f742b16b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 0x5578bbb9acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.133+0000 7f742b16b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f74201b0780 con 0x5578bbbac620 2024-04-01T01:40:26.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.133+0000 7f7425b25640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 0x5578bbb9a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.133+0000 7f7425b25640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 0x5578bbb9a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47400/0 (socket says 172.21.15.70:47400) 2024-04-01T01:40:26.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.133+0000 7f7425b25640 1 -- 172.21.15.70:0/3185351323 learned_addr learned my addr 172.21.15.70:0/3185351323 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.133+0000 7f742b16b740 1 -- 172.21.15.70:0/3185351323 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f74201b22c0 con 0x5578bbba74b0 2024-04-01T01:40:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.133+0000 7f742b16b740 1 -- 172.21.15.70:0/3185351323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f74201b0e50 con 0x5578bbba6ac0 2024-04-01T01:40:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.133+0000 7f7426326640 1 --2- 172.21.15.70:0/3185351323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbb98aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.133+0000 7f7426b27640 1 --2- 172.21.15.70:0/3185351323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 0x5578bbb9acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.134+0000 7f7425b25640 1 -- 172.21.15.70:0/3185351323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 msgr2=0x5578bbb98aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.134+0000 7f7425b25640 1 --2- 172.21.15.70:0/3185351323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbb98aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.134+0000 7f7425b25640 1 -- 172.21.15.70:0/3185351323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 msgr2=0x5578bbb9acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.134+0000 7f7425b25640 1 --2- 172.21.15.70:0/3185351323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 0x5578bbb9acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.134+0000 7f7425b25640 1 -- 172.21.15.70:0/3185351323 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578bbbc7e00 con 0x5578bbba74b0 2024-04-01T01:40:26.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.134+0000 7f7425b25640 1 --2- 172.21.15.70:0/3185351323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 0x5578bbb9a7b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f74100058c0 tx=0x7f741000ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b30b5b789122d8e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.134+0000 7f740bfff640 1 -- 172.21.15.70:0/3185351323 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74100140c0 con 0x5578bbba74b0 2024-04-01T01:40:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.134+0000 7f740bfff640 1 -- 172.21.15.70:0/3185351323 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7410014260 con 0x5578bbba74b0 2024-04-01T01:40:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.134+0000 7f740bfff640 1 -- 172.21.15.70:0/3185351323 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74100340d0 con 0x5578bbba74b0 2024-04-01T01:40:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.135+0000 7f742b16b740 1 -- 172.21.15.70:0/3185351323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 msgr2=0x5578bbb9a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.135+0000 7f742b16b740 1 --2- 172.21.15.70:0/3185351323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 0x5578bbb9a7b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f74100058c0 tx=0x7f741000ccc0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.136+0000 7f742b16b740 1 -- 172.21.15.70:0/3185351323 shutdown_connections 2024-04-01T01:40:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.136+0000 7f742b16b740 1 --2- 172.21.15.70:0/3185351323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbbac620 0x5578bbb9acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.136+0000 7f742b16b740 1 --2- 172.21.15.70:0/3185351323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbba74b0 0x5578bbb9a7b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.136+0000 7f742b16b740 1 --2- 172.21.15.70:0/3185351323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbb98aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.136+0000 7f742b16b740 1 -- 172.21.15.70:0/3185351323 >> 172.21.15.70:0/3185351323 conn(0x5578bbba2030 msgr2=0x5578bbb98390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:26.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.136+0000 7f742b16b740 1 -- 172.21.15.70:0/3185351323 shutdown_connections 2024-04-01T01:40:26.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.137+0000 7f742b16b740 1 -- 172.21.15.70:0/3185351323 wait complete. 2024-04-01T01:40:26.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.137+0000 7f742b16b740 1 -- start start 2024-04-01T01:40:26.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.138+0000 7f742b16b740 1 Processor -- start 2024-04-01T01:40:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.138+0000 7f742b16b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbbd69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.139+0000 7f7426326640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbbd69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.139+0000 7f7426326640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbbd69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:50462/0 (socket says 172.21.15.70:50462) 2024-04-01T01:40:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.139+0000 7f7426326640 1 -- 172.21.15.70:0/2499106468 learned_addr learned my addr 172.21.15.70:0/2499106468 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:26.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.139+0000 7f742b16b740 1 --2- 172.21.15.70:0/2499106468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbba74b0 0x5578bbbd6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.139+0000 7f742b16b740 1 --2- 172.21.15.70:0/2499106468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbbac620 0x5578bbbd7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.139+0000 7f7425b25640 1 --2- 172.21.15.70:0/2499106468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbba74b0 0x5578bbbd6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.139+0000 7f7426b27640 1 --2- 172.21.15.70:0/2499106468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbbac620 0x5578bbbd7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.140+0000 7f742b16b740 1 -- 172.21.15.70:0/2499106468 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f74201b1bf0 con 0x5578bbba74b0 2024-04-01T01:40:26.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.140+0000 7f742b16b740 1 -- 172.21.15.70:0/2499106468 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f74201b1520 con 0x5578bbbac620 2024-04-01T01:40:26.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.140+0000 7f742b16b740 1 -- 172.21.15.70:0/2499106468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f74201ae760 con 0x5578bbba6ac0 2024-04-01T01:40:26.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.141+0000 7f7426b27640 1 -- 172.21.15.70:0/2499106468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 msgr2=0x5578bbbd69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:26.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.141+0000 7f7426b27640 1 --2- 172.21.15.70:0/2499106468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbbd69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.141+0000 7f7426b27640 1 -- 172.21.15.70:0/2499106468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbba74b0 msgr2=0x5578bbbd6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:26.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.141+0000 7f7426b27640 1 --2- 172.21.15.70:0/2499106468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbba74b0 0x5578bbbd6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.141+0000 7f7426b27640 1 -- 172.21.15.70:0/2499106468 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578bb7d70f0 con 0x5578bbbac620 2024-04-01T01:40:26.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.141+0000 7f7426326640 1 --2- 172.21.15.70:0/2499106468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbbd69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:40:26.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.141+0000 7f7426b27640 1 --2- 172.21.15.70:0/2499106468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbbac620 0x5578bbbd7f30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f741809b730 tx=0x7f741809b100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db115a62b4483ad1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:26.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.141+0000 7f7409ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:40:26.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.141+0000 7f7409ffb640 1 -- 172.21.15.70:0/2499106468 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74180b3020 con 0x5578bbbac620 2024-04-01T01:40:26.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.141+0000 7f7409ffb640 1 -- 172.21.15.70:0/2499106468 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f74180a3d20 con 0x5578bbbac620 2024-04-01T01:40:26.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.141+0000 7f7409ffb640 10 client.0 inactive, discarding osd_map(94..94 src has 1..94) 2024-04-01T01:40:26.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.142+0000 7f742b16b740 10 client.0 _ll_register_callbacks cb 0x7f741c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:26.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.142+0000 7f742b16b740 1 -- 172.21.15.70:0/2499106468 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7410014260 con 0x5578bbbac620 2024-04-01T01:40:26.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.142+0000 7f7409ffb640 1 -- 172.21.15.70:0/2499106468 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f741809f8f0 con 0x5578bbbac620 2024-04-01T01:40:26.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.142+0000 7f742b16b740 10 client.27509 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:26.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.143+0000 7f742b16b740 1 -- 172.21.15.70:0/2499106468 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=31}) -- 0x5578bbc2b7d0 con 0x5578bbbac620 2024-04-01T01:40:26.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.143+0000 7f7409ffb640 1 -- 172.21.15.70:0/2499106468 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f7418005310 con 0x5578bbbac620 2024-04-01T01:40:26.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.143+0000 7f742b16b740 10 client.27509 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:26.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.143+0000 7f742b16b740 10 client.27509 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:40:26.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.143+0000 7f742b16b740 20 client.27509 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:26.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.143+0000 7f742b16b740 1 --2- 172.21.15.70:0/2499106468 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5578bbc2c060 0x5578bbc2e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.143+0000 7f742b16b740 4 client.27509 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:40:26.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.143+0000 7f742b16b740 1 -- 172.21.15.70:0/2499106468 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f7410014260 con 0x5578bbc2c060 2024-04-01T01:40:26.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.144+0000 7f7426326640 1 --2- 172.21.15.70:0/2499106468 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5578bbc2c060 0x5578bbc2e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.146+0000 7f7426326640 1 --2- 172.21.15.70:0/2499106468 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5578bbc2c060 0x5578bbc2e480 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e2fefd86198d75a server_cookie=b0dbd0739b38603 in_seq=0 out_seq=0 2024-04-01T01:40:26.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.147+0000 7f7409ffb640 10 client.27509 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:26.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.147+0000 7f7409ffb640 1 -- 172.21.15.70:0/2499106468 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f7410014260 con 0x5578bbc2c060 2024-04-01T01:40:26.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.147+0000 7f7409ffb640 10 client.27509 handle_command_reply: tid=0 2024-04-01T01:40:26.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.147+0000 7f742b16b740 1 client.27509 shutdown 2024-04-01T01:40:26.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.147+0000 7f742b16b740 1 -- 172.21.15.70:0/2499106468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbbac620 msgr2=0x5578bbbd7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.147+0000 7f742b16b740 1 --2- 172.21.15.70:0/2499106468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbbac620 0x5578bbbd7f30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f741809b730 tx=0x7f741809b100 comp rx=0 tx=0).stop 2024-04-01T01:40:26.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.147+0000 7f742b16b740 1 -- 172.21.15.70:0/2499106468 shutdown_connections 2024-04-01T01:40:26.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.147+0000 7f742b16b740 1 --2- 172.21.15.70:0/2499106468 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5578bbc2c060 0x5578bbc2e480 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.147+0000 7f742b16b740 1 --2- 172.21.15.70:0/2499106468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578bbbac620 0x5578bbbd7f30 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.147+0000 7f742b16b740 1 --2- 172.21.15.70:0/2499106468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578bbba74b0 0x5578bbbd6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.147+0000 7f7409ffb640 0 client.27509 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:26.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.147+0000 7f742b16b740 1 --2- 172.21.15.70:0/2499106468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578bbba6ac0 0x5578bbbd69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.148+0000 7f742b16b740 1 -- 172.21.15.70:0/2499106468 >> 172.21.15.70:0/2499106468 conn(0x5578bbba2030 msgr2=0x5578bbba5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:26.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.149+0000 7f742b16b740 1 -- 172.21.15.70:0/2499106468 shutdown_connections 2024-04-01T01:40:26.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.149+0000 7f742b16b740 1 -- 172.21.15.70:0/2499106468 wait complete. 2024-04-01T01:40:26.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.150+0000 7f742b16b740 20 client.27509 trim_cache size 0 max 16384 2024-04-01T01:40:26.149 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:40:26.149 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:40:26.149 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:40:26.150 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:40:26.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.151+0000 7f742858a640 1 -- 172.21.15.70:0/4114301788 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f740c0a25a0 msgr2=0x7f740c0a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.151+0000 7f742858a640 1 --2- 172.21.15.70:0/4114301788 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f740c0a25a0 0x7f740c0a4a60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f74201a3730 tx=0x7f7410016000 comp rx=0 tx=0).stop 2024-04-01T01:40:26.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.151+0000 7f742858a640 1 -- 172.21.15.70:0/4114301788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f742010dd10 msgr2=0x7f74201a97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.151+0000 7f742858a640 1 --2- 172.21.15.70:0/4114301788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f742010dd10 0x7f74201a97d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f741800eb10 tx=0x7f741800c2d0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.151+0000 7f742858a640 1 -- 172.21.15.70:0/4114301788 shutdown_connections 2024-04-01T01:40:26.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.152+0000 7f742858a640 1 --2- 172.21.15.70:0/4114301788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f742010dd10 0x7f74201a97d0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.152+0000 7f742858a640 1 --2- 172.21.15.70:0/4114301788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f742010d350 0x7f74201a2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.152+0000 7f742858a640 1 --2- 172.21.15.70:0/4114301788 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f740c0a25a0 0x7f740c0a4a60 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.152+0000 7f742858a640 1 --2- 172.21.15.70:0/4114301788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742010c160 0x7f74201a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.152+0000 7f742858a640 1 -- 172.21.15.70:0/4114301788 >> 172.21.15.70:0/4114301788 conn(0x7f742006c110 msgr2=0x7f7420110580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:26.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.152+0000 7f742858a640 1 -- 172.21.15.70:0/4114301788 shutdown_connections 2024-04-01T01:40:26.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.152+0000 7f742858a640 1 -- 172.21.15.70:0/4114301788 wait complete. 2024-04-01T01:40:26.175 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:40:26.175 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:40:26.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.350+0000 7fad22b03640 1 -- 172.21.15.70:0/2966593410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c101670 msgr2=0x7fad1c101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.350+0000 7fad22b03640 1 --2- 172.21.15.70:0/2966593410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c101670 0x7fad1c101af0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fad0c01a910 tx=0x7fad0c03cc70 comp rx=0 tx=0).stop 2024-04-01T01:40:26.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.350+0000 7fad22b03640 1 -- 172.21.15.70:0/2966593410 shutdown_connections 2024-04-01T01:40:26.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.350+0000 7fad22b03640 1 --2- 172.21.15.70:0/2966593410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad1c102030 0x7fad1c1088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.350+0000 7fad22b03640 1 --2- 172.21.15.70:0/2966593410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c101670 0x7fad1c101af0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.350+0000 7fad22b03640 1 --2- 172.21.15.70:0/2966593410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1c100480 0x7fad1c1008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.350+0000 7fad22b03640 1 -- 172.21.15.70:0/2966593410 >> 172.21.15.70:0/2966593410 conn(0x7fad1c0fb9d0 msgr2=0x7fad1c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:26.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.350+0000 7fad22b03640 1 -- 172.21.15.70:0/2966593410 shutdown_connections 2024-04-01T01:40:26.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.351+0000 7fad22b03640 1 -- 172.21.15.70:0/2966593410 wait complete. 2024-04-01T01:40:26.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.352+0000 7fad22b03640 1 Processor -- start 2024-04-01T01:40:26.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.352+0000 7fad22b03640 1 -- start start 2024-04-01T01:40:26.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.353+0000 7fad22b03640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1c100480 0x7fad1c18e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.353+0000 7fad22b03640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad1c101670 0x7fad1c18e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.353+0000 7fad22b03640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c102030 0x7fad1c1954b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.353+0000 7fad22b03640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c06dc70 con 0x7fad1c102030 2024-04-01T01:40:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.353+0000 7fad22b03640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c06daf0 con 0x7fad1c101670 2024-04-01T01:40:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.353+0000 7fad22b03640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c06ddf0 con 0x7fad1c100480 2024-04-01T01:40:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.353+0000 7fad22302640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c102030 0x7fad1c1954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.353+0000 7fad21b01640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1c100480 0x7fad1c18e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.353+0000 7fad22302640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c102030 0x7fad1c1954b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41442/0 (socket says 172.21.15.70:41442) 2024-04-01T01:40:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.353+0000 7fad22302640 1 -- 172.21.15.70:0/108732415 learned_addr learned my addr 172.21.15.70:0/108732415 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.354+0000 7fad21300640 1 --2- 172.21.15.70:0/108732415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad1c101670 0x7fad1c18e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.354+0000 7fad22302640 1 -- 172.21.15.70:0/108732415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1c100480 msgr2=0x7fad1c18e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.354+0000 7fad22302640 1 --2- 172.21.15.70:0/108732415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1c100480 0x7fad1c18e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.354+0000 7fad22302640 1 -- 172.21.15.70:0/108732415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad1c101670 msgr2=0x7fad1c18e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.354+0000 7fad22302640 1 --2- 172.21.15.70:0/108732415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad1c101670 0x7fad1c18e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.354+0000 7fad22302640 1 -- 172.21.15.70:0/108732415 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad1c195b20 con 0x7fad1c102030 2024-04-01T01:40:26.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.355+0000 7fad21b01640 1 --2- 172.21.15.70:0/108732415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1c100480 0x7fad1c18e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:40:26.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.355+0000 7fad22302640 1 --2- 172.21.15.70:0/108732415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c102030 0x7fad1c1954b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fad1400eaa0 tx=0x7fad1400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:26.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.355+0000 7fad12ffd640 1 -- 172.21.15.70:0/108732415 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1400cdb0 con 0x7fad1c102030 2024-04-01T01:40:26.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.355+0000 7fad22b03640 1 -- 172.21.15.70:0/108732415 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad1c069c40 con 0x7fad1c102030 2024-04-01T01:40:26.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.355+0000 7fad12ffd640 1 -- 172.21.15.70:0/108732415 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad14005020 con 0x7fad1c102030 2024-04-01T01:40:26.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.355+0000 7fad12ffd640 1 -- 172.21.15.70:0/108732415 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad140116d0 con 0x7fad1c102030 2024-04-01T01:40:26.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.356+0000 7fad22b03640 1 -- 172.21.15.70:0/108732415 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad1c06a160 con 0x7fad1c102030 2024-04-01T01:40:26.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.357+0000 7fad22b03640 1 -- 172.21.15.70:0/108732415 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad1c0688b0 con 0x7fad1c102030 2024-04-01T01:40:26.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.359+0000 7fad12ffd640 1 -- 172.21.15.70:0/108732415 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fad140051d0 con 0x7fad1c102030 2024-04-01T01:40:26.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.359+0000 7fad12ffd640 1 --2- 172.21.15.70:0/108732415 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fad080a25a0 0x7fad080a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.360+0000 7fad12ffd640 1 -- 172.21.15.70:0/108732415 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fad140cec30 con 0x7fad1c102030 2024-04-01T01:40:26.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.360+0000 7fad21b01640 1 --2- 172.21.15.70:0/108732415 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fad080a25a0 0x7fad080a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.363+0000 7fad21b01640 1 --2- 172.21.15.70:0/108732415 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fad080a25a0 0x7fad080a4a60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fad1c1014d0 tx=0x7fad18005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:26.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.367+0000 7fad12ffd640 1 -- 172.21.15.70:0/108732415 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fad14016030 con 0x7fad1c102030 2024-04-01T01:40:26.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.773+0000 7fad22b03640 1 -- 172.21.15.70:0/108732415 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad1c19a1c0 con 0x7fad1c102030 2024-04-01T01:40:26.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.777+0000 7fad12ffd640 1 -- 172.21.15.70:0/108732415 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7fad1409b370 con 0x7fad1c102030 2024-04-01T01:40:26.777 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:40:26.777 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:40:26.778 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:40:26.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.781+0000 7fad22b03640 1 -- 172.21.15.70:0/108732415 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fad080a25a0 msgr2=0x7fad080a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.781+0000 7fad22b03640 1 --2- 172.21.15.70:0/108732415 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fad080a25a0 0x7fad080a4a60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fad1c1014d0 tx=0x7fad18005f90 comp rx=0 tx=0).stop 2024-04-01T01:40:26.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.781+0000 7fad22b03640 1 -- 172.21.15.70:0/108732415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c102030 msgr2=0x7fad1c1954b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.781+0000 7fad22b03640 1 --2- 172.21.15.70:0/108732415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c102030 0x7fad1c1954b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fad1400eaa0 tx=0x7fad1400ef60 comp rx=0 tx=0).stop 2024-04-01T01:40:26.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.781+0000 7fad22b03640 1 -- 172.21.15.70:0/108732415 shutdown_connections 2024-04-01T01:40:26.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.781+0000 7fad22b03640 1 --2- 172.21.15.70:0/108732415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad1c102030 0x7fad1c1954b0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.781+0000 7fad22b03640 1 --2- 172.21.15.70:0/108732415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fad1c101670 0x7fad1c18e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.781+0000 7fad22b03640 1 --2- 172.21.15.70:0/108732415 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fad080a25a0 0x7fad080a4a60 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.781+0000 7fad22b03640 1 --2- 172.21.15.70:0/108732415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1c100480 0x7fad1c18e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.781+0000 7fad22b03640 1 -- 172.21.15.70:0/108732415 >> 172.21.15.70:0/108732415 conn(0x7fad1c0fb9d0 msgr2=0x7fad1c070760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:26.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.782+0000 7fad22b03640 1 -- 172.21.15.70:0/108732415 shutdown_connections 2024-04-01T01:40:26.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.782+0000 7fad22b03640 1 -- 172.21.15.70:0/108732415 wait complete. 2024-04-01T01:40:26.800 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:40:26.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 340.0 s, 558.6 tps, lat 256.879 ms stddev 521.417 2024-04-01T01:40:26.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.940+0000 7f1a9461c640 1 -- 172.21.15.70:0/3055831042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a8c0ffa30 msgr2=0x7f1a8c0ffe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.940+0000 7f1a9461c640 1 --2- 172.21.15.70:0/3055831042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a8c0ffa30 0x7f1a8c0ffe50 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f1a88009900 tx=0x7f1a8802bc80 comp rx=0 tx=0).stop 2024-04-01T01:40:26.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.940+0000 7f1a9461c640 1 -- 172.21.15.70:0/3055831042 shutdown_connections 2024-04-01T01:40:26.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.940+0000 7f1a9461c640 1 --2- 172.21.15.70:0/3055831042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a8c103460 0x7f1a8c109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.940+0000 7f1a9461c640 1 --2- 172.21.15.70:0/3055831042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a8c102aa0 0x7f1a8c102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.940+0000 7f1a9461c640 1 --2- 172.21.15.70:0/3055831042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a8c0ffa30 0x7f1a8c0ffe50 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.940+0000 7f1a9461c640 1 -- 172.21.15.70:0/3055831042 >> 172.21.15.70:0/3055831042 conn(0x7f1a8c0fb340 msgr2=0x7f1a8c0fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:26.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.940+0000 7f1a9461c640 1 -- 172.21.15.70:0/3055831042 shutdown_connections 2024-04-01T01:40:26.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.940+0000 7f1a9461c640 1 -- 172.21.15.70:0/3055831042 wait complete. 2024-04-01T01:40:26.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a9461c640 1 Processor -- start 2024-04-01T01:40:26.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a9461c640 1 -- start start 2024-04-01T01:40:26.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a9461c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a8c0ffa30 0x7f1a8c102410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a9461c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a8c102aa0 0x7f1a8c100a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a9461c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a8c103460 0x7f1a8c100f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a9461c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c10d2b0 con 0x7f1a8c102aa0 2024-04-01T01:40:26.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a9461c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c10d130 con 0x7f1a8c103460 2024-04-01T01:40:26.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a9461c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c10d430 con 0x7f1a8c0ffa30 2024-04-01T01:40:26.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a91bb7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a8c102aa0 0x7f1a8c100a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a91bb7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a8c102aa0 0x7f1a8c100a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47388/0 (socket says 172.21.15.70:47388) 2024-04-01T01:40:26.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a91bb7640 1 -- 172.21.15.70:0/3859410689 learned_addr learned my addr 172.21.15.70:0/3859410689 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:26.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a923b8640 1 --2- 172.21.15.70:0/3859410689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a8c0ffa30 0x7f1a8c102410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a92bb9640 1 --2- 172.21.15.70:0/3859410689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a8c103460 0x7f1a8c100f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a91bb7640 1 -- 172.21.15.70:0/3859410689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a8c0ffa30 msgr2=0x7f1a8c102410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a91bb7640 1 --2- 172.21.15.70:0/3859410689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a8c0ffa30 0x7f1a8c102410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a91bb7640 1 -- 172.21.15.70:0/3859410689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a8c103460 msgr2=0x7f1a8c100f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a91bb7640 1 --2- 172.21.15.70:0/3859410689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a8c103460 0x7f1a8c100f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a91bb7640 1 -- 172.21.15.70:0/3859410689 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a8c101820 con 0x7f1a8c102aa0 2024-04-01T01:40:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a91bb7640 1 --2- 172.21.15.70:0/3859410689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a8c102aa0 0x7f1a8c100a50 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c00e9f0 tx=0x7f1a7c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.941+0000 7f1a837fe640 1 -- 172.21.15.70:0/3859410689 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c00cd80 con 0x7f1a8c102aa0 2024-04-01T01:40:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.942+0000 7f1a837fe640 1 -- 172.21.15.70:0/3859410689 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a7c0108e0 con 0x7f1a8c102aa0 2024-04-01T01:40:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.942+0000 7f1a9461c640 1 -- 172.21.15.70:0/3859410689 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a8c1a1b00 con 0x7f1a8c102aa0 2024-04-01T01:40:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.942+0000 7f1a837fe640 1 -- 172.21.15.70:0/3859410689 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c018920 con 0x7f1a8c102aa0 2024-04-01T01:40:26.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.942+0000 7f1a9461c640 1 -- 172.21.15.70:0/3859410689 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a8c1a2050 con 0x7f1a8c102aa0 2024-04-01T01:40:26.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.949+0000 7f1a97200740 1 Processor -- start 2024-04-01T01:40:26.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.949+0000 7f1a97200740 1 -- start start 2024-04-01T01:40:26.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.949+0000 7f1a97200740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d030c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.949+0000 7f1a97200740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d03690 0x55d4b6d08190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.949+0000 7f1a97200740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d08800 0x55d4b6d0ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.949+0000 7f1a97200740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d4b6c5f1b0 con 0x55d4b6d02ca0 2024-04-01T01:40:26.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.949+0000 7f1a97200740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d4b6c07c00 con 0x55d4b6d03690 2024-04-01T01:40:26.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.949+0000 7f1a97200740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d4b6c5b250 con 0x55d4b6d08800 2024-04-01T01:40:26.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.949+0000 7f1a92bb9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.949+0000 7f1a92bb9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d030c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47394/0 (socket says 172.21.15.70:47394) 2024-04-01T01:40:26.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.950+0000 7f1a92bb9640 1 -- 172.21.15.70:0/1111740374 learned_addr learned my addr 172.21.15.70:0/1111740374 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:26.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.950+0000 7f1a923b8640 1 --2- 172.21.15.70:0/1111740374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d03690 0x55d4b6d08190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.950+0000 7f1a92bb9640 1 -- 172.21.15.70:0/1111740374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d08800 msgr2=0x55d4b6d0ac90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:40:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.950+0000 7f1a92bb9640 1 --2- 172.21.15.70:0/1111740374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d08800 0x55d4b6d0ac90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.950+0000 7f1a92bb9640 1 -- 172.21.15.70:0/1111740374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d03690 msgr2=0x55d4b6d08190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.950+0000 7f1a92bb9640 1 --2- 172.21.15.70:0/1111740374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d03690 0x55d4b6d08190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.950+0000 7f1a92bb9640 1 -- 172.21.15.70:0/1111740374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4b6c273d0 con 0x55d4b6d02ca0 2024-04-01T01:40:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.950+0000 7f1a92bb9640 1 --2- 172.21.15.70:0/1111740374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d030c0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f1a84009fd0 tx=0x7f1a840118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9785690711866d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.951+0000 7f1a923b8640 1 --2- 172.21.15.70:0/1111740374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d03690 0x55d4b6d08190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:40:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.951+0000 7f1a80bfe640 1 -- 172.21.15.70:0/1111740374 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84004d20 con 0x55d4b6d02ca0 2024-04-01T01:40:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.951+0000 7f1a80bfe640 1 -- 172.21.15.70:0/1111740374 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a84004050 con 0x55d4b6d02ca0 2024-04-01T01:40:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.951+0000 7f1a80bfe640 1 -- 172.21.15.70:0/1111740374 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a84005260 con 0x55d4b6d02ca0 2024-04-01T01:40:26.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.951+0000 7f1a837fe640 1 -- 172.21.15.70:0/3859410689 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f1a7c010a90 con 0x7f1a8c102aa0 2024-04-01T01:40:26.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.952+0000 7f1a837fe640 1 --2- 172.21.15.70:0/3859410689 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a780a24a0 0x7f1a780a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.952+0000 7f1a97200740 1 -- 172.21.15.70:0/1111740374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d02ca0 msgr2=0x55d4b6d030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.952+0000 7f1a97200740 1 --2- 172.21.15.70:0/1111740374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d030c0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f1a84009fd0 tx=0x7f1a840118d0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.952+0000 7f1a837fe640 1 -- 172.21.15.70:0/3859410689 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f1a7c0cedc0 con 0x7f1a8c102aa0 2024-04-01T01:40:26.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.952+0000 7f1a97200740 1 -- 172.21.15.70:0/1111740374 shutdown_connections 2024-04-01T01:40:26.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.952+0000 7f1a97200740 1 --2- 172.21.15.70:0/1111740374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d08800 0x55d4b6d0ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.952+0000 7f1a97200740 1 --2- 172.21.15.70:0/1111740374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d03690 0x55d4b6d08190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.952+0000 7f1a97200740 1 --2- 172.21.15.70:0/1111740374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d030c0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.952+0000 7f1a97200740 1 -- 172.21.15.70:0/1111740374 >> 172.21.15.70:0/1111740374 conn(0x55d4b6cfe210 msgr2=0x55d4b6cfe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.953+0000 7f1a923b8640 1 --2- 172.21.15.70:0/3859410689 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a780a24a0 0x7f1a780a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.953+0000 7f1a97200740 1 -- 172.21.15.70:0/1111740374 shutdown_connections 2024-04-01T01:40:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.953+0000 7f1a97200740 1 -- 172.21.15.70:0/1111740374 wait complete. 2024-04-01T01:40:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.953+0000 7f1a97200740 1 -- start start 2024-04-01T01:40:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.953+0000 7f1a97200740 1 Processor -- start 2024-04-01T01:40:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a97200740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d0d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a923b8640 1 --2- 172.21.15.70:0/3859410689 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a780a24a0 0x7f1a780a4960 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1a88005ab0 tx=0x7f1a88005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a97200740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d03690 0x55d4b6d0d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a97200740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d08800 0x55d4b6d14800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a97200740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d4b6d0b7b0 con 0x55d4b6d08800 2024-04-01T01:40:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a97200740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d4b6bfd470 con 0x55d4b6d03690 2024-04-01T01:40:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a97200740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d4b6d0b930 con 0x55d4b6d02ca0 2024-04-01T01:40:26.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a97200740 10 client.0 _ll_register_callbacks cb 0x55d4b6c1acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:26.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a91bb7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d03690 0x55d4b6d0d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a91bb7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d03690 0x55d4b6d0d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:58848/0 (socket says 172.21.15.70:58848) 2024-04-01T01:40:26.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a91bb7640 1 -- 172.21.15.70:0/438643914 learned_addr learned my addr 172.21.15.70:0/438643914 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:26.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a923b8640 1 --2- 172.21.15.70:0/438643914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d0d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a92bb9640 1 --2- 172.21.15.70:0/438643914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d08800 0x55d4b6d14800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a91bb7640 1 -- 172.21.15.70:0/438643914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d02ca0 msgr2=0x55d4b6d0d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a91bb7640 1 --2- 172.21.15.70:0/438643914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d0d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a91bb7640 1 -- 172.21.15.70:0/438643914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d08800 msgr2=0x55d4b6d14800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a91bb7640 1 --2- 172.21.15.70:0/438643914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d08800 0x55d4b6d14800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a91bb7640 1 -- 172.21.15.70:0/438643914 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4b69733f0 con 0x55d4b6d03690 2024-04-01T01:40:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.954+0000 7f1a91bb7640 1 --2- 172.21.15.70:0/438643914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d03690 0x55d4b6d0d7c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c00c6a0 tx=0x7f1a7c010de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaf8a8faaa5ccb11 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.955+0000 7f1a75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:40:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.955+0000 7f1a75ffb640 1 -- 172.21.15.70:0/438643914 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c0bb2c0 con 0x55d4b6d03690 2024-04-01T01:40:26.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.955+0000 7f1a75ffb640 1 -- 172.21.15.70:0/438643914 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f1a7c010430 con 0x55d4b6d03690 2024-04-01T01:40:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.955+0000 7f1a97200740 1 -- 172.21.15.70:0/438643914 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a84004050 con 0x55d4b6d03690 2024-04-01T01:40:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.955+0000 7f1a75ffb640 1 -- 172.21.15.70:0/438643914 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1a7c0a3ec0 con 0x55d4b6d03690 2024-04-01T01:40:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.955+0000 7f1a97200740 10 client.27527 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.955+0000 7f1a97200740 1 -- 172.21.15.70:0/438643914 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d4b6a3d7d0 con 0x55d4b6d03690 2024-04-01T01:40:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.955+0000 7f1a75ffb640 1 -- 172.21.15.70:0/438643914 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a84004050 con 0x55d4b6d03690 2024-04-01T01:40:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.955+0000 7f1a75ffb640 1 -- 172.21.15.70:0/438643914 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f1a7c084af0 con 0x55d4b6d03690 2024-04-01T01:40:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.955+0000 7f1a97200740 10 client.27527 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.955+0000 7f1a97200740 10 client.27527 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:40:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.955+0000 7f1a97200740 20 client.27527 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.956+0000 7f1a97200740 1 --2- 172.21.15.70:0/438643914 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55d4b6d18c90 0x55d4b6d1b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.956+0000 7f1a97200740 4 client.27527 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:40:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.956+0000 7f1a97200740 1 -- 172.21.15.70:0/438643914 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4b6c6c6f0 con 0x55d4b6d18c90 2024-04-01T01:40:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.956+0000 7f1a923b8640 1 --2- 172.21.15.70:0/438643914 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55d4b6d18c90 0x55d4b6d1b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.956+0000 7f1a75ffb640 10 client.27527 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:40:26.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.956+0000 7f1a923b8640 1 --2- 172.21.15.70:0/438643914 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55d4b6d18c90 0x55d4b6d1b0a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bd3bf2e7e0a3be5c server_cookie=43870f3b21d6e3eb in_seq=0 out_seq=0 2024-04-01T01:40:26.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.957+0000 7f1a75ffb640 1 -- 172.21.15.70:0/438643914 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d4b6c6c6f0 con 0x55d4b6d18c90 2024-04-01T01:40:26.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.957+0000 7f1a75ffb640 10 client.27527 handle_command_reply: tid=0 2024-04-01T01:40:26.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.957+0000 7f1a97200740 1 client.27527 shutdown 2024-04-01T01:40:26.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.958+0000 7f1a97200740 1 -- 172.21.15.70:0/438643914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d03690 msgr2=0x55d4b6d0d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.958+0000 7f1a97200740 1 --2- 172.21.15.70:0/438643914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d03690 0x55d4b6d0d7c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c00c6a0 tx=0x7f1a7c010de0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.958+0000 7f1a97200740 1 -- 172.21.15.70:0/438643914 shutdown_connections 2024-04-01T01:40:26.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.958+0000 7f1a97200740 1 --2- 172.21.15.70:0/438643914 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55d4b6d18c90 0x55d4b6d1b0a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.958+0000 7f1a97200740 1 --2- 172.21.15.70:0/438643914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d08800 0x55d4b6d14800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.958+0000 7f1a97200740 1 --2- 172.21.15.70:0/438643914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d03690 0x55d4b6d0d7c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.958+0000 7f1a97200740 1 --2- 172.21.15.70:0/438643914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d0d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.958+0000 7f1a97200740 1 -- 172.21.15.70:0/438643914 >> 172.21.15.70:0/438643914 conn(0x55d4b6cfe210 msgr2=0x55d4b6cf4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:26.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.958+0000 7f1a75ffb640 0 client.27527 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:40:26.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.958+0000 7f1a97200740 1 -- 172.21.15.70:0/438643914 shutdown_connections 2024-04-01T01:40:26.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.958+0000 7f1a97200740 1 -- 172.21.15.70:0/438643914 wait complete. 2024-04-01T01:40:26.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.958+0000 7f1a97200740 20 client.27527 trim_cache size 0 max 16384 2024-04-01T01:40:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.983+0000 7f1a97200740 1 Processor -- start 2024-04-01T01:40:26.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.983+0000 7f1a97200740 1 -- start start 2024-04-01T01:40:26.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.984+0000 7f1a97200740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d0bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.984+0000 7f1a97200740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d03690 0x55d4b6d147c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.984+0000 7f1a97200740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d08800 0x55d4b6d0c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:26.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.984+0000 7f1a97200740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c1a8290 con 0x55d4b6d08800 2024-04-01T01:40:26.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.984+0000 7f1a97200740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c1a9dd0 con 0x55d4b6d02ca0 2024-04-01T01:40:26.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.984+0000 7f1a97200740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c1a8960 con 0x55d4b6d03690 2024-04-01T01:40:26.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.984+0000 7f1a91bb7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d03690 0x55d4b6d147c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.984+0000 7f1a91bb7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d03690 0x55d4b6d147c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37350/0 (socket says 172.21.15.70:37350) 2024-04-01T01:40:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.984+0000 7f1a91bb7640 1 -- 172.21.15.70:0/357368232 learned_addr learned my addr 172.21.15.70:0/357368232 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.984+0000 7f1a92bb9640 1 --2- 172.21.15.70:0/357368232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d08800 0x55d4b6d0c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.984+0000 7f1a923b8640 1 --2- 172.21.15.70:0/357368232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d0bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.985+0000 7f1a91bb7640 1 -- 172.21.15.70:0/357368232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d02ca0 msgr2=0x55d4b6d0bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.985+0000 7f1a91bb7640 1 --2- 172.21.15.70:0/357368232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d0bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.985+0000 7f1a91bb7640 1 -- 172.21.15.70:0/357368232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d08800 msgr2=0x55d4b6d0c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:26.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.985+0000 7f1a91bb7640 1 --2- 172.21.15.70:0/357368232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d08800 0x55d4b6d0c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.985+0000 7f1a91bb7640 1 -- 172.21.15.70:0/357368232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4b6d028e0 con 0x55d4b6d03690 2024-04-01T01:40:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.985+0000 7f1a92bb9640 1 --2- 172.21.15.70:0/357368232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d08800 0x55d4b6d0c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:40:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.985+0000 7f1a91bb7640 1 --2- 172.21.15.70:0/357368232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d03690 0x55d4b6d147c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c09b4f0 tx=0x7f1a7c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fba1023720b0deca server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.988+0000 7f1a77fff640 1 -- 172.21.15.70:0/357368232 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c09f310 con 0x55d4b6d03690 2024-04-01T01:40:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.988+0000 7f1a77fff640 1 -- 172.21.15.70:0/357368232 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a7c097380 con 0x55d4b6d03690 2024-04-01T01:40:26.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.988+0000 7f1a77fff640 1 -- 172.21.15.70:0/357368232 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c0bb070 con 0x55d4b6d03690 2024-04-01T01:40:26.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.993+0000 7f1a97200740 1 -- 172.21.15.70:0/357368232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d03690 msgr2=0x55d4b6d147c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:26.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:26.993+0000 7f1a97200740 1 --2- 172.21.15.70:0/357368232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d03690 0x55d4b6d147c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c09b4f0 tx=0x7f1a7c000bc0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.001+0000 7f1a97200740 1 -- 172.21.15.70:0/357368232 shutdown_connections 2024-04-01T01:40:27.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.001+0000 7f1a97200740 1 --2- 172.21.15.70:0/357368232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d08800 0x55d4b6d0c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.001+0000 7f1a97200740 1 --2- 172.21.15.70:0/357368232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d03690 0x55d4b6d147c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.001+0000 7f1a97200740 1 --2- 172.21.15.70:0/357368232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d0bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.001+0000 7f1a97200740 1 -- 172.21.15.70:0/357368232 >> 172.21.15.70:0/357368232 conn(0x55d4b6cfe210 msgr2=0x55d4b6d04ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.005+0000 7f1a97200740 1 -- 172.21.15.70:0/357368232 shutdown_connections 2024-04-01T01:40:27.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.005+0000 7f1a97200740 1 -- 172.21.15.70:0/357368232 wait complete. 2024-04-01T01:40:27.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.006+0000 7f1a97200740 1 -- start start 2024-04-01T01:40:27.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.006+0000 7f1a97200740 1 Processor -- start 2024-04-01T01:40:27.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.006+0000 7f1a97200740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d12780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.006+0000 7f1a97200740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d03690 0x55d4b6d12cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.006+0000 7f1a97200740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d08800 0x55d4b6d13c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.006+0000 7f1a97200740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c1a9700 con 0x55d4b6d03690 2024-04-01T01:40:27.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.006+0000 7f1a97200740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c1a9030 con 0x55d4b6d08800 2024-04-01T01:40:27.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.006+0000 7f1a97200740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8c1a6270 con 0x55d4b6d02ca0 2024-04-01T01:40:27.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.006+0000 7f1a97200740 10 client.0 _ll_register_callbacks cb 0x7f1a7c09b0a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.007+0000 7f1a923b8640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d12780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.007+0000 7f1a923b8640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d12780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37352/0 (socket says 172.21.15.70:37352) 2024-04-01T01:40:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.007+0000 7f1a923b8640 1 -- 172.21.15.70:0/3278297359 learned_addr learned my addr 172.21.15.70:0/3278297359 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.007+0000 7f1a92bb9640 1 --2- 172.21.15.70:0/3278297359 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d08800 0x55d4b6d13c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.007+0000 7f1a923b8640 1 -- 172.21.15.70:0/3278297359 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d08800 msgr2=0x55d4b6d13c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.007+0000 7f1a923b8640 1 --2- 172.21.15.70:0/3278297359 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d08800 0x55d4b6d13c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.007+0000 7f1a923b8640 1 -- 172.21.15.70:0/3278297359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d03690 msgr2=0x55d4b6d12cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.007+0000 7f1a923b8640 1 --2- 172.21.15.70:0/3278297359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d03690 0x55d4b6d12cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.007+0000 7f1a923b8640 1 -- 172.21.15.70:0/3278297359 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4b6d14270 con 0x55d4b6d02ca0 2024-04-01T01:40:27.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.008+0000 7f1a923b8640 1 --2- 172.21.15.70:0/3278297359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d12780 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1a88005ec0 tx=0x7f1a88009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea9ffc4bb05ccfe server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.008+0000 7f1a75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:40:27.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.008+0000 7f1a75ffb640 1 -- 172.21.15.70:0/3278297359 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a880040c0 con 0x55d4b6d02ca0 2024-04-01T01:40:27.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.008+0000 7f1a97200740 1 -- 172.21.15.70:0/3278297359 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a7c097380 con 0x55d4b6d02ca0 2024-04-01T01:40:27.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.008+0000 7f1a75ffb640 1 -- 172.21.15.70:0/3278297359 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f1a88004260 con 0x55d4b6d02ca0 2024-04-01T01:40:27.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.009+0000 7f1a75ffb640 1 -- 172.21.15.70:0/3278297359 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1a88065050 con 0x55d4b6d02ca0 2024-04-01T01:40:27.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.010+0000 7f1a97200740 10 client.27385 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:27.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.010+0000 7f1a97200740 1 -- 172.21.15.70:0/3278297359 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d4b6cfae00 con 0x55d4b6d02ca0 2024-04-01T01:40:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.011+0000 7f1a75ffb640 1 -- 172.21.15.70:0/3278297359 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a7c097380 con 0x55d4b6d02ca0 2024-04-01T01:40:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.012+0000 7f1a75ffb640 1 -- 172.21.15.70:0/3278297359 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f1a880046c0 con 0x55d4b6d02ca0 2024-04-01T01:40:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.012+0000 7f1a97200740 10 client.27385 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.012+0000 7f1a97200740 10 client.27385 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:40:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.012+0000 7f1a97200740 20 client.27385 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.012+0000 7f1a97200740 1 --2- 172.21.15.70:0/3278297359 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55d4b6cfb800 0x55d4b6d78f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.012+0000 7f1a97200740 4 client.27385 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:40:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.012+0000 7f1a97200740 1 -- 172.21.15.70:0/3278297359 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d4b6c66a40 con 0x55d4b6cfb800 2024-04-01T01:40:27.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.014+0000 7f1a91bb7640 1 --2- 172.21.15.70:0/3278297359 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55d4b6cfb800 0x55d4b6d78f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.020+0000 7f1a91bb7640 1 --2- 172.21.15.70:0/3278297359 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55d4b6cfb800 0x55d4b6d78f70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f0b8b7591b3b53c8 server_cookie=335a18697289c2f8 in_seq=0 out_seq=0 2024-04-01T01:40:27.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.020+0000 7f1a75ffb640 10 client.27385 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:40:27.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.025+0000 7f1a75ffb640 1 -- 172.21.15.70:0/3278297359 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d4b6c66a40 con 0x55d4b6cfb800 2024-04-01T01:40:27.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.025+0000 7f1a75ffb640 10 client.27385 handle_command_reply: tid=0 2024-04-01T01:40:27.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.025+0000 7f1a97200740 1 client.27385 shutdown 2024-04-01T01:40:27.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.026+0000 7f1a97200740 1 -- 172.21.15.70:0/3278297359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d02ca0 msgr2=0x55d4b6d12780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.026+0000 7f1a97200740 1 --2- 172.21.15.70:0/3278297359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d12780 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1a88005ec0 tx=0x7f1a88009260 comp rx=0 tx=0).stop 2024-04-01T01:40:27.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.026+0000 7f1a97200740 1 -- 172.21.15.70:0/3278297359 shutdown_connections 2024-04-01T01:40:27.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.026+0000 7f1a97200740 1 --2- 172.21.15.70:0/3278297359 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55d4b6cfb800 0x55d4b6d78f70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.026+0000 7f1a97200740 1 --2- 172.21.15.70:0/3278297359 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d4b6d08800 0x55d4b6d13c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.026+0000 7f1a97200740 1 --2- 172.21.15.70:0/3278297359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d4b6d03690 0x55d4b6d12cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.026+0000 7f1a97200740 1 --2- 172.21.15.70:0/3278297359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b6d02ca0 0x55d4b6d12780 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.026+0000 7f1a97200740 1 -- 172.21.15.70:0/3278297359 >> 172.21.15.70:0/3278297359 conn(0x55d4b6cfe210 msgr2=0x55d4b6d0d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.026+0000 7f1a75ffb640 0 client.27385 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:40:27.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.028+0000 7f1a97200740 1 -- 172.21.15.70:0/3278297359 shutdown_connections 2024-04-01T01:40:27.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.028+0000 7f1a97200740 1 -- 172.21.15.70:0/3278297359 wait complete. 2024-04-01T01:40:27.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.028+0000 7f1a97200740 20 client.27385 trim_cache size 0 max 16384 2024-04-01T01:40:27.028 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:40:27.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.029+0000 7f1a9461c640 1 -- 172.21.15.70:0/3859410689 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a780a24a0 msgr2=0x7f1a780a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.029+0000 7f1a9461c640 1 --2- 172.21.15.70:0/3859410689 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a780a24a0 0x7f1a780a4960 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1a88005ab0 tx=0x7f1a88005fd0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.029+0000 7f1a9461c640 1 -- 172.21.15.70:0/3859410689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a8c102aa0 msgr2=0x7f1a8c100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.029+0000 7f1a9461c640 1 --2- 172.21.15.70:0/3859410689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a8c102aa0 0x7f1a8c100a50 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c00e9f0 tx=0x7f1a7c00eeb0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.029+0000 7f1a9461c640 1 -- 172.21.15.70:0/3859410689 shutdown_connections 2024-04-01T01:40:27.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.029+0000 7f1a9461c640 1 --2- 172.21.15.70:0/3859410689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a8c103460 0x7f1a8c100f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.029+0000 7f1a9461c640 1 --2- 172.21.15.70:0/3859410689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a8c102aa0 0x7f1a8c100a50 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.029+0000 7f1a9461c640 1 --2- 172.21.15.70:0/3859410689 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a780a24a0 0x7f1a780a4960 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.029+0000 7f1a9461c640 1 --2- 172.21.15.70:0/3859410689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a8c0ffa30 0x7f1a8c102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.029+0000 7f1a9461c640 1 -- 172.21.15.70:0/3859410689 >> 172.21.15.70:0/3859410689 conn(0x7f1a8c0fb340 msgr2=0x7f1a8c105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.029+0000 7f1a9461c640 1 -- 172.21.15.70:0/3859410689 shutdown_connections 2024-04-01T01:40:27.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.029+0000 7f1a9461c640 1 -- 172.21.15.70:0/3859410689 wait complete. 2024-04-01T01:40:27.047 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:40:27.095 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:26 smithi070 ceph-mon[32179]: pgmap v568: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 16 MiB/s wr, 1.13k op/s 2024-04-01T01:40:27.095 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:26 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/108732415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:40:27.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.233+0000 7f5ce94c2640 1 -- 172.21.15.70:0/1149160726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ce41090b0 msgr2=0x7f5ce410fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.233+0000 7f5ce94c2640 1 --2- 172.21.15.70:0/1149160726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ce41090b0 0x7f5ce410fa90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009900 tx=0x7f5cd802bc30 comp rx=0 tx=0).stop 2024-04-01T01:40:27.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.233+0000 7f5ce94c2640 1 -- 172.21.15.70:0/1149160726 shutdown_connections 2024-04-01T01:40:27.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.233+0000 7f5ce94c2640 1 --2- 172.21.15.70:0/1149160726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce410ffd0 0x7f5ce41123c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.233+0000 7f5ce94c2640 1 --2- 172.21.15.70:0/1149160726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ce41090b0 0x7f5ce410fa90 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.233+0000 7f5ce94c2640 1 --2- 172.21.15.70:0/1149160726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ce4108750 0x7f5ce4108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.233+0000 7f5ce94c2640 1 -- 172.21.15.70:0/1149160726 >> 172.21.15.70:0/1149160726 conn(0x7f5ce406e5b0 msgr2=0x7f5ce40709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.233+0000 7f5ce94c2640 1 -- 172.21.15.70:0/1149160726 shutdown_connections 2024-04-01T01:40:27.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.233+0000 7f5ce94c2640 1 -- 172.21.15.70:0/1149160726 wait complete. 2024-04-01T01:40:27.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.234+0000 7f5ce94c2640 1 Processor -- start 2024-04-01T01:40:27.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.234+0000 7f5ce94c2640 1 -- start start 2024-04-01T01:40:27.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.234+0000 7f5ce94c2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ce4108750 0x7f5ce41a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.234+0000 7f5ce94c2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce41090b0 0x7f5ce41a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.234+0000 7f5ce94c2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ce410ffd0 0x7f5ce41a77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.234+0000 7f5ce94c2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce4115ac0 con 0x7f5ce4108750 2024-04-01T01:40:27.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.234+0000 7f5ce94c2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce4115940 con 0x7f5ce410ffd0 2024-04-01T01:40:27.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.234+0000 7f5ce94c2640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce4115c40 con 0x7f5ce41090b0 2024-04-01T01:40:27.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.235+0000 7f5ce2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ce4108750 0x7f5ce41a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.235+0000 7f5ce2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ce4108750 0x7f5ce41a0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47448/0 (socket says 172.21.15.70:47448) 2024-04-01T01:40:27.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.235+0000 7f5ce2ffd640 1 -- 172.21.15.70:0/12692426 learned_addr learned my addr 172.21.15.70:0/12692426 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:27.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.235+0000 7f5ce37fe640 1 --2- 172.21.15.70:0/12692426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ce410ffd0 0x7f5ce41a77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.235+0000 7f5ce2ffd640 1 -- 172.21.15.70:0/12692426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce41090b0 msgr2=0x7f5ce41a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.235+0000 7f5ce2ffd640 1 --2- 172.21.15.70:0/12692426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce41090b0 0x7f5ce41a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.235+0000 7f5ce2ffd640 1 -- 172.21.15.70:0/12692426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ce410ffd0 msgr2=0x7f5ce41a77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.235+0000 7f5ce2ffd640 1 --2- 172.21.15.70:0/12692426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ce410ffd0 0x7f5ce41a77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.235+0000 7f5ce2ffd640 1 -- 172.21.15.70:0/12692426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce41a7e30 con 0x7f5ce4108750 2024-04-01T01:40:27.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.235+0000 7f5ce2ffd640 1 --2- 172.21.15.70:0/12692426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ce4108750 0x7f5ce41a0210 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400e9f0 tx=0x7f5cd400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.236+0000 7f5ccbfff640 1 -- 172.21.15.70:0/12692426 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd400cd80 con 0x7f5ce4108750 2024-04-01T01:40:27.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.236+0000 7f5ccbfff640 1 -- 172.21.15.70:0/12692426 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cd40108e0 con 0x7f5ce4108750 2024-04-01T01:40:27.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.236+0000 7f5ccbfff640 1 -- 172.21.15.70:0/12692426 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4018920 con 0x7f5ce4108750 2024-04-01T01:40:27.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.236+0000 7f5ce94c2640 1 -- 172.21.15.70:0/12692426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce41a8120 con 0x7f5ce4108750 2024-04-01T01:40:27.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.237+0000 7f5ce94c2640 1 -- 172.21.15.70:0/12692426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ce41a8560 con 0x7f5ce4108750 2024-04-01T01:40:27.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.240+0000 7f5ccbfff640 1 -- 172.21.15.70:0/12692426 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f5cd4010a90 con 0x7f5ce4108750 2024-04-01T01:40:27.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.241+0000 7f5ccbfff640 1 --2- 172.21.15.70:0/12692426 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ccc0a26a0 0x7f5ccc0a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.241+0000 7f5ccbfff640 1 -- 172.21.15.70:0/12692426 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5cd40cede0 con 0x7f5ce4108750 2024-04-01T01:40:27.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.241+0000 7f5cec176740 1 Processor -- start 2024-04-01T01:40:27.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.241+0000 7f5ce27fc640 1 --2- 172.21.15.70:0/12692426 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ccc0a26a0 0x7f5ccc0a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.242+0000 7f5cec176740 1 -- start start 2024-04-01T01:40:27.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.242+0000 7f5cec176740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 0x56441679d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.242+0000 7f5cec176740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56441679de80 0x5644167a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.242+0000 7f5cec176740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644167a7400 0x5644167a9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.242+0000 7f5cec176740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5644167021b0 con 0x56441679de80 2024-04-01T01:40:27.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.242+0000 7f5cec176740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5644166aac00 con 0x56441679d490 2024-04-01T01:40:27.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.242+0000 7f5cec176740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5644166fe250 con 0x5644167a7400 2024-04-01T01:40:27.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.242+0000 7f5ce37fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644167a7400 0x5644167a9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.242+0000 7f5ce37fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644167a7400 0x5644167a9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37384/0 (socket says 172.21.15.70:37384) 2024-04-01T01:40:27.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.242+0000 7f5ce37fe640 1 -- 172.21.15.70:0/1121017024 learned_addr learned my addr 172.21.15.70:0/1121017024 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:27.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.242+0000 7f5ce27fc640 1 --2- 172.21.15.70:0/1121017024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56441679de80 0x5644167a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.242+0000 7f5ce27fc640 1 --2- 172.21.15.70:0/12692426 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ccc0a26a0 0x7f5ccc0a4b60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8005ab0 tx=0x7f5cd8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.243+0000 7f5ce37fe640 1 -- 172.21.15.70:0/1121017024 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 msgr2=0x56441679d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.243+0000 7f5ce37fe640 1 --2- 172.21.15.70:0/1121017024 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 0x56441679d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.243+0000 7f5ce37fe640 1 -- 172.21.15.70:0/1121017024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56441679de80 msgr2=0x5644167a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.243+0000 7f5ce37fe640 1 --2- 172.21.15.70:0/1121017024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56441679de80 0x5644167a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.243+0000 7f5ce37fe640 1 -- 172.21.15.70:0/1121017024 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644166ca3d0 con 0x5644167a7400 2024-04-01T01:40:27.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.243+0000 7f5ce37fe640 1 --2- 172.21.15.70:0/1121017024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644167a7400 0x5644167a9890 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc009f50 tx=0x7f5cdc0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6370e35602f8671b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.243+0000 7f5cc97fa640 1 -- 172.21.15.70:0/1121017024 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc004b20 con 0x5644167a7400 2024-04-01T01:40:27.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.243+0000 7f5cc97fa640 1 -- 172.21.15.70:0/1121017024 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cdc004cc0 con 0x5644167a7400 2024-04-01T01:40:27.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.243+0000 7f5cc97fa640 1 -- 172.21.15.70:0/1121017024 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc005020 con 0x5644167a7400 2024-04-01T01:40:27.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.244+0000 7f5cec176740 1 -- 172.21.15.70:0/1121017024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644167a7400 msgr2=0x5644167a9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.244+0000 7f5cec176740 1 --2- 172.21.15.70:0/1121017024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644167a7400 0x5644167a9890 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc009f50 tx=0x7f5cdc0118d0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.245+0000 7f5cec176740 1 -- 172.21.15.70:0/1121017024 shutdown_connections 2024-04-01T01:40:27.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.245+0000 7f5cec176740 1 --2- 172.21.15.70:0/1121017024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644167a7400 0x5644167a9890 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.245+0000 7f5cec176740 1 --2- 172.21.15.70:0/1121017024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56441679de80 0x5644167a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.245+0000 7f5cec176740 1 --2- 172.21.15.70:0/1121017024 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 0x56441679d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.245+0000 7f5cec176740 1 -- 172.21.15.70:0/1121017024 >> 172.21.15.70:0/1121017024 conn(0x564416798a00 msgr2=0x564416798e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.245+0000 7f5cec176740 1 -- 172.21.15.70:0/1121017024 shutdown_connections 2024-04-01T01:40:27.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.245+0000 7f5cec176740 1 -- 172.21.15.70:0/1121017024 wait complete. 2024-04-01T01:40:27.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.245+0000 7f5cec176740 1 -- start start 2024-04-01T01:40:27.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.245+0000 7f5cec176740 1 Processor -- start 2024-04-01T01:40:27.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.245+0000 7f5cec176740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 0x5644167a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.245+0000 7f5cec176740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56441679de80 0x5644167a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.245+0000 7f5cec176740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644167a7400 0x5644167a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.246+0000 7f5cec176740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5644167a3320 con 0x5644167a7400 2024-04-01T01:40:27.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.246+0000 7f5cec176740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5644166a0470 con 0x56441679d490 2024-04-01T01:40:27.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.246+0000 7f5cec176740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5644167a34a0 con 0x56441679de80 2024-04-01T01:40:27.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.246+0000 7f5ce2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 0x5644167a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.246+0000 7f5ce2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 0x5644167a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:58908/0 (socket says 172.21.15.70:58908) 2024-04-01T01:40:27.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.246+0000 7f5ce27fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56441679de80 0x5644167a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.246+0000 7f5ce2ffd640 1 -- 172.21.15.70:0/2205815286 learned_addr learned my addr 172.21.15.70:0/2205815286 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:27.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.246+0000 7f5cec176740 10 client.0 _ll_register_callbacks cb 0x5644166bdcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:27.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.246+0000 7f5ce37fe640 1 --2- 172.21.15.70:0/2205815286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644167a7400 0x5644167a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.246+0000 7f5ce27fc640 1 -- 172.21.15.70:0/2205815286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 msgr2=0x5644167a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.246+0000 7f5ce27fc640 1 --2- 172.21.15.70:0/2205815286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 0x5644167a08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.246+0000 7f5ce27fc640 1 -- 172.21.15.70:0/2205815286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644167a7400 msgr2=0x5644167a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.246+0000 7f5ce27fc640 1 --2- 172.21.15.70:0/2205815286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644167a7400 0x5644167a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.246+0000 7f5ce27fc640 1 -- 172.21.15.70:0/2205815286 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644164163f0 con 0x56441679de80 2024-04-01T01:40:27.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.246+0000 7f5ce27fc640 1 --2- 172.21.15.70:0/2205815286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56441679de80 0x5644167a0e20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009230 tx=0x7f5cd8009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f999408992782cc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.246+0000 7f5cc6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:40:27.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.247+0000 7f5cc6ff5640 1 -- 172.21.15.70:0/2205815286 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd804e070 con 0x56441679de80 2024-04-01T01:40:27.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.247+0000 7f5cc6ff5640 1 -- 172.21.15.70:0/2205815286 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5cd8003d40 con 0x56441679de80 2024-04-01T01:40:27.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.247+0000 7f5cec176740 1 -- 172.21.15.70:0/2205815286 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cdc004cc0 con 0x56441679de80 2024-04-01T01:40:27.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.247+0000 7f5cc6ff5640 1 -- 172.21.15.70:0/2205815286 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5cd805a050 con 0x56441679de80 2024-04-01T01:40:27.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.247+0000 7f5cec176740 10 client.27394 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:27.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.247+0000 7f5cec176740 1 -- 172.21.15.70:0/2205815286 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5644164e07d0 con 0x56441679de80 2024-04-01T01:40:27.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.247+0000 7f5cc6ff5640 1 -- 172.21.15.70:0/2205815286 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cdc004cc0 con 0x56441679de80 2024-04-01T01:40:27.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.247+0000 7f5cc6ff5640 1 -- 172.21.15.70:0/2205815286 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5cd80344d0 con 0x56441679de80 2024-04-01T01:40:27.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.247+0000 7f5cec176740 10 client.27394 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:27.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.247+0000 7f5cec176740 10 client.27394 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:40:27.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.248+0000 7f5cec176740 20 client.27394 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:27.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.248+0000 7f5cec176740 1 --2- 172.21.15.70:0/2205815286 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5644167bbcd0 0x5644167be0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.248+0000 7f5cec176740 4 client.27394 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:40:27.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.248+0000 7f5cec176740 1 -- 172.21.15.70:0/2205815286 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56441670f6f0 con 0x5644167bbcd0 2024-04-01T01:40:27.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.248+0000 7f5ce2ffd640 1 --2- 172.21.15.70:0/2205815286 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5644167bbcd0 0x5644167be0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.248+0000 7f5cc6ff5640 10 client.27394 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:40:27.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.248+0000 7f5ce2ffd640 1 --2- 172.21.15.70:0/2205815286 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5644167bbcd0 0x5644167be0f0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cfc24837f920edc9 server_cookie=9d14ce738dab7634 in_seq=0 out_seq=0 2024-04-01T01:40:27.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.249+0000 7f5cc6ff5640 1 -- 172.21.15.70:0/2205815286 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56441670f6f0 con 0x5644167bbcd0 2024-04-01T01:40:27.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.249+0000 7f5cc6ff5640 10 client.27394 handle_command_reply: tid=0 2024-04-01T01:40:27.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.249+0000 7f5cec176740 1 client.27394 shutdown 2024-04-01T01:40:27.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.250+0000 7f5cec176740 1 -- 172.21.15.70:0/2205815286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56441679de80 msgr2=0x5644167a0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.250+0000 7f5cec176740 1 --2- 172.21.15.70:0/2205815286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56441679de80 0x5644167a0e20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009230 tx=0x7f5cd8009260 comp rx=0 tx=0).stop 2024-04-01T01:40:27.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.250+0000 7f5cec176740 1 -- 172.21.15.70:0/2205815286 shutdown_connections 2024-04-01T01:40:27.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.250+0000 7f5cec176740 1 --2- 172.21.15.70:0/2205815286 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5644167bbcd0 0x5644167be0f0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.250+0000 7f5cec176740 1 --2- 172.21.15.70:0/2205815286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644167a7400 0x5644167a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.250+0000 7f5cec176740 1 --2- 172.21.15.70:0/2205815286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56441679de80 0x5644167a0e20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.250+0000 7f5cec176740 1 --2- 172.21.15.70:0/2205815286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 0x5644167a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.250+0000 7f5cec176740 1 -- 172.21.15.70:0/2205815286 >> 172.21.15.70:0/2205815286 conn(0x564416798a00 msgr2=0x564416797d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.250+0000 7f5cc6ff5640 0 client.27394 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:40:27.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.250+0000 7f5cec176740 1 -- 172.21.15.70:0/2205815286 shutdown_connections 2024-04-01T01:40:27.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.250+0000 7f5cec176740 1 -- 172.21.15.70:0/2205815286 wait complete. 2024-04-01T01:40:27.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.250+0000 7f5cec176740 20 client.27394 trim_cache size 0 max 16384 2024-04-01T01:40:27.255 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:26 smithi189 ceph-mon[35753]: pgmap v568: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 16 MiB/s wr, 1.13k op/s 2024-04-01T01:40:27.255 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:26 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/108732415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:40:27.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.271+0000 7f5cec176740 1 Processor -- start 2024-04-01T01:40:27.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.271+0000 7f5cec176740 1 -- start start 2024-04-01T01:40:27.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.271+0000 7f5cec176740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56441679d490 0x56441679ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.271+0000 7f5cec176740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679de80 0x56441679f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.271+0000 7f5cec176740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644167a7400 0x5644167974b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.271+0000 7f5cec176740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce41b0ad0 con 0x56441679d490 2024-04-01T01:40:27.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.271+0000 7f5cec176740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce41b2610 con 0x56441679de80 2024-04-01T01:40:27.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.271+0000 7f5cec176740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce41b11a0 con 0x5644167a7400 2024-04-01T01:40:27.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.272+0000 7f5ce27fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679de80 0x56441679f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.272+0000 7f5ce37fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644167a7400 0x5644167974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.272+0000 7f5ce27fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679de80 0x56441679f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:58914/0 (socket says 172.21.15.70:58914) 2024-04-01T01:40:27.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.272+0000 7f5ce27fc640 1 -- 172.21.15.70:0/2358408369 learned_addr learned my addr 172.21.15.70:0/2358408369 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:27.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.272+0000 7f5ce2ffd640 1 --2- 172.21.15.70:0/2358408369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56441679d490 0x56441679ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.272+0000 7f5ce27fc640 1 -- 172.21.15.70:0/2358408369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644167a7400 msgr2=0x5644167974b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.272+0000 7f5ce27fc640 1 --2- 172.21.15.70:0/2358408369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644167a7400 0x5644167974b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.272+0000 7f5ce27fc640 1 -- 172.21.15.70:0/2358408369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56441679d490 msgr2=0x56441679ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.272+0000 7f5ce27fc640 1 --2- 172.21.15.70:0/2358408369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56441679d490 0x56441679ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.272+0000 7f5ce27fc640 1 -- 172.21.15.70:0/2358408369 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56441679e260 con 0x56441679de80 2024-04-01T01:40:27.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.272+0000 7f5ce27fc640 1 --2- 172.21.15.70:0/2358408369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679de80 0x56441679f480 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5cd80348f0 tx=0x7f5cd8009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fc72a669a455906 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.273+0000 7f5cc8ff9640 1 -- 172.21.15.70:0/2358408369 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd804e040 con 0x56441679de80 2024-04-01T01:40:27.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.273+0000 7f5cc8ff9640 1 -- 172.21.15.70:0/2358408369 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cd8063070 con 0x56441679de80 2024-04-01T01:40:27.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.273+0000 7f5cc8ff9640 1 -- 172.21.15.70:0/2358408369 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd803f6d0 con 0x56441679de80 2024-04-01T01:40:27.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.274+0000 7f5cec176740 1 -- 172.21.15.70:0/2358408369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679de80 msgr2=0x56441679f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.274+0000 7f5cec176740 1 --2- 172.21.15.70:0/2358408369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679de80 0x56441679f480 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5cd80348f0 tx=0x7f5cd8009620 comp rx=0 tx=0).stop 2024-04-01T01:40:27.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.274+0000 7f5cec176740 1 -- 172.21.15.70:0/2358408369 shutdown_connections 2024-04-01T01:40:27.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.274+0000 7f5cec176740 1 --2- 172.21.15.70:0/2358408369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644167a7400 0x5644167974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.274+0000 7f5cec176740 1 --2- 172.21.15.70:0/2358408369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679de80 0x56441679f480 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.274+0000 7f5cec176740 1 --2- 172.21.15.70:0/2358408369 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56441679d490 0x56441679ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.274+0000 7f5cec176740 1 -- 172.21.15.70:0/2358408369 >> 172.21.15.70:0/2358408369 conn(0x564416798a00 msgr2=0x5644167a43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.274+0000 7f5cec176740 1 -- 172.21.15.70:0/2358408369 shutdown_connections 2024-04-01T01:40:27.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.274+0000 7f5cec176740 1 -- 172.21.15.70:0/2358408369 wait complete. 2024-04-01T01:40:27.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.274+0000 7f5cec176740 1 -- start start 2024-04-01T01:40:27.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.274+0000 7f5cec176740 1 Processor -- start 2024-04-01T01:40:27.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.275+0000 7f5cec176740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 0x5644167b5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.275+0000 7f5cec176740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56441679de80 0x5644167b6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.275+0000 7f5cec176740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644167a7400 0x5644167b7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.275+0000 7f5ce2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 0x5644167b5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.275+0000 7f5ce2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 0x5644167b5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:58918/0 (socket says 172.21.15.70:58918) 2024-04-01T01:40:27.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.275+0000 7f5cec176740 1 -- 172.21.15.70:0/2655300309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce41b1f40 con 0x5644167a7400 2024-04-01T01:40:27.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.275+0000 7f5ce2ffd640 1 -- 172.21.15.70:0/2655300309 learned_addr learned my addr 172.21.15.70:0/2655300309 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:27.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.275+0000 7f5cec176740 1 -- 172.21.15.70:0/2655300309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce41b1870 con 0x56441679d490 2024-04-01T01:40:27.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.275+0000 7f5cec176740 1 -- 172.21.15.70:0/2655300309 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce41aeab0 con 0x56441679de80 2024-04-01T01:40:27.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.275+0000 7f5cec176740 10 client.0 _ll_register_callbacks cb 0x7f5cd802cca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:27.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.275+0000 7f5ce37fe640 1 --2- 172.21.15.70:0/2655300309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644167a7400 0x5644167b7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.275+0000 7f5ce27fc640 1 --2- 172.21.15.70:0/2655300309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56441679de80 0x5644167b6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.275+0000 7f5ce2ffd640 1 -- 172.21.15.70:0/2655300309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56441679de80 msgr2=0x5644167b6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.276+0000 7f5ce2ffd640 1 --2- 172.21.15.70:0/2655300309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56441679de80 0x5644167b6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.276+0000 7f5ce2ffd640 1 -- 172.21.15.70:0/2655300309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644167a7400 msgr2=0x5644167b7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.276+0000 7f5ce2ffd640 1 --2- 172.21.15.70:0/2655300309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644167a7400 0x5644167b7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.276+0000 7f5ce2ffd640 1 -- 172.21.15.70:0/2655300309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644167b7600 con 0x56441679d490 2024-04-01T01:40:27.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.276+0000 7f5ce27fc640 1 --2- 172.21.15.70:0/2655300309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56441679de80 0x5644167b6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:40:27.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.276+0000 7f5ce37fe640 1 --2- 172.21.15.70:0/2655300309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644167a7400 0x5644167b7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:40:27.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.276+0000 7f5ce2ffd640 1 --2- 172.21.15.70:0/2655300309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 0x5644167b5ac0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4097e40 tx=0x7f5cd40bb040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c3801aa310bcdeb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.276+0000 7f5cc6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:40:27.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.277+0000 7f5cc6ff5640 1 -- 172.21.15.70:0/2655300309 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd40b4060 con 0x56441679d490 2024-04-01T01:40:27.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.277+0000 7f5cc6ff5640 1 -- 172.21.15.70:0/2655300309 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5cd4097ae0 con 0x56441679d490 2024-04-01T01:40:27.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.277+0000 7f5cec176740 1 -- 172.21.15.70:0/2655300309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cd8063070 con 0x56441679d490 2024-04-01T01:40:27.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.278+0000 7f5cc6ff5640 1 -- 172.21.15.70:0/2655300309 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5cd40b4200 con 0x56441679d490 2024-04-01T01:40:27.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.278+0000 7f5cec176740 10 client.27548 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:27.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.278+0000 7f5cec176740 1 -- 172.21.15.70:0/2655300309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564416821ff0 con 0x56441679d490 2024-04-01T01:40:27.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.278+0000 7f5cc6ff5640 1 -- 172.21.15.70:0/2655300309 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cd8063070 con 0x56441679d490 2024-04-01T01:40:27.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.278+0000 7f5cc6ff5640 1 -- 172.21.15.70:0/2655300309 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5cd40a39f0 con 0x56441679d490 2024-04-01T01:40:27.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.278+0000 7f5cec176740 10 client.27548 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:27.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.278+0000 7f5cec176740 10 client.27548 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:40:27.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.278+0000 7f5cec176740 20 client.27548 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:27.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.278+0000 7f5cec176740 1 --2- 172.21.15.70:0/2655300309 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x564416822910 0x564416824d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.278+0000 7f5cec176740 4 client.27548 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:40:27.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.278+0000 7f5cec176740 1 -- 172.21.15.70:0/2655300309 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564416709a40 con 0x564416822910 2024-04-01T01:40:27.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.278+0000 7f5ce27fc640 1 --2- 172.21.15.70:0/2655300309 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x564416822910 0x564416824d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.279+0000 7f5ce27fc640 1 --2- 172.21.15.70:0/2655300309 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x564416822910 0x564416824d30 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2f312b61cd6a6409 server_cookie=7ab7d6060baab5db in_seq=0 out_seq=0 2024-04-01T01:40:27.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.279+0000 7f5cc6ff5640 10 client.27548 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:40:27.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.279+0000 7f5cc6ff5640 1 -- 172.21.15.70:0/2655300309 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564416709a40 con 0x564416822910 2024-04-01T01:40:27.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.279+0000 7f5cc6ff5640 10 client.27548 handle_command_reply: tid=0 2024-04-01T01:40:27.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.279+0000 7f5cec176740 1 client.27548 shutdown 2024-04-01T01:40:27.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.280+0000 7f5cec176740 1 -- 172.21.15.70:0/2655300309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 msgr2=0x5644167b5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.280+0000 7f5cec176740 1 --2- 172.21.15.70:0/2655300309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 0x5644167b5ac0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4097e40 tx=0x7f5cd40bb040 comp rx=0 tx=0).stop 2024-04-01T01:40:27.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.280+0000 7f5cec176740 1 -- 172.21.15.70:0/2655300309 shutdown_connections 2024-04-01T01:40:27.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.280+0000 7f5cec176740 1 --2- 172.21.15.70:0/2655300309 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x564416822910 0x564416824d30 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.280+0000 7f5cc6ff5640 0 client.27548 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:40:27.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.280+0000 7f5cec176740 1 --2- 172.21.15.70:0/2655300309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644167a7400 0x5644167b7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.280+0000 7f5cec176740 1 --2- 172.21.15.70:0/2655300309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56441679de80 0x5644167b6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.280+0000 7f5cec176740 1 --2- 172.21.15.70:0/2655300309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56441679d490 0x5644167b5ac0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.280+0000 7f5cec176740 1 -- 172.21.15.70:0/2655300309 >> 172.21.15.70:0/2655300309 conn(0x564416798a00 msgr2=0x56441679ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.280+0000 7f5cec176740 1 -- 172.21.15.70:0/2655300309 shutdown_connections 2024-04-01T01:40:27.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.280+0000 7f5cec176740 1 -- 172.21.15.70:0/2655300309 wait complete. 2024-04-01T01:40:27.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.280+0000 7f5cec176740 20 client.27548 trim_cache size 0 max 16384 2024-04-01T01:40:27.285 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:40:27.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.281+0000 7f5ce94c2640 1 -- 172.21.15.70:0/12692426 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ccc0a26a0 msgr2=0x7f5ccc0a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.281+0000 7f5ce94c2640 1 --2- 172.21.15.70:0/12692426 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ccc0a26a0 0x7f5ccc0a4b60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8005ab0 tx=0x7f5cd8000bc0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.281+0000 7f5ce94c2640 1 -- 172.21.15.70:0/12692426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ce4108750 msgr2=0x7f5ce41a0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.281+0000 7f5ce94c2640 1 --2- 172.21.15.70:0/12692426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ce4108750 0x7f5ce41a0210 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400e9f0 tx=0x7f5cd400eeb0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.282+0000 7f5ce94c2640 1 -- 172.21.15.70:0/12692426 shutdown_connections 2024-04-01T01:40:27.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.282+0000 7f5ce94c2640 1 --2- 172.21.15.70:0/12692426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ce410ffd0 0x7f5ce41a77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.282+0000 7f5ce94c2640 1 --2- 172.21.15.70:0/12692426 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ccc0a26a0 0x7f5ccc0a4b60 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.282+0000 7f5ce94c2640 1 --2- 172.21.15.70:0/12692426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce41090b0 0x7f5ce41a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.282+0000 7f5ce94c2640 1 --2- 172.21.15.70:0/12692426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ce4108750 0x7f5ce41a0210 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.282+0000 7f5ce94c2640 1 -- 172.21.15.70:0/12692426 >> 172.21.15.70:0/12692426 conn(0x7f5ce406e5b0 msgr2=0x7f5ce4070320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.282+0000 7f5ce94c2640 1 -- 172.21.15.70:0/12692426 shutdown_connections 2024-04-01T01:40:27.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.282+0000 7f5ce94c2640 1 -- 172.21.15.70:0/12692426 wait complete. 2024-04-01T01:40:27.298 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:40:27.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:26 smithi081 ceph-mon[35142]: pgmap v568: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 16 MiB/s wr, 1.13k op/s 2024-04-01T01:40:27.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:26 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/108732415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:40:27.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.475+0000 7f5ef196c640 1 -- 172.21.15.70:0/191604642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5eec102240 msgr2=0x7f5eec108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.475+0000 7f5ef196c640 1 --2- 172.21.15.70:0/191604642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5eec102240 0x7f5eec108ad0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0009900 tx=0x7f5ee002bc60 comp rx=0 tx=0).stop 2024-04-01T01:40:27.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.476+0000 7f5ef196c640 1 -- 172.21.15.70:0/191604642 shutdown_connections 2024-04-01T01:40:27.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.476+0000 7f5ef196c640 1 --2- 172.21.15.70:0/191604642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5eec102240 0x7f5eec108ad0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.476+0000 7f5ef196c640 1 --2- 172.21.15.70:0/191604642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eec101880 0x7f5eec101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.476+0000 7f5ef196c640 1 --2- 172.21.15.70:0/191604642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5eec100690 0x7f5eec100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.476+0000 7f5ef196c640 1 -- 172.21.15.70:0/191604642 >> 172.21.15.70:0/191604642 conn(0x7f5eec0fbc00 msgr2=0x7f5eec0fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.477+0000 7f5ef196c640 1 -- 172.21.15.70:0/191604642 shutdown_connections 2024-04-01T01:40:27.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.477+0000 7f5ef196c640 1 -- 172.21.15.70:0/191604642 wait complete. 2024-04-01T01:40:27.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.478+0000 7f5ef196c640 1 Processor -- start 2024-04-01T01:40:27.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.479+0000 7f5ef196c640 1 -- start start 2024-04-01T01:40:27.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.480+0000 7f5ef196c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5eec100690 0x7f5eec196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.480+0000 7f5ef196c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5eec101880 0x7f5eec196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.480+0000 7f5ef196c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eec102240 0x7f5eec19dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.480+0000 7f5ef196c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec10c240 con 0x7f5eec101880 2024-04-01T01:40:27.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.480+0000 7f5eeb7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eec102240 0x7f5eec19dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.480+0000 7f5eeb7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eec102240 0x7f5eec19dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37422/0 (socket says 172.21.15.70:37422) 2024-04-01T01:40:27.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.480+0000 7f5eeb7fe640 1 -- 172.21.15.70:0/3148243929 learned_addr learned my addr 172.21.15.70:0/3148243929 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:27.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.480+0000 7f5eea7fc640 1 --2- 172.21.15.70:0/3148243929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5eec101880 0x7f5eec196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.480+0000 7f5ef196c640 1 -- 172.21.15.70:0/3148243929 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec10c0c0 con 0x7f5eec100690 2024-04-01T01:40:27.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.480+0000 7f5ef196c640 1 -- 172.21.15.70:0/3148243929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec10c3c0 con 0x7f5eec102240 2024-04-01T01:40:27.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.481+0000 7f5eeb7fe640 1 -- 172.21.15.70:0/3148243929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5eec100690 msgr2=0x7f5eec196930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:40:27.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.481+0000 7f5eeb7fe640 1 --2- 172.21.15.70:0/3148243929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5eec100690 0x7f5eec196930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.481+0000 7f5eeb7fe640 1 -- 172.21.15.70:0/3148243929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5eec101880 msgr2=0x7f5eec196e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.481+0000 7f5eeb7fe640 1 --2- 172.21.15.70:0/3148243929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5eec101880 0x7f5eec196e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.481+0000 7f5eeb7fe640 1 -- 172.21.15.70:0/3148243929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eec19e550 con 0x7f5eec102240 2024-04-01T01:40:27.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.482+0000 7f5eeb7fe640 1 --2- 172.21.15.70:0/3148243929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eec102240 0x7f5eec19dee0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0036040 tx=0x7f5ee0004aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.482+0000 7f5ef096a640 1 -- 172.21.15.70:0/3148243929 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee003e650 con 0x7f5eec102240 2024-04-01T01:40:27.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.483+0000 7f5ef096a640 1 -- 172.21.15.70:0/3148243929 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ee003eca0 con 0x7f5eec102240 2024-04-01T01:40:27.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.483+0000 7f5ef096a640 1 -- 172.21.15.70:0/3148243929 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee00035f0 con 0x7f5eec102240 2024-04-01T01:40:27.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.483+0000 7f5ef196c640 1 -- 172.21.15.70:0/3148243929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eec069a90 con 0x7f5eec102240 2024-04-01T01:40:27.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.483+0000 7f5ef196c640 1 -- 172.21.15.70:0/3148243929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eec069f50 con 0x7f5eec102240 2024-04-01T01:40:27.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.486+0000 7f5ef096a640 1 -- 172.21.15.70:0/3148243929 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f5ee003e7f0 con 0x7f5eec102240 2024-04-01T01:40:27.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.487+0000 7f5ef096a640 1 --2- 172.21.15.70:0/3148243929 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ed40a26a0 0x7f5ed40a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.488+0000 7f5ef096a640 1 -- 172.21.15.70:0/3148243929 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5ee00f4a50 con 0x7f5eec102240 2024-04-01T01:40:27.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.488+0000 7f5eeaffd640 1 --2- 172.21.15.70:0/3148243929 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ed40a26a0 0x7f5ed40a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.489+0000 7f5eeaffd640 1 --2- 172.21.15.70:0/3148243929 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ed40a26a0 0x7f5ed40a4b60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5edc0058c0 tx=0x7f5edc005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.489+0000 7f5ef462b740 1 Processor -- start 2024-04-01T01:40:27.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.489+0000 7f5ef462b740 1 -- start start 2024-04-01T01:40:27.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.490+0000 7f5ef462b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ff490 0x555d785ff8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.490+0000 7f5ef462b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ffe80 0x555d78604980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.490+0000 7f5ef462b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d78604ff0 0x555d78607480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.490+0000 7f5eeaffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ff490 0x555d785ff8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.490+0000 7f5eeaffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ff490 0x555d785ff8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37436/0 (socket says 172.21.15.70:37436) 2024-04-01T01:40:27.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.490+0000 7f5eea7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ffe80 0x555d78604980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.491+0000 7f5eea7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ffe80 0x555d78604980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:58938/0 (socket says 172.21.15.70:58938) 2024-04-01T01:40:27.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.491+0000 7f5eeaffd640 1 -- 172.21.15.70:0/470006979 learned_addr learned my addr 172.21.15.70:0/470006979 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:27.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.492+0000 7f5ef462b740 1 -- 172.21.15.70:0/470006979 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555d785641b0 con 0x555d78604ff0 2024-04-01T01:40:27.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.492+0000 7f5ef462b740 1 -- 172.21.15.70:0/470006979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555d7850cc00 con 0x555d785ffe80 2024-04-01T01:40:27.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.492+0000 7f5ef462b740 1 -- 172.21.15.70:0/470006979 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555d78560250 con 0x555d785ff490 2024-04-01T01:40:27.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.493+0000 7f5eea7fc640 1 -- 172.21.15.70:0/470006979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ff490 msgr2=0x555d785ff8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.493+0000 7f5eea7fc640 1 --2- 172.21.15.70:0/470006979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ff490 0x555d785ff8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.493+0000 7f5eea7fc640 1 -- 172.21.15.70:0/470006979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d78604ff0 msgr2=0x555d78607480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.493+0000 7f5eea7fc640 1 --2- 172.21.15.70:0/470006979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d78604ff0 0x555d78607480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.493+0000 7f5eea7fc640 1 -- 172.21.15.70:0/470006979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d7852c3d0 con 0x555d785ffe80 2024-04-01T01:40:27.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.494+0000 7f5eea7fc640 1 --2- 172.21.15.70:0/470006979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ffe80 0x555d78604980 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4007170 tx=0x7f5ee400b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=343c9bc21d3c958 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.494+0000 7f5ed1ffb640 1 -- 172.21.15.70:0/470006979 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee4008040 con 0x555d785ffe80 2024-04-01T01:40:27.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.495+0000 7f5ed1ffb640 1 -- 172.21.15.70:0/470006979 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ee400b0d0 con 0x555d785ffe80 2024-04-01T01:40:27.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.495+0000 7f5ed1ffb640 1 -- 172.21.15.70:0/470006979 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee4014640 con 0x555d785ffe80 2024-04-01T01:40:27.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.496+0000 7f5ef462b740 1 -- 172.21.15.70:0/470006979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ffe80 msgr2=0x555d78604980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.496+0000 7f5ef462b740 1 --2- 172.21.15.70:0/470006979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ffe80 0x555d78604980 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4007170 tx=0x7f5ee400b8e0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.496+0000 7f5ef462b740 1 -- 172.21.15.70:0/470006979 shutdown_connections 2024-04-01T01:40:27.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.496+0000 7f5ef462b740 1 --2- 172.21.15.70:0/470006979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d78604ff0 0x555d78607480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.496+0000 7f5ef462b740 1 --2- 172.21.15.70:0/470006979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ffe80 0x555d78604980 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.496+0000 7f5ef462b740 1 --2- 172.21.15.70:0/470006979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ff490 0x555d785ff8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.496+0000 7f5ef462b740 1 -- 172.21.15.70:0/470006979 >> 172.21.15.70:0/470006979 conn(0x555d785faa00 msgr2=0x555d785fae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.496+0000 7f5ef462b740 1 -- 172.21.15.70:0/470006979 shutdown_connections 2024-04-01T01:40:27.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.496+0000 7f5ef462b740 1 -- 172.21.15.70:0/470006979 wait complete. 2024-04-01T01:40:27.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.497+0000 7f5ef462b740 1 -- start start 2024-04-01T01:40:27.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.498+0000 7f5ef462b740 1 Processor -- start 2024-04-01T01:40:27.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.498+0000 7f5ef462b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ff490 0x555d786099a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.498+0000 7f5ef462b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ffe80 0x555d78609ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.498+0000 7f5eeaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ff490 0x555d786099a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.499+0000 7f5eeaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ff490 0x555d786099a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:58948/0 (socket says 172.21.15.70:58948) 2024-04-01T01:40:27.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.499+0000 7f5eeaffd640 1 -- 172.21.15.70:0/514356259 learned_addr learned my addr 172.21.15.70:0/514356259 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:27.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.499+0000 7f5ef462b740 1 --2- 172.21.15.70:0/514356259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d78604ff0 0x555d78610f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.499+0000 7f5eea7fc640 1 --2- 172.21.15.70:0/514356259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ffe80 0x555d78609ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.500+0000 7f5eeb7fe640 1 --2- 172.21.15.70:0/514356259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d78604ff0 0x555d78610f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.500+0000 7f5ef462b740 1 -- 172.21.15.70:0/514356259 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555d78607f40 con 0x555d78604ff0 2024-04-01T01:40:27.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.500+0000 7f5ef462b740 1 -- 172.21.15.70:0/514356259 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555d78502470 con 0x555d785ff490 2024-04-01T01:40:27.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.500+0000 7f5ef462b740 1 -- 172.21.15.70:0/514356259 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555d786080c0 con 0x555d785ffe80 2024-04-01T01:40:27.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.500+0000 7f5ef462b740 10 client.0 _ll_register_callbacks cb 0x555d7851fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.501+0000 7f5eea7fc640 1 -- 172.21.15.70:0/514356259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ff490 msgr2=0x555d786099a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.501+0000 7f5eea7fc640 1 --2- 172.21.15.70:0/514356259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ff490 0x555d786099a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.501+0000 7f5eea7fc640 1 -- 172.21.15.70:0/514356259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d78604ff0 msgr2=0x555d78610f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.501+0000 7f5eea7fc640 1 --2- 172.21.15.70:0/514356259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d78604ff0 0x555d78610f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.501+0000 7f5eea7fc640 1 -- 172.21.15.70:0/514356259 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d782783f0 con 0x555d785ffe80 2024-04-01T01:40:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.501+0000 7f5eeb7fe640 1 --2- 172.21.15.70:0/514356259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d78604ff0 0x555d78610f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:40:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.501+0000 7f5eeaffd640 1 --2- 172.21.15.70:0/514356259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ff490 0x555d786099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:40:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.501+0000 7f5eea7fc640 1 --2- 172.21.15.70:0/514356259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ffe80 0x555d78609ee0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x555d785fb970 tx=0x7f5ee400b280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fe221246e54ab00 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.501+0000 7f5ecaffd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:40:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.502+0000 7f5ecaffd640 1 -- 172.21.15.70:0/514356259 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee4009990 con 0x555d785ffe80 2024-04-01T01:40:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.502+0000 7f5ecaffd640 1 -- 172.21.15.70:0/514356259 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5ee401e420 con 0x555d785ffe80 2024-04-01T01:40:27.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.503+0000 7f5ef462b740 1 -- 172.21.15.70:0/514356259 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ee400b0d0 con 0x555d785ffe80 2024-04-01T01:40:27.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.505+0000 7f5ecaffd640 1 -- 172.21.15.70:0/514356259 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5ee401e640 con 0x555d785ffe80 2024-04-01T01:40:27.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.505+0000 7f5ef462b740 10 client.27406 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:27.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.505+0000 7f5ef462b740 1 -- 172.21.15.70:0/514356259 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555d783427d0 con 0x555d785ffe80 2024-04-01T01:40:27.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.506+0000 7f5ecaffd640 1 -- 172.21.15.70:0/514356259 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ee401ec90 con 0x555d785ffe80 2024-04-01T01:40:27.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.506+0000 7f5ecaffd640 1 -- 172.21.15.70:0/514356259 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5ee4030a00 con 0x555d785ffe80 2024-04-01T01:40:27.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.507+0000 7f5ef462b740 10 client.27406 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:27.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.507+0000 7f5ef462b740 10 client.27406 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:40:27.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.507+0000 7f5ef462b740 20 client.27406 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:27.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.507+0000 7f5ef462b740 1 --2- 172.21.15.70:0/514356259 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x555d786150c0 0x555d786174b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.507+0000 7f5ef462b740 4 client.27406 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:40:27.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.507+0000 7f5ef462b740 1 -- 172.21.15.70:0/514356259 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5ee400b0d0 con 0x555d786150c0 2024-04-01T01:40:27.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.507+0000 7f5eeaffd640 1 --2- 172.21.15.70:0/514356259 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x555d786150c0 0x555d786174b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.508+0000 7f5eeaffd640 1 --2- 172.21.15.70:0/514356259 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x555d786150c0 0x555d786174b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5993b1ec78bed658 server_cookie=c093b94942874020 in_seq=0 out_seq=0 2024-04-01T01:40:27.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.508+0000 7f5ecaffd640 10 client.27406 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:40:27.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.509+0000 7f5ecaffd640 1 -- 172.21.15.70:0/514356259 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f5ee400b0d0 con 0x555d786150c0 2024-04-01T01:40:27.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.509+0000 7f5ecaffd640 10 client.27406 handle_command_reply: tid=0 2024-04-01T01:40:27.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.509+0000 7f5ef462b740 1 client.27406 shutdown 2024-04-01T01:40:27.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.510+0000 7f5ef462b740 1 -- 172.21.15.70:0/514356259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ffe80 msgr2=0x555d78609ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.510+0000 7f5ef462b740 1 --2- 172.21.15.70:0/514356259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ffe80 0x555d78609ee0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x555d785fb970 tx=0x7f5ee400b280 comp rx=0 tx=0).stop 2024-04-01T01:40:27.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.512+0000 7f5ef462b740 1 -- 172.21.15.70:0/514356259 shutdown_connections 2024-04-01T01:40:27.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.512+0000 7f5ef462b740 1 --2- 172.21.15.70:0/514356259 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x555d786150c0 0x555d786174b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.512+0000 7f5ef462b740 1 --2- 172.21.15.70:0/514356259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d78604ff0 0x555d78610f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.512+0000 7f5ecaffd640 0 client.27406 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:40:27.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.512+0000 7f5ef462b740 1 --2- 172.21.15.70:0/514356259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ffe80 0x555d78609ee0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.512+0000 7f5ef462b740 1 --2- 172.21.15.70:0/514356259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ff490 0x555d786099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.512+0000 7f5ef462b740 1 -- 172.21.15.70:0/514356259 >> 172.21.15.70:0/514356259 conn(0x555d785faa00 msgr2=0x555d785f9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.512+0000 7f5ef462b740 1 -- 172.21.15.70:0/514356259 shutdown_connections 2024-04-01T01:40:27.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.513+0000 7f5ef462b740 1 -- 172.21.15.70:0/514356259 wait complete. 2024-04-01T01:40:27.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.513+0000 7f5ef462b740 20 client.27406 trim_cache size 0 max 16384 2024-04-01T01:40:27.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.538+0000 7f5ef462b740 1 Processor -- start 2024-04-01T01:40:27.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.538+0000 7f5ef462b740 1 -- start start 2024-04-01T01:40:27.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.539+0000 7f5ef462b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785faa00 0x555d785fe370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.539+0000 7f5ef462b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d785ff490 0x555d78610ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.539+0000 7f5ef462b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ffe80 0x555d7861bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.539+0000 7f5ef462b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec1a8210 con 0x555d785ff490 2024-04-01T01:40:27.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.539+0000 7f5ef462b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec1a9d50 con 0x555d785ffe80 2024-04-01T01:40:27.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.539+0000 7f5ef462b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec1a88e0 con 0x555d785faa00 2024-04-01T01:40:27.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.539+0000 7f5eeb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ffe80 0x555d7861bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.539+0000 7f5eeb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ffe80 0x555d7861bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:58950/0 (socket says 172.21.15.70:58950) 2024-04-01T01:40:27.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.539+0000 7f5eeb7fe640 1 -- 172.21.15.70:0/1520600087 learned_addr learned my addr 172.21.15.70:0/1520600087 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:27.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.539+0000 7f5eeaffd640 1 --2- 172.21.15.70:0/1520600087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785faa00 0x555d785fe370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.539+0000 7f5eeb7fe640 1 -- 172.21.15.70:0/1520600087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785faa00 msgr2=0x555d785fe370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.539+0000 7f5eeb7fe640 1 --2- 172.21.15.70:0/1520600087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785faa00 0x555d785fe370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.539+0000 7f5eeb7fe640 1 -- 172.21.15.70:0/1520600087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d785ff490 msgr2=0x555d78610ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.539+0000 7f5eeb7fe640 1 --2- 172.21.15.70:0/1520600087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d785ff490 0x555d78610ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.539+0000 7f5eeb7fe640 1 -- 172.21.15.70:0/1520600087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d78608240 con 0x555d785ffe80 2024-04-01T01:40:27.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.540+0000 7f5eeaffd640 1 --2- 172.21.15.70:0/1520600087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785faa00 0x555d785fe370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:40:27.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.540+0000 7f5eeb7fe640 1 --2- 172.21.15.70:0/1520600087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ffe80 0x555d7861bcb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00f2120 tx=0x7f5ee0009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0a335ad73a7bcf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.540+0000 7f5ed17fa640 1 -- 172.21.15.70:0/1520600087 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee003ee50 con 0x555d785ffe80 2024-04-01T01:40:27.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.540+0000 7f5ed17fa640 1 -- 172.21.15.70:0/1520600087 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ee003eae0 con 0x555d785ffe80 2024-04-01T01:40:27.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.540+0000 7f5ed17fa640 1 -- 172.21.15.70:0/1520600087 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee00bd750 con 0x555d785ffe80 2024-04-01T01:40:27.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.541+0000 7f5ef462b740 1 -- 172.21.15.70:0/1520600087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ffe80 msgr2=0x555d7861bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.541+0000 7f5ef462b740 1 --2- 172.21.15.70:0/1520600087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ffe80 0x555d7861bcb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00f2120 tx=0x7f5ee0009680 comp rx=0 tx=0).stop 2024-04-01T01:40:27.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.542+0000 7f5ef462b740 1 -- 172.21.15.70:0/1520600087 shutdown_connections 2024-04-01T01:40:27.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.542+0000 7f5ef462b740 1 --2- 172.21.15.70:0/1520600087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785ffe80 0x555d7861bcb0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.542+0000 7f5ef462b740 1 --2- 172.21.15.70:0/1520600087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d785ff490 0x555d78610ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.542+0000 7f5ef462b740 1 --2- 172.21.15.70:0/1520600087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785faa00 0x555d785fe370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.542+0000 7f5ef462b740 1 -- 172.21.15.70:0/1520600087 >> 172.21.15.70:0/1520600087 conn(0x555d78617820 msgr2=0x555d785fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.542+0000 7f5ef462b740 1 -- 172.21.15.70:0/1520600087 shutdown_connections 2024-04-01T01:40:27.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.542+0000 7f5ef462b740 1 -- 172.21.15.70:0/1520600087 wait complete. 2024-04-01T01:40:27.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.543+0000 7f5ef462b740 1 -- start start 2024-04-01T01:40:27.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.543+0000 7f5ef462b740 1 Processor -- start 2024-04-01T01:40:27.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.543+0000 7f5ef462b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785faa00 0x555d7860ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.543+0000 7f5ef462b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ff490 0x555d7860f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.543+0000 7f5ef462b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d785ffe80 0x555d78610450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.543+0000 7f5ef462b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec1a9680 con 0x555d785ffe80 2024-04-01T01:40:27.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.543+0000 7f5ef462b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec1a8fb0 con 0x555d785faa00 2024-04-01T01:40:27.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.543+0000 7f5ef462b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5eec1a61f0 con 0x555d785ff490 2024-04-01T01:40:27.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.543+0000 7f5ef462b740 10 client.0 _ll_register_callbacks cb 0x7f5ee400bd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:27.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.544+0000 7f5eea7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ff490 0x555d7860f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.544+0000 7f5eeb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d785ffe80 0x555d78610450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.544+0000 7f5eea7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ff490 0x555d7860f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37472/0 (socket says 172.21.15.70:37472) 2024-04-01T01:40:27.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.544+0000 7f5eea7fc640 1 -- 172.21.15.70:0/3007796930 learned_addr learned my addr 172.21.15.70:0/3007796930 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:27.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.544+0000 7f5eeaffd640 1 --2- 172.21.15.70:0/3007796930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785faa00 0x555d7860ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.544+0000 7f5eeb7fe640 1 -- 172.21.15.70:0/3007796930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ff490 msgr2=0x555d7860f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.544+0000 7f5eeb7fe640 1 --2- 172.21.15.70:0/3007796930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ff490 0x555d7860f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.544+0000 7f5eeb7fe640 1 -- 172.21.15.70:0/3007796930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785faa00 msgr2=0x555d7860ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.544+0000 7f5eeb7fe640 1 --2- 172.21.15.70:0/3007796930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785faa00 0x555d7860ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.544+0000 7f5eeb7fe640 1 -- 172.21.15.70:0/3007796930 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d78610a40 con 0x555d785ffe80 2024-04-01T01:40:27.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.545+0000 7f5eeb7fe640 1 --2- 172.21.15.70:0/3007796930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d785ffe80 0x555d78610450 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00f2120 tx=0x7f5ee00bf5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934973992966e37 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.545+0000 7f5ecaffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:40:27.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.545+0000 7f5ecaffd640 1 -- 172.21.15.70:0/3007796930 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee00ce060 con 0x555d785ffe80 2024-04-01T01:40:27.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.545+0000 7f5ecaffd640 1 -- 172.21.15.70:0/3007796930 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5ee00c9860 con 0x555d785ffe80 2024-04-01T01:40:27.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.545+0000 7f5ef462b740 1 -- 172.21.15.70:0/3007796930 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555d785716f0 con 0x555d785ffe80 2024-04-01T01:40:27.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.546+0000 7f5ecaffd640 1 -- 172.21.15.70:0/3007796930 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5ee00c10e0 con 0x555d785ffe80 2024-04-01T01:40:27.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.547+0000 7f5ef462b740 10 client.17637 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:27.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.547+0000 7f5ef462b740 1 -- 172.21.15.70:0/3007796930 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555d7867bb10 con 0x555d785ffe80 2024-04-01T01:40:27.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.548+0000 7f5ecaffd640 1 -- 172.21.15.70:0/3007796930 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x555d785716f0 con 0x555d785ffe80 2024-04-01T01:40:27.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.548+0000 7f5ecaffd640 1 -- 172.21.15.70:0/3007796930 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5ee0051030 con 0x555d785ffe80 2024-04-01T01:40:27.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.549+0000 7f5ef462b740 10 client.17637 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:27.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.549+0000 7f5ef462b740 10 client.17637 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:40:27.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.549+0000 7f5ef462b740 20 client.17637 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:27.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.549+0000 7f5ef462b740 1 --2- 172.21.15.70:0/3007796930 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x555d78604ff0 0x555d7867e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.549+0000 7f5ef462b740 4 client.17637 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:40:27.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.549+0000 7f5ef462b740 1 -- 172.21.15.70:0/3007796930 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555d7856ba40 con 0x555d78604ff0 2024-04-01T01:40:27.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.550+0000 7f5eeaffd640 1 --2- 172.21.15.70:0/3007796930 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x555d78604ff0 0x555d7867e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.551+0000 7f5ecaffd640 10 client.17637 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:40:27.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.551+0000 7f5eeaffd640 1 --2- 172.21.15.70:0/3007796930 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x555d78604ff0 0x555d7867e360 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a9919a1505f20432 server_cookie=85fd9d5502d44e8e in_seq=0 out_seq=0 2024-04-01T01:40:27.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.552+0000 7f5ecaffd640 1 -- 172.21.15.70:0/3007796930 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555d7856ba40 con 0x555d78604ff0 2024-04-01T01:40:27.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.552+0000 7f5ecaffd640 10 client.17637 handle_command_reply: tid=0 2024-04-01T01:40:27.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.552+0000 7f5ef462b740 1 client.17637 shutdown 2024-04-01T01:40:27.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.552+0000 7f5ef462b740 1 -- 172.21.15.70:0/3007796930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d785ffe80 msgr2=0x555d78610450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.552+0000 7f5ef462b740 1 --2- 172.21.15.70:0/3007796930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d785ffe80 0x555d78610450 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00f2120 tx=0x7f5ee00bf5f0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.553+0000 7f5ef462b740 1 -- 172.21.15.70:0/3007796930 shutdown_connections 2024-04-01T01:40:27.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.553+0000 7f5ef462b740 1 --2- 172.21.15.70:0/3007796930 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x555d78604ff0 0x555d7867e360 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.553+0000 7f5ef462b740 1 --2- 172.21.15.70:0/3007796930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555d785ffe80 0x555d78610450 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.553+0000 7f5ef462b740 1 --2- 172.21.15.70:0/3007796930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555d785ff490 0x555d7860f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.553+0000 7f5ecaffd640 0 client.17637 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:40:27.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.553+0000 7f5ef462b740 1 --2- 172.21.15.70:0/3007796930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d785faa00 0x555d7860ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.553+0000 7f5ef462b740 1 -- 172.21.15.70:0/3007796930 >> 172.21.15.70:0/3007796930 conn(0x555d78617820 msgr2=0x555d785fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.555+0000 7f5ef462b740 1 -- 172.21.15.70:0/3007796930 shutdown_connections 2024-04-01T01:40:27.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.556+0000 7f5ef462b740 1 -- 172.21.15.70:0/3007796930 wait complete. 2024-04-01T01:40:27.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.556+0000 7f5ef462b740 20 client.17637 trim_cache size 0 max 16384 2024-04-01T01:40:27.555 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:40:27.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.557+0000 7f5ef196c640 1 -- 172.21.15.70:0/3148243929 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ed40a26a0 msgr2=0x7f5ed40a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.557+0000 7f5ef196c640 1 --2- 172.21.15.70:0/3148243929 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ed40a26a0 0x7f5ed40a4b60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5edc0058c0 tx=0x7f5edc005850 comp rx=0 tx=0).stop 2024-04-01T01:40:27.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.557+0000 7f5ef196c640 1 -- 172.21.15.70:0/3148243929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eec102240 msgr2=0x7f5eec19dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.557+0000 7f5ef196c640 1 --2- 172.21.15.70:0/3148243929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eec102240 0x7f5eec19dee0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0036040 tx=0x7f5ee0004aa0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.557+0000 7f5ef196c640 1 -- 172.21.15.70:0/3148243929 shutdown_connections 2024-04-01T01:40:27.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.557+0000 7f5ef196c640 1 --2- 172.21.15.70:0/3148243929 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ed40a26a0 0x7f5ed40a4b60 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.557+0000 7f5ef196c640 1 --2- 172.21.15.70:0/3148243929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eec102240 0x7f5eec19dee0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.557+0000 7f5ef196c640 1 --2- 172.21.15.70:0/3148243929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5eec101880 0x7f5eec196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.557+0000 7f5ef196c640 1 --2- 172.21.15.70:0/3148243929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5eec100690 0x7f5eec196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.558+0000 7f5ef196c640 1 -- 172.21.15.70:0/3148243929 >> 172.21.15.70:0/3148243929 conn(0x7f5eec0fbc00 msgr2=0x7f5eec104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.558+0000 7f5ef196c640 1 -- 172.21.15.70:0/3148243929 shutdown_connections 2024-04-01T01:40:27.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.558+0000 7f5ef196c640 1 -- 172.21.15.70:0/3148243929 wait complete. 2024-04-01T01:40:27.576 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:40:27.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.763+0000 7fdc8981c640 1 -- 172.21.15.70:0/708336512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc84067cb0 msgr2=0x7fdc84100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.763+0000 7fdc8981c640 1 --2- 172.21.15.70:0/708336512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc84067cb0 0x7fdc84100b90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdc74009ca0 tx=0x7fdc7402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.763+0000 7fdc8981c640 1 -- 172.21.15.70:0/708336512 shutdown_connections 2024-04-01T01:40:27.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.763+0000 7fdc8981c640 1 --2- 172.21.15.70:0/708336512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc841010d0 0x7fdc84101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.763+0000 7fdc8981c640 1 --2- 172.21.15.70:0/708336512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc84067cb0 0x7fdc84100b90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.763+0000 7fdc8981c640 1 --2- 172.21.15.70:0/708336512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc840ff9a0 0x7fdc840ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.763+0000 7fdc8981c640 1 -- 172.21.15.70:0/708336512 >> 172.21.15.70:0/708336512 conn(0x7fdc840faf10 msgr2=0x7fdc840fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.763+0000 7fdc8981c640 1 -- 172.21.15.70:0/708336512 shutdown_connections 2024-04-01T01:40:27.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.763+0000 7fdc8981c640 1 -- 172.21.15.70:0/708336512 wait complete. 2024-04-01T01:40:27.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.765+0000 7fdc8981c640 1 Processor -- start 2024-04-01T01:40:27.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.765+0000 7fdc8981c640 1 -- start start 2024-04-01T01:40:27.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.765+0000 7fdc8981c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc84067cb0 0x7fdc840701d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.765+0000 7fdc8981c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc840ff9a0 0x7fdc84070710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.766+0000 7fdc8981c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc841010d0 0x7fdc841a57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.766+0000 7fdc827fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc840ff9a0 0x7fdc84070710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.766+0000 7fdc827fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc840ff9a0 0x7fdc84070710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37500/0 (socket says 172.21.15.70:37500) 2024-04-01T01:40:27.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.766+0000 7fdc827fc640 1 -- 172.21.15.70:0/1227970068 learned_addr learned my addr 172.21.15.70:0/1227970068 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:27.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.766+0000 7fdc8981c640 1 -- 172.21.15.70:0/1227970068 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8410b120 con 0x7fdc841010d0 2024-04-01T01:40:27.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.766+0000 7fdc8981c640 1 -- 172.21.15.70:0/1227970068 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8410afa0 con 0x7fdc84067cb0 2024-04-01T01:40:27.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.766+0000 7fdc8981c640 1 -- 172.21.15.70:0/1227970068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdc8410b2a0 con 0x7fdc840ff9a0 2024-04-01T01:40:27.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.767+0000 7fdc837fe640 1 --2- 172.21.15.70:0/1227970068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc841010d0 0x7fdc841a57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.767+0000 7fdc827fc640 1 -- 172.21.15.70:0/1227970068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc84067cb0 msgr2=0x7fdc840701d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:40:27.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.767+0000 7fdc827fc640 1 --2- 172.21.15.70:0/1227970068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc84067cb0 0x7fdc840701d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.767+0000 7fdc827fc640 1 -- 172.21.15.70:0/1227970068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc841010d0 msgr2=0x7fdc841a57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.767+0000 7fdc827fc640 1 --2- 172.21.15.70:0/1227970068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc841010d0 0x7fdc841a57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.767+0000 7fdc827fc640 1 -- 172.21.15.70:0/1227970068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc841a5d30 con 0x7fdc840ff9a0 2024-04-01T01:40:27.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.767+0000 7fdc827fc640 1 --2- 172.21.15.70:0/1227970068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc840ff9a0 0x7fdc84070710 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdc74032380 tx=0x7fdc74032820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.768+0000 7fdc8881a640 1 -- 172.21.15.70:0/1227970068 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc74041020 con 0x7fdc840ff9a0 2024-04-01T01:40:27.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.768+0000 7fdc8881a640 1 -- 172.21.15.70:0/1227970068 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc74003e20 con 0x7fdc840ff9a0 2024-04-01T01:40:27.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.768+0000 7fdc8881a640 1 -- 172.21.15.70:0/1227970068 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc74039040 con 0x7fdc840ff9a0 2024-04-01T01:40:27.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.769+0000 7fdc8981c640 1 -- 172.21.15.70:0/1227970068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc841a5f00 con 0x7fdc840ff9a0 2024-04-01T01:40:27.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.769+0000 7fdc8981c640 1 -- 172.21.15.70:0/1227970068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc841a6310 con 0x7fdc840ff9a0 2024-04-01T01:40:27.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.772+0000 7fdc8c3fe740 1 Processor -- start 2024-04-01T01:40:27.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.772+0000 7fdc8c3fe740 1 -- start start 2024-04-01T01:40:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.772+0000 7fdc8c3fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a365b490 0x5602a365b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.772+0000 7fdc8c3fe740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a365be80 0x5602a3660980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.772+0000 7fdc8c3fe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a3660ff0 0x5602a3663480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.772+0000 7fdc8c3fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5602a35c01b0 con 0x5602a365b490 2024-04-01T01:40:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.772+0000 7fdc8c3fe740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5602a3568c00 con 0x5602a365be80 2024-04-01T01:40:27.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.772+0000 7fdc8c3fe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5602a35bc250 con 0x5602a3660ff0 2024-04-01T01:40:27.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.773+0000 7fdc8881a640 1 -- 172.21.15.70:0/1227970068 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fdc74044020 con 0x7fdc840ff9a0 2024-04-01T01:40:27.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.773+0000 7fdc82ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a365be80 0x5602a3660980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.773+0000 7fdc837fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a365b490 0x5602a365b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.773+0000 7fdc82ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a365be80 0x5602a3660980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:58986/0 (socket says 172.21.15.70:58986) 2024-04-01T01:40:27.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.773+0000 7fdc82ffd640 1 -- 172.21.15.70:0/2043604906 learned_addr learned my addr 172.21.15.70:0/2043604906 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:27.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.773+0000 7fdc827fc640 1 --2- 172.21.15.70:0/2043604906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a3660ff0 0x5602a3663480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.773+0000 7fdc837fe640 1 -- 172.21.15.70:0/2043604906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a3660ff0 msgr2=0x5602a3663480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.773+0000 7fdc837fe640 1 --2- 172.21.15.70:0/2043604906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a3660ff0 0x5602a3663480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.773+0000 7fdc837fe640 1 -- 172.21.15.70:0/2043604906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a365be80 msgr2=0x5602a3660980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.773+0000 7fdc837fe640 1 --2- 172.21.15.70:0/2043604906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a365be80 0x5602a3660980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.773+0000 7fdc837fe640 1 -- 172.21.15.70:0/2043604906 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602a35883d0 con 0x5602a365b490 2024-04-01T01:40:27.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.774+0000 7fdc837fe640 1 --2- 172.21.15.70:0/2043604906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a365b490 0x5602a365b8b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c009af0 tx=0x7fdc7c0113e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0df9c44edb33af server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.774+0000 7fdc8881a640 1 --2- 172.21.15.70:0/1227970068 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdc6c0a6a30 0x7fdc6c0a8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.774+0000 7fdc8881a640 1 -- 172.21.15.70:0/1227970068 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fdc740f0a90 con 0x7fdc840ff9a0 2024-04-01T01:40:27.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.774+0000 7fdc69ffb640 1 -- 172.21.15.70:0/2043604906 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c011e10 con 0x5602a365b490 2024-04-01T01:40:27.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.774+0000 7fdc82ffd640 1 --2- 172.21.15.70:0/1227970068 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdc6c0a6a30 0x7fdc6c0a8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.774+0000 7fdc69ffb640 1 -- 172.21.15.70:0/2043604906 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc7c012900 con 0x5602a365b490 2024-04-01T01:40:27.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.774+0000 7fdc69ffb640 1 -- 172.21.15.70:0/2043604906 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c00fa80 con 0x5602a365b490 2024-04-01T01:40:27.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.776+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/2043604906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a365b490 msgr2=0x5602a365b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.776+0000 7fdc82ffd640 1 --2- 172.21.15.70:0/1227970068 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdc6c0a6a30 0x7fdc6c0a8ef0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x5602a365c280 tx=0x7fdc7800d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.776+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/2043604906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a365b490 0x5602a365b8b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c009af0 tx=0x7fdc7c0113e0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.776+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/2043604906 shutdown_connections 2024-04-01T01:40:27.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.776+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/2043604906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a3660ff0 0x5602a3663480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.776+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/2043604906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a365be80 0x5602a3660980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.776+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/2043604906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a365b490 0x5602a365b8b0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.776+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/2043604906 >> 172.21.15.70:0/2043604906 conn(0x5602a3656a00 msgr2=0x5602a3656e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.776+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/2043604906 shutdown_connections 2024-04-01T01:40:27.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.776+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/2043604906 wait complete. 2024-04-01T01:40:27.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.777+0000 7fdc8c3fe740 1 -- start start 2024-04-01T01:40:27.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.777+0000 7fdc8c3fe740 1 Processor -- start 2024-04-01T01:40:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.778+0000 7fdc8c3fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a365b490 0x5602a3665a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.778+0000 7fdc8c3fe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a365be80 0x5602a3665f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.778+0000 7fdc8c3fe740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a3660ff0 0x5602a366cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.778+0000 7fdc8c3fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5602a3663fa0 con 0x5602a365b490 2024-04-01T01:40:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.778+0000 7fdc8c3fe740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5602a355e470 con 0x5602a3660ff0 2024-04-01T01:40:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.778+0000 7fdc8c3fe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5602a3664120 con 0x5602a365be80 2024-04-01T01:40:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.778+0000 7fdc837fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a3660ff0 0x5602a366cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.778+0000 7fdc837fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a3660ff0 0x5602a366cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:58996/0 (socket says 172.21.15.70:58996) 2024-04-01T01:40:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.778+0000 7fdc837fe640 1 -- 172.21.15.70:0/2417684448 learned_addr learned my addr 172.21.15.70:0/2417684448 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.778+0000 7fdc8c3fe740 10 client.0 _ll_register_callbacks cb 0x5602a357bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.778+0000 7fdc837fe640 1 -- 172.21.15.70:0/2417684448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a365be80 msgr2=0x5602a3665f50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:40:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.778+0000 7fdc837fe640 1 --2- 172.21.15.70:0/2417684448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a365be80 0x5602a3665f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.778+0000 7fdc837fe640 1 -- 172.21.15.70:0/2417684448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a365b490 msgr2=0x5602a3665a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.778+0000 7fdc837fe640 1 --2- 172.21.15.70:0/2417684448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a365b490 0x5602a3665a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.778+0000 7fdc837fe640 1 -- 172.21.15.70:0/2417684448 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602a32d43f0 con 0x5602a3660ff0 2024-04-01T01:40:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.779+0000 7fdc837fe640 1 --2- 172.21.15.70:0/2417684448 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a3660ff0 0x5602a366cf90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c012700 tx=0x7fdc7c005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a97de89f6c7197a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.779+0000 7fdc62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:40:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.779+0000 7fdc62ffd640 1 -- 172.21.15.70:0/2417684448 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c039020 con 0x5602a3660ff0 2024-04-01T01:40:27.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.779+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/2417684448 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc7c012900 con 0x5602a3660ff0 2024-04-01T01:40:27.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.779+0000 7fdc62ffd640 1 -- 172.21.15.70:0/2417684448 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fdc7c005070 con 0x5602a3660ff0 2024-04-01T01:40:27.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.779+0000 7fdc62ffd640 1 -- 172.21.15.70:0/2417684448 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdc7c044050 con 0x5602a3660ff0 2024-04-01T01:40:27.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.780+0000 7fdc8c3fe740 10 client.27572 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:27.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.780+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/2417684448 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5602a339e7d0 con 0x5602a3660ff0 2024-04-01T01:40:27.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.780+0000 7fdc62ffd640 1 -- 172.21.15.70:0/2417684448 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc7c012900 con 0x5602a3660ff0 2024-04-01T01:40:27.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.780+0000 7fdc62ffd640 1 -- 172.21.15.70:0/2417684448 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fdc7c0035f0 con 0x5602a3660ff0 2024-04-01T01:40:27.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.781+0000 7fdc8c3fe740 10 client.27572 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:27.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.781+0000 7fdc8c3fe740 10 client.27572 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:40:27.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.781+0000 7fdc8c3fe740 20 client.27572 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:27.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.781+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/2417684448 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5602a366ff00 0x5602a3672390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.781+0000 7fdc8c3fe740 4 client.27572 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:40:27.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.781+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/2417684448 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602a35cd6f0 con 0x5602a366ff00 2024-04-01T01:40:27.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.781+0000 7fdc82ffd640 1 --2- 172.21.15.70:0/2417684448 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5602a366ff00 0x5602a3672390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.782+0000 7fdc82ffd640 1 --2- 172.21.15.70:0/2417684448 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5602a366ff00 0x5602a3672390 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4f962071d4010f30 server_cookie=8f8c8541e6d1a9de in_seq=0 out_seq=0 2024-04-01T01:40:27.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.782+0000 7fdc62ffd640 10 client.27572 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:40:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.784+0000 7fdc62ffd640 1 -- 172.21.15.70:0/2417684448 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5602a35cd6f0 con 0x5602a366ff00 2024-04-01T01:40:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.784+0000 7fdc62ffd640 10 client.27572 handle_command_reply: tid=0 2024-04-01T01:40:27.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.784+0000 7fdc8c3fe740 1 client.27572 shutdown 2024-04-01T01:40:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.784+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/2417684448 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a3660ff0 msgr2=0x5602a366cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.784+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/2417684448 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a3660ff0 0x5602a366cf90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c012700 tx=0x7fdc7c005dc0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.784+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/2417684448 shutdown_connections 2024-04-01T01:40:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.784+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/2417684448 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5602a366ff00 0x5602a3672390 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.784+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/2417684448 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a3660ff0 0x5602a366cf90 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.784+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/2417684448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a365be80 0x5602a3665f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.784+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/2417684448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a365b490 0x5602a3665a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.784+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/2417684448 >> 172.21.15.70:0/2417684448 conn(0x5602a3656a00 msgr2=0x5602a3655d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.784+0000 7fdc62ffd640 0 client.27572 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:40:27.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.784+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/2417684448 shutdown_connections 2024-04-01T01:40:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.784+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/2417684448 wait complete. 2024-04-01T01:40:27.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.784+0000 7fdc8c3fe740 20 client.27572 trim_cache size 0 max 16384 2024-04-01T01:40:27.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.822+0000 7fdc8c3fe740 1 Processor -- start 2024-04-01T01:40:27.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.823+0000 7fdc8c3fe740 1 -- start start 2024-04-01T01:40:27.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.823+0000 7fdc8c3fe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a365b490 0x5602a36554b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.823+0000 7fdc8c3fe740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a365be80 0x5602a366cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.823+0000 7fdc8c3fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a3660ff0 0x5602a36559f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.823+0000 7fdc82ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a365b490 0x5602a36554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.823+0000 7fdc82ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a365b490 0x5602a36554b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37514/0 (socket says 172.21.15.70:37514) 2024-04-01T01:40:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.823+0000 7fdc827fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a365be80 0x5602a366cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.823+0000 7fdc827fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a365be80 0x5602a366cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:59004/0 (socket says 172.21.15.70:59004) 2024-04-01T01:40:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.823+0000 7fdc827fc640 1 -- 172.21.15.70:0/4263404690 learned_addr learned my addr 172.21.15.70:0/4263404690 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:27.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.823+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/4263404690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc841b0a10 con 0x5602a3660ff0 2024-04-01T01:40:27.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.823+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/4263404690 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdc841b2550 con 0x5602a365be80 2024-04-01T01:40:27.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.823+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/4263404690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdc841b10e0 con 0x5602a365b490 2024-04-01T01:40:27.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.824+0000 7fdc837fe640 1 --2- 172.21.15.70:0/4263404690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a3660ff0 0x5602a36559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.824+0000 7fdc837fe640 1 -- 172.21.15.70:0/4263404690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a365b490 msgr2=0x5602a36554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.824+0000 7fdc837fe640 1 --2- 172.21.15.70:0/4263404690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a365b490 0x5602a36554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.824+0000 7fdc837fe640 1 -- 172.21.15.70:0/4263404690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a365be80 msgr2=0x5602a366cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.824+0000 7fdc837fe640 1 --2- 172.21.15.70:0/4263404690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a365be80 0x5602a366cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.824+0000 7fdc837fe640 1 -- 172.21.15.70:0/4263404690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602a3652460 con 0x5602a3660ff0 2024-04-01T01:40:27.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.824+0000 7fdc837fe640 1 --2- 172.21.15.70:0/4263404690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a3660ff0 0x5602a36559f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00a010 tx=0x7fdc7c00ffb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d6a99350d3916fb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.825+0000 7fdc697fa640 1 -- 172.21.15.70:0/4263404690 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c04d070 con 0x5602a3660ff0 2024-04-01T01:40:27.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.825+0000 7fdc697fa640 1 -- 172.21.15.70:0/4263404690 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc7c005570 con 0x5602a3660ff0 2024-04-01T01:40:27.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.825+0000 7fdc697fa640 1 -- 172.21.15.70:0/4263404690 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c039040 con 0x5602a3660ff0 2024-04-01T01:40:27.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.832+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/4263404690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a3660ff0 msgr2=0x5602a36559f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.832+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/4263404690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a3660ff0 0x5602a36559f0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00a010 tx=0x7fdc7c00ffb0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.832+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/4263404690 shutdown_connections 2024-04-01T01:40:27.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.832+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/4263404690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a3660ff0 0x5602a36559f0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.832+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/4263404690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a365be80 0x5602a366cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.832+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/4263404690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a365b490 0x5602a36554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.832+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/4263404690 >> 172.21.15.70:0/4263404690 conn(0x5602a3656a00 msgr2=0x5602a3665a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.833+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/4263404690 shutdown_connections 2024-04-01T01:40:27.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.833+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/4263404690 wait complete. 2024-04-01T01:40:27.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.833+0000 7fdc8c3fe740 1 -- start start 2024-04-01T01:40:27.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.834+0000 7fdc8c3fe740 1 Processor -- start 2024-04-01T01:40:27.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.834+0000 7fdc8c3fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a365b490 0x5602a36690e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.834+0000 7fdc8c3fe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a365be80 0x5602a3669620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.834+0000 7fdc8c3fe740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a3660ff0 0x5602a366c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.835+0000 7fdc8c3fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc841b1e80 con 0x5602a365b490 2024-04-01T01:40:27.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.835+0000 7fdc8c3fe740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdc841b17b0 con 0x5602a3660ff0 2024-04-01T01:40:27.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.835+0000 7fdc837fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a3660ff0 0x5602a366c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.835+0000 7fdc837fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a3660ff0 0x5602a366c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:59020/0 (socket says 172.21.15.70:59020) 2024-04-01T01:40:27.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.835+0000 7fdc837fe640 1 -- 172.21.15.70:0/265444467 learned_addr learned my addr 172.21.15.70:0/265444467 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:27.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.835+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/265444467 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdc841ae9f0 con 0x5602a365be80 2024-04-01T01:40:27.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.835+0000 7fdc8c3fe740 10 client.0 _ll_register_callbacks cb 0x7fdc7c011f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:27.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.835+0000 7fdc82ffd640 1 --2- 172.21.15.70:0/265444467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a365b490 0x5602a36690e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.835+0000 7fdc837fe640 1 -- 172.21.15.70:0/265444467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a365be80 msgr2=0x5602a3669620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.835+0000 7fdc837fe640 1 --2- 172.21.15.70:0/265444467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a365be80 0x5602a3669620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.836+0000 7fdc837fe640 1 -- 172.21.15.70:0/265444467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a365b490 msgr2=0x5602a36690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:27.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.836+0000 7fdc837fe640 1 --2- 172.21.15.70:0/265444467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a365b490 0x5602a36690e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.836+0000 7fdc837fe640 1 -- 172.21.15.70:0/265444467 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602a366c760 con 0x5602a3660ff0 2024-04-01T01:40:27.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.836+0000 7fdc837fe640 1 --2- 172.21.15.70:0/265444467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a3660ff0 0x5602a366c120 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c001eb0 tx=0x7fdc7c003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f78ffe13bea77ddb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:27.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.836+0000 7fdc62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:40:27.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.837+0000 7fdc62ffd640 1 -- 172.21.15.70:0/265444467 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c04d070 con 0x5602a3660ff0 2024-04-01T01:40:27.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.837+0000 7fdc62ffd640 1 -- 172.21.15.70:0/265444467 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fdc7c00f5a0 con 0x5602a3660ff0 2024-04-01T01:40:27.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.837+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/265444467 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc7c005570 con 0x5602a3660ff0 2024-04-01T01:40:27.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.837+0000 7fdc62ffd640 1 -- 172.21.15.70:0/265444467 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdc7c059050 con 0x5602a3660ff0 2024-04-01T01:40:27.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.837+0000 7fdc8c3fe740 10 client.27578 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:27.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.837+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/265444467 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5602a36d6b40 con 0x5602a3660ff0 2024-04-01T01:40:27.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.838+0000 7fdc62ffd640 1 -- 172.21.15.70:0/265444467 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc7c005570 con 0x5602a3660ff0 2024-04-01T01:40:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.838+0000 7fdc62ffd640 1 -- 172.21.15.70:0/265444467 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fdc7c035470 con 0x5602a3660ff0 2024-04-01T01:40:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.838+0000 7fdc8c3fe740 10 client.27578 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.838+0000 7fdc8c3fe740 10 client.27578 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:40:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.838+0000 7fdc8c3fe740 20 client.27578 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.838+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/265444467 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5602a36d7430 0x5602a36d9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.838+0000 7fdc8c3fe740 4 client.27578 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:40:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.838+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/265444467 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5602a35c7a40 con 0x5602a36d7430 2024-04-01T01:40:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.838+0000 7fdc82ffd640 1 --2- 172.21.15.70:0/265444467 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5602a36d7430 0x5602a36d9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:27.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.839+0000 7fdc62ffd640 10 client.27578 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:40:27.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.840+0000 7fdc82ffd640 1 --2- 172.21.15.70:0/265444467 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5602a36d7430 0x5602a36d9850 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e5a9fac85e947b server_cookie=18cd8a9c595881d5 in_seq=0 out_seq=0 2024-04-01T01:40:27.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.840+0000 7fdc62ffd640 1 -- 172.21.15.70:0/265444467 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5602a35c7a40 con 0x5602a36d7430 2024-04-01T01:40:27.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.840+0000 7fdc62ffd640 10 client.27578 handle_command_reply: tid=0 2024-04-01T01:40:27.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.840+0000 7fdc8c3fe740 1 client.27578 shutdown 2024-04-01T01:40:27.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.841+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/265444467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a3660ff0 msgr2=0x5602a366c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.841+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/265444467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a3660ff0 0x5602a366c120 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c001eb0 tx=0x7fdc7c003820 comp rx=0 tx=0).stop 2024-04-01T01:40:27.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.841+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/265444467 shutdown_connections 2024-04-01T01:40:27.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.841+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/265444467 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5602a36d7430 0x5602a36d9850 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.841+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/265444467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602a3660ff0 0x5602a366c120 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.841+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/265444467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602a365be80 0x5602a3669620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.841+0000 7fdc62ffd640 0 client.27578 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:40:27.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.841+0000 7fdc8c3fe740 1 --2- 172.21.15.70:0/265444467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5602a365b490 0x5602a36690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.841+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/265444467 >> 172.21.15.70:0/265444467 conn(0x5602a3656a00 msgr2=0x5602a3659c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.841+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/265444467 shutdown_connections 2024-04-01T01:40:27.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.841+0000 7fdc8c3fe740 1 -- 172.21.15.70:0/265444467 wait complete. 2024-04-01T01:40:27.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.841+0000 7fdc8c3fe740 20 client.27578 trim_cache size 0 max 16384 2024-04-01T01:40:27.842 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:40:27.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.842+0000 7fdc8981c640 1 -- 172.21.15.70:0/1227970068 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdc6c0a6a30 msgr2=0x7fdc6c0a8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.842+0000 7fdc8981c640 1 --2- 172.21.15.70:0/1227970068 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdc6c0a6a30 0x7fdc6c0a8ef0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x5602a365c280 tx=0x7fdc7800d040 comp rx=0 tx=0).stop 2024-04-01T01:40:27.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.842+0000 7fdc8981c640 1 -- 172.21.15.70:0/1227970068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc840ff9a0 msgr2=0x7fdc84070710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:27.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.842+0000 7fdc8981c640 1 --2- 172.21.15.70:0/1227970068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc840ff9a0 0x7fdc84070710 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdc74032380 tx=0x7fdc74032820 comp rx=0 tx=0).stop 2024-04-01T01:40:27.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.843+0000 7fdc8981c640 1 -- 172.21.15.70:0/1227970068 shutdown_connections 2024-04-01T01:40:27.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.843+0000 7fdc8981c640 1 --2- 172.21.15.70:0/1227970068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc841010d0 0x7fdc841a57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.843+0000 7fdc8981c640 1 --2- 172.21.15.70:0/1227970068 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fdc6c0a6a30 0x7fdc6c0a8ef0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.843+0000 7fdc8981c640 1 --2- 172.21.15.70:0/1227970068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc840ff9a0 0x7fdc84070710 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.843+0000 7fdc8981c640 1 --2- 172.21.15.70:0/1227970068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc84067cb0 0x7fdc840701d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:27.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.843+0000 7fdc8981c640 1 -- 172.21.15.70:0/1227970068 >> 172.21.15.70:0/1227970068 conn(0x7fdc840faf10 msgr2=0x7fdc840fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:27.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.845+0000 7fdc8981c640 1 -- 172.21.15.70:0/1227970068 shutdown_connections 2024-04-01T01:40:27.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:27.845+0000 7fdc8981c640 1 -- 172.21.15.70:0/1227970068 wait complete. 2024-04-01T01:40:27.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:40:28.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.063+0000 7faa1b4b1640 1 -- 172.21.15.70:0/1531491094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0c0a6710 msgr2=0x7faa0c0a6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:28.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.063+0000 7faa1b4b1640 1 --2- 172.21.15.70:0/1531491094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0c0a6710 0x7faa0c0a6b90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faa14067420 tx=0x7faa1408f430 comp rx=0 tx=0).stop 2024-04-01T01:40:28.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.064+0000 7faa1b4b1640 1 -- 172.21.15.70:0/1531491094 shutdown_connections 2024-04-01T01:40:28.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.064+0000 7faa1b4b1640 1 --2- 172.21.15.70:0/1531491094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0c0a6710 0x7faa0c0a6b90 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.064+0000 7faa1b4b1640 1 --2- 172.21.15.70:0/1531491094 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa0c0182a0 0x7faa0c0a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.064+0000 7faa1b4b1640 1 --2- 172.21.15.70:0/1531491094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa0c0a4fe0 0x7faa0c0a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.064+0000 7faa1b4b1640 1 -- 172.21.15.70:0/1531491094 >> 172.21.15.70:0/1531491094 conn(0x7faa0c0a0530 msgr2=0x7faa0c0a2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:28.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.064+0000 7faa1b4b1640 1 -- 172.21.15.70:0/1531491094 shutdown_connections 2024-04-01T01:40:28.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.064+0000 7faa1b4b1640 1 -- 172.21.15.70:0/1531491094 wait complete. 2024-04-01T01:40:28.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.065+0000 7faa1b4b1640 1 Processor -- start 2024-04-01T01:40:28.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.065+0000 7faa1b4b1640 1 -- start start 2024-04-01T01:40:28.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.065+0000 7faa1b4b1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0c0182a0 0x7faa0c143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.065+0000 7faa1b4b1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa0c0a4fe0 0x7faa0c143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.065+0000 7faa1b4b1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa0c0a6710 0x7faa0c14aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.065+0000 7faa1b4b1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faa0c0b0640 con 0x7faa0c0a6710 2024-04-01T01:40:28.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.065+0000 7faa1b4b1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faa0c0b04c0 con 0x7faa0c0a4fe0 2024-04-01T01:40:28.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.065+0000 7faa1b4b1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faa0c0b07c0 con 0x7faa0c0182a0 2024-04-01T01:40:28.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.066+0000 7faa1acb0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa0c0a6710 0x7faa0c14aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:28.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.066+0000 7faa1acb0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa0c0a6710 0x7faa0c14aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47624/0 (socket says 172.21.15.70:47624) 2024-04-01T01:40:28.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.066+0000 7faa1acb0640 1 -- 172.21.15.70:0/2027328491 learned_addr learned my addr 172.21.15.70:0/2027328491 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:28.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.069+0000 7faa1acb0640 1 -- 172.21.15.70:0/2027328491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0c0182a0 msgr2=0x7faa0c143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:28.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.069+0000 7faa1acb0640 1 --2- 172.21.15.70:0/2027328491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0c0182a0 0x7faa0c143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.069+0000 7faa1acb0640 1 -- 172.21.15.70:0/2027328491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa0c0a4fe0 msgr2=0x7faa0c143ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:28.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.069+0000 7faa1acb0640 1 --2- 172.21.15.70:0/2027328491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa0c0a4fe0 0x7faa0c143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.069+0000 7faa1acb0640 1 -- 172.21.15.70:0/2027328491 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa0c13a3b0 con 0x7faa0c0a6710 2024-04-01T01:40:28.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.070+0000 7faa1acb0640 1 --2- 172.21.15.70:0/2027328491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa0c0a6710 0x7faa0c14aad0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7faa14067010 tx=0x7faa14067f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:28.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.071+0000 7faa0b7fe640 1 -- 172.21.15.70:0/2027328491 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa140698d0 con 0x7faa0c0a6710 2024-04-01T01:40:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.072+0000 7faa1b4b1640 1 -- 172.21.15.70:0/2027328491 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa0c13a640 con 0x7faa0c0a6710 2024-04-01T01:40:28.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.072+0000 7faa1b4b1640 1 -- 172.21.15.70:0/2027328491 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa0c13ab00 con 0x7faa0c0a6710 2024-04-01T01:40:28.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.075+0000 7faa1f3d2740 1 Processor -- start 2024-04-01T01:40:28.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.075+0000 7faa0b7fe640 1 -- 172.21.15.70:0/2027328491 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa140a7070 con 0x7faa0c0a6710 2024-04-01T01:40:28.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.076+0000 7faa0b7fe640 1 -- 172.21.15.70:0/2027328491 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa140978d0 con 0x7faa0c0a6710 2024-04-01T01:40:28.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.077+0000 7faa0b7fe640 1 -- 172.21.15.70:0/2027328491 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7faa14097b30 con 0x7faa0c0a6710 2024-04-01T01:40:28.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.078+0000 7faa0b7fe640 1 --2- 172.21.15.70:0/2027328491 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faa000a2450 0x7faa000a4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.078+0000 7faa1f3d2740 1 -- start start 2024-04-01T01:40:28.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.078+0000 7faa1f3d2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce54680 0x560ebce54aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.078+0000 7faa1f3d2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 0x560ebce59b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.078+0000 7faa1f3d2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce5a1e0 0x560ebce5c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.078+0000 7faa1f3d2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560ebcdb91b0 con 0x560ebce55070 2024-04-01T01:40:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.079+0000 7faa1f3d2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560ebcd61c00 con 0x560ebce5a1e0 2024-04-01T01:40:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.079+0000 7faa1f3d2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560ebcdb5250 con 0x560ebce54680 2024-04-01T01:40:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.079+0000 7faa19cae640 1 --2- 172.21.15.70:0/2027328491 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faa000a2450 0x7faa000a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.079+0000 7faa1acb0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce5a1e0 0x560ebce5c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.079+0000 7faa1acb0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce5a1e0 0x560ebce5c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:59044/0 (socket says 172.21.15.70:59044) 2024-04-01T01:40:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.079+0000 7faa1acb0640 1 -- 172.21.15.70:0/4208404708 learned_addr learned my addr 172.21.15.70:0/4208404708 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.079+0000 7faa1a4af640 1 --2- 172.21.15.70:0/4208404708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce54680 0x560ebce54aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:28.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.079+0000 7faa19cae640 1 --2- 172.21.15.70:0/4208404708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 0x560ebce59b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.080+0000 7faa0b7fe640 1 -- 172.21.15.70:0/2027328491 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7faa141541c0 con 0x7faa0c0a6710 2024-04-01T01:40:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.080+0000 7faa1acb0640 1 -- 172.21.15.70:0/4208404708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce54680 msgr2=0x560ebce54aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.080+0000 7faa1acb0640 1 --2- 172.21.15.70:0/4208404708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce54680 0x560ebce54aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.080+0000 7faa1acb0640 1 -- 172.21.15.70:0/4208404708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 msgr2=0x560ebce59b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.080+0000 7faa1acb0640 1 --2- 172.21.15.70:0/4208404708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 0x560ebce59b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.080+0000 7faa1acb0640 1 -- 172.21.15.70:0/4208404708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ebcd813d0 con 0x560ebce5a1e0 2024-04-01T01:40:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.081+0000 7faa1acb0640 1 --2- 172.21.15.70:0/4208404708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce5a1e0 0x560ebce5c670 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7faa141207e0 tx=0x7faa1409e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d8c9fc300696484 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:28.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.081+0000 7faa1b4b1640 1 -- 172.21.15.70:0/4208404708 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa14120850 con 0x560ebce5a1e0 2024-04-01T01:40:28.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.081+0000 7faa1b4b1640 1 -- 172.21.15.70:0/4208404708 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa1409d040 con 0x560ebce5a1e0 2024-04-01T01:40:28.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.081+0000 7faa1b4b1640 1 -- 172.21.15.70:0/4208404708 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa140964f0 con 0x560ebce5a1e0 2024-04-01T01:40:28.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.083+0000 7faa1a4af640 1 --2- 172.21.15.70:0/4208404708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce54680 0x560ebce54aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:40:28.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.083+0000 7faa1f3d2740 1 -- 172.21.15.70:0/4208404708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce5a1e0 msgr2=0x560ebce5c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:28.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.083+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/4208404708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce5a1e0 0x560ebce5c670 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7faa141207e0 tx=0x7faa1409e000 comp rx=0 tx=0).stop 2024-04-01T01:40:28.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.083+0000 7faa19cae640 1 --2- 172.21.15.70:0/4208404708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 0x560ebce59b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:40:28.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.084+0000 7faa19cae640 1 --2- 172.21.15.70:0/2027328491 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faa000a2450 0x7faa000a4910 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x560ebce55470 tx=0x7faa10009390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:28.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.084+0000 7faa1f3d2740 1 -- 172.21.15.70:0/4208404708 shutdown_connections 2024-04-01T01:40:28.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.084+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/4208404708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce5a1e0 0x560ebce5c670 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.084+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/4208404708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 0x560ebce59b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.084+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/4208404708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce54680 0x560ebce54aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.084+0000 7faa1f3d2740 1 -- 172.21.15.70:0/4208404708 >> 172.21.15.70:0/4208404708 conn(0x560ebce4fee0 msgr2=0x560ebce502e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:28.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.084+0000 7faa1f3d2740 1 -- 172.21.15.70:0/4208404708 shutdown_connections 2024-04-01T01:40:28.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.084+0000 7faa1f3d2740 1 -- 172.21.15.70:0/4208404708 wait complete. 2024-04-01T01:40:28.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.085+0000 7faa1f3d2740 1 -- start start 2024-04-01T01:40:28.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.085+0000 7faa1f3d2740 1 Processor -- start 2024-04-01T01:40:28.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.085+0000 7faa1f3d2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce54680 0x560ebce5e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.085+0000 7faa1f3d2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 0x560ebce5ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.085+0000 7faa1f3d2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce5a1e0 0x560ebce65e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.085+0000 7faa1f3d2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560ebce5cf50 con 0x560ebce55070 2024-04-01T01:40:28.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.085+0000 7faa1f3d2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560ebcd57470 con 0x560ebce54680 2024-04-01T01:40:28.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.085+0000 7faa1f3d2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560ebce5d0d0 con 0x560ebce5a1e0 2024-04-01T01:40:28.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.086+0000 7faa1f3d2740 10 client.0 _ll_register_callbacks cb 0x560ebcd74cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:28.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.086+0000 7faa1acb0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce5a1e0 0x560ebce65e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:28.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.086+0000 7faa19cae640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 0x560ebce5ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:28.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.086+0000 7faa1acb0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce5a1e0 0x560ebce65e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37560/0 (socket says 172.21.15.70:37560) 2024-04-01T01:40:28.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.086+0000 7faa19cae640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 0x560ebce5ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47632/0 (socket says 172.21.15.70:47632) 2024-04-01T01:40:28.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.086+0000 7faa19cae640 1 -- 172.21.15.70:0/8109298 learned_addr learned my addr 172.21.15.70:0/8109298 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:28.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.086+0000 7faa1a4af640 1 --2- 172.21.15.70:0/8109298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce54680 0x560ebce5e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:28.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.086+0000 7faa1acb0640 1 -- 172.21.15.70:0/8109298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce54680 msgr2=0x560ebce5e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:28.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.087+0000 7faa1acb0640 1 --2- 172.21.15.70:0/8109298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce54680 0x560ebce5e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.087+0000 7faa1acb0640 1 -- 172.21.15.70:0/8109298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 msgr2=0x560ebce5ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:28.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.087+0000 7faa1acb0640 1 --2- 172.21.15.70:0/8109298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 0x560ebce5ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.087+0000 7faa1acb0640 1 -- 172.21.15.70:0/8109298 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ebcacd3f0 con 0x560ebce5a1e0 2024-04-01T01:40:28.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.087+0000 7faa19cae640 1 --2- 172.21.15.70:0/8109298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 0x560ebce5ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:40:28.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.088+0000 7faa1acb0640 1 --2- 172.21.15.70:0/8109298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce5a1e0 0x560ebce65e40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faa14062c90 tx=0x7faa1408f430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49b7c6b559cfa32e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:28.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.088+0000 7fa9fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:40:28.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.088+0000 7fa9fdffb640 1 -- 172.21.15.70:0/8109298 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa140964f0 con 0x560ebce5a1e0 2024-04-01T01:40:28.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.088+0000 7fa9fdffb640 1 -- 172.21.15.70:0/8109298 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7faa1411cec0 con 0x560ebce5a1e0 2024-04-01T01:40:28.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.089+0000 7faa1f3d2740 1 -- 172.21.15.70:0/8109298 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa1409d040 con 0x560ebce5a1e0 2024-04-01T01:40:28.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.090+0000 7fa9fdffb640 1 -- 172.21.15.70:0/8109298 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faa14120850 con 0x560ebce5a1e0 2024-04-01T01:40:28.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.092+0000 7faa1f3d2740 10 client.27430 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:28.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.092+0000 7faa1f3d2740 1 -- 172.21.15.70:0/8109298 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560ebcb977d0 con 0x560ebce5a1e0 2024-04-01T01:40:28.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.093+0000 7fa9fdffb640 1 -- 172.21.15.70:0/8109298 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa1409d040 con 0x560ebce5a1e0 2024-04-01T01:40:28.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.093+0000 7fa9fdffb640 1 -- 172.21.15.70:0/8109298 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7faa1411c6c0 con 0x560ebce5a1e0 2024-04-01T01:40:28.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.093+0000 7faa1f3d2740 10 client.27430 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:28.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.093+0000 7faa1f3d2740 10 client.27430 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:40:28.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.093+0000 7faa1f3d2740 20 client.27430 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:28.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.093+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/8109298 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560ebce6a210 0x560ebce6c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.093+0000 7faa1f3d2740 4 client.27430 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:40:28.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.093+0000 7faa1f3d2740 1 -- 172.21.15.70:0/8109298 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ebcdc66f0 con 0x560ebce6a210 2024-04-01T01:40:28.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.094+0000 7faa1a4af640 1 --2- 172.21.15.70:0/8109298 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560ebce6a210 0x560ebce6c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:28.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.095+0000 7faa1a4af640 1 --2- 172.21.15.70:0/8109298 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560ebce6a210 0x560ebce6c630 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d6cfb57a61fc3cf server_cookie=125cefb1dd85b3f1 in_seq=0 out_seq=0 2024-04-01T01:40:28.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.095+0000 7fa9fdffb640 10 client.27430 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:28.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.095+0000 7fa9fdffb640 1 -- 172.21.15.70:0/8109298 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560ebcdc66f0 con 0x560ebce6a210 2024-04-01T01:40:28.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.095+0000 7fa9fdffb640 10 client.27430 handle_command_reply: tid=0 2024-04-01T01:40:28.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.096+0000 7faa1f3d2740 1 client.27430 shutdown 2024-04-01T01:40:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.096+0000 7faa1f3d2740 1 -- 172.21.15.70:0/8109298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce5a1e0 msgr2=0x560ebce65e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.096+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/8109298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce5a1e0 0x560ebce65e40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faa14062c90 tx=0x7faa1408f430 comp rx=0 tx=0).stop 2024-04-01T01:40:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.096+0000 7faa1f3d2740 1 -- 172.21.15.70:0/8109298 shutdown_connections 2024-04-01T01:40:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.096+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/8109298 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560ebce6a210 0x560ebce6c630 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.096+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/8109298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce5a1e0 0x560ebce65e40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.096+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/8109298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 0x560ebce5ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.096+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/8109298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce54680 0x560ebce5e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.097+0000 7fa9fdffb640 0 client.27430 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.097+0000 7faa1f3d2740 1 -- 172.21.15.70:0/8109298 >> 172.21.15.70:0/8109298 conn(0x560ebce4fee0 msgr2=0x560ebce4f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:28.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.097+0000 7faa1f3d2740 1 -- 172.21.15.70:0/8109298 shutdown_connections 2024-04-01T01:40:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.097+0000 7faa1f3d2740 1 -- 172.21.15.70:0/8109298 wait complete. 2024-04-01T01:40:28.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.097+0000 7faa1f3d2740 20 client.27430 trim_cache size 0 max 16384 2024-04-01T01:40:28.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.123+0000 7faa1f3d2740 1 Processor -- start 2024-04-01T01:40:28.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.123+0000 7faa1f3d2740 1 -- start start 2024-04-01T01:40:28.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.124+0000 7faa1f3d2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce54680 0x560ebce5e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.124+0000 7faa1f3d2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce55070 0x560ebce5ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.124+0000 7faa1f3d2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce5a1e0 0x560ebce65e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.124+0000 7faa1f3d2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f800fd40 con 0x560ebce54680 2024-04-01T01:40:28.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.124+0000 7faa1f3d2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f8011880 con 0x560ebce55070 2024-04-01T01:40:28.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.124+0000 7faa1f3d2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f8010410 con 0x560ebce5a1e0 2024-04-01T01:40:28.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.124+0000 7faa1acb0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce5a1e0 0x560ebce65e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:28.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.124+0000 7faa1a4af640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce54680 0x560ebce5e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:28.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.124+0000 7faa1acb0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce5a1e0 0x560ebce65e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37568/0 (socket says 172.21.15.70:37568) 2024-04-01T01:40:28.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.124+0000 7faa1acb0640 1 -- 172.21.15.70:0/206606418 learned_addr learned my addr 172.21.15.70:0/206606418 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:28.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.124+0000 7faa19cae640 1 --2- 172.21.15.70:0/206606418 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce55070 0x560ebce5ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:28.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.124+0000 7faa19cae640 1 -- 172.21.15.70:0/206606418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce5a1e0 msgr2=0x560ebce65e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:28.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.124+0000 7faa19cae640 1 --2- 172.21.15.70:0/206606418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce5a1e0 0x560ebce65e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.124+0000 7faa19cae640 1 -- 172.21.15.70:0/206606418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce54680 msgr2=0x560ebce5e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:28.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.124+0000 7faa19cae640 1 --2- 172.21.15.70:0/206606418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce54680 0x560ebce5e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.124+0000 7faa19cae640 1 -- 172.21.15.70:0/206606418 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ebce4ae20 con 0x560ebce55070 2024-04-01T01:40:28.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.125+0000 7faa19cae640 1 --2- 172.21.15.70:0/206606418 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce55070 0x560ebce5ee00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x560ebce5fda0 tx=0x7faa10005c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=968d28324061dc12 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:28.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.126+0000 7fa9fffff640 1 -- 172.21.15.70:0/206606418 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1001f8e0 con 0x560ebce55070 2024-04-01T01:40:28.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.126+0000 7fa9fffff640 1 -- 172.21.15.70:0/206606418 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa10010430 con 0x560ebce55070 2024-04-01T01:40:28.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.126+0000 7fa9fffff640 1 -- 172.21.15.70:0/206606418 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa10029650 con 0x560ebce55070 2024-04-01T01:40:28.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.127+0000 7faa1f3d2740 1 -- 172.21.15.70:0/206606418 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce55070 msgr2=0x560ebce5ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:28.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.127+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/206606418 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce55070 0x560ebce5ee00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x560ebce5fda0 tx=0x7faa10005c50 comp rx=0 tx=0).stop 2024-04-01T01:40:28.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.127+0000 7faa1f3d2740 1 -- 172.21.15.70:0/206606418 shutdown_connections 2024-04-01T01:40:28.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.127+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/206606418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce5a1e0 0x560ebce65e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.127+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/206606418 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce55070 0x560ebce5ee00 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.127+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/206606418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce54680 0x560ebce5e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.127+0000 7faa1f3d2740 1 -- 172.21.15.70:0/206606418 >> 172.21.15.70:0/206606418 conn(0x560ebce4fee0 msgr2=0x560ebce6c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:28.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.128+0000 7faa1f3d2740 1 -- 172.21.15.70:0/206606418 shutdown_connections 2024-04-01T01:40:28.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.128+0000 7faa1f3d2740 1 -- 172.21.15.70:0/206606418 wait complete. 2024-04-01T01:40:28.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.128+0000 7faa1f3d2740 1 -- start start 2024-04-01T01:40:28.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.128+0000 7faa1f3d2740 1 Processor -- start 2024-04-01T01:40:28.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.129+0000 7faa1f3d2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce54680 0x560ebce62ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.129+0000 7faa1f3d2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 0x560ebce63020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.129+0000 7faa1f3d2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce5a1e0 0x560ebcece520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.129+0000 7faa1a4af640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce54680 0x560ebce62ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:28.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.129+0000 7faa1a4af640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce54680 0x560ebce62ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37584/0 (socket says 172.21.15.70:37584) 2024-04-01T01:40:28.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.129+0000 7faa1a4af640 1 -- 172.21.15.70:0/467254927 learned_addr learned my addr 172.21.15.70:0/467254927 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:28.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.130+0000 7faa1f3d2740 1 -- 172.21.15.70:0/467254927 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f80111b0 con 0x560ebce55070 2024-04-01T01:40:28.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.130+0000 7faa1acb0640 1 --2- 172.21.15.70:0/467254927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce5a1e0 0x560ebcece520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:28.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.130+0000 7faa1f3d2740 1 -- 172.21.15.70:0/467254927 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f8010ae0 con 0x560ebce5a1e0 2024-04-01T01:40:28.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.130+0000 7faa1f3d2740 1 -- 172.21.15.70:0/467254927 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f800dd20 con 0x560ebce54680 2024-04-01T01:40:28.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.130+0000 7faa19cae640 1 --2- 172.21.15.70:0/467254927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 0x560ebce63020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:28.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.130+0000 7faa1f3d2740 10 client.0 _ll_register_callbacks cb 0x7faa141206b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:28.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.130+0000 7faa1acb0640 1 -- 172.21.15.70:0/467254927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce54680 msgr2=0x560ebce62ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:28.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.130+0000 7faa1acb0640 1 --2- 172.21.15.70:0/467254927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce54680 0x560ebce62ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.130+0000 7faa1acb0640 1 -- 172.21.15.70:0/467254927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 msgr2=0x560ebce63020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:28.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.130+0000 7faa1acb0640 1 --2- 172.21.15.70:0/467254927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 0x560ebce63020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.130+0000 7faa1acb0640 1 -- 172.21.15.70:0/467254927 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ebce63660 con 0x560ebce5a1e0 2024-04-01T01:40:28.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.130+0000 7faa19cae640 1 --2- 172.21.15.70:0/467254927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 0x560ebce63020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:40:28.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.131+0000 7faa1f3d2740 1 -- 172.21.15.70:0/467254927 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa10010430 con 0x560ebce5a1e0 2024-04-01T01:40:28.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.131+0000 7faa1acb0640 1 --2- 172.21.15.70:0/467254927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce5a1e0 0x560ebcece520 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7faa14067290 tx=0x7faa14069a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2a54e76baf35e6e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:28.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.131+0000 7fa9fdffb640 10 client.27593 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:40:28.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.131+0000 7fa9fdffb640 1 -- 172.21.15.70:0/467254927 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa140964f0 con 0x560ebce5a1e0 2024-04-01T01:40:28.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.131+0000 7fa9fdffb640 1 -- 172.21.15.70:0/467254927 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7faa14121d90 con 0x560ebce5a1e0 2024-04-01T01:40:28.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.132+0000 7fa9fdffb640 1 -- 172.21.15.70:0/467254927 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faa141201b0 con 0x560ebce5a1e0 2024-04-01T01:40:28.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.132+0000 7faa1f3d2740 10 client.27593 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:28.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.132+0000 7faa1f3d2740 1 -- 172.21.15.70:0/467254927 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=31}) -- 0x560ebce63660 con 0x560ebce5a1e0 2024-04-01T01:40:28.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.132+0000 7fa9fdffb640 1 -- 172.21.15.70:0/467254927 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7faa1411c5a0 con 0x560ebce5a1e0 2024-04-01T01:40:28.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.133+0000 7faa1f3d2740 10 client.27593 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:28.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.133+0000 7faa1f3d2740 10 client.27593 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:40:28.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.133+0000 7faa1f3d2740 20 client.27593 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:28.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.133+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/467254927 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560ebcecfec0 0x560ebced22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:28.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.133+0000 7faa1f3d2740 4 client.27593 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:40:28.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.133+0000 7faa1f3d2740 1 -- 172.21.15.70:0/467254927 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560ebcdc0a40 con 0x560ebcecfec0 2024-04-01T01:40:28.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.133+0000 7faa1a4af640 1 --2- 172.21.15.70:0/467254927 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560ebcecfec0 0x560ebced22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:28.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.138+0000 7faa1a4af640 1 --2- 172.21.15.70:0/467254927 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560ebcecfec0 0x560ebced22e0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21380947c6f2ea8a server_cookie=984ca9ba4105e72d in_seq=0 out_seq=0 2024-04-01T01:40:28.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.138+0000 7fa9fdffb640 10 client.27593 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:28.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.138+0000 7fa9fdffb640 1 -- 172.21.15.70:0/467254927 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560ebcdc0a40 con 0x560ebcecfec0 2024-04-01T01:40:28.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.138+0000 7fa9fdffb640 10 client.27593 handle_command_reply: tid=0 2024-04-01T01:40:28.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.138+0000 7faa1f3d2740 1 client.27593 shutdown 2024-04-01T01:40:28.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.139+0000 7faa1f3d2740 1 -- 172.21.15.70:0/467254927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce5a1e0 msgr2=0x560ebcece520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:28.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.139+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/467254927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce5a1e0 0x560ebcece520 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7faa14067290 tx=0x7faa14069a70 comp rx=0 tx=0).stop 2024-04-01T01:40:28.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.139+0000 7faa1f3d2740 1 -- 172.21.15.70:0/467254927 shutdown_connections 2024-04-01T01:40:28.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.139+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/467254927 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560ebcecfec0 0x560ebced22e0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.140+0000 7fa9fdffb640 0 client.27593 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:28.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.140+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/467254927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560ebce5a1e0 0x560ebcece520 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.140+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/467254927 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ebce55070 0x560ebce63020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.140+0000 7faa1f3d2740 1 --2- 172.21.15.70:0/467254927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560ebce54680 0x560ebce62ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.140+0000 7faa1f3d2740 1 -- 172.21.15.70:0/467254927 >> 172.21.15.70:0/467254927 conn(0x560ebce4fee0 msgr2=0x560ebce56530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:28.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.140+0000 7faa1f3d2740 1 -- 172.21.15.70:0/467254927 shutdown_connections 2024-04-01T01:40:28.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.140+0000 7faa1f3d2740 1 -- 172.21.15.70:0/467254927 wait complete. 2024-04-01T01:40:28.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.140+0000 7faa1f3d2740 20 client.27593 trim_cache size 0 max 16384 2024-04-01T01:40:28.140 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:40:28.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.142+0000 7faa097fa640 1 -- 172.21.15.70:0/2027328491 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faa000a2450 msgr2=0x7faa000a4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:28.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.142+0000 7faa097fa640 1 --2- 172.21.15.70:0/2027328491 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faa000a2450 0x7faa000a4910 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x560ebce55470 tx=0x7faa10009390 comp rx=0 tx=0).stop 2024-04-01T01:40:28.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.142+0000 7faa097fa640 1 -- 172.21.15.70:0/2027328491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa0c0a6710 msgr2=0x7faa0c14aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:28.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.142+0000 7faa097fa640 1 --2- 172.21.15.70:0/2027328491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa0c0a6710 0x7faa0c14aad0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7faa14067010 tx=0x7faa14067f20 comp rx=0 tx=0).stop 2024-04-01T01:40:28.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.142+0000 7faa097fa640 1 -- 172.21.15.70:0/2027328491 shutdown_connections 2024-04-01T01:40:28.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.143+0000 7faa097fa640 1 --2- 172.21.15.70:0/2027328491 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa0c0a6710 0x7faa0c14aad0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.143+0000 7faa097fa640 1 --2- 172.21.15.70:0/2027328491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa0c0a4fe0 0x7faa0c143ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.143+0000 7faa097fa640 1 --2- 172.21.15.70:0/2027328491 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faa000a2450 0x7faa000a4910 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.143+0000 7faa097fa640 1 --2- 172.21.15.70:0/2027328491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0c0182a0 0x7faa0c143570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:28.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.143+0000 7faa097fa640 1 -- 172.21.15.70:0/2027328491 >> 172.21.15.70:0/2027328491 conn(0x7faa0c0a0530 msgr2=0x7faa0c0a2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:28.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.144+0000 7faa097fa640 1 -- 172.21.15.70:0/2027328491 shutdown_connections 2024-04-01T01:40:28.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:28.144+0000 7faa097fa640 1 -- 172.21.15.70:0/2027328491 wait complete. 2024-04-01T01:40:29.164 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:40:29.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:40:29.165 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:40:29.212 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:28 smithi070 ceph-mon[32179]: pgmap v569: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 14 MiB/s wr, 1.06k op/s 2024-04-01T01:40:29.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:28 smithi189 ceph-mon[35753]: pgmap v569: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 14 MiB/s wr, 1.06k op/s 2024-04-01T01:40:29.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:28 smithi081 ceph-mon[35142]: pgmap v569: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 14 MiB/s wr, 1.06k op/s 2024-04-01T01:40:29.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.361+0000 7f69dd956640 1 -- 172.21.15.70:0/3579891473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69d8103330 msgr2=0x7f69d8109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.361+0000 7f69dd956640 1 --2- 172.21.15.70:0/3579891473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69d8103330 0x7f69d8109bc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f69d0009ca0 tx=0x7f69d0030530 comp rx=0 tx=0).stop 2024-04-01T01:40:29.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.361+0000 7f69dd956640 1 -- 172.21.15.70:0/3579891473 shutdown_connections 2024-04-01T01:40:29.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.361+0000 7f69dd956640 1 --2- 172.21.15.70:0/3579891473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69d8103330 0x7f69d8109bc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.361+0000 7f69dd956640 1 --2- 172.21.15.70:0/3579891473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69d8102970 0x7f69d8102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.361+0000 7f69dd956640 1 --2- 172.21.15.70:0/3579891473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69d8101780 0x7f69d8101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.361+0000 7f69dd956640 1 -- 172.21.15.70:0/3579891473 >> 172.21.15.70:0/3579891473 conn(0x7f69d80fccf0 msgr2=0x7f69d80ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:29.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.361+0000 7f69dd956640 1 -- 172.21.15.70:0/3579891473 shutdown_connections 2024-04-01T01:40:29.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.361+0000 7f69dd956640 1 -- 172.21.15.70:0/3579891473 wait complete. 2024-04-01T01:40:29.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.362+0000 7f69dd956640 1 Processor -- start 2024-04-01T01:40:29.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.362+0000 7f69dd956640 1 -- start start 2024-04-01T01:40:29.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.362+0000 7f69dd956640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69d8101780 0x7f69d81957d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.362+0000 7f69dd956640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69d8102970 0x7f69d8195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.363+0000 7f69dd956640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69d8103330 0x7f69d81ae030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.363+0000 7f69dd956640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f69d810b110 con 0x7f69d8101780 2024-04-01T01:40:29.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.363+0000 7f69dd956640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f69d810af90 con 0x7f69d8103330 2024-04-01T01:40:29.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.363+0000 7f69dd956640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f69d810b290 con 0x7f69d8102970 2024-04-01T01:40:29.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.363+0000 7f69d67fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69d8102970 0x7f69d8195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.363+0000 7f69d77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69d8103330 0x7f69d81ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.363+0000 7f69d67fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69d8102970 0x7f69d8195d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37610/0 (socket says 172.21.15.70:37610) 2024-04-01T01:40:29.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.363+0000 7f69d67fc640 1 -- 172.21.15.70:0/2793752931 learned_addr learned my addr 172.21.15.70:0/2793752931 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:29.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.364+0000 7f69d77fe640 1 -- 172.21.15.70:0/2793752931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69d8102970 msgr2=0x7f69d8195d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.364+0000 7f69d77fe640 1 --2- 172.21.15.70:0/2793752931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69d8102970 0x7f69d8195d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.364+0000 7f69d77fe640 1 -- 172.21.15.70:0/2793752931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69d8101780 msgr2=0x7f69d81957d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.364+0000 7f69d77fe640 1 --2- 172.21.15.70:0/2793752931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69d8101780 0x7f69d81957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.364+0000 7f69d77fe640 1 -- 172.21.15.70:0/2793752931 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d81ae570 con 0x7f69d8103330 2024-04-01T01:40:29.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.364+0000 7f69d77fe640 1 --2- 172.21.15.70:0/2793752931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69d8103330 0x7f69d81ae030 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f69d0009bf0 tx=0x7f69d00376a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:29.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.365+0000 7f69dc954640 1 -- 172.21.15.70:0/2793752931 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d0044020 con 0x7f69d8103330 2024-04-01T01:40:29.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.365+0000 7f69dd956640 1 -- 172.21.15.70:0/2793752931 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69d81ae740 con 0x7f69d8103330 2024-04-01T01:40:29.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.365+0000 7f69dd956640 1 -- 172.21.15.70:0/2793752931 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69d81aeba0 con 0x7f69d8103330 2024-04-01T01:40:29.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.366+0000 7f69dc954640 1 -- 172.21.15.70:0/2793752931 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69d0010520 con 0x7f69d8103330 2024-04-01T01:40:29.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.367+0000 7f69dc954640 1 -- 172.21.15.70:0/2793752931 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d0035b80 con 0x7f69d8103330 2024-04-01T01:40:29.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.369+0000 7f69e0619740 1 Processor -- start 2024-04-01T01:40:29.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.371+0000 7f69dc954640 1 -- 172.21.15.70:0/2793752931 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f69d0047020 con 0x7f69d8103330 2024-04-01T01:40:29.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.372+0000 7f69dc954640 1 --2- 172.21.15.70:0/2793752931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f69c00a26a0 0x7f69c00a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.372+0000 7f69e0619740 1 -- start start 2024-04-01T01:40:29.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.373+0000 7f69dc954640 1 -- 172.21.15.70:0/2793752931 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f69d00f1340 con 0x7f69d8103330 2024-04-01T01:40:29.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.373+0000 7f69e0619740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba74a10 0x55d0eba74e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.373+0000 7f69e0619740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba75400 0x55d0eba79f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.373+0000 7f69e0619740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba7a570 0x55d0eba7ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.373+0000 7f69e0619740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d0eb9760b0 con 0x55d0eba74a10 2024-04-01T01:40:29.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.373+0000 7f69e0619740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d0eb9d8ea0 con 0x55d0eba7a570 2024-04-01T01:40:29.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.373+0000 7f69e0619740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d0eb981c00 con 0x55d0eba75400 2024-04-01T01:40:29.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.373+0000 7f69d67fc640 1 --2- 172.21.15.70:0/2793752931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f69c00a26a0 0x7f69c00a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.373+0000 7f69d77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba7a570 0x55d0eba7ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.373+0000 7f69d77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba7a570 0x55d0eba7ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:59106/0 (socket says 172.21.15.70:59106) 2024-04-01T01:40:29.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.373+0000 7f69d77fe640 1 -- 172.21.15.70:0/507068021 learned_addr learned my addr 172.21.15.70:0/507068021 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:29.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.374+0000 7f69d77fe640 1 -- 172.21.15.70:0/507068021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba75400 msgr2=0x55d0eba79f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:29.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.374+0000 7f69d77fe640 1 --2- 172.21.15.70:0/507068021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba75400 0x55d0eba79f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.374+0000 7f69d77fe640 1 -- 172.21.15.70:0/507068021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba74a10 msgr2=0x55d0eba74e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:29.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.374+0000 7f69d77fe640 1 --2- 172.21.15.70:0/507068021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba74a10 0x55d0eba74e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.374+0000 7f69d77fe640 1 -- 172.21.15.70:0/507068021 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0eb7b57e0 con 0x55d0eba7a570 2024-04-01T01:40:29.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.374+0000 7f69d77fe640 1 --2- 172.21.15.70:0/507068021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba7a570 0x55d0eba7ca00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f69d0037a60 tx=0x7f69d00bd330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5b91265dbec9d24 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:29.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.374+0000 7f69d67fc640 1 --2- 172.21.15.70:0/2793752931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f69c00a26a0 0x7f69c00a4b60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f69d8196ce0 tx=0x7f69c80113e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:29.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.375+0000 7f69bdffb640 1 -- 172.21.15.70:0/507068021 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d00be0d0 con 0x55d0eba7a570 2024-04-01T01:40:29.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.375+0000 7f69bdffb640 1 -- 172.21.15.70:0/507068021 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69d00bd9d0 con 0x55d0eba7a570 2024-04-01T01:40:29.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.375+0000 7f69bdffb640 1 -- 172.21.15.70:0/507068021 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d00bbd60 con 0x55d0eba7a570 2024-04-01T01:40:29.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.376+0000 7f69e0619740 1 -- 172.21.15.70:0/507068021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba7a570 msgr2=0x55d0eba7ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.376+0000 7f69e0619740 1 --2- 172.21.15.70:0/507068021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba7a570 0x55d0eba7ca00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f69d0037a60 tx=0x7f69d00bd330 comp rx=0 tx=0).stop 2024-04-01T01:40:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.376+0000 7f69e0619740 1 -- 172.21.15.70:0/507068021 shutdown_connections 2024-04-01T01:40:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.376+0000 7f69e0619740 1 --2- 172.21.15.70:0/507068021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba7a570 0x55d0eba7ca00 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.376+0000 7f69e0619740 1 --2- 172.21.15.70:0/507068021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba75400 0x55d0eba79f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.376+0000 7f69e0619740 1 --2- 172.21.15.70:0/507068021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba74a10 0x55d0eba74e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.376+0000 7f69e0619740 1 -- 172.21.15.70:0/507068021 >> 172.21.15.70:0/507068021 conn(0x55d0eba6ff80 msgr2=0x55d0eba70380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.377+0000 7f69e0619740 1 -- 172.21.15.70:0/507068021 shutdown_connections 2024-04-01T01:40:29.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.377+0000 7f69e0619740 1 -- 172.21.15.70:0/507068021 wait complete. 2024-04-01T01:40:29.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.377+0000 7f69e0619740 1 -- start start 2024-04-01T01:40:29.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.378+0000 7f69e0619740 1 Processor -- start 2024-04-01T01:40:29.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.378+0000 7f69e0619740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba74a10 0x55d0eba7eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.378+0000 7f69e0619740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba75400 0x55d0eba7f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.378+0000 7f69e0619740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba7a570 0x55d0eba860e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.378+0000 7f69e0619740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d0eb9773d0 con 0x55d0eba74a10 2024-04-01T01:40:29.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.378+0000 7f69e0619740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d0eb9d4f40 con 0x55d0eba75400 2024-04-01T01:40:29.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.378+0000 7f69e0619740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d0eba7d2e0 con 0x55d0eba7a570 2024-04-01T01:40:29.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.378+0000 7f69e0619740 10 client.0 _ll_register_callbacks cb 0x55d0eb994b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.378+0000 7f69d67fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba75400 0x55d0eba7f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.378+0000 7f69d6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba74a10 0x55d0eba7eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.378+0000 7f69d67fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba75400 0x55d0eba7f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:59108/0 (socket says 172.21.15.70:59108) 2024-04-01T01:40:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.378+0000 7f69d67fc640 1 -- 172.21.15.70:0/3874121420 learned_addr learned my addr 172.21.15.70:0/3874121420 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.378+0000 7f69d77fe640 1 --2- 172.21.15.70:0/3874121420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba7a570 0x55d0eba860e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.379+0000 7f69d77fe640 1 -- 172.21.15.70:0/3874121420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba75400 msgr2=0x55d0eba7f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.379+0000 7f69d77fe640 1 --2- 172.21.15.70:0/3874121420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba75400 0x55d0eba7f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.379+0000 7f69d77fe640 1 -- 172.21.15.70:0/3874121420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba74a10 msgr2=0x55d0eba7eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.379+0000 7f69d77fe640 1 --2- 172.21.15.70:0/3874121420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba74a10 0x55d0eba7eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.379+0000 7f69d77fe640 1 -- 172.21.15.70:0/3874121420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0eb815f70 con 0x55d0eba7a570 2024-04-01T01:40:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.379+0000 7f69d67fc640 1 --2- 172.21.15.70:0/3874121420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba75400 0x55d0eba7f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:40:29.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.380+0000 7f69d77fe640 1 --2- 172.21.15.70:0/3874121420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba7a570 0x55d0eba860e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x55d0eba7d170 tx=0x7f69d00c1c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5c4a9102d5f17e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:29.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.380+0000 7f69b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:40:29.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.381+0000 7f69b6ffd640 1 -- 172.21.15.70:0/3874121420 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d00c5f50 con 0x55d0eba7a570 2024-04-01T01:40:29.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.381+0000 7f69b6ffd640 1 -- 172.21.15.70:0/3874121420 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f69d00c60f0 con 0x55d0eba7a570 2024-04-01T01:40:29.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.381+0000 7f69e0619740 1 -- 172.21.15.70:0/3874121420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69d00bd9d0 con 0x55d0eba7a570 2024-04-01T01:40:29.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.381+0000 7f69b6ffd640 1 -- 172.21.15.70:0/3874121420 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f69d00cb050 con 0x55d0eba7a570 2024-04-01T01:40:29.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.382+0000 7f69e0619740 10 client.27445 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:29.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.382+0000 7f69e0619740 1 -- 172.21.15.70:0/3874121420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d0eb6ad170 con 0x55d0eba7a570 2024-04-01T01:40:29.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.383+0000 7f69b6ffd640 1 -- 172.21.15.70:0/3874121420 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69d00bd9d0 con 0x55d0eba7a570 2024-04-01T01:40:29.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.383+0000 7f69b6ffd640 1 -- 172.21.15.70:0/3874121420 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f69d00b9720 con 0x55d0eba7a570 2024-04-01T01:40:29.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.383+0000 7f69e0619740 10 client.27445 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:29.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.383+0000 7f69e0619740 10 client.27445 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:40:29.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.383+0000 7f69e0619740 20 client.27445 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:29.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.383+0000 7f69e0619740 1 --2- 172.21.15.70:0/3874121420 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d0eba8a680 0x55d0eba8caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.383+0000 7f69e0619740 4 client.27445 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:40:29.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.383+0000 7f69e0619740 1 -- 172.21.15.70:0/3874121420 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0eb9e63e0 con 0x55d0eba8a680 2024-04-01T01:40:29.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.384+0000 7f69d6ffd640 1 --2- 172.21.15.70:0/3874121420 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d0eba8a680 0x55d0eba8caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.386+0000 7f69d6ffd640 1 --2- 172.21.15.70:0/3874121420 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d0eba8a680 0x55d0eba8caa0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb96871140a3cd2b server_cookie=64ed949e3fbb7f71 in_seq=0 out_seq=0 2024-04-01T01:40:29.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.386+0000 7f69b6ffd640 10 client.27445 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:29.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.386+0000 7f69b6ffd640 1 -- 172.21.15.70:0/3874121420 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d0eb9e63e0 con 0x55d0eba8a680 2024-04-01T01:40:29.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.386+0000 7f69b6ffd640 10 client.27445 handle_command_reply: tid=0 2024-04-01T01:40:29.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.387+0000 7f69e0619740 1 client.27445 shutdown 2024-04-01T01:40:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.388+0000 7f69e0619740 1 -- 172.21.15.70:0/3874121420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba7a570 msgr2=0x55d0eba860e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.388+0000 7f69e0619740 1 --2- 172.21.15.70:0/3874121420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba7a570 0x55d0eba860e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x55d0eba7d170 tx=0x7f69d00c1c10 comp rx=0 tx=0).stop 2024-04-01T01:40:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.389+0000 7f69e0619740 1 -- 172.21.15.70:0/3874121420 shutdown_connections 2024-04-01T01:40:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.389+0000 7f69e0619740 1 --2- 172.21.15.70:0/3874121420 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d0eba8a680 0x55d0eba8caa0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.389+0000 7f69e0619740 1 --2- 172.21.15.70:0/3874121420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba7a570 0x55d0eba860e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.389+0000 7f69e0619740 1 --2- 172.21.15.70:0/3874121420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba75400 0x55d0eba7f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.389+0000 7f69e0619740 1 --2- 172.21.15.70:0/3874121420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba74a10 0x55d0eba7eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.389+0000 7f69e0619740 1 -- 172.21.15.70:0/3874121420 >> 172.21.15.70:0/3874121420 conn(0x55d0eba6ff80 msgr2=0x55d0eba6f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.389+0000 7f69b6ffd640 0 client.27445 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.390+0000 7f69e0619740 1 -- 172.21.15.70:0/3874121420 shutdown_connections 2024-04-01T01:40:29.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.391+0000 7f69e0619740 1 -- 172.21.15.70:0/3874121420 wait complete. 2024-04-01T01:40:29.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.391+0000 7f69e0619740 20 client.27445 trim_cache size 0 max 16384 2024-04-01T01:40:29.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.429+0000 7f69e0619740 1 Processor -- start 2024-04-01T01:40:29.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.429+0000 7f69e0619740 1 -- start start 2024-04-01T01:40:29.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.429+0000 7f69e0619740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba74a10 0x55d0eba738f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.429+0000 7f69e0619740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba75400 0x55d0eba860a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.429+0000 7f69e0619740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba7a570 0x55d0eba91360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.430+0000 7f69e0619740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f69c400fdf0 con 0x55d0eba7a570 2024-04-01T01:40:29.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.430+0000 7f69e0619740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f69c4011930 con 0x55d0eba75400 2024-04-01T01:40:29.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.430+0000 7f69e0619740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f69c40104c0 con 0x55d0eba74a10 2024-04-01T01:40:29.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.430+0000 7f69d77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba7a570 0x55d0eba91360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.430+0000 7f69d77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba7a570 0x55d0eba91360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47692/0 (socket says 172.21.15.70:47692) 2024-04-01T01:40:29.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.430+0000 7f69d77fe640 1 -- 172.21.15.70:0/484743799 learned_addr learned my addr 172.21.15.70:0/484743799 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:29.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.430+0000 7f69d6ffd640 1 --2- 172.21.15.70:0/484743799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba74a10 0x55d0eba738f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.430+0000 7f69d67fc640 1 --2- 172.21.15.70:0/484743799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba75400 0x55d0eba860a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.430+0000 7f69d77fe640 1 -- 172.21.15.70:0/484743799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba74a10 msgr2=0x55d0eba738f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:29.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.430+0000 7f69d77fe640 1 --2- 172.21.15.70:0/484743799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba74a10 0x55d0eba738f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.430+0000 7f69d77fe640 1 -- 172.21.15.70:0/484743799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba75400 msgr2=0x55d0eba860a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:29.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.430+0000 7f69d77fe640 1 --2- 172.21.15.70:0/484743799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba75400 0x55d0eba860a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.430+0000 7f69d77fe640 1 -- 172.21.15.70:0/484743799 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0eba8ce10 con 0x55d0eba7a570 2024-04-01T01:40:29.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.430+0000 7f69d77fe640 1 --2- 172.21.15.70:0/484743799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba7a570 0x55d0eba91360 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f69d0036040 tx=0x7f69d00bd330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e74de4b1ba11e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:29.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.431+0000 7f69bd7fa640 1 -- 172.21.15.70:0/484743799 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d00eea90 con 0x55d0eba7a570 2024-04-01T01:40:29.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.431+0000 7f69bd7fa640 1 -- 172.21.15.70:0/484743799 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69d00c5f50 con 0x55d0eba7a570 2024-04-01T01:40:29.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.431+0000 7f69bd7fa640 1 -- 172.21.15.70:0/484743799 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d0004da0 con 0x55d0eba7a570 2024-04-01T01:40:29.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.432+0000 7f69e0619740 1 -- 172.21.15.70:0/484743799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba7a570 msgr2=0x55d0eba91360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.432+0000 7f69e0619740 1 --2- 172.21.15.70:0/484743799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba7a570 0x55d0eba91360 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f69d0036040 tx=0x7f69d00bd330 comp rx=0 tx=0).stop 2024-04-01T01:40:29.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.432+0000 7f69e0619740 1 -- 172.21.15.70:0/484743799 shutdown_connections 2024-04-01T01:40:29.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.432+0000 7f69e0619740 1 --2- 172.21.15.70:0/484743799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba7a570 0x55d0eba91360 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.432+0000 7f69e0619740 1 --2- 172.21.15.70:0/484743799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba75400 0x55d0eba860a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.432+0000 7f69e0619740 1 --2- 172.21.15.70:0/484743799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba74a10 0x55d0eba738f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.432+0000 7f69e0619740 1 -- 172.21.15.70:0/484743799 >> 172.21.15.70:0/484743799 conn(0x55d0eba6ff80 msgr2=0x55d0eba731e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:29.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.433+0000 7f69e0619740 1 -- 172.21.15.70:0/484743799 shutdown_connections 2024-04-01T01:40:29.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.433+0000 7f69e0619740 1 -- 172.21.15.70:0/484743799 wait complete. 2024-04-01T01:40:29.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.433+0000 7f69e0619740 1 -- start start 2024-04-01T01:40:29.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.433+0000 7f69e0619740 1 Processor -- start 2024-04-01T01:40:29.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69e0619740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba74a10 0x55d0eba83f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69e0619740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba75400 0x55d0eba84450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69e0619740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba7a570 0x55d0eba85460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69e0619740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f69c4011260 con 0x55d0eba75400 2024-04-01T01:40:29.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69e0619740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f69c4010b90 con 0x55d0eba74a10 2024-04-01T01:40:29.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69e0619740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f69c400ddd0 con 0x55d0eba7a570 2024-04-01T01:40:29.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69e0619740 10 client.0 _ll_register_callbacks cb 0x7f69d00c1b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:29.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69d6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba74a10 0x55d0eba83f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69d6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba74a10 0x55d0eba83f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:59140/0 (socket says 172.21.15.70:59140) 2024-04-01T01:40:29.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69d6ffd640 1 -- 172.21.15.70:0/3585941440 learned_addr learned my addr 172.21.15.70:0/3585941440 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:29.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69d77fe640 1 --2- 172.21.15.70:0/3585941440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba7a570 0x55d0eba85460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69d67fc640 1 --2- 172.21.15.70:0/3585941440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba75400 0x55d0eba84450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69d6ffd640 1 -- 172.21.15.70:0/3585941440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba7a570 msgr2=0x55d0eba85460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:29.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69d6ffd640 1 --2- 172.21.15.70:0/3585941440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba7a570 0x55d0eba85460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69d6ffd640 1 -- 172.21.15.70:0/3585941440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba75400 msgr2=0x55d0eba84450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:29.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69d6ffd640 1 --2- 172.21.15.70:0/3585941440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba75400 0x55d0eba84450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69d6ffd640 1 -- 172.21.15.70:0/3585941440 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0eba85a50 con 0x55d0eba74a10 2024-04-01T01:40:29.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.434+0000 7f69d67fc640 1 --2- 172.21.15.70:0/3585941440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba75400 0x55d0eba84450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:40:29.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.435+0000 7f69d6ffd640 1 --2- 172.21.15.70:0/3585941440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba74a10 0x55d0eba83f10 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f69cc006e80 tx=0x7f69cc016790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a35d105bd85f147 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:29.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.435+0000 7f69b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:40:29.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.435+0000 7f69b6ffd640 1 -- 172.21.15.70:0/3585941440 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc020020 con 0x55d0eba74a10 2024-04-01T01:40:29.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.435+0000 7f69b6ffd640 1 -- 172.21.15.70:0/3585941440 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f69cc005a30 con 0x55d0eba74a10 2024-04-01T01:40:29.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.435+0000 7f69e0619740 1 -- 172.21.15.70:0/3585941440 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69d00c5f50 con 0x55d0eba74a10 2024-04-01T01:40:29.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.436+0000 7f69b6ffd640 1 -- 172.21.15.70:0/3585941440 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f69cc005c30 con 0x55d0eba74a10 2024-04-01T01:40:29.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.436+0000 7f69e0619740 10 client.27608 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:29.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.436+0000 7f69e0619740 1 -- 172.21.15.70:0/3585941440 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d0ebaf1200 con 0x55d0eba74a10 2024-04-01T01:40:29.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.436+0000 7f69b6ffd640 1 -- 172.21.15.70:0/3585941440 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f69d00c5f50 con 0x55d0eba74a10 2024-04-01T01:40:29.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.437+0000 7f69b6ffd640 1 -- 172.21.15.70:0/3585941440 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f69cc006130 con 0x55d0eba74a10 2024-04-01T01:40:29.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.437+0000 7f69e0619740 10 client.27608 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:29.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.437+0000 7f69e0619740 10 client.27608 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:40:29.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.437+0000 7f69e0619740 20 client.27608 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:29.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.437+0000 7f69e0619740 1 --2- 172.21.15.70:0/3585941440 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d0ebaf1be0 0x55d0ebaf4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.437+0000 7f69e0619740 4 client.27608 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:40:29.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.437+0000 7f69e0619740 1 -- 172.21.15.70:0/3585941440 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d0eb9e0730 con 0x55d0ebaf1be0 2024-04-01T01:40:29.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.437+0000 7f69d67fc640 1 --2- 172.21.15.70:0/3585941440 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d0ebaf1be0 0x55d0ebaf4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.438+0000 7f69d67fc640 1 --2- 172.21.15.70:0/3585941440 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d0ebaf1be0 0x55d0ebaf4000 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53453217faa41ef3 server_cookie=e83d557304784c99 in_seq=0 out_seq=0 2024-04-01T01:40:29.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.438+0000 7f69b6ffd640 10 client.27608 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:29.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.440+0000 7f69b6ffd640 1 -- 172.21.15.70:0/3585941440 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d0eb9e0730 con 0x55d0ebaf1be0 2024-04-01T01:40:29.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.440+0000 7f69b6ffd640 10 client.27608 handle_command_reply: tid=0 2024-04-01T01:40:29.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.441+0000 7f69e0619740 1 client.27608 shutdown 2024-04-01T01:40:29.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.442+0000 7f69e0619740 1 -- 172.21.15.70:0/3585941440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba74a10 msgr2=0x55d0eba83f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.442+0000 7f69e0619740 1 --2- 172.21.15.70:0/3585941440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba74a10 0x55d0eba83f10 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f69cc006e80 tx=0x7f69cc016790 comp rx=0 tx=0).stop 2024-04-01T01:40:29.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.442+0000 7f69e0619740 1 -- 172.21.15.70:0/3585941440 shutdown_connections 2024-04-01T01:40:29.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.442+0000 7f69e0619740 1 --2- 172.21.15.70:0/3585941440 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d0ebaf1be0 0x55d0ebaf4000 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.442+0000 7f69e0619740 1 --2- 172.21.15.70:0/3585941440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d0eba7a570 0x55d0eba85460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.442+0000 7f69e0619740 1 --2- 172.21.15.70:0/3585941440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d0eba75400 0x55d0eba84450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.442+0000 7f69e0619740 1 --2- 172.21.15.70:0/3585941440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d0eba74a10 0x55d0eba83f10 secure :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f69cc006e80 tx=0x7f69cc016790 comp rx=0 tx=0).stop 2024-04-01T01:40:29.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.442+0000 7f69e0619740 1 -- 172.21.15.70:0/3585941440 >> 172.21.15.70:0/3585941440 conn(0x55d0eba6ff80 msgr2=0x55d0eba73a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:29.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.443+0000 7f69b6ffd640 0 client.27608 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:29.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.446+0000 7f69e0619740 1 -- 172.21.15.70:0/3585941440 shutdown_connections 2024-04-01T01:40:29.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.446+0000 7f69e0619740 1 -- 172.21.15.70:0/3585941440 wait complete. 2024-04-01T01:40:29.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.446+0000 7f69e0619740 20 client.27608 trim_cache size 0 max 16384 2024-04-01T01:40:29.446 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:40:29.446 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:40:29.446 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "8beb3e58-aa81-4197-8e44-80d70157508c", 2024-04-01T01:40:29.446 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:40:29.446 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:40:29.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.447+0000 7f69dd956640 1 -- 172.21.15.70:0/2793752931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f69c00a26a0 msgr2=0x7f69c00a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.447+0000 7f69dd956640 1 --2- 172.21.15.70:0/2793752931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f69c00a26a0 0x7f69c00a4b60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f69d8196ce0 tx=0x7f69c80113e0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.447+0000 7f69dd956640 1 -- 172.21.15.70:0/2793752931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69d8103330 msgr2=0x7f69d81ae030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.447+0000 7f69dd956640 1 --2- 172.21.15.70:0/2793752931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69d8103330 0x7f69d81ae030 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f69d0009bf0 tx=0x7f69d00376a0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.448+0000 7f69dd956640 1 -- 172.21.15.70:0/2793752931 shutdown_connections 2024-04-01T01:40:29.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.448+0000 7f69dd956640 1 --2- 172.21.15.70:0/2793752931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69d8103330 0x7f69d81ae030 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.448+0000 7f69dd956640 1 --2- 172.21.15.70:0/2793752931 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f69c00a26a0 0x7f69c00a4b60 secure :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f69d8196ce0 tx=0x7f69c80113e0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.448+0000 7f69dd956640 1 --2- 172.21.15.70:0/2793752931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69d8102970 0x7f69d8195d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.448+0000 7f69dd956640 1 --2- 172.21.15.70:0/2793752931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69d8101780 0x7f69d81957d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.448+0000 7f69dd956640 1 -- 172.21.15.70:0/2793752931 >> 172.21.15.70:0/2793752931 conn(0x7f69d80fccf0 msgr2=0x7f69d8105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:29.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.448+0000 7f69dd956640 1 -- 172.21.15.70:0/2793752931 shutdown_connections 2024-04-01T01:40:29.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.449+0000 7f69dd956640 1 -- 172.21.15.70:0/2793752931 wait complete. 2024-04-01T01:40:29.468 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:40:29.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:40:29.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.663+0000 7f005d3bc640 1 -- 172.21.15.70:0/2808718846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0058109940 msgr2=0x7f00581101d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.663+0000 7f005d3bc640 1 --2- 172.21.15.70:0/2808718846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0058109940 0x7f00581101d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0050009ca0 tx=0x7f005002c120 comp rx=0 tx=0).stop 2024-04-01T01:40:29.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.663+0000 7f005d3bc640 1 -- 172.21.15.70:0/2808718846 shutdown_connections 2024-04-01T01:40:29.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.663+0000 7f005d3bc640 1 --2- 172.21.15.70:0/2808718846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0058109940 0x7f00581101d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.663+0000 7f005d3bc640 1 --2- 172.21.15.70:0/2808718846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0058108f80 0x7f0058109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.663+0000 7f005d3bc640 1 --2- 172.21.15.70:0/2808718846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0058107d90 0x7f00581081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.663+0000 7f005d3bc640 1 -- 172.21.15.70:0/2808718846 >> 172.21.15.70:0/2808718846 conn(0x7f0058076d20 msgr2=0x7f0058079160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.663+0000 7f005d3bc640 1 -- 172.21.15.70:0/2808718846 shutdown_connections 2024-04-01T01:40:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.663+0000 7f005d3bc640 1 -- 172.21.15.70:0/2808718846 wait complete. 2024-04-01T01:40:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.664+0000 7f005d3bc640 1 Processor -- start 2024-04-01T01:40:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.665+0000 7f005d3bc640 1 -- start start 2024-04-01T01:40:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.665+0000 7f005d3bc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0058107d90 0x7f0058070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.665+0000 7f005d3bc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0058108f80 0x7f0058071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.665+0000 7f005d3bc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0058109940 0x7f0058072470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.665+0000 7f005d3bc640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f005806b640 con 0x7f0058107d90 2024-04-01T01:40:29.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.665+0000 7f005d3bc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f005806b4c0 con 0x7f0058109940 2024-04-01T01:40:29.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.665+0000 7f005d3bc640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f005806b7c0 con 0x7f0058108f80 2024-04-01T01:40:29.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.665+0000 7f0056ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0058107d90 0x7f0058070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.665+0000 7f0056ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0058107d90 0x7f0058070ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47714/0 (socket says 172.21.15.70:47714) 2024-04-01T01:40:29.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.665+0000 7f0056ffd640 1 -- 172.21.15.70:0/3311955233 learned_addr learned my addr 172.21.15.70:0/3311955233 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:29.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.665+0000 7f00577fe640 1 --2- 172.21.15.70:0/3311955233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0058109940 0x7f0058072470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.666+0000 7f0056ffd640 1 -- 172.21.15.70:0/3311955233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0058108f80 msgr2=0x7f0058071430 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:40:29.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.666+0000 7f0056ffd640 1 --2- 172.21.15.70:0/3311955233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0058108f80 0x7f0058071430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.666+0000 7f0056ffd640 1 -- 172.21.15.70:0/3311955233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0058109940 msgr2=0x7f0058072470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.666+0000 7f0056ffd640 1 --2- 172.21.15.70:0/3311955233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0058109940 0x7f0058072470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.666+0000 7f0056ffd640 1 -- 172.21.15.70:0/3311955233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0058072ae0 con 0x7f0058107d90 2024-04-01T01:40:29.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.667+0000 7f0056ffd640 1 --2- 172.21.15.70:0/3311955233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0058107d90 0x7f0058070ef0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f004800e9c0 tx=0x7f004800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:29.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.670+0000 7f003ffff640 1 -- 172.21.15.70:0/3311955233 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004800cd80 con 0x7f0058107d90 2024-04-01T01:40:29.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.670+0000 7f003ffff640 1 -- 172.21.15.70:0/3311955233 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00480035f0 con 0x7f0058107d90 2024-04-01T01:40:29.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.670+0000 7f003ffff640 1 -- 172.21.15.70:0/3311955233 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0048010690 con 0x7f0058107d90 2024-04-01T01:40:29.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.670+0000 7f005d3bc640 1 -- 172.21.15.70:0/3311955233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0058072dd0 con 0x7f0058107d90 2024-04-01T01:40:29.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.671+0000 7f005d3bc640 1 -- 172.21.15.70:0/3311955233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00580732d0 con 0x7f0058107d90 2024-04-01T01:40:29.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.675+0000 7f003ffff640 1 -- 172.21.15.70:0/3311955233 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f00480037a0 con 0x7f0058107d90 2024-04-01T01:40:29.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.675+0000 7f005ff9f740 1 Processor -- start 2024-04-01T01:40:29.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.676+0000 7f003ffff640 1 --2- 172.21.15.70:0/3311955233 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00400a26b0 0x7f00400a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.676+0000 7f003ffff640 1 -- 172.21.15.70:0/3311955233 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f00480ceee0 con 0x7f0058107d90 2024-04-01T01:40:29.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.678+0000 7f005ff9f740 1 -- start start 2024-04-01T01:40:29.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.678+0000 7f005ff9f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f52b0 0x55cc967f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.678+0000 7f005ff9f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967f5ca0 0x55cc967fa7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.678+0000 7f005ff9f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967fae10 0x55cc967fd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.678+0000 7f005ff9f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cc9675a180 con 0x55cc967f52b0 2024-04-01T01:40:29.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.678+0000 7f005ff9f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55cc96702b70 con 0x55cc967fae10 2024-04-01T01:40:29.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.678+0000 7f005ff9f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cc96756220 con 0x55cc967f5ca0 2024-04-01T01:40:29.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.678+0000 7f0056ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f52b0 0x55cc967f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.678+0000 7f0056ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f52b0 0x55cc967f56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47730/0 (socket says 172.21.15.70:47730) 2024-04-01T01:40:29.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.678+0000 7f0056ffd640 1 -- 172.21.15.70:0/975204212 learned_addr learned my addr 172.21.15.70:0/975204212 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:29.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.679+0000 7f0056ffd640 1 -- 172.21.15.70:0/975204212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967f5ca0 msgr2=0x55cc967fa7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:29.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.679+0000 7f0056ffd640 1 --2- 172.21.15.70:0/975204212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967f5ca0 0x55cc967fa7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.679+0000 7f0056ffd640 1 -- 172.21.15.70:0/975204212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967fae10 msgr2=0x55cc967fd2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:40:29.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.679+0000 7f0056ffd640 1 --2- 172.21.15.70:0/975204212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967fae10 0x55cc967fd2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.679+0000 7f0056ffd640 1 -- 172.21.15.70:0/975204212 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc967167c0 con 0x55cc967f52b0 2024-04-01T01:40:29.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.679+0000 7f00577fe640 1 --2- 172.21.15.70:0/3311955233 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00400a26b0 0x7f00400a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.679+0000 7f0056ffd640 1 --2- 172.21.15.70:0/975204212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f52b0 0x55cc967f56d0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f004809b500 tx=0x7f004809af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=653e57dcaafceb55 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:29.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.683+0000 7f003d7fa640 1 -- 172.21.15.70:0/975204212 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00480a3f80 con 0x55cc967f52b0 2024-04-01T01:40:29.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.683+0000 7f003d7fa640 1 -- 172.21.15.70:0/975204212 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0048016df0 con 0x55cc967f52b0 2024-04-01T01:40:29.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.685+0000 7f005ff9f740 1 -- 172.21.15.70:0/975204212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f52b0 msgr2=0x55cc967f56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.685+0000 7f005ff9f740 1 --2- 172.21.15.70:0/975204212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f52b0 0x55cc967f56d0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f004809b500 tx=0x7f004809af90 comp rx=0 tx=0).stop 2024-04-01T01:40:29.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.685+0000 7f00577fe640 1 --2- 172.21.15.70:0/3311955233 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00400a26b0 0x7f00400a4b70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0050037040 tx=0x7f00500324f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:29.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.685+0000 7f005ff9f740 1 -- 172.21.15.70:0/975204212 shutdown_connections 2024-04-01T01:40:29.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.685+0000 7f005ff9f740 1 --2- 172.21.15.70:0/975204212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967fae10 0x55cc967fd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.685+0000 7f005ff9f740 1 --2- 172.21.15.70:0/975204212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967f5ca0 0x55cc967fa7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.685+0000 7f005ff9f740 1 --2- 172.21.15.70:0/975204212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f52b0 0x55cc967f56d0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.685+0000 7f005ff9f740 1 -- 172.21.15.70:0/975204212 >> 172.21.15.70:0/975204212 conn(0x55cc967f0820 msgr2=0x55cc967f0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:29.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.688+0000 7f005ff9f740 1 -- 172.21.15.70:0/975204212 shutdown_connections 2024-04-01T01:40:29.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.688+0000 7f005ff9f740 1 -- 172.21.15.70:0/975204212 wait complete. 2024-04-01T01:40:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.689+0000 7f005ff9f740 1 -- start start 2024-04-01T01:40:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.689+0000 7f005ff9f740 1 Processor -- start 2024-04-01T01:40:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.689+0000 7f005ff9f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f52b0 0x55cc967ff810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.689+0000 7f005ff9f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f5ca0 0x55cc967ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.689+0000 7f005ff9f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967fae10 0x55cc96806d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.689+0000 7f005ff9f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cc967fddb0 con 0x55cc967f5ca0 2024-04-01T01:40:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.689+0000 7f005ff9f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55cc966f84a0 con 0x55cc967f52b0 2024-04-01T01:40:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.689+0000 7f005ff9f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cc967fdf30 con 0x55cc967fae10 2024-04-01T01:40:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.689+0000 7f0056ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f52b0 0x55cc967ff810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.689+0000 7f0056ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f52b0 0x55cc967ff810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:59164/0 (socket says 172.21.15.70:59164) 2024-04-01T01:40:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.689+0000 7f0056ffd640 1 -- 172.21.15.70:0/2128895688 learned_addr learned my addr 172.21.15.70:0/2128895688 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.690+0000 7f00577fe640 1 --2- 172.21.15.70:0/2128895688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967fae10 0x55cc96806d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.690+0000 7f005ff9f740 10 client.0 _ll_register_callbacks cb 0x55cc966fbd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.690+0000 7f00577fe640 1 -- 172.21.15.70:0/2128895688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f52b0 msgr2=0x55cc967ff810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.690+0000 7f00577fe640 1 --2- 172.21.15.70:0/2128895688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f52b0 0x55cc967ff810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.690+0000 7f00577fe640 1 -- 172.21.15.70:0/2128895688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f5ca0 msgr2=0x55cc967ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.690+0000 7f00577fe640 1 --2- 172.21.15.70:0/2128895688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f5ca0 0x55cc967ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.690+0000 7f00577fe640 1 -- 172.21.15.70:0/2128895688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc965387d0 con 0x55cc967fae10 2024-04-01T01:40:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.690+0000 7f00577fe640 1 --2- 172.21.15.70:0/2128895688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967fae10 0x55cc96806d90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0050009bf0 tx=0x7f00500095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=debb5fe4443f1a82 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.690+0000 7f003aff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:40:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.691+0000 7f003aff5640 1 -- 172.21.15.70:0/2128895688 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050004450 con 0x55cc967fae10 2024-04-01T01:40:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.691+0000 7f005ff9f740 1 -- 172.21.15.70:0/2128895688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0048016df0 con 0x55cc967fae10 2024-04-01T01:40:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.691+0000 7f003aff5640 1 -- 172.21.15.70:0/2128895688 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f00500045f0 con 0x55cc967fae10 2024-04-01T01:40:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.691+0000 7f003aff5640 1 -- 172.21.15.70:0/2128895688 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f005007a050 con 0x55cc967fae10 2024-04-01T01:40:29.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.691+0000 7f005ff9f740 10 client.27454 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:29.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.692+0000 7f005ff9f740 1 -- 172.21.15.70:0/2128895688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cc9646e3f0 con 0x55cc967fae10 2024-04-01T01:40:29.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.692+0000 7f003aff5640 1 -- 172.21.15.70:0/2128895688 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0048016df0 con 0x55cc967fae10 2024-04-01T01:40:29.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.693+0000 7f003aff5640 1 -- 172.21.15.70:0/2128895688 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f005006b070 con 0x55cc967fae10 2024-04-01T01:40:29.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.693+0000 7f005ff9f740 10 client.27454 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:29.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.693+0000 7f005ff9f740 10 client.27454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:40:29.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.693+0000 7f005ff9f740 20 client.27454 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:29.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.693+0000 7f005ff9f740 1 --2- 172.21.15.70:0/2128895688 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cc9680b720 0x55cc9680dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.693+0000 7f005ff9f740 4 client.27454 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:40:29.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.693+0000 7f005ff9f740 1 -- 172.21.15.70:0/2128895688 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc967676c0 con 0x55cc9680b720 2024-04-01T01:40:29.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.693+0000 7f0056ffd640 1 --2- 172.21.15.70:0/2128895688 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cc9680b720 0x55cc9680dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.694+0000 7f0056ffd640 1 --2- 172.21.15.70:0/2128895688 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cc9680b720 0x55cc9680dbc0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f9d9b0a89e636bc server_cookie=8a092a88176c3c72 in_seq=0 out_seq=0 2024-04-01T01:40:29.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.694+0000 7f003aff5640 10 client.27454 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:29.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.695+0000 7f003aff5640 1 -- 172.21.15.70:0/2128895688 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cc967676c0 con 0x55cc9680b720 2024-04-01T01:40:29.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.695+0000 7f003aff5640 10 client.27454 handle_command_reply: tid=0 2024-04-01T01:40:29.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.695+0000 7f005ff9f740 1 client.27454 shutdown 2024-04-01T01:40:29.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.696+0000 7f005ff9f740 1 -- 172.21.15.70:0/2128895688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967fae10 msgr2=0x55cc96806d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.696+0000 7f005ff9f740 1 --2- 172.21.15.70:0/2128895688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967fae10 0x55cc96806d90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0050009bf0 tx=0x7f00500095c0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.698+0000 7f005ff9f740 1 -- 172.21.15.70:0/2128895688 shutdown_connections 2024-04-01T01:40:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.698+0000 7f005ff9f740 1 --2- 172.21.15.70:0/2128895688 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cc9680b720 0x55cc9680dbc0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.698+0000 7f005ff9f740 1 --2- 172.21.15.70:0/2128895688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967fae10 0x55cc96806d90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.698+0000 7f005ff9f740 1 --2- 172.21.15.70:0/2128895688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f5ca0 0x55cc967ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.698+0000 7f005ff9f740 1 --2- 172.21.15.70:0/2128895688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f52b0 0x55cc967ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.698+0000 7f005ff9f740 1 -- 172.21.15.70:0/2128895688 >> 172.21.15.70:0/2128895688 conn(0x55cc967f0820 msgr2=0x55cc967efc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.698+0000 7f003aff5640 0 client.27454 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:29.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.700+0000 7f005ff9f740 1 -- 172.21.15.70:0/2128895688 shutdown_connections 2024-04-01T01:40:29.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.700+0000 7f005ff9f740 1 -- 172.21.15.70:0/2128895688 wait complete. 2024-04-01T01:40:29.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.700+0000 7f005ff9f740 20 client.27454 trim_cache size 0 max 16384 2024-04-01T01:40:29.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.736+0000 7f005ff9f740 1 Processor -- start 2024-04-01T01:40:29.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.736+0000 7f005ff9f740 1 -- start start 2024-04-01T01:40:29.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.737+0000 7f005ff9f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f52b0 0x55cc967efaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.737+0000 7f005ff9f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f5ca0 0x55cc96806d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.737+0000 7f005ff9f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967fae10 0x55cc968124c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.737+0000 7f005ff9f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f004400fd10 con 0x55cc967f52b0 2024-04-01T01:40:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.737+0000 7f00567fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f5ca0 0x55cc96806d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.737+0000 7f00567fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f5ca0 0x55cc96806d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:59166/0 (socket says 172.21.15.70:59166) 2024-04-01T01:40:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.737+0000 7f00567fc640 1 -- 172.21.15.70:0/1968763843 learned_addr learned my addr 172.21.15.70:0/1968763843 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.737+0000 7f00577fe640 1 --2- 172.21.15.70:0/1968763843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967fae10 0x55cc968124c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.737+0000 7f005ff9f740 1 -- 172.21.15.70:0/1968763843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0044011850 con 0x55cc967f5ca0 2024-04-01T01:40:29.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.737+0000 7f005ff9f740 1 -- 172.21.15.70:0/1968763843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f00440103e0 con 0x55cc967fae10 2024-04-01T01:40:29.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.738+0000 7f0056ffd640 1 --2- 172.21.15.70:0/1968763843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f52b0 0x55cc967efaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.738+0000 7f0056ffd640 1 -- 172.21.15.70:0/1968763843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967fae10 msgr2=0x55cc968124c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:29.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.738+0000 7f0056ffd640 1 --2- 172.21.15.70:0/1968763843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967fae10 0x55cc968124c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.738+0000 7f0056ffd640 1 -- 172.21.15.70:0/1968763843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f5ca0 msgr2=0x55cc96806d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:29.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.738+0000 7f0056ffd640 1 --2- 172.21.15.70:0/1968763843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f5ca0 0x55cc96806d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.738+0000 7f0056ffd640 1 -- 172.21.15.70:0/1968763843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc9642e0f0 con 0x55cc967f52b0 2024-04-01T01:40:29.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.739+0000 7f0056ffd640 1 --2- 172.21.15.70:0/1968763843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f52b0 0x55cc967efaa0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f004800cfd0 tx=0x7f00480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdbefafcb3a72605 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:29.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.739+0000 7f003cff9640 1 -- 172.21.15.70:0/1968763843 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0048016df0 con 0x55cc967f52b0 2024-04-01T01:40:29.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.740+0000 7f003cff9640 1 -- 172.21.15.70:0/1968763843 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0048099870 con 0x55cc967f52b0 2024-04-01T01:40:29.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.740+0000 7f003cff9640 1 -- 172.21.15.70:0/1968763843 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00480aae20 con 0x55cc967f52b0 2024-04-01T01:40:29.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.741+0000 7f005ff9f740 1 -- 172.21.15.70:0/1968763843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f52b0 msgr2=0x55cc967efaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.741+0000 7f005ff9f740 1 --2- 172.21.15.70:0/1968763843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f52b0 0x55cc967efaa0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f004800cfd0 tx=0x7f00480046c0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.742+0000 7f005ff9f740 1 -- 172.21.15.70:0/1968763843 shutdown_connections 2024-04-01T01:40:29.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.742+0000 7f005ff9f740 1 --2- 172.21.15.70:0/1968763843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967fae10 0x55cc968124c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.742+0000 7f005ff9f740 1 --2- 172.21.15.70:0/1968763843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f5ca0 0x55cc96806d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.742+0000 7f005ff9f740 1 --2- 172.21.15.70:0/1968763843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967f52b0 0x55cc967efaa0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.742+0000 7f005ff9f740 1 -- 172.21.15.70:0/1968763843 >> 172.21.15.70:0/1968763843 conn(0x55cc967f0820 msgr2=0x55cc967ef390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:29.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.743+0000 7f005ff9f740 1 -- 172.21.15.70:0/1968763843 shutdown_connections 2024-04-01T01:40:29.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.743+0000 7f005ff9f740 1 -- 172.21.15.70:0/1968763843 wait complete. 2024-04-01T01:40:29.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.743+0000 7f005ff9f740 1 -- start start 2024-04-01T01:40:29.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.743+0000 7f005ff9f740 1 Processor -- start 2024-04-01T01:40:29.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.744+0000 7f005ff9f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967f52b0 0x55cc96804dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.744+0000 7f005ff9f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f5ca0 0x55cc96805300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.744+0000 7f005ff9f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967fae10 0x55cc96806310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.744+0000 7f005ff9f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0044011180 con 0x55cc967fae10 2024-04-01T01:40:29.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.744+0000 7f005ff9f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0044010ab0 con 0x55cc967f5ca0 2024-04-01T01:40:29.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.744+0000 7f005ff9f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f004400dcf0 con 0x55cc967f52b0 2024-04-01T01:40:29.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.744+0000 7f005ff9f740 10 client.0 _ll_register_callbacks cb 0x7f0050043040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:29.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.744+0000 7f00567fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f5ca0 0x55cc96805300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.744+0000 7f00577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967fae10 0x55cc96806310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.744+0000 7f00577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967fae10 0x55cc96806310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47740/0 (socket says 172.21.15.70:47740) 2024-04-01T01:40:29.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.744+0000 7f00577fe640 1 -- 172.21.15.70:0/724716250 learned_addr learned my addr 172.21.15.70:0/724716250 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:29.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.745+0000 7f0056ffd640 1 --2- 172.21.15.70:0/724716250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967f52b0 0x55cc96804dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.745+0000 7f00577fe640 1 -- 172.21.15.70:0/724716250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967f52b0 msgr2=0x55cc96804dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:29.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.745+0000 7f00577fe640 1 --2- 172.21.15.70:0/724716250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967f52b0 0x55cc96804dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.745+0000 7f00577fe640 1 -- 172.21.15.70:0/724716250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f5ca0 msgr2=0x55cc96805300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:29.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.745+0000 7f00577fe640 1 --2- 172.21.15.70:0/724716250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f5ca0 0x55cc96805300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.745+0000 7f00577fe640 1 -- 172.21.15.70:0/724716250 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc96806900 con 0x55cc967fae10 2024-04-01T01:40:29.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.745+0000 7f00577fe640 1 --2- 172.21.15.70:0/724716250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967fae10 0x55cc96806310 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f0050009ca0 tx=0x7f0050004500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49786fddde417eb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:29.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.745+0000 7f003aff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:40:29.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.746+0000 7f003aff5640 1 -- 172.21.15.70:0/724716250 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050084020 con 0x55cc967fae10 2024-04-01T01:40:29.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.746+0000 7f005ff9f740 1 -- 172.21.15.70:0/724716250 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0048099870 con 0x55cc967fae10 2024-04-01T01:40:29.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.747+0000 7f003aff5640 1 -- 172.21.15.70:0/724716250 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f005006c7f0 con 0x55cc967fae10 2024-04-01T01:40:29.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.748+0000 7f003aff5640 1 -- 172.21.15.70:0/724716250 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f005008f050 con 0x55cc967fae10 2024-04-01T01:40:29.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.748+0000 7f005ff9f740 10 client.17688 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:29.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.748+0000 7f005ff9f740 1 -- 172.21.15.70:0/724716250 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cc96872400 con 0x55cc967fae10 2024-04-01T01:40:29.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.748+0000 7f003aff5640 1 -- 172.21.15.70:0/724716250 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0048099870 con 0x55cc967fae10 2024-04-01T01:40:29.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.749+0000 7f003aff5640 1 -- 172.21.15.70:0/724716250 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f005006b040 con 0x55cc967fae10 2024-04-01T01:40:29.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.750+0000 7f005ff9f740 10 client.17688 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:29.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.750+0000 7f005ff9f740 10 client.17688 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:40:29.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.750+0000 7f005ff9f740 20 client.17688 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:29.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.750+0000 7f005ff9f740 1 --2- 172.21.15.70:0/724716250 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cc96872ea0 0x55cc968752c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:29.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.750+0000 7f005ff9f740 4 client.17688 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:40:29.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.750+0000 7f005ff9f740 1 -- 172.21.15.70:0/724716250 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cc96761a10 con 0x55cc96872ea0 2024-04-01T01:40:29.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.750+0000 7f0056ffd640 1 --2- 172.21.15.70:0/724716250 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cc96872ea0 0x55cc968752c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:29.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.752+0000 7f0056ffd640 1 --2- 172.21.15.70:0/724716250 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cc96872ea0 0x55cc968752c0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f54115eb33114a9 server_cookie=8c603da5535fdde0 in_seq=0 out_seq=0 2024-04-01T01:40:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.752+0000 7f003aff5640 10 client.17688 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.752+0000 7f003aff5640 1 -- 172.21.15.70:0/724716250 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55cc96761a10 con 0x55cc96872ea0 2024-04-01T01:40:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.752+0000 7f003aff5640 10 client.17688 handle_command_reply: tid=0 2024-04-01T01:40:29.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.753+0000 7f005ff9f740 1 client.17688 shutdown 2024-04-01T01:40:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.753+0000 7f005ff9f740 1 -- 172.21.15.70:0/724716250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967fae10 msgr2=0x55cc96806310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.753+0000 7f005ff9f740 1 --2- 172.21.15.70:0/724716250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967fae10 0x55cc96806310 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f0050009ca0 tx=0x7f0050004500 comp rx=0 tx=0).stop 2024-04-01T01:40:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.754+0000 7f005ff9f740 1 -- 172.21.15.70:0/724716250 shutdown_connections 2024-04-01T01:40:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.754+0000 7f005ff9f740 1 --2- 172.21.15.70:0/724716250 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cc96872ea0 0x55cc968752c0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.754+0000 7f005ff9f740 1 --2- 172.21.15.70:0/724716250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc967fae10 0x55cc96806310 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.754+0000 7f005ff9f740 1 --2- 172.21.15.70:0/724716250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc967f5ca0 0x55cc96805300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.754+0000 7f005ff9f740 1 --2- 172.21.15.70:0/724716250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc967f52b0 0x55cc96804dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.754+0000 7f005ff9f740 1 -- 172.21.15.70:0/724716250 >> 172.21.15.70:0/724716250 conn(0x55cc967f0820 msgr2=0x55cc967efc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:29.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.756+0000 7f003aff5640 0 client.17688 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:29.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.756+0000 7f005ff9f740 1 -- 172.21.15.70:0/724716250 shutdown_connections 2024-04-01T01:40:29.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.756+0000 7f005ff9f740 1 -- 172.21.15.70:0/724716250 wait complete. 2024-04-01T01:40:29.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.756+0000 7f005ff9f740 20 client.17688 trim_cache size 0 max 16384 2024-04-01T01:40:29.759 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:40:29.759 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "scrub active (171 inodes in the stack)", 2024-04-01T01:40:29.759 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:40:29.759 INFO:teuthology.orchestra.run.smithi070.stdout: "8beb3e58-aa81-4197-8e44-80d70157508c": { 2024-04-01T01:40:29.759 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:40:29.759 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "8beb3e58-aa81-4197-8e44-80d70157508c", 2024-04-01T01:40:29.759 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:40:29.759 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:40:29.760 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:40:29.760 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:40:29.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.760+0000 7f005d3bc640 1 -- 172.21.15.70:0/3311955233 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00400a26b0 msgr2=0x7f00400a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.760+0000 7f005d3bc640 1 --2- 172.21.15.70:0/3311955233 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00400a26b0 0x7f00400a4b70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f0050037040 tx=0x7f00500324f0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.760+0000 7f005d3bc640 1 -- 172.21.15.70:0/3311955233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0058107d90 msgr2=0x7f0058070ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.760+0000 7f005d3bc640 1 --2- 172.21.15.70:0/3311955233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0058107d90 0x7f0058070ef0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f004800e9c0 tx=0x7f004800ee80 comp rx=0 tx=0).stop 2024-04-01T01:40:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.760+0000 7f005d3bc640 1 -- 172.21.15.70:0/3311955233 shutdown_connections 2024-04-01T01:40:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.760+0000 7f005d3bc640 1 --2- 172.21.15.70:0/3311955233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0058109940 0x7f0058072470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.760+0000 7f005d3bc640 1 --2- 172.21.15.70:0/3311955233 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00400a26b0 0x7f00400a4b70 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.760+0000 7f005d3bc640 1 --2- 172.21.15.70:0/3311955233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0058108f80 0x7f0058071430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.760+0000 7f005d3bc640 1 --2- 172.21.15.70:0/3311955233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0058107d90 0x7f0058070ef0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.760+0000 7f005d3bc640 1 -- 172.21.15.70:0/3311955233 >> 172.21.15.70:0/3311955233 conn(0x7f0058076d20 msgr2=0x7f005810c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.761+0000 7f005d3bc640 1 -- 172.21.15.70:0/3311955233 shutdown_connections 2024-04-01T01:40:29.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:29.761+0000 7f005d3bc640 1 -- 172.21.15.70:0/3311955233 wait complete. 2024-04-01T01:40:29.781 INFO:tasks.cephfs.filesystem:scrub status for tag:8beb3e58-aa81-4197-8e44-80d70157508c - {'path': '/', 'tag': '8beb3e58-aa81-4197-8e44-80d70157508c', 'options': 'recursive,force'} 2024-04-01T01:40:30.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:30 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:40:30.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:30 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:40:30.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:30 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:40:30.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:30 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:40:30.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:30 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:40:30.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:30 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:40:30.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:30 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:40:30.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:30 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:40:30.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:30 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:40:30.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:30 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:40:30.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:30 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:40:30.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:30 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:40:31.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:31 smithi189 ceph-mon[35753]: pgmap v570: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 13 MiB/s wr, 997 op/s 2024-04-01T01:40:31.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:31 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:40:31.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:31 smithi070 ceph-mon[32179]: pgmap v570: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 13 MiB/s wr, 997 op/s 2024-04-01T01:40:31.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:31 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:40:31.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:31 smithi081 ceph-mon[35142]: pgmap v570: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 13 MiB/s wr, 997 op/s 2024-04-01T01:40:31.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:31 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:40:31.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 345.0 s, 834.4 tps, lat 119.935 ms stddev 48.112 2024-04-01T01:40:33.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:32 smithi189 ceph-mon[35753]: pgmap v571: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 15 MiB/s wr, 1.15k op/s 2024-04-01T01:40:33.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:32 smithi070 ceph-mon[32179]: pgmap v571: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 15 MiB/s wr, 1.15k op/s 2024-04-01T01:40:33.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:32 smithi081 ceph-mon[35142]: pgmap v571: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 15 MiB/s wr, 1.15k op/s 2024-04-01T01:40:35.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:34 smithi189 ceph-mon[35753]: pgmap v572: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 12 MiB/s wr, 983 op/s 2024-04-01T01:40:35.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:34 smithi070 ceph-mon[32179]: pgmap v572: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 12 MiB/s wr, 983 op/s 2024-04-01T01:40:35.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:34 smithi081 ceph-mon[35142]: pgmap v572: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 12 MiB/s wr, 983 op/s 2024-04-01T01:40:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.461+0000 7f99140d4640 1 -- 172.21.15.70:0/2472337835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990c10da70 msgr2=0x7f990c10def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.461+0000 7f99140d4640 1 --2- 172.21.15.70:0/2472337835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990c10da70 0x7f990c10def0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f98fc009900 tx=0x7f98fc02bc60 comp rx=0 tx=0).stop 2024-04-01T01:40:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.465+0000 7f99140d4640 1 -- 172.21.15.70:0/2472337835 shutdown_connections 2024-04-01T01:40:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.465+0000 7f99140d4640 1 --2- 172.21.15.70:0/2472337835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990c10da70 0x7f990c10def0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.465+0000 7f99140d4640 1 --2- 172.21.15.70:0/2472337835 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f990c10b0c0 0x7f990c10d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.465+0000 7f99140d4640 1 --2- 172.21.15.70:0/2472337835 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f990c108750 0x7f990c10ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.465+0000 7f99140d4640 1 -- 172.21.15.70:0/2472337835 >> 172.21.15.70:0/2472337835 conn(0x7f990c06e790 msgr2=0x7f990c070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.466+0000 7f99140d4640 1 -- 172.21.15.70:0/2472337835 shutdown_connections 2024-04-01T01:40:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.466+0000 7f99140d4640 1 -- 172.21.15.70:0/2472337835 wait complete. 2024-04-01T01:40:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.467+0000 7f99140d4640 1 Processor -- start 2024-04-01T01:40:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.467+0000 7f99140d4640 1 -- start start 2024-04-01T01:40:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.467+0000 7f99140d4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f990c108750 0x7f990c1a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.467+0000 7f99140d4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990c10b0c0 0x7f990c1a6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.467+0000 7f99140d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f990c10da70 0x7f990c1adca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.467+0000 7f99140d4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f990c11c100 con 0x7f990c108750 2024-04-01T01:40:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.467+0000 7f99140d4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f990c11bf80 con 0x7f990c10da70 2024-04-01T01:40:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.467+0000 7f99140d4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f990c11c280 con 0x7f990c10b0c0 2024-04-01T01:40:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.467+0000 7f9911e70640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f990c108750 0x7f990c1a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.467+0000 7f9911e70640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f990c108750 0x7f990c1a6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47760/0 (socket says 172.21.15.70:47760) 2024-04-01T01:40:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.467+0000 7f9911e70640 1 -- 172.21.15.70:0/120400960 learned_addr learned my addr 172.21.15.70:0/120400960 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.467+0000 7f9912671640 1 --2- 172.21.15.70:0/120400960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f990c10da70 0x7f990c1adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.468+0000 7f9911e70640 1 -- 172.21.15.70:0/120400960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990c10b0c0 msgr2=0x7f990c1a6c80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:40:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.468+0000 7f9911e70640 1 --2- 172.21.15.70:0/120400960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990c10b0c0 0x7f990c1a6c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.468+0000 7f9911e70640 1 -- 172.21.15.70:0/120400960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f990c10da70 msgr2=0x7f990c1adca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.468+0000 7f9911e70640 1 --2- 172.21.15.70:0/120400960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f990c10da70 0x7f990c1adca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.468+0000 7f9911e70640 1 -- 172.21.15.70:0/120400960 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990c1ae310 con 0x7f990c108750 2024-04-01T01:40:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.468+0000 7f9911e70640 1 --2- 172.21.15.70:0/120400960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f990c108750 0x7f990c1a6740 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f990400ba50 tx=0x7f990400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.469+0000 7f9902ffd640 1 -- 172.21.15.70:0/120400960 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9904019070 con 0x7f990c108750 2024-04-01T01:40:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.469+0000 7f9902ffd640 1 -- 172.21.15.70:0/120400960 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f990400cc10 con 0x7f990c108750 2024-04-01T01:40:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.469+0000 7f9902ffd640 1 -- 172.21.15.70:0/120400960 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99040075f0 con 0x7f990c108750 2024-04-01T01:40:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.469+0000 7f99140d4640 1 -- 172.21.15.70:0/120400960 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f990c1ae600 con 0x7f990c108750 2024-04-01T01:40:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.469+0000 7f99140d4640 1 -- 172.21.15.70:0/120400960 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f990c1aeb80 con 0x7f990c108750 2024-04-01T01:40:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.475+0000 7f9902ffd640 1 -- 172.21.15.70:0/120400960 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f9904014440 con 0x7f990c108750 2024-04-01T01:40:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.476+0000 7f9902ffd640 1 --2- 172.21.15.70:0/120400960 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f98f80a2730 0x7f98f80a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.477+0000 7f9902ffd640 1 -- 172.21.15.70:0/120400960 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f99040cdf90 con 0x7f990c108750 2024-04-01T01:40:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.478+0000 7f9916d99740 1 Processor -- start 2024-04-01T01:40:36.482 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.478+0000 7f991166f640 1 --2- 172.21.15.70:0/120400960 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f98f80a2730 0x7f98f80a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.479+0000 7f9916d99740 1 -- start start 2024-04-01T01:40:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.479+0000 7f9916d99740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac268d0 0x5599dac26cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.479+0000 7f9916d99740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac272c0 0x5599dac2bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.479+0000 7f9916d99740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac2c430 0x5599dac2e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.479+0000 7f9916d99740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5599dab8a8f0 con 0x5599dac2c430 2024-04-01T01:40:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.479+0000 7f9916d99740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5599dab338d0 con 0x5599dac272c0 2024-04-01T01:40:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.479+0000 7f9916d99740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5599dab86990 con 0x5599dac268d0 2024-04-01T01:40:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.480+0000 7f9911e70640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac268d0 0x5599dac26cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.480+0000 7f9911e70640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac268d0 0x5599dac26cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37718/0 (socket says 172.21.15.70:37718) 2024-04-01T01:40:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.480+0000 7f9911e70640 1 -- 172.21.15.70:0/1510205899 learned_addr learned my addr 172.21.15.70:0/1510205899 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.480+0000 7f9912671640 1 --2- 172.21.15.70:0/1510205899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac2c430 0x5599dac2e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.480+0000 7f991166f640 1 --2- 172.21.15.70:0/1510205899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac272c0 0x5599dac2bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.481+0000 7f9912671640 1 -- 172.21.15.70:0/1510205899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac268d0 msgr2=0x5599dac26cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.481+0000 7f9912671640 1 --2- 172.21.15.70:0/1510205899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac268d0 0x5599dac26cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.481+0000 7f9912671640 1 -- 172.21.15.70:0/1510205899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac272c0 msgr2=0x5599dac2bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.481+0000 7f9912671640 1 --2- 172.21.15.70:0/1510205899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac272c0 0x5599dac2bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.481+0000 7f9912671640 1 -- 172.21.15.70:0/1510205899 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599da89f210 con 0x5599dac2c430 2024-04-01T01:40:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.481+0000 7f9911e70640 1 --2- 172.21.15.70:0/1510205899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac268d0 0x5599dac26cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:40:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.481+0000 7f9912671640 1 --2- 172.21.15.70:0/1510205899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac2c430 0x5599dac2e8c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f98fc02c640 tx=0x7f98fc005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b22514da7872a81 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.482+0000 7f98f7fff640 1 -- 172.21.15.70:0/1510205899 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc043030 con 0x5599dac2c430 2024-04-01T01:40:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.482+0000 7f98f7fff640 1 -- 172.21.15.70:0/1510205899 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98fc004040 con 0x5599dac2c430 2024-04-01T01:40:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.482+0000 7f98f7fff640 1 -- 172.21.15.70:0/1510205899 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc003990 con 0x5599dac2c430 2024-04-01T01:40:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.483+0000 7f9916d99740 1 -- 172.21.15.70:0/1510205899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac2c430 msgr2=0x5599dac2e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.483+0000 7f9916d99740 1 --2- 172.21.15.70:0/1510205899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac2c430 0x5599dac2e8c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f98fc02c640 tx=0x7f98fc005c00 comp rx=0 tx=0).stop 2024-04-01T01:40:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.483+0000 7f991166f640 1 --2- 172.21.15.70:0/120400960 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f98f80a2730 0x7f98f80a4bf0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f990c1a7c00 tx=0x7f990800bf20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.486+0000 7f9916d99740 1 -- 172.21.15.70:0/1510205899 shutdown_connections 2024-04-01T01:40:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.486+0000 7f9916d99740 1 --2- 172.21.15.70:0/1510205899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac2c430 0x5599dac2e8c0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.486+0000 7f9916d99740 1 --2- 172.21.15.70:0/1510205899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac272c0 0x5599dac2bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.486+0000 7f9916d99740 1 --2- 172.21.15.70:0/1510205899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac268d0 0x5599dac26cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.486+0000 7f9916d99740 1 -- 172.21.15.70:0/1510205899 >> 172.21.15.70:0/1510205899 conn(0x5599dac21e40 msgr2=0x5599dac22240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:36.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.486+0000 7f9916d99740 1 -- 172.21.15.70:0/1510205899 shutdown_connections 2024-04-01T01:40:36.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.487+0000 7f9916d99740 1 -- 172.21.15.70:0/1510205899 wait complete. 2024-04-01T01:40:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.487+0000 7f9916d99740 1 -- start start 2024-04-01T01:40:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.487+0000 7f9916d99740 1 Processor -- start 2024-04-01T01:40:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.488+0000 7f9916d99740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac268d0 0x5599dac41bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.488+0000 7f9916d99740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac272c0 0x5599dac42110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.488+0000 7f9916d99740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac2c430 0x5599dac49150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.488+0000 7f9916d99740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5599dac403e0 con 0x5599dac268d0 2024-04-01T01:40:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.488+0000 7f9916d99740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5599dab28f80 con 0x5599dac272c0 2024-04-01T01:40:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.488+0000 7f9916d99740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5599dac40560 con 0x5599dac2c430 2024-04-01T01:40:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.488+0000 7f9916d99740 10 client.0 _ll_register_callbacks cb 0x5599dab81290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.489+0000 7f9912671640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac2c430 0x5599dac49150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.489+0000 7f9912671640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac2c430 0x5599dac49150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37734/0 (socket says 172.21.15.70:37734) 2024-04-01T01:40:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.489+0000 7f9912671640 1 -- 172.21.15.70:0/2738149309 learned_addr learned my addr 172.21.15.70:0/2738149309 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.489+0000 7f9912671640 1 -- 172.21.15.70:0/2738149309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac272c0 msgr2=0x5599dac42110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:40:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.489+0000 7f9912671640 1 --2- 172.21.15.70:0/2738149309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac272c0 0x5599dac42110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.489+0000 7f9912671640 1 -- 172.21.15.70:0/2738149309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac268d0 msgr2=0x5599dac41bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:40:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.489+0000 7f9912671640 1 --2- 172.21.15.70:0/2738149309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac268d0 0x5599dac41bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.489+0000 7f9912671640 1 -- 172.21.15.70:0/2738149309 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599da96b620 con 0x5599dac2c430 2024-04-01T01:40:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.491+0000 7f9912671640 1 --2- 172.21.15.70:0/2738149309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac2c430 0x5599dac49150 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f98fc02c1a0 tx=0x7f98fc02cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eb8520213411884 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.491+0000 7f98f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:40:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.491+0000 7f98f57fa640 1 -- 172.21.15.70:0/2738149309 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc043070 con 0x5599dac2c430 2024-04-01T01:40:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.491+0000 7f98f57fa640 1 -- 172.21.15.70:0/2738149309 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f98fc003ca0 con 0x5599dac2c430 2024-04-01T01:40:36.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.491+0000 7f9916d99740 1 -- 172.21.15.70:0/2738149309 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98fc004040 con 0x5599dac2c430 2024-04-01T01:40:36.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.493+0000 7f98f57fa640 1 -- 172.21.15.70:0/2738149309 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f98fc02ce20 con 0x5599dac2c430 2024-04-01T01:40:36.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.494+0000 7f9916d99740 10 client.27469 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:36.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.494+0000 7f9916d99740 1 -- 172.21.15.70:0/2738149309 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5599dac403e0 con 0x5599dac2c430 2024-04-01T01:40:36.495 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.495+0000 7f98f57fa640 1 -- 172.21.15.70:0/2738149309 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98fc004040 con 0x5599dac2c430 2024-04-01T01:40:36.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.496+0000 7f98f57fa640 1 -- 172.21.15.70:0/2738149309 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f98fc034550 con 0x5599dac2c430 2024-04-01T01:40:36.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.496+0000 7f9916d99740 10 client.27469 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:36.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.496+0000 7f9916d99740 10 client.27469 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:40:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.496+0000 7f9916d99740 20 client.27469 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.496+0000 7f9916d99740 1 --2- 172.21.15.70:0/2738149309 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599dac6f990 0x5599dac71dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.496+0000 7f9916d99740 4 client.27469 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:40:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.496+0000 7f9916d99740 1 -- 172.21.15.70:0/2738149309 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599dab9e120 con 0x5599dac6f990 2024-04-01T01:40:36.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.496+0000 7f9911e70640 1 --2- 172.21.15.70:0/2738149309 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599dac6f990 0x5599dac71dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.498+0000 7f9911e70640 1 --2- 172.21.15.70:0/2738149309 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599dac6f990 0x5599dac71dd0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba1b2b84b3226f89 server_cookie=84e26da3c790af9c in_seq=0 out_seq=0 2024-04-01T01:40:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.498+0000 7f98f57fa640 10 client.27469 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.501+0000 7f98f57fa640 1 -- 172.21.15.70:0/2738149309 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5599dab9e120 con 0x5599dac6f990 2024-04-01T01:40:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.501+0000 7f98f57fa640 10 client.27469 handle_command_reply: tid=0 2024-04-01T01:40:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.501+0000 7f9916d99740 1 client.27469 shutdown 2024-04-01T01:40:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.501+0000 7f9916d99740 1 -- 172.21.15.70:0/2738149309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac2c430 msgr2=0x5599dac49150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.501+0000 7f9916d99740 1 --2- 172.21.15.70:0/2738149309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac2c430 0x5599dac49150 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f98fc02c1a0 tx=0x7f98fc02cc80 comp rx=0 tx=0).stop 2024-04-01T01:40:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.502+0000 7f9916d99740 1 -- 172.21.15.70:0/2738149309 shutdown_connections 2024-04-01T01:40:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.502+0000 7f9916d99740 1 --2- 172.21.15.70:0/2738149309 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599dac6f990 0x5599dac71dd0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.502+0000 7f9916d99740 1 --2- 172.21.15.70:0/2738149309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac2c430 0x5599dac49150 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.502+0000 7f9916d99740 1 --2- 172.21.15.70:0/2738149309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac272c0 0x5599dac42110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.502+0000 7f9916d99740 1 --2- 172.21.15.70:0/2738149309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac268d0 0x5599dac41bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.502+0000 7f9916d99740 1 -- 172.21.15.70:0/2738149309 >> 172.21.15.70:0/2738149309 conn(0x5599dac21e40 msgr2=0x5599dac20f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.502+0000 7f98f57fa640 0 client.27469 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.502+0000 7f9916d99740 1 -- 172.21.15.70:0/2738149309 shutdown_connections 2024-04-01T01:40:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.502+0000 7f9916d99740 1 -- 172.21.15.70:0/2738149309 wait complete. 2024-04-01T01:40:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.502+0000 7f9916d99740 20 client.27469 trim_cache size 0 max 16384 2024-04-01T01:40:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.530+0000 7f9916d99740 1 Processor -- start 2024-04-01T01:40:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.530+0000 7f9916d99740 1 -- start start 2024-04-01T01:40:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.530+0000 7f9916d99740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac268d0 0x5599dac776e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.531+0000 7f9916d99740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac272c0 0x5599dac49110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.531+0000 7f9911e70640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac268d0 0x5599dac776e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.531+0000 7f9911e70640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac268d0 0x5599dac776e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37742/0 (socket says 172.21.15.70:37742) 2024-04-01T01:40:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.531+0000 7f9911e70640 1 -- 172.21.15.70:0/3292938658 learned_addr learned my addr 172.21.15.70:0/3292938658 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.531+0000 7f9916d99740 1 --2- 172.21.15.70:0/3292938658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac2c430 0x5599dac250a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.531+0000 7f9916d99740 1 -- 172.21.15.70:0/3292938658 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f990c0754a0 con 0x5599dac272c0 2024-04-01T01:40:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.531+0000 7f9912671640 1 --2- 172.21.15.70:0/3292938658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac2c430 0x5599dac250a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.531+0000 7f9916d99740 1 -- 172.21.15.70:0/3292938658 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f990c076fe0 con 0x5599dac2c430 2024-04-01T01:40:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.531+0000 7f9916d99740 1 -- 172.21.15.70:0/3292938658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f990c075b70 con 0x5599dac268d0 2024-04-01T01:40:36.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.532+0000 7f9911e70640 1 -- 172.21.15.70:0/3292938658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac2c430 msgr2=0x5599dac250a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:36.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.532+0000 7f9911e70640 1 --2- 172.21.15.70:0/3292938658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac2c430 0x5599dac250a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.532+0000 7f9911e70640 1 -- 172.21.15.70:0/3292938658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac272c0 msgr2=0x5599dac49110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:36.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.532+0000 7f9911e70640 1 --2- 172.21.15.70:0/3292938658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac272c0 0x5599dac49110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.532+0000 7f9911e70640 1 -- 172.21.15.70:0/3292938658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599dac1d190 con 0x5599dac268d0 2024-04-01T01:40:36.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.532+0000 7f9911e70640 1 --2- 172.21.15.70:0/3292938658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac268d0 0x5599dac776e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f990400c7d0 tx=0x7f99040ba040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=324295ec19f7d112 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:36.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.532+0000 7f98f77fe640 1 -- 172.21.15.70:0/3292938658 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99040b3070 con 0x5599dac268d0 2024-04-01T01:40:36.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.532+0000 7f98f77fe640 1 -- 172.21.15.70:0/3292938658 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f990409e770 con 0x5599dac268d0 2024-04-01T01:40:36.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.533+0000 7f98f77fe640 1 -- 172.21.15.70:0/3292938658 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9904096d40 con 0x5599dac268d0 2024-04-01T01:40:36.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.534+0000 7f9916d99740 1 -- 172.21.15.70:0/3292938658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac268d0 msgr2=0x5599dac776e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:36.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.534+0000 7f9916d99740 1 --2- 172.21.15.70:0/3292938658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac268d0 0x5599dac776e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f990400c7d0 tx=0x7f99040ba040 comp rx=0 tx=0).stop 2024-04-01T01:40:36.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.534+0000 7f9916d99740 1 -- 172.21.15.70:0/3292938658 shutdown_connections 2024-04-01T01:40:36.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.534+0000 7f9916d99740 1 --2- 172.21.15.70:0/3292938658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac2c430 0x5599dac250a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.534+0000 7f9916d99740 1 --2- 172.21.15.70:0/3292938658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac272c0 0x5599dac49110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.534+0000 7f9916d99740 1 --2- 172.21.15.70:0/3292938658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac268d0 0x5599dac776e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.534+0000 7f9916d99740 1 -- 172.21.15.70:0/3292938658 >> 172.21.15.70:0/3292938658 conn(0x5599dac21e40 msgr2=0x5599dac41bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.534+0000 7f9916d99740 1 -- 172.21.15.70:0/3292938658 shutdown_connections 2024-04-01T01:40:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.534+0000 7f9916d99740 1 -- 172.21.15.70:0/3292938658 wait complete. 2024-04-01T01:40:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.534+0000 7f9916d99740 1 -- start start 2024-04-01T01:40:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.535+0000 7f9916d99740 1 Processor -- start 2024-04-01T01:40:36.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.536+0000 7f9916d99740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac268d0 0x5599dac46010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.536+0000 7f9916d99740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac272c0 0x5599dac46550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.536+0000 7f9916d99740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac2c430 0x5599dacd3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.536+0000 7f9911e70640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac268d0 0x5599dac46010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:36.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.536+0000 7f9911e70640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac268d0 0x5599dac46010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:59220/0 (socket says 172.21.15.70:59220) 2024-04-01T01:40:36.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.536+0000 7f991166f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac272c0 0x5599dac46550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:36.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.536+0000 7f991166f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac272c0 0x5599dac46550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37756/0 (socket says 172.21.15.70:37756) 2024-04-01T01:40:36.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.537+0000 7f9911e70640 1 -- 172.21.15.70:0/1859643981 learned_addr learned my addr 172.21.15.70:0/1859643981 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:36.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.537+0000 7f9916d99740 1 -- 172.21.15.70:0/1859643981 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f990c076910 con 0x5599dac2c430 2024-04-01T01:40:36.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.537+0000 7f9916d99740 1 -- 172.21.15.70:0/1859643981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f990c076240 con 0x5599dac268d0 2024-04-01T01:40:36.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.537+0000 7f9916d99740 1 -- 172.21.15.70:0/1859643981 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f990c118180 con 0x5599dac272c0 2024-04-01T01:40:36.538 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.537+0000 7f9912671640 1 --2- 172.21.15.70:0/1859643981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac2c430 0x5599dacd3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:36.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.537+0000 7f9916d99740 10 client.0 _ll_register_callbacks cb 0x7f99040962b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.538+0000 7f9911e70640 1 -- 172.21.15.70:0/1859643981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac272c0 msgr2=0x5599dac46550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.538+0000 7f9911e70640 1 --2- 172.21.15.70:0/1859643981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac272c0 0x5599dac46550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.538+0000 7f9911e70640 1 -- 172.21.15.70:0/1859643981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac2c430 msgr2=0x5599dacd3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.538+0000 7f9911e70640 1 --2- 172.21.15.70:0/1859643981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac2c430 0x5599dacd3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.538+0000 7f9911e70640 1 -- 172.21.15.70:0/1859643981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599dacd4240 con 0x5599dac268d0 2024-04-01T01:40:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.538+0000 7f9916d99740 1 -- 172.21.15.70:0/1859643981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f990409e770 con 0x5599dac268d0 2024-04-01T01:40:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.538+0000 7f9911e70640 1 --2- 172.21.15.70:0/1859643981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac268d0 0x5599dac46010 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f990409a3d0 tx=0x7f99040a3030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b24034f561abb37f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.538+0000 7f98f57fa640 10 client.27632 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:40:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.538+0000 7f98f57fa640 1 -- 172.21.15.70:0/1859643981 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99040b7020 con 0x5599dac268d0 2024-04-01T01:40:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.538+0000 7f98f57fa640 1 -- 172.21.15.70:0/1859643981 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f9904096d40 con 0x5599dac268d0 2024-04-01T01:40:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.538+0000 7f98f57fa640 1 -- 172.21.15.70:0/1859643981 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f99040b3040 con 0x5599dac268d0 2024-04-01T01:40:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.539+0000 7f9916d99740 10 client.27632 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.539+0000 7f9916d99740 1 -- 172.21.15.70:0/1859643981 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=31}) -- 0x5599dacd53a0 con 0x5599dac268d0 2024-04-01T01:40:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.540+0000 7f98f57fa640 1 -- 172.21.15.70:0/1859643981 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f99040b31e0 con 0x5599dac268d0 2024-04-01T01:40:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.540+0000 7f9916d99740 10 client.27632 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.540+0000 7f9916d99740 10 client.27632 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:40:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.540+0000 7f9916d99740 20 client.27632 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.540+0000 7f9916d99740 1 --2- 172.21.15.70:0/1859643981 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599dacd5bf0 0x5599dacd8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.540+0000 7f9916d99740 4 client.27632 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:40:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.540+0000 7f9916d99740 1 -- 172.21.15.70:0/1859643981 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f990409e770 con 0x5599dacd5bf0 2024-04-01T01:40:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.540+0000 7f991166f640 1 --2- 172.21.15.70:0/1859643981 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599dacd5bf0 0x5599dacd8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.541+0000 7f991166f640 1 --2- 172.21.15.70:0/1859643981 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599dacd5bf0 0x5599dacd8010 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55501180221aed1b server_cookie=6c64a7f3d70c1ff6 in_seq=0 out_seq=0 2024-04-01T01:40:36.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:36.541+0000 7f98f57fa640 10 client.27632 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:36.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 350.0 s, 852.4 tps, lat 117.293 ms stddev 81.785 2024-04-01T01:40:37.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:36 smithi189 ceph-mon[35753]: pgmap v573: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 13 MiB/s wr, 1.08k op/s 2024-04-01T01:40:37.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:36 smithi070 ceph-mon[32179]: pgmap v573: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 13 MiB/s wr, 1.08k op/s 2024-04-01T01:40:37.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:36 smithi081 ceph-mon[35142]: pgmap v573: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 13 MiB/s wr, 1.08k op/s 2024-04-01T01:40:39.129 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:38 smithi081 ceph-mon[35142]: pgmap v574: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-01T01:40:39.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:38 smithi189 ceph-mon[35753]: pgmap v574: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-01T01:40:39.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:38 smithi070 ceph-mon[32179]: pgmap v574: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-01T01:40:41.090 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.089+0000 7f98f57fa640 1 -- 172.21.15.70:0/1859643981 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f990409e770 con 0x5599dacd5bf0 2024-04-01T01:40:41.090 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.089+0000 7f98f57fa640 10 client.27632 handle_command_reply: tid=0 2024-04-01T01:40:41.090 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.089+0000 7f9916d99740 1 client.27632 shutdown 2024-04-01T01:40:41.090 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.090+0000 7f9916d99740 1 -- 172.21.15.70:0/1859643981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac268d0 msgr2=0x5599dac46010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:41.090 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.090+0000 7f9916d99740 1 --2- 172.21.15.70:0/1859643981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac268d0 0x5599dac46010 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f990409a3d0 tx=0x7f99040a3030 comp rx=0 tx=0).stop 2024-04-01T01:40:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.090+0000 7f9916d99740 1 -- 172.21.15.70:0/1859643981 shutdown_connections 2024-04-01T01:40:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.090+0000 7f9916d99740 1 --2- 172.21.15.70:0/1859643981 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599dacd5bf0 0x5599dacd8010 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.090+0000 7f9916d99740 1 --2- 172.21.15.70:0/1859643981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599dac2c430 0x5599dacd3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.090+0000 7f9916d99740 1 --2- 172.21.15.70:0/1859643981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599dac272c0 0x5599dac46550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.090+0000 7f9916d99740 1 --2- 172.21.15.70:0/1859643981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599dac268d0 0x5599dac46010 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.090+0000 7f9916d99740 1 -- 172.21.15.70:0/1859643981 >> 172.21.15.70:0/1859643981 conn(0x5599dac21e40 msgr2=0x5599dac250a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.090+0000 7f98f57fa640 0 client.27632 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.090+0000 7f9916d99740 1 -- 172.21.15.70:0/1859643981 shutdown_connections 2024-04-01T01:40:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.090+0000 7f9916d99740 1 -- 172.21.15.70:0/1859643981 wait complete. 2024-04-01T01:40:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.090+0000 7f9916d99740 20 client.27632 trim_cache size 0 max 16384 2024-04-01T01:40:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:40:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:40:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:40:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:40:41.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.092+0000 7f99140d4640 1 -- 172.21.15.70:0/120400960 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f98f80a2730 msgr2=0x7f98f80a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:41.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.092+0000 7f99140d4640 1 --2- 172.21.15.70:0/120400960 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f98f80a2730 0x7f98f80a4bf0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f990c1a7c00 tx=0x7f990800bf20 comp rx=0 tx=0).stop 2024-04-01T01:40:41.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.092+0000 7f99140d4640 1 -- 172.21.15.70:0/120400960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f990c108750 msgr2=0x7f990c1a6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:41.094 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.092+0000 7f99140d4640 1 --2- 172.21.15.70:0/120400960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f990c108750 0x7f990c1a6740 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f990400ba50 tx=0x7f990400bf10 comp rx=0 tx=0).stop 2024-04-01T01:40:41.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.092+0000 7f99140d4640 1 -- 172.21.15.70:0/120400960 shutdown_connections 2024-04-01T01:40:41.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.092+0000 7f99140d4640 1 --2- 172.21.15.70:0/120400960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f990c10da70 0x7f990c1adca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:41.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.092+0000 7f99140d4640 1 --2- 172.21.15.70:0/120400960 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f98f80a2730 0x7f98f80a4bf0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:41.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.092+0000 7f99140d4640 1 --2- 172.21.15.70:0/120400960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f990c10b0c0 0x7f990c1a6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:41.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.092+0000 7f99140d4640 1 --2- 172.21.15.70:0/120400960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f990c108750 0x7f990c1a6740 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:41.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.092+0000 7f99140d4640 1 -- 172.21.15.70:0/120400960 >> 172.21.15.70:0/120400960 conn(0x7f990c06e790 msgr2=0x7f990c10a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:41.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.093+0000 7f99140d4640 1 -- 172.21.15.70:0/120400960 shutdown_connections 2024-04-01T01:40:41.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:41.093+0000 7f99140d4640 1 -- 172.21.15.70:0/120400960 wait complete. 2024-04-01T01:40:41.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:40 smithi189 ceph-mon[35753]: pgmap v575: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 9.9 MiB/s wr, 954 op/s 2024-04-01T01:40:41.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:40 smithi070 ceph-mon[32179]: pgmap v575: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 9.9 MiB/s wr, 954 op/s 2024-04-01T01:40:41.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:40 smithi081 ceph-mon[35142]: pgmap v575: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 9.9 MiB/s wr, 954 op/s 2024-04-01T01:40:41.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 355.0 s, 887.0 tps, lat 112.484 ms stddev 54.113 2024-04-01T01:40:43.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:42 smithi189 ceph-mon[35753]: pgmap v576: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-04-01T01:40:43.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:42 smithi070 ceph-mon[32179]: pgmap v576: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-04-01T01:40:43.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:42 smithi081 ceph-mon[35142]: pgmap v576: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 12 MiB/s wr, 1.07k op/s 2024-04-01T01:40:45.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:44 smithi189 ceph-mon[35753]: pgmap v577: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 10 MiB/s wr, 915 op/s 2024-04-01T01:40:45.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:44 smithi070 ceph-mon[32179]: pgmap v577: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 10 MiB/s wr, 915 op/s 2024-04-01T01:40:45.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:44 smithi081 ceph-mon[35142]: pgmap v577: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 10 MiB/s wr, 915 op/s 2024-04-01T01:40:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 360.0 s, 762.8 tps, lat 130.877 ms stddev 119.715 2024-04-01T01:40:47.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:46 smithi189 ceph-mon[35753]: pgmap v578: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-04-01T01:40:47.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:46 smithi070 ceph-mon[32179]: pgmap v578: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-04-01T01:40:47.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:46 smithi081 ceph-mon[35142]: pgmap v578: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-04-01T01:40:49.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:48 smithi189 ceph-mon[35753]: pgmap v579: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-01T01:40:49.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:48 smithi070 ceph-mon[32179]: pgmap v579: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-01T01:40:49.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:48 smithi081 ceph-mon[35142]: pgmap v579: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 12 MiB/s wr, 1.03k op/s 2024-04-01T01:40:51.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:50 smithi189 ceph-mon[35753]: pgmap v580: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 11 MiB/s wr, 956 op/s 2024-04-01T01:40:51.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:50 smithi070 ceph-mon[32179]: pgmap v580: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 11 MiB/s wr, 956 op/s 2024-04-01T01:40:51.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:50 smithi081 ceph-mon[35142]: pgmap v580: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 11 MiB/s wr, 956 op/s 2024-04-01T01:40:51.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 365.0 s, 884.6 tps, lat 112.841 ms stddev 53.820 2024-04-01T01:40:53.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:52 smithi189 ceph-mon[35753]: pgmap v581: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 13 MiB/s wr, 1.08k op/s 2024-04-01T01:40:53.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:52 smithi070 ceph-mon[32179]: pgmap v581: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 13 MiB/s wr, 1.08k op/s 2024-04-01T01:40:53.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:52 smithi081 ceph-mon[35142]: pgmap v581: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 13 MiB/s wr, 1.08k op/s 2024-04-01T01:40:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.246+0000 7fb91effd640 1 -- 172.21.15.70:0/1844468918 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb92009dc60 con 0x7fb918097930 2024-04-01T01:40:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.246+0000 7fb925072640 1 -- 172.21.15.70:0/1844468918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb918097930 msgr2=0x7fb91809e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.246+0000 7fb925072640 1 --2- 172.21.15.70:0/1844468918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb918097930 0x7fb91809e1c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb920066e90 tx=0x7fb920097910 comp rx=0 tx=0).stop 2024-04-01T01:40:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.247+0000 7fb925072640 1 -- 172.21.15.70:0/1844468918 shutdown_connections 2024-04-01T01:40:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.248+0000 7fb925072640 1 --2- 172.21.15.70:0/1844468918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb918097930 0x7fb91809e1c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.248+0000 7fb925072640 1 --2- 172.21.15.70:0/1844468918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb918096f70 0x7fb9180973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.248+0000 7fb925072640 1 --2- 172.21.15.70:0/1844468918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb918095d80 0x7fb9180961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.248+0000 7fb925072640 1 -- 172.21.15.70:0/1844468918 >> 172.21.15.70:0/1844468918 conn(0x7fb9180912d0 msgr2=0x7fb918093730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.249+0000 7fb925072640 1 -- 172.21.15.70:0/1844468918 shutdown_connections 2024-04-01T01:40:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.249+0000 7fb925072640 1 -- 172.21.15.70:0/1844468918 wait complete. 2024-04-01T01:40:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.250+0000 7fb925072640 1 Processor -- start 2024-04-01T01:40:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.250+0000 7fb925072640 1 -- start start 2024-04-01T01:40:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.250+0000 7fb925072640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb918095d80 0x7fb9180a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.250+0000 7fb925072640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb918096f70 0x7fb9180a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.250+0000 7fb925072640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb918097930 0x7fb918142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.250+0000 7fb925072640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9180088e0 con 0x7fb918095d80 2024-04-01T01:40:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.250+0000 7fb925072640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb918008760 con 0x7fb918097930 2024-04-01T01:40:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.250+0000 7fb925072640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb918008a60 con 0x7fb918096f70 2024-04-01T01:40:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.251+0000 7fb91f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb918096f70 0x7fb9180a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.251+0000 7fb91f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb918096f70 0x7fb9180a54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40812/0 (socket says 172.21.15.70:40812) 2024-04-01T01:40:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.251+0000 7fb91f7fe640 1 -- 172.21.15.70:0/1664035830 learned_addr learned my addr 172.21.15.70:0/1664035830 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.251+0000 7fb924871640 1 --2- 172.21.15.70:0/1664035830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb918097930 0x7fb918142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.252+0000 7fb91f7fe640 1 -- 172.21.15.70:0/1664035830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb918097930 msgr2=0x7fb918142690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.252+0000 7fb91f7fe640 1 --2- 172.21.15.70:0/1664035830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb918097930 0x7fb918142690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.252+0000 7fb91f7fe640 1 -- 172.21.15.70:0/1664035830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb918095d80 msgr2=0x7fb9180a4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.252+0000 7fb91f7fe640 1 --2- 172.21.15.70:0/1664035830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb918095d80 0x7fb9180a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.252+0000 7fb91f7fe640 1 -- 172.21.15.70:0/1664035830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9180ac650 con 0x7fb918096f70 2024-04-01T01:40:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.252+0000 7fb91f7fe640 1 --2- 172.21.15.70:0/1664035830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb918096f70 0x7fb9180a54b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00e9f0 tx=0x7fb90c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.252+0000 7fb91d7fa640 1 -- 172.21.15.70:0/1664035830 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90c00cd80 con 0x7fb918096f70 2024-04-01T01:40:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.252+0000 7fb91d7fa640 1 -- 172.21.15.70:0/1664035830 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb90c0108e0 con 0x7fb918096f70 2024-04-01T01:40:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.252+0000 7fb91d7fa640 1 -- 172.21.15.70:0/1664035830 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90c018880 con 0x7fb918096f70 2024-04-01T01:40:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.253+0000 7fb925072640 1 -- 172.21.15.70:0/1664035830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb918142bd0 con 0x7fb918096f70 2024-04-01T01:40:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.253+0000 7fb925072640 1 -- 172.21.15.70:0/1664035830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9181430a0 con 0x7fb918096f70 2024-04-01T01:40:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.256+0000 7fb928f97740 1 Processor -- start 2024-04-01T01:40:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.257+0000 7fb928f97740 1 -- start start 2024-04-01T01:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.257+0000 7fb928f97740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc7fae0 0x55bdafc7ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.257+0000 7fb928f97740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc804d0 0x55bdafc84fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.257+0000 7fb928f97740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc85640 0x55bdafc87ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.257+0000 7fb928f97740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bdafbe38f0 con 0x55bdafc804d0 2024-04-01T01:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.257+0000 7fb928f97740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55bdafb8c8d0 con 0x55bdafc85640 2024-04-01T01:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.257+0000 7fb928f97740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bdafbdf990 con 0x55bdafc7fae0 2024-04-01T01:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.257+0000 7fb91d7fa640 1 -- 172.21.15.70:0/1664035830 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb90c010a90 con 0x7fb918096f70 2024-04-01T01:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.258+0000 7fb91ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc804d0 0x55bdafc84fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.258+0000 7fb91ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc804d0 0x55bdafc84fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59296/0 (socket says 172.21.15.70:59296) 2024-04-01T01:40:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.258+0000 7fb91ffff640 1 -- 172.21.15.70:0/2802382301 learned_addr learned my addr 172.21.15.70:0/2802382301 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.258+0000 7fb924871640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc7fae0 0x55bdafc7ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.258+0000 7fb91f7fe640 1 --2- 172.21.15.70:0/2802382301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc85640 0x55bdafc87ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.258+0000 7fb91ffff640 1 -- 172.21.15.70:0/2802382301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc7fae0 msgr2=0x55bdafc7ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.258+0000 7fb91ffff640 1 --2- 172.21.15.70:0/2802382301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc7fae0 0x55bdafc7ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.258+0000 7fb91ffff640 1 -- 172.21.15.70:0/2802382301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc85640 msgr2=0x55bdafc87ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.258+0000 7fb91ffff640 1 --2- 172.21.15.70:0/2802382301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc85640 0x55bdafc87ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.258+0000 7fb91ffff640 1 -- 172.21.15.70:0/2802382301 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdaf8f8210 con 0x55bdafc804d0 2024-04-01T01:40:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.258+0000 7fb91ffff640 1 --2- 172.21.15.70:0/2802382301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc804d0 0x55bdafc84fd0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb914009bf0 tx=0x7fb91400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d23bd3f062c2bd28 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.259+0000 7fb91d7fa640 1 --2- 172.21.15.70:0/1664035830 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9100a25f0 0x7fb9100a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.259+0000 7fb90a7fc640 1 -- 172.21.15.70:0/2802382301 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb914005020 con 0x55bdafc804d0 2024-04-01T01:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.259+0000 7fb91d7fa640 1 -- 172.21.15.70:0/1664035830 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb90c0ceed0 con 0x7fb918096f70 2024-04-01T01:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.259+0000 7fb90a7fc640 1 -- 172.21.15.70:0/2802382301 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9140047a0 con 0x55bdafc804d0 2024-04-01T01:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.259+0000 7fb924871640 1 --2- 172.21.15.70:0/1664035830 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9100a25f0 0x7fb9100a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.259+0000 7fb90a7fc640 1 -- 172.21.15.70:0/2802382301 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb914010690 con 0x55bdafc804d0 2024-04-01T01:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.260+0000 7fb928f97740 1 -- 172.21.15.70:0/2802382301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc804d0 msgr2=0x55bdafc84fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.260+0000 7fb924871640 1 --2- 172.21.15.70:0/1664035830 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9100a25f0 0x7fb9100a4ab0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb92002d700 tx=0x7fb920078040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.260+0000 7fb928f97740 1 --2- 172.21.15.70:0/2802382301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc804d0 0x55bdafc84fd0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb914009bf0 tx=0x7fb91400e720 comp rx=0 tx=0).stop 2024-04-01T01:40:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.260+0000 7fb928f97740 1 -- 172.21.15.70:0/2802382301 shutdown_connections 2024-04-01T01:40:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.260+0000 7fb928f97740 1 --2- 172.21.15.70:0/2802382301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc85640 0x55bdafc87ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.260+0000 7fb928f97740 1 --2- 172.21.15.70:0/2802382301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc804d0 0x55bdafc84fd0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.260+0000 7fb928f97740 1 --2- 172.21.15.70:0/2802382301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc7fae0 0x55bdafc7ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.260+0000 7fb928f97740 1 -- 172.21.15.70:0/2802382301 >> 172.21.15.70:0/2802382301 conn(0x55bdafc7b050 msgr2=0x55bdafc7b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.260+0000 7fb928f97740 1 -- 172.21.15.70:0/2802382301 shutdown_connections 2024-04-01T01:40:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.260+0000 7fb928f97740 1 -- 172.21.15.70:0/2802382301 wait complete. 2024-04-01T01:40:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.261+0000 7fb928f97740 1 -- start start 2024-04-01T01:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.261+0000 7fb928f97740 1 Processor -- start 2024-04-01T01:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.261+0000 7fb928f97740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc7fae0 0x55bdafc89e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.262+0000 7fb928f97740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc804d0 0x55bdafc8a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.262+0000 7fb928f97740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc85640 0x55bdafc91310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.262+0000 7fb91ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc7fae0 0x55bdafc89e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.262+0000 7fb91ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc7fae0 0x55bdafc89e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40828/0 (socket says 172.21.15.70:40828) 2024-04-01T01:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.262+0000 7fb91ffff640 1 -- 172.21.15.70:0/1942850726 learned_addr learned my addr 172.21.15.70:0/1942850726 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.262+0000 7fb91f7fe640 1 --2- 172.21.15.70:0/1942850726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc804d0 0x55bdafc8a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.262+0000 7fb928f97740 1 -- 172.21.15.70:0/1942850726 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bdafc88640 con 0x55bdafc804d0 2024-04-01T01:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.262+0000 7fb924871640 1 --2- 172.21.15.70:0/1942850726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc85640 0x55bdafc91310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.262+0000 7fb928f97740 1 -- 172.21.15.70:0/1942850726 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55bdafb81f80 con 0x55bdafc85640 2024-04-01T01:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.262+0000 7fb928f97740 1 -- 172.21.15.70:0/1942850726 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bdafc887c0 con 0x55bdafc7fae0 2024-04-01T01:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.262+0000 7fb928f97740 10 client.0 _ll_register_callbacks cb 0x55bdafbda290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.263+0000 7fb924871640 1 -- 172.21.15.70:0/1942850726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc7fae0 msgr2=0x55bdafc89e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.263+0000 7fb924871640 1 --2- 172.21.15.70:0/1942850726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc7fae0 0x55bdafc89e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.263+0000 7fb924871640 1 -- 172.21.15.70:0/1942850726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc804d0 msgr2=0x55bdafc8a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.263+0000 7fb924871640 1 --2- 172.21.15.70:0/1942850726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc804d0 0x55bdafc8a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.263+0000 7fb924871640 1 -- 172.21.15.70:0/1942850726 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdaf9c4620 con 0x55bdafc85640 2024-04-01T01:40:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.263+0000 7fb924871640 1 --2- 172.21.15.70:0/1942850726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc85640 0x55bdafc91310 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb920067070 tx=0x7fb920097910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14f02cec746a58fe server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.263+0000 7fb8ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:40:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.264+0000 7fb8ff7fe640 1 -- 172.21.15.70:0/1942850726 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb92009ceb0 con 0x55bdafc85640 2024-04-01T01:40:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.264+0000 7fb8ff7fe640 1 -- 172.21.15.70:0/1942850726 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb9200a2050 con 0x55bdafc85640 2024-04-01T01:40:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.265+0000 7fb928f97740 1 -- 172.21.15.70:0/1942850726 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9140047a0 con 0x55bdafc85640 2024-04-01T01:40:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.268+0000 7fb8ff7fe640 1 -- 172.21.15.70:0/1942850726 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb9200bf440 con 0x55bdafc85640 2024-04-01T01:40:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.269+0000 7fb928f97740 10 client.27644 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.269+0000 7fb928f97740 1 -- 172.21.15.70:0/1942850726 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55bdafc942f0 con 0x55bdafc85640 2024-04-01T01:40:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.270+0000 7fb8ff7fe640 1 -- 172.21.15.70:0/1942850726 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9200bfa90 con 0x55bdafc85640 2024-04-01T01:40:54.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.270+0000 7fb8ff7fe640 1 -- 172.21.15.70:0/1942850726 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb9200bd070 con 0x55bdafc85640 2024-04-01T01:40:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.270+0000 7fb928f97740 10 client.27644 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.270+0000 7fb928f97740 10 client.27644 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:40:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.270+0000 7fb928f97740 20 client.27644 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.270+0000 7fb928f97740 1 --2- 172.21.15.70:0/1942850726 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bdafc959d0 0x55bdafc97e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.270+0000 7fb928f97740 4 client.27644 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:40:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.270+0000 7fb928f97740 1 -- 172.21.15.70:0/1942850726 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb9140047a0 con 0x55bdafc959d0 2024-04-01T01:40:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.270+0000 7fb91ffff640 1 --2- 172.21.15.70:0/1942850726 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bdafc959d0 0x55bdafc97e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.271+0000 7fb8ff7fe640 10 client.27644 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.271+0000 7fb91ffff640 1 --2- 172.21.15.70:0/1942850726 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bdafc959d0 0x55bdafc97e50 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a78a97be59ddbd6 server_cookie=191c93d5e1e99945 in_seq=0 out_seq=0 2024-04-01T01:40:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.272+0000 7fb8ff7fe640 1 -- 172.21.15.70:0/1942850726 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fb9140047a0 con 0x55bdafc959d0 2024-04-01T01:40:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.272+0000 7fb8ff7fe640 10 client.27644 handle_command_reply: tid=0 2024-04-01T01:40:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.272+0000 7fb928f97740 1 client.27644 shutdown 2024-04-01T01:40:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.273+0000 7fb928f97740 1 -- 172.21.15.70:0/1942850726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc85640 msgr2=0x55bdafc91310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.273+0000 7fb928f97740 1 --2- 172.21.15.70:0/1942850726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc85640 0x55bdafc91310 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb920067070 tx=0x7fb920097910 comp rx=0 tx=0).stop 2024-04-01T01:40:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.273+0000 7fb928f97740 1 -- 172.21.15.70:0/1942850726 shutdown_connections 2024-04-01T01:40:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.273+0000 7fb928f97740 1 --2- 172.21.15.70:0/1942850726 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bdafc959d0 0x55bdafc97e50 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.273+0000 7fb8ff7fe640 0 client.27644 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.273+0000 7fb928f97740 1 --2- 172.21.15.70:0/1942850726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc85640 0x55bdafc91310 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.273+0000 7fb928f97740 1 --2- 172.21.15.70:0/1942850726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc804d0 0x55bdafc8a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.273+0000 7fb928f97740 1 --2- 172.21.15.70:0/1942850726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc7fae0 0x55bdafc89e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.273+0000 7fb928f97740 1 -- 172.21.15.70:0/1942850726 >> 172.21.15.70:0/1942850726 conn(0x55bdafc7b050 msgr2=0x55bdafc7a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.273+0000 7fb928f97740 1 -- 172.21.15.70:0/1942850726 shutdown_connections 2024-04-01T01:40:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.273+0000 7fb928f97740 1 -- 172.21.15.70:0/1942850726 wait complete. 2024-04-01T01:40:54.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.274+0000 7fb928f97740 20 client.27644 trim_cache size 0 max 16384 2024-04-01T01:40:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.303+0000 7fb928f97740 1 Processor -- start 2024-04-01T01:40:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.305+0000 7fb928f97740 1 -- start start 2024-04-01T01:40:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.305+0000 7fb928f97740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc7fae0 0x55bdafc84d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.305+0000 7fb928f97740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc804d0 0x55bdafc912d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.305+0000 7fb928f97740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc85640 0x55bdafc89e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.305+0000 7fb928f97740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb91814d810 con 0x55bdafc7fae0 2024-04-01T01:40:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.305+0000 7fb928f97740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb91814f350 con 0x55bdafc804d0 2024-04-01T01:40:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.305+0000 7fb928f97740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb91814dee0 con 0x55bdafc85640 2024-04-01T01:40:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.306+0000 7fb91ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc7fae0 0x55bdafc84d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.306+0000 7fb91ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc7fae0 0x55bdafc84d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59302/0 (socket says 172.21.15.70:59302) 2024-04-01T01:40:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.306+0000 7fb91ffff640 1 -- 172.21.15.70:0/2614796944 learned_addr learned my addr 172.21.15.70:0/2614796944 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.307+0000 7fb91f7fe640 1 --2- 172.21.15.70:0/2614796944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc804d0 0x55bdafc912d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.307+0000 7fb924871640 1 --2- 172.21.15.70:0/2614796944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc85640 0x55bdafc89e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.308+0000 7fb91f7fe640 1 -- 172.21.15.70:0/2614796944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc85640 msgr2=0x55bdafc89e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.308+0000 7fb91f7fe640 1 --2- 172.21.15.70:0/2614796944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc85640 0x55bdafc89e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.308+0000 7fb91f7fe640 1 -- 172.21.15.70:0/2614796944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc7fae0 msgr2=0x55bdafc84d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.308+0000 7fb91f7fe640 1 --2- 172.21.15.70:0/2614796944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc7fae0 0x55bdafc84d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.308+0000 7fb91f7fe640 1 -- 172.21.15.70:0/2614796944 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdafc808b0 con 0x55bdafc804d0 2024-04-01T01:40:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.308+0000 7fb91f7fe640 1 --2- 172.21.15.70:0/2614796944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc804d0 0x55bdafc912d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0cc5a0 tx=0x7fb90c0d2910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aff71bfe50d7507 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.309+0000 7fb909ffb640 1 -- 172.21.15.70:0/2614796944 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90c09aec0 con 0x55bdafc804d0 2024-04-01T01:40:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.309+0000 7fb909ffb640 1 -- 172.21.15.70:0/2614796944 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb90c010de0 con 0x55bdafc804d0 2024-04-01T01:40:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.309+0000 7fb909ffb640 1 -- 172.21.15.70:0/2614796944 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90c097bd0 con 0x55bdafc804d0 2024-04-01T01:40:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.310+0000 7fb928f97740 1 -- 172.21.15.70:0/2614796944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc804d0 msgr2=0x55bdafc912d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.310+0000 7fb928f97740 1 --2- 172.21.15.70:0/2614796944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc804d0 0x55bdafc912d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0cc5a0 tx=0x7fb90c0d2910 comp rx=0 tx=0).stop 2024-04-01T01:40:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.310+0000 7fb928f97740 1 -- 172.21.15.70:0/2614796944 shutdown_connections 2024-04-01T01:40:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.310+0000 7fb928f97740 1 --2- 172.21.15.70:0/2614796944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc85640 0x55bdafc89e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.310+0000 7fb928f97740 1 --2- 172.21.15.70:0/2614796944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc804d0 0x55bdafc912d0 secure :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0cc5a0 tx=0x7fb90c0d2910 comp rx=0 tx=0).stop 2024-04-01T01:40:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.310+0000 7fb928f97740 1 --2- 172.21.15.70:0/2614796944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc7fae0 0x55bdafc84d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.310+0000 7fb928f97740 1 -- 172.21.15.70:0/2614796944 >> 172.21.15.70:0/2614796944 conn(0x55bdafc7b050 msgr2=0x55bdafc7b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.311+0000 7fb928f97740 1 -- 172.21.15.70:0/2614796944 shutdown_connections 2024-04-01T01:40:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.311+0000 7fb928f97740 1 -- 172.21.15.70:0/2614796944 wait complete. 2024-04-01T01:40:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.312+0000 7fb928f97740 1 -- start start 2024-04-01T01:40:54.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.312+0000 7fb928f97740 1 Processor -- start 2024-04-01T01:40:54.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.313+0000 7fb928f97740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc7fae0 0x55bdafc8ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.313+0000 7fb928f97740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc85640 0x55bdafc8ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.313+0000 7fb928f97740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc959d0 0x55bdafc8ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.314+0000 7fb91f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc85640 0x55bdafc8ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.314+0000 7fb91f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc85640 0x55bdafc8ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40848/0 (socket says 172.21.15.70:40848) 2024-04-01T01:40:54.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.314+0000 7fb91f7fe640 1 -- 172.21.15.70:0/2268754202 learned_addr learned my addr 172.21.15.70:0/2268754202 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:54.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.314+0000 7fb924871640 1 --2- 172.21.15.70:0/2268754202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc959d0 0x55bdafc8ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.314+0000 7fb91ffff640 1 --2- 172.21.15.70:0/2268754202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc7fae0 0x55bdafc8ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.314+0000 7fb928f97740 1 -- 172.21.15.70:0/2268754202 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb91814ec80 con 0x55bdafc7fae0 2024-04-01T01:40:54.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.314+0000 7fb928f97740 1 -- 172.21.15.70:0/2268754202 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb91814e5b0 con 0x55bdafc959d0 2024-04-01T01:40:54.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.314+0000 7fb928f97740 1 -- 172.21.15.70:0/2268754202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb91814b7f0 con 0x55bdafc85640 2024-04-01T01:40:54.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.315+0000 7fb928f97740 10 client.0 _ll_register_callbacks cb 0x7fb914005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:40:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.315+0000 7fb91f7fe640 1 -- 172.21.15.70:0/2268754202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc959d0 msgr2=0x55bdafc8ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.315+0000 7fb91f7fe640 1 --2- 172.21.15.70:0/2268754202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc959d0 0x55bdafc8ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.315+0000 7fb91f7fe640 1 -- 172.21.15.70:0/2268754202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc7fae0 msgr2=0x55bdafc8ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:40:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.315+0000 7fb91f7fe640 1 --2- 172.21.15.70:0/2268754202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc7fae0 0x55bdafc8ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.315+0000 7fb91f7fe640 1 -- 172.21.15.70:0/2268754202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdafc90560 con 0x55bdafc85640 2024-04-01T01:40:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.315+0000 7fb91f7fe640 1 --2- 172.21.15.70:0/2268754202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc85640 0x55bdafc8ef60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0979c0 tx=0x7fb90c00c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e683c6aa10f295 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.315+0000 7fb928f97740 1 -- 172.21.15.70:0/2268754202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bdafbf7120 con 0x55bdafc85640 2024-04-01T01:40:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.316+0000 7fb8ff7fe640 10 client.27493 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:40:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.316+0000 7fb8ff7fe640 1 -- 172.21.15.70:0/2268754202 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90c0b3070 con 0x55bdafc85640 2024-04-01T01:40:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.316+0000 7fb8ff7fe640 1 -- 172.21.15.70:0/2268754202 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb90c0a3ae0 con 0x55bdafc85640 2024-04-01T01:40:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.317+0000 7fb8ff7fe640 1 -- 172.21.15.70:0/2268754202 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb90c0a3ce0 con 0x55bdafc85640 2024-04-01T01:40:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.317+0000 7fb928f97740 10 client.27493 fetch_fsmap learned FSMap version 31 2024-04-01T01:40:54.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.317+0000 7fb928f97740 1 -- 172.21.15.70:0/2268754202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55bdafd0cda0 con 0x55bdafc85640 2024-04-01T01:40:54.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.317+0000 7fb8ff7fe640 1 -- 172.21.15.70:0/2268754202 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb90c0a8070 con 0x55bdafc85640 2024-04-01T01:40:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.318+0000 7fb928f97740 10 client.27493 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:40:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.318+0000 7fb928f97740 10 client.27493 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:40:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.318+0000 7fb928f97740 20 client.27493 populate_metadata read hostname 'smithi070' 2024-04-01T01:40:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.318+0000 7fb928f97740 1 --2- 172.21.15.70:0/2268754202 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bdafd0d510 0x55bdafd0f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.318+0000 7fb928f97740 4 client.27493 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:40:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.318+0000 7fb928f97740 1 -- 172.21.15.70:0/2268754202 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bdafbf7120 con 0x55bdafd0d510 2024-04-01T01:40:54.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.321+0000 7fb91ffff640 1 --2- 172.21.15.70:0/2268754202 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bdafd0d510 0x55bdafd0f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:54.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.322+0000 7fb91ffff640 1 --2- 172.21.15.70:0/2268754202 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bdafd0d510 0x55bdafd0f930 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e98a1cef2ed1a77 server_cookie=addc66728a24f420 in_seq=0 out_seq=0 2024-04-01T01:40:54.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.322+0000 7fb8ff7fe640 10 client.27493 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:54.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.474+0000 7fb8ff7fe640 1 -- 172.21.15.70:0/2268754202 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bdafbf7120 con 0x55bdafd0d510 2024-04-01T01:40:54.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.474+0000 7fb8ff7fe640 10 client.27493 handle_command_reply: tid=0 2024-04-01T01:40:54.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.474+0000 7fb928f97740 1 client.27493 shutdown 2024-04-01T01:40:54.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.475+0000 7fb928f97740 1 -- 172.21.15.70:0/2268754202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc85640 msgr2=0x55bdafc8ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:54.475 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.475+0000 7fb928f97740 1 --2- 172.21.15.70:0/2268754202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc85640 0x55bdafc8ef60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb90c0979c0 tx=0x7fb90c00c840 comp rx=0 tx=0).stop 2024-04-01T01:40:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.475+0000 7fb928f97740 1 -- 172.21.15.70:0/2268754202 shutdown_connections 2024-04-01T01:40:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.475+0000 7fb928f97740 1 --2- 172.21.15.70:0/2268754202 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bdafd0d510 0x55bdafd0f930 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.475+0000 7fb928f97740 1 --2- 172.21.15.70:0/2268754202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bdafc959d0 0x55bdafc8ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.475+0000 7fb928f97740 1 --2- 172.21.15.70:0/2268754202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bdafc85640 0x55bdafc8ef60 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.475+0000 7fb928f97740 1 --2- 172.21.15.70:0/2268754202 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bdafc7fae0 0x55bdafc8ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.475+0000 7fb928f97740 1 -- 172.21.15.70:0/2268754202 >> 172.21.15.70:0/2268754202 conn(0x55bdafc7b050 msgr2=0x55bdafc89d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.476+0000 7fb8ff7fe640 0 client.27493 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:40:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.476+0000 7fb928f97740 1 -- 172.21.15.70:0/2268754202 shutdown_connections 2024-04-01T01:40:54.477 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.476+0000 7fb928f97740 1 -- 172.21.15.70:0/2268754202 wait complete. 2024-04-01T01:40:54.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.477+0000 7fb928f97740 20 client.27493 trim_cache size 0 max 16384 2024-04-01T01:40:54.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:40:54.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:40:54.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:40:54.478 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:40:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.477+0000 7fb925072640 1 -- 172.21.15.70:0/1664035830 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9100a25f0 msgr2=0x7fb9100a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.477+0000 7fb925072640 1 --2- 172.21.15.70:0/1664035830 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9100a25f0 0x7fb9100a4ab0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb92002d700 tx=0x7fb920078040 comp rx=0 tx=0).stop 2024-04-01T01:40:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.477+0000 7fb925072640 1 -- 172.21.15.70:0/1664035830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb918096f70 msgr2=0x7fb9180a54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.477+0000 7fb925072640 1 --2- 172.21.15.70:0/1664035830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb918096f70 0x7fb9180a54b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00e9f0 tx=0x7fb90c00eeb0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.478+0000 7fb925072640 1 -- 172.21.15.70:0/1664035830 shutdown_connections 2024-04-01T01:40:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.478+0000 7fb925072640 1 --2- 172.21.15.70:0/1664035830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb918097930 0x7fb918142690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.478+0000 7fb925072640 1 --2- 172.21.15.70:0/1664035830 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9100a25f0 0x7fb9100a4ab0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.478+0000 7fb925072640 1 --2- 172.21.15.70:0/1664035830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb918096f70 0x7fb9180a54b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.478+0000 7fb925072640 1 --2- 172.21.15.70:0/1664035830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb918095d80 0x7fb9180a4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.478+0000 7fb925072640 1 -- 172.21.15.70:0/1664035830 >> 172.21.15.70:0/1664035830 conn(0x7fb9180912d0 msgr2=0x7fb91809a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.478+0000 7fb925072640 1 -- 172.21.15.70:0/1664035830 shutdown_connections 2024-04-01T01:40:54.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:40:54.478+0000 7fb925072640 1 -- 172.21.15.70:0/1664035830 wait complete. 2024-04-01T01:40:55.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:54 smithi189 ceph-mon[35753]: pgmap v582: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 11 MiB/s wr, 945 op/s 2024-04-01T01:40:55.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:54 smithi070 ceph-mon[32179]: pgmap v582: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 11 MiB/s wr, 945 op/s 2024-04-01T01:40:55.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:54 smithi081 ceph-mon[35142]: pgmap v582: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 11 MiB/s wr, 945 op/s 2024-04-01T01:40:56.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 370.0 s, 502.6 tps, lat 200.402 ms stddev 293.166 2024-04-01T01:40:57.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:56 smithi189 ceph-mon[35753]: pgmap v583: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 12 MiB/s wr, 1.09k op/s 2024-04-01T01:40:57.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:56 smithi070 ceph-mon[32179]: pgmap v583: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 12 MiB/s wr, 1.09k op/s 2024-04-01T01:40:57.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:56 smithi081 ceph-mon[35142]: pgmap v583: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 12 MiB/s wr, 1.09k op/s 2024-04-01T01:40:59.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:40:58 smithi189 ceph-mon[35753]: pgmap v584: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-04-01T01:40:59.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:40:58 smithi070 ceph-mon[32179]: pgmap v584: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-04-01T01:40:59.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:40:58 smithi081 ceph-mon[35142]: pgmap v584: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-04-01T01:40:59.783 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:40:59.783 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:40:59.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.976+0000 7fb62cfe5640 1 -- 172.21.15.70:0/470443380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62810b0c0 msgr2=0x7fb628111e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:59.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.976+0000 7fb62cfe5640 1 --2- 172.21.15.70:0/470443380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62810b0c0 0x7fb628111e90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009860 tx=0x7fb61c02bb70 comp rx=0 tx=0).stop 2024-04-01T01:40:59.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.977+0000 7fb62cfe5640 1 -- 172.21.15.70:0/470443380 shutdown_connections 2024-04-01T01:40:59.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.977+0000 7fb62cfe5640 1 --2- 172.21.15.70:0/470443380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb62810b540 0x7fb6281143e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:59.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.977+0000 7fb62cfe5640 1 --2- 172.21.15.70:0/470443380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62810b0c0 0x7fb628111e90 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:59.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.977+0000 7fb62cfe5640 1 --2- 172.21.15.70:0/470443380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb628108750 0x7fb62810ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:59.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.977+0000 7fb62cfe5640 1 -- 172.21.15.70:0/470443380 >> 172.21.15.70:0/470443380 conn(0x7fb62806e820 msgr2=0x7fb628070c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:40:59.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.978+0000 7fb62cfe5640 1 -- 172.21.15.70:0/470443380 shutdown_connections 2024-04-01T01:40:59.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.981+0000 7fb62cfe5640 1 -- 172.21.15.70:0/470443380 wait complete. 2024-04-01T01:40:59.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.983+0000 7fb62cfe5640 1 Processor -- start 2024-04-01T01:40:59.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.986+0000 7fb62cfe5640 1 -- start start 2024-04-01T01:40:59.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.986+0000 7fb62cfe5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb628108750 0x7fb6281a66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:59.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.986+0000 7fb62cfe5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62810b0c0 0x7fb6281a6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:59.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.986+0000 7fb62cfe5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb62810b540 0x7fb6281adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:59.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.986+0000 7fb62cfe5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb62810cbe0 con 0x7fb628108750 2024-04-01T01:40:59.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.986+0000 7fb62cfe5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb62810ca60 con 0x7fb62810b0c0 2024-04-01T01:40:59.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.986+0000 7fb62cfe5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb62810cd60 con 0x7fb62810b540 2024-04-01T01:40:59.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.986+0000 7fb62659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb628108750 0x7fb6281a66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:59.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.987+0000 7fb62659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb628108750 0x7fb6281a66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38348/0 (socket says 172.21.15.70:38348) 2024-04-01T01:40:59.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.987+0000 7fb62659c640 1 -- 172.21.15.70:0/2069777242 learned_addr learned my addr 172.21.15.70:0/2069777242 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:40:59.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.987+0000 7fb626d9d640 1 --2- 172.21.15.70:0/2069777242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb62810b540 0x7fb6281adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:59.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.987+0000 7fb625d9b640 1 --2- 172.21.15.70:0/2069777242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62810b0c0 0x7fb6281a6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:59.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.988+0000 7fb62659c640 1 -- 172.21.15.70:0/2069777242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb62810b540 msgr2=0x7fb6281adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:59.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.988+0000 7fb62659c640 1 --2- 172.21.15.70:0/2069777242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb62810b540 0x7fb6281adc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:59.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.988+0000 7fb62659c640 1 -- 172.21.15.70:0/2069777242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62810b0c0 msgr2=0x7fb6281a6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:40:59.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.988+0000 7fb62659c640 1 --2- 172.21.15.70:0/2069777242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62810b0c0 0x7fb6281a6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:40:59.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.988+0000 7fb62659c640 1 -- 172.21.15.70:0/2069777242 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6281ae2e0 con 0x7fb628108750 2024-04-01T01:40:59.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.988+0000 7fb626d9d640 1 --2- 172.21.15.70:0/2069777242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb62810b540 0x7fb6281adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:40:59.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.989+0000 7fb62659c640 1 --2- 172.21.15.70:0/2069777242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb628108750 0x7fb6281a66c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb61800d770 tx=0x7fb61800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:40:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.993+0000 7fb6177fe640 1 -- 172.21.15.70:0/2069777242 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb618017020 con 0x7fb628108750 2024-04-01T01:40:59.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.993+0000 7fb62cfe5640 1 -- 172.21.15.70:0/2069777242 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6281ae5d0 con 0x7fb628108750 2024-04-01T01:40:59.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.993+0000 7fb62cfe5640 1 -- 172.21.15.70:0/2069777242 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6281aeaf0 con 0x7fb628108750 2024-04-01T01:40:59.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.994+0000 7fb6177fe640 1 -- 172.21.15.70:0/2069777242 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb618004440 con 0x7fb628108750 2024-04-01T01:40:59.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.994+0000 7fb6177fe640 1 -- 172.21.15.70:0/2069777242 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb618003cd0 con 0x7fb628108750 2024-04-01T01:40:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.999+0000 7fb62fbc6740 1 Processor -- start 2024-04-01T01:40:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.999+0000 7fb62fbc6740 1 -- start start 2024-04-01T01:40:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.999+0000 7fb62fbc6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a6c490 0x55a091a6c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.999+0000 7fb62fbc6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6ce80 0x55a091a71980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.999+0000 7fb62fbc6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 0x55a091a74480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:40:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.999+0000 7fb62fbc6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a0919d1180 con 0x55a091a6c490 2024-04-01T01:40:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.999+0000 7fb62fbc6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a091979b70 con 0x55a091a6ce80 2024-04-01T01:40:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:40:59.999+0000 7fb62fbc6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a0919cd220 con 0x55a091a71ff0 2024-04-01T01:40:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.000+0000 7fb625d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 0x55a091a74480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:40:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.000+0000 7fb625d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 0x55a091a74480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42924/0 (socket says 172.21.15.70:42924) 2024-04-01T01:40:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.000+0000 7fb625d9b640 1 -- 172.21.15.70:0/488513089 learned_addr learned my addr 172.21.15.70:0/488513089 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:00.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.000+0000 7fb6177fe640 1 -- 172.21.15.70:0/2069777242 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb61801a020 con 0x7fb628108750 2024-04-01T01:41:00.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.000+0000 7fb625d9b640 1 -- 172.21.15.70:0/488513089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6ce80 msgr2=0x55a091a71980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:00.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.000+0000 7fb625d9b640 1 --2- 172.21.15.70:0/488513089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6ce80 0x55a091a71980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.000+0000 7fb625d9b640 1 -- 172.21.15.70:0/488513089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a6c490 msgr2=0x55a091a6c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:00.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.000+0000 7fb625d9b640 1 --2- 172.21.15.70:0/488513089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a6c490 0x55a091a6c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.000+0000 7fb625d9b640 1 -- 172.21.15.70:0/488513089 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a09198d7c0 con 0x55a091a71ff0 2024-04-01T01:41:00.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.001+0000 7fb625d9b640 1 --2- 172.21.15.70:0/488513089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 0x55a091a74480 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009860 tx=0x7fb61c02c5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40da481ff20240b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:00.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.001+0000 7fb614ff9640 1 -- 172.21.15.70:0/488513089 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c043020 con 0x55a091a71ff0 2024-04-01T01:41:00.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.001+0000 7fb6177fe640 1 --2- 172.21.15.70:0/2069777242 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6100a2620 0x7fb6100a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:00.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.001+0000 7fb614ff9640 1 -- 172.21.15.70:0/488513089 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb61c02c7b0 con 0x55a091a71ff0 2024-04-01T01:41:00.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.001+0000 7fb6177fe640 1 -- 172.21.15.70:0/2069777242 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb6180ce600 con 0x7fb628108750 2024-04-01T01:41:00.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.001+0000 7fb626d9d640 1 --2- 172.21.15.70:0/2069777242 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6100a2620 0x7fb6100a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:00.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.002+0000 7fb614ff9640 1 -- 172.21.15.70:0/488513089 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c039040 con 0x55a091a71ff0 2024-04-01T01:41:00.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.002+0000 7fb62fbc6740 1 -- 172.21.15.70:0/488513089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 msgr2=0x55a091a74480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:00.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.002+0000 7fb626d9d640 1 --2- 172.21.15.70:0/2069777242 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6100a2620 0x7fb6100a4ae0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb620000c00 tx=0x7fb62000ae90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.002+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/488513089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 0x55a091a74480 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009860 tx=0x7fb61c02c5c0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.002+0000 7fb62fbc6740 1 -- 172.21.15.70:0/488513089 shutdown_connections 2024-04-01T01:41:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.002+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/488513089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 0x55a091a74480 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.002+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/488513089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6ce80 0x55a091a71980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.002+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/488513089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a6c490 0x55a091a6c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.003+0000 7fb62fbc6740 1 -- 172.21.15.70:0/488513089 >> 172.21.15.70:0/488513089 conn(0x55a091a67a00 msgr2=0x55a091a67e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.003+0000 7fb62fbc6740 1 -- 172.21.15.70:0/488513089 shutdown_connections 2024-04-01T01:41:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.003+0000 7fb62fbc6740 1 -- 172.21.15.70:0/488513089 wait complete. 2024-04-01T01:41:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.003+0000 7fb62fbc6740 1 -- start start 2024-04-01T01:41:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.004+0000 7fb62fbc6740 1 Processor -- start 2024-04-01T01:41:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.004+0000 7fb62fbc6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a6c490 0x55a091a769d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.004+0000 7fb62fbc6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6ce80 0x55a091a76f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.004+0000 7fb62fbc6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 0x55a091a7df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.004+0000 7fb62fbc6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a091a74fe0 con 0x55a091a6c490 2024-04-01T01:41:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.004+0000 7fb62fbc6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a09196f4a0 con 0x55a091a6ce80 2024-04-01T01:41:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.004+0000 7fb62fbc6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a091a75160 con 0x55a091a71ff0 2024-04-01T01:41:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.004+0000 7fb62fbc6740 10 client.0 _ll_register_callbacks cb 0x55a091972d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.004+0000 7fb62659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a6c490 0x55a091a769d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:00.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.004+0000 7fb626d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 0x55a091a7df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:00.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.004+0000 7fb62659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a6c490 0x55a091a769d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38376/0 (socket says 172.21.15.70:38376) 2024-04-01T01:41:00.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.004+0000 7fb62659c640 1 -- 172.21.15.70:0/3696031296 learned_addr learned my addr 172.21.15.70:0/3696031296 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:00.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.005+0000 7fb625d9b640 1 --2- 172.21.15.70:0/3696031296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6ce80 0x55a091a76f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:00.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.005+0000 7fb62659c640 1 -- 172.21.15.70:0/3696031296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 msgr2=0x55a091a7df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:00.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.005+0000 7fb62659c640 1 --2- 172.21.15.70:0/3696031296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 0x55a091a7df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.005+0000 7fb62659c640 1 -- 172.21.15.70:0/3696031296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6ce80 msgr2=0x55a091a76f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:00.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.005+0000 7fb62659c640 1 --2- 172.21.15.70:0/3696031296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6ce80 0x55a091a76f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.005+0000 7fb62659c640 1 -- 172.21.15.70:0/3696031296 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0917af7d0 con 0x55a091a6c490 2024-04-01T01:41:00.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.005+0000 7fb625d9b640 1 --2- 172.21.15.70:0/3696031296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6ce80 0x55a091a76f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:41:00.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.005+0000 7fb626d9d640 1 --2- 172.21.15.70:0/3696031296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 0x55a091a7df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:41:00.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.005+0000 7fb62659c640 1 --2- 172.21.15.70:0/3696031296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a6c490 0x55a091a769d0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb6180a3170 tx=0x7fb61809a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b89a1e413a0b0dfb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:00.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.005+0000 7fb60dffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:41:00.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.005+0000 7fb60dffb640 1 -- 172.21.15.70:0/3696031296 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61809ede0 con 0x55a091a6c490 2024-04-01T01:41:00.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.006+0000 7fb62fbc6740 1 -- 172.21.15.70:0/3696031296 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb61c02c7b0 con 0x55a091a6c490 2024-04-01T01:41:00.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.006+0000 7fb60dffb640 1 -- 172.21.15.70:0/3696031296 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb618097300 con 0x55a091a6c490 2024-04-01T01:41:00.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.006+0000 7fb60dffb640 1 -- 172.21.15.70:0/3696031296 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb6180a36a0 con 0x55a091a6c490 2024-04-01T01:41:00.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.007+0000 7fb62fbc6740 10 client.17724 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:00.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.007+0000 7fb62fbc6740 1 -- 172.21.15.70:0/3696031296 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a0916e53f0 con 0x55a091a6c490 2024-04-01T01:41:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.008+0000 7fb60dffb640 1 -- 172.21.15.70:0/3696031296 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb6180969e0 con 0x55a091a6c490 2024-04-01T01:41:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.008+0000 7fb60dffb640 1 -- 172.21.15.70:0/3696031296 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb618099cb0 con 0x55a091a6c490 2024-04-01T01:41:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.008+0000 7fb62fbc6740 10 client.17724 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.008+0000 7fb62fbc6740 10 client.17724 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.008+0000 7fb62fbc6740 20 client.17724 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.008+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/3696031296 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a091a82c50 0x55a091a850b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.008+0000 7fb62fbc6740 4 client.17724 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:00.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.008+0000 7fb62fbc6740 1 -- 172.21.15.70:0/3696031296 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb61c02c7b0 con 0x55a091a82c50 2024-04-01T01:41:00.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.009+0000 7fb625d9b640 1 --2- 172.21.15.70:0/3696031296 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a091a82c50 0x55a091a850b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:00.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.010+0000 7fb625d9b640 1 --2- 172.21.15.70:0/3696031296 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a091a82c50 0x55a091a850b0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f270ec1ddd7b657e server_cookie=26bec12ccffe4767 in_seq=0 out_seq=0 2024-04-01T01:41:00.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.010+0000 7fb60dffb640 10 client.17724 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:00.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.011+0000 7fb60dffb640 1 -- 172.21.15.70:0/3696031296 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fb61c02c7b0 con 0x55a091a82c50 2024-04-01T01:41:00.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.011+0000 7fb60dffb640 10 client.17724 handle_command_reply: tid=0 2024-04-01T01:41:00.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.012+0000 7fb62fbc6740 1 client.17724 shutdown 2024-04-01T01:41:00.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.013+0000 7fb62fbc6740 1 -- 172.21.15.70:0/3696031296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a6c490 msgr2=0x55a091a769d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:00.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.013+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/3696031296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a6c490 0x55a091a769d0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb6180a3170 tx=0x7fb61809a5f0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.014+0000 7fb62fbc6740 1 -- 172.21.15.70:0/3696031296 shutdown_connections 2024-04-01T01:41:00.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.014+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/3696031296 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a091a82c50 0x55a091a850b0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.014+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/3696031296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 0x55a091a7df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.015+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/3696031296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6ce80 0x55a091a76f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.015+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/3696031296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a6c490 0x55a091a769d0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.015+0000 7fb62fbc6740 1 -- 172.21.15.70:0/3696031296 >> 172.21.15.70:0/3696031296 conn(0x55a091a67a00 msgr2=0x55a091a66c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:00.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.015+0000 7fb60dffb640 0 client.17724 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:00.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.015+0000 7fb62fbc6740 1 -- 172.21.15.70:0/3696031296 shutdown_connections 2024-04-01T01:41:00.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.015+0000 7fb62fbc6740 1 -- 172.21.15.70:0/3696031296 wait complete. 2024-04-01T01:41:00.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.017+0000 7fb62fbc6740 20 client.17724 trim_cache size 0 max 16384 2024-04-01T01:41:00.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.058+0000 7fb62fbc6740 1 Processor -- start 2024-04-01T01:41:00.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.058+0000 7fb62fbc6740 1 -- start start 2024-04-01T01:41:00.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.058+0000 7fb62fbc6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6c490 0x55a091a67de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:00.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.059+0000 7fb62fbc6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a6ce80 0x55a091a7df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:00.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.059+0000 7fb62659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6c490 0x55a091a67de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:00.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.059+0000 7fb62659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6c490 0x55a091a67de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57138/0 (socket says 172.21.15.70:57138) 2024-04-01T01:41:00.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.059+0000 7fb62659c640 1 -- 172.21.15.70:0/2030118689 learned_addr learned my addr 172.21.15.70:0/2030118689 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:00.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.059+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/2030118689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 0x55a091a76cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:00.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.059+0000 7fb62fbc6740 1 -- 172.21.15.70:0/2030118689 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb60800fd90 con 0x55a091a6ce80 2024-04-01T01:41:00.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.059+0000 7fb62fbc6740 1 -- 172.21.15.70:0/2030118689 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb6080118d0 con 0x55a091a6c490 2024-04-01T01:41:00.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.059+0000 7fb62fbc6740 1 -- 172.21.15.70:0/2030118689 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb608010460 con 0x55a091a71ff0 2024-04-01T01:41:00.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.059+0000 7fb626d9d640 1 --2- 172.21.15.70:0/2030118689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 0x55a091a76cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:00.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.060+0000 7fb62659c640 1 -- 172.21.15.70:0/2030118689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 msgr2=0x55a091a76cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:00.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.060+0000 7fb62659c640 1 --2- 172.21.15.70:0/2030118689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 0x55a091a76cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.060+0000 7fb62659c640 1 -- 172.21.15.70:0/2030118689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a6ce80 msgr2=0x55a091a7df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:00.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.060+0000 7fb62659c640 1 --2- 172.21.15.70:0/2030118689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a6ce80 0x55a091a7df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.060+0000 7fb62659c640 1 -- 172.21.15.70:0/2030118689 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0916a50f0 con 0x55a091a6c490 2024-04-01T01:41:00.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.061+0000 7fb62659c640 1 --2- 172.21.15.70:0/2030118689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6c490 0x55a091a67de0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb618009610 tx=0x7fb61809ae10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e3b6ea8c6cfef85 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:00.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.061+0000 7fb60ffff640 1 -- 172.21.15.70:0/2030118689 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6180a8070 con 0x55a091a6c490 2024-04-01T01:41:00.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.061+0000 7fb60ffff640 1 -- 172.21.15.70:0/2030118689 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb618097300 con 0x55a091a6c490 2024-04-01T01:41:00.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.061+0000 7fb60ffff640 1 -- 172.21.15.70:0/2030118689 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb618099b50 con 0x55a091a6c490 2024-04-01T01:41:00.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.062+0000 7fb62fbc6740 1 -- 172.21.15.70:0/2030118689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6c490 msgr2=0x55a091a67de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:00.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.062+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/2030118689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6c490 0x55a091a67de0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fb618009610 tx=0x7fb61809ae10 comp rx=0 tx=0).stop 2024-04-01T01:41:00.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.063+0000 7fb62fbc6740 1 -- 172.21.15.70:0/2030118689 shutdown_connections 2024-04-01T01:41:00.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.063+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/2030118689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a71ff0 0x55a091a76cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.063+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/2030118689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a6ce80 0x55a091a7df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.063+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/2030118689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6c490 0x55a091a67de0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.063+0000 7fb62fbc6740 1 -- 172.21.15.70:0/2030118689 >> 172.21.15.70:0/2030118689 conn(0x55a091a67a00 msgr2=0x55a091a765e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:00.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.064+0000 7fb62fbc6740 1 -- 172.21.15.70:0/2030118689 shutdown_connections 2024-04-01T01:41:00.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.064+0000 7fb62fbc6740 1 -- 172.21.15.70:0/2030118689 wait complete. 2024-04-01T01:41:00.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.064+0000 7fb62fbc6740 1 -- start start 2024-04-01T01:41:00.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.065+0000 7fb62fbc6740 1 Processor -- start 2024-04-01T01:41:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.065+0000 7fb62fbc6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a6c490 0x55a091a7b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.065+0000 7fb62fbc6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6ce80 0x55a091a7bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.065+0000 7fb62fbc6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a71ff0 0x55a091a7cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.066+0000 7fb62fbc6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb608011200 con 0x55a091a71ff0 2024-04-01T01:41:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.066+0000 7fb62fbc6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb608010b30 con 0x55a091a6ce80 2024-04-01T01:41:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.066+0000 7fb62fbc6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb60800dd70 con 0x55a091a6c490 2024-04-01T01:41:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.066+0000 7fb62fbc6740 10 client.0 _ll_register_callbacks cb 0x7fb61800de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:00.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.066+0000 7fb626d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a71ff0 0x55a091a7cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:00.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.066+0000 7fb626d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a71ff0 0x55a091a7cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38390/0 (socket says 172.21.15.70:38390) 2024-04-01T01:41:00.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.066+0000 7fb626d9d640 1 -- 172.21.15.70:0/3880000876 learned_addr learned my addr 172.21.15.70:0/3880000876 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:00.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.066+0000 7fb626d9d640 1 -- 172.21.15.70:0/3880000876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a6c490 msgr2=0x55a091a7b810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:41:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.066+0000 7fb626d9d640 1 --2- 172.21.15.70:0/3880000876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a6c490 0x55a091a7b810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.066+0000 7fb626d9d640 1 -- 172.21.15.70:0/3880000876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6ce80 msgr2=0x55a091a7bd50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:41:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.066+0000 7fb626d9d640 1 --2- 172.21.15.70:0/3880000876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6ce80 0x55a091a7bd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.066+0000 7fb626d9d640 1 -- 172.21.15.70:0/3880000876 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a091a7d350 con 0x55a091a71ff0 2024-04-01T01:41:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.067+0000 7fb626d9d640 1 --2- 172.21.15.70:0/3880000876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a71ff0 0x55a091a7cd60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb620001440 tx=0x7fb62000a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d443b304002fb22 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.067+0000 7fb60dffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:41:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.067+0000 7fb60dffb640 1 -- 172.21.15.70:0/3880000876 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb620027450 con 0x55a091a71ff0 2024-04-01T01:41:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.068+0000 7fb60dffb640 1 -- 172.21.15.70:0/3880000876 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb6200275f0 con 0x55a091a71ff0 2024-04-01T01:41:00.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.068+0000 7fb62fbc6740 1 -- 172.21.15.70:0/3880000876 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb618097300 con 0x55a091a71ff0 2024-04-01T01:41:00.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.068+0000 7fb60dffb640 1 -- 172.21.15.70:0/3880000876 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb620035050 con 0x55a091a71ff0 2024-04-01T01:41:00.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.068+0000 7fb62fbc6740 10 client.17727 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:00.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.068+0000 7fb62fbc6740 1 -- 172.21.15.70:0/3880000876 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a091ae9210 con 0x55a091a71ff0 2024-04-01T01:41:00.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.069+0000 7fb60dffb640 1 -- 172.21.15.70:0/3880000876 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb618097300 con 0x55a091a71ff0 2024-04-01T01:41:00.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.069+0000 7fb60dffb640 1 -- 172.21.15.70:0/3880000876 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb620019960 con 0x55a091a71ff0 2024-04-01T01:41:00.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.069+0000 7fb62fbc6740 10 client.17727 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:00.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.069+0000 7fb62fbc6740 10 client.17727 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:00.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.069+0000 7fb62fbc6740 20 client.17727 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:00.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.069+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/3880000876 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a091ae9c40 0x55a091aec060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:00.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.069+0000 7fb62fbc6740 4 client.17727 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:41:00.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.069+0000 7fb62fbc6740 1 -- 172.21.15.70:0/3880000876 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a091a6ddf0 con 0x55a091ae9c40 2024-04-01T01:41:00.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.071+0000 7fb62659c640 1 --2- 172.21.15.70:0/3880000876 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a091ae9c40 0x55a091aec060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:00.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.072+0000 7fb60dffb640 10 client.17727 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:00.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.072+0000 7fb62659c640 1 --2- 172.21.15.70:0/3880000876 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a091ae9c40 0x55a091aec060 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bbfd3680ea2a85c server_cookie=6497b7d6b6f32e1 in_seq=0 out_seq=0 2024-04-01T01:41:00.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.073+0000 7fb60dffb640 1 -- 172.21.15.70:0/3880000876 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a091a6ddf0 con 0x55a091ae9c40 2024-04-01T01:41:00.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.073+0000 7fb60dffb640 10 client.17727 handle_command_reply: tid=0 2024-04-01T01:41:00.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.074+0000 7fb62fbc6740 1 client.17727 shutdown 2024-04-01T01:41:00.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.075+0000 7fb62fbc6740 1 -- 172.21.15.70:0/3880000876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a71ff0 msgr2=0x55a091a7cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:00.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.075+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/3880000876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a71ff0 0x55a091a7cd60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fb620001440 tx=0x7fb62000a680 comp rx=0 tx=0).stop 2024-04-01T01:41:00.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.075+0000 7fb62fbc6740 1 -- 172.21.15.70:0/3880000876 shutdown_connections 2024-04-01T01:41:00.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.075+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/3880000876 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a091ae9c40 0x55a091aec060 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.075+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/3880000876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a091a71ff0 0x55a091a7cd60 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.075+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/3880000876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a091a6ce80 0x55a091a7bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.075+0000 7fb62fbc6740 1 --2- 172.21.15.70:0/3880000876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a091a6c490 0x55a091a7b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.075+0000 7fb62fbc6740 1 -- 172.21.15.70:0/3880000876 >> 172.21.15.70:0/3880000876 conn(0x55a091a67a00 msgr2=0x55a091a66390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:00.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.077+0000 7fb60dffb640 0 client.17727 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:00.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.078+0000 7fb62fbc6740 1 -- 172.21.15.70:0/3880000876 shutdown_connections 2024-04-01T01:41:00.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.079+0000 7fb62fbc6740 1 -- 172.21.15.70:0/3880000876 wait complete. 2024-04-01T01:41:00.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.079+0000 7fb62fbc6740 20 client.17727 trim_cache size 0 max 16384 2024-04-01T01:41:00.079 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:41:00.079 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:41:00.079 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:41:00.080 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:41:00.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.085+0000 7fb62cfe5640 1 -- 172.21.15.70:0/2069777242 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6100a2620 msgr2=0x7fb6100a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:00.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.085+0000 7fb62cfe5640 1 --2- 172.21.15.70:0/2069777242 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6100a2620 0x7fb6100a4ae0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb620000c00 tx=0x7fb62000ae90 comp rx=0 tx=0).stop 2024-04-01T01:41:00.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.085+0000 7fb62cfe5640 1 -- 172.21.15.70:0/2069777242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb628108750 msgr2=0x7fb6281a66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:00.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.085+0000 7fb62cfe5640 1 --2- 172.21.15.70:0/2069777242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb628108750 0x7fb6281a66c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb61800d770 tx=0x7fb61800dc30 comp rx=0 tx=0).stop 2024-04-01T01:41:00.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.087+0000 7fb62cfe5640 1 -- 172.21.15.70:0/2069777242 shutdown_connections 2024-04-01T01:41:00.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.087+0000 7fb62cfe5640 1 --2- 172.21.15.70:0/2069777242 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb6100a2620 0x7fb6100a4ae0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.087+0000 7fb62cfe5640 1 --2- 172.21.15.70:0/2069777242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb62810b540 0x7fb6281adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.087+0000 7fb62cfe5640 1 --2- 172.21.15.70:0/2069777242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62810b0c0 0x7fb6281a6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.087+0000 7fb62cfe5640 1 --2- 172.21.15.70:0/2069777242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb628108750 0x7fb6281a66c0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.087+0000 7fb62cfe5640 1 -- 172.21.15.70:0/2069777242 >> 172.21.15.70:0/2069777242 conn(0x7fb62806e820 msgr2=0x7fb62806f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:00.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.088+0000 7fb62cfe5640 1 -- 172.21.15.70:0/2069777242 shutdown_connections 2024-04-01T01:41:00.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.088+0000 7fb62cfe5640 1 -- 172.21.15.70:0/2069777242 wait complete. 2024-04-01T01:41:00.113 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:41:00.113 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:41:00.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.314+0000 7f8512ffd640 1 -- 172.21.15.70:0/4279993615 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850402cbe0 con 0x7f8514072a30 2024-04-01T01:41:00.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.314+0000 7f85190bc640 1 -- 172.21.15.70:0/4279993615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8514072a30 msgr2=0x7f8514072e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:00.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.314+0000 7f85190bc640 1 --2- 172.21.15.70:0/4279993615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8514072a30 0x7f8514072e50 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f8504009900 tx=0x7f850402bc30 comp rx=0 tx=0).stop 2024-04-01T01:41:00.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.314+0000 7f85190bc640 1 -- 172.21.15.70:0/4279993615 shutdown_connections 2024-04-01T01:41:00.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.314+0000 7f85190bc640 1 --2- 172.21.15.70:0/4279993615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85141096d0 0x7f851410e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.314+0000 7f85190bc640 1 --2- 172.21.15.70:0/4279993615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8514108d10 0x7f8514109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.314+0000 7f85190bc640 1 --2- 172.21.15.70:0/4279993615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8514072a30 0x7f8514072e50 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.314+0000 7f85190bc640 1 -- 172.21.15.70:0/4279993615 >> 172.21.15.70:0/4279993615 conn(0x7f851406e540 msgr2=0x7f8514070980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:00.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.315+0000 7f85190bc640 1 -- 172.21.15.70:0/4279993615 shutdown_connections 2024-04-01T01:41:00.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.315+0000 7f85190bc640 1 -- 172.21.15.70:0/4279993615 wait complete. 2024-04-01T01:41:00.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.315+0000 7f85190bc640 1 Processor -- start 2024-04-01T01:41:00.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.316+0000 7f85190bc640 1 -- start start 2024-04-01T01:41:00.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.316+0000 7f85190bc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8514072a30 0x7f8514117cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:00.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.316+0000 7f85190bc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8514108d10 0x7f8514118210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:00.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.316+0000 7f85190bc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85141096d0 0x7f8514119250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:00.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.316+0000 7f85190bc640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8514110330 con 0x7f8514072a30 2024-04-01T01:41:00.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.316+0000 7f85190bc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f85141101b0 con 0x7f8514108d10 2024-04-01T01:41:00.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.316+0000 7f85190bc640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f85141104b0 con 0x7f85141096d0 2024-04-01T01:41:00.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.317+0000 7f85188bb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85141096d0 0x7f8514119250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:00.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.317+0000 7f85188bb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85141096d0 0x7f8514119250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42976/0 (socket says 172.21.15.70:42976) 2024-04-01T01:41:00.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.317+0000 7f85188bb640 1 -- 172.21.15.70:0/928440316 learned_addr learned my addr 172.21.15.70:0/928440316 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:00.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.317+0000 7f8513fff640 1 --2- 172.21.15.70:0/928440316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8514072a30 0x7f8514117cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:00.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.317+0000 7f85188bb640 1 -- 172.21.15.70:0/928440316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8514108d10 msgr2=0x7f8514118210 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:41:00.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.317+0000 7f85188bb640 1 --2- 172.21.15.70:0/928440316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8514108d10 0x7f8514118210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.317+0000 7f85188bb640 1 -- 172.21.15.70:0/928440316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8514072a30 msgr2=0x7f8514117cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:00.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.317+0000 7f85188bb640 1 --2- 172.21.15.70:0/928440316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8514072a30 0x7f8514117cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.317+0000 7f85188bb640 1 -- 172.21.15.70:0/928440316 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85141198c0 con 0x7f85141096d0 2024-04-01T01:41:00.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.318+0000 7f85188bb640 1 --2- 172.21.15.70:0/928440316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85141096d0 0x7f8514119250 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f850c00eaa0 tx=0x7f850c00ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:00.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.318+0000 7f85117fa640 1 -- 172.21.15.70:0/928440316 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850c00cdb0 con 0x7f85141096d0 2024-04-01T01:41:00.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.318+0000 7f85190bc640 1 -- 172.21.15.70:0/928440316 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8514119bb0 con 0x7f85141096d0 2024-04-01T01:41:00.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.318+0000 7f85190bc640 1 -- 172.21.15.70:0/928440316 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f851411a100 con 0x7f85141096d0 2024-04-01T01:41:00.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.318+0000 7f85117fa640 1 -- 172.21.15.70:0/928440316 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f850c005020 con 0x7f85141096d0 2024-04-01T01:41:00.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.318+0000 7f85117fa640 1 -- 172.21.15.70:0/928440316 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850c0116d0 con 0x7f85141096d0 2024-04-01T01:41:00.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.321+0000 7f85117fa640 1 -- 172.21.15.70:0/928440316 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f850c004a40 con 0x7f85141096d0 2024-04-01T01:41:00.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.322+0000 7f85190bc640 1 -- 172.21.15.70:0/928440316 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85000052c0 con 0x7f85141096d0 2024-04-01T01:41:00.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.322+0000 7f85117fa640 1 --2- 172.21.15.70:0/928440316 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f84fc0a25f0 0x7f84fc0a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:00.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.323+0000 7f85117fa640 1 -- 172.21.15.70:0/928440316 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f850c0ced10 con 0x7f85141096d0 2024-04-01T01:41:00.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.323+0000 7f8513fff640 1 --2- 172.21.15.70:0/928440316 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f84fc0a25f0 0x7f84fc0a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:00.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.324+0000 7f8513fff640 1 --2- 172.21.15.70:0/928440316 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f84fc0a25f0 0x7f84fc0a4ab0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8504006010 tx=0x7f8504000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:00.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.329+0000 7f85117fa640 1 -- 172.21.15.70:0/928440316 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f850c016030 con 0x7f85141096d0 2024-04-01T01:41:00.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.800+0000 7f85190bc640 1 -- 172.21.15.70:0/928440316 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85000055b0 con 0x7f85141096d0 2024-04-01T01:41:00.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.803+0000 7f85117fa640 1 -- 172.21.15.70:0/928440316 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7f850c09baa0 con 0x7f85141096d0 2024-04-01T01:41:00.804 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:41:00.804 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:41:00.804 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:41:00.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.810+0000 7f85190bc640 1 -- 172.21.15.70:0/928440316 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f84fc0a25f0 msgr2=0x7f84fc0a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:00.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.810+0000 7f85190bc640 1 --2- 172.21.15.70:0/928440316 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f84fc0a25f0 0x7f84fc0a4ab0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8504006010 tx=0x7f8504000bc0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.810+0000 7f85190bc640 1 -- 172.21.15.70:0/928440316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85141096d0 msgr2=0x7f8514119250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:00.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.810+0000 7f85190bc640 1 --2- 172.21.15.70:0/928440316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85141096d0 0x7f8514119250 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f850c00eaa0 tx=0x7f850c00ef60 comp rx=0 tx=0).stop 2024-04-01T01:41:00.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.810+0000 7f85190bc640 1 -- 172.21.15.70:0/928440316 shutdown_connections 2024-04-01T01:41:00.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.810+0000 7f85190bc640 1 --2- 172.21.15.70:0/928440316 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f84fc0a25f0 0x7f84fc0a4ab0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.810+0000 7f85190bc640 1 --2- 172.21.15.70:0/928440316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85141096d0 0x7f8514119250 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.810+0000 7f85190bc640 1 --2- 172.21.15.70:0/928440316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8514108d10 0x7f8514118210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.810+0000 7f85190bc640 1 --2- 172.21.15.70:0/928440316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8514072a30 0x7f8514117cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:00.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.810+0000 7f85190bc640 1 -- 172.21.15.70:0/928440316 >> 172.21.15.70:0/928440316 conn(0x7f851406e540 msgr2=0x7f851410c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:00.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.811+0000 7f85190bc640 1 -- 172.21.15.70:0/928440316 shutdown_connections 2024-04-01T01:41:00.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:00.811+0000 7f85190bc640 1 -- 172.21.15.70:0/928440316 wait complete. 2024-04-01T01:41:00.829 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:41:01.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.013+0000 7f424159a640 1 -- 172.21.15.70:0/1286966550 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423402cc60 con 0x7f424410ffd0 2024-04-01T01:41:01.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.014+0000 7f4248fd6640 1 -- 172.21.15.70:0/1286966550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424410ffd0 msgr2=0x7f42441123c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.014+0000 7f4248fd6640 1 --2- 172.21.15.70:0/1286966550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424410ffd0 0x7f42441123c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4234009900 tx=0x7f423402bc80 comp rx=0 tx=0).stop 2024-04-01T01:41:01.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.014+0000 7f4248fd6640 1 -- 172.21.15.70:0/1286966550 shutdown_connections 2024-04-01T01:41:01.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.014+0000 7f4248fd6640 1 --2- 172.21.15.70:0/1286966550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424410ffd0 0x7f42441123c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.014+0000 7f4248fd6640 1 --2- 172.21.15.70:0/1286966550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42441090b0 0x7f424410fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.014+0000 7f4248fd6640 1 --2- 172.21.15.70:0/1286966550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4244108750 0x7f4244108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.014+0000 7f4248fd6640 1 -- 172.21.15.70:0/1286966550 >> 172.21.15.70:0/1286966550 conn(0x7f424406e950 msgr2=0x7f4244070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.016+0000 7f4248fd6640 1 -- 172.21.15.70:0/1286966550 shutdown_connections 2024-04-01T01:41:01.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.017+0000 7f4248fd6640 1 -- 172.21.15.70:0/1286966550 wait complete. 2024-04-01T01:41:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.017+0000 7f4248fd6640 1 Processor -- start 2024-04-01T01:41:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.018+0000 7f4248fd6640 1 -- start start 2024-04-01T01:41:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.018+0000 7f4248fd6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4244108750 0x7f42441a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.018+0000 7f4248fd6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42441090b0 0x7f42441a6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.018+0000 7f4248fd6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f424410ffd0 0x7f42441ade40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.018+0000 7f4248fd6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f42441139a0 con 0x7f42441090b0 2024-04-01T01:41:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.018+0000 7f4248fd6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4244113820 con 0x7f4244108750 2024-04-01T01:41:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.018+0000 7f4248fd6640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4244113b20 con 0x7f424410ffd0 2024-04-01T01:41:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.018+0000 7f4242d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f424410ffd0 0x7f42441ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.018+0000 7f4242d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f424410ffd0 0x7f42441ade40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42996/0 (socket says 172.21.15.70:42996) 2024-04-01T01:41:01.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.018+0000 7f4242d9d640 1 -- 172.21.15.70:0/2263406055 learned_addr learned my addr 172.21.15.70:0/2263406055 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.019+0000 7f4241d9b640 1 --2- 172.21.15.70:0/2263406055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42441090b0 0x7f42441a6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.019+0000 7f4242d9d640 1 -- 172.21.15.70:0/2263406055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4244108750 msgr2=0x7f42441a68e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:41:01.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.019+0000 7f4242d9d640 1 --2- 172.21.15.70:0/2263406055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4244108750 0x7f42441a68e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.019+0000 7f4242d9d640 1 -- 172.21.15.70:0/2263406055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42441090b0 msgr2=0x7f42441a6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.019+0000 7f4242d9d640 1 --2- 172.21.15.70:0/2263406055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42441090b0 0x7f42441a6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.019+0000 7f4242d9d640 1 -- 172.21.15.70:0/2263406055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4244117500 con 0x7f424410ffd0 2024-04-01T01:41:01.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.019+0000 7f4241d9b640 1 --2- 172.21.15.70:0/2263406055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42441090b0 0x7f42441a6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:41:01.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.020+0000 7f4242d9d640 1 --2- 172.21.15.70:0/2263406055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f424410ffd0 0x7f42441ade40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4234009850 tx=0x7f4234004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.020+0000 7f42337fe640 1 -- 172.21.15.70:0/2263406055 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4234043020 con 0x7f424410ffd0 2024-04-01T01:41:01.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.020+0000 7f42337fe640 1 -- 172.21.15.70:0/2263406055 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f42340052b0 con 0x7f424410ffd0 2024-04-01T01:41:01.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.020+0000 7f4248fd6640 1 -- 172.21.15.70:0/2263406055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4244117790 con 0x7f424410ffd0 2024-04-01T01:41:01.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.020+0000 7f4248fd6640 1 -- 172.21.15.70:0/2263406055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4244117d60 con 0x7f424410ffd0 2024-04-01T01:41:01.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.021+0000 7f42337fe640 1 -- 172.21.15.70:0/2263406055 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4234003c90 con 0x7f424410ffd0 2024-04-01T01:41:01.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.024+0000 7f424bbb0740 1 Processor -- start 2024-04-01T01:41:01.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.026+0000 7f42337fe640 1 -- 172.21.15.70:0/2263406055 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f4234046020 con 0x7f424410ffd0 2024-04-01T01:41:01.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.027+0000 7f424bbb0740 1 -- start start 2024-04-01T01:41:01.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.027+0000 7f424bbb0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6650490 0x55bef66508b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.027+0000 7f424bbb0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650e80 0x55bef6655980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.027+0000 7f424bbb0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6655ff0 0x55bef6658480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.027+0000 7f424bbb0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bef65b51b0 con 0x55bef6650490 2024-04-01T01:41:01.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.027+0000 7f424bbb0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55bef655dc00 con 0x55bef6655ff0 2024-04-01T01:41:01.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.027+0000 7f424bbb0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bef65b1250 con 0x55bef6650e80 2024-04-01T01:41:01.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.027+0000 7f424259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650e80 0x55bef6655980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.027+0000 7f424259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650e80 0x55bef6655980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43006/0 (socket says 172.21.15.70:43006) 2024-04-01T01:41:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.027+0000 7f424259c640 1 -- 172.21.15.70:0/1043462054 learned_addr learned my addr 172.21.15.70:0/1043462054 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.027+0000 7f4241d9b640 1 --2- 172.21.15.70:0/1043462054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6655ff0 0x55bef6658480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.027+0000 7f4241d9b640 1 --2- 172.21.15.70:0/1043462054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6650490 0x55bef66508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.028+0000 7f424259c640 1 -- 172.21.15.70:0/1043462054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6655ff0 msgr2=0x55bef6658480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.028+0000 7f424259c640 1 --2- 172.21.15.70:0/1043462054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6655ff0 0x55bef6658480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.028+0000 7f424259c640 1 -- 172.21.15.70:0/1043462054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6650490 msgr2=0x55bef66508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.028+0000 7f42337fe640 1 --2- 172.21.15.70:0/2263406055 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f422c0a24f0 0x7f422c0a49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.028+0000 7f424259c640 1 --2- 172.21.15.70:0/1043462054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6650490 0x55bef66508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.028+0000 7f424259c640 1 -- 172.21.15.70:0/1043462054 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bef657d3d0 con 0x55bef6650e80 2024-04-01T01:41:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.028+0000 7f4241d9b640 1 --2- 172.21.15.70:0/1043462054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6650490 0x55bef66508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:41:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.028+0000 7f42337fe640 1 -- 172.21.15.70:0/2263406055 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f42340f0fc0 con 0x7f424410ffd0 2024-04-01T01:41:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.028+0000 7f4242d9d640 1 --2- 172.21.15.70:0/2263406055 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f422c0a24f0 0x7f422c0a49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.028+0000 7f424259c640 1 --2- 172.21.15.70:0/1043462054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650e80 0x55bef6655980 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4238009780 tx=0x7f4238009c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1571d9df7566b0c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.029+0000 7f4242d9d640 1 --2- 172.21.15.70:0/2263406055 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f422c0a24f0 0x7f422c0a49b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f42340ba1a0 tx=0x7f423402bc80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.030+0000 7f4230ff9640 1 -- 172.21.15.70:0/1043462054 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238015020 con 0x55bef6650e80 2024-04-01T01:41:01.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.030+0000 7f4230ff9640 1 -- 172.21.15.70:0/1043462054 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4238011420 con 0x55bef6650e80 2024-04-01T01:41:01.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.031+0000 7f424bbb0740 1 -- 172.21.15.70:0/1043462054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650e80 msgr2=0x55bef6655980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.031+0000 7f424bbb0740 1 --2- 172.21.15.70:0/1043462054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650e80 0x55bef6655980 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4238009780 tx=0x7f4238009c40 comp rx=0 tx=0).stop 2024-04-01T01:41:01.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.031+0000 7f424bbb0740 1 -- 172.21.15.70:0/1043462054 shutdown_connections 2024-04-01T01:41:01.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.031+0000 7f424bbb0740 1 --2- 172.21.15.70:0/1043462054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6655ff0 0x55bef6658480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.031+0000 7f424bbb0740 1 --2- 172.21.15.70:0/1043462054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650e80 0x55bef6655980 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.031+0000 7f424bbb0740 1 --2- 172.21.15.70:0/1043462054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6650490 0x55bef66508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.031+0000 7f424bbb0740 1 -- 172.21.15.70:0/1043462054 >> 172.21.15.70:0/1043462054 conn(0x55bef664ba00 msgr2=0x55bef664be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.033+0000 7f424bbb0740 1 -- 172.21.15.70:0/1043462054 shutdown_connections 2024-04-01T01:41:01.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.033+0000 7f424bbb0740 1 -- 172.21.15.70:0/1043462054 wait complete. 2024-04-01T01:41:01.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.033+0000 7f424bbb0740 1 -- start start 2024-04-01T01:41:01.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.034+0000 7f424bbb0740 1 Processor -- start 2024-04-01T01:41:01.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.034+0000 7f424bbb0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 0x55bef665a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.034+0000 7f424bbb0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6650e80 0x55bef665acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.034+0000 7f424bbb0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6655ff0 0x55bef6661d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.034+0000 7f424bbb0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bef6658ff0 con 0x55bef6655ff0 2024-04-01T01:41:01.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.034+0000 7f424bbb0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55bef6553470 con 0x55bef6650e80 2024-04-01T01:41:01.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.034+0000 7f424bbb0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bef6659170 con 0x55bef6650490 2024-04-01T01:41:01.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.034+0000 7f424259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6650e80 0x55bef665acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.034+0000 7f4241d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 0x55bef665a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.034+0000 7f4241d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 0x55bef665a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43016/0 (socket says 172.21.15.70:43016) 2024-04-01T01:41:01.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.034+0000 7f424259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6650e80 0x55bef665acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57216/0 (socket says 172.21.15.70:57216) 2024-04-01T01:41:01.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.034+0000 7f4241d9b640 1 -- 172.21.15.70:0/1992937105 learned_addr learned my addr 172.21.15.70:0/1992937105 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.034+0000 7f4241d9b640 1 --2- 172.21.15.70:0/1992937105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6655ff0 0x55bef6661d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.034+0000 7f424bbb0740 10 client.0 _ll_register_callbacks cb 0x55bef6570cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.035+0000 7f424259c640 1 -- 172.21.15.70:0/1992937105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 msgr2=0x55bef665a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.035+0000 7f424259c640 1 --2- 172.21.15.70:0/1992937105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 0x55bef665a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.035+0000 7f424259c640 1 -- 172.21.15.70:0/1992937105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6655ff0 msgr2=0x55bef6661d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.035+0000 7f424259c640 1 --2- 172.21.15.70:0/1992937105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6655ff0 0x55bef6661d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.035+0000 7f424259c640 1 -- 172.21.15.70:0/1992937105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bef62c93f0 con 0x55bef6650e80 2024-04-01T01:41:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.035+0000 7f424259c640 1 --2- 172.21.15.70:0/1992937105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6650e80 0x55bef665acc0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x55bef664c970 tx=0x7f4238011800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=146d04a5b0e4a097 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.035+0000 7f4229ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.035+0000 7f4229ffb640 1 -- 172.21.15.70:0/1992937105 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4238015030 con 0x55bef6650e80 2024-04-01T01:41:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.036+0000 7f4229ffb640 1 -- 172.21.15.70:0/1992937105 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f423800bce0 con 0x55bef6650e80 2024-04-01T01:41:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.036+0000 7f424bbb0740 1 -- 172.21.15.70:0/1992937105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4238011420 con 0x55bef6650e80 2024-04-01T01:41:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.036+0000 7f4229ffb640 1 -- 172.21.15.70:0/1992937105 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4238020070 con 0x55bef6650e80 2024-04-01T01:41:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.036+0000 7f424bbb0740 10 client.27674 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:01.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.036+0000 7f424bbb0740 1 -- 172.21.15.70:0/1992937105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55bef63937d0 con 0x55bef6650e80 2024-04-01T01:41:01.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.037+0000 7f4229ffb640 1 -- 172.21.15.70:0/1992937105 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4238010c90 con 0x55bef6650e80 2024-04-01T01:41:01.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.037+0000 7f4229ffb640 1 -- 172.21.15.70:0/1992937105 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f423801a860 con 0x55bef6650e80 2024-04-01T01:41:01.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.037+0000 7f424bbb0740 10 client.27674 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:01.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.037+0000 7f424bbb0740 10 client.27674 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:41:01.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.037+0000 7f424bbb0740 20 client.27674 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:01.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.037+0000 7f424bbb0740 1 --2- 172.21.15.70:0/1992937105 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55bef6665f00 0x55bef66682f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.037+0000 7f424bbb0740 4 client.27674 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:01.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.037+0000 7f424bbb0740 1 -- 172.21.15.70:0/1992937105 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4238011420 con 0x55bef6665f00 2024-04-01T01:41:01.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.037+0000 7f4241d9b640 1 --2- 172.21.15.70:0/1992937105 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55bef6665f00 0x55bef66682f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.038+0000 7f4241d9b640 1 --2- 172.21.15.70:0/1992937105 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55bef6665f00 0x55bef66682f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=980c72a75247426d server_cookie=5ab502bad8b9deb2 in_seq=0 out_seq=0 2024-04-01T01:41:01.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.039+0000 7f4229ffb640 10 client.27674 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:41:01.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.039+0000 7f4229ffb640 1 -- 172.21.15.70:0/1992937105 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f4238011420 con 0x55bef6665f00 2024-04-01T01:41:01.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.039+0000 7f4229ffb640 10 client.27674 handle_command_reply: tid=0 2024-04-01T01:41:01.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.039+0000 7f424bbb0740 1 client.27674 shutdown 2024-04-01T01:41:01.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.040+0000 7f424bbb0740 1 -- 172.21.15.70:0/1992937105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6650e80 msgr2=0x55bef665acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.040+0000 7f424bbb0740 1 --2- 172.21.15.70:0/1992937105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6650e80 0x55bef665acc0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x55bef664c970 tx=0x7f4238011800 comp rx=0 tx=0).stop 2024-04-01T01:41:01.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.040+0000 7f424bbb0740 1 -- 172.21.15.70:0/1992937105 shutdown_connections 2024-04-01T01:41:01.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.040+0000 7f424bbb0740 1 --2- 172.21.15.70:0/1992937105 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55bef6665f00 0x55bef66682f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.040+0000 7f4229ffb640 0 client.27674 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:41:01.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.040+0000 7f424bbb0740 1 --2- 172.21.15.70:0/1992937105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6655ff0 0x55bef6661d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.040+0000 7f424bbb0740 1 --2- 172.21.15.70:0/1992937105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6650e80 0x55bef665acc0 secure :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0x55bef664c970 tx=0x7f4238011800 comp rx=0 tx=0).stop 2024-04-01T01:41:01.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.040+0000 7f424bbb0740 1 --2- 172.21.15.70:0/1992937105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 0x55bef665a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.040+0000 7f424bbb0740 1 -- 172.21.15.70:0/1992937105 >> 172.21.15.70:0/1992937105 conn(0x55bef664ba00 msgr2=0x55bef664ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.040+0000 7f424bbb0740 1 -- 172.21.15.70:0/1992937105 shutdown_connections 2024-04-01T01:41:01.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.040+0000 7f424bbb0740 1 -- 172.21.15.70:0/1992937105 wait complete. 2024-04-01T01:41:01.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.040+0000 7f424bbb0740 20 client.27674 trim_cache size 0 max 16384 2024-04-01T01:41:01.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.067+0000 7f424bbb0740 1 Processor -- start 2024-04-01T01:41:01.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.067+0000 7f424bbb0740 1 -- start start 2024-04-01T01:41:01.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.068+0000 7f424bbb0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 0x55bef664a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.068+0000 7f424bbb0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6655ff0 0x55bef6661cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.068+0000 7f424bbb0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6647030 0x55bef664a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.068+0000 7f424bbb0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f42441c1c90 con 0x55bef6655ff0 2024-04-01T01:41:01.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.068+0000 7f424bbb0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f42441c37d0 con 0x55bef6647030 2024-04-01T01:41:01.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.068+0000 7f424bbb0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f42441c2360 con 0x55bef6650490 2024-04-01T01:41:01.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.068+0000 7f4241d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6647030 0x55bef664a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.068+0000 7f4241d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 0x55bef664a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.068+0000 7f4241d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 0x55bef664a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43030/0 (socket says 172.21.15.70:43030) 2024-04-01T01:41:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.068+0000 7f4241d9b640 1 -- 172.21.15.70:0/3708367670 learned_addr learned my addr 172.21.15.70:0/3708367670 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.068+0000 7f424259c640 1 --2- 172.21.15.70:0/3708367670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6655ff0 0x55bef6661cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.069+0000 7f4241d9b640 1 -- 172.21.15.70:0/3708367670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6647030 msgr2=0x55bef664a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.069+0000 7f4241d9b640 1 --2- 172.21.15.70:0/3708367670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6647030 0x55bef664a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.069+0000 7f4241d9b640 1 -- 172.21.15.70:0/3708367670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6655ff0 msgr2=0x55bef6661cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.069+0000 7f4241d9b640 1 --2- 172.21.15.70:0/3708367670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6655ff0 0x55bef6661cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.069+0000 7f4241d9b640 1 -- 172.21.15.70:0/3708367670 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bef6651260 con 0x55bef6650490 2024-04-01T01:41:01.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.069+0000 7f4241d9b640 1 --2- 172.21.15.70:0/3708367670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 0x55bef664a4b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x55bef6668b60 tx=0x7f423c008220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb7b38cd3fb367ff server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.070+0000 7f422bfff640 1 -- 172.21.15.70:0/3708367670 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c01e080 con 0x55bef6650490 2024-04-01T01:41:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.070+0000 7f422bfff640 1 -- 172.21.15.70:0/3708367670 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f423c01e220 con 0x55bef6650490 2024-04-01T01:41:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.070+0000 7f422bfff640 1 -- 172.21.15.70:0/3708367670 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c020760 con 0x55bef6650490 2024-04-01T01:41:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.071+0000 7f424bbb0740 1 -- 172.21.15.70:0/3708367670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 msgr2=0x55bef664a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.071+0000 7f424bbb0740 1 --2- 172.21.15.70:0/3708367670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 0x55bef664a4b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x55bef6668b60 tx=0x7f423c008220 comp rx=0 tx=0).stop 2024-04-01T01:41:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.071+0000 7f424bbb0740 1 -- 172.21.15.70:0/3708367670 shutdown_connections 2024-04-01T01:41:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.071+0000 7f424bbb0740 1 --2- 172.21.15.70:0/3708367670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6647030 0x55bef664a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.071+0000 7f424bbb0740 1 --2- 172.21.15.70:0/3708367670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6655ff0 0x55bef6661cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.071+0000 7f424bbb0740 1 --2- 172.21.15.70:0/3708367670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 0x55bef664a4b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.071+0000 7f424bbb0740 1 -- 172.21.15.70:0/3708367670 >> 172.21.15.70:0/3708367670 conn(0x55bef664ba00 msgr2=0x55bef66520c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.072+0000 7f424bbb0740 1 -- 172.21.15.70:0/3708367670 shutdown_connections 2024-04-01T01:41:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.072+0000 7f424bbb0740 1 -- 172.21.15.70:0/3708367670 wait complete. 2024-04-01T01:41:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.072+0000 7f424bbb0740 1 -- start start 2024-04-01T01:41:01.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.073+0000 7f424bbb0740 1 Processor -- start 2024-04-01T01:41:01.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.073+0000 7f424bbb0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6647030 0x55bef665fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.073+0000 7f424bbb0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 0x55bef6660090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.073+0000 7f424bbb0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6655ff0 0x55bef66610a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.073+0000 7f424bbb0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f42441c3100 con 0x55bef6647030 2024-04-01T01:41:01.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.073+0000 7f424bbb0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f42441c2a30 con 0x55bef6655ff0 2024-04-01T01:41:01.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.073+0000 7f424bbb0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f42441bfca0 con 0x55bef6650490 2024-04-01T01:41:01.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.073+0000 7f424bbb0740 10 client.0 _ll_register_callbacks cb 0x7f4238011cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:01.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.073+0000 7f424259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 0x55bef6660090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.073+0000 7f4241d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6647030 0x55bef665fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.073+0000 7f4241d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6647030 0x55bef665fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38478/0 (socket says 172.21.15.70:38478) 2024-04-01T01:41:01.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.073+0000 7f4241d9b640 1 -- 172.21.15.70:0/1758268139 learned_addr learned my addr 172.21.15.70:0/1758268139 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.074+0000 7f4241d9b640 1 --2- 172.21.15.70:0/1758268139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6655ff0 0x55bef66610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.075+0000 7f4241d9b640 1 -- 172.21.15.70:0/1758268139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 msgr2=0x55bef6660090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.075+0000 7f4241d9b640 1 --2- 172.21.15.70:0/1758268139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 0x55bef6660090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.075+0000 7f4241d9b640 1 -- 172.21.15.70:0/1758268139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6655ff0 msgr2=0x55bef66610a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.075+0000 7f4241d9b640 1 --2- 172.21.15.70:0/1758268139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6655ff0 0x55bef66610a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.075+0000 7f4241d9b640 1 -- 172.21.15.70:0/1758268139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bef6661690 con 0x55bef6647030 2024-04-01T01:41:01.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.075+0000 7f4241d9b640 1 --2- 172.21.15.70:0/1758268139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6647030 0x55bef665fb50 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f423c00a010 tx=0x7f423c009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4501d069e02d8860 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.075+0000 7f4229ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:41:01.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.076+0000 7f4229ffb640 1 -- 172.21.15.70:0/1758268139 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c033060 con 0x55bef6647030 2024-04-01T01:41:01.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.076+0000 7f4229ffb640 1 -- 172.21.15.70:0/1758268139 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f423c01e8b0 con 0x55bef6647030 2024-04-01T01:41:01.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.076+0000 7f424bbb0740 1 -- 172.21.15.70:0/1758268139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f423c01e220 con 0x55bef6647030 2024-04-01T01:41:01.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.076+0000 7f4229ffb640 1 -- 172.21.15.70:0/1758268139 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f423c01eab0 con 0x55bef6647030 2024-04-01T01:41:01.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.077+0000 7f424bbb0740 10 client.17745 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:01.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.077+0000 7f424bbb0740 1 -- 172.21.15.70:0/1758268139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55bef66cc930 con 0x55bef6647030 2024-04-01T01:41:01.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.079+0000 7f4229ffb640 1 -- 172.21.15.70:0/1758268139 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f423c01e220 con 0x55bef6647030 2024-04-01T01:41:01.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.079+0000 7f4229ffb640 1 -- 172.21.15.70:0/1758268139 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f423c02dc10 con 0x55bef6647030 2024-04-01T01:41:01.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.079+0000 7f424bbb0740 10 client.17745 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:01.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.079+0000 7f424bbb0740 10 client.17745 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:41:01.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.079+0000 7f424bbb0740 20 client.17745 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:01.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.079+0000 7f424bbb0740 1 --2- 172.21.15.70:0/1758268139 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55bef66cd1b0 0x55bef66cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.079+0000 7f424bbb0740 4 client.17745 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:41:01.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.079+0000 7f424bbb0740 1 -- 172.21.15.70:0/1758268139 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bef65c26f0 con 0x55bef66cd1b0 2024-04-01T01:41:01.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.080+0000 7f424259c640 1 --2- 172.21.15.70:0/1758268139 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55bef66cd1b0 0x55bef66cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.082+0000 7f424259c640 1 --2- 172.21.15.70:0/1758268139 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55bef66cd1b0 0x55bef66cf5d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c1e86c49fc9d062 server_cookie=5aeb398726705b87 in_seq=0 out_seq=0 2024-04-01T01:41:01.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.082+0000 7f4229ffb640 10 client.17745 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:41:01.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.082+0000 7f4229ffb640 1 -- 172.21.15.70:0/1758268139 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bef65c26f0 con 0x55bef66cd1b0 2024-04-01T01:41:01.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.083+0000 7f4229ffb640 10 client.17745 handle_command_reply: tid=0 2024-04-01T01:41:01.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.083+0000 7f424bbb0740 1 client.17745 shutdown 2024-04-01T01:41:01.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.083+0000 7f424bbb0740 1 -- 172.21.15.70:0/1758268139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6647030 msgr2=0x55bef665fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.083+0000 7f424bbb0740 1 --2- 172.21.15.70:0/1758268139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6647030 0x55bef665fb50 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f423c00a010 tx=0x7f423c009570 comp rx=0 tx=0).stop 2024-04-01T01:41:01.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.084+0000 7f424bbb0740 1 -- 172.21.15.70:0/1758268139 shutdown_connections 2024-04-01T01:41:01.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.084+0000 7f424bbb0740 1 --2- 172.21.15.70:0/1758268139 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55bef66cd1b0 0x55bef66cf5d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.084+0000 7f424bbb0740 1 --2- 172.21.15.70:0/1758268139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bef6655ff0 0x55bef66610a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.084+0000 7f424bbb0740 1 --2- 172.21.15.70:0/1758268139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bef6650490 0x55bef6660090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.084+0000 7f424bbb0740 1 --2- 172.21.15.70:0/1758268139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bef6647030 0x55bef665fb50 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.084+0000 7f424bbb0740 1 -- 172.21.15.70:0/1758268139 >> 172.21.15.70:0/1758268139 conn(0x55bef664ba00 msgr2=0x55bef664a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.084+0000 7f4229ffb640 0 client.17745 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:41:01.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.084+0000 7f424bbb0740 1 -- 172.21.15.70:0/1758268139 shutdown_connections 2024-04-01T01:41:01.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.084+0000 7f424bbb0740 1 -- 172.21.15.70:0/1758268139 wait complete. 2024-04-01T01:41:01.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.084+0000 7f424bbb0740 20 client.17745 trim_cache size 0 max 16384 2024-04-01T01:41:01.085 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:41:01.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.088+0000 7f4248fd6640 1 -- 172.21.15.70:0/2263406055 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f422c0a24f0 msgr2=0x7f422c0a49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.088+0000 7f4248fd6640 1 --2- 172.21.15.70:0/2263406055 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f422c0a24f0 0x7f422c0a49b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f42340ba1a0 tx=0x7f423402bc80 comp rx=0 tx=0).stop 2024-04-01T01:41:01.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.088+0000 7f4248fd6640 1 -- 172.21.15.70:0/2263406055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f424410ffd0 msgr2=0x7f42441ade40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.088+0000 7f4248fd6640 1 --2- 172.21.15.70:0/2263406055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f424410ffd0 0x7f42441ade40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4234009850 tx=0x7f4234004270 comp rx=0 tx=0).stop 2024-04-01T01:41:01.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.088+0000 7f4248fd6640 1 -- 172.21.15.70:0/2263406055 shutdown_connections 2024-04-01T01:41:01.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.089+0000 7f4248fd6640 1 --2- 172.21.15.70:0/2263406055 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f422c0a24f0 0x7f422c0a49b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.089+0000 7f4248fd6640 1 --2- 172.21.15.70:0/2263406055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f424410ffd0 0x7f42441ade40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.089+0000 7f4248fd6640 1 --2- 172.21.15.70:0/2263406055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42441090b0 0x7f42441a6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.089+0000 7f4248fd6640 1 --2- 172.21.15.70:0/2263406055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4244108750 0x7f42441a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.089+0000 7f4248fd6640 1 -- 172.21.15.70:0/2263406055 >> 172.21.15.70:0/2263406055 conn(0x7f424406e950 msgr2=0x7f4244114350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.089+0000 7f4248fd6640 1 -- 172.21.15.70:0/2263406055 shutdown_connections 2024-04-01T01:41:01.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.089+0000 7f4248fd6640 1 -- 172.21.15.70:0/2263406055 wait complete. 2024-04-01T01:41:01.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:00 smithi070 ceph-mon[32179]: pgmap v585: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 10 MiB/s wr, 900 op/s 2024-04-01T01:41:01.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:00 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/928440316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:41:01.110 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:41:01.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:00 smithi189 ceph-mon[35753]: pgmap v585: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 10 MiB/s wr, 900 op/s 2024-04-01T01:41:01.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:00 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/928440316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:41:01.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.302+0000 7ff766c07640 1 -- 172.21.15.70:0/1050084667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7580a5010 msgr2=0x7ff7580a5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.302+0000 7ff766c07640 1 --2- 172.21.15.70:0/1050084667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7580a5010 0x7ff7580a5430 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff760067420 tx=0x7ff7600911a0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.302+0000 7ff766c07640 1 -- 172.21.15.70:0/1050084667 shutdown_connections 2024-04-01T01:41:01.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.302+0000 7ff766c07640 1 --2- 172.21.15.70:0/1050084667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7580a6bc0 0x7ff7580ad450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.302+0000 7ff766c07640 1 --2- 172.21.15.70:0/1050084667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7580a6200 0x7ff7580a6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.302+0000 7ff766c07640 1 --2- 172.21.15.70:0/1050084667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7580a5010 0x7ff7580a5430 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.302+0000 7ff766c07640 1 -- 172.21.15.70:0/1050084667 >> 172.21.15.70:0/1050084667 conn(0x7ff7580a0560 msgr2=0x7ff7580a29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.303+0000 7ff766c07640 1 -- 172.21.15.70:0/1050084667 shutdown_connections 2024-04-01T01:41:01.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.303+0000 7ff766c07640 1 -- 172.21.15.70:0/1050084667 wait complete. 2024-04-01T01:41:01.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.304+0000 7ff766c07640 1 Processor -- start 2024-04-01T01:41:01.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.304+0000 7ff766c07640 1 -- start start 2024-04-01T01:41:01.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.304+0000 7ff766c07640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7580a5010 0x7ff7581435e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.304+0000 7ff766c07640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7580a6200 0x7ff758143b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.304+0000 7ff766c07640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7580a6bc0 0x7ff75814ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.304+0000 7ff766c07640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff7580afc90 con 0x7ff7580a6bc0 2024-04-01T01:41:01.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.304+0000 7ff766c07640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff7580afb10 con 0x7ff7580a5010 2024-04-01T01:41:01.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.304+0000 7ff766c07640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff7580afe10 con 0x7ff7580a6200 2024-04-01T01:41:01.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.304+0000 7ff7649a3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7580a5010 0x7ff7581435e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.304+0000 7ff7649a3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7580a5010 0x7ff7581435e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57266/0 (socket says 172.21.15.70:57266) 2024-04-01T01:41:01.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.305+0000 7ff7649a3640 1 -- 172.21.15.70:0/1255302885 learned_addr learned my addr 172.21.15.70:0/1255302885 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.305+0000 7ff75ffff640 1 --2- 172.21.15.70:0/1255302885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7580a6200 0x7ff758143b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.305+0000 7ff7649a3640 1 -- 172.21.15.70:0/1255302885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7580a6200 msgr2=0x7ff758143b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.305+0000 7ff7649a3640 1 --2- 172.21.15.70:0/1255302885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7580a6200 0x7ff758143b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.305+0000 7ff7649a3640 1 -- 172.21.15.70:0/1255302885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7580a6bc0 msgr2=0x7ff75814ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.305+0000 7ff7649a3640 1 --2- 172.21.15.70:0/1255302885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7580a6bc0 0x7ff75814ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.305+0000 7ff7649a3640 1 -- 172.21.15.70:0/1255302885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75814b200 con 0x7ff7580a5010 2024-04-01T01:41:01.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.306+0000 7ff7649a3640 1 --2- 172.21.15.70:0/1255302885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7580a5010 0x7ff7581435e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff760050720 tx=0x7ff76006d7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.306+0000 7ff75dffb640 1 -- 172.21.15.70:0/1255302885 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76006dd40 con 0x7ff7580a5010 2024-04-01T01:41:01.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.306+0000 7ff766c07640 1 -- 172.21.15.70:0/1255302885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff75814b490 con 0x7ff7580a5010 2024-04-01T01:41:01.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.307+0000 7ff766c07640 1 -- 172.21.15.70:0/1255302885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff75814b950 con 0x7ff7580a5010 2024-04-01T01:41:01.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.307+0000 7ff75dffb640 1 -- 172.21.15.70:0/1255302885 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff760069500 con 0x7ff7580a5010 2024-04-01T01:41:01.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.307+0000 7ff75dffb640 1 -- 172.21.15.70:0/1255302885 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76009ab10 con 0x7ff7580a5010 2024-04-01T01:41:01.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.310+0000 7ff75dffb640 1 -- 172.21.15.70:0/1255302885 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7ff76009acb0 con 0x7ff7580a5010 2024-04-01T01:41:01.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.311+0000 7ff7697eb740 1 Processor -- start 2024-04-01T01:41:01.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.311+0000 7ff7697eb740 1 -- start start 2024-04-01T01:41:01.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.311+0000 7ff7697eb740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a59516a0 0x55c7a5951ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.311+0000 7ff7697eb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a5952090 0x55c7a5956b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.311+0000 7ff7697eb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5957200 0x55c7a5959690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.311+0000 7ff7697eb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c7a58b61b0 con 0x55c7a5952090 2024-04-01T01:41:01.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.311+0000 7ff7697eb740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c7a585ec00 con 0x55c7a59516a0 2024-04-01T01:41:01.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.311+0000 7ff7697eb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c7a58b2250 con 0x55c7a5957200 2024-04-01T01:41:01.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.312+0000 7ff75ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5957200 0x55c7a5959690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.312+0000 7ff75ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5957200 0x55c7a5959690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43064/0 (socket says 172.21.15.70:43064) 2024-04-01T01:41:01.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.312+0000 7ff75ffff640 1 -- 172.21.15.70:0/2215939697 learned_addr learned my addr 172.21.15.70:0/2215939697 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.312+0000 7ff7651a4640 1 --2- 172.21.15.70:0/2215939697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a59516a0 0x55c7a5951ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.312+0000 7ff7651a4640 1 -- 172.21.15.70:0/2215939697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5957200 msgr2=0x55c7a5959690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.312+0000 7ff7651a4640 1 --2- 172.21.15.70:0/2215939697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5957200 0x55c7a5959690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.312+0000 7ff7651a4640 1 -- 172.21.15.70:0/2215939697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a5952090 msgr2=0x55c7a5956b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.312+0000 7ff7651a4640 1 --2- 172.21.15.70:0/2215939697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a5952090 0x55c7a5956b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.312+0000 7ff7651a4640 1 -- 172.21.15.70:0/2215939697 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7a587e3d0 con 0x55c7a59516a0 2024-04-01T01:41:01.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.312+0000 7ff7651a4640 1 --2- 172.21.15.70:0/2215939697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a59516a0 0x55c7a5951ac0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009b40 tx=0x7ff74c00e350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c8e4b48b1a88d75 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.313+0000 7ff75dffb640 1 --2- 172.21.15.70:0/1255302885 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7500a2620 0x7ff7500a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.313+0000 7ff74affd640 1 -- 172.21.15.70:0/2215939697 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c00ed00 con 0x55c7a59516a0 2024-04-01T01:41:01.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.313+0000 7ff74affd640 1 -- 172.21.15.70:0/2215939697 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff74c005020 con 0x55c7a59516a0 2024-04-01T01:41:01.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.313+0000 7ff7649a3640 1 --2- 172.21.15.70:0/1255302885 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7500a2620 0x7ff7500a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.313+0000 7ff75dffb640 1 -- 172.21.15.70:0/1255302885 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff760154ba0 con 0x7ff7580a5010 2024-04-01T01:41:01.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.314+0000 7ff74affd640 1 -- 172.21.15.70:0/2215939697 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c004a20 con 0x55c7a59516a0 2024-04-01T01:41:01.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.314+0000 7ff7697eb740 1 -- 172.21.15.70:0/2215939697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a59516a0 msgr2=0x55c7a5951ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.314+0000 7ff7697eb740 1 --2- 172.21.15.70:0/2215939697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a59516a0 0x55c7a5951ac0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009b40 tx=0x7ff74c00e350 comp rx=0 tx=0).stop 2024-04-01T01:41:01.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.314+0000 7ff7649a3640 1 --2- 172.21.15.70:0/1255302885 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7500a2620 0x7ff7500a4ae0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff760067290 tx=0x7ff760141040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.314+0000 7ff7697eb740 1 -- 172.21.15.70:0/2215939697 shutdown_connections 2024-04-01T01:41:01.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.314+0000 7ff7697eb740 1 --2- 172.21.15.70:0/2215939697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5957200 0x55c7a5959690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.314+0000 7ff7697eb740 1 --2- 172.21.15.70:0/2215939697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a5952090 0x55c7a5956b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.314+0000 7ff7697eb740 1 --2- 172.21.15.70:0/2215939697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a59516a0 0x55c7a5951ac0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.314+0000 7ff7697eb740 1 -- 172.21.15.70:0/2215939697 >> 172.21.15.70:0/2215939697 conn(0x55c7a594cc10 msgr2=0x55c7a594d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.314+0000 7ff7697eb740 1 -- 172.21.15.70:0/2215939697 shutdown_connections 2024-04-01T01:41:01.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.315+0000 7ff7697eb740 1 -- 172.21.15.70:0/2215939697 wait complete. 2024-04-01T01:41:01.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.315+0000 7ff7697eb740 1 -- start start 2024-04-01T01:41:01.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.315+0000 7ff7697eb740 1 Processor -- start 2024-04-01T01:41:01.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.316+0000 7ff7697eb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a59516a0 0x55c7a595bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.316+0000 7ff7697eb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5952090 0x55c7a595c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.316+0000 7ff7697eb740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a5957200 0x55c7a5963190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.316+0000 7ff7697eb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c7a595a200 con 0x55c7a59516a0 2024-04-01T01:41:01.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.316+0000 7ff7697eb740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c7a5854470 con 0x55c7a5957200 2024-04-01T01:41:01.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.316+0000 7ff7697eb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c7a595a380 con 0x55c7a5952090 2024-04-01T01:41:01.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.316+0000 7ff7697eb740 10 client.0 _ll_register_callbacks cb 0x55c7a5871cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:01.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.316+0000 7ff7651a4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a5957200 0x55c7a5963190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.316+0000 7ff7651a4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a5957200 0x55c7a5963190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57296/0 (socket says 172.21.15.70:57296) 2024-04-01T01:41:01.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.316+0000 7ff7651a4640 1 -- 172.21.15.70:0/3064914542 learned_addr learned my addr 172.21.15.70:0/3064914542 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.317+0000 7ff75ffff640 1 --2- 172.21.15.70:0/3064914542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5952090 0x55c7a595c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.317+0000 7ff7651a4640 1 --2- 172.21.15.70:0/3064914542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a59516a0 0x55c7a595bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.317+0000 7ff75ffff640 1 -- 172.21.15.70:0/3064914542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a5957200 msgr2=0x55c7a5963190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.317+0000 7ff75ffff640 1 --2- 172.21.15.70:0/3064914542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a5957200 0x55c7a5963190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.317+0000 7ff75ffff640 1 -- 172.21.15.70:0/3064914542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a59516a0 msgr2=0x55c7a595bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.317+0000 7ff75ffff640 1 --2- 172.21.15.70:0/3064914542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a59516a0 0x55c7a595bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.317+0000 7ff75ffff640 1 -- 172.21.15.70:0/3064914542 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7a55ca3f0 con 0x55c7a5952090 2024-04-01T01:41:01.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.317+0000 7ff7651a4640 1 --2- 172.21.15.70:0/3064914542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a5957200 0x55c7a5963190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:41:01.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.317+0000 7ff75ffff640 1 --2- 172.21.15.70:0/3064914542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5952090 0x55c7a595c150 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x55c7a594db80 tx=0x7ff754014d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0ce51a88f6f8658 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.317+0000 7ff73ffff640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:41:01.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.318+0000 7ff73ffff640 1 -- 172.21.15.70:0/3064914542 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff754012a20 con 0x55c7a5952090 2024-04-01T01:41:01.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.318+0000 7ff73ffff640 1 -- 172.21.15.70:0/3064914542 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff754012bc0 con 0x55c7a5952090 2024-04-01T01:41:01.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.318+0000 7ff7697eb740 1 -- 172.21.15.70:0/3064914542 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff74c005020 con 0x55c7a5952090 2024-04-01T01:41:01.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.318+0000 7ff73ffff640 1 -- 172.21.15.70:0/3064914542 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff754012dc0 con 0x55c7a5952090 2024-04-01T01:41:01.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.318+0000 7ff7697eb740 10 client.27541 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:01.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.318+0000 7ff7697eb740 1 -- 172.21.15.70:0/3064914542 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c7a56947d0 con 0x55c7a5952090 2024-04-01T01:41:01.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.319+0000 7ff73ffff640 1 -- 172.21.15.70:0/3064914542 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff754005020 con 0x55c7a5952090 2024-04-01T01:41:01.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.319+0000 7ff73ffff640 1 -- 172.21.15.70:0/3064914542 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff7540049a0 con 0x55c7a5952090 2024-04-01T01:41:01.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.319+0000 7ff7697eb740 10 client.27541 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:01.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.320+0000 7ff7697eb740 10 client.27541 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:41:01.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.320+0000 7ff7697eb740 20 client.27541 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:01.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.320+0000 7ff7697eb740 1 --2- 172.21.15.70:0/3064914542 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55c7a5967c00 0x55c7a596a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.320+0000 7ff7697eb740 4 client.27541 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:01.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.320+0000 7ff7697eb740 1 -- 172.21.15.70:0/3064914542 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff74c005020 con 0x55c7a5967c00 2024-04-01T01:41:01.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.322+0000 7ff7651a4640 1 --2- 172.21.15.70:0/3064914542 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55c7a5967c00 0x55c7a596a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.324+0000 7ff7651a4640 1 --2- 172.21.15.70:0/3064914542 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55c7a5967c00 0x55c7a596a040 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6f352ce65b5aa872 server_cookie=d66f9c40193b56f2 in_seq=0 out_seq=0 2024-04-01T01:41:01.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.324+0000 7ff73ffff640 10 client.27541 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:41:01.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.325+0000 7ff73ffff640 1 -- 172.21.15.70:0/3064914542 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7ff74c005020 con 0x55c7a5967c00 2024-04-01T01:41:01.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.325+0000 7ff73ffff640 10 client.27541 handle_command_reply: tid=0 2024-04-01T01:41:01.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.325+0000 7ff7697eb740 1 client.27541 shutdown 2024-04-01T01:41:01.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.326+0000 7ff7697eb740 1 -- 172.21.15.70:0/3064914542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5952090 msgr2=0x55c7a595c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.326+0000 7ff7697eb740 1 --2- 172.21.15.70:0/3064914542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5952090 0x55c7a595c150 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x55c7a594db80 tx=0x7ff754014d90 comp rx=0 tx=0).stop 2024-04-01T01:41:01.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.326+0000 7ff7697eb740 1 -- 172.21.15.70:0/3064914542 shutdown_connections 2024-04-01T01:41:01.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.326+0000 7ff7697eb740 1 --2- 172.21.15.70:0/3064914542 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55c7a5967c00 0x55c7a596a040 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.326+0000 7ff7697eb740 1 --2- 172.21.15.70:0/3064914542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a5957200 0x55c7a5963190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.326+0000 7ff7697eb740 1 --2- 172.21.15.70:0/3064914542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5952090 0x55c7a595c150 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.326+0000 7ff7697eb740 1 --2- 172.21.15.70:0/3064914542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a59516a0 0x55c7a595bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.326+0000 7ff7697eb740 1 -- 172.21.15.70:0/3064914542 >> 172.21.15.70:0/3064914542 conn(0x55c7a594cc10 msgr2=0x55c7a594bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.326+0000 7ff73ffff640 0 client.27541 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:41:01.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.326+0000 7ff7697eb740 1 -- 172.21.15.70:0/3064914542 shutdown_connections 2024-04-01T01:41:01.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.326+0000 7ff7697eb740 1 -- 172.21.15.70:0/3064914542 wait complete. 2024-04-01T01:41:01.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.326+0000 7ff7697eb740 20 client.27541 trim_cache size 0 max 16384 2024-04-01T01:41:01.330 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:00 smithi081 ceph-mon[35142]: pgmap v585: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 10 MiB/s wr, 900 op/s 2024-04-01T01:41:01.330 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:00 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/928440316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:41:01.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.353+0000 7ff7697eb740 1 Processor -- start 2024-04-01T01:41:01.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.353+0000 7ff7697eb740 1 -- start start 2024-04-01T01:41:01.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.354+0000 7ff7697eb740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a59516a0 0x55c7a594cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.354+0000 7ff7697eb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a5952090 0x55c7a5963150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.354+0000 7ff7697eb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5957200 0x55c7a595bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.354+0000 7ff7697eb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff74400fd10 con 0x55c7a5952090 2024-04-01T01:41:01.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.354+0000 7ff7697eb740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff744011850 con 0x55c7a59516a0 2024-04-01T01:41:01.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.354+0000 7ff7697eb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff7440103e0 con 0x55c7a5957200 2024-04-01T01:41:01.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.354+0000 7ff7651a4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a59516a0 0x55c7a594cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.354+0000 7ff7651a4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a59516a0 0x55c7a594cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57310/0 (socket says 172.21.15.70:57310) 2024-04-01T01:41:01.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.354+0000 7ff7651a4640 1 -- 172.21.15.70:0/1019025594 learned_addr learned my addr 172.21.15.70:0/1019025594 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.354+0000 7ff75ffff640 1 --2- 172.21.15.70:0/1019025594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a5952090 0x55c7a5963150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.354+0000 7ff7651a4640 1 --2- 172.21.15.70:0/1019025594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5957200 0x55c7a595bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.354+0000 7ff7651a4640 1 -- 172.21.15.70:0/1019025594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5957200 msgr2=0x55c7a595bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.354+0000 7ff7651a4640 1 --2- 172.21.15.70:0/1019025594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5957200 0x55c7a595bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.354+0000 7ff7651a4640 1 -- 172.21.15.70:0/1019025594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a5952090 msgr2=0x55c7a5963150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.354+0000 7ff7651a4640 1 --2- 172.21.15.70:0/1019025594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a5952090 0x55c7a5963150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.354+0000 7ff7651a4640 1 -- 172.21.15.70:0/1019025594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7a593de60 con 0x55c7a59516a0 2024-04-01T01:41:01.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.355+0000 7ff7651a4640 1 --2- 172.21.15.70:0/1019025594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a59516a0 0x55c7a594cff0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff74c006010 tx=0x7ff74c009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39e8ff3ef7f7c3e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.355+0000 7ff74a7fc640 1 -- 172.21.15.70:0/1019025594 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c00eca0 con 0x55c7a59516a0 2024-04-01T01:41:01.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.355+0000 7ff74a7fc640 1 -- 172.21.15.70:0/1019025594 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff74c00ee40 con 0x55c7a59516a0 2024-04-01T01:41:01.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.355+0000 7ff74a7fc640 1 -- 172.21.15.70:0/1019025594 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c004bc0 con 0x55c7a59516a0 2024-04-01T01:41:01.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.356+0000 7ff7697eb740 1 -- 172.21.15.70:0/1019025594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a59516a0 msgr2=0x55c7a594cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.356+0000 7ff7697eb740 1 --2- 172.21.15.70:0/1019025594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a59516a0 0x55c7a594cff0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff74c006010 tx=0x7ff74c009620 comp rx=0 tx=0).stop 2024-04-01T01:41:01.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.356+0000 7ff7697eb740 1 -- 172.21.15.70:0/1019025594 shutdown_connections 2024-04-01T01:41:01.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.356+0000 7ff7697eb740 1 --2- 172.21.15.70:0/1019025594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5957200 0x55c7a595bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.356+0000 7ff7697eb740 1 --2- 172.21.15.70:0/1019025594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a5952090 0x55c7a5963150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.356+0000 7ff7697eb740 1 --2- 172.21.15.70:0/1019025594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a59516a0 0x55c7a594cff0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.357+0000 7ff7697eb740 1 -- 172.21.15.70:0/1019025594 >> 172.21.15.70:0/1019025594 conn(0x55c7a594cc10 msgr2=0x55c7a5947e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.357+0000 7ff7697eb740 1 -- 172.21.15.70:0/1019025594 shutdown_connections 2024-04-01T01:41:01.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.357+0000 7ff7697eb740 1 -- 172.21.15.70:0/1019025594 wait complete. 2024-04-01T01:41:01.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.357+0000 7ff7697eb740 1 -- start start 2024-04-01T01:41:01.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.358+0000 7ff7697eb740 1 Processor -- start 2024-04-01T01:41:01.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.358+0000 7ff7697eb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a59516a0 0x55c7a5960930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.358+0000 7ff7697eb740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a5952090 0x55c7a5960e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.358+0000 7ff7697eb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5957200 0x55c7a5961e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.358+0000 7ff7697eb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff744011180 con 0x55c7a59516a0 2024-04-01T01:41:01.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.358+0000 7ff7697eb740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff744010ab0 con 0x55c7a5952090 2024-04-01T01:41:01.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.358+0000 7ff7697eb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff74400dcf0 con 0x55c7a5957200 2024-04-01T01:41:01.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.358+0000 7ff7697eb740 10 client.0 _ll_register_callbacks cb 0x7ff754014540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:01.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.358+0000 7ff75ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a5952090 0x55c7a5960e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.358+0000 7ff75ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a5952090 0x55c7a5960e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57320/0 (socket says 172.21.15.70:57320) 2024-04-01T01:41:01.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.359+0000 7ff75ffff640 1 -- 172.21.15.70:0/1518775928 learned_addr learned my addr 172.21.15.70:0/1518775928 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.359+0000 7ff7651a4640 1 --2- 172.21.15.70:0/1518775928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a59516a0 0x55c7a5960930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.359+0000 7ff7651a4640 1 --2- 172.21.15.70:0/1518775928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5957200 0x55c7a5961e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.359+0000 7ff75ffff640 1 -- 172.21.15.70:0/1518775928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5957200 msgr2=0x55c7a5961e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.359+0000 7ff75ffff640 1 --2- 172.21.15.70:0/1518775928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5957200 0x55c7a5961e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.359+0000 7ff75ffff640 1 -- 172.21.15.70:0/1518775928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a59516a0 msgr2=0x55c7a5960930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.359+0000 7ff75ffff640 1 --2- 172.21.15.70:0/1518775928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a59516a0 0x55c7a5960930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.359+0000 7ff75ffff640 1 -- 172.21.15.70:0/1518775928 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7a5962470 con 0x55c7a5952090 2024-04-01T01:41:01.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.360+0000 7ff75ffff640 1 --2- 172.21.15.70:0/1518775928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a5952090 0x55c7a5960e70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff754014650 tx=0x7ff75400aae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1fd48d42f7ee0c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.360+0000 7ff73ffff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:01.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.360+0000 7ff73ffff640 1 -- 172.21.15.70:0/1518775928 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7540054b0 con 0x55c7a5952090 2024-04-01T01:41:01.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.360+0000 7ff7697eb740 1 -- 172.21.15.70:0/1518775928 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff74c00ee40 con 0x55c7a5952090 2024-04-01T01:41:01.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.361+0000 7ff73ffff640 1 -- 172.21.15.70:0/1518775928 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff754005650 con 0x55c7a5952090 2024-04-01T01:41:01.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.361+0000 7ff73ffff640 1 -- 172.21.15.70:0/1518775928 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff754003640 con 0x55c7a5952090 2024-04-01T01:41:01.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.361+0000 7ff7697eb740 10 client.27698 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:01.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.361+0000 7ff7697eb740 1 -- 172.21.15.70:0/1518775928 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c7a59ce020 con 0x55c7a5952090 2024-04-01T01:41:01.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.361+0000 7ff73ffff640 1 -- 172.21.15.70:0/1518775928 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff74c00ee40 con 0x55c7a5952090 2024-04-01T01:41:01.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.362+0000 7ff73ffff640 1 -- 172.21.15.70:0/1518775928 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff75401e930 con 0x55c7a5952090 2024-04-01T01:41:01.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.362+0000 7ff7697eb740 10 client.27698 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:01.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.362+0000 7ff7697eb740 10 client.27698 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:41:01.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.362+0000 7ff7697eb740 20 client.27698 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:01.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.362+0000 7ff7697eb740 1 --2- 172.21.15.70:0/1518775928 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55c7a59ce9a0 0x55c7a59d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.362+0000 7ff7697eb740 4 client.27698 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:41:01.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.362+0000 7ff7697eb740 1 -- 172.21.15.70:0/1518775928 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c7a58c36f0 con 0x55c7a59ce9a0 2024-04-01T01:41:01.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.364+0000 7ff7651a4640 1 --2- 172.21.15.70:0/1518775928 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55c7a59ce9a0 0x55c7a59d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.366+0000 7ff7651a4640 1 --2- 172.21.15.70:0/1518775928 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55c7a59ce9a0 0x55c7a59d0dc0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d9264daa5ed42204 server_cookie=dcd6432ffa546ee3 in_seq=0 out_seq=0 2024-04-01T01:41:01.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.367+0000 7ff73ffff640 10 client.27698 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:41:01.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.367+0000 7ff73ffff640 1 -- 172.21.15.70:0/1518775928 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c7a58c36f0 con 0x55c7a59ce9a0 2024-04-01T01:41:01.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.367+0000 7ff73ffff640 10 client.27698 handle_command_reply: tid=0 2024-04-01T01:41:01.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.367+0000 7ff7697eb740 1 client.27698 shutdown 2024-04-01T01:41:01.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.367+0000 7ff7697eb740 1 -- 172.21.15.70:0/1518775928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a5952090 msgr2=0x55c7a5960e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.367+0000 7ff7697eb740 1 --2- 172.21.15.70:0/1518775928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a5952090 0x55c7a5960e70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff754014650 tx=0x7ff75400aae0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.370+0000 7ff7697eb740 1 -- 172.21.15.70:0/1518775928 shutdown_connections 2024-04-01T01:41:01.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.370+0000 7ff7697eb740 1 --2- 172.21.15.70:0/1518775928 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55c7a59ce9a0 0x55c7a59d0dc0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.370+0000 7ff7697eb740 1 --2- 172.21.15.70:0/1518775928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7a5957200 0x55c7a5961e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.370+0000 7ff7697eb740 1 --2- 172.21.15.70:0/1518775928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c7a5952090 0x55c7a5960e70 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.370+0000 7ff7697eb740 1 --2- 172.21.15.70:0/1518775928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a59516a0 0x55c7a5960930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.370+0000 7ff7697eb740 1 -- 172.21.15.70:0/1518775928 >> 172.21.15.70:0/1518775928 conn(0x55c7a594cc10 msgr2=0x55c7a594cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.370+0000 7ff73ffff640 0 client.27698 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:41:01.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.370+0000 7ff7697eb740 1 -- 172.21.15.70:0/1518775928 shutdown_connections 2024-04-01T01:41:01.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.370+0000 7ff7697eb740 1 -- 172.21.15.70:0/1518775928 wait complete. 2024-04-01T01:41:01.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.370+0000 7ff7697eb740 20 client.27698 trim_cache size 0 max 16384 2024-04-01T01:41:01.371 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:41:01.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.372+0000 7ff766c07640 1 -- 172.21.15.70:0/1255302885 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7500a2620 msgr2=0x7ff7500a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.372+0000 7ff766c07640 1 --2- 172.21.15.70:0/1255302885 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7500a2620 0x7ff7500a4ae0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff760067290 tx=0x7ff760141040 comp rx=0 tx=0).stop 2024-04-01T01:41:01.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.372+0000 7ff766c07640 1 -- 172.21.15.70:0/1255302885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7580a5010 msgr2=0x7ff7581435e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.372+0000 7ff766c07640 1 --2- 172.21.15.70:0/1255302885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7580a5010 0x7ff7581435e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff760050720 tx=0x7ff76006d7e0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.372+0000 7ff766c07640 1 -- 172.21.15.70:0/1255302885 shutdown_connections 2024-04-01T01:41:01.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.372+0000 7ff766c07640 1 --2- 172.21.15.70:0/1255302885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7580a6bc0 0x7ff75814ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.372+0000 7ff766c07640 1 --2- 172.21.15.70:0/1255302885 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7500a2620 0x7ff7500a4ae0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.372+0000 7ff766c07640 1 --2- 172.21.15.70:0/1255302885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7580a6200 0x7ff758143b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.372+0000 7ff766c07640 1 --2- 172.21.15.70:0/1255302885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7580a5010 0x7ff7581435e0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.372+0000 7ff766c07640 1 -- 172.21.15.70:0/1255302885 >> 172.21.15.70:0/1255302885 conn(0x7ff7580a0560 msgr2=0x7ff7580b27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.372+0000 7ff766c07640 1 -- 172.21.15.70:0/1255302885 shutdown_connections 2024-04-01T01:41:01.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.372+0000 7ff766c07640 1 -- 172.21.15.70:0/1255302885 wait complete. 2024-04-01T01:41:01.388 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:41:01.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.580+0000 7f45ecb81640 1 -- 172.21.15.70:0/4245077627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e8069470 msgr2=0x7f45e80ffd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.580+0000 7f45ecb81640 1 --2- 172.21.15.70:0/4245077627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e8069470 0x7f45e80ffd00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f45dc009900 tx=0x7f45dc02bc40 comp rx=0 tx=0).stop 2024-04-01T01:41:01.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.581+0000 7f45ecb81640 1 -- 172.21.15.70:0/4245077627 shutdown_connections 2024-04-01T01:41:01.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.581+0000 7f45ecb81640 1 --2- 172.21.15.70:0/4245077627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45e8100ef0 0x7f45e8101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.581+0000 7f45ecb81640 1 --2- 172.21.15.70:0/4245077627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e8069920 0x7f45e8100260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.581+0000 7f45ecb81640 1 --2- 172.21.15.70:0/4245077627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e8069470 0x7f45e80ffd00 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.581+0000 7f45ecb81640 1 -- 172.21.15.70:0/4245077627 >> 172.21.15.70:0/4245077627 conn(0x7f45e80fb270 msgr2=0x7f45e80fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.581+0000 7f45ecb81640 1 -- 172.21.15.70:0/4245077627 shutdown_connections 2024-04-01T01:41:01.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.581+0000 7f45ecb81640 1 -- 172.21.15.70:0/4245077627 wait complete. 2024-04-01T01:41:01.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.583+0000 7f45ecb81640 1 Processor -- start 2024-04-01T01:41:01.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.583+0000 7f45ecb81640 1 -- start start 2024-04-01T01:41:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.583+0000 7f45ecb81640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45e8069470 0x7f45e819dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.583+0000 7f45ecb81640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e8069920 0x7f45e819e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.583+0000 7f45ecb81640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e8100ef0 0x7f45e81a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.583+0000 7f45ecb81640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f45e8107050 con 0x7f45e8069920 2024-04-01T01:41:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.583+0000 7f45ecb81640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f45e8106ed0 con 0x7f45e8100ef0 2024-04-01T01:41:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.583+0000 7f45ecb81640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f45e81071d0 con 0x7f45e8069470 2024-04-01T01:41:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.583+0000 7f45e659c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45e8069470 0x7f45e819dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.583+0000 7f45e659c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45e8069470 0x7f45e819dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43126/0 (socket says 172.21.15.70:43126) 2024-04-01T01:41:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.583+0000 7f45e659c640 1 -- 172.21.15.70:0/2297465613 learned_addr learned my addr 172.21.15.70:0/2297465613 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.583+0000 7f45e6d9d640 1 --2- 172.21.15.70:0/2297465613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e8100ef0 0x7f45e81a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.583+0000 7f45e5d9b640 1 --2- 172.21.15.70:0/2297465613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e8069920 0x7f45e819e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.584+0000 7f45e6d9d640 1 -- 172.21.15.70:0/2297465613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45e8069470 msgr2=0x7f45e819dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.584+0000 7f45e6d9d640 1 --2- 172.21.15.70:0/2297465613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45e8069470 0x7f45e819dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.584+0000 7f45e6d9d640 1 -- 172.21.15.70:0/2297465613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e8069920 msgr2=0x7f45e819e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.584+0000 7f45e6d9d640 1 --2- 172.21.15.70:0/2297465613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e8069920 0x7f45e819e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.584+0000 7f45e6d9d640 1 -- 172.21.15.70:0/2297465613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e81a5c00 con 0x7f45e8100ef0 2024-04-01T01:41:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.584+0000 7f45e659c640 1 --2- 172.21.15.70:0/2297465613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45e8069470 0x7f45e819dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:41:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.584+0000 7f45e6d9d640 1 --2- 172.21.15.70:0/2297465613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e8100ef0 0x7f45e81a5590 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f45e000e940 tx=0x7f45e000ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.584+0000 7f45d77fe640 1 -- 172.21.15.70:0/2297465613 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e0004d20 con 0x7f45e8100ef0 2024-04-01T01:41:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.584+0000 7f45d77fe640 1 -- 172.21.15.70:0/2297465613 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f45e0005020 con 0x7f45e8100ef0 2024-04-01T01:41:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.585+0000 7f45ecb81640 1 -- 172.21.15.70:0/2297465613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45e81a5ef0 con 0x7f45e8100ef0 2024-04-01T01:41:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.585+0000 7f45ecb81640 1 -- 172.21.15.70:0/2297465613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45e81a63d0 con 0x7f45e8100ef0 2024-04-01T01:41:01.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.586+0000 7f45d77fe640 1 -- 172.21.15.70:0/2297465613 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e00116b0 con 0x7f45e8100ef0 2024-04-01T01:41:01.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.588+0000 7f45d77fe640 1 -- 172.21.15.70:0/2297465613 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f45e000cb20 con 0x7f45e8100ef0 2024-04-01T01:41:01.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.589+0000 7f45ef764740 1 Processor -- start 2024-04-01T01:41:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.589+0000 7f45ef764740 1 -- start start 2024-04-01T01:41:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.589+0000 7f45ef764740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 0x562caae17ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.589+0000 7f45ef764740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae18090 0x562caae1cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.589+0000 7f45ef764740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae1d200 0x562caae1f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.589+0000 7f45ef764740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562caad7c1b0 con 0x562caae18090 2024-04-01T01:41:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.589+0000 7f45ef764740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562caad24c00 con 0x562caae176a0 2024-04-01T01:41:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.589+0000 7f45ef764740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562caad78250 con 0x562caae1d200 2024-04-01T01:41:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.590+0000 7f45e5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 0x562caae17ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.590+0000 7f45e5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 0x562caae17ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57346/0 (socket says 172.21.15.70:57346) 2024-04-01T01:41:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.590+0000 7f45e5d9b640 1 -- 172.21.15.70:0/3221585237 learned_addr learned my addr 172.21.15.70:0/3221585237 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.590+0000 7f45e5d9b640 1 --2- 172.21.15.70:0/3221585237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae1d200 0x562caae1f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.590+0000 7f45e5d9b640 1 -- 172.21.15.70:0/3221585237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae1d200 msgr2=0x562caae1f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.590+0000 7f45e5d9b640 1 --2- 172.21.15.70:0/3221585237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae1d200 0x562caae1f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.590+0000 7f45e5d9b640 1 -- 172.21.15.70:0/3221585237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae18090 msgr2=0x562caae1cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.590+0000 7f45e5d9b640 1 --2- 172.21.15.70:0/3221585237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae18090 0x562caae1cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.591+0000 7f45e5d9b640 1 -- 172.21.15.70:0/3221585237 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562caad443d0 con 0x562caae176a0 2024-04-01T01:41:01.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.591+0000 7f45e5d9b640 1 --2- 172.21.15.70:0/3221585237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 0x562caae17ac0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f45d8016500 tx=0x7f45d800a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5448eb33ada85a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.591+0000 7f45d77fe640 1 --2- 172.21.15.70:0/2297465613 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f45d00a25f0 0x7f45d00a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.591+0000 7f45d77fe640 1 -- 172.21.15.70:0/2297465613 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f45e00ceb00 con 0x7f45e8100ef0 2024-04-01T01:41:01.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.591+0000 7f45d4ff9640 1 -- 172.21.15.70:0/3221585237 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8018430 con 0x562caae176a0 2024-04-01T01:41:01.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.592+0000 7f45d4ff9640 1 -- 172.21.15.70:0/3221585237 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f45d8018a80 con 0x562caae176a0 2024-04-01T01:41:01.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.592+0000 7f45d4ff9640 1 -- 172.21.15.70:0/3221585237 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d80178a0 con 0x562caae176a0 2024-04-01T01:41:01.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.592+0000 7f45e659c640 1 --2- 172.21.15.70:0/2297465613 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f45d00a25f0 0x7f45d00a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.593+0000 7f45ef764740 1 -- 172.21.15.70:0/3221585237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 msgr2=0x562caae17ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.593+0000 7f45ef764740 1 --2- 172.21.15.70:0/3221585237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 0x562caae17ac0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f45d8016500 tx=0x7f45d800a290 comp rx=0 tx=0).stop 2024-04-01T01:41:01.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.593+0000 7f45ef764740 1 -- 172.21.15.70:0/3221585237 shutdown_connections 2024-04-01T01:41:01.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.593+0000 7f45ef764740 1 --2- 172.21.15.70:0/3221585237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae1d200 0x562caae1f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.593+0000 7f45ef764740 1 --2- 172.21.15.70:0/3221585237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae18090 0x562caae1cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.593+0000 7f45ef764740 1 --2- 172.21.15.70:0/3221585237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 0x562caae17ac0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.593+0000 7f45ef764740 1 -- 172.21.15.70:0/3221585237 >> 172.21.15.70:0/3221585237 conn(0x562caae12c10 msgr2=0x562caae13010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.593+0000 7f45ef764740 1 -- 172.21.15.70:0/3221585237 shutdown_connections 2024-04-01T01:41:01.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.593+0000 7f45ef764740 1 -- 172.21.15.70:0/3221585237 wait complete. 2024-04-01T01:41:01.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.593+0000 7f45ef764740 1 -- start start 2024-04-01T01:41:01.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.593+0000 7f45e659c640 1 --2- 172.21.15.70:0/2297465613 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f45d00a25f0 0x7f45d00a4ab0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x562caae18490 tx=0x7f45dc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.593+0000 7f45ef764740 1 Processor -- start 2024-04-01T01:41:01.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.594+0000 7f45ef764740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae176a0 0x562caae32c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.594+0000 7f45ef764740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae18090 0x562caae331d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.594+0000 7f45ef764740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae1d200 0x562caae3a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.594+0000 7f45ef764740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562caae311b0 con 0x562caae176a0 2024-04-01T01:41:01.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.594+0000 7f45ef764740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562caad1a470 con 0x562caae18090 2024-04-01T01:41:01.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.594+0000 7f45ef764740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562caae31330 con 0x562caae1d200 2024-04-01T01:41:01.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.594+0000 7f45ef764740 10 client.0 _ll_register_callbacks cb 0x562caad37cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:01.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.595+0000 7f45e6d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae1d200 0x562caae3a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.595+0000 7f45e6d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae1d200 0x562caae3a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43148/0 (socket says 172.21.15.70:43148) 2024-04-01T01:41:01.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.595+0000 7f45e6d9d640 1 -- 172.21.15.70:0/2060996311 learned_addr learned my addr 172.21.15.70:0/2060996311 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.595+0000 7f45e659c640 1 --2- 172.21.15.70:0/2060996311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae176a0 0x562caae32c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.595+0000 7f45e6d9d640 1 -- 172.21.15.70:0/2060996311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae18090 msgr2=0x562caae331d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:41:01.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.595+0000 7f45e6d9d640 1 --2- 172.21.15.70:0/2060996311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae18090 0x562caae331d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.595+0000 7f45e6d9d640 1 -- 172.21.15.70:0/2060996311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae176a0 msgr2=0x562caae32c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.595+0000 7f45e6d9d640 1 --2- 172.21.15.70:0/2060996311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae176a0 0x562caae32c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.595+0000 7f45e6d9d640 1 -- 172.21.15.70:0/2060996311 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562caaa903f0 con 0x562caae1d200 2024-04-01T01:41:01.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.595+0000 7f45e6d9d640 1 --2- 172.21.15.70:0/2060996311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae1d200 0x562caae3a210 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f45e00a3670 tx=0x7f45e009b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e0e303012c3306a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.595+0000 7f45cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:41:01.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.596+0000 7f45cdffb640 1 -- 172.21.15.70:0/2060996311 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e001f040 con 0x562caae1d200 2024-04-01T01:41:01.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.596+0000 7f45cdffb640 1 -- 172.21.15.70:0/2060996311 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f45e00b9d70 con 0x562caae1d200 2024-04-01T01:41:01.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.596+0000 7f45ef764740 1 -- 172.21.15.70:0/2060996311 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45d8018a80 con 0x562caae1d200 2024-04-01T01:41:01.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.596+0000 7f45cdffb640 1 -- 172.21.15.70:0/2060996311 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f45e009f2e0 con 0x562caae1d200 2024-04-01T01:41:01.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.597+0000 7f45ef764740 10 client.27559 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:01.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.597+0000 7f45ef764740 1 -- 172.21.15.70:0/2060996311 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562caab5a7d0 con 0x562caae1d200 2024-04-01T01:41:01.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.597+0000 7f45cdffb640 1 -- 172.21.15.70:0/2060996311 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f45e00a3ba0 con 0x562caae1d200 2024-04-01T01:41:01.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.598+0000 7f45cdffb640 1 -- 172.21.15.70:0/2060996311 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f45e000f040 con 0x562caae1d200 2024-04-01T01:41:01.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.598+0000 7f45ef764740 10 client.27559 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:01.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.598+0000 7f45ef764740 10 client.27559 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:41:01.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.598+0000 7f45ef764740 20 client.27559 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:01.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.599+0000 7f45ef764740 1 --2- 172.21.15.70:0/2060996311 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x562caae3e6d0 0x562caae40b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.599+0000 7f45ef764740 4 client.27559 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:01.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.599+0000 7f45ef764740 1 -- 172.21.15.70:0/2060996311 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f45d8018a80 con 0x562caae3e6d0 2024-04-01T01:41:01.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.599+0000 7f45e659c640 1 --2- 172.21.15.70:0/2060996311 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x562caae3e6d0 0x562caae40b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.600+0000 7f45e659c640 1 --2- 172.21.15.70:0/2060996311 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x562caae3e6d0 0x562caae40b70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8893e1f92764149d server_cookie=fd97712fccb61e22 in_seq=0 out_seq=0 2024-04-01T01:41:01.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.600+0000 7f45cdffb640 10 client.27559 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:41:01.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.601+0000 7f45cdffb640 1 -- 172.21.15.70:0/2060996311 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f45d8018a80 con 0x562caae3e6d0 2024-04-01T01:41:01.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.602+0000 7f45cdffb640 10 client.27559 handle_command_reply: tid=0 2024-04-01T01:41:01.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.602+0000 7f45ef764740 1 client.27559 shutdown 2024-04-01T01:41:01.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.603+0000 7f45ef764740 1 -- 172.21.15.70:0/2060996311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae1d200 msgr2=0x562caae3a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.603+0000 7f45ef764740 1 --2- 172.21.15.70:0/2060996311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae1d200 0x562caae3a210 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f45e00a3670 tx=0x7f45e009b890 comp rx=0 tx=0).stop 2024-04-01T01:41:01.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.603+0000 7f45ef764740 1 -- 172.21.15.70:0/2060996311 shutdown_connections 2024-04-01T01:41:01.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.603+0000 7f45ef764740 1 --2- 172.21.15.70:0/2060996311 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x562caae3e6d0 0x562caae40b70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.603+0000 7f45ef764740 1 --2- 172.21.15.70:0/2060996311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae1d200 0x562caae3a210 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.603+0000 7f45ef764740 1 --2- 172.21.15.70:0/2060996311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae18090 0x562caae331d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.603+0000 7f45ef764740 1 --2- 172.21.15.70:0/2060996311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae176a0 0x562caae32c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.603+0000 7f45ef764740 1 -- 172.21.15.70:0/2060996311 >> 172.21.15.70:0/2060996311 conn(0x562caae12c10 msgr2=0x562caae11f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.603+0000 7f45cdffb640 0 client.27559 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:41:01.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.607+0000 7f45ef764740 1 -- 172.21.15.70:0/2060996311 shutdown_connections 2024-04-01T01:41:01.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.608+0000 7f45ef764740 1 -- 172.21.15.70:0/2060996311 wait complete. 2024-04-01T01:41:01.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.608+0000 7f45ef764740 20 client.27559 trim_cache size 0 max 16384 2024-04-01T01:41:01.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.639+0000 7f45ef764740 1 Processor -- start 2024-04-01T01:41:01.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.639+0000 7f45ef764740 1 -- start start 2024-04-01T01:41:01.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.640+0000 7f45ef764740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 0x562caae11e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.640+0000 7f45ef764740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae18090 0x562caae3a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.640+0000 7f45ef764740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae1d200 0x562caae45430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.640+0000 7f45ef764740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f45c800fcb0 con 0x562caae1d200 2024-04-01T01:41:01.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.640+0000 7f45ef764740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f45c80117f0 con 0x562caae176a0 2024-04-01T01:41:01.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.640+0000 7f45ef764740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f45c8010380 con 0x562caae18090 2024-04-01T01:41:01.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.640+0000 7f45e659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 0x562caae11e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.641+0000 7f45e5d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae18090 0x562caae3a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.641+0000 7f45e5d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae18090 0x562caae3a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43164/0 (socket says 172.21.15.70:43164) 2024-04-01T01:41:01.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.641+0000 7f45e5d9b640 1 -- 172.21.15.70:0/2292873323 learned_addr learned my addr 172.21.15.70:0/2292873323 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.641+0000 7f45e5d9b640 1 -- 172.21.15.70:0/2292873323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 msgr2=0x562caae11e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.641+0000 7f45e5d9b640 1 --2- 172.21.15.70:0/2292873323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 0x562caae11e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.641+0000 7f45e5d9b640 1 -- 172.21.15.70:0/2292873323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae1d200 msgr2=0x562caae45430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.641+0000 7f45e5d9b640 1 --2- 172.21.15.70:0/2292873323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae1d200 0x562caae45430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.642+0000 7f45e5d9b640 1 -- 172.21.15.70:0/2292873323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562caae1cf50 con 0x562caae18090 2024-04-01T01:41:01.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.642+0000 7f45e5d9b640 1 --2- 172.21.15.70:0/2292873323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae18090 0x562caae3a1d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f45d80164d0 tx=0x7f45d800a1b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=741e0815eed357b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.642+0000 7f45cffff640 1 -- 172.21.15.70:0/2292873323 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8026020 con 0x562caae18090 2024-04-01T01:41:01.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.642+0000 7f45cffff640 1 -- 172.21.15.70:0/2292873323 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f45d8021af0 con 0x562caae18090 2024-04-01T01:41:01.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.642+0000 7f45cffff640 1 -- 172.21.15.70:0/2292873323 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d80188f0 con 0x562caae18090 2024-04-01T01:41:01.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.644+0000 7f45ef764740 1 -- 172.21.15.70:0/2292873323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae18090 msgr2=0x562caae3a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.644+0000 7f45ef764740 1 --2- 172.21.15.70:0/2292873323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae18090 0x562caae3a1d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f45d80164d0 tx=0x7f45d800a1b0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.645+0000 7f45ef764740 1 -- 172.21.15.70:0/2292873323 shutdown_connections 2024-04-01T01:41:01.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.645+0000 7f45ef764740 1 --2- 172.21.15.70:0/2292873323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae1d200 0x562caae45430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.645+0000 7f45ef764740 1 --2- 172.21.15.70:0/2292873323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae18090 0x562caae3a1d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.645+0000 7f45ef764740 1 --2- 172.21.15.70:0/2292873323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 0x562caae11e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.645+0000 7f45ef764740 1 -- 172.21.15.70:0/2292873323 >> 172.21.15.70:0/2292873323 conn(0x562caae12c10 msgr2=0x562caae11700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.646+0000 7f45ef764740 1 -- 172.21.15.70:0/2292873323 shutdown_connections 2024-04-01T01:41:01.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.646+0000 7f45ef764740 1 -- 172.21.15.70:0/2292873323 wait complete. 2024-04-01T01:41:01.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.647+0000 7f45ef764740 1 -- start start 2024-04-01T01:41:01.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.647+0000 7f45ef764740 1 Processor -- start 2024-04-01T01:41:01.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.647+0000 7f45ef764740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 0x562caae37a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.647+0000 7f45ef764740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae18090 0x562caae37f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.647+0000 7f45ef764740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae1d200 0x562caae38f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.648+0000 7f45ef764740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f45c8011120 con 0x562caae1d200 2024-04-01T01:41:01.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.648+0000 7f45ef764740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f45c8010a50 con 0x562caae176a0 2024-04-01T01:41:01.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.648+0000 7f45ef764740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f45c800dc90 con 0x562caae18090 2024-04-01T01:41:01.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.648+0000 7f45e659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 0x562caae37a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.648+0000 7f45e6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae1d200 0x562caae38f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.648+0000 7f45e659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 0x562caae37a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57364/0 (socket says 172.21.15.70:57364) 2024-04-01T01:41:01.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.648+0000 7f45e659c640 1 -- 172.21.15.70:0/3578036216 learned_addr learned my addr 172.21.15.70:0/3578036216 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.648+0000 7f45ef764740 10 client.0 _ll_register_callbacks cb 0x7f45e0004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:01.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.648+0000 7f45e5d9b640 1 --2- 172.21.15.70:0/3578036216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae18090 0x562caae37f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.648+0000 7f45e6d9d640 1 -- 172.21.15.70:0/3578036216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae18090 msgr2=0x562caae37f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.648+0000 7f45e6d9d640 1 --2- 172.21.15.70:0/3578036216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae18090 0x562caae37f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.648+0000 7f45e6d9d640 1 -- 172.21.15.70:0/3578036216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 msgr2=0x562caae37a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.648+0000 7f45e6d9d640 1 --2- 172.21.15.70:0/3578036216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 0x562caae37a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.648+0000 7f45e6d9d640 1 -- 172.21.15.70:0/3578036216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562caae39520 con 0x562caae1d200 2024-04-01T01:41:01.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.648+0000 7f45e5d9b640 1 --2- 172.21.15.70:0/3578036216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae18090 0x562caae37f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:41:01.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.648+0000 7f45ef764740 1 -- 172.21.15.70:0/3578036216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45d8021af0 con 0x562caae1d200 2024-04-01T01:41:01.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.650+0000 7f45e6d9d640 1 --2- 172.21.15.70:0/3578036216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae1d200 0x562caae38f30 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f45e000c640 tx=0x7f45e00ce000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=975af2304a3b3a51 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.650+0000 7f45cdffb640 10 client.17766 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:41:01.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.654+0000 7f45cdffb640 1 -- 172.21.15.70:0/3578036216 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e001f070 con 0x562caae1d200 2024-04-01T01:41:01.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.654+0000 7f45cdffb640 1 -- 172.21.15.70:0/3578036216 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f45e00a0470 con 0x562caae1d200 2024-04-01T01:41:01.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.655+0000 7f45cdffb640 1 -- 172.21.15.70:0/3578036216 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f45e007e080 con 0x562caae1d200 2024-04-01T01:41:01.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.655+0000 7f45ef764740 10 client.17766 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:01.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.655+0000 7f45ef764740 1 -- 172.21.15.70:0/3578036216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({fsmap=31}) -- 0x562caaea4ae0 con 0x562caae1d200 2024-04-01T01:41:01.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.656+0000 7f45cdffb640 1 -- 172.21.15.70:0/3578036216 <== mon.0 v2:172.21.15.70:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f45e00a06b0 con 0x562caae1d200 2024-04-01T01:41:01.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.656+0000 7f45ef764740 10 client.17766 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:01.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.656+0000 7f45ef764740 10 client.17766 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:41:01.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.656+0000 7f45ef764740 20 client.17766 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:01.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.656+0000 7f45ef764740 1 --2- 172.21.15.70:0/3578036216 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x562caaea52c0 0x562caaea76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.657+0000 7f45ef764740 4 client.17766 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:41:01.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.657+0000 7f45ef764740 1 -- 172.21.15.70:0/3578036216 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562caad896f0 con 0x562caaea52c0 2024-04-01T01:41:01.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.657+0000 7f45e659c640 1 --2- 172.21.15.70:0/3578036216 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x562caaea52c0 0x562caaea76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.658+0000 7f45e659c640 1 --2- 172.21.15.70:0/3578036216 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x562caaea52c0 0x562caaea76e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e9cc3c0c17c7416c server_cookie=f24ae230db683fd5 in_seq=0 out_seq=0 2024-04-01T01:41:01.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.658+0000 7f45cdffb640 10 client.17766 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:41:01.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.658+0000 7f45cdffb640 1 -- 172.21.15.70:0/3578036216 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562caad896f0 con 0x562caaea52c0 2024-04-01T01:41:01.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.658+0000 7f45cdffb640 10 client.17766 handle_command_reply: tid=0 2024-04-01T01:41:01.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.659+0000 7f45ef764740 1 client.17766 shutdown 2024-04-01T01:41:01.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.659+0000 7f45ef764740 1 -- 172.21.15.70:0/3578036216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae1d200 msgr2=0x562caae38f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.659+0000 7f45ef764740 1 --2- 172.21.15.70:0/3578036216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae1d200 0x562caae38f30 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f45e000c640 tx=0x7f45e00ce000 comp rx=0 tx=0).stop 2024-04-01T01:41:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.661+0000 7f45ef764740 1 -- 172.21.15.70:0/3578036216 shutdown_connections 2024-04-01T01:41:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.661+0000 7f45ef764740 1 --2- 172.21.15.70:0/3578036216 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x562caaea52c0 0x562caaea76e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.661+0000 7f45ef764740 1 --2- 172.21.15.70:0/3578036216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562caae1d200 0x562caae38f30 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.661+0000 7f45ef764740 1 --2- 172.21.15.70:0/3578036216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562caae18090 0x562caae37f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.661+0000 7f45ef764740 1 --2- 172.21.15.70:0/3578036216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562caae176a0 0x562caae37a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.661+0000 7f45ef764740 1 -- 172.21.15.70:0/3578036216 >> 172.21.15.70:0/3578036216 conn(0x562caae12c10 msgr2=0x562caae16700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.661+0000 7f45cdffb640 0 client.17766 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:41:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.661+0000 7f45ef764740 1 -- 172.21.15.70:0/3578036216 shutdown_connections 2024-04-01T01:41:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.662+0000 7f45ef764740 1 -- 172.21.15.70:0/3578036216 wait complete. 2024-04-01T01:41:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.662+0000 7f45ef764740 20 client.17766 trim_cache size 0 max 16384 2024-04-01T01:41:01.662 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:41:01.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.663+0000 7f45ecb81640 1 -- 172.21.15.70:0/2297465613 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f45d00a25f0 msgr2=0x7f45d00a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.663+0000 7f45ecb81640 1 --2- 172.21.15.70:0/2297465613 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f45d00a25f0 0x7f45d00a4ab0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x562caae18490 tx=0x7f45dc009260 comp rx=0 tx=0).stop 2024-04-01T01:41:01.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.663+0000 7f45ecb81640 1 -- 172.21.15.70:0/2297465613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e8100ef0 msgr2=0x7f45e81a5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.663+0000 7f45ecb81640 1 --2- 172.21.15.70:0/2297465613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e8100ef0 0x7f45e81a5590 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f45e000e940 tx=0x7f45e000ee00 comp rx=0 tx=0).stop 2024-04-01T01:41:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.664+0000 7f45ecb81640 1 -- 172.21.15.70:0/2297465613 shutdown_connections 2024-04-01T01:41:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.664+0000 7f45ecb81640 1 --2- 172.21.15.70:0/2297465613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e8100ef0 0x7f45e81a5590 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.664+0000 7f45ecb81640 1 --2- 172.21.15.70:0/2297465613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e8069920 0x7f45e819e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.664+0000 7f45ecb81640 1 --2- 172.21.15.70:0/2297465613 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f45d00a25f0 0x7f45d00a4ab0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.664+0000 7f45ecb81640 1 --2- 172.21.15.70:0/2297465613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45e8069470 0x7f45e819dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.664+0000 7f45ecb81640 1 -- 172.21.15.70:0/2297465613 >> 172.21.15.70:0/2297465613 conn(0x7f45e80fb270 msgr2=0x7f45e8107350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.665+0000 7f45ecb81640 1 -- 172.21.15.70:0/2297465613 shutdown_connections 2024-04-01T01:41:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.665+0000 7f45ecb81640 1 -- 172.21.15.70:0/2297465613 wait complete. 2024-04-01T01:41:01.685 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:41:01.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 375.0 s, 823.2 tps, lat 120.329 ms stddev 58.360 2024-04-01T01:41:01.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.905+0000 7f0fbf249640 1 -- 172.21.15.70:0/2201469336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fb810b0c0 msgr2=0x7f0fb810b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.905+0000 7f0fbf249640 1 --2- 172.21.15.70:0/2201469336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fb810b0c0 0x7f0fb810b520 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009900 tx=0x7f0fac02bc50 comp rx=0 tx=0).stop 2024-04-01T01:41:01.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.908+0000 7f0fbf249640 1 -- 172.21.15.70:0/2201469336 shutdown_connections 2024-04-01T01:41:01.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.908+0000 7f0fbf249640 1 --2- 172.21.15.70:0/2201469336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb8111e90 0x7f0fb81142a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.908+0000 7f0fbf249640 1 --2- 172.21.15.70:0/2201469336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fb810b0c0 0x7f0fb810b520 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.908+0000 7f0fbf249640 1 --2- 172.21.15.70:0/2201469336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fb8108750 0x7f0fb810ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.908+0000 7f0fbf249640 1 -- 172.21.15.70:0/2201469336 >> 172.21.15.70:0/2201469336 conn(0x7f0fb806e5b0 msgr2=0x7f0fb80709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.912+0000 7f0fbf249640 1 -- 172.21.15.70:0/2201469336 shutdown_connections 2024-04-01T01:41:01.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.915+0000 7f0fbf249640 1 -- 172.21.15.70:0/2201469336 wait complete. 2024-04-01T01:41:01.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.916+0000 7f0fbf249640 1 Processor -- start 2024-04-01T01:41:01.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.918+0000 7f0fbf249640 1 -- start start 2024-04-01T01:41:01.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.918+0000 7f0fbf249640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb8108750 0x7f0fb819e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.918+0000 7f0fbf249640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fb810b0c0 0x7f0fb819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.918+0000 7f0fbf249640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fb8111e90 0x7f0fb81a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.918+0000 7f0fbf249640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb8117d80 con 0x7f0fb810b0c0 2024-04-01T01:41:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.918+0000 7f0fbf249640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb8117c00 con 0x7f0fb8108750 2024-04-01T01:41:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.918+0000 7f0fbf249640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb8117f00 con 0x7f0fb8111e90 2024-04-01T01:41:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.919+0000 7f0fbcfe5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb8108750 0x7f0fb819e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.919+0000 7f0fbcfe5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb8108750 0x7f0fb819e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57384/0 (socket says 172.21.15.70:57384) 2024-04-01T01:41:01.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.919+0000 7f0fbcfe5640 1 -- 172.21.15.70:0/2904023564 learned_addr learned my addr 172.21.15.70:0/2904023564 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.919+0000 7f0fbcfe5640 1 -- 172.21.15.70:0/2904023564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fb8111e90 msgr2=0x7f0fb81a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.919+0000 7f0fbcfe5640 1 --2- 172.21.15.70:0/2904023564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fb8111e90 0x7f0fb81a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.919+0000 7f0fbcfe5640 1 -- 172.21.15.70:0/2904023564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fb810b0c0 msgr2=0x7f0fb819e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.919+0000 7f0fbcfe5640 1 --2- 172.21.15.70:0/2904023564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fb810b0c0 0x7f0fb819e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.919+0000 7f0fbcfe5640 1 -- 172.21.15.70:0/2904023564 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb81b6d00 con 0x7f0fb8108750 2024-04-01T01:41:01.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.921+0000 7f0fbcfe5640 1 --2- 172.21.15.70:0/2904023564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb8108750 0x7f0fb819e130 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800b450 tx=0x7f0fa800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.924+0000 7f0fb5ffb640 1 -- 172.21.15.70:0/2904023564 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8015020 con 0x7f0fb8108750 2024-04-01T01:41:01.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.924+0000 7f0fb5ffb640 1 -- 172.21.15.70:0/2904023564 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0fa8011d40 con 0x7f0fb8108750 2024-04-01T01:41:01.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.924+0000 7f0fbf249640 1 -- 172.21.15.70:0/2904023564 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fb81b6ff0 con 0x7f0fb8108750 2024-04-01T01:41:01.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.924+0000 7f0fbf249640 1 -- 172.21.15.70:0/2904023564 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fb81b7530 con 0x7f0fb8108750 2024-04-01T01:41:01.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.925+0000 7f0fb5ffb640 1 -- 172.21.15.70:0/2904023564 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa800f340 con 0x7f0fb8108750 2024-04-01T01:41:01.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.929+0000 7f0fc1e20740 1 Processor -- start 2024-04-01T01:41:01.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.931+0000 7f0fb5ffb640 1 -- 172.21.15.70:0/2904023564 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f0fa8018020 con 0x7f0fb8108750 2024-04-01T01:41:01.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.932+0000 7f0fb5ffb640 1 --2- 172.21.15.70:0/2904023564 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0fa00a26a0 0x7f0fa00a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.932+0000 7f0fb5ffb640 1 -- 172.21.15.70:0/2904023564 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f0fa80ce520 con 0x7f0fb8108750 2024-04-01T01:41:01.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.934+0000 7f0fbd7e6640 1 --2- 172.21.15.70:0/2904023564 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0fa00a26a0 0x7f0fa00a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.934+0000 7f0fc1e20740 1 -- start start 2024-04-01T01:41:01.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.934+0000 7f0fc1e20740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b37e490 0x563b5b37e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.934+0000 7f0fc1e20740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37ee80 0x563b5b383980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.934+0000 7f0fc1e20740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b383ff0 0x563b5b386480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.934+0000 7f0fc1e20740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563b5b2e31b0 con 0x563b5b37ee80 2024-04-01T01:41:01.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.934+0000 7f0fc1e20740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563b5b28bc00 con 0x563b5b383ff0 2024-04-01T01:41:01.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.934+0000 7f0fc1e20740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563b5b2df250 con 0x563b5b37e490 2024-04-01T01:41:01.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.935+0000 7f0fbd7e6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b383ff0 0x563b5b386480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.935+0000 7f0fbcfe5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b37e490 0x563b5b37e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.936+0000 7f0fbcfe5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b37e490 0x563b5b37e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43206/0 (socket says 172.21.15.70:43206) 2024-04-01T01:41:01.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.936+0000 7f0fbcfe5640 1 -- 172.21.15.70:0/605635853 learned_addr learned my addr 172.21.15.70:0/605635853 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.938+0000 7f0fbcfe5640 1 -- 172.21.15.70:0/605635853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b383ff0 msgr2=0x563b5b386480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.938+0000 7f0fbcfe5640 1 --2- 172.21.15.70:0/605635853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b383ff0 0x563b5b386480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.938+0000 7f0fbcfe5640 1 -- 172.21.15.70:0/605635853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37ee80 msgr2=0x563b5b383980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.938+0000 7f0fbcfe5640 1 --2- 172.21.15.70:0/605635853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37ee80 0x563b5b383980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.938+0000 7f0fbcfe5640 1 -- 172.21.15.70:0/605635853 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b5b2ab3d0 con 0x563b5b37e490 2024-04-01T01:41:01.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.938+0000 7f0fbd7e6640 1 --2- 172.21.15.70:0/605635853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b383ff0 0x563b5b386480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:41:01.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.939+0000 7f0fbd7e6640 1 --2- 172.21.15.70:0/2904023564 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0fa00a26a0 0x7f0fa00a4b60 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0005c40 tx=0x7f0fb0012620 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.942+0000 7f0fbcfe5640 1 --2- 172.21.15.70:0/605635853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b37e490 0x563b5b37e8b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0fa80a3090 tx=0x7f0fa8010fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82d65a0d820d4227 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.942+0000 7f0f9effd640 1 -- 172.21.15.70:0/605635853 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa80cbd00 con 0x563b5b37e490 2024-04-01T01:41:01.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.942+0000 7f0f9effd640 1 -- 172.21.15.70:0/605635853 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0fa80b3070 con 0x563b5b37e490 2024-04-01T01:41:01.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.943+0000 7f0fc1e20740 1 -- 172.21.15.70:0/605635853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b37e490 msgr2=0x563b5b37e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.943+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/605635853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b37e490 0x563b5b37e8b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0fa80a3090 tx=0x7f0fa8010fc0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.946+0000 7f0fc1e20740 1 -- 172.21.15.70:0/605635853 shutdown_connections 2024-04-01T01:41:01.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.946+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/605635853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b383ff0 0x563b5b386480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.946+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/605635853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37ee80 0x563b5b383980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.946+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/605635853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b37e490 0x563b5b37e8b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.946+0000 7f0fc1e20740 1 -- 172.21.15.70:0/605635853 >> 172.21.15.70:0/605635853 conn(0x563b5b379a00 msgr2=0x563b5b379e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.946+0000 7f0fc1e20740 1 -- 172.21.15.70:0/605635853 shutdown_connections 2024-04-01T01:41:01.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.946+0000 7f0fc1e20740 1 -- 172.21.15.70:0/605635853 wait complete. 2024-04-01T01:41:01.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.947+0000 7f0fc1e20740 1 -- start start 2024-04-01T01:41:01.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.947+0000 7f0fc1e20740 1 Processor -- start 2024-04-01T01:41:01.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.947+0000 7f0fc1e20740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b37e490 0x563b5b388a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.947+0000 7f0fc1e20740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37ee80 0x563b5b388f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.947+0000 7f0fc1e20740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b383ff0 0x563b5b38ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.947+0000 7f0fc1e20740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563b5b386fe0 con 0x563b5b37ee80 2024-04-01T01:41:01.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.947+0000 7f0fc1e20740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563b5b281470 con 0x563b5b37e490 2024-04-01T01:41:01.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.947+0000 7f0fc1e20740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563b5b387160 con 0x563b5b383ff0 2024-04-01T01:41:01.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.947+0000 7f0fc1e20740 10 client.0 _ll_register_callbacks cb 0x563b5b29ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:01.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.948+0000 7f0fbd7e6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b383ff0 0x563b5b38ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.948+0000 7f0fbd7e6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b383ff0 0x563b5b38ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43218/0 (socket says 172.21.15.70:43218) 2024-04-01T01:41:01.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.948+0000 7f0fbd7e6640 1 -- 172.21.15.70:0/703769353 learned_addr learned my addr 172.21.15.70:0/703769353 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:01.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.951+0000 7f0fbcfe5640 1 --2- 172.21.15.70:0/703769353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b37e490 0x563b5b388a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.953+0000 7f0fb7fff640 1 --2- 172.21.15.70:0/703769353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37ee80 0x563b5b388f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.954+0000 7f0fbd7e6640 1 -- 172.21.15.70:0/703769353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b37e490 msgr2=0x563b5b388a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.954+0000 7f0fbd7e6640 1 --2- 172.21.15.70:0/703769353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b37e490 0x563b5b388a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.954+0000 7f0fbd7e6640 1 -- 172.21.15.70:0/703769353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37ee80 msgr2=0x563b5b388f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:01.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.955+0000 7f0fbd7e6640 1 --2- 172.21.15.70:0/703769353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37ee80 0x563b5b388f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.955+0000 7f0fbd7e6640 1 -- 172.21.15.70:0/703769353 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b5aff73f0 con 0x563b5b383ff0 2024-04-01T01:41:01.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.957+0000 7f0fbd7e6640 1 --2- 172.21.15.70:0/703769353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b383ff0 0x563b5b38ff80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0010d70 tx=0x7f0fb0018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b96352a56a035fb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:01.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.960+0000 7f0f97fff640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:41:01.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.961+0000 7f0f97fff640 1 -- 172.21.15.70:0/703769353 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0004b20 con 0x563b5b383ff0 2024-04-01T01:41:01.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.962+0000 7f0f97fff640 1 -- 172.21.15.70:0/703769353 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f0fb0004cc0 con 0x563b5b383ff0 2024-04-01T01:41:01.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.962+0000 7f0fc1e20740 1 -- 172.21.15.70:0/703769353 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fa80b3070 con 0x563b5b383ff0 2024-04-01T01:41:01.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.963+0000 7f0f97fff640 1 -- 172.21.15.70:0/703769353 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0fb0030050 con 0x563b5b383ff0 2024-04-01T01:41:01.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.965+0000 7f0fc1e20740 10 client.27568 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:01.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.965+0000 7f0fc1e20740 1 -- 172.21.15.70:0/703769353 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563b5b0c17d0 con 0x563b5b383ff0 2024-04-01T01:41:01.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.966+0000 7f0f97fff640 1 -- 172.21.15.70:0/703769353 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0fa80b3070 con 0x563b5b383ff0 2024-04-01T01:41:01.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.966+0000 7f0f97fff640 1 -- 172.21.15.70:0/703769353 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f0fb002a4d0 con 0x563b5b383ff0 2024-04-01T01:41:01.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.967+0000 7f0fc1e20740 10 client.27568 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:01.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.967+0000 7f0fc1e20740 10 client.27568 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:41:01.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.967+0000 7f0fc1e20740 20 client.27568 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:01.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.967+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/703769353 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x563b5b394500 0x563b5b396950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:01.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.967+0000 7f0fc1e20740 4 client.27568 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:01.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.967+0000 7f0fc1e20740 1 -- 172.21.15.70:0/703769353 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b5b2f06f0 con 0x563b5b394500 2024-04-01T01:41:01.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.969+0000 7f0fbcfe5640 1 --2- 172.21.15.70:0/703769353 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x563b5b394500 0x563b5b396950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:01.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.981+0000 7f0fbcfe5640 1 --2- 172.21.15.70:0/703769353 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x563b5b394500 0x563b5b396950 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d139471249f069f3 server_cookie=a854ba6003dfb99b in_seq=0 out_seq=0 2024-04-01T01:41:01.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.981+0000 7f0f97fff640 10 client.27568 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:41:01.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.983+0000 7f0f97fff640 1 -- 172.21.15.70:0/703769353 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563b5b2f06f0 con 0x563b5b394500 2024-04-01T01:41:01.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.983+0000 7f0f97fff640 10 client.27568 handle_command_reply: tid=0 2024-04-01T01:41:01.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.984+0000 7f0fc1e20740 1 client.27568 shutdown 2024-04-01T01:41:01.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.985+0000 7f0fc1e20740 1 -- 172.21.15.70:0/703769353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b383ff0 msgr2=0x563b5b38ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:01.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.985+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/703769353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b383ff0 0x563b5b38ff80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0010d70 tx=0x7f0fb0018000 comp rx=0 tx=0).stop 2024-04-01T01:41:01.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.986+0000 7f0fc1e20740 1 -- 172.21.15.70:0/703769353 shutdown_connections 2024-04-01T01:41:01.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.986+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/703769353 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x563b5b394500 0x563b5b396950 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.986+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/703769353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b383ff0 0x563b5b38ff80 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.986+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/703769353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37ee80 0x563b5b388f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.986+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/703769353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b37e490 0x563b5b388a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:01.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.986+0000 7f0fc1e20740 1 -- 172.21.15.70:0/703769353 >> 172.21.15.70:0/703769353 conn(0x563b5b379a00 msgr2=0x563b5b378d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:01.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.988+0000 7f0f97fff640 0 client.27568 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:41:01.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.989+0000 7f0fc1e20740 1 -- 172.21.15.70:0/703769353 shutdown_connections 2024-04-01T01:41:01.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.991+0000 7f0fc1e20740 1 -- 172.21.15.70:0/703769353 wait complete. 2024-04-01T01:41:01.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:01.992+0000 7f0fc1e20740 20 client.27568 trim_cache size 0 max 16384 2024-04-01T01:41:02.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.029+0000 7f0fc1e20740 1 Processor -- start 2024-04-01T01:41:02.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.029+0000 7f0fc1e20740 1 -- start start 2024-04-01T01:41:02.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.030+0000 7f0fc1e20740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37e490 0x563b5b378bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.030+0000 7f0fc1e20740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b37ee80 0x563b5b38fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.030+0000 7f0fc1e20740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b383ff0 0x563b5b3875b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.030+0000 7f0fc1e20740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa400fe30 con 0x563b5b37e490 2024-04-01T01:41:02.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.030+0000 7f0fb7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b37ee80 0x563b5b38fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.030+0000 7f0fb7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b37ee80 0x563b5b38fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57400/0 (socket says 172.21.15.70:57400) 2024-04-01T01:41:02.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.030+0000 7f0fb7fff640 1 -- 172.21.15.70:0/2461189271 learned_addr learned my addr 172.21.15.70:0/2461189271 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:02.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.030+0000 7f0fc1e20740 1 -- 172.21.15.70:0/2461189271 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa4011970 con 0x563b5b37ee80 2024-04-01T01:41:02.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.030+0000 7f0fbd7e6640 1 --2- 172.21.15.70:0/2461189271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b383ff0 0x563b5b3875b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.030+0000 7f0fc1e20740 1 -- 172.21.15.70:0/2461189271 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa4010500 con 0x563b5b383ff0 2024-04-01T01:41:02.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.031+0000 7f0fb7fff640 1 -- 172.21.15.70:0/2461189271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b383ff0 msgr2=0x563b5b3875b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:02.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.031+0000 7f0fb7fff640 1 --2- 172.21.15.70:0/2461189271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b383ff0 0x563b5b3875b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.031+0000 7f0fb7fff640 1 -- 172.21.15.70:0/2461189271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37e490 msgr2=0x563b5b378bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:41:02.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.031+0000 7f0fb7fff640 1 --2- 172.21.15.70:0/2461189271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37e490 0x563b5b378bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.031+0000 7f0fb7fff640 1 -- 172.21.15.70:0/2461189271 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b5b37a770 con 0x563b5b37ee80 2024-04-01T01:41:02.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.031+0000 7f0fb7fff640 1 --2- 172.21.15.70:0/2461189271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b37ee80 0x563b5b38fea0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f0fac036040 tx=0x7f0fac0056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af1ff8937b7415f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:02.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.035+0000 7f0f9e7fc640 1 -- 172.21.15.70:0/2461189271 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac042070 con 0x563b5b37ee80 2024-04-01T01:41:02.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.035+0000 7f0f9e7fc640 1 -- 172.21.15.70:0/2461189271 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0fac02cb00 con 0x563b5b37ee80 2024-04-01T01:41:02.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.036+0000 7f0fc1e20740 1 -- 172.21.15.70:0/2461189271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b37ee80 msgr2=0x563b5b38fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:02.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.036+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/2461189271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b37ee80 0x563b5b38fea0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f0fac036040 tx=0x7f0fac0056d0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.036+0000 7f0fc1e20740 1 -- 172.21.15.70:0/2461189271 shutdown_connections 2024-04-01T01:41:02.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.036+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/2461189271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b383ff0 0x563b5b3875b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.036+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/2461189271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b37ee80 0x563b5b38fea0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.036+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/2461189271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37e490 0x563b5b378bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.036+0000 7f0fc1e20740 1 -- 172.21.15.70:0/2461189271 >> 172.21.15.70:0/2461189271 conn(0x563b5b379a00 msgr2=0x563b5b3784b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:02.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.040+0000 7f0fc1e20740 1 -- 172.21.15.70:0/2461189271 shutdown_connections 2024-04-01T01:41:02.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.043+0000 7f0fc1e20740 1 -- 172.21.15.70:0/2461189271 wait complete. 2024-04-01T01:41:02.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.043+0000 7f0fc1e20740 1 -- start start 2024-04-01T01:41:02.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.043+0000 7f0fc1e20740 1 Processor -- start 2024-04-01T01:41:02.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.044+0000 7f0fc1e20740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37e490 0x563b5b38d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.044+0000 7f0fc1e20740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b383ff0 0x563b5b38dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.044+0000 7f0fc1e20740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b38ecb0 0x563b5b38f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.044+0000 7f0fc1e20740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa40112a0 con 0x563b5b37e490 2024-04-01T01:41:02.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.044+0000 7f0fc1e20740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa4010bd0 con 0x563b5b38ecb0 2024-04-01T01:41:02.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.044+0000 7f0fc1e20740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0fa400de10 con 0x563b5b383ff0 2024-04-01T01:41:02.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.044+0000 7f0fc1e20740 10 client.0 _ll_register_callbacks cb 0x7f0fb0010c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:02.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.044+0000 7f0fbd7e6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b38ecb0 0x563b5b38f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.044+0000 7f0fbd7e6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b38ecb0 0x563b5b38f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57414/0 (socket says 172.21.15.70:57414) 2024-04-01T01:41:02.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.044+0000 7f0fbd7e6640 1 -- 172.21.15.70:0/130650720 learned_addr learned my addr 172.21.15.70:0/130650720 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:02.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.045+0000 7f0fbcfe5640 1 --2- 172.21.15.70:0/130650720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37e490 0x563b5b38d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.046+0000 7f0fbcfe5640 1 -- 172.21.15.70:0/130650720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b383ff0 msgr2=0x563b5b38dca0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:41:02.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.046+0000 7f0fbcfe5640 1 --2- 172.21.15.70:0/130650720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b383ff0 0x563b5b38dca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.046+0000 7f0fbcfe5640 1 -- 172.21.15.70:0/130650720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b38ecb0 msgr2=0x563b5b38f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:02.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.046+0000 7f0fbcfe5640 1 --2- 172.21.15.70:0/130650720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b38ecb0 0x563b5b38f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.046+0000 7f0fbcfe5640 1 -- 172.21.15.70:0/130650720 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b5b38f670 con 0x563b5b37e490 2024-04-01T01:41:02.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.048+0000 7f0fbcfe5640 1 --2- 172.21.15.70:0/130650720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37e490 0x563b5b38d760 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800be10 tx=0x7f0fa809a820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2562b9635367e943 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:02.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.048+0000 7f0f97fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:41:02.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.049+0000 7f0f97fff640 1 -- 172.21.15.70:0/130650720 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa80b8020 con 0x563b5b37e490 2024-04-01T01:41:02.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.049+0000 7f0f97fff640 1 -- 172.21.15.70:0/130650720 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f0fa80b9220 con 0x563b5b37e490 2024-04-01T01:41:02.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.052+0000 7f0fc1e20740 1 -- 172.21.15.70:0/130650720 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563b5b2eaa40 con 0x563b5b37e490 2024-04-01T01:41:02.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.053+0000 7f0f97fff640 1 -- 172.21.15.70:0/130650720 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0fa80ce1c0 con 0x563b5b37e490 2024-04-01T01:41:02.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.053+0000 7f0fc1e20740 10 client.17775 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:02.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.053+0000 7f0fc1e20740 1 -- 172.21.15.70:0/130650720 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563b5b3faf20 con 0x563b5b37e490 2024-04-01T01:41:02.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.054+0000 7f0f97fff640 1 -- 172.21.15.70:0/130650720 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x563b5b2eaa40 con 0x563b5b37e490 2024-04-01T01:41:02.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.054+0000 7f0f97fff640 1 -- 172.21.15.70:0/130650720 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f0fa8017b10 con 0x563b5b37e490 2024-04-01T01:41:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.054+0000 7f0fc1e20740 10 client.17775 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.054+0000 7f0fc1e20740 10 client.17775 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:41:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.054+0000 7f0fc1e20740 20 client.17775 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.054+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/130650720 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x563b5b3fb900 0x563b5b3fdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.054+0000 7f0fc1e20740 4 client.17775 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:41:02.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.054+0000 7f0fc1e20740 1 -- 172.21.15.70:0/130650720 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563b5b2686a0 con 0x563b5b3fb900 2024-04-01T01:41:02.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.061+0000 7f0fb7fff640 1 --2- 172.21.15.70:0/130650720 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x563b5b3fb900 0x563b5b3fdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.064+0000 7f0fb7fff640 1 --2- 172.21.15.70:0/130650720 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x563b5b3fb900 0x563b5b3fdd20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b93d0ac9482d05f2 server_cookie=59ba46fec22e6559 in_seq=0 out_seq=0 2024-04-01T01:41:02.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.064+0000 7f0f97fff640 10 client.17775 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:41:02.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.064+0000 7f0f97fff640 1 -- 172.21.15.70:0/130650720 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563b5b2686a0 con 0x563b5b3fb900 2024-04-01T01:41:02.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.064+0000 7f0f97fff640 10 client.17775 handle_command_reply: tid=0 2024-04-01T01:41:02.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.064+0000 7f0fc1e20740 1 client.17775 shutdown 2024-04-01T01:41:02.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.068+0000 7f0fc1e20740 1 -- 172.21.15.70:0/130650720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37e490 msgr2=0x563b5b38d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:02.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.068+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/130650720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37e490 0x563b5b38d760 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800be10 tx=0x7f0fa809a820 comp rx=0 tx=0).stop 2024-04-01T01:41:02.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.069+0000 7f0fc1e20740 1 -- 172.21.15.70:0/130650720 shutdown_connections 2024-04-01T01:41:02.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.069+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/130650720 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x563b5b3fb900 0x563b5b3fdd20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.069+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/130650720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563b5b38ecb0 0x563b5b38f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.069+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/130650720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b5b383ff0 0x563b5b38dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.069+0000 7f0fc1e20740 1 --2- 172.21.15.70:0/130650720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563b5b37e490 0x563b5b38d760 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.069+0000 7f0fc1e20740 1 -- 172.21.15.70:0/130650720 >> 172.21.15.70:0/130650720 conn(0x563b5b379a00 msgr2=0x563b5b37d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:02.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.069+0000 7f0f97fff640 0 client.17775 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:41:02.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.075+0000 7f0fc1e20740 1 -- 172.21.15.70:0/130650720 shutdown_connections 2024-04-01T01:41:02.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.079+0000 7f0fc1e20740 1 -- 172.21.15.70:0/130650720 wait complete. 2024-04-01T01:41:02.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.079+0000 7f0fc1e20740 20 client.17775 trim_cache size 0 max 16384 2024-04-01T01:41:02.083 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:41:02.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.085+0000 7f0fbf249640 1 -- 172.21.15.70:0/2904023564 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0fa00a26a0 msgr2=0x7f0fa00a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:02.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.085+0000 7f0fbf249640 1 --2- 172.21.15.70:0/2904023564 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0fa00a26a0 0x7f0fa00a4b60 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0005c40 tx=0x7f0fb0012620 comp rx=0 tx=0).stop 2024-04-01T01:41:02.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.085+0000 7f0fbf249640 1 -- 172.21.15.70:0/2904023564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb8108750 msgr2=0x7f0fb819e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:02.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.085+0000 7f0fbf249640 1 --2- 172.21.15.70:0/2904023564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb8108750 0x7f0fb819e130 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800b450 tx=0x7f0fa800b910 comp rx=0 tx=0).stop 2024-04-01T01:41:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.086+0000 7f0fbf249640 1 -- 172.21.15.70:0/2904023564 shutdown_connections 2024-04-01T01:41:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.086+0000 7f0fbf249640 1 --2- 172.21.15.70:0/2904023564 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0fa00a26a0 0x7f0fa00a4b60 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.086+0000 7f0fbf249640 1 --2- 172.21.15.70:0/2904023564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0fb8111e90 0x7f0fb81a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.086+0000 7f0fbf249640 1 --2- 172.21.15.70:0/2904023564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0fb810b0c0 0x7f0fb819e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.086+0000 7f0fbf249640 1 --2- 172.21.15.70:0/2904023564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb8108750 0x7f0fb819e130 secure :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800b450 tx=0x7f0fa800b910 comp rx=0 tx=0).stop 2024-04-01T01:41:02.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.086+0000 7f0fbf249640 1 -- 172.21.15.70:0/2904023564 >> 172.21.15.70:0/2904023564 conn(0x7f0fb806e5b0 msgr2=0x7f0fb8109bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:02.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.089+0000 7f0fbf249640 1 -- 172.21.15.70:0/2904023564 shutdown_connections 2024-04-01T01:41:02.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.090+0000 7f0fbf249640 1 -- 172.21.15.70:0/2904023564 wait complete. 2024-04-01T01:41:02.123 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:41:02.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.343+0000 7f222d36a640 1 -- 172.21.15.70:0/3745436669 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222402c620 con 0x7f2228109a50 2024-04-01T01:41:02.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.344+0000 7f222f36e640 1 -- 172.21.15.70:0/3745436669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2228109a50 msgr2=0x7f2228112330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:02.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.344+0000 7f222f36e640 1 --2- 172.21.15.70:0/3745436669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2228109a50 0x7f2228112330 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2224009900 tx=0x7f222402bc80 comp rx=0 tx=0).stop 2024-04-01T01:41:02.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.347+0000 7f222f36e640 1 -- 172.21.15.70:0/3745436669 shutdown_connections 2024-04-01T01:41:02.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.347+0000 7f222f36e640 1 --2- 172.21.15.70:0/3745436669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2228109a50 0x7f2228112330 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.347+0000 7f222f36e640 1 --2- 172.21.15.70:0/3745436669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f22281090b0 0x7f2228109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.347+0000 7f222f36e640 1 --2- 172.21.15.70:0/3745436669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2228108750 0x7f2228108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.347+0000 7f222f36e640 1 -- 172.21.15.70:0/3745436669 >> 172.21.15.70:0/3745436669 conn(0x7f222806e630 msgr2=0x7f2228070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:02.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.347+0000 7f222f36e640 1 -- 172.21.15.70:0/3745436669 shutdown_connections 2024-04-01T01:41:02.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.347+0000 7f222f36e640 1 -- 172.21.15.70:0/3745436669 wait complete. 2024-04-01T01:41:02.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.348+0000 7f222f36e640 1 Processor -- start 2024-04-01T01:41:02.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.348+0000 7f222f36e640 1 -- start start 2024-04-01T01:41:02.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.348+0000 7f222f36e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2228108750 0x7f22281161e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.348+0000 7f222f36e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22281090b0 0x7f2228116720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.348+0000 7f222f36e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2228109a50 0x7f2228112d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.348+0000 7f222f36e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2228117bc0 con 0x7f2228108750 2024-04-01T01:41:02.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.348+0000 7f222f36e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2228117a40 con 0x7f2228109a50 2024-04-01T01:41:02.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.348+0000 7f222f36e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2228117d40 con 0x7f22281090b0 2024-04-01T01:41:02.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.348+0000 7f222eb6d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2228109a50 0x7f2228112d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.348+0000 7f222db6b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22281090b0 0x7f2228116720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.348+0000 7f222db6b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22281090b0 0x7f2228116720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43246/0 (socket says 172.21.15.70:43246) 2024-04-01T01:41:02.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.348+0000 7f222db6b640 1 -- 172.21.15.70:0/2187031133 learned_addr learned my addr 172.21.15.70:0/2187031133 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:02.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.349+0000 7f222e36c640 1 --2- 172.21.15.70:0/2187031133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2228108750 0x7f22281161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.349+0000 7f222db6b640 1 -- 172.21.15.70:0/2187031133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2228109a50 msgr2=0x7f2228112d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:02.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.349+0000 7f222db6b640 1 --2- 172.21.15.70:0/2187031133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2228109a50 0x7f2228112d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.349+0000 7f222db6b640 1 -- 172.21.15.70:0/2187031133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2228108750 msgr2=0x7f22281161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:02.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.349+0000 7f222db6b640 1 --2- 172.21.15.70:0/2187031133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2228108750 0x7f22281161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.349+0000 7f222db6b640 1 -- 172.21.15.70:0/2187031133 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2228113610 con 0x7f22281090b0 2024-04-01T01:41:02.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.349+0000 7f222db6b640 1 --2- 172.21.15.70:0/2187031133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22281090b0 0x7f2228116720 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f221800d950 tx=0x7f221800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:02.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.350+0000 7f221f7fe640 1 -- 172.21.15.70:0/2187031133 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2218017070 con 0x7f22281090b0 2024-04-01T01:41:02.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.350+0000 7f221f7fe640 1 -- 172.21.15.70:0/2187031133 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2218009c30 con 0x7f22281090b0 2024-04-01T01:41:02.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.350+0000 7f221f7fe640 1 -- 172.21.15.70:0/2187031133 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221800f620 con 0x7f22281090b0 2024-04-01T01:41:02.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.350+0000 7f222f36e640 1 -- 172.21.15.70:0/2187031133 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22281ae600 con 0x7f22281090b0 2024-04-01T01:41:02.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.350+0000 7f222f36e640 1 -- 172.21.15.70:0/2187031133 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22281aea10 con 0x7f22281090b0 2024-04-01T01:41:02.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.354+0000 7f22331b9740 1 Processor -- start 2024-04-01T01:41:02.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.357+0000 7f221f7fe640 1 -- 172.21.15.70:0/2187031133 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f22180109a0 con 0x7f22281090b0 2024-04-01T01:41:02.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.357+0000 7f221f7fe640 1 --2- 172.21.15.70:0/2187031133 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22140a25f0 0x7f22140a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.358+0000 7f221f7fe640 1 -- 172.21.15.70:0/2187031133 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f22180cdf30 con 0x7f22281090b0 2024-04-01T01:41:02.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.358+0000 7f22331b9740 1 -- start start 2024-04-01T01:41:02.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.358+0000 7f22331b9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f67826a0 0x5643f6782ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.358+0000 7f22331b9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6783090 0x5643f6787b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.358+0000 7f22331b9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6788200 0x5643f678a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.358+0000 7f22331b9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5643f66e71b0 con 0x5643f6788200 2024-04-01T01:41:02.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.358+0000 7f22331b9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5643f668fc00 con 0x5643f6783090 2024-04-01T01:41:02.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.358+0000 7f22331b9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5643f66e3250 con 0x5643f67826a0 2024-04-01T01:41:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.359+0000 7f222eb6d640 1 --2- 172.21.15.70:0/2187031133 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22140a25f0 0x7f22140a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.360+0000 7f222eb6d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6788200 0x5643f678a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.360+0000 7f222eb6d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6788200 0x5643f678a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38722/0 (socket says 172.21.15.70:38722) 2024-04-01T01:41:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.360+0000 7f222eb6d640 1 -- 172.21.15.70:0/747316386 learned_addr learned my addr 172.21.15.70:0/747316386 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.360+0000 7f222db6b640 1 --2- 172.21.15.70:0/747316386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6783090 0x5643f6787b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.360+0000 7f222e36c640 1 --2- 172.21.15.70:0/747316386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f67826a0 0x5643f6782ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.361+0000 7f222eb6d640 1 -- 172.21.15.70:0/747316386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f67826a0 msgr2=0x5643f6782ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:02.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.361+0000 7f222eb6d640 1 --2- 172.21.15.70:0/747316386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f67826a0 0x5643f6782ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.361+0000 7f222eb6d640 1 -- 172.21.15.70:0/747316386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6783090 msgr2=0x5643f6787b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:02.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.361+0000 7f222eb6d640 1 --2- 172.21.15.70:0/747316386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6783090 0x5643f6787b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.361+0000 7f222eb6d640 1 -- 172.21.15.70:0/747316386 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643f66af3d0 con 0x5643f6788200 2024-04-01T01:41:02.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.362+0000 7f222eb6d640 1 --2- 172.21.15.70:0/2187031133 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22140a25f0 0x7f22140a4ab0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2224006010 tx=0x7f22240047d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:02.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.362+0000 7f222eb6d640 1 --2- 172.21.15.70:0/747316386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6788200 0x5643f678a690 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f222402c290 tx=0x7f2224049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=907c203f4864309b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:02.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.363+0000 7f221cbfe640 1 -- 172.21.15.70:0/747316386 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224052070 con 0x5643f6788200 2024-04-01T01:41:02.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.363+0000 7f221cbfe640 1 -- 172.21.15.70:0/747316386 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f222402d5a0 con 0x5643f6788200 2024-04-01T01:41:02.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.363+0000 7f221cbfe640 1 -- 172.21.15.70:0/747316386 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224005170 con 0x5643f6788200 2024-04-01T01:41:02.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.364+0000 7f22331b9740 1 -- 172.21.15.70:0/747316386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6788200 msgr2=0x5643f678a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:02.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.364+0000 7f22331b9740 1 --2- 172.21.15.70:0/747316386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6788200 0x5643f678a690 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f222402c290 tx=0x7f2224049000 comp rx=0 tx=0).stop 2024-04-01T01:41:02.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.364+0000 7f22331b9740 1 -- 172.21.15.70:0/747316386 shutdown_connections 2024-04-01T01:41:02.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.364+0000 7f22331b9740 1 --2- 172.21.15.70:0/747316386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6788200 0x5643f678a690 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.364+0000 7f22331b9740 1 --2- 172.21.15.70:0/747316386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6783090 0x5643f6787b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.364+0000 7f22331b9740 1 --2- 172.21.15.70:0/747316386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f67826a0 0x5643f6782ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.364+0000 7f22331b9740 1 -- 172.21.15.70:0/747316386 >> 172.21.15.70:0/747316386 conn(0x5643f677dc10 msgr2=0x5643f677e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:02.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.365+0000 7f22331b9740 1 -- 172.21.15.70:0/747316386 shutdown_connections 2024-04-01T01:41:02.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.367+0000 7f22331b9740 1 -- 172.21.15.70:0/747316386 wait complete. 2024-04-01T01:41:02.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.367+0000 7f22331b9740 1 -- start start 2024-04-01T01:41:02.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.367+0000 7f22331b9740 1 Processor -- start 2024-04-01T01:41:02.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.368+0000 7f22331b9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f67826a0 0x5643f678cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.368+0000 7f22331b9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f6783090 0x5643f678d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.368+0000 7f22331b9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6788200 0x5643f6794160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.368+0000 7f22331b9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5643f678b150 con 0x5643f67826a0 2024-04-01T01:41:02.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.368+0000 7f22331b9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5643f6685470 con 0x5643f6788200 2024-04-01T01:41:02.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.368+0000 7f22331b9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5643f678b2d0 con 0x5643f6783090 2024-04-01T01:41:02.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.368+0000 7f22331b9740 10 client.0 _ll_register_callbacks cb 0x5643f66a2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:02.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.369+0000 7f222e36c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f67826a0 0x5643f678cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.369+0000 7f222e36c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f67826a0 0x5643f678cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38726/0 (socket says 172.21.15.70:38726) 2024-04-01T01:41:02.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.369+0000 7f222e36c640 1 -- 172.21.15.70:0/3115049026 learned_addr learned my addr 172.21.15.70:0/3115049026 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:02.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.369+0000 7f222db6b640 1 --2- 172.21.15.70:0/3115049026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f6783090 0x5643f678d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.369+0000 7f222eb6d640 1 --2- 172.21.15.70:0/3115049026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6788200 0x5643f6794160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.371+0000 7f222e36c640 1 -- 172.21.15.70:0/3115049026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f6783090 msgr2=0x5643f678d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:02.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.371+0000 7f222e36c640 1 --2- 172.21.15.70:0/3115049026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f6783090 0x5643f678d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.371+0000 7f222e36c640 1 -- 172.21.15.70:0/3115049026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6788200 msgr2=0x5643f6794160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:02.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.371+0000 7f222e36c640 1 --2- 172.21.15.70:0/3115049026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6788200 0x5643f6794160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.371+0000 7f222e36c640 1 -- 172.21.15.70:0/3115049026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643f63fb3f0 con 0x5643f67826a0 2024-04-01T01:41:02.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.376+0000 7f222e36c640 1 --2- 172.21.15.70:0/3115049026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f67826a0 0x5643f678cbe0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f2220011a40 tx=0x7f2220011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b4465c641f0dd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:02.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.376+0000 7f2211ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:41:02.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.377+0000 7f2211ffb640 1 -- 172.21.15.70:0/3115049026 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222000fcc0 con 0x5643f67826a0 2024-04-01T01:41:02.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.377+0000 7f2211ffb640 1 -- 172.21.15.70:0/3115049026 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f222001b3d0 con 0x5643f67826a0 2024-04-01T01:41:02.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.377+0000 7f22331b9740 1 -- 172.21.15.70:0/3115049026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f222402d5a0 con 0x5643f67826a0 2024-04-01T01:41:02.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.378+0000 7f2211ffb640 1 -- 172.21.15.70:0/3115049026 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f222000fe60 con 0x5643f67826a0 2024-04-01T01:41:02.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.379+0000 7f22331b9740 10 client.17787 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:02.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.379+0000 7f22331b9740 1 -- 172.21.15.70:0/3115049026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5643f64c57d0 con 0x5643f67826a0 2024-04-01T01:41:02.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.379+0000 7f2211ffb640 1 -- 172.21.15.70:0/3115049026 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f222402d5a0 con 0x5643f67826a0 2024-04-01T01:41:02.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.380+0000 7f2211ffb640 1 -- 172.21.15.70:0/3115049026 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f222001c900 con 0x5643f67826a0 2024-04-01T01:41:02.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.380+0000 7f22331b9740 10 client.17787 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:02.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.380+0000 7f22331b9740 10 client.17787 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:02.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.380+0000 7f22331b9740 20 client.17787 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:02.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.380+0000 7f22331b9740 1 --2- 172.21.15.70:0/3115049026 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5643f6798b10 0x5643f679afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.380+0000 7f22331b9740 4 client.17787 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:02.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.380+0000 7f22331b9740 1 -- 172.21.15.70:0/3115049026 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643f66f46f0 con 0x5643f6798b10 2024-04-01T01:41:02.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.381+0000 7f222db6b640 1 --2- 172.21.15.70:0/3115049026 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5643f6798b10 0x5643f679afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.383+0000 7f222db6b640 1 --2- 172.21.15.70:0/3115049026 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5643f6798b10 0x5643f679afb0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a61453516885819b server_cookie=ff1477f0b54ed673 in_seq=0 out_seq=0 2024-04-01T01:41:02.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.383+0000 7f2211ffb640 10 client.17787 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:02.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.384+0000 7f2211ffb640 1 -- 172.21.15.70:0/3115049026 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5643f66f46f0 con 0x5643f6798b10 2024-04-01T01:41:02.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.384+0000 7f2211ffb640 10 client.17787 handle_command_reply: tid=0 2024-04-01T01:41:02.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.386+0000 7f22331b9740 1 client.17787 shutdown 2024-04-01T01:41:02.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.387+0000 7f22331b9740 1 -- 172.21.15.70:0/3115049026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f67826a0 msgr2=0x5643f678cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:02.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.387+0000 7f22331b9740 1 --2- 172.21.15.70:0/3115049026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f67826a0 0x5643f678cbe0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f2220011a40 tx=0x7f2220011f00 comp rx=0 tx=0).stop 2024-04-01T01:41:02.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.390+0000 7f22331b9740 1 -- 172.21.15.70:0/3115049026 shutdown_connections 2024-04-01T01:41:02.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.390+0000 7f22331b9740 1 --2- 172.21.15.70:0/3115049026 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5643f6798b10 0x5643f679afb0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.390+0000 7f22331b9740 1 --2- 172.21.15.70:0/3115049026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6788200 0x5643f6794160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.390+0000 7f22331b9740 1 --2- 172.21.15.70:0/3115049026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f6783090 0x5643f678d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.390+0000 7f22331b9740 1 --2- 172.21.15.70:0/3115049026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f67826a0 0x5643f678cbe0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.390+0000 7f22331b9740 1 -- 172.21.15.70:0/3115049026 >> 172.21.15.70:0/3115049026 conn(0x5643f677dc10 msgr2=0x5643f677cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:02.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.390+0000 7f2211ffb640 0 client.17787 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:02.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.390+0000 7f22331b9740 1 -- 172.21.15.70:0/3115049026 shutdown_connections 2024-04-01T01:41:02.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.392+0000 7f22331b9740 1 -- 172.21.15.70:0/3115049026 wait complete. 2024-04-01T01:41:02.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.394+0000 7f22331b9740 20 client.17787 trim_cache size 0 max 16384 2024-04-01T01:41:02.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.430+0000 7f22331b9740 1 Processor -- start 2024-04-01T01:41:02.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.431+0000 7f22331b9740 1 -- start start 2024-04-01T01:41:02.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.433+0000 7f22331b9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f67826a0 0x5643f677dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.433+0000 7f22331b9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f6783090 0x5643f6794120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.433+0000 7f22331b9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6788200 0x5643f677c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.433+0000 7f222db6b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f6783090 0x5643f6794120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.433+0000 7f222db6b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f6783090 0x5643f6794120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43270/0 (socket says 172.21.15.70:43270) 2024-04-01T01:41:02.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.433+0000 7f222db6b640 1 -- 172.21.15.70:0/1309910492 learned_addr learned my addr 172.21.15.70:0/1309910492 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:02.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.433+0000 7f22331b9740 1 -- 172.21.15.70:0/1309910492 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f22281b9020 con 0x5643f6788200 2024-04-01T01:41:02.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.433+0000 7f22331b9740 1 -- 172.21.15.70:0/1309910492 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f22281bab60 con 0x5643f67826a0 2024-04-01T01:41:02.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.433+0000 7f22331b9740 1 -- 172.21.15.70:0/1309910492 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f22281b96f0 con 0x5643f6783090 2024-04-01T01:41:02.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.434+0000 7f222eb6d640 1 --2- 172.21.15.70:0/1309910492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6788200 0x5643f677c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.434+0000 7f222e36c640 1 --2- 172.21.15.70:0/1309910492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f67826a0 0x5643f677dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.434+0000 7f222eb6d640 1 -- 172.21.15.70:0/1309910492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f6783090 msgr2=0x5643f6794120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:02.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.434+0000 7f222eb6d640 1 --2- 172.21.15.70:0/1309910492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f6783090 0x5643f6794120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.434+0000 7f222eb6d640 1 -- 172.21.15.70:0/1309910492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f67826a0 msgr2=0x5643f677dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:02.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.434+0000 7f222eb6d640 1 --2- 172.21.15.70:0/1309910492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f67826a0 0x5643f677dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.434+0000 7f222eb6d640 1 -- 172.21.15.70:0/1309910492 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643f676ee60 con 0x5643f6788200 2024-04-01T01:41:02.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.435+0000 7f222eb6d640 1 --2- 172.21.15.70:0/1309910492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6788200 0x5643f677c700 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f2224009520 tx=0x7f2224004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8d347a22ac4a43 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:02.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.435+0000 7f2213fff640 1 -- 172.21.15.70:0/1309910492 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222405b070 con 0x5643f6788200 2024-04-01T01:41:02.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.435+0000 7f2213fff640 1 -- 172.21.15.70:0/1309910492 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f222402cc10 con 0x5643f6788200 2024-04-01T01:41:02.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.436+0000 7f2213fff640 1 -- 172.21.15.70:0/1309910492 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224039040 con 0x5643f6788200 2024-04-01T01:41:02.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.437+0000 7f22331b9740 1 -- 172.21.15.70:0/1309910492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6788200 msgr2=0x5643f677c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:02.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.437+0000 7f22331b9740 1 --2- 172.21.15.70:0/1309910492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6788200 0x5643f677c700 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f2224009520 tx=0x7f2224004be0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.437+0000 7f22331b9740 1 -- 172.21.15.70:0/1309910492 shutdown_connections 2024-04-01T01:41:02.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.437+0000 7f22331b9740 1 --2- 172.21.15.70:0/1309910492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6788200 0x5643f677c700 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.437+0000 7f22331b9740 1 --2- 172.21.15.70:0/1309910492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f6783090 0x5643f6794120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.437+0000 7f22331b9740 1 --2- 172.21.15.70:0/1309910492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f67826a0 0x5643f677dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.437+0000 7f22331b9740 1 -- 172.21.15.70:0/1309910492 >> 172.21.15.70:0/1309910492 conn(0x5643f677dc10 msgr2=0x5643f6783a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.438+0000 7f22331b9740 1 -- 172.21.15.70:0/1309910492 shutdown_connections 2024-04-01T01:41:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.438+0000 7f22331b9740 1 -- 172.21.15.70:0/1309910492 wait complete. 2024-04-01T01:41:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.438+0000 7f22331b9740 1 -- start start 2024-04-01T01:41:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.438+0000 7f22331b9740 1 Processor -- start 2024-04-01T01:41:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.439+0000 7f22331b9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f67826a0 0x5643f6794120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.439+0000 7f22331b9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6783090 0x5643f6790bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.439+0000 7f22331b9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6788200 0x5643f67911d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.439+0000 7f22331b9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f22281ba490 con 0x5643f6783090 2024-04-01T01:41:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.439+0000 7f22331b9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f22281b9dc0 con 0x5643f6788200 2024-04-01T01:41:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.439+0000 7f222eb6d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6788200 0x5643f67911d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.439+0000 7f222eb6d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6788200 0x5643f67911d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57458/0 (socket says 172.21.15.70:57458) 2024-04-01T01:41:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.439+0000 7f222eb6d640 1 -- 172.21.15.70:0/3909886035 learned_addr learned my addr 172.21.15.70:0/3909886035 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.439+0000 7f222db6b640 1 --2- 172.21.15.70:0/3909886035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6783090 0x5643f6790bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.439+0000 7f22331b9740 1 -- 172.21.15.70:0/3909886035 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f22281b7000 con 0x5643f67826a0 2024-04-01T01:41:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.440+0000 7f22331b9740 10 client.0 _ll_register_callbacks cb 0x7f2220011650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.440+0000 7f222eb6d640 1 -- 172.21.15.70:0/3909886035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f67826a0 msgr2=0x5643f6794120 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:41:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.440+0000 7f222eb6d640 1 --2- 172.21.15.70:0/3909886035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f67826a0 0x5643f6794120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.440+0000 7f222eb6d640 1 -- 172.21.15.70:0/3909886035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6783090 msgr2=0x5643f6790bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.440+0000 7f222eb6d640 1 --2- 172.21.15.70:0/3909886035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6783090 0x5643f6790bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.440+0000 7f222eb6d640 1 -- 172.21.15.70:0/3909886035 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643f67fcf60 con 0x5643f6788200 2024-04-01T01:41:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.440+0000 7f222eb6d640 1 --2- 172.21.15.70:0/3909886035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6788200 0x5643f67911d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2224005960 tx=0x7f2224005990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85a5045312ec0d9f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.440+0000 7f2211ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.440+0000 7f2211ffb640 1 -- 172.21.15.70:0/3909886035 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224063060 con 0x5643f6788200 2024-04-01T01:41:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.440+0000 7f2211ffb640 1 -- 172.21.15.70:0/3909886035 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f22240055f0 con 0x5643f6788200 2024-04-01T01:41:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.440+0000 7f22331b9740 1 -- 172.21.15.70:0/3909886035 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f222402cc10 con 0x5643f6788200 2024-04-01T01:41:02.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.441+0000 7f2211ffb640 1 -- 172.21.15.70:0/3909886035 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f222405b070 con 0x5643f6788200 2024-04-01T01:41:02.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.441+0000 7f22331b9740 10 client.27746 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:02.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.441+0000 7f22331b9740 1 -- 172.21.15.70:0/3909886035 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5643f67fe240 con 0x5643f6788200 2024-04-01T01:41:02.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.442+0000 7f2211ffb640 1 -- 172.21.15.70:0/3909886035 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f222402c9b0 con 0x5643f6788200 2024-04-01T01:41:02.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.442+0000 7f2211ffb640 1 -- 172.21.15.70:0/3909886035 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f2224055070 con 0x5643f6788200 2024-04-01T01:41:02.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.442+0000 7f22331b9740 10 client.27746 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:02.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.442+0000 7f22331b9740 10 client.27746 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:02.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.442+0000 7f22331b9740 20 client.27746 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:02.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.442+0000 7f22331b9740 1 --2- 172.21.15.70:0/3909886035 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5643f67fe690 0x5643f6800a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:02.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.442+0000 7f22331b9740 4 client.27746 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:41:02.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.442+0000 7f22331b9740 1 -- 172.21.15.70:0/3909886035 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f222402cc10 con 0x5643f67fe690 2024-04-01T01:41:02.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.442+0000 7f222e36c640 1 --2- 172.21.15.70:0/3909886035 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5643f67fe690 0x5643f6800a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:02.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.444+0000 7f222e36c640 1 --2- 172.21.15.70:0/3909886035 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5643f67fe690 0x5643f6800a80 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90ab22626412b7e9 server_cookie=6ae62fa130eadd6f in_seq=0 out_seq=0 2024-04-01T01:41:02.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.444+0000 7f2211ffb640 10 client.27746 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:02.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.444+0000 7f2211ffb640 1 -- 172.21.15.70:0/3909886035 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f222402cc10 con 0x5643f67fe690 2024-04-01T01:41:02.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.444+0000 7f2211ffb640 10 client.27746 handle_command_reply: tid=0 2024-04-01T01:41:02.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.444+0000 7f22331b9740 1 client.27746 shutdown 2024-04-01T01:41:02.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.444+0000 7f22331b9740 1 -- 172.21.15.70:0/3909886035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6788200 msgr2=0x5643f67911d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:02.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.444+0000 7f22331b9740 1 --2- 172.21.15.70:0/3909886035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6788200 0x5643f67911d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2224005960 tx=0x7f2224005990 comp rx=0 tx=0).stop 2024-04-01T01:41:02.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.445+0000 7f22331b9740 1 -- 172.21.15.70:0/3909886035 shutdown_connections 2024-04-01T01:41:02.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.445+0000 7f22331b9740 1 --2- 172.21.15.70:0/3909886035 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5643f67fe690 0x5643f6800a80 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.445+0000 7f22331b9740 1 --2- 172.21.15.70:0/3909886035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643f6788200 0x5643f67911d0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.445+0000 7f22331b9740 1 --2- 172.21.15.70:0/3909886035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643f6783090 0x5643f6790bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.445+0000 7f22331b9740 1 --2- 172.21.15.70:0/3909886035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643f67826a0 0x5643f6794120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.445+0000 7f22331b9740 1 -- 172.21.15.70:0/3909886035 >> 172.21.15.70:0/3909886035 conn(0x5643f677dc10 msgr2=0x5643f677dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:02.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.445+0000 7f2211ffb640 0 client.27746 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:02.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.445+0000 7f22331b9740 1 -- 172.21.15.70:0/3909886035 shutdown_connections 2024-04-01T01:41:02.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.446+0000 7f22331b9740 1 -- 172.21.15.70:0/3909886035 wait complete. 2024-04-01T01:41:02.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.446+0000 7f22331b9740 20 client.27746 trim_cache size 0 max 16384 2024-04-01T01:41:02.450 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:41:02.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.448+0000 7f222f36e640 1 -- 172.21.15.70:0/2187031133 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22140a25f0 msgr2=0x7f22140a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:02.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.448+0000 7f222f36e640 1 --2- 172.21.15.70:0/2187031133 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22140a25f0 0x7f22140a4ab0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2224006010 tx=0x7f22240047d0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.448+0000 7f222f36e640 1 -- 172.21.15.70:0/2187031133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22281090b0 msgr2=0x7f2228116720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:02.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.448+0000 7f222f36e640 1 --2- 172.21.15.70:0/2187031133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22281090b0 0x7f2228116720 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f221800d950 tx=0x7f221800de10 comp rx=0 tx=0).stop 2024-04-01T01:41:02.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.449+0000 7f222f36e640 1 -- 172.21.15.70:0/2187031133 shutdown_connections 2024-04-01T01:41:02.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.449+0000 7f222f36e640 1 --2- 172.21.15.70:0/2187031133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2228109a50 0x7f2228112d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.449+0000 7f222f36e640 1 --2- 172.21.15.70:0/2187031133 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22140a25f0 0x7f22140a4ab0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.449+0000 7f222f36e640 1 --2- 172.21.15.70:0/2187031133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22281090b0 0x7f2228116720 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.449+0000 7f222f36e640 1 --2- 172.21.15.70:0/2187031133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2228108750 0x7f22281161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:02.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.449+0000 7f222f36e640 1 -- 172.21.15.70:0/2187031133 >> 172.21.15.70:0/2187031133 conn(0x7f222806e630 msgr2=0x7f222810e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:02.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.450+0000 7f222f36e640 1 -- 172.21.15.70:0/2187031133 shutdown_connections 2024-04-01T01:41:02.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:02.450+0000 7f222f36e640 1 -- 172.21.15.70:0/2187031133 wait complete. 2024-04-01T01:41:03.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:02 smithi189 ceph-mon[35753]: pgmap v586: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-01T01:41:03.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:02 smithi070 ceph-mon[32179]: pgmap v586: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-01T01:41:03.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:02 smithi081 ceph-mon[35142]: pgmap v586: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-01T01:41:03.467 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:41:03.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:41:03.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:41:03.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.659+0000 7f3a6359e640 1 -- 172.21.15.70:0/153195639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c007580 msgr2=0x7f3a5c007a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:03.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.659+0000 7f3a6359e640 1 --2- 172.21.15.70:0/153195639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c007580 0x7f3a5c007a00 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009930 tx=0x7f3a5402bc30 comp rx=0 tx=0).stop 2024-04-01T01:41:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.659+0000 7f3a6359e640 1 -- 172.21.15.70:0/153195639 shutdown_connections 2024-04-01T01:41:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.659+0000 7f3a6359e640 1 --2- 172.21.15.70:0/153195639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5c0a2340 0x7f3a5c0a4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.659+0000 7f3a6359e640 1 --2- 172.21.15.70:0/153195639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c007580 0x7f3a5c007a00 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.659+0000 7f3a6359e640 1 --2- 172.21.15.70:0/153195639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a5c008b60 0x7f3a5c006fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.659+0000 7f3a6359e640 1 -- 172.21.15.70:0/153195639 >> 172.21.15.70:0/153195639 conn(0x7f3a5c0912e0 msgr2=0x7f3a5c093720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.659+0000 7f3a6359e640 1 -- 172.21.15.70:0/153195639 shutdown_connections 2024-04-01T01:41:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.659+0000 7f3a6359e640 1 -- 172.21.15.70:0/153195639 wait complete. 2024-04-01T01:41:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.660+0000 7f3a6359e640 1 Processor -- start 2024-04-01T01:41:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.660+0000 7f3a6359e640 1 -- start start 2024-04-01T01:41:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.660+0000 7f3a6359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a5c007580 0x7f3a5c132390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.660+0000 7f3a6359e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5c008b60 0x7f3a5c1328d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.661+0000 7f3a6359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c0a2340 0x7f3a5c139940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.661+0000 7f3a6259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a5c007580 0x7f3a5c132390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.661+0000 7f3a6259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a5c007580 0x7f3a5c132390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38776/0 (socket says 172.21.15.70:38776) 2024-04-01T01:41:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.661+0000 7f3a6359e640 1 -- 172.21.15.70:0/3671737024 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c0a7ec0 con 0x7f3a5c007580 2024-04-01T01:41:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.661+0000 7f3a6259c640 1 -- 172.21.15.70:0/3671737024 learned_addr learned my addr 172.21.15.70:0/3671737024 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.661+0000 7f3a6359e640 1 -- 172.21.15.70:0/3671737024 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c0a7d40 con 0x7f3a5c0a2340 2024-04-01T01:41:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.661+0000 7f3a6359e640 1 -- 172.21.15.70:0/3671737024 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c0a8040 con 0x7f3a5c008b60 2024-04-01T01:41:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.661+0000 7f3a62d9d640 1 --2- 172.21.15.70:0/3671737024 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c0a2340 0x7f3a5c139940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.661+0000 7f3a61d9b640 1 --2- 172.21.15.70:0/3671737024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5c008b60 0x7f3a5c1328d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.662+0000 7f3a62d9d640 1 -- 172.21.15.70:0/3671737024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5c008b60 msgr2=0x7f3a5c1328d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.662+0000 7f3a62d9d640 1 --2- 172.21.15.70:0/3671737024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5c008b60 0x7f3a5c1328d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.662+0000 7f3a62d9d640 1 -- 172.21.15.70:0/3671737024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a5c007580 msgr2=0x7f3a5c132390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.662+0000 7f3a62d9d640 1 --2- 172.21.15.70:0/3671737024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a5c007580 0x7f3a5c132390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.662+0000 7f3a62d9d640 1 -- 172.21.15.70:0/3671737024 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a5c139fb0 con 0x7f3a5c0a2340 2024-04-01T01:41:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.662+0000 7f3a6259c640 1 --2- 172.21.15.70:0/3671737024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a5c007580 0x7f3a5c132390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:41:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.662+0000 7f3a62d9d640 1 --2- 172.21.15.70:0/3671737024 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c0a2340 0x7f3a5c139940 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3a64067420 tx=0x7f3a6407aee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.662+0000 7f3a537fe640 1 -- 172.21.15.70:0/3671737024 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a64083020 con 0x7f3a5c0a2340 2024-04-01T01:41:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.662+0000 7f3a537fe640 1 -- 172.21.15.70:0/3671737024 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a64078c20 con 0x7f3a5c0a2340 2024-04-01T01:41:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.662+0000 7f3a6359e640 1 -- 172.21.15.70:0/3671737024 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a5c13a240 con 0x7f3a5c0a2340 2024-04-01T01:41:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.662+0000 7f3a537fe640 1 -- 172.21.15.70:0/3671737024 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a64069530 con 0x7f3a5c0a2340 2024-04-01T01:41:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.662+0000 7f3a6359e640 1 -- 172.21.15.70:0/3671737024 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a5c13a710 con 0x7f3a5c0a2340 2024-04-01T01:41:03.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.666+0000 7f3a6b5d2740 1 Processor -- start 2024-04-01T01:41:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.666+0000 7f3a537fe640 1 -- 172.21.15.70:0/3671737024 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f3a64086070 con 0x7f3a5c0a2340 2024-04-01T01:41:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.667+0000 7f3a6b5d2740 1 -- start start 2024-04-01T01:41:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.667+0000 7f3a6b5d2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c0297030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.667+0000 7f3a6b5d2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c0297600 0x5652c029c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.667+0000 7f3a6b5d2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 0x5652c029ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.667+0000 7f3a6b5d2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5652c01980b0 con 0x5652c0296c10 2024-04-01T01:41:03.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.667+0000 7f3a6b5d2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5652c01faea0 con 0x5652c029c770 2024-04-01T01:41:03.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.667+0000 7f3a6b5d2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5652c01a3c00 con 0x5652c0297600 2024-04-01T01:41:03.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.667+0000 7f3a61d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c0297030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.667+0000 7f3a61d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c0297030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38788/0 (socket says 172.21.15.70:38788) 2024-04-01T01:41:03.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.667+0000 7f3a61d9b640 1 -- 172.21.15.70:0/3069643690 learned_addr learned my addr 172.21.15.70:0/3069643690 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:03.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.667+0000 7f3a61d9b640 1 --2- 172.21.15.70:0/3069643690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 0x5652c029ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.668+0000 7f3a61d9b640 1 -- 172.21.15.70:0/3069643690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c0297600 msgr2=0x5652c029c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:03.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.668+0000 7f3a61d9b640 1 --2- 172.21.15.70:0/3069643690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c0297600 0x5652c029c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.668+0000 7f3a61d9b640 1 -- 172.21.15.70:0/3069643690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 msgr2=0x5652c029ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:03.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.668+0000 7f3a61d9b640 1 --2- 172.21.15.70:0/3069643690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 0x5652c029ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.668+0000 7f3a61d9b640 1 -- 172.21.15.70:0/3069643690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652bffd77e0 con 0x5652c0296c10 2024-04-01T01:41:03.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.668+0000 7f3a61d9b640 1 --2- 172.21.15.70:0/3069643690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c0297030 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009930 tx=0x7f3a5402c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8715c6cd31348e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:03.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.669+0000 7f3a537fe640 1 --2- 172.21.15.70:0/3671737024 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3a4c0a2570 0x7f3a4c0a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.669+0000 7f3a537fe640 1 -- 172.21.15.70:0/3671737024 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3a641392b0 con 0x7f3a5c0a2340 2024-04-01T01:41:03.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.669+0000 7f3a50ff9640 1 -- 172.21.15.70:0/3069643690 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a54004030 con 0x5652c0296c10 2024-04-01T01:41:03.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.669+0000 7f3a6259c640 1 --2- 172.21.15.70:0/3671737024 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3a4c0a2570 0x7f3a4c0a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.669+0000 7f3a50ff9640 1 -- 172.21.15.70:0/3069643690 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a54048070 con 0x5652c0296c10 2024-04-01T01:41:03.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.669+0000 7f3a50ff9640 1 -- 172.21.15.70:0/3069643690 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a54004920 con 0x5652c0296c10 2024-04-01T01:41:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.670+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/3069643690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 msgr2=0x5652c0297030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.670+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/3069643690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c0297030 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009930 tx=0x7f3a5402c670 comp rx=0 tx=0).stop 2024-04-01T01:41:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.671+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/3069643690 shutdown_connections 2024-04-01T01:41:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.671+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/3069643690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 0x5652c029ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.671+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/3069643690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c0297600 0x5652c029c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.671+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/3069643690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c0297030 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.671+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/3069643690 >> 172.21.15.70:0/3069643690 conn(0x5652c0292180 msgr2=0x5652c0292580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.671+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/3069643690 shutdown_connections 2024-04-01T01:41:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.671+0000 7f3a6259c640 1 --2- 172.21.15.70:0/3671737024 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3a4c0a2570 0x7f3a4c0a4a30 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3a58009fd0 tx=0x7f3a58009780 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.671+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/3069643690 wait complete. 2024-04-01T01:41:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.672+0000 7f3a6b5d2740 1 -- start start 2024-04-01T01:41:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.672+0000 7f3a6b5d2740 1 Processor -- start 2024-04-01T01:41:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.672+0000 7f3a6b5d2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c02a1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.672+0000 7f3a6b5d2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c0297600 0x5652c02a1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.673+0000 7f3a6b5d2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c029c770 0x5652c02a8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.673+0000 7f3a61d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c0297600 0x5652c02a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.673+0000 7f3a61d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c0297600 0x5652c02a1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57506/0 (socket says 172.21.15.70:57506) 2024-04-01T01:41:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.673+0000 7f3a6259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c02a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.673+0000 7f3a62d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c029c770 0x5652c02a8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.673+0000 7f3a62d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c029c770 0x5652c02a8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43312/0 (socket says 172.21.15.70:43312) 2024-04-01T01:41:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.673+0000 7f3a61d9b640 1 -- 172.21.15.70:0/1922406005 learned_addr learned my addr 172.21.15.70:0/1922406005 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.673+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/1922406005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5652c01993d0 con 0x5652c0296c10 2024-04-01T01:41:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.673+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/1922406005 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5652c01f6f40 con 0x5652c0297600 2024-04-01T01:41:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.673+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/1922406005 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5652c029f710 con 0x5652c029c770 2024-04-01T01:41:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.674+0000 7f3a61d9b640 1 -- 172.21.15.70:0/1922406005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c029c770 msgr2=0x5652c02a8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.674+0000 7f3a61d9b640 1 --2- 172.21.15.70:0/1922406005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c029c770 0x5652c02a8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.674+0000 7f3a61d9b640 1 -- 172.21.15.70:0/1922406005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 msgr2=0x5652c02a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.674+0000 7f3a61d9b640 1 --2- 172.21.15.70:0/1922406005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c02a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.674+0000 7f3a61d9b640 1 -- 172.21.15.70:0/1922406005 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652c0037f70 con 0x5652c0297600 2024-04-01T01:41:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.674+0000 7f3a6259c640 1 --2- 172.21.15.70:0/1922406005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c02a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:41:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.674+0000 7f3a61d9b640 1 --2- 172.21.15.70:0/1922406005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c0297600 0x5652c02a1540 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009230 tx=0x7f3a5402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10511f123a598397 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.674+0000 7f3a49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.674+0000 7f3a49ffb640 1 -- 172.21.15.70:0/1922406005 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a54053020 con 0x5652c0297600 2024-04-01T01:41:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.674+0000 7f3a49ffb640 1 -- 172.21.15.70:0/1922406005 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3a54003db0 con 0x5652c0297600 2024-04-01T01:41:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.674+0000 7f3a6b5d2740 10 client.0 _ll_register_callbacks cb 0x5652c01b6b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.674+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/1922406005 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a54048070 con 0x5652c0297600 2024-04-01T01:41:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.675+0000 7f3a49ffb640 1 -- 172.21.15.70:0/1922406005 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3a54043450 con 0x5652c0297600 2024-04-01T01:41:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.675+0000 7f3a6b5d2740 10 client.27758 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.675+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/1922406005 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=31}) -- 0x5652bfecf170 con 0x5652c0297600 2024-04-01T01:41:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.675+0000 7f3a49ffb640 1 -- 172.21.15.70:0/1922406005 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f3a540435f0 con 0x5652c0297600 2024-04-01T01:41:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.675+0000 7f3a6b5d2740 10 client.27758 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.675+0000 7f3a6b5d2740 10 client.27758 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:03.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.675+0000 7f3a6b5d2740 20 client.27758 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:03.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.676+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/1922406005 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5652c02ac410 0x5652c02ae800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.676+0000 7f3a6b5d2740 4 client.27758 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:03.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.676+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/1922406005 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3a54048070 con 0x5652c02ac410 2024-04-01T01:41:03.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.676+0000 7f3a6259c640 1 --2- 172.21.15.70:0/1922406005 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5652c02ac410 0x5652c02ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.677+0000 7f3a49ffb640 10 client.27758 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:03.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.677+0000 7f3a6259c640 1 --2- 172.21.15.70:0/1922406005 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5652c02ac410 0x5652c02ae800 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77fbcd4b08d87bff server_cookie=b3c6c4c02e25e518 in_seq=0 out_seq=0 2024-04-01T01:41:03.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.679+0000 7f3a49ffb640 1 -- 172.21.15.70:0/1922406005 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f3a54048070 con 0x5652c02ac410 2024-04-01T01:41:03.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.679+0000 7f3a49ffb640 10 client.27758 handle_command_reply: tid=0 2024-04-01T01:41:03.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.679+0000 7f3a6b5d2740 1 client.27758 shutdown 2024-04-01T01:41:03.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.679+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/1922406005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c0297600 msgr2=0x5652c02a1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:03.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.679+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/1922406005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c0297600 0x5652c02a1540 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009230 tx=0x7f3a5402bc30 comp rx=0 tx=0).stop 2024-04-01T01:41:03.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.679+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/1922406005 shutdown_connections 2024-04-01T01:41:03.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.679+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/1922406005 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5652c02ac410 0x5652c02ae800 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.679+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/1922406005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c029c770 0x5652c02a8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.679+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/1922406005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c0297600 0x5652c02a1540 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.679+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/1922406005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c02a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.679+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/1922406005 >> 172.21.15.70:0/1922406005 conn(0x5652c0292180 msgr2=0x5652c02913a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.679+0000 7f3a49ffb640 0 client.27758 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.680+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/1922406005 shutdown_connections 2024-04-01T01:41:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.680+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/1922406005 wait complete. 2024-04-01T01:41:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.680+0000 7f3a6b5d2740 20 client.27758 trim_cache size 0 max 16384 2024-04-01T01:41:03.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.716+0000 7f3a6b5d2740 1 Processor -- start 2024-04-01T01:41:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.717+0000 7f3a6b5d2740 1 -- start start 2024-04-01T01:41:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.717+0000 7f3a6b5d2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c0295af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.717+0000 7f3a6b5d2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c0297600 0x5652c02a8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.717+0000 7f3a6b5d2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 0x5652c02b3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.717+0000 7f3a6b5d2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c144c80 con 0x5652c0296c10 2024-04-01T01:41:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.717+0000 7f3a6b5d2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c1467c0 con 0x5652c029c770 2024-04-01T01:41:03.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.717+0000 7f3a6b5d2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c145350 con 0x5652c0297600 2024-04-01T01:41:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.717+0000 7f3a62d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 0x5652c02b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.717+0000 7f3a62d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 0x5652c02b3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57518/0 (socket says 172.21.15.70:57518) 2024-04-01T01:41:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.717+0000 7f3a61d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c0297600 0x5652c02a8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.717+0000 7f3a62d9d640 1 -- 172.21.15.70:0/2666398245 learned_addr learned my addr 172.21.15.70:0/2666398245 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.717+0000 7f3a6259c640 1 --2- 172.21.15.70:0/2666398245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c0295af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.718+0000 7f3a62d9d640 1 -- 172.21.15.70:0/2666398245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c0297600 msgr2=0x5652c02a8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.718+0000 7f3a62d9d640 1 --2- 172.21.15.70:0/2666398245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c0297600 0x5652c02a8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.718+0000 7f3a62d9d640 1 -- 172.21.15.70:0/2666398245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 msgr2=0x5652c0295af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.718+0000 7f3a62d9d640 1 --2- 172.21.15.70:0/2666398245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c0295af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.718+0000 7f3a62d9d640 1 -- 172.21.15.70:0/2666398245 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652c02a1ea0 con 0x5652c029c770 2024-04-01T01:41:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.718+0000 7f3a62d9d640 1 --2- 172.21.15.70:0/2666398245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 0x5652c02b3070 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3a64064c40 tx=0x7f3a640675f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b0c590e6325f027 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.718+0000 7f3a4bfff640 1 -- 172.21.15.70:0/2666398245 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a641059f0 con 0x5652c029c770 2024-04-01T01:41:03.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.718+0000 7f3a4bfff640 1 -- 172.21.15.70:0/2666398245 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a64109a90 con 0x5652c029c770 2024-04-01T01:41:03.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.718+0000 7f3a4bfff640 1 -- 172.21.15.70:0/2666398245 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a64104e40 con 0x5652c029c770 2024-04-01T01:41:03.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.720+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/2666398245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 msgr2=0x5652c02b3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:03.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.720+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/2666398245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 0x5652c02b3070 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3a64064c40 tx=0x7f3a640675f0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.720+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/2666398245 shutdown_connections 2024-04-01T01:41:03.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.720+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/2666398245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 0x5652c02b3070 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.720+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/2666398245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c0297600 0x5652c02a8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.720+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/2666398245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c0295af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.720+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/2666398245 >> 172.21.15.70:0/2666398245 conn(0x5652c0292180 msgr2=0x5652c02953e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:03.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.720+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/2666398245 shutdown_connections 2024-04-01T01:41:03.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.720+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/2666398245 wait complete. 2024-04-01T01:41:03.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.721+0000 7f3a6b5d2740 1 -- start start 2024-04-01T01:41:03.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.721+0000 7f3a6b5d2740 1 Processor -- start 2024-04-01T01:41:03.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.722+0000 7f3a6b5d2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c02a7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.722+0000 7f3a6b5d2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c0297600 0x5652c02a8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.722+0000 7f3a6259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c02a7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.722+0000 7f3a6b5d2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 0x5652c02a6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.722+0000 7f3a6b5d2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c1460f0 con 0x5652c0296c10 2024-04-01T01:41:03.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.722+0000 7f3a6b5d2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c145a20 con 0x5652c029c770 2024-04-01T01:41:03.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.722+0000 7f3a6b5d2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c142c60 con 0x5652c0297600 2024-04-01T01:41:03.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.722+0000 7f3a6b5d2740 10 client.0 _ll_register_callbacks cb 0x7f3a54004440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:03.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.722+0000 7f3a6259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c02a7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38808/0 (socket says 172.21.15.70:38808) 2024-04-01T01:41:03.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.722+0000 7f3a6259c640 1 -- 172.21.15.70:0/982351197 learned_addr learned my addr 172.21.15.70:0/982351197 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:03.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.722+0000 7f3a62d9d640 1 --2- 172.21.15.70:0/982351197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 0x5652c02a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.722+0000 7f3a61d9b640 1 --2- 172.21.15.70:0/982351197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c0297600 0x5652c02a8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.723+0000 7f3a62d9d640 1 -- 172.21.15.70:0/982351197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c0297600 msgr2=0x5652c02a8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:03.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.723+0000 7f3a62d9d640 1 --2- 172.21.15.70:0/982351197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c0297600 0x5652c02a8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.723+0000 7f3a62d9d640 1 -- 172.21.15.70:0/982351197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 msgr2=0x5652c02a7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:03.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.723+0000 7f3a62d9d640 1 --2- 172.21.15.70:0/982351197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c02a7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.723+0000 7f3a62d9d640 1 -- 172.21.15.70:0/982351197 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652c02a6f70 con 0x5652c029c770 2024-04-01T01:41:03.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.723+0000 7f3a6259c640 1 --2- 172.21.15.70:0/982351197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c02a7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:41:03.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.723+0000 7f3a62d9d640 1 --2- 172.21.15.70:0/982351197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 0x5652c02a6770 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3a64067010 tx=0x7f3a641390c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b1f7164ae1f3cbc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:03.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.723+0000 7f3a49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:03.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.723+0000 7f3a49ffb640 1 -- 172.21.15.70:0/982351197 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6411d070 con 0x5652c029c770 2024-04-01T01:41:03.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.723+0000 7f3a49ffb640 1 -- 172.21.15.70:0/982351197 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3a64078770 con 0x5652c029c770 2024-04-01T01:41:03.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.724+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/982351197 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a64109a90 con 0x5652c029c770 2024-04-01T01:41:03.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.724+0000 7f3a49ffb640 1 -- 172.21.15.70:0/982351197 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3a641059f0 con 0x5652c029c770 2024-04-01T01:41:03.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.724+0000 7f3a6b5d2740 10 client.27764 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:03.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.724+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/982351197 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5652c03132b0 con 0x5652c029c770 2024-04-01T01:41:03.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.724+0000 7f3a49ffb640 1 -- 172.21.15.70:0/982351197 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a64078970 con 0x5652c029c770 2024-04-01T01:41:03.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.725+0000 7f3a49ffb640 1 -- 172.21.15.70:0/982351197 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f3a640697b0 con 0x5652c029c770 2024-04-01T01:41:03.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.725+0000 7f3a6b5d2740 10 client.27764 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:03.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.725+0000 7f3a6b5d2740 10 client.27764 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:03.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.725+0000 7f3a6b5d2740 20 client.27764 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:03.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.725+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/982351197 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5652c0313b60 0x5652c0315f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.725+0000 7f3a6b5d2740 4 client.27764 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:41:03.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.725+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/982351197 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f3a64109a90 con 0x5652c0313b60 2024-04-01T01:41:03.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.725+0000 7f3a6259c640 1 --2- 172.21.15.70:0/982351197 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5652c0313b60 0x5652c0315f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.726+0000 7f3a49ffb640 10 client.27764 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:03.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.726+0000 7f3a6259c640 1 --2- 172.21.15.70:0/982351197 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5652c0313b60 0x5652c0315f50 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4da2e5544c71de29 server_cookie=9fa386a4532d8858 in_seq=0 out_seq=0 2024-04-01T01:41:03.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.728+0000 7f3a49ffb640 1 -- 172.21.15.70:0/982351197 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f3a64109a90 con 0x5652c0313b60 2024-04-01T01:41:03.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.728+0000 7f3a49ffb640 10 client.27764 handle_command_reply: tid=0 2024-04-01T01:41:03.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.728+0000 7f3a6b5d2740 1 client.27764 shutdown 2024-04-01T01:41:03.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.728+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/982351197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 msgr2=0x5652c02a6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:03.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.728+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/982351197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 0x5652c02a6770 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3a64067010 tx=0x7f3a641390c0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.728+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/982351197 shutdown_connections 2024-04-01T01:41:03.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.728+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/982351197 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5652c0313b60 0x5652c0315f50 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.728+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/982351197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5652c029c770 0x5652c02a6770 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.728+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/982351197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c0297600 0x5652c02a8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.728+0000 7f3a6b5d2740 1 --2- 172.21.15.70:0/982351197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5652c0296c10 0x5652c02a7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.728+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/982351197 >> 172.21.15.70:0/982351197 conn(0x5652c0292180 msgr2=0x5652c0295c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:03.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.728+0000 7f3a49ffb640 0 client.27764 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:03.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.728+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/982351197 shutdown_connections 2024-04-01T01:41:03.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.728+0000 7f3a6b5d2740 1 -- 172.21.15.70:0/982351197 wait complete. 2024-04-01T01:41:03.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.729+0000 7f3a6b5d2740 20 client.27764 trim_cache size 0 max 16384 2024-04-01T01:41:03.731 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:41:03.731 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:41:03.731 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "f9386fb8-72d3-4d7c-a37e-dd96014ddf28", 2024-04-01T01:41:03.731 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:41:03.731 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:41:03.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.730+0000 7f3a6359e640 1 -- 172.21.15.70:0/3671737024 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3a4c0a2570 msgr2=0x7f3a4c0a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:03.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.730+0000 7f3a6359e640 1 --2- 172.21.15.70:0/3671737024 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3a4c0a2570 0x7f3a4c0a4a30 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3a58009fd0 tx=0x7f3a58009780 comp rx=0 tx=0).stop 2024-04-01T01:41:03.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.730+0000 7f3a6359e640 1 -- 172.21.15.70:0/3671737024 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c0a2340 msgr2=0x7f3a5c139940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:03.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.730+0000 7f3a6359e640 1 --2- 172.21.15.70:0/3671737024 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c0a2340 0x7f3a5c139940 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3a64067420 tx=0x7f3a6407aee0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.730+0000 7f3a6359e640 1 -- 172.21.15.70:0/3671737024 shutdown_connections 2024-04-01T01:41:03.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.730+0000 7f3a6359e640 1 --2- 172.21.15.70:0/3671737024 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c0a2340 0x7f3a5c139940 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.730+0000 7f3a6359e640 1 --2- 172.21.15.70:0/3671737024 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3a4c0a2570 0x7f3a4c0a4a30 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.730+0000 7f3a6359e640 1 --2- 172.21.15.70:0/3671737024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a5c008b60 0x7f3a5c1328d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.730+0000 7f3a6359e640 1 --2- 172.21.15.70:0/3671737024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a5c007580 0x7f3a5c132390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.730+0000 7f3a6359e640 1 -- 172.21.15.70:0/3671737024 >> 172.21.15.70:0/3671737024 conn(0x7f3a5c0912e0 msgr2=0x7f3a5c0955d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:03.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.730+0000 7f3a6359e640 1 -- 172.21.15.70:0/3671737024 shutdown_connections 2024-04-01T01:41:03.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.730+0000 7f3a6359e640 1 -- 172.21.15.70:0/3671737024 wait complete. 2024-04-01T01:41:03.747 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:41:03.747 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:41:03.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.959+0000 7f23f3587640 1 -- 172.21.15.70:0/1995645806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23ec1097e0 msgr2=0x7f23ec10e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:03.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.959+0000 7f23f3587640 1 --2- 172.21.15.70:0/1995645806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23ec1097e0 0x7f23ec10e080 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f23d8009900 tx=0x7f23d802bc60 comp rx=0 tx=0).stop 2024-04-01T01:41:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.959+0000 7f23f3587640 1 -- 172.21.15.70:0/1995645806 shutdown_connections 2024-04-01T01:41:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.959+0000 7f23f3587640 1 --2- 172.21.15.70:0/1995645806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23ec1097e0 0x7f23ec10e080 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.959+0000 7f23f3587640 1 --2- 172.21.15.70:0/1995645806 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23ec108e20 0x7f23ec1092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.959+0000 7f23f3587640 1 --2- 172.21.15.70:0/1995645806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23ec072b60 0x7f23ec072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.959+0000 7f23f3587640 1 -- 172.21.15.70:0/1995645806 >> 172.21.15.70:0/1995645806 conn(0x7f23ec06e630 msgr2=0x7f23ec070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.959+0000 7f23f3587640 1 -- 172.21.15.70:0/1995645806 shutdown_connections 2024-04-01T01:41:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.959+0000 7f23f3587640 1 -- 172.21.15.70:0/1995645806 wait complete. 2024-04-01T01:41:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.960+0000 7f23f3587640 1 Processor -- start 2024-04-01T01:41:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.960+0000 7f23f3587640 1 -- start start 2024-04-01T01:41:03.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.960+0000 7f23f3587640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23ec072b60 0x7f23ec1a65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.960+0000 7f23f3587640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23ec108e20 0x7f23ec1a6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.960+0000 7f23f3587640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23ec1097e0 0x7f23ec1adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.960+0000 7f23f3587640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec11bf50 con 0x7f23ec1097e0 2024-04-01T01:41:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.960+0000 7f23f3587640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec11bdd0 con 0x7f23ec072b60 2024-04-01T01:41:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.960+0000 7f23f3587640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec11c0d0 con 0x7f23ec108e20 2024-04-01T01:41:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.961+0000 7f23f1d84640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23ec108e20 0x7f23ec1a6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.961+0000 7f23f1d84640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23ec108e20 0x7f23ec1a6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43352/0 (socket says 172.21.15.70:43352) 2024-04-01T01:41:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.961+0000 7f23f1d84640 1 -- 172.21.15.70:0/3642486266 learned_addr learned my addr 172.21.15.70:0/3642486266 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.961+0000 7f23f2585640 1 --2- 172.21.15.70:0/3642486266 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23ec072b60 0x7f23ec1a65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.961+0000 7f23f1d84640 1 -- 172.21.15.70:0/3642486266 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23ec072b60 msgr2=0x7f23ec1a65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.961+0000 7f23f1d84640 1 --2- 172.21.15.70:0/3642486266 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23ec072b60 0x7f23ec1a65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.961+0000 7f23f1d84640 1 -- 172.21.15.70:0/3642486266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23ec1097e0 msgr2=0x7f23ec1adb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.961+0000 7f23f1d84640 1 --2- 172.21.15.70:0/3642486266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23ec1097e0 0x7f23ec1adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.961+0000 7f23f1d84640 1 -- 172.21.15.70:0/3642486266 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23ec1ae1f0 con 0x7f23ec108e20 2024-04-01T01:41:03.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.962+0000 7f23f1d84640 1 --2- 172.21.15.70:0/3642486266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23ec108e20 0x7f23ec1a6b10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f23e000ebc0 tx=0x7f23e000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:03.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.962+0000 7f23eb7fe640 1 -- 172.21.15.70:0/3642486266 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e0010630 con 0x7f23ec108e20 2024-04-01T01:41:03.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.962+0000 7f23eb7fe640 1 -- 172.21.15.70:0/3642486266 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f23e000ce50 con 0x7f23ec108e20 2024-04-01T01:41:03.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.962+0000 7f23f3587640 1 -- 172.21.15.70:0/3642486266 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23ec1ae4e0 con 0x7f23ec108e20 2024-04-01T01:41:03.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.962+0000 7f23f3587640 1 -- 172.21.15.70:0/3642486266 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23ec1aea30 con 0x7f23ec108e20 2024-04-01T01:41:03.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.963+0000 7f23eb7fe640 1 -- 172.21.15.70:0/3642486266 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e0018ad0 con 0x7f23ec108e20 2024-04-01T01:41:03.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.968+0000 7f23f73d0740 1 Processor -- start 2024-04-01T01:41:03.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.968+0000 7f23f73d0740 1 -- start start 2024-04-01T01:41:03.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.969+0000 7f23f73d0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8490 0x557dbc2a88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.969+0000 7f23f73d0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2a8e80 0x557dbc2ad980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.969+0000 7f23f2d86640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8490 0x557dbc2a88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.969+0000 7f23f2d86640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8490 0x557dbc2a88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57562/0 (socket says 172.21.15.70:57562) 2024-04-01T01:41:03.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.969+0000 7f23f73d0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2adff0 0x557dbc2b0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.969+0000 7f23f2d86640 1 -- 172.21.15.70:0/2610438782 learned_addr learned my addr 172.21.15.70:0/2610438782 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:03.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.969+0000 7f23f73d0740 1 -- 172.21.15.70:0/2610438782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557dbc20d180 con 0x557dbc2a8e80 2024-04-01T01:41:03.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.969+0000 7f23f73d0740 1 -- 172.21.15.70:0/2610438782 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x557dbc1b5b70 con 0x557dbc2a8490 2024-04-01T01:41:03.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.969+0000 7f23f73d0740 1 -- 172.21.15.70:0/2610438782 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557dbc209220 con 0x557dbc2adff0 2024-04-01T01:41:03.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.970+0000 7f23eb7fe640 1 -- 172.21.15.70:0/3642486266 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f23e0018c70 con 0x7f23ec108e20 2024-04-01T01:41:03.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.970+0000 7f23f2d86640 1 -- 172.21.15.70:0/2610438782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2adff0 msgr2=0x557dbc2b0480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:41:03.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.970+0000 7f23f2d86640 1 --2- 172.21.15.70:0/2610438782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2adff0 0x557dbc2b0480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.970+0000 7f23f2d86640 1 -- 172.21.15.70:0/2610438782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2a8e80 msgr2=0x557dbc2ad980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:03.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.970+0000 7f23f2d86640 1 --2- 172.21.15.70:0/2610438782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2a8e80 0x557dbc2ad980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.970+0000 7f23f2d86640 1 -- 172.21.15.70:0/2610438782 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557dbc1c97c0 con 0x557dbc2a8490 2024-04-01T01:41:03.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.971+0000 7f23f2d86640 1 --2- 172.21.15.70:0/2610438782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8490 0x557dbc2a88b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f23d802c270 tx=0x7f23d80095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea9c47f7ca0436d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:03.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.971+0000 7f23eb7fe640 1 --2- 172.21.15.70:0/3642486266 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23dc0a2570 0x7f23dc0a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.971+0000 7f23e8ff9640 1 -- 172.21.15.70:0/2610438782 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d8039040 con 0x557dbc2a8490 2024-04-01T01:41:03.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.971+0000 7f23eb7fe640 1 -- 172.21.15.70:0/3642486266 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f23e00cf140 con 0x7f23ec108e20 2024-04-01T01:41:03.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.971+0000 7f23e8ff9640 1 -- 172.21.15.70:0/2610438782 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f23d8042070 con 0x557dbc2a8490 2024-04-01T01:41:03.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.971+0000 7f23e8ff9640 1 -- 172.21.15.70:0/2610438782 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d80035f0 con 0x557dbc2a8490 2024-04-01T01:41:03.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.972+0000 7f23f2585640 1 --2- 172.21.15.70:0/3642486266 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23dc0a2570 0x7f23dc0a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.972+0000 7f23f73d0740 1 -- 172.21.15.70:0/2610438782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8490 msgr2=0x557dbc2a88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:03.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.972+0000 7f23f73d0740 1 --2- 172.21.15.70:0/2610438782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8490 0x557dbc2a88b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f23d802c270 tx=0x7f23d80095f0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.972+0000 7f23f73d0740 1 -- 172.21.15.70:0/2610438782 shutdown_connections 2024-04-01T01:41:03.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.972+0000 7f23f73d0740 1 --2- 172.21.15.70:0/2610438782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2adff0 0x557dbc2b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.972+0000 7f23f73d0740 1 --2- 172.21.15.70:0/2610438782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2a8e80 0x557dbc2ad980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.973+0000 7f23f73d0740 1 --2- 172.21.15.70:0/2610438782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8490 0x557dbc2a88b0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.973+0000 7f23f73d0740 1 -- 172.21.15.70:0/2610438782 >> 172.21.15.70:0/2610438782 conn(0x557dbc2a3a00 msgr2=0x557dbc2a3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:03.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.976+0000 7f23f73d0740 1 -- 172.21.15.70:0/2610438782 shutdown_connections 2024-04-01T01:41:03.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.976+0000 7f23f2585640 1 --2- 172.21.15.70:0/3642486266 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23dc0a2570 0x7f23dc0a4a30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f23e400b620 tx=0x7f23e400b5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:03.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.976+0000 7f23f73d0740 1 -- 172.21.15.70:0/2610438782 wait complete. 2024-04-01T01:41:03.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.977+0000 7f23f73d0740 1 -- start start 2024-04-01T01:41:03.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.977+0000 7f23f73d0740 1 Processor -- start 2024-04-01T01:41:03.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.978+0000 7f23f73d0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2a8490 0x557dbc2b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.978+0000 7f23f73d0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8e80 0x557dbc2b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.978+0000 7f23f73d0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2adff0 0x557dbc2b9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.978+0000 7f23f73d0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557dbc2b0f40 con 0x557dbc2a8490 2024-04-01T01:41:03.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.978+0000 7f23f2585640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2a8490 0x557dbc2b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.978+0000 7f23f2585640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2a8490 0x557dbc2b27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38842/0 (socket says 172.21.15.70:38842) 2024-04-01T01:41:03.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.978+0000 7f23f2585640 1 -- 172.21.15.70:0/832078802 learned_addr learned my addr 172.21.15.70:0/832078802 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:03.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.979+0000 7f23f73d0740 1 -- 172.21.15.70:0/832078802 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x557dbc1ab4a0 con 0x557dbc2adff0 2024-04-01T01:41:03.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.979+0000 7f23f73d0740 1 -- 172.21.15.70:0/832078802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557dbc2b10c0 con 0x557dbc2a8e80 2024-04-01T01:41:03.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.979+0000 7f23f1d84640 1 --2- 172.21.15.70:0/832078802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8e80 0x557dbc2b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.979+0000 7f23f73d0740 10 client.0 _ll_register_callbacks cb 0x557dbc1aed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.979+0000 7f23f1d84640 1 -- 172.21.15.70:0/832078802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2adff0 msgr2=0x557dbc2b9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.979+0000 7f23f1d84640 1 --2- 172.21.15.70:0/832078802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2adff0 0x557dbc2b9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.979+0000 7f23f1d84640 1 -- 172.21.15.70:0/832078802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2a8490 msgr2=0x557dbc2b27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.979+0000 7f23f1d84640 1 --2- 172.21.15.70:0/832078802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2a8490 0x557dbc2b27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.979+0000 7f23f1d84640 1 -- 172.21.15.70:0/832078802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557dbbfeb7d0 con 0x557dbc2a8e80 2024-04-01T01:41:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.980+0000 7f23f1d84640 1 --2- 172.21.15.70:0/832078802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8e80 0x557dbc2b2d10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f23e009b760 tx=0x7f23e009b390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cec98111634ea829 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.980+0000 7f23d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:41:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.980+0000 7f23d5ffb640 1 -- 172.21.15.70:0/832078802 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e009bed0 con 0x557dbc2a8e80 2024-04-01T01:41:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.980+0000 7f23d5ffb640 1 -- 172.21.15.70:0/832078802 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f23e009ac20 con 0x557dbc2a8e80 2024-04-01T01:41:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.980+0000 7f23f73d0740 1 -- 172.21.15.70:0/832078802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23d8042070 con 0x557dbc2a8e80 2024-04-01T01:41:03.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.981+0000 7f23d5ffb640 1 -- 172.21.15.70:0/832078802 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f23e009b7d0 con 0x557dbc2a8e80 2024-04-01T01:41:03.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.981+0000 7f23f73d0740 10 client.27607 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:03.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.981+0000 7f23f73d0740 1 -- 172.21.15.70:0/832078802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557dbbf213f0 con 0x557dbc2a8e80 2024-04-01T01:41:03.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.982+0000 7f23d5ffb640 1 -- 172.21.15.70:0/832078802 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f23d8042070 con 0x557dbc2a8e80 2024-04-01T01:41:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.982+0000 7f23d5ffb640 1 -- 172.21.15.70:0/832078802 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f23e0097520 con 0x557dbc2a8e80 2024-04-01T01:41:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.982+0000 7f23f73d0740 10 client.27607 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.982+0000 7f23f73d0740 10 client.27607 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.982+0000 7f23f73d0740 20 client.27607 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.982+0000 7f23f73d0740 1 --2- 172.21.15.70:0/832078802 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557dbc2be5b0 0x557dbc2c09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.982+0000 7f23f73d0740 4 client.27607 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.982+0000 7f23f73d0740 1 -- 172.21.15.70:0/832078802 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557dbc21a6c0 con 0x557dbc2be5b0 2024-04-01T01:41:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.983+0000 7f23f2585640 1 --2- 172.21.15.70:0/832078802 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557dbc2be5b0 0x557dbc2c09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:03.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.983+0000 7f23d5ffb640 10 client.27607 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:03.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.984+0000 7f23f2585640 1 --2- 172.21.15.70:0/832078802 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557dbc2be5b0 0x557dbc2c09e0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d44fc9da336f3e2 server_cookie=f1815a4a009bc140 in_seq=0 out_seq=0 2024-04-01T01:41:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.986+0000 7f23d5ffb640 1 -- 172.21.15.70:0/832078802 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557dbc21a6c0 con 0x557dbc2be5b0 2024-04-01T01:41:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.986+0000 7f23d5ffb640 10 client.27607 handle_command_reply: tid=0 2024-04-01T01:41:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.986+0000 7f23f73d0740 1 client.27607 shutdown 2024-04-01T01:41:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.986+0000 7f23f73d0740 1 -- 172.21.15.70:0/832078802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8e80 msgr2=0x557dbc2b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.986+0000 7f23f73d0740 1 --2- 172.21.15.70:0/832078802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8e80 0x557dbc2b2d10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f23e009b760 tx=0x7f23e009b390 comp rx=0 tx=0).stop 2024-04-01T01:41:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.986+0000 7f23f73d0740 1 -- 172.21.15.70:0/832078802 shutdown_connections 2024-04-01T01:41:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.986+0000 7f23f73d0740 1 --2- 172.21.15.70:0/832078802 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557dbc2be5b0 0x557dbc2c09e0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.986+0000 7f23f73d0740 1 --2- 172.21.15.70:0/832078802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2adff0 0x557dbc2b9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.986+0000 7f23d5ffb640 0 client.27607 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.986+0000 7f23f73d0740 1 --2- 172.21.15.70:0/832078802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8e80 0x557dbc2b2d10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.986+0000 7f23f73d0740 1 --2- 172.21.15.70:0/832078802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2a8490 0x557dbc2b27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.986+0000 7f23f73d0740 1 -- 172.21.15.70:0/832078802 >> 172.21.15.70:0/832078802 conn(0x557dbc2a3a00 msgr2=0x557dbc2a2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.986+0000 7f23f73d0740 1 -- 172.21.15.70:0/832078802 shutdown_connections 2024-04-01T01:41:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.986+0000 7f23f73d0740 1 -- 172.21.15.70:0/832078802 wait complete. 2024-04-01T01:41:03.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:03.987+0000 7f23f73d0740 20 client.27607 trim_cache size 0 max 16384 2024-04-01T01:41:04.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.025+0000 7f23f73d0740 1 Processor -- start 2024-04-01T01:41:04.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.025+0000 7f23f73d0740 1 -- start start 2024-04-01T01:41:04.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.025+0000 7f23f73d0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8490 0x557dbc2a2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:04.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.025+0000 7f23f73d0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8e80 0x557dbc2b9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:04.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.025+0000 7f23f73d0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2adff0 0x557dbc2a28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:04.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.025+0000 7f23f1d84640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8e80 0x557dbc2b9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:04.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.025+0000 7f23f2585640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8490 0x557dbc2a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:04.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.026+0000 7f23f2585640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8490 0x557dbc2a2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57578/0 (socket says 172.21.15.70:57578) 2024-04-01T01:41:04.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.026+0000 7f23f2585640 1 -- 172.21.15.70:0/1023102782 learned_addr learned my addr 172.21.15.70:0/1023102782 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:04.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.026+0000 7f23f73d0740 1 -- 172.21.15.70:0/1023102782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec1b8f30 con 0x557dbc2adff0 2024-04-01T01:41:04.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.026+0000 7f23f73d0740 1 -- 172.21.15.70:0/1023102782 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec1baa70 con 0x557dbc2a8490 2024-04-01T01:41:04.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.026+0000 7f23f73d0740 1 -- 172.21.15.70:0/1023102782 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec1b9600 con 0x557dbc2a8e80 2024-04-01T01:41:04.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.026+0000 7f23f2d86640 1 --2- 172.21.15.70:0/1023102782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2adff0 0x557dbc2a28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:04.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.026+0000 7f23f2585640 1 -- 172.21.15.70:0/1023102782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8e80 msgr2=0x557dbc2b9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:04.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.026+0000 7f23f2585640 1 --2- 172.21.15.70:0/1023102782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8e80 0x557dbc2b9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.026+0000 7f23f2585640 1 -- 172.21.15.70:0/1023102782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2adff0 msgr2=0x557dbc2a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:04.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.026+0000 7f23f2585640 1 --2- 172.21.15.70:0/1023102782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2adff0 0x557dbc2a28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.026+0000 7f23f2585640 1 -- 172.21.15.70:0/1023102782 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557dbbee10f0 con 0x557dbc2a8490 2024-04-01T01:41:04.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.026+0000 7f23f2585640 1 --2- 172.21.15.70:0/1023102782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8490 0x557dbc2a2390 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f23e400bfd0 tx=0x7f23e4013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbad6a269d75c40a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:04.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.027+0000 7f23d7fff640 1 -- 172.21.15.70:0/1023102782 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e402c070 con 0x557dbc2a8490 2024-04-01T01:41:04.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.027+0000 7f23d7fff640 1 -- 172.21.15.70:0/1023102782 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f23e4003d70 con 0x557dbc2a8490 2024-04-01T01:41:04.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.027+0000 7f23d7fff640 1 -- 172.21.15.70:0/1023102782 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e40117a0 con 0x557dbc2a8490 2024-04-01T01:41:04.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.028+0000 7f23f73d0740 1 -- 172.21.15.70:0/1023102782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8490 msgr2=0x557dbc2a2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:04.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.028+0000 7f23f73d0740 1 --2- 172.21.15.70:0/1023102782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8490 0x557dbc2a2390 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f23e400bfd0 tx=0x7f23e4013000 comp rx=0 tx=0).stop 2024-04-01T01:41:04.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.028+0000 7f23f73d0740 1 -- 172.21.15.70:0/1023102782 shutdown_connections 2024-04-01T01:41:04.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.028+0000 7f23f73d0740 1 --2- 172.21.15.70:0/1023102782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2adff0 0x557dbc2a28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.028+0000 7f23f73d0740 1 --2- 172.21.15.70:0/1023102782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8e80 0x557dbc2b9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.028+0000 7f23f73d0740 1 --2- 172.21.15.70:0/1023102782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8490 0x557dbc2a2390 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.028+0000 7f23f73d0740 1 -- 172.21.15.70:0/1023102782 >> 172.21.15.70:0/1023102782 conn(0x557dbc2a3a00 msgr2=0x557dbc2b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:04.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.029+0000 7f23f73d0740 1 -- 172.21.15.70:0/1023102782 shutdown_connections 2024-04-01T01:41:04.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.029+0000 7f23f73d0740 1 -- 172.21.15.70:0/1023102782 wait complete. 2024-04-01T01:41:04.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.029+0000 7f23f73d0740 1 -- start start 2024-04-01T01:41:04.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.029+0000 7f23f73d0740 1 Processor -- start 2024-04-01T01:41:04.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.029+0000 7f23f73d0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8490 0x557dbc2b7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:04.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.029+0000 7f23f73d0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8e80 0x557dbc2b7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:04.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.030+0000 7f23f2585640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8490 0x557dbc2b7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:04.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.030+0000 7f23f2585640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8490 0x557dbc2b7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43386/0 (socket says 172.21.15.70:43386) 2024-04-01T01:41:04.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.030+0000 7f23f2585640 1 -- 172.21.15.70:0/1022501834 learned_addr learned my addr 172.21.15.70:0/1022501834 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:04.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.030+0000 7f23f73d0740 1 --2- 172.21.15.70:0/1022501834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2adff0 0x557dbc2b8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:04.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.030+0000 7f23f1d84640 1 --2- 172.21.15.70:0/1022501834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8e80 0x557dbc2b7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:04.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.030+0000 7f23f73d0740 1 -- 172.21.15.70:0/1022501834 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec1ba3a0 con 0x557dbc2adff0 2024-04-01T01:41:04.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.030+0000 7f23f73d0740 1 -- 172.21.15.70:0/1022501834 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec1b9cd0 con 0x557dbc2a8e80 2024-04-01T01:41:04.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.030+0000 7f23f73d0740 1 -- 172.21.15.70:0/1022501834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec1b6f10 con 0x557dbc2a8490 2024-04-01T01:41:04.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.030+0000 7f23f73d0740 10 client.0 _ll_register_callbacks cb 0x7f23e009b5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:04.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.031+0000 7f23f2d86640 1 --2- 172.21.15.70:0/1022501834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2adff0 0x557dbc2b8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:04.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.031+0000 7f23f2585640 1 -- 172.21.15.70:0/1022501834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8e80 msgr2=0x557dbc2b7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:04.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.031+0000 7f23f2585640 1 --2- 172.21.15.70:0/1022501834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8e80 0x557dbc2b7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.031+0000 7f23f2585640 1 -- 172.21.15.70:0/1022501834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2adff0 msgr2=0x557dbc2b8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:04.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.031+0000 7f23f2585640 1 --2- 172.21.15.70:0/1022501834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2adff0 0x557dbc2b8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.031+0000 7f23f2585640 1 -- 172.21.15.70:0/1022501834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557dbc2b93d0 con 0x557dbc2a8490 2024-04-01T01:41:04.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.031+0000 7f23f2585640 1 --2- 172.21.15.70:0/1022501834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8490 0x557dbc2b7890 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f23e4027a40 tx=0x7f23e4027a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f781d470fb410bae server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:04.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.031+0000 7f23d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:41:04.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.032+0000 7f23d5ffb640 1 -- 172.21.15.70:0/1022501834 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e4037020 con 0x557dbc2a8490 2024-04-01T01:41:04.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.032+0000 7f23d5ffb640 1 -- 172.21.15.70:0/1022501834 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f23e4003b70 con 0x557dbc2a8490 2024-04-01T01:41:04.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.032+0000 7f23f73d0740 1 -- 172.21.15.70:0/1022501834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23e4003d70 con 0x557dbc2a8490 2024-04-01T01:41:04.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.036+0000 7f23d5ffb640 1 -- 172.21.15.70:0/1022501834 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f23e402c030 con 0x557dbc2a8490 2024-04-01T01:41:04.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.036+0000 7f23f73d0740 10 client.27613 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:04.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.036+0000 7f23f73d0740 1 -- 172.21.15.70:0/1022501834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557dbc324c70 con 0x557dbc2a8490 2024-04-01T01:41:04.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.036+0000 7f23d5ffb640 1 -- 172.21.15.70:0/1022501834 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f23e40272e0 con 0x557dbc2a8490 2024-04-01T01:41:04.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.036+0000 7f23d5ffb640 1 -- 172.21.15.70:0/1022501834 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f23e4032070 con 0x557dbc2a8490 2024-04-01T01:41:04.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.037+0000 7f23f73d0740 10 client.27613 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:04.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.037+0000 7f23f73d0740 10 client.27613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:04.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.037+0000 7f23f73d0740 20 client.27613 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:04.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.037+0000 7f23f73d0740 1 --2- 172.21.15.70:0/1022501834 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557dbc2be5b0 0x557dbc3274e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:04.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.037+0000 7f23f73d0740 4 client.27613 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:41:04.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.037+0000 7f23f73d0740 1 -- 172.21.15.70:0/1022501834 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f23e4003d70 con 0x557dbc2be5b0 2024-04-01T01:41:04.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.037+0000 7f23f1d84640 1 --2- 172.21.15.70:0/1022501834 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557dbc2be5b0 0x557dbc3274e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:04.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.038+0000 7f23d5ffb640 10 client.27613 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:04.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.039+0000 7f23f1d84640 1 --2- 172.21.15.70:0/1022501834 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557dbc2be5b0 0x557dbc3274e0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2206a5ed3cd4e3b5 server_cookie=712658dcd4908331 in_seq=0 out_seq=0 2024-04-01T01:41:04.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.042+0000 7f23d5ffb640 1 -- 172.21.15.70:0/1022501834 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x7f23e4003d70 con 0x557dbc2be5b0 2024-04-01T01:41:04.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.042+0000 7f23d5ffb640 10 client.27613 handle_command_reply: tid=0 2024-04-01T01:41:04.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.042+0000 7f23f73d0740 1 client.27613 shutdown 2024-04-01T01:41:04.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.045+0000 7f23f73d0740 1 -- 172.21.15.70:0/1022501834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8490 msgr2=0x557dbc2b7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:04.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.045+0000 7f23f73d0740 1 --2- 172.21.15.70:0/1022501834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8490 0x557dbc2b7890 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f23e4027a40 tx=0x7f23e4027a70 comp rx=0 tx=0).stop 2024-04-01T01:41:04.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.045+0000 7f23f73d0740 1 -- 172.21.15.70:0/1022501834 shutdown_connections 2024-04-01T01:41:04.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.045+0000 7f23f73d0740 1 --2- 172.21.15.70:0/1022501834 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557dbc2be5b0 0x557dbc3274e0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.045+0000 7f23d5ffb640 0 client.27613 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:04.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.045+0000 7f23f73d0740 1 --2- 172.21.15.70:0/1022501834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557dbc2adff0 0x557dbc2b8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.046+0000 7f23f73d0740 1 --2- 172.21.15.70:0/1022501834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557dbc2a8e80 0x557dbc2b7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.046+0000 7f23f73d0740 1 --2- 172.21.15.70:0/1022501834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557dbc2a8490 0x557dbc2b7890 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.046+0000 7f23f73d0740 1 -- 172.21.15.70:0/1022501834 >> 172.21.15.70:0/1022501834 conn(0x557dbc2a3a00 msgr2=0x557dbc2a6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:04.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.046+0000 7f23f73d0740 1 -- 172.21.15.70:0/1022501834 shutdown_connections 2024-04-01T01:41:04.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.046+0000 7f23f73d0740 1 -- 172.21.15.70:0/1022501834 wait complete. 2024-04-01T01:41:04.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.046+0000 7f23f73d0740 20 client.27613 trim_cache size 0 max 16384 2024-04-01T01:41:04.048 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:41:04.048 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "scrub active (161 inodes in the stack)", 2024-04-01T01:41:04.048 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:41:04.048 INFO:teuthology.orchestra.run.smithi070.stdout: "f9386fb8-72d3-4d7c-a37e-dd96014ddf28": { 2024-04-01T01:41:04.048 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:41:04.048 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "f9386fb8-72d3-4d7c-a37e-dd96014ddf28", 2024-04-01T01:41:04.048 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:41:04.048 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:41:04.049 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:41:04.049 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:41:04.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.049+0000 7f23f3587640 1 -- 172.21.15.70:0/3642486266 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23dc0a2570 msgr2=0x7f23dc0a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:04.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.049+0000 7f23f3587640 1 --2- 172.21.15.70:0/3642486266 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23dc0a2570 0x7f23dc0a4a30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f23e400b620 tx=0x7f23e400b5b0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.049+0000 7f23f3587640 1 -- 172.21.15.70:0/3642486266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23ec108e20 msgr2=0x7f23ec1a6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:04.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.049+0000 7f23f3587640 1 --2- 172.21.15.70:0/3642486266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23ec108e20 0x7f23ec1a6b10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f23e000ebc0 tx=0x7f23e000c6a0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.049+0000 7f23f3587640 1 -- 172.21.15.70:0/3642486266 shutdown_connections 2024-04-01T01:41:04.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.049+0000 7f23f3587640 1 --2- 172.21.15.70:0/3642486266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23ec1097e0 0x7f23ec1adb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.049+0000 7f23f3587640 1 --2- 172.21.15.70:0/3642486266 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23dc0a2570 0x7f23dc0a4a30 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.049+0000 7f23f3587640 1 --2- 172.21.15.70:0/3642486266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23ec108e20 0x7f23ec1a6b10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.049+0000 7f23f3587640 1 --2- 172.21.15.70:0/3642486266 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23ec072b60 0x7f23ec1a65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:04.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.050+0000 7f23f3587640 1 -- 172.21.15.70:0/3642486266 >> 172.21.15.70:0/3642486266 conn(0x7f23ec06e630 msgr2=0x7f23ec070290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:04.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.050+0000 7f23f3587640 1 -- 172.21.15.70:0/3642486266 shutdown_connections 2024-04-01T01:41:04.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:04.052+0000 7f23f3587640 1 -- 172.21.15.70:0/3642486266 wait complete. 2024-04-01T01:41:04.080 INFO:tasks.cephfs.filesystem:scrub status for tag:f9386fb8-72d3-4d7c-a37e-dd96014ddf28 - {'path': '/', 'tag': 'f9386fb8-72d3-4d7c-a37e-dd96014ddf28', 'options': 'recursive,force'} 2024-04-01T01:41:04.081 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:03 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:41:04.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:03 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:41:04.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:03 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:41:05.149 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:04 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:41:05.149 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:04 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:41:05.149 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:04 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:41:05.149 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:04 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:41:05.149 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:04 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:41:05.149 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:04 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:41:05.149 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:04 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:41:05.149 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:04 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-04-01T01:41:05.150 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:04 smithi189 ceph-mon[35753]: Unable to set osd_memory_target on smithi189 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-04-01T01:41:05.150 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:04 smithi189 ceph-mon[35753]: pgmap v587: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 9.2 MiB/s wr, 871 op/s 2024-04-01T01:41:05.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:04 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:41:05.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:04 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:41:05.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:04 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:41:05.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:04 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:41:05.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:04 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:41:05.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:04 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:41:05.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:04 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:41:05.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:04 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-04-01T01:41:05.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:04 smithi070 ceph-mon[32179]: Unable to set osd_memory_target on smithi189 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-04-01T01:41:05.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:04 smithi070 ceph-mon[32179]: pgmap v587: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 9.2 MiB/s wr, 871 op/s 2024-04-01T01:41:05.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:04 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:41:05.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:04 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:41:05.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:04 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:41:05.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:04 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:41:05.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:04 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:41:05.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:04 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:41:05.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:04 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:41:05.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:04 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-04-01T01:41:05.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:04 smithi081 ceph-mon[35142]: Unable to set osd_memory_target on smithi189 to 184788889: error parsing value: Value '184788889' is below minimum 939524096 2024-04-01T01:41:05.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:04 smithi081 ceph-mon[35142]: pgmap v587: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 9.2 MiB/s wr, 871 op/s 2024-04-01T01:41:06.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:05 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:41:06.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:41:06.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:41:06.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:41:06.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:05 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:41:06.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:41:06.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:41:06.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:41:06.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:05 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:41:06.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:41:06.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:41:06.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:41:06.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 380.0 s, 793.0 tps, lat 126.758 ms stddev 49.220 2024-04-01T01:41:07.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:06 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:41:07.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:06 smithi189 ceph-mon[35753]: pgmap v588: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-01T01:41:07.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:06 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:41:07.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:06 smithi070 ceph-mon[32179]: pgmap v588: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-01T01:41:07.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:06 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:41:07.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:06 smithi081 ceph-mon[35142]: pgmap v588: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-01T01:41:07.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.627+0000 7fd17bccd640 1 -- 172.21.15.70:0/829486729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17410da70 msgr2=0x7fd17410def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:07.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.627+0000 7fd17bccd640 1 --2- 172.21.15.70:0/829486729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17410da70 0x7fd17410def0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd16c009be0 tx=0x7fd16c02bef0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.628+0000 7fd17bccd640 1 -- 172.21.15.70:0/829486729 shutdown_connections 2024-04-01T01:41:07.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.628+0000 7fd17bccd640 1 --2- 172.21.15.70:0/829486729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17410da70 0x7fd17410def0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.628+0000 7fd17bccd640 1 --2- 172.21.15.70:0/829486729 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd17410b0c0 0x7fd17410d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.628+0000 7fd17bccd640 1 --2- 172.21.15.70:0/829486729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd174108750 0x7fd17410ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.628+0000 7fd17bccd640 1 -- 172.21.15.70:0/829486729 >> 172.21.15.70:0/829486729 conn(0x7fd17406e790 msgr2=0x7fd174070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:07.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.628+0000 7fd17bccd640 1 -- 172.21.15.70:0/829486729 shutdown_connections 2024-04-01T01:41:07.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.628+0000 7fd17bccd640 1 -- 172.21.15.70:0/829486729 wait complete. 2024-04-01T01:41:07.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.629+0000 7fd17bccd640 1 Processor -- start 2024-04-01T01:41:07.630 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.629+0000 7fd17bccd640 1 -- start start 2024-04-01T01:41:07.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.629+0000 7fd17bccd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd174108750 0x7fd1741a6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.629+0000 7fd17bccd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd17410b0c0 0x7fd1741a6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.629+0000 7fd17bccd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17410da70 0x7fd1741adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.629+0000 7fd17bccd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd17411c130 con 0x7fd174108750 2024-04-01T01:41:07.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.629+0000 7fd17bccd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd17411bfb0 con 0x7fd17410b0c0 2024-04-01T01:41:07.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.629+0000 7fd17bccd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd17411c2b0 con 0x7fd17410da70 2024-04-01T01:41:07.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.629+0000 7fd17a26a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17410da70 0x7fd1741adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:07.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.629+0000 7fd179268640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd17410b0c0 0x7fd1741a6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:07.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.629+0000 7fd17a26a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17410da70 0x7fd1741adc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:48600/0 (socket says 172.21.15.70:48600) 2024-04-01T01:41:07.631 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.629+0000 7fd17a26a640 1 -- 172.21.15.70:0/3315875174 learned_addr learned my addr 172.21.15.70:0/3315875174 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:07.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.630+0000 7fd179a69640 1 --2- 172.21.15.70:0/3315875174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd174108750 0x7fd1741a6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:07.632 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.630+0000 7fd17a26a640 1 -- 172.21.15.70:0/3315875174 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd17410b0c0 msgr2=0x7fd1741a6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.630+0000 7fd17a26a640 1 --2- 172.21.15.70:0/3315875174 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd17410b0c0 0x7fd1741a6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.630+0000 7fd17a26a640 1 -- 172.21.15.70:0/3315875174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd174108750 msgr2=0x7fd1741a6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.630+0000 7fd17a26a640 1 --2- 172.21.15.70:0/3315875174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd174108750 0x7fd1741a6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.630+0000 7fd17a26a640 1 -- 172.21.15.70:0/3315875174 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1741ae2e0 con 0x7fd17410da70 2024-04-01T01:41:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.630+0000 7fd179a69640 1 --2- 172.21.15.70:0/3315875174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd174108750 0x7fd1741a6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:41:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.630+0000 7fd17a26a640 1 --2- 172.21.15.70:0/3315875174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17410da70 0x7fd1741adc70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd16c034fd0 tx=0x7fd16c033c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.632+0000 7fd16affd640 1 -- 172.21.15.70:0/3315875174 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c043020 con 0x7fd17410da70 2024-04-01T01:41:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.632+0000 7fd16affd640 1 -- 172.21.15.70:0/3315875174 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd16c003e20 con 0x7fd17410da70 2024-04-01T01:41:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.632+0000 7fd16affd640 1 -- 172.21.15.70:0/3315875174 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c034690 con 0x7fd17410da70 2024-04-01T01:41:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.632+0000 7fd17bccd640 1 -- 172.21.15.70:0/3315875174 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1741ae570 con 0x7fd17410da70 2024-04-01T01:41:07.633 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.632+0000 7fd17bccd640 1 -- 172.21.15.70:0/3315875174 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1741aeaf0 con 0x7fd17410da70 2024-04-01T01:41:07.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.636+0000 7fd16affd640 1 -- 172.21.15.70:0/3315875174 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fd16c046020 con 0x7fd17410da70 2024-04-01T01:41:07.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.636+0000 7fd17e993740 1 Processor -- start 2024-04-01T01:41:07.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.637+0000 7fd16affd640 1 --2- 172.21.15.70:0/3315875174 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1600a26b0 0x7fd1600a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.637+0000 7fd16affd640 1 -- 172.21.15.70:0/3315875174 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd16c0f12d0 con 0x7fd17410da70 2024-04-01T01:41:07.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.638+0000 7fd17e993740 1 -- start start 2024-04-01T01:41:07.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.638+0000 7fd17e993740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636745498e0 0x563674549d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.638+0000 7fd17e993740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56367454a2d0 0x56367454edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.638+0000 7fd17e993740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x5636745518d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.638+0000 7fd17e993740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56367449c8f0 con 0x5636745498e0 2024-04-01T01:41:07.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.638+0000 7fd17e993740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5636744458d0 con 0x56367454a2d0 2024-04-01T01:41:07.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.638+0000 7fd17e993740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563674498990 con 0x56367454f440 2024-04-01T01:41:07.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.638+0000 7fd17a26a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x5636745518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:07.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.638+0000 7fd17a26a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x5636745518d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:48606/0 (socket says 172.21.15.70:48606) 2024-04-01T01:41:07.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.638+0000 7fd17a26a640 1 -- 172.21.15.70:0/2345341485 learned_addr learned my addr 172.21.15.70:0/2345341485 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:07.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.639+0000 7fd17a26a640 1 -- 172.21.15.70:0/2345341485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56367454a2d0 msgr2=0x56367454edd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:41:07.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.639+0000 7fd17a26a640 1 --2- 172.21.15.70:0/2345341485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56367454a2d0 0x56367454edd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.639+0000 7fd17a26a640 1 -- 172.21.15.70:0/2345341485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636745498e0 msgr2=0x563674549d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:07.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.639+0000 7fd17a26a640 1 --2- 172.21.15.70:0/2345341485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636745498e0 0x563674549d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.639+0000 7fd17a26a640 1 -- 172.21.15.70:0/2345341485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636741b1210 con 0x56367454f440 2024-04-01T01:41:07.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.639+0000 7fd17a26a640 1 --2- 172.21.15.70:0/2345341485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x5636745518d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd16c0bda60 tx=0x7fd16c004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=842dfe8cbe8fcd3d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:07.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.639+0000 7fd15ffff640 1 -- 172.21.15.70:0/2345341485 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c0eee60 con 0x56367454f440 2024-04-01T01:41:07.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.639+0000 7fd15ffff640 1 -- 172.21.15.70:0/2345341485 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd16c0be080 con 0x56367454f440 2024-04-01T01:41:07.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.640+0000 7fd179268640 1 --2- 172.21.15.70:0/3315875174 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1600a26b0 0x7fd1600a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:07.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.640+0000 7fd17e993740 1 -- 172.21.15.70:0/2345341485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 msgr2=0x5636745518d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:07.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.640+0000 7fd17e993740 1 --2- 172.21.15.70:0/2345341485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x5636745518d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd16c0bda60 tx=0x7fd16c004040 comp rx=0 tx=0).stop 2024-04-01T01:41:07.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.641+0000 7fd17e993740 1 -- 172.21.15.70:0/2345341485 shutdown_connections 2024-04-01T01:41:07.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.641+0000 7fd17e993740 1 --2- 172.21.15.70:0/2345341485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x5636745518d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.641+0000 7fd17e993740 1 --2- 172.21.15.70:0/2345341485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56367454a2d0 0x56367454edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.641+0000 7fd17e993740 1 --2- 172.21.15.70:0/2345341485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636745498e0 0x563674549d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.641+0000 7fd17e993740 1 -- 172.21.15.70:0/2345341485 >> 172.21.15.70:0/2345341485 conn(0x563674533e40 msgr2=0x563674534240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:07.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.641+0000 7fd179268640 1 --2- 172.21.15.70:0/3315875174 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1600a26b0 0x7fd1600a4b70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x56367454a6d0 tx=0x7fd164009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:07.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.642+0000 7fd17e993740 1 -- 172.21.15.70:0/2345341485 shutdown_connections 2024-04-01T01:41:07.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.643+0000 7fd17e993740 1 -- 172.21.15.70:0/2345341485 wait complete. 2024-04-01T01:41:07.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.644+0000 7fd17e993740 1 -- start start 2024-04-01T01:41:07.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.644+0000 7fd17e993740 1 Processor -- start 2024-04-01T01:41:07.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.644+0000 7fd17e993740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636745498e0 0x5636745538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.644+0000 7fd17e993740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56367454a2d0 0x563674553e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.645+0000 7fd17e993740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x56367455ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.645+0000 7fd17e993740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5636745521b0 con 0x56367454a2d0 2024-04-01T01:41:07.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.645+0000 7fd17e993740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56367443af80 con 0x5636745498e0 2024-04-01T01:41:07.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.645+0000 7fd17e993740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563674552330 con 0x56367454f440 2024-04-01T01:41:07.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.645+0000 7fd17e993740 10 client.0 _ll_register_callbacks cb 0x563674493290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:07.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.645+0000 7fd179a69640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636745498e0 0x5636745538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:07.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.645+0000 7fd179a69640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636745498e0 0x5636745538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35616/0 (socket says 172.21.15.70:35616) 2024-04-01T01:41:07.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.645+0000 7fd179a69640 1 -- 172.21.15.70:0/255860105 learned_addr learned my addr 172.21.15.70:0/255860105 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:07.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.645+0000 7fd179268640 1 --2- 172.21.15.70:0/255860105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56367454a2d0 0x563674553e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:07.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.646+0000 7fd179a69640 1 -- 172.21.15.70:0/255860105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 msgr2=0x56367455ae60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:41:07.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.646+0000 7fd179a69640 1 --2- 172.21.15.70:0/255860105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x56367455ae60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.646+0000 7fd179a69640 1 -- 172.21.15.70:0/255860105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56367454a2d0 msgr2=0x563674553e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:07.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.646+0000 7fd179a69640 1 --2- 172.21.15.70:0/255860105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56367454a2d0 0x563674553e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.646+0000 7fd179a69640 1 -- 172.21.15.70:0/255860105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56367427d620 con 0x5636745498e0 2024-04-01T01:41:07.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.646+0000 7fd179a69640 1 --2- 172.21.15.70:0/255860105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636745498e0 0x5636745538e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd17000e370 tx=0x7fd17000e830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c7230c467f7679a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:07.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.646+0000 7fd15d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:07.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.646+0000 7fd15d7fa640 1 -- 172.21.15.70:0/255860105 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd170016070 con 0x5636745498e0 2024-04-01T01:41:07.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.646+0000 7fd17e993740 1 -- 172.21.15.70:0/255860105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd16c0be080 con 0x5636745498e0 2024-04-01T01:41:07.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.646+0000 7fd15d7fa640 1 -- 172.21.15.70:0/255860105 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd1700108a0 con 0x5636745498e0 2024-04-01T01:41:07.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.646+0000 7fd15d7fa640 1 -- 172.21.15.70:0/255860105 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd170022050 con 0x5636745498e0 2024-04-01T01:41:07.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.647+0000 7fd17e993740 10 client.27788 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:07.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.647+0000 7fd17e993740 1 -- 172.21.15.70:0/255860105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56367455df30 con 0x5636745498e0 2024-04-01T01:41:07.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.647+0000 7fd15d7fa640 1 -- 172.21.15.70:0/255860105 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd16c0be080 con 0x5636745498e0 2024-04-01T01:41:07.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.647+0000 7fd15d7fa640 1 -- 172.21.15.70:0/255860105 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd17000f040 con 0x5636745498e0 2024-04-01T01:41:07.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.647+0000 7fd17e993740 10 client.27788 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:07.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.647+0000 7fd17e993740 10 client.27788 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:07.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.647+0000 7fd17e993740 20 client.27788 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:07.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.647+0000 7fd17e993740 1 --2- 172.21.15.70:0/255860105 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56367455fa70 0x563674561f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.647+0000 7fd17e993740 4 client.27788 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:07.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.647+0000 7fd17e993740 1 -- 172.21.15.70:0/255860105 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636744b0120 con 0x56367455fa70 2024-04-01T01:41:07.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.648+0000 7fd179268640 1 --2- 172.21.15.70:0/255860105 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56367455fa70 0x563674561f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:07.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.651+0000 7fd15d7fa640 10 client.27788 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:07.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.651+0000 7fd179268640 1 --2- 172.21.15.70:0/255860105 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56367455fa70 0x563674561f30 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56ec9a1a33417aec server_cookie=1a7b1fb928d374f0 in_seq=0 out_seq=0 2024-04-01T01:41:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.655+0000 7fd15d7fa640 1 -- 172.21.15.70:0/255860105 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5636744b0120 con 0x56367455fa70 2024-04-01T01:41:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.655+0000 7fd15d7fa640 10 client.27788 handle_command_reply: tid=0 2024-04-01T01:41:07.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.655+0000 7fd17e993740 1 client.27788 shutdown 2024-04-01T01:41:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.656+0000 7fd17e993740 1 -- 172.21.15.70:0/255860105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636745498e0 msgr2=0x5636745538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:07.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.656+0000 7fd17e993740 1 --2- 172.21.15.70:0/255860105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636745498e0 0x5636745538e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd17000e370 tx=0x7fd17000e830 comp rx=0 tx=0).stop 2024-04-01T01:41:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.657+0000 7fd17e993740 1 -- 172.21.15.70:0/255860105 shutdown_connections 2024-04-01T01:41:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.657+0000 7fd17e993740 1 --2- 172.21.15.70:0/255860105 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56367455fa70 0x563674561f30 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.657+0000 7fd17e993740 1 --2- 172.21.15.70:0/255860105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x56367455ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.657+0000 7fd17e993740 1 --2- 172.21.15.70:0/255860105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56367454a2d0 0x563674553e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.657+0000 7fd17e993740 1 --2- 172.21.15.70:0/255860105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636745498e0 0x5636745538e0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.657+0000 7fd17e993740 1 -- 172.21.15.70:0/255860105 >> 172.21.15.70:0/255860105 conn(0x563674533e40 msgr2=0x563674532f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:07.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.658+0000 7fd15d7fa640 0 client.27788 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:07.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.658+0000 7fd17e993740 1 -- 172.21.15.70:0/255860105 shutdown_connections 2024-04-01T01:41:07.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.658+0000 7fd17e993740 1 -- 172.21.15.70:0/255860105 wait complete. 2024-04-01T01:41:07.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.658+0000 7fd17e993740 20 client.27788 trim_cache size 0 max 16384 2024-04-01T01:41:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.689+0000 7fd17e993740 1 Processor -- start 2024-04-01T01:41:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.690+0000 7fd17e993740 1 -- start start 2024-04-01T01:41:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.690+0000 7fd17e993740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636745498e0 0x563674561eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.690+0000 7fd17e993740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56367454a2d0 0x56367455ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.690+0000 7fd179a69640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636745498e0 0x563674561eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.690+0000 7fd17e993740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x563674534220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.690+0000 7fd179a69640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636745498e0 0x563674561eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:35628/0 (socket says 172.21.15.70:35628) 2024-04-01T01:41:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.690+0000 7fd17e993740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd15800fd10 con 0x56367454a2d0 2024-04-01T01:41:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.690+0000 7fd17e993740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd158011850 con 0x5636745498e0 2024-04-01T01:41:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.690+0000 7fd17e993740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd1580103e0 con 0x56367454f440 2024-04-01T01:41:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.690+0000 7fd179a69640 1 -- 172.21.15.70:0/3883861767 learned_addr learned my addr 172.21.15.70:0/3883861767 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.690+0000 7fd17a26a640 1 --2- 172.21.15.70:0/3883861767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x563674534220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:07.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.691+0000 7fd179a69640 1 -- 172.21.15.70:0/3883861767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 msgr2=0x563674534220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:07.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.691+0000 7fd179a69640 1 --2- 172.21.15.70:0/3883861767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x563674534220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.691+0000 7fd179a69640 1 -- 172.21.15.70:0/3883861767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56367454a2d0 msgr2=0x56367455ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:07.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.691+0000 7fd179a69640 1 --2- 172.21.15.70:0/3883861767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56367454a2d0 0x56367455ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.691+0000 7fd179a69640 1 -- 172.21.15.70:0/3883861767 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563674549520 con 0x5636745498e0 2024-04-01T01:41:07.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.691+0000 7fd179a69640 1 --2- 172.21.15.70:0/3883861767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636745498e0 0x563674561eb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd170009b40 tx=0x7fd170009c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfb7b7362ddf4f39 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:07.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.691+0000 7fd15f7fe640 1 -- 172.21.15.70:0/3883861767 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd170016070 con 0x5636745498e0 2024-04-01T01:41:07.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.691+0000 7fd15f7fe640 1 -- 172.21.15.70:0/3883861767 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1700119a0 con 0x5636745498e0 2024-04-01T01:41:07.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.692+0000 7fd15f7fe640 1 -- 172.21.15.70:0/3883861767 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd170010be0 con 0x5636745498e0 2024-04-01T01:41:07.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.692+0000 7fd17e993740 1 -- 172.21.15.70:0/3883861767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636745498e0 msgr2=0x563674561eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:07.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.692+0000 7fd17e993740 1 --2- 172.21.15.70:0/3883861767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636745498e0 0x563674561eb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd170009b40 tx=0x7fd170009c60 comp rx=0 tx=0).stop 2024-04-01T01:41:07.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.692+0000 7fd17e993740 1 -- 172.21.15.70:0/3883861767 shutdown_connections 2024-04-01T01:41:07.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.692+0000 7fd17e993740 1 --2- 172.21.15.70:0/3883861767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x563674534220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.692+0000 7fd17e993740 1 --2- 172.21.15.70:0/3883861767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56367454a2d0 0x56367455ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.692+0000 7fd17e993740 1 --2- 172.21.15.70:0/3883861767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636745498e0 0x563674561eb0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.692+0000 7fd17e993740 1 -- 172.21.15.70:0/3883861767 >> 172.21.15.70:0/3883861767 conn(0x563674533e40 msgr2=0x56367454eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:07.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.695+0000 7fd17e993740 1 -- 172.21.15.70:0/3883861767 shutdown_connections 2024-04-01T01:41:07.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.695+0000 7fd17e993740 1 -- 172.21.15.70:0/3883861767 wait complete. 2024-04-01T01:41:07.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.695+0000 7fd17e993740 1 -- start start 2024-04-01T01:41:07.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.695+0000 7fd17e993740 1 Processor -- start 2024-04-01T01:41:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.696+0000 7fd17e993740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636745498e0 0x56367455ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.696+0000 7fd17e993740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56367454a2d0 0x563674557830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.696+0000 7fd17e993740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x563674557d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.696+0000 7fd17e993740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd158011180 con 0x5636745498e0 2024-04-01T01:41:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.696+0000 7fd17e993740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd158010ab0 con 0x56367454a2d0 2024-04-01T01:41:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.696+0000 7fd17e993740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd15800dcf0 con 0x56367454f440 2024-04-01T01:41:07.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.696+0000 7fd17e993740 10 client.0 _ll_register_callbacks cb 0x7fd1700092e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.696+0000 7fd17a26a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x563674557d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.696+0000 7fd17a26a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x563674557d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:48636/0 (socket says 172.21.15.70:48636) 2024-04-01T01:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.696+0000 7fd17a26a640 1 -- 172.21.15.70:0/3753247813 learned_addr learned my addr 172.21.15.70:0/3753247813 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.696+0000 7fd179268640 1 --2- 172.21.15.70:0/3753247813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56367454a2d0 0x563674557830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.697+0000 7fd179268640 1 -- 172.21.15.70:0/3753247813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 msgr2=0x563674557d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:07.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.697+0000 7fd179268640 1 --2- 172.21.15.70:0/3753247813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x563674557d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.697+0000 7fd179268640 1 -- 172.21.15.70:0/3753247813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636745498e0 msgr2=0x56367455ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.697+0000 7fd179268640 1 --2- 172.21.15.70:0/3753247813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636745498e0 0x56367455ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.697+0000 7fd179268640 1 -- 172.21.15.70:0/3753247813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636745c3e60 con 0x56367454a2d0 2024-04-01T01:41:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.698+0000 7fd179268640 1 --2- 172.21.15.70:0/3753247813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56367454a2d0 0x563674557830 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x563674562780 tx=0x7fd16400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31c814a2c666167b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:07.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.698+0000 7fd15d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.699+0000 7fd15d7fa640 1 -- 172.21.15.70:0/3753247813 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16401b040 con 0x56367454a2d0 2024-04-01T01:41:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.699+0000 7fd15d7fa640 1 -- 172.21.15.70:0/3753247813 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd164015ae0 con 0x56367454a2d0 2024-04-01T01:41:07.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.699+0000 7fd17e993740 1 -- 172.21.15.70:0/3753247813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1700119a0 con 0x56367454a2d0 2024-04-01T01:41:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.699+0000 7fd15d7fa640 1 -- 172.21.15.70:0/3753247813 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd164015ce0 con 0x56367454a2d0 2024-04-01T01:41:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.699+0000 7fd17e993740 10 client.27794 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.699+0000 7fd17e993740 1 -- 172.21.15.70:0/3753247813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5636745c5120 con 0x56367454a2d0 2024-04-01T01:41:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.700+0000 7fd15d7fa640 1 -- 172.21.15.70:0/3753247813 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1700119a0 con 0x56367454a2d0 2024-04-01T01:41:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.700+0000 7fd15d7fa640 1 -- 172.21.15.70:0/3753247813 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd164022810 con 0x56367454a2d0 2024-04-01T01:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.700+0000 7fd17e993740 10 client.27794 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.700+0000 7fd17e993740 10 client.27794 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.700+0000 7fd17e993740 20 client.27794 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.700+0000 7fd17e993740 1 --2- 172.21.15.70:0/3753247813 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5636745c5ad0 0x5636745c7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.700+0000 7fd17e993740 4 client.27794 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:41:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.700+0000 7fd17e993740 1 -- 172.21.15.70:0/3753247813 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5636744a9e30 con 0x5636745c5ad0 2024-04-01T01:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.701+0000 7fd179a69640 1 --2- 172.21.15.70:0/3753247813 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5636745c5ad0 0x5636745c7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.702+0000 7fd15d7fa640 10 client.27794 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.702+0000 7fd179a69640 1 --2- 172.21.15.70:0/3753247813 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5636745c5ad0 0x5636745c7ef0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca2bd98961019e1d server_cookie=9f5cdcc415d2cd50 in_seq=0 out_seq=0 2024-04-01T01:41:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.748+0000 7fd15d7fa640 1 -- 172.21.15.70:0/3753247813 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5636744a9e30 con 0x5636745c5ad0 2024-04-01T01:41:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.748+0000 7fd15d7fa640 10 client.27794 handle_command_reply: tid=0 2024-04-01T01:41:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.748+0000 7fd17e993740 1 client.27794 shutdown 2024-04-01T01:41:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.748+0000 7fd17e993740 1 -- 172.21.15.70:0/3753247813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56367454a2d0 msgr2=0x563674557830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.748+0000 7fd17e993740 1 --2- 172.21.15.70:0/3753247813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56367454a2d0 0x563674557830 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x563674562780 tx=0x7fd16400eeb0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.751+0000 7fd17e993740 1 -- 172.21.15.70:0/3753247813 shutdown_connections 2024-04-01T01:41:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.751+0000 7fd17e993740 1 --2- 172.21.15.70:0/3753247813 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5636745c5ad0 0x5636745c7ef0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.751+0000 7fd17e993740 1 --2- 172.21.15.70:0/3753247813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56367454f440 0x563674557d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.751+0000 7fd17e993740 1 --2- 172.21.15.70:0/3753247813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56367454a2d0 0x563674557830 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.751+0000 7fd17e993740 1 --2- 172.21.15.70:0/3753247813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5636745498e0 0x56367455ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.751+0000 7fd17e993740 1 -- 172.21.15.70:0/3753247813 >> 172.21.15.70:0/3753247813 conn(0x563674533e40 msgr2=0x56367454b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.751+0000 7fd15d7fa640 0 client.27794 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.751+0000 7fd17e993740 1 -- 172.21.15.70:0/3753247813 shutdown_connections 2024-04-01T01:41:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.751+0000 7fd17e993740 1 -- 172.21.15.70:0/3753247813 wait complete. 2024-04-01T01:41:07.752 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.752+0000 7fd17e993740 20 client.27794 trim_cache size 0 max 16384 2024-04-01T01:41:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:41:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:41:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:41:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:41:07.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.758+0000 7fd17bccd640 1 -- 172.21.15.70:0/3315875174 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1600a26b0 msgr2=0x7fd1600a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:07.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.758+0000 7fd17bccd640 1 --2- 172.21.15.70:0/3315875174 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1600a26b0 0x7fd1600a4b70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x56367454a6d0 tx=0x7fd164009110 comp rx=0 tx=0).stop 2024-04-01T01:41:07.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.758+0000 7fd17bccd640 1 -- 172.21.15.70:0/3315875174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17410da70 msgr2=0x7fd1741adc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:07.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.758+0000 7fd17bccd640 1 --2- 172.21.15.70:0/3315875174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17410da70 0x7fd1741adc70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd16c034fd0 tx=0x7fd16c033c40 comp rx=0 tx=0).stop 2024-04-01T01:41:07.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.758+0000 7fd17bccd640 1 -- 172.21.15.70:0/3315875174 shutdown_connections 2024-04-01T01:41:07.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.758+0000 7fd17bccd640 1 --2- 172.21.15.70:0/3315875174 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1600a26b0 0x7fd1600a4b70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.758+0000 7fd17bccd640 1 --2- 172.21.15.70:0/3315875174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd17410da70 0x7fd1741adc70 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.758+0000 7fd17bccd640 1 --2- 172.21.15.70:0/3315875174 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd17410b0c0 0x7fd1741a6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.758+0000 7fd17bccd640 1 --2- 172.21.15.70:0/3315875174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd174108750 0x7fd1741a6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:07.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.758+0000 7fd17bccd640 1 -- 172.21.15.70:0/3315875174 >> 172.21.15.70:0/3315875174 conn(0x7fd17406e790 msgr2=0x7fd17410a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:07.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.761+0000 7fd17bccd640 1 -- 172.21.15.70:0/3315875174 shutdown_connections 2024-04-01T01:41:07.763 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:07.763+0000 7fd17bccd640 1 -- 172.21.15.70:0/3315875174 wait complete. 2024-04-01T01:41:09.128 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:08 smithi081 ceph-mon[35142]: pgmap v589: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 9.3 MiB/s wr, 978 op/s 2024-04-01T01:41:09.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:08 smithi189 ceph-mon[35753]: pgmap v589: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 9.3 MiB/s wr, 978 op/s 2024-04-01T01:41:09.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:08 smithi070 ceph-mon[32179]: pgmap v589: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 9.3 MiB/s wr, 978 op/s 2024-04-01T01:41:11.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:10 smithi189 ceph-mon[35753]: pgmap v590: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 8.0 MiB/s wr, 891 op/s 2024-04-01T01:41:11.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:10 smithi070 ceph-mon[32179]: pgmap v590: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 8.0 MiB/s wr, 891 op/s 2024-04-01T01:41:11.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:10 smithi081 ceph-mon[35142]: pgmap v590: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 8.0 MiB/s wr, 891 op/s 2024-04-01T01:41:11.897 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 385.0 s, 844.6 tps, lat 115.814 ms stddev 86.937 2024-04-01T01:41:13.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:12 smithi189 ceph-mon[35753]: pgmap v591: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 9.2 MiB/s wr, 1.03k op/s 2024-04-01T01:41:13.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:12 smithi070 ceph-mon[32179]: pgmap v591: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 9.2 MiB/s wr, 1.03k op/s 2024-04-01T01:41:13.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:12 smithi081 ceph-mon[35142]: pgmap v591: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 9.2 MiB/s wr, 1.03k op/s 2024-04-01T01:41:15.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:14 smithi189 ceph-mon[35753]: pgmap v592: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 8.1 MiB/s wr, 917 op/s 2024-04-01T01:41:15.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:14 smithi070 ceph-mon[32179]: pgmap v592: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 8.1 MiB/s wr, 917 op/s 2024-04-01T01:41:15.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:14 smithi081 ceph-mon[35142]: pgmap v592: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 8.1 MiB/s wr, 917 op/s 2024-04-01T01:41:16.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 390.0 s, 879.0 tps, lat 115.371 ms stddev 120.166 2024-04-01T01:41:17.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:16 smithi189 ceph-mon[35753]: pgmap v593: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 9.1 MiB/s wr, 1.02k op/s 2024-04-01T01:41:17.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:16 smithi070 ceph-mon[32179]: pgmap v593: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 9.1 MiB/s wr, 1.02k op/s 2024-04-01T01:41:17.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:16 smithi081 ceph-mon[35142]: pgmap v593: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 9.1 MiB/s wr, 1.02k op/s 2024-04-01T01:41:19.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:18 smithi189 ceph-mon[35753]: pgmap v594: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 8.3 MiB/s wr, 935 op/s 2024-04-01T01:41:19.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:18 smithi070 ceph-mon[32179]: pgmap v594: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 8.3 MiB/s wr, 935 op/s 2024-04-01T01:41:19.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:18 smithi081 ceph-mon[35142]: pgmap v594: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 8.3 MiB/s wr, 935 op/s 2024-04-01T01:41:20.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.917+0000 7fae80ceb640 1 -- 172.21.15.70:0/77376291 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae7c108750 msgr2=0x7fae7c10ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:20.918 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.917+0000 7fae80ceb640 1 --2- 172.21.15.70:0/77376291 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae7c108750 0x7fae7c10ab80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009900 tx=0x7fae6c02bc80 comp rx=0 tx=0).stop 2024-04-01T01:41:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.918+0000 7fae80ceb640 1 -- 172.21.15.70:0/77376291 shutdown_connections 2024-04-01T01:41:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.918+0000 7fae80ceb640 1 --2- 172.21.15.70:0/77376291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7c10da70 0x7fae7c10def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.918+0000 7fae80ceb640 1 --2- 172.21.15.70:0/77376291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae7c10b0c0 0x7fae7c10d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.918+0000 7fae80ceb640 1 --2- 172.21.15.70:0/77376291 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae7c108750 0x7fae7c10ab80 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.918+0000 7fae80ceb640 1 -- 172.21.15.70:0/77376291 >> 172.21.15.70:0/77376291 conn(0x7fae7c06e790 msgr2=0x7fae7c070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.920+0000 7fae80ceb640 1 -- 172.21.15.70:0/77376291 shutdown_connections 2024-04-01T01:41:20.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.920+0000 7fae80ceb640 1 -- 172.21.15.70:0/77376291 wait complete. 2024-04-01T01:41:20.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.921+0000 7fae80ceb640 1 Processor -- start 2024-04-01T01:41:20.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.921+0000 7fae80ceb640 1 -- start start 2024-04-01T01:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.922+0000 7fae80ceb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae7c108750 0x7fae7c1a6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.922+0000 7fae80ceb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae7c10b0c0 0x7fae7c1a6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.922+0000 7fae80ceb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7c10da70 0x7fae7c1ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.922+0000 7fae80ceb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c11c1d0 con 0x7fae7c10da70 2024-04-01T01:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.922+0000 7fae80ceb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c11c050 con 0x7fae7c10b0c0 2024-04-01T01:41:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.922+0000 7fae80ceb640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c11c350 con 0x7fae7c108750 2024-04-01T01:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.922+0000 7fae7a59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae7c108750 0x7fae7c1a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.922+0000 7fae7a59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae7c108750 0x7fae7c1a6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:47316/0 (socket says 172.21.15.70:47316) 2024-04-01T01:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.922+0000 7fae7a59c640 1 -- 172.21.15.70:0/1675254166 learned_addr learned my addr 172.21.15.70:0/1675254166 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.922+0000 7fae7ad9d640 1 --2- 172.21.15.70:0/1675254166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7c10da70 0x7fae7c1ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.923+0000 7fae7a59c640 1 -- 172.21.15.70:0/1675254166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae7c10b0c0 msgr2=0x7fae7c1a6db0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:41:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.923+0000 7fae7a59c640 1 --2- 172.21.15.70:0/1675254166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae7c10b0c0 0x7fae7c1a6db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.923+0000 7fae7a59c640 1 -- 172.21.15.70:0/1675254166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7c10da70 msgr2=0x7fae7c1ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.923+0000 7fae7a59c640 1 --2- 172.21.15.70:0/1675254166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7c10da70 0x7fae7c1ade20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.923+0000 7fae7a59c640 1 -- 172.21.15.70:0/1675254166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae7c1ae490 con 0x7fae7c108750 2024-04-01T01:41:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.923+0000 7fae7a59c640 1 --2- 172.21.15.70:0/1675254166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae7c108750 0x7fae7c1a6870 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009230 tx=0x7fae6c0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.924+0000 7fae73fff640 1 -- 172.21.15.70:0/1675254166 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c043020 con 0x7fae7c108750 2024-04-01T01:41:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.924+0000 7fae73fff640 1 -- 172.21.15.70:0/1675254166 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae6c0054c0 con 0x7fae7c108750 2024-04-01T01:41:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.924+0000 7fae73fff640 1 -- 172.21.15.70:0/1675254166 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c003d10 con 0x7fae7c108750 2024-04-01T01:41:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.924+0000 7fae80ceb640 1 -- 172.21.15.70:0/1675254166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae7c1ae720 con 0x7fae7c108750 2024-04-01T01:41:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.924+0000 7fae80ceb640 1 -- 172.21.15.70:0/1675254166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae7c1aeca0 con 0x7fae7c108750 2024-04-01T01:41:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.927+0000 7fae73fff640 1 -- 172.21.15.70:0/1675254166 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fae6c046020 con 0x7fae7c108750 2024-04-01T01:41:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.928+0000 7fae73fff640 1 --2- 172.21.15.70:0/1675254166 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fae680a25c0 0x7fae680a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.928+0000 7fae73fff640 1 -- 172.21.15.70:0/1675254166 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fae6c0f0e60 con 0x7fae7c108750 2024-04-01T01:41:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.929+0000 7fae71d9b640 1 --2- 172.21.15.70:0/1675254166 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fae680a25c0 0x7fae680a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.929+0000 7fae839a4740 1 Processor -- start 2024-04-01T01:41:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.929+0000 7fae839a4740 1 -- start start 2024-04-01T01:41:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.929+0000 7fae71d9b640 1 --2- 172.21.15.70:0/1675254166 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fae680a25c0 0x7fae680a4a80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fae7c1a7d80 tx=0x7fae64006bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:20.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.929+0000 7fae839a4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a2708d0 0x55b05a270cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.929+0000 7fae839a4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2712c0 0x55b05a275dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.929+0000 7fae839a4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a276430 0x55b05a2788c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.929+0000 7fae839a4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b05a1d48f0 con 0x55b05a2712c0 2024-04-01T01:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.929+0000 7fae839a4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b05a17d8d0 con 0x55b05a276430 2024-04-01T01:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.929+0000 7fae839a4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b05a1d0990 con 0x55b05a2708d0 2024-04-01T01:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.930+0000 7fae71d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2712c0 0x55b05a275dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.930+0000 7fae71d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2712c0 0x55b05a275dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41074/0 (socket says 172.21.15.70:41074) 2024-04-01T01:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.930+0000 7fae71d9b640 1 -- 172.21.15.70:0/337900910 learned_addr learned my addr 172.21.15.70:0/337900910 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.930+0000 7fae7a59c640 1 --2- 172.21.15.70:0/337900910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a2708d0 0x55b05a270cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.930+0000 7fae7ad9d640 1 --2- 172.21.15.70:0/337900910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a276430 0x55b05a2788c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.930+0000 7fae7a59c640 1 -- 172.21.15.70:0/337900910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a276430 msgr2=0x55b05a2788c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.930+0000 7fae7a59c640 1 --2- 172.21.15.70:0/337900910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a276430 0x55b05a2788c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.930+0000 7fae7a59c640 1 -- 172.21.15.70:0/337900910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2712c0 msgr2=0x55b05a275dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:20.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.930+0000 7fae7a59c640 1 --2- 172.21.15.70:0/337900910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2712c0 0x55b05a275dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.930+0000 7fae7a59c640 1 -- 172.21.15.70:0/337900910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b059ee9210 con 0x55b05a2708d0 2024-04-01T01:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.930+0000 7fae7a59c640 1 --2- 172.21.15.70:0/337900910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a2708d0 0x55b05a270cf0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0ee530 tx=0x7fae6c0095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ba216c46e8286e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.930+0000 7fae70d99640 1 -- 172.21.15.70:0/337900910 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c0d5070 con 0x55b05a2708d0 2024-04-01T01:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.931+0000 7fae70d99640 1 -- 172.21.15.70:0/337900910 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae6c0b9040 con 0x55b05a2708d0 2024-04-01T01:41:20.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.931+0000 7fae70d99640 1 -- 172.21.15.70:0/337900910 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c0c6020 con 0x55b05a2708d0 2024-04-01T01:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.932+0000 7fae839a4740 1 -- 172.21.15.70:0/337900910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a2708d0 msgr2=0x55b05a270cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.932+0000 7fae839a4740 1 --2- 172.21.15.70:0/337900910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a2708d0 0x55b05a270cf0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0ee530 tx=0x7fae6c0095e0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.933+0000 7fae839a4740 1 -- 172.21.15.70:0/337900910 shutdown_connections 2024-04-01T01:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.933+0000 7fae839a4740 1 --2- 172.21.15.70:0/337900910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a276430 0x55b05a2788c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.933+0000 7fae839a4740 1 --2- 172.21.15.70:0/337900910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2712c0 0x55b05a275dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.933+0000 7fae839a4740 1 --2- 172.21.15.70:0/337900910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a2708d0 0x55b05a270cf0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.933+0000 7fae839a4740 1 -- 172.21.15.70:0/337900910 >> 172.21.15.70:0/337900910 conn(0x55b05a26be40 msgr2=0x55b05a26c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.933+0000 7fae839a4740 1 -- 172.21.15.70:0/337900910 shutdown_connections 2024-04-01T01:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.933+0000 7fae839a4740 1 -- 172.21.15.70:0/337900910 wait complete. 2024-04-01T01:41:20.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.934+0000 7fae839a4740 1 -- start start 2024-04-01T01:41:20.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.934+0000 7fae839a4740 1 Processor -- start 2024-04-01T01:41:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.934+0000 7fae839a4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2708d0 0x55b05a27a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.935+0000 7fae839a4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a2712c0 0x55b05a27ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.935+0000 7fae7a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2708d0 0x55b05a27a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.935+0000 7fae7a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2708d0 0x55b05a27a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41084/0 (socket says 172.21.15.70:41084) 2024-04-01T01:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.935+0000 7fae7a59c640 1 -- 172.21.15.70:0/2609062936 learned_addr learned my addr 172.21.15.70:0/2609062936 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.935+0000 7fae839a4740 1 --2- 172.21.15.70:0/2609062936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a276430 0x55b05a281d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.935+0000 7fae839a4740 1 -- 172.21.15.70:0/2609062936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b05a2791a0 con 0x55b05a2708d0 2024-04-01T01:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.935+0000 7fae7ad9d640 1 --2- 172.21.15.70:0/2609062936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a276430 0x55b05a281d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.935+0000 7fae71d9b640 1 --2- 172.21.15.70:0/2609062936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a2712c0 0x55b05a27ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.935+0000 7fae839a4740 1 -- 172.21.15.70:0/2609062936 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b05a172f80 con 0x55b05a2712c0 2024-04-01T01:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.935+0000 7fae839a4740 1 -- 172.21.15.70:0/2609062936 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b05a279320 con 0x55b05a276430 2024-04-01T01:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.935+0000 7fae839a4740 10 client.0 _ll_register_callbacks cb 0x55b05a1cb290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.936+0000 7fae71d9b640 1 -- 172.21.15.70:0/2609062936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a276430 msgr2=0x55b05a281d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.936+0000 7fae71d9b640 1 --2- 172.21.15.70:0/2609062936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a276430 0x55b05a281d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.936+0000 7fae71d9b640 1 -- 172.21.15.70:0/2609062936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2708d0 msgr2=0x55b05a27a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.936+0000 7fae71d9b640 1 --2- 172.21.15.70:0/2609062936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2708d0 0x55b05a27a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.936+0000 7fae71d9b640 1 -- 172.21.15.70:0/2609062936 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b059fb5620 con 0x55b05a2712c0 2024-04-01T01:41:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.936+0000 7fae71d9b640 1 --2- 172.21.15.70:0/2609062936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a2712c0 0x55b05a27ad50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fae64009ed0 tx=0x7fae640046e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1c757796565392b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:20.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.936+0000 7fae61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.936+0000 7fae61ffb640 1 -- 172.21.15.70:0/2609062936 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae64023020 con 0x55b05a2712c0 2024-04-01T01:41:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.937+0000 7fae61ffb640 1 -- 172.21.15.70:0/2609062936 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fae6401f850 con 0x55b05a2712c0 2024-04-01T01:41:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.937+0000 7fae839a4740 1 -- 172.21.15.70:0/2609062936 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae6c0b9040 con 0x55b05a2712c0 2024-04-01T01:41:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.937+0000 7fae61ffb640 1 -- 172.21.15.70:0/2609062936 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fae6402e050 con 0x55b05a2712c0 2024-04-01T01:41:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.937+0000 7fae839a4740 10 client.27803 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.937+0000 7fae839a4740 1 -- 172.21.15.70:0/2609062936 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b05a284e30 con 0x55b05a2712c0 2024-04-01T01:41:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.938+0000 7fae61ffb640 1 -- 172.21.15.70:0/2609062936 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae6c0b9040 con 0x55b05a2712c0 2024-04-01T01:41:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.938+0000 7fae61ffb640 1 -- 172.21.15.70:0/2609062936 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fae64012070 con 0x55b05a2712c0 2024-04-01T01:41:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.938+0000 7fae839a4740 10 client.27803 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.938+0000 7fae839a4740 10 client.27803 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.938+0000 7fae839a4740 20 client.27803 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.938+0000 7fae839a4740 1 --2- 172.21.15.70:0/2609062936 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b05a286570 0x55b05a288980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.938+0000 7fae839a4740 4 client.27803 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:20.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.938+0000 7fae839a4740 1 -- 172.21.15.70:0/2609062936 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b05a1e8120 con 0x55b05a286570 2024-04-01T01:41:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.939+0000 7fae7a59c640 1 --2- 172.21.15.70:0/2609062936 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b05a286570 0x55b05a288980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.940+0000 7fae7a59c640 1 --2- 172.21.15.70:0/2609062936 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b05a286570 0x55b05a288980 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a64f1e168f60510 server_cookie=c62a50d96f9e0721 in_seq=0 out_seq=0 2024-04-01T01:41:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.940+0000 7fae61ffb640 10 client.27803 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.941+0000 7fae61ffb640 1 -- 172.21.15.70:0/2609062936 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b05a1e8120 con 0x55b05a286570 2024-04-01T01:41:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.941+0000 7fae61ffb640 10 client.27803 handle_command_reply: tid=0 2024-04-01T01:41:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.941+0000 7fae839a4740 1 client.27803 shutdown 2024-04-01T01:41:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.942+0000 7fae839a4740 1 -- 172.21.15.70:0/2609062936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a2712c0 msgr2=0x55b05a27ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:20.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.942+0000 7fae839a4740 1 --2- 172.21.15.70:0/2609062936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a2712c0 0x55b05a27ad50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fae64009ed0 tx=0x7fae640046e0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.943+0000 7fae839a4740 1 -- 172.21.15.70:0/2609062936 shutdown_connections 2024-04-01T01:41:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.943+0000 7fae839a4740 1 --2- 172.21.15.70:0/2609062936 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b05a286570 0x55b05a288980 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.943+0000 7fae839a4740 1 --2- 172.21.15.70:0/2609062936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a276430 0x55b05a281d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.943+0000 7fae839a4740 1 --2- 172.21.15.70:0/2609062936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a2712c0 0x55b05a27ad50 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.943+0000 7fae839a4740 1 --2- 172.21.15.70:0/2609062936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2708d0 0x55b05a27a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.943+0000 7fae839a4740 1 -- 172.21.15.70:0/2609062936 >> 172.21.15.70:0/2609062936 conn(0x55b05a26be40 msgr2=0x55b05a26af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.944+0000 7fae61ffb640 0 client.27803 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.944+0000 7fae839a4740 1 -- 172.21.15.70:0/2609062936 shutdown_connections 2024-04-01T01:41:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.944+0000 7fae839a4740 1 -- 172.21.15.70:0/2609062936 wait complete. 2024-04-01T01:41:20.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.944+0000 7fae839a4740 20 client.27803 trim_cache size 0 max 16384 2024-04-01T01:41:20.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.972+0000 7fae839a4740 1 Processor -- start 2024-04-01T01:41:20.972 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.972+0000 7fae839a4740 1 -- start start 2024-04-01T01:41:20.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.973+0000 7fae839a4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a2708d0 0x55b05a26c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.973+0000 7fae839a4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2712c0 0x55b05a281d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.973+0000 7fae7a59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a2708d0 0x55b05a26c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.973+0000 7fae7a59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a2708d0 0x55b05a26c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:47336/0 (socket says 172.21.15.70:47336) 2024-04-01T01:41:20.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.973+0000 7fae7a59c640 1 -- 172.21.15.70:0/2349695221 learned_addr learned my addr 172.21.15.70:0/2349695221 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:20.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.973+0000 7fae839a4740 1 --2- 172.21.15.70:0/2349695221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a276430 0x55b05a28e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.973+0000 7fae839a4740 1 -- 172.21.15.70:0/2349695221 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c1b9200 con 0x55b05a2712c0 2024-04-01T01:41:20.974 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.973+0000 7fae71d9b640 1 --2- 172.21.15.70:0/2349695221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2712c0 0x55b05a281d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.973+0000 7fae839a4740 1 -- 172.21.15.70:0/2349695221 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c1bad40 con 0x55b05a276430 2024-04-01T01:41:20.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.974+0000 7fae7ad9d640 1 --2- 172.21.15.70:0/2349695221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a276430 0x55b05a28e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.974+0000 7fae839a4740 1 -- 172.21.15.70:0/2349695221 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c1b98d0 con 0x55b05a2708d0 2024-04-01T01:41:20.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.974+0000 7fae7a59c640 1 -- 172.21.15.70:0/2349695221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a276430 msgr2=0x55b05a28e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:20.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.974+0000 7fae7a59c640 1 --2- 172.21.15.70:0/2349695221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a276430 0x55b05a28e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.974+0000 7fae7a59c640 1 -- 172.21.15.70:0/2349695221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2712c0 msgr2=0x55b05a281d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.974+0000 7fae7a59c640 1 --2- 172.21.15.70:0/2349695221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2712c0 0x55b05a281d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.974+0000 7fae7a59c640 1 -- 172.21.15.70:0/2349695221 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b05a209240 con 0x55b05a2708d0 2024-04-01T01:41:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.975+0000 7fae7a59c640 1 --2- 172.21.15.70:0/2349695221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a2708d0 0x55b05a26c220 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0bd0e0 tx=0x7fae6c0044e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77ce99f15a82e4b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.975+0000 7fae63fff640 1 -- 172.21.15.70:0/2349695221 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c0d5040 con 0x55b05a2708d0 2024-04-01T01:41:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.975+0000 7fae63fff640 1 -- 172.21.15.70:0/2349695221 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae6c087070 con 0x55b05a2708d0 2024-04-01T01:41:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.975+0000 7fae63fff640 1 -- 172.21.15.70:0/2349695221 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c0ee640 con 0x55b05a2708d0 2024-04-01T01:41:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.976+0000 7fae839a4740 1 -- 172.21.15.70:0/2349695221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a2708d0 msgr2=0x55b05a26c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.976+0000 7fae839a4740 1 --2- 172.21.15.70:0/2349695221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a2708d0 0x55b05a26c220 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0bd0e0 tx=0x7fae6c0044e0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.976+0000 7fae839a4740 1 -- 172.21.15.70:0/2349695221 shutdown_connections 2024-04-01T01:41:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.976+0000 7fae839a4740 1 --2- 172.21.15.70:0/2349695221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a276430 0x55b05a28e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.977+0000 7fae839a4740 1 --2- 172.21.15.70:0/2349695221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2712c0 0x55b05a281d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.977+0000 7fae839a4740 1 --2- 172.21.15.70:0/2349695221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a2708d0 0x55b05a26c220 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.977+0000 7fae839a4740 1 -- 172.21.15.70:0/2349695221 >> 172.21.15.70:0/2349695221 conn(0x55b05a26be40 msgr2=0x55b05a275af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.977+0000 7fae839a4740 1 -- 172.21.15.70:0/2349695221 shutdown_connections 2024-04-01T01:41:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.977+0000 7fae839a4740 1 -- 172.21.15.70:0/2349695221 wait complete. 2024-04-01T01:41:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.977+0000 7fae839a4740 1 -- start start 2024-04-01T01:41:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.978+0000 7fae839a4740 1 Processor -- start 2024-04-01T01:41:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.978+0000 7fae839a4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2708d0 0x55b05a27eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.978+0000 7fae839a4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a2712c0 0x55b05a27efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.979+0000 7fae839a4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a276430 0x55b05a2ea8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.979+0000 7fae71d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a2712c0 0x55b05a27efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.979+0000 7fae71d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a2712c0 0x55b05a27efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:36960/0 (socket says 172.21.15.70:36960) 2024-04-01T01:41:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.979+0000 7fae7a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2708d0 0x55b05a27eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.979+0000 7fae7a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2708d0 0x55b05a27eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41112/0 (socket says 172.21.15.70:41112) 2024-04-01T01:41:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.979+0000 7fae71d9b640 1 -- 172.21.15.70:0/1261028069 learned_addr learned my addr 172.21.15.70:0/1261028069 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.979+0000 7fae839a4740 1 -- 172.21.15.70:0/1261028069 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c1ba670 con 0x55b05a2708d0 2024-04-01T01:41:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.979+0000 7fae839a4740 1 -- 172.21.15.70:0/1261028069 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c1b9fa0 con 0x55b05a2712c0 2024-04-01T01:41:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.979+0000 7fae839a4740 1 -- 172.21.15.70:0/1261028069 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c1b71e0 con 0x55b05a276430 2024-04-01T01:41:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.979+0000 7fae7ad9d640 1 --2- 172.21.15.70:0/1261028069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a276430 0x55b05a2ea8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.979+0000 7fae839a4740 10 client.0 _ll_register_callbacks cb 0x7fae64013e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.980+0000 7fae71d9b640 1 -- 172.21.15.70:0/1261028069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a276430 msgr2=0x55b05a2ea8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.980+0000 7fae71d9b640 1 --2- 172.21.15.70:0/1261028069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a276430 0x55b05a2ea8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.980+0000 7fae71d9b640 1 -- 172.21.15.70:0/1261028069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2708d0 msgr2=0x55b05a27eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.980+0000 7fae71d9b640 1 --2- 172.21.15.70:0/1261028069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2708d0 0x55b05a27eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.980+0000 7fae71d9b640 1 -- 172.21.15.70:0/1261028069 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b05a27f680 con 0x55b05a2712c0 2024-04-01T01:41:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.980+0000 7fae71d9b640 1 --2- 172.21.15.70:0/1261028069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a2712c0 0x55b05a27efe0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fae64012360 tx=0x7fae640134f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8c5282fcc90f898 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.980+0000 7fae61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.980+0000 7fae61ffb640 1 -- 172.21.15.70:0/1261028069 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae64036070 con 0x55b05a2712c0 2024-04-01T01:41:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.980+0000 7fae61ffb640 1 -- 172.21.15.70:0/1261028069 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fae6401e7b0 con 0x55b05a2712c0 2024-04-01T01:41:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.980+0000 7fae839a4740 1 -- 172.21.15.70:0/1261028069 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae6c087070 con 0x55b05a2712c0 2024-04-01T01:41:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.981+0000 7fae61ffb640 1 -- 172.21.15.70:0/1261028069 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fae6403f050 con 0x55b05a2712c0 2024-04-01T01:41:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.981+0000 7fae839a4740 10 client.27809 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:20.982 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.981+0000 7fae839a4740 1 -- 172.21.15.70:0/1261028069 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b05a2ebf90 con 0x55b05a2712c0 2024-04-01T01:41:20.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.981+0000 7fae61ffb640 1 -- 172.21.15.70:0/1261028069 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae6c087070 con 0x55b05a2712c0 2024-04-01T01:41:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.987+0000 7fae61ffb640 1 -- 172.21.15.70:0/1261028069 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fae64012040 con 0x55b05a2712c0 2024-04-01T01:41:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.987+0000 7fae839a4740 10 client.27809 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.987+0000 7fae839a4740 10 client.27809 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.987+0000 7fae839a4740 20 client.27809 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.987+0000 7fae839a4740 1 --2- 172.21.15.70:0/1261028069 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b05a2ec910 0x55b05a2eed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.987+0000 7fae839a4740 4 client.27809 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:41:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.987+0000 7fae839a4740 1 -- 172.21.15.70:0/1261028069 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b05a1e1e30 con 0x55b05a2ec910 2024-04-01T01:41:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.988+0000 7fae7a59c640 1 --2- 172.21.15.70:0/1261028069 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b05a2ec910 0x55b05a2eed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.988+0000 7fae7a59c640 1 --2- 172.21.15.70:0/1261028069 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b05a2ec910 0x55b05a2eed30 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5e911573942461d server_cookie=3cfda3596be8c63b in_seq=0 out_seq=0 2024-04-01T01:41:20.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:20.989+0000 7fae61ffb640 10 client.27809 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.062+0000 7fae61ffb640 1 -- 172.21.15.70:0/1261028069 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b05a1e1e30 con 0x55b05a2ec910 2024-04-01T01:41:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.062+0000 7fae61ffb640 10 client.27809 handle_command_reply: tid=0 2024-04-01T01:41:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.062+0000 7fae839a4740 1 client.27809 shutdown 2024-04-01T01:41:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.062+0000 7fae839a4740 1 -- 172.21.15.70:0/1261028069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a2712c0 msgr2=0x55b05a27efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.062+0000 7fae839a4740 1 --2- 172.21.15.70:0/1261028069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a2712c0 0x55b05a27efe0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fae64012360 tx=0x7fae640134f0 comp rx=0 tx=0).stop 2024-04-01T01:41:21.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.063+0000 7fae839a4740 1 -- 172.21.15.70:0/1261028069 shutdown_connections 2024-04-01T01:41:21.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.063+0000 7fae839a4740 1 --2- 172.21.15.70:0/1261028069 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b05a2ec910 0x55b05a2eed30 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:21.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.063+0000 7fae839a4740 1 --2- 172.21.15.70:0/1261028069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b05a276430 0x55b05a2ea8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:21.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.063+0000 7fae839a4740 1 --2- 172.21.15.70:0/1261028069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b05a2712c0 0x55b05a27efe0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:21.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.063+0000 7fae839a4740 1 --2- 172.21.15.70:0/1261028069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b05a2708d0 0x55b05a27eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.063+0000 7fae839a4740 1 -- 172.21.15.70:0/1261028069 >> 172.21.15.70:0/1261028069 conn(0x55b05a26be40 msgr2=0x55b05a26c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.063+0000 7fae61ffb640 0 client.27809 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.063+0000 7fae839a4740 1 -- 172.21.15.70:0/1261028069 shutdown_connections 2024-04-01T01:41:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.063+0000 7fae839a4740 1 -- 172.21.15.70:0/1261028069 wait complete. 2024-04-01T01:41:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.063+0000 7fae839a4740 20 client.27809 trim_cache size 0 max 16384 2024-04-01T01:41:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:41:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:41:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:41:21.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:41:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.064+0000 7fae80ceb640 1 -- 172.21.15.70:0/1675254166 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fae680a25c0 msgr2=0x7fae680a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.064+0000 7fae80ceb640 1 --2- 172.21.15.70:0/1675254166 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fae680a25c0 0x7fae680a4a80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fae7c1a7d80 tx=0x7fae64006bd0 comp rx=0 tx=0).stop 2024-04-01T01:41:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.064+0000 7fae80ceb640 1 -- 172.21.15.70:0/1675254166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae7c108750 msgr2=0x7fae7c1a6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.064+0000 7fae80ceb640 1 --2- 172.21.15.70:0/1675254166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae7c108750 0x7fae7c1a6870 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009230 tx=0x7fae6c0042f0 comp rx=0 tx=0).stop 2024-04-01T01:41:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.064+0000 7fae80ceb640 1 -- 172.21.15.70:0/1675254166 shutdown_connections 2024-04-01T01:41:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.064+0000 7fae80ceb640 1 --2- 172.21.15.70:0/1675254166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7c10da70 0x7fae7c1ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:21.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.064+0000 7fae80ceb640 1 --2- 172.21.15.70:0/1675254166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae7c10b0c0 0x7fae7c1a6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:21.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.064+0000 7fae80ceb640 1 --2- 172.21.15.70:0/1675254166 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fae680a25c0 0x7fae680a4a80 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:21.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.064+0000 7fae80ceb640 1 --2- 172.21.15.70:0/1675254166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae7c108750 0x7fae7c1a6870 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:21.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.064+0000 7fae80ceb640 1 -- 172.21.15.70:0/1675254166 >> 172.21.15.70:0/1675254166 conn(0x7fae7c06e790 msgr2=0x7fae7c10b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:21.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.065+0000 7fae80ceb640 1 -- 172.21.15.70:0/1675254166 shutdown_connections 2024-04-01T01:41:21.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:21.065+0000 7fae80ceb640 1 -- 172.21.15.70:0/1675254166 wait complete. 2024-04-01T01:41:21.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:20 smithi189 ceph-mon[35753]: pgmap v595: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.8 MiB/s wr, 846 op/s 2024-04-01T01:41:21.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:20 smithi070 ceph-mon[32179]: pgmap v595: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.8 MiB/s wr, 846 op/s 2024-04-01T01:41:21.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:20 smithi081 ceph-mon[35142]: pgmap v595: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.8 MiB/s wr, 846 op/s 2024-04-01T01:41:21.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 395.0 s, 947.2 tps, lat 106.459 ms stddev 77.141 2024-04-01T01:41:23.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:22 smithi189 ceph-mon[35753]: pgmap v596: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 9.5 MiB/s wr, 983 op/s 2024-04-01T01:41:23.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:22 smithi070 ceph-mon[32179]: pgmap v596: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 9.5 MiB/s wr, 983 op/s 2024-04-01T01:41:23.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:22 smithi081 ceph-mon[35142]: pgmap v596: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 9.5 MiB/s wr, 983 op/s 2024-04-01T01:41:25.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:24 smithi189 ceph-mon[35753]: pgmap v597: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 8.8 MiB/s wr, 894 op/s 2024-04-01T01:41:25.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:24 smithi070 ceph-mon[32179]: pgmap v597: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 8.8 MiB/s wr, 894 op/s 2024-04-01T01:41:25.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:24 smithi081 ceph-mon[35142]: pgmap v597: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 8.8 MiB/s wr, 894 op/s 2024-04-01T01:41:26.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 400.0 s, 955.4 tps, lat 104.755 ms stddev 54.080 2024-04-01T01:41:27.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:26 smithi189 ceph-mon[35753]: pgmap v598: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 9.2 MiB/s wr, 989 op/s 2024-04-01T01:41:27.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:26 smithi070 ceph-mon[32179]: pgmap v598: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 9.2 MiB/s wr, 989 op/s 2024-04-01T01:41:27.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:26 smithi081 ceph-mon[35142]: pgmap v598: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 9.2 MiB/s wr, 989 op/s 2024-04-01T01:41:29.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:28 smithi189 ceph-mon[35753]: pgmap v599: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 8.9 MiB/s wr, 961 op/s 2024-04-01T01:41:29.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:28 smithi070 ceph-mon[32179]: pgmap v599: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 8.9 MiB/s wr, 961 op/s 2024-04-01T01:41:29.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:28 smithi081 ceph-mon[35142]: pgmap v599: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 8.9 MiB/s wr, 961 op/s 2024-04-01T01:41:31.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:30 smithi189 ceph-mon[35753]: pgmap v600: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.2 MiB/s wr, 884 op/s 2024-04-01T01:41:31.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:30 smithi070 ceph-mon[32179]: pgmap v600: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.2 MiB/s wr, 884 op/s 2024-04-01T01:41:31.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:30 smithi081 ceph-mon[35142]: pgmap v600: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 8.2 MiB/s wr, 884 op/s 2024-04-01T01:41:31.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 405.0 s, 830.6 tps, lat 118.597 ms stddev 102.221 2024-04-01T01:41:33.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:32 smithi189 ceph-mon[35753]: pgmap v601: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 9.1 MiB/s wr, 973 op/s 2024-04-01T01:41:33.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:32 smithi070 ceph-mon[32179]: pgmap v601: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 9.1 MiB/s wr, 973 op/s 2024-04-01T01:41:33.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:32 smithi081 ceph-mon[35142]: pgmap v601: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 9.1 MiB/s wr, 973 op/s 2024-04-01T01:41:34.081 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:41:34.081 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:41:34.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.222+0000 7f9e0cb3f640 1 -- 172.21.15.70:0/3739316929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e00096830 msgr2=0x7f9e0009d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.222+0000 7f9e0cb3f640 1 --2- 172.21.15.70:0/3739316929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e00096830 0x7f9e0009d0c0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9e08066e90 tx=0x7f9e08097940 comp rx=0 tx=0).stop 2024-04-01T01:41:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.222+0000 7f9e0cb3f640 1 -- 172.21.15.70:0/3739316929 shutdown_connections 2024-04-01T01:41:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.222+0000 7f9e0cb3f640 1 --2- 172.21.15.70:0/3739316929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e00096830 0x7f9e0009d0c0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.222+0000 7f9e0cb3f640 1 --2- 172.21.15.70:0/3739316929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e00095e70 0x7f9e000962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.222+0000 7f9e0cb3f640 1 --2- 172.21.15.70:0/3739316929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e00094c80 0x7f9e000950a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.222+0000 7f9e0cb3f640 1 -- 172.21.15.70:0/3739316929 >> 172.21.15.70:0/3739316929 conn(0x7f9e000901d0 msgr2=0x7f9e00092630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:34.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.222+0000 7f9e0cb3f640 1 -- 172.21.15.70:0/3739316929 shutdown_connections 2024-04-01T01:41:34.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.222+0000 7f9e0cb3f640 1 -- 172.21.15.70:0/3739316929 wait complete. 2024-04-01T01:41:34.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.223+0000 7f9e0cb3f640 1 Processor -- start 2024-04-01T01:41:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.224+0000 7f9e0cb3f640 1 -- start start 2024-04-01T01:41:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.224+0000 7f9e0cb3f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e00094c80 0x7f9e00132540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.224+0000 7f9e0cb3f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e00095e70 0x7f9e00132a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.224+0000 7f9e0cb3f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e00096830 0x7f9e00139af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.224+0000 7f9e0cb3f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0009f650 con 0x7f9e00096830 2024-04-01T01:41:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.224+0000 7f9e0cb3f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0009f4d0 con 0x7f9e00094c80 2024-04-01T01:41:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.224+0000 7f9e0cb3f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0009f7d0 con 0x7f9e00095e70 2024-04-01T01:41:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.225+0000 7f9e077fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e00094c80 0x7f9e00132540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.225+0000 7f9e06ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e00095e70 0x7f9e00132a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.225+0000 7f9e077fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e00094c80 0x7f9e00132540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57556/0 (socket says 172.21.15.70:57556) 2024-04-01T01:41:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.225+0000 7f9e077fe640 1 -- 172.21.15.70:0/3515752711 learned_addr learned my addr 172.21.15.70:0/3515752711 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.225+0000 7f9e07fff640 1 --2- 172.21.15.70:0/3515752711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e00096830 0x7f9e00139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.225+0000 7f9e06ffd640 1 -- 172.21.15.70:0/3515752711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e00094c80 msgr2=0x7f9e00132540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.225+0000 7f9e06ffd640 1 --2- 172.21.15.70:0/3515752711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e00094c80 0x7f9e00132540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.225+0000 7f9e06ffd640 1 -- 172.21.15.70:0/3515752711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e00096830 msgr2=0x7f9e00139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.225+0000 7f9e06ffd640 1 --2- 172.21.15.70:0/3515752711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e00096830 0x7f9e00139af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.225+0000 7f9e06ffd640 1 -- 172.21.15.70:0/3515752711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e0013a160 con 0x7f9e00095e70 2024-04-01T01:41:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.225+0000 7f9e07fff640 1 --2- 172.21.15.70:0/3515752711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e00096830 0x7f9e00139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:41:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.226+0000 7f9e06ffd640 1 --2- 172.21.15.70:0/3515752711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e00095e70 0x7f9e00132a80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009fd0 tx=0x7f9df400b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.226+0000 7f9e04ff9640 1 -- 172.21.15.70:0/3515752711 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df4015020 con 0x7f9e00095e70 2024-04-01T01:41:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.226+0000 7f9e04ff9640 1 -- 172.21.15.70:0/3515752711 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9df4011ce0 con 0x7f9e00095e70 2024-04-01T01:41:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.226+0000 7f9e04ff9640 1 -- 172.21.15.70:0/3515752711 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df4015020 con 0x7f9e00095e70 2024-04-01T01:41:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.227+0000 7f9e0cb3f640 1 -- 172.21.15.70:0/3515752711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e0013a450 con 0x7f9e00095e70 2024-04-01T01:41:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.227+0000 7f9e0cb3f640 1 -- 172.21.15.70:0/3515752711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e0013a9d0 con 0x7f9e00095e70 2024-04-01T01:41:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.231+0000 7f9e10984740 1 Processor -- start 2024-04-01T01:41:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.232+0000 7f9e04ff9640 1 -- 172.21.15.70:0/3515752711 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f9df4018020 con 0x7f9e00095e70 2024-04-01T01:41:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.233+0000 7f9e10984740 1 -- start start 2024-04-01T01:41:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.234+0000 7f9e10984740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf8754af0 0x555bf8754f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.234+0000 7f9e10984740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf87554e0 0x555bf8759fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.234+0000 7f9e10984740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf875a650 0x555bf875cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.234+0000 7f9e10984740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555bf86a78f0 con 0x555bf87554e0 2024-04-01T01:41:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.234+0000 7f9e10984740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555bf86508d0 con 0x555bf8754af0 2024-04-01T01:41:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.234+0000 7f9e10984740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555bf86a3990 con 0x555bf875a650 2024-04-01T01:41:34.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.234+0000 7f9e04ff9640 1 --2- 172.21.15.70:0/3515752711 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9df80a28e0 0x7f9df80a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.234+0000 7f9e04ff9640 1 -- 172.21.15.70:0/3515752711 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f9df4011630 con 0x7f9e00095e70 2024-04-01T01:41:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.235+0000 7f9e07fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf8754af0 0x555bf8754f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.235+0000 7f9e07fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf8754af0 0x555bf8754f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57572/0 (socket says 172.21.15.70:57572) 2024-04-01T01:41:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.235+0000 7f9e07fff640 1 -- 172.21.15.70:0/3852361845 learned_addr learned my addr 172.21.15.70:0/3852361845 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.235+0000 7f9e077fe640 1 --2- 172.21.15.70:0/3852361845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf87554e0 0x555bf8759fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.235+0000 7f9e06ffd640 1 --2- 172.21.15.70:0/3852361845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf875a650 0x555bf875cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.235+0000 7f9e07fff640 1 -- 172.21.15.70:0/3852361845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf875a650 msgr2=0x555bf875cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.235+0000 7f9e07fff640 1 --2- 172.21.15.70:0/3852361845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf875a650 0x555bf875cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.235+0000 7f9e07fff640 1 -- 172.21.15.70:0/3852361845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf87554e0 msgr2=0x555bf8759fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.235+0000 7f9e07fff640 1 --2- 172.21.15.70:0/3852361845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf87554e0 0x555bf8759fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.235+0000 7f9e07fff640 1 -- 172.21.15.70:0/3852361845 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bf83bc210 con 0x555bf8754af0 2024-04-01T01:41:34.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.236+0000 7f9e07fff640 1 --2- 172.21.15.70:0/3852361845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf8754af0 0x555bf8754f10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9e0804fbf0 tx=0x7f9e0809cb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e716e5ab845585 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:34.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.236+0000 7f9df1ffb640 1 -- 172.21.15.70:0/3852361845 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e080a0040 con 0x555bf8754af0 2024-04-01T01:41:34.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.236+0000 7f9df1ffb640 1 -- 172.21.15.70:0/3852361845 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e0809bd70 con 0x555bf8754af0 2024-04-01T01:41:34.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.237+0000 7f9df1ffb640 1 -- 172.21.15.70:0/3852361845 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e080a6690 con 0x555bf8754af0 2024-04-01T01:41:34.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.237+0000 7f9e10984740 1 -- 172.21.15.70:0/3852361845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf8754af0 msgr2=0x555bf8754f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.237+0000 7f9e10984740 1 --2- 172.21.15.70:0/3852361845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf8754af0 0x555bf8754f10 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9e0804fbf0 tx=0x7f9e0809cb60 comp rx=0 tx=0).stop 2024-04-01T01:41:34.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.237+0000 7f9e10984740 1 -- 172.21.15.70:0/3852361845 shutdown_connections 2024-04-01T01:41:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.237+0000 7f9e10984740 1 --2- 172.21.15.70:0/3852361845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf875a650 0x555bf875cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.238+0000 7f9e10984740 1 --2- 172.21.15.70:0/3852361845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf87554e0 0x555bf8759fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.238+0000 7f9e10984740 1 --2- 172.21.15.70:0/3852361845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf8754af0 0x555bf8754f10 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.238+0000 7f9e10984740 1 -- 172.21.15.70:0/3852361845 >> 172.21.15.70:0/3852361845 conn(0x555bf873f050 msgr2=0x555bf873f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.238+0000 7f9e07fff640 1 --2- 172.21.15.70:0/3515752711 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9df80a28e0 0x7f9df80a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.239+0000 7f9e07fff640 1 --2- 172.21.15.70:0/3515752711 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9df80a28e0 0x7f9df80a4da0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9e08050a00 tx=0x7f9e080a9040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:34.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.241+0000 7f9e10984740 1 -- 172.21.15.70:0/3852361845 shutdown_connections 2024-04-01T01:41:34.241 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.242+0000 7f9e10984740 1 -- 172.21.15.70:0/3852361845 wait complete. 2024-04-01T01:41:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.242+0000 7f9e10984740 1 -- start start 2024-04-01T01:41:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.242+0000 7f9e10984740 1 Processor -- start 2024-04-01T01:41:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.243+0000 7f9e10984740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf8754af0 0x555bf8745260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.243+0000 7f9e10984740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf87554e0 0x555bf87457a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.243+0000 7f9e10984740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf875a650 0x555bf874c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.243+0000 7f9e10984740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555bf87439e0 con 0x555bf8754af0 2024-04-01T01:41:34.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.243+0000 7f9e10984740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555bf8645f80 con 0x555bf87554e0 2024-04-01T01:41:34.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.243+0000 7f9e10984740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555bf8743b60 con 0x555bf875a650 2024-04-01T01:41:34.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.243+0000 7f9e10984740 10 client.0 _ll_register_callbacks cb 0x555bf869e290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:34.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.243+0000 7f9e077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf8754af0 0x555bf8745260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.243+0000 7f9e077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf8754af0 0x555bf8745260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33868/0 (socket says 172.21.15.70:33868) 2024-04-01T01:41:34.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.243+0000 7f9e077fe640 1 -- 172.21.15.70:0/504845478 learned_addr learned my addr 172.21.15.70:0/504845478 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:34.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.244+0000 7f9e07fff640 1 --2- 172.21.15.70:0/504845478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf875a650 0x555bf874c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.244+0000 7f9e06ffd640 1 --2- 172.21.15.70:0/504845478 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf87554e0 0x555bf87457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.245+0000 7f9e07fff640 1 -- 172.21.15.70:0/504845478 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf87554e0 msgr2=0x555bf87457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.245+0000 7f9e07fff640 1 --2- 172.21.15.70:0/504845478 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf87554e0 0x555bf87457a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.245+0000 7f9e07fff640 1 -- 172.21.15.70:0/504845478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf8754af0 msgr2=0x555bf8745260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.245+0000 7f9e07fff640 1 --2- 172.21.15.70:0/504845478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf8754af0 0x555bf8745260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.245+0000 7f9e07fff640 1 -- 172.21.15.70:0/504845478 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bf8488620 con 0x555bf875a650 2024-04-01T01:41:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.249+0000 7f9e07fff640 1 --2- 172.21.15.70:0/504845478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf875a650 0x555bf874c7e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9e08067040 tx=0x7f9e0804faa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c028d76172083aa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.250+0000 7f9de6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:41:34.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.250+0000 7f9de6ffd640 1 -- 172.21.15.70:0/504845478 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e080b8070 con 0x555bf875a650 2024-04-01T01:41:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.251+0000 7f9e10984740 1 -- 172.21.15.70:0/504845478 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e0809bd70 con 0x555bf875a650 2024-04-01T01:41:34.250 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.251+0000 7f9de6ffd640 1 -- 172.21.15.70:0/504845478 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f9e080b2420 con 0x555bf875a650 2024-04-01T01:41:34.254 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.255+0000 7f9de6ffd640 1 -- 172.21.15.70:0/504845478 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9e080a0050 con 0x555bf875a650 2024-04-01T01:41:34.254 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.255+0000 7f9e10984740 10 client.27652 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:34.254 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.255+0000 7f9e10984740 1 -- 172.21.15.70:0/504845478 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555bf87439e0 con 0x555bf875a650 2024-04-01T01:41:34.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.256+0000 7f9de6ffd640 1 -- 172.21.15.70:0/504845478 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e0809bd70 con 0x555bf875a650 2024-04-01T01:41:34.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.256+0000 7f9de6ffd640 1 -- 172.21.15.70:0/504845478 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f9e080bf430 con 0x555bf875a650 2024-04-01T01:41:34.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.257+0000 7f9e10984740 10 client.27652 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:34.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.257+0000 7f9e10984740 10 client.27652 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:34.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.257+0000 7f9e10984740 20 client.27652 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:34.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.257+0000 7f9e10984740 1 --2- 172.21.15.70:0/504845478 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555bf8750ac0 0x555bf8752ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.257+0000 7f9e10984740 4 client.27652 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:34.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.257+0000 7f9e10984740 1 -- 172.21.15.70:0/504845478 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bf86bb120 con 0x555bf8750ac0 2024-04-01T01:41:34.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.257+0000 7f9e077fe640 1 --2- 172.21.15.70:0/504845478 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555bf8750ac0 0x555bf8752ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.258+0000 7f9e077fe640 1 --2- 172.21.15.70:0/504845478 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555bf8750ac0 0x555bf8752ef0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fbf09194ee20645 server_cookie=80878532005765d6 in_seq=0 out_seq=0 2024-04-01T01:41:34.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.258+0000 7f9de6ffd640 10 client.27652 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:34.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.259+0000 7f9de6ffd640 1 -- 172.21.15.70:0/504845478 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555bf86bb120 con 0x555bf8750ac0 2024-04-01T01:41:34.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.259+0000 7f9de6ffd640 10 client.27652 handle_command_reply: tid=0 2024-04-01T01:41:34.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.259+0000 7f9e10984740 1 client.27652 shutdown 2024-04-01T01:41:34.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.260+0000 7f9e10984740 1 -- 172.21.15.70:0/504845478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf875a650 msgr2=0x555bf874c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.260+0000 7f9e10984740 1 --2- 172.21.15.70:0/504845478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf875a650 0x555bf874c7e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9e08067040 tx=0x7f9e0804faa0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.260+0000 7f9e10984740 1 -- 172.21.15.70:0/504845478 shutdown_connections 2024-04-01T01:41:34.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.260+0000 7f9e10984740 1 --2- 172.21.15.70:0/504845478 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555bf8750ac0 0x555bf8752ef0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.260+0000 7f9e10984740 1 --2- 172.21.15.70:0/504845478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf875a650 0x555bf874c7e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.260+0000 7f9e10984740 1 --2- 172.21.15.70:0/504845478 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf87554e0 0x555bf87457a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.260+0000 7f9e10984740 1 --2- 172.21.15.70:0/504845478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf8754af0 0x555bf8745260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.260+0000 7f9e10984740 1 -- 172.21.15.70:0/504845478 >> 172.21.15.70:0/504845478 conn(0x555bf873f050 msgr2=0x555bf873e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:34.260 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.260+0000 7f9de6ffd640 0 client.27652 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:34.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.261+0000 7f9e10984740 1 -- 172.21.15.70:0/504845478 shutdown_connections 2024-04-01T01:41:34.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.261+0000 7f9e10984740 1 -- 172.21.15.70:0/504845478 wait complete. 2024-04-01T01:41:34.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.261+0000 7f9e10984740 20 client.27652 trim_cache size 0 max 16384 2024-04-01T01:41:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.292+0000 7f9e10984740 1 Processor -- start 2024-04-01T01:41:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.292+0000 7f9e10984740 1 -- start start 2024-04-01T01:41:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.293+0000 7f9e10984740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf8754af0 0x555bf8753730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.293+0000 7f9e10984740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf87554e0 0x555bf874c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.293+0000 7f9e10984740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf875a650 0x555bf8753c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.293+0000 7f9e10984740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0013c7f0 con 0x555bf875a650 2024-04-01T01:41:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.293+0000 7f9e10984740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0013e330 con 0x555bf8754af0 2024-04-01T01:41:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.293+0000 7f9e10984740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0013cec0 con 0x555bf87554e0 2024-04-01T01:41:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.293+0000 7f9e077fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf8754af0 0x555bf8753730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.293+0000 7f9e07fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf875a650 0x555bf8753c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.293+0000 7f9e077fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf8754af0 0x555bf8753730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57602/0 (socket says 172.21.15.70:57602) 2024-04-01T01:41:34.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.293+0000 7f9e077fe640 1 -- 172.21.15.70:0/3235919292 learned_addr learned my addr 172.21.15.70:0/3235919292 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:34.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.293+0000 7f9e06ffd640 1 --2- 172.21.15.70:0/3235919292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf87554e0 0x555bf874c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.294+0000 7f9e077fe640 1 -- 172.21.15.70:0/3235919292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf87554e0 msgr2=0x555bf874c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:34.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.294+0000 7f9e077fe640 1 --2- 172.21.15.70:0/3235919292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf87554e0 0x555bf874c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.294+0000 7f9e077fe640 1 -- 172.21.15.70:0/3235919292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf875a650 msgr2=0x555bf8753c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:34.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.294+0000 7f9e077fe640 1 --2- 172.21.15.70:0/3235919292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf875a650 0x555bf8753c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.294+0000 7f9e077fe640 1 -- 172.21.15.70:0/3235919292 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bf87534d0 con 0x555bf8754af0 2024-04-01T01:41:34.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.294+0000 7f9e077fe640 1 --2- 172.21.15.70:0/3235919292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf8754af0 0x555bf8753730 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00bfd0 tx=0x7f9dfc00b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f998ce4557d6245 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:34.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.295+0000 7f9df17fa640 1 -- 172.21.15.70:0/3235919292 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc004220 con 0x555bf8754af0 2024-04-01T01:41:34.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.295+0000 7f9df17fa640 1 -- 172.21.15.70:0/3235919292 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9dfc024030 con 0x555bf8754af0 2024-04-01T01:41:34.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.295+0000 7f9df17fa640 1 -- 172.21.15.70:0/3235919292 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc0047a0 con 0x555bf8754af0 2024-04-01T01:41:34.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.295+0000 7f5bc6380640 1 -- 172.21.15.70:0/425682833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bb8099910 msgr2=0x7f5bb8099d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.295+0000 7f5bc6380640 1 --2- 172.21.15.70:0/425682833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bb8099910 0x7f5bb8099d90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0067420 tx=0x7f5bc0097850 comp rx=0 tx=0).stop 2024-04-01T01:41:34.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.295+0000 7f5bc6380640 1 -- 172.21.15.70:0/425682833 shutdown_connections 2024-04-01T01:41:34.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.295+0000 7f5bc6380640 1 --2- 172.21.15.70:0/425682833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bb8099910 0x7f5bb8099d90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.295+0000 7f5bc6380640 1 --2- 172.21.15.70:0/425682833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb8096f60 0x7f5bb80993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.295+0000 7f5bc6380640 1 --2- 172.21.15.70:0/425682833 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5bb80945f0 0x7f5bb8096a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.295+0000 7f5bc6380640 1 -- 172.21.15.70:0/425682833 >> 172.21.15.70:0/425682833 conn(0x7f5bb80901f0 msgr2=0x7f5bb8092630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:34.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.296+0000 7f9e10984740 1 -- 172.21.15.70:0/3235919292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf8754af0 msgr2=0x555bf8753730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.296+0000 7f9e10984740 1 --2- 172.21.15.70:0/3235919292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf8754af0 0x555bf8753730 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00bfd0 tx=0x7f9dfc00b410 comp rx=0 tx=0).stop 2024-04-01T01:41:34.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.296+0000 7f5bc6380640 1 -- 172.21.15.70:0/425682833 shutdown_connections 2024-04-01T01:41:34.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.296+0000 7f9e10984740 1 -- 172.21.15.70:0/3235919292 shutdown_connections 2024-04-01T01:41:34.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.296+0000 7f9e10984740 1 --2- 172.21.15.70:0/3235919292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf875a650 0x555bf8753c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.296+0000 7f9e10984740 1 --2- 172.21.15.70:0/3235919292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf87554e0 0x555bf874c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.296+0000 7f9e10984740 1 --2- 172.21.15.70:0/3235919292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf8754af0 0x555bf8753730 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.296+0000 7f9e10984740 1 -- 172.21.15.70:0/3235919292 >> 172.21.15.70:0/3235919292 conn(0x555bf873f050 msgr2=0x555bf873e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:34.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.299+0000 7f5bc6380640 1 -- 172.21.15.70:0/425682833 wait complete. 2024-04-01T01:41:34.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.300+0000 7f5bc6380640 1 Processor -- start 2024-04-01T01:41:34.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.300+0000 7f5bc6380640 1 -- start start 2024-04-01T01:41:34.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.300+0000 7f5bc6380640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bb80945f0 0x7f5bb8143380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.300+0000 7f5bc6380640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb8096f60 0x7f5bb81438c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.300+0000 7f5bc6380640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5bb8099910 0x7f5bb814a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.300+0000 7f5bc6380640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb80a7dd0 con 0x7f5bb8099910 2024-04-01T01:41:34.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.300+0000 7f5bc6380640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb80a7c50 con 0x7f5bb8096f60 2024-04-01T01:41:34.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.300+0000 7f5bc6380640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb80a7f50 con 0x7f5bb80945f0 2024-04-01T01:41:34.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.301+0000 7f9e10984740 1 -- 172.21.15.70:0/3235919292 shutdown_connections 2024-04-01T01:41:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.301+0000 7f5bc5b7f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5bb8099910 0x7f5bb814a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.301+0000 7f5bc5b7f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5bb8099910 0x7f5bb814a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33906/0 (socket says 172.21.15.70:33906) 2024-04-01T01:41:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.301+0000 7f5bc5b7f640 1 -- 172.21.15.70:0/3491392456 learned_addr learned my addr 172.21.15.70:0/3491392456 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.301+0000 7f5bc537e640 1 --2- 172.21.15.70:0/3491392456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bb80945f0 0x7f5bb8143380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.301+0000 7f5bc4b7d640 1 --2- 172.21.15.70:0/3491392456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb8096f60 0x7f5bb81438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.301+0000 7f5bc5b7f640 1 -- 172.21.15.70:0/3491392456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bb80945f0 msgr2=0x7f5bb8143380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.301+0000 7f5bc5b7f640 1 --2- 172.21.15.70:0/3491392456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bb80945f0 0x7f5bb8143380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.301+0000 7f5bc5b7f640 1 -- 172.21.15.70:0/3491392456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb8096f60 msgr2=0x7f5bb81438c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.301+0000 7f5bc5b7f640 1 --2- 172.21.15.70:0/3491392456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb8096f60 0x7f5bb81438c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.301+0000 7f5bc5b7f640 1 -- 172.21.15.70:0/3491392456 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb809ad40 con 0x7f5bb8099910 2024-04-01T01:41:34.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.302+0000 7f5bc5b7f640 1 --2- 172.21.15.70:0/3491392456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5bb8099910 0x7f5bb814a930 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0050db0 tx=0x7f5bc00699e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:34.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.302+0000 7f9e10984740 1 -- 172.21.15.70:0/3235919292 wait complete. 2024-04-01T01:41:34.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.303+0000 7f9e10984740 1 -- start start 2024-04-01T01:41:34.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.303+0000 7f9e10984740 1 Processor -- start 2024-04-01T01:41:34.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.306+0000 7f9e10984740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf8754af0 0x555bf874c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.306+0000 7f9e10984740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf87554e0 0x555bf874c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.306+0000 7f9e10984740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf875a650 0x555bf8748b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.306+0000 7f9e10984740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0013dc60 con 0x555bf8754af0 2024-04-01T01:41:34.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.306+0000 7f9e10984740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0013d590 con 0x555bf875a650 2024-04-01T01:41:34.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.306+0000 7f9e10984740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e000acfb0 con 0x555bf87554e0 2024-04-01T01:41:34.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.306+0000 7f9e10984740 10 client.0 _ll_register_callbacks cb 0x7f9dfc00b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:34.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.306+0000 7f5bb67fc640 1 -- 172.21.15.70:0/3491392456 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc006c040 con 0x7f5bb8099910 2024-04-01T01:41:34.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.306+0000 7f5bb67fc640 1 -- 172.21.15.70:0/3491392456 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bc00692d0 con 0x7f5bb8099910 2024-04-01T01:41:34.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.306+0000 7f5bb67fc640 1 -- 172.21.15.70:0/3491392456 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc009e440 con 0x7f5bb8099910 2024-04-01T01:41:34.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.306+0000 7f5bc6380640 1 -- 172.21.15.70:0/3491392456 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb809afd0 con 0x7f5bb8099910 2024-04-01T01:41:34.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.307+0000 7f5bc6380640 1 -- 172.21.15.70:0/3491392456 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb809b4a0 con 0x7f5bb8099910 2024-04-01T01:41:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.307+0000 7f9e077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf8754af0 0x555bf874c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.307+0000 7f9e077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf8754af0 0x555bf874c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33922/0 (socket says 172.21.15.70:33922) 2024-04-01T01:41:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.307+0000 7f9e077fe640 1 -- 172.21.15.70:0/1921362159 learned_addr learned my addr 172.21.15.70:0/1921362159 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.307+0000 7f9e077fe640 1 -- 172.21.15.70:0/1921362159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf87554e0 msgr2=0x555bf874c5b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:41:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.307+0000 7f9e077fe640 1 --2- 172.21.15.70:0/1921362159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf87554e0 0x555bf874c5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.307+0000 7f9e077fe640 1 -- 172.21.15.70:0/1921362159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf875a650 msgr2=0x555bf8748b10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:41:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.307+0000 7f9e077fe640 1 --2- 172.21.15.70:0/1921362159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf875a650 0x555bf8748b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.307+0000 7f9e077fe640 1 -- 172.21.15.70:0/1921362159 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bf875a3a0 con 0x555bf8754af0 2024-04-01T01:41:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.308+0000 7f9e077fe640 1 --2- 172.21.15.70:0/1921362159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf8754af0 0x555bf874c070 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc011d60 tx=0x7f9dfc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b3c8495375d488 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.308+0000 7f9de6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:41:34.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.310+0000 7f9de6ffd640 1 -- 172.21.15.70:0/1921362159 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc020420 con 0x555bf8754af0 2024-04-01T01:41:34.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.310+0000 7f9de6ffd640 1 -- 172.21.15.70:0/1921362159 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f9dfc0205c0 con 0x555bf8754af0 2024-04-01T01:41:34.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.311+0000 7f9e10984740 1 -- 172.21.15.70:0/1921362159 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dfc024030 con 0x555bf8754af0 2024-04-01T01:41:34.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.314+0000 7f9de6ffd640 1 -- 172.21.15.70:0/1921362159 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9dfc038050 con 0x555bf8754af0 2024-04-01T01:41:34.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.314+0000 7f9e10984740 10 client.17859 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:34.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.314+0000 7f9e10984740 1 -- 172.21.15.70:0/1921362159 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555bf87d0450 con 0x555bf8754af0 2024-04-01T01:41:34.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.315+0000 7f9de6ffd640 1 -- 172.21.15.70:0/1921362159 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9dfc024030 con 0x555bf8754af0 2024-04-01T01:41:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.310+0000 7f5bca1bf740 1 Processor -- start 2024-04-01T01:41:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.313+0000 7f5bca1bf740 1 -- start start 2024-04-01T01:41:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.313+0000 7f5bca1bf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b518b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.313+0000 7f5bca1bf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 0x562193b56980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.313+0000 7f5bca1bf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b56ff0 0x562193b59480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.313+0000 7f5bca1bf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562193ab6180 con 0x562193b56ff0 2024-04-01T01:41:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.313+0000 7f5bca1bf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562193a5eb70 con 0x562193b51490 2024-04-01T01:41:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.313+0000 7f5bca1bf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562193ab2220 con 0x562193b51e80 2024-04-01T01:41:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.313+0000 7f5bc4b7d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b56ff0 0x562193b59480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.313+0000 7f5bb67fc640 1 -- 172.21.15.70:0/3491392456 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f5bc00a5070 con 0x7f5bb8099910 2024-04-01T01:41:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.313+0000 7f5bc4b7d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b56ff0 0x562193b59480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33934/0 (socket says 172.21.15.70:33934) 2024-04-01T01:41:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.313+0000 7f5bc4b7d640 1 -- 172.21.15.70:0/347656085 learned_addr learned my addr 172.21.15.70:0/347656085 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.313+0000 7f5bc4b7d640 1 --2- 172.21.15.70:0/347656085 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.314+0000 7f5bc537e640 1 --2- 172.21.15.70:0/347656085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 0x562193b56980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.314+0000 7f5bc4b7d640 1 -- 172.21.15.70:0/347656085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 msgr2=0x562193b56980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.314+0000 7f5bc4b7d640 1 --2- 172.21.15.70:0/347656085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 0x562193b56980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.314+0000 7f5bc4b7d640 1 -- 172.21.15.70:0/347656085 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 msgr2=0x562193b518b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.314+0000 7f5bc4b7d640 1 --2- 172.21.15.70:0/347656085 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b518b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.314+0000 7f5bc4b7d640 1 -- 172.21.15.70:0/347656085 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562193a727c0 con 0x562193b56ff0 2024-04-01T01:41:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.314+0000 7f5bb67fc640 1 --2- 172.21.15.70:0/3491392456 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5bac0a2670 0x7f5bac0a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.314+0000 7f5bc4b7d640 1 --2- 172.21.15.70:0/347656085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b56ff0 0x562193b59480 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0034760 tx=0x7f5bb00096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b74f417e8f2c3a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.314+0000 7f5bb67fc640 1 -- 172.21.15.70:0/3491392456 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5bc01536e0 con 0x7f5bb8099910 2024-04-01T01:41:34.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.316+0000 7f5bc537e640 1 --2- 172.21.15.70:0/3491392456 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5bac0a2670 0x7f5bac0a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.316+0000 7f9de6ffd640 1 -- 172.21.15.70:0/1921362159 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f9dfc004730 con 0x555bf8754af0 2024-04-01T01:41:34.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.316+0000 7f9e10984740 10 client.17859 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:34.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.316+0000 7f9e10984740 10 client.17859 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:34.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.316+0000 7f9e10984740 20 client.17859 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:34.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.316+0000 7f9e10984740 1 --2- 172.21.15.70:0/1921362159 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555bf87d0d00 0x555bf87d3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.316+0000 7f9e10984740 4 client.17859 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:41:34.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.316+0000 7f9e10984740 1 -- 172.21.15.70:0/1921362159 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555bf86b4e30 con 0x555bf87d0d00 2024-04-01T01:41:34.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.317+0000 7f9e06ffd640 1 --2- 172.21.15.70:0/1921362159 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555bf87d0d00 0x555bf87d3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.319+0000 7f9e06ffd640 1 --2- 172.21.15.70:0/1921362159 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555bf87d0d00 0x555bf87d3120 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb7833b55338342a server_cookie=ff538115093528c1 in_seq=0 out_seq=0 2024-04-01T01:41:34.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.319+0000 7f9de6ffd640 10 client.17859 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:34.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.316+0000 7f5bc537e640 1 --2- 172.21.15.70:0/3491392456 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5bac0a2670 0x7f5bac0a4b30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x562193b52280 tx=0x7f5bbc005ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.318+0000 7f5bab7fe640 1 -- 172.21.15.70:0/347656085 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0034ce0 con 0x562193b56ff0 2024-04-01T01:41:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.318+0000 7f5bab7fe640 1 -- 172.21.15.70:0/347656085 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bb00327e0 con 0x562193b56ff0 2024-04-01T01:41:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.319+0000 7f5bca1bf740 1 -- 172.21.15.70:0/347656085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b56ff0 msgr2=0x562193b59480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.319+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/347656085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b56ff0 0x562193b59480 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0034760 tx=0x7f5bb00096e0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.319+0000 7f5bca1bf740 1 -- 172.21.15.70:0/347656085 shutdown_connections 2024-04-01T01:41:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.319+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/347656085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b56ff0 0x562193b59480 secure :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0034760 tx=0x7f5bb00096e0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.319+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/347656085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 0x562193b56980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.319+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/347656085 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.319+0000 7f5bca1bf740 1 -- 172.21.15.70:0/347656085 >> 172.21.15.70:0/347656085 conn(0x562193b4ca00 msgr2=0x562193b4ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:34.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.324+0000 7f5bca1bf740 1 -- 172.21.15.70:0/347656085 shutdown_connections 2024-04-01T01:41:34.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.325+0000 7f5bca1bf740 1 -- 172.21.15.70:0/347656085 wait complete. 2024-04-01T01:41:34.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.326+0000 7f5bca1bf740 1 -- start start 2024-04-01T01:41:34.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.326+0000 7f5bca1bf740 1 Processor -- start 2024-04-01T01:41:34.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.326+0000 7f5bca1bf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b5bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.326+0000 7f5bca1bf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b51e80 0x562193b5c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.326+0000 7f5bca1bf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b610e0 0x562193b634d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.326+0000 7f5bca1bf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562193b59ff0 con 0x562193b51e80 2024-04-01T01:41:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.326+0000 7f5bca1bf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562193a544a0 con 0x562193b51490 2024-04-01T01:41:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.326+0000 7f5bca1bf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562193b5a170 con 0x562193b610e0 2024-04-01T01:41:34.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.326+0000 7f5bca1bf740 10 client.0 _ll_register_callbacks cb 0x562193a57d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.327+0000 7f5bc537e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b5bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.327+0000 7f5bc537e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b5bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57648/0 (socket says 172.21.15.70:57648) 2024-04-01T01:41:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.327+0000 7f5bc537e640 1 -- 172.21.15.70:0/1831101093 learned_addr learned my addr 172.21.15.70:0/1831101093 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.327+0000 7f5bc5b7f640 1 --2- 172.21.15.70:0/1831101093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b610e0 0x562193b634d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.327+0000 7f5bc4b7d640 1 --2- 172.21.15.70:0/1831101093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b51e80 0x562193b5c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.327+0000 7f5bc5b7f640 1 -- 172.21.15.70:0/1831101093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 msgr2=0x562193b5bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.327+0000 7f5bc5b7f640 1 --2- 172.21.15.70:0/1831101093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b5bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.327+0000 7f5bc5b7f640 1 -- 172.21.15.70:0/1831101093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b51e80 msgr2=0x562193b5c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.327+0000 7f5bc5b7f640 1 --2- 172.21.15.70:0/1831101093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b51e80 0x562193b5c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.327+0000 7f5bc5b7f640 1 -- 172.21.15.70:0/1831101093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621938947d0 con 0x562193b610e0 2024-04-01T01:41:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.328+0000 7f5bc537e640 1 --2- 172.21.15.70:0/1831101093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b5bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:41:34.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.328+0000 7f5bc4b7d640 1 --2- 172.21.15.70:0/1831101093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b51e80 0x562193b5c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:41:34.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.328+0000 7f5bc5b7f640 1 --2- 172.21.15.70:0/1831101093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b610e0 0x562193b634d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0067010 tx=0x7f5bc011f8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d82d77c941742afb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:34.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.328+0000 7f5ba8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:41:34.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.331+0000 7f5ba8ff9640 1 -- 172.21.15.70:0/1831101093 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc00ac040 con 0x562193b610e0 2024-04-01T01:41:34.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.331+0000 7f5ba8ff9640 1 -- 172.21.15.70:0/1831101093 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5bc0069480 con 0x562193b610e0 2024-04-01T01:41:34.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.332+0000 7f5bca1bf740 1 -- 172.21.15.70:0/1831101093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bb00327e0 con 0x562193b610e0 2024-04-01T01:41:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.332+0000 7f5ba8ff9640 1 -- 172.21.15.70:0/1831101093 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5bc013d030 con 0x562193b610e0 2024-04-01T01:41:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.332+0000 7f5bca1bf740 10 client.27664 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.332+0000 7f5bca1bf740 1 -- 172.21.15.70:0/1831101093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5621937ca3f0 con 0x562193b610e0 2024-04-01T01:41:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.333+0000 7f5ba8ff9640 1 -- 172.21.15.70:0/1831101093 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bb00327e0 con 0x562193b610e0 2024-04-01T01:41:34.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.333+0000 7f5ba8ff9640 1 -- 172.21.15.70:0/1831101093 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5bc006f070 con 0x562193b610e0 2024-04-01T01:41:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.333+0000 7f5bca1bf740 10 client.27664 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.333+0000 7f5bca1bf740 10 client.27664 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.333+0000 7f5bca1bf740 20 client.27664 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.333+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/1831101093 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562193b67e40 0x562193b6a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.333+0000 7f5bca1bf740 4 client.27664 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:34.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.333+0000 7f5bca1bf740 1 -- 172.21.15.70:0/1831101093 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562193ac36c0 con 0x562193b67e40 2024-04-01T01:41:34.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.336+0000 7f5bc537e640 1 --2- 172.21.15.70:0/1831101093 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562193b67e40 0x562193b6a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.338+0000 7f5bc537e640 1 --2- 172.21.15.70:0/1831101093 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562193b67e40 0x562193b6a250 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=470dafc10ae5e045 server_cookie=686ed3296618f9dd in_seq=0 out_seq=0 2024-04-01T01:41:34.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.338+0000 7f5ba8ff9640 10 client.27664 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.408+0000 7f9de6ffd640 1 -- 172.21.15.70:0/1921362159 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555bf86b4e30 con 0x555bf87d0d00 2024-04-01T01:41:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.408+0000 7f9de6ffd640 10 client.17859 handle_command_reply: tid=0 2024-04-01T01:41:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.408+0000 7f9e10984740 1 client.17859 shutdown 2024-04-01T01:41:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.408+0000 7f9e10984740 1 -- 172.21.15.70:0/1921362159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf8754af0 msgr2=0x555bf874c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.408+0000 7f9e10984740 1 --2- 172.21.15.70:0/1921362159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf8754af0 0x555bf874c070 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc011d60 tx=0x7f9dfc000bc0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.411+0000 7f5ba8ff9640 1 -- 172.21.15.70:0/1831101093 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562193ac36c0 con 0x562193b67e40 2024-04-01T01:41:34.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.411+0000 7f5ba8ff9640 10 client.27664 handle_command_reply: tid=0 2024-04-01T01:41:34.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.411+0000 7f5bca1bf740 1 client.27664 shutdown 2024-04-01T01:41:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.412+0000 7f9e10984740 1 -- 172.21.15.70:0/1921362159 shutdown_connections 2024-04-01T01:41:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.412+0000 7f9e10984740 1 --2- 172.21.15.70:0/1921362159 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555bf87d0d00 0x555bf87d3120 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.412+0000 7f9e10984740 1 --2- 172.21.15.70:0/1921362159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555bf875a650 0x555bf8748b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.412+0000 7f9e10984740 1 --2- 172.21.15.70:0/1921362159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555bf87554e0 0x555bf874c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.412+0000 7f9e10984740 1 --2- 172.21.15.70:0/1921362159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555bf8754af0 0x555bf874c070 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.412+0000 7f9e10984740 1 -- 172.21.15.70:0/1921362159 >> 172.21.15.70:0/1921362159 conn(0x555bf873f050 msgr2=0x555bf87422b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:34.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.413+0000 7f9de6ffd640 0 client.17859 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:34.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.415+0000 7f9e10984740 1 -- 172.21.15.70:0/1921362159 shutdown_connections 2024-04-01T01:41:34.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.416+0000 7f5bca1bf740 1 -- 172.21.15.70:0/1831101093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b610e0 msgr2=0x562193b634d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.416+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/1831101093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b610e0 0x562193b634d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0067010 tx=0x7f5bc011f8b0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.419+0000 7f9e10984740 1 -- 172.21.15.70:0/1921362159 wait complete. 2024-04-01T01:41:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:41:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:41:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:41:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:41:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.419+0000 7f9e10984740 20 client.17859 trim_cache size 0 max 16384 2024-04-01T01:41:34.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.420+0000 7f5bca1bf740 1 -- 172.21.15.70:0/1831101093 shutdown_connections 2024-04-01T01:41:34.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.420+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/1831101093 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562193b67e40 0x562193b6a250 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.420+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/1831101093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b610e0 0x562193b634d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.420+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/1831101093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b51e80 0x562193b5c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.420+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/1831101093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b5bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.420+0000 7f5bca1bf740 1 -- 172.21.15.70:0/1831101093 >> 172.21.15.70:0/1831101093 conn(0x562193b4ca00 msgr2=0x562193b4bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:34.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.420+0000 7f5ba8ff9640 0 client.27664 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:34.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.424+0000 7f5bca1bf740 1 -- 172.21.15.70:0/1831101093 shutdown_connections 2024-04-01T01:41:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.425+0000 7f9e0cb3f640 1 -- 172.21.15.70:0/3515752711 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9df80a28e0 msgr2=0x7f9df80a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.426+0000 7f9e0cb3f640 1 --2- 172.21.15.70:0/3515752711 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9df80a28e0 0x7f9df80a4da0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9e08050a00 tx=0x7f9e080a9040 comp rx=0 tx=0).stop 2024-04-01T01:41:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.426+0000 7f9e0cb3f640 1 -- 172.21.15.70:0/3515752711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e00095e70 msgr2=0x7f9e00132a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.426+0000 7f9e0cb3f640 1 --2- 172.21.15.70:0/3515752711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e00095e70 0x7f9e00132a80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009fd0 tx=0x7f9df400b870 comp rx=0 tx=0).stop 2024-04-01T01:41:34.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.426+0000 7f5bca1bf740 1 -- 172.21.15.70:0/1831101093 wait complete. 2024-04-01T01:41:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.428+0000 7f9e0cb3f640 1 -- 172.21.15.70:0/3515752711 shutdown_connections 2024-04-01T01:41:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.428+0000 7f9e0cb3f640 1 --2- 172.21.15.70:0/3515752711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e00096830 0x7f9e00139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.428+0000 7f9e0cb3f640 1 --2- 172.21.15.70:0/3515752711 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9df80a28e0 0x7f9df80a4da0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.428+0000 7f9e0cb3f640 1 --2- 172.21.15.70:0/3515752711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e00095e70 0x7f9e00132a80 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.428+0000 7f9e0cb3f640 1 --2- 172.21.15.70:0/3515752711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e00094c80 0x7f9e00132540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.428+0000 7f9e0cb3f640 1 -- 172.21.15.70:0/3515752711 >> 172.21.15.70:0/3515752711 conn(0x7f9e000901d0 msgr2=0x7f9e000a2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:34.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.430+0000 7f5bca1bf740 20 client.27664 trim_cache size 0 max 16384 2024-04-01T01:41:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.433+0000 7f9e0cb3f640 1 -- 172.21.15.70:0/3515752711 shutdown_connections 2024-04-01T01:41:34.435 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:34.433+0000 7f9e0cb3f640 1 -- 172.21.15.70:0/3515752711 wait complete. 2024-04-01T01:41:34.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.469+0000 7f5bca1bf740 1 Processor -- start 2024-04-01T01:41:34.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.469+0000 7f5bca1bf740 1 -- start start 2024-04-01T01:41:34.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bca1bf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b5a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bca1bf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 0x562193b47ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bca1bf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b610e0 0x562193b5bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bca1bf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb812fb50 con 0x562193b610e0 2024-04-01T01:41:34.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bca1bf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb8131690 con 0x562193b51490 2024-04-01T01:41:34.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bca1bf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb8130220 con 0x562193b51e80 2024-04-01T01:41:34.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bc5b7f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b610e0 0x562193b5bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bc5b7f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b610e0 0x562193b5bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33940/0 (socket says 172.21.15.70:33940) 2024-04-01T01:41:34.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bc5b7f640 1 -- 172.21.15.70:0/3944351638 learned_addr learned my addr 172.21.15.70:0/3944351638 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:34.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bc537e640 1 --2- 172.21.15.70:0/3944351638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b5a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bc4b7d640 1 --2- 172.21.15.70:0/3944351638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 0x562193b47ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bc5b7f640 1 -- 172.21.15.70:0/3944351638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 msgr2=0x562193b47ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:34.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bc5b7f640 1 --2- 172.21.15.70:0/3944351638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 0x562193b47ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bc5b7f640 1 -- 172.21.15.70:0/3944351638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 msgr2=0x562193b5a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:34.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bc5b7f640 1 --2- 172.21.15.70:0/3944351638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b5a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bc5b7f640 1 -- 172.21.15.70:0/3944351638 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56219378a0f0 con 0x562193b610e0 2024-04-01T01:41:34.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.470+0000 7f5bc4b7d640 1 --2- 172.21.15.70:0/3944351638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 0x562193b47ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:41:34.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.471+0000 7f5bc5b7f640 1 --2- 172.21.15.70:0/3944351638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b610e0 0x562193b5bb70 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0062c90 tx=0x7f5bc011e280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43658a02f0fa77f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:34.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.471+0000 7f5baaffd640 1 -- 172.21.15.70:0/3944351638 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc00ac070 con 0x562193b610e0 2024-04-01T01:41:34.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.471+0000 7f5baaffd640 1 -- 172.21.15.70:0/3944351638 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bc0138060 con 0x562193b610e0 2024-04-01T01:41:34.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.471+0000 7f5baaffd640 1 -- 172.21.15.70:0/3944351638 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc013e2f0 con 0x562193b610e0 2024-04-01T01:41:34.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.472+0000 7f5bca1bf740 1 -- 172.21.15.70:0/3944351638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b610e0 msgr2=0x562193b5bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.472+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/3944351638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b610e0 0x562193b5bb70 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0062c90 tx=0x7f5bc011e280 comp rx=0 tx=0).stop 2024-04-01T01:41:34.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.472+0000 7f5bca1bf740 1 -- 172.21.15.70:0/3944351638 shutdown_connections 2024-04-01T01:41:34.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.472+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/3944351638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b610e0 0x562193b5bb70 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.472+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/3944351638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 0x562193b47ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.472+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/3944351638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b5a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.472+0000 7f5bca1bf740 1 -- 172.21.15.70:0/3944351638 >> 172.21.15.70:0/3944351638 conn(0x562193b4ca00 msgr2=0x562193b5b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:34.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.472+0000 7f5bca1bf740 1 -- 172.21.15.70:0/3944351638 shutdown_connections 2024-04-01T01:41:34.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.473+0000 7f5bca1bf740 1 -- 172.21.15.70:0/3944351638 wait complete. 2024-04-01T01:41:34.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.473+0000 7f5bca1bf740 1 -- start start 2024-04-01T01:41:34.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.473+0000 7f5bca1bf740 1 Processor -- start 2024-04-01T01:41:34.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.474+0000 7f5bca1bf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b78ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.474+0000 7f5bca1bf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 0x562193b79000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.474+0000 7f5bca1bf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b610e0 0x562193b7a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.474+0000 7f5bca1bf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb8130fc0 con 0x562193b610e0 2024-04-01T01:41:34.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.474+0000 7f5bca1bf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb81308f0 con 0x562193b51490 2024-04-01T01:41:34.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.474+0000 7f5bca1bf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb8005a60 con 0x562193b51e80 2024-04-01T01:41:34.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.474+0000 7f5bca1bf740 10 client.0 _ll_register_callbacks cb 0x7f5bc0062be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:34.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.474+0000 7f5bc4b7d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 0x562193b79000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.474+0000 7f5bc4b7d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 0x562193b79000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57714/0 (socket says 172.21.15.70:57714) 2024-04-01T01:41:34.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.474+0000 7f5bc4b7d640 1 -- 172.21.15.70:0/2082712370 learned_addr learned my addr 172.21.15.70:0/2082712370 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:34.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.474+0000 7f5bc537e640 1 --2- 172.21.15.70:0/2082712370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b78ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.475+0000 7f5bc5b7f640 1 --2- 172.21.15.70:0/2082712370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b610e0 0x562193b7a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.475+0000 7f5bc5b7f640 1 -- 172.21.15.70:0/2082712370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 msgr2=0x562193b79000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:34.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.475+0000 7f5bc5b7f640 1 --2- 172.21.15.70:0/2082712370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 0x562193b79000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.475+0000 7f5bc5b7f640 1 -- 172.21.15.70:0/2082712370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 msgr2=0x562193b78ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:34.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.475+0000 7f5bc5b7f640 1 --2- 172.21.15.70:0/2082712370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b78ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.475+0000 7f5bc5b7f640 1 -- 172.21.15.70:0/2082712370 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562193b7a600 con 0x562193b610e0 2024-04-01T01:41:34.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.475+0000 7f5bc4b7d640 1 --2- 172.21.15.70:0/2082712370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 0x562193b79000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:41:34.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.475+0000 7f5bc5b7f640 1 --2- 172.21.15.70:0/2082712370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b610e0 0x562193b7a010 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0050c90 tx=0x7f5bc011f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b87aa662185a44 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:34.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.476+0000 7f5ba8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:41:34.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.476+0000 7f5ba8ff9640 1 -- 172.21.15.70:0/2082712370 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc00ac050 con 0x562193b610e0 2024-04-01T01:41:34.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.476+0000 7f5ba8ff9640 1 -- 172.21.15.70:0/2082712370 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5bc013e450 con 0x562193b610e0 2024-04-01T01:41:34.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.476+0000 7f5bca1bf740 1 -- 172.21.15.70:0/2082712370 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bc0138060 con 0x562193b610e0 2024-04-01T01:41:34.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.476+0000 7f5ba8ff9640 1 -- 172.21.15.70:0/2082712370 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5bc0069480 con 0x562193b610e0 2024-04-01T01:41:34.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.476+0000 7f5bca1bf740 10 client.17868 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:34.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.476+0000 7f5bca1bf740 1 -- 172.21.15.70:0/2082712370 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562193b7a600 con 0x562193b610e0 2024-04-01T01:41:34.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.477+0000 7f5ba8ff9640 1 -- 172.21.15.70:0/2082712370 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bc0138060 con 0x562193b610e0 2024-04-01T01:41:34.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.477+0000 7f5ba8ff9640 1 -- 172.21.15.70:0/2082712370 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5bc006f070 con 0x562193b610e0 2024-04-01T01:41:34.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.477+0000 7f5bca1bf740 10 client.17868 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:34.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.477+0000 7f5bca1bf740 10 client.17868 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:34.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.477+0000 7f5bca1bf740 20 client.17868 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:34.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.477+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/2082712370 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562193bce110 0x562193bd0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.477+0000 7f5bca1bf740 4 client.17868 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:41:34.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.477+0000 7f5bca1bf740 1 -- 172.21.15.70:0/2082712370 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562193abda10 con 0x562193bce110 2024-04-01T01:41:34.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.477+0000 7f5bc537e640 1 --2- 172.21.15.70:0/2082712370 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562193bce110 0x562193bd0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.478+0000 7f5ba8ff9640 10 client.17868 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:34.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.478+0000 7f5bc537e640 1 --2- 172.21.15.70:0/2082712370 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562193bce110 0x562193bd0530 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=666d9496aeb7a850 server_cookie=33c096e302e27f00 in_seq=0 out_seq=0 2024-04-01T01:41:34.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.478+0000 7f5ba8ff9640 1 -- 172.21.15.70:0/2082712370 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562193abda10 con 0x562193bce110 2024-04-01T01:41:34.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.478+0000 7f5ba8ff9640 10 client.17868 handle_command_reply: tid=0 2024-04-01T01:41:34.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.478+0000 7f5bca1bf740 1 client.17868 shutdown 2024-04-01T01:41:34.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.478+0000 7f5bca1bf740 1 -- 172.21.15.70:0/2082712370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b610e0 msgr2=0x562193b7a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.478+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/2082712370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b610e0 0x562193b7a010 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0050c90 tx=0x7f5bc011f9e0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.479+0000 7f5bca1bf740 1 -- 172.21.15.70:0/2082712370 shutdown_connections 2024-04-01T01:41:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.479+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/2082712370 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562193bce110 0x562193bd0530 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.479+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/2082712370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562193b610e0 0x562193b7a010 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.479+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/2082712370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562193b51e80 0x562193b79000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.479+0000 7f5bca1bf740 1 --2- 172.21.15.70:0/2082712370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562193b51490 0x562193b78ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.479+0000 7f5bca1bf740 1 -- 172.21.15.70:0/2082712370 >> 172.21.15.70:0/2082712370 conn(0x562193b4ca00 msgr2=0x562193b47ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.479+0000 7f5ba8ff9640 0 client.17868 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.479+0000 7f5bca1bf740 1 -- 172.21.15.70:0/2082712370 shutdown_connections 2024-04-01T01:41:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.479+0000 7f5bca1bf740 1 -- 172.21.15.70:0/2082712370 wait complete. 2024-04-01T01:41:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.479+0000 7f5bca1bf740 20 client.17868 trim_cache size 0 max 16384 2024-04-01T01:41:34.486 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:41:34.486 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:41:34.486 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:41:34.486 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:41:34.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.480+0000 7f5bc6380640 1 -- 172.21.15.70:0/3491392456 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5bac0a2670 msgr2=0x7f5bac0a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.480+0000 7f5bc6380640 1 --2- 172.21.15.70:0/3491392456 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5bac0a2670 0x7f5bac0a4b30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x562193b52280 tx=0x7f5bbc005ec0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.480+0000 7f5bc6380640 1 -- 172.21.15.70:0/3491392456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5bb8099910 msgr2=0x7f5bb814a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.480+0000 7f5bc6380640 1 --2- 172.21.15.70:0/3491392456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5bb8099910 0x7f5bb814a930 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0050db0 tx=0x7f5bc00699e0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.480+0000 7f5bc6380640 1 -- 172.21.15.70:0/3491392456 shutdown_connections 2024-04-01T01:41:34.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.480+0000 7f5bc6380640 1 --2- 172.21.15.70:0/3491392456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5bb8099910 0x7f5bb814a930 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.480+0000 7f5bc6380640 1 --2- 172.21.15.70:0/3491392456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb8096f60 0x7f5bb81438c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.480+0000 7f5bc6380640 1 --2- 172.21.15.70:0/3491392456 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5bac0a2670 0x7f5bac0a4b30 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.480+0000 7f5bc6380640 1 --2- 172.21.15.70:0/3491392456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5bb80945f0 0x7f5bb8143380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.480+0000 7f5bc6380640 1 -- 172.21.15.70:0/3491392456 >> 172.21.15.70:0/3491392456 conn(0x7f5bb80901f0 msgr2=0x7f5bb80978c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:34.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.480+0000 7f5bc6380640 1 -- 172.21.15.70:0/3491392456 shutdown_connections 2024-04-01T01:41:34.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.480+0000 7f5bc6380640 1 -- 172.21.15.70:0/3491392456 wait complete. 2024-04-01T01:41:34.495 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:41:34.495 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:41:34.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.667+0000 7ffbf517a640 1 -- 172.21.15.70:0/15922769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbf0108750 msgr2=0x7ffbf010ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.667+0000 7ffbf517a640 1 --2- 172.21.15.70:0/15922769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbf0108750 0x7ffbf010ab80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0009860 tx=0x7ffbe002bb90 comp rx=0 tx=0).stop 2024-04-01T01:41:34.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.668+0000 7ffbf517a640 1 -- 172.21.15.70:0/15922769 shutdown_connections 2024-04-01T01:41:34.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.668+0000 7ffbf517a640 1 --2- 172.21.15.70:0/15922769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbf010da90 0x7ffbf010df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.668+0000 7ffbf517a640 1 --2- 172.21.15.70:0/15922769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbf010b0c0 0x7ffbf010d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.668+0000 7ffbf517a640 1 --2- 172.21.15.70:0/15922769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbf0108750 0x7ffbf010ab80 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.668+0000 7ffbf517a640 1 -- 172.21.15.70:0/15922769 >> 172.21.15.70:0/15922769 conn(0x7ffbf006e5c0 msgr2=0x7ffbf0070a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:34.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.671+0000 7ffbf517a640 1 -- 172.21.15.70:0/15922769 shutdown_connections 2024-04-01T01:41:34.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.672+0000 7ffbf517a640 1 -- 172.21.15.70:0/15922769 wait complete. 2024-04-01T01:41:34.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.675+0000 7ffbf517a640 1 Processor -- start 2024-04-01T01:41:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.677+0000 7ffbf517a640 1 -- start start 2024-04-01T01:41:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.677+0000 7ffbf517a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbf0108750 0x7ffbf01a6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.677+0000 7ffbf517a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbf010b0c0 0x7ffbf01a6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.677+0000 7ffbf517a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbf010da90 0x7ffbf01add70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.677+0000 7ffbf517a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf011c0f0 con 0x7ffbf010b0c0 2024-04-01T01:41:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.677+0000 7ffbf517a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf011bf70 con 0x7ffbf0108750 2024-04-01T01:41:34.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.677+0000 7ffbf517a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf011c270 con 0x7ffbf010da90 2024-04-01T01:41:34.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.678+0000 7ffbef59e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbf010da90 0x7ffbf01add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.678+0000 7ffbef59e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbf010da90 0x7ffbf01add70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57744/0 (socket says 172.21.15.70:57744) 2024-04-01T01:41:34.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.678+0000 7ffbef59e640 1 -- 172.21.15.70:0/1453606326 learned_addr learned my addr 172.21.15.70:0/1453606326 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:34.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.678+0000 7ffbee59c640 1 --2- 172.21.15.70:0/1453606326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbf010b0c0 0x7ffbf01a6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.678+0000 7ffbef59e640 1 -- 172.21.15.70:0/1453606326 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbf0108750 msgr2=0x7ffbf01a6810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:41:34.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.678+0000 7ffbef59e640 1 --2- 172.21.15.70:0/1453606326 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbf0108750 0x7ffbf01a6810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.678+0000 7ffbef59e640 1 -- 172.21.15.70:0/1453606326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbf010b0c0 msgr2=0x7ffbf01a6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:34.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.678+0000 7ffbef59e640 1 --2- 172.21.15.70:0/1453606326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbf010b0c0 0x7ffbf01a6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:34.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.678+0000 7ffbef59e640 1 -- 172.21.15.70:0/1453606326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbf01ae3e0 con 0x7ffbf010da90 2024-04-01T01:41:34.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.679+0000 7ffbef59e640 1 --2- 172.21.15.70:0/1453606326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbf010da90 0x7ffbf01add70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ffbe400e970 tx=0x7ffbe400ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:34.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.679+0000 7ffbd7fff640 1 -- 172.21.15.70:0/1453606326 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe400ccc0 con 0x7ffbf010da90 2024-04-01T01:41:34.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.679+0000 7ffbd7fff640 1 -- 172.21.15.70:0/1453606326 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffbe40054d0 con 0x7ffbf010da90 2024-04-01T01:41:34.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.679+0000 7ffbf517a640 1 -- 172.21.15.70:0/1453606326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbf01ae6d0 con 0x7ffbf010da90 2024-04-01T01:41:34.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.680+0000 7ffbd7fff640 1 -- 172.21.15.70:0/1453606326 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe4010430 con 0x7ffbf010da90 2024-04-01T01:41:34.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.680+0000 7ffbf517a640 1 -- 172.21.15.70:0/1453606326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbf01aec20 con 0x7ffbf010da90 2024-04-01T01:41:34.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.681+0000 7ffbf517a640 1 -- 172.21.15.70:0/1453606326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbf00710c0 con 0x7ffbf010da90 2024-04-01T01:41:34.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.685+0000 7ffbd7fff640 1 -- 172.21.15.70:0/1453606326 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7ffbe40046c0 con 0x7ffbf010da90 2024-04-01T01:41:34.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.687+0000 7ffbd7fff640 1 --2- 172.21.15.70:0/1453606326 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ffbd80a25a0 0x7ffbd80a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:34.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.687+0000 7ffbeed9d640 1 --2- 172.21.15.70:0/1453606326 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ffbd80a25a0 0x7ffbd80a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:34.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.687+0000 7ffbd7fff640 1 -- 172.21.15.70:0/1453606326 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ffbe40cf220 con 0x7ffbf010da90 2024-04-01T01:41:34.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.688+0000 7ffbeed9d640 1 --2- 172.21.15.70:0/1453606326 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ffbd80a25a0 0x7ffbd80a4a60 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ffbe002c0d0 tx=0x7ffbe00058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:34.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:34.694+0000 7ffbd7fff640 1 -- 172.21.15.70:0/1453606326 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ffbe409c040 con 0x7ffbf010da90 2024-04-01T01:41:35.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.115+0000 7ffbf517a640 1 -- 172.21.15.70:0/1453606326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffbf0062350 con 0x7ffbf010da90 2024-04-01T01:41:35.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.117+0000 7ffbd7fff640 1 -- 172.21.15.70:0/1453606326 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7ffbe409b940 con 0x7ffbf010da90 2024-04-01T01:41:35.116 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:41:35.116 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:41:35.117 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:41:35.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.120+0000 7ffbf517a640 1 -- 172.21.15.70:0/1453606326 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ffbd80a25a0 msgr2=0x7ffbd80a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.120+0000 7ffbf517a640 1 --2- 172.21.15.70:0/1453606326 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ffbd80a25a0 0x7ffbd80a4a60 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ffbe002c0d0 tx=0x7ffbe00058e0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.120+0000 7ffbf517a640 1 -- 172.21.15.70:0/1453606326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbf010da90 msgr2=0x7ffbf01add70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.120+0000 7ffbf517a640 1 --2- 172.21.15.70:0/1453606326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbf010da90 0x7ffbf01add70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ffbe400e970 tx=0x7ffbe400ee30 comp rx=0 tx=0).stop 2024-04-01T01:41:35.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.121+0000 7ffbf517a640 1 -- 172.21.15.70:0/1453606326 shutdown_connections 2024-04-01T01:41:35.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.121+0000 7ffbf517a640 1 --2- 172.21.15.70:0/1453606326 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ffbd80a25a0 0x7ffbd80a4a60 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.121+0000 7ffbf517a640 1 --2- 172.21.15.70:0/1453606326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbf010da90 0x7ffbf01add70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.121+0000 7ffbf517a640 1 --2- 172.21.15.70:0/1453606326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffbf010b0c0 0x7ffbf01a6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.121+0000 7ffbf517a640 1 --2- 172.21.15.70:0/1453606326 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbf0108750 0x7ffbf01a6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.121+0000 7ffbf517a640 1 -- 172.21.15.70:0/1453606326 >> 172.21.15.70:0/1453606326 conn(0x7ffbf006e5c0 msgr2=0x7ffbf010b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.122+0000 7ffbf517a640 1 -- 172.21.15.70:0/1453606326 shutdown_connections 2024-04-01T01:41:35.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.122+0000 7ffbf517a640 1 -- 172.21.15.70:0/1453606326 wait complete. 2024-04-01T01:41:35.141 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:34 smithi070 ceph-mon[32179]: pgmap v602: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.8 MiB/s wr, 842 op/s 2024-04-01T01:41:35.143 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:41:35.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:34 smithi189 ceph-mon[35753]: pgmap v602: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.8 MiB/s wr, 842 op/s 2024-04-01T01:41:35.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.323+0000 7fc508f1f640 1 -- 172.21.15.70:0/1139682816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc50410fb20 msgr2=0x7fc504111f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.323+0000 7fc508f1f640 1 --2- 172.21.15.70:0/1139682816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc50410fb20 0x7fc504111f70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009ca0 tx=0x7fc4fc02c100 comp rx=0 tx=0).stop 2024-04-01T01:41:35.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.323+0000 7fc508f1f640 1 -- 172.21.15.70:0/1139682816 shutdown_connections 2024-04-01T01:41:35.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.323+0000 7fc508f1f640 1 --2- 172.21.15.70:0/1139682816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc50410fb20 0x7fc504111f70 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.323+0000 7fc508f1f640 1 --2- 172.21.15.70:0/1139682816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc504072d90 0x7fc504073210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.323+0000 7fc508f1f640 1 --2- 172.21.15.70:0/1139682816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5041093f0 0x7fc5040727c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.323+0000 7fc508f1f640 1 -- 172.21.15.70:0/1139682816 >> 172.21.15.70:0/1139682816 conn(0x7fc50406e630 msgr2=0x7fc504070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.324+0000 7fc508f1f640 1 -- 172.21.15.70:0/1139682816 shutdown_connections 2024-04-01T01:41:35.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.324+0000 7fc508f1f640 1 -- 172.21.15.70:0/1139682816 wait complete. 2024-04-01T01:41:35.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.325+0000 7fc508f1f640 1 Processor -- start 2024-04-01T01:41:35.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.325+0000 7fc508f1f640 1 -- start start 2024-04-01T01:41:35.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.326+0000 7fc508f1f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc504072d90 0x7fc50419ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.326+0000 7fc508f1f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5041093f0 0x7fc5041a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.326+0000 7fc508f1f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc50410fb20 0x7fc5041a7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.326+0000 7fc508f1f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc504115930 con 0x7fc504072d90 2024-04-01T01:41:35.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.326+0000 7fc508f1f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc5041157b0 con 0x7fc5041093f0 2024-04-01T01:41:35.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.326+0000 7fc508f1f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc504115ab0 con 0x7fc50410fb20 2024-04-01T01:41:35.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.327+0000 7fc503fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc50410fb20 0x7fc5041a7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.327+0000 7fc503fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc50410fb20 0x7fc5041a7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57750/0 (socket says 172.21.15.70:57750) 2024-04-01T01:41:35.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.327+0000 7fc503fff640 1 -- 172.21.15.70:0/2350604499 learned_addr learned my addr 172.21.15.70:0/2350604499 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:35.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.327+0000 7fc5037fe640 1 --2- 172.21.15.70:0/2350604499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc504072d90 0x7fc50419ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.327+0000 7fc5037fe640 1 -- 172.21.15.70:0/2350604499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc50410fb20 msgr2=0x7fc5041a7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.327+0000 7fc5037fe640 1 --2- 172.21.15.70:0/2350604499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc50410fb20 0x7fc5041a7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.327+0000 7fc5037fe640 1 -- 172.21.15.70:0/2350604499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5041093f0 msgr2=0x7fc5041a04d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:41:35.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.327+0000 7fc5037fe640 1 --2- 172.21.15.70:0/2350604499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5041093f0 0x7fc5041a04d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.327+0000 7fc5037fe640 1 -- 172.21.15.70:0/2350604499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5041a7bb0 con 0x7fc504072d90 2024-04-01T01:41:35.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.327+0000 7fc5037fe640 1 --2- 172.21.15.70:0/2350604499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc504072d90 0x7fc50419ff90 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800c940 tx=0x7fc4f800ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.328+0000 7fc500ff9640 1 -- 172.21.15.70:0/2350604499 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8007bb0 con 0x7fc504072d90 2024-04-01T01:41:35.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.328+0000 7fc500ff9640 1 -- 172.21.15.70:0/2350604499 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4f8007d50 con 0x7fc504072d90 2024-04-01T01:41:35.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.328+0000 7fc500ff9640 1 -- 172.21.15.70:0/2350604499 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8018780 con 0x7fc504072d90 2024-04-01T01:41:35.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.328+0000 7fc508f1f640 1 -- 172.21.15.70:0/2350604499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5041098c0 con 0x7fc504072d90 2024-04-01T01:41:35.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.328+0000 7fc508f1f640 1 -- 172.21.15.70:0/2350604499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc504109dc0 con 0x7fc504072d90 2024-04-01T01:41:35.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:34 smithi081 ceph-mon[35142]: pgmap v602: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.8 MiB/s wr, 842 op/s 2024-04-01T01:41:35.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.334+0000 7fc50cd67740 1 Processor -- start 2024-04-01T01:41:35.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.334+0000 7fc500ff9640 1 -- 172.21.15.70:0/2350604499 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fc4f800f450 con 0x7fc504072d90 2024-04-01T01:41:35.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.334+0000 7fc50cd67740 1 -- start start 2024-04-01T01:41:35.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.334+0000 7fc50cd67740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae3eb0 0x5594a9ae42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.334+0000 7fc50cd67740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae48a0 0x5594a9ae93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.334+0000 7fc50cd67740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae9a10 0x5594a9aebea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.334+0000 7fc50cd67740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5594a9a401b0 con 0x5594a9ae9a10 2024-04-01T01:41:35.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.334+0000 7fc50cd67740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5594a99e8c00 con 0x5594a9ae48a0 2024-04-01T01:41:35.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.334+0000 7fc50cd67740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5594a9a3c250 con 0x5594a9ae3eb0 2024-04-01T01:41:35.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.334+0000 7fc502ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae9a10 0x5594a9aebea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.334+0000 7fc503fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae3eb0 0x5594a9ae42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.335+0000 7fc503fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae3eb0 0x5594a9ae42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57756/0 (socket says 172.21.15.70:57756) 2024-04-01T01:41:35.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.335+0000 7fc5037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae48a0 0x5594a9ae93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.335+0000 7fc503fff640 1 -- 172.21.15.70:0/1762328488 learned_addr learned my addr 172.21.15.70:0/1762328488 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:35.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.336+0000 7fc500ff9640 1 --2- 172.21.15.70:0/2350604499 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc4ec0a25e0 0x7fc4ec0a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.336+0000 7fc500ff9640 1 -- 172.21.15.70:0/2350604499 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc4f80ceeb0 con 0x7fc504072d90 2024-04-01T01:41:35.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.337+0000 7fc503fff640 1 --2- 172.21.15.70:0/2350604499 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc4ec0a25e0 0x7fc4ec0a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.337+0000 7fc502ffd640 1 -- 172.21.15.70:0/1762328488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae3eb0 msgr2=0x5594a9ae42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.337+0000 7fc502ffd640 1 --2- 172.21.15.70:0/1762328488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae3eb0 0x5594a9ae42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.337+0000 7fc502ffd640 1 -- 172.21.15.70:0/1762328488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae48a0 msgr2=0x5594a9ae93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.337+0000 7fc502ffd640 1 --2- 172.21.15.70:0/1762328488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae48a0 0x5594a9ae93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.337+0000 7fc502ffd640 1 -- 172.21.15.70:0/1762328488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594a9a083d0 con 0x5594a9ae9a10 2024-04-01T01:41:35.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.337+0000 7fc502ffd640 1 --2- 172.21.15.70:0/1762328488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae9a10 0x5594a9aebea0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4009aa0 tx=0x7fc4f4009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=302cb4374d253342 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.337+0000 7fc4e9ffb640 1 -- 172.21.15.70:0/1762328488 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f400ece0 con 0x5594a9ae9a10 2024-04-01T01:41:35.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.337+0000 7fc4e9ffb640 1 -- 172.21.15.70:0/1762328488 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4f400ee80 con 0x5594a9ae9a10 2024-04-01T01:41:35.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.338+0000 7fc4e9ffb640 1 -- 172.21.15.70:0/1762328488 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f40049e0 con 0x5594a9ae9a10 2024-04-01T01:41:35.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.338+0000 7fc503fff640 1 --2- 172.21.15.70:0/2350604499 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc4ec0a25e0 0x7fc4ec0a4aa0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x5594a9ae5100 tx=0x7fc4fc02c100 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.338+0000 7fc50cd67740 1 -- 172.21.15.70:0/1762328488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae9a10 msgr2=0x5594a9aebea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.339+0000 7fc50cd67740 1 --2- 172.21.15.70:0/1762328488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae9a10 0x5594a9aebea0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4009aa0 tx=0x7fc4f4009f60 comp rx=0 tx=0).stop 2024-04-01T01:41:35.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.339+0000 7fc50cd67740 1 -- 172.21.15.70:0/1762328488 shutdown_connections 2024-04-01T01:41:35.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.339+0000 7fc50cd67740 1 --2- 172.21.15.70:0/1762328488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae9a10 0x5594a9aebea0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.339+0000 7fc50cd67740 1 --2- 172.21.15.70:0/1762328488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae48a0 0x5594a9ae93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.339+0000 7fc50cd67740 1 --2- 172.21.15.70:0/1762328488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae3eb0 0x5594a9ae42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.339+0000 7fc50cd67740 1 -- 172.21.15.70:0/1762328488 >> 172.21.15.70:0/1762328488 conn(0x5594a9adf420 msgr2=0x5594a9adf820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.339+0000 7fc50cd67740 1 -- 172.21.15.70:0/1762328488 shutdown_connections 2024-04-01T01:41:35.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.339+0000 7fc50cd67740 1 -- 172.21.15.70:0/1762328488 wait complete. 2024-04-01T01:41:35.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.339+0000 7fc50cd67740 1 -- start start 2024-04-01T01:41:35.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.340+0000 7fc50cd67740 1 Processor -- start 2024-04-01T01:41:35.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.340+0000 7fc50cd67740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae3eb0 0x5594a9aee460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.340+0000 7fc50cd67740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae48a0 0x5594a9aee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.340+0000 7fc50cd67740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae9a10 0x5594a9af59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.340+0000 7fc50cd67740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5594a9aeca10 con 0x5594a9ae3eb0 2024-04-01T01:41:35.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.340+0000 7fc50cd67740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5594a99de470 con 0x5594a9ae48a0 2024-04-01T01:41:35.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.340+0000 7fc50cd67740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5594a9aecb90 con 0x5594a9ae9a10 2024-04-01T01:41:35.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.340+0000 7fc50cd67740 10 client.0 _ll_register_callbacks cb 0x5594a99fbcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:35.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.341+0000 7fc502ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae48a0 0x5594a9aee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.341+0000 7fc5037fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae3eb0 0x5594a9aee460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.341+0000 7fc5037fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae3eb0 0x5594a9aee460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34022/0 (socket says 172.21.15.70:34022) 2024-04-01T01:41:35.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.341+0000 7fc5037fe640 1 -- 172.21.15.70:0/3124790177 learned_addr learned my addr 172.21.15.70:0/3124790177 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:35.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.341+0000 7fc5037fe640 1 -- 172.21.15.70:0/3124790177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae9a10 msgr2=0x5594a9af59e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:41:35.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.341+0000 7fc5037fe640 1 --2- 172.21.15.70:0/3124790177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae9a10 0x5594a9af59e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.341+0000 7fc5037fe640 1 -- 172.21.15.70:0/3124790177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae48a0 msgr2=0x5594a9aee9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.341+0000 7fc5037fe640 1 --2- 172.21.15.70:0/3124790177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae48a0 0x5594a9aee9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.341+0000 7fc5037fe640 1 -- 172.21.15.70:0/3124790177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594a97543f0 con 0x5594a9ae3eb0 2024-04-01T01:41:35.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.341+0000 7fc5037fe640 1 --2- 172.21.15.70:0/3124790177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae3eb0 0x5594a9aee460 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8097f90 tx=0x7fc4f8097950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ca0e5b7dc34bad server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.341+0000 7fc4e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:41:35.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.342+0000 7fc4e2ffd640 1 -- 172.21.15.70:0/3124790177 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f800fa90 con 0x5594a9ae3eb0 2024-04-01T01:41:35.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.342+0000 7fc50cd67740 1 -- 172.21.15.70:0/3124790177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5594a9a4d6f0 con 0x5594a9ae3eb0 2024-04-01T01:41:35.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.342+0000 7fc4e2ffd640 1 -- 172.21.15.70:0/3124790177 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc4f809aa10 con 0x5594a9ae3eb0 2024-04-01T01:41:35.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.342+0000 7fc4e2ffd640 1 -- 172.21.15.70:0/3124790177 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc4f809bcc0 con 0x5594a9ae3eb0 2024-04-01T01:41:35.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.343+0000 7fc50cd67740 10 client.17886 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:35.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.343+0000 7fc50cd67740 1 -- 172.21.15.70:0/3124790177 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5594a981e7d0 con 0x5594a9ae3eb0 2024-04-01T01:41:35.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.344+0000 7fc4e2ffd640 1 -- 172.21.15.70:0/3124790177 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5594a9a4d6f0 con 0x5594a9ae3eb0 2024-04-01T01:41:35.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.344+0000 7fc4e2ffd640 1 -- 172.21.15.70:0/3124790177 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc4f8020070 con 0x5594a9ae3eb0 2024-04-01T01:41:35.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.344+0000 7fc50cd67740 10 client.17886 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:35.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.344+0000 7fc50cd67740 10 client.17886 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:41:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.344+0000 7fc50cd67740 20 client.17886 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.344+0000 7fc50cd67740 1 --2- 172.21.15.70:0/3124790177 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5594a9af9f50 0x5594a9afc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.344+0000 7fc50cd67740 4 client.17886 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.344+0000 7fc50cd67740 1 -- 172.21.15.70:0/3124790177 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594a9a47a40 con 0x5594a9af9f50 2024-04-01T01:41:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.345+0000 7fc502ffd640 1 --2- 172.21.15.70:0/3124790177 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5594a9af9f50 0x5594a9afc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.345+0000 7fc4e2ffd640 10 client.17886 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:41:35.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.346+0000 7fc502ffd640 1 --2- 172.21.15.70:0/3124790177 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5594a9af9f50 0x5594a9afc3e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c7c4e1fffe92b167 server_cookie=920989c31842fcb5 in_seq=0 out_seq=0 2024-04-01T01:41:35.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.347+0000 7fc4e2ffd640 1 -- 172.21.15.70:0/3124790177 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5594a9a47a40 con 0x5594a9af9f50 2024-04-01T01:41:35.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.347+0000 7fc4e2ffd640 10 client.17886 handle_command_reply: tid=0 2024-04-01T01:41:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.347+0000 7fc50cd67740 1 client.17886 shutdown 2024-04-01T01:41:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.347+0000 7fc50cd67740 1 -- 172.21.15.70:0/3124790177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae3eb0 msgr2=0x5594a9aee460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.347+0000 7fc50cd67740 1 --2- 172.21.15.70:0/3124790177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae3eb0 0x5594a9aee460 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8097f90 tx=0x7fc4f8097950 comp rx=0 tx=0).stop 2024-04-01T01:41:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.347+0000 7fc50cd67740 1 -- 172.21.15.70:0/3124790177 shutdown_connections 2024-04-01T01:41:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.347+0000 7fc50cd67740 1 --2- 172.21.15.70:0/3124790177 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5594a9af9f50 0x5594a9afc3e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.347+0000 7fc50cd67740 1 --2- 172.21.15.70:0/3124790177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae9a10 0x5594a9af59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.347+0000 7fc50cd67740 1 --2- 172.21.15.70:0/3124790177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae48a0 0x5594a9aee9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.347+0000 7fc50cd67740 1 --2- 172.21.15.70:0/3124790177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae3eb0 0x5594a9aee460 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.348+0000 7fc50cd67740 1 -- 172.21.15.70:0/3124790177 >> 172.21.15.70:0/3124790177 conn(0x5594a9adf420 msgr2=0x5594a9ad5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.348+0000 7fc4e2ffd640 0 client.17886 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:41:35.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.348+0000 7fc50cd67740 1 -- 172.21.15.70:0/3124790177 shutdown_connections 2024-04-01T01:41:35.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.348+0000 7fc50cd67740 1 -- 172.21.15.70:0/3124790177 wait complete. 2024-04-01T01:41:35.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.348+0000 7fc50cd67740 20 client.17886 trim_cache size 0 max 16384 2024-04-01T01:41:35.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.378+0000 7fc50cd67740 1 Processor -- start 2024-04-01T01:41:35.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.378+0000 7fc50cd67740 1 -- start start 2024-04-01T01:41:35.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.378+0000 7fc50cd67740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae3eb0 0x5594a9aee780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.378+0000 7fc50cd67740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae48a0 0x5594a9af59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.378+0000 7fc50cd67740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae9a10 0x5594a9ad5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.378+0000 7fc50cd67740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc5041b0770 con 0x5594a9ae48a0 2024-04-01T01:41:35.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.378+0000 7fc50cd67740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc5041b22b0 con 0x5594a9ae9a10 2024-04-01T01:41:35.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.378+0000 7fc50cd67740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc5041b0e40 con 0x5594a9ae3eb0 2024-04-01T01:41:35.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.379+0000 7fc503fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae9a10 0x5594a9ad5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.379+0000 7fc503fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae9a10 0x5594a9ad5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57730/0 (socket says 172.21.15.70:57730) 2024-04-01T01:41:35.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.379+0000 7fc503fff640 1 -- 172.21.15.70:0/991146639 learned_addr learned my addr 172.21.15.70:0/991146639 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:35.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.380+0000 7fc502ffd640 1 --2- 172.21.15.70:0/991146639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae48a0 0x5594a9af59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.380+0000 7fc5037fe640 1 --2- 172.21.15.70:0/991146639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae3eb0 0x5594a9aee780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.380+0000 7fc5037fe640 1 -- 172.21.15.70:0/991146639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae9a10 msgr2=0x5594a9ad5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.380+0000 7fc5037fe640 1 --2- 172.21.15.70:0/991146639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae9a10 0x5594a9ad5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.381+0000 7fc5037fe640 1 -- 172.21.15.70:0/991146639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae48a0 msgr2=0x5594a9af59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.381+0000 7fc5037fe640 1 --2- 172.21.15.70:0/991146639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae48a0 0x5594a9af59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.381+0000 7fc5037fe640 1 -- 172.21.15.70:0/991146639 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594a9ae9760 con 0x5594a9ae3eb0 2024-04-01T01:41:35.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.381+0000 7fc5037fe640 1 --2- 172.21.15.70:0/991146639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae3eb0 0x5594a9aee780 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc4f809b2c0 tx=0x7fc4f809ae70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d2d6212a8507f43 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.381+0000 7fc4e97fa640 1 -- 172.21.15.70:0/991146639 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f800fa90 con 0x5594a9ae3eb0 2024-04-01T01:41:35.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.382+0000 7fc4e97fa640 1 -- 172.21.15.70:0/991146639 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4f809f710 con 0x5594a9ae3eb0 2024-04-01T01:41:35.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.382+0000 7fc4e97fa640 1 -- 172.21.15.70:0/991146639 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f809f170 con 0x5594a9ae3eb0 2024-04-01T01:41:35.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.382+0000 7fc50cd67740 1 -- 172.21.15.70:0/991146639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae3eb0 msgr2=0x5594a9aee780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.382+0000 7fc50cd67740 1 --2- 172.21.15.70:0/991146639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae3eb0 0x5594a9aee780 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc4f809b2c0 tx=0x7fc4f809ae70 comp rx=0 tx=0).stop 2024-04-01T01:41:35.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.383+0000 7fc50cd67740 1 -- 172.21.15.70:0/991146639 shutdown_connections 2024-04-01T01:41:35.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.383+0000 7fc50cd67740 1 --2- 172.21.15.70:0/991146639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae9a10 0x5594a9ad5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.383+0000 7fc50cd67740 1 --2- 172.21.15.70:0/991146639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae48a0 0x5594a9af59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.383+0000 7fc50cd67740 1 --2- 172.21.15.70:0/991146639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae3eb0 0x5594a9aee780 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.383+0000 7fc50cd67740 1 -- 172.21.15.70:0/991146639 >> 172.21.15.70:0/991146639 conn(0x5594a9adf420 msgr2=0x5594a9aee070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.383+0000 7fc50cd67740 1 -- 172.21.15.70:0/991146639 shutdown_connections 2024-04-01T01:41:35.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.383+0000 7fc50cd67740 1 -- 172.21.15.70:0/991146639 wait complete. 2024-04-01T01:41:35.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.383+0000 7fc50cd67740 1 -- start start 2024-04-01T01:41:35.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.384+0000 7fc50cd67740 1 Processor -- start 2024-04-01T01:41:35.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.384+0000 7fc50cd67740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae3eb0 0x5594a9af5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.384+0000 7fc50cd67740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae48a0 0x5594a9af5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.384+0000 7fc50cd67740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae9a10 0x5594a9af1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.384+0000 7fc50cd67740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc5041b1be0 con 0x5594a9ae3eb0 2024-04-01T01:41:35.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.384+0000 7fc50cd67740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc5041b1510 con 0x5594a9ae48a0 2024-04-01T01:41:35.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.384+0000 7fc50cd67740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc5041ae750 con 0x5594a9ae9a10 2024-04-01T01:41:35.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.384+0000 7fc50cd67740 10 client.0 _ll_register_callbacks cb 0x7fc4f80d2420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:35.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.384+0000 7fc5037fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae3eb0 0x5594a9af5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.384+0000 7fc5037fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae3eb0 0x5594a9af5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34034/0 (socket says 172.21.15.70:34034) 2024-04-01T01:41:35.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.384+0000 7fc5037fe640 1 -- 172.21.15.70:0/142915884 learned_addr learned my addr 172.21.15.70:0/142915884 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:35.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.384+0000 7fc503fff640 1 --2- 172.21.15.70:0/142915884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae9a10 0x5594a9af1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.384+0000 7fc502ffd640 1 --2- 172.21.15.70:0/142915884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae48a0 0x5594a9af5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.385+0000 7fc503fff640 1 -- 172.21.15.70:0/142915884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae48a0 msgr2=0x5594a9af5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.385+0000 7fc503fff640 1 --2- 172.21.15.70:0/142915884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae48a0 0x5594a9af5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.385+0000 7fc503fff640 1 -- 172.21.15.70:0/142915884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae3eb0 msgr2=0x5594a9af5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.385+0000 7fc503fff640 1 --2- 172.21.15.70:0/142915884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae3eb0 0x5594a9af5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.385+0000 7fc503fff640 1 -- 172.21.15.70:0/142915884 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594a9af2810 con 0x5594a9ae9a10 2024-04-01T01:41:35.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.385+0000 7fc503fff640 1 --2- 172.21.15.70:0/142915884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae9a10 0x5594a9af1fe0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc037040 tx=0x7fc4fc009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f1afb39cc5bae53 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.387+0000 7fc4e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:41:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.387+0000 7fc4e2ffd640 1 -- 172.21.15.70:0/142915884 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc053070 con 0x5594a9ae9a10 2024-04-01T01:41:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.388+0000 7fc50cd67740 1 -- 172.21.15.70:0/142915884 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4f809f710 con 0x5594a9ae9a10 2024-04-01T01:41:35.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.388+0000 7fc4e2ffd640 1 -- 172.21.15.70:0/142915884 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc4fc004450 con 0x5594a9ae9a10 2024-04-01T01:41:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.388+0000 7fc4e2ffd640 1 -- 172.21.15.70:0/142915884 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc4fc05f050 con 0x5594a9ae9a10 2024-04-01T01:41:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.388+0000 7fc50cd67740 10 client.27688 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.388+0000 7fc50cd67740 1 -- 172.21.15.70:0/142915884 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5594a9ada640 con 0x5594a9ae9a10 2024-04-01T01:41:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.389+0000 7fc4e2ffd640 1 -- 172.21.15.70:0/142915884 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4f809f710 con 0x5594a9ae9a10 2024-04-01T01:41:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.389+0000 7fc4e2ffd640 1 -- 172.21.15.70:0/142915884 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc4fc034c60 con 0x5594a9ae9a10 2024-04-01T01:41:35.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.389+0000 7fc50cd67740 10 client.27688 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.389+0000 7fc50cd67740 10 client.27688 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:41:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.389+0000 7fc50cd67740 20 client.27688 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.389+0000 7fc50cd67740 1 --2- 172.21.15.70:0/142915884 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5594a9adb070 0x5594a9add490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.389+0000 7fc50cd67740 4 client.27688 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:41:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.389+0000 7fc50cd67740 1 -- 172.21.15.70:0/142915884 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5594a9ae58b0 con 0x5594a9adb070 2024-04-01T01:41:35.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.390+0000 7fc5037fe640 1 --2- 172.21.15.70:0/142915884 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5594a9adb070 0x5594a9add490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.392+0000 7fc5037fe640 1 --2- 172.21.15.70:0/142915884 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5594a9adb070 0x5594a9add490 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=34e5a310ccd6a35 server_cookie=de55d6ee31d1efe in_seq=0 out_seq=0 2024-04-01T01:41:35.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.392+0000 7fc4e2ffd640 10 client.27688 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:41:35.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.392+0000 7fc4e2ffd640 1 -- 172.21.15.70:0/142915884 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5594a9ae58b0 con 0x5594a9adb070 2024-04-01T01:41:35.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.392+0000 7fc4e2ffd640 10 client.27688 handle_command_reply: tid=0 2024-04-01T01:41:35.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.392+0000 7fc50cd67740 1 client.27688 shutdown 2024-04-01T01:41:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.393+0000 7fc50cd67740 1 -- 172.21.15.70:0/142915884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae9a10 msgr2=0x5594a9af1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.393+0000 7fc50cd67740 1 --2- 172.21.15.70:0/142915884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae9a10 0x5594a9af1fe0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc037040 tx=0x7fc4fc009800 comp rx=0 tx=0).stop 2024-04-01T01:41:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.393+0000 7fc50cd67740 1 -- 172.21.15.70:0/142915884 shutdown_connections 2024-04-01T01:41:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.393+0000 7fc50cd67740 1 --2- 172.21.15.70:0/142915884 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5594a9adb070 0x5594a9add490 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.393+0000 7fc50cd67740 1 --2- 172.21.15.70:0/142915884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594a9ae9a10 0x5594a9af1fe0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.393+0000 7fc50cd67740 1 --2- 172.21.15.70:0/142915884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594a9ae48a0 0x5594a9af5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.393+0000 7fc50cd67740 1 --2- 172.21.15.70:0/142915884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594a9ae3eb0 0x5594a9af5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.393+0000 7fc50cd67740 1 -- 172.21.15.70:0/142915884 >> 172.21.15.70:0/142915884 conn(0x5594a9adf420 msgr2=0x5594a9ae2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.393+0000 7fc4e2ffd640 0 client.27688 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:41:35.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.393+0000 7fc50cd67740 1 -- 172.21.15.70:0/142915884 shutdown_connections 2024-04-01T01:41:35.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.393+0000 7fc50cd67740 1 -- 172.21.15.70:0/142915884 wait complete. 2024-04-01T01:41:35.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.394+0000 7fc50cd67740 20 client.27688 trim_cache size 0 max 16384 2024-04-01T01:41:35.394 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:41:35.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.396+0000 7fc508f1f640 1 -- 172.21.15.70:0/2350604499 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc4ec0a25e0 msgr2=0x7fc4ec0a4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.396+0000 7fc508f1f640 1 --2- 172.21.15.70:0/2350604499 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc4ec0a25e0 0x7fc4ec0a4aa0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x5594a9ae5100 tx=0x7fc4fc02c100 comp rx=0 tx=0).stop 2024-04-01T01:41:35.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.396+0000 7fc508f1f640 1 -- 172.21.15.70:0/2350604499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc504072d90 msgr2=0x7fc50419ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.396+0000 7fc508f1f640 1 --2- 172.21.15.70:0/2350604499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc504072d90 0x7fc50419ff90 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800c940 tx=0x7fc4f800ce00 comp rx=0 tx=0).stop 2024-04-01T01:41:35.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.397+0000 7fc508f1f640 1 -- 172.21.15.70:0/2350604499 shutdown_connections 2024-04-01T01:41:35.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.397+0000 7fc508f1f640 1 --2- 172.21.15.70:0/2350604499 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc4ec0a25e0 0x7fc4ec0a4aa0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.397+0000 7fc508f1f640 1 --2- 172.21.15.70:0/2350604499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc50410fb20 0x7fc5041a7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.397+0000 7fc508f1f640 1 --2- 172.21.15.70:0/2350604499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5041093f0 0x7fc5041a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.397+0000 7fc508f1f640 1 --2- 172.21.15.70:0/2350604499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc504072d90 0x7fc50419ff90 secure :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800c940 tx=0x7fc4f800ce00 comp rx=0 tx=0).stop 2024-04-01T01:41:35.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.397+0000 7fc508f1f640 1 -- 172.21.15.70:0/2350604499 >> 172.21.15.70:0/2350604499 conn(0x7fc50406e630 msgr2=0x7fc50410ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.401+0000 7fc508f1f640 1 -- 172.21.15.70:0/2350604499 shutdown_connections 2024-04-01T01:41:35.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.401+0000 7fc508f1f640 1 -- 172.21.15.70:0/2350604499 wait complete. 2024-04-01T01:41:35.425 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:41:35.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.608+0000 7fa2461a1640 1 -- 172.21.15.70:0/2039560508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa240102110 msgr2=0x7fa2401089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.608+0000 7fa2461a1640 1 --2- 172.21.15.70:0/2039560508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa240102110 0x7fa2401089a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fa238009ca0 tx=0x7fa23802c100 comp rx=0 tx=0).stop 2024-04-01T01:41:35.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.609+0000 7fa2461a1640 1 -- 172.21.15.70:0/2039560508 shutdown_connections 2024-04-01T01:41:35.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.609+0000 7fa2461a1640 1 --2- 172.21.15.70:0/2039560508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa240102110 0x7fa2401089a0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.609+0000 7fa2461a1640 1 --2- 172.21.15.70:0/2039560508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa240101750 0x7fa240101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.609+0000 7fa2461a1640 1 --2- 172.21.15.70:0/2039560508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa240100560 0x7fa240100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.609+0000 7fa2461a1640 1 -- 172.21.15.70:0/2039560508 >> 172.21.15.70:0/2039560508 conn(0x7fa2400fbaf0 msgr2=0x7fa2400fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.609+0000 7fa2461a1640 1 -- 172.21.15.70:0/2039560508 shutdown_connections 2024-04-01T01:41:35.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.609+0000 7fa2461a1640 1 -- 172.21.15.70:0/2039560508 wait complete. 2024-04-01T01:41:35.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.610+0000 7fa2461a1640 1 Processor -- start 2024-04-01T01:41:35.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.610+0000 7fa2461a1640 1 -- start start 2024-04-01T01:41:35.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.610+0000 7fa2461a1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa240100560 0x7fa2401999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.610+0000 7fa2461a1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa240101750 0x7fa240199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.610+0000 7fa2461a1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa240102110 0x7fa2401a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.610+0000 7fa2461a1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa24010af50 con 0x7fa240102110 2024-04-01T01:41:35.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.610+0000 7fa2461a1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa24010add0 con 0x7fa240101750 2024-04-01T01:41:35.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.610+0000 7fa2461a1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa24010b0d0 con 0x7fa240100560 2024-04-01T01:41:35.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.611+0000 7fa23effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa240101750 0x7fa240199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.611+0000 7fa23f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa240100560 0x7fa2401999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.611+0000 7fa23effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa240101750 0x7fa240199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57752/0 (socket says 172.21.15.70:57752) 2024-04-01T01:41:35.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.611+0000 7fa23effd640 1 -- 172.21.15.70:0/3112832978 learned_addr learned my addr 172.21.15.70:0/3112832978 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:35.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.611+0000 7fa23ffff640 1 --2- 172.21.15.70:0/3112832978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa240102110 0x7fa2401a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.611+0000 7fa23effd640 1 -- 172.21.15.70:0/3112832978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa240100560 msgr2=0x7fa2401999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.611+0000 7fa23effd640 1 --2- 172.21.15.70:0/3112832978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa240100560 0x7fa2401999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.611+0000 7fa23effd640 1 -- 172.21.15.70:0/3112832978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa240102110 msgr2=0x7fa2401a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.611+0000 7fa23effd640 1 --2- 172.21.15.70:0/3112832978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa240102110 0x7fa2401a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.611+0000 7fa23effd640 1 -- 172.21.15.70:0/3112832978 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2401a1600 con 0x7fa240101750 2024-04-01T01:41:35.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.611+0000 7fa23f7fe640 1 --2- 172.21.15.70:0/3112832978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa240100560 0x7fa2401999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:41:35.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.611+0000 7fa23effd640 1 --2- 172.21.15.70:0/3112832978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa240101750 0x7fa240199f20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa23400ea60 tx=0x7fa23400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.612+0000 7fa23cff9640 1 -- 172.21.15.70:0/3112832978 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23400cde0 con 0x7fa240101750 2024-04-01T01:41:35.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.612+0000 7fa2461a1640 1 -- 172.21.15.70:0/3112832978 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2401a18f0 con 0x7fa240101750 2024-04-01T01:41:35.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.612+0000 7fa23cff9640 1 -- 172.21.15.70:0/3112832978 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa2340035f0 con 0x7fa240101750 2024-04-01T01:41:35.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.612+0000 7fa2461a1640 1 -- 172.21.15.70:0/3112832978 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2401a1e90 con 0x7fa240101750 2024-04-01T01:41:35.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.613+0000 7fa23cff9640 1 -- 172.21.15.70:0/3112832978 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa234010690 con 0x7fa240101750 2024-04-01T01:41:35.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.615+0000 7fa23cff9640 1 -- 172.21.15.70:0/3112832978 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fa234010830 con 0x7fa240101750 2024-04-01T01:41:35.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.616+0000 7fa248d85740 1 Processor -- start 2024-04-01T01:41:35.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.616+0000 7fa23cff9640 1 --2- 172.21.15.70:0/3112832978 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa2280a2500 0x7fa2280a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.616+0000 7fa248d85740 1 -- start start 2024-04-01T01:41:35.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.616+0000 7fa23cff9640 1 -- 172.21.15.70:0/3112832978 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa2340cedd0 con 0x7fa240101750 2024-04-01T01:41:35.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.616+0000 7fa248d85740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866d9beb0 0x563866d9c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.616+0000 7fa248d85740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9c8a0 0x563866da13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.616+0000 7fa248d85740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866da1a10 0x563866da3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.616+0000 7fa248d85740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563866cf81b0 con 0x563866da1a10 2024-04-01T01:41:35.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.616+0000 7fa248d85740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563866ca0c00 con 0x563866d9beb0 2024-04-01T01:41:35.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.616+0000 7fa248d85740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563866cf4250 con 0x563866d9c8a0 2024-04-01T01:41:35.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.616+0000 7fa23ffff640 1 --2- 172.21.15.70:0/3112832978 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa2280a2500 0x7fa2280a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.616+0000 7fa23f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866d9beb0 0x563866d9c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.616+0000 7fa23f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866d9beb0 0x563866d9c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57760/0 (socket says 172.21.15.70:57760) 2024-04-01T01:41:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.616+0000 7fa23f7fe640 1 -- 172.21.15.70:0/787573836 learned_addr learned my addr 172.21.15.70:0/787573836 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.616+0000 7fa23ffff640 1 --2- 172.21.15.70:0/787573836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866da1a10 0x563866da3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.617+0000 7fa23f7fe640 1 -- 172.21.15.70:0/787573836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9c8a0 msgr2=0x563866da13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.617+0000 7fa23f7fe640 1 --2- 172.21.15.70:0/787573836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9c8a0 0x563866da13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.617+0000 7fa23f7fe640 1 -- 172.21.15.70:0/787573836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866da1a10 msgr2=0x563866da3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.617+0000 7fa23f7fe640 1 --2- 172.21.15.70:0/787573836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866da1a10 0x563866da3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.617+0000 7fa23f7fe640 1 -- 172.21.15.70:0/787573836 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563866cc03d0 con 0x563866d9beb0 2024-04-01T01:41:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.617+0000 7fa23f7fe640 1 --2- 172.21.15.70:0/787573836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866d9beb0 0x563866d9c2d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa2300114c0 tx=0x7fa230011980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c062b2d91e10138e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.617+0000 7fa23ffff640 1 --2- 172.21.15.70:0/3112832978 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa2280a2500 0x7fa2280a49c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa238006010 tx=0x7fa238005ef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.617+0000 7fa225ffb640 1 -- 172.21.15.70:0/787573836 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230004020 con 0x563866d9beb0 2024-04-01T01:41:35.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.617+0000 7fa225ffb640 1 -- 172.21.15.70:0/787573836 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa2300054d0 con 0x563866d9beb0 2024-04-01T01:41:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.617+0000 7fa225ffb640 1 -- 172.21.15.70:0/787573836 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230004d30 con 0x563866d9beb0 2024-04-01T01:41:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.618+0000 7fa248d85740 1 -- 172.21.15.70:0/787573836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866d9beb0 msgr2=0x563866d9c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.618+0000 7fa248d85740 1 --2- 172.21.15.70:0/787573836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866d9beb0 0x563866d9c2d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa2300114c0 tx=0x7fa230011980 comp rx=0 tx=0).stop 2024-04-01T01:41:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.618+0000 7fa248d85740 1 -- 172.21.15.70:0/787573836 shutdown_connections 2024-04-01T01:41:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.618+0000 7fa248d85740 1 --2- 172.21.15.70:0/787573836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866da1a10 0x563866da3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.618+0000 7fa248d85740 1 --2- 172.21.15.70:0/787573836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9c8a0 0x563866da13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.618+0000 7fa248d85740 1 --2- 172.21.15.70:0/787573836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866d9beb0 0x563866d9c2d0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.618+0000 7fa248d85740 1 -- 172.21.15.70:0/787573836 >> 172.21.15.70:0/787573836 conn(0x563866d8ec10 msgr2=0x563866d8f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.618+0000 7fa248d85740 1 -- 172.21.15.70:0/787573836 shutdown_connections 2024-04-01T01:41:35.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.618+0000 7fa248d85740 1 -- 172.21.15.70:0/787573836 wait complete. 2024-04-01T01:41:35.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.619+0000 7fa248d85740 1 -- start start 2024-04-01T01:41:35.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.619+0000 7fa248d85740 1 Processor -- start 2024-04-01T01:41:35.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.619+0000 7fa248d85740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9beb0 0x563866da6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.619+0000 7fa248d85740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866d9c8a0 0x563866da69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.619+0000 7fa248d85740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866da1a10 0x563866dad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.619+0000 7fa248d85740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563866da49c0 con 0x563866d9c8a0 2024-04-01T01:41:35.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.619+0000 7fa248d85740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563866c96470 con 0x563866da1a10 2024-04-01T01:41:35.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.619+0000 7fa248d85740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563866da4b40 con 0x563866d9beb0 2024-04-01T01:41:35.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.619+0000 7fa23ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866da1a10 0x563866dad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.619+0000 7fa23ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866da1a10 0x563866dad9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57774/0 (socket says 172.21.15.70:57774) 2024-04-01T01:41:35.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.619+0000 7fa23ffff640 1 -- 172.21.15.70:0/2749399845 learned_addr learned my addr 172.21.15.70:0/2749399845 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:35.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.619+0000 7fa248d85740 10 client.0 _ll_register_callbacks cb 0x563866cb3cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:35.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.619+0000 7fa23effd640 1 --2- 172.21.15.70:0/2749399845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866d9c8a0 0x563866da69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.619+0000 7fa23f7fe640 1 --2- 172.21.15.70:0/2749399845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9beb0 0x563866da6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.620+0000 7fa23ffff640 1 -- 172.21.15.70:0/2749399845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9beb0 msgr2=0x563866da6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.620+0000 7fa23ffff640 1 --2- 172.21.15.70:0/2749399845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9beb0 0x563866da6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.620+0000 7fa23ffff640 1 -- 172.21.15.70:0/2749399845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866d9c8a0 msgr2=0x563866da69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.620+0000 7fa23ffff640 1 --2- 172.21.15.70:0/2749399845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866d9c8a0 0x563866da69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.620+0000 7fa23ffff640 1 -- 172.21.15.70:0/2749399845 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563866a0c3f0 con 0x563866da1a10 2024-04-01T01:41:35.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.620+0000 7fa23ffff640 1 --2- 172.21.15.70:0/2749399845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866da1a10 0x563866dad9e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa238036000 tx=0x7fa238032480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82cf9bd68cc4213a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.620+0000 7fa21effd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:35.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.620+0000 7fa21effd640 1 -- 172.21.15.70:0/2749399845 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa238053060 con 0x563866da1a10 2024-04-01T01:41:35.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.620+0000 7fa248d85740 1 -- 172.21.15.70:0/2749399845 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2300054d0 con 0x563866da1a10 2024-04-01T01:41:35.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.620+0000 7fa21effd640 1 -- 172.21.15.70:0/2749399845 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa23802ccb0 con 0x563866da1a10 2024-04-01T01:41:35.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.620+0000 7fa21effd640 1 -- 172.21.15.70:0/2749399845 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa238039070 con 0x563866da1a10 2024-04-01T01:41:35.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.620+0000 7fa248d85740 10 client.27866 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:35.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.620+0000 7fa248d85740 1 -- 172.21.15.70:0/2749399845 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563866ad67d0 con 0x563866da1a10 2024-04-01T01:41:35.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.621+0000 7fa21effd640 1 -- 172.21.15.70:0/2749399845 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa23804e470 con 0x563866da1a10 2024-04-01T01:41:35.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.621+0000 7fa21effd640 1 -- 172.21.15.70:0/2749399845 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fa238034440 con 0x563866da1a10 2024-04-01T01:41:35.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.621+0000 7fa248d85740 10 client.27866 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:35.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.621+0000 7fa248d85740 10 client.27866 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:41:35.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.621+0000 7fa248d85740 20 client.27866 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:35.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.621+0000 7fa248d85740 1 --2- 172.21.15.70:0/2749399845 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x563866db1f70 0x563866db43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.621+0000 7fa248d85740 4 client.27866 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:35.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.621+0000 7fa248d85740 1 -- 172.21.15.70:0/2749399845 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa2300054d0 con 0x563866db1f70 2024-04-01T01:41:35.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.621+0000 7fa23f7fe640 1 --2- 172.21.15.70:0/2749399845 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x563866db1f70 0x563866db43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.622+0000 7fa23f7fe640 1 --2- 172.21.15.70:0/2749399845 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x563866db1f70 0x563866db43d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7167d6076a8b713d server_cookie=38872adf02033ebe in_seq=0 out_seq=0 2024-04-01T01:41:35.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.622+0000 7fa21effd640 10 client.27866 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:41:35.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.622+0000 7fa21effd640 1 -- 172.21.15.70:0/2749399845 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fa2300054d0 con 0x563866db1f70 2024-04-01T01:41:35.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.622+0000 7fa21effd640 10 client.27866 handle_command_reply: tid=0 2024-04-01T01:41:35.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.622+0000 7fa248d85740 1 client.27866 shutdown 2024-04-01T01:41:35.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.623+0000 7fa248d85740 1 -- 172.21.15.70:0/2749399845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866da1a10 msgr2=0x563866dad9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.623+0000 7fa248d85740 1 --2- 172.21.15.70:0/2749399845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866da1a10 0x563866dad9e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa238036000 tx=0x7fa238032480 comp rx=0 tx=0).stop 2024-04-01T01:41:35.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.623+0000 7fa248d85740 1 -- 172.21.15.70:0/2749399845 shutdown_connections 2024-04-01T01:41:35.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.623+0000 7fa248d85740 1 --2- 172.21.15.70:0/2749399845 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x563866db1f70 0x563866db43d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.623+0000 7fa248d85740 1 --2- 172.21.15.70:0/2749399845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866da1a10 0x563866dad9e0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.623+0000 7fa248d85740 1 --2- 172.21.15.70:0/2749399845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866d9c8a0 0x563866da69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.623+0000 7fa248d85740 1 --2- 172.21.15.70:0/2749399845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9beb0 0x563866da6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.623+0000 7fa248d85740 1 -- 172.21.15.70:0/2749399845 >> 172.21.15.70:0/2749399845 conn(0x563866d8ec10 msgr2=0x563866d8df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.623+0000 7fa21effd640 0 client.27866 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:41:35.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.623+0000 7fa248d85740 1 -- 172.21.15.70:0/2749399845 shutdown_connections 2024-04-01T01:41:35.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.623+0000 7fa248d85740 1 -- 172.21.15.70:0/2749399845 wait complete. 2024-04-01T01:41:35.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.623+0000 7fa248d85740 20 client.27866 trim_cache size 0 max 16384 2024-04-01T01:41:35.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.648+0000 7fa248d85740 1 Processor -- start 2024-04-01T01:41:35.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.648+0000 7fa248d85740 1 -- start start 2024-04-01T01:41:35.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.649+0000 7fa248d85740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9beb0 0x563866da5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.649+0000 7fa248d85740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866d9c8a0 0x563866dad900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.649+0000 7fa248d85740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866da1a10 0x563866da5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.649+0000 7fa248d85740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa22c00fcd0 con 0x563866d9c8a0 2024-04-01T01:41:35.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.649+0000 7fa248d85740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa22c011810 con 0x563866da1a10 2024-04-01T01:41:35.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.649+0000 7fa248d85740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa22c0103a0 con 0x563866d9beb0 2024-04-01T01:41:35.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.649+0000 7fa23f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9beb0 0x563866da5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.649+0000 7fa23ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866da1a10 0x563866da5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.649+0000 7fa23ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866da1a10 0x563866da5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57790/0 (socket says 172.21.15.70:57790) 2024-04-01T01:41:35.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.649+0000 7fa23ffff640 1 -- 172.21.15.70:0/3129707124 learned_addr learned my addr 172.21.15.70:0/3129707124 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:35.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.649+0000 7fa23effd640 1 --2- 172.21.15.70:0/3129707124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866d9c8a0 0x563866dad900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.649+0000 7fa23ffff640 1 -- 172.21.15.70:0/3129707124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9beb0 msgr2=0x563866da5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.649+0000 7fa23ffff640 1 --2- 172.21.15.70:0/3129707124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9beb0 0x563866da5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.649+0000 7fa23ffff640 1 -- 172.21.15.70:0/3129707124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866d9c8a0 msgr2=0x563866dad900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.649+0000 7fa23ffff640 1 --2- 172.21.15.70:0/3129707124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866d9c8a0 0x563866dad900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.649+0000 7fa23ffff640 1 -- 172.21.15.70:0/3129707124 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563866db0900 con 0x563866da1a10 2024-04-01T01:41:35.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.650+0000 7fa23ffff640 1 --2- 172.21.15.70:0/3129707124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866da1a10 0x563866da5c90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa238037040 tx=0x7fa2380098d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58a9d1b6876863cc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.650+0000 7fa2257fa640 1 -- 172.21.15.70:0/3129707124 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa238053070 con 0x563866da1a10 2024-04-01T01:41:35.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.650+0000 7fa2257fa640 1 -- 172.21.15.70:0/3129707124 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa238034b20 con 0x563866da1a10 2024-04-01T01:41:35.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.651+0000 7fa2257fa640 1 -- 172.21.15.70:0/3129707124 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23804e7f0 con 0x563866da1a10 2024-04-01T01:41:35.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.651+0000 7fa248d85740 1 -- 172.21.15.70:0/3129707124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866da1a10 msgr2=0x563866da5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.651+0000 7fa248d85740 1 --2- 172.21.15.70:0/3129707124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866da1a10 0x563866da5c90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa238037040 tx=0x7fa2380098d0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.651+0000 7fa248d85740 1 -- 172.21.15.70:0/3129707124 shutdown_connections 2024-04-01T01:41:35.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.651+0000 7fa248d85740 1 --2- 172.21.15.70:0/3129707124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866da1a10 0x563866da5c90 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.651+0000 7fa248d85740 1 --2- 172.21.15.70:0/3129707124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866d9c8a0 0x563866dad900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.651+0000 7fa248d85740 1 --2- 172.21.15.70:0/3129707124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9beb0 0x563866da5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.651+0000 7fa248d85740 1 -- 172.21.15.70:0/3129707124 >> 172.21.15.70:0/3129707124 conn(0x563866d8ec10 msgr2=0x563866d9de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.652+0000 7fa248d85740 1 -- 172.21.15.70:0/3129707124 shutdown_connections 2024-04-01T01:41:35.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.652+0000 7fa248d85740 1 -- 172.21.15.70:0/3129707124 wait complete. 2024-04-01T01:41:35.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.652+0000 7fa248d85740 1 -- start start 2024-04-01T01:41:35.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.652+0000 7fa248d85740 1 Processor -- start 2024-04-01T01:41:35.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.652+0000 7fa248d85740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866d9beb0 0x563866daaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.652+0000 7fa248d85740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9c8a0 0x563866daafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.652+0000 7fa248d85740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866da1a10 0x563866e162c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.652+0000 7fa248d85740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa22c011140 con 0x563866da1a10 2024-04-01T01:41:35.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.652+0000 7fa248d85740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa22c010a70 con 0x563866d9beb0 2024-04-01T01:41:35.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.652+0000 7fa248d85740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa22c00dcb0 con 0x563866d9c8a0 2024-04-01T01:41:35.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.653+0000 7fa248d85740 10 client.0 _ll_register_callbacks cb 0x7fa238009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:35.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.653+0000 7fa23f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866d9beb0 0x563866daaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.653+0000 7fa23f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866d9beb0 0x563866daaa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57794/0 (socket says 172.21.15.70:57794) 2024-04-01T01:41:35.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.653+0000 7fa23f7fe640 1 -- 172.21.15.70:0/633293286 learned_addr learned my addr 172.21.15.70:0/633293286 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:35.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.654+0000 7fa23ffff640 1 --2- 172.21.15.70:0/633293286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866da1a10 0x563866e162c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.655+0000 7fa23f7fe640 1 -- 172.21.15.70:0/633293286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9c8a0 msgr2=0x563866daafb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:41:35.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.655+0000 7fa23f7fe640 1 --2- 172.21.15.70:0/633293286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9c8a0 0x563866daafb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.655+0000 7fa23f7fe640 1 -- 172.21.15.70:0/633293286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866da1a10 msgr2=0x563866e162c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.655+0000 7fa23f7fe640 1 --2- 172.21.15.70:0/633293286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866da1a10 0x563866e162c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.655+0000 7fa23f7fe640 1 -- 172.21.15.70:0/633293286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563866e16880 con 0x563866d9beb0 2024-04-01T01:41:35.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.655+0000 7fa23ffff640 1 --2- 172.21.15.70:0/633293286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866da1a10 0x563866e162c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:41:35.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.655+0000 7fa23f7fe640 1 --2- 172.21.15.70:0/633293286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866d9beb0 0x563866daaa70 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa230009730 tx=0x7fa230011310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1300136c87f67801 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.655+0000 7fa21effd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:35.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.656+0000 7fa21effd640 1 -- 172.21.15.70:0/633293286 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23002e020 con 0x563866d9beb0 2024-04-01T01:41:35.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.656+0000 7fa21effd640 1 -- 172.21.15.70:0/633293286 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa230005280 con 0x563866d9beb0 2024-04-01T01:41:35.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.656+0000 7fa248d85740 1 -- 172.21.15.70:0/633293286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa238034b20 con 0x563866d9beb0 2024-04-01T01:41:35.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.656+0000 7fa21effd640 1 -- 172.21.15.70:0/633293286 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa23001d040 con 0x563866d9beb0 2024-04-01T01:41:35.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.656+0000 7fa248d85740 10 client.27872 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:35.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.656+0000 7fa248d85740 1 -- 172.21.15.70:0/633293286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563866e17bf0 con 0x563866d9beb0 2024-04-01T01:41:35.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.656+0000 7fa21effd640 1 -- 172.21.15.70:0/633293286 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa238034b20 con 0x563866d9beb0 2024-04-01T01:41:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.657+0000 7fa21effd640 1 -- 172.21.15.70:0/633293286 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fa23000f700 con 0x563866d9beb0 2024-04-01T01:41:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.657+0000 7fa248d85740 10 client.27872 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.657+0000 7fa248d85740 10 client.27872 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:41:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.657+0000 7fa248d85740 20 client.27872 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.657+0000 7fa248d85740 1 --2- 172.21.15.70:0/633293286 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x563866e18370 0x563866e1a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.657+0000 7fa248d85740 4 client.27872 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:41:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.657+0000 7fa248d85740 1 -- 172.21.15.70:0/633293286 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563866d056f0 con 0x563866e18370 2024-04-01T01:41:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.659+0000 7fa23effd640 1 --2- 172.21.15.70:0/633293286 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x563866e18370 0x563866e1a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.659+0000 7fa23effd640 1 --2- 172.21.15.70:0/633293286 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x563866e18370 0x563866e1a790 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e52601f12ffe9fd9 server_cookie=2747a94f065b096d in_seq=0 out_seq=0 2024-04-01T01:41:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.659+0000 7fa21effd640 10 client.27872 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:41:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.659+0000 7fa21effd640 1 -- 172.21.15.70:0/633293286 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563866d056f0 con 0x563866e18370 2024-04-01T01:41:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.660+0000 7fa21effd640 10 client.27872 handle_command_reply: tid=0 2024-04-01T01:41:35.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.660+0000 7fa248d85740 1 client.27872 shutdown 2024-04-01T01:41:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.660+0000 7fa248d85740 1 -- 172.21.15.70:0/633293286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866d9beb0 msgr2=0x563866daaa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.660+0000 7fa248d85740 1 --2- 172.21.15.70:0/633293286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866d9beb0 0x563866daaa70 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa230009730 tx=0x7fa230011310 comp rx=0 tx=0).stop 2024-04-01T01:41:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.660+0000 7fa248d85740 1 -- 172.21.15.70:0/633293286 shutdown_connections 2024-04-01T01:41:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.660+0000 7fa248d85740 1 --2- 172.21.15.70:0/633293286 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x563866e18370 0x563866e1a790 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.660+0000 7fa248d85740 1 --2- 172.21.15.70:0/633293286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563866da1a10 0x563866e162c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.660+0000 7fa248d85740 1 --2- 172.21.15.70:0/633293286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563866d9c8a0 0x563866daafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.660+0000 7fa248d85740 1 --2- 172.21.15.70:0/633293286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563866d9beb0 0x563866daaa70 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.660+0000 7fa248d85740 1 -- 172.21.15.70:0/633293286 >> 172.21.15.70:0/633293286 conn(0x563866d8ec10 msgr2=0x563866d92700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.660+0000 7fa21effd640 0 client.27872 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:41:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.660+0000 7fa248d85740 1 -- 172.21.15.70:0/633293286 shutdown_connections 2024-04-01T01:41:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.660+0000 7fa248d85740 1 -- 172.21.15.70:0/633293286 wait complete. 2024-04-01T01:41:35.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.660+0000 7fa248d85740 20 client.27872 trim_cache size 0 max 16384 2024-04-01T01:41:35.664 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:41:35.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.661+0000 7fa2461a1640 1 -- 172.21.15.70:0/3112832978 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa2280a2500 msgr2=0x7fa2280a49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.661+0000 7fa2461a1640 1 --2- 172.21.15.70:0/3112832978 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa2280a2500 0x7fa2280a49c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa238006010 tx=0x7fa238005ef0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.661+0000 7fa2461a1640 1 -- 172.21.15.70:0/3112832978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa240101750 msgr2=0x7fa240199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.661+0000 7fa2461a1640 1 --2- 172.21.15.70:0/3112832978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa240101750 0x7fa240199f20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa23400ea60 tx=0x7fa23400ef20 comp rx=0 tx=0).stop 2024-04-01T01:41:35.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.661+0000 7fa2461a1640 1 -- 172.21.15.70:0/3112832978 shutdown_connections 2024-04-01T01:41:35.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.661+0000 7fa2461a1640 1 --2- 172.21.15.70:0/3112832978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa240102110 0x7fa2401a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.661+0000 7fa2461a1640 1 --2- 172.21.15.70:0/3112832978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa240101750 0x7fa240199f20 secure :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa23400ea60 tx=0x7fa23400ef20 comp rx=0 tx=0).stop 2024-04-01T01:41:35.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.661+0000 7fa2461a1640 1 --2- 172.21.15.70:0/3112832978 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa2280a2500 0x7fa2280a49c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.661+0000 7fa2461a1640 1 --2- 172.21.15.70:0/3112832978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa240100560 0x7fa2401999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.661+0000 7fa2461a1640 1 -- 172.21.15.70:0/3112832978 >> 172.21.15.70:0/3112832978 conn(0x7fa2400fbaf0 msgr2=0x7fa240104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.662+0000 7fa2461a1640 1 -- 172.21.15.70:0/3112832978 shutdown_connections 2024-04-01T01:41:35.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.665+0000 7fa2461a1640 1 -- 172.21.15.70:0/3112832978 wait complete. 2024-04-01T01:41:35.685 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:41:35.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.857+0000 7f5efd125640 1 -- 172.21.15.70:0/2177610639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8102930 msgr2=0x7f5ef8104da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.857+0000 7f5efd125640 1 --2- 172.21.15.70:0/2177610639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8102930 0x7f5ef8104da0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0009ca0 tx=0x7f5ef002c100 comp rx=0 tx=0).stop 2024-04-01T01:41:35.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.858+0000 7f5efd125640 1 -- 172.21.15.70:0/2177610639 shutdown_connections 2024-04-01T01:41:35.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.858+0000 7f5efd125640 1 --2- 172.21.15.70:0/2177610639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ef81052e0 0x7f5ef8105760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.858+0000 7f5efd125640 1 --2- 172.21.15.70:0/2177610639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8102930 0x7f5ef8104da0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.858+0000 7f5efd125640 1 --2- 172.21.15.70:0/2177610639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ef80fffc0 0x7f5ef81023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.858+0000 7f5efd125640 1 -- 172.21.15.70:0/2177610639 >> 172.21.15.70:0/2177610639 conn(0x7f5ef80fbbe0 msgr2=0x7f5ef80fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.858+0000 7f5efd125640 1 -- 172.21.15.70:0/2177610639 shutdown_connections 2024-04-01T01:41:35.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.858+0000 7f5efd125640 1 -- 172.21.15.70:0/2177610639 wait complete. 2024-04-01T01:41:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.860+0000 7f5efd125640 1 Processor -- start 2024-04-01T01:41:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.860+0000 7f5efd125640 1 -- start start 2024-04-01T01:41:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.861+0000 7f5efd125640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ef80fffc0 0x7f5ef819df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.861+0000 7f5efd125640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8102930 0x7f5ef819e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.861+0000 7f5efd125640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ef81052e0 0x7f5ef81a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.861+0000 7f5efd125640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef8113850 con 0x7f5ef8102930 2024-04-01T01:41:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.861+0000 7f5efd125640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef81136d0 con 0x7f5ef80fffc0 2024-04-01T01:41:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.861+0000 7f5efd125640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef81139d0 con 0x7f5ef81052e0 2024-04-01T01:41:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.861+0000 7f5ef659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8102930 0x7f5ef819e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.861+0000 7f5ef659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8102930 0x7f5ef819e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34102/0 (socket says 172.21.15.70:34102) 2024-04-01T01:41:35.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.861+0000 7f5ef759e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ef81052e0 0x7f5ef81a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.861+0000 7f5ef759e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ef81052e0 0x7f5ef81a54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57862/0 (socket says 172.21.15.70:57862) 2024-04-01T01:41:35.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.862+0000 7f5ef759e640 1 -- 172.21.15.70:0/2545345047 learned_addr learned my addr 172.21.15.70:0/2545345047 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:35.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.862+0000 7f5ef759e640 1 -- 172.21.15.70:0/2545345047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ef80fffc0 msgr2=0x7f5ef819df20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:41:35.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.862+0000 7f5ef759e640 1 --2- 172.21.15.70:0/2545345047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ef80fffc0 0x7f5ef819df20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.862+0000 7f5ef759e640 1 -- 172.21.15.70:0/2545345047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8102930 msgr2=0x7f5ef819e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.862+0000 7f5ef759e640 1 --2- 172.21.15.70:0/2545345047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8102930 0x7f5ef819e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.862+0000 7f5ef759e640 1 -- 172.21.15.70:0/2545345047 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef81a5b40 con 0x7f5ef81052e0 2024-04-01T01:41:35.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.862+0000 7f5ef759e640 1 --2- 172.21.15.70:0/2545345047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ef81052e0 0x7f5ef81a54d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5eec00c970 tx=0x7f5eec00ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.863+0000 7f5edffff640 1 -- 172.21.15.70:0/2545345047 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eec007bb0 con 0x7f5ef81052e0 2024-04-01T01:41:35.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.863+0000 7f5edffff640 1 -- 172.21.15.70:0/2545345047 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5eec007d50 con 0x7f5ef81052e0 2024-04-01T01:41:35.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.863+0000 7f5efd125640 1 -- 172.21.15.70:0/2545345047 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ef81a5e30 con 0x7f5ef81052e0 2024-04-01T01:41:35.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.863+0000 7f5efd125640 1 -- 172.21.15.70:0/2545345047 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ef81a62f0 con 0x7f5ef81052e0 2024-04-01T01:41:35.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.864+0000 7f5edffff640 1 -- 172.21.15.70:0/2545345047 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eec018770 con 0x7f5ef81052e0 2024-04-01T01:41:35.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.866+0000 7f5effde7740 1 Processor -- start 2024-04-01T01:41:35.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.867+0000 7f5edffff640 1 -- 172.21.15.70:0/2545345047 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f5eec018910 con 0x7f5ef81052e0 2024-04-01T01:41:35.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.867+0000 7f5effde7740 1 -- start start 2024-04-01T01:41:35.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.867+0000 7f5effde7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5480 0x5586f6ae58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.867+0000 7f5effde7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 0x5586f6aea970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.867+0000 7f5effde7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6aeafe0 0x5586f6aed470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.867+0000 7f5effde7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5586f6a4a1b0 con 0x5586f6aeafe0 2024-04-01T01:41:35.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.867+0000 7f5effde7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5586f69f2c00 con 0x5586f6ae5e70 2024-04-01T01:41:35.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.867+0000 7f5effde7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5586f6a46250 con 0x5586f6ae5480 2024-04-01T01:41:35.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.867+0000 7f5ef659c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5480 0x5586f6ae58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.867+0000 7f5ef659c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5480 0x5586f6ae58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57866/0 (socket says 172.21.15.70:57866) 2024-04-01T01:41:35.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.867+0000 7f5ef659c640 1 -- 172.21.15.70:0/25024380 learned_addr learned my addr 172.21.15.70:0/25024380 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:35.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.868+0000 7f5ef659c640 1 --2- 172.21.15.70:0/25024380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6aeafe0 0x5586f6aed470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.868+0000 7f5edffff640 1 --2- 172.21.15.70:0/2545345047 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ee00a25a0 0x7f5ee00a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.868+0000 7f5ef659c640 1 -- 172.21.15.70:0/25024380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 msgr2=0x5586f6aea970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:41:35.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.868+0000 7f5ef659c640 1 --2- 172.21.15.70:0/25024380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 0x5586f6aea970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.868+0000 7f5ef659c640 1 -- 172.21.15.70:0/25024380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6aeafe0 msgr2=0x5586f6aed470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.868+0000 7f5ef659c640 1 --2- 172.21.15.70:0/25024380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6aeafe0 0x5586f6aed470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.868+0000 7f5ef659c640 1 -- 172.21.15.70:0/25024380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586f6a123d0 con 0x5586f6ae5480 2024-04-01T01:41:35.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.868+0000 7f5edffff640 1 -- 172.21.15.70:0/2545345047 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5eec0cec80 con 0x7f5ef81052e0 2024-04-01T01:41:35.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.868+0000 7f5ef659c640 1 --2- 172.21.15.70:0/25024380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5480 0x5586f6ae58a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0006010 tx=0x7f5ef002e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc9eb477fc3d057a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.869+0000 7f5ef759e640 1 --2- 172.21.15.70:0/2545345047 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ee00a25a0 0x7f5ee00a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.869+0000 7f5efd125640 1 -- 172.21.15.70:0/25024380 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef004a070 con 0x5586f6ae5480 2024-04-01T01:41:35.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.869+0000 7f5efd125640 1 -- 172.21.15.70:0/25024380 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ef0032950 con 0x5586f6ae5480 2024-04-01T01:41:35.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.869+0000 7f5efd125640 1 -- 172.21.15.70:0/25024380 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0039040 con 0x5586f6ae5480 2024-04-01T01:41:35.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.870+0000 7f5ef759e640 1 --2- 172.21.15.70:0/2545345047 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ee00a25a0 0x7f5ee00a4a60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5eec0985e0 tx=0x7f5eec013000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.870+0000 7f5effde7740 1 -- 172.21.15.70:0/25024380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5480 msgr2=0x5586f6ae58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.870+0000 7f5effde7740 1 --2- 172.21.15.70:0/25024380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5480 0x5586f6ae58a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0006010 tx=0x7f5ef002e040 comp rx=0 tx=0).stop 2024-04-01T01:41:35.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.870+0000 7f5effde7740 1 -- 172.21.15.70:0/25024380 shutdown_connections 2024-04-01T01:41:35.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.870+0000 7f5effde7740 1 --2- 172.21.15.70:0/25024380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6aeafe0 0x5586f6aed470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.870+0000 7f5effde7740 1 --2- 172.21.15.70:0/25024380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 0x5586f6aea970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.870+0000 7f5effde7740 1 --2- 172.21.15.70:0/25024380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5480 0x5586f6ae58a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.870+0000 7f5effde7740 1 -- 172.21.15.70:0/25024380 >> 172.21.15.70:0/25024380 conn(0x5586f6ae0ce0 msgr2=0x5586f6ae10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.870+0000 7f5effde7740 1 -- 172.21.15.70:0/25024380 shutdown_connections 2024-04-01T01:41:35.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.870+0000 7f5effde7740 1 -- 172.21.15.70:0/25024380 wait complete. 2024-04-01T01:41:35.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.871+0000 7f5effde7740 1 -- start start 2024-04-01T01:41:35.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.871+0000 7f5effde7740 1 Processor -- start 2024-04-01T01:41:35.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.871+0000 7f5effde7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6ae5480 0x5586f6b00870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.871+0000 7f5effde7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5e70 0x5586f6b00db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.871+0000 7f5effde7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6aeafe0 0x5586f6b07df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.871+0000 7f5effde7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5586f6afef90 con 0x5586f6ae5480 2024-04-01T01:41:35.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.871+0000 7f5effde7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5586f69e8470 con 0x5586f6aeafe0 2024-04-01T01:41:35.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.871+0000 7f5effde7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5586f6aff110 con 0x5586f6ae5e70 2024-04-01T01:41:35.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.871+0000 7f5effde7740 10 client.0 _ll_register_callbacks cb 0x5586f6a05cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:35.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.872+0000 7f5ef6d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5e70 0x5586f6b00db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.872+0000 7f5ef6d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5e70 0x5586f6b00db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57868/0 (socket says 172.21.15.70:57868) 2024-04-01T01:41:35.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.872+0000 7f5ef6d9d640 1 -- 172.21.15.70:0/4106381837 learned_addr learned my addr 172.21.15.70:0/4106381837 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:35.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.872+0000 7f5ef659c640 1 --2- 172.21.15.70:0/4106381837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6ae5480 0x5586f6b00870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.872+0000 7f5ef659c640 1 --2- 172.21.15.70:0/4106381837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6aeafe0 0x5586f6b07df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.872+0000 7f5ef6d9d640 1 -- 172.21.15.70:0/4106381837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6aeafe0 msgr2=0x5586f6b07df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.872+0000 7f5ef6d9d640 1 --2- 172.21.15.70:0/4106381837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6aeafe0 0x5586f6b07df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.872+0000 7f5ef6d9d640 1 -- 172.21.15.70:0/4106381837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6ae5480 msgr2=0x5586f6b00870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.872+0000 7f5ef6d9d640 1 --2- 172.21.15.70:0/4106381837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6ae5480 0x5586f6b00870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.872+0000 7f5ef6d9d640 1 -- 172.21.15.70:0/4106381837 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586f675e3f0 con 0x5586f6ae5e70 2024-04-01T01:41:35.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.873+0000 7f5ef6d9d640 1 --2- 172.21.15.70:0/4106381837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5e70 0x5586f6b00db0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5ee800bb60 tx=0x7f5ee800d6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c4b8e8777cb133 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.873+0000 7f5edaff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:41:35.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.873+0000 7f5edaff5640 1 -- 172.21.15.70:0/4106381837 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8016070 con 0x5586f6ae5e70 2024-04-01T01:41:35.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.873+0000 7f5edaff5640 1 -- 172.21.15.70:0/4106381837 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5ee8010660 con 0x5586f6ae5e70 2024-04-01T01:41:35.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.873+0000 7f5effde7740 1 -- 172.21.15.70:0/4106381837 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ef0032950 con 0x5586f6ae5e70 2024-04-01T01:41:35.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.874+0000 7f5edaff5640 1 -- 172.21.15.70:0/4106381837 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5ee8022050 con 0x5586f6ae5e70 2024-04-01T01:41:35.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.874+0000 7f5effde7740 10 client.27712 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:35.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.874+0000 7f5effde7740 1 -- 172.21.15.70:0/4106381837 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5586f68287d0 con 0x5586f6ae5e70 2024-04-01T01:41:35.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.874+0000 7f5edaff5640 1 -- 172.21.15.70:0/4106381837 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ef0032950 con 0x5586f6ae5e70 2024-04-01T01:41:35.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.874+0000 7f5edaff5640 1 -- 172.21.15.70:0/4106381837 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5ee800f680 con 0x5586f6ae5e70 2024-04-01T01:41:35.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.874+0000 7f5effde7740 10 client.27712 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:35.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.874+0000 7f5effde7740 10 client.27712 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:41:35.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.874+0000 7f5effde7740 20 client.27712 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:35.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.874+0000 7f5effde7740 1 --2- 172.21.15.70:0/4106381837 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5586f6b0c730 0x5586f6b0eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.874+0000 7f5effde7740 4 client.27712 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:35.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.874+0000 7f5effde7740 1 -- 172.21.15.70:0/4106381837 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586f6a576f0 con 0x5586f6b0c730 2024-04-01T01:41:35.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.875+0000 7f5ef659c640 1 --2- 172.21.15.70:0/4106381837 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5586f6b0c730 0x5586f6b0eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.876+0000 7f5ef659c640 1 --2- 172.21.15.70:0/4106381837 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5586f6b0c730 0x5586f6b0eb90 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1fd87c26fc1c3a97 server_cookie=cd7fb615411ac459 in_seq=0 out_seq=0 2024-04-01T01:41:35.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.877+0000 7f5edaff5640 10 client.27712 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:41:35.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.878+0000 7f5edaff5640 1 -- 172.21.15.70:0/4106381837 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5586f6a576f0 con 0x5586f6b0c730 2024-04-01T01:41:35.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.878+0000 7f5edaff5640 10 client.27712 handle_command_reply: tid=0 2024-04-01T01:41:35.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.878+0000 7f5effde7740 1 client.27712 shutdown 2024-04-01T01:41:35.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.882+0000 7f5effde7740 1 -- 172.21.15.70:0/4106381837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5e70 msgr2=0x5586f6b00db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.882+0000 7f5effde7740 1 --2- 172.21.15.70:0/4106381837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5e70 0x5586f6b00db0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5ee800bb60 tx=0x7f5ee800d6b0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.882+0000 7f5effde7740 1 -- 172.21.15.70:0/4106381837 shutdown_connections 2024-04-01T01:41:35.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.882+0000 7f5effde7740 1 --2- 172.21.15.70:0/4106381837 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5586f6b0c730 0x5586f6b0eb90 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.882+0000 7f5effde7740 1 --2- 172.21.15.70:0/4106381837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6aeafe0 0x5586f6b07df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.882+0000 7f5effde7740 1 --2- 172.21.15.70:0/4106381837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5e70 0x5586f6b00db0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.882+0000 7f5effde7740 1 --2- 172.21.15.70:0/4106381837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6ae5480 0x5586f6b00870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.882+0000 7f5effde7740 1 -- 172.21.15.70:0/4106381837 >> 172.21.15.70:0/4106381837 conn(0x5586f6ae0ce0 msgr2=0x5586f6adfe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.882+0000 7f5edaff5640 0 client.27712 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:41:35.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.886+0000 7f5effde7740 1 -- 172.21.15.70:0/4106381837 shutdown_connections 2024-04-01T01:41:35.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.886+0000 7f5effde7740 1 -- 172.21.15.70:0/4106381837 wait complete. 2024-04-01T01:41:35.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.886+0000 7f5effde7740 20 client.27712 trim_cache size 0 max 16384 2024-04-01T01:41:35.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.910+0000 7f5effde7740 1 Processor -- start 2024-04-01T01:41:35.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.910+0000 7f5effde7740 1 -- start start 2024-04-01T01:41:35.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.910+0000 7f5effde7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5480 0x5586f6adbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.910+0000 7f5effde7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 0x5586f6adc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.910+0000 7f5effde7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6aeafe0 0x5586f6b07db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.910+0000 7f5effde7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef81a8220 con 0x5586f6aeafe0 2024-04-01T01:41:35.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.910+0000 7f5effde7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef81a9d60 con 0x5586f6ae5e70 2024-04-01T01:41:35.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.910+0000 7f5effde7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef81a88f0 con 0x5586f6ae5480 2024-04-01T01:41:35.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.911+0000 7f5ef6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 0x5586f6adc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.911+0000 7f5ef6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 0x5586f6adc110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57842/0 (socket says 172.21.15.70:57842) 2024-04-01T01:41:35.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.911+0000 7f5ef6d9d640 1 -- 172.21.15.70:0/1959648121 learned_addr learned my addr 172.21.15.70:0/1959648121 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:35.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.911+0000 7f5ef659c640 1 --2- 172.21.15.70:0/1959648121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6aeafe0 0x5586f6b07db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.911+0000 7f5ef659c640 1 -- 172.21.15.70:0/1959648121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5480 msgr2=0x5586f6adbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.911+0000 7f5ef659c640 1 --2- 172.21.15.70:0/1959648121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5480 0x5586f6adbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.911+0000 7f5ef659c640 1 -- 172.21.15.70:0/1959648121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 msgr2=0x5586f6adc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.911+0000 7f5ef659c640 1 --2- 172.21.15.70:0/1959648121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 0x5586f6adc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.911+0000 7f5ef659c640 1 -- 172.21.15.70:0/1959648121 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586f6ae1a50 con 0x5586f6aeafe0 2024-04-01T01:41:35.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.913+0000 7f5ef659c640 1 --2- 172.21.15.70:0/1959648121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6aeafe0 0x5586f6b07db0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0009bf0 tx=0x7f5ef0000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f1c81e314b65e6c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.913+0000 7f5edcff9640 1 -- 172.21.15.70:0/1959648121 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef004a040 con 0x5586f6aeafe0 2024-04-01T01:41:35.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.914+0000 7f5edcff9640 1 -- 172.21.15.70:0/1959648121 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ef00328d0 con 0x5586f6aeafe0 2024-04-01T01:41:35.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.914+0000 7f5edcff9640 1 -- 172.21.15.70:0/1959648121 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0039040 con 0x5586f6aeafe0 2024-04-01T01:41:35.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.914+0000 7f5effde7740 1 -- 172.21.15.70:0/1959648121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6aeafe0 msgr2=0x5586f6b07db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.915+0000 7f5effde7740 1 --2- 172.21.15.70:0/1959648121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6aeafe0 0x5586f6b07db0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0009bf0 tx=0x7f5ef0000f40 comp rx=0 tx=0).stop 2024-04-01T01:41:35.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.915+0000 7f5effde7740 1 -- 172.21.15.70:0/1959648121 shutdown_connections 2024-04-01T01:41:35.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.915+0000 7f5effde7740 1 --2- 172.21.15.70:0/1959648121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6aeafe0 0x5586f6b07db0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.915+0000 7f5effde7740 1 --2- 172.21.15.70:0/1959648121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 0x5586f6adc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.915+0000 7f5effde7740 1 --2- 172.21.15.70:0/1959648121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6ae5480 0x5586f6adbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.915+0000 7f5effde7740 1 -- 172.21.15.70:0/1959648121 >> 172.21.15.70:0/1959648121 conn(0x5586f6ae0ce0 msgr2=0x5586f6b00870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.916+0000 7f5effde7740 1 -- 172.21.15.70:0/1959648121 shutdown_connections 2024-04-01T01:41:35.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.917+0000 7f5effde7740 1 -- 172.21.15.70:0/1959648121 wait complete. 2024-04-01T01:41:35.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.917+0000 7f5effde7740 1 -- start start 2024-04-01T01:41:35.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.917+0000 7f5effde7740 1 Processor -- start 2024-04-01T01:41:35.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.917+0000 7f5effde7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6ae5480 0x5586f6b04bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.917+0000 7f5effde7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 0x5586f6b050f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.918+0000 7f5effde7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6aeafe0 0x5586f6b70ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.918+0000 7f5effde7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef81a9690 con 0x5586f6ae5480 2024-04-01T01:41:35.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.918+0000 7f5effde7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef81a8fc0 con 0x5586f6ae5e70 2024-04-01T01:41:35.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.918+0000 7f5effde7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef810e9b0 con 0x5586f6aeafe0 2024-04-01T01:41:35.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.918+0000 7f5effde7740 10 client.0 _ll_register_callbacks cb 0x7f5ee800b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:35.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.918+0000 7f5ef6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 0x5586f6b050f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.918+0000 7f5ef6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 0x5586f6b050f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57844/0 (socket says 172.21.15.70:57844) 2024-04-01T01:41:35.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.918+0000 7f5ef6d9d640 1 -- 172.21.15.70:0/2745819733 learned_addr learned my addr 172.21.15.70:0/2745819733 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:35.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.918+0000 7f5ef659c640 1 --2- 172.21.15.70:0/2745819733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6ae5480 0x5586f6b04bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.918+0000 7f5ef659c640 1 --2- 172.21.15.70:0/2745819733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6aeafe0 0x5586f6b70ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.918+0000 7f5ef6d9d640 1 -- 172.21.15.70:0/2745819733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6aeafe0 msgr2=0x5586f6b70ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.918+0000 7f5ef6d9d640 1 --2- 172.21.15.70:0/2745819733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6aeafe0 0x5586f6b70ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.918+0000 7f5ef6d9d640 1 -- 172.21.15.70:0/2745819733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6ae5480 msgr2=0x5586f6b04bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:35.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.918+0000 7f5ef6d9d640 1 --2- 172.21.15.70:0/2745819733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6ae5480 0x5586f6b04bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.918+0000 7f5ef6d9d640 1 -- 172.21.15.70:0/2745819733 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586f6b05760 con 0x5586f6ae5e70 2024-04-01T01:41:35.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.919+0000 7f5ef6d9d640 1 --2- 172.21.15.70:0/2745819733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 0x5586f6b050f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5ee800b830 tx=0x7f5ee800b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a47b9aaf4be4f97e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:35.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.919+0000 7f5edaff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:35.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.920+0000 7f5edaff5640 1 -- 172.21.15.70:0/2745819733 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8016070 con 0x5586f6ae5e70 2024-04-01T01:41:35.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.920+0000 7f5edaff5640 1 -- 172.21.15.70:0/2745819733 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5ee800f5c0 con 0x5586f6ae5e70 2024-04-01T01:41:35.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.920+0000 7f5effde7740 1 -- 172.21.15.70:0/2745819733 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ef00328d0 con 0x5586f6ae5e70 2024-04-01T01:41:35.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.920+0000 7f5edaff5640 1 -- 172.21.15.70:0/2745819733 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5ee8034050 con 0x5586f6ae5e70 2024-04-01T01:41:35.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.920+0000 7f5effde7740 10 client.27881 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:35.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.920+0000 7f5effde7740 1 -- 172.21.15.70:0/2745819733 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5586f6b72180 con 0x5586f6ae5e70 2024-04-01T01:41:35.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.921+0000 7f5edaff5640 1 -- 172.21.15.70:0/2745819733 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ef00328d0 con 0x5586f6ae5e70 2024-04-01T01:41:35.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.921+0000 7f5edaff5640 1 -- 172.21.15.70:0/2745819733 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5ee8003870 con 0x5586f6ae5e70 2024-04-01T01:41:35.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.921+0000 7f5effde7740 10 client.27881 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:35.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.922+0000 7f5effde7740 10 client.27881 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:41:35.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.922+0000 7f5effde7740 20 client.27881 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:35.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.922+0000 7f5effde7740 1 --2- 172.21.15.70:0/2745819733 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5586f6b72b00 0x5586f6b74f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:35.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.922+0000 7f5effde7740 4 client.27881 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:41:35.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.922+0000 7f5effde7740 1 -- 172.21.15.70:0/2745819733 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5586f6a51a40 con 0x5586f6b72b00 2024-04-01T01:41:35.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.922+0000 7f5ef659c640 1 --2- 172.21.15.70:0/2745819733 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5586f6b72b00 0x5586f6b74f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:35.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.927+0000 7f5ef659c640 1 --2- 172.21.15.70:0/2745819733 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5586f6b72b00 0x5586f6b74f20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c07088393640e80 server_cookie=47777eb2ce614700 in_seq=0 out_seq=0 2024-04-01T01:41:35.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.927+0000 7f5edaff5640 10 client.27881 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:41:35.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.928+0000 7f5edaff5640 1 -- 172.21.15.70:0/2745819733 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5586f6a51a40 con 0x5586f6b72b00 2024-04-01T01:41:35.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.928+0000 7f5edaff5640 10 client.27881 handle_command_reply: tid=0 2024-04-01T01:41:35.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.928+0000 7f5effde7740 1 client.27881 shutdown 2024-04-01T01:41:35.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.928+0000 7f5effde7740 1 -- 172.21.15.70:0/2745819733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 msgr2=0x5586f6b050f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.928+0000 7f5effde7740 1 --2- 172.21.15.70:0/2745819733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 0x5586f6b050f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5ee800b830 tx=0x7f5ee800b390 comp rx=0 tx=0).stop 2024-04-01T01:41:35.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.929+0000 7f5effde7740 1 -- 172.21.15.70:0/2745819733 shutdown_connections 2024-04-01T01:41:35.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.929+0000 7f5effde7740 1 --2- 172.21.15.70:0/2745819733 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5586f6b72b00 0x5586f6b74f20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.929+0000 7f5effde7740 1 --2- 172.21.15.70:0/2745819733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5586f6aeafe0 0x5586f6b70ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.929+0000 7f5effde7740 1 --2- 172.21.15.70:0/2745819733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586f6ae5e70 0x5586f6b050f0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.929+0000 7f5effde7740 1 --2- 172.21.15.70:0/2745819733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5586f6ae5480 0x5586f6b04bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.929+0000 7f5effde7740 1 -- 172.21.15.70:0/2745819733 >> 172.21.15.70:0/2745819733 conn(0x5586f6ae0ce0 msgr2=0x5586f6ae7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.929+0000 7f5edaff5640 0 client.27881 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:41:35.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.929+0000 7f5effde7740 1 -- 172.21.15.70:0/2745819733 shutdown_connections 2024-04-01T01:41:35.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.929+0000 7f5effde7740 1 -- 172.21.15.70:0/2745819733 wait complete. 2024-04-01T01:41:35.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.930+0000 7f5effde7740 20 client.27881 trim_cache size 0 max 16384 2024-04-01T01:41:35.930 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:41:35.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.931+0000 7f5eddffb640 1 -- 172.21.15.70:0/2545345047 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ee00a25a0 msgr2=0x7f5ee00a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.931+0000 7f5eddffb640 1 --2- 172.21.15.70:0/2545345047 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ee00a25a0 0x7f5ee00a4a60 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5eec0985e0 tx=0x7f5eec013000 comp rx=0 tx=0).stop 2024-04-01T01:41:35.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.931+0000 7f5eddffb640 1 -- 172.21.15.70:0/2545345047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ef81052e0 msgr2=0x7f5ef81a54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:35.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.931+0000 7f5eddffb640 1 --2- 172.21.15.70:0/2545345047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ef81052e0 0x7f5ef81a54d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5eec00c970 tx=0x7f5eec00ce30 comp rx=0 tx=0).stop 2024-04-01T01:41:35.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.932+0000 7f5eddffb640 1 -- 172.21.15.70:0/2545345047 shutdown_connections 2024-04-01T01:41:35.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.932+0000 7f5eddffb640 1 --2- 172.21.15.70:0/2545345047 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5ee00a25a0 0x7f5ee00a4a60 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.932+0000 7f5eddffb640 1 --2- 172.21.15.70:0/2545345047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ef81052e0 0x7f5ef81a54d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.932+0000 7f5eddffb640 1 --2- 172.21.15.70:0/2545345047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ef8102930 0x7f5ef819e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.932+0000 7f5eddffb640 1 --2- 172.21.15.70:0/2545345047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ef80fffc0 0x7f5ef819df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:35.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.932+0000 7f5eddffb640 1 -- 172.21.15.70:0/2545345047 >> 172.21.15.70:0/2545345047 conn(0x7f5ef80fbbe0 msgr2=0x7f5ef8100940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:35.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.932+0000 7f5eddffb640 1 -- 172.21.15.70:0/2545345047 shutdown_connections 2024-04-01T01:41:35.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:35.932+0000 7f5eddffb640 1 -- 172.21.15.70:0/2545345047 wait complete. 2024-04-01T01:41:35.950 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:35 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1453606326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:41:35.951 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:41:36.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.135+0000 7f747da02640 1 -- 172.21.15.70:0/3284592525 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7478107430 msgr2=0x7f7478109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.135+0000 7f747da02640 1 --2- 172.21.15.70:0/3284592525 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7478107430 0x7f7478109880 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7470009900 tx=0x7f747002bc30 comp rx=0 tx=0).stop 2024-04-01T01:41:36.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.135+0000 7f747da02640 1 -- 172.21.15.70:0/3284592525 shutdown_connections 2024-04-01T01:41:36.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.135+0000 7f747da02640 1 --2- 172.21.15.70:0/3284592525 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7478107430 0x7f7478109880 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.135+0000 7f747da02640 1 --2- 172.21.15.70:0/3284592525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7478104b00 0x7f7478106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.135+0000 7f747da02640 1 --2- 172.21.15.70:0/3284592525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74781021d0 0x7f74781045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.135+0000 7f747da02640 1 -- 172.21.15.70:0/3284592525 >> 172.21.15.70:0/3284592525 conn(0x7f74780fbbe0 msgr2=0x7f74780fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:36.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.136+0000 7f747da02640 1 -- 172.21.15.70:0/3284592525 shutdown_connections 2024-04-01T01:41:36.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.136+0000 7f747da02640 1 -- 172.21.15.70:0/3284592525 wait complete. 2024-04-01T01:41:36.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.136+0000 7f747da02640 1 Processor -- start 2024-04-01T01:41:36.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.136+0000 7f747da02640 1 -- start start 2024-04-01T01:41:36.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.137+0000 7f747da02640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74781021d0 0x7f747819e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.137+0000 7f747da02640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7478104b00 0x7f747819e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.137+0000 7f747da02640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7478107430 0x7f74781a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.137+0000 7f747da02640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f747810b130 con 0x7f7478107430 2024-04-01T01:41:36.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.137+0000 7f747da02640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f747810afb0 con 0x7f7478104b00 2024-04-01T01:41:36.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.137+0000 7f747da02640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f747810b2b0 con 0x7f74781021d0 2024-04-01T01:41:36.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.137+0000 7f746ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7478104b00 0x7f747819e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.137+0000 7f74777fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7478107430 0x7f74781a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.137+0000 7f74777fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7478107430 0x7f74781a55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34156/0 (socket says 172.21.15.70:34156) 2024-04-01T01:41:36.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.137+0000 7f74777fe640 1 -- 172.21.15.70:0/187217965 learned_addr learned my addr 172.21.15.70:0/187217965 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:36.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.137+0000 7f7476ffd640 1 --2- 172.21.15.70:0/187217965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74781021d0 0x7f747819e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.138+0000 7f74777fe640 1 -- 172.21.15.70:0/187217965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74781021d0 msgr2=0x7f747819e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.138+0000 7f74777fe640 1 --2- 172.21.15.70:0/187217965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74781021d0 0x7f747819e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.138+0000 7f74777fe640 1 -- 172.21.15.70:0/187217965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7478104b00 msgr2=0x7f747819e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.138+0000 7f74777fe640 1 --2- 172.21.15.70:0/187217965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7478104b00 0x7f747819e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.138+0000 7f74777fe640 1 -- 172.21.15.70:0/187217965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74781a5c30 con 0x7f7478107430 2024-04-01T01:41:36.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.138+0000 7f7476ffd640 1 --2- 172.21.15.70:0/187217965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74781021d0 0x7f747819e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:41:36.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.138+0000 7f746ffff640 1 --2- 172.21.15.70:0/187217965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7478104b00 0x7f747819e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:41:36.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.138+0000 7f74777fe640 1 --2- 172.21.15.70:0/187217965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7478107430 0x7f74781a55c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7470036040 tx=0x7f747002cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:36.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.138+0000 7f747ca00640 1 -- 172.21.15.70:0/187217965 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7470043020 con 0x7f7478107430 2024-04-01T01:41:36.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.139+0000 7f747da02640 1 -- 172.21.15.70:0/187217965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74781a5ec0 con 0x7f7478107430 2024-04-01T01:41:36.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.139+0000 7f747da02640 1 -- 172.21.15.70:0/187217965 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74781a6400 con 0x7f7478107430 2024-04-01T01:41:36.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.139+0000 7f747ca00640 1 -- 172.21.15.70:0/187217965 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7470004400 con 0x7f7478107430 2024-04-01T01:41:36.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.140+0000 7f747ca00640 1 -- 172.21.15.70:0/187217965 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7470003b80 con 0x7f7478107430 2024-04-01T01:41:36.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.142+0000 7f74805e5740 1 Processor -- start 2024-04-01T01:41:36.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.144+0000 7f74805e5740 1 -- start start 2024-04-01T01:41:36.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.144+0000 7f74805e5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21653490 0x55bb216538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.144+0000 7f74805e5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21653e80 0x55bb21658980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.144+0000 7f74805e5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21658ff0 0x55bb2165b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.144+0000 7f74805e5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bb215b81b0 con 0x55bb21653490 2024-04-01T01:41:36.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.144+0000 7f74805e5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55bb21560c00 con 0x55bb21658ff0 2024-04-01T01:41:36.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.144+0000 7f74805e5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bb215b4250 con 0x55bb21653e80 2024-04-01T01:41:36.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.144+0000 7f747ca00640 1 -- 172.21.15.70:0/187217965 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f7470046020 con 0x7f7478107430 2024-04-01T01:41:36.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.144+0000 7f746ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21653490 0x55bb216538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.144+0000 7f746ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21653490 0x55bb216538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34170/0 (socket says 172.21.15.70:34170) 2024-04-01T01:41:36.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.144+0000 7f746ffff640 1 -- 172.21.15.70:0/334646931 learned_addr learned my addr 172.21.15.70:0/334646931 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:36.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.144+0000 7f746ffff640 1 --2- 172.21.15.70:0/334646931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21658ff0 0x55bb2165b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.144+0000 7f7476ffd640 1 --2- 172.21.15.70:0/334646931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21653e80 0x55bb21658980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.145+0000 7f746ffff640 1 -- 172.21.15.70:0/334646931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21653e80 msgr2=0x55bb21658980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:36.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.145+0000 7f746ffff640 1 --2- 172.21.15.70:0/334646931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21653e80 0x55bb21658980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.145+0000 7f746ffff640 1 -- 172.21.15.70:0/334646931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21658ff0 msgr2=0x55bb2165b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:36.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.145+0000 7f746ffff640 1 --2- 172.21.15.70:0/334646931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21658ff0 0x55bb2165b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.145+0000 7f746ffff640 1 -- 172.21.15.70:0/334646931 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb215803d0 con 0x55bb21653490 2024-04-01T01:41:36.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.145+0000 7f746ffff640 1 --2- 172.21.15.70:0/334646931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21653490 0x55bb216538b0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f74680049d0 tx=0x7f7468009a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1f153434d8cac8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:36.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.145+0000 7f747ca00640 1 --2- 172.21.15.70:0/187217965 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f74640a2650 0x7f74640a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.145+0000 7f7476ffd640 1 --2- 172.21.15.70:0/187217965 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f74640a2650 0x7f74640a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.145+0000 7f747ca00640 1 -- 172.21.15.70:0/187217965 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f74700f0d20 con 0x7f7478107430 2024-04-01T01:41:36.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.146+0000 7f746dffb640 1 -- 172.21.15.70:0/334646931 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7468012d50 con 0x55bb21653490 2024-04-01T01:41:36.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.146+0000 7f746dffb640 1 -- 172.21.15.70:0/334646931 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7468012110 con 0x55bb21653490 2024-04-01T01:41:36.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.146+0000 7f746dffb640 1 -- 172.21.15.70:0/334646931 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7468010b30 con 0x55bb21653490 2024-04-01T01:41:36.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.147+0000 7f7476ffd640 1 --2- 172.21.15.70:0/187217965 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f74640a2650 0x7f74640a4b10 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x55bb21654280 tx=0x7f7460009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:36.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.147+0000 7f74805e5740 1 -- 172.21.15.70:0/334646931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21653490 msgr2=0x55bb216538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.147+0000 7f74805e5740 1 --2- 172.21.15.70:0/334646931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21653490 0x55bb216538b0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f74680049d0 tx=0x7f7468009a80 comp rx=0 tx=0).stop 2024-04-01T01:41:36.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.147+0000 7f74805e5740 1 -- 172.21.15.70:0/334646931 shutdown_connections 2024-04-01T01:41:36.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.147+0000 7f74805e5740 1 --2- 172.21.15.70:0/334646931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21658ff0 0x55bb2165b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.147+0000 7f74805e5740 1 --2- 172.21.15.70:0/334646931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21653e80 0x55bb21658980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.147+0000 7f74805e5740 1 --2- 172.21.15.70:0/334646931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21653490 0x55bb216538b0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.147+0000 7f74805e5740 1 -- 172.21.15.70:0/334646931 >> 172.21.15.70:0/334646931 conn(0x55bb2164ea00 msgr2=0x55bb2164ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:36.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.147+0000 7f74805e5740 1 -- 172.21.15.70:0/334646931 shutdown_connections 2024-04-01T01:41:36.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.147+0000 7f74805e5740 1 -- 172.21.15.70:0/334646931 wait complete. 2024-04-01T01:41:36.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.148+0000 7f74805e5740 1 -- start start 2024-04-01T01:41:36.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.148+0000 7f74805e5740 1 Processor -- start 2024-04-01T01:41:36.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.148+0000 7f74805e5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21653e80 0x55bb2165db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.149+0000 7f74805e5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21658ff0 0x55bb2165e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.149+0000 7f74805e5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21663100 0x55bb216654f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.149+0000 7f74805e5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bb2165bf90 con 0x55bb21653e80 2024-04-01T01:41:36.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.149+0000 7f74805e5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55bb21556470 con 0x55bb21658ff0 2024-04-01T01:41:36.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.149+0000 7f74805e5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bb2165c110 con 0x55bb21663100 2024-04-01T01:41:36.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.149+0000 7f74805e5740 10 client.0 _ll_register_callbacks cb 0x55bb21573cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:36.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.149+0000 7f746ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21658ff0 0x55bb2165e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.149+0000 7f7476ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21653e80 0x55bb2165db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.149+0000 7f746ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21658ff0 0x55bb2165e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57888/0 (socket says 172.21.15.70:57888) 2024-04-01T01:41:36.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.149+0000 7f746ffff640 1 -- 172.21.15.70:0/637727204 learned_addr learned my addr 172.21.15.70:0/637727204 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:36.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.149+0000 7f74777fe640 1 --2- 172.21.15.70:0/637727204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21663100 0x55bb216654f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.149+0000 7f746ffff640 1 -- 172.21.15.70:0/637727204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21663100 msgr2=0x55bb216654f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:36.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.149+0000 7f746ffff640 1 --2- 172.21.15.70:0/637727204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21663100 0x55bb216654f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.149+0000 7f746ffff640 1 -- 172.21.15.70:0/637727204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21653e80 msgr2=0x55bb2165db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:36.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.149+0000 7f746ffff640 1 --2- 172.21.15.70:0/637727204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21653e80 0x55bb2165db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.149+0000 7f746ffff640 1 -- 172.21.15.70:0/637727204 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb212cc3f0 con 0x55bb21658ff0 2024-04-01T01:41:36.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.150+0000 7f746ffff640 1 --2- 172.21.15.70:0/637727204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21658ff0 0x55bb2165e0d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7468006010 tx=0x7f7468012960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f66e7555c1fcbbc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:36.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.150+0000 7f7456ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:36.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.150+0000 7f7456ffd640 1 -- 172.21.15.70:0/637727204 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7468005450 con 0x55bb21658ff0 2024-04-01T01:41:36.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.150+0000 7f7456ffd640 1 -- 172.21.15.70:0/637727204 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f74680055f0 con 0x55bb21658ff0 2024-04-01T01:41:36.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.150+0000 7f74805e5740 1 -- 172.21.15.70:0/637727204 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7468012110 con 0x55bb21658ff0 2024-04-01T01:41:36.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.151+0000 7f7456ffd640 1 -- 172.21.15.70:0/637727204 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7468030050 con 0x55bb21658ff0 2024-04-01T01:41:36.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.151+0000 7f74805e5740 10 client.27893 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:36.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.151+0000 7f74805e5740 1 -- 172.21.15.70:0/637727204 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55bb213967d0 con 0x55bb21658ff0 2024-04-01T01:41:36.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.151+0000 7f7456ffd640 1 -- 172.21.15.70:0/637727204 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7468023030 con 0x55bb21658ff0 2024-04-01T01:41:36.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.151+0000 7f7456ffd640 1 -- 172.21.15.70:0/637727204 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f74680108d0 con 0x55bb21658ff0 2024-04-01T01:41:36.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.151+0000 7f74805e5740 10 client.27893 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:36.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.151+0000 7f74805e5740 10 client.27893 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:41:36.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.151+0000 7f74805e5740 20 client.27893 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:36.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.151+0000 7f74805e5740 1 --2- 172.21.15.70:0/637727204 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bb21669b90 0x55bb2166bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.151+0000 7f74805e5740 4 client.27893 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:36.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.151+0000 7f74805e5740 1 -- 172.21.15.70:0/637727204 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7468012110 con 0x55bb21669b90 2024-04-01T01:41:36.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.152+0000 7f7476ffd640 1 --2- 172.21.15.70:0/637727204 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bb21669b90 0x55bb2166bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.153+0000 7f7476ffd640 1 --2- 172.21.15.70:0/637727204 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bb21669b90 0x55bb2166bff0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b5f4483899965d4 server_cookie=9c1e52912506be3e in_seq=0 out_seq=0 2024-04-01T01:41:36.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.153+0000 7f7456ffd640 10 client.27893 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:41:36.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.154+0000 7f7456ffd640 1 -- 172.21.15.70:0/637727204 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f7468012110 con 0x55bb21669b90 2024-04-01T01:41:36.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.154+0000 7f7456ffd640 10 client.27893 handle_command_reply: tid=0 2024-04-01T01:41:36.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.154+0000 7f74805e5740 1 client.27893 shutdown 2024-04-01T01:41:36.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.154+0000 7f74805e5740 1 -- 172.21.15.70:0/637727204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21658ff0 msgr2=0x55bb2165e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.154+0000 7f74805e5740 1 --2- 172.21.15.70:0/637727204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21658ff0 0x55bb2165e0d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7468006010 tx=0x7f7468012960 comp rx=0 tx=0).stop 2024-04-01T01:41:36.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.155+0000 7f74805e5740 1 -- 172.21.15.70:0/637727204 shutdown_connections 2024-04-01T01:41:36.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.155+0000 7f74805e5740 1 --2- 172.21.15.70:0/637727204 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bb21669b90 0x55bb2166bff0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.155+0000 7f74805e5740 1 --2- 172.21.15.70:0/637727204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21663100 0x55bb216654f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.155+0000 7f74805e5740 1 --2- 172.21.15.70:0/637727204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21658ff0 0x55bb2165e0d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.155+0000 7f74805e5740 1 --2- 172.21.15.70:0/637727204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21653e80 0x55bb2165db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.155+0000 7f74805e5740 1 -- 172.21.15.70:0/637727204 >> 172.21.15.70:0/637727204 conn(0x55bb2164ea00 msgr2=0x55bb2164dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:36.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.155+0000 7f7456ffd640 0 client.27893 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:41:36.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.155+0000 7f74805e5740 1 -- 172.21.15.70:0/637727204 shutdown_connections 2024-04-01T01:41:36.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.155+0000 7f74805e5740 1 -- 172.21.15.70:0/637727204 wait complete. 2024-04-01T01:41:36.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.155+0000 7f74805e5740 20 client.27893 trim_cache size 0 max 16384 2024-04-01T01:41:36.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.181+0000 7f74805e5740 1 Processor -- start 2024-04-01T01:41:36.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.181+0000 7f74805e5740 1 -- start start 2024-04-01T01:41:36.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.181+0000 7f74805e5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21653e80 0x55bb21654980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.181+0000 7f74805e5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21658ff0 0x55bb2165db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.181+0000 7f74805e5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21663100 0x55bb2165e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.181+0000 7f74805e5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f74781b0ab0 con 0x55bb21658ff0 2024-04-01T01:41:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.181+0000 7f74805e5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f74781b25f0 con 0x55bb21663100 2024-04-01T01:41:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.181+0000 7f74805e5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f74781b1180 con 0x55bb21653e80 2024-04-01T01:41:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.182+0000 7f746ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21658ff0 0x55bb2165db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.182+0000 7f746ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21658ff0 0x55bb2165db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34190/0 (socket says 172.21.15.70:34190) 2024-04-01T01:41:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.182+0000 7f746ffff640 1 -- 172.21.15.70:0/4021126999 learned_addr learned my addr 172.21.15.70:0/4021126999 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.182+0000 7f74777fe640 1 --2- 172.21.15.70:0/4021126999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21663100 0x55bb2165e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.182+0000 7f7476ffd640 1 --2- 172.21.15.70:0/4021126999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21653e80 0x55bb21654980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.182+0000 7f746ffff640 1 -- 172.21.15.70:0/4021126999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21653e80 msgr2=0x55bb21654980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:36.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.182+0000 7f746ffff640 1 --2- 172.21.15.70:0/4021126999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21653e80 0x55bb21654980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.182+0000 7f746ffff640 1 -- 172.21.15.70:0/4021126999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21663100 msgr2=0x55bb2165e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:36.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.182+0000 7f746ffff640 1 --2- 172.21.15.70:0/4021126999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21663100 0x55bb2165e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.182+0000 7f746ffff640 1 -- 172.21.15.70:0/4021126999 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb2164a460 con 0x55bb21658ff0 2024-04-01T01:41:36.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.183+0000 7f746ffff640 1 --2- 172.21.15.70:0/4021126999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21658ff0 0x55bb2165db90 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f746802a760 tx=0x7f7468004e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2dd53f815f5860f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:36.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.183+0000 7f746d7fa640 1 -- 172.21.15.70:0/4021126999 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7468012110 con 0x55bb21658ff0 2024-04-01T01:41:36.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.183+0000 7f746d7fa640 1 -- 172.21.15.70:0/4021126999 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7468023040 con 0x55bb21658ff0 2024-04-01T01:41:36.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.183+0000 7f746d7fa640 1 -- 172.21.15.70:0/4021126999 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7468010420 con 0x55bb21658ff0 2024-04-01T01:41:36.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.184+0000 7f74805e5740 1 -- 172.21.15.70:0/4021126999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21658ff0 msgr2=0x55bb2165db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.184+0000 7f74805e5740 1 --2- 172.21.15.70:0/4021126999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21658ff0 0x55bb2165db90 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f746802a760 tx=0x7f7468004e70 comp rx=0 tx=0).stop 2024-04-01T01:41:36.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.184+0000 7f74805e5740 1 -- 172.21.15.70:0/4021126999 shutdown_connections 2024-04-01T01:41:36.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.184+0000 7f74805e5740 1 --2- 172.21.15.70:0/4021126999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb21663100 0x55bb2165e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.184+0000 7f74805e5740 1 --2- 172.21.15.70:0/4021126999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21658ff0 0x55bb2165db90 secure :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f746802a760 tx=0x7f7468004e70 comp rx=0 tx=0).stop 2024-04-01T01:41:36.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.184+0000 7f74805e5740 1 --2- 172.21.15.70:0/4021126999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21653e80 0x55bb21654980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.184+0000 7f74805e5740 1 -- 172.21.15.70:0/4021126999 >> 172.21.15.70:0/4021126999 conn(0x55bb2164ea00 msgr2=0x55bb2164ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:36.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.184+0000 7f74805e5740 1 -- 172.21.15.70:0/4021126999 shutdown_connections 2024-04-01T01:41:36.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.185+0000 7f74805e5740 1 -- 172.21.15.70:0/4021126999 wait complete. 2024-04-01T01:41:36.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.185+0000 7f74805e5740 1 -- start start 2024-04-01T01:41:36.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.185+0000 7f74805e5740 1 Processor -- start 2024-04-01T01:41:36.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.189+0000 7f74805e5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21653e80 0x55bb2167a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.189+0000 7f74805e5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21663100 0x55bb2167a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.189+0000 7f74805e5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb2167b9e0 0x55bb2167be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.189+0000 7f74805e5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f74781b1f20 con 0x55bb21653e80 2024-04-01T01:41:36.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.189+0000 7f74805e5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f74781b1850 con 0x55bb2167b9e0 2024-04-01T01:41:36.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.189+0000 7f74805e5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f74781aea90 con 0x55bb21663100 2024-04-01T01:41:36.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.189+0000 7f74805e5740 10 client.0 _ll_register_callbacks cb 0x7f7468012ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:36.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.189+0000 7f74777fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb2167b9e0 0x55bb2167be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.189+0000 7f74777fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb2167b9e0 0x55bb2167be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57912/0 (socket says 172.21.15.70:57912) 2024-04-01T01:41:36.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.189+0000 7f74777fe640 1 -- 172.21.15.70:0/88704845 learned_addr learned my addr 172.21.15.70:0/88704845 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:36.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.191+0000 7f74777fe640 1 -- 172.21.15.70:0/88704845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21663100 msgr2=0x55bb2167a9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:41:36.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.191+0000 7f74777fe640 1 --2- 172.21.15.70:0/88704845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21663100 0x55bb2167a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.191+0000 7f74777fe640 1 -- 172.21.15.70:0/88704845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21653e80 msgr2=0x55bb2167a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:36.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.191+0000 7f74777fe640 1 --2- 172.21.15.70:0/88704845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21653e80 0x55bb2167a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.191+0000 7f74777fe640 1 -- 172.21.15.70:0/88704845 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb2167c3a0 con 0x55bb2167b9e0 2024-04-01T01:41:36.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.191+0000 7f74777fe640 1 --2- 172.21.15.70:0/88704845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb2167b9e0 0x55bb2167be60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f74700bd340 tx=0x7f74700ee2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e153b6cef8e51cb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:36.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.191+0000 7f7456ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:36.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.192+0000 7f7456ffd640 1 -- 172.21.15.70:0/88704845 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74700d5070 con 0x55bb2167b9e0 2024-04-01T01:41:36.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.192+0000 7f7456ffd640 1 -- 172.21.15.70:0/88704845 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f74700ee500 con 0x55bb2167b9e0 2024-04-01T01:41:36.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.192+0000 7f74805e5740 1 -- 172.21.15.70:0/88704845 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7468023040 con 0x55bb2167b9e0 2024-04-01T01:41:36.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.192+0000 7f7456ffd640 1 -- 172.21.15.70:0/88704845 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f74700c1500 con 0x55bb2167b9e0 2024-04-01T01:41:36.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.194+0000 7f74805e5740 10 client.27899 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:36.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.194+0000 7f74805e5740 1 -- 172.21.15.70:0/88704845 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55bb216e0a30 con 0x55bb2167b9e0 2024-04-01T01:41:36.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.195+0000 7f7456ffd640 1 -- 172.21.15.70:0/88704845 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7468023040 con 0x55bb2167b9e0 2024-04-01T01:41:36.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.195+0000 7f7456ffd640 1 -- 172.21.15.70:0/88704845 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f74700c5930 con 0x55bb2167b9e0 2024-04-01T01:41:36.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.195+0000 7f74805e5740 10 client.27899 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:36.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.195+0000 7f74805e5740 10 client.27899 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:41:36.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.195+0000 7f74805e5740 20 client.27899 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:36.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.195+0000 7f74805e5740 1 --2- 172.21.15.70:0/88704845 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bb216e1360 0x55bb216e37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.195+0000 7f74805e5740 4 client.27899 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:41:36.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.195+0000 7f74805e5740 1 -- 172.21.15.70:0/88704845 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bb215c56f0 con 0x55bb216e1360 2024-04-01T01:41:36.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.198+0000 7f7476ffd640 1 --2- 172.21.15.70:0/88704845 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bb216e1360 0x55bb216e37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.201+0000 7f7476ffd640 1 --2- 172.21.15.70:0/88704845 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bb216e1360 0x55bb216e37a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=505b62017d9271f5 server_cookie=13a9e0689cf0ad87 in_seq=0 out_seq=0 2024-04-01T01:41:36.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.201+0000 7f7456ffd640 10 client.27899 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:41:36.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.201+0000 7f7456ffd640 1 -- 172.21.15.70:0/88704845 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bb215c56f0 con 0x55bb216e1360 2024-04-01T01:41:36.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.201+0000 7f7456ffd640 10 client.27899 handle_command_reply: tid=0 2024-04-01T01:41:36.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.203+0000 7f74805e5740 1 client.27899 shutdown 2024-04-01T01:41:36.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.206+0000 7f74805e5740 1 -- 172.21.15.70:0/88704845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb2167b9e0 msgr2=0x55bb2167be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.206+0000 7f74805e5740 1 --2- 172.21.15.70:0/88704845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb2167b9e0 0x55bb2167be60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f74700bd340 tx=0x7f74700ee2c0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.208+0000 7f74805e5740 1 -- 172.21.15.70:0/88704845 shutdown_connections 2024-04-01T01:41:36.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.208+0000 7f74805e5740 1 --2- 172.21.15.70:0/88704845 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bb216e1360 0x55bb216e37a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.208+0000 7f74805e5740 1 --2- 172.21.15.70:0/88704845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb2167b9e0 0x55bb2167be60 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.208+0000 7f74805e5740 1 --2- 172.21.15.70:0/88704845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb21663100 0x55bb2167a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.208+0000 7f74805e5740 1 --2- 172.21.15.70:0/88704845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb21653e80 0x55bb2167a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.208+0000 7f74805e5740 1 -- 172.21.15.70:0/88704845 >> 172.21.15.70:0/88704845 conn(0x55bb2164ea00 msgr2=0x55bb2166bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:36.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.208+0000 7f7456ffd640 0 client.27899 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:41:36.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.208+0000 7f74805e5740 1 -- 172.21.15.70:0/88704845 shutdown_connections 2024-04-01T01:41:36.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.209+0000 7f74805e5740 1 -- 172.21.15.70:0/88704845 wait complete. 2024-04-01T01:41:36.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.209+0000 7f74805e5740 20 client.27899 trim_cache size 0 max 16384 2024-04-01T01:41:36.209 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:41:36.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.211+0000 7f747da02640 1 -- 172.21.15.70:0/187217965 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f74640a2650 msgr2=0x7f74640a4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.211+0000 7f747da02640 1 --2- 172.21.15.70:0/187217965 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f74640a2650 0x7f74640a4b10 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x55bb21654280 tx=0x7f7460009110 comp rx=0 tx=0).stop 2024-04-01T01:41:36.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.211+0000 7f747da02640 1 -- 172.21.15.70:0/187217965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7478107430 msgr2=0x7f74781a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.211+0000 7f747da02640 1 --2- 172.21.15.70:0/187217965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7478107430 0x7f74781a55c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7470036040 tx=0x7f747002cdd0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.211+0000 7f747da02640 1 -- 172.21.15.70:0/187217965 shutdown_connections 2024-04-01T01:41:36.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.211+0000 7f747da02640 1 --2- 172.21.15.70:0/187217965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7478107430 0x7f74781a55c0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.211+0000 7f747da02640 1 --2- 172.21.15.70:0/187217965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7478104b00 0x7f747819e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.211+0000 7f747da02640 1 --2- 172.21.15.70:0/187217965 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f74640a2650 0x7f74640a4b10 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.211+0000 7f747da02640 1 --2- 172.21.15.70:0/187217965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74781021d0 0x7f747819e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.211+0000 7f747da02640 1 -- 172.21.15.70:0/187217965 >> 172.21.15.70:0/187217965 conn(0x7f74780fbbe0 msgr2=0x7f7478102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:36.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.212+0000 7f747da02640 1 -- 172.21.15.70:0/187217965 shutdown_connections 2024-04-01T01:41:36.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.212+0000 7f747da02640 1 -- 172.21.15.70:0/187217965 wait complete. 2024-04-01T01:41:36.234 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:41:36.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:35 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1453606326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:41:36.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:35 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1453606326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:41:36.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.433+0000 7ff338822640 1 -- 172.21.15.70:0/1583644544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff32c0a7f80 msgr2=0x7ff32c0a8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.433+0000 7ff338822640 1 --2- 172.21.15.70:0/1583644544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff32c0a7f80 0x7ff32c0a8400 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff328009900 tx=0x7ff32802bc30 comp rx=0 tx=0).stop 2024-04-01T01:41:36.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.433+0000 7ff338822640 1 -- 172.21.15.70:0/1583644544 shutdown_connections 2024-04-01T01:41:36.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.434+0000 7ff338822640 1 --2- 172.21.15.70:0/1583644544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff32c0a8940 0x7ff32c0af1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.434+0000 7ff338822640 1 --2- 172.21.15.70:0/1583644544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff32c0a7f80 0x7ff32c0a8400 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.434+0000 7ff338822640 1 --2- 172.21.15.70:0/1583644544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff32c0a6d90 0x7ff32c0a71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.434+0000 7ff338822640 1 -- 172.21.15.70:0/1583644544 >> 172.21.15.70:0/1583644544 conn(0x7ff32c0a2300 msgr2=0x7ff32c0a4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:36.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.434+0000 7ff338822640 1 -- 172.21.15.70:0/1583644544 shutdown_connections 2024-04-01T01:41:36.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.434+0000 7ff338822640 1 -- 172.21.15.70:0/1583644544 wait complete. 2024-04-01T01:41:36.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.435+0000 7ff338822640 1 Processor -- start 2024-04-01T01:41:36.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.436+0000 7ff338822640 1 -- start start 2024-04-01T01:41:36.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.436+0000 7ff338822640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff32c0a6d90 0x7ff32c1435b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.436+0000 7ff338822640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff32c0a7f80 0x7ff32c143af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.436+0000 7ff338822640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff32c0a8940 0x7ff32c14ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.436+0000 7ff338822640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c019b60 con 0x7ff32c0a8940 2024-04-01T01:41:36.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.436+0000 7ff338822640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c0199e0 con 0x7ff32c0a7f80 2024-04-01T01:41:36.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.436+0000 7ff338822640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c019ce0 con 0x7ff32c0a6d90 2024-04-01T01:41:36.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.436+0000 7ff33359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff32c0a8940 0x7ff32c14ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.436+0000 7ff332d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff32c0a6d90 0x7ff32c1435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.436+0000 7ff33359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff32c0a8940 0x7ff32c14ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34218/0 (socket says 172.21.15.70:34218) 2024-04-01T01:41:36.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.436+0000 7ff33359e640 1 -- 172.21.15.70:0/763373010 learned_addr learned my addr 172.21.15.70:0/763373010 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:36.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.437+0000 7ff33359e640 1 -- 172.21.15.70:0/763373010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff32c0a6d90 msgr2=0x7ff32c1435b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.437+0000 7ff33359e640 1 --2- 172.21.15.70:0/763373010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff32c0a6d90 0x7ff32c1435b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.437+0000 7ff33359e640 1 -- 172.21.15.70:0/763373010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff32c0a7f80 msgr2=0x7ff32c143af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.437+0000 7ff33359e640 1 --2- 172.21.15.70:0/763373010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff32c0a7f80 0x7ff32c143af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.437+0000 7ff33359e640 1 -- 172.21.15.70:0/763373010 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32c14b1d0 con 0x7ff32c0a8940 2024-04-01T01:41:36.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.437+0000 7ff332d9d640 1 --2- 172.21.15.70:0/763373010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff32c0a6d90 0x7ff32c1435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:41:36.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.437+0000 7ff33359e640 1 --2- 172.21.15.70:0/763373010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff32c0a8940 0x7ff32c14ab60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7ff324009fd0 tx=0x7ff32400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:36.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.438+0000 7ff31bfff640 1 -- 172.21.15.70:0/763373010 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324011830 con 0x7ff32c0a8940 2024-04-01T01:41:36.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.438+0000 7ff31bfff640 1 -- 172.21.15.70:0/763373010 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff324016070 con 0x7ff32c0a8940 2024-04-01T01:41:36.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.438+0000 7ff31bfff640 1 -- 172.21.15.70:0/763373010 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324010e50 con 0x7ff32c0a8940 2024-04-01T01:41:36.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.438+0000 7ff338822640 1 -- 172.21.15.70:0/763373010 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff32c14b4c0 con 0x7ff32c0a8940 2024-04-01T01:41:36.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.438+0000 7ff338822640 1 -- 172.21.15.70:0/763373010 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff32c14b9e0 con 0x7ff32c0a8940 2024-04-01T01:41:36.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.442+0000 7ff31bfff640 1 -- 172.21.15.70:0/763373010 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7ff3240119d0 con 0x7ff32c0a8940 2024-04-01T01:41:36.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.442+0000 7ff33bc06740 1 Processor -- start 2024-04-01T01:41:36.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.443+0000 7ff31bfff640 1 --2- 172.21.15.70:0/763373010 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff31c0a2810 0x7ff31c0a4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.443+0000 7ff31bfff640 1 -- 172.21.15.70:0/763373010 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff3240ce110 con 0x7ff32c0a8940 2024-04-01T01:41:36.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.443+0000 7ff33259c640 1 --2- 172.21.15.70:0/763373010 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff31c0a2810 0x7ff31c0a4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.444+0000 7ff33bc06740 1 -- start start 2024-04-01T01:41:36.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.444+0000 7ff33bc06740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c69826a0 0x5567c6982ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.444+0000 7ff33bc06740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6983090 0x5567c6987b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.444+0000 7ff33bc06740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6988200 0x5567c698a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.444+0000 7ff33bc06740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5567c68e71b0 con 0x5567c6983090 2024-04-01T01:41:36.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.444+0000 7ff33bc06740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5567c688fc00 con 0x5567c6988200 2024-04-01T01:41:36.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.444+0000 7ff33bc06740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5567c68e3250 con 0x5567c69826a0 2024-04-01T01:41:36.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.444+0000 7ff33259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6983090 0x5567c6987b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.444+0000 7ff33259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6983090 0x5567c6987b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34234/0 (socket says 172.21.15.70:34234) 2024-04-01T01:41:36.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.444+0000 7ff33259c640 1 -- 172.21.15.70:0/1337415980 learned_addr learned my addr 172.21.15.70:0/1337415980 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:36.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.444+0000 7ff33359e640 1 --2- 172.21.15.70:0/1337415980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6988200 0x5567c698a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.445+0000 7ff332d9d640 1 --2- 172.21.15.70:0/1337415980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c69826a0 0x5567c6982ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.445+0000 7ff33359e640 1 -- 172.21.15.70:0/1337415980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c69826a0 msgr2=0x5567c6982ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:36.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.445+0000 7ff33359e640 1 --2- 172.21.15.70:0/1337415980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c69826a0 0x5567c6982ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.445+0000 7ff33359e640 1 -- 172.21.15.70:0/1337415980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6983090 msgr2=0x5567c6987b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:36.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.445+0000 7ff33359e640 1 --2- 172.21.15.70:0/1337415980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6983090 0x5567c6987b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.445+0000 7ff33359e640 1 -- 172.21.15.70:0/1337415980 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567c68af3d0 con 0x5567c6988200 2024-04-01T01:41:36.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.445+0000 7ff33259c640 1 --2- 172.21.15.70:0/763373010 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff31c0a2810 0x7ff31c0a4cd0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff328005ec0 tx=0x7ff328005560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:36.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.445+0000 7ff33359e640 1 --2- 172.21.15.70:0/1337415980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6988200 0x5567c698a690 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ff324006fd0 tx=0x7ff324011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a6c4d108ebe1506 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:36.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.446+0000 7ff3197fa640 1 -- 172.21.15.70:0/1337415980 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3240b3070 con 0x5567c6988200 2024-04-01T01:41:36.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.446+0000 7ff3197fa640 1 -- 172.21.15.70:0/1337415980 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff3240ba050 con 0x5567c6988200 2024-04-01T01:41:36.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.446+0000 7ff3197fa640 1 -- 172.21.15.70:0/1337415980 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324011cc0 con 0x5567c6988200 2024-04-01T01:41:36.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.447+0000 7ff33bc06740 1 -- 172.21.15.70:0/1337415980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6988200 msgr2=0x5567c698a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.447+0000 7ff33bc06740 1 --2- 172.21.15.70:0/1337415980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6988200 0x5567c698a690 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ff324006fd0 tx=0x7ff324011000 comp rx=0 tx=0).stop 2024-04-01T01:41:36.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.447+0000 7ff33bc06740 1 -- 172.21.15.70:0/1337415980 shutdown_connections 2024-04-01T01:41:36.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.447+0000 7ff33bc06740 1 --2- 172.21.15.70:0/1337415980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6988200 0x5567c698a690 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.447+0000 7ff33bc06740 1 --2- 172.21.15.70:0/1337415980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6983090 0x5567c6987b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.447+0000 7ff33bc06740 1 --2- 172.21.15.70:0/1337415980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c69826a0 0x5567c6982ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.447+0000 7ff33bc06740 1 -- 172.21.15.70:0/1337415980 >> 172.21.15.70:0/1337415980 conn(0x5567c697dc10 msgr2=0x5567c697e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:36.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.447+0000 7ff33bc06740 1 -- 172.21.15.70:0/1337415980 shutdown_connections 2024-04-01T01:41:36.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.447+0000 7ff33bc06740 1 -- 172.21.15.70:0/1337415980 wait complete. 2024-04-01T01:41:36.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.448+0000 7ff33bc06740 1 -- start start 2024-04-01T01:41:36.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.448+0000 7ff33bc06740 1 Processor -- start 2024-04-01T01:41:36.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.448+0000 7ff33bc06740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c69826a0 0x5567c699dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.448+0000 7ff33bc06740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6983090 0x5567c699e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.448+0000 7ff33bc06740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6988200 0x5567c69a5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.448+0000 7ff33bc06740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5567c699c1c0 con 0x5567c6983090 2024-04-01T01:41:36.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.448+0000 7ff33bc06740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5567c6885470 con 0x5567c6988200 2024-04-01T01:41:36.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.448+0000 7ff33bc06740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5567c699c340 con 0x5567c69826a0 2024-04-01T01:41:36.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.448+0000 7ff33bc06740 10 client.0 _ll_register_callbacks cb 0x5567c68a2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:36.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.448+0000 7ff33259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6983090 0x5567c699e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.449+0000 7ff33259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6983090 0x5567c699e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34236/0 (socket says 172.21.15.70:34236) 2024-04-01T01:41:36.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.449+0000 7ff332d9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c69826a0 0x5567c699dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.449+0000 7ff33259c640 1 -- 172.21.15.70:0/1977044096 learned_addr learned my addr 172.21.15.70:0/1977044096 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:36.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.449+0000 7ff33359e640 1 --2- 172.21.15.70:0/1977044096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6988200 0x5567c69a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.449+0000 7ff33259c640 1 -- 172.21.15.70:0/1977044096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c69826a0 msgr2=0x5567c699dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:36.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.449+0000 7ff33259c640 1 --2- 172.21.15.70:0/1977044096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c69826a0 0x5567c699dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.449+0000 7ff33259c640 1 -- 172.21.15.70:0/1977044096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6988200 msgr2=0x5567c69a5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:36.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.449+0000 7ff33259c640 1 --2- 172.21.15.70:0/1977044096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6988200 0x5567c69a5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.449+0000 7ff33259c640 1 -- 172.21.15.70:0/1977044096 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567c65fb3f0 con 0x5567c6983090 2024-04-01T01:41:36.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.449+0000 7ff33259c640 1 --2- 172.21.15.70:0/1977044096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6983090 0x5567c699e100 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7ff32802c170 tx=0x7ff328005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd56931b48510bfa server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:36.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.449+0000 7ff316ff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:41:36.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.450+0000 7ff316ff5640 1 -- 172.21.15.70:0/1977044096 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff328052060 con 0x5567c6983090 2024-04-01T01:41:36.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.450+0000 7ff33bc06740 1 -- 172.21.15.70:0/1977044096 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3240ba050 con 0x5567c6983090 2024-04-01T01:41:36.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.450+0000 7ff316ff5640 1 -- 172.21.15.70:0/1977044096 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff328005020 con 0x5567c6983090 2024-04-01T01:41:36.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.451+0000 7ff316ff5640 1 -- 172.21.15.70:0/1977044096 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff328059050 con 0x5567c6983090 2024-04-01T01:41:36.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.451+0000 7ff33bc06740 10 client.17946 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:36.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.451+0000 7ff33bc06740 1 -- 172.21.15.70:0/1977044096 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5567c66c57d0 con 0x5567c6983090 2024-04-01T01:41:36.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.451+0000 7ff316ff5640 1 -- 172.21.15.70:0/1977044096 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff3240ba050 con 0x5567c6983090 2024-04-01T01:41:36.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.452+0000 7ff316ff5640 1 -- 172.21.15.70:0/1977044096 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff3280039d0 con 0x5567c6983090 2024-04-01T01:41:36.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.452+0000 7ff33bc06740 10 client.17946 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:36.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.452+0000 7ff33bc06740 10 client.17946 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:36.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.452+0000 7ff33bc06740 20 client.17946 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:36.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.452+0000 7ff33bc06740 1 --2- 172.21.15.70:0/1977044096 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5567c69a9bc0 0x5567c69ac040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.452+0000 7ff33bc06740 4 client.17946 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:36.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.452+0000 7ff33bc06740 1 -- 172.21.15.70:0/1977044096 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567c68f46f0 con 0x5567c69a9bc0 2024-04-01T01:41:36.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.453+0000 7ff332d9d640 1 --2- 172.21.15.70:0/1977044096 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5567c69a9bc0 0x5567c69ac040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.454+0000 7ff332d9d640 1 --2- 172.21.15.70:0/1977044096 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5567c69a9bc0 0x5567c69ac040 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac086dac0a91755f server_cookie=7b7cba94858a36b in_seq=0 out_seq=0 2024-04-01T01:41:36.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.454+0000 7ff316ff5640 10 client.17946 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:36.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.455+0000 7ff316ff5640 1 -- 172.21.15.70:0/1977044096 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5567c68f46f0 con 0x5567c69a9bc0 2024-04-01T01:41:36.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.455+0000 7ff316ff5640 10 client.17946 handle_command_reply: tid=0 2024-04-01T01:41:36.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.455+0000 7ff33bc06740 1 client.17946 shutdown 2024-04-01T01:41:36.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.455+0000 7ff33bc06740 1 -- 172.21.15.70:0/1977044096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6983090 msgr2=0x5567c699e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.455+0000 7ff33bc06740 1 --2- 172.21.15.70:0/1977044096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6983090 0x5567c699e100 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7ff32802c170 tx=0x7ff328005fd0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.458+0000 7ff33bc06740 1 -- 172.21.15.70:0/1977044096 shutdown_connections 2024-04-01T01:41:36.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.458+0000 7ff33bc06740 1 --2- 172.21.15.70:0/1977044096 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5567c69a9bc0 0x5567c69ac040 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.458+0000 7ff33bc06740 1 --2- 172.21.15.70:0/1977044096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6988200 0x5567c69a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.458+0000 7ff33bc06740 1 --2- 172.21.15.70:0/1977044096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6983090 0x5567c699e100 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.458+0000 7ff33bc06740 1 --2- 172.21.15.70:0/1977044096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c69826a0 0x5567c699dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.458+0000 7ff33bc06740 1 -- 172.21.15.70:0/1977044096 >> 172.21.15.70:0/1977044096 conn(0x5567c697dc10 msgr2=0x5567c697cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:36.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.458+0000 7ff316ff5640 0 client.17946 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:36.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.458+0000 7ff33bc06740 1 -- 172.21.15.70:0/1977044096 shutdown_connections 2024-04-01T01:41:36.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.459+0000 7ff33bc06740 1 -- 172.21.15.70:0/1977044096 wait complete. 2024-04-01T01:41:36.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.459+0000 7ff33bc06740 20 client.17946 trim_cache size 0 max 16384 2024-04-01T01:41:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.492+0000 7ff33bc06740 1 Processor -- start 2024-04-01T01:41:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.493+0000 7ff33bc06740 1 -- start start 2024-04-01T01:41:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.493+0000 7ff33bc06740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c69826a0 0x5567c699d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.493+0000 7ff33bc06740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c6983090 0x5567c69a5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.493+0000 7ff33bc06740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6988200 0x5567c697c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.493+0000 7ff332d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c69826a0 0x5567c699d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.493+0000 7ff332d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c69826a0 0x5567c699d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57970/0 (socket says 172.21.15.70:57970) 2024-04-01T01:41:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.493+0000 7ff33259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c6983090 0x5567c69a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.493+0000 7ff33259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c6983090 0x5567c69a5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57944/0 (socket says 172.21.15.70:57944) 2024-04-01T01:41:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.493+0000 7ff33359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6988200 0x5567c697c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.493+0000 7ff33359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6988200 0x5567c697c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34248/0 (socket says 172.21.15.70:34248) 2024-04-01T01:41:36.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.493+0000 7ff332d9d640 1 -- 172.21.15.70:0/1231353579 learned_addr learned my addr 172.21.15.70:0/1231353579 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:36.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.493+0000 7ff33bc06740 1 -- 172.21.15.70:0/1231353579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c14d660 con 0x5567c6988200 2024-04-01T01:41:36.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.493+0000 7ff33bc06740 1 -- 172.21.15.70:0/1231353579 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c14f1a0 con 0x5567c69826a0 2024-04-01T01:41:36.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.493+0000 7ff33bc06740 1 -- 172.21.15.70:0/1231353579 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c14dd30 con 0x5567c6983090 2024-04-01T01:41:36.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.494+0000 7ff33259c640 1 -- 172.21.15.70:0/1231353579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c69826a0 msgr2=0x5567c699d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:36.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.494+0000 7ff33259c640 1 --2- 172.21.15.70:0/1231353579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c69826a0 0x5567c699d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.494+0000 7ff33259c640 1 -- 172.21.15.70:0/1231353579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6988200 msgr2=0x5567c697c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:36.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.494+0000 7ff33259c640 1 --2- 172.21.15.70:0/1231353579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6988200 0x5567c697c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.494+0000 7ff33259c640 1 -- 172.21.15.70:0/1231353579 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567c6983470 con 0x5567c6983090 2024-04-01T01:41:36.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.494+0000 7ff33259c640 1 --2- 172.21.15.70:0/1231353579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c6983090 0x5567c69a5100 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff328036040 tx=0x7ff328004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=162668f30ecf079d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:36.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.495+0000 7ff318ff9640 1 -- 172.21.15.70:0/1231353579 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff328052070 con 0x5567c6983090 2024-04-01T01:41:36.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.495+0000 7ff318ff9640 1 -- 172.21.15.70:0/1231353579 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff328049040 con 0x5567c6983090 2024-04-01T01:41:36.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.495+0000 7ff318ff9640 1 -- 172.21.15.70:0/1231353579 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff328005220 con 0x5567c6983090 2024-04-01T01:41:36.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.496+0000 7ff33bc06740 1 -- 172.21.15.70:0/1231353579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c6983090 msgr2=0x5567c69a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.496+0000 7ff33bc06740 1 --2- 172.21.15.70:0/1231353579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c6983090 0x5567c69a5100 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff328036040 tx=0x7ff328004610 comp rx=0 tx=0).stop 2024-04-01T01:41:36.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.496+0000 7ff33bc06740 1 -- 172.21.15.70:0/1231353579 shutdown_connections 2024-04-01T01:41:36.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.496+0000 7ff33bc06740 1 --2- 172.21.15.70:0/1231353579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c6988200 0x5567c697c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.496+0000 7ff33bc06740 1 --2- 172.21.15.70:0/1231353579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c6983090 0x5567c69a5100 secure :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff328036040 tx=0x7ff328004610 comp rx=0 tx=0).stop 2024-04-01T01:41:36.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.496+0000 7ff33bc06740 1 --2- 172.21.15.70:0/1231353579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c69826a0 0x5567c699d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.496+0000 7ff33bc06740 1 -- 172.21.15.70:0/1231353579 >> 172.21.15.70:0/1231353579 conn(0x5567c697dc10 msgr2=0x5567c699ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:36.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.498+0000 7ff33bc06740 1 -- 172.21.15.70:0/1231353579 shutdown_connections 2024-04-01T01:41:36.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.498+0000 7ff33bc06740 1 -- 172.21.15.70:0/1231353579 wait complete. 2024-04-01T01:41:36.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.499+0000 7ff33bc06740 1 -- start start 2024-04-01T01:41:36.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.499+0000 7ff33bc06740 1 Processor -- start 2024-04-01T01:41:36.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.499+0000 7ff33bc06740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c69826a0 0x5567c69a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.499+0000 7ff33bc06740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c6988200 0x5567c69a2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.499+0000 7ff33bc06740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6a0bf60 0x5567c6a0e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.499+0000 7ff33bc06740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c14ead0 con 0x5567c69826a0 2024-04-01T01:41:36.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.499+0000 7ff33bc06740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c14e400 con 0x5567c6a0bf60 2024-04-01T01:41:36.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.499+0000 7ff33bc06740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c0bdf20 con 0x5567c6988200 2024-04-01T01:41:36.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.500+0000 7ff33bc06740 10 client.0 _ll_register_callbacks cb 0x7ff328005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:36.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.500+0000 7ff33259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c6988200 0x5567c69a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.500+0000 7ff33259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c6988200 0x5567c69a2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57954/0 (socket says 172.21.15.70:57954) 2024-04-01T01:41:36.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.500+0000 7ff33259c640 1 -- 172.21.15.70:0/874573626 learned_addr learned my addr 172.21.15.70:0/874573626 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:36.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.500+0000 7ff332d9d640 1 --2- 172.21.15.70:0/874573626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c69826a0 0x5567c69a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.500+0000 7ff33359e640 1 --2- 172.21.15.70:0/874573626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6a0bf60 0x5567c6a0e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.500+0000 7ff332d9d640 1 -- 172.21.15.70:0/874573626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c6988200 msgr2=0x5567c69a2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:36.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.500+0000 7ff332d9d640 1 --2- 172.21.15.70:0/874573626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c6988200 0x5567c69a2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.500+0000 7ff332d9d640 1 -- 172.21.15.70:0/874573626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6a0bf60 msgr2=0x5567c6a0e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:36.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.500+0000 7ff332d9d640 1 --2- 172.21.15.70:0/874573626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6a0bf60 0x5567c6a0e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.500+0000 7ff332d9d640 1 -- 172.21.15.70:0/874573626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567c69a29b0 con 0x5567c69826a0 2024-04-01T01:41:36.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.500+0000 7ff33359e640 1 --2- 172.21.15.70:0/874573626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6a0bf60 0x5567c6a0e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:41:36.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.501+0000 7ff332d9d640 1 --2- 172.21.15.70:0/874573626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c69826a0 0x5567c69a1e30 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff334050db0 tx=0x7ff334071740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453b3904d14d1669 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:36.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.501+0000 7ff316ff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:41:36.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.501+0000 7ff316ff5640 1 -- 172.21.15.70:0/874573626 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff334084020 con 0x5567c69826a0 2024-04-01T01:41:36.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.501+0000 7ff316ff5640 1 -- 172.21.15.70:0/874573626 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff334075c30 con 0x5567c69826a0 2024-04-01T01:41:36.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.501+0000 7ff33bc06740 1 -- 172.21.15.70:0/874573626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff328049040 con 0x5567c69826a0 2024-04-01T01:41:36.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.502+0000 7ff316ff5640 1 -- 172.21.15.70:0/874573626 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff334075e30 con 0x5567c69826a0 2024-04-01T01:41:36.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.502+0000 7ff33bc06740 10 client.17952 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:36.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.502+0000 7ff33bc06740 1 -- 172.21.15.70:0/874573626 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5567c6a0f9b0 con 0x5567c69826a0 2024-04-01T01:41:36.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.502+0000 7ff316ff5640 1 -- 172.21.15.70:0/874573626 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff3340807c0 con 0x5567c69826a0 2024-04-01T01:41:36.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.502+0000 7ff316ff5640 1 -- 172.21.15.70:0/874573626 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff334069830 con 0x5567c69826a0 2024-04-01T01:41:36.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.503+0000 7ff33bc06740 10 client.17952 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:36.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.503+0000 7ff33bc06740 10 client.17952 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:36.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.503+0000 7ff33bc06740 20 client.17952 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:36.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.503+0000 7ff33bc06740 1 --2- 172.21.15.70:0/874573626 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5567c6a10360 0x5567c6a12780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:36.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.503+0000 7ff33bc06740 4 client.17952 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:41:36.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.503+0000 7ff33bc06740 1 -- 172.21.15.70:0/874573626 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7ff328049040 con 0x5567c6a10360 2024-04-01T01:41:36.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.503+0000 7ff33259c640 1 --2- 172.21.15.70:0/874573626 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5567c6a10360 0x5567c6a12780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:36.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.504+0000 7ff33259c640 1 --2- 172.21.15.70:0/874573626 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5567c6a10360 0x5567c6a12780 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fefefd9031803c9 server_cookie=33a0f09c47a65aff in_seq=0 out_seq=0 2024-04-01T01:41:36.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.504+0000 7ff316ff5640 10 client.17952 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:36.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.504+0000 7ff316ff5640 1 -- 172.21.15.70:0/874573626 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7ff328049040 con 0x5567c6a10360 2024-04-01T01:41:36.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.504+0000 7ff316ff5640 10 client.17952 handle_command_reply: tid=0 2024-04-01T01:41:36.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.504+0000 7ff33bc06740 1 client.17952 shutdown 2024-04-01T01:41:36.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.504+0000 7ff33bc06740 1 -- 172.21.15.70:0/874573626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c69826a0 msgr2=0x5567c69a1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.504+0000 7ff33bc06740 1 --2- 172.21.15.70:0/874573626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c69826a0 0x5567c69a1e30 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff334050db0 tx=0x7ff334071740 comp rx=0 tx=0).stop 2024-04-01T01:41:36.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.506+0000 7ff33bc06740 1 -- 172.21.15.70:0/874573626 shutdown_connections 2024-04-01T01:41:36.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.506+0000 7ff33bc06740 1 --2- 172.21.15.70:0/874573626 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5567c6a10360 0x5567c6a12780 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.506+0000 7ff33bc06740 1 --2- 172.21.15.70:0/874573626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5567c6a0bf60 0x5567c6a0e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.506+0000 7ff33bc06740 1 --2- 172.21.15.70:0/874573626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c6988200 0x5567c69a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.506+0000 7ff33bc06740 1 --2- 172.21.15.70:0/874573626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567c69826a0 0x5567c69a1e30 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.506+0000 7ff33bc06740 1 -- 172.21.15.70:0/874573626 >> 172.21.15.70:0/874573626 conn(0x5567c697dc10 msgr2=0x5567c6980e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:36.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.506+0000 7ff316ff5640 0 client.17952 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:36.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.507+0000 7ff33bc06740 1 -- 172.21.15.70:0/874573626 shutdown_connections 2024-04-01T01:41:36.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.507+0000 7ff33bc06740 1 -- 172.21.15.70:0/874573626 wait complete. 2024-04-01T01:41:36.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.507+0000 7ff33bc06740 20 client.17952 trim_cache size 0 max 16384 2024-04-01T01:41:36.507 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:41:36.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.509+0000 7ff338822640 1 -- 172.21.15.70:0/763373010 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff31c0a2810 msgr2=0x7ff31c0a4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.509+0000 7ff338822640 1 --2- 172.21.15.70:0/763373010 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff31c0a2810 0x7ff31c0a4cd0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff328005ec0 tx=0x7ff328005560 comp rx=0 tx=0).stop 2024-04-01T01:41:36.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.509+0000 7ff338822640 1 -- 172.21.15.70:0/763373010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff32c0a8940 msgr2=0x7ff32c14ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:36.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.509+0000 7ff338822640 1 --2- 172.21.15.70:0/763373010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff32c0a8940 0x7ff32c14ab60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7ff324009fd0 tx=0x7ff32400b8e0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.509+0000 7ff338822640 1 -- 172.21.15.70:0/763373010 shutdown_connections 2024-04-01T01:41:36.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.509+0000 7ff338822640 1 --2- 172.21.15.70:0/763373010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff32c0a8940 0x7ff32c14ab60 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.509+0000 7ff338822640 1 --2- 172.21.15.70:0/763373010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff32c0a7f80 0x7ff32c143af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.509+0000 7ff338822640 1 --2- 172.21.15.70:0/763373010 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff31c0a2810 0x7ff31c0a4cd0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.509+0000 7ff338822640 1 --2- 172.21.15.70:0/763373010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff32c0a6d90 0x7ff32c1435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:36.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.509+0000 7ff338822640 1 -- 172.21.15.70:0/763373010 >> 172.21.15.70:0/763373010 conn(0x7ff32c0a2300 msgr2=0x7ff32c0a3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:36.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.510+0000 7ff338822640 1 -- 172.21.15.70:0/763373010 shutdown_connections 2024-04-01T01:41:36.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:36.510+0000 7ff338822640 1 -- 172.21.15.70:0/763373010 wait complete. 2024-04-01T01:41:36.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 410.0 s, 820.3 tps, lat 123.449 ms stddev 125.093 2024-04-01T01:41:37.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:36 smithi189 ceph-mon[35753]: pgmap v603: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.0 MiB/s wr, 907 op/s 2024-04-01T01:41:37.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:36 smithi070 ceph-mon[32179]: pgmap v603: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.0 MiB/s wr, 907 op/s 2024-04-01T01:41:37.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:36 smithi081 ceph-mon[35142]: pgmap v603: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 8.0 MiB/s wr, 907 op/s 2024-04-01T01:41:37.529 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:41:37.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:41:37.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:41:37.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.709+0000 7fe28bfff640 1 -- 172.21.15.70:0/3207272591 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28402cc10 con 0x7fe28c0a66f0 2024-04-01T01:41:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.710+0000 7fe29a042640 1 -- 172.21.15.70:0/3207272591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe28c0a66f0 msgr2=0x7fe28c0aafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.710+0000 7fe29a042640 1 --2- 172.21.15.70:0/3207272591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe28c0a66f0 0x7fe28c0aafb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe284009900 tx=0x7fe28402bc60 comp rx=0 tx=0).stop 2024-04-01T01:41:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.710+0000 7fe29a042640 1 -- 172.21.15.70:0/3207272591 shutdown_connections 2024-04-01T01:41:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.710+0000 7fe29a042640 1 --2- 172.21.15.70:0/3207272591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe28c0a66f0 0x7fe28c0aafb0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.710+0000 7fe29a042640 1 --2- 172.21.15.70:0/3207272591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe28c0a5d50 0x7fe28c0a61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.710+0000 7fe29a042640 1 --2- 172.21.15.70:0/3207272591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe28c0a4c20 0x7fe28c0a5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.710+0000 7fe29a042640 1 -- 172.21.15.70:0/3207272591 >> 172.21.15.70:0/3207272591 conn(0x7fe28c0a00b0 msgr2=0x7fe28c0a2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.710+0000 7fe29a042640 1 -- 172.21.15.70:0/3207272591 shutdown_connections 2024-04-01T01:41:37.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.710+0000 7fe29a042640 1 -- 172.21.15.70:0/3207272591 wait complete. 2024-04-01T01:41:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.711+0000 7fe29a042640 1 Processor -- start 2024-04-01T01:41:37.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.711+0000 7fe29a042640 1 -- start start 2024-04-01T01:41:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.711+0000 7fe29a042640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe28c0a4c20 0x7fe28c143700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.711+0000 7fe29a042640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe28c0a5d50 0x7fe28c143c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.711+0000 7fe29a042640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe28c0a66f0 0x7fe28c14ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.711+0000 7fe29a042640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c0b8fd0 con 0x7fe28c0a5d50 2024-04-01T01:41:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.711+0000 7fe29a042640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c0b8e50 con 0x7fe28c0a66f0 2024-04-01T01:41:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.711+0000 7fe29a042640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c0b9150 con 0x7fe28c0a4c20 2024-04-01T01:41:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.712+0000 7fe29883f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe28c0a5d50 0x7fe28c143c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.712+0000 7fe299841640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe28c0a66f0 0x7fe28c14ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.712+0000 7fe299841640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe28c0a66f0 0x7fe28c14ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48472/0 (socket says 172.21.15.70:48472) 2024-04-01T01:41:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.712+0000 7fe299841640 1 -- 172.21.15.70:0/334522537 learned_addr learned my addr 172.21.15.70:0/334522537 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.712+0000 7fe29883f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe28c0a5d50 0x7fe28c143c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60708/0 (socket says 172.21.15.70:60708) 2024-04-01T01:41:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.712+0000 7fe299040640 1 --2- 172.21.15.70:0/334522537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe28c0a4c20 0x7fe28c143700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:37.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.712+0000 7fe299040640 1 -- 172.21.15.70:0/334522537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe28c0a66f0 msgr2=0x7fe28c14ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.712+0000 7fe299040640 1 --2- 172.21.15.70:0/334522537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe28c0a66f0 0x7fe28c14ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.712+0000 7fe299040640 1 -- 172.21.15.70:0/334522537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe28c0a5d50 msgr2=0x7fe28c143c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.712+0000 7fe299040640 1 --2- 172.21.15.70:0/334522537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe28c0a5d50 0x7fe28c143c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.712+0000 7fe299040640 1 -- 172.21.15.70:0/334522537 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28c14b2d0 con 0x7fe28c0a4c20 2024-04-01T01:41:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.712+0000 7fe29883f640 1 --2- 172.21.15.70:0/334522537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe28c0a5d50 0x7fe28c143c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:41:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.712+0000 7fe299040640 1 --2- 172.21.15.70:0/334522537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe28c0a4c20 0x7fe28c143700 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe294000b60 tx=0x7fe29406d430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.713+0000 7fe28a7fc640 1 -- 172.21.15.70:0/334522537 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe294079030 con 0x7fe28c0a4c20 2024-04-01T01:41:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.713+0000 7fe29a042640 1 -- 172.21.15.70:0/334522537 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28c14b5c0 con 0x7fe28c0a4c20 2024-04-01T01:41:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.713+0000 7fe29a042640 1 -- 172.21.15.70:0/334522537 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe28c14bb10 con 0x7fe28c0a4c20 2024-04-01T01:41:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.713+0000 7fe28a7fc640 1 -- 172.21.15.70:0/334522537 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe29406dd20 con 0x7fe28c0a4c20 2024-04-01T01:41:37.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.713+0000 7fe28a7fc640 1 -- 172.21.15.70:0/334522537 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2940628f0 con 0x7fe28c0a4c20 2024-04-01T01:41:37.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.717+0000 7fe29df64740 1 Processor -- start 2024-04-01T01:41:37.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.717+0000 7fe29df64740 1 -- start start 2024-04-01T01:41:37.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.717+0000 7fe29df64740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90c5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.717+0000 7fe29df64740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c5600 0x5555a90ca100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.717+0000 7fe29df64740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90ca770 0x5555a90ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.717+0000 7fe29df64740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5555a8fc60b0 con 0x5555a90ca770 2024-04-01T01:41:37.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.717+0000 7fe29df64740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5555a9028ea0 con 0x5555a90c5600 2024-04-01T01:41:37.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.717+0000 7fe29df64740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5555a8fd1c00 con 0x5555a90c4c10 2024-04-01T01:41:37.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.718+0000 7fe299841640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:37.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.718+0000 7fe299841640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90c5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:56128/0 (socket says 172.21.15.70:56128) 2024-04-01T01:41:37.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.718+0000 7fe299841640 1 -- 172.21.15.70:0/2216882442 learned_addr learned my addr 172.21.15.70:0/2216882442 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:37.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.718+0000 7fe29883f640 1 --2- 172.21.15.70:0/2216882442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90ca770 0x5555a90ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:37.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.718+0000 7fe299841640 1 -- 172.21.15.70:0/2216882442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c5600 msgr2=0x5555a90ca100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:41:37.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.718+0000 7fe299841640 1 --2- 172.21.15.70:0/2216882442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c5600 0x5555a90ca100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.718+0000 7fe299841640 1 -- 172.21.15.70:0/2216882442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90ca770 msgr2=0x5555a90ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:37.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.718+0000 7fe299841640 1 --2- 172.21.15.70:0/2216882442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90ca770 0x5555a90ccc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.718+0000 7fe299841640 1 -- 172.21.15.70:0/2216882442 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555a8e057e0 con 0x5555a90c4c10 2024-04-01T01:41:37.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.718+0000 7fe299841640 1 --2- 172.21.15.70:0/2216882442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90c5030 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe2840096a0 tx=0x7fe28402c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e8606ad6829179d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:37.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.719+0000 7fe27f7fe640 1 -- 172.21.15.70:0/2216882442 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284045020 con 0x5555a90c4c10 2024-04-01T01:41:37.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.719+0000 7fe27f7fe640 1 -- 172.21.15.70:0/2216882442 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe28402cbf0 con 0x5555a90c4c10 2024-04-01T01:41:37.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.719+0000 7fe27f7fe640 1 -- 172.21.15.70:0/2216882442 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284004920 con 0x5555a90c4c10 2024-04-01T01:41:37.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.719+0000 7fe28a7fc640 1 -- 172.21.15.70:0/334522537 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fe29407c070 con 0x7fe28c0a4c20 2024-04-01T01:41:37.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.720+0000 7fe29df64740 1 -- 172.21.15.70:0/2216882442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 msgr2=0x5555a90c5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:37.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.720+0000 7fe28a7fc640 1 --2- 172.21.15.70:0/334522537 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2800a25a0 0x7fe2800a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.720+0000 7fe29df64740 1 --2- 172.21.15.70:0/2216882442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90c5030 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe2840096a0 tx=0x7fe28402c910 comp rx=0 tx=0).stop 2024-04-01T01:41:37.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.720+0000 7fe28a7fc640 1 -- 172.21.15.70:0/334522537 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe294131550 con 0x7fe28c0a4c20 2024-04-01T01:41:37.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.721+0000 7fe29df64740 1 -- 172.21.15.70:0/2216882442 shutdown_connections 2024-04-01T01:41:37.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.721+0000 7fe29df64740 1 --2- 172.21.15.70:0/2216882442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90ca770 0x5555a90ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.721+0000 7fe29df64740 1 --2- 172.21.15.70:0/2216882442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c5600 0x5555a90ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.721+0000 7fe29df64740 1 --2- 172.21.15.70:0/2216882442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90c5030 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.721+0000 7fe29df64740 1 -- 172.21.15.70:0/2216882442 >> 172.21.15.70:0/2216882442 conn(0x5555a90c0180 msgr2=0x5555a90c0580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:37.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.721+0000 7fe299040640 1 --2- 172.21.15.70:0/334522537 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2800a25a0 0x7fe2800a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:37.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.722+0000 7fe299040640 1 --2- 172.21.15.70:0/334522537 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2800a25a0 0x7fe2800a4a60 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe2940fdb70 tx=0x7fe2940675f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:37.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.722+0000 7fe29df64740 1 -- 172.21.15.70:0/2216882442 shutdown_connections 2024-04-01T01:41:37.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.722+0000 7fe29df64740 1 -- 172.21.15.70:0/2216882442 wait complete. 2024-04-01T01:41:37.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.723+0000 7fe29df64740 1 -- start start 2024-04-01T01:41:37.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.723+0000 7fe29df64740 1 Processor -- start 2024-04-01T01:41:37.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.723+0000 7fe29df64740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.724+0000 7fe29df64740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c5600 0x5555a90cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.724+0000 7fe29df64740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90ca770 0x5555a90d6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.724+0000 7fe29df64740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5555a8fc73d0 con 0x5555a90ca770 2024-04-01T01:41:37.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.724+0000 7fe29df64740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5555a9024f40 con 0x5555a90c5600 2024-04-01T01:41:37.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.724+0000 7fe299841640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:37.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.724+0000 7fe29df64740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5555a90cd6c0 con 0x5555a90c4c10 2024-04-01T01:41:37.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.724+0000 7fe299841640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90cef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:56138/0 (socket says 172.21.15.70:56138) 2024-04-01T01:41:37.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.724+0000 7fe299841640 1 -- 172.21.15.70:0/204612333 learned_addr learned my addr 172.21.15.70:0/204612333 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:37.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.724+0000 7fe29883f640 1 --2- 172.21.15.70:0/204612333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c5600 0x5555a90cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:37.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.724+0000 7fe29df64740 10 client.0 _ll_register_callbacks cb 0x5555a8fe4b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:37.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.724+0000 7fe299841640 1 -- 172.21.15.70:0/204612333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c5600 msgr2=0x5555a90cf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:37.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.724+0000 7fe299841640 1 --2- 172.21.15.70:0/204612333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c5600 0x5555a90cf4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.724+0000 7fe299841640 1 -- 172.21.15.70:0/204612333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90ca770 msgr2=0x5555a90d6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:37.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.724+0000 7fe299841640 1 --2- 172.21.15.70:0/204612333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90ca770 0x5555a90d6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.724+0000 7fe299841640 1 -- 172.21.15.70:0/204612333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555a8e65f70 con 0x5555a90c4c10 2024-04-01T01:41:37.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.725+0000 7fe299841640 1 --2- 172.21.15.70:0/204612333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90cef80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe284005c40 tx=0x7fe284005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b12857d4da9e257b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:37.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.725+0000 7fe27cff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:41:37.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.725+0000 7fe27cff9640 1 -- 172.21.15.70:0/204612333 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284053060 con 0x5555a90c4c10 2024-04-01T01:41:37.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.725+0000 7fe27cff9640 1 -- 172.21.15.70:0/204612333 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe284004980 con 0x5555a90c4c10 2024-04-01T01:41:37.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.725+0000 7fe29df64740 1 -- 172.21.15.70:0/204612333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe28402cbf0 con 0x5555a90c4c10 2024-04-01T01:41:37.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.726+0000 7fe27cff9640 1 -- 172.21.15.70:0/204612333 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe284045070 con 0x5555a90c4c10 2024-04-01T01:41:37.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.726+0000 7fe29df64740 10 client.27760 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:37.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.726+0000 7fe29df64740 1 -- 172.21.15.70:0/204612333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5555a8cfd170 con 0x5555a90c4c10 2024-04-01T01:41:37.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.726+0000 7fe27cff9640 1 -- 172.21.15.70:0/204612333 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe284004b80 con 0x5555a90c4c10 2024-04-01T01:41:37.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.726+0000 7fe27cff9640 1 -- 172.21.15.70:0/204612333 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe284056050 con 0x5555a90c4c10 2024-04-01T01:41:37.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.727+0000 7fe29df64740 10 client.27760 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:37.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.727+0000 7fe29df64740 10 client.27760 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:37.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.727+0000 7fe29df64740 20 client.27760 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:37.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.727+0000 7fe29df64740 1 --2- 172.21.15.70:0/204612333 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5555a90da620 0x5555a90dca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.727+0000 7fe29df64740 4 client.27760 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:37.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.727+0000 7fe29df64740 1 -- 172.21.15.70:0/204612333 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe28402cbf0 con 0x5555a90da620 2024-04-01T01:41:37.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.728+0000 7fe29883f640 1 --2- 172.21.15.70:0/204612333 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5555a90da620 0x5555a90dca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:37.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.728+0000 7fe27cff9640 10 client.27760 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:37.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.728+0000 7fe29883f640 1 --2- 172.21.15.70:0/204612333 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5555a90da620 0x5555a90dca10 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9cc206229968fbf server_cookie=5caf11c4e0b4579d in_seq=0 out_seq=0 2024-04-01T01:41:37.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.729+0000 7fe27cff9640 1 -- 172.21.15.70:0/204612333 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fe28402cbf0 con 0x5555a90da620 2024-04-01T01:41:37.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.729+0000 7fe27cff9640 10 client.27760 handle_command_reply: tid=0 2024-04-01T01:41:37.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.729+0000 7fe29df64740 1 client.27760 shutdown 2024-04-01T01:41:37.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.730+0000 7fe29df64740 1 -- 172.21.15.70:0/204612333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 msgr2=0x5555a90cef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:37.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.730+0000 7fe29df64740 1 --2- 172.21.15.70:0/204612333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90cef80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe284005c40 tx=0x7fe284005740 comp rx=0 tx=0).stop 2024-04-01T01:41:37.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.730+0000 7fe29df64740 1 -- 172.21.15.70:0/204612333 shutdown_connections 2024-04-01T01:41:37.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.730+0000 7fe29df64740 1 --2- 172.21.15.70:0/204612333 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5555a90da620 0x5555a90dca10 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.730+0000 7fe29df64740 1 --2- 172.21.15.70:0/204612333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90ca770 0x5555a90d6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.730+0000 7fe29df64740 1 --2- 172.21.15.70:0/204612333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c5600 0x5555a90cf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.730+0000 7fe29df64740 1 --2- 172.21.15.70:0/204612333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90cef80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.730+0000 7fe29df64740 1 -- 172.21.15.70:0/204612333 >> 172.21.15.70:0/204612333 conn(0x5555a90c0180 msgr2=0x5555a90bf3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:37.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.730+0000 7fe27cff9640 0 client.27760 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:37.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.730+0000 7fe29df64740 1 -- 172.21.15.70:0/204612333 shutdown_connections 2024-04-01T01:41:37.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.730+0000 7fe29df64740 1 -- 172.21.15.70:0/204612333 wait complete. 2024-04-01T01:41:37.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.730+0000 7fe29df64740 20 client.27760 trim_cache size 0 max 16384 2024-04-01T01:41:37.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.762+0000 7fe29df64740 1 Processor -- start 2024-04-01T01:41:37.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.762+0000 7fe29df64740 1 -- start start 2024-04-01T01:41:37.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.762+0000 7fe29df64740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c4c10 0x5555a90cf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.762+0000 7fe29df64740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90c5600 0x5555a90d6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.762+0000 7fe29df64740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90ca770 0x5555a90c3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.762+0000 7fe29df64740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe27800fd90 con 0x5555a90c5600 2024-04-01T01:41:37.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.762+0000 7fe29df64740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe2780118d0 con 0x5555a90c4c10 2024-04-01T01:41:37.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.762+0000 7fe29df64740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe278010460 con 0x5555a90ca770 2024-04-01T01:41:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.765+0000 7fe299841640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c4c10 0x5555a90cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.765+0000 7fe299841640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c4c10 0x5555a90cf420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48492/0 (socket says 172.21.15.70:48492) 2024-04-01T01:41:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.765+0000 7fe299841640 1 -- 172.21.15.70:0/2674667713 learned_addr learned my addr 172.21.15.70:0/2674667713 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.766+0000 7fe29883f640 1 --2- 172.21.15.70:0/2674667713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90c5600 0x5555a90d6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.766+0000 7fe29883f640 1 -- 172.21.15.70:0/2674667713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90ca770 msgr2=0x5555a90c3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.766+0000 7fe29883f640 1 --2- 172.21.15.70:0/2674667713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90ca770 0x5555a90c3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.766+0000 7fe29883f640 1 -- 172.21.15.70:0/2674667713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c4c10 msgr2=0x5555a90cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.766+0000 7fe29883f640 1 --2- 172.21.15.70:0/2674667713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c4c10 0x5555a90cf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.766+0000 7fe29883f640 1 -- 172.21.15.70:0/2674667713 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555a90d0100 con 0x5555a90c5600 2024-04-01T01:41:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.767+0000 7fe29883f640 1 --2- 172.21.15.70:0/2674667713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90c5600 0x5555a90d6490 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x5555a90dd100 tx=0x7fe2900047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e7ddba857ca71ca server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:37.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.768+0000 7fe27effd640 1 -- 172.21.15.70:0/2674667713 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe290014c10 con 0x5555a90c5600 2024-04-01T01:41:37.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.769+0000 7fe27effd640 1 -- 172.21.15.70:0/2674667713 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe290025070 con 0x5555a90c5600 2024-04-01T01:41:37.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.770+0000 7fe29df64740 1 -- 172.21.15.70:0/2674667713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90c5600 msgr2=0x5555a90d6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:37.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.770+0000 7fe29df64740 1 --2- 172.21.15.70:0/2674667713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90c5600 0x5555a90d6490 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x5555a90dd100 tx=0x7fe2900047d0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.772+0000 7fe29df64740 1 -- 172.21.15.70:0/2674667713 shutdown_connections 2024-04-01T01:41:37.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.772+0000 7fe29df64740 1 --2- 172.21.15.70:0/2674667713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90ca770 0x5555a90c3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.772+0000 7fe29df64740 1 --2- 172.21.15.70:0/2674667713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90c5600 0x5555a90d6490 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.772+0000 7fe29df64740 1 --2- 172.21.15.70:0/2674667713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c4c10 0x5555a90cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.772+0000 7fe29df64740 1 -- 172.21.15.70:0/2674667713 >> 172.21.15.70:0/2674667713 conn(0x5555a90c0180 msgr2=0x5555a90c33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:37.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.772+0000 7fe29df64740 1 -- 172.21.15.70:0/2674667713 shutdown_connections 2024-04-01T01:41:37.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.772+0000 7fe29df64740 1 -- 172.21.15.70:0/2674667713 wait complete. 2024-04-01T01:41:37.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.773+0000 7fe29df64740 1 -- start start 2024-04-01T01:41:37.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.773+0000 7fe29df64740 1 Processor -- start 2024-04-01T01:41:37.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.773+0000 7fe29df64740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90d5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.773+0000 7fe29df64740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c5600 0x5555a90d64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.773+0000 7fe29df64740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90ca770 0x5555a90d4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.773+0000 7fe29df64740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe278011200 con 0x5555a90ca770 2024-04-01T01:41:37.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.773+0000 7fe29df64740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe278010b30 con 0x5555a90c5600 2024-04-01T01:41:37.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.773+0000 7fe29df64740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe27800dd70 con 0x5555a90c4c10 2024-04-01T01:41:37.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.773+0000 7fe29df64740 10 client.0 _ll_register_callbacks cb 0x7fe2840095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:37.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.774+0000 7fe29883f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c5600 0x5555a90d64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:37.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.774+0000 7fe29883f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c5600 0x5555a90d64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48508/0 (socket says 172.21.15.70:48508) 2024-04-01T01:41:37.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.774+0000 7fe29883f640 1 -- 172.21.15.70:0/1548687954 learned_addr learned my addr 172.21.15.70:0/1548687954 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:37.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.774+0000 7fe299841640 1 --2- 172.21.15.70:0/1548687954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90d5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:37.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.774+0000 7fe299841640 1 -- 172.21.15.70:0/1548687954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c5600 msgr2=0x5555a90d64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:37.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.774+0000 7fe299841640 1 --2- 172.21.15.70:0/1548687954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c5600 0x5555a90d64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.774+0000 7fe299841640 1 -- 172.21.15.70:0/1548687954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90ca770 msgr2=0x5555a90d4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:37.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.774+0000 7fe299841640 1 --2- 172.21.15.70:0/1548687954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90ca770 0x5555a90d4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.774+0000 7fe299841640 1 -- 172.21.15.70:0/1548687954 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555a90d4e80 con 0x5555a90c4c10 2024-04-01T01:41:37.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.774+0000 7fe29883f640 1 --2- 172.21.15.70:0/1548687954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c5600 0x5555a90d64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:41:37.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.775+0000 7fe299841640 1 --2- 172.21.15.70:0/1548687954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90d5fa0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe284005a40 tx=0x7fe284005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b433bb6c0fdf3740 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:37.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.775+0000 7fe27cff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:41:37.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.775+0000 7fe27cff9640 1 -- 172.21.15.70:0/1548687954 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28405c030 con 0x5555a90c4c10 2024-04-01T01:41:37.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.775+0000 7fe27cff9640 1 -- 172.21.15.70:0/1548687954 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe28402c910 con 0x5555a90c4c10 2024-04-01T01:41:37.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.775+0000 7fe29df64740 1 -- 172.21.15.70:0/1548687954 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe290025070 con 0x5555a90c4c10 2024-04-01T01:41:37.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.775+0000 7fe27cff9640 1 -- 172.21.15.70:0/1548687954 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe284066050 con 0x5555a90c4c10 2024-04-01T01:41:37.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.776+0000 7fe29df64740 10 client.27763 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:37.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.776+0000 7fe29df64740 1 -- 172.21.15.70:0/1548687954 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5555a9152450 con 0x5555a90c4c10 2024-04-01T01:41:37.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.777+0000 7fe27cff9640 1 -- 172.21.15.70:0/1548687954 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe290025070 con 0x5555a90c4c10 2024-04-01T01:41:37.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.778+0000 7fe27cff9640 1 -- 172.21.15.70:0/1548687954 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe284056040 con 0x5555a90c4c10 2024-04-01T01:41:37.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.778+0000 7fe29df64740 10 client.27763 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:37.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.778+0000 7fe29df64740 10 client.27763 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:37.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.778+0000 7fe29df64740 20 client.27763 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.778+0000 7fe29df64740 1 --2- 172.21.15.70:0/1548687954 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5555a9152e30 0x5555a9155250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.778+0000 7fe29df64740 4 client.27763 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:41:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.778+0000 7fe29df64740 1 -- 172.21.15.70:0/1548687954 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5555a90363e0 con 0x5555a9152e30 2024-04-01T01:41:37.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.779+0000 7fe29883f640 1 --2- 172.21.15.70:0/1548687954 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5555a9152e30 0x5555a9155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:37.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.779+0000 7fe29883f640 1 --2- 172.21.15.70:0/1548687954 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5555a9152e30 0x5555a9155250 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3196ae6570cd1001 server_cookie=ba1127849030c44 in_seq=0 out_seq=0 2024-04-01T01:41:37.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.780+0000 7fe27cff9640 10 client.27763 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:37.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.782+0000 7fe27cff9640 1 -- 172.21.15.70:0/1548687954 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5555a90363e0 con 0x5555a9152e30 2024-04-01T01:41:37.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.783+0000 7fe27cff9640 10 client.27763 handle_command_reply: tid=0 2024-04-01T01:41:37.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.783+0000 7fe29df64740 1 client.27763 shutdown 2024-04-01T01:41:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.784+0000 7fe29df64740 1 -- 172.21.15.70:0/1548687954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 msgr2=0x5555a90d5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.784+0000 7fe29df64740 1 --2- 172.21.15.70:0/1548687954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90d5fa0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe284005a40 tx=0x7fe284005c90 comp rx=0 tx=0).stop 2024-04-01T01:41:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.784+0000 7fe29df64740 1 -- 172.21.15.70:0/1548687954 shutdown_connections 2024-04-01T01:41:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.784+0000 7fe29df64740 1 --2- 172.21.15.70:0/1548687954 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5555a9152e30 0x5555a9155250 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.784+0000 7fe29df64740 1 --2- 172.21.15.70:0/1548687954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5555a90ca770 0x5555a90d4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.784+0000 7fe29df64740 1 --2- 172.21.15.70:0/1548687954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5555a90c5600 0x5555a90d64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.784+0000 7fe29df64740 1 --2- 172.21.15.70:0/1548687954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5555a90c4c10 0x5555a90d5fa0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.784+0000 7fe29df64740 1 -- 172.21.15.70:0/1548687954 >> 172.21.15.70:0/1548687954 conn(0x5555a90c0180 msgr2=0x5555a90c3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:37.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.784+0000 7fe27cff9640 0 client.27763 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:37.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.784+0000 7fe29df64740 1 -- 172.21.15.70:0/1548687954 shutdown_connections 2024-04-01T01:41:37.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.785+0000 7fe29df64740 1 -- 172.21.15.70:0/1548687954 wait complete. 2024-04-01T01:41:37.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.785+0000 7fe29df64740 20 client.27763 trim_cache size 0 max 16384 2024-04-01T01:41:37.784 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:41:37.784 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:41:37.784 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "1b05d11c-4705-4e58-8fb0-0a13c9732569", 2024-04-01T01:41:37.785 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:41:37.785 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:41:37.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.786+0000 7fe29a042640 1 -- 172.21.15.70:0/334522537 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2800a25a0 msgr2=0x7fe2800a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:37.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.786+0000 7fe29a042640 1 --2- 172.21.15.70:0/334522537 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2800a25a0 0x7fe2800a4a60 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe2940fdb70 tx=0x7fe2940675f0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.786+0000 7fe29a042640 1 -- 172.21.15.70:0/334522537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe28c0a4c20 msgr2=0x7fe28c143700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:37.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.786+0000 7fe29a042640 1 --2- 172.21.15.70:0/334522537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe28c0a4c20 0x7fe28c143700 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe294000b60 tx=0x7fe29406d430 comp rx=0 tx=0).stop 2024-04-01T01:41:37.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.787+0000 7fe29a042640 1 -- 172.21.15.70:0/334522537 shutdown_connections 2024-04-01T01:41:37.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.787+0000 7fe29a042640 1 --2- 172.21.15.70:0/334522537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe28c0a66f0 0x7fe28c14ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.787+0000 7fe29a042640 1 --2- 172.21.15.70:0/334522537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe28c0a5d50 0x7fe28c143c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.787+0000 7fe29a042640 1 --2- 172.21.15.70:0/334522537 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2800a25a0 0x7fe2800a4a60 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.787+0000 7fe29a042640 1 --2- 172.21.15.70:0/334522537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe28c0a4c20 0x7fe28c143700 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:37.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.787+0000 7fe29a042640 1 -- 172.21.15.70:0/334522537 >> 172.21.15.70:0/334522537 conn(0x7fe28c0a00b0 msgr2=0x7fe28c0a6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:37.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.790+0000 7fe29a042640 1 -- 172.21.15.70:0/334522537 shutdown_connections 2024-04-01T01:41:37.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:37.790+0000 7fe29a042640 1 -- 172.21.15.70:0/334522537 wait complete. 2024-04-01T01:41:37.808 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:41:37.808 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:41:38.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.002+0000 7fa8e7fff640 1 -- 172.21.15.70:0/3422213385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8e81097e0 msgr2=0x7fa8e810a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:38.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.002+0000 7fa8e7fff640 1 --2- 172.21.15.70:0/3422213385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8e81097e0 0x7fa8e810a060 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009860 tx=0x7fa8d802bb90 comp rx=0 tx=0).stop 2024-04-01T01:41:38.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.002+0000 7fa8e7fff640 1 -- 172.21.15.70:0/3422213385 shutdown_connections 2024-04-01T01:41:38.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.002+0000 7fa8e7fff640 1 --2- 172.21.15.70:0/3422213385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8e81097e0 0x7fa8e810a060 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.002+0000 7fa8e7fff640 1 --2- 172.21.15.70:0/3422213385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e8108e20 0x7fa8e81092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.002+0000 7fa8e7fff640 1 --2- 172.21.15.70:0/3422213385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e8072be0 0x7fa8e8073000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.002+0000 7fa8e7fff640 1 -- 172.21.15.70:0/3422213385 >> 172.21.15.70:0/3422213385 conn(0x7fa8e806e630 msgr2=0x7fa8e8070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:38.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.002+0000 7fa8e7fff640 1 -- 172.21.15.70:0/3422213385 shutdown_connections 2024-04-01T01:41:38.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.002+0000 7fa8e7fff640 1 -- 172.21.15.70:0/3422213385 wait complete. 2024-04-01T01:41:38.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.003+0000 7fa8e7fff640 1 Processor -- start 2024-04-01T01:41:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.003+0000 7fa8e7fff640 1 -- start start 2024-04-01T01:41:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.003+0000 7fa8e7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e8072be0 0x7fa8e81a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.004+0000 7fa8e7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8e8108e20 0x7fa8e81a2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.004+0000 7fa8e7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e81097e0 0x7fa8e81a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.004+0000 7fa8e7fff640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e8117b80 con 0x7fa8e8072be0 2024-04-01T01:41:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.004+0000 7fa8e7fff640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e8117a00 con 0x7fa8e8108e20 2024-04-01T01:41:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.004+0000 7fa8e7fff640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e8117d00 con 0x7fa8e81097e0 2024-04-01T01:41:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.004+0000 7fa8e6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e8072be0 0x7fa8e81a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:38.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.004+0000 7fa8e6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e8072be0 0x7fa8e81a2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60748/0 (socket says 172.21.15.70:60748) 2024-04-01T01:41:38.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.004+0000 7fa8e6ffd640 1 -- 172.21.15.70:0/747834808 learned_addr learned my addr 172.21.15.70:0/747834808 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:38.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.004+0000 7fa8e77fe640 1 --2- 172.21.15.70:0/747834808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e81097e0 0x7fa8e81a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:38.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.004+0000 7fa8e67fc640 1 --2- 172.21.15.70:0/747834808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8e8108e20 0x7fa8e81a2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:38.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.004+0000 7fa8e6ffd640 1 -- 172.21.15.70:0/747834808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e81097e0 msgr2=0x7fa8e81a97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:38.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.004+0000 7fa8e6ffd640 1 --2- 172.21.15.70:0/747834808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e81097e0 0x7fa8e81a97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.004+0000 7fa8e6ffd640 1 -- 172.21.15.70:0/747834808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8e8108e20 msgr2=0x7fa8e81a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:38.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.004+0000 7fa8e6ffd640 1 --2- 172.21.15.70:0/747834808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8e8108e20 0x7fa8e81a2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.004+0000 7fa8e6ffd640 1 -- 172.21.15.70:0/747834808 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e81a9e40 con 0x7fa8e8072be0 2024-04-01T01:41:38.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.005+0000 7fa8e6ffd640 1 --2- 172.21.15.70:0/747834808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e8072be0 0x7fa8e81a2220 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000efb0 tx=0x7fa8e000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:38.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.006+0000 7fa8cffff640 1 -- 172.21.15.70:0/747834808 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0038620 con 0x7fa8e8072be0 2024-04-01T01:41:38.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.006+0000 7fa8cffff640 1 -- 172.21.15.70:0/747834808 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8e0038c70 con 0x7fa8e8072be0 2024-04-01T01:41:38.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.006+0000 7fa8cffff640 1 -- 172.21.15.70:0/747834808 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0037cd0 con 0x7fa8e8072be0 2024-04-01T01:41:38.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.007+0000 7fa8e7fff640 1 -- 172.21.15.70:0/747834808 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8e80739e0 con 0x7fa8e8072be0 2024-04-01T01:41:38.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.007+0000 7fa8e7fff640 1 -- 172.21.15.70:0/747834808 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8e8073ee0 con 0x7fa8e8072be0 2024-04-01T01:41:38.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.011+0000 7fa8cffff640 1 -- 172.21.15.70:0/747834808 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fa8e00387c0 con 0x7fa8e8072be0 2024-04-01T01:41:38.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.011+0000 7fa8effce740 1 Processor -- start 2024-04-01T01:41:38.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.012+0000 7fa8effce740 1 -- start start 2024-04-01T01:41:38.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.012+0000 7fa8effce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853a4ca0 0x5643853a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.012+0000 7fa8effce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a5690 0x5643853aa190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.012+0000 7fa8effce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853acc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.012+0000 7fa8effce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564385301180 con 0x5643853aa800 2024-04-01T01:41:38.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.012+0000 7fa8effce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5643852a9b70 con 0x5643853a5690 2024-04-01T01:41:38.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.012+0000 7fa8effce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5643852fd220 con 0x5643853a4ca0 2024-04-01T01:41:38.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.013+0000 7fa8e6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a5690 0x5643853aa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:38.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.013+0000 7fa8cffff640 1 --2- 172.21.15.70:0/747834808 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8d00a25a0 0x7fa8d00a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.013+0000 7fa8e6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a5690 0x5643853aa190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48546/0 (socket says 172.21.15.70:48546) 2024-04-01T01:41:38.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.013+0000 7fa8e6ffd640 1 -- 172.21.15.70:0/2039208007 learned_addr learned my addr 172.21.15.70:0/2039208007 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:38.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.013+0000 7fa8e77fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853a4ca0 0x5643853a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:38.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.013+0000 7fa8e67fc640 1 --2- 172.21.15.70:0/2039208007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853acc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:38.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.013+0000 7fa8cffff640 1 -- 172.21.15.70:0/747834808 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa8e00f11b0 con 0x7fa8e8072be0 2024-04-01T01:41:38.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.013+0000 7fa8e6ffd640 1 -- 172.21.15.70:0/2039208007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853a4ca0 msgr2=0x5643853a50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:38.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.013+0000 7fa8e6ffd640 1 --2- 172.21.15.70:0/2039208007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853a4ca0 0x5643853a50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.013+0000 7fa8e6ffd640 1 -- 172.21.15.70:0/2039208007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 msgr2=0x5643853acc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:38.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.013+0000 7fa8e6ffd640 1 --2- 172.21.15.70:0/2039208007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853acc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.013+0000 7fa8e6ffd640 1 -- 172.21.15.70:0/2039208007 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643852bd7c0 con 0x5643853a5690 2024-04-01T01:41:38.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.013+0000 7fa8e6ffd640 1 --2- 172.21.15.70:0/2039208007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a5690 0x5643853aa190 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000cfd0 tx=0x7fa8e00b94e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=523777c426534596 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:38.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.014+0000 7fa8cd7fa640 1 -- 172.21.15.70:0/2039208007 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0037e70 con 0x5643853a5690 2024-04-01T01:41:38.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.014+0000 7fa8cd7fa640 1 -- 172.21.15.70:0/2039208007 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8e0038ab0 con 0x5643853a5690 2024-04-01T01:41:38.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.014+0000 7fa8cd7fa640 1 -- 172.21.15.70:0/2039208007 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e00b9eb0 con 0x5643853a5690 2024-04-01T01:41:38.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.014+0000 7fa8e77fe640 1 --2- 172.21.15.70:0/747834808 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8d00a25a0 0x7fa8d00a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:38.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.015+0000 7fa8effce740 1 -- 172.21.15.70:0/2039208007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a5690 msgr2=0x5643853aa190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:38.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.015+0000 7fa8effce740 1 --2- 172.21.15.70:0/2039208007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a5690 0x5643853aa190 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000cfd0 tx=0x7fa8e00b94e0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.015+0000 7fa8e77fe640 1 --2- 172.21.15.70:0/747834808 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8d00a25a0 0x7fa8d00a4a60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8034b30 tx=0x7fa8d802e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:38.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.015+0000 7fa8effce740 1 -- 172.21.15.70:0/2039208007 shutdown_connections 2024-04-01T01:41:38.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.015+0000 7fa8effce740 1 --2- 172.21.15.70:0/2039208007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853acc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.015+0000 7fa8effce740 1 --2- 172.21.15.70:0/2039208007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a5690 0x5643853aa190 secure :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000cfd0 tx=0x7fa8e00b94e0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.015+0000 7fa8effce740 1 --2- 172.21.15.70:0/2039208007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853a4ca0 0x5643853a50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.015+0000 7fa8effce740 1 -- 172.21.15.70:0/2039208007 >> 172.21.15.70:0/2039208007 conn(0x5643853a0210 msgr2=0x5643853a0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:38.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.015+0000 7fa8effce740 1 -- 172.21.15.70:0/2039208007 shutdown_connections 2024-04-01T01:41:38.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.015+0000 7fa8effce740 1 -- 172.21.15.70:0/2039208007 wait complete. 2024-04-01T01:41:38.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.015+0000 7fa8effce740 1 -- start start 2024-04-01T01:41:38.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.016+0000 7fa8effce740 1 Processor -- start 2024-04-01T01:41:38.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.016+0000 7fa8effce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a4ca0 0x5643853af300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.016+0000 7fa8effce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.016+0000 7fa8effce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853b4870 0x5643853b6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.016+0000 7fa8effce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5643853ad6f0 con 0x5643853aa800 2024-04-01T01:41:38.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.016+0000 7fa8effce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56438529f4a0 con 0x5643853a4ca0 2024-04-01T01:41:38.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.016+0000 7fa8effce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5643853ad870 con 0x5643853b4870 2024-04-01T01:41:38.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.016+0000 7fa8effce740 10 client.0 _ll_register_callbacks cb 0x5643852a2d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:38.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.017+0000 7fa8e67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:38.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.017+0000 7fa8e67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60768/0 (socket says 172.21.15.70:60768) 2024-04-01T01:41:38.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.017+0000 7fa8e67fc640 1 -- 172.21.15.70:0/1279628278 learned_addr learned my addr 172.21.15.70:0/1279628278 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:38.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.017+0000 7fa8e67fc640 1 -- 172.21.15.70:0/1279628278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853b4870 msgr2=0x5643853b6c60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:41:38.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.018+0000 7fa8e67fc640 1 --2- 172.21.15.70:0/1279628278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853b4870 0x5643853b6c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.018+0000 7fa8e67fc640 1 -- 172.21.15.70:0/1279628278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a4ca0 msgr2=0x5643853af300 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:41:38.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.018+0000 7fa8e67fc640 1 --2- 172.21.15.70:0/1279628278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a4ca0 0x5643853af300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.018+0000 7fa8e67fc640 1 -- 172.21.15.70:0/1279628278 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643850df7d0 con 0x5643853aa800 2024-04-01T01:41:38.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.018+0000 7fa8e67fc640 1 --2- 172.21.15.70:0/1279628278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853af840 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc012700 tx=0x7fa8dc012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4558ec390c96c135 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:38.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.018+0000 7fa8caff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:41:38.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.018+0000 7fa8caff5640 1 -- 172.21.15.70:0/1279628278 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc004260 con 0x5643853aa800 2024-04-01T01:41:38.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.018+0000 7fa8caff5640 1 -- 172.21.15.70:0/1279628278 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa8dc004400 con 0x5643853aa800 2024-04-01T01:41:38.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.019+0000 7fa8effce740 1 -- 172.21.15.70:0/1279628278 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56438530e6c0 con 0x5643853aa800 2024-04-01T01:41:38.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.019+0000 7fa8caff5640 1 -- 172.21.15.70:0/1279628278 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa8dc011940 con 0x5643853aa800 2024-04-01T01:41:38.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.019+0000 7fa8effce740 10 client.17973 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:38.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.019+0000 7fa8effce740 1 -- 172.21.15.70:0/1279628278 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5643850153f0 con 0x5643853aa800 2024-04-01T01:41:38.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.020+0000 7fa8caff5640 1 -- 172.21.15.70:0/1279628278 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56438530e6c0 con 0x5643853aa800 2024-04-01T01:41:38.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.020+0000 7fa8caff5640 1 -- 172.21.15.70:0/1279628278 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fa8dc00f440 con 0x5643853aa800 2024-04-01T01:41:38.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.022+0000 7fa8effce740 10 client.17973 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:38.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.022+0000 7fa8effce740 10 client.17973 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:38.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.022+0000 7fa8effce740 20 client.17973 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:38.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.022+0000 7fa8effce740 1 --2- 172.21.15.70:0/1279628278 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5643853bb560 0x5643853bda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.022+0000 7fa8effce740 4 client.17973 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:38.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.022+0000 7fa8effce740 1 -- 172.21.15.70:0/1279628278 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564385308a10 con 0x5643853bb560 2024-04-01T01:41:38.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.022+0000 7fa8e6ffd640 1 --2- 172.21.15.70:0/1279628278 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5643853bb560 0x5643853bda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:38.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.025+0000 7fa8e6ffd640 1 --2- 172.21.15.70:0/1279628278 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5643853bb560 0x5643853bda00 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=122722fd46382a0d server_cookie=c0de9bf52a2aca25 in_seq=0 out_seq=0 2024-04-01T01:41:38.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.025+0000 7fa8caff5640 10 client.17973 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:38.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.026+0000 7fa8caff5640 1 -- 172.21.15.70:0/1279628278 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564385308a10 con 0x5643853bb560 2024-04-01T01:41:38.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.026+0000 7fa8caff5640 10 client.17973 handle_command_reply: tid=0 2024-04-01T01:41:38.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.026+0000 7fa8effce740 1 client.17973 shutdown 2024-04-01T01:41:38.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.026+0000 7fa8effce740 1 -- 172.21.15.70:0/1279628278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 msgr2=0x5643853af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:38.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.026+0000 7fa8effce740 1 --2- 172.21.15.70:0/1279628278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853af840 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc012700 tx=0x7fa8dc012bc0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.026+0000 7fa8effce740 1 -- 172.21.15.70:0/1279628278 shutdown_connections 2024-04-01T01:41:38.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.026+0000 7fa8effce740 1 --2- 172.21.15.70:0/1279628278 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5643853bb560 0x5643853bda00 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.026+0000 7fa8effce740 1 --2- 172.21.15.70:0/1279628278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853b4870 0x5643853b6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.027+0000 7fa8effce740 1 --2- 172.21.15.70:0/1279628278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853af840 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.027+0000 7fa8effce740 1 --2- 172.21.15.70:0/1279628278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a4ca0 0x5643853af300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.027+0000 7fa8effce740 1 -- 172.21.15.70:0/1279628278 >> 172.21.15.70:0/1279628278 conn(0x5643853a0210 msgr2=0x564385396c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:38.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.027+0000 7fa8caff5640 0 client.17973 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:38.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.027+0000 7fa8effce740 1 -- 172.21.15.70:0/1279628278 shutdown_connections 2024-04-01T01:41:38.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.027+0000 7fa8effce740 1 -- 172.21.15.70:0/1279628278 wait complete. 2024-04-01T01:41:38.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.027+0000 7fa8effce740 20 client.17973 trim_cache size 0 max 16384 2024-04-01T01:41:38.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.063+0000 7fa8effce740 1 Processor -- start 2024-04-01T01:41:38.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.063+0000 7fa8effce740 1 -- start start 2024-04-01T01:41:38.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.063+0000 7fa8effce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853a4ca0 0x5643853a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.063+0000 7fa8effce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853adc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.063+0000 7fa8effce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853b4870 0x5643853ae180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.063+0000 7fa8effce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e807a2e0 con 0x5643853aa800 2024-04-01T01:41:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.063+0000 7fa8effce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e807be20 con 0x5643853b4870 2024-04-01T01:41:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.063+0000 7fa8effce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e807a9b0 con 0x5643853a4ca0 2024-04-01T01:41:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.064+0000 7fa8e77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853b4870 0x5643853ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.064+0000 7fa8e67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853adc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.064+0000 7fa8e67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853adc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60782/0 (socket says 172.21.15.70:60782) 2024-04-01T01:41:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.064+0000 7fa8e77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853b4870 0x5643853ae180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48568/0 (socket says 172.21.15.70:48568) 2024-04-01T01:41:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.064+0000 7fa8e67fc640 1 -- 172.21.15.70:0/1114580096 learned_addr learned my addr 172.21.15.70:0/1114580096 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.064+0000 7fa8e6ffd640 1 --2- 172.21.15.70:0/1114580096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853a4ca0 0x5643853a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.064+0000 7fa8e77fe640 1 -- 172.21.15.70:0/1114580096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853a4ca0 msgr2=0x5643853a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.064+0000 7fa8e77fe640 1 --2- 172.21.15.70:0/1114580096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853a4ca0 0x5643853a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.064+0000 7fa8e77fe640 1 -- 172.21.15.70:0/1114580096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 msgr2=0x5643853adc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:38.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.064+0000 7fa8e77fe640 1 --2- 172.21.15.70:0/1114580096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853adc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.064+0000 7fa8e77fe640 1 -- 172.21.15.70:0/1114580096 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564384fd50f0 con 0x5643853b4870 2024-04-01T01:41:38.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.064+0000 7fa8e67fc640 1 --2- 172.21.15.70:0/1114580096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853adc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:41:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.064+0000 7fa8e77fe640 1 --2- 172.21.15.70:0/1114580096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853b4870 0x5643853ae180 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa8d802c0d0 tx=0x7fa8d8034ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61ff236762df9d89 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.065+0000 7fa8ccff9640 1 -- 172.21.15.70:0/1114580096 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8048040 con 0x5643853b4870 2024-04-01T01:41:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.065+0000 7fa8ccff9640 1 -- 172.21.15.70:0/1114580096 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8d8051070 con 0x5643853b4870 2024-04-01T01:41:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.065+0000 7fa8ccff9640 1 -- 172.21.15.70:0/1114580096 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8039070 con 0x5643853b4870 2024-04-01T01:41:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.065+0000 7fa8effce740 1 -- 172.21.15.70:0/1114580096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853b4870 msgr2=0x5643853ae180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.065+0000 7fa8effce740 1 --2- 172.21.15.70:0/1114580096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853b4870 0x5643853ae180 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa8d802c0d0 tx=0x7fa8d8034ac0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.066+0000 7fa8effce740 1 -- 172.21.15.70:0/1114580096 shutdown_connections 2024-04-01T01:41:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.066+0000 7fa8effce740 1 --2- 172.21.15.70:0/1114580096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853b4870 0x5643853ae180 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.066+0000 7fa8effce740 1 --2- 172.21.15.70:0/1114580096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853adc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.066+0000 7fa8effce740 1 --2- 172.21.15.70:0/1114580096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853a4ca0 0x5643853a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.066+0000 7fa8effce740 1 -- 172.21.15.70:0/1114580096 >> 172.21.15.70:0/1114580096 conn(0x5643853a0210 msgr2=0x5643853a6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.066+0000 7fa8effce740 1 -- 172.21.15.70:0/1114580096 shutdown_connections 2024-04-01T01:41:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.066+0000 7fa8effce740 1 -- 172.21.15.70:0/1114580096 wait complete. 2024-04-01T01:41:38.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.067+0000 7fa8effce740 1 -- start start 2024-04-01T01:41:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.067+0000 7fa8effce740 1 Processor -- start 2024-04-01T01:41:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.067+0000 7fa8effce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a4ca0 0x5643853cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.067+0000 7fa8e6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a4ca0 0x5643853cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.068+0000 7fa8e6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a4ca0 0x5643853cda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48578/0 (socket says 172.21.15.70:48578) 2024-04-01T01:41:38.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.068+0000 7fa8e6ffd640 1 -- 172.21.15.70:0/608672240 learned_addr learned my addr 172.21.15.70:0/608672240 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.068+0000 7fa8effce740 1 --2- 172.21.15.70:0/608672240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.068+0000 7fa8effce740 1 --2- 172.21.15.70:0/608672240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853b4870 0x56438539c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.068+0000 7fa8effce740 1 -- 172.21.15.70:0/608672240 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e807b750 con 0x5643853aa800 2024-04-01T01:41:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.068+0000 7fa8effce740 1 -- 172.21.15.70:0/608672240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e807b080 con 0x5643853a4ca0 2024-04-01T01:41:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.068+0000 7fa8effce740 1 -- 172.21.15.70:0/608672240 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e80782c0 con 0x5643853b4870 2024-04-01T01:41:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.068+0000 7fa8e67fc640 1 --2- 172.21.15.70:0/608672240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.068+0000 7fa8effce740 10 client.0 _ll_register_callbacks cb 0x7fa8dc0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.068+0000 7fa8e77fe640 1 --2- 172.21.15.70:0/608672240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853b4870 0x56438539c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.068+0000 7fa8e6ffd640 1 -- 172.21.15.70:0/608672240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853b4870 msgr2=0x56438539c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.068+0000 7fa8e6ffd640 1 --2- 172.21.15.70:0/608672240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853b4870 0x56438539c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.068+0000 7fa8e6ffd640 1 -- 172.21.15.70:0/608672240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 msgr2=0x5643853cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.068+0000 7fa8e6ffd640 1 --2- 172.21.15.70:0/608672240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.068+0000 7fa8e6ffd640 1 -- 172.21.15.70:0/608672240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56438539c900 con 0x5643853a4ca0 2024-04-01T01:41:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.069+0000 7fa8e6ffd640 1 --2- 172.21.15.70:0/608672240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a4ca0 0x5643853cda00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00bd7d0 tx=0x7fa8e00bd740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a04d2aef534967b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:38.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.069+0000 7fa8caff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:38.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.071+0000 7fa8caff5640 1 -- 172.21.15.70:0/608672240 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e00bdf40 con 0x5643853a4ca0 2024-04-01T01:41:38.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.071+0000 7fa8caff5640 1 -- 172.21.15.70:0/608672240 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa8e00373c0 con 0x5643853a4ca0 2024-04-01T01:41:38.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.071+0000 7fa8effce740 1 -- 172.21.15.70:0/608672240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8d8051070 con 0x5643853a4ca0 2024-04-01T01:41:38.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.072+0000 7fa8caff5640 1 -- 172.21.15.70:0/608672240 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa8e00bd840 con 0x5643853a4ca0 2024-04-01T01:41:38.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.072+0000 7fa8effce740 10 client.27941 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:38.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.073+0000 7fa8effce740 1 -- 172.21.15.70:0/608672240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56438539dc30 con 0x5643853a4ca0 2024-04-01T01:41:38.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.073+0000 7fa8caff5640 1 -- 172.21.15.70:0/608672240 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8d8051070 con 0x5643853a4ca0 2024-04-01T01:41:38.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.073+0000 7fa8caff5640 1 -- 172.21.15.70:0/608672240 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fa8e000b040 con 0x5643853a4ca0 2024-04-01T01:41:38.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.073+0000 7fa8effce740 10 client.27941 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:38.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.073+0000 7fa8effce740 10 client.27941 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:38.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.073+0000 7fa8effce740 20 client.27941 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:38.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.073+0000 7fa8effce740 1 --2- 172.21.15.70:0/608672240 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56438539e210 0x56438539e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:38.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.074+0000 7fa8effce740 4 client.27941 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:41:38.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.074+0000 7fa8effce740 1 -- 172.21.15.70:0/608672240 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564385286540 con 0x56438539e210 2024-04-01T01:41:38.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.074+0000 7fa8e67fc640 1 --2- 172.21.15.70:0/608672240 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56438539e210 0x56438539e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:38.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.075+0000 7fa8caff5640 10 client.27941 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:38.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.075+0000 7fa8e67fc640 1 --2- 172.21.15.70:0/608672240 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56438539e210 0x56438539e620 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36c12792ab1664f5 server_cookie=9eefe696b45a5b39 in_seq=0 out_seq=0 2024-04-01T01:41:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.076+0000 7fa8caff5640 1 -- 172.21.15.70:0/608672240 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564385286540 con 0x56438539e210 2024-04-01T01:41:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.076+0000 7fa8caff5640 10 client.27941 handle_command_reply: tid=0 2024-04-01T01:41:38.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.076+0000 7fa8effce740 1 client.27941 shutdown 2024-04-01T01:41:38.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.079+0000 7fa8effce740 1 -- 172.21.15.70:0/608672240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a4ca0 msgr2=0x5643853cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:38.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.079+0000 7fa8effce740 1 --2- 172.21.15.70:0/608672240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a4ca0 0x5643853cda00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00bd7d0 tx=0x7fa8e00bd740 comp rx=0 tx=0).stop 2024-04-01T01:41:38.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.080+0000 7fa8effce740 1 -- 172.21.15.70:0/608672240 shutdown_connections 2024-04-01T01:41:38.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.080+0000 7fa8effce740 1 --2- 172.21.15.70:0/608672240 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56438539e210 0x56438539e620 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.080+0000 7fa8effce740 1 --2- 172.21.15.70:0/608672240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643853b4870 0x56438539c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.080+0000 7fa8caff5640 0 client.27941 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:38.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.081+0000 7fa8effce740 1 --2- 172.21.15.70:0/608672240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643853aa800 0x5643853cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.081+0000 7fa8effce740 1 --2- 172.21.15.70:0/608672240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643853a4ca0 0x5643853cda00 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.081+0000 7fa8effce740 1 -- 172.21.15.70:0/608672240 >> 172.21.15.70:0/608672240 conn(0x5643853a0210 msgr2=0x5643853adc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:38.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.081+0000 7fa8effce740 1 -- 172.21.15.70:0/608672240 shutdown_connections 2024-04-01T01:41:38.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.081+0000 7fa8effce740 1 -- 172.21.15.70:0/608672240 wait complete. 2024-04-01T01:41:38.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.081+0000 7fa8effce740 20 client.27941 trim_cache size 0 max 16384 2024-04-01T01:41:38.081 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:41:38.082 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:41:38.082 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:41:38.082 INFO:teuthology.orchestra.run.smithi070.stdout: "1b05d11c-4705-4e58-8fb0-0a13c9732569": { 2024-04-01T01:41:38.082 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:41:38.082 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "1b05d11c-4705-4e58-8fb0-0a13c9732569", 2024-04-01T01:41:38.082 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:41:38.082 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:41:38.082 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:41:38.082 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:41:38.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.084+0000 7fa8e7fff640 1 -- 172.21.15.70:0/747834808 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8d00a25a0 msgr2=0x7fa8d00a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:38.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.084+0000 7fa8e7fff640 1 --2- 172.21.15.70:0/747834808 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8d00a25a0 0x7fa8d00a4a60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8034b30 tx=0x7fa8d802e040 comp rx=0 tx=0).stop 2024-04-01T01:41:38.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.084+0000 7fa8e7fff640 1 -- 172.21.15.70:0/747834808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e8072be0 msgr2=0x7fa8e81a2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:38.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.084+0000 7fa8e7fff640 1 --2- 172.21.15.70:0/747834808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e8072be0 0x7fa8e81a2220 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa8e000efb0 tx=0x7fa8e000c800 comp rx=0 tx=0).stop 2024-04-01T01:41:38.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.085+0000 7fa8e7fff640 1 -- 172.21.15.70:0/747834808 shutdown_connections 2024-04-01T01:41:38.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.085+0000 7fa8e7fff640 1 --2- 172.21.15.70:0/747834808 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8d00a25a0 0x7fa8d00a4a60 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.085+0000 7fa8e7fff640 1 --2- 172.21.15.70:0/747834808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e81097e0 0x7fa8e81a97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.085+0000 7fa8e7fff640 1 --2- 172.21.15.70:0/747834808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8e8108e20 0x7fa8e81a2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.085+0000 7fa8e7fff640 1 --2- 172.21.15.70:0/747834808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8e8072be0 0x7fa8e81a2220 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:38.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.085+0000 7fa8e7fff640 1 -- 172.21.15.70:0/747834808 >> 172.21.15.70:0/747834808 conn(0x7fa8e806e630 msgr2=0x7fa8e8070df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:38.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.085+0000 7fa8e7fff640 1 -- 172.21.15.70:0/747834808 shutdown_connections 2024-04-01T01:41:38.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:41:38.085+0000 7fa8e7fff640 1 -- 172.21.15.70:0/747834808 wait complete. 2024-04-01T01:41:38.107 INFO:tasks.cephfs.filesystem:scrub status for tag:1b05d11c-4705-4e58-8fb0-0a13c9732569 - {'path': '/', 'tag': '1b05d11c-4705-4e58-8fb0-0a13c9732569', 'options': 'recursive,force'} 2024-04-01T01:41:38.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:38 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:41:38.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:38 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:41:38.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:38 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:41:38.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:38 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:41:38.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:38 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:41:38.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:38 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:41:38.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:38 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:41:38.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:38 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:41:38.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:38 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:41:38.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:38 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:41:38.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:38 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:41:38.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:38 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:41:39.478 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:39 smithi081 ceph-mon[35142]: pgmap v604: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 8.1 MiB/s wr, 896 op/s 2024-04-01T01:41:39.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:39 smithi189 ceph-mon[35753]: pgmap v604: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 8.1 MiB/s wr, 896 op/s 2024-04-01T01:41:39.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:39 smithi070 ceph-mon[32179]: pgmap v604: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 8.1 MiB/s wr, 896 op/s 2024-04-01T01:41:41.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 415.0 s, 861.3 tps, lat 113.808 ms stddev 75.373 2024-04-01T01:41:41.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:41 smithi189 ceph-mon[35753]: pgmap v605: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 7.5 MiB/s wr, 840 op/s 2024-04-01T01:41:41.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:41 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:41:42.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:41 smithi070 ceph-mon[32179]: pgmap v605: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 7.5 MiB/s wr, 840 op/s 2024-04-01T01:41:42.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:41 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:41:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:41 smithi081 ceph-mon[35142]: pgmap v605: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 7.5 MiB/s wr, 840 op/s 2024-04-01T01:41:42.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:41 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:41:43.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:42 smithi189 ceph-mon[35753]: pgmap v606: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 8.6 MiB/s wr, 999 op/s 2024-04-01T01:41:43.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:42 smithi070 ceph-mon[32179]: pgmap v606: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 8.6 MiB/s wr, 999 op/s 2024-04-01T01:41:43.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:42 smithi081 ceph-mon[35142]: pgmap v606: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 8.6 MiB/s wr, 999 op/s 2024-04-01T01:41:45.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:44 smithi189 ceph-mon[35753]: pgmap v607: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 7.5 MiB/s wr, 885 op/s 2024-04-01T01:41:45.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:44 smithi070 ceph-mon[32179]: pgmap v607: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 7.5 MiB/s wr, 885 op/s 2024-04-01T01:41:45.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:44 smithi081 ceph-mon[35142]: pgmap v607: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 7.5 MiB/s wr, 885 op/s 2024-04-01T01:41:46.895 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 420.0 s, 865.6 tps, lat 116.974 ms stddev 130.575 2024-04-01T01:41:47.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:46 smithi189 ceph-mon[35753]: pgmap v608: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 8.2 MiB/s wr, 970 op/s 2024-04-01T01:41:47.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:46 smithi070 ceph-mon[32179]: pgmap v608: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 8.2 MiB/s wr, 970 op/s 2024-04-01T01:41:47.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:46 smithi081 ceph-mon[35142]: pgmap v608: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 8.2 MiB/s wr, 970 op/s 2024-04-01T01:41:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.581+0000 7f9e756a0640 1 -- 172.21.15.70:0/3884342471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e680946a0 msgr2=0x7f9e680951a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.581+0000 7f9e756a0640 1 --2- 172.21.15.70:0/3884342471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e680946a0 0x7f9e680951a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c009860 tx=0x7f9e5c02bb70 comp rx=0 tx=0).stop 2024-04-01T01:41:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.582+0000 7f9e756a0640 1 -- 172.21.15.70:0/3884342471 shutdown_connections 2024-04-01T01:41:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.582+0000 7f9e756a0640 1 --2- 172.21.15.70:0/3884342471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e680956e0 0x7f9e68095b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.582+0000 7f9e756a0640 1 --2- 172.21.15.70:0/3884342471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e680946a0 0x7f9e680951a0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.582+0000 7f9e756a0640 1 --2- 172.21.15.70:0/3884342471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e68093d40 0x7f9e68094160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.582+0000 7f9e756a0640 1 -- 172.21.15.70:0/3884342471 >> 172.21.15.70:0/3884342471 conn(0x7f9e6808f500 msgr2=0x7f9e68091960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:47.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.584+0000 7f9e756a0640 1 -- 172.21.15.70:0/3884342471 shutdown_connections 2024-04-01T01:41:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.586+0000 7f9e756a0640 1 -- 172.21.15.70:0/3884342471 wait complete. 2024-04-01T01:41:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.587+0000 7f9e756a0640 1 Processor -- start 2024-04-01T01:41:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.587+0000 7f9e756a0640 1 -- start start 2024-04-01T01:41:47.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.587+0000 7f9e756a0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e68093d40 0x7f9e68132660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.587+0000 7f9e756a0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e680946a0 0x7f9e68132ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.587+0000 7f9e756a0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e680956e0 0x7f9e68139ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.587+0000 7f9e756a0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9e680a18c0 con 0x7f9e680946a0 2024-04-01T01:41:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.587+0000 7f9e756a0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9e680a1740 con 0x7f9e68093d40 2024-04-01T01:41:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.587+0000 7f9e756a0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e680a1a40 con 0x7f9e680956e0 2024-04-01T01:41:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.587+0000 7f9e6ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e68093d40 0x7f9e68132660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.588+0000 7f9e6ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e68093d40 0x7f9e68132660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60996/0 (socket says 172.21.15.70:60996) 2024-04-01T01:41:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.588+0000 7f9e6ffff640 1 -- 172.21.15.70:0/581890614 learned_addr learned my addr 172.21.15.70:0/581890614 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.588+0000 7f9e6f7fe640 1 --2- 172.21.15.70:0/581890614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e680946a0 0x7f9e68132ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.588+0000 7f9e74e9f640 1 --2- 172.21.15.70:0/581890614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e680956e0 0x7f9e68139ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.588+0000 7f9e6ffff640 1 -- 172.21.15.70:0/581890614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e680956e0 msgr2=0x7f9e68139ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.588+0000 7f9e6ffff640 1 --2- 172.21.15.70:0/581890614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e680956e0 0x7f9e68139ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.588+0000 7f9e6ffff640 1 -- 172.21.15.70:0/581890614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e680946a0 msgr2=0x7f9e68132ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.588+0000 7f9e6ffff640 1 --2- 172.21.15.70:0/581890614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e680946a0 0x7f9e68132ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.588+0000 7f9e6ffff640 1 -- 172.21.15.70:0/581890614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e6813a210 con 0x7f9e68093d40 2024-04-01T01:41:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.588+0000 7f9e6ffff640 1 --2- 172.21.15.70:0/581890614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e68093d40 0x7f9e68132660 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9e6400e9f0 tx=0x7f9e6400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.589+0000 7f9e6d7fa640 1 -- 172.21.15.70:0/581890614 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6400ccc0 con 0x7f9e68093d40 2024-04-01T01:41:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.589+0000 7f9e756a0640 1 -- 172.21.15.70:0/581890614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e6813a500 con 0x7f9e68093d40 2024-04-01T01:41:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.589+0000 7f9e6d7fa640 1 -- 172.21.15.70:0/581890614 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e640035f0 con 0x7f9e68093d40 2024-04-01T01:41:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.589+0000 7f9e756a0640 1 -- 172.21.15.70:0/581890614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e6813aa70 con 0x7f9e68093d40 2024-04-01T01:41:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.589+0000 7f9e6d7fa640 1 -- 172.21.15.70:0/581890614 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6400ce60 con 0x7f9e68093d40 2024-04-01T01:41:47.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.592+0000 7f9e795c3740 1 Processor -- start 2024-04-01T01:41:47.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.592+0000 7f9e795c3740 1 -- start start 2024-04-01T01:41:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.593+0000 7f9e6d7fa640 1 -- 172.21.15.70:0/581890614 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f9e640037a0 con 0x7f9e68093d40 2024-04-01T01:41:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.593+0000 7f9e795c3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c2ae0 0x562a754c2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.593+0000 7f9e795c3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c34d0 0x562a754c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.593+0000 7f9e795c3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c8640 0x562a754caad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.593+0000 7f9e795c3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562a754268f0 con 0x562a754c34d0 2024-04-01T01:41:47.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.593+0000 7f9e795c3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562a753cf8d0 con 0x562a754c2ae0 2024-04-01T01:41:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.593+0000 7f9e795c3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562a75422990 con 0x562a754c8640 2024-04-01T01:41:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.593+0000 7f9e6ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c34d0 0x562a754c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.593+0000 7f9e74e9f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c2ae0 0x562a754c2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.593+0000 7f9e74e9f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c2ae0 0x562a754c2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:32768/0 (socket says 172.21.15.70:32768) 2024-04-01T01:41:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.593+0000 7f9e74e9f640 1 -- 172.21.15.70:0/1299764269 learned_addr learned my addr 172.21.15.70:0/1299764269 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.594+0000 7f9e6f7fe640 1 --2- 172.21.15.70:0/1299764269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c8640 0x562a754caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.594+0000 7f9e6f7fe640 1 -- 172.21.15.70:0/1299764269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c2ae0 msgr2=0x562a754c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.594+0000 7f9e6f7fe640 1 --2- 172.21.15.70:0/1299764269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c2ae0 0x562a754c2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.594+0000 7f9e6f7fe640 1 -- 172.21.15.70:0/1299764269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c34d0 msgr2=0x562a754c7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.594+0000 7f9e6f7fe640 1 --2- 172.21.15.70:0/1299764269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c34d0 0x562a754c7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.594+0000 7f9e6f7fe640 1 -- 172.21.15.70:0/1299764269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a7513b210 con 0x562a754c8640 2024-04-01T01:41:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.595+0000 7f9e6d7fa640 1 --2- 172.21.15.70:0/581890614 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e600a25a0 0x7f9e600a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.595+0000 7f9e6d7fa640 1 -- 172.21.15.70:0/581890614 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f9e640cede0 con 0x7f9e68093d40 2024-04-01T01:41:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.596+0000 7f9e6f7fe640 1 --2- 172.21.15.70:0/1299764269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c8640 0x562a754caad0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c037040 tx=0x7f9e5c034440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=869e9b3a1349e265 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.596+0000 7f9e5a7fc640 1 -- 172.21.15.70:0/1299764269 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c039040 con 0x562a754c8640 2024-04-01T01:41:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.598+0000 7f9e5a7fc640 1 -- 172.21.15.70:0/1299764269 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e5c043070 con 0x562a754c8640 2024-04-01T01:41:47.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.599+0000 7f9e74e9f640 1 --2- 172.21.15.70:0/581890614 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e600a25a0 0x7f9e600a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:47.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.599+0000 7f9e795c3740 1 -- 172.21.15.70:0/1299764269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c8640 msgr2=0x562a754caad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:47.599 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.599+0000 7f9e795c3740 1 --2- 172.21.15.70:0/1299764269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c8640 0x562a754caad0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c037040 tx=0x7f9e5c034440 comp rx=0 tx=0).stop 2024-04-01T01:41:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.601+0000 7f9e74e9f640 1 --2- 172.21.15.70:0/581890614 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e600a25a0 0x7f9e600a4a60 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9e70050db0 tx=0x7f9e70067d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.601+0000 7f9e795c3740 1 -- 172.21.15.70:0/1299764269 shutdown_connections 2024-04-01T01:41:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.601+0000 7f9e795c3740 1 --2- 172.21.15.70:0/1299764269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c8640 0x562a754caad0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.601+0000 7f9e795c3740 1 --2- 172.21.15.70:0/1299764269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c34d0 0x562a754c7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.601+0000 7f9e795c3740 1 --2- 172.21.15.70:0/1299764269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c2ae0 0x562a754c2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.601+0000 7f9e795c3740 1 -- 172.21.15.70:0/1299764269 >> 172.21.15.70:0/1299764269 conn(0x562a754be050 msgr2=0x562a754be450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.601+0000 7f9e795c3740 1 -- 172.21.15.70:0/1299764269 shutdown_connections 2024-04-01T01:41:47.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.602+0000 7f9e795c3740 1 -- 172.21.15.70:0/1299764269 wait complete. 2024-04-01T01:41:47.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.602+0000 7f9e795c3740 1 -- start start 2024-04-01T01:41:47.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.602+0000 7f9e795c3740 1 Processor -- start 2024-04-01T01:41:47.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.603+0000 7f9e795c3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c2ae0 0x562a754ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.603+0000 7f9e795c3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c34d0 0x562a754cd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.603+0000 7f9e795c3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c8640 0x562a754d4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.603+0000 7f9e795c3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562a754cb630 con 0x562a754c34d0 2024-04-01T01:41:47.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.603+0000 7f9e795c3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562a753c4f80 con 0x562a754c2ae0 2024-04-01T01:41:47.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.603+0000 7f9e795c3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562a754cb7b0 con 0x562a754c8640 2024-04-01T01:41:47.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.603+0000 7f9e795c3740 10 client.0 _ll_register_callbacks cb 0x562a7541d290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:47.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.603+0000 7f9e6ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c2ae0 0x562a754ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:47.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.603+0000 7f9e6ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c2ae0 0x562a754ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:32772/0 (socket says 172.21.15.70:32772) 2024-04-01T01:41:47.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.603+0000 7f9e6ffff640 1 -- 172.21.15.70:0/2616368225 learned_addr learned my addr 172.21.15.70:0/2616368225 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:47.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.603+0000 7f9e74e9f640 1 --2- 172.21.15.70:0/2616368225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c8640 0x562a754d4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:47.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.604+0000 7f9e6ffff640 1 -- 172.21.15.70:0/2616368225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c8640 msgr2=0x562a754d4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:47.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.604+0000 7f9e6ffff640 1 --2- 172.21.15.70:0/2616368225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c8640 0x562a754d4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.604+0000 7f9e6ffff640 1 -- 172.21.15.70:0/2616368225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c34d0 msgr2=0x562a754cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:47.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.604+0000 7f9e6ffff640 1 --2- 172.21.15.70:0/2616368225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c34d0 0x562a754cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.604+0000 7f9e6ffff640 1 -- 172.21.15.70:0/2616368225 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a75207620 con 0x562a754c2ae0 2024-04-01T01:41:47.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.604+0000 7f9e74e9f640 1 --2- 172.21.15.70:0/2616368225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c8640 0x562a754d4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:41:47.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.604+0000 7f9e6ffff640 1 --2- 172.21.15.70:0/2616368225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c2ae0 0x562a754ccd40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9e640a3950 tx=0x7f9e64097710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c4a753987831043 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:47.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.604+0000 7f9e4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:47.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.604+0000 7f9e4f7fe640 1 -- 172.21.15.70:0/2616368225 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6409add0 con 0x562a754c2ae0 2024-04-01T01:41:47.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.604+0000 7f9e795c3740 1 -- 172.21.15.70:0/2616368225 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e5c043070 con 0x562a754c2ae0 2024-04-01T01:41:47.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.604+0000 7f9e4f7fe640 1 -- 172.21.15.70:0/2616368225 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f9e6409f120 con 0x562a754c2ae0 2024-04-01T01:41:47.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.605+0000 7f9e4f7fe640 1 -- 172.21.15.70:0/2616368225 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9e640b9050 con 0x562a754c2ae0 2024-04-01T01:41:47.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.605+0000 7f9e795c3740 10 client.27953 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:47.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.605+0000 7f9e795c3740 1 -- 172.21.15.70:0/2616368225 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562a754d73a0 con 0x562a754c2ae0 2024-04-01T01:41:47.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.606+0000 7f9e4f7fe640 1 -- 172.21.15.70:0/2616368225 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e5c043070 con 0x562a754c2ae0 2024-04-01T01:41:47.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.606+0000 7f9e4f7fe640 1 -- 172.21.15.70:0/2616368225 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f9e640971c0 con 0x562a754c2ae0 2024-04-01T01:41:47.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.606+0000 7f9e795c3740 10 client.27953 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:47.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.606+0000 7f9e795c3740 10 client.27953 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:47.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.606+0000 7f9e795c3740 20 client.27953 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:47.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.606+0000 7f9e795c3740 1 --2- 172.21.15.70:0/2616368225 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562a754d8f40 0x562a754db400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.606+0000 7f9e795c3740 4 client.27953 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:41:47.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.606+0000 7f9e795c3740 1 -- 172.21.15.70:0/2616368225 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a7543a120 con 0x562a754d8f40 2024-04-01T01:41:47.608 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.607+0000 7f9e6f7fe640 1 --2- 172.21.15.70:0/2616368225 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562a754d8f40 0x562a754db400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.613+0000 7f9e6f7fe640 1 --2- 172.21.15.70:0/2616368225 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562a754d8f40 0x562a754db400 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b6d8a16fb561a9f server_cookie=f5eb3a22f4c28581 in_seq=0 out_seq=0 2024-04-01T01:41:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.613+0000 7f9e4f7fe640 10 client.27953 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.620+0000 7f9e4f7fe640 1 -- 172.21.15.70:0/2616368225 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562a7543a120 con 0x562a754d8f40 2024-04-01T01:41:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.620+0000 7f9e4f7fe640 10 client.27953 handle_command_reply: tid=0 2024-04-01T01:41:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.621+0000 7f9e795c3740 1 client.27953 shutdown 2024-04-01T01:41:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.621+0000 7f9e795c3740 1 -- 172.21.15.70:0/2616368225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c2ae0 msgr2=0x562a754ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.622+0000 7f9e795c3740 1 --2- 172.21.15.70:0/2616368225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c2ae0 0x562a754ccd40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9e640a3950 tx=0x7f9e64097710 comp rx=0 tx=0).stop 2024-04-01T01:41:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.622+0000 7f9e795c3740 1 -- 172.21.15.70:0/2616368225 shutdown_connections 2024-04-01T01:41:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.622+0000 7f9e795c3740 1 --2- 172.21.15.70:0/2616368225 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562a754d8f40 0x562a754db400 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.622+0000 7f9e795c3740 1 --2- 172.21.15.70:0/2616368225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c8640 0x562a754d4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.622+0000 7f9e795c3740 1 --2- 172.21.15.70:0/2616368225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c34d0 0x562a754cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.622+0000 7f9e795c3740 1 --2- 172.21.15.70:0/2616368225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c2ae0 0x562a754ccd40 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.622+0000 7f9e795c3740 1 -- 172.21.15.70:0/2616368225 >> 172.21.15.70:0/2616368225 conn(0x562a754be050 msgr2=0x562a754bd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.623+0000 7f9e4f7fe640 0 client.27953 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.628+0000 7f9e795c3740 1 -- 172.21.15.70:0/2616368225 shutdown_connections 2024-04-01T01:41:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.628+0000 7f9e795c3740 1 -- 172.21.15.70:0/2616368225 wait complete. 2024-04-01T01:41:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.629+0000 7f9e795c3740 20 client.27953 trim_cache size 0 max 16384 2024-04-01T01:41:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.659+0000 7f9e795c3740 1 Processor -- start 2024-04-01T01:41:47.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.660+0000 7f9e795c3740 1 -- start start 2024-04-01T01:41:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.660+0000 7f9e795c3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c2ae0 0x562a754be430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.660+0000 7f9e795c3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c34d0 0x562a754d4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.660+0000 7f9e795c3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c8640 0x562a754c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.660+0000 7f9e795c3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9e68145030 con 0x562a754c34d0 2024-04-01T01:41:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.660+0000 7f9e795c3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9e68146b70 con 0x562a754c8640 2024-04-01T01:41:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.660+0000 7f9e74e9f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c8640 0x562a754c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.660+0000 7f9e74e9f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c8640 0x562a754c7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:32788/0 (socket says 172.21.15.70:32788) 2024-04-01T01:41:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.660+0000 7f9e74e9f640 1 -- 172.21.15.70:0/163034740 learned_addr learned my addr 172.21.15.70:0/163034740 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:47.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.660+0000 7f9e795c3740 1 -- 172.21.15.70:0/163034740 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e68145700 con 0x562a754c2ae0 2024-04-01T01:41:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.661+0000 7f9e74e9f640 1 -- 172.21.15.70:0/163034740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c2ae0 msgr2=0x562a754be430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:41:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.661+0000 7f9e74e9f640 1 --2- 172.21.15.70:0/163034740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c2ae0 0x562a754be430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.661+0000 7f9e74e9f640 1 -- 172.21.15.70:0/163034740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c34d0 msgr2=0x562a754d4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.661+0000 7f9e74e9f640 1 --2- 172.21.15.70:0/163034740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c34d0 0x562a754d4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.661+0000 7f9e74e9f640 1 -- 172.21.15.70:0/163034740 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a754d71c0 con 0x562a754c8640 2024-04-01T01:41:47.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.661+0000 7f9e74e9f640 1 --2- 172.21.15.70:0/163034740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c8640 0x562a754c7c80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9e7004f780 tx=0x7f9e70084000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5ffd04483890fc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.662+0000 7f9e59ffb640 1 -- 172.21.15.70:0/163034740 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7008a440 con 0x562a754c8640 2024-04-01T01:41:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.662+0000 7f9e59ffb640 1 -- 172.21.15.70:0/163034740 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e7008aa90 con 0x562a754c8640 2024-04-01T01:41:47.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.662+0000 7f9e59ffb640 1 -- 172.21.15.70:0/163034740 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e700977a0 con 0x562a754c8640 2024-04-01T01:41:47.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.663+0000 7f9e795c3740 1 -- 172.21.15.70:0/163034740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c8640 msgr2=0x562a754c7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:47.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.663+0000 7f9e795c3740 1 --2- 172.21.15.70:0/163034740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c8640 0x562a754c7c80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9e7004f780 tx=0x7f9e70084000 comp rx=0 tx=0).stop 2024-04-01T01:41:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.664+0000 7f9e795c3740 1 -- 172.21.15.70:0/163034740 shutdown_connections 2024-04-01T01:41:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.664+0000 7f9e795c3740 1 --2- 172.21.15.70:0/163034740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c8640 0x562a754c7c80 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.664+0000 7f9e795c3740 1 --2- 172.21.15.70:0/163034740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c34d0 0x562a754d4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.664+0000 7f9e795c3740 1 --2- 172.21.15.70:0/163034740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c2ae0 0x562a754be430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.665+0000 7f9e795c3740 1 -- 172.21.15.70:0/163034740 >> 172.21.15.70:0/163034740 conn(0x562a754be050 msgr2=0x562a754b8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.665+0000 7f9e795c3740 1 -- 172.21.15.70:0/163034740 shutdown_connections 2024-04-01T01:41:47.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.665+0000 7f9e795c3740 1 -- 172.21.15.70:0/163034740 wait complete. 2024-04-01T01:41:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.665+0000 7f9e795c3740 1 -- start start 2024-04-01T01:41:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.666+0000 7f9e795c3740 1 Processor -- start 2024-04-01T01:41:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.666+0000 7f9e795c3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c2ae0 0x562a754d0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.666+0000 7f9e795c3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c34d0 0x562a754d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.666+0000 7f9e795c3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c8640 0x562a7553d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.666+0000 7f9e795c3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9e681464a0 con 0x562a754c2ae0 2024-04-01T01:41:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.666+0000 7f9e795c3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9e68145dd0 con 0x562a754c34d0 2024-04-01T01:41:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.666+0000 7f9e795c3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e68143010 con 0x562a754c8640 2024-04-01T01:41:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.666+0000 7f9e795c3740 10 client.0 _ll_register_callbacks cb 0x7f9e640d2490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:41:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.666+0000 7f9e6f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c34d0 0x562a754d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:47.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.667+0000 7f9e6f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c34d0 0x562a754d13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:32794/0 (socket says 172.21.15.70:32794) 2024-04-01T01:41:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.667+0000 7f9e6f7fe640 1 -- 172.21.15.70:0/3202065957 learned_addr learned my addr 172.21.15.70:0/3202065957 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:41:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.667+0000 7f9e6ffff640 1 --2- 172.21.15.70:0/3202065957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c2ae0 0x562a754d0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:47.668 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.668+0000 7f9e6f7fe640 1 -- 172.21.15.70:0/3202065957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c8640 msgr2=0x562a7553d330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:41:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.668+0000 7f9e6f7fe640 1 --2- 172.21.15.70:0/3202065957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c8640 0x562a7553d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.668+0000 7f9e6f7fe640 1 -- 172.21.15.70:0/3202065957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c2ae0 msgr2=0x562a754d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:41:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.668+0000 7f9e6f7fe640 1 --2- 172.21.15.70:0/3202065957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c2ae0 0x562a754d0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.668+0000 7f9e6f7fe640 1 -- 172.21.15.70:0/3202065957 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a754d4420 con 0x562a754c34d0 2024-04-01T01:41:47.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.670+0000 7f9e6f7fe640 1 --2- 172.21.15.70:0/3202065957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c34d0 0x562a754d13a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c02f020 tx=0x7f9e5c034440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39d5beb498603c4c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:41:47.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.670+0000 7f9e4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:41:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.670+0000 7f9e4f7fe640 1 -- 172.21.15.70:0/3202065957 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c039040 con 0x562a754c34d0 2024-04-01T01:41:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.670+0000 7f9e795c3740 1 -- 172.21.15.70:0/3202065957 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e7008aa90 con 0x562a754c34d0 2024-04-01T01:41:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.671+0000 7f9e4f7fe640 1 -- 172.21.15.70:0/3202065957 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f9e5c034a80 con 0x562a754c34d0 2024-04-01T01:41:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.671+0000 7f9e4f7fe640 1 -- 172.21.15.70:0/3202065957 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9e5c034c80 con 0x562a754c34d0 2024-04-01T01:41:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.671+0000 7f9e795c3740 10 client.27959 fetch_fsmap learned FSMap version 31 2024-04-01T01:41:47.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.671+0000 7f9e795c3740 1 -- 172.21.15.70:0/3202065957 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562a7553e8c0 con 0x562a754c34d0 2024-04-01T01:41:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.671+0000 7f9e4f7fe640 1 -- 172.21.15.70:0/3202065957 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e7008aa90 con 0x562a754c34d0 2024-04-01T01:41:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.671+0000 7f9e4f7fe640 1 -- 172.21.15.70:0/3202065957 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f9e5c004040 con 0x562a754c34d0 2024-04-01T01:41:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.671+0000 7f9e795c3740 10 client.27959 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:41:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.671+0000 7f9e795c3740 10 client.27959 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:41:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.671+0000 7f9e795c3740 20 client.27959 populate_metadata read hostname 'smithi070' 2024-04-01T01:41:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.671+0000 7f9e795c3740 1 --2- 172.21.15.70:0/3202065957 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562a7553f290 0x562a755416b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:41:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.671+0000 7f9e795c3740 4 client.27959 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:41:47.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.671+0000 7f9e795c3740 1 -- 172.21.15.70:0/3202065957 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562a75433e30 con 0x562a7553f290 2024-04-01T01:41:47.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.674+0000 7f9e6ffff640 1 --2- 172.21.15.70:0/3202065957 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562a7553f290 0x562a755416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:41:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.674+0000 7f9e6ffff640 1 --2- 172.21.15.70:0/3202065957 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562a7553f290 0x562a755416b0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c40885e1070d9fc server_cookie=7c3c5052c68b78ac in_seq=0 out_seq=0 2024-04-01T01:41:47.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.675+0000 7f9e4f7fe640 10 client.27959 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.701+0000 7f9e4f7fe640 1 -- 172.21.15.70:0/3202065957 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562a75433e30 con 0x562a7553f290 2024-04-01T01:41:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.701+0000 7f9e4f7fe640 10 client.27959 handle_command_reply: tid=0 2024-04-01T01:41:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.701+0000 7f9e795c3740 1 client.27959 shutdown 2024-04-01T01:41:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.701+0000 7f9e795c3740 1 -- 172.21.15.70:0/3202065957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c34d0 msgr2=0x562a754d13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.701+0000 7f9e795c3740 1 --2- 172.21.15.70:0/3202065957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c34d0 0x562a754d13a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c02f020 tx=0x7f9e5c034440 comp rx=0 tx=0).stop 2024-04-01T01:41:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.701+0000 7f9e795c3740 1 -- 172.21.15.70:0/3202065957 shutdown_connections 2024-04-01T01:41:47.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.701+0000 7f9e795c3740 1 --2- 172.21.15.70:0/3202065957 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562a7553f290 0x562a755416b0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.701+0000 7f9e795c3740 1 --2- 172.21.15.70:0/3202065957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a754c8640 0x562a7553d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.701+0000 7f9e4f7fe640 0 client.27959 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:41:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.701+0000 7f9e795c3740 1 --2- 172.21.15.70:0/3202065957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a754c34d0 0x562a754d13a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.701+0000 7f9e795c3740 1 --2- 172.21.15.70:0/3202065957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a754c2ae0 0x562a754d0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.701+0000 7f9e795c3740 1 -- 172.21.15.70:0/3202065957 >> 172.21.15.70:0/3202065957 conn(0x562a754be050 msgr2=0x562a754be430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.701+0000 7f9e795c3740 1 -- 172.21.15.70:0/3202065957 shutdown_connections 2024-04-01T01:41:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.701+0000 7f9e795c3740 1 -- 172.21.15.70:0/3202065957 wait complete. 2024-04-01T01:41:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.702+0000 7f9e795c3740 20 client.27959 trim_cache size 0 max 16384 2024-04-01T01:41:47.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:41:47.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:41:47.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:41:47.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:41:47.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.703+0000 7f9e756a0640 1 -- 172.21.15.70:0/581890614 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e600a25a0 msgr2=0x7f9e600a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:47.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.703+0000 7f9e756a0640 1 --2- 172.21.15.70:0/581890614 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e600a25a0 0x7f9e600a4a60 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9e70050db0 tx=0x7f9e70067d30 comp rx=0 tx=0).stop 2024-04-01T01:41:47.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.703+0000 7f9e756a0640 1 -- 172.21.15.70:0/581890614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e68093d40 msgr2=0x7f9e68132660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:41:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.703+0000 7f9e756a0640 1 --2- 172.21.15.70:0/581890614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e68093d40 0x7f9e68132660 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9e6400e9f0 tx=0x7f9e6400eeb0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.703+0000 7f9e756a0640 1 -- 172.21.15.70:0/581890614 shutdown_connections 2024-04-01T01:41:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.703+0000 7f9e756a0640 1 --2- 172.21.15.70:0/581890614 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e600a25a0 0x7f9e600a4a60 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.703+0000 7f9e756a0640 1 --2- 172.21.15.70:0/581890614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e680956e0 0x7f9e68139ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.703+0000 7f9e756a0640 1 --2- 172.21.15.70:0/581890614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e680946a0 0x7f9e68132ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.703+0000 7f9e756a0640 1 --2- 172.21.15.70:0/581890614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e68093d40 0x7f9e68132660 secure :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9e6400e9f0 tx=0x7f9e6400eeb0 comp rx=0 tx=0).stop 2024-04-01T01:41:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.703+0000 7f9e756a0640 1 -- 172.21.15.70:0/581890614 >> 172.21.15.70:0/581890614 conn(0x7f9e6808f500 msgr2=0x7f9e68090fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:41:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.703+0000 7f9e756a0640 1 -- 172.21.15.70:0/581890614 shutdown_connections 2024-04-01T01:41:47.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:41:47.704+0000 7f9e756a0640 1 -- 172.21.15.70:0/581890614 wait complete. 2024-04-01T01:41:49.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:48 smithi189 ceph-mon[35753]: pgmap v609: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 7.9 MiB/s wr, 932 op/s 2024-04-01T01:41:49.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:48 smithi070 ceph-mon[32179]: pgmap v609: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 7.9 MiB/s wr, 932 op/s 2024-04-01T01:41:49.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:48 smithi081 ceph-mon[35142]: pgmap v609: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 7.9 MiB/s wr, 932 op/s 2024-04-01T01:41:51.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:50 smithi189 ceph-mon[35753]: pgmap v610: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 6.7 MiB/s wr, 829 op/s 2024-04-01T01:41:51.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:50 smithi070 ceph-mon[32179]: pgmap v610: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 6.7 MiB/s wr, 829 op/s 2024-04-01T01:41:51.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:50 smithi081 ceph-mon[35142]: pgmap v610: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 6.7 MiB/s wr, 829 op/s 2024-04-01T01:41:51.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 425.0 s, 886.6 tps, lat 112.530 ms stddev 94.408 2024-04-01T01:41:53.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:52 smithi189 ceph-mon[35753]: pgmap v611: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 8.0 MiB/s wr, 930 op/s 2024-04-01T01:41:53.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:52 smithi070 ceph-mon[32179]: pgmap v611: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 8.0 MiB/s wr, 930 op/s 2024-04-01T01:41:53.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:52 smithi081 ceph-mon[35142]: pgmap v611: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 8.0 MiB/s wr, 930 op/s 2024-04-01T01:41:55.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:54 smithi189 ceph-mon[35753]: pgmap v612: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.8 MiB/s wr, 767 op/s 2024-04-01T01:41:55.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:54 smithi070 ceph-mon[32179]: pgmap v612: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.8 MiB/s wr, 767 op/s 2024-04-01T01:41:55.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:54 smithi081 ceph-mon[35142]: pgmap v612: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.8 MiB/s wr, 767 op/s 2024-04-01T01:41:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 430.0 s, 923.4 tps, lat 108.291 ms stddev 101.553 2024-04-01T01:41:57.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:56 smithi189 ceph-mon[35753]: pgmap v613: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 8.5 MiB/s wr, 922 op/s 2024-04-01T01:41:57.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:56 smithi070 ceph-mon[32179]: pgmap v613: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 8.5 MiB/s wr, 922 op/s 2024-04-01T01:41:57.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:56 smithi081 ceph-mon[35142]: pgmap v613: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 8.5 MiB/s wr, 922 op/s 2024-04-01T01:41:59.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:41:58 smithi189 ceph-mon[35753]: pgmap v614: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.8 MiB/s wr, 886 op/s 2024-04-01T01:41:59.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:41:58 smithi070 ceph-mon[32179]: pgmap v614: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.8 MiB/s wr, 886 op/s 2024-04-01T01:41:59.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:41:58 smithi081 ceph-mon[35142]: pgmap v614: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.8 MiB/s wr, 886 op/s 2024-04-01T01:42:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.863+0000 7f62e2ffd640 1 -- 172.21.15.70:0/417562515 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d402c620 con 0x7f62e4108dc0 2024-04-01T01:42:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.864+0000 7f62e926b640 1 -- 172.21.15.70:0/417562515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e4108dc0 msgr2=0x7f62e4109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:00.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.864+0000 7f62e926b640 1 --2- 172.21.15.70:0/417562515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e4108dc0 0x7f62e4109240 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009900 tx=0x7f62d402bc80 comp rx=0 tx=0).stop 2024-04-01T01:42:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.865+0000 7f62e926b640 1 -- 172.21.15.70:0/417562515 shutdown_connections 2024-04-01T01:42:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.865+0000 7f62e926b640 1 --2- 172.21.15.70:0/417562515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62e4109780 0x7f62e4110010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.865+0000 7f62e926b640 1 --2- 172.21.15.70:0/417562515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e4108dc0 0x7f62e4109240 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.865+0000 7f62e926b640 1 --2- 172.21.15.70:0/417562515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62e4107bd0 0x7f62e4107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.865+0000 7f62e926b640 1 -- 172.21.15.70:0/417562515 >> 172.21.15.70:0/417562515 conn(0x7f62e4076b60 msgr2=0x7f62e4078fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.865+0000 7f62e926b640 1 -- 172.21.15.70:0/417562515 shutdown_connections 2024-04-01T01:42:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.865+0000 7f62e926b640 1 -- 172.21.15.70:0/417562515 wait complete. 2024-04-01T01:42:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.866+0000 7f62e926b640 1 Processor -- start 2024-04-01T01:42:00.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.866+0000 7f62e926b640 1 -- start start 2024-04-01T01:42:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.866+0000 7f62e926b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62e4107bd0 0x7f62e419de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.866+0000 7f62e926b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e4108dc0 0x7f62e419e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.866+0000 7f62e926b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62e4109780 0x7f62e41a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.867+0000 7f62e926b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62e41137c0 con 0x7f62e4109780 2024-04-01T01:42:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.867+0000 7f62e926b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f62e4113640 con 0x7f62e4108dc0 2024-04-01T01:42:00.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.867+0000 7f62e926b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f62e4113940 con 0x7f62e4107bd0 2024-04-01T01:42:00.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.867+0000 7f62e3fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62e4107bd0 0x7f62e419de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:00.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.867+0000 7f62e3fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62e4107bd0 0x7f62e419de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37756/0 (socket says 172.21.15.70:37756) 2024-04-01T01:42:00.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.867+0000 7f62e3fff640 1 -- 172.21.15.70:0/272924082 learned_addr learned my addr 172.21.15.70:0/272924082 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:00.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.867+0000 7f62e37fe640 1 --2- 172.21.15.70:0/272924082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e4108dc0 0x7f62e419e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:00.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.867+0000 7f62e8a6a640 1 --2- 172.21.15.70:0/272924082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62e4109780 0x7f62e41a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:00.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.867+0000 7f62e3fff640 1 -- 172.21.15.70:0/272924082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e4108dc0 msgr2=0x7f62e419e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:00.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.867+0000 7f62e3fff640 1 --2- 172.21.15.70:0/272924082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e4108dc0 0x7f62e419e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.867+0000 7f62e3fff640 1 -- 172.21.15.70:0/272924082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62e4109780 msgr2=0x7f62e41a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:00.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.867+0000 7f62e3fff640 1 --2- 172.21.15.70:0/272924082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62e4109780 0x7f62e41a5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.867+0000 7f62e3fff640 1 -- 172.21.15.70:0/272924082 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62e41a5a90 con 0x7f62e4107bd0 2024-04-01T01:42:00.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.868+0000 7f62e3fff640 1 --2- 172.21.15.70:0/272924082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62e4107bd0 0x7f62e419de70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f62dc00b9c0 tx=0x7f62dc00be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:00.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.869+0000 7f62e17fa640 1 -- 172.21.15.70:0/272924082 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc019070 con 0x7f62e4107bd0 2024-04-01T01:42:00.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.869+0000 7f62e17fa640 1 -- 172.21.15.70:0/272924082 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62dc00cba0 con 0x7f62e4107bd0 2024-04-01T01:42:00.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.869+0000 7f62e926b640 1 -- 172.21.15.70:0/272924082 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62e41a5d80 con 0x7f62e4107bd0 2024-04-01T01:42:00.870 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.869+0000 7f62e926b640 1 -- 172.21.15.70:0/272924082 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62e41a6260 con 0x7f62e4107bd0 2024-04-01T01:42:00.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.870+0000 7f62e17fa640 1 -- 172.21.15.70:0/272924082 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0125a0 con 0x7f62e4107bd0 2024-04-01T01:42:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.872+0000 7f62ed181740 1 Processor -- start 2024-04-01T01:42:00.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.873+0000 7f62e17fa640 1 -- 172.21.15.70:0/272924082 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f62dc01c070 con 0x7f62e4107bd0 2024-04-01T01:42:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.873+0000 7f62ed181740 1 -- start start 2024-04-01T01:42:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.873+0000 7f62ed181740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd176ae0 0x55bbfd176f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.873+0000 7f62ed181740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd1774d0 0x55bbfd17bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.873+0000 7f62ed181740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd17c640 0x55bbfd17ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.873+0000 7f62ed181740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bbfd0da8f0 con 0x55bbfd1774d0 2024-04-01T01:42:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.873+0000 7f62ed181740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55bbfd0838d0 con 0x55bbfd176ae0 2024-04-01T01:42:00.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.873+0000 7f62ed181740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bbfd0d6990 con 0x55bbfd17c640 2024-04-01T01:42:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.874+0000 7f62e8a6a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd176ae0 0x55bbfd176f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.874+0000 7f62e8a6a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd176ae0 0x55bbfd176f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:51192/0 (socket says 172.21.15.70:51192) 2024-04-01T01:42:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.874+0000 7f62e8a6a640 1 -- 172.21.15.70:0/3394280801 learned_addr learned my addr 172.21.15.70:0/3394280801 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.874+0000 7f62e3fff640 1 --2- 172.21.15.70:0/3394280801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd1774d0 0x55bbfd17bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.874+0000 7f62e8a6a640 1 -- 172.21.15.70:0/3394280801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd17c640 msgr2=0x55bbfd17ead0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:42:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.874+0000 7f62e8a6a640 1 --2- 172.21.15.70:0/3394280801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd17c640 0x55bbfd17ead0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.874+0000 7f62e8a6a640 1 -- 172.21.15.70:0/3394280801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd1774d0 msgr2=0x55bbfd17bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.874+0000 7f62e8a6a640 1 --2- 172.21.15.70:0/3394280801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd1774d0 0x55bbfd17bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.874+0000 7f62e8a6a640 1 -- 172.21.15.70:0/3394280801 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbfcdef210 con 0x55bbfd176ae0 2024-04-01T01:42:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.874+0000 7f62e17fa640 1 --2- 172.21.15.70:0/272924082 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62cc0a25e0 0x7f62cc0a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.875+0000 7f62e8a6a640 1 --2- 172.21.15.70:0/3394280801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd176ae0 0x55bbfd176f00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009c10 tx=0x7f62d80117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc6956465d9bd255 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.875+0000 7f62e3fff640 1 --2- 172.21.15.70:0/272924082 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62cc0a25e0 0x7f62cc0a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.875+0000 7f62ca7fc640 1 -- 172.21.15.70:0/3394280801 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8012450 con 0x55bbfd176ae0 2024-04-01T01:42:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.875+0000 7f62ca7fc640 1 -- 172.21.15.70:0/3394280801 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62d8012aa0 con 0x55bbfd176ae0 2024-04-01T01:42:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.875+0000 7f62ca7fc640 1 -- 172.21.15.70:0/3394280801 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d800f9b0 con 0x55bbfd176ae0 2024-04-01T01:42:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.875+0000 7f62e17fa640 1 -- 172.21.15.70:0/272924082 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f62dc0cdf60 con 0x7f62e4107bd0 2024-04-01T01:42:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.876+0000 7f62ed181740 1 -- 172.21.15.70:0/3394280801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd176ae0 msgr2=0x55bbfd176f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.876+0000 7f62ed181740 1 --2- 172.21.15.70:0/3394280801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd176ae0 0x55bbfd176f00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009c10 tx=0x7f62d80117b0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.876+0000 7f62e3fff640 1 --2- 172.21.15.70:0/272924082 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62cc0a25e0 0x7f62cc0a4aa0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0cba00 tx=0x7f62dc004b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.876+0000 7f62ed181740 1 -- 172.21.15.70:0/3394280801 shutdown_connections 2024-04-01T01:42:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.876+0000 7f62ed181740 1 --2- 172.21.15.70:0/3394280801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd17c640 0x55bbfd17ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.876+0000 7f62ed181740 1 --2- 172.21.15.70:0/3394280801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd1774d0 0x55bbfd17bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.876+0000 7f62ed181740 1 --2- 172.21.15.70:0/3394280801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd176ae0 0x55bbfd176f00 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.876+0000 7f62ed181740 1 -- 172.21.15.70:0/3394280801 >> 172.21.15.70:0/3394280801 conn(0x55bbfd172050 msgr2=0x55bbfd172450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.877+0000 7f62ed181740 1 -- 172.21.15.70:0/3394280801 shutdown_connections 2024-04-01T01:42:00.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.877+0000 7f62ed181740 1 -- 172.21.15.70:0/3394280801 wait complete. 2024-04-01T01:42:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.877+0000 7f62ed181740 1 -- start start 2024-04-01T01:42:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.878+0000 7f62ed181740 1 Processor -- start 2024-04-01T01:42:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.878+0000 7f62ed181740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd176ae0 0x55bbfd180df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.878+0000 7f62ed181740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd1774d0 0x55bbfd181330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.878+0000 7f62ed181740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd17c640 0x55bbfd188330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.878+0000 7f62ed181740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bbfd17f5f0 con 0x55bbfd1774d0 2024-04-01T01:42:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.878+0000 7f62ed181740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55bbfd078f80 con 0x55bbfd17c640 2024-04-01T01:42:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.878+0000 7f62ed181740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bbfd17f770 con 0x55bbfd176ae0 2024-04-01T01:42:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.878+0000 7f62ed181740 10 client.0 _ll_register_callbacks cb 0x55bbfd0d1290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.879+0000 7f62e8a6a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd17c640 0x55bbfd188330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.879+0000 7f62e8a6a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd17c640 0x55bbfd188330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:51200/0 (socket says 172.21.15.70:51200) 2024-04-01T01:42:00.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.879+0000 7f62e8a6a640 1 -- 172.21.15.70:0/2336026306 learned_addr learned my addr 172.21.15.70:0/2336026306 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:00.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.879+0000 7f62e8a6a640 1 --2- 172.21.15.70:0/2336026306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd176ae0 0x55bbfd180df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:00.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.879+0000 7f62e37fe640 1 --2- 172.21.15.70:0/2336026306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd1774d0 0x55bbfd181330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:00.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.879+0000 7f62e8a6a640 1 -- 172.21.15.70:0/2336026306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd176ae0 msgr2=0x55bbfd180df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:00.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.879+0000 7f62e8a6a640 1 --2- 172.21.15.70:0/2336026306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd176ae0 0x55bbfd180df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.879+0000 7f62e8a6a640 1 -- 172.21.15.70:0/2336026306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd1774d0 msgr2=0x55bbfd181330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:00.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.879+0000 7f62e8a6a640 1 --2- 172.21.15.70:0/2336026306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd1774d0 0x55bbfd181330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.879+0000 7f62e8a6a640 1 -- 172.21.15.70:0/2336026306 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbfcebb620 con 0x55bbfd17c640 2024-04-01T01:42:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.880+0000 7f62e8a6a640 1 --2- 172.21.15.70:0/2336026306 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd17c640 0x55bbfd188330 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009fb0 tx=0x7f62d8014040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=961d7e80fc44f9d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.880+0000 7f62c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.880+0000 7f62c37fe640 1 -- 172.21.15.70:0/2336026306 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d8012900 con 0x55bbfd17c640 2024-04-01T01:42:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.881+0000 7f62ed181740 1 -- 172.21.15.70:0/2336026306 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62d8012aa0 con 0x55bbfd17c640 2024-04-01T01:42:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.881+0000 7f62c37fe640 1 -- 172.21.15.70:0/2336026306 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f62d8005020 con 0x55bbfd17c640 2024-04-01T01:42:00.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.882+0000 7f62c37fe640 1 -- 172.21.15.70:0/2336026306 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f62d8036050 con 0x55bbfd17c640 2024-04-01T01:42:00.883 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.882+0000 7f62ed181740 10 client.27971 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:00.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.883+0000 7f62ed181740 1 -- 172.21.15.70:0/2336026306 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55bbfd19c370 con 0x55bbfd17c640 2024-04-01T01:42:00.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.883+0000 7f62c37fe640 1 -- 172.21.15.70:0/2336026306 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62d8012aa0 con 0x55bbfd17c640 2024-04-01T01:42:00.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.883+0000 7f62c37fe640 1 -- 172.21.15.70:0/2336026306 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f62d8026040 con 0x55bbfd17c640 2024-04-01T01:42:00.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.883+0000 7f62ed181740 10 client.27971 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:00.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.883+0000 7f62ed181740 10 client.27971 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:00.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.883+0000 7f62ed181740 20 client.27971 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:00.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.883+0000 7f62ed181740 1 --2- 172.21.15.70:0/2336026306 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bbfd19de50 0x55bbfd1a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.883+0000 7f62ed181740 4 client.27971 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:00.884 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.883+0000 7f62ed181740 1 -- 172.21.15.70:0/2336026306 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bbfd0ee120 con 0x55bbfd19de50 2024-04-01T01:42:00.885 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.884+0000 7f62e37fe640 1 --2- 172.21.15.70:0/2336026306 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bbfd19de50 0x55bbfd1a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:00.885 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.884+0000 7f62c37fe640 10 client.27971 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:00.886 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.886+0000 7f62e37fe640 1 --2- 172.21.15.70:0/2336026306 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bbfd19de50 0x55bbfd1a0290 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa05446ff78eeaf8 server_cookie=f0178cb173d14838 in_seq=0 out_seq=0 2024-04-01T01:42:00.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.886+0000 7f62c37fe640 1 -- 172.21.15.70:0/2336026306 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bbfd0ee120 con 0x55bbfd19de50 2024-04-01T01:42:00.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.886+0000 7f62c37fe640 10 client.27971 handle_command_reply: tid=0 2024-04-01T01:42:00.887 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.887+0000 7f62ed181740 1 client.27971 shutdown 2024-04-01T01:42:00.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.888+0000 7f62ed181740 1 -- 172.21.15.70:0/2336026306 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd17c640 msgr2=0x55bbfd188330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:00.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.888+0000 7f62ed181740 1 --2- 172.21.15.70:0/2336026306 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd17c640 0x55bbfd188330 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f62d8009fb0 tx=0x7f62d8014040 comp rx=0 tx=0).stop 2024-04-01T01:42:00.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.888+0000 7f62ed181740 1 -- 172.21.15.70:0/2336026306 shutdown_connections 2024-04-01T01:42:00.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.888+0000 7f62ed181740 1 --2- 172.21.15.70:0/2336026306 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bbfd19de50 0x55bbfd1a0290 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.888+0000 7f62ed181740 1 --2- 172.21.15.70:0/2336026306 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd17c640 0x55bbfd188330 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.888+0000 7f62ed181740 1 --2- 172.21.15.70:0/2336026306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd1774d0 0x55bbfd181330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.888+0000 7f62ed181740 1 --2- 172.21.15.70:0/2336026306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd176ae0 0x55bbfd180df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.888+0000 7f62ed181740 1 -- 172.21.15.70:0/2336026306 >> 172.21.15.70:0/2336026306 conn(0x55bbfd172050 msgr2=0x55bbfd171190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:00.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.888+0000 7f62c37fe640 0 client.27971 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:00.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.888+0000 7f62ed181740 1 -- 172.21.15.70:0/2336026306 shutdown_connections 2024-04-01T01:42:00.889 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.888+0000 7f62ed181740 1 -- 172.21.15.70:0/2336026306 wait complete. 2024-04-01T01:42:00.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.888+0000 7f62ed181740 20 client.27971 trim_cache size 0 max 16384 2024-04-01T01:42:00.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.919+0000 7f62ed181740 1 Processor -- start 2024-04-01T01:42:00.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.919+0000 7f62ed181740 1 -- start start 2024-04-01T01:42:00.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.919+0000 7f62ed181740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd176ae0 0x55bbfd172430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.919+0000 7f62ed181740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd1774d0 0x55bbfd188200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.919+0000 7f62e8a6a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd176ae0 0x55bbfd172430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.919+0000 7f62e8a6a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd176ae0 0x55bbfd172430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49368/0 (socket says 172.21.15.70:49368) 2024-04-01T01:42:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.919+0000 7f62e8a6a640 1 -- 172.21.15.70:0/1098476825 learned_addr learned my addr 172.21.15.70:0/1098476825 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.920+0000 7f62ed181740 1 --2- 172.21.15.70:0/1098476825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd17c640 0x55bbfd1a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.920+0000 7f62ed181740 1 -- 172.21.15.70:0/1098476825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62d000fde0 con 0x55bbfd176ae0 2024-04-01T01:42:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.920+0000 7f62ed181740 1 -- 172.21.15.70:0/1098476825 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f62d0011920 con 0x55bbfd1774d0 2024-04-01T01:42:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.920+0000 7f62ed181740 1 -- 172.21.15.70:0/1098476825 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f62d00104b0 con 0x55bbfd17c640 2024-04-01T01:42:00.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.920+0000 7f62e8a6a640 1 -- 172.21.15.70:0/1098476825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd17c640 msgr2=0x55bbfd1a5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.920+0000 7f62e8a6a640 1 --2- 172.21.15.70:0/1098476825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd17c640 0x55bbfd1a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.920+0000 7f62e8a6a640 1 -- 172.21.15.70:0/1098476825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd1774d0 msgr2=0x55bbfd188200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:42:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.920+0000 7f62e8a6a640 1 --2- 172.21.15.70:0/1098476825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd1774d0 0x55bbfd188200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.920+0000 7f62e8a6a640 1 -- 172.21.15.70:0/1098476825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbfd177d30 con 0x55bbfd176ae0 2024-04-01T01:42:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.922+0000 7f62e8a6a640 1 --2- 172.21.15.70:0/1098476825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd176ae0 0x55bbfd172430 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f62d8000c00 tx=0x7f62d8009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac12a265468a232 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:00.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.923+0000 7f62c9ffb640 1 -- 172.21.15.70:0/1098476825 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d803f070 con 0x55bbfd176ae0 2024-04-01T01:42:00.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.924+0000 7f62c9ffb640 1 -- 172.21.15.70:0/1098476825 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62d80095f0 con 0x55bbfd176ae0 2024-04-01T01:42:00.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.924+0000 7f62c9ffb640 1 -- 172.21.15.70:0/1098476825 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d80049d0 con 0x55bbfd176ae0 2024-04-01T01:42:00.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.925+0000 7f62ed181740 1 -- 172.21.15.70:0/1098476825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd176ae0 msgr2=0x55bbfd172430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:00.926 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.925+0000 7f62ed181740 1 --2- 172.21.15.70:0/1098476825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd176ae0 0x55bbfd172430 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f62d8000c00 tx=0x7f62d8009b90 comp rx=0 tx=0).stop 2024-04-01T01:42:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.926+0000 7f62ed181740 1 -- 172.21.15.70:0/1098476825 shutdown_connections 2024-04-01T01:42:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.926+0000 7f62ed181740 1 --2- 172.21.15.70:0/1098476825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd17c640 0x55bbfd1a5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.926+0000 7f62ed181740 1 --2- 172.21.15.70:0/1098476825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd1774d0 0x55bbfd188200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.926+0000 7f62ed181740 1 --2- 172.21.15.70:0/1098476825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd176ae0 0x55bbfd172430 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.927 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.926+0000 7f62ed181740 1 -- 172.21.15.70:0/1098476825 >> 172.21.15.70:0/1098476825 conn(0x55bbfd172050 msgr2=0x55bbfd17bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.929+0000 7f62ed181740 1 -- 172.21.15.70:0/1098476825 shutdown_connections 2024-04-01T01:42:00.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.931+0000 7f62ed181740 1 -- 172.21.15.70:0/1098476825 wait complete. 2024-04-01T01:42:00.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.931+0000 7f62ed181740 1 -- start start 2024-04-01T01:42:00.931 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.932+0000 7f62ed181740 1 Processor -- start 2024-04-01T01:42:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.932+0000 7f62ed181740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd176ae0 0x55bbfd188170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.932+0000 7f62ed181740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd1774d0 0x55bbfd184c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.932+0000 7f62ed181740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd17c640 0x55bbfd1851c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.932+0000 7f62e8a6a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd176ae0 0x55bbfd188170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.932+0000 7f62e8a6a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd176ae0 0x55bbfd188170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:51204/0 (socket says 172.21.15.70:51204) 2024-04-01T01:42:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.932+0000 7f62e8a6a640 1 -- 172.21.15.70:0/3649493025 learned_addr learned my addr 172.21.15.70:0/3649493025 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.933+0000 7f62ed181740 1 -- 172.21.15.70:0/3649493025 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62d0011250 con 0x55bbfd17c640 2024-04-01T01:42:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.933+0000 7f62ed181740 1 -- 172.21.15.70:0/3649493025 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f62d0010b80 con 0x55bbfd176ae0 2024-04-01T01:42:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.933+0000 7f62ed181740 1 -- 172.21.15.70:0/3649493025 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f62d000ddc0 con 0x55bbfd1774d0 2024-04-01T01:42:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.933+0000 7f62e37fe640 1 --2- 172.21.15.70:0/3649493025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd1774d0 0x55bbfd184c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.933+0000 7f62ed181740 10 client.0 _ll_register_callbacks cb 0x7f62d8009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.933+0000 7f62e8a6a640 1 --2- 172.21.15.70:0/3649493025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd17c640 0x55bbfd1851c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.933+0000 7f62e8a6a640 1 -- 172.21.15.70:0/3649493025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd1774d0 msgr2=0x55bbfd184c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.933+0000 7f62e8a6a640 1 --2- 172.21.15.70:0/3649493025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd1774d0 0x55bbfd184c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.933+0000 7f62e8a6a640 1 -- 172.21.15.70:0/3649493025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd17c640 msgr2=0x55bbfd1851c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.933+0000 7f62e8a6a640 1 --2- 172.21.15.70:0/3649493025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd17c640 0x55bbfd1851c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.933+0000 7f62e8a6a640 1 -- 172.21.15.70:0/3649493025 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbfd2021c0 con 0x55bbfd176ae0 2024-04-01T01:42:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.934+0000 7f62e8a6a640 1 --2- 172.21.15.70:0/3649493025 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd176ae0 0x55bbfd188170 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f62d800f6e0 tx=0x7f62d800f650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8098f88b736948a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.934+0000 7f62c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.934+0000 7f62c37fe640 1 -- 172.21.15.70:0/3649493025 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d803f070 con 0x55bbfd176ae0 2024-04-01T01:42:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.934+0000 7f62c37fe640 1 -- 172.21.15.70:0/3649493025 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f62d800fc10 con 0x55bbfd176ae0 2024-04-01T01:42:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.934+0000 7f62ed181740 1 -- 172.21.15.70:0/3649493025 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62d80095f0 con 0x55bbfd176ae0 2024-04-01T01:42:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.934+0000 7f62c37fe640 1 -- 172.21.15.70:0/3649493025 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f62d804c050 con 0x55bbfd176ae0 2024-04-01T01:42:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.935+0000 7f62ed181740 10 client.27974 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.935+0000 7f62ed181740 1 -- 172.21.15.70:0/3649493025 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55bbfd203440 con 0x55bbfd176ae0 2024-04-01T01:42:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.935+0000 7f62c37fe640 1 -- 172.21.15.70:0/3649493025 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62d80095f0 con 0x55bbfd176ae0 2024-04-01T01:42:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.935+0000 7f62c37fe640 1 -- 172.21.15.70:0/3649493025 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f62d80050a0 con 0x55bbfd176ae0 2024-04-01T01:42:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.935+0000 7f62ed181740 10 client.27974 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.935+0000 7f62ed181740 10 client.27974 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.936+0000 7f62ed181740 20 client.27974 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.936+0000 7f62ed181740 1 --2- 172.21.15.70:0/3649493025 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bbfd203d30 0x55bbfd206150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.936+0000 7f62ed181740 4 client.27974 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:42:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.936+0000 7f62ed181740 1 -- 172.21.15.70:0/3649493025 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bbfd0e7e30 con 0x55bbfd203d30 2024-04-01T01:42:00.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.937+0000 7f62e37fe640 1 --2- 172.21.15.70:0/3649493025 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bbfd203d30 0x55bbfd206150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:00.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.938+0000 7f62c37fe640 10 client.27974 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:00.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:00.938+0000 7f62e37fe640 1 --2- 172.21.15.70:0/3649493025 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bbfd203d30 0x55bbfd206150 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7be91559174ebfb8 server_cookie=2bc6a8b34fc31a3 in_seq=0 out_seq=0 2024-04-01T01:42:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.065+0000 7f62c37fe640 1 -- 172.21.15.70:0/3649493025 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bbfd0e7e30 con 0x55bbfd203d30 2024-04-01T01:42:01.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.065+0000 7f62c37fe640 10 client.27974 handle_command_reply: tid=0 2024-04-01T01:42:01.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.065+0000 7f62ed181740 1 client.27974 shutdown 2024-04-01T01:42:01.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.066+0000 7f62ed181740 1 -- 172.21.15.70:0/3649493025 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd176ae0 msgr2=0x55bbfd188170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:01.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.066+0000 7f62ed181740 1 --2- 172.21.15.70:0/3649493025 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd176ae0 0x55bbfd188170 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f62d800f6e0 tx=0x7f62d800f650 comp rx=0 tx=0).stop 2024-04-01T01:42:01.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.067+0000 7f62ed181740 1 -- 172.21.15.70:0/3649493025 shutdown_connections 2024-04-01T01:42:01.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.067+0000 7f62ed181740 1 --2- 172.21.15.70:0/3649493025 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55bbfd203d30 0x55bbfd206150 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:01.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.067+0000 7f62ed181740 1 --2- 172.21.15.70:0/3649493025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bbfd17c640 0x55bbfd1851c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:01.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.067+0000 7f62c37fe640 0 client.27974 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:01.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.067+0000 7f62ed181740 1 --2- 172.21.15.70:0/3649493025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bbfd1774d0 0x55bbfd184c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:01.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.067+0000 7f62ed181740 1 --2- 172.21.15.70:0/3649493025 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bbfd176ae0 0x55bbfd188170 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:01.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.067+0000 7f62ed181740 1 -- 172.21.15.70:0/3649493025 >> 172.21.15.70:0/3649493025 conn(0x55bbfd172050 msgr2=0x55bbfd17b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:01.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.071+0000 7f62ed181740 1 -- 172.21.15.70:0/3649493025 shutdown_connections 2024-04-01T01:42:01.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.071+0000 7f62ed181740 1 -- 172.21.15.70:0/3649493025 wait complete. 2024-04-01T01:42:01.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.071+0000 7f62ed181740 20 client.27974 trim_cache size 0 max 16384 2024-04-01T01:42:01.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:42:01.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:42:01.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:42:01.072 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:42:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.073+0000 7f62e926b640 1 -- 172.21.15.70:0/272924082 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62cc0a25e0 msgr2=0x7f62cc0a4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:01.073 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.073+0000 7f62e926b640 1 --2- 172.21.15.70:0/272924082 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62cc0a25e0 0x7f62cc0a4aa0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0cba00 tx=0x7f62dc004b50 comp rx=0 tx=0).stop 2024-04-01T01:42:01.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.073+0000 7f62e926b640 1 -- 172.21.15.70:0/272924082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62e4107bd0 msgr2=0x7f62e419de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:01.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.073+0000 7f62e926b640 1 --2- 172.21.15.70:0/272924082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62e4107bd0 0x7f62e419de70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f62dc00b9c0 tx=0x7f62dc00be80 comp rx=0 tx=0).stop 2024-04-01T01:42:01.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.073+0000 7f62e926b640 1 -- 172.21.15.70:0/272924082 shutdown_connections 2024-04-01T01:42:01.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.073+0000 7f62e926b640 1 --2- 172.21.15.70:0/272924082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62e4109780 0x7f62e41a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:01.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.073+0000 7f62e926b640 1 --2- 172.21.15.70:0/272924082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e4108dc0 0x7f62e419e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:01.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.073+0000 7f62e926b640 1 --2- 172.21.15.70:0/272924082 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62cc0a25e0 0x7f62cc0a4aa0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:01.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.073+0000 7f62e926b640 1 --2- 172.21.15.70:0/272924082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62e4107bd0 0x7f62e419de70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:01.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.073+0000 7f62e926b640 1 -- 172.21.15.70:0/272924082 >> 172.21.15.70:0/272924082 conn(0x7f62e4076b60 msgr2=0x7f62e410c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:01.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.073+0000 7f62e926b640 1 -- 172.21.15.70:0/272924082 shutdown_connections 2024-04-01T01:42:01.074 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:01.073+0000 7f62e926b640 1 -- 172.21.15.70:0/272924082 wait complete. 2024-04-01T01:42:01.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:00 smithi189 ceph-mon[35753]: pgmap v615: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.3 MiB/s wr, 821 op/s 2024-04-01T01:42:01.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:00 smithi070 ceph-mon[32179]: pgmap v615: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.3 MiB/s wr, 821 op/s 2024-04-01T01:42:01.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:00 smithi081 ceph-mon[35142]: pgmap v615: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.3 MiB/s wr, 821 op/s 2024-04-01T01:42:01.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 435.0 s, 814.2 tps, lat 121.913 ms stddev 122.813 2024-04-01T01:42:03.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:02 smithi189 ceph-mon[35753]: pgmap v616: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 906 op/s 2024-04-01T01:42:03.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:02 smithi070 ceph-mon[32179]: pgmap v616: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 906 op/s 2024-04-01T01:42:03.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:02 smithi081 ceph-mon[35142]: pgmap v616: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 MiB/s wr, 906 op/s 2024-04-01T01:42:05.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:04 smithi189 ceph-mon[35753]: pgmap v617: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.9 MiB/s wr, 779 op/s 2024-04-01T01:42:05.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:04 smithi070 ceph-mon[32179]: pgmap v617: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.9 MiB/s wr, 779 op/s 2024-04-01T01:42:05.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:04 smithi081 ceph-mon[35142]: pgmap v617: 129 pgs: 129 active+clean; 1.4 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.9 MiB/s wr, 779 op/s 2024-04-01T01:42:06.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:05 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:42:06.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:05 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:42:06.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:05 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:42:06.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 440.0 s, 759.6 tps, lat 134.005 ms stddev 127.847 2024-04-01T01:42:07.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:06 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:42:07.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:06 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:42:07.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:06 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:42:07.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:06 smithi189 ceph-mon[35753]: pgmap v618: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.0 MiB/s wr, 880 op/s 2024-04-01T01:42:07.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:06 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:42:07.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:06 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:42:07.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:06 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:42:07.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:06 smithi070 ceph-mon[32179]: pgmap v618: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.0 MiB/s wr, 880 op/s 2024-04-01T01:42:07.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:06 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:42:07.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:06 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:42:07.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:06 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:42:07.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:06 smithi081 ceph-mon[35142]: pgmap v618: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.0 MiB/s wr, 880 op/s 2024-04-01T01:42:08.109 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:42:08.109 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:42:08.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.285+0000 7f8945640640 1 -- 172.21.15.70:0/3525781240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938096860 msgr2=0x7f893809d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:08.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.285+0000 7f8945640640 1 --2- 172.21.15.70:0/3525781240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938096860 0x7f893809d0f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8940067420 tx=0x7f8940097850 comp rx=0 tx=0).stop 2024-04-01T01:42:08.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.286+0000 7f8945640640 1 -- 172.21.15.70:0/3525781240 shutdown_connections 2024-04-01T01:42:08.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.286+0000 7f8945640640 1 --2- 172.21.15.70:0/3525781240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938096860 0x7f893809d0f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.286+0000 7f8945640640 1 --2- 172.21.15.70:0/3525781240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8938095ea0 0x7f8938096320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.286+0000 7f8945640640 1 --2- 172.21.15.70:0/3525781240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8938094cb0 0x7f89380950d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.286+0000 7f8945640640 1 -- 172.21.15.70:0/3525781240 >> 172.21.15.70:0/3525781240 conn(0x7f8938090200 msgr2=0x7f8938092660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:08.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.286+0000 7f8945640640 1 -- 172.21.15.70:0/3525781240 shutdown_connections 2024-04-01T01:42:08.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.287+0000 7f8945640640 1 -- 172.21.15.70:0/3525781240 wait complete. 2024-04-01T01:42:08.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.287+0000 7f8945640640 1 Processor -- start 2024-04-01T01:42:08.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.287+0000 7f8945640640 1 -- start start 2024-04-01T01:42:08.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.288+0000 7f8945640640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8938094cb0 0x7f8938134cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.288+0000 7f8945640640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8938095ea0 0x7f8938137210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.288+0000 7f8945640640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938096860 0x7f89381397c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.288+0000 7f8945640640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f893809f5f0 con 0x7f8938095ea0 2024-04-01T01:42:08.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.288+0000 7f8945640640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f893809f470 con 0x7f8938094cb0 2024-04-01T01:42:08.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.288+0000 7f8945640640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f893809f770 con 0x7f8938096860 2024-04-01T01:42:08.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.288+0000 7f893f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938096860 0x7f89381397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.288+0000 7f893f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938096860 0x7f89381397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:56662/0 (socket says 172.21.15.70:56662) 2024-04-01T01:42:08.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.288+0000 7f893f7fe640 1 -- 172.21.15.70:0/3853790836 learned_addr learned my addr 172.21.15.70:0/3853790836 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.288+0000 7f893e7fc640 1 --2- 172.21.15.70:0/3853790836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8938095ea0 0x7f8938137210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.288+0000 7f893effd640 1 --2- 172.21.15.70:0/3853790836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8938094cb0 0x7f8938134cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.289+0000 7f893f7fe640 1 -- 172.21.15.70:0/3853790836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8938094cb0 msgr2=0x7f8938134cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.289+0000 7f893f7fe640 1 --2- 172.21.15.70:0/3853790836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8938094cb0 0x7f8938134cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.289+0000 7f893f7fe640 1 -- 172.21.15.70:0/3853790836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8938095ea0 msgr2=0x7f8938137210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.289+0000 7f893f7fe640 1 --2- 172.21.15.70:0/3853790836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8938095ea0 0x7f8938137210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.289+0000 7f893f7fe640 1 -- 172.21.15.70:0/3853790836 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f893813a080 con 0x7f8938096860 2024-04-01T01:42:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.289+0000 7f893e7fc640 1 --2- 172.21.15.70:0/3853790836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8938095ea0 0x7f8938137210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:42:08.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.289+0000 7f893f7fe640 1 --2- 172.21.15.70:0/3853790836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938096860 0x7f89381397c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f894002d700 tx=0x7f8940069960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:08.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.289+0000 7f8927fff640 1 -- 172.21.15.70:0/3853790836 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89400ad030 con 0x7f8938096860 2024-04-01T01:42:08.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.290+0000 7f8927fff640 1 -- 172.21.15.70:0/3853790836 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f89400692e0 con 0x7f8938096860 2024-04-01T01:42:08.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.290+0000 7f8927fff640 1 -- 172.21.15.70:0/3853790836 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89400a9cb0 con 0x7f8938096860 2024-04-01T01:42:08.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.290+0000 7f8945640640 1 -- 172.21.15.70:0/3853790836 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89380a55b0 con 0x7f8938096860 2024-04-01T01:42:08.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.290+0000 7f8945640640 1 -- 172.21.15.70:0/3853790836 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89380a59c0 con 0x7f8938096860 2024-04-01T01:42:08.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.293+0000 7f8948227740 1 Processor -- start 2024-04-01T01:42:08.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.293+0000 7f8927fff640 1 -- 172.21.15.70:0/3853790836 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f89400b0070 con 0x7f8938096860 2024-04-01T01:42:08.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.294+0000 7f8948227740 1 -- start start 2024-04-01T01:42:08.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.294+0000 7f8948227740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c188490 0x563d6c1888b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.294+0000 7f8948227740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c188e80 0x563d6c18d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.294+0000 7f8948227740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c18dff0 0x563d6c190480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.294+0000 7f8948227740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563d6c0ed180 con 0x563d6c188490 2024-04-01T01:42:08.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.294+0000 7f8948227740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563d6c095b70 con 0x563d6c18dff0 2024-04-01T01:42:08.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.294+0000 7f8948227740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563d6c0e9220 con 0x563d6c188e80 2024-04-01T01:42:08.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.294+0000 7f893e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c18dff0 0x563d6c190480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.294+0000 7f893e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c18dff0 0x563d6c190480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60646/0 (socket says 172.21.15.70:60646) 2024-04-01T01:42:08.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.294+0000 7f893e7fc640 1 -- 172.21.15.70:0/3111071967 learned_addr learned my addr 172.21.15.70:0/3111071967 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:08.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.295+0000 7f8927fff640 1 --2- 172.21.15.70:0/3853790836 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f89280a26a0 0x7f89280a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.295+0000 7f893effd640 1 --2- 172.21.15.70:0/3111071967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c188e80 0x563d6c18d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.295+0000 7f893e7fc640 1 --2- 172.21.15.70:0/3111071967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c188490 0x563d6c1888b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.295+0000 7f8927fff640 1 -- 172.21.15.70:0/3853790836 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f894015c250 con 0x7f8938096860 2024-04-01T01:42:08.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.295+0000 7f893f7fe640 1 --2- 172.21.15.70:0/3853790836 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f89280a26a0 0x7f89280a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.296+0000 7f893effd640 1 -- 172.21.15.70:0/3111071967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c18dff0 msgr2=0x563d6c190480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:08.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.296+0000 7f893effd640 1 --2- 172.21.15.70:0/3111071967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c18dff0 0x563d6c190480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.296+0000 7f893effd640 1 -- 172.21.15.70:0/3111071967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c188490 msgr2=0x563d6c1888b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:08.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.297+0000 7f893f7fe640 1 --2- 172.21.15.70:0/3853790836 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f89280a26a0 0x7f89280a4b60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8940067290 tx=0x7f8940069490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:08.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.298+0000 7f893effd640 1 --2- 172.21.15.70:0/3111071967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c188490 0x563d6c1888b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.298+0000 7f893effd640 1 -- 172.21.15.70:0/3111071967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d6c0a97c0 con 0x563d6c188e80 2024-04-01T01:42:08.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.298+0000 7f893effd640 1 --2- 172.21.15.70:0/3111071967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c188e80 0x563d6c18d980 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8938095d00 tx=0x7f89300113e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ce92b6f2ca0fc0a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:08.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.298+0000 7f89257fa640 1 -- 172.21.15.70:0/3111071967 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8930011de0 con 0x563d6c188e80 2024-04-01T01:42:08.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.299+0000 7f89257fa640 1 -- 172.21.15.70:0/3111071967 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8930012900 con 0x563d6c188e80 2024-04-01T01:42:08.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.299+0000 7f89257fa640 1 -- 172.21.15.70:0/3111071967 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893000f9a0 con 0x563d6c188e80 2024-04-01T01:42:08.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.301+0000 7f8948227740 1 -- 172.21.15.70:0/3111071967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c188e80 msgr2=0x563d6c18d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:08.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.301+0000 7f8948227740 1 --2- 172.21.15.70:0/3111071967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c188e80 0x563d6c18d980 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8938095d00 tx=0x7f89300113e0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.302+0000 7f8948227740 1 -- 172.21.15.70:0/3111071967 shutdown_connections 2024-04-01T01:42:08.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.302+0000 7f8948227740 1 --2- 172.21.15.70:0/3111071967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c18dff0 0x563d6c190480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.302+0000 7f8948227740 1 --2- 172.21.15.70:0/3111071967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c188e80 0x563d6c18d980 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.302+0000 7f8948227740 1 --2- 172.21.15.70:0/3111071967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c188490 0x563d6c1888b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.302+0000 7f8948227740 1 -- 172.21.15.70:0/3111071967 >> 172.21.15.70:0/3111071967 conn(0x563d6c183a00 msgr2=0x563d6c183e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:08.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.303+0000 7f8948227740 1 -- 172.21.15.70:0/3111071967 shutdown_connections 2024-04-01T01:42:08.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.303+0000 7f8948227740 1 -- 172.21.15.70:0/3111071967 wait complete. 2024-04-01T01:42:08.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.303+0000 7f8948227740 1 -- start start 2024-04-01T01:42:08.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.304+0000 7f8948227740 1 Processor -- start 2024-04-01T01:42:08.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.305+0000 7f8948227740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188490 0x563d6c192820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.305+0000 7f8948227740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c188e80 0x563d6c192d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.305+0000 7f893e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188490 0x563d6c192820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.305+0000 7f893e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188490 0x563d6c192820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60652/0 (socket says 172.21.15.70:60652) 2024-04-01T01:42:08.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.305+0000 7f893e7fc640 1 -- 172.21.15.70:0/1043161956 learned_addr learned my addr 172.21.15.70:0/1043161956 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:08.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.305+0000 7f8948227740 1 --2- 172.21.15.70:0/1043161956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c18dff0 0x563d6c199da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.305+0000 7f8948227740 1 -- 172.21.15.70:0/1043161956 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563d6c190fa0 con 0x563d6c18dff0 2024-04-01T01:42:08.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.305+0000 7f8948227740 1 -- 172.21.15.70:0/1043161956 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563d6c08b4a0 con 0x563d6c188490 2024-04-01T01:42:08.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.305+0000 7f8948227740 1 -- 172.21.15.70:0/1043161956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563d6c191120 con 0x563d6c188e80 2024-04-01T01:42:08.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.305+0000 7f893effd640 1 --2- 172.21.15.70:0/1043161956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c188e80 0x563d6c192d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.306+0000 7f8948227740 10 client.0 _ll_register_callbacks cb 0x563d6c08ed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:08.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.308+0000 7f893effd640 1 -- 172.21.15.70:0/1043161956 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188490 msgr2=0x563d6c192820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:08.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.308+0000 7f893effd640 1 --2- 172.21.15.70:0/1043161956 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188490 0x563d6c192820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.308+0000 7f893effd640 1 -- 172.21.15.70:0/1043161956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c18dff0 msgr2=0x563d6c199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:08.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.308+0000 7f893effd640 1 --2- 172.21.15.70:0/1043161956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c18dff0 0x563d6c199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.308+0000 7f893effd640 1 -- 172.21.15.70:0/1043161956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d6becb7d0 con 0x563d6c188e80 2024-04-01T01:42:08.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.309+0000 7f893effd640 1 --2- 172.21.15.70:0/1043161956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c188e80 0x563d6c192d60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x563d6c189280 tx=0x7f8930012680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5595a2c7cf8ec433 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:08.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.309+0000 7f8922ff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:42:08.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.309+0000 7f8922ff5640 1 -- 172.21.15.70:0/1043161956 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8930028020 con 0x563d6c188e80 2024-04-01T01:42:08.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.309+0000 7f8922ff5640 1 -- 172.21.15.70:0/1043161956 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8930012ab0 con 0x563d6c188e80 2024-04-01T01:42:08.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.311+0000 7f8948227740 1 -- 172.21.15.70:0/1043161956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8930012900 con 0x563d6c188e80 2024-04-01T01:42:08.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.311+0000 7f8922ff5640 1 -- 172.21.15.70:0/1043161956 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8930012cb0 con 0x563d6c188e80 2024-04-01T01:42:08.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.311+0000 7f8948227740 10 client.27811 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:08.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.311+0000 7f8948227740 1 -- 172.21.15.70:0/1043161956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563d6be013f0 con 0x563d6c188e80 2024-04-01T01:42:08.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.312+0000 7f8922ff5640 1 -- 172.21.15.70:0/1043161956 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8930012e50 con 0x563d6c188e80 2024-04-01T01:42:08.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.312+0000 7f8922ff5640 1 -- 172.21.15.70:0/1043161956 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8930003800 con 0x563d6c188e80 2024-04-01T01:42:08.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.312+0000 7f8948227740 10 client.27811 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:08.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.312+0000 7f8948227740 10 client.27811 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:08.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.312+0000 7f8948227740 20 client.27811 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:08.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.312+0000 7f8948227740 1 --2- 172.21.15.70:0/1043161956 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563d6c19e920 0x563d6c1a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.313+0000 7f8948227740 4 client.27811 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:08.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.313+0000 7f8948227740 1 -- 172.21.15.70:0/1043161956 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8930012900 con 0x563d6c19e920 2024-04-01T01:42:08.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.314+0000 7f893e7fc640 1 --2- 172.21.15.70:0/1043161956 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563d6c19e920 0x563d6c1a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.315+0000 7f893e7fc640 1 --2- 172.21.15.70:0/1043161956 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563d6c19e920 0x563d6c1a0d40 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26ad86a27a0ebd72 server_cookie=2515c7395d3e6ad0 in_seq=0 out_seq=0 2024-04-01T01:42:08.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.315+0000 7f8922ff5640 10 client.27811 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:08.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.316+0000 7f8922ff5640 1 -- 172.21.15.70:0/1043161956 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f8930012900 con 0x563d6c19e920 2024-04-01T01:42:08.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.316+0000 7f8922ff5640 10 client.27811 handle_command_reply: tid=0 2024-04-01T01:42:08.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.316+0000 7f8948227740 1 client.27811 shutdown 2024-04-01T01:42:08.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.320+0000 7f8948227740 1 -- 172.21.15.70:0/1043161956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c188e80 msgr2=0x563d6c192d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:08.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.320+0000 7f8948227740 1 --2- 172.21.15.70:0/1043161956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c188e80 0x563d6c192d60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x563d6c189280 tx=0x7f8930012680 comp rx=0 tx=0).stop 2024-04-01T01:42:08.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.322+0000 7f8948227740 1 -- 172.21.15.70:0/1043161956 shutdown_connections 2024-04-01T01:42:08.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.322+0000 7f8948227740 1 --2- 172.21.15.70:0/1043161956 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563d6c19e920 0x563d6c1a0d40 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.322+0000 7f8948227740 1 --2- 172.21.15.70:0/1043161956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c18dff0 0x563d6c199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.322+0000 7f8948227740 1 --2- 172.21.15.70:0/1043161956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c188e80 0x563d6c192d60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.322+0000 7f8948227740 1 --2- 172.21.15.70:0/1043161956 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188490 0x563d6c192820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.323+0000 7f8948227740 1 -- 172.21.15.70:0/1043161956 >> 172.21.15.70:0/1043161956 conn(0x563d6c183a00 msgr2=0x563d6c182c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:08.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.323+0000 7f8922ff5640 0 client.27811 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:08.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.324+0000 7f8948227740 1 -- 172.21.15.70:0/1043161956 shutdown_connections 2024-04-01T01:42:08.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.324+0000 7f8948227740 1 -- 172.21.15.70:0/1043161956 wait complete. 2024-04-01T01:42:08.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.325+0000 7f8948227740 20 client.27811 trim_cache size 0 max 16384 2024-04-01T01:42:08.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.358+0000 7f8948227740 1 Processor -- start 2024-04-01T01:42:08.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.358+0000 7f8948227740 1 -- start start 2024-04-01T01:42:08.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.358+0000 7f8948227740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188490 0x563d6c192820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.358+0000 7f8948227740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c188e80 0x563d6c192d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.358+0000 7f8948227740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c18dff0 0x563d6c199d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.358+0000 7f8948227740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f893813c6d0 con 0x563d6c188e80 2024-04-01T01:42:08.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.358+0000 7f8948227740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f893813e210 con 0x563d6c188490 2024-04-01T01:42:08.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.358+0000 7f8948227740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f893813cda0 con 0x563d6c18dff0 2024-04-01T01:42:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.359+0000 7f893e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188490 0x563d6c192820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.359+0000 7f893e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188490 0x563d6c192820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60654/0 (socket says 172.21.15.70:60654) 2024-04-01T01:42:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.359+0000 7f893e7fc640 1 -- 172.21.15.70:0/3600663685 learned_addr learned my addr 172.21.15.70:0/3600663685 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.359+0000 7f893e7fc640 1 --2- 172.21.15.70:0/3600663685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c18dff0 0x563d6c199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.359+0000 7f893e7fc640 1 -- 172.21.15.70:0/3600663685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c18dff0 msgr2=0x563d6c199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.359+0000 7f893e7fc640 1 --2- 172.21.15.70:0/3600663685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c18dff0 0x563d6c199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.359+0000 7f893e7fc640 1 -- 172.21.15.70:0/3600663685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c188e80 msgr2=0x563d6c192d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:42:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.359+0000 7f893e7fc640 1 --2- 172.21.15.70:0/3600663685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c188e80 0x563d6c192d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.359+0000 7f893e7fc640 1 -- 172.21.15.70:0/3600663685 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d6bdc10f0 con 0x563d6c188490 2024-04-01T01:42:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.360+0000 7f893e7fc640 1 --2- 172.21.15.70:0/3600663685 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188490 0x563d6c192820 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8934009730 tx=0x7f8934004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c432e757431bbde3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.360+0000 7f8924ff9640 1 -- 172.21.15.70:0/3600663685 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8934005020 con 0x563d6c188490 2024-04-01T01:42:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.360+0000 7f8924ff9640 1 -- 172.21.15.70:0/3600663685 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f89340047a0 con 0x563d6c188490 2024-04-01T01:42:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.360+0000 7f8924ff9640 1 -- 172.21.15.70:0/3600663685 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8934031450 con 0x563d6c188490 2024-04-01T01:42:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.362+0000 7f8948227740 1 -- 172.21.15.70:0/3600663685 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188490 msgr2=0x563d6c192820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.362+0000 7f8948227740 1 --2- 172.21.15.70:0/3600663685 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188490 0x563d6c192820 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8934009730 tx=0x7f8934004730 comp rx=0 tx=0).stop 2024-04-01T01:42:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.362+0000 7f8948227740 1 -- 172.21.15.70:0/3600663685 shutdown_connections 2024-04-01T01:42:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.362+0000 7f8948227740 1 --2- 172.21.15.70:0/3600663685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c18dff0 0x563d6c199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.362+0000 7f8948227740 1 --2- 172.21.15.70:0/3600663685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c188e80 0x563d6c192d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.362+0000 7f8948227740 1 --2- 172.21.15.70:0/3600663685 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188490 0x563d6c192820 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.362+0000 7f8948227740 1 -- 172.21.15.70:0/3600663685 >> 172.21.15.70:0/3600663685 conn(0x563d6c183a00 msgr2=0x563d6c17eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.363+0000 7f8948227740 1 -- 172.21.15.70:0/3600663685 shutdown_connections 2024-04-01T01:42:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.363+0000 7f8948227740 1 -- 172.21.15.70:0/3600663685 wait complete. 2024-04-01T01:42:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.363+0000 7f8948227740 1 -- start start 2024-04-01T01:42:08.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.364+0000 7f8948227740 1 Processor -- start 2024-04-01T01:42:08.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.364+0000 7f8948227740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c188490 0x563d6c196b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.365+0000 7f8948227740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188e80 0x563d6c1970c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.365+0000 7f8948227740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c18dff0 0x563d6c202c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.365+0000 7f8948227740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f893813db40 con 0x563d6c188490 2024-04-01T01:42:08.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.365+0000 7f8948227740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f893813d470 con 0x563d6c188e80 2024-04-01T01:42:08.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.365+0000 7f8948227740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f89380ad910 con 0x563d6c18dff0 2024-04-01T01:42:08.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.365+0000 7f8948227740 10 client.0 _ll_register_callbacks cb 0x7f8934009b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:08.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.365+0000 7f893effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188e80 0x563d6c1970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.365+0000 7f893effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188e80 0x563d6c1970c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60670/0 (socket says 172.21.15.70:60670) 2024-04-01T01:42:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.365+0000 7f893effd640 1 -- 172.21.15.70:0/379464965 learned_addr learned my addr 172.21.15.70:0/379464965 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.365+0000 7f893e7fc640 1 --2- 172.21.15.70:0/379464965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c188490 0x563d6c196b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.365+0000 7f893e7fc640 1 --2- 172.21.15.70:0/379464965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c18dff0 0x563d6c202c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.365+0000 7f893effd640 1 -- 172.21.15.70:0/379464965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c18dff0 msgr2=0x563d6c202c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.365+0000 7f893effd640 1 --2- 172.21.15.70:0/379464965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c18dff0 0x563d6c202c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.365+0000 7f893effd640 1 -- 172.21.15.70:0/379464965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c188490 msgr2=0x563d6c196b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.365+0000 7f893effd640 1 --2- 172.21.15.70:0/379464965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c188490 0x563d6c196b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.365+0000 7f893effd640 1 -- 172.21.15.70:0/379464965 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d6c197730 con 0x563d6c188e80 2024-04-01T01:42:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.366+0000 7f893effd640 1 --2- 172.21.15.70:0/379464965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188e80 0x563d6c1970c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f893002f040 tx=0x7f8930005510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1fc4c2aedea7721 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.366+0000 7f8922ff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.366+0000 7f8922ff5640 1 -- 172.21.15.70:0/379464965 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8930028040 con 0x563d6c188e80 2024-04-01T01:42:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.366+0000 7f8922ff5640 1 -- 172.21.15.70:0/379464965 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8930012900 con 0x563d6c188e80 2024-04-01T01:42:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.366+0000 7f8948227740 1 -- 172.21.15.70:0/379464965 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89340047a0 con 0x563d6c188e80 2024-04-01T01:42:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.367+0000 7f8922ff5640 1 -- 172.21.15.70:0/379464965 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8930045050 con 0x563d6c188e80 2024-04-01T01:42:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.367+0000 7f8948227740 10 client.27992 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.367+0000 7f8948227740 1 -- 172.21.15.70:0/379464965 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563d6c2042b0 con 0x563d6c188e80 2024-04-01T01:42:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.367+0000 7f8922ff5640 1 -- 172.21.15.70:0/379464965 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f89340047a0 con 0x563d6c188e80 2024-04-01T01:42:08.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.368+0000 7f8922ff5640 1 -- 172.21.15.70:0/379464965 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8930024b80 con 0x563d6c188e80 2024-04-01T01:42:08.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.368+0000 7f8948227740 10 client.27992 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:08.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.368+0000 7f8948227740 10 client.27992 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:08.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.368+0000 7f8948227740 20 client.27992 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:08.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.368+0000 7f8948227740 1 --2- 172.21.15.70:0/379464965 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563d6c204c50 0x563d6c207070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.368+0000 7f8948227740 4 client.27992 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:42:08.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.368+0000 7f8948227740 1 -- 172.21.15.70:0/379464965 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563d6c0fa6c0 con 0x563d6c204c50 2024-04-01T01:42:08.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.368+0000 7f893e7fc640 1 --2- 172.21.15.70:0/379464965 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563d6c204c50 0x563d6c207070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.369+0000 7f893e7fc640 1 --2- 172.21.15.70:0/379464965 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563d6c204c50 0x563d6c207070 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57f12bb0d51394f8 server_cookie=17cc64cbf8a9e108 in_seq=0 out_seq=0 2024-04-01T01:42:08.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.369+0000 7f8922ff5640 10 client.27992 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:08.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.369+0000 7f8922ff5640 1 -- 172.21.15.70:0/379464965 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563d6c0fa6c0 con 0x563d6c204c50 2024-04-01T01:42:08.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.369+0000 7f8922ff5640 10 client.27992 handle_command_reply: tid=0 2024-04-01T01:42:08.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.369+0000 7f8948227740 1 client.27992 shutdown 2024-04-01T01:42:08.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.370+0000 7f8948227740 1 -- 172.21.15.70:0/379464965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188e80 msgr2=0x563d6c1970c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:08.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.370+0000 7f8948227740 1 --2- 172.21.15.70:0/379464965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188e80 0x563d6c1970c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f893002f040 tx=0x7f8930005510 comp rx=0 tx=0).stop 2024-04-01T01:42:08.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.371+0000 7f8948227740 1 -- 172.21.15.70:0/379464965 shutdown_connections 2024-04-01T01:42:08.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.371+0000 7f8948227740 1 --2- 172.21.15.70:0/379464965 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563d6c204c50 0x563d6c207070 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.371+0000 7f8948227740 1 --2- 172.21.15.70:0/379464965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d6c18dff0 0x563d6c202c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.371+0000 7f8948227740 1 --2- 172.21.15.70:0/379464965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d6c188e80 0x563d6c1970c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.371+0000 7f8948227740 1 --2- 172.21.15.70:0/379464965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d6c188490 0x563d6c196b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.371+0000 7f8948227740 1 -- 172.21.15.70:0/379464965 >> 172.21.15.70:0/379464965 conn(0x563d6c183a00 msgr2=0x563d6c18a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:08.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.371+0000 7f8922ff5640 0 client.27992 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:08.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.374+0000 7f8948227740 1 -- 172.21.15.70:0/379464965 shutdown_connections 2024-04-01T01:42:08.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.375+0000 7f8948227740 1 -- 172.21.15.70:0/379464965 wait complete. 2024-04-01T01:42:08.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.375+0000 7f8948227740 20 client.27992 trim_cache size 0 max 16384 2024-04-01T01:42:08.376 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:42:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:42:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:42:08.376 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:42:08.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.377+0000 7f8945640640 1 -- 172.21.15.70:0/3853790836 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f89280a26a0 msgr2=0x7f89280a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:08.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.377+0000 7f8945640640 1 --2- 172.21.15.70:0/3853790836 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f89280a26a0 0x7f89280a4b60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8940067290 tx=0x7f8940069490 comp rx=0 tx=0).stop 2024-04-01T01:42:08.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.377+0000 7f8945640640 1 -- 172.21.15.70:0/3853790836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938096860 msgr2=0x7f89381397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:08.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.377+0000 7f8945640640 1 --2- 172.21.15.70:0/3853790836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938096860 0x7f89381397c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f894002d700 tx=0x7f8940069960 comp rx=0 tx=0).stop 2024-04-01T01:42:08.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.377+0000 7f8945640640 1 -- 172.21.15.70:0/3853790836 shutdown_connections 2024-04-01T01:42:08.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.377+0000 7f8945640640 1 --2- 172.21.15.70:0/3853790836 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f89280a26a0 0x7f89280a4b60 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.377+0000 7f8945640640 1 --2- 172.21.15.70:0/3853790836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8938096860 0x7f89381397c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.377+0000 7f8945640640 1 --2- 172.21.15.70:0/3853790836 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8938095ea0 0x7f8938137210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.377+0000 7f8945640640 1 --2- 172.21.15.70:0/3853790836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8938094cb0 0x7f8938134cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.377+0000 7f8945640640 1 -- 172.21.15.70:0/3853790836 >> 172.21.15.70:0/3853790836 conn(0x7f8938090200 msgr2=0x7f89380a2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:08.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.379+0000 7f8945640640 1 -- 172.21.15.70:0/3853790836 shutdown_connections 2024-04-01T01:42:08.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.379+0000 7f8945640640 1 -- 172.21.15.70:0/3853790836 wait complete. 2024-04-01T01:42:08.401 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:42:08.401 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:42:08.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.575+0000 7f79952f4640 1 -- 172.21.15.70:0/1880811870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79900fffe0 msgr2=0x7f7990100400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:08.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.575+0000 7f79952f4640 1 --2- 172.21.15.70:0/1880811870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79900fffe0 0x7f7990100400 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f7980009900 tx=0x7f798002bc30 comp rx=0 tx=0).stop 2024-04-01T01:42:08.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.575+0000 7f79952f4640 1 -- 172.21.15.70:0/1880811870 shutdown_connections 2024-04-01T01:42:08.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.575+0000 7f79952f4640 1 --2- 172.21.15.70:0/1880811870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79901012e0 0x7f7990101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.575+0000 7f79952f4640 1 --2- 172.21.15.70:0/1880811870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7990100940 0x7f7990100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.575+0000 7f79952f4640 1 --2- 172.21.15.70:0/1880811870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79900fffe0 0x7f7990100400 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.575+0000 7f79952f4640 1 -- 172.21.15.70:0/1880811870 >> 172.21.15.70:0/1880811870 conn(0x7f79900fbbe0 msgr2=0x7f79900fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:08.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.575+0000 7f79952f4640 1 -- 172.21.15.70:0/1880811870 shutdown_connections 2024-04-01T01:42:08.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.575+0000 7f79952f4640 1 -- 172.21.15.70:0/1880811870 wait complete. 2024-04-01T01:42:08.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.576+0000 7f79952f4640 1 Processor -- start 2024-04-01T01:42:08.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.576+0000 7f79952f4640 1 -- start start 2024-04-01T01:42:08.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.576+0000 7f79952f4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79900fffe0 0x7f799019dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.576+0000 7f79952f4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7990100940 0x7f799019e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.576+0000 7f79952f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79901012e0 0x7f79901a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.576+0000 7f79952f4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7990102af0 con 0x7f79901012e0 2024-04-01T01:42:08.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.576+0000 7f79952f4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7990102970 con 0x7f79900fffe0 2024-04-01T01:42:08.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.576+0000 7f79952f4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7990102c70 con 0x7f7990100940 2024-04-01T01:42:08.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.576+0000 7f798effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79900fffe0 0x7f799019dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.577+0000 7f798e7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7990100940 0x7f799019e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.577+0000 7f798effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79900fffe0 0x7f799019dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60690/0 (socket says 172.21.15.70:60690) 2024-04-01T01:42:08.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.577+0000 7f798effd640 1 -- 172.21.15.70:0/1568426914 learned_addr learned my addr 172.21.15.70:0/1568426914 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:08.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.577+0000 7f798e7fc640 1 -- 172.21.15.70:0/1568426914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79900fffe0 msgr2=0x7f799019dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:08.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.577+0000 7f798e7fc640 1 --2- 172.21.15.70:0/1568426914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79900fffe0 0x7f799019dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.577+0000 7f798e7fc640 1 -- 172.21.15.70:0/1568426914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79901012e0 msgr2=0x7f79901a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:08.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.577+0000 7f798e7fc640 1 --2- 172.21.15.70:0/1568426914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79901012e0 0x7f79901a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:08.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.577+0000 7f798e7fc640 1 -- 172.21.15.70:0/1568426914 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79901a5c10 con 0x7f7990100940 2024-04-01T01:42:08.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.577+0000 7f798e7fc640 1 --2- 172.21.15.70:0/1568426914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7990100940 0x7f799019e530 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f798400e9f0 tx=0x7f798400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:08.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.578+0000 7f7977fff640 1 -- 172.21.15.70:0/1568426914 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798400cd80 con 0x7f7990100940 2024-04-01T01:42:08.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.578+0000 7f7977fff640 1 -- 172.21.15.70:0/1568426914 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79840035f0 con 0x7f7990100940 2024-04-01T01:42:08.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.578+0000 7f7977fff640 1 -- 172.21.15.70:0/1568426914 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7984010670 con 0x7f7990100940 2024-04-01T01:42:08.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.578+0000 7f79952f4640 1 -- 172.21.15.70:0/1568426914 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79901a5f00 con 0x7f7990100940 2024-04-01T01:42:08.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.579+0000 7f79952f4640 1 -- 172.21.15.70:0/1568426914 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79901a6440 con 0x7f7990100940 2024-04-01T01:42:08.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.580+0000 7f79952f4640 1 -- 172.21.15.70:0/1568426914 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7990107dd0 con 0x7f7990100940 2024-04-01T01:42:08.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.582+0000 7f7977fff640 1 -- 172.21.15.70:0/1568426914 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f79840037a0 con 0x7f7990100940 2024-04-01T01:42:08.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.583+0000 7f7977fff640 1 --2- 172.21.15.70:0/1568426914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79780a2620 0x7f79780a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:08.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.583+0000 7f7977fff640 1 -- 172.21.15.70:0/1568426914 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f79840ced70 con 0x7f7990100940 2024-04-01T01:42:08.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.583+0000 7f798effd640 1 --2- 172.21.15.70:0/1568426914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79780a2620 0x7f79780a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:08.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.584+0000 7f798effd640 1 --2- 172.21.15.70:0/1568426914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79780a2620 0x7f79780a4ae0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7980005ab0 tx=0x7f7980005a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:08.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:08.591+0000 7f7977fff640 1 -- 172.21.15.70:0/1568426914 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f798409bb00 con 0x7f7990100940 2024-04-01T01:42:09.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.018+0000 7f79952f4640 1 -- 172.21.15.70:0/1568426914 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7990062350 con 0x7f7990100940 2024-04-01T01:42:09.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.018+0000 7f7977fff640 1 -- 172.21.15.70:0/1568426914 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7f798409b400 con 0x7f7990100940 2024-04-01T01:42:09.022 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:42:09.022 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:42:09.024 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:42:09.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.022+0000 7f79952f4640 1 -- 172.21.15.70:0/1568426914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79780a2620 msgr2=0x7f79780a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.022+0000 7f79952f4640 1 --2- 172.21.15.70:0/1568426914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79780a2620 0x7f79780a4ae0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7980005ab0 tx=0x7f7980005a40 comp rx=0 tx=0).stop 2024-04-01T01:42:09.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.023+0000 7f79952f4640 1 -- 172.21.15.70:0/1568426914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7990100940 msgr2=0x7f799019e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.023+0000 7f79952f4640 1 --2- 172.21.15.70:0/1568426914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7990100940 0x7f799019e530 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f798400e9f0 tx=0x7f798400eeb0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.023+0000 7f79952f4640 1 -- 172.21.15.70:0/1568426914 shutdown_connections 2024-04-01T01:42:09.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.023+0000 7f79952f4640 1 --2- 172.21.15.70:0/1568426914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79901012e0 0x7f79901a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.023+0000 7f79952f4640 1 --2- 172.21.15.70:0/1568426914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79780a2620 0x7f79780a4ae0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.023+0000 7f79952f4640 1 --2- 172.21.15.70:0/1568426914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7990100940 0x7f799019e530 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.023+0000 7f79952f4640 1 --2- 172.21.15.70:0/1568426914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79900fffe0 0x7f799019dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.023+0000 7f79952f4640 1 -- 172.21.15.70:0/1568426914 >> 172.21.15.70:0/1568426914 conn(0x7f79900fbbe0 msgr2=0x7f79900fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.023+0000 7f79952f4640 1 -- 172.21.15.70:0/1568426914 shutdown_connections 2024-04-01T01:42:09.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.024+0000 7f79952f4640 1 -- 172.21.15.70:0/1568426914 wait complete. 2024-04-01T01:42:09.044 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:42:09.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.227+0000 7f0a25a22640 1 -- 172.21.15.70:0/1283398560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a20072d90 msgr2=0x7f0a20073210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.227+0000 7f0a25a22640 1 --2- 172.21.15.70:0/1283398560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a20072d90 0x7f0a20073210 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009ca0 tx=0x7f0a1802c100 comp rx=0 tx=0).stop 2024-04-01T01:42:09.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.227+0000 7f0a25a22640 1 -- 172.21.15.70:0/1283398560 shutdown_connections 2024-04-01T01:42:09.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.227+0000 7f0a25a22640 1 --2- 172.21.15.70:0/1283398560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a2010fb20 0x7f0a20111f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.227+0000 7f0a25a22640 1 --2- 172.21.15.70:0/1283398560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a20072d90 0x7f0a20073210 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.227+0000 7f0a25a22640 1 --2- 172.21.15.70:0/1283398560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a201093f0 0x7f0a200727c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.227+0000 7f0a25a22640 1 -- 172.21.15.70:0/1283398560 >> 172.21.15.70:0/1283398560 conn(0x7f0a2006e630 msgr2=0x7f0a20070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.229+0000 7f0a25a22640 1 -- 172.21.15.70:0/1283398560 shutdown_connections 2024-04-01T01:42:09.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.229+0000 7f0a25a22640 1 -- 172.21.15.70:0/1283398560 wait complete. 2024-04-01T01:42:09.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.230+0000 7f0a25a22640 1 Processor -- start 2024-04-01T01:42:09.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.230+0000 7f0a25a22640 1 -- start start 2024-04-01T01:42:09.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.230+0000 7f0a25a22640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a20072d90 0x7f0a2010e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.230+0000 7f0a25a22640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a201093f0 0x7f0a2010ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.230+0000 7f0a25a22640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a2010fb20 0x7f0a201a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.230+0000 7f0a25a22640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0a20115a50 con 0x7f0a201093f0 2024-04-01T01:42:09.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.230+0000 7f0a25a22640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0a201158d0 con 0x7f0a2010fb20 2024-04-01T01:42:09.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.230+0000 7f0a25a22640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a20115bd0 con 0x7f0a20072d90 2024-04-01T01:42:09.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.230+0000 7f0a24a20640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a20072d90 0x7f0a2010e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.231+0000 7f0a24a20640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a20072d90 0x7f0a2010e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:56734/0 (socket says 172.21.15.70:56734) 2024-04-01T01:42:09.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.231+0000 7f0a24a20640 1 -- 172.21.15.70:0/4031661026 learned_addr learned my addr 172.21.15.70:0/4031661026 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:09.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.231+0000 7f0a25221640 1 --2- 172.21.15.70:0/4031661026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a2010fb20 0x7f0a201a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.231+0000 7f0a1ffff640 1 --2- 172.21.15.70:0/4031661026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a201093f0 0x7f0a2010ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.231+0000 7f0a24a20640 1 -- 172.21.15.70:0/4031661026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a2010fb20 msgr2=0x7f0a201a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.231+0000 7f0a24a20640 1 --2- 172.21.15.70:0/4031661026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a2010fb20 0x7f0a201a55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.231+0000 7f0a24a20640 1 -- 172.21.15.70:0/4031661026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a201093f0 msgr2=0x7f0a2010ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.231+0000 7f0a24a20640 1 --2- 172.21.15.70:0/4031661026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a201093f0 0x7f0a2010ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.231+0000 7f0a24a20640 1 -- 172.21.15.70:0/4031661026 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a201a5bf0 con 0x7f0a20072d90 2024-04-01T01:42:09.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.231+0000 7f0a24a20640 1 --2- 172.21.15.70:0/4031661026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a20072d90 0x7f0a2010e500 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400b450 tx=0x7f0a1400b850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.232+0000 7f0a1dffb640 1 -- 172.21.15.70:0/4031661026 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a14015020 con 0x7f0a20072d90 2024-04-01T01:42:09.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.232+0000 7f0a25a22640 1 -- 172.21.15.70:0/4031661026 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a201a5ee0 con 0x7f0a20072d90 2024-04-01T01:42:09.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.232+0000 7f0a25a22640 1 -- 172.21.15.70:0/4031661026 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a201a6500 con 0x7f0a20072d90 2024-04-01T01:42:09.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.232+0000 7f0a1dffb640 1 -- 172.21.15.70:0/4031661026 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a14011ce0 con 0x7f0a20072d90 2024-04-01T01:42:09.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.232+0000 7f0a1dffb640 1 -- 172.21.15.70:0/4031661026 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a14015020 con 0x7f0a20072d90 2024-04-01T01:42:09.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.236+0000 7f0a29864740 1 Processor -- start 2024-04-01T01:42:09.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.236+0000 7f0a29864740 1 -- start start 2024-04-01T01:42:09.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.237+0000 7f0a29864740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33d6a0 0x560e6d33dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.237+0000 7f0a29864740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d33e090 0x560e6d342b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.237+0000 7f0a29864740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d343200 0x560e6d345690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.237+0000 7f0a29864740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560e6d2a21b0 con 0x560e6d33e090 2024-04-01T01:42:09.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.237+0000 7f0a29864740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560e6d24ac00 con 0x560e6d343200 2024-04-01T01:42:09.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.237+0000 7f0a29864740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560e6d29e250 con 0x560e6d33d6a0 2024-04-01T01:42:09.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.237+0000 7f0a25221640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33d6a0 0x560e6d33dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.237+0000 7f0a25221640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33d6a0 0x560e6d33dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:56740/0 (socket says 172.21.15.70:56740) 2024-04-01T01:42:09.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.237+0000 7f0a25221640 1 -- 172.21.15.70:0/1057854648 learned_addr learned my addr 172.21.15.70:0/1057854648 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:09.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.237+0000 7f0a1ffff640 1 --2- 172.21.15.70:0/1057854648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d343200 0x560e6d345690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.238+0000 7f0a1ffff640 1 -- 172.21.15.70:0/1057854648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33d6a0 msgr2=0x560e6d33dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.238+0000 7f0a1ffff640 1 --2- 172.21.15.70:0/1057854648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33d6a0 0x560e6d33dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.238+0000 7f0a1ffff640 1 -- 172.21.15.70:0/1057854648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d33e090 msgr2=0x560e6d342b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:42:09.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.238+0000 7f0a1ffff640 1 --2- 172.21.15.70:0/1057854648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d33e090 0x560e6d342b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.238+0000 7f0a1ffff640 1 -- 172.21.15.70:0/1057854648 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e6d26a3d0 con 0x560e6d343200 2024-04-01T01:42:09.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.238+0000 7f0a1ffff640 1 --2- 172.21.15.70:0/1057854648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d343200 0x560e6d345690 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0a1802c710 tx=0x7f0a18032bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de3990703190c675 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.239+0000 7f0a06ffd640 1 -- 172.21.15.70:0/1057854648 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18043020 con 0x560e6d343200 2024-04-01T01:42:09.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.239+0000 7f0a06ffd640 1 -- 172.21.15.70:0/1057854648 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a18003970 con 0x560e6d343200 2024-04-01T01:42:09.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.239+0000 7f0a06ffd640 1 -- 172.21.15.70:0/1057854648 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18039040 con 0x560e6d343200 2024-04-01T01:42:09.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.241+0000 7f0a29864740 1 -- 172.21.15.70:0/1057854648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d343200 msgr2=0x560e6d345690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.241+0000 7f0a29864740 1 --2- 172.21.15.70:0/1057854648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d343200 0x560e6d345690 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0a1802c710 tx=0x7f0a18032bc0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.241+0000 7f0a1dffb640 1 -- 172.21.15.70:0/4031661026 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f0a14018020 con 0x7f0a20072d90 2024-04-01T01:42:09.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.242+0000 7f0a1dffb640 1 --2- 172.21.15.70:0/4031661026 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a080a27e0 0x7f0a080a4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.242+0000 7f0a1dffb640 1 -- 172.21.15.70:0/4031661026 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f0a140cdee0 con 0x7f0a20072d90 2024-04-01T01:42:09.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.243+0000 7f0a25221640 1 --2- 172.21.15.70:0/4031661026 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a080a27e0 0x7f0a080a4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.243+0000 7f0a29864740 1 -- 172.21.15.70:0/1057854648 shutdown_connections 2024-04-01T01:42:09.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.243+0000 7f0a29864740 1 --2- 172.21.15.70:0/1057854648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d343200 0x560e6d345690 secure :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0a1802c710 tx=0x7f0a18032bc0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.243+0000 7f0a29864740 1 --2- 172.21.15.70:0/1057854648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d33e090 0x560e6d342b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.243+0000 7f0a29864740 1 --2- 172.21.15.70:0/1057854648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33d6a0 0x560e6d33dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.243+0000 7f0a29864740 1 -- 172.21.15.70:0/1057854648 >> 172.21.15.70:0/1057854648 conn(0x560e6d338c10 msgr2=0x560e6d339010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.243+0000 7f0a29864740 1 -- 172.21.15.70:0/1057854648 shutdown_connections 2024-04-01T01:42:09.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.245+0000 7f0a29864740 1 -- 172.21.15.70:0/1057854648 wait complete. 2024-04-01T01:42:09.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.246+0000 7f0a29864740 1 -- start start 2024-04-01T01:42:09.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.246+0000 7f0a25221640 1 --2- 172.21.15.70:0/4031661026 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a080a27e0 0x7f0a080a4ca0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x560e6d33e8f0 tx=0x7f0a10005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.246+0000 7f0a29864740 1 Processor -- start 2024-04-01T01:42:09.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.247+0000 7f0a29864740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d33d6a0 0x560e6d347d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.247+0000 7f0a29864740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d33e090 0x560e6d3482b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.247+0000 7f0a29864740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d34d2a0 0x560e6d34f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.247+0000 7f0a1ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d33e090 0x560e6d3482b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.247+0000 7f0a1ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d33e090 0x560e6d3482b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40006/0 (socket says 172.21.15.70:40006) 2024-04-01T01:42:09.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.247+0000 7f0a1ffff640 1 -- 172.21.15.70:0/3492544620 learned_addr learned my addr 172.21.15.70:0/3492544620 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:09.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.247+0000 7f0a29864740 1 -- 172.21.15.70:0/3492544620 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560e6d346150 con 0x560e6d33e090 2024-04-01T01:42:09.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.248+0000 7f0a29864740 1 -- 172.21.15.70:0/3492544620 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560e6d240470 con 0x560e6d33d6a0 2024-04-01T01:42:09.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.248+0000 7f0a25221640 1 --2- 172.21.15.70:0/3492544620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d34d2a0 0x560e6d34f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.248+0000 7f0a29864740 1 -- 172.21.15.70:0/3492544620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560e6d3462d0 con 0x560e6d34d2a0 2024-04-01T01:42:09.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.248+0000 7f0a29864740 10 client.0 _ll_register_callbacks cb 0x560e6d25dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:09.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.248+0000 7f0a25221640 1 -- 172.21.15.70:0/3492544620 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d33d6a0 msgr2=0x560e6d347d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:42:09.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.248+0000 7f0a25221640 1 --2- 172.21.15.70:0/3492544620 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d33d6a0 0x560e6d347d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.248+0000 7f0a25221640 1 -- 172.21.15.70:0/3492544620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d33e090 msgr2=0x560e6d3482b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.248+0000 7f0a25221640 1 --2- 172.21.15.70:0/3492544620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d33e090 0x560e6d3482b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.248+0000 7f0a25221640 1 -- 172.21.15.70:0/3492544620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e6cfb63f0 con 0x560e6d34d2a0 2024-04-01T01:42:09.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:08 smithi189 ceph-mon[35753]: pgmap v619: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.8 MiB/s wr, 779 op/s 2024-04-01T01:42:09.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.249+0000 7f0a25221640 1 --2- 172.21.15.70:0/3492544620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d34d2a0 0x560e6d34f690 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0a10004620 tx=0x7f0a10004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b36f7201e590bfd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.251+0000 7f09fffff640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:42:09.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.251+0000 7f09fffff640 1 -- 172.21.15.70:0/3492544620 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1001f640 con 0x560e6d34d2a0 2024-04-01T01:42:09.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.252+0000 7f0a29864740 1 -- 172.21.15.70:0/3492544620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a18003970 con 0x560e6d34d2a0 2024-04-01T01:42:09.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.252+0000 7f09fffff640 1 -- 172.21.15.70:0/3492544620 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f0a1001f7e0 con 0x560e6d34d2a0 2024-04-01T01:42:09.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.252+0000 7f09fffff640 1 -- 172.21.15.70:0/3492544620 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0a10032050 con 0x560e6d34d2a0 2024-04-01T01:42:09.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.252+0000 7f0a29864740 10 client.27835 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:09.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.252+0000 7f0a29864740 1 -- 172.21.15.70:0/3492544620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560e6d0807d0 con 0x560e6d34d2a0 2024-04-01T01:42:09.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.252+0000 7f09fffff640 1 -- 172.21.15.70:0/3492544620 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a10029440 con 0x560e6d34d2a0 2024-04-01T01:42:09.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.253+0000 7f09fffff640 1 -- 172.21.15.70:0/3492544620 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f0a10027020 con 0x560e6d34d2a0 2024-04-01T01:42:09.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.254+0000 7f0a29864740 10 client.27835 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:09.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.254+0000 7f0a29864740 10 client.27835 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:42:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.254+0000 7f0a29864740 20 client.27835 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.254+0000 7f0a29864740 1 --2- 172.21.15.70:0/3492544620 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x560e6d354170 0x560e6d356610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.254+0000 7f0a29864740 4 client.27835 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.254+0000 7f0a29864740 1 -- 172.21.15.70:0/3492544620 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0a18003970 con 0x560e6d354170 2024-04-01T01:42:09.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:08 smithi070 ceph-mon[32179]: pgmap v619: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.8 MiB/s wr, 779 op/s 2024-04-01T01:42:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.254+0000 7f0a24a20640 1 --2- 172.21.15.70:0/3492544620 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x560e6d354170 0x560e6d356610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.256+0000 7f0a24a20640 1 --2- 172.21.15.70:0/3492544620 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x560e6d354170 0x560e6d356610 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=32deb116a8fe6b7b server_cookie=8767944224d0f783 in_seq=0 out_seq=0 2024-04-01T01:42:09.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.256+0000 7f09fffff640 10 client.27835 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:42:09.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.259+0000 7f09fffff640 1 -- 172.21.15.70:0/3492544620 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f0a18003970 con 0x560e6d354170 2024-04-01T01:42:09.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.259+0000 7f09fffff640 10 client.27835 handle_command_reply: tid=0 2024-04-01T01:42:09.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.261+0000 7f0a29864740 1 client.27835 shutdown 2024-04-01T01:42:09.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.265+0000 7f0a29864740 1 -- 172.21.15.70:0/3492544620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d34d2a0 msgr2=0x560e6d34f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.265+0000 7f0a29864740 1 --2- 172.21.15.70:0/3492544620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d34d2a0 0x560e6d34f690 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0a10004620 tx=0x7f0a10004730 comp rx=0 tx=0).stop 2024-04-01T01:42:09.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.265+0000 7f0a29864740 1 -- 172.21.15.70:0/3492544620 shutdown_connections 2024-04-01T01:42:09.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.265+0000 7f0a29864740 1 --2- 172.21.15.70:0/3492544620 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x560e6d354170 0x560e6d356610 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.265+0000 7f0a29864740 1 --2- 172.21.15.70:0/3492544620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d34d2a0 0x560e6d34f690 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.265+0000 7f0a29864740 1 --2- 172.21.15.70:0/3492544620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d33e090 0x560e6d3482b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.265+0000 7f0a29864740 1 --2- 172.21.15.70:0/3492544620 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d33d6a0 0x560e6d347d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.265+0000 7f0a29864740 1 -- 172.21.15.70:0/3492544620 >> 172.21.15.70:0/3492544620 conn(0x560e6d338c10 msgr2=0x560e6d337f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.266+0000 7f09fffff640 0 client.27835 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:42:09.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.268+0000 7f0a29864740 1 -- 172.21.15.70:0/3492544620 shutdown_connections 2024-04-01T01:42:09.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.269+0000 7f0a29864740 1 -- 172.21.15.70:0/3492544620 wait complete. 2024-04-01T01:42:09.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.273+0000 7f0a29864740 20 client.27835 trim_cache size 0 max 16384 2024-04-01T01:42:09.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.310+0000 7f0a29864740 1 Processor -- start 2024-04-01T01:42:09.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.310+0000 7f0a29864740 1 -- start start 2024-04-01T01:42:09.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.311+0000 7f0a29864740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33d6a0 0x560e6d33f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.311+0000 7f0a29864740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d33e090 0x560e6d33fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.311+0000 7f0a29864740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d34d2a0 0x560e6d347d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.311+0000 7f0a24a20640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33d6a0 0x560e6d33f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.311+0000 7f0a29864740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0c00fc80 con 0x560e6d34d2a0 2024-04-01T01:42:09.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.311+0000 7f0a24a20640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33d6a0 0x560e6d33f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:56758/0 (socket says 172.21.15.70:56758) 2024-04-01T01:42:09.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.311+0000 7f0a29864740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0c0117c0 con 0x560e6d33e090 2024-04-01T01:42:09.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.311+0000 7f0a24a20640 1 -- 172.21.15.70:0/3310453551 learned_addr learned my addr 172.21.15.70:0/3310453551 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:09.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.311+0000 7f0a29864740 1 -- 172.21.15.70:0/3310453551 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0c010350 con 0x560e6d33d6a0 2024-04-01T01:42:09.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.311+0000 7f0a25221640 1 --2- 172.21.15.70:0/3310453551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d34d2a0 0x560e6d347d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.312+0000 7f0a24a20640 1 -- 172.21.15.70:0/3310453551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d33e090 msgr2=0x560e6d33fcb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:42:09.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.312+0000 7f0a24a20640 1 --2- 172.21.15.70:0/3310453551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d33e090 0x560e6d33fcb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.312+0000 7f0a24a20640 1 -- 172.21.15.70:0/3310453551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d34d2a0 msgr2=0x560e6d347d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.312+0000 7f0a24a20640 1 --2- 172.21.15.70:0/3310453551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d34d2a0 0x560e6d347d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.312+0000 7f0a24a20640 1 -- 172.21.15.70:0/3310453551 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e6d329e60 con 0x560e6d33d6a0 2024-04-01T01:42:09.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.312+0000 7f0a24a20640 1 --2- 172.21.15.70:0/3310453551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33d6a0 0x560e6d33f770 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400bd50 tx=0x7f0a14000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2478cd7b5c11ff43 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.312+0000 7f0a067fc640 1 -- 172.21.15.70:0/3310453551 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1409a570 con 0x560e6d33d6a0 2024-04-01T01:42:09.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.312+0000 7f0a067fc640 1 -- 172.21.15.70:0/3310453551 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a1409e6c0 con 0x560e6d33d6a0 2024-04-01T01:42:09.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.312+0000 7f0a067fc640 1 -- 172.21.15.70:0/3310453551 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a140a2af0 con 0x560e6d33d6a0 2024-04-01T01:42:09.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.313+0000 7f0a29864740 1 -- 172.21.15.70:0/3310453551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33d6a0 msgr2=0x560e6d33f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.313+0000 7f0a29864740 1 --2- 172.21.15.70:0/3310453551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33d6a0 0x560e6d33f770 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400bd50 tx=0x7f0a14000bc0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.313+0000 7f0a29864740 1 -- 172.21.15.70:0/3310453551 shutdown_connections 2024-04-01T01:42:09.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.313+0000 7f0a29864740 1 --2- 172.21.15.70:0/3310453551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d34d2a0 0x560e6d347d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.313+0000 7f0a29864740 1 --2- 172.21.15.70:0/3310453551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d33e090 0x560e6d33fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.313+0000 7f0a29864740 1 --2- 172.21.15.70:0/3310453551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33d6a0 0x560e6d33f770 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.313+0000 7f0a29864740 1 -- 172.21.15.70:0/3310453551 >> 172.21.15.70:0/3310453551 conn(0x560e6d338c10 msgr2=0x560e6d356180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.313+0000 7f0a29864740 1 -- 172.21.15.70:0/3310453551 shutdown_connections 2024-04-01T01:42:09.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.314+0000 7f0a29864740 1 -- 172.21.15.70:0/3310453551 wait complete. 2024-04-01T01:42:09.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.314+0000 7f0a29864740 1 -- start start 2024-04-01T01:42:09.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.314+0000 7f0a29864740 1 Processor -- start 2024-04-01T01:42:09.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.315+0000 7f0a29864740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d33d6a0 0x560e6d364dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.315+0000 7f0a29864740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33e090 0x560e6d365300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.315+0000 7f0a29864740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d34d2a0 0x560e6d3662c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.315+0000 7f0a29864740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0c0110f0 con 0x560e6d33d6a0 2024-04-01T01:42:09.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.315+0000 7f0a29864740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0c010a20 con 0x560e6d34d2a0 2024-04-01T01:42:09.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.315+0000 7f0a29864740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0c00dc60 con 0x560e6d33e090 2024-04-01T01:42:09.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.315+0000 7f0a29864740 10 client.0 _ll_register_callbacks cb 0x7f0a100044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:09.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.315+0000 7f0a1ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33e090 0x560e6d365300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.315+0000 7f0a1ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33e090 0x560e6d365300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:56764/0 (socket says 172.21.15.70:56764) 2024-04-01T01:42:09.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.315+0000 7f0a1ffff640 1 -- 172.21.15.70:0/4134682786 learned_addr learned my addr 172.21.15.70:0/4134682786 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:09.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.315+0000 7f0a24a20640 1 --2- 172.21.15.70:0/4134682786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d33d6a0 0x560e6d364dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.316+0000 7f0a1ffff640 1 -- 172.21.15.70:0/4134682786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d34d2a0 msgr2=0x560e6d3662c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:42:09.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.316+0000 7f0a1ffff640 1 --2- 172.21.15.70:0/4134682786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d34d2a0 0x560e6d3662c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.316+0000 7f0a1ffff640 1 -- 172.21.15.70:0/4134682786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d33d6a0 msgr2=0x560e6d364dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.316+0000 7f0a1ffff640 1 --2- 172.21.15.70:0/4134682786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d33d6a0 0x560e6d364dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.316+0000 7f0a1ffff640 1 -- 172.21.15.70:0/4134682786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e6d3668b0 con 0x560e6d33e090 2024-04-01T01:42:09.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.316+0000 7f0a1ffff640 1 --2- 172.21.15.70:0/4134682786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33e090 0x560e6d365300 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009ca0 tx=0x7f0a18032a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b77896c7be29977 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.316+0000 7f09fffff640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:42:09.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.316+0000 7f09fffff640 1 -- 172.21.15.70:0/4134682786 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1804c020 con 0x560e6d33e090 2024-04-01T01:42:09.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.317+0000 7f09fffff640 1 -- 172.21.15.70:0/4134682786 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f0a18003b70 con 0x560e6d33e090 2024-04-01T01:42:09.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.317+0000 7f0a29864740 1 -- 172.21.15.70:0/4134682786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a1409e6c0 con 0x560e6d33e090 2024-04-01T01:42:09.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.317+0000 7f09fffff640 1 -- 172.21.15.70:0/4134682786 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0a18043070 con 0x560e6d33e090 2024-04-01T01:42:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.318+0000 7f0a29864740 10 client.27841 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.318+0000 7f0a29864740 1 -- 172.21.15.70:0/4134682786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560e6d3b9f00 con 0x560e6d33e090 2024-04-01T01:42:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.318+0000 7f09fffff640 1 -- 172.21.15.70:0/4134682786 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a1409e6c0 con 0x560e6d33e090 2024-04-01T01:42:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.318+0000 7f09fffff640 1 -- 172.21.15.70:0/4134682786 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f0a1803f470 con 0x560e6d33e090 2024-04-01T01:42:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.318+0000 7f0a29864740 10 client.27841 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.318+0000 7f0a29864740 10 client.27841 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:42:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.318+0000 7f0a29864740 20 client.27841 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.318+0000 7f0a29864740 1 --2- 172.21.15.70:0/4134682786 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x560e6d3ba8d0 0x560e6d3bccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.318+0000 7f0a29864740 4 client.27841 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:42:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.318+0000 7f0a29864740 1 -- 172.21.15.70:0/4134682786 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560e6d2af6f0 con 0x560e6d3ba8d0 2024-04-01T01:42:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.319+0000 7f0a24a20640 1 --2- 172.21.15.70:0/4134682786 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x560e6d3ba8d0 0x560e6d3bccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.320+0000 7f09fffff640 10 client.27841 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:42:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.320+0000 7f0a24a20640 1 --2- 172.21.15.70:0/4134682786 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x560e6d3ba8d0 0x560e6d3bccf0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=512c95614757bb7 server_cookie=1269dd0c7eb79fa7 in_seq=0 out_seq=0 2024-04-01T01:42:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.320+0000 7f09fffff640 1 -- 172.21.15.70:0/4134682786 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560e6d2af6f0 con 0x560e6d3ba8d0 2024-04-01T01:42:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.320+0000 7f09fffff640 10 client.27841 handle_command_reply: tid=0 2024-04-01T01:42:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.321+0000 7f0a29864740 1 client.27841 shutdown 2024-04-01T01:42:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.321+0000 7f0a29864740 1 -- 172.21.15.70:0/4134682786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33e090 msgr2=0x560e6d365300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.321+0000 7f0a29864740 1 --2- 172.21.15.70:0/4134682786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33e090 0x560e6d365300 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009ca0 tx=0x7f0a18032a50 comp rx=0 tx=0).stop 2024-04-01T01:42:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.321+0000 7f0a29864740 1 -- 172.21.15.70:0/4134682786 shutdown_connections 2024-04-01T01:42:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.321+0000 7f0a29864740 1 --2- 172.21.15.70:0/4134682786 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x560e6d3ba8d0 0x560e6d3bccf0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.321+0000 7f0a29864740 1 --2- 172.21.15.70:0/4134682786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560e6d34d2a0 0x560e6d3662c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.321+0000 7f09fffff640 0 client.27841 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:42:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.321+0000 7f0a29864740 1 --2- 172.21.15.70:0/4134682786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e6d33e090 0x560e6d365300 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.321+0000 7f0a29864740 1 --2- 172.21.15.70:0/4134682786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560e6d33d6a0 0x560e6d364dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.321+0000 7f0a29864740 1 -- 172.21.15.70:0/4134682786 >> 172.21.15.70:0/4134682786 conn(0x560e6d338c10 msgr2=0x560e6d33f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.321+0000 7f0a29864740 1 -- 172.21.15.70:0/4134682786 shutdown_connections 2024-04-01T01:42:09.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.321+0000 7f0a29864740 1 -- 172.21.15.70:0/4134682786 wait complete. 2024-04-01T01:42:09.322 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:42:09.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.321+0000 7f0a29864740 20 client.27841 trim_cache size 0 max 16384 2024-04-01T01:42:09.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.322+0000 7f0a25a22640 1 -- 172.21.15.70:0/4031661026 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a080a27e0 msgr2=0x7f0a080a4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.322+0000 7f0a25a22640 1 --2- 172.21.15.70:0/4031661026 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a080a27e0 0x7f0a080a4ca0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x560e6d33e8f0 tx=0x7f0a10005f90 comp rx=0 tx=0).stop 2024-04-01T01:42:09.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.322+0000 7f0a25a22640 1 -- 172.21.15.70:0/4031661026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a20072d90 msgr2=0x7f0a2010e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.322+0000 7f0a25a22640 1 --2- 172.21.15.70:0/4031661026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a20072d90 0x7f0a2010e500 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400b450 tx=0x7f0a1400b850 comp rx=0 tx=0).stop 2024-04-01T01:42:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.322+0000 7f0a25a22640 1 -- 172.21.15.70:0/4031661026 shutdown_connections 2024-04-01T01:42:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.323+0000 7f0a25a22640 1 --2- 172.21.15.70:0/4031661026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a2010fb20 0x7f0a201a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.323+0000 7f0a25a22640 1 --2- 172.21.15.70:0/4031661026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a201093f0 0x7f0a2010ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.323+0000 7f0a25a22640 1 --2- 172.21.15.70:0/4031661026 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0a080a27e0 0x7f0a080a4ca0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.323+0000 7f0a25a22640 1 --2- 172.21.15.70:0/4031661026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a20072d90 0x7f0a2010e500 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.323+0000 7f0a25a22640 1 -- 172.21.15.70:0/4031661026 >> 172.21.15.70:0/4031661026 conn(0x7f0a2006e630 msgr2=0x7f0a20116420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.323+0000 7f0a25a22640 1 -- 172.21.15.70:0/4031661026 shutdown_connections 2024-04-01T01:42:09.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.323+0000 7f0a25a22640 1 -- 172.21.15.70:0/4031661026 wait complete. 2024-04-01T01:42:09.327 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:08 smithi081 ceph-mon[35142]: pgmap v619: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.8 MiB/s wr, 779 op/s 2024-04-01T01:42:09.344 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:42:09.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.525+0000 7f2fd24d4640 1 -- 172.21.15.70:0/142895341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fc4099910 msgr2=0x7f2fc4099d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.525+0000 7f2fd24d4640 1 --2- 172.21.15.70:0/142895341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fc4099910 0x7f2fc4099d90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc04f780 tx=0x7f2fcc0977d0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.525+0000 7f2fd24d4640 1 -- 172.21.15.70:0/142895341 shutdown_connections 2024-04-01T01:42:09.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.525+0000 7f2fd24d4640 1 --2- 172.21.15.70:0/142895341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fc4099910 0x7f2fc4099d90 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.525+0000 7f2fd24d4640 1 --2- 172.21.15.70:0/142895341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fc4096f60 0x7f2fc40993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.525+0000 7f2fd24d4640 1 --2- 172.21.15.70:0/142895341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fc40945f0 0x7f2fc4096a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.525+0000 7f2fd24d4640 1 -- 172.21.15.70:0/142895341 >> 172.21.15.70:0/142895341 conn(0x7f2fc40901d0 msgr2=0x7f2fc4092630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.526+0000 7f2fd24d4640 1 -- 172.21.15.70:0/142895341 shutdown_connections 2024-04-01T01:42:09.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.526+0000 7f2fd24d4640 1 -- 172.21.15.70:0/142895341 wait complete. 2024-04-01T01:42:09.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.528+0000 7f2fd24d4640 1 Processor -- start 2024-04-01T01:42:09.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.528+0000 7f2fd24d4640 1 -- start start 2024-04-01T01:42:09.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.529+0000 7f2fd24d4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fc40945f0 0x7f2fc4132470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.529+0000 7f2fd24d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fc4096f60 0x7f2fc41329b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.529+0000 7f2fd24d4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fc4099910 0x7f2fc4139a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.529+0000 7f2fd24d4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc40a7e80 con 0x7f2fc40945f0 2024-04-01T01:42:09.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.529+0000 7f2fd24d4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc40a7d00 con 0x7f2fc4096f60 2024-04-01T01:42:09.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.529+0000 7f2fd24d4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc40a8000 con 0x7f2fc4099910 2024-04-01T01:42:09.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.529+0000 7f2fcb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fc4096f60 0x7f2fc41329b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.529+0000 7f2fcb7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fc4096f60 0x7f2fc41329b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60748/0 (socket says 172.21.15.70:60748) 2024-04-01T01:42:09.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.529+0000 7f2fcb7fe640 1 -- 172.21.15.70:0/3006055550 learned_addr learned my addr 172.21.15.70:0/3006055550 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:09.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.529+0000 7f2fcbfff640 1 --2- 172.21.15.70:0/3006055550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fc40945f0 0x7f2fc4132470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.530+0000 7f2fcb7fe640 1 -- 172.21.15.70:0/3006055550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fc4099910 msgr2=0x7f2fc4139a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.530+0000 7f2fcb7fe640 1 --2- 172.21.15.70:0/3006055550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fc4099910 0x7f2fc4139a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.530+0000 7f2fcb7fe640 1 -- 172.21.15.70:0/3006055550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fc40945f0 msgr2=0x7f2fc4132470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.530+0000 7f2fcb7fe640 1 --2- 172.21.15.70:0/3006055550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fc40945f0 0x7f2fc4132470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.530+0000 7f2fcb7fe640 1 -- 172.21.15.70:0/3006055550 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc413a090 con 0x7f2fc4096f60 2024-04-01T01:42:09.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.530+0000 7f2fcb7fe640 1 --2- 172.21.15.70:0/3006055550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fc4096f60 0x7f2fc41329b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800e9c0 tx=0x7f2fb800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.530+0000 7f2fc97fa640 1 -- 172.21.15.70:0/3006055550 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb800ccc0 con 0x7f2fc4096f60 2024-04-01T01:42:09.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.530+0000 7f2fd24d4640 1 -- 172.21.15.70:0/3006055550 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fc413a380 con 0x7f2fc4096f60 2024-04-01T01:42:09.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.530+0000 7f2fd24d4640 1 -- 172.21.15.70:0/3006055550 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fc413a8d0 con 0x7f2fc4096f60 2024-04-01T01:42:09.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.531+0000 7f2fc97fa640 1 -- 172.21.15.70:0/3006055550 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2fb80054d0 con 0x7f2fc4096f60 2024-04-01T01:42:09.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.531+0000 7f2fc97fa640 1 -- 172.21.15.70:0/3006055550 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8010430 con 0x7f2fc4096f60 2024-04-01T01:42:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.534+0000 7f2fc97fa640 1 -- 172.21.15.70:0/3006055550 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f2fb80046c0 con 0x7f2fc4096f60 2024-04-01T01:42:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.535+0000 7f2fc97fa640 1 --2- 172.21.15.70:0/3006055550 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2fbc0a2670 0x7f2fbc0a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.535+0000 7f2fc97fa640 1 -- 172.21.15.70:0/3006055550 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f2fb80cedf0 con 0x7f2fc4096f60 2024-04-01T01:42:09.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.536+0000 7f2fcbfff640 1 --2- 172.21.15.70:0/3006055550 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2fbc0a2670 0x7f2fbc0a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.537+0000 7f2fd518e740 1 Processor -- start 2024-04-01T01:42:09.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.537+0000 7f2fd518e740 1 -- start start 2024-04-01T01:42:09.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.538+0000 7f2fd518e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8bdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.538+0000 7f2fd518e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8be090 0x55ee8e8c2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.538+0000 7f2fd518e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8c3200 0x55ee8e8c5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.538+0000 7f2fd518e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ee8e8221b0 con 0x55ee8e8c3200 2024-04-01T01:42:09.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.538+0000 7f2fd518e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ee8e7cac00 con 0x55ee8e8be090 2024-04-01T01:42:09.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.538+0000 7f2fd518e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ee8e81e250 con 0x55ee8e8bd6a0 2024-04-01T01:42:09.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.539+0000 7f2fcbfff640 1 --2- 172.21.15.70:0/3006055550 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2fbc0a2670 0x7f2fbc0a4b30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0005ef0 tx=0x7f2fc0005e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.539+0000 7f2fd0a71640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8c3200 0x55ee8e8c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.539+0000 7f2fd0a71640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8c3200 0x55ee8e8c5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40052/0 (socket says 172.21.15.70:40052) 2024-04-01T01:42:09.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.539+0000 7f2fd0a71640 1 -- 172.21.15.70:0/3357626985 learned_addr learned my addr 172.21.15.70:0/3357626985 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:09.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.539+0000 7f2fcbfff640 1 --2- 172.21.15.70:0/3357626985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8bdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.541+0000 7f2fd0a71640 1 -- 172.21.15.70:0/3357626985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8bd6a0 msgr2=0x55ee8e8bdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.541+0000 7f2fd0a71640 1 --2- 172.21.15.70:0/3357626985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8bdac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.541+0000 7f2fd0a71640 1 -- 172.21.15.70:0/3357626985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8be090 msgr2=0x55ee8e8c2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.540+0000 7f2fcb7fe640 1 --2- 172.21.15.70:0/3357626985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8be090 0x55ee8e8c2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.541+0000 7f2fd0a71640 1 --2- 172.21.15.70:0/3357626985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8be090 0x55ee8e8c2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.541+0000 7f2fd0a71640 1 -- 172.21.15.70:0/3357626985 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee8e7ea3d0 con 0x55ee8e8c3200 2024-04-01T01:42:09.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.541+0000 7f2fcb7fe640 1 --2- 172.21.15.70:0/3357626985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8be090 0x55ee8e8c2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:42:09.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.542+0000 7f2fd0a71640 1 --2- 172.21.15.70:0/3357626985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8c3200 0x55ee8e8c5690 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc050c90 tx=0x7f2fcc067920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3e879a697b1709 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.542+0000 7f2fb67fc640 1 -- 172.21.15.70:0/3357626985 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc0a4040 con 0x55ee8e8c3200 2024-04-01T01:42:09.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.542+0000 7f2fb67fc640 1 -- 172.21.15.70:0/3357626985 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2fcc067bd0 con 0x55ee8e8c3200 2024-04-01T01:42:09.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.542+0000 7f2fb67fc640 1 -- 172.21.15.70:0/3357626985 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc069330 con 0x55ee8e8c3200 2024-04-01T01:42:09.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.543+0000 7f2fd518e740 1 -- 172.21.15.70:0/3357626985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8c3200 msgr2=0x55ee8e8c5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.543+0000 7f2fd518e740 1 --2- 172.21.15.70:0/3357626985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8c3200 0x55ee8e8c5690 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc050c90 tx=0x7f2fcc067920 comp rx=0 tx=0).stop 2024-04-01T01:42:09.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.543+0000 7f2fd518e740 1 -- 172.21.15.70:0/3357626985 shutdown_connections 2024-04-01T01:42:09.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.543+0000 7f2fd518e740 1 --2- 172.21.15.70:0/3357626985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8c3200 0x55ee8e8c5690 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.543+0000 7f2fd518e740 1 --2- 172.21.15.70:0/3357626985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8be090 0x55ee8e8c2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.543+0000 7f2fd518e740 1 --2- 172.21.15.70:0/3357626985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8bdac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.543+0000 7f2fd518e740 1 -- 172.21.15.70:0/3357626985 >> 172.21.15.70:0/3357626985 conn(0x55ee8e8b8c10 msgr2=0x55ee8e8b9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.543+0000 7f2fd518e740 1 -- 172.21.15.70:0/3357626985 shutdown_connections 2024-04-01T01:42:09.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.543+0000 7f2fd518e740 1 -- 172.21.15.70:0/3357626985 wait complete. 2024-04-01T01:42:09.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.544+0000 7f2fd518e740 1 -- start start 2024-04-01T01:42:09.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.544+0000 7f2fd518e740 1 Processor -- start 2024-04-01T01:42:09.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.544+0000 7f2fd518e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8d8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.544+0000 7f2fd518e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8be090 0x55ee8e8d90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.544+0000 7f2fd518e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8c3200 0x55ee8e8e0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.544+0000 7f2fd518e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ee8e8c6150 con 0x55ee8e8c3200 2024-04-01T01:42:09.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.544+0000 7f2fd518e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ee8e7c0470 con 0x55ee8e8be090 2024-04-01T01:42:09.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.544+0000 7f2fd518e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ee8e8c62d0 con 0x55ee8e8bd6a0 2024-04-01T01:42:09.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.545+0000 7f2fd518e740 10 client.0 _ll_register_callbacks cb 0x55ee8e7ddcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:09.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.545+0000 7f2fcbfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8d8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.545+0000 7f2fd0a71640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8c3200 0x55ee8e8e0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.545+0000 7f2fd0a71640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8c3200 0x55ee8e8e0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40064/0 (socket says 172.21.15.70:40064) 2024-04-01T01:42:09.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.545+0000 7f2fd0a71640 1 -- 172.21.15.70:0/3173085691 learned_addr learned my addr 172.21.15.70:0/3173085691 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:09.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.545+0000 7f2fcb7fe640 1 --2- 172.21.15.70:0/3173085691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8be090 0x55ee8e8d90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.546+0000 7f2fd0a71640 1 -- 172.21.15.70:0/3173085691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8bd6a0 msgr2=0x55ee8e8d8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.546+0000 7f2fd0a71640 1 --2- 172.21.15.70:0/3173085691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8d8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.546+0000 7f2fd0a71640 1 -- 172.21.15.70:0/3173085691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8be090 msgr2=0x55ee8e8d90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.546+0000 7f2fd0a71640 1 --2- 172.21.15.70:0/3173085691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8be090 0x55ee8e8d90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.546+0000 7f2fd0a71640 1 -- 172.21.15.70:0/3173085691 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee8e5363f0 con 0x55ee8e8c3200 2024-04-01T01:42:09.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.547+0000 7f2fd0a71640 1 --2- 172.21.15.70:0/3173085691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8c3200 0x55ee8e8e0110 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc067290 tx=0x7f2fcc09ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=355364bd16a88fa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.547+0000 7f2fab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:42:09.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.548+0000 7f2fab7fe640 1 -- 172.21.15.70:0/3173085691 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc06c030 con 0x55ee8e8c3200 2024-04-01T01:42:09.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.548+0000 7f2fab7fe640 1 -- 172.21.15.70:0/3173085691 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f2fcc0af070 con 0x55ee8e8c3200 2024-04-01T01:42:09.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.548+0000 7f2fd518e740 1 -- 172.21.15.70:0/3173085691 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fcc067bd0 con 0x55ee8e8c3200 2024-04-01T01:42:09.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.549+0000 7f2fab7fe640 1 -- 172.21.15.70:0/3173085691 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2fcc073060 con 0x55ee8e8c3200 2024-04-01T01:42:09.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.549+0000 7f2fd518e740 10 client.18042 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:09.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.549+0000 7f2fd518e740 1 -- 172.21.15.70:0/3173085691 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ee8e6007d0 con 0x55ee8e8c3200 2024-04-01T01:42:09.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.550+0000 7f2fab7fe640 1 -- 172.21.15.70:0/3173085691 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2fcc06e050 con 0x55ee8e8c3200 2024-04-01T01:42:09.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.550+0000 7f2fab7fe640 1 -- 172.21.15.70:0/3173085691 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f2fcc0a7030 con 0x55ee8e8c3200 2024-04-01T01:42:09.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.550+0000 7f2fd518e740 10 client.18042 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:09.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.550+0000 7f2fd518e740 10 client.18042 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:42:09.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.550+0000 7f2fd518e740 20 client.18042 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:09.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.550+0000 7f2fd518e740 1 --2- 172.21.15.70:0/3173085691 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55ee8e8e41e0 0x55ee8e8e65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.550+0000 7f2fd518e740 4 client.18042 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:09.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.550+0000 7f2fd518e740 1 -- 172.21.15.70:0/3173085691 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2fcc067bd0 con 0x55ee8e8e41e0 2024-04-01T01:42:09.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.551+0000 7f2fcbfff640 1 --2- 172.21.15.70:0/3173085691 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55ee8e8e41e0 0x55ee8e8e65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.552+0000 7f2fab7fe640 10 client.18042 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:42:09.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.552+0000 7f2fcbfff640 1 --2- 172.21.15.70:0/3173085691 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55ee8e8e41e0 0x55ee8e8e65d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7d8651518471ff0f server_cookie=175e614a8d1198a8 in_seq=0 out_seq=0 2024-04-01T01:42:09.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.552+0000 7f2fab7fe640 1 -- 172.21.15.70:0/3173085691 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f2fcc067bd0 con 0x55ee8e8e41e0 2024-04-01T01:42:09.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.552+0000 7f2fab7fe640 10 client.18042 handle_command_reply: tid=0 2024-04-01T01:42:09.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.553+0000 7f2fd518e740 1 client.18042 shutdown 2024-04-01T01:42:09.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.554+0000 7f2fd518e740 1 -- 172.21.15.70:0/3173085691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8c3200 msgr2=0x55ee8e8e0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.554+0000 7f2fd518e740 1 --2- 172.21.15.70:0/3173085691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8c3200 0x55ee8e8e0110 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc067290 tx=0x7f2fcc09ebe0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.556+0000 7f2fd518e740 1 -- 172.21.15.70:0/3173085691 shutdown_connections 2024-04-01T01:42:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.556+0000 7f2fd518e740 1 --2- 172.21.15.70:0/3173085691 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55ee8e8e41e0 0x55ee8e8e65d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.556+0000 7f2fd518e740 1 --2- 172.21.15.70:0/3173085691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8c3200 0x55ee8e8e0110 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.556+0000 7f2fab7fe640 0 client.18042 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:42:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.556+0000 7f2fd518e740 1 --2- 172.21.15.70:0/3173085691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8be090 0x55ee8e8d90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.556+0000 7f2fd518e740 1 --2- 172.21.15.70:0/3173085691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8d8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.557+0000 7f2fd518e740 1 -- 172.21.15.70:0/3173085691 >> 172.21.15.70:0/3173085691 conn(0x55ee8e8b8c10 msgr2=0x55ee8e8b7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.557+0000 7f2fd518e740 1 -- 172.21.15.70:0/3173085691 shutdown_connections 2024-04-01T01:42:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.557+0000 7f2fd518e740 1 -- 172.21.15.70:0/3173085691 wait complete. 2024-04-01T01:42:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.557+0000 7f2fd518e740 20 client.18042 trim_cache size 0 max 16384 2024-04-01T01:42:09.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.585+0000 7f2fd518e740 1 Processor -- start 2024-04-01T01:42:09.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.586+0000 7f2fd518e740 1 -- start start 2024-04-01T01:42:09.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.586+0000 7f2fd518e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.586+0000 7f2fd518e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8be090 0x55ee8e8e00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.586+0000 7f2fd518e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8c3200 0x55ee8e8c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.586+0000 7f2fd518e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb000fd90 con 0x55ee8e8bd6a0 2024-04-01T01:42:09.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.586+0000 7f2fd518e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb00118d0 con 0x55ee8e8be090 2024-04-01T01:42:09.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.586+0000 7f2fd518e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb0010460 con 0x55ee8e8c3200 2024-04-01T01:42:09.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.586+0000 7f2fcbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.586+0000 7f2fcbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8c66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40076/0 (socket says 172.21.15.70:40076) 2024-04-01T01:42:09.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.586+0000 7f2fcbfff640 1 -- 172.21.15.70:0/739703808 learned_addr learned my addr 172.21.15.70:0/739703808 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:09.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.586+0000 7f2fd0a71640 1 --2- 172.21.15.70:0/739703808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8c3200 0x55ee8e8c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.587+0000 7f2fcb7fe640 1 --2- 172.21.15.70:0/739703808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8be090 0x55ee8e8e00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.587+0000 7f2fd0a71640 1 -- 172.21.15.70:0/739703808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8be090 msgr2=0x55ee8e8e00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.587+0000 7f2fd0a71640 1 --2- 172.21.15.70:0/739703808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8be090 0x55ee8e8e00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.587+0000 7f2fd0a71640 1 -- 172.21.15.70:0/739703808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8bd6a0 msgr2=0x55ee8e8c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.587+0000 7f2fd0a71640 1 --2- 172.21.15.70:0/739703808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8c66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.587+0000 7f2fd0a71640 1 -- 172.21.15.70:0/739703808 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee8e8a9e60 con 0x55ee8e8c3200 2024-04-01T01:42:09.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.587+0000 7f2fcbfff640 1 --2- 172.21.15.70:0/739703808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:42:09.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.587+0000 7f2fd0a71640 1 --2- 172.21.15.70:0/739703808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8c3200 0x55ee8e8c6be0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc069650 tx=0x7f2fcc097e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed7215888402d8a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.588+0000 7f2fb5ffb640 1 -- 172.21.15.70:0/739703808 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc06c030 con 0x55ee8e8c3200 2024-04-01T01:42:09.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.588+0000 7f2fb5ffb640 1 -- 172.21.15.70:0/739703808 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2fcc067bd0 con 0x55ee8e8c3200 2024-04-01T01:42:09.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.589+0000 7f2fd518e740 1 -- 172.21.15.70:0/739703808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8c3200 msgr2=0x55ee8e8c6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.589+0000 7f2fd518e740 1 --2- 172.21.15.70:0/739703808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8c3200 0x55ee8e8c6be0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc069650 tx=0x7f2fcc097e50 comp rx=0 tx=0).stop 2024-04-01T01:42:09.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.589+0000 7f2fd518e740 1 -- 172.21.15.70:0/739703808 shutdown_connections 2024-04-01T01:42:09.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.589+0000 7f2fd518e740 1 --2- 172.21.15.70:0/739703808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8c3200 0x55ee8e8c6be0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.589+0000 7f2fd518e740 1 --2- 172.21.15.70:0/739703808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8be090 0x55ee8e8e00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.589+0000 7f2fd518e740 1 --2- 172.21.15.70:0/739703808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.589+0000 7f2fd518e740 1 -- 172.21.15.70:0/739703808 >> 172.21.15.70:0/739703808 conn(0x55ee8e8b8c10 msgr2=0x55ee8e8b8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.590+0000 7f2fd518e740 1 -- 172.21.15.70:0/739703808 shutdown_connections 2024-04-01T01:42:09.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.590+0000 7f2fd518e740 1 -- 172.21.15.70:0/739703808 wait complete. 2024-04-01T01:42:09.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.591+0000 7f2fd518e740 1 -- start start 2024-04-01T01:42:09.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.591+0000 7f2fd518e740 1 Processor -- start 2024-04-01T01:42:09.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.591+0000 7f2fd518e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8dd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.591+0000 7f2fd518e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8be090 0x55ee8e8ddd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.591+0000 7f2fd518e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8e41e0 0x55ee8e8ded30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.591+0000 7f2fd518e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb0011200 con 0x55ee8e8be090 2024-04-01T01:42:09.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.591+0000 7f2fd518e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb0010b30 con 0x55ee8e8bd6a0 2024-04-01T01:42:09.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.591+0000 7f2fd518e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb000dd70 con 0x55ee8e8e41e0 2024-04-01T01:42:09.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.591+0000 7f2fd518e740 10 client.0 _ll_register_callbacks cb 0x7f2fcc09ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:09.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.592+0000 7f2fcbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8dd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.592+0000 7f2fcbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8dd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60790/0 (socket says 172.21.15.70:60790) 2024-04-01T01:42:09.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.592+0000 7f2fcbfff640 1 -- 172.21.15.70:0/3996626964 learned_addr learned my addr 172.21.15.70:0/3996626964 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:09.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.592+0000 7f2fd0a71640 1 --2- 172.21.15.70:0/3996626964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8e41e0 0x55ee8e8ded30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.593+0000 7f2fcb7fe640 1 --2- 172.21.15.70:0/3996626964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8be090 0x55ee8e8ddd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.593+0000 7f2fcbfff640 1 -- 172.21.15.70:0/3996626964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8e41e0 msgr2=0x55ee8e8ded30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.593+0000 7f2fcbfff640 1 --2- 172.21.15.70:0/3996626964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8e41e0 0x55ee8e8ded30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.593+0000 7f2fcbfff640 1 -- 172.21.15.70:0/3996626964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8be090 msgr2=0x55ee8e8ddd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.593+0000 7f2fcbfff640 1 --2- 172.21.15.70:0/3996626964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8be090 0x55ee8e8ddd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.593+0000 7f2fcbfff640 1 -- 172.21.15.70:0/3996626964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee8e8df320 con 0x55ee8e8bd6a0 2024-04-01T01:42:09.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.594+0000 7f2fcbfff640 1 --2- 172.21.15.70:0/3996626964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8dd830 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0039de0 tx=0x7f2fc0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43166ef65f454085 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.594+0000 7f2fab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:09.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.595+0000 7f2fab7fe640 1 -- 172.21.15.70:0/3996626964 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0039300 con 0x55ee8e8bd6a0 2024-04-01T01:42:09.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.595+0000 7f2fd518e740 1 -- 172.21.15.70:0/3996626964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ee8e82f6f0 con 0x55ee8e8bd6a0 2024-04-01T01:42:09.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.595+0000 7f2fab7fe640 1 -- 172.21.15.70:0/3996626964 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f2fc00394a0 con 0x55ee8e8bd6a0 2024-04-01T01:42:09.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.595+0000 7f2fab7fe640 1 -- 172.21.15.70:0/3996626964 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2fc004b050 con 0x55ee8e8bd6a0 2024-04-01T01:42:09.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.595+0000 7f2fd518e740 10 client.28022 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:09.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.595+0000 7f2fd518e740 1 -- 172.21.15.70:0/3996626964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ee8e94a5a0 con 0x55ee8e8bd6a0 2024-04-01T01:42:09.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.597+0000 7f2fab7fe640 1 -- 172.21.15.70:0/3996626964 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55ee8e82f6f0 con 0x55ee8e8bd6a0 2024-04-01T01:42:09.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.598+0000 7f2fab7fe640 1 -- 172.21.15.70:0/3996626964 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f2fc0010430 con 0x55ee8e8bd6a0 2024-04-01T01:42:09.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.598+0000 7f2fd518e740 10 client.28022 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:09.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.598+0000 7f2fd518e740 10 client.28022 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:42:09.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.598+0000 7f2fd518e740 20 client.28022 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:09.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.598+0000 7f2fd518e740 1 --2- 172.21.15.70:0/3996626964 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55ee8e94af30 0x55ee8e94d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.598+0000 7f2fd518e740 4 client.28022 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:42:09.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.598+0000 7f2fd518e740 1 -- 172.21.15.70:0/3996626964 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ee8e829a40 con 0x55ee8e94af30 2024-04-01T01:42:09.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.598+0000 7f2fcb7fe640 1 --2- 172.21.15.70:0/3996626964 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55ee8e94af30 0x55ee8e94d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.599+0000 7f2fcb7fe640 1 --2- 172.21.15.70:0/3996626964 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55ee8e94af30 0x55ee8e94d350 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6e49b3e7985f30ca server_cookie=f6a42e014f03e010 in_seq=0 out_seq=0 2024-04-01T01:42:09.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.600+0000 7f2fab7fe640 10 client.28022 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:42:09.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.600+0000 7f2fab7fe640 1 -- 172.21.15.70:0/3996626964 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ee8e829a40 con 0x55ee8e94af30 2024-04-01T01:42:09.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.600+0000 7f2fab7fe640 10 client.28022 handle_command_reply: tid=0 2024-04-01T01:42:09.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.600+0000 7f2fd518e740 1 client.28022 shutdown 2024-04-01T01:42:09.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.601+0000 7f2fd518e740 1 -- 172.21.15.70:0/3996626964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8bd6a0 msgr2=0x55ee8e8dd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.601+0000 7f2fd518e740 1 --2- 172.21.15.70:0/3996626964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8dd830 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0039de0 tx=0x7f2fc0004730 comp rx=0 tx=0).stop 2024-04-01T01:42:09.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.601+0000 7f2fd518e740 1 -- 172.21.15.70:0/3996626964 shutdown_connections 2024-04-01T01:42:09.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.601+0000 7f2fd518e740 1 --2- 172.21.15.70:0/3996626964 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55ee8e94af30 0x55ee8e94d350 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.601+0000 7f2fd518e740 1 --2- 172.21.15.70:0/3996626964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ee8e8e41e0 0x55ee8e8ded30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.601+0000 7f2fd518e740 1 --2- 172.21.15.70:0/3996626964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ee8e8be090 0x55ee8e8ddd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.601+0000 7f2fd518e740 1 --2- 172.21.15.70:0/3996626964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ee8e8bd6a0 0x55ee8e8dd830 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.601+0000 7f2fd518e740 1 -- 172.21.15.70:0/3996626964 >> 172.21.15.70:0/3996626964 conn(0x55ee8e8b8c10 msgr2=0x55ee8e8d8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.601+0000 7f2fab7fe640 0 client.28022 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:42:09.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.601+0000 7f2fd518e740 1 -- 172.21.15.70:0/3996626964 shutdown_connections 2024-04-01T01:42:09.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.601+0000 7f2fd518e740 1 -- 172.21.15.70:0/3996626964 wait complete. 2024-04-01T01:42:09.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.601+0000 7f2fd518e740 20 client.28022 trim_cache size 0 max 16384 2024-04-01T01:42:09.602 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:42:09.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.602+0000 7f2fd24d4640 1 -- 172.21.15.70:0/3006055550 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2fbc0a2670 msgr2=0x7f2fbc0a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.602+0000 7f2fd24d4640 1 --2- 172.21.15.70:0/3006055550 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2fbc0a2670 0x7f2fbc0a4b30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0005ef0 tx=0x7f2fc0005e80 comp rx=0 tx=0).stop 2024-04-01T01:42:09.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.602+0000 7f2fd24d4640 1 -- 172.21.15.70:0/3006055550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fc4096f60 msgr2=0x7f2fc41329b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.602+0000 7f2fd24d4640 1 --2- 172.21.15.70:0/3006055550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fc4096f60 0x7f2fc41329b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800e9c0 tx=0x7f2fb800ee80 comp rx=0 tx=0).stop 2024-04-01T01:42:09.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.602+0000 7f2fd24d4640 1 -- 172.21.15.70:0/3006055550 shutdown_connections 2024-04-01T01:42:09.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.602+0000 7f2fd24d4640 1 --2- 172.21.15.70:0/3006055550 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2fbc0a2670 0x7f2fbc0a4b30 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.602+0000 7f2fd24d4640 1 --2- 172.21.15.70:0/3006055550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fc4099910 0x7f2fc4139a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.602+0000 7f2fd24d4640 1 --2- 172.21.15.70:0/3006055550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fc4096f60 0x7f2fc41329b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.602+0000 7f2fd24d4640 1 --2- 172.21.15.70:0/3006055550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2fc40945f0 0x7f2fc4132470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.602+0000 7f2fd24d4640 1 -- 172.21.15.70:0/3006055550 >> 172.21.15.70:0/3006055550 conn(0x7f2fc40901d0 msgr2=0x7f2fc4097940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.603+0000 7f2fd24d4640 1 -- 172.21.15.70:0/3006055550 shutdown_connections 2024-04-01T01:42:09.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.603+0000 7f2fd24d4640 1 -- 172.21.15.70:0/3006055550 wait complete. 2024-04-01T01:42:09.623 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:42:09.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.818+0000 7fba789c7640 1 -- 172.21.15.70:0/3188544177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba74108750 msgr2=0x7fba7410ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.818+0000 7fba789c7640 1 --2- 172.21.15.70:0/3188544177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba74108750 0x7fba7410ab80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fba68009900 tx=0x7fba6802bc60 comp rx=0 tx=0).stop 2024-04-01T01:42:09.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.824+0000 7fba789c7640 1 -- 172.21.15.70:0/3188544177 shutdown_connections 2024-04-01T01:42:09.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.824+0000 7fba789c7640 1 --2- 172.21.15.70:0/3188544177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba7410da70 0x7fba7410def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.824+0000 7fba789c7640 1 --2- 172.21.15.70:0/3188544177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba7410b0c0 0x7fba7410d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.824+0000 7fba789c7640 1 --2- 172.21.15.70:0/3188544177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba74108750 0x7fba7410ab80 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.824+0000 7fba789c7640 1 -- 172.21.15.70:0/3188544177 >> 172.21.15.70:0/3188544177 conn(0x7fba7406e5b0 msgr2=0x7fba740709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.827+0000 7fba789c7640 1 -- 172.21.15.70:0/3188544177 shutdown_connections 2024-04-01T01:42:09.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.828+0000 7fba789c7640 1 -- 172.21.15.70:0/3188544177 wait complete. 2024-04-01T01:42:09.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.829+0000 7fba789c7640 1 Processor -- start 2024-04-01T01:42:09.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.829+0000 7fba789c7640 1 -- start start 2024-04-01T01:42:09.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.829+0000 7fba789c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba74108750 0x7fba741a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.829+0000 7fba789c7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7410b0c0 0x7fba741a6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.829+0000 7fba789c7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba7410da70 0x7fba741adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.829+0000 7fba789c7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fba7411c1c0 con 0x7fba7410da70 2024-04-01T01:42:09.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.829+0000 7fba789c7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fba7411c040 con 0x7fba74108750 2024-04-01T01:42:09.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.829+0000 7fba789c7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fba7411c340 con 0x7fba7410b0c0 2024-04-01T01:42:09.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.829+0000 7fba71d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7410b0c0 0x7fba741a6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.829+0000 7fba71d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7410b0c0 0x7fba741a6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:56836/0 (socket says 172.21.15.70:56836) 2024-04-01T01:42:09.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.829+0000 7fba71d9b640 1 -- 172.21.15.70:0/2336907592 learned_addr learned my addr 172.21.15.70:0/2336907592 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:09.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.829+0000 7fba7259c640 1 --2- 172.21.15.70:0/2336907592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba74108750 0x7fba741a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.830+0000 7fba71d9b640 1 -- 172.21.15.70:0/2336907592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba74108750 msgr2=0x7fba741a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.830+0000 7fba71d9b640 1 --2- 172.21.15.70:0/2336907592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba74108750 0x7fba741a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.830+0000 7fba71d9b640 1 -- 172.21.15.70:0/2336907592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba7410da70 msgr2=0x7fba741adde0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:42:09.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.830+0000 7fba71d9b640 1 --2- 172.21.15.70:0/2336907592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba7410da70 0x7fba741adde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.830+0000 7fba71d9b640 1 -- 172.21.15.70:0/2336907592 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba7410ffb0 con 0x7fba7410b0c0 2024-04-01T01:42:09.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.830+0000 7fba7259c640 1 --2- 172.21.15.70:0/2336907592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba74108750 0x7fba741a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:42:09.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.831+0000 7fba71d9b640 1 --2- 172.21.15.70:0/2336907592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7410b0c0 0x7fba741a6dc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fba6c00ba50 tx=0x7fba6c00bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.831+0000 7fba637fe640 1 -- 172.21.15.70:0/2336907592 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6c019070 con 0x7fba7410b0c0 2024-04-01T01:42:09.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.831+0000 7fba789c7640 1 -- 172.21.15.70:0/2336907592 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba741102a0 con 0x7fba7410b0c0 2024-04-01T01:42:09.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.831+0000 7fba789c7640 1 -- 172.21.15.70:0/2336907592 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba741108c0 con 0x7fba7410b0c0 2024-04-01T01:42:09.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.831+0000 7fba637fe640 1 -- 172.21.15.70:0/2336907592 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba6c00cbe0 con 0x7fba7410b0c0 2024-04-01T01:42:09.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.832+0000 7fba637fe640 1 -- 172.21.15.70:0/2336907592 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6c0075d0 con 0x7fba7410b0c0 2024-04-01T01:42:09.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.834+0000 7fba7b5a2740 1 Processor -- start 2024-04-01T01:42:09.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.834+0000 7fba7b5a2740 1 -- start start 2024-04-01T01:42:09.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.835+0000 7fba7b5a2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d935246490 0x55d9352468b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.835+0000 7fba7b5a2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246e80 0x55d93524b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.835+0000 7fba7b5a2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d93524bff0 0x55d93524e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.835+0000 7fba7b5a2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d9351ab1b0 con 0x55d935246490 2024-04-01T01:42:09.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.835+0000 7fba7b5a2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d935153c00 con 0x55d935246e80 2024-04-01T01:42:09.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.835+0000 7fba7b5a2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d9351a7250 con 0x55d93524bff0 2024-04-01T01:42:09.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.835+0000 7fba7259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246e80 0x55d93524b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.835+0000 7fba7259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246e80 0x55d93524b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60816/0 (socket says 172.21.15.70:60816) 2024-04-01T01:42:09.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.835+0000 7fba7259c640 1 -- 172.21.15.70:0/2242532039 learned_addr learned my addr 172.21.15.70:0/2242532039 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:09.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.835+0000 7fba637fe640 1 -- 172.21.15.70:0/2336907592 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fba6c01c070 con 0x7fba7410b0c0 2024-04-01T01:42:09.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.835+0000 7fba71d9b640 1 --2- 172.21.15.70:0/2242532039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d93524bff0 0x55d93524e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.835+0000 7fba7259c640 1 -- 172.21.15.70:0/2242532039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d93524bff0 msgr2=0x55d93524e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.836+0000 7fba7259c640 1 --2- 172.21.15.70:0/2242532039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d93524bff0 0x55d93524e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.836+0000 7fba7259c640 1 -- 172.21.15.70:0/2242532039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d935246490 msgr2=0x55d9352468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.836+0000 7fba637fe640 1 --2- 172.21.15.70:0/2336907592 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba5c0a25e0 0x7fba5c0a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.836+0000 7fba72d9d640 1 --2- 172.21.15.70:0/2242532039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d935246490 0x55d9352468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.836+0000 7fba7259c640 1 --2- 172.21.15.70:0/2242532039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d935246490 0x55d9352468b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.836+0000 7fba7259c640 1 -- 172.21.15.70:0/2242532039 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9351733d0 con 0x55d935246e80 2024-04-01T01:42:09.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.837+0000 7fba71d9b640 1 --2- 172.21.15.70:0/2336907592 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba5c0a25e0 0x7fba5c0a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.837+0000 7fba637fe640 1 -- 172.21.15.70:0/2336907592 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fba6c0ce050 con 0x7fba7410b0c0 2024-04-01T01:42:09.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.837+0000 7fba7259c640 1 --2- 172.21.15.70:0/2242532039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246e80 0x55d93524b980 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fba68009850 tx=0x7fba68009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f031ebb6fa332f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.837+0000 7fba60ff9640 1 -- 172.21.15.70:0/2242532039 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6802cc10 con 0x55d935246e80 2024-04-01T01:42:09.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.837+0000 7fba60ff9640 1 -- 172.21.15.70:0/2242532039 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba68005040 con 0x55d935246e80 2024-04-01T01:42:09.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.838+0000 7fba60ff9640 1 -- 172.21.15.70:0/2242532039 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba68004a40 con 0x55d935246e80 2024-04-01T01:42:09.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.838+0000 7fba7b5a2740 1 -- 172.21.15.70:0/2242532039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246e80 msgr2=0x55d93524b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.838+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/2242532039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246e80 0x55d93524b980 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fba68009850 tx=0x7fba68009610 comp rx=0 tx=0).stop 2024-04-01T01:42:09.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.838+0000 7fba71d9b640 1 --2- 172.21.15.70:0/2336907592 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba5c0a25e0 0x7fba5c0a4aa0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fba6c096b40 tx=0x7fba6c06a590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.838+0000 7fba7b5a2740 1 -- 172.21.15.70:0/2242532039 shutdown_connections 2024-04-01T01:42:09.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.838+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/2242532039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d93524bff0 0x55d93524e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.838+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/2242532039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246e80 0x55d93524b980 secure :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fba68009850 tx=0x7fba68009610 comp rx=0 tx=0).stop 2024-04-01T01:42:09.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.838+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/2242532039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d935246490 0x55d9352468b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.838+0000 7fba7b5a2740 1 -- 172.21.15.70:0/2242532039 >> 172.21.15.70:0/2242532039 conn(0x55d935241a00 msgr2=0x55d935241e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.839+0000 7fba7b5a2740 1 -- 172.21.15.70:0/2242532039 shutdown_connections 2024-04-01T01:42:09.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.839+0000 7fba7b5a2740 1 -- 172.21.15.70:0/2242532039 wait complete. 2024-04-01T01:42:09.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.839+0000 7fba7b5a2740 1 -- start start 2024-04-01T01:42:09.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.839+0000 7fba7b5a2740 1 Processor -- start 2024-04-01T01:42:09.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.839+0000 7fba7b5a2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d935246490 0x55d935250a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.839+0000 7fba7b5a2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d93524bff0 0x55d935250f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.839+0000 7fba7b5a2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d935255f80 0x55d935258370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.839+0000 7fba7b5a2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d93524ef90 con 0x55d935246490 2024-04-01T01:42:09.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.839+0000 7fba7b5a2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d935149470 con 0x55d93524bff0 2024-04-01T01:42:09.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.840+0000 7fba7b5a2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d93524f110 con 0x55d935255f80 2024-04-01T01:42:09.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.840+0000 7fba7b5a2740 10 client.0 _ll_register_callbacks cb 0x55d935166cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:09.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.840+0000 7fba7259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d93524bff0 0x55d935250f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.840+0000 7fba7259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d93524bff0 0x55d935250f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60828/0 (socket says 172.21.15.70:60828) 2024-04-01T01:42:09.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.840+0000 7fba7259c640 1 -- 172.21.15.70:0/3279637006 learned_addr learned my addr 172.21.15.70:0/3279637006 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:09.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.840+0000 7fba72d9d640 1 --2- 172.21.15.70:0/3279637006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d935255f80 0x55d935258370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.840+0000 7fba72d9d640 1 --2- 172.21.15.70:0/3279637006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d935246490 0x55d935250a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.840+0000 7fba7259c640 1 -- 172.21.15.70:0/3279637006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d935255f80 msgr2=0x55d935258370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.840+0000 7fba7259c640 1 --2- 172.21.15.70:0/3279637006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d935255f80 0x55d935258370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.840+0000 7fba7259c640 1 -- 172.21.15.70:0/3279637006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d935246490 msgr2=0x55d935250a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.840+0000 7fba7259c640 1 --2- 172.21.15.70:0/3279637006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d935246490 0x55d935250a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.840+0000 7fba7259c640 1 -- 172.21.15.70:0/3279637006 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d934ebf3f0 con 0x55d93524bff0 2024-04-01T01:42:09.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.841+0000 7fba7259c640 1 --2- 172.21.15.70:0/3279637006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d93524bff0 0x55d935250f50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fba68000c00 tx=0x7fba68004fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e69028706a67284 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.841+0000 7fba59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:09.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.841+0000 7fba59ffb640 1 -- 172.21.15.70:0/3279637006 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6804c020 con 0x55d93524bff0 2024-04-01T01:42:09.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.841+0000 7fba59ffb640 1 -- 172.21.15.70:0/3279637006 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fba68005500 con 0x55d93524bff0 2024-04-01T01:42:09.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.841+0000 7fba7b5a2740 1 -- 172.21.15.70:0/3279637006 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba68005040 con 0x55d93524bff0 2024-04-01T01:42:09.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.842+0000 7fba59ffb640 1 -- 172.21.15.70:0/3279637006 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fba68057050 con 0x55d93524bff0 2024-04-01T01:42:09.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.842+0000 7fba7b5a2740 10 client.28028 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:09.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.842+0000 7fba7b5a2740 1 -- 172.21.15.70:0/3279637006 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d934f897d0 con 0x55d93524bff0 2024-04-01T01:42:09.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.842+0000 7fba59ffb640 1 -- 172.21.15.70:0/3279637006 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba68005040 con 0x55d93524bff0 2024-04-01T01:42:09.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.842+0000 7fba59ffb640 1 -- 172.21.15.70:0/3279637006 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fba68034440 con 0x55d93524bff0 2024-04-01T01:42:09.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.842+0000 7fba7b5a2740 10 client.28028 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:09.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.842+0000 7fba7b5a2740 10 client.28028 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:42:09.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.842+0000 7fba7b5a2740 20 client.28028 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:09.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.842+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/3279637006 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55d93525c4d0 0x55d93525e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.842+0000 7fba7b5a2740 4 client.28028 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:09.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.842+0000 7fba7b5a2740 1 -- 172.21.15.70:0/3279637006 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9351b86f0 con 0x55d93525c4d0 2024-04-01T01:42:09.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.842+0000 7fba72d9d640 1 --2- 172.21.15.70:0/3279637006 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55d93525c4d0 0x55d93525e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.846+0000 7fba72d9d640 1 --2- 172.21.15.70:0/3279637006 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55d93525c4d0 0x55d93525e900 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=373a4185c65f3eaf server_cookie=5673595e84149017 in_seq=0 out_seq=0 2024-04-01T01:42:09.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.846+0000 7fba59ffb640 10 client.28028 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:42:09.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.847+0000 7fba59ffb640 1 -- 172.21.15.70:0/3279637006 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d9351b86f0 con 0x55d93525c4d0 2024-04-01T01:42:09.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.847+0000 7fba59ffb640 10 client.28028 handle_command_reply: tid=0 2024-04-01T01:42:09.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.847+0000 7fba7b5a2740 1 client.28028 shutdown 2024-04-01T01:42:09.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.847+0000 7fba7b5a2740 1 -- 172.21.15.70:0/3279637006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d93524bff0 msgr2=0x55d935250f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.847+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/3279637006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d93524bff0 0x55d935250f50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fba68000c00 tx=0x7fba68004fd0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.848+0000 7fba7b5a2740 1 -- 172.21.15.70:0/3279637006 shutdown_connections 2024-04-01T01:42:09.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.848+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/3279637006 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55d93525c4d0 0x55d93525e900 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.848+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/3279637006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d935255f80 0x55d935258370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.848+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/3279637006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d93524bff0 0x55d935250f50 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.848+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/3279637006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d935246490 0x55d935250a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.848+0000 7fba7b5a2740 1 -- 172.21.15.70:0/3279637006 >> 172.21.15.70:0/3279637006 conn(0x55d935241a00 msgr2=0x55d935240d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.848+0000 7fba59ffb640 0 client.28028 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:42:09.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.848+0000 7fba7b5a2740 1 -- 172.21.15.70:0/3279637006 shutdown_connections 2024-04-01T01:42:09.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.848+0000 7fba7b5a2740 1 -- 172.21.15.70:0/3279637006 wait complete. 2024-04-01T01:42:09.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.848+0000 7fba7b5a2740 20 client.28028 trim_cache size 0 max 16384 2024-04-01T01:42:09.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.872+0000 7fba7b5a2740 1 Processor -- start 2024-04-01T01:42:09.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.872+0000 7fba7b5a2740 1 -- start start 2024-04-01T01:42:09.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.872+0000 7fba7b5a2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246490 0x55d935241de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.872+0000 7fba7b5a2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d93524bff0 0x55d9352483e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.872+0000 7fba7b5a2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d935255f80 0x55d935248920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.872+0000 7fba7b5a2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fba5400fd90 con 0x55d93524bff0 2024-04-01T01:42:09.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.872+0000 7fba7b5a2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fba540118d0 con 0x55d935246490 2024-04-01T01:42:09.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.872+0000 7fba7b5a2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fba54010460 con 0x55d935255f80 2024-04-01T01:42:09.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.872+0000 7fba7259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d93524bff0 0x55d9352483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.872+0000 7fba7259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d93524bff0 0x55d9352483e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40122/0 (socket says 172.21.15.70:40122) 2024-04-01T01:42:09.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.872+0000 7fba7259c640 1 -- 172.21.15.70:0/2769664589 learned_addr learned my addr 172.21.15.70:0/2769664589 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:09.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.873+0000 7fba7259c640 1 -- 172.21.15.70:0/2769664589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d935255f80 msgr2=0x55d935248920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.873+0000 7fba7259c640 1 --2- 172.21.15.70:0/2769664589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d935255f80 0x55d935248920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.873+0000 7fba7259c640 1 -- 172.21.15.70:0/2769664589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246490 msgr2=0x55d935241de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.873+0000 7fba72d9d640 1 --2- 172.21.15.70:0/2769664589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246490 0x55d935241de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.873+0000 7fba7259c640 1 --2- 172.21.15.70:0/2769664589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246490 0x55d935241de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.873+0000 7fba7259c640 1 -- 172.21.15.70:0/2769664589 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d93524b660 con 0x55d93524bff0 2024-04-01T01:42:09.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.873+0000 7fba72d9d640 1 --2- 172.21.15.70:0/2769664589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246490 0x55d935241de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:42:09.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.874+0000 7fba7259c640 1 --2- 172.21.15.70:0/2769664589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d93524bff0 0x55d9352483e0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fba6802cd80 tx=0x7fba6802bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40a63ad3c06130e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.874+0000 7fba5bfff640 1 -- 172.21.15.70:0/2769664589 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6804c040 con 0x55d93524bff0 2024-04-01T01:42:09.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.874+0000 7fba5bfff640 1 -- 172.21.15.70:0/2769664589 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba680044d0 con 0x55d93524bff0 2024-04-01T01:42:09.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.874+0000 7fba5bfff640 1 -- 172.21.15.70:0/2769664589 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba68004a40 con 0x55d93524bff0 2024-04-01T01:42:09.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.875+0000 7fba7b5a2740 1 -- 172.21.15.70:0/2769664589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d93524bff0 msgr2=0x55d9352483e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.875+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/2769664589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d93524bff0 0x55d9352483e0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fba6802cd80 tx=0x7fba6802bc60 comp rx=0 tx=0).stop 2024-04-01T01:42:09.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.875+0000 7fba7b5a2740 1 -- 172.21.15.70:0/2769664589 shutdown_connections 2024-04-01T01:42:09.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.875+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/2769664589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d935255f80 0x55d935248920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.875+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/2769664589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d93524bff0 0x55d9352483e0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.875+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/2769664589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246490 0x55d935241de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.875+0000 7fba7b5a2740 1 -- 172.21.15.70:0/2769664589 >> 172.21.15.70:0/2769664589 conn(0x55d935241a00 msgr2=0x55d93524f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.876+0000 7fba7b5a2740 1 -- 172.21.15.70:0/2769664589 shutdown_connections 2024-04-01T01:42:09.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.876+0000 7fba7b5a2740 1 -- 172.21.15.70:0/2769664589 wait complete. 2024-04-01T01:42:09.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.876+0000 7fba7b5a2740 1 -- start start 2024-04-01T01:42:09.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.876+0000 7fba7b5a2740 1 Processor -- start 2024-04-01T01:42:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.877+0000 7fba7b5a2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246490 0x55d93526d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.877+0000 7fba7b5a2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d93524bff0 0x55d93526d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.877+0000 7fba7b5a2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d935255f80 0x55d93526e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.877+0000 7fba7259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d93524bff0 0x55d93526d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.877+0000 7fba7259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d93524bff0 0x55d93526d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40134/0 (socket says 172.21.15.70:40134) 2024-04-01T01:42:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.877+0000 7fba72d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246490 0x55d93526d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.877+0000 7fba72d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246490 0x55d93526d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60856/0 (socket says 172.21.15.70:60856) 2024-04-01T01:42:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.877+0000 7fba7259c640 1 -- 172.21.15.70:0/1882706034 learned_addr learned my addr 172.21.15.70:0/1882706034 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.877+0000 7fba7b5a2740 1 -- 172.21.15.70:0/1882706034 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fba54011200 con 0x55d93524bff0 2024-04-01T01:42:09.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.877+0000 7fba7b5a2740 1 -- 172.21.15.70:0/1882706034 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fba54010b30 con 0x55d935246490 2024-04-01T01:42:09.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.877+0000 7fba7b5a2740 1 -- 172.21.15.70:0/1882706034 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fba5400dd70 con 0x55d935255f80 2024-04-01T01:42:09.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.877+0000 7fba7b5a2740 10 client.0 _ll_register_callbacks cb 0x7fba68004cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:09.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.878+0000 7fba7259c640 1 -- 172.21.15.70:0/1882706034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d935255f80 msgr2=0x55d93526e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.878+0000 7fba7259c640 1 --2- 172.21.15.70:0/1882706034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d935255f80 0x55d93526e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.878+0000 7fba7259c640 1 -- 172.21.15.70:0/1882706034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246490 msgr2=0x55d93526d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:09.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.878+0000 7fba7259c640 1 --2- 172.21.15.70:0/1882706034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246490 0x55d93526d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.878+0000 7fba7259c640 1 -- 172.21.15.70:0/1882706034 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d93526efe0 con 0x55d93524bff0 2024-04-01T01:42:09.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.879+0000 7fba7259c640 1 --2- 172.21.15.70:0/1882706034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d93524bff0 0x55d93526d9e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fba68005c40 tx=0x7fba680346e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604bb8f8958c4c5a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:09.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.879+0000 7fba59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:42:09.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.879+0000 7fba59ffb640 1 -- 172.21.15.70:0/1882706034 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba68060070 con 0x55d93524bff0 2024-04-01T01:42:09.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.879+0000 7fba59ffb640 1 -- 172.21.15.70:0/1882706034 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fba68034c50 con 0x55d93524bff0 2024-04-01T01:42:09.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.879+0000 7fba7b5a2740 1 -- 172.21.15.70:0/1882706034 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba680044d0 con 0x55d93524bff0 2024-04-01T01:42:09.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.881+0000 7fba59ffb640 1 -- 172.21.15.70:0/1882706034 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fba68069050 con 0x55d93524bff0 2024-04-01T01:42:09.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.881+0000 7fba7b5a2740 10 client.18057 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:09.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.881+0000 7fba7b5a2740 1 -- 172.21.15.70:0/1882706034 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d9352c2500 con 0x55d93524bff0 2024-04-01T01:42:09.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.881+0000 7fba59ffb640 1 -- 172.21.15.70:0/1882706034 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba680044d0 con 0x55d93524bff0 2024-04-01T01:42:09.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.882+0000 7fba59ffb640 1 -- 172.21.15.70:0/1882706034 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fba6805cc60 con 0x55d93524bff0 2024-04-01T01:42:09.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.882+0000 7fba7b5a2740 10 client.18057 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:09.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.882+0000 7fba7b5a2740 10 client.18057 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:42:09.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.882+0000 7fba7b5a2740 20 client.18057 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:09.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.882+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/1882706034 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55d9352c2ec0 0x55d9352c52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:09.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.882+0000 7fba7b5a2740 4 client.18057 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:42:09.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.882+0000 7fba7b5a2740 1 -- 172.21.15.70:0/1882706034 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d9351b2a40 con 0x55d9352c2ec0 2024-04-01T01:42:09.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.882+0000 7fba72d9d640 1 --2- 172.21.15.70:0/1882706034 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55d9352c2ec0 0x55d9352c52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:09.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.885+0000 7fba72d9d640 1 --2- 172.21.15.70:0/1882706034 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55d9352c2ec0 0x55d9352c52e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b55bb97c5c711ab5 server_cookie=5d21a979a575d165 in_seq=0 out_seq=0 2024-04-01T01:42:09.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.886+0000 7fba59ffb640 10 client.18057 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:42:09.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.886+0000 7fba59ffb640 1 -- 172.21.15.70:0/1882706034 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d9351b2a40 con 0x55d9352c2ec0 2024-04-01T01:42:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.886+0000 7fba59ffb640 10 client.18057 handle_command_reply: tid=0 2024-04-01T01:42:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.886+0000 7fba7b5a2740 1 client.18057 shutdown 2024-04-01T01:42:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.886+0000 7fba7b5a2740 1 -- 172.21.15.70:0/1882706034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d93524bff0 msgr2=0x55d93526d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.886+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/1882706034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d93524bff0 0x55d93526d9e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fba68005c40 tx=0x7fba680346e0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.887+0000 7fba7b5a2740 1 -- 172.21.15.70:0/1882706034 shutdown_connections 2024-04-01T01:42:09.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.887+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/1882706034 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55d9352c2ec0 0x55d9352c52e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.887+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/1882706034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d935255f80 0x55d93526e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.887+0000 7fba59ffb640 0 client.18057 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:42:09.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.887+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/1882706034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d93524bff0 0x55d93526d9e0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.887+0000 7fba7b5a2740 1 --2- 172.21.15.70:0/1882706034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d935246490 0x55d93526d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.887+0000 7fba7b5a2740 1 -- 172.21.15.70:0/1882706034 >> 172.21.15.70:0/1882706034 conn(0x55d935241a00 msgr2=0x55d935241de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.887+0000 7fba7b5a2740 1 -- 172.21.15.70:0/1882706034 shutdown_connections 2024-04-01T01:42:09.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.887+0000 7fba7b5a2740 1 -- 172.21.15.70:0/1882706034 wait complete. 2024-04-01T01:42:09.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.887+0000 7fba7b5a2740 20 client.18057 trim_cache size 0 max 16384 2024-04-01T01:42:09.888 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:42:09.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.890+0000 7fba789c7640 1 -- 172.21.15.70:0/2336907592 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba5c0a25e0 msgr2=0x7fba5c0a4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.890+0000 7fba789c7640 1 --2- 172.21.15.70:0/2336907592 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba5c0a25e0 0x7fba5c0a4aa0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fba6c096b40 tx=0x7fba6c06a590 comp rx=0 tx=0).stop 2024-04-01T01:42:09.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.890+0000 7fba789c7640 1 -- 172.21.15.70:0/2336907592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7410b0c0 msgr2=0x7fba741a6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:09.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.890+0000 7fba789c7640 1 --2- 172.21.15.70:0/2336907592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7410b0c0 0x7fba741a6dc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fba6c00ba50 tx=0x7fba6c00bf10 comp rx=0 tx=0).stop 2024-04-01T01:42:09.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.890+0000 7fba789c7640 1 -- 172.21.15.70:0/2336907592 shutdown_connections 2024-04-01T01:42:09.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.891+0000 7fba789c7640 1 --2- 172.21.15.70:0/2336907592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fba7410da70 0x7fba741adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.891+0000 7fba789c7640 1 --2- 172.21.15.70:0/2336907592 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fba5c0a25e0 0x7fba5c0a4aa0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.891+0000 7fba789c7640 1 --2- 172.21.15.70:0/2336907592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7410b0c0 0x7fba741a6dc0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.891+0000 7fba789c7640 1 --2- 172.21.15.70:0/2336907592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba74108750 0x7fba741a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:09.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.891+0000 7fba789c7640 1 -- 172.21.15.70:0/2336907592 >> 172.21.15.70:0/2336907592 conn(0x7fba7406e5b0 msgr2=0x7fba7410c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:09.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.891+0000 7fba789c7640 1 -- 172.21.15.70:0/2336907592 shutdown_connections 2024-04-01T01:42:09.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:09.891+0000 7fba789c7640 1 -- 172.21.15.70:0/2336907592 wait complete. 2024-04-01T01:42:09.911 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:42:10.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.106+0000 7f41e10eb640 1 -- 172.21.15.70:0/3340885421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41dc10da70 msgr2=0x7f41dc10def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.106+0000 7f41e10eb640 1 --2- 172.21.15.70:0/3340885421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41dc10da70 0x7f41dc10def0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f41d4009ca0 tx=0x7f41d402c120 comp rx=0 tx=0).stop 2024-04-01T01:42:10.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.106+0000 7f41e10eb640 1 -- 172.21.15.70:0/3340885421 shutdown_connections 2024-04-01T01:42:10.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.106+0000 7f41e10eb640 1 --2- 172.21.15.70:0/3340885421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41dc10da70 0x7f41dc10def0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.106+0000 7f41e10eb640 1 --2- 172.21.15.70:0/3340885421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41dc10b0c0 0x7f41dc10d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.106+0000 7f41e10eb640 1 --2- 172.21.15.70:0/3340885421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41dc108750 0x7f41dc10ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.106+0000 7f41e10eb640 1 -- 172.21.15.70:0/3340885421 >> 172.21.15.70:0/3340885421 conn(0x7f41dc06e5b0 msgr2=0x7f41dc0709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:10.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.110+0000 7f41e10eb640 1 -- 172.21.15.70:0/3340885421 shutdown_connections 2024-04-01T01:42:10.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.110+0000 7f41e10eb640 1 -- 172.21.15.70:0/3340885421 wait complete. 2024-04-01T01:42:10.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.111+0000 7f41e10eb640 1 Processor -- start 2024-04-01T01:42:10.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.112+0000 7f41e10eb640 1 -- start start 2024-04-01T01:42:10.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.112+0000 7f41e10eb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41dc108750 0x7f41dc1a6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.112+0000 7f41e10eb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41dc10b0c0 0x7f41dc1a6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.112+0000 7f41e10eb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41dc10da70 0x7f41dc1addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.112+0000 7f41e10eb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc11c0d0 con 0x7f41dc108750 2024-04-01T01:42:10.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.112+0000 7f41e10eb640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc11bf50 con 0x7f41dc10da70 2024-04-01T01:42:10.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.112+0000 7f41e10eb640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc11c250 con 0x7f41dc10b0c0 2024-04-01T01:42:10.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.112+0000 7f41db59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41dc10da70 0x7f41dc1addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.112+0000 7f41dad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41dc108750 0x7f41dc1a6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.112+0000 7f41dad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41dc108750 0x7f41dc1a6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40148/0 (socket says 172.21.15.70:40148) 2024-04-01T01:42:10.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.112+0000 7f41dad9d640 1 -- 172.21.15.70:0/1470120635 learned_addr learned my addr 172.21.15.70:0/1470120635 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:10.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.113+0000 7f41db59e640 1 -- 172.21.15.70:0/1470120635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41dc10b0c0 msgr2=0x7f41dc1a6d80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:42:10.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.113+0000 7f41db59e640 1 --2- 172.21.15.70:0/1470120635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41dc10b0c0 0x7f41dc1a6d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.113+0000 7f41db59e640 1 -- 172.21.15.70:0/1470120635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41dc108750 msgr2=0x7f41dc1a6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.113+0000 7f41db59e640 1 --2- 172.21.15.70:0/1470120635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41dc108750 0x7f41dc1a6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.113+0000 7f41db59e640 1 -- 172.21.15.70:0/1470120635 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41dc1ae460 con 0x7f41dc10da70 2024-04-01T01:42:10.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.117+0000 7f41db59e640 1 --2- 172.21.15.70:0/1470120635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41dc10da70 0x7f41dc1addf0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f41d4037040 tx=0x7f41d4034880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:10.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.118+0000 7f41c3fff640 1 -- 172.21.15.70:0/1470120635 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4041020 con 0x7f41dc10da70 2024-04-01T01:42:10.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.118+0000 7f41c3fff640 1 -- 172.21.15.70:0/1470120635 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41d4034e10 con 0x7f41dc10da70 2024-04-01T01:42:10.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.118+0000 7f41c3fff640 1 -- 172.21.15.70:0/1470120635 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d4039040 con 0x7f41dc10da70 2024-04-01T01:42:10.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.120+0000 7f41e10eb640 1 -- 172.21.15.70:0/1470120635 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41dc1ae6f0 con 0x7f41dc10da70 2024-04-01T01:42:10.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.120+0000 7f41e10eb640 1 -- 172.21.15.70:0/1470120635 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41dc1aebe0 con 0x7f41dc10da70 2024-04-01T01:42:10.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.123+0000 7f41e3da5740 1 Processor -- start 2024-04-01T01:42:10.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.125+0000 7f41e3da5740 1 -- start start 2024-04-01T01:42:10.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.125+0000 7f41e3da5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2a490 0x560fdff2a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.125+0000 7f41e3da5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2ae80 0x560fdff2f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.125+0000 7f41c3fff640 1 -- 172.21.15.70:0/1470120635 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f41d4044020 con 0x7f41dc10da70 2024-04-01T01:42:10.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.125+0000 7f41e3da5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2fff0 0x560fdff32480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.125+0000 7f41e3da5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560fdfe8f1b0 con 0x560fdff2fff0 2024-04-01T01:42:10.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.125+0000 7f41e3da5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560fdfe37c00 con 0x560fdff2ae80 2024-04-01T01:42:10.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.125+0000 7f41e3da5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560fdfe8b250 con 0x560fdff2a490 2024-04-01T01:42:10.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.126+0000 7f41da59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2a490 0x560fdff2a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.126+0000 7f41da59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2a490 0x560fdff2a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:56908/0 (socket says 172.21.15.70:56908) 2024-04-01T01:42:10.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.126+0000 7f41da59c640 1 -- 172.21.15.70:0/3202477563 learned_addr learned my addr 172.21.15.70:0/3202477563 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:10.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.126+0000 7f41dad9d640 1 --2- 172.21.15.70:0/3202477563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2ae80 0x560fdff2f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.126+0000 7f41da59c640 1 --2- 172.21.15.70:0/3202477563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2fff0 0x560fdff32480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.126+0000 7f41da59c640 1 -- 172.21.15.70:0/3202477563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2ae80 msgr2=0x560fdff2f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:10.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.126+0000 7f41da59c640 1 --2- 172.21.15.70:0/3202477563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2ae80 0x560fdff2f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.126+0000 7f41da59c640 1 -- 172.21.15.70:0/3202477563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2fff0 msgr2=0x560fdff32480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:10.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.126+0000 7f41da59c640 1 --2- 172.21.15.70:0/3202477563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2fff0 0x560fdff32480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.126+0000 7f41da59c640 1 -- 172.21.15.70:0/3202477563 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fdfe573d0 con 0x560fdff2a490 2024-04-01T01:42:10.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.127+0000 7f41dad9d640 1 --2- 172.21.15.70:0/3202477563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2ae80 0x560fdff2f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:42:10.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.127+0000 7f41c3fff640 1 --2- 172.21.15.70:0/1470120635 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41c40a25f0 0x7f41c40a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.127+0000 7f41c3fff640 1 -- 172.21.15.70:0/1470120635 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f41d40f0a30 con 0x7f41dc10da70 2024-04-01T01:42:10.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.127+0000 7f41da59c640 1 --2- 172.21.15.70:0/3202477563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2a490 0x560fdff2a8b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f41d0001ab0 tx=0x7f41d0009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d897cbbdc9268d95 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:10.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.127+0000 7f41dad9d640 1 --2- 172.21.15.70:0/1470120635 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41c40a25f0 0x7f41c40a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.128+0000 7f41dad9d640 1 --2- 172.21.15.70:0/1470120635 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41c40a25f0 0x7f41c40a4ab0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x560fdff2b280 tx=0x7f41cc00b480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:10.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.128+0000 7f41c17fa640 1 -- 172.21.15.70:0/3202477563 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d000ea70 con 0x560fdff2a490 2024-04-01T01:42:10.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.128+0000 7f41c17fa640 1 -- 172.21.15.70:0/3202477563 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41d0008e30 con 0x560fdff2a490 2024-04-01T01:42:10.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.128+0000 7f41c17fa640 1 -- 172.21.15.70:0/3202477563 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d00047d0 con 0x560fdff2a490 2024-04-01T01:42:10.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.129+0000 7f41e3da5740 1 -- 172.21.15.70:0/3202477563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2a490 msgr2=0x560fdff2a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.129+0000 7f41e3da5740 1 --2- 172.21.15.70:0/3202477563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2a490 0x560fdff2a8b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f41d0001ab0 tx=0x7f41d0009f90 comp rx=0 tx=0).stop 2024-04-01T01:42:10.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.132+0000 7f41e3da5740 1 -- 172.21.15.70:0/3202477563 shutdown_connections 2024-04-01T01:42:10.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.132+0000 7f41e3da5740 1 --2- 172.21.15.70:0/3202477563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2fff0 0x560fdff32480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.132+0000 7f41e3da5740 1 --2- 172.21.15.70:0/3202477563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2ae80 0x560fdff2f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.132+0000 7f41e3da5740 1 --2- 172.21.15.70:0/3202477563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2a490 0x560fdff2a8b0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.132+0000 7f41e3da5740 1 -- 172.21.15.70:0/3202477563 >> 172.21.15.70:0/3202477563 conn(0x560fdff25a00 msgr2=0x560fdff25e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:10.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.132+0000 7f41e3da5740 1 -- 172.21.15.70:0/3202477563 shutdown_connections 2024-04-01T01:42:10.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.132+0000 7f41e3da5740 1 -- 172.21.15.70:0/3202477563 wait complete. 2024-04-01T01:42:10.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.132+0000 7f41e3da5740 1 -- start start 2024-04-01T01:42:10.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.133+0000 7f41e3da5740 1 Processor -- start 2024-04-01T01:42:10.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.133+0000 7f41e3da5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2a490 0x560fdff45a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.133+0000 7f41e3da5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2ae80 0x560fdff45fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.133+0000 7f41e3da5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2fff0 0x560fdff4cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.133+0000 7f41e3da5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560fdff43fb0 con 0x560fdff2ae80 2024-04-01T01:42:10.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.133+0000 7f41e3da5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560fdfe2d470 con 0x560fdff2fff0 2024-04-01T01:42:10.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.133+0000 7f41e3da5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560fdff44130 con 0x560fdff2a490 2024-04-01T01:42:10.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.133+0000 7f41e3da5740 10 client.0 _ll_register_callbacks cb 0x560fdfe4acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:10.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.134+0000 7f41db59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2fff0 0x560fdff4cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.134+0000 7f41db59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2fff0 0x560fdff4cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60880/0 (socket says 172.21.15.70:60880) 2024-04-01T01:42:10.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.134+0000 7f41db59e640 1 -- 172.21.15.70:0/2699229329 learned_addr learned my addr 172.21.15.70:0/2699229329 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:10.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.134+0000 7f41dad9d640 1 --2- 172.21.15.70:0/2699229329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2a490 0x560fdff45a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.134+0000 7f41db59e640 1 -- 172.21.15.70:0/2699229329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2a490 msgr2=0x560fdff45a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:10.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.134+0000 7f41db59e640 1 --2- 172.21.15.70:0/2699229329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2a490 0x560fdff45a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.134+0000 7f41db59e640 1 -- 172.21.15.70:0/2699229329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2ae80 msgr2=0x560fdff45fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:10.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.134+0000 7f41db59e640 1 --2- 172.21.15.70:0/2699229329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2ae80 0x560fdff45fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.135+0000 7f41db59e640 1 -- 172.21.15.70:0/2699229329 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fdfba33f0 con 0x560fdff2fff0 2024-04-01T01:42:10.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.135+0000 7f41db59e640 1 --2- 172.21.15.70:0/2699229329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2fff0 0x560fdff4cff0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f41d4009c70 tx=0x7f41d40bccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de2d87b4859e0c10 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:10.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.137+0000 7f41beff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:10.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.137+0000 7f41beff5640 1 -- 172.21.15.70:0/2699229329 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d40bd0c0 con 0x560fdff2fff0 2024-04-01T01:42:10.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.137+0000 7f41beff5640 1 -- 172.21.15.70:0/2699229329 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f41d40bd320 con 0x560fdff2fff0 2024-04-01T01:42:10.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.137+0000 7f41e3da5740 1 -- 172.21.15.70:0/2699229329 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41d0008e30 con 0x560fdff2fff0 2024-04-01T01:42:10.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.138+0000 7f41beff5640 1 -- 172.21.15.70:0/2699229329 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f41d40c1210 con 0x560fdff2fff0 2024-04-01T01:42:10.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.141+0000 7f41e3da5740 10 client.28040 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:10.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.141+0000 7f41e3da5740 1 -- 172.21.15.70:0/2699229329 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560fdfc6d7d0 con 0x560fdff2fff0 2024-04-01T01:42:10.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.144+0000 7f41beff5640 1 -- 172.21.15.70:0/2699229329 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41d0008e30 con 0x560fdff2fff0 2024-04-01T01:42:10.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.144+0000 7f41beff5640 1 -- 172.21.15.70:0/2699229329 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f41d404e060 con 0x560fdff2fff0 2024-04-01T01:42:10.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.144+0000 7f41e3da5740 10 client.28040 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:10.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.144+0000 7f41e3da5740 10 client.28040 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:42:10.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.144+0000 7f41e3da5740 20 client.28040 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:10.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.144+0000 7f41e3da5740 1 --2- 172.21.15.70:0/2699229329 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560fdff73540 0x560fdff75a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.144+0000 7f41e3da5740 4 client.28040 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:10.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.144+0000 7f41e3da5740 1 -- 172.21.15.70:0/2699229329 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fdfe9c6f0 con 0x560fdff73540 2024-04-01T01:42:10.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.149+0000 7f41dad9d640 1 --2- 172.21.15.70:0/2699229329 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560fdff73540 0x560fdff75a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.152+0000 7f41beff5640 10 client.28040 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:42:10.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.152+0000 7f41dad9d640 1 --2- 172.21.15.70:0/2699229329 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560fdff73540 0x560fdff75a00 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=171f9769d99d5c03 server_cookie=e7b064b53c897b1c in_seq=0 out_seq=0 2024-04-01T01:42:10.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.153+0000 7f41beff5640 1 -- 172.21.15.70:0/2699229329 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560fdfe9c6f0 con 0x560fdff73540 2024-04-01T01:42:10.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.153+0000 7f41beff5640 10 client.28040 handle_command_reply: tid=0 2024-04-01T01:42:10.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.153+0000 7f41e3da5740 1 client.28040 shutdown 2024-04-01T01:42:10.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.155+0000 7f41e3da5740 1 -- 172.21.15.70:0/2699229329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2fff0 msgr2=0x560fdff4cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.155+0000 7f41e3da5740 1 --2- 172.21.15.70:0/2699229329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2fff0 0x560fdff4cff0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f41d4009c70 tx=0x7f41d40bccc0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.155+0000 7f41e3da5740 1 -- 172.21.15.70:0/2699229329 shutdown_connections 2024-04-01T01:42:10.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.155+0000 7f41e3da5740 1 --2- 172.21.15.70:0/2699229329 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560fdff73540 0x560fdff75a00 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.155+0000 7f41e3da5740 1 --2- 172.21.15.70:0/2699229329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2fff0 0x560fdff4cff0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.155+0000 7f41e3da5740 1 --2- 172.21.15.70:0/2699229329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2ae80 0x560fdff45fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.155+0000 7f41e3da5740 1 --2- 172.21.15.70:0/2699229329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2a490 0x560fdff45a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.155+0000 7f41e3da5740 1 -- 172.21.15.70:0/2699229329 >> 172.21.15.70:0/2699229329 conn(0x560fdff25a00 msgr2=0x560fdff24d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:10.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.155+0000 7f41beff5640 0 client.28040 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:42:10.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.157+0000 7f41e3da5740 1 -- 172.21.15.70:0/2699229329 shutdown_connections 2024-04-01T01:42:10.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.157+0000 7f41e3da5740 1 -- 172.21.15.70:0/2699229329 wait complete. 2024-04-01T01:42:10.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.157+0000 7f41e3da5740 20 client.28040 trim_cache size 0 max 16384 2024-04-01T01:42:10.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.196+0000 7f41e3da5740 1 Processor -- start 2024-04-01T01:42:10.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.197+0000 7f41e3da5740 1 -- start start 2024-04-01T01:42:10.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.197+0000 7f41e3da5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2a490 0x560fdff444d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.197+0000 7f41e3da5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2ae80 0x560fdff4cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.197+0000 7f41e3da5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2fff0 0x560fdff44a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.197+0000 7f41da59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2ae80 0x560fdff4cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.197+0000 7f41dad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2a490 0x560fdff444d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.197+0000 7f41da59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2ae80 0x560fdff4cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40172/0 (socket says 172.21.15.70:40172) 2024-04-01T01:42:10.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.197+0000 7f41dad9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2a490 0x560fdff444d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60892/0 (socket says 172.21.15.70:60892) 2024-04-01T01:42:10.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.198+0000 7f41da59c640 1 -- 172.21.15.70:0/2660418364 learned_addr learned my addr 172.21.15.70:0/2660418364 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:10.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.198+0000 7f41e3da5740 1 -- 172.21.15.70:0/2660418364 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc0755e0 con 0x560fdff2ae80 2024-04-01T01:42:10.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.198+0000 7f41e3da5740 1 -- 172.21.15.70:0/2660418364 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc077120 con 0x560fdff2a490 2024-04-01T01:42:10.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.198+0000 7f41db59e640 1 --2- 172.21.15.70:0/2660418364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2fff0 0x560fdff44a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.198+0000 7f41e3da5740 1 -- 172.21.15.70:0/2660418364 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc075cb0 con 0x560fdff2fff0 2024-04-01T01:42:10.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.198+0000 7f41da59c640 1 -- 172.21.15.70:0/2660418364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2fff0 msgr2=0x560fdff44a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:10.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.198+0000 7f41da59c640 1 --2- 172.21.15.70:0/2660418364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2fff0 0x560fdff44a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.198+0000 7f41da59c640 1 -- 172.21.15.70:0/2660418364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2a490 msgr2=0x560fdff444d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:10.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.198+0000 7f41da59c640 1 --2- 172.21.15.70:0/2660418364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2a490 0x560fdff444d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.198+0000 7f41da59c640 1 -- 172.21.15.70:0/2660418364 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fdff44270 con 0x560fdff2ae80 2024-04-01T01:42:10.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.199+0000 7f41da59c640 1 --2- 172.21.15.70:0/2660418364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2ae80 0x560fdff4cfb0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f41d0001a80 tx=0x7f41d0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=173e93dc3dc0ecbc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:10.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.199+0000 7f41c0ff9640 1 -- 172.21.15.70:0/2660418364 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d000ea70 con 0x560fdff2ae80 2024-04-01T01:42:10.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.199+0000 7f41c0ff9640 1 -- 172.21.15.70:0/2660418364 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41d000ec10 con 0x560fdff2ae80 2024-04-01T01:42:10.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.199+0000 7f41c0ff9640 1 -- 172.21.15.70:0/2660418364 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d0007170 con 0x560fdff2ae80 2024-04-01T01:42:10.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.200+0000 7f41e3da5740 1 -- 172.21.15.70:0/2660418364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2ae80 msgr2=0x560fdff4cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.200+0000 7f41e3da5740 1 --2- 172.21.15.70:0/2660418364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2ae80 0x560fdff4cfb0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f41d0001a80 tx=0x7f41d0001e70 comp rx=0 tx=0).stop 2024-04-01T01:42:10.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.201+0000 7f41e3da5740 1 -- 172.21.15.70:0/2660418364 shutdown_connections 2024-04-01T01:42:10.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.201+0000 7f41e3da5740 1 --2- 172.21.15.70:0/2660418364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2fff0 0x560fdff44a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.201+0000 7f41e3da5740 1 --2- 172.21.15.70:0/2660418364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2ae80 0x560fdff4cfb0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.201+0000 7f41e3da5740 1 --2- 172.21.15.70:0/2660418364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2a490 0x560fdff444d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.201+0000 7f41e3da5740 1 -- 172.21.15.70:0/2660418364 >> 172.21.15.70:0/2660418364 conn(0x560fdff25a00 msgr2=0x560fdff25de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.201+0000 7f41e3da5740 1 -- 172.21.15.70:0/2660418364 shutdown_connections 2024-04-01T01:42:10.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.201+0000 7f41e3da5740 1 -- 172.21.15.70:0/2660418364 wait complete. 2024-04-01T01:42:10.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.201+0000 7f41e3da5740 1 -- start start 2024-04-01T01:42:10.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.201+0000 7f41e3da5740 1 Processor -- start 2024-04-01T01:42:10.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.202+0000 7f41e3da5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2a490 0x560fdff4a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.202+0000 7f41e3da5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2ae80 0x560fdff4aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.202+0000 7f41e3da5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2fff0 0x560fdff4baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.202+0000 7f41e3da5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc076a50 con 0x560fdff2a490 2024-04-01T01:42:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.202+0000 7f41e3da5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc076380 con 0x560fdff2fff0 2024-04-01T01:42:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.202+0000 7f41e3da5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc1182d0 con 0x560fdff2ae80 2024-04-01T01:42:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.202+0000 7f41dad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2a490 0x560fdff4a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.202+0000 7f41da59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2ae80 0x560fdff4aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.202+0000 7f41dad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2a490 0x560fdff4a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40180/0 (socket says 172.21.15.70:40180) 2024-04-01T01:42:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.202+0000 7f41dad9d640 1 -- 172.21.15.70:0/3275618395 learned_addr learned my addr 172.21.15.70:0/3275618395 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.202+0000 7f41db59e640 1 --2- 172.21.15.70:0/3275618395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2fff0 0x560fdff4baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.203+0000 7f41e3da5740 10 client.0 _ll_register_callbacks cb 0x7f41d40b9430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:10.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.203+0000 7f41dad9d640 1 -- 172.21.15.70:0/3275618395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2ae80 msgr2=0x560fdff4aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:10.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.203+0000 7f41dad9d640 1 --2- 172.21.15.70:0/3275618395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2ae80 0x560fdff4aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.203+0000 7f41dad9d640 1 -- 172.21.15.70:0/3275618395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2fff0 msgr2=0x560fdff4baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:10.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.203+0000 7f41dad9d640 1 --2- 172.21.15.70:0/3275618395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2fff0 0x560fdff4baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.203+0000 7f41dad9d640 1 -- 172.21.15.70:0/3275618395 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fdff4c0e0 con 0x560fdff2a490 2024-04-01T01:42:10.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.204+0000 7f41dad9d640 1 --2- 172.21.15.70:0/3275618395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2a490 0x560fdff4a5a0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x560fdff2b8f0 tx=0x7f41cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6addd69ae8f30fab server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:10.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.204+0000 7f41beff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:42:10.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.206+0000 7f41beff5640 1 -- 172.21.15.70:0/3275618395 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc02a750 con 0x560fdff2a490 2024-04-01T01:42:10.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.207+0000 7f41e3da5740 1 -- 172.21.15.70:0/3275618395 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41d000ec10 con 0x560fdff2a490 2024-04-01T01:42:10.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.207+0000 7f41beff5640 1 -- 172.21.15.70:0/3275618395 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f41cc02ab90 con 0x560fdff2a490 2024-04-01T01:42:10.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.208+0000 7f41beff5640 1 -- 172.21.15.70:0/3275618395 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f41cc03b050 con 0x560fdff2a490 2024-04-01T01:42:10.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.208+0000 7f41e3da5740 10 client.18072 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:10.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.208+0000 7f41e3da5740 1 -- 172.21.15.70:0/3275618395 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560fdffd9960 con 0x560fdff2a490 2024-04-01T01:42:10.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.208+0000 7f41beff5640 1 -- 172.21.15.70:0/3275618395 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f41d000ec10 con 0x560fdff2a490 2024-04-01T01:42:10.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.209+0000 7f41beff5640 1 -- 172.21.15.70:0/3275618395 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f41cc0054a0 con 0x560fdff2a490 2024-04-01T01:42:10.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.209+0000 7f41e3da5740 10 client.18072 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:10.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.209+0000 7f41e3da5740 10 client.18072 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:42:10.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.209+0000 7f41e3da5740 20 client.18072 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:10.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.209+0000 7f41e3da5740 1 --2- 172.21.15.70:0/3275618395 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560fdffda360 0x560fdffdc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.209+0000 7f41e3da5740 4 client.18072 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:42:10.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.209+0000 7f41e3da5740 1 -- 172.21.15.70:0/3275618395 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560fdfe96a40 con 0x560fdffda360 2024-04-01T01:42:10.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.213+0000 7f41da59c640 1 --2- 172.21.15.70:0/3275618395 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560fdffda360 0x560fdffdc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.219+0000 7f41da59c640 1 --2- 172.21.15.70:0/3275618395 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560fdffda360 0x560fdffdc780 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=95c21d5f1f267f9e server_cookie=4894f2af333f04a2 in_seq=0 out_seq=0 2024-04-01T01:42:10.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.219+0000 7f41beff5640 10 client.18072 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:42:10.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.220+0000 7f41beff5640 1 -- 172.21.15.70:0/3275618395 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560fdfe96a40 con 0x560fdffda360 2024-04-01T01:42:10.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.220+0000 7f41beff5640 10 client.18072 handle_command_reply: tid=0 2024-04-01T01:42:10.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.220+0000 7f41e3da5740 1 client.18072 shutdown 2024-04-01T01:42:10.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.221+0000 7f41e3da5740 1 -- 172.21.15.70:0/3275618395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2a490 msgr2=0x560fdff4a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.221+0000 7f41e3da5740 1 --2- 172.21.15.70:0/3275618395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2a490 0x560fdff4a5a0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x560fdff2b8f0 tx=0x7f41cc004730 comp rx=0 tx=0).stop 2024-04-01T01:42:10.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.222+0000 7f41e3da5740 1 -- 172.21.15.70:0/3275618395 shutdown_connections 2024-04-01T01:42:10.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.222+0000 7f41e3da5740 1 --2- 172.21.15.70:0/3275618395 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x560fdffda360 0x560fdffdc780 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.222+0000 7f41e3da5740 1 --2- 172.21.15.70:0/3275618395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560fdff2fff0 0x560fdff4baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.222+0000 7f41e3da5740 1 --2- 172.21.15.70:0/3275618395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fdff2ae80 0x560fdff4aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.222+0000 7f41e3da5740 1 --2- 172.21.15.70:0/3275618395 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560fdff2a490 0x560fdff4a5a0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.222+0000 7f41e3da5740 1 -- 172.21.15.70:0/3275618395 >> 172.21.15.70:0/3275618395 conn(0x560fdff25a00 msgr2=0x560fdff2c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:10.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.222+0000 7f41beff5640 0 client.18072 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:42:10.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.223+0000 7f41e3da5740 1 -- 172.21.15.70:0/3275618395 shutdown_connections 2024-04-01T01:42:10.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.223+0000 7f41e3da5740 1 -- 172.21.15.70:0/3275618395 wait complete. 2024-04-01T01:42:10.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.226+0000 7f41e3da5740 20 client.18072 trim_cache size 0 max 16384 2024-04-01T01:42:10.225 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:42:10.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.227+0000 7f41e10eb640 1 -- 172.21.15.70:0/1470120635 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41c40a25f0 msgr2=0x7f41c40a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.227+0000 7f41e10eb640 1 --2- 172.21.15.70:0/1470120635 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41c40a25f0 0x7f41c40a4ab0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x560fdff2b280 tx=0x7f41cc00b480 comp rx=0 tx=0).stop 2024-04-01T01:42:10.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.227+0000 7f41e10eb640 1 -- 172.21.15.70:0/1470120635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41dc10da70 msgr2=0x7f41dc1addf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.227+0000 7f41e10eb640 1 --2- 172.21.15.70:0/1470120635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41dc10da70 0x7f41dc1addf0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f41d4037040 tx=0x7f41d4034880 comp rx=0 tx=0).stop 2024-04-01T01:42:10.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.228+0000 7f41e10eb640 1 -- 172.21.15.70:0/1470120635 shutdown_connections 2024-04-01T01:42:10.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.228+0000 7f41e10eb640 1 --2- 172.21.15.70:0/1470120635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41dc10da70 0x7f41dc1addf0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.228+0000 7f41e10eb640 1 --2- 172.21.15.70:0/1470120635 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f41c40a25f0 0x7f41c40a4ab0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.228+0000 7f41e10eb640 1 --2- 172.21.15.70:0/1470120635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41dc10b0c0 0x7f41dc1a6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.228+0000 7f41e10eb640 1 --2- 172.21.15.70:0/1470120635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41dc108750 0x7f41dc1a6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.228+0000 7f41e10eb640 1 -- 172.21.15.70:0/1470120635 >> 172.21.15.70:0/1470120635 conn(0x7f41dc06e5b0 msgr2=0x7f41dc108b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:10.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.233+0000 7f41e10eb640 1 -- 172.21.15.70:0/1470120635 shutdown_connections 2024-04-01T01:42:10.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.233+0000 7f41e10eb640 1 -- 172.21.15.70:0/1470120635 wait complete. 2024-04-01T01:42:10.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:09 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1568426914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:42:10.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:09 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1568426914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:42:10.260 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:42:10.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:09 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1568426914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:42:10.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.455+0000 7f5fb6dac640 1 -- 172.21.15.70:0/2270866966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fb0077cf0 msgr2=0x7f5fb0078590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.455+0000 7f5fb6dac640 1 --2- 172.21.15.70:0/2270866966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fb0077cf0 0x7f5fb0078590 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009be0 tx=0x7f5fa802bef0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.455+0000 7f5fb6dac640 1 -- 172.21.15.70:0/2270866966 shutdown_connections 2024-04-01T01:42:10.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.455+0000 7f5fb6dac640 1 --2- 172.21.15.70:0/2270866966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fb0077cf0 0x7f5fb0078590 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.455+0000 7f5fb6dac640 1 --2- 172.21.15.70:0/2270866966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb0077350 0x7f5fb00777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.455+0000 7f5fb6dac640 1 --2- 172.21.15.70:0/2270866966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fb0076110 0x7f5fb0076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.455+0000 7f5fb6dac640 1 -- 172.21.15.70:0/2270866966 >> 172.21.15.70:0/2270866966 conn(0x7f5fb00fedc0 msgr2=0x7f5fb0101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:10.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.456+0000 7f5fb6dac640 1 -- 172.21.15.70:0/2270866966 shutdown_connections 2024-04-01T01:42:10.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.456+0000 7f5fb6dac640 1 -- 172.21.15.70:0/2270866966 wait complete. 2024-04-01T01:42:10.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.457+0000 7f5fb6dac640 1 Processor -- start 2024-04-01T01:42:10.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.457+0000 7f5fb6dac640 1 -- start start 2024-04-01T01:42:10.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.457+0000 7f5fb6dac640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb0076110 0x7f5fb0106cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.457+0000 7f5fb6dac640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fb0077350 0x7f5fb0107200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.457+0000 7f5fb6dac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fb0077cf0 0x7f5fb010e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.457+0000 7f5fb6dac640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb0103b40 con 0x7f5fb0077350 2024-04-01T01:42:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.457+0000 7f5fb6dac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb01039c0 con 0x7f5fb0077cf0 2024-04-01T01:42:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.457+0000 7f5fb6dac640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb0103cc0 con 0x7f5fb0076110 2024-04-01T01:42:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.458+0000 7f5fb55a9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fb0077350 0x7f5fb0107200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.458+0000 7f5fb5daa640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb0076110 0x7f5fb0106cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.458+0000 7f5fb65ab640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fb0077cf0 0x7f5fb010e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.458+0000 7f5fb5daa640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb0076110 0x7f5fb0106cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:56950/0 (socket says 172.21.15.70:56950) 2024-04-01T01:42:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.458+0000 7f5fb5daa640 1 -- 172.21.15.70:0/2565875786 learned_addr learned my addr 172.21.15.70:0/2565875786 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.459+0000 7f5fb5daa640 1 -- 172.21.15.70:0/2565875786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fb0077cf0 msgr2=0x7f5fb010e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.459+0000 7f5fb5daa640 1 --2- 172.21.15.70:0/2565875786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fb0077cf0 0x7f5fb010e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.459+0000 7f5fb5daa640 1 -- 172.21.15.70:0/2565875786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fb0077350 msgr2=0x7f5fb0107200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.459+0000 7f5fb5daa640 1 --2- 172.21.15.70:0/2565875786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fb0077350 0x7f5fb0107200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.459+0000 7f5fb5daa640 1 -- 172.21.15.70:0/2565875786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb010e8e0 con 0x7f5fb0076110 2024-04-01T01:42:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.459+0000 7f5fb65ab640 1 --2- 172.21.15.70:0/2565875786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fb0077cf0 0x7f5fb010e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:42:10.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.459+0000 7f5fb5daa640 1 --2- 172.21.15.70:0/2565875786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb0076110 0x7f5fb0106cc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000d950 tx=0x7f5fa000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:10.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.460+0000 7f5fa6ffd640 1 -- 172.21.15.70:0/2565875786 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0017070 con 0x7f5fb0076110 2024-04-01T01:42:10.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.460+0000 7f5fa6ffd640 1 -- 172.21.15.70:0/2565875786 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fa0009c30 con 0x7f5fb0076110 2024-04-01T01:42:10.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.461+0000 7f5fa6ffd640 1 -- 172.21.15.70:0/2565875786 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa000f3c0 con 0x7f5fb0076110 2024-04-01T01:42:10.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.461+0000 7f5fb6dac640 1 -- 172.21.15.70:0/2565875786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb010ebd0 con 0x7f5fb0076110 2024-04-01T01:42:10.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.461+0000 7f5fb6dac640 1 -- 172.21.15.70:0/2565875786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb010f120 con 0x7f5fb0076110 2024-04-01T01:42:10.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.466+0000 7f5fa6ffd640 1 -- 172.21.15.70:0/2565875786 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f5fa0010460 con 0x7f5fb0076110 2024-04-01T01:42:10.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.466+0000 7f5fbabf5740 1 Processor -- start 2024-04-01T01:42:10.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.466+0000 7f5fbabf5740 1 -- start start 2024-04-01T01:42:10.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.467+0000 7f5fbabf5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843c86a0 0x5587843c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.467+0000 7f5fbabf5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843c9090 0x5587843cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.467+0000 7f5fbabf5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843ce200 0x5587843d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.467+0000 7f5fbabf5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55878432d1b0 con 0x5587843c9090 2024-04-01T01:42:10.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.467+0000 7f5fbabf5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5587842d5c00 con 0x5587843c86a0 2024-04-01T01:42:10.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.467+0000 7f5fbabf5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558784329250 con 0x5587843ce200 2024-04-01T01:42:10.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.467+0000 7f5fb65ab640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843c86a0 0x5587843c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.467+0000 7f5fb65ab640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843c86a0 0x5587843c8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60928/0 (socket says 172.21.15.70:60928) 2024-04-01T01:42:10.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.467+0000 7f5fb65ab640 1 -- 172.21.15.70:0/2723862181 learned_addr learned my addr 172.21.15.70:0/2723862181 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:10.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.467+0000 7f5fb5daa640 1 --2- 172.21.15.70:0/2723862181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843c9090 0x5587843cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.467+0000 7f5fb55a9640 1 --2- 172.21.15.70:0/2723862181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843ce200 0x5587843d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.467+0000 7f5fb65ab640 1 -- 172.21.15.70:0/2723862181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843ce200 msgr2=0x5587843d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:10.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.468+0000 7f5fb65ab640 1 --2- 172.21.15.70:0/2723862181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843ce200 0x5587843d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.468+0000 7f5fb65ab640 1 -- 172.21.15.70:0/2723862181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843c9090 msgr2=0x5587843cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:10.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.468+0000 7f5fb65ab640 1 --2- 172.21.15.70:0/2723862181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843c9090 0x5587843cdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.468+0000 7f5fb65ab640 1 -- 172.21.15.70:0/2723862181 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587842f53d0 con 0x5587843c86a0 2024-04-01T01:42:10.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.468+0000 7f5fb65ab640 1 --2- 172.21.15.70:0/2723862181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843c86a0 0x5587843c8ac0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8004af0 tx=0x7f5fa80098f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a894701eb04e973 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:10.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.469+0000 7f5f9bfff640 1 -- 172.21.15.70:0/2723862181 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8043030 con 0x5587843c86a0 2024-04-01T01:42:10.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.469+0000 7f5f9bfff640 1 -- 172.21.15.70:0/2723862181 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fa8003970 con 0x5587843c86a0 2024-04-01T01:42:10.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.469+0000 7f5f9bfff640 1 -- 172.21.15.70:0/2723862181 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa802cb90 con 0x5587843c86a0 2024-04-01T01:42:10.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.470+0000 7f5fbabf5740 1 -- 172.21.15.70:0/2723862181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843c86a0 msgr2=0x5587843c8ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.470+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/2723862181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843c86a0 0x5587843c8ac0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8004af0 tx=0x7f5fa80098f0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.470+0000 7f5fbabf5740 1 -- 172.21.15.70:0/2723862181 shutdown_connections 2024-04-01T01:42:10.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.470+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/2723862181 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843ce200 0x5587843d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.470+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/2723862181 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843c9090 0x5587843cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.470+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/2723862181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843c86a0 0x5587843c8ac0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.470+0000 7f5fbabf5740 1 -- 172.21.15.70:0/2723862181 >> 172.21.15.70:0/2723862181 conn(0x5587843c3c10 msgr2=0x5587843c4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:10.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.470+0000 7f5fa6ffd640 1 --2- 172.21.15.70:0/2565875786 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f9c0a25d0 0x7f5f9c0a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.471+0000 7f5fb55a9640 1 --2- 172.21.15.70:0/2565875786 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f9c0a25d0 0x7f5f9c0a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.471+0000 7f5fbabf5740 1 -- 172.21.15.70:0/2723862181 shutdown_connections 2024-04-01T01:42:10.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.471+0000 7f5fa6ffd640 1 -- 172.21.15.70:0/2565875786 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5fa00ce6c0 con 0x7f5fb0076110 2024-04-01T01:42:10.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.471+0000 7f5fbabf5740 1 -- 172.21.15.70:0/2723862181 wait complete. 2024-04-01T01:42:10.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.471+0000 7f5fbabf5740 1 -- start start 2024-04-01T01:42:10.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.471+0000 7f5fbabf5740 1 Processor -- start 2024-04-01T01:42:10.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.472+0000 7f5fbabf5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843c86a0 0x5587843d2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.472+0000 7f5fbabf5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843c9090 0x5587843d30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.472+0000 7f5fbabf5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843ce200 0x5587843da100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.472+0000 7f5fbabf5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5587843d1150 con 0x5587843ce200 2024-04-01T01:42:10.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.472+0000 7f5fbabf5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5587842cb470 con 0x5587843c9090 2024-04-01T01:42:10.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.472+0000 7f5fbabf5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5587843d12d0 con 0x5587843c86a0 2024-04-01T01:42:10.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.472+0000 7f5fbabf5740 10 client.0 _ll_register_callbacks cb 0x5587842e8cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:10.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.472+0000 7f5fb65ab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843ce200 0x5587843da100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.472+0000 7f5fb65ab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843ce200 0x5587843da100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40230/0 (socket says 172.21.15.70:40230) 2024-04-01T01:42:10.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.472+0000 7f5fb65ab640 1 -- 172.21.15.70:0/1336926583 learned_addr learned my addr 172.21.15.70:0/1336926583 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:10.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.472+0000 7f5fb5daa640 1 --2- 172.21.15.70:0/1336926583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843c86a0 0x5587843d2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.473+0000 7f5fb55a9640 1 --2- 172.21.15.70:0/1336926583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843c9090 0x5587843d30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.473+0000 7f5fb65ab640 1 -- 172.21.15.70:0/1336926583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843c86a0 msgr2=0x5587843d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:10.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.473+0000 7f5fb65ab640 1 --2- 172.21.15.70:0/1336926583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843c86a0 0x5587843d2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.473+0000 7f5fb65ab640 1 -- 172.21.15.70:0/1336926583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843c9090 msgr2=0x5587843d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:10.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.473+0000 7f5fb65ab640 1 --2- 172.21.15.70:0/1336926583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843c9090 0x5587843d30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.473+0000 7f5fb65ab640 1 -- 172.21.15.70:0/1336926583 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587840413f0 con 0x5587843ce200 2024-04-01T01:42:10.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.473+0000 7f5fb5daa640 1 --2- 172.21.15.70:0/1336926583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843c86a0 0x5587843d2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:42:10.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.473+0000 7f5fb55a9640 1 --2- 172.21.15.70:0/2565875786 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f9c0a25d0 0x7f5f9c0a4a90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x5587843c4b80 tx=0x7f5fac009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:10.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.473+0000 7f5fb65ab640 1 --2- 172.21.15.70:0/1336926583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843ce200 0x5587843da100 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8036040 tx=0x7f5fa8003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7960e409aae8dc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:10.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.473+0000 7f5f997fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:42:10.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.473+0000 7f5f997fa640 1 -- 172.21.15.70:0/1336926583 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa804c070 con 0x5587843ce200 2024-04-01T01:42:10.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.474+0000 7f5fbabf5740 1 -- 172.21.15.70:0/1336926583 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fa8003970 con 0x5587843ce200 2024-04-01T01:42:10.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.474+0000 7f5f997fa640 1 -- 172.21.15.70:0/1336926583 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5fa8033a90 con 0x5587843ce200 2024-04-01T01:42:10.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.474+0000 7f5f997fa640 1 -- 172.21.15.70:0/1336926583 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5fa8058050 con 0x5587843ce200 2024-04-01T01:42:10.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.474+0000 7f5fbabf5740 10 client.18084 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:10.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.474+0000 7f5fbabf5740 1 -- 172.21.15.70:0/1336926583 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55878410b7d0 con 0x5587843ce200 2024-04-01T01:42:10.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.474+0000 7f5f997fa640 1 -- 172.21.15.70:0/1336926583 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fa8003970 con 0x5587843ce200 2024-04-01T01:42:10.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.475+0000 7f5f997fa640 1 -- 172.21.15.70:0/1336926583 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5fa80344c0 con 0x5587843ce200 2024-04-01T01:42:10.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.475+0000 7f5fbabf5740 10 client.18084 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:10.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.475+0000 7f5fbabf5740 10 client.18084 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:10.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.475+0000 7f5fbabf5740 20 client.18084 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:10.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.475+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/1336926583 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5587843de6e0 0x5587843e0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.475+0000 7f5fbabf5740 4 client.18084 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:10.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.475+0000 7f5fbabf5740 1 -- 172.21.15.70:0/1336926583 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55878433a6f0 con 0x5587843de6e0 2024-04-01T01:42:10.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.476+0000 7f5fb5daa640 1 --2- 172.21.15.70:0/1336926583 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5587843de6e0 0x5587843e0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.476+0000 7f5fb5daa640 1 --2- 172.21.15.70:0/1336926583 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5587843de6e0 0x5587843e0b30 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf82ccabca8dff5a server_cookie=1025c94c80eea7d4 in_seq=0 out_seq=0 2024-04-01T01:42:10.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.476+0000 7f5f997fa640 10 client.18084 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:10.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.477+0000 7f5f997fa640 1 -- 172.21.15.70:0/1336926583 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55878433a6f0 con 0x5587843de6e0 2024-04-01T01:42:10.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.477+0000 7f5f997fa640 10 client.18084 handle_command_reply: tid=0 2024-04-01T01:42:10.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.478+0000 7f5fbabf5740 1 client.18084 shutdown 2024-04-01T01:42:10.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.478+0000 7f5fbabf5740 1 -- 172.21.15.70:0/1336926583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843ce200 msgr2=0x5587843da100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.478+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/1336926583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843ce200 0x5587843da100 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8036040 tx=0x7f5fa8003f10 comp rx=0 tx=0).stop 2024-04-01T01:42:10.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.478+0000 7f5fbabf5740 1 -- 172.21.15.70:0/1336926583 shutdown_connections 2024-04-01T01:42:10.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.478+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/1336926583 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5587843de6e0 0x5587843e0b30 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.478+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/1336926583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843ce200 0x5587843da100 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.478+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/1336926583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843c9090 0x5587843d30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.478+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/1336926583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843c86a0 0x5587843d2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.478+0000 7f5fbabf5740 1 -- 172.21.15.70:0/1336926583 >> 172.21.15.70:0/1336926583 conn(0x5587843c3c10 msgr2=0x5587843c2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:10.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.478+0000 7f5f997fa640 0 client.18084 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:10.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.478+0000 7f5fbabf5740 1 -- 172.21.15.70:0/1336926583 shutdown_connections 2024-04-01T01:42:10.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.478+0000 7f5fbabf5740 1 -- 172.21.15.70:0/1336926583 wait complete. 2024-04-01T01:42:10.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.479+0000 7f5fbabf5740 20 client.18084 trim_cache size 0 max 16384 2024-04-01T01:42:10.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.505+0000 7f5fbabf5740 1 Processor -- start 2024-04-01T01:42:10.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.505+0000 7f5fbabf5740 1 -- start start 2024-04-01T01:42:10.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.505+0000 7f5fbabf5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843c86a0 0x5587843c2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.506+0000 7f5fbabf5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843c9090 0x5587843da0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.506+0000 7f5fbabf5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843ce200 0x5587843c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.506+0000 7f5fbabf5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb01c17c0 con 0x5587843ce200 2024-04-01T01:42:10.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.506+0000 7f5fbabf5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb01c3300 con 0x5587843c86a0 2024-04-01T01:42:10.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.506+0000 7f5fbabf5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb01c1e90 con 0x5587843c9090 2024-04-01T01:42:10.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.506+0000 7f5fb65ab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843ce200 0x5587843c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.506+0000 7f5fb65ab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843ce200 0x5587843c2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40240/0 (socket says 172.21.15.70:40240) 2024-04-01T01:42:10.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.506+0000 7f5fb65ab640 1 -- 172.21.15.70:0/2501260107 learned_addr learned my addr 172.21.15.70:0/2501260107 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:10.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.507+0000 7f5fb65ab640 1 -- 172.21.15.70:0/2501260107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843c9090 msgr2=0x5587843da0c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:42:10.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.507+0000 7f5fb65ab640 1 --2- 172.21.15.70:0/2501260107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843c9090 0x5587843da0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.507+0000 7f5fb65ab640 1 -- 172.21.15.70:0/2501260107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843c86a0 msgr2=0x5587843c2700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:42:10.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.507+0000 7f5fb65ab640 1 --2- 172.21.15.70:0/2501260107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843c86a0 0x5587843c2700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.507+0000 7f5fb65ab640 1 -- 172.21.15.70:0/2501260107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587843d1c40 con 0x5587843ce200 2024-04-01T01:42:10.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.507+0000 7f5fb65ab640 1 --2- 172.21.15.70:0/2501260107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843ce200 0x5587843c2c40 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8034ab0 tx=0x7f5fa80098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9437b3ebd92e29cc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:10.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.508+0000 7f5f9b7fe640 1 -- 172.21.15.70:0/2501260107 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa804c040 con 0x5587843ce200 2024-04-01T01:42:10.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.508+0000 7f5f9b7fe640 1 -- 172.21.15.70:0/2501260107 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fa8055070 con 0x5587843ce200 2024-04-01T01:42:10.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.508+0000 7f5f9b7fe640 1 -- 172.21.15.70:0/2501260107 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8043040 con 0x5587843ce200 2024-04-01T01:42:10.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.509+0000 7f5fbabf5740 1 -- 172.21.15.70:0/2501260107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843ce200 msgr2=0x5587843c2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.509+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/2501260107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843ce200 0x5587843c2c40 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8034ab0 tx=0x7f5fa80098f0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.510+0000 7f5fbabf5740 1 -- 172.21.15.70:0/2501260107 shutdown_connections 2024-04-01T01:42:10.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.510+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/2501260107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843ce200 0x5587843c2c40 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.510+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/2501260107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843c9090 0x5587843da0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.510+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/2501260107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843c86a0 0x5587843c2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.510+0000 7f5fbabf5740 1 -- 172.21.15.70:0/2501260107 >> 172.21.15.70:0/2501260107 conn(0x5587843c3c10 msgr2=0x5587843d2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:10.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.514+0000 7f5fbabf5740 1 -- 172.21.15.70:0/2501260107 shutdown_connections 2024-04-01T01:42:10.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.514+0000 7f5fbabf5740 1 -- 172.21.15.70:0/2501260107 wait complete. 2024-04-01T01:42:10.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.515+0000 7f5fbabf5740 1 -- start start 2024-04-01T01:42:10.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.515+0000 7f5fbabf5740 1 Processor -- start 2024-04-01T01:42:10.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.516+0000 7f5fbabf5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843c86a0 0x5587843d77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.516+0000 7f5fbabf5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843c9090 0x5587843d7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.516+0000 7f5fbabf5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843ce200 0x5587843d8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.516+0000 7f5fbabf5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb01c2c30 con 0x5587843c86a0 2024-04-01T01:42:10.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.516+0000 7f5fbabf5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb01c2560 con 0x5587843ce200 2024-04-01T01:42:10.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.516+0000 7f5fbabf5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb01bf7a0 con 0x5587843c9090 2024-04-01T01:42:10.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.516+0000 7f5fb55a9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843c9090 0x5587843d7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.516+0000 7f5fbabf5740 10 client.0 _ll_register_callbacks cb 0x7f5fa8009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:10.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.516+0000 7f5fb65ab640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843ce200 0x5587843d8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.516+0000 7f5fb65ab640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843ce200 0x5587843d8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60942/0 (socket says 172.21.15.70:60942) 2024-04-01T01:42:10.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.516+0000 7f5fb65ab640 1 -- 172.21.15.70:0/1094543110 learned_addr learned my addr 172.21.15.70:0/1094543110 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:10.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.517+0000 7f5fb65ab640 1 -- 172.21.15.70:0/1094543110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843c9090 msgr2=0x5587843d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:10.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.517+0000 7f5fb65ab640 1 --2- 172.21.15.70:0/1094543110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843c9090 0x5587843d7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.517+0000 7f5fb65ab640 1 -- 172.21.15.70:0/1094543110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843c86a0 msgr2=0x5587843d77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:10.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.517+0000 7f5fb65ab640 1 --2- 172.21.15.70:0/1094543110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843c86a0 0x5587843d77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.517+0000 7f5fb65ab640 1 -- 172.21.15.70:0/1094543110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587843d92e0 con 0x5587843ce200 2024-04-01T01:42:10.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.520+0000 7f5fb65ab640 1 --2- 172.21.15.70:0/1094543110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843ce200 0x5587843d8cf0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8034a30 tx=0x7f5fa8035fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9374173c0774ce23 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:10.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.520+0000 7f5f997fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:10.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.520+0000 7f5f997fa640 1 -- 172.21.15.70:0/1094543110 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8043030 con 0x5587843ce200 2024-04-01T01:42:10.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.520+0000 7f5fbabf5740 1 -- 172.21.15.70:0/1094543110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fa8055070 con 0x5587843ce200 2024-04-01T01:42:10.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.520+0000 7f5f997fa640 1 -- 172.21.15.70:0/1094543110 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5fa8034640 con 0x5587843ce200 2024-04-01T01:42:10.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.521+0000 7f5f997fa640 1 -- 172.21.15.70:0/1094543110 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5fa806d050 con 0x5587843ce200 2024-04-01T01:42:10.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.521+0000 7f5fbabf5740 10 client.28061 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:10.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.521+0000 7f5fbabf5740 1 -- 172.21.15.70:0/1094543110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558784444a70 con 0x5587843ce200 2024-04-01T01:42:10.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.521+0000 7f5f997fa640 1 -- 172.21.15.70:0/1094543110 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fa8055070 con 0x5587843ce200 2024-04-01T01:42:10.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.521+0000 7f5f997fa640 1 -- 172.21.15.70:0/1094543110 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5fa803f440 con 0x5587843ce200 2024-04-01T01:42:10.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.521+0000 7f5fbabf5740 10 client.28061 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:10.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.521+0000 7f5fbabf5740 10 client.28061 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:10.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.521+0000 7f5fbabf5740 20 client.28061 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:10.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.521+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/1094543110 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558784445350 0x558784447770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:10.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.521+0000 7f5fbabf5740 4 client.28061 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:42:10.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.521+0000 7f5fbabf5740 1 -- 172.21.15.70:0/1094543110 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558784334a40 con 0x558784445350 2024-04-01T01:42:10.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.522+0000 7f5fb5daa640 1 --2- 172.21.15.70:0/1094543110 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558784445350 0x558784447770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:10.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.522+0000 7f5f997fa640 10 client.28061 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:10.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.522+0000 7f5fb5daa640 1 --2- 172.21.15.70:0/1094543110 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558784445350 0x558784447770 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99eaaa4d92fd4c53 server_cookie=a585e10ddadfad60 in_seq=0 out_seq=0 2024-04-01T01:42:10.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.522+0000 7f5f997fa640 1 -- 172.21.15.70:0/1094543110 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558784334a40 con 0x558784445350 2024-04-01T01:42:10.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.522+0000 7f5f997fa640 10 client.28061 handle_command_reply: tid=0 2024-04-01T01:42:10.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.523+0000 7f5fbabf5740 1 client.28061 shutdown 2024-04-01T01:42:10.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.523+0000 7f5fbabf5740 1 -- 172.21.15.70:0/1094543110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843ce200 msgr2=0x5587843d8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.523+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/1094543110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843ce200 0x5587843d8cf0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8034a30 tx=0x7f5fa8035fe0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.523+0000 7f5fbabf5740 1 -- 172.21.15.70:0/1094543110 shutdown_connections 2024-04-01T01:42:10.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.523+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/1094543110 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558784445350 0x558784447770 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.523+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/1094543110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587843ce200 0x5587843d8cf0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.523+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/1094543110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587843c9090 0x5587843d7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.523+0000 7f5fbabf5740 1 --2- 172.21.15.70:0/1094543110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587843c86a0 0x5587843d77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.523+0000 7f5fbabf5740 1 -- 172.21.15.70:0/1094543110 >> 172.21.15.70:0/1094543110 conn(0x5587843c3c10 msgr2=0x5587843c6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:10.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.524+0000 7f5f997fa640 0 client.28061 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:10.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.524+0000 7f5fbabf5740 1 -- 172.21.15.70:0/1094543110 shutdown_connections 2024-04-01T01:42:10.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.524+0000 7f5fbabf5740 1 -- 172.21.15.70:0/1094543110 wait complete. 2024-04-01T01:42:10.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.524+0000 7f5fbabf5740 20 client.28061 trim_cache size 0 max 16384 2024-04-01T01:42:10.525 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:42:10.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.526+0000 7f5fb6dac640 1 -- 172.21.15.70:0/2565875786 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f9c0a25d0 msgr2=0x7f5f9c0a4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.526+0000 7f5fb6dac640 1 --2- 172.21.15.70:0/2565875786 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f9c0a25d0 0x7f5f9c0a4a90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x5587843c4b80 tx=0x7f5fac009260 comp rx=0 tx=0).stop 2024-04-01T01:42:10.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.526+0000 7f5fb6dac640 1 -- 172.21.15.70:0/2565875786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb0076110 msgr2=0x7f5fb0106cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:10.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.526+0000 7f5fb6dac640 1 --2- 172.21.15.70:0/2565875786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb0076110 0x7f5fb0106cc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000d950 tx=0x7f5fa000de10 comp rx=0 tx=0).stop 2024-04-01T01:42:10.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.526+0000 7f5fb6dac640 1 -- 172.21.15.70:0/2565875786 shutdown_connections 2024-04-01T01:42:10.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.526+0000 7f5fb6dac640 1 --2- 172.21.15.70:0/2565875786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fb0077cf0 0x7f5fb010e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.526+0000 7f5fb6dac640 1 --2- 172.21.15.70:0/2565875786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fb0077350 0x7f5fb0107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.526+0000 7f5fb6dac640 1 --2- 172.21.15.70:0/2565875786 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f9c0a25d0 0x7f5f9c0a4a90 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.526+0000 7f5fb6dac640 1 --2- 172.21.15.70:0/2565875786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb0076110 0x7f5fb0106cc0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:10.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.526+0000 7f5fb6dac640 1 -- 172.21.15.70:0/2565875786 >> 172.21.15.70:0/2565875786 conn(0x7f5fb00fedc0 msgr2=0x7f5fb01003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:10.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.528+0000 7f5fb6dac640 1 -- 172.21.15.70:0/2565875786 shutdown_connections 2024-04-01T01:42:10.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:10.528+0000 7f5fb6dac640 1 -- 172.21.15.70:0/2565875786 wait complete. 2024-04-01T01:42:11.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:10 smithi189 ceph-mon[35753]: pgmap v620: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 6.2 MiB/s wr, 707 op/s 2024-04-01T01:42:11.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:10 smithi070 ceph-mon[32179]: pgmap v620: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 6.2 MiB/s wr, 707 op/s 2024-04-01T01:42:11.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:10 smithi081 ceph-mon[35142]: pgmap v620: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 6.2 MiB/s wr, 707 op/s 2024-04-01T01:42:11.549 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:42:11.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:42:11.549 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:42:11.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.729+0000 7efc9feed640 1 -- 172.21.15.70:0/3412496456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc981090b0 msgr2=0x7efc9810fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:11.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.729+0000 7efc9feed640 1 --2- 172.21.15.70:0/3412496456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc981090b0 0x7efc9810fa90 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7efc88009900 tx=0x7efc8802bc40 comp rx=0 tx=0).stop 2024-04-01T01:42:11.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.730+0000 7efc9feed640 1 -- 172.21.15.70:0/3412496456 shutdown_connections 2024-04-01T01:42:11.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.730+0000 7efc9feed640 1 --2- 172.21.15.70:0/3412496456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc9810ffd0 0x7efc981123c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.730+0000 7efc9feed640 1 --2- 172.21.15.70:0/3412496456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc981090b0 0x7efc9810fa90 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.730+0000 7efc9feed640 1 --2- 172.21.15.70:0/3412496456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc98108750 0x7efc98108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.730+0000 7efc9feed640 1 -- 172.21.15.70:0/3412496456 >> 172.21.15.70:0/3412496456 conn(0x7efc9806e950 msgr2=0x7efc98070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:11.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.730+0000 7efc9feed640 1 -- 172.21.15.70:0/3412496456 shutdown_connections 2024-04-01T01:42:11.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.730+0000 7efc9feed640 1 -- 172.21.15.70:0/3412496456 wait complete. 2024-04-01T01:42:11.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.731+0000 7efc9feed640 1 Processor -- start 2024-04-01T01:42:11.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.731+0000 7efc9feed640 1 -- start start 2024-04-01T01:42:11.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.731+0000 7efc9feed640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc98108750 0x7efc981a6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.731+0000 7efc9feed640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc981090b0 0x7efc981a6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.731+0000 7efc9feed640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc9810ffd0 0x7efc981ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.731+0000 7efc9feed640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efc9810b350 con 0x7efc9810ffd0 2024-04-01T01:42:11.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.731+0000 7efc9feed640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efc9810b1d0 con 0x7efc981090b0 2024-04-01T01:42:11.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.731+0000 7efc9feed640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efc9810b4d0 con 0x7efc98108750 2024-04-01T01:42:11.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.731+0000 7efc9e48a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc9810ffd0 0x7efc981ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.731+0000 7efc9dc89640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc98108750 0x7efc981a6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.731+0000 7efc9dc89640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc98108750 0x7efc981a6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:56992/0 (socket says 172.21.15.70:56992) 2024-04-01T01:42:11.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.731+0000 7efc9e48a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc9810ffd0 0x7efc981ade10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40264/0 (socket says 172.21.15.70:40264) 2024-04-01T01:42:11.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.731+0000 7efc9dc89640 1 -- 172.21.15.70:0/3321758897 learned_addr learned my addr 172.21.15.70:0/3321758897 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:11.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.732+0000 7efc9d488640 1 --2- 172.21.15.70:0/3321758897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc981090b0 0x7efc981a6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.732+0000 7efc9dc89640 1 -- 172.21.15.70:0/3321758897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc981090b0 msgr2=0x7efc981a6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:11.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.732+0000 7efc9dc89640 1 --2- 172.21.15.70:0/3321758897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc981090b0 0x7efc981a6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.732+0000 7efc9dc89640 1 -- 172.21.15.70:0/3321758897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc9810ffd0 msgr2=0x7efc981ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:11.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.732+0000 7efc9dc89640 1 --2- 172.21.15.70:0/3321758897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc9810ffd0 0x7efc981ade10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.732+0000 7efc9dc89640 1 -- 172.21.15.70:0/3321758897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc981ae480 con 0x7efc98108750 2024-04-01T01:42:11.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.735+0000 7efc9dc89640 1 --2- 172.21.15.70:0/3321758897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc98108750 0x7efc981a6860 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7efc9000b9c0 tx=0x7efc9000be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:11.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.736+0000 7efc8effd640 1 -- 172.21.15.70:0/3321758897 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90019070 con 0x7efc98108750 2024-04-01T01:42:11.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.736+0000 7efc8effd640 1 -- 172.21.15.70:0/3321758897 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc9000cba0 con 0x7efc98108750 2024-04-01T01:42:11.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.736+0000 7efc8effd640 1 -- 172.21.15.70:0/3321758897 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc900125a0 con 0x7efc98108750 2024-04-01T01:42:11.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.737+0000 7efc9feed640 1 -- 172.21.15.70:0/3321758897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc981ae770 con 0x7efc98108750 2024-04-01T01:42:11.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.737+0000 7efc9feed640 1 -- 172.21.15.70:0/3321758897 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc981aecc0 con 0x7efc98108750 2024-04-01T01:42:11.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.740+0000 7efc8effd640 1 -- 172.21.15.70:0/3321758897 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7efc90014440 con 0x7efc98108750 2024-04-01T01:42:11.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.741+0000 7efca2baf740 1 Processor -- start 2024-04-01T01:42:11.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.741+0000 7efca2baf740 1 -- start start 2024-04-01T01:42:11.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.741+0000 7efca2baf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b38e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.741+0000 7efca2baf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 0x55d667b3df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.742+0000 7efca2baf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 0x55d667b40a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.742+0000 7efca2baf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d667a3a0b0 con 0x55d667b38a10 2024-04-01T01:42:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.742+0000 7efca2baf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d667a9cea0 con 0x55d667b3e570 2024-04-01T01:42:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.742+0000 7efca2baf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d667a45c00 con 0x55d667b39400 2024-04-01T01:42:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.742+0000 7efc8effd640 1 --2- 172.21.15.70:0/3321758897 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc840a25f0 0x7efc840a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.742+0000 7efc9d488640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 0x55d667b40a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.742+0000 7efc9d488640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 0x55d667b40a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60976/0 (socket says 172.21.15.70:60976) 2024-04-01T01:42:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.742+0000 7efc9d488640 1 -- 172.21.15.70:0/1751877054 learned_addr learned my addr 172.21.15.70:0/1751877054 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.742+0000 7efc9dc89640 1 --2- 172.21.15.70:0/1751877054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 0x55d667b3df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.742+0000 7efc8effd640 1 -- 172.21.15.70:0/3321758897 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7efc900cdf20 con 0x7efc98108750 2024-04-01T01:42:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.742+0000 7efc9d488640 1 -- 172.21.15.70:0/1751877054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 msgr2=0x55d667b3df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.742+0000 7efc9d488640 1 --2- 172.21.15.70:0/1751877054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 0x55d667b3df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.742+0000 7efc9d488640 1 -- 172.21.15.70:0/1751877054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 msgr2=0x55d667b38e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:11.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.742+0000 7efc9d488640 1 --2- 172.21.15.70:0/1751877054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b38e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.742+0000 7efc9d488640 1 -- 172.21.15.70:0/1751877054 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6678797e0 con 0x55d667b3e570 2024-04-01T01:42:11.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.743+0000 7efc9d488640 1 --2- 172.21.15.70:0/1751877054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 0x55d667b40a00 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7efc88009230 tx=0x7efc8802c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=735e13c3fd50c13d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:11.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.743+0000 7efc9e48a640 1 --2- 172.21.15.70:0/3321758897 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc840a25f0 0x7efc840a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.744+0000 7efc9e48a640 1 --2- 172.21.15.70:0/3321758897 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc840a25f0 0x7efc840a4ab0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7efc9400b590 tx=0x7efc9400b520 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:11.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.744+0000 7efc83fff640 1 -- 172.21.15.70:0/1751877054 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc88045020 con 0x55d667b3e570 2024-04-01T01:42:11.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.744+0000 7efc83fff640 1 -- 172.21.15.70:0/1751877054 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc8802cba0 con 0x55d667b3e570 2024-04-01T01:42:11.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.744+0000 7efc83fff640 1 -- 172.21.15.70:0/1751877054 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc88004920 con 0x55d667b3e570 2024-04-01T01:42:11.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.745+0000 7efca2baf740 1 -- 172.21.15.70:0/1751877054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 msgr2=0x55d667b40a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:11.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.745+0000 7efca2baf740 1 --2- 172.21.15.70:0/1751877054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 0x55d667b40a00 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7efc88009230 tx=0x7efc8802c8f0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.745+0000 7efca2baf740 1 -- 172.21.15.70:0/1751877054 shutdown_connections 2024-04-01T01:42:11.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.745+0000 7efca2baf740 1 --2- 172.21.15.70:0/1751877054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 0x55d667b40a00 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.745+0000 7efca2baf740 1 --2- 172.21.15.70:0/1751877054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 0x55d667b3df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.745+0000 7efca2baf740 1 --2- 172.21.15.70:0/1751877054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b38e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.745+0000 7efca2baf740 1 -- 172.21.15.70:0/1751877054 >> 172.21.15.70:0/1751877054 conn(0x55d667b33f80 msgr2=0x55d667b34380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:11.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.745+0000 7efca2baf740 1 -- 172.21.15.70:0/1751877054 shutdown_connections 2024-04-01T01:42:11.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.745+0000 7efca2baf740 1 -- 172.21.15.70:0/1751877054 wait complete. 2024-04-01T01:42:11.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.746+0000 7efca2baf740 1 -- start start 2024-04-01T01:42:11.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.746+0000 7efca2baf740 1 Processor -- start 2024-04-01T01:42:11.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.746+0000 7efca2baf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b42e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.746+0000 7efca2baf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 0x55d667b43350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.746+0000 7efca2baf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 0x55d667b4a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.746+0000 7efca2baf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d667a3b3d0 con 0x55d667b38a10 2024-04-01T01:42:11.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.746+0000 7efca2baf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d667a98f40 con 0x55d667b3e570 2024-04-01T01:42:11.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.746+0000 7efca2baf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d667b41510 con 0x55d667b39400 2024-04-01T01:42:11.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.746+0000 7efca2baf740 10 client.0 _ll_register_callbacks cb 0x55d667a58b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:11.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.747+0000 7efc9d488640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 0x55d667b43350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.747+0000 7efc9d488640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 0x55d667b43350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57000/0 (socket says 172.21.15.70:57000) 2024-04-01T01:42:11.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.747+0000 7efc9d488640 1 -- 172.21.15.70:0/2840143726 learned_addr learned my addr 172.21.15.70:0/2840143726 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:11.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.747+0000 7efc9dc89640 1 --2- 172.21.15.70:0/2840143726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b42e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.747+0000 7efc9e48a640 1 --2- 172.21.15.70:0/2840143726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 0x55d667b4a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.747+0000 7efc9d488640 1 -- 172.21.15.70:0/2840143726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 msgr2=0x55d667b4a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:11.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.747+0000 7efc9d488640 1 --2- 172.21.15.70:0/2840143726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 0x55d667b4a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.747+0000 7efc9d488640 1 -- 172.21.15.70:0/2840143726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 msgr2=0x55d667b42e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:11.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.747+0000 7efc9d488640 1 --2- 172.21.15.70:0/2840143726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b42e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.747+0000 7efc9d488640 1 -- 172.21.15.70:0/2840143726 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6678d9f70 con 0x55d667b39400 2024-04-01T01:42:11.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.748+0000 7efc9d488640 1 --2- 172.21.15.70:0/2840143726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 0x55d667b43350 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7efc8802c180 tx=0x7efc88003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44403dfe9a4c73ac server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:11.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.748+0000 7efc817fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:42:11.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.748+0000 7efc817fa640 1 -- 172.21.15.70:0/2840143726 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc88045070 con 0x55d667b39400 2024-04-01T01:42:11.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.748+0000 7efca2baf740 1 -- 172.21.15.70:0/2840143726 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc8802cba0 con 0x55d667b39400 2024-04-01T01:42:11.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.748+0000 7efc817fa640 1 -- 172.21.15.70:0/2840143726 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7efc88003da0 con 0x55d667b39400 2024-04-01T01:42:11.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.748+0000 7efc817fa640 1 -- 172.21.15.70:0/2840143726 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efc88059050 con 0x55d667b39400 2024-04-01T01:42:11.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.748+0000 7efca2baf740 10 client.27898 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:11.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.748+0000 7efca2baf740 1 -- 172.21.15.70:0/2840143726 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d667771170 con 0x55d667b39400 2024-04-01T01:42:11.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.749+0000 7efc817fa640 1 -- 172.21.15.70:0/2840143726 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc8802cba0 con 0x55d667b39400 2024-04-01T01:42:11.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.749+0000 7efc817fa640 1 -- 172.21.15.70:0/2840143726 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7efc88054030 con 0x55d667b39400 2024-04-01T01:42:11.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.749+0000 7efca2baf740 10 client.27898 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:11.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.749+0000 7efca2baf740 10 client.27898 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:11.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.749+0000 7efca2baf740 20 client.27898 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:11.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.749+0000 7efca2baf740 1 --2- 172.21.15.70:0/2840143726 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d667b4e920 0x55d667b50d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.749+0000 7efca2baf740 4 client.27898 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:11.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.749+0000 7efca2baf740 1 -- 172.21.15.70:0/2840143726 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d667aaa3e0 con 0x55d667b4e920 2024-04-01T01:42:11.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.751+0000 7efc9dc89640 1 --2- 172.21.15.70:0/2840143726 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d667b4e920 0x55d667b50d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.753+0000 7efc9dc89640 1 --2- 172.21.15.70:0/2840143726 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d667b4e920 0x55d667b50d30 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d955a6bfa719e2ec server_cookie=5468b77ffc2ce82e in_seq=0 out_seq=0 2024-04-01T01:42:11.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.753+0000 7efc817fa640 10 client.27898 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:11.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.754+0000 7efc817fa640 1 -- 172.21.15.70:0/2840143726 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d667aaa3e0 con 0x55d667b4e920 2024-04-01T01:42:11.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.754+0000 7efc817fa640 10 client.27898 handle_command_reply: tid=0 2024-04-01T01:42:11.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.754+0000 7efca2baf740 1 client.27898 shutdown 2024-04-01T01:42:11.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.754+0000 7efca2baf740 1 -- 172.21.15.70:0/2840143726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 msgr2=0x55d667b43350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:11.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.754+0000 7efca2baf740 1 --2- 172.21.15.70:0/2840143726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 0x55d667b43350 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7efc8802c180 tx=0x7efc88003820 comp rx=0 tx=0).stop 2024-04-01T01:42:11.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.754+0000 7efca2baf740 1 -- 172.21.15.70:0/2840143726 shutdown_connections 2024-04-01T01:42:11.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.754+0000 7efca2baf740 1 --2- 172.21.15.70:0/2840143726 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d667b4e920 0x55d667b50d30 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.754+0000 7efca2baf740 1 --2- 172.21.15.70:0/2840143726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 0x55d667b4a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.754+0000 7efca2baf740 1 --2- 172.21.15.70:0/2840143726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 0x55d667b43350 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.754+0000 7efca2baf740 1 --2- 172.21.15.70:0/2840143726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b42e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.754+0000 7efca2baf740 1 -- 172.21.15.70:0/2840143726 >> 172.21.15.70:0/2840143726 conn(0x55d667b33f80 msgr2=0x55d667b33180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:11.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.755+0000 7efc817fa640 0 client.27898 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:11.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.755+0000 7efca2baf740 1 -- 172.21.15.70:0/2840143726 shutdown_connections 2024-04-01T01:42:11.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.755+0000 7efca2baf740 1 -- 172.21.15.70:0/2840143726 wait complete. 2024-04-01T01:42:11.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.755+0000 7efca2baf740 20 client.27898 trim_cache size 0 max 16384 2024-04-01T01:42:11.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.790+0000 7efca2baf740 1 Processor -- start 2024-04-01T01:42:11.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.790+0000 7efca2baf740 1 -- start start 2024-04-01T01:42:11.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.791+0000 7efca2baf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b328f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.791+0000 7efca2baf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 0x55d667b4a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.791+0000 7efca2baf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 0x55d667b32e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.791+0000 7efca2baf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efc981b9290 con 0x55d667b38a10 2024-04-01T01:42:11.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.791+0000 7efca2baf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efc981badd0 con 0x55d667b3e570 2024-04-01T01:42:11.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.791+0000 7efca2baf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efc981b9960 con 0x55d667b39400 2024-04-01T01:42:11.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.791+0000 7efc9e48a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 0x55d667b32e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.791+0000 7efc9d488640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 0x55d667b4a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.791+0000 7efc9e48a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 0x55d667b32e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60994/0 (socket says 172.21.15.70:60994) 2024-04-01T01:42:11.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.791+0000 7efc9e48a640 1 -- 172.21.15.70:0/1920033513 learned_addr learned my addr 172.21.15.70:0/1920033513 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:11.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.791+0000 7efc9dc89640 1 --2- 172.21.15.70:0/1920033513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b328f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.791+0000 7efc9e48a640 1 -- 172.21.15.70:0/1920033513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 msgr2=0x55d667b4a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:11.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.792+0000 7efc9e48a640 1 --2- 172.21.15.70:0/1920033513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 0x55d667b4a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.792+0000 7efc9e48a640 1 -- 172.21.15.70:0/1920033513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 msgr2=0x55d667b328f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:11.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.792+0000 7efc9e48a640 1 --2- 172.21.15.70:0/1920033513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b328f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.792+0000 7efc9e48a640 1 -- 172.21.15.70:0/1920033513 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d667b38650 con 0x55d667b3e570 2024-04-01T01:42:11.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.792+0000 7efc9dc89640 1 --2- 172.21.15.70:0/1920033513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b328f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:42:11.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.792+0000 7efc9d488640 1 --2- 172.21.15.70:0/1920033513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 0x55d667b4a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:42:11.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.792+0000 7efc9e48a640 1 --2- 172.21.15.70:0/1920033513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 0x55d667b32e30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7efc94010fa0 tx=0x7efc94013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=786df66c14e442fe server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:11.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.792+0000 7efc837fe640 1 -- 172.21.15.70:0/1920033513 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc94005220 con 0x55d667b3e570 2024-04-01T01:42:11.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.792+0000 7efc837fe640 1 -- 172.21.15.70:0/1920033513 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc94023070 con 0x55d667b3e570 2024-04-01T01:42:11.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.792+0000 7efc837fe640 1 -- 172.21.15.70:0/1920033513 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9401f690 con 0x55d667b3e570 2024-04-01T01:42:11.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.793+0000 7efca2baf740 1 -- 172.21.15.70:0/1920033513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 msgr2=0x55d667b32e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:11.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.793+0000 7efca2baf740 1 --2- 172.21.15.70:0/1920033513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 0x55d667b32e30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7efc94010fa0 tx=0x7efc94013000 comp rx=0 tx=0).stop 2024-04-01T01:42:11.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.793+0000 7efca2baf740 1 -- 172.21.15.70:0/1920033513 shutdown_connections 2024-04-01T01:42:11.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.793+0000 7efca2baf740 1 --2- 172.21.15.70:0/1920033513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b3e570 0x55d667b32e30 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.793+0000 7efca2baf740 1 --2- 172.21.15.70:0/1920033513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b39400 0x55d667b4a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.793+0000 7efca2baf740 1 --2- 172.21.15.70:0/1920033513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b328f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.793+0000 7efca2baf740 1 -- 172.21.15.70:0/1920033513 >> 172.21.15.70:0/1920033513 conn(0x55d667b33f80 msgr2=0x55d667b3a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:11.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.793+0000 7efca2baf740 1 -- 172.21.15.70:0/1920033513 shutdown_connections 2024-04-01T01:42:11.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.794+0000 7efca2baf740 1 -- 172.21.15.70:0/1920033513 wait complete. 2024-04-01T01:42:11.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.794+0000 7efca2baf740 1 -- start start 2024-04-01T01:42:11.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.794+0000 7efca2baf740 1 Processor -- start 2024-04-01T01:42:11.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.795+0000 7efca2baf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b478f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.795+0000 7efca2baf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b39400 0x55d667b47e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.795+0000 7efca2baf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b3e570 0x55d667b48d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.795+0000 7efca2baf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efc981ba700 con 0x55d667b38a10 2024-04-01T01:42:11.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.795+0000 7efca2baf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efc981ba030 con 0x55d667b39400 2024-04-01T01:42:11.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.795+0000 7efca2baf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efc981b7270 con 0x55d667b3e570 2024-04-01T01:42:11.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.795+0000 7efca2baf740 10 client.0 _ll_register_callbacks cb 0x7efc88004420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:11.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.795+0000 7efc9dc89640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b478f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.795+0000 7efc9dc89640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b478f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40294/0 (socket says 172.21.15.70:40294) 2024-04-01T01:42:11.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.795+0000 7efc9dc89640 1 -- 172.21.15.70:0/4185038794 learned_addr learned my addr 172.21.15.70:0/4185038794 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:11.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.795+0000 7efc9d488640 1 --2- 172.21.15.70:0/4185038794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b39400 0x55d667b47e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.795+0000 7efc9e48a640 1 --2- 172.21.15.70:0/4185038794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b3e570 0x55d667b48d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.796+0000 7efc9e48a640 1 -- 172.21.15.70:0/4185038794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b39400 msgr2=0x55d667b47e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:11.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.796+0000 7efc9e48a640 1 --2- 172.21.15.70:0/4185038794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b39400 0x55d667b47e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.796+0000 7efc9e48a640 1 -- 172.21.15.70:0/4185038794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 msgr2=0x55d667b478f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:11.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.796+0000 7efc9e48a640 1 --2- 172.21.15.70:0/4185038794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b478f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.796+0000 7efc9e48a640 1 -- 172.21.15.70:0/4185038794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d667b49370 con 0x55d667b3e570 2024-04-01T01:42:11.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.796+0000 7efc9e48a640 1 --2- 172.21.15.70:0/4185038794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b3e570 0x55d667b48d80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efc9400ba30 tx=0x7efc94005230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30b82039cdb7b40a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:11.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.796+0000 7efc817fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:42:11.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.796+0000 7efc817fa640 1 -- 172.21.15.70:0/4185038794 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc94036060 con 0x55d667b3e570 2024-04-01T01:42:11.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.796+0000 7efc817fa640 1 -- 172.21.15.70:0/4185038794 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7efc940048d0 con 0x55d667b3e570 2024-04-01T01:42:11.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.796+0000 7efca2baf740 1 -- 172.21.15.70:0/4185038794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc94023070 con 0x55d667b3e570 2024-04-01T01:42:11.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.797+0000 7efc817fa640 1 -- 172.21.15.70:0/4185038794 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efc9403d050 con 0x55d667b3e570 2024-04-01T01:42:11.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.797+0000 7efca2baf740 10 client.27904 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:11.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.797+0000 7efca2baf740 1 -- 172.21.15.70:0/4185038794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d667b4a6b0 con 0x55d667b3e570 2024-04-01T01:42:11.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.798+0000 7efc817fa640 1 -- 172.21.15.70:0/4185038794 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc94023070 con 0x55d667b3e570 2024-04-01T01:42:11.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.798+0000 7efc817fa640 1 -- 172.21.15.70:0/4185038794 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7efc94030070 con 0x55d667b3e570 2024-04-01T01:42:11.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.798+0000 7efca2baf740 10 client.27904 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:11.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.798+0000 7efca2baf740 10 client.27904 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:11.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.798+0000 7efca2baf740 20 client.27904 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:11.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.798+0000 7efca2baf740 1 --2- 172.21.15.70:0/4185038794 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d667bb5340 0x55d667bb7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:11.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.798+0000 7efca2baf740 4 client.27904 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:42:11.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.798+0000 7efca2baf740 1 -- 172.21.15.70:0/4185038794 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d667b3e2e0 con 0x55d667bb5340 2024-04-01T01:42:11.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.798+0000 7efc9dc89640 1 --2- 172.21.15.70:0/4185038794 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d667bb5340 0x55d667bb7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:11.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.799+0000 7efc817fa640 10 client.27904 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:11.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.799+0000 7efc9dc89640 1 --2- 172.21.15.70:0/4185038794 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d667bb5340 0x55d667bb7760 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f226bdf9d273057 server_cookie=15c48e855fbfdb12 in_seq=0 out_seq=0 2024-04-01T01:42:11.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.800+0000 7efc817fa640 1 -- 172.21.15.70:0/4185038794 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d667b3e2e0 con 0x55d667bb5340 2024-04-01T01:42:11.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.800+0000 7efc817fa640 10 client.27904 handle_command_reply: tid=0 2024-04-01T01:42:11.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.800+0000 7efca2baf740 1 client.27904 shutdown 2024-04-01T01:42:11.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.800+0000 7efca2baf740 1 -- 172.21.15.70:0/4185038794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b3e570 msgr2=0x55d667b48d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:11.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.801+0000 7efca2baf740 1 --2- 172.21.15.70:0/4185038794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b3e570 0x55d667b48d80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efc9400ba30 tx=0x7efc94005230 comp rx=0 tx=0).stop 2024-04-01T01:42:11.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.801+0000 7efca2baf740 1 -- 172.21.15.70:0/4185038794 shutdown_connections 2024-04-01T01:42:11.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.801+0000 7efca2baf740 1 --2- 172.21.15.70:0/4185038794 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d667bb5340 0x55d667bb7760 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.801+0000 7efca2baf740 1 --2- 172.21.15.70:0/4185038794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d667b3e570 0x55d667b48d80 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.801+0000 7efca2baf740 1 --2- 172.21.15.70:0/4185038794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d667b39400 0x55d667b47e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.801+0000 7efca2baf740 1 --2- 172.21.15.70:0/4185038794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d667b38a10 0x55d667b478f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.801+0000 7efca2baf740 1 -- 172.21.15.70:0/4185038794 >> 172.21.15.70:0/4185038794 conn(0x55d667b33f80 msgr2=0x55d667b371e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:11.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.801+0000 7efc817fa640 0 client.27904 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:11.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.802+0000 7efca2baf740 1 -- 172.21.15.70:0/4185038794 shutdown_connections 2024-04-01T01:42:11.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.802+0000 7efca2baf740 1 -- 172.21.15.70:0/4185038794 wait complete. 2024-04-01T01:42:11.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.802+0000 7efca2baf740 20 client.27904 trim_cache size 0 max 16384 2024-04-01T01:42:11.804 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:42:11.804 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:42:11.804 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "702dc7e2-ba31-41f8-869d-ce8ad2217cd9", 2024-04-01T01:42:11.804 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:42:11.804 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:42:11.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.804+0000 7efc9feed640 1 -- 172.21.15.70:0/3321758897 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc840a25f0 msgr2=0x7efc840a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:11.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.804+0000 7efc9feed640 1 --2- 172.21.15.70:0/3321758897 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc840a25f0 0x7efc840a4ab0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7efc9400b590 tx=0x7efc9400b520 comp rx=0 tx=0).stop 2024-04-01T01:42:11.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.804+0000 7efc9feed640 1 -- 172.21.15.70:0/3321758897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc98108750 msgr2=0x7efc981a6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:11.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.804+0000 7efc9feed640 1 --2- 172.21.15.70:0/3321758897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc98108750 0x7efc981a6860 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7efc9000b9c0 tx=0x7efc9000be80 comp rx=0 tx=0).stop 2024-04-01T01:42:11.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.805+0000 7efc9feed640 1 -- 172.21.15.70:0/3321758897 shutdown_connections 2024-04-01T01:42:11.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.805+0000 7efc9feed640 1 --2- 172.21.15.70:0/3321758897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc9810ffd0 0x7efc981ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.805+0000 7efc9feed640 1 --2- 172.21.15.70:0/3321758897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc981090b0 0x7efc981a6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.805+0000 7efc9feed640 1 --2- 172.21.15.70:0/3321758897 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc840a25f0 0x7efc840a4ab0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.805+0000 7efc9feed640 1 --2- 172.21.15.70:0/3321758897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc98108750 0x7efc981a6860 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:11.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.805+0000 7efc9feed640 1 -- 172.21.15.70:0/3321758897 >> 172.21.15.70:0/3321758897 conn(0x7efc9806e950 msgr2=0x7efc98114370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:11.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.806+0000 7efc9feed640 1 -- 172.21.15.70:0/3321758897 shutdown_connections 2024-04-01T01:42:11.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:11.807+0000 7efc9feed640 1 -- 172.21.15.70:0/3321758897 wait complete. 2024-04-01T01:42:11.827 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:42:11.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:42:11.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 445.0 s, 816.7 tps, lat 122.321 ms stddev 59.195 2024-04-01T01:42:12.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.017+0000 7f190878a640 1 -- 172.21.15.70:0/2460257001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1900076110 msgr2=0x7f1900076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:12.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.017+0000 7f190878a640 1 --2- 172.21.15.70:0/2460257001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1900076110 0x7f1900076530 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f18f8009900 tx=0x7f18f802bc40 comp rx=0 tx=0).stop 2024-04-01T01:42:12.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.017+0000 7f190878a640 1 -- 172.21.15.70:0/2460257001 shutdown_connections 2024-04-01T01:42:12.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.017+0000 7f190878a640 1 --2- 172.21.15.70:0/2460257001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1900077cf0 0x7f1900078590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.017+0000 7f190878a640 1 --2- 172.21.15.70:0/2460257001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1900077350 0x7f19000777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.017+0000 7f190878a640 1 --2- 172.21.15.70:0/2460257001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1900076110 0x7f1900076530 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.017+0000 7f190878a640 1 -- 172.21.15.70:0/2460257001 >> 172.21.15.70:0/2460257001 conn(0x7f19000fef00 msgr2=0x7f1900101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:12.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.017+0000 7f190878a640 1 -- 172.21.15.70:0/2460257001 shutdown_connections 2024-04-01T01:42:12.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.017+0000 7f190878a640 1 -- 172.21.15.70:0/2460257001 wait complete. 2024-04-01T01:42:12.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.018+0000 7f190878a640 1 Processor -- start 2024-04-01T01:42:12.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.018+0000 7f190878a640 1 -- start start 2024-04-01T01:42:12.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.019+0000 7f190878a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1900076110 0x7f190019de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.019+0000 7f190878a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1900077350 0x7f190019e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.019+0000 7f190878a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1900077cf0 0x7f19001a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.019+0000 7f1906526640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1900076110 0x7f190019de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:12.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.019+0000 7f1906526640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1900076110 0x7f190019de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57034/0 (socket says 172.21.15.70:57034) 2024-04-01T01:42:12.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.019+0000 7f1906526640 1 -- 172.21.15.70:0/318722514 learned_addr learned my addr 172.21.15.70:0/318722514 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:12.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.019+0000 7f190878a640 1 -- 172.21.15.70:0/318722514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1900113910 con 0x7f1900077cf0 2024-04-01T01:42:12.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.019+0000 7f190878a640 1 -- 172.21.15.70:0/318722514 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1900113790 con 0x7f1900077350 2024-04-01T01:42:12.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.019+0000 7f190878a640 1 -- 172.21.15.70:0/318722514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1900113a90 con 0x7f1900076110 2024-04-01T01:42:12.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.019+0000 7f1905d25640 1 --2- 172.21.15.70:0/318722514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1900077350 0x7f190019e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:12.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.020+0000 7f1906d27640 1 --2- 172.21.15.70:0/318722514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1900077cf0 0x7f19001a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:12.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.020+0000 7f1906526640 1 -- 172.21.15.70:0/318722514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1900077350 msgr2=0x7f190019e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:12.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.020+0000 7f1906526640 1 --2- 172.21.15.70:0/318722514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1900077350 0x7f190019e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.020+0000 7f1906526640 1 -- 172.21.15.70:0/318722514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1900077cf0 msgr2=0x7f19001a53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:12.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.020+0000 7f1906526640 1 --2- 172.21.15.70:0/318722514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1900077cf0 0x7f19001a53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.020+0000 7f1906526640 1 -- 172.21.15.70:0/318722514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19001a5a20 con 0x7f1900076110 2024-04-01T01:42:12.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.020+0000 7f1906526640 1 --2- 172.21.15.70:0/318722514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1900076110 0x7f190019de00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f18f8009850 tx=0x7f18f80042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:12.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.021+0000 7f18f77fe640 1 -- 172.21.15.70:0/318722514 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f8043020 con 0x7f1900076110 2024-04-01T01:42:12.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.021+0000 7f18f77fe640 1 -- 172.21.15.70:0/318722514 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18f80054c0 con 0x7f1900076110 2024-04-01T01:42:12.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.021+0000 7f18f77fe640 1 -- 172.21.15.70:0/318722514 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f8003d40 con 0x7f1900076110 2024-04-01T01:42:12.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.021+0000 7f190878a640 1 -- 172.21.15.70:0/318722514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19001a5cb0 con 0x7f1900076110 2024-04-01T01:42:12.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.021+0000 7f190878a640 1 -- 172.21.15.70:0/318722514 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19001a60f0 con 0x7f1900076110 2024-04-01T01:42:12.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.024+0000 7f18f77fe640 1 -- 172.21.15.70:0/318722514 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f18f8046020 con 0x7f1900076110 2024-04-01T01:42:12.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.024+0000 7f190b36c740 1 Processor -- start 2024-04-01T01:42:12.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.025+0000 7f190b36c740 1 -- start start 2024-04-01T01:42:12.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.025+0000 7f190b36c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe17412b0 0x564fe17416d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.025+0000 7f190b36c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe1741ca0 0x564fe17467a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.025+0000 7f190b36c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1746e10 0x564fe17492a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.025+0000 7f190b36c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564fe16a6180 con 0x564fe1746e10 2024-04-01T01:42:12.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.025+0000 7f190b36c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x564fe164eb70 con 0x564fe1741ca0 2024-04-01T01:42:12.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.025+0000 7f190b36c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564fe16a2220 con 0x564fe17412b0 2024-04-01T01:42:12.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.025+0000 7f1906d27640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe17412b0 0x564fe17416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:12.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.025+0000 7f1906d27640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe17412b0 0x564fe17416d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57046/0 (socket says 172.21.15.70:57046) 2024-04-01T01:42:12.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.025+0000 7f1906d27640 1 -- 172.21.15.70:0/270920538 learned_addr learned my addr 172.21.15.70:0/270920538 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:12.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.025+0000 7f1906526640 1 --2- 172.21.15.70:0/270920538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe1741ca0 0x564fe17467a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:12.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.025+0000 7f1905d25640 1 --2- 172.21.15.70:0/270920538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1746e10 0x564fe17492a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:12.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.026+0000 7f1906d27640 1 -- 172.21.15.70:0/270920538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe1741ca0 msgr2=0x564fe17467a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:12.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.026+0000 7f1906d27640 1 --2- 172.21.15.70:0/270920538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe1741ca0 0x564fe17467a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.026+0000 7f1906d27640 1 -- 172.21.15.70:0/270920538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1746e10 msgr2=0x564fe17492a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:12.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.026+0000 7f1906d27640 1 --2- 172.21.15.70:0/270920538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1746e10 0x564fe17492a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.026+0000 7f1906d27640 1 -- 172.21.15.70:0/270920538 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fe16627c0 con 0x564fe17412b0 2024-04-01T01:42:12.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.026+0000 7f18f77fe640 1 --2- 172.21.15.70:0/318722514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f18ec0a25a0 0x7f18ec0a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.026+0000 7f1906d27640 1 --2- 172.21.15.70:0/270920538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe17412b0 0x564fe17416d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009fd0 tx=0x7f18fc0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=700d32cba4d08c81 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:12.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.026+0000 7f18f77fe640 1 -- 172.21.15.70:0/318722514 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f18f80f0f40 con 0x7f1900076110 2024-04-01T01:42:12.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.026+0000 7f1906526640 1 --2- 172.21.15.70:0/318722514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f18ec0a25a0 0x7f18ec0a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:12.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.027+0000 7f18f4ff9640 1 -- 172.21.15.70:0/270920538 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18fc004d20 con 0x564fe17412b0 2024-04-01T01:42:12.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.027+0000 7f18f4ff9640 1 -- 172.21.15.70:0/270920538 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18fc004050 con 0x564fe17412b0 2024-04-01T01:42:12.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.027+0000 7f18f4ff9640 1 -- 172.21.15.70:0/270920538 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18fc005280 con 0x564fe17412b0 2024-04-01T01:42:12.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.028+0000 7f1906526640 1 --2- 172.21.15.70:0/318722514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f18ec0a25a0 0x7f18ec0a4a60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f18f80ba0c0 tx=0x7f18f80f4900 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:12.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.028+0000 7f190b36c740 1 -- 172.21.15.70:0/270920538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe17412b0 msgr2=0x564fe17416d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:12.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.028+0000 7f190b36c740 1 --2- 172.21.15.70:0/270920538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe17412b0 0x564fe17416d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009fd0 tx=0x7f18fc0118d0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.028+0000 7f190b36c740 1 -- 172.21.15.70:0/270920538 shutdown_connections 2024-04-01T01:42:12.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.028+0000 7f190b36c740 1 --2- 172.21.15.70:0/270920538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1746e10 0x564fe17492a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.028+0000 7f190b36c740 1 --2- 172.21.15.70:0/270920538 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe1741ca0 0x564fe17467a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.028+0000 7f190b36c740 1 --2- 172.21.15.70:0/270920538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe17412b0 0x564fe17416d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.028+0000 7f190b36c740 1 -- 172.21.15.70:0/270920538 >> 172.21.15.70:0/270920538 conn(0x564fe173c820 msgr2=0x564fe173cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:12.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.028+0000 7f190b36c740 1 -- 172.21.15.70:0/270920538 shutdown_connections 2024-04-01T01:42:12.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.028+0000 7f190b36c740 1 -- 172.21.15.70:0/270920538 wait complete. 2024-04-01T01:42:12.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.029+0000 7f190b36c740 1 -- start start 2024-04-01T01:42:12.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.029+0000 7f190b36c740 1 Processor -- start 2024-04-01T01:42:12.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.029+0000 7f190b36c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe17412b0 0x564fe174b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.029+0000 7f190b36c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1741ca0 0x564fe174bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.030+0000 7f190b36c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe1746e10 0x564fe1752da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.030+0000 7f1906d27640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe17412b0 0x564fe174b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:12.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.030+0000 7f1906d27640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe17412b0 0x564fe174b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57048/0 (socket says 172.21.15.70:57048) 2024-04-01T01:42:12.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.030+0000 7f1906d27640 1 -- 172.21.15.70:0/3851669714 learned_addr learned my addr 172.21.15.70:0/3851669714 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:12.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.030+0000 7f190b36c740 1 -- 172.21.15.70:0/3851669714 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564fe1749dc0 con 0x564fe1741ca0 2024-04-01T01:42:12.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.030+0000 7f190b36c740 1 -- 172.21.15.70:0/3851669714 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x564fe16444a0 con 0x564fe1746e10 2024-04-01T01:42:12.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.030+0000 7f190b36c740 1 -- 172.21.15.70:0/3851669714 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564fe1749f40 con 0x564fe17412b0 2024-04-01T01:42:12.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.030+0000 7f1906d27640 1 --2- 172.21.15.70:0/3851669714 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe1746e10 0x564fe1752da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:12.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.030+0000 7f1906d27640 1 -- 172.21.15.70:0/3851669714 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe1746e10 msgr2=0x564fe1752da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:12.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.030+0000 7f1906d27640 1 --2- 172.21.15.70:0/3851669714 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe1746e10 0x564fe1752da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.030+0000 7f1905d25640 1 --2- 172.21.15.70:0/3851669714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1741ca0 0x564fe174bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:12.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.030+0000 7f1906d27640 1 -- 172.21.15.70:0/3851669714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1741ca0 msgr2=0x564fe174bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:12.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.030+0000 7f190b36c740 10 client.0 _ll_register_callbacks cb 0x564fe1647d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:12.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.030+0000 7f1906d27640 1 --2- 172.21.15.70:0/3851669714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1741ca0 0x564fe174bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.030+0000 7f1906d27640 1 -- 172.21.15.70:0/3851669714 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fe14847d0 con 0x564fe17412b0 2024-04-01T01:42:12.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.030+0000 7f1905d25640 1 --2- 172.21.15.70:0/3851669714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1741ca0 0x564fe174bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:42:12.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.031+0000 7f1906d27640 1 --2- 172.21.15.70:0/3851669714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe17412b0 0x564fe174b820 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f18fc004700 tx=0x7f18fc004760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=796fde77592f39da server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:12.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.031+0000 7f18e9ffb640 10 client.27916 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:42:12.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.031+0000 7f190b36c740 1 -- 172.21.15.70:0/3851669714 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18fc004050 con 0x564fe17412b0 2024-04-01T01:42:12.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.031+0000 7f18e9ffb640 1 -- 172.21.15.70:0/3851669714 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18fc00f6f0 con 0x564fe17412b0 2024-04-01T01:42:12.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.031+0000 7f18e9ffb640 1 -- 172.21.15.70:0/3851669714 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f18fc00f890 con 0x564fe17412b0 2024-04-01T01:42:12.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.031+0000 7f18e9ffb640 1 -- 172.21.15.70:0/3851669714 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f18fc034050 con 0x564fe17412b0 2024-04-01T01:42:12.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.031+0000 7f190b36c740 10 client.27916 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:12.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.031+0000 7f190b36c740 1 -- 172.21.15.70:0/3851669714 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=31}) -- 0x564fe13ba3f0 con 0x564fe17412b0 2024-04-01T01:42:12.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.032+0000 7f18e9ffb640 1 -- 172.21.15.70:0/3851669714 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f18fc00fa90 con 0x564fe17412b0 2024-04-01T01:42:12.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.032+0000 7f190b36c740 10 client.27916 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:12.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.032+0000 7f190b36c740 10 client.27916 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:12.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.032+0000 7f190b36c740 20 client.27916 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:12.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.032+0000 7f190b36c740 1 --2- 172.21.15.70:0/3851669714 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564fe1757230 0x564fe1759620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.032+0000 7f190b36c740 4 client.27916 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:12.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.032+0000 7f190b36c740 1 -- 172.21.15.70:0/3851669714 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f18fc004050 con 0x564fe1757230 2024-04-01T01:42:12.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.032+0000 7f1905d25640 1 --2- 172.21.15.70:0/3851669714 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564fe1757230 0x564fe1759620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:12.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.033+0000 7f1905d25640 1 --2- 172.21.15.70:0/3851669714 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564fe1757230 0x564fe1759620 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ea3689df9c578c0 server_cookie=759b7ee010e10da1 in_seq=0 out_seq=0 2024-04-01T01:42:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.033+0000 7f18e9ffb640 10 client.27916 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.035+0000 7f18e9ffb640 1 -- 172.21.15.70:0/3851669714 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f18fc004050 con 0x564fe1757230 2024-04-01T01:42:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.035+0000 7f18e9ffb640 10 client.27916 handle_command_reply: tid=0 2024-04-01T01:42:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.035+0000 7f190b36c740 1 client.27916 shutdown 2024-04-01T01:42:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.035+0000 7f190b36c740 1 -- 172.21.15.70:0/3851669714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe17412b0 msgr2=0x564fe174b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.035+0000 7f190b36c740 1 --2- 172.21.15.70:0/3851669714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe17412b0 0x564fe174b820 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f18fc004700 tx=0x7f18fc004760 comp rx=0 tx=0).stop 2024-04-01T01:42:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.035+0000 7f190b36c740 1 -- 172.21.15.70:0/3851669714 shutdown_connections 2024-04-01T01:42:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.035+0000 7f190b36c740 1 --2- 172.21.15.70:0/3851669714 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564fe1757230 0x564fe1759620 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.035+0000 7f190b36c740 1 --2- 172.21.15.70:0/3851669714 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe1746e10 0x564fe1752da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.035+0000 7f190b36c740 1 --2- 172.21.15.70:0/3851669714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1741ca0 0x564fe174bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.035+0000 7f190b36c740 1 --2- 172.21.15.70:0/3851669714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe17412b0 0x564fe174b820 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.035+0000 7f190b36c740 1 -- 172.21.15.70:0/3851669714 >> 172.21.15.70:0/3851669714 conn(0x564fe173c820 msgr2=0x564fe173bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.036+0000 7f18e9ffb640 0 client.27916 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:12.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.036+0000 7f190b36c740 1 -- 172.21.15.70:0/3851669714 shutdown_connections 2024-04-01T01:42:12.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.036+0000 7f190b36c740 1 -- 172.21.15.70:0/3851669714 wait complete. 2024-04-01T01:42:12.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.036+0000 7f190b36c740 20 client.27916 trim_cache size 0 max 16384 2024-04-01T01:42:12.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.069+0000 7f190b36c740 1 Processor -- start 2024-04-01T01:42:12.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.069+0000 7f190b36c740 1 -- start start 2024-04-01T01:42:12.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.069+0000 7f190b36c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe17412b0 0x564fe174b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.070+0000 7f190b36c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe1741ca0 0x564fe1752d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.070+0000 7f1906d27640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe17412b0 0x564fe174b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.070+0000 7f1906d27640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe17412b0 0x564fe174b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:32818/0 (socket says 172.21.15.70:32818) 2024-04-01T01:42:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.070+0000 7f1906d27640 1 -- 172.21.15.70:0/2605757088 learned_addr learned my addr 172.21.15.70:0/2605757088 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.070+0000 7f190b36c740 1 --2- 172.21.15.70:0/2605757088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1746e10 0x564fe174b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.070+0000 7f190b36c740 1 -- 172.21.15.70:0/2605757088 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f19001b0780 con 0x564fe1746e10 2024-04-01T01:42:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.070+0000 7f190b36c740 1 -- 172.21.15.70:0/2605757088 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f19001b22c0 con 0x564fe17412b0 2024-04-01T01:42:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.070+0000 7f1905d25640 1 --2- 172.21.15.70:0/2605757088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe1741ca0 0x564fe1752d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.070+0000 7f190b36c740 1 -- 172.21.15.70:0/2605757088 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f19001b0e50 con 0x564fe1741ca0 2024-04-01T01:42:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.071+0000 7f1906d27640 1 -- 172.21.15.70:0/2605757088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe1741ca0 msgr2=0x564fe1752d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.071+0000 7f1906d27640 1 --2- 172.21.15.70:0/2605757088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe1741ca0 0x564fe1752d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.071+0000 7f1906d27640 1 -- 172.21.15.70:0/2605757088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1746e10 msgr2=0x564fe174b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.071+0000 7f1906d27640 1 --2- 172.21.15.70:0/2605757088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1746e10 0x564fe174b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.071+0000 7f1906d27640 1 -- 172.21.15.70:0/2605757088 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fe137a0f0 con 0x564fe17412b0 2024-04-01T01:42:12.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.071+0000 7f1905d25640 1 --2- 172.21.15.70:0/2605757088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe1741ca0 0x564fe1752d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:42:12.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.071+0000 7f1906d27640 1 --2- 172.21.15.70:0/2605757088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe17412b0 0x564fe174b430 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f18fc002230 tx=0x7f18fc024000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c4cacc54a951b66 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:12.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.072+0000 7f18ebfff640 1 -- 172.21.15.70:0/2605757088 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18fc03d070 con 0x564fe17412b0 2024-04-01T01:42:12.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.072+0000 7f18ebfff640 1 -- 172.21.15.70:0/2605757088 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18fc00fba0 con 0x564fe17412b0 2024-04-01T01:42:12.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.072+0000 7f18ebfff640 1 -- 172.21.15.70:0/2605757088 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18fc019730 con 0x564fe17412b0 2024-04-01T01:42:12.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.074+0000 7f190b36c740 1 -- 172.21.15.70:0/2605757088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe17412b0 msgr2=0x564fe174b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:12.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.074+0000 7f190b36c740 1 --2- 172.21.15.70:0/2605757088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe17412b0 0x564fe174b430 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f18fc002230 tx=0x7f18fc024000 comp rx=0 tx=0).stop 2024-04-01T01:42:12.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.074+0000 7f190b36c740 1 -- 172.21.15.70:0/2605757088 shutdown_connections 2024-04-01T01:42:12.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.074+0000 7f190b36c740 1 --2- 172.21.15.70:0/2605757088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1746e10 0x564fe174b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.074+0000 7f190b36c740 1 --2- 172.21.15.70:0/2605757088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe1741ca0 0x564fe1752d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.074+0000 7f190b36c740 1 --2- 172.21.15.70:0/2605757088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe17412b0 0x564fe174b430 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.074+0000 7f190b36c740 1 -- 172.21.15.70:0/2605757088 >> 172.21.15.70:0/2605757088 conn(0x564fe173c820 msgr2=0x564fe1743230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:12.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.074+0000 7f190b36c740 1 -- 172.21.15.70:0/2605757088 shutdown_connections 2024-04-01T01:42:12.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.074+0000 7f190b36c740 1 -- 172.21.15.70:0/2605757088 wait complete. 2024-04-01T01:42:12.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.075+0000 7f190b36c740 1 -- start start 2024-04-01T01:42:12.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.075+0000 7f190b36c740 1 Processor -- start 2024-04-01T01:42:12.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.075+0000 7f190b36c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe17412b0 0x564fe174fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.075+0000 7f190b36c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe1741ca0 0x564fe1750260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.075+0000 7f190b36c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1746e10 0x564fe1751270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.075+0000 7f190b36c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f19001b1bf0 con 0x564fe1746e10 2024-04-01T01:42:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.075+0000 7f190b36c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f19001b1520 con 0x564fe17412b0 2024-04-01T01:42:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.075+0000 7f190b36c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f19001ae760 con 0x564fe1741ca0 2024-04-01T01:42:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.075+0000 7f190b36c740 10 client.0 _ll_register_callbacks cb 0x7f18f80f47d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.076+0000 7f1906d27640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe17412b0 0x564fe174fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.076+0000 7f1906d27640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe17412b0 0x564fe174fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:32826/0 (socket says 172.21.15.70:32826) 2024-04-01T01:42:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.076+0000 7f1906d27640 1 -- 172.21.15.70:0/1516449276 learned_addr learned my addr 172.21.15.70:0/1516449276 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.076+0000 7f1906d27640 1 --2- 172.21.15.70:0/1516449276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1746e10 0x564fe1751270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.076+0000 7f1906d27640 1 -- 172.21.15.70:0/1516449276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe1741ca0 msgr2=0x564fe1750260 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:42:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.076+0000 7f1906d27640 1 --2- 172.21.15.70:0/1516449276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe1741ca0 0x564fe1750260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.076+0000 7f1906d27640 1 -- 172.21.15.70:0/1516449276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1746e10 msgr2=0x564fe1751270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.076+0000 7f1906d27640 1 --2- 172.21.15.70:0/1516449276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1746e10 0x564fe1751270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.076+0000 7f1906d27640 1 -- 172.21.15.70:0/1516449276 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fe1751860 con 0x564fe17412b0 2024-04-01T01:42:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.076+0000 7f1906d27640 1 --2- 172.21.15.70:0/1516449276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe17412b0 0x564fe174fd20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f18fc00fdd0 tx=0x7f18fc005cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33b99ecc6feb44f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.076+0000 7f18e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.077+0000 7f18e9ffb640 1 -- 172.21.15.70:0/1516449276 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18fc03d040 con 0x564fe17412b0 2024-04-01T01:42:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.077+0000 7f18e9ffb640 1 -- 172.21.15.70:0/1516449276 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f18fc00f6f0 con 0x564fe17412b0 2024-04-01T01:42:12.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.077+0000 7f190b36c740 1 -- 172.21.15.70:0/1516449276 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18fc00fba0 con 0x564fe17412b0 2024-04-01T01:42:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.077+0000 7f18e9ffb640 1 -- 172.21.15.70:0/1516449276 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f18fc00f8f0 con 0x564fe17412b0 2024-04-01T01:42:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.077+0000 7f190b36c740 10 client.28097 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.077+0000 7f190b36c740 1 -- 172.21.15.70:0/1516449276 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564fe1752ae0 con 0x564fe17412b0 2024-04-01T01:42:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.078+0000 7f18e9ffb640 1 -- 172.21.15.70:0/1516449276 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f18fc00fba0 con 0x564fe17412b0 2024-04-01T01:42:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.078+0000 7f18e9ffb640 1 -- 172.21.15.70:0/1516449276 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f18fc043050 con 0x564fe17412b0 2024-04-01T01:42:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.078+0000 7f190b36c740 10 client.28097 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.078+0000 7f190b36c740 10 client.28097 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.078+0000 7f190b36c740 20 client.28097 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.078+0000 7f190b36c740 1 --2- 172.21.15.70:0/1516449276 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564fe17bd520 0x564fe17bf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.078+0000 7f190b36c740 4 client.28097 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:42:12.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.078+0000 7f190b36c740 1 -- 172.21.15.70:0/1516449276 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564fe16b36c0 con 0x564fe17bd520 2024-04-01T01:42:12.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.082+0000 7f1905d25640 1 --2- 172.21.15.70:0/1516449276 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564fe17bd520 0x564fe17bf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.083+0000 7f1905d25640 1 --2- 172.21.15.70:0/1516449276 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564fe17bd520 0x564fe17bf910 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f6da40790a39f55 server_cookie=52ad220d7e7a1bce in_seq=0 out_seq=0 2024-04-01T01:42:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.083+0000 7f18e9ffb640 10 client.28097 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.083+0000 7f18e9ffb640 1 -- 172.21.15.70:0/1516449276 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564fe16b36c0 con 0x564fe17bd520 2024-04-01T01:42:12.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.083+0000 7f18e9ffb640 10 client.28097 handle_command_reply: tid=0 2024-04-01T01:42:12.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.083+0000 7f190b36c740 1 client.28097 shutdown 2024-04-01T01:42:12.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.084+0000 7f190b36c740 1 -- 172.21.15.70:0/1516449276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe17412b0 msgr2=0x564fe174fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:12.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.084+0000 7f190b36c740 1 --2- 172.21.15.70:0/1516449276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe17412b0 0x564fe174fd20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f18fc00fdd0 tx=0x7f18fc005cf0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.084+0000 7f190b36c740 1 -- 172.21.15.70:0/1516449276 shutdown_connections 2024-04-01T01:42:12.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.084+0000 7f190b36c740 1 --2- 172.21.15.70:0/1516449276 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564fe17bd520 0x564fe17bf910 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.084+0000 7f190b36c740 1 --2- 172.21.15.70:0/1516449276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564fe1746e10 0x564fe1751270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.084+0000 7f18e9ffb640 0 client.28097 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:12.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.084+0000 7f190b36c740 1 --2- 172.21.15.70:0/1516449276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564fe1741ca0 0x564fe1750260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.084+0000 7f190b36c740 1 --2- 172.21.15.70:0/1516449276 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564fe17412b0 0x564fe174fd20 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.084+0000 7f190b36c740 1 -- 172.21.15.70:0/1516449276 >> 172.21.15.70:0/1516449276 conn(0x564fe173c820 msgr2=0x564fe174b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:12.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.084+0000 7f190b36c740 1 -- 172.21.15.70:0/1516449276 shutdown_connections 2024-04-01T01:42:12.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.084+0000 7f190b36c740 1 -- 172.21.15.70:0/1516449276 wait complete. 2024-04-01T01:42:12.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.085+0000 7f190b36c740 20 client.28097 trim_cache size 0 max 16384 2024-04-01T01:42:12.086 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:42:12.086 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:42:12.086 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:42:12.087 INFO:teuthology.orchestra.run.smithi070.stdout: "702dc7e2-ba31-41f8-869d-ce8ad2217cd9": { 2024-04-01T01:42:12.087 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:42:12.087 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "702dc7e2-ba31-41f8-869d-ce8ad2217cd9", 2024-04-01T01:42:12.087 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:42:12.087 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:42:12.087 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:42:12.087 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:42:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.089+0000 7f190878a640 1 -- 172.21.15.70:0/318722514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f18ec0a25a0 msgr2=0x7f18ec0a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.089+0000 7f190878a640 1 --2- 172.21.15.70:0/318722514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f18ec0a25a0 0x7f18ec0a4a60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f18f80ba0c0 tx=0x7f18f80f4900 comp rx=0 tx=0).stop 2024-04-01T01:42:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.089+0000 7f190878a640 1 -- 172.21.15.70:0/318722514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1900076110 msgr2=0x7f190019de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.089+0000 7f190878a640 1 --2- 172.21.15.70:0/318722514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1900076110 0x7f190019de00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f18f8009850 tx=0x7f18f80042f0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.090+0000 7f190878a640 1 -- 172.21.15.70:0/318722514 shutdown_connections 2024-04-01T01:42:12.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.090+0000 7f190878a640 1 --2- 172.21.15.70:0/318722514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1900077cf0 0x7f19001a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.090+0000 7f190878a640 1 --2- 172.21.15.70:0/318722514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1900077350 0x7f190019e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.090+0000 7f190878a640 1 --2- 172.21.15.70:0/318722514 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f18ec0a25a0 0x7f18ec0a4a60 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.090+0000 7f190878a640 1 --2- 172.21.15.70:0/318722514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1900076110 0x7f190019de00 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:12.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.091+0000 7f190878a640 1 -- 172.21.15.70:0/318722514 >> 172.21.15.70:0/318722514 conn(0x7f19000fef00 msgr2=0x7f1900100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:12.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.091+0000 7f190878a640 1 -- 172.21.15.70:0/318722514 shutdown_connections 2024-04-01T01:42:12.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:12.091+0000 7f190878a640 1 -- 172.21.15.70:0/318722514 wait complete. 2024-04-01T01:42:12.112 INFO:tasks.cephfs.filesystem:scrub status for tag:702dc7e2-ba31-41f8-869d-ce8ad2217cd9 - {'path': '/', 'tag': '702dc7e2-ba31-41f8-869d-ce8ad2217cd9', 'options': 'recursive,force'} 2024-04-01T01:42:12.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:12 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:42:12.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:12 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:42:12.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:12 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:42:12.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:12 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:42:12.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:12 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:42:12.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:12 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:42:12.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:12 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:42:12.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:12 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:42:12.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:12 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:42:12.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:12 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:42:12.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:12 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:42:12.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:12 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:42:13.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:13 smithi189 ceph-mon[35753]: pgmap v621: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 7.4 MiB/s wr, 844 op/s 2024-04-01T01:42:13.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:13 smithi070 ceph-mon[32179]: pgmap v621: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 7.4 MiB/s wr, 844 op/s 2024-04-01T01:42:13.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:13 smithi081 ceph-mon[35142]: pgmap v621: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 7.4 MiB/s wr, 844 op/s 2024-04-01T01:42:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.221+0000 7f8e60a4e640 1 -- 172.21.15.70:0/638282693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e5c109960 msgr2=0x7f8e5c1101f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:14.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.221+0000 7f8e60a4e640 1 --2- 172.21.15.70:0/638282693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e5c109960 0x7f8e5c1101f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8e54009ca0 tx=0x7f8e5402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.221+0000 7f8e60a4e640 1 -- 172.21.15.70:0/638282693 shutdown_connections 2024-04-01T01:42:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.221+0000 7f8e60a4e640 1 --2- 172.21.15.70:0/638282693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e5c109960 0x7f8e5c1101f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.221+0000 7f8e60a4e640 1 --2- 172.21.15.70:0/638282693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e5c108fa0 0x7f8e5c109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.221+0000 7f8e60a4e640 1 --2- 172.21.15.70:0/638282693 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e5c107db0 0x7f8e5c1081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.221+0000 7f8e60a4e640 1 -- 172.21.15.70:0/638282693 >> 172.21.15.70:0/638282693 conn(0x7f8e5c076d50 msgr2=0x7f8e5c079190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.221+0000 7f8e60a4e640 1 -- 172.21.15.70:0/638282693 shutdown_connections 2024-04-01T01:42:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.221+0000 7f8e60a4e640 1 -- 172.21.15.70:0/638282693 wait complete. 2024-04-01T01:42:14.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.222+0000 7f8e60a4e640 1 Processor -- start 2024-04-01T01:42:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.222+0000 7f8e60a4e640 1 -- start start 2024-04-01T01:42:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.222+0000 7f8e60a4e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e5c107db0 0x7f8e5c19df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.222+0000 7f8e60a4e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e5c108fa0 0x7f8e5c19e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.222+0000 7f8e60a4e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e5c109960 0x7f8e5c1a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.222+0000 7f8e60a4e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c1138c0 con 0x7f8e5c109960 2024-04-01T01:42:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.222+0000 7f8e60a4e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c113740 con 0x7f8e5c108fa0 2024-04-01T01:42:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.222+0000 7f8e60a4e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c113a40 con 0x7f8e5c107db0 2024-04-01T01:42:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.223+0000 7f8e5ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e5c109960 0x7f8e5c1a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.223+0000 7f8e5ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e5c109960 0x7f8e5c1a54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40368/0 (socket says 172.21.15.70:40368) 2024-04-01T01:42:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.223+0000 7f8e5ad9d640 1 -- 172.21.15.70:0/2087515710 learned_addr learned my addr 172.21.15.70:0/2087515710 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.223+0000 7f8e59d9b640 1 --2- 172.21.15.70:0/2087515710 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e5c108fa0 0x7f8e5c19e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.223+0000 7f8e5a59c640 1 --2- 172.21.15.70:0/2087515710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e5c107db0 0x7f8e5c19df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:14.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.223+0000 7f8e59d9b640 1 -- 172.21.15.70:0/2087515710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e5c107db0 msgr2=0x7f8e5c19df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.223+0000 7f8e59d9b640 1 --2- 172.21.15.70:0/2087515710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e5c107db0 0x7f8e5c19df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.223+0000 7f8e59d9b640 1 -- 172.21.15.70:0/2087515710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e5c109960 msgr2=0x7f8e5c1a54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.223+0000 7f8e59d9b640 1 --2- 172.21.15.70:0/2087515710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e5c109960 0x7f8e5c1a54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.223+0000 7f8e59d9b640 1 -- 172.21.15.70:0/2087515710 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e5c1a5b20 con 0x7f8e5c108fa0 2024-04-01T01:42:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.223+0000 7f8e5ad9d640 1 --2- 172.21.15.70:0/2087515710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e5c109960 0x7f8e5c1a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:42:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.224+0000 7f8e59d9b640 1 --2- 172.21.15.70:0/2087515710 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e5c108fa0 0x7f8e5c19e440 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000e9f0 tx=0x7f8e5000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.224+0000 7f8e4b7fe640 1 -- 172.21.15.70:0/2087515710 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5000ccc0 con 0x7f8e5c108fa0 2024-04-01T01:42:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.224+0000 7f8e4b7fe640 1 -- 172.21.15.70:0/2087515710 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e500108e0 con 0x7f8e5c108fa0 2024-04-01T01:42:14.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.224+0000 7f8e60a4e640 1 -- 172.21.15.70:0/2087515710 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e5c1a5e10 con 0x7f8e5c108fa0 2024-04-01T01:42:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.224+0000 7f8e4b7fe640 1 -- 172.21.15.70:0/2087515710 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e500187e0 con 0x7f8e5c108fa0 2024-04-01T01:42:14.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.224+0000 7f8e60a4e640 1 -- 172.21.15.70:0/2087515710 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e5c1a6260 con 0x7f8e5c108fa0 2024-04-01T01:42:14.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.230+0000 7f8e63632740 1 Processor -- start 2024-04-01T01:42:14.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.230+0000 7f8e4b7fe640 1 -- 172.21.15.70:0/2087515710 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f8e50010a90 con 0x7f8e5c108fa0 2024-04-01T01:42:14.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.231+0000 7f8e4b7fe640 1 --2- 172.21.15.70:0/2087515710 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e440a2690 0x7f8e440a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.231+0000 7f8e4b7fe640 1 -- 172.21.15.70:0/2087515710 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8e500cedf0 con 0x7f8e5c108fa0 2024-04-01T01:42:14.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.234+0000 7f8e5ad9d640 1 --2- 172.21.15.70:0/2087515710 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e440a2690 0x7f8e440a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.234+0000 7f8e63632740 1 -- start start 2024-04-01T01:42:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.234+0000 7f8e63632740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d7060fe8d0 0x55d7060fecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.234+0000 7f8e63632740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 0x55d706103dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.234+0000 7f8e63632740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d706104430 0x55d7061068c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.234+0000 7f8e63632740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d7060628f0 con 0x55d7060fe8d0 2024-04-01T01:42:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.234+0000 7f8e63632740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d70600b8d0 con 0x55d7060ff2c0 2024-04-01T01:42:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.234+0000 7f8e63632740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d70605e990 con 0x55d706104430 2024-04-01T01:42:14.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.235+0000 7f8e59d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 0x55d706103dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.235+0000 7f8e59d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 0x55d706103dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:32852/0 (socket says 172.21.15.70:32852) 2024-04-01T01:42:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.235+0000 7f8e59d9b640 1 -- 172.21.15.70:0/1524700166 learned_addr learned my addr 172.21.15.70:0/1524700166 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.235+0000 7f8e5a59c640 1 --2- 172.21.15.70:0/1524700166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d7060fe8d0 0x55d7060fecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.235+0000 7f8e5ad9d640 1 --2- 172.21.15.70:0/1524700166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d706104430 0x55d7061068c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.235+0000 7f8e5ad9d640 1 --2- 172.21.15.70:0/2087515710 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e440a2690 0x7f8e440a4b50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8e54032380 tx=0x7f8e54005ef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.235+0000 7f8e59d9b640 1 -- 172.21.15.70:0/1524700166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d706104430 msgr2=0x55d7061068c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.235+0000 7f8e59d9b640 1 --2- 172.21.15.70:0/1524700166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d706104430 0x55d7061068c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.235+0000 7f8e59d9b640 1 -- 172.21.15.70:0/1524700166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d7060fe8d0 msgr2=0x55d7060fecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.235+0000 7f8e59d9b640 1 --2- 172.21.15.70:0/1524700166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d7060fe8d0 0x55d7060fecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.235+0000 7f8e59d9b640 1 -- 172.21.15.70:0/1524700166 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d705d77210 con 0x55d7060ff2c0 2024-04-01T01:42:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.235+0000 7f8e59d9b640 1 --2- 172.21.15.70:0/1524700166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 0x55d706103dc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8e50097210 tx=0x7f8e50010f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d2376eab21b10e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:14.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.236+0000 7f8e48ff9640 1 -- 172.21.15.70:0/1524700166 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e500b9e60 con 0x55d7060ff2c0 2024-04-01T01:42:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.236+0000 7f8e48ff9640 1 -- 172.21.15.70:0/1524700166 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e500cc5d0 con 0x55d7060ff2c0 2024-04-01T01:42:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.236+0000 7f8e48ff9640 1 -- 172.21.15.70:0/1524700166 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e500bb320 con 0x55d7060ff2c0 2024-04-01T01:42:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.237+0000 7f8e63632740 1 -- 172.21.15.70:0/1524700166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 msgr2=0x55d706103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.237+0000 7f8e63632740 1 --2- 172.21.15.70:0/1524700166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 0x55d706103dc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8e50097210 tx=0x7f8e50010f20 comp rx=0 tx=0).stop 2024-04-01T01:42:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.238+0000 7f8e63632740 1 -- 172.21.15.70:0/1524700166 shutdown_connections 2024-04-01T01:42:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.238+0000 7f8e63632740 1 --2- 172.21.15.70:0/1524700166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d706104430 0x55d7061068c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.238+0000 7f8e63632740 1 --2- 172.21.15.70:0/1524700166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 0x55d706103dc0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.238+0000 7f8e63632740 1 --2- 172.21.15.70:0/1524700166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d7060fe8d0 0x55d7060fecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.238+0000 7f8e63632740 1 -- 172.21.15.70:0/1524700166 >> 172.21.15.70:0/1524700166 conn(0x55d7060f9e40 msgr2=0x55d7060fa240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.238+0000 7f8e63632740 1 -- 172.21.15.70:0/1524700166 shutdown_connections 2024-04-01T01:42:14.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.238+0000 7f8e63632740 1 -- 172.21.15.70:0/1524700166 wait complete. 2024-04-01T01:42:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.239+0000 7f8e63632740 1 -- start start 2024-04-01T01:42:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.239+0000 7f8e63632740 1 Processor -- start 2024-04-01T01:42:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.239+0000 7f8e63632740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7060fe8d0 0x55d706108c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.239+0000 7f8e63632740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 0x55d7061091c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.239+0000 7f8e63632740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d706104430 0x55d7061101c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.240+0000 7f8e63632740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d706107320 con 0x55d706104430 2024-04-01T01:42:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.240+0000 7f8e59d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 0x55d7061091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.240+0000 7f8e59d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 0x55d7061091c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:32860/0 (socket says 172.21.15.70:32860) 2024-04-01T01:42:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.240+0000 7f8e59d9b640 1 -- 172.21.15.70:0/1819285469 learned_addr learned my addr 172.21.15.70:0/1819285469 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.240+0000 7f8e5a59c640 1 --2- 172.21.15.70:0/1819285469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7060fe8d0 0x55d706108c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.240+0000 7f8e63632740 1 -- 172.21.15.70:0/1819285469 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d706000f80 con 0x55d7060ff2c0 2024-04-01T01:42:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.240+0000 7f8e5ad9d640 1 --2- 172.21.15.70:0/1819285469 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d706104430 0x55d7061101c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.240+0000 7f8e63632740 1 -- 172.21.15.70:0/1819285469 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d7061074a0 con 0x55d7060fe8d0 2024-04-01T01:42:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.240+0000 7f8e63632740 10 client.0 _ll_register_callbacks cb 0x55d706059290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:14.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.240+0000 7f8e59d9b640 1 -- 172.21.15.70:0/1819285469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7060fe8d0 msgr2=0x55d706108c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.240+0000 7f8e59d9b640 1 --2- 172.21.15.70:0/1819285469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7060fe8d0 0x55d706108c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.240+0000 7f8e59d9b640 1 -- 172.21.15.70:0/1819285469 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d706104430 msgr2=0x55d7061101c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.240+0000 7f8e59d9b640 1 --2- 172.21.15.70:0/1819285469 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d706104430 0x55d7061101c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.240+0000 7f8e59d9b640 1 -- 172.21.15.70:0/1819285469 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d705e43620 con 0x55d7060ff2c0 2024-04-01T01:42:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.241+0000 7f8e5a59c640 1 --2- 172.21.15.70:0/1819285469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7060fe8d0 0x55d706108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:42:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.241+0000 7f8e59d9b640 1 --2- 172.21.15.70:0/1819285469 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 0x55d7061091c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f8e500997e0 tx=0x7f8e500a4010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc609fc985b5aee0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.241+0000 7f8e41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.241+0000 7f8e41ffb640 1 -- 172.21.15.70:0/1819285469 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e500b3070 con 0x55d7060ff2c0 2024-04-01T01:42:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.241+0000 7f8e41ffb640 1 -- 172.21.15.70:0/1819285469 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8e5009ab90 con 0x55d7060ff2c0 2024-04-01T01:42:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.241+0000 7f8e63632740 1 -- 172.21.15.70:0/1819285469 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e500cc5d0 con 0x55d7060ff2c0 2024-04-01T01:42:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.241+0000 7f8e41ffb640 1 -- 172.21.15.70:0/1819285469 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8e500b9e60 con 0x55d7060ff2c0 2024-04-01T01:42:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.241+0000 7f8e63632740 10 client.28109 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:14.244 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.241+0000 7f8e63632740 1 -- 172.21.15.70:0/1819285469 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d706113160 con 0x55d7060ff2c0 2024-04-01T01:42:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.242+0000 7f8e41ffb640 1 -- 172.21.15.70:0/1819285469 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e500cc5d0 con 0x55d7060ff2c0 2024-04-01T01:42:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.242+0000 7f8e41ffb640 1 -- 172.21.15.70:0/1819285469 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8e50084ac0 con 0x55d7060ff2c0 2024-04-01T01:42:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.242+0000 7f8e63632740 10 client.28109 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.242+0000 7f8e63632740 10 client.28109 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.242+0000 7f8e63632740 20 client.28109 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.242+0000 7f8e63632740 1 --2- 172.21.15.70:0/1819285469 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d7061144b0 0x55d7061168a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.242+0000 7f8e63632740 4 client.28109 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.242+0000 7f8e63632740 1 -- 172.21.15.70:0/1819285469 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d706076120 con 0x55d7061144b0 2024-04-01T01:42:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.242+0000 7f8e5a59c640 1 --2- 172.21.15.70:0/1819285469 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d7061144b0 0x55d7061168a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.243+0000 7f8e5a59c640 1 --2- 172.21.15.70:0/1819285469 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d7061144b0 0x55d7061168a0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f11fdcb5812a766f server_cookie=f3145da9afd51e5 in_seq=0 out_seq=0 2024-04-01T01:42:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.243+0000 7f8e41ffb640 10 client.28109 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.244+0000 7f8e41ffb640 1 -- 172.21.15.70:0/1819285469 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d706076120 con 0x55d7061144b0 2024-04-01T01:42:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.244+0000 7f8e41ffb640 10 client.28109 handle_command_reply: tid=0 2024-04-01T01:42:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.244+0000 7f8e63632740 1 client.28109 shutdown 2024-04-01T01:42:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.246+0000 7f8e63632740 1 -- 172.21.15.70:0/1819285469 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 msgr2=0x55d7061091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.246+0000 7f8e63632740 1 --2- 172.21.15.70:0/1819285469 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 0x55d7061091c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f8e500997e0 tx=0x7f8e500a4010 comp rx=0 tx=0).stop 2024-04-01T01:42:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.246+0000 7f8e63632740 1 -- 172.21.15.70:0/1819285469 shutdown_connections 2024-04-01T01:42:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.246+0000 7f8e63632740 1 --2- 172.21.15.70:0/1819285469 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d7061144b0 0x55d7061168a0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.246+0000 7f8e63632740 1 --2- 172.21.15.70:0/1819285469 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d706104430 0x55d7061101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.246+0000 7f8e63632740 1 --2- 172.21.15.70:0/1819285469 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 0x55d7061091c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.246+0000 7f8e63632740 1 --2- 172.21.15.70:0/1819285469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7060fe8d0 0x55d706108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.246+0000 7f8e63632740 1 -- 172.21.15.70:0/1819285469 >> 172.21.15.70:0/1819285469 conn(0x55d7060f9e40 msgr2=0x55d7060f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.246+0000 7f8e41ffb640 0 client.28109 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.246+0000 7f8e63632740 1 -- 172.21.15.70:0/1819285469 shutdown_connections 2024-04-01T01:42:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.246+0000 7f8e63632740 1 -- 172.21.15.70:0/1819285469 wait complete. 2024-04-01T01:42:14.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.247+0000 7f8e63632740 20 client.28109 trim_cache size 0 max 16384 2024-04-01T01:42:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.279+0000 7f8e63632740 1 Processor -- start 2024-04-01T01:42:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.279+0000 7f8e63632740 1 -- start start 2024-04-01T01:42:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.279+0000 7f8e63632740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d7060fe8d0 0x55d7061164c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.279+0000 7f8e63632740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7060ff2c0 0x55d706116a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.279+0000 7f8e63632740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d706104430 0x55d706110180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.279+0000 7f8e63632740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c1b0980 con 0x55d7060fe8d0 2024-04-01T01:42:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.279+0000 7f8e63632740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c1b24c0 con 0x55d706104430 2024-04-01T01:42:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.279+0000 7f8e63632740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c1b1050 con 0x55d7060ff2c0 2024-04-01T01:42:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.279+0000 7f8e5a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d7060fe8d0 0x55d7061164c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.279+0000 7f8e5a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d7060fe8d0 0x55d7061164c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40398/0 (socket says 172.21.15.70:40398) 2024-04-01T01:42:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.279+0000 7f8e5a59c640 1 -- 172.21.15.70:0/981993791 learned_addr learned my addr 172.21.15.70:0/981993791 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.279+0000 7f8e5ad9d640 1 --2- 172.21.15.70:0/981993791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d706104430 0x55d706110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:14.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.280+0000 7f8e59d9b640 1 --2- 172.21.15.70:0/981993791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7060ff2c0 0x55d706116a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.280+0000 7f8e5ad9d640 1 -- 172.21.15.70:0/981993791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7060ff2c0 msgr2=0x55d706116a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.280+0000 7f8e5ad9d640 1 --2- 172.21.15.70:0/981993791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7060ff2c0 0x55d706116a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.280+0000 7f8e5ad9d640 1 -- 172.21.15.70:0/981993791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d7060fe8d0 msgr2=0x55d7061164c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:14.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.280+0000 7f8e5ad9d640 1 --2- 172.21.15.70:0/981993791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d7060fe8d0 0x55d7061164c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.280+0000 7f8e5ad9d640 1 -- 172.21.15.70:0/981993791 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d706097240 con 0x55d706104430 2024-04-01T01:42:14.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.280+0000 7f8e5a59c640 1 --2- 172.21.15.70:0/981993791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d7060fe8d0 0x55d7061164c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:42:14.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.280+0000 7f8e5ad9d640 1 --2- 172.21.15.70:0/981993791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d706104430 0x55d706110180 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8e54033000 tx=0x7f8e54043000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c15b1e8f84fd9ace server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:14.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.280+0000 7f8e43fff640 1 -- 172.21.15.70:0/981993791 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e54044070 con 0x55d706104430 2024-04-01T01:42:14.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.281+0000 7f8e43fff640 1 -- 172.21.15.70:0/981993791 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e54004630 con 0x55d706104430 2024-04-01T01:42:14.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.281+0000 7f8e43fff640 1 -- 172.21.15.70:0/981993791 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e54039040 con 0x55d706104430 2024-04-01T01:42:14.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.282+0000 7f8e63632740 1 -- 172.21.15.70:0/981993791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d706104430 msgr2=0x55d706110180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:14.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.282+0000 7f8e63632740 1 --2- 172.21.15.70:0/981993791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d706104430 0x55d706110180 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8e54033000 tx=0x7f8e54043000 comp rx=0 tx=0).stop 2024-04-01T01:42:14.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.283+0000 7f8e63632740 1 -- 172.21.15.70:0/981993791 shutdown_connections 2024-04-01T01:42:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.283+0000 7f8e63632740 1 --2- 172.21.15.70:0/981993791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d706104430 0x55d706110180 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.283+0000 7f8e63632740 1 --2- 172.21.15.70:0/981993791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7060ff2c0 0x55d706116a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.283+0000 7f8e63632740 1 --2- 172.21.15.70:0/981993791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d7060fe8d0 0x55d7061164c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.283+0000 7f8e63632740 1 -- 172.21.15.70:0/981993791 >> 172.21.15.70:0/981993791 conn(0x55d7060f9e40 msgr2=0x55d7060fa220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.286+0000 7f8e63632740 1 -- 172.21.15.70:0/981993791 shutdown_connections 2024-04-01T01:42:14.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.286+0000 7f8e63632740 1 -- 172.21.15.70:0/981993791 wait complete. 2024-04-01T01:42:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.286+0000 7f8e63632740 1 -- start start 2024-04-01T01:42:14.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.287+0000 7f8e63632740 1 Processor -- start 2024-04-01T01:42:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.287+0000 7f8e63632740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d7060fe8d0 0x55d70610d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.287+0000 7f8e63632740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 0x55d70610d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.287+0000 7f8e63632740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d706104430 0x55d70610e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.287+0000 7f8e63632740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c1b1df0 con 0x55d7060fe8d0 2024-04-01T01:42:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.287+0000 7f8e63632740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c1b1720 con 0x55d7060ff2c0 2024-04-01T01:42:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.287+0000 7f8e63632740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c1ae960 con 0x55d706104430 2024-04-01T01:42:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.288+0000 7f8e63632740 10 client.0 _ll_register_callbacks cb 0x7f8e4c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.289+0000 7f8e5ad9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d706104430 0x55d70610e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.289+0000 7f8e5ad9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d706104430 0x55d70610e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57106/0 (socket says 172.21.15.70:57106) 2024-04-01T01:42:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.289+0000 7f8e5ad9d640 1 -- 172.21.15.70:0/982693730 learned_addr learned my addr 172.21.15.70:0/982693730 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.289+0000 7f8e5ad9d640 1 -- 172.21.15.70:0/982693730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 msgr2=0x55d70610d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.289+0000 7f8e5ad9d640 1 --2- 172.21.15.70:0/982693730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 0x55d70610d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.289+0000 7f8e5ad9d640 1 -- 172.21.15.70:0/982693730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d7060fe8d0 msgr2=0x55d70610d240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:42:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.289+0000 7f8e5ad9d640 1 --2- 172.21.15.70:0/982693730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d7060fe8d0 0x55d70610d240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.289+0000 7f8e5ad9d640 1 -- 172.21.15.70:0/982693730 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d70610ed80 con 0x55d706104430 2024-04-01T01:42:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.290+0000 7f8e5ad9d640 1 --2- 172.21.15.70:0/982693730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d706104430 0x55d70610e790 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8e5402c680 tx=0x7f8e5403f5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70ba276fd802447 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.290+0000 7f8e41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:42:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.291+0000 7f8e41ffb640 1 -- 172.21.15.70:0/982693730 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e54058030 con 0x55d706104430 2024-04-01T01:42:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.291+0000 7f8e41ffb640 1 -- 172.21.15.70:0/982693730 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8e54003f80 con 0x55d706104430 2024-04-01T01:42:14.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.291+0000 7f8e63632740 1 -- 172.21.15.70:0/982693730 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e54004630 con 0x55d706104430 2024-04-01T01:42:14.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.293+0000 7f8e41ffb640 1 -- 172.21.15.70:0/982693730 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8e54065050 con 0x55d706104430 2024-04-01T01:42:14.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.294+0000 7f8e63632740 10 client.27934 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:14.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.294+0000 7f8e63632740 1 -- 172.21.15.70:0/982693730 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d706110090 con 0x55d706104430 2024-04-01T01:42:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.295+0000 7f8e41ffb640 1 -- 172.21.15.70:0/982693730 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e5403fd80 con 0x55d706104430 2024-04-01T01:42:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.295+0000 7f8e41ffb640 1 -- 172.21.15.70:0/982693730 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8e540349a0 con 0x55d706104430 2024-04-01T01:42:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.295+0000 7f8e63632740 10 client.27934 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.295+0000 7f8e63632740 10 client.27934 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.295+0000 7f8e63632740 20 client.27934 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.295+0000 7f8e63632740 1 --2- 172.21.15.70:0/982693730 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d70617a950 0x55d70617cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.295+0000 7f8e63632740 4 client.27934 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:42:14.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.295+0000 7f8e63632740 1 -- 172.21.15.70:0/982693730 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f8e54004630 con 0x55d70617a950 2024-04-01T01:42:14.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.297+0000 7f8e5a59c640 1 --2- 172.21.15.70:0/982693730 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d70617a950 0x55d70617cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:14.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.297+0000 7f8e5a59c640 1 --2- 172.21.15.70:0/982693730 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d70617a950 0x55d70617cd70 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=980991fdd87b94eb server_cookie=56bb1bfa5168a579 in_seq=0 out_seq=0 2024-04-01T01:42:14.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.297+0000 7f8e41ffb640 10 client.27934 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.333+0000 7f8e41ffb640 1 -- 172.21.15.70:0/982693730 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f8e54004630 con 0x55d70617a950 2024-04-01T01:42:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.334+0000 7f8e41ffb640 10 client.27934 handle_command_reply: tid=0 2024-04-01T01:42:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.334+0000 7f8e63632740 1 client.27934 shutdown 2024-04-01T01:42:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.334+0000 7f8e63632740 1 -- 172.21.15.70:0/982693730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d706104430 msgr2=0x55d70610e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.334+0000 7f8e63632740 1 --2- 172.21.15.70:0/982693730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d706104430 0x55d70610e790 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8e5402c680 tx=0x7f8e5403f5d0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.334+0000 7f8e63632740 1 -- 172.21.15.70:0/982693730 shutdown_connections 2024-04-01T01:42:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.334+0000 7f8e63632740 1 --2- 172.21.15.70:0/982693730 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d70617a950 0x55d70617cd70 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.334+0000 7f8e63632740 1 --2- 172.21.15.70:0/982693730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d706104430 0x55d70610e790 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.334+0000 7f8e41ffb640 0 client.27934 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.334+0000 7f8e63632740 1 --2- 172.21.15.70:0/982693730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d7060ff2c0 0x55d70610d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.334+0000 7f8e63632740 1 --2- 172.21.15.70:0/982693730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d7060fe8d0 0x55d70610d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.334+0000 7f8e63632740 1 -- 172.21.15.70:0/982693730 >> 172.21.15.70:0/982693730 conn(0x55d7060f9e40 msgr2=0x55d706100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.335+0000 7f8e63632740 1 -- 172.21.15.70:0/982693730 shutdown_connections 2024-04-01T01:42:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.335+0000 7f8e63632740 1 -- 172.21.15.70:0/982693730 wait complete. 2024-04-01T01:42:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.335+0000 7f8e63632740 20 client.27934 trim_cache size 0 max 16384 2024-04-01T01:42:14.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:42:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:42:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:42:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:42:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.342+0000 7f8e60a4e640 1 -- 172.21.15.70:0/2087515710 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e440a2690 msgr2=0x7f8e440a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.342+0000 7f8e60a4e640 1 --2- 172.21.15.70:0/2087515710 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e440a2690 0x7f8e440a4b50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8e54032380 tx=0x7f8e54005ef0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.342+0000 7f8e60a4e640 1 -- 172.21.15.70:0/2087515710 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e5c108fa0 msgr2=0x7f8e5c19e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.342+0000 7f8e60a4e640 1 --2- 172.21.15.70:0/2087515710 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e5c108fa0 0x7f8e5c19e440 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000e9f0 tx=0x7f8e5000eeb0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.343+0000 7f8e60a4e640 1 -- 172.21.15.70:0/2087515710 shutdown_connections 2024-04-01T01:42:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.343+0000 7f8e60a4e640 1 --2- 172.21.15.70:0/2087515710 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e5c109960 0x7f8e5c1a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.343+0000 7f8e60a4e640 1 --2- 172.21.15.70:0/2087515710 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e5c108fa0 0x7f8e5c19e440 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.343+0000 7f8e60a4e640 1 --2- 172.21.15.70:0/2087515710 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e440a2690 0x7f8e440a4b50 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.343+0000 7f8e60a4e640 1 --2- 172.21.15.70:0/2087515710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e5c107db0 0x7f8e5c19df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.343+0000 7f8e60a4e640 1 -- 172.21.15.70:0/2087515710 >> 172.21.15.70:0/2087515710 conn(0x7f8e5c076d50 msgr2=0x7f8e5c10c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.343+0000 7f8e60a4e640 1 -- 172.21.15.70:0/2087515710 shutdown_connections 2024-04-01T01:42:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:14.343+0000 7f8e60a4e640 1 -- 172.21.15.70:0/2087515710 wait complete. 2024-04-01T01:42:15.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:14 smithi189 ceph-mon[35753]: pgmap v622: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 6.4 MiB/s wr, 761 op/s 2024-04-01T01:42:15.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:14 smithi070 ceph-mon[32179]: pgmap v622: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 6.4 MiB/s wr, 761 op/s 2024-04-01T01:42:15.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:14 smithi081 ceph-mon[35142]: pgmap v622: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 6.4 MiB/s wr, 761 op/s 2024-04-01T01:42:16.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:16 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:42:16.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:16 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:42:16.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:16 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:42:16.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 450.0 s, 857.1 tps, lat 115.308 ms stddev 112.515 2024-04-01T01:42:17.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:17 smithi189 ceph-mon[35753]: pgmap v623: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 7.8 MiB/s wr, 953 op/s 2024-04-01T01:42:17.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:17 smithi070 ceph-mon[32179]: pgmap v623: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 7.8 MiB/s wr, 953 op/s 2024-04-01T01:42:17.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:17 smithi081 ceph-mon[35142]: pgmap v623: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 7.8 MiB/s wr, 953 op/s 2024-04-01T01:42:19.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:18 smithi189 ceph-mon[35753]: pgmap v624: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 7.3 MiB/s wr, 898 op/s 2024-04-01T01:42:19.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:18 smithi070 ceph-mon[32179]: pgmap v624: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 7.3 MiB/s wr, 898 op/s 2024-04-01T01:42:19.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:18 smithi081 ceph-mon[35142]: pgmap v624: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 7.3 MiB/s wr, 898 op/s 2024-04-01T01:42:21.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:20 smithi189 ceph-mon[35753]: pgmap v625: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 7.0 MiB/s wr, 858 op/s 2024-04-01T01:42:21.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:20 smithi070 ceph-mon[32179]: pgmap v625: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 7.0 MiB/s wr, 858 op/s 2024-04-01T01:42:21.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:20 smithi081 ceph-mon[35142]: pgmap v625: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 7.0 MiB/s wr, 858 op/s 2024-04-01T01:42:21.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 455.0 s, 872.4 tps, lat 115.137 ms stddev 120.751 2024-04-01T01:42:23.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:22 smithi189 ceph-mon[35753]: pgmap v626: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 7.7 MiB/s wr, 942 op/s 2024-04-01T01:42:23.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:22 smithi070 ceph-mon[32179]: pgmap v626: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 7.7 MiB/s wr, 942 op/s 2024-04-01T01:42:23.330 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:22 smithi081 ceph-mon[35142]: pgmap v626: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 7.7 MiB/s wr, 942 op/s 2024-04-01T01:42:25.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:24 smithi189 ceph-mon[35753]: pgmap v627: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 6.9 MiB/s wr, 826 op/s 2024-04-01T01:42:25.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:24 smithi070 ceph-mon[32179]: pgmap v627: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 6.9 MiB/s wr, 826 op/s 2024-04-01T01:42:25.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:24 smithi081 ceph-mon[35142]: pgmap v627: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 6.9 MiB/s wr, 826 op/s 2024-04-01T01:42:26.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 460.0 s, 774.0 tps, lat 129.115 ms stddev 131.754 2024-04-01T01:42:27.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:26 smithi189 ceph-mon[35753]: pgmap v628: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 7.9 MiB/s wr, 922 op/s 2024-04-01T01:42:27.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:26 smithi070 ceph-mon[32179]: pgmap v628: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 7.9 MiB/s wr, 922 op/s 2024-04-01T01:42:27.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:26 smithi081 ceph-mon[35142]: pgmap v628: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 7.9 MiB/s wr, 922 op/s 2024-04-01T01:42:27.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.479+0000 7efc4b7a7640 1 -- 172.21.15.70:0/1641191217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc44102ac0 msgr2=0x7efc44102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:27.479 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.479+0000 7efc4b7a7640 1 --2- 172.21.15.70:0/1641191217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc44102ac0 0x7efc44102f40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7efc34009860 tx=0x7efc3402bb70 comp rx=0 tx=0).stop 2024-04-01T01:42:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.480+0000 7efc4b7a7640 1 -- 172.21.15.70:0/1641191217 shutdown_connections 2024-04-01T01:42:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.480+0000 7efc4b7a7640 1 --2- 172.21.15.70:0/1641191217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc44103480 0x7efc44109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.480+0000 7efc4b7a7640 1 --2- 172.21.15.70:0/1641191217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc44102ac0 0x7efc44102f40 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.480+0000 7efc4b7a7640 1 --2- 172.21.15.70:0/1641191217 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc440ffa50 0x7efc440ffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.480 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.480+0000 7efc4b7a7640 1 -- 172.21.15.70:0/1641191217 >> 172.21.15.70:0/1641191217 conn(0x7efc440fb3a0 msgr2=0x7efc440fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:27.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.480+0000 7efc4b7a7640 1 -- 172.21.15.70:0/1641191217 shutdown_connections 2024-04-01T01:42:27.481 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.480+0000 7efc4b7a7640 1 -- 172.21.15.70:0/1641191217 wait complete. 2024-04-01T01:42:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.481+0000 7efc4b7a7640 1 Processor -- start 2024-04-01T01:42:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.481+0000 7efc4b7a7640 1 -- start start 2024-04-01T01:42:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.481+0000 7efc4b7a7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc440ffa50 0x7efc4419dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.481+0000 7efc4b7a7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc44102ac0 0x7efc4419e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.481+0000 7efc4b7a7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc44103480 0x7efc441a52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.481+0000 7efc4b7a7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efc4410f480 con 0x7efc44103480 2024-04-01T01:42:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.481+0000 7efc4b7a7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efc4410f300 con 0x7efc44102ac0 2024-04-01T01:42:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.481+0000 7efc48d42640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc44102ac0 0x7efc4419e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.481+0000 7efc48d42640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc44102ac0 0x7efc4419e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47652/0 (socket says 172.21.15.70:47652) 2024-04-01T01:42:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.481+0000 7efc48d42640 1 -- 172.21.15.70:0/3279029477 learned_addr learned my addr 172.21.15.70:0/3279029477 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.482+0000 7efc49543640 1 --2- 172.21.15.70:0/3279029477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc440ffa50 0x7efc4419dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:27.483 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.482+0000 7efc4b7a7640 1 -- 172.21.15.70:0/3279029477 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efc4410f600 con 0x7efc440ffa50 2024-04-01T01:42:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.482+0000 7efc49d44640 1 --2- 172.21.15.70:0/3279029477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc44103480 0x7efc441a52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.482+0000 7efc48d42640 1 -- 172.21.15.70:0/3279029477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc440ffa50 msgr2=0x7efc4419dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.482+0000 7efc48d42640 1 --2- 172.21.15.70:0/3279029477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc440ffa50 0x7efc4419dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.482+0000 7efc48d42640 1 -- 172.21.15.70:0/3279029477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc44103480 msgr2=0x7efc441a52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.482+0000 7efc48d42640 1 --2- 172.21.15.70:0/3279029477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc44103480 0x7efc441a52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.482+0000 7efc48d42640 1 -- 172.21.15.70:0/3279029477 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc441a5950 con 0x7efc44102ac0 2024-04-01T01:42:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.482+0000 7efc49d44640 1 --2- 172.21.15.70:0/3279029477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc44103480 0x7efc441a52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:42:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.482+0000 7efc48d42640 1 --2- 172.21.15.70:0/3279029477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc44102ac0 0x7efc4419e280 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7efc3402c180 tx=0x7efc340042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.482+0000 7efc3a7fc640 1 -- 172.21.15.70:0/3279029477 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc34041020 con 0x7efc44102ac0 2024-04-01T01:42:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.483+0000 7efc3a7fc640 1 -- 172.21.15.70:0/3279029477 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc340035f0 con 0x7efc44102ac0 2024-04-01T01:42:27.484 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.483+0000 7efc4b7a7640 1 -- 172.21.15.70:0/3279029477 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc441a5be0 con 0x7efc44102ac0 2024-04-01T01:42:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.483+0000 7efc3a7fc640 1 -- 172.21.15.70:0/3279029477 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc34039040 con 0x7efc44102ac0 2024-04-01T01:42:27.485 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.483+0000 7efc4b7a7640 1 -- 172.21.15.70:0/3279029477 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc441a6120 con 0x7efc44102ac0 2024-04-01T01:42:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.485+0000 7efc3a7fc640 1 -- 172.21.15.70:0/3279029477 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7efc34044020 con 0x7efc44102ac0 2024-04-01T01:42:27.486 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.486+0000 7efc3a7fc640 1 --2- 172.21.15.70:0/3279029477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc300a2640 0x7efc300a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.487+0000 7efc3a7fc640 1 -- 172.21.15.70:0/3279029477 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7efc340f07b0 con 0x7efc44102ac0 2024-04-01T01:42:27.487 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.487+0000 7efc49543640 1 --2- 172.21.15.70:0/3279029477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc300a2640 0x7efc300a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:27.488 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.488+0000 7efc49543640 1 --2- 172.21.15.70:0/3279029477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc300a2640 0x7efc300a4b00 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7efc4419f210 tx=0x7efc40009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:27.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.489+0000 7efc4e382740 1 Processor -- start 2024-04-01T01:42:27.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.489+0000 7efc4e382740 1 -- start start 2024-04-01T01:42:27.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.489+0000 7efc4e382740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d38d0 0x555fcb7d3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.489+0000 7efc4e382740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d42c0 0x555fcb7d8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.490+0000 7efc4e382740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d9430 0x555fcb7db8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.490+0000 7efc4e382740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555fcb7378f0 con 0x555fcb7d9430 2024-04-01T01:42:27.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.490+0000 7efc4e382740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555fcb6e08d0 con 0x555fcb7d42c0 2024-04-01T01:42:27.490 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.490+0000 7efc4e382740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555fcb733990 con 0x555fcb7d38d0 2024-04-01T01:42:27.491 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.490+0000 7efc49543640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d38d0 0x555fcb7d3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:27.491 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.490+0000 7efc49543640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d38d0 0x555fcb7d3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45158/0 (socket says 172.21.15.70:45158) 2024-04-01T01:42:27.491 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.490+0000 7efc49543640 1 -- 172.21.15.70:0/2950939698 learned_addr learned my addr 172.21.15.70:0/2950939698 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:27.491 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.490+0000 7efc48d42640 1 --2- 172.21.15.70:0/2950939698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d42c0 0x555fcb7d8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:27.491 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.490+0000 7efc48d42640 1 -- 172.21.15.70:0/2950939698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d38d0 msgr2=0x555fcb7d3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:27.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.490+0000 7efc48d42640 1 --2- 172.21.15.70:0/2950939698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d38d0 0x555fcb7d3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.490+0000 7efc48d42640 1 -- 172.21.15.70:0/2950939698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d9430 msgr2=0x555fcb7db8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:27.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.490+0000 7efc48d42640 1 --2- 172.21.15.70:0/2950939698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d9430 0x555fcb7db8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.490+0000 7efc48d42640 1 -- 172.21.15.70:0/2950939698 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fcb44c210 con 0x555fcb7d42c0 2024-04-01T01:42:27.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.491+0000 7efc48d42640 1 --2- 172.21.15.70:0/2950939698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d42c0 0x555fcb7d8dc0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7efc340bcdd0 tx=0x7efc340bc7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6227e60e4a4356f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:27.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.491+0000 7efc2f7fe640 1 -- 172.21.15.70:0/2950939698 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc340bf070 con 0x555fcb7d42c0 2024-04-01T01:42:27.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.491+0000 7efc2f7fe640 1 -- 172.21.15.70:0/2950939698 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc340bd540 con 0x555fcb7d42c0 2024-04-01T01:42:27.492 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.491+0000 7efc2f7fe640 1 -- 172.21.15.70:0/2950939698 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc340b9420 con 0x555fcb7d42c0 2024-04-01T01:42:27.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.492+0000 7efc4e382740 1 -- 172.21.15.70:0/2950939698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d42c0 msgr2=0x555fcb7d8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:27.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.492+0000 7efc4e382740 1 --2- 172.21.15.70:0/2950939698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d42c0 0x555fcb7d8dc0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7efc340bcdd0 tx=0x7efc340bc7a0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.492+0000 7efc4e382740 1 -- 172.21.15.70:0/2950939698 shutdown_connections 2024-04-01T01:42:27.493 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.492+0000 7efc4e382740 1 --2- 172.21.15.70:0/2950939698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d9430 0x555fcb7db8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.492+0000 7efc4e382740 1 --2- 172.21.15.70:0/2950939698 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d42c0 0x555fcb7d8dc0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.492+0000 7efc4e382740 1 --2- 172.21.15.70:0/2950939698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d38d0 0x555fcb7d3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.492+0000 7efc4e382740 1 -- 172.21.15.70:0/2950939698 >> 172.21.15.70:0/2950939698 conn(0x555fcb7cee40 msgr2=0x555fcb7cf240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.492+0000 7efc4e382740 1 -- 172.21.15.70:0/2950939698 shutdown_connections 2024-04-01T01:42:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.492+0000 7efc4e382740 1 -- 172.21.15.70:0/2950939698 wait complete. 2024-04-01T01:42:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.493+0000 7efc4e382740 1 -- start start 2024-04-01T01:42:27.494 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.493+0000 7efc4e382740 1 Processor -- start 2024-04-01T01:42:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.494+0000 7efc4e382740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d38d0 0x555fcb7dd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.496 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.494+0000 7efc4e382740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d42c0 0x555fcb7ddf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.494+0000 7efc4e382740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d9430 0x555fcb7e4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.494+0000 7efc4e382740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555fcb7dc1a0 con 0x555fcb7d38d0 2024-04-01T01:42:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.494+0000 7efc4e382740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555fcb6d5f80 con 0x555fcb7d9430 2024-04-01T01:42:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.494+0000 7efc48d42640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d42c0 0x555fcb7ddf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.494+0000 7efc48d42640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d42c0 0x555fcb7ddf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45168/0 (socket says 172.21.15.70:45168) 2024-04-01T01:42:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.494+0000 7efc48d42640 1 -- 172.21.15.70:0/3810051329 learned_addr learned my addr 172.21.15.70:0/3810051329 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.494+0000 7efc4e382740 1 -- 172.21.15.70:0/3810051329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555fcb7dc320 con 0x555fcb7d42c0 2024-04-01T01:42:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.494+0000 7efc49d44640 1 --2- 172.21.15.70:0/3810051329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d9430 0x555fcb7e4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.494+0000 7efc4e382740 10 client.0 _ll_register_callbacks cb 0x555fcb72e290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.494+0000 7efc49543640 1 --2- 172.21.15.70:0/3810051329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d38d0 0x555fcb7dd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.494+0000 7efc48d42640 1 -- 172.21.15.70:0/3810051329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d9430 msgr2=0x555fcb7e4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.494+0000 7efc48d42640 1 --2- 172.21.15.70:0/3810051329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d9430 0x555fcb7e4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.494+0000 7efc48d42640 1 -- 172.21.15.70:0/3810051329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d38d0 msgr2=0x555fcb7dd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:27.497 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.495+0000 7efc48d42640 1 --2- 172.21.15.70:0/3810051329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d38d0 0x555fcb7dd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.495+0000 7efc48d42640 1 -- 172.21.15.70:0/3810051329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fcb518620 con 0x555fcb7d42c0 2024-04-01T01:42:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.495+0000 7efc49d44640 1 --2- 172.21.15.70:0/3810051329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d9430 0x555fcb7e4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:42:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.495+0000 7efc4e382740 1 -- 172.21.15.70:0/3810051329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc340bd540 con 0x555fcb7d42c0 2024-04-01T01:42:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.495+0000 7efc48d42640 1 --2- 172.21.15.70:0/3810051329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d42c0 0x555fcb7ddf20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efc34048ff0 tx=0x7efc340094d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6511ca7ad1c2e6d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.495+0000 7efc2cff9640 10 client.27946 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:42:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.495+0000 7efc2cff9640 1 -- 172.21.15.70:0/3810051329 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc340a0060 con 0x555fcb7d42c0 2024-04-01T01:42:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.495+0000 7efc2cff9640 1 -- 172.21.15.70:0/3810051329 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7efc340bd0a0 con 0x555fcb7d42c0 2024-04-01T01:42:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.495+0000 7efc2cff9640 1 -- 172.21.15.70:0/3810051329 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efc340bf070 con 0x555fcb7d42c0 2024-04-01T01:42:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.495+0000 7efc4e382740 10 client.27946 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.495+0000 7efc4e382740 1 -- 172.21.15.70:0/3810051329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=31}) -- 0x555fcb7dc1a0 con 0x555fcb7d42c0 2024-04-01T01:42:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.496+0000 7efc2cff9640 1 -- 172.21.15.70:0/3810051329 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7efc3403c030 con 0x555fcb7d42c0 2024-04-01T01:42:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.496+0000 7efc4e382740 10 client.27946 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.496+0000 7efc4e382740 10 client.27946 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:27.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.496+0000 7efc4e382740 20 client.27946 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:27.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.496+0000 7efc4e382740 1 --2- 172.21.15.70:0/3810051329 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555fcb7e9030 0x555fcb7eb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.496+0000 7efc4e382740 4 client.27946 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:27.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.496+0000 7efc4e382740 1 -- 172.21.15.70:0/3810051329 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555fcb74b120 con 0x555fcb7e9030 2024-04-01T01:42:27.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.496+0000 7efc49543640 1 --2- 172.21.15.70:0/3810051329 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555fcb7e9030 0x555fcb7eb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:27.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.496+0000 7efc2cff9640 10 client.27946 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:27.499 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.496+0000 7efc49543640 1 --2- 172.21.15.70:0/3810051329 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555fcb7e9030 0x555fcb7eb420 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=422c5aea5359501d server_cookie=2e70a8a592de2346 in_seq=0 out_seq=0 2024-04-01T01:42:27.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.497+0000 7efc2cff9640 1 -- 172.21.15.70:0/3810051329 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555fcb74b120 con 0x555fcb7e9030 2024-04-01T01:42:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.497+0000 7efc2cff9640 10 client.27946 handle_command_reply: tid=0 2024-04-01T01:42:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.497+0000 7efc4e382740 1 client.27946 shutdown 2024-04-01T01:42:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.497+0000 7efc4e382740 1 -- 172.21.15.70:0/3810051329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d42c0 msgr2=0x555fcb7ddf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.497+0000 7efc4e382740 1 --2- 172.21.15.70:0/3810051329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d42c0 0x555fcb7ddf20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7efc34048ff0 tx=0x7efc340094d0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.498+0000 7efc4e382740 1 -- 172.21.15.70:0/3810051329 shutdown_connections 2024-04-01T01:42:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.498+0000 7efc4e382740 1 --2- 172.21.15.70:0/3810051329 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555fcb7e9030 0x555fcb7eb420 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.498+0000 7efc4e382740 1 --2- 172.21.15.70:0/3810051329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d9430 0x555fcb7e4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.498+0000 7efc4e382740 1 --2- 172.21.15.70:0/3810051329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d42c0 0x555fcb7ddf20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.498+0000 7efc4e382740 1 --2- 172.21.15.70:0/3810051329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d38d0 0x555fcb7dd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.498+0000 7efc4e382740 1 -- 172.21.15.70:0/3810051329 >> 172.21.15.70:0/3810051329 conn(0x555fcb7cee40 msgr2=0x555fcb7cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.498+0000 7efc2cff9640 0 client.27946 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.498+0000 7efc4e382740 1 -- 172.21.15.70:0/3810051329 shutdown_connections 2024-04-01T01:42:27.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.498+0000 7efc4e382740 1 -- 172.21.15.70:0/3810051329 wait complete. 2024-04-01T01:42:27.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.498+0000 7efc4e382740 20 client.27946 trim_cache size 0 max 16384 2024-04-01T01:42:27.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.522+0000 7efc4e382740 1 Processor -- start 2024-04-01T01:42:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.526+0000 7efc4e382740 1 -- start start 2024-04-01T01:42:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.526+0000 7efc4e382740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d38d0 0x555fcb7cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.526+0000 7efc4e382740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d42c0 0x555fcb7e4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.526+0000 7efc4e382740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d9430 0x555fcb7d5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.527+0000 7efc4e382740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efc441b0720 con 0x555fcb7d38d0 2024-04-01T01:42:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.527+0000 7efc4e382740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efc441b2260 con 0x555fcb7d9430 2024-04-01T01:42:27.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.527+0000 7efc4e382740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efc441b0df0 con 0x555fcb7d42c0 2024-04-01T01:42:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.527+0000 7efc48d42640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d42c0 0x555fcb7e4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.527+0000 7efc48d42640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d42c0 0x555fcb7e4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45174/0 (socket says 172.21.15.70:45174) 2024-04-01T01:42:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.527+0000 7efc48d42640 1 -- 172.21.15.70:0/1587897284 learned_addr learned my addr 172.21.15.70:0/1587897284 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.527+0000 7efc49d44640 1 --2- 172.21.15.70:0/1587897284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d9430 0x555fcb7d5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.527+0000 7efc48d42640 1 -- 172.21.15.70:0/1587897284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d9430 msgr2=0x555fcb7d5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.527+0000 7efc48d42640 1 --2- 172.21.15.70:0/1587897284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d9430 0x555fcb7d5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.527+0000 7efc48d42640 1 -- 172.21.15.70:0/1587897284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d38d0 msgr2=0x555fcb7cf220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:42:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.527+0000 7efc48d42640 1 --2- 172.21.15.70:0/1587897284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d38d0 0x555fcb7cf220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.527+0000 7efc48d42640 1 -- 172.21.15.70:0/1587897284 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fcb7d46a0 con 0x555fcb7d42c0 2024-04-01T01:42:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.528+0000 7efc48d42640 1 --2- 172.21.15.70:0/1587897284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d42c0 0x555fcb7e4f20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7efc34003ce0 tx=0x7efc340d9000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c331aca2d7b6a8e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.529+0000 7efc2effd640 1 -- 172.21.15.70:0/1587897284 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc34003b30 con 0x555fcb7d42c0 2024-04-01T01:42:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.529+0000 7efc2effd640 1 -- 172.21.15.70:0/1587897284 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc340bd540 con 0x555fcb7d42c0 2024-04-01T01:42:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.529+0000 7efc2effd640 1 -- 172.21.15.70:0/1587897284 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc340bd0a0 con 0x555fcb7d42c0 2024-04-01T01:42:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.530+0000 7efc4e382740 1 -- 172.21.15.70:0/1587897284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d42c0 msgr2=0x555fcb7e4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.530+0000 7efc4e382740 1 --2- 172.21.15.70:0/1587897284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d42c0 0x555fcb7e4f20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7efc34003ce0 tx=0x7efc340d9000 comp rx=0 tx=0).stop 2024-04-01T01:42:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.530+0000 7efc4e382740 1 -- 172.21.15.70:0/1587897284 shutdown_connections 2024-04-01T01:42:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.530+0000 7efc4e382740 1 --2- 172.21.15.70:0/1587897284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d9430 0x555fcb7d5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.530+0000 7efc4e382740 1 --2- 172.21.15.70:0/1587897284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d42c0 0x555fcb7e4f20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.530+0000 7efc4e382740 1 --2- 172.21.15.70:0/1587897284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d38d0 0x555fcb7cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.530+0000 7efc4e382740 1 -- 172.21.15.70:0/1587897284 >> 172.21.15.70:0/1587897284 conn(0x555fcb7cee40 msgr2=0x555fcb7c9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.530+0000 7efc4e382740 1 -- 172.21.15.70:0/1587897284 shutdown_connections 2024-04-01T01:42:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.530+0000 7efc4e382740 1 -- 172.21.15.70:0/1587897284 wait complete. 2024-04-01T01:42:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.530+0000 7efc4e382740 1 -- start start 2024-04-01T01:42:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.530+0000 7efc4e382740 1 Processor -- start 2024-04-01T01:42:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.531+0000 7efc4e382740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d38d0 0x555fcb7e1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.531+0000 7efc4e382740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d42c0 0x555fcb7e23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.531+0000 7efc4e382740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d9430 0x555fcb84d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.531+0000 7efc4e382740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efc441b1b90 con 0x555fcb7d42c0 2024-04-01T01:42:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.531+0000 7efc4e382740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efc441b14c0 con 0x555fcb7d38d0 2024-04-01T01:42:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.531+0000 7efc4e382740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efc441ae700 con 0x555fcb7d9430 2024-04-01T01:42:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.531+0000 7efc4e382740 10 client.0 _ll_register_callbacks cb 0x7efc40009940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:27.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.531+0000 7efc49d44640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d9430 0x555fcb84d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.531+0000 7efc49d44640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d9430 0x555fcb84d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45188/0 (socket says 172.21.15.70:45188) 2024-04-01T01:42:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.531+0000 7efc49d44640 1 -- 172.21.15.70:0/715938746 learned_addr learned my addr 172.21.15.70:0/715938746 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.531+0000 7efc49543640 1 --2- 172.21.15.70:0/715938746 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d38d0 0x555fcb7e1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.531+0000 7efc48d42640 1 --2- 172.21.15.70:0/715938746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d42c0 0x555fcb7e23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.532+0000 7efc49d44640 1 -- 172.21.15.70:0/715938746 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d38d0 msgr2=0x555fcb7e1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.532+0000 7efc49d44640 1 --2- 172.21.15.70:0/715938746 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d38d0 0x555fcb7e1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.532+0000 7efc49d44640 1 -- 172.21.15.70:0/715938746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d42c0 msgr2=0x555fcb7e23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.532+0000 7efc49d44640 1 --2- 172.21.15.70:0/715938746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d42c0 0x555fcb7e23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.532+0000 7efc49d44640 1 -- 172.21.15.70:0/715938746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fcb84d8a0 con 0x555fcb7d9430 2024-04-01T01:42:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.532+0000 7efc49d44640 1 --2- 172.21.15.70:0/715938746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d9430 0x555fcb84d330 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7efc3c005fd0 tx=0x7efc3c010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eade1e970ddfdfb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.532+0000 7efc2cff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:42:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.533+0000 7efc2cff9640 1 -- 172.21.15.70:0/715938746 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c020060 con 0x555fcb7d9430 2024-04-01T01:42:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.533+0000 7efc4e382740 1 -- 172.21.15.70:0/715938746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc340bd540 con 0x555fcb7d9430 2024-04-01T01:42:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.533+0000 7efc2cff9640 1 -- 172.21.15.70:0/715938746 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7efc3c00f5d0 con 0x555fcb7d9430 2024-04-01T01:42:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.533+0000 7efc2cff9640 1 -- 172.21.15.70:0/715938746 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efc3c027050 con 0x555fcb7d9430 2024-04-01T01:42:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.533+0000 7efc4e382740 10 client.27952 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.533+0000 7efc4e382740 1 -- 172.21.15.70:0/715938746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555fcb84ecb0 con 0x555fcb7d9430 2024-04-01T01:42:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.534+0000 7efc2cff9640 1 -- 172.21.15.70:0/715938746 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc340bd540 con 0x555fcb7d9430 2024-04-01T01:42:27.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.534+0000 7efc2cff9640 1 -- 172.21.15.70:0/715938746 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7efc3c01b830 con 0x555fcb7d9430 2024-04-01T01:42:27.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.535+0000 7efc4e382740 10 client.27952 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:27.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.535+0000 7efc4e382740 10 client.27952 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:27.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.535+0000 7efc4e382740 20 client.27952 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:27.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.535+0000 7efc4e382740 1 --2- 172.21.15.70:0/715938746 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555fcb84f610 0x555fcb851a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:27.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.535+0000 7efc4e382740 4 client.27952 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:42:27.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.535+0000 7efc4e382740 1 -- 172.21.15.70:0/715938746 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555fcb744e30 con 0x555fcb84f610 2024-04-01T01:42:27.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.535+0000 7efc49543640 1 --2- 172.21.15.70:0/715938746 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555fcb84f610 0x555fcb851a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:27.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.536+0000 7efc49543640 1 --2- 172.21.15.70:0/715938746 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555fcb84f610 0x555fcb851a30 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38193f7122c42827 server_cookie=f0eb755fa123955d in_seq=0 out_seq=0 2024-04-01T01:42:27.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.536+0000 7efc2cff9640 10 client.27952 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.590+0000 7efc2cff9640 1 -- 172.21.15.70:0/715938746 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555fcb744e30 con 0x555fcb84f610 2024-04-01T01:42:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.590+0000 7efc2cff9640 10 client.27952 handle_command_reply: tid=0 2024-04-01T01:42:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.590+0000 7efc4e382740 1 client.27952 shutdown 2024-04-01T01:42:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.590+0000 7efc4e382740 1 -- 172.21.15.70:0/715938746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d9430 msgr2=0x555fcb84d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.590+0000 7efc4e382740 1 --2- 172.21.15.70:0/715938746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d9430 0x555fcb84d330 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7efc3c005fd0 tx=0x7efc3c010800 comp rx=0 tx=0).stop 2024-04-01T01:42:27.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.590+0000 7efc4e382740 1 -- 172.21.15.70:0/715938746 shutdown_connections 2024-04-01T01:42:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.590+0000 7efc4e382740 1 --2- 172.21.15.70:0/715938746 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555fcb84f610 0x555fcb851a30 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.591+0000 7efc4e382740 1 --2- 172.21.15.70:0/715938746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fcb7d9430 0x555fcb84d330 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.591+0000 7efc4e382740 1 --2- 172.21.15.70:0/715938746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555fcb7d42c0 0x555fcb7e23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.591+0000 7efc4e382740 1 --2- 172.21.15.70:0/715938746 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555fcb7d38d0 0x555fcb7e1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.591+0000 7efc4e382740 1 -- 172.21.15.70:0/715938746 >> 172.21.15.70:0/715938746 conn(0x555fcb7cee40 msgr2=0x555fcb7cf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.591+0000 7efc2cff9640 0 client.27952 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.591+0000 7efc4e382740 1 -- 172.21.15.70:0/715938746 shutdown_connections 2024-04-01T01:42:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.591+0000 7efc4e382740 1 -- 172.21.15.70:0/715938746 wait complete. 2024-04-01T01:42:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.591+0000 7efc4e382740 20 client.27952 trim_cache size 0 max 16384 2024-04-01T01:42:27.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:42:27.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:42:27.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:42:27.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:42:27.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.593+0000 7efc4b7a7640 1 -- 172.21.15.70:0/3279029477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc300a2640 msgr2=0x7efc300a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:27.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.593+0000 7efc4b7a7640 1 --2- 172.21.15.70:0/3279029477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc300a2640 0x7efc300a4b00 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7efc4419f210 tx=0x7efc40009260 comp rx=0 tx=0).stop 2024-04-01T01:42:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.593+0000 7efc4b7a7640 1 -- 172.21.15.70:0/3279029477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc44102ac0 msgr2=0x7efc4419e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.593+0000 7efc4b7a7640 1 --2- 172.21.15.70:0/3279029477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc44102ac0 0x7efc4419e280 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7efc3402c180 tx=0x7efc340042f0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.593+0000 7efc4b7a7640 1 -- 172.21.15.70:0/3279029477 shutdown_connections 2024-04-01T01:42:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.593+0000 7efc4b7a7640 1 --2- 172.21.15.70:0/3279029477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc44103480 0x7efc441a52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.593+0000 7efc4b7a7640 1 --2- 172.21.15.70:0/3279029477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc44102ac0 0x7efc4419e280 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.593+0000 7efc4b7a7640 1 --2- 172.21.15.70:0/3279029477 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc300a2640 0x7efc300a4b00 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.593+0000 7efc4b7a7640 1 --2- 172.21.15.70:0/3279029477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc440ffa50 0x7efc4419dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.593+0000 7efc4b7a7640 1 -- 172.21.15.70:0/3279029477 >> 172.21.15.70:0/3279029477 conn(0x7efc440fb3a0 msgr2=0x7efc4410c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.594+0000 7efc4b7a7640 1 -- 172.21.15.70:0/3279029477 shutdown_connections 2024-04-01T01:42:27.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:27.594+0000 7efc4b7a7640 1 -- 172.21.15.70:0/3279029477 wait complete. 2024-04-01T01:42:29.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:28 smithi189 ceph-mon[35753]: pgmap v629: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 6.8 MiB/s wr, 769 op/s 2024-04-01T01:42:29.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:28 smithi070 ceph-mon[32179]: pgmap v629: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 6.8 MiB/s wr, 769 op/s 2024-04-01T01:42:29.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:28 smithi081 ceph-mon[35142]: pgmap v629: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 6.8 MiB/s wr, 769 op/s 2024-04-01T01:42:31.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:30 smithi189 ceph-mon[35753]: pgmap v630: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.5 MiB/s wr, 730 op/s 2024-04-01T01:42:31.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:30 smithi070 ceph-mon[32179]: pgmap v630: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.5 MiB/s wr, 730 op/s 2024-04-01T01:42:31.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:30 smithi081 ceph-mon[35142]: pgmap v630: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.5 MiB/s wr, 730 op/s 2024-04-01T01:42:31.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 465.0 s, 809.8 tps, lat 122.995 ms stddev 129.125 2024-04-01T01:42:33.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:32 smithi189 ceph-mon[35753]: pgmap v631: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 7.0 MiB/s wr, 792 op/s 2024-04-01T01:42:33.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:32 smithi070 ceph-mon[32179]: pgmap v631: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 7.0 MiB/s wr, 792 op/s 2024-04-01T01:42:33.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:32 smithi081 ceph-mon[35142]: pgmap v631: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 7.0 MiB/s wr, 792 op/s 2024-04-01T01:42:35.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:34 smithi189 ceph-mon[35753]: pgmap v632: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 6.2 MiB/s wr, 703 op/s 2024-04-01T01:42:35.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:34 smithi070 ceph-mon[32179]: pgmap v632: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 6.2 MiB/s wr, 703 op/s 2024-04-01T01:42:35.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:34 smithi081 ceph-mon[35142]: pgmap v632: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 6.2 MiB/s wr, 703 op/s 2024-04-01T01:42:36.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 470.0 s, 783.8 tps, lat 127.555 ms stddev 147.592 2024-04-01T01:42:37.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:36 smithi189 ceph-mon[35753]: pgmap v633: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.9 MiB/s wr, 794 op/s 2024-04-01T01:42:37.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:36 smithi070 ceph-mon[32179]: pgmap v633: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.9 MiB/s wr, 794 op/s 2024-04-01T01:42:37.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:36 smithi081 ceph-mon[35142]: pgmap v633: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.9 MiB/s wr, 794 op/s 2024-04-01T01:42:39.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:38 smithi189 ceph-mon[35753]: pgmap v634: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 6.4 MiB/s wr, 730 op/s 2024-04-01T01:42:39.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:38 smithi070 ceph-mon[32179]: pgmap v634: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 6.4 MiB/s wr, 730 op/s 2024-04-01T01:42:39.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:38 smithi081 ceph-mon[35142]: pgmap v634: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 6.4 MiB/s wr, 730 op/s 2024-04-01T01:42:40.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.750+0000 7f7ae31e9640 1 -- 172.21.15.70:0/499091498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7adc108fa0 msgr2=0x7f7adc109800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:40.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.750+0000 7f7ae31e9640 1 --2- 172.21.15.70:0/499091498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7adc108fa0 0x7f7adc109800 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0009900 tx=0x7f7ad002bc30 comp rx=0 tx=0).stop 2024-04-01T01:42:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.751+0000 7f7ae31e9640 1 -- 172.21.15.70:0/499091498 shutdown_connections 2024-04-01T01:42:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.751+0000 7f7ae31e9640 1 --2- 172.21.15.70:0/499091498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7adc109d40 0x7f7adc10a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.751+0000 7f7ae31e9640 1 --2- 172.21.15.70:0/499091498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7adc108fa0 0x7f7adc109800 secure :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0009900 tx=0x7f7ad002bc30 comp rx=0 tx=0).stop 2024-04-01T01:42:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.751+0000 7f7ae31e9640 1 --2- 172.21.15.70:0/499091498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7adc072e20 0x7f7adc108a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.751+0000 7f7ae31e9640 1 -- 172.21.15.70:0/499091498 >> 172.21.15.70:0/499091498 conn(0x7f7adc06e790 msgr2=0x7f7adc070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:40.752 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.752+0000 7f7ae31e9640 1 -- 172.21.15.70:0/499091498 shutdown_connections 2024-04-01T01:42:40.752 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.753+0000 7f7ae31e9640 1 -- 172.21.15.70:0/499091498 wait complete. 2024-04-01T01:42:40.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.753+0000 7f7ae31e9640 1 Processor -- start 2024-04-01T01:42:40.753 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.754+0000 7f7ae31e9640 1 -- start start 2024-04-01T01:42:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.754+0000 7f7ae31e9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7adc072e20 0x7f7adc10b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.754+0000 7f7ae31e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7adc109d40 0x7f7adc1a7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.754+0000 7f7ae0f85640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7adc072e20 0x7f7adc10b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.754+0000 7f7ae31e9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7adc1a7950 0x7f7adc1a9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.754+0000 7f7ae0f85640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7adc072e20 0x7f7adc10b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40718/0 (socket says 172.21.15.70:40718) 2024-04-01T01:42:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.754+0000 7f7ae31e9640 1 -- 172.21.15.70:0/3377169874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc1149c0 con 0x7f7adc109d40 2024-04-01T01:42:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.754+0000 7f7ae31e9640 1 -- 172.21.15.70:0/3377169874 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc114840 con 0x7f7adc1a7950 2024-04-01T01:42:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.754+0000 7f7ae31e9640 1 -- 172.21.15.70:0/3377169874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc114b40 con 0x7f7adc072e20 2024-04-01T01:42:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.754+0000 7f7adbfff640 1 --2- 172.21.15.70:0/3377169874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7adc109d40 0x7f7adc1a7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.754+0000 7f7ae0f85640 1 -- 172.21.15.70:0/3377169874 learned_addr learned my addr 172.21.15.70:0/3377169874 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.755+0000 7f7ae1786640 1 --2- 172.21.15.70:0/3377169874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7adc1a7950 0x7f7adc1a9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.755+0000 7f7ae0f85640 1 -- 172.21.15.70:0/3377169874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7adc1a7950 msgr2=0x7f7adc1a9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.755+0000 7f7ae0f85640 1 --2- 172.21.15.70:0/3377169874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7adc1a7950 0x7f7adc1a9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.755+0000 7f7ae0f85640 1 -- 172.21.15.70:0/3377169874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7adc109d40 msgr2=0x7f7adc1a7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.755+0000 7f7ae0f85640 1 --2- 172.21.15.70:0/3377169874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7adc109d40 0x7f7adc1a7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.755+0000 7f7ae0f85640 1 -- 172.21.15.70:0/3377169874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7adc1aa510 con 0x7f7adc072e20 2024-04-01T01:42:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.755+0000 7f7ae0f85640 1 --2- 172.21.15.70:0/3377169874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7adc072e20 0x7f7adc10b5b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7acc00e9f0 tx=0x7f7acc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.755+0000 7f7ad9ffb640 1 -- 172.21.15.70:0/3377169874 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7acc00cd80 con 0x7f7adc072e20 2024-04-01T01:42:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.755+0000 7f7ae31e9640 1 -- 172.21.15.70:0/3377169874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7adc1aa800 con 0x7f7adc072e20 2024-04-01T01:42:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.756+0000 7f7ad9ffb640 1 -- 172.21.15.70:0/3377169874 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7acc0108e0 con 0x7f7adc072e20 2024-04-01T01:42:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.756+0000 7f7ad9ffb640 1 -- 172.21.15.70:0/3377169874 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7acc0188e0 con 0x7f7adc072e20 2024-04-01T01:42:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.756+0000 7f7ae31e9640 1 -- 172.21.15.70:0/3377169874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7adc1aad70 con 0x7f7adc072e20 2024-04-01T01:42:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.759+0000 7f7ad9ffb640 1 -- 172.21.15.70:0/3377169874 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f7acc010a90 con 0x7f7adc072e20 2024-04-01T01:42:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.759+0000 7f7ae5dd0740 1 Processor -- start 2024-04-01T01:42:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.760+0000 7f7ae5dd0740 1 -- start start 2024-04-01T01:42:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.760+0000 7f7ae5dd0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f330e0 0x55b571f33500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.760+0000 7f7ae5dd0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f33ad0 0x55b571f385d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.760+0000 7f7ae5dd0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f38c40 0x55b571f3b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.760+0000 7f7ae5dd0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b571e8e8f0 con 0x55b571f38c40 2024-04-01T01:42:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.760+0000 7f7ae5dd0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b571e378d0 con 0x55b571f33ad0 2024-04-01T01:42:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.760+0000 7f7ae5dd0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b571e8a990 con 0x55b571f330e0 2024-04-01T01:42:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.760+0000 7f7ae1786640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f330e0 0x55b571f33500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.760+0000 7f7adbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f38c40 0x55b571f3b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.760+0000 7f7ae1786640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f330e0 0x55b571f33500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40720/0 (socket says 172.21.15.70:40720) 2024-04-01T01:42:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.760+0000 7f7ae1786640 1 -- 172.21.15.70:0/1766355688 learned_addr learned my addr 172.21.15.70:0/1766355688 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.760+0000 7f7ae0f85640 1 --2- 172.21.15.70:0/1766355688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f33ad0 0x55b571f385d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.760+0000 7f7ad9ffb640 1 --2- 172.21.15.70:0/3377169874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7ac40a2690 0x7f7ac40a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.761+0000 7f7ad9ffb640 1 -- 172.21.15.70:0/3377169874 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f7acc0cee10 con 0x7f7adc072e20 2024-04-01T01:42:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.761+0000 7f7adbfff640 1 -- 172.21.15.70:0/1766355688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f330e0 msgr2=0x55b571f33500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.761+0000 7f7adbfff640 1 --2- 172.21.15.70:0/1766355688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f330e0 0x55b571f33500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.761+0000 7f7adbfff640 1 -- 172.21.15.70:0/1766355688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f33ad0 msgr2=0x55b571f385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.761+0000 7f7adbfff640 1 --2- 172.21.15.70:0/1766355688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f33ad0 0x55b571f385d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.761+0000 7f7adbfff640 1 -- 172.21.15.70:0/1766355688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b571ba3210 con 0x55b571f38c40 2024-04-01T01:42:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.761+0000 7f7ae1786640 1 --2- 172.21.15.70:0/3377169874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7ac40a2690 0x7f7ac40a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.761+0000 7f7adbfff640 1 --2- 172.21.15.70:0/1766355688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f38c40 0x55b571f3b0d0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0008000 tx=0x7f7ad002c610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45bc9c6dac928e48 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.761+0000 7f7ac2ffd640 1 -- 172.21.15.70:0/1766355688 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0045020 con 0x55b571f38c40 2024-04-01T01:42:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.762+0000 7f7ae1786640 1 --2- 172.21.15.70:0/3377169874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7ac40a2690 0x7f7ac40a4b50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4009fd0 tx=0x7f7ad4004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.762+0000 7f7ac2ffd640 1 -- 172.21.15.70:0/1766355688 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ad0004040 con 0x55b571f38c40 2024-04-01T01:42:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.762+0000 7f7ac2ffd640 1 -- 172.21.15.70:0/1766355688 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0005530 con 0x55b571f38c40 2024-04-01T01:42:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.762+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/1766355688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f38c40 msgr2=0x55b571f3b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.762+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/1766355688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f38c40 0x55b571f3b0d0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0008000 tx=0x7f7ad002c610 comp rx=0 tx=0).stop 2024-04-01T01:42:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.763+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/1766355688 shutdown_connections 2024-04-01T01:42:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.763+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/1766355688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f38c40 0x55b571f3b0d0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.763+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/1766355688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f33ad0 0x55b571f385d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.763+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/1766355688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f330e0 0x55b571f33500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.763+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/1766355688 >> 172.21.15.70:0/1766355688 conn(0x55b571f2e650 msgr2=0x55b571f2ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.763+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/1766355688 shutdown_connections 2024-04-01T01:42:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.763+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/1766355688 wait complete. 2024-04-01T01:42:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.763+0000 7f7ae5dd0740 1 -- start start 2024-04-01T01:42:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.763+0000 7f7ae5dd0740 1 Processor -- start 2024-04-01T01:42:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7ae5dd0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f330e0 0x55b571f3d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7ae5dd0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f33ad0 0x55b571f3d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7ae5dd0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f38c40 0x55b571f44870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7ae5dd0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b571f3bb90 con 0x55b571f33ad0 2024-04-01T01:42:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7ae5dd0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b571e2cf80 con 0x55b571f330e0 2024-04-01T01:42:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7ae5dd0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b571f3bd10 con 0x55b571f38c40 2024-04-01T01:42:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7ae0f85640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f330e0 0x55b571f3d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7ae0f85640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f330e0 0x55b571f3d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47364/0 (socket says 172.21.15.70:47364) 2024-04-01T01:42:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7ae0f85640 1 -- 172.21.15.70:0/3024064473 learned_addr learned my addr 172.21.15.70:0/3024064473 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7ae5dd0740 10 client.0 _ll_register_callbacks cb 0x55b571e85290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7adbfff640 1 --2- 172.21.15.70:0/3024064473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f33ad0 0x55b571f3d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7ae0f85640 1 -- 172.21.15.70:0/3024064473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f38c40 msgr2=0x55b571f44870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:42:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7ae0f85640 1 --2- 172.21.15.70:0/3024064473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f38c40 0x55b571f44870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7ae0f85640 1 -- 172.21.15.70:0/3024064473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f33ad0 msgr2=0x55b571f3d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7ae0f85640 1 --2- 172.21.15.70:0/3024064473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f33ad0 0x55b571f3d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7ae0f85640 1 -- 172.21.15.70:0/3024064473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b571c6f620 con 0x55b571f330e0 2024-04-01T01:42:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7ae0f85640 1 --2- 172.21.15.70:0/3024064473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f330e0 0x55b571f3d360 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7acc0cf0f0 tx=0x7f7acc09b150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37fee8afa0c20f9d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.764+0000 7f7abbfff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.765+0000 7f7abbfff640 1 -- 172.21.15.70:0/3024064473 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7acc0bb270 con 0x55b571f330e0 2024-04-01T01:42:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.765+0000 7f7abbfff640 1 -- 172.21.15.70:0/3024064473 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f7acc010430 con 0x55b571f330e0 2024-04-01T01:42:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.765+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/3024064473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ad0004040 con 0x55b571f330e0 2024-04-01T01:42:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.765+0000 7f7abbfff640 1 -- 172.21.15.70:0/3024064473 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7acc09f5f0 con 0x55b571f330e0 2024-04-01T01:42:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.765+0000 7f7ae5dd0740 10 client.28142 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.765+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/3024064473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b571f47910 con 0x55b571f330e0 2024-04-01T01:42:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.765+0000 7f7abbfff640 1 -- 172.21.15.70:0/3024064473 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ad0004040 con 0x55b571f330e0 2024-04-01T01:42:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.766+0000 7f7abbfff640 1 -- 172.21.15.70:0/3024064473 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f7acc09f110 con 0x55b571f330e0 2024-04-01T01:42:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.766+0000 7f7ae5dd0740 10 client.28142 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.766+0000 7f7ae5dd0740 10 client.28142 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.766+0000 7f7ae5dd0740 20 client.28142 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.766+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/3024064473 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b571f48fa0 0x55b571f4b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.766+0000 7f7ae5dd0740 4 client.28142 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.766+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/3024064473 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b571ea2120 con 0x55b571f48fa0 2024-04-01T01:42:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.766+0000 7f7adbfff640 1 --2- 172.21.15.70:0/3024064473 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b571f48fa0 0x55b571f4b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.766+0000 7f7adbfff640 1 --2- 172.21.15.70:0/3024064473 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b571f48fa0 0x55b571f4b3f0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4732857b6f1c9ca server_cookie=b52c7a2ba1625a1a in_seq=0 out_seq=0 2024-04-01T01:42:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.766+0000 7f7abbfff640 10 client.28142 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.767+0000 7f7abbfff640 1 -- 172.21.15.70:0/3024064473 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b571ea2120 con 0x55b571f48fa0 2024-04-01T01:42:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.767+0000 7f7abbfff640 10 client.28142 handle_command_reply: tid=0 2024-04-01T01:42:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.767+0000 7f7ae5dd0740 1 client.28142 shutdown 2024-04-01T01:42:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.767+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/3024064473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f330e0 msgr2=0x55b571f3d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.767+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/3024064473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f330e0 0x55b571f3d360 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7acc0cf0f0 tx=0x7f7acc09b150 comp rx=0 tx=0).stop 2024-04-01T01:42:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.767+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/3024064473 shutdown_connections 2024-04-01T01:42:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.767+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/3024064473 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b571f48fa0 0x55b571f4b3f0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.767+0000 7f7abbfff640 0 client.28142 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.767+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/3024064473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f38c40 0x55b571f44870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.767+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/3024064473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f33ad0 0x55b571f3d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.767+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/3024064473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f330e0 0x55b571f3d360 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.768+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/3024064473 >> 172.21.15.70:0/3024064473 conn(0x55b571f2e650 msgr2=0x55b571f24f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.768+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/3024064473 shutdown_connections 2024-04-01T01:42:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.768+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/3024064473 wait complete. 2024-04-01T01:42:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.768+0000 7f7ae5dd0740 20 client.28142 trim_cache size 0 max 16384 2024-04-01T01:42:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.789+0000 7f7ae5dd0740 1 Processor -- start 2024-04-01T01:42:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.789+0000 7f7ae5dd0740 1 -- start start 2024-04-01T01:42:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7ae5dd0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f330e0 0x55b571f3c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7ae5dd0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f33ad0 0x55b571f3b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7ae5dd0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f38c40 0x55b571f2ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7ae0f85640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f330e0 0x55b571f3c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7ae0f85640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f330e0 0x55b571f3c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51966/0 (socket says 172.21.15.70:51966) 2024-04-01T01:42:40.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7ae0f85640 1 -- 172.21.15.70:0/4136239114 learned_addr learned my addr 172.21.15.70:0/4136239114 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:40.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/4136239114 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc1b0e20 con 0x55b571f330e0 2024-04-01T01:42:40.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/4136239114 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc1b2960 con 0x55b571f33ad0 2024-04-01T01:42:40.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/4136239114 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc1b14f0 con 0x55b571f38c40 2024-04-01T01:42:40.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7adbfff640 1 --2- 172.21.15.70:0/4136239114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f33ad0 0x55b571f3b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7ae1786640 1 --2- 172.21.15.70:0/4136239114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f38c40 0x55b571f2ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7ae0f85640 1 -- 172.21.15.70:0/4136239114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f38c40 msgr2=0x55b571f2ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:40.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7ae0f85640 1 --2- 172.21.15.70:0/4136239114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f38c40 0x55b571f2ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7ae0f85640 1 -- 172.21.15.70:0/4136239114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f33ad0 msgr2=0x55b571f3b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:40.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7ae0f85640 1 --2- 172.21.15.70:0/4136239114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f33ad0 0x55b571f3b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7ae0f85640 1 -- 172.21.15.70:0/4136239114 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b571f38040 con 0x55b571f330e0 2024-04-01T01:42:40.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.790+0000 7f7adbfff640 1 --2- 172.21.15.70:0/4136239114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f33ad0 0x55b571f3b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:42:40.792 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.791+0000 7f7ae0f85640 1 --2- 172.21.15.70:0/4136239114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f330e0 0x55b571f3c980 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7acc097ed0 tx=0x7f7acc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6db546d3f9a6ae0f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:40.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.791+0000 7f7ac27fc640 1 -- 172.21.15.70:0/4136239114 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7acc010430 con 0x55b571f330e0 2024-04-01T01:42:40.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.791+0000 7f7ac27fc640 1 -- 172.21.15.70:0/4136239114 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7acc09f110 con 0x55b571f330e0 2024-04-01T01:42:40.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.791+0000 7f7ac27fc640 1 -- 172.21.15.70:0/4136239114 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7acc0bb340 con 0x55b571f330e0 2024-04-01T01:42:40.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.792+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/4136239114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f330e0 msgr2=0x55b571f3c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:40.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.792+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/4136239114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f330e0 0x55b571f3c980 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7acc097ed0 tx=0x7f7acc000bc0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.792+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/4136239114 shutdown_connections 2024-04-01T01:42:40.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.792+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/4136239114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f38c40 0x55b571f2ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.792+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/4136239114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f33ad0 0x55b571f3b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.792+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/4136239114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f330e0 0x55b571f3c980 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.793 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.792+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/4136239114 >> 172.21.15.70:0/4136239114 conn(0x55b571f2e650 msgr2=0x55b571f383c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:40.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.792+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/4136239114 shutdown_connections 2024-04-01T01:42:40.796 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.792+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/4136239114 wait complete. 2024-04-01T01:42:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.793+0000 7f7ae5dd0740 1 -- start start 2024-04-01T01:42:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.793+0000 7f7ae5dd0740 1 Processor -- start 2024-04-01T01:42:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.793+0000 7f7ae5dd0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f330e0 0x55b571f41b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.793+0000 7f7ae5dd0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f33ad0 0x55b571f42050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.793+0000 7f7ae5dd0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f38c40 0x55b571f43060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.793+0000 7f7ae5dd0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc1b2290 con 0x55b571f33ad0 2024-04-01T01:42:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.793+0000 7f7ae5dd0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc1b1bc0 con 0x55b571f38c40 2024-04-01T01:42:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.793+0000 7f7ae5dd0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7adc1aee00 con 0x55b571f330e0 2024-04-01T01:42:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.793+0000 7f7ae0f85640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f330e0 0x55b571f41b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.793+0000 7f7ae5dd0740 10 client.0 _ll_register_callbacks cb 0x7f7acc0d24c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.793+0000 7f7ae0f85640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f330e0 0x55b571f41b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40758/0 (socket says 172.21.15.70:40758) 2024-04-01T01:42:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.793+0000 7f7ae0f85640 1 -- 172.21.15.70:0/210475368 learned_addr learned my addr 172.21.15.70:0/210475368 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:40.797 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.793+0000 7f7ae1786640 1 --2- 172.21.15.70:0/210475368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f38c40 0x55b571f43060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.794+0000 7f7adbfff640 1 --2- 172.21.15.70:0/210475368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f33ad0 0x55b571f42050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.794+0000 7f7ae0f85640 1 -- 172.21.15.70:0/210475368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f38c40 msgr2=0x55b571f43060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.794+0000 7f7ae0f85640 1 --2- 172.21.15.70:0/210475368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f38c40 0x55b571f43060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.794+0000 7f7ae0f85640 1 -- 172.21.15.70:0/210475368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f33ad0 msgr2=0x55b571f42050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.794+0000 7f7ae0f85640 1 --2- 172.21.15.70:0/210475368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f33ad0 0x55b571f42050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.794+0000 7f7ae0f85640 1 -- 172.21.15.70:0/210475368 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b571f43650 con 0x55b571f330e0 2024-04-01T01:42:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.794+0000 7f7adbfff640 1 --2- 172.21.15.70:0/210475368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f33ad0 0x55b571f42050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:42:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.795+0000 7f7ae0f85640 1 --2- 172.21.15.70:0/210475368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f330e0 0x55b571f41b10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7acc097f20 tx=0x7f7acc00e960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4afe8813ed36e54e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.795+0000 7f7abbfff640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:42:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.795+0000 7f7abbfff640 1 -- 172.21.15.70:0/210475368 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7acc0a3eb0 con 0x55b571f330e0 2024-04-01T01:42:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.795+0000 7f7abbfff640 1 -- 172.21.15.70:0/210475368 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f7acc09abb0 con 0x55b571f330e0 2024-04-01T01:42:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.795+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/210475368 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7acc09f110 con 0x55b571f330e0 2024-04-01T01:42:40.798 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.795+0000 7f7abbfff640 1 -- 172.21.15.70:0/210475368 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7acc0972a0 con 0x55b571f330e0 2024-04-01T01:42:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.795+0000 7f7ae5dd0740 10 client.27967 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.795+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/210475368 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b571faf330 con 0x55b571f330e0 2024-04-01T01:42:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.795+0000 7f7abbfff640 1 -- 172.21.15.70:0/210475368 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7acc09f110 con 0x55b571f330e0 2024-04-01T01:42:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.795+0000 7f7abbfff640 1 -- 172.21.15.70:0/210475368 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f7acc0af190 con 0x55b571f330e0 2024-04-01T01:42:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.796+0000 7f7ae5dd0740 10 client.27967 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.796+0000 7f7ae5dd0740 10 client.27967 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.796+0000 7f7ae5dd0740 20 client.27967 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.796+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/210475368 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b571f48fa0 0x55b571fb19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.796+0000 7f7ae5dd0740 4 client.27967 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:42:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.796+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/210475368 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b571e9be30 con 0x55b571f48fa0 2024-04-01T01:42:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.796+0000 7f7adbfff640 1 --2- 172.21.15.70:0/210475368 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b571f48fa0 0x55b571fb19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.797+0000 7f7abbfff640 10 client.27967 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:40.799 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.797+0000 7f7adbfff640 1 --2- 172.21.15.70:0/210475368 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b571f48fa0 0x55b571fb19a0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5850345291f8842 server_cookie=65a552227ad315fb in_seq=0 out_seq=0 2024-04-01T01:42:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.861+0000 7f7abbfff640 1 -- 172.21.15.70:0/210475368 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b571e9be30 con 0x55b571f48fa0 2024-04-01T01:42:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.861+0000 7f7abbfff640 10 client.27967 handle_command_reply: tid=0 2024-04-01T01:42:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.861+0000 7f7ae5dd0740 1 client.27967 shutdown 2024-04-01T01:42:40.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.862+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/210475368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f330e0 msgr2=0x55b571f41b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:40.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.862+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/210475368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f330e0 0x55b571f41b10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7acc097f20 tx=0x7f7acc00e960 comp rx=0 tx=0).stop 2024-04-01T01:42:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.863+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/210475368 shutdown_connections 2024-04-01T01:42:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.863+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/210475368 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b571f48fa0 0x55b571fb19a0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.863+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/210475368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b571f38c40 0x55b571f43060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.863+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/210475368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b571f33ad0 0x55b571f42050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.863+0000 7f7ae5dd0740 1 --2- 172.21.15.70:0/210475368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b571f330e0 0x55b571f41b10 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.863+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/210475368 >> 172.21.15.70:0/210475368 conn(0x55b571f2e650 msgr2=0x55b571f318b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:40.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.863+0000 7f7abbfff640 0 client.27967 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:40.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.864+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/210475368 shutdown_connections 2024-04-01T01:42:40.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.864+0000 7f7ae5dd0740 1 -- 172.21.15.70:0/210475368 wait complete. 2024-04-01T01:42:40.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.864+0000 7f7ae5dd0740 20 client.27967 trim_cache size 0 max 16384 2024-04-01T01:42:40.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:42:40.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:42:40.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:42:40.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:42:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.870+0000 7f7ae31e9640 1 -- 172.21.15.70:0/3377169874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7ac40a2690 msgr2=0x7f7ac40a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.870+0000 7f7ae31e9640 1 --2- 172.21.15.70:0/3377169874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7ac40a2690 0x7f7ac40a4b50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4009fd0 tx=0x7f7ad4004a40 comp rx=0 tx=0).stop 2024-04-01T01:42:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.870+0000 7f7ae31e9640 1 -- 172.21.15.70:0/3377169874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7adc072e20 msgr2=0x7f7adc10b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.870+0000 7f7ae31e9640 1 --2- 172.21.15.70:0/3377169874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7adc072e20 0x7f7adc10b5b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7acc00e9f0 tx=0x7f7acc00eeb0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.870+0000 7f7ae31e9640 1 -- 172.21.15.70:0/3377169874 shutdown_connections 2024-04-01T01:42:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.870+0000 7f7ae31e9640 1 --2- 172.21.15.70:0/3377169874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7adc1a7950 0x7f7adc1a9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.870+0000 7f7ae31e9640 1 --2- 172.21.15.70:0/3377169874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7adc109d40 0x7f7adc1a7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.870+0000 7f7ae31e9640 1 --2- 172.21.15.70:0/3377169874 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7ac40a2690 0x7f7ac40a4b50 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.870+0000 7f7ae31e9640 1 --2- 172.21.15.70:0/3377169874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7adc072e20 0x7f7adc10b5b0 secure :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7acc00e9f0 tx=0x7f7acc00eeb0 comp rx=0 tx=0).stop 2024-04-01T01:42:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.870+0000 7f7ae31e9640 1 -- 172.21.15.70:0/3377169874 >> 172.21.15.70:0/3377169874 conn(0x7f7adc06e790 msgr2=0x7f7adc070f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.871+0000 7f7ae31e9640 1 -- 172.21.15.70:0/3377169874 shutdown_connections 2024-04-01T01:42:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:40.871+0000 7f7ae31e9640 1 -- 172.21.15.70:0/3377169874 wait complete. 2024-04-01T01:42:41.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:40 smithi189 ceph-mon[35753]: pgmap v635: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.2 MiB/s wr, 700 op/s 2024-04-01T01:42:41.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:40 smithi070 ceph-mon[32179]: pgmap v635: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.2 MiB/s wr, 700 op/s 2024-04-01T01:42:41.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:40 smithi081 ceph-mon[35142]: pgmap v635: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.2 MiB/s wr, 700 op/s 2024-04-01T01:42:41.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 475.0 s, 858.0 tps, lat 117.407 ms stddev 115.375 2024-04-01T01:42:42.112 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:42:42.113 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:42:42.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.281+0000 7efc2f59e640 1 -- 172.21.15.70:0/1308112542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc30108e20 msgr2=0x7efc301092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:42.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.281+0000 7efc2f59e640 1 --2- 172.21.15.70:0/1308112542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc30108e20 0x7efc301092a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7efc2401a910 tx=0x7efc2403cc60 comp rx=0 tx=0).stop 2024-04-01T01:42:42.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.281+0000 7efc2f59e640 1 -- 172.21.15.70:0/1308112542 shutdown_connections 2024-04-01T01:42:42.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.281+0000 7efc2f59e640 1 --2- 172.21.15.70:0/1308112542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc301097e0 0x7efc3010e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.281+0000 7efc2f59e640 1 --2- 172.21.15.70:0/1308112542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc30108e20 0x7efc301092a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.281+0000 7efc2f59e640 1 --2- 172.21.15.70:0/1308112542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc30072b60 0x7efc30072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.281+0000 7efc2f59e640 1 -- 172.21.15.70:0/1308112542 >> 172.21.15.70:0/1308112542 conn(0x7efc3006e630 msgr2=0x7efc30070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:42.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.283+0000 7efc2f59e640 1 -- 172.21.15.70:0/1308112542 shutdown_connections 2024-04-01T01:42:42.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.283+0000 7efc2f59e640 1 -- 172.21.15.70:0/1308112542 wait complete. 2024-04-01T01:42:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.284+0000 7efc2f59e640 1 Processor -- start 2024-04-01T01:42:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.284+0000 7efc2f59e640 1 -- start start 2024-04-01T01:42:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.284+0000 7efc2f59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc30072b60 0x7efc3019de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.284+0000 7efc2f59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc30108e20 0x7efc3019e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.284+0000 7efc2f59e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc301097e0 0x7efc301a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.284+0000 7efc2f59e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efc30111930 con 0x7efc30108e20 2024-04-01T01:42:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.284+0000 7efc2f59e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efc301117b0 con 0x7efc30072b60 2024-04-01T01:42:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.284+0000 7efc2f59e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efc30111ab0 con 0x7efc301097e0 2024-04-01T01:42:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.285+0000 7efc2e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc30072b60 0x7efc3019de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.285+0000 7efc2e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc30072b60 0x7efc3019de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47390/0 (socket says 172.21.15.70:47390) 2024-04-01T01:42:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.285+0000 7efc2e59c640 1 -- 172.21.15.70:0/4171282444 learned_addr learned my addr 172.21.15.70:0/4171282444 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.285+0000 7efc2ed9d640 1 --2- 172.21.15.70:0/4171282444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc301097e0 0x7efc301a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.285+0000 7efc2dd9b640 1 --2- 172.21.15.70:0/4171282444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc30108e20 0x7efc3019e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.285+0000 7efc2dd9b640 1 -- 172.21.15.70:0/4171282444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc301097e0 msgr2=0x7efc301a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:42.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.285+0000 7efc2dd9b640 1 --2- 172.21.15.70:0/4171282444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc301097e0 0x7efc301a5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.285+0000 7efc2dd9b640 1 -- 172.21.15.70:0/4171282444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc30072b60 msgr2=0x7efc3019de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:42.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.285+0000 7efc2dd9b640 1 --2- 172.21.15.70:0/4171282444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc30072b60 0x7efc3019de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.285+0000 7efc2dd9b640 1 -- 172.21.15.70:0/4171282444 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc301a5a80 con 0x7efc30108e20 2024-04-01T01:42:42.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.286+0000 7efc2e59c640 1 --2- 172.21.15.70:0/4171282444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc30072b60 0x7efc3019de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:42:42.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.286+0000 7efc2dd9b640 1 --2- 172.21.15.70:0/4171282444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc30108e20 0x7efc3019e3a0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7efc2401a860 tx=0x7efc2403def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:42.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.287+0000 7efc1f7fe640 1 -- 172.21.15.70:0/4171282444 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc24052020 con 0x7efc30108e20 2024-04-01T01:42:42.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.287+0000 7efc1f7fe640 1 -- 172.21.15.70:0/4171282444 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc240053f0 con 0x7efc30108e20 2024-04-01T01:42:42.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.287+0000 7efc1f7fe640 1 -- 172.21.15.70:0/4171282444 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc24003bf0 con 0x7efc30108e20 2024-04-01T01:42:42.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.287+0000 7efc2f59e640 1 -- 172.21.15.70:0/4171282444 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc301a5d10 con 0x7efc30108e20 2024-04-01T01:42:42.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.287+0000 7efc2f59e640 1 -- 172.21.15.70:0/4171282444 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc301a61b0 con 0x7efc30108e20 2024-04-01T01:42:42.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.290+0000 7efc1f7fe640 1 -- 172.21.15.70:0/4171282444 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7efc24055020 con 0x7efc30108e20 2024-04-01T01:42:42.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.291+0000 7efc37816740 1 Processor -- start 2024-04-01T01:42:42.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.291+0000 7efc37816740 1 -- start start 2024-04-01T01:42:42.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.291+0000 7efc37816740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115490 0x558ac41158b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.291+0000 7efc37816740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115e80 0x558ac411a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.291+0000 7efc37816740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 0x558ac411d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.292+0000 7efc37816740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558ac407a180 con 0x558ac4115e80 2024-04-01T01:42:42.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.292+0000 7efc37816740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x558ac4022b70 con 0x558ac4115490 2024-04-01T01:42:42.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.292+0000 7efc37816740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558ac4076220 con 0x558ac411aff0 2024-04-01T01:42:42.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.292+0000 7efc1f7fe640 1 --2- 172.21.15.70:0/4171282444 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc180a25f0 0x7efc180a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.292+0000 7efc2dd9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 0x558ac411d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.292+0000 7efc2ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115490 0x558ac41158b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.292+0000 7efc2ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115490 0x558ac41158b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47404/0 (socket says 172.21.15.70:47404) 2024-04-01T01:42:42.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.292+0000 7efc2ed9d640 1 -- 172.21.15.70:0/567753814 learned_addr learned my addr 172.21.15.70:0/567753814 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:42.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.292+0000 7efc2ed9d640 1 --2- 172.21.15.70:0/4171282444 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc180a25f0 0x7efc180a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.292+0000 7efc1f7fe640 1 -- 172.21.15.70:0/4171282444 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7efc24101740 con 0x7efc30108e20 2024-04-01T01:42:42.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.292+0000 7efc2ed9d640 1 -- 172.21.15.70:0/567753814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 msgr2=0x558ac411d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:42.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.292+0000 7efc2ed9d640 1 --2- 172.21.15.70:0/567753814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 0x558ac411d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.292+0000 7efc2ed9d640 1 -- 172.21.15.70:0/567753814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115e80 msgr2=0x558ac411a980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:42:42.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.292+0000 7efc2ed9d640 1 --2- 172.21.15.70:0/567753814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115e80 0x558ac411a980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.292+0000 7efc2ed9d640 1 -- 172.21.15.70:0/567753814 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ac40367c0 con 0x558ac4115490 2024-04-01T01:42:42.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.292+0000 7efc2dd9b640 1 --2- 172.21.15.70:0/567753814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 0x558ac411d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:42:42.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.293+0000 7efc2ed9d640 1 --2- 172.21.15.70:0/567753814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115490 0x558ac41158b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7efc20011590 tx=0x7efc20011a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4ce751f09a01253 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:42.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.293+0000 7efc2ed9d640 1 --2- 172.21.15.70:0/4171282444 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc180a25f0 0x7efc180a4ab0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7efc2001d000 tx=0x7efc2001f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:42.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.293+0000 7efc1cff9640 1 -- 172.21.15.70:0/567753814 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2000fc50 con 0x558ac4115490 2024-04-01T01:42:42.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.293+0000 7efc1cff9640 1 -- 172.21.15.70:0/567753814 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc20022070 con 0x558ac4115490 2024-04-01T01:42:42.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.293+0000 7efc1cff9640 1 -- 172.21.15.70:0/567753814 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc200279b0 con 0x558ac4115490 2024-04-01T01:42:42.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.294+0000 7efc37816740 1 -- 172.21.15.70:0/567753814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115490 msgr2=0x558ac41158b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:42.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.294+0000 7efc37816740 1 --2- 172.21.15.70:0/567753814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115490 0x558ac41158b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7efc20011590 tx=0x7efc20011a50 comp rx=0 tx=0).stop 2024-04-01T01:42:42.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.294+0000 7efc37816740 1 -- 172.21.15.70:0/567753814 shutdown_connections 2024-04-01T01:42:42.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.294+0000 7efc37816740 1 --2- 172.21.15.70:0/567753814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 0x558ac411d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.294+0000 7efc37816740 1 --2- 172.21.15.70:0/567753814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115e80 0x558ac411a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.294+0000 7efc37816740 1 --2- 172.21.15.70:0/567753814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115490 0x558ac41158b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.294+0000 7efc37816740 1 -- 172.21.15.70:0/567753814 >> 172.21.15.70:0/567753814 conn(0x558ac4110a00 msgr2=0x558ac4110e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:42.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.294+0000 7efc37816740 1 -- 172.21.15.70:0/567753814 shutdown_connections 2024-04-01T01:42:42.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.294+0000 7efc37816740 1 -- 172.21.15.70:0/567753814 wait complete. 2024-04-01T01:42:42.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.295+0000 7efc37816740 1 -- start start 2024-04-01T01:42:42.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.295+0000 7efc37816740 1 Processor -- start 2024-04-01T01:42:42.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.295+0000 7efc37816740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115490 0x558ac411fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.295+0000 7efc37816740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac4115e80 0x558ac411ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.295+0000 7efc37816740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac411aff0 0x558ac4126f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.295+0000 7efc37816740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558ac411df40 con 0x558ac4115490 2024-04-01T01:42:42.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.295+0000 7efc2e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115490 0x558ac411fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.295+0000 7efc37816740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x558ac40184a0 con 0x558ac411aff0 2024-04-01T01:42:42.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.295+0000 7efc37816740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558ac411e0c0 con 0x558ac4115e80 2024-04-01T01:42:42.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.295+0000 7efc2e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115490 0x558ac411fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51998/0 (socket says 172.21.15.70:51998) 2024-04-01T01:42:42.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.295+0000 7efc2e59c640 1 -- 172.21.15.70:0/1716534458 learned_addr learned my addr 172.21.15.70:0/1716534458 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:42.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.295+0000 7efc37816740 10 client.0 _ll_register_callbacks cb 0x558ac401bd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:42.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.296+0000 7efc2ed9d640 1 --2- 172.21.15.70:0/1716534458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac411aff0 0x558ac4126f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.296+0000 7efc2e59c640 1 -- 172.21.15.70:0/1716534458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac4115e80 msgr2=0x558ac411ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:42.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.296+0000 7efc2e59c640 1 --2- 172.21.15.70:0/1716534458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac4115e80 0x558ac411ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.296+0000 7efc2e59c640 1 -- 172.21.15.70:0/1716534458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac411aff0 msgr2=0x558ac4126f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:42.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.296+0000 7efc2e59c640 1 --2- 172.21.15.70:0/1716534458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac411aff0 0x558ac4126f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.296+0000 7efc2e59c640 1 -- 172.21.15.70:0/1716534458 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ac3e587d0 con 0x558ac4115490 2024-04-01T01:42:42.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.296+0000 7efc2e59c640 1 --2- 172.21.15.70:0/1716534458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115490 0x558ac411fa60 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7efc2800ec40 tx=0x7efc28010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa71fc21b8fc1adc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:42.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.296+0000 7efc15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:42:42.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.297+0000 7efc15ffb640 1 -- 172.21.15.70:0/1716534458 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc28012650 con 0x558ac4115490 2024-04-01T01:42:42.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.297+0000 7efc37816740 1 -- 172.21.15.70:0/1716534458 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc20022070 con 0x558ac4115490 2024-04-01T01:42:42.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.297+0000 7efc15ffb640 1 -- 172.21.15.70:0/1716534458 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7efc280127f0 con 0x558ac4115490 2024-04-01T01:42:42.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.297+0000 7efc15ffb640 1 -- 172.21.15.70:0/1716534458 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efc28012a30 con 0x558ac4115490 2024-04-01T01:42:42.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.298+0000 7efc37816740 10 client.18165 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:42.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.298+0000 7efc37816740 1 -- 172.21.15.70:0/1716534458 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558ac3d8e3f0 con 0x558ac4115490 2024-04-01T01:42:42.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.298+0000 7efc15ffb640 1 -- 172.21.15.70:0/1716534458 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc20022070 con 0x558ac4115490 2024-04-01T01:42:42.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.299+0000 7efc15ffb640 1 -- 172.21.15.70:0/1716534458 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7efc2800f040 con 0x558ac4115490 2024-04-01T01:42:42.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.299+0000 7efc37816740 10 client.18165 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:42.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.299+0000 7efc37816740 10 client.18165 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:42.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.299+0000 7efc37816740 20 client.18165 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:42.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.299+0000 7efc37816740 1 --2- 172.21.15.70:0/1716534458 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558ac412b820 0x558ac412dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.299+0000 7efc37816740 4 client.18165 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:42.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.299+0000 7efc37816740 1 -- 172.21.15.70:0/1716534458 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ac40876c0 con 0x558ac412b820 2024-04-01T01:42:42.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.299+0000 7efc2dd9b640 1 --2- 172.21.15.70:0/1716534458 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558ac412b820 0x558ac412dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.300+0000 7efc2dd9b640 1 --2- 172.21.15.70:0/1716534458 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558ac412b820 0x558ac412dce0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94a4b57040776809 server_cookie=e19456901d306f26 in_seq=0 out_seq=0 2024-04-01T01:42:42.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.300+0000 7efc15ffb640 10 client.18165 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:42.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.301+0000 7efc15ffb640 1 -- 172.21.15.70:0/1716534458 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558ac40876c0 con 0x558ac412b820 2024-04-01T01:42:42.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.301+0000 7efc15ffb640 10 client.18165 handle_command_reply: tid=0 2024-04-01T01:42:42.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.301+0000 7efc37816740 1 client.18165 shutdown 2024-04-01T01:42:42.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.301+0000 7efc37816740 1 -- 172.21.15.70:0/1716534458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115490 msgr2=0x558ac411fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:42.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.301+0000 7efc37816740 1 --2- 172.21.15.70:0/1716534458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115490 0x558ac411fa60 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7efc2800ec40 tx=0x7efc28010740 comp rx=0 tx=0).stop 2024-04-01T01:42:42.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.301+0000 7efc37816740 1 -- 172.21.15.70:0/1716534458 shutdown_connections 2024-04-01T01:42:42.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.301+0000 7efc37816740 1 --2- 172.21.15.70:0/1716534458 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558ac412b820 0x558ac412dce0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.301+0000 7efc37816740 1 --2- 172.21.15.70:0/1716534458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac411aff0 0x558ac4126f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.301+0000 7efc15ffb640 0 client.18165 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:42.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.301+0000 7efc37816740 1 --2- 172.21.15.70:0/1716534458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac4115e80 0x558ac411ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.301+0000 7efc37816740 1 --2- 172.21.15.70:0/1716534458 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115490 0x558ac411fa60 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.302+0000 7efc37816740 1 -- 172.21.15.70:0/1716534458 >> 172.21.15.70:0/1716534458 conn(0x558ac4110a00 msgr2=0x558ac410fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:42.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.302+0000 7efc37816740 1 -- 172.21.15.70:0/1716534458 shutdown_connections 2024-04-01T01:42:42.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.302+0000 7efc37816740 1 -- 172.21.15.70:0/1716534458 wait complete. 2024-04-01T01:42:42.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.302+0000 7efc37816740 20 client.18165 trim_cache size 0 max 16384 2024-04-01T01:42:42.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.335+0000 7efc37816740 1 Processor -- start 2024-04-01T01:42:42.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.335+0000 7efc37816740 1 -- start start 2024-04-01T01:42:42.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.336+0000 7efc37816740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115490 0x558ac4114370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.336+0000 7efc37816740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115e80 0x558ac4126f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.336+0000 7efc37816740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 0x558ac411f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.336+0000 7efc37816740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efc301b07b0 con 0x558ac4115e80 2024-04-01T01:42:42.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.336+0000 7efc37816740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efc301b22f0 con 0x558ac4115490 2024-04-01T01:42:42.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.337+0000 7efc2ed9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 0x558ac411f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.337+0000 7efc2ed9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 0x558ac411f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40804/0 (socket says 172.21.15.70:40804) 2024-04-01T01:42:42.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.337+0000 7efc2ed9d640 1 -- 172.21.15.70:0/1995336087 learned_addr learned my addr 172.21.15.70:0/1995336087 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:42.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.337+0000 7efc2dd9b640 1 --2- 172.21.15.70:0/1995336087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115e80 0x558ac4126f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.337+0000 7efc37816740 1 -- 172.21.15.70:0/1995336087 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efc301b0e80 con 0x558ac411aff0 2024-04-01T01:42:42.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.337+0000 7efc2ed9d640 1 -- 172.21.15.70:0/1995336087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115490 msgr2=0x558ac4114370 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:42:42.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.337+0000 7efc2ed9d640 1 --2- 172.21.15.70:0/1995336087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115490 0x558ac4114370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.337+0000 7efc2ed9d640 1 -- 172.21.15.70:0/1995336087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115e80 msgr2=0x558ac4126f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:42.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.337+0000 7efc2ed9d640 1 --2- 172.21.15.70:0/1995336087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115e80 0x558ac4126f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.337+0000 7efc2ed9d640 1 -- 172.21.15.70:0/1995336087 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ac3d4e0f0 con 0x558ac411aff0 2024-04-01T01:42:42.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.338+0000 7efc2ed9d640 1 --2- 172.21.15.70:0/1995336087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 0x558ac411f080 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efc20009700 tx=0x7efc2001eda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f866e3b6f44848d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:42.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.338+0000 7efc17fff640 1 -- 172.21.15.70:0/1995336087 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2002b070 con 0x558ac411aff0 2024-04-01T01:42:42.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.338+0000 7efc17fff640 1 -- 172.21.15.70:0/1995336087 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc20035070 con 0x558ac411aff0 2024-04-01T01:42:42.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.338+0000 7efc17fff640 1 -- 172.21.15.70:0/1995336087 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc20027c30 con 0x558ac411aff0 2024-04-01T01:42:42.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.339+0000 7efc37816740 1 -- 172.21.15.70:0/1995336087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 msgr2=0x558ac411f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:42.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.339+0000 7efc37816740 1 --2- 172.21.15.70:0/1995336087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 0x558ac411f080 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efc20009700 tx=0x7efc2001eda0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.339+0000 7efc37816740 1 -- 172.21.15.70:0/1995336087 shutdown_connections 2024-04-01T01:42:42.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.339+0000 7efc37816740 1 --2- 172.21.15.70:0/1995336087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 0x558ac411f080 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.339+0000 7efc37816740 1 --2- 172.21.15.70:0/1995336087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115e80 0x558ac4126f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.339+0000 7efc37816740 1 --2- 172.21.15.70:0/1995336087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115490 0x558ac4114370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.339+0000 7efc37816740 1 -- 172.21.15.70:0/1995336087 >> 172.21.15.70:0/1995336087 conn(0x558ac4110a00 msgr2=0x558ac4113c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:42.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.340+0000 7efc37816740 1 -- 172.21.15.70:0/1995336087 shutdown_connections 2024-04-01T01:42:42.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.340+0000 7efc37816740 1 -- 172.21.15.70:0/1995336087 wait complete. 2024-04-01T01:42:42.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.341+0000 7efc37816740 1 -- start start 2024-04-01T01:42:42.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.341+0000 7efc37816740 1 Processor -- start 2024-04-01T01:42:42.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.342+0000 7efc37816740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115490 0x558ac4124060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.342+0000 7efc37816740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115e80 0x558ac41245a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.342+0000 7efc2e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115490 0x558ac4124060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.342+0000 7efc2e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115490 0x558ac4124060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52014/0 (socket says 172.21.15.70:52014) 2024-04-01T01:42:42.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.342+0000 7efc37816740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 0x558ac41255b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.342+0000 7efc2dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115e80 0x558ac41245a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.342+0000 7efc2dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115e80 0x558ac41245a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47434/0 (socket says 172.21.15.70:47434) 2024-04-01T01:42:42.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.342+0000 7efc2e59c640 1 -- 172.21.15.70:0/1226067184 learned_addr learned my addr 172.21.15.70:0/1226067184 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:42.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.342+0000 7efc37816740 1 -- 172.21.15.70:0/1226067184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efc301b1c20 con 0x558ac4115490 2024-04-01T01:42:42.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.342+0000 7efc2ed9d640 1 --2- 172.21.15.70:0/1226067184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 0x558ac41255b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.342+0000 7efc37816740 1 -- 172.21.15.70:0/1226067184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efc301b1550 con 0x558ac4115e80 2024-04-01T01:42:42.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.342+0000 7efc37816740 1 -- 172.21.15.70:0/1226067184 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efc301ae790 con 0x558ac411aff0 2024-04-01T01:42:42.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.343+0000 7efc37816740 10 client.0 _ll_register_callbacks cb 0x7efc2800e2f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:42.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.343+0000 7efc2dd9b640 1 -- 172.21.15.70:0/1226067184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 msgr2=0x558ac41255b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:42.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.343+0000 7efc2dd9b640 1 --2- 172.21.15.70:0/1226067184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 0x558ac41255b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.343+0000 7efc2dd9b640 1 -- 172.21.15.70:0/1226067184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115490 msgr2=0x558ac4124060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:42.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.343+0000 7efc2dd9b640 1 --2- 172.21.15.70:0/1226067184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115490 0x558ac4124060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.343+0000 7efc2dd9b640 1 -- 172.21.15.70:0/1226067184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ac4125ba0 con 0x558ac4115e80 2024-04-01T01:42:42.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.343+0000 7efc2dd9b640 1 --2- 172.21.15.70:0/1226067184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115e80 0x558ac41245a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7efc240cdad0 tx=0x7efc2409dc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d7ddee7cf747d26 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:42.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.343+0000 7efc15ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:42.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.343+0000 7efc15ffb640 1 -- 172.21.15.70:0/1226067184 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc240d1040 con 0x558ac4115e80 2024-04-01T01:42:42.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.343+0000 7efc15ffb640 1 -- 172.21.15.70:0/1226067184 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7efc240cc0d0 con 0x558ac4115e80 2024-04-01T01:42:42.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.343+0000 7efc37816740 1 -- 172.21.15.70:0/1226067184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc20035070 con 0x558ac4115e80 2024-04-01T01:42:42.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.344+0000 7efc15ffb640 1 -- 172.21.15.70:0/1226067184 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efc2403d810 con 0x558ac4115e80 2024-04-01T01:42:42.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.345+0000 7efc37816740 10 client.28163 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:42.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.345+0000 7efc37816740 1 -- 172.21.15.70:0/1226067184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558ac4126ee0 con 0x558ac4115e80 2024-04-01T01:42:42.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.345+0000 7efc15ffb640 1 -- 172.21.15.70:0/1226067184 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efc20035070 con 0x558ac4115e80 2024-04-01T01:42:42.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.345+0000 7efc15ffb640 1 -- 172.21.15.70:0/1226067184 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7efc241013b0 con 0x558ac4115e80 2024-04-01T01:42:42.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.345+0000 7efc37816740 10 client.28163 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:42.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.346+0000 7efc37816740 10 client.28163 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:42.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.346+0000 7efc37816740 20 client.28163 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:42.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.346+0000 7efc37816740 1 --2- 172.21.15.70:0/1226067184 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558ac412b820 0x558ac4194080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.346+0000 7efc37816740 4 client.28163 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:42:42.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.346+0000 7efc37816740 1 -- 172.21.15.70:0/1226067184 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558ac4081a10 con 0x558ac412b820 2024-04-01T01:42:42.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.346+0000 7efc2e59c640 1 --2- 172.21.15.70:0/1226067184 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558ac412b820 0x558ac4194080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.347+0000 7efc15ffb640 10 client.28163 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:42.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.347+0000 7efc2e59c640 1 --2- 172.21.15.70:0/1226067184 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558ac412b820 0x558ac4194080 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd4fcb8f0e3f1bf4 server_cookie=403b4a5a6830d47f in_seq=0 out_seq=0 2024-04-01T01:42:42.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.347+0000 7efc15ffb640 1 -- 172.21.15.70:0/1226067184 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558ac4081a10 con 0x558ac412b820 2024-04-01T01:42:42.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.347+0000 7efc15ffb640 10 client.28163 handle_command_reply: tid=0 2024-04-01T01:42:42.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.348+0000 7efc37816740 1 client.28163 shutdown 2024-04-01T01:42:42.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.348+0000 7efc37816740 1 -- 172.21.15.70:0/1226067184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115e80 msgr2=0x558ac41245a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:42.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.348+0000 7efc37816740 1 --2- 172.21.15.70:0/1226067184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115e80 0x558ac41245a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7efc240cdad0 tx=0x7efc2409dc80 comp rx=0 tx=0).stop 2024-04-01T01:42:42.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.349+0000 7efc37816740 1 -- 172.21.15.70:0/1226067184 shutdown_connections 2024-04-01T01:42:42.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.349+0000 7efc37816740 1 --2- 172.21.15.70:0/1226067184 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558ac412b820 0x558ac4194080 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.349+0000 7efc37816740 1 --2- 172.21.15.70:0/1226067184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ac411aff0 0x558ac41255b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.349+0000 7efc37816740 1 --2- 172.21.15.70:0/1226067184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558ac4115e80 0x558ac41245a0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.349+0000 7efc37816740 1 --2- 172.21.15.70:0/1226067184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558ac4115490 0x558ac4124060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.349+0000 7efc37816740 1 -- 172.21.15.70:0/1226067184 >> 172.21.15.70:0/1226067184 conn(0x558ac4110a00 msgr2=0x558ac410f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.349+0000 7efc15ffb640 0 client.28163 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.349+0000 7efc37816740 1 -- 172.21.15.70:0/1226067184 shutdown_connections 2024-04-01T01:42:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.349+0000 7efc37816740 1 -- 172.21.15.70:0/1226067184 wait complete. 2024-04-01T01:42:42.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.350+0000 7efc37816740 20 client.28163 trim_cache size 0 max 16384 2024-04-01T01:42:42.351 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:42:42.351 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:42:42.351 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:42:42.351 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:42:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.352+0000 7efc2f59e640 1 -- 172.21.15.70:0/4171282444 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc180a25f0 msgr2=0x7efc180a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.352+0000 7efc2f59e640 1 --2- 172.21.15.70:0/4171282444 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc180a25f0 0x7efc180a4ab0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7efc2001d000 tx=0x7efc2001f000 comp rx=0 tx=0).stop 2024-04-01T01:42:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.352+0000 7efc2f59e640 1 -- 172.21.15.70:0/4171282444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc30108e20 msgr2=0x7efc3019e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.352+0000 7efc2f59e640 1 --2- 172.21.15.70:0/4171282444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc30108e20 0x7efc3019e3a0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7efc2401a860 tx=0x7efc2403def0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.352+0000 7efc2f59e640 1 -- 172.21.15.70:0/4171282444 shutdown_connections 2024-04-01T01:42:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.352+0000 7efc2f59e640 1 --2- 172.21.15.70:0/4171282444 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efc180a25f0 0x7efc180a4ab0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.352+0000 7efc2f59e640 1 --2- 172.21.15.70:0/4171282444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc301097e0 0x7efc301a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.352+0000 7efc2f59e640 1 --2- 172.21.15.70:0/4171282444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efc30108e20 0x7efc3019e3a0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.352+0000 7efc2f59e640 1 --2- 172.21.15.70:0/4171282444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc30072b60 0x7efc3019de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.352+0000 7efc2f59e640 1 -- 172.21.15.70:0/4171282444 >> 172.21.15.70:0/4171282444 conn(0x7efc3006e630 msgr2=0x7efc3010c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.353+0000 7efc2f59e640 1 -- 172.21.15.70:0/4171282444 shutdown_connections 2024-04-01T01:42:42.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.353+0000 7efc2f59e640 1 -- 172.21.15.70:0/4171282444 wait complete. 2024-04-01T01:42:42.375 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:42:42.376 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:42:42.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.548+0000 7f7e7b7fe640 1 -- 172.21.15.70:0/3440310974 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c039070 con 0x7f7e7c107bb0 2024-04-01T01:42:42.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.549+0000 7f7e81d67640 1 -- 172.21.15.70:0/3440310974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e7c107bb0 msgr2=0x7f7e7c107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:42.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.549+0000 7f7e81d67640 1 --2- 172.21.15.70:0/3440310974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e7c107bb0 0x7f7e7c107fd0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009860 tx=0x7f7e6c02bb70 comp rx=0 tx=0).stop 2024-04-01T01:42:42.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.554+0000 7f7e81d67640 1 -- 172.21.15.70:0/3440310974 shutdown_connections 2024-04-01T01:42:42.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.554+0000 7f7e81d67640 1 --2- 172.21.15.70:0/3440310974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e7c109760 0x7f7e7c10fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.554+0000 7f7e81d67640 1 --2- 172.21.15.70:0/3440310974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e7c108da0 0x7f7e7c109220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.554+0000 7f7e81d67640 1 --2- 172.21.15.70:0/3440310974 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e7c107bb0 0x7f7e7c107fd0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.554+0000 7f7e81d67640 1 -- 172.21.15.70:0/3440310974 >> 172.21.15.70:0/3440310974 conn(0x7f7e7c076b60 msgr2=0x7f7e7c078f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:42.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.557+0000 7f7e81d67640 1 -- 172.21.15.70:0/3440310974 shutdown_connections 2024-04-01T01:42:42.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.557+0000 7f7e81d67640 1 -- 172.21.15.70:0/3440310974 wait complete. 2024-04-01T01:42:42.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.557+0000 7f7e81d67640 1 Processor -- start 2024-04-01T01:42:42.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.557+0000 7f7e81d67640 1 -- start start 2024-04-01T01:42:42.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.558+0000 7f7e81d67640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e7c107bb0 0x7f7e7c080110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.558+0000 7f7e81d67640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e7c108da0 0x7f7e7c080650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.558+0000 7f7e81d67640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e7c109760 0x7f7e7c081690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.558+0000 7f7e81d67640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c07a700 con 0x7f7e7c108da0 2024-04-01T01:42:42.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.558+0000 7f7e81d67640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c07a580 con 0x7f7e7c109760 2024-04-01T01:42:42.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.558+0000 7f7e81d67640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c07a880 con 0x7f7e7c107bb0 2024-04-01T01:42:42.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.558+0000 7f7e7bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e7c108da0 0x7f7e7c080650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.558+0000 7f7e7bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e7c108da0 0x7f7e7c080650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52028/0 (socket says 172.21.15.70:52028) 2024-04-01T01:42:42.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.558+0000 7f7e7bfff640 1 -- 172.21.15.70:0/699392872 learned_addr learned my addr 172.21.15.70:0/699392872 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:42.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.558+0000 7f7e81566640 1 --2- 172.21.15.70:0/699392872 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e7c109760 0x7f7e7c081690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.559+0000 7f7e80d65640 1 --2- 172.21.15.70:0/699392872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e7c107bb0 0x7f7e7c080110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.559+0000 7f7e7bfff640 1 -- 172.21.15.70:0/699392872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e7c107bb0 msgr2=0x7f7e7c080110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:42.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.559+0000 7f7e7bfff640 1 --2- 172.21.15.70:0/699392872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e7c107bb0 0x7f7e7c080110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.559+0000 7f7e7bfff640 1 -- 172.21.15.70:0/699392872 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e7c109760 msgr2=0x7f7e7c081690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:42.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.559+0000 7f7e7bfff640 1 --2- 172.21.15.70:0/699392872 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e7c109760 0x7f7e7c081690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.559+0000 7f7e7bfff640 1 -- 172.21.15.70:0/699392872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7c081d00 con 0x7f7e7c108da0 2024-04-01T01:42:42.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.559+0000 7f7e81566640 1 --2- 172.21.15.70:0/699392872 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e7c109760 0x7f7e7c081690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:42:42.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.559+0000 7f7e7bfff640 1 --2- 172.21.15.70:0/699392872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e7c108da0 0x7f7e7c080650 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f7e7400ed80 tx=0x7f7e7400c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:42.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.559+0000 7f7e79ffb640 1 -- 172.21.15.70:0/699392872 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7401c070 con 0x7f7e7c108da0 2024-04-01T01:42:42.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.560+0000 7f7e81d67640 1 -- 172.21.15.70:0/699392872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e7c081ff0 con 0x7f7e7c108da0 2024-04-01T01:42:42.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.560+0000 7f7e79ffb640 1 -- 172.21.15.70:0/699392872 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e74017af0 con 0x7f7e7c108da0 2024-04-01T01:42:42.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.560+0000 7f7e79ffb640 1 -- 172.21.15.70:0/699392872 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7400fb70 con 0x7f7e7c108da0 2024-04-01T01:42:42.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.560+0000 7f7e81d67640 1 -- 172.21.15.70:0/699392872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e7c119000 con 0x7f7e7c108da0 2024-04-01T01:42:42.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.561+0000 7f7e81d67640 1 -- 172.21.15.70:0/699392872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e7c068460 con 0x7f7e7c108da0 2024-04-01T01:42:42.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.563+0000 7f7e79ffb640 1 -- 172.21.15.70:0/699392872 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f7e7401f070 con 0x7f7e7c108da0 2024-04-01T01:42:42.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.564+0000 7f7e79ffb640 1 --2- 172.21.15.70:0/699392872 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7e640a2520 0x7f7e640a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:42.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.565+0000 7f7e79ffb640 1 -- 172.21.15.70:0/699392872 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f7e740cee70 con 0x7f7e7c108da0 2024-04-01T01:42:42.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.566+0000 7f7e80d65640 1 --2- 172.21.15.70:0/699392872 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7e640a2520 0x7f7e640a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:42.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.567+0000 7f7e80d65640 1 --2- 172.21.15.70:0/699392872 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7e640a2520 0x7f7e640a49e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0348e0 tx=0x7f7e6c0349b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:42.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.571+0000 7f7e79ffb640 1 -- 172.21.15.70:0/699392872 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7e7409bc00 con 0x7f7e7c108da0 2024-04-01T01:42:42.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.973+0000 7f7e81d67640 1 -- 172.21.15.70:0/699392872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e7c02d520 con 0x7f7e7c108da0 2024-04-01T01:42:42.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.974+0000 7f7e79ffb640 1 -- 172.21.15.70:0/699392872 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7f7e7c02d520 con 0x7f7e7c108da0 2024-04-01T01:42:42.975 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:42:42.975 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:42:42.975 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:42:42.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.977+0000 7f7e81d67640 1 -- 172.21.15.70:0/699392872 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7e640a2520 msgr2=0x7f7e640a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:42.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.978+0000 7f7e81d67640 1 --2- 172.21.15.70:0/699392872 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7e640a2520 0x7f7e640a49e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0348e0 tx=0x7f7e6c0349b0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.978+0000 7f7e81d67640 1 -- 172.21.15.70:0/699392872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e7c108da0 msgr2=0x7f7e7c080650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:42.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.978+0000 7f7e81d67640 1 --2- 172.21.15.70:0/699392872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e7c108da0 0x7f7e7c080650 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f7e7400ed80 tx=0x7f7e7400c6e0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.979+0000 7f7e81d67640 1 -- 172.21.15.70:0/699392872 shutdown_connections 2024-04-01T01:42:42.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.979+0000 7f7e81d67640 1 --2- 172.21.15.70:0/699392872 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e7c109760 0x7f7e7c081690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.979+0000 7f7e81d67640 1 --2- 172.21.15.70:0/699392872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e7c108da0 0x7f7e7c080650 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.979+0000 7f7e81d67640 1 --2- 172.21.15.70:0/699392872 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7e640a2520 0x7f7e640a49e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.979+0000 7f7e81d67640 1 --2- 172.21.15.70:0/699392872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e7c107bb0 0x7f7e7c080110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:42.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.979+0000 7f7e81d67640 1 -- 172.21.15.70:0/699392872 >> 172.21.15.70:0/699392872 conn(0x7f7e7c076b60 msgr2=0x7f7e7c10c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:42.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.979+0000 7f7e81d67640 1 -- 172.21.15.70:0/699392872 shutdown_connections 2024-04-01T01:42:42.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:42.979+0000 7f7e81d67640 1 -- 172.21.15.70:0/699392872 wait complete. 2024-04-01T01:42:42.997 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:42:43.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.188+0000 7f0b1444e640 1 -- 172.21.15.70:0/1136289150 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b0c1090b0 msgr2=0x7f0b0c10fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.188+0000 7f0b1444e640 1 --2- 172.21.15.70:0/1136289150 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b0c1090b0 0x7f0b0c10fa90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0b00009900 tx=0x7f0b0002bc40 comp rx=0 tx=0).stop 2024-04-01T01:42:43.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.188+0000 7f0b1444e640 1 -- 172.21.15.70:0/1136289150 shutdown_connections 2024-04-01T01:42:43.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.188+0000 7f0b1444e640 1 --2- 172.21.15.70:0/1136289150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b0c10ffd0 0x7f0b0c1123c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.189+0000 7f0b1444e640 1 --2- 172.21.15.70:0/1136289150 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b0c1090b0 0x7f0b0c10fa90 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.189+0000 7f0b1444e640 1 --2- 172.21.15.70:0/1136289150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b0c108750 0x7f0b0c108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.189+0000 7f0b1444e640 1 -- 172.21.15.70:0/1136289150 >> 172.21.15.70:0/1136289150 conn(0x7f0b0c06e5b0 msgr2=0x7f0b0c0709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.189+0000 7f0b1444e640 1 -- 172.21.15.70:0/1136289150 shutdown_connections 2024-04-01T01:42:43.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.189+0000 7f0b1444e640 1 -- 172.21.15.70:0/1136289150 wait complete. 2024-04-01T01:42:43.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.190+0000 7f0b1444e640 1 Processor -- start 2024-04-01T01:42:43.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.190+0000 7f0b1444e640 1 -- start start 2024-04-01T01:42:43.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.190+0000 7f0b1444e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b0c108750 0x7f0b0c10b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.191+0000 7f0b1444e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b0c1090b0 0x7f0b0c109a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.191+0000 7f0b1444e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b0c10ffd0 0x7f0b0c109f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.191+0000 7f0b1444e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0c115ac0 con 0x7f0b0c10ffd0 2024-04-01T01:42:43.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.191+0000 7f0b1444e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0c115940 con 0x7f0b0c108750 2024-04-01T01:42:43.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.191+0000 7f0b1444e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0c115c40 con 0x7f0b0c1090b0 2024-04-01T01:42:43.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.191+0000 7f0b119e9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b0c1090b0 0x7f0b0c109a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.191+0000 7f0b119e9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b0c1090b0 0x7f0b0c109a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40846/0 (socket says 172.21.15.70:40846) 2024-04-01T01:42:43.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.191+0000 7f0b119e9640 1 -- 172.21.15.70:0/3825226285 learned_addr learned my addr 172.21.15.70:0/3825226285 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:43.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.191+0000 7f0b121ea640 1 --2- 172.21.15.70:0/3825226285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b0c108750 0x7f0b0c10b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.191+0000 7f0b129eb640 1 --2- 172.21.15.70:0/3825226285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b0c10ffd0 0x7f0b0c109f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.191+0000 7f0b119e9640 1 -- 172.21.15.70:0/3825226285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b0c108750 msgr2=0x7f0b0c10b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.191+0000 7f0b119e9640 1 --2- 172.21.15.70:0/3825226285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b0c108750 0x7f0b0c10b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.191+0000 7f0b119e9640 1 -- 172.21.15.70:0/3825226285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b0c10ffd0 msgr2=0x7f0b0c109f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.191+0000 7f0b119e9640 1 --2- 172.21.15.70:0/3825226285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b0c10ffd0 0x7f0b0c109f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.192+0000 7f0b119e9640 1 -- 172.21.15.70:0/3825226285 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b0c10a7f0 con 0x7f0b0c1090b0 2024-04-01T01:42:43.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.192+0000 7f0b119e9640 1 --2- 172.21.15.70:0/3825226285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b0c1090b0 0x7f0b0c109a20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0b00000c00 tx=0x7f0b000040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.192+0000 7f0aff7fe640 1 -- 172.21.15.70:0/3825226285 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b00043020 con 0x7f0b0c1090b0 2024-04-01T01:42:43.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.192+0000 7f0aff7fe640 1 -- 172.21.15.70:0/3825226285 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b000044c0 con 0x7f0b0c1090b0 2024-04-01T01:42:43.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.192+0000 7f0aff7fe640 1 -- 172.21.15.70:0/3825226285 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b00003c30 con 0x7f0b0c1090b0 2024-04-01T01:42:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.193+0000 7f0b1444e640 1 -- 172.21.15.70:0/3825226285 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b0c1aa360 con 0x7f0b0c1090b0 2024-04-01T01:42:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.193+0000 7f0b1444e640 1 -- 172.21.15.70:0/3825226285 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b0c1aa7a0 con 0x7f0b0c1090b0 2024-04-01T01:42:43.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.197+0000 7f0b17035740 1 Processor -- start 2024-04-01T01:42:43.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.197+0000 7f0aff7fe640 1 -- 172.21.15.70:0/3825226285 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f0b00046020 con 0x7f0b0c1090b0 2024-04-01T01:42:43.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.198+0000 7f0aff7fe640 1 --2- 172.21.15.70:0/3825226285 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0af80a2670 0x7f0af80a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.199+0000 7f0aff7fe640 1 -- 172.21.15.70:0/3825226285 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f0b000f0d00 con 0x7f0b0c1090b0 2024-04-01T01:42:43.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.199+0000 7f0b17035740 1 -- start start 2024-04-01T01:42:43.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.199+0000 7f0b17035740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c810ca0 0x560f8c8110c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.199+0000 7f0b17035740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c811690 0x560f8c816190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.199+0000 7f0b17035740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c816800 0x560f8c818c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.199+0000 7f0b17035740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560f8c76d1b0 con 0x560f8c810ca0 2024-04-01T01:42:43.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.199+0000 7f0b129eb640 1 --2- 172.21.15.70:0/3825226285 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0af80a2670 0x7f0af80a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.199+0000 7f0b17035740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560f8c715c00 con 0x560f8c811690 2024-04-01T01:42:43.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.199+0000 7f0b17035740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560f8c769250 con 0x560f8c816800 2024-04-01T01:42:43.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.199+0000 7f0b129eb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c816800 0x560f8c818c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.199+0000 7f0b121ea640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c810ca0 0x560f8c8110c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.200+0000 7f0b121ea640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c810ca0 0x560f8c8110c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52048/0 (socket says 172.21.15.70:52048) 2024-04-01T01:42:43.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.200+0000 7f0b121ea640 1 -- 172.21.15.70:0/776904897 learned_addr learned my addr 172.21.15.70:0/776904897 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:43.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.200+0000 7f0b119e9640 1 --2- 172.21.15.70:0/776904897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c811690 0x560f8c816190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.203+0000 7f0b119e9640 1 -- 172.21.15.70:0/776904897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c816800 msgr2=0x560f8c818c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.203+0000 7f0b119e9640 1 --2- 172.21.15.70:0/776904897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c816800 0x560f8c818c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.203+0000 7f0b119e9640 1 -- 172.21.15.70:0/776904897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c810ca0 msgr2=0x560f8c8110c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.203+0000 7f0b119e9640 1 --2- 172.21.15.70:0/776904897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c810ca0 0x560f8c8110c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.203+0000 7f0b119e9640 1 -- 172.21.15.70:0/776904897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f8c7353d0 con 0x560f8c811690 2024-04-01T01:42:43.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.203+0000 7f0b129eb640 1 --2- 172.21.15.70:0/776904897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c816800 0x560f8c818c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:42:43.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.204+0000 7f0b119e9640 1 --2- 172.21.15.70:0/776904897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c811690 0x560f8c816190 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f0b000bd090 tx=0x7f0b0002bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c63432d60120203 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.204+0000 7f0b129eb640 1 --2- 172.21.15.70:0/3825226285 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0af80a2670 0x7f0af80a4b30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0b04005c40 tx=0x7f0b040098e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.204+0000 7f0afcbfe640 1 -- 172.21.15.70:0/776904897 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b000bda90 con 0x560f8c811690 2024-04-01T01:42:43.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.204+0000 7f0afcbfe640 1 -- 172.21.15.70:0/776904897 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b000bd390 con 0x560f8c811690 2024-04-01T01:42:43.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.204+0000 7f0afcbfe640 1 -- 172.21.15.70:0/776904897 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b000b9a00 con 0x560f8c811690 2024-04-01T01:42:43.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.205+0000 7f0b17035740 1 -- 172.21.15.70:0/776904897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c811690 msgr2=0x560f8c816190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.205+0000 7f0b17035740 1 --2- 172.21.15.70:0/776904897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c811690 0x560f8c816190 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f0b000bd090 tx=0x7f0b0002bc40 comp rx=0 tx=0).stop 2024-04-01T01:42:43.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.205+0000 7f0b17035740 1 -- 172.21.15.70:0/776904897 shutdown_connections 2024-04-01T01:42:43.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.205+0000 7f0b17035740 1 --2- 172.21.15.70:0/776904897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c816800 0x560f8c818c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.205+0000 7f0b17035740 1 --2- 172.21.15.70:0/776904897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c811690 0x560f8c816190 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.205+0000 7f0b17035740 1 --2- 172.21.15.70:0/776904897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c810ca0 0x560f8c8110c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.205+0000 7f0b17035740 1 -- 172.21.15.70:0/776904897 >> 172.21.15.70:0/776904897 conn(0x560f8c803a00 msgr2=0x560f8c803e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.206+0000 7f0b17035740 1 -- 172.21.15.70:0/776904897 shutdown_connections 2024-04-01T01:42:43.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.206+0000 7f0b17035740 1 -- 172.21.15.70:0/776904897 wait complete. 2024-04-01T01:42:43.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.206+0000 7f0b17035740 1 -- start start 2024-04-01T01:42:43.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.206+0000 7f0b17035740 1 Processor -- start 2024-04-01T01:42:43.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.207+0000 7f0b17035740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c810ca0 0x560f8c81b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.207+0000 7f0b17035740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c811690 0x560f8c81b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.207+0000 7f0b17035740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c816800 0x560f8c822530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.207+0000 7f0b17035740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560f8c819570 con 0x560f8c811690 2024-04-01T01:42:43.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.207+0000 7f0b17035740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560f8c70b470 con 0x560f8c816800 2024-04-01T01:42:43.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.207+0000 7f0b17035740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560f8c8196f0 con 0x560f8c810ca0 2024-04-01T01:42:43.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.207+0000 7f0b17035740 10 client.0 _ll_register_callbacks cb 0x560f8c728cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:43.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.207+0000 7f0b121ea640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c810ca0 0x560f8c81b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.207+0000 7f0b121ea640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c810ca0 0x560f8c81b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40872/0 (socket says 172.21.15.70:40872) 2024-04-01T01:42:43.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.207+0000 7f0b121ea640 1 -- 172.21.15.70:0/3176025612 learned_addr learned my addr 172.21.15.70:0/3176025612 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:43.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.208+0000 7f0b129eb640 1 --2- 172.21.15.70:0/3176025612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c816800 0x560f8c822530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.208+0000 7f0b121ea640 1 -- 172.21.15.70:0/3176025612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c816800 msgr2=0x560f8c822530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.208+0000 7f0b121ea640 1 --2- 172.21.15.70:0/3176025612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c816800 0x560f8c822530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.208+0000 7f0b121ea640 1 -- 172.21.15.70:0/3176025612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c811690 msgr2=0x560f8c81b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.208+0000 7f0b121ea640 1 --2- 172.21.15.70:0/3176025612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c811690 0x560f8c81b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.208+0000 7f0b121ea640 1 -- 172.21.15.70:0/3176025612 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f8c4813f0 con 0x560f8c810ca0 2024-04-01T01:42:43.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.208+0000 7f0b121ea640 1 --2- 172.21.15.70:0/3176025612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c810ca0 0x560f8c81b020 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0b08012740 tx=0x7f0b08012c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46a8e42c5995c7cb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.208+0000 7f0af5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:42:43.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.209+0000 7f0af5ffb640 1 -- 172.21.15.70:0/3176025612 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b08004050 con 0x560f8c810ca0 2024-04-01T01:42:43.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.209+0000 7f0b17035740 1 -- 172.21.15.70:0/3176025612 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b000bd390 con 0x560f8c810ca0 2024-04-01T01:42:43.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.209+0000 7f0af5ffb640 1 -- 172.21.15.70:0/3176025612 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f0b080041f0 con 0x560f8c810ca0 2024-04-01T01:42:43.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.210+0000 7f0af5ffb640 1 -- 172.21.15.70:0/3176025612 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0b08025050 con 0x560f8c810ca0 2024-04-01T01:42:43.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.210+0000 7f0b17035740 10 client.27997 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:43.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.210+0000 7f0b17035740 1 -- 172.21.15.70:0/3176025612 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560f8c54b7d0 con 0x560f8c810ca0 2024-04-01T01:42:43.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.210+0000 7f0af5ffb640 1 -- 172.21.15.70:0/3176025612 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b000bd390 con 0x560f8c810ca0 2024-04-01T01:42:43.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.210+0000 7f0af5ffb640 1 -- 172.21.15.70:0/3176025612 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f0b080038b0 con 0x560f8c810ca0 2024-04-01T01:42:43.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.210+0000 7f0b17035740 10 client.27997 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:43.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.210+0000 7f0b17035740 10 client.27997 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:42:43.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.210+0000 7f0b17035740 20 client.27997 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:43.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.210+0000 7f0b17035740 1 --2- 172.21.15.70:0/3176025612 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x560f8c826f00 0x560f8c8293c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.210+0000 7f0b17035740 4 client.27997 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:43.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.210+0000 7f0b17035740 1 -- 172.21.15.70:0/3176025612 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f8c77a6f0 con 0x560f8c826f00 2024-04-01T01:42:43.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.211+0000 7f0b119e9640 1 --2- 172.21.15.70:0/3176025612 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x560f8c826f00 0x560f8c8293c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.212+0000 7f0b119e9640 1 --2- 172.21.15.70:0/3176025612 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x560f8c826f00 0x560f8c8293c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c08512febf330f08 server_cookie=ee75edfec6e40f73 in_seq=0 out_seq=0 2024-04-01T01:42:43.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.212+0000 7f0af5ffb640 10 client.27997 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:42:43.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.213+0000 7f0af5ffb640 1 -- 172.21.15.70:0/3176025612 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560f8c77a6f0 con 0x560f8c826f00 2024-04-01T01:42:43.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.213+0000 7f0af5ffb640 10 client.27997 handle_command_reply: tid=0 2024-04-01T01:42:43.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.213+0000 7f0b17035740 1 client.27997 shutdown 2024-04-01T01:42:43.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.213+0000 7f0b17035740 1 -- 172.21.15.70:0/3176025612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c810ca0 msgr2=0x560f8c81b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.213+0000 7f0b17035740 1 --2- 172.21.15.70:0/3176025612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c810ca0 0x560f8c81b020 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0b08012740 tx=0x7f0b08012c00 comp rx=0 tx=0).stop 2024-04-01T01:42:43.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.213+0000 7f0b17035740 1 -- 172.21.15.70:0/3176025612 shutdown_connections 2024-04-01T01:42:43.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.213+0000 7f0b17035740 1 --2- 172.21.15.70:0/3176025612 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x560f8c826f00 0x560f8c8293c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.213+0000 7f0b17035740 1 --2- 172.21.15.70:0/3176025612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c816800 0x560f8c822530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.213+0000 7f0b17035740 1 --2- 172.21.15.70:0/3176025612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c811690 0x560f8c81b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.213+0000 7f0b17035740 1 --2- 172.21.15.70:0/3176025612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c810ca0 0x560f8c81b020 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.213+0000 7f0b17035740 1 -- 172.21.15.70:0/3176025612 >> 172.21.15.70:0/3176025612 conn(0x560f8c803a00 msgr2=0x560f8c802d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.213+0000 7f0af5ffb640 0 client.27997 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:42:43.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.214+0000 7f0b17035740 1 -- 172.21.15.70:0/3176025612 shutdown_connections 2024-04-01T01:42:43.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.214+0000 7f0b17035740 1 -- 172.21.15.70:0/3176025612 wait complete. 2024-04-01T01:42:43.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.214+0000 7f0b17035740 20 client.27997 trim_cache size 0 max 16384 2024-04-01T01:42:43.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.242+0000 7f0b17035740 1 Processor -- start 2024-04-01T01:42:43.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.242+0000 7f0b17035740 1 -- start start 2024-04-01T01:42:43.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.242+0000 7f0b17035740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c810ca0 0x560f8c81a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.242+0000 7f0b17035740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c811690 0x560f8c8224f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.242+0000 7f0b17035740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c816800 0x560f8c81ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.242+0000 7f0b17035740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0c1b0af0 con 0x560f8c816800 2024-04-01T01:42:43.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.242+0000 7f0b17035740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0c1b2630 con 0x560f8c811690 2024-04-01T01:42:43.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.242+0000 7f0b17035740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0c1b11c0 con 0x560f8c810ca0 2024-04-01T01:42:43.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.244+0000 7f0b119e9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c811690 0x560f8c8224f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.244+0000 7f0b119e9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c811690 0x560f8c8224f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47498/0 (socket says 172.21.15.70:47498) 2024-04-01T01:42:43.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.244+0000 7f0b119e9640 1 -- 172.21.15.70:0/3839000092 learned_addr learned my addr 172.21.15.70:0/3839000092 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:43.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.244+0000 7f0b121ea640 1 --2- 172.21.15.70:0/3839000092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c810ca0 0x560f8c81a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.244+0000 7f0b119e9640 1 -- 172.21.15.70:0/3839000092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c810ca0 msgr2=0x560f8c81a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.244+0000 7f0b119e9640 1 --2- 172.21.15.70:0/3839000092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c810ca0 0x560f8c81a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.244+0000 7f0b119e9640 1 -- 172.21.15.70:0/3839000092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c816800 msgr2=0x560f8c81ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.244+0000 7f0b119e9640 1 --2- 172.21.15.70:0/3839000092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c816800 0x560f8c81ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.244+0000 7f0b119e9640 1 -- 172.21.15.70:0/3839000092 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f8c811a70 con 0x560f8c811690 2024-04-01T01:42:43.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.245+0000 7f0b119e9640 1 --2- 172.21.15.70:0/3839000092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c811690 0x560f8c8224f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0b0002c6b0 tx=0x7f0b00005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b864412c1a57250c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.245+0000 7f0af7fff640 1 -- 172.21.15.70:0/3839000092 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b000b9630 con 0x560f8c811690 2024-04-01T01:42:43.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.245+0000 7f0af7fff640 1 -- 172.21.15.70:0/3839000092 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b000bda90 con 0x560f8c811690 2024-04-01T01:42:43.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.246+0000 7f0b17035740 1 -- 172.21.15.70:0/3839000092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c811690 msgr2=0x560f8c8224f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.246+0000 7f0b17035740 1 --2- 172.21.15.70:0/3839000092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c811690 0x560f8c8224f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0b0002c6b0 tx=0x7f0b00005c90 comp rx=0 tx=0).stop 2024-04-01T01:42:43.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.246+0000 7f0b17035740 1 -- 172.21.15.70:0/3839000092 shutdown_connections 2024-04-01T01:42:43.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.246+0000 7f0b17035740 1 --2- 172.21.15.70:0/3839000092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c816800 0x560f8c81ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.246+0000 7f0b17035740 1 --2- 172.21.15.70:0/3839000092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c811690 0x560f8c8224f0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.246+0000 7f0b17035740 1 --2- 172.21.15.70:0/3839000092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c810ca0 0x560f8c81a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.246+0000 7f0b17035740 1 -- 172.21.15.70:0/3839000092 >> 172.21.15.70:0/3839000092 conn(0x560f8c803a00 msgr2=0x560f8c8074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.247+0000 7f0b17035740 1 -- 172.21.15.70:0/3839000092 shutdown_connections 2024-04-01T01:42:43.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.247+0000 7f0b17035740 1 -- 172.21.15.70:0/3839000092 wait complete. 2024-04-01T01:42:43.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.247+0000 7f0b17035740 1 -- start start 2024-04-01T01:42:43.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.247+0000 7f0b17035740 1 Processor -- start 2024-04-01T01:42:43.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.248+0000 7f0b17035740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c810ca0 0x560f8c81fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.248+0000 7f0b17035740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c811690 0x560f8c820240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.248+0000 7f0b17035740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c816800 0x560f8c821250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.248+0000 7f0b17035740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0c1b1f60 con 0x560f8c816800 2024-04-01T01:42:43.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.248+0000 7f0b17035740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0c1b1890 con 0x560f8c810ca0 2024-04-01T01:42:43.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.248+0000 7f0b17035740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0c1aead0 con 0x560f8c811690 2024-04-01T01:42:43.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.248+0000 7f0b17035740 10 client.0 _ll_register_callbacks cb 0x7f0b08009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:43.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:42 smithi189 ceph-mon[35753]: pgmap v636: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.0 MiB/s wr, 804 op/s 2024-04-01T01:42:43.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.248+0000 7f0b129eb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c816800 0x560f8c821250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.248+0000 7f0b119e9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c811690 0x560f8c820240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.248+0000 7f0b129eb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c816800 0x560f8c821250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52060/0 (socket says 172.21.15.70:52060) 2024-04-01T01:42:43.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.248+0000 7f0b129eb640 1 -- 172.21.15.70:0/2433496151 learned_addr learned my addr 172.21.15.70:0/2433496151 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:43.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.248+0000 7f0b121ea640 1 --2- 172.21.15.70:0/2433496151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c810ca0 0x560f8c81fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.249+0000 7f0b129eb640 1 -- 172.21.15.70:0/2433496151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c811690 msgr2=0x560f8c820240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.249+0000 7f0b129eb640 1 --2- 172.21.15.70:0/2433496151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c811690 0x560f8c820240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.249+0000 7f0b129eb640 1 -- 172.21.15.70:0/2433496151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c810ca0 msgr2=0x560f8c81fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.249+0000 7f0b129eb640 1 --2- 172.21.15.70:0/2433496151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c810ca0 0x560f8c81fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.249+0000 7f0b129eb640 1 -- 172.21.15.70:0/2433496151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f8c821840 con 0x560f8c816800 2024-04-01T01:42:43.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.249+0000 7f0b119e9640 1 --2- 172.21.15.70:0/2433496151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c811690 0x560f8c820240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:42:43.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.249+0000 7f0b121ea640 1 --2- 172.21.15.70:0/2433496151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c810ca0 0x560f8c81fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:42:43.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.249+0000 7f0b129eb640 1 --2- 172.21.15.70:0/2433496151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c816800 0x560f8c821250 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009350 tx=0x7f0b04009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec0f9d7a3a42ed3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.249+0000 7f0af5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:42:43.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.249+0000 7f0af5ffb640 1 -- 172.21.15.70:0/2433496151 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0403d060 con 0x560f8c816800 2024-04-01T01:42:43.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.250+0000 7f0af5ffb640 1 -- 172.21.15.70:0/2433496151 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f0b04038400 con 0x560f8c816800 2024-04-01T01:42:43.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.250+0000 7f0b17035740 1 -- 172.21.15.70:0/2433496151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b000bda90 con 0x560f8c816800 2024-04-01T01:42:43.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.250+0000 7f0af5ffb640 1 -- 172.21.15.70:0/2433496151 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0b04018070 con 0x560f8c816800 2024-04-01T01:42:43.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.250+0000 7f0b17035740 10 client.18189 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:43.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.250+0000 7f0b17035740 1 -- 172.21.15.70:0/2433496151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560f8c808200 con 0x560f8c816800 2024-04-01T01:42:43.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.251+0000 7f0af5ffb640 1 -- 172.21.15.70:0/2433496151 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0b000bda90 con 0x560f8c816800 2024-04-01T01:42:43.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.251+0000 7f0af5ffb640 1 -- 172.21.15.70:0/2433496151 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f0b0401e5b0 con 0x560f8c816800 2024-04-01T01:42:43.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.251+0000 7f0b17035740 10 client.18189 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:43.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.251+0000 7f0b17035740 10 client.18189 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:42:43.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.251+0000 7f0b17035740 20 client.18189 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:43.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.251+0000 7f0b17035740 1 --2- 172.21.15.70:0/2433496151 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x560f8c808c30 0x560f8c80b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.251+0000 7f0b17035740 4 client.18189 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:42:43.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.251+0000 7f0b17035740 1 -- 172.21.15.70:0/2433496151 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560f8c774a40 con 0x560f8c808c30 2024-04-01T01:42:43.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:42 smithi070 ceph-mon[32179]: pgmap v636: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.0 MiB/s wr, 804 op/s 2024-04-01T01:42:43.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.257+0000 7f0b121ea640 1 --2- 172.21.15.70:0/2433496151 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x560f8c808c30 0x560f8c80b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.257+0000 7f0b121ea640 1 --2- 172.21.15.70:0/2433496151 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x560f8c808c30 0x560f8c80b050 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f6ce2a83575d63cf server_cookie=6747e8ef26fd5445 in_seq=0 out_seq=0 2024-04-01T01:42:43.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.258+0000 7f0af5ffb640 10 client.18189 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:42:43.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.258+0000 7f0af5ffb640 1 -- 172.21.15.70:0/2433496151 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560f8c774a40 con 0x560f8c808c30 2024-04-01T01:42:43.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.258+0000 7f0af5ffb640 10 client.18189 handle_command_reply: tid=0 2024-04-01T01:42:43.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.258+0000 7f0b17035740 1 client.18189 shutdown 2024-04-01T01:42:43.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.258+0000 7f0b17035740 1 -- 172.21.15.70:0/2433496151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c816800 msgr2=0x560f8c821250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.258+0000 7f0b17035740 1 --2- 172.21.15.70:0/2433496151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c816800 0x560f8c821250 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009350 tx=0x7f0b04009b90 comp rx=0 tx=0).stop 2024-04-01T01:42:43.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.259+0000 7f0b17035740 1 -- 172.21.15.70:0/2433496151 shutdown_connections 2024-04-01T01:42:43.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.259+0000 7f0b17035740 1 --2- 172.21.15.70:0/2433496151 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x560f8c808c30 0x560f8c80b050 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.259+0000 7f0b17035740 1 --2- 172.21.15.70:0/2433496151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f8c816800 0x560f8c821250 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.259+0000 7f0b17035740 1 --2- 172.21.15.70:0/2433496151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8c811690 0x560f8c820240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.259+0000 7f0b17035740 1 --2- 172.21.15.70:0/2433496151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f8c810ca0 0x560f8c81fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.259+0000 7f0b17035740 1 -- 172.21.15.70:0/2433496151 >> 172.21.15.70:0/2433496151 conn(0x560f8c803a00 msgr2=0x560f8c8024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.259+0000 7f0af5ffb640 0 client.18189 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:42:43.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.259+0000 7f0b17035740 1 -- 172.21.15.70:0/2433496151 shutdown_connections 2024-04-01T01:42:43.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.259+0000 7f0b17035740 1 -- 172.21.15.70:0/2433496151 wait complete. 2024-04-01T01:42:43.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.259+0000 7f0b17035740 20 client.18189 trim_cache size 0 max 16384 2024-04-01T01:42:43.260 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:42:43.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.262+0000 7f0b1444e640 1 -- 172.21.15.70:0/3825226285 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0af80a2670 msgr2=0x7f0af80a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.262+0000 7f0b1444e640 1 --2- 172.21.15.70:0/3825226285 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0af80a2670 0x7f0af80a4b30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0b04005c40 tx=0x7f0b040098e0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.262+0000 7f0b1444e640 1 -- 172.21.15.70:0/3825226285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b0c1090b0 msgr2=0x7f0b0c109a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.262+0000 7f0b1444e640 1 --2- 172.21.15.70:0/3825226285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b0c1090b0 0x7f0b0c109a20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0b00000c00 tx=0x7f0b000040c0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.262+0000 7f0b1444e640 1 -- 172.21.15.70:0/3825226285 shutdown_connections 2024-04-01T01:42:43.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.262+0000 7f0b1444e640 1 --2- 172.21.15.70:0/3825226285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b0c10ffd0 0x7f0b0c109f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.262+0000 7f0b1444e640 1 --2- 172.21.15.70:0/3825226285 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0af80a2670 0x7f0af80a4b30 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.262+0000 7f0b1444e640 1 --2- 172.21.15.70:0/3825226285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b0c1090b0 0x7f0b0c109a20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.262+0000 7f0b1444e640 1 --2- 172.21.15.70:0/3825226285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b0c108750 0x7f0b0c10b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.262+0000 7f0b1444e640 1 -- 172.21.15.70:0/3825226285 >> 172.21.15.70:0/3825226285 conn(0x7f0b0c06e5b0 msgr2=0x7f0b0c06f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.262+0000 7f0b1444e640 1 -- 172.21.15.70:0/3825226285 shutdown_connections 2024-04-01T01:42:43.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.262+0000 7f0b1444e640 1 -- 172.21.15.70:0/3825226285 wait complete. 2024-04-01T01:42:43.282 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:42:43.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:42 smithi081 ceph-mon[35142]: pgmap v636: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.0 MiB/s wr, 804 op/s 2024-04-01T01:42:43.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.475+0000 7fe84a5a5640 1 -- 172.21.15.70:0/1816022311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe844072b60 msgr2=0x7fe844072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.475+0000 7fe84a5a5640 1 --2- 172.21.15.70:0/1816022311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe844072b60 0x7fe844072f80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe840009be0 tx=0x7fe84002bf30 comp rx=0 tx=0).stop 2024-04-01T01:42:43.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.477+0000 7fe84a5a5640 1 -- 172.21.15.70:0/1816022311 shutdown_connections 2024-04-01T01:42:43.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.477+0000 7fe84a5a5640 1 --2- 172.21.15.70:0/1816022311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8441097e0 0x7fe84410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.477+0000 7fe84a5a5640 1 --2- 172.21.15.70:0/1816022311 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe844108e20 0x7fe8441092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.477+0000 7fe84a5a5640 1 --2- 172.21.15.70:0/1816022311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe844072b60 0x7fe844072f80 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.477+0000 7fe84a5a5640 1 -- 172.21.15.70:0/1816022311 >> 172.21.15.70:0/1816022311 conn(0x7fe84406e630 msgr2=0x7fe844070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.477+0000 7fe84a5a5640 1 -- 172.21.15.70:0/1816022311 shutdown_connections 2024-04-01T01:42:43.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.478+0000 7fe84a5a5640 1 -- 172.21.15.70:0/1816022311 wait complete. 2024-04-01T01:42:43.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.479+0000 7fe84a5a5640 1 Processor -- start 2024-04-01T01:42:43.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.479+0000 7fe84a5a5640 1 -- start start 2024-04-01T01:42:43.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.479+0000 7fe84a5a5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe844072b60 0x7fe844114e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.479+0000 7fe84a5a5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe844108e20 0x7fe8441153b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.479+0000 7fe84a5a5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8441097e0 0x7fe8441b6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.479+0000 7fe84a5a5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe84411c020 con 0x7fe844108e20 2024-04-01T01:42:43.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.479+0000 7fe84a5a5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe84411bea0 con 0x7fe844072b60 2024-04-01T01:42:43.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.479+0000 7fe84a5a5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe84411c1a0 con 0x7fe8441097e0 2024-04-01T01:42:43.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.480+0000 7fe849da4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8441097e0 0x7fe8441b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.480+0000 7fe849da4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8441097e0 0x7fe8441b6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40902/0 (socket says 172.21.15.70:40902) 2024-04-01T01:42:43.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.480+0000 7fe849da4640 1 -- 172.21.15.70:0/1895995921 learned_addr learned my addr 172.21.15.70:0/1895995921 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:43.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.480+0000 7fe848da2640 1 --2- 172.21.15.70:0/1895995921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe844108e20 0x7fe8441153b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.480+0000 7fe8495a3640 1 --2- 172.21.15.70:0/1895995921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe844072b60 0x7fe844114e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.481+0000 7fe848da2640 1 -- 172.21.15.70:0/1895995921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8441097e0 msgr2=0x7fe8441b6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.481+0000 7fe848da2640 1 --2- 172.21.15.70:0/1895995921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8441097e0 0x7fe8441b6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.481+0000 7fe848da2640 1 -- 172.21.15.70:0/1895995921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe844072b60 msgr2=0x7fe844114e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.481+0000 7fe848da2640 1 --2- 172.21.15.70:0/1895995921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe844072b60 0x7fe844114e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.482+0000 7fe848da2640 1 -- 172.21.15.70:0/1895995921 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8441b6c70 con 0x7fe844108e20 2024-04-01T01:42:43.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.482+0000 7fe848da2640 1 --2- 172.21.15.70:0/1895995921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe844108e20 0x7fe8441153b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe83400d6d0 tx=0x7fe83400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.482+0000 7fe8495a3640 1 --2- 172.21.15.70:0/1895995921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe844072b60 0x7fe844114e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:42:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.482+0000 7fe849da4640 1 --2- 172.21.15.70:0/1895995921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8441097e0 0x7fe8441b6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:42:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.483+0000 7fe83a7fc640 1 -- 172.21.15.70:0/1895995921 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe834017070 con 0x7fe844108e20 2024-04-01T01:42:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.483+0000 7fe84a5a5640 1 -- 172.21.15.70:0/1895995921 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8441b6e40 con 0x7fe844108e20 2024-04-01T01:42:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.483+0000 7fe84a5a5640 1 -- 172.21.15.70:0/1895995921 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8441b7330 con 0x7fe844108e20 2024-04-01T01:42:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.483+0000 7fe83a7fc640 1 -- 172.21.15.70:0/1895995921 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe834010460 con 0x7fe844108e20 2024-04-01T01:42:43.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.483+0000 7fe83a7fc640 1 -- 172.21.15.70:0/1895995921 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83400f620 con 0x7fe844108e20 2024-04-01T01:42:43.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.487+0000 7fe83a7fc640 1 -- 172.21.15.70:0/1895995921 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fe83401a070 con 0x7fe844108e20 2024-04-01T01:42:43.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.488+0000 7fe84e3ef740 1 Processor -- start 2024-04-01T01:42:43.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.488+0000 7fe83a7fc640 1 --2- 172.21.15.70:0/1895995921 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8300a25f0 0x7fe8300a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.488+0000 7fe83a7fc640 1 -- 172.21.15.70:0/1895995921 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe8340cf070 con 0x7fe844108e20 2024-04-01T01:42:43.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.489+0000 7fe849da4640 1 --2- 172.21.15.70:0/1895995921 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8300a25f0 0x7fe8300a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.489+0000 7fe84e3ef740 1 -- start start 2024-04-01T01:42:43.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.489+0000 7fe84e3ef740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce19a6a0 0x5599ce19aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.489+0000 7fe84e3ef740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19b090 0x5599ce19fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.489+0000 7fe84e3ef740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce1a0200 0x5599ce1a2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.489+0000 7fe84e3ef740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5599ce0ff1b0 con 0x5599ce19a6a0 2024-04-01T01:42:43.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.489+0000 7fe84e3ef740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5599ce0a7c00 con 0x5599ce19b090 2024-04-01T01:42:43.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.489+0000 7fe84e3ef740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5599ce0fb250 con 0x5599ce1a0200 2024-04-01T01:42:43.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.490+0000 7fe848da2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19b090 0x5599ce19fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.490+0000 7fe848da2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19b090 0x5599ce19fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47546/0 (socket says 172.21.15.70:47546) 2024-04-01T01:42:43.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.490+0000 7fe848da2640 1 -- 172.21.15.70:0/725553448 learned_addr learned my addr 172.21.15.70:0/725553448 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:43.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.490+0000 7fe8495a3640 1 --2- 172.21.15.70:0/725553448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce19a6a0 0x5599ce19aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.490+0000 7fe848da2640 1 -- 172.21.15.70:0/725553448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce1a0200 msgr2=0x5599ce1a2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.490+0000 7fe849da4640 1 --2- 172.21.15.70:0/725553448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce1a0200 0x5599ce1a2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.490+0000 7fe848da2640 1 --2- 172.21.15.70:0/725553448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce1a0200 0x5599ce1a2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.490+0000 7fe848da2640 1 -- 172.21.15.70:0/725553448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce19a6a0 msgr2=0x5599ce19aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.490+0000 7fe848da2640 1 --2- 172.21.15.70:0/725553448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce19a6a0 0x5599ce19aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.490+0000 7fe848da2640 1 -- 172.21.15.70:0/725553448 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599ce0c73d0 con 0x5599ce19b090 2024-04-01T01:42:43.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.490+0000 7fe8495a3640 1 --2- 172.21.15.70:0/725553448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce19a6a0 0x5599ce19aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:42:43.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.490+0000 7fe848da2640 1 --2- 172.21.15.70:0/725553448 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19b090 0x5599ce19fb90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe834009610 tx=0x7fe83409be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bedd056445adace server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.490+0000 7fe849da4640 1 --2- 172.21.15.70:0/725553448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce1a0200 0x5599ce1a2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:42:43.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.491+0000 7fe82f7fe640 1 -- 172.21.15.70:0/725553448 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe834021050 con 0x5599ce19b090 2024-04-01T01:42:43.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.491+0000 7fe82f7fe640 1 -- 172.21.15.70:0/725553448 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8340a4110 con 0x5599ce19b090 2024-04-01T01:42:43.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.491+0000 7fe82f7fe640 1 -- 172.21.15.70:0/725553448 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8340977c0 con 0x5599ce19b090 2024-04-01T01:42:43.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.491+0000 7fe849da4640 1 --2- 172.21.15.70:0/1895995921 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8300a25f0 0x7fe8300a4ab0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0122c0 tx=0x7fe83c0126c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.493+0000 7fe84e3ef740 1 -- 172.21.15.70:0/725553448 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19b090 msgr2=0x5599ce19fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.493+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/725553448 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19b090 0x5599ce19fb90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe834009610 tx=0x7fe83409be00 comp rx=0 tx=0).stop 2024-04-01T01:42:43.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.494+0000 7fe84e3ef740 1 -- 172.21.15.70:0/725553448 shutdown_connections 2024-04-01T01:42:43.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.494+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/725553448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce1a0200 0x5599ce1a2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.494+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/725553448 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19b090 0x5599ce19fb90 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.494+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/725553448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce19a6a0 0x5599ce19aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.494+0000 7fe84e3ef740 1 -- 172.21.15.70:0/725553448 >> 172.21.15.70:0/725553448 conn(0x5599ce195c10 msgr2=0x5599ce196010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.495+0000 7fe84e3ef740 1 -- 172.21.15.70:0/725553448 shutdown_connections 2024-04-01T01:42:43.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.495+0000 7fe84e3ef740 1 -- 172.21.15.70:0/725553448 wait complete. 2024-04-01T01:42:43.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.496+0000 7fe84e3ef740 1 -- start start 2024-04-01T01:42:43.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.496+0000 7fe84e3ef740 1 Processor -- start 2024-04-01T01:42:43.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.496+0000 7fe84e3ef740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 0x5599ce1a4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.496+0000 7fe84e3ef740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce19b090 0x5599ce1a5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.496+0000 7fe84e3ef740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce1a0200 0x5599ce1ac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.496+0000 7fe84e3ef740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5599ce1a31b0 con 0x5599ce19b090 2024-04-01T01:42:43.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.496+0000 7fe84e3ef740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5599ce09d470 con 0x5599ce19a6a0 2024-04-01T01:42:43.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.496+0000 7fe84e3ef740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5599ce1a3330 con 0x5599ce1a0200 2024-04-01T01:42:43.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.496+0000 7fe84e3ef740 10 client.0 _ll_register_callbacks cb 0x5599ce0bacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:43.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.497+0000 7fe849da4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce1a0200 0x5599ce1ac0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.497+0000 7fe849da4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce1a0200 0x5599ce1ac0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40908/0 (socket says 172.21.15.70:40908) 2024-04-01T01:42:43.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.497+0000 7fe849da4640 1 -- 172.21.15.70:0/718433456 learned_addr learned my addr 172.21.15.70:0/718433456 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:43.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.497+0000 7fe8495a3640 1 --2- 172.21.15.70:0/718433456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 0x5599ce1a4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.497+0000 7fe848da2640 1 --2- 172.21.15.70:0/718433456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce19b090 0x5599ce1a5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.497+0000 7fe8495a3640 1 -- 172.21.15.70:0/718433456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce1a0200 msgr2=0x5599ce1ac0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.497+0000 7fe8495a3640 1 --2- 172.21.15.70:0/718433456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce1a0200 0x5599ce1ac0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.497+0000 7fe8495a3640 1 -- 172.21.15.70:0/718433456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce19b090 msgr2=0x5599ce1a5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.497+0000 7fe8495a3640 1 --2- 172.21.15.70:0/718433456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce19b090 0x5599ce1a5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.497+0000 7fe8495a3640 1 -- 172.21.15.70:0/718433456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599cde133f0 con 0x5599ce19a6a0 2024-04-01T01:42:43.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.498+0000 7fe8495a3640 1 --2- 172.21.15.70:0/718433456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 0x5599ce1a4b20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fe840036040 tx=0x7fe84002cb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80bef0686c3905aa server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.498+0000 7fe82cff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:43.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.498+0000 7fe82cff9640 1 -- 172.21.15.70:0/718433456 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe840033e20 con 0x5599ce19a6a0 2024-04-01T01:42:43.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.498+0000 7fe84e3ef740 1 -- 172.21.15.70:0/718433456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8340a4110 con 0x5599ce19a6a0 2024-04-01T01:42:43.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.498+0000 7fe82cff9640 1 -- 172.21.15.70:0/718433456 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe840034430 con 0x5599ce19a6a0 2024-04-01T01:42:43.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.498+0000 7fe82cff9640 1 -- 172.21.15.70:0/718433456 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe840034630 con 0x5599ce19a6a0 2024-04-01T01:42:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.498+0000 7fe84e3ef740 10 client.28196 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.498+0000 7fe84e3ef740 1 -- 172.21.15.70:0/718433456 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5599cdedd7d0 con 0x5599ce19a6a0 2024-04-01T01:42:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.499+0000 7fe82cff9640 1 -- 172.21.15.70:0/718433456 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8340a4110 con 0x5599ce19a6a0 2024-04-01T01:42:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.499+0000 7fe82cff9640 1 -- 172.21.15.70:0/718433456 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe84003d990 con 0x5599ce19a6a0 2024-04-01T01:42:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.499+0000 7fe84e3ef740 10 client.28196 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.499+0000 7fe84e3ef740 10 client.28196 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:42:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.499+0000 7fe84e3ef740 20 client.28196 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.499+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/718433456 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5599ce1b09e0 0x5599ce1b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.499+0000 7fe84e3ef740 4 client.28196 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.499+0000 7fe84e3ef740 1 -- 172.21.15.70:0/718433456 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599ce10c6f0 con 0x5599ce1b09e0 2024-04-01T01:42:43.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.502+0000 7fe848da2640 1 --2- 172.21.15.70:0/718433456 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5599ce1b09e0 0x5599ce1b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.503+0000 7fe848da2640 1 --2- 172.21.15.70:0/718433456 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5599ce1b09e0 0x5599ce1b2e80 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7d1086ed9bd49354 server_cookie=e175c637707d2a4c in_seq=0 out_seq=0 2024-04-01T01:42:43.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.503+0000 7fe82cff9640 10 client.28196 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:42:43.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.504+0000 7fe82cff9640 1 -- 172.21.15.70:0/718433456 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5599ce10c6f0 con 0x5599ce1b09e0 2024-04-01T01:42:43.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.504+0000 7fe82cff9640 10 client.28196 handle_command_reply: tid=0 2024-04-01T01:42:43.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.504+0000 7fe84e3ef740 1 client.28196 shutdown 2024-04-01T01:42:43.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.505+0000 7fe84e3ef740 1 -- 172.21.15.70:0/718433456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 msgr2=0x5599ce1a4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.505+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/718433456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 0x5599ce1a4b20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fe840036040 tx=0x7fe84002cb00 comp rx=0 tx=0).stop 2024-04-01T01:42:43.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.505+0000 7fe84e3ef740 1 -- 172.21.15.70:0/718433456 shutdown_connections 2024-04-01T01:42:43.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.505+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/718433456 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5599ce1b09e0 0x5599ce1b2e80 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.505+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/718433456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce1a0200 0x5599ce1ac0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.505+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/718433456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce19b090 0x5599ce1a5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.505+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/718433456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 0x5599ce1a4b20 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.505+0000 7fe84e3ef740 1 -- 172.21.15.70:0/718433456 >> 172.21.15.70:0/718433456 conn(0x5599ce195c10 msgr2=0x5599ce194f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.505+0000 7fe82cff9640 0 client.28196 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:42:43.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.505+0000 7fe84e3ef740 1 -- 172.21.15.70:0/718433456 shutdown_connections 2024-04-01T01:42:43.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.505+0000 7fe84e3ef740 1 -- 172.21.15.70:0/718433456 wait complete. 2024-04-01T01:42:43.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.505+0000 7fe84e3ef740 20 client.28196 trim_cache size 0 max 16384 2024-04-01T01:42:43.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.532+0000 7fe84e3ef740 1 Processor -- start 2024-04-01T01:42:43.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.532+0000 7fe84e3ef740 1 -- start start 2024-04-01T01:42:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.533+0000 7fe84e3ef740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 0x5599ce194700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.533+0000 7fe84e3ef740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce19b090 0x5599ce194c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.533+0000 7fe84e3ef740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce1a0200 0x5599ce195180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.533+0000 7fe84e3ef740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe82800fd90 con 0x5599ce1a0200 2024-04-01T01:42:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.533+0000 7fe84e3ef740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe8280118d0 con 0x5599ce19a6a0 2024-04-01T01:42:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.533+0000 7fe849da4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce1a0200 0x5599ce195180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.533+0000 7fe849da4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce1a0200 0x5599ce195180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52106/0 (socket says 172.21.15.70:52106) 2024-04-01T01:42:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.534+0000 7fe849da4640 1 -- 172.21.15.70:0/2520505949 learned_addr learned my addr 172.21.15.70:0/2520505949 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.534+0000 7fe84e3ef740 1 -- 172.21.15.70:0/2520505949 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe828010460 con 0x5599ce19b090 2024-04-01T01:42:43.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.534+0000 7fe8495a3640 1 --2- 172.21.15.70:0/2520505949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 0x5599ce194700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.534+0000 7fe849da4640 1 -- 172.21.15.70:0/2520505949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce19b090 msgr2=0x5599ce194c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:42:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.534+0000 7fe849da4640 1 --2- 172.21.15.70:0/2520505949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce19b090 0x5599ce194c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.535+0000 7fe849da4640 1 -- 172.21.15.70:0/2520505949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 msgr2=0x5599ce194700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.535+0000 7fe849da4640 1 --2- 172.21.15.70:0/2520505949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 0x5599ce194700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.535+0000 7fe849da4640 1 -- 172.21.15.70:0/2520505949 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599ce19f7e0 con 0x5599ce1a0200 2024-04-01T01:42:43.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.535+0000 7fe849da4640 1 --2- 172.21.15.70:0/2520505949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce1a0200 0x5599ce195180 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0156f0 tx=0x7fe83c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0b350de3653511 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.535+0000 7fe8495a3640 1 --2- 172.21.15.70:0/2520505949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 0x5599ce194700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:42:43.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.536+0000 7fe82effd640 1 -- 172.21.15.70:0/2520505949 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c01e040 con 0x5599ce1a0200 2024-04-01T01:42:43.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.536+0000 7fe82effd640 1 -- 172.21.15.70:0/2520505949 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe83c029070 con 0x5599ce1a0200 2024-04-01T01:42:43.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.536+0000 7fe82effd640 1 -- 172.21.15.70:0/2520505949 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c024bc0 con 0x5599ce1a0200 2024-04-01T01:42:43.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.537+0000 7fe84e3ef740 1 -- 172.21.15.70:0/2520505949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce1a0200 msgr2=0x5599ce195180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.537+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/2520505949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce1a0200 0x5599ce195180 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0156f0 tx=0x7fe83c019000 comp rx=0 tx=0).stop 2024-04-01T01:42:43.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.537+0000 7fe84e3ef740 1 -- 172.21.15.70:0/2520505949 shutdown_connections 2024-04-01T01:42:43.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.537+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/2520505949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce1a0200 0x5599ce195180 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.537+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/2520505949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce19b090 0x5599ce194c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.537+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/2520505949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 0x5599ce194700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.537+0000 7fe84e3ef740 1 -- 172.21.15.70:0/2520505949 >> 172.21.15.70:0/2520505949 conn(0x5599ce195c10 msgr2=0x5599ce195ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.537+0000 7fe84e3ef740 1 -- 172.21.15.70:0/2520505949 shutdown_connections 2024-04-01T01:42:43.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.537+0000 7fe84e3ef740 1 -- 172.21.15.70:0/2520505949 wait complete. 2024-04-01T01:42:43.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.538+0000 7fe84e3ef740 1 -- start start 2024-04-01T01:42:43.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.538+0000 7fe84e3ef740 1 Processor -- start 2024-04-01T01:42:43.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.538+0000 7fe84e3ef740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 0x5599ce1a9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.538+0000 7fe84e3ef740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce19b090 0x5599ce1a9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.538+0000 7fe84e3ef740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce1a0200 0x5599ce1aa710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.538+0000 7fe84e3ef740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe828011200 con 0x5599ce1a0200 2024-04-01T01:42:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.538+0000 7fe84e3ef740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe828010b30 con 0x5599ce19a6a0 2024-04-01T01:42:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.538+0000 7fe84e3ef740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe82800dd70 con 0x5599ce19b090 2024-04-01T01:42:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.538+0000 7fe84e3ef740 10 client.0 _ll_register_callbacks cb 0x7fe840005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.539+0000 7fe8495a3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 0x5599ce1a9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.539+0000 7fe8495a3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 0x5599ce1a9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47572/0 (socket says 172.21.15.70:47572) 2024-04-01T01:42:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.539+0000 7fe8495a3640 1 -- 172.21.15.70:0/236585004 learned_addr learned my addr 172.21.15.70:0/236585004 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.539+0000 7fe848da2640 1 --2- 172.21.15.70:0/236585004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce19b090 0x5599ce1a9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.539+0000 7fe849da4640 1 --2- 172.21.15.70:0/236585004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce1a0200 0x5599ce1aa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.539+0000 7fe8495a3640 1 -- 172.21.15.70:0/236585004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce19b090 msgr2=0x5599ce1a9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.539+0000 7fe8495a3640 1 --2- 172.21.15.70:0/236585004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce19b090 0x5599ce1a9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.539+0000 7fe8495a3640 1 -- 172.21.15.70:0/236585004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce1a0200 msgr2=0x5599ce1aa710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.539+0000 7fe8495a3640 1 --2- 172.21.15.70:0/236585004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce1a0200 0x5599ce1aa710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.539+0000 7fe8495a3640 1 -- 172.21.15.70:0/236585004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599ce1aad00 con 0x5599ce19a6a0 2024-04-01T01:42:43.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.540+0000 7fe82cff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:43.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.540+0000 7fe8495a3640 1 --2- 172.21.15.70:0/236585004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 0x5599ce1a9210 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fe84003ddd0 tx=0x7fe84002c5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38767b57735e2b02 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.540+0000 7fe82cff9640 1 -- 172.21.15.70:0/236585004 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe840059070 con 0x5599ce19a6a0 2024-04-01T01:42:43.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.540+0000 7fe84e3ef740 1 -- 172.21.15.70:0/236585004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe83c029070 con 0x5599ce19a6a0 2024-04-01T01:42:43.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.540+0000 7fe82cff9640 1 -- 172.21.15.70:0/236585004 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe840034430 con 0x5599ce19a6a0 2024-04-01T01:42:43.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.541+0000 7fe82cff9640 1 -- 172.21.15.70:0/236585004 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe840063050 con 0x5599ce19a6a0 2024-04-01T01:42:43.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.541+0000 7fe84e3ef740 10 client.28199 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:43.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.541+0000 7fe84e3ef740 1 -- 172.21.15.70:0/236585004 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5599ce1abfe0 con 0x5599ce19a6a0 2024-04-01T01:42:43.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.541+0000 7fe82cff9640 1 -- 172.21.15.70:0/236585004 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe83c029070 con 0x5599ce19a6a0 2024-04-01T01:42:43.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.542+0000 7fe82cff9640 1 -- 172.21.15.70:0/236585004 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe84003d8a0 con 0x5599ce19a6a0 2024-04-01T01:42:43.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.542+0000 7fe84e3ef740 10 client.28199 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:43.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.542+0000 7fe84e3ef740 10 client.28199 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:42:43.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.542+0000 7fe84e3ef740 20 client.28199 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:43.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.542+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/236585004 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5599ce217370 0x5599ce219790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.542+0000 7fe84e3ef740 4 client.28199 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:42:43.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.542+0000 7fe84e3ef740 1 -- 172.21.15.70:0/236585004 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5599ce106a40 con 0x5599ce217370 2024-04-01T01:42:43.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.542+0000 7fe848da2640 1 --2- 172.21.15.70:0/236585004 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5599ce217370 0x5599ce219790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.543+0000 7fe848da2640 1 --2- 172.21.15.70:0/236585004 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5599ce217370 0x5599ce219790 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1ad788f20d61c923 server_cookie=da10acc5914143f in_seq=0 out_seq=0 2024-04-01T01:42:43.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.543+0000 7fe82cff9640 10 client.28199 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:42:43.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.543+0000 7fe82cff9640 1 -- 172.21.15.70:0/236585004 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5599ce106a40 con 0x5599ce217370 2024-04-01T01:42:43.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.543+0000 7fe82cff9640 10 client.28199 handle_command_reply: tid=0 2024-04-01T01:42:43.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.543+0000 7fe84e3ef740 1 client.28199 shutdown 2024-04-01T01:42:43.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.543+0000 7fe84e3ef740 1 -- 172.21.15.70:0/236585004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 msgr2=0x5599ce1a9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.543+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/236585004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 0x5599ce1a9210 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fe84003ddd0 tx=0x7fe84002c5b0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.544+0000 7fe84e3ef740 1 -- 172.21.15.70:0/236585004 shutdown_connections 2024-04-01T01:42:43.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.544+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/236585004 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5599ce217370 0x5599ce219790 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.544+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/236585004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599ce1a0200 0x5599ce1aa710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.544+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/236585004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599ce19b090 0x5599ce1a9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.544+0000 7fe84e3ef740 1 --2- 172.21.15.70:0/236585004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599ce19a6a0 0x5599ce1a9210 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.544+0000 7fe84e3ef740 1 -- 172.21.15.70:0/236585004 >> 172.21.15.70:0/236585004 conn(0x5599ce195c10 msgr2=0x5599ce194700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.544+0000 7fe82cff9640 0 client.28199 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:42:43.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.545+0000 7fe84e3ef740 1 -- 172.21.15.70:0/236585004 shutdown_connections 2024-04-01T01:42:43.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.545+0000 7fe84e3ef740 1 -- 172.21.15.70:0/236585004 wait complete. 2024-04-01T01:42:43.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.545+0000 7fe84e3ef740 20 client.28199 trim_cache size 0 max 16384 2024-04-01T01:42:43.545 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:42:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.546+0000 7fe84a5a5640 1 -- 172.21.15.70:0/1895995921 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8300a25f0 msgr2=0x7fe8300a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.546+0000 7fe84a5a5640 1 --2- 172.21.15.70:0/1895995921 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8300a25f0 0x7fe8300a4ab0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0122c0 tx=0x7fe83c0126c0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.546+0000 7fe84a5a5640 1 -- 172.21.15.70:0/1895995921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe844108e20 msgr2=0x7fe8441153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.546+0000 7fe84a5a5640 1 --2- 172.21.15.70:0/1895995921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe844108e20 0x7fe8441153b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe83400d6d0 tx=0x7fe83400db90 comp rx=0 tx=0).stop 2024-04-01T01:42:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.546+0000 7fe84a5a5640 1 -- 172.21.15.70:0/1895995921 shutdown_connections 2024-04-01T01:42:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.546+0000 7fe84a5a5640 1 --2- 172.21.15.70:0/1895995921 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe8300a25f0 0x7fe8300a4ab0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.546+0000 7fe84a5a5640 1 --2- 172.21.15.70:0/1895995921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8441097e0 0x7fe8441b6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.546+0000 7fe84a5a5640 1 --2- 172.21.15.70:0/1895995921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe844108e20 0x7fe8441153b0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.546+0000 7fe84a5a5640 1 --2- 172.21.15.70:0/1895995921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe844072b60 0x7fe844114e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.546+0000 7fe84a5a5640 1 -- 172.21.15.70:0/1895995921 >> 172.21.15.70:0/1895995921 conn(0x7fe84406e630 msgr2=0x7fe844070270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.546+0000 7fe84a5a5640 1 -- 172.21.15.70:0/1895995921 shutdown_connections 2024-04-01T01:42:43.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.546+0000 7fe84a5a5640 1 -- 172.21.15.70:0/1895995921 wait complete. 2024-04-01T01:42:43.565 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:42:43.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.699+0000 7fcb05353640 1 -- 172.21.15.70:0/2360048578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb00072b60 msgr2=0x7fcb00072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.699+0000 7fcb05353640 1 --2- 172.21.15.70:0/2360048578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb00072b60 0x7fcb00072f80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8009ca0 tx=0x7fcaf802c140 comp rx=0 tx=0).stop 2024-04-01T01:42:43.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.702+0000 7fcb05353640 1 -- 172.21.15.70:0/2360048578 shutdown_connections 2024-04-01T01:42:43.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.702+0000 7fcb05353640 1 --2- 172.21.15.70:0/2360048578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb001097e0 0x7fcb0010e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.702+0000 7fcb05353640 1 --2- 172.21.15.70:0/2360048578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb00108e20 0x7fcb001092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.702+0000 7fcb05353640 1 --2- 172.21.15.70:0/2360048578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb00072b60 0x7fcb00072f80 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.702+0000 7fcb05353640 1 -- 172.21.15.70:0/2360048578 >> 172.21.15.70:0/2360048578 conn(0x7fcb0006e630 msgr2=0x7fcb00070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.702+0000 7fcb05353640 1 -- 172.21.15.70:0/2360048578 shutdown_connections 2024-04-01T01:42:43.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.702+0000 7fcb05353640 1 -- 172.21.15.70:0/2360048578 wait complete. 2024-04-01T01:42:43.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.703+0000 7fcb05353640 1 Processor -- start 2024-04-01T01:42:43.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.703+0000 7fcb05353640 1 -- start start 2024-04-01T01:42:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.703+0000 7fcb05353640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb00072b60 0x7fcb001a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.703+0000 7fcb05353640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb00108e20 0x7fcb001a6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.703+0000 7fcb05353640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb001097e0 0x7fcb001adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.703+0000 7fcb05353640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0011c0a0 con 0x7fcb00072b60 2024-04-01T01:42:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.703+0000 7fcb05353640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0011bf20 con 0x7fcb001097e0 2024-04-01T01:42:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.703+0000 7fcb05353640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0011c220 con 0x7fcb00108e20 2024-04-01T01:42:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.704+0000 7fcb04b52640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb001097e0 0x7fcb001adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.704+0000 7fcb04b52640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb001097e0 0x7fcb001adc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47590/0 (socket says 172.21.15.70:47590) 2024-04-01T01:42:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.704+0000 7fcb04b52640 1 -- 172.21.15.70:0/709115656 learned_addr learned my addr 172.21.15.70:0/709115656 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.704+0000 7fcaff7fe640 1 --2- 172.21.15.70:0/709115656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb00108e20 0x7fcb001a6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.704+0000 7fcb04b52640 1 -- 172.21.15.70:0/709115656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb00108e20 msgr2=0x7fcb001a6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.704+0000 7fcb04b52640 1 --2- 172.21.15.70:0/709115656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb00108e20 0x7fcb001a6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.704+0000 7fcb04b52640 1 -- 172.21.15.70:0/709115656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb00072b60 msgr2=0x7fcb001a6650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:42:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.704+0000 7fcb04b52640 1 --2- 172.21.15.70:0/709115656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb00072b60 0x7fcb001a6650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.704+0000 7fcb04b52640 1 -- 172.21.15.70:0/709115656 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb001ae270 con 0x7fcb001097e0 2024-04-01T01:42:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.705+0000 7fcb04b52640 1 --2- 172.21.15.70:0/709115656 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb001097e0 0x7fcb001adc00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fcaf000b3f0 tx=0x7fcaf000b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.705+0000 7fcafd7fa640 1 -- 172.21.15.70:0/709115656 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf0015020 con 0x7fcb001097e0 2024-04-01T01:42:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.705+0000 7fcafd7fa640 1 -- 172.21.15.70:0/709115656 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcaf0011ce0 con 0x7fcb001097e0 2024-04-01T01:42:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.705+0000 7fcb05353640 1 -- 172.21.15.70:0/709115656 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb001ae560 con 0x7fcb001097e0 2024-04-01T01:42:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.705+0000 7fcb05353640 1 -- 172.21.15.70:0/709115656 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb001aead0 con 0x7fcb001097e0 2024-04-01T01:42:43.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.707+0000 7fcafd7fa640 1 -- 172.21.15.70:0/709115656 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf0010df0 con 0x7fcb001097e0 2024-04-01T01:42:43.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.709+0000 7fcafd7fa640 1 -- 172.21.15.70:0/709115656 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fcaf0018020 con 0x7fcb001097e0 2024-04-01T01:42:43.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.710+0000 7fcb09197740 1 Processor -- start 2024-04-01T01:42:43.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.711+0000 7fcafd7fa640 1 --2- 172.21.15.70:0/709115656 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcae80a24c0 0x7fcae80a4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.711+0000 7fcafd7fa640 1 -- 172.21.15.70:0/709115656 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fcaf00ce330 con 0x7fcb001097e0 2024-04-01T01:42:43.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.712+0000 7fcb09197740 1 -- start start 2024-04-01T01:42:43.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.712+0000 7fcb09197740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d7716a0 0x55ea9d771ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.712+0000 7fcb09197740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d776b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.712+0000 7fcb09197740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d777200 0x55ea9d779690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.712+0000 7fcb09197740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ea9d6d61b0 con 0x55ea9d777200 2024-04-01T01:42:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.712+0000 7fcb09197740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ea9d67ec00 con 0x55ea9d7716a0 2024-04-01T01:42:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.712+0000 7fcb09197740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ea9d6d2250 con 0x55ea9d772090 2024-04-01T01:42:43.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.714+0000 7fcb04b52640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d777200 0x55ea9d779690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.714+0000 7fcb04b52640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d777200 0x55ea9d779690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52132/0 (socket says 172.21.15.70:52132) 2024-04-01T01:42:43.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.714+0000 7fcb04b52640 1 -- 172.21.15.70:0/3930322523 learned_addr learned my addr 172.21.15.70:0/3930322523 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:43.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.714+0000 7fcaff7fe640 1 --2- 172.21.15.70:0/709115656 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcae80a24c0 0x7fcae80a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.714+0000 7fcaff7fe640 1 --2- 172.21.15.70:0/3930322523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d776b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.714+0000 7fcaff7fe640 1 --2- 172.21.15.70:0/709115656 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcae80a24c0 0x7fcae80a4980 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4006e10 tx=0x7fcaf4006d60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.715+0000 7fcaff7fe640 1 -- 172.21.15.70:0/3930322523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d7716a0 msgr2=0x55ea9d771ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:42:43.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.715+0000 7fcaff7fe640 1 --2- 172.21.15.70:0/3930322523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d7716a0 0x55ea9d771ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.715+0000 7fcaff7fe640 1 -- 172.21.15.70:0/3930322523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d777200 msgr2=0x55ea9d779690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.715+0000 7fcaff7fe640 1 --2- 172.21.15.70:0/3930322523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d777200 0x55ea9d779690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.715+0000 7fcaff7fe640 1 -- 172.21.15.70:0/3930322523 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea9d69e3d0 con 0x55ea9d772090 2024-04-01T01:42:43.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.715+0000 7fcaff7fe640 1 --2- 172.21.15.70:0/3930322523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d776b90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4010c10 tx=0x7fcaf4019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47f9cf402344f8ed server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.718+0000 7fcae67fc640 1 -- 172.21.15.70:0/3930322523 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf401b040 con 0x55ea9d772090 2024-04-01T01:42:43.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.718+0000 7fcae67fc640 1 -- 172.21.15.70:0/3930322523 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcaf4024070 con 0x55ea9d772090 2024-04-01T01:42:43.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.718+0000 7fcae67fc640 1 -- 172.21.15.70:0/3930322523 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf401f3f0 con 0x55ea9d772090 2024-04-01T01:42:43.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.719+0000 7fcb09197740 1 -- 172.21.15.70:0/3930322523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 msgr2=0x55ea9d776b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.719+0000 7fcb09197740 1 --2- 172.21.15.70:0/3930322523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d776b90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4010c10 tx=0x7fcaf4019000 comp rx=0 tx=0).stop 2024-04-01T01:42:43.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.721+0000 7fcb09197740 1 -- 172.21.15.70:0/3930322523 shutdown_connections 2024-04-01T01:42:43.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.721+0000 7fcb09197740 1 --2- 172.21.15.70:0/3930322523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d777200 0x55ea9d779690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.721+0000 7fcb09197740 1 --2- 172.21.15.70:0/3930322523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d776b90 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.721+0000 7fcb09197740 1 --2- 172.21.15.70:0/3930322523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d7716a0 0x55ea9d771ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.721+0000 7fcb09197740 1 -- 172.21.15.70:0/3930322523 >> 172.21.15.70:0/3930322523 conn(0x55ea9d76cc10 msgr2=0x55ea9d76d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.723+0000 7fcb09197740 1 -- 172.21.15.70:0/3930322523 shutdown_connections 2024-04-01T01:42:43.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.723+0000 7fcb09197740 1 -- 172.21.15.70:0/3930322523 wait complete. 2024-04-01T01:42:43.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.723+0000 7fcb09197740 1 -- start start 2024-04-01T01:42:43.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.724+0000 7fcb09197740 1 Processor -- start 2024-04-01T01:42:43.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.724+0000 7fcb09197740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d7716a0 0x55ea9d77bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.724+0000 7fcb09197740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d77c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.724+0000 7fcb09197740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d777200 0x55ea9d783270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.724+0000 7fcb09197740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ea9d77a1a0 con 0x55ea9d777200 2024-04-01T01:42:43.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.724+0000 7fcb09197740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ea9d674470 con 0x55ea9d7716a0 2024-04-01T01:42:43.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.724+0000 7fcb09197740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ea9d77a320 con 0x55ea9d772090 2024-04-01T01:42:43.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.724+0000 7fcb09197740 10 client.0 _ll_register_callbacks cb 0x55ea9d691cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:43.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.725+0000 7fcb04b52640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d777200 0x55ea9d783270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.725+0000 7fcb04b52640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d777200 0x55ea9d783270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52136/0 (socket says 172.21.15.70:52136) 2024-04-01T01:42:43.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.725+0000 7fcb04b52640 1 -- 172.21.15.70:0/2217435009 learned_addr learned my addr 172.21.15.70:0/2217435009 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:43.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.725+0000 7fcafffff640 1 --2- 172.21.15.70:0/2217435009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d7716a0 0x55ea9d77bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.725+0000 7fcafffff640 1 -- 172.21.15.70:0/2217435009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 msgr2=0x55ea9d77c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.725+0000 7fcafffff640 1 --2- 172.21.15.70:0/2217435009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d77c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.725+0000 7fcafffff640 1 -- 172.21.15.70:0/2217435009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d777200 msgr2=0x55ea9d783270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.725+0000 7fcafffff640 1 --2- 172.21.15.70:0/2217435009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d777200 0x55ea9d783270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.725+0000 7fcafffff640 1 -- 172.21.15.70:0/2217435009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea9d3ea3f0 con 0x55ea9d7716a0 2024-04-01T01:42:43.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.727+0000 7fcafffff640 1 --2- 172.21.15.70:0/2217435009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d7716a0 0x55ea9d77bd30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x55ea9d76db80 tx=0x7fcaf8032570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26f18eea5df5b8e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.727+0000 7fcadf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:43.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.727+0000 7fcadf7fe640 1 -- 172.21.15.70:0/2217435009 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf8039040 con 0x55ea9d7716a0 2024-04-01T01:42:43.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.728+0000 7fcb09197740 1 -- 172.21.15.70:0/2217435009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcaf4024070 con 0x55ea9d7716a0 2024-04-01T01:42:43.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.728+0000 7fcadf7fe640 1 -- 172.21.15.70:0/2217435009 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fcaf80327e0 con 0x55ea9d7716a0 2024-04-01T01:42:43.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.728+0000 7fcadf7fe640 1 -- 172.21.15.70:0/2217435009 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcaf804d050 con 0x55ea9d7716a0 2024-04-01T01:42:43.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.728+0000 7fcb09197740 10 client.28208 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:43.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.728+0000 7fcb09197740 1 -- 172.21.15.70:0/2217435009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ea9d4b47d0 con 0x55ea9d7716a0 2024-04-01T01:42:43.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.729+0000 7fcadf7fe640 1 -- 172.21.15.70:0/2217435009 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcaf4024070 con 0x55ea9d7716a0 2024-04-01T01:42:43.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.729+0000 7fcadf7fe640 1 -- 172.21.15.70:0/2217435009 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fcaf803c030 con 0x55ea9d7716a0 2024-04-01T01:42:43.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.730+0000 7fcb09197740 10 client.28208 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:43.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.730+0000 7fcb09197740 10 client.28208 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:42:43.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.730+0000 7fcb09197740 20 client.28208 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:43.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.730+0000 7fcb09197740 1 --2- 172.21.15.70:0/2217435009 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55ea9d798800 0x55ea9d79ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.730+0000 7fcb09197740 4 client.28208 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:43.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.730+0000 7fcb09197740 1 -- 172.21.15.70:0/2217435009 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea9d6e36f0 con 0x55ea9d798800 2024-04-01T01:42:43.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.732+0000 7fcaff7fe640 1 --2- 172.21.15.70:0/2217435009 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55ea9d798800 0x55ea9d79ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.735+0000 7fcaff7fe640 1 --2- 172.21.15.70:0/2217435009 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55ea9d798800 0x55ea9d79ac80 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3891aaba647986bc server_cookie=3d6add1b7009bebf in_seq=0 out_seq=0 2024-04-01T01:42:43.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.735+0000 7fcadf7fe640 10 client.28208 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:42:43.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.736+0000 7fcadf7fe640 1 -- 172.21.15.70:0/2217435009 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ea9d6e36f0 con 0x55ea9d798800 2024-04-01T01:42:43.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.736+0000 7fcadf7fe640 10 client.28208 handle_command_reply: tid=0 2024-04-01T01:42:43.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.736+0000 7fcb09197740 1 client.28208 shutdown 2024-04-01T01:42:43.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.737+0000 7fcb09197740 1 -- 172.21.15.70:0/2217435009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d7716a0 msgr2=0x55ea9d77bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.737+0000 7fcb09197740 1 --2- 172.21.15.70:0/2217435009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d7716a0 0x55ea9d77bd30 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x55ea9d76db80 tx=0x7fcaf8032570 comp rx=0 tx=0).stop 2024-04-01T01:42:43.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.737+0000 7fcb09197740 1 -- 172.21.15.70:0/2217435009 shutdown_connections 2024-04-01T01:42:43.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.737+0000 7fcb09197740 1 --2- 172.21.15.70:0/2217435009 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55ea9d798800 0x55ea9d79ac80 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.737+0000 7fcb09197740 1 --2- 172.21.15.70:0/2217435009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d777200 0x55ea9d783270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.737+0000 7fcb09197740 1 --2- 172.21.15.70:0/2217435009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d77c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.737+0000 7fcb09197740 1 --2- 172.21.15.70:0/2217435009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d7716a0 0x55ea9d77bd30 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.737+0000 7fcb09197740 1 -- 172.21.15.70:0/2217435009 >> 172.21.15.70:0/2217435009 conn(0x55ea9d76cc10 msgr2=0x55ea9d76bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.738+0000 7fcadf7fe640 0 client.28208 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:42:43.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.738+0000 7fcb09197740 1 -- 172.21.15.70:0/2217435009 shutdown_connections 2024-04-01T01:42:43.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.738+0000 7fcb09197740 1 -- 172.21.15.70:0/2217435009 wait complete. 2024-04-01T01:42:43.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.738+0000 7fcb09197740 20 client.28208 trim_cache size 0 max 16384 2024-04-01T01:42:43.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.764+0000 7fcb09197740 1 Processor -- start 2024-04-01T01:42:43.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.764+0000 7fcb09197740 1 -- start start 2024-04-01T01:42:43.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.764+0000 7fcb09197740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d7716a0 0x55ea9d77c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.765+0000 7fcafffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d7716a0 0x55ea9d77c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.765+0000 7fcb09197740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d76b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.765+0000 7fcafffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d7716a0 0x55ea9d77c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52144/0 (socket says 172.21.15.70:52144) 2024-04-01T01:42:43.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.765+0000 7fcafffff640 1 -- 172.21.15.70:0/54801603 learned_addr learned my addr 172.21.15.70:0/54801603 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:43.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.765+0000 7fcb09197740 1 --2- 172.21.15.70:0/54801603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d777200 0x55ea9d76bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.765+0000 7fcb09197740 1 -- 172.21.15.70:0/54801603 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec00fd60 con 0x55ea9d7716a0 2024-04-01T01:42:43.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.765+0000 7fcb09197740 1 -- 172.21.15.70:0/54801603 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec0118a0 con 0x55ea9d777200 2024-04-01T01:42:43.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.765+0000 7fcb09197740 1 -- 172.21.15.70:0/54801603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec010430 con 0x55ea9d772090 2024-04-01T01:42:43.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.765+0000 7fcaff7fe640 1 --2- 172.21.15.70:0/54801603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d76b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.766+0000 7fcb04b52640 1 --2- 172.21.15.70:0/54801603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d777200 0x55ea9d76bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.766+0000 7fcaff7fe640 1 -- 172.21.15.70:0/54801603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d777200 msgr2=0x55ea9d76bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.766+0000 7fcaff7fe640 1 --2- 172.21.15.70:0/54801603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d777200 0x55ea9d76bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.766+0000 7fcaff7fe640 1 -- 172.21.15.70:0/54801603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d7716a0 msgr2=0x55ea9d77c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.766+0000 7fcaff7fe640 1 --2- 172.21.15.70:0/54801603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d7716a0 0x55ea9d77c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.766+0000 7fcaff7fe640 1 -- 172.21.15.70:0/54801603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea9d76d980 con 0x55ea9d772090 2024-04-01T01:42:43.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.766+0000 7fcaff7fe640 1 --2- 172.21.15.70:0/54801603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d76b700 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcaf40098d0 tx=0x7fcaf401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ca895448d1d9259 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.768+0000 7fcae5ffb640 1 -- 172.21.15.70:0/54801603 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4034020 con 0x55ea9d772090 2024-04-01T01:42:43.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.768+0000 7fcae5ffb640 1 -- 172.21.15.70:0/54801603 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcaf4024040 con 0x55ea9d772090 2024-04-01T01:42:43.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.768+0000 7fcae5ffb640 1 -- 172.21.15.70:0/54801603 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf401b040 con 0x55ea9d772090 2024-04-01T01:42:43.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.769+0000 7fcb09197740 1 -- 172.21.15.70:0/54801603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 msgr2=0x55ea9d76b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.769+0000 7fcb09197740 1 --2- 172.21.15.70:0/54801603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d76b700 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcaf40098d0 tx=0x7fcaf401a000 comp rx=0 tx=0).stop 2024-04-01T01:42:43.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.770+0000 7fcb09197740 1 -- 172.21.15.70:0/54801603 shutdown_connections 2024-04-01T01:42:43.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.770+0000 7fcb09197740 1 --2- 172.21.15.70:0/54801603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d777200 0x55ea9d76bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.770+0000 7fcb09197740 1 --2- 172.21.15.70:0/54801603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d76b700 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.770+0000 7fcb09197740 1 --2- 172.21.15.70:0/54801603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d7716a0 0x55ea9d77c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.770+0000 7fcb09197740 1 -- 172.21.15.70:0/54801603 >> 172.21.15.70:0/54801603 conn(0x55ea9d76cc10 msgr2=0x55ea9d77bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.771+0000 7fcb09197740 1 -- 172.21.15.70:0/54801603 shutdown_connections 2024-04-01T01:42:43.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.771+0000 7fcb09197740 1 -- 172.21.15.70:0/54801603 wait complete. 2024-04-01T01:42:43.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.772+0000 7fcb09197740 1 -- start start 2024-04-01T01:42:43.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.772+0000 7fcb09197740 1 Processor -- start 2024-04-01T01:42:43.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.773+0000 7fcb09197740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d7716a0 0x55ea9d7801c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.773+0000 7fcb09197740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d780700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.773+0000 7fcb09197740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d777200 0x55ea9d781710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.774+0000 7fcb09197740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec0111d0 con 0x55ea9d7716a0 2024-04-01T01:42:43.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.774+0000 7fcb04b52640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d777200 0x55ea9d781710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.774+0000 7fcb04b52640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d777200 0x55ea9d781710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47628/0 (socket says 172.21.15.70:47628) 2024-04-01T01:42:43.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.774+0000 7fcb04b52640 1 -- 172.21.15.70:0/195789092 learned_addr learned my addr 172.21.15.70:0/195789092 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:43.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.774+0000 7fcaff7fe640 1 --2- 172.21.15.70:0/195789092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d780700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.774+0000 7fcb09197740 1 -- 172.21.15.70:0/195789092 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec010b00 con 0x55ea9d777200 2024-04-01T01:42:43.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.774+0000 7fcb09197740 1 -- 172.21.15.70:0/195789092 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec00dd40 con 0x55ea9d772090 2024-04-01T01:42:43.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.774+0000 7fcafffff640 1 --2- 172.21.15.70:0/195789092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d7716a0 0x55ea9d7801c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.774+0000 7fcb09197740 10 client.0 _ll_register_callbacks cb 0x7fcaf8037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:43.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.775+0000 7fcafffff640 1 -- 172.21.15.70:0/195789092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 msgr2=0x55ea9d780700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.775+0000 7fcafffff640 1 --2- 172.21.15.70:0/195789092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d780700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.775+0000 7fcafffff640 1 -- 172.21.15.70:0/195789092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d777200 msgr2=0x55ea9d781710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:43.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.775+0000 7fcafffff640 1 --2- 172.21.15.70:0/195789092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d777200 0x55ea9d781710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.775+0000 7fcafffff640 1 -- 172.21.15.70:0/195789092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea9d781d00 con 0x55ea9d7716a0 2024-04-01T01:42:43.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.775+0000 7fcafffff640 1 --2- 172.21.15.70:0/195789092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d7716a0 0x55ea9d7801c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8034630 tx=0x7fcaf802cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ffad57bd9c0eec5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:43.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.775+0000 7fcadf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:42:43.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.775+0000 7fcadf7fe640 1 -- 172.21.15.70:0/195789092 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf8039040 con 0x55ea9d7716a0 2024-04-01T01:42:43.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.776+0000 7fcadf7fe640 1 -- 172.21.15.70:0/195789092 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fcaf8034d50 con 0x55ea9d7716a0 2024-04-01T01:42:43.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.776+0000 7fcb09197740 1 -- 172.21.15.70:0/195789092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcaf4024040 con 0x55ea9d7716a0 2024-04-01T01:42:43.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.776+0000 7fcadf7fe640 1 -- 172.21.15.70:0/195789092 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcaf805e050 con 0x55ea9d7716a0 2024-04-01T01:42:43.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.776+0000 7fcb09197740 10 client.18216 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:43.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.776+0000 7fcb09197740 1 -- 172.21.15.70:0/195789092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ea9d782fe0 con 0x55ea9d7716a0 2024-04-01T01:42:43.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.777+0000 7fcadf7fe640 1 -- 172.21.15.70:0/195789092 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcaf4024040 con 0x55ea9d7716a0 2024-04-01T01:42:43.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.777+0000 7fcadf7fe640 1 -- 172.21.15.70:0/195789092 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fcaf8059030 con 0x55ea9d7716a0 2024-04-01T01:42:43.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.777+0000 7fcb09197740 10 client.18216 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:43.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.777+0000 7fcb09197740 10 client.18216 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:42:43.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.777+0000 7fcb09197740 20 client.18216 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:43.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.777+0000 7fcb09197740 1 --2- 172.21.15.70:0/195789092 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55ea9d7ff090 0x55ea9d8014b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:43.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.777+0000 7fcb09197740 4 client.18216 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:42:43.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.777+0000 7fcb09197740 1 -- 172.21.15.70:0/195789092 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ea9d6dda40 con 0x55ea9d7ff090 2024-04-01T01:42:43.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.782+0000 7fcaff7fe640 1 --2- 172.21.15.70:0/195789092 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55ea9d7ff090 0x55ea9d8014b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.783+0000 7fcaff7fe640 1 --2- 172.21.15.70:0/195789092 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55ea9d7ff090 0x55ea9d8014b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4b5c971e2a195ca server_cookie=c9ec5aafd0bd64d7 in_seq=0 out_seq=0 2024-04-01T01:42:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.783+0000 7fcadf7fe640 10 client.18216 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:42:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.783+0000 7fcadf7fe640 1 -- 172.21.15.70:0/195789092 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ea9d6dda40 con 0x55ea9d7ff090 2024-04-01T01:42:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.783+0000 7fcadf7fe640 10 client.18216 handle_command_reply: tid=0 2024-04-01T01:42:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.783+0000 7fcb09197740 1 client.18216 shutdown 2024-04-01T01:42:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.784+0000 7fcb09197740 1 -- 172.21.15.70:0/195789092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d7716a0 msgr2=0x55ea9d7801c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.784+0000 7fcb09197740 1 --2- 172.21.15.70:0/195789092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d7716a0 0x55ea9d7801c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8034630 tx=0x7fcaf802cf20 comp rx=0 tx=0).stop 2024-04-01T01:42:43.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.785+0000 7fcb09197740 1 -- 172.21.15.70:0/195789092 shutdown_connections 2024-04-01T01:42:43.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.785+0000 7fcb09197740 1 --2- 172.21.15.70:0/195789092 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55ea9d7ff090 0x55ea9d8014b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.785+0000 7fcb09197740 1 --2- 172.21.15.70:0/195789092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ea9d777200 0x55ea9d781710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.785+0000 7fcb09197740 1 --2- 172.21.15.70:0/195789092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ea9d772090 0x55ea9d780700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.785+0000 7fcb09197740 1 --2- 172.21.15.70:0/195789092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea9d7716a0 0x55ea9d7801c0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.785+0000 7fcb09197740 1 -- 172.21.15.70:0/195789092 >> 172.21.15.70:0/195789092 conn(0x55ea9d76cc10 msgr2=0x55ea9d770700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.785+0000 7fcadf7fe640 0 client.18216 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:42:43.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.786+0000 7fcb09197740 1 -- 172.21.15.70:0/195789092 shutdown_connections 2024-04-01T01:42:43.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.786+0000 7fcb09197740 1 -- 172.21.15.70:0/195789092 wait complete. 2024-04-01T01:42:43.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.790+0000 7fcb09197740 20 client.18216 trim_cache size 0 max 16384 2024-04-01T01:42:43.794 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:42:43.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.799+0000 7fcb05353640 1 -- 172.21.15.70:0/709115656 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcae80a24c0 msgr2=0x7fcae80a4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.799+0000 7fcb05353640 1 --2- 172.21.15.70:0/709115656 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcae80a24c0 0x7fcae80a4980 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4006e10 tx=0x7fcaf4006d60 comp rx=0 tx=0).stop 2024-04-01T01:42:43.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.799+0000 7fcb05353640 1 -- 172.21.15.70:0/709115656 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb001097e0 msgr2=0x7fcb001adc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:43.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.799+0000 7fcb05353640 1 --2- 172.21.15.70:0/709115656 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb001097e0 0x7fcb001adc00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fcaf000b3f0 tx=0x7fcaf000b7f0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.802+0000 7fcb05353640 1 -- 172.21.15.70:0/709115656 shutdown_connections 2024-04-01T01:42:43.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.802+0000 7fcb05353640 1 --2- 172.21.15.70:0/709115656 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb001097e0 0x7fcb001adc00 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.802+0000 7fcb05353640 1 --2- 172.21.15.70:0/709115656 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcae80a24c0 0x7fcae80a4980 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.802+0000 7fcb05353640 1 --2- 172.21.15.70:0/709115656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb00108e20 0x7fcb001a6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.802+0000 7fcb05353640 1 --2- 172.21.15.70:0/709115656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb00072b60 0x7fcb001a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:43.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.802+0000 7fcb05353640 1 -- 172.21.15.70:0/709115656 >> 172.21.15.70:0/709115656 conn(0x7fcb0006e630 msgr2=0x7fcb0010a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:43.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.803+0000 7fcb05353640 1 -- 172.21.15.70:0/709115656 shutdown_connections 2024-04-01T01:42:43.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:43.803+0000 7fcb05353640 1 -- 172.21.15.70:0/709115656 wait complete. 2024-04-01T01:42:43.826 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:42:44.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.014+0000 7f09d22a0640 1 -- 172.21.15.70:0/3649172303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09c4096f60 msgr2=0x7f09c40993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.014+0000 7f09d22a0640 1 --2- 172.21.15.70:0/3649172303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09c4096f60 0x7f09c40993d0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009860 tx=0x7f09c802bbb0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.014+0000 7f09d22a0640 1 -- 172.21.15.70:0/3649172303 shutdown_connections 2024-04-01T01:42:44.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.014+0000 7f09d22a0640 1 --2- 172.21.15.70:0/3649172303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c4099910 0x7f09c4099d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.014+0000 7f09d22a0640 1 --2- 172.21.15.70:0/3649172303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09c4096f60 0x7f09c40993d0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.014+0000 7f09d22a0640 1 --2- 172.21.15.70:0/3649172303 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09c40945f0 0x7f09c4096a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.014+0000 7f09d22a0640 1 -- 172.21.15.70:0/3649172303 >> 172.21.15.70:0/3649172303 conn(0x7f09c40901d0 msgr2=0x7f09c4092630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:44.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.014+0000 7f09d22a0640 1 -- 172.21.15.70:0/3649172303 shutdown_connections 2024-04-01T01:42:44.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.014+0000 7f09d22a0640 1 -- 172.21.15.70:0/3649172303 wait complete. 2024-04-01T01:42:44.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.015+0000 7f09d22a0640 1 Processor -- start 2024-04-01T01:42:44.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.015+0000 7f09d22a0640 1 -- start start 2024-04-01T01:42:44.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.015+0000 7f09d22a0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c40945f0 0x7f09c4132660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.015+0000 7f09d22a0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09c4096f60 0x7f09c4132ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.015+0000 7f09d22a0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09c4099910 0x7f09c4139c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.015+0000 7f09d22a0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f09c40a7ea0 con 0x7f09c4099910 2024-04-01T01:42:44.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.015+0000 7f09d22a0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f09c40a7d20 con 0x7f09c4096f60 2024-04-01T01:42:44.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.015+0000 7f09d22a0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f09c40a8020 con 0x7f09c40945f0 2024-04-01T01:42:44.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.015+0000 7f09d129e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c40945f0 0x7f09c4132660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.015+0000 7f09d129e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c40945f0 0x7f09c4132660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40986/0 (socket says 172.21.15.70:40986) 2024-04-01T01:42:44.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.015+0000 7f09d129e640 1 -- 172.21.15.70:0/948861087 learned_addr learned my addr 172.21.15.70:0/948861087 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:44.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.015+0000 7f09d0a9d640 1 --2- 172.21.15.70:0/948861087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09c4096f60 0x7f09c4132ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.016+0000 7f09d1a9f640 1 --2- 172.21.15.70:0/948861087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09c4099910 0x7f09c4139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.016+0000 7f09d129e640 1 -- 172.21.15.70:0/948861087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09c4096f60 msgr2=0x7f09c4132ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.016+0000 7f09d129e640 1 --2- 172.21.15.70:0/948861087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09c4096f60 0x7f09c4132ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.016+0000 7f09d129e640 1 -- 172.21.15.70:0/948861087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09c4099910 msgr2=0x7f09c4139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.016+0000 7f09d129e640 1 --2- 172.21.15.70:0/948861087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09c4099910 0x7f09c4139c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.016+0000 7f09d129e640 1 -- 172.21.15.70:0/948861087 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c414b290 con 0x7f09c40945f0 2024-04-01T01:42:44.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.016+0000 7f09d129e640 1 --2- 172.21.15.70:0/948861087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c40945f0 0x7f09c4132660 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00b4a0 tx=0x7f09bc00b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:44.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.016+0000 7f09c27fc640 1 -- 172.21.15.70:0/948861087 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc015020 con 0x7f09c40945f0 2024-04-01T01:42:44.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.016+0000 7f09c27fc640 1 -- 172.21.15.70:0/948861087 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09bc011ce0 con 0x7f09c40945f0 2024-04-01T01:42:44.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.016+0000 7f09c27fc640 1 -- 172.21.15.70:0/948861087 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc010dc0 con 0x7f09c40945f0 2024-04-01T01:42:44.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.017+0000 7f09d22a0640 1 -- 172.21.15.70:0/948861087 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c414b580 con 0x7f09c40945f0 2024-04-01T01:42:44.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.017+0000 7f09d22a0640 1 -- 172.21.15.70:0/948861087 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c414bad0 con 0x7f09c40945f0 2024-04-01T01:42:44.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.020+0000 7f09c27fc640 1 -- 172.21.15.70:0/948861087 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f09bc018020 con 0x7f09c40945f0 2024-04-01T01:42:44.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.020+0000 7f09d61c4740 1 Processor -- start 2024-04-01T01:42:44.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.022+0000 7f09c27fc640 1 --2- 172.21.15.70:0/948861087 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09b80a25a0 0x7f09b80a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.022+0000 7f09c27fc640 1 -- 172.21.15.70:0/948861087 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f09bc0ce2f0 con 0x7f09c40945f0 2024-04-01T01:42:44.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.022+0000 7f09d1a9f640 1 --2- 172.21.15.70:0/948861087 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09b80a25a0 0x7f09b80a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.023+0000 7f09d61c4740 1 -- start start 2024-04-01T01:42:44.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.023+0000 7f09d61c4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b5056a0 0x559d4b505ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.023+0000 7f09d61c4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b506090 0x559d4b50ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.023+0000 7f09d61c4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b50b200 0x559d4b50d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.023+0000 7f09d61c4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559d4b46a1b0 con 0x559d4b506090 2024-04-01T01:42:44.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.023+0000 7f09d61c4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x559d4b412c00 con 0x559d4b5056a0 2024-04-01T01:42:44.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.023+0000 7f09d61c4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559d4b466250 con 0x559d4b50b200 2024-04-01T01:42:44.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.023+0000 7f09d0a9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b506090 0x559d4b50ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.023+0000 7f09d0a9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b506090 0x559d4b50ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52164/0 (socket says 172.21.15.70:52164) 2024-04-01T01:42:44.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.023+0000 7f09d0a9d640 1 -- 172.21.15.70:0/2456341778 learned_addr learned my addr 172.21.15.70:0/2456341778 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:44.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.024+0000 7f09d0a9d640 1 -- 172.21.15.70:0/2456341778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b50b200 msgr2=0x559d4b50d690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:42:44.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.024+0000 7f09d0a9d640 1 --2- 172.21.15.70:0/2456341778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b50b200 0x559d4b50d690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.024+0000 7f09d0a9d640 1 -- 172.21.15.70:0/2456341778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b5056a0 msgr2=0x559d4b505ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:42:44.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.024+0000 7f09d0a9d640 1 --2- 172.21.15.70:0/2456341778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b5056a0 0x559d4b505ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.024+0000 7f09d0a9d640 1 -- 172.21.15.70:0/2456341778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d4b4323d0 con 0x559d4b506090 2024-04-01T01:42:44.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.024+0000 7f09d0a9d640 1 --2- 172.21.15.70:0/2456341778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b506090 0x559d4b50ab90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009830 tx=0x7f09c8034440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b4b065bc7f8767 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:44.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.025+0000 7f09b77fe640 1 -- 172.21.15.70:0/2456341778 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c8039040 con 0x559d4b506090 2024-04-01T01:42:44.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.025+0000 7f09b77fe640 1 -- 172.21.15.70:0/2456341778 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09c8034cd0 con 0x559d4b506090 2024-04-01T01:42:44.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.025+0000 7f09b77fe640 1 -- 172.21.15.70:0/2456341778 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c8005640 con 0x559d4b506090 2024-04-01T01:42:44.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.026+0000 7f09d61c4740 1 -- 172.21.15.70:0/2456341778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b506090 msgr2=0x559d4b50ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.026+0000 7f09d1a9f640 1 --2- 172.21.15.70:0/948861087 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09b80a25a0 0x7f09b80a4a60 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f09cc067290 tx=0x7f09cc077040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:44.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.026+0000 7f09d61c4740 1 --2- 172.21.15.70:0/2456341778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b506090 0x559d4b50ab90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009830 tx=0x7f09c8034440 comp rx=0 tx=0).stop 2024-04-01T01:42:44.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.027+0000 7f09d61c4740 1 -- 172.21.15.70:0/2456341778 shutdown_connections 2024-04-01T01:42:44.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.027+0000 7f09d61c4740 1 --2- 172.21.15.70:0/2456341778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b50b200 0x559d4b50d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.027+0000 7f09d61c4740 1 --2- 172.21.15.70:0/2456341778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b506090 0x559d4b50ab90 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.027+0000 7f09d61c4740 1 --2- 172.21.15.70:0/2456341778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b5056a0 0x559d4b505ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.027+0000 7f09d61c4740 1 -- 172.21.15.70:0/2456341778 >> 172.21.15.70:0/2456341778 conn(0x559d4b500c10 msgr2=0x559d4b501010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:44.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.027+0000 7f09d61c4740 1 -- 172.21.15.70:0/2456341778 shutdown_connections 2024-04-01T01:42:44.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.027+0000 7f09d61c4740 1 -- 172.21.15.70:0/2456341778 wait complete. 2024-04-01T01:42:44.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.027+0000 7f09d61c4740 1 -- start start 2024-04-01T01:42:44.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.028+0000 7f09d61c4740 1 Processor -- start 2024-04-01T01:42:44.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.028+0000 7f09d61c4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b5056a0 0x559d4b520c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.028+0000 7f09d61c4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b506090 0x559d4b521190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.028+0000 7f09d61c4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b50b200 0x559d4b5281d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.029+0000 7f09d61c4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559d4b51f160 con 0x559d4b5056a0 2024-04-01T01:42:44.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.029+0000 7f09d129e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b5056a0 0x559d4b520c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.029+0000 7f09d61c4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x559d4b408470 con 0x559d4b50b200 2024-04-01T01:42:44.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.029+0000 7f09d61c4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559d4b51f2e0 con 0x559d4b506090 2024-04-01T01:42:44.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.029+0000 7f09d129e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b5056a0 0x559d4b520c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52166/0 (socket says 172.21.15.70:52166) 2024-04-01T01:42:44.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.029+0000 7f09d129e640 1 -- 172.21.15.70:0/542735589 learned_addr learned my addr 172.21.15.70:0/542735589 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:44.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.029+0000 7f09d61c4740 10 client.0 _ll_register_callbacks cb 0x559d4b425cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:44.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.029+0000 7f09d0a9d640 1 --2- 172.21.15.70:0/542735589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b506090 0x559d4b521190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.029+0000 7f09d0a9d640 1 -- 172.21.15.70:0/542735589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b50b200 msgr2=0x559d4b5281d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:44.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.029+0000 7f09d0a9d640 1 --2- 172.21.15.70:0/542735589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b50b200 0x559d4b5281d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.029+0000 7f09d0a9d640 1 -- 172.21.15.70:0/542735589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b5056a0 msgr2=0x559d4b520c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:44.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.029+0000 7f09d0a9d640 1 --2- 172.21.15.70:0/542735589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b5056a0 0x559d4b520c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.029+0000 7f09d0a9d640 1 -- 172.21.15.70:0/542735589 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d4b17e3f0 con 0x559d4b506090 2024-04-01T01:42:44.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.030+0000 7f09d0a9d640 1 --2- 172.21.15.70:0/542735589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b506090 0x559d4b521190 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x559d4b501b80 tx=0x7f09c8034e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ace75da7dfd0c91c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:44.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.030+0000 7f09b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:42:44.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.030+0000 7f09b4ff9640 1 -- 172.21.15.70:0/542735589 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c8039070 con 0x559d4b506090 2024-04-01T01:42:44.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.030+0000 7f09d61c4740 1 -- 172.21.15.70:0/542735589 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09c8034cd0 con 0x559d4b506090 2024-04-01T01:42:44.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.030+0000 7f09b4ff9640 1 -- 172.21.15.70:0/542735589 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f09c8004450 con 0x559d4b506090 2024-04-01T01:42:44.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.031+0000 7f09b4ff9640 1 -- 172.21.15.70:0/542735589 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f09c8059050 con 0x559d4b506090 2024-04-01T01:42:44.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.031+0000 7f09d61c4740 10 client.28030 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:44.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.031+0000 7f09d61c4740 1 -- 172.21.15.70:0/542735589 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559d4b2487d0 con 0x559d4b506090 2024-04-01T01:42:44.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.032+0000 7f09b4ff9640 1 -- 172.21.15.70:0/542735589 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09c802ca00 con 0x559d4b506090 2024-04-01T01:42:44.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.032+0000 7f09b4ff9640 1 -- 172.21.15.70:0/542735589 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f09c8003be0 con 0x559d4b506090 2024-04-01T01:42:44.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.032+0000 7f09d61c4740 10 client.28030 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:44.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.032+0000 7f09d61c4740 10 client.28030 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:42:44.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.032+0000 7f09d61c4740 20 client.28030 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:44.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.032+0000 7f09d61c4740 1 --2- 172.21.15.70:0/542735589 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x559d4b52c330 0x559d4b52e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.032+0000 7f09d61c4740 4 client.28030 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:44.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.032+0000 7f09d61c4740 1 -- 172.21.15.70:0/542735589 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f09c8034cd0 con 0x559d4b52c330 2024-04-01T01:42:44.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.033+0000 7f09d129e640 1 --2- 172.21.15.70:0/542735589 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x559d4b52c330 0x559d4b52e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.033+0000 7f09b4ff9640 10 client.28030 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:42:44.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.033+0000 7f09d129e640 1 --2- 172.21.15.70:0/542735589 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x559d4b52c330 0x559d4b52e720 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4198947c8a17eb5f server_cookie=2caf810339c08500 in_seq=0 out_seq=0 2024-04-01T01:42:44.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.034+0000 7f09b4ff9640 1 -- 172.21.15.70:0/542735589 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f09c8034cd0 con 0x559d4b52c330 2024-04-01T01:42:44.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.034+0000 7f09b4ff9640 10 client.28030 handle_command_reply: tid=0 2024-04-01T01:42:44.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.035+0000 7f09d61c4740 1 client.28030 shutdown 2024-04-01T01:42:44.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.035+0000 7f09d61c4740 1 -- 172.21.15.70:0/542735589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b506090 msgr2=0x559d4b521190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.035+0000 7f09d61c4740 1 --2- 172.21.15.70:0/542735589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b506090 0x559d4b521190 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x559d4b501b80 tx=0x7f09c8034e80 comp rx=0 tx=0).stop 2024-04-01T01:42:44.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.036+0000 7f09d61c4740 1 -- 172.21.15.70:0/542735589 shutdown_connections 2024-04-01T01:42:44.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.036+0000 7f09d61c4740 1 --2- 172.21.15.70:0/542735589 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x559d4b52c330 0x559d4b52e720 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.036+0000 7f09d61c4740 1 --2- 172.21.15.70:0/542735589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b50b200 0x559d4b5281d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.036+0000 7f09d61c4740 1 --2- 172.21.15.70:0/542735589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b506090 0x559d4b521190 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.036+0000 7f09d61c4740 1 --2- 172.21.15.70:0/542735589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b5056a0 0x559d4b520c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.036+0000 7f09d61c4740 1 -- 172.21.15.70:0/542735589 >> 172.21.15.70:0/542735589 conn(0x559d4b500c10 msgr2=0x559d4b4fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:44.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.036+0000 7f09b4ff9640 0 client.28030 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:42:44.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.036+0000 7f09d61c4740 1 -- 172.21.15.70:0/542735589 shutdown_connections 2024-04-01T01:42:44.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.036+0000 7f09d61c4740 1 -- 172.21.15.70:0/542735589 wait complete. 2024-04-01T01:42:44.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.036+0000 7f09d61c4740 20 client.28030 trim_cache size 0 max 16384 2024-04-01T01:42:44.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.065+0000 7f09d61c4740 1 Processor -- start 2024-04-01T01:42:44.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.065+0000 7f09d61c4740 1 -- start start 2024-04-01T01:42:44.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.066+0000 7f09d61c4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b5056a0 0x559d4b500ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.066+0000 7f09d61c4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b506090 0x559d4b528190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.066+0000 7f09d61c4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b50b200 0x559d4b520c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.066+0000 7f09d61c4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f09c414d9e0 con 0x559d4b5056a0 2024-04-01T01:42:44.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.066+0000 7f09d61c4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f09c414f520 con 0x559d4b506090 2024-04-01T01:42:44.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.066+0000 7f09d61c4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f09c414e0b0 con 0x559d4b50b200 2024-04-01T01:42:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.066+0000 7f09d1a9f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b50b200 0x559d4b520c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.066+0000 7f09d1a9f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b50b200 0x559d4b520c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40990/0 (socket says 172.21.15.70:40990) 2024-04-01T01:42:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.066+0000 7f09d1a9f640 1 -- 172.21.15.70:0/3801447515 learned_addr learned my addr 172.21.15.70:0/3801447515 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.066+0000 7f09d129e640 1 --2- 172.21.15.70:0/3801447515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b5056a0 0x559d4b500ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.067+0000 7f09d1a9f640 1 -- 172.21.15.70:0/3801447515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b506090 msgr2=0x559d4b528190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.067+0000 7f09d1a9f640 1 --2- 172.21.15.70:0/3801447515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b506090 0x559d4b528190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.067+0000 7f09d1a9f640 1 -- 172.21.15.70:0/3801447515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b5056a0 msgr2=0x559d4b500ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.067+0000 7f09d1a9f640 1 --2- 172.21.15.70:0/3801447515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b5056a0 0x559d4b500ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.067+0000 7f09d1a9f640 1 -- 172.21.15.70:0/3801447515 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d4b501980 con 0x559d4b50b200 2024-04-01T01:42:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.067+0000 7f09d1a9f640 1 --2- 172.21.15.70:0/3801447515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b50b200 0x559d4b520c50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f09cc064c40 tx=0x7f09cc0674f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b10107813125f10a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.067+0000 7f09b6ffd640 1 -- 172.21.15.70:0/3801447515 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc07c070 con 0x559d4b50b200 2024-04-01T01:42:44.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.067+0000 7f09b6ffd640 1 -- 172.21.15.70:0/3801447515 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09cc083b70 con 0x559d4b50b200 2024-04-01T01:42:44.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.067+0000 7f09b6ffd640 1 -- 172.21.15.70:0/3801447515 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc06c040 con 0x559d4b50b200 2024-04-01T01:42:44.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.068+0000 7f09d61c4740 1 -- 172.21.15.70:0/3801447515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b50b200 msgr2=0x559d4b520c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.068+0000 7f09d61c4740 1 --2- 172.21.15.70:0/3801447515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b50b200 0x559d4b520c50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f09cc064c40 tx=0x7f09cc0674f0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.068+0000 7f09d61c4740 1 -- 172.21.15.70:0/3801447515 shutdown_connections 2024-04-01T01:42:44.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.068+0000 7f09d61c4740 1 --2- 172.21.15.70:0/3801447515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b50b200 0x559d4b520c50 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.068+0000 7f09d61c4740 1 --2- 172.21.15.70:0/3801447515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b506090 0x559d4b528190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.068+0000 7f09d61c4740 1 --2- 172.21.15.70:0/3801447515 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b5056a0 0x559d4b500ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.068+0000 7f09d61c4740 1 -- 172.21.15.70:0/3801447515 >> 172.21.15.70:0/3801447515 conn(0x559d4b500c10 msgr2=0x559d4b51f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:44.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.069+0000 7f09d61c4740 1 -- 172.21.15.70:0/3801447515 shutdown_connections 2024-04-01T01:42:44.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.069+0000 7f09d61c4740 1 -- 172.21.15.70:0/3801447515 wait complete. 2024-04-01T01:42:44.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.069+0000 7f09d61c4740 1 -- start start 2024-04-01T01:42:44.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.069+0000 7f09d61c4740 1 Processor -- start 2024-04-01T01:42:44.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.069+0000 7f09d61c4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b5056a0 0x559d4b525160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.070+0000 7f09d61c4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b506090 0x559d4b5256a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.070+0000 7f09d129e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b5056a0 0x559d4b525160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.070+0000 7f09d129e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b5056a0 0x559d4b525160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47670/0 (socket says 172.21.15.70:47670) 2024-04-01T01:42:44.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.070+0000 7f09d129e640 1 -- 172.21.15.70:0/2388576149 learned_addr learned my addr 172.21.15.70:0/2388576149 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:44.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.070+0000 7f09d61c4740 1 --2- 172.21.15.70:0/2388576149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b50b200 0x559d4b590480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.070+0000 7f09d0a9d640 1 --2- 172.21.15.70:0/2388576149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b506090 0x559d4b5256a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.070+0000 7f09d61c4740 1 -- 172.21.15.70:0/2388576149 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f09c414ee50 con 0x559d4b50b200 2024-04-01T01:42:44.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.070+0000 7f09d1a9f640 1 --2- 172.21.15.70:0/2388576149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b50b200 0x559d4b590480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.070+0000 7f09d61c4740 1 -- 172.21.15.70:0/2388576149 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f09c414e780 con 0x559d4b5056a0 2024-04-01T01:42:44.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.071+0000 7f09d61c4740 1 -- 172.21.15.70:0/2388576149 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f09c40a3070 con 0x559d4b506090 2024-04-01T01:42:44.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.071+0000 7f09d61c4740 10 client.0 _ll_register_callbacks cb 0x7f09c80346f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:44.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.071+0000 7f09d0a9d640 1 -- 172.21.15.70:0/2388576149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b5056a0 msgr2=0x559d4b525160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.071+0000 7f09d0a9d640 1 --2- 172.21.15.70:0/2388576149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b5056a0 0x559d4b525160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.071+0000 7f09d0a9d640 1 -- 172.21.15.70:0/2388576149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b50b200 msgr2=0x559d4b590480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.071+0000 7f09d0a9d640 1 --2- 172.21.15.70:0/2388576149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b50b200 0x559d4b590480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.071+0000 7f09d0a9d640 1 -- 172.21.15.70:0/2388576149 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d4b5909f0 con 0x559d4b506090 2024-04-01T01:42:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.072+0000 7f09d0a9d640 1 --2- 172.21.15.70:0/2388576149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b506090 0x559d4b5256a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x559d4b4fc990 tx=0x7f09c8004de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c269853831af8218 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.072+0000 7f09b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:42:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.072+0000 7f09b4ff9640 1 -- 172.21.15.70:0/2388576149 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c8039070 con 0x559d4b506090 2024-04-01T01:42:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.072+0000 7f09b4ff9640 1 -- 172.21.15.70:0/2388576149 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f09c802c550 con 0x559d4b506090 2024-04-01T01:42:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.072+0000 7f09d61c4740 1 -- 172.21.15.70:0/2388576149 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09cc083b70 con 0x559d4b506090 2024-04-01T01:42:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.075+0000 7f09b4ff9640 1 -- 172.21.15.70:0/2388576149 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f09c8005850 con 0x559d4b506090 2024-04-01T01:42:44.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.075+0000 7f09d61c4740 10 client.28036 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:44.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.075+0000 7f09d61c4740 1 -- 172.21.15.70:0/2388576149 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559d4b591cf0 con 0x559d4b506090 2024-04-01T01:42:44.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.077+0000 7f09b4ff9640 1 -- 172.21.15.70:0/2388576149 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f09cc083b70 con 0x559d4b506090 2024-04-01T01:42:44.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.079+0000 7f09b4ff9640 1 -- 172.21.15.70:0/2388576149 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f09c802c7b0 con 0x559d4b506090 2024-04-01T01:42:44.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.079+0000 7f09d61c4740 10 client.28036 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:44.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.079+0000 7f09d61c4740 10 client.28036 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:42:44.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.079+0000 7f09d61c4740 20 client.28036 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:44.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.080+0000 7f09d61c4740 1 --2- 172.21.15.70:0/2388576149 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x559d4b592590 0x559d4b594980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.080+0000 7f09d61c4740 4 client.28036 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:42:44.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.080+0000 7f09d61c4740 1 -- 172.21.15.70:0/2388576149 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559d4b4776f0 con 0x559d4b592590 2024-04-01T01:42:44.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.080+0000 7f09d129e640 1 --2- 172.21.15.70:0/2388576149 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x559d4b592590 0x559d4b594980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.081+0000 7f09d129e640 1 --2- 172.21.15.70:0/2388576149 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x559d4b592590 0x559d4b594980 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cfb9f9483656d65 server_cookie=3febf7d03a0618a3 in_seq=0 out_seq=0 2024-04-01T01:42:44.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.081+0000 7f09b4ff9640 10 client.28036 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:42:44.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.081+0000 7f09b4ff9640 1 -- 172.21.15.70:0/2388576149 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559d4b4776f0 con 0x559d4b592590 2024-04-01T01:42:44.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.081+0000 7f09b4ff9640 10 client.28036 handle_command_reply: tid=0 2024-04-01T01:42:44.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.081+0000 7f09d61c4740 1 client.28036 shutdown 2024-04-01T01:42:44.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.085+0000 7f09d61c4740 1 -- 172.21.15.70:0/2388576149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b506090 msgr2=0x559d4b5256a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.085+0000 7f09d61c4740 1 --2- 172.21.15.70:0/2388576149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b506090 0x559d4b5256a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x559d4b4fc990 tx=0x7f09c8004de0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.086+0000 7f09d61c4740 1 -- 172.21.15.70:0/2388576149 shutdown_connections 2024-04-01T01:42:44.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.086+0000 7f09d61c4740 1 --2- 172.21.15.70:0/2388576149 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x559d4b592590 0x559d4b594980 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.086+0000 7f09d61c4740 1 --2- 172.21.15.70:0/2388576149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d4b50b200 0x559d4b590480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.087+0000 7f09d61c4740 1 --2- 172.21.15.70:0/2388576149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d4b506090 0x559d4b5256a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.087+0000 7f09b4ff9640 0 client.28036 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:42:44.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.087+0000 7f09d61c4740 1 --2- 172.21.15.70:0/2388576149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d4b5056a0 0x559d4b525160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.087+0000 7f09d61c4740 1 -- 172.21.15.70:0/2388576149 >> 172.21.15.70:0/2388576149 conn(0x559d4b500c10 msgr2=0x559d4b500ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:44.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.087+0000 7f09d61c4740 1 -- 172.21.15.70:0/2388576149 shutdown_connections 2024-04-01T01:42:44.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.088+0000 7f09d61c4740 1 -- 172.21.15.70:0/2388576149 wait complete. 2024-04-01T01:42:44.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.088+0000 7f09d61c4740 20 client.28036 trim_cache size 0 max 16384 2024-04-01T01:42:44.088 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:42:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.090+0000 7f09d22a0640 1 -- 172.21.15.70:0/948861087 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09b80a25a0 msgr2=0x7f09b80a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.090+0000 7f09d22a0640 1 --2- 172.21.15.70:0/948861087 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09b80a25a0 0x7f09b80a4a60 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f09cc067290 tx=0x7f09cc077040 comp rx=0 tx=0).stop 2024-04-01T01:42:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.090+0000 7f09d22a0640 1 -- 172.21.15.70:0/948861087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c40945f0 msgr2=0x7f09c4132660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.090+0000 7f09d22a0640 1 --2- 172.21.15.70:0/948861087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c40945f0 0x7f09c4132660 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00b4a0 tx=0x7f09bc00b8a0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.091+0000 7f09d22a0640 1 -- 172.21.15.70:0/948861087 shutdown_connections 2024-04-01T01:42:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.091+0000 7f09d22a0640 1 --2- 172.21.15.70:0/948861087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f09c4099910 0x7f09c4139c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.091+0000 7f09d22a0640 1 --2- 172.21.15.70:0/948861087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09c4096f60 0x7f09c4132ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.091+0000 7f09d22a0640 1 --2- 172.21.15.70:0/948861087 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09b80a25a0 0x7f09b80a4a60 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.091+0000 7f09d22a0640 1 --2- 172.21.15.70:0/948861087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c40945f0 0x7f09c4132660 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.091+0000 7f09d22a0640 1 -- 172.21.15.70:0/948861087 >> 172.21.15.70:0/948861087 conn(0x7f09c40901d0 msgr2=0x7f09c40954e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.091+0000 7f09d22a0640 1 -- 172.21.15.70:0/948861087 shutdown_connections 2024-04-01T01:42:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.092+0000 7f09d22a0640 1 -- 172.21.15.70:0/948861087 wait complete. 2024-04-01T01:42:44.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:43 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/699392872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:42:44.112 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:42:44.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:43 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/699392872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:42:44.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.289+0000 7f1a4ca3b640 1 -- 172.21.15.70:0/3744361210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a48100650 msgr2=0x7f1a48100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.289+0000 7f1a4ca3b640 1 --2- 172.21.15.70:0/3744361210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a48100650 0x7f1a48100a70 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f1a38009be0 tx=0x7f1a3802bf10 comp rx=0 tx=0).stop 2024-04-01T01:42:44.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.290+0000 7f1a4ca3b640 1 -- 172.21.15.70:0/3744361210 shutdown_connections 2024-04-01T01:42:44.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.290+0000 7f1a4ca3b640 1 --2- 172.21.15.70:0/3744361210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a48102200 0x7f1a48108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.290+0000 7f1a4ca3b640 1 --2- 172.21.15.70:0/3744361210 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a48101840 0x7f1a48101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.290+0000 7f1a4ca3b640 1 --2- 172.21.15.70:0/3744361210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a48100650 0x7f1a48100a70 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.290+0000 7f1a4ca3b640 1 -- 172.21.15.70:0/3744361210 >> 172.21.15.70:0/3744361210 conn(0x7f1a480fbbe0 msgr2=0x7f1a480fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:44.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.291+0000 7f1a4ca3b640 1 -- 172.21.15.70:0/3744361210 shutdown_connections 2024-04-01T01:42:44.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.291+0000 7f1a4ca3b640 1 -- 172.21.15.70:0/3744361210 wait complete. 2024-04-01T01:42:44.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.292+0000 7f1a4ca3b640 1 Processor -- start 2024-04-01T01:42:44.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.292+0000 7f1a4ca3b640 1 -- start start 2024-04-01T01:42:44.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.293+0000 7f1a4ca3b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a48100650 0x7f1a4810f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.293+0000 7f1a4ca3b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a48101840 0x7f1a4810fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.293+0000 7f1a4ca3b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a48102200 0x7f1a48110eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.293+0000 7f1a4ca3b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1a48109fa0 con 0x7f1a48100650 2024-04-01T01:42:44.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.293+0000 7f1a4ca3b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1a48109e20 con 0x7f1a48102200 2024-04-01T01:42:44.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.293+0000 7f1a4ca3b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4810a120 con 0x7f1a48101840 2024-04-01T01:42:44.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.293+0000 7f1a46d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a48102200 0x7f1a48110eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.293+0000 7f1a46d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a48102200 0x7f1a48110eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47690/0 (socket says 172.21.15.70:47690) 2024-04-01T01:42:44.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.293+0000 7f1a46d9d640 1 -- 172.21.15.70:0/1856601839 learned_addr learned my addr 172.21.15.70:0/1856601839 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:44.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.293+0000 7f1a4659c640 1 --2- 172.21.15.70:0/1856601839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a48100650 0x7f1a4810f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.294+0000 7f1a45d9b640 1 --2- 172.21.15.70:0/1856601839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a48101840 0x7f1a4810fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.294+0000 7f1a46d9d640 1 -- 172.21.15.70:0/1856601839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a48101840 msgr2=0x7f1a4810fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.294+0000 7f1a46d9d640 1 --2- 172.21.15.70:0/1856601839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a48101840 0x7f1a4810fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.294+0000 7f1a46d9d640 1 -- 172.21.15.70:0/1856601839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a48100650 msgr2=0x7f1a4810f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.294+0000 7f1a46d9d640 1 --2- 172.21.15.70:0/1856601839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a48100650 0x7f1a4810f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.294+0000 7f1a46d9d640 1 -- 172.21.15.70:0/1856601839 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a48111520 con 0x7f1a48102200 2024-04-01T01:42:44.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.294+0000 7f1a4659c640 1 --2- 172.21.15.70:0/1856601839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a48100650 0x7f1a4810f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:42:44.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.294+0000 7f1a46d9d640 1 --2- 172.21.15.70:0/1856601839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a48102200 0x7f1a48110eb0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c00e970 tx=0x7f1a3c00ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:44.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.295+0000 7f1a377fe640 1 -- 172.21.15.70:0/1856601839 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c00ccc0 con 0x7f1a48102200 2024-04-01T01:42:44.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.295+0000 7f1a377fe640 1 -- 172.21.15.70:0/1856601839 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a3c00ce60 con 0x7f1a48102200 2024-04-01T01:42:44.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.295+0000 7f1a4ca3b640 1 -- 172.21.15.70:0/1856601839 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a48111810 con 0x7f1a48102200 2024-04-01T01:42:44.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.295+0000 7f1a4ca3b640 1 -- 172.21.15.70:0/1856601839 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a48111d10 con 0x7f1a48102200 2024-04-01T01:42:44.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.295+0000 7f1a377fe640 1 -- 172.21.15.70:0/1856601839 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c010430 con 0x7f1a48102200 2024-04-01T01:42:44.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.299+0000 7f1a4f616740 1 Processor -- start 2024-04-01T01:42:44.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.300+0000 7f1a377fe640 1 -- 172.21.15.70:0/1856601839 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f1a3c0046c0 con 0x7f1a48102200 2024-04-01T01:42:44.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.300+0000 7f1a4f616740 1 -- start start 2024-04-01T01:42:44.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.300+0000 7f1a4f616740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cd490 0x55c24b6cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.300+0000 7f1a4f616740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6cde80 0x55c24b6d2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.300+0000 7f1a4f616740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6d2ff0 0x55c24b6d5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.300+0000 7f1a4f616740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c24b6321b0 con 0x55c24b6d2ff0 2024-04-01T01:42:44.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.300+0000 7f1a4f616740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c24b5dac00 con 0x55c24b6cde80 2024-04-01T01:42:44.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.300+0000 7f1a4f616740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c24b62e250 con 0x55c24b6cd490 2024-04-01T01:42:44.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.301+0000 7f1a4659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6cde80 0x55c24b6d2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.301+0000 7f1a4659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6cde80 0x55c24b6d2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47706/0 (socket says 172.21.15.70:47706) 2024-04-01T01:42:44.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.301+0000 7f1a4659c640 1 -- 172.21.15.70:0/3532400047 learned_addr learned my addr 172.21.15.70:0/3532400047 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.301+0000 7f1a45d9b640 1 --2- 172.21.15.70:0/3532400047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6d2ff0 0x55c24b6d5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.301+0000 7f1a45d9b640 1 --2- 172.21.15.70:0/3532400047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cd490 0x55c24b6cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.301+0000 7f1a4659c640 1 -- 172.21.15.70:0/3532400047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cd490 msgr2=0x55c24b6cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.301+0000 7f1a4659c640 1 --2- 172.21.15.70:0/3532400047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cd490 0x55c24b6cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.301+0000 7f1a4659c640 1 -- 172.21.15.70:0/3532400047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6d2ff0 msgr2=0x55c24b6d5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.301+0000 7f1a4659c640 1 --2- 172.21.15.70:0/3532400047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6d2ff0 0x55c24b6d5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.302+0000 7f1a4659c640 1 -- 172.21.15.70:0/3532400047 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c24b5fa3d0 con 0x55c24b6cde80 2024-04-01T01:42:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.302+0000 7f1a4659c640 1 --2- 172.21.15.70:0/3532400047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6cde80 0x55c24b6d2980 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1a38000f80 tx=0x7f1a38009280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d438ba72dfd2286b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.302+0000 7f1a377fe640 1 --2- 172.21.15.70:0/1856601839 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a300a2520 0x7f1a300a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.303+0000 7f1a34ff9640 1 -- 172.21.15.70:0/3532400047 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a38043070 con 0x55c24b6cde80 2024-04-01T01:42:44.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.303+0000 7f1a34ff9640 1 -- 172.21.15.70:0/3532400047 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a38003e20 con 0x55c24b6cde80 2024-04-01T01:42:44.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.303+0000 7f1a34ff9640 1 -- 172.21.15.70:0/3532400047 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3802ca90 con 0x55c24b6cde80 2024-04-01T01:42:44.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.303+0000 7f1a377fe640 1 -- 172.21.15.70:0/1856601839 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f1a3c0cf040 con 0x7f1a48102200 2024-04-01T01:42:44.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.304+0000 7f1a45d9b640 1 --2- 172.21.15.70:0/1856601839 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a300a2520 0x7f1a300a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.304+0000 7f1a4f616740 1 -- 172.21.15.70:0/3532400047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6cde80 msgr2=0x55c24b6d2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.304+0000 7f1a4f616740 1 --2- 172.21.15.70:0/3532400047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6cde80 0x55c24b6d2980 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f1a38000f80 tx=0x7f1a38009280 comp rx=0 tx=0).stop 2024-04-01T01:42:44.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.304+0000 7f1a4f616740 1 -- 172.21.15.70:0/3532400047 shutdown_connections 2024-04-01T01:42:44.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.304+0000 7f1a4f616740 1 --2- 172.21.15.70:0/3532400047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6d2ff0 0x55c24b6d5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.304+0000 7f1a4f616740 1 --2- 172.21.15.70:0/3532400047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6cde80 0x55c24b6d2980 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.304+0000 7f1a4f616740 1 --2- 172.21.15.70:0/3532400047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cd490 0x55c24b6cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.304+0000 7f1a4f616740 1 -- 172.21.15.70:0/3532400047 >> 172.21.15.70:0/3532400047 conn(0x55c24b6c8a00 msgr2=0x55c24b6c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:44.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.305+0000 7f1a45d9b640 1 --2- 172.21.15.70:0/1856601839 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a300a2520 0x7f1a300a49e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x55c24b6ce6e0 tx=0x7f1a40009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:44.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.308+0000 7f1a4f616740 1 -- 172.21.15.70:0/3532400047 shutdown_connections 2024-04-01T01:42:44.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.308+0000 7f1a4f616740 1 -- 172.21.15.70:0/3532400047 wait complete. 2024-04-01T01:42:44.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.308+0000 7f1a4f616740 1 -- start start 2024-04-01T01:42:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.308+0000 7f1a4f616740 1 Processor -- start 2024-04-01T01:42:44.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.309+0000 7f1a4f616740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cd490 0x55c24b6d77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.309+0000 7f1a4f616740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6cde80 0x55c24b6d7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.309+0000 7f1a4f616740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6d2ff0 0x55c24b6ded70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.309+0000 7f1a4659c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cd490 0x55c24b6d77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.309+0000 7f1a4659c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cd490 0x55c24b6d77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41022/0 (socket says 172.21.15.70:41022) 2024-04-01T01:42:44.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.310+0000 7f1a4659c640 1 -- 172.21.15.70:0/2794103185 learned_addr learned my addr 172.21.15.70:0/2794103185 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:44.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.310+0000 7f1a4f616740 1 -- 172.21.15.70:0/2794103185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c24b6d5f40 con 0x55c24b6cde80 2024-04-01T01:42:44.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.310+0000 7f1a46d9d640 1 --2- 172.21.15.70:0/2794103185 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6d2ff0 0x55c24b6ded70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.310+0000 7f1a4f616740 1 -- 172.21.15.70:0/2794103185 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c24b5d0470 con 0x55c24b6d2ff0 2024-04-01T01:42:44.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.310+0000 7f1a45d9b640 1 --2- 172.21.15.70:0/2794103185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6cde80 0x55c24b6d7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.310+0000 7f1a4f616740 1 -- 172.21.15.70:0/2794103185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c24b6d60c0 con 0x55c24b6cd490 2024-04-01T01:42:44.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.310+0000 7f1a46d9d640 1 -- 172.21.15.70:0/2794103185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cd490 msgr2=0x55c24b6d77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:44.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.310+0000 7f1a46d9d640 1 --2- 172.21.15.70:0/2794103185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cd490 0x55c24b6d77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.310+0000 7f1a46d9d640 1 -- 172.21.15.70:0/2794103185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6cde80 msgr2=0x55c24b6d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:44.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.310+0000 7f1a46d9d640 1 --2- 172.21.15.70:0/2794103185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6cde80 0x55c24b6d7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.310+0000 7f1a46d9d640 1 -- 172.21.15.70:0/2794103185 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c24b3463f0 con 0x55c24b6d2ff0 2024-04-01T01:42:44.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.311+0000 7f1a46d9d640 1 --2- 172.21.15.70:0/2794103185 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6d2ff0 0x55c24b6ded70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c00c6a0 tx=0x7f1a3c09bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38688105c83c9e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:44.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.311+0000 7f1a2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:44.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.311+0000 7f1a2dffb640 1 -- 172.21.15.70:0/2794103185 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3c0b3020 con 0x55c24b6d2ff0 2024-04-01T01:42:44.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.311+0000 7f1a2dffb640 1 -- 172.21.15.70:0/2794103185 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f1a3c0b4250 con 0x55c24b6d2ff0 2024-04-01T01:42:44.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.311+0000 7f1a4f616740 10 client.0 _ll_register_callbacks cb 0x55c24b5edcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:44.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.311+0000 7f1a4f616740 1 -- 172.21.15.70:0/2794103185 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a38003e20 con 0x55c24b6d2ff0 2024-04-01T01:42:44.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.312+0000 7f1a2dffb640 1 -- 172.21.15.70:0/2794103185 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1a3c0b9050 con 0x55c24b6d2ff0 2024-04-01T01:42:44.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.312+0000 7f1a4f616740 10 client.28235 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:44.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.312+0000 7f1a4f616740 1 -- 172.21.15.70:0/2794103185 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55c24b4107d0 con 0x55c24b6d2ff0 2024-04-01T01:42:44.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.312+0000 7f1a2dffb640 1 -- 172.21.15.70:0/2794103185 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f1a3c0ccbd0 con 0x55c24b6d2ff0 2024-04-01T01:42:44.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.312+0000 7f1a4f616740 10 client.28235 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:44.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.312+0000 7f1a4f616740 10 client.28235 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:44.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.313+0000 7f1a4f616740 20 client.28235 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:44.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.313+0000 7f1a4f616740 1 --2- 172.21.15.70:0/2794103185 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c24b6e3580 0x55c24b6e5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.313+0000 7f1a4f616740 4 client.28235 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:44.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.313+0000 7f1a4f616740 1 -- 172.21.15.70:0/2794103185 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1a38003e20 con 0x55c24b6e3580 2024-04-01T01:42:44.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.313+0000 7f1a4659c640 1 --2- 172.21.15.70:0/2794103185 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c24b6e3580 0x55c24b6e5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.313+0000 7f1a4659c640 1 --2- 172.21.15.70:0/2794103185 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c24b6e3580 0x55c24b6e5a40 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4863a3ece9b0fa5c server_cookie=60847b3f315e8875 in_seq=0 out_seq=0 2024-04-01T01:42:44.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.314+0000 7f1a2dffb640 10 client.28235 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:44.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.314+0000 7f1a2dffb640 1 -- 172.21.15.70:0/2794103185 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f1a38003e20 con 0x55c24b6e3580 2024-04-01T01:42:44.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.314+0000 7f1a2dffb640 10 client.28235 handle_command_reply: tid=0 2024-04-01T01:42:44.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.314+0000 7f1a4f616740 1 client.28235 shutdown 2024-04-01T01:42:44.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.318+0000 7f1a4f616740 1 -- 172.21.15.70:0/2794103185 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6d2ff0 msgr2=0x55c24b6ded70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.318+0000 7f1a4f616740 1 --2- 172.21.15.70:0/2794103185 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6d2ff0 0x55c24b6ded70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c00c6a0 tx=0x7f1a3c09bf10 comp rx=0 tx=0).stop 2024-04-01T01:42:44.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.318+0000 7f1a4f616740 1 -- 172.21.15.70:0/2794103185 shutdown_connections 2024-04-01T01:42:44.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.318+0000 7f1a4f616740 1 --2- 172.21.15.70:0/2794103185 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c24b6e3580 0x55c24b6e5a40 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.318+0000 7f1a4f616740 1 --2- 172.21.15.70:0/2794103185 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6d2ff0 0x55c24b6ded70 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.318+0000 7f1a4f616740 1 --2- 172.21.15.70:0/2794103185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6cde80 0x55c24b6d7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.318+0000 7f1a4f616740 1 --2- 172.21.15.70:0/2794103185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cd490 0x55c24b6d77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.318+0000 7f1a4f616740 1 -- 172.21.15.70:0/2794103185 >> 172.21.15.70:0/2794103185 conn(0x55c24b6c8a00 msgr2=0x55c24b6c7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:44.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.319+0000 7f1a2dffb640 0 client.28235 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:44.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.319+0000 7f1a4f616740 1 -- 172.21.15.70:0/2794103185 shutdown_connections 2024-04-01T01:42:44.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.319+0000 7f1a4f616740 1 -- 172.21.15.70:0/2794103185 wait complete. 2024-04-01T01:42:44.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.319+0000 7f1a4f616740 20 client.28235 trim_cache size 0 max 16384 2024-04-01T01:42:44.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:43 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/699392872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:42:44.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.342+0000 7f1a4f616740 1 Processor -- start 2024-04-01T01:42:44.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.343+0000 7f1a4f616740 1 -- start start 2024-04-01T01:42:44.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.343+0000 7f1a4f616740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6cd490 0x55c24b6d7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.343+0000 7f1a4f616740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cde80 0x55c24b6c74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.343+0000 7f1a4f616740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6d2ff0 0x55c24b6c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.343+0000 7f1a4f616740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1a481b9250 con 0x55c24b6cd490 2024-04-01T01:42:44.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.344+0000 7f1a45d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cde80 0x55c24b6c74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.344+0000 7f1a45d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cde80 0x55c24b6c74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41026/0 (socket says 172.21.15.70:41026) 2024-04-01T01:42:44.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.344+0000 7f1a45d9b640 1 -- 172.21.15.70:0/1265556790 learned_addr learned my addr 172.21.15.70:0/1265556790 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:44.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.344+0000 7f1a46d9d640 1 --2- 172.21.15.70:0/1265556790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6d2ff0 0x55c24b6c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.344+0000 7f1a4f616740 1 -- 172.21.15.70:0/1265556790 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1a481bad90 con 0x55c24b6d2ff0 2024-04-01T01:42:44.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.344+0000 7f1a4f616740 1 -- 172.21.15.70:0/1265556790 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1a481b9920 con 0x55c24b6cde80 2024-04-01T01:42:44.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.344+0000 7f1a4659c640 1 --2- 172.21.15.70:0/1265556790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6cd490 0x55c24b6d7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.344+0000 7f1a4659c640 1 -- 172.21.15.70:0/1265556790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cde80 msgr2=0x55c24b6c74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:44.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.344+0000 7f1a4659c640 1 --2- 172.21.15.70:0/1265556790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cde80 0x55c24b6c74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.344+0000 7f1a4659c640 1 -- 172.21.15.70:0/1265556790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6d2ff0 msgr2=0x55c24b6c79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:44.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.344+0000 7f1a4659c640 1 --2- 172.21.15.70:0/1265556790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6d2ff0 0x55c24b6c79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.344+0000 7f1a4659c640 1 -- 172.21.15.70:0/1265556790 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c24b6d24f0 con 0x55c24b6cd490 2024-04-01T01:42:44.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.345+0000 7f1a4659c640 1 --2- 172.21.15.70:0/1265556790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6cd490 0x55c24b6d7b10 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f1a38034ee0 tx=0x7f1a3802bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=764d6f60d26296a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:44.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.346+0000 7f1a2ffff640 1 -- 172.21.15.70:0/1265556790 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a38043040 con 0x55c24b6cd490 2024-04-01T01:42:44.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.346+0000 7f1a2ffff640 1 -- 172.21.15.70:0/1265556790 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a38052d80 con 0x55c24b6cd490 2024-04-01T01:42:44.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.346+0000 7f1a2ffff640 1 -- 172.21.15.70:0/1265556790 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a38033740 con 0x55c24b6cd490 2024-04-01T01:42:44.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.347+0000 7f1a4f616740 1 -- 172.21.15.70:0/1265556790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6cd490 msgr2=0x55c24b6d7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.347+0000 7f1a4f616740 1 --2- 172.21.15.70:0/1265556790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6cd490 0x55c24b6d7b10 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f1a38034ee0 tx=0x7f1a3802bf10 comp rx=0 tx=0).stop 2024-04-01T01:42:44.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.347+0000 7f1a4f616740 1 -- 172.21.15.70:0/1265556790 shutdown_connections 2024-04-01T01:42:44.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.347+0000 7f1a4f616740 1 --2- 172.21.15.70:0/1265556790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6d2ff0 0x55c24b6c79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.347+0000 7f1a4f616740 1 --2- 172.21.15.70:0/1265556790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cde80 0x55c24b6c74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.347+0000 7f1a4f616740 1 --2- 172.21.15.70:0/1265556790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6cd490 0x55c24b6d7b10 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.347+0000 7f1a4f616740 1 -- 172.21.15.70:0/1265556790 >> 172.21.15.70:0/1265556790 conn(0x55c24b6c8a00 msgr2=0x55c24b6d7400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:44.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.348+0000 7f1a4f616740 1 -- 172.21.15.70:0/1265556790 shutdown_connections 2024-04-01T01:42:44.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.348+0000 7f1a4f616740 1 -- 172.21.15.70:0/1265556790 wait complete. 2024-04-01T01:42:44.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.348+0000 7f1a4f616740 1 -- start start 2024-04-01T01:42:44.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.348+0000 7f1a4f616740 1 Processor -- start 2024-04-01T01:42:44.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.348+0000 7f1a4f616740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cd490 0x55c24b6dc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.349+0000 7f1a4f616740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6cde80 0x55c24b6dc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.349+0000 7f1a4f616740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6d2ff0 0x55c24b6dd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.349+0000 7f1a4f616740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1a481ba6c0 con 0x55c24b6d2ff0 2024-04-01T01:42:44.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.349+0000 7f1a4f616740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1a481b9ff0 con 0x55c24b6cde80 2024-04-01T01:42:44.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.349+0000 7f1a4f616740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1a481b71c0 con 0x55c24b6cd490 2024-04-01T01:42:44.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.349+0000 7f1a4f616740 10 client.0 _ll_register_callbacks cb 0x7f1a3c09b030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:44.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.351+0000 7f1a45d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6cde80 0x55c24b6dc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.351+0000 7f1a45d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6cde80 0x55c24b6dc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47738/0 (socket says 172.21.15.70:47738) 2024-04-01T01:42:44.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.351+0000 7f1a45d9b640 1 -- 172.21.15.70:0/3859556493 learned_addr learned my addr 172.21.15.70:0/3859556493 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:44.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.352+0000 7f1a45d9b640 1 -- 172.21.15.70:0/3859556493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cd490 msgr2=0x55c24b6dc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:44.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.352+0000 7f1a45d9b640 1 --2- 172.21.15.70:0/3859556493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cd490 0x55c24b6dc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.352+0000 7f1a45d9b640 1 -- 172.21.15.70:0/3859556493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6d2ff0 msgr2=0x55c24b6dd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:44.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.352+0000 7f1a45d9b640 1 --2- 172.21.15.70:0/3859556493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6d2ff0 0x55c24b6dd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.352+0000 7f1a45d9b640 1 -- 172.21.15.70:0/3859556493 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c24b6dde10 con 0x55c24b6cde80 2024-04-01T01:42:44.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.353+0000 7f1a45d9b640 1 --2- 172.21.15.70:0/3859556493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6cde80 0x55c24b6dc810 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1a4000fc10 tx=0x7f1a40009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98807913d31aa3c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:44.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.354+0000 7f1a2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:44.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.354+0000 7f1a2dffb640 1 -- 172.21.15.70:0/3859556493 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a40030060 con 0x55c24b6cde80 2024-04-01T01:42:44.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.355+0000 7f1a4f616740 1 -- 172.21.15.70:0/3859556493 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a38052d80 con 0x55c24b6cde80 2024-04-01T01:42:44.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.355+0000 7f1a2dffb640 1 -- 172.21.15.70:0/3859556493 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f1a40023dc0 con 0x55c24b6cde80 2024-04-01T01:42:44.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.355+0000 7f1a2dffb640 1 -- 172.21.15.70:0/3859556493 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1a40037050 con 0x55c24b6cde80 2024-04-01T01:42:44.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.355+0000 7f1a4f616740 10 client.28241 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:44.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.355+0000 7f1a4f616740 1 -- 172.21.15.70:0/3859556493 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c24b76bbb0 con 0x55c24b6cde80 2024-04-01T01:42:44.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.355+0000 7f1a2dffb640 1 -- 172.21.15.70:0/3859556493 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a38052d80 con 0x55c24b6cde80 2024-04-01T01:42:44.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.356+0000 7f1a2dffb640 1 -- 172.21.15.70:0/3859556493 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f1a4002a030 con 0x55c24b6cde80 2024-04-01T01:42:44.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.356+0000 7f1a4f616740 10 client.28241 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:44.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.356+0000 7f1a4f616740 10 client.28241 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:44.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.356+0000 7f1a4f616740 20 client.28241 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:44.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.356+0000 7f1a4f616740 1 --2- 172.21.15.70:0/3859556493 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c24b76c5b0 0x55c24b76ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:44.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.356+0000 7f1a4f616740 4 client.28241 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:42:44.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.356+0000 7f1a4f616740 1 -- 172.21.15.70:0/3859556493 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c24b63f6f0 con 0x55c24b76c5b0 2024-04-01T01:42:44.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.356+0000 7f1a4659c640 1 --2- 172.21.15.70:0/3859556493 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c24b76c5b0 0x55c24b76ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:44.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.357+0000 7f1a4659c640 1 --2- 172.21.15.70:0/3859556493 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c24b76c5b0 0x55c24b76ea30 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57fe9b55b91749b8 server_cookie=47dcd8b2c8bb4b68 in_seq=0 out_seq=0 2024-04-01T01:42:44.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.357+0000 7f1a2dffb640 10 client.28241 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:44.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.357+0000 7f1a2dffb640 1 -- 172.21.15.70:0/3859556493 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c24b63f6f0 con 0x55c24b76c5b0 2024-04-01T01:42:44.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.357+0000 7f1a2dffb640 10 client.28241 handle_command_reply: tid=0 2024-04-01T01:42:44.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.358+0000 7f1a4f616740 1 client.28241 shutdown 2024-04-01T01:42:44.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.358+0000 7f1a4f616740 1 -- 172.21.15.70:0/3859556493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6cde80 msgr2=0x55c24b6dc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.358+0000 7f1a4f616740 1 --2- 172.21.15.70:0/3859556493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6cde80 0x55c24b6dc810 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1a4000fc10 tx=0x7f1a40009260 comp rx=0 tx=0).stop 2024-04-01T01:42:44.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.358+0000 7f1a4f616740 1 -- 172.21.15.70:0/3859556493 shutdown_connections 2024-04-01T01:42:44.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.358+0000 7f1a4f616740 1 --2- 172.21.15.70:0/3859556493 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c24b76c5b0 0x55c24b76ea30 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.358+0000 7f1a4f616740 1 --2- 172.21.15.70:0/3859556493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c24b6d2ff0 0x55c24b6dd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.358+0000 7f1a2dffb640 0 client.28241 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:44.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.358+0000 7f1a4f616740 1 --2- 172.21.15.70:0/3859556493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c24b6cde80 0x55c24b6dc810 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.358+0000 7f1a4f616740 1 --2- 172.21.15.70:0/3859556493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c24b6cd490 0x55c24b6dc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.358+0000 7f1a4f616740 1 -- 172.21.15.70:0/3859556493 >> 172.21.15.70:0/3859556493 conn(0x55c24b6c8a00 msgr2=0x55c24b6cbc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:44.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.358+0000 7f1a4f616740 1 -- 172.21.15.70:0/3859556493 shutdown_connections 2024-04-01T01:42:44.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.358+0000 7f1a4f616740 1 -- 172.21.15.70:0/3859556493 wait complete. 2024-04-01T01:42:44.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.359+0000 7f1a4f616740 20 client.28241 trim_cache size 0 max 16384 2024-04-01T01:42:44.359 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:42:44.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.360+0000 7f1a4ca3b640 1 -- 172.21.15.70:0/1856601839 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a300a2520 msgr2=0x7f1a300a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.360+0000 7f1a4ca3b640 1 --2- 172.21.15.70:0/1856601839 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a300a2520 0x7f1a300a49e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x55c24b6ce6e0 tx=0x7f1a40009680 comp rx=0 tx=0).stop 2024-04-01T01:42:44.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.360+0000 7f1a4ca3b640 1 -- 172.21.15.70:0/1856601839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a48102200 msgr2=0x7f1a48110eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:44.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.360+0000 7f1a4ca3b640 1 --2- 172.21.15.70:0/1856601839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a48102200 0x7f1a48110eb0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c00e970 tx=0x7f1a3c00ed70 comp rx=0 tx=0).stop 2024-04-01T01:42:44.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.361+0000 7f1a4ca3b640 1 -- 172.21.15.70:0/1856601839 shutdown_connections 2024-04-01T01:42:44.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.361+0000 7f1a4ca3b640 1 --2- 172.21.15.70:0/1856601839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a48102200 0x7f1a48110eb0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.361+0000 7f1a4ca3b640 1 --2- 172.21.15.70:0/1856601839 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1a300a2520 0x7f1a300a49e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.361+0000 7f1a4ca3b640 1 --2- 172.21.15.70:0/1856601839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a48101840 0x7f1a4810fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.361+0000 7f1a4ca3b640 1 --2- 172.21.15.70:0/1856601839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a48100650 0x7f1a4810f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:44.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.361+0000 7f1a4ca3b640 1 -- 172.21.15.70:0/1856601839 >> 172.21.15.70:0/1856601839 conn(0x7f1a480fbbe0 msgr2=0x7f1a48104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:44.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.362+0000 7f1a4ca3b640 1 -- 172.21.15.70:0/1856601839 shutdown_connections 2024-04-01T01:42:44.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:44.362+0000 7f1a4ca3b640 1 -- 172.21.15.70:0/1856601839 wait complete. 2024-04-01T01:42:45.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:44 smithi189 ceph-mon[35753]: pgmap v637: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.4 MiB/s wr, 734 op/s 2024-04-01T01:42:45.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:44 smithi070 ceph-mon[32179]: pgmap v637: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.4 MiB/s wr, 734 op/s 2024-04-01T01:42:45.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:44 smithi081 ceph-mon[35142]: pgmap v637: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.4 MiB/s wr, 734 op/s 2024-04-01T01:42:45.382 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:42:45.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:42:45.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:42:45.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.563+0000 7fbff708f640 1 -- 172.21.15.70:0/1824432167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbff01052e0 msgr2=0x7fbff0105760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.563+0000 7fbff708f640 1 --2- 172.21.15.70:0/1824432167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbff01052e0 0x7fbff0105760 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fbfe001a910 tx=0x7fbfe003cca0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.563+0000 7fbff708f640 1 -- 172.21.15.70:0/1824432167 shutdown_connections 2024-04-01T01:42:45.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.563+0000 7fbff708f640 1 --2- 172.21.15.70:0/1824432167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbff01052e0 0x7fbff0105760 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.563+0000 7fbff708f640 1 --2- 172.21.15.70:0/1824432167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbff0102930 0x7fbff0104da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.563+0000 7fbff708f640 1 --2- 172.21.15.70:0/1824432167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbff00fffc0 0x7fbff01023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.563+0000 7fbff708f640 1 -- 172.21.15.70:0/1824432167 >> 172.21.15.70:0/1824432167 conn(0x7fbff00fbbe0 msgr2=0x7fbff00fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:45.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.564+0000 7fbff708f640 1 -- 172.21.15.70:0/1824432167 shutdown_connections 2024-04-01T01:42:45.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.564+0000 7fbff708f640 1 -- 172.21.15.70:0/1824432167 wait complete. 2024-04-01T01:42:45.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.565+0000 7fbff708f640 1 Processor -- start 2024-04-01T01:42:45.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.565+0000 7fbff708f640 1 -- start start 2024-04-01T01:42:45.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.565+0000 7fbff708f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbff00fffc0 0x7fbff006b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.565+0000 7fbff708f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbff0102930 0x7fbff006b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.565+0000 7fbff708f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbff01052e0 0x7fbff0072520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.565+0000 7fbff708f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbff0109160 con 0x7fbff00fffc0 2024-04-01T01:42:45.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.565+0000 7fbff708f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbff0108fe0 con 0x7fbff0102930 2024-04-01T01:42:45.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.565+0000 7fbff708f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbff01092e0 con 0x7fbff01052e0 2024-04-01T01:42:45.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.565+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbff0102930 0x7fbff006b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.565+0000 7fbff562c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbff01052e0 0x7fbff0072520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.566+0000 7fbff562c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbff01052e0 0x7fbff0072520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41046/0 (socket says 172.21.15.70:41046) 2024-04-01T01:42:45.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.566+0000 7fbff562c640 1 -- 172.21.15.70:0/3845162164 learned_addr learned my addr 172.21.15.70:0/3845162164 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:45.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.566+0000 7fbff4e2b640 1 --2- 172.21.15.70:0/3845162164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbff00fffc0 0x7fbff006b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.566+0000 7fbff4e2b640 1 -- 172.21.15.70:0/3845162164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbff01052e0 msgr2=0x7fbff0072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.566+0000 7fbff4e2b640 1 --2- 172.21.15.70:0/3845162164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbff01052e0 0x7fbff0072520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.566+0000 7fbff4e2b640 1 -- 172.21.15.70:0/3845162164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbff0102930 msgr2=0x7fbff006b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.566+0000 7fbff4e2b640 1 --2- 172.21.15.70:0/3845162164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbff0102930 0x7fbff006b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.566+0000 7fbff4e2b640 1 -- 172.21.15.70:0/3845162164 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff0072b90 con 0x7fbff00fffc0 2024-04-01T01:42:45.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.571+0000 7fbff4e2b640 1 --2- 172.21.15.70:0/3845162164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbff00fffc0 0x7fbff006b370 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800be20 tx=0x7fbfe8009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:45.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.572+0000 7fbfedffb640 1 -- 172.21.15.70:0/3845162164 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8019070 con 0x7fbff00fffc0 2024-04-01T01:42:45.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.572+0000 7fbfedffb640 1 -- 172.21.15.70:0/3845162164 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfe80148f0 con 0x7fbff00fffc0 2024-04-01T01:42:45.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.572+0000 7fbfedffb640 1 -- 172.21.15.70:0/3845162164 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe80128f0 con 0x7fbff00fffc0 2024-04-01T01:42:45.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.572+0000 7fbff708f640 1 -- 172.21.15.70:0/3845162164 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff0072e80 con 0x7fbff00fffc0 2024-04-01T01:42:45.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.573+0000 7fbff708f640 1 -- 172.21.15.70:0/3845162164 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff0073400 con 0x7fbff00fffc0 2024-04-01T01:42:45.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.578+0000 7fbff9c7c740 1 Processor -- start 2024-04-01T01:42:45.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.578+0000 7fbff9c7c740 1 -- start start 2024-04-01T01:42:45.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.579+0000 7fbff9c7c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9aa10 0x5581b4d9ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.579+0000 7fbff9c7c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9b400 0x5581b4d9ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.579+0000 7fbff9c7c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4da2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.579+0000 7fbff9c7c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5581b4c9c0b0 con 0x5581b4d9aa10 2024-04-01T01:42:45.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.579+0000 7fbff9c7c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5581b4cfeea0 con 0x5581b4da0570 2024-04-01T01:42:45.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.579+0000 7fbff9c7c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5581b4ca7c00 con 0x5581b4d9b400 2024-04-01T01:42:45.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.580+0000 7fbff562c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9aa10 0x5581b4d9ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.580+0000 7fbff562c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9aa10 0x5581b4d9ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52280/0 (socket says 172.21.15.70:52280) 2024-04-01T01:42:45.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.580+0000 7fbff562c640 1 -- 172.21.15.70:0/3106070167 learned_addr learned my addr 172.21.15.70:0/3106070167 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:45.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.581+0000 7fbfedffb640 1 -- 172.21.15.70:0/3845162164 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fbfe801c070 con 0x7fbff00fffc0 2024-04-01T01:42:45.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.581+0000 7fbff4e2b640 1 --2- 172.21.15.70:0/3106070167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9b400 0x5581b4d9ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.581+0000 7fbfeffff640 1 --2- 172.21.15.70:0/3106070167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4da2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.582+0000 7fbff4e2b640 1 -- 172.21.15.70:0/3106070167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 msgr2=0x5581b4da2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:45.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.582+0000 7fbff4e2b640 1 --2- 172.21.15.70:0/3106070167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4da2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.582+0000 7fbff4e2b640 1 -- 172.21.15.70:0/3106070167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9aa10 msgr2=0x5581b4d9ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:45.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.582+0000 7fbff4e2b640 1 --2- 172.21.15.70:0/3106070167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9aa10 0x5581b4d9ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.582+0000 7fbff4e2b640 1 -- 172.21.15.70:0/3106070167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581b4adb7e0 con 0x5581b4d9b400 2024-04-01T01:42:45.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.582+0000 7fbff4e2b640 1 --2- 172.21.15.70:0/3106070167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9b400 0x5581b4d9ff00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbfe80cb970 tx=0x7fbfe80d62d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10788593ef31ab54 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:45.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.582+0000 7fbfedffb640 1 --2- 172.21.15.70:0/3845162164 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbfd80a2620 0x7fbfd80a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.582+0000 7fbfd6ffd640 1 -- 172.21.15.70:0/3106070167 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe80e1070 con 0x5581b4d9b400 2024-04-01T01:42:45.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.582+0000 7fbfd6ffd640 1 -- 172.21.15.70:0/3106070167 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfe80ce6f0 con 0x5581b4d9b400 2024-04-01T01:42:45.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.582+0000 7fbfd6ffd640 1 -- 172.21.15.70:0/3106070167 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8096790 con 0x5581b4d9b400 2024-04-01T01:42:45.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.583+0000 7fbfedffb640 1 -- 172.21.15.70:0/3845162164 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fbfe80ce300 con 0x7fbff00fffc0 2024-04-01T01:42:45.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.583+0000 7fbff562c640 1 --2- 172.21.15.70:0/3845162164 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbfd80a2620 0x7fbfd80a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.583+0000 7fbff9c7c740 1 -- 172.21.15.70:0/3106070167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9b400 msgr2=0x5581b4d9ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.583+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/3106070167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9b400 0x5581b4d9ff00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbfe80cb970 tx=0x7fbfe80d62d0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.583+0000 7fbff562c640 1 --2- 172.21.15.70:0/3845162164 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbfd80a2620 0x7fbfd80a4ae0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0005c40 tx=0x7fbfe003f040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:45.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.584+0000 7fbff9c7c740 1 -- 172.21.15.70:0/3106070167 shutdown_connections 2024-04-01T01:42:45.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.584+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/3106070167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4da2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.584+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/3106070167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9b400 0x5581b4d9ff00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.584+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/3106070167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9aa10 0x5581b4d9ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.584+0000 7fbff9c7c740 1 -- 172.21.15.70:0/3106070167 >> 172.21.15.70:0/3106070167 conn(0x5581b4d95f80 msgr2=0x5581b4d96380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:45.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.584+0000 7fbff9c7c740 1 -- 172.21.15.70:0/3106070167 shutdown_connections 2024-04-01T01:42:45.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.584+0000 7fbff9c7c740 1 -- 172.21.15.70:0/3106070167 wait complete. 2024-04-01T01:42:45.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.584+0000 7fbff9c7c740 1 -- start start 2024-04-01T01:42:45.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.584+0000 7fbff9c7c740 1 Processor -- start 2024-04-01T01:42:45.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.585+0000 7fbff9c7c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9aa10 0x5581b4da4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.585+0000 7fbff9c7c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9b400 0x5581b4da5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.585+0000 7fbff9c7c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4dac3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.585+0000 7fbff9c7c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5581b4c9d3d0 con 0x5581b4d9b400 2024-04-01T01:42:45.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.585+0000 7fbff9c7c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5581b4cfaf40 con 0x5581b4da0570 2024-04-01T01:42:45.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.585+0000 7fbff9c7c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5581b4da3520 con 0x5581b4d9aa10 2024-04-01T01:42:45.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.585+0000 7fbff9c7c740 10 client.0 _ll_register_callbacks cb 0x5581b4cbab70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:45.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.585+0000 7fbff4e2b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9aa10 0x5581b4da4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.586+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9b400 0x5581b4da5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.586+0000 7fbff4e2b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9aa10 0x5581b4da4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41058/0 (socket says 172.21.15.70:41058) 2024-04-01T01:42:45.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.586+0000 7fbff4e2b640 1 -- 172.21.15.70:0/2687732736 learned_addr learned my addr 172.21.15.70:0/2687732736 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:45.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.585+0000 7fbff562c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4dac3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.586+0000 7fbff562c640 1 -- 172.21.15.70:0/2687732736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9aa10 msgr2=0x5581b4da4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:45.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.586+0000 7fbff562c640 1 --2- 172.21.15.70:0/2687732736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9aa10 0x5581b4da4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.586+0000 7fbff562c640 1 -- 172.21.15.70:0/2687732736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9b400 msgr2=0x5581b4da5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:45.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.586+0000 7fbff562c640 1 --2- 172.21.15.70:0/2687732736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9b400 0x5581b4da5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.586+0000 7fbff562c640 1 -- 172.21.15.70:0/2687732736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581b4b3bf70 con 0x5581b4da0570 2024-04-01T01:42:45.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.586+0000 7fbfeffff640 1 --2- 172.21.15.70:0/2687732736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9b400 0x5581b4da5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:42:45.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.587+0000 7fbff562c640 1 --2- 172.21.15.70:0/2687732736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4dac3d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0005940 tx=0x7fbfe001a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bde2ce07f309b1a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:45.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.587+0000 7fbfcffff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:45.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.587+0000 7fbfcffff640 1 -- 172.21.15.70:0/2687732736 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe005f070 con 0x5581b4da0570 2024-04-01T01:42:45.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.587+0000 7fbfcffff640 1 -- 172.21.15.70:0/2687732736 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fbfe0005440 con 0x5581b4da0570 2024-04-01T01:42:45.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.587+0000 7fbff9c7c740 1 -- 172.21.15.70:0/2687732736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfe80ce6f0 con 0x5581b4da0570 2024-04-01T01:42:45.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.589+0000 7fbfcffff640 1 -- 172.21.15.70:0/2687732736 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbfe0055070 con 0x5581b4da0570 2024-04-01T01:42:45.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.589+0000 7fbff9c7c740 10 client.28250 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:45.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.589+0000 7fbff9c7c740 1 -- 172.21.15.70:0/2687732736 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5581b49d3170 con 0x5581b4da0570 2024-04-01T01:42:45.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.596+0000 7fbfcffff640 1 -- 172.21.15.70:0/2687732736 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfe80ce6f0 con 0x5581b4da0570 2024-04-01T01:42:45.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.596+0000 7fbfcffff640 1 -- 172.21.15.70:0/2687732736 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fbfe00645f0 con 0x5581b4da0570 2024-04-01T01:42:45.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.596+0000 7fbff9c7c740 10 client.28250 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:45.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.596+0000 7fbff9c7c740 10 client.28250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:45.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.596+0000 7fbff9c7c740 20 client.28250 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:45.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.596+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/2687732736 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5581b4db0990 0x5581b4db2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.596+0000 7fbff9c7c740 4 client.28250 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:45.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.596+0000 7fbff9c7c740 1 -- 172.21.15.70:0/2687732736 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581b4d0c3e0 con 0x5581b4db0990 2024-04-01T01:42:45.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.598+0000 7fbff4e2b640 1 --2- 172.21.15.70:0/2687732736 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5581b4db0990 0x5581b4db2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.599+0000 7fbff4e2b640 1 --2- 172.21.15.70:0/2687732736 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5581b4db0990 0x5581b4db2df0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62ff1006a2efa32b server_cookie=db213849d76b94a7 in_seq=0 out_seq=0 2024-04-01T01:42:45.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.599+0000 7fbfcffff640 10 client.28250 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:45.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.600+0000 7fbfcffff640 1 -- 172.21.15.70:0/2687732736 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5581b4d0c3e0 con 0x5581b4db0990 2024-04-01T01:42:45.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.600+0000 7fbfcffff640 10 client.28250 handle_command_reply: tid=0 2024-04-01T01:42:45.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.600+0000 7fbff9c7c740 1 client.28250 shutdown 2024-04-01T01:42:45.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.601+0000 7fbff9c7c740 1 -- 172.21.15.70:0/2687732736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 msgr2=0x5581b4dac3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.601+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/2687732736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4dac3d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0005940 tx=0x7fbfe001a270 comp rx=0 tx=0).stop 2024-04-01T01:42:45.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.601+0000 7fbff9c7c740 1 -- 172.21.15.70:0/2687732736 shutdown_connections 2024-04-01T01:42:45.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.601+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/2687732736 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5581b4db0990 0x5581b4db2df0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.601+0000 7fbfcffff640 0 client.28250 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:45.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.601+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/2687732736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4dac3d0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.601+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/2687732736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9b400 0x5581b4da5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.601+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/2687732736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9aa10 0x5581b4da4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.601+0000 7fbff9c7c740 1 -- 172.21.15.70:0/2687732736 >> 172.21.15.70:0/2687732736 conn(0x5581b4d95f80 msgr2=0x5581b4d95180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:45.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.602+0000 7fbff9c7c740 1 -- 172.21.15.70:0/2687732736 shutdown_connections 2024-04-01T01:42:45.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.602+0000 7fbff9c7c740 1 -- 172.21.15.70:0/2687732736 wait complete. 2024-04-01T01:42:45.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.602+0000 7fbff9c7c740 20 client.28250 trim_cache size 0 max 16384 2024-04-01T01:42:45.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff9c7c740 1 Processor -- start 2024-04-01T01:42:45.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff9c7c740 1 -- start start 2024-04-01T01:42:45.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff9c7c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9aa10 0x5581b4d948f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff9c7c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9b400 0x5581b4dac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff9c7c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4d94e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff9c7c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbff01b0bd0 con 0x5581b4d9aa10 2024-04-01T01:42:45.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff9c7c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbff01b2710 con 0x5581b4da0570 2024-04-01T01:42:45.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff9c7c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbff01b12a0 con 0x5581b4d9b400 2024-04-01T01:42:45.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff562c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4d94e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff4e2b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9aa10 0x5581b4d948f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff562c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4d94e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47780/0 (socket says 172.21.15.70:47780) 2024-04-01T01:42:45.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9b400 0x5581b4dac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff562c640 1 -- 172.21.15.70:0/1487351936 learned_addr learned my addr 172.21.15.70:0/1487351936 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:45.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff562c640 1 -- 172.21.15.70:0/1487351936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9b400 msgr2=0x5581b4dac390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:45.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff562c640 1 --2- 172.21.15.70:0/1487351936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9b400 0x5581b4dac390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff562c640 1 -- 172.21.15.70:0/1487351936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9aa10 msgr2=0x5581b4d948f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:45.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff562c640 1 --2- 172.21.15.70:0/1487351936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9aa10 0x5581b4d948f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.637+0000 7fbff562c640 1 -- 172.21.15.70:0/1487351936 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581b4d9fc80 con 0x5581b4da0570 2024-04-01T01:42:45.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.638+0000 7fbff562c640 1 --2- 172.21.15.70:0/1487351936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4d94e30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0064bd0 tx=0x7fbfe003da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c2fe5b7e86756d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:45.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.638+0000 7fbfd67fc640 1 -- 172.21.15.70:0/1487351936 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe0075020 con 0x5581b4da0570 2024-04-01T01:42:45.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.638+0000 7fbfd67fc640 1 -- 172.21.15.70:0/1487351936 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfe0064660 con 0x5581b4da0570 2024-04-01T01:42:45.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.638+0000 7fbfd67fc640 1 -- 172.21.15.70:0/1487351936 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe005f040 con 0x5581b4da0570 2024-04-01T01:42:45.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.639+0000 7fbff9c7c740 1 -- 172.21.15.70:0/1487351936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 msgr2=0x5581b4d94e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.639+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/1487351936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4d94e30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0064bd0 tx=0x7fbfe003da60 comp rx=0 tx=0).stop 2024-04-01T01:42:45.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.639+0000 7fbff9c7c740 1 -- 172.21.15.70:0/1487351936 shutdown_connections 2024-04-01T01:42:45.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.639+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/1487351936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4d94e30 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.639+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/1487351936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9b400 0x5581b4dac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.639+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/1487351936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9aa10 0x5581b4d948f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.639+0000 7fbff9c7c740 1 -- 172.21.15.70:0/1487351936 >> 172.21.15.70:0/1487351936 conn(0x5581b4d95f80 msgr2=0x5581b4da3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:45.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.640+0000 7fbff9c7c740 1 -- 172.21.15.70:0/1487351936 shutdown_connections 2024-04-01T01:42:45.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.640+0000 7fbff9c7c740 1 -- 172.21.15.70:0/1487351936 wait complete. 2024-04-01T01:42:45.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.640+0000 7fbff9c7c740 1 -- start start 2024-04-01T01:42:45.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.640+0000 7fbff9c7c740 1 Processor -- start 2024-04-01T01:42:45.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.641+0000 7fbff9c7c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9aa10 0x5581b4da99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.641+0000 7fbff9c7c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9b400 0x5581b4da9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.641+0000 7fbff9c7c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4daaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.641+0000 7fbff9c7c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbff01b2040 con 0x5581b4d9b400 2024-04-01T01:42:45.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.641+0000 7fbff9c7c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbff01b1970 con 0x5581b4da0570 2024-04-01T01:42:45.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.642+0000 7fbff9c7c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbff01aebb0 con 0x5581b4d9aa10 2024-04-01T01:42:45.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.642+0000 7fbff9c7c740 10 client.0 _ll_register_callbacks cb 0x7fbfe00044d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:45.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.642+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9b400 0x5581b4da9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.642+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9b400 0x5581b4da9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52300/0 (socket says 172.21.15.70:52300) 2024-04-01T01:42:45.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.642+0000 7fbfeffff640 1 -- 172.21.15.70:0/612149893 learned_addr learned my addr 172.21.15.70:0/612149893 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:45.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.642+0000 7fbff562c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4daaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.642+0000 7fbff4e2b640 1 --2- 172.21.15.70:0/612149893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9aa10 0x5581b4da99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.643+0000 7fbff562c640 1 -- 172.21.15.70:0/612149893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9aa10 msgr2=0x5581b4da99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:45.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.643+0000 7fbff562c640 1 --2- 172.21.15.70:0/612149893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9aa10 0x5581b4da99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.643+0000 7fbff562c640 1 -- 172.21.15.70:0/612149893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9b400 msgr2=0x5581b4da9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:45.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.643+0000 7fbff562c640 1 --2- 172.21.15.70:0/612149893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9b400 0x5581b4da9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.643+0000 7fbff562c640 1 -- 172.21.15.70:0/612149893 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581b4dab4e0 con 0x5581b4da0570 2024-04-01T01:42:45.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.643+0000 7fbff562c640 1 --2- 172.21.15.70:0/612149893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4daaef0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbfe001a530 tx=0x7fbfe0003bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=729c6b742c352956 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:45.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.643+0000 7fbfcffff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:45.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.644+0000 7fbfcffff640 1 -- 172.21.15.70:0/612149893 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe007f060 con 0x5581b4da0570 2024-04-01T01:42:45.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.644+0000 7fbfcffff640 1 -- 172.21.15.70:0/612149893 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fbfe0064810 con 0x5581b4da0570 2024-04-01T01:42:45.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.644+0000 7fbff9c7c740 1 -- 172.21.15.70:0/612149893 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfe0064660 con 0x5581b4da0570 2024-04-01T01:42:45.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.644+0000 7fbfcffff640 1 -- 172.21.15.70:0/612149893 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbfe0075030 con 0x5581b4da0570 2024-04-01T01:42:45.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.646+0000 7fbff9c7c740 10 client.28256 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:45.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.646+0000 7fbff9c7c740 1 -- 172.21.15.70:0/612149893 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5581b4e16cf0 con 0x5581b4da0570 2024-04-01T01:42:45.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.650+0000 7fbfcffff640 1 -- 172.21.15.70:0/612149893 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbfe0064660 con 0x5581b4da0570 2024-04-01T01:42:45.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.650+0000 7fbfcffff640 1 -- 172.21.15.70:0/612149893 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fbfe0045440 con 0x5581b4da0570 2024-04-01T01:42:45.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.650+0000 7fbff9c7c740 10 client.28256 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:45.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.650+0000 7fbff9c7c740 10 client.28256 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:45.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.650+0000 7fbff9c7c740 20 client.28256 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:45.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.650+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/612149893 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5581b4db0990 0x5581b4e19520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.650+0000 7fbff9c7c740 4 client.28256 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:42:45.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.650+0000 7fbff9c7c740 1 -- 172.21.15.70:0/612149893 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5581b4d06730 con 0x5581b4db0990 2024-04-01T01:42:45.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.651+0000 7fbff4e2b640 1 --2- 172.21.15.70:0/612149893 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5581b4db0990 0x5581b4e19520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.656+0000 7fbff4e2b640 1 --2- 172.21.15.70:0/612149893 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5581b4db0990 0x5581b4e19520 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ccf49b226a266c6 server_cookie=d70a2a935cbd3634 in_seq=0 out_seq=0 2024-04-01T01:42:45.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.656+0000 7fbfcffff640 10 client.28256 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:45.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.658+0000 7fbfcffff640 1 -- 172.21.15.70:0/612149893 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5581b4d06730 con 0x5581b4db0990 2024-04-01T01:42:45.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.658+0000 7fbfcffff640 10 client.28256 handle_command_reply: tid=0 2024-04-01T01:42:45.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.658+0000 7fbff9c7c740 1 client.28256 shutdown 2024-04-01T01:42:45.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.658+0000 7fbff9c7c740 1 -- 172.21.15.70:0/612149893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 msgr2=0x5581b4daaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.658+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/612149893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4daaef0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbfe001a530 tx=0x7fbfe0003bb0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.658+0000 7fbff9c7c740 1 -- 172.21.15.70:0/612149893 shutdown_connections 2024-04-01T01:42:45.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.658+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/612149893 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5581b4db0990 0x5581b4e19520 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.658+0000 7fbfcffff640 0 client.28256 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:45.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.658+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/612149893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b4da0570 0x5581b4daaef0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.658+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/612149893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b4d9b400 0x5581b4da9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.658+0000 7fbff9c7c740 1 --2- 172.21.15.70:0/612149893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b4d9aa10 0x5581b4da99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.658+0000 7fbff9c7c740 1 -- 172.21.15.70:0/612149893 >> 172.21.15.70:0/612149893 conn(0x5581b4d95f80 msgr2=0x5581b4d991e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:45.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.658+0000 7fbff9c7c740 1 -- 172.21.15.70:0/612149893 shutdown_connections 2024-04-01T01:42:45.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.658+0000 7fbff9c7c740 1 -- 172.21.15.70:0/612149893 wait complete. 2024-04-01T01:42:45.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.658+0000 7fbff9c7c740 20 client.28256 trim_cache size 0 max 16384 2024-04-01T01:42:45.659 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:42:45.659 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:42:45.659 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "d03d710c-8301-4ed6-95d7-6715f08dfa7a", 2024-04-01T01:42:45.659 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:42:45.660 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:42:45.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.659+0000 7fbff708f640 1 -- 172.21.15.70:0/3845162164 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbfd80a2620 msgr2=0x7fbfd80a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.659+0000 7fbff708f640 1 --2- 172.21.15.70:0/3845162164 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbfd80a2620 0x7fbfd80a4ae0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0005c40 tx=0x7fbfe003f040 comp rx=0 tx=0).stop 2024-04-01T01:42:45.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.659+0000 7fbff708f640 1 -- 172.21.15.70:0/3845162164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbff00fffc0 msgr2=0x7fbff006b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.659+0000 7fbff708f640 1 --2- 172.21.15.70:0/3845162164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbff00fffc0 0x7fbff006b370 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800be20 tx=0x7fbfe8009f70 comp rx=0 tx=0).stop 2024-04-01T01:42:45.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.659+0000 7fbff708f640 1 -- 172.21.15.70:0/3845162164 shutdown_connections 2024-04-01T01:42:45.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.659+0000 7fbff708f640 1 --2- 172.21.15.70:0/3845162164 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbfd80a2620 0x7fbfd80a4ae0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.659+0000 7fbff708f640 1 --2- 172.21.15.70:0/3845162164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbff01052e0 0x7fbff0072520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.659+0000 7fbff708f640 1 --2- 172.21.15.70:0/3845162164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbff0102930 0x7fbff006b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.659+0000 7fbff708f640 1 --2- 172.21.15.70:0/3845162164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbff00fffc0 0x7fbff006b370 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.659+0000 7fbff708f640 1 -- 172.21.15.70:0/3845162164 >> 172.21.15.70:0/3845162164 conn(0x7fbff00fbbe0 msgr2=0x7fbff00fbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:45.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.660+0000 7fbff708f640 1 -- 172.21.15.70:0/3845162164 shutdown_connections 2024-04-01T01:42:45.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.660+0000 7fbff708f640 1 -- 172.21.15.70:0/3845162164 wait complete. 2024-04-01T01:42:45.671 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:42:45.672 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:42:45.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.830+0000 7f79b93a8640 1 -- 172.21.15.70:0/2516588063 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a402c600 con 0x7f79b4101840 2024-04-01T01:42:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.831+0000 7f79bc60e640 1 -- 172.21.15.70:0/2516588063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b4101840 msgr2=0x7f79b4101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.831+0000 7f79bc60e640 1 --2- 172.21.15.70:0/2516588063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b4101840 0x7f79b4101cc0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009900 tx=0x7f79a402bc60 comp rx=0 tx=0).stop 2024-04-01T01:42:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.831+0000 7f79bc60e640 1 -- 172.21.15.70:0/2516588063 shutdown_connections 2024-04-01T01:42:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.832+0000 7f79bc60e640 1 --2- 172.21.15.70:0/2516588063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79b4102200 0x7f79b4108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.832+0000 7f79bc60e640 1 --2- 172.21.15.70:0/2516588063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b4101840 0x7f79b4101cc0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.832+0000 7f79bc60e640 1 --2- 172.21.15.70:0/2516588063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79b4100650 0x7f79b4100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.832+0000 7f79bc60e640 1 -- 172.21.15.70:0/2516588063 >> 172.21.15.70:0/2516588063 conn(0x7f79b40fbbe0 msgr2=0x7f79b40fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.832+0000 7f79bc60e640 1 -- 172.21.15.70:0/2516588063 shutdown_connections 2024-04-01T01:42:45.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.832+0000 7f79bc60e640 1 -- 172.21.15.70:0/2516588063 wait complete. 2024-04-01T01:42:45.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.832+0000 7f79bc60e640 1 Processor -- start 2024-04-01T01:42:45.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.832+0000 7f79bc60e640 1 -- start start 2024-04-01T01:42:45.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.833+0000 7f79bc60e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79b4100650 0x7f79b41979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.833+0000 7f79bc60e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b4101840 0x7f79b4197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.833+0000 7f79bc60e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79b4102200 0x7f79b419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.833+0000 7f79bc60e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f79b410c1b0 con 0x7f79b4101840 2024-04-01T01:42:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.833+0000 7f79bc60e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f79b410c030 con 0x7f79b4100650 2024-04-01T01:42:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.833+0000 7f79bc60e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f79b410c330 con 0x7f79b4102200 2024-04-01T01:42:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.833+0000 7f79b9ba9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b4101840 0x7f79b4197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.833+0000 7f79babab640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79b4102200 0x7f79b419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.833+0000 7f79b9ba9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b4101840 0x7f79b4197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52318/0 (socket says 172.21.15.70:52318) 2024-04-01T01:42:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.833+0000 7f79b9ba9640 1 -- 172.21.15.70:0/4098415873 learned_addr learned my addr 172.21.15.70:0/4098415873 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:45.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.833+0000 7f79babab640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79b4102200 0x7f79b419ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41104/0 (socket says 172.21.15.70:41104) 2024-04-01T01:42:45.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.833+0000 7f79ba3aa640 1 --2- 172.21.15.70:0/4098415873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79b4100650 0x7f79b41979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.834+0000 7f79babab640 1 -- 172.21.15.70:0/4098415873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79b4100650 msgr2=0x7f79b41979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.834+0000 7f79babab640 1 --2- 172.21.15.70:0/4098415873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79b4100650 0x7f79b41979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.834+0000 7f79babab640 1 -- 172.21.15.70:0/4098415873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b4101840 msgr2=0x7f79b4197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.834+0000 7f79babab640 1 --2- 172.21.15.70:0/4098415873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b4101840 0x7f79b4197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.834+0000 7f79babab640 1 -- 172.21.15.70:0/4098415873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b4102b90 con 0x7f79b4102200 2024-04-01T01:42:45.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.834+0000 7f79babab640 1 --2- 172.21.15.70:0/4098415873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79b4102200 0x7f79b419ef90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f79ac009fd0 tx=0x7f79ac00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:45.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.835+0000 7f79ab7fe640 1 -- 172.21.15.70:0/4098415873 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ac00ce20 con 0x7f79b4102200 2024-04-01T01:42:45.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.835+0000 7f79ab7fe640 1 -- 172.21.15.70:0/4098415873 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79ac005020 con 0x7f79b4102200 2024-04-01T01:42:45.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.835+0000 7f79ab7fe640 1 -- 172.21.15.70:0/4098415873 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ac010690 con 0x7f79b4102200 2024-04-01T01:42:45.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.836+0000 7f79bc60e640 1 -- 172.21.15.70:0/4098415873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b4102e80 con 0x7f79b4102200 2024-04-01T01:42:45.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.836+0000 7f79bc60e640 1 -- 172.21.15.70:0/4098415873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b4103320 con 0x7f79b4102200 2024-04-01T01:42:45.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.842+0000 7f79bf1f3740 1 Processor -- start 2024-04-01T01:42:45.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.842+0000 7f79ab7fe640 1 -- 172.21.15.70:0/4098415873 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f79ac0051d0 con 0x7f79b4102200 2024-04-01T01:42:45.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.843+0000 7f79ab7fe640 1 --2- 172.21.15.70:0/4098415873 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79a00a2670 0x7f79a00a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.843+0000 7f79ab7fe640 1 -- 172.21.15.70:0/4098415873 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f79ac0cf120 con 0x7f79b4102200 2024-04-01T01:42:45.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.844+0000 7f79bf1f3740 1 -- start start 2024-04-01T01:42:45.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.844+0000 7f79bf1f3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b8d2b0 0x55f818b8d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.844+0000 7f79bf1f3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b8dca0 0x55f818b927a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.844+0000 7f79bf1f3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b92e10 0x55f818b952a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.844+0000 7f79bf1f3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f818af2180 con 0x55f818b92e10 2024-04-01T01:42:45.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.844+0000 7f79bf1f3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f818a9ab70 con 0x55f818b8d2b0 2024-04-01T01:42:45.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.844+0000 7f79bf1f3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f818aee220 con 0x55f818b8dca0 2024-04-01T01:42:45.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.845+0000 7f79babab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b92e10 0x55f818b952a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.845+0000 7f79babab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b92e10 0x55f818b952a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52330/0 (socket says 172.21.15.70:52330) 2024-04-01T01:42:45.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.845+0000 7f79babab640 1 -- 172.21.15.70:0/2404239889 learned_addr learned my addr 172.21.15.70:0/2404239889 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:45.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.845+0000 7f79b9ba9640 1 --2- 172.21.15.70:0/2404239889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b8dca0 0x55f818b927a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.845+0000 7f79b9ba9640 1 --2- 172.21.15.70:0/4098415873 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79a00a2670 0x7f79a00a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.845+0000 7f79babab640 1 -- 172.21.15.70:0/2404239889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b8dca0 msgr2=0x55f818b927a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:45.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.845+0000 7f79babab640 1 --2- 172.21.15.70:0/2404239889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b8dca0 0x55f818b927a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.845+0000 7f79babab640 1 -- 172.21.15.70:0/2404239889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b8d2b0 msgr2=0x55f818b8d6d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:42:45.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.845+0000 7f79babab640 1 --2- 172.21.15.70:0/2404239889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b8d2b0 0x55f818b8d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.845+0000 7f79babab640 1 -- 172.21.15.70:0/2404239889 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f818aae7c0 con 0x55f818b92e10 2024-04-01T01:42:45.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.845+0000 7f79babab640 1 --2- 172.21.15.70:0/2404239889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b92e10 0x55f818b952a0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f79ac09b740 tx=0x7f79ac09bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f012a1306de231e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:45.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.846+0000 7f79a8bfe640 1 -- 172.21.15.70:0/2404239889 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ac0b3020 con 0x55f818b92e10 2024-04-01T01:42:45.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.846+0000 7f79a8bfe640 1 -- 172.21.15.70:0/2404239889 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79ac0a41c0 con 0x55f818b92e10 2024-04-01T01:42:45.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.847+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2404239889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b92e10 msgr2=0x55f818b952a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.847+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2404239889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b92e10 0x55f818b952a0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f79ac09b740 tx=0x7f79ac09bf70 comp rx=0 tx=0).stop 2024-04-01T01:42:45.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.847+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2404239889 shutdown_connections 2024-04-01T01:42:45.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.847+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2404239889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b92e10 0x55f818b952a0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.847+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2404239889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b8dca0 0x55f818b927a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.847+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2404239889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b8d2b0 0x55f818b8d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.847+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2404239889 >> 172.21.15.70:0/2404239889 conn(0x55f818b88820 msgr2=0x55f818b88c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:45.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.850+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2404239889 shutdown_connections 2024-04-01T01:42:45.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.852+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2404239889 wait complete. 2024-04-01T01:42:45.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.852+0000 7f79bf1f3740 1 -- start start 2024-04-01T01:42:45.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.852+0000 7f79b9ba9640 1 --2- 172.21.15.70:0/4098415873 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79a00a2670 0x7f79a00a4b30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f79a4004610 tx=0x7f79a4004bb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:45.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.852+0000 7f79bf1f3740 1 Processor -- start 2024-04-01T01:42:45.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.853+0000 7f79bf1f3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b8d2b0 0x55f818b97740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.853+0000 7f79bf1f3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b97c80 0x55f818b9ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.853+0000 7f79bf1f3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b98100 0x55f818b9f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.853+0000 7f79bf1f3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f818b95e10 con 0x55f818b8d2b0 2024-04-01T01:42:45.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.853+0000 7f79bf1f3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f818a904a0 con 0x55f818b98100 2024-04-01T01:42:45.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.853+0000 7f79bf1f3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f818b95f90 con 0x55f818b97c80 2024-04-01T01:42:45.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.853+0000 7f79bf1f3740 10 client.0 _ll_register_callbacks cb 0x55f818a93d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:45.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.853+0000 7f79b9ba9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b97c80 0x55f818b9ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.853+0000 7f79ba3aa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b8d2b0 0x55f818b97740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.853+0000 7f79b9ba9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b97c80 0x55f818b9ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41118/0 (socket says 172.21.15.70:41118) 2024-04-01T01:42:45.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.854+0000 7f79b9ba9640 1 -- 172.21.15.70:0/2490854433 learned_addr learned my addr 172.21.15.70:0/2490854433 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:45.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.854+0000 7f79babab640 1 --2- 172.21.15.70:0/2490854433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b98100 0x55f818b9f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.854+0000 7f79b9ba9640 1 -- 172.21.15.70:0/2490854433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b98100 msgr2=0x55f818b9f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:45.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.854+0000 7f79b9ba9640 1 --2- 172.21.15.70:0/2490854433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b98100 0x55f818b9f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.854+0000 7f79b9ba9640 1 -- 172.21.15.70:0/2490854433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b8d2b0 msgr2=0x55f818b97740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:45.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.854+0000 7f79b9ba9640 1 --2- 172.21.15.70:0/2490854433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b8d2b0 0x55f818b97740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.854+0000 7f79b9ba9640 1 -- 172.21.15.70:0/2490854433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8188d07d0 con 0x55f818b97c80 2024-04-01T01:42:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.855+0000 7f79b9ba9640 1 --2- 172.21.15.70:0/2490854433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b97c80 0x55f818b9ccb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f79a4006010 tx=0x7f79a402bc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14b90c60aed6e37 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.855+0000 7f799dffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:42:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.855+0000 7f799dffb640 1 -- 172.21.15.70:0/2490854433 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a4005610 con 0x55f818b97c80 2024-04-01T01:42:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.855+0000 7f799dffb640 1 -- 172.21.15.70:0/2490854433 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f79a4039070 con 0x55f818b97c80 2024-04-01T01:42:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.856+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2490854433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f818aff6c0 con 0x55f818b97c80 2024-04-01T01:42:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.856+0000 7f799dffb640 1 -- 172.21.15.70:0/2490854433 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f79a4034440 con 0x55f818b97c80 2024-04-01T01:42:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.856+0000 7f79bf1f3740 10 client.28078 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.856+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2490854433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f8188063f0 con 0x55f818b97c80 2024-04-01T01:42:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.859+0000 7f799dffb640 1 -- 172.21.15.70:0/2490854433 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f818aff6c0 con 0x55f818b97c80 2024-04-01T01:42:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.859+0000 7f799dffb640 1 -- 172.21.15.70:0/2490854433 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f79a403f040 con 0x55f818b97c80 2024-04-01T01:42:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.860+0000 7f79bf1f3740 10 client.28078 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.860+0000 7f79bf1f3740 10 client.28078 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.860+0000 7f79bf1f3740 20 client.28078 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.860+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2490854433 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f818bb52b0 0x55f818bb76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.860+0000 7f79bf1f3740 4 client.28078 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.860+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2490854433 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f818af9a10 con 0x55f818bb52b0 2024-04-01T01:42:45.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.864+0000 7f79ba3aa640 1 --2- 172.21.15.70:0/2490854433 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f818bb52b0 0x55f818bb76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.868+0000 7f79ba3aa640 1 --2- 172.21.15.70:0/2490854433 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f818bb52b0 0x55f818bb76d0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=956c7ec4b63dd8c3 server_cookie=535e11bf3c477bf8 in_seq=0 out_seq=0 2024-04-01T01:42:45.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.869+0000 7f799dffb640 10 client.28078 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:45.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.871+0000 7f799dffb640 1 -- 172.21.15.70:0/2490854433 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f79b0009d50 con 0x55f818bb52b0 2024-04-01T01:42:45.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.871+0000 7f799dffb640 10 client.28078 handle_command_reply: tid=0 2024-04-01T01:42:45.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.871+0000 7f79bf1f3740 1 client.28078 shutdown 2024-04-01T01:42:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.874+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2490854433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b97c80 msgr2=0x55f818b9ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.874+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2490854433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b97c80 0x55f818b9ccb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f79a4006010 tx=0x7f79a402bc60 comp rx=0 tx=0).stop 2024-04-01T01:42:45.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.878+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2490854433 shutdown_connections 2024-04-01T01:42:45.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.878+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2490854433 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f818bb52b0 0x55f818bb76d0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.878+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2490854433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b98100 0x55f818b9f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.878+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2490854433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b97c80 0x55f818b9ccb0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.878+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2490854433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b8d2b0 0x55f818b97740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.878+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2490854433 >> 172.21.15.70:0/2490854433 conn(0x55f818b88820 msgr2=0x55f818b87c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:45.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.878+0000 7f799dffb640 0 client.28078 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:45.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.880+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2490854433 shutdown_connections 2024-04-01T01:42:45.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.880+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2490854433 wait complete. 2024-04-01T01:42:45.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.880+0000 7f79bf1f3740 20 client.28078 trim_cache size 0 max 16384 2024-04-01T01:42:45.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.915+0000 7f79bf1f3740 1 Processor -- start 2024-04-01T01:42:45.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.915+0000 7f79bf1f3740 1 -- start start 2024-04-01T01:42:45.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.916+0000 7f79bf1f3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b8d2b0 0x55f818b88c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.916+0000 7f79bf1f3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b97c80 0x55f818bbcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.916+0000 7f79ba3aa640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b8d2b0 0x55f818b88c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.916+0000 7f79ba3aa640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b8d2b0 0x55f818b88c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41120/0 (socket says 172.21.15.70:41120) 2024-04-01T01:42:45.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.917+0000 7f79ba3aa640 1 -- 172.21.15.70:0/2932325613 learned_addr learned my addr 172.21.15.70:0/2932325613 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:45.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.917+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2932325613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b98100 0x55f818b9ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.917+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2932325613 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f79b406faf0 con 0x55f818b97c80 2024-04-01T01:42:45.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.917+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2932325613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f79b4071630 con 0x55f818b98100 2024-04-01T01:42:45.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.917+0000 7f79babab640 1 --2- 172.21.15.70:0/2932325613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b98100 0x55f818b9ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.917+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2932325613 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f79b40701c0 con 0x55f818b8d2b0 2024-04-01T01:42:45.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.918+0000 7f79b9ba9640 1 --2- 172.21.15.70:0/2932325613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b97c80 0x55f818bbcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.918+0000 7f79b9ba9640 1 -- 172.21.15.70:0/2932325613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b8d2b0 msgr2=0x55f818b88c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:45.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.918+0000 7f79b9ba9640 1 --2- 172.21.15.70:0/2932325613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b8d2b0 0x55f818b88c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.918+0000 7f79b9ba9640 1 -- 172.21.15.70:0/2932325613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b98100 msgr2=0x55f818b9ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:45.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.918+0000 7f79b9ba9640 1 --2- 172.21.15.70:0/2932325613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b98100 0x55f818b9ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.918+0000 7f79b9ba9640 1 -- 172.21.15.70:0/2932325613 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8187c60f0 con 0x55f818b97c80 2024-04-01T01:42:45.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.918+0000 7f79b9ba9640 1 --2- 172.21.15.70:0/2932325613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b97c80 0x55f818bbcfe0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f79a40058a0 tx=0x7f79a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b741bd7b9cf9c15d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:45.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.920+0000 7f799ffff640 1 -- 172.21.15.70:0/2932325613 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a4005610 con 0x55f818b97c80 2024-04-01T01:42:45.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.921+0000 7f799ffff640 1 -- 172.21.15.70:0/2932325613 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79a4040400 con 0x55f818b97c80 2024-04-01T01:42:45.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.921+0000 7f799ffff640 1 -- 172.21.15.70:0/2932325613 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a405c440 con 0x55f818b97c80 2024-04-01T01:42:45.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.922+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2932325613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b97c80 msgr2=0x55f818bbcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.922+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2932325613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b97c80 0x55f818bbcfe0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f79a40058a0 tx=0x7f79a40046c0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.923+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2932325613 shutdown_connections 2024-04-01T01:42:45.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.923+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2932325613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b98100 0x55f818b9ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.923+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2932325613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b97c80 0x55f818bbcfe0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.923+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2932325613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b8d2b0 0x55f818b88c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.923+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2932325613 >> 172.21.15.70:0/2932325613 conn(0x55f818b88820 msgr2=0x55f818b83ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:45.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.924+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2932325613 shutdown_connections 2024-04-01T01:42:45.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.925+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2932325613 wait complete. 2024-04-01T01:42:45.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.926+0000 7f79bf1f3740 1 -- start start 2024-04-01T01:42:45.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.926+0000 7f79bf1f3740 1 Processor -- start 2024-04-01T01:42:45.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.926+0000 7f79bf1f3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b8d2b0 0x55f818b9bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.926+0000 7f79bf1f3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b97c80 0x55f818b9c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.927+0000 7f79ba3aa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b8d2b0 0x55f818b9bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.927+0000 7f79ba3aa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b8d2b0 0x55f818b9bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47830/0 (socket says 172.21.15.70:47830) 2024-04-01T01:42:45.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.927+0000 7f79ba3aa640 1 -- 172.21.15.70:0/2223848373 learned_addr learned my addr 172.21.15.70:0/2223848373 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:45.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.927+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2223848373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b98100 0x55f818c19600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.927+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2223848373 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f79b4070f60 con 0x55f818b97c80 2024-04-01T01:42:45.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.927+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2223848373 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f79b4070890 con 0x55f818b8d2b0 2024-04-01T01:42:45.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.927+0000 7f79babab640 1 --2- 172.21.15.70:0/2223848373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b98100 0x55f818c19600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.927+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2223848373 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f79b406dad0 con 0x55f818b98100 2024-04-01T01:42:45.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.927+0000 7f79bf1f3740 10 client.0 _ll_register_callbacks cb 0x7f79ac0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:45.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.928+0000 7f79ba3aa640 1 -- 172.21.15.70:0/2223848373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b98100 msgr2=0x55f818c19600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:45.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.928+0000 7f79ba3aa640 1 --2- 172.21.15.70:0/2223848373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b98100 0x55f818c19600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.928+0000 7f79ba3aa640 1 -- 172.21.15.70:0/2223848373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b97c80 msgr2=0x55f818b9c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:45.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.928+0000 7f79b9ba9640 1 --2- 172.21.15.70:0/2223848373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b97c80 0x55f818b9c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.928+0000 7f79ba3aa640 1 --2- 172.21.15.70:0/2223848373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b97c80 0x55f818b9c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.928+0000 7f79ba3aa640 1 -- 172.21.15.70:0/2223848373 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f818b9f310 con 0x55f818b8d2b0 2024-04-01T01:42:45.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.928+0000 7f79b9ba9640 1 --2- 172.21.15.70:0/2223848373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b97c80 0x55f818b9c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:42:45.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.929+0000 7f79ba3aa640 1 --2- 172.21.15.70:0/2223848373 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b8d2b0 0x55f818b9bda0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009e30 tx=0x7f79b001df60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2bb3c585a032d8c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:45.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.929+0000 7f799dffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:45.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.929+0000 7f799dffb640 1 -- 172.21.15.70:0/2223848373 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b0004050 con 0x55f818b8d2b0 2024-04-01T01:42:45.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.929+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2223848373 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79a4040400 con 0x55f818b8d2b0 2024-04-01T01:42:45.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.929+0000 7f799dffb640 1 -- 172.21.15.70:0/2223848373 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f79b00041f0 con 0x55f818b8d2b0 2024-04-01T01:42:45.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.930+0000 7f799dffb640 1 -- 172.21.15.70:0/2223848373 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f79b002c050 con 0x55f818b8d2b0 2024-04-01T01:42:45.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.930+0000 7f79bf1f3740 10 client.28271 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:45.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.930+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2223848373 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f818c1aa60 con 0x55f818b8d2b0 2024-04-01T01:42:45.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.931+0000 7f799dffb640 1 -- 172.21.15.70:0/2223848373 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79a4040400 con 0x55f818b8d2b0 2024-04-01T01:42:45.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.931+0000 7f799dffb640 1 -- 172.21.15.70:0/2223848373 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f79b0004730 con 0x55f818b8d2b0 2024-04-01T01:42:45.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.931+0000 7f79bf1f3740 10 client.28271 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:45.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.931+0000 7f79bf1f3740 10 client.28271 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:45.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.931+0000 7f79bf1f3740 20 client.28271 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:45.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.931+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2223848373 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f818c1b460 0x55f818c1d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:45.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.931+0000 7f79bf1f3740 4 client.28271 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:42:45.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.931+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2223848373 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f818a77540 con 0x55f818c1b460 2024-04-01T01:42:45.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.931+0000 7f79b9ba9640 1 --2- 172.21.15.70:0/2223848373 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f818c1b460 0x55f818c1d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:45.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.932+0000 7f799dffb640 10 client.28271 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:45.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.932+0000 7f79b9ba9640 1 --2- 172.21.15.70:0/2223848373 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f818c1b460 0x55f818c1d880 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92a810a3651adbbd server_cookie=1efbbd6746b117da in_seq=0 out_seq=0 2024-04-01T01:42:45.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.932+0000 7f799dffb640 1 -- 172.21.15.70:0/2223848373 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55f818a77540 con 0x55f818c1b460 2024-04-01T01:42:45.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.932+0000 7f799dffb640 10 client.28271 handle_command_reply: tid=0 2024-04-01T01:42:45.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.932+0000 7f79bf1f3740 1 client.28271 shutdown 2024-04-01T01:42:45.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.934+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2223848373 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b8d2b0 msgr2=0x55f818b9bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.934+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2223848373 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b8d2b0 0x55f818b9bda0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009e30 tx=0x7f79b001df60 comp rx=0 tx=0).stop 2024-04-01T01:42:45.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.934+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2223848373 shutdown_connections 2024-04-01T01:42:45.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.935+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2223848373 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f818c1b460 0x55f818c1d880 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.935+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2223848373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f818b98100 0x55f818c19600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.935+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2223848373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f818b97c80 0x55f818b9c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.935+0000 7f79bf1f3740 1 --2- 172.21.15.70:0/2223848373 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f818b8d2b0 0x55f818b9bda0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.935+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2223848373 >> 172.21.15.70:0/2223848373 conn(0x55f818b88820 msgr2=0x55f818b92190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:45.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.935+0000 7f799dffb640 0 client.28271 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:45.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.935+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2223848373 shutdown_connections 2024-04-01T01:42:45.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.935+0000 7f79bf1f3740 1 -- 172.21.15.70:0/2223848373 wait complete. 2024-04-01T01:42:45.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.935+0000 7f79bf1f3740 20 client.28271 trim_cache size 0 max 16384 2024-04-01T01:42:45.936 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:42:45.936 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "scrub active (108 inodes in the stack)", 2024-04-01T01:42:45.936 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:42:45.937 INFO:teuthology.orchestra.run.smithi070.stdout: "d03d710c-8301-4ed6-95d7-6715f08dfa7a": { 2024-04-01T01:42:45.937 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:42:45.937 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "d03d710c-8301-4ed6-95d7-6715f08dfa7a", 2024-04-01T01:42:45.937 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:42:45.937 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:42:45.937 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:42:45.937 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:42:45.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.938+0000 7f79bc60e640 1 -- 172.21.15.70:0/4098415873 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79a00a2670 msgr2=0x7f79a00a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.938+0000 7f79bc60e640 1 --2- 172.21.15.70:0/4098415873 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79a00a2670 0x7f79a00a4b30 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f79a4004610 tx=0x7f79a4004bb0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.938+0000 7f79bc60e640 1 -- 172.21.15.70:0/4098415873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79b4102200 msgr2=0x7f79b419ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:45.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.938+0000 7f79bc60e640 1 --2- 172.21.15.70:0/4098415873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79b4102200 0x7f79b419ef90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f79ac009fd0 tx=0x7f79ac00ef10 comp rx=0 tx=0).stop 2024-04-01T01:42:45.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.938+0000 7f79bc60e640 1 -- 172.21.15.70:0/4098415873 shutdown_connections 2024-04-01T01:42:45.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.938+0000 7f79bc60e640 1 --2- 172.21.15.70:0/4098415873 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f79a00a2670 0x7f79a00a4b30 secure :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f79a4004610 tx=0x7f79a4004bb0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.938+0000 7f79bc60e640 1 --2- 172.21.15.70:0/4098415873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79b4102200 0x7f79b419ef90 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.938+0000 7f79bc60e640 1 --2- 172.21.15.70:0/4098415873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b4101840 0x7f79b4197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.938+0000 7f79bc60e640 1 --2- 172.21.15.70:0/4098415873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79b4100650 0x7f79b41979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:45.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.938+0000 7f79bc60e640 1 -- 172.21.15.70:0/4098415873 >> 172.21.15.70:0/4098415873 conn(0x7f79b40fbbe0 msgr2=0x7f79b4068920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:45.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.941+0000 7f79bc60e640 1 -- 172.21.15.70:0/4098415873 shutdown_connections 2024-04-01T01:42:45.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:42:45.942+0000 7f79bc60e640 1 -- 172.21.15.70:0/4098415873 wait complete. 2024-04-01T01:42:45.959 INFO:tasks.cephfs.filesystem:scrub status for tag:d03d710c-8301-4ed6-95d7-6715f08dfa7a - {'path': '/', 'tag': 'd03d710c-8301-4ed6-95d7-6715f08dfa7a', 'options': 'recursive,force'} 2024-04-01T01:42:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 480.0 s, 772.2 tps, lat 127.886 ms stddev 111.007 2024-04-01T01:42:46.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:46 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:42:46.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:46 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:42:46.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:46 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:42:46.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:46 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:42:46.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:46 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:42:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:46 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:42:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:46 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:42:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:46 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:42:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:46 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:42:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:46 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:42:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:46 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:42:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:46 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:42:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:46 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:42:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:46 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:42:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:46 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:42:47.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:47 smithi189 ceph-mon[35753]: pgmap v638: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 7.1 MiB/s wr, 831 op/s 2024-04-01T01:42:48.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:47 smithi070 ceph-mon[32179]: pgmap v638: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 7.1 MiB/s wr, 831 op/s 2024-04-01T01:42:48.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:47 smithi081 ceph-mon[35142]: pgmap v638: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 7.1 MiB/s wr, 831 op/s 2024-04-01T01:42:49.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:48 smithi189 ceph-mon[35753]: pgmap v639: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 6.8 MiB/s wr, 848 op/s 2024-04-01T01:42:49.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:48 smithi070 ceph-mon[32179]: pgmap v639: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 6.8 MiB/s wr, 848 op/s 2024-04-01T01:42:49.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:48 smithi081 ceph-mon[35142]: pgmap v639: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 6.8 MiB/s wr, 848 op/s 2024-04-01T01:42:51.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:50 smithi189 ceph-mon[35753]: pgmap v640: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 6.4 MiB/s wr, 825 op/s 2024-04-01T01:42:51.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:50 smithi070 ceph-mon[32179]: pgmap v640: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 6.4 MiB/s wr, 825 op/s 2024-04-01T01:42:51.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:50 smithi081 ceph-mon[35142]: pgmap v640: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 6.4 MiB/s wr, 825 op/s 2024-04-01T01:42:51.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 485.0 s, 844.8 tps, lat 119.512 ms stddev 135.599 2024-04-01T01:42:53.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:52 smithi189 ceph-mon[35753]: pgmap v641: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 7.5 MiB/s wr, 925 op/s 2024-04-01T01:42:53.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:52 smithi070 ceph-mon[32179]: pgmap v641: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 7.5 MiB/s wr, 925 op/s 2024-04-01T01:42:53.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:52 smithi081 ceph-mon[35142]: pgmap v641: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 7.5 MiB/s wr, 925 op/s 2024-04-01T01:42:54.012 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.010+0000 7fcd0a15e640 1 -- 172.21.15.70:0/2631224236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd0410de20 msgr2=0x7fcd041102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:54.012 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.010+0000 7fcd0a15e640 1 --2- 172.21.15.70:0/2631224236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd0410de20 0x7fcd041102e0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fccfc009ca0 tx=0x7fccfc02c100 comp rx=0 tx=0).stop 2024-04-01T01:42:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.014+0000 7fcd0a15e640 1 -- 172.21.15.70:0/2631224236 shutdown_connections 2024-04-01T01:42:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.014+0000 7fcd0a15e640 1 --2- 172.21.15.70:0/2631224236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd0410de20 0x7fcd041102e0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.014+0000 7fcd0a15e640 1 --2- 172.21.15.70:0/2631224236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd0410b380 0x7fcd0410d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.014+0000 7fcd0a15e640 1 --2- 172.21.15.70:0/2631224236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd041089c0 0x7fcd0410adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.014+0000 7fcd0a15e640 1 -- 172.21.15.70:0/2631224236 >> 172.21.15.70:0/2631224236 conn(0x7fcd0406e950 msgr2=0x7fcd04070d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.015+0000 7fcd0a15e640 1 -- 172.21.15.70:0/2631224236 shutdown_connections 2024-04-01T01:42:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.015+0000 7fcd0a15e640 1 -- 172.21.15.70:0/2631224236 wait complete. 2024-04-01T01:42:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.015+0000 7fcd0a15e640 1 Processor -- start 2024-04-01T01:42:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.015+0000 7fcd0a15e640 1 -- start start 2024-04-01T01:42:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.016+0000 7fcd0a15e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd041089c0 0x7fcd041a2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.016+0000 7fcd0a15e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd0410b380 0x7fcd041a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.016+0000 7fcd0a15e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd0410de20 0x7fcd040799b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.016+0000 7fcd0a15e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcd04113880 con 0x7fcd0410de20 2024-04-01T01:42:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.016+0000 7fcd0a15e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcd04113700 con 0x7fcd041089c0 2024-04-01T01:42:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.016+0000 7fcd0a15e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcd04113a00 con 0x7fcd0410b380 2024-04-01T01:42:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.016+0000 7fcd037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd041089c0 0x7fcd041a2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.016+0000 7fcd02ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd0410b380 0x7fcd041a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.016+0000 7fcd037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd041089c0 0x7fcd041a2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:59096/0 (socket says 172.21.15.70:59096) 2024-04-01T01:42:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.016+0000 7fcd03fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd0410de20 0x7fcd040799b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.016+0000 7fcd037fe640 1 -- 172.21.15.70:0/212492450 learned_addr learned my addr 172.21.15.70:0/212492450 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.016+0000 7fcd037fe640 1 -- 172.21.15.70:0/212492450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd0410b380 msgr2=0x7fcd041a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.016+0000 7fcd037fe640 1 --2- 172.21.15.70:0/212492450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd0410b380 0x7fcd041a2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.016+0000 7fcd037fe640 1 -- 172.21.15.70:0/212492450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd0410de20 msgr2=0x7fcd040799b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.016+0000 7fcd037fe640 1 --2- 172.21.15.70:0/212492450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd0410de20 0x7fcd040799b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.016+0000 7fcd037fe640 1 -- 172.21.15.70:0/212492450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd0407a050 con 0x7fcd041089c0 2024-04-01T01:42:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.017+0000 7fcd037fe640 1 --2- 172.21.15.70:0/212492450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd041089c0 0x7fcd041a2000 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fccf800e9f0 tx=0x7fccf800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.017+0000 7fcd00ff9640 1 -- 172.21.15.70:0/212492450 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf800ccc0 con 0x7fcd041089c0 2024-04-01T01:42:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.017+0000 7fcd00ff9640 1 -- 172.21.15.70:0/212492450 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fccf80108e0 con 0x7fcd041089c0 2024-04-01T01:42:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.017+0000 7fcd00ff9640 1 -- 172.21.15.70:0/212492450 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf80187c0 con 0x7fcd041089c0 2024-04-01T01:42:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.017+0000 7fcd0a15e640 1 -- 172.21.15.70:0/212492450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd0407a340 con 0x7fcd041089c0 2024-04-01T01:42:54.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.018+0000 7fcd0a15e640 1 -- 172.21.15.70:0/212492450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd0407a840 con 0x7fcd041089c0 2024-04-01T01:42:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.021+0000 7fcd0ce1e740 1 Processor -- start 2024-04-01T01:42:54.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.021+0000 7fcd0ce1e740 1 -- start start 2024-04-01T01:42:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.021+0000 7fcd00ff9640 1 -- 172.21.15.70:0/212492450 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fccf8010a90 con 0x7fcd041089c0 2024-04-01T01:42:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.022+0000 7fcd0ce1e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03e30e0 0x5557e03e3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.022+0000 7fcd0ce1e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e3ad0 0x5557e03e85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.022+0000 7fcd0ce1e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e8c40 0x5557e03eb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.022+0000 7fcd0ce1e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5557e033e8f0 con 0x5557e03e30e0 2024-04-01T01:42:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.022+0000 7fcd0ce1e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5557e02e78d0 con 0x5557e03e3ad0 2024-04-01T01:42:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.022+0000 7fcd0ce1e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5557e033a990 con 0x5557e03e8c40 2024-04-01T01:42:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.022+0000 7fcd037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e3ad0 0x5557e03e85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.022+0000 7fcd037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e3ad0 0x5557e03e85d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:59098/0 (socket says 172.21.15.70:59098) 2024-04-01T01:42:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.022+0000 7fcd037fe640 1 -- 172.21.15.70:0/1706446902 learned_addr learned my addr 172.21.15.70:0/1706446902 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.022+0000 7fcd02ffd640 1 --2- 172.21.15.70:0/1706446902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e8c40 0x5557e03eb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.023+0000 7fcd02ffd640 1 -- 172.21.15.70:0/1706446902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e3ad0 msgr2=0x5557e03e85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.023+0000 7fcd02ffd640 1 --2- 172.21.15.70:0/1706446902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e3ad0 0x5557e03e85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.023+0000 7fcd02ffd640 1 -- 172.21.15.70:0/1706446902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03e30e0 msgr2=0x5557e03e3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.023+0000 7fcd02ffd640 1 --2- 172.21.15.70:0/1706446902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03e30e0 0x5557e03e3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.023+0000 7fcd02ffd640 1 -- 172.21.15.70:0/1706446902 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557e0053210 con 0x5557e03e8c40 2024-04-01T01:42:54.024 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.023+0000 7fcd02ffd640 1 --2- 172.21.15.70:0/1706446902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e8c40 0x5557e03eb0d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcd040738b0 tx=0x7fccf400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c1b56a83d40bbb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.023+0000 7fcce9ffb640 1 -- 172.21.15.70:0/1706446902 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4005220 con 0x5557e03e8c40 2024-04-01T01:42:54.025 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.023+0000 7fcce9ffb640 1 -- 172.21.15.70:0/1706446902 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fccf4004c50 con 0x5557e03e8c40 2024-04-01T01:42:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.024+0000 7fcce9ffb640 1 -- 172.21.15.70:0/1706446902 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4018730 con 0x5557e03e8c40 2024-04-01T01:42:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.024+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/1706446902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e8c40 msgr2=0x5557e03eb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.024+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/1706446902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e8c40 0x5557e03eb0d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcd040738b0 tx=0x7fccf400f790 comp rx=0 tx=0).stop 2024-04-01T01:42:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.024+0000 7fcd00ff9640 1 --2- 172.21.15.70:0/212492450 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fccec0a2610 0x7fccec0a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.025+0000 7fcd03fff640 1 --2- 172.21.15.70:0/212492450 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fccec0a2610 0x7fccec0a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:54.026 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.025+0000 7fcd00ff9640 1 -- 172.21.15.70:0/212492450 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fccf80cef20 con 0x7fcd041089c0 2024-04-01T01:42:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.026+0000 7fcd03fff640 1 --2- 172.21.15.70:0/212492450 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fccec0a2610 0x7fccec0a4ad0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fccfc005e30 tx=0x7fccfc009800 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.026+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/1706446902 shutdown_connections 2024-04-01T01:42:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.026+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/1706446902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e8c40 0x5557e03eb0d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.026+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/1706446902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e3ad0 0x5557e03e85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.026+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/1706446902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03e30e0 0x5557e03e3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.026+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/1706446902 >> 172.21.15.70:0/1706446902 conn(0x5557e03d5e40 msgr2=0x5557e03d6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:54.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.027+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/1706446902 shutdown_connections 2024-04-01T01:42:54.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.027+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/1706446902 wait complete. 2024-04-01T01:42:54.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.028+0000 7fcd0ce1e740 1 -- start start 2024-04-01T01:42:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.028+0000 7fcd0ce1e740 1 Processor -- start 2024-04-01T01:42:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.028+0000 7fcd0ce1e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03e30e0 0x5557e03ed420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.028+0000 7fcd0ce1e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e3ad0 0x5557e03ed960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.028+0000 7fcd0ce1e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e8c40 0x5557e03f49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.028+0000 7fcd0ce1e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5557e03ebc90 con 0x5557e03e30e0 2024-04-01T01:42:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.028+0000 7fcd0ce1e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5557e02dcf80 con 0x5557e03e8c40 2024-04-01T01:42:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.028+0000 7fcd0ce1e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5557e03ebe10 con 0x5557e03e3ad0 2024-04-01T01:42:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.029+0000 7fcd0ce1e740 10 client.0 _ll_register_callbacks cb 0x5557e0335290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.029+0000 7fcd02ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e3ad0 0x5557e03ed960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.029+0000 7fcd037fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03e30e0 0x5557e03ed420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.029+0000 7fcd037fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03e30e0 0x5557e03ed420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60480/0 (socket says 172.21.15.70:60480) 2024-04-01T01:42:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.029+0000 7fcd037fe640 1 -- 172.21.15.70:0/4091362313 learned_addr learned my addr 172.21.15.70:0/4091362313 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:54.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.030+0000 7fcd02ffd640 1 -- 172.21.15.70:0/4091362313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e8c40 msgr2=0x5557e03f49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:54.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.030+0000 7fcd02ffd640 1 --2- 172.21.15.70:0/4091362313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e8c40 0x5557e03f49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.030+0000 7fcd02ffd640 1 -- 172.21.15.70:0/4091362313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03e30e0 msgr2=0x5557e03ed420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:54.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.030+0000 7fcd02ffd640 1 --2- 172.21.15.70:0/4091362313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03e30e0 0x5557e03ed420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.030+0000 7fcd02ffd640 1 -- 172.21.15.70:0/4091362313 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557e011f620 con 0x5557e03e3ad0 2024-04-01T01:42:54.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.030+0000 7fcd02ffd640 1 --2- 172.21.15.70:0/4091362313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e3ad0 0x5557e03ed960 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fccf401a040 tx=0x7fccf4005030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d3ab38a1cdac62 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:54.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.030+0000 7fcce2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:42:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.030+0000 7fcce2ffd640 1 -- 172.21.15.70:0/4091362313 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4027070 con 0x5557e03e3ad0 2024-04-01T01:42:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.030+0000 7fcce2ffd640 1 -- 172.21.15.70:0/4091362313 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fccf40184b0 con 0x5557e03e3ad0 2024-04-01T01:42:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.030+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/4091362313 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccf4004c50 con 0x5557e03e3ad0 2024-04-01T01:42:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.031+0000 7fcce2ffd640 1 -- 172.21.15.70:0/4091362313 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fccf4031050 con 0x5557e03e3ad0 2024-04-01T01:42:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.031+0000 7fcd0ce1e740 10 client.28093 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.031+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/4091362313 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5557e03f7980 con 0x5557e03e3ad0 2024-04-01T01:42:54.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.032+0000 7fcce2ffd640 1 -- 172.21.15.70:0/4091362313 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fccf4004c50 con 0x5557e03e3ad0 2024-04-01T01:42:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.032+0000 7fcce2ffd640 1 -- 172.21.15.70:0/4091362313 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fccf4023430 con 0x5557e03e3ad0 2024-04-01T01:42:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.033+0000 7fcd0ce1e740 10 client.28093 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.033+0000 7fcd0ce1e740 10 client.28093 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.033+0000 7fcd0ce1e740 20 client.28093 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.033+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/4091362313 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5557e03f8f60 0x5557e03fb350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.033+0000 7fcd0ce1e740 4 client.28093 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:42:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.033+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/4091362313 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557e0352120 con 0x5557e03f8f60 2024-04-01T01:42:54.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.033+0000 7fcd037fe640 1 --2- 172.21.15.70:0/4091362313 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5557e03f8f60 0x5557e03fb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:54.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.034+0000 7fcd037fe640 1 --2- 172.21.15.70:0/4091362313 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5557e03f8f60 0x5557e03fb350 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9bed65798dc2ba2 server_cookie=6366b0f2c83cebcd in_seq=0 out_seq=0 2024-04-01T01:42:54.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.034+0000 7fcce2ffd640 10 client.28093 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:54.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.036+0000 7fcce2ffd640 1 -- 172.21.15.70:0/4091362313 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5557e0352120 con 0x5557e03f8f60 2024-04-01T01:42:54.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.036+0000 7fcce2ffd640 10 client.28093 handle_command_reply: tid=0 2024-04-01T01:42:54.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.036+0000 7fcd0ce1e740 1 client.28093 shutdown 2024-04-01T01:42:54.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.037+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/4091362313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e3ad0 msgr2=0x5557e03ed960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:54.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.037+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/4091362313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e3ad0 0x5557e03ed960 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fccf401a040 tx=0x7fccf4005030 comp rx=0 tx=0).stop 2024-04-01T01:42:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.039+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/4091362313 shutdown_connections 2024-04-01T01:42:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.039+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/4091362313 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5557e03f8f60 0x5557e03fb350 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.039+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/4091362313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e8c40 0x5557e03f49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.039+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/4091362313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e3ad0 0x5557e03ed960 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.039+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/4091362313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03e30e0 0x5557e03ed420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.039+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/4091362313 >> 172.21.15.70:0/4091362313 conn(0x5557e03d5e40 msgr2=0x5557e03d4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.039+0000 7fcce2ffd640 0 client.28093 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.039+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/4091362313 shutdown_connections 2024-04-01T01:42:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.039+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/4091362313 wait complete. 2024-04-01T01:42:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.040+0000 7fcd0ce1e740 20 client.28093 trim_cache size 0 max 16384 2024-04-01T01:42:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.068+0000 7fcd0ce1e740 1 Processor -- start 2024-04-01T01:42:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.068+0000 7fcd0ce1e740 1 -- start start 2024-04-01T01:42:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.068+0000 7fcd0ce1e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03d5e40 0x5557e0400c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.068+0000 7fcd0ce1e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e30e0 0x5557e03f4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.068+0000 7fcd0ce1e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03e3ad0 0x5557e03e4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.068+0000 7fcd0ce1e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcd04080b80 con 0x5557e03e3ad0 2024-04-01T01:42:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.068+0000 7fcd0ce1e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcd040826c0 con 0x5557e03d5e40 2024-04-01T01:42:54.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.068+0000 7fcd0ce1e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcd04081250 con 0x5557e03e30e0 2024-04-01T01:42:54.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.068+0000 7fcd02ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e30e0 0x5557e03f4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:54.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.068+0000 7fcd02ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e30e0 0x5557e03f4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39826/0 (socket says 172.21.15.70:39826) 2024-04-01T01:42:54.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.068+0000 7fcd02ffd640 1 -- 172.21.15.70:0/1377173161 learned_addr learned my addr 172.21.15.70:0/1377173161 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:54.070 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.069+0000 7fcd037fe640 1 --2- 172.21.15.70:0/1377173161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03d5e40 0x5557e0400c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.069+0000 7fcd02ffd640 1 -- 172.21.15.70:0/1377173161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03d5e40 msgr2=0x5557e0400c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.069+0000 7fcd02ffd640 1 --2- 172.21.15.70:0/1377173161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03d5e40 0x5557e0400c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.069+0000 7fcd02ffd640 1 -- 172.21.15.70:0/1377173161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03e3ad0 msgr2=0x5557e03e4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.069+0000 7fcd02ffd640 1 --2- 172.21.15.70:0/1377173161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03e3ad0 0x5557e03e4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.069+0000 7fcd02ffd640 1 -- 172.21.15.70:0/1377173161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557e03d6bb0 con 0x5557e03e30e0 2024-04-01T01:42:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.070+0000 7fcd02ffd640 1 --2- 172.21.15.70:0/1377173161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e30e0 0x5557e03f4960 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fccf4023a40 tx=0x7fccf40050a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f520c2b61f1f382f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.070+0000 7fcce97fa640 1 -- 172.21.15.70:0/1377173161 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4027040 con 0x5557e03e30e0 2024-04-01T01:42:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.070+0000 7fcce97fa640 1 -- 172.21.15.70:0/1377173161 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fccf403a070 con 0x5557e03e30e0 2024-04-01T01:42:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.071+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/1377173161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e30e0 msgr2=0x5557e03f4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.071+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/1377173161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e30e0 0x5557e03f4960 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fccf4023a40 tx=0x7fccf40050a0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.072+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/1377173161 shutdown_connections 2024-04-01T01:42:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.072+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/1377173161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03e3ad0 0x5557e03e4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.072+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/1377173161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e30e0 0x5557e03f4960 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.072+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/1377173161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03d5e40 0x5557e0400c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.072+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/1377173161 >> 172.21.15.70:0/1377173161 conn(0x5557e03e8710 msgr2=0x5557e03d6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.072+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/1377173161 shutdown_connections 2024-04-01T01:42:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.072+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/1377173161 wait complete. 2024-04-01T01:42:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.072+0000 7fcd0ce1e740 1 -- start start 2024-04-01T01:42:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.073+0000 7fcd0ce1e740 1 Processor -- start 2024-04-01T01:42:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.073+0000 7fcd0ce1e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03d5e40 0x5557e03f1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.073+0000 7fcd0ce1e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e30e0 0x5557e03f2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.073+0000 7fcd0ce1e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e3ad0 0x5557e03f3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.073+0000 7fcd0ce1e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcd04081ff0 con 0x5557e03d5e40 2024-04-01T01:42:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.073+0000 7fcd0ce1e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcd04081920 con 0x5557e03e3ad0 2024-04-01T01:42:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.073+0000 7fcd0ce1e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcd0407eb80 con 0x5557e03e30e0 2024-04-01T01:42:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.073+0000 7fcd0ce1e740 10 client.0 _ll_register_callbacks cb 0x7fccf80d27b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:42:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.073+0000 7fcd03fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e3ad0 0x5557e03f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.073+0000 7fcd037fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03d5e40 0x5557e03f1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.073+0000 7fcd03fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e3ad0 0x5557e03f3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:59130/0 (socket says 172.21.15.70:59130) 2024-04-01T01:42:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.073+0000 7fcd03fff640 1 -- 172.21.15.70:0/770130744 learned_addr learned my addr 172.21.15.70:0/770130744 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:42:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.073+0000 7fcd02ffd640 1 --2- 172.21.15.70:0/770130744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e30e0 0x5557e03f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.074+0000 7fcd03fff640 1 -- 172.21.15.70:0/770130744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e30e0 msgr2=0x5557e03f2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.074+0000 7fcd03fff640 1 --2- 172.21.15.70:0/770130744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e30e0 0x5557e03f2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.074+0000 7fcd03fff640 1 -- 172.21.15.70:0/770130744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03d5e40 msgr2=0x5557e03f1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:42:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.074+0000 7fcd03fff640 1 --2- 172.21.15.70:0/770130744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03d5e40 0x5557e03f1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.074+0000 7fcd03fff640 1 -- 172.21.15.70:0/770130744 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557e03ec4e0 con 0x5557e03e3ad0 2024-04-01T01:42:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.074+0000 7fcd03fff640 1 --2- 172.21.15.70:0/770130744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e3ad0 0x5557e03f3220 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fccfc036000 tx=0x7fccfc032340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2444408768c06485 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:42:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.074+0000 7fcce2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:42:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.074+0000 7fcce2ffd640 1 -- 172.21.15.70:0/770130744 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccfc039040 con 0x5557e03e3ad0 2024-04-01T01:42:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.074+0000 7fcce2ffd640 1 -- 172.21.15.70:0/770130744 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fccfc004450 con 0x5557e03e3ad0 2024-04-01T01:42:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.074+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/770130744 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccf403a070 con 0x5557e03e3ad0 2024-04-01T01:42:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.075+0000 7fcce2ffd640 1 -- 172.21.15.70:0/770130744 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fccfc058050 con 0x5557e03e3ad0 2024-04-01T01:42:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.075+0000 7fcd0ce1e740 10 client.28286 fetch_fsmap learned FSMap version 31 2024-04-01T01:42:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.075+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/770130744 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5557e03f49c0 con 0x5557e03e3ad0 2024-04-01T01:42:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.075+0000 7fcce2ffd640 1 -- 172.21.15.70:0/770130744 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fccf403a070 con 0x5557e03e3ad0 2024-04-01T01:42:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.075+0000 7fcce2ffd640 1 -- 172.21.15.70:0/770130744 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fccfc04ac40 con 0x5557e03e3ad0 2024-04-01T01:42:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.075+0000 7fcd0ce1e740 10 client.28286 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:42:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.075+0000 7fcd0ce1e740 10 client.28286 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:42:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.075+0000 7fcd0ce1e740 20 client.28286 populate_metadata read hostname 'smithi070' 2024-04-01T01:42:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.076+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/770130744 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5557e03e8c40 0x5557e03dcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:42:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.076+0000 7fcd0ce1e740 4 client.28286 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:42:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.076+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/770130744 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5557e034be30 con 0x5557e03e8c40 2024-04-01T01:42:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.076+0000 7fcd037fe640 1 --2- 172.21.15.70:0/770130744 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5557e03e8c40 0x5557e03dcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:42:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.079+0000 7fcd037fe640 1 --2- 172.21.15.70:0/770130744 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5557e03e8c40 0x5557e03dcbc0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=337aa546a40791fb server_cookie=ab2f436bab2c3c5b in_seq=0 out_seq=0 2024-04-01T01:42:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.079+0000 7fcce2ffd640 10 client.28286 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.113+0000 7fcce2ffd640 1 -- 172.21.15.70:0/770130744 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5557e034be30 con 0x5557e03e8c40 2024-04-01T01:42:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.113+0000 7fcce2ffd640 10 client.28286 handle_command_reply: tid=0 2024-04-01T01:42:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.114+0000 7fcd0ce1e740 1 client.28286 shutdown 2024-04-01T01:42:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.116+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/770130744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e3ad0 msgr2=0x5557e03f3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.116+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/770130744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e3ad0 0x5557e03f3220 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fccfc036000 tx=0x7fccfc032340 comp rx=0 tx=0).stop 2024-04-01T01:42:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.116+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/770130744 shutdown_connections 2024-04-01T01:42:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.116+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/770130744 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5557e03e8c40 0x5557e03dcbc0 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.116+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/770130744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5557e03e3ad0 0x5557e03f3220 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.116+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/770130744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557e03e30e0 0x5557e03f2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.116+0000 7fcd0ce1e740 1 --2- 172.21.15.70:0/770130744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5557e03d5e40 0x5557e03f1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.116+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/770130744 >> 172.21.15.70:0/770130744 conn(0x5557e03e8710 msgr2=0x5557e03e4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.116+0000 7fcce2ffd640 0 client.28286 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:42:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.116+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/770130744 shutdown_connections 2024-04-01T01:42:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.116+0000 7fcd0ce1e740 1 -- 172.21.15.70:0/770130744 wait complete. 2024-04-01T01:42:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.117+0000 7fcd0ce1e740 20 client.28286 trim_cache size 0 max 16384 2024-04-01T01:42:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:42:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:42:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:42:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:42:54.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.119+0000 7fcd0a15e640 1 -- 172.21.15.70:0/212492450 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fccec0a2610 msgr2=0x7fccec0a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:54.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.119+0000 7fcd0a15e640 1 --2- 172.21.15.70:0/212492450 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fccec0a2610 0x7fccec0a4ad0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fccfc005e30 tx=0x7fccfc009800 comp rx=0 tx=0).stop 2024-04-01T01:42:54.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.119+0000 7fcd0a15e640 1 -- 172.21.15.70:0/212492450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd041089c0 msgr2=0x7fcd041a2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:42:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.119+0000 7fcd0a15e640 1 --2- 172.21.15.70:0/212492450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd041089c0 0x7fcd041a2000 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fccf800e9f0 tx=0x7fccf800edf0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.119+0000 7fcd0a15e640 1 -- 172.21.15.70:0/212492450 shutdown_connections 2024-04-01T01:42:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.119+0000 7fcd0a15e640 1 --2- 172.21.15.70:0/212492450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd0410de20 0x7fcd040799b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.119+0000 7fcd0a15e640 1 --2- 172.21.15.70:0/212492450 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fccec0a2610 0x7fccec0a4ad0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.119+0000 7fcd0a15e640 1 --2- 172.21.15.70:0/212492450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd0410b380 0x7fcd041a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.119+0000 7fcd0a15e640 1 --2- 172.21.15.70:0/212492450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd041089c0 0x7fcd041a2000 secure :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fccf800e9f0 tx=0x7fccf800edf0 comp rx=0 tx=0).stop 2024-04-01T01:42:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.119+0000 7fcd0a15e640 1 -- 172.21.15.70:0/212492450 >> 172.21.15.70:0/212492450 conn(0x7fcd0406e950 msgr2=0x7fcd0410e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:42:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.119+0000 7fcd0a15e640 1 -- 172.21.15.70:0/212492450 shutdown_connections 2024-04-01T01:42:54.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:42:54.120+0000 7fcd0a15e640 1 -- 172.21.15.70:0/212492450 wait complete. 2024-04-01T01:42:55.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:54 smithi189 ceph-mon[35753]: pgmap v642: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 6.3 MiB/s wr, 806 op/s 2024-04-01T01:42:55.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:54 smithi070 ceph-mon[32179]: pgmap v642: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 6.3 MiB/s wr, 806 op/s 2024-04-01T01:42:55.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:54 smithi081 ceph-mon[35142]: pgmap v642: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 6.3 MiB/s wr, 806 op/s 2024-04-01T01:42:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 490.0 s, 882.2 tps, lat 113.494 ms stddev 99.393 2024-04-01T01:42:57.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:56 smithi189 ceph-mon[35753]: pgmap v643: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 7.0 MiB/s wr, 889 op/s 2024-04-01T01:42:57.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:56 smithi070 ceph-mon[32179]: pgmap v643: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 7.0 MiB/s wr, 889 op/s 2024-04-01T01:42:57.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:56 smithi081 ceph-mon[35142]: pgmap v643: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 7.0 MiB/s wr, 889 op/s 2024-04-01T01:42:59.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:42:58 smithi189 ceph-mon[35753]: pgmap v644: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.5 MiB/s wr, 910 op/s 2024-04-01T01:42:59.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:42:58 smithi070 ceph-mon[32179]: pgmap v644: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.5 MiB/s wr, 910 op/s 2024-04-01T01:42:59.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:42:58 smithi081 ceph-mon[35142]: pgmap v644: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.5 MiB/s wr, 910 op/s 2024-04-01T01:43:01.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:00 smithi189 ceph-mon[35753]: pgmap v645: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 6.7 MiB/s wr, 786 op/s 2024-04-01T01:43:01.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:00 smithi070 ceph-mon[32179]: pgmap v645: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 6.7 MiB/s wr, 786 op/s 2024-04-01T01:43:01.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:00 smithi081 ceph-mon[35142]: pgmap v645: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 6.7 MiB/s wr, 786 op/s 2024-04-01T01:43:01.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 495.0 s, 878.0 tps, lat 113.969 ms stddev 106.469 2024-04-01T01:43:03.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:02 smithi189 ceph-mon[35753]: pgmap v646: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.9 MiB/s wr, 887 op/s 2024-04-01T01:43:03.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:02 smithi070 ceph-mon[32179]: pgmap v646: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.9 MiB/s wr, 887 op/s 2024-04-01T01:43:03.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:02 smithi081 ceph-mon[35142]: pgmap v646: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.9 MiB/s wr, 887 op/s 2024-04-01T01:43:05.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:04 smithi189 ceph-mon[35753]: pgmap v647: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.7 MiB/s wr, 777 op/s 2024-04-01T01:43:05.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:04 smithi070 ceph-mon[32179]: pgmap v647: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.7 MiB/s wr, 777 op/s 2024-04-01T01:43:05.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:04 smithi081 ceph-mon[35142]: pgmap v647: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.7 MiB/s wr, 777 op/s 2024-04-01T01:43:06.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 500.0 s, 779.1 tps, lat 127.527 ms stddev 138.393 2024-04-01T01:43:07.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:06 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:43:07.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:06 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:43:07.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:06 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:43:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.277+0000 7fa8bff9a640 1 -- 172.21.15.70:0/3071589898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8b8100760 msgr2=0x7fa8b8100bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.277+0000 7fa8bff9a640 1 --2- 172.21.15.70:0/3071589898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8b8100760 0x7fa8b8100bc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8009860 tx=0x7fa8a802bb70 comp rx=0 tx=0).stop 2024-04-01T01:43:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.278+0000 7fa8bff9a640 1 -- 172.21.15.70:0/3071589898 shutdown_connections 2024-04-01T01:43:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.278+0000 7fa8bff9a640 1 --2- 172.21.15.70:0/3071589898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8b8101100 0x7fa8b81019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.278+0000 7fa8bff9a640 1 --2- 172.21.15.70:0/3071589898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8b8100760 0x7fa8b8100bc0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.278+0000 7fa8bff9a640 1 --2- 172.21.15.70:0/3071589898 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8b80ff630 0x7fa8b80ffa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.278+0000 7fa8bff9a640 1 -- 172.21.15.70:0/3071589898 >> 172.21.15.70:0/3071589898 conn(0x7fa8b80faf30 msgr2=0x7fa8b80fd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.278+0000 7fa8bff9a640 1 -- 172.21.15.70:0/3071589898 shutdown_connections 2024-04-01T01:43:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.278+0000 7fa8bff9a640 1 -- 172.21.15.70:0/3071589898 wait complete. 2024-04-01T01:43:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.278+0000 7fa8bff9a640 1 Processor -- start 2024-04-01T01:43:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.279+0000 7fa8bff9a640 1 -- start start 2024-04-01T01:43:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.279+0000 7fa8bff9a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8b80ff630 0x7fa8b8199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.279+0000 7fa8bff9a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8b8100760 0x7fa8b819a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.279+0000 7fa8bff9a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8b8101100 0x7fa8b81a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.279+0000 7fa8bff9a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b810f4d0 con 0x7fa8b8100760 2024-04-01T01:43:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.279+0000 7fa8bff9a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b810f350 con 0x7fa8b8101100 2024-04-01T01:43:07.281 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.279+0000 7fa8bff9a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b810f650 con 0x7fa8b80ff630 2024-04-01T01:43:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.279+0000 7fa8be537640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8b8101100 0x7fa8b81a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.279+0000 7fa8be537640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8b8101100 0x7fa8b81a1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41700/0 (socket says 172.21.15.70:41700) 2024-04-01T01:43:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.279+0000 7fa8be537640 1 -- 172.21.15.70:0/419230500 learned_addr learned my addr 172.21.15.70:0/419230500 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.280+0000 7fa8be537640 1 -- 172.21.15.70:0/419230500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8b80ff630 msgr2=0x7fa8b8199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.280+0000 7fa8bdd36640 1 --2- 172.21.15.70:0/419230500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8b80ff630 0x7fa8b8199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.280+0000 7fa8be537640 1 --2- 172.21.15.70:0/419230500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8b80ff630 0x7fa8b8199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.280+0000 7fa8be537640 1 -- 172.21.15.70:0/419230500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8b8100760 msgr2=0x7fa8b819a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.280+0000 7fa8be537640 1 --2- 172.21.15.70:0/419230500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8b8100760 0x7fa8b819a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.280+0000 7fa8be537640 1 -- 172.21.15.70:0/419230500 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b81a1700 con 0x7fa8b8101100 2024-04-01T01:43:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.280+0000 7fa8bdd36640 1 --2- 172.21.15.70:0/419230500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8b80ff630 0x7fa8b8199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:43:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.280+0000 7fa8be537640 1 --2- 172.21.15.70:0/419230500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8b8101100 0x7fa8b81a1090 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fa8b000eb10 tx=0x7fa8b000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:07.282 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.281+0000 7fa8aeffd640 1 -- 172.21.15.70:0/419230500 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0004b20 con 0x7fa8b8101100 2024-04-01T01:43:07.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.281+0000 7fa8aeffd640 1 -- 172.21.15.70:0/419230500 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8b0004cc0 con 0x7fa8b8101100 2024-04-01T01:43:07.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.282+0000 7fa8aeffd640 1 -- 172.21.15.70:0/419230500 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0010690 con 0x7fa8b8101100 2024-04-01T01:43:07.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.282+0000 7fa8bff9a640 1 -- 172.21.15.70:0/419230500 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8b81a19f0 con 0x7fa8b8101100 2024-04-01T01:43:07.283 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.282+0000 7fa8bff9a640 1 -- 172.21.15.70:0/419230500 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8b81a1ec0 con 0x7fa8b8101100 2024-04-01T01:43:07.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.286+0000 7fa8c2b7c740 1 Processor -- start 2024-04-01T01:43:07.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.287+0000 7fa8aeffd640 1 -- 172.21.15.70:0/419230500 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fa8b0005000 con 0x7fa8b8101100 2024-04-01T01:43:07.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.288+0000 7fa8c2b7c740 1 -- start start 2024-04-01T01:43:07.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.288+0000 7fa8c2b7c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e35748d0 0x5584e3574cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.288+0000 7fa8c2b7c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 0x5584e3579dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.288+0000 7fa8c2b7c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e357a430 0x5584e357c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.288+0000 7fa8c2b7c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5584e34d88f0 con 0x5584e35748d0 2024-04-01T01:43:07.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.288+0000 7fa8c2b7c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5584e34818d0 con 0x5584e357a430 2024-04-01T01:43:07.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.289+0000 7fa8c2b7c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5584e34d4990 con 0x5584e35752c0 2024-04-01T01:43:07.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.289+0000 7fa8aeffd640 1 --2- 172.21.15.70:0/419230500 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8a40a2500 0x7fa8a40a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.289+0000 7fa8bd535640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e357a430 0x5584e357c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.289+0000 7fa8bd535640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e35748d0 0x5584e3574cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.289+0000 7fa8bd535640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e35748d0 0x5584e3574cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46194/0 (socket says 172.21.15.70:46194) 2024-04-01T01:43:07.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.289+0000 7fa8bd535640 1 -- 172.21.15.70:0/4205910840 learned_addr learned my addr 172.21.15.70:0/4205910840 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:07.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.289+0000 7fa8be537640 1 --2- 172.21.15.70:0/419230500 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8a40a2500 0x7fa8a40a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.289+0000 7fa8aeffd640 1 -- 172.21.15.70:0/419230500 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa8b00cf1b0 con 0x7fa8b8101100 2024-04-01T01:43:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.289+0000 7fa8bdd36640 1 --2- 172.21.15.70:0/4205910840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 0x5584e3579dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.290+0000 7fa8bd535640 1 -- 172.21.15.70:0/4205910840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 msgr2=0x5584e3579dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.290+0000 7fa8bd535640 1 --2- 172.21.15.70:0/4205910840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 0x5584e3579dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.290+0000 7fa8bd535640 1 -- 172.21.15.70:0/4205910840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e35748d0 msgr2=0x5584e3574cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.290+0000 7fa8bd535640 1 --2- 172.21.15.70:0/4205910840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e35748d0 0x5584e3574cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.290+0000 7fa8bd535640 1 -- 172.21.15.70:0/4205910840 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584e31ed210 con 0x5584e357a430 2024-04-01T01:43:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.290+0000 7fa8bd535640 1 --2- 172.21.15.70:0/4205910840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e357a430 0x5584e357c8c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa8b819aff0 tx=0x7fa8a8005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c9c130f02e08b1c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.290+0000 7fa8be537640 1 --2- 172.21.15.70:0/419230500 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8a40a2500 0x7fa8a40a49c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa8b00b7120 tx=0x7fa8b0097440 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.291+0000 7fa8a3fff640 1 -- 172.21.15.70:0/4205910840 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8039030 con 0x5584e357a430 2024-04-01T01:43:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.291+0000 7fa8a3fff640 1 -- 172.21.15.70:0/4205910840 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8a8034e30 con 0x5584e357a430 2024-04-01T01:43:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.291+0000 7fa8a3fff640 1 -- 172.21.15.70:0/4205910840 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8004920 con 0x5584e357a430 2024-04-01T01:43:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.292+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/4205910840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e357a430 msgr2=0x5584e357c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.292+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/4205910840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e357a430 0x5584e357c8c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa8b819aff0 tx=0x7fa8a8005f90 comp rx=0 tx=0).stop 2024-04-01T01:43:07.293 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.292+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/4205910840 shutdown_connections 2024-04-01T01:43:07.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.292+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/4205910840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e357a430 0x5584e357c8c0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.292+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/4205910840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 0x5584e3579dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.292+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/4205910840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e35748d0 0x5584e3574cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.292+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/4205910840 >> 172.21.15.70:0/4205910840 conn(0x5584e356fe40 msgr2=0x5584e3570240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:07.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.292+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/4205910840 shutdown_connections 2024-04-01T01:43:07.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.293+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/4205910840 wait complete. 2024-04-01T01:43:07.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.293+0000 7fa8c2b7c740 1 -- start start 2024-04-01T01:43:07.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.294+0000 7fa8c2b7c740 1 Processor -- start 2024-04-01T01:43:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.295+0000 7fa8c2b7c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 0x5584e358fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.295+0000 7fa8c2b7c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 0x5584e3590140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.295+0000 7fa8c2b7c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e357a430 0x5584e3597180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.295+0000 7fa8c2b7c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5584e358e390 con 0x5584e357a430 2024-04-01T01:43:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.296+0000 7fa8c2b7c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5584e3476f80 con 0x5584e35748d0 2024-04-01T01:43:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.296+0000 7fa8c2b7c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5584e358e510 con 0x5584e35752c0 2024-04-01T01:43:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.296+0000 7fa8c2b7c740 10 client.0 _ll_register_callbacks cb 0x5584e34cf290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:07.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.305+0000 7fa8bd535640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 0x5584e358fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.305+0000 7fa8bd535640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 0x5584e358fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41720/0 (socket says 172.21.15.70:41720) 2024-04-01T01:43:07.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.305+0000 7fa8bd535640 1 -- 172.21.15.70:0/150590790 learned_addr learned my addr 172.21.15.70:0/150590790 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:07.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.305+0000 7fa8bd535640 1 --2- 172.21.15.70:0/150590790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e357a430 0x5584e3597180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.306 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.305+0000 7fa8bdd36640 1 --2- 172.21.15.70:0/150590790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 0x5584e3590140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.306+0000 7fa8bd535640 1 -- 172.21.15.70:0/150590790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 msgr2=0x5584e3590140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.306+0000 7fa8bd535640 1 --2- 172.21.15.70:0/150590790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 0x5584e3590140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.306+0000 7fa8bd535640 1 -- 172.21.15.70:0/150590790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e357a430 msgr2=0x5584e3597180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.306+0000 7fa8bd535640 1 --2- 172.21.15.70:0/150590790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e357a430 0x5584e3597180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.306+0000 7fa8bd535640 1 -- 172.21.15.70:0/150590790 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584e32b9620 con 0x5584e35748d0 2024-04-01T01:43:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.306+0000 7fa8bd535640 1 --2- 172.21.15.70:0/150590790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 0x5584e358fc00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fa8a802c260 tx=0x7fa8a8005d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59655343b0e73ff4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.306+0000 7fa8a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.307+0000 7fa8a17fa640 1 -- 172.21.15.70:0/150590790 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8039020 con 0x5584e35748d0 2024-04-01T01:43:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.307+0000 7fa8a17fa640 1 -- 172.21.15.70:0/150590790 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa8a8004a90 con 0x5584e35748d0 2024-04-01T01:43:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.307+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/150590790 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8a8034e30 con 0x5584e35748d0 2024-04-01T01:43:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.307+0000 7fa8a17fa640 1 -- 172.21.15.70:0/150590790 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa8a8058050 con 0x5584e35748d0 2024-04-01T01:43:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.307+0000 7fa8c2b7c740 10 client.28298 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:07.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.307+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/150590790 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5584e359a160 con 0x5584e35748d0 2024-04-01T01:43:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.307+0000 7fa8a17fa640 1 -- 172.21.15.70:0/150590790 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8a8034e30 con 0x5584e35748d0 2024-04-01T01:43:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.307+0000 7fa8a17fa640 1 -- 172.21.15.70:0/150590790 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fa8a804f070 con 0x5584e35748d0 2024-04-01T01:43:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.308+0000 7fa8c2b7c740 10 client.28298 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.308+0000 7fa8c2b7c740 10 client.28298 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.308+0000 7fa8c2b7c740 20 client.28298 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.308+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/150590790 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5584e359b4b0 0x5584e359d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.308+0000 7fa8c2b7c740 4 client.28298 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.308+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/150590790 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584e34ec120 con 0x5584e359b4b0 2024-04-01T01:43:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.308+0000 7fa8bdd36640 1 --2- 172.21.15.70:0/150590790 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5584e359b4b0 0x5584e359d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.308+0000 7fa8a17fa640 10 client.28298 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.308+0000 7fa8bdd36640 1 --2- 172.21.15.70:0/150590790 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5584e359b4b0 0x5584e359d8a0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab5b90e0e39dc100 server_cookie=226caad30ed4009a in_seq=0 out_seq=0 2024-04-01T01:43:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.309+0000 7fa8a17fa640 1 -- 172.21.15.70:0/150590790 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5584e34ec120 con 0x5584e359b4b0 2024-04-01T01:43:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.309+0000 7fa8a17fa640 10 client.28298 handle_command_reply: tid=0 2024-04-01T01:43:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.309+0000 7fa8c2b7c740 1 client.28298 shutdown 2024-04-01T01:43:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.309+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/150590790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 msgr2=0x5584e358fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.309+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/150590790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 0x5584e358fc00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fa8a802c260 tx=0x7fa8a8005d80 comp rx=0 tx=0).stop 2024-04-01T01:43:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.309+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/150590790 shutdown_connections 2024-04-01T01:43:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.309+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/150590790 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5584e359b4b0 0x5584e359d8a0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.309+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/150590790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e357a430 0x5584e3597180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.309+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/150590790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 0x5584e3590140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.309+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/150590790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 0x5584e358fc00 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.309+0000 7fa8a17fa640 0 client.28298 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:07.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.309+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/150590790 >> 172.21.15.70:0/150590790 conn(0x5584e356fe40 msgr2=0x5584e356ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:07.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.310+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/150590790 shutdown_connections 2024-04-01T01:43:07.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.310+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/150590790 wait complete. 2024-04-01T01:43:07.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.310+0000 7fa8c2b7c740 20 client.28298 trim_cache size 0 max 16384 2024-04-01T01:43:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.337+0000 7fa8c2b7c740 1 Processor -- start 2024-04-01T01:43:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.337+0000 7fa8c2b7c740 1 -- start start 2024-04-01T01:43:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.338+0000 7fa8c2b7c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 0x5584e35a3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.338+0000 7fa8c2b7c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e35752c0 0x5584e3597110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.338+0000 7fa8c2b7c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e357a430 0x5584e35730a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.338+0000 7fa8c2b7c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b8070ee0 con 0x5584e35752c0 2024-04-01T01:43:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.338+0000 7fa8c2b7c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b8072a20 con 0x5584e35748d0 2024-04-01T01:43:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.338+0000 7fa8c2b7c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b80715b0 con 0x5584e357a430 2024-04-01T01:43:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.338+0000 7fa8bd535640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 0x5584e35a3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.338+0000 7fa8bd535640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 0x5584e35a3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41722/0 (socket says 172.21.15.70:41722) 2024-04-01T01:43:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.338+0000 7fa8bd535640 1 -- 172.21.15.70:0/1903409131 learned_addr learned my addr 172.21.15.70:0/1903409131 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.338+0000 7fa8bdd36640 1 --2- 172.21.15.70:0/1903409131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e35752c0 0x5584e3597110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.338+0000 7fa8bd535640 1 --2- 172.21.15.70:0/1903409131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e357a430 0x5584e35730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.338+0000 7fa8bd535640 1 -- 172.21.15.70:0/1903409131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e357a430 msgr2=0x5584e35730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.338+0000 7fa8bd535640 1 --2- 172.21.15.70:0/1903409131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e357a430 0x5584e35730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.338+0000 7fa8bd535640 1 -- 172.21.15.70:0/1903409131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e35752c0 msgr2=0x5584e3597110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.338+0000 7fa8bd535640 1 --2- 172.21.15.70:0/1903409131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e35752c0 0x5584e3597110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.338+0000 7fa8bd535640 1 -- 172.21.15.70:0/1903409131 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584e356a170 con 0x5584e35748d0 2024-04-01T01:43:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.339+0000 7fa8bd535640 1 --2- 172.21.15.70:0/1903409131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 0x5584e35a3010 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa8a802f020 tx=0x7fa8a8005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfa90a32e71860e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.339+0000 7fa8a37fe640 1 -- 172.21.15.70:0/1903409131 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8039040 con 0x5584e35748d0 2024-04-01T01:43:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.339+0000 7fa8a37fe640 1 -- 172.21.15.70:0/1903409131 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8a803a040 con 0x5584e35748d0 2024-04-01T01:43:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.339+0000 7fa8a37fe640 1 -- 172.21.15.70:0/1903409131 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8049040 con 0x5584e35748d0 2024-04-01T01:43:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.340+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/1903409131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 msgr2=0x5584e35a3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.340+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/1903409131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 0x5584e35a3010 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa8a802f020 tx=0x7fa8a8005b20 comp rx=0 tx=0).stop 2024-04-01T01:43:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.340+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/1903409131 shutdown_connections 2024-04-01T01:43:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.340+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/1903409131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e357a430 0x5584e35730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.340+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/1903409131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e35752c0 0x5584e3597110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.340+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/1903409131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 0x5584e35a3010 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.340+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/1903409131 >> 172.21.15.70:0/1903409131 conn(0x5584e356fe40 msgr2=0x5584e3570220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.340+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/1903409131 shutdown_connections 2024-04-01T01:43:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.340+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/1903409131 wait complete. 2024-04-01T01:43:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.341+0000 7fa8c2b7c740 1 -- start start 2024-04-01T01:43:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.341+0000 7fa8c2b7c740 1 Processor -- start 2024-04-01T01:43:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.341+0000 7fa8c2b7c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 0x5584e3594430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.342+0000 7fa8c2b7c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 0x5584e3594970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.342+0000 7fa8c2b7c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e357a430 0x5584e3595930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.342+0000 7fa8c2b7c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b8072350 con 0x5584e357a430 2024-04-01T01:43:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.342+0000 7fa8c2b7c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b8071c80 con 0x5584e35748d0 2024-04-01T01:43:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.342+0000 7fa8c2b7c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b806eec0 con 0x5584e35752c0 2024-04-01T01:43:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.342+0000 7fa8c2b7c740 10 client.0 _ll_register_callbacks cb 0x7fa8b00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.342+0000 7fa8bd535640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 0x5584e3594430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.342+0000 7fa8bdd36640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 0x5584e3594970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.342+0000 7fa8bdd36640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 0x5584e3594970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:54780/0 (socket says 172.21.15.70:54780) 2024-04-01T01:43:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.342+0000 7fa8bdd36640 1 -- 172.21.15.70:0/3731749409 learned_addr learned my addr 172.21.15.70:0/3731749409 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.342+0000 7fa8bd535640 1 --2- 172.21.15.70:0/3731749409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e357a430 0x5584e3595930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.342+0000 7fa8bdd36640 1 -- 172.21.15.70:0/3731749409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 msgr2=0x5584e3594430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.342+0000 7fa8bdd36640 1 --2- 172.21.15.70:0/3731749409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 0x5584e3594430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.342+0000 7fa8bdd36640 1 -- 172.21.15.70:0/3731749409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e357a430 msgr2=0x5584e3595930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.342+0000 7fa8bdd36640 1 --2- 172.21.15.70:0/3731749409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e357a430 0x5584e3595930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.342+0000 7fa8bdd36640 1 -- 172.21.15.70:0/3731749409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584e3595f20 con 0x5584e35752c0 2024-04-01T01:43:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.343+0000 7fa8bdd36640 1 --2- 172.21.15.70:0/3731749409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 0x5584e3594970 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x5584e35756c0 tx=0x7fa8b4009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6cb4d703f676aa7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.343+0000 7fa8a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:43:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.343+0000 7fa8a17fa640 1 -- 172.21.15.70:0/3731749409 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b40270e0 con 0x5584e35752c0 2024-04-01T01:43:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.343+0000 7fa8a17fa640 1 -- 172.21.15.70:0/3731749409 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa8b4027280 con 0x5584e35752c0 2024-04-01T01:43:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.343+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/3731749409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8a803a040 con 0x5584e35752c0 2024-04-01T01:43:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.344+0000 7fa8a17fa640 1 -- 172.21.15.70:0/3731749409 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa8b4027480 con 0x5584e35752c0 2024-04-01T01:43:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.344+0000 7fa8c2b7c740 10 client.28114 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:07.346 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.344+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/3731749409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5584e3597330 con 0x5584e35752c0 2024-04-01T01:43:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.344+0000 7fa8a17fa640 1 -- 172.21.15.70:0/3731749409 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa8b4027620 con 0x5584e35752c0 2024-04-01T01:43:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.344+0000 7fa8a17fa640 1 -- 172.21.15.70:0/3731749409 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fa8b400f040 con 0x5584e35752c0 2024-04-01T01:43:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.344+0000 7fa8c2b7c740 10 client.28114 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.344+0000 7fa8c2b7c740 10 client.28114 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.344+0000 7fa8c2b7c740 20 client.28114 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.344+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/3731749409 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5584e3601c40 0x5584e3604060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.344+0000 7fa8c2b7c740 4 client.28114 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:43:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.344+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/3731749409 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fa8a803a040 con 0x5584e3601c40 2024-04-01T01:43:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.345+0000 7fa8bd535640 1 --2- 172.21.15.70:0/3731749409 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5584e3601c40 0x5584e3604060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.345+0000 7fa8a17fa640 10 client.28114 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.345+0000 7fa8bd535640 1 --2- 172.21.15.70:0/3731749409 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5584e3601c40 0x5584e3604060 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4360ae95734a436d server_cookie=687223a8993f3b96 in_seq=0 out_seq=0 2024-04-01T01:43:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.376+0000 7fa8a17fa640 1 -- 172.21.15.70:0/3731749409 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fa8a803a040 con 0x5584e3601c40 2024-04-01T01:43:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.376+0000 7fa8a17fa640 10 client.28114 handle_command_reply: tid=0 2024-04-01T01:43:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.377+0000 7fa8c2b7c740 1 client.28114 shutdown 2024-04-01T01:43:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.377+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/3731749409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 msgr2=0x5584e3594970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.377+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/3731749409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 0x5584e3594970 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x5584e35756c0 tx=0x7fa8b4009b80 comp rx=0 tx=0).stop 2024-04-01T01:43:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.377+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/3731749409 shutdown_connections 2024-04-01T01:43:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.377+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/3731749409 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5584e3601c40 0x5584e3604060 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.377+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/3731749409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584e357a430 0x5584e3595930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.377+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/3731749409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584e35752c0 0x5584e3594970 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.377+0000 7fa8c2b7c740 1 --2- 172.21.15.70:0/3731749409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5584e35748d0 0x5584e3594430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.377+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/3731749409 >> 172.21.15.70:0/3731749409 conn(0x5584e356fe40 msgr2=0x5584e35730a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.377+0000 7fa8a17fa640 0 client.28114 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.378+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/3731749409 shutdown_connections 2024-04-01T01:43:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.378+0000 7fa8c2b7c740 1 -- 172.21.15.70:0/3731749409 wait complete. 2024-04-01T01:43:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.378+0000 7fa8c2b7c740 20 client.28114 trim_cache size 0 max 16384 2024-04-01T01:43:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:43:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:43:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:43:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:43:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.380+0000 7fa8bff9a640 1 -- 172.21.15.70:0/419230500 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8a40a2500 msgr2=0x7fa8a40a49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.380+0000 7fa8bff9a640 1 --2- 172.21.15.70:0/419230500 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8a40a2500 0x7fa8a40a49c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa8b00b7120 tx=0x7fa8b0097440 comp rx=0 tx=0).stop 2024-04-01T01:43:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.380+0000 7fa8bff9a640 1 -- 172.21.15.70:0/419230500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8b8101100 msgr2=0x7fa8b81a1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:07.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.380+0000 7fa8bff9a640 1 --2- 172.21.15.70:0/419230500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8b8101100 0x7fa8b81a1090 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fa8b000eb10 tx=0x7fa8b000c2d0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.381+0000 7fa8bff9a640 1 -- 172.21.15.70:0/419230500 shutdown_connections 2024-04-01T01:43:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.381+0000 7fa8bff9a640 1 --2- 172.21.15.70:0/419230500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8b8101100 0x7fa8b81a1090 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.381+0000 7fa8bff9a640 1 --2- 172.21.15.70:0/419230500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8b8100760 0x7fa8b819a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.381+0000 7fa8bff9a640 1 --2- 172.21.15.70:0/419230500 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa8a40a2500 0x7fa8a40a49c0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.381+0000 7fa8bff9a640 1 --2- 172.21.15.70:0/419230500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8b80ff630 0x7fa8b8199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.381+0000 7fa8bff9a640 1 -- 172.21.15.70:0/419230500 >> 172.21.15.70:0/419230500 conn(0x7fa8b80faf30 msgr2=0x7fa8b80fc650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.382+0000 7fa8bff9a640 1 -- 172.21.15.70:0/419230500 shutdown_connections 2024-04-01T01:43:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:07.382+0000 7fa8bff9a640 1 -- 172.21.15.70:0/419230500 wait complete. 2024-04-01T01:43:08.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:07 smithi081 ceph-mon[35142]: pgmap v648: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.3 MiB/s wr, 851 op/s 2024-04-01T01:43:08.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:07 smithi189 ceph-mon[35753]: pgmap v648: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.3 MiB/s wr, 851 op/s 2024-04-01T01:43:08.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:07 smithi070 ceph-mon[32179]: pgmap v648: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.3 MiB/s wr, 851 op/s 2024-04-01T01:43:09.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:08 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:43:09.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:08 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:43:09.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:08 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:43:09.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:08 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:43:09.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:08 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:43:09.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:08 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:43:09.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:08 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:43:09.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:08 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:43:09.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:08 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:43:10.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:09 smithi189 ceph-mon[35753]: pgmap v649: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.4 MiB/s wr, 845 op/s 2024-04-01T01:43:10.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:09 smithi070 ceph-mon[32179]: pgmap v649: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.4 MiB/s wr, 845 op/s 2024-04-01T01:43:10.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:09 smithi081 ceph-mon[35142]: pgmap v649: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 7.4 MiB/s wr, 845 op/s 2024-04-01T01:43:11.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 505.0 s, 706.6 tps, lat 142.002 ms stddev 146.642 2024-04-01T01:43:12.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:11 smithi189 ceph-mon[35753]: pgmap v650: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.6 MiB/s wr, 755 op/s 2024-04-01T01:43:12.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:11 smithi070 ceph-mon[32179]: pgmap v650: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.6 MiB/s wr, 755 op/s 2024-04-01T01:43:12.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:11 smithi081 ceph-mon[35142]: pgmap v650: 129 pgs: 129 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.6 MiB/s wr, 755 op/s 2024-04-01T01:43:13.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:12 smithi189 ceph-mon[35753]: pgmap v651: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.6 MiB/s wr, 822 op/s 2024-04-01T01:43:13.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:12 smithi070 ceph-mon[32179]: pgmap v651: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.6 MiB/s wr, 822 op/s 2024-04-01T01:43:13.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:12 smithi081 ceph-mon[35142]: pgmap v651: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.6 MiB/s wr, 822 op/s 2024-04-01T01:43:15.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:14 smithi189 ceph-mon[35753]: pgmap v652: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 6.3 MiB/s wr, 691 op/s 2024-04-01T01:43:15.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:14 smithi070 ceph-mon[32179]: pgmap v652: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 6.3 MiB/s wr, 691 op/s 2024-04-01T01:43:15.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:14 smithi081 ceph-mon[35142]: pgmap v652: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 6.3 MiB/s wr, 691 op/s 2024-04-01T01:43:15.960 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:43:15.961 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:43:16.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.153+0000 7f5ab9a02640 1 -- 172.21.15.70:0/3158445240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ab4100590 msgr2=0x7f5ab41009b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:16.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.153+0000 7f5ab9a02640 1 --2- 172.21.15.70:0/3158445240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ab4100590 0x7f5ab41009b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4009900 tx=0x7f5aa402bc30 comp rx=0 tx=0).stop 2024-04-01T01:43:16.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.153+0000 7f5ab9a02640 1 -- 172.21.15.70:0/3158445240 shutdown_connections 2024-04-01T01:43:16.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.153+0000 7f5ab9a02640 1 --2- 172.21.15.70:0/3158445240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ab4102140 0x7f5ab41089d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.153+0000 7f5ab9a02640 1 --2- 172.21.15.70:0/3158445240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ab4101780 0x7f5ab4101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.153+0000 7f5ab9a02640 1 --2- 172.21.15.70:0/3158445240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ab4100590 0x7f5ab41009b0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.153+0000 7f5ab9a02640 1 -- 172.21.15.70:0/3158445240 >> 172.21.15.70:0/3158445240 conn(0x7f5ab40fbb00 msgr2=0x7f5ab40fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:16.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.153+0000 7f5ab9a02640 1 -- 172.21.15.70:0/3158445240 shutdown_connections 2024-04-01T01:43:16.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.153+0000 7f5ab9a02640 1 -- 172.21.15.70:0/3158445240 wait complete. 2024-04-01T01:43:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.154+0000 7f5ab9a02640 1 Processor -- start 2024-04-01T01:43:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.154+0000 7f5ab9a02640 1 -- start start 2024-04-01T01:43:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.155+0000 7f5ab9a02640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ab4100590 0x7f5ab4199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.155+0000 7f5ab9a02640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ab4101780 0x7f5ab419a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.155+0000 7f5ab9a02640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ab4102140 0x7f5ab41a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.155+0000 7f5ab9a02640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab410c0d0 con 0x7f5ab4101780 2024-04-01T01:43:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.155+0000 7f5ab2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ab4100590 0x7f5ab4199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.155+0000 7f5ab2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ab4100590 0x7f5ab4199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41744/0 (socket says 172.21.15.70:41744) 2024-04-01T01:43:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.155+0000 7f5ab2ffd640 1 -- 172.21.15.70:0/3542308265 learned_addr learned my addr 172.21.15.70:0/3542308265 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.155+0000 7f5ab37fe640 1 --2- 172.21.15.70:0/3542308265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ab4102140 0x7f5ab41a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.155+0000 7f5ab27fc640 1 --2- 172.21.15.70:0/3542308265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ab4101780 0x7f5ab419a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.155+0000 7f5ab9a02640 1 -- 172.21.15.70:0/3542308265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab410bf50 con 0x7f5ab4100590 2024-04-01T01:43:16.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.155+0000 7f5ab9a02640 1 -- 172.21.15.70:0/3542308265 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab410c250 con 0x7f5ab4102140 2024-04-01T01:43:16.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.155+0000 7f5ab2ffd640 1 -- 172.21.15.70:0/3542308265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ab4102140 msgr2=0x7f5ab41a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:16.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.155+0000 7f5ab2ffd640 1 --2- 172.21.15.70:0/3542308265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ab4102140 0x7f5ab41a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.155+0000 7f5ab2ffd640 1 -- 172.21.15.70:0/3542308265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ab4101780 msgr2=0x7f5ab419a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:16.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.155+0000 7f5ab2ffd640 1 --2- 172.21.15.70:0/3542308265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ab4101780 0x7f5ab419a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.155+0000 7f5ab2ffd640 1 -- 172.21.15.70:0/3542308265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab41a1720 con 0x7f5ab4100590 2024-04-01T01:43:16.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.156+0000 7f5ab2ffd640 1 --2- 172.21.15.70:0/3542308265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ab4100590 0x7f5ab4199b00 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4006010 tx=0x7f5aa402de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:16.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.156+0000 7f5ab8a00640 1 -- 172.21.15.70:0/3542308265 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4043020 con 0x7f5ab4100590 2024-04-01T01:43:16.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.156+0000 7f5ab9a02640 1 -- 172.21.15.70:0/3542308265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab41a19b0 con 0x7f5ab4100590 2024-04-01T01:43:16.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.156+0000 7f5ab8a00640 1 -- 172.21.15.70:0/3542308265 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5aa40048d0 con 0x7f5ab4100590 2024-04-01T01:43:16.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.157+0000 7f5ab8a00640 1 -- 172.21.15.70:0/3542308265 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4005220 con 0x7f5ab4100590 2024-04-01T01:43:16.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.157+0000 7f5ab9a02640 1 -- 172.21.15.70:0/3542308265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab41a1e40 con 0x7f5ab4100590 2024-04-01T01:43:16.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.159+0000 7f5ab8a00640 1 -- 172.21.15.70:0/3542308265 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f5aa4046020 con 0x7f5ab4100590 2024-04-01T01:43:16.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.159+0000 7f5ab8a00640 1 --2- 172.21.15.70:0/3542308265 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5a9c0a25f0 0x7f5a9c0a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.159+0000 7f5ab8a00640 1 -- 172.21.15.70:0/3542308265 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5aa40f0c60 con 0x7f5ab4100590 2024-04-01T01:43:16.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.159+0000 7f5ab27fc640 1 --2- 172.21.15.70:0/3542308265 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5a9c0a25f0 0x7f5a9c0a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.160+0000 7f5ab27fc640 1 --2- 172.21.15.70:0/3542308265 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5a9c0a25f0 0x7f5a9c0a4ab0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f5ab419b010 tx=0x7f5aa8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.160+0000 7f5abc5e7740 1 Processor -- start 2024-04-01T01:43:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.160+0000 7f5abc5e7740 1 -- start start 2024-04-01T01:43:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5abc5e7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b752b490 0x5580b752b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5abc5e7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752be80 0x5580b7530980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5abc5e7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7530ff0 0x5580b7533480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5abc5e7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5580b7490180 con 0x5580b752be80 2024-04-01T01:43:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5abc5e7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5580b7438b70 con 0x5580b752b490 2024-04-01T01:43:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5abc5e7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5580b748c220 con 0x5580b7530ff0 2024-04-01T01:43:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5ab2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b752b490 0x5580b752b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5ab27fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752be80 0x5580b7530980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5ab27fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752be80 0x5580b7530980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46242/0 (socket says 172.21.15.70:46242) 2024-04-01T01:43:16.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5ab37fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7530ff0 0x5580b7533480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5ab27fc640 1 -- 172.21.15.70:0/3407881400 learned_addr learned my addr 172.21.15.70:0/3407881400 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:16.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5ab2ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b752b490 0x5580b752b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41752/0 (socket says 172.21.15.70:41752) 2024-04-01T01:43:16.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5ab27fc640 1 -- 172.21.15.70:0/3407881400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7530ff0 msgr2=0x5580b7533480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:16.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5ab27fc640 1 --2- 172.21.15.70:0/3407881400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7530ff0 0x5580b7533480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5ab27fc640 1 -- 172.21.15.70:0/3407881400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b752b490 msgr2=0x5580b752b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:16.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5ab27fc640 1 --2- 172.21.15.70:0/3407881400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b752b490 0x5580b752b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5ab27fc640 1 -- 172.21.15.70:0/3407881400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580b744c7c0 con 0x5580b752be80 2024-04-01T01:43:16.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5ab37fe640 1 --2- 172.21.15.70:0/3407881400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7530ff0 0x5580b7533480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:43:16.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.161+0000 7f5ab27fc640 1 --2- 172.21.15.70:0/3407881400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752be80 0x5580b7530980 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8011f90 tx=0x7f5aa8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c430ecd9bf7274 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:16.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.162+0000 7f5a99ffb640 1 -- 172.21.15.70:0/3407881400 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa801f620 con 0x5580b752be80 2024-04-01T01:43:16.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.162+0000 7f5a99ffb640 1 -- 172.21.15.70:0/3407881400 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5aa801fc70 con 0x5580b752be80 2024-04-01T01:43:16.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.162+0000 7f5a99ffb640 1 -- 172.21.15.70:0/3407881400 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8027c70 con 0x5580b752be80 2024-04-01T01:43:16.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.163+0000 7f5abc5e7740 1 -- 172.21.15.70:0/3407881400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752be80 msgr2=0x5580b7530980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:16.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.163+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/3407881400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752be80 0x5580b7530980 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8011f90 tx=0x7f5aa8014000 comp rx=0 tx=0).stop 2024-04-01T01:43:16.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.164+0000 7f5abc5e7740 1 -- 172.21.15.70:0/3407881400 shutdown_connections 2024-04-01T01:43:16.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.164+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/3407881400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b7530ff0 0x5580b7533480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.164+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/3407881400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752be80 0x5580b7530980 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.164+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/3407881400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b752b490 0x5580b752b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.164+0000 7f5abc5e7740 1 -- 172.21.15.70:0/3407881400 >> 172.21.15.70:0/3407881400 conn(0x5580b7526a00 msgr2=0x5580b7526e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:16.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.164+0000 7f5abc5e7740 1 -- 172.21.15.70:0/3407881400 shutdown_connections 2024-04-01T01:43:16.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.164+0000 7f5abc5e7740 1 -- 172.21.15.70:0/3407881400 wait complete. 2024-04-01T01:43:16.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.165+0000 7f5abc5e7740 1 -- start start 2024-04-01T01:43:16.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.165+0000 7f5abc5e7740 1 Processor -- start 2024-04-01T01:43:16.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.166+0000 7f5abc5e7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752b490 0x5580b7535a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.166+0000 7f5abc5e7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752be80 0x5580b7535f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.166+0000 7f5abc5e7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 0x5580b753cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.166+0000 7f5abc5e7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5580b7533fa0 con 0x5580b752be80 2024-04-01T01:43:16.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.166+0000 7f5ab2ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752b490 0x5580b7535a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.166+0000 7f5ab2ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752b490 0x5580b7535a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:54820/0 (socket says 172.21.15.70:54820) 2024-04-01T01:43:16.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.166+0000 7f5ab2ffd640 1 -- 172.21.15.70:0/3636565323 learned_addr learned my addr 172.21.15.70:0/3636565323 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:16.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.166+0000 7f5ab27fc640 1 --2- 172.21.15.70:0/3636565323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752be80 0x5580b7535f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.166+0000 7f5ab37fe640 1 --2- 172.21.15.70:0/3636565323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 0x5580b753cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.166+0000 7f5abc5e7740 1 -- 172.21.15.70:0/3636565323 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5580b742e4a0 con 0x5580b7530ff0 2024-04-01T01:43:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.167+0000 7f5abc5e7740 1 -- 172.21.15.70:0/3636565323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5580b7534120 con 0x5580b752b490 2024-04-01T01:43:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.167+0000 7f5abc5e7740 10 client.0 _ll_register_callbacks cb 0x5580b7431d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.167+0000 7f5ab37fe640 1 -- 172.21.15.70:0/3636565323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752b490 msgr2=0x5580b7535a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.167+0000 7f5ab37fe640 1 --2- 172.21.15.70:0/3636565323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752b490 0x5580b7535a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.167+0000 7f5ab37fe640 1 -- 172.21.15.70:0/3636565323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752be80 msgr2=0x5580b7535f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.167+0000 7f5ab37fe640 1 --2- 172.21.15.70:0/3636565323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752be80 0x5580b7535f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.167+0000 7f5ab37fe640 1 -- 172.21.15.70:0/3636565323 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580b726e7d0 con 0x5580b7530ff0 2024-04-01T01:43:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.167+0000 7f5ab2ffd640 1 --2- 172.21.15.70:0/3636565323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752b490 0x5580b7535a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:43:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.167+0000 7f5ab37fe640 1 --2- 172.21.15.70:0/3636565323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 0x5580b753cf80 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5aac005fd0 tx=0x7f5aac011e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a5ac65251ba4a81 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.167+0000 7f5a92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.168+0000 7f5a92ffd640 1 -- 172.21.15.70:0/3636565323 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac003c40 con 0x5580b7530ff0 2024-04-01T01:43:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.168+0000 7f5a92ffd640 1 -- 172.21.15.70:0/3636565323 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5aac003de0 con 0x5580b7530ff0 2024-04-01T01:43:16.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.168+0000 7f5abc5e7740 1 -- 172.21.15.70:0/3636565323 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5aa801fc70 con 0x5580b7530ff0 2024-04-01T01:43:16.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.168+0000 7f5a92ffd640 1 -- 172.21.15.70:0/3636565323 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5aac00f950 con 0x5580b7530ff0 2024-04-01T01:43:16.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.168+0000 7f5abc5e7740 10 client.28316 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:16.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.168+0000 7f5abc5e7740 1 -- 172.21.15.70:0/3636565323 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5580b71a43f0 con 0x5580b7530ff0 2024-04-01T01:43:16.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.169+0000 7f5a92ffd640 1 -- 172.21.15.70:0/3636565323 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5aa801fc70 con 0x5580b7530ff0 2024-04-01T01:43:16.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.169+0000 7f5a92ffd640 1 -- 172.21.15.70:0/3636565323 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5aac01a410 con 0x5580b7530ff0 2024-04-01T01:43:16.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.169+0000 7f5abc5e7740 10 client.28316 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:16.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.169+0000 7f5abc5e7740 10 client.28316 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:16.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.169+0000 7f5abc5e7740 20 client.28316 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:16.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.170+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/3636565323 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5580b7541c60 0x5580b7544100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.170+0000 7f5abc5e7740 4 client.28316 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:16.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.170+0000 7f5abc5e7740 1 -- 172.21.15.70:0/3636565323 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580b749d6c0 con 0x5580b7541c60 2024-04-01T01:43:16.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.170+0000 7f5ab2ffd640 1 --2- 172.21.15.70:0/3636565323 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5580b7541c60 0x5580b7544100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.170+0000 7f5a92ffd640 10 client.28316 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:16.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.170+0000 7f5ab2ffd640 1 --2- 172.21.15.70:0/3636565323 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5580b7541c60 0x5580b7544100 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=443421586422f7fe server_cookie=98d58e337581ddb3 in_seq=0 out_seq=0 2024-04-01T01:43:16.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.171+0000 7f5a92ffd640 1 -- 172.21.15.70:0/3636565323 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5580b749d6c0 con 0x5580b7541c60 2024-04-01T01:43:16.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.171+0000 7f5a92ffd640 10 client.28316 handle_command_reply: tid=0 2024-04-01T01:43:16.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.171+0000 7f5abc5e7740 1 client.28316 shutdown 2024-04-01T01:43:16.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.171+0000 7f5abc5e7740 1 -- 172.21.15.70:0/3636565323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 msgr2=0x5580b753cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:16.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.171+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/3636565323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 0x5580b753cf80 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5aac005fd0 tx=0x7f5aac011e60 comp rx=0 tx=0).stop 2024-04-01T01:43:16.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.171+0000 7f5abc5e7740 1 -- 172.21.15.70:0/3636565323 shutdown_connections 2024-04-01T01:43:16.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.171+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/3636565323 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5580b7541c60 0x5580b7544100 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.171+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/3636565323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 0x5580b753cf80 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.171+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/3636565323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752be80 0x5580b7535f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.171+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/3636565323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752b490 0x5580b7535a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.171+0000 7f5a92ffd640 0 client.28316 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:16.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.171+0000 7f5abc5e7740 1 -- 172.21.15.70:0/3636565323 >> 172.21.15.70:0/3636565323 conn(0x5580b7526a00 msgr2=0x5580b7525c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:16.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.171+0000 7f5abc5e7740 1 -- 172.21.15.70:0/3636565323 shutdown_connections 2024-04-01T01:43:16.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.171+0000 7f5abc5e7740 1 -- 172.21.15.70:0/3636565323 wait complete. 2024-04-01T01:43:16.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.171+0000 7f5abc5e7740 20 client.28316 trim_cache size 0 max 16384 2024-04-01T01:43:16.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.198+0000 7f5abc5e7740 1 Processor -- start 2024-04-01T01:43:16.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.198+0000 7f5abc5e7740 1 -- start start 2024-04-01T01:43:16.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.198+0000 7f5abc5e7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752b490 0x5580b75344c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.199+0000 7f5abc5e7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752be80 0x5580b753cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.199+0000 7f5abc5e7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 0x5580b7535a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.199+0000 7f5abc5e7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab41ac430 con 0x5580b752b490 2024-04-01T01:43:16.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.199+0000 7f5abc5e7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab41adf70 con 0x5580b7530ff0 2024-04-01T01:43:16.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.199+0000 7f5ab37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 0x5580b7535a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.199+0000 7f5ab37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 0x5580b7535a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41756/0 (socket says 172.21.15.70:41756) 2024-04-01T01:43:16.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.199+0000 7f5ab37fe640 1 -- 172.21.15.70:0/2025427624 learned_addr learned my addr 172.21.15.70:0/2025427624 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:16.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.199+0000 7f5abc5e7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab41acb00 con 0x5580b752be80 2024-04-01T01:43:16.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.199+0000 7f5ab2ffd640 1 --2- 172.21.15.70:0/2025427624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752b490 0x5580b75344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.199+0000 7f5ab27fc640 1 --2- 172.21.15.70:0/2025427624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752be80 0x5580b753cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.200+0000 7f5ab37fe640 1 -- 172.21.15.70:0/2025427624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752be80 msgr2=0x5580b753cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:16.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.200+0000 7f5ab37fe640 1 --2- 172.21.15.70:0/2025427624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752be80 0x5580b753cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.200+0000 7f5ab37fe640 1 -- 172.21.15.70:0/2025427624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752b490 msgr2=0x5580b75344c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:16.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.200+0000 7f5ab37fe640 1 --2- 172.21.15.70:0/2025427624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752b490 0x5580b75344c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.200+0000 7f5ab37fe640 1 -- 172.21.15.70:0/2025427624 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580b71640f0 con 0x5580b7530ff0 2024-04-01T01:43:16.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.200+0000 7f5ab37fe640 1 --2- 172.21.15.70:0/2025427624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 0x5580b7535a00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00a010 tx=0x7f5aac01adb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=388a079bf9cbc463 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:16.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.200+0000 7f5a997fa640 1 -- 172.21.15.70:0/2025427624 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac02f070 con 0x5580b7530ff0 2024-04-01T01:43:16.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.200+0000 7f5a997fa640 1 -- 172.21.15.70:0/2025427624 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5aac009570 con 0x5580b7530ff0 2024-04-01T01:43:16.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.201+0000 7f5a997fa640 1 -- 172.21.15.70:0/2025427624 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac005740 con 0x5580b7530ff0 2024-04-01T01:43:16.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.201+0000 7f5abc5e7740 1 -- 172.21.15.70:0/2025427624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 msgr2=0x5580b7535a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:16.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.201+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/2025427624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 0x5580b7535a00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00a010 tx=0x7f5aac01adb0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.202+0000 7f5abc5e7740 1 -- 172.21.15.70:0/2025427624 shutdown_connections 2024-04-01T01:43:16.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.202+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/2025427624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 0x5580b7535a00 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.202+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/2025427624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752be80 0x5580b753cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.202+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/2025427624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752b490 0x5580b75344c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.202+0000 7f5abc5e7740 1 -- 172.21.15.70:0/2025427624 >> 172.21.15.70:0/2025427624 conn(0x5580b7526a00 msgr2=0x5580b752d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:16.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.202+0000 7f5abc5e7740 1 -- 172.21.15.70:0/2025427624 shutdown_connections 2024-04-01T01:43:16.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.202+0000 7f5abc5e7740 1 -- 172.21.15.70:0/2025427624 wait complete. 2024-04-01T01:43:16.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.202+0000 7f5abc5e7740 1 -- start start 2024-04-01T01:43:16.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.203+0000 7f5abc5e7740 1 Processor -- start 2024-04-01T01:43:16.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.203+0000 7f5abc5e7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752b490 0x5580b753a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.203+0000 7f5abc5e7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752be80 0x5580b753a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.203+0000 7f5abc5e7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 0x5580b753b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.203+0000 7f5abc5e7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab41ad8a0 con 0x5580b752b490 2024-04-01T01:43:16.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.203+0000 7f5abc5e7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab41ad1d0 con 0x5580b7530ff0 2024-04-01T01:43:16.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.203+0000 7f5abc5e7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab41aa410 con 0x5580b752be80 2024-04-01T01:43:16.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.203+0000 7f5abc5e7740 10 client.0 _ll_register_callbacks cb 0x7f5aac009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:16.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.203+0000 7f5ab27fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752be80 0x5580b753a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.203+0000 7f5ab27fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752be80 0x5580b753a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:54844/0 (socket says 172.21.15.70:54844) 2024-04-01T01:43:16.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.203+0000 7f5ab27fc640 1 -- 172.21.15.70:0/1941389579 learned_addr learned my addr 172.21.15.70:0/1941389579 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:16.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.203+0000 7f5ab2ffd640 1 --2- 172.21.15.70:0/1941389579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752b490 0x5580b753a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.203+0000 7f5ab37fe640 1 --2- 172.21.15.70:0/1941389579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 0x5580b753b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.203+0000 7f5ab2ffd640 1 -- 172.21.15.70:0/1941389579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752be80 msgr2=0x5580b753a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:16.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.203+0000 7f5ab2ffd640 1 --2- 172.21.15.70:0/1941389579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752be80 0x5580b753a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.204+0000 7f5ab2ffd640 1 -- 172.21.15.70:0/1941389579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 msgr2=0x5580b753b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:16.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.204+0000 7f5ab2ffd640 1 --2- 172.21.15.70:0/1941389579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 0x5580b753b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.204+0000 7f5ab2ffd640 1 -- 172.21.15.70:0/1941389579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580b753bc10 con 0x5580b752b490 2024-04-01T01:43:16.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.204+0000 7f5ab37fe640 1 --2- 172.21.15.70:0/1941389579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 0x5580b753b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:43:16.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.204+0000 7f5ab2ffd640 1 --2- 172.21.15.70:0/1941389579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752b490 0x5580b753a0d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f5aa40ee3e0 tx=0x7f5aa4004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e86db1b151b2be3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:16.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.204+0000 7f5a92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:43:16.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.204+0000 7f5a92ffd640 1 -- 172.21.15.70:0/1941389579 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa40c14a0 con 0x5580b752b490 2024-04-01T01:43:16.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.204+0000 7f5a92ffd640 1 -- 172.21.15.70:0/1941389579 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5aa40035f0 con 0x5580b752b490 2024-04-01T01:43:16.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.204+0000 7f5abc5e7740 1 -- 172.21.15.70:0/1941389579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5aac009570 con 0x5580b752b490 2024-04-01T01:43:16.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.204+0000 7f5a92ffd640 1 -- 172.21.15.70:0/1941389579 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5aa40c5d60 con 0x5580b752b490 2024-04-01T01:43:16.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.204+0000 7f5abc5e7740 10 client.18321 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:16.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.204+0000 7f5abc5e7740 1 -- 172.21.15.70:0/1941389579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5580b753cf90 con 0x5580b752b490 2024-04-01T01:43:16.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.205+0000 7f5a92ffd640 1 -- 172.21.15.70:0/1941389579 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5aac009570 con 0x5580b752b490 2024-04-01T01:43:16.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.205+0000 7f5a92ffd640 1 -- 172.21.15.70:0/1941389579 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5aa40d7070 con 0x5580b752b490 2024-04-01T01:43:16.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.205+0000 7f5abc5e7740 10 client.18321 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:16.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.205+0000 7f5abc5e7740 10 client.18321 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:16.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.205+0000 7f5abc5e7740 20 client.18321 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:16.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.205+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/1941389579 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5580b75b0cd0 0x5580b75b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.205+0000 7f5abc5e7740 4 client.18321 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:43:16.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.205+0000 7f5abc5e7740 1 -- 172.21.15.70:0/1941389579 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5580b7497a10 con 0x5580b75b0cd0 2024-04-01T01:43:16.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.206+0000 7f5ab27fc640 1 --2- 172.21.15.70:0/1941389579 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5580b75b0cd0 0x5580b75b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.206+0000 7f5ab27fc640 1 --2- 172.21.15.70:0/1941389579 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5580b75b0cd0 0x5580b75b30f0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1de3fcd9ce4b0554 server_cookie=c55efaf627e3c80d in_seq=0 out_seq=0 2024-04-01T01:43:16.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.206+0000 7f5a92ffd640 10 client.18321 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:16.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.207+0000 7f5a92ffd640 1 -- 172.21.15.70:0/1941389579 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5580b7497a10 con 0x5580b75b0cd0 2024-04-01T01:43:16.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.207+0000 7f5a92ffd640 10 client.18321 handle_command_reply: tid=0 2024-04-01T01:43:16.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.207+0000 7f5abc5e7740 1 client.18321 shutdown 2024-04-01T01:43:16.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.208+0000 7f5abc5e7740 1 -- 172.21.15.70:0/1941389579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752b490 msgr2=0x5580b753a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:16.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.208+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/1941389579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752b490 0x5580b753a0d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f5aa40ee3e0 tx=0x7f5aa4004b00 comp rx=0 tx=0).stop 2024-04-01T01:43:16.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.208+0000 7f5abc5e7740 1 -- 172.21.15.70:0/1941389579 shutdown_connections 2024-04-01T01:43:16.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.208+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/1941389579 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5580b75b0cd0 0x5580b75b30f0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.208+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/1941389579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580b7530ff0 0x5580b753b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.208+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/1941389579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5580b752be80 0x5580b753a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.208+0000 7f5abc5e7740 1 --2- 172.21.15.70:0/1941389579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b752b490 0x5580b753a0d0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.208+0000 7f5abc5e7740 1 -- 172.21.15.70:0/1941389579 >> 172.21.15.70:0/1941389579 conn(0x5580b7526a00 msgr2=0x5580b7535940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:16.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.208+0000 7f5a92ffd640 0 client.18321 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:16.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.208+0000 7f5abc5e7740 1 -- 172.21.15.70:0/1941389579 shutdown_connections 2024-04-01T01:43:16.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.208+0000 7f5abc5e7740 1 -- 172.21.15.70:0/1941389579 wait complete. 2024-04-01T01:43:16.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.208+0000 7f5abc5e7740 20 client.18321 trim_cache size 0 max 16384 2024-04-01T01:43:16.213 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:43:16.213 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:43:16.213 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:43:16.213 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:43:16.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.209+0000 7f5ab9a02640 1 -- 172.21.15.70:0/3542308265 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5a9c0a25f0 msgr2=0x7f5a9c0a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:16.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.209+0000 7f5ab9a02640 1 --2- 172.21.15.70:0/3542308265 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5a9c0a25f0 0x7f5a9c0a4ab0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f5ab419b010 tx=0x7f5aa8005f90 comp rx=0 tx=0).stop 2024-04-01T01:43:16.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.210+0000 7f5ab9a02640 1 -- 172.21.15.70:0/3542308265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ab4100590 msgr2=0x7f5ab4199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:16.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.210+0000 7f5ab9a02640 1 --2- 172.21.15.70:0/3542308265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ab4100590 0x7f5ab4199b00 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4006010 tx=0x7f5aa402de10 comp rx=0 tx=0).stop 2024-04-01T01:43:16.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.210+0000 7f5ab9a02640 1 -- 172.21.15.70:0/3542308265 shutdown_connections 2024-04-01T01:43:16.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.210+0000 7f5ab9a02640 1 --2- 172.21.15.70:0/3542308265 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5a9c0a25f0 0x7f5a9c0a4ab0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.210+0000 7f5ab9a02640 1 --2- 172.21.15.70:0/3542308265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ab4102140 0x7f5ab41a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.210+0000 7f5ab9a02640 1 --2- 172.21.15.70:0/3542308265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ab4101780 0x7f5ab419a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.210+0000 7f5ab9a02640 1 --2- 172.21.15.70:0/3542308265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ab4100590 0x7f5ab4199b00 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.210+0000 7f5ab9a02640 1 -- 172.21.15.70:0/3542308265 >> 172.21.15.70:0/3542308265 conn(0x7f5ab40fbb00 msgr2=0x7f5ab40fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:16.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.211+0000 7f5ab9a02640 1 -- 172.21.15.70:0/3542308265 shutdown_connections 2024-04-01T01:43:16.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.211+0000 7f5ab9a02640 1 -- 172.21.15.70:0/3542308265 wait complete. 2024-04-01T01:43:16.229 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:43:16.229 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:43:16.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.414+0000 7f66b610a640 1 -- 172.21.15.70:0/3773240201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b0102030 msgr2=0x7f66b01088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:16.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.414+0000 7f66b610a640 1 --2- 172.21.15.70:0/3773240201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b0102030 0x7f66b01088c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009ca0 tx=0x7f66a802c0f0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.414+0000 7f66b610a640 1 -- 172.21.15.70:0/3773240201 shutdown_connections 2024-04-01T01:43:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.414+0000 7f66b610a640 1 --2- 172.21.15.70:0/3773240201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b0102030 0x7f66b01088c0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.414+0000 7f66b610a640 1 --2- 172.21.15.70:0/3773240201 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66b0101670 0x7f66b0101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.414+0000 7f66b610a640 1 --2- 172.21.15.70:0/3773240201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b0100480 0x7f66b01008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.414+0000 7f66b610a640 1 -- 172.21.15.70:0/3773240201 >> 172.21.15.70:0/3773240201 conn(0x7f66b00fb9d0 msgr2=0x7f66b00fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.415+0000 7f66b610a640 1 -- 172.21.15.70:0/3773240201 shutdown_connections 2024-04-01T01:43:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.415+0000 7f66b610a640 1 -- 172.21.15.70:0/3773240201 wait complete. 2024-04-01T01:43:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.416+0000 7f66b610a640 1 Processor -- start 2024-04-01T01:43:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.416+0000 7f66b610a640 1 -- start start 2024-04-01T01:43:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.416+0000 7f66b610a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b0100480 0x7f66b019de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.416+0000 7f66b610a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66b0101670 0x7f66b019e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.416+0000 7f66b610a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b0102030 0x7f66b01a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.416+0000 7f66b610a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f66b010aeb0 con 0x7f66b0101670 2024-04-01T01:43:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.416+0000 7f66b610a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f66b010ad30 con 0x7f66b0102030 2024-04-01T01:43:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.417+0000 7f66b610a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f66b010b030 con 0x7f66b0100480 2024-04-01T01:43:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.417+0000 7f66b5108640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b0100480 0x7f66b019de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.417+0000 7f66b5108640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b0100480 0x7f66b019de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:54856/0 (socket says 172.21.15.70:54856) 2024-04-01T01:43:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.417+0000 7f66b5108640 1 -- 172.21.15.70:0/3187291778 learned_addr learned my addr 172.21.15.70:0/3187291778 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.417+0000 7f66b5909640 1 --2- 172.21.15.70:0/3187291778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b0102030 0x7f66b01a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.417+0000 7f66b4907640 1 --2- 172.21.15.70:0/3187291778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66b0101670 0x7f66b019e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.417+0000 7f66b5909640 1 -- 172.21.15.70:0/3187291778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b0100480 msgr2=0x7f66b019de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.417+0000 7f66b5909640 1 --2- 172.21.15.70:0/3187291778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b0100480 0x7f66b019de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.417+0000 7f66b5909640 1 -- 172.21.15.70:0/3187291778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66b0101670 msgr2=0x7f66b019e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:16.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.417+0000 7f66b5909640 1 --2- 172.21.15.70:0/3187291778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66b0101670 0x7f66b019e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.417+0000 7f66b5909640 1 -- 172.21.15.70:0/3187291778 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b01a5a30 con 0x7f66b0102030 2024-04-01T01:43:16.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.418+0000 7f66b5909640 1 --2- 172.21.15.70:0/3187291778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b0102030 0x7f66b01a53c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f66a8037040 tx=0x7f66a8032820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:16.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.418+0000 7f66a67fc640 1 -- 172.21.15.70:0/3187291778 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8041020 con 0x7f66b0102030 2024-04-01T01:43:16.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.418+0000 7f66a67fc640 1 -- 172.21.15.70:0/3187291778 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f66a8003970 con 0x7f66b0102030 2024-04-01T01:43:16.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.418+0000 7f66b610a640 1 -- 172.21.15.70:0/3187291778 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b01a5cc0 con 0x7f66b0102030 2024-04-01T01:43:16.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.419+0000 7f66a67fc640 1 -- 172.21.15.70:0/3187291778 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8039040 con 0x7f66b0102030 2024-04-01T01:43:16.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.419+0000 7f66b610a640 1 -- 172.21.15.70:0/3187291778 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b01a61b0 con 0x7f66b0102030 2024-04-01T01:43:16.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.421+0000 7f66b610a640 1 -- 172.21.15.70:0/3187291778 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66b01056c0 con 0x7f66b0102030 2024-04-01T01:43:16.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.422+0000 7f66a67fc640 1 -- 172.21.15.70:0/3187291778 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f66a8044020 con 0x7f66b0102030 2024-04-01T01:43:16.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.423+0000 7f66a67fc640 1 --2- 172.21.15.70:0/3187291778 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f669c0a25d0 0x7f669c0a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:16.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.423+0000 7f66a67fc640 1 -- 172.21.15.70:0/3187291778 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f66a80f0a30 con 0x7f66b0102030 2024-04-01T01:43:16.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.423+0000 7f66b5108640 1 --2- 172.21.15.70:0/3187291778 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f669c0a25d0 0x7f669c0a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:16.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.427+0000 7f66b5108640 1 --2- 172.21.15.70:0/3187291778 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f669c0a25d0 0x7f669c0a4a90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f66a000bc80 tx=0x7f66a000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:16.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.431+0000 7f66a67fc640 1 -- 172.21.15.70:0/3187291778 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f66a804b020 con 0x7f66b0102030 2024-04-01T01:43:16.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.841+0000 7f66b610a640 1 -- 172.21.15.70:0/3187291778 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66b002d170 con 0x7f66b0102030 2024-04-01T01:43:16.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.843+0000 7f66a67fc640 1 -- 172.21.15.70:0/3187291778 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7f66a80bd7c0 con 0x7f66b0102030 2024-04-01T01:43:16.842 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:43:16.843 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:43:16.843 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:43:16.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.848+0000 7f66b610a640 1 -- 172.21.15.70:0/3187291778 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f669c0a25d0 msgr2=0x7f669c0a4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:16.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.848+0000 7f66b610a640 1 --2- 172.21.15.70:0/3187291778 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f669c0a25d0 0x7f669c0a4a90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f66a000bc80 tx=0x7f66a000b410 comp rx=0 tx=0).stop 2024-04-01T01:43:16.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.848+0000 7f66b610a640 1 -- 172.21.15.70:0/3187291778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b0102030 msgr2=0x7f66b01a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:16.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.848+0000 7f66b610a640 1 --2- 172.21.15.70:0/3187291778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b0102030 0x7f66b01a53c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f66a8037040 tx=0x7f66a8032820 comp rx=0 tx=0).stop 2024-04-01T01:43:16.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.848+0000 7f66b610a640 1 -- 172.21.15.70:0/3187291778 shutdown_connections 2024-04-01T01:43:16.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.848+0000 7f66b610a640 1 --2- 172.21.15.70:0/3187291778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66b0102030 0x7f66b01a53c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.848+0000 7f66b610a640 1 --2- 172.21.15.70:0/3187291778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66b0101670 0x7f66b019e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.848+0000 7f66b610a640 1 --2- 172.21.15.70:0/3187291778 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f669c0a25d0 0x7f669c0a4a90 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.848+0000 7f66b610a640 1 --2- 172.21.15.70:0/3187291778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b0100480 0x7f66b019de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:16.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.848+0000 7f66b610a640 1 -- 172.21.15.70:0/3187291778 >> 172.21.15.70:0/3187291778 conn(0x7f66b00fb9d0 msgr2=0x7f66b0102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:16.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.851+0000 7f66b610a640 1 -- 172.21.15.70:0/3187291778 shutdown_connections 2024-04-01T01:43:16.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:16.851+0000 7f66b610a640 1 -- 172.21.15.70:0/3187291778 wait complete. 2024-04-01T01:43:16.868 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:43:16.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 510.0 s, 782.0 tps, lat 127.488 ms stddev 140.708 2024-04-01T01:43:17.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.024+0000 7f861760b640 1 -- 172.21.15.70:0/1779534636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8610102950 msgr2=0x7f8610102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.024+0000 7f861760b640 1 --2- 172.21.15.70:0/1779534636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8610102950 0x7f8610102dd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f860c009860 tx=0x7f860c02bb70 comp rx=0 tx=0).stop 2024-04-01T01:43:17.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.024+0000 7f861760b640 1 -- 172.21.15.70:0/1779534636 shutdown_connections 2024-04-01T01:43:17.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.024+0000 7f861760b640 1 --2- 172.21.15.70:0/1779534636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8610103310 0x7f8610109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.024+0000 7f861760b640 1 --2- 172.21.15.70:0/1779534636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8610102950 0x7f8610102dd0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.024+0000 7f861760b640 1 --2- 172.21.15.70:0/1779534636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8610101760 0x7f8610101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.024+0000 7f861760b640 1 -- 172.21.15.70:0/1779534636 >> 172.21.15.70:0/1779534636 conn(0x7f86100fccd0 msgr2=0x7f86100ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.025+0000 7f861760b640 1 -- 172.21.15.70:0/1779534636 shutdown_connections 2024-04-01T01:43:17.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.025+0000 7f861760b640 1 -- 172.21.15.70:0/1779534636 wait complete. 2024-04-01T01:43:17.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.026+0000 7f861760b640 1 Processor -- start 2024-04-01T01:43:17.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.026+0000 7f861760b640 1 -- start start 2024-04-01T01:43:17.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.026+0000 7f861760b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8610101760 0x7f8610195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.026+0000 7f861760b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8610102950 0x7f8610195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.026+0000 7f861760b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8610103310 0x7f86101ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.026+0000 7f861760b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f861010d270 con 0x7f8610103310 2024-04-01T01:43:17.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.026+0000 7f861760b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f861010d0f0 con 0x7f8610102950 2024-04-01T01:43:17.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.026+0000 7f861760b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f861010d3f0 con 0x7f8610101760 2024-04-01T01:43:17.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.026+0000 7f8614ba6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8610102950 0x7f8610195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.026+0000 7f8614ba6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8610102950 0x7f8610195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49180/0 (socket says 172.21.15.70:49180) 2024-04-01T01:43:17.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.027+0000 7f8614ba6640 1 -- 172.21.15.70:0/2797502102 learned_addr learned my addr 172.21.15.70:0/2797502102 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.027+0000 7f8615ba8640 1 --2- 172.21.15.70:0/2797502102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8610103310 0x7f86101ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.027+0000 7f86153a7640 1 --2- 172.21.15.70:0/2797502102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8610101760 0x7f8610195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.027+0000 7f8614ba6640 1 -- 172.21.15.70:0/2797502102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8610101760 msgr2=0x7f8610195810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.027+0000 7f8614ba6640 1 --2- 172.21.15.70:0/2797502102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8610101760 0x7f8610195810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.027+0000 7f8614ba6640 1 -- 172.21.15.70:0/2797502102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8610103310 msgr2=0x7f86101ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.027+0000 7f8614ba6640 1 --2- 172.21.15.70:0/2797502102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8610103310 0x7f86101ae180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.027+0000 7f8614ba6640 1 -- 172.21.15.70:0/2797502102 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f861019aee0 con 0x7f8610102950 2024-04-01T01:43:17.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.027+0000 7f8614ba6640 1 --2- 172.21.15.70:0/2797502102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8610102950 0x7f8610195d50 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f860c0057c0 tx=0x7f860c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.027+0000 7f86067fc640 1 -- 172.21.15.70:0/2797502102 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c041030 con 0x7f8610102950 2024-04-01T01:43:17.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.028+0000 7f86067fc640 1 -- 172.21.15.70:0/2797502102 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f860c02c9c0 con 0x7f8610102950 2024-04-01T01:43:17.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.028+0000 7f86067fc640 1 -- 172.21.15.70:0/2797502102 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c039040 con 0x7f8610102950 2024-04-01T01:43:17.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.028+0000 7f861760b640 1 -- 172.21.15.70:0/2797502102 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86101ae720 con 0x7f8610102950 2024-04-01T01:43:17.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.028+0000 7f861760b640 1 -- 172.21.15.70:0/2797502102 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86101aec10 con 0x7f8610102950 2024-04-01T01:43:17.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.031+0000 7f86067fc640 1 -- 172.21.15.70:0/2797502102 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f860c044070 con 0x7f8610102950 2024-04-01T01:43:17.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.031+0000 7f861a1df740 1 Processor -- start 2024-04-01T01:43:17.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.032+0000 7f861a1df740 1 -- start start 2024-04-01T01:43:17.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.032+0000 7f861a1df740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 0x55fc6476d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.032+0000 7f861a1df740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc6476de80 0x55fc64772980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.032+0000 7f861a1df740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc64772ff0 0x55fc64775480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.032+0000 7f861a1df740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fc646d21b0 con 0x55fc6476de80 2024-04-01T01:43:17.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.032+0000 7f861a1df740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55fc6467ac00 con 0x55fc6476d490 2024-04-01T01:43:17.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.032+0000 7f861a1df740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fc646ce250 con 0x55fc64772ff0 2024-04-01T01:43:17.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.033+0000 7f86067fc640 1 --2- 172.21.15.70:0/2797502102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85fc0a25a0 0x7f85fc0a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.033+0000 7f86153a7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc6476de80 0x55fc64772980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.033+0000 7f86153a7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc6476de80 0x55fc64772980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34080/0 (socket says 172.21.15.70:34080) 2024-04-01T01:43:17.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.033+0000 7f86153a7640 1 -- 172.21.15.70:0/2158201222 learned_addr learned my addr 172.21.15.70:0/2158201222 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.034+0000 7f86067fc640 1 -- 172.21.15.70:0/2797502102 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f860c0f0070 con 0x7f8610102950 2024-04-01T01:43:17.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.034+0000 7f8615ba8640 1 --2- 172.21.15.70:0/2158201222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 0x55fc6476d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.034+0000 7f8614ba6640 1 --2- 172.21.15.70:0/2158201222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc64772ff0 0x55fc64775480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.034+0000 7f8615ba8640 1 --2- 172.21.15.70:0/2797502102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85fc0a25a0 0x7f85fc0a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.034+0000 7f8614ba6640 1 -- 172.21.15.70:0/2158201222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 msgr2=0x55fc6476d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.035+0000 7f8614ba6640 1 --2- 172.21.15.70:0/2158201222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 0x55fc6476d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.035+0000 7f8614ba6640 1 -- 172.21.15.70:0/2158201222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc6476de80 msgr2=0x55fc64772980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.035+0000 7f8614ba6640 1 --2- 172.21.15.70:0/2158201222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc6476de80 0x55fc64772980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.035+0000 7f8614ba6640 1 -- 172.21.15.70:0/2158201222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc6469a3d0 con 0x55fc64772ff0 2024-04-01T01:43:17.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.035+0000 7f8614ba6640 1 --2- 172.21.15.70:0/2158201222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc64772ff0 0x55fc64775480 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f860c0bc690 tx=0x7f860c039f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a66a5f418980c553 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.035+0000 7f85fb7fe640 1 -- 172.21.15.70:0/2158201222 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c0d5070 con 0x55fc64772ff0 2024-04-01T01:43:17.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.036+0000 7f8615ba8640 1 --2- 172.21.15.70:0/2797502102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85fc0a25a0 0x7f85fc0a4a60 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x55fc6476e6e0 tx=0x7f8608004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.039+0000 7f85fb7fe640 1 -- 172.21.15.70:0/2158201222 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f860c0bce00 con 0x55fc64772ff0 2024-04-01T01:43:17.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.040+0000 7f861a1df740 1 -- 172.21.15.70:0/2158201222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc64772ff0 msgr2=0x55fc64775480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.040+0000 7f861a1df740 1 --2- 172.21.15.70:0/2158201222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc64772ff0 0x55fc64775480 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f860c0bc690 tx=0x7f860c039f90 comp rx=0 tx=0).stop 2024-04-01T01:43:17.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.041+0000 7f861a1df740 1 -- 172.21.15.70:0/2158201222 shutdown_connections 2024-04-01T01:43:17.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.041+0000 7f861a1df740 1 --2- 172.21.15.70:0/2158201222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc64772ff0 0x55fc64775480 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.041+0000 7f861a1df740 1 --2- 172.21.15.70:0/2158201222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc6476de80 0x55fc64772980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.041+0000 7f861a1df740 1 --2- 172.21.15.70:0/2158201222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 0x55fc6476d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.041+0000 7f861a1df740 1 -- 172.21.15.70:0/2158201222 >> 172.21.15.70:0/2158201222 conn(0x55fc64768a00 msgr2=0x55fc64768e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.041+0000 7f861a1df740 1 -- 172.21.15.70:0/2158201222 shutdown_connections 2024-04-01T01:43:17.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.041+0000 7f861a1df740 1 -- 172.21.15.70:0/2158201222 wait complete. 2024-04-01T01:43:17.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.042+0000 7f861a1df740 1 -- start start 2024-04-01T01:43:17.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.042+0000 7f861a1df740 1 Processor -- start 2024-04-01T01:43:17.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.042+0000 7f861a1df740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 0x55fc647775b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.042+0000 7f861a1df740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc6476de80 0x55fc64777af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.042+0000 7f861a1df740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc64772ff0 0x55fc6477eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.042+0000 7f861a1df740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fc646ce250 con 0x55fc64772ff0 2024-04-01T01:43:17.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.042+0000 7f861a1df740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55fc64670470 con 0x55fc6476d490 2024-04-01T01:43:17.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.042+0000 7f861a1df740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fc64775d60 con 0x55fc6476de80 2024-04-01T01:43:17.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.042+0000 7f861a1df740 10 client.0 _ll_register_callbacks cb 0x55fc6468dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:17.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.043+0000 7f86153a7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 0x55fc647775b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.043+0000 7f8615ba8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc64772ff0 0x55fc6477eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.043+0000 7f86153a7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 0x55fc647775b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49198/0 (socket says 172.21.15.70:49198) 2024-04-01T01:43:17.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.043+0000 7f86153a7640 1 -- 172.21.15.70:0/3005115367 learned_addr learned my addr 172.21.15.70:0/3005115367 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.043+0000 7f8614ba6640 1 --2- 172.21.15.70:0/3005115367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc6476de80 0x55fc64777af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.043+0000 7f86153a7640 1 -- 172.21.15.70:0/3005115367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc6476de80 msgr2=0x55fc64777af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.043+0000 7f86153a7640 1 --2- 172.21.15.70:0/3005115367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc6476de80 0x55fc64777af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.043+0000 7f86153a7640 1 -- 172.21.15.70:0/3005115367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc64772ff0 msgr2=0x55fc6477eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.043+0000 7f86153a7640 1 --2- 172.21.15.70:0/3005115367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc64772ff0 0x55fc6477eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.043+0000 7f86153a7640 1 -- 172.21.15.70:0/3005115367 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc643e63f0 con 0x55fc6476d490 2024-04-01T01:43:17.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.044+0000 7f86153a7640 1 --2- 172.21.15.70:0/3005115367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 0x55fc647775b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8600009b90 tx=0x7f86000039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ca62bc9d073729e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.044+0000 7f85f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.044+0000 7f85f8ff9640 1 -- 172.21.15.70:0/3005115367 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8600012860 con 0x55fc6476d490 2024-04-01T01:43:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.044+0000 7f861a1df740 1 -- 172.21.15.70:0/3005115367 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f860c0bce00 con 0x55fc6476d490 2024-04-01T01:43:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.044+0000 7f85f8ff9640 1 -- 172.21.15.70:0/3005115367 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8600012a00 con 0x55fc6476d490 2024-04-01T01:43:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.044+0000 7f85f8ff9640 1 -- 172.21.15.70:0/3005115367 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8600025050 con 0x55fc6476d490 2024-04-01T01:43:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.045+0000 7f861a1df740 10 client.28340 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.045+0000 7f861a1df740 1 -- 172.21.15.70:0/3005115367 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fc644b07d0 con 0x55fc6476d490 2024-04-01T01:43:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.045+0000 7f85f8ff9640 1 -- 172.21.15.70:0/3005115367 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f860c0bce00 con 0x55fc6476d490 2024-04-01T01:43:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.045+0000 7f85f8ff9640 1 -- 172.21.15.70:0/3005115367 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f860001a020 con 0x55fc6476d490 2024-04-01T01:43:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.045+0000 7f861a1df740 10 client.28340 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.045+0000 7f861a1df740 10 client.28340 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:43:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.045+0000 7f861a1df740 20 client.28340 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.045+0000 7f861a1df740 1 --2- 172.21.15.70:0/3005115367 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55fc64783480 0x55fc64785920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.045+0000 7f861a1df740 4 client.28340 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.046+0000 7f861a1df740 1 -- 172.21.15.70:0/3005115367 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc646df6f0 con 0x55fc64783480 2024-04-01T01:43:17.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.046+0000 7f8614ba6640 1 --2- 172.21.15.70:0/3005115367 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55fc64783480 0x55fc64785920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.046+0000 7f8614ba6640 1 --2- 172.21.15.70:0/3005115367 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55fc64783480 0x55fc64785920 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=709a8828a059d0f4 server_cookie=bf2359ac571e029d in_seq=0 out_seq=0 2024-04-01T01:43:17.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.046+0000 7f85f8ff9640 10 client.28340 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:43:17.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.047+0000 7f85f8ff9640 1 -- 172.21.15.70:0/3005115367 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fc646df6f0 con 0x55fc64783480 2024-04-01T01:43:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.047+0000 7f85f8ff9640 10 client.28340 handle_command_reply: tid=0 2024-04-01T01:43:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.047+0000 7f861a1df740 1 client.28340 shutdown 2024-04-01T01:43:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.048+0000 7f861a1df740 1 -- 172.21.15.70:0/3005115367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 msgr2=0x55fc647775b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.048+0000 7f861a1df740 1 --2- 172.21.15.70:0/3005115367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 0x55fc647775b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8600009b90 tx=0x7f86000039c0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.048+0000 7f861a1df740 1 -- 172.21.15.70:0/3005115367 shutdown_connections 2024-04-01T01:43:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.048+0000 7f861a1df740 1 --2- 172.21.15.70:0/3005115367 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55fc64783480 0x55fc64785920 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.048+0000 7f861a1df740 1 --2- 172.21.15.70:0/3005115367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc64772ff0 0x55fc6477eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.048+0000 7f861a1df740 1 --2- 172.21.15.70:0/3005115367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc6476de80 0x55fc64777af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.048+0000 7f861a1df740 1 --2- 172.21.15.70:0/3005115367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 0x55fc647775b0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.048+0000 7f861a1df740 1 -- 172.21.15.70:0/3005115367 >> 172.21.15.70:0/3005115367 conn(0x55fc64768a00 msgr2=0x55fc64767d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.048+0000 7f85f8ff9640 0 client.28340 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:43:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.049+0000 7f861a1df740 1 -- 172.21.15.70:0/3005115367 shutdown_connections 2024-04-01T01:43:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.049+0000 7f861a1df740 1 -- 172.21.15.70:0/3005115367 wait complete. 2024-04-01T01:43:17.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.049+0000 7f861a1df740 20 client.28340 trim_cache size 0 max 16384 2024-04-01T01:43:17.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.075+0000 7f861a1df740 1 Processor -- start 2024-04-01T01:43:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.075+0000 7f861a1df740 1 -- start start 2024-04-01T01:43:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.075+0000 7f861a1df740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 0x55fc64776bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.075+0000 7f861a1df740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc6476de80 0x55fc6477eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.075+0000 7f861a1df740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc64772ff0 0x55fc64777110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.075+0000 7f861a1df740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f86101b9270 con 0x55fc64772ff0 2024-04-01T01:43:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.075+0000 7f861a1df740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f86101badb0 con 0x55fc6476d490 2024-04-01T01:43:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.075+0000 7f861a1df740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f86101b9940 con 0x55fc6476de80 2024-04-01T01:43:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.075+0000 7f86153a7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 0x55fc64776bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.075+0000 7f86153a7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 0x55fc64776bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49202/0 (socket says 172.21.15.70:49202) 2024-04-01T01:43:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.075+0000 7f86153a7640 1 -- 172.21.15.70:0/2346897342 learned_addr learned my addr 172.21.15.70:0/2346897342 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.076+0000 7f86153a7640 1 -- 172.21.15.70:0/2346897342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc6476de80 msgr2=0x55fc6477eaf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:43:17.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.076+0000 7f86153a7640 1 --2- 172.21.15.70:0/2346897342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc6476de80 0x55fc6477eaf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.076+0000 7f86153a7640 1 -- 172.21.15.70:0/2346897342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc64772ff0 msgr2=0x55fc64777110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.076+0000 7f86153a7640 1 --2- 172.21.15.70:0/2346897342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc64772ff0 0x55fc64777110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.076+0000 7f86153a7640 1 -- 172.21.15.70:0/2346897342 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc647725c0 con 0x55fc6476d490 2024-04-01T01:43:17.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.077+0000 7f86153a7640 1 --2- 172.21.15.70:0/2346897342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 0x55fc64776bd0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f860001ce90 tx=0x7f8600004000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=863cf4a67ed240de server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.077+0000 7f85faffd640 1 -- 172.21.15.70:0/2346897342 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8600005040 con 0x55fc6476d490 2024-04-01T01:43:17.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.077+0000 7f85faffd640 1 -- 172.21.15.70:0/2346897342 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f860002e070 con 0x55fc6476d490 2024-04-01T01:43:17.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.077+0000 7f85faffd640 1 -- 172.21.15.70:0/2346897342 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860001c440 con 0x55fc6476d490 2024-04-01T01:43:17.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.078+0000 7f861a1df740 1 -- 172.21.15.70:0/2346897342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 msgr2=0x55fc64776bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.078+0000 7f861a1df740 1 --2- 172.21.15.70:0/2346897342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 0x55fc64776bd0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f860001ce90 tx=0x7f8600004000 comp rx=0 tx=0).stop 2024-04-01T01:43:17.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.079+0000 7f861a1df740 1 -- 172.21.15.70:0/2346897342 shutdown_connections 2024-04-01T01:43:17.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.079+0000 7f861a1df740 1 --2- 172.21.15.70:0/2346897342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc64772ff0 0x55fc64777110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.079+0000 7f861a1df740 1 --2- 172.21.15.70:0/2346897342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc6476de80 0x55fc6477eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.079+0000 7f861a1df740 1 --2- 172.21.15.70:0/2346897342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476d490 0x55fc64776bd0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.079+0000 7f861a1df740 1 -- 172.21.15.70:0/2346897342 >> 172.21.15.70:0/2346897342 conn(0x55fc64768a00 msgr2=0x55fc647674b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.080+0000 7f861a1df740 1 -- 172.21.15.70:0/2346897342 shutdown_connections 2024-04-01T01:43:17.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.080+0000 7f861a1df740 1 -- 172.21.15.70:0/2346897342 wait complete. 2024-04-01T01:43:17.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.080+0000 7f861a1df740 1 -- start start 2024-04-01T01:43:17.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.080+0000 7f861a1df740 1 Processor -- start 2024-04-01T01:43:17.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f861a1df740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc6476d490 0x55fc6477c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f861a1df740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476de80 0x55fc6477c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f861a1df740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc64772ff0 0x55fc6477d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f861a1df740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f86101ba6e0 con 0x55fc64772ff0 2024-04-01T01:43:17.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f861a1df740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f86101ba010 con 0x55fc6476de80 2024-04-01T01:43:17.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f861a1df740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f86101b7250 con 0x55fc6476d490 2024-04-01T01:43:17.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f861a1df740 10 client.0 _ll_register_callbacks cb 0x7f8600004560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:17.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f8614ba6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476de80 0x55fc6477c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f8614ba6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476de80 0x55fc6477c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49216/0 (socket says 172.21.15.70:49216) 2024-04-01T01:43:17.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f8614ba6640 1 -- 172.21.15.70:0/1512953487 learned_addr learned my addr 172.21.15.70:0/1512953487 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f8615ba8640 1 --2- 172.21.15.70:0/1512953487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc64772ff0 0x55fc6477d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f86153a7640 1 --2- 172.21.15.70:0/1512953487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc6476d490 0x55fc6477c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f8614ba6640 1 -- 172.21.15.70:0/1512953487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc6476d490 msgr2=0x55fc6477c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f8614ba6640 1 --2- 172.21.15.70:0/1512953487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc6476d490 0x55fc6477c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f8614ba6640 1 -- 172.21.15.70:0/1512953487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc64772ff0 msgr2=0x55fc6477d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f8614ba6640 1 --2- 172.21.15.70:0/1512953487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc64772ff0 0x55fc6477d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.081+0000 7f8614ba6640 1 -- 172.21.15.70:0/1512953487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc6477df00 con 0x55fc6476de80 2024-04-01T01:43:17.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.082+0000 7f86153a7640 1 --2- 172.21.15.70:0/1512953487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc6476d490 0x55fc6477c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:43:17.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.082+0000 7f8614ba6640 1 --2- 172.21.15.70:0/1512953487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476de80 0x55fc6477c950 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f860c000c00 tx=0x7f860c02c1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e58d5129ab86c67 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.082+0000 7f85f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:17.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.082+0000 7f85f8ff9640 1 -- 172.21.15.70:0/1512953487 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c0d5040 con 0x55fc6476de80 2024-04-01T01:43:17.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.082+0000 7f861a1df740 1 -- 172.21.15.70:0/1512953487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f860002e070 con 0x55fc6476de80 2024-04-01T01:43:17.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.082+0000 7f85f8ff9640 1 -- 172.21.15.70:0/1512953487 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f860c0baa00 con 0x55fc6476de80 2024-04-01T01:43:17.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.082+0000 7f85f8ff9640 1 -- 172.21.15.70:0/1512953487 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f860c0bc060 con 0x55fc6476de80 2024-04-01T01:43:17.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.082+0000 7f861a1df740 10 client.28346 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:17.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.082+0000 7f861a1df740 1 -- 172.21.15.70:0/1512953487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fc647e9a40 con 0x55fc6476de80 2024-04-01T01:43:17.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.083+0000 7f85f8ff9640 1 -- 172.21.15.70:0/1512953487 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f860002e070 con 0x55fc6476de80 2024-04-01T01:43:17.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.084+0000 7f85f8ff9640 1 -- 172.21.15.70:0/1512953487 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f860c04e060 con 0x55fc6476de80 2024-04-01T01:43:17.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.084+0000 7f861a1df740 10 client.28346 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:17.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.084+0000 7f861a1df740 10 client.28346 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:43:17.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.084+0000 7f861a1df740 20 client.28346 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:17.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.084+0000 7f861a1df740 1 --2- 172.21.15.70:0/1512953487 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55fc647ea230 0x55fc647ec650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.084+0000 7f861a1df740 4 client.28346 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:43:17.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.084+0000 7f861a1df740 1 -- 172.21.15.70:0/1512953487 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fc646d9a40 con 0x55fc647ea230 2024-04-01T01:43:17.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.084+0000 7f86153a7640 1 --2- 172.21.15.70:0/1512953487 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55fc647ea230 0x55fc647ec650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.085+0000 7f86153a7640 1 --2- 172.21.15.70:0/1512953487 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55fc647ea230 0x55fc647ec650 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c901f3ff1a6f6362 server_cookie=62d1e68479a41d69 in_seq=0 out_seq=0 2024-04-01T01:43:17.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.085+0000 7f85f8ff9640 10 client.28346 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:43:17.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.086+0000 7f85f8ff9640 1 -- 172.21.15.70:0/1512953487 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fc646d9a40 con 0x55fc647ea230 2024-04-01T01:43:17.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.086+0000 7f85f8ff9640 10 client.28346 handle_command_reply: tid=0 2024-04-01T01:43:17.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.086+0000 7f861a1df740 1 client.28346 shutdown 2024-04-01T01:43:17.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.086+0000 7f861a1df740 1 -- 172.21.15.70:0/1512953487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476de80 msgr2=0x55fc6477c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.086+0000 7f861a1df740 1 --2- 172.21.15.70:0/1512953487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476de80 0x55fc6477c950 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f860c000c00 tx=0x7f860c02c1f0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.087+0000 7f861a1df740 1 -- 172.21.15.70:0/1512953487 shutdown_connections 2024-04-01T01:43:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.087+0000 7f861a1df740 1 --2- 172.21.15.70:0/1512953487 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55fc647ea230 0x55fc647ec650 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.087+0000 7f861a1df740 1 --2- 172.21.15.70:0/1512953487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fc64772ff0 0x55fc6477d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.087+0000 7f861a1df740 1 --2- 172.21.15.70:0/1512953487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fc6476de80 0x55fc6477c950 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.087+0000 7f861a1df740 1 --2- 172.21.15.70:0/1512953487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc6476d490 0x55fc6477c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.087+0000 7f861a1df740 1 -- 172.21.15.70:0/1512953487 >> 172.21.15.70:0/1512953487 conn(0x55fc64768a00 msgr2=0x55fc6476bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.087+0000 7f85f8ff9640 0 client.28346 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:43:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.087+0000 7f861a1df740 1 -- 172.21.15.70:0/1512953487 shutdown_connections 2024-04-01T01:43:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.087+0000 7f861a1df740 1 -- 172.21.15.70:0/1512953487 wait complete. 2024-04-01T01:43:17.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.088+0000 7f861a1df740 20 client.28346 trim_cache size 0 max 16384 2024-04-01T01:43:17.089 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:43:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.090+0000 7f861760b640 1 -- 172.21.15.70:0/2797502102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85fc0a25a0 msgr2=0x7f85fc0a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.090+0000 7f861760b640 1 --2- 172.21.15.70:0/2797502102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85fc0a25a0 0x7f85fc0a4a60 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x55fc6476e6e0 tx=0x7f8608004ab0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.090+0000 7f861760b640 1 -- 172.21.15.70:0/2797502102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8610102950 msgr2=0x7f8610195d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.090+0000 7f861760b640 1 --2- 172.21.15.70:0/2797502102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8610102950 0x7f8610195d50 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f860c0057c0 tx=0x7f860c0042f0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.090+0000 7f861760b640 1 -- 172.21.15.70:0/2797502102 shutdown_connections 2024-04-01T01:43:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.090+0000 7f861760b640 1 --2- 172.21.15.70:0/2797502102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8610103310 0x7f86101ae180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.090+0000 7f861760b640 1 --2- 172.21.15.70:0/2797502102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8610102950 0x7f8610195d50 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.091+0000 7f861760b640 1 --2- 172.21.15.70:0/2797502102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85fc0a25a0 0x7f85fc0a4a60 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.091+0000 7f861760b640 1 --2- 172.21.15.70:0/2797502102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8610101760 0x7f8610195810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.091+0000 7f861760b640 1 -- 172.21.15.70:0/2797502102 >> 172.21.15.70:0/2797502102 conn(0x7f86100fccd0 msgr2=0x7f8610073980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.091+0000 7f861760b640 1 -- 172.21.15.70:0/2797502102 shutdown_connections 2024-04-01T01:43:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.091+0000 7f861760b640 1 -- 172.21.15.70:0/2797502102 wait complete. 2024-04-01T01:43:17.112 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:43:17.190 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:16 smithi070 ceph-mon[32179]: pgmap v653: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.1 MiB/s wr, 762 op/s 2024-04-01T01:43:17.190 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:16 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3187291778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:43:17.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:16 smithi189 ceph-mon[35753]: pgmap v653: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.1 MiB/s wr, 762 op/s 2024-04-01T01:43:17.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:16 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3187291778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:43:17.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:16 smithi081 ceph-mon[35142]: pgmap v653: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 7.1 MiB/s wr, 762 op/s 2024-04-01T01:43:17.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:16 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3187291778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:43:17.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.337+0000 7fa55759e640 1 -- 172.21.15.70:0/1841323878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa558108e20 msgr2=0x7fa5581092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.337+0000 7fa55759e640 1 --2- 172.21.15.70:0/1841323878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa558108e20 0x7fa5581092a0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fa54c009860 tx=0x7fa54c02bb90 comp rx=0 tx=0).stop 2024-04-01T01:43:17.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.340+0000 7fa55759e640 1 -- 172.21.15.70:0/1841323878 shutdown_connections 2024-04-01T01:43:17.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.340+0000 7fa55759e640 1 --2- 172.21.15.70:0/1841323878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5581097e0 0x7fa55810e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.340+0000 7fa55759e640 1 --2- 172.21.15.70:0/1841323878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa558108e20 0x7fa5581092a0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.340+0000 7fa55759e640 1 --2- 172.21.15.70:0/1841323878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa558072b60 0x7fa558072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.340+0000 7fa55759e640 1 -- 172.21.15.70:0/1841323878 >> 172.21.15.70:0/1841323878 conn(0x7fa55806e630 msgr2=0x7fa558070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.341+0000 7fa55759e640 1 -- 172.21.15.70:0/1841323878 shutdown_connections 2024-04-01T01:43:17.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.342+0000 7fa55759e640 1 -- 172.21.15.70:0/1841323878 wait complete. 2024-04-01T01:43:17.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.343+0000 7fa55759e640 1 Processor -- start 2024-04-01T01:43:17.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.343+0000 7fa55759e640 1 -- start start 2024-04-01T01:43:17.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.343+0000 7fa55759e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa558072b60 0x7fa5581a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.343+0000 7fa55759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa558108e20 0x7fa5581a6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.343+0000 7fa55759e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5581097e0 0x7fa5581adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.343+0000 7fa55759e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa55811bf50 con 0x7fa558108e20 2024-04-01T01:43:17.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.343+0000 7fa55759e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa55811bdd0 con 0x7fa5581097e0 2024-04-01T01:43:17.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.343+0000 7fa55759e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa55811c0d0 con 0x7fa558072b60 2024-04-01T01:43:17.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.344+0000 7fa55659c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa558072b60 0x7fa5581a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.344+0000 7fa556d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5581097e0 0x7fa5581adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.344+0000 7fa55659c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa558072b60 0x7fa5581a6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46536/0 (socket says 172.21.15.70:46536) 2024-04-01T01:43:17.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.344+0000 7fa555d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa558108e20 0x7fa5581a6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.344+0000 7fa55659c640 1 -- 172.21.15.70:0/1744531546 learned_addr learned my addr 172.21.15.70:0/1744531546 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.344+0000 7fa556d9d640 1 -- 172.21.15.70:0/1744531546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa558072b60 msgr2=0x7fa5581a6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.344+0000 7fa556d9d640 1 --2- 172.21.15.70:0/1744531546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa558072b60 0x7fa5581a6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.344+0000 7fa556d9d640 1 -- 172.21.15.70:0/1744531546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa558108e20 msgr2=0x7fa5581a6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.344+0000 7fa556d9d640 1 --2- 172.21.15.70:0/1744531546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa558108e20 0x7fa5581a6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.344+0000 7fa556d9d640 1 -- 172.21.15.70:0/1744531546 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5581ae270 con 0x7fa5581097e0 2024-04-01T01:43:17.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.344+0000 7fa555d9b640 1 --2- 172.21.15.70:0/1744531546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa558108e20 0x7fa5581a6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:43:17.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.344+0000 7fa556d9d640 1 --2- 172.21.15.70:0/1744531546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5581097e0 0x7fa5581adc00 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa550009fd0 tx=0x7fa55000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.345+0000 7fa5477fe640 1 -- 172.21.15.70:0/1744531546 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55000ce20 con 0x7fa5581097e0 2024-04-01T01:43:17.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.345+0000 7fa5477fe640 1 -- 172.21.15.70:0/1744531546 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa550005020 con 0x7fa5581097e0 2024-04-01T01:43:17.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.345+0000 7fa5477fe640 1 -- 172.21.15.70:0/1744531546 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa550010690 con 0x7fa5581097e0 2024-04-01T01:43:17.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.345+0000 7fa55759e640 1 -- 172.21.15.70:0/1744531546 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5581ae560 con 0x7fa5581097e0 2024-04-01T01:43:17.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.345+0000 7fa55759e640 1 -- 172.21.15.70:0/1744531546 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5581aea30 con 0x7fa5581097e0 2024-04-01T01:43:17.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.348+0000 7fa55fa32740 1 Processor -- start 2024-04-01T01:43:17.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.349+0000 7fa5477fe640 1 -- 172.21.15.70:0/1744531546 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fa5500051d0 con 0x7fa5581097e0 2024-04-01T01:43:17.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.350+0000 7fa5477fe640 1 --2- 172.21.15.70:0/1744531546 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa5400a2520 0x7fa5400a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.350+0000 7fa5477fe640 1 -- 172.21.15.70:0/1744531546 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa5500cf1d0 con 0x7fa5581097e0 2024-04-01T01:43:17.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.351+0000 7fa55fa32740 1 -- start start 2024-04-01T01:43:17.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.351+0000 7fa555d9b640 1 --2- 172.21.15.70:0/1744531546 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa5400a2520 0x7fa5400a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.351+0000 7fa55fa32740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505cffe6a0 0x56505cffeac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.351+0000 7fa55fa32740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505cfff090 0x56505d003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.351+0000 7fa55fa32740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505d004200 0x56505d006690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.351+0000 7fa55fa32740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56505cf631b0 con 0x56505d004200 2024-04-01T01:43:17.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.351+0000 7fa55fa32740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56505cf0bc00 con 0x56505cffe6a0 2024-04-01T01:43:17.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.351+0000 7fa55fa32740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56505cf5f250 con 0x56505cfff090 2024-04-01T01:43:17.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.351+0000 7fa55659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505cffe6a0 0x56505cffeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.351+0000 7fa55659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505cffe6a0 0x56505cffeac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49244/0 (socket says 172.21.15.70:49244) 2024-04-01T01:43:17.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.351+0000 7fa55659c640 1 -- 172.21.15.70:0/4013156125 learned_addr learned my addr 172.21.15.70:0/4013156125 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.351+0000 7fa556d9d640 1 --2- 172.21.15.70:0/4013156125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505d004200 0x56505d006690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.351+0000 7fa555d9b640 1 --2- 172.21.15.70:0/1744531546 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa5400a2520 0x7fa5400a49e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa54c000c00 tx=0x7fa54c005780 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.351+0000 7fa555d9b640 1 --2- 172.21.15.70:0/4013156125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505cfff090 0x56505d003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.352+0000 7fa55659c640 1 -- 172.21.15.70:0/4013156125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505cfff090 msgr2=0x56505d003b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.352+0000 7fa55659c640 1 --2- 172.21.15.70:0/4013156125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505cfff090 0x56505d003b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.352+0000 7fa55659c640 1 -- 172.21.15.70:0/4013156125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505d004200 msgr2=0x56505d006690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.352+0000 7fa55659c640 1 --2- 172.21.15.70:0/4013156125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505d004200 0x56505d006690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.352+0000 7fa55659c640 1 -- 172.21.15.70:0/4013156125 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56505cf2b3d0 con 0x56505cffe6a0 2024-04-01T01:43:17.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.352+0000 7fa555d9b640 1 --2- 172.21.15.70:0/4013156125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505cfff090 0x56505d003b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:43:17.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.352+0000 7fa556d9d640 1 --2- 172.21.15.70:0/4013156125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505d004200 0x56505d006690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:43:17.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.352+0000 7fa55659c640 1 --2- 172.21.15.70:0/4013156125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505cffe6a0 0x56505cffeac0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa548009db0 tx=0x7fa548011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=608a1cdf14b420bb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.352+0000 7fa544ff9640 1 -- 172.21.15.70:0/4013156125 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa548004220 con 0x56505cffe6a0 2024-04-01T01:43:17.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.353+0000 7fa544ff9640 1 -- 172.21.15.70:0/4013156125 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa5480043c0 con 0x56505cffe6a0 2024-04-01T01:43:17.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.353+0000 7fa544ff9640 1 -- 172.21.15.70:0/4013156125 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa548019850 con 0x56505cffe6a0 2024-04-01T01:43:17.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.353+0000 7fa55fa32740 1 -- 172.21.15.70:0/4013156125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505cffe6a0 msgr2=0x56505cffeac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.353+0000 7fa55fa32740 1 --2- 172.21.15.70:0/4013156125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505cffe6a0 0x56505cffeac0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa548009db0 tx=0x7fa548011810 comp rx=0 tx=0).stop 2024-04-01T01:43:17.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.354+0000 7fa55fa32740 1 -- 172.21.15.70:0/4013156125 shutdown_connections 2024-04-01T01:43:17.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.354+0000 7fa55fa32740 1 --2- 172.21.15.70:0/4013156125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505d004200 0x56505d006690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.354+0000 7fa55fa32740 1 --2- 172.21.15.70:0/4013156125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505cfff090 0x56505d003b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.354+0000 7fa55fa32740 1 --2- 172.21.15.70:0/4013156125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505cffe6a0 0x56505cffeac0 secure :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa548009db0 tx=0x7fa548011810 comp rx=0 tx=0).stop 2024-04-01T01:43:17.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.354+0000 7fa55fa32740 1 -- 172.21.15.70:0/4013156125 >> 172.21.15.70:0/4013156125 conn(0x56505cff9c10 msgr2=0x56505cffa010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.355+0000 7fa55fa32740 1 -- 172.21.15.70:0/4013156125 shutdown_connections 2024-04-01T01:43:17.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.355+0000 7fa55fa32740 1 -- 172.21.15.70:0/4013156125 wait complete. 2024-04-01T01:43:17.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.356+0000 7fa55fa32740 1 -- start start 2024-04-01T01:43:17.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.356+0000 7fa55fa32740 1 Processor -- start 2024-04-01T01:43:17.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.356+0000 7fa55fa32740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505cfff090 0x56505d008d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.356+0000 7fa55fa32740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505d004200 0x56505d0092a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.356+0000 7fa55fa32740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 0x56505d010680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.356+0000 7fa55fa32740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56505d007200 con 0x56505d004200 2024-04-01T01:43:17.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.356+0000 7fa55fa32740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56505cf01470 con 0x56505d00e290 2024-04-01T01:43:17.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.356+0000 7fa55fa32740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56505d007380 con 0x56505cfff090 2024-04-01T01:43:17.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.356+0000 7fa55fa32740 10 client.0 _ll_register_callbacks cb 0x56505cf1ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:17.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.357+0000 7fa556d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 0x56505d010680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.357+0000 7fa556d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 0x56505d010680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49248/0 (socket says 172.21.15.70:49248) 2024-04-01T01:43:17.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.357+0000 7fa556d9d640 1 -- 172.21.15.70:0/3302695634 learned_addr learned my addr 172.21.15.70:0/3302695634 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.357+0000 7fa555d9b640 1 --2- 172.21.15.70:0/3302695634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505d004200 0x56505d0092a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.357+0000 7fa556d9d640 1 -- 172.21.15.70:0/3302695634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505cfff090 msgr2=0x56505d008d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.357+0000 7fa556d9d640 1 --2- 172.21.15.70:0/3302695634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505cfff090 0x56505d008d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.357+0000 7fa556d9d640 1 -- 172.21.15.70:0/3302695634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505d004200 msgr2=0x56505d0092a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.357+0000 7fa556d9d640 1 --2- 172.21.15.70:0/3302695634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505d004200 0x56505d0092a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.357+0000 7fa556d9d640 1 -- 172.21.15.70:0/3302695634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56505cc773f0 con 0x56505d00e290 2024-04-01T01:43:17.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.357+0000 7fa556d9d640 1 --2- 172.21.15.70:0/3302695634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 0x56505d010680 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa5500058c0 tx=0x7fa55009b920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f2837e7563d6a18 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.357+0000 7fa53dffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:17.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.358+0000 7fa53dffb640 1 -- 172.21.15.70:0/3302695634 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5500b3020 con 0x56505d00e290 2024-04-01T01:43:17.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.358+0000 7fa53dffb640 1 -- 172.21.15.70:0/3302695634 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa550005540 con 0x56505d00e290 2024-04-01T01:43:17.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.358+0000 7fa55fa32740 1 -- 172.21.15.70:0/3302695634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5480043c0 con 0x56505d00e290 2024-04-01T01:43:17.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.358+0000 7fa53dffb640 1 -- 172.21.15.70:0/3302695634 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa5500860b0 con 0x56505d00e290 2024-04-01T01:43:17.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.359+0000 7fa55fa32740 10 client.28358 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:17.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.359+0000 7fa55fa32740 1 -- 172.21.15.70:0/3302695634 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56505cd417d0 con 0x56505d00e290 2024-04-01T01:43:17.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.359+0000 7fa53dffb640 1 -- 172.21.15.70:0/3302695634 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa5480043c0 con 0x56505d00e290 2024-04-01T01:43:17.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.359+0000 7fa53dffb640 1 -- 172.21.15.70:0/3302695634 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fa55009cb90 con 0x56505d00e290 2024-04-01T01:43:17.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.359+0000 7fa55fa32740 10 client.28358 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:17.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.359+0000 7fa55fa32740 10 client.28358 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:43:17.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.359+0000 7fa55fa32740 20 client.28358 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:17.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.359+0000 7fa55fa32740 1 --2- 172.21.15.70:0/3302695634 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56505d025cb0 0x56505d028120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.359+0000 7fa55fa32740 4 client.28358 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:17.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.359+0000 7fa55fa32740 1 -- 172.21.15.70:0/3302695634 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56505cf706f0 con 0x56505d025cb0 2024-04-01T01:43:17.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.360+0000 7fa55659c640 1 --2- 172.21.15.70:0/3302695634 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56505d025cb0 0x56505d028120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.361+0000 7fa55659c640 1 --2- 172.21.15.70:0/3302695634 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56505d025cb0 0x56505d028120 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9d0df3c189c6e713 server_cookie=e39b3a7cb9d05917 in_seq=0 out_seq=0 2024-04-01T01:43:17.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.361+0000 7fa53dffb640 10 client.28358 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:43:17.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.363+0000 7fa53dffb640 1 -- 172.21.15.70:0/3302695634 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56505cf706f0 con 0x56505d025cb0 2024-04-01T01:43:17.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.363+0000 7fa53dffb640 10 client.28358 handle_command_reply: tid=0 2024-04-01T01:43:17.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.363+0000 7fa55fa32740 1 client.28358 shutdown 2024-04-01T01:43:17.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.364+0000 7fa55fa32740 1 -- 172.21.15.70:0/3302695634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 msgr2=0x56505d010680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.364+0000 7fa55fa32740 1 --2- 172.21.15.70:0/3302695634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 0x56505d010680 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa5500058c0 tx=0x7fa55009b920 comp rx=0 tx=0).stop 2024-04-01T01:43:17.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.365+0000 7fa55fa32740 1 -- 172.21.15.70:0/3302695634 shutdown_connections 2024-04-01T01:43:17.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.365+0000 7fa55fa32740 1 --2- 172.21.15.70:0/3302695634 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56505d025cb0 0x56505d028120 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.365+0000 7fa55fa32740 1 --2- 172.21.15.70:0/3302695634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 0x56505d010680 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.365+0000 7fa55fa32740 1 --2- 172.21.15.70:0/3302695634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505d004200 0x56505d0092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.365+0000 7fa55fa32740 1 --2- 172.21.15.70:0/3302695634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505cfff090 0x56505d008d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.365+0000 7fa55fa32740 1 -- 172.21.15.70:0/3302695634 >> 172.21.15.70:0/3302695634 conn(0x56505cff9c10 msgr2=0x56505cff8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.366+0000 7fa53dffb640 0 client.28358 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:43:17.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.367+0000 7fa55fa32740 1 -- 172.21.15.70:0/3302695634 shutdown_connections 2024-04-01T01:43:17.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.367+0000 7fa55fa32740 1 -- 172.21.15.70:0/3302695634 wait complete. 2024-04-01T01:43:17.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.367+0000 7fa55fa32740 20 client.28358 trim_cache size 0 max 16384 2024-04-01T01:43:17.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.394+0000 7fa55fa32740 1 Processor -- start 2024-04-01T01:43:17.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.394+0000 7fa55fa32740 1 -- start start 2024-04-01T01:43:17.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.394+0000 7fa55fa32740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505cfff090 0x56505cff8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.395+0000 7fa55fa32740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505d004200 0x56505cff8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.395+0000 7fa55fa32740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 0x56505cff9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.395+0000 7fa555d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505d004200 0x56505cff8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.395+0000 7fa555d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505d004200 0x56505cff8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46576/0 (socket says 172.21.15.70:46576) 2024-04-01T01:43:17.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.395+0000 7fa555d9b640 1 -- 172.21.15.70:0/706610155 learned_addr learned my addr 172.21.15.70:0/706610155 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.395+0000 7fa55fa32740 1 -- 172.21.15.70:0/706610155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581b08e0 con 0x56505cfff090 2024-04-01T01:43:17.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.395+0000 7fa556d9d640 1 --2- 172.21.15.70:0/706610155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 0x56505cff9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.395+0000 7fa55fa32740 1 -- 172.21.15.70:0/706610155 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581b2420 con 0x56505d00e290 2024-04-01T01:43:17.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.395+0000 7fa55fa32740 1 -- 172.21.15.70:0/706610155 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581b0fb0 con 0x56505d004200 2024-04-01T01:43:17.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.396+0000 7fa55659c640 1 --2- 172.21.15.70:0/706610155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505cfff090 0x56505cff8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.396+0000 7fa556d9d640 1 -- 172.21.15.70:0/706610155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505d004200 msgr2=0x56505cff8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.396+0000 7fa556d9d640 1 --2- 172.21.15.70:0/706610155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505d004200 0x56505cff8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.396+0000 7fa556d9d640 1 -- 172.21.15.70:0/706610155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505cfff090 msgr2=0x56505cff8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.396+0000 7fa556d9d640 1 --2- 172.21.15.70:0/706610155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505cfff090 0x56505cff8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.396+0000 7fa556d9d640 1 -- 172.21.15.70:0/706610155 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56505cff5670 con 0x56505d00e290 2024-04-01T01:43:17.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.397+0000 7fa556d9d640 1 --2- 172.21.15.70:0/706610155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 0x56505cff9180 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa5500afb70 tx=0x7fa550099d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca834198b4855fcc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.398+0000 7fa53ffff640 1 -- 172.21.15.70:0/706610155 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5500b3040 con 0x56505d00e290 2024-04-01T01:43:17.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.399+0000 7fa53ffff640 1 -- 172.21.15.70:0/706610155 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa550005540 con 0x56505d00e290 2024-04-01T01:43:17.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.400+0000 7fa55fa32740 1 -- 172.21.15.70:0/706610155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 msgr2=0x56505cff9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.400+0000 7fa55fa32740 1 --2- 172.21.15.70:0/706610155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 0x56505cff9180 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa5500afb70 tx=0x7fa550099d70 comp rx=0 tx=0).stop 2024-04-01T01:43:17.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.400+0000 7fa55fa32740 1 -- 172.21.15.70:0/706610155 shutdown_connections 2024-04-01T01:43:17.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.400+0000 7fa55fa32740 1 --2- 172.21.15.70:0/706610155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 0x56505cff9180 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.400+0000 7fa55fa32740 1 --2- 172.21.15.70:0/706610155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505d004200 0x56505cff8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.400+0000 7fa55fa32740 1 --2- 172.21.15.70:0/706610155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505cfff090 0x56505cff8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.400+0000 7fa55fa32740 1 -- 172.21.15.70:0/706610155 >> 172.21.15.70:0/706610155 conn(0x56505cff9c10 msgr2=0x56505d0006a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.402+0000 7fa55fa32740 1 -- 172.21.15.70:0/706610155 shutdown_connections 2024-04-01T01:43:17.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.403+0000 7fa55fa32740 1 -- 172.21.15.70:0/706610155 wait complete. 2024-04-01T01:43:17.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.403+0000 7fa55fa32740 1 -- start start 2024-04-01T01:43:17.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.403+0000 7fa55fa32740 1 Processor -- start 2024-04-01T01:43:17.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa55fa32740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505cfff090 0x56505d0370b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa55fa32740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505d004200 0x56505d0375f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa55fa32740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 0x56505d038600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa55fa32740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581b1d50 con 0x56505d004200 2024-04-01T01:43:17.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa55fa32740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581b1680 con 0x56505d00e290 2024-04-01T01:43:17.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa55fa32740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa5581181f0 con 0x56505cfff090 2024-04-01T01:43:17.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa55fa32740 10 client.0 _ll_register_callbacks cb 0x7fa55009bf60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:17.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa556d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 0x56505d038600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa55659c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505cfff090 0x56505d0370b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa556d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 0x56505d038600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49270/0 (socket says 172.21.15.70:49270) 2024-04-01T01:43:17.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa556d9d640 1 -- 172.21.15.70:0/21756737 learned_addr learned my addr 172.21.15.70:0/21756737 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa555d9b640 1 --2- 172.21.15.70:0/21756737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505d004200 0x56505d0375f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa55659c640 1 -- 172.21.15.70:0/21756737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 msgr2=0x56505d038600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa55659c640 1 --2- 172.21.15.70:0/21756737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 0x56505d038600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa55659c640 1 -- 172.21.15.70:0/21756737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505d004200 msgr2=0x56505d0375f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa55659c640 1 --2- 172.21.15.70:0/21756737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505d004200 0x56505d0375f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.404+0000 7fa55659c640 1 -- 172.21.15.70:0/21756737 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56505d038bf0 con 0x56505cfff090 2024-04-01T01:43:17.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.405+0000 7fa55659c640 1 --2- 172.21.15.70:0/21756737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505cfff090 0x56505d0370b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x56505cffabb0 tx=0x7fa548009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d2ec5e97ecc63a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.405+0000 7fa53dffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:43:17.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.405+0000 7fa53dffb640 1 -- 172.21.15.70:0/21756737 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa548005450 con 0x56505cfff090 2024-04-01T01:43:17.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.405+0000 7fa55fa32740 1 -- 172.21.15.70:0/21756737 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56505cf6aa40 con 0x56505cfff090 2024-04-01T01:43:17.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.405+0000 7fa53dffb640 1 -- 172.21.15.70:0/21756737 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa5480055f0 con 0x56505cfff090 2024-04-01T01:43:17.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.405+0000 7fa53dffb640 1 -- 172.21.15.70:0/21756737 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa54803b050 con 0x56505cfff090 2024-04-01T01:43:17.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.408+0000 7fa55fa32740 10 client.28168 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:17.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.408+0000 7fa55fa32740 1 -- 172.21.15.70:0/21756737 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56505d08c260 con 0x56505cfff090 2024-04-01T01:43:17.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.409+0000 7fa53dffb640 1 -- 172.21.15.70:0/21756737 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56505cf6aa40 con 0x56505cfff090 2024-04-01T01:43:17.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.409+0000 7fa53dffb640 1 -- 172.21.15.70:0/21756737 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fa5480038e0 con 0x56505cfff090 2024-04-01T01:43:17.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.409+0000 7fa55fa32740 10 client.28168 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:17.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.409+0000 7fa55fa32740 10 client.28168 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:43:17.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.409+0000 7fa55fa32740 20 client.28168 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:17.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.409+0000 7fa55fa32740 1 --2- 172.21.15.70:0/21756737 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56505d08ca40 0x56505d08ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.409+0000 7fa55fa32740 4 client.28168 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:43:17.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.409+0000 7fa55fa32740 1 -- 172.21.15.70:0/21756737 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56505cee86a0 con 0x56505d08ca40 2024-04-01T01:43:17.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.409+0000 7fa555d9b640 1 --2- 172.21.15.70:0/21756737 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56505d08ca40 0x56505d08ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.410+0000 7fa555d9b640 1 --2- 172.21.15.70:0/21756737 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56505d08ca40 0x56505d08ee60 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2e5a1ce482d5fa65 server_cookie=78ccf7594dd41087 in_seq=0 out_seq=0 2024-04-01T01:43:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.410+0000 7fa53dffb640 10 client.28168 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:43:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.411+0000 7fa53dffb640 1 -- 172.21.15.70:0/21756737 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56505cee86a0 con 0x56505d08ca40 2024-04-01T01:43:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.411+0000 7fa53dffb640 10 client.28168 handle_command_reply: tid=0 2024-04-01T01:43:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.411+0000 7fa55fa32740 1 client.28168 shutdown 2024-04-01T01:43:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.414+0000 7fa55fa32740 1 -- 172.21.15.70:0/21756737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505cfff090 msgr2=0x56505d0370b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.414+0000 7fa55fa32740 1 --2- 172.21.15.70:0/21756737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505cfff090 0x56505d0370b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x56505cffabb0 tx=0x7fa548009cb0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.414+0000 7fa55fa32740 1 -- 172.21.15.70:0/21756737 shutdown_connections 2024-04-01T01:43:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.414+0000 7fa55fa32740 1 --2- 172.21.15.70:0/21756737 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x56505d08ca40 0x56505d08ee60 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.414+0000 7fa55fa32740 1 --2- 172.21.15.70:0/21756737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56505d00e290 0x56505d038600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.414+0000 7fa55fa32740 1 --2- 172.21.15.70:0/21756737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56505d004200 0x56505d0375f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.414+0000 7fa55fa32740 1 --2- 172.21.15.70:0/21756737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56505cfff090 0x56505d0370b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.414+0000 7fa55fa32740 1 -- 172.21.15.70:0/21756737 >> 172.21.15.70:0/21756737 conn(0x56505cff9c10 msgr2=0x56505cff8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.416+0000 7fa53dffb640 0 client.28168 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:43:17.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.416+0000 7fa55fa32740 1 -- 172.21.15.70:0/21756737 shutdown_connections 2024-04-01T01:43:17.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.416+0000 7fa55fa32740 1 -- 172.21.15.70:0/21756737 wait complete. 2024-04-01T01:43:17.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.416+0000 7fa55fa32740 20 client.28168 trim_cache size 0 max 16384 2024-04-01T01:43:17.417 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:43:17.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.419+0000 7fa55759e640 1 -- 172.21.15.70:0/1744531546 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa5400a2520 msgr2=0x7fa5400a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.419+0000 7fa55759e640 1 --2- 172.21.15.70:0/1744531546 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa5400a2520 0x7fa5400a49e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa54c000c00 tx=0x7fa54c005780 comp rx=0 tx=0).stop 2024-04-01T01:43:17.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.419+0000 7fa55759e640 1 -- 172.21.15.70:0/1744531546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5581097e0 msgr2=0x7fa5581adc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.419+0000 7fa55759e640 1 --2- 172.21.15.70:0/1744531546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5581097e0 0x7fa5581adc00 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa550009fd0 tx=0x7fa55000ef10 comp rx=0 tx=0).stop 2024-04-01T01:43:17.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.422+0000 7fa55759e640 1 -- 172.21.15.70:0/1744531546 shutdown_connections 2024-04-01T01:43:17.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.423+0000 7fa55759e640 1 --2- 172.21.15.70:0/1744531546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5581097e0 0x7fa5581adc00 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.423+0000 7fa55759e640 1 --2- 172.21.15.70:0/1744531546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa558108e20 0x7fa5581a6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.423+0000 7fa55759e640 1 --2- 172.21.15.70:0/1744531546 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa5400a2520 0x7fa5400a49e0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.423+0000 7fa55759e640 1 --2- 172.21.15.70:0/1744531546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa558072b60 0x7fa5581a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.423+0000 7fa55759e640 1 -- 172.21.15.70:0/1744531546 >> 172.21.15.70:0/1744531546 conn(0x7fa55806e630 msgr2=0x7fa5580702e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.424+0000 7fa55759e640 1 -- 172.21.15.70:0/1744531546 shutdown_connections 2024-04-01T01:43:17.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.424+0000 7fa55759e640 1 -- 172.21.15.70:0/1744531546 wait complete. 2024-04-01T01:43:17.447 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:43:17.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 -- 172.21.15.70:0/3642115761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb744102210 msgr2=0x7fb744108aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 --2- 172.21.15.70:0/3642115761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb744102210 0x7fb744108aa0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009ca0 tx=0x7fb73c02c0f0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 -- 172.21.15.70:0/3642115761 shutdown_connections 2024-04-01T01:43:17.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 --2- 172.21.15.70:0/3642115761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb744102210 0x7fb744108aa0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 --2- 172.21.15.70:0/3642115761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb744101850 0x7fb744101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 --2- 172.21.15.70:0/3642115761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb744100660 0x7fb744100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 -- 172.21.15.70:0/3642115761 >> 172.21.15.70:0/3642115761 conn(0x7fb7440fbbf0 msgr2=0x7fb7440fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 -- 172.21.15.70:0/3642115761 shutdown_connections 2024-04-01T01:43:17.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 -- 172.21.15.70:0/3642115761 wait complete. 2024-04-01T01:43:17.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 Processor -- start 2024-04-01T01:43:17.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 -- start start 2024-04-01T01:43:17.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb744100660 0x7fb744196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb744101850 0x7fb744196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb744102210 0x7fb74419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb74410c190 con 0x7fb744101850 2024-04-01T01:43:17.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb74410c010 con 0x7fb744102210 2024-04-01T01:43:17.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.642+0000 7fb74c1a1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb74410c310 con 0x7fb744100660 2024-04-01T01:43:17.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.643+0000 7fb74973c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb744101850 0x7fb744196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.643+0000 7fb74973c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb744101850 0x7fb744196db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34198/0 (socket says 172.21.15.70:34198) 2024-04-01T01:43:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.643+0000 7fb74973c640 1 -- 172.21.15.70:0/2136208870 learned_addr learned my addr 172.21.15.70:0/2136208870 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.643+0000 7fb749f3d640 1 --2- 172.21.15.70:0/2136208870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb744100660 0x7fb744196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.643+0000 7fb74a73e640 1 --2- 172.21.15.70:0/2136208870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb744102210 0x7fb74419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.643+0000 7fb74973c640 1 -- 172.21.15.70:0/2136208870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb744100660 msgr2=0x7fb744196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.643+0000 7fb74973c640 1 --2- 172.21.15.70:0/2136208870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb744100660 0x7fb744196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.643+0000 7fb74973c640 1 -- 172.21.15.70:0/2136208870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb744102210 msgr2=0x7fb74419de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.643+0000 7fb74973c640 1 --2- 172.21.15.70:0/2136208870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb744102210 0x7fb74419de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.643+0000 7fb74973c640 1 -- 172.21.15.70:0/2136208870 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74419e490 con 0x7fb744101850 2024-04-01T01:43:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.643+0000 7fb74973c640 1 --2- 172.21.15.70:0/2136208870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb744101850 0x7fb744196db0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb73400eb00 tx=0x7fb734009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.643+0000 7fb73affd640 1 -- 172.21.15.70:0/2136208870 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73401c070 con 0x7fb744101850 2024-04-01T01:43:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.643+0000 7fb73affd640 1 -- 172.21.15.70:0/2136208870 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb73400bc30 con 0x7fb744101850 2024-04-01T01:43:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.643+0000 7fb74c1a1640 1 -- 172.21.15.70:0/2136208870 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb744069da0 con 0x7fb744101850 2024-04-01T01:43:17.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.643+0000 7fb73affd640 1 -- 172.21.15.70:0/2136208870 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb734016610 con 0x7fb744101850 2024-04-01T01:43:17.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.643+0000 7fb74c1a1640 1 -- 172.21.15.70:0/2136208870 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb74406a340 con 0x7fb744101850 2024-04-01T01:43:17.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.646+0000 7fb74ed88740 1 Processor -- start 2024-04-01T01:43:17.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.646+0000 7fb73affd640 1 -- 172.21.15.70:0/2136208870 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb73401f070 con 0x7fb744101850 2024-04-01T01:43:17.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.646+0000 7fb74ed88740 1 -- start start 2024-04-01T01:43:17.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.646+0000 7fb74ed88740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634ffefa8a0 0x5634ffefacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.646+0000 7fb74ed88740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefb290 0x5634ffeffd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.646+0000 7fb74ed88740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634fff00400 0x5634fff02890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.646+0000 7fb74ed88740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5634ffe5b1b0 con 0x5634fff00400 2024-04-01T01:43:17.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.646+0000 7fb74ed88740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5634ffe03c00 con 0x5634ffefb290 2024-04-01T01:43:17.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.646+0000 7fb74ed88740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5634ffe57250 con 0x5634ffefa8a0 2024-04-01T01:43:17.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.646+0000 7fb74973c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634fff00400 0x5634fff02890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.646+0000 7fb74a73e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634ffefa8a0 0x5634ffefacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.646+0000 7fb74a73e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634ffefa8a0 0x5634ffefacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46622/0 (socket says 172.21.15.70:46622) 2024-04-01T01:43:17.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.646+0000 7fb74a73e640 1 -- 172.21.15.70:0/2776668662 learned_addr learned my addr 172.21.15.70:0/2776668662 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.646+0000 7fb749f3d640 1 --2- 172.21.15.70:0/2776668662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefb290 0x5634ffeffd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.647+0000 7fb73affd640 1 --2- 172.21.15.70:0/2136208870 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7300a2520 0x7fb7300a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.647+0000 7fb74a73e640 1 -- 172.21.15.70:0/2776668662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefb290 msgr2=0x5634ffeffd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.647+0000 7fb74a73e640 1 --2- 172.21.15.70:0/2776668662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefb290 0x5634ffeffd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.647+0000 7fb74a73e640 1 -- 172.21.15.70:0/2776668662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634fff00400 msgr2=0x5634fff02890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.647+0000 7fb74a73e640 1 --2- 172.21.15.70:0/2776668662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634fff00400 0x5634fff02890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.647+0000 7fb74a73e640 1 -- 172.21.15.70:0/2776668662 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634ffe233d0 con 0x5634ffefa8a0 2024-04-01T01:43:17.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.647+0000 7fb749f3d640 1 --2- 172.21.15.70:0/2776668662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefb290 0x5634ffeffd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:43:17.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.647+0000 7fb74973c640 1 --2- 172.21.15.70:0/2776668662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634fff00400 0x5634fff02890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:43:17.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.647+0000 7fb73affd640 1 -- 172.21.15.70:0/2136208870 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb7340cfff0 con 0x7fb744101850 2024-04-01T01:43:17.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.647+0000 7fb74a73e640 1 --2- 172.21.15.70:0/2136208870 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7300a2520 0x7fb7300a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.647+0000 7fb74a73e640 1 --2- 172.21.15.70:0/2776668662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634ffefa8a0 0x5634ffefacc0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb73c005e30 tx=0x7fb73c009880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fd8897582f08706 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.647+0000 7fb72ffff640 1 -- 172.21.15.70:0/2776668662 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c039040 con 0x5634ffefa8a0 2024-04-01T01:43:17.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.647+0000 7fb72ffff640 1 -- 172.21.15.70:0/2776668662 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb73c003cb0 con 0x5634ffefa8a0 2024-04-01T01:43:17.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.647+0000 7fb72ffff640 1 -- 172.21.15.70:0/2776668662 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c034680 con 0x5634ffefa8a0 2024-04-01T01:43:17.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.647+0000 7fb74a73e640 1 --2- 172.21.15.70:0/2136208870 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7300a2520 0x7fb7300a49e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb73c04c000 tx=0x7fb73c045000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.648+0000 7fb74ed88740 1 -- 172.21.15.70:0/2776668662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634ffefa8a0 msgr2=0x5634ffefacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.648+0000 7fb74ed88740 1 --2- 172.21.15.70:0/2776668662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634ffefa8a0 0x5634ffefacc0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb73c005e30 tx=0x7fb73c009880 comp rx=0 tx=0).stop 2024-04-01T01:43:17.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.648+0000 7fb74ed88740 1 -- 172.21.15.70:0/2776668662 shutdown_connections 2024-04-01T01:43:17.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.648+0000 7fb74ed88740 1 --2- 172.21.15.70:0/2776668662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634fff00400 0x5634fff02890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.648+0000 7fb74ed88740 1 --2- 172.21.15.70:0/2776668662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefb290 0x5634ffeffd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.648+0000 7fb74ed88740 1 --2- 172.21.15.70:0/2776668662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634ffefa8a0 0x5634ffefacc0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.648+0000 7fb74ed88740 1 -- 172.21.15.70:0/2776668662 >> 172.21.15.70:0/2776668662 conn(0x5634ffef1a00 msgr2=0x5634ffef1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.648+0000 7fb74ed88740 1 -- 172.21.15.70:0/2776668662 shutdown_connections 2024-04-01T01:43:17.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.648+0000 7fb74ed88740 1 -- 172.21.15.70:0/2776668662 wait complete. 2024-04-01T01:43:17.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.648+0000 7fb74ed88740 1 -- start start 2024-04-01T01:43:17.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.648+0000 7fb74ed88740 1 Processor -- start 2024-04-01T01:43:17.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb74ed88740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634ffefa8a0 0x5634fff04df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb74ed88740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefb290 0x5634fff05330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb74ed88740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634fff00400 0x5634fff0c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb74ed88740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5634fff033b0 con 0x5634ffefa8a0 2024-04-01T01:43:17.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb74ed88740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5634ffdf9470 con 0x5634ffefb290 2024-04-01T01:43:17.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb74ed88740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5634fff03530 con 0x5634fff00400 2024-04-01T01:43:17.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb74ed88740 10 client.0 _ll_register_callbacks cb 0x5634ffe16cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:17.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb749f3d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634ffefa8a0 0x5634fff04df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb74973c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefb290 0x5634fff05330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb749f3d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634ffefa8a0 0x5634fff04df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34208/0 (socket says 172.21.15.70:34208) 2024-04-01T01:43:17.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb749f3d640 1 -- 172.21.15.70:0/3533216152 learned_addr learned my addr 172.21.15.70:0/3533216152 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb74a73e640 1 --2- 172.21.15.70:0/3533216152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634fff00400 0x5634fff0c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb749f3d640 1 -- 172.21.15.70:0/3533216152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634fff00400 msgr2=0x5634fff0c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb749f3d640 1 --2- 172.21.15.70:0/3533216152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634fff00400 0x5634fff0c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb749f3d640 1 -- 172.21.15.70:0/3533216152 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefb290 msgr2=0x5634fff05330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb749f3d640 1 --2- 172.21.15.70:0/3533216152 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefb290 0x5634fff05330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb749f3d640 1 -- 172.21.15.70:0/3533216152 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634ffb6f3f0 con 0x5634ffefa8a0 2024-04-01T01:43:17.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb74973c640 1 --2- 172.21.15.70:0/3533216152 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefb290 0x5634fff05330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:43:17.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb749f3d640 1 --2- 172.21.15.70:0/3533216152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634ffefa8a0 0x5634fff04df0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fb740012700 tx=0x7fb740012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946b51c6d1e0dc4f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb72d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:43:17.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb72d7fa640 1 -- 172.21.15.70:0/3533216152 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb740004030 con 0x5634ffefa8a0 2024-04-01T01:43:17.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb72d7fa640 1 -- 172.21.15.70:0/3533216152 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb7400041d0 con 0x5634ffefa8a0 2024-04-01T01:43:17.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.649+0000 7fb74ed88740 1 -- 172.21.15.70:0/3533216152 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb73c003cb0 con 0x5634ffefa8a0 2024-04-01T01:43:17.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.650+0000 7fb72d7fa640 1 -- 172.21.15.70:0/3533216152 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb7400043d0 con 0x5634ffefa8a0 2024-04-01T01:43:17.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.650+0000 7fb74ed88740 10 client.18372 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:17.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.650+0000 7fb74ed88740 1 -- 172.21.15.70:0/3533216152 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5634ffc397d0 con 0x5634ffefa8a0 2024-04-01T01:43:17.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.650+0000 7fb72d7fa640 1 -- 172.21.15.70:0/3533216152 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb73c003cb0 con 0x5634ffefa8a0 2024-04-01T01:43:17.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.650+0000 7fb72d7fa640 1 -- 172.21.15.70:0/3533216152 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb740003a00 con 0x5634ffefa8a0 2024-04-01T01:43:17.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.650+0000 7fb74ed88740 10 client.18372 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:17.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.650+0000 7fb74ed88740 10 client.18372 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:43:17.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.650+0000 7fb74ed88740 20 client.18372 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:17.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.650+0000 7fb74ed88740 1 --2- 172.21.15.70:0/3533216152 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5634fff109a0 0x5634fff12e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.650+0000 7fb74ed88740 4 client.18372 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:17.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.650+0000 7fb74ed88740 1 -- 172.21.15.70:0/3533216152 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634ffe686f0 con 0x5634fff109a0 2024-04-01T01:43:17.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.650+0000 7fb74973c640 1 --2- 172.21.15.70:0/3533216152 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5634fff109a0 0x5634fff12e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.651+0000 7fb74973c640 1 --2- 172.21.15.70:0/3533216152 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5634fff109a0 0x5634fff12e20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3c3bd3e086b7e17a server_cookie=55e4ad9bd8f261cc in_seq=0 out_seq=0 2024-04-01T01:43:17.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.651+0000 7fb72d7fa640 10 client.18372 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:43:17.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.651+0000 7fb72d7fa640 1 -- 172.21.15.70:0/3533216152 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5634ffe686f0 con 0x5634fff109a0 2024-04-01T01:43:17.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.651+0000 7fb72d7fa640 10 client.18372 handle_command_reply: tid=0 2024-04-01T01:43:17.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.651+0000 7fb74ed88740 1 client.18372 shutdown 2024-04-01T01:43:17.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.651+0000 7fb74ed88740 1 -- 172.21.15.70:0/3533216152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634ffefa8a0 msgr2=0x5634fff04df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.651+0000 7fb74ed88740 1 --2- 172.21.15.70:0/3533216152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634ffefa8a0 0x5634fff04df0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fb740012700 tx=0x7fb740012bc0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.651+0000 7fb74ed88740 1 -- 172.21.15.70:0/3533216152 shutdown_connections 2024-04-01T01:43:17.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.651+0000 7fb74ed88740 1 --2- 172.21.15.70:0/3533216152 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5634fff109a0 0x5634fff12e20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.651+0000 7fb74ed88740 1 --2- 172.21.15.70:0/3533216152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634fff00400 0x5634fff0c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.651+0000 7fb74ed88740 1 --2- 172.21.15.70:0/3533216152 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefb290 0x5634fff05330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.651+0000 7fb74ed88740 1 --2- 172.21.15.70:0/3533216152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634ffefa8a0 0x5634fff04df0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.651+0000 7fb72d7fa640 0 client.18372 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:43:17.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.651+0000 7fb74ed88740 1 -- 172.21.15.70:0/3533216152 >> 172.21.15.70:0/3533216152 conn(0x5634ffef1a00 msgr2=0x5634ffef0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.651+0000 7fb74ed88740 1 -- 172.21.15.70:0/3533216152 shutdown_connections 2024-04-01T01:43:17.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.652+0000 7fb74ed88740 1 -- 172.21.15.70:0/3533216152 wait complete. 2024-04-01T01:43:17.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.652+0000 7fb74ed88740 20 client.18372 trim_cache size 0 max 16384 2024-04-01T01:43:17.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74ed88740 1 Processor -- start 2024-04-01T01:43:17.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74ed88740 1 -- start start 2024-04-01T01:43:17.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74ed88740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefa8a0 0x5634fff176e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74ed88740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634ffefb290 0x5634fff0c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74ed88740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634fff00400 0x5634fff17c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74ed88740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb7441a81a0 con 0x5634fff00400 2024-04-01T01:43:17.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74ed88740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb7441a9ce0 con 0x5634ffefa8a0 2024-04-01T01:43:17.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74ed88740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb7441a8870 con 0x5634ffefb290 2024-04-01T01:43:17.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74a73e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634fff00400 0x5634fff17c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74a73e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634fff00400 0x5634fff17c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34218/0 (socket says 172.21.15.70:34218) 2024-04-01T01:43:17.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74a73e640 1 -- 172.21.15.70:0/4043267739 learned_addr learned my addr 172.21.15.70:0/4043267739 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74973c640 1 --2- 172.21.15.70:0/4043267739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634ffefb290 0x5634fff0c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb749f3d640 1 --2- 172.21.15.70:0/4043267739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefa8a0 0x5634fff176e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74a73e640 1 -- 172.21.15.70:0/4043267739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634ffefb290 msgr2=0x5634fff0c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74a73e640 1 --2- 172.21.15.70:0/4043267739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634ffefb290 0x5634fff0c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74a73e640 1 -- 172.21.15.70:0/4043267739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefa8a0 msgr2=0x5634fff176e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74a73e640 1 --2- 172.21.15.70:0/4043267739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefa8a0 0x5634fff176e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74a73e640 1 -- 172.21.15.70:0/4043267739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634ffefb670 con 0x5634fff00400 2024-04-01T01:43:17.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb74a73e640 1 --2- 172.21.15.70:0/4043267739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634fff00400 0x5634fff17c20 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009ca0 tx=0x7fb73c0345d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e742af53790227fd server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb72f7fe640 1 -- 172.21.15.70:0/4043267739 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c039070 con 0x5634fff00400 2024-04-01T01:43:17.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb72f7fe640 1 -- 172.21.15.70:0/4043267739 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb73c05c070 con 0x5634fff00400 2024-04-01T01:43:17.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.665+0000 7fb72f7fe640 1 -- 172.21.15.70:0/4043267739 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c004040 con 0x5634fff00400 2024-04-01T01:43:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.666+0000 7fb74ed88740 1 -- 172.21.15.70:0/4043267739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634fff00400 msgr2=0x5634fff17c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.666+0000 7fb74ed88740 1 --2- 172.21.15.70:0/4043267739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634fff00400 0x5634fff17c20 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009ca0 tx=0x7fb73c0345d0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.666+0000 7fb74ed88740 1 -- 172.21.15.70:0/4043267739 shutdown_connections 2024-04-01T01:43:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.666+0000 7fb74ed88740 1 --2- 172.21.15.70:0/4043267739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634fff00400 0x5634fff17c20 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.666+0000 7fb74ed88740 1 --2- 172.21.15.70:0/4043267739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634ffefb290 0x5634fff0c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.666+0000 7fb74ed88740 1 --2- 172.21.15.70:0/4043267739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefa8a0 0x5634fff176e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.666+0000 7fb74ed88740 1 -- 172.21.15.70:0/4043267739 >> 172.21.15.70:0/4043267739 conn(0x5634ffef1a00 msgr2=0x5634ffef54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.666+0000 7fb74ed88740 1 -- 172.21.15.70:0/4043267739 shutdown_connections 2024-04-01T01:43:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.666+0000 7fb74ed88740 1 -- 172.21.15.70:0/4043267739 wait complete. 2024-04-01T01:43:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.666+0000 7fb74ed88740 1 -- start start 2024-04-01T01:43:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.666+0000 7fb74ed88740 1 Processor -- start 2024-04-01T01:43:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74ed88740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefa8a0 0x5634fff09a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74ed88740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634ffefb290 0x5634fff09fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74ed88740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634fff00400 0x5634fff0afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74ed88740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb7441a9610 con 0x5634ffefb290 2024-04-01T01:43:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74ed88740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb7441a8f40 con 0x5634ffefa8a0 2024-04-01T01:43:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74ed88740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb7441a6180 con 0x5634fff00400 2024-04-01T01:43:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74ed88740 10 client.0 _ll_register_callbacks cb 0x7fb740009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74973c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634ffefb290 0x5634fff09fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb749f3d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefa8a0 0x5634fff09a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74973c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634ffefb290 0x5634fff09fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34224/0 (socket says 172.21.15.70:34224) 2024-04-01T01:43:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74973c640 1 -- 172.21.15.70:0/1343159172 learned_addr learned my addr 172.21.15.70:0/1343159172 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74a73e640 1 --2- 172.21.15.70:0/1343159172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634fff00400 0x5634fff0afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74973c640 1 -- 172.21.15.70:0/1343159172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634fff00400 msgr2=0x5634fff0afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74973c640 1 --2- 172.21.15.70:0/1343159172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634fff00400 0x5634fff0afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74973c640 1 -- 172.21.15.70:0/1343159172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefa8a0 msgr2=0x5634fff09a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74973c640 1 --2- 172.21.15.70:0/1343159172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefa8a0 0x5634fff09a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74973c640 1 -- 172.21.15.70:0/1343159172 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634fff0b5a0 con 0x5634ffefb290 2024-04-01T01:43:17.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb749f3d640 1 --2- 172.21.15.70:0/1343159172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefa8a0 0x5634fff09a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:43:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74973c640 1 --2- 172.21.15.70:0/1343159172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634ffefb290 0x5634fff09fa0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb73400ead0 tx=0x7fb734017f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4885fcaf870a2a38 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb72d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:43:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb72d7fa640 1 -- 172.21.15.70:0/1343159172 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73400bde0 con 0x5634ffefb290 2024-04-01T01:43:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb72d7fa640 1 -- 172.21.15.70:0/1343159172 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb7340870b0 con 0x5634ffefb290 2024-04-01T01:43:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74ed88740 1 -- 172.21.15.70:0/1343159172 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb73c05c070 con 0x5634ffefb290 2024-04-01T01:43:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb72d7fa640 1 -- 172.21.15.70:0/1343159172 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb73409c790 con 0x5634ffefb290 2024-04-01T01:43:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.667+0000 7fb74ed88740 10 client.18378 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.668+0000 7fb74ed88740 1 -- 172.21.15.70:0/1343159172 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5634fff76df0 con 0x5634ffefb290 2024-04-01T01:43:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.668+0000 7fb72d7fa640 1 -- 172.21.15.70:0/1343159172 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb73c05c070 con 0x5634ffefb290 2024-04-01T01:43:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.668+0000 7fb72d7fa640 1 -- 172.21.15.70:0/1343159172 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb7340d1040 con 0x5634ffefb290 2024-04-01T01:43:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.668+0000 7fb74ed88740 10 client.18378 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.668+0000 7fb74ed88740 10 client.18378 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:43:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.668+0000 7fb74ed88740 20 client.18378 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.668+0000 7fb74ed88740 1 --2- 172.21.15.70:0/1343159172 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5634fff776c0 0x5634fff79ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.668+0000 7fb74ed88740 4 client.18378 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:43:17.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.668+0000 7fb74ed88740 1 -- 172.21.15.70:0/1343159172 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5634ffe62a40 con 0x5634fff776c0 2024-04-01T01:43:17.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.668+0000 7fb749f3d640 1 --2- 172.21.15.70:0/1343159172 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5634fff776c0 0x5634fff79ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.668+0000 7fb72d7fa640 10 client.18378 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:43:17.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.668+0000 7fb749f3d640 1 --2- 172.21.15.70:0/1343159172 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5634fff776c0 0x5634fff79ae0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9b242af9da74c4f2 server_cookie=373cf5252911c94d in_seq=0 out_seq=0 2024-04-01T01:43:17.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb72d7fa640 1 -- 172.21.15.70:0/1343159172 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5634ffe62a40 con 0x5634fff776c0 2024-04-01T01:43:17.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb72d7fa640 10 client.18378 handle_command_reply: tid=0 2024-04-01T01:43:17.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb74ed88740 1 client.18378 shutdown 2024-04-01T01:43:17.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb74ed88740 1 -- 172.21.15.70:0/1343159172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634ffefb290 msgr2=0x5634fff09fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb74ed88740 1 --2- 172.21.15.70:0/1343159172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634ffefb290 0x5634fff09fa0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb73400ead0 tx=0x7fb734017f90 comp rx=0 tx=0).stop 2024-04-01T01:43:17.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb74ed88740 1 -- 172.21.15.70:0/1343159172 shutdown_connections 2024-04-01T01:43:17.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb74ed88740 1 --2- 172.21.15.70:0/1343159172 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5634fff776c0 0x5634fff79ae0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb74ed88740 1 --2- 172.21.15.70:0/1343159172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634fff00400 0x5634fff0afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb74ed88740 1 --2- 172.21.15.70:0/1343159172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634ffefb290 0x5634fff09fa0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb74ed88740 1 --2- 172.21.15.70:0/1343159172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634ffefa8a0 0x5634fff09a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb72d7fa640 0 client.18378 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:43:17.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb74ed88740 1 -- 172.21.15.70:0/1343159172 >> 172.21.15.70:0/1343159172 conn(0x5634ffef1a00 msgr2=0x5634ffef04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb74ed88740 1 -- 172.21.15.70:0/1343159172 shutdown_connections 2024-04-01T01:43:17.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb74ed88740 1 -- 172.21.15.70:0/1343159172 wait complete. 2024-04-01T01:43:17.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb74ed88740 20 client.18378 trim_cache size 0 max 16384 2024-04-01T01:43:17.675 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:43:17.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb74c1a1640 1 -- 172.21.15.70:0/2136208870 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7300a2520 msgr2=0x7fb7300a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb74c1a1640 1 --2- 172.21.15.70:0/2136208870 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7300a2520 0x7fb7300a49e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb73c04c000 tx=0x7fb73c045000 comp rx=0 tx=0).stop 2024-04-01T01:43:17.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb74c1a1640 1 -- 172.21.15.70:0/2136208870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb744101850 msgr2=0x7fb744196db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.669+0000 7fb74c1a1640 1 --2- 172.21.15.70:0/2136208870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb744101850 0x7fb744196db0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fb73400eb00 tx=0x7fb734009f20 comp rx=0 tx=0).stop 2024-04-01T01:43:17.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.670+0000 7fb74c1a1640 1 -- 172.21.15.70:0/2136208870 shutdown_connections 2024-04-01T01:43:17.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.670+0000 7fb74c1a1640 1 --2- 172.21.15.70:0/2136208870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb744102210 0x7fb74419de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.670+0000 7fb74c1a1640 1 --2- 172.21.15.70:0/2136208870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb744101850 0x7fb744196db0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.670+0000 7fb74c1a1640 1 --2- 172.21.15.70:0/2136208870 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7300a2520 0x7fb7300a49e0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.670+0000 7fb74c1a1640 1 --2- 172.21.15.70:0/2136208870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb744100660 0x7fb744196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.670+0000 7fb74c1a1640 1 -- 172.21.15.70:0/2136208870 >> 172.21.15.70:0/2136208870 conn(0x7fb7440fbbf0 msgr2=0x7fb744104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.670+0000 7fb74c1a1640 1 -- 172.21.15.70:0/2136208870 shutdown_connections 2024-04-01T01:43:17.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.670+0000 7fb74c1a1640 1 -- 172.21.15.70:0/2136208870 wait complete. 2024-04-01T01:43:17.679 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:43:17.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.819+0000 7ff22affd640 1 -- 172.21.15.70:0/672161855 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c02c600 con 0x7ff22c1097e0 2024-04-01T01:43:17.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.819+0000 7ff2315ba640 1 -- 172.21.15.70:0/672161855 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff22c1097e0 msgr2=0x7ff22c10e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.819+0000 7ff2315ba640 1 --2- 172.21.15.70:0/672161855 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff22c1097e0 0x7ff22c10e080 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff21c009900 tx=0x7ff21c02bc60 comp rx=0 tx=0).stop 2024-04-01T01:43:17.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.821+0000 7ff2315ba640 1 -- 172.21.15.70:0/672161855 shutdown_connections 2024-04-01T01:43:17.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.821+0000 7ff2315ba640 1 --2- 172.21.15.70:0/672161855 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff22c1097e0 0x7ff22c10e080 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.821+0000 7ff2315ba640 1 --2- 172.21.15.70:0/672161855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff22c108e20 0x7ff22c1092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.821+0000 7ff2315ba640 1 --2- 172.21.15.70:0/672161855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff22c072b60 0x7ff22c072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.821+0000 7ff2315ba640 1 -- 172.21.15.70:0/672161855 >> 172.21.15.70:0/672161855 conn(0x7ff22c06e630 msgr2=0x7ff22c070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.823+0000 7ff2315ba640 1 -- 172.21.15.70:0/672161855 shutdown_connections 2024-04-01T01:43:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.823+0000 7ff2315ba640 1 -- 172.21.15.70:0/672161855 wait complete. 2024-04-01T01:43:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.824+0000 7ff2315ba640 1 Processor -- start 2024-04-01T01:43:17.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.824+0000 7ff2315ba640 1 -- start start 2024-04-01T01:43:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.824+0000 7ff2315ba640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff22c072b60 0x7ff22c117cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.825+0000 7ff2315ba640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff22c108e20 0x7ff22c1181f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.825+0000 7ff2315ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff22c1097e0 0x7ff22c119230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.825+0000 7ff2315ba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c110330 con 0x7ff22c1097e0 2024-04-01T01:43:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.825+0000 7ff2315ba640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1101b0 con 0x7ff22c108e20 2024-04-01T01:43:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.825+0000 7ff2315ba640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1104b0 con 0x7ff22c072b60 2024-04-01T01:43:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.825+0000 7ff22bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff22c072b60 0x7ff22c117cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.825+0000 7ff22b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff22c108e20 0x7ff22c1181f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.825+0000 7ff22b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff22c108e20 0x7ff22c1181f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49332/0 (socket says 172.21.15.70:49332) 2024-04-01T01:43:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.825+0000 7ff22b7fe640 1 -- 172.21.15.70:0/3656794572 learned_addr learned my addr 172.21.15.70:0/3656794572 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.825+0000 7ff230db9640 1 --2- 172.21.15.70:0/3656794572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff22c1097e0 0x7ff22c119230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.825+0000 7ff22b7fe640 1 -- 172.21.15.70:0/3656794572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff22c072b60 msgr2=0x7ff22c117cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.825+0000 7ff22b7fe640 1 --2- 172.21.15.70:0/3656794572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff22c072b60 0x7ff22c117cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.825+0000 7ff22b7fe640 1 -- 172.21.15.70:0/3656794572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff22c1097e0 msgr2=0x7ff22c119230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.825+0000 7ff22b7fe640 1 --2- 172.21.15.70:0/3656794572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff22c1097e0 0x7ff22c119230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.825+0000 7ff22b7fe640 1 -- 172.21.15.70:0/3656794572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff22c1198a0 con 0x7ff22c108e20 2024-04-01T01:43:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.826+0000 7ff22b7fe640 1 --2- 172.21.15.70:0/3656794572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff22c108e20 0x7ff22c1181f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff22400eab0 tx=0x7ff22400ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.826+0000 7ff2297fa640 1 -- 172.21.15.70:0/3656794572 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22400ce40 con 0x7ff22c108e20 2024-04-01T01:43:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.826+0000 7ff2297fa640 1 -- 172.21.15.70:0/3656794572 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff224011920 con 0x7ff22c108e20 2024-04-01T01:43:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.826+0000 7ff2297fa640 1 -- 172.21.15.70:0/3656794572 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff224010a00 con 0x7ff22c108e20 2024-04-01T01:43:17.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.826+0000 7ff2315ba640 1 -- 172.21.15.70:0/3656794572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff22c119b90 con 0x7ff22c108e20 2024-04-01T01:43:17.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.826+0000 7ff2315ba640 1 -- 172.21.15.70:0/3656794572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff22c11a130 con 0x7ff22c108e20 2024-04-01T01:43:17.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.829+0000 7ff2353ff740 1 Processor -- start 2024-04-01T01:43:17.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.831+0000 7ff2297fa640 1 -- 172.21.15.70:0/3656794572 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7ff224011ad0 con 0x7ff22c108e20 2024-04-01T01:43:17.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.831+0000 7ff2297fa640 1 --2- 172.21.15.70:0/3656794572 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2140a25f0 0x7ff2140a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.832+0000 7ff2297fa640 1 -- 172.21.15.70:0/3656794572 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff2240cf150 con 0x7ff22c108e20 2024-04-01T01:43:17.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.832+0000 7ff2353ff740 1 -- start start 2024-04-01T01:43:17.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.832+0000 7ff2353ff740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e6d6a0 0x55bb40e6dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.832+0000 7ff2353ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e72b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.833+0000 7ff2353ff740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e73200 0x55bb40e75690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.833+0000 7ff2353ff740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bb40dd21b0 con 0x55bb40e6d6a0 2024-04-01T01:43:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.833+0000 7ff2353ff740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55bb40d7ac00 con 0x55bb40e73200 2024-04-01T01:43:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.833+0000 7ff2353ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bb40dce250 con 0x55bb40e6e090 2024-04-01T01:43:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.833+0000 7ff22b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e72b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.833+0000 7ff22b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e72b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46682/0 (socket says 172.21.15.70:46682) 2024-04-01T01:43:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.833+0000 7ff22b7fe640 1 -- 172.21.15.70:0/2236699447 learned_addr learned my addr 172.21.15.70:0/2236699447 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.834+0000 7ff22b7fe640 1 -- 172.21.15.70:0/2236699447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e73200 msgr2=0x55bb40e75690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:43:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.834+0000 7ff22b7fe640 1 --2- 172.21.15.70:0/2236699447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e73200 0x55bb40e75690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.834+0000 7ff22b7fe640 1 -- 172.21.15.70:0/2236699447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e6d6a0 msgr2=0x55bb40e6dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.834+0000 7ff22b7fe640 1 --2- 172.21.15.70:0/2236699447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e6d6a0 0x55bb40e6dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.834+0000 7ff22b7fe640 1 -- 172.21.15.70:0/2236699447 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb40d9a3d0 con 0x55bb40e6e090 2024-04-01T01:43:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.834+0000 7ff230db9640 1 --2- 172.21.15.70:0/3656794572 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2140a25f0 0x7ff2140a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.835+0000 7ff230db9640 1 --2- 172.21.15.70:0/3656794572 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2140a25f0 0x7ff2140a4ab0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff21c006010 tx=0x7ff21c004bb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.835+0000 7ff22b7fe640 1 --2- 172.21.15.70:0/2236699447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e72b90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff2240981b0 tx=0x7ff224011e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f10703e8c30b25b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.835+0000 7ff2127fc640 1 -- 172.21.15.70:0/2236699447 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22409b7e0 con 0x55bb40e6e090 2024-04-01T01:43:17.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.836+0000 7ff2127fc640 1 -- 172.21.15.70:0/2236699447 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff22409f930 con 0x55bb40e6e090 2024-04-01T01:43:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.836+0000 7ff2127fc640 1 -- 172.21.15.70:0/2236699447 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2240202b0 con 0x55bb40e6e090 2024-04-01T01:43:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.837+0000 7ff2353ff740 1 -- 172.21.15.70:0/2236699447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 msgr2=0x55bb40e72b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.837+0000 7ff2353ff740 1 --2- 172.21.15.70:0/2236699447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e72b90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff2240981b0 tx=0x7ff224011e00 comp rx=0 tx=0).stop 2024-04-01T01:43:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.837+0000 7ff2353ff740 1 -- 172.21.15.70:0/2236699447 shutdown_connections 2024-04-01T01:43:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.837+0000 7ff2353ff740 1 --2- 172.21.15.70:0/2236699447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e73200 0x55bb40e75690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.837+0000 7ff2353ff740 1 --2- 172.21.15.70:0/2236699447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e72b90 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.837+0000 7ff2353ff740 1 --2- 172.21.15.70:0/2236699447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e6d6a0 0x55bb40e6dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.837+0000 7ff2353ff740 1 -- 172.21.15.70:0/2236699447 >> 172.21.15.70:0/2236699447 conn(0x55bb40e68c10 msgr2=0x55bb40e69010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.837+0000 7ff2353ff740 1 -- 172.21.15.70:0/2236699447 shutdown_connections 2024-04-01T01:43:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.837+0000 7ff2353ff740 1 -- 172.21.15.70:0/2236699447 wait complete. 2024-04-01T01:43:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.837+0000 7ff2353ff740 1 -- start start 2024-04-01T01:43:17.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.838+0000 7ff2353ff740 1 Processor -- start 2024-04-01T01:43:17.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.838+0000 7ff2353ff740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e6d6a0 0x55bb40e77ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.839+0000 7ff2353ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e781e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.839+0000 7ff22bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e6d6a0 0x55bb40e77ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.839+0000 7ff2353ff740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 0x55bb40e7f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.839+0000 7ff22bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e6d6a0 0x55bb40e77ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49348/0 (socket says 172.21.15.70:49348) 2024-04-01T01:43:17.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.839+0000 7ff22b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e781e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.839+0000 7ff2353ff740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bb40e761b0 con 0x55bb40e73200 2024-04-01T01:43:17.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.839+0000 7ff2353ff740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55bb40d70470 con 0x55bb40e6d6a0 2024-04-01T01:43:17.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.839+0000 7ff2353ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bb40e76330 con 0x55bb40e6e090 2024-04-01T01:43:17.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.839+0000 7ff22bfff640 1 -- 172.21.15.70:0/1557812858 learned_addr learned my addr 172.21.15.70:0/1557812858 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.839+0000 7ff2353ff740 10 client.0 _ll_register_callbacks cb 0x55bb40d8dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:17.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.839+0000 7ff230db9640 1 --2- 172.21.15.70:0/1557812858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 0x55bb40e7f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.840+0000 7ff22bfff640 1 -- 172.21.15.70:0/1557812858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 msgr2=0x55bb40e781e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.840+0000 7ff22bfff640 1 --2- 172.21.15.70:0/1557812858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e781e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.840+0000 7ff22bfff640 1 -- 172.21.15.70:0/1557812858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 msgr2=0x55bb40e7f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.840+0000 7ff22bfff640 1 --2- 172.21.15.70:0/1557812858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 0x55bb40e7f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.840+0000 7ff22bfff640 1 -- 172.21.15.70:0/1557812858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb40ae63f0 con 0x55bb40e6d6a0 2024-04-01T01:43:17.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.840+0000 7ff22bfff640 1 --2- 172.21.15.70:0/1557812858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e6d6a0 0x55bb40e77ca0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff22000dde0 tx=0x7ff22000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fd6e8b0ad32387a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.840+0000 7ff20b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:17.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.840+0000 7ff20b7fe640 1 -- 172.21.15.70:0/1557812858 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff220015070 con 0x55bb40e6d6a0 2024-04-01T01:43:17.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.840+0000 7ff20b7fe640 1 -- 172.21.15.70:0/1557812858 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff220011850 con 0x55bb40e6d6a0 2024-04-01T01:43:17.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.840+0000 7ff2353ff740 1 -- 172.21.15.70:0/1557812858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff22409f930 con 0x55bb40e6d6a0 2024-04-01T01:43:17.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.841+0000 7ff20b7fe640 1 -- 172.21.15.70:0/1557812858 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff220011a50 con 0x55bb40e6d6a0 2024-04-01T01:43:17.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.841+0000 7ff2353ff740 10 client.28391 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:17.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.841+0000 7ff2353ff740 1 -- 172.21.15.70:0/1557812858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55bb40bb07d0 con 0x55bb40e6d6a0 2024-04-01T01:43:17.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.842+0000 7ff20b7fe640 1 -- 172.21.15.70:0/1557812858 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff220011bf0 con 0x55bb40e6d6a0 2024-04-01T01:43:17.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.843+0000 7ff20b7fe640 1 -- 172.21.15.70:0/1557812858 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff220018360 con 0x55bb40e6d6a0 2024-04-01T01:43:17.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.843+0000 7ff2353ff740 10 client.28391 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:17.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.843+0000 7ff2353ff740 10 client.28391 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:43:17.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.843+0000 7ff2353ff740 20 client.28391 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:17.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.843+0000 7ff2353ff740 1 --2- 172.21.15.70:0/1557812858 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bb40e83740 0x55bb40e85b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.844+0000 7ff2353ff740 4 client.28391 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:17.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.844+0000 7ff2353ff740 1 -- 172.21.15.70:0/1557812858 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff22409f930 con 0x55bb40e83740 2024-04-01T01:43:17.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.844+0000 7ff22b7fe640 1 --2- 172.21.15.70:0/1557812858 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bb40e83740 0x55bb40e85b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.844+0000 7ff22b7fe640 1 --2- 172.21.15.70:0/1557812858 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bb40e83740 0x55bb40e85b80 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=29319c7c4b6b5d65 server_cookie=e0950619ad36885f in_seq=0 out_seq=0 2024-04-01T01:43:17.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.845+0000 7ff20b7fe640 10 client.28391 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:43:17.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.846+0000 7ff20b7fe640 1 -- 172.21.15.70:0/1557812858 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7ff22409f930 con 0x55bb40e83740 2024-04-01T01:43:17.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.846+0000 7ff20b7fe640 10 client.28391 handle_command_reply: tid=0 2024-04-01T01:43:17.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.846+0000 7ff2353ff740 1 client.28391 shutdown 2024-04-01T01:43:17.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.847+0000 7ff2353ff740 1 -- 172.21.15.70:0/1557812858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e6d6a0 msgr2=0x55bb40e77ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.847+0000 7ff2353ff740 1 --2- 172.21.15.70:0/1557812858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e6d6a0 0x55bb40e77ca0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff22000dde0 tx=0x7ff22000f730 comp rx=0 tx=0).stop 2024-04-01T01:43:17.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.848+0000 7ff2353ff740 1 -- 172.21.15.70:0/1557812858 shutdown_connections 2024-04-01T01:43:17.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.848+0000 7ff2353ff740 1 --2- 172.21.15.70:0/1557812858 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bb40e83740 0x55bb40e85b80 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.848+0000 7ff2353ff740 1 --2- 172.21.15.70:0/1557812858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 0x55bb40e7f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.848+0000 7ff2353ff740 1 --2- 172.21.15.70:0/1557812858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e781e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.848+0000 7ff2353ff740 1 --2- 172.21.15.70:0/1557812858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e6d6a0 0x55bb40e77ca0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.848+0000 7ff2353ff740 1 -- 172.21.15.70:0/1557812858 >> 172.21.15.70:0/1557812858 conn(0x55bb40e68c10 msgr2=0x55bb40e67f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.849+0000 7ff20b7fe640 0 client.28391 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:43:17.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.849+0000 7ff2353ff740 1 -- 172.21.15.70:0/1557812858 shutdown_connections 2024-04-01T01:43:17.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.849+0000 7ff2353ff740 1 -- 172.21.15.70:0/1557812858 wait complete. 2024-04-01T01:43:17.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.850+0000 7ff2353ff740 20 client.28391 trim_cache size 0 max 16384 2024-04-01T01:43:17.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.879+0000 7ff2353ff740 1 Processor -- start 2024-04-01T01:43:17.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.881+0000 7ff2353ff740 1 -- start start 2024-04-01T01:43:17.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.882+0000 7ff2353ff740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e6d6a0 0x55bb40e68ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.882+0000 7ff2353ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e7f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.882+0000 7ff2353ff740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 0x55bb40e778b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.882+0000 7ff2353ff740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1c17e0 con 0x55bb40e73200 2024-04-01T01:43:17.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.882+0000 7ff2353ff740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1c3320 con 0x55bb40e6d6a0 2024-04-01T01:43:17.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.882+0000 7ff2353ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1c1eb0 con 0x55bb40e6e090 2024-04-01T01:43:17.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.886+0000 7ff230db9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 0x55bb40e778b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.886+0000 7ff230db9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 0x55bb40e778b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34294/0 (socket says 172.21.15.70:34294) 2024-04-01T01:43:17.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.886+0000 7ff230db9640 1 -- 172.21.15.70:0/650847610 learned_addr learned my addr 172.21.15.70:0/650847610 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.886+0000 7ff22bfff640 1 --2- 172.21.15.70:0/650847610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e6d6a0 0x55bb40e68ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.886+0000 7ff22b7fe640 1 --2- 172.21.15.70:0/650847610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e7f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.887+0000 7ff230db9640 1 -- 172.21.15.70:0/650847610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 msgr2=0x55bb40e7f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.887+0000 7ff230db9640 1 --2- 172.21.15.70:0/650847610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e7f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.887+0000 7ff230db9640 1 -- 172.21.15.70:0/650847610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e6d6a0 msgr2=0x55bb40e68ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.887+0000 7ff230db9640 1 --2- 172.21.15.70:0/650847610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e6d6a0 0x55bb40e68ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.887+0000 7ff230db9640 1 -- 172.21.15.70:0/650847610 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb40e64670 con 0x55bb40e73200 2024-04-01T01:43:17.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.888+0000 7ff230db9640 1 --2- 172.21.15.70:0/650847610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 0x55bb40e778b0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ff21c004a20 tx=0x7ff21c03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31cf97052049ecd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.888+0000 7ff211ffb640 1 -- 172.21.15.70:0/650847610 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c02db90 con 0x55bb40e73200 2024-04-01T01:43:17.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.888+0000 7ff211ffb640 1 -- 172.21.15.70:0/650847610 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff21c051070 con 0x55bb40e73200 2024-04-01T01:43:17.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.889+0000 7ff2353ff740 1 -- 172.21.15.70:0/650847610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 msgr2=0x55bb40e778b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.889+0000 7ff2353ff740 1 --2- 172.21.15.70:0/650847610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 0x55bb40e778b0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ff21c004a20 tx=0x7ff21c03d000 comp rx=0 tx=0).stop 2024-04-01T01:43:17.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.889+0000 7ff2353ff740 1 -- 172.21.15.70:0/650847610 shutdown_connections 2024-04-01T01:43:17.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.889+0000 7ff2353ff740 1 --2- 172.21.15.70:0/650847610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 0x55bb40e778b0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.889+0000 7ff2353ff740 1 --2- 172.21.15.70:0/650847610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e7f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.889+0000 7ff2353ff740 1 --2- 172.21.15.70:0/650847610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e6d6a0 0x55bb40e68ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.889+0000 7ff2353ff740 1 -- 172.21.15.70:0/650847610 >> 172.21.15.70:0/650847610 conn(0x55bb40e68c10 msgr2=0x55bb40e6eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.892+0000 7ff2353ff740 1 -- 172.21.15.70:0/650847610 shutdown_connections 2024-04-01T01:43:17.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.892+0000 7ff2353ff740 1 -- 172.21.15.70:0/650847610 wait complete. 2024-04-01T01:43:17.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.893+0000 7ff2353ff740 1 -- start start 2024-04-01T01:43:17.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.893+0000 7ff2353ff740 1 Processor -- start 2024-04-01T01:43:17.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.893+0000 7ff2353ff740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e6d6a0 0x55bb40e7c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.893+0000 7ff2353ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e7c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.893+0000 7ff2353ff740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 0x55bb40e7d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.893+0000 7ff2353ff740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1c2c50 con 0x55bb40e73200 2024-04-01T01:43:17.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.893+0000 7ff2353ff740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1c2580 con 0x55bb40e6d6a0 2024-04-01T01:43:17.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.893+0000 7ff2353ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff22c1bf7c0 con 0x55bb40e6e090 2024-04-01T01:43:17.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.893+0000 7ff2353ff740 10 client.0 _ll_register_callbacks cb 0x7ff22000d970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:17.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.894+0000 7ff230db9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 0x55bb40e7d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.894+0000 7ff230db9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 0x55bb40e7d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34300/0 (socket says 172.21.15.70:34300) 2024-04-01T01:43:17.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.894+0000 7ff230db9640 1 -- 172.21.15.70:0/1056941500 learned_addr learned my addr 172.21.15.70:0/1056941500 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:17.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.895+0000 7ff230db9640 1 -- 172.21.15.70:0/1056941500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 msgr2=0x55bb40e7c8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:43:17.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.895+0000 7ff230db9640 1 --2- 172.21.15.70:0/1056941500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e7c8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.895+0000 7ff230db9640 1 -- 172.21.15.70:0/1056941500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e6d6a0 msgr2=0x55bb40e7c3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:43:17.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.895+0000 7ff230db9640 1 --2- 172.21.15.70:0/1056941500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e6d6a0 0x55bb40e7c3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.895+0000 7ff230db9640 1 -- 172.21.15.70:0/1056941500 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb40e7dee0 con 0x55bb40e73200 2024-04-01T01:43:17.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.895+0000 7ff230db9640 1 --2- 172.21.15.70:0/1056941500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 0x55bb40e7d8f0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ff21c005d90 tx=0x7ff21c004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd52c9655709951 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:17.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.895+0000 7ff20b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:43:17.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.895+0000 7ff20b7fe640 1 -- 172.21.15.70:0/1056941500 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21c005580 con 0x55bb40e73200 2024-04-01T01:43:17.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.895+0000 7ff20b7fe640 1 -- 172.21.15.70:0/1056941500 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff21c0035f0 con 0x55bb40e73200 2024-04-01T01:43:17.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.896+0000 7ff2353ff740 1 -- 172.21.15.70:0/1056941500 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff21c051070 con 0x55bb40e73200 2024-04-01T01:43:17.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.896+0000 7ff20b7fe640 1 -- 172.21.15.70:0/1056941500 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff21c0037f0 con 0x55bb40e73200 2024-04-01T01:43:17.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.896+0000 7ff2353ff740 10 client.18393 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:17.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.896+0000 7ff2353ff740 1 -- 172.21.15.70:0/1056941500 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55bb40e7f210 con 0x55bb40e73200 2024-04-01T01:43:17.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.896+0000 7ff20b7fe640 1 -- 172.21.15.70:0/1056941500 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff21c051070 con 0x55bb40e73200 2024-04-01T01:43:17.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.896+0000 7ff20b7fe640 1 -- 172.21.15.70:0/1056941500 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff21c040cb0 con 0x55bb40e73200 2024-04-01T01:43:17.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.897+0000 7ff2353ff740 10 client.18393 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:17.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.897+0000 7ff2353ff740 10 client.18393 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:43:17.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.897+0000 7ff2353ff740 20 client.18393 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:17.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.897+0000 7ff2353ff740 1 --2- 172.21.15.70:0/1056941500 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bb40ee9fc0 0x55bb40eec3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:17.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.897+0000 7ff2353ff740 4 client.18393 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:43:17.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.897+0000 7ff2353ff740 1 -- 172.21.15.70:0/1056941500 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bb40ddf6f0 con 0x55bb40ee9fc0 2024-04-01T01:43:17.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.897+0000 7ff22bfff640 1 --2- 172.21.15.70:0/1056941500 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bb40ee9fc0 0x55bb40eec3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:17.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.898+0000 7ff22bfff640 1 --2- 172.21.15.70:0/1056941500 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bb40ee9fc0 0x55bb40eec3e0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=72ea910f400daef server_cookie=a84a0d2e0a29131c in_seq=0 out_seq=0 2024-04-01T01:43:17.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.898+0000 7ff20b7fe640 10 client.18393 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:43:17.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.898+0000 7ff20b7fe640 1 -- 172.21.15.70:0/1056941500 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bb40ddf6f0 con 0x55bb40ee9fc0 2024-04-01T01:43:17.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.898+0000 7ff20b7fe640 10 client.18393 handle_command_reply: tid=0 2024-04-01T01:43:17.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.898+0000 7ff2353ff740 1 client.18393 shutdown 2024-04-01T01:43:17.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.899+0000 7ff2353ff740 1 -- 172.21.15.70:0/1056941500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 msgr2=0x55bb40e7d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.899+0000 7ff2353ff740 1 --2- 172.21.15.70:0/1056941500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 0x55bb40e7d8f0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ff21c005d90 tx=0x7ff21c004590 comp rx=0 tx=0).stop 2024-04-01T01:43:17.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.899+0000 7ff2353ff740 1 -- 172.21.15.70:0/1056941500 shutdown_connections 2024-04-01T01:43:17.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.899+0000 7ff2353ff740 1 --2- 172.21.15.70:0/1056941500 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55bb40ee9fc0 0x55bb40eec3e0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.899+0000 7ff2353ff740 1 --2- 172.21.15.70:0/1056941500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bb40e73200 0x55bb40e7d8f0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.899+0000 7ff2353ff740 1 --2- 172.21.15.70:0/1056941500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb40e6e090 0x55bb40e7c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.899+0000 7ff2353ff740 1 --2- 172.21.15.70:0/1056941500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bb40e6d6a0 0x55bb40e7c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.899+0000 7ff2353ff740 1 -- 172.21.15.70:0/1056941500 >> 172.21.15.70:0/1056941500 conn(0x55bb40e68c10 msgr2=0x55bb40e68ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.899+0000 7ff20b7fe640 0 client.18393 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:43:17.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.899+0000 7ff2353ff740 1 -- 172.21.15.70:0/1056941500 shutdown_connections 2024-04-01T01:43:17.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.899+0000 7ff2353ff740 1 -- 172.21.15.70:0/1056941500 wait complete. 2024-04-01T01:43:17.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.899+0000 7ff2353ff740 20 client.18393 trim_cache size 0 max 16384 2024-04-01T01:43:17.900 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:43:17.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.900+0000 7ff2315ba640 1 -- 172.21.15.70:0/3656794572 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2140a25f0 msgr2=0x7ff2140a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.900+0000 7ff2315ba640 1 --2- 172.21.15.70:0/3656794572 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2140a25f0 0x7ff2140a4ab0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff21c006010 tx=0x7ff21c004bb0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.900+0000 7ff2315ba640 1 -- 172.21.15.70:0/3656794572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff22c108e20 msgr2=0x7ff22c1181f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:17.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.900+0000 7ff2315ba640 1 --2- 172.21.15.70:0/3656794572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff22c108e20 0x7ff22c1181f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff22400eab0 tx=0x7ff22400ef70 comp rx=0 tx=0).stop 2024-04-01T01:43:17.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.901+0000 7ff2315ba640 1 -- 172.21.15.70:0/3656794572 shutdown_connections 2024-04-01T01:43:17.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.901+0000 7ff2315ba640 1 --2- 172.21.15.70:0/3656794572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff22c1097e0 0x7ff22c119230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.901+0000 7ff2315ba640 1 --2- 172.21.15.70:0/3656794572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff22c108e20 0x7ff22c1181f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.901+0000 7ff2315ba640 1 --2- 172.21.15.70:0/3656794572 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff2140a25f0 0x7ff2140a4ab0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.901+0000 7ff2315ba640 1 --2- 172.21.15.70:0/3656794572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff22c072b60 0x7ff22c117cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:17.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.901+0000 7ff2315ba640 1 -- 172.21.15.70:0/3656794572 >> 172.21.15.70:0/3656794572 conn(0x7ff22c06e630 msgr2=0x7ff22c10c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:17.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.901+0000 7ff2315ba640 1 -- 172.21.15.70:0/3656794572 shutdown_connections 2024-04-01T01:43:17.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:17.901+0000 7ff2315ba640 1 -- 172.21.15.70:0/3656794572 wait complete. 2024-04-01T01:43:17.921 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:43:18.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.089+0000 7f8b1d9e7640 1 -- 172.21.15.70:0/3764271069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b18108e40 msgr2=0x7f8b181092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:18.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.089+0000 7f8b1d9e7640 1 --2- 172.21.15.70:0/3764271069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b18108e40 0x7f8b181092c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8b08009860 tx=0x7f8b0802bbb0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.090+0000 7f8b1d9e7640 1 -- 172.21.15.70:0/3764271069 shutdown_connections 2024-04-01T01:43:18.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.090+0000 7f8b1d9e7640 1 --2- 172.21.15.70:0/3764271069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b18109800 0x7f8b18110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.090+0000 7f8b1d9e7640 1 --2- 172.21.15.70:0/3764271069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b18108e40 0x7f8b181092c0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.090+0000 7f8b1d9e7640 1 --2- 172.21.15.70:0/3764271069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b18107c50 0x7f8b18108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.090+0000 7f8b1d9e7640 1 -- 172.21.15.70:0/3764271069 >> 172.21.15.70:0/3764271069 conn(0x7f8b18076bf0 msgr2=0x7f8b18079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:18.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.091+0000 7f8b1d9e7640 1 -- 172.21.15.70:0/3764271069 shutdown_connections 2024-04-01T01:43:18.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.091+0000 7f8b1d9e7640 1 -- 172.21.15.70:0/3764271069 wait complete. 2024-04-01T01:43:18.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.092+0000 7f8b1d9e7640 1 Processor -- start 2024-04-01T01:43:18.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.092+0000 7f8b1d9e7640 1 -- start start 2024-04-01T01:43:18.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.092+0000 7f8b1d9e7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b18107c50 0x7f8b1819dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.092+0000 7f8b1d9e7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b18108e40 0x7f8b1819e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.092+0000 7f8b1d9e7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b18109800 0x7f8b181a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.092+0000 7f8b1d9e7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8b181137f0 con 0x7f8b18109800 2024-04-01T01:43:18.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.092+0000 7f8b1d9e7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8b18113670 con 0x7f8b18108e40 2024-04-01T01:43:18.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.092+0000 7f8b1d9e7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8b18113970 con 0x7f8b18107c50 2024-04-01T01:43:18.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.092+0000 7f8b17fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b18108e40 0x7f8b1819e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:18.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.092+0000 7f8b17fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b18108e40 0x7f8b1819e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49368/0 (socket says 172.21.15.70:49368) 2024-04-01T01:43:18.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.092+0000 7f8b17fff640 1 -- 172.21.15.70:0/2979752154 learned_addr learned my addr 172.21.15.70:0/2979752154 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:18.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.092+0000 7f8b1c9e5640 1 --2- 172.21.15.70:0/2979752154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b18107c50 0x7f8b1819dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.093+0000 7f8b17fff640 1 -- 172.21.15.70:0/2979752154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b18107c50 msgr2=0x7f8b1819dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.093+0000 7f8b17fff640 1 --2- 172.21.15.70:0/2979752154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b18107c50 0x7f8b1819dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.093+0000 7f8b17fff640 1 -- 172.21.15.70:0/2979752154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b18109800 msgr2=0x7f8b181a5470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:43:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.093+0000 7f8b17fff640 1 --2- 172.21.15.70:0/2979752154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b18109800 0x7f8b181a5470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.093+0000 7f8b17fff640 1 -- 172.21.15.70:0/2979752154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b1807a0d0 con 0x7f8b18108e40 2024-04-01T01:43:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.093+0000 7f8b17fff640 1 --2- 172.21.15.70:0/2979752154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b18108e40 0x7f8b1819e400 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8b08009460 tx=0x7f8b080042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.094+0000 7f8b15ffb640 1 -- 172.21.15.70:0/2979752154 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08041020 con 0x7f8b18108e40 2024-04-01T01:43:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.094+0000 7f8b15ffb640 1 -- 172.21.15.70:0/2979752154 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b080035f0 con 0x7f8b18108e40 2024-04-01T01:43:18.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.094+0000 7f8b15ffb640 1 -- 172.21.15.70:0/2979752154 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08039040 con 0x7f8b18108e40 2024-04-01T01:43:18.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.094+0000 7f8b1d9e7640 1 -- 172.21.15.70:0/2979752154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b1807a360 con 0x7f8b18108e40 2024-04-01T01:43:18.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.095+0000 7f8b1d9e7640 1 -- 172.21.15.70:0/2979752154 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b1807a880 con 0x7f8b18108e40 2024-04-01T01:43:18.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.104+0000 7f8b2181f740 1 Processor -- start 2024-04-01T01:43:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.104+0000 7f8b2181f740 1 -- start start 2024-04-01T01:43:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.105+0000 7f8b15ffb640 1 -- 172.21.15.70:0/2979752154 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f8b08044020 con 0x7f8b18108e40 2024-04-01T01:43:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.105+0000 7f8b2181f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e876a0 0x55dbd1e87ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.105+0000 7f8b2181f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e88090 0x55dbd1e8cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.105+0000 7f8b2181f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 0x55dbd1e8f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.105+0000 7f8b2181f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dbd1dec1b0 con 0x55dbd1e8d200 2024-04-01T01:43:18.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.105+0000 7f8b2181f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55dbd1d94c00 con 0x55dbd1e876a0 2024-04-01T01:43:18.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.105+0000 7f8b2181f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dbd1de8250 con 0x55dbd1e88090 2024-04-01T01:43:18.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.105+0000 7f8b1c9e5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e88090 0x55dbd1e8cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:18.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.105+0000 7f8b1c9e5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e88090 0x55dbd1e8cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46714/0 (socket says 172.21.15.70:46714) 2024-04-01T01:43:18.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.105+0000 7f8b1c9e5640 1 -- 172.21.15.70:0/1254955593 learned_addr learned my addr 172.21.15.70:0/1254955593 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:18.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.106+0000 7f8b1d1e6640 1 --2- 172.21.15.70:0/1254955593 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e876a0 0x55dbd1e87ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:18.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.106+0000 7f8b15ffb640 1 --2- 172.21.15.70:0/2979752154 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8b000a2670 0x7f8b000a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.106+0000 7f8b17fff640 1 --2- 172.21.15.70:0/1254955593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 0x55dbd1e8f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:18.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.107+0000 7f8b1c9e5640 1 -- 172.21.15.70:0/1254955593 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e876a0 msgr2=0x55dbd1e87ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:18.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.107+0000 7f8b1c9e5640 1 --2- 172.21.15.70:0/1254955593 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e876a0 0x55dbd1e87ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.107+0000 7f8b1c9e5640 1 -- 172.21.15.70:0/1254955593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 msgr2=0x55dbd1e8f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:18.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.107+0000 7f8b1c9e5640 1 --2- 172.21.15.70:0/1254955593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 0x55dbd1e8f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.107+0000 7f8b1c9e5640 1 -- 172.21.15.70:0/1254955593 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbd1db43d0 con 0x55dbd1e88090 2024-04-01T01:43:18.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.107+0000 7f8b15ffb640 1 -- 172.21.15.70:0/2979752154 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8b080efbf0 con 0x7f8b18108e40 2024-04-01T01:43:18.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.107+0000 7f8b1c9e5640 1 --2- 172.21.15.70:0/1254955593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e88090 0x55dbd1e8cb90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009a30 tx=0x7f8b0c009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9354aebfebe9262b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:18.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.108+0000 7f8afeffd640 1 -- 172.21.15.70:0/1254955593 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c017070 con 0x55dbd1e88090 2024-04-01T01:43:18.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.108+0000 7f8afeffd640 1 -- 172.21.15.70:0/1254955593 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b0c013470 con 0x55dbd1e88090 2024-04-01T01:43:18.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.108+0000 7f8afeffd640 1 -- 172.21.15.70:0/1254955593 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c012630 con 0x55dbd1e88090 2024-04-01T01:43:18.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.110+0000 7f8b2181f740 1 -- 172.21.15.70:0/1254955593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e88090 msgr2=0x55dbd1e8cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:18.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.110+0000 7f8b1d1e6640 1 --2- 172.21.15.70:0/2979752154 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8b000a2670 0x7f8b000a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:18.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.110+0000 7f8b2181f740 1 --2- 172.21.15.70:0/1254955593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e88090 0x55dbd1e8cb90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009a30 tx=0x7f8b0c009ef0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.111+0000 7f8b1d1e6640 1 --2- 172.21.15.70:0/2979752154 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8b000a2670 0x7f8b000a4b30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x55dbd1e888f0 tx=0x7f8b10004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:18.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.111+0000 7f8b2181f740 1 -- 172.21.15.70:0/1254955593 shutdown_connections 2024-04-01T01:43:18.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.111+0000 7f8b2181f740 1 --2- 172.21.15.70:0/1254955593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 0x55dbd1e8f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.111+0000 7f8b2181f740 1 --2- 172.21.15.70:0/1254955593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e88090 0x55dbd1e8cb90 secure :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009a30 tx=0x7f8b0c009ef0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.111+0000 7f8b2181f740 1 --2- 172.21.15.70:0/1254955593 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e876a0 0x55dbd1e87ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.111+0000 7f8b2181f740 1 -- 172.21.15.70:0/1254955593 >> 172.21.15.70:0/1254955593 conn(0x55dbd1e82c10 msgr2=0x55dbd1e83010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:18.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.112+0000 7f8b2181f740 1 -- 172.21.15.70:0/1254955593 shutdown_connections 2024-04-01T01:43:18.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.112+0000 7f8b2181f740 1 -- 172.21.15.70:0/1254955593 wait complete. 2024-04-01T01:43:18.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.113+0000 7f8b2181f740 1 -- start start 2024-04-01T01:43:18.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.113+0000 7f8b2181f740 1 Processor -- start 2024-04-01T01:43:18.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.113+0000 7f8b2181f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e876a0 0x55dbd1e91d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.113+0000 7f8b2181f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e8d200 0x55dbd1e922b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.114+0000 7f8b2181f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e972e0 0x55dbd1e996d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.114+0000 7f8b2181f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dbd1e901b0 con 0x55dbd1e972e0 2024-04-01T01:43:18.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.114+0000 7f8b2181f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55dbd1d8a470 con 0x55dbd1e8d200 2024-04-01T01:43:18.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.114+0000 7f8b2181f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dbd1e90330 con 0x55dbd1e876a0 2024-04-01T01:43:18.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.114+0000 7f8b1c9e5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e876a0 0x55dbd1e91d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:18.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.114+0000 7f8b2181f740 10 client.0 _ll_register_callbacks cb 0x55dbd1da7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:18.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.114+0000 7f8b1c9e5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e876a0 0x55dbd1e91d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46720/0 (socket says 172.21.15.70:46720) 2024-04-01T01:43:18.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.114+0000 7f8b1c9e5640 1 -- 172.21.15.70:0/1407873221 learned_addr learned my addr 172.21.15.70:0/1407873221 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:18.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.114+0000 7f8b17fff640 1 --2- 172.21.15.70:0/1407873221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e8d200 0x55dbd1e922b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:18.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.114+0000 7f8b1d1e6640 1 --2- 172.21.15.70:0/1407873221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e972e0 0x55dbd1e996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:18.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.114+0000 7f8b1c9e5640 1 -- 172.21.15.70:0/1407873221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e8d200 msgr2=0x55dbd1e922b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:18.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.114+0000 7f8b1c9e5640 1 --2- 172.21.15.70:0/1407873221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e8d200 0x55dbd1e922b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.114+0000 7f8b1c9e5640 1 -- 172.21.15.70:0/1407873221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e972e0 msgr2=0x55dbd1e996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:18.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.114+0000 7f8b1c9e5640 1 --2- 172.21.15.70:0/1407873221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e972e0 0x55dbd1e996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.114+0000 7f8b1c9e5640 1 -- 172.21.15.70:0/1407873221 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbd1b003f0 con 0x55dbd1e876a0 2024-04-01T01:43:18.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.114+0000 7f8b1d1e6640 1 --2- 172.21.15.70:0/1407873221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e972e0 0x55dbd1e996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:43:18.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.115+0000 7f8b1c9e5640 1 --2- 172.21.15.70:0/1407873221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e876a0 0x55dbd1e91d70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009a30 tx=0x7f8b0c013850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11d26dc0d4fdd0b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:18.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.115+0000 7f8af7fff640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:43:18.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.119+0000 7f8af7fff640 1 -- 172.21.15.70:0/1407873221 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c022070 con 0x55dbd1e876a0 2024-04-01T01:43:18.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.119+0000 7f8af7fff640 1 -- 172.21.15.70:0/1407873221 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8b0c01c650 con 0x55dbd1e876a0 2024-04-01T01:43:18.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.119+0000 7f8b2181f740 1 -- 172.21.15.70:0/1407873221 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b0c013470 con 0x55dbd1e876a0 2024-04-01T01:43:18.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.121+0000 7f8af7fff640 1 -- 172.21.15.70:0/1407873221 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8b0c017040 con 0x55dbd1e876a0 2024-04-01T01:43:18.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.121+0000 7f8b2181f740 10 client.28210 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:18.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.121+0000 7f8b2181f740 1 -- 172.21.15.70:0/1407873221 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55dbd1bca7d0 con 0x55dbd1e876a0 2024-04-01T01:43:18.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.121+0000 7f8af7fff640 1 -- 172.21.15.70:0/1407873221 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b0c013470 con 0x55dbd1e876a0 2024-04-01T01:43:18.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.121+0000 7f8af7fff640 1 -- 172.21.15.70:0/1407873221 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8b0c030780 con 0x55dbd1e876a0 2024-04-01T01:43:18.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.121+0000 7f8b2181f740 10 client.28210 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:18.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.121+0000 7f8b2181f740 10 client.28210 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:18.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.121+0000 7f8b2181f740 20 client.28210 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:18.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.121+0000 7f8b2181f740 1 --2- 172.21.15.70:0/1407873221 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55dbd1e9d830 0x55dbd1e9fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.121+0000 7f8b2181f740 4 client.28210 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:18.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.121+0000 7f8b2181f740 1 -- 172.21.15.70:0/1407873221 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbd1df96f0 con 0x55dbd1e9d830 2024-04-01T01:43:18.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.122+0000 7f8b17fff640 1 --2- 172.21.15.70:0/1407873221 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55dbd1e9d830 0x55dbd1e9fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:18.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.123+0000 7f8b17fff640 1 --2- 172.21.15.70:0/1407873221 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55dbd1e9d830 0x55dbd1e9fc20 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=264b34d30dc5e892 server_cookie=fcb2d7498fe0eebc in_seq=0 out_seq=0 2024-04-01T01:43:18.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.123+0000 7f8af7fff640 10 client.28210 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:18.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.124+0000 7f8af7fff640 1 -- 172.21.15.70:0/1407873221 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dbd1df96f0 con 0x55dbd1e9d830 2024-04-01T01:43:18.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.124+0000 7f8af7fff640 10 client.28210 handle_command_reply: tid=0 2024-04-01T01:43:18.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.124+0000 7f8b2181f740 1 client.28210 shutdown 2024-04-01T01:43:18.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.125+0000 7f8b2181f740 1 -- 172.21.15.70:0/1407873221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e876a0 msgr2=0x55dbd1e91d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:18.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.125+0000 7f8b2181f740 1 --2- 172.21.15.70:0/1407873221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e876a0 0x55dbd1e91d70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009a30 tx=0x7f8b0c013850 comp rx=0 tx=0).stop 2024-04-01T01:43:18.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.125+0000 7f8b2181f740 1 -- 172.21.15.70:0/1407873221 shutdown_connections 2024-04-01T01:43:18.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.125+0000 7f8b2181f740 1 --2- 172.21.15.70:0/1407873221 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55dbd1e9d830 0x55dbd1e9fc20 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.125+0000 7f8b2181f740 1 --2- 172.21.15.70:0/1407873221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e972e0 0x55dbd1e996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.125+0000 7f8b2181f740 1 --2- 172.21.15.70:0/1407873221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e8d200 0x55dbd1e922b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.125+0000 7f8b2181f740 1 --2- 172.21.15.70:0/1407873221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e876a0 0x55dbd1e91d70 secure :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009a30 tx=0x7f8b0c013850 comp rx=0 tx=0).stop 2024-04-01T01:43:18.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.125+0000 7f8b2181f740 1 -- 172.21.15.70:0/1407873221 >> 172.21.15.70:0/1407873221 conn(0x55dbd1e82c10 msgr2=0x55dbd1e81f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:18.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.128+0000 7f8af7fff640 0 client.28210 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:18.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.128+0000 7f8b2181f740 1 -- 172.21.15.70:0/1407873221 shutdown_connections 2024-04-01T01:43:18.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.129+0000 7f8b2181f740 1 -- 172.21.15.70:0/1407873221 wait complete. 2024-04-01T01:43:18.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.129+0000 7f8b2181f740 20 client.28210 trim_cache size 0 max 16384 2024-04-01T01:43:18.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.162+0000 7f8b2181f740 1 Processor -- start 2024-04-01T01:43:18.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.162+0000 7f8b2181f740 1 -- start start 2024-04-01T01:43:18.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.162+0000 7f8b2181f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 0x55dbd1e91980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.162+0000 7f8b2181f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e972e0 0x55dbd1e91ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.162+0000 7f8b2181f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e92400 0x55dbd1e81700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.162+0000 7f8b2181f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8b181b9100 con 0x55dbd1e8d200 2024-04-01T01:43:18.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.162+0000 7f8b2181f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8b181bac40 con 0x55dbd1e972e0 2024-04-01T01:43:18.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.162+0000 7f8b2181f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8b181b97d0 con 0x55dbd1e92400 2024-04-01T01:43:18.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.163+0000 7f8b1d1e6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e92400 0x55dbd1e81700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:18.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.163+0000 7f8b1d1e6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e92400 0x55dbd1e81700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46734/0 (socket says 172.21.15.70:46734) 2024-04-01T01:43:18.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.163+0000 7f8b1d1e6640 1 -- 172.21.15.70:0/2686652392 learned_addr learned my addr 172.21.15.70:0/2686652392 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:18.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.163+0000 7f8b17fff640 1 --2- 172.21.15.70:0/2686652392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e972e0 0x55dbd1e91ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:18.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.163+0000 7f8b1c9e5640 1 --2- 172.21.15.70:0/2686652392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 0x55dbd1e91980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:18.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.163+0000 7f8b1d1e6640 1 -- 172.21.15.70:0/2686652392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e972e0 msgr2=0x55dbd1e91ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:18.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.163+0000 7f8b1d1e6640 1 --2- 172.21.15.70:0/2686652392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e972e0 0x55dbd1e91ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.163+0000 7f8b1d1e6640 1 -- 172.21.15.70:0/2686652392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 msgr2=0x55dbd1e91980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:18.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.163+0000 7f8b1d1e6640 1 --2- 172.21.15.70:0/2686652392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 0x55dbd1e91980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.163+0000 7f8b1d1e6640 1 -- 172.21.15.70:0/2686652392 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbd1e73e60 con 0x55dbd1e92400 2024-04-01T01:43:18.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.164+0000 7f8b1d1e6640 1 --2- 172.21.15.70:0/2686652392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e92400 0x55dbd1e81700 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8b10010140 tx=0x7f8b10009b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f3573841f90f5f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:18.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.165+0000 7f8afe7fc640 1 -- 172.21.15.70:0/2686652392 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b10003b70 con 0x55dbd1e92400 2024-04-01T01:43:18.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.165+0000 7f8afe7fc640 1 -- 172.21.15.70:0/2686652392 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b10003d10 con 0x55dbd1e92400 2024-04-01T01:43:18.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.165+0000 7f8afe7fc640 1 -- 172.21.15.70:0/2686652392 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b100045d0 con 0x55dbd1e92400 2024-04-01T01:43:18.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.166+0000 7f8b2181f740 1 -- 172.21.15.70:0/2686652392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e92400 msgr2=0x55dbd1e81700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:18.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.166+0000 7f8b2181f740 1 --2- 172.21.15.70:0/2686652392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e92400 0x55dbd1e81700 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8b10010140 tx=0x7f8b10009b40 comp rx=0 tx=0).stop 2024-04-01T01:43:18.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.166+0000 7f8b2181f740 1 -- 172.21.15.70:0/2686652392 shutdown_connections 2024-04-01T01:43:18.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.166+0000 7f8b2181f740 1 --2- 172.21.15.70:0/2686652392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e92400 0x55dbd1e81700 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.166+0000 7f8b2181f740 1 --2- 172.21.15.70:0/2686652392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e972e0 0x55dbd1e91ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.166+0000 7f8b2181f740 1 --2- 172.21.15.70:0/2686652392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 0x55dbd1e91980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.166+0000 7f8b2181f740 1 -- 172.21.15.70:0/2686652392 >> 172.21.15.70:0/2686652392 conn(0x55dbd1e82c10 msgr2=0x55dbd1e897e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:18.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.166+0000 7f8b2181f740 1 -- 172.21.15.70:0/2686652392 shutdown_connections 2024-04-01T01:43:18.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.166+0000 7f8b2181f740 1 -- 172.21.15.70:0/2686652392 wait complete. 2024-04-01T01:43:18.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.167+0000 7f8b2181f740 1 -- start start 2024-04-01T01:43:18.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.167+0000 7f8b2181f740 1 Processor -- start 2024-04-01T01:43:18.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.167+0000 7f8b2181f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 0x55dbd1eaeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.168+0000 7f8b2181f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e92400 0x55dbd1eaf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.168+0000 7f8b2181f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e972e0 0x55dbd1eb0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.168+0000 7f8b1c9e5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 0x55dbd1eaeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:18.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.168+0000 7f8b1c9e5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 0x55dbd1eaeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34328/0 (socket says 172.21.15.70:34328) 2024-04-01T01:43:18.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.168+0000 7f8b17fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e92400 0x55dbd1eaf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:18.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.168+0000 7f8b1c9e5640 1 -- 172.21.15.70:0/1657870416 learned_addr learned my addr 172.21.15.70:0/1657870416 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:18.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.168+0000 7f8b2181f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8b181ba570 con 0x55dbd1e8d200 2024-04-01T01:43:18.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.168+0000 7f8b2181f740 1 -- 172.21.15.70:0/1657870416 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8b181b9ea0 con 0x55dbd1e92400 2024-04-01T01:43:18.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.168+0000 7f8b2181f740 1 -- 172.21.15.70:0/1657870416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8b180825c0 con 0x55dbd1e972e0 2024-04-01T01:43:18.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.168+0000 7f8b2181f740 10 client.0 _ll_register_callbacks cb 0x7f8b0c00ef50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:18.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.169+0000 7f8b1c9e5640 1 -- 172.21.15.70:0/1657870416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e972e0 msgr2=0x55dbd1eb0060 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:43:18.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.169+0000 7f8b1c9e5640 1 --2- 172.21.15.70:0/1657870416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e972e0 0x55dbd1eb0060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.169+0000 7f8b1c9e5640 1 -- 172.21.15.70:0/1657870416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e92400 msgr2=0x55dbd1eaf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:18.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.169+0000 7f8b1c9e5640 1 --2- 172.21.15.70:0/1657870416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e92400 0x55dbd1eaf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.169+0000 7f8b1c9e5640 1 -- 172.21.15.70:0/1657870416 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbd1eb0650 con 0x55dbd1e8d200 2024-04-01T01:43:18.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.170+0000 7f8b1c9e5640 1 --2- 172.21.15.70:0/1657870416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 0x55dbd1eaeb10 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c000c00 tx=0x7f8b0c005c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de4c827ae8e61d11 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:18.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.170+0000 7f8af7fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:43:18.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.170+0000 7f8af7fff640 1 -- 172.21.15.70:0/1657870416 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c022040 con 0x55dbd1e8d200 2024-04-01T01:43:18.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.171+0000 7f8b2181f740 1 -- 172.21.15.70:0/1657870416 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b10003d10 con 0x55dbd1e8d200 2024-04-01T01:43:18.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.171+0000 7f8af7fff640 1 -- 172.21.15.70:0/1657870416 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8b0c030890 con 0x55dbd1e8d200 2024-04-01T01:43:18.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.171+0000 7f8af7fff640 1 -- 172.21.15.70:0/1657870416 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8b0c013470 con 0x55dbd1e8d200 2024-04-01T01:43:18.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.172+0000 7f8b2181f740 10 client.18408 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:18.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.172+0000 7f8b2181f740 1 -- 172.21.15.70:0/1657870416 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55dbd1f03bf0 con 0x55dbd1e8d200 2024-04-01T01:43:18.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.173+0000 7f8af7fff640 1 -- 172.21.15.70:0/1657870416 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b10003d10 con 0x55dbd1e8d200 2024-04-01T01:43:18.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.173+0000 7f8af7fff640 1 -- 172.21.15.70:0/1657870416 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8b0c003690 con 0x55dbd1e8d200 2024-04-01T01:43:18.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.173+0000 7f8b2181f740 10 client.18408 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:18.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.173+0000 7f8b2181f740 10 client.18408 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:18.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.173+0000 7f8b2181f740 20 client.18408 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:18.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.173+0000 7f8b2181f740 1 --2- 172.21.15.70:0/1657870416 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55dbd1f045a0 0x55dbd1f069c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:18.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.173+0000 7f8b2181f740 4 client.18408 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:43:18.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.173+0000 7f8b2181f740 1 -- 172.21.15.70:0/1657870416 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dbd1df3a40 con 0x55dbd1f045a0 2024-04-01T01:43:18.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.173+0000 7f8b17fff640 1 --2- 172.21.15.70:0/1657870416 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55dbd1f045a0 0x55dbd1f069c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:18.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.175+0000 7f8af7fff640 10 client.18408 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:18.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.175+0000 7f8b17fff640 1 --2- 172.21.15.70:0/1657870416 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55dbd1f045a0 0x55dbd1f069c0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3adebda2ffa24a2 server_cookie=593d400c237d104 in_seq=0 out_seq=0 2024-04-01T01:43:18.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.175+0000 7f8af7fff640 1 -- 172.21.15.70:0/1657870416 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dbd1df3a40 con 0x55dbd1f045a0 2024-04-01T01:43:18.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.175+0000 7f8af7fff640 10 client.18408 handle_command_reply: tid=0 2024-04-01T01:43:18.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.176+0000 7f8b2181f740 1 client.18408 shutdown 2024-04-01T01:43:18.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.176+0000 7f8b2181f740 1 -- 172.21.15.70:0/1657870416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 msgr2=0x55dbd1eaeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:18.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.176+0000 7f8b2181f740 1 --2- 172.21.15.70:0/1657870416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 0x55dbd1eaeb10 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c000c00 tx=0x7f8b0c005c20 comp rx=0 tx=0).stop 2024-04-01T01:43:18.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.176+0000 7f8b2181f740 1 -- 172.21.15.70:0/1657870416 shutdown_connections 2024-04-01T01:43:18.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.176+0000 7f8b2181f740 1 --2- 172.21.15.70:0/1657870416 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55dbd1f045a0 0x55dbd1f069c0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.176+0000 7f8b2181f740 1 --2- 172.21.15.70:0/1657870416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dbd1e972e0 0x55dbd1eb0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.176+0000 7f8b2181f740 1 --2- 172.21.15.70:0/1657870416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55dbd1e92400 0x55dbd1eaf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.176+0000 7f8b2181f740 1 --2- 172.21.15.70:0/1657870416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dbd1e8d200 0x55dbd1eaeb10 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.176+0000 7f8b2181f740 1 -- 172.21.15.70:0/1657870416 >> 172.21.15.70:0/1657870416 conn(0x55dbd1e82c10 msgr2=0x55dbd1e897e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:18.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.176+0000 7f8af7fff640 0 client.18408 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:18.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.176+0000 7f8b2181f740 1 -- 172.21.15.70:0/1657870416 shutdown_connections 2024-04-01T01:43:18.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.178+0000 7f8b2181f740 1 -- 172.21.15.70:0/1657870416 wait complete. 2024-04-01T01:43:18.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.179+0000 7f8b2181f740 20 client.18408 trim_cache size 0 max 16384 2024-04-01T01:43:18.179 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:43:18.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.179+0000 7f8b1d9e7640 1 -- 172.21.15.70:0/2979752154 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8b000a2670 msgr2=0x7f8b000a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:18.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.179+0000 7f8b1d9e7640 1 --2- 172.21.15.70:0/2979752154 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8b000a2670 0x7f8b000a4b30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x55dbd1e888f0 tx=0x7f8b10004a40 comp rx=0 tx=0).stop 2024-04-01T01:43:18.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.179+0000 7f8b1d9e7640 1 -- 172.21.15.70:0/2979752154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b18108e40 msgr2=0x7f8b1819e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:18.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.179+0000 7f8b1d9e7640 1 --2- 172.21.15.70:0/2979752154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b18108e40 0x7f8b1819e400 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8b08009460 tx=0x7f8b080042f0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.180+0000 7f8b1d9e7640 1 -- 172.21.15.70:0/2979752154 shutdown_connections 2024-04-01T01:43:18.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.180+0000 7f8b1d9e7640 1 --2- 172.21.15.70:0/2979752154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b18109800 0x7f8b181a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.180+0000 7f8b1d9e7640 1 --2- 172.21.15.70:0/2979752154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b18108e40 0x7f8b1819e400 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.180+0000 7f8b1d9e7640 1 --2- 172.21.15.70:0/2979752154 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8b000a2670 0x7f8b000a4b30 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.180+0000 7f8b1d9e7640 1 --2- 172.21.15.70:0/2979752154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b18107c50 0x7f8b1819dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:18.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.180+0000 7f8b1d9e7640 1 -- 172.21.15.70:0/2979752154 >> 172.21.15.70:0/2979752154 conn(0x7f8b18076bf0 msgr2=0x7f8b1810c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:18.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.181+0000 7f8b1d9e7640 1 -- 172.21.15.70:0/2979752154 shutdown_connections 2024-04-01T01:43:18.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:18.181+0000 7f8b1d9e7640 1 -- 172.21.15.70:0/2979752154 wait complete. 2024-04-01T01:43:19.201 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:43:19.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:43:19.201 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:43:19.241 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:18 smithi070 ceph-mon[32179]: pgmap v654: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.9 MiB/s wr, 743 op/s 2024-04-01T01:43:19.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:18 smithi189 ceph-mon[35753]: pgmap v654: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.9 MiB/s wr, 743 op/s 2024-04-01T01:43:19.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:18 smithi081 ceph-mon[35142]: pgmap v654: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.9 MiB/s wr, 743 op/s 2024-04-01T01:43:19.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.381+0000 7fc1a111f640 1 -- 172.21.15.70:0/542073011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1940a4d70 msgr2=0x7fc1940a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.381+0000 7fc1a111f640 1 --2- 172.21.15.70:0/542073011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1940a4d70 0x7fc1940a5190 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc188009900 tx=0x7fc18802bc30 comp rx=0 tx=0).stop 2024-04-01T01:43:19.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.383+0000 7fc1a111f640 1 -- 172.21.15.70:0/542073011 shutdown_connections 2024-04-01T01:43:19.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.383+0000 7fc1a111f640 1 --2- 172.21.15.70:0/542073011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1940a6710 0x7fc1940a6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.383+0000 7fc1a111f640 1 --2- 172.21.15.70:0/542073011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1940a56d0 0x7fc1940a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.383+0000 7fc1a111f640 1 --2- 172.21.15.70:0/542073011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1940a4d70 0x7fc1940a5190 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.383+0000 7fc1a111f640 1 -- 172.21.15.70:0/542073011 >> 172.21.15.70:0/542073011 conn(0x7fc1940a0530 msgr2=0x7fc1940a2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:19.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.384+0000 7fc1a111f640 1 -- 172.21.15.70:0/542073011 shutdown_connections 2024-04-01T01:43:19.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.385+0000 7fc1a111f640 1 -- 172.21.15.70:0/542073011 wait complete. 2024-04-01T01:43:19.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.385+0000 7fc1a111f640 1 Processor -- start 2024-04-01T01:43:19.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.385+0000 7fc1a111f640 1 -- start start 2024-04-01T01:43:19.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.386+0000 7fc1a111f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1940a4d70 0x7fc19413ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.386+0000 7fc1a111f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1940a56d0 0x7fc19413b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.386+0000 7fc1a111f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1940a6710 0x7fc194153610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.386+0000 7fc1a111f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc1940b0730 con 0x7fc1940a56d0 2024-04-01T01:43:19.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.386+0000 7fc1a111f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc1940b05b0 con 0x7fc1940a6710 2024-04-01T01:43:19.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.386+0000 7fc1a111f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc1940b08b0 con 0x7fc1940a4d70 2024-04-01T01:43:19.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.386+0000 7fc19b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1940a56d0 0x7fc19413b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.386+0000 7fc1a091e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1940a6710 0x7fc194153610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.386+0000 7fc1a091e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1940a6710 0x7fc194153610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49418/0 (socket says 172.21.15.70:49418) 2024-04-01T01:43:19.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.386+0000 7fc1a091e640 1 -- 172.21.15.70:0/2951246987 learned_addr learned my addr 172.21.15.70:0/2951246987 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:19.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.386+0000 7fc19bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1940a4d70 0x7fc19413ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.386+0000 7fc1a091e640 1 -- 172.21.15.70:0/2951246987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1940a4d70 msgr2=0x7fc19413ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.386+0000 7fc1a091e640 1 --2- 172.21.15.70:0/2951246987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1940a4d70 0x7fc19413ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.386+0000 7fc1a091e640 1 -- 172.21.15.70:0/2951246987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1940a56d0 msgr2=0x7fc19413b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.387+0000 7fc1a091e640 1 --2- 172.21.15.70:0/2951246987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1940a56d0 0x7fc19413b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.387+0000 7fc1a091e640 1 -- 172.21.15.70:0/2951246987 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc194153b50 con 0x7fc1940a6710 2024-04-01T01:43:19.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.387+0000 7fc1a091e640 1 --2- 172.21.15.70:0/2951246987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1940a6710 0x7fc194153610 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc19c050260 tx=0x7fc19c070990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:19.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.387+0000 7fc1997fa640 1 -- 172.21.15.70:0/2951246987 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c078070 con 0x7fc1940a6710 2024-04-01T01:43:19.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.387+0000 7fc1a111f640 1 -- 172.21.15.70:0/2951246987 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc194153d80 con 0x7fc1940a6710 2024-04-01T01:43:19.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.387+0000 7fc1997fa640 1 -- 172.21.15.70:0/2951246987 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc19c069580 con 0x7fc1940a6710 2024-04-01T01:43:19.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.387+0000 7fc1997fa640 1 -- 172.21.15.70:0/2951246987 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c074a40 con 0x7fc1940a6710 2024-04-01T01:43:19.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.387+0000 7fc1a111f640 1 -- 172.21.15.70:0/2951246987 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1941541f0 con 0x7fc1940a6710 2024-04-01T01:43:19.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.390+0000 7fc1997fa640 1 -- 172.21.15.70:0/2951246987 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fc19c07b070 con 0x7fc1940a6710 2024-04-01T01:43:19.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.391+0000 7fc1997fa640 1 --2- 172.21.15.70:0/2951246987 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc18c0a24a0 0x7fc18c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.391+0000 7fc1997fa640 1 -- 172.21.15.70:0/2951246987 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc19c0fcab0 con 0x7fc1940a6710 2024-04-01T01:43:19.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.391+0000 7fc19bfff640 1 --2- 172.21.15.70:0/2951246987 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc18c0a24a0 0x7fc18c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.392+0000 7fc19bfff640 1 --2- 172.21.15.70:0/2951246987 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc18c0a24a0 0x7fc18c0a4960 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc18802c240 tx=0x7fc188005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:19.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.394+0000 7fc1a4f67740 1 Processor -- start 2024-04-01T01:43:19.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.395+0000 7fc1a4f67740 1 -- start start 2024-04-01T01:43:19.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.396+0000 7fc1a4f67740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d1bf0 0x55d33e8d2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.396+0000 7fc1a4f67740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d25e0 0x55d33e8d70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.396+0000 7fc1a4f67740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d7750 0x55d33e8d9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.396+0000 7fc1a4f67740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d33e7d30b0 con 0x55d33e8d7750 2024-04-01T01:43:19.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.396+0000 7fc1a4f67740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d33e835ea0 con 0x55d33e8d1bf0 2024-04-01T01:43:19.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.396+0000 7fc1a4f67740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d33e7dec00 con 0x55d33e8d25e0 2024-04-01T01:43:19.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.396+0000 7fc19bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d1bf0 0x55d33e8d2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.396+0000 7fc19bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d1bf0 0x55d33e8d2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49428/0 (socket says 172.21.15.70:49428) 2024-04-01T01:43:19.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.396+0000 7fc19bfff640 1 -- 172.21.15.70:0/2574743432 learned_addr learned my addr 172.21.15.70:0/2574743432 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:19.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.396+0000 7fc1a091e640 1 --2- 172.21.15.70:0/2574743432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d7750 0x55d33e8d9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.396+0000 7fc19bfff640 1 -- 172.21.15.70:0/2574743432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d25e0 msgr2=0x55d33e8d70e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:43:19.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.396+0000 7fc19bfff640 1 --2- 172.21.15.70:0/2574743432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d25e0 0x55d33e8d70e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.396+0000 7fc19bfff640 1 -- 172.21.15.70:0/2574743432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d7750 msgr2=0x55d33e8d9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:19.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.396+0000 7fc19bfff640 1 --2- 172.21.15.70:0/2574743432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d7750 0x55d33e8d9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.396+0000 7fc19bfff640 1 -- 172.21.15.70:0/2574743432 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d33e6127e0 con 0x55d33e8d1bf0 2024-04-01T01:43:19.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.396+0000 7fc1a091e640 1 --2- 172.21.15.70:0/2574743432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d7750 0x55d33e8d9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:43:19.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.397+0000 7fc19bfff640 1 --2- 172.21.15.70:0/2574743432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d1bf0 0x55d33e8d2010 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc188005ec0 tx=0x7fc18803d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e122c43bde85aa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:19.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.398+0000 7fc1867fc640 1 -- 172.21.15.70:0/2574743432 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18804e070 con 0x55d33e8d1bf0 2024-04-01T01:43:19.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.399+0000 7fc1867fc640 1 -- 172.21.15.70:0/2574743432 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc188044070 con 0x55d33e8d1bf0 2024-04-01T01:43:19.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.399+0000 7fc1867fc640 1 -- 172.21.15.70:0/2574743432 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1880035f0 con 0x55d33e8d1bf0 2024-04-01T01:43:19.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.399+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2574743432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d1bf0 msgr2=0x55d33e8d2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.400+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/2574743432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d1bf0 0x55d33e8d2010 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc188005ec0 tx=0x7fc18803d000 comp rx=0 tx=0).stop 2024-04-01T01:43:19.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.400+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2574743432 shutdown_connections 2024-04-01T01:43:19.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.400+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/2574743432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d7750 0x55d33e8d9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.400+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/2574743432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d25e0 0x55d33e8d70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.400+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/2574743432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d1bf0 0x55d33e8d2010 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.400+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2574743432 >> 172.21.15.70:0/2574743432 conn(0x55d33e8cd160 msgr2=0x55d33e8cd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:19.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.400+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2574743432 shutdown_connections 2024-04-01T01:43:19.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.400+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2574743432 wait complete. 2024-04-01T01:43:19.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.401+0000 7fc1a4f67740 1 -- start start 2024-04-01T01:43:19.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.401+0000 7fc1a4f67740 1 Processor -- start 2024-04-01T01:43:19.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.402+0000 7fc1a4f67740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d1bf0 0x55d33e8dbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.402+0000 7fc1a4f67740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d25e0 0x55d33e8dc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.402+0000 7fc19bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d1bf0 0x55d33e8dbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.402+0000 7fc19bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d1bf0 0x55d33e8dbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49430/0 (socket says 172.21.15.70:49430) 2024-04-01T01:43:19.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.402+0000 7fc19bfff640 1 -- 172.21.15.70:0/247312712 learned_addr learned my addr 172.21.15.70:0/247312712 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:19.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.402+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/247312712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d7750 0x55d33e8e3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.402+0000 7fc1a4f67740 1 -- 172.21.15.70:0/247312712 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d33e7d43d0 con 0x55d33e8d25e0 2024-04-01T01:43:19.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.402+0000 7fc1a4f67740 1 -- 172.21.15.70:0/247312712 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55d33e831f40 con 0x55d33e8d1bf0 2024-04-01T01:43:19.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.403+0000 7fc1a4f67740 1 -- 172.21.15.70:0/247312712 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d33e8da6f0 con 0x55d33e8d7750 2024-04-01T01:43:19.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.403+0000 7fc19b7fe640 1 --2- 172.21.15.70:0/247312712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d25e0 0x55d33e8dc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.403+0000 7fc1a4f67740 10 client.0 _ll_register_callbacks cb 0x55d33e7f1b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:19.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.403+0000 7fc19bfff640 1 -- 172.21.15.70:0/247312712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d7750 msgr2=0x55d33e8e3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:19.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.403+0000 7fc19bfff640 1 --2- 172.21.15.70:0/247312712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d7750 0x55d33e8e3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.403+0000 7fc19bfff640 1 -- 172.21.15.70:0/247312712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d25e0 msgr2=0x55d33e8dc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:19.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.403+0000 7fc19bfff640 1 --2- 172.21.15.70:0/247312712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d25e0 0x55d33e8dc510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.403+0000 7fc19bfff640 1 -- 172.21.15.70:0/247312712 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d33e672f70 con 0x55d33e8d1bf0 2024-04-01T01:43:19.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.403+0000 7fc19bfff640 1 --2- 172.21.15.70:0/247312712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d1bf0 0x55d33e8dbfd0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc18802c170 tx=0x7fc188004f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db1aa48277e5c675 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:19.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.404+0000 7fc17b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:19.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.404+0000 7fc17b7fe640 1 -- 172.21.15.70:0/247312712 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18804e040 con 0x55d33e8d1bf0 2024-04-01T01:43:19.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.404+0000 7fc17b7fe640 1 -- 172.21.15.70:0/247312712 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc188005550 con 0x55d33e8d1bf0 2024-04-01T01:43:19.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.404+0000 7fc1a4f67740 1 -- 172.21.15.70:0/247312712 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc188044070 con 0x55d33e8d1bf0 2024-04-01T01:43:19.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.404+0000 7fc17b7fe640 1 -- 172.21.15.70:0/247312712 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc1880035f0 con 0x55d33e8d1bf0 2024-04-01T01:43:19.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.404+0000 7fc1a4f67740 10 client.28424 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:19.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.404+0000 7fc1a4f67740 1 -- 172.21.15.70:0/247312712 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d33e50a170 con 0x55d33e8d1bf0 2024-04-01T01:43:19.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.405+0000 7fc17b7fe640 1 -- 172.21.15.70:0/247312712 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc18805b070 con 0x55d33e8d1bf0 2024-04-01T01:43:19.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.406+0000 7fc17b7fe640 1 -- 172.21.15.70:0/247312712 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc188049070 con 0x55d33e8d1bf0 2024-04-01T01:43:19.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.407+0000 7fc1a4f67740 10 client.28424 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:19.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.407+0000 7fc1a4f67740 10 client.28424 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:19.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.407+0000 7fc1a4f67740 20 client.28424 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:19.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.407+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/247312712 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d33e8e7690 0x55d33e8e9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.407+0000 7fc1a4f67740 4 client.28424 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:19.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.407+0000 7fc1a4f67740 1 -- 172.21.15.70:0/247312712 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc188044070 con 0x55d33e8e7690 2024-04-01T01:43:19.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.407+0000 7fc19b7fe640 1 --2- 172.21.15.70:0/247312712 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d33e8e7690 0x55d33e8e9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.408+0000 7fc19b7fe640 1 --2- 172.21.15.70:0/247312712 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d33e8e7690 0x55d33e8e9a80 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a14ab16cd864817f server_cookie=3dad40b97c3464af in_seq=0 out_seq=0 2024-04-01T01:43:19.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.408+0000 7fc17b7fe640 10 client.28424 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:19.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.410+0000 7fc17b7fe640 1 -- 172.21.15.70:0/247312712 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fc188044070 con 0x55d33e8e7690 2024-04-01T01:43:19.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.410+0000 7fc17b7fe640 10 client.28424 handle_command_reply: tid=0 2024-04-01T01:43:19.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.410+0000 7fc1a4f67740 1 client.28424 shutdown 2024-04-01T01:43:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.416+0000 7fc1a4f67740 1 -- 172.21.15.70:0/247312712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d1bf0 msgr2=0x55d33e8dbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.416+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/247312712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d1bf0 0x55d33e8dbfd0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc18802c170 tx=0x7fc188004f30 comp rx=0 tx=0).stop 2024-04-01T01:43:19.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.423+0000 7fc1a4f67740 1 -- 172.21.15.70:0/247312712 shutdown_connections 2024-04-01T01:43:19.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.423+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/247312712 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d33e8e7690 0x55d33e8e9a80 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.423+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/247312712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d7750 0x55d33e8e3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.424+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/247312712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d25e0 0x55d33e8dc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.424+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/247312712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d1bf0 0x55d33e8dbfd0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.424+0000 7fc17b7fe640 0 client.28424 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:19.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.424+0000 7fc1a4f67740 1 -- 172.21.15.70:0/247312712 >> 172.21.15.70:0/247312712 conn(0x55d33e8cd160 msgr2=0x55d33e8cc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:19.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.425+0000 7fc1a4f67740 1 -- 172.21.15.70:0/247312712 shutdown_connections 2024-04-01T01:43:19.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.425+0000 7fc1a4f67740 1 -- 172.21.15.70:0/247312712 wait complete. 2024-04-01T01:43:19.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.426+0000 7fc1a4f67740 20 client.28424 trim_cache size 0 max 16384 2024-04-01T01:43:19.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.468+0000 7fc1a4f67740 1 Processor -- start 2024-04-01T01:43:19.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.468+0000 7fc1a4f67740 1 -- start start 2024-04-01T01:43:19.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.468+0000 7fc1a4f67740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d1bf0 0x55d33e8dc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.468+0000 7fc1a4f67740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d25e0 0x55d33e8e34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.469+0000 7fc19b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d25e0 0x55d33e8e34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.469+0000 7fc19bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d1bf0 0x55d33e8dc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.469+0000 7fc19bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d1bf0 0x55d33e8dc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46788/0 (socket says 172.21.15.70:46788) 2024-04-01T01:43:19.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.469+0000 7fc19bfff640 1 -- 172.21.15.70:0/2874913765 learned_addr learned my addr 172.21.15.70:0/2874913765 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:19.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.469+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/2874913765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d7750 0x55d33e8cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.469+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2874913765 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc19415e8d0 con 0x55d33e8d25e0 2024-04-01T01:43:19.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.469+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2874913765 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc194160410 con 0x55d33e8d7750 2024-04-01T01:43:19.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.469+0000 7fc1a091e640 1 --2- 172.21.15.70:0/2874913765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d7750 0x55d33e8cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.469+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2874913765 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc19415efa0 con 0x55d33e8d1bf0 2024-04-01T01:43:19.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.470+0000 7fc19b7fe640 1 -- 172.21.15.70:0/2874913765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d1bf0 msgr2=0x55d33e8dc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:19.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.470+0000 7fc19b7fe640 1 --2- 172.21.15.70:0/2874913765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d1bf0 0x55d33e8dc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.470+0000 7fc19b7fe640 1 -- 172.21.15.70:0/2874913765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d7750 msgr2=0x55d33e8cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:19.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.470+0000 7fc19b7fe640 1 --2- 172.21.15.70:0/2874913765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d7750 0x55d33e8cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.470+0000 7fc19b7fe640 1 -- 172.21.15.70:0/2874913765 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d33e8c7440 con 0x55d33e8d25e0 2024-04-01T01:43:19.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.471+0000 7fc19b7fe640 1 --2- 172.21.15.70:0/2874913765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d25e0 0x55d33e8e34e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc19000b3c0 tx=0x7fc190005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da7a4d893d0b36d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:19.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.471+0000 7fc185ffb640 1 -- 172.21.15.70:0/2874913765 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc190003c40 con 0x55d33e8d25e0 2024-04-01T01:43:19.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.472+0000 7fc185ffb640 1 -- 172.21.15.70:0/2874913765 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc190021030 con 0x55d33e8d25e0 2024-04-01T01:43:19.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.472+0000 7fc185ffb640 1 -- 172.21.15.70:0/2874913765 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1900253d0 con 0x55d33e8d25e0 2024-04-01T01:43:19.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.473+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2874913765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d25e0 msgr2=0x55d33e8e34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.473+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/2874913765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d25e0 0x55d33e8e34e0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fc19000b3c0 tx=0x7fc190005f90 comp rx=0 tx=0).stop 2024-04-01T01:43:19.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.473+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2874913765 shutdown_connections 2024-04-01T01:43:19.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.473+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/2874913765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d7750 0x55d33e8cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.474+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/2874913765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d25e0 0x55d33e8e34e0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.474+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/2874913765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d1bf0 0x55d33e8dc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.474+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2874913765 >> 172.21.15.70:0/2874913765 conn(0x55d33e8cd160 msgr2=0x55d33e8dbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:19.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.474+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2874913765 shutdown_connections 2024-04-01T01:43:19.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.474+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2874913765 wait complete. 2024-04-01T01:43:19.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.475+0000 7fc1a4f67740 1 -- start start 2024-04-01T01:43:19.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.475+0000 7fc1a4f67740 1 Processor -- start 2024-04-01T01:43:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc1a4f67740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d1bf0 0x55d33e8e1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc1a4f67740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d25e0 0x55d33e8e16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc1a4f67740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d7750 0x55d33e8e2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc1a4f67740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc19415fd40 con 0x55d33e8d25e0 2024-04-01T01:43:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc1a4f67740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc19415f670 con 0x55d33e8d7750 2024-04-01T01:43:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc1a4f67740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc19415c8b0 con 0x55d33e8d1bf0 2024-04-01T01:43:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc1a4f67740 10 client.0 _ll_register_callbacks cb 0x7fc18802ccc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc19bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d1bf0 0x55d33e8e1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc19bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d1bf0 0x55d33e8e1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46794/0 (socket says 172.21.15.70:46794) 2024-04-01T01:43:19.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc19bfff640 1 -- 172.21.15.70:0/2950862649 learned_addr learned my addr 172.21.15.70:0/2950862649 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc1a091e640 1 --2- 172.21.15.70:0/2950862649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d7750 0x55d33e8e2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc19b7fe640 1 --2- 172.21.15.70:0/2950862649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d25e0 0x55d33e8e16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc1a091e640 1 -- 172.21.15.70:0/2950862649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d1bf0 msgr2=0x55d33e8e1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc1a091e640 1 --2- 172.21.15.70:0/2950862649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d1bf0 0x55d33e8e1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc1a091e640 1 -- 172.21.15.70:0/2950862649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d25e0 msgr2=0x55d33e8e16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc1a091e640 1 --2- 172.21.15.70:0/2950862649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d25e0 0x55d33e8e16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.476+0000 7fc1a091e640 1 -- 172.21.15.70:0/2950862649 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d33e8e2c60 con 0x55d33e8d7750 2024-04-01T01:43:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.477+0000 7fc1a091e640 1 --2- 172.21.15.70:0/2950862649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d7750 0x55d33e8e2670 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc19c0f94e0 tx=0x7fc19c0f9510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf2283046907fe0c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.477+0000 7fc17b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.477+0000 7fc17b7fe640 1 -- 172.21.15.70:0/2950862649 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c0fb440 con 0x55d33e8d7750 2024-04-01T01:43:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.477+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2950862649 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc190021030 con 0x55d33e8d7750 2024-04-01T01:43:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.477+0000 7fc17b7fe640 1 -- 172.21.15.70:0/2950862649 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc19c0faae0 con 0x55d33e8d7750 2024-04-01T01:43:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.477+0000 7fc17b7fe640 1 -- 172.21.15.70:0/2950862649 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc19c082070 con 0x55d33e8d7750 2024-04-01T01:43:19.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.478+0000 7fc1a4f67740 10 client.28430 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.478+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2950862649 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d33e94de90 con 0x55d33e8d7750 2024-04-01T01:43:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.478+0000 7fc17b7fe640 1 -- 172.21.15.70:0/2950862649 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc190021030 con 0x55d33e8d7750 2024-04-01T01:43:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.478+0000 7fc17b7fe640 1 -- 172.21.15.70:0/2950862649 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc19c118020 con 0x55d33e8d7750 2024-04-01T01:43:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.478+0000 7fc1a4f67740 10 client.28430 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.478+0000 7fc1a4f67740 10 client.28430 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.478+0000 7fc1a4f67740 20 client.28430 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.478+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/2950862649 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d33e8e7690 0x55d33e950820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.478+0000 7fc1a4f67740 4 client.28430 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:43:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.478+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2950862649 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d33e8433e0 con 0x55d33e8e7690 2024-04-01T01:43:19.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.478+0000 7fc19bfff640 1 --2- 172.21.15.70:0/2950862649 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d33e8e7690 0x55d33e950820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.479+0000 7fc17b7fe640 10 client.28430 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:19.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.479+0000 7fc19bfff640 1 --2- 172.21.15.70:0/2950862649 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d33e8e7690 0x55d33e950820 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10ae0537def97cf3 server_cookie=93451d99f022bf1f in_seq=0 out_seq=0 2024-04-01T01:43:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.483+0000 7fc17b7fe640 1 -- 172.21.15.70:0/2950862649 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d33e8433e0 con 0x55d33e8e7690 2024-04-01T01:43:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.483+0000 7fc17b7fe640 10 client.28430 handle_command_reply: tid=0 2024-04-01T01:43:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.484+0000 7fc1a4f67740 1 client.28430 shutdown 2024-04-01T01:43:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.484+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2950862649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d7750 msgr2=0x55d33e8e2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.484+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/2950862649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d7750 0x55d33e8e2670 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc19c0f94e0 tx=0x7fc19c0f9510 comp rx=0 tx=0).stop 2024-04-01T01:43:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.484+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2950862649 shutdown_connections 2024-04-01T01:43:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.484+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/2950862649 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55d33e8e7690 0x55d33e950820 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.484+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/2950862649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d33e8d7750 0x55d33e8e2670 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.484+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/2950862649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d33e8d25e0 0x55d33e8e16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.484+0000 7fc1a4f67740 1 --2- 172.21.15.70:0/2950862649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d33e8d1bf0 0x55d33e8e1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.485+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2950862649 >> 172.21.15.70:0/2950862649 conn(0x55d33e8cd160 msgr2=0x55d33e8cba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.485+0000 7fc17b7fe640 0 client.28430 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.485+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2950862649 shutdown_connections 2024-04-01T01:43:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.485+0000 7fc1a4f67740 1 -- 172.21.15.70:0/2950862649 wait complete. 2024-04-01T01:43:19.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.485+0000 7fc1a4f67740 20 client.28430 trim_cache size 0 max 16384 2024-04-01T01:43:19.486 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:43:19.486 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:43:19.486 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "cefb5ce3-c6c5-4378-b90f-0772927086a3", 2024-04-01T01:43:19.487 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:43:19.487 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:43:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.486+0000 7fc1a111f640 1 -- 172.21.15.70:0/2951246987 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc18c0a24a0 msgr2=0x7fc18c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.486+0000 7fc1a111f640 1 --2- 172.21.15.70:0/2951246987 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc18c0a24a0 0x7fc18c0a4960 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc18802c240 tx=0x7fc188005fd0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.486+0000 7fc1a111f640 1 -- 172.21.15.70:0/2951246987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1940a6710 msgr2=0x7fc194153610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.486+0000 7fc1a111f640 1 --2- 172.21.15.70:0/2951246987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1940a6710 0x7fc194153610 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc19c050260 tx=0x7fc19c070990 comp rx=0 tx=0).stop 2024-04-01T01:43:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.486+0000 7fc1a111f640 1 -- 172.21.15.70:0/2951246987 shutdown_connections 2024-04-01T01:43:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.486+0000 7fc1a111f640 1 --2- 172.21.15.70:0/2951246987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1940a6710 0x7fc194153610 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.486+0000 7fc1a111f640 1 --2- 172.21.15.70:0/2951246987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1940a56d0 0x7fc19413b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.486+0000 7fc1a111f640 1 --2- 172.21.15.70:0/2951246987 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc18c0a24a0 0x7fc18c0a4960 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.486+0000 7fc1a111f640 1 --2- 172.21.15.70:0/2951246987 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1940a4d70 0x7fc19413ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.486+0000 7fc1a111f640 1 -- 172.21.15.70:0/2951246987 >> 172.21.15.70:0/2951246987 conn(0x7fc1940a0530 msgr2=0x7fc1940a2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.487+0000 7fc1a111f640 1 -- 172.21.15.70:0/2951246987 shutdown_connections 2024-04-01T01:43:19.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.487+0000 7fc1a111f640 1 -- 172.21.15.70:0/2951246987 wait complete. 2024-04-01T01:43:19.505 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:43:19.505 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:43:19.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.728+0000 7f12d646b640 1 -- 172.21.15.70:0/905127968 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d010b0c0 msgr2=0x7f12d010d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.728+0000 7f12d646b640 1 --2- 172.21.15.70:0/905127968 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d010b0c0 0x7f12d010d530 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f12c0009900 tx=0x7f12c002bc50 comp rx=0 tx=0).stop 2024-04-01T01:43:19.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.732+0000 7f12d646b640 1 -- 172.21.15.70:0/905127968 shutdown_connections 2024-04-01T01:43:19.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.732+0000 7f12d646b640 1 --2- 172.21.15.70:0/905127968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d010da70 0x7f12d010def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.732+0000 7f12d646b640 1 --2- 172.21.15.70:0/905127968 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d010b0c0 0x7f12d010d530 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.732+0000 7f12d646b640 1 --2- 172.21.15.70:0/905127968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0108750 0x7f12d010ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.732+0000 7f12d646b640 1 -- 172.21.15.70:0/905127968 >> 172.21.15.70:0/905127968 conn(0x7f12d006e5b0 msgr2=0x7f12d00709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:19.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.734+0000 7f12d646b640 1 -- 172.21.15.70:0/905127968 shutdown_connections 2024-04-01T01:43:19.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.735+0000 7f12d646b640 1 -- 172.21.15.70:0/905127968 wait complete. 2024-04-01T01:43:19.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.735+0000 7f12d646b640 1 Processor -- start 2024-04-01T01:43:19.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.736+0000 7f12d646b640 1 -- start start 2024-04-01T01:43:19.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.736+0000 7f12d646b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0108750 0x7f12d01a6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.736+0000 7f12d646b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d010b0c0 0x7f12d01a6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.736+0000 7f12d646b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d010da70 0x7f12d01ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.736+0000 7f12d646b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f12d011c140 con 0x7f12d0108750 2024-04-01T01:43:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.736+0000 7f12d646b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f12d011bfc0 con 0x7f12d010da70 2024-04-01T01:43:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.736+0000 7f12d646b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f12d011c2c0 con 0x7f12d010b0c0 2024-04-01T01:43:19.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.737+0000 7f12cffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0108750 0x7f12d01a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.737+0000 7f12cffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0108750 0x7f12d01a6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34392/0 (socket says 172.21.15.70:34392) 2024-04-01T01:43:19.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.737+0000 7f12cffff640 1 -- 172.21.15.70:0/665507803 learned_addr learned my addr 172.21.15.70:0/665507803 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:19.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.737+0000 7f12cf7fe640 1 --2- 172.21.15.70:0/665507803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d010b0c0 0x7f12d01a6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.737+0000 7f12d4a08640 1 --2- 172.21.15.70:0/665507803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d010da70 0x7f12d01ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.737+0000 7f12cffff640 1 -- 172.21.15.70:0/665507803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d010b0c0 msgr2=0x7f12d01a6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.737+0000 7f12cffff640 1 --2- 172.21.15.70:0/665507803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d010b0c0 0x7f12d01a6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.737+0000 7f12cffff640 1 -- 172.21.15.70:0/665507803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d010da70 msgr2=0x7f12d01ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.737+0000 7f12cffff640 1 --2- 172.21.15.70:0/665507803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d010da70 0x7f12d01ade20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.737+0000 7f12cffff640 1 -- 172.21.15.70:0/665507803 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d01ae490 con 0x7f12d0108750 2024-04-01T01:43:19.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.737+0000 7f12d4a08640 1 --2- 172.21.15.70:0/665507803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d010da70 0x7f12d01ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:43:19.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.738+0000 7f12cffff640 1 --2- 172.21.15.70:0/665507803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0108750 0x7f12d01a6870 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f12c800ba50 tx=0x7f12c800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:19.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.738+0000 7f12cd7fa640 1 -- 172.21.15.70:0/665507803 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c8019070 con 0x7f12d0108750 2024-04-01T01:43:19.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.738+0000 7f12d646b640 1 -- 172.21.15.70:0/665507803 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12d01ae780 con 0x7f12d0108750 2024-04-01T01:43:19.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.738+0000 7f12cd7fa640 1 -- 172.21.15.70:0/665507803 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12c800cbe0 con 0x7f12d0108750 2024-04-01T01:43:19.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.738+0000 7f12d646b640 1 -- 172.21.15.70:0/665507803 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12d01aed20 con 0x7f12d0108750 2024-04-01T01:43:19.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.738+0000 7f12cd7fa640 1 -- 172.21.15.70:0/665507803 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c80075d0 con 0x7f12d0108750 2024-04-01T01:43:19.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.742+0000 7f12d9046740 1 Processor -- start 2024-04-01T01:43:19.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.742+0000 7f12d9046740 1 -- start start 2024-04-01T01:43:19.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.742+0000 7f12d9046740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119d36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.742+0000 7f12d9046740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d3ca0 0x5608119d87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.742+0000 7f12d9046740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d8e10 0x5608119db2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.742+0000 7f12d9046740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560811938180 con 0x5608119d8e10 2024-04-01T01:43:19.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.742+0000 7f12d9046740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5608118e0b70 con 0x5608119d3ca0 2024-04-01T01:43:19.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.742+0000 7f12d9046740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560811934220 con 0x5608119d32b0 2024-04-01T01:43:19.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.743+0000 7f12d4a08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119d36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.743+0000 7f12d4a08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119d36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46816/0 (socket says 172.21.15.70:46816) 2024-04-01T01:43:19.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.743+0000 7f12d4a08640 1 -- 172.21.15.70:0/2681324767 learned_addr learned my addr 172.21.15.70:0/2681324767 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:19.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.743+0000 7f12cffff640 1 --2- 172.21.15.70:0/2681324767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d3ca0 0x5608119d87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.743+0000 7f12d4a08640 1 -- 172.21.15.70:0/2681324767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d3ca0 msgr2=0x5608119d87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:19.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.743+0000 7f12d4a08640 1 --2- 172.21.15.70:0/2681324767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d3ca0 0x5608119d87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.743+0000 7f12d4a08640 1 -- 172.21.15.70:0/2681324767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d8e10 msgr2=0x5608119db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:19.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.743+0000 7f12d4a08640 1 --2- 172.21.15.70:0/2681324767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d8e10 0x5608119db2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.743+0000 7f12d4a08640 1 -- 172.21.15.70:0/2681324767 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608118f47c0 con 0x5608119d32b0 2024-04-01T01:43:19.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.744+0000 7f12d4a08640 1 --2- 172.21.15.70:0/2681324767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119d36d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009c60 tx=0x7f12c40117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91ff02ea3236f6f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:19.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.743+0000 7f12cd7fa640 1 -- 172.21.15.70:0/665507803 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f12c801c070 con 0x7f12d0108750 2024-04-01T01:43:19.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.746+0000 7f12cd7fa640 1 --2- 172.21.15.70:0/665507803 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12b80a25f0 0x7f12b80a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.746+0000 7f12cffff640 1 --2- 172.21.15.70:0/665507803 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12b80a25f0 0x7f12b80a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.746+0000 7f12cd7fa640 1 -- 172.21.15.70:0/665507803 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f12c80ce110 con 0x7f12d0108750 2024-04-01T01:43:19.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.746+0000 7f12cffff640 1 --2- 172.21.15.70:0/665507803 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12b80a25f0 0x7f12b80a4ab0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f12c809a550 tx=0x7f12c80c50d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:19.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.747+0000 7f12b67fc640 1 -- 172.21.15.70:0/2681324767 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c4012450 con 0x5608119d32b0 2024-04-01T01:43:19.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.747+0000 7f12b67fc640 1 -- 172.21.15.70:0/2681324767 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12c4012aa0 con 0x5608119d32b0 2024-04-01T01:43:19.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.747+0000 7f12b67fc640 1 -- 172.21.15.70:0/2681324767 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c400faa0 con 0x5608119d32b0 2024-04-01T01:43:19.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.748+0000 7f12d9046740 1 -- 172.21.15.70:0/2681324767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 msgr2=0x5608119d36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.748+0000 7f12d9046740 1 --2- 172.21.15.70:0/2681324767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119d36d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009c60 tx=0x7f12c40117b0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.749+0000 7f12d9046740 1 -- 172.21.15.70:0/2681324767 shutdown_connections 2024-04-01T01:43:19.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.749+0000 7f12d9046740 1 --2- 172.21.15.70:0/2681324767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d8e10 0x5608119db2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.749+0000 7f12d9046740 1 --2- 172.21.15.70:0/2681324767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d3ca0 0x5608119d87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.749+0000 7f12d9046740 1 --2- 172.21.15.70:0/2681324767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119d36d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.749+0000 7f12d9046740 1 -- 172.21.15.70:0/2681324767 >> 172.21.15.70:0/2681324767 conn(0x5608119ce820 msgr2=0x5608119cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:19.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.753+0000 7f12d9046740 1 -- 172.21.15.70:0/2681324767 shutdown_connections 2024-04-01T01:43:19.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.754+0000 7f12d9046740 1 -- 172.21.15.70:0/2681324767 wait complete. 2024-04-01T01:43:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.755+0000 7f12d9046740 1 -- start start 2024-04-01T01:43:19.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.755+0000 7f12d9046740 1 Processor -- start 2024-04-01T01:43:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.755+0000 7f12d9046740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119dd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.755+0000 7f12d9046740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d3ca0 0x5608119ddd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.755+0000 7f12d9046740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d8e10 0x5608119e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.755+0000 7f12d9046740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5608119dbdc0 con 0x5608119d8e10 2024-04-01T01:43:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.755+0000 7f12d9046740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5608118d64a0 con 0x5608119d3ca0 2024-04-01T01:43:19.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.755+0000 7f12d9046740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5608119dbf40 con 0x5608119d32b0 2024-04-01T01:43:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.756+0000 7f12d9046740 10 client.0 _ll_register_callbacks cb 0x5608118d9d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.756+0000 7f12d4a08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119dd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.756+0000 7f12d4a08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119dd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46830/0 (socket says 172.21.15.70:46830) 2024-04-01T01:43:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.756+0000 7f12d4a08640 1 -- 172.21.15.70:0/4176695643 learned_addr learned my addr 172.21.15.70:0/4176695643 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:19.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.756+0000 7f12d4a08640 1 --2- 172.21.15.70:0/4176695643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d8e10 0x5608119e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.756+0000 7f12cf7fe640 1 --2- 172.21.15.70:0/4176695643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d3ca0 0x5608119ddd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.757+0000 7f12cf7fe640 1 -- 172.21.15.70:0/4176695643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 msgr2=0x5608119dd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.757+0000 7f12cf7fe640 1 --2- 172.21.15.70:0/4176695643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119dd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.757+0000 7f12cf7fe640 1 -- 172.21.15.70:0/4176695643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d8e10 msgr2=0x5608119e4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.757+0000 7f12cf7fe640 1 --2- 172.21.15.70:0/4176695643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d8e10 0x5608119e4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.757+0000 7f12cf7fe640 1 -- 172.21.15.70:0/4176695643 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608117167d0 con 0x5608119d3ca0 2024-04-01T01:43:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.757+0000 7f12d4a08640 1 --2- 172.21.15.70:0/4176695643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d8e10 0x5608119e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:43:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.757+0000 7f12cf7fe640 1 --2- 172.21.15.70:0/4176695643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d3ca0 0x5608119ddd60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f12c0009850 tx=0x7f12c0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a757e9095babb071 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.757+0000 7f12af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.757+0000 7f12af7fe640 1 -- 172.21.15.70:0/4176695643 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c0045020 con 0x5608119d3ca0 2024-04-01T01:43:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.757+0000 7f12af7fe640 1 -- 172.21.15.70:0/4176695643 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f12c0005250 con 0x5608119d3ca0 2024-04-01T01:43:19.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.758+0000 7f12d9046740 1 -- 172.21.15.70:0/4176695643 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12c4012aa0 con 0x5608119d3ca0 2024-04-01T01:43:19.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.758+0000 7f12af7fe640 1 -- 172.21.15.70:0/4176695643 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f12c0050050 con 0x5608119d3ca0 2024-04-01T01:43:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.758+0000 7f12d9046740 10 client.28439 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.758+0000 7f12d9046740 1 -- 172.21.15.70:0/4176695643 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56081164c3f0 con 0x5608119d3ca0 2024-04-01T01:43:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.758+0000 7f12af7fe640 1 -- 172.21.15.70:0/4176695643 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12c4012aa0 con 0x5608119d3ca0 2024-04-01T01:43:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.759+0000 7f12af7fe640 1 -- 172.21.15.70:0/4176695643 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f12c00035f0 con 0x5608119d3ca0 2024-04-01T01:43:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.759+0000 7f12d9046740 10 client.28439 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.759+0000 7f12d9046740 10 client.28439 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.759+0000 7f12d9046740 20 client.28439 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.759+0000 7f12d9046740 1 --2- 172.21.15.70:0/4176695643 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5608119e9ab0 0x5608119ebed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.759+0000 7f12d9046740 4 client.28439 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:19.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.759+0000 7f12d9046740 1 -- 172.21.15.70:0/4176695643 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608119456c0 con 0x5608119e9ab0 2024-04-01T01:43:19.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.760+0000 7f12d4a08640 1 --2- 172.21.15.70:0/4176695643 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5608119e9ab0 0x5608119ebed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.761+0000 7f12d4a08640 1 --2- 172.21.15.70:0/4176695643 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5608119e9ab0 0x5608119ebed0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bae494a2fe252d03 server_cookie=e00bcee6ce57bd in_seq=0 out_seq=0 2024-04-01T01:43:19.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.761+0000 7f12af7fe640 10 client.28439 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:19.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.764+0000 7f12af7fe640 1 -- 172.21.15.70:0/4176695643 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5608119456c0 con 0x5608119e9ab0 2024-04-01T01:43:19.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.764+0000 7f12af7fe640 10 client.28439 handle_command_reply: tid=0 2024-04-01T01:43:19.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.765+0000 7f12d9046740 1 client.28439 shutdown 2024-04-01T01:43:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.770+0000 7f12d9046740 1 -- 172.21.15.70:0/4176695643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d3ca0 msgr2=0x5608119ddd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.770+0000 7f12d9046740 1 --2- 172.21.15.70:0/4176695643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d3ca0 0x5608119ddd60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f12c0009850 tx=0x7f12c0004270 comp rx=0 tx=0).stop 2024-04-01T01:43:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.770+0000 7f12d9046740 1 -- 172.21.15.70:0/4176695643 shutdown_connections 2024-04-01T01:43:19.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.770+0000 7f12d9046740 1 --2- 172.21.15.70:0/4176695643 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5608119e9ab0 0x5608119ebed0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.770+0000 7f12d9046740 1 --2- 172.21.15.70:0/4176695643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d8e10 0x5608119e4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.770+0000 7f12d9046740 1 --2- 172.21.15.70:0/4176695643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d3ca0 0x5608119ddd60 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.770+0000 7f12d9046740 1 --2- 172.21.15.70:0/4176695643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119dd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.770+0000 7f12d9046740 1 -- 172.21.15.70:0/4176695643 >> 172.21.15.70:0/4176695643 conn(0x5608119ce820 msgr2=0x5608119cdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.770+0000 7f12af7fe640 0 client.28439 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.771+0000 7f12d9046740 1 -- 172.21.15.70:0/4176695643 shutdown_connections 2024-04-01T01:43:19.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.771+0000 7f12d9046740 1 -- 172.21.15.70:0/4176695643 wait complete. 2024-04-01T01:43:19.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.772+0000 7f12d9046740 20 client.28439 trim_cache size 0 max 16384 2024-04-01T01:43:19.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.814+0000 7f12d9046740 1 Processor -- start 2024-04-01T01:43:19.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.814+0000 7f12d9046740 1 -- start start 2024-04-01T01:43:19.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.814+0000 7f12d9046740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.815+0000 7f12d9046740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d3ca0 0x5608119e4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.815+0000 7f12d4a08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.815+0000 7f12d4a08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46832/0 (socket says 172.21.15.70:46832) 2024-04-01T01:43:19.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.815+0000 7f12d4a08640 1 -- 172.21.15.70:0/2039940506 learned_addr learned my addr 172.21.15.70:0/2039940506 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:19.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.815+0000 7f12d9046740 1 --2- 172.21.15.70:0/2039940506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d8e10 0x5608119dd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.815+0000 7f12d9046740 1 -- 172.21.15.70:0/2039940506 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f12d01b9320 con 0x5608119d3ca0 2024-04-01T01:43:19.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.815+0000 7f12d9046740 1 -- 172.21.15.70:0/2039940506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f12d01bae60 con 0x5608119d8e10 2024-04-01T01:43:19.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.815+0000 7f12d9046740 1 -- 172.21.15.70:0/2039940506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f12d01b99f0 con 0x5608119d32b0 2024-04-01T01:43:19.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.815+0000 7f12cf7fe640 1 --2- 172.21.15.70:0/2039940506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d3ca0 0x5608119e4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.816+0000 7f12d4a08640 1 --2- 172.21.15.70:0/2039940506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d8e10 0x5608119dd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.816+0000 7f12d4a08640 1 -- 172.21.15.70:0/2039940506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d8e10 msgr2=0x5608119dd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:19.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.816+0000 7f12d4a08640 1 --2- 172.21.15.70:0/2039940506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d8e10 0x5608119dd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.816+0000 7f12d4a08640 1 -- 172.21.15.70:0/2039940506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d3ca0 msgr2=0x5608119e4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:19.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.816+0000 7f12d4a08640 1 --2- 172.21.15.70:0/2039940506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d3ca0 0x5608119e4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.816+0000 7f12d4a08640 1 -- 172.21.15.70:0/2039940506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56081160c0f0 con 0x5608119d32b0 2024-04-01T01:43:19.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.817+0000 7f12d4a08640 1 --2- 172.21.15.70:0/2039940506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119cec00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009830 tx=0x7f12c40117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12005b2462be5a2b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:19.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.817+0000 7f12b5ffb640 1 -- 172.21.15.70:0/2039940506 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c4005230 con 0x5608119d32b0 2024-04-01T01:43:19.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.817+0000 7f12b5ffb640 1 -- 172.21.15.70:0/2039940506 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12c4036070 con 0x5608119d32b0 2024-04-01T01:43:19.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.817+0000 7f12b5ffb640 1 -- 172.21.15.70:0/2039940506 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c4005230 con 0x5608119d32b0 2024-04-01T01:43:19.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.818+0000 7f12d9046740 1 -- 172.21.15.70:0/2039940506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 msgr2=0x5608119cec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.818+0000 7f12d9046740 1 --2- 172.21.15.70:0/2039940506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119cec00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f12c4009830 tx=0x7f12c40117b0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.818+0000 7f12d9046740 1 -- 172.21.15.70:0/2039940506 shutdown_connections 2024-04-01T01:43:19.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.818+0000 7f12d9046740 1 --2- 172.21.15.70:0/2039940506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d8e10 0x5608119dd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.818+0000 7f12d9046740 1 --2- 172.21.15.70:0/2039940506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d3ca0 0x5608119e4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.818+0000 7f12d9046740 1 --2- 172.21.15.70:0/2039940506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119cec00 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.818+0000 7f12d9046740 1 -- 172.21.15.70:0/2039940506 >> 172.21.15.70:0/2039940506 conn(0x5608119ce820 msgr2=0x5608119dc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:19.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.818+0000 7f12d9046740 1 -- 172.21.15.70:0/2039940506 shutdown_connections 2024-04-01T01:43:19.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.818+0000 7f12d9046740 1 -- 172.21.15.70:0/2039940506 wait complete. 2024-04-01T01:43:19.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.819+0000 7f12d9046740 1 -- start start 2024-04-01T01:43:19.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.819+0000 7f12d9046740 1 Processor -- start 2024-04-01T01:43:19.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.820+0000 7f12d9046740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119e4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.820+0000 7f12d9046740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d3ca0 0x5608119e4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.820+0000 7f12d9046740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d8e10 0x5608119e14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.820+0000 7f12d9046740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f12d01ba790 con 0x5608119d3ca0 2024-04-01T01:43:19.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.820+0000 7f12d9046740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f12d01ba0c0 con 0x5608119d8e10 2024-04-01T01:43:19.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.820+0000 7f12d9046740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f12d01b7300 con 0x5608119d32b0 2024-04-01T01:43:19.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.820+0000 7f12d9046740 10 client.0 _ll_register_callbacks cb 0x7f12c0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:19.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.820+0000 7f12d4a08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119e4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.820+0000 7f12cf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d3ca0 0x5608119e4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.820+0000 7f12cf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d3ca0 0x5608119e4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34426/0 (socket says 172.21.15.70:34426) 2024-04-01T01:43:19.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.820+0000 7f12cf7fe640 1 -- 172.21.15.70:0/2724084949 learned_addr learned my addr 172.21.15.70:0/2724084949 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:19.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.820+0000 7f12d4a08640 1 --2- 172.21.15.70:0/2724084949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d8e10 0x5608119e14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.820+0000 7f12d4a08640 1 -- 172.21.15.70:0/2724084949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d8e10 msgr2=0x5608119e14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:19.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.820+0000 7f12d4a08640 1 --2- 172.21.15.70:0/2724084949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d8e10 0x5608119e14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.820+0000 7f12d4a08640 1 -- 172.21.15.70:0/2724084949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d3ca0 msgr2=0x5608119e4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:19.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.821+0000 7f12d4a08640 1 --2- 172.21.15.70:0/2724084949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d3ca0 0x5608119e4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.821+0000 7f12d4a08640 1 -- 172.21.15.70:0/2724084949 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608119e1d10 con 0x5608119d32b0 2024-04-01T01:43:19.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.821+0000 7f12cf7fe640 1 --2- 172.21.15.70:0/2724084949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d3ca0 0x5608119e4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:43:19.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.822+0000 7f12af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:43:19.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.822+0000 7f12d4a08640 1 --2- 172.21.15.70:0/2724084949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119e4900 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f12c4032040 tx=0x7f12c4009fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21e02efc82d97089 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:19.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.822+0000 7f12af7fe640 1 -- 172.21.15.70:0/2724084949 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c4041070 con 0x5608119d32b0 2024-04-01T01:43:19.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.822+0000 7f12af7fe640 1 -- 172.21.15.70:0/2724084949 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f12c4012530 con 0x5608119d32b0 2024-04-01T01:43:19.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.822+0000 7f12d9046740 1 -- 172.21.15.70:0/2724084949 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12c4036070 con 0x5608119d32b0 2024-04-01T01:43:19.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.824+0000 7f12af7fe640 1 -- 172.21.15.70:0/2724084949 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f12c4012730 con 0x5608119d32b0 2024-04-01T01:43:19.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.824+0000 7f12d9046740 10 client.28243 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:19.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.824+0000 7f12d9046740 1 -- 172.21.15.70:0/2724084949 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560811a4ec70 con 0x5608119d32b0 2024-04-01T01:43:19.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.825+0000 7f12af7fe640 1 -- 172.21.15.70:0/2724084949 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12c4036070 con 0x5608119d32b0 2024-04-01T01:43:19.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.826+0000 7f12af7fe640 1 -- 172.21.15.70:0/2724084949 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f12c4026040 con 0x5608119d32b0 2024-04-01T01:43:19.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.826+0000 7f12d9046740 10 client.28243 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:19.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.826+0000 7f12d9046740 10 client.28243 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:19.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.826+0000 7f12d9046740 20 client.28243 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:19.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.826+0000 7f12d9046740 1 --2- 172.21.15.70:0/2724084949 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560811a4f460 0x560811a51880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:19.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.826+0000 7f12d9046740 4 client.28243 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:43:19.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.826+0000 7f12d9046740 1 -- 172.21.15.70:0/2724084949 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56081193fa10 con 0x560811a4f460 2024-04-01T01:43:19.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.826+0000 7f12cf7fe640 1 --2- 172.21.15.70:0/2724084949 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560811a4f460 0x560811a51880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:19.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.827+0000 7f12cf7fe640 1 --2- 172.21.15.70:0/2724084949 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560811a4f460 0x560811a51880 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68a630cfcb05fe11 server_cookie=977b60ccec35492f in_seq=0 out_seq=0 2024-04-01T01:43:19.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.827+0000 7f12af7fe640 10 client.28243 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:19.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.828+0000 7f12af7fe640 1 -- 172.21.15.70:0/2724084949 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x56081193fa10 con 0x560811a4f460 2024-04-01T01:43:19.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.828+0000 7f12af7fe640 10 client.28243 handle_command_reply: tid=0 2024-04-01T01:43:19.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.831+0000 7f12d9046740 1 client.28243 shutdown 2024-04-01T01:43:19.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.831+0000 7f12d9046740 1 -- 172.21.15.70:0/2724084949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 msgr2=0x5608119e4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.831+0000 7f12d9046740 1 --2- 172.21.15.70:0/2724084949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119e4900 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f12c4032040 tx=0x7f12c4009fa0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.834+0000 7f12d9046740 1 -- 172.21.15.70:0/2724084949 shutdown_connections 2024-04-01T01:43:19.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.834+0000 7f12d9046740 1 --2- 172.21.15.70:0/2724084949 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560811a4f460 0x560811a51880 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.834+0000 7f12d9046740 1 --2- 172.21.15.70:0/2724084949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5608119d8e10 0x5608119e14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.834+0000 7f12d9046740 1 --2- 172.21.15.70:0/2724084949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5608119d3ca0 0x5608119e4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.834+0000 7f12d9046740 1 --2- 172.21.15.70:0/2724084949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5608119d32b0 0x5608119e4900 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.834+0000 7f12d9046740 1 -- 172.21.15.70:0/2724084949 >> 172.21.15.70:0/2724084949 conn(0x5608119ce820 msgr2=0x5608119cec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:19.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.834+0000 7f12af7fe640 0 client.28243 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:19.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.834+0000 7f12d9046740 1 -- 172.21.15.70:0/2724084949 shutdown_connections 2024-04-01T01:43:19.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.835+0000 7f12d9046740 1 -- 172.21.15.70:0/2724084949 wait complete. 2024-04-01T01:43:19.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.835+0000 7f12d9046740 20 client.28243 trim_cache size 0 max 16384 2024-04-01T01:43:19.836 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:43:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "scrub active (28 inodes in the stack)", 2024-04-01T01:43:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:43:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: "cefb5ce3-c6c5-4378-b90f-0772927086a3": { 2024-04-01T01:43:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:43:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "cefb5ce3-c6c5-4378-b90f-0772927086a3", 2024-04-01T01:43:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:43:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:43:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:43:19.837 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:43:19.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.836+0000 7f12d646b640 1 -- 172.21.15.70:0/665507803 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12b80a25f0 msgr2=0x7f12b80a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.836+0000 7f12d646b640 1 --2- 172.21.15.70:0/665507803 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12b80a25f0 0x7f12b80a4ab0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f12c809a550 tx=0x7f12c80c50d0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.836+0000 7f12d646b640 1 -- 172.21.15.70:0/665507803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0108750 msgr2=0x7f12d01a6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:19.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.836+0000 7f12d646b640 1 --2- 172.21.15.70:0/665507803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0108750 0x7f12d01a6870 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f12c800ba50 tx=0x7f12c800bf10 comp rx=0 tx=0).stop 2024-04-01T01:43:19.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.837+0000 7f12d646b640 1 -- 172.21.15.70:0/665507803 shutdown_connections 2024-04-01T01:43:19.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.837+0000 7f12d646b640 1 --2- 172.21.15.70:0/665507803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12d010da70 0x7f12d01ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.837+0000 7f12d646b640 1 --2- 172.21.15.70:0/665507803 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12b80a25f0 0x7f12b80a4ab0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.837+0000 7f12d646b640 1 --2- 172.21.15.70:0/665507803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12d010b0c0 0x7f12d01a6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.838+0000 7f12d646b640 1 --2- 172.21.15.70:0/665507803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12d0108750 0x7f12d01a6870 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:19.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.838+0000 7f12d646b640 1 -- 172.21.15.70:0/665507803 >> 172.21.15.70:0/665507803 conn(0x7f12d006e5b0 msgr2=0x7f12d0109a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:19.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.838+0000 7f12d646b640 1 -- 172.21.15.70:0/665507803 shutdown_connections 2024-04-01T01:43:19.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:19.838+0000 7f12d646b640 1 -- 172.21.15.70:0/665507803 wait complete. 2024-04-01T01:43:19.868 INFO:tasks.cephfs.filesystem:scrub status for tag:cefb5ce3-c6c5-4378-b90f-0772927086a3 - {'path': '/', 'tag': 'cefb5ce3-c6c5-4378-b90f-0772927086a3', 'options': 'recursive,force'} 2024-04-01T01:43:20.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:20 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:43:20.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:20 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:43:20.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:20 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:43:20.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:20 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:43:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:20 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:43:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:20 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:43:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:20 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:43:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:20 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:43:20.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.530+0000 7f4315609640 1 -- 172.21.15.70:0/1168643426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4308099120 msgr2=0x7f430809b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:20.531 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.530+0000 7f4315609640 1 --2- 172.21.15.70:0/1168643426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4308099120 0x7f430809b510 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f42fc009900 tx=0x7f42fc02bc30 comp rx=0 tx=0).stop 2024-04-01T01:43:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.531+0000 7f4315609640 1 -- 172.21.15.70:0/1168643426 shutdown_connections 2024-04-01T01:43:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.531+0000 7f4315609640 1 --2- 172.21.15.70:0/1168643426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f430809ba50 0x7f430809dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.531+0000 7f4315609640 1 --2- 172.21.15.70:0/1168643426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4308099120 0x7f430809b510 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.531+0000 7f4315609640 1 --2- 172.21.15.70:0/1168643426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43080967f0 0x7f4308098be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.531+0000 7f4315609640 1 -- 172.21.15.70:0/1168643426 >> 172.21.15.70:0/1168643426 conn(0x7f4308090220 msgr2=0x7f4308092640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.531+0000 7f4315609640 1 -- 172.21.15.70:0/1168643426 shutdown_connections 2024-04-01T01:43:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.531+0000 7f4315609640 1 -- 172.21.15.70:0/1168643426 wait complete. 2024-04-01T01:43:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.532+0000 7f4315609640 1 Processor -- start 2024-04-01T01:43:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.532+0000 7f4315609640 1 -- start start 2024-04-01T01:43:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.533+0000 7f4315609640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43080967f0 0x7f430812e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.533+0000 7f4315609640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4308099120 0x7f430812e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.533+0000 7f4315609640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f430809ba50 0x7f43081355d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.533+0000 7f430ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43080967f0 0x7f430812e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.533+0000 7f430f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4308099120 0x7f430812e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.533+0000 7f430f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4308099120 0x7f430812e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34434/0 (socket says 172.21.15.70:34434) 2024-04-01T01:43:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.533+0000 7f4315609640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4308094d40 con 0x7f4308099120 2024-04-01T01:43:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.533+0000 7f4315609640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4308094bc0 con 0x7f430809ba50 2024-04-01T01:43:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.533+0000 7f4315609640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4308094ec0 con 0x7f43080967f0 2024-04-01T01:43:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.533+0000 7f430f7fe640 1 -- 172.21.15.70:0/358675955 learned_addr learned my addr 172.21.15.70:0/358675955 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.533+0000 7f430ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43080967f0 0x7f430812e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46864/0 (socket says 172.21.15.70:46864) 2024-04-01T01:43:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.533+0000 7f4314e08640 1 --2- 172.21.15.70:0/358675955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f430809ba50 0x7f43081355d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.534+0000 7f430ffff640 1 -- 172.21.15.70:0/358675955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f430809ba50 msgr2=0x7f43081355d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.534+0000 7f430ffff640 1 --2- 172.21.15.70:0/358675955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f430809ba50 0x7f43081355d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.534+0000 7f430ffff640 1 -- 172.21.15.70:0/358675955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4308099120 msgr2=0x7f430812e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.534+0000 7f430ffff640 1 --2- 172.21.15.70:0/358675955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4308099120 0x7f430812e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.534+0000 7f430ffff640 1 -- 172.21.15.70:0/358675955 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4308135c40 con 0x7f43080967f0 2024-04-01T01:43:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.534+0000 7f430f7fe640 1 --2- 172.21.15.70:0/358675955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4308099120 0x7f430812e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:43:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.534+0000 7f430ffff640 1 --2- 172.21.15.70:0/358675955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43080967f0 0x7f430812e020 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f430400d6d0 tx=0x7f430400db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.535+0000 7f430d7fa640 1 -- 172.21.15.70:0/358675955 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4304017070 con 0x7f43080967f0 2024-04-01T01:43:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.535+0000 7f430d7fa640 1 -- 172.21.15.70:0/358675955 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4304009820 con 0x7f43080967f0 2024-04-01T01:43:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.535+0000 7f430d7fa640 1 -- 172.21.15.70:0/358675955 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4304017070 con 0x7f43080967f0 2024-04-01T01:43:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.535+0000 7f4315609640 1 -- 172.21.15.70:0/358675955 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4308135f30 con 0x7f43080967f0 2024-04-01T01:43:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.535+0000 7f4315609640 1 -- 172.21.15.70:0/358675955 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4308136340 con 0x7f43080967f0 2024-04-01T01:43:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.538+0000 7f4319446740 1 Processor -- start 2024-04-01T01:43:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.538+0000 7f430d7fa640 1 -- 172.21.15.70:0/358675955 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f4304004980 con 0x7f43080967f0 2024-04-01T01:43:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.539+0000 7f430d7fa640 1 --2- 172.21.15.70:0/358675955 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f43000a2890 0x7f43000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.539+0000 7f430d7fa640 1 -- 172.21.15.70:0/358675955 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f4304005020 con 0x7f43080967f0 2024-04-01T01:43:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.539+0000 7f4319446740 1 -- start start 2024-04-01T01:43:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.540+0000 7f4314e08640 1 --2- 172.21.15.70:0/358675955 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f43000a2890 0x7f43000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.540+0000 7f4319446740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60d3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.540+0000 7f4319446740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60d87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.540+0000 7f4319446740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 0x55cfd60db2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.540+0000 7f4319446740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cfd602e8f0 con 0x55cfd60d8e50 2024-04-01T01:43:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.540+0000 7f4319446740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55cfd5fd78d0 con 0x55cfd60d3ce0 2024-04-01T01:43:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.540+0000 7f4319446740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cfd602a990 con 0x55cfd60d32f0 2024-04-01T01:43:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.540+0000 7f430f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60d87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.540+0000 7f430ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60d3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.540+0000 7f430f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60d87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49540/0 (socket says 172.21.15.70:49540) 2024-04-01T01:43:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.540+0000 7f430ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60d3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46878/0 (socket says 172.21.15.70:46878) 2024-04-01T01:43:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.540+0000 7f430f7fe640 1 -- 172.21.15.70:0/2061714467 learned_addr learned my addr 172.21.15.70:0/2061714467 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.541+0000 7f4314e08640 1 --2- 172.21.15.70:0/2061714467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 0x55cfd60db2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.541+0000 7f430f7fe640 1 -- 172.21.15.70:0/2061714467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 msgr2=0x55cfd60d3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.541+0000 7f430f7fe640 1 --2- 172.21.15.70:0/2061714467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60d3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.541+0000 7f430f7fe640 1 -- 172.21.15.70:0/2061714467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 msgr2=0x55cfd60db2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.541+0000 7f430f7fe640 1 --2- 172.21.15.70:0/2061714467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 0x55cfd60db2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.541+0000 7f430f7fe640 1 -- 172.21.15.70:0/2061714467 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfd5d43210 con 0x55cfd60d3ce0 2024-04-01T01:43:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.541+0000 7f4314e08640 1 --2- 172.21.15.70:0/358675955 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f43000a2890 0x7f43000a4d50 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4310067000 tx=0x7f4310073d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.541+0000 7f430f7fe640 1 --2- 172.21.15.70:0/2061714467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60d87e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f42fc005e00 tx=0x7f42fc005e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d051ec7ba3b4f45f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.542+0000 7f42fa7fc640 1 -- 172.21.15.70:0/2061714467 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc045070 con 0x55cfd60d3ce0 2024-04-01T01:43:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.542+0000 7f42fa7fc640 1 -- 172.21.15.70:0/2061714467 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f42fc02cbe0 con 0x55cfd60d3ce0 2024-04-01T01:43:20.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.542+0000 7f42fa7fc640 1 -- 172.21.15.70:0/2061714467 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc03f440 con 0x55cfd60d3ce0 2024-04-01T01:43:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.543+0000 7f4319446740 1 -- 172.21.15.70:0/2061714467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 msgr2=0x55cfd60d87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.543+0000 7f4319446740 1 --2- 172.21.15.70:0/2061714467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60d87e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f42fc005e00 tx=0x7f42fc005e30 comp rx=0 tx=0).stop 2024-04-01T01:43:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.544+0000 7f4319446740 1 -- 172.21.15.70:0/2061714467 shutdown_connections 2024-04-01T01:43:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.544+0000 7f4319446740 1 --2- 172.21.15.70:0/2061714467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 0x55cfd60db2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.544+0000 7f4319446740 1 --2- 172.21.15.70:0/2061714467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60d87e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.544+0000 7f4319446740 1 --2- 172.21.15.70:0/2061714467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60d3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.544+0000 7f4319446740 1 -- 172.21.15.70:0/2061714467 >> 172.21.15.70:0/2061714467 conn(0x55cfd60c6050 msgr2=0x55cfd60c6450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.544+0000 7f4319446740 1 -- 172.21.15.70:0/2061714467 shutdown_connections 2024-04-01T01:43:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.544+0000 7f4319446740 1 -- 172.21.15.70:0/2061714467 wait complete. 2024-04-01T01:43:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.544+0000 7f4319446740 1 -- start start 2024-04-01T01:43:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.545+0000 7f4319446740 1 Processor -- start 2024-04-01T01:43:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.545+0000 7f4319446740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60dd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.545+0000 7f4319446740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60ddbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.545+0000 7f4319446740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 0x55cfd60e4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.545+0000 7f430f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60ddbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.545+0000 7f430f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60ddbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49546/0 (socket says 172.21.15.70:49546) 2024-04-01T01:43:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.545+0000 7f430ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60dd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.546+0000 7f430f7fe640 1 -- 172.21.15.70:0/2864033648 learned_addr learned my addr 172.21.15.70:0/2864033648 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.546+0000 7f4319446740 1 -- 172.21.15.70:0/2864033648 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cfd60dbda0 con 0x55cfd60d8e50 2024-04-01T01:43:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.546+0000 7f4319446740 1 -- 172.21.15.70:0/2864033648 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55cfd5fccf80 con 0x55cfd60d3ce0 2024-04-01T01:43:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.546+0000 7f4319446740 1 -- 172.21.15.70:0/2864033648 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cfd60dbf20 con 0x55cfd60d32f0 2024-04-01T01:43:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.546+0000 7f4314e08640 1 --2- 172.21.15.70:0/2864033648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 0x55cfd60e4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.546+0000 7f4319446740 10 client.0 _ll_register_callbacks cb 0x55cfd6025290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:20.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.547+0000 7f4314e08640 1 -- 172.21.15.70:0/2864033648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 msgr2=0x55cfd60dd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:20.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.547+0000 7f4314e08640 1 --2- 172.21.15.70:0/2864033648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60dd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.547+0000 7f4314e08640 1 -- 172.21.15.70:0/2864033648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 msgr2=0x55cfd60ddbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.547+0000 7f4314e08640 1 --2- 172.21.15.70:0/2864033648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60ddbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.547+0000 7f4314e08640 1 -- 172.21.15.70:0/2864033648 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfd5e0f620 con 0x55cfd60d8e50 2024-04-01T01:43:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.547+0000 7f430ffff640 1 --2- 172.21.15.70:0/2864033648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60dd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:43:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.547+0000 7f4314e08640 1 --2- 172.21.15.70:0/2864033648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 0x55cfd60e4c00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f431004f780 tx=0x7f43100679f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f301cfc9abddf825 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.547+0000 7f42ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:43:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.547+0000 7f42ef7fe640 1 -- 172.21.15.70:0/2864033648 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4310087030 con 0x55cfd60d8e50 2024-04-01T01:43:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.547+0000 7f4319446740 1 -- 172.21.15.70:0/2864033648 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42fc02cbe0 con 0x55cfd60d8e50 2024-04-01T01:43:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.548+0000 7f42ef7fe640 1 -- 172.21.15.70:0/2864033648 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f431007fe00 con 0x55cfd60d8e50 2024-04-01T01:43:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.548+0000 7f42ef7fe640 1 -- 172.21.15.70:0/2864033648 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f431009e050 con 0x55cfd60d8e50 2024-04-01T01:43:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.548+0000 7f4319446740 10 client.18450 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.548+0000 7f4319446740 1 -- 172.21.15.70:0/2864033648 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cfd60e7ca0 con 0x55cfd60d8e50 2024-04-01T01:43:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.548+0000 7f42ef7fe640 1 -- 172.21.15.70:0/2864033648 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f42fc02cbe0 con 0x55cfd60d8e50 2024-04-01T01:43:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.549+0000 7f42ef7fe640 1 -- 172.21.15.70:0/2864033648 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f431008ec30 con 0x55cfd60d8e50 2024-04-01T01:43:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.549+0000 7f4319446740 10 client.18450 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.549+0000 7f4319446740 10 client.18450 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.549+0000 7f4319446740 20 client.18450 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.549+0000 7f4319446740 1 --2- 172.21.15.70:0/2864033648 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cfd60e9390 0x55cfd60eb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.549+0000 7f4319446740 4 client.18450 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.549+0000 7f4319446740 1 -- 172.21.15.70:0/2864033648 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfd6042120 con 0x55cfd60e9390 2024-04-01T01:43:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.549+0000 7f430ffff640 1 --2- 172.21.15.70:0/2864033648 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cfd60e9390 0x55cfd60eb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.550+0000 7f430ffff640 1 --2- 172.21.15.70:0/2864033648 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cfd60e9390 0x55cfd60eb7a0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=878589f0023205ee server_cookie=64228e49a0ce6cb4 in_seq=0 out_seq=0 2024-04-01T01:43:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.550+0000 7f42ef7fe640 10 client.18450 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.551+0000 7f42ef7fe640 1 -- 172.21.15.70:0/2864033648 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cfd6042120 con 0x55cfd60e9390 2024-04-01T01:43:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.551+0000 7f42ef7fe640 10 client.18450 handle_command_reply: tid=0 2024-04-01T01:43:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.551+0000 7f4319446740 1 client.18450 shutdown 2024-04-01T01:43:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.551+0000 7f4319446740 1 -- 172.21.15.70:0/2864033648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 msgr2=0x55cfd60e4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.551+0000 7f4319446740 1 --2- 172.21.15.70:0/2864033648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 0x55cfd60e4c00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f431004f780 tx=0x7f43100679f0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.551+0000 7f4319446740 1 -- 172.21.15.70:0/2864033648 shutdown_connections 2024-04-01T01:43:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.551+0000 7f4319446740 1 --2- 172.21.15.70:0/2864033648 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cfd60e9390 0x55cfd60eb7a0 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.551+0000 7f4319446740 1 --2- 172.21.15.70:0/2864033648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 0x55cfd60e4c00 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.551+0000 7f4319446740 1 --2- 172.21.15.70:0/2864033648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60ddbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.551+0000 7f42ef7fe640 0 client.18450 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.551+0000 7f4319446740 1 --2- 172.21.15.70:0/2864033648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60dd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.551+0000 7f4319446740 1 -- 172.21.15.70:0/2864033648 >> 172.21.15.70:0/2864033648 conn(0x55cfd60c6050 msgr2=0x55cfd60c5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.551+0000 7f4319446740 1 -- 172.21.15.70:0/2864033648 shutdown_connections 2024-04-01T01:43:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.552+0000 7f4319446740 1 -- 172.21.15.70:0/2864033648 wait complete. 2024-04-01T01:43:20.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.552+0000 7f4319446740 20 client.18450 trim_cache size 0 max 16384 2024-04-01T01:43:20.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:20 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:43:20.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:20 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:43:20.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:20 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:43:20.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:20 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:43:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.585+0000 7f4319446740 1 Processor -- start 2024-04-01T01:43:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.585+0000 7f4319446740 1 -- start start 2024-04-01T01:43:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.585+0000 7f4319446740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60dc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.585+0000 7f4319446740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d3ce0 0x55cfd60e4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.585+0000 7f430ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60dc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.585+0000 7f4319446740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d8e50 0x55cfd60dce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.585+0000 7f430ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60dc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46900/0 (socket says 172.21.15.70:46900) 2024-04-01T01:43:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.585+0000 7f430ffff640 1 -- 172.21.15.70:0/2234191639 learned_addr learned my addr 172.21.15.70:0/2234191639 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.585+0000 7f4319446740 1 -- 172.21.15.70:0/2234191639 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f42f400fcc0 con 0x55cfd60d3ce0 2024-04-01T01:43:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.585+0000 7f4319446740 1 -- 172.21.15.70:0/2234191639 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f42f4011800 con 0x55cfd60d8e50 2024-04-01T01:43:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.586+0000 7f4319446740 1 -- 172.21.15.70:0/2234191639 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f42f4010390 con 0x55cfd60d32f0 2024-04-01T01:43:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.586+0000 7f4314e08640 1 --2- 172.21.15.70:0/2234191639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d8e50 0x55cfd60dce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.586+0000 7f430f7fe640 1 --2- 172.21.15.70:0/2234191639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d3ce0 0x55cfd60e4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.586+0000 7f4314e08640 1 -- 172.21.15.70:0/2234191639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 msgr2=0x55cfd60dc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.586+0000 7f4314e08640 1 --2- 172.21.15.70:0/2234191639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60dc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.586+0000 7f4314e08640 1 -- 172.21.15.70:0/2234191639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d3ce0 msgr2=0x55cfd60e4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.586+0000 7f4314e08640 1 --2- 172.21.15.70:0/2234191639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d3ce0 0x55cfd60e4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.586+0000 7f4314e08640 1 -- 172.21.15.70:0/2234191639 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfd60d40c0 con 0x55cfd60d8e50 2024-04-01T01:43:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.586+0000 7f430f7fe640 1 --2- 172.21.15.70:0/2234191639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d3ce0 0x55cfd60e4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:43:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.586+0000 7f4314e08640 1 --2- 172.21.15.70:0/2234191639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d8e50 0x55cfd60dce90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f431009c040 tx=0x7f431004fbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b89ceeefabc892c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:20.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.587+0000 7f42f9ffb640 1 -- 172.21.15.70:0/2234191639 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43100a8070 con 0x55cfd60d8e50 2024-04-01T01:43:20.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.587+0000 7f42f9ffb640 1 -- 172.21.15.70:0/2234191639 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4310088040 con 0x55cfd60d8e50 2024-04-01T01:43:20.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.588+0000 7f4319446740 1 -- 172.21.15.70:0/2234191639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d8e50 msgr2=0x55cfd60dce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.588+0000 7f4319446740 1 --2- 172.21.15.70:0/2234191639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d8e50 0x55cfd60dce90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f431009c040 tx=0x7f431004fbd0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.589+0000 7f4319446740 1 -- 172.21.15.70:0/2234191639 shutdown_connections 2024-04-01T01:43:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.589+0000 7f4319446740 1 --2- 172.21.15.70:0/2234191639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d8e50 0x55cfd60dce90 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.589+0000 7f4319446740 1 --2- 172.21.15.70:0/2234191639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d3ce0 0x55cfd60e4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.589+0000 7f4319446740 1 --2- 172.21.15.70:0/2234191639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60dc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.589+0000 7f4319446740 1 -- 172.21.15.70:0/2234191639 >> 172.21.15.70:0/2234191639 conn(0x55cfd60c6050 msgr2=0x55cfd60c6430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.589+0000 7f4319446740 1 -- 172.21.15.70:0/2234191639 shutdown_connections 2024-04-01T01:43:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.589+0000 7f4319446740 1 -- 172.21.15.70:0/2234191639 wait complete. 2024-04-01T01:43:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.590+0000 7f4319446740 1 -- start start 2024-04-01T01:43:20.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.590+0000 7f4319446740 1 Processor -- start 2024-04-01T01:43:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.590+0000 7f4319446740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60e1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.591+0000 7f4319446740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60e21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.591+0000 7f4319446740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 0x55cfd60e3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.591+0000 7f430f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60e21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.591+0000 7f430f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60e21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49560/0 (socket says 172.21.15.70:49560) 2024-04-01T01:43:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.591+0000 7f430f7fe640 1 -- 172.21.15.70:0/1928364160 learned_addr learned my addr 172.21.15.70:0/1928364160 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.591+0000 7f4314e08640 1 --2- 172.21.15.70:0/1928364160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 0x55cfd60e3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.591+0000 7f4319446740 1 -- 172.21.15.70:0/1928364160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f42f4011130 con 0x55cfd60d8e50 2024-04-01T01:43:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.591+0000 7f4319446740 1 -- 172.21.15.70:0/1928364160 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f42f4010a60 con 0x55cfd60d3ce0 2024-04-01T01:43:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.591+0000 7f4319446740 1 -- 172.21.15.70:0/1928364160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f42f400dca0 con 0x55cfd60d32f0 2024-04-01T01:43:20.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.592+0000 7f4319446740 10 client.0 _ll_register_callbacks cb 0x7f431007e900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.592+0000 7f4314e08640 1 -- 172.21.15.70:0/1928364160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 msgr2=0x55cfd60e1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.592+0000 7f4314e08640 1 --2- 172.21.15.70:0/1928364160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60e1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.592+0000 7f4314e08640 1 -- 172.21.15.70:0/1928364160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 msgr2=0x55cfd60e21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.592+0000 7f4314e08640 1 --2- 172.21.15.70:0/1928364160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60e21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.592+0000 7f4314e08640 1 -- 172.21.15.70:0/1928364160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfd60e3730 con 0x55cfd60d8e50 2024-04-01T01:43:20.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.592+0000 7f430f7fe640 1 --2- 172.21.15.70:0/1928364160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60e21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:43:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.593+0000 7f4314e08640 1 --2- 172.21.15.70:0/1928364160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 0x55cfd60e3140 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f4310074010 tx=0x7f431007cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa1a805ddb70c6d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.593+0000 7f42ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:43:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.593+0000 7f42ef7fe640 1 -- 172.21.15.70:0/1928364160 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43100a8040 con 0x55cfd60d8e50 2024-04-01T01:43:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.593+0000 7f4319446740 1 -- 172.21.15.70:0/1928364160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4310088040 con 0x55cfd60d8e50 2024-04-01T01:43:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.593+0000 7f42ef7fe640 1 -- 172.21.15.70:0/1928364160 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f431008e420 con 0x55cfd60d8e50 2024-04-01T01:43:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.593+0000 7f42ef7fe640 1 -- 172.21.15.70:0/1928364160 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f43100b6050 con 0x55cfd60d8e50 2024-04-01T01:43:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.593+0000 7f4319446740 10 client.18456 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.594+0000 7f4319446740 1 -- 172.21.15.70:0/1928364160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cfd60e4a20 con 0x55cfd60d8e50 2024-04-01T01:43:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.594+0000 7f42ef7fe640 1 -- 172.21.15.70:0/1928364160 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4310088040 con 0x55cfd60d8e50 2024-04-01T01:43:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.594+0000 7f42ef7fe640 1 -- 172.21.15.70:0/1928364160 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f431008ead0 con 0x55cfd60d8e50 2024-04-01T01:43:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.594+0000 7f4319446740 10 client.18456 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.594+0000 7f4319446740 10 client.18456 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.594+0000 7f4319446740 20 client.18456 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.594+0000 7f4319446740 1 --2- 172.21.15.70:0/1928364160 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cfd60e9390 0x55cfd61518c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.595+0000 7f4319446740 4 client.18456 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:43:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.595+0000 7f4319446740 1 -- 172.21.15.70:0/1928364160 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cfd603be30 con 0x55cfd60e9390 2024-04-01T01:43:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.595+0000 7f430ffff640 1 --2- 172.21.15.70:0/1928364160 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cfd60e9390 0x55cfd61518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.595+0000 7f430ffff640 1 --2- 172.21.15.70:0/1928364160 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cfd60e9390 0x55cfd61518c0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=932e994a63ebd18d server_cookie=5ce349c228733f9a in_seq=0 out_seq=0 2024-04-01T01:43:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.595+0000 7f42ef7fe640 10 client.18456 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.670+0000 7f42ef7fe640 1 -- 172.21.15.70:0/1928364160 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cfd603be30 con 0x55cfd60e9390 2024-04-01T01:43:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.671+0000 7f42ef7fe640 10 client.18456 handle_command_reply: tid=0 2024-04-01T01:43:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.671+0000 7f4319446740 1 client.18456 shutdown 2024-04-01T01:43:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.675+0000 7f4319446740 1 -- 172.21.15.70:0/1928364160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 msgr2=0x55cfd60e3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.675+0000 7f4319446740 1 --2- 172.21.15.70:0/1928364160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 0x55cfd60e3140 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f4310074010 tx=0x7f431007cd50 comp rx=0 tx=0).stop 2024-04-01T01:43:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.676+0000 7f4319446740 1 -- 172.21.15.70:0/1928364160 shutdown_connections 2024-04-01T01:43:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.676+0000 7f4319446740 1 --2- 172.21.15.70:0/1928364160 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cfd60e9390 0x55cfd61518c0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.676+0000 7f4319446740 1 --2- 172.21.15.70:0/1928364160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cfd60d8e50 0x55cfd60e3140 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.676+0000 7f4319446740 1 --2- 172.21.15.70:0/1928364160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cfd60d3ce0 0x55cfd60e21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.676+0000 7f4319446740 1 --2- 172.21.15.70:0/1928364160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cfd60d32f0 0x55cfd60e1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.676+0000 7f4319446740 1 -- 172.21.15.70:0/1928364160 >> 172.21.15.70:0/1928364160 conn(0x55cfd60c6050 msgr2=0x55cfd60dc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.676+0000 7f42ef7fe640 0 client.18456 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.677+0000 7f4319446740 1 -- 172.21.15.70:0/1928364160 shutdown_connections 2024-04-01T01:43:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.677+0000 7f4319446740 1 -- 172.21.15.70:0/1928364160 wait complete. 2024-04-01T01:43:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.677+0000 7f4319446740 20 client.18456 trim_cache size 0 max 16384 2024-04-01T01:43:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:43:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:43:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:43:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:43:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.681+0000 7f4315609640 1 -- 172.21.15.70:0/358675955 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f43000a2890 msgr2=0x7f43000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.682+0000 7f4315609640 1 --2- 172.21.15.70:0/358675955 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f43000a2890 0x7f43000a4d50 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4310067000 tx=0x7f4310073d50 comp rx=0 tx=0).stop 2024-04-01T01:43:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.682+0000 7f4315609640 1 -- 172.21.15.70:0/358675955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43080967f0 msgr2=0x7f430812e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.682+0000 7f4315609640 1 --2- 172.21.15.70:0/358675955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43080967f0 0x7f430812e020 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f430400d6d0 tx=0x7f430400db90 comp rx=0 tx=0).stop 2024-04-01T01:43:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.682+0000 7f4315609640 1 -- 172.21.15.70:0/358675955 shutdown_connections 2024-04-01T01:43:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.682+0000 7f4315609640 1 --2- 172.21.15.70:0/358675955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f430809ba50 0x7f43081355d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.682+0000 7f4315609640 1 --2- 172.21.15.70:0/358675955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4308099120 0x7f430812e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.682+0000 7f4315609640 1 --2- 172.21.15.70:0/358675955 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f43000a2890 0x7f43000a4d50 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.682+0000 7f4315609640 1 --2- 172.21.15.70:0/358675955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43080967f0 0x7f430812e020 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.682+0000 7f4315609640 1 -- 172.21.15.70:0/358675955 >> 172.21.15.70:0/358675955 conn(0x7f4308090220 msgr2=0x7f430809a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.682+0000 7f4315609640 1 -- 172.21.15.70:0/358675955 shutdown_connections 2024-04-01T01:43:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:20.682+0000 7f4315609640 1 -- 172.21.15.70:0/358675955 wait complete. 2024-04-01T01:43:21.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:21 smithi189 ceph-mon[35753]: pgmap v655: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.4 MiB/s wr, 699 op/s 2024-04-01T01:43:21.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:21 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:43:21.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:21 smithi070 ceph-mon[32179]: pgmap v655: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.4 MiB/s wr, 699 op/s 2024-04-01T01:43:21.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:21 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:43:21.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:21 smithi081 ceph-mon[35142]: pgmap v655: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.4 MiB/s wr, 699 op/s 2024-04-01T01:43:21.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:21 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:43:21.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 515.0 s, 705.6 tps, lat 142.951 ms stddev 124.755 2024-04-01T01:43:23.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:22 smithi189 ceph-mon[35753]: pgmap v656: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.7 MiB/s wr, 805 op/s 2024-04-01T01:43:23.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:22 smithi070 ceph-mon[32179]: pgmap v656: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.7 MiB/s wr, 805 op/s 2024-04-01T01:43:23.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:22 smithi081 ceph-mon[35142]: pgmap v656: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.7 MiB/s wr, 805 op/s 2024-04-01T01:43:25.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:24 smithi189 ceph-mon[35753]: pgmap v657: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 5.5 MiB/s wr, 741 op/s 2024-04-01T01:43:25.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:24 smithi070 ceph-mon[32179]: pgmap v657: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 5.5 MiB/s wr, 741 op/s 2024-04-01T01:43:25.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:24 smithi081 ceph-mon[35142]: pgmap v657: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 5.5 MiB/s wr, 741 op/s 2024-04-01T01:43:26.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 520.0 s, 862.4 tps, lat 115.771 ms stddev 101.959 2024-04-01T01:43:27.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:26 smithi189 ceph-mon[35753]: pgmap v658: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 6.3 MiB/s wr, 834 op/s 2024-04-01T01:43:27.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:26 smithi070 ceph-mon[32179]: pgmap v658: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 6.3 MiB/s wr, 834 op/s 2024-04-01T01:43:27.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:26 smithi081 ceph-mon[35142]: pgmap v658: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 6.3 MiB/s wr, 834 op/s 2024-04-01T01:43:29.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:28 smithi189 ceph-mon[35753]: pgmap v659: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 6.6 MiB/s wr, 862 op/s 2024-04-01T01:43:29.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:28 smithi070 ceph-mon[32179]: pgmap v659: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 6.6 MiB/s wr, 862 op/s 2024-04-01T01:43:29.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:28 smithi081 ceph-mon[35142]: pgmap v659: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 6.6 MiB/s wr, 862 op/s 2024-04-01T01:43:31.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:30 smithi189 ceph-mon[35753]: pgmap v660: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 6.1 MiB/s wr, 803 op/s 2024-04-01T01:43:31.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:30 smithi070 ceph-mon[32179]: pgmap v660: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 6.1 MiB/s wr, 803 op/s 2024-04-01T01:43:31.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:30 smithi081 ceph-mon[35142]: pgmap v660: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 6.1 MiB/s wr, 803 op/s 2024-04-01T01:43:31.897 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 525.0 s, 880.0 tps, lat 114.105 ms stddev 88.377 2024-04-01T01:43:33.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:32 smithi189 ceph-mon[35753]: pgmap v661: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 7.0 MiB/s wr, 882 op/s 2024-04-01T01:43:33.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:32 smithi070 ceph-mon[32179]: pgmap v661: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 7.0 MiB/s wr, 882 op/s 2024-04-01T01:43:33.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:32 smithi081 ceph-mon[35142]: pgmap v661: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 7.0 MiB/s wr, 882 op/s 2024-04-01T01:43:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.833+0000 7fe47802a640 1 -- 172.21.15.70:0/4189319272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4701033f0 msgr2=0x7fe470109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.833+0000 7fe47802a640 1 --2- 172.21.15.70:0/4189319272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4701033f0 0x7fe470109c80 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fe4680097e0 tx=0x7fe4680345f0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.833+0000 7fe47802a640 1 -- 172.21.15.70:0/4189319272 shutdown_connections 2024-04-01T01:43:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.833+0000 7fe47802a640 1 --2- 172.21.15.70:0/4189319272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4701033f0 0x7fe470109c80 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.833+0000 7fe47802a640 1 --2- 172.21.15.70:0/4189319272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe470102a30 0x7fe470102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.833+0000 7fe47802a640 1 --2- 172.21.15.70:0/4189319272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4700ffa60 0x7fe4700ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.833+0000 7fe47802a640 1 -- 172.21.15.70:0/4189319272 >> 172.21.15.70:0/4189319272 conn(0x7fe4700fb3b0 msgr2=0x7fe4700fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.834+0000 7fe47802a640 1 -- 172.21.15.70:0/4189319272 shutdown_connections 2024-04-01T01:43:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.836+0000 7fe47802a640 1 -- 172.21.15.70:0/4189319272 wait complete. 2024-04-01T01:43:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.837+0000 7fe47802a640 1 Processor -- start 2024-04-01T01:43:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.837+0000 7fe47802a640 1 -- start start 2024-04-01T01:43:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.837+0000 7fe47802a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4700ffa60 0x7fe47006fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.837+0000 7fe47802a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe470102a30 0x7fe470070530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.837+0000 7fe47802a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4701033f0 0x7fe4701a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.837+0000 7fe47802a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe47010b140 con 0x7fe4701033f0 2024-04-01T01:43:33.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.837+0000 7fe47802a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe47010afc0 con 0x7fe4700ffa60 2024-04-01T01:43:33.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.837+0000 7fe47802a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe47010b2c0 con 0x7fe470102a30 2024-04-01T01:43:33.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.838+0000 7fe4765c7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4701033f0 0x7fe4701a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:33.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.838+0000 7fe4755c5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe470102a30 0x7fe470070530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:33.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.838+0000 7fe4755c5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe470102a30 0x7fe470070530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:51676/0 (socket says 172.21.15.70:51676) 2024-04-01T01:43:33.839 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.838+0000 7fe4755c5640 1 -- 172.21.15.70:0/2738043277 learned_addr learned my addr 172.21.15.70:0/2738043277 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.839+0000 7fe4755c5640 1 -- 172.21.15.70:0/2738043277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4700ffa60 msgr2=0x7fe47006fff0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:43:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.839+0000 7fe4755c5640 1 --2- 172.21.15.70:0/2738043277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4700ffa60 0x7fe47006fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.840+0000 7fe4755c5640 1 -- 172.21.15.70:0/2738043277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4701033f0 msgr2=0x7fe4701a5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.840+0000 7fe4755c5640 1 --2- 172.21.15.70:0/2738043277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4701033f0 0x7fe4701a5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.840 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.840+0000 7fe4755c5640 1 -- 172.21.15.70:0/2738043277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4700736b0 con 0x7fe470102a30 2024-04-01T01:43:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.841+0000 7fe4755c5640 1 --2- 172.21.15.70:0/2738043277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe470102a30 0x7fe470070530 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe46000bbb0 tx=0x7fe460008600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.842+0000 7fe466ffd640 1 -- 172.21.15.70:0/2738043277 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe460008e30 con 0x7fe470102a30 2024-04-01T01:43:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.842+0000 7fe466ffd640 1 -- 172.21.15.70:0/2738043277 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe4600035f0 con 0x7fe470102a30 2024-04-01T01:43:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.842+0000 7fe466ffd640 1 -- 172.21.15.70:0/2738043277 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4600166e0 con 0x7fe470102a30 2024-04-01T01:43:33.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.844+0000 7fe47802a640 1 -- 172.21.15.70:0/2738043277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4701b6dc0 con 0x7fe470102a30 2024-04-01T01:43:33.846 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.844+0000 7fe47802a640 1 -- 172.21.15.70:0/2738043277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4701b7230 con 0x7fe470102a30 2024-04-01T01:43:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.847+0000 7fe47ac0e740 1 Processor -- start 2024-04-01T01:43:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.847+0000 7fe47ac0e740 1 -- start start 2024-04-01T01:43:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.848+0000 7fe47ac0e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e873b8d0 0x5605e873bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.848+0000 7fe47ac0e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e873c2c0 0x5605e8740dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.848+0000 7fe47ac0e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e8741430 0x5605e87438c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.848+0000 7fe47ac0e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5605e869f8f0 con 0x5605e8741430 2024-04-01T01:43:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.848+0000 7fe47ac0e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5605e86488d0 con 0x5605e873b8d0 2024-04-01T01:43:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.848+0000 7fe47ac0e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5605e869b990 con 0x5605e873c2c0 2024-04-01T01:43:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.848+0000 7fe475dc6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e873c2c0 0x5605e8740dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.848+0000 7fe475dc6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e873c2c0 0x5605e8740dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:51682/0 (socket says 172.21.15.70:51682) 2024-04-01T01:43:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.848+0000 7fe475dc6640 1 -- 172.21.15.70:0/936320325 learned_addr learned my addr 172.21.15.70:0/936320325 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.848+0000 7fe4765c7640 1 --2- 172.21.15.70:0/936320325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e873b8d0 0x5605e873bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.848+0000 7fe4765c7640 1 -- 172.21.15.70:0/936320325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e873c2c0 msgr2=0x5605e8740dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.848+0000 7fe4765c7640 1 --2- 172.21.15.70:0/936320325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e873c2c0 0x5605e8740dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.848+0000 7fe4765c7640 1 -- 172.21.15.70:0/936320325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e8741430 msgr2=0x5605e87438c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.848+0000 7fe4765c7640 1 --2- 172.21.15.70:0/936320325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e8741430 0x5605e87438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.848+0000 7fe4765c7640 1 -- 172.21.15.70:0/936320325 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605e83b4210 con 0x5605e873b8d0 2024-04-01T01:43:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.849+0000 7fe4765c7640 1 --2- 172.21.15.70:0/936320325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e873b8d0 0x5605e873bcf0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fe468034b30 tx=0x7fe4680093f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81395735b235e60f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.849+0000 7fe45bfff640 1 -- 172.21.15.70:0/936320325 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46803b400 con 0x5605e873b8d0 2024-04-01T01:43:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.849+0000 7fe45bfff640 1 -- 172.21.15.70:0/936320325 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe468003e20 con 0x5605e873b8d0 2024-04-01T01:43:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.849+0000 7fe45bfff640 1 -- 172.21.15.70:0/936320325 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4680138e0 con 0x5605e873b8d0 2024-04-01T01:43:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.849+0000 7fe466ffd640 1 -- 172.21.15.70:0/2738043277 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fe46001d020 con 0x7fe470102a30 2024-04-01T01:43:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.850+0000 7fe466ffd640 1 --2- 172.21.15.70:0/2738043277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe45c0a25c0 0x7fe45c0a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.850+0000 7fe466ffd640 1 -- 172.21.15.70:0/2738043277 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe46009b180 con 0x7fe470102a30 2024-04-01T01:43:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.852+0000 7fe47ac0e740 1 -- 172.21.15.70:0/936320325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e873b8d0 msgr2=0x5605e873bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.852+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/936320325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e873b8d0 0x5605e873bcf0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fe468034b30 tx=0x7fe4680093f0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.852+0000 7fe475dc6640 1 --2- 172.21.15.70:0/2738043277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe45c0a25c0 0x7fe45c0a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.852+0000 7fe47ac0e740 1 -- 172.21.15.70:0/936320325 shutdown_connections 2024-04-01T01:43:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.852+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/936320325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e8741430 0x5605e87438c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.852+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/936320325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e873c2c0 0x5605e8740dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.852+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/936320325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e873b8d0 0x5605e873bcf0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.852+0000 7fe47ac0e740 1 -- 172.21.15.70:0/936320325 >> 172.21.15.70:0/936320325 conn(0x5605e8736e40 msgr2=0x5605e8737240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.858+0000 7fe47ac0e740 1 -- 172.21.15.70:0/936320325 shutdown_connections 2024-04-01T01:43:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.858+0000 7fe475dc6640 1 --2- 172.21.15.70:0/2738043277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe45c0a25c0 0x7fe45c0a4a80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x5605e8737db0 tx=0x7fe46c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.860+0000 7fe47ac0e740 1 -- 172.21.15.70:0/936320325 wait complete. 2024-04-01T01:43:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.860+0000 7fe47ac0e740 1 -- start start 2024-04-01T01:43:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.860+0000 7fe47ac0e740 1 Processor -- start 2024-04-01T01:43:33.861 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.861+0000 7fe47ac0e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e873b8d0 0x5605e8745c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.861+0000 7fe47ac0e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e873c2c0 0x5605e87461b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.861+0000 7fe47ac0e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e8741430 0x5605e874d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.861+0000 7fe47ac0e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5605e8744380 con 0x5605e873b8d0 2024-04-01T01:43:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.861+0000 7fe47ac0e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5605e863df80 con 0x5605e873c2c0 2024-04-01T01:43:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.861+0000 7fe47ac0e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5605e8744500 con 0x5605e8741430 2024-04-01T01:43:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.861+0000 7fe47ac0e740 10 client.0 _ll_register_callbacks cb 0x5605e8696290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.861+0000 7fe4765c7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e8741430 0x5605e874d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.861+0000 7fe4765c7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e8741430 0x5605e874d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:51686/0 (socket says 172.21.15.70:51686) 2024-04-01T01:43:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.861+0000 7fe4765c7640 1 -- 172.21.15.70:0/2873703736 learned_addr learned my addr 172.21.15.70:0/2873703736 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:33.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.862+0000 7fe4765c7640 1 -- 172.21.15.70:0/2873703736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e873c2c0 msgr2=0x5605e87461b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.862+0000 7fe4765c7640 1 --2- 172.21.15.70:0/2873703736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e873c2c0 0x5605e87461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.862+0000 7fe4765c7640 1 -- 172.21.15.70:0/2873703736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e873b8d0 msgr2=0x5605e8745c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:43:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.862+0000 7fe4765c7640 1 --2- 172.21.15.70:0/2873703736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e873b8d0 0x5605e8745c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.862+0000 7fe4765c7640 1 -- 172.21.15.70:0/2873703736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605e8480620 con 0x5605e8741430 2024-04-01T01:43:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.862+0000 7fe4765c7640 1 --2- 172.21.15.70:0/2873703736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e8741430 0x5605e874d180 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe46803ba60 tx=0x7fe468034ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=feac4ccdc91f74ea server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.862+0000 7fe4597fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:43:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.863+0000 7fe4597fa640 1 -- 172.21.15.70:0/2873703736 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4680138f0 con 0x5605e8741430 2024-04-01T01:43:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.863+0000 7fe4597fa640 1 -- 172.21.15.70:0/2873703736 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe46803b400 con 0x5605e8741430 2024-04-01T01:43:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.863+0000 7fe47ac0e740 1 -- 172.21.15.70:0/2873703736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe468003e20 con 0x5605e8741430 2024-04-01T01:43:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.864+0000 7fe4597fa640 1 -- 172.21.15.70:0/2873703736 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe46805c050 con 0x5605e8741430 2024-04-01T01:43:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.866+0000 7fe47ac0e740 10 client.28270 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:33.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.866+0000 7fe47ac0e740 1 -- 172.21.15.70:0/2873703736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5605e8744380 con 0x5605e8741430 2024-04-01T01:43:33.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.868+0000 7fe4597fa640 1 -- 172.21.15.70:0/2873703736 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe468053070 con 0x5605e8741430 2024-04-01T01:43:33.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.868+0000 7fe4597fa640 1 -- 172.21.15.70:0/2873703736 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe46804d070 con 0x5605e8741430 2024-04-01T01:43:33.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.870+0000 7fe47ac0e740 10 client.28270 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.871+0000 7fe47ac0e740 10 client.28270 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.871+0000 7fe47ac0e740 20 client.28270 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.871+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/2873703736 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5605e8751250 0x5605e8753640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.871+0000 7fe47ac0e740 4 client.28270 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.871+0000 7fe47ac0e740 1 -- 172.21.15.70:0/2873703736 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe468003e20 con 0x5605e8751250 2024-04-01T01:43:33.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.872+0000 7fe475dc6640 1 --2- 172.21.15.70:0/2873703736 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5605e8751250 0x5605e8753640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:33.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.873+0000 7fe475dc6640 1 --2- 172.21.15.70:0/2873703736 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5605e8751250 0x5605e8753640 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f87e16905a034ef server_cookie=e3e0cf3fe90d22bc in_seq=0 out_seq=0 2024-04-01T01:43:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.873+0000 7fe4597fa640 10 client.28270 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.875+0000 7fe4597fa640 1 -- 172.21.15.70:0/2873703736 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fe468003e20 con 0x5605e8751250 2024-04-01T01:43:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.875+0000 7fe4597fa640 10 client.28270 handle_command_reply: tid=0 2024-04-01T01:43:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.875+0000 7fe47ac0e740 1 client.28270 shutdown 2024-04-01T01:43:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.877+0000 7fe47ac0e740 1 -- 172.21.15.70:0/2873703736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e8741430 msgr2=0x5605e874d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.877+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/2873703736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e8741430 0x5605e874d180 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe46803ba60 tx=0x7fe468034ef0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.877+0000 7fe47ac0e740 1 -- 172.21.15.70:0/2873703736 shutdown_connections 2024-04-01T01:43:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.877+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/2873703736 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5605e8751250 0x5605e8753640 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.877+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/2873703736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e8741430 0x5605e874d180 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.877+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/2873703736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e873c2c0 0x5605e87461b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.877+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/2873703736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e873b8d0 0x5605e8745c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.877+0000 7fe47ac0e740 1 -- 172.21.15.70:0/2873703736 >> 172.21.15.70:0/2873703736 conn(0x5605e8736e40 msgr2=0x5605e8735f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.877+0000 7fe4597fa640 0 client.28270 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.877+0000 7fe47ac0e740 1 -- 172.21.15.70:0/2873703736 shutdown_connections 2024-04-01T01:43:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.878+0000 7fe47ac0e740 1 -- 172.21.15.70:0/2873703736 wait complete. 2024-04-01T01:43:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.879+0000 7fe47ac0e740 20 client.28270 trim_cache size 0 max 16384 2024-04-01T01:43:33.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.907+0000 7fe47ac0e740 1 Processor -- start 2024-04-01T01:43:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.907+0000 7fe47ac0e740 1 -- start start 2024-04-01T01:43:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.907+0000 7fe47ac0e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e873b8d0 0x5605e8735e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.907+0000 7fe47ac0e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e873c2c0 0x5605e874d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe47ac0e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e8741430 0x5605e8757eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe47ac0e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe4701c1940 con 0x5605e873b8d0 2024-04-01T01:43:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe47ac0e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe4701c3480 con 0x5605e873c2c0 2024-04-01T01:43:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe47ac0e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe4701c2010 con 0x5605e8741430 2024-04-01T01:43:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe475dc6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e873b8d0 0x5605e8735e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe475dc6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e873b8d0 0x5605e8735e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40710/0 (socket says 172.21.15.70:40710) 2024-04-01T01:43:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe475dc6640 1 -- 172.21.15.70:0/2290365913 learned_addr learned my addr 172.21.15.70:0/2290365913 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe4755c5640 1 --2- 172.21.15.70:0/2290365913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e873c2c0 0x5605e874d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe4765c7640 1 --2- 172.21.15.70:0/2290365913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e8741430 0x5605e8757eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe475dc6640 1 -- 172.21.15.70:0/2290365913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e8741430 msgr2=0x5605e8757eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe475dc6640 1 --2- 172.21.15.70:0/2290365913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e8741430 0x5605e8757eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe475dc6640 1 -- 172.21.15.70:0/2290365913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e873c2c0 msgr2=0x5605e874d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe475dc6640 1 --2- 172.21.15.70:0/2290365913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e873c2c0 0x5605e874d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe475dc6640 1 -- 172.21.15.70:0/2290365913 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605e8753c20 con 0x5605e873b8d0 2024-04-01T01:43:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe4765c7640 1 --2- 172.21.15.70:0/2290365913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e8741430 0x5605e8757eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:43:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe4755c5640 1 --2- 172.21.15.70:0/2290365913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e873c2c0 0x5605e874d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:43:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.908+0000 7fe475dc6640 1 --2- 172.21.15.70:0/2290365913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e873b8d0 0x5605e8735e00 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x5605e873c6c0 tx=0x7fe46c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea8f1eb30a38cce server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.909+0000 7fe45b7fe640 1 -- 172.21.15.70:0/2290365913 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46c02b030 con 0x5605e873b8d0 2024-04-01T01:43:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.909+0000 7fe45b7fe640 1 -- 172.21.15.70:0/2290365913 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe46c0267e0 con 0x5605e873b8d0 2024-04-01T01:43:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.909+0000 7fe45b7fe640 1 -- 172.21.15.70:0/2290365913 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe46c02f3c0 con 0x5605e873b8d0 2024-04-01T01:43:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.910+0000 7fe47ac0e740 1 -- 172.21.15.70:0/2290365913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e873b8d0 msgr2=0x5605e8735e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.910+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/2290365913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e873b8d0 0x5605e8735e00 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x5605e873c6c0 tx=0x7fe46c014000 comp rx=0 tx=0).stop 2024-04-01T01:43:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.910+0000 7fe47ac0e740 1 -- 172.21.15.70:0/2290365913 shutdown_connections 2024-04-01T01:43:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.910+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/2290365913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e8741430 0x5605e8757eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.910+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/2290365913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e873c2c0 0x5605e874d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.910+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/2290365913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e873b8d0 0x5605e8735e00 secure :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0x5605e873c6c0 tx=0x7fe46c014000 comp rx=0 tx=0).stop 2024-04-01T01:43:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.910+0000 7fe47ac0e740 1 -- 172.21.15.70:0/2290365913 >> 172.21.15.70:0/2290365913 conn(0x5605e8736e40 msgr2=0x5605e87356f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.910+0000 7fe47ac0e740 1 -- 172.21.15.70:0/2290365913 shutdown_connections 2024-04-01T01:43:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.910+0000 7fe47ac0e740 1 -- 172.21.15.70:0/2290365913 wait complete. 2024-04-01T01:43:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.910+0000 7fe47ac0e740 1 -- start start 2024-04-01T01:43:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.911+0000 7fe47ac0e740 1 Processor -- start 2024-04-01T01:43:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.911+0000 7fe47ac0e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e873c2c0 0x5605e874ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.911+0000 7fe47ac0e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e8741430 0x5605e874b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.911+0000 7fe47ac0e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e874c070 0x5605e874c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.912+0000 7fe4755c5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e8741430 0x5605e874b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.912+0000 7fe4755c5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e8741430 0x5605e874b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53954/0 (socket says 172.21.15.70:53954) 2024-04-01T01:43:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.912+0000 7fe4755c5640 1 -- 172.21.15.70:0/269886720 learned_addr learned my addr 172.21.15.70:0/269886720 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.912+0000 7fe4765c7640 1 --2- 172.21.15.70:0/269886720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e874c070 0x5605e874c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.912+0000 7fe47ac0e740 1 -- 172.21.15.70:0/269886720 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe4701c2db0 con 0x5605e873c2c0 2024-04-01T01:43:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.912+0000 7fe47ac0e740 1 -- 172.21.15.70:0/269886720 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe4701c26e0 con 0x5605e8741430 2024-04-01T01:43:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.912+0000 7fe47ac0e740 1 -- 172.21.15.70:0/269886720 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe4701bf920 con 0x5605e874c070 2024-04-01T01:43:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.912+0000 7fe47ac0e740 10 client.0 _ll_register_callbacks cb 0x7fe46c009630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.914+0000 7fe4765c7640 1 -- 172.21.15.70:0/269886720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e8741430 msgr2=0x5605e874b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.914+0000 7fe4765c7640 1 --2- 172.21.15.70:0/269886720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e8741430 0x5605e874b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.914+0000 7fe4765c7640 1 -- 172.21.15.70:0/269886720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e873c2c0 msgr2=0x5605e874ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.914+0000 7fe4765c7640 1 --2- 172.21.15.70:0/269886720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e873c2c0 0x5605e874ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.914+0000 7fe4765c7640 1 -- 172.21.15.70:0/269886720 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605e874ca30 con 0x5605e874c070 2024-04-01T01:43:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.916+0000 7fe4765c7640 1 --2- 172.21.15.70:0/269886720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e874c070 0x5605e874c4f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe468006010 tx=0x7fe468013ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6c074630f33ba63 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.916+0000 7fe4597fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:43:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.916+0000 7fe4597fa640 1 -- 172.21.15.70:0/269886720 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe468039ca0 con 0x5605e874c070 2024-04-01T01:43:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.916+0000 7fe4597fa640 1 -- 172.21.15.70:0/269886720 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe468039e40 con 0x5605e874c070 2024-04-01T01:43:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.916+0000 7fe47ac0e740 1 -- 172.21.15.70:0/269886720 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe46c0267e0 con 0x5605e874c070 2024-04-01T01:43:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.917+0000 7fe4597fa640 1 -- 172.21.15.70:0/269886720 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe468039800 con 0x5605e874c070 2024-04-01T01:43:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.917+0000 7fe47ac0e740 10 client.28276 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.917+0000 7fe47ac0e740 1 -- 172.21.15.70:0/269886720 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5605e87b7ac0 con 0x5605e874c070 2024-04-01T01:43:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.917+0000 7fe4597fa640 1 -- 172.21.15.70:0/269886720 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe468053070 con 0x5605e874c070 2024-04-01T01:43:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.917+0000 7fe4597fa640 1 -- 172.21.15.70:0/269886720 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe46800b040 con 0x5605e874c070 2024-04-01T01:43:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.917+0000 7fe47ac0e740 10 client.28276 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.917+0000 7fe47ac0e740 10 client.28276 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.917+0000 7fe47ac0e740 20 client.28276 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.918+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/269886720 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5605e87b8440 0x5605e87ba860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.918+0000 7fe47ac0e740 4 client.28276 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:43:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.918+0000 7fe47ac0e740 1 -- 172.21.15.70:0/269886720 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fe46c0267e0 con 0x5605e87b8440 2024-04-01T01:43:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.918+0000 7fe475dc6640 1 --2- 172.21.15.70:0/269886720 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5605e87b8440 0x5605e87ba860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.919+0000 7fe475dc6640 1 --2- 172.21.15.70:0/269886720 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5605e87b8440 0x5605e87ba860 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d0a3ccb6212773c server_cookie=eb54a80b587a395a in_seq=0 out_seq=0 2024-04-01T01:43:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.919+0000 7fe4597fa640 10 client.28276 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:33.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.944+0000 7fe4597fa640 1 -- 172.21.15.70:0/269886720 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fe46c0267e0 con 0x5605e87b8440 2024-04-01T01:43:33.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.944+0000 7fe4597fa640 10 client.28276 handle_command_reply: tid=0 2024-04-01T01:43:33.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.944+0000 7fe47ac0e740 1 client.28276 shutdown 2024-04-01T01:43:33.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.946+0000 7fe47ac0e740 1 -- 172.21.15.70:0/269886720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e874c070 msgr2=0x5605e874c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:33.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.946+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/269886720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e874c070 0x5605e874c4f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe468006010 tx=0x7fe468013ec0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.946+0000 7fe47ac0e740 1 -- 172.21.15.70:0/269886720 shutdown_connections 2024-04-01T01:43:33.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.946+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/269886720 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5605e87b8440 0x5605e87ba860 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.946+0000 7fe4597fa640 0 client.28276 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.946+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/269886720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5605e874c070 0x5605e874c4f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.946+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/269886720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605e8741430 0x5605e874b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.946+0000 7fe47ac0e740 1 --2- 172.21.15.70:0/269886720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605e873c2c0 0x5605e874ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.946+0000 7fe47ac0e740 1 -- 172.21.15.70:0/269886720 >> 172.21.15.70:0/269886720 conn(0x5605e8736e40 msgr2=0x5605e8735f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.947+0000 7fe47ac0e740 1 -- 172.21.15.70:0/269886720 shutdown_connections 2024-04-01T01:43:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.947+0000 7fe47ac0e740 1 -- 172.21.15.70:0/269886720 wait complete. 2024-04-01T01:43:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.947+0000 7fe47ac0e740 20 client.28276 trim_cache size 0 max 16384 2024-04-01T01:43:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:43:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:43:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:43:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:43:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.950+0000 7fe47802a640 1 -- 172.21.15.70:0/2738043277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe45c0a25c0 msgr2=0x7fe45c0a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.950+0000 7fe47802a640 1 --2- 172.21.15.70:0/2738043277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe45c0a25c0 0x7fe45c0a4a80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x5605e8737db0 tx=0x7fe46c005f90 comp rx=0 tx=0).stop 2024-04-01T01:43:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.950+0000 7fe47802a640 1 -- 172.21.15.70:0/2738043277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe470102a30 msgr2=0x7fe470070530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.950+0000 7fe47802a640 1 --2- 172.21.15.70:0/2738043277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe470102a30 0x7fe470070530 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe46000bbb0 tx=0x7fe460008600 comp rx=0 tx=0).stop 2024-04-01T01:43:33.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.951+0000 7fe47802a640 1 -- 172.21.15.70:0/2738043277 shutdown_connections 2024-04-01T01:43:33.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.951+0000 7fe47802a640 1 --2- 172.21.15.70:0/2738043277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4701033f0 0x7fe4701a5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.951+0000 7fe47802a640 1 --2- 172.21.15.70:0/2738043277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe45c0a25c0 0x7fe45c0a4a80 secure :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0x5605e8737db0 tx=0x7fe46c005f90 comp rx=0 tx=0).stop 2024-04-01T01:43:33.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.951+0000 7fe47802a640 1 --2- 172.21.15.70:0/2738043277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe470102a30 0x7fe470070530 secure :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe46000bbb0 tx=0x7fe460008600 comp rx=0 tx=0).stop 2024-04-01T01:43:33.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.951+0000 7fe47802a640 1 --2- 172.21.15.70:0/2738043277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4700ffa60 0x7fe47006fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:33.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.951+0000 7fe47802a640 1 -- 172.21.15.70:0/2738043277 >> 172.21.15.70:0/2738043277 conn(0x7fe4700fb3b0 msgr2=0x7fe470105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:33.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.953+0000 7fe47802a640 1 -- 172.21.15.70:0/2738043277 shutdown_connections 2024-04-01T01:43:33.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:33.954+0000 7fe47802a640 1 -- 172.21.15.70:0/2738043277 wait complete. 2024-04-01T01:43:35.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:34 smithi189 ceph-mon[35753]: pgmap v662: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 6.4 MiB/s wr, 759 op/s 2024-04-01T01:43:35.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:34 smithi070 ceph-mon[32179]: pgmap v662: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 6.4 MiB/s wr, 759 op/s 2024-04-01T01:43:35.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:34 smithi081 ceph-mon[35142]: pgmap v662: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 6.4 MiB/s wr, 759 op/s 2024-04-01T01:43:36.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 530.0 s, 866.6 tps, lat 115.567 ms stddev 65.846 2024-04-01T01:43:37.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:36 smithi189 ceph-mon[35753]: pgmap v663: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.1 MiB/s wr, 822 op/s 2024-04-01T01:43:37.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:36 smithi070 ceph-mon[32179]: pgmap v663: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.1 MiB/s wr, 822 op/s 2024-04-01T01:43:37.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:36 smithi081 ceph-mon[35142]: pgmap v663: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.1 MiB/s wr, 822 op/s 2024-04-01T01:43:39.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:38 smithi189 ceph-mon[35753]: pgmap v664: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.4 MiB/s wr, 852 op/s 2024-04-01T01:43:39.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:38 smithi070 ceph-mon[32179]: pgmap v664: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.4 MiB/s wr, 852 op/s 2024-04-01T01:43:39.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:38 smithi081 ceph-mon[35142]: pgmap v664: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.4 MiB/s wr, 852 op/s 2024-04-01T01:43:41.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:40 smithi189 ceph-mon[35753]: pgmap v665: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.7 MiB/s wr, 769 op/s 2024-04-01T01:43:41.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:40 smithi070 ceph-mon[32179]: pgmap v665: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.7 MiB/s wr, 769 op/s 2024-04-01T01:43:41.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:40 smithi081 ceph-mon[35142]: pgmap v665: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.7 MiB/s wr, 769 op/s 2024-04-01T01:43:41.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 535.0 s, 818.2 tps, lat 120.998 ms stddev 129.032 2024-04-01T01:43:43.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:42 smithi189 ceph-mon[35753]: pgmap v666: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.9 MiB/s wr, 886 op/s 2024-04-01T01:43:43.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:42 smithi070 ceph-mon[32179]: pgmap v666: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.9 MiB/s wr, 886 op/s 2024-04-01T01:43:43.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:42 smithi081 ceph-mon[35142]: pgmap v666: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.9 MiB/s wr, 886 op/s 2024-04-01T01:43:45.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:44 smithi189 ceph-mon[35753]: pgmap v667: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.2 MiB/s wr, 807 op/s 2024-04-01T01:43:45.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:44 smithi070 ceph-mon[32179]: pgmap v667: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.2 MiB/s wr, 807 op/s 2024-04-01T01:43:45.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:44 smithi081 ceph-mon[35142]: pgmap v667: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.2 MiB/s wr, 807 op/s 2024-04-01T01:43:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 540.0 s, 946.4 tps, lat 106.431 ms stddev 88.718 2024-04-01T01:43:47.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.113+0000 7f53908bf640 1 -- 172.21.15.70:0/3224682384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f538c06d340 msgr2=0x7f538c06d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:47.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.113+0000 7f53908bf640 1 --2- 172.21.15.70:0/3224682384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f538c06d340 0x7f538c06d760 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f537c009900 tx=0x7f537c02bc80 comp rx=0 tx=0).stop 2024-04-01T01:43:47.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.117+0000 7f53908bf640 1 -- 172.21.15.70:0/3224682384 shutdown_connections 2024-04-01T01:43:47.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.117+0000 7f53908bf640 1 --2- 172.21.15.70:0/3224682384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f538c06ef20 0x7f538c06f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.118+0000 7f53908bf640 1 --2- 172.21.15.70:0/3224682384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f538c06e580 0x7f538c06e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.118+0000 7f53908bf640 1 --2- 172.21.15.70:0/3224682384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f538c06d340 0x7f538c06d760 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.118+0000 7f53908bf640 1 -- 172.21.15.70:0/3224682384 >> 172.21.15.70:0/3224682384 conn(0x7f538c06c030 msgr2=0x7f538c06c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:47.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.121+0000 7f53908bf640 1 -- 172.21.15.70:0/3224682384 shutdown_connections 2024-04-01T01:43:47.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.121+0000 7f53908bf640 1 -- 172.21.15.70:0/3224682384 wait complete. 2024-04-01T01:43:47.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.122+0000 7f53908bf640 1 Processor -- start 2024-04-01T01:43:47.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.122+0000 7f53908bf640 1 -- start start 2024-04-01T01:43:47.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.122+0000 7f53908bf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f538c06d340 0x7f538c1a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.122+0000 7f53908bf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f538c06e580 0x7f538c1a6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.122+0000 7f53908bf640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f538c06ef20 0x7f538c1add60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.122+0000 7f53908bf640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f538c11c040 con 0x7f538c06d340 2024-04-01T01:43:47.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.122+0000 7f53908bf640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f538c11bec0 con 0x7f538c06e580 2024-04-01T01:43:47.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.122+0000 7f53908bf640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f538c11c1c0 con 0x7f538c06ef20 2024-04-01T01:43:47.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.123+0000 7f538ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f538c06d340 0x7f538c1a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.123+0000 7f538b59e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f538c06ef20 0x7f538c1add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.123+0000 7f538b59e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f538c06ef20 0x7f538c1add60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42276/0 (socket says 172.21.15.70:42276) 2024-04-01T01:43:47.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.123+0000 7f538ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f538c06d340 0x7f538c1a67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47366/0 (socket says 172.21.15.70:47366) 2024-04-01T01:43:47.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.123+0000 7f538b59e640 1 -- 172.21.15.70:0/1538772057 learned_addr learned my addr 172.21.15.70:0/1538772057 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:47.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.123+0000 7f538b59e640 1 -- 172.21.15.70:0/1538772057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f538c06e580 msgr2=0x7f538c1a6cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:43:47.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.123+0000 7f538b59e640 1 --2- 172.21.15.70:0/1538772057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f538c06e580 0x7f538c1a6cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.123+0000 7f538b59e640 1 -- 172.21.15.70:0/1538772057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f538c06d340 msgr2=0x7f538c1a67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.123+0000 7f538b59e640 1 --2- 172.21.15.70:0/1538772057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f538c06d340 0x7f538c1a67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.123+0000 7f538b59e640 1 -- 172.21.15.70:0/1538772057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f538c10bc60 con 0x7f538c06ef20 2024-04-01T01:43:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.124+0000 7f538b59e640 1 --2- 172.21.15.70:0/1538772057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f538c06ef20 0x7f538c1add60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f538400b9e0 tx=0x7f538400bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.124+0000 7f5373fff640 1 -- 172.21.15.70:0/1538772057 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5384019070 con 0x7f538c06ef20 2024-04-01T01:43:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.124+0000 7f53908bf640 1 -- 172.21.15.70:0/1538772057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f538c10bf50 con 0x7f538c06ef20 2024-04-01T01:43:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.124+0000 7f53908bf640 1 -- 172.21.15.70:0/1538772057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f538c10c520 con 0x7f538c06ef20 2024-04-01T01:43:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.124+0000 7f5373fff640 1 -- 172.21.15.70:0/1538772057 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5384014440 con 0x7f538c06ef20 2024-04-01T01:43:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.124+0000 7f5373fff640 1 -- 172.21.15.70:0/1538772057 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53840075d0 con 0x7f538c06ef20 2024-04-01T01:43:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.127+0000 7f5373fff640 1 -- 172.21.15.70:0/1538772057 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f538401c070 con 0x7f538c06ef20 2024-04-01T01:43:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.127+0000 7f5393d7e740 1 Processor -- start 2024-04-01T01:43:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.127+0000 7f5393d7e740 1 -- start start 2024-04-01T01:43:47.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.128+0000 7f5393d7e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3cae0 0x5589fca3cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.128+0000 7f5393d7e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca3d4d0 0x5589fca41fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.128+0000 7f5393d7e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 0x5589fca44ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.128+0000 7f5393d7e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5589fc9a08f0 con 0x5589fca3d4d0 2024-04-01T01:43:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.128+0000 7f5393d7e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5589fc9498d0 con 0x5589fca3cae0 2024-04-01T01:43:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.128+0000 7f5393d7e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5589fc99c990 con 0x5589fca42640 2024-04-01T01:43:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.128+0000 7f538ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca3d4d0 0x5589fca41fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.128+0000 7f5373fff640 1 --2- 172.21.15.70:0/1538772057 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53740a2590 0x7f53740a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.128+0000 7f538a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3cae0 0x5589fca3cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.128+0000 7f538a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3cae0 0x5589fca3cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41560/0 (socket says 172.21.15.70:41560) 2024-04-01T01:43:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.128+0000 7f538a59c640 1 -- 172.21.15.70:0/326630005 learned_addr learned my addr 172.21.15.70:0/326630005 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.128+0000 7f538a59c640 1 --2- 172.21.15.70:0/326630005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 0x5589fca44ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.128+0000 7f538b59e640 1 --2- 172.21.15.70:0/1538772057 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53740a2590 0x7f53740a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.129+0000 7f5373fff640 1 -- 172.21.15.70:0/1538772057 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f53840ce030 con 0x7f538c06ef20 2024-04-01T01:43:47.135 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.129+0000 7f538a59c640 1 -- 172.21.15.70:0/326630005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 msgr2=0x5589fca44ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.129+0000 7f538a59c640 1 --2- 172.21.15.70:0/326630005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 0x5589fca44ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.129+0000 7f538a59c640 1 -- 172.21.15.70:0/326630005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca3d4d0 msgr2=0x5589fca41fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.129+0000 7f538a59c640 1 --2- 172.21.15.70:0/326630005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca3d4d0 0x5589fca41fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.129+0000 7f538a59c640 1 -- 172.21.15.70:0/326630005 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589fc6b5210 con 0x5589fca3cae0 2024-04-01T01:43:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.129+0000 7f538a59c640 1 --2- 172.21.15.70:0/326630005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3cae0 0x5589fca3cf00 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f53800143a0 tx=0x7f538000b0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa4dd80d90121848 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.129+0000 7f538b59e640 1 --2- 172.21.15.70:0/1538772057 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53740a2590 0x7f53740a4a50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5384021f10 tx=0x7f538406b5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.129+0000 7f53717fa640 1 -- 172.21.15.70:0/326630005 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5380011040 con 0x5589fca3cae0 2024-04-01T01:43:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.129+0000 7f53717fa640 1 -- 172.21.15.70:0/326630005 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53800065a0 con 0x5589fca3cae0 2024-04-01T01:43:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.130+0000 7f53717fa640 1 -- 172.21.15.70:0/326630005 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5380018690 con 0x5589fca3cae0 2024-04-01T01:43:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.130+0000 7f5393d7e740 1 -- 172.21.15.70:0/326630005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3cae0 msgr2=0x5589fca3cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.130+0000 7f5393d7e740 1 --2- 172.21.15.70:0/326630005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3cae0 0x5589fca3cf00 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f53800143a0 tx=0x7f538000b0a0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.131+0000 7f5393d7e740 1 -- 172.21.15.70:0/326630005 shutdown_connections 2024-04-01T01:43:47.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.131+0000 7f5393d7e740 1 --2- 172.21.15.70:0/326630005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 0x5589fca44ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.131+0000 7f5393d7e740 1 --2- 172.21.15.70:0/326630005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca3d4d0 0x5589fca41fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.131+0000 7f5393d7e740 1 --2- 172.21.15.70:0/326630005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3cae0 0x5589fca3cf00 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.131+0000 7f5393d7e740 1 -- 172.21.15.70:0/326630005 >> 172.21.15.70:0/326630005 conn(0x5589fca38050 msgr2=0x5589fca38450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.131+0000 7f5393d7e740 1 -- 172.21.15.70:0/326630005 shutdown_connections 2024-04-01T01:43:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.131+0000 7f5393d7e740 1 -- 172.21.15.70:0/326630005 wait complete. 2024-04-01T01:43:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.131+0000 7f5393d7e740 1 -- start start 2024-04-01T01:43:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.131+0000 7f5393d7e740 1 Processor -- start 2024-04-01T01:43:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f5393d7e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3cae0 0x5589fca46de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f5393d7e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca3d4d0 0x5589fca47320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f5393d7e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 0x5589fca4e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f5393d7e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5589fca45590 con 0x5589fca3d4d0 2024-04-01T01:43:47.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f5393d7e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5589fc93ef80 con 0x5589fca3cae0 2024-04-01T01:43:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f5393d7e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5589fca45710 con 0x5589fca42640 2024-04-01T01:43:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f5393d7e740 10 client.0 _ll_register_callbacks cb 0x5589fc997290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f538a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3cae0 0x5589fca46de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f538a59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3cae0 0x5589fca46de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41572/0 (socket says 172.21.15.70:41572) 2024-04-01T01:43:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f538a59c640 1 -- 172.21.15.70:0/3554900773 learned_addr learned my addr 172.21.15.70:0/3554900773 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f538a59c640 1 --2- 172.21.15.70:0/3554900773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 0x5589fca4e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f538ad9d640 1 --2- 172.21.15.70:0/3554900773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca3d4d0 0x5589fca47320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f538a59c640 1 -- 172.21.15.70:0/3554900773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 msgr2=0x5589fca4e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f538a59c640 1 --2- 172.21.15.70:0/3554900773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 0x5589fca4e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f538a59c640 1 -- 172.21.15.70:0/3554900773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca3d4d0 msgr2=0x5589fca47320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f538a59c640 1 --2- 172.21.15.70:0/3554900773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca3d4d0 0x5589fca47320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.132+0000 7f538a59c640 1 -- 172.21.15.70:0/3554900773 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589fc781620 con 0x5589fca3cae0 2024-04-01T01:43:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.133+0000 7f538a59c640 1 --2- 172.21.15.70:0/3554900773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3cae0 0x5589fca46de0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f538001a040 tx=0x7f538000a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82d38b7e918a5fce server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.133+0000 7f536eff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.133+0000 7f536eff5640 1 -- 172.21.15.70:0/3554900773 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538002a030 con 0x5589fca3cae0 2024-04-01T01:43:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.133+0000 7f536eff5640 1 -- 172.21.15.70:0/3554900773 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f538001e040 con 0x5589fca3cae0 2024-04-01T01:43:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.133+0000 7f5393d7e740 1 -- 172.21.15.70:0/3554900773 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53800065a0 con 0x5589fca3cae0 2024-04-01T01:43:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.133+0000 7f536eff5640 1 -- 172.21.15.70:0/3554900773 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5380004e60 con 0x5589fca3cae0 2024-04-01T01:43:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.133+0000 7f5393d7e740 10 client.28481 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:47.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.133+0000 7f5393d7e740 1 -- 172.21.15.70:0/3554900773 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5589fca51350 con 0x5589fca3cae0 2024-04-01T01:43:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.133+0000 7f536eff5640 1 -- 172.21.15.70:0/3554900773 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53800065a0 con 0x5589fca3cae0 2024-04-01T01:43:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.134+0000 7f536eff5640 1 -- 172.21.15.70:0/3554900773 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5380018990 con 0x5589fca3cae0 2024-04-01T01:43:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.134+0000 7f5393d7e740 10 client.28481 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.134+0000 7f5393d7e740 10 client.28481 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.134+0000 7f5393d7e740 20 client.28481 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.134+0000 7f5393d7e740 1 --2- 172.21.15.70:0/3554900773 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5589fca52900 0x5589fca54d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.134+0000 7f5393d7e740 4 client.28481 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.134+0000 7f5393d7e740 1 -- 172.21.15.70:0/3554900773 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589fc9b4120 con 0x5589fca52900 2024-04-01T01:43:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.134+0000 7f538ad9d640 1 --2- 172.21.15.70:0/3554900773 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5589fca52900 0x5589fca54d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.135+0000 7f536eff5640 10 client.28481 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.135+0000 7f538ad9d640 1 --2- 172.21.15.70:0/3554900773 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5589fca52900 0x5589fca54d20 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46ed0676cad3d11a server_cookie=91a6114394d8e8f4 in_seq=0 out_seq=0 2024-04-01T01:43:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.135+0000 7f536eff5640 1 -- 172.21.15.70:0/3554900773 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5589fc9b4120 con 0x5589fca52900 2024-04-01T01:43:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.135+0000 7f536eff5640 10 client.28481 handle_command_reply: tid=0 2024-04-01T01:43:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.136+0000 7f5393d7e740 1 client.28481 shutdown 2024-04-01T01:43:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.136+0000 7f5393d7e740 1 -- 172.21.15.70:0/3554900773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3cae0 msgr2=0x5589fca46de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.136+0000 7f5393d7e740 1 --2- 172.21.15.70:0/3554900773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3cae0 0x5589fca46de0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f538001a040 tx=0x7f538000a040 comp rx=0 tx=0).stop 2024-04-01T01:43:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.136+0000 7f5393d7e740 1 -- 172.21.15.70:0/3554900773 shutdown_connections 2024-04-01T01:43:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.136+0000 7f5393d7e740 1 --2- 172.21.15.70:0/3554900773 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5589fca52900 0x5589fca54d20 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.136+0000 7f5393d7e740 1 --2- 172.21.15.70:0/3554900773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 0x5589fca4e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.136+0000 7f5393d7e740 1 --2- 172.21.15.70:0/3554900773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca3d4d0 0x5589fca47320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.136+0000 7f5393d7e740 1 --2- 172.21.15.70:0/3554900773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3cae0 0x5589fca46de0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.136+0000 7f536eff5640 0 client.28481 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.136+0000 7f5393d7e740 1 -- 172.21.15.70:0/3554900773 >> 172.21.15.70:0/3554900773 conn(0x5589fca38050 msgr2=0x5589fca37190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.136+0000 7f5393d7e740 1 -- 172.21.15.70:0/3554900773 shutdown_connections 2024-04-01T01:43:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.136+0000 7f5393d7e740 1 -- 172.21.15.70:0/3554900773 wait complete. 2024-04-01T01:43:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.136+0000 7f5393d7e740 20 client.28481 trim_cache size 0 max 16384 2024-04-01T01:43:47.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.160+0000 7f5393d7e740 1 Processor -- start 2024-04-01T01:43:47.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.161+0000 7f5393d7e740 1 -- start start 2024-04-01T01:43:47.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.161+0000 7f5393d7e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca3cae0 0x5589fca5a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.161+0000 7f5393d7e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3d4d0 0x5589fca4e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.161+0000 7f5393d7e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca42640 0x5589fca460b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.161+0000 7f5393d7e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f537800fcc0 con 0x5589fca42640 2024-04-01T01:43:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.161+0000 7f5393d7e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5378011800 con 0x5589fca3d4d0 2024-04-01T01:43:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.161+0000 7f5393d7e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5378010390 con 0x5589fca3cae0 2024-04-01T01:43:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.161+0000 7f538a59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca3cae0 0x5589fca5a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.161+0000 7f538a59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca3cae0 0x5589fca5a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42288/0 (socket says 172.21.15.70:42288) 2024-04-01T01:43:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.161+0000 7f538a59c640 1 -- 172.21.15.70:0/3623281041 learned_addr learned my addr 172.21.15.70:0/3623281041 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.161+0000 7f538ad9d640 1 --2- 172.21.15.70:0/3623281041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3d4d0 0x5589fca4e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.162+0000 7f538a59c640 1 --2- 172.21.15.70:0/3623281041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca42640 0x5589fca460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.162+0000 7f538ad9d640 1 -- 172.21.15.70:0/3623281041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca3cae0 msgr2=0x5589fca5a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.162+0000 7f538ad9d640 1 --2- 172.21.15.70:0/3623281041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca3cae0 0x5589fca5a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.162+0000 7f538ad9d640 1 -- 172.21.15.70:0/3623281041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca42640 msgr2=0x5589fca460b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.162+0000 7f538ad9d640 1 --2- 172.21.15.70:0/3623281041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca42640 0x5589fca460b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.162+0000 7f538ad9d640 1 -- 172.21.15.70:0/3623281041 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589fca3c720 con 0x5589fca3d4d0 2024-04-01T01:43:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.162+0000 7f538ad9d640 1 --2- 172.21.15.70:0/3623281041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3d4d0 0x5589fca4e2e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f537c02c1c0 tx=0x7f537c02c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=650a50e1fc60b737 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:47.164 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.162+0000 7f5370ff9640 1 -- 172.21.15.70:0/3623281041 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c047050 con 0x5589fca3d4d0 2024-04-01T01:43:47.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.162+0000 7f5370ff9640 1 -- 172.21.15.70:0/3623281041 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f537c02ccd0 con 0x5589fca3d4d0 2024-04-01T01:43:47.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.163+0000 7f5370ff9640 1 -- 172.21.15.70:0/3623281041 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c049550 con 0x5589fca3d4d0 2024-04-01T01:43:47.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.163+0000 7f5393d7e740 1 -- 172.21.15.70:0/3623281041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3d4d0 msgr2=0x5589fca4e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:47.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.163+0000 7f5393d7e740 1 --2- 172.21.15.70:0/3623281041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3d4d0 0x5589fca4e2e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f537c02c1c0 tx=0x7f537c02c830 comp rx=0 tx=0).stop 2024-04-01T01:43:47.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.164+0000 7f5393d7e740 1 -- 172.21.15.70:0/3623281041 shutdown_connections 2024-04-01T01:43:47.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.164+0000 7f5393d7e740 1 --2- 172.21.15.70:0/3623281041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca42640 0x5589fca460b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.164+0000 7f5393d7e740 1 --2- 172.21.15.70:0/3623281041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3d4d0 0x5589fca4e2e0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.164+0000 7f5393d7e740 1 --2- 172.21.15.70:0/3623281041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca3cae0 0x5589fca5a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.164+0000 7f5393d7e740 1 -- 172.21.15.70:0/3623281041 >> 172.21.15.70:0/3623281041 conn(0x5589fca38050 msgr2=0x5589fca3b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:47.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.164+0000 7f5393d7e740 1 -- 172.21.15.70:0/3623281041 shutdown_connections 2024-04-01T01:43:47.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.164+0000 7f5393d7e740 1 -- 172.21.15.70:0/3623281041 wait complete. 2024-04-01T01:43:47.165 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.164+0000 7f5393d7e740 1 -- start start 2024-04-01T01:43:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.164+0000 7f5393d7e740 1 Processor -- start 2024-04-01T01:43:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.165+0000 7f5393d7e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca3cae0 0x5589fca4c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.165+0000 7f5393d7e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3d4d0 0x5589fca4c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.165+0000 7f5393d7e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 0x5589fca4d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.165+0000 7f5393d7e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5378011130 con 0x5589fca3cae0 2024-04-01T01:43:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.165+0000 7f5393d7e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5378010a60 con 0x5589fca3d4d0 2024-04-01T01:43:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.165+0000 7f5393d7e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f537800dca0 con 0x5589fca42640 2024-04-01T01:43:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.165+0000 7f5393d7e740 10 client.0 _ll_register_callbacks cb 0x7f538000ba50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.165+0000 7f538a59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 0x5589fca4d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.166 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.165+0000 7f538a59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 0x5589fca4d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42298/0 (socket says 172.21.15.70:42298) 2024-04-01T01:43:47.167 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.165+0000 7f538a59c640 1 -- 172.21.15.70:0/163856982 learned_addr learned my addr 172.21.15.70:0/163856982 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:47.167 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.165+0000 7f538ad9d640 1 --2- 172.21.15.70:0/163856982 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3d4d0 0x5589fca4c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.167 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.165+0000 7f538a59c640 1 --2- 172.21.15.70:0/163856982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca3cae0 0x5589fca4c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.166+0000 7f538a59c640 1 -- 172.21.15.70:0/163856982 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3d4d0 msgr2=0x5589fca4c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.166+0000 7f538a59c640 1 --2- 172.21.15.70:0/163856982 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3d4d0 0x5589fca4c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.166+0000 7f538a59c640 1 -- 172.21.15.70:0/163856982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca3cae0 msgr2=0x5589fca4c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.166+0000 7f538a59c640 1 --2- 172.21.15.70:0/163856982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca3cae0 0x5589fca4c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.166+0000 7f538a59c640 1 -- 172.21.15.70:0/163856982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589fca4dc30 con 0x5589fca42640 2024-04-01T01:43:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.167+0000 7f538a59c640 1 --2- 172.21.15.70:0/163856982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 0x5589fca4d640 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5380018f20 tx=0x7f538000a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=453f10a8c9c257d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.167+0000 7f536eff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:43:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.167+0000 7f536eff5640 1 -- 172.21.15.70:0/163856982 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538002a040 con 0x5589fca42640 2024-04-01T01:43:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.167+0000 7f536eff5640 1 -- 172.21.15.70:0/163856982 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5380018870 con 0x5589fca42640 2024-04-01T01:43:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.167+0000 7f5393d7e740 1 -- 172.21.15.70:0/163856982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5589fc9ade30 con 0x5589fca42640 2024-04-01T01:43:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.168+0000 7f536eff5640 1 -- 172.21.15.70:0/163856982 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5380018a70 con 0x5589fca42640 2024-04-01T01:43:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.168+0000 7f5393d7e740 10 client.28291 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.168+0000 7f5393d7e740 1 -- 172.21.15.70:0/163856982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5589fcab92d0 con 0x5589fca42640 2024-04-01T01:43:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.168+0000 7f536eff5640 1 -- 172.21.15.70:0/163856982 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5589fc9ade30 con 0x5589fca42640 2024-04-01T01:43:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.168+0000 7f536eff5640 1 -- 172.21.15.70:0/163856982 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f53800063a0 con 0x5589fca42640 2024-04-01T01:43:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.169+0000 7f5393d7e740 10 client.28291 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.169+0000 7f5393d7e740 10 client.28291 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.169+0000 7f5393d7e740 20 client.28291 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.169+0000 7f5393d7e740 1 --2- 172.21.15.70:0/163856982 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5589fcab9c20 0x5589fcabc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.169+0000 7f5393d7e740 4 client.28291 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:43:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.169+0000 7f5393d7e740 1 -- 172.21.15.70:0/163856982 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5589fc9a8180 con 0x5589fcab9c20 2024-04-01T01:43:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.169+0000 7f538ad9d640 1 --2- 172.21.15.70:0/163856982 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5589fcab9c20 0x5589fcabc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.170+0000 7f536eff5640 10 client.28291 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.170+0000 7f538ad9d640 1 --2- 172.21.15.70:0/163856982 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5589fcab9c20 0x5589fcabc040 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3c8518659a2de84 server_cookie=f12ee638b7613f5c in_seq=0 out_seq=0 2024-04-01T01:43:47.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.219+0000 7f536eff5640 1 -- 172.21.15.70:0/163856982 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5589fc9a8180 con 0x5589fcab9c20 2024-04-01T01:43:47.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.219+0000 7f536eff5640 10 client.28291 handle_command_reply: tid=0 2024-04-01T01:43:47.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.219+0000 7f5393d7e740 1 client.28291 shutdown 2024-04-01T01:43:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.222+0000 7f5393d7e740 1 -- 172.21.15.70:0/163856982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 msgr2=0x5589fca4d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.222+0000 7f5393d7e740 1 --2- 172.21.15.70:0/163856982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 0x5589fca4d640 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5380018f20 tx=0x7f538000a040 comp rx=0 tx=0).stop 2024-04-01T01:43:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.223+0000 7f5393d7e740 1 -- 172.21.15.70:0/163856982 shutdown_connections 2024-04-01T01:43:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.223+0000 7f5393d7e740 1 --2- 172.21.15.70:0/163856982 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5589fcab9c20 0x5589fcabc040 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.223+0000 7f5393d7e740 1 --2- 172.21.15.70:0/163856982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589fca42640 0x5589fca4d640 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.223+0000 7f5393d7e740 1 --2- 172.21.15.70:0/163856982 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589fca3d4d0 0x5589fca4c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.223+0000 7f5393d7e740 1 --2- 172.21.15.70:0/163856982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5589fca3cae0 0x5589fca4c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.223+0000 7f5393d7e740 1 -- 172.21.15.70:0/163856982 >> 172.21.15.70:0/163856982 conn(0x5589fca38050 msgr2=0x5589fca3b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:47.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.223+0000 7f536eff5640 0 client.28291 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.224+0000 7f5393d7e740 1 -- 172.21.15.70:0/163856982 shutdown_connections 2024-04-01T01:43:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.224+0000 7f5393d7e740 1 -- 172.21.15.70:0/163856982 wait complete. 2024-04-01T01:43:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.224+0000 7f5393d7e740 20 client.28291 trim_cache size 0 max 16384 2024-04-01T01:43:47.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:43:47.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:43:47.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:43:47.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:43:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.228+0000 7f53908bf640 1 -- 172.21.15.70:0/1538772057 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53740a2590 msgr2=0x7f53740a4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:47.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.228+0000 7f53908bf640 1 --2- 172.21.15.70:0/1538772057 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53740a2590 0x7f53740a4a50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f5384021f10 tx=0x7f538406b5b0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.228+0000 7f53908bf640 1 -- 172.21.15.70:0/1538772057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f538c06ef20 msgr2=0x7f538c1add60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.228+0000 7f53908bf640 1 --2- 172.21.15.70:0/1538772057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f538c06ef20 0x7f538c1add60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f538400b9e0 tx=0x7f538400bea0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.228+0000 7f53908bf640 1 -- 172.21.15.70:0/1538772057 shutdown_connections 2024-04-01T01:43:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.228+0000 7f53908bf640 1 --2- 172.21.15.70:0/1538772057 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f53740a2590 0x7f53740a4a50 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.228+0000 7f53908bf640 1 --2- 172.21.15.70:0/1538772057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f538c06ef20 0x7f538c1add60 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.228+0000 7f53908bf640 1 --2- 172.21.15.70:0/1538772057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f538c06e580 0x7f538c1a6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.228+0000 7f53908bf640 1 --2- 172.21.15.70:0/1538772057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f538c06d340 0x7f538c1a67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:47.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.228+0000 7f53908bf640 1 -- 172.21.15.70:0/1538772057 >> 172.21.15.70:0/1538772057 conn(0x7f538c06c030 msgr2=0x7f538c10a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:47.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.230+0000 7f53908bf640 1 -- 172.21.15.70:0/1538772057 shutdown_connections 2024-04-01T01:43:47.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:43:47.230+0000 7f53908bf640 1 -- 172.21.15.70:0/1538772057 wait complete. 2024-04-01T01:43:47.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:46 smithi189 ceph-mon[35753]: pgmap v668: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.8 MiB/s wr, 877 op/s 2024-04-01T01:43:47.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:46 smithi070 ceph-mon[32179]: pgmap v668: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.8 MiB/s wr, 877 op/s 2024-04-01T01:43:47.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:46 smithi081 ceph-mon[35142]: pgmap v668: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.8 MiB/s wr, 877 op/s 2024-04-01T01:43:49.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:48 smithi189 ceph-mon[35753]: pgmap v669: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.9 MiB/s wr, 889 op/s 2024-04-01T01:43:49.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:48 smithi070 ceph-mon[32179]: pgmap v669: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.9 MiB/s wr, 889 op/s 2024-04-01T01:43:49.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:48 smithi081 ceph-mon[35142]: pgmap v669: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.9 MiB/s wr, 889 op/s 2024-04-01T01:43:49.869 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:43:49.869 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:43:50.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.055+0000 7f5723b19640 1 -- 172.21.15.70:0/1442689590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f571c107d70 msgr2=0x7f571c108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:50.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.055+0000 7f5723b19640 1 --2- 172.21.15.70:0/1442689590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f571c107d70 0x7f571c108190 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f570c009900 tx=0x7f570c02bc40 comp rx=0 tx=0).stop 2024-04-01T01:43:50.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.055+0000 7f5723b19640 1 -- 172.21.15.70:0/1442689590 shutdown_connections 2024-04-01T01:43:50.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.055+0000 7f5723b19640 1 --2- 172.21.15.70:0/1442689590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f571c109920 0x7f571c1101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.055+0000 7f5723b19640 1 --2- 172.21.15.70:0/1442689590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571c108f60 0x7f571c1093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.055+0000 7f5723b19640 1 --2- 172.21.15.70:0/1442689590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f571c107d70 0x7f571c108190 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.055+0000 7f5723b19640 1 -- 172.21.15.70:0/1442689590 >> 172.21.15.70:0/1442689590 conn(0x7f571c076d00 msgr2=0x7f571c079140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:50.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.056+0000 7f5723b19640 1 -- 172.21.15.70:0/1442689590 shutdown_connections 2024-04-01T01:43:50.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.056+0000 7f5723b19640 1 -- 172.21.15.70:0/1442689590 wait complete. 2024-04-01T01:43:50.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.057+0000 7f5723b19640 1 Processor -- start 2024-04-01T01:43:50.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.058+0000 7f5723b19640 1 -- start start 2024-04-01T01:43:50.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.059+0000 7f5723b19640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571c107d70 0x7f571c06ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.059+0000 7f5723b19640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f571c108f60 0x7f571c070510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.059+0000 7f57218b5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571c107d70 0x7f571c06ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.059+0000 7f57218b5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571c107d70 0x7f571c06ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41616/0 (socket says 172.21.15.70:41616) 2024-04-01T01:43:50.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.059+0000 7f5723b19640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f571c109920 0x7f571c1adfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.059+0000 7f57218b5640 1 -- 172.21.15.70:0/4226590877 learned_addr learned my addr 172.21.15.70:0/4226590877 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:50.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.059+0000 7f5723b19640 1 -- 172.21.15.70:0/4226590877 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f571c1138c0 con 0x7f571c108f60 2024-04-01T01:43:50.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.059+0000 7f5723b19640 1 -- 172.21.15.70:0/4226590877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f571c113740 con 0x7f571c107d70 2024-04-01T01:43:50.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.059+0000 7f5723b19640 1 -- 172.21.15.70:0/4226590877 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f571c113a40 con 0x7f571c109920 2024-04-01T01:43:50.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.060+0000 7f57220b6640 1 --2- 172.21.15.70:0/4226590877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f571c109920 0x7f571c1adfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.060+0000 7f57218b5640 1 -- 172.21.15.70:0/4226590877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f571c109920 msgr2=0x7f571c1adfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:50.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.060+0000 7f57218b5640 1 --2- 172.21.15.70:0/4226590877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f571c109920 0x7f571c1adfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.060+0000 7f57218b5640 1 -- 172.21.15.70:0/4226590877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f571c108f60 msgr2=0x7f571c070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:50.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.060+0000 7f57218b5640 1 --2- 172.21.15.70:0/4226590877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f571c108f60 0x7f571c070510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.060+0000 7f57218b5640 1 -- 172.21.15.70:0/4226590877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571c073690 con 0x7f571c107d70 2024-04-01T01:43:50.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.060+0000 7f57218b5640 1 --2- 172.21.15.70:0/4226590877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571c107d70 0x7f571c06ffd0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f570c009850 tx=0x7f570c004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:50.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.061+0000 7f5712ffd640 1 -- 172.21.15.70:0/4226590877 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c054020 con 0x7f571c107d70 2024-04-01T01:43:50.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.061+0000 7f5712ffd640 1 -- 172.21.15.70:0/4226590877 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f570c004440 con 0x7f571c107d70 2024-04-01T01:43:50.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.061+0000 7f5712ffd640 1 -- 172.21.15.70:0/4226590877 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c003bc0 con 0x7f571c107d70 2024-04-01T01:43:50.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.061+0000 7f5723b19640 1 -- 172.21.15.70:0/4226590877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f571c079fa0 con 0x7f571c107d70 2024-04-01T01:43:50.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.061+0000 7f5723b19640 1 -- 172.21.15.70:0/4226590877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f571c07a440 con 0x7f571c107d70 2024-04-01T01:43:50.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.064+0000 7f5712ffd640 1 -- 172.21.15.70:0/4226590877 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f570c057020 con 0x7f571c107d70 2024-04-01T01:43:50.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.065+0000 7f5712ffd640 1 --2- 172.21.15.70:0/4226590877 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57080a2620 0x7f57080a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.065+0000 7f5712ffd640 1 -- 172.21.15.70:0/4226590877 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f570c101ef0 con 0x7f571c107d70 2024-04-01T01:43:50.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.066+0000 7f57267ce740 1 Processor -- start 2024-04-01T01:43:50.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.066+0000 7f57210b4640 1 --2- 172.21.15.70:0/4226590877 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57080a2620 0x7f57080a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.066+0000 7f57267ce740 1 -- start start 2024-04-01T01:43:50.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.066+0000 7f57267ce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 0x5579166806d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.067+0000 7f57267ce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557916680ca0 0x5579166857a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.067+0000 7f57267ce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916685e10 0x5579166882a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.067+0000 7f57267ce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5579165e5180 con 0x5579166802b0 2024-04-01T01:43:50.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.067+0000 7f57267ce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55791658db70 con 0x557916685e10 2024-04-01T01:43:50.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.067+0000 7f57267ce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5579165e1220 con 0x557916680ca0 2024-04-01T01:43:50.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.067+0000 7f57218b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 0x5579166806d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.067+0000 7f57218b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 0x5579166806d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47420/0 (socket says 172.21.15.70:47420) 2024-04-01T01:43:50.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.067+0000 7f57218b5640 1 -- 172.21.15.70:0/3147315612 learned_addr learned my addr 172.21.15.70:0/3147315612 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:50.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.067+0000 7f57220b6640 1 --2- 172.21.15.70:0/3147315612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916685e10 0x5579166882a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.067+0000 7f57210b4640 1 --2- 172.21.15.70:0/3147315612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557916680ca0 0x5579166857a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.067+0000 7f57218b5640 1 -- 172.21.15.70:0/3147315612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557916680ca0 msgr2=0x5579166857a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:50.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.067+0000 7f57218b5640 1 --2- 172.21.15.70:0/3147315612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557916680ca0 0x5579166857a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.067+0000 7f57218b5640 1 -- 172.21.15.70:0/3147315612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916685e10 msgr2=0x5579166882a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:50.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.067+0000 7f57218b5640 1 --2- 172.21.15.70:0/3147315612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916685e10 0x5579166882a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.067+0000 7f57218b5640 1 -- 172.21.15.70:0/3147315612 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579165a17c0 con 0x5579166802b0 2024-04-01T01:43:50.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.068+0000 7f57210b4640 1 --2- 172.21.15.70:0/4226590877 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57080a2620 0x7f57080a4ae0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f57180058c0 tx=0x7f5718005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:50.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.068+0000 7f57218b5640 1 --2- 172.21.15.70:0/3147315612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 0x5579166806d0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f570c0d6ac0 tx=0x7f570c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680b5bd9634a8c57 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:50.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.068+0000 7f5707fff640 1 -- 172.21.15.70:0/3147315612 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c0ce5e0 con 0x5579166802b0 2024-04-01T01:43:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.068+0000 7f5707fff640 1 -- 172.21.15.70:0/3147315612 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f570c0d2730 con 0x5579166802b0 2024-04-01T01:43:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.068+0000 7f5707fff640 1 -- 172.21.15.70:0/3147315612 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c0ce840 con 0x5579166802b0 2024-04-01T01:43:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.069+0000 7f57267ce740 1 -- 172.21.15.70:0/3147315612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 msgr2=0x5579166806d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.069+0000 7f57267ce740 1 --2- 172.21.15.70:0/3147315612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 0x5579166806d0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f570c0d6ac0 tx=0x7f570c005fd0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.070+0000 7f57267ce740 1 -- 172.21.15.70:0/3147315612 shutdown_connections 2024-04-01T01:43:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.070+0000 7f57267ce740 1 --2- 172.21.15.70:0/3147315612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916685e10 0x5579166882a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.070+0000 7f57267ce740 1 --2- 172.21.15.70:0/3147315612 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557916680ca0 0x5579166857a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.070+0000 7f57267ce740 1 --2- 172.21.15.70:0/3147315612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 0x5579166806d0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.070+0000 7f57267ce740 1 -- 172.21.15.70:0/3147315612 >> 172.21.15.70:0/3147315612 conn(0x55791667b820 msgr2=0x55791667bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.071+0000 7f57267ce740 1 -- 172.21.15.70:0/3147315612 shutdown_connections 2024-04-01T01:43:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.072+0000 7f57267ce740 1 -- 172.21.15.70:0/3147315612 wait complete. 2024-04-01T01:43:50.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.072+0000 7f57267ce740 1 -- start start 2024-04-01T01:43:50.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.072+0000 7f57267ce740 1 Processor -- start 2024-04-01T01:43:50.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.073+0000 7f57267ce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579166802b0 0x55791668a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.073+0000 7f57267ce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557916680ca0 0x55791668a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.073+0000 7f57267ce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916685e10 0x557916691760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.073+0000 7f57267ce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5579165e5180 con 0x557916680ca0 2024-04-01T01:43:50.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.073+0000 7f57267ce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5579165834a0 con 0x557916685e10 2024-04-01T01:43:50.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.073+0000 7f57267ce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557916688b80 con 0x5579166802b0 2024-04-01T01:43:50.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.073+0000 7f57267ce740 10 client.0 _ll_register_callbacks cb 0x557916586d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:50.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.073+0000 7f57218b5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579166802b0 0x55791668a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.074+0000 7f57218b5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579166802b0 0x55791668a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42340/0 (socket says 172.21.15.70:42340) 2024-04-01T01:43:50.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.074+0000 7f57218b5640 1 -- 172.21.15.70:0/1167173561 learned_addr learned my addr 172.21.15.70:0/1167173561 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:50.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.074+0000 7f57220b6640 1 --2- 172.21.15.70:0/1167173561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916685e10 0x557916691760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.074+0000 7f57210b4640 1 --2- 172.21.15.70:0/1167173561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557916680ca0 0x55791668a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.074+0000 7f57218b5640 1 -- 172.21.15.70:0/1167173561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916685e10 msgr2=0x557916691760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:50.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.074+0000 7f57218b5640 1 --2- 172.21.15.70:0/1167173561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916685e10 0x557916691760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.074+0000 7f57218b5640 1 -- 172.21.15.70:0/1167173561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557916680ca0 msgr2=0x55791668a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:50.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.074+0000 7f57218b5640 1 --2- 172.21.15.70:0/1167173561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557916680ca0 0x55791668a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.074+0000 7f57218b5640 1 -- 172.21.15.70:0/1167173561 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579163c37d0 con 0x5579166802b0 2024-04-01T01:43:50.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.074+0000 7f57218b5640 1 --2- 172.21.15.70:0/1167173561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579166802b0 0x55791668a1e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f570c0ff990 tx=0x7f570c003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3092a50115083ac server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:50.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.074+0000 7f57057fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:43:50.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.075+0000 7f57057fa640 1 -- 172.21.15.70:0/1167173561 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c0eb020 con 0x5579166802b0 2024-04-01T01:43:50.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.075+0000 7f57057fa640 1 -- 172.21.15.70:0/1167173561 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f570c0ce840 con 0x5579166802b0 2024-04-01T01:43:50.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.076+0000 7f57267ce740 1 -- 172.21.15.70:0/1167173561 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f570c0d2730 con 0x5579166802b0 2024-04-01T01:43:50.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.076+0000 7f57057fa640 1 -- 172.21.15.70:0/1167173561 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f570c0cea80 con 0x5579166802b0 2024-04-01T01:43:50.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.076+0000 7f57267ce740 10 client.28300 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:50.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.076+0000 7f57267ce740 1 -- 172.21.15.70:0/1167173561 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5579162f93f0 con 0x5579166802b0 2024-04-01T01:43:50.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.077+0000 7f57057fa640 1 -- 172.21.15.70:0/1167173561 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f570c0d2730 con 0x5579166802b0 2024-04-01T01:43:50.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.077+0000 7f57057fa640 1 -- 172.21.15.70:0/1167173561 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f570c0ffae0 con 0x5579166802b0 2024-04-01T01:43:50.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.077+0000 7f57267ce740 10 client.28300 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:50.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.077+0000 7f57267ce740 10 client.28300 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:50.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.077+0000 7f57267ce740 20 client.28300 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:50.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.077+0000 7f57267ce740 1 --2- 172.21.15.70:0/1167173561 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557916695ca0 0x5579166980d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.077+0000 7f57267ce740 4 client.28300 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:50.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.077+0000 7f57267ce740 1 -- 172.21.15.70:0/1167173561 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579165f26c0 con 0x557916695ca0 2024-04-01T01:43:50.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.077+0000 7f57210b4640 1 --2- 172.21.15.70:0/1167173561 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557916695ca0 0x5579166980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.080+0000 7f57057fa640 10 client.28300 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:50.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.080+0000 7f57210b4640 1 --2- 172.21.15.70:0/1167173561 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557916695ca0 0x5579166980d0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b3edbb18a98ba00 server_cookie=800215f691d9cb9a in_seq=0 out_seq=0 2024-04-01T01:43:50.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.081+0000 7f57057fa640 1 -- 172.21.15.70:0/1167173561 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5579165f26c0 con 0x557916695ca0 2024-04-01T01:43:50.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.081+0000 7f57057fa640 10 client.28300 handle_command_reply: tid=0 2024-04-01T01:43:50.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.081+0000 7f57267ce740 1 client.28300 shutdown 2024-04-01T01:43:50.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.081+0000 7f57267ce740 1 -- 172.21.15.70:0/1167173561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579166802b0 msgr2=0x55791668a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:50.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.081+0000 7f57267ce740 1 --2- 172.21.15.70:0/1167173561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579166802b0 0x55791668a1e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f570c0ff990 tx=0x7f570c003f90 comp rx=0 tx=0).stop 2024-04-01T01:43:50.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.082+0000 7f57267ce740 1 -- 172.21.15.70:0/1167173561 shutdown_connections 2024-04-01T01:43:50.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.082+0000 7f57267ce740 1 --2- 172.21.15.70:0/1167173561 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557916695ca0 0x5579166980d0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.082+0000 7f57267ce740 1 --2- 172.21.15.70:0/1167173561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916685e10 0x557916691760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.082+0000 7f57267ce740 1 --2- 172.21.15.70:0/1167173561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557916680ca0 0x55791668a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.082+0000 7f57267ce740 1 --2- 172.21.15.70:0/1167173561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5579166802b0 0x55791668a1e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.082+0000 7f57267ce740 1 -- 172.21.15.70:0/1167173561 >> 172.21.15.70:0/1167173561 conn(0x55791667b820 msgr2=0x55791667ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:50.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.082+0000 7f57057fa640 0 client.28300 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:50.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.082+0000 7f57267ce740 1 -- 172.21.15.70:0/1167173561 shutdown_connections 2024-04-01T01:43:50.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.082+0000 7f57267ce740 1 -- 172.21.15.70:0/1167173561 wait complete. 2024-04-01T01:43:50.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.082+0000 7f57267ce740 20 client.28300 trim_cache size 0 max 16384 2024-04-01T01:43:50.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.119+0000 7f57267ce740 1 Processor -- start 2024-04-01T01:43:50.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.120+0000 7f57267ce740 1 -- start start 2024-04-01T01:43:50.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.120+0000 7f57267ce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 0x55791668a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.120+0000 7f57267ce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916680ca0 0x5579166916f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.120+0000 7f57267ce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557916685e10 0x55791669c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.120+0000 7f57267ce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f571c1b0860 con 0x5579166802b0 2024-04-01T01:43:50.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.120+0000 7f57267ce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f571c1b23a0 con 0x557916680ca0 2024-04-01T01:43:50.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.120+0000 7f57267ce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f571c1b0f30 con 0x557916685e10 2024-04-01T01:43:50.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.120+0000 7f57220b6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557916685e10 0x55791669c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.120+0000 7f57220b6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557916685e10 0x55791669c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42350/0 (socket says 172.21.15.70:42350) 2024-04-01T01:43:50.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.120+0000 7f57220b6640 1 -- 172.21.15.70:0/2326363326 learned_addr learned my addr 172.21.15.70:0/2326363326 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:50.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.120+0000 7f57218b5640 1 --2- 172.21.15.70:0/2326363326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 0x55791668a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.120+0000 7f57210b4640 1 --2- 172.21.15.70:0/2326363326 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916680ca0 0x5579166916f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.121+0000 7f57220b6640 1 -- 172.21.15.70:0/2326363326 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916680ca0 msgr2=0x5579166916f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:50.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.121+0000 7f57220b6640 1 --2- 172.21.15.70:0/2326363326 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916680ca0 0x5579166916f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.121+0000 7f57220b6640 1 -- 172.21.15.70:0/2326363326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 msgr2=0x55791668a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:50.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.121+0000 7f57220b6640 1 --2- 172.21.15.70:0/2326363326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 0x55791668a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.121+0000 7f57220b6640 1 -- 172.21.15.70:0/2326363326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579162b90f0 con 0x557916685e10 2024-04-01T01:43:50.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.121+0000 7f57220b6640 1 --2- 172.21.15.70:0/2326363326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557916685e10 0x55791669c990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f57140041f0 tx=0x7f57140046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=473814dec8e53377 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:50.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.121+0000 7f57077fe640 1 -- 172.21.15.70:0/2326363326 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5714009d60 con 0x557916685e10 2024-04-01T01:43:50.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.122+0000 7f57077fe640 1 -- 172.21.15.70:0/2326363326 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f571401b8f0 con 0x557916685e10 2024-04-01T01:43:50.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.122+0000 7f57077fe640 1 -- 172.21.15.70:0/2326363326 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5714011800 con 0x557916685e10 2024-04-01T01:43:50.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.122+0000 7f57267ce740 1 -- 172.21.15.70:0/2326363326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557916685e10 msgr2=0x55791669c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:50.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.122+0000 7f57267ce740 1 --2- 172.21.15.70:0/2326363326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557916685e10 0x55791669c990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f57140041f0 tx=0x7f57140046b0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.123+0000 7f57267ce740 1 -- 172.21.15.70:0/2326363326 shutdown_connections 2024-04-01T01:43:50.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.123+0000 7f57267ce740 1 --2- 172.21.15.70:0/2326363326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557916685e10 0x55791669c990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.123+0000 7f57267ce740 1 --2- 172.21.15.70:0/2326363326 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916680ca0 0x5579166916f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.123+0000 7f57267ce740 1 --2- 172.21.15.70:0/2326363326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 0x55791668a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.123+0000 7f57267ce740 1 -- 172.21.15.70:0/2326363326 >> 172.21.15.70:0/2326363326 conn(0x55791667b820 msgr2=0x557916689df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:50.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.125+0000 7f57267ce740 1 -- 172.21.15.70:0/2326363326 shutdown_connections 2024-04-01T01:43:50.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.125+0000 7f57267ce740 1 -- 172.21.15.70:0/2326363326 wait complete. 2024-04-01T01:43:50.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.125+0000 7f57267ce740 1 -- start start 2024-04-01T01:43:50.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.125+0000 7f57267ce740 1 Processor -- start 2024-04-01T01:43:50.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.126+0000 7f57267ce740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 0x55791668f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.126+0000 7f57267ce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916680ca0 0x55791668f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.126+0000 7f57267ce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557916685e10 0x557916690940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.126+0000 7f57267ce740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f571c1b1cd0 con 0x5579166802b0 2024-04-01T01:43:50.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.126+0000 7f57267ce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f571c1b1600 con 0x557916680ca0 2024-04-01T01:43:50.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.126+0000 7f57267ce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f571c1ae840 con 0x557916685e10 2024-04-01T01:43:50.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.126+0000 7f57267ce740 10 client.0 _ll_register_callbacks cb 0x7f570c0ce3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:50.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.126+0000 7f57210b4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916680ca0 0x55791668f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.126+0000 7f57210b4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916680ca0 0x55791668f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41650/0 (socket says 172.21.15.70:41650) 2024-04-01T01:43:50.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.126+0000 7f57210b4640 1 -- 172.21.15.70:0/4190308177 learned_addr learned my addr 172.21.15.70:0/4190308177 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:50.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.126+0000 7f57218b5640 1 --2- 172.21.15.70:0/4190308177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 0x55791668f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.127+0000 7f57210b4640 1 -- 172.21.15.70:0/4190308177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557916685e10 msgr2=0x557916690940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:50.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.127+0000 7f57210b4640 1 --2- 172.21.15.70:0/4190308177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557916685e10 0x557916690940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.127+0000 7f57210b4640 1 -- 172.21.15.70:0/4190308177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 msgr2=0x55791668f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:50.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.127+0000 7f57210b4640 1 --2- 172.21.15.70:0/4190308177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 0x55791668f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.127+0000 7f57210b4640 1 -- 172.21.15.70:0/4190308177 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557916690f30 con 0x557916680ca0 2024-04-01T01:43:50.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.127+0000 7f57210b4640 1 --2- 172.21.15.70:0/4190308177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916680ca0 0x55791668f980 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f571c0714e0 tx=0x7f5718005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42278b1b5b9e5142 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:50.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.127+0000 7f57057fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:50.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.127+0000 7f57057fa640 1 -- 172.21.15.70:0/4190308177 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5718005020 con 0x557916680ca0 2024-04-01T01:43:50.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.127+0000 7f57267ce740 1 -- 172.21.15.70:0/4190308177 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f571401b8f0 con 0x557916680ca0 2024-04-01T01:43:50.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.128+0000 7f57057fa640 1 -- 172.21.15.70:0/4190308177 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f57180051c0 con 0x557916680ca0 2024-04-01T01:43:50.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.128+0000 7f57057fa640 1 -- 172.21.15.70:0/4190308177 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5718037050 con 0x557916680ca0 2024-04-01T01:43:50.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.128+0000 7f57267ce740 10 client.28505 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:50.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.128+0000 7f57267ce740 1 -- 172.21.15.70:0/4190308177 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55791671e6e0 con 0x557916680ca0 2024-04-01T01:43:50.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.128+0000 7f57057fa640 1 -- 172.21.15.70:0/4190308177 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f571401b8f0 con 0x557916680ca0 2024-04-01T01:43:50.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.129+0000 7f57057fa640 1 -- 172.21.15.70:0/4190308177 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f571802a650 con 0x557916680ca0 2024-04-01T01:43:50.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.129+0000 7f57267ce740 10 client.28505 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:50.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.129+0000 7f57267ce740 10 client.28505 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:50.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.129+0000 7f57267ce740 20 client.28505 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:50.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.129+0000 7f57267ce740 1 --2- 172.21.15.70:0/4190308177 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55791671f0e0 0x557916721500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.129+0000 7f57267ce740 4 client.28505 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:43:50.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.129+0000 7f57267ce740 1 -- 172.21.15.70:0/4190308177 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5579165eca10 con 0x55791671f0e0 2024-04-01T01:43:50.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.129+0000 7f57218b5640 1 --2- 172.21.15.70:0/4190308177 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55791671f0e0 0x557916721500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.130+0000 7f57057fa640 10 client.28505 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:50.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.130+0000 7f57218b5640 1 --2- 172.21.15.70:0/4190308177 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55791671f0e0 0x557916721500 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f87100d4a41b0d8c server_cookie=3ef7d93b87cf433c in_seq=0 out_seq=0 2024-04-01T01:43:50.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.130+0000 7f57057fa640 1 -- 172.21.15.70:0/4190308177 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5579165eca10 con 0x55791671f0e0 2024-04-01T01:43:50.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.130+0000 7f57057fa640 10 client.28505 handle_command_reply: tid=0 2024-04-01T01:43:50.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.130+0000 7f57267ce740 1 client.28505 shutdown 2024-04-01T01:43:50.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.130+0000 7f57267ce740 1 -- 172.21.15.70:0/4190308177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916680ca0 msgr2=0x55791668f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:50.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.130+0000 7f57267ce740 1 --2- 172.21.15.70:0/4190308177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916680ca0 0x55791668f980 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f571c0714e0 tx=0x7f5718005f90 comp rx=0 tx=0).stop 2024-04-01T01:43:50.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.131+0000 7f57267ce740 1 -- 172.21.15.70:0/4190308177 shutdown_connections 2024-04-01T01:43:50.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.131+0000 7f57267ce740 1 --2- 172.21.15.70:0/4190308177 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55791671f0e0 0x557916721500 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.131+0000 7f57267ce740 1 --2- 172.21.15.70:0/4190308177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557916685e10 0x557916690940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.131+0000 7f57267ce740 1 --2- 172.21.15.70:0/4190308177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557916680ca0 0x55791668f980 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.131+0000 7f57267ce740 1 --2- 172.21.15.70:0/4190308177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5579166802b0 0x55791668f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.131+0000 7f57267ce740 1 -- 172.21.15.70:0/4190308177 >> 172.21.15.70:0/4190308177 conn(0x55791667b820 msgr2=0x55791667a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:50.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.131+0000 7f57057fa640 0 client.28505 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:50.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.131+0000 7f57267ce740 1 -- 172.21.15.70:0/4190308177 shutdown_connections 2024-04-01T01:43:50.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.131+0000 7f57267ce740 1 -- 172.21.15.70:0/4190308177 wait complete. 2024-04-01T01:43:50.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.131+0000 7f57267ce740 20 client.28505 trim_cache size 0 max 16384 2024-04-01T01:43:50.133 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:43:50.133 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:43:50.133 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:43:50.133 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:43:50.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.132+0000 7f5723b19640 1 -- 172.21.15.70:0/4226590877 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57080a2620 msgr2=0x7f57080a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:50.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.132+0000 7f5723b19640 1 --2- 172.21.15.70:0/4226590877 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57080a2620 0x7f57080a4ae0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f57180058c0 tx=0x7f5718005850 comp rx=0 tx=0).stop 2024-04-01T01:43:50.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.132+0000 7f5723b19640 1 -- 172.21.15.70:0/4226590877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571c107d70 msgr2=0x7f571c06ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:50.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.132+0000 7f5723b19640 1 --2- 172.21.15.70:0/4226590877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571c107d70 0x7f571c06ffd0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f570c009850 tx=0x7f570c004040 comp rx=0 tx=0).stop 2024-04-01T01:43:50.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.132+0000 7f5723b19640 1 -- 172.21.15.70:0/4226590877 shutdown_connections 2024-04-01T01:43:50.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.132+0000 7f5723b19640 1 --2- 172.21.15.70:0/4226590877 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57080a2620 0x7f57080a4ae0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.132+0000 7f5723b19640 1 --2- 172.21.15.70:0/4226590877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f571c109920 0x7f571c1adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.132+0000 7f5723b19640 1 --2- 172.21.15.70:0/4226590877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f571c108f60 0x7f571c070510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.132+0000 7f5723b19640 1 --2- 172.21.15.70:0/4226590877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571c107d70 0x7f571c06ffd0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.132+0000 7f5723b19640 1 -- 172.21.15.70:0/4226590877 >> 172.21.15.70:0/4226590877 conn(0x7f571c076d00 msgr2=0x7f571c10c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:50.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.132+0000 7f5723b19640 1 -- 172.21.15.70:0/4226590877 shutdown_connections 2024-04-01T01:43:50.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.132+0000 7f5723b19640 1 -- 172.21.15.70:0/4226590877 wait complete. 2024-04-01T01:43:50.149 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:43:50.149 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:43:50.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.345+0000 7f39d3c47640 1 -- 172.21.15.70:0/93626084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39cc109940 msgr2=0x7f39cc1101d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:50.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.345+0000 7f39d3c47640 1 --2- 172.21.15.70:0/93626084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39cc109940 0x7f39cc1101d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f39c4009be0 tx=0x7f39c402bef0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.345+0000 7f39d3c47640 1 -- 172.21.15.70:0/93626084 shutdown_connections 2024-04-01T01:43:50.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.345+0000 7f39d3c47640 1 --2- 172.21.15.70:0/93626084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39cc109940 0x7f39cc1101d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.345+0000 7f39d3c47640 1 --2- 172.21.15.70:0/93626084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc108f80 0x7f39cc109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.345+0000 7f39d3c47640 1 --2- 172.21.15.70:0/93626084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39cc107d90 0x7f39cc1081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.345+0000 7f39d3c47640 1 -- 172.21.15.70:0/93626084 >> 172.21.15.70:0/93626084 conn(0x7f39cc076d30 msgr2=0x7f39cc079170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:50.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.345+0000 7f39d3c47640 1 -- 172.21.15.70:0/93626084 shutdown_connections 2024-04-01T01:43:50.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.346+0000 7f39d3c47640 1 -- 172.21.15.70:0/93626084 wait complete. 2024-04-01T01:43:50.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.347+0000 7f39d3c47640 1 Processor -- start 2024-04-01T01:43:50.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.347+0000 7f39d3c47640 1 -- start start 2024-04-01T01:43:50.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.347+0000 7f39d3c47640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39cc107d90 0x7f39cc19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.347+0000 7f39d3c47640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39cc108f80 0x7f39cc19e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.347+0000 7f39d3c47640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc109940 0x7f39cc1a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.347+0000 7f39d3c47640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc1138a0 con 0x7f39cc107d90 2024-04-01T01:43:50.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.347+0000 7f39d3c47640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc113720 con 0x7f39cc108f80 2024-04-01T01:43:50.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.347+0000 7f39d3c47640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc113a20 con 0x7f39cc109940 2024-04-01T01:43:50.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.347+0000 7f39d21e4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc109940 0x7f39cc1a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.347+0000 7f39d21e4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc109940 0x7f39cc1a5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42374/0 (socket says 172.21.15.70:42374) 2024-04-01T01:43:50.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.347+0000 7f39d21e4640 1 -- 172.21.15.70:0/3363456214 learned_addr learned my addr 172.21.15.70:0/3363456214 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:50.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.347+0000 7f39d11e2640 1 --2- 172.21.15.70:0/3363456214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39cc108f80 0x7f39cc19e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.348+0000 7f39d11e2640 1 -- 172.21.15.70:0/3363456214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc109940 msgr2=0x7f39cc1a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:50.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.348+0000 7f39d11e2640 1 --2- 172.21.15.70:0/3363456214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc109940 0x7f39cc1a5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.348+0000 7f39d11e2640 1 -- 172.21.15.70:0/3363456214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39cc107d90 msgr2=0x7f39cc19dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:50.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.348+0000 7f39d11e2640 1 --2- 172.21.15.70:0/3363456214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39cc107d90 0x7f39cc19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.348+0000 7f39d11e2640 1 -- 172.21.15.70:0/3363456214 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39cc1a5b70 con 0x7f39cc108f80 2024-04-01T01:43:50.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.348+0000 7f39d11e2640 1 --2- 172.21.15.70:0/3363456214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39cc108f80 0x7f39cc19e4e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f39bc005fd0 tx=0x7f39bc00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:50.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.349+0000 7f39c2ffd640 1 -- 172.21.15.70:0/3363456214 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc00cd00 con 0x7f39cc108f80 2024-04-01T01:43:50.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.349+0000 7f39c2ffd640 1 -- 172.21.15.70:0/3363456214 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39bc0035f0 con 0x7f39cc108f80 2024-04-01T01:43:50.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.349+0000 7f39c2ffd640 1 -- 172.21.15.70:0/3363456214 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39bc010690 con 0x7f39cc108f80 2024-04-01T01:43:50.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.349+0000 7f39d3c47640 1 -- 172.21.15.70:0/3363456214 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39cc1a5e60 con 0x7f39cc108f80 2024-04-01T01:43:50.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.349+0000 7f39d3c47640 1 -- 172.21.15.70:0/3363456214 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39cc1a6270 con 0x7f39cc108f80 2024-04-01T01:43:50.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.350+0000 7f39c0ff9640 1 -- 172.21.15.70:0/3363456214 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39b4005290 con 0x7f39cc108f80 2024-04-01T01:43:50.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.352+0000 7f39c2ffd640 1 -- 172.21.15.70:0/3363456214 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f39bc0037a0 con 0x7f39cc108f80 2024-04-01T01:43:50.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.353+0000 7f39c2ffd640 1 --2- 172.21.15.70:0/3363456214 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39b80a26a0 0x7f39b80a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:50.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.353+0000 7f39c2ffd640 1 -- 172.21.15.70:0/3363456214 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f39bc0cec10 con 0x7f39cc108f80 2024-04-01T01:43:50.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.354+0000 7f39d19e3640 1 --2- 172.21.15.70:0/3363456214 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39b80a26a0 0x7f39b80a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:50.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.355+0000 7f39d19e3640 1 --2- 172.21.15.70:0/3363456214 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39b80a26a0 0x7f39b80a4b60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f39cc108de0 tx=0x7f39c8009630 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:50.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.361+0000 7f39c2ffd640 1 -- 172.21.15.70:0/3363456214 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f39bc009510 con 0x7f39cc108f80 2024-04-01T01:43:50.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.789+0000 7f39c0ff9640 1 -- 172.21.15.70:0/3363456214 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39b4005580 con 0x7f39cc108f80 2024-04-01T01:43:50.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.790+0000 7f39c2ffd640 1 -- 172.21.15.70:0/3363456214 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7f39bc09b9a0 con 0x7f39cc108f80 2024-04-01T01:43:50.790 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:43:50.790 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:43:50.790 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:43:50.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.794+0000 7f39c0ff9640 1 -- 172.21.15.70:0/3363456214 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39b80a26a0 msgr2=0x7f39b80a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:50.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.794+0000 7f39c0ff9640 1 --2- 172.21.15.70:0/3363456214 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39b80a26a0 0x7f39b80a4b60 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f39cc108de0 tx=0x7f39c8009630 comp rx=0 tx=0).stop 2024-04-01T01:43:50.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.794+0000 7f39c0ff9640 1 -- 172.21.15.70:0/3363456214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39cc108f80 msgr2=0x7f39cc19e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:50.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.794+0000 7f39c0ff9640 1 --2- 172.21.15.70:0/3363456214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39cc108f80 0x7f39cc19e4e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f39bc005fd0 tx=0x7f39bc00ee50 comp rx=0 tx=0).stop 2024-04-01T01:43:50.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.794+0000 7f39c0ff9640 1 -- 172.21.15.70:0/3363456214 shutdown_connections 2024-04-01T01:43:50.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.794+0000 7f39c0ff9640 1 --2- 172.21.15.70:0/3363456214 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39b80a26a0 0x7f39b80a4b60 secure :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f39cc108de0 tx=0x7f39c8009630 comp rx=0 tx=0).stop 2024-04-01T01:43:50.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.794+0000 7f39c0ff9640 1 --2- 172.21.15.70:0/3363456214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39cc109940 0x7f39cc1a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.794+0000 7f39c0ff9640 1 --2- 172.21.15.70:0/3363456214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39cc108f80 0x7f39cc19e4e0 secure :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f39bc005fd0 tx=0x7f39bc00ee50 comp rx=0 tx=0).stop 2024-04-01T01:43:50.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.794+0000 7f39c0ff9640 1 --2- 172.21.15.70:0/3363456214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39cc107d90 0x7f39cc19dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:50.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.794+0000 7f39c0ff9640 1 -- 172.21.15.70:0/3363456214 >> 172.21.15.70:0/3363456214 conn(0x7f39cc076d30 msgr2=0x7f39cc10c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:50.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.794+0000 7f39c0ff9640 1 -- 172.21.15.70:0/3363456214 shutdown_connections 2024-04-01T01:43:50.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:50.795+0000 7f39c0ff9640 1 -- 172.21.15.70:0/3363456214 wait complete. 2024-04-01T01:43:50.815 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:43:51.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.003+0000 7fc7d4f6e640 1 -- 172.21.15.70:0/281107423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d006e6d0 msgr2=0x7fc7d006eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.003+0000 7fc7d4f6e640 1 --2- 172.21.15.70:0/281107423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d006e6d0 0x7fc7d006eb30 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009900 tx=0x7fc7c402bc80 comp rx=0 tx=0).stop 2024-04-01T01:43:51.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.006+0000 7fc7d4f6e640 1 -- 172.21.15.70:0/281107423 shutdown_connections 2024-04-01T01:43:51.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.006+0000 7fc7d4f6e640 1 --2- 172.21.15.70:0/281107423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7d006f070 0x7fc7d006f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.006+0000 7fc7d4f6e640 1 --2- 172.21.15.70:0/281107423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d006e6d0 0x7fc7d006eb30 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.006+0000 7fc7d4f6e640 1 --2- 172.21.15.70:0/281107423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d006d490 0x7fc7d006d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.006+0000 7fc7d4f6e640 1 -- 172.21.15.70:0/281107423 >> 172.21.15.70:0/281107423 conn(0x7fc7d006c110 msgr2=0x7fc7d006c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.009+0000 7fc7d4f6e640 1 -- 172.21.15.70:0/281107423 shutdown_connections 2024-04-01T01:43:51.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.010+0000 7fc7d4f6e640 1 -- 172.21.15.70:0/281107423 wait complete. 2024-04-01T01:43:51.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.011+0000 7fc7d4f6e640 1 Processor -- start 2024-04-01T01:43:51.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.012+0000 7fc7d4f6e640 1 -- start start 2024-04-01T01:43:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.012+0000 7fc7d4f6e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d006d490 0x7fc7d01a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.012+0000 7fc7d4f6e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7d006e6d0 0x7fc7d01a6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.012+0000 7fc7d4f6e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d006f070 0x7fc7d01adca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.013+0000 7fc7d4f6e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d011c020 con 0x7fc7d006e6d0 2024-04-01T01:43:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.013+0000 7fc7d4f6e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d011bea0 con 0x7fc7d006d490 2024-04-01T01:43:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.013+0000 7fc7d4f6e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d011c1a0 con 0x7fc7d006f070 2024-04-01T01:43:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.013+0000 7fc7cf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d006d490 0x7fc7d01a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.013+0000 7fc7cf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d006d490 0x7fc7d01a66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41692/0 (socket says 172.21.15.70:41692) 2024-04-01T01:43:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.013+0000 7fc7cf7fe640 1 -- 172.21.15.70:0/2992944875 learned_addr learned my addr 172.21.15.70:0/2992944875 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.013+0000 7fc7ceffd640 1 --2- 172.21.15.70:0/2992944875 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7d006e6d0 0x7fc7d01a6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.013+0000 7fc7cf7fe640 1 -- 172.21.15.70:0/2992944875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d006f070 msgr2=0x7fc7d01adca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.013+0000 7fc7cf7fe640 1 --2- 172.21.15.70:0/2992944875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d006f070 0x7fc7d01adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.013+0000 7fc7cf7fe640 1 -- 172.21.15.70:0/2992944875 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7d006e6d0 msgr2=0x7fc7d01a6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.013+0000 7fc7cf7fe640 1 --2- 172.21.15.70:0/2992944875 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7d006e6d0 0x7fc7d01a6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.013+0000 7fc7cf7fe640 1 -- 172.21.15.70:0/2992944875 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d01ae310 con 0x7fc7d006d490 2024-04-01T01:43:51.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.014+0000 7fc7cf7fe640 1 --2- 172.21.15.70:0/2992944875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d006d490 0x7fc7d01a66f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000b450 tx=0x7fc7c000b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.014+0000 7fc7ccff9640 1 -- 172.21.15.70:0/2992944875 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0015020 con 0x7fc7d006d490 2024-04-01T01:43:51.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.014+0000 7fc7ccff9640 1 -- 172.21.15.70:0/2992944875 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7c0011d40 con 0x7fc7d006d490 2024-04-01T01:43:51.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.014+0000 7fc7ccff9640 1 -- 172.21.15.70:0/2992944875 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c000f340 con 0x7fc7d006d490 2024-04-01T01:43:51.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.014+0000 7fc7d4f6e640 1 -- 172.21.15.70:0/2992944875 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d01ae600 con 0x7fc7d006d490 2024-04-01T01:43:51.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.014+0000 7fc7d4f6e640 1 -- 172.21.15.70:0/2992944875 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d01aeb50 con 0x7fc7d006d490 2024-04-01T01:43:51.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.018+0000 7fc7d8db8740 1 Processor -- start 2024-04-01T01:43:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.021+0000 7fc7ccff9640 1 -- 172.21.15.70:0/2992944875 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fc7c0018020 con 0x7fc7d006d490 2024-04-01T01:43:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.021+0000 7fc7d8db8740 1 -- start start 2024-04-01T01:43:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.021+0000 7fc7d8db8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd5476a0 0x5603bd547ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.021+0000 7fc7d8db8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 0x5603bd54cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.021+0000 7fc7d8db8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd54d200 0x5603bd54f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.021+0000 7fc7d8db8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5603bd4ac1b0 con 0x5603bd548090 2024-04-01T01:43:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.021+0000 7fc7d8db8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5603bd454c00 con 0x5603bd5476a0 2024-04-01T01:43:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.021+0000 7fc7d8db8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5603bd4a8250 con 0x5603bd54d200 2024-04-01T01:43:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.021+0000 7fc7ceffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd54d200 0x5603bd54f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.021+0000 7fc7ceffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd54d200 0x5603bd54f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42392/0 (socket says 172.21.15.70:42392) 2024-04-01T01:43:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.021+0000 7fc7ceffd640 1 -- 172.21.15.70:0/1459065644 learned_addr learned my addr 172.21.15.70:0/1459065644 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.021+0000 7fc7cffff640 1 --2- 172.21.15.70:0/1459065644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd5476a0 0x5603bd547ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.022+0000 7fc7cffff640 1 -- 172.21.15.70:0/1459065644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd54d200 msgr2=0x5603bd54f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.022+0000 7fc7cffff640 1 --2- 172.21.15.70:0/1459065644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd54d200 0x5603bd54f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.022+0000 7fc7cffff640 1 -- 172.21.15.70:0/1459065644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 msgr2=0x5603bd54cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.022+0000 7fc7cffff640 1 --2- 172.21.15.70:0/1459065644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 0x5603bd54cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.022+0000 7fc7cffff640 1 -- 172.21.15.70:0/1459065644 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603bd4743d0 con 0x5603bd5476a0 2024-04-01T01:43:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.022+0000 7fc7cffff640 1 --2- 172.21.15.70:0/1459065644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd5476a0 0x5603bd547ac0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009cc0 tx=0x7fc7c800e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68d80b525c3fef7d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.022+0000 7fc7b5ffb640 1 -- 172.21.15.70:0/1459065644 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8016070 con 0x5603bd5476a0 2024-04-01T01:43:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.022+0000 7fc7b5ffb640 1 -- 172.21.15.70:0/1459065644 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7c8005020 con 0x5603bd5476a0 2024-04-01T01:43:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.022+0000 7fc7b5ffb640 1 -- 172.21.15.70:0/1459065644 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c80116d0 con 0x5603bd5476a0 2024-04-01T01:43:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.023+0000 7fc7ccff9640 1 --2- 172.21.15.70:0/2992944875 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc7b80a26a0 0x7fc7b80a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.023+0000 7fc7cf7fe640 1 --2- 172.21.15.70:0/2992944875 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc7b80a26a0 0x7fc7b80a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.023+0000 7fc7ccff9640 1 -- 172.21.15.70:0/2992944875 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc7c00ce4f0 con 0x7fc7d006d490 2024-04-01T01:43:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.023+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1459065644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd5476a0 msgr2=0x5603bd547ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.023+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/1459065644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd5476a0 0x5603bd547ac0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009cc0 tx=0x7fc7c800e6c0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.024+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1459065644 shutdown_connections 2024-04-01T01:43:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.024+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/1459065644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd54d200 0x5603bd54f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.024+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/1459065644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 0x5603bd54cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.024+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/1459065644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd5476a0 0x5603bd547ac0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.024+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1459065644 >> 172.21.15.70:0/1459065644 conn(0x5603bd542c10 msgr2=0x5603bd543010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.024+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1459065644 shutdown_connections 2024-04-01T01:43:51.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.024+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1459065644 wait complete. 2024-04-01T01:43:51.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.024+0000 7fc7cf7fe640 1 --2- 172.21.15.70:0/2992944875 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc7b80a26a0 0x7fc7b80a4b60 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc7c009a880 tx=0x7fc7c00c54b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.024+0000 7fc7d8db8740 1 -- start start 2024-04-01T01:43:51.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.025+0000 7fc7d8db8740 1 Processor -- start 2024-04-01T01:43:51.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.025+0000 7fc7d8db8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd5476a0 0x5603bd551c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.025+0000 7fc7d8db8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 0x5603bd552160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.025+0000 7fc7d8db8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd54d200 0x5603bd5591a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.025+0000 7fc7d8db8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5603bd5501b0 con 0x5603bd548090 2024-04-01T01:43:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.025+0000 7fc7d8db8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5603bd44a470 con 0x5603bd5476a0 2024-04-01T01:43:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.025+0000 7fc7d8db8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5603bd550330 con 0x5603bd54d200 2024-04-01T01:43:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.025+0000 7fc7d8db8740 10 client.0 _ll_register_callbacks cb 0x5603bd467cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.026+0000 7fc7ceffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 0x5603bd552160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.026+0000 7fc7ceffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 0x5603bd552160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47490/0 (socket says 172.21.15.70:47490) 2024-04-01T01:43:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.026+0000 7fc7ceffd640 1 -- 172.21.15.70:0/1928754398 learned_addr learned my addr 172.21.15.70:0/1928754398 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.026+0000 7fc7cffff640 1 --2- 172.21.15.70:0/1928754398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd5476a0 0x5603bd551c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.026+0000 7fc7cffff640 1 --2- 172.21.15.70:0/1928754398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd54d200 0x5603bd5591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.027+0000 7fc7cffff640 1 -- 172.21.15.70:0/1928754398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd5476a0 msgr2=0x5603bd551c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.027+0000 7fc7cffff640 1 --2- 172.21.15.70:0/1928754398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd5476a0 0x5603bd551c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.027+0000 7fc7cffff640 1 -- 172.21.15.70:0/1928754398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 msgr2=0x5603bd552160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.027+0000 7fc7cffff640 1 --2- 172.21.15.70:0/1928754398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 0x5603bd552160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.027+0000 7fc7cffff640 1 -- 172.21.15.70:0/1928754398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603bd1c03f0 con 0x5603bd54d200 2024-04-01T01:43:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.027+0000 7fc7cffff640 1 --2- 172.21.15.70:0/1928754398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd54d200 0x5603bd5591a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009240 tx=0x7fc7c80095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54f91c9f09a900b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.027+0000 7fc7aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:43:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.027+0000 7fc7aeffd640 1 -- 172.21.15.70:0/1928754398 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8023040 con 0x5603bd54d200 2024-04-01T01:43:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.027+0000 7fc7aeffd640 1 -- 172.21.15.70:0/1928754398 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc7c8005600 con 0x5603bd54d200 2024-04-01T01:43:51.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.028+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1928754398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7c8005020 con 0x5603bd54d200 2024-04-01T01:43:51.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.028+0000 7fc7aeffd640 1 -- 172.21.15.70:0/1928754398 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc7c8016040 con 0x5603bd54d200 2024-04-01T01:43:51.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.029+0000 7fc7d8db8740 10 client.28315 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:51.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.029+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1928754398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5603bd28a7d0 con 0x5603bd54d200 2024-04-01T01:43:51.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.029+0000 7fc7aeffd640 1 -- 172.21.15.70:0/1928754398 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7c8005020 con 0x5603bd54d200 2024-04-01T01:43:51.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.029+0000 7fc7aeffd640 1 -- 172.21.15.70:0/1928754398 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc7c800f040 con 0x5603bd54d200 2024-04-01T01:43:51.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.029+0000 7fc7d8db8740 10 client.28315 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:51.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.029+0000 7fc7d8db8740 10 client.28315 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:43:51.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.029+0000 7fc7d8db8740 20 client.28315 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:51.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.030+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/1928754398 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5603bd56eb30 0x5603bd570f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.030+0000 7fc7d8db8740 4 client.28315 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:51.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.030+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1928754398 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603bd4b96f0 con 0x5603bd56eb30 2024-04-01T01:43:51.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.033+0000 7fc7ceffd640 1 --2- 172.21.15.70:0/1928754398 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5603bd56eb30 0x5603bd570f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.034+0000 7fc7ceffd640 1 --2- 172.21.15.70:0/1928754398 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5603bd56eb30 0x5603bd570f70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1eaad2cec7413df2 server_cookie=3ffb56dc1c1ab62b in_seq=0 out_seq=0 2024-04-01T01:43:51.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.034+0000 7fc7aeffd640 10 client.28315 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:43:51.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.036+0000 7fc7aeffd640 1 -- 172.21.15.70:0/1928754398 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5603bd4b96f0 con 0x5603bd56eb30 2024-04-01T01:43:51.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.036+0000 7fc7aeffd640 10 client.28315 handle_command_reply: tid=0 2024-04-01T01:43:51.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.036+0000 7fc7d8db8740 1 client.28315 shutdown 2024-04-01T01:43:51.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.036+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1928754398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd54d200 msgr2=0x5603bd5591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.037+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/1928754398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd54d200 0x5603bd5591a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009240 tx=0x7fc7c80095e0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.037+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1928754398 shutdown_connections 2024-04-01T01:43:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.037+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/1928754398 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5603bd56eb30 0x5603bd570f70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.037+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/1928754398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd54d200 0x5603bd5591a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.037+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/1928754398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 0x5603bd552160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.037+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/1928754398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd5476a0 0x5603bd551c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.037+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1928754398 >> 172.21.15.70:0/1928754398 conn(0x5603bd542c10 msgr2=0x5603bd541f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.037+0000 7fc7aeffd640 0 client.28315 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:43:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.037+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1928754398 shutdown_connections 2024-04-01T01:43:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.037+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1928754398 wait complete. 2024-04-01T01:43:51.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.037+0000 7fc7d8db8740 20 client.28315 trim_cache size 0 max 16384 2024-04-01T01:43:51.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.061+0000 7fc7d8db8740 1 Processor -- start 2024-04-01T01:43:51.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.063+0000 7fc7d8db8740 1 -- start start 2024-04-01T01:43:51.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.063+0000 7fc7d8db8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd5476a0 0x5603bd549590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.063+0000 7fc7d8db8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 0x5603bd549ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.063+0000 7fc7d8db8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd54d200 0x5603bd559020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.063+0000 7fc7d8db8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc00fd40 con 0x5603bd548090 2024-04-01T01:43:51.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.063+0000 7fc7d8db8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc011880 con 0x5603bd54d200 2024-04-01T01:43:51.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.063+0000 7fc7d8db8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc010410 con 0x5603bd5476a0 2024-04-01T01:43:51.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.064+0000 7fc7ceffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 0x5603bd549ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.064+0000 7fc7ceffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 0x5603bd549ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47506/0 (socket says 172.21.15.70:47506) 2024-04-01T01:43:51.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.064+0000 7fc7ceffd640 1 -- 172.21.15.70:0/1722808929 learned_addr learned my addr 172.21.15.70:0/1722808929 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.064+0000 7fc7cffff640 1 --2- 172.21.15.70:0/1722808929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd5476a0 0x5603bd549590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.064+0000 7fc7cffff640 1 --2- 172.21.15.70:0/1722808929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd54d200 0x5603bd559020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.065+0000 7fc7ceffd640 1 -- 172.21.15.70:0/1722808929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd5476a0 msgr2=0x5603bd549590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.065+0000 7fc7ceffd640 1 --2- 172.21.15.70:0/1722808929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd5476a0 0x5603bd549590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.065+0000 7fc7ceffd640 1 -- 172.21.15.70:0/1722808929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd54d200 msgr2=0x5603bd559020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.065+0000 7fc7ceffd640 1 --2- 172.21.15.70:0/1722808929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd54d200 0x5603bd559020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.065+0000 7fc7ceffd640 1 -- 172.21.15.70:0/1722808929 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603bd533e60 con 0x5603bd548090 2024-04-01T01:43:51.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.065+0000 7fc7cffff640 1 --2- 172.21.15.70:0/1722808929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd5476a0 0x5603bd549590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:43:51.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.066+0000 7fc7ceffd640 1 --2- 172.21.15.70:0/1722808929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 0x5603bd549ad0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4036040 tx=0x7fc7c4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c559b01df133012 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.067+0000 7fc7b57fa640 1 -- 172.21.15.70:0/1722808929 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4050070 con 0x5603bd548090 2024-04-01T01:43:51.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.067+0000 7fc7b57fa640 1 -- 172.21.15.70:0/1722808929 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7c4004110 con 0x5603bd548090 2024-04-01T01:43:51.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.067+0000 7fc7b57fa640 1 -- 172.21.15.70:0/1722808929 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4005040 con 0x5603bd548090 2024-04-01T01:43:51.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.068+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1722808929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 msgr2=0x5603bd549ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.068+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/1722808929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 0x5603bd549ad0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4036040 tx=0x7fc7c4009260 comp rx=0 tx=0).stop 2024-04-01T01:43:51.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.068+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1722808929 shutdown_connections 2024-04-01T01:43:51.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.068+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/1722808929 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd54d200 0x5603bd559020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.068+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/1722808929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd548090 0x5603bd549ad0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.068+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/1722808929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd5476a0 0x5603bd549590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.068+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1722808929 >> 172.21.15.70:0/1722808929 conn(0x5603bd542c10 msgr2=0x5603bd542ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.069+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1722808929 shutdown_connections 2024-04-01T01:43:51.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.069+0000 7fc7d8db8740 1 -- 172.21.15.70:0/1722808929 wait complete. 2024-04-01T01:43:51.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.069+0000 7fc7d8db8740 1 -- start start 2024-04-01T01:43:51.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.070+0000 7fc7d8db8740 1 Processor -- start 2024-04-01T01:43:51.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.070+0000 7fc7d8db8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd5476a0 0x5603bd556640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.070+0000 7fc7d8db8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd548090 0x5603bd556b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.070+0000 7fc7d8db8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd54d200 0x5603bd557b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.070+0000 7fc7d8db8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc0111b0 con 0x5603bd54d200 2024-04-01T01:43:51.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.070+0000 7fc7d8db8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc010ae0 con 0x5603bd548090 2024-04-01T01:43:51.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.070+0000 7fc7d8db8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc00dd20 con 0x5603bd5476a0 2024-04-01T01:43:51.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.070+0000 7fc7d8db8740 10 client.0 _ll_register_callbacks cb 0x7fc7c8011670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:51.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.070+0000 7fc7cffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd5476a0 0x5603bd556640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.070+0000 7fc7cffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd5476a0 0x5603bd556640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42412/0 (socket says 172.21.15.70:42412) 2024-04-01T01:43:51.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.070+0000 7fc7cffff640 1 -- 172.21.15.70:0/2789689957 learned_addr learned my addr 172.21.15.70:0/2789689957 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.071+0000 7fc7ceffd640 1 --2- 172.21.15.70:0/2789689957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd548090 0x5603bd556b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.071+0000 7fc7cffff640 1 -- 172.21.15.70:0/2789689957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd548090 msgr2=0x5603bd556b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.071+0000 7fc7cffff640 1 --2- 172.21.15.70:0/2789689957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd548090 0x5603bd556b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.071+0000 7fc7cffff640 1 -- 172.21.15.70:0/2789689957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd54d200 msgr2=0x5603bd557b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.071+0000 7fc7cffff640 1 --2- 172.21.15.70:0/2789689957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd54d200 0x5603bd557b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.071+0000 7fc7cffff640 1 -- 172.21.15.70:0/2789689957 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603bd558180 con 0x5603bd5476a0 2024-04-01T01:43:51.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.072+0000 7fc7cffff640 1 --2- 172.21.15.70:0/2789689957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd5476a0 0x5603bd556640 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8002230 tx=0x7fc7c8005c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19e354a6d008497b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.073+0000 7fc7aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:43:51.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.073+0000 7fc7aeffd640 1 -- 172.21.15.70:0/2789689957 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8004fd0 con 0x5603bd5476a0 2024-04-01T01:43:51.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.074+0000 7fc7aeffd640 1 -- 172.21.15.70:0/2789689957 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc7c8004050 con 0x5603bd5476a0 2024-04-01T01:43:51.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.074+0000 7fc7d8db8740 1 -- 172.21.15.70:0/2789689957 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7c4004110 con 0x5603bd5476a0 2024-04-01T01:43:51.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.074+0000 7fc7aeffd640 1 -- 172.21.15.70:0/2789689957 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc7c8044050 con 0x5603bd5476a0 2024-04-01T01:43:51.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.074+0000 7fc7d8db8740 10 client.28321 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:51.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.074+0000 7fc7d8db8740 1 -- 172.21.15.70:0/2789689957 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5603bd559480 con 0x5603bd5476a0 2024-04-01T01:43:51.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.074+0000 7fc7aeffd640 1 -- 172.21.15.70:0/2789689957 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7c4004110 con 0x5603bd5476a0 2024-04-01T01:43:51.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.075+0000 7fc7aeffd640 1 -- 172.21.15.70:0/2789689957 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc7c803f030 con 0x5603bd5476a0 2024-04-01T01:43:51.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.075+0000 7fc7d8db8740 10 client.28321 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:51.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.075+0000 7fc7d8db8740 10 client.28321 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:43:51.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.075+0000 7fc7d8db8740 20 client.28321 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:51.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.075+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/2789689957 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5603bd5d5520 0x5603bd5d7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.075+0000 7fc7d8db8740 4 client.28321 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:43:51.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.075+0000 7fc7d8db8740 1 -- 172.21.15.70:0/2789689957 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5603bd4b3a40 con 0x5603bd5d5520 2024-04-01T01:43:51.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.075+0000 7fc7ceffd640 1 --2- 172.21.15.70:0/2789689957 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5603bd5d5520 0x5603bd5d7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.075+0000 7fc7aeffd640 10 client.28321 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:43:51.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.075+0000 7fc7ceffd640 1 --2- 172.21.15.70:0/2789689957 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5603bd5d5520 0x5603bd5d7940 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a3bf091096dec56c server_cookie=f318449d3023f7df in_seq=0 out_seq=0 2024-04-01T01:43:51.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.076+0000 7fc7aeffd640 1 -- 172.21.15.70:0/2789689957 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5603bd4b3a40 con 0x5603bd5d5520 2024-04-01T01:43:51.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.076+0000 7fc7aeffd640 10 client.28321 handle_command_reply: tid=0 2024-04-01T01:43:51.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.076+0000 7fc7d8db8740 1 client.28321 shutdown 2024-04-01T01:43:51.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.076+0000 7fc7d8db8740 1 -- 172.21.15.70:0/2789689957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd5476a0 msgr2=0x5603bd556640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.076+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/2789689957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd5476a0 0x5603bd556640 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8002230 tx=0x7fc7c8005c30 comp rx=0 tx=0).stop 2024-04-01T01:43:51.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.076+0000 7fc7d8db8740 1 -- 172.21.15.70:0/2789689957 shutdown_connections 2024-04-01T01:43:51.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.076+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/2789689957 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5603bd5d5520 0x5603bd5d7940 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.076+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/2789689957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603bd54d200 0x5603bd557b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.076+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/2789689957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603bd548090 0x5603bd556b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.076+0000 7fc7d8db8740 1 --2- 172.21.15.70:0/2789689957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603bd5476a0 0x5603bd556640 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.076+0000 7fc7d8db8740 1 -- 172.21.15.70:0/2789689957 >> 172.21.15.70:0/2789689957 conn(0x5603bd542c10 msgr2=0x5603bd551c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.076+0000 7fc7aeffd640 0 client.28321 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:43:51.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.076+0000 7fc7d8db8740 1 -- 172.21.15.70:0/2789689957 shutdown_connections 2024-04-01T01:43:51.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.077+0000 7fc7d8db8740 1 -- 172.21.15.70:0/2789689957 wait complete. 2024-04-01T01:43:51.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.077+0000 7fc7d8db8740 20 client.28321 trim_cache size 0 max 16384 2024-04-01T01:43:51.078 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:43:51.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.079+0000 7fc7d4f6e640 1 -- 172.21.15.70:0/2992944875 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc7b80a26a0 msgr2=0x7fc7b80a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.079+0000 7fc7d4f6e640 1 --2- 172.21.15.70:0/2992944875 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc7b80a26a0 0x7fc7b80a4b60 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc7c009a880 tx=0x7fc7c00c54b0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.079+0000 7fc7d4f6e640 1 -- 172.21.15.70:0/2992944875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d006d490 msgr2=0x7fc7d01a66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.079+0000 7fc7d4f6e640 1 --2- 172.21.15.70:0/2992944875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d006d490 0x7fc7d01a66f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000b450 tx=0x7fc7c000b910 comp rx=0 tx=0).stop 2024-04-01T01:43:51.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.080+0000 7fc7d4f6e640 1 -- 172.21.15.70:0/2992944875 shutdown_connections 2024-04-01T01:43:51.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.080+0000 7fc7d4f6e640 1 --2- 172.21.15.70:0/2992944875 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc7b80a26a0 0x7fc7b80a4b60 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.080+0000 7fc7d4f6e640 1 --2- 172.21.15.70:0/2992944875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d006f070 0x7fc7d01adca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.080+0000 7fc7d4f6e640 1 --2- 172.21.15.70:0/2992944875 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7d006e6d0 0x7fc7d01a6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.080+0000 7fc7d4f6e640 1 --2- 172.21.15.70:0/2992944875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d006d490 0x7fc7d01a66f0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.080+0000 7fc7d4f6e640 1 -- 172.21.15.70:0/2992944875 >> 172.21.15.70:0/2992944875 conn(0x7fc7d006c110 msgr2=0x7fc7d01095e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.081+0000 7fc7d4f6e640 1 -- 172.21.15.70:0/2992944875 shutdown_connections 2024-04-01T01:43:51.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.081+0000 7fc7d4f6e640 1 -- 172.21.15.70:0/2992944875 wait complete. 2024-04-01T01:43:51.099 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:50 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3363456214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:43:51.100 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:43:51.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:50 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3363456214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:43:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.272+0000 7ff268735640 1 -- 172.21.15.70:0/2804935024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff260100770 msgr2=0x7ff260100bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.272+0000 7ff268735640 1 --2- 172.21.15.70:0/2804935024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff260100770 0x7ff260100bf0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff250009900 tx=0x7ff25002bc30 comp rx=0 tx=0).stop 2024-04-01T01:43:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.273+0000 7ff268735640 1 -- 172.21.15.70:0/2804935024 shutdown_connections 2024-04-01T01:43:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.273+0000 7ff268735640 1 --2- 172.21.15.70:0/2804935024 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff260101130 0x7ff2601059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.273+0000 7ff268735640 1 --2- 172.21.15.70:0/2804935024 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff260100770 0x7ff260100bf0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.273+0000 7ff268735640 1 --2- 172.21.15.70:0/2804935024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2600ffa10 0x7ff2600ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.273+0000 7ff268735640 1 -- 172.21.15.70:0/2804935024 >> 172.21.15.70:0/2804935024 conn(0x7ff2600fb340 msgr2=0x7ff2600fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.274+0000 7ff268735640 1 -- 172.21.15.70:0/2804935024 shutdown_connections 2024-04-01T01:43:51.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.274+0000 7ff268735640 1 -- 172.21.15.70:0/2804935024 wait complete. 2024-04-01T01:43:51.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.275+0000 7ff268735640 1 Processor -- start 2024-04-01T01:43:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.275+0000 7ff268735640 1 -- start start 2024-04-01T01:43:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.275+0000 7ff268735640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2600ffa10 0x7ff26019e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.275+0000 7ff268735640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff260100770 0x7ff26019e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.275+0000 7ff268735640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff260101130 0x7ff2601a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.275+0000 7ff268735640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff2601138f0 con 0x7ff260100770 2024-04-01T01:43:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.275+0000 7ff2664d1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2600ffa10 0x7ff26019e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.275+0000 7ff2664d1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2600ffa10 0x7ff26019e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41758/0 (socket says 172.21.15.70:41758) 2024-04-01T01:43:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.275+0000 7ff2664d1640 1 -- 172.21.15.70:0/3425190617 learned_addr learned my addr 172.21.15.70:0/3425190617 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.275+0000 7ff265cd0640 1 --2- 172.21.15.70:0/3425190617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff260100770 0x7ff26019e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.275+0000 7ff268735640 1 -- 172.21.15.70:0/3425190617 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff260113770 con 0x7ff2600ffa10 2024-04-01T01:43:51.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.275+0000 7ff268735640 1 -- 172.21.15.70:0/3425190617 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff260113a70 con 0x7ff260101130 2024-04-01T01:43:51.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.276+0000 7ff266cd2640 1 --2- 172.21.15.70:0/3425190617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff260101130 0x7ff2601a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.276+0000 7ff266cd2640 1 -- 172.21.15.70:0/3425190617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2600ffa10 msgr2=0x7ff26019e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.276+0000 7ff266cd2640 1 --2- 172.21.15.70:0/3425190617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2600ffa10 0x7ff26019e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.276+0000 7ff266cd2640 1 -- 172.21.15.70:0/3425190617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff260100770 msgr2=0x7ff26019e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.276+0000 7ff266cd2640 1 --2- 172.21.15.70:0/3425190617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff260100770 0x7ff26019e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.276+0000 7ff266cd2640 1 -- 172.21.15.70:0/3425190617 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2601a5c40 con 0x7ff260101130 2024-04-01T01:43:51.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.276+0000 7ff266cd2640 1 --2- 172.21.15.70:0/3425190617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff260101130 0x7ff2601a55d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff25800da10 tx=0x7ff25800ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.277+0000 7ff2577fe640 1 -- 172.21.15.70:0/3425190617 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25800bd70 con 0x7ff260101130 2024-04-01T01:43:51.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.277+0000 7ff2577fe640 1 -- 172.21.15.70:0/3425190617 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff258003e20 con 0x7ff260101130 2024-04-01T01:43:51.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.277+0000 7ff268735640 1 -- 172.21.15.70:0/3425190617 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2601a5f30 con 0x7ff260101130 2024-04-01T01:43:51.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.277+0000 7ff268735640 1 -- 172.21.15.70:0/3425190617 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2601a63a0 con 0x7ff260101130 2024-04-01T01:43:51.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.277+0000 7ff2577fe640 1 -- 172.21.15.70:0/3425190617 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25800bd70 con 0x7ff260101130 2024-04-01T01:43:51.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.280+0000 7ff26b3f0740 1 Processor -- start 2024-04-01T01:43:51.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.281+0000 7ff2577fe640 1 -- 172.21.15.70:0/3425190617 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7ff258003970 con 0x7ff260101130 2024-04-01T01:43:51.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.282+0000 7ff2577fe640 1 --2- 172.21.15.70:0/3425190617 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff24c0a2890 0x7ff24c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.282+0000 7ff2577fe640 1 -- 172.21.15.70:0/3425190617 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff25800bb70 con 0x7ff260101130 2024-04-01T01:43:51.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.283+0000 7ff26b3f0740 1 -- start start 2024-04-01T01:43:51.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.283+0000 7ff26b3f0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee81f480 0x561fee81f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.283+0000 7ff26b3f0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81fe70 0x561fee824970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.283+0000 7ff26b3f0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee824fe0 0x561fee827470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.283+0000 7ff26b3f0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561fee7841b0 con 0x561fee81f480 2024-04-01T01:43:51.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.283+0000 7ff26b3f0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561fee72cc00 con 0x561fee81fe70 2024-04-01T01:43:51.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.283+0000 7ff26b3f0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561fee780250 con 0x561fee824fe0 2024-04-01T01:43:51.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.283+0000 7ff265cd0640 1 --2- 172.21.15.70:0/3425190617 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff24c0a2890 0x7ff24c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.284+0000 7ff2664d1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee81f480 0x561fee81f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.284+0000 7ff2664d1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee81f480 0x561fee81f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47540/0 (socket says 172.21.15.70:47540) 2024-04-01T01:43:51.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.284+0000 7ff2664d1640 1 -- 172.21.15.70:0/1987659936 learned_addr learned my addr 172.21.15.70:0/1987659936 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.284+0000 7ff2664d1640 1 -- 172.21.15.70:0/1987659936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee824fe0 msgr2=0x561fee827470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:43:51.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.284+0000 7ff2664d1640 1 --2- 172.21.15.70:0/1987659936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee824fe0 0x561fee827470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.284+0000 7ff2664d1640 1 -- 172.21.15.70:0/1987659936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81fe70 msgr2=0x561fee824970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:43:51.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.284+0000 7ff2664d1640 1 --2- 172.21.15.70:0/1987659936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81fe70 0x561fee824970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.284+0000 7ff2664d1640 1 -- 172.21.15.70:0/1987659936 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fee74c3d0 con 0x561fee81f480 2024-04-01T01:43:51.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.285+0000 7ff2664d1640 1 --2- 172.21.15.70:0/1987659936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee81f480 0x561fee81f8a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff25c009e10 tx=0x7ff25c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc8e0869af66eb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.285+0000 7ff265cd0640 1 --2- 172.21.15.70:0/3425190617 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff24c0a2890 0x7ff24c0a4d50 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff250005ab0 tx=0x7ff250005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.286+0000 7ff268735640 1 -- 172.21.15.70:0/1987659936 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25c02a620 con 0x561fee81f480 2024-04-01T01:43:51.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.287+0000 7ff268735640 1 -- 172.21.15.70:0/1987659936 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff25c02ac70 con 0x561fee81f480 2024-04-01T01:43:51.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.287+0000 7ff268735640 1 -- 172.21.15.70:0/1987659936 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25c029d10 con 0x561fee81f480 2024-04-01T01:43:51.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.288+0000 7ff26b3f0740 1 -- 172.21.15.70:0/1987659936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee81f480 msgr2=0x561fee81f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.288+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/1987659936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee81f480 0x561fee81f8a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff25c009e10 tx=0x7ff25c011810 comp rx=0 tx=0).stop 2024-04-01T01:43:51.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.288+0000 7ff26b3f0740 1 -- 172.21.15.70:0/1987659936 shutdown_connections 2024-04-01T01:43:51.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.288+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/1987659936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee824fe0 0x561fee827470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.289+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/1987659936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81fe70 0x561fee824970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.289+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/1987659936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee81f480 0x561fee81f8a0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.289+0000 7ff26b3f0740 1 -- 172.21.15.70:0/1987659936 >> 172.21.15.70:0/1987659936 conn(0x561fee81ace0 msgr2=0x561fee81b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.289+0000 7ff26b3f0740 1 -- 172.21.15.70:0/1987659936 shutdown_connections 2024-04-01T01:43:51.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.289+0000 7ff26b3f0740 1 -- 172.21.15.70:0/1987659936 wait complete. 2024-04-01T01:43:51.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.290+0000 7ff26b3f0740 1 -- start start 2024-04-01T01:43:51.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.290+0000 7ff26b3f0740 1 Processor -- start 2024-04-01T01:43:51.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.290+0000 7ff26b3f0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee81f480 0x561fee829ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.290+0000 7ff26b3f0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81fe70 0x561fee82a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.290+0000 7ff26b3f0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee824fe0 0x561fee831060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.291+0000 7ff2664d1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee81f480 0x561fee829ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.291+0000 7ff2664d1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee81f480 0x561fee829ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47548/0 (socket says 172.21.15.70:47548) 2024-04-01T01:43:51.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.291+0000 7ff2664d1640 1 -- 172.21.15.70:0/2241501733 learned_addr learned my addr 172.21.15.70:0/2241501733 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.291+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2241501733 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561fee827f90 con 0x561fee81f480 2024-04-01T01:43:51.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.291+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2241501733 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561fee722470 con 0x561fee824fe0 2024-04-01T01:43:51.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.291+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2241501733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561fee828110 con 0x561fee81fe70 2024-04-01T01:43:51.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.291+0000 7ff26b3f0740 10 client.0 _ll_register_callbacks cb 0x561fee73fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:51.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.291+0000 7ff265cd0640 1 --2- 172.21.15.70:0/2241501733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81fe70 0x561fee82a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.291+0000 7ff266cd2640 1 --2- 172.21.15.70:0/2241501733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee824fe0 0x561fee831060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.291+0000 7ff265cd0640 1 -- 172.21.15.70:0/2241501733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee824fe0 msgr2=0x561fee831060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.291+0000 7ff265cd0640 1 --2- 172.21.15.70:0/2241501733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee824fe0 0x561fee831060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.291+0000 7ff265cd0640 1 -- 172.21.15.70:0/2241501733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee81f480 msgr2=0x561fee829ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.291+0000 7ff265cd0640 1 --2- 172.21.15.70:0/2241501733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee81f480 0x561fee829ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.291+0000 7ff265cd0640 1 -- 172.21.15.70:0/2241501733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fee4983f0 con 0x561fee81fe70 2024-04-01T01:43:51.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.292+0000 7ff265cd0640 1 --2- 172.21.15.70:0/2241501733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81fe70 0x561fee82a020 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff250036040 tx=0x7ff250009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40e1b685c1a65c3d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.292+0000 7ff249ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:43:51.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.292+0000 7ff249ffb640 1 -- 172.21.15.70:0/2241501733 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25004f020 con 0x561fee81fe70 2024-04-01T01:43:51.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.292+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2241501733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff25c02ac70 con 0x561fee81fe70 2024-04-01T01:43:51.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.292+0000 7ff249ffb640 1 -- 172.21.15.70:0/2241501733 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff2500046c0 con 0x561fee81fe70 2024-04-01T01:43:51.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.292+0000 7ff249ffb640 1 -- 172.21.15.70:0/2241501733 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff25005a050 con 0x561fee81fe70 2024-04-01T01:43:51.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.292+0000 7ff26b3f0740 10 client.28330 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:51.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.293+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2241501733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561fee5627d0 con 0x561fee81fe70 2024-04-01T01:43:51.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.293+0000 7ff249ffb640 1 -- 172.21.15.70:0/2241501733 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff250004d70 con 0x561fee81fe70 2024-04-01T01:43:51.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.293+0000 7ff249ffb640 1 -- 172.21.15.70:0/2241501733 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff2500564b0 con 0x561fee81fe70 2024-04-01T01:43:51.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.293+0000 7ff26b3f0740 10 client.28330 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:51.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.293+0000 7ff26b3f0740 10 client.28330 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:43:51.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.293+0000 7ff26b3f0740 20 client.28330 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:51.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.293+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/2241501733 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x561fee835590 0x561fee837a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.293+0000 7ff26b3f0740 4 client.28330 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:51.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.293+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2241501733 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff25c02ac70 con 0x561fee835590 2024-04-01T01:43:51.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.295+0000 7ff2664d1640 1 --2- 172.21.15.70:0/2241501733 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x561fee835590 0x561fee837a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.296+0000 7ff2664d1640 1 --2- 172.21.15.70:0/2241501733 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x561fee835590 0x561fee837a10 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c3c64d0a90c1fdd7 server_cookie=a5e02fc2ab410d0f in_seq=0 out_seq=0 2024-04-01T01:43:51.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.296+0000 7ff249ffb640 10 client.28330 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:43:51.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.299+0000 7ff249ffb640 1 -- 172.21.15.70:0/2241501733 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7ff25c02ac70 con 0x561fee835590 2024-04-01T01:43:51.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.299+0000 7ff249ffb640 10 client.28330 handle_command_reply: tid=0 2024-04-01T01:43:51.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.299+0000 7ff26b3f0740 1 client.28330 shutdown 2024-04-01T01:43:51.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.299+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2241501733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81fe70 msgr2=0x561fee82a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.299+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/2241501733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81fe70 0x561fee82a020 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff250036040 tx=0x7ff250009260 comp rx=0 tx=0).stop 2024-04-01T01:43:51.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.301+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2241501733 shutdown_connections 2024-04-01T01:43:51.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.301+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/2241501733 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x561fee835590 0x561fee837a10 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.301+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/2241501733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee824fe0 0x561fee831060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.301+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/2241501733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81fe70 0x561fee82a020 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.301+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/2241501733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee81f480 0x561fee829ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.301+0000 7ff249ffb640 0 client.28330 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:43:51.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.301+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2241501733 >> 172.21.15.70:0/2241501733 conn(0x561fee81ace0 msgr2=0x561fee819e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.301+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2241501733 shutdown_connections 2024-04-01T01:43:51.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.301+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2241501733 wait complete. 2024-04-01T01:43:51.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.301+0000 7ff26b3f0740 20 client.28330 trim_cache size 0 max 16384 2024-04-01T01:43:51.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.328+0000 7ff26b3f0740 1 Processor -- start 2024-04-01T01:43:51.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.328+0000 7ff26b3f0740 1 -- start start 2024-04-01T01:43:51.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:50 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3363456214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:43:51.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.328+0000 7ff26b3f0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81f480 0x561fee819ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.328+0000 7ff26b3f0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81fe70 0x561fee831020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.328+0000 7ff26b3f0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee824fe0 0x561fee83c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.329+0000 7ff26b3f0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff2601a8220 con 0x561fee824fe0 2024-04-01T01:43:51.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.329+0000 7ff26b3f0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff2601a9d60 con 0x561fee81f480 2024-04-01T01:43:51.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.329+0000 7ff26b3f0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff2601a88f0 con 0x561fee81fe70 2024-04-01T01:43:51.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.329+0000 7ff2664d1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81f480 0x561fee819ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.329+0000 7ff265cd0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81fe70 0x561fee831020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.329+0000 7ff265cd0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81fe70 0x561fee831020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42450/0 (socket says 172.21.15.70:42450) 2024-04-01T01:43:51.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.329+0000 7ff265cd0640 1 -- 172.21.15.70:0/4026431097 learned_addr learned my addr 172.21.15.70:0/4026431097 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.329+0000 7ff2664d1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81f480 0x561fee819ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41786/0 (socket says 172.21.15.70:41786) 2024-04-01T01:43:51.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.329+0000 7ff266cd2640 1 --2- 172.21.15.70:0/4026431097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee824fe0 0x561fee83c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.329+0000 7ff265cd0640 1 -- 172.21.15.70:0/4026431097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81f480 msgr2=0x561fee819ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.329+0000 7ff265cd0640 1 --2- 172.21.15.70:0/4026431097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81f480 0x561fee819ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.329+0000 7ff265cd0640 1 -- 172.21.15.70:0/4026431097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee824fe0 msgr2=0x561fee83c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.329+0000 7ff265cd0640 1 --2- 172.21.15.70:0/4026431097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee824fe0 0x561fee83c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.329+0000 7ff265cd0640 1 -- 172.21.15.70:0/4026431097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561fee820250 con 0x561fee81fe70 2024-04-01T01:43:51.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.329+0000 7ff2664d1640 1 --2- 172.21.15.70:0/4026431097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81f480 0x561fee819ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:43:51.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.330+0000 7ff265cd0640 1 --2- 172.21.15.70:0/4026431097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81fe70 0x561fee831020 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff250009900 tx=0x7ff25002cd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d27d63fb028c930 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.330+0000 7ff24bfff640 1 -- 172.21.15.70:0/4026431097 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff250062020 con 0x561fee81fe70 2024-04-01T01:43:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.330+0000 7ff24bfff640 1 -- 172.21.15.70:0/4026431097 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2500565a0 con 0x561fee81fe70 2024-04-01T01:43:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.330+0000 7ff24bfff640 1 -- 172.21.15.70:0/4026431097 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25004f040 con 0x561fee81fe70 2024-04-01T01:43:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.331+0000 7ff26b3f0740 1 -- 172.21.15.70:0/4026431097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81fe70 msgr2=0x561fee831020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.331+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/4026431097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81fe70 0x561fee831020 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff250009900 tx=0x7ff25002cd70 comp rx=0 tx=0).stop 2024-04-01T01:43:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.332+0000 7ff26b3f0740 1 -- 172.21.15.70:0/4026431097 shutdown_connections 2024-04-01T01:43:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.332+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/4026431097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee824fe0 0x561fee83c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.332+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/4026431097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81fe70 0x561fee831020 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.332+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/4026431097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81f480 0x561fee819ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.332+0000 7ff26b3f0740 1 -- 172.21.15.70:0/4026431097 >> 172.21.15.70:0/4026431097 conn(0x561fee81ace0 msgr2=0x561fee819590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.336+0000 7ff26b3f0740 1 -- 172.21.15.70:0/4026431097 shutdown_connections 2024-04-01T01:43:51.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.337+0000 7ff26b3f0740 1 -- 172.21.15.70:0/4026431097 wait complete. 2024-04-01T01:43:51.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.338+0000 7ff26b3f0740 1 -- start start 2024-04-01T01:43:51.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.338+0000 7ff26b3f0740 1 Processor -- start 2024-04-01T01:43:51.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.338+0000 7ff26b3f0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81f480 0x561fee830bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.338+0000 7ff26b3f0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81fe70 0x561fee82f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.339+0000 7ff26b3f0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee824fe0 0x561fee82f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.339+0000 7ff265cd0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81fe70 0x561fee82f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.339+0000 7ff265cd0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81fe70 0x561fee82f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41800/0 (socket says 172.21.15.70:41800) 2024-04-01T01:43:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.339+0000 7ff265cd0640 1 -- 172.21.15.70:0/2314923216 learned_addr learned my addr 172.21.15.70:0/2314923216 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.339+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2314923216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff2601a9690 con 0x561fee824fe0 2024-04-01T01:43:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.339+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2314923216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff2601a8fc0 con 0x561fee81fe70 2024-04-01T01:43:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.339+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2314923216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff26010f170 con 0x561fee81f480 2024-04-01T01:43:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.339+0000 7ff26b3f0740 10 client.0 _ll_register_callbacks cb 0x7ff250005e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.339+0000 7ff266cd2640 1 --2- 172.21.15.70:0/2314923216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee824fe0 0x561fee82f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.340+0000 7ff265cd0640 1 -- 172.21.15.70:0/2314923216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81f480 msgr2=0x561fee830bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:43:51.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.340+0000 7ff265cd0640 1 --2- 172.21.15.70:0/2314923216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81f480 0x561fee830bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.340+0000 7ff265cd0640 1 -- 172.21.15.70:0/2314923216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee824fe0 msgr2=0x561fee82f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.340+0000 7ff265cd0640 1 --2- 172.21.15.70:0/2314923216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee824fe0 0x561fee82f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.340+0000 7ff265cd0640 1 -- 172.21.15.70:0/2314923216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561fee82fc50 con 0x561fee81fe70 2024-04-01T01:43:51.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.341+0000 7ff265cd0640 1 --2- 172.21.15.70:0/2314923216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81fe70 0x561fee82f1d0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff250056300 tx=0x7ff25002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c0eb858449701ed server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.341+0000 7ff249ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:51.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.341+0000 7ff249ffb640 1 -- 172.21.15.70:0/2314923216 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25006b030 con 0x561fee81fe70 2024-04-01T01:43:51.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.341+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2314923216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2500565a0 con 0x561fee81fe70 2024-04-01T01:43:51.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.341+0000 7ff249ffb640 1 -- 172.21.15.70:0/2314923216 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff25003a040 con 0x561fee81fe70 2024-04-01T01:43:51.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.342+0000 7ff249ffb640 1 -- 172.21.15.70:0/2314923216 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff25004f070 con 0x561fee81fe70 2024-04-01T01:43:51.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.342+0000 7ff26b3f0740 10 client.28541 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:51.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.342+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2314923216 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561fee89c550 con 0x561fee81fe70 2024-04-01T01:43:51.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.342+0000 7ff249ffb640 1 -- 172.21.15.70:0/2314923216 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff2500035f0 con 0x561fee81fe70 2024-04-01T01:43:51.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.343+0000 7ff249ffb640 1 -- 172.21.15.70:0/2314923216 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff250041460 con 0x561fee81fe70 2024-04-01T01:43:51.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.343+0000 7ff26b3f0740 10 client.28541 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:51.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.343+0000 7ff26b3f0740 10 client.28541 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:43:51.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.343+0000 7ff26b3f0740 20 client.28541 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:51.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.343+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/2314923216 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x561fee89cf20 0x561fee89f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.343+0000 7ff26b3f0740 4 client.28541 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:43:51.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.343+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2314923216 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7ff2500565a0 con 0x561fee89cf20 2024-04-01T01:43:51.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.343+0000 7ff2664d1640 1 --2- 172.21.15.70:0/2314923216 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x561fee89cf20 0x561fee89f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.344+0000 7ff2664d1640 1 --2- 172.21.15.70:0/2314923216 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x561fee89cf20 0x561fee89f340 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=320e19e4af542031 server_cookie=8f5d87964d6d939b in_seq=0 out_seq=0 2024-04-01T01:43:51.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.344+0000 7ff249ffb640 10 client.28541 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:43:51.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.344+0000 7ff249ffb640 1 -- 172.21.15.70:0/2314923216 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7ff2500565a0 con 0x561fee89cf20 2024-04-01T01:43:51.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.344+0000 7ff249ffb640 10 client.28541 handle_command_reply: tid=0 2024-04-01T01:43:51.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.344+0000 7ff26b3f0740 1 client.28541 shutdown 2024-04-01T01:43:51.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.345+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2314923216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81fe70 msgr2=0x561fee82f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.345+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/2314923216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81fe70 0x561fee82f1d0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff250056300 tx=0x7ff25002bc30 comp rx=0 tx=0).stop 2024-04-01T01:43:51.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.345+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2314923216 shutdown_connections 2024-04-01T01:43:51.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.345+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/2314923216 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x561fee89cf20 0x561fee89f340 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.345+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/2314923216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561fee824fe0 0x561fee82f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.345+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/2314923216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561fee81fe70 0x561fee82f1d0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.345+0000 7ff26b3f0740 1 --2- 172.21.15.70:0/2314923216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561fee81f480 0x561fee830bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.345+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2314923216 >> 172.21.15.70:0/2314923216 conn(0x561fee81ace0 msgr2=0x561fee819e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.345+0000 7ff249ffb640 0 client.28541 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:43:51.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.345+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2314923216 shutdown_connections 2024-04-01T01:43:51.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.345+0000 7ff26b3f0740 1 -- 172.21.15.70:0/2314923216 wait complete. 2024-04-01T01:43:51.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.346+0000 7ff26b3f0740 20 client.28541 trim_cache size 0 max 16384 2024-04-01T01:43:51.346 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:43:51.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.347+0000 7ff2557fa640 1 -- 172.21.15.70:0/3425190617 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff24c0a2890 msgr2=0x7ff24c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.347+0000 7ff2557fa640 1 --2- 172.21.15.70:0/3425190617 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff24c0a2890 0x7ff24c0a4d50 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7ff250005ab0 tx=0x7ff250005fd0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.347+0000 7ff2557fa640 1 -- 172.21.15.70:0/3425190617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff260101130 msgr2=0x7ff2601a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.347+0000 7ff2557fa640 1 --2- 172.21.15.70:0/3425190617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff260101130 0x7ff2601a55d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff25800da10 tx=0x7ff25800ded0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.350+0000 7ff2557fa640 1 -- 172.21.15.70:0/3425190617 shutdown_connections 2024-04-01T01:43:51.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.350+0000 7ff2557fa640 1 --2- 172.21.15.70:0/3425190617 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff24c0a2890 0x7ff24c0a4d50 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.350+0000 7ff2557fa640 1 --2- 172.21.15.70:0/3425190617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff260101130 0x7ff2601a55d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.350+0000 7ff2557fa640 1 --2- 172.21.15.70:0/3425190617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff260100770 0x7ff26019e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.350+0000 7ff2557fa640 1 --2- 172.21.15.70:0/3425190617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2600ffa10 0x7ff26019e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.350+0000 7ff2557fa640 1 -- 172.21.15.70:0/3425190617 >> 172.21.15.70:0/3425190617 conn(0x7ff2600fb340 msgr2=0x7ff2601019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.353+0000 7ff2557fa640 1 -- 172.21.15.70:0/3425190617 shutdown_connections 2024-04-01T01:43:51.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.353+0000 7ff2557fa640 1 -- 172.21.15.70:0/3425190617 wait complete. 2024-04-01T01:43:51.376 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:43:51.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.571+0000 7fb134a53640 1 -- 172.21.15.70:0/3081785985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb130104a40 msgr2=0x7fb130104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.571+0000 7fb134a53640 1 --2- 172.21.15.70:0/3081785985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb130104a40 0x7fb130104ec0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fb120009900 tx=0x7fb12002bc30 comp rx=0 tx=0).stop 2024-04-01T01:43:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.571+0000 7fb134a53640 1 -- 172.21.15.70:0/3081785985 shutdown_connections 2024-04-01T01:43:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.571+0000 7fb134a53640 1 --2- 172.21.15.70:0/3081785985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb130105400 0x7fb13010bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.571+0000 7fb134a53640 1 --2- 172.21.15.70:0/3081785985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb130104a40 0x7fb130104ec0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.571+0000 7fb134a53640 1 --2- 172.21.15.70:0/3081785985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb130103850 0x7fb130103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.571+0000 7fb134a53640 1 -- 172.21.15.70:0/3081785985 >> 172.21.15.70:0/3081785985 conn(0x7fb1300fedc0 msgr2=0x7fb130101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.571+0000 7fb134a53640 1 -- 172.21.15.70:0/3081785985 shutdown_connections 2024-04-01T01:43:51.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.571+0000 7fb134a53640 1 -- 172.21.15.70:0/3081785985 wait complete. 2024-04-01T01:43:51.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.572+0000 7fb134a53640 1 Processor -- start 2024-04-01T01:43:51.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.572+0000 7fb134a53640 1 -- start start 2024-04-01T01:43:51.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.572+0000 7fb134a53640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb130103850 0x7fb130199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.572+0000 7fb134a53640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb130104a40 0x7fb130199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.572+0000 7fb134a53640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb130105400 0x7fb1301a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.572+0000 7fb134a53640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb13010f3b0 con 0x7fb130105400 2024-04-01T01:43:51.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.572+0000 7fb134a53640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb13010f230 con 0x7fb130104a40 2024-04-01T01:43:51.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.572+0000 7fb134a53640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb13010f530 con 0x7fb130103850 2024-04-01T01:43:51.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.572+0000 7fb127fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb130104a40 0x7fb130199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.572+0000 7fb12f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb130103850 0x7fb130199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.572+0000 7fb127fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb130104a40 0x7fb130199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41806/0 (socket says 172.21.15.70:41806) 2024-04-01T01:43:51.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.572+0000 7fb127fff640 1 -- 172.21.15.70:0/1706433699 learned_addr learned my addr 172.21.15.70:0/1706433699 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.572+0000 7fb12ffff640 1 --2- 172.21.15.70:0/1706433699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb130105400 0x7fb1301a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.573+0000 7fb127fff640 1 -- 172.21.15.70:0/1706433699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb130103850 msgr2=0x7fb130199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.573+0000 7fb127fff640 1 --2- 172.21.15.70:0/1706433699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb130103850 0x7fb130199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.573+0000 7fb127fff640 1 -- 172.21.15.70:0/1706433699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb130105400 msgr2=0x7fb1301a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.573+0000 7fb127fff640 1 --2- 172.21.15.70:0/1706433699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb130105400 0x7fb1301a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.573+0000 7fb127fff640 1 -- 172.21.15.70:0/1706433699 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1301a1490 con 0x7fb130104a40 2024-04-01T01:43:51.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.573+0000 7fb127fff640 1 --2- 172.21.15.70:0/1706433699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb130104a40 0x7fb130199db0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fb120009850 tx=0x7fb12002cdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.574+0000 7fb12d7fa640 1 -- 172.21.15.70:0/1706433699 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb120043020 con 0x7fb130104a40 2024-04-01T01:43:51.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.574+0000 7fb12d7fa640 1 -- 172.21.15.70:0/1706433699 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1200052b0 con 0x7fb130104a40 2024-04-01T01:43:51.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.574+0000 7fb134a53640 1 -- 172.21.15.70:0/1706433699 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1301a1720 con 0x7fb130104a40 2024-04-01T01:43:51.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.574+0000 7fb134a53640 1 -- 172.21.15.70:0/1706433699 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1301a1c70 con 0x7fb130104a40 2024-04-01T01:43:51.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.574+0000 7fb12d7fa640 1 -- 172.21.15.70:0/1706433699 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb120003d90 con 0x7fb130104a40 2024-04-01T01:43:51.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.577+0000 7fb12d7fa640 1 -- 172.21.15.70:0/1706433699 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb120046020 con 0x7fb130104a40 2024-04-01T01:43:51.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.577+0000 7fb138973740 1 Processor -- start 2024-04-01T01:43:51.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.578+0000 7fb138973740 1 -- start start 2024-04-01T01:43:51.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.578+0000 7fb138973740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 0x5562c225d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.578+0000 7fb138973740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c225d8a0 0x5562c22623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.578+0000 7fb138973740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c2262a10 0x5562c2264ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.578+0000 7fb138973740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5562c21b91b0 con 0x5562c225d8a0 2024-04-01T01:43:51.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.578+0000 7fb138973740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5562c2161c00 con 0x5562c2262a10 2024-04-01T01:43:51.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.578+0000 7fb138973740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5562c21b5250 con 0x5562c225ceb0 2024-04-01T01:43:51.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.578+0000 7fb127fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c2262a10 0x5562c2264ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.578+0000 7fb127fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c2262a10 0x5562c2264ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41816/0 (socket says 172.21.15.70:41816) 2024-04-01T01:43:51.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.578+0000 7fb127fff640 1 -- 172.21.15.70:0/2075635921 learned_addr learned my addr 172.21.15.70:0/2075635921 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.578+0000 7fb12ffff640 1 --2- 172.21.15.70:0/2075635921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 0x5562c225d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.579+0000 7fb12f7fe640 1 --2- 172.21.15.70:0/2075635921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c225d8a0 0x5562c22623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.579+0000 7fb12d7fa640 1 --2- 172.21.15.70:0/1706433699 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb11c0a26a0 0x7fb11c0a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.579+0000 7fb12ffff640 1 -- 172.21.15.70:0/2075635921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c2262a10 msgr2=0x5562c2264ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.579+0000 7fb12ffff640 1 --2- 172.21.15.70:0/2075635921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c2262a10 0x5562c2264ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.579+0000 7fb12ffff640 1 -- 172.21.15.70:0/2075635921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c225d8a0 msgr2=0x5562c22623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.579+0000 7fb12ffff640 1 --2- 172.21.15.70:0/2075635921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c225d8a0 0x5562c22623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.579+0000 7fb12ffff640 1 -- 172.21.15.70:0/2075635921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562c21813d0 con 0x5562c225ceb0 2024-04-01T01:43:51.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.579+0000 7fb12f7fe640 1 --2- 172.21.15.70:0/2075635921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c225d8a0 0x5562c22623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:43:51.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.579+0000 7fb12d7fa640 1 -- 172.21.15.70:0/1706433699 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb1200f0e60 con 0x7fb130104a40 2024-04-01T01:43:51.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.579+0000 7fb12ffff640 1 --2- 172.21.15.70:0/1706433699 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb11c0a26a0 0x7fb11c0a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.580+0000 7fb12ffff640 1 --2- 172.21.15.70:0/2075635921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 0x5562c225d2d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb128009f00 tx=0x7fb128011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40c36046b2beef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.580+0000 7fb125ffb640 1 -- 172.21.15.70:0/2075635921 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1280045c0 con 0x5562c225ceb0 2024-04-01T01:43:51.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.580+0000 7fb125ffb640 1 -- 172.21.15.70:0/2075635921 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb128004760 con 0x5562c225ceb0 2024-04-01T01:43:51.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.580+0000 7fb125ffb640 1 -- 172.21.15.70:0/2075635921 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12801c660 con 0x5562c225ceb0 2024-04-01T01:43:51.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.581+0000 7fb138973740 1 -- 172.21.15.70:0/2075635921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 msgr2=0x5562c225d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.581+0000 7fb138973740 1 --2- 172.21.15.70:0/2075635921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 0x5562c225d2d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb128009f00 tx=0x7fb128011810 comp rx=0 tx=0).stop 2024-04-01T01:43:51.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.581+0000 7fb12ffff640 1 --2- 172.21.15.70:0/1706433699 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb11c0a26a0 0x7fb11c0a4b60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb128020000 tx=0x7fb128021000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.581+0000 7fb138973740 1 -- 172.21.15.70:0/2075635921 shutdown_connections 2024-04-01T01:43:51.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.581+0000 7fb138973740 1 --2- 172.21.15.70:0/2075635921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c2262a10 0x5562c2264ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.581+0000 7fb138973740 1 --2- 172.21.15.70:0/2075635921 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c225d8a0 0x5562c22623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.581+0000 7fb138973740 1 --2- 172.21.15.70:0/2075635921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 0x5562c225d2d0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.581+0000 7fb138973740 1 -- 172.21.15.70:0/2075635921 >> 172.21.15.70:0/2075635921 conn(0x5562c224fc10 msgr2=0x5562c2250010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.582+0000 7fb138973740 1 -- 172.21.15.70:0/2075635921 shutdown_connections 2024-04-01T01:43:51.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.582+0000 7fb138973740 1 -- 172.21.15.70:0/2075635921 wait complete. 2024-04-01T01:43:51.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.583+0000 7fb138973740 1 -- start start 2024-04-01T01:43:51.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.583+0000 7fb138973740 1 Processor -- start 2024-04-01T01:43:51.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.583+0000 7fb138973740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 0x5562c2256310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.583+0000 7fb138973740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c225d8a0 0x5562c2256850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.583+0000 7fb138973740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c2262a10 0x5562c225b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.583+0000 7fb138973740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5562c2254840 con 0x5562c225d8a0 2024-04-01T01:43:51.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.583+0000 7fb138973740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5562c2157470 con 0x5562c2262a10 2024-04-01T01:43:51.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.583+0000 7fb138973740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5562c22549c0 con 0x5562c225ceb0 2024-04-01T01:43:51.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.583+0000 7fb138973740 10 client.0 _ll_register_callbacks cb 0x5562c2174cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:51.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.584+0000 7fb12ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c2262a10 0x5562c225b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.584+0000 7fb12ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c2262a10 0x5562c225b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41818/0 (socket says 172.21.15.70:41818) 2024-04-01T01:43:51.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.584+0000 7fb12f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 0x5562c2256310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.584+0000 7fb12ffff640 1 -- 172.21.15.70:0/1101035705 learned_addr learned my addr 172.21.15.70:0/1101035705 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.584+0000 7fb127fff640 1 --2- 172.21.15.70:0/1101035705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c225d8a0 0x5562c2256850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.584+0000 7fb127fff640 1 -- 172.21.15.70:0/1101035705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 msgr2=0x5562c2256310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.584+0000 7fb127fff640 1 --2- 172.21.15.70:0/1101035705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 0x5562c2256310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.584+0000 7fb127fff640 1 -- 172.21.15.70:0/1101035705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c2262a10 msgr2=0x5562c225b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.584+0000 7fb127fff640 1 --2- 172.21.15.70:0/1101035705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c2262a10 0x5562c225b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.584+0000 7fb127fff640 1 -- 172.21.15.70:0/1101035705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562c1ecd3f0 con 0x5562c225d8a0 2024-04-01T01:43:51.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.585+0000 7fb127fff640 1 --2- 172.21.15.70:0/1101035705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c225d8a0 0x5562c2256850 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb1200b9950 tx=0x7fb120005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da89ecabbffdde60 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.585+0000 7fb10effd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:43:51.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.585+0000 7fb10effd640 1 -- 172.21.15.70:0/1101035705 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1200b9790 con 0x5562c225d8a0 2024-04-01T01:43:51.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.585+0000 7fb10effd640 1 -- 172.21.15.70:0/1101035705 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb1200c5a70 con 0x5562c225d8a0 2024-04-01T01:43:51.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.585+0000 7fb138973740 1 -- 172.21.15.70:0/1101035705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb128004760 con 0x5562c225d8a0 2024-04-01T01:43:51.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.586+0000 7fb10effd640 1 -- 172.21.15.70:0/1101035705 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb120005460 con 0x5562c225d8a0 2024-04-01T01:43:51.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.586+0000 7fb138973740 10 client.18534 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:51.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.586+0000 7fb138973740 1 -- 172.21.15.70:0/1101035705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5562c1f977d0 con 0x5562c225d8a0 2024-04-01T01:43:51.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.587+0000 7fb10effd640 1 -- 172.21.15.70:0/1101035705 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb128004760 con 0x5562c225d8a0 2024-04-01T01:43:51.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.587+0000 7fb10effd640 1 -- 172.21.15.70:0/1101035705 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb1200dd0b0 con 0x5562c225d8a0 2024-04-01T01:43:51.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.587+0000 7fb138973740 10 client.18534 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:51.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.587+0000 7fb138973740 10 client.18534 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:43:51.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.587+0000 7fb138973740 20 client.18534 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:51.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.587+0000 7fb138973740 1 --2- 172.21.15.70:0/1101035705 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5562c227b7d0 0x5562c227dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.587+0000 7fb138973740 4 client.18534 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:51.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.587+0000 7fb138973740 1 -- 172.21.15.70:0/1101035705 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562c21c66f0 con 0x5562c227b7d0 2024-04-01T01:43:51.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.589+0000 7fb12f7fe640 1 --2- 172.21.15.70:0/1101035705 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5562c227b7d0 0x5562c227dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.589+0000 7fb12f7fe640 1 --2- 172.21.15.70:0/1101035705 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5562c227b7d0 0x5562c227dc30 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d3f3e4c692951ec7 server_cookie=dfa42736a587f0ca in_seq=0 out_seq=0 2024-04-01T01:43:51.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.589+0000 7fb10effd640 10 client.18534 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:43:51.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.591+0000 7fb10effd640 1 -- 172.21.15.70:0/1101035705 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5562c21c66f0 con 0x5562c227b7d0 2024-04-01T01:43:51.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.591+0000 7fb10effd640 10 client.18534 handle_command_reply: tid=0 2024-04-01T01:43:51.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.591+0000 7fb138973740 1 client.18534 shutdown 2024-04-01T01:43:51.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.591+0000 7fb138973740 1 -- 172.21.15.70:0/1101035705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c225d8a0 msgr2=0x5562c2256850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.591+0000 7fb138973740 1 --2- 172.21.15.70:0/1101035705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c225d8a0 0x5562c2256850 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb1200b9950 tx=0x7fb120005c90 comp rx=0 tx=0).stop 2024-04-01T01:43:51.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.591+0000 7fb138973740 1 -- 172.21.15.70:0/1101035705 shutdown_connections 2024-04-01T01:43:51.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.591+0000 7fb138973740 1 --2- 172.21.15.70:0/1101035705 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5562c227b7d0 0x5562c227dc30 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.591+0000 7fb138973740 1 --2- 172.21.15.70:0/1101035705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c2262a10 0x5562c225b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.591+0000 7fb138973740 1 --2- 172.21.15.70:0/1101035705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c225d8a0 0x5562c2256850 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.591+0000 7fb10effd640 0 client.18534 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:43:51.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.591+0000 7fb138973740 1 --2- 172.21.15.70:0/1101035705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 0x5562c2256310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.591+0000 7fb138973740 1 -- 172.21.15.70:0/1101035705 >> 172.21.15.70:0/1101035705 conn(0x5562c224fc10 msgr2=0x5562c224ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.591+0000 7fb138973740 1 -- 172.21.15.70:0/1101035705 shutdown_connections 2024-04-01T01:43:51.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.591+0000 7fb138973740 1 -- 172.21.15.70:0/1101035705 wait complete. 2024-04-01T01:43:51.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.592+0000 7fb138973740 20 client.18534 trim_cache size 0 max 16384 2024-04-01T01:43:51.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.622+0000 7fb138973740 1 Processor -- start 2024-04-01T01:43:51.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.623+0000 7fb138973740 1 -- start start 2024-04-01T01:43:51.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.623+0000 7fb138973740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c225ceb0 0x5562c2255d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.623+0000 7fb138973740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c225d8a0 0x5562c224e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.624+0000 7fb138973740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c2262a10 0x5562c224ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.624+0000 7fb138973740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb11000fde0 con 0x5562c225d8a0 2024-04-01T01:43:51.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.624+0000 7fb12ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c2262a10 0x5562c224ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.624+0000 7fb12ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c2262a10 0x5562c224ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42504/0 (socket says 172.21.15.70:42504) 2024-04-01T01:43:51.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.624+0000 7fb12ffff640 1 -- 172.21.15.70:0/2343180882 learned_addr learned my addr 172.21.15.70:0/2343180882 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.624+0000 7fb138973740 1 -- 172.21.15.70:0/2343180882 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb110011920 con 0x5562c225ceb0 2024-04-01T01:43:51.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.624+0000 7fb138973740 1 -- 172.21.15.70:0/2343180882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb1100104b0 con 0x5562c2262a10 2024-04-01T01:43:51.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.625+0000 7fb12ffff640 1 -- 172.21.15.70:0/2343180882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c225ceb0 msgr2=0x5562c2255d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.625+0000 7fb12ffff640 1 --2- 172.21.15.70:0/2343180882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c225ceb0 0x5562c2255d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.625+0000 7fb12ffff640 1 -- 172.21.15.70:0/2343180882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c225d8a0 msgr2=0x5562c224e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.625+0000 7fb12ffff640 1 --2- 172.21.15.70:0/2343180882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c225d8a0 0x5562c224e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.625+0000 7fb12ffff640 1 -- 172.21.15.70:0/2343180882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562c2250b80 con 0x5562c2262a10 2024-04-01T01:43:51.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.625+0000 7fb12ffff640 1 --2- 172.21.15.70:0/2343180882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c2262a10 0x5562c224ec40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb128012fd0 tx=0x7fb128027000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19225ba4d8d44821 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.625+0000 7fb1257fa640 1 -- 172.21.15.70:0/2343180882 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb128036020 con 0x5562c2262a10 2024-04-01T01:43:51.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.625+0000 7fb1257fa640 1 -- 172.21.15.70:0/2343180882 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1280045c0 con 0x5562c2262a10 2024-04-01T01:43:51.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.625+0000 7fb1257fa640 1 -- 172.21.15.70:0/2343180882 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12802e040 con 0x5562c2262a10 2024-04-01T01:43:51.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.626+0000 7fb138973740 1 -- 172.21.15.70:0/2343180882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c2262a10 msgr2=0x5562c224ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.626+0000 7fb138973740 1 --2- 172.21.15.70:0/2343180882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c2262a10 0x5562c224ec40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb128012fd0 tx=0x7fb128027000 comp rx=0 tx=0).stop 2024-04-01T01:43:51.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.627+0000 7fb138973740 1 -- 172.21.15.70:0/2343180882 shutdown_connections 2024-04-01T01:43:51.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.627+0000 7fb138973740 1 --2- 172.21.15.70:0/2343180882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c2262a10 0x5562c224ec40 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.627+0000 7fb138973740 1 --2- 172.21.15.70:0/2343180882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c225d8a0 0x5562c224e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.627+0000 7fb138973740 1 --2- 172.21.15.70:0/2343180882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c225ceb0 0x5562c2255d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.627+0000 7fb138973740 1 -- 172.21.15.70:0/2343180882 >> 172.21.15.70:0/2343180882 conn(0x5562c224fc10 msgr2=0x5562c2255600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.628+0000 7fb138973740 1 -- 172.21.15.70:0/2343180882 shutdown_connections 2024-04-01T01:43:51.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.628+0000 7fb138973740 1 -- 172.21.15.70:0/2343180882 wait complete. 2024-04-01T01:43:51.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.629+0000 7fb138973740 1 -- start start 2024-04-01T01:43:51.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.629+0000 7fb138973740 1 Processor -- start 2024-04-01T01:43:51.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.629+0000 7fb138973740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 0x5562c228cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.630+0000 7fb138973740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c225d8a0 0x5562c228d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.630+0000 7fb138973740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c2262a10 0x5562c228e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.630+0000 7fb138973740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb110011250 con 0x5562c2262a10 2024-04-01T01:43:51.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.630+0000 7fb138973740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb110010b80 con 0x5562c225d8a0 2024-04-01T01:43:51.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.630+0000 7fb138973740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb11000ddc0 con 0x5562c225ceb0 2024-04-01T01:43:51.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.630+0000 7fb138973740 10 client.0 _ll_register_callbacks cb 0x7fb120003f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:51.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.631+0000 7fb12f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 0x5562c228cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.631+0000 7fb12f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 0x5562c228cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42510/0 (socket says 172.21.15.70:42510) 2024-04-01T01:43:51.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.631+0000 7fb12f7fe640 1 -- 172.21.15.70:0/1656182774 learned_addr learned my addr 172.21.15.70:0/1656182774 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.632+0000 7fb12f7fe640 1 -- 172.21.15.70:0/1656182774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c225d8a0 msgr2=0x5562c228d1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:43:51.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.632+0000 7fb12f7fe640 1 --2- 172.21.15.70:0/1656182774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c225d8a0 0x5562c228d1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.632+0000 7fb12f7fe640 1 -- 172.21.15.70:0/1656182774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c2262a10 msgr2=0x5562c228e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.632+0000 7fb12f7fe640 1 --2- 172.21.15.70:0/1656182774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c2262a10 0x5562c228e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.632+0000 7fb12f7fe640 1 -- 172.21.15.70:0/1656182774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562c228e7e0 con 0x5562c225ceb0 2024-04-01T01:43:51.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.632+0000 7fb12f7fe640 1 --2- 172.21.15.70:0/1656182774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 0x5562c228cca0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb118009d70 tx=0x7fb11801ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=148dbdbd79112803 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.635+0000 7fb10effd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:43:51.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.635+0000 7fb10effd640 1 -- 172.21.15.70:0/1656182774 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11801e630 con 0x5562c225ceb0 2024-04-01T01:43:51.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.635+0000 7fb10effd640 1 -- 172.21.15.70:0/1656182774 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb118014a60 con 0x5562c225ceb0 2024-04-01T01:43:51.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.635+0000 7fb138973740 1 -- 172.21.15.70:0/1656182774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1280045c0 con 0x5562c225ceb0 2024-04-01T01:43:51.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.636+0000 7fb10effd640 1 -- 172.21.15.70:0/1656182774 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb11801e7d0 con 0x5562c225ceb0 2024-04-01T01:43:51.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.636+0000 7fb138973740 10 client.28351 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:51.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.636+0000 7fb138973740 1 -- 172.21.15.70:0/1656182774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5562c22e1e70 con 0x5562c225ceb0 2024-04-01T01:43:51.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.637+0000 7fb10effd640 1 -- 172.21.15.70:0/1656182774 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1280045c0 con 0x5562c225ceb0 2024-04-01T01:43:51.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.637+0000 7fb10effd640 1 -- 172.21.15.70:0/1656182774 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb118004980 con 0x5562c225ceb0 2024-04-01T01:43:51.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.637+0000 7fb138973740 10 client.28351 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:51.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.637+0000 7fb138973740 10 client.28351 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:43:51.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.637+0000 7fb138973740 20 client.28351 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:51.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.638+0000 7fb138973740 1 --2- 172.21.15.70:0/1656182774 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5562c22e2850 0x5562c22e4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.638+0000 7fb138973740 4 client.28351 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:43:51.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.638+0000 7fb138973740 1 -- 172.21.15.70:0/1656182774 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5562c21c0a40 con 0x5562c22e2850 2024-04-01T01:43:51.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.639+0000 7fb127fff640 1 --2- 172.21.15.70:0/1656182774 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5562c22e2850 0x5562c22e4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.640+0000 7fb10effd640 10 client.28351 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:43:51.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.640+0000 7fb127fff640 1 --2- 172.21.15.70:0/1656182774 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5562c22e2850 0x5562c22e4c70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=219897da4bd118c6 server_cookie=916199f578018222 in_seq=0 out_seq=0 2024-04-01T01:43:51.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.640+0000 7fb10effd640 1 -- 172.21.15.70:0/1656182774 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5562c21c0a40 con 0x5562c22e2850 2024-04-01T01:43:51.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.640+0000 7fb10effd640 10 client.28351 handle_command_reply: tid=0 2024-04-01T01:43:51.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.641+0000 7fb138973740 1 client.28351 shutdown 2024-04-01T01:43:51.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.647+0000 7fb138973740 1 -- 172.21.15.70:0/1656182774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 msgr2=0x5562c228cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.647+0000 7fb138973740 1 --2- 172.21.15.70:0/1656182774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 0x5562c228cca0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb118009d70 tx=0x7fb11801ef60 comp rx=0 tx=0).stop 2024-04-01T01:43:51.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.648+0000 7fb138973740 1 -- 172.21.15.70:0/1656182774 shutdown_connections 2024-04-01T01:43:51.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.648+0000 7fb138973740 1 --2- 172.21.15.70:0/1656182774 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5562c22e2850 0x5562c22e4c70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.649+0000 7fb138973740 1 --2- 172.21.15.70:0/1656182774 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562c2262a10 0x5562c228e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.649+0000 7fb138973740 1 --2- 172.21.15.70:0/1656182774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5562c225d8a0 0x5562c228d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.649+0000 7fb138973740 1 --2- 172.21.15.70:0/1656182774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562c225ceb0 0x5562c228cca0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.649+0000 7fb138973740 1 -- 172.21.15.70:0/1656182774 >> 172.21.15.70:0/1656182774 conn(0x5562c224fc10 msgr2=0x5562c2252e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.649+0000 7fb10effd640 0 client.28351 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:43:51.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.652+0000 7fb138973740 1 -- 172.21.15.70:0/1656182774 shutdown_connections 2024-04-01T01:43:51.652 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:43:51.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.652+0000 7fb138973740 1 -- 172.21.15.70:0/1656182774 wait complete. 2024-04-01T01:43:51.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.652+0000 7fb138973740 20 client.28351 trim_cache size 0 max 16384 2024-04-01T01:43:51.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.653+0000 7fb134a53640 1 -- 172.21.15.70:0/1706433699 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb11c0a26a0 msgr2=0x7fb11c0a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.653+0000 7fb134a53640 1 --2- 172.21.15.70:0/1706433699 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb11c0a26a0 0x7fb11c0a4b60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb128020000 tx=0x7fb128021000 comp rx=0 tx=0).stop 2024-04-01T01:43:51.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.653+0000 7fb134a53640 1 -- 172.21.15.70:0/1706433699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb130104a40 msgr2=0x7fb130199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.653+0000 7fb134a53640 1 --2- 172.21.15.70:0/1706433699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb130104a40 0x7fb130199db0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fb120009850 tx=0x7fb12002cdd0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.655+0000 7fb134a53640 1 -- 172.21.15.70:0/1706433699 shutdown_connections 2024-04-01T01:43:51.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.655+0000 7fb134a53640 1 --2- 172.21.15.70:0/1706433699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb130105400 0x7fb1301a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.655+0000 7fb134a53640 1 --2- 172.21.15.70:0/1706433699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb130104a40 0x7fb130199db0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.655+0000 7fb134a53640 1 --2- 172.21.15.70:0/1706433699 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb11c0a26a0 0x7fb11c0a4b60 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.655+0000 7fb134a53640 1 --2- 172.21.15.70:0/1706433699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb130103850 0x7fb130199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.655+0000 7fb134a53640 1 -- 172.21.15.70:0/1706433699 >> 172.21.15.70:0/1706433699 conn(0x7fb1300fedc0 msgr2=0x7fb130107c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.655+0000 7fb134a53640 1 -- 172.21.15.70:0/1706433699 shutdown_connections 2024-04-01T01:43:51.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.655+0000 7fb134a53640 1 -- 172.21.15.70:0/1706433699 wait complete. 2024-04-01T01:43:51.673 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:43:51.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.866+0000 7f337ffff640 1 -- 172.21.15.70:0/2058927485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33780a5470 msgr2=0x7f33780ac470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.866+0000 7f337ffff640 1 --2- 172.21.15.70:0/2058927485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33780a5470 0x7f33780ac470 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3374009900 tx=0x7f337402bc40 comp rx=0 tx=0).stop 2024-04-01T01:43:51.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.866+0000 7f337ffff640 1 -- 172.21.15.70:0/2058927485 shutdown_connections 2024-04-01T01:43:51.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.866+0000 7f337ffff640 1 --2- 172.21.15.70:0/2058927485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33780aca00 0x7f33780aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.866+0000 7f337ffff640 1 --2- 172.21.15.70:0/2058927485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33780a5470 0x7f33780ac470 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.866+0000 7f337ffff640 1 --2- 172.21.15.70:0/2058927485 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33780a4b10 0x7f33780a4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.866+0000 7f337ffff640 1 -- 172.21.15.70:0/2058927485 >> 172.21.15.70:0/2058927485 conn(0x7f33780a00d0 msgr2=0x7f33780a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.866+0000 7f337ffff640 1 -- 172.21.15.70:0/2058927485 shutdown_connections 2024-04-01T01:43:51.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.866+0000 7f337ffff640 1 -- 172.21.15.70:0/2058927485 wait complete. 2024-04-01T01:43:51.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.867+0000 7f337ffff640 1 Processor -- start 2024-04-01T01:43:51.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.867+0000 7f337ffff640 1 -- start start 2024-04-01T01:43:51.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.867+0000 7f337ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33780a4b10 0x7f337813f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.867+0000 7f337ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33780a5470 0x7f337813f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.867+0000 7f337ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33780aca00 0x7f3378146730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.868+0000 7f337ffff640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33780b28e0 con 0x7f33780a5470 2024-04-01T01:43:51.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.868+0000 7f337ffff640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f33780b2760 con 0x7f33780aca00 2024-04-01T01:43:51.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.868+0000 7f337ffff640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f33780b2a60 con 0x7f33780a4b10 2024-04-01T01:43:51.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.868+0000 7f337effd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33780a4b10 0x7f337813f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.868+0000 7f337effd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33780a4b10 0x7f337813f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42532/0 (socket says 172.21.15.70:42532) 2024-04-01T01:43:51.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.868+0000 7f337effd640 1 -- 172.21.15.70:0/886520926 learned_addr learned my addr 172.21.15.70:0/886520926 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.868+0000 7f337e7fc640 1 --2- 172.21.15.70:0/886520926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33780a5470 0x7f337813f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.868+0000 7f337f7fe640 1 --2- 172.21.15.70:0/886520926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33780aca00 0x7f3378146730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.868+0000 7f337e7fc640 1 -- 172.21.15.70:0/886520926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33780a4b10 msgr2=0x7f337813f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.868+0000 7f337e7fc640 1 --2- 172.21.15.70:0/886520926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33780a4b10 0x7f337813f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.869+0000 7f337e7fc640 1 -- 172.21.15.70:0/886520926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33780aca00 msgr2=0x7f3378146730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.869+0000 7f337e7fc640 1 --2- 172.21.15.70:0/886520926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33780aca00 0x7f3378146730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.869+0000 7f337e7fc640 1 -- 172.21.15.70:0/886520926 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3378146da0 con 0x7f33780a5470 2024-04-01T01:43:51.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.869+0000 7f337e7fc640 1 --2- 172.21.15.70:0/886520926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33780a5470 0x7f337813f710 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f3374036040 tx=0x7f33740040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.870+0000 7f3367fff640 1 -- 172.21.15.70:0/886520926 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3374043020 con 0x7f33780a5470 2024-04-01T01:43:51.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.870+0000 7f3367fff640 1 -- 172.21.15.70:0/886520926 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f33740044c0 con 0x7f33780a5470 2024-04-01T01:43:51.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.870+0000 7f3367fff640 1 -- 172.21.15.70:0/886520926 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3374003ae0 con 0x7f33780a5470 2024-04-01T01:43:51.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.870+0000 7f337ffff640 1 -- 172.21.15.70:0/886520926 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3378147030 con 0x7f33780a5470 2024-04-01T01:43:51.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.870+0000 7f337ffff640 1 -- 172.21.15.70:0/886520926 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3378147500 con 0x7f33780a5470 2024-04-01T01:43:51.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.874+0000 7f3367fff640 1 -- 172.21.15.70:0/886520926 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f3374046020 con 0x7f33780a5470 2024-04-01T01:43:51.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.874+0000 7f3388431740 1 Processor -- start 2024-04-01T01:43:51.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.875+0000 7f3388431740 1 -- start start 2024-04-01T01:43:51.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.875+0000 7f3388431740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3d9eb0 0x5585db3da2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.875+0000 7f3388431740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3da8a0 0x5585db3df3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.875+0000 7f3388431740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3dfa10 0x5585db3e1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.875+0000 7f3388431740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5585db3361b0 con 0x5585db3da8a0 2024-04-01T01:43:51.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.875+0000 7f3388431740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5585db2dec00 con 0x5585db3d9eb0 2024-04-01T01:43:51.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.875+0000 7f3388431740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5585db332250 con 0x5585db3dfa10 2024-04-01T01:43:51.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.875+0000 7f337effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3da8a0 0x5585db3df3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.875+0000 7f337effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3da8a0 0x5585db3df3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47658/0 (socket says 172.21.15.70:47658) 2024-04-01T01:43:51.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.875+0000 7f337effd640 1 -- 172.21.15.70:0/3199570289 learned_addr learned my addr 172.21.15.70:0/3199570289 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.875+0000 7f337f7fe640 1 --2- 172.21.15.70:0/3199570289 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3d9eb0 0x5585db3da2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.876+0000 7f337e7fc640 1 --2- 172.21.15.70:0/3199570289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3dfa10 0x5585db3e1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.876+0000 7f337f7fe640 1 -- 172.21.15.70:0/3199570289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3dfa10 msgr2=0x5585db3e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.876+0000 7f337f7fe640 1 --2- 172.21.15.70:0/3199570289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3dfa10 0x5585db3e1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.876+0000 7f337f7fe640 1 -- 172.21.15.70:0/3199570289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3da8a0 msgr2=0x5585db3df3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.876+0000 7f337f7fe640 1 --2- 172.21.15.70:0/3199570289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3da8a0 0x5585db3df3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.876+0000 7f337f7fe640 1 -- 172.21.15.70:0/3199570289 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585db2fe3d0 con 0x5585db3d9eb0 2024-04-01T01:43:51.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.876+0000 7f3367fff640 1 --2- 172.21.15.70:0/886520926 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f33680a25f0 0x7f33680a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.877+0000 7f3367fff640 1 -- 172.21.15.70:0/886520926 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f33740f0b20 con 0x7f33780a5470 2024-04-01T01:43:51.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.877+0000 7f337f7fe640 1 --2- 172.21.15.70:0/3199570289 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3d9eb0 0x5585db3da2d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3380067420 tx=0x7f3380072ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2112b27f20bf3aef server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.878+0000 7f33657fa640 1 -- 172.21.15.70:0/3199570289 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3380067dd0 con 0x5585db3d9eb0 2024-04-01T01:43:51.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.878+0000 7f33657fa640 1 -- 172.21.15.70:0/3199570289 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3380075e40 con 0x5585db3d9eb0 2024-04-01T01:43:51.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.878+0000 7f33657fa640 1 -- 172.21.15.70:0/3199570289 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3380069650 con 0x5585db3d9eb0 2024-04-01T01:43:51.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.878+0000 7f337effd640 1 --2- 172.21.15.70:0/886520926 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f33680a25f0 0x7f33680a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.879+0000 7f337effd640 1 --2- 172.21.15.70:0/886520926 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f33680a25f0 0x7f33680a4ab0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x5585db3daca0 tx=0x7f337000b480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.879+0000 7f3388431740 1 -- 172.21.15.70:0/3199570289 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3d9eb0 msgr2=0x5585db3da2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.879+0000 7f3388431740 1 --2- 172.21.15.70:0/3199570289 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3d9eb0 0x5585db3da2d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3380067420 tx=0x7f3380072ca0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.879+0000 7f3388431740 1 -- 172.21.15.70:0/3199570289 shutdown_connections 2024-04-01T01:43:51.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.879+0000 7f3388431740 1 --2- 172.21.15.70:0/3199570289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3dfa10 0x5585db3e1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.879+0000 7f3388431740 1 --2- 172.21.15.70:0/3199570289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3da8a0 0x5585db3df3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.879+0000 7f3388431740 1 --2- 172.21.15.70:0/3199570289 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3d9eb0 0x5585db3da2d0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.879+0000 7f3388431740 1 -- 172.21.15.70:0/3199570289 >> 172.21.15.70:0/3199570289 conn(0x5585db3d5420 msgr2=0x5585db3d5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.880+0000 7f3388431740 1 -- 172.21.15.70:0/3199570289 shutdown_connections 2024-04-01T01:43:51.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.880+0000 7f3388431740 1 -- 172.21.15.70:0/3199570289 wait complete. 2024-04-01T01:43:51.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.880+0000 7f3388431740 1 -- start start 2024-04-01T01:43:51.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.880+0000 7f3388431740 1 Processor -- start 2024-04-01T01:43:51.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.881+0000 7f3388431740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3d9eb0 0x5585db3e4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.881+0000 7f3388431740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3da8a0 0x5585db3e4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.882+0000 7f3388431740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 0x5585db3eb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.882+0000 7f3388431740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5585db3e29c0 con 0x5585db3dfa10 2024-04-01T01:43:51.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.882+0000 7f3388431740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5585db2d4470 con 0x5585db3da8a0 2024-04-01T01:43:51.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.882+0000 7f3388431740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5585db3e2b40 con 0x5585db3d9eb0 2024-04-01T01:43:51.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.882+0000 7f3388431740 10 client.0 _ll_register_callbacks cb 0x5585db2f1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:51.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.882+0000 7f337f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 0x5585db3eb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.882+0000 7f337f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 0x5585db3eb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47668/0 (socket says 172.21.15.70:47668) 2024-04-01T01:43:51.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.882+0000 7f337f7fe640 1 -- 172.21.15.70:0/1704762794 learned_addr learned my addr 172.21.15.70:0/1704762794 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.882+0000 7f337effd640 1 --2- 172.21.15.70:0/1704762794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3d9eb0 0x5585db3e4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.882+0000 7f337f7fe640 1 -- 172.21.15.70:0/1704762794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3d9eb0 msgr2=0x5585db3e4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.882+0000 7f337f7fe640 1 --2- 172.21.15.70:0/1704762794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3d9eb0 0x5585db3e4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.882+0000 7f337f7fe640 1 -- 172.21.15.70:0/1704762794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3da8a0 msgr2=0x5585db3e4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.882+0000 7f337f7fe640 1 --2- 172.21.15.70:0/1704762794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3da8a0 0x5585db3e4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.882+0000 7f337f7fe640 1 -- 172.21.15.70:0/1704762794 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585db04a3f0 con 0x5585db3dfa10 2024-04-01T01:43:51.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.882+0000 7f337effd640 1 --2- 172.21.15.70:0/1704762794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3d9eb0 0x5585db3e4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:43:51.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.884+0000 7f337f7fe640 1 --2- 172.21.15.70:0/1704762794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 0x5585db3eb9d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f3380073010 tx=0x7f3380067e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8da506e2646963f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.884+0000 7f3362ff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:43:51.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.885+0000 7f3362ff5640 1 -- 172.21.15.70:0/1704762794 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3380075c60 con 0x5585db3dfa10 2024-04-01T01:43:51.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.885+0000 7f3388431740 1 -- 172.21.15.70:0/1704762794 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5585db3436f0 con 0x5585db3dfa10 2024-04-01T01:43:51.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.885+0000 7f3362ff5640 1 -- 172.21.15.70:0/1704762794 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f33800693d0 con 0x5585db3dfa10 2024-04-01T01:43:51.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.886+0000 7f3362ff5640 1 -- 172.21.15.70:0/1704762794 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f33800695d0 con 0x5585db3dfa10 2024-04-01T01:43:51.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.886+0000 7f3388431740 10 client.18546 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:51.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.886+0000 7f3388431740 1 -- 172.21.15.70:0/1704762794 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5585db1147d0 con 0x5585db3dfa10 2024-04-01T01:43:51.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.886+0000 7f3362ff5640 1 -- 172.21.15.70:0/1704762794 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3380069770 con 0x5585db3dfa10 2024-04-01T01:43:51.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.886+0000 7f3362ff5640 1 -- 172.21.15.70:0/1704762794 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f3380089420 con 0x5585db3dfa10 2024-04-01T01:43:51.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.886+0000 7f3388431740 10 client.18546 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:51.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.886+0000 7f3388431740 10 client.18546 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:43:51.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.886+0000 7f3388431740 20 client.18546 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:51.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.886+0000 7f3388431740 1 --2- 172.21.15.70:0/1704762794 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5585db3f02e0 0x5585db3f2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.886+0000 7f3388431740 4 client.18546 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:51.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.886+0000 7f3388431740 1 -- 172.21.15.70:0/1704762794 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585db3436f0 con 0x5585db3f02e0 2024-04-01T01:43:51.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.887+0000 7f337effd640 1 --2- 172.21.15.70:0/1704762794 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5585db3f02e0 0x5585db3f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.890+0000 7f337effd640 1 --2- 172.21.15.70:0/1704762794 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5585db3f02e0 0x5585db3f2720 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e7ab0e23ffb2422d server_cookie=d589ce613e74f836 in_seq=0 out_seq=0 2024-04-01T01:43:51.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.890+0000 7f3362ff5640 10 client.18546 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:43:51.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.893+0000 7f3362ff5640 1 -- 172.21.15.70:0/1704762794 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5585db3436f0 con 0x5585db3f02e0 2024-04-01T01:43:51.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.893+0000 7f3362ff5640 10 client.18546 handle_command_reply: tid=0 2024-04-01T01:43:51.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.893+0000 7f3388431740 1 client.18546 shutdown 2024-04-01T01:43:51.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.894+0000 7f3388431740 1 -- 172.21.15.70:0/1704762794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 msgr2=0x5585db3eb9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.894+0000 7f3388431740 1 --2- 172.21.15.70:0/1704762794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 0x5585db3eb9d0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f3380073010 tx=0x7f3380067e00 comp rx=0 tx=0).stop 2024-04-01T01:43:51.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.895+0000 7f3388431740 1 -- 172.21.15.70:0/1704762794 shutdown_connections 2024-04-01T01:43:51.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.895+0000 7f3388431740 1 --2- 172.21.15.70:0/1704762794 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5585db3f02e0 0x5585db3f2720 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.895+0000 7f3388431740 1 --2- 172.21.15.70:0/1704762794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 0x5585db3eb9d0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.895+0000 7f3388431740 1 --2- 172.21.15.70:0/1704762794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3da8a0 0x5585db3e4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.895+0000 7f3388431740 1 --2- 172.21.15.70:0/1704762794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3d9eb0 0x5585db3e4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.895+0000 7f3388431740 1 -- 172.21.15.70:0/1704762794 >> 172.21.15.70:0/1704762794 conn(0x5585db3d5420 msgr2=0x5585db3d6670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.895+0000 7f3362ff5640 0 client.18546 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:43:51.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.895+0000 7f3388431740 1 -- 172.21.15.70:0/1704762794 shutdown_connections 2024-04-01T01:43:51.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.895+0000 7f3388431740 1 -- 172.21.15.70:0/1704762794 wait complete. 2024-04-01T01:43:51.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.895+0000 7f3388431740 20 client.18546 trim_cache size 0 max 16384 2024-04-01T01:43:51.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 545.0 s, 864.6 tps, lat 115.196 ms stddev 47.355 2024-04-01T01:43:51.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.922+0000 7f3388431740 1 Processor -- start 2024-04-01T01:43:51.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.922+0000 7f3388431740 1 -- start start 2024-04-01T01:43:51.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.922+0000 7f3388431740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3d9eb0 0x5585db3cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.922+0000 7f3388431740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3da8a0 0x5585db3eb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.922+0000 7f3388431740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 0x5585db3e2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.922+0000 7f3388431740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f337814d7e0 con 0x5585db3dfa10 2024-04-01T01:43:51.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.922+0000 7f3388431740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f337814f320 con 0x5585db3da8a0 2024-04-01T01:43:51.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.922+0000 7f3388431740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f337814deb0 con 0x5585db3d9eb0 2024-04-01T01:43:51.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.922+0000 7f337f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 0x5585db3e2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.922+0000 7f337f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 0x5585db3e2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47670/0 (socket says 172.21.15.70:47670) 2024-04-01T01:43:51.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.922+0000 7f337f7fe640 1 -- 172.21.15.70:0/1439719931 learned_addr learned my addr 172.21.15.70:0/1439719931 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.922+0000 7f337f7fe640 1 -- 172.21.15.70:0/1439719931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3d9eb0 msgr2=0x5585db3cbe10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:43:51.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.922+0000 7f337f7fe640 1 --2- 172.21.15.70:0/1439719931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3d9eb0 0x5585db3cbe10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.922+0000 7f337f7fe640 1 -- 172.21.15.70:0/1439719931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3da8a0 msgr2=0x5585db3eb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.922+0000 7f337e7fc640 1 --2- 172.21.15.70:0/1439719931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3da8a0 0x5585db3eb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.923+0000 7f337f7fe640 1 --2- 172.21.15.70:0/1439719931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3da8a0 0x5585db3eb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.923+0000 7f337f7fe640 1 -- 172.21.15.70:0/1439719931 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585db3c8670 con 0x5585db3dfa10 2024-04-01T01:43:51.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.923+0000 7f337e7fc640 1 --2- 172.21.15.70:0/1439719931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3da8a0 0x5585db3eb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:43:51.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.923+0000 7f337f7fe640 1 --2- 172.21.15.70:0/1439719931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 0x5585db3e2f40 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f338004f780 tx=0x7f338004fea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=328dd1282f06a4b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.923+0000 7f3364ff9640 1 -- 172.21.15.70:0/1439719931 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3380082040 con 0x5585db3dfa10 2024-04-01T01:43:51.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.923+0000 7f3364ff9640 1 -- 172.21.15.70:0/1439719931 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3380089a10 con 0x5585db3dfa10 2024-04-01T01:43:51.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.923+0000 7f3364ff9640 1 -- 172.21.15.70:0/1439719931 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33800699d0 con 0x5585db3dfa10 2024-04-01T01:43:51.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.925+0000 7f3388431740 1 -- 172.21.15.70:0/1439719931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 msgr2=0x5585db3e2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.926+0000 7f3388431740 1 --2- 172.21.15.70:0/1439719931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 0x5585db3e2f40 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f338004f780 tx=0x7f338004fea0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.926+0000 7f3388431740 1 -- 172.21.15.70:0/1439719931 shutdown_connections 2024-04-01T01:43:51.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.926+0000 7f3388431740 1 --2- 172.21.15.70:0/1439719931 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 0x5585db3e2f40 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.926+0000 7f3388431740 1 --2- 172.21.15.70:0/1439719931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3da8a0 0x5585db3eb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.926+0000 7f3388431740 1 --2- 172.21.15.70:0/1439719931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3d9eb0 0x5585db3cbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.926+0000 7f3388431740 1 -- 172.21.15.70:0/1439719931 >> 172.21.15.70:0/1439719931 conn(0x5585db3d5420 msgr2=0x5585db3cb700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.926+0000 7f3388431740 1 -- 172.21.15.70:0/1439719931 shutdown_connections 2024-04-01T01:43:51.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.926+0000 7f3388431740 1 -- 172.21.15.70:0/1439719931 wait complete. 2024-04-01T01:43:51.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.927+0000 7f3388431740 1 -- start start 2024-04-01T01:43:51.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.927+0000 7f3388431740 1 Processor -- start 2024-04-01T01:43:51.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.927+0000 7f3388431740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3d9eb0 0x5585db3e8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.927+0000 7f3388431740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3da8a0 0x5585db3e8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.927+0000 7f3388431740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 0x5585db454610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.927+0000 7f3388431740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f337814ec50 con 0x5585db3dfa10 2024-04-01T01:43:51.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.927+0000 7f3388431740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f337814e580 con 0x5585db3d9eb0 2024-04-01T01:43:51.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.927+0000 7f3388431740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f337814b7c0 con 0x5585db3da8a0 2024-04-01T01:43:51.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.927+0000 7f3388431740 10 client.0 _ll_register_callbacks cb 0x7f33800724e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:51.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.927+0000 7f337effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3d9eb0 0x5585db3e8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.927+0000 7f337effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3d9eb0 0x5585db3e8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41892/0 (socket says 172.21.15.70:41892) 2024-04-01T01:43:51.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.927+0000 7f337effd640 1 -- 172.21.15.70:0/143988866 learned_addr learned my addr 172.21.15.70:0/143988866 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:51.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.927+0000 7f337f7fe640 1 --2- 172.21.15.70:0/143988866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 0x5585db454610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.927+0000 7f337e7fc640 1 --2- 172.21.15.70:0/143988866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3da8a0 0x5585db3e8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.928+0000 7f337effd640 1 -- 172.21.15.70:0/143988866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3da8a0 msgr2=0x5585db3e8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.928+0000 7f337effd640 1 --2- 172.21.15.70:0/143988866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3da8a0 0x5585db3e8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.928+0000 7f337effd640 1 -- 172.21.15.70:0/143988866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 msgr2=0x5585db454610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:51.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.928+0000 7f337effd640 1 --2- 172.21.15.70:0/143988866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 0x5585db454610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.928+0000 7f337effd640 1 -- 172.21.15.70:0/143988866 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585db454bd0 con 0x5585db3d9eb0 2024-04-01T01:43:51.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.929+0000 7f337effd640 1 --2- 172.21.15.70:0/143988866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3d9eb0 0x5585db3e8a10 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x5585db3d6390 tx=0x7f33700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9653a7de4342dc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:51.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.929+0000 7f3362ff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:51.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.929+0000 7f3362ff5640 1 -- 172.21.15.70:0/143988866 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3370004020 con 0x5585db3d9eb0 2024-04-01T01:43:51.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.929+0000 7f3362ff5640 1 -- 172.21.15.70:0/143988866 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f33700041c0 con 0x5585db3d9eb0 2024-04-01T01:43:51.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.929+0000 7f3388431740 1 -- 172.21.15.70:0/143988866 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3380089a10 con 0x5585db3d9eb0 2024-04-01T01:43:51.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.929+0000 7f3362ff5640 1 -- 172.21.15.70:0/143988866 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3370038050 con 0x5585db3d9eb0 2024-04-01T01:43:51.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.930+0000 7f3388431740 10 client.28565 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:51.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.930+0000 7f3388431740 1 -- 172.21.15.70:0/143988866 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5585db455f10 con 0x5585db3d9eb0 2024-04-01T01:43:51.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.930+0000 7f3362ff5640 1 -- 172.21.15.70:0/143988866 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3380089a10 con 0x5585db3d9eb0 2024-04-01T01:43:51.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.930+0000 7f3362ff5640 1 -- 172.21.15.70:0/143988866 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f3370003a50 con 0x5585db3d9eb0 2024-04-01T01:43:51.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.930+0000 7f3388431740 10 client.28565 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:51.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.931+0000 7f3388431740 10 client.28565 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:43:51.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.931+0000 7f3388431740 20 client.28565 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:51.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.931+0000 7f3388431740 1 --2- 172.21.15.70:0/143988866 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5585db456970 0x5585db458d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:51.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.931+0000 7f3388431740 4 client.28565 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:43:51.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.931+0000 7f3388431740 1 -- 172.21.15.70:0/143988866 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5585db33da40 con 0x5585db456970 2024-04-01T01:43:51.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.932+0000 7f337e7fc640 1 --2- 172.21.15.70:0/143988866 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5585db456970 0x5585db458d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:51.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.937+0000 7f337e7fc640 1 --2- 172.21.15.70:0/143988866 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5585db456970 0x5585db458d90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=db68e4b7995fb6bd server_cookie=751c190d3af2e6c6 in_seq=0 out_seq=0 2024-04-01T01:43:51.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.937+0000 7f3362ff5640 10 client.28565 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:43:51.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.937+0000 7f3362ff5640 1 -- 172.21.15.70:0/143988866 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5585db33da40 con 0x5585db456970 2024-04-01T01:43:51.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.937+0000 7f3362ff5640 10 client.28565 handle_command_reply: tid=0 2024-04-01T01:43:51.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.938+0000 7f3388431740 1 client.28565 shutdown 2024-04-01T01:43:51.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.938+0000 7f3388431740 1 -- 172.21.15.70:0/143988866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3d9eb0 msgr2=0x5585db3e8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.938+0000 7f3388431740 1 --2- 172.21.15.70:0/143988866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3d9eb0 0x5585db3e8a10 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x5585db3d6390 tx=0x7f33700046c0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.938+0000 7f3388431740 1 -- 172.21.15.70:0/143988866 shutdown_connections 2024-04-01T01:43:51.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.938+0000 7f3388431740 1 --2- 172.21.15.70:0/143988866 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5585db456970 0x5585db458d90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.938+0000 7f3388431740 1 --2- 172.21.15.70:0/143988866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5585db3dfa10 0x5585db454610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.938+0000 7f3362ff5640 0 client.28565 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:43:51.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.938+0000 7f3388431740 1 --2- 172.21.15.70:0/143988866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585db3da8a0 0x5585db3e8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.938+0000 7f3388431740 1 --2- 172.21.15.70:0/143988866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5585db3d9eb0 0x5585db3e8a10 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.938+0000 7f3388431740 1 -- 172.21.15.70:0/143988866 >> 172.21.15.70:0/143988866 conn(0x5585db3d5420 msgr2=0x5585db3d8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.938+0000 7f3388431740 1 -- 172.21.15.70:0/143988866 shutdown_connections 2024-04-01T01:43:51.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.938+0000 7f3388431740 1 -- 172.21.15.70:0/143988866 wait complete. 2024-04-01T01:43:51.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.939+0000 7f3388431740 20 client.28565 trim_cache size 0 max 16384 2024-04-01T01:43:51.939 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:43:51.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.940+0000 7f337ffff640 1 -- 172.21.15.70:0/886520926 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f33680a25f0 msgr2=0x7f33680a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.940+0000 7f337ffff640 1 --2- 172.21.15.70:0/886520926 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f33680a25f0 0x7f33680a4ab0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x5585db3daca0 tx=0x7f337000b480 comp rx=0 tx=0).stop 2024-04-01T01:43:51.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.940+0000 7f337ffff640 1 -- 172.21.15.70:0/886520926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33780a5470 msgr2=0x7f337813f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:51.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.940+0000 7f337ffff640 1 --2- 172.21.15.70:0/886520926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33780a5470 0x7f337813f710 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f3374036040 tx=0x7f33740040c0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.940+0000 7f337ffff640 1 -- 172.21.15.70:0/886520926 shutdown_connections 2024-04-01T01:43:51.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.940+0000 7f337ffff640 1 --2- 172.21.15.70:0/886520926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33780aca00 0x7f3378146730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.940+0000 7f337ffff640 1 --2- 172.21.15.70:0/886520926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33780a5470 0x7f337813f710 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.940+0000 7f337ffff640 1 --2- 172.21.15.70:0/886520926 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f33680a25f0 0x7f33680a4ab0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.940+0000 7f337ffff640 1 --2- 172.21.15.70:0/886520926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33780a4b10 0x7f337813f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:51.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.940+0000 7f337ffff640 1 -- 172.21.15.70:0/886520926 >> 172.21.15.70:0/886520926 conn(0x7f33780a00d0 msgr2=0x7f33780a24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:51.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.940+0000 7f337ffff640 1 -- 172.21.15.70:0/886520926 shutdown_connections 2024-04-01T01:43:51.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:51.941+0000 7f337ffff640 1 -- 172.21.15.70:0/886520926 wait complete. 2024-04-01T01:43:51.959 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:43:51.959 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:51 smithi070 ceph-mon[32179]: pgmap v670: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.0 MiB/s wr, 791 op/s 2024-04-01T01:43:52.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.145+0000 7fbc13d7a640 1 -- 172.21.15.70:0/489795317 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc0c105520 msgr2=0x7fbc0c10bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:52.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.145+0000 7fbc13d7a640 1 --2- 172.21.15.70:0/489795317 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc0c105520 0x7fbc0c10bdb0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fbc04009be0 tx=0x7fbc0402bf30 comp rx=0 tx=0).stop 2024-04-01T01:43:52.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.146+0000 7fbc13d7a640 1 -- 172.21.15.70:0/489795317 shutdown_connections 2024-04-01T01:43:52.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.146+0000 7fbc13d7a640 1 --2- 172.21.15.70:0/489795317 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc0c105520 0x7fbc0c10bdb0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.146+0000 7fbc13d7a640 1 --2- 172.21.15.70:0/489795317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc0c104b60 0x7fbc0c104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.146+0000 7fbc13d7a640 1 --2- 172.21.15.70:0/489795317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc0c103970 0x7fbc0c103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.146+0000 7fbc13d7a640 1 -- 172.21.15.70:0/489795317 >> 172.21.15.70:0/489795317 conn(0x7fbc0c0feec0 msgr2=0x7fbc0c101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:52.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.146+0000 7fbc13d7a640 1 -- 172.21.15.70:0/489795317 shutdown_connections 2024-04-01T01:43:52.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.146+0000 7fbc13d7a640 1 -- 172.21.15.70:0/489795317 wait complete. 2024-04-01T01:43:52.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.147+0000 7fbc13d7a640 1 Processor -- start 2024-04-01T01:43:52.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.147+0000 7fbc13d7a640 1 -- start start 2024-04-01T01:43:52.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.147+0000 7fbc13d7a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc0c104b60 0x7fbc0c110ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.147+0000 7fbc13d7a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc0c105520 0x7fbc0c110ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.147+0000 7fbc13d7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc0c111580 0x7fbc0c118450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.147+0000 7fbc13d7a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c0778b0 con 0x7fbc0c104b60 2024-04-01T01:43:52.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.147+0000 7fbc13d7a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c077730 con 0x7fbc0c111580 2024-04-01T01:43:52.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.147+0000 7fbc13d7a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0c077a30 con 0x7fbc0c105520 2024-04-01T01:43:52.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.147+0000 7fbc12317640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc0c111580 0x7fbc0c118450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:52.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.147+0000 7fbc12317640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc0c111580 0x7fbc0c118450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41902/0 (socket says 172.21.15.70:41902) 2024-04-01T01:43:52.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.147+0000 7fbc12317640 1 -- 172.21.15.70:0/3865884596 learned_addr learned my addr 172.21.15.70:0/3865884596 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:52.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.147+0000 7fbc11b16640 1 --2- 172.21.15.70:0/3865884596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc0c104b60 0x7fbc0c110ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:52.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.148+0000 7fbc11315640 1 --2- 172.21.15.70:0/3865884596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc0c105520 0x7fbc0c110ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:52.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.148+0000 7fbc12317640 1 -- 172.21.15.70:0/3865884596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc0c105520 msgr2=0x7fbc0c110ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:52.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.148+0000 7fbc12317640 1 --2- 172.21.15.70:0/3865884596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc0c105520 0x7fbc0c110ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.148+0000 7fbc12317640 1 -- 172.21.15.70:0/3865884596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc0c104b60 msgr2=0x7fbc0c110ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:52.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.148+0000 7fbc12317640 1 --2- 172.21.15.70:0/3865884596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc0c104b60 0x7fbc0c110ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.148+0000 7fbc12317640 1 -- 172.21.15.70:0/3865884596 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0c118c80 con 0x7fbc0c111580 2024-04-01T01:43:52.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.149+0000 7fbc12317640 1 --2- 172.21.15.70:0/3865884596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc0c111580 0x7fbc0c118450 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbc04036040 tx=0x7fbc04033780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:52.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.149+0000 7fbc02ffd640 1 -- 172.21.15.70:0/3865884596 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04043020 con 0x7fbc0c111580 2024-04-01T01:43:52.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.150+0000 7fbc02ffd640 1 -- 172.21.15.70:0/3865884596 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc04003b80 con 0x7fbc0c111580 2024-04-01T01:43:52.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.150+0000 7fbc02ffd640 1 -- 172.21.15.70:0/3865884596 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04034690 con 0x7fbc0c111580 2024-04-01T01:43:52.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.150+0000 7fbc13d7a640 1 -- 172.21.15.70:0/3865884596 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc0c118f10 con 0x7fbc0c111580 2024-04-01T01:43:52.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.150+0000 7fbc13d7a640 1 -- 172.21.15.70:0/3865884596 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc0c119490 con 0x7fbc0c111580 2024-04-01T01:43:52.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.153+0000 7fbc02ffd640 1 -- 172.21.15.70:0/3865884596 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fbc04046020 con 0x7fbc0c111580 2024-04-01T01:43:52.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.155+0000 7fbc165fd740 1 Processor -- start 2024-04-01T01:43:52.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.155+0000 7fbc02ffd640 1 --2- 172.21.15.70:0/3865884596 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbf80a26a0 0x7fbbf80a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.155+0000 7fbc02ffd640 1 -- 172.21.15.70:0/3865884596 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fbc040f1570 con 0x7fbc0c111580 2024-04-01T01:43:52.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.156+0000 7fbc165fd740 1 -- start start 2024-04-01T01:43:52.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.156+0000 7fbc11315640 1 --2- 172.21.15.70:0/3865884596 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbf80a26a0 0x7fbbf80a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:52.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.157+0000 7fbc165fd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b12b490 0x55e47b12b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.157+0000 7fbc165fd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12be80 0x55e47b130980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.157+0000 7fbc165fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b130ff0 0x55e47b133480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.157+0000 7fbc165fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e47b0901b0 con 0x55e47b130ff0 2024-04-01T01:43:52.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.157+0000 7fbc165fd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e47b038c00 con 0x55e47b12be80 2024-04-01T01:43:52.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.157+0000 7fbc11315640 1 --2- 172.21.15.70:0/3865884596 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbf80a26a0 0x7fbbf80a4b60 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c112190 tx=0x7fbbfc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:52.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.157+0000 7fbc11315640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12be80 0x55e47b130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:52.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.157+0000 7fbc11315640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12be80 0x55e47b130980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41918/0 (socket says 172.21.15.70:41918) 2024-04-01T01:43:52.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.157+0000 7fbc11315640 1 -- 172.21.15.70:0/4168696527 learned_addr learned my addr 172.21.15.70:0/4168696527 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:52.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.157+0000 7fbc165fd740 1 -- 172.21.15.70:0/4168696527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e47b08c250 con 0x55e47b12b490 2024-04-01T01:43:52.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.158+0000 7fbc11315640 1 -- 172.21.15.70:0/4168696527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b12b490 msgr2=0x55e47b12b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:52.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.158+0000 7fbc11315640 1 --2- 172.21.15.70:0/4168696527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b12b490 0x55e47b12b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.158+0000 7fbc11315640 1 -- 172.21.15.70:0/4168696527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b130ff0 msgr2=0x55e47b133480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:43:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.158+0000 7fbc11315640 1 --2- 172.21.15.70:0/4168696527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b130ff0 0x55e47b133480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.158+0000 7fbc11315640 1 -- 172.21.15.70:0/4168696527 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e47b0583d0 con 0x55e47b12be80 2024-04-01T01:43:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.158+0000 7fbc11315640 1 --2- 172.21.15.70:0/4168696527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12be80 0x55e47b130980 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00caf0 tx=0x7fbbfc00f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2588246183f9127b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.159+0000 7fbbf7fff640 1 -- 172.21.15.70:0/4168696527 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc019040 con 0x55e47b12be80 2024-04-01T01:43:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.159+0000 7fbbf7fff640 1 -- 172.21.15.70:0/4168696527 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbfc022070 con 0x55e47b12be80 2024-04-01T01:43:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.159+0000 7fbbf7fff640 1 -- 172.21.15.70:0/4168696527 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc0113d0 con 0x55e47b12be80 2024-04-01T01:43:52.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.160+0000 7fbc165fd740 1 -- 172.21.15.70:0/4168696527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12be80 msgr2=0x55e47b130980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:52.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.160+0000 7fbc165fd740 1 --2- 172.21.15.70:0/4168696527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12be80 0x55e47b130980 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00caf0 tx=0x7fbbfc00f000 comp rx=0 tx=0).stop 2024-04-01T01:43:52.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.160+0000 7fbc165fd740 1 -- 172.21.15.70:0/4168696527 shutdown_connections 2024-04-01T01:43:52.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.160+0000 7fbc165fd740 1 --2- 172.21.15.70:0/4168696527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b130ff0 0x55e47b133480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.160+0000 7fbc165fd740 1 --2- 172.21.15.70:0/4168696527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12be80 0x55e47b130980 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.160+0000 7fbc165fd740 1 --2- 172.21.15.70:0/4168696527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b12b490 0x55e47b12b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.160+0000 7fbc165fd740 1 -- 172.21.15.70:0/4168696527 >> 172.21.15.70:0/4168696527 conn(0x55e47b126a00 msgr2=0x55e47b126e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:52.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.162+0000 7fbc165fd740 1 -- 172.21.15.70:0/4168696527 shutdown_connections 2024-04-01T01:43:52.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.162+0000 7fbc165fd740 1 -- 172.21.15.70:0/4168696527 wait complete. 2024-04-01T01:43:52.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.162+0000 7fbc165fd740 1 -- start start 2024-04-01T01:43:52.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.162+0000 7fbc165fd740 1 Processor -- start 2024-04-01T01:43:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.163+0000 7fbc165fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b12b490 0x55e47b135a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.163+0000 7fbc165fd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12be80 0x55e47b135f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.163+0000 7fbc165fd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b130ff0 0x55e47b13cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.163+0000 7fbc165fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e47b133fa0 con 0x55e47b12b490 2024-04-01T01:43:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.163+0000 7fbc165fd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e47b02e470 con 0x55e47b12be80 2024-04-01T01:43:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.163+0000 7fbc165fd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e47b134120 con 0x55e47b130ff0 2024-04-01T01:43:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.163+0000 7fbc165fd740 10 client.0 _ll_register_callbacks cb 0x55e47b04bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.163+0000 7fbc11315640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12be80 0x55e47b135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.163+0000 7fbc12317640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b130ff0 0x55e47b13cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.163+0000 7fbc12317640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b130ff0 0x55e47b13cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42612/0 (socket says 172.21.15.70:42612) 2024-04-01T01:43:52.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.163+0000 7fbc12317640 1 -- 172.21.15.70:0/226543401 learned_addr learned my addr 172.21.15.70:0/226543401 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.164+0000 7fbc11315640 1 -- 172.21.15.70:0/226543401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b130ff0 msgr2=0x55e47b13cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.164+0000 7fbc11315640 1 --2- 172.21.15.70:0/226543401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b130ff0 0x55e47b13cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.164+0000 7fbc11315640 1 -- 172.21.15.70:0/226543401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b12b490 msgr2=0x55e47b135a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.164+0000 7fbc11315640 1 --2- 172.21.15.70:0/226543401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b12b490 0x55e47b135a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.164+0000 7fbc11315640 1 -- 172.21.15.70:0/226543401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e47ada43f0 con 0x55e47b12be80 2024-04-01T01:43:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.164+0000 7fbc12317640 1 --2- 172.21.15.70:0/226543401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b130ff0 0x55e47b13cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:43:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.165+0000 7fbc11315640 1 --2- 172.21.15.70:0/226543401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12be80 0x55e47b135f70 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00cfd0 tx=0x7fbbfc01d9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69df186ec3ff5048 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:52.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.165+0000 7fbbf57fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:52.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.165+0000 7fbbf57fa640 1 -- 172.21.15.70:0/226543401 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc02c070 con 0x55e47b12be80 2024-04-01T01:43:52.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.165+0000 7fbbf57fa640 1 -- 172.21.15.70:0/226543401 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fbbfc013980 con 0x55e47b12be80 2024-04-01T01:43:52.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.165+0000 7fbc165fd740 1 -- 172.21.15.70:0/226543401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbfc022070 con 0x55e47b12be80 2024-04-01T01:43:52.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.166+0000 7fbbf57fa640 1 -- 172.21.15.70:0/226543401 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbbfc035050 con 0x55e47b12be80 2024-04-01T01:43:52.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.166+0000 7fbc165fd740 10 client.28574 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:52.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.166+0000 7fbc165fd740 1 -- 172.21.15.70:0/226543401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e47ae6e7d0 con 0x55e47b12be80 2024-04-01T01:43:52.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.166+0000 7fbbf57fa640 1 -- 172.21.15.70:0/226543401 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbfc022070 con 0x55e47b12be80 2024-04-01T01:43:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.167+0000 7fbbf57fa640 1 -- 172.21.15.70:0/226543401 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fbbfc01c030 con 0x55e47b12be80 2024-04-01T01:43:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.167+0000 7fbc165fd740 10 client.28574 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.167+0000 7fbc165fd740 10 client.28574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.167+0000 7fbc165fd740 20 client.28574 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.167+0000 7fbc165fd740 1 --2- 172.21.15.70:0/226543401 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e47b1411f0 0x55e47b1435e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.168+0000 7fbc165fd740 4 client.28574 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:52.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.168+0000 7fbc11b16640 1 --2- 172.21.15.70:0/226543401 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e47b1411f0 0x55e47b1435e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:52.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.169+0000 7fbc165fd740 1 -- 172.21.15.70:0/226543401 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e47b09d6f0 con 0x55e47b1411f0 2024-04-01T01:43:52.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.169+0000 7fbc11b16640 1 --2- 172.21.15.70:0/226543401 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e47b1411f0 0x55e47b1435e0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=423cc4703e1762ea server_cookie=803b5eff72c841c9 in_seq=0 out_seq=0 2024-04-01T01:43:52.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.169+0000 7fbbf57fa640 10 client.28574 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:52.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.171+0000 7fbbf57fa640 1 -- 172.21.15.70:0/226543401 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e47b09d6f0 con 0x55e47b1411f0 2024-04-01T01:43:52.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.171+0000 7fbbf57fa640 10 client.28574 handle_command_reply: tid=0 2024-04-01T01:43:52.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.171+0000 7fbc165fd740 1 client.28574 shutdown 2024-04-01T01:43:52.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.172+0000 7fbc165fd740 1 -- 172.21.15.70:0/226543401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12be80 msgr2=0x55e47b135f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:52.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.172+0000 7fbc165fd740 1 --2- 172.21.15.70:0/226543401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12be80 0x55e47b135f70 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc00cfd0 tx=0x7fbbfc01d9b0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.172+0000 7fbc165fd740 1 -- 172.21.15.70:0/226543401 shutdown_connections 2024-04-01T01:43:52.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.172+0000 7fbc165fd740 1 --2- 172.21.15.70:0/226543401 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e47b1411f0 0x55e47b1435e0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.172+0000 7fbbf57fa640 0 client.28574 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:52.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.172+0000 7fbc165fd740 1 --2- 172.21.15.70:0/226543401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b130ff0 0x55e47b13cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.172+0000 7fbc165fd740 1 --2- 172.21.15.70:0/226543401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12be80 0x55e47b135f70 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.172+0000 7fbc165fd740 1 --2- 172.21.15.70:0/226543401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b12b490 0x55e47b135a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.173+0000 7fbc165fd740 1 -- 172.21.15.70:0/226543401 >> 172.21.15.70:0/226543401 conn(0x55e47b126a00 msgr2=0x55e47b125d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:52.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.173+0000 7fbc165fd740 1 -- 172.21.15.70:0/226543401 shutdown_connections 2024-04-01T01:43:52.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.173+0000 7fbc165fd740 1 -- 172.21.15.70:0/226543401 wait complete. 2024-04-01T01:43:52.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.175+0000 7fbc165fd740 20 client.28574 trim_cache size 0 max 16384 2024-04-01T01:43:52.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.202+0000 7fbc165fd740 1 Processor -- start 2024-04-01T01:43:52.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.202+0000 7fbc165fd740 1 -- start start 2024-04-01T01:43:52.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.202+0000 7fbc165fd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12b490 0x55e47b1254b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.202+0000 7fbc165fd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b12be80 0x55e47b13ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.202+0000 7fbc165fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b130ff0 0x55e47b1259f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.202+0000 7fbc165fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf000fde0 con 0x55e47b130ff0 2024-04-01T01:43:52.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.202+0000 7fbc165fd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf0011920 con 0x55e47b12b490 2024-04-01T01:43:52.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.202+0000 7fbc165fd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf00104b0 con 0x55e47b12be80 2024-04-01T01:43:52.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.203+0000 7fbc11b16640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12b490 0x55e47b1254b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:52.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.203+0000 7fbc11315640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b12be80 0x55e47b13ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:52.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.203+0000 7fbc12317640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b130ff0 0x55e47b1259f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:52.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.203+0000 7fbc11315640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b12be80 0x55e47b13ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42614/0 (socket says 172.21.15.70:42614) 2024-04-01T01:43:52.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.203+0000 7fbc11315640 1 -- 172.21.15.70:0/412542495 learned_addr learned my addr 172.21.15.70:0/412542495 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:52.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.203+0000 7fbc11b16640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12b490 0x55e47b1254b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41940/0 (socket says 172.21.15.70:41940) 2024-04-01T01:43:52.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.203+0000 7fbc11315640 1 -- 172.21.15.70:0/412542495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12b490 msgr2=0x55e47b1254b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:52.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.203+0000 7fbc11315640 1 --2- 172.21.15.70:0/412542495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12b490 0x55e47b1254b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.203+0000 7fbc11315640 1 -- 172.21.15.70:0/412542495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b130ff0 msgr2=0x55e47b1259f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:52.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.203+0000 7fbc11315640 1 --2- 172.21.15.70:0/412542495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b130ff0 0x55e47b1259f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.203+0000 7fbc11315640 1 -- 172.21.15.70:0/412542495 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e47b122460 con 0x55e47b12be80 2024-04-01T01:43:52.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.203+0000 7fbc12317640 1 --2- 172.21.15.70:0/412542495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b130ff0 0x55e47b1259f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:43:52.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.203+0000 7fbc11315640 1 --2- 172.21.15.70:0/412542495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b12be80 0x55e47b13ced0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc008010 tx=0x7fbbfc012ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6efcb0416378a96 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:52.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.204+0000 7fbbf77fe640 1 -- 172.21.15.70:0/412542495 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc02c070 con 0x55e47b12be80 2024-04-01T01:43:52.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.204+0000 7fbbf77fe640 1 -- 172.21.15.70:0/412542495 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbfc022070 con 0x55e47b12be80 2024-04-01T01:43:52.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.204+0000 7fbbf77fe640 1 -- 172.21.15.70:0/412542495 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbfc0113d0 con 0x55e47b12be80 2024-04-01T01:43:52.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.205+0000 7fbc165fd740 1 -- 172.21.15.70:0/412542495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b12be80 msgr2=0x55e47b13ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:52.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.205+0000 7fbc165fd740 1 --2- 172.21.15.70:0/412542495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b12be80 0x55e47b13ced0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc008010 tx=0x7fbbfc012ac0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.208+0000 7fbc165fd740 1 -- 172.21.15.70:0/412542495 shutdown_connections 2024-04-01T01:43:52.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.208+0000 7fbc165fd740 1 --2- 172.21.15.70:0/412542495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b130ff0 0x55e47b1259f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.208+0000 7fbc165fd740 1 --2- 172.21.15.70:0/412542495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b12be80 0x55e47b13ced0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.208+0000 7fbc165fd740 1 --2- 172.21.15.70:0/412542495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12b490 0x55e47b1254b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.208+0000 7fbc165fd740 1 -- 172.21.15.70:0/412542495 >> 172.21.15.70:0/412542495 conn(0x55e47b126a00 msgr2=0x55e47b12d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:52.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.208+0000 7fbc165fd740 1 -- 172.21.15.70:0/412542495 shutdown_connections 2024-04-01T01:43:52.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.209+0000 7fbc165fd740 1 -- 172.21.15.70:0/412542495 wait complete. 2024-04-01T01:43:52.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.209+0000 7fbc165fd740 1 -- start start 2024-04-01T01:43:52.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.209+0000 7fbc165fd740 1 Processor -- start 2024-04-01T01:43:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.210+0000 7fbc165fd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12b490 0x55e47b13a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.210+0000 7fbc165fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b12be80 0x55e47b13a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.210+0000 7fbc165fd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b130ff0 0x55e47b13b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.210+0000 7fbc11b16640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12b490 0x55e47b13a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:52.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.210+0000 7fbc11b16640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12b490 0x55e47b13a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41944/0 (socket says 172.21.15.70:41944) 2024-04-01T01:43:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.210+0000 7fbc11b16640 1 -- 172.21.15.70:0/735313954 learned_addr learned my addr 172.21.15.70:0/735313954 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.211+0000 7fbc12317640 1 --2- 172.21.15.70:0/735313954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b130ff0 0x55e47b13b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.211+0000 7fbc165fd740 1 -- 172.21.15.70:0/735313954 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf0011250 con 0x55e47b12be80 2024-04-01T01:43:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.211+0000 7fbc11315640 1 --2- 172.21.15.70:0/735313954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b12be80 0x55e47b13a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.211+0000 7fbc165fd740 1 -- 172.21.15.70:0/735313954 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf0010b80 con 0x55e47b12b490 2024-04-01T01:43:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.211+0000 7fbc165fd740 1 -- 172.21.15.70:0/735313954 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf000dd60 con 0x55e47b130ff0 2024-04-01T01:43:52.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.211+0000 7fbc165fd740 10 client.0 _ll_register_callbacks cb 0x7fbbfc00c7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:52.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.211+0000 7fbc11b16640 1 -- 172.21.15.70:0/735313954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b130ff0 msgr2=0x55e47b13b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:52.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.211+0000 7fbc11b16640 1 --2- 172.21.15.70:0/735313954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b130ff0 0x55e47b13b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.211+0000 7fbc11b16640 1 -- 172.21.15.70:0/735313954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b12be80 msgr2=0x55e47b13a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:52.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.211+0000 7fbc11b16640 1 --2- 172.21.15.70:0/735313954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b12be80 0x55e47b13a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.211+0000 7fbc11b16640 1 -- 172.21.15.70:0/735313954 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e47b13bf30 con 0x55e47b12b490 2024-04-01T01:43:52.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.212+0000 7fbc11b16640 1 --2- 172.21.15.70:0/735313954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12b490 0x55e47b13a3f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c1033c0 tx=0x7fbc0801bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c77feec0222aa0a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:52.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.212+0000 7fbbf57fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:52.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.212+0000 7fbbf57fa640 1 -- 172.21.15.70:0/735313954 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc08004050 con 0x55e47b12b490 2024-04-01T01:43:52.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.212+0000 7fbbf57fa640 1 -- 172.21.15.70:0/735313954 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fbc080041f0 con 0x55e47b12b490 2024-04-01T01:43:52.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.212+0000 7fbc165fd740 1 -- 172.21.15.70:0/735313954 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbfc022070 con 0x55e47b12b490 2024-04-01T01:43:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.216+0000 7fbbf57fa640 1 -- 172.21.15.70:0/735313954 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbc080043f0 con 0x55e47b12b490 2024-04-01T01:43:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.216+0000 7fbc165fd740 10 client.28580 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.216+0000 7fbc165fd740 1 -- 172.21.15.70:0/735313954 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e47b1a7400 con 0x55e47b12b490 2024-04-01T01:43:52.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.216+0000 7fbbf57fa640 1 -- 172.21.15.70:0/735313954 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc0801f020 con 0x55e47b12b490 2024-04-01T01:43:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.216+0000 7fbbf57fa640 1 -- 172.21.15.70:0/735313954 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fbc080035f0 con 0x55e47b12b490 2024-04-01T01:43:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.216+0000 7fbc165fd740 10 client.28580 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.216+0000 7fbc165fd740 10 client.28580 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:52.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.217+0000 7fbc165fd740 20 client.28580 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:52.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.217+0000 7fbc165fd740 1 --2- 172.21.15.70:0/735313954 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e47b1411f0 0x55e47b1a9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:52.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.217+0000 7fbc165fd740 4 client.28580 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:43:52.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.217+0000 7fbc165fd740 1 -- 172.21.15.70:0/735313954 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fbbfc022070 con 0x55e47b1411f0 2024-04-01T01:43:52.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.218+0000 7fbc11315640 1 --2- 172.21.15.70:0/735313954 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e47b1411f0 0x55e47b1a9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:52.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.223+0000 7fbc11315640 1 --2- 172.21.15.70:0/735313954 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e47b1411f0 0x55e47b1a9cb0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6831419dd2d24325 server_cookie=51d740694e2ff58b in_seq=0 out_seq=0 2024-04-01T01:43:52.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.223+0000 7fbbf57fa640 10 client.28580 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:52.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.223+0000 7fbbf57fa640 1 -- 172.21.15.70:0/735313954 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fbbfc022070 con 0x55e47b1411f0 2024-04-01T01:43:52.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.223+0000 7fbbf57fa640 10 client.28580 handle_command_reply: tid=0 2024-04-01T01:43:52.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.224+0000 7fbc165fd740 1 client.28580 shutdown 2024-04-01T01:43:52.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.224+0000 7fbc165fd740 1 -- 172.21.15.70:0/735313954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12b490 msgr2=0x55e47b13a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:52.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.224+0000 7fbc165fd740 1 --2- 172.21.15.70:0/735313954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12b490 0x55e47b13a3f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c1033c0 tx=0x7fbc0801bee0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.224+0000 7fbc165fd740 1 -- 172.21.15.70:0/735313954 shutdown_connections 2024-04-01T01:43:52.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.224+0000 7fbc165fd740 1 --2- 172.21.15.70:0/735313954 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e47b1411f0 0x55e47b1a9cb0 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.224+0000 7fbc165fd740 1 --2- 172.21.15.70:0/735313954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e47b130ff0 0x55e47b13b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.224+0000 7fbc165fd740 1 --2- 172.21.15.70:0/735313954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e47b12be80 0x55e47b13a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.224+0000 7fbc165fd740 1 --2- 172.21.15.70:0/735313954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e47b12b490 0x55e47b13a3f0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.224+0000 7fbc165fd740 1 -- 172.21.15.70:0/735313954 >> 172.21.15.70:0/735313954 conn(0x55e47b126a00 msgr2=0x55e47b1254b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:52.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.224+0000 7fbbf57fa640 0 client.28580 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:52.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.224+0000 7fbc165fd740 1 -- 172.21.15.70:0/735313954 shutdown_connections 2024-04-01T01:43:52.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.224+0000 7fbc165fd740 1 -- 172.21.15.70:0/735313954 wait complete. 2024-04-01T01:43:52.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.225+0000 7fbc165fd740 20 client.28580 trim_cache size 0 max 16384 2024-04-01T01:43:52.225 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:43:52.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.229+0000 7fbc13d7a640 1 -- 172.21.15.70:0/3865884596 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbf80a26a0 msgr2=0x7fbbf80a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:52.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.229+0000 7fbc13d7a640 1 --2- 172.21.15.70:0/3865884596 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbf80a26a0 0x7fbbf80a4b60 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c112190 tx=0x7fbbfc008040 comp rx=0 tx=0).stop 2024-04-01T01:43:52.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.229+0000 7fbc13d7a640 1 -- 172.21.15.70:0/3865884596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc0c111580 msgr2=0x7fbc0c118450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:52.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.229+0000 7fbc13d7a640 1 --2- 172.21.15.70:0/3865884596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc0c111580 0x7fbc0c118450 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fbc04036040 tx=0x7fbc04033780 comp rx=0 tx=0).stop 2024-04-01T01:43:52.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.230+0000 7fbc13d7a640 1 -- 172.21.15.70:0/3865884596 shutdown_connections 2024-04-01T01:43:52.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.230+0000 7fbc13d7a640 1 --2- 172.21.15.70:0/3865884596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc0c111580 0x7fbc0c118450 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.230+0000 7fbc13d7a640 1 --2- 172.21.15.70:0/3865884596 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbbf80a26a0 0x7fbbf80a4b60 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.230+0000 7fbc13d7a640 1 --2- 172.21.15.70:0/3865884596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc0c105520 0x7fbc0c110ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.230+0000 7fbc13d7a640 1 --2- 172.21.15.70:0/3865884596 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc0c104b60 0x7fbc0c110ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:52.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.230+0000 7fbc13d7a640 1 -- 172.21.15.70:0/3865884596 >> 172.21.15.70:0/3865884596 conn(0x7fbc0c0feec0 msgr2=0x7fbc0c0ffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:52.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.233+0000 7fbc13d7a640 1 -- 172.21.15.70:0/3865884596 shutdown_connections 2024-04-01T01:43:52.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:52.233+0000 7fbc13d7a640 1 -- 172.21.15.70:0/3865884596 wait complete. 2024-04-01T01:43:52.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:51 smithi189 ceph-mon[35753]: pgmap v670: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.0 MiB/s wr, 791 op/s 2024-04-01T01:43:52.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:51 smithi081 ceph-mon[35142]: pgmap v670: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.0 MiB/s wr, 791 op/s 2024-04-01T01:43:53.255 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:43:53.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:43:53.255 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:43:53.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.459+0000 7f57f786d640 1 -- 172.21.15.70:0/1743259068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57f010da70 msgr2=0x7f57f010def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:53.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.459+0000 7f57f786d640 1 --2- 172.21.15.70:0/1743259068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57f010da70 0x7f57f010def0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f57e0009900 tx=0x7f57e002bc30 comp rx=0 tx=0).stop 2024-04-01T01:43:53.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.460+0000 7f57f786d640 1 -- 172.21.15.70:0/1743259068 shutdown_connections 2024-04-01T01:43:53.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.461+0000 7f57f786d640 1 --2- 172.21.15.70:0/1743259068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57f010da70 0x7f57f010def0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.461+0000 7f57f786d640 1 --2- 172.21.15.70:0/1743259068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f010b0c0 0x7f57f010d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.461+0000 7f57f786d640 1 --2- 172.21.15.70:0/1743259068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57f0108750 0x7f57f010ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.461+0000 7f57f786d640 1 -- 172.21.15.70:0/1743259068 >> 172.21.15.70:0/1743259068 conn(0x7f57f006e5b0 msgr2=0x7f57f00709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:53.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.461+0000 7f57f786d640 1 -- 172.21.15.70:0/1743259068 shutdown_connections 2024-04-01T01:43:53.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.462+0000 7f57f786d640 1 -- 172.21.15.70:0/1743259068 wait complete. 2024-04-01T01:43:53.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.463+0000 7f57f786d640 1 Processor -- start 2024-04-01T01:43:53.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.463+0000 7f57f786d640 1 -- start start 2024-04-01T01:43:53.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.463+0000 7f57f786d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57f0108750 0x7f57f01a6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.463+0000 7f57f786d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57f010b0c0 0x7f57f01a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.463+0000 7f57f786d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f010da70 0x7f57f01addb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.463+0000 7f57f786d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57f011c0d0 con 0x7f57f010da70 2024-04-01T01:43:53.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.464+0000 7f57f786d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f57f011bf50 con 0x7f57f010b0c0 2024-04-01T01:43:53.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.464+0000 7f57f5e0a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f010da70 0x7f57f01addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.464+0000 7f57f5e0a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f010da70 0x7f57f01addb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47744/0 (socket says 172.21.15.70:47744) 2024-04-01T01:43:53.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.464+0000 7f57f5e0a640 1 -- 172.21.15.70:0/979931331 learned_addr learned my addr 172.21.15.70:0/979931331 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:53.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.464+0000 7f57f786d640 1 -- 172.21.15.70:0/979931331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f57f011c250 con 0x7f57f0108750 2024-04-01T01:43:53.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.464+0000 7f57f5e0a640 1 -- 172.21.15.70:0/979931331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57f0108750 msgr2=0x7f57f01a6850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:43:53.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.464+0000 7f57f5e0a640 1 --2- 172.21.15.70:0/979931331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57f0108750 0x7f57f01a6850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.464+0000 7f57f5e0a640 1 -- 172.21.15.70:0/979931331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57f010b0c0 msgr2=0x7f57f01a6d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:43:53.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.464+0000 7f57f5e0a640 1 --2- 172.21.15.70:0/979931331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57f010b0c0 0x7f57f01a6d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.464+0000 7f57f5e0a640 1 -- 172.21.15.70:0/979931331 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f01ae420 con 0x7f57f010da70 2024-04-01T01:43:53.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.465+0000 7f57f5e0a640 1 --2- 172.21.15.70:0/979931331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f010da70 0x7f57f01addb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f57e0036040 tx=0x7f57e0004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:53.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.467+0000 7f57e67fc640 1 -- 172.21.15.70:0/979931331 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0043020 con 0x7f57f010da70 2024-04-01T01:43:53.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.467+0000 7f57e67fc640 1 -- 172.21.15.70:0/979931331 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57e00043e0 con 0x7f57f010da70 2024-04-01T01:43:53.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.467+0000 7f57e67fc640 1 -- 172.21.15.70:0/979931331 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0003b40 con 0x7f57f010da70 2024-04-01T01:43:53.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.468+0000 7f57f786d640 1 -- 172.21.15.70:0/979931331 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f01ae6b0 con 0x7f57f010da70 2024-04-01T01:43:53.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.468+0000 7f57f786d640 1 -- 172.21.15.70:0/979931331 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f01aeb70 con 0x7f57f010da70 2024-04-01T01:43:53.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.473+0000 7f57fa445740 1 Processor -- start 2024-04-01T01:43:53.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.474+0000 7f57e67fc640 1 -- 172.21.15.70:0/979931331 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f57e0046020 con 0x7f57f010da70 2024-04-01T01:43:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.475+0000 7f57fa445740 1 -- start start 2024-04-01T01:43:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.475+0000 7f57fa445740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e76a10 0x560294e76e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.475+0000 7f57fa445740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e77400 0x560294e7bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.475+0000 7f57fa445740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e7c570 0x560294e7ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.475+0000 7f57fa445740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560294d780b0 con 0x560294e76a10 2024-04-01T01:43:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.475+0000 7f57fa445740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560294ddaea0 con 0x560294e7c570 2024-04-01T01:43:53.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.475+0000 7f57fa445740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560294d83c00 con 0x560294e77400 2024-04-01T01:43:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.475+0000 7f57e67fc640 1 --2- 172.21.15.70:0/979931331 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57dc0a2620 0x7f57dc0a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.476+0000 7f57e67fc640 1 -- 172.21.15.70:0/979931331 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f57e00f0f70 con 0x7f57f010da70 2024-04-01T01:43:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.476+0000 7f57f4e08640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e7c570 0x560294e7ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.476+0000 7f57f4e08640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e7c570 0x560294e7ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41966/0 (socket says 172.21.15.70:41966) 2024-04-01T01:43:53.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.476+0000 7f57f4e08640 1 -- 172.21.15.70:0/1787381704 learned_addr learned my addr 172.21.15.70:0/1787381704 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:53.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.476+0000 7f57f5609640 1 --2- 172.21.15.70:0/1787381704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e77400 0x560294e7bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.476+0000 7f57f4e08640 1 --2- 172.21.15.70:0/1787381704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e76a10 0x560294e76e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.476+0000 7f57f5e0a640 1 --2- 172.21.15.70:0/979931331 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57dc0a2620 0x7f57dc0a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.477+0000 7f57f4e08640 1 -- 172.21.15.70:0/1787381704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e77400 msgr2=0x560294e7bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.477+0000 7f57f4e08640 1 --2- 172.21.15.70:0/1787381704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e77400 0x560294e7bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.477+0000 7f57f4e08640 1 -- 172.21.15.70:0/1787381704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e76a10 msgr2=0x560294e76e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.477+0000 7f57f4e08640 1 --2- 172.21.15.70:0/1787381704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e76a10 0x560294e76e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.477+0000 7f57f4e08640 1 -- 172.21.15.70:0/1787381704 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560294bb77e0 con 0x560294e7c570 2024-04-01T01:43:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.477+0000 7f57f4e08640 1 --2- 172.21.15.70:0/1787381704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e7c570 0x560294e7ea00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f57e800bfb0 tx=0x7f57e800b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23dba51e42fcbcc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.477+0000 7f57f5e0a640 1 --2- 172.21.15.70:0/979931331 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57dc0a2620 0x7f57dc0a4ae0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f57e00bd590 tx=0x7f57e00dd040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.477+0000 7f57db7fe640 1 -- 172.21.15.70:0/1787381704 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8004ba0 con 0x560294e7c570 2024-04-01T01:43:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.477+0000 7f57f5609640 1 --2- 172.21.15.70:0/1787381704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e77400 0x560294e7bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:43:53.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.477+0000 7f57db7fe640 1 -- 172.21.15.70:0/1787381704 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57e8003e20 con 0x560294e7c570 2024-04-01T01:43:53.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.477+0000 7f57db7fe640 1 -- 172.21.15.70:0/1787381704 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8004820 con 0x560294e7c570 2024-04-01T01:43:53.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.478+0000 7f57fa445740 1 -- 172.21.15.70:0/1787381704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e7c570 msgr2=0x560294e7ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:53.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.478+0000 7f57fa445740 1 --2- 172.21.15.70:0/1787381704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e7c570 0x560294e7ea00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f57e800bfb0 tx=0x7f57e800b6e0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.479+0000 7f57fa445740 1 -- 172.21.15.70:0/1787381704 shutdown_connections 2024-04-01T01:43:53.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.479+0000 7f57fa445740 1 --2- 172.21.15.70:0/1787381704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e7c570 0x560294e7ea00 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.479+0000 7f57fa445740 1 --2- 172.21.15.70:0/1787381704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e77400 0x560294e7bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.479+0000 7f57fa445740 1 --2- 172.21.15.70:0/1787381704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e76a10 0x560294e76e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.479+0000 7f57fa445740 1 -- 172.21.15.70:0/1787381704 >> 172.21.15.70:0/1787381704 conn(0x560294e71f80 msgr2=0x560294e72380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:53.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.479+0000 7f57fa445740 1 -- 172.21.15.70:0/1787381704 shutdown_connections 2024-04-01T01:43:53.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.480+0000 7f57fa445740 1 -- 172.21.15.70:0/1787381704 wait complete. 2024-04-01T01:43:53.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.480+0000 7f57fa445740 1 -- start start 2024-04-01T01:43:53.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.480+0000 7f57fa445740 1 Processor -- start 2024-04-01T01:43:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.481+0000 7f57fa445740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e76a10 0x560294e80de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.481+0000 7f57fa445740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 0x560294e81320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.481+0000 7f57fa445740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e7c570 0x560294e88360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.481+0000 7f57fa445740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560294d793d0 con 0x560294e77400 2024-04-01T01:43:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.481+0000 7f57fa445740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560294dd6f40 con 0x560294e76a10 2024-04-01T01:43:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.481+0000 7f57fa445740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560294e7f520 con 0x560294e7c570 2024-04-01T01:43:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.481+0000 7f57fa445740 10 client.0 _ll_register_callbacks cb 0x560294d96b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.481+0000 7f57f5609640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 0x560294e81320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.481+0000 7f57f5609640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 0x560294e81320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47770/0 (socket says 172.21.15.70:47770) 2024-04-01T01:43:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.481+0000 7f57f5609640 1 -- 172.21.15.70:0/5041600 learned_addr learned my addr 172.21.15.70:0/5041600 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.481+0000 7f57f4e08640 1 --2- 172.21.15.70:0/5041600 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e76a10 0x560294e80de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.481+0000 7f57f4e08640 1 --2- 172.21.15.70:0/5041600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e7c570 0x560294e88360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.482+0000 7f57f5609640 1 -- 172.21.15.70:0/5041600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e7c570 msgr2=0x560294e88360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.482+0000 7f57f5609640 1 --2- 172.21.15.70:0/5041600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e7c570 0x560294e88360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.482+0000 7f57f5609640 1 -- 172.21.15.70:0/5041600 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e76a10 msgr2=0x560294e80de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.482+0000 7f57f5609640 1 --2- 172.21.15.70:0/5041600 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e76a10 0x560294e80de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.482+0000 7f57f5609640 1 -- 172.21.15.70:0/5041600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560294c17f70 con 0x560294e77400 2024-04-01T01:43:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.482+0000 7f57f5609640 1 --2- 172.21.15.70:0/5041600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 0x560294e81320 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0105f0 tx=0x7f57ec010ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab59969c60317e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.482+0000 7f57d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:43:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.482+0000 7f57d8ff9640 1 -- 172.21.15.70:0/5041600 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ec005220 con 0x560294e77400 2024-04-01T01:43:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.482+0000 7f57fa445740 1 -- 172.21.15.70:0/5041600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57e8003e20 con 0x560294e77400 2024-04-01T01:43:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.482+0000 7f57d8ff9640 1 -- 172.21.15.70:0/5041600 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f57ec0053c0 con 0x560294e77400 2024-04-01T01:43:53.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.483+0000 7f57d8ff9640 1 -- 172.21.15.70:0/5041600 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f57ec023050 con 0x560294e77400 2024-04-01T01:43:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.483+0000 7f57fa445740 10 client.18573 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:53.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.483+0000 7f57fa445740 1 -- 172.21.15.70:0/5041600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560294aaf170 con 0x560294e77400 2024-04-01T01:43:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.483+0000 7f57d8ff9640 1 -- 172.21.15.70:0/5041600 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57e8003e20 con 0x560294e77400 2024-04-01T01:43:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.483+0000 7f57d8ff9640 1 -- 172.21.15.70:0/5041600 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f57ec00f040 con 0x560294e77400 2024-04-01T01:43:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.483+0000 7f57fa445740 10 client.18573 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.483+0000 7f57fa445740 10 client.18573 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:53.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.483+0000 7f57fa445740 20 client.18573 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:53.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.483+0000 7f57fa445740 1 --2- 172.21.15.70:0/5041600 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560294e9dca0 0x560294ea0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.483+0000 7f57fa445740 4 client.18573 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:53.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.483+0000 7f57fa445740 1 -- 172.21.15.70:0/5041600 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560294de83e0 con 0x560294e9dca0 2024-04-01T01:43:53.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.484+0000 7f57f4e08640 1 --2- 172.21.15.70:0/5041600 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560294e9dca0 0x560294ea0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.484+0000 7f57d8ff9640 10 client.18573 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:53.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.484+0000 7f57f4e08640 1 --2- 172.21.15.70:0/5041600 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560294e9dca0 0x560294ea0100 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34959b4541b42bd4 server_cookie=d96d35abbc0bf73c in_seq=0 out_seq=0 2024-04-01T01:43:53.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.485+0000 7f57d8ff9640 1 -- 172.21.15.70:0/5041600 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560294de83e0 con 0x560294e9dca0 2024-04-01T01:43:53.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.485+0000 7f57d8ff9640 10 client.18573 handle_command_reply: tid=0 2024-04-01T01:43:53.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.485+0000 7f57fa445740 1 client.18573 shutdown 2024-04-01T01:43:53.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.487+0000 7f57fa445740 1 -- 172.21.15.70:0/5041600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 msgr2=0x560294e81320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:53.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.487+0000 7f57fa445740 1 --2- 172.21.15.70:0/5041600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 0x560294e81320 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0105f0 tx=0x7f57ec010ab0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.487+0000 7f57fa445740 1 -- 172.21.15.70:0/5041600 shutdown_connections 2024-04-01T01:43:53.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.487+0000 7f57fa445740 1 --2- 172.21.15.70:0/5041600 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560294e9dca0 0x560294ea0100 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.487+0000 7f57fa445740 1 --2- 172.21.15.70:0/5041600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e7c570 0x560294e88360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.487+0000 7f57fa445740 1 --2- 172.21.15.70:0/5041600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 0x560294e81320 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.487+0000 7f57fa445740 1 --2- 172.21.15.70:0/5041600 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e76a10 0x560294e80de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.487+0000 7f57fa445740 1 -- 172.21.15.70:0/5041600 >> 172.21.15.70:0/5041600 conn(0x560294e71f80 msgr2=0x560294e71180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:53.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.487+0000 7f57d8ff9640 0 client.18573 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:53.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.487+0000 7f57fa445740 1 -- 172.21.15.70:0/5041600 shutdown_connections 2024-04-01T01:43:53.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.487+0000 7f57fa445740 1 -- 172.21.15.70:0/5041600 wait complete. 2024-04-01T01:43:53.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.487+0000 7f57fa445740 20 client.18573 trim_cache size 0 max 16384 2024-04-01T01:43:53.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.529+0000 7f57fa445740 1 Processor -- start 2024-04-01T01:43:53.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.529+0000 7f57fa445740 1 -- start start 2024-04-01T01:43:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.530+0000 7f57fa445740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e76a10 0x560294e758f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.530+0000 7f57fa445740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 0x560294e88320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.530+0000 7f57fa445740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e7c570 0x560294ea4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.530+0000 7f57f4e08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e76a10 0x560294e758f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.530+0000 7f57f4e08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e76a10 0x560294e758f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42652/0 (socket says 172.21.15.70:42652) 2024-04-01T01:43:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.530+0000 7f57f4e08640 1 -- 172.21.15.70:0/2751286982 learned_addr learned my addr 172.21.15.70:0/2751286982 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.530+0000 7f57fa445740 1 -- 172.21.15.70:0/2751286982 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57f01b0b00 con 0x560294e77400 2024-04-01T01:43:53.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.530+0000 7f57fa445740 1 -- 172.21.15.70:0/2751286982 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f57f01b2640 con 0x560294e7c570 2024-04-01T01:43:53.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.530+0000 7f57fa445740 1 -- 172.21.15.70:0/2751286982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f57f01b11d0 con 0x560294e76a10 2024-04-01T01:43:53.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.531+0000 7f57f5609640 1 --2- 172.21.15.70:0/2751286982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 0x560294e88320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.531+0000 7f57f4e08640 1 --2- 172.21.15.70:0/2751286982 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e7c570 0x560294ea4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.531+0000 7f57f4e08640 1 -- 172.21.15.70:0/2751286982 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e7c570 msgr2=0x560294ea4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:53.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.531+0000 7f57f4e08640 1 --2- 172.21.15.70:0/2751286982 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e7c570 0x560294ea4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.531+0000 7f57f4e08640 1 -- 172.21.15.70:0/2751286982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 msgr2=0x560294e88320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:53.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.531+0000 7f57f4e08640 1 --2- 172.21.15.70:0/2751286982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 0x560294e88320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.531+0000 7f57f4e08640 1 -- 172.21.15.70:0/2751286982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560294ea0780 con 0x560294e76a10 2024-04-01T01:43:53.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.531+0000 7f57f4e08640 1 --2- 172.21.15.70:0/2751286982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e76a10 0x560294e758f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f57e80091f0 tx=0x7f57e800bde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27a8a68843f8674a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:53.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.531+0000 7f57daffd640 1 -- 172.21.15.70:0/2751286982 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e8009040 con 0x560294e76a10 2024-04-01T01:43:53.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.532+0000 7f57daffd640 1 -- 172.21.15.70:0/2751286982 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57e8003e20 con 0x560294e76a10 2024-04-01T01:43:53.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.532+0000 7f57daffd640 1 -- 172.21.15.70:0/2751286982 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e802c040 con 0x560294e76a10 2024-04-01T01:43:53.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.533+0000 7f57fa445740 1 -- 172.21.15.70:0/2751286982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e76a10 msgr2=0x560294e758f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:53.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.533+0000 7f57fa445740 1 --2- 172.21.15.70:0/2751286982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e76a10 0x560294e758f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f57e80091f0 tx=0x7f57e800bde0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.533+0000 7f57fa445740 1 -- 172.21.15.70:0/2751286982 shutdown_connections 2024-04-01T01:43:53.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.533+0000 7f57fa445740 1 --2- 172.21.15.70:0/2751286982 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e7c570 0x560294ea4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.533+0000 7f57fa445740 1 --2- 172.21.15.70:0/2751286982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 0x560294e88320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.533+0000 7f57fa445740 1 --2- 172.21.15.70:0/2751286982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e76a10 0x560294e758f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.533+0000 7f57fa445740 1 -- 172.21.15.70:0/2751286982 >> 172.21.15.70:0/2751286982 conn(0x560294e71f80 msgr2=0x560294e751e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:53.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.533+0000 7f57fa445740 1 -- 172.21.15.70:0/2751286982 shutdown_connections 2024-04-01T01:43:53.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.533+0000 7f57fa445740 1 -- 172.21.15.70:0/2751286982 wait complete. 2024-04-01T01:43:53.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.534+0000 7f57fa445740 1 -- start start 2024-04-01T01:43:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.534+0000 7f57fa445740 1 Processor -- start 2024-04-01T01:43:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.534+0000 7f57fa445740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e76a10 0x560294e87f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.534+0000 7f57fa445740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 0x560294e86580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.534+0000 7f57f4e08640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e76a10 0x560294e87f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57f4e08640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e76a10 0x560294e87f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41990/0 (socket says 172.21.15.70:41990) 2024-04-01T01:43:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57f4e08640 1 -- 172.21.15.70:0/2568118430 learned_addr learned my addr 172.21.15.70:0/2568118430 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57fa445740 1 --2- 172.21.15.70:0/2568118430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e7c570 0x560294e86ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57fa445740 1 -- 172.21.15.70:0/2568118430 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57f01b1f70 con 0x560294e77400 2024-04-01T01:43:53.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57fa445740 1 -- 172.21.15.70:0/2568118430 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f57f01b18a0 con 0x560294e76a10 2024-04-01T01:43:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57fa445740 1 -- 172.21.15.70:0/2568118430 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f57f0118390 con 0x560294e7c570 2024-04-01T01:43:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57f5609640 1 --2- 172.21.15.70:0/2568118430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 0x560294e86580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57fa445740 10 client.0 _ll_register_callbacks cb 0x7f57ec009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57f4e08640 1 --2- 172.21.15.70:0/2568118430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e7c570 0x560294e86ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57f4e08640 1 -- 172.21.15.70:0/2568118430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e7c570 msgr2=0x560294e86ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57f4e08640 1 --2- 172.21.15.70:0/2568118430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e7c570 0x560294e86ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57f4e08640 1 -- 172.21.15.70:0/2568118430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 msgr2=0x560294e86580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57f4e08640 1 --2- 172.21.15.70:0/2568118430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 0x560294e86580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57f4e08640 1 -- 172.21.15.70:0/2568118430 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560294e81690 con 0x560294e76a10 2024-04-01T01:43:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57f4e08640 1 --2- 172.21.15.70:0/2568118430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e76a10 0x560294e87f20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f57e800b7c0 tx=0x7f57e8009000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=577c926e08f1fe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.535+0000 7f57d8ff9640 1 -- 172.21.15.70:0/2568118430 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e802c120 con 0x560294e76a10 2024-04-01T01:43:53.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.536+0000 7f57fa445740 1 -- 172.21.15.70:0/2568118430 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57e8003e20 con 0x560294e76a10 2024-04-01T01:43:53.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.536+0000 7f57d8ff9640 1 -- 172.21.15.70:0/2568118430 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f57e802c2c0 con 0x560294e76a10 2024-04-01T01:43:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.536+0000 7f57d8ff9640 1 -- 172.21.15.70:0/2568118430 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f57e802c4c0 con 0x560294e76a10 2024-04-01T01:43:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.536+0000 7f57fa445740 10 client.28595 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.536+0000 7f57fa445740 1 -- 172.21.15.70:0/2568118430 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560294e72cf0 con 0x560294e76a10 2024-04-01T01:43:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.536+0000 7f57d8ff9640 1 -- 172.21.15.70:0/2568118430 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57e8003e20 con 0x560294e76a10 2024-04-01T01:43:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.537+0000 7f57d8ff9640 1 -- 172.21.15.70:0/2568118430 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f57e8026040 con 0x560294e76a10 2024-04-01T01:43:53.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.537+0000 7f57fa445740 10 client.28595 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.537+0000 7f57fa445740 10 client.28595 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.537+0000 7f57fa445740 20 client.28595 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.537+0000 7f57fa445740 1 --2- 172.21.15.70:0/2568118430 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560294f05180 0x560294f075a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.537+0000 7f57fa445740 4 client.28595 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:43:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.537+0000 7f57fa445740 1 -- 172.21.15.70:0/2568118430 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560294de2730 con 0x560294f05180 2024-04-01T01:43:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.538+0000 7f57f5609640 1 --2- 172.21.15.70:0/2568118430 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560294f05180 0x560294f075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.538+0000 7f57f5609640 1 --2- 172.21.15.70:0/2568118430 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560294f05180 0x560294f075a0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7286aff486fab7e5 server_cookie=c49d0abe31e7af16 in_seq=0 out_seq=0 2024-04-01T01:43:53.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.538+0000 7f57d8ff9640 10 client.28595 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.539+0000 7f57d8ff9640 1 -- 172.21.15.70:0/2568118430 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560294de2730 con 0x560294f05180 2024-04-01T01:43:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.539+0000 7f57d8ff9640 10 client.28595 handle_command_reply: tid=0 2024-04-01T01:43:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.539+0000 7f57fa445740 1 client.28595 shutdown 2024-04-01T01:43:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.540+0000 7f57fa445740 1 -- 172.21.15.70:0/2568118430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e76a10 msgr2=0x560294e87f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.540+0000 7f57fa445740 1 --2- 172.21.15.70:0/2568118430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e76a10 0x560294e87f20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f57e800b7c0 tx=0x7f57e8009000 comp rx=0 tx=0).stop 2024-04-01T01:43:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.541+0000 7f57fa445740 1 -- 172.21.15.70:0/2568118430 shutdown_connections 2024-04-01T01:43:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.541+0000 7f57fa445740 1 --2- 172.21.15.70:0/2568118430 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560294f05180 0x560294f075a0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.541+0000 7f57fa445740 1 --2- 172.21.15.70:0/2568118430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560294e7c570 0x560294e86ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.541+0000 7f57fa445740 1 --2- 172.21.15.70:0/2568118430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560294e77400 0x560294e86580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.541+0000 7f57fa445740 1 --2- 172.21.15.70:0/2568118430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560294e76a10 0x560294e87f20 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.541+0000 7f57fa445740 1 -- 172.21.15.70:0/2568118430 >> 172.21.15.70:0/2568118430 conn(0x560294e71f80 msgr2=0x560294e75a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.542+0000 7f57d8ff9640 0 client.28595 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.543+0000 7f57fa445740 1 -- 172.21.15.70:0/2568118430 shutdown_connections 2024-04-01T01:43:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.543+0000 7f57fa445740 1 -- 172.21.15.70:0/2568118430 wait complete. 2024-04-01T01:43:53.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.543+0000 7f57fa445740 20 client.28595 trim_cache size 0 max 16384 2024-04-01T01:43:53.545 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:43:53.545 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:43:53.545 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "529aca45-eb38-4b96-b6a1-c4efa95c9a53", 2024-04-01T01:43:53.545 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:43:53.545 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:43:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.544+0000 7f57f786d640 1 -- 172.21.15.70:0/979931331 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57dc0a2620 msgr2=0x7f57dc0a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.544+0000 7f57f786d640 1 --2- 172.21.15.70:0/979931331 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57dc0a2620 0x7f57dc0a4ae0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f57e00bd590 tx=0x7f57e00dd040 comp rx=0 tx=0).stop 2024-04-01T01:43:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.545+0000 7f57f786d640 1 -- 172.21.15.70:0/979931331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f010da70 msgr2=0x7f57f01addb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.545+0000 7f57f786d640 1 --2- 172.21.15.70:0/979931331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f010da70 0x7f57f01addb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f57e0036040 tx=0x7f57e0004060 comp rx=0 tx=0).stop 2024-04-01T01:43:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.545+0000 7f57f786d640 1 -- 172.21.15.70:0/979931331 shutdown_connections 2024-04-01T01:43:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.545+0000 7f57f786d640 1 --2- 172.21.15.70:0/979931331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57f010da70 0x7f57f01addb0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.545+0000 7f57f786d640 1 --2- 172.21.15.70:0/979931331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57f010b0c0 0x7f57f01a6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.545+0000 7f57f786d640 1 --2- 172.21.15.70:0/979931331 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57dc0a2620 0x7f57dc0a4ae0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.545+0000 7f57f786d640 1 --2- 172.21.15.70:0/979931331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57f0108750 0x7f57f01a6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.545+0000 7f57f786d640 1 -- 172.21.15.70:0/979931331 >> 172.21.15.70:0/979931331 conn(0x7f57f006e5b0 msgr2=0x7f57f0109da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:53.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.545+0000 7f57f786d640 1 -- 172.21.15.70:0/979931331 shutdown_connections 2024-04-01T01:43:53.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.545+0000 7f57f786d640 1 -- 172.21.15.70:0/979931331 wait complete. 2024-04-01T01:43:53.565 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:43:53.565 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:43:53.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.752+0000 7f046e519640 1 -- 172.21.15.70:0/1580849608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f046810ffd0 msgr2=0x7f04681123c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:53.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.752+0000 7f046e519640 1 --2- 172.21.15.70:0/1580849608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f046810ffd0 0x7f04681123c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0450009860 tx=0x7f045002bb70 comp rx=0 tx=0).stop 2024-04-01T01:43:53.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.752+0000 7f046e519640 1 -- 172.21.15.70:0/1580849608 shutdown_connections 2024-04-01T01:43:53.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.752+0000 7f046e519640 1 --2- 172.21.15.70:0/1580849608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f046810ffd0 0x7f04681123c0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.752+0000 7f046e519640 1 --2- 172.21.15.70:0/1580849608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04681090b0 0x7f046810fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.752+0000 7f046e519640 1 --2- 172.21.15.70:0/1580849608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0468108750 0x7f0468108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.752+0000 7f046e519640 1 -- 172.21.15.70:0/1580849608 >> 172.21.15.70:0/1580849608 conn(0x7f046806e5b0 msgr2=0x7f04680709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:53.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.753+0000 7f046e519640 1 -- 172.21.15.70:0/1580849608 shutdown_connections 2024-04-01T01:43:53.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.753+0000 7f046e519640 1 -- 172.21.15.70:0/1580849608 wait complete. 2024-04-01T01:43:53.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.753+0000 7f046e519640 1 Processor -- start 2024-04-01T01:43:53.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.754+0000 7f046e519640 1 -- start start 2024-04-01T01:43:53.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.754+0000 7f046e519640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0468108750 0x7f04681a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.754+0000 7f046e519640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04681090b0 0x7f04681a6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.754+0000 7f046e519640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f046810ffd0 0x7f04681add00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.754+0000 7f046e519640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0468117cc0 con 0x7f0468108750 2024-04-01T01:43:53.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.754+0000 7f046e519640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0468117b40 con 0x7f046810ffd0 2024-04-01T01:43:53.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.754+0000 7f046e519640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0468117e40 con 0x7f04681090b0 2024-04-01T01:43:53.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.754+0000 7f0467fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0468108750 0x7f04681a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.754+0000 7f0467fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0468108750 0x7f04681a6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47812/0 (socket says 172.21.15.70:47812) 2024-04-01T01:43:53.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.754+0000 7f0467fff640 1 -- 172.21.15.70:0/2374031846 learned_addr learned my addr 172.21.15.70:0/2374031846 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:53.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.754+0000 7f046cab6640 1 --2- 172.21.15.70:0/2374031846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f046810ffd0 0x7f04681add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.755+0000 7f04677fe640 1 --2- 172.21.15.70:0/2374031846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04681090b0 0x7f04681a6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.755+0000 7f0467fff640 1 -- 172.21.15.70:0/2374031846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04681090b0 msgr2=0x7f04681a6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:53.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.755+0000 7f0467fff640 1 --2- 172.21.15.70:0/2374031846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04681090b0 0x7f04681a6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.755+0000 7f0467fff640 1 -- 172.21.15.70:0/2374031846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f046810ffd0 msgr2=0x7f04681add00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:53.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.755+0000 7f0467fff640 1 --2- 172.21.15.70:0/2374031846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f046810ffd0 0x7f04681add00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.755+0000 7f0467fff640 1 -- 172.21.15.70:0/2374031846 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04681ae370 con 0x7f0468108750 2024-04-01T01:43:53.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.756+0000 7f0467fff640 1 --2- 172.21.15.70:0/2374031846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0468108750 0x7f04681a6750 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f045800d950 tx=0x7f045800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:53.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.759+0000 7f04657fa640 1 -- 172.21.15.70:0/2374031846 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458017070 con 0x7f0468108750 2024-04-01T01:43:53.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.759+0000 7f04657fa640 1 -- 172.21.15.70:0/2374031846 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0458009c30 con 0x7f0468108750 2024-04-01T01:43:53.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.759+0000 7f04657fa640 1 -- 172.21.15.70:0/2374031846 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045800f620 con 0x7f0468108750 2024-04-01T01:43:53.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.759+0000 7f046e519640 1 -- 172.21.15.70:0/2374031846 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04681ae660 con 0x7f0468108750 2024-04-01T01:43:53.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.759+0000 7f046e519640 1 -- 172.21.15.70:0/2374031846 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04681aeb60 con 0x7f0468108750 2024-04-01T01:43:53.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.762+0000 7f04711d0740 1 Processor -- start 2024-04-01T01:43:53.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.762+0000 7f04711d0740 1 -- start start 2024-04-01T01:43:53.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.763+0000 7f04711d0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab32b0 0x55b5ffab36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.763+0000 7f04711d0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab3ca0 0x55b5ffab87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.763+0000 7f04711d0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab8e10 0x55b5ffabb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.763+0000 7f04711d0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b5ffa18180 con 0x55b5ffab3ca0 2024-04-01T01:43:53.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.763+0000 7f04711d0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b5ff9c0b70 con 0x55b5ffab32b0 2024-04-01T01:43:53.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.763+0000 7f04711d0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b5ffa14220 con 0x55b5ffab8e10 2024-04-01T01:43:53.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.764+0000 7f046cab6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab32b0 0x55b5ffab36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.764+0000 7f046cab6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab32b0 0x55b5ffab36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42020/0 (socket says 172.21.15.70:42020) 2024-04-01T01:43:53.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.764+0000 7f046cab6640 1 -- 172.21.15.70:0/1988129256 learned_addr learned my addr 172.21.15.70:0/1988129256 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:53.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.765+0000 7f046cab6640 1 -- 172.21.15.70:0/1988129256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab8e10 msgr2=0x55b5ffabb2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:43:53.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.765+0000 7f046cab6640 1 --2- 172.21.15.70:0/1988129256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab8e10 0x55b5ffabb2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.765+0000 7f046cab6640 1 -- 172.21.15.70:0/1988129256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab3ca0 msgr2=0x55b5ffab87a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:43:53.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.765+0000 7f04657fa640 1 -- 172.21.15.70:0/2374031846 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f04580109a0 con 0x7f0468108750 2024-04-01T01:43:53.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.766+0000 7f04657fa640 1 --2- 172.21.15.70:0/2374031846 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f04540a2670 0x7f04540a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.766+0000 7f04657fa640 1 -- 172.21.15.70:0/2374031846 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f04580ce750 con 0x7f0468108750 2024-04-01T01:43:53.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.766+0000 7f0467fff640 1 --2- 172.21.15.70:0/1988129256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab3ca0 0x55b5ffab87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.766+0000 7f046cab6640 1 --2- 172.21.15.70:0/1988129256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab3ca0 0x55b5ffab87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.767+0000 7f046cab6640 1 -- 172.21.15.70:0/1988129256 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5ff9d47c0 con 0x55b5ffab32b0 2024-04-01T01:43:53.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.767+0000 7f04677fe640 1 --2- 172.21.15.70:0/2374031846 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f04540a2670 0x7f04540a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.768+0000 7f04677fe640 1 --2- 172.21.15.70:0/2374031846 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f04540a2670 0x7f04540a4b30 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f0460005c40 tx=0x7f0460005bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:53.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.769+0000 7f046cab6640 1 --2- 172.21.15.70:0/1988129256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab32b0 0x55b5ffab36d0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0450009830 tx=0x7f0450009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a0b31d7c1c059fe server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:53.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.769+0000 7f045e7fc640 1 -- 172.21.15.70:0/1988129256 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0450042070 con 0x55b5ffab32b0 2024-04-01T01:43:53.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.769+0000 7f045e7fc640 1 -- 172.21.15.70:0/1988129256 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0450034500 con 0x55b5ffab32b0 2024-04-01T01:43:53.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.769+0000 7f045e7fc640 1 -- 172.21.15.70:0/1988129256 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0450039070 con 0x55b5ffab32b0 2024-04-01T01:43:53.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.771+0000 7f04711d0740 1 -- 172.21.15.70:0/1988129256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab32b0 msgr2=0x55b5ffab36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:53.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.771+0000 7f04711d0740 1 --2- 172.21.15.70:0/1988129256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab32b0 0x55b5ffab36d0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0450009830 tx=0x7f0450009510 comp rx=0 tx=0).stop 2024-04-01T01:43:53.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.772+0000 7f04711d0740 1 -- 172.21.15.70:0/1988129256 shutdown_connections 2024-04-01T01:43:53.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.772+0000 7f04711d0740 1 --2- 172.21.15.70:0/1988129256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab8e10 0x55b5ffabb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.772+0000 7f04711d0740 1 --2- 172.21.15.70:0/1988129256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab3ca0 0x55b5ffab87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.772+0000 7f04711d0740 1 --2- 172.21.15.70:0/1988129256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab32b0 0x55b5ffab36d0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.772+0000 7f04711d0740 1 -- 172.21.15.70:0/1988129256 >> 172.21.15.70:0/1988129256 conn(0x55b5ffaae820 msgr2=0x55b5ffaaec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:53.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.774+0000 7f04711d0740 1 -- 172.21.15.70:0/1988129256 shutdown_connections 2024-04-01T01:43:53.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.774+0000 7f04711d0740 1 -- 172.21.15.70:0/1988129256 wait complete. 2024-04-01T01:43:53.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.775+0000 7f04711d0740 1 -- start start 2024-04-01T01:43:53.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.775+0000 7f04711d0740 1 Processor -- start 2024-04-01T01:43:53.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.776+0000 7f04711d0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab32b0 0x55b5fface830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.776+0000 7f04711d0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab3ca0 0x55b5ffaced70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.776+0000 7f04711d0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab8e10 0x55b5ffad5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.776+0000 7f04711d0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b5ffaccd70 con 0x55b5ffab8e10 2024-04-01T01:43:53.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.776+0000 7f04711d0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b5ff9b64a0 con 0x55b5ffab3ca0 2024-04-01T01:43:53.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.776+0000 7f04711d0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b5ffaccef0 con 0x55b5ffab32b0 2024-04-01T01:43:53.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.776+0000 7f04711d0740 10 client.0 _ll_register_callbacks cb 0x55b5ff9b9d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:53.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.776+0000 7f0467fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab32b0 0x55b5fface830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.776+0000 7f0467fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab32b0 0x55b5fface830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42692/0 (socket says 172.21.15.70:42692) 2024-04-01T01:43:53.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.776+0000 7f0467fff640 1 -- 172.21.15.70:0/3875486690 learned_addr learned my addr 172.21.15.70:0/3875486690 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:53.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.776+0000 7f04677fe640 1 --2- 172.21.15.70:0/3875486690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab3ca0 0x55b5ffaced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.779+0000 7f04677fe640 1 -- 172.21.15.70:0/3875486690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab32b0 msgr2=0x55b5fface830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:53.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.779+0000 7f04677fe640 1 --2- 172.21.15.70:0/3875486690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab32b0 0x55b5fface830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.779+0000 7f04677fe640 1 -- 172.21.15.70:0/3875486690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab8e10 msgr2=0x55b5ffad5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:53.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.779+0000 7f04677fe640 1 --2- 172.21.15.70:0/3875486690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab8e10 0x55b5ffad5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.779+0000 7f04677fe640 1 -- 172.21.15.70:0/3875486690 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5ff7f67d0 con 0x55b5ffab3ca0 2024-04-01T01:43:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.779+0000 7f04677fe640 1 --2- 172.21.15.70:0/3875486690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab3ca0 0x55b5ffaced70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x55b5ffaaf790 tx=0x7f0460004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3390a0e5ea4c6a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.779+0000 7f04477fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.779+0000 7f04477fe640 1 -- 172.21.15.70:0/3875486690 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460023020 con 0x55b5ffab3ca0 2024-04-01T01:43:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.779+0000 7f04477fe640 1 -- 172.21.15.70:0/3875486690 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f046001f890 con 0x55b5ffab3ca0 2024-04-01T01:43:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.779+0000 7f04711d0740 1 -- 172.21.15.70:0/3875486690 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0450034500 con 0x55b5ffab3ca0 2024-04-01T01:43:53.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.780+0000 7f04477fe640 1 -- 172.21.15.70:0/3875486690 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f046001fa90 con 0x55b5ffab3ca0 2024-04-01T01:43:53.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.780+0000 7f04711d0740 10 client.28607 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:53.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.780+0000 7f04711d0740 1 -- 172.21.15.70:0/3875486690 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b5ff72c3f0 con 0x55b5ffab3ca0 2024-04-01T01:43:53.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.780+0000 7f04477fe640 1 -- 172.21.15.70:0/3875486690 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0450034500 con 0x55b5ffab3ca0 2024-04-01T01:43:53.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.781+0000 7f04477fe640 1 -- 172.21.15.70:0/3875486690 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f046000f040 con 0x55b5ffab3ca0 2024-04-01T01:43:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.782+0000 7f04711d0740 10 client.28607 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.782+0000 7f04711d0740 10 client.28607 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.782+0000 7f04711d0740 20 client.28607 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.782+0000 7f04711d0740 1 --2- 172.21.15.70:0/3875486690 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b5ffada680 0x55b5ffadcac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.782+0000 7f04711d0740 4 client.28607 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:43:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.782+0000 7f04711d0740 1 -- 172.21.15.70:0/3875486690 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5ffa256c0 con 0x55b5ffada680 2024-04-01T01:43:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.783+0000 7f0467fff640 1 --2- 172.21.15.70:0/3875486690 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b5ffada680 0x55b5ffadcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.784+0000 7f0467fff640 1 --2- 172.21.15.70:0/3875486690 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b5ffada680 0x55b5ffadcac0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=781442e7941990d5 server_cookie=1e83a7bbb0e4114c in_seq=0 out_seq=0 2024-04-01T01:43:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.784+0000 7f04477fe640 10 client.28607 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.785+0000 7f04477fe640 1 -- 172.21.15.70:0/3875486690 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b5ffa256c0 con 0x55b5ffada680 2024-04-01T01:43:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.785+0000 7f04477fe640 10 client.28607 handle_command_reply: tid=0 2024-04-01T01:43:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.785+0000 7f04711d0740 1 client.28607 shutdown 2024-04-01T01:43:53.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.785+0000 7f04711d0740 1 -- 172.21.15.70:0/3875486690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab3ca0 msgr2=0x55b5ffaced70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:53.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.785+0000 7f04711d0740 1 --2- 172.21.15.70:0/3875486690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab3ca0 0x55b5ffaced70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x55b5ffaaf790 tx=0x7f0460004a40 comp rx=0 tx=0).stop 2024-04-01T01:43:53.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.786+0000 7f04711d0740 1 -- 172.21.15.70:0/3875486690 shutdown_connections 2024-04-01T01:43:53.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.786+0000 7f04711d0740 1 --2- 172.21.15.70:0/3875486690 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b5ffada680 0x55b5ffadcac0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.786+0000 7f04711d0740 1 --2- 172.21.15.70:0/3875486690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab8e10 0x55b5ffad5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.786+0000 7f04711d0740 1 --2- 172.21.15.70:0/3875486690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab3ca0 0x55b5ffaced70 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.786+0000 7f04477fe640 0 client.28607 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:53.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.786+0000 7f04711d0740 1 --2- 172.21.15.70:0/3875486690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab32b0 0x55b5fface830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.786+0000 7f04711d0740 1 -- 172.21.15.70:0/3875486690 >> 172.21.15.70:0/3875486690 conn(0x55b5ffaae820 msgr2=0x55b5ffaadc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:53.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.786+0000 7f04711d0740 1 -- 172.21.15.70:0/3875486690 shutdown_connections 2024-04-01T01:43:53.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.786+0000 7f04711d0740 1 -- 172.21.15.70:0/3875486690 wait complete. 2024-04-01T01:43:53.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.786+0000 7f04711d0740 20 client.28607 trim_cache size 0 max 16384 2024-04-01T01:43:53.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.828+0000 7f04711d0740 1 Processor -- start 2024-04-01T01:43:53.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.828+0000 7f04711d0740 1 -- start start 2024-04-01T01:43:53.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.829+0000 7f04711d0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab32b0 0x55b5ffacd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.829+0000 7f04711d0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab3ca0 0x55b5ffad5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.829+0000 7f04711d0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab8e10 0x55b5ffacd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.829+0000 7f04711d0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f04681b0a20 con 0x55b5ffab32b0 2024-04-01T01:43:53.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.829+0000 7f04711d0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f04681b2560 con 0x55b5ffab8e10 2024-04-01T01:43:53.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.829+0000 7f04711d0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f04681b10f0 con 0x55b5ffab3ca0 2024-04-01T01:43:53.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.829+0000 7f0467fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab32b0 0x55b5ffacd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.829+0000 7f0467fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab32b0 0x55b5ffacd290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47838/0 (socket says 172.21.15.70:47838) 2024-04-01T01:43:53.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.829+0000 7f0467fff640 1 -- 172.21.15.70:0/3850861011 learned_addr learned my addr 172.21.15.70:0/3850861011 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:53.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.829+0000 7f04677fe640 1 --2- 172.21.15.70:0/3850861011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab3ca0 0x55b5ffad5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.829+0000 7f046cab6640 1 --2- 172.21.15.70:0/3850861011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab8e10 0x55b5ffacd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.829+0000 7f0467fff640 1 -- 172.21.15.70:0/3850861011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab3ca0 msgr2=0x55b5ffad5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.829+0000 7f0467fff640 1 --2- 172.21.15.70:0/3850861011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab3ca0 0x55b5ffad5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.830+0000 7f0467fff640 1 -- 172.21.15.70:0/3850861011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab8e10 msgr2=0x55b5ffacd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.830+0000 7f0467fff640 1 --2- 172.21.15.70:0/3850861011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab8e10 0x55b5ffacd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.830+0000 7f0467fff640 1 -- 172.21.15.70:0/3850861011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5ff6ec0f0 con 0x55b5ffab32b0 2024-04-01T01:43:53.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.830+0000 7f0467fff640 1 --2- 172.21.15.70:0/3850861011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab32b0 0x55b5ffacd290 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f0458000c00 tx=0x7f04580971b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d496679dea833839 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.830+0000 7f045dffb640 1 -- 172.21.15.70:0/3850861011 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458020040 con 0x55b5ffab32b0 2024-04-01T01:43:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.830+0000 7f045dffb640 1 -- 172.21.15.70:0/3850861011 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f04580a37f0 con 0x55b5ffab32b0 2024-04-01T01:43:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.830+0000 7f045dffb640 1 -- 172.21.15.70:0/3850861011 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458097450 con 0x55b5ffab32b0 2024-04-01T01:43:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.831+0000 7f04711d0740 1 -- 172.21.15.70:0/3850861011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab32b0 msgr2=0x55b5ffacd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.831+0000 7f04711d0740 1 --2- 172.21.15.70:0/3850861011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab32b0 0x55b5ffacd290 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f0458000c00 tx=0x7f04580971b0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.832+0000 7f04711d0740 1 -- 172.21.15.70:0/3850861011 shutdown_connections 2024-04-01T01:43:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.832+0000 7f04711d0740 1 --2- 172.21.15.70:0/3850861011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab8e10 0x55b5ffacd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.832+0000 7f04711d0740 1 --2- 172.21.15.70:0/3850861011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab3ca0 0x55b5ffad5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.832+0000 7f04711d0740 1 --2- 172.21.15.70:0/3850861011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab32b0 0x55b5ffacd290 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.832+0000 7f04711d0740 1 -- 172.21.15.70:0/3850861011 >> 172.21.15.70:0/3850861011 conn(0x55b5ffaae820 msgr2=0x55b5fface830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.832+0000 7f04711d0740 1 -- 172.21.15.70:0/3850861011 shutdown_connections 2024-04-01T01:43:53.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.832+0000 7f04711d0740 1 -- 172.21.15.70:0/3850861011 wait complete. 2024-04-01T01:43:53.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.833+0000 7f04711d0740 1 -- start start 2024-04-01T01:43:53.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.834+0000 7f04711d0740 1 Processor -- start 2024-04-01T01:43:53.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.834+0000 7f04711d0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab32b0 0x55b5ffad31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.834+0000 7f04711d0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab3ca0 0x55b5ffad3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.834+0000 7f04711d0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab8e10 0x55b5ffad4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.834+0000 7f04711d0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f04681b1e90 con 0x55b5ffab3ca0 2024-04-01T01:43:53.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.834+0000 7f04711d0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f04681b17c0 con 0x55b5ffab32b0 2024-04-01T01:43:53.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.834+0000 7f04711d0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f04681a5780 con 0x55b5ffab8e10 2024-04-01T01:43:53.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.834+0000 7f04711d0740 10 client.0 _ll_register_callbacks cb 0x7f0460016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:43:53.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.835+0000 7f0467fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab32b0 0x55b5ffad31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.835+0000 7f0467fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab32b0 0x55b5ffad31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42056/0 (socket says 172.21.15.70:42056) 2024-04-01T01:43:53.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.835+0000 7f0467fff640 1 -- 172.21.15.70:0/3459309606 learned_addr learned my addr 172.21.15.70:0/3459309606 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:43:53.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.835+0000 7f04677fe640 1 --2- 172.21.15.70:0/3459309606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab3ca0 0x55b5ffad3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.835+0000 7f046cab6640 1 --2- 172.21.15.70:0/3459309606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab8e10 0x55b5ffad4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.835+0000 7f0467fff640 1 -- 172.21.15.70:0/3459309606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab8e10 msgr2=0x55b5ffad4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.835+0000 7f0467fff640 1 --2- 172.21.15.70:0/3459309606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab8e10 0x55b5ffad4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.835+0000 7f0467fff640 1 -- 172.21.15.70:0/3459309606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab3ca0 msgr2=0x55b5ffad3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:43:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.835+0000 7f0467fff640 1 --2- 172.21.15.70:0/3459309606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab3ca0 0x55b5ffad3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.835+0000 7f0467fff640 1 -- 172.21.15.70:0/3459309606 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5ffad4d00 con 0x55b5ffab32b0 2024-04-01T01:43:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.836+0000 7f0467fff640 1 --2- 172.21.15.70:0/3459309606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab32b0 0x55b5ffad31c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f045809aae0 tx=0x7f045809ace0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a200e6c74a60d50 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:43:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.836+0000 7f04477fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:43:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.836+0000 7f04477fe640 1 -- 172.21.15.70:0/3459309606 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458020030 con 0x55b5ffab32b0 2024-04-01T01:43:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.836+0000 7f04477fe640 1 -- 172.21.15.70:0/3459309606 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f04580ce440 con 0x55b5ffab32b0 2024-04-01T01:43:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.836+0000 7f04711d0740 1 -- 172.21.15.70:0/3459309606 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04580a37f0 con 0x55b5ffab32b0 2024-04-01T01:43:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.836+0000 7f04477fe640 1 -- 172.21.15.70:0/3459309606 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0458097450 con 0x55b5ffab32b0 2024-04-01T01:43:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.836+0000 7f04711d0740 10 client.28613 fetch_fsmap learned FSMap version 31 2024-04-01T01:43:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.837+0000 7f04711d0740 1 -- 172.21.15.70:0/3459309606 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b5ffad6010 con 0x55b5ffab32b0 2024-04-01T01:43:53.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.837+0000 7f04477fe640 1 -- 172.21.15.70:0/3459309606 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f04580a37f0 con 0x55b5ffab32b0 2024-04-01T01:43:53.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.837+0000 7f04477fe640 1 -- 172.21.15.70:0/3459309606 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f0458010460 con 0x55b5ffab32b0 2024-04-01T01:43:53.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.837+0000 7f04711d0740 10 client.28613 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:43:53.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.837+0000 7f04711d0740 10 client.28613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:43:53.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.837+0000 7f04711d0740 20 client.28613 populate_metadata read hostname 'smithi070' 2024-04-01T01:43:53.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.837+0000 7f04711d0740 1 --2- 172.21.15.70:0/3459309606 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b5ffada680 0x55b5ffb430a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:43:53.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.837+0000 7f04711d0740 4 client.28613 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:43:53.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.837+0000 7f04711d0740 1 -- 172.21.15.70:0/3459309606 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b5ffa1fa10 con 0x55b5ffada680 2024-04-01T01:43:53.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.838+0000 7f04677fe640 1 --2- 172.21.15.70:0/3459309606 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b5ffada680 0x55b5ffb430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:43:53.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.839+0000 7f04677fe640 1 --2- 172.21.15.70:0/3459309606 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b5ffada680 0x55b5ffb430a0 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5fd50e7fba3870d server_cookie=4119dc020d0fe7f4 in_seq=0 out_seq=0 2024-04-01T01:43:53.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.839+0000 7f04477fe640 10 client.28613 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:53.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.843+0000 7f04477fe640 1 -- 172.21.15.70:0/3459309606 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55b5ffa1fa10 con 0x55b5ffada680 2024-04-01T01:43:53.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.843+0000 7f04477fe640 10 client.28613 handle_command_reply: tid=0 2024-04-01T01:43:53.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.843+0000 7f04711d0740 1 client.28613 shutdown 2024-04-01T01:43:53.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.844+0000 7f04711d0740 1 -- 172.21.15.70:0/3459309606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab32b0 msgr2=0x55b5ffad31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:53.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.844+0000 7f04711d0740 1 --2- 172.21.15.70:0/3459309606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab32b0 0x55b5ffad31c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f045809aae0 tx=0x7f045809ace0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.844+0000 7f04711d0740 1 -- 172.21.15.70:0/3459309606 shutdown_connections 2024-04-01T01:43:53.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.844+0000 7f04711d0740 1 --2- 172.21.15.70:0/3459309606 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55b5ffada680 0x55b5ffb430a0 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.844+0000 7f04711d0740 1 --2- 172.21.15.70:0/3459309606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5ffab8e10 0x55b5ffad4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.845+0000 7f04711d0740 1 --2- 172.21.15.70:0/3459309606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b5ffab3ca0 0x55b5ffad3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.845+0000 7f04711d0740 1 --2- 172.21.15.70:0/3459309606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b5ffab32b0 0x55b5ffad31c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.845+0000 7f04477fe640 0 client.28613 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:43:53.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.845+0000 7f04711d0740 1 -- 172.21.15.70:0/3459309606 >> 172.21.15.70:0/3459309606 conn(0x55b5ffaae820 msgr2=0x55b5ffacd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:53.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.845+0000 7f04711d0740 1 -- 172.21.15.70:0/3459309606 shutdown_connections 2024-04-01T01:43:53.847 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:43:53.847 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-01T01:43:53.847 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:43:53.847 INFO:teuthology.orchestra.run.smithi070.stdout: "529aca45-eb38-4b96-b6a1-c4efa95c9a53": { 2024-04-01T01:43:53.847 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:43:53.848 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "529aca45-eb38-4b96-b6a1-c4efa95c9a53", 2024-04-01T01:43:53.848 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:43:53.848 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:43:53.848 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:43:53.848 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:43:53.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.846+0000 7f04711d0740 1 -- 172.21.15.70:0/3459309606 wait complete. 2024-04-01T01:43:53.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.846+0000 7f04711d0740 20 client.28613 trim_cache size 0 max 16384 2024-04-01T01:43:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.848+0000 7f046e519640 1 -- 172.21.15.70:0/2374031846 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f04540a2670 msgr2=0x7f04540a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.848+0000 7f046e519640 1 --2- 172.21.15.70:0/2374031846 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f04540a2670 0x7f04540a4b30 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f0460005c40 tx=0x7f0460005bd0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.848+0000 7f046e519640 1 -- 172.21.15.70:0/2374031846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0468108750 msgr2=0x7f04681a6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:43:53.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.848+0000 7f046e519640 1 --2- 172.21.15.70:0/2374031846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0468108750 0x7f04681a6750 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f045800d950 tx=0x7f045800de10 comp rx=0 tx=0).stop 2024-04-01T01:43:53.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.851+0000 7f046e519640 1 -- 172.21.15.70:0/2374031846 shutdown_connections 2024-04-01T01:43:53.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.851+0000 7f046e519640 1 --2- 172.21.15.70:0/2374031846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f046810ffd0 0x7f04681add00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.851+0000 7f046e519640 1 --2- 172.21.15.70:0/2374031846 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f04540a2670 0x7f04540a4b30 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.851+0000 7f046e519640 1 --2- 172.21.15.70:0/2374031846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f04681090b0 0x7f04681a6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.851+0000 7f046e519640 1 --2- 172.21.15.70:0/2374031846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0468108750 0x7f04681a6750 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:43:53.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.851+0000 7f046e519640 1 -- 172.21.15.70:0/2374031846 >> 172.21.15.70:0/2374031846 conn(0x7f046806e5b0 msgr2=0x7f0468070290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:43:53.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.854+0000 7f046e519640 1 -- 172.21.15.70:0/2374031846 shutdown_connections 2024-04-01T01:43:53.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:43:53.858+0000 7f046e519640 1 -- 172.21.15.70:0/2374031846 wait complete. 2024-04-01T01:43:53.885 INFO:tasks.cephfs.filesystem:scrub status for tag:529aca45-eb38-4b96-b6a1-c4efa95c9a53 - {'path': '/', 'tag': '529aca45-eb38-4b96-b6a1-c4efa95c9a53', 'options': 'recursive,force'} 2024-04-01T01:43:54.253 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:53 smithi189 ceph-mon[35753]: pgmap v671: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.8 MiB/s wr, 889 op/s 2024-04-01T01:43:54.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:53 smithi070 ceph-mon[32179]: pgmap v671: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.8 MiB/s wr, 889 op/s 2024-04-01T01:43:54.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:53 smithi081 ceph-mon[35142]: pgmap v671: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.8 MiB/s wr, 889 op/s 2024-04-01T01:43:55.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:54 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:43:55.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:54 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:43:55.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:54 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:43:55.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:54 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:43:55.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:54 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:43:55.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:54 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:43:55.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:54 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:43:55.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:54 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:43:55.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:54 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:43:55.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:54 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:43:55.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:54 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:43:55.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:54 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:43:56.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:55 smithi189 ceph-mon[35753]: pgmap v672: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.6 MiB/s wr, 772 op/s 2024-04-01T01:43:56.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:55 smithi070 ceph-mon[32179]: pgmap v672: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.6 MiB/s wr, 772 op/s 2024-04-01T01:43:56.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:55 smithi081 ceph-mon[35142]: pgmap v672: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.6 MiB/s wr, 772 op/s 2024-04-01T01:43:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 550.0 s, 802.8 tps, lat 125.038 ms stddev 58.061 2024-04-01T01:43:57.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:56 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:43:57.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:56 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:43:57.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:56 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:43:58.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:57 smithi189 ceph-mon[35753]: pgmap v673: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 6.8 MiB/s wr, 857 op/s 2024-04-01T01:43:58.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:57 smithi070 ceph-mon[32179]: pgmap v673: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 6.8 MiB/s wr, 857 op/s 2024-04-01T01:43:58.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:57 smithi081 ceph-mon[35142]: pgmap v673: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 6.8 MiB/s wr, 857 op/s 2024-04-01T01:43:59.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:43:58 smithi189 ceph-mon[35753]: pgmap v674: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 7.1 MiB/s wr, 913 op/s 2024-04-01T01:43:59.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:43:58 smithi070 ceph-mon[32179]: pgmap v674: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 7.1 MiB/s wr, 913 op/s 2024-04-01T01:43:59.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:43:58 smithi081 ceph-mon[35142]: pgmap v674: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 7.1 MiB/s wr, 913 op/s 2024-04-01T01:44:00.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.376+0000 7ff7f8d02640 1 -- 172.21.15.70:0/2295197925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7f410da70 msgr2=0x7ff7f410def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.376+0000 7ff7f8d02640 1 --2- 172.21.15.70:0/2295197925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7f410da70 0x7ff7f410def0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009ca0 tx=0x7ff7ec02c120 comp rx=0 tx=0).stop 2024-04-01T01:44:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.377+0000 7ff7f8d02640 1 -- 172.21.15.70:0/2295197925 shutdown_connections 2024-04-01T01:44:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.377+0000 7ff7f8d02640 1 --2- 172.21.15.70:0/2295197925 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7f410da70 0x7ff7f410def0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.377+0000 7ff7f8d02640 1 --2- 172.21.15.70:0/2295197925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7f410b0c0 0x7ff7f410d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.377+0000 7ff7f8d02640 1 --2- 172.21.15.70:0/2295197925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7f4108750 0x7ff7f410ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.377+0000 7ff7f8d02640 1 -- 172.21.15.70:0/2295197925 >> 172.21.15.70:0/2295197925 conn(0x7ff7f406e790 msgr2=0x7ff7f4070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.377+0000 7ff7f8d02640 1 -- 172.21.15.70:0/2295197925 shutdown_connections 2024-04-01T01:44:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.377+0000 7ff7f8d02640 1 -- 172.21.15.70:0/2295197925 wait complete. 2024-04-01T01:44:00.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.378+0000 7ff7f8d02640 1 Processor -- start 2024-04-01T01:44:00.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.382+0000 7ff7f8d02640 1 -- start start 2024-04-01T01:44:00.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.382+0000 7ff7f8d02640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7f4108750 0x7ff7f41a66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.382+0000 7ff7f8d02640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7f410b0c0 0x7ff7f41a6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.382+0000 7ff7f8d02640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7f410da70 0x7ff7f41adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.382+0000 7ff7f8d02640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f411c0a0 con 0x7ff7f4108750 2024-04-01T01:44:00.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.382+0000 7ff7f8d02640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f411bf20 con 0x7ff7f410da70 2024-04-01T01:44:00.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.382+0000 7ff7f8d02640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f411c220 con 0x7ff7f410b0c0 2024-04-01T01:44:00.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.382+0000 7ff7f2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7f410da70 0x7ff7f41adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:00.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.382+0000 7ff7f2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7f410da70 0x7ff7f41adc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57758/0 (socket says 172.21.15.70:57758) 2024-04-01T01:44:00.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.382+0000 7ff7f2d9d640 1 -- 172.21.15.70:0/664298663 learned_addr learned my addr 172.21.15.70:0/664298663 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:00.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.383+0000 7ff7f2d9d640 1 -- 172.21.15.70:0/664298663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7f410b0c0 msgr2=0x7ff7f41a6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:00.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.383+0000 7ff7f2d9d640 1 --2- 172.21.15.70:0/664298663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7f410b0c0 0x7ff7f41a6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.383+0000 7ff7f2d9d640 1 -- 172.21.15.70:0/664298663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7f4108750 msgr2=0x7ff7f41a66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:00.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.383+0000 7ff7f2d9d640 1 --2- 172.21.15.70:0/664298663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7f4108750 0x7ff7f41a66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.383+0000 7ff7f2d9d640 1 -- 172.21.15.70:0/664298663 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f41ae270 con 0x7ff7f410da70 2024-04-01T01:44:00.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.383+0000 7ff7f2d9d640 1 --2- 172.21.15.70:0/664298663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7f410da70 0x7ff7f41adc00 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec02c660 tx=0x7ff7ec035ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:00.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.383+0000 7ff7e37fe640 1 -- 172.21.15.70:0/664298663 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec039040 con 0x7ff7f410da70 2024-04-01T01:44:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.383+0000 7ff7e37fe640 1 -- 172.21.15.70:0/664298663 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7ec044070 con 0x7ff7f410da70 2024-04-01T01:44:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.383+0000 7ff7e37fe640 1 -- 172.21.15.70:0/664298663 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec0043c0 con 0x7ff7f410da70 2024-04-01T01:44:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.384+0000 7ff7f8d02640 1 -- 172.21.15.70:0/664298663 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f41ae500 con 0x7ff7f410da70 2024-04-01T01:44:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.384+0000 7ff7f8d02640 1 -- 172.21.15.70:0/664298663 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f41ae9a0 con 0x7ff7f410da70 2024-04-01T01:44:00.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.388+0000 7ff7fb9c3740 1 Processor -- start 2024-04-01T01:44:00.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.389+0000 7ff7e37fe640 1 -- 172.21.15.70:0/664298663 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7ff7ec004560 con 0x7ff7f410da70 2024-04-01T01:44:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.389+0000 7ff7fb9c3740 1 -- start start 2024-04-01T01:44:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.389+0000 7ff7fb9c3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f38d0 0x55e05a3f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.389+0000 7ff7fb9c3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f42c0 0x55e05a3f8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.389+0000 7ff7fb9c3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f9430 0x55e05a3fb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.389+0000 7ff7fb9c3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e05a3578f0 con 0x55e05a3f38d0 2024-04-01T01:44:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.389+0000 7ff7fb9c3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e05a3008d0 con 0x55e05a3f9430 2024-04-01T01:44:00.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.389+0000 7ff7fb9c3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e05a353990 con 0x55e05a3f42c0 2024-04-01T01:44:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.390+0000 7ff7f1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f38d0 0x55e05a3f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.390+0000 7ff7f1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f38d0 0x55e05a3f3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58544/0 (socket says 172.21.15.70:58544) 2024-04-01T01:44:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.390+0000 7ff7f1d9b640 1 -- 172.21.15.70:0/2079372266 learned_addr learned my addr 172.21.15.70:0/2079372266 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.390+0000 7ff7f259c640 1 --2- 172.21.15.70:0/2079372266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f42c0 0x55e05a3f8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.390+0000 7ff7f1d9b640 1 --2- 172.21.15.70:0/2079372266 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f9430 0x55e05a3fb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.390+0000 7ff7e37fe640 1 --2- 172.21.15.70:0/664298663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7dc0a26b0 0x7ff7dc0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.391+0000 7ff7e37fe640 1 -- 172.21.15.70:0/664298663 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff7ec0f12a0 con 0x7ff7f410da70 2024-04-01T01:44:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.391+0000 7ff7f1d9b640 1 -- 172.21.15.70:0/2079372266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f42c0 msgr2=0x55e05a3f8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.391+0000 7ff7f1d9b640 1 --2- 172.21.15.70:0/2079372266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f42c0 0x55e05a3f8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.391+0000 7ff7f1d9b640 1 -- 172.21.15.70:0/2079372266 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f9430 msgr2=0x55e05a3fb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.391+0000 7ff7f1d9b640 1 --2- 172.21.15.70:0/2079372266 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f9430 0x55e05a3fb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.391+0000 7ff7f1d9b640 1 -- 172.21.15.70:0/2079372266 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e05a06c210 con 0x55e05a3f38d0 2024-04-01T01:44:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.391+0000 7ff7f2d9d640 1 --2- 172.21.15.70:0/664298663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7dc0a26b0 0x7ff7dc0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.391+0000 7ff7f1d9b640 1 --2- 172.21.15.70:0/2079372266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f38d0 0x55e05a3f3cf0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80137e0 tx=0x7ff7e8009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68192cd9793bf665 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.392+0000 7ff7f2d9d640 1 --2- 172.21.15.70:0/664298663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7dc0a26b0 0x7ff7dc0a4b70 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec033000 tx=0x7ff7ec02c120 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.392+0000 7ff7e0ff9640 1 -- 172.21.15.70:0/2079372266 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e80132d0 con 0x55e05a3f38d0 2024-04-01T01:44:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.392+0000 7ff7e0ff9640 1 -- 172.21.15.70:0/2079372266 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7e8013e20 con 0x55e05a3f38d0 2024-04-01T01:44:00.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.393+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/2079372266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f38d0 msgr2=0x55e05a3f3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.393+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/2079372266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f38d0 0x55e05a3f3cf0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80137e0 tx=0x7ff7e8009650 comp rx=0 tx=0).stop 2024-04-01T01:44:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.395+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/2079372266 shutdown_connections 2024-04-01T01:44:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.395+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/2079372266 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f9430 0x55e05a3fb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.395+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/2079372266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f42c0 0x55e05a3f8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.395+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/2079372266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f38d0 0x55e05a3f3cf0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.396+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/2079372266 >> 172.21.15.70:0/2079372266 conn(0x55e05a3eee40 msgr2=0x55e05a3ef240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:00.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.396+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/2079372266 shutdown_connections 2024-04-01T01:44:00.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.396+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/2079372266 wait complete. 2024-04-01T01:44:00.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.397+0000 7ff7fb9c3740 1 -- start start 2024-04-01T01:44:00.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.397+0000 7ff7fb9c3740 1 Processor -- start 2024-04-01T01:44:00.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.398+0000 7ff7fb9c3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f38d0 0x55e05a40eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.398+0000 7ff7fb9c3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f42c0 0x55e05a40f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.398+0000 7ff7fb9c3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f9430 0x55e05a416100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.398+0000 7ff7f1d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f38d0 0x55e05a40eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:00.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.398+0000 7ff7f1d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f38d0 0x55e05a40eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45770/0 (socket says 172.21.15.70:45770) 2024-04-01T01:44:00.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.398+0000 7ff7f1d9b640 1 -- 172.21.15.70:0/1080475684 learned_addr learned my addr 172.21.15.70:0/1080475684 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:00.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.398+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/1080475684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e05a40d430 con 0x55e05a3f9430 2024-04-01T01:44:00.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.398+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/1080475684 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e05a2f5f80 con 0x55e05a3f42c0 2024-04-01T01:44:00.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.398+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/1080475684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e05a40d5b0 con 0x55e05a3f38d0 2024-04-01T01:44:00.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.398+0000 7ff7fb9c3740 10 client.0 _ll_register_callbacks cb 0x55e05a34e290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:00.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.399+0000 7ff7f1d9b640 1 -- 172.21.15.70:0/1080475684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f42c0 msgr2=0x55e05a40f0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:44:00.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.399+0000 7ff7f1d9b640 1 --2- 172.21.15.70:0/1080475684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f42c0 0x55e05a40f0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.399+0000 7ff7f1d9b640 1 -- 172.21.15.70:0/1080475684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f9430 msgr2=0x55e05a416100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:00.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.399+0000 7ff7f1d9b640 1 --2- 172.21.15.70:0/1080475684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f9430 0x55e05a416100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.399+0000 7ff7f1d9b640 1 -- 172.21.15.70:0/1080475684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e05a138620 con 0x55e05a3f38d0 2024-04-01T01:44:00.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.399+0000 7ff7f1d9b640 1 --2- 172.21.15.70:0/1080475684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f38d0 0x55e05a40eb80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800e9b0 tx=0x7ff7e800e9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77d178c8535d469b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:00.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.400+0000 7ff7d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:44:00.400 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.400+0000 7ff7d9ffb640 1 -- 172.21.15.70:0/1080475684 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8029450 con 0x55e05a3f38d0 2024-04-01T01:44:00.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.400+0000 7ff7d9ffb640 1 -- 172.21.15.70:0/1080475684 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff7e80295f0 con 0x55e05a3f38d0 2024-04-01T01:44:00.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.400+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/1080475684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7e8013e20 con 0x55e05a3f38d0 2024-04-01T01:44:00.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.400+0000 7ff7d9ffb640 1 -- 172.21.15.70:0/1080475684 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff7e80297f0 con 0x55e05a3f38d0 2024-04-01T01:44:00.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.400+0000 7ff7fb9c3740 10 client.28414 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:00.401 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.401+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/1080475684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e05a4190a0 con 0x55e05a3f38d0 2024-04-01T01:44:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.405+0000 7ff7d9ffb640 1 -- 172.21.15.70:0/1080475684 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7e8013e20 con 0x55e05a3f38d0 2024-04-01T01:44:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.405+0000 7ff7d9ffb640 1 -- 172.21.15.70:0/1080475684 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff7e8033420 con 0x55e05a3f38d0 2024-04-01T01:44:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.405+0000 7ff7fb9c3740 10 client.28414 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.405+0000 7ff7fb9c3740 10 client.28414 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.405+0000 7ff7fb9c3740 20 client.28414 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.405+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/1080475684 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e05a41a4a0 0x55e05a41c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.405+0000 7ff7fb9c3740 4 client.28414 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:00.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.405+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/1080475684 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e05a36b120 con 0x55e05a41a4a0 2024-04-01T01:44:00.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.406+0000 7ff7f259c640 1 --2- 172.21.15.70:0/1080475684 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e05a41a4a0 0x55e05a41c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:00.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.407+0000 7ff7f259c640 1 --2- 172.21.15.70:0/1080475684 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e05a41a4a0 0x55e05a41c890 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fd23b1fc708e6bb server_cookie=46caff46d7502d17 in_seq=0 out_seq=0 2024-04-01T01:44:00.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.407+0000 7ff7d9ffb640 10 client.28414 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:00.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.408+0000 7ff7d9ffb640 1 -- 172.21.15.70:0/1080475684 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e05a36b120 con 0x55e05a41a4a0 2024-04-01T01:44:00.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.408+0000 7ff7d9ffb640 10 client.28414 handle_command_reply: tid=0 2024-04-01T01:44:00.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.408+0000 7ff7fb9c3740 1 client.28414 shutdown 2024-04-01T01:44:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.408+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/1080475684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f38d0 msgr2=0x55e05a40eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.408+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/1080475684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f38d0 0x55e05a40eb80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800e9b0 tx=0x7ff7e800e9e0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.409+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/1080475684 shutdown_connections 2024-04-01T01:44:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.409+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/1080475684 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e05a41a4a0 0x55e05a41c890 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.409+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/1080475684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f9430 0x55e05a416100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.409+0000 7ff7d9ffb640 0 client.28414 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.409+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/1080475684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f42c0 0x55e05a40f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.409+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/1080475684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f38d0 0x55e05a40eb80 secure :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800e9b0 tx=0x7ff7e800e9e0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.409+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/1080475684 >> 172.21.15.70:0/1080475684 conn(0x55e05a3eee40 msgr2=0x55e05a3edf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.410+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/1080475684 shutdown_connections 2024-04-01T01:44:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.413+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/1080475684 wait complete. 2024-04-01T01:44:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.413+0000 7ff7fb9c3740 20 client.28414 trim_cache size 0 max 16384 2024-04-01T01:44:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.445+0000 7ff7fb9c3740 1 Processor -- start 2024-04-01T01:44:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.446+0000 7ff7fb9c3740 1 -- start start 2024-04-01T01:44:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.446+0000 7ff7fb9c3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f42c0 0x55e05a40f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.446+0000 7ff7fb9c3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f9430 0x55e05a416090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.446+0000 7ff7fb9c3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a41a4a0 0x55e05a3ef220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.447+0000 7ff7f1d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f42c0 0x55e05a40f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.447+0000 7ff7fb9c3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f41b0950 con 0x55e05a41a4a0 2024-04-01T01:44:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.447+0000 7ff7fb9c3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f41b2490 con 0x55e05a3f9430 2024-04-01T01:44:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.447+0000 7ff7fb9c3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f41b1020 con 0x55e05a3f42c0 2024-04-01T01:44:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.447+0000 7ff7f1d9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f42c0 0x55e05a40f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:45784/0 (socket says 172.21.15.70:45784) 2024-04-01T01:44:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.447+0000 7ff7f1d9b640 1 -- 172.21.15.70:0/567355240 learned_addr learned my addr 172.21.15.70:0/567355240 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.447+0000 7ff7f259c640 1 --2- 172.21.15.70:0/567355240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f9430 0x55e05a416090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.447+0000 7ff7f1d9b640 1 --2- 172.21.15.70:0/567355240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a41a4a0 0x55e05a3ef220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.447+0000 7ff7f1d9b640 1 -- 172.21.15.70:0/567355240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f9430 msgr2=0x55e05a416090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.447+0000 7ff7f1d9b640 1 --2- 172.21.15.70:0/567355240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f9430 0x55e05a416090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.447+0000 7ff7f1d9b640 1 -- 172.21.15.70:0/567355240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a41a4a0 msgr2=0x55e05a3ef220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.447+0000 7ff7f1d9b640 1 --2- 172.21.15.70:0/567355240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a41a4a0 0x55e05a3ef220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.447+0000 7ff7f1d9b640 1 -- 172.21.15.70:0/567355240 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e05a41cc00 con 0x55e05a3f42c0 2024-04-01T01:44:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.448+0000 7ff7f1d9b640 1 --2- 172.21.15.70:0/567355240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f42c0 0x55e05a40f020 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800a010 tx=0x7ff7e8016040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba646c8844940557 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.448+0000 7ff7dbfff640 1 -- 172.21.15.70:0/567355240 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8013e20 con 0x55e05a3f42c0 2024-04-01T01:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.448+0000 7ff7dbfff640 1 -- 172.21.15.70:0/567355240 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7e80132e0 con 0x55e05a3f42c0 2024-04-01T01:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.449+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/567355240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f42c0 msgr2=0x55e05a40f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.449+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/567355240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f42c0 0x55e05a40f020 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800a010 tx=0x7ff7e8016040 comp rx=0 tx=0).stop 2024-04-01T01:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.449+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/567355240 shutdown_connections 2024-04-01T01:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.449+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/567355240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a41a4a0 0x55e05a3ef220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.449+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/567355240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f9430 0x55e05a416090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.449+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/567355240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a3f42c0 0x55e05a40f020 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.449+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/567355240 >> 172.21.15.70:0/567355240 conn(0x55e05a3eee40 msgr2=0x55e05a3e9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.450+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/567355240 shutdown_connections 2024-04-01T01:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.450+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/567355240 wait complete. 2024-04-01T01:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.450+0000 7ff7fb9c3740 1 -- start start 2024-04-01T01:44:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.450+0000 7ff7fb9c3740 1 Processor -- start 2024-04-01T01:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.450+0000 7ff7fb9c3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f42c0 0x55e05a413980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.450+0000 7ff7fb9c3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f9430 0x55e05a413ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.450+0000 7ff7fb9c3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a41a4a0 0x55e05a414ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.450+0000 7ff7fb9c3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f41b1dc0 con 0x55e05a3f9430 2024-04-01T01:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.450+0000 7ff7fb9c3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f41b16f0 con 0x55e05a3f42c0 2024-04-01T01:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.450+0000 7ff7fb9c3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f4118300 con 0x55e05a41a4a0 2024-04-01T01:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.450+0000 7ff7fb9c3740 10 client.0 _ll_register_callbacks cb 0x7ff7e8009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.451+0000 7ff7f259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f9430 0x55e05a413ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.451+0000 7ff7f259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f9430 0x55e05a413ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58570/0 (socket says 172.21.15.70:58570) 2024-04-01T01:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.451+0000 7ff7f259c640 1 -- 172.21.15.70:0/2402588613 learned_addr learned my addr 172.21.15.70:0/2402588613 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.451+0000 7ff7f1d9b640 1 --2- 172.21.15.70:0/2402588613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f42c0 0x55e05a413980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.451+0000 7ff7f1d9b640 1 --2- 172.21.15.70:0/2402588613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a41a4a0 0x55e05a414ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.451+0000 7ff7f1d9b640 1 -- 172.21.15.70:0/2402588613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a41a4a0 msgr2=0x55e05a414ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.451+0000 7ff7f1d9b640 1 --2- 172.21.15.70:0/2402588613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a41a4a0 0x55e05a414ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.451+0000 7ff7f1d9b640 1 -- 172.21.15.70:0/2402588613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f9430 msgr2=0x55e05a413ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.451+0000 7ff7f1d9b640 1 --2- 172.21.15.70:0/2402588613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f9430 0x55e05a413ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.451+0000 7ff7f1d9b640 1 -- 172.21.15.70:0/2402588613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e05a4154c0 con 0x55e05a3f42c0 2024-04-01T01:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.452+0000 7ff7f259c640 1 --2- 172.21.15.70:0/2402588613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f9430 0x55e05a413ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.452+0000 7ff7f1d9b640 1 --2- 172.21.15.70:0/2402588613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f42c0 0x55e05a413980 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800e720 tx=0x7ff7e800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33314cac067ce587 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.452+0000 7ff7d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:44:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.452+0000 7ff7d9ffb640 1 -- 172.21.15.70:0/2402588613 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8013e20 con 0x55e05a3f42c0 2024-04-01T01:44:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.452+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/2402588613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7e80132e0 con 0x55e05a3f42c0 2024-04-01T01:44:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.452+0000 7ff7d9ffb640 1 -- 172.21.15.70:0/2402588613 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff7e8029da0 con 0x55e05a3f42c0 2024-04-01T01:44:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.453+0000 7ff7d9ffb640 1 -- 172.21.15.70:0/2402588613 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff7e8071050 con 0x55e05a3f42c0 2024-04-01T01:44:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.453+0000 7ff7fb9c3740 10 client.28628 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.453+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/2402588613 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e05a402150 con 0x55e05a3f42c0 2024-04-01T01:44:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.453+0000 7ff7d9ffb640 1 -- 172.21.15.70:0/2402588613 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff7e80132e0 con 0x55e05a3f42c0 2024-04-01T01:44:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.453+0000 7ff7d9ffb640 1 -- 172.21.15.70:0/2402588613 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff7e8011af0 con 0x55e05a3f42c0 2024-04-01T01:44:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.454+0000 7ff7fb9c3740 10 client.28628 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.454+0000 7ff7fb9c3740 10 client.28628 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.454+0000 7ff7fb9c3740 20 client.28628 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.454+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/2402588613 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e05a402b10 0x55e05a404f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.454+0000 7ff7fb9c3740 4 client.28628 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:44:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.454+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/2402588613 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e05a364e30 con 0x55e05a402b10 2024-04-01T01:44:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.455+0000 7ff7f259c640 1 --2- 172.21.15.70:0/2402588613 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e05a402b10 0x55e05a404f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:00.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.458+0000 7ff7f259c640 1 --2- 172.21.15.70:0/2402588613 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e05a402b10 0x55e05a404f30 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55c7e25f149f6b0c server_cookie=87e86fb8ba5e0bc8 in_seq=0 out_seq=0 2024-04-01T01:44:00.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.458+0000 7ff7d9ffb640 10 client.28628 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.519+0000 7ff7d9ffb640 1 -- 172.21.15.70:0/2402588613 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e05a364e30 con 0x55e05a402b10 2024-04-01T01:44:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.519+0000 7ff7d9ffb640 10 client.28628 handle_command_reply: tid=0 2024-04-01T01:44:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.519+0000 7ff7fb9c3740 1 client.28628 shutdown 2024-04-01T01:44:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.520+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/2402588613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f42c0 msgr2=0x55e05a413980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.520+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/2402588613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f42c0 0x55e05a413980 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800e720 tx=0x7ff7e800ef10 comp rx=0 tx=0).stop 2024-04-01T01:44:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.520+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/2402588613 shutdown_connections 2024-04-01T01:44:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.520+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/2402588613 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e05a402b10 0x55e05a404f30 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.520+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/2402588613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e05a41a4a0 0x55e05a414ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.520+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/2402588613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e05a3f9430 0x55e05a413ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.520+0000 7ff7fb9c3740 1 --2- 172.21.15.70:0/2402588613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e05a3f42c0 0x55e05a413980 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.520+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/2402588613 >> 172.21.15.70:0/2402588613 conn(0x55e05a3eee40 msgr2=0x55e05a3ef220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.520+0000 7ff7d9ffb640 0 client.28628 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.521+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/2402588613 shutdown_connections 2024-04-01T01:44:00.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.521+0000 7ff7fb9c3740 1 -- 172.21.15.70:0/2402588613 wait complete. 2024-04-01T01:44:00.522 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.521+0000 7ff7fb9c3740 20 client.28628 trim_cache size 0 max 16384 2024-04-01T01:44:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:44:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:44:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:44:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:44:00.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.526+0000 7ff7f8d02640 1 -- 172.21.15.70:0/664298663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7dc0a26b0 msgr2=0x7ff7dc0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:00.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.526+0000 7ff7f8d02640 1 --2- 172.21.15.70:0/664298663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7dc0a26b0 0x7ff7dc0a4b70 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec033000 tx=0x7ff7ec02c120 comp rx=0 tx=0).stop 2024-04-01T01:44:00.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.526+0000 7ff7f8d02640 1 -- 172.21.15.70:0/664298663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7f410da70 msgr2=0x7ff7f41adc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:00.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.526+0000 7ff7f8d02640 1 --2- 172.21.15.70:0/664298663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7f410da70 0x7ff7f41adc00 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec02c660 tx=0x7ff7ec035ce0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.526+0000 7ff7f8d02640 1 -- 172.21.15.70:0/664298663 shutdown_connections 2024-04-01T01:44:00.527 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.526+0000 7ff7f8d02640 1 --2- 172.21.15.70:0/664298663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7f410da70 0x7ff7f41adc00 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.526+0000 7ff7f8d02640 1 --2- 172.21.15.70:0/664298663 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff7dc0a26b0 0x7ff7dc0a4b70 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.526+0000 7ff7f8d02640 1 --2- 172.21.15.70:0/664298663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7f410b0c0 0x7ff7f41a6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.526+0000 7ff7f8d02640 1 --2- 172.21.15.70:0/664298663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7f4108750 0x7ff7f41a66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.526+0000 7ff7f8d02640 1 -- 172.21.15.70:0/664298663 >> 172.21.15.70:0/664298663 conn(0x7ff7f406e790 msgr2=0x7ff7f410a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.527+0000 7ff7f8d02640 1 -- 172.21.15.70:0/664298663 shutdown_connections 2024-04-01T01:44:00.528 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:00.527+0000 7ff7f8d02640 1 -- 172.21.15.70:0/664298663 wait complete. 2024-04-01T01:44:01.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:00 smithi189 ceph-mon[35753]: pgmap v675: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 6.4 MiB/s wr, 832 op/s 2024-04-01T01:44:01.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:00 smithi070 ceph-mon[32179]: pgmap v675: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 6.4 MiB/s wr, 832 op/s 2024-04-01T01:44:01.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:00 smithi081 ceph-mon[35142]: pgmap v675: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 6.4 MiB/s wr, 832 op/s 2024-04-01T01:44:01.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 555.0 s, 847.8 tps, lat 117.933 ms stddev 54.083 2024-04-01T01:44:03.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:02 smithi189 ceph-mon[35753]: pgmap v676: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 7.3 MiB/s wr, 932 op/s 2024-04-01T01:44:03.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:02 smithi070 ceph-mon[32179]: pgmap v676: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 7.3 MiB/s wr, 932 op/s 2024-04-01T01:44:03.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:02 smithi081 ceph-mon[35142]: pgmap v676: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 7.3 MiB/s wr, 932 op/s 2024-04-01T01:44:05.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:04 smithi189 ceph-mon[35753]: pgmap v677: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 6.2 MiB/s wr, 828 op/s 2024-04-01T01:44:05.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:04 smithi070 ceph-mon[32179]: pgmap v677: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 6.2 MiB/s wr, 828 op/s 2024-04-01T01:44:05.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:04 smithi081 ceph-mon[35142]: pgmap v677: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 6.2 MiB/s wr, 828 op/s 2024-04-01T01:44:06.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 560.0 s, 868.2 tps, lat 114.669 ms stddev 95.920 2024-04-01T01:44:07.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:06 smithi189 ceph-mon[35753]: pgmap v678: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 8.0 MiB/s wr, 941 op/s 2024-04-01T01:44:07.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:06 smithi070 ceph-mon[32179]: pgmap v678: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 8.0 MiB/s wr, 941 op/s 2024-04-01T01:44:07.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:06 smithi081 ceph-mon[35142]: pgmap v678: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 8.0 MiB/s wr, 941 op/s 2024-04-01T01:44:08.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:08 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:44:08.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:08 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:44:08.753 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:08 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:44:09.579 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:09 smithi081 ceph-mon[35142]: pgmap v679: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 9.7 MiB/s wr, 953 op/s 2024-04-01T01:44:09.673 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:09 smithi189 ceph-mon[35753]: pgmap v679: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 9.7 MiB/s wr, 953 op/s 2024-04-01T01:44:09.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:09 smithi070 ceph-mon[32179]: pgmap v679: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 9.7 MiB/s wr, 953 op/s 2024-04-01T01:44:11.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 565.0 s, 811.6 tps, lat 123.639 ms stddev 109.385 2024-04-01T01:44:12.101 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:11 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:12.101 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:11 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:12.101 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:11 smithi070 ceph-mon[32179]: pgmap v680: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 10 MiB/s wr, 851 op/s 2024-04-01T01:44:12.101 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:11 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:12.101 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:11 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:12.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:11 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:12.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:11 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:12.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:11 smithi189 ceph-mon[35753]: pgmap v680: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 10 MiB/s wr, 851 op/s 2024-04-01T01:44:12.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:11 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:12.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:11 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:12.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:11 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:12.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:11 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:12.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:11 smithi081 ceph-mon[35142]: pgmap v680: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 10 MiB/s wr, 851 op/s 2024-04-01T01:44:12.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:11 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:12.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:11 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:13.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:12 smithi189 ceph-mon[35753]: pgmap v681: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 12 MiB/s wr, 944 op/s 2024-04-01T01:44:13.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:12 smithi070 ceph-mon[32179]: pgmap v681: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 12 MiB/s wr, 944 op/s 2024-04-01T01:44:13.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:12 smithi081 ceph-mon[35142]: pgmap v681: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 12 MiB/s wr, 944 op/s 2024-04-01T01:44:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.692+0000 7f552b4df640 1 -- 172.21.15.70:0/3035817713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55241008d0 msgr2=0x7f5524100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.692+0000 7f552b4df640 1 --2- 172.21.15.70:0/3035817713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55241008d0 0x7f5524100d50 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f5514009860 tx=0x7f551402bb70 comp rx=0 tx=0).stop 2024-04-01T01:44:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.692+0000 7f552b4df640 1 -- 172.21.15.70:0/3035817713 shutdown_connections 2024-04-01T01:44:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.692+0000 7f552b4df640 1 --2- 172.21.15.70:0/3035817713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5524101290 0x7f5524108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.692+0000 7f552b4df640 1 --2- 172.21.15.70:0/3035817713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55241008d0 0x7f5524100d50 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.692+0000 7f552b4df640 1 --2- 172.21.15.70:0/3035817713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5524101c80 0x7f55241020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.692+0000 7f552b4df640 1 -- 172.21.15.70:0/3035817713 >> 172.21.15.70:0/3035817713 conn(0x7f55240fb3a0 msgr2=0x7f55240fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.693+0000 7f552b4df640 1 -- 172.21.15.70:0/3035817713 shutdown_connections 2024-04-01T01:44:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.693+0000 7f552b4df640 1 -- 172.21.15.70:0/3035817713 wait complete. 2024-04-01T01:44:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.693+0000 7f552b4df640 1 Processor -- start 2024-04-01T01:44:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.694+0000 7f552b4df640 1 -- start start 2024-04-01T01:44:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.694+0000 7f552b4df640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55241008d0 0x7f5524199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.694+0000 7f552b4df640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5524101290 0x7f552419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.694+0000 7f552b4df640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5524101c80 0x7f55241a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.694+0000 7f552b4df640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f552410c1d0 con 0x7f5524101c80 2024-04-01T01:44:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.694+0000 7f552b4df640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f552410c050 con 0x7f55241008d0 2024-04-01T01:44:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.694+0000 7f552b4df640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f552410c350 con 0x7f5524101290 2024-04-01T01:44:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.694+0000 7f5529a7c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5524101c80 0x7f55241a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.694+0000 7f552927b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55241008d0 0x7f5524199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.694+0000 7f5529a7c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5524101c80 0x7f55241a10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40292/0 (socket says 172.21.15.70:40292) 2024-04-01T01:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.694+0000 7f5529a7c640 1 -- 172.21.15.70:0/1814828488 learned_addr learned my addr 172.21.15.70:0/1814828488 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.694+0000 7f5528a7a640 1 --2- 172.21.15.70:0/1814828488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5524101290 0x7f552419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.695+0000 7f5529a7c640 1 -- 172.21.15.70:0/1814828488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5524101290 msgr2=0x7f552419a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.695+0000 7f5529a7c640 1 --2- 172.21.15.70:0/1814828488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5524101290 0x7f552419a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.695+0000 7f5529a7c640 1 -- 172.21.15.70:0/1814828488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55241008d0 msgr2=0x7f5524199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.695+0000 7f5529a7c640 1 --2- 172.21.15.70:0/1814828488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55241008d0 0x7f5524199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.695+0000 7f5529a7c640 1 -- 172.21.15.70:0/1814828488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55241a1730 con 0x7f5524101c80 2024-04-01T01:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.695+0000 7f5529a7c640 1 --2- 172.21.15.70:0/1814828488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5524101c80 0x7f55241a10c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f551c00eb10 tx=0x7f551c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.695+0000 7f551a7fc640 1 -- 172.21.15.70:0/1814828488 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c00ce20 con 0x7f5524101c80 2024-04-01T01:44:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.695+0000 7f551a7fc640 1 -- 172.21.15.70:0/1814828488 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f551c005020 con 0x7f5524101c80 2024-04-01T01:44:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.696+0000 7f551a7fc640 1 -- 172.21.15.70:0/1814828488 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c010690 con 0x7f5524101c80 2024-04-01T01:44:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.696+0000 7f552b4df640 1 -- 172.21.15.70:0/1814828488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55241a1a20 con 0x7f5524101c80 2024-04-01T01:44:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.696+0000 7f552b4df640 1 -- 172.21.15.70:0/1814828488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55241a1ef0 con 0x7f5524101c80 2024-04-01T01:44:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.700+0000 7f551a7fc640 1 -- 172.21.15.70:0/1814828488 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f551c0051d0 con 0x7f5524101c80 2024-04-01T01:44:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.700+0000 7f552e19d740 1 Processor -- start 2024-04-01T01:44:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.700+0000 7f552e19d740 1 -- start start 2024-04-01T01:44:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.700+0000 7f552e19d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a08d0 0x555a0f8a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.700+0000 7f552e19d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a12c0 0x555a0f8a5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.700+0000 7f552e19d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 0x555a0f8a88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.700+0000 7f552e19d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555a0f8048f0 con 0x555a0f8a6430 2024-04-01T01:44:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.700+0000 7f552e19d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555a0f7ad8d0 con 0x555a0f8a08d0 2024-04-01T01:44:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.701+0000 7f552e19d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555a0f800990 con 0x555a0f8a12c0 2024-04-01T01:44:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.701+0000 7f552927b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a12c0 0x555a0f8a5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.701+0000 7f552927b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a12c0 0x555a0f8a5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:44324/0 (socket says 172.21.15.70:44324) 2024-04-01T01:44:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.701+0000 7f552927b640 1 -- 172.21.15.70:0/1650770463 learned_addr learned my addr 172.21.15.70:0/1650770463 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.701+0000 7f5528a7a640 1 --2- 172.21.15.70:0/1650770463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a08d0 0x555a0f8a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.701+0000 7f5528a7a640 1 --2- 172.21.15.70:0/1650770463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 0x555a0f8a88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.701+0000 7f551a7fc640 1 --2- 172.21.15.70:0/1814828488 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f55100a25c0 0x7f55100a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.701+0000 7f552927b640 1 -- 172.21.15.70:0/1650770463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a08d0 msgr2=0x555a0f8a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.701+0000 7f552927b640 1 --2- 172.21.15.70:0/1650770463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a08d0 0x555a0f8a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.701+0000 7f552927b640 1 -- 172.21.15.70:0/1650770463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 msgr2=0x555a0f8a88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.701+0000 7f552927b640 1 --2- 172.21.15.70:0/1650770463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 0x555a0f8a88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.701+0000 7f552927b640 1 -- 172.21.15.70:0/1650770463 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a0f519210 con 0x555a0f8a12c0 2024-04-01T01:44:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.701+0000 7f551a7fc640 1 -- 172.21.15.70:0/1814828488 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f551c0cf080 con 0x7f5524101c80 2024-04-01T01:44:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.701+0000 7f552927b640 1 --2- 172.21.15.70:0/1650770463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a12c0 0x555a0f8a5dc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5520009ca0 tx=0x7f5520011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed1246c4e185db7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.701+0000 7f5529a7c640 1 --2- 172.21.15.70:0/1814828488 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f55100a25c0 0x7f55100a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.702+0000 7f550f7fe640 1 -- 172.21.15.70:0/1650770463 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5520004230 con 0x555a0f8a12c0 2024-04-01T01:44:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.702+0000 7f550f7fe640 1 -- 172.21.15.70:0/1650770463 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55200043d0 con 0x555a0f8a12c0 2024-04-01T01:44:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.702+0000 7f550f7fe640 1 -- 172.21.15.70:0/1650770463 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5520004c90 con 0x555a0f8a12c0 2024-04-01T01:44:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.702+0000 7f552e19d740 1 -- 172.21.15.70:0/1650770463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a12c0 msgr2=0x555a0f8a5dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.702+0000 7f552e19d740 1 --2- 172.21.15.70:0/1650770463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a12c0 0x555a0f8a5dc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5520009ca0 tx=0x7f5520011810 comp rx=0 tx=0).stop 2024-04-01T01:44:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.702+0000 7f5529a7c640 1 --2- 172.21.15.70:0/1814828488 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f55100a25c0 0x7f55100a4a80 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f551c00c760 tx=0x7f551c005b20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.702+0000 7f552e19d740 1 -- 172.21.15.70:0/1650770463 shutdown_connections 2024-04-01T01:44:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.702+0000 7f552e19d740 1 --2- 172.21.15.70:0/1650770463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 0x555a0f8a88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.702+0000 7f552e19d740 1 --2- 172.21.15.70:0/1650770463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a12c0 0x555a0f8a5dc0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.702+0000 7f552e19d740 1 --2- 172.21.15.70:0/1650770463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a08d0 0x555a0f8a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.703+0000 7f552e19d740 1 -- 172.21.15.70:0/1650770463 >> 172.21.15.70:0/1650770463 conn(0x555a0f89be40 msgr2=0x555a0f89c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.703+0000 7f552e19d740 1 -- 172.21.15.70:0/1650770463 shutdown_connections 2024-04-01T01:44:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.703+0000 7f552e19d740 1 -- 172.21.15.70:0/1650770463 wait complete. 2024-04-01T01:44:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.703+0000 7f552e19d740 1 -- start start 2024-04-01T01:44:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.703+0000 7f552e19d740 1 Processor -- start 2024-04-01T01:44:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f552e19d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a08d0 0x555a0f8aab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f552e19d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a12c0 0x555a0f8ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f552e19d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 0x555a0f8b2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f552e19d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555a0f8a93e0 con 0x555a0f8a6430 2024-04-01T01:44:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f552e19d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555a0f7a2f80 con 0x555a0f8a08d0 2024-04-01T01:44:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f552e19d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555a0f8a9560 con 0x555a0f8a12c0 2024-04-01T01:44:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f5528a7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a08d0 0x555a0f8aab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f5528a7a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a08d0 0x555a0f8aab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:55212/0 (socket says 172.21.15.70:55212) 2024-04-01T01:44:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f5528a7a640 1 -- 172.21.15.70:0/3965210738 learned_addr learned my addr 172.21.15.70:0/3965210738 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f552927b640 1 --2- 172.21.15.70:0/3965210738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a12c0 0x555a0f8ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f552e19d740 10 client.0 _ll_register_callbacks cb 0x555a0f7fb290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f5528a7a640 1 --2- 172.21.15.70:0/3965210738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 0x555a0f8b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f552927b640 1 -- 172.21.15.70:0/3965210738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a08d0 msgr2=0x555a0f8aab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f552927b640 1 --2- 172.21.15.70:0/3965210738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a08d0 0x555a0f8aab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f552927b640 1 -- 172.21.15.70:0/3965210738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 msgr2=0x555a0f8b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f552927b640 1 --2- 172.21.15.70:0/3965210738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 0x555a0f8b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.704+0000 7f552927b640 1 -- 172.21.15.70:0/3965210738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a0f5e5620 con 0x555a0f8a12c0 2024-04-01T01:44:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.705+0000 7f5528a7a640 1 --2- 172.21.15.70:0/3965210738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a08d0 0x555a0f8aab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.705+0000 7f552927b640 1 --2- 172.21.15.70:0/3965210738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a12c0 0x555a0f8ab0c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x555a0f89cdb0 tx=0x7f5520019b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5a00f1d6806940 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.705+0000 7f550cff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:44:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.705+0000 7f550cff9640 1 -- 172.21.15.70:0/3965210738 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5520028020 con 0x555a0f8a12c0 2024-04-01T01:44:13.708 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.705+0000 7f550cff9640 1 -- 172.21.15.70:0/3965210738 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5520024620 con 0x555a0f8a12c0 2024-04-01T01:44:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.705+0000 7f552e19d740 1 -- 172.21.15.70:0/3965210738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55200043d0 con 0x555a0f8a12c0 2024-04-01T01:44:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.705+0000 7f550cff9640 1 -- 172.21.15.70:0/3965210738 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5520033050 con 0x555a0f8a12c0 2024-04-01T01:44:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.706+0000 7f552e19d740 10 client.28432 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.706+0000 7f552e19d740 1 -- 172.21.15.70:0/3965210738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555a0f8b50e0 con 0x555a0f8a12c0 2024-04-01T01:44:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.706+0000 7f550cff9640 1 -- 172.21.15.70:0/3965210738 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55200043d0 con 0x555a0f8a12c0 2024-04-01T01:44:13.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.706+0000 7f550cff9640 1 -- 172.21.15.70:0/3965210738 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f552002f4d0 con 0x555a0f8a12c0 2024-04-01T01:44:13.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.707+0000 7f552e19d740 10 client.28432 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:13.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.707+0000 7f552e19d740 10 client.28432 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:13.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.707+0000 7f552e19d740 20 client.28432 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:13.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.707+0000 7f552e19d740 1 --2- 172.21.15.70:0/3965210738 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555a0f8b65f0 0x555a0f8b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.707+0000 7f552e19d740 4 client.28432 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:13.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.707+0000 7f552e19d740 1 -- 172.21.15.70:0/3965210738 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a0f818120 con 0x555a0f8b65f0 2024-04-01T01:44:13.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.707+0000 7f5528a7a640 1 --2- 172.21.15.70:0/3965210738 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555a0f8b65f0 0x555a0f8b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.707+0000 7f5528a7a640 1 --2- 172.21.15.70:0/3965210738 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555a0f8b65f0 0x555a0f8b89e0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=446b12581157a237 server_cookie=652e5ffcfa87e62a in_seq=0 out_seq=0 2024-04-01T01:44:13.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.707+0000 7f550cff9640 10 client.28432 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:13.711 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.708+0000 7f550cff9640 1 -- 172.21.15.70:0/3965210738 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555a0f818120 con 0x555a0f8b65f0 2024-04-01T01:44:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.708+0000 7f550cff9640 10 client.28432 handle_command_reply: tid=0 2024-04-01T01:44:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.708+0000 7f552e19d740 1 client.28432 shutdown 2024-04-01T01:44:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.708+0000 7f552e19d740 1 -- 172.21.15.70:0/3965210738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a12c0 msgr2=0x555a0f8ab0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.709+0000 7f552e19d740 1 --2- 172.21.15.70:0/3965210738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a12c0 0x555a0f8ab0c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x555a0f89cdb0 tx=0x7f5520019b50 comp rx=0 tx=0).stop 2024-04-01T01:44:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.709+0000 7f552e19d740 1 -- 172.21.15.70:0/3965210738 shutdown_connections 2024-04-01T01:44:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.709+0000 7f552e19d740 1 --2- 172.21.15.70:0/3965210738 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555a0f8b65f0 0x555a0f8b89e0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.709+0000 7f552e19d740 1 --2- 172.21.15.70:0/3965210738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 0x555a0f8b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.709+0000 7f550cff9640 0 client.28432 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.709+0000 7f552e19d740 1 --2- 172.21.15.70:0/3965210738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a12c0 0x555a0f8ab0c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.709+0000 7f552e19d740 1 --2- 172.21.15.70:0/3965210738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a08d0 0x555a0f8aab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.709+0000 7f552e19d740 1 -- 172.21.15.70:0/3965210738 >> 172.21.15.70:0/3965210738 conn(0x555a0f89be40 msgr2=0x555a0f89af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.709+0000 7f552e19d740 1 -- 172.21.15.70:0/3965210738 shutdown_connections 2024-04-01T01:44:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.709+0000 7f552e19d740 1 -- 172.21.15.70:0/3965210738 wait complete. 2024-04-01T01:44:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.709+0000 7f552e19d740 20 client.28432 trim_cache size 0 max 16384 2024-04-01T01:44:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.735+0000 7f552e19d740 1 Processor -- start 2024-04-01T01:44:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.737+0000 7f552e19d740 1 -- start start 2024-04-01T01:44:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.737+0000 7f552e19d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a08d0 0x555a0f89c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.737+0000 7f552e19d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a12c0 0x555a0f8b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.738+0000 7f552e19d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 0x555a0f8a2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.738+0000 7f552e19d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f55241ac5b0 con 0x555a0f8a6430 2024-04-01T01:44:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.738+0000 7f552e19d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f55241ae0f0 con 0x555a0f8a12c0 2024-04-01T01:44:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.738+0000 7f552e19d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f55241acc80 con 0x555a0f8a08d0 2024-04-01T01:44:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.738+0000 7f5528a7a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 0x555a0f8a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.738+0000 7f5528a7a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 0x555a0f8a2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:40312/0 (socket says 172.21.15.70:40312) 2024-04-01T01:44:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.738+0000 7f5528a7a640 1 -- 172.21.15.70:0/532837677 learned_addr learned my addr 172.21.15.70:0/532837677 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.738+0000 7f5528a7a640 1 --2- 172.21.15.70:0/532837677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a08d0 0x555a0f89c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.738+0000 7f552927b640 1 --2- 172.21.15.70:0/532837677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a12c0 0x555a0f8b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.738+0000 7f5528a7a640 1 -- 172.21.15.70:0/532837677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a08d0 msgr2=0x555a0f89c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.738+0000 7f5528a7a640 1 --2- 172.21.15.70:0/532837677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a08d0 0x555a0f89c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.738+0000 7f5528a7a640 1 -- 172.21.15.70:0/532837677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a12c0 msgr2=0x555a0f8b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.738+0000 7f5528a7a640 1 --2- 172.21.15.70:0/532837677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a12c0 0x555a0f8b20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.738+0000 7f5528a7a640 1 -- 172.21.15.70:0/532837677 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a0f8b8d50 con 0x555a0f8a6430 2024-04-01T01:44:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.739+0000 7f5528a7a640 1 --2- 172.21.15.70:0/532837677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 0x555a0f8a2540 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f5514037040 tx=0x7f5514005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a71666b2c3200f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.739+0000 7f550effd640 1 -- 172.21.15.70:0/532837677 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5514039040 con 0x555a0f8a6430 2024-04-01T01:44:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.739+0000 7f550effd640 1 -- 172.21.15.70:0/532837677 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f551404e770 con 0x555a0f8a6430 2024-04-01T01:44:13.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.739+0000 7f550effd640 1 -- 172.21.15.70:0/532837677 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551402caa0 con 0x555a0f8a6430 2024-04-01T01:44:13.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.740+0000 7f552e19d740 1 -- 172.21.15.70:0/532837677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 msgr2=0x555a0f8a2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:13.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.740+0000 7f552e19d740 1 --2- 172.21.15.70:0/532837677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 0x555a0f8a2540 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f5514037040 tx=0x7f5514005f90 comp rx=0 tx=0).stop 2024-04-01T01:44:13.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.740+0000 7f552e19d740 1 -- 172.21.15.70:0/532837677 shutdown_connections 2024-04-01T01:44:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.740+0000 7f552e19d740 1 --2- 172.21.15.70:0/532837677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a6430 0x555a0f8a2540 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.740+0000 7f552e19d740 1 --2- 172.21.15.70:0/532837677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a12c0 0x555a0f8b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.740+0000 7f552e19d740 1 --2- 172.21.15.70:0/532837677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a08d0 0x555a0f89c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.740+0000 7f552e19d740 1 -- 172.21.15.70:0/532837677 >> 172.21.15.70:0/532837677 conn(0x555a0f89be40 msgr2=0x555a0f8a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.741+0000 7f552e19d740 1 -- 172.21.15.70:0/532837677 shutdown_connections 2024-04-01T01:44:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.742+0000 7f552e19d740 1 -- 172.21.15.70:0/532837677 wait complete. 2024-04-01T01:44:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.742+0000 7f552e19d740 1 -- start start 2024-04-01T01:44:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.742+0000 7f552e19d740 1 Processor -- start 2024-04-01T01:44:13.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.743+0000 7f552e19d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a08d0 0x555a0f8aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.743+0000 7f552e19d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a12c0 0x555a0f8af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.743+0000 7f552e19d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a6430 0x555a0f9230d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.743+0000 7f552e19d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f55241ada20 con 0x555a0f8a08d0 2024-04-01T01:44:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.743+0000 7f552e19d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f55241ad350 con 0x555a0f8a12c0 2024-04-01T01:44:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.743+0000 7f552e19d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f55241aa590 con 0x555a0f8a6430 2024-04-01T01:44:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.743+0000 7f552927b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a12c0 0x555a0f8af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.743+0000 7f552927b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a12c0 0x555a0f8af4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:55226/0 (socket says 172.21.15.70:55226) 2024-04-01T01:44:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.743+0000 7f552927b640 1 -- 172.21.15.70:0/1780718007 learned_addr learned my addr 172.21.15.70:0/1780718007 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.743+0000 7f552e19d740 10 client.0 _ll_register_callbacks cb 0x7f551c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.743+0000 7f5528a7a640 1 --2- 172.21.15.70:0/1780718007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a6430 0x555a0f9230d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.744+0000 7f5528a7a640 1 --2- 172.21.15.70:0/1780718007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a08d0 0x555a0f8aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.744+0000 7f552927b640 1 -- 172.21.15.70:0/1780718007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a6430 msgr2=0x555a0f9230d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.744+0000 7f552927b640 1 --2- 172.21.15.70:0/1780718007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a6430 0x555a0f9230d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.744+0000 7f552927b640 1 -- 172.21.15.70:0/1780718007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a08d0 msgr2=0x555a0f8aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.744+0000 7f552927b640 1 --2- 172.21.15.70:0/1780718007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a08d0 0x555a0f8aefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.744+0000 7f552927b640 1 -- 172.21.15.70:0/1780718007 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a0f923610 con 0x555a0f8a12c0 2024-04-01T01:44:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.744+0000 7f5528a7a640 1 --2- 172.21.15.70:0/1780718007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a08d0 0x555a0f8aefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:44:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.744+0000 7f552927b640 1 --2- 172.21.15.70:0/1780718007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a12c0 0x555a0f8af4e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f55200197c0 tx=0x7f552002fe10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93d0db6c1aaa1462 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.744+0000 7f550cff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:44:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.744+0000 7f550cff9640 1 -- 172.21.15.70:0/1780718007 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5520015020 con 0x555a0f8a12c0 2024-04-01T01:44:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.745+0000 7f552e19d740 1 -- 172.21.15.70:0/1780718007 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f551404e770 con 0x555a0f8a12c0 2024-04-01T01:44:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.745+0000 7f550cff9640 1 -- 172.21.15.70:0/1780718007 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5520005450 con 0x555a0f8a12c0 2024-04-01T01:44:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.745+0000 7f550cff9640 1 -- 172.21.15.70:0/1780718007 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5520028040 con 0x555a0f8a12c0 2024-04-01T01:44:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.745+0000 7f552e19d740 10 client.28646 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.746+0000 7f552e19d740 1 -- 172.21.15.70:0/1780718007 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555a0f924840 con 0x555a0f8a12c0 2024-04-01T01:44:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.746+0000 7f550cff9640 1 -- 172.21.15.70:0/1780718007 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f551404e770 con 0x555a0f8a12c0 2024-04-01T01:44:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.746+0000 7f550cff9640 1 -- 172.21.15.70:0/1780718007 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f552002f560 con 0x555a0f8a12c0 2024-04-01T01:44:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.746+0000 7f552e19d740 10 client.28646 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.746+0000 7f552e19d740 10 client.28646 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.746+0000 7f552e19d740 20 client.28646 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.746+0000 7f552e19d740 1 --2- 172.21.15.70:0/1780718007 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555a0f925270 0x555a0f927690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.746+0000 7f552e19d740 4 client.28646 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:44:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.747+0000 7f552e19d740 1 -- 172.21.15.70:0/1780718007 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555a0f811e30 con 0x555a0f925270 2024-04-01T01:44:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.747+0000 7f5528a7a640 1 --2- 172.21.15.70:0/1780718007 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555a0f925270 0x555a0f927690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.747+0000 7f5528a7a640 1 --2- 172.21.15.70:0/1780718007 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555a0f925270 0x555a0f927690 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae2fead5df0823b9 server_cookie=28e78281cab4ab30 in_seq=0 out_seq=0 2024-04-01T01:44:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.747+0000 7f550cff9640 10 client.28646 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:13.808 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.807+0000 7f550cff9640 1 -- 172.21.15.70:0/1780718007 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555a0f811e30 con 0x555a0f925270 2024-04-01T01:44:13.808 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.807+0000 7f550cff9640 10 client.28646 handle_command_reply: tid=0 2024-04-01T01:44:13.808 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.807+0000 7f552e19d740 1 client.28646 shutdown 2024-04-01T01:44:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.809+0000 7f552e19d740 1 -- 172.21.15.70:0/1780718007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a12c0 msgr2=0x555a0f8af4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.809+0000 7f552e19d740 1 --2- 172.21.15.70:0/1780718007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a12c0 0x555a0f8af4e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f55200197c0 tx=0x7f552002fe10 comp rx=0 tx=0).stop 2024-04-01T01:44:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.810+0000 7f552e19d740 1 -- 172.21.15.70:0/1780718007 shutdown_connections 2024-04-01T01:44:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.810+0000 7f552e19d740 1 --2- 172.21.15.70:0/1780718007 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x555a0f925270 0x555a0f927690 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.810 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.810+0000 7f552e19d740 1 --2- 172.21.15.70:0/1780718007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a0f8a6430 0x555a0f9230d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.810+0000 7f552e19d740 1 --2- 172.21.15.70:0/1780718007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a0f8a12c0 0x555a0f8af4e0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.810+0000 7f552e19d740 1 --2- 172.21.15.70:0/1780718007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555a0f8a08d0 0x555a0f8aefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.810+0000 7f552e19d740 1 -- 172.21.15.70:0/1780718007 >> 172.21.15.70:0/1780718007 conn(0x555a0f89be40 msgr2=0x555a0f8aab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.810+0000 7f550cff9640 0 client.28646 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.810+0000 7f552e19d740 1 -- 172.21.15.70:0/1780718007 shutdown_connections 2024-04-01T01:44:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.810+0000 7f552e19d740 1 -- 172.21.15.70:0/1780718007 wait complete. 2024-04-01T01:44:13.811 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.811+0000 7f552e19d740 20 client.28646 trim_cache size 0 max 16384 2024-04-01T01:44:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:44:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:44:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:44:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:44:13.814 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.813+0000 7f552b4df640 1 -- 172.21.15.70:0/1814828488 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f55100a25c0 msgr2=0x7f55100a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.813+0000 7f552b4df640 1 --2- 172.21.15.70:0/1814828488 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f55100a25c0 0x7f55100a4a80 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f551c00c760 tx=0x7f551c005b20 comp rx=0 tx=0).stop 2024-04-01T01:44:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.813+0000 7f552b4df640 1 -- 172.21.15.70:0/1814828488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5524101c80 msgr2=0x7f55241a10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.813+0000 7f552b4df640 1 --2- 172.21.15.70:0/1814828488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5524101c80 0x7f55241a10c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f551c00eb10 tx=0x7f551c00ef10 comp rx=0 tx=0).stop 2024-04-01T01:44:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.814+0000 7f552b4df640 1 -- 172.21.15.70:0/1814828488 shutdown_connections 2024-04-01T01:44:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.814+0000 7f552b4df640 1 --2- 172.21.15.70:0/1814828488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5524101c80 0x7f55241a10c0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.814+0000 7f552b4df640 1 --2- 172.21.15.70:0/1814828488 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f55100a25c0 0x7f55100a4a80 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.814+0000 7f552b4df640 1 --2- 172.21.15.70:0/1814828488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5524101290 0x7f552419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.814+0000 7f552b4df640 1 --2- 172.21.15.70:0/1814828488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55241008d0 0x7f5524199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.814+0000 7f552b4df640 1 -- 172.21.15.70:0/1814828488 >> 172.21.15.70:0/1814828488 conn(0x7f55240fb3a0 msgr2=0x7f55240fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.814+0000 7f552b4df640 1 -- 172.21.15.70:0/1814828488 shutdown_connections 2024-04-01T01:44:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:13.814+0000 7f552b4df640 1 -- 172.21.15.70:0/1814828488 wait complete. 2024-04-01T01:44:14.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:14 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:14.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:14 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:14.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:14 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:14.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:14 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:14.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:14 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:14.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:14 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:15.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:15 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:44:15.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:15 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:44:15.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:15 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:15.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:15 smithi189 ceph-mon[35753]: pgmap v682: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 11 MiB/s wr, 814 op/s 2024-04-01T01:44:15.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:15 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:44:15.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:15 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:44:15.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:15 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:15.505 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:15 smithi070 ceph-mon[32179]: pgmap v682: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 11 MiB/s wr, 814 op/s 2024-04-01T01:44:15.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:15 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:44:15.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:15 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:44:15.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:15 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:44:15.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:15 smithi081 ceph-mon[35142]: pgmap v682: 129 pgs: 129 active+clean; 1.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 11 MiB/s wr, 814 op/s 2024-04-01T01:44:16.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 570.0 s, 939.2 tps, lat 104.970 ms stddev 82.835 2024-04-01T01:44:17.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:16 smithi070 ceph-mon[32179]: pgmap v683: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 13 MiB/s wr, 887 op/s 2024-04-01T01:44:17.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:16 smithi189 ceph-mon[35753]: pgmap v683: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 13 MiB/s wr, 887 op/s 2024-04-01T01:44:17.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:16 smithi081 ceph-mon[35142]: pgmap v683: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 13 MiB/s wr, 887 op/s 2024-04-01T01:44:19.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:18 smithi189 ceph-mon[35753]: pgmap v684: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 14 MiB/s wr, 880 op/s 2024-04-01T01:44:19.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:18 smithi070 ceph-mon[32179]: pgmap v684: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 14 MiB/s wr, 880 op/s 2024-04-01T01:44:19.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:18 smithi081 ceph-mon[35142]: pgmap v684: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 14 MiB/s wr, 880 op/s 2024-04-01T01:44:21.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:20 smithi189 ceph-mon[35753]: pgmap v685: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 13 MiB/s wr, 763 op/s 2024-04-01T01:44:21.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:20 smithi070 ceph-mon[32179]: pgmap v685: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 13 MiB/s wr, 763 op/s 2024-04-01T01:44:21.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:20 smithi081 ceph-mon[35142]: pgmap v685: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 13 MiB/s wr, 763 op/s 2024-04-01T01:44:21.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 575.0 s, 863.4 tps, lat 117.307 ms stddev 99.817 2024-04-01T01:44:23.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:22 smithi189 ceph-mon[35753]: pgmap v686: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 13 MiB/s wr, 829 op/s 2024-04-01T01:44:23.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:22 smithi070 ceph-mon[32179]: pgmap v686: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 13 MiB/s wr, 829 op/s 2024-04-01T01:44:23.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:22 smithi081 ceph-mon[35142]: pgmap v686: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 13 MiB/s wr, 829 op/s 2024-04-01T01:44:23.886 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:44:23.887 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:44:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.069+0000 7f6f576cd640 1 -- 172.21.15.70:0/1127545898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f50076110 msgr2=0x7f6f50076530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.069+0000 7f6f576cd640 1 --2- 172.21.15.70:0/1127545898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f50076110 0x7f6f50076530 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009900 tx=0x7f6f4c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:44:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.070+0000 7f6f576cd640 1 -- 172.21.15.70:0/1127545898 shutdown_connections 2024-04-01T01:44:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.070+0000 7f6f576cd640 1 --2- 172.21.15.70:0/1127545898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f50077cf0 0x7f6f50078590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.070+0000 7f6f576cd640 1 --2- 172.21.15.70:0/1127545898 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f50077350 0x7f6f500777b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.070+0000 7f6f576cd640 1 --2- 172.21.15.70:0/1127545898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f50076110 0x7f6f50076530 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.070+0000 7f6f576cd640 1 -- 172.21.15.70:0/1127545898 >> 172.21.15.70:0/1127545898 conn(0x7f6f500fef00 msgr2=0x7f6f50101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:24.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.070+0000 7f6f576cd640 1 -- 172.21.15.70:0/1127545898 shutdown_connections 2024-04-01T01:44:24.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.070+0000 7f6f576cd640 1 -- 172.21.15.70:0/1127545898 wait complete. 2024-04-01T01:44:24.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.071+0000 7f6f576cd640 1 Processor -- start 2024-04-01T01:44:24.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.071+0000 7f6f576cd640 1 -- start start 2024-04-01T01:44:24.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.071+0000 7f6f576cd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f50076110 0x7f6f5019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.071+0000 7f6f576cd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f50077350 0x7f6f5019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.071+0000 7f6f576cd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f50077cf0 0x7f6f501a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.072+0000 7f6f576cd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6f50113910 con 0x7f6f50077350 2024-04-01T01:44:24.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.072+0000 7f6f576cd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f50113790 con 0x7f6f50077cf0 2024-04-01T01:44:24.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.072+0000 7f6f576cd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6f50113a90 con 0x7f6f50076110 2024-04-01T01:44:24.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.072+0000 7f6f54c68640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f50077350 0x7f6f5019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.072+0000 7f6f54c68640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f50077350 0x7f6f5019e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35522/0 (socket says 172.21.15.70:35522) 2024-04-01T01:44:24.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.072+0000 7f6f54c68640 1 -- 172.21.15.70:0/2226441102 learned_addr learned my addr 172.21.15.70:0/2226441102 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:24.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.072+0000 7f6f55c6a640 1 --2- 172.21.15.70:0/2226441102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f50077cf0 0x7f6f501a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.073+0000 7f6f55469640 1 --2- 172.21.15.70:0/2226441102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f50076110 0x7f6f5019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.075+0000 7f6f55469640 1 -- 172.21.15.70:0/2226441102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f50077cf0 msgr2=0x7f6f501a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:24.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.075+0000 7f6f55469640 1 --2- 172.21.15.70:0/2226441102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f50077cf0 0x7f6f501a55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.075+0000 7f6f55469640 1 -- 172.21.15.70:0/2226441102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f50077350 msgr2=0x7f6f5019e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:24.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.075+0000 7f6f55469640 1 --2- 172.21.15.70:0/2226441102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f50077350 0x7f6f5019e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.075+0000 7f6f55469640 1 -- 172.21.15.70:0/2226441102 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f501a5c20 con 0x7f6f50076110 2024-04-01T01:44:24.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.075+0000 7f6f55469640 1 --2- 172.21.15.70:0/2226441102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f50076110 0x7f6f5019e000 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009850 tx=0x7f6f4c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:24.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.075+0000 7f6f427fc640 1 -- 172.21.15.70:0/2226441102 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c043020 con 0x7f6f50076110 2024-04-01T01:44:24.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.075+0000 7f6f427fc640 1 -- 172.21.15.70:0/2226441102 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f4c0052b0 con 0x7f6f50076110 2024-04-01T01:44:24.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.075+0000 7f6f427fc640 1 -- 172.21.15.70:0/2226441102 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c043020 con 0x7f6f50076110 2024-04-01T01:44:24.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.076+0000 7f6f576cd640 1 -- 172.21.15.70:0/2226441102 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f501a5eb0 con 0x7f6f50076110 2024-04-01T01:44:24.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.076+0000 7f6f576cd640 1 -- 172.21.15.70:0/2226441102 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f501a62c0 con 0x7f6f50076110 2024-04-01T01:44:24.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.079+0000 7f6f5a382740 1 Processor -- start 2024-04-01T01:44:24.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.079+0000 7f6f5a382740 1 -- start start 2024-04-01T01:44:24.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.080+0000 7f6f5a382740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fab2c0 0x564998fab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.080+0000 7f6f427fc640 1 -- 172.21.15.70:0/2226441102 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f6f4c046020 con 0x7f6f50076110 2024-04-01T01:44:24.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.080+0000 7f6f5a382740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fabcb0 0x564998fb07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.080+0000 7f6f5a382740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fb0e20 0x564998fb32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.080+0000 7f6f5a382740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564998eff180 con 0x564998fab2c0 2024-04-01T01:44:24.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.080+0000 7f6f5a382740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x564998ea7b70 con 0x564998fb0e20 2024-04-01T01:44:24.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.080+0000 7f6f5a382740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564998efb220 con 0x564998fabcb0 2024-04-01T01:44:24.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.080+0000 7f6f55c6a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fab2c0 0x564998fab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.080+0000 7f6f55c6a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fab2c0 0x564998fab6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35532/0 (socket says 172.21.15.70:35532) 2024-04-01T01:44:24.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.080+0000 7f6f55c6a640 1 -- 172.21.15.70:0/3612885265 learned_addr learned my addr 172.21.15.70:0/3612885265 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:24.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.080+0000 7f6f54c68640 1 --2- 172.21.15.70:0/3612885265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fb0e20 0x564998fb32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.081+0000 7f6f427fc640 1 --2- 172.21.15.70:0/2226441102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f3c0a28e0 0x7f6f3c0a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.081+0000 7f6f427fc640 1 -- 172.21.15.70:0/2226441102 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6f4c03a040 con 0x7f6f50076110 2024-04-01T01:44:24.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.081+0000 7f6f54c68640 1 -- 172.21.15.70:0/3612885265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fabcb0 msgr2=0x564998fb07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:24.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.081+0000 7f6f54c68640 1 --2- 172.21.15.70:0/3612885265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fabcb0 0x564998fb07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.081+0000 7f6f54c68640 1 -- 172.21.15.70:0/3612885265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fab2c0 msgr2=0x564998fab6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:24.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.081+0000 7f6f54c68640 1 --2- 172.21.15.70:0/3612885265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fab2c0 0x564998fab6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.081+0000 7f6f54c68640 1 -- 172.21.15.70:0/3612885265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564998ebb7c0 con 0x564998fb0e20 2024-04-01T01:44:24.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.081+0000 7f6f55c6a640 1 --2- 172.21.15.70:0/3612885265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fab2c0 0x564998fab6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:44:24.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.082+0000 7f6f55c6a640 1 --2- 172.21.15.70:0/2226441102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f3c0a28e0 0x7f6f3c0a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.082+0000 7f6f54c68640 1 --2- 172.21.15.70:0/3612885265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fb0e20 0x564998fb32b0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f6f44005fd0 tx=0x7f6f44004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83380b751c1280f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:24.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.083+0000 7f6f55c6a640 1 --2- 172.21.15.70:0/2226441102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f3c0a28e0 0x7f6f3c0a4da0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x564998fac510 tx=0x7f6f48004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:24.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.085+0000 7f6f3b7fe640 1 -- 172.21.15.70:0/3612885265 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f44003dd0 con 0x564998fb0e20 2024-04-01T01:44:24.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.085+0000 7f6f3b7fe640 1 -- 172.21.15.70:0/3612885265 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f44015070 con 0x564998fb0e20 2024-04-01T01:44:24.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.085+0000 7f6f3b7fe640 1 -- 172.21.15.70:0/3612885265 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f44011440 con 0x564998fb0e20 2024-04-01T01:44:24.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.086+0000 7f6f5a382740 1 -- 172.21.15.70:0/3612885265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fb0e20 msgr2=0x564998fb32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:24.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.086+0000 7f6f5a382740 1 --2- 172.21.15.70:0/3612885265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fb0e20 0x564998fb32b0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f6f44005fd0 tx=0x7f6f44004610 comp rx=0 tx=0).stop 2024-04-01T01:44:24.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.086+0000 7f6f5a382740 1 -- 172.21.15.70:0/3612885265 shutdown_connections 2024-04-01T01:44:24.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.086+0000 7f6f5a382740 1 --2- 172.21.15.70:0/3612885265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fb0e20 0x564998fb32b0 secure :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f6f44005fd0 tx=0x7f6f44004610 comp rx=0 tx=0).stop 2024-04-01T01:44:24.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.086+0000 7f6f5a382740 1 --2- 172.21.15.70:0/3612885265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fabcb0 0x564998fb07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.086+0000 7f6f5a382740 1 --2- 172.21.15.70:0/3612885265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fab2c0 0x564998fab6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.086+0000 7f6f5a382740 1 -- 172.21.15.70:0/3612885265 >> 172.21.15.70:0/3612885265 conn(0x564998f95820 msgr2=0x564998f95c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:24.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.086+0000 7f6f5a382740 1 -- 172.21.15.70:0/3612885265 shutdown_connections 2024-04-01T01:44:24.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.087+0000 7f6f5a382740 1 -- 172.21.15.70:0/3612885265 wait complete. 2024-04-01T01:44:24.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.087+0000 7f6f5a382740 1 -- start start 2024-04-01T01:44:24.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.087+0000 7f6f5a382740 1 Processor -- start 2024-04-01T01:44:24.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.087+0000 7f6f5a382740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fab2c0 0x564998fc6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.087+0000 7f6f5a382740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fc6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.088+0000 7f6f5a382740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fcbfc0 0x564998fce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.088+0000 7f6f5a382740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564998fb3dd0 con 0x564998fcbfc0 2024-04-01T01:44:24.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.088+0000 7f6f5a382740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x564998e9d4a0 con 0x564998fabcb0 2024-04-01T01:44:24.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.088+0000 7f6f5a382740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564998fb3f50 con 0x564998fab2c0 2024-04-01T01:44:24.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.088+0000 7f6f5a382740 10 client.0 _ll_register_callbacks cb 0x564998ea0d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:24.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.088+0000 7f6f54c68640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fc6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.088+0000 7f6f55469640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fab2c0 0x564998fc6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.088+0000 7f6f54c68640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fc6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53400/0 (socket says 172.21.15.70:53400) 2024-04-01T01:44:24.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.088+0000 7f6f54c68640 1 -- 172.21.15.70:0/2534849058 learned_addr learned my addr 172.21.15.70:0/2534849058 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:24.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.088+0000 7f6f55c6a640 1 --2- 172.21.15.70:0/2534849058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fcbfc0 0x564998fce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.089+0000 7f6f54c68640 1 -- 172.21.15.70:0/2534849058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fab2c0 msgr2=0x564998fc6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:24.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.089+0000 7f6f54c68640 1 --2- 172.21.15.70:0/2534849058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fab2c0 0x564998fc6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.089+0000 7f6f54c68640 1 -- 172.21.15.70:0/2534849058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fcbfc0 msgr2=0x564998fce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:24.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.089+0000 7f6f54c68640 1 --2- 172.21.15.70:0/2534849058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fcbfc0 0x564998fce3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.089+0000 7f6f54c68640 1 -- 172.21.15.70:0/2534849058 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564998cdd7d0 con 0x564998fabcb0 2024-04-01T01:44:24.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.089+0000 7f6f54c68640 1 --2- 172.21.15.70:0/2534849058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fc6f90 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f6f44000c00 tx=0x7f6f44003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd1bdc890beb3d7b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:24.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.089+0000 7f6f38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:44:24.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.089+0000 7f6f38ff9640 1 -- 172.21.15.70:0/2534849058 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4400be50 con 0x564998fabcb0 2024-04-01T01:44:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.090+0000 7f6f38ff9640 1 -- 172.21.15.70:0/2534849058 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6f44011650 con 0x564998fabcb0 2024-04-01T01:44:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.090+0000 7f6f5a382740 1 -- 172.21.15.70:0/2534849058 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f44015070 con 0x564998fabcb0 2024-04-01T01:44:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.090+0000 7f6f38ff9640 1 -- 172.21.15.70:0/2534849058 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6f44030050 con 0x564998fabcb0 2024-04-01T01:44:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.091+0000 7f6f5a382740 10 client.28658 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.091+0000 7f6f5a382740 1 -- 172.21.15.70:0/2534849058 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564998c133f0 con 0x564998fabcb0 2024-04-01T01:44:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.091+0000 7f6f38ff9640 1 -- 172.21.15.70:0/2534849058 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f44015070 con 0x564998fabcb0 2024-04-01T01:44:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.092+0000 7f6f38ff9640 1 -- 172.21.15.70:0/2534849058 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6f440258a0 con 0x564998fabcb0 2024-04-01T01:44:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.092+0000 7f6f5a382740 10 client.28658 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.092+0000 7f6f5a382740 10 client.28658 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.092+0000 7f6f5a382740 20 client.28658 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.092+0000 7f6f5a382740 1 --2- 172.21.15.70:0/2534849058 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564998fb87f0 0x564998fbac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.092+0000 7f6f5a382740 4 client.28658 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.092+0000 7f6f5a382740 1 -- 172.21.15.70:0/2534849058 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564998f0c6c0 con 0x564998fb87f0 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.092+0000 7f6f55469640 1 --2- 172.21.15.70:0/2534849058 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564998fb87f0 0x564998fbac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.093+0000 7f6f55469640 1 --2- 172.21.15.70:0/2534849058 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564998fb87f0 0x564998fbac10 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd3c87c22cfb9b9e server_cookie=ca35dfed052c616a in_seq=0 out_seq=0 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.093+0000 7f6f38ff9640 10 client.28658 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.094+0000 7f6f38ff9640 1 -- 172.21.15.70:0/2534849058 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564998f0c6c0 con 0x564998fb87f0 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.094+0000 7f6f38ff9640 10 client.28658 handle_command_reply: tid=0 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.094+0000 7f6f5a382740 1 client.28658 shutdown 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.095+0000 7f6f5a382740 1 -- 172.21.15.70:0/2534849058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 msgr2=0x564998fc6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.095+0000 7f6f5a382740 1 --2- 172.21.15.70:0/2534849058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fc6f90 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f6f44000c00 tx=0x7f6f44003e00 comp rx=0 tx=0).stop 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.095+0000 7f6f5a382740 1 -- 172.21.15.70:0/2534849058 shutdown_connections 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.095+0000 7f6f5a382740 1 --2- 172.21.15.70:0/2534849058 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564998fb87f0 0x564998fbac10 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.095+0000 7f6f5a382740 1 --2- 172.21.15.70:0/2534849058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fcbfc0 0x564998fce3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.095+0000 7f6f5a382740 1 --2- 172.21.15.70:0/2534849058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fc6f90 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.095+0000 7f6f5a382740 1 --2- 172.21.15.70:0/2534849058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fab2c0 0x564998fc6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.095+0000 7f6f5a382740 1 -- 172.21.15.70:0/2534849058 >> 172.21.15.70:0/2534849058 conn(0x564998f95820 msgr2=0x564998f94c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.095+0000 7f6f38ff9640 0 client.28658 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.096+0000 7f6f5a382740 1 -- 172.21.15.70:0/2534849058 shutdown_connections 2024-04-01T01:44:24.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.096+0000 7f6f5a382740 1 -- 172.21.15.70:0/2534849058 wait complete. 2024-04-01T01:44:24.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.096+0000 7f6f5a382740 20 client.28658 trim_cache size 0 max 16384 2024-04-01T01:44:24.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.143+0000 7f6f5a382740 1 Processor -- start 2024-04-01T01:44:24.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.143+0000 7f6f5a382740 1 -- start start 2024-04-01T01:44:24.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.143+0000 7f6f5a382740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fab2c0 0x564998fc6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.143+0000 7f6f5a382740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fc6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.143+0000 7f6f5a382740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fcbfc0 0x564998fad390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.143+0000 7f6f5a382740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3400fcc0 con 0x564998fab2c0 2024-04-01T01:44:24.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.143+0000 7f6f5a382740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f34011800 con 0x564998fabcb0 2024-04-01T01:44:24.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.143+0000 7f6f5a382740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6f34010390 con 0x564998fcbfc0 2024-04-01T01:44:24.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.144+0000 7f6f54c68640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fc6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.144+0000 7f6f54c68640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fc6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53402/0 (socket says 172.21.15.70:53402) 2024-04-01T01:44:24.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.144+0000 7f6f54c68640 1 -- 172.21.15.70:0/3451847954 learned_addr learned my addr 172.21.15.70:0/3451847954 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:24.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.144+0000 7f6f55c6a640 1 --2- 172.21.15.70:0/3451847954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fcbfc0 0x564998fad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.144+0000 7f6f54c68640 1 -- 172.21.15.70:0/3451847954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fcbfc0 msgr2=0x564998fad390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:24.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.144+0000 7f6f54c68640 1 --2- 172.21.15.70:0/3451847954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fcbfc0 0x564998fad390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.144+0000 7f6f54c68640 1 -- 172.21.15.70:0/3451847954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fab2c0 msgr2=0x564998fc6a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:44:24.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.144+0000 7f6f54c68640 1 --2- 172.21.15.70:0/3451847954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fab2c0 0x564998fc6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.144+0000 7f6f54c68640 1 -- 172.21.15.70:0/3451847954 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564998bd30f0 con 0x564998fabcb0 2024-04-01T01:44:24.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.145+0000 7f6f54c68640 1 --2- 172.21.15.70:0/3451847954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fc6f90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6f44003a40 tx=0x7f6f44004320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7d24f0b0140e80c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:24.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.145+0000 7f6f3affd640 1 -- 172.21.15.70:0/3451847954 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f44019420 con 0x564998fabcb0 2024-04-01T01:44:24.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.145+0000 7f6f3affd640 1 -- 172.21.15.70:0/3451847954 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f44015040 con 0x564998fabcb0 2024-04-01T01:44:24.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.146+0000 7f6f5a382740 1 -- 172.21.15.70:0/3451847954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 msgr2=0x564998fc6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:24.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.146+0000 7f6f5a382740 1 --2- 172.21.15.70:0/3451847954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fc6f90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6f44003a40 tx=0x7f6f44004320 comp rx=0 tx=0).stop 2024-04-01T01:44:24.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.146+0000 7f6f5a382740 1 -- 172.21.15.70:0/3451847954 shutdown_connections 2024-04-01T01:44:24.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.146+0000 7f6f5a382740 1 --2- 172.21.15.70:0/3451847954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fcbfc0 0x564998fad390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.146+0000 7f6f5a382740 1 --2- 172.21.15.70:0/3451847954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fc6f90 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.146+0000 7f6f5a382740 1 --2- 172.21.15.70:0/3451847954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fab2c0 0x564998fc6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.146+0000 7f6f5a382740 1 -- 172.21.15.70:0/3451847954 >> 172.21.15.70:0/3451847954 conn(0x564998f95820 msgr2=0x564998fb4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:24.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.151+0000 7f6f5a382740 1 -- 172.21.15.70:0/3451847954 shutdown_connections 2024-04-01T01:44:24.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.151+0000 7f6f5a382740 1 -- 172.21.15.70:0/3451847954 wait complete. 2024-04-01T01:44:24.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.152+0000 7f6f5a382740 1 -- start start 2024-04-01T01:44:24.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.152+0000 7f6f5a382740 1 Processor -- start 2024-04-01T01:44:24.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.152+0000 7f6f5a382740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fab2c0 0x564998fbda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.152+0000 7f6f5a382740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fbdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.152+0000 7f6f5a382740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fcbfc0 0x564998fbef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.152+0000 7f6f5a382740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6f34011130 con 0x564998fab2c0 2024-04-01T01:44:24.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.152+0000 7f6f5a382740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f34010a60 con 0x564998fabcb0 2024-04-01T01:44:24.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.152+0000 7f6f5a382740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3400dca0 con 0x564998fcbfc0 2024-04-01T01:44:24.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.152+0000 7f6f5a382740 10 client.0 _ll_register_callbacks cb 0x7f6f44004490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:24.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.153+0000 7f6f55c6a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fcbfc0 0x564998fbef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.153+0000 7f6f55c6a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fcbfc0 0x564998fbef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52930/0 (socket says 172.21.15.70:52930) 2024-04-01T01:44:24.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.153+0000 7f6f55c6a640 1 -- 172.21.15.70:0/298022826 learned_addr learned my addr 172.21.15.70:0/298022826 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:24.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.153+0000 7f6f55469640 1 --2- 172.21.15.70:0/298022826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fab2c0 0x564998fbda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.153+0000 7f6f54c68640 1 --2- 172.21.15.70:0/298022826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fbdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.154+0000 7f6f54c68640 1 -- 172.21.15.70:0/298022826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fcbfc0 msgr2=0x564998fbef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:24.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.154+0000 7f6f54c68640 1 --2- 172.21.15.70:0/298022826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fcbfc0 0x564998fbef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.154+0000 7f6f54c68640 1 -- 172.21.15.70:0/298022826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fab2c0 msgr2=0x564998fbda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:24.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.154+0000 7f6f54c68640 1 --2- 172.21.15.70:0/298022826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fab2c0 0x564998fbda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.154+0000 7f6f54c68640 1 -- 172.21.15.70:0/298022826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564998fbf580 con 0x564998fabcb0 2024-04-01T01:44:24.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.154+0000 7f6f54c68640 1 --2- 172.21.15.70:0/298022826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fbdfd0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f6f44025c60 tx=0x7f6f440040e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d42f489787dce62 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:24.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.154+0000 7f6f38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:44:24.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.155+0000 7f6f38ff9640 1 -- 172.21.15.70:0/298022826 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f440256c0 con 0x564998fabcb0 2024-04-01T01:44:24.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.155+0000 7f6f5a382740 1 -- 172.21.15.70:0/298022826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564998f06a10 con 0x564998fabcb0 2024-04-01T01:44:24.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.155+0000 7f6f38ff9640 1 -- 172.21.15.70:0/298022826 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6f44021040 con 0x564998fabcb0 2024-04-01T01:44:24.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.155+0000 7f6f38ff9640 1 -- 172.21.15.70:0/298022826 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6f44044050 con 0x564998fabcb0 2024-04-01T01:44:24.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.155+0000 7f6f5a382740 10 client.28664 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:24.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.155+0000 7f6f5a382740 1 -- 172.21.15.70:0/298022826 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564998fc0950 con 0x564998fabcb0 2024-04-01T01:44:24.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.157+0000 7f6f38ff9640 1 -- 172.21.15.70:0/298022826 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x564998f06a10 con 0x564998fabcb0 2024-04-01T01:44:24.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.157+0000 7f6f38ff9640 1 -- 172.21.15.70:0/298022826 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6f44004690 con 0x564998fabcb0 2024-04-01T01:44:24.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.158+0000 7f6f5a382740 10 client.28664 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:24.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.158+0000 7f6f5a382740 10 client.28664 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:24.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.158+0000 7f6f5a382740 20 client.28664 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:24.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.158+0000 7f6f5a382740 1 --2- 172.21.15.70:0/298022826 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564999049e10 0x56499904c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.158+0000 7f6f5a382740 4 client.28664 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:44:24.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.158+0000 7f6f5a382740 1 -- 172.21.15.70:0/298022826 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564998e84540 con 0x564999049e10 2024-04-01T01:44:24.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.158+0000 7f6f55469640 1 --2- 172.21.15.70:0/298022826 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564999049e10 0x56499904c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.160+0000 7f6f55469640 1 --2- 172.21.15.70:0/298022826 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564999049e10 0x56499904c230 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4a68db5f2abdf1a server_cookie=8e9fe2dd0b50c72 in_seq=0 out_seq=0 2024-04-01T01:44:24.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.160+0000 7f6f38ff9640 10 client.28664 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:24.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.161+0000 7f6f38ff9640 1 -- 172.21.15.70:0/298022826 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564998e84540 con 0x564999049e10 2024-04-01T01:44:24.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.161+0000 7f6f38ff9640 10 client.28664 handle_command_reply: tid=0 2024-04-01T01:44:24.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.161+0000 7f6f5a382740 1 client.28664 shutdown 2024-04-01T01:44:24.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.161+0000 7f6f5a382740 1 -- 172.21.15.70:0/298022826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 msgr2=0x564998fbdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:24.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.161+0000 7f6f5a382740 1 --2- 172.21.15.70:0/298022826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fbdfd0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f6f44025c60 tx=0x7f6f440040e0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.161+0000 7f6f5a382740 1 -- 172.21.15.70:0/298022826 shutdown_connections 2024-04-01T01:44:24.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.161+0000 7f6f5a382740 1 --2- 172.21.15.70:0/298022826 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564999049e10 0x56499904c230 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.161+0000 7f6f5a382740 1 --2- 172.21.15.70:0/298022826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564998fcbfc0 0x564998fbef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.161+0000 7f6f5a382740 1 --2- 172.21.15.70:0/298022826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564998fabcb0 0x564998fbdfd0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.161+0000 7f6f5a382740 1 --2- 172.21.15.70:0/298022826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564998fab2c0 0x564998fbda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.161+0000 7f6f5a382740 1 -- 172.21.15.70:0/298022826 >> 172.21.15.70:0/298022826 conn(0x564998f95820 msgr2=0x564998fad390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:24.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.161+0000 7f6f38ff9640 0 client.28664 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:24.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.161+0000 7f6f5a382740 1 -- 172.21.15.70:0/298022826 shutdown_connections 2024-04-01T01:44:24.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.162+0000 7f6f5a382740 1 -- 172.21.15.70:0/298022826 wait complete. 2024-04-01T01:44:24.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.162+0000 7f6f5a382740 20 client.28664 trim_cache size 0 max 16384 2024-04-01T01:44:24.162 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:44:24.162 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:44:24.162 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:44:24.163 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:44:24.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.163+0000 7f6f576cd640 1 -- 172.21.15.70:0/2226441102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f3c0a28e0 msgr2=0x7f6f3c0a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:24.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.163+0000 7f6f576cd640 1 --2- 172.21.15.70:0/2226441102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f3c0a28e0 0x7f6f3c0a4da0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x564998fac510 tx=0x7f6f48004ab0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.163+0000 7f6f576cd640 1 -- 172.21.15.70:0/2226441102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f50076110 msgr2=0x7f6f5019e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:24.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.163+0000 7f6f576cd640 1 --2- 172.21.15.70:0/2226441102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f50076110 0x7f6f5019e000 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009850 tx=0x7f6f4c004270 comp rx=0 tx=0).stop 2024-04-01T01:44:24.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.163+0000 7f6f576cd640 1 -- 172.21.15.70:0/2226441102 shutdown_connections 2024-04-01T01:44:24.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.164+0000 7f6f576cd640 1 --2- 172.21.15.70:0/2226441102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f50077cf0 0x7f6f501a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.164+0000 7f6f576cd640 1 --2- 172.21.15.70:0/2226441102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f50077350 0x7f6f5019e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.164+0000 7f6f576cd640 1 --2- 172.21.15.70:0/2226441102 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f3c0a28e0 0x7f6f3c0a4da0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.164+0000 7f6f576cd640 1 --2- 172.21.15.70:0/2226441102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f50076110 0x7f6f5019e000 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.164+0000 7f6f576cd640 1 -- 172.21.15.70:0/2226441102 >> 172.21.15.70:0/2226441102 conn(0x7f6f500fef00 msgr2=0x7f6f50100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:24.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.164+0000 7f6f576cd640 1 -- 172.21.15.70:0/2226441102 shutdown_connections 2024-04-01T01:44:24.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.164+0000 7f6f576cd640 1 -- 172.21.15.70:0/2226441102 wait complete. 2024-04-01T01:44:24.182 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:44:24.183 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:44:24.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.358+0000 7f776202d640 1 -- 172.21.15.70:0/2518730512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f775c101670 msgr2=0x7f775c101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:24.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.358+0000 7f776202d640 1 --2- 172.21.15.70:0/2518730512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f775c101670 0x7f775c101af0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f774c009900 tx=0x7f774c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:44:24.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.358+0000 7f776202d640 1 -- 172.21.15.70:0/2518730512 shutdown_connections 2024-04-01T01:44:24.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.358+0000 7f776202d640 1 --2- 172.21.15.70:0/2518730512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775c102030 0x7f775c1088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.358+0000 7f776202d640 1 --2- 172.21.15.70:0/2518730512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f775c101670 0x7f775c101af0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.358+0000 7f776202d640 1 --2- 172.21.15.70:0/2518730512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f775c100480 0x7f775c1008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.358+0000 7f776202d640 1 -- 172.21.15.70:0/2518730512 >> 172.21.15.70:0/2518730512 conn(0x7f775c0fb9d0 msgr2=0x7f775c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:24.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.358+0000 7f776202d640 1 -- 172.21.15.70:0/2518730512 shutdown_connections 2024-04-01T01:44:24.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.358+0000 7f776202d640 1 -- 172.21.15.70:0/2518730512 wait complete. 2024-04-01T01:44:24.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.360+0000 7f776202d640 1 Processor -- start 2024-04-01T01:44:24.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.360+0000 7f776202d640 1 -- start start 2024-04-01T01:44:24.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.360+0000 7f776202d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f775c100480 0x7f775c1999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.361+0000 7f776202d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775c101670 0x7f775c199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.361+0000 7f776102b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f775c100480 0x7f775c1999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.361+0000 7f776102b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f775c100480 0x7f775c1999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53446/0 (socket says 172.21.15.70:53446) 2024-04-01T01:44:24.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.361+0000 7f776102b640 1 -- 172.21.15.70:0/3191521012 learned_addr learned my addr 172.21.15.70:0/3191521012 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:24.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.361+0000 7f776202d640 1 --2- 172.21.15.70:0/3191521012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f775c102030 0x7f775c1a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.361+0000 7f776082a640 1 --2- 172.21.15.70:0/3191521012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775c101670 0x7f775c199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.361+0000 7f776202d640 1 -- 172.21.15.70:0/3191521012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f775c10af00 con 0x7f775c101670 2024-04-01T01:44:24.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.361+0000 7f776202d640 1 -- 172.21.15.70:0/3191521012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f775c10ad80 con 0x7f775c100480 2024-04-01T01:44:24.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.361+0000 7f776182c640 1 --2- 172.21.15.70:0/3191521012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f775c102030 0x7f775c1a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.361+0000 7f776202d640 1 -- 172.21.15.70:0/3191521012 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f775c10b080 con 0x7f775c102030 2024-04-01T01:44:24.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.362+0000 7f776102b640 1 -- 172.21.15.70:0/3191521012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f775c102030 msgr2=0x7f775c1a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:24.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.362+0000 7f776102b640 1 --2- 172.21.15.70:0/3191521012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f775c102030 0x7f775c1a0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.362+0000 7f776102b640 1 -- 172.21.15.70:0/3191521012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775c101670 msgr2=0x7f775c199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:24.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.362+0000 7f776102b640 1 --2- 172.21.15.70:0/3191521012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775c101670 0x7f775c199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.362+0000 7f776102b640 1 -- 172.21.15.70:0/3191521012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f775c1a1570 con 0x7f775c100480 2024-04-01T01:44:24.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.362+0000 7f776102b640 1 --2- 172.21.15.70:0/3191521012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f775c100480 0x7f775c1999a0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f775800e9c0 tx=0x7f775800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:24.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.362+0000 7f77527fc640 1 -- 172.21.15.70:0/3191521012 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775800cde0 con 0x7f775c100480 2024-04-01T01:44:24.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.362+0000 7f776202d640 1 -- 172.21.15.70:0/3191521012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f775c1a1860 con 0x7f775c100480 2024-04-01T01:44:24.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.362+0000 7f77527fc640 1 -- 172.21.15.70:0/3191521012 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77580108e0 con 0x7f775c100480 2024-04-01T01:44:24.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.363+0000 7f77527fc640 1 -- 172.21.15.70:0/3191521012 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7758018900 con 0x7f775c100480 2024-04-01T01:44:24.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.363+0000 7f776202d640 1 -- 172.21.15.70:0/3191521012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f775c1a1cc0 con 0x7f775c100480 2024-04-01T01:44:24.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.365+0000 7f776202d640 1 -- 172.21.15.70:0/3191521012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f775c107840 con 0x7f775c100480 2024-04-01T01:44:24.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.368+0000 7f77527fc640 1 -- 172.21.15.70:0/3191521012 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f775801f020 con 0x7f775c100480 2024-04-01T01:44:24.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.369+0000 7f77527fc640 1 --2- 172.21.15.70:0/3191521012 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77480a25a0 0x7f77480a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:24.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.369+0000 7f77527fc640 1 -- 172.21.15.70:0/3191521012 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f77580cee10 con 0x7f775c100480 2024-04-01T01:44:24.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.370+0000 7f776082a640 1 --2- 172.21.15.70:0/3191521012 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77480a25a0 0x7f77480a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:24.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.378+0000 7f77527fc640 1 -- 172.21.15.70:0/3191521012 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f775809bba0 con 0x7f775c100480 2024-04-01T01:44:24.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.380+0000 7f776082a640 1 --2- 172.21.15.70:0/3191521012 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77480a25a0 0x7f77480a4a60 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f774c000c00 tx=0x7f774c02c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:24.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.803+0000 7f776202d640 1 -- 172.21.15.70:0/3191521012 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f775c02d170 con 0x7f775c100480 2024-04-01T01:44:24.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.808+0000 7f77527fc640 1 -- 172.21.15.70:0/3191521012 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7f775809b4a0 con 0x7f775c100480 2024-04-01T01:44:24.808 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:44:24.808 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:44:24.809 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:44:24.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.819+0000 7f776202d640 1 -- 172.21.15.70:0/3191521012 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77480a25a0 msgr2=0x7f77480a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:24.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.819+0000 7f776202d640 1 --2- 172.21.15.70:0/3191521012 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77480a25a0 0x7f77480a4a60 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f774c000c00 tx=0x7f774c02c7e0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.819+0000 7f776202d640 1 -- 172.21.15.70:0/3191521012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f775c100480 msgr2=0x7f775c1999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:24.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.819+0000 7f776202d640 1 --2- 172.21.15.70:0/3191521012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f775c100480 0x7f775c1999a0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f775800e9c0 tx=0x7f775800ee80 comp rx=0 tx=0).stop 2024-04-01T01:44:24.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.819+0000 7f776202d640 1 -- 172.21.15.70:0/3191521012 shutdown_connections 2024-04-01T01:44:24.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.819+0000 7f776202d640 1 --2- 172.21.15.70:0/3191521012 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f77480a25a0 0x7f77480a4a60 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.819+0000 7f776202d640 1 --2- 172.21.15.70:0/3191521012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f775c102030 0x7f775c1a0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.819+0000 7f776202d640 1 --2- 172.21.15.70:0/3191521012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775c101670 0x7f775c199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.819+0000 7f776202d640 1 --2- 172.21.15.70:0/3191521012 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f775c100480 0x7f775c1999a0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:24.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.819+0000 7f776202d640 1 -- 172.21.15.70:0/3191521012 >> 172.21.15.70:0/3191521012 conn(0x7f775c0fb9d0 msgr2=0x7f775c104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:24.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.819+0000 7f776202d640 1 -- 172.21.15.70:0/3191521012 shutdown_connections 2024-04-01T01:44:24.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:24.819+0000 7f776202d640 1 -- 172.21.15.70:0/3191521012 wait complete. 2024-04-01T01:44:24.836 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:44:25.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.012+0000 7f3940007640 1 -- 172.21.15.70:0/2801134805 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f393810a1a0 msgr2=0x7f3938110a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.012+0000 7f3940007640 1 --2- 172.21.15.70:0/2801134805 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f393810a1a0 0x7f3938110a50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f393000adb0 tx=0x7f393002cff0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.012+0000 7f3940007640 1 -- 172.21.15.70:0/2801134805 shutdown_connections 2024-04-01T01:44:25.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.012+0000 7f3940007640 1 --2- 172.21.15.70:0/2801134805 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f393810a1a0 0x7f3938110a50 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.012+0000 7f3940007640 1 --2- 172.21.15.70:0/2801134805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3938109800 0x7f3938109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.012+0000 7f3940007640 1 --2- 172.21.15.70:0/2801134805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3938108b40 0x7f3938108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.012+0000 7f3940007640 1 -- 172.21.15.70:0/2801134805 >> 172.21.15.70:0/2801134805 conn(0x7f393806e950 msgr2=0x7f3938070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.013+0000 7f3940007640 1 -- 172.21.15.70:0/2801134805 shutdown_connections 2024-04-01T01:44:25.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.013+0000 7f3940007640 1 -- 172.21.15.70:0/2801134805 wait complete. 2024-04-01T01:44:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.013+0000 7f3940007640 1 Processor -- start 2024-04-01T01:44:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.013+0000 7f3940007640 1 -- start start 2024-04-01T01:44:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f3940007640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3938108b40 0x7f393819dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f3940007640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3938109800 0x7f393819e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f3940007640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f393810a1a0 0x7f39381a7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f3940007640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3938113890 con 0x7f3938109800 2024-04-01T01:44:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f3940007640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3938113710 con 0x7f3938108b40 2024-04-01T01:44:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f3940007640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3938113a10 con 0x7f393810a1a0 2024-04-01T01:44:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f393dda3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3938108b40 0x7f393819dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f393e5a4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f393810a1a0 0x7f39381a7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f393dda3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3938108b40 0x7f393819dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53470/0 (socket says 172.21.15.70:53470) 2024-04-01T01:44:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f393e5a4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f393810a1a0 0x7f39381a7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:52980/0 (socket says 172.21.15.70:52980) 2024-04-01T01:44:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f393dda3640 1 -- 172.21.15.70:0/1238949044 learned_addr learned my addr 172.21.15.70:0/1238949044 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f393d5a2640 1 --2- 172.21.15.70:0/1238949044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3938109800 0x7f393819e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f393dda3640 1 -- 172.21.15.70:0/1238949044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f393810a1a0 msgr2=0x7f39381a7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f393dda3640 1 --2- 172.21.15.70:0/1238949044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f393810a1a0 0x7f39381a7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f393dda3640 1 -- 172.21.15.70:0/1238949044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3938109800 msgr2=0x7f393819e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f393dda3640 1 --2- 172.21.15.70:0/1238949044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3938109800 0x7f393819e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.014+0000 7f393dda3640 1 -- 172.21.15.70:0/1238949044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39381a1660 con 0x7f3938108b40 2024-04-01T01:44:25.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.015+0000 7f393dda3640 1 --2- 172.21.15.70:0/1238949044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3938108b40 0x7f393819dfa0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f392800e9f0 tx=0x7f392800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.015+0000 7f3936ffd640 1 -- 172.21.15.70:0/1238949044 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392800cd80 con 0x7f3938108b40 2024-04-01T01:44:25.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.015+0000 7f3940007640 1 -- 172.21.15.70:0/1238949044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39381a1950 con 0x7f3938108b40 2024-04-01T01:44:25.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.015+0000 7f3940007640 1 -- 172.21.15.70:0/1238949044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39381a1ea0 con 0x7f3938108b40 2024-04-01T01:44:25.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.015+0000 7f3936ffd640 1 -- 172.21.15.70:0/1238949044 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39280054d0 con 0x7f3938108b40 2024-04-01T01:44:25.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.016+0000 7f3936ffd640 1 -- 172.21.15.70:0/1238949044 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39280186d0 con 0x7f3938108b40 2024-04-01T01:44:25.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.019+0000 7f3942bec740 1 Processor -- start 2024-04-01T01:44:25.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.021+0000 7f3936ffd640 1 -- 172.21.15.70:0/1238949044 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f3928018870 con 0x7f3938108b40 2024-04-01T01:44:25.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.022+0000 7f3936ffd640 1 --2- 172.21.15.70:0/1238949044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39200a2570 0x7f39200a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.022+0000 7f3942bec740 1 -- start start 2024-04-01T01:44:25.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.022+0000 7f3942bec740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c2328490 0x5582c23288b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.022+0000 7f3936ffd640 1 -- 172.21.15.70:0/1238949044 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f39280cedc0 con 0x7f3938108b40 2024-04-01T01:44:25.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.022+0000 7f3942bec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328e80 0x5582c232d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.022+0000 7f3942bec740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c232dff0 0x5582c2330480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.022+0000 7f3942bec740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5582c228d1b0 con 0x5582c232dff0 2024-04-01T01:44:25.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.022+0000 7f3942bec740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5582c2235c00 con 0x5582c2328490 2024-04-01T01:44:25.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.023+0000 7f3942bec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5582c2289250 con 0x5582c2328e80 2024-04-01T01:44:25.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.023+0000 7f393dda3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c2328490 0x5582c23288b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.023+0000 7f393e5a4640 1 --2- 172.21.15.70:0/1238949044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39200a2570 0x7f39200a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.023+0000 7f393dda3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c2328490 0x5582c23288b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53486/0 (socket says 172.21.15.70:53486) 2024-04-01T01:44:25.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.023+0000 7f393dda3640 1 -- 172.21.15.70:0/2647903392 learned_addr learned my addr 172.21.15.70:0/2647903392 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.024+0000 7f393d5a2640 1 --2- 172.21.15.70:0/2647903392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328e80 0x5582c232d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.024+0000 7f393dda3640 1 -- 172.21.15.70:0/2647903392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328e80 msgr2=0x5582c232d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.024+0000 7f393dda3640 1 --2- 172.21.15.70:0/2647903392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328e80 0x5582c232d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.024+0000 7f393dda3640 1 -- 172.21.15.70:0/2647903392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c232dff0 msgr2=0x5582c2330480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.024+0000 7f393dda3640 1 --2- 172.21.15.70:0/2647903392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c232dff0 0x5582c2330480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.024+0000 7f393dda3640 1 -- 172.21.15.70:0/2647903392 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582c22553d0 con 0x5582c2328490 2024-04-01T01:44:25.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.024+0000 7f393d5a2640 1 --2- 172.21.15.70:0/2647903392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328e80 0x5582c232d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:44:25.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.024+0000 7f393e5a4640 1 --2- 172.21.15.70:0/1238949044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39200a2570 0x7f39200a4a30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3930033410 tx=0x7f393002f040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.025+0000 7f393dda3640 1 --2- 172.21.15.70:0/2647903392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c2328490 0x5582c23288b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f392809b1b0 tx=0x7f392809b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce2cb464793ef686 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.025+0000 7f391ffff640 1 -- 172.21.15.70:0/2647903392 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392809f600 con 0x5582c2328490 2024-04-01T01:44:25.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.025+0000 7f391ffff640 1 -- 172.21.15.70:0/2647903392 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39280a3ec0 con 0x5582c2328490 2024-04-01T01:44:25.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.025+0000 7f391ffff640 1 -- 172.21.15.70:0/2647903392 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39280bb0c0 con 0x5582c2328490 2024-04-01T01:44:25.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.026+0000 7f3942bec740 1 -- 172.21.15.70:0/2647903392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c2328490 msgr2=0x5582c23288b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.026+0000 7f3942bec740 1 --2- 172.21.15.70:0/2647903392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c2328490 0x5582c23288b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f392809b1b0 tx=0x7f392809b3b0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.026+0000 7f3942bec740 1 -- 172.21.15.70:0/2647903392 shutdown_connections 2024-04-01T01:44:25.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.026+0000 7f3942bec740 1 --2- 172.21.15.70:0/2647903392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c232dff0 0x5582c2330480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.026+0000 7f3942bec740 1 --2- 172.21.15.70:0/2647903392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328e80 0x5582c232d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.026+0000 7f3942bec740 1 --2- 172.21.15.70:0/2647903392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c2328490 0x5582c23288b0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.026+0000 7f3942bec740 1 -- 172.21.15.70:0/2647903392 >> 172.21.15.70:0/2647903392 conn(0x5582c2323a00 msgr2=0x5582c2323e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.027+0000 7f3942bec740 1 -- 172.21.15.70:0/2647903392 shutdown_connections 2024-04-01T01:44:25.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.027+0000 7f3942bec740 1 -- 172.21.15.70:0/2647903392 wait complete. 2024-04-01T01:44:25.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.027+0000 7f3942bec740 1 -- start start 2024-04-01T01:44:25.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.027+0000 7f3942bec740 1 Processor -- start 2024-04-01T01:44:25.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.028+0000 7f3942bec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328490 0x5582c233b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.028+0000 7f3942bec740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c2328e80 0x5582c233b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.028+0000 7f3942bec740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c232dff0 0x5582c2342840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.028+0000 7f3942bec740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5582c23397b0 con 0x5582c2328e80 2024-04-01T01:44:25.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.028+0000 7f3942bec740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5582c222b470 con 0x5582c232dff0 2024-04-01T01:44:25.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.028+0000 7f3942bec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5582c2339930 con 0x5582c2328490 2024-04-01T01:44:25.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.028+0000 7f393e5a4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c232dff0 0x5582c2342840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.028+0000 7f393e5a4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c232dff0 0x5582c2342840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53502/0 (socket says 172.21.15.70:53502) 2024-04-01T01:44:25.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.028+0000 7f393e5a4640 1 -- 172.21.15.70:0/305524063 learned_addr learned my addr 172.21.15.70:0/305524063 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.028+0000 7f3942bec740 10 client.0 _ll_register_callbacks cb 0x5582c2248cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.028+0000 7f393d5a2640 1 --2- 172.21.15.70:0/305524063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c2328e80 0x5582c233b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.028+0000 7f393dda3640 1 --2- 172.21.15.70:0/305524063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328490 0x5582c233b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.028+0000 7f393e5a4640 1 -- 172.21.15.70:0/305524063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328490 msgr2=0x5582c233b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.029+0000 7f393e5a4640 1 --2- 172.21.15.70:0/305524063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328490 0x5582c233b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.029+0000 7f393e5a4640 1 -- 172.21.15.70:0/305524063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c2328e80 msgr2=0x5582c233b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.029+0000 7f393e5a4640 1 --2- 172.21.15.70:0/305524063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c2328e80 0x5582c233b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.029+0000 7f393e5a4640 1 -- 172.21.15.70:0/305524063 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582c1fa13f0 con 0x5582c232dff0 2024-04-01T01:44:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.029+0000 7f393d5a2640 1 --2- 172.21.15.70:0/305524063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c2328e80 0x5582c233b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:44:25.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.029+0000 7f393dda3640 1 --2- 172.21.15.70:0/305524063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328490 0x5582c233b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:44:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.029+0000 7f393e5a4640 1 --2- 172.21.15.70:0/305524063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c232dff0 0x5582c2342840 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f3930005e30 tx=0x7f393002dba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=520f330f7f870bc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.029+0000 7f391d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:44:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.029+0000 7f391d7fa640 1 -- 172.21.15.70:0/305524063 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393004f020 con 0x5582c232dff0 2024-04-01T01:44:25.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.029+0000 7f391d7fa640 1 -- 172.21.15.70:0/305524063 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f39300356e0 con 0x5582c232dff0 2024-04-01T01:44:25.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.030+0000 7f3942bec740 1 -- 172.21.15.70:0/305524063 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39280a3ec0 con 0x5582c232dff0 2024-04-01T01:44:25.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.030+0000 7f391d7fa640 1 -- 172.21.15.70:0/305524063 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f393003a040 con 0x5582c232dff0 2024-04-01T01:44:25.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.030+0000 7f3942bec740 10 client.28682 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:25.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.030+0000 7f3942bec740 1 -- 172.21.15.70:0/305524063 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5582c206b7d0 con 0x5582c232dff0 2024-04-01T01:44:25.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.030+0000 7f391d7fa640 1 -- 172.21.15.70:0/305524063 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f39300334b0 con 0x5582c232dff0 2024-04-01T01:44:25.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.031+0000 7f391d7fa640 1 -- 172.21.15.70:0/305524063 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f3930040040 con 0x5582c232dff0 2024-04-01T01:44:25.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.031+0000 7f3942bec740 10 client.28682 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:25.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.031+0000 7f3942bec740 10 client.28682 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:44:25.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.031+0000 7f3942bec740 20 client.28682 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:25.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.031+0000 7f3942bec740 1 --2- 172.21.15.70:0/305524063 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5582c2346cc0 0x5582c2349140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.032+0000 7f3942bec740 4 client.28682 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:25.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.032+0000 7f3942bec740 1 -- 172.21.15.70:0/305524063 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f39280a3ec0 con 0x5582c2346cc0 2024-04-01T01:44:25.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.032+0000 7f393dda3640 1 --2- 172.21.15.70:0/305524063 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5582c2346cc0 0x5582c2349140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.033+0000 7f391d7fa640 10 client.28682 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:44:25.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.033+0000 7f393dda3640 1 --2- 172.21.15.70:0/305524063 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5582c2346cc0 0x5582c2349140 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4a8d4a4ad9dd2dc0 server_cookie=1587e42fc30cdd9d in_seq=0 out_seq=0 2024-04-01T01:44:25.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.034+0000 7f391d7fa640 1 -- 172.21.15.70:0/305524063 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f39280a3ec0 con 0x5582c2346cc0 2024-04-01T01:44:25.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.034+0000 7f391d7fa640 10 client.28682 handle_command_reply: tid=0 2024-04-01T01:44:25.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.035+0000 7f3942bec740 1 client.28682 shutdown 2024-04-01T01:44:25.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.035+0000 7f3942bec740 1 -- 172.21.15.70:0/305524063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c232dff0 msgr2=0x5582c2342840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.036+0000 7f3942bec740 1 --2- 172.21.15.70:0/305524063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c232dff0 0x5582c2342840 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f3930005e30 tx=0x7f393002dba0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.037+0000 7f3942bec740 1 -- 172.21.15.70:0/305524063 shutdown_connections 2024-04-01T01:44:25.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.037+0000 7f3942bec740 1 --2- 172.21.15.70:0/305524063 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5582c2346cc0 0x5582c2349140 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.037+0000 7f3942bec740 1 --2- 172.21.15.70:0/305524063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c232dff0 0x5582c2342840 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.037+0000 7f3942bec740 1 --2- 172.21.15.70:0/305524063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c2328e80 0x5582c233b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.037+0000 7f3942bec740 1 --2- 172.21.15.70:0/305524063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328490 0x5582c233b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.037+0000 7f3942bec740 1 -- 172.21.15.70:0/305524063 >> 172.21.15.70:0/305524063 conn(0x5582c2323a00 msgr2=0x5582c2322d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.037+0000 7f391d7fa640 0 client.28682 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:44:25.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.037+0000 7f3942bec740 1 -- 172.21.15.70:0/305524063 shutdown_connections 2024-04-01T01:44:25.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.037+0000 7f3942bec740 1 -- 172.21.15.70:0/305524063 wait complete. 2024-04-01T01:44:25.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.038+0000 7f3942bec740 20 client.28682 trim_cache size 0 max 16384 2024-04-01T01:44:25.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.063+0000 7f3942bec740 1 Processor -- start 2024-04-01T01:44:25.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.063+0000 7f3942bec740 1 -- start start 2024-04-01T01:44:25.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.063+0000 7f3942bec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328490 0x5582c23224b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.063+0000 7f3942bec740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c2328e80 0x5582c2342800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.063+0000 7f3942bec740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c232dff0 0x5582c23229f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.063+0000 7f3942bec740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f392400fd40 con 0x5582c2328e80 2024-04-01T01:44:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.064+0000 7f393d5a2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c2328e80 0x5582c2342800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.064+0000 7f393d5a2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c2328e80 0x5582c2342800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35630/0 (socket says 172.21.15.70:35630) 2024-04-01T01:44:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.064+0000 7f393d5a2640 1 -- 172.21.15.70:0/1610166245 learned_addr learned my addr 172.21.15.70:0/1610166245 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.064+0000 7f393dda3640 1 --2- 172.21.15.70:0/1610166245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328490 0x5582c23224b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.064+0000 7f3942bec740 1 -- 172.21.15.70:0/1610166245 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3924011880 con 0x5582c232dff0 2024-04-01T01:44:25.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.065+0000 7f393e5a4640 1 --2- 172.21.15.70:0/1610166245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c232dff0 0x5582c23229f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.065+0000 7f3942bec740 1 -- 172.21.15.70:0/1610166245 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3924010410 con 0x5582c2328490 2024-04-01T01:44:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.065+0000 7f393e5a4640 1 -- 172.21.15.70:0/1610166245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328490 msgr2=0x5582c23224b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.065+0000 7f393e5a4640 1 --2- 172.21.15.70:0/1610166245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328490 0x5582c23224b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.065+0000 7f393e5a4640 1 -- 172.21.15.70:0/1610166245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c2328e80 msgr2=0x5582c2342800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.065+0000 7f393e5a4640 1 --2- 172.21.15.70:0/1610166245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c2328e80 0x5582c2342800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.065+0000 7f393e5a4640 1 -- 172.21.15.70:0/1610166245 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582c232dd10 con 0x5582c232dff0 2024-04-01T01:44:25.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.066+0000 7f393e5a4640 1 --2- 172.21.15.70:0/1610166245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c232dff0 0x5582c23229f0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3930035fa0 tx=0x7f393002dd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbbd84cabfbf1172 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.066+0000 7f391f7fe640 1 -- 172.21.15.70:0/1610166245 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393004f070 con 0x5582c232dff0 2024-04-01T01:44:25.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.066+0000 7f391f7fe640 1 -- 172.21.15.70:0/1610166245 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3930035b90 con 0x5582c232dff0 2024-04-01T01:44:25.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.066+0000 7f391f7fe640 1 -- 172.21.15.70:0/1610166245 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39300334b0 con 0x5582c232dff0 2024-04-01T01:44:25.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.067+0000 7f3942bec740 1 -- 172.21.15.70:0/1610166245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c232dff0 msgr2=0x5582c23229f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.067+0000 7f3942bec740 1 --2- 172.21.15.70:0/1610166245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c232dff0 0x5582c23229f0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3930035fa0 tx=0x7f393002dd50 comp rx=0 tx=0).stop 2024-04-01T01:44:25.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.067+0000 7f3942bec740 1 -- 172.21.15.70:0/1610166245 shutdown_connections 2024-04-01T01:44:25.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.067+0000 7f3942bec740 1 --2- 172.21.15.70:0/1610166245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c232dff0 0x5582c23229f0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.067+0000 7f3942bec740 1 --2- 172.21.15.70:0/1610166245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c2328e80 0x5582c2342800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.067+0000 7f3942bec740 1 --2- 172.21.15.70:0/1610166245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c2328490 0x5582c23224b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.067+0000 7f3942bec740 1 -- 172.21.15.70:0/1610166245 >> 172.21.15.70:0/1610166245 conn(0x5582c2323a00 msgr2=0x5582c232a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.067+0000 7f3942bec740 1 -- 172.21.15.70:0/1610166245 shutdown_connections 2024-04-01T01:44:25.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.068+0000 7f3942bec740 1 -- 172.21.15.70:0/1610166245 wait complete. 2024-04-01T01:44:25.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.068+0000 7f3942bec740 1 -- start start 2024-04-01T01:44:25.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.068+0000 7f3942bec740 1 Processor -- start 2024-04-01T01:44:25.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.069+0000 7f3942bec740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c2328490 0x5582c233fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.069+0000 7f3942bec740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c2328e80 0x5582c23402b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.069+0000 7f3942bec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c232dff0 0x5582c23412c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.069+0000 7f3942bec740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39240111b0 con 0x5582c2328e80 2024-04-01T01:44:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.069+0000 7f3942bec740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3924010ae0 con 0x5582c2328490 2024-04-01T01:44:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.069+0000 7f3942bec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f392400dd20 con 0x5582c232dff0 2024-04-01T01:44:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.069+0000 7f3942bec740 10 client.0 _ll_register_callbacks cb 0x7f393000ac30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.069+0000 7f393dda3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c2328490 0x5582c233fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.069+0000 7f393e5a4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c232dff0 0x5582c23412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.070+0000 7f393e5a4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c232dff0 0x5582c23412c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:53004/0 (socket says 172.21.15.70:53004) 2024-04-01T01:44:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.070+0000 7f393e5a4640 1 -- 172.21.15.70:0/4173539999 learned_addr learned my addr 172.21.15.70:0/4173539999 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.070+0000 7f393e5a4640 1 -- 172.21.15.70:0/4173539999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c2328490 msgr2=0x5582c233fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.070+0000 7f393e5a4640 1 --2- 172.21.15.70:0/4173539999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c2328490 0x5582c233fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.070+0000 7f393e5a4640 1 -- 172.21.15.70:0/4173539999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c2328e80 msgr2=0x5582c23402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.070+0000 7f393e5a4640 1 --2- 172.21.15.70:0/4173539999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c2328e80 0x5582c23402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.070+0000 7f393e5a4640 1 -- 172.21.15.70:0/4173539999 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582c23418b0 con 0x5582c232dff0 2024-04-01T01:44:25.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.073+0000 7f393e5a4640 1 --2- 172.21.15.70:0/4173539999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c232dff0 0x5582c23412c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3930035e10 tx=0x7f39300354e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6535154f6b449b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.073 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.074+0000 7f391d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:44:25.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.074+0000 7f391d7fa640 1 -- 172.21.15.70:0/4173539999 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393004f040 con 0x5582c232dff0 2024-04-01T01:44:25.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.074+0000 7f391d7fa640 1 -- 172.21.15.70:0/4173539999 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f39300336b0 con 0x5582c232dff0 2024-04-01T01:44:25.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.074+0000 7f3942bec740 1 -- 172.21.15.70:0/4173539999 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3930035b90 con 0x5582c232dff0 2024-04-01T01:44:25.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.075+0000 7f391d7fa640 1 -- 172.21.15.70:0/4173539999 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f39300359a0 con 0x5582c232dff0 2024-04-01T01:44:25.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.077+0000 7f3942bec740 10 client.28468 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:25.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.077+0000 7f3942bec740 1 -- 172.21.15.70:0/4173539999 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5582c23ad050 con 0x5582c232dff0 2024-04-01T01:44:25.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.077+0000 7f391d7fa640 1 -- 172.21.15.70:0/4173539999 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3930035b90 con 0x5582c232dff0 2024-04-01T01:44:25.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.077+0000 7f391d7fa640 1 -- 172.21.15.70:0/4173539999 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f3930040040 con 0x5582c232dff0 2024-04-01T01:44:25.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.080+0000 7f3942bec740 10 client.28468 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:25.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.080+0000 7f3942bec740 10 client.28468 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:44:25.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.080+0000 7f3942bec740 20 client.28468 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:25.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.080+0000 7f3942bec740 1 --2- 172.21.15.70:0/4173539999 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5582c23ad580 0x5582c23af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.080+0000 7f3942bec740 4 client.28468 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:44:25.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.080+0000 7f3942bec740 1 -- 172.21.15.70:0/4173539999 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5582c229a6f0 con 0x5582c23ad580 2024-04-01T01:44:25.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.081+0000 7f393dda3640 1 --2- 172.21.15.70:0/4173539999 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5582c23ad580 0x5582c23af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.081+0000 7f393dda3640 1 --2- 172.21.15.70:0/4173539999 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5582c23ad580 0x5582c23af9a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c9c8846eefbe748e server_cookie=ac715396d68b2cb9 in_seq=0 out_seq=0 2024-04-01T01:44:25.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.082+0000 7f391d7fa640 10 client.28468 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:44:25.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.082+0000 7f391d7fa640 1 -- 172.21.15.70:0/4173539999 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5582c229a6f0 con 0x5582c23ad580 2024-04-01T01:44:25.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.082+0000 7f391d7fa640 10 client.28468 handle_command_reply: tid=0 2024-04-01T01:44:25.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.082+0000 7f3942bec740 1 client.28468 shutdown 2024-04-01T01:44:25.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.082+0000 7f3942bec740 1 -- 172.21.15.70:0/4173539999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c232dff0 msgr2=0x5582c23412c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.082+0000 7f3942bec740 1 --2- 172.21.15.70:0/4173539999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c232dff0 0x5582c23412c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3930035e10 tx=0x7f39300354e0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.088+0000 7f3942bec740 1 -- 172.21.15.70:0/4173539999 shutdown_connections 2024-04-01T01:44:25.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.088+0000 7f3942bec740 1 --2- 172.21.15.70:0/4173539999 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5582c23ad580 0x5582c23af9a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.088+0000 7f3942bec740 1 --2- 172.21.15.70:0/4173539999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582c232dff0 0x5582c23412c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.088+0000 7f391d7fa640 0 client.28468 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:44:25.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.088+0000 7f3942bec740 1 --2- 172.21.15.70:0/4173539999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5582c2328e80 0x5582c23402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.088+0000 7f3942bec740 1 --2- 172.21.15.70:0/4173539999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5582c2328490 0x5582c233fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.088+0000 7f3942bec740 1 -- 172.21.15.70:0/4173539999 >> 172.21.15.70:0/4173539999 conn(0x5582c2323a00 msgr2=0x5582c2326c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.088+0000 7f3942bec740 1 -- 172.21.15.70:0/4173539999 shutdown_connections 2024-04-01T01:44:25.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.088+0000 7f3942bec740 1 -- 172.21.15.70:0/4173539999 wait complete. 2024-04-01T01:44:25.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.088+0000 7f3942bec740 20 client.28468 trim_cache size 0 max 16384 2024-04-01T01:44:25.089 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:44:25.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.089+0000 7f3940007640 1 -- 172.21.15.70:0/1238949044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39200a2570 msgr2=0x7f39200a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.090+0000 7f3940007640 1 --2- 172.21.15.70:0/1238949044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39200a2570 0x7f39200a4a30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3930033410 tx=0x7f393002f040 comp rx=0 tx=0).stop 2024-04-01T01:44:25.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.090+0000 7f3940007640 1 -- 172.21.15.70:0/1238949044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3938108b40 msgr2=0x7f393819dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.090+0000 7f3940007640 1 --2- 172.21.15.70:0/1238949044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3938108b40 0x7f393819dfa0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f392800e9f0 tx=0x7f392800edf0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.090+0000 7f3940007640 1 -- 172.21.15.70:0/1238949044 shutdown_connections 2024-04-01T01:44:25.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.090+0000 7f3940007640 1 --2- 172.21.15.70:0/1238949044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f39200a2570 0x7f39200a4a30 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.090+0000 7f3940007640 1 --2- 172.21.15.70:0/1238949044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f393810a1a0 0x7f39381a7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.090+0000 7f3940007640 1 --2- 172.21.15.70:0/1238949044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3938109800 0x7f393819e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.090+0000 7f3940007640 1 --2- 172.21.15.70:0/1238949044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3938108b40 0x7f393819dfa0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.090+0000 7f3940007640 1 -- 172.21.15.70:0/1238949044 >> 172.21.15.70:0/1238949044 conn(0x7f393806e950 msgr2=0x7f393810cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.090+0000 7f3940007640 1 -- 172.21.15.70:0/1238949044 shutdown_connections 2024-04-01T01:44:25.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.090+0000 7f3940007640 1 -- 172.21.15.70:0/1238949044 wait complete. 2024-04-01T01:44:25.108 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:24 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3191521012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:44:25.109 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:44:25.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:24 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3191521012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:44:25.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.303+0000 7fab09165640 1 -- 172.21.15.70:0/2905072826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab04072b60 msgr2=0x7fab04072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.303+0000 7fab09165640 1 --2- 172.21.15.70:0/2905072826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab04072b60 0x7fab04072f80 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7faaf8009900 tx=0x7faaf802bc30 comp rx=0 tx=0).stop 2024-04-01T01:44:25.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.304+0000 7fab09165640 1 -- 172.21.15.70:0/2905072826 shutdown_connections 2024-04-01T01:44:25.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.304+0000 7fab09165640 1 --2- 172.21.15.70:0/2905072826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab041097e0 0x7fab0410e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.304+0000 7fab09165640 1 --2- 172.21.15.70:0/2905072826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab04108e20 0x7fab041092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.304+0000 7fab09165640 1 --2- 172.21.15.70:0/2905072826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab04072b60 0x7fab04072f80 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.304+0000 7fab09165640 1 -- 172.21.15.70:0/2905072826 >> 172.21.15.70:0/2905072826 conn(0x7fab0406e630 msgr2=0x7fab04070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.304+0000 7fab09165640 1 -- 172.21.15.70:0/2905072826 shutdown_connections 2024-04-01T01:44:25.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.304+0000 7fab09165640 1 -- 172.21.15.70:0/2905072826 wait complete. 2024-04-01T01:44:25.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.305+0000 7fab09165640 1 Processor -- start 2024-04-01T01:44:25.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.305+0000 7fab09165640 1 -- start start 2024-04-01T01:44:25.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.305+0000 7fab09165640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab04072b60 0x7fab04114e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.305+0000 7fab09165640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab04108e20 0x7fab041153a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.305+0000 7fab09165640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab041097e0 0x7fab041b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.305+0000 7fab09165640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab0411bfe0 con 0x7fab04072b60 2024-04-01T01:44:25.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.305+0000 7fab09165640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fab0411be60 con 0x7fab04108e20 2024-04-01T01:44:25.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.305+0000 7fab09165640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fab0411c160 con 0x7fab041097e0 2024-04-01T01:44:25.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.305+0000 7fab037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab04108e20 0x7fab041153a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.305+0000 7fab037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab04108e20 0x7fab041153a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53534/0 (socket says 172.21.15.70:53534) 2024-04-01T01:44:25.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.305+0000 7fab037fe640 1 -- 172.21.15.70:0/908277642 learned_addr learned my addr 172.21.15.70:0/908277642 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.305+0000 7fab08964640 1 --2- 172.21.15.70:0/908277642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab041097e0 0x7fab041b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.306+0000 7fab03fff640 1 --2- 172.21.15.70:0/908277642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab04072b60 0x7fab04114e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.306+0000 7fab08964640 1 -- 172.21.15.70:0/908277642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab04108e20 msgr2=0x7fab041153a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.306+0000 7fab08964640 1 --2- 172.21.15.70:0/908277642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab04108e20 0x7fab041153a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.306+0000 7fab08964640 1 -- 172.21.15.70:0/908277642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab04072b60 msgr2=0x7fab04114e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.306+0000 7fab08964640 1 --2- 172.21.15.70:0/908277642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab04072b60 0x7fab04114e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.306+0000 7fab08964640 1 -- 172.21.15.70:0/908277642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab041b6bf0 con 0x7fab041097e0 2024-04-01T01:44:25.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.306+0000 7fab037fe640 1 --2- 172.21.15.70:0/908277642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab04108e20 0x7fab041153a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:25.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.310+0000 7fab08964640 1 --2- 172.21.15.70:0/908277642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab041097e0 0x7fab041b66b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7faaf400e970 tx=0x7faaf400ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.312+0000 7fab017fa640 1 -- 172.21.15.70:0/908277642 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf400ccc0 con 0x7fab041097e0 2024-04-01T01:44:25.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.312+0000 7fab017fa640 1 -- 172.21.15.70:0/908277642 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faaf4005020 con 0x7fab041097e0 2024-04-01T01:44:25.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.313+0000 7fab017fa640 1 -- 172.21.15.70:0/908277642 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf4004a00 con 0x7fab041097e0 2024-04-01T01:44:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.313+0000 7fab09165640 1 -- 172.21.15.70:0/908277642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab041b6dc0 con 0x7fab041097e0 2024-04-01T01:44:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.313+0000 7fab09165640 1 -- 172.21.15.70:0/908277642 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab041b7280 con 0x7fab041097e0 2024-04-01T01:44:25.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.318+0000 7fab017fa640 1 -- 172.21.15.70:0/908277642 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7faaf4004ba0 con 0x7fab041097e0 2024-04-01T01:44:25.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.318+0000 7fab0cfa1740 1 Processor -- start 2024-04-01T01:44:25.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.322+0000 7fab0cfa1740 1 -- start start 2024-04-01T01:44:25.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.322+0000 7fab0cfa1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d076a6a0 0x5603d076aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.322+0000 7fab0cfa1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076b090 0x5603d076fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.322+0000 7fab0cfa1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d0770200 0x5603d0772690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.322+0000 7fab0cfa1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5603d06cf1b0 con 0x5603d076a6a0 2024-04-01T01:44:25.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.322+0000 7fab0cfa1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5603d0677c00 con 0x5603d076b090 2024-04-01T01:44:25.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.322+0000 7fab0cfa1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5603d06cb250 con 0x5603d0770200 2024-04-01T01:44:25.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.323+0000 7fab017fa640 1 --2- 172.21.15.70:0/908277642 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faaec0a2410 0x7faaec0a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.323+0000 7fab03fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076b090 0x5603d076fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.323+0000 7fab017fa640 1 -- 172.21.15.70:0/908277642 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7faaf40cefb0 con 0x7fab041097e0 2024-04-01T01:44:25.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.323+0000 7fab03fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076b090 0x5603d076fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53546/0 (socket says 172.21.15.70:53546) 2024-04-01T01:44:25.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.323+0000 7fab03fff640 1 -- 172.21.15.70:0/731737435 learned_addr learned my addr 172.21.15.70:0/731737435 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.323+0000 7fab037fe640 1 --2- 172.21.15.70:0/731737435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d076a6a0 0x5603d076aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.323+0000 7fab037fe640 1 --2- 172.21.15.70:0/731737435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d0770200 0x5603d0772690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.324+0000 7fab03fff640 1 -- 172.21.15.70:0/731737435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d0770200 msgr2=0x5603d0772690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.324+0000 7fab03fff640 1 --2- 172.21.15.70:0/731737435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d0770200 0x5603d0772690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.324+0000 7fab03fff640 1 -- 172.21.15.70:0/731737435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d076a6a0 msgr2=0x5603d076aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.324+0000 7fab03fff640 1 --2- 172.21.15.70:0/731737435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d076a6a0 0x5603d076aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.324+0000 7fab03fff640 1 -- 172.21.15.70:0/731737435 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603d06973d0 con 0x5603d076b090 2024-04-01T01:44:25.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.324+0000 7fab037fe640 1 --2- 172.21.15.70:0/731737435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d0770200 0x5603d0772690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.326+0000 7fab03fff640 1 --2- 172.21.15.70:0/731737435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076b090 0x5603d076fb90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7faaf802c1c0 tx=0x7faaf80095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=351e8dadfd563b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.326+0000 7fab08964640 1 --2- 172.21.15.70:0/908277642 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faaec0a2410 0x7faaec0a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.326+0000 7faaea7fc640 1 -- 172.21.15.70:0/731737435 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf8044020 con 0x5603d076b090 2024-04-01T01:44:25.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.326+0000 7faaea7fc640 1 -- 172.21.15.70:0/731737435 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faaf802cc90 con 0x5603d076b090 2024-04-01T01:44:25.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.327+0000 7fab0cfa1740 1 -- 172.21.15.70:0/731737435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076b090 msgr2=0x5603d076fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.327+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/731737435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076b090 0x5603d076fb90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7faaf802c1c0 tx=0x7faaf80095f0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:24 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3191521012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:44:25.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.329+0000 7fab0cfa1740 1 -- 172.21.15.70:0/731737435 shutdown_connections 2024-04-01T01:44:25.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.329+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/731737435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d0770200 0x5603d0772690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.329+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/731737435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076b090 0x5603d076fb90 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.329+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/731737435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d076a6a0 0x5603d076aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.329+0000 7fab0cfa1740 1 -- 172.21.15.70:0/731737435 >> 172.21.15.70:0/731737435 conn(0x5603d0765c10 msgr2=0x5603d0766010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.331+0000 7fab08964640 1 --2- 172.21.15.70:0/908277642 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faaec0a2410 0x7faaec0a48d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7faaf4004770 tx=0x7faaf4097220 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.332+0000 7fab0cfa1740 1 -- 172.21.15.70:0/731737435 shutdown_connections 2024-04-01T01:44:25.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.333+0000 7fab0cfa1740 1 -- 172.21.15.70:0/731737435 wait complete. 2024-04-01T01:44:25.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.334+0000 7fab0cfa1740 1 -- start start 2024-04-01T01:44:25.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.334+0000 7fab0cfa1740 1 Processor -- start 2024-04-01T01:44:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.334+0000 7fab0cfa1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d076a6a0 0x5603d0774b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.334+0000 7fab0cfa1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076b090 0x5603d07750a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.334+0000 7fab0cfa1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d0770200 0x5603d077c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.334+0000 7fab0cfa1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5603d07731f0 con 0x5603d0770200 2024-04-01T01:44:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.334+0000 7fab0cfa1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5603d066d470 con 0x5603d076b090 2024-04-01T01:44:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.334+0000 7fab0cfa1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5603d0773370 con 0x5603d076a6a0 2024-04-01T01:44:25.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.334+0000 7fab0cfa1740 10 client.0 _ll_register_callbacks cb 0x5603d068acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:25.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.335+0000 7fab037fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d076a6a0 0x5603d0774b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.335+0000 7fab037fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d076a6a0 0x5603d0774b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:53044/0 (socket says 172.21.15.70:53044) 2024-04-01T01:44:25.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.335+0000 7fab037fe640 1 -- 172.21.15.70:0/1244975442 learned_addr learned my addr 172.21.15.70:0/1244975442 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.335+0000 7fab03fff640 1 --2- 172.21.15.70:0/1244975442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076b090 0x5603d07750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.335+0000 7fab037fe640 1 --2- 172.21.15.70:0/1244975442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d0770200 0x5603d077c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.336+0000 7fab037fe640 1 -- 172.21.15.70:0/1244975442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076b090 msgr2=0x5603d07750a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.336+0000 7fab037fe640 1 --2- 172.21.15.70:0/1244975442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076b090 0x5603d07750a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.336+0000 7fab037fe640 1 -- 172.21.15.70:0/1244975442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d0770200 msgr2=0x5603d077c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.336+0000 7fab037fe640 1 --2- 172.21.15.70:0/1244975442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d0770200 0x5603d077c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.336+0000 7fab037fe640 1 -- 172.21.15.70:0/1244975442 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603d03e33f0 con 0x5603d076a6a0 2024-04-01T01:44:25.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.336+0000 7fab03fff640 1 --2- 172.21.15.70:0/1244975442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076b090 0x5603d07750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:25.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.336+0000 7fab037fe640 1 --2- 172.21.15.70:0/1244975442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d076a6a0 0x5603d0774b60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7faafc011a00 tx=0x7faafc004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12cd65eb8a874d83 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.336+0000 7faae37fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:44:25.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.337+0000 7faae37fe640 1 -- 172.21.15.70:0/1244975442 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faafc01c040 con 0x5603d076a6a0 2024-04-01T01:44:25.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.337+0000 7faae37fe640 1 -- 172.21.15.70:0/1244975442 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7faafc003b80 con 0x5603d076a6a0 2024-04-01T01:44:25.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.337+0000 7fab0cfa1740 1 -- 172.21.15.70:0/1244975442 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faaf802cc90 con 0x5603d076a6a0 2024-04-01T01:44:25.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.337+0000 7faae37fe640 1 -- 172.21.15.70:0/1244975442 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faafc003d80 con 0x5603d076a6a0 2024-04-01T01:44:25.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.337+0000 7fab0cfa1740 10 client.28480 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:25.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.337+0000 7fab0cfa1740 1 -- 172.21.15.70:0/1244975442 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5603d04ad7d0 con 0x5603d076a6a0 2024-04-01T01:44:25.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.337+0000 7faae37fe640 1 -- 172.21.15.70:0/1244975442 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faaf802cc90 con 0x5603d076a6a0 2024-04-01T01:44:25.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.338+0000 7faae37fe640 1 -- 172.21.15.70:0/1244975442 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7faafc004540 con 0x5603d076a6a0 2024-04-01T01:44:25.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.338+0000 7fab0cfa1740 10 client.28480 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:25.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.338+0000 7fab0cfa1740 10 client.28480 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:44:25.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.338+0000 7fab0cfa1740 20 client.28480 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:25.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.338+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/1244975442 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5603d0780660 0x5603d0782b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.338+0000 7fab0cfa1740 4 client.28480 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:25.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.338+0000 7fab0cfa1740 1 -- 172.21.15.70:0/1244975442 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603d06dc6f0 con 0x5603d0780660 2024-04-01T01:44:25.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.338+0000 7fab03fff640 1 --2- 172.21.15.70:0/1244975442 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5603d0780660 0x5603d0782b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.339+0000 7fab03fff640 1 --2- 172.21.15.70:0/1244975442 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5603d0780660 0x5603d0782b00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cc9cb583563224e2 server_cookie=d249518ecdf12887 in_seq=0 out_seq=0 2024-04-01T01:44:25.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.339+0000 7faae37fe640 10 client.28480 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:44:25.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.340+0000 7faae37fe640 1 -- 172.21.15.70:0/1244975442 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5603d06dc6f0 con 0x5603d0780660 2024-04-01T01:44:25.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.340+0000 7faae37fe640 10 client.28480 handle_command_reply: tid=0 2024-04-01T01:44:25.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.340+0000 7fab0cfa1740 1 client.28480 shutdown 2024-04-01T01:44:25.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.340+0000 7fab0cfa1740 1 -- 172.21.15.70:0/1244975442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d076a6a0 msgr2=0x5603d0774b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.340+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/1244975442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d076a6a0 0x5603d0774b60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7faafc011a00 tx=0x7faafc004ab0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.341+0000 7fab0cfa1740 1 -- 172.21.15.70:0/1244975442 shutdown_connections 2024-04-01T01:44:25.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.341+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/1244975442 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5603d0780660 0x5603d0782b00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.341+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/1244975442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d0770200 0x5603d077c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.341+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/1244975442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076b090 0x5603d07750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.341+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/1244975442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d076a6a0 0x5603d0774b60 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.341+0000 7fab0cfa1740 1 -- 172.21.15.70:0/1244975442 >> 172.21.15.70:0/1244975442 conn(0x5603d0765c10 msgr2=0x5603d0764f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.341+0000 7faae37fe640 0 client.28480 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:44:25.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.341+0000 7fab0cfa1740 1 -- 172.21.15.70:0/1244975442 shutdown_connections 2024-04-01T01:44:25.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.341+0000 7fab0cfa1740 1 -- 172.21.15.70:0/1244975442 wait complete. 2024-04-01T01:44:25.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.342+0000 7fab0cfa1740 20 client.28480 trim_cache size 0 max 16384 2024-04-01T01:44:25.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.368+0000 7fab0cfa1740 1 Processor -- start 2024-04-01T01:44:25.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.368+0000 7fab0cfa1740 1 -- start start 2024-04-01T01:44:25.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.369+0000 7fab0cfa1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076a6a0 0x5603d0774770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.369+0000 7fab0cfa1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d076b090 0x5603d077c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.369+0000 7fab0cfa1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d0770200 0x5603d0774cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.369+0000 7fab037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076a6a0 0x5603d0774770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.369+0000 7fab037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076a6a0 0x5603d0774770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53574/0 (socket says 172.21.15.70:53574) 2024-04-01T01:44:25.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.369+0000 7fab037fe640 1 -- 172.21.15.70:0/331667498 learned_addr learned my addr 172.21.15.70:0/331667498 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.369+0000 7fab0cfa1740 1 -- 172.21.15.70:0/331667498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab041c1720 con 0x5603d076b090 2024-04-01T01:44:25.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.369+0000 7fab0cfa1740 1 -- 172.21.15.70:0/331667498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fab041c3260 con 0x5603d076a6a0 2024-04-01T01:44:25.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.369+0000 7fab0cfa1740 1 -- 172.21.15.70:0/331667498 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fab041c1df0 con 0x5603d0770200 2024-04-01T01:44:25.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.370+0000 7fab037fe640 1 --2- 172.21.15.70:0/331667498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d0770200 0x5603d0774cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.370+0000 7fab037fe640 1 -- 172.21.15.70:0/331667498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d0770200 msgr2=0x5603d0774cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.370+0000 7fab037fe640 1 --2- 172.21.15.70:0/331667498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d0770200 0x5603d0774cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.370+0000 7fab037fe640 1 -- 172.21.15.70:0/331667498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d076b090 msgr2=0x5603d077c0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:44:25.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.370+0000 7fab037fe640 1 --2- 172.21.15.70:0/331667498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d076b090 0x5603d077c0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.370+0000 7fab037fe640 1 -- 172.21.15.70:0/331667498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603d07657b0 con 0x5603d076a6a0 2024-04-01T01:44:25.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.370+0000 7fab037fe640 1 --2- 172.21.15.70:0/331667498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076a6a0 0x5603d0774770 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7faafc011550 tx=0x7faafc0049b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90e8886ae108901d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.370+0000 7faae9ffb640 1 -- 172.21.15.70:0/331667498 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faafc013220 con 0x5603d076a6a0 2024-04-01T01:44:25.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.371+0000 7faae9ffb640 1 -- 172.21.15.70:0/331667498 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faafc010c50 con 0x5603d076a6a0 2024-04-01T01:44:25.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.371+0000 7faae9ffb640 1 -- 172.21.15.70:0/331667498 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faafc0045a0 con 0x5603d076a6a0 2024-04-01T01:44:25.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.372+0000 7fab0cfa1740 1 -- 172.21.15.70:0/331667498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076a6a0 msgr2=0x5603d0774770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.372+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/331667498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076a6a0 0x5603d0774770 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7faafc011550 tx=0x7faafc0049b0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.372+0000 7fab0cfa1740 1 -- 172.21.15.70:0/331667498 shutdown_connections 2024-04-01T01:44:25.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.372+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/331667498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d0770200 0x5603d0774cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.372+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/331667498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d076b090 0x5603d077c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.372+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/331667498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d076a6a0 0x5603d0774770 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.372+0000 7fab0cfa1740 1 -- 172.21.15.70:0/331667498 >> 172.21.15.70:0/331667498 conn(0x5603d0765c10 msgr2=0x5603d076c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.372+0000 7fab0cfa1740 1 -- 172.21.15.70:0/331667498 shutdown_connections 2024-04-01T01:44:25.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.372+0000 7fab0cfa1740 1 -- 172.21.15.70:0/331667498 wait complete. 2024-04-01T01:44:25.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.372+0000 7fab0cfa1740 1 -- start start 2024-04-01T01:44:25.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.373+0000 7fab0cfa1740 1 Processor -- start 2024-04-01T01:44:25.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.373+0000 7fab0cfa1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d076a6a0 0x5603d0779220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.373+0000 7fab0cfa1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d076b090 0x5603d0779760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.373+0000 7fab0cfa1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d0770200 0x5603d077a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.373+0000 7fab0cfa1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab041c2b90 con 0x5603d076a6a0 2024-04-01T01:44:25.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.373+0000 7fab0cfa1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fab041c24c0 con 0x5603d0770200 2024-04-01T01:44:25.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.373+0000 7fab0cfa1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fab041bf700 con 0x5603d076b090 2024-04-01T01:44:25.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.373+0000 7fab0cfa1740 10 client.0 _ll_register_callbacks cb 0x7faafc011610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:25.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.373+0000 7fab037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d0770200 0x5603d077a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.373+0000 7fab037fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d0770200 0x5603d077a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53578/0 (socket says 172.21.15.70:53578) 2024-04-01T01:44:25.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.373+0000 7fab037fe640 1 -- 172.21.15.70:0/4149593868 learned_addr learned my addr 172.21.15.70:0/4149593868 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.373+0000 7fab037fe640 1 --2- 172.21.15.70:0/4149593868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d076a6a0 0x5603d0779220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.373+0000 7fab03fff640 1 --2- 172.21.15.70:0/4149593868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d076b090 0x5603d0779760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.374+0000 7fab03fff640 1 -- 172.21.15.70:0/4149593868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d0770200 msgr2=0x5603d077a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.374+0000 7fab03fff640 1 --2- 172.21.15.70:0/4149593868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d0770200 0x5603d077a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.374+0000 7fab03fff640 1 -- 172.21.15.70:0/4149593868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d076a6a0 msgr2=0x5603d0779220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.374+0000 7fab03fff640 1 --2- 172.21.15.70:0/4149593868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d076a6a0 0x5603d0779220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.374+0000 7fab03fff640 1 -- 172.21.15.70:0/4149593868 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603d077ad60 con 0x5603d076b090 2024-04-01T01:44:25.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.374+0000 7fab03fff640 1 --2- 172.21.15.70:0/4149593868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d076b090 0x5603d0779760 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x5603d0766b80 tx=0x7faaf8005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9044c680e5a089f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.374+0000 7faae37fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:44:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.375+0000 7faae37fe640 1 -- 172.21.15.70:0/4149593868 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf8044040 con 0x5603d076b090 2024-04-01T01:44:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.375+0000 7fab0cfa1740 1 -- 172.21.15.70:0/4149593868 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faafc010c50 con 0x5603d076b090 2024-04-01T01:44:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.375+0000 7faae37fe640 1 -- 172.21.15.70:0/4149593868 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7faaf8053cb0 con 0x5603d076b090 2024-04-01T01:44:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.375+0000 7faae37fe640 1 -- 172.21.15.70:0/4149593868 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faaf805c050 con 0x5603d076b090 2024-04-01T01:44:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.375+0000 7fab0cfa1740 10 client.28486 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.375+0000 7fab0cfa1740 1 -- 172.21.15.70:0/4149593868 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5603d077c0d0 con 0x5603d076b090 2024-04-01T01:44:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.375+0000 7faae37fe640 1 -- 172.21.15.70:0/4149593868 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faafc010c50 con 0x5603d076b090 2024-04-01T01:44:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.376+0000 7faae37fe640 1 -- 172.21.15.70:0/4149593868 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7faafc013ce0 con 0x5603d076b090 2024-04-01T01:44:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.376+0000 7fab0cfa1740 10 client.28486 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.376+0000 7fab0cfa1740 10 client.28486 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:44:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.376+0000 7fab0cfa1740 20 client.28486 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:25.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.376+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/4149593868 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5603d07e6a00 0x5603d07e8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.376+0000 7fab0cfa1740 4 client.28486 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:44:25.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.376+0000 7fab0cfa1740 1 -- 172.21.15.70:0/4149593868 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5603d06d6a40 con 0x5603d07e6a00 2024-04-01T01:44:25.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.377+0000 7fab037fe640 1 --2- 172.21.15.70:0/4149593868 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5603d07e6a00 0x5603d07e8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.377+0000 7fab037fe640 1 --2- 172.21.15.70:0/4149593868 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5603d07e6a00 0x5603d07e8df0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=622a4fbe070af3dc server_cookie=b2a654684148a426 in_seq=0 out_seq=0 2024-04-01T01:44:25.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.377+0000 7faae37fe640 10 client.28486 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:44:25.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.377+0000 7faae37fe640 1 -- 172.21.15.70:0/4149593868 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5603d06d6a40 con 0x5603d07e6a00 2024-04-01T01:44:25.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.377+0000 7faae37fe640 10 client.28486 handle_command_reply: tid=0 2024-04-01T01:44:25.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.377+0000 7fab0cfa1740 1 client.28486 shutdown 2024-04-01T01:44:25.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.378+0000 7fab0cfa1740 1 -- 172.21.15.70:0/4149593868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d076b090 msgr2=0x5603d0779760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.378+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/4149593868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d076b090 0x5603d0779760 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x5603d0766b80 tx=0x7faaf8005fd0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.378+0000 7fab0cfa1740 1 -- 172.21.15.70:0/4149593868 shutdown_connections 2024-04-01T01:44:25.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.378+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/4149593868 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5603d07e6a00 0x5603d07e8df0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.378+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/4149593868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603d0770200 0x5603d077a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.378+0000 7faae37fe640 0 client.28486 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:44:25.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.378+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/4149593868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603d076b090 0x5603d0779760 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.378+0000 7fab0cfa1740 1 --2- 172.21.15.70:0/4149593868 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603d076a6a0 0x5603d0779220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.378+0000 7fab0cfa1740 1 -- 172.21.15.70:0/4149593868 >> 172.21.15.70:0/4149593868 conn(0x5603d0765c10 msgr2=0x5603d07746e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.378+0000 7fab0cfa1740 1 -- 172.21.15.70:0/4149593868 shutdown_connections 2024-04-01T01:44:25.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.379+0000 7fab0cfa1740 1 -- 172.21.15.70:0/4149593868 wait complete. 2024-04-01T01:44:25.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.379+0000 7fab0cfa1740 20 client.28486 trim_cache size 0 max 16384 2024-04-01T01:44:25.381 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:44:25.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.388+0000 7fab09165640 1 -- 172.21.15.70:0/908277642 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faaec0a2410 msgr2=0x7faaec0a48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.388+0000 7fab09165640 1 --2- 172.21.15.70:0/908277642 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faaec0a2410 0x7faaec0a48d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7faaf4004770 tx=0x7faaf4097220 comp rx=0 tx=0).stop 2024-04-01T01:44:25.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.388+0000 7fab09165640 1 -- 172.21.15.70:0/908277642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab041097e0 msgr2=0x7fab041b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.388+0000 7fab09165640 1 --2- 172.21.15.70:0/908277642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab041097e0 0x7fab041b66b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7faaf400e970 tx=0x7faaf400ee30 comp rx=0 tx=0).stop 2024-04-01T01:44:25.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.389+0000 7fab09165640 1 -- 172.21.15.70:0/908277642 shutdown_connections 2024-04-01T01:44:25.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.389+0000 7fab09165640 1 --2- 172.21.15.70:0/908277642 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faaec0a2410 0x7faaec0a48d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.389+0000 7fab09165640 1 --2- 172.21.15.70:0/908277642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab041097e0 0x7fab041b66b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.389+0000 7fab09165640 1 --2- 172.21.15.70:0/908277642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab04108e20 0x7fab041153a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.389+0000 7fab09165640 1 --2- 172.21.15.70:0/908277642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab04072b60 0x7fab04114e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.389+0000 7fab09165640 1 -- 172.21.15.70:0/908277642 >> 172.21.15.70:0/908277642 conn(0x7fab0406e630 msgr2=0x7fab040702e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.389+0000 7fab09165640 1 -- 172.21.15.70:0/908277642 shutdown_connections 2024-04-01T01:44:25.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.389+0000 7fab09165640 1 -- 172.21.15.70:0/908277642 wait complete. 2024-04-01T01:44:25.411 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:44:25.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.590+0000 7f23c4f83640 1 -- 172.21.15.70:0/102221502 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23bc004cb0 con 0x7f23c010dce0 2024-04-01T01:44:25.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.590+0000 7f23c6f87640 1 -- 172.21.15.70:0/102221502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23c010dce0 msgr2=0x7f23c01101a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.590+0000 7f23c6f87640 1 --2- 172.21.15.70:0/102221502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23c010dce0 0x7f23c01101a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f23bc00af00 tx=0x7f23bc033fe0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.590+0000 7f23c6f87640 1 -- 172.21.15.70:0/102221502 shutdown_connections 2024-04-01T01:44:25.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.590+0000 7f23c6f87640 1 --2- 172.21.15.70:0/102221502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23c010dce0 0x7f23c01101a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.590+0000 7f23c6f87640 1 --2- 172.21.15.70:0/102221502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23c010b260 0x7f23c010d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.590+0000 7f23c6f87640 1 --2- 172.21.15.70:0/102221502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23c0108930 0x7f23c010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.590+0000 7f23c6f87640 1 -- 172.21.15.70:0/102221502 >> 172.21.15.70:0/102221502 conn(0x7f23c006e820 msgr2=0x7f23c0070c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.591+0000 7f23c6f87640 1 -- 172.21.15.70:0/102221502 shutdown_connections 2024-04-01T01:44:25.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.591+0000 7f23c6f87640 1 -- 172.21.15.70:0/102221502 wait complete. 2024-04-01T01:44:25.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.592+0000 7f23c6f87640 1 Processor -- start 2024-04-01T01:44:25.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.592+0000 7f23c6f87640 1 -- start start 2024-04-01T01:44:25.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.593+0000 7f23c6f87640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23c0108930 0x7f23c01aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.593+0000 7f23c6f87640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23c010b260 0x7f23c01af400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.593+0000 7f23c6f87640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23c010dce0 0x7f23c01b6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.593+0000 7f23c6f87640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f23c0124870 con 0x7f23c010b260 2024-04-01T01:44:25.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.593+0000 7f23c6f87640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f23c01246f0 con 0x7f23c0108930 2024-04-01T01:44:25.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.593+0000 7f23c6f87640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f23c01249f0 con 0x7f23c010dce0 2024-04-01T01:44:25.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.593+0000 7f23c5f85640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23c0108930 0x7f23c01aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.593+0000 7f23c5f85640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23c0108930 0x7f23c01aeec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53594/0 (socket says 172.21.15.70:53594) 2024-04-01T01:44:25.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.593+0000 7f23c5f85640 1 -- 172.21.15.70:0/2429021000 learned_addr learned my addr 172.21.15.70:0/2429021000 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.593+0000 7f23c5784640 1 --2- 172.21.15.70:0/2429021000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23c010b260 0x7f23c01af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.593+0000 7f23c6786640 1 --2- 172.21.15.70:0/2429021000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23c010dce0 0x7f23c01b6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.593+0000 7f23c5f85640 1 -- 172.21.15.70:0/2429021000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23c010dce0 msgr2=0x7f23c01b6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.593+0000 7f23c5f85640 1 --2- 172.21.15.70:0/2429021000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23c010dce0 0x7f23c01b6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.593+0000 7f23c5f85640 1 -- 172.21.15.70:0/2429021000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23c010b260 msgr2=0x7f23c01af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.593+0000 7f23c5f85640 1 --2- 172.21.15.70:0/2429021000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23c010b260 0x7f23c01af400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.593+0000 7f23c5f85640 1 -- 172.21.15.70:0/2429021000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c01b6ae0 con 0x7f23c0108930 2024-04-01T01:44:25.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.594+0000 7f23c5f85640 1 --2- 172.21.15.70:0/2429021000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23c0108930 0x7f23c01aeec0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f23b000b450 tx=0x7f23b000b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.594+0000 7f23b6ffd640 1 -- 172.21.15.70:0/2429021000 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b0036070 con 0x7f23c0108930 2024-04-01T01:44:25.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.594+0000 7f23b6ffd640 1 -- 172.21.15.70:0/2429021000 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f23b0032cb0 con 0x7f23c0108930 2024-04-01T01:44:25.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.594+0000 7f23b6ffd640 1 -- 172.21.15.70:0/2429021000 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b0039de0 con 0x7f23c0108930 2024-04-01T01:44:25.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.594+0000 7f23c6f87640 1 -- 172.21.15.70:0/2429021000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23c0111cb0 con 0x7f23c0108930 2024-04-01T01:44:25.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.594+0000 7f23c6f87640 1 -- 172.21.15.70:0/2429021000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23c0112230 con 0x7f23c0108930 2024-04-01T01:44:25.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.599+0000 7f23b6ffd640 1 -- 172.21.15.70:0/2429021000 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f23b0039070 con 0x7f23c0108930 2024-04-01T01:44:25.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.600+0000 7f23b6ffd640 1 --2- 172.21.15.70:0/2429021000 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23ac0a2620 0x7f23ac0a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.600+0000 7f23cadd2740 1 Processor -- start 2024-04-01T01:44:25.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.600+0000 7f23b6ffd640 1 -- 172.21.15.70:0/2429021000 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f23b00effa0 con 0x7f23c0108930 2024-04-01T01:44:25.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.600+0000 7f23c5784640 1 --2- 172.21.15.70:0/2429021000 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23ac0a2620 0x7f23ac0a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.600+0000 7f23cadd2740 1 -- start start 2024-04-01T01:44:25.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.600+0000 7f23cadd2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfa6a0 0x560f52bfaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.600+0000 7f23cadd2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfb090 0x560f52bffb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.600+0000 7f23cadd2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52c00200 0x560f52c02690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.600+0000 7f23cadd2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560f52b5f1b0 con 0x560f52bfb090 2024-04-01T01:44:25.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.601+0000 7f23cadd2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560f52b07c00 con 0x560f52bfa6a0 2024-04-01T01:44:25.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.601+0000 7f23cadd2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560f52b5b250 con 0x560f52c00200 2024-04-01T01:44:25.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.601+0000 7f23c5f85640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfa6a0 0x560f52bfaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.601+0000 7f23c5f85640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfa6a0 0x560f52bfaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53596/0 (socket says 172.21.15.70:53596) 2024-04-01T01:44:25.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.601+0000 7f23c5f85640 1 -- 172.21.15.70:0/711947787 learned_addr learned my addr 172.21.15.70:0/711947787 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.601+0000 7f23c6786640 1 --2- 172.21.15.70:0/711947787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52c00200 0x560f52c02690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.601+0000 7f23c5784640 1 --2- 172.21.15.70:0/711947787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfb090 0x560f52bffb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.601+0000 7f23c5f85640 1 -- 172.21.15.70:0/711947787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52c00200 msgr2=0x560f52c02690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.601+0000 7f23c5f85640 1 --2- 172.21.15.70:0/711947787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52c00200 0x560f52c02690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.601+0000 7f23c5f85640 1 -- 172.21.15.70:0/711947787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfb090 msgr2=0x560f52bffb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.601+0000 7f23c5f85640 1 --2- 172.21.15.70:0/711947787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfb090 0x560f52bffb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.601+0000 7f23c5f85640 1 -- 172.21.15.70:0/711947787 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f52b273d0 con 0x560f52bfa6a0 2024-04-01T01:44:25.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.601+0000 7f23c5784640 1 --2- 172.21.15.70:0/711947787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfb090 0x560f52bffb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:44:25.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.602+0000 7f23c5f85640 1 --2- 172.21.15.70:0/711947787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfa6a0 0x560f52bfaac0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f23b00bc620 tx=0x7f23b00bc290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ca818168cec421d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.602+0000 7f23c5784640 1 --2- 172.21.15.70:0/2429021000 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23ac0a2620 0x7f23ac0a4ae0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f23c01b03d0 tx=0x7f23b80166e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.602+0000 7f23abfff640 1 -- 172.21.15.70:0/711947787 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b0040070 con 0x560f52bfa6a0 2024-04-01T01:44:25.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.602+0000 7f23abfff640 1 -- 172.21.15.70:0/711947787 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f23b00bc690 con 0x560f52bfa6a0 2024-04-01T01:44:25.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.602+0000 7f23abfff640 1 -- 172.21.15.70:0/711947787 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b00edb90 con 0x560f52bfa6a0 2024-04-01T01:44:25.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.603+0000 7f23cadd2740 1 -- 172.21.15.70:0/711947787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfa6a0 msgr2=0x560f52bfaac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.603+0000 7f23cadd2740 1 --2- 172.21.15.70:0/711947787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfa6a0 0x560f52bfaac0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f23b00bc620 tx=0x7f23b00bc290 comp rx=0 tx=0).stop 2024-04-01T01:44:25.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.603+0000 7f23cadd2740 1 -- 172.21.15.70:0/711947787 shutdown_connections 2024-04-01T01:44:25.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.603+0000 7f23cadd2740 1 --2- 172.21.15.70:0/711947787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52c00200 0x560f52c02690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.603+0000 7f23cadd2740 1 --2- 172.21.15.70:0/711947787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfb090 0x560f52bffb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.603+0000 7f23cadd2740 1 --2- 172.21.15.70:0/711947787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfa6a0 0x560f52bfaac0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.603+0000 7f23cadd2740 1 -- 172.21.15.70:0/711947787 >> 172.21.15.70:0/711947787 conn(0x560f52bf5c10 msgr2=0x560f52bf6010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.603+0000 7f23cadd2740 1 -- 172.21.15.70:0/711947787 shutdown_connections 2024-04-01T01:44:25.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.604+0000 7f23cadd2740 1 -- 172.21.15.70:0/711947787 wait complete. 2024-04-01T01:44:25.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.604+0000 7f23cadd2740 1 -- start start 2024-04-01T01:44:25.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.604+0000 7f23cadd2740 1 Processor -- start 2024-04-01T01:44:25.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.605+0000 7f23cadd2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52bfa6a0 0x560f52c04a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.605+0000 7f23cadd2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfb090 0x560f52c04f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.605+0000 7f23cadd2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52c00200 0x560f52c0bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.605+0000 7f23cadd2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560f52c03150 con 0x560f52bfb090 2024-04-01T01:44:25.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.605+0000 7f23cadd2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560f52afd470 con 0x560f52c00200 2024-04-01T01:44:25.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.605+0000 7f23cadd2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560f52c032d0 con 0x560f52bfa6a0 2024-04-01T01:44:25.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.605+0000 7f23cadd2740 10 client.0 _ll_register_callbacks cb 0x560f52b1acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:25.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.605+0000 7f23c5784640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfb090 0x560f52c04f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.605+0000 7f23c5f85640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52bfa6a0 0x560f52c04a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.605+0000 7f23c5784640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfb090 0x560f52c04f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35736/0 (socket says 172.21.15.70:35736) 2024-04-01T01:44:25.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.605+0000 7f23c5784640 1 -- 172.21.15.70:0/1011390166 learned_addr learned my addr 172.21.15.70:0/1011390166 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.605+0000 7f23c6786640 1 --2- 172.21.15.70:0/1011390166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52c00200 0x560f52c0bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.606+0000 7f23c5f85640 1 -- 172.21.15.70:0/1011390166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52c00200 msgr2=0x560f52c0bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.606+0000 7f23c5f85640 1 --2- 172.21.15.70:0/1011390166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52c00200 0x560f52c0bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.606+0000 7f23c5f85640 1 -- 172.21.15.70:0/1011390166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfb090 msgr2=0x560f52c04f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.606+0000 7f23c5f85640 1 --2- 172.21.15.70:0/1011390166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfb090 0x560f52c04f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.606+0000 7f23c5f85640 1 -- 172.21.15.70:0/1011390166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f528733f0 con 0x560f52bfa6a0 2024-04-01T01:44:25.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.606+0000 7f23c5f85640 1 --2- 172.21.15.70:0/1011390166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52bfa6a0 0x560f52c04a30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f23b00bc620 tx=0x7f23b00ede40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6277a57930446d39 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.606+0000 7f23a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:44:25.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.606+0000 7f23a97fa640 1 -- 172.21.15.70:0/1011390166 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b0040070 con 0x560f52bfa6a0 2024-04-01T01:44:25.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.606+0000 7f23a97fa640 1 -- 172.21.15.70:0/1011390166 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f23b0032a00 con 0x560f52bfa6a0 2024-04-01T01:44:25.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.606+0000 7f23cadd2740 1 -- 172.21.15.70:0/1011390166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23b00bc690 con 0x560f52bfa6a0 2024-04-01T01:44:25.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.607+0000 7f23a97fa640 1 -- 172.21.15.70:0/1011390166 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f23b00c50a0 con 0x560f52bfa6a0 2024-04-01T01:44:25.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.607+0000 7f23cadd2740 10 client.28498 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:25.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.607+0000 7f23cadd2740 1 -- 172.21.15.70:0/1011390166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560f5293d7d0 con 0x560f52bfa6a0 2024-04-01T01:44:25.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.607+0000 7f23a97fa640 1 -- 172.21.15.70:0/1011390166 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f23b00c07e0 con 0x560f52bfa6a0 2024-04-01T01:44:25.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.607+0000 7f23a97fa640 1 -- 172.21.15.70:0/1011390166 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f23b00c6060 con 0x560f52bfa6a0 2024-04-01T01:44:25.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.607+0000 7f23cadd2740 10 client.28498 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:25.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.607+0000 7f23cadd2740 10 client.28498 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:44:25.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.607+0000 7f23cadd2740 20 client.28498 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:25.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.607+0000 7f23cadd2740 1 --2- 172.21.15.70:0/1011390166 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x560f52c10010 0x560f52c12400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.607+0000 7f23cadd2740 4 client.28498 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:25.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.607+0000 7f23cadd2740 1 -- 172.21.15.70:0/1011390166 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f23b00bc690 con 0x560f52c10010 2024-04-01T01:44:25.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.607+0000 7f23c5784640 1 --2- 172.21.15.70:0/1011390166 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x560f52c10010 0x560f52c12400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.608+0000 7f23a97fa640 10 client.28498 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:44:25.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.608+0000 7f23c5784640 1 --2- 172.21.15.70:0/1011390166 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x560f52c10010 0x560f52c12400 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=53dc79e9901f169d server_cookie=6e07a16535cd0f4d in_seq=0 out_seq=0 2024-04-01T01:44:25.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.609+0000 7f23a97fa640 1 -- 172.21.15.70:0/1011390166 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f23b00bc690 con 0x560f52c10010 2024-04-01T01:44:25.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.609+0000 7f23a97fa640 10 client.28498 handle_command_reply: tid=0 2024-04-01T01:44:25.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.609+0000 7f23cadd2740 1 client.28498 shutdown 2024-04-01T01:44:25.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.611+0000 7f23cadd2740 1 -- 172.21.15.70:0/1011390166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52bfa6a0 msgr2=0x560f52c04a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.611+0000 7f23cadd2740 1 --2- 172.21.15.70:0/1011390166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52bfa6a0 0x560f52c04a30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f23b00bc620 tx=0x7f23b00ede40 comp rx=0 tx=0).stop 2024-04-01T01:44:25.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.611+0000 7f23cadd2740 1 -- 172.21.15.70:0/1011390166 shutdown_connections 2024-04-01T01:44:25.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.611+0000 7f23cadd2740 1 --2- 172.21.15.70:0/1011390166 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x560f52c10010 0x560f52c12400 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.611+0000 7f23cadd2740 1 --2- 172.21.15.70:0/1011390166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52c00200 0x560f52c0bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.611+0000 7f23cadd2740 1 --2- 172.21.15.70:0/1011390166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfb090 0x560f52c04f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.611+0000 7f23cadd2740 1 --2- 172.21.15.70:0/1011390166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52bfa6a0 0x560f52c04a30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.611+0000 7f23cadd2740 1 -- 172.21.15.70:0/1011390166 >> 172.21.15.70:0/1011390166 conn(0x560f52bf5c10 msgr2=0x560f52bf4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.612+0000 7f23a97fa640 0 client.28498 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:44:25.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.612+0000 7f23cadd2740 1 -- 172.21.15.70:0/1011390166 shutdown_connections 2024-04-01T01:44:25.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.613+0000 7f23cadd2740 1 -- 172.21.15.70:0/1011390166 wait complete. 2024-04-01T01:44:25.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.613+0000 7f23cadd2740 20 client.28498 trim_cache size 0 max 16384 2024-04-01T01:44:25.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.646+0000 7f23cadd2740 1 Processor -- start 2024-04-01T01:44:25.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.646+0000 7f23cadd2740 1 -- start start 2024-04-01T01:44:25.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.646+0000 7f23cadd2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfa6a0 0x560f52c04d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.646+0000 7f23cadd2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfb090 0x560f52c0bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.646+0000 7f23cadd2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52c00200 0x560f52c16ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.647+0000 7f23cadd2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f23c011c970 con 0x560f52bfa6a0 2024-04-01T01:44:25.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.647+0000 7f23cadd2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f23c011e4b0 con 0x560f52bfb090 2024-04-01T01:44:25.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.647+0000 7f23cadd2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f23c011d040 con 0x560f52c00200 2024-04-01T01:44:25.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.647+0000 7f23c5f85640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfa6a0 0x560f52c04d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.647+0000 7f23c5f85640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfa6a0 0x560f52c04d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35748/0 (socket says 172.21.15.70:35748) 2024-04-01T01:44:25.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.647+0000 7f23c5f85640 1 -- 172.21.15.70:0/3675818149 learned_addr learned my addr 172.21.15.70:0/3675818149 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.647+0000 7f23c5784640 1 --2- 172.21.15.70:0/3675818149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfb090 0x560f52c0bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.647+0000 7f23c5f85640 1 -- 172.21.15.70:0/3675818149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52c00200 msgr2=0x560f52c16ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.647+0000 7f23c5f85640 1 --2- 172.21.15.70:0/3675818149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52c00200 0x560f52c16ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.648+0000 7f23c5f85640 1 -- 172.21.15.70:0/3675818149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfb090 msgr2=0x560f52c0bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.648+0000 7f23c5f85640 1 --2- 172.21.15.70:0/3675818149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfb090 0x560f52c0bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.648+0000 7f23c5f85640 1 -- 172.21.15.70:0/3675818149 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f52bf1670 con 0x560f52bfa6a0 2024-04-01T01:44:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.648+0000 7f23c5784640 1 --2- 172.21.15.70:0/3675818149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfb090 0x560f52c0bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:44:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.648+0000 7f23c5f85640 1 --2- 172.21.15.70:0/3675818149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfa6a0 0x560f52c04d50 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f23b00b8af0 tx=0x7f23b0039650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed99cf2e6397131 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.649+0000 7f23ab7fe640 1 -- 172.21.15.70:0/3675818149 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b0040070 con 0x560f52bfa6a0 2024-04-01T01:44:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.649+0000 7f23ab7fe640 1 -- 172.21.15.70:0/3675818149 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f23b00c07e0 con 0x560f52bfa6a0 2024-04-01T01:44:25.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.649+0000 7f23ab7fe640 1 -- 172.21.15.70:0/3675818149 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23b00bc690 con 0x560f52bfa6a0 2024-04-01T01:44:25.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.650+0000 7f23cadd2740 1 -- 172.21.15.70:0/3675818149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfa6a0 msgr2=0x560f52c04d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.650+0000 7f23cadd2740 1 --2- 172.21.15.70:0/3675818149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfa6a0 0x560f52c04d50 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f23b00b8af0 tx=0x7f23b0039650 comp rx=0 tx=0).stop 2024-04-01T01:44:25.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.650+0000 7f23cadd2740 1 -- 172.21.15.70:0/3675818149 shutdown_connections 2024-04-01T01:44:25.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.650+0000 7f23cadd2740 1 --2- 172.21.15.70:0/3675818149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52c00200 0x560f52c16ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.650+0000 7f23cadd2740 1 --2- 172.21.15.70:0/3675818149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfb090 0x560f52c0bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.650+0000 7f23cadd2740 1 --2- 172.21.15.70:0/3675818149 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52bfa6a0 0x560f52c04d50 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.650+0000 7f23cadd2740 1 -- 172.21.15.70:0/3675818149 >> 172.21.15.70:0/3675818149 conn(0x560f52bf5c10 msgr2=0x560f52c04640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.651+0000 7f23cadd2740 1 -- 172.21.15.70:0/3675818149 shutdown_connections 2024-04-01T01:44:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.651+0000 7f23cadd2740 1 -- 172.21.15.70:0/3675818149 wait complete. 2024-04-01T01:44:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.652+0000 7f23cadd2740 1 -- start start 2024-04-01T01:44:25.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.652+0000 7f23cadd2740 1 Processor -- start 2024-04-01T01:44:25.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.653+0000 7f23cadd2740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfa6a0 0x560f52c09b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.653+0000 7f23cadd2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52bfb090 0x560f52c0a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.653+0000 7f23cadd2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52c00200 0x560f52c0b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.653+0000 7f23cadd2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f23c011dde0 con 0x560f52c00200 2024-04-01T01:44:25.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.653+0000 7f23cadd2740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f23c011d710 con 0x560f52bfa6a0 2024-04-01T01:44:25.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.653+0000 7f23cadd2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f23c011a950 con 0x560f52bfb090 2024-04-01T01:44:25.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.653+0000 7f23cadd2740 10 client.0 _ll_register_callbacks cb 0x7f23b00bce50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:25.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.654+0000 7f23c5f85640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfa6a0 0x560f52c09b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.654+0000 7f23c5784640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52bfb090 0x560f52c0a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.654+0000 7f23c5f85640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfa6a0 0x560f52c09b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53624/0 (socket says 172.21.15.70:53624) 2024-04-01T01:44:25.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.654+0000 7f23c5784640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52bfb090 0x560f52c0a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:53110/0 (socket says 172.21.15.70:53110) 2024-04-01T01:44:25.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.654+0000 7f23c5f85640 1 -- 172.21.15.70:0/3207990768 learned_addr learned my addr 172.21.15.70:0/3207990768 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.654+0000 7f23c6786640 1 --2- 172.21.15.70:0/3207990768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52c00200 0x560f52c0b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.654+0000 7f23c6786640 1 -- 172.21.15.70:0/3207990768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52bfb090 msgr2=0x560f52c0a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.654+0000 7f23c6786640 1 --2- 172.21.15.70:0/3207990768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52bfb090 0x560f52c0a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.654+0000 7f23c6786640 1 -- 172.21.15.70:0/3207990768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfa6a0 msgr2=0x560f52c09b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.654+0000 7f23c6786640 1 --2- 172.21.15.70:0/3207990768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfa6a0 0x560f52c09b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.654+0000 7f23c6786640 1 -- 172.21.15.70:0/3207990768 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f52c0b650 con 0x560f52c00200 2024-04-01T01:44:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.655+0000 7f23c6786640 1 --2- 172.21.15.70:0/3207990768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52c00200 0x560f52c0b060 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f23bc006010 tx=0x7f23bc00a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd3fcd9adac0de9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.655+0000 7f23a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:44:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.656+0000 7f23a97fa640 1 -- 172.21.15.70:0/3207990768 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23bc00a370 con 0x560f52c00200 2024-04-01T01:44:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.656+0000 7f23cadd2740 1 -- 172.21.15.70:0/3207990768 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23b00c07e0 con 0x560f52c00200 2024-04-01T01:44:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.656+0000 7f23a97fa640 1 -- 172.21.15.70:0/3207990768 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f23bc00a510 con 0x560f52c00200 2024-04-01T01:44:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.656+0000 7f23a97fa640 1 -- 172.21.15.70:0/3207990768 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f23bc00a710 con 0x560f52c00200 2024-04-01T01:44:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.657+0000 7f23cadd2740 10 client.18681 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:25.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.657+0000 7f23cadd2740 1 -- 172.21.15.70:0/3207990768 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560f52c765e0 con 0x560f52c00200 2024-04-01T01:44:25.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.657+0000 7f23a97fa640 1 -- 172.21.15.70:0/3207990768 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f23b00c07e0 con 0x560f52c00200 2024-04-01T01:44:25.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.657+0000 7f23a97fa640 1 -- 172.21.15.70:0/3207990768 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f23bc03cc90 con 0x560f52c00200 2024-04-01T01:44:25.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.658+0000 7f23cadd2740 10 client.18681 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:25.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.658+0000 7f23cadd2740 10 client.18681 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:44:25.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.658+0000 7f23cadd2740 20 client.18681 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:25.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.658+0000 7f23cadd2740 1 --2- 172.21.15.70:0/3207990768 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x560f52c76fb0 0x560f52c793d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.658+0000 7f23cadd2740 4 client.18681 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:44:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.658+0000 7f23cadd2740 1 -- 172.21.15.70:0/3207990768 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560f52b6c6f0 con 0x560f52c76fb0 2024-04-01T01:44:25.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.659+0000 7f23c5f85640 1 --2- 172.21.15.70:0/3207990768 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x560f52c76fb0 0x560f52c793d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.661+0000 7f23c5f85640 1 --2- 172.21.15.70:0/3207990768 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x560f52c76fb0 0x560f52c793d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=49b238ca80955c3a server_cookie=2909774634c5f9a0 in_seq=0 out_seq=0 2024-04-01T01:44:25.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.661+0000 7f23a97fa640 10 client.18681 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:44:25.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.661+0000 7f23a97fa640 1 -- 172.21.15.70:0/3207990768 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560f52b6c6f0 con 0x560f52c76fb0 2024-04-01T01:44:25.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.661+0000 7f23a97fa640 10 client.18681 handle_command_reply: tid=0 2024-04-01T01:44:25.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.661+0000 7f23cadd2740 1 client.18681 shutdown 2024-04-01T01:44:25.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.664+0000 7f23cadd2740 1 -- 172.21.15.70:0/3207990768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52c00200 msgr2=0x560f52c0b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.664+0000 7f23cadd2740 1 --2- 172.21.15.70:0/3207990768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52c00200 0x560f52c0b060 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f23bc006010 tx=0x7f23bc00a300 comp rx=0 tx=0).stop 2024-04-01T01:44:25.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.664+0000 7f23cadd2740 1 -- 172.21.15.70:0/3207990768 shutdown_connections 2024-04-01T01:44:25.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.664+0000 7f23cadd2740 1 --2- 172.21.15.70:0/3207990768 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x560f52c76fb0 0x560f52c793d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.664+0000 7f23cadd2740 1 --2- 172.21.15.70:0/3207990768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f52c00200 0x560f52c0b060 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.664+0000 7f23cadd2740 1 --2- 172.21.15.70:0/3207990768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f52bfb090 0x560f52c0a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.664+0000 7f23cadd2740 1 --2- 172.21.15.70:0/3207990768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560f52bfa6a0 0x560f52c09b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.664+0000 7f23cadd2740 1 -- 172.21.15.70:0/3207990768 >> 172.21.15.70:0/3207990768 conn(0x560f52bf5c10 msgr2=0x560f52c04e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.664+0000 7f23a97fa640 0 client.18681 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:44:25.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.664+0000 7f23cadd2740 1 -- 172.21.15.70:0/3207990768 shutdown_connections 2024-04-01T01:44:25.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.665+0000 7f23cadd2740 1 -- 172.21.15.70:0/3207990768 wait complete. 2024-04-01T01:44:25.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.665+0000 7f23cadd2740 20 client.18681 trim_cache size 0 max 16384 2024-04-01T01:44:25.665 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:44:25.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.666+0000 7f23c6f87640 1 -- 172.21.15.70:0/2429021000 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23ac0a2620 msgr2=0x7f23ac0a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.666+0000 7f23c6f87640 1 --2- 172.21.15.70:0/2429021000 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23ac0a2620 0x7f23ac0a4ae0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f23c01b03d0 tx=0x7f23b80166e0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.666+0000 7f23c6f87640 1 -- 172.21.15.70:0/2429021000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23c0108930 msgr2=0x7f23c01aeec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.666+0000 7f23c6f87640 1 --2- 172.21.15.70:0/2429021000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23c0108930 0x7f23c01aeec0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f23b000b450 tx=0x7f23b000b910 comp rx=0 tx=0).stop 2024-04-01T01:44:25.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.667+0000 7f23c6f87640 1 -- 172.21.15.70:0/2429021000 shutdown_connections 2024-04-01T01:44:25.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.667+0000 7f23c6f87640 1 --2- 172.21.15.70:0/2429021000 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23ac0a2620 0x7f23ac0a4ae0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.667+0000 7f23c6f87640 1 --2- 172.21.15.70:0/2429021000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23c010dce0 0x7f23c01b6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.667+0000 7f23c6f87640 1 --2- 172.21.15.70:0/2429021000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f23c010b260 0x7f23c01af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.667+0000 7f23c6f87640 1 --2- 172.21.15.70:0/2429021000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23c0108930 0x7f23c01aeec0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.667+0000 7f23c6f87640 1 -- 172.21.15.70:0/2429021000 >> 172.21.15.70:0/2429021000 conn(0x7f23c006e820 msgr2=0x7f23c0109270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.667+0000 7f23c6f87640 1 -- 172.21.15.70:0/2429021000 shutdown_connections 2024-04-01T01:44:25.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.668+0000 7f23c6f87640 1 -- 172.21.15.70:0/2429021000 wait complete. 2024-04-01T01:44:25.685 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:44:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.882+0000 7fa9948de640 1 -- 172.21.15.70:0/4196439195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9901020e0 msgr2=0x7fa990108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.882+0000 7fa9948de640 1 --2- 172.21.15.70:0/4196439195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9901020e0 0x7fa990108970 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa984009900 tx=0x7fa98402bc50 comp rx=0 tx=0).stop 2024-04-01T01:44:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.883+0000 7fa9948de640 1 -- 172.21.15.70:0/4196439195 shutdown_connections 2024-04-01T01:44:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.883+0000 7fa9948de640 1 --2- 172.21.15.70:0/4196439195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9901020e0 0x7fa990108970 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.883+0000 7fa9948de640 1 --2- 172.21.15.70:0/4196439195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa990101720 0x7fa990101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.883+0000 7fa9948de640 1 --2- 172.21.15.70:0/4196439195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa990100530 0x7fa990100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.883+0000 7fa9948de640 1 -- 172.21.15.70:0/4196439195 >> 172.21.15.70:0/4196439195 conn(0x7fa9900fbac0 msgr2=0x7fa9900fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.883+0000 7fa9948de640 1 -- 172.21.15.70:0/4196439195 shutdown_connections 2024-04-01T01:44:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.883+0000 7fa9948de640 1 -- 172.21.15.70:0/4196439195 wait complete. 2024-04-01T01:44:25.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.884+0000 7fa9948de640 1 Processor -- start 2024-04-01T01:44:25.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.885+0000 7fa9948de640 1 -- start start 2024-04-01T01:44:25.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.885+0000 7fa9948de640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa990100530 0x7fa99019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.885+0000 7fa9948de640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa990101720 0x7fa99019e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.885+0000 7fa9948de640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9901020e0 0x7fa9901a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.885+0000 7fa9948de640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa99010afa0 con 0x7fa990101720 2024-04-01T01:44:25.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.885+0000 7fa9948de640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa99010ae20 con 0x7fa9901020e0 2024-04-01T01:44:25.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.885+0000 7fa9948de640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa99010b120 con 0x7fa990100530 2024-04-01T01:44:25.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.885+0000 7fa98ed9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa990100530 0x7fa99019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.885+0000 7fa98ed9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa990100530 0x7fa99019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:53122/0 (socket says 172.21.15.70:53122) 2024-04-01T01:44:25.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.885+0000 7fa98ed9d640 1 -- 172.21.15.70:0/3598669433 learned_addr learned my addr 172.21.15.70:0/3598669433 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.885+0000 7fa98f59e640 1 --2- 172.21.15.70:0/3598669433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9901020e0 0x7fa9901a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.886+0000 7fa98e59c640 1 --2- 172.21.15.70:0/3598669433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa990101720 0x7fa99019e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.886+0000 7fa98f59e640 1 -- 172.21.15.70:0/3598669433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa990100530 msgr2=0x7fa99019de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.886+0000 7fa98f59e640 1 --2- 172.21.15.70:0/3598669433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa990100530 0x7fa99019de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.886+0000 7fa98f59e640 1 -- 172.21.15.70:0/3598669433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa990101720 msgr2=0x7fa99019e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.886+0000 7fa98f59e640 1 --2- 172.21.15.70:0/3598669433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa990101720 0x7fa99019e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.886+0000 7fa98f59e640 1 -- 172.21.15.70:0/3598669433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9901a5a40 con 0x7fa9901020e0 2024-04-01T01:44:25.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.886+0000 7fa98ed9d640 1 --2- 172.21.15.70:0/3598669433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa990100530 0x7fa99019de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:25.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.886+0000 7fa98e59c640 1 --2- 172.21.15.70:0/3598669433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa990101720 0x7fa99019e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:25.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.887+0000 7fa98f59e640 1 --2- 172.21.15.70:0/3598669433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9901020e0 0x7fa9901a53d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa984009230 tx=0x7fa9840048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.888+0000 7fa977fff640 1 -- 172.21.15.70:0/3598669433 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa984041020 con 0x7fa9901020e0 2024-04-01T01:44:25.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.888+0000 7fa977fff640 1 -- 172.21.15.70:0/3598669433 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa984005600 con 0x7fa9901020e0 2024-04-01T01:44:25.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.888+0000 7fa977fff640 1 -- 172.21.15.70:0/3598669433 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa984041020 con 0x7fa9901020e0 2024-04-01T01:44:25.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.888+0000 7fa9948de640 1 -- 172.21.15.70:0/3598669433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9901a5cd0 con 0x7fa9901020e0 2024-04-01T01:44:25.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.888+0000 7fa9948de640 1 -- 172.21.15.70:0/3598669433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9901a6170 con 0x7fa9901020e0 2024-04-01T01:44:25.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.892+0000 7fa977fff640 1 -- 172.21.15.70:0/3598669433 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fa984044020 con 0x7fa9901020e0 2024-04-01T01:44:25.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.893+0000 7fa997d9d740 1 Processor -- start 2024-04-01T01:44:25.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.893+0000 7fa997d9d740 1 -- start start 2024-04-01T01:44:25.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.893+0000 7fa997d9d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d06b0 0x55a3028d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.893+0000 7fa997d9d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d10a0 0x55a3028d5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.893+0000 7fa997d9d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d6210 0x55a3028d86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.894+0000 7fa997d9d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a3028241b0 con 0x55a3028d06b0 2024-04-01T01:44:25.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.894+0000 7fa997d9d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a3027ccc00 con 0x55a3028d10a0 2024-04-01T01:44:25.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.894+0000 7fa997d9d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a302820250 con 0x55a3028d6210 2024-04-01T01:44:25.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.894+0000 7fa98e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d06b0 0x55a3028d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.894+0000 7fa98e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d06b0 0x55a3028d0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35782/0 (socket says 172.21.15.70:35782) 2024-04-01T01:44:25.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.894+0000 7fa98e59c640 1 -- 172.21.15.70:0/795494593 learned_addr learned my addr 172.21.15.70:0/795494593 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.894+0000 7fa977fff640 1 --2- 172.21.15.70:0/3598669433 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa9780a2910 0x7fa9780a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.894+0000 7fa98e59c640 1 --2- 172.21.15.70:0/795494593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d6210 0x55a3028d86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.894+0000 7fa98f59e640 1 --2- 172.21.15.70:0/3598669433 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa9780a2910 0x7fa9780a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.895+0000 7fa98ed9d640 1 --2- 172.21.15.70:0/795494593 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d10a0 0x55a3028d5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.895+0000 7fa977fff640 1 -- 172.21.15.70:0/3598669433 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa984039050 con 0x7fa9901020e0 2024-04-01T01:44:25.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.895+0000 7fa98e59c640 1 -- 172.21.15.70:0/795494593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d6210 msgr2=0x55a3028d86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.895+0000 7fa98e59c640 1 --2- 172.21.15.70:0/795494593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d6210 0x55a3028d86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.895+0000 7fa98e59c640 1 -- 172.21.15.70:0/795494593 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d10a0 msgr2=0x55a3028d5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.895+0000 7fa98e59c640 1 --2- 172.21.15.70:0/795494593 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d10a0 0x55a3028d5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.895+0000 7fa98e59c640 1 -- 172.21.15.70:0/795494593 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3027ec3d0 con 0x55a3028d06b0 2024-04-01T01:44:25.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.895+0000 7fa98e59c640 1 --2- 172.21.15.70:0/795494593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d06b0 0x55a3028d0ad0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa9800378f0 tx=0x7fa980009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6718d43f01cb4935 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.896+0000 7fa98f59e640 1 --2- 172.21.15.70:0/3598669433 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa9780a2910 0x7fa9780a4dd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa984034000 tx=0x7fa984005890 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.896+0000 7fa9757fa640 1 -- 172.21.15.70:0/795494593 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98003f070 con 0x55a3028d06b0 2024-04-01T01:44:25.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.896+0000 7fa9757fa640 1 -- 172.21.15.70:0/795494593 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa98002ec20 con 0x55a3028d06b0 2024-04-01T01:44:25.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.896+0000 7fa9757fa640 1 -- 172.21.15.70:0/795494593 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98003aab0 con 0x55a3028d06b0 2024-04-01T01:44:25.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.897+0000 7fa997d9d740 1 -- 172.21.15.70:0/795494593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d06b0 msgr2=0x55a3028d0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.897+0000 7fa997d9d740 1 --2- 172.21.15.70:0/795494593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d06b0 0x55a3028d0ad0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa9800378f0 tx=0x7fa980009890 comp rx=0 tx=0).stop 2024-04-01T01:44:25.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.898+0000 7fa997d9d740 1 -- 172.21.15.70:0/795494593 shutdown_connections 2024-04-01T01:44:25.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.898+0000 7fa997d9d740 1 --2- 172.21.15.70:0/795494593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d6210 0x55a3028d86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.898+0000 7fa997d9d740 1 --2- 172.21.15.70:0/795494593 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d10a0 0x55a3028d5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.898+0000 7fa997d9d740 1 --2- 172.21.15.70:0/795494593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d06b0 0x55a3028d0ad0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.898+0000 7fa997d9d740 1 -- 172.21.15.70:0/795494593 >> 172.21.15.70:0/795494593 conn(0x55a3028bac10 msgr2=0x55a3028bb010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.898+0000 7fa997d9d740 1 -- 172.21.15.70:0/795494593 shutdown_connections 2024-04-01T01:44:25.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.898+0000 7fa997d9d740 1 -- 172.21.15.70:0/795494593 wait complete. 2024-04-01T01:44:25.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.899+0000 7fa997d9d740 1 -- start start 2024-04-01T01:44:25.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.899+0000 7fa997d9d740 1 Processor -- start 2024-04-01T01:44:25.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.899+0000 7fa997d9d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d06b0 0x55a3028c10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.899+0000 7fa997d9d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d10a0 0x55a3028c1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.899+0000 7fa997d9d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d6210 0x55a3028c8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.899+0000 7fa997d9d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a3028bf500 con 0x55a3028d10a0 2024-04-01T01:44:25.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.899+0000 7fa997d9d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a3027c2470 con 0x55a3028d6210 2024-04-01T01:44:25.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.899+0000 7fa997d9d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a3028bf680 con 0x55a3028d06b0 2024-04-01T01:44:25.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.899+0000 7fa997d9d740 10 client.0 _ll_register_callbacks cb 0x55a3027dfcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:25.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.899+0000 7fa98ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d10a0 0x55a3028c1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.899+0000 7fa98ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d10a0 0x55a3028c1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35788/0 (socket says 172.21.15.70:35788) 2024-04-01T01:44:25.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.899+0000 7fa98ed9d640 1 -- 172.21.15.70:0/1534462743 learned_addr learned my addr 172.21.15.70:0/1534462743 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.899+0000 7fa98e59c640 1 --2- 172.21.15.70:0/1534462743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d06b0 0x55a3028c10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.899+0000 7fa98e59c640 1 --2- 172.21.15.70:0/1534462743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d6210 0x55a3028c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.900+0000 7fa98e59c640 1 -- 172.21.15.70:0/1534462743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d6210 msgr2=0x55a3028c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.900+0000 7fa98e59c640 1 --2- 172.21.15.70:0/1534462743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d6210 0x55a3028c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.900+0000 7fa98e59c640 1 -- 172.21.15.70:0/1534462743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d10a0 msgr2=0x55a3028c1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.900+0000 7fa98e59c640 1 --2- 172.21.15.70:0/1534462743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d10a0 0x55a3028c1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.900+0000 7fa98e59c640 1 -- 172.21.15.70:0/1534462743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3025383f0 con 0x55a3028d06b0 2024-04-01T01:44:25.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.900+0000 7fa98e59c640 1 --2- 172.21.15.70:0/1534462743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d06b0 0x55a3028c10e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa9800378f0 tx=0x7fa980009e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4baca172b68596d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.900+0000 7fa972ff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:44:25.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.901+0000 7fa972ff5640 1 -- 172.21.15.70:0/1534462743 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98003f040 con 0x55a3028d06b0 2024-04-01T01:44:25.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.901+0000 7fa997d9d740 1 -- 172.21.15.70:0/1534462743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa98002ec20 con 0x55a3028d06b0 2024-04-01T01:44:25.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.901+0000 7fa972ff5640 1 -- 172.21.15.70:0/1534462743 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa9800393f0 con 0x55a3028d06b0 2024-04-01T01:44:25.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.901+0000 7fa972ff5640 1 -- 172.21.15.70:0/1534462743 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa9800395f0 con 0x55a3028d06b0 2024-04-01T01:44:25.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.901+0000 7fa997d9d740 10 client.28510 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:25.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.901+0000 7fa997d9d740 1 -- 172.21.15.70:0/1534462743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a3026027d0 con 0x55a3028d06b0 2024-04-01T01:44:25.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.902+0000 7fa972ff5640 1 -- 172.21.15.70:0/1534462743 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa98002ec20 con 0x55a3028d06b0 2024-04-01T01:44:25.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.902+0000 7fa972ff5640 1 -- 172.21.15.70:0/1534462743 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fa980045070 con 0x55a3028d06b0 2024-04-01T01:44:25.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.902+0000 7fa997d9d740 10 client.28510 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:25.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.902+0000 7fa997d9d740 10 client.28510 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:44:25.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.902+0000 7fa997d9d740 20 client.28510 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:25.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.902+0000 7fa997d9d740 1 --2- 172.21.15.70:0/1534462743 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55a3028cc8c0 0x55a3028cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.902+0000 7fa997d9d740 4 client.28510 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:25.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.902+0000 7fa997d9d740 1 -- 172.21.15.70:0/1534462743 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3028316f0 con 0x55a3028cc8c0 2024-04-01T01:44:25.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.902+0000 7fa98ed9d640 1 --2- 172.21.15.70:0/1534462743 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55a3028cc8c0 0x55a3028cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.903+0000 7fa98ed9d640 1 --2- 172.21.15.70:0/1534462743 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55a3028cc8c0 0x55a3028cecd0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f1c1e00417b09746 server_cookie=a8951da7894a0308 in_seq=0 out_seq=0 2024-04-01T01:44:25.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.903+0000 7fa972ff5640 10 client.28510 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:44:25.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.904+0000 7fa972ff5640 1 -- 172.21.15.70:0/1534462743 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a3028316f0 con 0x55a3028cc8c0 2024-04-01T01:44:25.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.904+0000 7fa972ff5640 10 client.28510 handle_command_reply: tid=0 2024-04-01T01:44:25.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.904+0000 7fa997d9d740 1 client.28510 shutdown 2024-04-01T01:44:25.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.904+0000 7fa997d9d740 1 -- 172.21.15.70:0/1534462743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d06b0 msgr2=0x55a3028c10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.904+0000 7fa997d9d740 1 --2- 172.21.15.70:0/1534462743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d06b0 0x55a3028c10e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa9800378f0 tx=0x7fa980009e60 comp rx=0 tx=0).stop 2024-04-01T01:44:25.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.905+0000 7fa997d9d740 1 -- 172.21.15.70:0/1534462743 shutdown_connections 2024-04-01T01:44:25.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.905+0000 7fa997d9d740 1 --2- 172.21.15.70:0/1534462743 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55a3028cc8c0 0x55a3028cecd0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.905+0000 7fa997d9d740 1 --2- 172.21.15.70:0/1534462743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d6210 0x55a3028c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.905+0000 7fa997d9d740 1 --2- 172.21.15.70:0/1534462743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d10a0 0x55a3028c1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.905+0000 7fa997d9d740 1 --2- 172.21.15.70:0/1534462743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d06b0 0x55a3028c10e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.905+0000 7fa997d9d740 1 -- 172.21.15.70:0/1534462743 >> 172.21.15.70:0/1534462743 conn(0x55a3028bac10 msgr2=0x55a3028b9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.905+0000 7fa972ff5640 0 client.28510 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:44:25.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.905+0000 7fa997d9d740 1 -- 172.21.15.70:0/1534462743 shutdown_connections 2024-04-01T01:44:25.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.905+0000 7fa997d9d740 1 -- 172.21.15.70:0/1534462743 wait complete. 2024-04-01T01:44:25.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.905+0000 7fa997d9d740 20 client.28510 trim_cache size 0 max 16384 2024-04-01T01:44:25.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.938+0000 7fa997d9d740 1 Processor -- start 2024-04-01T01:44:25.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.939+0000 7fa997d9d740 1 -- start start 2024-04-01T01:44:25.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.939+0000 7fa997d9d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d06b0 0x55a3028c1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.939+0000 7fa997d9d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d10a0 0x55a3028cf570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.939+0000 7fa997d9d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d6210 0x55a3028cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.939+0000 7fa997d9d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c00fda0 con 0x55a3028d06b0 2024-04-01T01:44:25.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.939+0000 7fa997d9d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c0118e0 con 0x55a3028d10a0 2024-04-01T01:44:25.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.939+0000 7fa997d9d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c010470 con 0x55a3028d6210 2024-04-01T01:44:25.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.940+0000 7fa98ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d10a0 0x55a3028cf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.940+0000 7fa98ed9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d10a0 0x55a3028cf570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53656/0 (socket says 172.21.15.70:53656) 2024-04-01T01:44:25.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.940+0000 7fa98ed9d640 1 -- 172.21.15.70:0/3380215107 learned_addr learned my addr 172.21.15.70:0/3380215107 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.940+0000 7fa98e59c640 1 --2- 172.21.15.70:0/3380215107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d06b0 0x55a3028c1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.940+0000 7fa98e59c640 1 --2- 172.21.15.70:0/3380215107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d6210 0x55a3028cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.940+0000 7fa98ed9d640 1 -- 172.21.15.70:0/3380215107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d6210 msgr2=0x55a3028cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.940+0000 7fa98ed9d640 1 --2- 172.21.15.70:0/3380215107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d6210 0x55a3028cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.940+0000 7fa98ed9d640 1 -- 172.21.15.70:0/3380215107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d06b0 msgr2=0x55a3028c1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.940+0000 7fa98ed9d640 1 --2- 172.21.15.70:0/3380215107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d06b0 0x55a3028c1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.940+0000 7fa98ed9d640 1 -- 172.21.15.70:0/3380215107 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3028bb680 con 0x55a3028d10a0 2024-04-01T01:44:25.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.941+0000 7fa98ed9d640 1 --2- 172.21.15.70:0/3380215107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d10a0 0x55a3028cf570 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa988003e70 tx=0x7fa988009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=376779bb7734ded4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.942+0000 7fa974ff9640 1 -- 172.21.15.70:0/3380215107 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa988004d20 con 0x55a3028d10a0 2024-04-01T01:44:25.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.942+0000 7fa974ff9640 1 -- 172.21.15.70:0/3380215107 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa988023030 con 0x55a3028d10a0 2024-04-01T01:44:25.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.942+0000 7fa974ff9640 1 -- 172.21.15.70:0/3380215107 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa988005690 con 0x55a3028d10a0 2024-04-01T01:44:25.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.944+0000 7fa997d9d740 1 -- 172.21.15.70:0/3380215107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d10a0 msgr2=0x55a3028cf570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.944+0000 7fa997d9d740 1 --2- 172.21.15.70:0/3380215107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d10a0 0x55a3028cf570 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa988003e70 tx=0x7fa988009500 comp rx=0 tx=0).stop 2024-04-01T01:44:25.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.944+0000 7fa997d9d740 1 -- 172.21.15.70:0/3380215107 shutdown_connections 2024-04-01T01:44:25.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.944+0000 7fa997d9d740 1 --2- 172.21.15.70:0/3380215107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d6210 0x55a3028cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.944+0000 7fa997d9d740 1 --2- 172.21.15.70:0/3380215107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d10a0 0x55a3028cf570 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.944+0000 7fa997d9d740 1 --2- 172.21.15.70:0/3380215107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d06b0 0x55a3028c1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.944+0000 7fa997d9d740 1 -- 172.21.15.70:0/3380215107 >> 172.21.15.70:0/3380215107 conn(0x55a3028bac10 msgr2=0x55a3028c0cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.944+0000 7fa997d9d740 1 -- 172.21.15.70:0/3380215107 shutdown_connections 2024-04-01T01:44:25.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.944+0000 7fa997d9d740 1 -- 172.21.15.70:0/3380215107 wait complete. 2024-04-01T01:44:25.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.945+0000 7fa997d9d740 1 -- start start 2024-04-01T01:44:25.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.945+0000 7fa997d9d740 1 Processor -- start 2024-04-01T01:44:25.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.945+0000 7fa997d9d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d06b0 0x55a3028dde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.945+0000 7fa997d9d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d10a0 0x55a3028de390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.945+0000 7fa98e59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d06b0 0x55a3028dde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.946+0000 7fa98e59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d06b0 0x55a3028dde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:53138/0 (socket says 172.21.15.70:53138) 2024-04-01T01:44:25.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.946+0000 7fa98e59c640 1 -- 172.21.15.70:0/3750446513 learned_addr learned my addr 172.21.15.70:0/3750446513 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:25.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.946+0000 7fa997d9d740 1 --2- 172.21.15.70:0/3750446513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d6210 0x55a3028dc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.946+0000 7fa997d9d740 1 -- 172.21.15.70:0/3750446513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c011210 con 0x55a3028d10a0 2024-04-01T01:44:25.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.946+0000 7fa98ed9d640 1 --2- 172.21.15.70:0/3750446513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d10a0 0x55a3028de390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.946+0000 7fa997d9d740 1 -- 172.21.15.70:0/3750446513 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c010b40 con 0x55a3028d6210 2024-04-01T01:44:25.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.946+0000 7fa997d9d740 1 -- 172.21.15.70:0/3750446513 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c00dd80 con 0x55a3028d06b0 2024-04-01T01:44:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.946+0000 7fa997d9d740 10 client.0 _ll_register_callbacks cb 0x7fa98002eaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.946+0000 7fa98e59c640 1 --2- 172.21.15.70:0/3750446513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d6210 0x55a3028dc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.946+0000 7fa98e59c640 1 -- 172.21.15.70:0/3750446513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d6210 msgr2=0x55a3028dc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.946+0000 7fa98e59c640 1 --2- 172.21.15.70:0/3750446513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d6210 0x55a3028dc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.946+0000 7fa98e59c640 1 -- 172.21.15.70:0/3750446513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d10a0 msgr2=0x55a3028de390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.946+0000 7fa98e59c640 1 --2- 172.21.15.70:0/3750446513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d10a0 0x55a3028de390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.946+0000 7fa98e59c640 1 -- 172.21.15.70:0/3750446513 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3028dcd10 con 0x55a3028d06b0 2024-04-01T01:44:25.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.946+0000 7fa98ed9d640 1 --2- 172.21.15.70:0/3750446513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d10a0 0x55a3028de390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:44:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.947+0000 7fa98e59c640 1 --2- 172.21.15.70:0/3750446513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d06b0 0x55a3028dde50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa980037380 tx=0x7fa9800046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee48f304b575535b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.947+0000 7fa972ff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:44:25.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.947+0000 7fa972ff5640 1 -- 172.21.15.70:0/3750446513 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98003f070 con 0x55a3028d06b0 2024-04-01T01:44:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.947+0000 7fa972ff5640 1 -- 172.21.15.70:0/3750446513 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa98002eb50 con 0x55a3028d06b0 2024-04-01T01:44:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.947+0000 7fa997d9d740 1 -- 172.21.15.70:0/3750446513 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa988023030 con 0x55a3028d06b0 2024-04-01T01:44:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.948+0000 7fa972ff5640 1 -- 172.21.15.70:0/3750446513 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa980071050 con 0x55a3028d06b0 2024-04-01T01:44:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.948+0000 7fa997d9d740 10 client.28516 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.948+0000 7fa997d9d740 1 -- 172.21.15.70:0/3750446513 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a30293b150 con 0x55a3028d06b0 2024-04-01T01:44:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.948+0000 7fa972ff5640 1 -- 172.21.15.70:0/3750446513 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9800473e0 con 0x55a3028d06b0 2024-04-01T01:44:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.948+0000 7fa972ff5640 1 -- 172.21.15.70:0/3750446513 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fa980067040 con 0x55a3028d06b0 2024-04-01T01:44:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.948+0000 7fa997d9d740 10 client.28516 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.948+0000 7fa997d9d740 10 client.28516 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:44:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.948+0000 7fa997d9d740 20 client.28516 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.949+0000 7fa997d9d740 1 --2- 172.21.15.70:0/3750446513 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55a30293ba60 0x55a30293de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.949+0000 7fa997d9d740 4 client.28516 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:44:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.949+0000 7fa997d9d740 1 -- 172.21.15.70:0/3750446513 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fa988023030 con 0x55a30293ba60 2024-04-01T01:44:25.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.949+0000 7fa98ed9d640 1 --2- 172.21.15.70:0/3750446513 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55a30293ba60 0x55a30293de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:25.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.949+0000 7fa98ed9d640 1 --2- 172.21.15.70:0/3750446513 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55a30293ba60 0x55a30293de80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dc1d72d03247540b server_cookie=5b4f9f74a39e5b15 in_seq=0 out_seq=0 2024-04-01T01:44:25.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.949+0000 7fa972ff5640 10 client.28516 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:44:25.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.950+0000 7fa972ff5640 1 -- 172.21.15.70:0/3750446513 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fa988023030 con 0x55a30293ba60 2024-04-01T01:44:25.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.950+0000 7fa972ff5640 10 client.28516 handle_command_reply: tid=0 2024-04-01T01:44:25.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.950+0000 7fa997d9d740 1 client.28516 shutdown 2024-04-01T01:44:25.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.951+0000 7fa997d9d740 1 -- 172.21.15.70:0/3750446513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d06b0 msgr2=0x55a3028dde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.951+0000 7fa997d9d740 1 --2- 172.21.15.70:0/3750446513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d06b0 0x55a3028dde50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa980037380 tx=0x7fa9800046c0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.952+0000 7fa997d9d740 1 -- 172.21.15.70:0/3750446513 shutdown_connections 2024-04-01T01:44:25.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.952+0000 7fa997d9d740 1 --2- 172.21.15.70:0/3750446513 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55a30293ba60 0x55a30293de80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.952+0000 7fa997d9d740 1 --2- 172.21.15.70:0/3750446513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3028d6210 0x55a3028dc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.952+0000 7fa997d9d740 1 --2- 172.21.15.70:0/3750446513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3028d10a0 0x55a3028de390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.952+0000 7fa997d9d740 1 --2- 172.21.15.70:0/3750446513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3028d06b0 0x55a3028dde50 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.952+0000 7fa972ff5640 0 client.28516 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:44:25.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.952+0000 7fa997d9d740 1 -- 172.21.15.70:0/3750446513 >> 172.21.15.70:0/3750446513 conn(0x55a3028bac10 msgr2=0x55a3028cf4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.952+0000 7fa997d9d740 1 -- 172.21.15.70:0/3750446513 shutdown_connections 2024-04-01T01:44:25.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.952+0000 7fa997d9d740 1 -- 172.21.15.70:0/3750446513 wait complete. 2024-04-01T01:44:25.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.952+0000 7fa997d9d740 20 client.28516 trim_cache size 0 max 16384 2024-04-01T01:44:25.953 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:44:25.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.954+0000 7fa9948de640 1 -- 172.21.15.70:0/3598669433 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa9780a2910 msgr2=0x7fa9780a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.954+0000 7fa9948de640 1 --2- 172.21.15.70:0/3598669433 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa9780a2910 0x7fa9780a4dd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa984034000 tx=0x7fa984005890 comp rx=0 tx=0).stop 2024-04-01T01:44:25.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.954+0000 7fa9948de640 1 -- 172.21.15.70:0/3598669433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9901020e0 msgr2=0x7fa9901a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:25.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.954+0000 7fa9948de640 1 --2- 172.21.15.70:0/3598669433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9901020e0 0x7fa9901a53d0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fa984009230 tx=0x7fa9840048f0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.955+0000 7fa9948de640 1 -- 172.21.15.70:0/3598669433 shutdown_connections 2024-04-01T01:44:25.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.955+0000 7fa9948de640 1 --2- 172.21.15.70:0/3598669433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9901020e0 0x7fa9901a53d0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.955+0000 7fa9948de640 1 --2- 172.21.15.70:0/3598669433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa990101720 0x7fa99019e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.955+0000 7fa9948de640 1 --2- 172.21.15.70:0/3598669433 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa9780a2910 0x7fa9780a4dd0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.955+0000 7fa9948de640 1 --2- 172.21.15.70:0/3598669433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa990100530 0x7fa99019de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:25.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.955+0000 7fa9948de640 1 -- 172.21.15.70:0/3598669433 >> 172.21.15.70:0/3598669433 conn(0x7fa9900fbac0 msgr2=0x7fa99010da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:25.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.955+0000 7fa9948de640 1 -- 172.21.15.70:0/3598669433 shutdown_connections 2024-04-01T01:44:25.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:25.955+0000 7fa9948de640 1 -- 172.21.15.70:0/3598669433 wait complete. 2024-04-01T01:44:25.973 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:25 smithi070 ceph-mon[32179]: pgmap v687: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 12 MiB/s wr, 742 op/s 2024-04-01T01:44:25.974 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:44:26.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.169+0000 7f571be3b640 1 -- 172.21.15.70:0/409702071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5714108930 msgr2=0x7f571410ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:26.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.169+0000 7f571be3b640 1 --2- 172.21.15.70:0/409702071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5714108930 0x7f571410ad20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f570c009ca0 tx=0x7f570c02c120 comp rx=0 tx=0).stop 2024-04-01T01:44:26.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.170+0000 7f571be3b640 1 -- 172.21.15.70:0/409702071 shutdown_connections 2024-04-01T01:44:26.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.170+0000 7f571be3b640 1 --2- 172.21.15.70:0/409702071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f571410dd20 0x7f57141101e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.170+0000 7f571be3b640 1 --2- 172.21.15.70:0/409702071 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f571410b260 0x7f571410d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.170+0000 7f571be3b640 1 --2- 172.21.15.70:0/409702071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5714108930 0x7f571410ad20 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.170+0000 7f571be3b640 1 -- 172.21.15.70:0/409702071 >> 172.21.15.70:0/409702071 conn(0x7f571406e630 msgr2=0x7f5714070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:26.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.170+0000 7f571be3b640 1 -- 172.21.15.70:0/409702071 shutdown_connections 2024-04-01T01:44:26.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.171+0000 7f571be3b640 1 -- 172.21.15.70:0/409702071 wait complete. 2024-04-01T01:44:26.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.172+0000 7f571be3b640 1 Processor -- start 2024-04-01T01:44:26.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.172+0000 7f571be3b640 1 -- start start 2024-04-01T01:44:26.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.172+0000 7f571be3b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5714108930 0x7f571419e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.172+0000 7f571be3b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f571410b260 0x7f571419e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.172+0000 7f571be3b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f571410dd20 0x7f57141a58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.173+0000 7f571be3b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5714113c40 con 0x7f571410dd20 2024-04-01T01:44:26.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.173+0000 7f571be3b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5714113ac0 con 0x7f5714108930 2024-04-01T01:44:26.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.173+0000 7f571be3b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5714113dc0 con 0x7f571410b260 2024-04-01T01:44:26.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.173+0000 7f571a3d8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f571410dd20 0x7f57141a58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.173+0000 7f571a3d8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f571410dd20 0x7f57141a58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35818/0 (socket says 172.21.15.70:35818) 2024-04-01T01:44:26.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.173+0000 7f571a3d8640 1 -- 172.21.15.70:0/3723327069 learned_addr learned my addr 172.21.15.70:0/3723327069 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:26.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.175+0000 7f571a3d8640 1 -- 172.21.15.70:0/3723327069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f571410b260 msgr2=0x7f571419e830 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:44:26.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.175+0000 7f571a3d8640 1 --2- 172.21.15.70:0/3723327069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f571410b260 0x7f571419e830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.175+0000 7f571a3d8640 1 -- 172.21.15.70:0/3723327069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5714108930 msgr2=0x7f571419e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:26.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.175+0000 7f571a3d8640 1 --2- 172.21.15.70:0/3723327069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5714108930 0x7f571419e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.175+0000 7f571a3d8640 1 -- 172.21.15.70:0/3723327069 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57141a5f10 con 0x7f571410dd20 2024-04-01T01:44:26.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.176+0000 7f571a3d8640 1 --2- 172.21.15.70:0/3723327069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f571410dd20 0x7f57141a58a0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f570400c7c0 tx=0x7f570400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:26.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.177+0000 7f570affd640 1 -- 172.21.15.70:0/3723327069 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5704007bb0 con 0x7f571410dd20 2024-04-01T01:44:26.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.177+0000 7f570affd640 1 -- 172.21.15.70:0/3723327069 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5704007d50 con 0x7f571410dd20 2024-04-01T01:44:26.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.178+0000 7f570affd640 1 -- 172.21.15.70:0/3723327069 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5704018640 con 0x7f571410dd20 2024-04-01T01:44:26.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.179+0000 7f571be3b640 1 -- 172.21.15.70:0/3723327069 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57140727c0 con 0x7f571410dd20 2024-04-01T01:44:26.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.179+0000 7f571be3b640 1 -- 172.21.15.70:0/3723327069 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57141a6200 con 0x7f571410dd20 2024-04-01T01:44:26.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.182+0000 7f571ea19740 1 Processor -- start 2024-04-01T01:44:26.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.185+0000 7f571ea19740 1 -- start start 2024-04-01T01:44:26.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.185+0000 7f571ea19740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae8a20 0x5566aaae8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.185+0000 7f571ea19740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae9380 0x5566aaaf6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.185+0000 7f571ea19740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 0x5566aaaf8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.185+0000 7f571ea19740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5566aaa4d1b0 con 0x5566aaae8a20 2024-04-01T01:44:26.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.185+0000 7f571ea19740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5566aa9f5c00 con 0x5566aaaf6a20 2024-04-01T01:44:26.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.185+0000 7f571ea19740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5566aaa49250 con 0x5566aaae9380 2024-04-01T01:44:26.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.186+0000 7f5719bd7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae9380 0x5566aaaf6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.186+0000 7f5719bd7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae9380 0x5566aaaf6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:53160/0 (socket says 172.21.15.70:53160) 2024-04-01T01:44:26.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.186+0000 7f5719bd7640 1 -- 172.21.15.70:0/2239607142 learned_addr learned my addr 172.21.15.70:0/2239607142 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:26.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.186+0000 7f57193d6640 1 --2- 172.21.15.70:0/2239607142 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 0x5566aaaf8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.187+0000 7f57193d6640 1 --2- 172.21.15.70:0/2239607142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae8a20 0x5566aaae8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.187+0000 7f57193d6640 1 -- 172.21.15.70:0/2239607142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae9380 msgr2=0x5566aaaf6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:26.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.187+0000 7f57193d6640 1 --2- 172.21.15.70:0/2239607142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae9380 0x5566aaaf6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.187+0000 7f57193d6640 1 -- 172.21.15.70:0/2239607142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae8a20 msgr2=0x5566aaae8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:26.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.187+0000 7f57193d6640 1 --2- 172.21.15.70:0/2239607142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae8a20 0x5566aaae8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.187+0000 7f57193d6640 1 -- 172.21.15.70:0/2239607142 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566aaa153d0 con 0x5566aaaf6a20 2024-04-01T01:44:26.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.188+0000 7f57193d6640 1 --2- 172.21.15.70:0/2239607142 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 0x5566aaaf8e90 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5714073950 tx=0x7f571000d320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ba17a455e3fa90e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:26.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.188+0000 7f56fffff640 1 -- 172.21.15.70:0/2239607142 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571000acb0 con 0x5566aaaf6a20 2024-04-01T01:44:26.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.188+0000 7f56fffff640 1 -- 172.21.15.70:0/2239607142 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f571000ae50 con 0x5566aaaf6a20 2024-04-01T01:44:26.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.189+0000 7f570affd640 1 -- 172.21.15.70:0/3723327069 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f570400f450 con 0x7f571410dd20 2024-04-01T01:44:26.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.189+0000 7f570affd640 1 --2- 172.21.15.70:0/3723327069 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57000a2590 0x7f57000a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.190+0000 7f571ea19740 1 -- 172.21.15.70:0/2239607142 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 msgr2=0x5566aaaf8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:26.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.190+0000 7f571ea19740 1 --2- 172.21.15.70:0/2239607142 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 0x5566aaaf8e90 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5714073950 tx=0x7f571000d320 comp rx=0 tx=0).stop 2024-04-01T01:44:26.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.190+0000 7f570affd640 1 -- 172.21.15.70:0/3723327069 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f57040ce740 con 0x7f571410dd20 2024-04-01T01:44:26.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.190+0000 7f571ea19740 1 -- 172.21.15.70:0/2239607142 shutdown_connections 2024-04-01T01:44:26.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.190+0000 7f571ea19740 1 --2- 172.21.15.70:0/2239607142 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 0x5566aaaf8e90 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.190+0000 7f571ea19740 1 --2- 172.21.15.70:0/2239607142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae9380 0x5566aaaf6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.190+0000 7f571ea19740 1 --2- 172.21.15.70:0/2239607142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae8a20 0x5566aaae8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.190+0000 7f571ea19740 1 -- 172.21.15.70:0/2239607142 >> 172.21.15.70:0/2239607142 conn(0x5566aaae51d0 msgr2=0x5566aaae3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:26.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.191+0000 7f5719bd7640 1 --2- 172.21.15.70:0/3723327069 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57000a2590 0x7f57000a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.194+0000 7f571ea19740 1 -- 172.21.15.70:0/2239607142 shutdown_connections 2024-04-01T01:44:26.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.194+0000 7f5719bd7640 1 --2- 172.21.15.70:0/3723327069 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57000a2590 0x7f57000a4a50 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f570c006010 tx=0x7f570c009630 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:26.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.194+0000 7f571ea19740 1 -- 172.21.15.70:0/2239607142 wait complete. 2024-04-01T01:44:26.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.195+0000 7f571ea19740 1 -- start start 2024-04-01T01:44:26.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.195+0000 7f571ea19740 1 Processor -- start 2024-04-01T01:44:26.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.195+0000 7f571ea19740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaae8a20 0x5566aaafb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.195+0000 7f571ea19740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae9380 0x5566aaafb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.196+0000 7f571ea19740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaaf6a20 0x5566aab028c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.196+0000 7f571ea19740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5566aaaf9950 con 0x5566aaaf6a20 2024-04-01T01:44:26.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.196+0000 7f571ea19740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5566aa9eb470 con 0x5566aaae8a20 2024-04-01T01:44:26.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.196+0000 7f571ea19740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5566aaaf9ad0 con 0x5566aaae9380 2024-04-01T01:44:26.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.196+0000 7f5719bd7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaae8a20 0x5566aaafb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.196+0000 7f5719bd7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaae8a20 0x5566aaafb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53710/0 (socket says 172.21.15.70:53710) 2024-04-01T01:44:26.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.196+0000 7f5719bd7640 1 -- 172.21.15.70:0/4166967018 learned_addr learned my addr 172.21.15.70:0/4166967018 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:26.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.196+0000 7f571ea19740 10 client.0 _ll_register_callbacks cb 0x5566aaa08cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:26.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.196+0000 7f5719bd7640 1 -- 172.21.15.70:0/4166967018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae9380 msgr2=0x5566aaafb880 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:44:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.196+0000 7f5719bd7640 1 --2- 172.21.15.70:0/4166967018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae9380 0x5566aaafb880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.196+0000 7f5719bd7640 1 -- 172.21.15.70:0/4166967018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaaf6a20 msgr2=0x5566aab028c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.196+0000 7f5719bd7640 1 --2- 172.21.15.70:0/4166967018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaaf6a20 0x5566aab028c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.196+0000 7f5719bd7640 1 -- 172.21.15.70:0/4166967018 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566aa7613f0 con 0x5566aaae8a20 2024-04-01T01:44:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.197+0000 7f5719bd7640 1 --2- 172.21.15.70:0/4166967018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaae8a20 0x5566aaafb340 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f570c009bf0 tx=0x7f570c005d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6d47eee16644a1e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.197+0000 7f56fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:44:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.197+0000 7f56fd7fa640 1 -- 172.21.15.70:0/4166967018 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c050060 con 0x5566aaae8a20 2024-04-01T01:44:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.197+0000 7f56fd7fa640 1 -- 172.21.15.70:0/4166967018 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f570c0346e0 con 0x5566aaae8a20 2024-04-01T01:44:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.197+0000 7f571ea19740 1 -- 172.21.15.70:0/4166967018 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f571000ae50 con 0x5566aaae8a20 2024-04-01T01:44:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.197+0000 7f56fd7fa640 1 -- 172.21.15.70:0/4166967018 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f570c044070 con 0x5566aaae8a20 2024-04-01T01:44:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.197+0000 7f571ea19740 10 client.28742 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:26.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.197+0000 7f571ea19740 1 -- 172.21.15.70:0/4166967018 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5566aa82b7d0 con 0x5566aaae8a20 2024-04-01T01:44:26.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.197+0000 7f56fd7fa640 1 -- 172.21.15.70:0/4166967018 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f571000ae50 con 0x5566aaae8a20 2024-04-01T01:44:26.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.198+0000 7f56fd7fa640 1 -- 172.21.15.70:0/4166967018 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f570c0329b0 con 0x5566aaae8a20 2024-04-01T01:44:26.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.198+0000 7f571ea19740 10 client.28742 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:26.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.198+0000 7f571ea19740 10 client.28742 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:26.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.198+0000 7f571ea19740 20 client.28742 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:26.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.198+0000 7f571ea19740 1 --2- 172.21.15.70:0/4166967018 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5566aab06ec0 0x5566aab09380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.198+0000 7f571ea19740 4 client.28742 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:26.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.198+0000 7f571ea19740 1 -- 172.21.15.70:0/4166967018 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566aaa5a6f0 con 0x5566aab06ec0 2024-04-01T01:44:26.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.198+0000 7f57193d6640 1 --2- 172.21.15.70:0/4166967018 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5566aab06ec0 0x5566aab09380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.199+0000 7f56fd7fa640 10 client.28742 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:26.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.199+0000 7f57193d6640 1 --2- 172.21.15.70:0/4166967018 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5566aab06ec0 0x5566aab09380 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f7c30d606e5fe95 server_cookie=3ebda631aede641c in_seq=0 out_seq=0 2024-04-01T01:44:26.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.200+0000 7f56fd7fa640 1 -- 172.21.15.70:0/4166967018 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5566aaa5a6f0 con 0x5566aab06ec0 2024-04-01T01:44:26.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.200+0000 7f56fd7fa640 10 client.28742 handle_command_reply: tid=0 2024-04-01T01:44:26.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.200+0000 7f571ea19740 1 client.28742 shutdown 2024-04-01T01:44:26.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.200+0000 7f571ea19740 1 -- 172.21.15.70:0/4166967018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaae8a20 msgr2=0x5566aaafb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:26.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.200+0000 7f571ea19740 1 --2- 172.21.15.70:0/4166967018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaae8a20 0x5566aaafb340 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f570c009bf0 tx=0x7f570c005d80 comp rx=0 tx=0).stop 2024-04-01T01:44:26.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.200+0000 7f571ea19740 1 -- 172.21.15.70:0/4166967018 shutdown_connections 2024-04-01T01:44:26.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.200+0000 7f571ea19740 1 --2- 172.21.15.70:0/4166967018 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5566aab06ec0 0x5566aab09380 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.200+0000 7f571ea19740 1 --2- 172.21.15.70:0/4166967018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaaf6a20 0x5566aab028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.200+0000 7f56fd7fa640 0 client.28742 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:26.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.200+0000 7f571ea19740 1 --2- 172.21.15.70:0/4166967018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae9380 0x5566aaafb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.200+0000 7f571ea19740 1 --2- 172.21.15.70:0/4166967018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaae8a20 0x5566aaafb340 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.200+0000 7f571ea19740 1 -- 172.21.15.70:0/4166967018 >> 172.21.15.70:0/4166967018 conn(0x5566aaae51d0 msgr2=0x5566aaae36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:26.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.200+0000 7f571ea19740 1 -- 172.21.15.70:0/4166967018 shutdown_connections 2024-04-01T01:44:26.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.200+0000 7f571ea19740 1 -- 172.21.15.70:0/4166967018 wait complete. 2024-04-01T01:44:26.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.201+0000 7f571ea19740 20 client.28742 trim_cache size 0 max 16384 2024-04-01T01:44:26.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.228+0000 7f571ea19740 1 Processor -- start 2024-04-01T01:44:26.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.228+0000 7f571ea19740 1 -- start start 2024-04-01T01:44:26.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.228+0000 7f571ea19740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae8a20 0x5566aaae3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.229+0000 7f571ea19740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae9380 0x5566aab02880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.229+0000 7f571ea19740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 0x5566aab0dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.229+0000 7f5719bd7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae8a20 0x5566aaae3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.229+0000 7f5719bd7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae8a20 0x5566aaae3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:53176/0 (socket says 172.21.15.70:53176) 2024-04-01T01:44:26.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.229+0000 7f57193d6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae9380 0x5566aab02880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.229+0000 7f57193d6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae9380 0x5566aab02880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35836/0 (socket says 172.21.15.70:35836) 2024-04-01T01:44:26.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.229+0000 7f5719bd7640 1 -- 172.21.15.70:0/358896590 learned_addr learned my addr 172.21.15.70:0/358896590 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:26.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.229+0000 7f571ea19740 1 -- 172.21.15.70:0/358896590 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57141b0660 con 0x5566aaae9380 2024-04-01T01:44:26.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.229+0000 7f571ea19740 1 -- 172.21.15.70:0/358896590 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f57141b21a0 con 0x5566aaaf6a20 2024-04-01T01:44:26.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.229+0000 7f571ea19740 1 -- 172.21.15.70:0/358896590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f57141b0d30 con 0x5566aaae8a20 2024-04-01T01:44:26.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.229+0000 7f571a3d8640 1 --2- 172.21.15.70:0/358896590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 0x5566aab0dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.229+0000 7f5719bd7640 1 -- 172.21.15.70:0/358896590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 msgr2=0x5566aab0dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:26.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.229+0000 7f5719bd7640 1 --2- 172.21.15.70:0/358896590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 0x5566aab0dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.229+0000 7f5719bd7640 1 -- 172.21.15.70:0/358896590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae9380 msgr2=0x5566aab02880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:26.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.229+0000 7f5719bd7640 1 --2- 172.21.15.70:0/358896590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae9380 0x5566aab02880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.230+0000 7f5719bd7640 1 -- 172.21.15.70:0/358896590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566aab057e0 con 0x5566aaae8a20 2024-04-01T01:44:26.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.230+0000 7f57193d6640 1 --2- 172.21.15.70:0/358896590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae9380 0x5566aab02880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:26.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.230+0000 7f571a3d8640 1 --2- 172.21.15.70:0/358896590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 0x5566aab0dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:44:26.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.230+0000 7f5719bd7640 1 --2- 172.21.15.70:0/358896590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae8a20 0x5566aaae3710 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f570c0096c0 tx=0x7f570c003d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3ae75d096ba74db server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:26.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.230+0000 7f56ff7fe640 1 -- 172.21.15.70:0/358896590 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c050070 con 0x5566aaae8a20 2024-04-01T01:44:26.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.231+0000 7f56ff7fe640 1 -- 172.21.15.70:0/358896590 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f570c03fd70 con 0x5566aaae8a20 2024-04-01T01:44:26.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.231+0000 7f56ff7fe640 1 -- 172.21.15.70:0/358896590 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c039040 con 0x5566aaae8a20 2024-04-01T01:44:26.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.232+0000 7f571ea19740 1 -- 172.21.15.70:0/358896590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae8a20 msgr2=0x5566aaae3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:26.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.232+0000 7f571ea19740 1 --2- 172.21.15.70:0/358896590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae8a20 0x5566aaae3710 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f570c0096c0 tx=0x7f570c003d70 comp rx=0 tx=0).stop 2024-04-01T01:44:26.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.232+0000 7f571ea19740 1 -- 172.21.15.70:0/358896590 shutdown_connections 2024-04-01T01:44:26.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.232+0000 7f571ea19740 1 --2- 172.21.15.70:0/358896590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 0x5566aab0dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.232+0000 7f571ea19740 1 --2- 172.21.15.70:0/358896590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae9380 0x5566aab02880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.232+0000 7f571ea19740 1 --2- 172.21.15.70:0/358896590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae8a20 0x5566aaae3710 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.232+0000 7f571ea19740 1 -- 172.21.15.70:0/358896590 >> 172.21.15.70:0/358896590 conn(0x5566aaae51d0 msgr2=0x5566aaae7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:26.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.232+0000 7f571ea19740 1 -- 172.21.15.70:0/358896590 shutdown_connections 2024-04-01T01:44:26.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.232+0000 7f571ea19740 1 -- 172.21.15.70:0/358896590 wait complete. 2024-04-01T01:44:26.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.233+0000 7f571ea19740 1 -- start start 2024-04-01T01:44:26.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.233+0000 7f571ea19740 1 Processor -- start 2024-04-01T01:44:26.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.233+0000 7f571ea19740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae8a20 0x5566aab00770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.233+0000 7f571ea19740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae9380 0x5566aab00cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571ea19740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 0x5566aab01cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571ea19740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57141b1ad0 con 0x5566aaae8a20 2024-04-01T01:44:26.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571ea19740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f57141b1400 con 0x5566aaaf6a20 2024-04-01T01:44:26.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571ea19740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f57141ae640 con 0x5566aaae9380 2024-04-01T01:44:26.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571a3d8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 0x5566aab01cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571a3d8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 0x5566aab01cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53736/0 (socket says 172.21.15.70:53736) 2024-04-01T01:44:26.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571a3d8640 1 -- 172.21.15.70:0/3099410716 learned_addr learned my addr 172.21.15.70:0/3099410716 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:26.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571ea19740 10 client.0 _ll_register_callbacks cb 0x7f570c003de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:26.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f57193d6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae9380 0x5566aab00cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f5719bd7640 1 --2- 172.21.15.70:0/3099410716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae8a20 0x5566aab00770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571a3d8640 1 -- 172.21.15.70:0/3099410716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae9380 msgr2=0x5566aab00cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:26.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571a3d8640 1 --2- 172.21.15.70:0/3099410716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae9380 0x5566aab00cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571a3d8640 1 -- 172.21.15.70:0/3099410716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae8a20 msgr2=0x5566aab00770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:26.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571a3d8640 1 --2- 172.21.15.70:0/3099410716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae8a20 0x5566aab00770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571a3d8640 1 -- 172.21.15.70:0/3099410716 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566aab022b0 con 0x5566aaaf6a20 2024-04-01T01:44:26.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571a3d8640 1 --2- 172.21.15.70:0/3099410716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 0x5566aab01cc0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f57040972b0 tx=0x7f570409af90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f93fd7d7b803979 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:26.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f56fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:44:26.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f56fd7fa640 1 -- 172.21.15.70:0/3099410716 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57040b8060 con 0x5566aaaf6a20 2024-04-01T01:44:26.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f56fd7fa640 1 -- 172.21.15.70:0/3099410716 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f570409a2a0 con 0x5566aaaf6a20 2024-04-01T01:44:26.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571ea19740 1 -- 172.21.15.70:0/3099410716 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f570c03fd70 con 0x5566aaaf6a20 2024-04-01T01:44:26.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f56fd7fa640 1 -- 172.21.15.70:0/3099410716 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f57040a8050 con 0x5566aaaf6a20 2024-04-01T01:44:26.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571ea19740 10 client.28745 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:26.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f571ea19740 1 -- 172.21.15.70:0/3099410716 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5566aab6da20 con 0x5566aaaf6a20 2024-04-01T01:44:26.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.234+0000 7f56fd7fa640 1 -- 172.21.15.70:0/3099410716 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f570c03fd70 con 0x5566aaaf6a20 2024-04-01T01:44:26.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.235+0000 7f56fd7fa640 1 -- 172.21.15.70:0/3099410716 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5704099e30 con 0x5566aaaf6a20 2024-04-01T01:44:26.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.235+0000 7f571ea19740 10 client.28745 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:26.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.235+0000 7f571ea19740 10 client.28745 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:26.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.235+0000 7f571ea19740 20 client.28745 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:26.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.235+0000 7f571ea19740 1 --2- 172.21.15.70:0/3099410716 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5566aab6e3f0 0x5566aab70810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.235+0000 7f571ea19740 4 client.28745 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:44:26.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.235+0000 7f571ea19740 1 -- 172.21.15.70:0/3099410716 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5566aaa54a40 con 0x5566aab6e3f0 2024-04-01T01:44:26.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.235+0000 7f5719bd7640 1 --2- 172.21.15.70:0/3099410716 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5566aab6e3f0 0x5566aab70810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.236+0000 7f56fd7fa640 10 client.28745 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:26.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.236+0000 7f5719bd7640 1 --2- 172.21.15.70:0/3099410716 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5566aab6e3f0 0x5566aab70810 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c271fb6208992845 server_cookie=f84969b6d65504b7 in_seq=0 out_seq=0 2024-04-01T01:44:26.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.236+0000 7f56fd7fa640 1 -- 172.21.15.70:0/3099410716 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5566aaa54a40 con 0x5566aab6e3f0 2024-04-01T01:44:26.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.236+0000 7f56fd7fa640 10 client.28745 handle_command_reply: tid=0 2024-04-01T01:44:26.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.236+0000 7f571ea19740 1 client.28745 shutdown 2024-04-01T01:44:26.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.237+0000 7f571ea19740 1 -- 172.21.15.70:0/3099410716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 msgr2=0x5566aab01cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:26.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.237+0000 7f571ea19740 1 --2- 172.21.15.70:0/3099410716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 0x5566aab01cc0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f57040972b0 tx=0x7f570409af90 comp rx=0 tx=0).stop 2024-04-01T01:44:26.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.238+0000 7f571ea19740 1 -- 172.21.15.70:0/3099410716 shutdown_connections 2024-04-01T01:44:26.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.238+0000 7f571ea19740 1 --2- 172.21.15.70:0/3099410716 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5566aab6e3f0 0x5566aab70810 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.238+0000 7f56fd7fa640 0 client.28745 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:26.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.238+0000 7f571ea19740 1 --2- 172.21.15.70:0/3099410716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566aaaf6a20 0x5566aab01cc0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.238+0000 7f571ea19740 1 --2- 172.21.15.70:0/3099410716 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566aaae9380 0x5566aab00cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.238+0000 7f571ea19740 1 --2- 172.21.15.70:0/3099410716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566aaae8a20 0x5566aab00770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.238+0000 7f571ea19740 1 -- 172.21.15.70:0/3099410716 >> 172.21.15.70:0/3099410716 conn(0x5566aaae51d0 msgr2=0x5566aaae2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:26.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.239+0000 7f571ea19740 1 -- 172.21.15.70:0/3099410716 shutdown_connections 2024-04-01T01:44:26.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.239+0000 7f571ea19740 1 -- 172.21.15.70:0/3099410716 wait complete. 2024-04-01T01:44:26.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.239+0000 7f571ea19740 20 client.28745 trim_cache size 0 max 16384 2024-04-01T01:44:26.241 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:44:26.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.244+0000 7f571be3b640 1 -- 172.21.15.70:0/3723327069 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57000a2590 msgr2=0x7f57000a4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:26.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.244+0000 7f571be3b640 1 --2- 172.21.15.70:0/3723327069 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57000a2590 0x7f57000a4a50 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f570c006010 tx=0x7f570c009630 comp rx=0 tx=0).stop 2024-04-01T01:44:26.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.244+0000 7f571be3b640 1 -- 172.21.15.70:0/3723327069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f571410dd20 msgr2=0x7f57141a58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:26.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.244+0000 7f571be3b640 1 --2- 172.21.15.70:0/3723327069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f571410dd20 0x7f57141a58a0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f570400c7c0 tx=0x7f570400cc80 comp rx=0 tx=0).stop 2024-04-01T01:44:26.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.245+0000 7f571be3b640 1 -- 172.21.15.70:0/3723327069 shutdown_connections 2024-04-01T01:44:26.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.245+0000 7f571be3b640 1 --2- 172.21.15.70:0/3723327069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f571410dd20 0x7f57141a58a0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.245+0000 7f571be3b640 1 --2- 172.21.15.70:0/3723327069 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f57000a2590 0x7f57000a4a50 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.245+0000 7f571be3b640 1 --2- 172.21.15.70:0/3723327069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f571410b260 0x7f571419e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.245+0000 7f571be3b640 1 --2- 172.21.15.70:0/3723327069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5714108930 0x7f571419e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.245+0000 7f571be3b640 1 -- 172.21.15.70:0/3723327069 >> 172.21.15.70:0/3723327069 conn(0x7f571406e630 msgr2=0x7f571410aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:26.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.246+0000 7f571be3b640 1 -- 172.21.15.70:0/3723327069 shutdown_connections 2024-04-01T01:44:26.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:26.246+0000 7f571be3b640 1 -- 172.21.15.70:0/3723327069 wait complete. 2024-04-01T01:44:26.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:25 smithi189 ceph-mon[35753]: pgmap v687: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 12 MiB/s wr, 742 op/s 2024-04-01T01:44:26.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:25 smithi081 ceph-mon[35142]: pgmap v687: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 12 MiB/s wr, 742 op/s 2024-04-01T01:44:26.898 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 580.0 s, 893.1 tps, lat 111.915 ms stddev 65.028 2024-04-01T01:44:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.975+0000 7fc7da925640 1 -- 172.21.15.70:0/3763109255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d41007e0 msgr2=0x7fc7d4100c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.975+0000 7fc7da925640 1 --2- 172.21.15.70:0/3763109255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d41007e0 0x7fc7d4100c60 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009900 tx=0x7fc7c802bc80 comp rx=0 tx=0).stop 2024-04-01T01:44:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.975+0000 7fc7da925640 1 -- 172.21.15.70:0/3763109255 shutdown_connections 2024-04-01T01:44:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.975+0000 7fc7da925640 1 --2- 172.21.15.70:0/3763109255 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7d41011a0 0x7fc7d4105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.975+0000 7fc7da925640 1 --2- 172.21.15.70:0/3763109255 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d41007e0 0x7fc7d4100c60 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.975+0000 7fc7da925640 1 --2- 172.21.15.70:0/3763109255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d40ffa80 0x7fc7d40ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.975+0000 7fc7da925640 1 -- 172.21.15.70:0/3763109255 >> 172.21.15.70:0/3763109255 conn(0x7fc7d40fb3b0 msgr2=0x7fc7d40fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:26.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.976+0000 7fc7da925640 1 -- 172.21.15.70:0/3763109255 shutdown_connections 2024-04-01T01:44:26.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.976+0000 7fc7da925640 1 -- 172.21.15.70:0/3763109255 wait complete. 2024-04-01T01:44:26.977 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.977+0000 7fc7da925640 1 Processor -- start 2024-04-01T01:44:26.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.977+0000 7fc7da925640 1 -- start start 2024-04-01T01:44:26.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.978+0000 7fc7da925640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7d40ffa80 0x7fc7d419df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.978+0000 7fc7da925640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d41007e0 0x7fc7d419e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.978+0000 7fc7da925640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d41011a0 0x7fc7d41a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.978+0000 7fc7da925640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d41139d0 con 0x7fc7d40ffa80 2024-04-01T01:44:26.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.978+0000 7fc7da925640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d4113850 con 0x7fc7d41011a0 2024-04-01T01:44:26.978 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.978+0000 7fc7da925640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d4113b50 con 0x7fc7d41007e0 2024-04-01T01:44:26.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.978+0000 7fc7d3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7d40ffa80 0x7fc7d419df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.978+0000 7fc7d3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7d40ffa80 0x7fc7d419df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59502/0 (socket says 172.21.15.70:59502) 2024-04-01T01:44:26.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.978+0000 7fc7d3fff640 1 -- 172.21.15.70:0/1879064696 learned_addr learned my addr 172.21.15.70:0/1879064696 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:26.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.978+0000 7fc7d37fe640 1 --2- 172.21.15.70:0/1879064696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d41007e0 0x7fc7d419e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.978+0000 7fc7d3fff640 1 -- 172.21.15.70:0/1879064696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d41007e0 msgr2=0x7fc7d419e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:26.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.978+0000 7fc7d3fff640 1 --2- 172.21.15.70:0/1879064696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d41007e0 0x7fc7d419e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.978+0000 7fc7d3fff640 1 -- 172.21.15.70:0/1879064696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d41011a0 msgr2=0x7fc7d41a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:26.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.978+0000 7fc7d3fff640 1 --2- 172.21.15.70:0/1879064696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d41011a0 0x7fc7d41a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.979 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.978+0000 7fc7d3fff640 1 -- 172.21.15.70:0/1879064696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d41a5b70 con 0x7fc7d40ffa80 2024-04-01T01:44:26.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.979+0000 7fc7d3fff640 1 --2- 172.21.15.70:0/1879064696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7d40ffa80 0x7fc7d419df50 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00eab0 tx=0x7fc7cc00ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:26.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.979+0000 7fc7d17fa640 1 -- 172.21.15.70:0/1879064696 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc00ce40 con 0x7fc7d40ffa80 2024-04-01T01:44:26.980 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.979+0000 7fc7da925640 1 -- 172.21.15.70:0/1879064696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d41a5e60 con 0x7fc7d40ffa80 2024-04-01T01:44:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.979+0000 7fc7da925640 1 -- 172.21.15.70:0/1879064696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d41a63a0 con 0x7fc7d40ffa80 2024-04-01T01:44:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.979+0000 7fc7d17fa640 1 -- 172.21.15.70:0/1879064696 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7cc011920 con 0x7fc7d40ffa80 2024-04-01T01:44:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.979+0000 7fc7d17fa640 1 -- 172.21.15.70:0/1879064696 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc010a00 con 0x7fc7d40ffa80 2024-04-01T01:44:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.983+0000 7fc7d17fa640 1 -- 172.21.15.70:0/1879064696 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fc7cc010ba0 con 0x7fc7d40ffa80 2024-04-01T01:44:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.983+0000 7fc7dd5e5740 1 Processor -- start 2024-04-01T01:44:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.984+0000 7fc7dd5e5740 1 -- start start 2024-04-01T01:44:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.984+0000 7fc7dd5e5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e58d0 0x5654363e5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.984+0000 7fc7dd5e5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e62c0 0x5654363eadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.984+0000 7fc7dd5e5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363eb430 0x5654363ed8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.984+0000 7fc7dd5e5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5654363498f0 con 0x5654363eb430 2024-04-01T01:44:26.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.984+0000 7fc7dd5e5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5654362f28d0 con 0x5654363e62c0 2024-04-01T01:44:26.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.984+0000 7fc7dd5e5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x565436345990 con 0x5654363e58d0 2024-04-01T01:44:26.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.984+0000 7fc7d17fa640 1 --2- 172.21.15.70:0/1879064696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc7bc0a2690 0x7fc7bc0a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.984+0000 7fc7d8ec2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e58d0 0x5654363e5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.984+0000 7fc7d8ec2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e58d0 0x5654363e5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57804/0 (socket says 172.21.15.70:57804) 2024-04-01T01:44:26.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.984+0000 7fc7d17fa640 1 -- 172.21.15.70:0/1879064696 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc7cc0cf0d0 con 0x7fc7d40ffa80 2024-04-01T01:44:26.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.984+0000 7fc7d8ec2640 1 -- 172.21.15.70:0/1222683042 learned_addr learned my addr 172.21.15.70:0/1222683042 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:26.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.985+0000 7fc7d8ec2640 1 --2- 172.21.15.70:0/1879064696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc7bc0a2690 0x7fc7bc0a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.985+0000 7fc7d3fff640 1 --2- 172.21.15.70:0/1222683042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e62c0 0x5654363eadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.985+0000 7fc7d8ec2640 1 -- 172.21.15.70:0/1222683042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e62c0 msgr2=0x5654363eadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:26.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.985+0000 7fc7d8ec2640 1 --2- 172.21.15.70:0/1222683042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e62c0 0x5654363eadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.985+0000 7fc7d8ec2640 1 -- 172.21.15.70:0/1222683042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363eb430 msgr2=0x5654363ed8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:26.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.985+0000 7fc7d8ec2640 1 --2- 172.21.15.70:0/1222683042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363eb430 0x5654363ed8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.985+0000 7fc7d8ec2640 1 -- 172.21.15.70:0/1222683042 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56543605e210 con 0x5654363e58d0 2024-04-01T01:44:26.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.985+0000 7fc7d8ec2640 1 --2- 172.21.15.70:0/1222683042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e58d0 0x5654363e5cf0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4012310 tx=0x7fc7c40127d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2e464f3d5b17358 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:26.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.986+0000 7fc7ba7fc640 1 -- 172.21.15.70:0/1222683042 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c400bd50 con 0x5654363e58d0 2024-04-01T01:44:26.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.986+0000 7fc7ba7fc640 1 -- 172.21.15.70:0/1222683042 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7c4020990 con 0x5654363e58d0 2024-04-01T01:44:26.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.986+0000 7fc7ba7fc640 1 -- 172.21.15.70:0/1222683042 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c40167a0 con 0x5654363e58d0 2024-04-01T01:44:26.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.987+0000 7fc7d8ec2640 1 --2- 172.21.15.70:0/1879064696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc7bc0a2690 0x7fc7bc0a4b50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc7c401d000 tx=0x7fc7c4018000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:26.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.987+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/1222683042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e58d0 msgr2=0x5654363e5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:26.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.987+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/1222683042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e58d0 0x5654363e5cf0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4012310 tx=0x7fc7c40127d0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.987+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/1222683042 shutdown_connections 2024-04-01T01:44:26.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.987+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/1222683042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363eb430 0x5654363ed8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.987+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/1222683042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e62c0 0x5654363eadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.987+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/1222683042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e58d0 0x5654363e5cf0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.987+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/1222683042 >> 172.21.15.70:0/1222683042 conn(0x5654363e0e40 msgr2=0x5654363e1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:26.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.987+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/1222683042 shutdown_connections 2024-04-01T01:44:26.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.987+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/1222683042 wait complete. 2024-04-01T01:44:26.989 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.988+0000 7fc7dd5e5740 1 -- start start 2024-04-01T01:44:26.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.988+0000 7fc7dd5e5740 1 Processor -- start 2024-04-01T01:44:26.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7dd5e5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e58d0 0x565436400bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7dd5e5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e62c0 0x565436401130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7dd5e5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363eb430 0x565436408100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e58d0 0x565436400bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e58d0 0x565436400bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33458/0 (socket says 172.21.15.70:33458) 2024-04-01T01:44:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7d37fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e62c0 0x565436401130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7d37fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e62c0 0x565436401130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57814/0 (socket says 172.21.15.70:57814) 2024-04-01T01:44:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7d3fff640 1 -- 172.21.15.70:0/125211073 learned_addr learned my addr 172.21.15.70:0/125211073 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/125211073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5654363ff3f0 con 0x5654363eb430 2024-04-01T01:44:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/125211073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5654362e7f80 con 0x5654363e58d0 2024-04-01T01:44:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/125211073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5654363ff570 con 0x5654363e62c0 2024-04-01T01:44:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7d8ec2640 1 --2- 172.21.15.70:0/125211073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363eb430 0x565436408100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7d3fff640 1 -- 172.21.15.70:0/125211073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e62c0 msgr2=0x565436401130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7d3fff640 1 --2- 172.21.15.70:0/125211073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e62c0 0x565436401130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7d3fff640 1 -- 172.21.15.70:0/125211073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363eb430 msgr2=0x565436408100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7d3fff640 1 --2- 172.21.15.70:0/125211073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363eb430 0x565436408100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7d3fff640 1 -- 172.21.15.70:0/125211073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56543612a620 con 0x5654363e58d0 2024-04-01T01:44:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.989+0000 7fc7dd5e5740 10 client.0 _ll_register_callbacks cb 0x565436340290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.990+0000 7fc7d3fff640 1 --2- 172.21.15.70:0/125211073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e58d0 0x565436400bf0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc099100 tx=0x7fc7cc011e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71dab53453f3fbb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.990+0000 7fc7b37fe640 10 client.28754 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:44:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.990+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/125211073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7c4020990 con 0x5654363e58d0 2024-04-01T01:44:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.990+0000 7fc7b37fe640 1 -- 172.21.15.70:0/125211073 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc09c730 con 0x5654363e58d0 2024-04-01T01:44:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.990+0000 7fc7b37fe640 1 -- 172.21.15.70:0/125211073 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc7cc098da0 con 0x5654363e58d0 2024-04-01T01:44:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.990+0000 7fc7b37fe640 1 -- 172.21.15.70:0/125211073 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc7cc0a0880 con 0x5654363e58d0 2024-04-01T01:44:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.990+0000 7fc7dd5e5740 10 client.28754 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.990+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/125211073 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=31}) -- 0x56543640b050 con 0x5654363e58d0 2024-04-01T01:44:26.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.991+0000 7fc7b37fe640 1 -- 172.21.15.70:0/125211073 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc7cc0b7030 con 0x5654363e58d0 2024-04-01T01:44:26.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.991+0000 7fc7dd5e5740 10 client.28754 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:26.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.991+0000 7fc7dd5e5740 10 client.28754 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:26.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.991+0000 7fc7dd5e5740 20 client.28754 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:26.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.991+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/125211073 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56543640c670 0x56543640eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:26.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.991+0000 7fc7dd5e5740 4 client.28754 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:26.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.992+0000 7fc7d37fe640 1 --2- 172.21.15.70:0/125211073 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56543640c670 0x56543640eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:26.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.992+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/125211073 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc7c4020990 con 0x56543640c670 2024-04-01T01:44:26.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.992+0000 7fc7d37fe640 1 --2- 172.21.15.70:0/125211073 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56543640c670 0x56543640eb30 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18c992c3505ed217 server_cookie=caa01e4d81cdcb3c in_seq=0 out_seq=0 2024-04-01T01:44:26.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.992+0000 7fc7b37fe640 10 client.28754 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.995+0000 7fc7b37fe640 1 -- 172.21.15.70:0/125211073 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fc7c4020990 con 0x56543640c670 2024-04-01T01:44:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.995+0000 7fc7b37fe640 10 client.28754 handle_command_reply: tid=0 2024-04-01T01:44:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.996+0000 7fc7dd5e5740 1 client.28754 shutdown 2024-04-01T01:44:26.997 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.997+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/125211073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e58d0 msgr2=0x565436400bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:26.997 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.997+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/125211073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e58d0 0x565436400bf0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc099100 tx=0x7fc7cc011e00 comp rx=0 tx=0).stop 2024-04-01T01:44:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.997+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/125211073 shutdown_connections 2024-04-01T01:44:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.998+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/125211073 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56543640c670 0x56543640eb30 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.998+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/125211073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363eb430 0x565436408100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.998+0000 7fc7b37fe640 0 client.28754 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.998+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/125211073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e62c0 0x565436401130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.998+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/125211073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e58d0 0x565436400bf0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.998+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/125211073 >> 172.21.15.70:0/125211073 conn(0x5654363e0e40 msgr2=0x5654363dff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.999+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/125211073 shutdown_connections 2024-04-01T01:44:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.999+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/125211073 wait complete. 2024-04-01T01:44:27.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:26.999+0000 7fc7dd5e5740 20 client.28754 trim_cache size 0 max 16384 2024-04-01T01:44:27.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.032+0000 7fc7dd5e5740 1 Processor -- start 2024-04-01T01:44:27.032 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.032+0000 7fc7dd5e5740 1 -- start start 2024-04-01T01:44:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.032+0000 7fc7dd5e5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e58d0 0x565436400800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.032+0000 7fc7dd5e5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e62c0 0x5654364080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.032+0000 7fc7dd5e5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363eb430 0x565436400d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.032+0000 7fc7dd5e5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c000fc40 con 0x5654363eb430 2024-04-01T01:44:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.032+0000 7fc7dd5e5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c0011780 con 0x5654363e62c0 2024-04-01T01:44:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.032+0000 7fc7dd5e5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c0010310 con 0x5654363e58d0 2024-04-01T01:44:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.032+0000 7fc7d3fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e58d0 0x565436400800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.032+0000 7fc7d37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e62c0 0x5654364080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.032+0000 7fc7d3fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e58d0 0x565436400800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57828/0 (socket says 172.21.15.70:57828) 2024-04-01T01:44:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.032+0000 7fc7d3fff640 1 -- 172.21.15.70:0/1337708145 learned_addr learned my addr 172.21.15.70:0/1337708145 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.032+0000 7fc7d8ec2640 1 --2- 172.21.15.70:0/1337708145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363eb430 0x565436400d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.033+0000 7fc7d3fff640 1 -- 172.21.15.70:0/1337708145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e62c0 msgr2=0x5654364080c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.033+0000 7fc7d3fff640 1 --2- 172.21.15.70:0/1337708145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e62c0 0x5654364080c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.033+0000 7fc7d3fff640 1 -- 172.21.15.70:0/1337708145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363eb430 msgr2=0x565436400d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.033+0000 7fc7d3fff640 1 --2- 172.21.15.70:0/1337708145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363eb430 0x565436400d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.033+0000 7fc7d3fff640 1 -- 172.21.15.70:0/1337708145 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654363e5cb0 con 0x5654363e58d0 2024-04-01T01:44:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.033+0000 7fc7d8ec2640 1 --2- 172.21.15.70:0/1337708145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363eb430 0x565436400d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.033+0000 7fc7d3fff640 1 --2- 172.21.15.70:0/1337708145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e58d0 0x565436400800 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc09c430 tx=0x7fc7cc087000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af3db150b99433b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.033+0000 7fc7b9ffb640 1 -- 172.21.15.70:0/1337708145 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc080070 con 0x5654363e58d0 2024-04-01T01:44:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.034+0000 7fc7b9ffb640 1 -- 172.21.15.70:0/1337708145 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7cc0cc6e0 con 0x5654363e58d0 2024-04-01T01:44:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.034+0000 7fc7b9ffb640 1 -- 172.21.15.70:0/1337708145 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc098480 con 0x5654363e58d0 2024-04-01T01:44:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.035+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/1337708145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e58d0 msgr2=0x565436400800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.035+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/1337708145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e58d0 0x565436400800 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc09c430 tx=0x7fc7cc087000 comp rx=0 tx=0).stop 2024-04-01T01:44:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.035+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/1337708145 shutdown_connections 2024-04-01T01:44:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.035+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/1337708145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363eb430 0x565436400d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.035+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/1337708145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363e62c0 0x5654364080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.036+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/1337708145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654363e58d0 0x565436400800 secure :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc09c430 tx=0x7fc7cc087000 comp rx=0 tx=0).stop 2024-04-01T01:44:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.036+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/1337708145 >> 172.21.15.70:0/1337708145 conn(0x5654363e0e40 msgr2=0x5654363e1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.036+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/1337708145 shutdown_connections 2024-04-01T01:44:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.036+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/1337708145 wait complete. 2024-04-01T01:44:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.036+0000 7fc7dd5e5740 1 -- start start 2024-04-01T01:44:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.037+0000 7fc7dd5e5740 1 Processor -- start 2024-04-01T01:44:27.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.038+0000 7fc7dd5e5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363e62c0 0x565436405b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.038+0000 7fc7dd5e5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363eb430 0x565436406080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.038 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.038+0000 7fc7dd5e5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56543640c670 0x565436407020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.038+0000 7fc7d37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363eb430 0x565436406080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.039 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.038+0000 7fc7d37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363eb430 0x565436406080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33474/0 (socket says 172.21.15.70:33474) 2024-04-01T01:44:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.039+0000 7fc7d37fe640 1 -- 172.21.15.70:0/750743119 learned_addr learned my addr 172.21.15.70:0/750743119 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.039+0000 7fc7d8ec2640 1 --2- 172.21.15.70:0/750743119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56543640c670 0x565436407020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.039+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/750743119 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c00110b0 con 0x5654363e62c0 2024-04-01T01:44:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.039+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/750743119 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c00109e0 con 0x5654363eb430 2024-04-01T01:44:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.039+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/750743119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c000dc20 con 0x56543640c670 2024-04-01T01:44:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.039+0000 7fc7d3fff640 1 --2- 172.21.15.70:0/750743119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363e62c0 0x565436405b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.039+0000 7fc7dd5e5740 10 client.0 _ll_register_callbacks cb 0x7fc7cc09c300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.039+0000 7fc7d37fe640 1 -- 172.21.15.70:0/750743119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56543640c670 msgr2=0x565436407020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.039+0000 7fc7d37fe640 1 --2- 172.21.15.70:0/750743119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56543640c670 0x565436407020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.039+0000 7fc7d37fe640 1 -- 172.21.15.70:0/750743119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363e62c0 msgr2=0x565436405b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.039+0000 7fc7d37fe640 1 --2- 172.21.15.70:0/750743119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363e62c0 0x565436405b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.039+0000 7fc7d37fe640 1 -- 172.21.15.70:0/750743119 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565436407610 con 0x5654363eb430 2024-04-01T01:44:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.040+0000 7fc7d37fe640 1 --2- 172.21.15.70:0/750743119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363eb430 0x565436406080 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8000c00 tx=0x7fc7c8004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e6b2daf235d6aae server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.041+0000 7fc7b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:44:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.041+0000 7fc7b37fe640 1 -- 172.21.15.70:0/750743119 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c804d070 con 0x5654363eb430 2024-04-01T01:44:27.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.041+0000 7fc7b37fe640 1 -- 172.21.15.70:0/750743119 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc7c8004480 con 0x5654363eb430 2024-04-01T01:44:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.041+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/750743119 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7cc0cc6e0 con 0x5654363eb430 2024-04-01T01:44:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.041+0000 7fc7b37fe640 1 -- 172.21.15.70:0/750743119 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc7c8058050 con 0x5654363eb430 2024-04-01T01:44:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.041+0000 7fc7dd5e5740 10 client.28760 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.042+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/750743119 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x565436472d40 con 0x5654363eb430 2024-04-01T01:44:27.043 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.042+0000 7fc7b37fe640 1 -- 172.21.15.70:0/750743119 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7c80054f0 con 0x5654363eb430 2024-04-01T01:44:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.042+0000 7fc7b37fe640 1 -- 172.21.15.70:0/750743119 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc7c80035f0 con 0x5654363eb430 2024-04-01T01:44:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.042+0000 7fc7dd5e5740 10 client.28760 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.042+0000 7fc7dd5e5740 10 client.28760 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.042+0000 7fc7dd5e5740 20 client.28760 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.042+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/750743119 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5654364736c0 0x565436475ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.042+0000 7fc7dd5e5740 4 client.28760 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:44:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.043+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/750743119 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fc7cc0cc6e0 con 0x5654364736c0 2024-04-01T01:44:27.044 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.043+0000 7fc7d3fff640 1 --2- 172.21.15.70:0/750743119 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5654364736c0 0x565436475ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.043+0000 7fc7d3fff640 1 --2- 172.21.15.70:0/750743119 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5654364736c0 0x565436475ae0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d8e603626657c4a server_cookie=8c2d7b334b331112 in_seq=0 out_seq=0 2024-04-01T01:44:27.045 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.043+0000 7fc7b37fe640 10 client.28760 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.102+0000 7fc7b37fe640 1 -- 172.21.15.70:0/750743119 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fc7cc0cc6e0 con 0x5654364736c0 2024-04-01T01:44:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.102+0000 7fc7b37fe640 10 client.28760 handle_command_reply: tid=0 2024-04-01T01:44:27.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.102+0000 7fc7dd5e5740 1 client.28760 shutdown 2024-04-01T01:44:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.103+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/750743119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363eb430 msgr2=0x565436406080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.104+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/750743119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363eb430 0x565436406080 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8000c00 tx=0x7fc7c8004040 comp rx=0 tx=0).stop 2024-04-01T01:44:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.104+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/750743119 shutdown_connections 2024-04-01T01:44:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.104+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/750743119 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5654364736c0 0x565436475ae0 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.104+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/750743119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56543640c670 0x565436407020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.104+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/750743119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5654363eb430 0x565436406080 secure :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8000c00 tx=0x7fc7c8004040 comp rx=0 tx=0).stop 2024-04-01T01:44:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.104+0000 7fc7dd5e5740 1 --2- 172.21.15.70:0/750743119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5654363e62c0 0x565436405b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.104+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/750743119 >> 172.21.15.70:0/750743119 conn(0x5654363e0e40 msgr2=0x5654363df6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.104+0000 7fc7b37fe640 0 client.28760 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.104+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/750743119 shutdown_connections 2024-04-01T01:44:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.105+0000 7fc7dd5e5740 1 -- 172.21.15.70:0/750743119 wait complete. 2024-04-01T01:44:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.105+0000 7fc7dd5e5740 20 client.28760 trim_cache size 0 max 16384 2024-04-01T01:44:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:44:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:44:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:44:27.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:44:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.110+0000 7fc7da925640 1 -- 172.21.15.70:0/1879064696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc7bc0a2690 msgr2=0x7fc7bc0a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.110+0000 7fc7da925640 1 --2- 172.21.15.70:0/1879064696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc7bc0a2690 0x7fc7bc0a4b50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc7c401d000 tx=0x7fc7c4018000 comp rx=0 tx=0).stop 2024-04-01T01:44:27.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.110+0000 7fc7da925640 1 -- 172.21.15.70:0/1879064696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7d40ffa80 msgr2=0x7fc7d419df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.110+0000 7fc7da925640 1 --2- 172.21.15.70:0/1879064696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7d40ffa80 0x7fc7d419df50 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00eab0 tx=0x7fc7cc00ef70 comp rx=0 tx=0).stop 2024-04-01T01:44:27.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.111+0000 7fc7da925640 1 -- 172.21.15.70:0/1879064696 shutdown_connections 2024-04-01T01:44:27.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.111+0000 7fc7da925640 1 --2- 172.21.15.70:0/1879064696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7d41011a0 0x7fc7d41a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.111+0000 7fc7da925640 1 --2- 172.21.15.70:0/1879064696 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc7bc0a2690 0x7fc7bc0a4b50 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.111+0000 7fc7da925640 1 --2- 172.21.15.70:0/1879064696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d41007e0 0x7fc7d419e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.111+0000 7fc7da925640 1 --2- 172.21.15.70:0/1879064696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7d40ffa80 0x7fc7d419df50 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.111+0000 7fc7da925640 1 -- 172.21.15.70:0/1879064696 >> 172.21.15.70:0/1879064696 conn(0x7fc7d40fb3b0 msgr2=0x7fc7d40fc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:27.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.115+0000 7fc7da925640 1 -- 172.21.15.70:0/1879064696 shutdown_connections 2024-04-01T01:44:27.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:27.115+0000 7fc7da925640 1 -- 172.21.15.70:0/1879064696 wait complete. 2024-04-01T01:44:27.271 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:44:27.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:44:27.271 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:44:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.464+0000 7f6c6bfff640 1 -- 172.21.15.70:0/2921377713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c640990e0 msgr2=0x7f6c64099560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.464+0000 7f6c6bfff640 1 --2- 172.21.15.70:0/2921377713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c640990e0 0x7f6c64099560 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c009900 tx=0x7f6c5c02bc80 comp rx=0 tx=0).stop 2024-04-01T01:44:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.464+0000 7f6c6bfff640 1 -- 172.21.15.70:0/2921377713 shutdown_connections 2024-04-01T01:44:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.464+0000 7f6c6bfff640 1 --2- 172.21.15.70:0/2921377713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c64099aa0 0x7f6c640a03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.464+0000 7f6c6bfff640 1 --2- 172.21.15.70:0/2921377713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c640990e0 0x7f6c64099560 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.464+0000 7f6c6bfff640 1 --2- 172.21.15.70:0/2921377713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c64097f10 0x7f6c64098350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.464+0000 7f6c6bfff640 1 -- 172.21.15.70:0/2921377713 >> 172.21.15.70:0/2921377713 conn(0x7f6c64093510 msgr2=0x7f6c64095930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:27.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.464+0000 7f6c6bfff640 1 -- 172.21.15.70:0/2921377713 shutdown_connections 2024-04-01T01:44:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.465+0000 7f6c6bfff640 1 -- 172.21.15.70:0/2921377713 wait complete. 2024-04-01T01:44:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.465+0000 7f6c6bfff640 1 Processor -- start 2024-04-01T01:44:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.466+0000 7f6c6bfff640 1 -- start start 2024-04-01T01:44:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.466+0000 7f6c6bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c64097f10 0x7f6c64129fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.466+0000 7f6c6bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c640990e0 0x7f6c6412a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.466+0000 7f6c6bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c64099aa0 0x7f6c64131500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.466+0000 7f6c6bfff640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6c640a3bf0 con 0x7f6c640990e0 2024-04-01T01:44:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.466+0000 7f6c6bfff640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6c640a3a70 con 0x7f6c64099aa0 2024-04-01T01:44:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.466+0000 7f6c6bfff640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6c640a3d70 con 0x7f6c64097f10 2024-04-01T01:44:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.466+0000 7f6c6b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c64099aa0 0x7f6c64131500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.466+0000 7f6c6affd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c64097f10 0x7f6c64129fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.466+0000 7f6c6affd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c64097f10 0x7f6c64129fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57844/0 (socket says 172.21.15.70:57844) 2024-04-01T01:44:27.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.466+0000 7f6c6affd640 1 -- 172.21.15.70:0/1135376954 learned_addr learned my addr 172.21.15.70:0/1135376954 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:27.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.466+0000 7f6c6a7fc640 1 --2- 172.21.15.70:0/1135376954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c640990e0 0x7f6c6412a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.467+0000 7f6c6affd640 1 -- 172.21.15.70:0/1135376954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c64099aa0 msgr2=0x7f6c64131500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.467+0000 7f6c6affd640 1 --2- 172.21.15.70:0/1135376954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c64099aa0 0x7f6c64131500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.467+0000 7f6c6affd640 1 -- 172.21.15.70:0/1135376954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c640990e0 msgr2=0x7f6c6412a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.467+0000 7f6c6affd640 1 --2- 172.21.15.70:0/1135376954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c640990e0 0x7f6c6412a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.467+0000 7f6c6affd640 1 -- 172.21.15.70:0/1135376954 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c64142ba0 con 0x7f6c64097f10 2024-04-01T01:44:27.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.467+0000 7f6c6b7fe640 1 --2- 172.21.15.70:0/1135376954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c64099aa0 0x7f6c64131500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:27.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.468+0000 7f6c6affd640 1 --2- 172.21.15.70:0/1135376954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c64097f10 0x7f6c64129fa0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6c6000b470 tx=0x7f6c6000b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:27.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.468+0000 7f6c53fff640 1 -- 172.21.15.70:0/1135376954 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c60015020 con 0x7f6c64097f10 2024-04-01T01:44:27.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.468+0000 7f6c6bfff640 1 -- 172.21.15.70:0/1135376954 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c64142e90 con 0x7f6c64097f10 2024-04-01T01:44:27.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.468+0000 7f6c53fff640 1 -- 172.21.15.70:0/1135376954 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c60011d40 con 0x7f6c64097f10 2024-04-01T01:44:27.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.468+0000 7f6c6bfff640 1 -- 172.21.15.70:0/1135376954 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c64143370 con 0x7f6c64097f10 2024-04-01T01:44:27.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.468+0000 7f6c53fff640 1 -- 172.21.15.70:0/1135376954 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6000f340 con 0x7f6c64097f10 2024-04-01T01:44:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.472+0000 7f6c53fff640 1 -- 172.21.15.70:0/1135376954 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f6c60018020 con 0x7f6c64097f10 2024-04-01T01:44:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.472+0000 7f6c74275740 1 Processor -- start 2024-04-01T01:44:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.472+0000 7f6c74275740 1 -- start start 2024-04-01T01:44:27.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.472+0000 7f6c74275740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e680c10 0x56331e681030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.472+0000 7f6c74275740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e681600 0x56331e686100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.472+0000 7f6c74275740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e686770 0x56331e688c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.472+0000 7f6c74275740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56331e5820b0 con 0x56331e681600 2024-04-01T01:44:27.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.472+0000 7f6c74275740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56331e5e4ea0 con 0x56331e680c10 2024-04-01T01:44:27.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.472+0000 7f6c74275740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56331e58dc00 con 0x56331e686770 2024-04-01T01:44:27.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.472+0000 7f6c53fff640 1 --2- 172.21.15.70:0/1135376954 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c540a2520 0x7f6c540a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.472+0000 7f6c6b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e680c10 0x56331e681030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.473+0000 7f6c6b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e680c10 0x56331e681030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33506/0 (socket says 172.21.15.70:33506) 2024-04-01T01:44:27.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.473+0000 7f6c6b7fe640 1 -- 172.21.15.70:0/2172128493 learned_addr learned my addr 172.21.15.70:0/2172128493 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:27.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.473+0000 7f6c6a7fc640 1 --2- 172.21.15.70:0/2172128493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e686770 0x56331e688c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.473+0000 7f6c53fff640 1 -- 172.21.15.70:0/1135376954 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6c600ce480 con 0x7f6c64097f10 2024-04-01T01:44:27.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.473+0000 7f6c6affd640 1 --2- 172.21.15.70:0/2172128493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e681600 0x56331e686100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.473+0000 7f6c6b7fe640 1 --2- 172.21.15.70:0/1135376954 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c540a2520 0x7f6c540a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.473+0000 7f6c6a7fc640 1 -- 172.21.15.70:0/2172128493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e680c10 msgr2=0x56331e681030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.473+0000 7f6c6a7fc640 1 --2- 172.21.15.70:0/2172128493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e680c10 0x56331e681030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.473+0000 7f6c6a7fc640 1 -- 172.21.15.70:0/2172128493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e681600 msgr2=0x56331e686100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.473+0000 7f6c6a7fc640 1 --2- 172.21.15.70:0/2172128493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e681600 0x56331e686100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.473+0000 7f6c6a7fc640 1 -- 172.21.15.70:0/2172128493 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56331e3c17e0 con 0x56331e686770 2024-04-01T01:44:27.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.473+0000 7f6c6a7fc640 1 --2- 172.21.15.70:0/2172128493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e686770 0x56331e688c00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c009230 tx=0x7f6c5c02c9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df95847ba40735ad server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:27.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.474+0000 7f6c6b7fe640 1 --2- 172.21.15.70:0/1135376954 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c540a2520 0x7f6c540a49e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c04f780 tx=0x7f6c6c0675f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:27.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.474+0000 7f6c517fa640 1 -- 172.21.15.70:0/2172128493 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c045020 con 0x56331e686770 2024-04-01T01:44:27.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.474+0000 7f6c517fa640 1 -- 172.21.15.70:0/2172128493 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c5c02cba0 con 0x56331e686770 2024-04-01T01:44:27.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.474+0000 7f6c517fa640 1 -- 172.21.15.70:0/2172128493 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c004920 con 0x56331e686770 2024-04-01T01:44:27.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.475+0000 7f6c74275740 1 -- 172.21.15.70:0/2172128493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e686770 msgr2=0x56331e688c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.475+0000 7f6c74275740 1 --2- 172.21.15.70:0/2172128493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e686770 0x56331e688c00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c009230 tx=0x7f6c5c02c9b0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.475+0000 7f6c74275740 1 -- 172.21.15.70:0/2172128493 shutdown_connections 2024-04-01T01:44:27.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.475+0000 7f6c74275740 1 --2- 172.21.15.70:0/2172128493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e686770 0x56331e688c00 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.476+0000 7f6c74275740 1 --2- 172.21.15.70:0/2172128493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e681600 0x56331e686100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.476+0000 7f6c74275740 1 --2- 172.21.15.70:0/2172128493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e680c10 0x56331e681030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.476+0000 7f6c74275740 1 -- 172.21.15.70:0/2172128493 >> 172.21.15.70:0/2172128493 conn(0x56331e67c180 msgr2=0x56331e67c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:27.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.476+0000 7f6c74275740 1 -- 172.21.15.70:0/2172128493 shutdown_connections 2024-04-01T01:44:27.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.476+0000 7f6c74275740 1 -- 172.21.15.70:0/2172128493 wait complete. 2024-04-01T01:44:27.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.477+0000 7f6c74275740 1 -- start start 2024-04-01T01:44:27.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.477+0000 7f6c74275740 1 Processor -- start 2024-04-01T01:44:27.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.477+0000 7f6c74275740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e680c10 0x56331e68af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.478+0000 7f6c74275740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e681600 0x56331e68b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.478+0000 7f6c74275740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e686770 0x56331e692490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.478+0000 7f6c6affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e680c10 0x56331e68af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.478+0000 7f6c6affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e680c10 0x56331e68af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59566/0 (socket says 172.21.15.70:59566) 2024-04-01T01:44:27.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.478+0000 7f6c6a7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e681600 0x56331e68b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.478+0000 7f6c6a7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e681600 0x56331e68b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57862/0 (socket says 172.21.15.70:57862) 2024-04-01T01:44:27.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.478+0000 7f6c6affd640 1 -- 172.21.15.70:0/1149893775 learned_addr learned my addr 172.21.15.70:0/1149893775 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:27.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.478+0000 7f6c6b7fe640 1 --2- 172.21.15.70:0/1149893775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e686770 0x56331e692490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.478+0000 7f6c74275740 1 -- 172.21.15.70:0/1149893775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56331e5833d0 con 0x56331e680c10 2024-04-01T01:44:27.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.478+0000 7f6c74275740 1 -- 172.21.15.70:0/1149893775 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56331e5e0f40 con 0x56331e686770 2024-04-01T01:44:27.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.478+0000 7f6c74275740 1 -- 172.21.15.70:0/1149893775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56331e6896c0 con 0x56331e681600 2024-04-01T01:44:27.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.479+0000 7f6c6b7fe640 1 -- 172.21.15.70:0/1149893775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e681600 msgr2=0x56331e68b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.479+0000 7f6c6b7fe640 1 --2- 172.21.15.70:0/1149893775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e681600 0x56331e68b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.479+0000 7f6c6b7fe640 1 -- 172.21.15.70:0/1149893775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e680c10 msgr2=0x56331e68af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.479+0000 7f6c6b7fe640 1 --2- 172.21.15.70:0/1149893775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e680c10 0x56331e68af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.479+0000 7f6c6b7fe640 1 -- 172.21.15.70:0/1149893775 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56331e421f70 con 0x56331e686770 2024-04-01T01:44:27.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.479+0000 7f6c74275740 10 client.0 _ll_register_callbacks cb 0x56331e5a0b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:27.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.479+0000 7f6c6b7fe640 1 --2- 172.21.15.70:0/1149893775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e686770 0x56331e692490 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c04faa0 tx=0x7f6c6c050420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88e8973698a2051 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:27.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.479+0000 7f6c4eff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:44:27.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.479+0000 7f6c74275740 1 -- 172.21.15.70:0/1149893775 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c5c02cba0 con 0x56331e686770 2024-04-01T01:44:27.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.479+0000 7f6c4eff5640 1 -- 172.21.15.70:0/1149893775 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c07c040 con 0x56331e686770 2024-04-01T01:44:27.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.479+0000 7f6c4eff5640 1 -- 172.21.15.70:0/1149893775 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6c6c061b90 con 0x56331e686770 2024-04-01T01:44:27.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.480+0000 7f6c4eff5640 1 -- 172.21.15.70:0/1149893775 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6c6c061d90 con 0x56331e686770 2024-04-01T01:44:27.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.480+0000 7f6c74275740 10 client.28772 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:27.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.480+0000 7f6c74275740 1 -- 172.21.15.70:0/1149893775 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=31}) -- 0x56331e2b9170 con 0x56331e686770 2024-04-01T01:44:27.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.480+0000 7f6c4eff5640 1 -- 172.21.15.70:0/1149893775 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6c6c0837c0 con 0x56331e686770 2024-04-01T01:44:27.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.480+0000 7f6c74275740 10 client.28772 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:27.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.480+0000 7f6c74275740 10 client.28772 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:27.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.480+0000 7f6c74275740 20 client.28772 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:27.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.480+0000 7f6c74275740 1 --2- 172.21.15.70:0/1149893775 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56331e696840 0x56331e698d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.480+0000 7f6c74275740 4 client.28772 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:27.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.481+0000 7f6c6affd640 1 --2- 172.21.15.70:0/1149893775 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56331e696840 0x56331e698d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.481+0000 7f6c74275740 1 -- 172.21.15.70:0/1149893775 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6c5c02cba0 con 0x56331e696840 2024-04-01T01:44:27.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.481+0000 7f6c6affd640 1 --2- 172.21.15.70:0/1149893775 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56331e696840 0x56331e698d00 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f92fab736c877ba0 server_cookie=4c49a300bbb3b47e in_seq=0 out_seq=0 2024-04-01T01:44:27.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.481+0000 7f6c4eff5640 10 client.28772 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:27.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.482+0000 7f6c4eff5640 1 -- 172.21.15.70:0/1149893775 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f6c5c02cba0 con 0x56331e696840 2024-04-01T01:44:27.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.482+0000 7f6c4eff5640 10 client.28772 handle_command_reply: tid=0 2024-04-01T01:44:27.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.482+0000 7f6c74275740 1 client.28772 shutdown 2024-04-01T01:44:27.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.482+0000 7f6c74275740 1 -- 172.21.15.70:0/1149893775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e686770 msgr2=0x56331e692490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.482+0000 7f6c74275740 1 --2- 172.21.15.70:0/1149893775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e686770 0x56331e692490 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c04faa0 tx=0x7f6c6c050420 comp rx=0 tx=0).stop 2024-04-01T01:44:27.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.482+0000 7f6c74275740 1 -- 172.21.15.70:0/1149893775 shutdown_connections 2024-04-01T01:44:27.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.482+0000 7f6c74275740 1 --2- 172.21.15.70:0/1149893775 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56331e696840 0x56331e698d00 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.482+0000 7f6c74275740 1 --2- 172.21.15.70:0/1149893775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e686770 0x56331e692490 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.482+0000 7f6c74275740 1 --2- 172.21.15.70:0/1149893775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e681600 0x56331e68b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.482+0000 7f6c74275740 1 --2- 172.21.15.70:0/1149893775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e680c10 0x56331e68af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.482+0000 7f6c74275740 1 -- 172.21.15.70:0/1149893775 >> 172.21.15.70:0/1149893775 conn(0x56331e67c180 msgr2=0x56331e67b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:27.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.483+0000 7f6c4eff5640 0 client.28772 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:27.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.483+0000 7f6c74275740 1 -- 172.21.15.70:0/1149893775 shutdown_connections 2024-04-01T01:44:27.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.483+0000 7f6c74275740 1 -- 172.21.15.70:0/1149893775 wait complete. 2024-04-01T01:44:27.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.483+0000 7f6c74275740 20 client.28772 trim_cache size 0 max 16384 2024-04-01T01:44:27.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.516+0000 7f6c74275740 1 Processor -- start 2024-04-01T01:44:27.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.516+0000 7f6c74275740 1 -- start start 2024-04-01T01:44:27.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.516+0000 7f6c74275740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e680c10 0x56331e682ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.516+0000 7f6c74275740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e681600 0x56331e692450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.516+0000 7f6c74275740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e686770 0x56331e683000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.516+0000 7f6c74275740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5800fd20 con 0x56331e680c10 2024-04-01T01:44:27.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.516+0000 7f6c74275740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6c58011860 con 0x56331e686770 2024-04-01T01:44:27.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.516+0000 7f6c74275740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6c580103f0 con 0x56331e681600 2024-04-01T01:44:27.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.516+0000 7f6c6affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e680c10 0x56331e682ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.517+0000 7f6c6affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e680c10 0x56331e682ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59578/0 (socket says 172.21.15.70:59578) 2024-04-01T01:44:27.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.517+0000 7f6c6affd640 1 -- 172.21.15.70:0/1654557839 learned_addr learned my addr 172.21.15.70:0/1654557839 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:27.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.517+0000 7f6c6affd640 1 -- 172.21.15.70:0/1654557839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e681600 msgr2=0x56331e692450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.517+0000 7f6c6affd640 1 --2- 172.21.15.70:0/1654557839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e681600 0x56331e692450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.517+0000 7f6c6affd640 1 -- 172.21.15.70:0/1654557839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e686770 msgr2=0x56331e683000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.517+0000 7f6c6affd640 1 --2- 172.21.15.70:0/1654557839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e686770 0x56331e683000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.517+0000 7f6c6affd640 1 -- 172.21.15.70:0/1654557839 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56331e6819e0 con 0x56331e680c10 2024-04-01T01:44:27.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.517+0000 7f6c6affd640 1 --2- 172.21.15.70:0/1654557839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e680c10 0x56331e682ac0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f6c600cbc00 tx=0x7f6c60096f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca8c62de6349b69 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:27.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.518+0000 7f6c50ff9640 1 -- 172.21.15.70:0/1654557839 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c600a3580 con 0x56331e680c10 2024-04-01T01:44:27.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.518+0000 7f6c50ff9640 1 -- 172.21.15.70:0/1654557839 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c600b3070 con 0x56331e680c10 2024-04-01T01:44:27.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.519+0000 7f6c50ff9640 1 -- 172.21.15.70:0/1654557839 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c600103e0 con 0x56331e680c10 2024-04-01T01:44:27.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.519+0000 7f6c74275740 1 -- 172.21.15.70:0/1654557839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e680c10 msgr2=0x56331e682ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.519+0000 7f6c74275740 1 --2- 172.21.15.70:0/1654557839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e680c10 0x56331e682ac0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f6c600cbc00 tx=0x7f6c60096f40 comp rx=0 tx=0).stop 2024-04-01T01:44:27.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.520+0000 7f6c74275740 1 -- 172.21.15.70:0/1654557839 shutdown_connections 2024-04-01T01:44:27.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.520+0000 7f6c74275740 1 --2- 172.21.15.70:0/1654557839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e686770 0x56331e683000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.520+0000 7f6c74275740 1 --2- 172.21.15.70:0/1654557839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e681600 0x56331e692450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.520+0000 7f6c74275740 1 --2- 172.21.15.70:0/1654557839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e680c10 0x56331e682ac0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.520+0000 7f6c74275740 1 -- 172.21.15.70:0/1654557839 >> 172.21.15.70:0/1654557839 conn(0x56331e67c180 msgr2=0x56331e68af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:27.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.520+0000 7f6c74275740 1 -- 172.21.15.70:0/1654557839 shutdown_connections 2024-04-01T01:44:27.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.520+0000 7f6c74275740 1 -- 172.21.15.70:0/1654557839 wait complete. 2024-04-01T01:44:27.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.521+0000 7f6c74275740 1 -- start start 2024-04-01T01:44:27.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.521+0000 7f6c74275740 1 Processor -- start 2024-04-01T01:44:27.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.522+0000 7f6c74275740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e680c10 0x56331e6903b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.522+0000 7f6c74275740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e681600 0x56331e6908f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.522+0000 7f6c74275740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e686770 0x56331e691890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.522+0000 7f6c74275740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6c58011190 con 0x56331e681600 2024-04-01T01:44:27.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.522+0000 7f6c74275740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6c58010ac0 con 0x56331e680c10 2024-04-01T01:44:27.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.522+0000 7f6c74275740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5800dc90 con 0x56331e686770 2024-04-01T01:44:27.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.522+0000 7f6c74275740 10 client.0 _ll_register_callbacks cb 0x7f6c6c086870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:27.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.523+0000 7f6c6affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e680c10 0x56331e6903b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.523+0000 7f6c6b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e686770 0x56331e691890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.523+0000 7f6c6b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e686770 0x56331e691890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57876/0 (socket says 172.21.15.70:57876) 2024-04-01T01:44:27.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.523+0000 7f6c6b7fe640 1 -- 172.21.15.70:0/3207008948 learned_addr learned my addr 172.21.15.70:0/3207008948 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:27.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.523+0000 7f6c6a7fc640 1 --2- 172.21.15.70:0/3207008948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e681600 0x56331e6908f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.523+0000 7f6c6b7fe640 1 -- 172.21.15.70:0/3207008948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e680c10 msgr2=0x56331e6903b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.523+0000 7f6c6b7fe640 1 --2- 172.21.15.70:0/3207008948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e680c10 0x56331e6903b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.523+0000 7f6c6b7fe640 1 -- 172.21.15.70:0/3207008948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e681600 msgr2=0x56331e6908f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.523+0000 7f6c6b7fe640 1 --2- 172.21.15.70:0/3207008948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e681600 0x56331e6908f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.523+0000 7f6c6b7fe640 1 -- 172.21.15.70:0/3207008948 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56331e691e80 con 0x56331e686770 2024-04-01T01:44:27.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.524+0000 7f6c6b7fe640 1 --2- 172.21.15.70:0/3207008948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e686770 0x56331e691890 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c067290 tx=0x7f6c6c073af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baffcdfa745b86ea server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:27.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.524+0000 7f6c4eff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:44:27.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.524+0000 7f6c4eff5640 1 -- 172.21.15.70:0/3207008948 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c06c070 con 0x56331e686770 2024-04-01T01:44:27.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.524+0000 7f6c4eff5640 1 -- 172.21.15.70:0/3207008948 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6c6c061b90 con 0x56331e686770 2024-04-01T01:44:27.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.524+0000 7f6c74275740 1 -- 172.21.15.70:0/3207008948 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c600b3070 con 0x56331e686770 2024-04-01T01:44:27.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.525+0000 7f6c4eff5640 1 -- 172.21.15.70:0/3207008948 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6c6c083c70 con 0x56331e686770 2024-04-01T01:44:27.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.525+0000 7f6c74275740 10 client.28558 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:27.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.525+0000 7f6c74275740 1 -- 172.21.15.70:0/3207008948 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56331e70e3c0 con 0x56331e686770 2024-04-01T01:44:27.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.525+0000 7f6c4eff5640 1 -- 172.21.15.70:0/3207008948 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c600b3070 con 0x56331e686770 2024-04-01T01:44:27.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.525+0000 7f6c4eff5640 1 -- 172.21.15.70:0/3207008948 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6c6c0735b0 con 0x56331e686770 2024-04-01T01:44:27.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.526+0000 7f6c74275740 10 client.28558 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:27.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.526+0000 7f6c74275740 10 client.28558 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:27.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.526+0000 7f6c74275740 20 client.28558 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:27.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.526+0000 7f6c74275740 1 --2- 172.21.15.70:0/3207008948 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56331e70ede0 0x56331e711200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.526+0000 7f6c74275740 4 client.28558 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:44:27.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.526+0000 7f6c74275740 1 -- 172.21.15.70:0/3207008948 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56331e5f23e0 con 0x56331e70ede0 2024-04-01T01:44:27.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.527+0000 7f6c6affd640 1 --2- 172.21.15.70:0/3207008948 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56331e70ede0 0x56331e711200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.528+0000 7f6c6affd640 1 --2- 172.21.15.70:0/3207008948 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56331e70ede0 0x56331e711200 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b0ebb38fc10f8f1 server_cookie=e70ad0e6d20a1541 in_seq=0 out_seq=0 2024-04-01T01:44:27.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.528+0000 7f6c4eff5640 10 client.28558 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:27.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.529+0000 7f6c4eff5640 1 -- 172.21.15.70:0/3207008948 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56331e5f23e0 con 0x56331e70ede0 2024-04-01T01:44:27.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.529+0000 7f6c4eff5640 10 client.28558 handle_command_reply: tid=0 2024-04-01T01:44:27.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.529+0000 7f6c74275740 1 client.28558 shutdown 2024-04-01T01:44:27.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.529+0000 7f6c74275740 1 -- 172.21.15.70:0/3207008948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e686770 msgr2=0x56331e691890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.529+0000 7f6c74275740 1 --2- 172.21.15.70:0/3207008948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e686770 0x56331e691890 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c067290 tx=0x7f6c6c073af0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.530+0000 7f6c74275740 1 -- 172.21.15.70:0/3207008948 shutdown_connections 2024-04-01T01:44:27.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.530+0000 7f6c74275740 1 --2- 172.21.15.70:0/3207008948 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56331e70ede0 0x56331e711200 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.530+0000 7f6c74275740 1 --2- 172.21.15.70:0/3207008948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56331e686770 0x56331e691890 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.530+0000 7f6c74275740 1 --2- 172.21.15.70:0/3207008948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56331e681600 0x56331e6908f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.530+0000 7f6c74275740 1 --2- 172.21.15.70:0/3207008948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56331e680c10 0x56331e6903b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.530+0000 7f6c74275740 1 -- 172.21.15.70:0/3207008948 >> 172.21.15.70:0/3207008948 conn(0x56331e67c180 msgr2=0x56331e68aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:27.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.530+0000 7f6c4eff5640 0 client.28558 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:27.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.530+0000 7f6c74275740 1 -- 172.21.15.70:0/3207008948 shutdown_connections 2024-04-01T01:44:27.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.530+0000 7f6c74275740 1 -- 172.21.15.70:0/3207008948 wait complete. 2024-04-01T01:44:27.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.530+0000 7f6c74275740 20 client.28558 trim_cache size 0 max 16384 2024-04-01T01:44:27.532 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:44:27.532 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:44:27.532 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "5ff18394-0713-49a7-9339-908da8dab257", 2024-04-01T01:44:27.532 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:44:27.532 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:44:27.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.531+0000 7f6c6bfff640 1 -- 172.21.15.70:0/1135376954 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c540a2520 msgr2=0x7f6c540a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.531+0000 7f6c6bfff640 1 --2- 172.21.15.70:0/1135376954 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c540a2520 0x7f6c540a49e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c04f780 tx=0x7f6c6c0675f0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.531+0000 7f6c6bfff640 1 -- 172.21.15.70:0/1135376954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c64097f10 msgr2=0x7f6c64129fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.531+0000 7f6c6bfff640 1 --2- 172.21.15.70:0/1135376954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c64097f10 0x7f6c64129fa0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6c6000b470 tx=0x7f6c6000b930 comp rx=0 tx=0).stop 2024-04-01T01:44:27.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.531+0000 7f6c6bfff640 1 -- 172.21.15.70:0/1135376954 shutdown_connections 2024-04-01T01:44:27.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.531+0000 7f6c6bfff640 1 --2- 172.21.15.70:0/1135376954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c64099aa0 0x7f6c64131500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.531+0000 7f6c6bfff640 1 --2- 172.21.15.70:0/1135376954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c640990e0 0x7f6c6412a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.531+0000 7f6c6bfff640 1 --2- 172.21.15.70:0/1135376954 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c540a2520 0x7f6c540a49e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.531+0000 7f6c6bfff640 1 --2- 172.21.15.70:0/1135376954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c64097f10 0x7f6c64129fa0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.531+0000 7f6c6bfff640 1 -- 172.21.15.70:0/1135376954 >> 172.21.15.70:0/1135376954 conn(0x7f6c64093510 msgr2=0x7f6c6400b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:27.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.532+0000 7f6c6bfff640 1 -- 172.21.15.70:0/1135376954 shutdown_connections 2024-04-01T01:44:27.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.532+0000 7f6c6bfff640 1 -- 172.21.15.70:0/1135376954 wait complete. 2024-04-01T01:44:27.559 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:44:27.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:44:27.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.741+0000 7f83c668d640 1 -- 172.21.15.70:0/3064470314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c010b0c0 msgr2=0x7f83c010d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.741+0000 7f83c668d640 1 --2- 172.21.15.70:0/3064470314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c010b0c0 0x7f83c010d530 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009860 tx=0x7f83b402bb70 comp rx=0 tx=0).stop 2024-04-01T01:44:27.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.741+0000 7f83c668d640 1 -- 172.21.15.70:0/3064470314 shutdown_connections 2024-04-01T01:44:27.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.741+0000 7f83c668d640 1 --2- 172.21.15.70:0/3064470314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c010da70 0x7f83c010def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.741+0000 7f83c668d640 1 --2- 172.21.15.70:0/3064470314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c010b0c0 0x7f83c010d530 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.741+0000 7f83c668d640 1 --2- 172.21.15.70:0/3064470314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83c0108750 0x7f83c010ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.741+0000 7f83c668d640 1 -- 172.21.15.70:0/3064470314 >> 172.21.15.70:0/3064470314 conn(0x7f83c006e5b0 msgr2=0x7f83c00709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:27.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.741+0000 7f83c668d640 1 -- 172.21.15.70:0/3064470314 shutdown_connections 2024-04-01T01:44:27.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.741+0000 7f83c668d640 1 -- 172.21.15.70:0/3064470314 wait complete. 2024-04-01T01:44:27.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.742+0000 7f83c668d640 1 Processor -- start 2024-04-01T01:44:27.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.742+0000 7f83c668d640 1 -- start start 2024-04-01T01:44:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.742+0000 7f83c668d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c0108750 0x7f83c01a6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.742+0000 7f83c668d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83c010b0c0 0x7f83c01a6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.742+0000 7f83c668d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c010da70 0x7f83c01adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.742+0000 7f83c668d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f83c011c0b0 con 0x7f83c010da70 2024-04-01T01:44:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.742+0000 7f83c668d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f83c011bf30 con 0x7f83c010b0c0 2024-04-01T01:44:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.742+0000 7f83c668d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f83c011c230 con 0x7f83c0108750 2024-04-01T01:44:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.742+0000 7f83bf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83c010b0c0 0x7f83c01a6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.742+0000 7f83bffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c0108750 0x7f83c01a6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.743+0000 7f83bffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c0108750 0x7f83c01a6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57894/0 (socket says 172.21.15.70:57894) 2024-04-01T01:44:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.743+0000 7f83bffff640 1 -- 172.21.15.70:0/3555000480 learned_addr learned my addr 172.21.15.70:0/3555000480 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.743+0000 7f83c4c2a640 1 --2- 172.21.15.70:0/3555000480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c010da70 0x7f83c01adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.743+0000 7f83bffff640 1 -- 172.21.15.70:0/3555000480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83c010b0c0 msgr2=0x7f83c01a6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.743+0000 7f83bffff640 1 --2- 172.21.15.70:0/3555000480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83c010b0c0 0x7f83c01a6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.743+0000 7f83bffff640 1 -- 172.21.15.70:0/3555000480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c010da70 msgr2=0x7f83c01adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.743+0000 7f83bffff640 1 --2- 172.21.15.70:0/3555000480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c010da70 0x7f83c01adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.743+0000 7f83bffff640 1 -- 172.21.15.70:0/3555000480 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c01ae450 con 0x7f83c0108750 2024-04-01T01:44:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.743+0000 7f83bffff640 1 --2- 172.21.15.70:0/3555000480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c0108750 0x7f83c01a6830 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f83b800eab0 tx=0x7f83b800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.744+0000 7f83bd7fa640 1 -- 172.21.15.70:0/3555000480 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b800cd80 con 0x7f83c0108750 2024-04-01T01:44:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.744+0000 7f83bd7fa640 1 -- 172.21.15.70:0/3555000480 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83b8011920 con 0x7f83c0108750 2024-04-01T01:44:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.744+0000 7f83c668d640 1 -- 172.21.15.70:0/3555000480 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83c01ae740 con 0x7f83c0108750 2024-04-01T01:44:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.744+0000 7f83c668d640 1 -- 172.21.15.70:0/3555000480 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83c01aece0 con 0x7f83c0108750 2024-04-01T01:44:27.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.744+0000 7f83bd7fa640 1 -- 172.21.15.70:0/3555000480 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b8010940 con 0x7f83c0108750 2024-04-01T01:44:27.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.747+0000 7f83bd7fa640 1 -- 172.21.15.70:0/3555000480 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f83b8011ad0 con 0x7f83c0108750 2024-04-01T01:44:27.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.748+0000 7f83c926a740 1 Processor -- start 2024-04-01T01:44:27.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.748+0000 7f83bd7fa640 1 --2- 172.21.15.70:0/3555000480 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f83a80a25a0 0x7f83a80a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.749+0000 7f83bd7fa640 1 -- 172.21.15.70:0/3555000480 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f83b80cf110 con 0x7f83c0108750 2024-04-01T01:44:27.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.749+0000 7f83c926a740 1 -- start start 2024-04-01T01:44:27.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.749+0000 7f83c926a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2231582b0 0x55c2231586d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.749+0000 7f83c926a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c223158ca0 0x55c22315d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.750+0000 7f83c926a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c22315de10 0x55c2231602a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.750+0000 7f83c926a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c2230bd180 con 0x55c223158ca0 2024-04-01T01:44:27.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.750+0000 7f83c926a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c223065b70 con 0x55c22315de10 2024-04-01T01:44:27.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.750+0000 7f83c926a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c2230b9220 con 0x55c2231582b0 2024-04-01T01:44:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.750+0000 7f83bffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2231582b0 0x55c2231586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.750+0000 7f83bffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2231582b0 0x55c2231586d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57910/0 (socket says 172.21.15.70:57910) 2024-04-01T01:44:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.750+0000 7f83bffff640 1 -- 172.21.15.70:0/2896969115 learned_addr learned my addr 172.21.15.70:0/2896969115 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:27.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.750+0000 7f83bf7fe640 1 --2- 172.21.15.70:0/2896969115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c223158ca0 0x55c22315d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.750+0000 7f83bffff640 1 -- 172.21.15.70:0/2896969115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c22315de10 msgr2=0x55c2231602a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:44:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.750+0000 7f83bffff640 1 --2- 172.21.15.70:0/2896969115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c22315de10 0x55c2231602a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.750+0000 7f83bffff640 1 -- 172.21.15.70:0/2896969115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c223158ca0 msgr2=0x55c22315d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.750+0000 7f83bffff640 1 --2- 172.21.15.70:0/2896969115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c223158ca0 0x55c22315d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.750+0000 7f83bffff640 1 -- 172.21.15.70:0/2896969115 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2230797c0 con 0x55c2231582b0 2024-04-01T01:44:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.751+0000 7f83bffff640 1 --2- 172.21.15.70:0/2896969115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2231582b0 0x55c2231586d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f83b8097c00 tx=0x7f83b8011ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2a35f487cff0d02 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.751+0000 7f83c4c2a640 1 --2- 172.21.15.70:0/3555000480 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f83a80a25a0 0x7f83a80a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.751+0000 7f83a67fc640 1 -- 172.21.15.70:0/2896969115 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b80b4070 con 0x55c2231582b0 2024-04-01T01:44:27.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.751+0000 7f83a67fc640 1 -- 172.21.15.70:0/2896969115 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83b80bb040 con 0x55c2231582b0 2024-04-01T01:44:27.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.752+0000 7f83a67fc640 1 -- 172.21.15.70:0/2896969115 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b80861a0 con 0x55c2231582b0 2024-04-01T01:44:27.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.752+0000 7f83c926a740 1 -- 172.21.15.70:0/2896969115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2231582b0 msgr2=0x55c2231586d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.752+0000 7f83c926a740 1 --2- 172.21.15.70:0/2896969115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2231582b0 0x55c2231586d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f83b8097c00 tx=0x7f83b8011ee0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.752+0000 7f83c4c2a640 1 --2- 172.21.15.70:0/3555000480 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f83a80a25a0 0x7f83a80a4a60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f83c006b440 tx=0x7f83b0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:27.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.752+0000 7f83c926a740 1 -- 172.21.15.70:0/2896969115 shutdown_connections 2024-04-01T01:44:27.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.752+0000 7f83c926a740 1 --2- 172.21.15.70:0/2896969115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c22315de10 0x55c2231602a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.752+0000 7f83c926a740 1 --2- 172.21.15.70:0/2896969115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c223158ca0 0x55c22315d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.752+0000 7f83c926a740 1 --2- 172.21.15.70:0/2896969115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c2231582b0 0x55c2231586d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.752+0000 7f83c926a740 1 -- 172.21.15.70:0/2896969115 >> 172.21.15.70:0/2896969115 conn(0x55c223153820 msgr2=0x55c223153c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:27.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.753+0000 7f83c926a740 1 -- 172.21.15.70:0/2896969115 shutdown_connections 2024-04-01T01:44:27.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.754+0000 7f83c926a740 1 -- 172.21.15.70:0/2896969115 wait complete. 2024-04-01T01:44:27.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.755+0000 7f83c926a740 1 -- start start 2024-04-01T01:44:27.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.755+0000 7f83c926a740 1 Processor -- start 2024-04-01T01:44:27.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.755+0000 7f83c926a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2231582b0 0x55c223162880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.755+0000 7f83c926a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c223158ca0 0x55c223162dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.756+0000 7f83c926a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c22315de10 0x55c223169e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.756+0000 7f83c926a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c223160e60 con 0x55c2231582b0 2024-04-01T01:44:27.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.756+0000 7f83bf7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c223158ca0 0x55c223162dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.756+0000 7f83bf7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c223158ca0 0x55c223162dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57926/0 (socket says 172.21.15.70:57926) 2024-04-01T01:44:27.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.756+0000 7f83c4c2a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c22315de10 0x55c223169e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.756+0000 7f83bf7fe640 1 -- 172.21.15.70:0/2709736320 learned_addr learned my addr 172.21.15.70:0/2709736320 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:27.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.756+0000 7f83c926a740 1 -- 172.21.15.70:0/2709736320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c22305b4a0 con 0x55c22315de10 2024-04-01T01:44:27.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.756+0000 7f83c926a740 1 -- 172.21.15.70:0/2709736320 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c223160fe0 con 0x55c223158ca0 2024-04-01T01:44:27.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.756+0000 7f83bffff640 1 --2- 172.21.15.70:0/2709736320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2231582b0 0x55c223162880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.756+0000 7f83c926a740 10 client.0 _ll_register_callbacks cb 0x55c22305ed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:27.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.756+0000 7f83c4c2a640 1 -- 172.21.15.70:0/2709736320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c223158ca0 msgr2=0x55c223162dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.756+0000 7f83c4c2a640 1 --2- 172.21.15.70:0/2709736320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c223158ca0 0x55c223162dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.756+0000 7f83c4c2a640 1 -- 172.21.15.70:0/2709736320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2231582b0 msgr2=0x55c223162880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.757+0000 7f83c4c2a640 1 --2- 172.21.15.70:0/2709736320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2231582b0 0x55c223162880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.757+0000 7f83c4c2a640 1 -- 172.21.15.70:0/2709736320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c222e9b7d0 con 0x55c22315de10 2024-04-01T01:44:27.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.757+0000 7f83bffff640 1 --2- 172.21.15.70:0/2709736320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2231582b0 0x55c223162880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:44:27.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.757+0000 7f83c926a740 1 -- 172.21.15.70:0/2709736320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83b80bb040 con 0x55c22315de10 2024-04-01T01:44:27.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.757+0000 7f83c4c2a640 1 --2- 172.21.15.70:0/2709736320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c22315de10 0x55c223169e00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f83b000e860 tx=0x7f83b0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70abf8b011f9aeb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:27.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.757+0000 7f839f7fe640 10 client.28781 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:44:27.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.758+0000 7f839f7fe640 1 -- 172.21.15.70:0/2709736320 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b0026630 con 0x55c22315de10 2024-04-01T01:44:27.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.758+0000 7f839f7fe640 1 -- 172.21.15.70:0/2709736320 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f83b00267d0 con 0x55c22315de10 2024-04-01T01:44:27.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.758+0000 7f839f7fe640 1 -- 172.21.15.70:0/2709736320 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f83b0026a10 con 0x55c22315de10 2024-04-01T01:44:27.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.758+0000 7f83c926a740 10 client.28781 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:27.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.758+0000 7f83c926a740 1 -- 172.21.15.70:0/2709736320 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55c222dd13f0 con 0x55c22315de10 2024-04-01T01:44:27.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.759+0000 7f839f7fe640 1 -- 172.21.15.70:0/2709736320 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f83b0030470 con 0x55c22315de10 2024-04-01T01:44:27.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.759+0000 7f83c926a740 10 client.28781 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:27.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.759+0000 7f83c926a740 10 client.28781 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:27.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.759+0000 7f83c926a740 20 client.28781 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:27.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.759+0000 7f83c926a740 1 --2- 172.21.15.70:0/2709736320 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c22316e9e0 0x55c223170e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.759+0000 7f83c926a740 4 client.28781 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:27.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.759+0000 7f83c926a740 1 -- 172.21.15.70:0/2709736320 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2230ca6c0 con 0x55c22316e9e0 2024-04-01T01:44:27.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.759+0000 7f83bffff640 1 --2- 172.21.15.70:0/2709736320 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c22316e9e0 0x55c223170e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.759+0000 7f83bffff640 1 --2- 172.21.15.70:0/2709736320 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c22316e9e0 0x55c223170e40 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0132428910ce530 server_cookie=371a4834e08cba24 in_seq=0 out_seq=0 2024-04-01T01:44:27.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.759+0000 7f839f7fe640 10 client.28781 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:27.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.760+0000 7f839f7fe640 1 -- 172.21.15.70:0/2709736320 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c2230ca6c0 con 0x55c22316e9e0 2024-04-01T01:44:27.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.760+0000 7f839f7fe640 10 client.28781 handle_command_reply: tid=0 2024-04-01T01:44:27.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.760+0000 7f83c926a740 1 client.28781 shutdown 2024-04-01T01:44:27.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.761+0000 7f83c926a740 1 -- 172.21.15.70:0/2709736320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c22315de10 msgr2=0x55c223169e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.761+0000 7f83c926a740 1 --2- 172.21.15.70:0/2709736320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c22315de10 0x55c223169e00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f83b000e860 tx=0x7f83b0004730 comp rx=0 tx=0).stop 2024-04-01T01:44:27.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.761+0000 7f83c926a740 1 -- 172.21.15.70:0/2709736320 shutdown_connections 2024-04-01T01:44:27.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.761+0000 7f83c926a740 1 --2- 172.21.15.70:0/2709736320 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c22316e9e0 0x55c223170e40 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.761+0000 7f83c926a740 1 --2- 172.21.15.70:0/2709736320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c22315de10 0x55c223169e00 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.761+0000 7f83c926a740 1 --2- 172.21.15.70:0/2709736320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c223158ca0 0x55c223162dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.761+0000 7f83c926a740 1 --2- 172.21.15.70:0/2709736320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2231582b0 0x55c223162880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.761+0000 7f83c926a740 1 -- 172.21.15.70:0/2709736320 >> 172.21.15.70:0/2709736320 conn(0x55c223153820 msgr2=0x55c223152c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:27.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.761+0000 7f839f7fe640 0 client.28781 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:27.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.761+0000 7f83c926a740 1 -- 172.21.15.70:0/2709736320 shutdown_connections 2024-04-01T01:44:27.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.761+0000 7f83c926a740 1 -- 172.21.15.70:0/2709736320 wait complete. 2024-04-01T01:44:27.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.761+0000 7f83c926a740 20 client.28781 trim_cache size 0 max 16384 2024-04-01T01:44:27.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.792+0000 7f83c926a740 1 Processor -- start 2024-04-01T01:44:27.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.793+0000 7f83c926a740 1 -- start start 2024-04-01T01:44:27.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.793+0000 7f83c926a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2231582b0 0x55c223161380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.793+0000 7f83c926a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c223158ca0 0x55c223169dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.793+0000 7f83c926a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22315de10 0x55c223162880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.793+0000 7f83c926a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f83c01b92f0 con 0x55c2231582b0 2024-04-01T01:44:27.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.793+0000 7f83c926a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f83c01bae30 con 0x55c223158ca0 2024-04-01T01:44:27.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.793+0000 7f83c926a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f83c01b99c0 con 0x55c22315de10 2024-04-01T01:44:27.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.793+0000 7f83c4c2a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22315de10 0x55c223162880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.793+0000 7f83c4c2a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22315de10 0x55c223162880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57934/0 (socket says 172.21.15.70:57934) 2024-04-01T01:44:27.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.793+0000 7f83c4c2a640 1 -- 172.21.15.70:0/1247968126 learned_addr learned my addr 172.21.15.70:0/1247968126 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:27.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.794+0000 7f83bffff640 1 --2- 172.21.15.70:0/1247968126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2231582b0 0x55c223161380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.794+0000 7f83bffff640 1 -- 172.21.15.70:0/1247968126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22315de10 msgr2=0x55c223162880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.794+0000 7f83bffff640 1 --2- 172.21.15.70:0/1247968126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22315de10 0x55c223162880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.794+0000 7f83bffff640 1 -- 172.21.15.70:0/1247968126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c223158ca0 msgr2=0x55c223169dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:44:27.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.794+0000 7f83bffff640 1 --2- 172.21.15.70:0/1247968126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c223158ca0 0x55c223169dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.794+0000 7f83bffff640 1 -- 172.21.15.70:0/1247968126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c222d910f0 con 0x55c2231582b0 2024-04-01T01:44:27.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.795+0000 7f83bffff640 1 --2- 172.21.15.70:0/1247968126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2231582b0 0x55c223161380 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f83b800cfd0 tx=0x7f83b809bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de79fbb4bb2d7ae server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:27.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.795+0000 7f83a5ffb640 1 -- 172.21.15.70:0/1247968126 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b80b4040 con 0x55c2231582b0 2024-04-01T01:44:27.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.795+0000 7f83a5ffb640 1 -- 172.21.15.70:0/1247968126 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83b8097e10 con 0x55c2231582b0 2024-04-01T01:44:27.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.796+0000 7f83a5ffb640 1 -- 172.21.15.70:0/1247968126 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b80a3d20 con 0x55c2231582b0 2024-04-01T01:44:27.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.797+0000 7f83c926a740 1 -- 172.21.15.70:0/1247968126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2231582b0 msgr2=0x55c223161380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.797+0000 7f83c926a740 1 --2- 172.21.15.70:0/1247968126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2231582b0 0x55c223161380 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f83b800cfd0 tx=0x7f83b809bf60 comp rx=0 tx=0).stop 2024-04-01T01:44:27.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.798+0000 7f83c926a740 1 -- 172.21.15.70:0/1247968126 shutdown_connections 2024-04-01T01:44:27.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.798+0000 7f83c926a740 1 --2- 172.21.15.70:0/1247968126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c22315de10 0x55c223162880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.798+0000 7f83c926a740 1 --2- 172.21.15.70:0/1247968126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c223158ca0 0x55c223169dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.798+0000 7f83c926a740 1 --2- 172.21.15.70:0/1247968126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c2231582b0 0x55c223161380 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.798+0000 7f83c926a740 1 -- 172.21.15.70:0/1247968126 >> 172.21.15.70:0/1247968126 conn(0x55c223153820 msgr2=0x55c223153c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:27.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.799+0000 7f83c926a740 1 -- 172.21.15.70:0/1247968126 shutdown_connections 2024-04-01T01:44:27.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.799+0000 7f83c926a740 1 -- 172.21.15.70:0/1247968126 wait complete. 2024-04-01T01:44:27.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.799+0000 7f83c926a740 1 -- start start 2024-04-01T01:44:27.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.799+0000 7f83c926a740 1 Processor -- start 2024-04-01T01:44:27.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.800+0000 7f83c926a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c2231582b0 0x55c2231668e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.800+0000 7f83c926a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c223158ca0 0x55c223166e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.800+0000 7f83c926a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c22315de10 0x55c2231d2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.800+0000 7f83c926a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f83c01ba760 con 0x55c22315de10 2024-04-01T01:44:27.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.800+0000 7f83c926a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f83c01ba090 con 0x55c2231582b0 2024-04-01T01:44:27.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.800+0000 7f83bf7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c223158ca0 0x55c223166e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.800+0000 7f83bf7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c223158ca0 0x55c223166e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57942/0 (socket says 172.21.15.70:57942) 2024-04-01T01:44:27.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.800+0000 7f83bf7fe640 1 -- 172.21.15.70:0/1881095507 learned_addr learned my addr 172.21.15.70:0/1881095507 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:27.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.801+0000 7f83bffff640 1 --2- 172.21.15.70:0/1881095507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c2231582b0 0x55c2231668e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.801+0000 7f83c926a740 1 -- 172.21.15.70:0/1881095507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f83c01b72d0 con 0x55c223158ca0 2024-04-01T01:44:27.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.801+0000 7f83c926a740 10 client.0 _ll_register_callbacks cb 0x7f83b000e510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:27.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.801+0000 7f83c4c2a640 1 --2- 172.21.15.70:0/1881095507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c22315de10 0x55c2231d2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.801+0000 7f83bffff640 1 -- 172.21.15.70:0/1881095507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c223158ca0 msgr2=0x55c223166e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.801+0000 7f83bffff640 1 --2- 172.21.15.70:0/1881095507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c223158ca0 0x55c223166e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.801+0000 7f83bffff640 1 -- 172.21.15.70:0/1881095507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c22315de10 msgr2=0x55c2231d2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.801+0000 7f83bffff640 1 --2- 172.21.15.70:0/1881095507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c22315de10 0x55c2231d2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.801+0000 7f83bffff640 1 -- 172.21.15.70:0/1881095507 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c223169e80 con 0x55c2231582b0 2024-04-01T01:44:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.802+0000 7f83bffff640 1 --2- 172.21.15.70:0/1881095507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c2231582b0 0x55c2231668e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f83b8000f80 tx=0x7f83b8011f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f981260fb5d302b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.802+0000 7f839f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:44:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.802+0000 7f839f7fe640 1 -- 172.21.15.70:0/1881095507 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b807e070 con 0x55c2231582b0 2024-04-01T01:44:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.802+0000 7f839f7fe640 1 -- 172.21.15.70:0/1881095507 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f83b809bb50 con 0x55c2231582b0 2024-04-01T01:44:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.802+0000 7f83c926a740 1 -- 172.21.15.70:0/1881095507 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83b8097e10 con 0x55c2231582b0 2024-04-01T01:44:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.802+0000 7f839f7fe640 1 -- 172.21.15.70:0/1881095507 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f83b80b4050 con 0x55c2231582b0 2024-04-01T01:44:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.803+0000 7f83c926a740 10 client.28784 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.803+0000 7f83c926a740 1 -- 172.21.15.70:0/1881095507 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c2231d4120 con 0x55c2231582b0 2024-04-01T01:44:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.803+0000 7f839f7fe640 1 -- 172.21.15.70:0/1881095507 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83b80cc860 con 0x55c2231582b0 2024-04-01T01:44:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.803+0000 7f839f7fe640 1 -- 172.21.15.70:0/1881095507 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f83b800f040 con 0x55c2231582b0 2024-04-01T01:44:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.803+0000 7f83c926a740 10 client.28784 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.803+0000 7f83c926a740 10 client.28784 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.803+0000 7f83c926a740 20 client.28784 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.803+0000 7f83c926a740 1 --2- 172.21.15.70:0/1881095507 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c2231d41e0 0x55c2231d65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.803+0000 7f83c926a740 4 client.28784 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:44:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.803+0000 7f83c926a740 1 -- 172.21.15.70:0/1881095507 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f83b8097e10 con 0x55c2231d41e0 2024-04-01T01:44:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.803+0000 7f83bf7fe640 1 --2- 172.21.15.70:0/1881095507 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c2231d41e0 0x55c2231d65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.804+0000 7f839f7fe640 10 client.28784 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.805+0000 7f83bf7fe640 1 --2- 172.21.15.70:0/1881095507 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c2231d41e0 0x55c2231d65f0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a66d2ed1faa9904 server_cookie=660845e7ba2b5c52 in_seq=0 out_seq=0 2024-04-01T01:44:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.805+0000 7f839f7fe640 1 -- 172.21.15.70:0/1881095507 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f83b8097e10 con 0x55c2231d41e0 2024-04-01T01:44:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.805+0000 7f839f7fe640 10 client.28784 handle_command_reply: tid=0 2024-04-01T01:44:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.806+0000 7f83c926a740 1 client.28784 shutdown 2024-04-01T01:44:27.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.807+0000 7f83c926a740 1 -- 172.21.15.70:0/1881095507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c2231582b0 msgr2=0x55c2231668e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.807+0000 7f83c926a740 1 --2- 172.21.15.70:0/1881095507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c2231582b0 0x55c2231668e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f83b8000f80 tx=0x7f83b8011f50 comp rx=0 tx=0).stop 2024-04-01T01:44:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.809+0000 7f83c926a740 1 -- 172.21.15.70:0/1881095507 shutdown_connections 2024-04-01T01:44:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.809+0000 7f83c926a740 1 --2- 172.21.15.70:0/1881095507 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c2231d41e0 0x55c2231d65f0 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.809+0000 7f83c926a740 1 --2- 172.21.15.70:0/1881095507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c22315de10 0x55c2231d2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.809+0000 7f839f7fe640 0 client.28784 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.809+0000 7f83c926a740 1 --2- 172.21.15.70:0/1881095507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c223158ca0 0x55c223166e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.809+0000 7f83c926a740 1 --2- 172.21.15.70:0/1881095507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c2231582b0 0x55c2231668e0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.809+0000 7f83c926a740 1 -- 172.21.15.70:0/1881095507 >> 172.21.15.70:0/1881095507 conn(0x55c223153820 msgr2=0x55c223161380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.809+0000 7f83c926a740 1 -- 172.21.15.70:0/1881095507 shutdown_connections 2024-04-01T01:44:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.809+0000 7f83c926a740 1 -- 172.21.15.70:0/1881095507 wait complete. 2024-04-01T01:44:27.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.809+0000 7f83c926a740 20 client.28784 trim_cache size 0 max 16384 2024-04-01T01:44:27.811 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:44:27.811 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:44:27.811 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:44:27.811 INFO:teuthology.orchestra.run.smithi070.stdout: "5ff18394-0713-49a7-9339-908da8dab257": { 2024-04-01T01:44:27.811 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:44:27.811 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "5ff18394-0713-49a7-9339-908da8dab257", 2024-04-01T01:44:27.811 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:44:27.811 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:44:27.811 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:44:27.811 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:44:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.811+0000 7f83c668d640 1 -- 172.21.15.70:0/3555000480 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f83a80a25a0 msgr2=0x7f83a80a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.811+0000 7f83c668d640 1 --2- 172.21.15.70:0/3555000480 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f83a80a25a0 0x7f83a80a4a60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f83c006b440 tx=0x7f83b0009260 comp rx=0 tx=0).stop 2024-04-01T01:44:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.811+0000 7f83c668d640 1 -- 172.21.15.70:0/3555000480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c0108750 msgr2=0x7f83c01a6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.811+0000 7f83c668d640 1 --2- 172.21.15.70:0/3555000480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c0108750 0x7f83c01a6830 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f83b800eab0 tx=0x7f83b800eeb0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.811+0000 7f83c668d640 1 -- 172.21.15.70:0/3555000480 shutdown_connections 2024-04-01T01:44:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.811+0000 7f83c668d640 1 --2- 172.21.15.70:0/3555000480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c010da70 0x7f83c01adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.811+0000 7f83c668d640 1 --2- 172.21.15.70:0/3555000480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83c010b0c0 0x7f83c01a6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.811+0000 7f83c668d640 1 --2- 172.21.15.70:0/3555000480 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f83a80a25a0 0x7f83a80a4a60 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.811+0000 7f83c668d640 1 --2- 172.21.15.70:0/3555000480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c0108750 0x7f83c01a6830 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.811+0000 7f83c668d640 1 -- 172.21.15.70:0/3555000480 >> 172.21.15.70:0/3555000480 conn(0x7f83c006e5b0 msgr2=0x7f83c0108b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.812+0000 7f83c668d640 1 -- 172.21.15.70:0/3555000480 shutdown_connections 2024-04-01T01:44:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:27.812+0000 7f83c668d640 1 -- 172.21.15.70:0/3555000480 wait complete. 2024-04-01T01:44:27.833 INFO:tasks.cephfs.filesystem:scrub status for tag:5ff18394-0713-49a7-9339-908da8dab257 - {'path': '/', 'tag': '5ff18394-0713-49a7-9339-908da8dab257', 'options': 'recursive,force'} 2024-04-01T01:44:28.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:27 smithi189 ceph-mon[35753]: pgmap v688: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 13 MiB/s wr, 850 op/s 2024-04-01T01:44:28.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:27 smithi070 ceph-mon[32179]: pgmap v688: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 13 MiB/s wr, 850 op/s 2024-04-01T01:44:28.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:27 smithi081 ceph-mon[35142]: pgmap v688: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 13 MiB/s wr, 850 op/s 2024-04-01T01:44:29.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:28 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:44:29.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:28 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:44:29.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:28 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:44:29.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:28 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:44:29.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:28 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:44:29.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:28 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:44:29.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:28 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:44:29.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:28 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:44:29.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:28 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:44:29.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:28 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:44:29.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:28 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:44:29.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:28 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:44:30.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:29 smithi189 ceph-mon[35753]: pgmap v689: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 13 MiB/s wr, 869 op/s 2024-04-01T01:44:30.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:29 smithi070 ceph-mon[32179]: pgmap v689: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 13 MiB/s wr, 869 op/s 2024-04-01T01:44:30.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:29 smithi081 ceph-mon[35142]: pgmap v689: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 13 MiB/s wr, 869 op/s 2024-04-01T01:44:31.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:30 smithi189 ceph-mon[35753]: pgmap v690: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 12 MiB/s wr, 813 op/s 2024-04-01T01:44:31.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:30 smithi070 ceph-mon[32179]: pgmap v690: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 12 MiB/s wr, 813 op/s 2024-04-01T01:44:31.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:30 smithi081 ceph-mon[35142]: pgmap v690: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 12 MiB/s wr, 813 op/s 2024-04-01T01:44:31.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 585.0 s, 894.5 tps, lat 112.603 ms stddev 79.818 2024-04-01T01:44:32.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:31 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:44:32.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:31 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:44:32.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:31 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:44:33.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:32 smithi189 ceph-mon[35753]: pgmap v691: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 13 MiB/s wr, 982 op/s 2024-04-01T01:44:33.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:32 smithi070 ceph-mon[32179]: pgmap v691: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 13 MiB/s wr, 982 op/s 2024-04-01T01:44:33.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:32 smithi081 ceph-mon[35142]: pgmap v691: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 13 MiB/s wr, 982 op/s 2024-04-01T01:44:35.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:34 smithi189 ceph-mon[35753]: pgmap v692: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 11 MiB/s wr, 931 op/s 2024-04-01T01:44:35.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:34 smithi070 ceph-mon[32179]: pgmap v692: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 11 MiB/s wr, 931 op/s 2024-04-01T01:44:35.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:34 smithi081 ceph-mon[35142]: pgmap v692: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 11 MiB/s wr, 931 op/s 2024-04-01T01:44:36.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 590.0 s, 914.6 tps, lat 109.232 ms stddev 84.266 2024-04-01T01:44:37.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:36 smithi189 ceph-mon[35753]: pgmap v693: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 11 MiB/s wr, 963 op/s 2024-04-01T01:44:37.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:36 smithi070 ceph-mon[32179]: pgmap v693: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 11 MiB/s wr, 963 op/s 2024-04-01T01:44:37.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:36 smithi081 ceph-mon[35142]: pgmap v693: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 11 MiB/s wr, 963 op/s 2024-04-01T01:44:39.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:38 smithi189 ceph-mon[35753]: pgmap v694: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 12 MiB/s wr, 961 op/s 2024-04-01T01:44:39.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:38 smithi070 ceph-mon[32179]: pgmap v694: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 12 MiB/s wr, 961 op/s 2024-04-01T01:44:39.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:38 smithi081 ceph-mon[35142]: pgmap v694: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 12 MiB/s wr, 961 op/s 2024-04-01T01:44:40.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.253+0000 7fe2fd59a640 1 -- 172.21.15.70:0/3277919241 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0003da0 con 0x7fe3000764f0 2024-04-01T01:44:40.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.254+0000 7fe30487c640 1 -- 172.21.15.70:0/3277919241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3000764f0 msgr2=0x7fe300078960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:40.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.254+0000 7fe30487c640 1 --2- 172.21.15.70:0/3277919241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3000764f0 0x7fe300078960 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fe2f00097d0 tx=0x7fe2f002bd70 comp rx=0 tx=0).stop 2024-04-01T01:44:40.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.254+0000 7fe30487c640 1 -- 172.21.15.70:0/3277919241 shutdown_connections 2024-04-01T01:44:40.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.254+0000 7fe30487c640 1 --2- 172.21.15.70:0/3277919241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe300078ea0 0x7fe30007b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.254+0000 7fe30487c640 1 --2- 172.21.15.70:0/3277919241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3000764f0 0x7fe300078960 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.254+0000 7fe30487c640 1 --2- 172.21.15.70:0/3277919241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe300073610 0x7fe300073a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.254+0000 7fe30487c640 1 -- 172.21.15.70:0/3277919241 >> 172.21.15.70:0/3277919241 conn(0x7fe30006e790 msgr2=0x7fe300070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.254+0000 7fe30487c640 1 -- 172.21.15.70:0/3277919241 shutdown_connections 2024-04-01T01:44:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.254+0000 7fe30487c640 1 -- 172.21.15.70:0/3277919241 wait complete. 2024-04-01T01:44:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.255+0000 7fe30487c640 1 Processor -- start 2024-04-01T01:44:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.255+0000 7fe30487c640 1 -- start start 2024-04-01T01:44:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.255+0000 7fe30487c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe300073610 0x7fe3001af690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.255+0000 7fe30487c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3000764f0 0x7fe3001afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.255+0000 7fe30487c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe300078ea0 0x7fe3001b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.255+0000 7fe30487c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe30007ec20 con 0x7fe300073610 2024-04-01T01:44:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.255+0000 7fe30487c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe30007eaa0 con 0x7fe3000764f0 2024-04-01T01:44:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.255+0000 7fe30487c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe30007eda0 con 0x7fe300078ea0 2024-04-01T01:44:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.256+0000 7fe2fed9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe300078ea0 0x7fe3001b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.256+0000 7fe2fe59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe300073610 0x7fe3001af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.256+0000 7fe2fed9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe300078ea0 0x7fe3001b64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43054/0 (socket says 172.21.15.70:43054) 2024-04-01T01:44:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.256+0000 7fe2fed9d640 1 -- 172.21.15.70:0/435257315 learned_addr learned my addr 172.21.15.70:0/435257315 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.256+0000 7fe2fdd9b640 1 --2- 172.21.15.70:0/435257315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3000764f0 0x7fe3001afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.256+0000 7fe2fed9d640 1 -- 172.21.15.70:0/435257315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3000764f0 msgr2=0x7fe3001afbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.256+0000 7fe2fed9d640 1 --2- 172.21.15.70:0/435257315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3000764f0 0x7fe3001afbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.256+0000 7fe2fed9d640 1 -- 172.21.15.70:0/435257315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe300073610 msgr2=0x7fe3001af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.256+0000 7fe2fed9d640 1 --2- 172.21.15.70:0/435257315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe300073610 0x7fe3001af690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.256+0000 7fe2fed9d640 1 -- 172.21.15.70:0/435257315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3001b6bc0 con 0x7fe300078ea0 2024-04-01T01:44:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.256+0000 7fe2fed9d640 1 --2- 172.21.15.70:0/435257315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe300078ea0 0x7fe3001b64f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800cbe0 tx=0x7fe2f80100c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.257+0000 7fe2ef7fe640 1 -- 172.21.15.70:0/435257315 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f801a030 con 0x7fe300078ea0 2024-04-01T01:44:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.257+0000 7fe2ef7fe640 1 -- 172.21.15.70:0/435257315 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2f8010b00 con 0x7fe300078ea0 2024-04-01T01:44:40.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.257+0000 7fe2ef7fe640 1 -- 172.21.15.70:0/435257315 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8005790 con 0x7fe300078ea0 2024-04-01T01:44:40.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.257+0000 7fe30487c640 1 -- 172.21.15.70:0/435257315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3001b6e50 con 0x7fe300078ea0 2024-04-01T01:44:40.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.257+0000 7fe30487c640 1 -- 172.21.15.70:0/435257315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3001b7340 con 0x7fe300078ea0 2024-04-01T01:44:40.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.260+0000 7fe2ef7fe640 1 -- 172.21.15.70:0/435257315 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fe2f801d070 con 0x7fe300078ea0 2024-04-01T01:44:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.260+0000 7fe30745a740 1 Processor -- start 2024-04-01T01:44:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe30745a740 1 -- start start 2024-04-01T01:44:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe30745a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a803512100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe30745a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a8035171d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe30745a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a803519cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe30745a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a8034718f0 con 0x55a8035126d0 2024-04-01T01:44:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe30745a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a80341a8d0 con 0x55a803517840 2024-04-01T01:44:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe30745a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a80346d990 con 0x55a803511ce0 2024-04-01T01:44:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe2fe59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a8035171d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe2fe59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a8035171d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58534/0 (socket says 172.21.15.70:58534) 2024-04-01T01:44:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe2fdd9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a803512100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe2fe59c640 1 -- 172.21.15.70:0/3094280666 learned_addr learned my addr 172.21.15.70:0/3094280666 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:40.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe2fdd9b640 1 --2- 172.21.15.70:0/3094280666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a803519cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe2ef7fe640 1 --2- 172.21.15.70:0/435257315 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2e80a25c0 0x7fe2e80a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe2fe59c640 1 -- 172.21.15.70:0/3094280666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 msgr2=0x55a803512100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:40.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe2fe59c640 1 --2- 172.21.15.70:0/3094280666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a803512100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe2fe59c640 1 -- 172.21.15.70:0/3094280666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 msgr2=0x55a803519cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:40.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe2fe59c640 1 --2- 172.21.15.70:0/3094280666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a803519cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.264 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.261+0000 7fe2fe59c640 1 -- 172.21.15.70:0/3094280666 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a803186210 con 0x55a8035126d0 2024-04-01T01:44:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.262+0000 7fe2fdd9b640 1 --2- 172.21.15.70:0/3094280666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a803519cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:44:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.262+0000 7fe2fed9d640 1 --2- 172.21.15.70:0/435257315 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2e80a25c0 0x7fe2e80a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.262+0000 7fe2ef7fe640 1 -- 172.21.15.70:0/435257315 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe2f80ce4f0 con 0x7fe300078ea0 2024-04-01T01:44:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.262+0000 7fe2fe59c640 1 --2- 172.21.15.70:0/3094280666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a8035171d0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4009a70 tx=0x7fe2f4009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf4ccab617f0cdac server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.262+0000 7fe2ecff9640 1 -- 172.21.15.70:0/3094280666 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f4024070 con 0x55a8035126d0 2024-04-01T01:44:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.262+0000 7fe2ecff9640 1 -- 172.21.15.70:0/3094280666 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2f4017d80 con 0x55a8035126d0 2024-04-01T01:44:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.262+0000 7fe2fed9d640 1 --2- 172.21.15.70:0/435257315 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2e80a25c0 0x7fe2e80a4a80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe2f809cb20 tx=0x7fe2f8092300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.262+0000 7fe2ecff9640 1 -- 172.21.15.70:0/3094280666 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f401fb90 con 0x55a8035126d0 2024-04-01T01:44:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.263+0000 7fe30745a740 1 -- 172.21.15.70:0/3094280666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 msgr2=0x55a8035171d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.263+0000 7fe30745a740 1 --2- 172.21.15.70:0/3094280666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a8035171d0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4009a70 tx=0x7fe2f4009f30 comp rx=0 tx=0).stop 2024-04-01T01:44:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.263+0000 7fe30745a740 1 -- 172.21.15.70:0/3094280666 shutdown_connections 2024-04-01T01:44:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.263+0000 7fe30745a740 1 --2- 172.21.15.70:0/3094280666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a803519cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.263+0000 7fe30745a740 1 --2- 172.21.15.70:0/3094280666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a8035171d0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.263+0000 7fe30745a740 1 --2- 172.21.15.70:0/3094280666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a803512100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.263+0000 7fe30745a740 1 -- 172.21.15.70:0/3094280666 >> 172.21.15.70:0/3094280666 conn(0x55a80350d250 msgr2=0x55a80350d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.264+0000 7fe30745a740 1 -- 172.21.15.70:0/3094280666 shutdown_connections 2024-04-01T01:44:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.264+0000 7fe30745a740 1 -- 172.21.15.70:0/3094280666 wait complete. 2024-04-01T01:44:40.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.264+0000 7fe30745a740 1 -- start start 2024-04-01T01:44:40.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.264+0000 7fe30745a740 1 Processor -- start 2024-04-01T01:44:40.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.264+0000 7fe30745a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a803514de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.264+0000 7fe30745a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a803515320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.264+0000 7fe30745a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a803516330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.264+0000 7fe30745a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a803508ab0 con 0x55a8035126d0 2024-04-01T01:44:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.264+0000 7fe30745a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a80340ff80 con 0x55a803517840 2024-04-01T01:44:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.264+0000 7fe30745a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a803508c30 con 0x55a803511ce0 2024-04-01T01:44:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.265+0000 7fe2fe59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a803515320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.265+0000 7fe2fdd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a803516330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.265+0000 7fe2fdd9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a803514de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.265+0000 7fe2fe59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a803515320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:58548/0 (socket says 172.21.15.70:58548) 2024-04-01T01:44:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.265+0000 7fe2fe59c640 1 -- 172.21.15.70:0/83190898 learned_addr learned my addr 172.21.15.70:0/83190898 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.264+0000 7fe30745a740 10 client.0 _ll_register_callbacks cb 0x55a803468290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.265+0000 7fe2fe59c640 1 -- 172.21.15.70:0/83190898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 msgr2=0x55a803514de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.265+0000 7fe2fe59c640 1 --2- 172.21.15.70:0/83190898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a803514de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.265+0000 7fe2fe59c640 1 -- 172.21.15.70:0/83190898 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 msgr2=0x55a803516330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.265+0000 7fe2fe59c640 1 --2- 172.21.15.70:0/83190898 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a803516330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.265+0000 7fe2fe59c640 1 -- 172.21.15.70:0/83190898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a803252620 con 0x55a8035126d0 2024-04-01T01:44:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.265+0000 7fe2fe59c640 1 --2- 172.21.15.70:0/83190898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a803515320 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4000c00 tx=0x7fe2f401fe80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83041a0e788745a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.265+0000 7fe2e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:44:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.266+0000 7fe2e5ffb640 1 -- 172.21.15.70:0/83190898 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f402c020 con 0x55a8035126d0 2024-04-01T01:44:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.266+0000 7fe2e5ffb640 1 -- 172.21.15.70:0/83190898 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe2f400e810 con 0x55a8035126d0 2024-04-01T01:44:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.266+0000 7fe30745a740 1 -- 172.21.15.70:0/83190898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2f4017d80 con 0x55a8035126d0 2024-04-01T01:44:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.266+0000 7fe2e5ffb640 1 -- 172.21.15.70:0/83190898 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe2f4037050 con 0x55a8035126d0 2024-04-01T01:44:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.266+0000 7fe30745a740 10 client.18762 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.266+0000 7fe30745a740 1 -- 172.21.15.70:0/83190898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a803522170 con 0x55a8035126d0 2024-04-01T01:44:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.267+0000 7fe2e5ffb640 1 -- 172.21.15.70:0/83190898 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2f4017d80 con 0x55a8035126d0 2024-04-01T01:44:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.267+0000 7fe2e5ffb640 1 -- 172.21.15.70:0/83190898 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe2f4028930 con 0x55a8035126d0 2024-04-01T01:44:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.267+0000 7fe30745a740 10 client.18762 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.267+0000 7fe30745a740 10 client.18762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.267+0000 7fe30745a740 20 client.18762 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.267+0000 7fe30745a740 1 --2- 172.21.15.70:0/83190898 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a803523520 0x55a803525910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.267+0000 7fe30745a740 4 client.18762 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.267+0000 7fe30745a740 1 -- 172.21.15.70:0/83190898 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a803485120 con 0x55a803523520 2024-04-01T01:44:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.267+0000 7fe2fdd9b640 1 --2- 172.21.15.70:0/83190898 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a803523520 0x55a803525910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.268+0000 7fe2fdd9b640 1 --2- 172.21.15.70:0/83190898 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a803523520 0x55a803525910 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=388afdd578125d83 server_cookie=135b196536549d59 in_seq=0 out_seq=0 2024-04-01T01:44:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.268+0000 7fe2e5ffb640 10 client.18762 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.270+0000 7fe2e5ffb640 1 -- 172.21.15.70:0/83190898 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a803485120 con 0x55a803523520 2024-04-01T01:44:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.270+0000 7fe2e5ffb640 10 client.18762 handle_command_reply: tid=0 2024-04-01T01:44:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.270+0000 7fe30745a740 1 client.18762 shutdown 2024-04-01T01:44:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.270+0000 7fe30745a740 1 -- 172.21.15.70:0/83190898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 msgr2=0x55a803515320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.270+0000 7fe30745a740 1 --2- 172.21.15.70:0/83190898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a803515320 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4000c00 tx=0x7fe2f401fe80 comp rx=0 tx=0).stop 2024-04-01T01:44:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.270+0000 7fe30745a740 1 -- 172.21.15.70:0/83190898 shutdown_connections 2024-04-01T01:44:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.270+0000 7fe30745a740 1 --2- 172.21.15.70:0/83190898 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a803523520 0x55a803525910 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.271+0000 7fe30745a740 1 --2- 172.21.15.70:0/83190898 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a803516330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.271+0000 7fe30745a740 1 --2- 172.21.15.70:0/83190898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a803515320 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.271+0000 7fe30745a740 1 --2- 172.21.15.70:0/83190898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a803514de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.271+0000 7fe2e5ffb640 0 client.18762 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.271+0000 7fe30745a740 1 -- 172.21.15.70:0/83190898 >> 172.21.15.70:0/83190898 conn(0x55a80350d250 msgr2=0x55a803507f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.271+0000 7fe30745a740 1 -- 172.21.15.70:0/83190898 shutdown_connections 2024-04-01T01:44:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.271+0000 7fe30745a740 1 -- 172.21.15.70:0/83190898 wait complete. 2024-04-01T01:44:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.271+0000 7fe30745a740 20 client.18762 trim_cache size 0 max 16384 2024-04-01T01:44:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.293+0000 7fe30745a740 1 Processor -- start 2024-04-01T01:44:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.293+0000 7fe30745a740 1 -- start start 2024-04-01T01:44:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.293+0000 7fe30745a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a8035076f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.293+0000 7fe30745a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a803507c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.293+0000 7fe30745a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a803508170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.293+0000 7fe30745a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001c1ba0 con 0x55a8035126d0 2024-04-01T01:44:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.293+0000 7fe30745a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001c36e0 con 0x55a803517840 2024-04-01T01:44:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.293+0000 7fe30745a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001c2270 con 0x55a803511ce0 2024-04-01T01:44:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.294+0000 7fe2fdd9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a8035076f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.294+0000 7fe2fdd9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a8035076f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43086/0 (socket says 172.21.15.70:43086) 2024-04-01T01:44:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.294+0000 7fe2fdd9b640 1 -- 172.21.15.70:0/958972207 learned_addr learned my addr 172.21.15.70:0/958972207 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.294+0000 7fe2fdd9b640 1 --2- 172.21.15.70:0/958972207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a803508170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.294+0000 7fe2fe59c640 1 --2- 172.21.15.70:0/958972207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a803507c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.294+0000 7fe2fdd9b640 1 -- 172.21.15.70:0/958972207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 msgr2=0x55a803508170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.294+0000 7fe2fdd9b640 1 --2- 172.21.15.70:0/958972207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a803508170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.294+0000 7fe2fdd9b640 1 -- 172.21.15.70:0/958972207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 msgr2=0x55a803507c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.294+0000 7fe2fdd9b640 1 --2- 172.21.15.70:0/958972207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a803507c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.294+0000 7fe2fdd9b640 1 -- 172.21.15.70:0/958972207 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a803520470 con 0x55a803511ce0 2024-04-01T01:44:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.294+0000 7fe2fdd9b640 1 --2- 172.21.15.70:0/958972207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a8035076f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe2f00038d0 tx=0x7fe2f0009f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38ff1fed07163f45 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.295+0000 7fe2e7fff640 1 -- 172.21.15.70:0/958972207 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0051d60 con 0x55a803511ce0 2024-04-01T01:44:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.295+0000 7fe2e7fff640 1 -- 172.21.15.70:0/958972207 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2f0050bb0 con 0x55a803511ce0 2024-04-01T01:44:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.295+0000 7fe2e7fff640 1 -- 172.21.15.70:0/958972207 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f002ce10 con 0x55a803511ce0 2024-04-01T01:44:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.296+0000 7fe30745a740 1 -- 172.21.15.70:0/958972207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 msgr2=0x55a8035076f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.296+0000 7fe30745a740 1 --2- 172.21.15.70:0/958972207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a8035076f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe2f00038d0 tx=0x7fe2f0009f40 comp rx=0 tx=0).stop 2024-04-01T01:44:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.296+0000 7fe30745a740 1 -- 172.21.15.70:0/958972207 shutdown_connections 2024-04-01T01:44:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.297+0000 7fe30745a740 1 --2- 172.21.15.70:0/958972207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a803508170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.297+0000 7fe30745a740 1 --2- 172.21.15.70:0/958972207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a803507c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.297+0000 7fe30745a740 1 --2- 172.21.15.70:0/958972207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a8035076f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.297+0000 7fe30745a740 1 -- 172.21.15.70:0/958972207 >> 172.21.15.70:0/958972207 conn(0x55a80350d250 msgr2=0x55a803513460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.297+0000 7fe30745a740 1 -- 172.21.15.70:0/958972207 shutdown_connections 2024-04-01T01:44:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.297+0000 7fe30745a740 1 -- 172.21.15.70:0/958972207 wait complete. 2024-04-01T01:44:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.297+0000 7fe30745a740 1 -- start start 2024-04-01T01:44:40.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.298+0000 7fe30745a740 1 Processor -- start 2024-04-01T01:44:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.298+0000 7fe30745a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a80351dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.298+0000 7fe30745a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a80351e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.298+0000 7fe2fdd9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a80351dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.298+0000 7fe2fdd9b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a80351dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43098/0 (socket says 172.21.15.70:43098) 2024-04-01T01:44:40.300 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.299+0000 7fe2fdd9b640 1 -- 172.21.15.70:0/1959976481 learned_addr learned my addr 172.21.15.70:0/1959976481 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.299+0000 7fe2fe59c640 1 --2- 172.21.15.70:0/1959976481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a80351e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.299+0000 7fe30745a740 1 --2- 172.21.15.70:0/1959976481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a80351f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.299+0000 7fe30745a740 1 -- 172.21.15.70:0/1959976481 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001c3010 con 0x55a8035126d0 2024-04-01T01:44:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.299+0000 7fe30745a740 1 -- 172.21.15.70:0/1959976481 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001c2940 con 0x55a803517840 2024-04-01T01:44:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.299+0000 7fe30745a740 1 -- 172.21.15.70:0/1959976481 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe3001bfb80 con 0x55a803511ce0 2024-04-01T01:44:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.299+0000 7fe30745a740 10 client.0 _ll_register_callbacks cb 0x7fe2f4022070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.299+0000 7fe2fdd9b640 1 --2- 172.21.15.70:0/1959976481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a80351f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.299+0000 7fe2fdd9b640 1 -- 172.21.15.70:0/1959976481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 msgr2=0x55a80351f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.299+0000 7fe2fdd9b640 1 --2- 172.21.15.70:0/1959976481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a80351f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.299+0000 7fe2fdd9b640 1 -- 172.21.15.70:0/1959976481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 msgr2=0x55a80351e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.299+0000 7fe2fdd9b640 1 --2- 172.21.15.70:0/1959976481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a80351e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.299+0000 7fe2fdd9b640 1 -- 172.21.15.70:0/1959976481 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a80351f690 con 0x55a803511ce0 2024-04-01T01:44:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.300+0000 7fe2fdd9b640 1 --2- 172.21.15.70:0/1959976481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a80351dbc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0003750 tx=0x7fe2f002bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb43c0192ce615af server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.300+0000 7fe2e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:44:40.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.301+0000 7fe2e5ffb640 1 -- 172.21.15.70:0/1959976481 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f002cde0 con 0x55a803511ce0 2024-04-01T01:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.301+0000 7fe2e5ffb640 1 -- 172.21.15.70:0/1959976481 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe2f0050440 con 0x55a803511ce0 2024-04-01T01:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.301+0000 7fe30745a740 1 -- 172.21.15.70:0/1959976481 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2f0050bb0 con 0x55a803511ce0 2024-04-01T01:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.301+0000 7fe2e5ffb640 1 -- 172.21.15.70:0/1959976481 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe2f0050640 con 0x55a803511ce0 2024-04-01T01:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.301+0000 7fe30745a740 10 client.28594 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.301+0000 7fe30745a740 1 -- 172.21.15.70:0/1959976481 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a803589ce0 con 0x55a803511ce0 2024-04-01T01:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.302+0000 7fe2e5ffb640 1 -- 172.21.15.70:0/1959976481 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe2f00507e0 con 0x55a803511ce0 2024-04-01T01:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.302+0000 7fe2e5ffb640 1 -- 172.21.15.70:0/1959976481 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe2f0003ca0 con 0x55a803511ce0 2024-04-01T01:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.302+0000 7fe30745a740 10 client.28594 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.302+0000 7fe30745a740 10 client.28594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.302+0000 7fe30745a740 20 client.28594 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.302+0000 7fe30745a740 1 --2- 172.21.15.70:0/1959976481 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a80358a4e0 0x55a80358c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.302+0000 7fe30745a740 4 client.28594 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.302+0000 7fe30745a740 1 -- 172.21.15.70:0/1959976481 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fe2f0050bb0 con 0x55a80358a4e0 2024-04-01T01:44:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.302+0000 7fe2fe59c640 1 --2- 172.21.15.70:0/1959976481 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a80358a4e0 0x55a80358c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:40.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.303+0000 7fe2fe59c640 1 --2- 172.21.15.70:0/1959976481 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a80358a4e0 0x55a80358c900 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aef1cc5ecd7c711a server_cookie=3240b9e8a5d86d8d in_seq=0 out_seq=0 2024-04-01T01:44:40.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.303+0000 7fe2e5ffb640 10 client.28594 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.389+0000 7fe2e5ffb640 1 -- 172.21.15.70:0/1959976481 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fe2f0050bb0 con 0x55a80358a4e0 2024-04-01T01:44:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.389+0000 7fe2e5ffb640 10 client.28594 handle_command_reply: tid=0 2024-04-01T01:44:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.389+0000 7fe30745a740 1 client.28594 shutdown 2024-04-01T01:44:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.389+0000 7fe30745a740 1 -- 172.21.15.70:0/1959976481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 msgr2=0x55a80351dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.389+0000 7fe30745a740 1 --2- 172.21.15.70:0/1959976481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a80351dbc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0003750 tx=0x7fe2f002bd70 comp rx=0 tx=0).stop 2024-04-01T01:44:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.389+0000 7fe30745a740 1 -- 172.21.15.70:0/1959976481 shutdown_connections 2024-04-01T01:44:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.389+0000 7fe30745a740 1 --2- 172.21.15.70:0/1959976481 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a80358a4e0 0x55a80358c900 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.389+0000 7fe2e5ffb640 0 client.28594 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.389+0000 7fe30745a740 1 --2- 172.21.15.70:0/1959976481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a803517840 0x55a80351f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.389+0000 7fe30745a740 1 --2- 172.21.15.70:0/1959976481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a8035126d0 0x55a80351e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.389+0000 7fe30745a740 1 --2- 172.21.15.70:0/1959976481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a803511ce0 0x55a80351dbc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.390+0000 7fe30745a740 1 -- 172.21.15.70:0/1959976481 >> 172.21.15.70:0/1959976481 conn(0x55a80350d250 msgr2=0x55a803513980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.390+0000 7fe30745a740 1 -- 172.21.15.70:0/1959976481 shutdown_connections 2024-04-01T01:44:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.390+0000 7fe30745a740 1 -- 172.21.15.70:0/1959976481 wait complete. 2024-04-01T01:44:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.390+0000 7fe30745a740 20 client.28594 trim_cache size 0 max 16384 2024-04-01T01:44:40.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:44:40.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:44:40.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:44:40.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:44:40.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.390+0000 7fe30487c640 1 -- 172.21.15.70:0/435257315 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2e80a25c0 msgr2=0x7fe2e80a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:40.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.391+0000 7fe30487c640 1 --2- 172.21.15.70:0/435257315 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2e80a25c0 0x7fe2e80a4a80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe2f809cb20 tx=0x7fe2f8092300 comp rx=0 tx=0).stop 2024-04-01T01:44:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.391+0000 7fe30487c640 1 -- 172.21.15.70:0/435257315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe300078ea0 msgr2=0x7fe3001b64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.391+0000 7fe30487c640 1 --2- 172.21.15.70:0/435257315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe300078ea0 0x7fe3001b64f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800cbe0 tx=0x7fe2f80100c0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.391+0000 7fe30487c640 1 -- 172.21.15.70:0/435257315 shutdown_connections 2024-04-01T01:44:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.391+0000 7fe30487c640 1 --2- 172.21.15.70:0/435257315 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe2e80a25c0 0x7fe2e80a4a80 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.391+0000 7fe30487c640 1 --2- 172.21.15.70:0/435257315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe300078ea0 0x7fe3001b64f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.391+0000 7fe30487c640 1 --2- 172.21.15.70:0/435257315 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3000764f0 0x7fe3001afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.391+0000 7fe30487c640 1 --2- 172.21.15.70:0/435257315 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe300073610 0x7fe3001af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.391+0000 7fe30487c640 1 -- 172.21.15.70:0/435257315 >> 172.21.15.70:0/435257315 conn(0x7fe30006e790 msgr2=0x7fe30007b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:40.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.391+0000 7fe30487c640 1 -- 172.21.15.70:0/435257315 shutdown_connections 2024-04-01T01:44:40.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:40.391+0000 7fe30487c640 1 -- 172.21.15.70:0/435257315 wait complete. 2024-04-01T01:44:41.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:40 smithi189 ceph-mon[35753]: pgmap v695: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 10 MiB/s wr, 858 op/s 2024-04-01T01:44:41.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:40 smithi070 ceph-mon[32179]: pgmap v695: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 10 MiB/s wr, 858 op/s 2024-04-01T01:44:41.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:40 smithi081 ceph-mon[35142]: pgmap v695: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 10 MiB/s wr, 858 op/s 2024-04-01T01:44:41.897 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 595.0 s, 844.5 tps, lat 118.272 ms stddev 97.243 2024-04-01T01:44:43.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:42 smithi189 ceph-mon[35753]: pgmap v696: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 11 MiB/s wr, 910 op/s 2024-04-01T01:44:43.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:42 smithi070 ceph-mon[32179]: pgmap v696: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 11 MiB/s wr, 910 op/s 2024-04-01T01:44:43.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:42 smithi081 ceph-mon[35142]: pgmap v696: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 11 MiB/s wr, 910 op/s 2024-04-01T01:44:45.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:44 smithi189 ceph-mon[35753]: pgmap v697: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 9.8 MiB/s wr, 774 op/s 2024-04-01T01:44:45.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:44 smithi070 ceph-mon[32179]: pgmap v697: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 9.8 MiB/s wr, 774 op/s 2024-04-01T01:44:45.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:44 smithi081 ceph-mon[35142]: pgmap v697: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 9.8 MiB/s wr, 774 op/s 2024-04-01T01:44:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 600.0 s, 817.3 tps, lat 121.410 ms stddev 103.896 2024-04-01T01:44:47.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:46 smithi189 ceph-mon[35753]: pgmap v698: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 11 MiB/s wr, 790 op/s 2024-04-01T01:44:47.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:46 smithi070 ceph-mon[32179]: pgmap v698: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 11 MiB/s wr, 790 op/s 2024-04-01T01:44:47.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:46 smithi081 ceph-mon[35142]: pgmap v698: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 11 MiB/s wr, 790 op/s 2024-04-01T01:44:49.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:48 smithi189 ceph-mon[35753]: pgmap v699: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 12 MiB/s wr, 830 op/s 2024-04-01T01:44:49.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:48 smithi070 ceph-mon[32179]: pgmap v699: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 12 MiB/s wr, 830 op/s 2024-04-01T01:44:49.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:48 smithi081 ceph-mon[35142]: pgmap v699: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 12 MiB/s wr, 830 op/s 2024-04-01T01:44:51.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:50 smithi189 ceph-mon[35753]: pgmap v700: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 10 MiB/s wr, 747 op/s 2024-04-01T01:44:51.256 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:50 smithi070 ceph-mon[32179]: pgmap v700: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 10 MiB/s wr, 747 op/s 2024-04-01T01:44:51.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:50 smithi081 ceph-mon[35142]: pgmap v700: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 10 MiB/s wr, 747 op/s 2024-04-01T01:44:51.898 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 605.0 s, 851.2 tps, lat 118.258 ms stddev 108.399 2024-04-01T01:44:53.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:52 smithi189 ceph-mon[35753]: pgmap v701: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 11 MiB/s wr, 848 op/s 2024-04-01T01:44:53.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:52 smithi070 ceph-mon[32179]: pgmap v701: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 11 MiB/s wr, 848 op/s 2024-04-01T01:44:53.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:52 smithi081 ceph-mon[35142]: pgmap v701: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 11 MiB/s wr, 848 op/s 2024-04-01T01:44:53.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.541+0000 7fb196765640 1 -- 172.21.15.70:0/1325664713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb190109960 msgr2=0x7fb1901101f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.541+0000 7fb196765640 1 --2- 172.21.15.70:0/1325664713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb190109960 0x7fb1901101f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb188009ca0 tx=0x7fb18802c100 comp rx=0 tx=0).stop 2024-04-01T01:44:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.542+0000 7fb196765640 1 -- 172.21.15.70:0/1325664713 shutdown_connections 2024-04-01T01:44:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.542+0000 7fb196765640 1 --2- 172.21.15.70:0/1325664713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb190109960 0x7fb1901101f0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.542+0000 7fb196765640 1 --2- 172.21.15.70:0/1325664713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb190108fa0 0x7fb190109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.542+0000 7fb196765640 1 --2- 172.21.15.70:0/1325664713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb190107db0 0x7fb1901081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.542+0000 7fb196765640 1 -- 172.21.15.70:0/1325664713 >> 172.21.15.70:0/1325664713 conn(0x7fb190076d70 msgr2=0x7fb190079190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.543+0000 7fb196765640 1 -- 172.21.15.70:0/1325664713 shutdown_connections 2024-04-01T01:44:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.543+0000 7fb196765640 1 -- 172.21.15.70:0/1325664713 wait complete. 2024-04-01T01:44:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.544+0000 7fb196765640 1 Processor -- start 2024-04-01T01:44:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.544+0000 7fb196765640 1 -- start start 2024-04-01T01:44:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.544+0000 7fb196765640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb190107db0 0x7fb19019dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.544+0000 7fb196765640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb190108fa0 0x7fb19019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.544+0000 7fb196765640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb190109960 0x7fb1901a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.544+0000 7fb196765640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb1901138c0 con 0x7fb190108fa0 2024-04-01T01:44:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.544+0000 7fb196765640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb190113740 con 0x7fb190107db0 2024-04-01T01:44:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.544+0000 7fb196765640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb190113a40 con 0x7fb190109960 2024-04-01T01:44:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.544+0000 7fb18ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb190107db0 0x7fb19019dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.544+0000 7fb18ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb190107db0 0x7fb19019dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33620/0 (socket says 172.21.15.70:33620) 2024-04-01T01:44:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.544+0000 7fb18ffff640 1 -- 172.21.15.70:0/3051713531 learned_addr learned my addr 172.21.15.70:0/3051713531 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.545+0000 7fb194d02640 1 --2- 172.21.15.70:0/3051713531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb190109960 0x7fb1901a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.545+0000 7fb18f7fe640 1 --2- 172.21.15.70:0/3051713531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb190108fa0 0x7fb19019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.545+0000 7fb18ffff640 1 -- 172.21.15.70:0/3051713531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb190109960 msgr2=0x7fb1901a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.545+0000 7fb18ffff640 1 --2- 172.21.15.70:0/3051713531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb190109960 0x7fb1901a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.545+0000 7fb18ffff640 1 -- 172.21.15.70:0/3051713531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb190108fa0 msgr2=0x7fb19019e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.545+0000 7fb18ffff640 1 --2- 172.21.15.70:0/3051713531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb190108fa0 0x7fb19019e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.545+0000 7fb18ffff640 1 -- 172.21.15.70:0/3051713531 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1901a5ac0 con 0x7fb190107db0 2024-04-01T01:44:53.546 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.545+0000 7fb18ffff640 1 --2- 172.21.15.70:0/3051713531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb190107db0 0x7fb19019dea0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb18000d770 tx=0x7fb18000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:53.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.546+0000 7fb18d7fa640 1 -- 172.21.15.70:0/3051713531 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb180017070 con 0x7fb190107db0 2024-04-01T01:44:53.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.546+0000 7fb196765640 1 -- 172.21.15.70:0/3051713531 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1901a5db0 con 0x7fb190107db0 2024-04-01T01:44:53.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.546+0000 7fb196765640 1 -- 172.21.15.70:0/3051713531 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1901a6220 con 0x7fb190107db0 2024-04-01T01:44:53.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.547+0000 7fb18d7fa640 1 -- 172.21.15.70:0/3051713531 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb180009c30 con 0x7fb190107db0 2024-04-01T01:44:53.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.547+0000 7fb18d7fa640 1 -- 172.21.15.70:0/3051713531 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb180003d00 con 0x7fb190107db0 2024-04-01T01:44:53.548 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.549+0000 7fb18d7fa640 1 -- 172.21.15.70:0/3051713531 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb180010460 con 0x7fb190107db0 2024-04-01T01:44:53.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.550+0000 7fb18d7fa640 1 --2- 172.21.15.70:0/3051713531 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb1780a2730 0x7fb1780a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.549 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.550+0000 7fb18d7fa640 1 -- 172.21.15.70:0/3051713531 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb1800cf360 con 0x7fb190107db0 2024-04-01T01:44:53.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.551+0000 7fb18f7fe640 1 --2- 172.21.15.70:0/3051713531 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb1780a2730 0x7fb1780a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.552+0000 7fb18f7fe640 1 --2- 172.21.15.70:0/3051713531 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb1780a2730 0x7fb1780a4bf0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb184005fd0 tx=0x7fb184005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.552+0000 7fb199425740 1 Processor -- start 2024-04-01T01:44:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.552+0000 7fb199425740 1 -- start start 2024-04-01T01:44:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb199425740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e18a8d0 0x56317e18acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb199425740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e18b2c0 0x56317e18fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb199425740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e190430 0x56317e1928c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb18ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e18a8d0 0x56317e18acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb18ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e18a8d0 0x56317e18acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:47996/0 (socket says 172.21.15.70:47996) 2024-04-01T01:44:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb18ffff640 1 -- 172.21.15.70:0/1308122254 learned_addr learned my addr 172.21.15.70:0/1308122254 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb199425740 1 -- 172.21.15.70:0/1308122254 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56317e0ee8f0 con 0x56317e18b2c0 2024-04-01T01:44:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb199425740 1 -- 172.21.15.70:0/1308122254 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56317e0978d0 con 0x56317e190430 2024-04-01T01:44:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb199425740 1 -- 172.21.15.70:0/1308122254 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56317e0ea990 con 0x56317e18a8d0 2024-04-01T01:44:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb18f7fe640 1 --2- 172.21.15.70:0/1308122254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e18b2c0 0x56317e18fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb194d02640 1 --2- 172.21.15.70:0/1308122254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e190430 0x56317e1928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb18ffff640 1 -- 172.21.15.70:0/1308122254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e190430 msgr2=0x56317e1928c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb18ffff640 1 --2- 172.21.15.70:0/1308122254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e190430 0x56317e1928c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb18ffff640 1 -- 172.21.15.70:0/1308122254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e18b2c0 msgr2=0x56317e18fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb18ffff640 1 --2- 172.21.15.70:0/1308122254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e18b2c0 0x56317e18fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb18ffff640 1 -- 172.21.15.70:0/1308122254 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56317de03210 con 0x56317e18a8d0 2024-04-01T01:44:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.553+0000 7fb18f7fe640 1 --2- 172.21.15.70:0/1308122254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e18b2c0 0x56317e18fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.554+0000 7fb18ffff640 1 --2- 172.21.15.70:0/1308122254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e18a8d0 0x56317e18acf0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb1800038b0 tx=0x7fb18009c1b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8ce89929c1c869a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.554+0000 7fb1767fc640 1 -- 172.21.15.70:0/1308122254 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb180020040 con 0x56317e18a8d0 2024-04-01T01:44:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.554+0000 7fb1767fc640 1 -- 172.21.15.70:0/1308122254 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1800a4400 con 0x56317e18a8d0 2024-04-01T01:44:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.555+0000 7fb1767fc640 1 -- 172.21.15.70:0/1308122254 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb180097740 con 0x56317e18a8d0 2024-04-01T01:44:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.555+0000 7fb199425740 1 -- 172.21.15.70:0/1308122254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e18a8d0 msgr2=0x56317e18acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.555+0000 7fb199425740 1 --2- 172.21.15.70:0/1308122254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e18a8d0 0x56317e18acf0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb1800038b0 tx=0x7fb18009c1b0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.555+0000 7fb199425740 1 -- 172.21.15.70:0/1308122254 shutdown_connections 2024-04-01T01:44:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.555+0000 7fb199425740 1 --2- 172.21.15.70:0/1308122254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e190430 0x56317e1928c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.555+0000 7fb199425740 1 --2- 172.21.15.70:0/1308122254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e18b2c0 0x56317e18fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.555+0000 7fb199425740 1 --2- 172.21.15.70:0/1308122254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e18a8d0 0x56317e18acf0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.556+0000 7fb199425740 1 -- 172.21.15.70:0/1308122254 >> 172.21.15.70:0/1308122254 conn(0x56317e185e40 msgr2=0x56317e186240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.556+0000 7fb199425740 1 -- 172.21.15.70:0/1308122254 shutdown_connections 2024-04-01T01:44:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.556+0000 7fb199425740 1 -- 172.21.15.70:0/1308122254 wait complete. 2024-04-01T01:44:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.556+0000 7fb199425740 1 -- start start 2024-04-01T01:44:53.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.557+0000 7fb199425740 1 Processor -- start 2024-04-01T01:44:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.557+0000 7fb199425740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e18a8d0 0x56317e194ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.557+0000 7fb199425740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18b2c0 0x56317e195010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.557+0000 7fb199425740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e190430 0x56317e19c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.557+0000 7fb199425740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56317e193370 con 0x56317e190430 2024-04-01T01:44:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.557+0000 7fb199425740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56317e08cf80 con 0x56317e18b2c0 2024-04-01T01:44:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.557+0000 7fb199425740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56317e1934f0 con 0x56317e18a8d0 2024-04-01T01:44:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.557+0000 7fb199425740 10 client.0 _ll_register_callbacks cb 0x56317e0e5290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.557+0000 7fb18ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e18a8d0 0x56317e194ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.557+0000 7fb18ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e18a8d0 0x56317e194ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:48002/0 (socket says 172.21.15.70:48002) 2024-04-01T01:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.557+0000 7fb18ffff640 1 -- 172.21.15.70:0/3491139660 learned_addr learned my addr 172.21.15.70:0/3491139660 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.558+0000 7fb194d02640 1 --2- 172.21.15.70:0/3491139660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e190430 0x56317e19c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.558+0000 7fb194d02640 1 -- 172.21.15.70:0/3491139660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e18a8d0 msgr2=0x56317e194ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.558+0000 7fb194d02640 1 --2- 172.21.15.70:0/3491139660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e18a8d0 0x56317e194ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.558+0000 7fb18f7fe640 1 --2- 172.21.15.70:0/3491139660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18b2c0 0x56317e195010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.558+0000 7fb194d02640 1 -- 172.21.15.70:0/3491139660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18b2c0 msgr2=0x56317e195010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.558+0000 7fb194d02640 1 --2- 172.21.15.70:0/3491139660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18b2c0 0x56317e195010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.558+0000 7fb194d02640 1 -- 172.21.15.70:0/3491139660 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56317decf620 con 0x56317e190430 2024-04-01T01:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.559+0000 7fb194d02640 1 --2- 172.21.15.70:0/3491139660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e190430 0x56317e19c050 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fb188037040 tx=0x7fb188005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9194f89ad0498eae server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.559+0000 7fb16f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.559+0000 7fb16f7fe640 1 -- 172.21.15.70:0/3491139660 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb188039070 con 0x56317e190430 2024-04-01T01:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.559+0000 7fb16f7fe640 1 -- 172.21.15.70:0/3491139660 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb1880329a0 con 0x56317e190430 2024-04-01T01:44:53.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.559+0000 7fb199425740 1 -- 172.21.15.70:0/3491139660 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1800a4400 con 0x56317e190430 2024-04-01T01:44:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.560+0000 7fb16f7fe640 1 -- 172.21.15.70:0/3491139660 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb188032ba0 con 0x56317e190430 2024-04-01T01:44:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.560+0000 7fb199425740 10 client.18780 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.560+0000 7fb199425740 1 -- 172.21.15.70:0/3491139660 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56317e19f0a0 con 0x56317e190430 2024-04-01T01:44:53.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.560+0000 7fb16f7fe640 1 -- 172.21.15.70:0/3491139660 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1800a4400 con 0x56317e190430 2024-04-01T01:44:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.560+0000 7fb16f7fe640 1 -- 172.21.15.70:0/3491139660 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb18803c070 con 0x56317e190430 2024-04-01T01:44:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.560+0000 7fb199425740 10 client.18780 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.560+0000 7fb199425740 10 client.18780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.560+0000 7fb199425740 20 client.18780 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.560+0000 7fb199425740 1 --2- 172.21.15.70:0/3491139660 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56317e1a0770 0x56317e1a2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.561+0000 7fb199425740 4 client.18780 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.561+0000 7fb199425740 1 -- 172.21.15.70:0/3491139660 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56317e102120 con 0x56317e1a0770 2024-04-01T01:44:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.562+0000 7fb18ffff640 1 --2- 172.21.15.70:0/3491139660 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56317e1a0770 0x56317e1a2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.562+0000 7fb18ffff640 1 --2- 172.21.15.70:0/3491139660 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56317e1a0770 0x56317e1a2c00 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bd569be40cdee2 server_cookie=31a9a0c1a84c32ac in_seq=0 out_seq=0 2024-04-01T01:44:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.562+0000 7fb16f7fe640 10 client.18780 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.563+0000 7fb16f7fe640 1 -- 172.21.15.70:0/3491139660 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56317e102120 con 0x56317e1a0770 2024-04-01T01:44:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.563+0000 7fb16f7fe640 10 client.18780 handle_command_reply: tid=0 2024-04-01T01:44:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.563+0000 7fb199425740 1 client.18780 shutdown 2024-04-01T01:44:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.564+0000 7fb199425740 1 -- 172.21.15.70:0/3491139660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e190430 msgr2=0x56317e19c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.564+0000 7fb199425740 1 --2- 172.21.15.70:0/3491139660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e190430 0x56317e19c050 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fb188037040 tx=0x7fb188005f90 comp rx=0 tx=0).stop 2024-04-01T01:44:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.564+0000 7fb199425740 1 -- 172.21.15.70:0/3491139660 shutdown_connections 2024-04-01T01:44:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.564+0000 7fb199425740 1 --2- 172.21.15.70:0/3491139660 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56317e1a0770 0x56317e1a2c00 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.564+0000 7fb199425740 1 --2- 172.21.15.70:0/3491139660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e190430 0x56317e19c050 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.564+0000 7fb199425740 1 --2- 172.21.15.70:0/3491139660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18b2c0 0x56317e195010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.564+0000 7fb199425740 1 --2- 172.21.15.70:0/3491139660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e18a8d0 0x56317e194ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.564+0000 7fb199425740 1 -- 172.21.15.70:0/3491139660 >> 172.21.15.70:0/3491139660 conn(0x56317e185e40 msgr2=0x56317e184f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:53.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.564+0000 7fb16f7fe640 0 client.18780 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:53.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.565+0000 7fb199425740 1 -- 172.21.15.70:0/3491139660 shutdown_connections 2024-04-01T01:44:53.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.565+0000 7fb199425740 1 -- 172.21.15.70:0/3491139660 wait complete. 2024-04-01T01:44:53.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.565+0000 7fb199425740 20 client.18780 trim_cache size 0 max 16384 2024-04-01T01:44:53.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.594+0000 7fb199425740 1 Processor -- start 2024-04-01T01:44:53.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.594+0000 7fb199425740 1 -- start start 2024-04-01T01:44:53.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.595+0000 7fb199425740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18a8d0 0x56317e1a2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.595+0000 7fb199425740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e18b2c0 0x56317e19c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.595+0000 7fb18ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18a8d0 0x56317e1a2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.595+0000 7fb18ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18a8d0 0x56317e1a2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33644/0 (socket says 172.21.15.70:33644) 2024-04-01T01:44:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.595+0000 7fb18ffff640 1 -- 172.21.15.70:0/2333780383 learned_addr learned my addr 172.21.15.70:0/2333780383 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:53.595 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.595+0000 7fb199425740 1 --2- 172.21.15.70:0/2333780383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e190430 0x56317e1a8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.596+0000 7fb194d02640 1 --2- 172.21.15.70:0/2333780383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e190430 0x56317e1a8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.595+0000 7fb199425740 1 -- 172.21.15.70:0/2333780383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb17c00fd90 con 0x56317e18b2c0 2024-04-01T01:44:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.596+0000 7fb199425740 1 -- 172.21.15.70:0/2333780383 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb17c0118d0 con 0x56317e18a8d0 2024-04-01T01:44:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.596+0000 7fb199425740 1 -- 172.21.15.70:0/2333780383 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb17c010460 con 0x56317e190430 2024-04-01T01:44:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.596+0000 7fb18f7fe640 1 --2- 172.21.15.70:0/2333780383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e18b2c0 0x56317e19c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.597+0000 7fb194d02640 1 -- 172.21.15.70:0/2333780383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18a8d0 msgr2=0x56317e1a2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:53.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.597+0000 7fb194d02640 1 --2- 172.21.15.70:0/2333780383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18a8d0 0x56317e1a2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.597+0000 7fb194d02640 1 -- 172.21.15.70:0/2333780383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e18b2c0 msgr2=0x56317e19c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.597+0000 7fb194d02640 1 --2- 172.21.15.70:0/2333780383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e18b2c0 0x56317e19c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.597+0000 7fb194d02640 1 -- 172.21.15.70:0/2333780383 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56317e18bf90 con 0x56317e190430 2024-04-01T01:44:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.597+0000 7fb194d02640 1 --2- 172.21.15.70:0/2333780383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e190430 0x56317e1a8510 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb18802c640 tx=0x7fb188005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1b097009a54a148 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.598+0000 7fb175ffb640 1 -- 172.21.15.70:0/2333780383 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18805a070 con 0x56317e190430 2024-04-01T01:44:53.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.598+0000 7fb175ffb640 1 -- 172.21.15.70:0/2333780383 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb18804d030 con 0x56317e190430 2024-04-01T01:44:53.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.598+0000 7fb175ffb640 1 -- 172.21.15.70:0/2333780383 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb188039040 con 0x56317e190430 2024-04-01T01:44:53.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.599+0000 7fb199425740 1 -- 172.21.15.70:0/2333780383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e190430 msgr2=0x56317e1a8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:53.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.599+0000 7fb199425740 1 --2- 172.21.15.70:0/2333780383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e190430 0x56317e1a8510 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb18802c640 tx=0x7fb188005ef0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.601+0000 7fb199425740 1 -- 172.21.15.70:0/2333780383 shutdown_connections 2024-04-01T01:44:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.602+0000 7fb199425740 1 --2- 172.21.15.70:0/2333780383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e190430 0x56317e1a8510 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.602+0000 7fb199425740 1 --2- 172.21.15.70:0/2333780383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e18b2c0 0x56317e19c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.602+0000 7fb199425740 1 --2- 172.21.15.70:0/2333780383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18a8d0 0x56317e1a2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.602+0000 7fb199425740 1 -- 172.21.15.70:0/2333780383 >> 172.21.15.70:0/2333780383 conn(0x56317e185e40 msgr2=0x56317e186220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.602+0000 7fb199425740 1 -- 172.21.15.70:0/2333780383 shutdown_connections 2024-04-01T01:44:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.602+0000 7fb199425740 1 -- 172.21.15.70:0/2333780383 wait complete. 2024-04-01T01:44:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.602+0000 7fb199425740 1 -- start start 2024-04-01T01:44:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.603+0000 7fb199425740 1 Processor -- start 2024-04-01T01:44:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.603+0000 7fb199425740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18a8d0 0x56317e199170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.603+0000 7fb199425740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e18b2c0 0x56317e1996b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.603+0000 7fb199425740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e190430 0x56317e19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.603+0000 7fb199425740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb17c011200 con 0x56317e18b2c0 2024-04-01T01:44:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.603+0000 7fb199425740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb17c010b30 con 0x56317e18a8d0 2024-04-01T01:44:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.603+0000 7fb199425740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb17c00dd70 con 0x56317e190430 2024-04-01T01:44:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.603+0000 7fb199425740 10 client.0 _ll_register_callbacks cb 0x7fb188009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.604+0000 7fb18ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18a8d0 0x56317e199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.604+0000 7fb18ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18a8d0 0x56317e199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33650/0 (socket says 172.21.15.70:33650) 2024-04-01T01:44:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.604+0000 7fb18ffff640 1 -- 172.21.15.70:0/1397343002 learned_addr learned my addr 172.21.15.70:0/1397343002 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:53.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.604+0000 7fb18ffff640 1 -- 172.21.15.70:0/1397343002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e190430 msgr2=0x56317e19a6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:44:53.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.604+0000 7fb18ffff640 1 --2- 172.21.15.70:0/1397343002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e190430 0x56317e19a6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.604+0000 7fb18ffff640 1 -- 172.21.15.70:0/1397343002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e18b2c0 msgr2=0x56317e1996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:53.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.604+0000 7fb18ffff640 1 --2- 172.21.15.70:0/1397343002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e18b2c0 0x56317e1996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:53.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.604+0000 7fb18ffff640 1 -- 172.21.15.70:0/1397343002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56317e19acb0 con 0x56317e18a8d0 2024-04-01T01:44:53.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.604+0000 7fb18ffff640 1 --2- 172.21.15.70:0/1397343002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18a8d0 0x56317e199170 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb180004610 tx=0x7fb18001e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93c064d629390af5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:53.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.605+0000 7fb16f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:44:53.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.605+0000 7fb16f7fe640 1 -- 172.21.15.70:0/1397343002 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb180020040 con 0x56317e18a8d0 2024-04-01T01:44:53.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.605+0000 7fb199425740 1 -- 172.21.15.70:0/1397343002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb18804d030 con 0x56317e18a8d0 2024-04-01T01:44:53.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.605+0000 7fb16f7fe640 1 -- 172.21.15.70:0/1397343002 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb18009ae40 con 0x56317e18a8d0 2024-04-01T01:44:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.605+0000 7fb16f7fe640 1 -- 172.21.15.70:0/1397343002 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb1800a4400 con 0x56317e18a8d0 2024-04-01T01:44:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.605+0000 7fb199425740 10 client.28817 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.605+0000 7fb199425740 1 -- 172.21.15.70:0/1397343002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56317e19bfc0 con 0x56317e18a8d0 2024-04-01T01:44:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.606+0000 7fb16f7fe640 1 -- 172.21.15.70:0/1397343002 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb18804d030 con 0x56317e18a8d0 2024-04-01T01:44:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.606+0000 7fb16f7fe640 1 -- 172.21.15.70:0/1397343002 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb18009a8b0 con 0x56317e18a8d0 2024-04-01T01:44:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.606+0000 7fb199425740 10 client.28817 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.606+0000 7fb199425740 10 client.28817 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.606+0000 7fb199425740 20 client.28817 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.606+0000 7fb199425740 1 --2- 172.21.15.70:0/1397343002 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56317e217b50 0x56317e219f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.606+0000 7fb199425740 4 client.28817 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:44:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.606+0000 7fb199425740 1 -- 172.21.15.70:0/1397343002 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56317e0fbe30 con 0x56317e217b50 2024-04-01T01:44:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.606+0000 7fb18f7fe640 1 --2- 172.21.15.70:0/1397343002 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56317e217b50 0x56317e219f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.607+0000 7fb16f7fe640 10 client.28817 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:53.607+0000 7fb18f7fe640 1 --2- 172.21.15.70:0/1397343002 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56317e217b50 0x56317e219f40 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b3b895e9683191d server_cookie=115bef040fe2769d in_seq=0 out_seq=0 2024-04-01T01:44:55.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:54 smithi189 ceph-mon[35753]: pgmap v702: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 9.7 MiB/s wr, 794 op/s 2024-04-01T01:44:55.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:54 smithi070 ceph-mon[32179]: pgmap v702: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 9.7 MiB/s wr, 794 op/s 2024-04-01T01:44:55.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:54 smithi081 ceph-mon[35142]: pgmap v702: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 9.7 MiB/s wr, 794 op/s 2024-04-01T01:44:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.113+0000 7fb16f7fe640 1 -- 172.21.15.70:0/1397343002 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56317e0fbe30 con 0x56317e217b50 2024-04-01T01:44:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.113+0000 7fb16f7fe640 10 client.28817 handle_command_reply: tid=0 2024-04-01T01:44:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.114+0000 7fb199425740 1 client.28817 shutdown 2024-04-01T01:44:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.114+0000 7fb199425740 1 -- 172.21.15.70:0/1397343002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18a8d0 msgr2=0x56317e199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.114+0000 7fb199425740 1 --2- 172.21.15.70:0/1397343002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18a8d0 0x56317e199170 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb180004610 tx=0x7fb18001e000 comp rx=0 tx=0).stop 2024-04-01T01:44:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.114+0000 7fb199425740 1 -- 172.21.15.70:0/1397343002 shutdown_connections 2024-04-01T01:44:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.114+0000 7fb199425740 1 --2- 172.21.15.70:0/1397343002 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56317e217b50 0x56317e219f40 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.114+0000 7fb199425740 1 --2- 172.21.15.70:0/1397343002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56317e190430 0x56317e19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.114+0000 7fb199425740 1 --2- 172.21.15.70:0/1397343002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56317e18b2c0 0x56317e1996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.114+0000 7fb199425740 1 --2- 172.21.15.70:0/1397343002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56317e18a8d0 0x56317e199170 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:56.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.114+0000 7fb199425740 1 -- 172.21.15.70:0/1397343002 >> 172.21.15.70:0/1397343002 conn(0x56317e185e40 msgr2=0x56317e18c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:56.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.114+0000 7fb16f7fe640 0 client.28817 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:56.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.114+0000 7fb199425740 1 -- 172.21.15.70:0/1397343002 shutdown_connections 2024-04-01T01:44:56.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.114+0000 7fb199425740 1 -- 172.21.15.70:0/1397343002 wait complete. 2024-04-01T01:44:56.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.115+0000 7fb199425740 20 client.28817 trim_cache size 0 max 16384 2024-04-01T01:44:56.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:44:56.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:44:56.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:44:56.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:44:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.120+0000 7fb196765640 1 -- 172.21.15.70:0/3051713531 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb1780a2730 msgr2=0x7fb1780a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.120+0000 7fb196765640 1 --2- 172.21.15.70:0/3051713531 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb1780a2730 0x7fb1780a4bf0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fb184005fd0 tx=0x7fb184005850 comp rx=0 tx=0).stop 2024-04-01T01:44:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.120+0000 7fb196765640 1 -- 172.21.15.70:0/3051713531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb190107db0 msgr2=0x7fb19019dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.120+0000 7fb196765640 1 --2- 172.21.15.70:0/3051713531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb190107db0 0x7fb19019dea0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb18000d770 tx=0x7fb18000dc30 comp rx=0 tx=0).stop 2024-04-01T01:44:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.120+0000 7fb196765640 1 -- 172.21.15.70:0/3051713531 shutdown_connections 2024-04-01T01:44:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.120+0000 7fb196765640 1 --2- 172.21.15.70:0/3051713531 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb1780a2730 0x7fb1780a4bf0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:56.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.120+0000 7fb196765640 1 --2- 172.21.15.70:0/3051713531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb190109960 0x7fb1901a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:56.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.120+0000 7fb196765640 1 --2- 172.21.15.70:0/3051713531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb190108fa0 0x7fb19019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:56.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.120+0000 7fb196765640 1 --2- 172.21.15.70:0/3051713531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb190107db0 0x7fb19019dea0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:56.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.120+0000 7fb196765640 1 -- 172.21.15.70:0/3051713531 >> 172.21.15.70:0/3051713531 conn(0x7fb190076d70 msgr2=0x7fb19010c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:56.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.121+0000 7fb196765640 1 -- 172.21.15.70:0/3051713531 shutdown_connections 2024-04-01T01:44:56.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:44:56.122+0000 7fb196765640 1 -- 172.21.15.70:0/3051713531 wait complete. 2024-04-01T01:44:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 610.0 s, 859.4 tps, lat 115.970 ms stddev 80.474 2024-04-01T01:44:57.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:56 smithi189 ceph-mon[35753]: pgmap v703: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 9.6 MiB/s wr, 800 op/s 2024-04-01T01:44:57.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:56 smithi070 ceph-mon[32179]: pgmap v703: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 9.6 MiB/s wr, 800 op/s 2024-04-01T01:44:57.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:56 smithi081 ceph-mon[35142]: pgmap v703: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 9.6 MiB/s wr, 800 op/s 2024-04-01T01:44:57.834 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:44:57.835 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:44:58.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.020+0000 7f7e0d8a0640 1 -- 172.21.15.70:0/1514268976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e081097e0 msgr2=0x7f7e0810e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:58.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.020+0000 7f7e0d8a0640 1 --2- 172.21.15.70:0/1514268976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e081097e0 0x7f7e0810e080 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7e00009ca0 tx=0x7f7e0002c100 comp rx=0 tx=0).stop 2024-04-01T01:44:58.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.020+0000 7f7e0d8a0640 1 -- 172.21.15.70:0/1514268976 shutdown_connections 2024-04-01T01:44:58.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.020+0000 7f7e0d8a0640 1 --2- 172.21.15.70:0/1514268976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e081097e0 0x7f7e0810e080 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.020+0000 7f7e0d8a0640 1 --2- 172.21.15.70:0/1514268976 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e08108e20 0x7f7e081092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.020+0000 7f7e0d8a0640 1 --2- 172.21.15.70:0/1514268976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e08072b60 0x7f7e08072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.020+0000 7f7e0d8a0640 1 -- 172.21.15.70:0/1514268976 >> 172.21.15.70:0/1514268976 conn(0x7f7e0806e630 msgr2=0x7f7e08070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:58.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.021+0000 7f7e0d8a0640 1 -- 172.21.15.70:0/1514268976 shutdown_connections 2024-04-01T01:44:58.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.021+0000 7f7e0d8a0640 1 -- 172.21.15.70:0/1514268976 wait complete. 2024-04-01T01:44:58.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.022+0000 7f7e0d8a0640 1 Processor -- start 2024-04-01T01:44:58.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.022+0000 7f7e0d8a0640 1 -- start start 2024-04-01T01:44:58.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.022+0000 7f7e0d8a0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e08072b60 0x7f7e081a65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.022+0000 7f7e0d8a0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e08108e20 0x7f7e081a6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.022+0000 7f7e0d8a0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e081097e0 0x7f7e081adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.022+0000 7f7e0d8a0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0811bf90 con 0x7f7e08072b60 2024-04-01T01:44:58.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.022+0000 7f7e0d8a0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0811be10 con 0x7f7e08108e20 2024-04-01T01:44:58.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.022+0000 7f7e0d8a0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0811c110 con 0x7f7e081097e0 2024-04-01T01:44:58.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.022+0000 7f7e0d09f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e081097e0 0x7f7e081adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.022+0000 7f7e0d09f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e081097e0 0x7f7e081adb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39190/0 (socket says 172.21.15.70:39190) 2024-04-01T01:44:58.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.022+0000 7f7e0d09f640 1 -- 172.21.15.70:0/3650668824 learned_addr learned my addr 172.21.15.70:0/3650668824 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:58.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.023+0000 7f7e07fff640 1 --2- 172.21.15.70:0/3650668824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e08108e20 0x7f7e081a6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.023+0000 7f7e07fff640 1 -- 172.21.15.70:0/3650668824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e081097e0 msgr2=0x7f7e081adb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:58.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.023+0000 7f7e07fff640 1 --2- 172.21.15.70:0/3650668824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e081097e0 0x7f7e081adb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.023+0000 7f7e07fff640 1 -- 172.21.15.70:0/3650668824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e08072b60 msgr2=0x7f7e081a65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:58.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.023+0000 7f7e07fff640 1 --2- 172.21.15.70:0/3650668824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e08072b60 0x7f7e081a65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.023+0000 7f7e07fff640 1 -- 172.21.15.70:0/3650668824 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e0810ffb0 con 0x7f7e08108e20 2024-04-01T01:44:58.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.023+0000 7f7e07fff640 1 --2- 172.21.15.70:0/3650668824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e08108e20 0x7f7e081a6b10 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00e9f0 tx=0x7f7dfc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:58.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.025+0000 7f7e05ffb640 1 -- 172.21.15.70:0/3650668824 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc00ccc0 con 0x7f7e08108e20 2024-04-01T01:44:58.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.025+0000 7f7e0d8a0640 1 -- 172.21.15.70:0/3650668824 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e081102a0 con 0x7f7e08108e20 2024-04-01T01:44:58.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.025+0000 7f7e0d8a0640 1 -- 172.21.15.70:0/3650668824 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e081107f0 con 0x7f7e08108e20 2024-04-01T01:44:58.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.025+0000 7f7e05ffb640 1 -- 172.21.15.70:0/3650668824 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7dfc0108e0 con 0x7f7e08108e20 2024-04-01T01:44:58.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.025+0000 7f7e05ffb640 1 -- 172.21.15.70:0/3650668824 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc018840 con 0x7f7e08108e20 2024-04-01T01:44:58.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.029+0000 7f7e05ffb640 1 -- 172.21.15.70:0/3650668824 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f7dfc010430 con 0x7f7e08108e20 2024-04-01T01:44:58.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.030+0000 7f7e05ffb640 1 --2- 172.21.15.70:0/3650668824 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7df00a2670 0x7f7df00a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.030+0000 7f7e05ffb640 1 -- 172.21.15.70:0/3650668824 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f7dfc0ced60 con 0x7f7e08108e20 2024-04-01T01:44:58.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.030+0000 7f7e117c0740 1 Processor -- start 2024-04-01T01:44:58.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.030+0000 7f7e117c0740 1 -- start start 2024-04-01T01:44:58.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.030+0000 7f7e117c0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958490 0x55a7219588b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.030+0000 7f7e117c0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a721958e80 0x55a72195d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.031+0000 7f7e117c0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 0x55a721960480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.031+0000 7f7e117c0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a7218bd180 con 0x55a721958e80 2024-04-01T01:44:58.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.031+0000 7f7e117c0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a721865b70 con 0x55a721958490 2024-04-01T01:44:58.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.031+0000 7f7e117c0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a7218b9220 con 0x55a72195dff0 2024-04-01T01:44:58.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.031+0000 7f7e0c89e640 1 --2- 172.21.15.70:0/3650668824 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7df00a2670 0x7f7df00a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.031+0000 7f7e0d09f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 0x55a721960480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.031+0000 7f7e0d09f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 0x55a721960480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39198/0 (socket says 172.21.15.70:39198) 2024-04-01T01:44:58.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.031+0000 7f7e0d09f640 1 -- 172.21.15.70:0/3512709682 learned_addr learned my addr 172.21.15.70:0/3512709682 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:58.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.031+0000 7f7e07fff640 1 --2- 172.21.15.70:0/3512709682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a721958e80 0x55a72195d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.032+0000 7f7e0d09f640 1 -- 172.21.15.70:0/3512709682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958490 msgr2=0x55a7219588b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:44:58.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.032+0000 7f7e0d09f640 1 --2- 172.21.15.70:0/3512709682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958490 0x55a7219588b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.032+0000 7f7e0d09f640 1 -- 172.21.15.70:0/3512709682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a721958e80 msgr2=0x55a72195d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:58.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.032+0000 7f7e0d09f640 1 --2- 172.21.15.70:0/3512709682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a721958e80 0x55a72195d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.032+0000 7f7e0d09f640 1 -- 172.21.15.70:0/3512709682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7218797c0 con 0x55a72195dff0 2024-04-01T01:44:58.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.032+0000 7f7e07fff640 1 --2- 172.21.15.70:0/3512709682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a721958e80 0x55a72195d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:58.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.032+0000 7f7e0d09f640 1 --2- 172.21.15.70:0/3512709682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 0x55a721960480 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7e00037040 tx=0x7f7e00009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c466f8f1772ae442 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:58.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.034+0000 7f7deeffd640 1 -- 172.21.15.70:0/3512709682 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e00043070 con 0x55a72195dff0 2024-04-01T01:44:58.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.034+0000 7f7deeffd640 1 -- 172.21.15.70:0/3512709682 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e00032dc0 con 0x55a72195dff0 2024-04-01T01:44:58.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.034+0000 7f7deeffd640 1 -- 172.21.15.70:0/3512709682 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e00039040 con 0x55a72195dff0 2024-04-01T01:44:58.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.037+0000 7f7e117c0740 1 -- 172.21.15.70:0/3512709682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 msgr2=0x55a721960480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:58.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.037+0000 7f7e117c0740 1 --2- 172.21.15.70:0/3512709682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 0x55a721960480 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7e00037040 tx=0x7f7e00009800 comp rx=0 tx=0).stop 2024-04-01T01:44:58.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.037+0000 7f7e117c0740 1 -- 172.21.15.70:0/3512709682 shutdown_connections 2024-04-01T01:44:58.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.037+0000 7f7e117c0740 1 --2- 172.21.15.70:0/3512709682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 0x55a721960480 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.037+0000 7f7e117c0740 1 --2- 172.21.15.70:0/3512709682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a721958e80 0x55a72195d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.037+0000 7f7e117c0740 1 --2- 172.21.15.70:0/3512709682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958490 0x55a7219588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.037+0000 7f7e117c0740 1 -- 172.21.15.70:0/3512709682 >> 172.21.15.70:0/3512709682 conn(0x55a721953a00 msgr2=0x55a721953e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:58.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.038+0000 7f7e117c0740 1 -- 172.21.15.70:0/3512709682 shutdown_connections 2024-04-01T01:44:58.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.038+0000 7f7e0c89e640 1 --2- 172.21.15.70:0/3650668824 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7df00a2670 0x7f7df00a4b30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x55a7219596e0 tx=0x7f7df800b460 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:58.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.038+0000 7f7e117c0740 1 -- 172.21.15.70:0/3512709682 wait complete. 2024-04-01T01:44:58.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.039+0000 7f7e117c0740 1 -- start start 2024-04-01T01:44:58.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.039+0000 7f7e117c0740 1 Processor -- start 2024-04-01T01:44:58.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.042+0000 7f7e117c0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958490 0x55a721962970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.042+0000 7f7e117c0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a721958e80 0x55a721962eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.042+0000 7f7e117c0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 0x55a721969ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.042+0000 7f7e117c0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a721960f40 con 0x55a721958e80 2024-04-01T01:44:58.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.042+0000 7f7e117c0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a72185b4a0 con 0x55a721958490 2024-04-01T01:44:58.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.042+0000 7f7e117c0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a7219610c0 con 0x55a72195dff0 2024-04-01T01:44:58.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.042+0000 7f7e117c0740 10 client.0 _ll_register_callbacks cb 0x55a72185ed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:58.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.042+0000 7f7e0d09f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 0x55a721969ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.042+0000 7f7e0d09f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 0x55a721969ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39212/0 (socket says 172.21.15.70:39212) 2024-04-01T01:44:58.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.042+0000 7f7e0d09f640 1 -- 172.21.15.70:0/254714937 learned_addr learned my addr 172.21.15.70:0/254714937 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:58.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.043+0000 7f7e0c89e640 1 --2- 172.21.15.70:0/254714937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958490 0x55a721962970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.043+0000 7f7e07fff640 1 --2- 172.21.15.70:0/254714937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a721958e80 0x55a721962eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.043+0000 7f7e0d09f640 1 -- 172.21.15.70:0/254714937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958490 msgr2=0x55a721962970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:58.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.043+0000 7f7e0d09f640 1 --2- 172.21.15.70:0/254714937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958490 0x55a721962970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.043+0000 7f7e0d09f640 1 -- 172.21.15.70:0/254714937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a721958e80 msgr2=0x55a721962eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:58.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.043+0000 7f7e0d09f640 1 --2- 172.21.15.70:0/254714937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a721958e80 0x55a721962eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.043+0000 7f7e0d09f640 1 -- 172.21.15.70:0/254714937 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a72169b7d0 con 0x55a72195dff0 2024-04-01T01:44:58.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.044+0000 7f7e07fff640 1 --2- 172.21.15.70:0/254714937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a721958e80 0x55a721962eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:44:58.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.044+0000 7f7e0d09f640 1 --2- 172.21.15.70:0/254714937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 0x55a721969ef0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7e00006010 tx=0x7f7e0002c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=973c748b271f7980 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:58.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.044+0000 7f7de7fff640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:44:58.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.044+0000 7f7de7fff640 1 -- 172.21.15.70:0/254714937 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0004c030 con 0x55a72195dff0 2024-04-01T01:44:58.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.044+0000 7f7de7fff640 1 -- 172.21.15.70:0/254714937 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f7e00032bc0 con 0x55a72195dff0 2024-04-01T01:44:58.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.045+0000 7f7e117c0740 1 -- 172.21.15.70:0/254714937 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e00032dc0 con 0x55a72195dff0 2024-04-01T01:44:58.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.046+0000 7f7de7fff640 1 -- 172.21.15.70:0/254714937 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7e00057050 con 0x55a72195dff0 2024-04-01T01:44:58.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.046+0000 7f7e117c0740 10 client.28621 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:58.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.046+0000 7f7e117c0740 1 -- 172.21.15.70:0/254714937 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a7215d13f0 con 0x55a72195dff0 2024-04-01T01:44:58.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.047+0000 7f7de7fff640 1 -- 172.21.15.70:0/254714937 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e00032dc0 con 0x55a72195dff0 2024-04-01T01:44:58.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.047+0000 7f7de7fff640 1 -- 172.21.15.70:0/254714937 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f7e00048680 con 0x55a72195dff0 2024-04-01T01:44:58.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.047+0000 7f7e117c0740 10 client.28621 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:58.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.047+0000 7f7e117c0740 10 client.28621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:58.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.047+0000 7f7e117c0740 20 client.28621 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:58.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.047+0000 7f7e117c0740 1 --2- 172.21.15.70:0/254714937 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a72196e3e0 0x55a7219707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.047+0000 7f7e117c0740 4 client.28621 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:58.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.047+0000 7f7e117c0740 1 -- 172.21.15.70:0/254714937 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7218ca6c0 con 0x55a72196e3e0 2024-04-01T01:44:58.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.048+0000 7f7e0c89e640 1 --2- 172.21.15.70:0/254714937 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a72196e3e0 0x55a7219707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.048+0000 7f7e0c89e640 1 --2- 172.21.15.70:0/254714937 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a72196e3e0 0x55a7219707d0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d792e3287ad9f2b server_cookie=14d9fcf413745192 in_seq=0 out_seq=0 2024-04-01T01:44:58.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.049+0000 7f7de7fff640 10 client.28621 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:58.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.049+0000 7f7de7fff640 1 -- 172.21.15.70:0/254714937 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a7218ca6c0 con 0x55a72196e3e0 2024-04-01T01:44:58.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.049+0000 7f7de7fff640 10 client.28621 handle_command_reply: tid=0 2024-04-01T01:44:58.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.050+0000 7f7e117c0740 1 client.28621 shutdown 2024-04-01T01:44:58.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.051+0000 7f7e117c0740 1 -- 172.21.15.70:0/254714937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 msgr2=0x55a721969ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:58.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.051+0000 7f7e117c0740 1 --2- 172.21.15.70:0/254714937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 0x55a721969ef0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7e00006010 tx=0x7f7e0002c100 comp rx=0 tx=0).stop 2024-04-01T01:44:58.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.051+0000 7f7e117c0740 1 -- 172.21.15.70:0/254714937 shutdown_connections 2024-04-01T01:44:58.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.051+0000 7f7e117c0740 1 --2- 172.21.15.70:0/254714937 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a72196e3e0 0x55a7219707d0 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.051+0000 7f7e117c0740 1 --2- 172.21.15.70:0/254714937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 0x55a721969ef0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.051+0000 7f7e117c0740 1 --2- 172.21.15.70:0/254714937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a721958e80 0x55a721962eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.051+0000 7f7e117c0740 1 --2- 172.21.15.70:0/254714937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958490 0x55a721962970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.051+0000 7f7e117c0740 1 -- 172.21.15.70:0/254714937 >> 172.21.15.70:0/254714937 conn(0x55a721953a00 msgr2=0x55a721952c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:58.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.052+0000 7f7de7fff640 0 client.28621 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:58.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.052+0000 7f7e117c0740 1 -- 172.21.15.70:0/254714937 shutdown_connections 2024-04-01T01:44:58.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.052+0000 7f7e117c0740 1 -- 172.21.15.70:0/254714937 wait complete. 2024-04-01T01:44:58.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.052+0000 7f7e117c0740 20 client.28621 trim_cache size 0 max 16384 2024-04-01T01:44:58.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.091+0000 7f7e117c0740 1 Processor -- start 2024-04-01T01:44:58.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.091+0000 7f7e117c0740 1 -- start start 2024-04-01T01:44:58.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.091+0000 7f7e117c0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a721958490 0x55a721952390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.091+0000 7f7e117c0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958e80 0x55a721969eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.091+0000 7f7e117c0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 0x55a7219528d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.092+0000 7f7e117c0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7df400fda0 con 0x55a721958490 2024-04-01T01:44:58.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.092+0000 7f7e07fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958e80 0x55a721969eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.092+0000 7f7e07fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958e80 0x55a721969eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42346/0 (socket says 172.21.15.70:42346) 2024-04-01T01:44:58.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.092+0000 7f7e07fff640 1 -- 172.21.15.70:0/3282561797 learned_addr learned my addr 172.21.15.70:0/3282561797 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:58.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.092+0000 7f7e117c0740 1 -- 172.21.15.70:0/3282561797 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7df40118e0 con 0x55a721958e80 2024-04-01T01:44:58.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.092+0000 7f7e117c0740 1 -- 172.21.15.70:0/3282561797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7df4010470 con 0x55a72195dff0 2024-04-01T01:44:58.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.092+0000 7f7e0d09f640 1 --2- 172.21.15.70:0/3282561797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 0x55a7219528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.092+0000 7f7e07fff640 1 -- 172.21.15.70:0/3282561797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 msgr2=0x55a7219528d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:58.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.092+0000 7f7e07fff640 1 --2- 172.21.15.70:0/3282561797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 0x55a7219528d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.092+0000 7f7e07fff640 1 -- 172.21.15.70:0/3282561797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a721958490 msgr2=0x55a721952390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:58.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.092+0000 7f7e07fff640 1 --2- 172.21.15.70:0/3282561797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a721958490 0x55a721952390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.092+0000 7f7e07fff640 1 -- 172.21.15.70:0/3282561797 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7215910f0 con 0x55a721958e80 2024-04-01T01:44:58.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.092+0000 7f7e07fff640 1 --2- 172.21.15.70:0/3282561797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958e80 0x55a721969eb0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc0cc430 tx=0x7f7dfc097040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7009548b2924b02 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:58.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.093+0000 7f7dee7fc640 1 -- 172.21.15.70:0/3282561797 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc0cc540 con 0x55a721958e80 2024-04-01T01:44:58.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.093+0000 7f7dee7fc640 1 -- 172.21.15.70:0/3282561797 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7dfc097a60 con 0x55a721958e80 2024-04-01T01:44:58.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.093+0000 7f7dee7fc640 1 -- 172.21.15.70:0/3282561797 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc097140 con 0x55a721958e80 2024-04-01T01:44:58.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.094+0000 7f7e117c0740 1 -- 172.21.15.70:0/3282561797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958e80 msgr2=0x55a721969eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:58.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.094+0000 7f7e117c0740 1 --2- 172.21.15.70:0/3282561797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958e80 0x55a721969eb0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc0cc430 tx=0x7f7dfc097040 comp rx=0 tx=0).stop 2024-04-01T01:44:58.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.095+0000 7f7e117c0740 1 -- 172.21.15.70:0/3282561797 shutdown_connections 2024-04-01T01:44:58.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.095+0000 7f7e117c0740 1 --2- 172.21.15.70:0/3282561797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a72195dff0 0x55a7219528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.095+0000 7f7e117c0740 1 --2- 172.21.15.70:0/3282561797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958e80 0x55a721969eb0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.095+0000 7f7e117c0740 1 --2- 172.21.15.70:0/3282561797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a721958490 0x55a721952390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.095+0000 7f7e117c0740 1 -- 172.21.15.70:0/3282561797 >> 172.21.15.70:0/3282561797 conn(0x55a721953a00 msgr2=0x55a721961c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:58.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.095+0000 7f7e117c0740 1 -- 172.21.15.70:0/3282561797 shutdown_connections 2024-04-01T01:44:58.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.095+0000 7f7e117c0740 1 -- 172.21.15.70:0/3282561797 wait complete. 2024-04-01T01:44:58.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.096+0000 7f7e117c0740 1 -- start start 2024-04-01T01:44:58.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.096+0000 7f7e117c0740 1 Processor -- start 2024-04-01T01:44:58.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.096+0000 7f7e117c0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958490 0x55a721967b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.096+0000 7f7e117c0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a721958e80 0x55a7219680a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.096+0000 7f7e117c0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a72195dff0 0x55a7219690b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.096+0000 7f7e0c89e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958490 0x55a721967b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.096+0000 7f7e0c89e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958490 0x55a721967b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42360/0 (socket says 172.21.15.70:42360) 2024-04-01T01:44:58.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.096+0000 7f7e0c89e640 1 -- 172.21.15.70:0/2651132296 learned_addr learned my addr 172.21.15.70:0/2651132296 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:58.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.096+0000 7f7e117c0740 1 -- 172.21.15.70:0/2651132296 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7df4011210 con 0x55a72195dff0 2024-04-01T01:44:58.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.096+0000 7f7e117c0740 1 -- 172.21.15.70:0/2651132296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7df4010b40 con 0x55a721958490 2024-04-01T01:44:58.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.096+0000 7f7e117c0740 1 -- 172.21.15.70:0/2651132296 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7df400dd80 con 0x55a721958e80 2024-04-01T01:44:58.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.096+0000 7f7e117c0740 10 client.0 _ll_register_callbacks cb 0x7f7dfc097690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:58.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.097+0000 7f7e0d09f640 1 --2- 172.21.15.70:0/2651132296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a72195dff0 0x55a7219690b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.098+0000 7f7e0c89e640 1 -- 172.21.15.70:0/2651132296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a721958e80 msgr2=0x55a7219680a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:58.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.098+0000 7f7e0c89e640 1 --2- 172.21.15.70:0/2651132296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a721958e80 0x55a7219680a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.098+0000 7f7e0c89e640 1 -- 172.21.15.70:0/2651132296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a72195dff0 msgr2=0x55a7219690b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:58.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.098+0000 7f7e0c89e640 1 --2- 172.21.15.70:0/2651132296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a72195dff0 0x55a7219690b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.098+0000 7f7e0c89e640 1 -- 172.21.15.70:0/2651132296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7219696a0 con 0x55a721958490 2024-04-01T01:44:58.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.098+0000 7f7e0c89e640 1 --2- 172.21.15.70:0/2651132296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958490 0x55a721967b60 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f7df8005fd0 tx=0x7f7df800b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97c5071db19acfe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:58.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.098+0000 7f7de7fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:44:58.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.099+0000 7f7de7fff640 1 -- 172.21.15.70:0/2651132296 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df8010cc0 con 0x55a721958490 2024-04-01T01:44:58.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.099+0000 7f7de7fff640 1 -- 172.21.15.70:0/2651132296 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f7df801e440 con 0x55a721958490 2024-04-01T01:44:58.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.099+0000 7f7e117c0740 1 -- 172.21.15.70:0/2651132296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dfc097a60 con 0x55a721958490 2024-04-01T01:44:58.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.099+0000 7f7de7fff640 1 -- 172.21.15.70:0/2651132296 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7df8032050 con 0x55a721958490 2024-04-01T01:44:58.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.100+0000 7f7e117c0740 10 client.28829 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:58.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.100+0000 7f7e117c0740 1 -- 172.21.15.70:0/2651132296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a7219d4c80 con 0x55a721958490 2024-04-01T01:44:58.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.102+0000 7f7de7fff640 1 -- 172.21.15.70:0/2651132296 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7df8017040 con 0x55a721958490 2024-04-01T01:44:58.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.102+0000 7f7de7fff640 1 -- 172.21.15.70:0/2651132296 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f7df801c040 con 0x55a721958490 2024-04-01T01:44:58.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.102+0000 7f7e117c0740 10 client.28829 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:58.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.102+0000 7f7e117c0740 10 client.28829 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:44:58.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.102+0000 7f7e117c0740 20 client.28829 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:58.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.102+0000 7f7e117c0740 1 --2- 172.21.15.70:0/2651132296 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a7219d5680 0x55a7219d7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.102+0000 7f7e117c0740 4 client.28829 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:44:58.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.102+0000 7f7e117c0740 1 -- 172.21.15.70:0/2651132296 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f7dfc097a60 con 0x55a7219d5680 2024-04-01T01:44:58.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.103+0000 7f7e07fff640 1 --2- 172.21.15.70:0/2651132296 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a7219d5680 0x55a7219d7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.104+0000 7f7e07fff640 1 --2- 172.21.15.70:0/2651132296 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a7219d5680 0x55a7219d7aa0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a820542f246f1c3b server_cookie=35449ac681090ad1 in_seq=0 out_seq=0 2024-04-01T01:44:58.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.104+0000 7f7de7fff640 10 client.28829 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:58.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.104+0000 7f7de7fff640 1 -- 172.21.15.70:0/2651132296 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f7dfc097a60 con 0x55a7219d5680 2024-04-01T01:44:58.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.104+0000 7f7de7fff640 10 client.28829 handle_command_reply: tid=0 2024-04-01T01:44:58.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.108+0000 7f7e117c0740 1 client.28829 shutdown 2024-04-01T01:44:58.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.109+0000 7f7e117c0740 1 -- 172.21.15.70:0/2651132296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958490 msgr2=0x55a721967b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:58.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.109+0000 7f7e117c0740 1 --2- 172.21.15.70:0/2651132296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958490 0x55a721967b60 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f7df8005fd0 tx=0x7f7df800b820 comp rx=0 tx=0).stop 2024-04-01T01:44:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.112+0000 7f7e117c0740 1 -- 172.21.15.70:0/2651132296 shutdown_connections 2024-04-01T01:44:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.112+0000 7f7e117c0740 1 --2- 172.21.15.70:0/2651132296 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a7219d5680 0x55a7219d7aa0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.112+0000 7f7e117c0740 1 --2- 172.21.15.70:0/2651132296 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a72195dff0 0x55a7219690b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.112+0000 7f7e117c0740 1 --2- 172.21.15.70:0/2651132296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a721958e80 0x55a7219680a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.112+0000 7f7e117c0740 1 --2- 172.21.15.70:0/2651132296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a721958490 0x55a721967b60 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.112+0000 7f7e117c0740 1 -- 172.21.15.70:0/2651132296 >> 172.21.15.70:0/2651132296 conn(0x55a721953a00 msgr2=0x55a721956c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.112+0000 7f7de7fff640 0 client.28829 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:44:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.113+0000 7f7e117c0740 1 -- 172.21.15.70:0/2651132296 shutdown_connections 2024-04-01T01:44:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.113+0000 7f7e117c0740 1 -- 172.21.15.70:0/2651132296 wait complete. 2024-04-01T01:44:58.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.114+0000 7f7e117c0740 20 client.28829 trim_cache size 0 max 16384 2024-04-01T01:44:58.115 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:44:58.116 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:44:58.116 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:44:58.116 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:44:58.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.119+0000 7f7e0d8a0640 1 -- 172.21.15.70:0/3650668824 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7df00a2670 msgr2=0x7f7df00a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:58.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.119+0000 7f7e0d8a0640 1 --2- 172.21.15.70:0/3650668824 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7df00a2670 0x7f7df00a4b30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x55a7219596e0 tx=0x7f7df800b460 comp rx=0 tx=0).stop 2024-04-01T01:44:58.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.119+0000 7f7e0d8a0640 1 -- 172.21.15.70:0/3650668824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e08108e20 msgr2=0x7f7e081a6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:58.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.119+0000 7f7e0d8a0640 1 --2- 172.21.15.70:0/3650668824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e08108e20 0x7f7e081a6b10 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00e9f0 tx=0x7f7dfc00eeb0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.120+0000 7f7e0d8a0640 1 -- 172.21.15.70:0/3650668824 shutdown_connections 2024-04-01T01:44:58.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.120+0000 7f7e0d8a0640 1 --2- 172.21.15.70:0/3650668824 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f7df00a2670 0x7f7df00a4b30 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.120+0000 7f7e0d8a0640 1 --2- 172.21.15.70:0/3650668824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e081097e0 0x7f7e081adb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.120+0000 7f7e0d8a0640 1 --2- 172.21.15.70:0/3650668824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e08108e20 0x7f7e081a6b10 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.120+0000 7f7e0d8a0640 1 --2- 172.21.15.70:0/3650668824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e08072b60 0x7f7e081a65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.120+0000 7f7e0d8a0640 1 -- 172.21.15.70:0/3650668824 >> 172.21.15.70:0/3650668824 conn(0x7f7e0806e630 msgr2=0x7f7e080702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.126+0000 7f7e0d8a0640 1 -- 172.21.15.70:0/3650668824 shutdown_connections 2024-04-01T01:44:58.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.127+0000 7f7e0d8a0640 1 -- 172.21.15.70:0/3650668824 wait complete. 2024-04-01T01:44:58.145 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:44:58.146 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:44:58.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.333+0000 7f30e2a1c640 1 -- 172.21.15.70:0/665960113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30dc10d830 msgr2=0x7f30dc10dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.333+0000 7f30e2a1c640 1 --2- 172.21.15.70:0/665960113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30dc10d830 0x7f30dc10dcb0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009be0 tx=0x7f30d402bf10 comp rx=0 tx=0).stop 2024-04-01T01:44:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.333+0000 7f30e2a1c640 1 -- 172.21.15.70:0/665960113 shutdown_connections 2024-04-01T01:44:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.333+0000 7f30e2a1c640 1 --2- 172.21.15.70:0/665960113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30dc10d830 0x7f30dc10dcb0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.333+0000 7f30e2a1c640 1 --2- 172.21.15.70:0/665960113 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30dc10ad30 0x7f30dc10d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.333+0000 7f30e2a1c640 1 --2- 172.21.15.70:0/665960113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30dc06aa20 0x7f30dc10a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.333+0000 7f30e2a1c640 1 -- 172.21.15.70:0/665960113 >> 172.21.15.70:0/665960113 conn(0x7f30dc06f440 msgr2=0x7f30dc071880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.333+0000 7f30e2a1c640 1 -- 172.21.15.70:0/665960113 shutdown_connections 2024-04-01T01:44:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.333+0000 7f30e2a1c640 1 -- 172.21.15.70:0/665960113 wait complete. 2024-04-01T01:44:58.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.334+0000 7f30e2a1c640 1 Processor -- start 2024-04-01T01:44:58.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.334+0000 7f30e2a1c640 1 -- start start 2024-04-01T01:44:58.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.334+0000 7f30e2a1c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30dc06aa20 0x7f30dc113d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.334+0000 7f30e2a1c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30dc10ad30 0x7f30dc114270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.334+0000 7f30e2a1c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30dc10d830 0x7f30dc1a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.334+0000 7f30e2a1c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f30dc11bf40 con 0x7f30dc10ad30 2024-04-01T01:44:58.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.334+0000 7f30e2a1c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f30dc11bdc0 con 0x7f30dc06aa20 2024-04-01T01:44:58.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.334+0000 7f30e2a1c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f30dc11c0c0 con 0x7f30dc10d830 2024-04-01T01:44:58.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.335+0000 7f30e0fb9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30dc10d830 0x7f30dc1a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.335+0000 7f30e0fb9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30dc10d830 0x7f30dc1a5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39238/0 (socket says 172.21.15.70:39238) 2024-04-01T01:44:58.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.335+0000 7f30e0fb9640 1 -- 172.21.15.70:0/4239946896 learned_addr learned my addr 172.21.15.70:0/4239946896 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:58.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.335+0000 7f30db7fe640 1 --2- 172.21.15.70:0/4239946896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30dc10ad30 0x7f30dc114270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.335+0000 7f30e0fb9640 1 -- 172.21.15.70:0/4239946896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30dc06aa20 msgr2=0x7f30dc113d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:58.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.335+0000 7f30e0fb9640 1 --2- 172.21.15.70:0/4239946896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30dc06aa20 0x7f30dc113d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.335+0000 7f30e0fb9640 1 -- 172.21.15.70:0/4239946896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30dc10ad30 msgr2=0x7f30dc114270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:58.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.335+0000 7f30e0fb9640 1 --2- 172.21.15.70:0/4239946896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30dc10ad30 0x7f30dc114270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.335+0000 7f30e0fb9640 1 -- 172.21.15.70:0/4239946896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30dc1a5b80 con 0x7f30dc10d830 2024-04-01T01:44:58.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.335+0000 7f30db7fe640 1 --2- 172.21.15.70:0/4239946896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30dc10ad30 0x7f30dc114270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:44:58.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.335+0000 7f30e0fb9640 1 --2- 172.21.15.70:0/4239946896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30dc10d830 0x7f30dc1a5640 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f30d4034fd0 tx=0x7f30d4033850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:58.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.336+0000 7f30d97fa640 1 -- 172.21.15.70:0/4239946896 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d4043020 con 0x7f30dc10d830 2024-04-01T01:44:58.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.336+0000 7f30e2a1c640 1 -- 172.21.15.70:0/4239946896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30dc1a5e10 con 0x7f30dc10d830 2024-04-01T01:44:58.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.336+0000 7f30e2a1c640 1 -- 172.21.15.70:0/4239946896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30dc1a6430 con 0x7f30dc10d830 2024-04-01T01:44:58.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.336+0000 7f30d97fa640 1 -- 172.21.15.70:0/4239946896 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30d4003b80 con 0x7f30dc10d830 2024-04-01T01:44:58.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.336+0000 7f30d97fa640 1 -- 172.21.15.70:0/4239946896 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d4034690 con 0x7f30dc10d830 2024-04-01T01:44:58.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.337+0000 7f30e2a1c640 1 -- 172.21.15.70:0/4239946896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30c00052c0 con 0x7f30dc10d830 2024-04-01T01:44:58.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.339+0000 7f30d97fa640 1 -- 172.21.15.70:0/4239946896 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f30d4046020 con 0x7f30dc10d830 2024-04-01T01:44:58.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.340+0000 7f30d97fa640 1 --2- 172.21.15.70:0/4239946896 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30cc0a25f0 0x7f30cc0a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:58.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.340+0000 7f30dbfff640 1 --2- 172.21.15.70:0/4239946896 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30cc0a25f0 0x7f30cc0a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:58.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.341+0000 7f30d97fa640 1 -- 172.21.15.70:0/4239946896 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f30d40f1d30 con 0x7f30dc10d830 2024-04-01T01:44:58.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.341+0000 7f30dbfff640 1 --2- 172.21.15.70:0/4239946896 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30cc0a25f0 0x7f30cc0a4ab0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f30dc115280 tx=0x7f30d0006c40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:58.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.347+0000 7f30d97fa640 1 -- 172.21.15.70:0/4239946896 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f30d4004820 con 0x7f30dc10d830 2024-04-01T01:44:58.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.771+0000 7f30e2a1c640 1 -- 172.21.15.70:0/4239946896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30c00055b0 con 0x7f30dc10d830 2024-04-01T01:44:58.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.773+0000 7f30d97fa640 1 -- 172.21.15.70:0/4239946896 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7f30d40beac0 con 0x7f30dc10d830 2024-04-01T01:44:58.773 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:44:58.774 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:44:58.774 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:44:58.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.777+0000 7f30e2a1c640 1 -- 172.21.15.70:0/4239946896 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30cc0a25f0 msgr2=0x7f30cc0a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:58.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.777+0000 7f30e2a1c640 1 --2- 172.21.15.70:0/4239946896 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30cc0a25f0 0x7f30cc0a4ab0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f30dc115280 tx=0x7f30d0006c40 comp rx=0 tx=0).stop 2024-04-01T01:44:58.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.777+0000 7f30e2a1c640 1 -- 172.21.15.70:0/4239946896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30dc10d830 msgr2=0x7f30dc1a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:58.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.777+0000 7f30e2a1c640 1 --2- 172.21.15.70:0/4239946896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30dc10d830 0x7f30dc1a5640 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f30d4034fd0 tx=0x7f30d4033850 comp rx=0 tx=0).stop 2024-04-01T01:44:58.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.777+0000 7f30e2a1c640 1 -- 172.21.15.70:0/4239946896 shutdown_connections 2024-04-01T01:44:58.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.777+0000 7f30e2a1c640 1 --2- 172.21.15.70:0/4239946896 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30cc0a25f0 0x7f30cc0a4ab0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.777+0000 7f30e2a1c640 1 --2- 172.21.15.70:0/4239946896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30dc10d830 0x7f30dc1a5640 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.777+0000 7f30e2a1c640 1 --2- 172.21.15.70:0/4239946896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f30dc10ad30 0x7f30dc114270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.777+0000 7f30e2a1c640 1 --2- 172.21.15.70:0/4239946896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30dc06aa20 0x7f30dc113d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.777+0000 7f30e2a1c640 1 -- 172.21.15.70:0/4239946896 >> 172.21.15.70:0/4239946896 conn(0x7f30dc06f440 msgr2=0x7f30dc108cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:58.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.779+0000 7f30e2a1c640 1 -- 172.21.15.70:0/4239946896 shutdown_connections 2024-04-01T01:44:58.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.779+0000 7f30e2a1c640 1 -- 172.21.15.70:0/4239946896 wait complete. 2024-04-01T01:44:58.800 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:44:58.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.999+0000 7fb9f846b640 1 -- 172.21.15.70:0/1178496616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f0108750 msgr2=0x7fb9f010ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:58.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.999+0000 7fb9f846b640 1 --2- 172.21.15.70:0/1178496616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f0108750 0x7fb9f010ab80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8009be0 tx=0x7fb9e802bef0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.999+0000 7fb9f846b640 1 -- 172.21.15.70:0/1178496616 shutdown_connections 2024-04-01T01:44:58.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.999+0000 7fb9f846b640 1 --2- 172.21.15.70:0/1178496616 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f010da70 0x7fb9f010def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.999+0000 7fb9f846b640 1 --2- 172.21.15.70:0/1178496616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9f010b0c0 0x7fb9f010d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.999+0000 7fb9f846b640 1 --2- 172.21.15.70:0/1178496616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f0108750 0x7fb9f010ab80 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:58.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:58.999+0000 7fb9f846b640 1 -- 172.21.15.70:0/1178496616 >> 172.21.15.70:0/1178496616 conn(0x7fb9f006e5b0 msgr2=0x7fb9f00709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.001+0000 7fb9f846b640 1 -- 172.21.15.70:0/1178496616 shutdown_connections 2024-04-01T01:44:59.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.001+0000 7fb9f846b640 1 -- 172.21.15.70:0/1178496616 wait complete. 2024-04-01T01:44:59.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.002+0000 7fb9f846b640 1 Processor -- start 2024-04-01T01:44:59.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.002+0000 7fb9f846b640 1 -- start start 2024-04-01T01:44:59.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.002+0000 7fb9f846b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f0108750 0x7fb9f01a67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.002+0000 7fb9f846b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f010b0c0 0x7fb9f01a6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.002+0000 7fb9f846b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9f010da70 0x7fb9f01add80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.002+0000 7fb9f846b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f011c100 con 0x7fb9f010b0c0 2024-04-01T01:44:59.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.002+0000 7fb9f846b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f011bf80 con 0x7fb9f0108750 2024-04-01T01:44:59.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.002+0000 7fb9f846b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f011c280 con 0x7fb9f010da70 2024-04-01T01:44:59.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.003+0000 7fb9f5a06640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f010b0c0 0x7fb9f01a6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.003+0000 7fb9f6207640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f0108750 0x7fb9f01a67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.003+0000 7fb9f5a06640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f010b0c0 0x7fb9f01a6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54128/0 (socket says 172.21.15.70:54128) 2024-04-01T01:44:59.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.003+0000 7fb9f6207640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f0108750 0x7fb9f01a67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42390/0 (socket says 172.21.15.70:42390) 2024-04-01T01:44:59.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.003+0000 7fb9f6a08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9f010da70 0x7fb9f01add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.003+0000 7fb9f5a06640 1 -- 172.21.15.70:0/3564740913 learned_addr learned my addr 172.21.15.70:0/3564740913 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.003+0000 7fb9f5a06640 1 -- 172.21.15.70:0/3564740913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9f010da70 msgr2=0x7fb9f01add80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.003+0000 7fb9f5a06640 1 --2- 172.21.15.70:0/3564740913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9f010da70 0x7fb9f01add80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.003+0000 7fb9f5a06640 1 -- 172.21.15.70:0/3564740913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f0108750 msgr2=0x7fb9f01a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.003+0000 7fb9f5a06640 1 --2- 172.21.15.70:0/3564740913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f0108750 0x7fb9f01a67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.003+0000 7fb9f5a06640 1 -- 172.21.15.70:0/3564740913 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f01ae3f0 con 0x7fb9f010b0c0 2024-04-01T01:44:59.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.004+0000 7fb9f5a06640 1 --2- 172.21.15.70:0/3564740913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f010b0c0 0x7fb9f01a6d10 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb9e000d950 tx=0x7fb9e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.004+0000 7fb9e77fe640 1 -- 172.21.15.70:0/3564740913 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e0017070 con 0x7fb9f010b0c0 2024-04-01T01:44:59.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.004+0000 7fb9e77fe640 1 -- 172.21.15.70:0/3564740913 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9e0009c30 con 0x7fb9f010b0c0 2024-04-01T01:44:59.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.004+0000 7fb9e77fe640 1 -- 172.21.15.70:0/3564740913 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e000f620 con 0x7fb9f010b0c0 2024-04-01T01:44:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.005+0000 7fb9f846b640 1 -- 172.21.15.70:0/3564740913 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9f01ae6e0 con 0x7fb9f010b0c0 2024-04-01T01:44:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.006+0000 7fb9f846b640 1 -- 172.21.15.70:0/3564740913 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9f01aebb0 con 0x7fb9f010b0c0 2024-04-01T01:44:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.010+0000 7fb9e77fe640 1 -- 172.21.15.70:0/3564740913 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb9e00109a0 con 0x7fb9f010b0c0 2024-04-01T01:44:59.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.011+0000 7fb9e77fe640 1 --2- 172.21.15.70:0/3564740913 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9dc0a2570 0x7fb9dc0a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.012+0000 7fb9e77fe640 1 -- 172.21.15.70:0/3564740913 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb9e00ce4e0 con 0x7fb9f010b0c0 2024-04-01T01:44:59.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.012+0000 7fb9f6207640 1 --2- 172.21.15.70:0/3564740913 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9dc0a2570 0x7fb9dc0a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.013+0000 7fb9f6207640 1 --2- 172.21.15.70:0/3564740913 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9dc0a2570 0x7fb9dc0a4a30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8006010 tx=0x7fb9e8000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.014+0000 7fb9fb04a740 1 Processor -- start 2024-04-01T01:44:59.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.014+0000 7fb9fb04a740 1 -- start start 2024-04-01T01:44:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.014+0000 7fb9fb04a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f336c490 0x5577f336c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.015+0000 7fb9fb04a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336ce80 0x5577f3371980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.015+0000 7fb9fb04a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f3371ff0 0x5577f3374480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.015+0000 7fb9f6207640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f336c490 0x5577f336c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.015+0000 7fb9f6207640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f336c490 0x5577f336c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39258/0 (socket says 172.21.15.70:39258) 2024-04-01T01:44:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.015+0000 7fb9f6207640 1 -- 172.21.15.70:0/2252635714 learned_addr learned my addr 172.21.15.70:0/2252635714 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.015+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2252635714 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5577f32d11b0 con 0x5577f336ce80 2024-04-01T01:44:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.015+0000 7fb9f6a08640 1 --2- 172.21.15.70:0/2252635714 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f3371ff0 0x5577f3374480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.015+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2252635714 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5577f3279c00 con 0x5577f3371ff0 2024-04-01T01:44:59.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.015+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2252635714 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5577f32cd250 con 0x5577f336c490 2024-04-01T01:44:59.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.016+0000 7fb9f6a08640 1 -- 172.21.15.70:0/2252635714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f336c490 msgr2=0x5577f336c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.016+0000 7fb9f6a08640 1 --2- 172.21.15.70:0/2252635714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f336c490 0x5577f336c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.016+0000 7fb9f6a08640 1 -- 172.21.15.70:0/2252635714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336ce80 msgr2=0x5577f3371980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:44:59.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.016+0000 7fb9f6a08640 1 --2- 172.21.15.70:0/2252635714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336ce80 0x5577f3371980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.016+0000 7fb9f6a08640 1 -- 172.21.15.70:0/2252635714 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577f32993d0 con 0x5577f3371ff0 2024-04-01T01:44:59.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.017+0000 7fb9f6a08640 1 --2- 172.21.15.70:0/2252635714 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f3371ff0 0x5577f3374480 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec005fd0 tx=0x7fb9ec0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a64e8ef9c5e1b866 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.018+0000 7fb9e4bfe640 1 -- 172.21.15.70:0/2252635714 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec012450 con 0x5577f3371ff0 2024-04-01T01:44:59.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.018+0000 7fb9e4bfe640 1 -- 172.21.15.70:0/2252635714 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9ec012aa0 con 0x5577f3371ff0 2024-04-01T01:44:59.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.019+0000 7fb9e4bfe640 1 -- 172.21.15.70:0/2252635714 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec00fa70 con 0x5577f3371ff0 2024-04-01T01:44:59.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.019+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2252635714 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f3371ff0 msgr2=0x5577f3374480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.019+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/2252635714 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f3371ff0 0x5577f3374480 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec005fd0 tx=0x7fb9ec0117b0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.020+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2252635714 shutdown_connections 2024-04-01T01:44:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.020+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/2252635714 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f3371ff0 0x5577f3374480 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.020+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/2252635714 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336ce80 0x5577f3371980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.020+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/2252635714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f336c490 0x5577f336c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.020+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2252635714 >> 172.21.15.70:0/2252635714 conn(0x5577f3367a00 msgr2=0x5577f3367e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.020+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2252635714 shutdown_connections 2024-04-01T01:44:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.020+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2252635714 wait complete. 2024-04-01T01:44:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.020+0000 7fb9fb04a740 1 -- start start 2024-04-01T01:44:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.020+0000 7fb9fb04a740 1 Processor -- start 2024-04-01T01:44:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.021+0000 7fb9fb04a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336c490 0x5577f3376740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.021+0000 7fb9fb04a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336ce80 0x5577f3376c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.021+0000 7fb9fb04a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 0x5577f337e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.021+0000 7fb9fb04a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5577f3374ff0 con 0x5577f336ce80 2024-04-01T01:44:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.021+0000 7fb9fb04a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5577f326f470 con 0x5577f336c490 2024-04-01T01:44:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.021+0000 7fb9fb04a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5577f3375170 con 0x5577f337bcb0 2024-04-01T01:44:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.021+0000 7fb9fb04a740 10 client.0 _ll_register_callbacks cb 0x5577f328ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.021+0000 7fb9f6a08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 0x5577f337e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.021+0000 7fb9f6a08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 0x5577f337e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39260/0 (socket says 172.21.15.70:39260) 2024-04-01T01:44:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.021+0000 7fb9f6a08640 1 -- 172.21.15.70:0/281167651 learned_addr learned my addr 172.21.15.70:0/281167651 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.022+0000 7fb9f6207640 1 --2- 172.21.15.70:0/281167651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336c490 0x5577f3376740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.022+0000 7fb9f5a06640 1 --2- 172.21.15.70:0/281167651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336ce80 0x5577f3376c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.022+0000 7fb9f6a08640 1 -- 172.21.15.70:0/281167651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336c490 msgr2=0x5577f3376740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.022+0000 7fb9f6a08640 1 --2- 172.21.15.70:0/281167651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336c490 0x5577f3376740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.022+0000 7fb9f6a08640 1 -- 172.21.15.70:0/281167651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336ce80 msgr2=0x5577f3376c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.022+0000 7fb9f6a08640 1 --2- 172.21.15.70:0/281167651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336ce80 0x5577f3376c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.022+0000 7fb9f6a08640 1 -- 172.21.15.70:0/281167651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577f2fe53f0 con 0x5577f337bcb0 2024-04-01T01:44:59.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.023+0000 7fb9f6a08640 1 --2- 172.21.15.70:0/281167651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 0x5577f337e0a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec01c040 tx=0x7fb9ec009ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb7eac3da4c4a380 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.023+0000 7fb9d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:44:59.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.024+0000 7fb9d9ffb640 1 -- 172.21.15.70:0/281167651 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec028070 con 0x5577f337bcb0 2024-04-01T01:44:59.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.024+0000 7fb9d9ffb640 1 -- 172.21.15.70:0/281167651 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb9ec012860 con 0x5577f337bcb0 2024-04-01T01:44:59.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.024+0000 7fb9fb04a740 1 -- 172.21.15.70:0/281167651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9ec012aa0 con 0x5577f337bcb0 2024-04-01T01:44:59.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.024+0000 7fb9d9ffb640 1 -- 172.21.15.70:0/281167651 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb9ec032050 con 0x5577f337bcb0 2024-04-01T01:44:59.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.024+0000 7fb9fb04a740 10 client.28642 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:59.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.024+0000 7fb9fb04a740 1 -- 172.21.15.70:0/281167651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5577f30af7d0 con 0x5577f337bcb0 2024-04-01T01:44:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.025+0000 7fb9d9ffb640 1 -- 172.21.15.70:0/281167651 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9ec012aa0 con 0x5577f337bcb0 2024-04-01T01:44:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.025+0000 7fb9d9ffb640 1 -- 172.21.15.70:0/281167651 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb9ec00fc30 con 0x5577f337bcb0 2024-04-01T01:44:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.026+0000 7fb9fb04a740 10 client.28642 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.026+0000 7fb9fb04a740 10 client.28642 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:44:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.026+0000 7fb9fb04a740 20 client.28642 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.026+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/281167651 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5577f3393320 0x5577f3395710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.026+0000 7fb9fb04a740 4 client.28642 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.026+0000 7fb9fb04a740 1 -- 172.21.15.70:0/281167651 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577f32de6f0 con 0x5577f3393320 2024-04-01T01:44:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.026+0000 7fb9f6207640 1 --2- 172.21.15.70:0/281167651 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5577f3393320 0x5577f3395710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.027+0000 7fb9f6207640 1 --2- 172.21.15.70:0/281167651 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5577f3393320 0x5577f3395710 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9899735da266db3f server_cookie=f65573f2ef0cac0e in_seq=0 out_seq=0 2024-04-01T01:44:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.027+0000 7fb9d9ffb640 10 client.28642 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:44:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.027+0000 7fb9d9ffb640 1 -- 172.21.15.70:0/281167651 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5577f32de6f0 con 0x5577f3393320 2024-04-01T01:44:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.027+0000 7fb9d9ffb640 10 client.28642 handle_command_reply: tid=0 2024-04-01T01:44:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.027+0000 7fb9fb04a740 1 client.28642 shutdown 2024-04-01T01:44:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.028+0000 7fb9fb04a740 1 -- 172.21.15.70:0/281167651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 msgr2=0x5577f337e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.028+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/281167651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 0x5577f337e0a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec01c040 tx=0x7fb9ec009ec0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.028+0000 7fb9fb04a740 1 -- 172.21.15.70:0/281167651 shutdown_connections 2024-04-01T01:44:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.028+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/281167651 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5577f3393320 0x5577f3395710 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.028+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/281167651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 0x5577f337e0a0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.028+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/281167651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336ce80 0x5577f3376c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.028+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/281167651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336c490 0x5577f3376740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.028+0000 7fb9d9ffb640 0 client.28642 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:44:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.028+0000 7fb9fb04a740 1 -- 172.21.15.70:0/281167651 >> 172.21.15.70:0/281167651 conn(0x5577f3367a00 msgr2=0x5577f3374210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.028+0000 7fb9fb04a740 1 -- 172.21.15.70:0/281167651 shutdown_connections 2024-04-01T01:44:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.028+0000 7fb9fb04a740 1 -- 172.21.15.70:0/281167651 wait complete. 2024-04-01T01:44:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.030+0000 7fb9fb04a740 20 client.28642 trim_cache size 0 max 16384 2024-04-01T01:44:59.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.056+0000 7fb9fb04a740 1 Processor -- start 2024-04-01T01:44:59.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.056+0000 7fb9fb04a740 1 -- start start 2024-04-01T01:44:59.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9fb04a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336c490 0x5577f3367de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9fb04a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336ce80 0x5577f336ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9fb04a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 0x5577f336b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9fb04a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f01b0a60 con 0x5577f336c490 2024-04-01T01:44:59.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9fb04a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f01b25a0 con 0x5577f336ce80 2024-04-01T01:44:59.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9fb04a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f01b1130 con 0x5577f337bcb0 2024-04-01T01:44:59.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9f5a06640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336ce80 0x5577f336ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9f5a06640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336ce80 0x5577f336ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42414/0 (socket says 172.21.15.70:42414) 2024-04-01T01:44:59.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9f5a06640 1 -- 172.21.15.70:0/2143189757 learned_addr learned my addr 172.21.15.70:0/2143189757 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9f6a08640 1 --2- 172.21.15.70:0/2143189757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 0x5577f336b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9f6207640 1 --2- 172.21.15.70:0/2143189757 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336c490 0x5577f3367de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9f5a06640 1 -- 172.21.15.70:0/2143189757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 msgr2=0x5577f336b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9f5a06640 1 --2- 172.21.15.70:0/2143189757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 0x5577f336b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9f5a06640 1 -- 172.21.15.70:0/2143189757 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336c490 msgr2=0x5577f3367de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9f5a06640 1 --2- 172.21.15.70:0/2143189757 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336c490 0x5577f3367de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9f5a06640 1 -- 172.21.15.70:0/2143189757 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577f336db50 con 0x5577f336ce80 2024-04-01T01:44:59.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.057+0000 7fb9f5a06640 1 --2- 172.21.15.70:0/2143189757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336ce80 0x5577f336ac60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb9e00975a0 tx=0x7fb9e0096fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38c5fd4639d7076a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.058+0000 7fb9dbfff640 1 -- 172.21.15.70:0/2143189757 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e009abd0 con 0x5577f336ce80 2024-04-01T01:44:59.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.058+0000 7fb9dbfff640 1 -- 172.21.15.70:0/2143189757 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9e009ed20 con 0x5577f336ce80 2024-04-01T01:44:59.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.058+0000 7fb9dbfff640 1 -- 172.21.15.70:0/2143189757 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e0097240 con 0x5577f336ce80 2024-04-01T01:44:59.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.059+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2143189757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336ce80 msgr2=0x5577f336ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.059+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/2143189757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336ce80 0x5577f336ac60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb9e00975a0 tx=0x7fb9e0096fa0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.059+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2143189757 shutdown_connections 2024-04-01T01:44:59.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.059+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/2143189757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 0x5577f336b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.059+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/2143189757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336ce80 0x5577f336ac60 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.059+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/2143189757 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336c490 0x5577f3367de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.059+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2143189757 >> 172.21.15.70:0/2143189757 conn(0x5577f3367a00 msgr2=0x5577f3362bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.060+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2143189757 shutdown_connections 2024-04-01T01:44:59.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.060+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2143189757 wait complete. 2024-04-01T01:44:59.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.060+0000 7fb9fb04a740 1 -- start start 2024-04-01T01:44:59.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.061+0000 7fb9fb04a740 1 Processor -- start 2024-04-01T01:44:59.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.061+0000 7fb9fb04a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336c490 0x5577f33a4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.061+0000 7fb9fb04a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336ce80 0x5577f33a4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.061+0000 7fb9fb04a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 0x5577f33a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.061+0000 7fb9fb04a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f01b1ed0 con 0x5577f336c490 2024-04-01T01:44:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.061+0000 7fb9fb04a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f01b1800 con 0x5577f336ce80 2024-04-01T01:44:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.061+0000 7fb9fb04a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f01181f0 con 0x5577f337bcb0 2024-04-01T01:44:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.061+0000 7fb9fb04a740 10 client.0 _ll_register_callbacks cb 0x7fb9ec00f4e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.061+0000 7fb9f5a06640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336ce80 0x5577f33a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.061+0000 7fb9f5a06640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336ce80 0x5577f33a4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42418/0 (socket says 172.21.15.70:42418) 2024-04-01T01:44:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.061+0000 7fb9f5a06640 1 -- 172.21.15.70:0/2758669217 learned_addr learned my addr 172.21.15.70:0/2758669217 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.062+0000 7fb9f6a08640 1 --2- 172.21.15.70:0/2758669217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 0x5577f33a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.062+0000 7fb9f6a08640 1 -- 172.21.15.70:0/2758669217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336ce80 msgr2=0x5577f33a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.062+0000 7fb9f6a08640 1 --2- 172.21.15.70:0/2758669217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336ce80 0x5577f33a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.062+0000 7fb9f6a08640 1 -- 172.21.15.70:0/2758669217 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336c490 msgr2=0x5577f33a4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.062+0000 7fb9f6a08640 1 --2- 172.21.15.70:0/2758669217 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336c490 0x5577f33a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.062+0000 7fb9f6a08640 1 -- 172.21.15.70:0/2758669217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577f33a5f90 con 0x5577f337bcb0 2024-04-01T01:44:59.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.062+0000 7fb9f5a06640 1 --2- 172.21.15.70:0/2758669217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336ce80 0x5577f33a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:59.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.062+0000 7fb9f6a08640 1 --2- 172.21.15.70:0/2758669217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 0x5577f33a59a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec001eb0 tx=0x7fb9ec005d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6822844043a8ad8f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.062+0000 7fb9d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:44:59.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.062+0000 7fb9d9ffb640 1 -- 172.21.15.70:0/2758669217 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec03b070 con 0x5577f337bcb0 2024-04-01T01:44:59.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.062+0000 7fb9d9ffb640 1 -- 172.21.15.70:0/2758669217 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb9ec004040 con 0x5577f337bcb0 2024-04-01T01:44:59.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.062+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2758669217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9e009ed20 con 0x5577f337bcb0 2024-04-01T01:44:59.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.063+0000 7fb9d9ffb640 1 -- 172.21.15.70:0/2758669217 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb9ec028040 con 0x5577f337bcb0 2024-04-01T01:44:59.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.063+0000 7fb9fb04a740 10 client.28648 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:59.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.063+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2758669217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5577f33f9590 con 0x5577f337bcb0 2024-04-01T01:44:59.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.063+0000 7fb9d9ffb640 1 -- 172.21.15.70:0/2758669217 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9e009ed20 con 0x5577f337bcb0 2024-04-01T01:44:59.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.063+0000 7fb9d9ffb640 1 -- 172.21.15.70:0/2758669217 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb9ec03e020 con 0x5577f337bcb0 2024-04-01T01:44:59.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.064+0000 7fb9fb04a740 10 client.28648 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:59.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.064+0000 7fb9fb04a740 10 client.28648 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:44:59.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.064+0000 7fb9fb04a740 20 client.28648 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:59.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.064+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/2758669217 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5577f33f9f80 0x5577f33fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.064+0000 7fb9fb04a740 4 client.28648 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:44:59.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.064+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2758669217 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5577f32d8a40 con 0x5577f33f9f80 2024-04-01T01:44:59.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.065+0000 7fb9f6207640 1 --2- 172.21.15.70:0/2758669217 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5577f33f9f80 0x5577f33fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.065+0000 7fb9f6207640 1 --2- 172.21.15.70:0/2758669217 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5577f33f9f80 0x5577f33fc3a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3198dfa210a14100 server_cookie=891f556c2bc74d6a in_seq=0 out_seq=0 2024-04-01T01:44:59.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.065+0000 7fb9d9ffb640 10 client.28648 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:44:59.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.065+0000 7fb9d9ffb640 1 -- 172.21.15.70:0/2758669217 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5577f32d8a40 con 0x5577f33f9f80 2024-04-01T01:44:59.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.066+0000 7fb9d9ffb640 10 client.28648 handle_command_reply: tid=0 2024-04-01T01:44:59.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.066+0000 7fb9fb04a740 1 client.28648 shutdown 2024-04-01T01:44:59.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.066+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2758669217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 msgr2=0x5577f33a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.066+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/2758669217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 0x5577f33a59a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec001eb0 tx=0x7fb9ec005d20 comp rx=0 tx=0).stop 2024-04-01T01:44:59.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.067+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2758669217 shutdown_connections 2024-04-01T01:44:59.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.067+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/2758669217 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5577f33f9f80 0x5577f33fc3a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.067+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/2758669217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577f337bcb0 0x5577f33a59a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.067+0000 7fb9d9ffb640 0 client.28648 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:44:59.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.067+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/2758669217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5577f336ce80 0x5577f33a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.067+0000 7fb9fb04a740 1 --2- 172.21.15.70:0/2758669217 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577f336c490 0x5577f33a4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.067+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2758669217 >> 172.21.15.70:0/2758669217 conn(0x5577f3367a00 msgr2=0x5577f3367de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.068+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2758669217 shutdown_connections 2024-04-01T01:44:59.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.068+0000 7fb9fb04a740 1 -- 172.21.15.70:0/2758669217 wait complete. 2024-04-01T01:44:59.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.069+0000 7fb9fb04a740 20 client.28648 trim_cache size 0 max 16384 2024-04-01T01:44:59.070 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:44:59.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.070+0000 7fb9f846b640 1 -- 172.21.15.70:0/3564740913 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9dc0a2570 msgr2=0x7fb9dc0a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.070+0000 7fb9f846b640 1 --2- 172.21.15.70:0/3564740913 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9dc0a2570 0x7fb9dc0a4a30 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8006010 tx=0x7fb9e8000f40 comp rx=0 tx=0).stop 2024-04-01T01:44:59.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.070+0000 7fb9f846b640 1 -- 172.21.15.70:0/3564740913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f010b0c0 msgr2=0x7fb9f01a6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.070+0000 7fb9f846b640 1 --2- 172.21.15.70:0/3564740913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f010b0c0 0x7fb9f01a6d10 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb9e000d950 tx=0x7fb9e000de10 comp rx=0 tx=0).stop 2024-04-01T01:44:59.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.070+0000 7fb9f846b640 1 -- 172.21.15.70:0/3564740913 shutdown_connections 2024-04-01T01:44:59.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.070+0000 7fb9f846b640 1 --2- 172.21.15.70:0/3564740913 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9dc0a2570 0x7fb9dc0a4a30 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.070+0000 7fb9f846b640 1 --2- 172.21.15.70:0/3564740913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9f010da70 0x7fb9f01add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.070+0000 7fb9f846b640 1 --2- 172.21.15.70:0/3564740913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f010b0c0 0x7fb9f01a6d10 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.070+0000 7fb9f846b640 1 --2- 172.21.15.70:0/3564740913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f0108750 0x7fb9f01a67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.070+0000 7fb9f846b640 1 -- 172.21.15.70:0/3564740913 >> 172.21.15.70:0/3564740913 conn(0x7fb9f006e5b0 msgr2=0x7fb9f010b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.070+0000 7fb9f846b640 1 -- 172.21.15.70:0/3564740913 shutdown_connections 2024-04-01T01:44:59.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.070+0000 7fb9f846b640 1 -- 172.21.15.70:0/3564740913 wait complete. 2024-04-01T01:44:59.089 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:58 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/4239946896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:44:59.090 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:44:59.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:58 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/4239946896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:44:59.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.287+0000 7f25e1fb6640 1 -- 172.21.15.70:0/3547950285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25dc100650 msgr2=0x7f25dc100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.287+0000 7f25e1fb6640 1 --2- 172.21.15.70:0/3547950285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25dc100650 0x7f25dc100a70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f25d0009900 tx=0x7f25d002bc30 comp rx=0 tx=0).stop 2024-04-01T01:44:59.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.287+0000 7f25e1fb6640 1 -- 172.21.15.70:0/3547950285 shutdown_connections 2024-04-01T01:44:59.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.287+0000 7f25e1fb6640 1 --2- 172.21.15.70:0/3547950285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25dc102200 0x7f25dc108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.287+0000 7f25e1fb6640 1 --2- 172.21.15.70:0/3547950285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25dc101840 0x7f25dc101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.287+0000 7f25e1fb6640 1 --2- 172.21.15.70:0/3547950285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25dc100650 0x7f25dc100a70 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.287+0000 7f25e1fb6640 1 -- 172.21.15.70:0/3547950285 >> 172.21.15.70:0/3547950285 conn(0x7f25dc0fbbe0 msgr2=0x7f25dc0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.288+0000 7f25e1fb6640 1 -- 172.21.15.70:0/3547950285 shutdown_connections 2024-04-01T01:44:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.288+0000 7f25e1fb6640 1 -- 172.21.15.70:0/3547950285 wait complete. 2024-04-01T01:44:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.289+0000 7f25e1fb6640 1 Processor -- start 2024-04-01T01:44:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.289+0000 7f25e1fb6640 1 -- start start 2024-04-01T01:44:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.289+0000 7f25e1fb6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25dc100650 0x7f25dc06f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.289+0000 7f25e1fb6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25dc101840 0x7f25dc06f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.289+0000 7f25e1fb6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25dc102200 0x7f25dc072630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.289+0000 7f25e1fb6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc10c1d0 con 0x7f25dc100650 2024-04-01T01:44:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.289+0000 7f25e1fb6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc10c050 con 0x7f25dc101840 2024-04-01T01:44:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.289+0000 7f25e1fb6640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc10c350 con 0x7f25dc102200 2024-04-01T01:44:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.289+0000 7f25daffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25dc101840 0x7f25dc06f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.289+0000 7f25daffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25dc101840 0x7f25dc06f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42428/0 (socket says 172.21.15.70:42428) 2024-04-01T01:44:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.289+0000 7f25daffd640 1 -- 172.21.15.70:0/3641601910 learned_addr learned my addr 172.21.15.70:0/3641601910 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.290+0000 7f25dbfff640 1 --2- 172.21.15.70:0/3641601910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25dc102200 0x7f25dc072630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.290+0000 7f25db7fe640 1 --2- 172.21.15.70:0/3641601910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25dc100650 0x7f25dc06f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.290+0000 7f25daffd640 1 -- 172.21.15.70:0/3641601910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25dc102200 msgr2=0x7f25dc072630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.290+0000 7f25daffd640 1 --2- 172.21.15.70:0/3641601910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25dc102200 0x7f25dc072630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.290+0000 7f25daffd640 1 -- 172.21.15.70:0/3641601910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25dc100650 msgr2=0x7f25dc06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.290+0000 7f25daffd640 1 --2- 172.21.15.70:0/3641601910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25dc100650 0x7f25dc06f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.290+0000 7f25daffd640 1 -- 172.21.15.70:0/3641601910 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25dc072ca0 con 0x7f25dc101840 2024-04-01T01:44:59.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.290+0000 7f25daffd640 1 --2- 172.21.15.70:0/3641601910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25dc101840 0x7f25dc06f5e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f25cc00e9f0 tx=0x7f25cc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.292+0000 7f25d8ff9640 1 -- 172.21.15.70:0/3641601910 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc00cd80 con 0x7f25dc101840 2024-04-01T01:44:59.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.292+0000 7f25d8ff9640 1 -- 172.21.15.70:0/3641601910 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25cc0054d0 con 0x7f25dc101840 2024-04-01T01:44:59.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.292+0000 7f25d8ff9640 1 -- 172.21.15.70:0/3641601910 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc010490 con 0x7f25dc101840 2024-04-01T01:44:59.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.293+0000 7f25e1fb6640 1 -- 172.21.15.70:0/3641601910 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25dc072f90 con 0x7f25dc101840 2024-04-01T01:44:59.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.293+0000 7f25e1fb6640 1 -- 172.21.15.70:0/3641601910 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25dc073490 con 0x7f25dc101840 2024-04-01T01:44:59.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.296+0000 7f25e4b90740 1 Processor -- start 2024-04-01T01:44:59.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.296+0000 7f25d8ff9640 1 -- 172.21.15.70:0/3641601910 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f25cc005020 con 0x7f25dc101840 2024-04-01T01:44:59.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.296+0000 7f25e4b90740 1 -- start start 2024-04-01T01:44:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.296+0000 7f25e4b90740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867490 0x559d048678b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.296+0000 7f25e4b90740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d04867e80 0x559d0486c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.296+0000 7f25e4b90740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d0486cff0 0x559d0486f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.296+0000 7f25e4b90740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559d047cc1b0 con 0x559d0486cff0 2024-04-01T01:44:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.296+0000 7f25e4b90740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x559d04774c00 con 0x559d04867e80 2024-04-01T01:44:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.296+0000 7f25e4b90740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559d047c8250 con 0x559d04867490 2024-04-01T01:44:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.297+0000 7f25dbfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867490 0x559d048678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.297+0000 7f25db7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d04867e80 0x559d0486c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.297+0000 7f25daffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d0486cff0 0x559d0486f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.297+0000 7f25db7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d04867e80 0x559d0486c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42434/0 (socket says 172.21.15.70:42434) 2024-04-01T01:44:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.297+0000 7f25dbfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867490 0x559d048678b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39302/0 (socket says 172.21.15.70:39302) 2024-04-01T01:44:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.297+0000 7f25db7fe640 1 -- 172.21.15.70:0/3760884281 learned_addr learned my addr 172.21.15.70:0/3760884281 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.297+0000 7f25d8ff9640 1 --2- 172.21.15.70:0/3641601910 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f25c40a2620 0x7f25c40a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.297+0000 7f25d8ff9640 1 -- 172.21.15.70:0/3641601910 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f25cc0cee90 con 0x7f25dc101840 2024-04-01T01:44:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.297+0000 7f25dbfff640 1 -- 172.21.15.70:0/3760884281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d04867e80 msgr2=0x559d0486c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.297+0000 7f25dbfff640 1 --2- 172.21.15.70:0/3760884281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d04867e80 0x559d0486c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.297+0000 7f25dbfff640 1 -- 172.21.15.70:0/3760884281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d0486cff0 msgr2=0x559d0486f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.297+0000 7f25dbfff640 1 --2- 172.21.15.70:0/3760884281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d0486cff0 0x559d0486f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.297+0000 7f25dbfff640 1 -- 172.21.15.70:0/3760884281 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d047943d0 con 0x559d04867490 2024-04-01T01:44:59.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.297+0000 7f25dbfff640 1 --2- 172.21.15.70:0/3641601910 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f25c40a2620 0x7f25c40a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.298+0000 7f25dbfff640 1 --2- 172.21.15.70:0/3760884281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867490 0x559d048678b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f25d4010440 tx=0x7f25d4010900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c52c2e439a7544c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.298+0000 7f25c1ffb640 1 -- 172.21.15.70:0/3760884281 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d4022070 con 0x559d04867490 2024-04-01T01:44:59.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.298+0000 7f25c1ffb640 1 -- 172.21.15.70:0/3760884281 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25d401d430 con 0x559d04867490 2024-04-01T01:44:59.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.298+0000 7f25c1ffb640 1 -- 172.21.15.70:0/3760884281 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d401c5f0 con 0x559d04867490 2024-04-01T01:44:59.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.299+0000 7f25e4b90740 1 -- 172.21.15.70:0/3760884281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867490 msgr2=0x559d048678b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.299+0000 7f25dbfff640 1 --2- 172.21.15.70:0/3641601910 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f25c40a2620 0x7f25c40a4ae0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f25d401f000 tx=0x7f25d401cca0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.299+0000 7f25e4b90740 1 --2- 172.21.15.70:0/3760884281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867490 0x559d048678b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f25d4010440 tx=0x7f25d4010900 comp rx=0 tx=0).stop 2024-04-01T01:44:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.299+0000 7f25e4b90740 1 -- 172.21.15.70:0/3760884281 shutdown_connections 2024-04-01T01:44:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.299+0000 7f25e4b90740 1 --2- 172.21.15.70:0/3760884281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d0486cff0 0x559d0486f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.299+0000 7f25e4b90740 1 --2- 172.21.15.70:0/3760884281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d04867e80 0x559d0486c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.299+0000 7f25e4b90740 1 --2- 172.21.15.70:0/3760884281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867490 0x559d048678b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.299+0000 7f25e4b90740 1 -- 172.21.15.70:0/3760884281 >> 172.21.15.70:0/3760884281 conn(0x559d04862a00 msgr2=0x559d04862e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.299+0000 7f25e4b90740 1 -- 172.21.15.70:0/3760884281 shutdown_connections 2024-04-01T01:44:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.299+0000 7f25e4b90740 1 -- 172.21.15.70:0/3760884281 wait complete. 2024-04-01T01:44:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.299+0000 7f25e4b90740 1 -- start start 2024-04-01T01:44:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.300+0000 7f25e4b90740 1 Processor -- start 2024-04-01T01:44:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.300+0000 7f25e4b90740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d04867490 0x559d04871a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.300+0000 7f25e4b90740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d04867e80 0x559d04871fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.300+0000 7f25e4b90740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d0486cff0 0x559d04878fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.300+0000 7f25e4b90740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559d0486fff0 con 0x559d04867e80 2024-04-01T01:44:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.300+0000 7f25e4b90740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x559d0476a470 con 0x559d04867490 2024-04-01T01:44:59.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.300+0000 7f25e4b90740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559d04870170 con 0x559d0486cff0 2024-04-01T01:44:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.300+0000 7f25e4b90740 10 client.0 _ll_register_callbacks cb 0x559d04787cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.300+0000 7f25dbfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d0486cff0 0x559d04878fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.300+0000 7f25daffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d04867e80 0x559d04871fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.300+0000 7f25dbfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d0486cff0 0x559d04878fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39314/0 (socket says 172.21.15.70:39314) 2024-04-01T01:44:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.300+0000 7f25dbfff640 1 -- 172.21.15.70:0/2905726711 learned_addr learned my addr 172.21.15.70:0/2905726711 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.300+0000 7f25db7fe640 1 --2- 172.21.15.70:0/2905726711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d04867490 0x559d04871a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.301+0000 7f25dbfff640 1 -- 172.21.15.70:0/2905726711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d04867490 msgr2=0x559d04871a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.301+0000 7f25dbfff640 1 --2- 172.21.15.70:0/2905726711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d04867490 0x559d04871a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.301+0000 7f25dbfff640 1 -- 172.21.15.70:0/2905726711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d04867e80 msgr2=0x559d04871fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.301+0000 7f25dbfff640 1 --2- 172.21.15.70:0/2905726711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d04867e80 0x559d04871fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.301+0000 7f25dbfff640 1 -- 172.21.15.70:0/2905726711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d044e03f0 con 0x559d0486cff0 2024-04-01T01:44:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.301+0000 7f25dbfff640 1 --2- 172.21.15.70:0/2905726711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d0486cff0 0x559d04878fe0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009b00 tx=0x7f25d4018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=372b77f630f48ca5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.301+0000 7f25baffd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:44:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.301+0000 7f25baffd640 1 -- 172.21.15.70:0/2905726711 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d4033060 con 0x559d0486cff0 2024-04-01T01:44:59.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.301+0000 7f25baffd640 1 -- 172.21.15.70:0/2905726711 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f25d4004a40 con 0x559d0486cff0 2024-04-01T01:44:59.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.301+0000 7f25e4b90740 1 -- 172.21.15.70:0/2905726711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25d401d430 con 0x559d0486cff0 2024-04-01T01:44:59.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.302+0000 7f25baffd640 1 -- 172.21.15.70:0/2905726711 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f25d403a050 con 0x559d0486cff0 2024-04-01T01:44:59.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.302+0000 7f25e4b90740 10 client.28660 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:59.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.302+0000 7f25e4b90740 1 -- 172.21.15.70:0/2905726711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559d045aa7d0 con 0x559d0486cff0 2024-04-01T01:44:59.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.302+0000 7f25baffd640 1 -- 172.21.15.70:0/2905726711 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25d401d430 con 0x559d0486cff0 2024-04-01T01:44:59.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.302+0000 7f25baffd640 1 -- 172.21.15.70:0/2905726711 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f25d400f040 con 0x559d0486cff0 2024-04-01T01:44:59.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.302+0000 7f25e4b90740 10 client.28660 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:59.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.302+0000 7f25e4b90740 10 client.28660 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:44:59.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.302+0000 7f25e4b90740 20 client.28660 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:59.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.302+0000 7f25e4b90740 1 --2- 172.21.15.70:0/2905726711 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x559d0487d8c0 0x559d0487fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.303+0000 7f25e4b90740 4 client.28660 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:59.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.303+0000 7f25e4b90740 1 -- 172.21.15.70:0/2905726711 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d047d96f0 con 0x559d0487d8c0 2024-04-01T01:44:59.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.303+0000 7f25db7fe640 1 --2- 172.21.15.70:0/2905726711 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x559d0487d8c0 0x559d0487fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.303+0000 7f25baffd640 10 client.28660 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:44:59.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.304+0000 7f25db7fe640 1 --2- 172.21.15.70:0/2905726711 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x559d0487d8c0 0x559d0487fd50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=376691e5804f58f7 server_cookie=40431053e4bebb26 in_seq=0 out_seq=0 2024-04-01T01:44:59.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.307+0000 7f25baffd640 1 -- 172.21.15.70:0/2905726711 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559d047d96f0 con 0x559d0487d8c0 2024-04-01T01:44:59.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.307+0000 7f25baffd640 10 client.28660 handle_command_reply: tid=0 2024-04-01T01:44:59.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.307+0000 7f25e4b90740 1 client.28660 shutdown 2024-04-01T01:44:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.307+0000 7f25e4b90740 1 -- 172.21.15.70:0/2905726711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d0486cff0 msgr2=0x559d04878fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.307+0000 7f25e4b90740 1 --2- 172.21.15.70:0/2905726711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d0486cff0 0x559d04878fe0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009b00 tx=0x7f25d4018000 comp rx=0 tx=0).stop 2024-04-01T01:44:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.308+0000 7f25e4b90740 1 -- 172.21.15.70:0/2905726711 shutdown_connections 2024-04-01T01:44:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.308+0000 7f25e4b90740 1 --2- 172.21.15.70:0/2905726711 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x559d0487d8c0 0x559d0487fd50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.308+0000 7f25e4b90740 1 --2- 172.21.15.70:0/2905726711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d0486cff0 0x559d04878fe0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.308+0000 7f25e4b90740 1 --2- 172.21.15.70:0/2905726711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d04867e80 0x559d04871fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.308+0000 7f25e4b90740 1 --2- 172.21.15.70:0/2905726711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d04867490 0x559d04871a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.308+0000 7f25e4b90740 1 -- 172.21.15.70:0/2905726711 >> 172.21.15.70:0/2905726711 conn(0x559d04862a00 msgr2=0x559d04861d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.308+0000 7f25baffd640 0 client.28660 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:44:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.308+0000 7f25e4b90740 1 -- 172.21.15.70:0/2905726711 shutdown_connections 2024-04-01T01:44:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.308+0000 7f25e4b90740 1 -- 172.21.15.70:0/2905726711 wait complete. 2024-04-01T01:44:59.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.309+0000 7f25e4b90740 20 client.28660 trim_cache size 0 max 16384 2024-04-01T01:44:59.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:58 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/4239946896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:44:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.336+0000 7f25e4b90740 1 Processor -- start 2024-04-01T01:44:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.336+0000 7f25e4b90740 1 -- start start 2024-04-01T01:44:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.336+0000 7f25e4b90740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d04867490 0x559d048693c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.336+0000 7f25e4b90740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867e80 0x559d04869900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.337+0000 7f25e4b90740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d0486cff0 0x559d04878fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.337+0000 7f25e4b90740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc1b9280 con 0x559d04867490 2024-04-01T01:44:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.337+0000 7f25e4b90740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc1badc0 con 0x559d0486cff0 2024-04-01T01:44:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.337+0000 7f25e4b90740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc1b9950 con 0x559d04867e80 2024-04-01T01:44:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.337+0000 7f25daffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867e80 0x559d04869900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.337+0000 7f25daffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867e80 0x559d04869900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39316/0 (socket says 172.21.15.70:39316) 2024-04-01T01:44:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.337+0000 7f25daffd640 1 -- 172.21.15.70:0/2501085212 learned_addr learned my addr 172.21.15.70:0/2501085212 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.337+0000 7f25dbfff640 1 --2- 172.21.15.70:0/2501085212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d0486cff0 0x559d04878fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.337+0000 7f25dbfff640 1 -- 172.21.15.70:0/2501085212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867e80 msgr2=0x559d04869900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.337+0000 7f25dbfff640 1 --2- 172.21.15.70:0/2501085212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867e80 0x559d04869900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.337+0000 7f25dbfff640 1 -- 172.21.15.70:0/2501085212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d04867490 msgr2=0x559d048693c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:44:59.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.337+0000 7f25dbfff640 1 --2- 172.21.15.70:0/2501085212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d04867490 0x559d048693c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.337+0000 7f25dbfff640 1 -- 172.21.15.70:0/2501085212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d04868260 con 0x559d0486cff0 2024-04-01T01:44:59.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.338+0000 7f25dbfff640 1 --2- 172.21.15.70:0/2501085212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d0486cff0 0x559d04878fa0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f25d4010440 tx=0x7f25d4005c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbfff2e9714d15bd server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.338+0000 7f25c17fa640 1 -- 172.21.15.70:0/2501085212 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d4033070 con 0x559d0486cff0 2024-04-01T01:44:59.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.338+0000 7f25c17fa640 1 -- 172.21.15.70:0/2501085212 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25d402a040 con 0x559d0486cff0 2024-04-01T01:44:59.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.339+0000 7f25c17fa640 1 -- 172.21.15.70:0/2501085212 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d4022070 con 0x559d0486cff0 2024-04-01T01:44:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.339+0000 7f25e4b90740 1 -- 172.21.15.70:0/2501085212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d0486cff0 msgr2=0x559d04878fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.339+0000 7f25e4b90740 1 --2- 172.21.15.70:0/2501085212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d0486cff0 0x559d04878fa0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f25d4010440 tx=0x7f25d4005c70 comp rx=0 tx=0).stop 2024-04-01T01:44:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.339+0000 7f25e4b90740 1 -- 172.21.15.70:0/2501085212 shutdown_connections 2024-04-01T01:44:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.339+0000 7f25e4b90740 1 --2- 172.21.15.70:0/2501085212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d0486cff0 0x559d04878fa0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.339+0000 7f25e4b90740 1 --2- 172.21.15.70:0/2501085212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867e80 0x559d04869900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.339+0000 7f25e4b90740 1 --2- 172.21.15.70:0/2501085212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d04867490 0x559d048693c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.339+0000 7f25e4b90740 1 -- 172.21.15.70:0/2501085212 >> 172.21.15.70:0/2501085212 conn(0x559d04862a00 msgr2=0x559d0485dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.339+0000 7f25e4b90740 1 -- 172.21.15.70:0/2501085212 shutdown_connections 2024-04-01T01:44:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.339+0000 7f25e4b90740 1 -- 172.21.15.70:0/2501085212 wait complete. 2024-04-01T01:44:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.340+0000 7f25e4b90740 1 -- start start 2024-04-01T01:44:59.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.340+0000 7f25e4b90740 1 Processor -- start 2024-04-01T01:44:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.340+0000 7f25e4b90740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867490 0x559d048760b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.340+0000 7f25e4b90740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d04867e80 0x559d048765f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.340+0000 7f25e4b90740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d0486cff0 0x559d04877600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.340+0000 7f25e4b90740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc1ba6f0 con 0x559d04867e80 2024-04-01T01:44:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.340+0000 7f25e4b90740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc1ba020 con 0x559d0486cff0 2024-04-01T01:44:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.340+0000 7f25e4b90740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc1b7260 con 0x559d04867490 2024-04-01T01:44:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.340+0000 7f25e4b90740 10 client.0 _ll_register_callbacks cb 0x7f25cc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.340+0000 7f25db7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867490 0x559d048760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.340+0000 7f25db7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867490 0x559d048760b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39318/0 (socket says 172.21.15.70:39318) 2024-04-01T01:44:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.340+0000 7f25db7fe640 1 -- 172.21.15.70:0/1292779171 learned_addr learned my addr 172.21.15.70:0/1292779171 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.341+0000 7f25dbfff640 1 --2- 172.21.15.70:0/1292779171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d0486cff0 0x559d04877600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.341+0000 7f25daffd640 1 --2- 172.21.15.70:0/1292779171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d04867e80 0x559d048765f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.341+0000 7f25db7fe640 1 -- 172.21.15.70:0/1292779171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d0486cff0 msgr2=0x559d04877600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.341+0000 7f25db7fe640 1 --2- 172.21.15.70:0/1292779171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d0486cff0 0x559d04877600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.341+0000 7f25db7fe640 1 -- 172.21.15.70:0/1292779171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d04867e80 msgr2=0x559d048765f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.341+0000 7f25db7fe640 1 --2- 172.21.15.70:0/1292779171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d04867e80 0x559d048765f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.341+0000 7f25db7fe640 1 -- 172.21.15.70:0/1292779171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d04877bf0 con 0x559d04867490 2024-04-01T01:44:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.341+0000 7f25dbfff640 1 --2- 172.21.15.70:0/1292779171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d0486cff0 0x559d04877600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.341+0000 7f25daffd640 1 --2- 172.21.15.70:0/1292779171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d04867e80 0x559d048765f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:44:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.341+0000 7f25db7fe640 1 --2- 172.21.15.70:0/1292779171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867490 0x559d048760b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f25d0036040 tx=0x7f25d002cba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eab269072354b3e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.341+0000 7f25baffd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:44:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.342+0000 7f25baffd640 1 -- 172.21.15.70:0/1292779171 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d004f060 con 0x559d04867490 2024-04-01T01:44:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.342+0000 7f25e4b90740 1 -- 172.21.15.70:0/1292779171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25d402a040 con 0x559d04867490 2024-04-01T01:44:59.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.342+0000 7f25baffd640 1 -- 172.21.15.70:0/1292779171 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f25d0004250 con 0x559d04867490 2024-04-01T01:44:59.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.342+0000 7f25baffd640 1 -- 172.21.15.70:0/1292779171 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f25d0056050 con 0x559d04867490 2024-04-01T01:44:59.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.344+0000 7f25e4b90740 10 client.28666 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:59.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.344+0000 7f25e4b90740 1 -- 172.21.15.70:0/1292779171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559d04878f60 con 0x559d04867490 2024-04-01T01:44:59.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.344+0000 7f25baffd640 1 -- 172.21.15.70:0/1292779171 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25d402a040 con 0x559d04867490 2024-04-01T01:44:59.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.345+0000 7f25baffd640 1 -- 172.21.15.70:0/1292779171 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f25d00046c0 con 0x559d04867490 2024-04-01T01:44:59.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.345+0000 7f25e4b90740 10 client.28666 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:59.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.345+0000 7f25e4b90740 10 client.28666 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:44:59.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.345+0000 7f25e4b90740 20 client.28666 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:59.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.345+0000 7f25e4b90740 1 --2- 172.21.15.70:0/1292779171 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x559d048e40f0 0x559d048e6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.345+0000 7f25e4b90740 4 client.28666 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:44:59.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.345+0000 7f25e4b90740 1 -- 172.21.15.70:0/1292779171 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559d047d3a40 con 0x559d048e40f0 2024-04-01T01:44:59.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.346+0000 7f25daffd640 1 --2- 172.21.15.70:0/1292779171 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x559d048e40f0 0x559d048e6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.346+0000 7f25daffd640 1 --2- 172.21.15.70:0/1292779171 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x559d048e40f0 0x559d048e6510 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ef469ca6cc9684d9 server_cookie=8156a4393916f3b0 in_seq=0 out_seq=0 2024-04-01T01:44:59.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.347+0000 7f25baffd640 10 client.28666 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:44:59.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.347+0000 7f25baffd640 1 -- 172.21.15.70:0/1292779171 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559d047d3a40 con 0x559d048e40f0 2024-04-01T01:44:59.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.347+0000 7f25baffd640 10 client.28666 handle_command_reply: tid=0 2024-04-01T01:44:59.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.347+0000 7f25e4b90740 1 client.28666 shutdown 2024-04-01T01:44:59.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.347+0000 7f25e4b90740 1 -- 172.21.15.70:0/1292779171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867490 msgr2=0x559d048760b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.347+0000 7f25e4b90740 1 --2- 172.21.15.70:0/1292779171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867490 0x559d048760b0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f25d0036040 tx=0x7f25d002cba0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.347+0000 7f25e4b90740 1 -- 172.21.15.70:0/1292779171 shutdown_connections 2024-04-01T01:44:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.347+0000 7f25e4b90740 1 --2- 172.21.15.70:0/1292779171 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x559d048e40f0 0x559d048e6510 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.347+0000 7f25e4b90740 1 --2- 172.21.15.70:0/1292779171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559d0486cff0 0x559d04877600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.347+0000 7f25e4b90740 1 --2- 172.21.15.70:0/1292779171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559d04867e80 0x559d048765f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.347+0000 7f25e4b90740 1 --2- 172.21.15.70:0/1292779171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d04867490 0x559d048760b0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.347+0000 7f25e4b90740 1 -- 172.21.15.70:0/1292779171 >> 172.21.15.70:0/1292779171 conn(0x559d04862a00 msgr2=0x559d048693c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.347+0000 7f25baffd640 0 client.28666 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:44:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.348+0000 7f25e4b90740 1 -- 172.21.15.70:0/1292779171 shutdown_connections 2024-04-01T01:44:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.348+0000 7f25e4b90740 1 -- 172.21.15.70:0/1292779171 wait complete. 2024-04-01T01:44:59.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.348+0000 7f25e4b90740 20 client.28666 trim_cache size 0 max 16384 2024-04-01T01:44:59.349 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:44:59.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.349+0000 7f25e1fb6640 1 -- 172.21.15.70:0/3641601910 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f25c40a2620 msgr2=0x7f25c40a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.349+0000 7f25e1fb6640 1 --2- 172.21.15.70:0/3641601910 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f25c40a2620 0x7f25c40a4ae0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f25d401f000 tx=0x7f25d401cca0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.349+0000 7f25e1fb6640 1 -- 172.21.15.70:0/3641601910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25dc101840 msgr2=0x7f25dc06f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.349+0000 7f25e1fb6640 1 --2- 172.21.15.70:0/3641601910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25dc101840 0x7f25dc06f5e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f25cc00e9f0 tx=0x7f25cc00edf0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.350+0000 7f25e1fb6640 1 -- 172.21.15.70:0/3641601910 shutdown_connections 2024-04-01T01:44:59.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.350+0000 7f25e1fb6640 1 --2- 172.21.15.70:0/3641601910 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f25c40a2620 0x7f25c40a4ae0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.350+0000 7f25e1fb6640 1 --2- 172.21.15.70:0/3641601910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25dc102200 0x7f25dc072630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.350+0000 7f25e1fb6640 1 --2- 172.21.15.70:0/3641601910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25dc101840 0x7f25dc06f5e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.350+0000 7f25e1fb6640 1 --2- 172.21.15.70:0/3641601910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25dc100650 0x7f25dc06f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.350+0000 7f25e1fb6640 1 -- 172.21.15.70:0/3641601910 >> 172.21.15.70:0/3641601910 conn(0x7f25dc0fbbe0 msgr2=0x7f25dc0688e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.351+0000 7f25e1fb6640 1 -- 172.21.15.70:0/3641601910 shutdown_connections 2024-04-01T01:44:59.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.351+0000 7f25e1fb6640 1 -- 172.21.15.70:0/3641601910 wait complete. 2024-04-01T01:44:59.369 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:44:59.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.543+0000 7fb2af68a640 1 -- 172.21.15.70:0/191696545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2a8103460 msgr2=0x7fb2a8109cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.543+0000 7fb2af68a640 1 --2- 172.21.15.70:0/191696545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2a8103460 0x7fb2a8109cf0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0009cd0 tx=0x7fb2a002c0f0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.543+0000 7fb2af68a640 1 -- 172.21.15.70:0/191696545 shutdown_connections 2024-04-01T01:44:59.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.543+0000 7fb2af68a640 1 --2- 172.21.15.70:0/191696545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2a8103460 0x7fb2a8109cf0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.543+0000 7fb2af68a640 1 --2- 172.21.15.70:0/191696545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2a8102aa0 0x7fb2a8102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.543+0000 7fb2af68a640 1 --2- 172.21.15.70:0/191696545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a80ffa30 0x7fb2a80ffe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.543+0000 7fb2af68a640 1 -- 172.21.15.70:0/191696545 >> 172.21.15.70:0/191696545 conn(0x7fb2a80fb340 msgr2=0x7fb2a80fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.543+0000 7fb2af68a640 1 -- 172.21.15.70:0/191696545 shutdown_connections 2024-04-01T01:44:59.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.543+0000 7fb2af68a640 1 -- 172.21.15.70:0/191696545 wait complete. 2024-04-01T01:44:59.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.544+0000 7fb2af68a640 1 Processor -- start 2024-04-01T01:44:59.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.544+0000 7fb2af68a640 1 -- start start 2024-04-01T01:44:59.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.544+0000 7fb2af68a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a80ffa30 0x7fb2a81979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.544+0000 7fb2af68a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2a8102aa0 0x7fb2a8197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.544+0000 7fb2af68a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2a8103460 0x7fb2a819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.544+0000 7fb2af68a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a810d290 con 0x7fb2a8103460 2024-04-01T01:44:59.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.544+0000 7fb2af68a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a810d110 con 0x7fb2a80ffa30 2024-04-01T01:44:59.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.544+0000 7fb2af68a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a810d410 con 0x7fb2a8102aa0 2024-04-01T01:44:59.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.544+0000 7fb2adc27640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2a8103460 0x7fb2a819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.544+0000 7fb2ad426640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a80ffa30 0x7fb2a81979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.544+0000 7fb2ad426640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a80ffa30 0x7fb2a81979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42488/0 (socket says 172.21.15.70:42488) 2024-04-01T01:44:59.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.544+0000 7fb2ad426640 1 -- 172.21.15.70:0/198312535 learned_addr learned my addr 172.21.15.70:0/198312535 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.544+0000 7fb2acc25640 1 --2- 172.21.15.70:0/198312535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2a8102aa0 0x7fb2a8197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.544+0000 7fb2ad426640 1 -- 172.21.15.70:0/198312535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2a8102aa0 msgr2=0x7fb2a8197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.544+0000 7fb2ad426640 1 --2- 172.21.15.70:0/198312535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2a8102aa0 0x7fb2a8197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.544+0000 7fb2ad426640 1 -- 172.21.15.70:0/198312535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2a8103460 msgr2=0x7fb2a819ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.545+0000 7fb2ad426640 1 --2- 172.21.15.70:0/198312535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2a8103460 0x7fb2a819ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.545+0000 7fb2ad426640 1 -- 172.21.15.70:0/198312535 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a819f5e0 con 0x7fb2a80ffa30 2024-04-01T01:44:59.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.545+0000 7fb2ad426640 1 --2- 172.21.15.70:0/198312535 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a80ffa30 0x7fb2a81979c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb29800e9c0 tx=0x7fb29800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.546+0000 7fb29e7fc640 1 -- 172.21.15.70:0/198312535 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29800cd80 con 0x7fb2a80ffa30 2024-04-01T01:44:59.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.546+0000 7fb29e7fc640 1 -- 172.21.15.70:0/198312535 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2980035f0 con 0x7fb2a80ffa30 2024-04-01T01:44:59.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.546+0000 7fb29e7fc640 1 -- 172.21.15.70:0/198312535 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb298010690 con 0x7fb2a80ffa30 2024-04-01T01:44:59.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.546+0000 7fb2af68a640 1 -- 172.21.15.70:0/198312535 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2a819f8d0 con 0x7fb2a80ffa30 2024-04-01T01:44:59.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.546+0000 7fb2af68a640 1 -- 172.21.15.70:0/198312535 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2a819fdd0 con 0x7fb2a80ffa30 2024-04-01T01:44:59.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.549+0000 7fb29e7fc640 1 -- 172.21.15.70:0/198312535 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb2980037a0 con 0x7fb2a80ffa30 2024-04-01T01:44:59.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.550+0000 7fb2b225f740 1 Processor -- start 2024-04-01T01:44:59.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.550+0000 7fb2b225f740 1 -- start start 2024-04-01T01:44:59.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.550+0000 7fb2b225f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29f48a0 0x55a5d29f4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.550+0000 7fb2b225f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f5290 0x55a5d29f9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.550+0000 7fb2b225f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 0x55a5d29fc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.550+0000 7fb2b225f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a5d29551b0 con 0x55a5d29f5290 2024-04-01T01:44:59.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.550+0000 7fb2b225f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a5d28fdc00 con 0x55a5d29fa400 2024-04-01T01:44:59.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.550+0000 7fb2b225f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a5d2951250 con 0x55a5d29f48a0 2024-04-01T01:44:59.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.550+0000 7fb29e7fc640 1 --2- 172.21.15.70:0/198312535 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2940a2520 0x7fb2940a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.551+0000 7fb2acc25640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 0x55a5d29fc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.551+0000 7fb2adc27640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29f48a0 0x55a5d29f4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.551+0000 7fb2adc27640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29f48a0 0x55a5d29f4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39356/0 (socket says 172.21.15.70:39356) 2024-04-01T01:44:59.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.551+0000 7fb2acc25640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 0x55a5d29fc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42492/0 (socket says 172.21.15.70:42492) 2024-04-01T01:44:59.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.551+0000 7fb2adc27640 1 -- 172.21.15.70:0/1508930415 learned_addr learned my addr 172.21.15.70:0/1508930415 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.551+0000 7fb29e7fc640 1 -- 172.21.15.70:0/198312535 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb2980cede0 con 0x7fb2a80ffa30 2024-04-01T01:44:59.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.551+0000 7fb2adc27640 1 --2- 172.21.15.70:0/198312535 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2940a2520 0x7fb2940a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.551+0000 7fb2acc25640 1 -- 172.21.15.70:0/1508930415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29f48a0 msgr2=0x55a5d29f4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.551+0000 7fb2acc25640 1 --2- 172.21.15.70:0/1508930415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29f48a0 0x55a5d29f4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.551+0000 7fb2acc25640 1 -- 172.21.15.70:0/1508930415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f5290 msgr2=0x55a5d29f9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.551+0000 7fb2acc25640 1 --2- 172.21.15.70:0/1508930415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f5290 0x55a5d29f9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.551+0000 7fb2acc25640 1 -- 172.21.15.70:0/1508930415 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5d291d3d0 con 0x55a5d29fa400 2024-04-01T01:44:59.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.551+0000 7fb2adc27640 1 --2- 172.21.15.70:0/1508930415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29f48a0 0x55a5d29f4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:59.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.551+0000 7fb2acc25640 1 --2- 172.21.15.70:0/1508930415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 0x55a5d29fc890 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4009bd0 tx=0x7fb2a4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a01f05c9a8f07a2e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.551+0000 7fb2adc27640 1 --2- 172.21.15.70:0/198312535 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2940a2520 0x7fb2940a49e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0009a60 tx=0x7fb2a00324f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.551+0000 7fb2937fe640 1 -- 172.21.15.70:0/1508930415 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4004230 con 0x55a5d29fa400 2024-04-01T01:44:59.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.552+0000 7fb2937fe640 1 -- 172.21.15.70:0/1508930415 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2a4009e40 con 0x55a5d29fa400 2024-04-01T01:44:59.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.552+0000 7fb2937fe640 1 -- 172.21.15.70:0/1508930415 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4004cc0 con 0x55a5d29fa400 2024-04-01T01:44:59.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.552+0000 7fb2b225f740 1 -- 172.21.15.70:0/1508930415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 msgr2=0x55a5d29fc890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.552+0000 7fb2b225f740 1 --2- 172.21.15.70:0/1508930415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 0x55a5d29fc890 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4009bd0 tx=0x7fb2a4011810 comp rx=0 tx=0).stop 2024-04-01T01:44:59.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.553+0000 7fb2b225f740 1 -- 172.21.15.70:0/1508930415 shutdown_connections 2024-04-01T01:44:59.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.553+0000 7fb2b225f740 1 --2- 172.21.15.70:0/1508930415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 0x55a5d29fc890 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.553+0000 7fb2b225f740 1 --2- 172.21.15.70:0/1508930415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f5290 0x55a5d29f9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.553+0000 7fb2b225f740 1 --2- 172.21.15.70:0/1508930415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29f48a0 0x55a5d29f4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.553+0000 7fb2b225f740 1 -- 172.21.15.70:0/1508930415 >> 172.21.15.70:0/1508930415 conn(0x55a5d29eba00 msgr2=0x55a5d29ebe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.553+0000 7fb2b225f740 1 -- 172.21.15.70:0/1508930415 shutdown_connections 2024-04-01T01:44:59.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.553+0000 7fb2b225f740 1 -- 172.21.15.70:0/1508930415 wait complete. 2024-04-01T01:44:59.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.553+0000 7fb2b225f740 1 -- start start 2024-04-01T01:44:59.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.553+0000 7fb2b225f740 1 Processor -- start 2024-04-01T01:44:59.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.554+0000 7fb2b225f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29f48a0 0x55a5d29fed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.554+0000 7fb2b225f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f5290 0x55a5d29ff2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.554+0000 7fb2b225f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 0x55a5d2a06310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.554+0000 7fb2b225f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a5d29fd3b0 con 0x55a5d29f5290 2024-04-01T01:44:59.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.554+0000 7fb2b225f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a5d28f3470 con 0x55a5d29fa400 2024-04-01T01:44:59.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.554+0000 7fb2b225f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a5d29fd530 con 0x55a5d29f48a0 2024-04-01T01:44:59.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.554+0000 7fb2b225f740 10 client.0 _ll_register_callbacks cb 0x55a5d2910cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:59.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.554+0000 7fb2acc25640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f5290 0x55a5d29ff2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.554+0000 7fb2acc25640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f5290 0x55a5d29ff2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54250/0 (socket says 172.21.15.70:54250) 2024-04-01T01:44:59.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.554+0000 7fb2acc25640 1 -- 172.21.15.70:0/4047468858 learned_addr learned my addr 172.21.15.70:0/4047468858 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.554+0000 7fb2adc27640 1 --2- 172.21.15.70:0/4047468858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 0x55a5d2a06310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.554+0000 7fb2ad426640 1 --2- 172.21.15.70:0/4047468858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29f48a0 0x55a5d29fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.555+0000 7fb2acc25640 1 -- 172.21.15.70:0/4047468858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29f48a0 msgr2=0x55a5d29fed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.555+0000 7fb2acc25640 1 --2- 172.21.15.70:0/4047468858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29f48a0 0x55a5d29fed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.555+0000 7fb2acc25640 1 -- 172.21.15.70:0/4047468858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 msgr2=0x55a5d2a06310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.555+0000 7fb2acc25640 1 --2- 172.21.15.70:0/4047468858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 0x55a5d2a06310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.555+0000 7fb2acc25640 1 -- 172.21.15.70:0/4047468858 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5d26693f0 con 0x55a5d29f5290 2024-04-01T01:44:59.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.555+0000 7fb2acc25640 1 --2- 172.21.15.70:0/4047468858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f5290 0x55a5d29ff2d0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4005bc0 tx=0x7fb2a4019b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30de3f3fdaa0551 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.555+0000 7fb290ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:44:59.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.555+0000 7fb290ff9640 1 -- 172.21.15.70:0/4047468858 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4028020 con 0x55a5d29f5290 2024-04-01T01:44:59.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.555+0000 7fb2b225f740 1 -- 172.21.15.70:0/4047468858 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a5d29626f0 con 0x55a5d29f5290 2024-04-01T01:44:59.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.555+0000 7fb290ff9640 1 -- 172.21.15.70:0/4047468858 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb2a4024470 con 0x55a5d29f5290 2024-04-01T01:44:59.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.556+0000 7fb290ff9640 1 -- 172.21.15.70:0/4047468858 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb2a4033050 con 0x55a5d29f5290 2024-04-01T01:44:59.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.556+0000 7fb2b225f740 10 client.18831 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:59.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.556+0000 7fb2b225f740 1 -- 172.21.15.70:0/4047468858 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a5d27337d0 con 0x55a5d29f5290 2024-04-01T01:44:59.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.556+0000 7fb290ff9640 1 -- 172.21.15.70:0/4047468858 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a5d29626f0 con 0x55a5d29f5290 2024-04-01T01:44:59.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.556+0000 7fb290ff9640 1 -- 172.21.15.70:0/4047468858 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb2a4018070 con 0x55a5d29f5290 2024-04-01T01:44:59.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.556+0000 7fb2b225f740 10 client.18831 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:59.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.556+0000 7fb2b225f740 10 client.18831 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:44:59.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.556+0000 7fb2b225f740 20 client.18831 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:59.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.557+0000 7fb2b225f740 1 --2- 172.21.15.70:0/4047468858 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55a5d2a0ab60 0x55a5d2a0d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.557+0000 7fb2b225f740 4 client.18831 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:59.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.557+0000 7fb2b225f740 1 -- 172.21.15.70:0/4047468858 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5d295ca40 con 0x55a5d2a0ab60 2024-04-01T01:44:59.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.557+0000 7fb2ad426640 1 --2- 172.21.15.70:0/4047468858 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55a5d2a0ab60 0x55a5d2a0d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.557+0000 7fb2ad426640 1 --2- 172.21.15.70:0/4047468858 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55a5d2a0ab60 0x55a5d2a0d000 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c6a8e4a759d6e989 server_cookie=e7eccb9036a147a8 in_seq=0 out_seq=0 2024-04-01T01:44:59.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.557+0000 7fb290ff9640 10 client.18831 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:44:59.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.558+0000 7fb290ff9640 1 -- 172.21.15.70:0/4047468858 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a5d295ca40 con 0x55a5d2a0ab60 2024-04-01T01:44:59.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.558+0000 7fb290ff9640 10 client.18831 handle_command_reply: tid=0 2024-04-01T01:44:59.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.558+0000 7fb2b225f740 1 client.18831 shutdown 2024-04-01T01:44:59.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.558+0000 7fb2b225f740 1 -- 172.21.15.70:0/4047468858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f5290 msgr2=0x55a5d29ff2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.558+0000 7fb2b225f740 1 --2- 172.21.15.70:0/4047468858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f5290 0x55a5d29ff2d0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4005bc0 tx=0x7fb2a4019b80 comp rx=0 tx=0).stop 2024-04-01T01:44:59.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.559+0000 7fb2b225f740 1 -- 172.21.15.70:0/4047468858 shutdown_connections 2024-04-01T01:44:59.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.559+0000 7fb2b225f740 1 --2- 172.21.15.70:0/4047468858 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55a5d2a0ab60 0x55a5d2a0d000 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.559+0000 7fb2b225f740 1 --2- 172.21.15.70:0/4047468858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 0x55a5d2a06310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.559+0000 7fb2b225f740 1 --2- 172.21.15.70:0/4047468858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f5290 0x55a5d29ff2d0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.559+0000 7fb2b225f740 1 --2- 172.21.15.70:0/4047468858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29f48a0 0x55a5d29fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.559+0000 7fb2b225f740 1 -- 172.21.15.70:0/4047468858 >> 172.21.15.70:0/4047468858 conn(0x55a5d29eba00 msgr2=0x55a5d29ead40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.559+0000 7fb290ff9640 0 client.18831 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:44:59.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.559+0000 7fb2b225f740 1 -- 172.21.15.70:0/4047468858 shutdown_connections 2024-04-01T01:44:59.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.559+0000 7fb2b225f740 1 -- 172.21.15.70:0/4047468858 wait complete. 2024-04-01T01:44:59.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.559+0000 7fb2b225f740 20 client.18831 trim_cache size 0 max 16384 2024-04-01T01:44:59.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.583+0000 7fb2b225f740 1 Processor -- start 2024-04-01T01:44:59.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.584+0000 7fb2b225f740 1 -- start start 2024-04-01T01:44:59.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.584+0000 7fb2b225f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f48a0 0x55a5d29fed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.584+0000 7fb2b225f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29f5290 0x55a5d29ff2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.584+0000 7fb2b225f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 0x55a5d29ea4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.584+0000 7fb2b225f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81a8230 con 0x55a5d29f48a0 2024-04-01T01:44:59.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.584+0000 7fb2b225f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81a9d70 con 0x55a5d29fa400 2024-04-01T01:44:59.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.584+0000 7fb2b225f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81a8900 con 0x55a5d29f5290 2024-04-01T01:44:59.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.584+0000 7fb2adc27640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 0x55a5d29ea4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.584+0000 7fb2adc27640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 0x55a5d29ea4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42522/0 (socket says 172.21.15.70:42522) 2024-04-01T01:44:59.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.584+0000 7fb2adc27640 1 -- 172.21.15.70:0/979659117 learned_addr learned my addr 172.21.15.70:0/979659117 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.585+0000 7fb2adc27640 1 -- 172.21.15.70:0/979659117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29f5290 msgr2=0x55a5d29ff2d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:44:59.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.585+0000 7fb2adc27640 1 --2- 172.21.15.70:0/979659117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29f5290 0x55a5d29ff2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.585+0000 7fb2adc27640 1 -- 172.21.15.70:0/979659117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f48a0 msgr2=0x55a5d29fed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.585+0000 7fb2adc27640 1 --2- 172.21.15.70:0/979659117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f48a0 0x55a5d29fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.585+0000 7fb2adc27640 1 -- 172.21.15.70:0/979659117 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5d29f9a40 con 0x55a5d29fa400 2024-04-01T01:44:59.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.585+0000 7fb2adc27640 1 --2- 172.21.15.70:0/979659117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 0x55a5d29ea4b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0009cd0 tx=0x7fb2a002e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=298828acae2dfd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.590+0000 7fb292ffd640 1 -- 172.21.15.70:0/979659117 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0039040 con 0x55a5d29fa400 2024-04-01T01:44:59.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.590+0000 7fb292ffd640 1 -- 172.21.15.70:0/979659117 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2a0054070 con 0x55a5d29fa400 2024-04-01T01:44:59.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.591+0000 7fb292ffd640 1 -- 172.21.15.70:0/979659117 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a004f6c0 con 0x55a5d29fa400 2024-04-01T01:44:59.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.592+0000 7fb2b225f740 1 -- 172.21.15.70:0/979659117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 msgr2=0x55a5d29ea4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.592+0000 7fb2b225f740 1 --2- 172.21.15.70:0/979659117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 0x55a5d29ea4b0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0009cd0 tx=0x7fb2a002e040 comp rx=0 tx=0).stop 2024-04-01T01:44:59.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.592+0000 7fb2b225f740 1 -- 172.21.15.70:0/979659117 shutdown_connections 2024-04-01T01:44:59.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.592+0000 7fb2b225f740 1 --2- 172.21.15.70:0/979659117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29fa400 0x55a5d29ea4b0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.592+0000 7fb2b225f740 1 --2- 172.21.15.70:0/979659117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29f5290 0x55a5d29ff2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.592+0000 7fb2b225f740 1 --2- 172.21.15.70:0/979659117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f48a0 0x55a5d29fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.592+0000 7fb2b225f740 1 -- 172.21.15.70:0/979659117 >> 172.21.15.70:0/979659117 conn(0x55a5d29eba00 msgr2=0x55a5d29ebde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.595+0000 7fb2b225f740 1 -- 172.21.15.70:0/979659117 shutdown_connections 2024-04-01T01:44:59.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.595+0000 7fb2b225f740 1 -- 172.21.15.70:0/979659117 wait complete. 2024-04-01T01:44:59.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.595+0000 7fb2b225f740 1 -- start start 2024-04-01T01:44:59.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.596+0000 7fb2b225f740 1 Processor -- start 2024-04-01T01:44:59.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.596+0000 7fb2b225f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f48a0 0x55a5d2a0e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.596+0000 7fb2b225f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29f5290 0x55a5d2a0e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.596+0000 7fb2b225f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29fa400 0x55a5d2a117e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.596+0000 7fb2b225f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81a96a0 con 0x55a5d29f48a0 2024-04-01T01:44:59.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.596+0000 7fb2b225f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81a8fd0 con 0x55a5d29f5290 2024-04-01T01:44:59.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.596+0000 7fb2b225f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81a6210 con 0x55a5d29fa400 2024-04-01T01:44:59.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.596+0000 7fb2b225f740 10 client.0 _ll_register_callbacks cb 0x7fb2a4019790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:59.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.596+0000 7fb2acc25640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29f5290 0x55a5d2a0e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.596+0000 7fb2acc25640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29f5290 0x55a5d2a0e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42530/0 (socket says 172.21.15.70:42530) 2024-04-01T01:44:59.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.596+0000 7fb2acc25640 1 -- 172.21.15.70:0/447653327 learned_addr learned my addr 172.21.15.70:0/447653327 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.596+0000 7fb2adc27640 1 --2- 172.21.15.70:0/447653327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29fa400 0x55a5d2a117e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.597+0000 7fb2ad426640 1 --2- 172.21.15.70:0/447653327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f48a0 0x55a5d2a0e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.597+0000 7fb2acc25640 1 -- 172.21.15.70:0/447653327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29fa400 msgr2=0x55a5d2a117e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.597+0000 7fb2acc25640 1 --2- 172.21.15.70:0/447653327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29fa400 0x55a5d2a117e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.597+0000 7fb2acc25640 1 -- 172.21.15.70:0/447653327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f48a0 msgr2=0x55a5d2a0e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.597+0000 7fb2acc25640 1 --2- 172.21.15.70:0/447653327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f48a0 0x55a5d2a0e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.597+0000 7fb2acc25640 1 -- 172.21.15.70:0/447653327 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5d2a11dd0 con 0x55a5d29f5290 2024-04-01T01:44:59.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.597+0000 7fb2adc27640 1 --2- 172.21.15.70:0/447653327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29fa400 0x55a5d2a117e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:44:59.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.597+0000 7fb2acc25640 1 --2- 172.21.15.70:0/447653327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29f5290 0x55a5d2a0e830 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4009bd0 tx=0x7fb2a4011d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d5e7fbe38b07ce1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.597+0000 7fb290ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:44:59.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.597+0000 7fb290ff9640 1 -- 172.21.15.70:0/447653327 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4028070 con 0x55a5d29f5290 2024-04-01T01:44:59.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.598+0000 7fb290ff9640 1 -- 172.21.15.70:0/447653327 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb2a4005040 con 0x55a5d29f5290 2024-04-01T01:44:59.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.598+0000 7fb2b225f740 1 -- 172.21.15.70:0/447653327 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2a0054070 con 0x55a5d29f5290 2024-04-01T01:44:59.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.598+0000 7fb290ff9640 1 -- 172.21.15.70:0/447653327 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb2a401e040 con 0x55a5d29f5290 2024-04-01T01:44:59.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.598+0000 7fb2b225f740 10 client.28880 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:59.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.598+0000 7fb2b225f740 1 -- 172.21.15.70:0/447653327 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a5d2a6ce90 con 0x55a5d29f5290 2024-04-01T01:44:59.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.598+0000 7fb290ff9640 1 -- 172.21.15.70:0/447653327 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2a0054070 con 0x55a5d29f5290 2024-04-01T01:44:59.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.598+0000 7fb290ff9640 1 -- 172.21.15.70:0/447653327 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb2a4018040 con 0x55a5d29f5290 2024-04-01T01:44:59.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.599+0000 7fb2b225f740 10 client.28880 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:59.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.599+0000 7fb2b225f740 10 client.28880 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:44:59.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.599+0000 7fb2b225f740 20 client.28880 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:59.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.599+0000 7fb2b225f740 1 --2- 172.21.15.70:0/447653327 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55a5d2a6d810 0x55a5d2a6fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.599+0000 7fb2b225f740 4 client.28880 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:44:59.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.599+0000 7fb2b225f740 1 -- 172.21.15.70:0/447653327 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a5d28da6a0 con 0x55a5d2a6d810 2024-04-01T01:44:59.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.599+0000 7fb2ad426640 1 --2- 172.21.15.70:0/447653327 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55a5d2a6d810 0x55a5d2a6fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.600+0000 7fb2ad426640 1 --2- 172.21.15.70:0/447653327 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55a5d2a6d810 0x55a5d2a6fc30 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2bdd230756d9a6f6 server_cookie=b79dea1fe276d6c9 in_seq=0 out_seq=0 2024-04-01T01:44:59.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.600+0000 7fb290ff9640 10 client.28880 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:44:59.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.600+0000 7fb290ff9640 1 -- 172.21.15.70:0/447653327 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a5d28da6a0 con 0x55a5d2a6d810 2024-04-01T01:44:59.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.600+0000 7fb290ff9640 10 client.28880 handle_command_reply: tid=0 2024-04-01T01:44:59.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.600+0000 7fb2b225f740 1 client.28880 shutdown 2024-04-01T01:44:59.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.600+0000 7fb2b225f740 1 -- 172.21.15.70:0/447653327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29f5290 msgr2=0x55a5d2a0e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.600+0000 7fb2b225f740 1 --2- 172.21.15.70:0/447653327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29f5290 0x55a5d2a0e830 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4009bd0 tx=0x7fb2a4011d90 comp rx=0 tx=0).stop 2024-04-01T01:44:59.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.600+0000 7fb2b225f740 1 -- 172.21.15.70:0/447653327 shutdown_connections 2024-04-01T01:44:59.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.600+0000 7fb2b225f740 1 --2- 172.21.15.70:0/447653327 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55a5d2a6d810 0x55a5d2a6fc30 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.600+0000 7fb2b225f740 1 --2- 172.21.15.70:0/447653327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a5d29fa400 0x55a5d2a117e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.600+0000 7fb2b225f740 1 --2- 172.21.15.70:0/447653327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a5d29f5290 0x55a5d2a0e830 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.600+0000 7fb2b225f740 1 --2- 172.21.15.70:0/447653327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5d29f48a0 0x55a5d2a0e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.600+0000 7fb2b225f740 1 -- 172.21.15.70:0/447653327 >> 172.21.15.70:0/447653327 conn(0x55a5d29eba00 msgr2=0x55a5d29f6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.600+0000 7fb290ff9640 0 client.28880 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:44:59.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.600+0000 7fb2b225f740 1 -- 172.21.15.70:0/447653327 shutdown_connections 2024-04-01T01:44:59.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.600+0000 7fb2b225f740 1 -- 172.21.15.70:0/447653327 wait complete. 2024-04-01T01:44:59.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.601+0000 7fb2b225f740 20 client.28880 trim_cache size 0 max 16384 2024-04-01T01:44:59.603 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:44:59.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.601+0000 7fb2af68a640 1 -- 172.21.15.70:0/198312535 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2940a2520 msgr2=0x7fb2940a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.601+0000 7fb2af68a640 1 --2- 172.21.15.70:0/198312535 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2940a2520 0x7fb2940a49e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0009a60 tx=0x7fb2a00324f0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.601+0000 7fb2af68a640 1 -- 172.21.15.70:0/198312535 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a80ffa30 msgr2=0x7fb2a81979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.601+0000 7fb2af68a640 1 --2- 172.21.15.70:0/198312535 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a80ffa30 0x7fb2a81979c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb29800e9c0 tx=0x7fb29800ee80 comp rx=0 tx=0).stop 2024-04-01T01:44:59.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.601+0000 7fb2af68a640 1 -- 172.21.15.70:0/198312535 shutdown_connections 2024-04-01T01:44:59.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.601+0000 7fb2af68a640 1 --2- 172.21.15.70:0/198312535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2a8103460 0x7fb2a819ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.601+0000 7fb2af68a640 1 --2- 172.21.15.70:0/198312535 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2940a2520 0x7fb2940a49e0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.601+0000 7fb2af68a640 1 --2- 172.21.15.70:0/198312535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2a8102aa0 0x7fb2a8197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.601+0000 7fb2af68a640 1 --2- 172.21.15.70:0/198312535 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a80ffa30 0x7fb2a81979c0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.601+0000 7fb2af68a640 1 -- 172.21.15.70:0/198312535 >> 172.21.15.70:0/198312535 conn(0x7fb2a80fb340 msgr2=0x7fb2a8105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.602+0000 7fb2af68a640 1 -- 172.21.15.70:0/198312535 shutdown_connections 2024-04-01T01:44:59.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.602+0000 7fb2af68a640 1 -- 172.21.15.70:0/198312535 wait complete. 2024-04-01T01:44:59.620 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:44:59.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.769+0000 7fc37291e640 1 -- 172.21.15.70:0/830322150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc36c108f60 msgr2=0x7fc36c1093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.769+0000 7fc37291e640 1 --2- 172.21.15.70:0/830322150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc36c108f60 0x7fc36c1093e0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fc360009900 tx=0x7fc36002bc30 comp rx=0 tx=0).stop 2024-04-01T01:44:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.769+0000 7fc37291e640 1 -- 172.21.15.70:0/830322150 shutdown_connections 2024-04-01T01:44:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.769+0000 7fc37291e640 1 --2- 172.21.15.70:0/830322150 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc36c109920 0x7fc36c1101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.769+0000 7fc37291e640 1 --2- 172.21.15.70:0/830322150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc36c108f60 0x7fc36c1093e0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.769+0000 7fc37291e640 1 --2- 172.21.15.70:0/830322150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36c107d70 0x7fc36c108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.769+0000 7fc37291e640 1 -- 172.21.15.70:0/830322150 >> 172.21.15.70:0/830322150 conn(0x7fc36c076d00 msgr2=0x7fc36c079140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.769+0000 7fc37291e640 1 -- 172.21.15.70:0/830322150 shutdown_connections 2024-04-01T01:44:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.769+0000 7fc37291e640 1 -- 172.21.15.70:0/830322150 wait complete. 2024-04-01T01:44:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.769+0000 7fc37291e640 1 Processor -- start 2024-04-01T01:44:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.769+0000 7fc37291e640 1 -- start start 2024-04-01T01:44:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.769+0000 7fc37291e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36c107d70 0x7fc36c19dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.769+0000 7fc37291e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc36c108f60 0x7fc36c19e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc37291e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc36c109920 0x7fc36c1a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc37291e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c113880 con 0x7fc36c108f60 2024-04-01T01:44:59.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc37291e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c113700 con 0x7fc36c109920 2024-04-01T01:44:59.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc37291e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c113a00 con 0x7fc36c107d70 2024-04-01T01:44:59.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc36b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc36c108f60 0x7fc36c19e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc36b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc36c108f60 0x7fc36c19e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54272/0 (socket says 172.21.15.70:54272) 2024-04-01T01:44:59.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc36b7fe640 1 -- 172.21.15.70:0/1977440281 learned_addr learned my addr 172.21.15.70:0/1977440281 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc36bfff640 1 --2- 172.21.15.70:0/1977440281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36c107d70 0x7fc36c19dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc370ebb640 1 --2- 172.21.15.70:0/1977440281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc36c109920 0x7fc36c1a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc36b7fe640 1 -- 172.21.15.70:0/1977440281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36c107d70 msgr2=0x7fc36c19dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc36b7fe640 1 --2- 172.21.15.70:0/1977440281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36c107d70 0x7fc36c19dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc36b7fe640 1 -- 172.21.15.70:0/1977440281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc36c109920 msgr2=0x7fc36c1a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc36b7fe640 1 --2- 172.21.15.70:0/1977440281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc36c109920 0x7fc36c1a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc36b7fe640 1 -- 172.21.15.70:0/1977440281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc36c1a5bf0 con 0x7fc36c108f60 2024-04-01T01:44:59.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc36b7fe640 1 --2- 172.21.15.70:0/1977440281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc36c108f60 0x7fc36c19e510 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc360006010 tx=0x7fc360004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc3697fa640 1 -- 172.21.15.70:0/1977440281 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc360043020 con 0x7fc36c108f60 2024-04-01T01:44:59.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc3697fa640 1 -- 172.21.15.70:0/1977440281 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc360004440 con 0x7fc36c108f60 2024-04-01T01:44:59.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc37291e640 1 -- 172.21.15.70:0/1977440281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc36c1a5e80 con 0x7fc36c108f60 2024-04-01T01:44:59.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc3697fa640 1 -- 172.21.15.70:0/1977440281 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc360003bf0 con 0x7fc36c108f60 2024-04-01T01:44:59.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.770+0000 7fc37291e640 1 -- 172.21.15.70:0/1977440281 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc36c1a6390 con 0x7fc36c108f60 2024-04-01T01:44:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.773+0000 7fc3697fa640 1 -- 172.21.15.70:0/1977440281 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fc360046020 con 0x7fc36c108f60 2024-04-01T01:44:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.774+0000 7fc3755d8740 1 Processor -- start 2024-04-01T01:44:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.774+0000 7fc3697fa640 1 --2- 172.21.15.70:0/1977440281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc3540a2520 0x7fc3540a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.774+0000 7fc3755d8740 1 -- start start 2024-04-01T01:44:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.774+0000 7fc3697fa640 1 -- 172.21.15.70:0/1977440281 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc3600f0e00 con 0x7fc36c108f60 2024-04-01T01:44:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc3755d8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cae490 0x562483cae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc3755d8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483caee80 0x562483cb3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc3755d8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483cb3ff0 0x562483cb6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc3755d8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562483c131b0 con 0x562483cb3ff0 2024-04-01T01:44:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc3755d8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562483bbbc00 con 0x562483caee80 2024-04-01T01:44:59.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc3755d8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562483c0f250 con 0x562483cae490 2024-04-01T01:44:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc370ebb640 1 --2- 172.21.15.70:0/1977440281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc3540a2520 0x7fc3540a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc36bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cae490 0x562483cae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc36bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cae490 0x562483cae8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39396/0 (socket says 172.21.15.70:39396) 2024-04-01T01:44:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc36bfff640 1 -- 172.21.15.70:0/2726528278 learned_addr learned my addr 172.21.15.70:0/2726528278 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc36b7fe640 1 --2- 172.21.15.70:0/2726528278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483caee80 0x562483cb3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc370ebb640 1 --2- 172.21.15.70:0/2726528278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483cb3ff0 0x562483cb6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc36bfff640 1 -- 172.21.15.70:0/2726528278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483caee80 msgr2=0x562483cb3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc36bfff640 1 --2- 172.21.15.70:0/2726528278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483caee80 0x562483cb3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc36bfff640 1 -- 172.21.15.70:0/2726528278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483cb3ff0 msgr2=0x562483cb6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc36bfff640 1 --2- 172.21.15.70:0/2726528278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483cb3ff0 0x562483cb6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc36bfff640 1 -- 172.21.15.70:0/2726528278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562483bdb3d0 con 0x562483cae490 2024-04-01T01:44:59.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc36b7fe640 1 --2- 172.21.15.70:0/2726528278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483caee80 0x562483cb3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:59.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc370ebb640 1 --2- 172.21.15.70:0/1977440281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc3540a2520 0x7fc3540a49e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc364005fd0 tx=0x7fc364009c60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.775+0000 7fc36bfff640 1 --2- 172.21.15.70:0/2726528278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cae490 0x562483cae8b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc35c00bcb0 tx=0x7fc35c0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=229524c750ef1d4b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.776+0000 7fc3527fc640 1 -- 172.21.15.70:0/2726528278 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc35c004020 con 0x562483cae490 2024-04-01T01:44:59.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.776+0000 7fc3527fc640 1 -- 172.21.15.70:0/2726528278 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc35c0054d0 con 0x562483cae490 2024-04-01T01:44:59.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.776+0000 7fc3527fc640 1 -- 172.21.15.70:0/2726528278 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc35c004c40 con 0x562483cae490 2024-04-01T01:44:59.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.777+0000 7fc3755d8740 1 -- 172.21.15.70:0/2726528278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cae490 msgr2=0x562483cae8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.777+0000 7fc3755d8740 1 --2- 172.21.15.70:0/2726528278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cae490 0x562483cae8b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc35c00bcb0 tx=0x7fc35c0107a0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.777+0000 7fc3755d8740 1 -- 172.21.15.70:0/2726528278 shutdown_connections 2024-04-01T01:44:59.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.777+0000 7fc3755d8740 1 --2- 172.21.15.70:0/2726528278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483cb3ff0 0x562483cb6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.777+0000 7fc3755d8740 1 --2- 172.21.15.70:0/2726528278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483caee80 0x562483cb3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.777+0000 7fc3755d8740 1 --2- 172.21.15.70:0/2726528278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cae490 0x562483cae8b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.777+0000 7fc3755d8740 1 -- 172.21.15.70:0/2726528278 >> 172.21.15.70:0/2726528278 conn(0x562483ca9a00 msgr2=0x562483ca9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.777+0000 7fc3755d8740 1 -- 172.21.15.70:0/2726528278 shutdown_connections 2024-04-01T01:44:59.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.777+0000 7fc3755d8740 1 -- 172.21.15.70:0/2726528278 wait complete. 2024-04-01T01:44:59.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.777+0000 7fc3755d8740 1 -- start start 2024-04-01T01:44:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.777+0000 7fc3755d8740 1 Processor -- start 2024-04-01T01:44:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc3755d8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483cae490 0x562483cb88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc3755d8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483caee80 0x562483cb8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc3755d8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cb3ff0 0x562483cbfe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc3755d8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562483cb6fa0 con 0x562483cae490 2024-04-01T01:44:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc3755d8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562483bb1470 con 0x562483caee80 2024-04-01T01:44:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc3755d8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562483cb7120 con 0x562483cb3ff0 2024-04-01T01:44:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc3755d8740 10 client.0 _ll_register_callbacks cb 0x562483bcecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc370ebb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cb3ff0 0x562483cbfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc370ebb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cb3ff0 0x562483cbfe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39410/0 (socket says 172.21.15.70:39410) 2024-04-01T01:44:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc370ebb640 1 -- 172.21.15.70:0/3088998887 learned_addr learned my addr 172.21.15.70:0/3088998887 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc36b7fe640 1 --2- 172.21.15.70:0/3088998887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483caee80 0x562483cb8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc36bfff640 1 --2- 172.21.15.70:0/3088998887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483cae490 0x562483cb88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc370ebb640 1 -- 172.21.15.70:0/3088998887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483caee80 msgr2=0x562483cb8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc370ebb640 1 --2- 172.21.15.70:0/3088998887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483caee80 0x562483cb8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc370ebb640 1 -- 172.21.15.70:0/3088998887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483cae490 msgr2=0x562483cb88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc370ebb640 1 --2- 172.21.15.70:0/3088998887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483cae490 0x562483cb88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc370ebb640 1 -- 172.21.15.70:0/3088998887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624839273f0 con 0x562483cb3ff0 2024-04-01T01:44:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc36bfff640 1 --2- 172.21.15.70:0/3088998887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483cae490 0x562483cb88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:44:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc370ebb640 1 --2- 172.21.15.70:0/3088998887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cb3ff0 0x562483cbfe60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc3640114a0 tx=0x7fc364009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8927597ec86242a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc34b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:44:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.778+0000 7fc34b7fe640 1 -- 172.21.15.70:0/3088998887 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc364010b90 con 0x562483cb3ff0 2024-04-01T01:44:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.779+0000 7fc34b7fe640 1 -- 172.21.15.70:0/3088998887 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc364010d30 con 0x562483cb3ff0 2024-04-01T01:44:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.779+0000 7fc3755d8740 1 -- 172.21.15.70:0/3088998887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc35c0054d0 con 0x562483cb3ff0 2024-04-01T01:44:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.779+0000 7fc34b7fe640 1 -- 172.21.15.70:0/3088998887 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc364004b20 con 0x562483cb3ff0 2024-04-01T01:44:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.779+0000 7fc3755d8740 10 client.28690 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.779+0000 7fc3755d8740 1 -- 172.21.15.70:0/3088998887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5624839f17d0 con 0x562483cb3ff0 2024-04-01T01:44:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.779+0000 7fc34b7fe640 1 -- 172.21.15.70:0/3088998887 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc364004cc0 con 0x562483cb3ff0 2024-04-01T01:44:59.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.779+0000 7fc34b7fe640 1 -- 172.21.15.70:0/3088998887 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc36401e070 con 0x562483cb3ff0 2024-04-01T01:44:59.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.779+0000 7fc3755d8740 10 client.28690 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:59.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.779+0000 7fc3755d8740 10 client.28690 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:44:59.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.779+0000 7fc3755d8740 20 client.28690 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:59.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.779+0000 7fc3755d8740 1 --2- 172.21.15.70:0/3088998887 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562483cc4860 0x562483cc6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.779+0000 7fc3755d8740 4 client.28690 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:44:59.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.779+0000 7fc3755d8740 1 -- 172.21.15.70:0/3088998887 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc35c0054d0 con 0x562483cc4860 2024-04-01T01:44:59.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.779+0000 7fc36bfff640 1 --2- 172.21.15.70:0/3088998887 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562483cc4860 0x562483cc6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.780+0000 7fc36bfff640 1 --2- 172.21.15.70:0/3088998887 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562483cc4860 0x562483cc6d20 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aa3c09066dfbab30 server_cookie=8b1de9c085477aa in_seq=0 out_seq=0 2024-04-01T01:44:59.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.780+0000 7fc34b7fe640 10 client.28690 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:44:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.781+0000 7fc34b7fe640 1 -- 172.21.15.70:0/3088998887 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fc35c0054d0 con 0x562483cc4860 2024-04-01T01:44:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.781+0000 7fc34b7fe640 10 client.28690 handle_command_reply: tid=0 2024-04-01T01:44:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.781+0000 7fc3755d8740 1 client.28690 shutdown 2024-04-01T01:44:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.781+0000 7fc3755d8740 1 -- 172.21.15.70:0/3088998887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cb3ff0 msgr2=0x562483cbfe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.781+0000 7fc3755d8740 1 --2- 172.21.15.70:0/3088998887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cb3ff0 0x562483cbfe60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc3640114a0 tx=0x7fc364009500 comp rx=0 tx=0).stop 2024-04-01T01:44:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.781+0000 7fc3755d8740 1 -- 172.21.15.70:0/3088998887 shutdown_connections 2024-04-01T01:44:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.781+0000 7fc3755d8740 1 --2- 172.21.15.70:0/3088998887 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562483cc4860 0x562483cc6d20 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.781+0000 7fc3755d8740 1 --2- 172.21.15.70:0/3088998887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cb3ff0 0x562483cbfe60 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.781+0000 7fc3755d8740 1 --2- 172.21.15.70:0/3088998887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483caee80 0x562483cb8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.781+0000 7fc3755d8740 1 --2- 172.21.15.70:0/3088998887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483cae490 0x562483cb88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.781+0000 7fc3755d8740 1 -- 172.21.15.70:0/3088998887 >> 172.21.15.70:0/3088998887 conn(0x562483ca9a00 msgr2=0x562483ca8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.781+0000 7fc34b7fe640 0 client.28690 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:44:59.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.781+0000 7fc3755d8740 1 -- 172.21.15.70:0/3088998887 shutdown_connections 2024-04-01T01:44:59.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.782+0000 7fc3755d8740 1 -- 172.21.15.70:0/3088998887 wait complete. 2024-04-01T01:44:59.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.782+0000 7fc3755d8740 20 client.28690 trim_cache size 0 max 16384 2024-04-01T01:44:59.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.799+0000 7fc3755d8740 1 Processor -- start 2024-04-01T01:44:59.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.799+0000 7fc3755d8740 1 -- start start 2024-04-01T01:44:59.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.799+0000 7fc3755d8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cae490 0x562483cb88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.799+0000 7fc3755d8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483caee80 0x562483cb8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.799+0000 7fc3755d8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483cb3ff0 0x562483cbfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.799+0000 7fc3755d8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c1b09f0 con 0x562483caee80 2024-04-01T01:44:59.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.799+0000 7fc3755d8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c1b2530 con 0x562483cb3ff0 2024-04-01T01:44:59.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.799+0000 7fc3755d8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c1b10c0 con 0x562483cae490 2024-04-01T01:44:59.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.799+0000 7fc36b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483caee80 0x562483cb8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.799+0000 7fc36b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483caee80 0x562483cb8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54292/0 (socket says 172.21.15.70:54292) 2024-04-01T01:44:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.799+0000 7fc36b7fe640 1 -- 172.21.15.70:0/4161080730 learned_addr learned my addr 172.21.15.70:0/4161080730 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.799+0000 7fc36bfff640 1 --2- 172.21.15.70:0/4161080730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cae490 0x562483cb88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.799+0000 7fc370ebb640 1 --2- 172.21.15.70:0/4161080730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483cb3ff0 0x562483cbfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.800+0000 7fc36b7fe640 1 -- 172.21.15.70:0/4161080730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cae490 msgr2=0x562483cb88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.800+0000 7fc36b7fe640 1 --2- 172.21.15.70:0/4161080730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cae490 0x562483cb88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.800+0000 7fc36b7fe640 1 -- 172.21.15.70:0/4161080730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483cb3ff0 msgr2=0x562483cbfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.800+0000 7fc36b7fe640 1 --2- 172.21.15.70:0/4161080730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483cb3ff0 0x562483cbfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.800+0000 7fc36b7fe640 1 -- 172.21.15.70:0/4161080730 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562483caf260 con 0x562483caee80 2024-04-01T01:44:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.800+0000 7fc36b7fe640 1 --2- 172.21.15.70:0/4161080730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483caee80 0x562483cb8e20 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fc3600b98f0 tx=0x7fc3600dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f1009afefc47ac server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.800+0000 7fc351ffb640 1 -- 172.21.15.70:0/4161080730 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3600bcdf0 con 0x562483caee80 2024-04-01T01:44:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.800+0000 7fc351ffb640 1 -- 172.21.15.70:0/4161080730 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc360005560 con 0x562483caee80 2024-04-01T01:44:59.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.800+0000 7fc351ffb640 1 -- 172.21.15.70:0/4161080730 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3600bb790 con 0x562483caee80 2024-04-01T01:44:59.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.801+0000 7fc3755d8740 1 -- 172.21.15.70:0/4161080730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483caee80 msgr2=0x562483cb8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.801+0000 7fc3755d8740 1 --2- 172.21.15.70:0/4161080730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483caee80 0x562483cb8e20 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fc3600b98f0 tx=0x7fc3600dd040 comp rx=0 tx=0).stop 2024-04-01T01:44:59.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.801+0000 7fc3755d8740 1 -- 172.21.15.70:0/4161080730 shutdown_connections 2024-04-01T01:44:59.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.801+0000 7fc3755d8740 1 --2- 172.21.15.70:0/4161080730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483cb3ff0 0x562483cbfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.801+0000 7fc3755d8740 1 --2- 172.21.15.70:0/4161080730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483caee80 0x562483cb8e20 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.801+0000 7fc3755d8740 1 --2- 172.21.15.70:0/4161080730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cae490 0x562483cb88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.801+0000 7fc3755d8740 1 -- 172.21.15.70:0/4161080730 >> 172.21.15.70:0/4161080730 conn(0x562483ca9a00 msgr2=0x562483ca4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.801+0000 7fc3755d8740 1 -- 172.21.15.70:0/4161080730 shutdown_connections 2024-04-01T01:44:59.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.801+0000 7fc3755d8740 1 -- 172.21.15.70:0/4161080730 wait complete. 2024-04-01T01:44:59.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.802+0000 7fc3755d8740 1 -- start start 2024-04-01T01:44:59.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.802+0000 7fc3755d8740 1 Processor -- start 2024-04-01T01:44:59.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.802+0000 7fc3755d8740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483cae490 0x562483cbd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.802+0000 7fc3755d8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483caee80 0x562483cbd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.802+0000 7fc3755d8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cb3ff0 0x562483cbe6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.802+0000 7fc3755d8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c1b1e60 con 0x562483caee80 2024-04-01T01:44:59.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.802+0000 7fc3755d8740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c1b1790 con 0x562483cae490 2024-04-01T01:44:59.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.802+0000 7fc3755d8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c1ae9d0 con 0x562483cb3ff0 2024-04-01T01:44:59.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.803+0000 7fc370ebb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cb3ff0 0x562483cbe6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.803+0000 7fc3755d8740 10 client.0 _ll_register_callbacks cb 0x7fc364011120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:44:59.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.802+0000 7fc36bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483cae490 0x562483cbd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.803+0000 7fc370ebb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cb3ff0 0x562483cbe6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39418/0 (socket says 172.21.15.70:39418) 2024-04-01T01:44:59.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.803+0000 7fc370ebb640 1 -- 172.21.15.70:0/1389071063 learned_addr learned my addr 172.21.15.70:0/1389071063 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.803+0000 7fc36b7fe640 1 --2- 172.21.15.70:0/1389071063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483caee80 0x562483cbd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.803+0000 7fc36bfff640 1 -- 172.21.15.70:0/1389071063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cb3ff0 msgr2=0x562483cbe6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.803+0000 7fc36bfff640 1 --2- 172.21.15.70:0/1389071063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cb3ff0 0x562483cbe6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.803+0000 7fc36bfff640 1 -- 172.21.15.70:0/1389071063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483caee80 msgr2=0x562483cbd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.803+0000 7fc36bfff640 1 --2- 172.21.15.70:0/1389071063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483caee80 0x562483cbd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.803+0000 7fc36bfff640 1 -- 172.21.15.70:0/1389071063 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562483cbeca0 con 0x562483cae490 2024-04-01T01:44:59.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.803+0000 7fc370ebb640 1 --2- 172.21.15.70:0/1389071063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cb3ff0 0x562483cbe6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:59.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.803+0000 7fc36b7fe640 1 --2- 172.21.15.70:0/1389071063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483caee80 0x562483cbd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:44:59.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.803+0000 7fc36bfff640 1 --2- 172.21.15.70:0/1389071063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483cae490 0x562483cbd1b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc35c00bc80 tx=0x7fc35c005d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3b3a2eefe50bf51 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.803+0000 7fc34b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:44:59.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.803+0000 7fc34b7fe640 1 -- 172.21.15.70:0/1389071063 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc35c005240 con 0x562483cae490 2024-04-01T01:44:59.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.804+0000 7fc3755d8740 1 -- 172.21.15.70:0/1389071063 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562483c206f0 con 0x562483cae490 2024-04-01T01:44:59.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.804+0000 7fc34b7fe640 1 -- 172.21.15.70:0/1389071063 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc35c0053e0 con 0x562483cae490 2024-04-01T01:44:59.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.804+0000 7fc34b7fe640 1 -- 172.21.15.70:0/1389071063 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc35c005640 con 0x562483cae490 2024-04-01T01:44:59.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.804+0000 7fc3755d8740 10 client.28898 fetch_fsmap learned FSMap version 31 2024-04-01T01:44:59.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.804+0000 7fc3755d8740 1 -- 172.21.15.70:0/1389071063 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562483cc0010 con 0x562483cae490 2024-04-01T01:44:59.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.804+0000 7fc34b7fe640 1 -- 172.21.15.70:0/1389071063 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x562483c206f0 con 0x562483cae490 2024-04-01T01:44:59.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.804+0000 7fc34b7fe640 1 -- 172.21.15.70:0/1389071063 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc35c0047b0 con 0x562483cae490 2024-04-01T01:44:59.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.804+0000 7fc3755d8740 10 client.28898 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:44:59.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.804+0000 7fc3755d8740 10 client.28898 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:44:59.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.804+0000 7fc3755d8740 20 client.28898 populate_metadata read hostname 'smithi070' 2024-04-01T01:44:59.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.804+0000 7fc3755d8740 1 --2- 172.21.15.70:0/1389071063 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562483d2b110 0x562483d2d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.804+0000 7fc3755d8740 4 client.28898 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:44:59.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.804+0000 7fc3755d8740 1 -- 172.21.15.70:0/1389071063 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562483c1aa40 con 0x562483d2b110 2024-04-01T01:44:59.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc36b7fe640 1 --2- 172.21.15.70:0/1389071063 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562483d2b110 0x562483d2d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc34b7fe640 10 client.28898 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:44:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc36b7fe640 1 --2- 172.21.15.70:0/1389071063 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562483d2b110 0x562483d2d530 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fed0740553a4319a server_cookie=b8c839a25f619b34 in_seq=0 out_seq=0 2024-04-01T01:44:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc34b7fe640 1 -- 172.21.15.70:0/1389071063 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562483c1aa40 con 0x562483d2b110 2024-04-01T01:44:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc34b7fe640 10 client.28898 handle_command_reply: tid=0 2024-04-01T01:44:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc3755d8740 1 client.28898 shutdown 2024-04-01T01:44:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc3755d8740 1 -- 172.21.15.70:0/1389071063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483cae490 msgr2=0x562483cbd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc3755d8740 1 --2- 172.21.15.70:0/1389071063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483cae490 0x562483cbd1b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc35c00bc80 tx=0x7fc35c005d10 comp rx=0 tx=0).stop 2024-04-01T01:44:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc3755d8740 1 -- 172.21.15.70:0/1389071063 shutdown_connections 2024-04-01T01:44:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc3755d8740 1 --2- 172.21.15.70:0/1389071063 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562483d2b110 0x562483d2d530 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc3755d8740 1 --2- 172.21.15.70:0/1389071063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562483cb3ff0 0x562483cbe6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc34b7fe640 0 client.28898 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:44:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc3755d8740 1 --2- 172.21.15.70:0/1389071063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562483caee80 0x562483cbd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc3755d8740 1 --2- 172.21.15.70:0/1389071063 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562483cae490 0x562483cbd1b0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc3755d8740 1 -- 172.21.15.70:0/1389071063 >> 172.21.15.70:0/1389071063 conn(0x562483ca9a00 msgr2=0x562483cb02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc3755d8740 1 -- 172.21.15.70:0/1389071063 shutdown_connections 2024-04-01T01:44:59.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.805+0000 7fc3755d8740 1 -- 172.21.15.70:0/1389071063 wait complete. 2024-04-01T01:44:59.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.806+0000 7fc3755d8740 20 client.28898 trim_cache size 0 max 16384 2024-04-01T01:44:59.812 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:44:59.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.806+0000 7fc37291e640 1 -- 172.21.15.70:0/1977440281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc3540a2520 msgr2=0x7fc3540a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.806+0000 7fc37291e640 1 --2- 172.21.15.70:0/1977440281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc3540a2520 0x7fc3540a49e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc364005fd0 tx=0x7fc364009c60 comp rx=0 tx=0).stop 2024-04-01T01:44:59.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.806+0000 7fc37291e640 1 -- 172.21.15.70:0/1977440281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc36c108f60 msgr2=0x7fc36c19e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.806+0000 7fc37291e640 1 --2- 172.21.15.70:0/1977440281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc36c108f60 0x7fc36c19e510 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc360006010 tx=0x7fc360004040 comp rx=0 tx=0).stop 2024-04-01T01:44:59.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.806+0000 7fc37291e640 1 -- 172.21.15.70:0/1977440281 shutdown_connections 2024-04-01T01:44:59.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.806+0000 7fc37291e640 1 --2- 172.21.15.70:0/1977440281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc36c109920 0x7fc36c1a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.806+0000 7fc37291e640 1 --2- 172.21.15.70:0/1977440281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc36c108f60 0x7fc36c19e510 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.806+0000 7fc37291e640 1 --2- 172.21.15.70:0/1977440281 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc3540a2520 0x7fc3540a49e0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.806+0000 7fc37291e640 1 --2- 172.21.15.70:0/1977440281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc36c107d70 0x7fc36c19dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.806+0000 7fc37291e640 1 -- 172.21.15.70:0/1977440281 >> 172.21.15.70:0/1977440281 conn(0x7fc36c076d00 msgr2=0x7fc36c10c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.806+0000 7fc37291e640 1 -- 172.21.15.70:0/1977440281 shutdown_connections 2024-04-01T01:44:59.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.806+0000 7fc37291e640 1 -- 172.21.15.70:0/1977440281 wait complete. 2024-04-01T01:44:59.819 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:44:59.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.991+0000 7f96c33d0640 1 -- 172.21.15.70:0/2649029814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96bc104b20 msgr2=0x7f96bc106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.991+0000 7f96c33d0640 1 --2- 172.21.15.70:0/2649029814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96bc104b20 0x7f96bc106f10 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009900 tx=0x7f96ac02bc30 comp rx=0 tx=0).stop 2024-04-01T01:44:59.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.991+0000 7f96c33d0640 1 -- 172.21.15.70:0/2649029814 shutdown_connections 2024-04-01T01:44:59.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.991+0000 7f96c33d0640 1 --2- 172.21.15.70:0/2649029814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96bc107450 0x7f96bc1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.991+0000 7f96c33d0640 1 --2- 172.21.15.70:0/2649029814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96bc104b20 0x7f96bc106f10 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.991+0000 7f96c33d0640 1 --2- 172.21.15.70:0/2649029814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96bc1021f0 0x7f96bc1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.991+0000 7f96c33d0640 1 -- 172.21.15.70:0/2649029814 >> 172.21.15.70:0/2649029814 conn(0x7f96bc0fbc00 msgr2=0x7f96bc0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:44:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.992+0000 7f96c33d0640 1 -- 172.21.15.70:0/2649029814 shutdown_connections 2024-04-01T01:44:59.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.992+0000 7f96c33d0640 1 -- 172.21.15.70:0/2649029814 wait complete. 2024-04-01T01:44:59.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.992+0000 7f96c33d0640 1 Processor -- start 2024-04-01T01:44:59.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c33d0640 1 -- start start 2024-04-01T01:44:59.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c33d0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96bc1021f0 0x7f96bc101b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c33d0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96bc107450 0x7f96bc1001a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c33d0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96bc1006e0 0x7f96bc100b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c33d0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc10d290 con 0x7f96bc107450 2024-04-01T01:44:59.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c33d0640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc10d110 con 0x7f96bc1006e0 2024-04-01T01:44:59.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c33d0640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc10d410 con 0x7f96bc1021f0 2024-04-01T01:44:59.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c116c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96bc1021f0 0x7f96bc101b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c096b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96bc107450 0x7f96bc1001a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c116c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96bc1021f0 0x7f96bc101b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39428/0 (socket says 172.21.15.70:39428) 2024-04-01T01:44:59.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c116c640 1 -- 172.21.15.70:0/3913749632 learned_addr learned my addr 172.21.15.70:0/3913749632 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c196d640 1 --2- 172.21.15.70:0/3913749632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96bc1006e0 0x7f96bc100b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c116c640 1 -- 172.21.15.70:0/3913749632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96bc1006e0 msgr2=0x7f96bc100b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c116c640 1 --2- 172.21.15.70:0/3913749632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96bc1006e0 0x7f96bc100b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c116c640 1 -- 172.21.15.70:0/3913749632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96bc107450 msgr2=0x7f96bc1001a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:44:59.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c116c640 1 --2- 172.21.15.70:0/3913749632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96bc107450 0x7f96bc1001a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.993+0000 7f96c116c640 1 -- 172.21.15.70:0/3913749632 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96bc1a1e50 con 0x7f96bc1021f0 2024-04-01T01:44:59.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.994+0000 7f96c116c640 1 --2- 172.21.15.70:0/3913749632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96bc1021f0 0x7f96bc101b60 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f96b800e9f0 tx=0x7f96b800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.994+0000 7f96b27fc640 1 -- 172.21.15.70:0/3913749632 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b800cd80 con 0x7f96bc1021f0 2024-04-01T01:44:59.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.994+0000 7f96c33d0640 1 -- 172.21.15.70:0/3913749632 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96bc1a2140 con 0x7f96bc1021f0 2024-04-01T01:44:59.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.994+0000 7f96b27fc640 1 -- 172.21.15.70:0/3913749632 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96b80108e0 con 0x7f96bc1021f0 2024-04-01T01:44:59.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.994+0000 7f96b27fc640 1 -- 172.21.15.70:0/3913749632 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b8018920 con 0x7f96bc1021f0 2024-04-01T01:44:59.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.994+0000 7f96c33d0640 1 -- 172.21.15.70:0/3913749632 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96bc1a2660 con 0x7f96bc1021f0 2024-04-01T01:44:59.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.997+0000 7f96c5fbc740 1 Processor -- start 2024-04-01T01:44:59.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.997+0000 7f96c5fbc740 1 -- start start 2024-04-01T01:44:59.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.997+0000 7f96b27fc640 1 -- 172.21.15.70:0/3913749632 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f96b8010a90 con 0x7f96bc1021f0 2024-04-01T01:44:59.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.997+0000 7f96c5fbc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f67c490 0x563a5f67c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.997+0000 7f96c5fbc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67ce80 0x563a5f681980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.997+0000 7f96c5fbc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f681ff0 0x563a5f684480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.997+0000 7f96c5fbc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563a5f5e11b0 con 0x563a5f67ce80 2024-04-01T01:44:59.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.997+0000 7f96c5fbc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563a5f589c00 con 0x563a5f67c490 2024-04-01T01:44:59.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.997+0000 7f96c5fbc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563a5f5dd250 con 0x563a5f681ff0 2024-04-01T01:44:59.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.997+0000 7f96c196d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f67c490 0x563a5f67c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.997+0000 7f96c116c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67ce80 0x563a5f681980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.997+0000 7f96c196d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f67c490 0x563a5f67c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42634/0 (socket says 172.21.15.70:42634) 2024-04-01T01:44:59.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.997+0000 7f96c196d640 1 -- 172.21.15.70:0/118730521 learned_addr learned my addr 172.21.15.70:0/118730521 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:44:59.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.997+0000 7f96c096b640 1 --2- 172.21.15.70:0/118730521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f681ff0 0x563a5f684480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.998+0000 7f96c196d640 1 -- 172.21.15.70:0/118730521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f681ff0 msgr2=0x563a5f684480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.998+0000 7f96c196d640 1 --2- 172.21.15.70:0/118730521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f681ff0 0x563a5f684480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.998+0000 7f96c196d640 1 -- 172.21.15.70:0/118730521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67ce80 msgr2=0x563a5f681980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:44:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.998+0000 7f96c196d640 1 --2- 172.21.15.70:0/118730521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67ce80 0x563a5f681980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:44:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.998+0000 7f96c196d640 1 -- 172.21.15.70:0/118730521 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a5f5a93d0 con 0x563a5f67c490 2024-04-01T01:44:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.998+0000 7f96b27fc640 1 --2- 172.21.15.70:0/3913749632 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96a80a24a0 0x7f96a80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:44:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.998+0000 7f96c116c640 1 --2- 172.21.15.70:0/3913749632 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96a80a24a0 0x7f96a80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:44:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.998+0000 7f96c196d640 1 --2- 172.21.15.70:0/118730521 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f67c490 0x563a5f67c8b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f96b40039f0 tx=0x7f96b4003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8ba5fe644f67d66 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:44:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.998+0000 7f96a77fe640 1 -- 172.21.15.70:0/118730521 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b401a470 con 0x563a5f67c490 2024-04-01T01:44:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.999+0000 7f96a77fe640 1 -- 172.21.15.70:0/118730521 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96b4015070 con 0x563a5f67c490 2024-04-01T01:44:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.999+0000 7f96a77fe640 1 -- 172.21.15.70:0/118730521 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b40107e0 con 0x563a5f67c490 2024-04-01T01:44:59.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.999+0000 7f96b27fc640 1 -- 172.21.15.70:0/3913749632 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f96b80cedc0 con 0x7f96bc1021f0 2024-04-01T01:45:00.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.999+0000 7f96c116c640 1 --2- 172.21.15.70:0/3913749632 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96a80a24a0 0x7f96a80a4960 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f96b80cc860 tx=0x7f96b80d28a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:00.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.999+0000 7f96c5fbc740 1 -- 172.21.15.70:0/118730521 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f67c490 msgr2=0x563a5f67c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:00.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:44:59.999+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/118730521 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f67c490 0x563a5f67c8b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f96b40039f0 tx=0x7f96b4003eb0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.000+0000 7f96c5fbc740 1 -- 172.21.15.70:0/118730521 shutdown_connections 2024-04-01T01:45:00.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.000+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/118730521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f681ff0 0x563a5f684480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.000+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/118730521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67ce80 0x563a5f681980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.000+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/118730521 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f67c490 0x563a5f67c8b0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.000+0000 7f96c5fbc740 1 -- 172.21.15.70:0/118730521 >> 172.21.15.70:0/118730521 conn(0x563a5f677a00 msgr2=0x563a5f677e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.000+0000 7f96c5fbc740 1 -- 172.21.15.70:0/118730521 shutdown_connections 2024-04-01T01:45:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.000+0000 7f96c5fbc740 1 -- 172.21.15.70:0/118730521 wait complete. 2024-04-01T01:45:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.000+0000 7f96c5fbc740 1 -- start start 2024-04-01T01:45:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.000+0000 7f96c5fbc740 1 Processor -- start 2024-04-01T01:45:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.001+0000 7f96c5fbc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67c490 0x563a5f6869b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.001+0000 7f96c5fbc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f67ce80 0x563a5f686ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.001+0000 7f96c196d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67c490 0x563a5f6869b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.001+0000 7f96c196d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67c490 0x563a5f6869b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54332/0 (socket says 172.21.15.70:54332) 2024-04-01T01:45:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.001+0000 7f96c196d640 1 -- 172.21.15.70:0/3906238055 learned_addr learned my addr 172.21.15.70:0/3906238055 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.001+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/3906238055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f681ff0 0x563a5f68df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.001+0000 7f96c5fbc740 1 -- 172.21.15.70:0/3906238055 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563a5f684fa0 con 0x563a5f67c490 2024-04-01T01:45:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.001+0000 7f96c096b640 1 --2- 172.21.15.70:0/3906238055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f67ce80 0x563a5f686ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.001+0000 7f96c5fbc740 1 -- 172.21.15.70:0/3906238055 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563a5f57f470 con 0x563a5f681ff0 2024-04-01T01:45:00.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.001+0000 7f96c5fbc740 1 -- 172.21.15.70:0/3906238055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563a5f685120 con 0x563a5f67ce80 2024-04-01T01:45:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.002+0000 7f96c196d640 1 --2- 172.21.15.70:0/3906238055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f681ff0 0x563a5f68df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.002+0000 7f96c096b640 1 -- 172.21.15.70:0/3906238055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f681ff0 msgr2=0x563a5f68df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.002+0000 7f96c096b640 1 --2- 172.21.15.70:0/3906238055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f681ff0 0x563a5f68df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.002+0000 7f96c096b640 1 -- 172.21.15.70:0/3906238055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67c490 msgr2=0x563a5f6869b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.002+0000 7f96c096b640 1 --2- 172.21.15.70:0/3906238055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67c490 0x563a5f6869b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.002+0000 7f96c096b640 1 -- 172.21.15.70:0/3906238055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a5f2f53f0 con 0x563a5f67ce80 2024-04-01T01:45:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.002+0000 7f96c196d640 1 --2- 172.21.15.70:0/3906238055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67c490 0x563a5f6869b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:45:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.002+0000 7f96c5fbc740 10 client.0 _ll_register_callbacks cb 0x563a5f59ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.002+0000 7f96c096b640 1 --2- 172.21.15.70:0/3906238055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f67ce80 0x563a5f686ef0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f96bc06a0b0 tx=0x7f96ac004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa32a76ebe5605dd server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.002+0000 7f96a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:45:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.002+0000 7f96c5fbc740 1 -- 172.21.15.70:0/3906238055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96b4015070 con 0x563a5f67ce80 2024-04-01T01:45:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.002+0000 7f96a4ff9640 1 -- 172.21.15.70:0/3906238055 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac005d50 con 0x563a5f67ce80 2024-04-01T01:45:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.002+0000 7f96a4ff9640 1 -- 172.21.15.70:0/3906238055 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f96ac005030 con 0x563a5f67ce80 2024-04-01T01:45:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.003+0000 7f96a4ff9640 1 -- 172.21.15.70:0/3906238055 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f96ac005230 con 0x563a5f67ce80 2024-04-01T01:45:00.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.003+0000 7f96c5fbc740 10 client.28708 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:00.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.003+0000 7f96c5fbc740 1 -- 172.21.15.70:0/3906238055 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=31}) -- 0x563a5f3bf7d0 con 0x563a5f67ce80 2024-04-01T01:45:00.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.003+0000 7f96a4ff9640 1 -- 172.21.15.70:0/3906238055 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f96ac0053d0 con 0x563a5f67ce80 2024-04-01T01:45:00.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.003+0000 7f96c5fbc740 10 client.28708 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:00.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.003+0000 7f96c5fbc740 10 client.28708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:00.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.003+0000 7f96c5fbc740 20 client.28708 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:00.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.003+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/3906238055 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563a5f692290 0x563a5f694720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:00.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.003+0000 7f96c5fbc740 4 client.28708 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:45:00.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.003+0000 7f96c5fbc740 1 -- 172.21.15.70:0/3906238055 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a5f5ee6f0 con 0x563a5f692290 2024-04-01T01:45:00.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.003+0000 7f96c196d640 1 --2- 172.21.15.70:0/3906238055 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563a5f692290 0x563a5f694720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:00.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.004+0000 7f96c196d640 1 --2- 172.21.15.70:0/3906238055 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563a5f692290 0x563a5f694720 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d47ff362e6ac412 server_cookie=5c01a1326df6d512 in_seq=0 out_seq=0 2024-04-01T01:45:00.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.004+0000 7f96a4ff9640 10 client.28708 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:00.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:44:59 smithi070 ceph-mon[32179]: pgmap v704: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 9.6 MiB/s wr, 826 op/s 2024-04-01T01:45:00.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.005+0000 7f96a4ff9640 1 -- 172.21.15.70:0/3906238055 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563a5f5ee6f0 con 0x563a5f692290 2024-04-01T01:45:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.005+0000 7f96a4ff9640 10 client.28708 handle_command_reply: tid=0 2024-04-01T01:45:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.005+0000 7f96c5fbc740 1 client.28708 shutdown 2024-04-01T01:45:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.005+0000 7f96c5fbc740 1 -- 172.21.15.70:0/3906238055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f67ce80 msgr2=0x563a5f686ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.005+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/3906238055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f67ce80 0x563a5f686ef0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f96bc06a0b0 tx=0x7f96ac004270 comp rx=0 tx=0).stop 2024-04-01T01:45:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.005+0000 7f96c5fbc740 1 -- 172.21.15.70:0/3906238055 shutdown_connections 2024-04-01T01:45:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.005+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/3906238055 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563a5f692290 0x563a5f694720 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.005+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/3906238055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f681ff0 0x563a5f68df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.005+0000 7f96a4ff9640 0 client.28708 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.005+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/3906238055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f67ce80 0x563a5f686ef0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.005+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/3906238055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67c490 0x563a5f6869b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.005+0000 7f96c5fbc740 1 -- 172.21.15.70:0/3906238055 >> 172.21.15.70:0/3906238055 conn(0x563a5f677a00 msgr2=0x563a5f676d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.005+0000 7f96c5fbc740 1 -- 172.21.15.70:0/3906238055 shutdown_connections 2024-04-01T01:45:00.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.006+0000 7f96c5fbc740 1 -- 172.21.15.70:0/3906238055 wait complete. 2024-04-01T01:45:00.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.006+0000 7f96c5fbc740 20 client.28708 trim_cache size 0 max 16384 2024-04-01T01:45:00.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.030+0000 7f96c5fbc740 1 Processor -- start 2024-04-01T01:45:00.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.031+0000 7f96c5fbc740 1 -- start start 2024-04-01T01:45:00.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.031+0000 7f96c5fbc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67c490 0x563a5f67e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:00.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.031+0000 7f96c5fbc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f67ce80 0x563a5f67e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:00.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.031+0000 7f96c5fbc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f681ff0 0x563a5f68def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:00.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.031+0000 7f96c5fbc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc1acc50 con 0x563a5f67c490 2024-04-01T01:45:00.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.031+0000 7f96c5fbc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc1ae790 con 0x563a5f67ce80 2024-04-01T01:45:00.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.031+0000 7f96c5fbc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc1ad320 con 0x563a5f681ff0 2024-04-01T01:45:00.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.031+0000 7f96c196d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67c490 0x563a5f67e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:00.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.031+0000 7f96c096b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f67ce80 0x563a5f67e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:00.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.031+0000 7f96c196d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67c490 0x563a5f67e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54344/0 (socket says 172.21.15.70:54344) 2024-04-01T01:45:00.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.031+0000 7f96c196d640 1 -- 172.21.15.70:0/3383060602 learned_addr learned my addr 172.21.15.70:0/3383060602 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:00.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.032+0000 7f96c196d640 1 --2- 172.21.15.70:0/3383060602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f681ff0 0x563a5f68def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:00.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.032+0000 7f96c196d640 1 -- 172.21.15.70:0/3383060602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f681ff0 msgr2=0x563a5f68def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:00.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.032+0000 7f96c196d640 1 --2- 172.21.15.70:0/3383060602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f681ff0 0x563a5f68def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.032+0000 7f96c196d640 1 -- 172.21.15.70:0/3383060602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f67ce80 msgr2=0x563a5f67e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:00.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.032+0000 7f96c196d640 1 --2- 172.21.15.70:0/3383060602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f67ce80 0x563a5f67e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.032+0000 7f96c196d640 1 -- 172.21.15.70:0/3383060602 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a5f685260 con 0x563a5f67c490 2024-04-01T01:45:00.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.032+0000 7f96c196d640 1 --2- 172.21.15.70:0/3383060602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67c490 0x563a5f67e3e0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f96b4011dc0 tx=0x7f96b4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e92ea0e6463585 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:00.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.033+0000 7f96a6ffd640 1 -- 172.21.15.70:0/3383060602 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b402c760 con 0x563a5f67c490 2024-04-01T01:45:00.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.033+0000 7f96a6ffd640 1 -- 172.21.15.70:0/3383060602 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96b402c110 con 0x563a5f67c490 2024-04-01T01:45:00.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.033+0000 7f96a6ffd640 1 -- 172.21.15.70:0/3383060602 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b401a8d0 con 0x563a5f67c490 2024-04-01T01:45:00.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.034+0000 7f96c5fbc740 1 -- 172.21.15.70:0/3383060602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67c490 msgr2=0x563a5f67e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:00.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.034+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/3383060602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67c490 0x563a5f67e3e0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f96b4011dc0 tx=0x7f96b4005fd0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.035+0000 7f96c5fbc740 1 -- 172.21.15.70:0/3383060602 shutdown_connections 2024-04-01T01:45:00.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.035+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/3383060602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f681ff0 0x563a5f68def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.035+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/3383060602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f67ce80 0x563a5f67e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.035+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/3383060602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67c490 0x563a5f67e3e0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.035+0000 7f96c5fbc740 1 -- 172.21.15.70:0/3383060602 >> 172.21.15.70:0/3383060602 conn(0x563a5f677a00 msgr2=0x563a5f677de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:00.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.036+0000 7f96c5fbc740 1 -- 172.21.15.70:0/3383060602 shutdown_connections 2024-04-01T01:45:00.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.037+0000 7f96c5fbc740 1 -- 172.21.15.70:0/3383060602 wait complete. 2024-04-01T01:45:00.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.037+0000 7f96c5fbc740 1 -- start start 2024-04-01T01:45:00.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.037+0000 7f96c5fbc740 1 Processor -- start 2024-04-01T01:45:00.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.037+0000 7f96c5fbc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f67c490 0x563a5f6899b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:00.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.037+0000 7f96c5fbc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67ce80 0x563a5f689ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:00.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.038+0000 7f96c5fbc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f681ff0 0x563a5f68cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:00.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.038+0000 7f96c5fbc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc1ae0c0 con 0x563a5f67ce80 2024-04-01T01:45:00.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.038+0000 7f96c5fbc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc1ad9f0 con 0x563a5f681ff0 2024-04-01T01:45:00.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.038+0000 7f96c5fbc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f96bc1aac30 con 0x563a5f67c490 2024-04-01T01:45:00.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.038+0000 7f96c5fbc740 10 client.0 _ll_register_callbacks cb 0x7f96ac02ded0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:00.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.038+0000 7f96c096b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67ce80 0x563a5f689ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:00.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.038+0000 7f96c096b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67ce80 0x563a5f689ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54360/0 (socket says 172.21.15.70:54360) 2024-04-01T01:45:00.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.038+0000 7f96c096b640 1 -- 172.21.15.70:0/792788045 learned_addr learned my addr 172.21.15.70:0/792788045 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:00.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.039+0000 7f96c196d640 1 --2- 172.21.15.70:0/792788045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f67c490 0x563a5f6899b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:00.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.039+0000 7f96c196d640 1 --2- 172.21.15.70:0/792788045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f681ff0 0x563a5f68cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:00.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.039+0000 7f96c196d640 1 -- 172.21.15.70:0/792788045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f681ff0 msgr2=0x563a5f68cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:00.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.039+0000 7f96c196d640 1 --2- 172.21.15.70:0/792788045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f681ff0 0x563a5f68cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.039+0000 7f96c196d640 1 -- 172.21.15.70:0/792788045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67ce80 msgr2=0x563a5f689ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:00.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.039+0000 7f96c196d640 1 --2- 172.21.15.70:0/792788045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67ce80 0x563a5f689ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.039+0000 7f96c196d640 1 -- 172.21.15.70:0/792788045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a5f68d500 con 0x563a5f67c490 2024-04-01T01:45:00.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.040+0000 7f96c196d640 1 --2- 172.21.15.70:0/792788045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f67c490 0x563a5f6899b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f96b4009c20 tx=0x7f96b4009880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8abdb11c2c78dff3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:00.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.040+0000 7f96a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:45:00.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.040+0000 7f96a4ff9640 1 -- 172.21.15.70:0/792788045 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b4011640 con 0x563a5f67c490 2024-04-01T01:45:00.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.040+0000 7f96c5fbc740 1 -- 172.21.15.70:0/792788045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96b402c110 con 0x563a5f67c490 2024-04-01T01:45:00.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.040+0000 7f96a4ff9640 1 -- 172.21.15.70:0/792788045 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f96b40117e0 con 0x563a5f67c490 2024-04-01T01:45:00.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.041+0000 7f96a4ff9640 1 -- 172.21.15.70:0/792788045 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f96b40119e0 con 0x563a5f67c490 2024-04-01T01:45:00.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.041+0000 7f96c5fbc740 10 client.28714 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:00.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.041+0000 7f96c5fbc740 1 -- 172.21.15.70:0/792788045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563a5f705900 con 0x563a5f67c490 2024-04-01T01:45:00.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.041+0000 7f96a4ff9640 1 -- 172.21.15.70:0/792788045 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96b402c110 con 0x563a5f67c490 2024-04-01T01:45:00.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.041+0000 7f96a4ff9640 1 -- 172.21.15.70:0/792788045 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f96b402a440 con 0x563a5f67c490 2024-04-01T01:45:00.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.042+0000 7f96c5fbc740 10 client.28714 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:00.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.042+0000 7f96c5fbc740 10 client.28714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:00.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.042+0000 7f96c5fbc740 20 client.28714 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:00.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.042+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/792788045 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563a5f705e70 0x563a5f708260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:00.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.042+0000 7f96c5fbc740 4 client.28714 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:45:00.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.042+0000 7f96c5fbc740 1 -- 172.21.15.70:0/792788045 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563a5f5e8a40 con 0x563a5f705e70 2024-04-01T01:45:00.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.043+0000 7f96c096b640 1 --2- 172.21.15.70:0/792788045 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563a5f705e70 0x563a5f708260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:00.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.045+0000 7f96c096b640 1 --2- 172.21.15.70:0/792788045 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563a5f705e70 0x563a5f708260 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=888b1808258361ea server_cookie=47bdaaf8cb51e584 in_seq=0 out_seq=0 2024-04-01T01:45:00.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.045+0000 7f96a4ff9640 10 client.28714 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:00.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.045+0000 7f96a4ff9640 1 -- 172.21.15.70:0/792788045 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563a5f5e8a40 con 0x563a5f705e70 2024-04-01T01:45:00.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.045+0000 7f96a4ff9640 10 client.28714 handle_command_reply: tid=0 2024-04-01T01:45:00.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.045+0000 7f96c5fbc740 1 client.28714 shutdown 2024-04-01T01:45:00.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.045+0000 7f96c5fbc740 1 -- 172.21.15.70:0/792788045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f67c490 msgr2=0x563a5f6899b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:00.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.045+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/792788045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f67c490 0x563a5f6899b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f96b4009c20 tx=0x7f96b4009880 comp rx=0 tx=0).stop 2024-04-01T01:45:00.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.045+0000 7f96c5fbc740 1 -- 172.21.15.70:0/792788045 shutdown_connections 2024-04-01T01:45:00.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.045+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/792788045 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563a5f705e70 0x563a5f708260 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.045+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/792788045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a5f681ff0 0x563a5f68cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.045+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/792788045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a5f67ce80 0x563a5f689ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.045+0000 7f96c5fbc740 1 --2- 172.21.15.70:0/792788045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a5f67c490 0x563a5f6899b0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.045+0000 7f96a4ff9640 0 client.28714 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:00.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.045+0000 7f96c5fbc740 1 -- 172.21.15.70:0/792788045 >> 172.21.15.70:0/792788045 conn(0x563a5f677a00 msgr2=0x563a5f67e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:00.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.046+0000 7f96c5fbc740 1 -- 172.21.15.70:0/792788045 shutdown_connections 2024-04-01T01:45:00.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.046+0000 7f96c5fbc740 1 -- 172.21.15.70:0/792788045 wait complete. 2024-04-01T01:45:00.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.046+0000 7f96c5fbc740 20 client.28714 trim_cache size 0 max 16384 2024-04-01T01:45:00.047 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:45:00.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.047+0000 7f96c33d0640 1 -- 172.21.15.70:0/3913749632 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96a80a24a0 msgr2=0x7f96a80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:00.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.047+0000 7f96c33d0640 1 --2- 172.21.15.70:0/3913749632 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96a80a24a0 0x7f96a80a4960 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f96b80cc860 tx=0x7f96b80d28a0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.047+0000 7f96c33d0640 1 -- 172.21.15.70:0/3913749632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96bc1021f0 msgr2=0x7f96bc101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:00.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.047+0000 7f96c33d0640 1 --2- 172.21.15.70:0/3913749632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96bc1021f0 0x7f96bc101b60 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f96b800e9f0 tx=0x7f96b800eeb0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.048+0000 7f96c33d0640 1 -- 172.21.15.70:0/3913749632 shutdown_connections 2024-04-01T01:45:00.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.048+0000 7f96c33d0640 1 --2- 172.21.15.70:0/3913749632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96bc1006e0 0x7f96bc100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.048+0000 7f96c33d0640 1 --2- 172.21.15.70:0/3913749632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f96bc107450 0x7f96bc1001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.048+0000 7f96c33d0640 1 --2- 172.21.15.70:0/3913749632 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f96a80a24a0 0x7f96a80a4960 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.048+0000 7f96c33d0640 1 --2- 172.21.15.70:0/3913749632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96bc1021f0 0x7f96bc101b60 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:00.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.048+0000 7f96c33d0640 1 -- 172.21.15.70:0/3913749632 >> 172.21.15.70:0/3913749632 conn(0x7f96bc0fbc00 msgr2=0x7f96bc1055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:00.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.048+0000 7f96c33d0640 1 -- 172.21.15.70:0/3913749632 shutdown_connections 2024-04-01T01:45:00.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:00.048+0000 7f96c33d0640 1 -- 172.21.15.70:0/3913749632 wait complete. 2024-04-01T01:45:00.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:44:59 smithi189 ceph-mon[35753]: pgmap v704: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 9.6 MiB/s wr, 826 op/s 2024-04-01T01:45:00.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:44:59 smithi081 ceph-mon[35142]: pgmap v704: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 9.6 MiB/s wr, 826 op/s 2024-04-01T01:45:01.070 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:45:01.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:45:01.070 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:45:01.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:00 smithi189 ceph-mon[35753]: pgmap v705: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 9.0 MiB/s wr, 768 op/s 2024-04-01T01:45:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:00 smithi070 ceph-mon[32179]: pgmap v705: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 9.0 MiB/s wr, 768 op/s 2024-04-01T01:45:01.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.272+0000 7f6ae8854640 1 -- 172.21.15.70:0/2580990572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6adc0a6710 msgr2=0x7f6adc0aafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.272+0000 7f6ae8854640 1 --2- 172.21.15.70:0/2580990572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6adc0a6710 0x7f6adc0aafd0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4067420 tx=0x7f6ae408f450 comp rx=0 tx=0).stop 2024-04-01T01:45:01.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.272+0000 7f6ae8854640 1 -- 172.21.15.70:0/2580990572 shutdown_connections 2024-04-01T01:45:01.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.272+0000 7f6ae8854640 1 --2- 172.21.15.70:0/2580990572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6adc0a6710 0x7f6adc0aafd0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.272+0000 7f6ae8854640 1 --2- 172.21.15.70:0/2580990572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6adc0a5d70 0x7f6adc0a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.272+0000 7f6ae8854640 1 --2- 172.21.15.70:0/2580990572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6adc0a4c40 0x7f6adc0a5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.272+0000 7f6ae8854640 1 -- 172.21.15.70:0/2580990572 >> 172.21.15.70:0/2580990572 conn(0x7f6adc0a00d0 msgr2=0x7f6adc0a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:01.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.272+0000 7f6ae8854640 1 -- 172.21.15.70:0/2580990572 shutdown_connections 2024-04-01T01:45:01.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.272+0000 7f6ae8854640 1 -- 172.21.15.70:0/2580990572 wait complete. 2024-04-01T01:45:01.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.273+0000 7f6ae8854640 1 Processor -- start 2024-04-01T01:45:01.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.273+0000 7f6ae8854640 1 -- start start 2024-04-01T01:45:01.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.274+0000 7f6ae8854640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6adc0a4c40 0x7f6adc13ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.274+0000 7f6ae8854640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6adc0a5d70 0x7f6adc13b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.274+0000 7f6ae8854640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6adc0a6710 0x7f6adc142230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.274+0000 7f6ae8854640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc0ae700 con 0x7f6adc0a6710 2024-04-01T01:45:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.274+0000 7f6ae8854640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc0ae580 con 0x7f6adc0a4c40 2024-04-01T01:45:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.274+0000 7f6ae8854640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc0ae880 con 0x7f6adc0a5d70 2024-04-01T01:45:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.274+0000 7f6ae2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6adc0a4c40 0x7f6adc13ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.274+0000 7f6ae359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6adc0a6710 0x7f6adc142230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.274+0000 7f6ae359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6adc0a6710 0x7f6adc142230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54388/0 (socket says 172.21.15.70:54388) 2024-04-01T01:45:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.274+0000 7f6ae359e640 1 -- 172.21.15.70:0/2490927697 learned_addr learned my addr 172.21.15.70:0/2490927697 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.274+0000 7f6ae259c640 1 --2- 172.21.15.70:0/2490927697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6adc0a5d70 0x7f6adc13b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.275+0000 7f6ae2d9d640 1 -- 172.21.15.70:0/2490927697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6adc0a5d70 msgr2=0x7f6adc13b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.275+0000 7f6ae2d9d640 1 --2- 172.21.15.70:0/2490927697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6adc0a5d70 0x7f6adc13b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.275+0000 7f6ae2d9d640 1 -- 172.21.15.70:0/2490927697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6adc0a6710 msgr2=0x7f6adc142230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.275+0000 7f6ae2d9d640 1 --2- 172.21.15.70:0/2490927697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6adc0a6710 0x7f6adc142230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.275+0000 7f6ae2d9d640 1 -- 172.21.15.70:0/2490927697 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6adc1428d0 con 0x7f6adc0a4c40 2024-04-01T01:45:01.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.275+0000 7f6ae259c640 1 --2- 172.21.15.70:0/2490927697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6adc0a5d70 0x7f6adc13b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:45:01.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.275+0000 7f6ae2d9d640 1 --2- 172.21.15.70:0/2490927697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6adc0a4c40 0x7f6adc13ac90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f6ad401e990 tx=0x7f6ad401ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:01.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.276+0000 7f6acbfff640 1 -- 172.21.15.70:0/2490927697 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4028070 con 0x7f6adc0a4c40 2024-04-01T01:45:01.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.276+0000 7f6acbfff640 1 -- 172.21.15.70:0/2490927697 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ad4021470 con 0x7f6adc0a4c40 2024-04-01T01:45:01.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.276+0000 7f6acbfff640 1 -- 172.21.15.70:0/2490927697 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4020630 con 0x7f6adc0a4c40 2024-04-01T01:45:01.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.276+0000 7f6ae8854640 1 -- 172.21.15.70:0/2490927697 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6adc142bc0 con 0x7f6adc0a4c40 2024-04-01T01:45:01.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.277+0000 7f6ae8854640 1 -- 172.21.15.70:0/2490927697 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6adc1430c0 con 0x7f6adc0a4c40 2024-04-01T01:45:01.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.280+0000 7f6acbfff640 1 -- 172.21.15.70:0/2490927697 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f6ad402b070 con 0x7f6adc0a4c40 2024-04-01T01:45:01.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.281+0000 7f6acbfff640 1 --2- 172.21.15.70:0/2490927697 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6acc0a26a0 0x7f6acc0a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.281+0000 7f6acbfff640 1 -- 172.21.15.70:0/2490927697 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6ad40dec90 con 0x7f6adc0a4c40 2024-04-01T01:45:01.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.281+0000 7f6aebc3a740 1 Processor -- start 2024-04-01T01:45:01.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.281+0000 7f6ae259c640 1 --2- 172.21.15.70:0/2490927697 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6acc0a26a0 0x7f6acc0a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.282+0000 7f6aebc3a740 1 -- start start 2024-04-01T01:45:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.282+0000 7f6aebc3a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907c8bf0 0x5587907c9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.282+0000 7f6aebc3a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c95e0 0x5587907ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.282+0000 7f6aebc3a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 0x5587907d0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.282+0000 7f6aebc3a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5587906ca0b0 con 0x5587907c95e0 2024-04-01T01:45:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.282+0000 7f6aebc3a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55879072cea0 con 0x5587907ce750 2024-04-01T01:45:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.282+0000 7f6aebc3a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5587906d5c00 con 0x5587907c8bf0 2024-04-01T01:45:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.282+0000 7f6ae359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 0x5587907d0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.282+0000 7f6ae359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 0x5587907d0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42676/0 (socket says 172.21.15.70:42676) 2024-04-01T01:45:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.282+0000 7f6ae359e640 1 -- 172.21.15.70:0/61352 learned_addr learned my addr 172.21.15.70:0/61352 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:01.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.282+0000 7f6ae259c640 1 --2- 172.21.15.70:0/61352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c95e0 0x5587907ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.283+0000 7f6ae359e640 1 -- 172.21.15.70:0/61352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907c8bf0 msgr2=0x5587907c9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:01.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.283+0000 7f6ae359e640 1 --2- 172.21.15.70:0/61352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907c8bf0 0x5587907c9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.283+0000 7f6ae359e640 1 -- 172.21.15.70:0/61352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c95e0 msgr2=0x5587907ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:01.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.283+0000 7f6ae359e640 1 --2- 172.21.15.70:0/61352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c95e0 0x5587907ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.283+0000 7f6ae359e640 1 -- 172.21.15.70:0/61352 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587905097e0 con 0x5587907ce750 2024-04-01T01:45:01.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.283+0000 7f6ae259c640 1 --2- 172.21.15.70:0/2490927697 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6acc0a26a0 0x7f6acc0a4b60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6adc13c190 tx=0x7f6ad8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:01.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.283+0000 7f6ae359e640 1 --2- 172.21.15.70:0/61352 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 0x5587907d0be0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6ae40966a0 tx=0x7f6ae4067cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ec3e6ca63bb19b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:01.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.283+0000 7f6ac97fa640 1 -- 172.21.15.70:0/61352 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae40a8020 con 0x5587907ce750 2024-04-01T01:45:01.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.283+0000 7f6ac97fa640 1 -- 172.21.15.70:0/61352 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ae409d040 con 0x5587907ce750 2024-04-01T01:45:01.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.283+0000 7f6ac97fa640 1 -- 172.21.15.70:0/61352 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae4069760 con 0x5587907ce750 2024-04-01T01:45:01.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.284+0000 7f6aebc3a740 1 -- 172.21.15.70:0/61352 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 msgr2=0x5587907d0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.284+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/61352 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 0x5587907d0be0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6ae40966a0 tx=0x7f6ae4067cf0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.284+0000 7f6aebc3a740 1 -- 172.21.15.70:0/61352 shutdown_connections 2024-04-01T01:45:01.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.284+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/61352 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 0x5587907d0be0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.285+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/61352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c95e0 0x5587907ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.285+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/61352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907c8bf0 0x5587907c9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.285+0000 7f6aebc3a740 1 -- 172.21.15.70:0/61352 >> 172.21.15.70:0/61352 conn(0x5587907c4160 msgr2=0x5587907c4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:01.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.285+0000 7f6aebc3a740 1 -- 172.21.15.70:0/61352 shutdown_connections 2024-04-01T01:45:01.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.285+0000 7f6aebc3a740 1 -- 172.21.15.70:0/61352 wait complete. 2024-04-01T01:45:01.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.285+0000 7f6aebc3a740 1 -- start start 2024-04-01T01:45:01.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.286+0000 7f6aebc3a740 1 Processor -- start 2024-04-01T01:45:01.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.286+0000 7f6aebc3a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907c8bf0 0x5587907d2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.286+0000 7f6aebc3a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c95e0 0x5587907d3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.286+0000 7f6aebc3a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 0x5587907da570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.286+0000 7f6aebc3a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5587906cb3d0 con 0x5587907c95e0 2024-04-01T01:45:01.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.286+0000 7f6aebc3a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x558790728f40 con 0x5587907ce750 2024-04-01T01:45:01.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.286+0000 7f6aebc3a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5587907d16a0 con 0x5587907c8bf0 2024-04-01T01:45:01.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.286+0000 7f6aebc3a740 10 client.0 _ll_register_callbacks cb 0x5587906e8b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:01.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.286+0000 7f6ae359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 0x5587907da570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.286+0000 7f6ae359e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 0x5587907da570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42684/0 (socket says 172.21.15.70:42684) 2024-04-01T01:45:01.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.286+0000 7f6ae359e640 1 -- 172.21.15.70:0/257055298 learned_addr learned my addr 172.21.15.70:0/257055298 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:01.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.287+0000 7f6ae259c640 1 --2- 172.21.15.70:0/257055298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c95e0 0x5587907d3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.287+0000 7f6ae2d9d640 1 --2- 172.21.15.70:0/257055298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907c8bf0 0x5587907d2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.287+0000 7f6ae259c640 1 -- 172.21.15.70:0/257055298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907c8bf0 msgr2=0x5587907d2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:01.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.287+0000 7f6ae259c640 1 --2- 172.21.15.70:0/257055298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907c8bf0 0x5587907d2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.287+0000 7f6ae259c640 1 -- 172.21.15.70:0/257055298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 msgr2=0x5587907da570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:01.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.287+0000 7f6ae259c640 1 --2- 172.21.15.70:0/257055298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 0x5587907da570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.287+0000 7f6ae259c640 1 -- 172.21.15.70:0/257055298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558790569f70 con 0x5587907c95e0 2024-04-01T01:45:01.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.289+0000 7f6ae259c640 1 --2- 172.21.15.70:0/257055298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c95e0 0x5587907d3530 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x5587907c99e0 tx=0x7f6ad8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d907b4f7ae8fc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:01.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.289+0000 7f6ac6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:45:01.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.290+0000 7f6ac6ff5640 1 -- 172.21.15.70:0/257055298 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad802a060 con 0x5587907c95e0 2024-04-01T01:45:01.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.290+0000 7f6ac6ff5640 1 -- 172.21.15.70:0/257055298 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6ad800f810 con 0x5587907c95e0 2024-04-01T01:45:01.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.290+0000 7f6aebc3a740 1 -- 172.21.15.70:0/257055298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ae409d040 con 0x5587907c95e0 2024-04-01T01:45:01.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.291+0000 7f6ac6ff5640 1 -- 172.21.15.70:0/257055298 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6ad800fa10 con 0x5587907c95e0 2024-04-01T01:45:01.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.291+0000 7f6aebc3a740 10 client.18879 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:01.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.291+0000 7f6aebc3a740 1 -- 172.21.15.70:0/257055298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558790401170 con 0x5587907c95e0 2024-04-01T01:45:01.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.292+0000 7f6ac6ff5640 1 -- 172.21.15.70:0/257055298 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ae409d040 con 0x5587907c95e0 2024-04-01T01:45:01.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.292+0000 7f6ac6ff5640 1 -- 172.21.15.70:0/257055298 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6ad801b070 con 0x5587907c95e0 2024-04-01T01:45:01.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.292+0000 7f6aebc3a740 10 client.18879 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:01.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.292+0000 7f6aebc3a740 10 client.18879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:01.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.292+0000 7f6aebc3a740 20 client.18879 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:01.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.292+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/257055298 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5587907deb60 0x5587907e0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.292+0000 7f6aebc3a740 4 client.18879 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:45:01.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.292+0000 7f6aebc3a740 1 -- 172.21.15.70:0/257055298 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55879073a3e0 con 0x5587907deb60 2024-04-01T01:45:01.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.293+0000 7f6ae2d9d640 1 --2- 172.21.15.70:0/257055298 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5587907deb60 0x5587907e0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.294+0000 7f6ae2d9d640 1 --2- 172.21.15.70:0/257055298 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5587907deb60 0x5587907e0f80 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4f1aadbb84d6283 server_cookie=9decee98e497b0c3 in_seq=0 out_seq=0 2024-04-01T01:45:01.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.294+0000 7f6ac6ff5640 10 client.18879 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:01.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.295+0000 7f6ac6ff5640 1 -- 172.21.15.70:0/257055298 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55879073a3e0 con 0x5587907deb60 2024-04-01T01:45:01.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.295+0000 7f6ac6ff5640 10 client.18879 handle_command_reply: tid=0 2024-04-01T01:45:01.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.295+0000 7f6aebc3a740 1 client.18879 shutdown 2024-04-01T01:45:01.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.296+0000 7f6aebc3a740 1 -- 172.21.15.70:0/257055298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c95e0 msgr2=0x5587907d3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.296+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/257055298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c95e0 0x5587907d3530 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x5587907c99e0 tx=0x7f6ad8004730 comp rx=0 tx=0).stop 2024-04-01T01:45:01.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.296+0000 7f6aebc3a740 1 -- 172.21.15.70:0/257055298 shutdown_connections 2024-04-01T01:45:01.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.296+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/257055298 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5587907deb60 0x5587907e0f80 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.296+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/257055298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 0x5587907da570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.296+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/257055298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c95e0 0x5587907d3530 secure :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0x5587907c99e0 tx=0x7f6ad8004730 comp rx=0 tx=0).stop 2024-04-01T01:45:01.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.296+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/257055298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907c8bf0 0x5587907d2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.296+0000 7f6ac6ff5640 0 client.18879 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:01.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.296+0000 7f6aebc3a740 1 -- 172.21.15.70:0/257055298 >> 172.21.15.70:0/257055298 conn(0x5587907c4160 msgr2=0x5587907c32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:01.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.296+0000 7f6aebc3a740 1 -- 172.21.15.70:0/257055298 shutdown_connections 2024-04-01T01:45:01.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.296+0000 7f6aebc3a740 1 -- 172.21.15.70:0/257055298 wait complete. 2024-04-01T01:45:01.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.297+0000 7f6aebc3a740 20 client.18879 trim_cache size 0 max 16384 2024-04-01T01:45:01.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:00 smithi081 ceph-mon[35142]: pgmap v705: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 9.0 MiB/s wr, 768 op/s 2024-04-01T01:45:01.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.337+0000 7f6aebc3a740 1 Processor -- start 2024-04-01T01:45:01.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.337+0000 7f6aebc3a740 1 -- start start 2024-04-01T01:45:01.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.337+0000 7f6aebc3a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907c8bf0 0x5587907d29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.337+0000 7f6aebc3a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 0x5587907da530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.337+0000 7f6aebc3a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c2a20 0x5587907c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.337+0000 7f6aebc3a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc14d830 con 0x5587907c2a20 2024-04-01T01:45:01.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.337+0000 7f6aebc3a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc14f370 con 0x5587907ce750 2024-04-01T01:45:01.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.337+0000 7f6aebc3a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc14df00 con 0x5587907c8bf0 2024-04-01T01:45:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.338+0000 7f6ae359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c2a20 0x5587907c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.338+0000 7f6ae359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c2a20 0x5587907c2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54416/0 (socket says 172.21.15.70:54416) 2024-04-01T01:45:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.338+0000 7f6ae359e640 1 -- 172.21.15.70:0/1050556886 learned_addr learned my addr 172.21.15.70:0/1050556886 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.338+0000 7f6ae259c640 1 --2- 172.21.15.70:0/1050556886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 0x5587907da530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.339+0000 7f6ae259c640 1 -- 172.21.15.70:0/1050556886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907c8bf0 msgr2=0x5587907d29f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:45:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.339+0000 7f6ae259c640 1 --2- 172.21.15.70:0/1050556886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907c8bf0 0x5587907d29f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.339+0000 7f6ae259c640 1 -- 172.21.15.70:0/1050556886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c2a20 msgr2=0x5587907c2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.339+0000 7f6ae259c640 1 --2- 172.21.15.70:0/1050556886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c2a20 0x5587907c2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.339+0000 7f6ae259c640 1 -- 172.21.15.70:0/1050556886 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587907dd490 con 0x5587907ce750 2024-04-01T01:45:01.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.339+0000 7f6ae259c640 1 --2- 172.21.15.70:0/1050556886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 0x5587907da530 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6ad801d000 tx=0x7f6ad80059b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a34d273aaf62fb1b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.339+0000 7f6ac8ff9640 1 -- 172.21.15.70:0/1050556886 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad802a070 con 0x5587907ce750 2024-04-01T01:45:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.339+0000 7f6ac8ff9640 1 -- 172.21.15.70:0/1050556886 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ad800f810 con 0x5587907ce750 2024-04-01T01:45:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.340+0000 7f6ac8ff9640 1 -- 172.21.15.70:0/1050556886 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8034080 con 0x5587907ce750 2024-04-01T01:45:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.340+0000 7f6aebc3a740 1 -- 172.21.15.70:0/1050556886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 msgr2=0x5587907da530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.340+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/1050556886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 0x5587907da530 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6ad801d000 tx=0x7f6ad80059b0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.341+0000 7f6aebc3a740 1 -- 172.21.15.70:0/1050556886 shutdown_connections 2024-04-01T01:45:01.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.341+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/1050556886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c2a20 0x5587907c2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.341+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/1050556886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907ce750 0x5587907da530 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.341+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/1050556886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907c8bf0 0x5587907d29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.341+0000 7f6aebc3a740 1 -- 172.21.15.70:0/1050556886 >> 172.21.15.70:0/1050556886 conn(0x5587907c4160 msgr2=0x5587907d22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:01.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.341+0000 7f6aebc3a740 1 -- 172.21.15.70:0/1050556886 shutdown_connections 2024-04-01T01:45:01.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.341+0000 7f6aebc3a740 1 -- 172.21.15.70:0/1050556886 wait complete. 2024-04-01T01:45:01.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.342+0000 7f6aebc3a740 1 -- start start 2024-04-01T01:45:01.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.342+0000 7f6aebc3a740 1 Processor -- start 2024-04-01T01:45:01.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.342+0000 7f6aebc3a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907c2a20 0x5587907da1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.342+0000 7f6aebc3a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c8bf0 0x5587907d8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.342+0000 7f6aebc3a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907ce750 0x5587907d8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.342+0000 7f6aebc3a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc14eca0 con 0x5587907c8bf0 2024-04-01T01:45:01.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.342+0000 7f6aebc3a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc14e5d0 con 0x5587907c2a20 2024-04-01T01:45:01.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.342+0000 7f6aebc3a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc14b810 con 0x5587907ce750 2024-04-01T01:45:01.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.342+0000 7f6aebc3a740 10 client.0 _ll_register_callbacks cb 0x7f6ad8004550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:01.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.344+0000 7f6ae2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907c2a20 0x5587907da1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.344+0000 7f6ae2d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907c2a20 0x5587907da1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42710/0 (socket says 172.21.15.70:42710) 2024-04-01T01:45:01.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.344+0000 7f6ae2d9d640 1 -- 172.21.15.70:0/3858188155 learned_addr learned my addr 172.21.15.70:0/3858188155 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:01.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.344+0000 7f6ae359e640 1 --2- 172.21.15.70:0/3858188155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907ce750 0x5587907d8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.346+0000 7f6ae259c640 1 --2- 172.21.15.70:0/3858188155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c8bf0 0x5587907d8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.347+0000 7f6ae259c640 1 -- 172.21.15.70:0/3858188155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907ce750 msgr2=0x5587907d8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:01.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.347+0000 7f6ae259c640 1 --2- 172.21.15.70:0/3858188155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907ce750 0x5587907d8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.347+0000 7f6ae259c640 1 -- 172.21.15.70:0/3858188155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907c2a20 msgr2=0x5587907da1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:01.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.347+0000 7f6ae259c640 1 --2- 172.21.15.70:0/3858188155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907c2a20 0x5587907da1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.347+0000 7f6ae259c640 1 -- 172.21.15.70:0/3858188155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587907d92d0 con 0x5587907c8bf0 2024-04-01T01:45:01.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.347+0000 7f6ae259c640 1 --2- 172.21.15.70:0/3858188155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c8bf0 0x5587907d8850 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6ad801d000 tx=0x7f6ad800ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cea8124fbdff511 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:01.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.347+0000 7f6ac6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:45:01.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.347+0000 7f6ac6ff5640 1 -- 172.21.15.70:0/3858188155 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad803d060 con 0x5587907c8bf0 2024-04-01T01:45:01.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.348+0000 7f6aebc3a740 1 -- 172.21.15.70:0/3858188155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ad800f810 con 0x5587907c8bf0 2024-04-01T01:45:01.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.348+0000 7f6ac6ff5640 1 -- 172.21.15.70:0/3858188155 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6ad8005130 con 0x5587907c8bf0 2024-04-01T01:45:01.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.348+0000 7f6ac6ff5640 1 -- 172.21.15.70:0/3858188155 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6ad8044050 con 0x5587907c8bf0 2024-04-01T01:45:01.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.348+0000 7f6aebc3a740 10 client.18885 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:01.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.348+0000 7f6aebc3a740 1 -- 172.21.15.70:0/3858188155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558790845bb0 con 0x5587907c8bf0 2024-04-01T01:45:01.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.348+0000 7f6ac6ff5640 1 -- 172.21.15.70:0/3858188155 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ad800f810 con 0x5587907c8bf0 2024-04-01T01:45:01.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.349+0000 7f6ac6ff5640 1 -- 172.21.15.70:0/3858188155 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6ad801c550 con 0x5587907c8bf0 2024-04-01T01:45:01.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.349+0000 7f6aebc3a740 10 client.18885 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:01.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.349+0000 7f6aebc3a740 10 client.18885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:01.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.349+0000 7f6aebc3a740 20 client.18885 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:01.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.349+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/3858188155 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5587908464a0 0x5587908488c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.349+0000 7f6aebc3a740 4 client.18885 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:45:01.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.349+0000 7f6aebc3a740 1 -- 172.21.15.70:0/3858188155 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558790734730 con 0x5587908464a0 2024-04-01T01:45:01.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.349+0000 7f6ae2d9d640 1 --2- 172.21.15.70:0/3858188155 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5587908464a0 0x5587908488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.350+0000 7f6ae2d9d640 1 --2- 172.21.15.70:0/3858188155 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5587908464a0 0x5587908488c0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=889ddc05453a801f server_cookie=c75b3997ceda8945 in_seq=0 out_seq=0 2024-04-01T01:45:01.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.350+0000 7f6ac6ff5640 10 client.18885 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:01.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.352+0000 7f6ac6ff5640 1 -- 172.21.15.70:0/3858188155 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558790734730 con 0x5587908464a0 2024-04-01T01:45:01.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.352+0000 7f6ac6ff5640 10 client.18885 handle_command_reply: tid=0 2024-04-01T01:45:01.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.352+0000 7f6aebc3a740 1 client.18885 shutdown 2024-04-01T01:45:01.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.353+0000 7f6aebc3a740 1 -- 172.21.15.70:0/3858188155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c8bf0 msgr2=0x5587907d8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.353+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/3858188155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c8bf0 0x5587907d8850 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6ad801d000 tx=0x7f6ad800ffd0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.355+0000 7f6aebc3a740 1 -- 172.21.15.70:0/3858188155 shutdown_connections 2024-04-01T01:45:01.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.355+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/3858188155 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5587908464a0 0x5587908488c0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.355+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/3858188155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5587907ce750 0x5587907d8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.355+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/3858188155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5587907c8bf0 0x5587907d8850 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.355+0000 7f6aebc3a740 1 --2- 172.21.15.70:0/3858188155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5587907c2a20 0x5587907da1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.355+0000 7f6aebc3a740 1 -- 172.21.15.70:0/3858188155 >> 172.21.15.70:0/3858188155 conn(0x5587907c4160 msgr2=0x5587907d22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:01.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.355+0000 7f6ac6ff5640 0 client.18885 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:01.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.356+0000 7f6aebc3a740 1 -- 172.21.15.70:0/3858188155 shutdown_connections 2024-04-01T01:45:01.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.357+0000 7f6aebc3a740 1 -- 172.21.15.70:0/3858188155 wait complete. 2024-04-01T01:45:01.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.357+0000 7f6aebc3a740 20 client.18885 trim_cache size 0 max 16384 2024-04-01T01:45:01.357 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:45:01.357 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:45:01.357 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "7becb7aa-fb2c-404f-8946-38eb43b1fcdc", 2024-04-01T01:45:01.358 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:45:01.358 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:45:01.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.358+0000 7f6ae8854640 1 -- 172.21.15.70:0/2490927697 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6acc0a26a0 msgr2=0x7f6acc0a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.359+0000 7f6ae8854640 1 --2- 172.21.15.70:0/2490927697 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6acc0a26a0 0x7f6acc0a4b60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6adc13c190 tx=0x7f6ad8005f90 comp rx=0 tx=0).stop 2024-04-01T01:45:01.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.359+0000 7f6ae8854640 1 -- 172.21.15.70:0/2490927697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6adc0a4c40 msgr2=0x7f6adc13ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.359+0000 7f6ae8854640 1 --2- 172.21.15.70:0/2490927697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6adc0a4c40 0x7f6adc13ac90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f6ad401e990 tx=0x7f6ad401ee50 comp rx=0 tx=0).stop 2024-04-01T01:45:01.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.360+0000 7f6ae8854640 1 -- 172.21.15.70:0/2490927697 shutdown_connections 2024-04-01T01:45:01.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.360+0000 7f6ae8854640 1 --2- 172.21.15.70:0/2490927697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6adc0a6710 0x7f6adc142230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.360+0000 7f6ae8854640 1 --2- 172.21.15.70:0/2490927697 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6acc0a26a0 0x7f6acc0a4b60 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.360+0000 7f6ae8854640 1 --2- 172.21.15.70:0/2490927697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6adc0a5d70 0x7f6adc13b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.360+0000 7f6ae8854640 1 --2- 172.21.15.70:0/2490927697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6adc0a4c40 0x7f6adc13ac90 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.360+0000 7f6ae8854640 1 -- 172.21.15.70:0/2490927697 >> 172.21.15.70:0/2490927697 conn(0x7f6adc0a00d0 msgr2=0x7f6adc0a9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:01.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.360+0000 7f6ae8854640 1 -- 172.21.15.70:0/2490927697 shutdown_connections 2024-04-01T01:45:01.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.360+0000 7f6ae8854640 1 -- 172.21.15.70:0/2490927697 wait complete. 2024-04-01T01:45:01.383 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:45:01.384 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:45:01.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.564+0000 7fd05e8ca640 1 -- 172.21.15.70:0/166215748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd05009d560 msgr2=0x7fd05009d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.564+0000 7fd05e8ca640 1 --2- 172.21.15.70:0/166215748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd05009d560 0x7fd05009d9e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd048009900 tx=0x7fd04802bc60 comp rx=0 tx=0).stop 2024-04-01T01:45:01.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.565+0000 7fd05e8ca640 1 -- 172.21.15.70:0/166215748 shutdown_connections 2024-04-01T01:45:01.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.565+0000 7fd05e8ca640 1 --2- 172.21.15.70:0/166215748 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd05009df20 0x7fd0500a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.565+0000 7fd05e8ca640 1 --2- 172.21.15.70:0/166215748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd05009d560 0x7fd05009d9e0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.565+0000 7fd05e8ca640 1 --2- 172.21.15.70:0/166215748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05009c370 0x7fd05009c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.565+0000 7fd05e8ca640 1 -- 172.21.15.70:0/166215748 >> 172.21.15.70:0/166215748 conn(0x7fd05000b4d0 msgr2=0x7fd05009a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:01.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.565+0000 7fd05e8ca640 1 -- 172.21.15.70:0/166215748 shutdown_connections 2024-04-01T01:45:01.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.565+0000 7fd05e8ca640 1 -- 172.21.15.70:0/166215748 wait complete. 2024-04-01T01:45:01.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.566+0000 7fd05e8ca640 1 Processor -- start 2024-04-01T01:45:01.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.567+0000 7fd05e8ca640 1 -- start start 2024-04-01T01:45:01.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.567+0000 7fd05e8ca640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd05009c370 0x7fd050013c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.567+0000 7fd05e8ca640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05009d560 0x7fd0500141a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.567+0000 7fd05e8ca640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd05009df20 0x7fd0500151e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.567+0000 7fd05e8ca640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd05000e4b0 con 0x7fd05009df20 2024-04-01T01:45:01.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.567+0000 7fd05e8ca640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd05000e330 con 0x7fd05009c370 2024-04-01T01:45:01.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.567+0000 7fd05e8ca640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd05000e630 con 0x7fd05009d560 2024-04-01T01:45:01.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.567+0000 7fd05d8c8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd05009c370 0x7fd050013c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.567+0000 7fd05d8c8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd05009c370 0x7fd050013c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42738/0 (socket says 172.21.15.70:42738) 2024-04-01T01:45:01.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.567+0000 7fd05d8c8640 1 -- 172.21.15.70:0/2144389694 learned_addr learned my addr 172.21.15.70:0/2144389694 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:01.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.567+0000 7fd05d0c7640 1 --2- 172.21.15.70:0/2144389694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05009d560 0x7fd0500141a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.568+0000 7fd05e0c9640 1 --2- 172.21.15.70:0/2144389694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd05009df20 0x7fd0500151e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.568+0000 7fd05d8c8640 1 -- 172.21.15.70:0/2144389694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05009d560 msgr2=0x7fd0500141a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.568+0000 7fd05d8c8640 1 --2- 172.21.15.70:0/2144389694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05009d560 0x7fd0500141a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.568+0000 7fd05d8c8640 1 -- 172.21.15.70:0/2144389694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd05009df20 msgr2=0x7fd0500151e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.568+0000 7fd05d8c8640 1 --2- 172.21.15.70:0/2144389694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd05009df20 0x7fd0500151e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.568+0000 7fd05d8c8640 1 -- 172.21.15.70:0/2144389694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd050015850 con 0x7fd05009c370 2024-04-01T01:45:01.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.568+0000 7fd05e0c9640 1 --2- 172.21.15.70:0/2144389694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd05009df20 0x7fd0500151e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:45:01.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.568+0000 7fd05d8c8640 1 --2- 172.21.15.70:0/2144389694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd05009c370 0x7fd050013c60 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd05400d770 tx=0x7fd05400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:01.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.569+0000 7fd04effd640 1 -- 172.21.15.70:0/2144389694 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054017070 con 0x7fd05009c370 2024-04-01T01:45:01.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.569+0000 7fd04effd640 1 -- 172.21.15.70:0/2144389694 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd054009c30 con 0x7fd05009c370 2024-04-01T01:45:01.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.569+0000 7fd04effd640 1 -- 172.21.15.70:0/2144389694 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054003c90 con 0x7fd05009c370 2024-04-01T01:45:01.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.569+0000 7fd05e8ca640 1 -- 172.21.15.70:0/2144389694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd050015b40 con 0x7fd05009c370 2024-04-01T01:45:01.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.569+0000 7fd05e8ca640 1 -- 172.21.15.70:0/2144389694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd050016090 con 0x7fd05009c370 2024-04-01T01:45:01.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.572+0000 7fd0627ee740 1 Processor -- start 2024-04-01T01:45:01.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.573+0000 7fd0627ee740 1 -- start start 2024-04-01T01:45:01.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.573+0000 7fd0627ee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208490 0x5622dc2088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.573+0000 7fd0627ee740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc208e80 0x5622dc20d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.573+0000 7fd0627ee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc20dff0 0x5622dc210480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.573+0000 7fd05e0c9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208490 0x5622dc2088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.573+0000 7fd05e0c9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208490 0x5622dc2088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42752/0 (socket says 172.21.15.70:42752) 2024-04-01T01:45:01.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.573+0000 7fd05e0c9640 1 -- 172.21.15.70:0/92776513 learned_addr learned my addr 172.21.15.70:0/92776513 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:01.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.573+0000 7fd0627ee740 1 -- 172.21.15.70:0/92776513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5622dc16d180 con 0x5622dc208e80 2024-04-01T01:45:01.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.573+0000 7fd0627ee740 1 -- 172.21.15.70:0/92776513 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5622dc115b70 con 0x5622dc208490 2024-04-01T01:45:01.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.573+0000 7fd0627ee740 1 -- 172.21.15.70:0/92776513 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5622dc169220 con 0x5622dc20dff0 2024-04-01T01:45:01.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.574+0000 7fd05d0c7640 1 --2- 172.21.15.70:0/92776513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc20dff0 0x5622dc210480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.574+0000 7fd04effd640 1 -- 172.21.15.70:0/2144389694 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fd054004020 con 0x7fd05009c370 2024-04-01T01:45:01.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.574+0000 7fd05d8c8640 1 --2- 172.21.15.70:0/92776513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc208e80 0x5622dc20d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.575+0000 7fd04effd640 1 --2- 172.21.15.70:0/2144389694 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0440a26a0 0x7fd0440a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.575+0000 7fd05d0c7640 1 -- 172.21.15.70:0/92776513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208490 msgr2=0x5622dc2088b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:01.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.575+0000 7fd05d0c7640 1 --2- 172.21.15.70:0/92776513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208490 0x5622dc2088b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.575+0000 7fd05d0c7640 1 -- 172.21.15.70:0/92776513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc208e80 msgr2=0x5622dc20d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:01.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.575+0000 7fd05d0c7640 1 --2- 172.21.15.70:0/92776513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc208e80 0x5622dc20d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.575+0000 7fd05d0c7640 1 -- 172.21.15.70:0/92776513 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622dc1297c0 con 0x5622dc20dff0 2024-04-01T01:45:01.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.575+0000 7fd04effd640 1 -- 172.21.15.70:0/2144389694 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd0540ce3d0 con 0x7fd05009c370 2024-04-01T01:45:01.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.575+0000 7fd05e0c9640 1 --2- 172.21.15.70:0/2144389694 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0440a26a0 0x7fd0440a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.576+0000 7fd05e0c9640 1 --2- 172.21.15.70:0/2144389694 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0440a26a0 0x7fd0440a4b60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x5622dc2096e0 tx=0x7fd0580675f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:01.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.579+0000 7fd05d0c7640 1 --2- 172.21.15.70:0/92776513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc20dff0 0x5622dc210480 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd048009850 tx=0x7fd048009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5e3b250b814183e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:01.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.579+0000 7fd043fff640 1 -- 172.21.15.70:0/92776513 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd048045020 con 0x5622dc20dff0 2024-04-01T01:45:01.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.579+0000 7fd043fff640 1 -- 172.21.15.70:0/92776513 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd04802cbd0 con 0x5622dc20dff0 2024-04-01T01:45:01.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.579+0000 7fd043fff640 1 -- 172.21.15.70:0/92776513 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0480046c0 con 0x5622dc20dff0 2024-04-01T01:45:01.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.580+0000 7fd0627ee740 1 -- 172.21.15.70:0/92776513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc20dff0 msgr2=0x5622dc210480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.581+0000 7fd0627ee740 1 --2- 172.21.15.70:0/92776513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc20dff0 0x5622dc210480 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd048009850 tx=0x7fd048009560 comp rx=0 tx=0).stop 2024-04-01T01:45:01.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.581+0000 7fd0627ee740 1 -- 172.21.15.70:0/92776513 shutdown_connections 2024-04-01T01:45:01.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.581+0000 7fd0627ee740 1 --2- 172.21.15.70:0/92776513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc20dff0 0x5622dc210480 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.581+0000 7fd0627ee740 1 --2- 172.21.15.70:0/92776513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc208e80 0x5622dc20d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.581+0000 7fd0627ee740 1 --2- 172.21.15.70:0/92776513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208490 0x5622dc2088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.581+0000 7fd0627ee740 1 -- 172.21.15.70:0/92776513 >> 172.21.15.70:0/92776513 conn(0x5622dc203a00 msgr2=0x5622dc203e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:01.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.581+0000 7fd0627ee740 1 -- 172.21.15.70:0/92776513 shutdown_connections 2024-04-01T01:45:01.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.581+0000 7fd0627ee740 1 -- 172.21.15.70:0/92776513 wait complete. 2024-04-01T01:45:01.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.582+0000 7fd0627ee740 1 -- start start 2024-04-01T01:45:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.582+0000 7fd0627ee740 1 Processor -- start 2024-04-01T01:45:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.582+0000 7fd0627ee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc208490 0x5622dc2127b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.582+0000 7fd0627ee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 0x5622dc212cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.582+0000 7fd0627ee740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc20dff0 0x5622dc219d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.582+0000 7fd0627ee740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5622dc210f40 con 0x5622dc20dff0 2024-04-01T01:45:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.582+0000 7fd0627ee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5622dc10b4a0 con 0x5622dc208e80 2024-04-01T01:45:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.582+0000 7fd0627ee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5622dc2110c0 con 0x5622dc208490 2024-04-01T01:45:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.582+0000 7fd0627ee740 10 client.0 _ll_register_callbacks cb 0x5622dc10ed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:01.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.582+0000 7fd05d8c8640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc208490 0x5622dc2127b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.583+0000 7fd05d8c8640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc208490 0x5622dc2127b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39556/0 (socket says 172.21.15.70:39556) 2024-04-01T01:45:01.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.583+0000 7fd05d8c8640 1 -- 172.21.15.70:0/3191774990 learned_addr learned my addr 172.21.15.70:0/3191774990 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:01.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.583+0000 7fd05d0c7640 1 --2- 172.21.15.70:0/3191774990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 0x5622dc212cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.583+0000 7fd05d8c8640 1 -- 172.21.15.70:0/3191774990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 msgr2=0x5622dc212cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.583+0000 7fd05d8c8640 1 --2- 172.21.15.70:0/3191774990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 0x5622dc212cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.583+0000 7fd05d8c8640 1 -- 172.21.15.70:0/3191774990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc20dff0 msgr2=0x5622dc219d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.583+0000 7fd05d8c8640 1 --2- 172.21.15.70:0/3191774990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc20dff0 0x5622dc219d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.583+0000 7fd05d8c8640 1 -- 172.21.15.70:0/3191774990 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622dbf4b7d0 con 0x5622dc208490 2024-04-01T01:45:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.583+0000 7fd05d0c7640 1 --2- 172.21.15.70:0/3191774990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 0x5622dc212cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:45:01.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.583+0000 7fd05d8c8640 1 --2- 172.21.15.70:0/3191774990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc208490 0x5622dc2127b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd054096ec0 tx=0x7fd05409a830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ee462152c367097 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:01.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.584+0000 7fd0417fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:45:01.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.584+0000 7fd0417fa640 1 -- 172.21.15.70:0/3191774990 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054020040 con 0x5622dc208490 2024-04-01T01:45:01.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.584+0000 7fd0417fa640 1 -- 172.21.15.70:0/3191774990 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd054004310 con 0x5622dc208490 2024-04-01T01:45:01.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.584+0000 7fd0627ee740 1 -- 172.21.15.70:0/3191774990 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd04802cbd0 con 0x5622dc208490 2024-04-01T01:45:01.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.584+0000 7fd0417fa640 1 -- 172.21.15.70:0/3191774990 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd0540035f0 con 0x5622dc208490 2024-04-01T01:45:01.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.585+0000 7fd0627ee740 10 client.28738 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:01.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.585+0000 7fd0627ee740 1 -- 172.21.15.70:0/3191774990 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5622dbe813f0 con 0x5622dc208490 2024-04-01T01:45:01.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.589+0000 7fd0417fa640 1 -- 172.21.15.70:0/3191774990 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd04802cbd0 con 0x5622dc208490 2024-04-01T01:45:01.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.589+0000 7fd0417fa640 1 -- 172.21.15.70:0/3191774990 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd0540970d0 con 0x5622dc208490 2024-04-01T01:45:01.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.589+0000 7fd0627ee740 10 client.28738 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.589+0000 7fd0627ee740 10 client.28738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.589+0000 7fd0627ee740 20 client.28738 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.589+0000 7fd0627ee740 1 --2- 172.21.15.70:0/3191774990 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5622dc21e650 0x5622dc220b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.589+0000 7fd0627ee740 4 client.28738 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:45:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.589+0000 7fd0627ee740 1 -- 172.21.15.70:0/3191774990 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622dc17a6c0 con 0x5622dc21e650 2024-04-01T01:45:01.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.589+0000 7fd05d0c7640 1 --2- 172.21.15.70:0/3191774990 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5622dc21e650 0x5622dc220b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.590+0000 7fd05d0c7640 1 --2- 172.21.15.70:0/3191774990 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5622dc21e650 0x5622dc220b10 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=331589b6c9b32b99 server_cookie=bc8af87a1d555180 in_seq=0 out_seq=0 2024-04-01T01:45:01.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.591+0000 7fd0417fa640 10 client.28738 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:01.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.592+0000 7fd0417fa640 1 -- 172.21.15.70:0/3191774990 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5622dc17a6c0 con 0x5622dc21e650 2024-04-01T01:45:01.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.592+0000 7fd0417fa640 10 client.28738 handle_command_reply: tid=0 2024-04-01T01:45:01.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.592+0000 7fd0627ee740 1 client.28738 shutdown 2024-04-01T01:45:01.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.596+0000 7fd0627ee740 1 -- 172.21.15.70:0/3191774990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc208490 msgr2=0x5622dc2127b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.596+0000 7fd0627ee740 1 --2- 172.21.15.70:0/3191774990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc208490 0x5622dc2127b0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd054096ec0 tx=0x7fd05409a830 comp rx=0 tx=0).stop 2024-04-01T01:45:01.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.596+0000 7fd0627ee740 1 -- 172.21.15.70:0/3191774990 shutdown_connections 2024-04-01T01:45:01.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.596+0000 7fd0627ee740 1 --2- 172.21.15.70:0/3191774990 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5622dc21e650 0x5622dc220b10 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.596+0000 7fd0627ee740 1 --2- 172.21.15.70:0/3191774990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc20dff0 0x5622dc219d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.596+0000 7fd0627ee740 1 --2- 172.21.15.70:0/3191774990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 0x5622dc212cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.596+0000 7fd0627ee740 1 --2- 172.21.15.70:0/3191774990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc208490 0x5622dc2127b0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.596+0000 7fd0627ee740 1 -- 172.21.15.70:0/3191774990 >> 172.21.15.70:0/3191774990 conn(0x5622dc203a00 msgr2=0x5622dc202c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:01.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.597+0000 7fd0417fa640 0 client.28738 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:01.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.597+0000 7fd0627ee740 1 -- 172.21.15.70:0/3191774990 shutdown_connections 2024-04-01T01:45:01.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.598+0000 7fd0627ee740 1 -- 172.21.15.70:0/3191774990 wait complete. 2024-04-01T01:45:01.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.598+0000 7fd0627ee740 20 client.28738 trim_cache size 0 max 16384 2024-04-01T01:45:01.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.641+0000 7fd0627ee740 1 Processor -- start 2024-04-01T01:45:01.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.644+0000 7fd0627ee740 1 -- start start 2024-04-01T01:45:01.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.644+0000 7fd0627ee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc203a00 0x5622dc211c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.644+0000 7fd0627ee740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc208490 0x5622dc219cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.644+0000 7fd0627ee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 0x5622dc2123c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.645+0000 7fd0627ee740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd05014d660 con 0x5622dc208490 2024-04-01T01:45:01.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.645+0000 7fd0627ee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd05014f1a0 con 0x5622dc208e80 2024-04-01T01:45:01.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.645+0000 7fd0627ee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd05014dd30 con 0x5622dc203a00 2024-04-01T01:45:01.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.647+0000 7fd05d0c7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc208490 0x5622dc219cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.647+0000 7fd05d0c7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc208490 0x5622dc219cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54470/0 (socket says 172.21.15.70:54470) 2024-04-01T01:45:01.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.647+0000 7fd05d0c7640 1 -- 172.21.15.70:0/2918256599 learned_addr learned my addr 172.21.15.70:0/2918256599 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:01.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.648+0000 7fd05e0c9640 1 --2- 172.21.15.70:0/2918256599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 0x5622dc2123c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.648+0000 7fd05d8c8640 1 --2- 172.21.15.70:0/2918256599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc203a00 0x5622dc211c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.652+0000 7fd05e0c9640 1 -- 172.21.15.70:0/2918256599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc203a00 msgr2=0x5622dc211c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:01.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.652+0000 7fd05e0c9640 1 --2- 172.21.15.70:0/2918256599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc203a00 0x5622dc211c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.652+0000 7fd05e0c9640 1 -- 172.21.15.70:0/2918256599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc208490 msgr2=0x5622dc219cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:01.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.652+0000 7fd05e0c9640 1 --2- 172.21.15.70:0/2918256599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc208490 0x5622dc219cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.652+0000 7fd05e0c9640 1 -- 172.21.15.70:0/2918256599 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622dbe410f0 con 0x5622dc208e80 2024-04-01T01:45:01.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.653+0000 7fd05e0c9640 1 --2- 172.21.15.70:0/2918256599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 0x5622dc2123c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd058064c40 tx=0x7fd058075040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f284d662941e2adf server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:01.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.654+0000 7fd0437fe640 1 -- 172.21.15.70:0/2918256599 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05806d030 con 0x5622dc208e80 2024-04-01T01:45:01.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.655+0000 7fd0437fe640 1 -- 172.21.15.70:0/2918256599 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd058071040 con 0x5622dc208e80 2024-04-01T01:45:01.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.655+0000 7fd0437fe640 1 -- 172.21.15.70:0/2918256599 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05807c040 con 0x5622dc208e80 2024-04-01T01:45:01.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.656+0000 7fd0627ee740 1 -- 172.21.15.70:0/2918256599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 msgr2=0x5622dc2123c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.656+0000 7fd0627ee740 1 --2- 172.21.15.70:0/2918256599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 0x5622dc2123c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fd058064c40 tx=0x7fd058075040 comp rx=0 tx=0).stop 2024-04-01T01:45:01.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.656+0000 7fd0627ee740 1 -- 172.21.15.70:0/2918256599 shutdown_connections 2024-04-01T01:45:01.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.656+0000 7fd0627ee740 1 --2- 172.21.15.70:0/2918256599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 0x5622dc2123c0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.656+0000 7fd0627ee740 1 --2- 172.21.15.70:0/2918256599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc208490 0x5622dc219cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.656+0000 7fd0627ee740 1 --2- 172.21.15.70:0/2918256599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc203a00 0x5622dc211c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.656+0000 7fd0627ee740 1 -- 172.21.15.70:0/2918256599 >> 172.21.15.70:0/2918256599 conn(0x5622dc20daf0 msgr2=0x5622dc211510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:01.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.656+0000 7fd0627ee740 1 -- 172.21.15.70:0/2918256599 shutdown_connections 2024-04-01T01:45:01.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.656+0000 7fd0627ee740 1 -- 172.21.15.70:0/2918256599 wait complete. 2024-04-01T01:45:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.657+0000 7fd0627ee740 1 -- start start 2024-04-01T01:45:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.657+0000 7fd0627ee740 1 Processor -- start 2024-04-01T01:45:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.657+0000 7fd0627ee740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc203a00 0x5622dc217410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.657+0000 7fd0627ee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc208490 0x5622dc217950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.658+0000 7fd0627ee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 0x5622dc2188f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.658+0000 7fd0627ee740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd05014ead0 con 0x5622dc203a00 2024-04-01T01:45:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.658+0000 7fd0627ee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd05014e400 con 0x5622dc208e80 2024-04-01T01:45:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.658+0000 7fd0627ee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd05014b640 con 0x5622dc208490 2024-04-01T01:45:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.658+0000 7fd0627ee740 10 client.0 _ll_register_callbacks cb 0x7fd054096d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.658+0000 7fd05d0c7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc208490 0x5622dc217950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.658+0000 7fd05d8c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc203a00 0x5622dc217410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.658+0000 7fd05d0c7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc208490 0x5622dc217950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:39578/0 (socket says 172.21.15.70:39578) 2024-04-01T01:45:01.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.658+0000 7fd05d0c7640 1 -- 172.21.15.70:0/1005797416 learned_addr learned my addr 172.21.15.70:0/1005797416 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.658+0000 7fd05e0c9640 1 --2- 172.21.15.70:0/1005797416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 0x5622dc2188f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.658+0000 7fd05d0c7640 1 -- 172.21.15.70:0/1005797416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 msgr2=0x5622dc2188f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.658+0000 7fd05d0c7640 1 --2- 172.21.15.70:0/1005797416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 0x5622dc2188f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.658+0000 7fd05d0c7640 1 -- 172.21.15.70:0/1005797416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc203a00 msgr2=0x5622dc217410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.658+0000 7fd05d0c7640 1 --2- 172.21.15.70:0/1005797416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc203a00 0x5622dc217410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.658+0000 7fd05d0c7640 1 -- 172.21.15.70:0/1005797416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622dc218ee0 con 0x5622dc208490 2024-04-01T01:45:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.658+0000 7fd05e0c9640 1 --2- 172.21.15.70:0/1005797416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 0x5622dc2188f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:45:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.659+0000 7fd05d8c8640 1 --2- 172.21.15.70:0/1005797416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc203a00 0x5622dc217410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:45:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.659+0000 7fd05d0c7640 1 --2- 172.21.15.70:0/1005797416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc208490 0x5622dc217950 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd048009850 tx=0x7fd04802cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f47596c4d6429aa server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.659+0000 7fd0417fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:45:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.659+0000 7fd0417fa640 1 -- 172.21.15.70:0/1005797416 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd048056070 con 0x5622dc208490 2024-04-01T01:45:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.659+0000 7fd0627ee740 1 -- 172.21.15.70:0/1005797416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd058071040 con 0x5622dc208490 2024-04-01T01:45:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.659+0000 7fd0417fa640 1 -- 172.21.15.70:0/1005797416 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd048051330 con 0x5622dc208490 2024-04-01T01:45:01.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.659+0000 7fd0417fa640 1 -- 172.21.15.70:0/1005797416 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd048062050 con 0x5622dc208490 2024-04-01T01:45:01.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.659+0000 7fd0627ee740 10 client.28744 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:01.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.660+0000 7fd0627ee740 1 -- 172.21.15.70:0/1005797416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5622dc284a80 con 0x5622dc208490 2024-04-01T01:45:01.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.660+0000 7fd0417fa640 1 -- 172.21.15.70:0/1005797416 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd04802cbd0 con 0x5622dc208490 2024-04-01T01:45:01.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.660+0000 7fd0417fa640 1 -- 172.21.15.70:0/1005797416 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd048041480 con 0x5622dc208490 2024-04-01T01:45:01.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.660+0000 7fd0627ee740 10 client.28744 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:01.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.660+0000 7fd0627ee740 10 client.28744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:01.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.660+0000 7fd0627ee740 20 client.28744 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.660+0000 7fd0627ee740 1 --2- 172.21.15.70:0/1005797416 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5622dc20dff0 0x5622dc287070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.660+0000 7fd0627ee740 4 client.28744 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:45:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.660+0000 7fd0627ee740 1 -- 172.21.15.70:0/1005797416 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fd058071040 con 0x5622dc20dff0 2024-04-01T01:45:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.661+0000 7fd05d8c8640 1 --2- 172.21.15.70:0/1005797416 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5622dc20dff0 0x5622dc287070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.661+0000 7fd05d8c8640 1 --2- 172.21.15.70:0/1005797416 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5622dc20dff0 0x5622dc287070 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5db4c12a1519fc81 server_cookie=a4de2d3d596200ef in_seq=0 out_seq=0 2024-04-01T01:45:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.661+0000 7fd0417fa640 10 client.28744 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.662+0000 7fd0417fa640 1 -- 172.21.15.70:0/1005797416 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7fd058071040 con 0x5622dc20dff0 2024-04-01T01:45:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.662+0000 7fd0417fa640 10 client.28744 handle_command_reply: tid=0 2024-04-01T01:45:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.662+0000 7fd0627ee740 1 client.28744 shutdown 2024-04-01T01:45:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.662+0000 7fd0627ee740 1 -- 172.21.15.70:0/1005797416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc208490 msgr2=0x5622dc217950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.662+0000 7fd0627ee740 1 --2- 172.21.15.70:0/1005797416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc208490 0x5622dc217950 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd048009850 tx=0x7fd04802cd80 comp rx=0 tx=0).stop 2024-04-01T01:45:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.662+0000 7fd0627ee740 1 -- 172.21.15.70:0/1005797416 shutdown_connections 2024-04-01T01:45:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.662+0000 7fd0627ee740 1 --2- 172.21.15.70:0/1005797416 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5622dc20dff0 0x5622dc287070 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.662+0000 7fd0417fa640 0 client.28744 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:01.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.662+0000 7fd0627ee740 1 --2- 172.21.15.70:0/1005797416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5622dc208e80 0x5622dc2188f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.662+0000 7fd0627ee740 1 --2- 172.21.15.70:0/1005797416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622dc208490 0x5622dc217950 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.662+0000 7fd0627ee740 1 --2- 172.21.15.70:0/1005797416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5622dc203a00 0x5622dc217410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.662+0000 7fd0627ee740 1 -- 172.21.15.70:0/1005797416 >> 172.21.15.70:0/1005797416 conn(0x5622dc20daf0 msgr2=0x5622dc2123c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:01.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.662+0000 7fd0627ee740 1 -- 172.21.15.70:0/1005797416 shutdown_connections 2024-04-01T01:45:01.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.663+0000 7fd0627ee740 1 -- 172.21.15.70:0/1005797416 wait complete. 2024-04-01T01:45:01.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.663+0000 7fd0627ee740 20 client.28744 trim_cache size 0 max 16384 2024-04-01T01:45:01.665 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:45:01.666 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:45:01.666 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:45:01.666 INFO:teuthology.orchestra.run.smithi070.stdout: "7becb7aa-fb2c-404f-8946-38eb43b1fcdc": { 2024-04-01T01:45:01.666 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:45:01.666 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "7becb7aa-fb2c-404f-8946-38eb43b1fcdc", 2024-04-01T01:45:01.666 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:45:01.666 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:45:01.666 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:45:01.666 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:45:01.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.665+0000 7fd05e8ca640 1 -- 172.21.15.70:0/2144389694 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0440a26a0 msgr2=0x7fd0440a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.665+0000 7fd05e8ca640 1 --2- 172.21.15.70:0/2144389694 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0440a26a0 0x7fd0440a4b60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x5622dc2096e0 tx=0x7fd0580675f0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.665+0000 7fd05e8ca640 1 -- 172.21.15.70:0/2144389694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd05009c370 msgr2=0x7fd050013c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:01.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.665+0000 7fd05e8ca640 1 --2- 172.21.15.70:0/2144389694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd05009c370 0x7fd050013c60 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd05400d770 tx=0x7fd05400dc30 comp rx=0 tx=0).stop 2024-04-01T01:45:01.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.666+0000 7fd05e8ca640 1 -- 172.21.15.70:0/2144389694 shutdown_connections 2024-04-01T01:45:01.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.666+0000 7fd05e8ca640 1 --2- 172.21.15.70:0/2144389694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd05009df20 0x7fd0500151e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.666+0000 7fd05e8ca640 1 --2- 172.21.15.70:0/2144389694 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd0440a26a0 0x7fd0440a4b60 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.666+0000 7fd05e8ca640 1 --2- 172.21.15.70:0/2144389694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05009d560 0x7fd0500141a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.666+0000 7fd05e8ca640 1 --2- 172.21.15.70:0/2144389694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd05009c370 0x7fd050013c60 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:01.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.666+0000 7fd05e8ca640 1 -- 172.21.15.70:0/2144389694 >> 172.21.15.70:0/2144389694 conn(0x7fd05000b4d0 msgr2=0x7fd0500a0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:01.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.666+0000 7fd05e8ca640 1 -- 172.21.15.70:0/2144389694 shutdown_connections 2024-04-01T01:45:01.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:01.666+0000 7fd05e8ca640 1 -- 172.21.15.70:0/2144389694 wait complete. 2024-04-01T01:45:01.687 INFO:tasks.cephfs.filesystem:scrub status for tag:7becb7aa-fb2c-404f-8946-38eb43b1fcdc - {'path': '/', 'tag': '7becb7aa-fb2c-404f-8946-38eb43b1fcdc', 'options': 'recursive,force'} 2024-04-01T01:45:01.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 615.0 s, 841.4 tps, lat 119.347 ms stddev 84.359 2024-04-01T01:45:02.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:02 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:45:02.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:02 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:45:02.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:02 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:45:02.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:02 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:45:02.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:02 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:45:02.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:02 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:45:02.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:02 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:45:02.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:02 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:45:02.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:02 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:45:02.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:02 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:45:02.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:02 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:45:02.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:02 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:45:03.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:03 smithi189 ceph-mon[35753]: pgmap v706: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 9.7 MiB/s wr, 885 op/s 2024-04-01T01:45:03.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:03 smithi070 ceph-mon[32179]: pgmap v706: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 9.7 MiB/s wr, 885 op/s 2024-04-01T01:45:03.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:03 smithi081 ceph-mon[35142]: pgmap v706: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 9.7 MiB/s wr, 885 op/s 2024-04-01T01:45:05.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:04 smithi189 ceph-mon[35753]: pgmap v707: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 9.4 MiB/s wr, 855 op/s 2024-04-01T01:45:05.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:04 smithi070 ceph-mon[32179]: pgmap v707: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 9.4 MiB/s wr, 855 op/s 2024-04-01T01:45:05.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:04 smithi081 ceph-mon[35142]: pgmap v707: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 9.4 MiB/s wr, 855 op/s 2024-04-01T01:45:06.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:06 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:45:06.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:06 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:45:06.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:06 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:45:06.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 620.0 s, 831.8 tps, lat 117.307 ms stddev 103.768 2024-04-01T01:45:07.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:07 smithi189 ceph-mon[35753]: pgmap v708: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 9.6 MiB/s wr, 879 op/s 2024-04-01T01:45:07.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:07 smithi070 ceph-mon[32179]: pgmap v708: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 9.6 MiB/s wr, 879 op/s 2024-04-01T01:45:07.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:07 smithi081 ceph-mon[35142]: pgmap v708: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 9.6 MiB/s wr, 879 op/s 2024-04-01T01:45:09.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:08 smithi189 ceph-mon[35753]: pgmap v709: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 9.8 MiB/s wr, 910 op/s 2024-04-01T01:45:09.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:08 smithi070 ceph-mon[32179]: pgmap v709: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 9.8 MiB/s wr, 910 op/s 2024-04-01T01:45:09.276 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.275+0000 7f23321e5640 1 -- 172.21.15.70:0/1537607832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f232c073570 msgr2=0x7f232c10b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:09.276 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.275+0000 7f23321e5640 1 --2- 172.21.15.70:0/1537607832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f232c073570 0x7f232c10b870 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2328009900 tx=0x7f232802bc50 comp rx=0 tx=0).stop 2024-04-01T01:45:09.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.284+0000 7f23321e5640 1 -- 172.21.15.70:0/1537607832 shutdown_connections 2024-04-01T01:45:09.284 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.284+0000 7f23321e5640 1 --2- 172.21.15.70:0/1537607832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f232c10e920 0x7f232c110de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.284+0000 7f23321e5640 1 --2- 172.21.15.70:0/1537607832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f232c10be40 0x7f232c10e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.284+0000 7f23321e5640 1 --2- 172.21.15.70:0/1537607832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f232c073570 0x7f232c10b870 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.284+0000 7f23321e5640 1 -- 172.21.15.70:0/1537607832 >> 172.21.15.70:0/1537607832 conn(0x7f232c06e780 msgr2=0x7f232c070bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.290+0000 7f23321e5640 1 -- 172.21.15.70:0/1537607832 shutdown_connections 2024-04-01T01:45:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.294+0000 7f23321e5640 1 -- 172.21.15.70:0/1537607832 wait complete. 2024-04-01T01:45:09.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.294+0000 7f23321e5640 1 Processor -- start 2024-04-01T01:45:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.295+0000 7f23321e5640 1 -- start start 2024-04-01T01:45:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.296+0000 7f23321e5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f232c073570 0x7f232c1a6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.296+0000 7f23321e5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f232c10be40 0x7f232c1a6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.296+0000 7f23321e5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f232c10e920 0x7f232c1adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.296+0000 7f23321e5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f232c1137d0 con 0x7f232c10e920 2024-04-01T01:45:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.296+0000 7f23321e5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f232c113650 con 0x7f232c073570 2024-04-01T01:45:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.296+0000 7f23321e5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f232c113950 con 0x7f232c10be40 2024-04-01T01:45:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.299+0000 7f23309e2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f232c10be40 0x7f232c1a6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.299+0000 7f23309e2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f232c10be40 0x7f232c1a6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:60554/0 (socket says 172.21.15.70:60554) 2024-04-01T01:45:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.299+0000 7f23309e2640 1 -- 172.21.15.70:0/2139555103 learned_addr learned my addr 172.21.15.70:0/2139555103 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.299+0000 7f23319e4640 1 --2- 172.21.15.70:0/2139555103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f232c10e920 0x7f232c1adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.300+0000 7f23311e3640 1 --2- 172.21.15.70:0/2139555103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f232c073570 0x7f232c1a6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.300+0000 7f23309e2640 1 -- 172.21.15.70:0/2139555103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f232c073570 msgr2=0x7f232c1a6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.300+0000 7f23309e2640 1 --2- 172.21.15.70:0/2139555103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f232c073570 0x7f232c1a6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.300+0000 7f23309e2640 1 -- 172.21.15.70:0/2139555103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f232c10e920 msgr2=0x7f232c1adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.301+0000 7f23309e2640 1 --2- 172.21.15.70:0/2139555103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f232c10e920 0x7f232c1adcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.301+0000 7f23309e2640 1 -- 172.21.15.70:0/2139555103 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f232c19dc50 con 0x7f232c10be40 2024-04-01T01:45:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.301+0000 7f23309e2640 1 --2- 172.21.15.70:0/2139555103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f232c10be40 0x7f232c1a6c50 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f231c00d740 tx=0x7f231c00dc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.303+0000 7f23227fc640 1 -- 172.21.15.70:0/2139555103 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c017070 con 0x7f232c10be40 2024-04-01T01:45:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.304+0000 7f23227fc640 1 -- 172.21.15.70:0/2139555103 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f231c00de50 con 0x7f232c10be40 2024-04-01T01:45:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.304+0000 7f23227fc640 1 -- 172.21.15.70:0/2139555103 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c004d60 con 0x7f232c10be40 2024-04-01T01:45:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.304+0000 7f23321e5640 1 -- 172.21.15.70:0/2139555103 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f232c19df40 con 0x7f232c10be40 2024-04-01T01:45:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.304+0000 7f23321e5640 1 -- 172.21.15.70:0/2139555103 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f232c19e4c0 con 0x7f232c10be40 2024-04-01T01:45:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.308+0000 7f233602c740 1 Processor -- start 2024-04-01T01:45:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.309+0000 7f23227fc640 1 -- 172.21.15.70:0/2139555103 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f231c009c30 con 0x7f232c10be40 2024-04-01T01:45:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.310+0000 7f23227fc640 1 --2- 172.21.15.70:0/2139555103 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23180a26a0 0x7f23180a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.310+0000 7f23227fc640 1 -- 172.21.15.70:0/2139555103 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f231c0ce710 con 0x7f232c10be40 2024-04-01T01:45:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.311+0000 7f23319e4640 1 --2- 172.21.15.70:0/2139555103 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23180a26a0 0x7f23180a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.312+0000 7f233602c740 1 -- start start 2024-04-01T01:45:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.312+0000 7f233602c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a81ae0 0x563d88a81f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.312+0000 7f233602c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 0x563d88a86fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.312+0000 7f233602c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a87640 0x563d88a89ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.312+0000 7f233602c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563d889e58f0 con 0x563d88a87640 2024-04-01T01:45:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.312+0000 7f233602c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563d8898e8d0 con 0x563d88a824d0 2024-04-01T01:45:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.312+0000 7f233602c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563d889e1990 con 0x563d88a81ae0 2024-04-01T01:45:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.312+0000 7f23311e3640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a81ae0 0x563d88a81f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.312+0000 7f23311e3640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a81ae0 0x563d88a81f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:60568/0 (socket says 172.21.15.70:60568) 2024-04-01T01:45:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.312+0000 7f23311e3640 1 -- 172.21.15.70:0/1895677654 learned_addr learned my addr 172.21.15.70:0/1895677654 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.312+0000 7f23319e4640 1 --2- 172.21.15.70:0/1895677654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a87640 0x563d88a89ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.312+0000 7f23309e2640 1 --2- 172.21.15.70:0/1895677654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 0x563d88a86fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.313+0000 7f23311e3640 1 -- 172.21.15.70:0/1895677654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 msgr2=0x563d88a86fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.313+0000 7f23311e3640 1 --2- 172.21.15.70:0/1895677654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 0x563d88a86fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.313+0000 7f23311e3640 1 -- 172.21.15.70:0/1895677654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a87640 msgr2=0x563d88a89ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.313+0000 7f23311e3640 1 --2- 172.21.15.70:0/1895677654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a87640 0x563d88a89ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.313+0000 7f23311e3640 1 -- 172.21.15.70:0/1895677654 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d886fa210 con 0x563d88a81ae0 2024-04-01T01:45:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.313+0000 7f23319e4640 1 --2- 172.21.15.70:0/2139555103 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23180a26a0 0x7f23180a4b60 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2324009ec0 tx=0x7f2324009430 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.313+0000 7f23311e3640 1 --2- 172.21.15.70:0/1895677654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a81ae0 0x563d88a81f00 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2328000c00 tx=0x7f2328009610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87edf707103453ff server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.313+0000 7f23177fe640 1 -- 172.21.15.70:0/1895677654 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2328044020 con 0x563d88a81ae0 2024-04-01T01:45:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.313+0000 7f23177fe640 1 -- 172.21.15.70:0/1895677654 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f232802cc40 con 0x563d88a81ae0 2024-04-01T01:45:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.313+0000 7f23177fe640 1 -- 172.21.15.70:0/1895677654 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2328004920 con 0x563d88a81ae0 2024-04-01T01:45:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.314+0000 7f233602c740 1 -- 172.21.15.70:0/1895677654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a81ae0 msgr2=0x563d88a81f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.314+0000 7f233602c740 1 --2- 172.21.15.70:0/1895677654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a81ae0 0x563d88a81f00 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2328000c00 tx=0x7f2328009610 comp rx=0 tx=0).stop 2024-04-01T01:45:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.314+0000 7f233602c740 1 -- 172.21.15.70:0/1895677654 shutdown_connections 2024-04-01T01:45:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.314+0000 7f233602c740 1 --2- 172.21.15.70:0/1895677654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a87640 0x563d88a89ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.314+0000 7f233602c740 1 --2- 172.21.15.70:0/1895677654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 0x563d88a86fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.314+0000 7f233602c740 1 --2- 172.21.15.70:0/1895677654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a81ae0 0x563d88a81f00 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.314+0000 7f233602c740 1 -- 172.21.15.70:0/1895677654 >> 172.21.15.70:0/1895677654 conn(0x563d88a7d050 msgr2=0x563d88a7d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.315+0000 7f233602c740 1 -- 172.21.15.70:0/1895677654 shutdown_connections 2024-04-01T01:45:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.315+0000 7f233602c740 1 -- 172.21.15.70:0/1895677654 wait complete. 2024-04-01T01:45:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.315+0000 7f233602c740 1 -- start start 2024-04-01T01:45:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.315+0000 7f233602c740 1 Processor -- start 2024-04-01T01:45:09.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.316+0000 7f233602c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 0x563d88a8bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.316+0000 7f233602c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a824d0 0x563d88a8c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.316+0000 7f233602c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a87640 0x563d88a93260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.316+0000 7f233602c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563d88a8a590 con 0x563d88a81ae0 2024-04-01T01:45:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.316+0000 7f233602c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563d88983f80 con 0x563d88a87640 2024-04-01T01:45:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.316+0000 7f233602c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563d88a8a710 con 0x563d88a824d0 2024-04-01T01:45:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.316+0000 7f233602c740 10 client.0 _ll_register_callbacks cb 0x563d889dc290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.316+0000 7f23311e3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 0x563d88a8bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.316+0000 7f23311e3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 0x563d88a8bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33592/0 (socket says 172.21.15.70:33592) 2024-04-01T01:45:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.316+0000 7f23311e3640 1 -- 172.21.15.70:0/562165595 learned_addr learned my addr 172.21.15.70:0/562165595 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.316+0000 7f23309e2640 1 --2- 172.21.15.70:0/562165595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a824d0 0x563d88a8c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.316+0000 7f23311e3640 1 -- 172.21.15.70:0/562165595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a824d0 msgr2=0x563d88a8c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.316+0000 7f23311e3640 1 --2- 172.21.15.70:0/562165595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a824d0 0x563d88a8c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.316+0000 7f23311e3640 1 -- 172.21.15.70:0/562165595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a87640 msgr2=0x563d88a93260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.316+0000 7f23311e3640 1 --2- 172.21.15.70:0/562165595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a87640 0x563d88a93260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.316+0000 7f23311e3640 1 -- 172.21.15.70:0/562165595 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d887c6620 con 0x563d88a81ae0 2024-04-01T01:45:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.318+0000 7f23311e3640 1 --2- 172.21.15.70:0/562165595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 0x563d88a8bcf0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2328009850 tx=0x7f2328009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b76173e5e081163 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.318+0000 7f2314ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:45:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.318+0000 7f2314ff9640 1 -- 172.21.15.70:0/562165595 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2328044030 con 0x563d88a81ae0 2024-04-01T01:45:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.318+0000 7f2314ff9640 1 -- 172.21.15.70:0/562165595 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f2328003a00 con 0x563d88a81ae0 2024-04-01T01:45:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.318+0000 7f233602c740 1 -- 172.21.15.70:0/562165595 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f232802cc40 con 0x563d88a81ae0 2024-04-01T01:45:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.318+0000 7f2314ff9640 1 -- 172.21.15.70:0/562165595 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2328003c00 con 0x563d88a81ae0 2024-04-01T01:45:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.318+0000 7f233602c740 10 client.18903 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.318+0000 7f233602c740 1 -- 172.21.15.70:0/562165595 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563d88a96240 con 0x563d88a81ae0 2024-04-01T01:45:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.319+0000 7f2314ff9640 1 -- 172.21.15.70:0/562165595 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f232802cc40 con 0x563d88a81ae0 2024-04-01T01:45:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.319+0000 7f2314ff9640 1 -- 172.21.15.70:0/562165595 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f232804b460 con 0x563d88a81ae0 2024-04-01T01:45:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.319+0000 7f233602c740 10 client.18903 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.319+0000 7f233602c740 10 client.18903 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.319+0000 7f233602c740 20 client.18903 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.319+0000 7f233602c740 1 --2- 172.21.15.70:0/562165595 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563d88a97600 0x563d88a99a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.319+0000 7f233602c740 4 client.18903 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:45:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.319+0000 7f233602c740 1 -- 172.21.15.70:0/562165595 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d889f9120 con 0x563d88a97600 2024-04-01T01:45:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.320+0000 7f23309e2640 1 --2- 172.21.15.70:0/562165595 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563d88a97600 0x563d88a99a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.320+0000 7f23309e2640 1 --2- 172.21.15.70:0/562165595 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563d88a97600 0x563d88a99a90 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e2954635e21afda server_cookie=3cc63e500331b4c3 in_seq=0 out_seq=0 2024-04-01T01:45:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.320+0000 7f2314ff9640 10 client.18903 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.321+0000 7f2314ff9640 1 -- 172.21.15.70:0/562165595 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563d889f9120 con 0x563d88a97600 2024-04-01T01:45:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.321+0000 7f2314ff9640 10 client.18903 handle_command_reply: tid=0 2024-04-01T01:45:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.321+0000 7f233602c740 1 client.18903 shutdown 2024-04-01T01:45:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.321+0000 7f233602c740 1 -- 172.21.15.70:0/562165595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 msgr2=0x563d88a8bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.321+0000 7f233602c740 1 --2- 172.21.15.70:0/562165595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 0x563d88a8bcf0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2328009850 tx=0x7f2328009680 comp rx=0 tx=0).stop 2024-04-01T01:45:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.321+0000 7f233602c740 1 -- 172.21.15.70:0/562165595 shutdown_connections 2024-04-01T01:45:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.321+0000 7f233602c740 1 --2- 172.21.15.70:0/562165595 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563d88a97600 0x563d88a99a90 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.321+0000 7f233602c740 1 --2- 172.21.15.70:0/562165595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a87640 0x563d88a93260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.321+0000 7f233602c740 1 --2- 172.21.15.70:0/562165595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a824d0 0x563d88a8c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.321+0000 7f233602c740 1 --2- 172.21.15.70:0/562165595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 0x563d88a8bcf0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.321+0000 7f233602c740 1 -- 172.21.15.70:0/562165595 >> 172.21.15.70:0/562165595 conn(0x563d88a7d050 msgr2=0x563d88a7c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.322+0000 7f2314ff9640 0 client.18903 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.322+0000 7f233602c740 1 -- 172.21.15.70:0/562165595 shutdown_connections 2024-04-01T01:45:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.322+0000 7f233602c740 1 -- 172.21.15.70:0/562165595 wait complete. 2024-04-01T01:45:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.322+0000 7f233602c740 20 client.18903 trim_cache size 0 max 16384 2024-04-01T01:45:09.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:08 smithi081 ceph-mon[35142]: pgmap v709: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 9.8 MiB/s wr, 910 op/s 2024-04-01T01:45:09.355 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.355+0000 7f233602c740 1 Processor -- start 2024-04-01T01:45:09.355 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.355+0000 7f233602c740 1 -- start start 2024-04-01T01:45:09.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.355+0000 7f233602c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 0x563d88a78030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.356+0000 7f233602c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 0x563d88a931f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.356+0000 7f233602c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a87640 0x563d88a8aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.356+0000 7f23309e2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 0x563d88a931f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.356+0000 7f23309e2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 0x563d88a931f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:54824/0 (socket says 172.21.15.70:54824) 2024-04-01T01:45:09.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.356+0000 7f23309e2640 1 -- 172.21.15.70:0/2154582437 learned_addr learned my addr 172.21.15.70:0/2154582437 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:09.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.356+0000 7f23319e4640 1 --2- 172.21.15.70:0/2154582437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a87640 0x563d88a8aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.356+0000 7f233602c740 1 -- 172.21.15.70:0/2154582437 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f231000fd20 con 0x563d88a81ae0 2024-04-01T01:45:09.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.356+0000 7f233602c740 1 -- 172.21.15.70:0/2154582437 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2310011860 con 0x563d88a824d0 2024-04-01T01:45:09.357 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.356+0000 7f233602c740 1 -- 172.21.15.70:0/2154582437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f23100103f0 con 0x563d88a87640 2024-04-01T01:45:09.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.358+0000 7f23311e3640 1 --2- 172.21.15.70:0/2154582437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 0x563d88a78030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.358 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.358+0000 7f23309e2640 1 -- 172.21.15.70:0/2154582437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a87640 msgr2=0x563d88a8aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:09.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.358+0000 7f23309e2640 1 --2- 172.21.15.70:0/2154582437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a87640 0x563d88a8aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.358+0000 7f23309e2640 1 -- 172.21.15.70:0/2154582437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 msgr2=0x563d88a78030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:09.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.358+0000 7f23309e2640 1 --2- 172.21.15.70:0/2154582437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 0x563d88a78030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.359 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.358+0000 7f23309e2640 1 -- 172.21.15.70:0/2154582437 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d88a83190 con 0x563d88a824d0 2024-04-01T01:45:09.360 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.359+0000 7f23309e2640 1 --2- 172.21.15.70:0/2154582437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 0x563d88a931f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f231c09ab20 tx=0x7f231c09b660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b319be0df1b4def3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:09.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.365+0000 7f2316ffd640 1 -- 172.21.15.70:0/2154582437 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c099120 con 0x563d88a824d0 2024-04-01T01:45:09.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.365+0000 7f2316ffd640 1 -- 172.21.15.70:0/2154582437 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f231c0a8070 con 0x563d88a824d0 2024-04-01T01:45:09.365 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.365+0000 7f2316ffd640 1 -- 172.21.15.70:0/2154582437 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c0a33a0 con 0x563d88a824d0 2024-04-01T01:45:09.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.366+0000 7f233602c740 1 -- 172.21.15.70:0/2154582437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 msgr2=0x563d88a931f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:09.366 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.366+0000 7f233602c740 1 --2- 172.21.15.70:0/2154582437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 0x563d88a931f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f231c09ab20 tx=0x7f231c09b660 comp rx=0 tx=0).stop 2024-04-01T01:45:09.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.366+0000 7f233602c740 1 -- 172.21.15.70:0/2154582437 shutdown_connections 2024-04-01T01:45:09.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.366+0000 7f233602c740 1 --2- 172.21.15.70:0/2154582437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a87640 0x563d88a8aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.366+0000 7f233602c740 1 --2- 172.21.15.70:0/2154582437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 0x563d88a931f0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.366+0000 7f233602c740 1 --2- 172.21.15.70:0/2154582437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 0x563d88a78030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.366+0000 7f233602c740 1 -- 172.21.15.70:0/2154582437 >> 172.21.15.70:0/2154582437 conn(0x563d88a7d050 msgr2=0x563d88a7d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.372+0000 7f233602c740 1 -- 172.21.15.70:0/2154582437 shutdown_connections 2024-04-01T01:45:09.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.372+0000 7f233602c740 1 -- 172.21.15.70:0/2154582437 wait complete. 2024-04-01T01:45:09.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.373+0000 7f233602c740 1 -- start start 2024-04-01T01:45:09.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.373+0000 7f233602c740 1 Processor -- start 2024-04-01T01:45:09.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.373+0000 7f233602c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 0x563d88a8ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.373+0000 7f233602c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 0x563d88a90500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.373+0000 7f233602c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a87640 0x563d88afb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.373+0000 7f233602c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2310011190 con 0x563d88a81ae0 2024-04-01T01:45:09.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.373+0000 7f233602c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2310010ac0 con 0x563d88a824d0 2024-04-01T01:45:09.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.373+0000 7f233602c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f231000dd00 con 0x563d88a87640 2024-04-01T01:45:09.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.373+0000 7f233602c740 10 client.0 _ll_register_callbacks cb 0x7f2328009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:09.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.374+0000 7f23319e4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a87640 0x563d88afb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.374+0000 7f23319e4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a87640 0x563d88afb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:60604/0 (socket says 172.21.15.70:60604) 2024-04-01T01:45:09.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.374+0000 7f23319e4640 1 -- 172.21.15.70:0/4185642477 learned_addr learned my addr 172.21.15.70:0/4185642477 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:09.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.374+0000 7f23309e2640 1 --2- 172.21.15.70:0/4185642477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 0x563d88a90500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.374+0000 7f23311e3640 1 --2- 172.21.15.70:0/4185642477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 0x563d88a8ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.374+0000 7f23319e4640 1 -- 172.21.15.70:0/4185642477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 msgr2=0x563d88a90500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:09.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.374+0000 7f23319e4640 1 --2- 172.21.15.70:0/4185642477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 0x563d88a90500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.374+0000 7f23319e4640 1 -- 172.21.15.70:0/4185642477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 msgr2=0x563d88a8ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:09.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.374+0000 7f23319e4640 1 --2- 172.21.15.70:0/4185642477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 0x563d88a8ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.375+0000 7f23319e4640 1 -- 172.21.15.70:0/4185642477 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d88afbec0 con 0x563d88a87640 2024-04-01T01:45:09.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.375+0000 7f23319e4640 1 --2- 172.21.15.70:0/4185642477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a87640 0x563d88afb980 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2324017cc0 tx=0x7f2324004af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0a255154079ad27 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:09.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.375+0000 7f2314ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:45:09.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.375+0000 7f2314ff9640 1 -- 172.21.15.70:0/4185642477 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2324016ab0 con 0x563d88a87640 2024-04-01T01:45:09.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.375+0000 7f2314ff9640 1 -- 172.21.15.70:0/4185642477 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f2324016c50 con 0x563d88a87640 2024-04-01T01:45:09.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.376+0000 7f233602c740 1 -- 172.21.15.70:0/4185642477 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f231c0a8070 con 0x563d88a87640 2024-04-01T01:45:09.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.376+0000 7f2314ff9640 1 -- 172.21.15.70:0/4185642477 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2324016e50 con 0x563d88a87640 2024-04-01T01:45:09.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.380+0000 7f233602c740 10 client.28759 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:09.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.380+0000 7f233602c740 1 -- 172.21.15.70:0/4185642477 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563d88afd110 con 0x563d88a87640 2024-04-01T01:45:09.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.383+0000 7f2314ff9640 1 -- 172.21.15.70:0/4185642477 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f231c0a8070 con 0x563d88a87640 2024-04-01T01:45:09.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.383+0000 7f2314ff9640 1 -- 172.21.15.70:0/4185642477 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f232402c030 con 0x563d88a87640 2024-04-01T01:45:09.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.383+0000 7f233602c740 10 client.28759 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:09.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.383+0000 7f233602c740 10 client.28759 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:09.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.383+0000 7f233602c740 20 client.28759 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:09.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.383+0000 7f233602c740 1 --2- 172.21.15.70:0/4185642477 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563d88afdb20 0x563d88afff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:09.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.383+0000 7f233602c740 4 client.28759 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:45:09.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.383+0000 7f233602c740 1 -- 172.21.15.70:0/4185642477 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563d889f2e30 con 0x563d88afdb20 2024-04-01T01:45:09.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.386+0000 7f23311e3640 1 --2- 172.21.15.70:0/4185642477 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563d88afdb20 0x563d88afff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:09.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.415+0000 7f23311e3640 1 --2- 172.21.15.70:0/4185642477 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563d88afdb20 0x563d88afff40 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a463dea89fd41e3 server_cookie=4fbc22e1d23d13b7 in_seq=0 out_seq=0 2024-04-01T01:45:09.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.416+0000 7f2314ff9640 10 client.28759 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.831+0000 7f2314ff9640 1 -- 172.21.15.70:0/4185642477 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563d889f2e30 con 0x563d88afdb20 2024-04-01T01:45:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.831+0000 7f2314ff9640 10 client.28759 handle_command_reply: tid=0 2024-04-01T01:45:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.831+0000 7f233602c740 1 client.28759 shutdown 2024-04-01T01:45:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.832+0000 7f233602c740 1 -- 172.21.15.70:0/4185642477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a87640 msgr2=0x563d88afb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.832+0000 7f233602c740 1 --2- 172.21.15.70:0/4185642477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a87640 0x563d88afb980 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2324017cc0 tx=0x7f2324004af0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.832+0000 7f233602c740 1 -- 172.21.15.70:0/4185642477 shutdown_connections 2024-04-01T01:45:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.832+0000 7f233602c740 1 --2- 172.21.15.70:0/4185642477 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563d88afdb20 0x563d88afff40 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.832+0000 7f233602c740 1 --2- 172.21.15.70:0/4185642477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563d88a87640 0x563d88afb980 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.832+0000 7f233602c740 1 --2- 172.21.15.70:0/4185642477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d88a824d0 0x563d88a90500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.832+0000 7f233602c740 1 --2- 172.21.15.70:0/4185642477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563d88a81ae0 0x563d88a8ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.832+0000 7f233602c740 1 -- 172.21.15.70:0/4185642477 >> 172.21.15.70:0/4185642477 conn(0x563d88a7d050 msgr2=0x563d88a7d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.832+0000 7f2314ff9640 0 client.28759 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.832+0000 7f233602c740 1 -- 172.21.15.70:0/4185642477 shutdown_connections 2024-04-01T01:45:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.832+0000 7f233602c740 1 -- 172.21.15.70:0/4185642477 wait complete. 2024-04-01T01:45:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.832+0000 7f233602c740 20 client.28759 trim_cache size 0 max 16384 2024-04-01T01:45:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:45:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:45:09.834 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:45:09.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:45:09.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.836+0000 7f23321e5640 1 -- 172.21.15.70:0/2139555103 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23180a26a0 msgr2=0x7f23180a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:09.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.836+0000 7f23321e5640 1 --2- 172.21.15.70:0/2139555103 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23180a26a0 0x7f23180a4b60 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2324009ec0 tx=0x7f2324009430 comp rx=0 tx=0).stop 2024-04-01T01:45:09.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.836+0000 7f23321e5640 1 -- 172.21.15.70:0/2139555103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f232c10be40 msgr2=0x7f232c1a6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.836+0000 7f23321e5640 1 --2- 172.21.15.70:0/2139555103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f232c10be40 0x7f232c1a6c50 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f231c00d740 tx=0x7f231c00dc00 comp rx=0 tx=0).stop 2024-04-01T01:45:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.836+0000 7f23321e5640 1 -- 172.21.15.70:0/2139555103 shutdown_connections 2024-04-01T01:45:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.836+0000 7f23321e5640 1 --2- 172.21.15.70:0/2139555103 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f232c10e920 0x7f232c1adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.836+0000 7f23321e5640 1 --2- 172.21.15.70:0/2139555103 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f23180a26a0 0x7f23180a4b60 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.836+0000 7f23321e5640 1 --2- 172.21.15.70:0/2139555103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f232c10be40 0x7f232c1a6c50 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.836+0000 7f23321e5640 1 --2- 172.21.15.70:0/2139555103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f232c073570 0x7f232c1a6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.836+0000 7f23321e5640 1 -- 172.21.15.70:0/2139555103 >> 172.21.15.70:0/2139555103 conn(0x7f232c06e780 msgr2=0x7f232c10f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.837+0000 7f23321e5640 1 -- 172.21.15.70:0/2139555103 shutdown_connections 2024-04-01T01:45:09.838 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:09.837+0000 7f23321e5640 1 -- 172.21.15.70:0/2139555103 wait complete. 2024-04-01T01:45:11.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:10 smithi189 ceph-mon[35753]: pgmap v710: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 9.2 MiB/s wr, 866 op/s 2024-04-01T01:45:11.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:10 smithi070 ceph-mon[32179]: pgmap v710: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 9.2 MiB/s wr, 866 op/s 2024-04-01T01:45:11.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:10 smithi081 ceph-mon[35142]: pgmap v710: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 9.2 MiB/s wr, 866 op/s 2024-04-01T01:45:11.899 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 625.0 s, 729.4 tps, lat 135.845 ms stddev 218.922 2024-04-01T01:45:13.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:12 smithi189 ceph-mon[35753]: pgmap v711: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 10 MiB/s wr, 916 op/s 2024-04-01T01:45:13.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:12 smithi070 ceph-mon[32179]: pgmap v711: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 10 MiB/s wr, 916 op/s 2024-04-01T01:45:13.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:12 smithi081 ceph-mon[35142]: pgmap v711: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 10 MiB/s wr, 916 op/s 2024-04-01T01:45:14.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:14 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:45:14.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:14 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:45:15.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:14 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:45:15.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:15 smithi081 ceph-mon[35142]: pgmap v712: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 10 MiB/s wr, 814 op/s 2024-04-01T01:45:15.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:15 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:15.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:15 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:15.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:15 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:15.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:15 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:15.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:15 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:15.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:15 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:15.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:15 smithi189 ceph-mon[35753]: pgmap v712: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 10 MiB/s wr, 814 op/s 2024-04-01T01:45:15.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:15 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:15.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:15 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:15.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:15 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:15.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:15 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:15.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:15 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:15.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:15 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:16.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:15 smithi070 ceph-mon[32179]: pgmap v712: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 10 MiB/s wr, 814 op/s 2024-04-01T01:45:16.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:15 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:16.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:15 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:16.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:15 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:16.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:15 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:16.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:15 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:16.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:15 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:16.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:45:16.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:45:16.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:16 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:16.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 630.0 s, 669.8 tps, lat 127.738 ms stddev 131.708 2024-04-01T01:45:16.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:45:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:45:16.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:16 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:17.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:45:17.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:45:17.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:16 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:45:17.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:17 smithi081 ceph-mon[35142]: pgmap v713: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 10 MiB/s wr, 782 op/s 2024-04-01T01:45:17.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:17 smithi189 ceph-mon[35753]: pgmap v713: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 10 MiB/s wr, 782 op/s 2024-04-01T01:45:18.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:17 smithi070 ceph-mon[32179]: pgmap v713: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 10 MiB/s wr, 782 op/s 2024-04-01T01:45:19.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:18 smithi189 ceph-mon[35753]: pgmap v714: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 799 op/s 2024-04-01T01:45:19.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:18 smithi070 ceph-mon[32179]: pgmap v714: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 799 op/s 2024-04-01T01:45:19.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:18 smithi081 ceph-mon[35142]: pgmap v714: 129 pgs: 129 active+clean; 1.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 11 MiB/s wr, 799 op/s 2024-04-01T01:45:21.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 635.0 s, 988.5 tps, lat 117.920 ms stddev 181.026 2024-04-01T01:45:22.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:21 smithi070 ceph-mon[32179]: pgmap v715: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 12 MiB/s wr, 756 op/s 2024-04-01T01:45:22.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:21 smithi081 ceph-mon[35142]: pgmap v715: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 12 MiB/s wr, 756 op/s 2024-04-01T01:45:22.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:21 smithi189 ceph-mon[35753]: pgmap v715: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 12 MiB/s wr, 756 op/s 2024-04-01T01:45:23.013 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.011+0000 7f4845a02640 1 -- 172.21.15.70:0/3957971007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f484010b0c0 msgr2=0x7f484010d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:23.013 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.011+0000 7f4845a02640 1 --2- 172.21.15.70:0/3957971007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f484010b0c0 0x7f484010d530 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f483801acb0 tx=0x7f483803d130 comp rx=0 tx=0).stop 2024-04-01T01:45:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.011+0000 7f4845a02640 1 -- 172.21.15.70:0/3957971007 shutdown_connections 2024-04-01T01:45:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.011+0000 7f4845a02640 1 --2- 172.21.15.70:0/3957971007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f484010da70 0x7f484010def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.011+0000 7f4845a02640 1 --2- 172.21.15.70:0/3957971007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f484010b0c0 0x7f484010d530 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.011+0000 7f4845a02640 1 --2- 172.21.15.70:0/3957971007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4840108750 0x7f484010ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.011+0000 7f4845a02640 1 -- 172.21.15.70:0/3957971007 >> 172.21.15.70:0/3957971007 conn(0x7f484006e790 msgr2=0x7f4840070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:23.018 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.016+0000 7f4845a02640 1 -- 172.21.15.70:0/3957971007 shutdown_connections 2024-04-01T01:45:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.018+0000 7f4845a02640 1 -- 172.21.15.70:0/3957971007 wait complete. 2024-04-01T01:45:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.018+0000 7f4845a02640 1 Processor -- start 2024-04-01T01:45:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.019+0000 7f4845a02640 1 -- start start 2024-04-01T01:45:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.019+0000 7f4845a02640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4840108750 0x7f484019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.019+0000 7f4845a02640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f484010b0c0 0x7f484019e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.019+0000 7f4845a02640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f484010da70 0x7f48401a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.019 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.019+0000 7f4845a02640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4840111880 con 0x7f4840108750 2024-04-01T01:45:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.019+0000 7f4845a02640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4840111700 con 0x7f484010da70 2024-04-01T01:45:23.020 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.019+0000 7f4845a02640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4840111a00 con 0x7f484010b0c0 2024-04-01T01:45:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.020+0000 7f483e7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f484010b0c0 0x7f484019e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.020+0000 7f483f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f484010da70 0x7f48401a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.020+0000 7f483f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f484010da70 0x7f48401a5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53322/0 (socket says 172.21.15.70:53322) 2024-04-01T01:45:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.020+0000 7f483f7fe640 1 -- 172.21.15.70:0/1537639417 learned_addr learned my addr 172.21.15.70:0/1537639417 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.021+0000 7f483f7fe640 1 -- 172.21.15.70:0/1537639417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f484010b0c0 msgr2=0x7f484019e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:23.021 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.021+0000 7f483f7fe640 1 --2- 172.21.15.70:0/1537639417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f484010b0c0 0x7f484019e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.021+0000 7f483f7fe640 1 -- 172.21.15.70:0/1537639417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4840108750 msgr2=0x7f484019de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.021+0000 7f483f7fe640 1 --2- 172.21.15.70:0/1537639417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4840108750 0x7f484019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.021+0000 7f483f7fe640 1 -- 172.21.15.70:0/1537639417 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48401a5aa0 con 0x7f484010da70 2024-04-01T01:45:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.021+0000 7f483f7fe640 1 --2- 172.21.15.70:0/1537639417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f484010da70 0x7f48401a5430 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f483000d950 tx=0x7f483000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:23.022 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.022+0000 7f4844a00640 1 -- 172.21.15.70:0/1537639417 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830017030 con 0x7f484010da70 2024-04-01T01:45:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.022+0000 7f4845a02640 1 -- 172.21.15.70:0/1537639417 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48401a5d90 con 0x7f484010da70 2024-04-01T01:45:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.022+0000 7f4845a02640 1 -- 172.21.15.70:0/1537639417 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48401a6280 con 0x7f484010da70 2024-04-01T01:45:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.022+0000 7f4844a00640 1 -- 172.21.15.70:0/1537639417 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4830004020 con 0x7f484010da70 2024-04-01T01:45:23.023 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.022+0000 7f4844a00640 1 -- 172.21.15.70:0/1537639417 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830003b20 con 0x7f484010da70 2024-04-01T01:45:23.027 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.027+0000 7f48485dd740 1 Processor -- start 2024-04-01T01:45:23.028 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.028+0000 7f4844a00640 1 -- 172.21.15.70:0/1537639417 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f483001a070 con 0x7f484010da70 2024-04-01T01:45:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.029+0000 7f4844a00640 1 --2- 172.21.15.70:0/1537639417 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f48280a2730 0x7f48280a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.029 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.029+0000 7f4844a00640 1 -- 172.21.15.70:0/1537639417 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f48300ce520 con 0x7f484010da70 2024-04-01T01:45:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.031+0000 7f48485dd740 1 -- start start 2024-04-01T01:45:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.031+0000 7f48485dd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 0x557adf84dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.031+0000 7f48485dd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf84e2c0 0x557adf852dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.031+0000 7f48485dd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf853430 0x557adf8558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.031+0000 7f48485dd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557adf7b18f0 con 0x557adf84d8d0 2024-04-01T01:45:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.031+0000 7f48485dd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x557adf75a8d0 con 0x557adf853430 2024-04-01T01:45:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.031+0000 7f48485dd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557adf7ad990 con 0x557adf84e2c0 2024-04-01T01:45:23.033 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.031+0000 7f483e7fc640 1 --2- 172.21.15.70:0/1537639417 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f48280a2730 0x7f48280a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.033+0000 7f483f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf853430 0x557adf8558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.034+0000 7f483f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf853430 0x557adf8558c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53332/0 (socket says 172.21.15.70:53332) 2024-04-01T01:45:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.034+0000 7f483f7fe640 1 -- 172.21.15.70:0/3737681501 learned_addr learned my addr 172.21.15.70:0/3737681501 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.034+0000 7f483f7fe640 1 -- 172.21.15.70:0/3737681501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf84e2c0 msgr2=0x557adf852dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:45:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.034+0000 7f483f7fe640 1 --2- 172.21.15.70:0/3737681501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf84e2c0 0x557adf852dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.034+0000 7f483f7fe640 1 -- 172.21.15.70:0/3737681501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 msgr2=0x557adf84dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.034+0000 7f483f7fe640 1 --2- 172.21.15.70:0/3737681501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 0x557adf84dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.035 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.034+0000 7f483f7fe640 1 -- 172.21.15.70:0/3737681501 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557adf4c6210 con 0x557adf853430 2024-04-01T01:45:23.040 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.040+0000 7f483e7fc640 1 --2- 172.21.15.70:0/1537639417 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f48280a2730 0x7f48280a4bf0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x557adf84ed30 tx=0x7f4838000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:23.041 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.041+0000 7f483f7fe640 1 --2- 172.21.15.70:0/3737681501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf853430 0x557adf8558c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f483009a8b0 tx=0x7f483009b2b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5c0d5ed973a1cf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:23.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.047+0000 7f4825ffb640 1 -- 172.21.15.70:0/3737681501 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48300b3070 con 0x557adf853430 2024-04-01T01:45:23.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.047+0000 7f4825ffb640 1 -- 172.21.15.70:0/3737681501 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f483009ed00 con 0x557adf853430 2024-04-01T01:45:23.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.048+0000 7f48485dd740 1 -- 172.21.15.70:0/3737681501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf853430 msgr2=0x557adf8558c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:23.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.048+0000 7f48485dd740 1 --2- 172.21.15.70:0/3737681501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf853430 0x557adf8558c0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f483009a8b0 tx=0x7f483009b2b0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.050+0000 7f48485dd740 1 -- 172.21.15.70:0/3737681501 shutdown_connections 2024-04-01T01:45:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.050+0000 7f48485dd740 1 --2- 172.21.15.70:0/3737681501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf853430 0x557adf8558c0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.050+0000 7f48485dd740 1 --2- 172.21.15.70:0/3737681501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf84e2c0 0x557adf852dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.050+0000 7f48485dd740 1 --2- 172.21.15.70:0/3737681501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 0x557adf84dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.050+0000 7f48485dd740 1 -- 172.21.15.70:0/3737681501 >> 172.21.15.70:0/3737681501 conn(0x557adf848e40 msgr2=0x557adf849240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:23.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.054+0000 7f48485dd740 1 -- 172.21.15.70:0/3737681501 shutdown_connections 2024-04-01T01:45:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.056+0000 7f48485dd740 1 -- 172.21.15.70:0/3737681501 wait complete. 2024-04-01T01:45:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.056+0000 7f48485dd740 1 -- start start 2024-04-01T01:45:23.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.056+0000 7f48485dd740 1 Processor -- start 2024-04-01T01:45:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.060+0000 7f48485dd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 0x557adf857e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.060+0000 7f48485dd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf84e2c0 0x557adf858370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.061 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.060+0000 7f48485dd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf853430 0x557adf85f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.060+0000 7f48485dd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557adf856430 con 0x557adf84d8d0 2024-04-01T01:45:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.060+0000 7f48485dd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x557adf74ff80 con 0x557adf84e2c0 2024-04-01T01:45:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.060+0000 7f48485dd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557adf8565b0 con 0x557adf853430 2024-04-01T01:45:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.060+0000 7f48485dd740 10 client.0 _ll_register_callbacks cb 0x557adf7a8290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.061+0000 7f483effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 0x557adf857e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.061+0000 7f483effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 0x557adf857e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:59954/0 (socket says 172.21.15.70:59954) 2024-04-01T01:45:23.062 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.061+0000 7f483effd640 1 -- 172.21.15.70:0/1724642230 learned_addr learned my addr 172.21.15.70:0/1724642230 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:23.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.061+0000 7f483effd640 1 -- 172.21.15.70:0/1724642230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf853430 msgr2=0x557adf85f3b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:45:23.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.061+0000 7f483effd640 1 --2- 172.21.15.70:0/1724642230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf853430 0x557adf85f3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.061+0000 7f483effd640 1 -- 172.21.15.70:0/1724642230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf84e2c0 msgr2=0x557adf858370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:23.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.061+0000 7f483effd640 1 --2- 172.21.15.70:0/1724642230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf84e2c0 0x557adf858370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.062+0000 7f483effd640 1 -- 172.21.15.70:0/1724642230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557adf592620 con 0x557adf84d8d0 2024-04-01T01:45:23.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.062+0000 7f483effd640 1 --2- 172.21.15.70:0/1724642230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 0x557adf857e30 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f483400bc90 tx=0x7f483400d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c268f5dde83f04b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:23.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.062+0000 7f481effd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:45:23.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.063+0000 7f481effd640 1 -- 172.21.15.70:0/1724642230 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4834016020 con 0x557adf84d8d0 2024-04-01T01:45:23.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.063+0000 7f48485dd740 1 -- 172.21.15.70:0/1724642230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557adf7c5120 con 0x557adf84d8d0 2024-04-01T01:45:23.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.063+0000 7f481effd640 1 -- 172.21.15.70:0/1724642230 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f4834010660 con 0x557adf84d8d0 2024-04-01T01:45:23.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.063+0000 7f481effd640 1 -- 172.21.15.70:0/1724642230 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4834021050 con 0x557adf84d8d0 2024-04-01T01:45:23.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.063+0000 7f48485dd740 10 client.18915 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:23.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.064+0000 7f48485dd740 1 -- 172.21.15.70:0/1724642230 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557adf862520 con 0x557adf84d8d0 2024-04-01T01:45:23.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.064+0000 7f481effd640 1 -- 172.21.15.70:0/1724642230 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x557adf7c5120 con 0x557adf84d8d0 2024-04-01T01:45:23.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.064+0000 7f481effd640 1 -- 172.21.15.70:0/1724642230 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f483400f9e0 con 0x557adf84d8d0 2024-04-01T01:45:23.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.065+0000 7f48485dd740 10 client.18915 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:23.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.065+0000 7f48485dd740 10 client.18915 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:23.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.065+0000 7f48485dd740 20 client.18915 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:23.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.065+0000 7f48485dd740 1 --2- 172.21.15.70:0/1724642230 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557adf8640a0 0x557adf866560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.065+0000 7f48485dd740 4 client.18915 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:45:23.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.065+0000 7f48485dd740 1 -- 172.21.15.70:0/1724642230 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557adf7bee30 con 0x557adf8640a0 2024-04-01T01:45:23.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.066+0000 7f483e7fc640 1 --2- 172.21.15.70:0/1724642230 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557adf8640a0 0x557adf866560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.067+0000 7f483e7fc640 1 --2- 172.21.15.70:0/1724642230 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557adf8640a0 0x557adf866560 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b85accc7c124f26 server_cookie=ab2648100ff3c15f in_seq=0 out_seq=0 2024-04-01T01:45:23.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.067+0000 7f481effd640 10 client.18915 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:23.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.068+0000 7f481effd640 1 -- 172.21.15.70:0/1724642230 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557adf7bee30 con 0x557adf8640a0 2024-04-01T01:45:23.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.068+0000 7f481effd640 10 client.18915 handle_command_reply: tid=0 2024-04-01T01:45:23.068 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.068+0000 7f48485dd740 1 client.18915 shutdown 2024-04-01T01:45:23.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.074+0000 7f48485dd740 1 -- 172.21.15.70:0/1724642230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 msgr2=0x557adf857e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:23.075 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.074+0000 7f48485dd740 1 --2- 172.21.15.70:0/1724642230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 0x557adf857e30 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f483400bc90 tx=0x7f483400d6b0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.076+0000 7f48485dd740 1 -- 172.21.15.70:0/1724642230 shutdown_connections 2024-04-01T01:45:23.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.076+0000 7f48485dd740 1 --2- 172.21.15.70:0/1724642230 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557adf8640a0 0x557adf866560 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.076+0000 7f48485dd740 1 --2- 172.21.15.70:0/1724642230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf853430 0x557adf85f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.076+0000 7f48485dd740 1 --2- 172.21.15.70:0/1724642230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf84e2c0 0x557adf858370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.076+0000 7f48485dd740 1 --2- 172.21.15.70:0/1724642230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 0x557adf857e30 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.076+0000 7f48485dd740 1 -- 172.21.15.70:0/1724642230 >> 172.21.15.70:0/1724642230 conn(0x557adf848e40 msgr2=0x557adf847f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:23.077 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.076+0000 7f481effd640 0 client.18915 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:23.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.077+0000 7f48485dd740 1 -- 172.21.15.70:0/1724642230 shutdown_connections 2024-04-01T01:45:23.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.077+0000 7f48485dd740 1 -- 172.21.15.70:0/1724642230 wait complete. 2024-04-01T01:45:23.078 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.077+0000 7f48485dd740 20 client.18915 trim_cache size 0 max 16384 2024-04-01T01:45:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.112+0000 7f48485dd740 1 Processor -- start 2024-04-01T01:45:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.114+0000 7f48485dd740 1 -- start start 2024-04-01T01:45:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.114+0000 7f48485dd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 0x557adf84c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.114+0000 7f48485dd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf84e2c0 0x557adf85f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.114+0000 7f48485dd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf853430 0x557adf84c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.114+0000 7f48485dd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f482c00fe30 con 0x557adf84d8d0 2024-04-01T01:45:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.114+0000 7f48485dd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f482c011970 con 0x557adf84e2c0 2024-04-01T01:45:23.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.114+0000 7f48485dd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f482c010500 con 0x557adf853430 2024-04-01T01:45:23.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.115+0000 7f483f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf853430 0x557adf84c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:23.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.115+0000 7f483f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf853430 0x557adf84c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46900/0 (socket says 172.21.15.70:46900) 2024-04-01T01:45:23.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.115+0000 7f483f7fe640 1 -- 172.21.15.70:0/2870495584 learned_addr learned my addr 172.21.15.70:0/2870495584 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:23.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.115+0000 7f483e7fc640 1 --2- 172.21.15.70:0/2870495584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf84e2c0 0x557adf85f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.115+0000 7f483f7fe640 1 -- 172.21.15.70:0/2870495584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf84e2c0 msgr2=0x557adf85f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.115+0000 7f483f7fe640 1 --2- 172.21.15.70:0/2870495584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf84e2c0 0x557adf85f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.115+0000 7f483f7fe640 1 -- 172.21.15.70:0/2870495584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 msgr2=0x557adf84c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.115+0000 7f483f7fe640 1 --2- 172.21.15.70:0/2870495584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 0x557adf84c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.115+0000 7f483f7fe640 1 -- 172.21.15.70:0/2870495584 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557adf84cee0 con 0x557adf853430 2024-04-01T01:45:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.115+0000 7f483f7fe640 1 --2- 172.21.15.70:0/2870495584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf853430 0x557adf84c5e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4830097580 tx=0x7f483000b600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9c3d3c567fc1354 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.119+0000 7f48257fa640 1 -- 172.21.15.70:0/2870495584 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48300b7070 con 0x557adf853430 2024-04-01T01:45:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.119+0000 7f48257fa640 1 -- 172.21.15.70:0/2870495584 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f483009ed00 con 0x557adf853430 2024-04-01T01:45:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.119+0000 7f48257fa640 1 -- 172.21.15.70:0/2870495584 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48300b3050 con 0x557adf853430 2024-04-01T01:45:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.120+0000 7f48485dd740 1 -- 172.21.15.70:0/2870495584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf853430 msgr2=0x557adf84c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.120+0000 7f48485dd740 1 --2- 172.21.15.70:0/2870495584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf853430 0x557adf84c5e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4830097580 tx=0x7f483000b600 comp rx=0 tx=0).stop 2024-04-01T01:45:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.120+0000 7f48485dd740 1 -- 172.21.15.70:0/2870495584 shutdown_connections 2024-04-01T01:45:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.120+0000 7f48485dd740 1 --2- 172.21.15.70:0/2870495584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf853430 0x557adf84c5e0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.120+0000 7f48485dd740 1 --2- 172.21.15.70:0/2870495584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf84e2c0 0x557adf85f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.120+0000 7f48485dd740 1 --2- 172.21.15.70:0/2870495584 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 0x557adf84c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.120+0000 7f48485dd740 1 -- 172.21.15.70:0/2870495584 >> 172.21.15.70:0/2870495584 conn(0x557adf848e40 msgr2=0x557adf849220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:23.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.122+0000 7f48485dd740 1 -- 172.21.15.70:0/2870495584 shutdown_connections 2024-04-01T01:45:23.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.124+0000 7f48485dd740 1 -- 172.21.15.70:0/2870495584 wait complete. 2024-04-01T01:45:23.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.125+0000 7f48485dd740 1 -- start start 2024-04-01T01:45:23.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.125+0000 7f48485dd740 1 Processor -- start 2024-04-01T01:45:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.128+0000 7f48485dd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 0x557adf85f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.128+0000 7f48485dd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf84e2c0 0x557adf85bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.128+0000 7f48485dd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf853430 0x557adf85c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.128+0000 7f48485dd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f482c0112a0 con 0x557adf84d8d0 2024-04-01T01:45:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.128+0000 7f48485dd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f482c010bd0 con 0x557adf853430 2024-04-01T01:45:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.128+0000 7f48485dd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f482c00de10 con 0x557adf84e2c0 2024-04-01T01:45:23.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.128+0000 7f48485dd740 10 client.0 _ll_register_callbacks cb 0x7f483400b2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.128+0000 7f483e7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf84e2c0 0x557adf85bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.129+0000 7f483e7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf84e2c0 0x557adf85bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46904/0 (socket says 172.21.15.70:46904) 2024-04-01T01:45:23.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.129+0000 7f483e7fc640 1 -- 172.21.15.70:0/2306629656 learned_addr learned my addr 172.21.15.70:0/2306629656 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:23.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.129+0000 7f483e7fc640 1 -- 172.21.15.70:0/2306629656 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf853430 msgr2=0x557adf85c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.129+0000 7f483e7fc640 1 --2- 172.21.15.70:0/2306629656 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf853430 0x557adf85c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.129+0000 7f483e7fc640 1 -- 172.21.15.70:0/2306629656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 msgr2=0x557adf85f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.129+0000 7f483e7fc640 1 --2- 172.21.15.70:0/2306629656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 0x557adf85f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.129+0000 7f483e7fc640 1 -- 172.21.15.70:0/2306629656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557adf8c8490 con 0x557adf84e2c0 2024-04-01T01:45:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.129+0000 7f483e7fc640 1 --2- 172.21.15.70:0/2306629656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf84e2c0 0x557adf85bc00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f483803d740 tx=0x7f483801a980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89ddf204ed02dd68 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.129+0000 7f481effd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:45:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.130+0000 7f481effd640 1 -- 172.21.15.70:0/2306629656 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483804a070 con 0x557adf84e2c0 2024-04-01T01:45:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.130+0000 7f48485dd740 1 -- 172.21.15.70:0/2306629656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f483009ed00 con 0x557adf84e2c0 2024-04-01T01:45:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.130+0000 7f481effd640 1 -- 172.21.15.70:0/2306629656 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f4838061560 con 0x557adf84e2c0 2024-04-01T01:45:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.131+0000 7f481effd640 1 -- 172.21.15.70:0/2306629656 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4838061760 con 0x557adf84e2c0 2024-04-01T01:45:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.132+0000 7f48485dd740 10 client.28768 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:23.134 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.132+0000 7f48485dd740 1 -- 172.21.15.70:0/2306629656 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557adf8c9730 con 0x557adf84e2c0 2024-04-01T01:45:23.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.134+0000 7f481effd640 1 -- 172.21.15.70:0/2306629656 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f483009ed00 con 0x557adf84e2c0 2024-04-01T01:45:23.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.135+0000 7f481effd640 1 -- 172.21.15.70:0/2306629656 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f483805a040 con 0x557adf84e2c0 2024-04-01T01:45:23.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.136+0000 7f48485dd740 10 client.28768 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:23.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.136+0000 7f48485dd740 10 client.28768 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:23.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.136+0000 7f48485dd740 20 client.28768 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:23.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.136+0000 7f48485dd740 1 --2- 172.21.15.70:0/2306629656 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557adf8ca090 0x557adf8cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:23.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.136+0000 7f48485dd740 4 client.28768 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:45:23.137 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.136+0000 7f48485dd740 1 -- 172.21.15.70:0/2306629656 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557adf7b9180 con 0x557adf8ca090 2024-04-01T01:45:23.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.137+0000 7f483effd640 1 --2- 172.21.15.70:0/2306629656 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557adf8ca090 0x557adf8cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.140+0000 7f483effd640 1 --2- 172.21.15.70:0/2306629656 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557adf8ca090 0x557adf8cc4d0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88452f754b4aee24 server_cookie=937ed6d80a9d0546 in_seq=0 out_seq=0 2024-04-01T01:45:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.140+0000 7f481effd640 10 client.28768 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.562+0000 7f481effd640 1 -- 172.21.15.70:0/2306629656 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557adf7b9180 con 0x557adf8ca090 2024-04-01T01:45:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.562+0000 7f481effd640 10 client.28768 handle_command_reply: tid=0 2024-04-01T01:45:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.562+0000 7f48485dd740 1 client.28768 shutdown 2024-04-01T01:45:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.563+0000 7f48485dd740 1 -- 172.21.15.70:0/2306629656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf84e2c0 msgr2=0x557adf85bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:23.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.563+0000 7f48485dd740 1 --2- 172.21.15.70:0/2306629656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf84e2c0 0x557adf85bc00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f483803d740 tx=0x7f483801a980 comp rx=0 tx=0).stop 2024-04-01T01:45:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.566+0000 7f48485dd740 1 -- 172.21.15.70:0/2306629656 shutdown_connections 2024-04-01T01:45:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.566+0000 7f48485dd740 1 --2- 172.21.15.70:0/2306629656 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557adf8ca090 0x557adf8cc4d0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.566+0000 7f48485dd740 1 --2- 172.21.15.70:0/2306629656 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557adf853430 0x557adf85c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.566+0000 7f48485dd740 1 --2- 172.21.15.70:0/2306629656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557adf84e2c0 0x557adf85bc00 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.566+0000 7f48485dd740 1 --2- 172.21.15.70:0/2306629656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557adf84d8d0 0x557adf85f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.566+0000 7f48485dd740 1 -- 172.21.15.70:0/2306629656 >> 172.21.15.70:0/2306629656 conn(0x557adf848e40 msgr2=0x557adf84c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.566+0000 7f481effd640 0 client.28768 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.566+0000 7f48485dd740 1 -- 172.21.15.70:0/2306629656 shutdown_connections 2024-04-01T01:45:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.566+0000 7f48485dd740 1 -- 172.21.15.70:0/2306629656 wait complete. 2024-04-01T01:45:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.567+0000 7f48485dd740 20 client.28768 trim_cache size 0 max 16384 2024-04-01T01:45:23.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:45:23.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:45:23.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:45:23.567 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:45:23.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.571+0000 7f4845a02640 1 -- 172.21.15.70:0/1537639417 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f48280a2730 msgr2=0x7f48280a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.571+0000 7f4845a02640 1 --2- 172.21.15.70:0/1537639417 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f48280a2730 0x7f48280a4bf0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x557adf84ed30 tx=0x7f4838000f40 comp rx=0 tx=0).stop 2024-04-01T01:45:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.571+0000 7f4845a02640 1 -- 172.21.15.70:0/1537639417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f484010da70 msgr2=0x7f48401a5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.571+0000 7f4845a02640 1 --2- 172.21.15.70:0/1537639417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f484010da70 0x7f48401a5430 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f483000d950 tx=0x7f483000de10 comp rx=0 tx=0).stop 2024-04-01T01:45:23.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.573+0000 7f4845a02640 1 -- 172.21.15.70:0/1537639417 shutdown_connections 2024-04-01T01:45:23.572 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.573+0000 7f4845a02640 1 --2- 172.21.15.70:0/1537639417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f484010da70 0x7f48401a5430 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.573+0000 7f4845a02640 1 --2- 172.21.15.70:0/1537639417 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f48280a2730 0x7f48280a4bf0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.573+0000 7f4845a02640 1 --2- 172.21.15.70:0/1537639417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f484010b0c0 0x7f484019e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.573+0000 7f4845a02640 1 --2- 172.21.15.70:0/1537639417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4840108750 0x7f484019de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.573+0000 7f4845a02640 1 -- 172.21.15.70:0/1537639417 >> 172.21.15.70:0/1537639417 conn(0x7f484006e790 msgr2=0x7f484010b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.573+0000 7f4845a02640 1 -- 172.21.15.70:0/1537639417 shutdown_connections 2024-04-01T01:45:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:23.573+0000 7f4845a02640 1 -- 172.21.15.70:0/1537639417 wait complete. 2024-04-01T01:45:24.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:23 smithi189 ceph-mon[35753]: pgmap v716: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 11 MiB/s wr, 780 op/s 2024-04-01T01:45:24.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:23 smithi070 ceph-mon[32179]: pgmap v716: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 11 MiB/s wr, 780 op/s 2024-04-01T01:45:24.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:23 smithi081 ceph-mon[35142]: pgmap v716: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 11 MiB/s wr, 780 op/s 2024-04-01T01:45:26.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:25 smithi189 ceph-mon[35753]: pgmap v717: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 11 MiB/s wr, 716 op/s 2024-04-01T01:45:26.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:25 smithi070 ceph-mon[32179]: pgmap v717: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 11 MiB/s wr, 716 op/s 2024-04-01T01:45:26.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:25 smithi081 ceph-mon[35142]: pgmap v717: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 11 MiB/s wr, 716 op/s 2024-04-01T01:45:26.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 640.0 s, 814.4 tps, lat 122.422 ms stddev 194.390 2024-04-01T01:45:28.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:27 smithi189 ceph-mon[35753]: pgmap v718: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 10 MiB/s wr, 774 op/s 2024-04-01T01:45:28.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:27 smithi070 ceph-mon[32179]: pgmap v718: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 10 MiB/s wr, 774 op/s 2024-04-01T01:45:28.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:27 smithi081 ceph-mon[35142]: pgmap v718: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 10 MiB/s wr, 774 op/s 2024-04-01T01:45:30.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:29 smithi189 ceph-mon[35753]: pgmap v719: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 12 MiB/s wr, 827 op/s 2024-04-01T01:45:30.256 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:29 smithi070 ceph-mon[32179]: pgmap v719: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 12 MiB/s wr, 827 op/s 2024-04-01T01:45:30.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:29 smithi081 ceph-mon[35142]: pgmap v719: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 12 MiB/s wr, 827 op/s 2024-04-01T01:45:31.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:45:31.688 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:45:31.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.881+0000 7f2223b56640 1 -- 172.21.15.70:0/1006717231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f221c1089c0 msgr2=0x7f221c10adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:31.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.881+0000 7f2223b56640 1 --2- 172.21.15.70:0/1006717231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f221c1089c0 0x7f221c10adb0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f220c009860 tx=0x7f220c02bb70 comp rx=0 tx=0).stop 2024-04-01T01:45:31.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.882+0000 7f2223b56640 1 -- 172.21.15.70:0/1006717231 shutdown_connections 2024-04-01T01:45:31.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.882+0000 7f2223b56640 1 --2- 172.21.15.70:0/1006717231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f221c10de00 0x7f221c1102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.882+0000 7f2223b56640 1 --2- 172.21.15.70:0/1006717231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f221c10b380 0x7f221c10d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.882+0000 7f2223b56640 1 --2- 172.21.15.70:0/1006717231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f221c1089c0 0x7f221c10adb0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.882+0000 7f2223b56640 1 -- 172.21.15.70:0/1006717231 >> 172.21.15.70:0/1006717231 conn(0x7f221c06e950 msgr2=0x7f221c070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:31.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.882+0000 7f2223b56640 1 -- 172.21.15.70:0/1006717231 shutdown_connections 2024-04-01T01:45:31.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.882+0000 7f2223b56640 1 -- 172.21.15.70:0/1006717231 wait complete. 2024-04-01T01:45:31.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.883+0000 7f2223b56640 1 Processor -- start 2024-04-01T01:45:31.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.883+0000 7f2223b56640 1 -- start start 2024-04-01T01:45:31.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.883+0000 7f2223b56640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f221c1089c0 0x7f221c19e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.883+0000 7f2223b56640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f221c10b380 0x7f221c19e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.883+0000 7f2223b56640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f221c10de00 0x7f221c1a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.883+0000 7f2223b56640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f221c113890 con 0x7f221c10b380 2024-04-01T01:45:31.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.883+0000 7f2223b56640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f221c113710 con 0x7f221c1089c0 2024-04-01T01:45:31.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.883+0000 7f2223b56640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f221c113a10 con 0x7f221c10de00 2024-04-01T01:45:31.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.883+0000 7f22210f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f221c10b380 0x7f221c19e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:31.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.883+0000 7f22210f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f221c10b380 0x7f221c19e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36028/0 (socket says 172.21.15.70:36028) 2024-04-01T01:45:31.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.883+0000 7f22210f1640 1 -- 172.21.15.70:0/381900020 learned_addr learned my addr 172.21.15.70:0/381900020 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:31.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.884+0000 7f22210f1640 1 -- 172.21.15.70:0/381900020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f221c10de00 msgr2=0x7f221c1a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:31.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.884+0000 7f22210f1640 1 --2- 172.21.15.70:0/381900020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f221c10de00 0x7f221c1a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.884+0000 7f22210f1640 1 -- 172.21.15.70:0/381900020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f221c1089c0 msgr2=0x7f221c19e000 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:45:31.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.884+0000 7f22210f1640 1 --2- 172.21.15.70:0/381900020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f221c1089c0 0x7f221c19e000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.884+0000 7f22210f1640 1 -- 172.21.15.70:0/381900020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f221c1b6c30 con 0x7f221c10b380 2024-04-01T01:45:31.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.885+0000 7f22210f1640 1 --2- 172.21.15.70:0/381900020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f221c10b380 0x7f221c19e540 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f221400a660 tx=0x7f221400ab20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:31.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.886+0000 7f2212ffd640 1 -- 172.21.15.70:0/381900020 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221400e770 con 0x7f221c10b380 2024-04-01T01:45:31.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.886+0000 7f2212ffd640 1 -- 172.21.15.70:0/381900020 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2214019070 con 0x7f221c10b380 2024-04-01T01:45:31.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.886+0000 7f2212ffd640 1 -- 172.21.15.70:0/381900020 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2214007d20 con 0x7f221c10b380 2024-04-01T01:45:31.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.886+0000 7f2223b56640 1 -- 172.21.15.70:0/381900020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f221c1b6ec0 con 0x7f221c10b380 2024-04-01T01:45:31.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.886+0000 7f2223b56640 1 -- 172.21.15.70:0/381900020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f221c1b7330 con 0x7f221c10b380 2024-04-01T01:45:31.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.891+0000 7f2226812740 1 Processor -- start 2024-04-01T01:45:31.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.891+0000 7f2226812740 1 -- start start 2024-04-01T01:45:31.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.891+0000 7f2226812740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 0x55fba7d1b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.891+0000 7f2226812740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 0x55fba7d207a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.891+0000 7f2226812740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 0x55fba7d232a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.891+0000 7f2226812740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fba7c80180 con 0x55fba7d1bca0 2024-04-01T01:45:31.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.891+0000 7f2226812740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55fba7c28b70 con 0x55fba7d20e10 2024-04-01T01:45:31.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.891+0000 7f2226812740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fba7c7c220 con 0x55fba7d1b2b0 2024-04-01T01:45:31.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.892+0000 7f22220f3640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 0x55fba7d1b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:31.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.892+0000 7f22220f3640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 0x55fba7d1b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58266/0 (socket says 172.21.15.70:58266) 2024-04-01T01:45:31.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.892+0000 7f22220f3640 1 -- 172.21.15.70:0/2866709435 learned_addr learned my addr 172.21.15.70:0/2866709435 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:31.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.892+0000 7f22220f3640 1 -- 172.21.15.70:0/2866709435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 msgr2=0x55fba7d232a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:45:31.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.892+0000 7f22220f3640 1 --2- 172.21.15.70:0/2866709435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 0x55fba7d232a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.892+0000 7f22220f3640 1 -- 172.21.15.70:0/2866709435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 msgr2=0x55fba7d207a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:31.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.892+0000 7f22220f3640 1 --2- 172.21.15.70:0/2866709435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 0x55fba7d207a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.892+0000 7f22220f3640 1 -- 172.21.15.70:0/2866709435 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fba7c3c7c0 con 0x55fba7d1b2b0 2024-04-01T01:45:31.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.893+0000 7f22220f3640 1 --2- 172.21.15.70:0/2866709435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 0x55fba7d1b6d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2218009c40 tx=0x7f221800e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32a6a6796b5d02e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:31.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.894+0000 7f2207fff640 1 -- 172.21.15.70:0/2866709435 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2218005020 con 0x55fba7d1b2b0 2024-04-01T01:45:31.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.894+0000 7f2207fff640 1 -- 172.21.15.70:0/2866709435 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22180047a0 con 0x55fba7d1b2b0 2024-04-01T01:45:31.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.894+0000 7f2212ffd640 1 -- 172.21.15.70:0/381900020 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f2214007530 con 0x7f221c10b380 2024-04-01T01:45:31.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.894+0000 7f2207fff640 1 -- 172.21.15.70:0/2866709435 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2218010690 con 0x55fba7d1b2b0 2024-04-01T01:45:31.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.895+0000 7f2226812740 1 -- 172.21.15.70:0/2866709435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 msgr2=0x55fba7d1b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:31.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.895+0000 7f2226812740 1 --2- 172.21.15.70:0/2866709435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 0x55fba7d1b6d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2218009c40 tx=0x7f221800e720 comp rx=0 tx=0).stop 2024-04-01T01:45:31.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.895+0000 7f2212ffd640 1 --2- 172.21.15.70:0/381900020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22080a25f0 0x7f22080a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.895+0000 7f2226812740 1 -- 172.21.15.70:0/2866709435 shutdown_connections 2024-04-01T01:45:31.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.895+0000 7f2226812740 1 --2- 172.21.15.70:0/2866709435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 0x55fba7d232a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.895+0000 7f2226812740 1 --2- 172.21.15.70:0/2866709435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 0x55fba7d207a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.895+0000 7f2226812740 1 --2- 172.21.15.70:0/2866709435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 0x55fba7d1b6d0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.895+0000 7f2226812740 1 -- 172.21.15.70:0/2866709435 >> 172.21.15.70:0/2866709435 conn(0x55fba7d16820 msgr2=0x55fba7d16c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:31.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.895+0000 7f2212ffd640 1 -- 172.21.15.70:0/381900020 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f22140ce6a0 con 0x7f221c10b380 2024-04-01T01:45:31.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.896+0000 7f22220f3640 1 --2- 172.21.15.70:0/381900020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22080a25f0 0x7f22080a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:31.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 645.0 s, 841.4 tps, lat 119.793 ms stddev 111.828 2024-04-01T01:45:31.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.896+0000 7f22220f3640 1 --2- 172.21.15.70:0/381900020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22080a25f0 0x7f22080a4ab0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f22180057c0 tx=0x7f2218005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:31.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.898+0000 7f2226812740 1 -- 172.21.15.70:0/2866709435 shutdown_connections 2024-04-01T01:45:31.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.898+0000 7f2226812740 1 -- 172.21.15.70:0/2866709435 wait complete. 2024-04-01T01:45:31.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.898+0000 7f2226812740 1 -- start start 2024-04-01T01:45:31.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.898+0000 7f2226812740 1 Processor -- start 2024-04-01T01:45:31.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.899+0000 7f2226812740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 0x55fba7d25810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.899+0000 7f2226812740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 0x55fba7d25d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.899+0000 7f2226812740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 0x55fba7d2cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.899+0000 7f2226812740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fba7d23e60 con 0x55fba7d1bca0 2024-04-01T01:45:31.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.899+0000 7f2226812740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55fba7c1e4a0 con 0x55fba7d20e10 2024-04-01T01:45:31.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.899+0000 7f2226812740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fba7d23fe0 con 0x55fba7d1b2b0 2024-04-01T01:45:31.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.899+0000 7f2226812740 10 client.0 _ll_register_callbacks cb 0x55fba7c21d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:31.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.899+0000 7f22220f3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 0x55fba7d2cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:31.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.899+0000 7f22220f3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 0x55fba7d2cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41004/0 (socket says 172.21.15.70:41004) 2024-04-01T01:45:31.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.899+0000 7f22220f3640 1 -- 172.21.15.70:0/1490873678 learned_addr learned my addr 172.21.15.70:0/1490873678 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:31.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.900+0000 7f22218f2640 1 --2- 172.21.15.70:0/1490873678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 0x55fba7d25810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:31.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.900+0000 7f22220f3640 1 -- 172.21.15.70:0/1490873678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 msgr2=0x55fba7d25810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:31.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.900+0000 7f22220f3640 1 --2- 172.21.15.70:0/1490873678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 0x55fba7d25810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.900+0000 7f22220f3640 1 -- 172.21.15.70:0/1490873678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 msgr2=0x55fba7d25d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:31.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.900+0000 7f22220f3640 1 --2- 172.21.15.70:0/1490873678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 0x55fba7d25d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.900+0000 7f22220f3640 1 -- 172.21.15.70:0/1490873678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fba7a5e7d0 con 0x55fba7d20e10 2024-04-01T01:45:31.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.905+0000 7f22220f3640 1 --2- 172.21.15.70:0/1490873678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 0x55fba7d2cd90 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f2218018000 tx=0x7f221800e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93ec8b7ccdc3ac69 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:31.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.905+0000 7f22057fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:45:31.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.905+0000 7f22057fa640 1 -- 172.21.15.70:0/1490873678 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221802a070 con 0x55fba7d20e10 2024-04-01T01:45:31.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.905+0000 7f2226812740 1 -- 172.21.15.70:0/1490873678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22180047a0 con 0x55fba7d20e10 2024-04-01T01:45:31.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.906+0000 7f22057fa640 1 -- 172.21.15.70:0/1490873678 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f2218023e00 con 0x55fba7d20e10 2024-04-01T01:45:31.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.907+0000 7f22057fa640 1 -- 172.21.15.70:0/1490873678 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f221803c050 con 0x55fba7d20e10 2024-04-01T01:45:31.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.907+0000 7f2226812740 10 client.28985 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:31.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.907+0000 7f2226812740 1 -- 172.21.15.70:0/1490873678 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fba79943f0 con 0x55fba7d20e10 2024-04-01T01:45:31.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.908+0000 7f22057fa640 1 -- 172.21.15.70:0/1490873678 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f221802b040 con 0x55fba7d20e10 2024-04-01T01:45:31.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.908+0000 7f22057fa640 1 -- 172.21.15.70:0/1490873678 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f2218037030 con 0x55fba7d20e10 2024-04-01T01:45:31.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.908+0000 7f2226812740 10 client.28985 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:31.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.908+0000 7f2226812740 10 client.28985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:31.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.908+0000 7f2226812740 20 client.28985 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:31.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.908+0000 7f2226812740 1 --2- 172.21.15.70:0/1490873678 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55fba7d31ab0 0x55fba7d33f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.908+0000 7f2226812740 4 client.28985 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:45:31.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.908+0000 7f2226812740 1 -- 172.21.15.70:0/1490873678 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f22180047a0 con 0x55fba7d31ab0 2024-04-01T01:45:31.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.910+0000 7f22218f2640 1 --2- 172.21.15.70:0/1490873678 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55fba7d31ab0 0x55fba7d33f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:31.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.911+0000 7f22218f2640 1 --2- 172.21.15.70:0/1490873678 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55fba7d31ab0 0x55fba7d33f10 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfff60112dfd8bb1 server_cookie=e98301f189e1695 in_seq=0 out_seq=0 2024-04-01T01:45:31.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.913+0000 7f22057fa640 10 client.28985 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:31.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.913+0000 7f22057fa640 1 -- 172.21.15.70:0/1490873678 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f22180047a0 con 0x55fba7d31ab0 2024-04-01T01:45:31.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.913+0000 7f22057fa640 10 client.28985 handle_command_reply: tid=0 2024-04-01T01:45:31.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.913+0000 7f2226812740 1 client.28985 shutdown 2024-04-01T01:45:31.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.922+0000 7f2226812740 1 -- 172.21.15.70:0/1490873678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 msgr2=0x55fba7d2cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:31.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.922+0000 7f2226812740 1 --2- 172.21.15.70:0/1490873678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 0x55fba7d2cd90 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f2218018000 tx=0x7f221800e720 comp rx=0 tx=0).stop 2024-04-01T01:45:31.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.922+0000 7f2226812740 1 -- 172.21.15.70:0/1490873678 shutdown_connections 2024-04-01T01:45:31.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.922+0000 7f2226812740 1 --2- 172.21.15.70:0/1490873678 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55fba7d31ab0 0x55fba7d33f10 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.922+0000 7f2226812740 1 --2- 172.21.15.70:0/1490873678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 0x55fba7d2cd90 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.922+0000 7f2226812740 1 --2- 172.21.15.70:0/1490873678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 0x55fba7d25d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.922+0000 7f2226812740 1 --2- 172.21.15.70:0/1490873678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 0x55fba7d25810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.922+0000 7f2226812740 1 -- 172.21.15.70:0/1490873678 >> 172.21.15.70:0/1490873678 conn(0x55fba7d16820 msgr2=0x55fba7d15c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:31.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.922+0000 7f22057fa640 0 client.28985 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:31.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.923+0000 7f2226812740 1 -- 172.21.15.70:0/1490873678 shutdown_connections 2024-04-01T01:45:31.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.923+0000 7f2226812740 1 -- 172.21.15.70:0/1490873678 wait complete. 2024-04-01T01:45:31.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.925+0000 7f2226812740 20 client.28985 trim_cache size 0 max 16384 2024-04-01T01:45:31.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.961+0000 7f2226812740 1 Processor -- start 2024-04-01T01:45:31.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.961+0000 7f2226812740 1 -- start start 2024-04-01T01:45:31.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.962+0000 7f2226812740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d1b2b0 0x55fba7d16c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.962+0000 7f2226812740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 0x55fba7d2cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.962+0000 7f2226812740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d20e10 0x55fba7d15390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.962+0000 7f22218f2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d1b2b0 0x55fba7d16c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:31.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.962+0000 7f2226812740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f220000fd70 con 0x55fba7d1bca0 2024-04-01T01:45:31.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.962+0000 7f22218f2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d1b2b0 0x55fba7d16c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41018/0 (socket says 172.21.15.70:41018) 2024-04-01T01:45:31.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.962+0000 7f22218f2640 1 -- 172.21.15.70:0/560989313 learned_addr learned my addr 172.21.15.70:0/560989313 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:31.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.962+0000 7f2226812740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f22000118b0 con 0x55fba7d1b2b0 2024-04-01T01:45:31.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.962+0000 7f22210f1640 1 --2- 172.21.15.70:0/560989313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 0x55fba7d2cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:31.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.962+0000 7f2226812740 1 -- 172.21.15.70:0/560989313 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2200010440 con 0x55fba7d20e10 2024-04-01T01:45:31.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.962+0000 7f22220f3640 1 --2- 172.21.15.70:0/560989313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d20e10 0x55fba7d15390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:31.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.962+0000 7f22218f2640 1 -- 172.21.15.70:0/560989313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d20e10 msgr2=0x55fba7d15390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:31.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.962+0000 7f22218f2640 1 --2- 172.21.15.70:0/560989313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d20e10 0x55fba7d15390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.962+0000 7f22218f2640 1 -- 172.21.15.70:0/560989313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 msgr2=0x55fba7d2cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:31.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.962+0000 7f22218f2640 1 --2- 172.21.15.70:0/560989313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 0x55fba7d2cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.962+0000 7f22218f2640 1 -- 172.21.15.70:0/560989313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fba79540f0 con 0x55fba7d1b2b0 2024-04-01T01:45:31.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.963+0000 7f22220f3640 1 --2- 172.21.15.70:0/560989313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d20e10 0x55fba7d15390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:45:31.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.963+0000 7f22218f2640 1 --2- 172.21.15.70:0/560989313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d1b2b0 0x55fba7d16c00 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x55fba7d1c710 tx=0x7f220c005a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7be084f0b2ef87fa server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:31.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.963+0000 7f22077fe640 1 -- 172.21.15.70:0/560989313 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220c039040 con 0x55fba7d1b2b0 2024-04-01T01:45:31.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.963+0000 7f22077fe640 1 -- 172.21.15.70:0/560989313 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f220c004040 con 0x55fba7d1b2b0 2024-04-01T01:45:31.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.963+0000 7f22077fe640 1 -- 172.21.15.70:0/560989313 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220c043680 con 0x55fba7d1b2b0 2024-04-01T01:45:31.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.964+0000 7f2226812740 1 -- 172.21.15.70:0/560989313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d1b2b0 msgr2=0x55fba7d16c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:31.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.964+0000 7f2226812740 1 --2- 172.21.15.70:0/560989313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d1b2b0 0x55fba7d16c00 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x55fba7d1c710 tx=0x7f220c005a90 comp rx=0 tx=0).stop 2024-04-01T01:45:31.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.964+0000 7f2226812740 1 -- 172.21.15.70:0/560989313 shutdown_connections 2024-04-01T01:45:31.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.964+0000 7f2226812740 1 --2- 172.21.15.70:0/560989313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d20e10 0x55fba7d15390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.964+0000 7f2226812740 1 --2- 172.21.15.70:0/560989313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 0x55fba7d2cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.964+0000 7f2226812740 1 --2- 172.21.15.70:0/560989313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d1b2b0 0x55fba7d16c00 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.965+0000 7f2226812740 1 -- 172.21.15.70:0/560989313 >> 172.21.15.70:0/560989313 conn(0x55fba7d16820 msgr2=0x55fba7d1d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:31.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.965+0000 7f2226812740 1 -- 172.21.15.70:0/560989313 shutdown_connections 2024-04-01T01:45:31.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.965+0000 7f2226812740 1 -- 172.21.15.70:0/560989313 wait complete. 2024-04-01T01:45:31.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.965+0000 7f2226812740 1 -- start start 2024-04-01T01:45:31.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.965+0000 7f2226812740 1 Processor -- start 2024-04-01T01:45:31.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.966+0000 7f2226812740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 0x55fba7d2a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.966+0000 7f2226812740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 0x55fba7d2a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.966+0000 7f2226812740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 0x55fba7d2b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.966+0000 7f22210f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 0x55fba7d2a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:31.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.966+0000 7f22218f2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 0x55fba7d2a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:31.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.966+0000 7f22210f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 0x55fba7d2a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36062/0 (socket says 172.21.15.70:36062) 2024-04-01T01:45:31.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.966+0000 7f22210f1640 1 -- 172.21.15.70:0/565336915 learned_addr learned my addr 172.21.15.70:0/565336915 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:31.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.966+0000 7f2226812740 1 -- 172.21.15.70:0/565336915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f22000111e0 con 0x55fba7d1bca0 2024-04-01T01:45:31.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.966+0000 7f2226812740 1 -- 172.21.15.70:0/565336915 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2200010b10 con 0x55fba7d20e10 2024-04-01T01:45:31.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.966+0000 7f2226812740 1 -- 172.21.15.70:0/565336915 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f220000dd50 con 0x55fba7d1b2b0 2024-04-01T01:45:31.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.966+0000 7f22220f3640 1 --2- 172.21.15.70:0/565336915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 0x55fba7d2b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:31.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.966+0000 7f2226812740 10 client.0 _ll_register_callbacks cb 0x7f2218009b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:31.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.967+0000 7f22220f3640 1 -- 172.21.15.70:0/565336915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 msgr2=0x55fba7d2a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:31.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.967+0000 7f22220f3640 1 --2- 172.21.15.70:0/565336915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 0x55fba7d2a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.967+0000 7f22220f3640 1 -- 172.21.15.70:0/565336915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 msgr2=0x55fba7d2a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:31.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.967+0000 7f22220f3640 1 --2- 172.21.15.70:0/565336915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 0x55fba7d2a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.967+0000 7f22220f3640 1 -- 172.21.15.70:0/565336915 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fba7d2beb0 con 0x55fba7d20e10 2024-04-01T01:45:31.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.967+0000 7f22210f1640 1 --2- 172.21.15.70:0/565336915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 0x55fba7d2a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:45:31.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.967+0000 7f22220f3640 1 --2- 172.21.15.70:0/565336915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 0x55fba7d2b8c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f2218023040 tx=0x7f22180098d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2ff13cf44e1ad47 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:31.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.970+0000 7f22057fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:45:31.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.970+0000 7f22057fa640 1 -- 172.21.15.70:0/565336915 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2218044070 con 0x55fba7d20e10 2024-04-01T01:45:31.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.970+0000 7f22057fa640 1 -- 172.21.15.70:0/565336915 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f2218010430 con 0x55fba7d20e10 2024-04-01T01:45:31.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.970+0000 7f2226812740 1 -- 172.21.15.70:0/565336915 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f220c004040 con 0x55fba7d20e10 2024-04-01T01:45:31.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.971+0000 7f22057fa640 1 -- 172.21.15.70:0/565336915 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f221804e050 con 0x55fba7d20e10 2024-04-01T01:45:31.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.971+0000 7f2226812740 10 client.28991 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:31.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.971+0000 7f2226812740 1 -- 172.21.15.70:0/565336915 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fba7d97e10 con 0x55fba7d20e10 2024-04-01T01:45:31.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.971+0000 7f22057fa640 1 -- 172.21.15.70:0/565336915 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2218023840 con 0x55fba7d20e10 2024-04-01T01:45:31.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.972+0000 7f22057fa640 1 -- 172.21.15.70:0/565336915 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f2218037070 con 0x55fba7d20e10 2024-04-01T01:45:31.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.972+0000 7f2226812740 10 client.28991 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:31.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.972+0000 7f2226812740 10 client.28991 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:31.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.972+0000 7f2226812740 20 client.28991 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:31.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.972+0000 7f2226812740 1 --2- 172.21.15.70:0/565336915 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55fba7d986a0 0x55fba7d9aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:31.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.972+0000 7f2226812740 4 client.28991 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:45:31.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.972+0000 7f2226812740 1 -- 172.21.15.70:0/565336915 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f220c004040 con 0x55fba7d986a0 2024-04-01T01:45:31.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.973+0000 7f22218f2640 1 --2- 172.21.15.70:0/565336915 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55fba7d986a0 0x55fba7d9aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:31.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.973+0000 7f22218f2640 1 --2- 172.21.15.70:0/565336915 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55fba7d986a0 0x55fba7d9aac0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1373f892f7b96919 server_cookie=e0e18a0e6adacada in_seq=0 out_seq=0 2024-04-01T01:45:31.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.973+0000 7f22057fa640 10 client.28991 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:31.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.973+0000 7f22057fa640 1 -- 172.21.15.70:0/565336915 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f220c004040 con 0x55fba7d986a0 2024-04-01T01:45:31.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.974+0000 7f22057fa640 10 client.28991 handle_command_reply: tid=0 2024-04-01T01:45:31.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.974+0000 7f2226812740 1 client.28991 shutdown 2024-04-01T01:45:31.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.974+0000 7f2226812740 1 -- 172.21.15.70:0/565336915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 msgr2=0x55fba7d2b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:31.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.974+0000 7f2226812740 1 --2- 172.21.15.70:0/565336915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 0x55fba7d2b8c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f2218023040 tx=0x7f22180098d0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.974+0000 7f2226812740 1 -- 172.21.15.70:0/565336915 shutdown_connections 2024-04-01T01:45:31.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.974+0000 7f2226812740 1 --2- 172.21.15.70:0/565336915 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55fba7d986a0 0x55fba7d9aac0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.974+0000 7f2226812740 1 --2- 172.21.15.70:0/565336915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fba7d20e10 0x55fba7d2b8c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.974+0000 7f2226812740 1 --2- 172.21.15.70:0/565336915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fba7d1bca0 0x55fba7d2a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.974+0000 7f2226812740 1 --2- 172.21.15.70:0/565336915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba7d1b2b0 0x55fba7d2a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.974+0000 7f2226812740 1 -- 172.21.15.70:0/565336915 >> 172.21.15.70:0/565336915 conn(0x55fba7d16820 msgr2=0x55fba7d19a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:31.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.974+0000 7f22057fa640 0 client.28991 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:31.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.974+0000 7f2226812740 1 -- 172.21.15.70:0/565336915 shutdown_connections 2024-04-01T01:45:31.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.974+0000 7f2226812740 1 -- 172.21.15.70:0/565336915 wait complete. 2024-04-01T01:45:31.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.975+0000 7f2226812740 20 client.28991 trim_cache size 0 max 16384 2024-04-01T01:45:31.975 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:45:31.975 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:45:31.975 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:45:31.975 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:45:31.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.976+0000 7f2223b56640 1 -- 172.21.15.70:0/381900020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22080a25f0 msgr2=0x7f22080a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:31.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.976+0000 7f2223b56640 1 --2- 172.21.15.70:0/381900020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22080a25f0 0x7f22080a4ab0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f22180057c0 tx=0x7f2218005f90 comp rx=0 tx=0).stop 2024-04-01T01:45:31.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.976+0000 7f2223b56640 1 -- 172.21.15.70:0/381900020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f221c10b380 msgr2=0x7f221c19e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:31.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.976+0000 7f2223b56640 1 --2- 172.21.15.70:0/381900020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f221c10b380 0x7f221c19e540 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f221400a660 tx=0x7f221400ab20 comp rx=0 tx=0).stop 2024-04-01T01:45:31.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.977+0000 7f2223b56640 1 -- 172.21.15.70:0/381900020 shutdown_connections 2024-04-01T01:45:31.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.977+0000 7f2223b56640 1 --2- 172.21.15.70:0/381900020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f22080a25f0 0x7f22080a4ab0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.977+0000 7f2223b56640 1 --2- 172.21.15.70:0/381900020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f221c10de00 0x7f221c1a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.977+0000 7f2223b56640 1 --2- 172.21.15.70:0/381900020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f221c10b380 0x7f221c19e540 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.977+0000 7f2223b56640 1 --2- 172.21.15.70:0/381900020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f221c1089c0 0x7f221c19e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:31.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.977+0000 7f2223b56640 1 -- 172.21.15.70:0/381900020 >> 172.21.15.70:0/381900020 conn(0x7f221c06e950 msgr2=0x7f221c10a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:31.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.979+0000 7f2223b56640 1 -- 172.21.15.70:0/381900020 shutdown_connections 2024-04-01T01:45:31.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:31.979+0000 7f2223b56640 1 -- 172.21.15.70:0/381900020 wait complete. 2024-04-01T01:45:31.996 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:31 smithi070 ceph-mon[32179]: pgmap v720: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 10 MiB/s wr, 763 op/s 2024-04-01T01:45:31.997 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:45:31.997 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:45:32.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.182+0000 7f45bfe15640 1 -- 172.21.15.70:0/2860469886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45b81004b0 msgr2=0x7f45b81008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:32.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.182+0000 7f45bfe15640 1 --2- 172.21.15.70:0/2860469886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45b81004b0 0x7f45b81008d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009900 tx=0x7f45a802bc30 comp rx=0 tx=0).stop 2024-04-01T01:45:32.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.182+0000 7f45bfe15640 1 -- 172.21.15.70:0/2860469886 shutdown_connections 2024-04-01T01:45:32.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.182+0000 7f45bfe15640 1 --2- 172.21.15.70:0/2860469886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45b8102060 0x7f45b81088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.182+0000 7f45bfe15640 1 --2- 172.21.15.70:0/2860469886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45b81016a0 0x7f45b8101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.182+0000 7f45bfe15640 1 --2- 172.21.15.70:0/2860469886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45b81004b0 0x7f45b81008d0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.182+0000 7f45bfe15640 1 -- 172.21.15.70:0/2860469886 >> 172.21.15.70:0/2860469886 conn(0x7f45b80fba00 msgr2=0x7f45b80fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:32.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.182+0000 7f45bfe15640 1 -- 172.21.15.70:0/2860469886 shutdown_connections 2024-04-01T01:45:32.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.182+0000 7f45bfe15640 1 -- 172.21.15.70:0/2860469886 wait complete. 2024-04-01T01:45:32.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.183+0000 7f45bfe15640 1 Processor -- start 2024-04-01T01:45:32.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.183+0000 7f45bfe15640 1 -- start start 2024-04-01T01:45:32.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.183+0000 7f45bfe15640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45b81004b0 0x7f45b81955f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.183+0000 7f45bfe15640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45b81016a0 0x7f45b8195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.183+0000 7f45bfe15640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45b8102060 0x7f45b819cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.183+0000 7f45bfe15640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f45b810af30 con 0x7f45b81016a0 2024-04-01T01:45:32.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.183+0000 7f45bfe15640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f45b810adb0 con 0x7f45b81004b0 2024-04-01T01:45:32.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.183+0000 7f45bfe15640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f45b810b0b0 con 0x7f45b8102060 2024-04-01T01:45:32.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.183+0000 7f45bd3b0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45b81016a0 0x7f45b8195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.183+0000 7f45bdbb1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45b81004b0 0x7f45b81955f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.183+0000 7f45be3b2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45b8102060 0x7f45b819cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.184+0000 7f45be3b2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45b8102060 0x7f45b819cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58306/0 (socket says 172.21.15.70:58306) 2024-04-01T01:45:32.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.184+0000 7f45bd3b0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45b81016a0 0x7f45b8195b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36072/0 (socket says 172.21.15.70:36072) 2024-04-01T01:45:32.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.184+0000 7f45be3b2640 1 -- 172.21.15.70:0/2757727089 learned_addr learned my addr 172.21.15.70:0/2757727089 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:32.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.184+0000 7f45bdbb1640 1 -- 172.21.15.70:0/2757727089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45b8102060 msgr2=0x7f45b819cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:32.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.184+0000 7f45bdbb1640 1 --2- 172.21.15.70:0/2757727089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45b8102060 0x7f45b819cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.184+0000 7f45bdbb1640 1 -- 172.21.15.70:0/2757727089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45b81016a0 msgr2=0x7f45b8195b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:32.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.184+0000 7f45bdbb1640 1 --2- 172.21.15.70:0/2757727089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45b81016a0 0x7f45b8195b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.184+0000 7f45bdbb1640 1 -- 172.21.15.70:0/2757727089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b819d210 con 0x7f45b81004b0 2024-04-01T01:45:32.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.184+0000 7f45bd3b0640 1 --2- 172.21.15.70:0/2757727089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45b81016a0 0x7f45b8195b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:45:32.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.184+0000 7f45bdbb1640 1 --2- 172.21.15.70:0/2757727089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45b81004b0 0x7f45b81955f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f45a8036040 tx=0x7f45a80040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:32.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.185+0000 7f45aeffd640 1 -- 172.21.15.70:0/2757727089 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a8043020 con 0x7f45b81004b0 2024-04-01T01:45:32.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.185+0000 7f45bfe15640 1 -- 172.21.15.70:0/2757727089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45b819d4a0 con 0x7f45b81004b0 2024-04-01T01:45:32.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.185+0000 7f45bfe15640 1 -- 172.21.15.70:0/2757727089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45b819da10 con 0x7f45b81004b0 2024-04-01T01:45:32.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.185+0000 7f45aeffd640 1 -- 172.21.15.70:0/2757727089 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f45a80044c0 con 0x7f45b81004b0 2024-04-01T01:45:32.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.185+0000 7f45aeffd640 1 -- 172.21.15.70:0/2757727089 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a8004fb0 con 0x7f45b81004b0 2024-04-01T01:45:32.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.186+0000 7f45bfe15640 1 -- 172.21.15.70:0/2757727089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f459c0052c0 con 0x7f45b81004b0 2024-04-01T01:45:32.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.188+0000 7f45aeffd640 1 -- 172.21.15.70:0/2757727089 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f45a8046020 con 0x7f45b81004b0 2024-04-01T01:45:32.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.189+0000 7f45aeffd640 1 --2- 172.21.15.70:0/2757727089 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f45a40a25a0 0x7f45a40a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.190+0000 7f45aeffd640 1 -- 172.21.15.70:0/2757727089 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f45a80f0b70 con 0x7f45b81004b0 2024-04-01T01:45:32.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.190+0000 7f45bd3b0640 1 --2- 172.21.15.70:0/2757727089 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f45a40a25a0 0x7f45a40a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.191+0000 7f45bd3b0640 1 --2- 172.21.15.70:0/2757727089 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f45a40a25a0 0x7f45a40a4a60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f45b8196b00 tx=0x7f45b4015040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:32.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.194+0000 7f45aeffd640 1 -- 172.21.15.70:0/2757727089 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f45a804d020 con 0x7f45b81004b0 2024-04-01T01:45:32.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:31 smithi189 ceph-mon[35753]: pgmap v720: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 10 MiB/s wr, 763 op/s 2024-04-01T01:45:32.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:31 smithi081 ceph-mon[35142]: pgmap v720: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 10 MiB/s wr, 763 op/s 2024-04-01T01:45:32.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.624+0000 7f45bfe15640 1 -- 172.21.15.70:0/2757727089 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f459c0055b0 con 0x7f45b81004b0 2024-04-01T01:45:32.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.626+0000 7f45aeffd640 1 -- 172.21.15.70:0/2757727089 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7f45a80bda20 con 0x7f45b81004b0 2024-04-01T01:45:32.625 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:45:32.625 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:45:32.626 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:45:32.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.629+0000 7f45bfe15640 1 -- 172.21.15.70:0/2757727089 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f45a40a25a0 msgr2=0x7f45a40a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.629+0000 7f45bfe15640 1 --2- 172.21.15.70:0/2757727089 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f45a40a25a0 0x7f45a40a4a60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f45b8196b00 tx=0x7f45b4015040 comp rx=0 tx=0).stop 2024-04-01T01:45:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.629+0000 7f45bfe15640 1 -- 172.21.15.70:0/2757727089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45b81004b0 msgr2=0x7f45b81955f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.629+0000 7f45bfe15640 1 --2- 172.21.15.70:0/2757727089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45b81004b0 0x7f45b81955f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f45a8036040 tx=0x7f45a80040c0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.630+0000 7f45bfe15640 1 -- 172.21.15.70:0/2757727089 shutdown_connections 2024-04-01T01:45:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.630+0000 7f45bfe15640 1 --2- 172.21.15.70:0/2757727089 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f45a40a25a0 0x7f45a40a4a60 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.630+0000 7f45bfe15640 1 --2- 172.21.15.70:0/2757727089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45b8102060 0x7f45b819cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.630+0000 7f45bfe15640 1 --2- 172.21.15.70:0/2757727089 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45b81016a0 0x7f45b8195b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.630+0000 7f45bfe15640 1 --2- 172.21.15.70:0/2757727089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45b81004b0 0x7f45b81955f0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.630+0000 7f45bfe15640 1 -- 172.21.15.70:0/2757727089 >> 172.21.15.70:0/2757727089 conn(0x7f45b80fba00 msgr2=0x7f45b8104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.630+0000 7f45bfe15640 1 -- 172.21.15.70:0/2757727089 shutdown_connections 2024-04-01T01:45:32.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.630+0000 7f45bfe15640 1 -- 172.21.15.70:0/2757727089 wait complete. 2024-04-01T01:45:32.648 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:45:32.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.818+0000 7f0978806640 1 -- 172.21.15.70:0/1917872109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0974101410 msgr2=0x7f0974107ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:32.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.818+0000 7f0978806640 1 --2- 172.21.15.70:0/1917872109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0974101410 0x7f0974107ca0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f096c009370 tx=0x7f096c02cea0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.819+0000 7f0978806640 1 -- 172.21.15.70:0/1917872109 shutdown_connections 2024-04-01T01:45:32.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.819+0000 7f0978806640 1 --2- 172.21.15.70:0/1917872109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0974101410 0x7f0974107ca0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.819+0000 7f0978806640 1 --2- 172.21.15.70:0/1917872109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0974100a50 0x7f0974100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.819+0000 7f0978806640 1 --2- 172.21.15.70:0/1917872109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09740ff860 0x7f09740ffc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.819+0000 7f0978806640 1 -- 172.21.15.70:0/1917872109 >> 172.21.15.70:0/1917872109 conn(0x7f09740fadd0 msgr2=0x7f09740fd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:32.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.823+0000 7f0978806640 1 -- 172.21.15.70:0/1917872109 shutdown_connections 2024-04-01T01:45:32.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.824+0000 7f0978806640 1 -- 172.21.15.70:0/1917872109 wait complete. 2024-04-01T01:45:32.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.825+0000 7f0978806640 1 Processor -- start 2024-04-01T01:45:32.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.826+0000 7f0978806640 1 -- start start 2024-04-01T01:45:32.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.826+0000 7f0978806640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09740ff860 0x7f097419ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.826+0000 7f0978806640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0974100a50 0x7f097419e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.826+0000 7f0978806640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0974101410 0x7f09741a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.826+0000 7f0978806640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f097410b0b0 con 0x7f0974100a50 2024-04-01T01:45:32.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.826+0000 7f0978806640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f097410af30 con 0x7f09740ff860 2024-04-01T01:45:32.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.826+0000 7f0978806640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f097410b230 con 0x7f0974101410 2024-04-01T01:45:32.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.826+0000 7f097359e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0974101410 0x7f09741a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.826+0000 7f097359e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0974101410 0x7f09741a5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58318/0 (socket says 172.21.15.70:58318) 2024-04-01T01:45:32.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.826+0000 7f097359e640 1 -- 172.21.15.70:0/1338910914 learned_addr learned my addr 172.21.15.70:0/1338910914 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:32.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.827+0000 7f097359e640 1 -- 172.21.15.70:0/1338910914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09740ff860 msgr2=0x7f097419ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:32.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.827+0000 7f097359e640 1 --2- 172.21.15.70:0/1338910914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09740ff860 0x7f097419ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.827+0000 7f097359e640 1 -- 172.21.15.70:0/1338910914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0974100a50 msgr2=0x7f097419e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:32.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.827+0000 7f097359e640 1 --2- 172.21.15.70:0/1338910914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0974100a50 0x7f097419e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.827+0000 7f097359e640 1 -- 172.21.15.70:0/1338910914 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0974194cd0 con 0x7f0974101410 2024-04-01T01:45:32.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.827+0000 7f097359e640 1 --2- 172.21.15.70:0/1338910914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0974101410 0x7f09741a5380 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f096c009340 tx=0x7f096c005010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:32.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.831+0000 7f0970d99640 1 -- 172.21.15.70:0/1338910914 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096c043020 con 0x7f0974101410 2024-04-01T01:45:32.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.831+0000 7f0970d99640 1 -- 172.21.15.70:0/1338910914 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f096c0359d0 con 0x7f0974101410 2024-04-01T01:45:32.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.831+0000 7f0970d99640 1 -- 172.21.15.70:0/1338910914 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096c03fb30 con 0x7f0974101410 2024-04-01T01:45:32.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.832+0000 7f0978806640 1 -- 172.21.15.70:0/1338910914 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0974194f60 con 0x7f0974101410 2024-04-01T01:45:32.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.832+0000 7f0978806640 1 -- 172.21.15.70:0/1338910914 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09741954e0 con 0x7f0974101410 2024-04-01T01:45:32.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.836+0000 7f097bbee740 1 Processor -- start 2024-04-01T01:45:32.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.837+0000 7f0970d99640 1 -- 172.21.15.70:0/1338910914 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f096c046020 con 0x7f0974101410 2024-04-01T01:45:32.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.837+0000 7f097bbee740 1 -- start start 2024-04-01T01:45:32.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.837+0000 7f097bbee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 0x55e713a36ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.837+0000 7f097bbee740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 0x55e713a3bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.837+0000 7f097bbee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a3c200 0x55e713a3e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.837+0000 7f097bbee740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e71399b1b0 con 0x55e713a37090 2024-04-01T01:45:32.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.837+0000 7f097bbee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e713943c00 con 0x55e713a3c200 2024-04-01T01:45:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.837+0000 7f097bbee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e713997250 con 0x55e713a366a0 2024-04-01T01:45:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.837+0000 7f096bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 0x55e713a36ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.837+0000 7f096bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 0x55e713a36ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58330/0 (socket says 172.21.15.70:58330) 2024-04-01T01:45:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.837+0000 7f096bfff640 1 -- 172.21.15.70:0/2042850065 learned_addr learned my addr 172.21.15.70:0/2042850065 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.837+0000 7f096bfff640 1 --2- 172.21.15.70:0/2042850065 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a3c200 0x55e713a3e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.837+0000 7f0972d9d640 1 --2- 172.21.15.70:0/2042850065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 0x55e713a3bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.837+0000 7f0970d99640 1 --2- 172.21.15.70:0/1338910914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09600a25f0 0x7f09600a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.838+0000 7f096bfff640 1 -- 172.21.15.70:0/2042850065 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a3c200 msgr2=0x55e713a3e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.838+0000 7f096bfff640 1 --2- 172.21.15.70:0/2042850065 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a3c200 0x55e713a3e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.838+0000 7f0970d99640 1 -- 172.21.15.70:0/1338910914 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f096c0f11e0 con 0x7f0974101410 2024-04-01T01:45:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.838+0000 7f096bfff640 1 -- 172.21.15.70:0/2042850065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 msgr2=0x55e713a3bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.838+0000 7f096bfff640 1 --2- 172.21.15.70:0/2042850065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 0x55e713a3bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.838+0000 7f096bfff640 1 -- 172.21.15.70:0/2042850065 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7139633d0 con 0x55e713a366a0 2024-04-01T01:45:32.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.838+0000 7f097359e640 1 --2- 172.21.15.70:0/1338910914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09600a25f0 0x7f09600a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.838+0000 7f096bfff640 1 --2- 172.21.15.70:0/2042850065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 0x55e713a36ac0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0964013990 tx=0x7f0964001aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d22b7136fb9aebd server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:32.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.838+0000 7f09697fa640 1 -- 172.21.15.70:0/2042850065 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0964012040 con 0x55e713a366a0 2024-04-01T01:45:32.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.839+0000 7f09697fa640 1 -- 172.21.15.70:0/2042850065 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0964004cf0 con 0x55e713a366a0 2024-04-01T01:45:32.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.839+0000 7f09697fa640 1 -- 172.21.15.70:0/2042850065 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0964007850 con 0x55e713a366a0 2024-04-01T01:45:32.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.839+0000 7f097359e640 1 --2- 172.21.15.70:0/1338910914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09600a25f0 0x7f09600a4ab0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f096c0c5e70 tx=0x7f096c02cea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:32.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.840+0000 7f097bbee740 1 -- 172.21.15.70:0/2042850065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 msgr2=0x55e713a36ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:32.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.840+0000 7f097bbee740 1 --2- 172.21.15.70:0/2042850065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 0x55e713a36ac0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0964013990 tx=0x7f0964001aa0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.840+0000 7f097bbee740 1 -- 172.21.15.70:0/2042850065 shutdown_connections 2024-04-01T01:45:32.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.840+0000 7f097bbee740 1 --2- 172.21.15.70:0/2042850065 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a3c200 0x55e713a3e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.840+0000 7f097bbee740 1 --2- 172.21.15.70:0/2042850065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 0x55e713a3bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.840+0000 7f097bbee740 1 --2- 172.21.15.70:0/2042850065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 0x55e713a36ac0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.840+0000 7f097bbee740 1 -- 172.21.15.70:0/2042850065 >> 172.21.15.70:0/2042850065 conn(0x55e713a31c10 msgr2=0x55e713a32010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:32.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.840+0000 7f097bbee740 1 -- 172.21.15.70:0/2042850065 shutdown_connections 2024-04-01T01:45:32.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.840+0000 7f097bbee740 1 -- 172.21.15.70:0/2042850065 wait complete. 2024-04-01T01:45:32.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.840+0000 7f097bbee740 1 -- start start 2024-04-01T01:45:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.840+0000 7f097bbee740 1 Processor -- start 2024-04-01T01:45:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.841+0000 7f097bbee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 0x55e713a40c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.841+0000 7f097bbee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a37090 0x55e713a41150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.841+0000 7f097bbee740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a3c200 0x55e713a48190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.841+0000 7f097bbee740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e713a3f150 con 0x55e713a3c200 2024-04-01T01:45:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.841+0000 7f097bbee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e713939470 con 0x55e713a37090 2024-04-01T01:45:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.841+0000 7f097bbee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e713a3f2d0 con 0x55e713a366a0 2024-04-01T01:45:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.842+0000 7f097bbee740 10 client.0 _ll_register_callbacks cb 0x55e713956cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.842+0000 7f096bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 0x55e713a40c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.842+0000 7f096bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 0x55e713a40c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58344/0 (socket says 172.21.15.70:58344) 2024-04-01T01:45:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.842+0000 7f096bfff640 1 -- 172.21.15.70:0/2205117042 learned_addr learned my addr 172.21.15.70:0/2205117042 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.842+0000 7f0972d9d640 1 --2- 172.21.15.70:0/2205117042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a37090 0x55e713a41150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.842+0000 7f096bfff640 1 --2- 172.21.15.70:0/2205117042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a3c200 0x55e713a48190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.842+0000 7f0972d9d640 1 -- 172.21.15.70:0/2205117042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 msgr2=0x55e713a40c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.842+0000 7f0972d9d640 1 --2- 172.21.15.70:0/2205117042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 0x55e713a40c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.842+0000 7f0972d9d640 1 -- 172.21.15.70:0/2205117042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a3c200 msgr2=0x55e713a48190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.842+0000 7f0972d9d640 1 --2- 172.21.15.70:0/2205117042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a3c200 0x55e713a48190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.842+0000 7f0972d9d640 1 -- 172.21.15.70:0/2205117042 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7136af3f0 con 0x55e713a37090 2024-04-01T01:45:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.843+0000 7f0972d9d640 1 --2- 172.21.15.70:0/2205117042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a37090 0x55e713a41150 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x55e713a37490 tx=0x7f095c009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67465684589c4055 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.843+0000 7f095a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:45:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.843+0000 7f095a7fc640 1 -- 172.21.15.70:0/2205117042 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c017070 con 0x55e713a37090 2024-04-01T01:45:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.843+0000 7f097bbee740 1 -- 172.21.15.70:0/2205117042 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0964004cf0 con 0x55e713a37090 2024-04-01T01:45:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.843+0000 7f095a7fc640 1 -- 172.21.15.70:0/2205117042 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f095c013470 con 0x55e713a37090 2024-04-01T01:45:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.843+0000 7f095a7fc640 1 -- 172.21.15.70:0/2205117042 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f095c023050 con 0x55e713a37090 2024-04-01T01:45:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.843+0000 7f097bbee740 10 client.29003 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.843+0000 7f097bbee740 1 -- 172.21.15.70:0/2205117042 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e7137797d0 con 0x55e713a37090 2024-04-01T01:45:32.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.844+0000 7f095a7fc640 1 -- 172.21.15.70:0/2205117042 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0964004cf0 con 0x55e713a37090 2024-04-01T01:45:32.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.844+0000 7f095a7fc640 1 -- 172.21.15.70:0/2205117042 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f095c0128a0 con 0x55e713a37090 2024-04-01T01:45:32.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.844+0000 7f097bbee740 10 client.29003 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:32.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.844+0000 7f097bbee740 10 client.29003 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:45:32.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.844+0000 7f097bbee740 20 client.29003 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:32.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.844+0000 7f097bbee740 1 --2- 172.21.15.70:0/2205117042 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55e713a4cbf0 0x55e713a4f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.844+0000 7f097bbee740 4 client.29003 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:45:32.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.844+0000 7f097bbee740 1 -- 172.21.15.70:0/2205117042 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7139a86f0 con 0x55e713a4cbf0 2024-04-01T01:45:32.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.845+0000 7f096bfff640 1 --2- 172.21.15.70:0/2205117042 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55e713a4cbf0 0x55e713a4f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.846+0000 7f096bfff640 1 --2- 172.21.15.70:0/2205117042 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55e713a4cbf0 0x55e713a4f010 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a29605dc445ff98e server_cookie=93471989f4a16281 in_seq=0 out_seq=0 2024-04-01T01:45:32.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.846+0000 7f095a7fc640 10 client.29003 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:45:32.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.847+0000 7f095a7fc640 1 -- 172.21.15.70:0/2205117042 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e7139a86f0 con 0x55e713a4cbf0 2024-04-01T01:45:32.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.847+0000 7f095a7fc640 10 client.29003 handle_command_reply: tid=0 2024-04-01T01:45:32.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.847+0000 7f097bbee740 1 client.29003 shutdown 2024-04-01T01:45:32.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.847+0000 7f097bbee740 1 -- 172.21.15.70:0/2205117042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a37090 msgr2=0x55e713a41150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:32.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.847+0000 7f097bbee740 1 --2- 172.21.15.70:0/2205117042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a37090 0x55e713a41150 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x55e713a37490 tx=0x7f095c009f90 comp rx=0 tx=0).stop 2024-04-01T01:45:32.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.847+0000 7f097bbee740 1 -- 172.21.15.70:0/2205117042 shutdown_connections 2024-04-01T01:45:32.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.847+0000 7f097bbee740 1 --2- 172.21.15.70:0/2205117042 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55e713a4cbf0 0x55e713a4f010 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.848+0000 7f097bbee740 1 --2- 172.21.15.70:0/2205117042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a3c200 0x55e713a48190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.848+0000 7f097bbee740 1 --2- 172.21.15.70:0/2205117042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a37090 0x55e713a41150 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.848+0000 7f097bbee740 1 --2- 172.21.15.70:0/2205117042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 0x55e713a40c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.848+0000 7f097bbee740 1 -- 172.21.15.70:0/2205117042 >> 172.21.15.70:0/2205117042 conn(0x55e713a31c10 msgr2=0x55e713a30f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:32.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.848+0000 7f095a7fc640 0 client.29003 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:45:32.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.848+0000 7f097bbee740 1 -- 172.21.15.70:0/2205117042 shutdown_connections 2024-04-01T01:45:32.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.848+0000 7f097bbee740 1 -- 172.21.15.70:0/2205117042 wait complete. 2024-04-01T01:45:32.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.849+0000 7f097bbee740 20 client.29003 trim_cache size 0 max 16384 2024-04-01T01:45:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.877+0000 7f097bbee740 1 Processor -- start 2024-04-01T01:45:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.877+0000 7f097bbee740 1 -- start start 2024-04-01T01:45:32.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.878+0000 7f097bbee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 0x55e713a35580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.878+0000 7f097bbee740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 0x55e713a48150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.878+0000 7f096bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 0x55e713a35580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.878+0000 7f096bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 0x55e713a35580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58358/0 (socket says 172.21.15.70:58358) 2024-04-01T01:45:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.878+0000 7f096bfff640 1 -- 172.21.15.70:0/3669902161 learned_addr learned my addr 172.21.15.70:0/3669902161 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.878+0000 7f0972d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 0x55e713a48150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.879+0000 7f097bbee740 1 --2- 172.21.15.70:0/3669902161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a3c200 0x55e713a3f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.879+0000 7f097bbee740 1 -- 172.21.15.70:0/3669902161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f095400fca0 con 0x55e713a37090 2024-04-01T01:45:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.879+0000 7f097bbee740 1 -- 172.21.15.70:0/3669902161 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f09540117e0 con 0x55e713a3c200 2024-04-01T01:45:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.879+0000 7f097bbee740 1 -- 172.21.15.70:0/3669902161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0954010370 con 0x55e713a366a0 2024-04-01T01:45:32.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.879+0000 7f096bfff640 1 --2- 172.21.15.70:0/3669902161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a3c200 0x55e713a3f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.879+0000 7f0972d9d640 1 -- 172.21.15.70:0/3669902161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 msgr2=0x55e713a35580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.879+0000 7f0972d9d640 1 --2- 172.21.15.70:0/3669902161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 0x55e713a35580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.879+0000 7f0972d9d640 1 -- 172.21.15.70:0/3669902161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a3c200 msgr2=0x55e713a3f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.879+0000 7f0972d9d640 1 --2- 172.21.15.70:0/3669902161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a3c200 0x55e713a3f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.879+0000 7f0972d9d640 1 -- 172.21.15.70:0/3669902161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e713a22e60 con 0x55e713a37090 2024-04-01T01:45:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.880+0000 7f0972d9d640 1 --2- 172.21.15.70:0/3669902161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 0x55e713a48150 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f095c012eb0 tx=0x7f095c012f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4121fbdf381511 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.881+0000 7f0968ff9640 1 -- 172.21.15.70:0/3669902161 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c02b020 con 0x55e713a37090 2024-04-01T01:45:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.881+0000 7f0968ff9640 1 -- 172.21.15.70:0/3669902161 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f095c0044d0 con 0x55e713a37090 2024-04-01T01:45:32.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.881+0000 7f0968ff9640 1 -- 172.21.15.70:0/3669902161 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c017050 con 0x55e713a37090 2024-04-01T01:45:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.882+0000 7f097bbee740 1 -- 172.21.15.70:0/3669902161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 msgr2=0x55e713a48150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.882+0000 7f097bbee740 1 --2- 172.21.15.70:0/3669902161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 0x55e713a48150 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f095c012eb0 tx=0x7f095c012f30 comp rx=0 tx=0).stop 2024-04-01T01:45:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.883+0000 7f097bbee740 1 -- 172.21.15.70:0/3669902161 shutdown_connections 2024-04-01T01:45:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.883+0000 7f097bbee740 1 --2- 172.21.15.70:0/3669902161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a3c200 0x55e713a3f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.883+0000 7f097bbee740 1 --2- 172.21.15.70:0/3669902161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 0x55e713a48150 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.883+0000 7f097bbee740 1 --2- 172.21.15.70:0/3669902161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a366a0 0x55e713a35580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.883+0000 7f097bbee740 1 -- 172.21.15.70:0/3669902161 >> 172.21.15.70:0/3669902161 conn(0x55e713a31c10 msgr2=0x55e713a34e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.883+0000 7f097bbee740 1 -- 172.21.15.70:0/3669902161 shutdown_connections 2024-04-01T01:45:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.883+0000 7f097bbee740 1 -- 172.21.15.70:0/3669902161 wait complete. 2024-04-01T01:45:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.883+0000 7f097bbee740 1 -- start start 2024-04-01T01:45:32.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.884+0000 7f097bbee740 1 Processor -- start 2024-04-01T01:45:32.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.884+0000 7f097bbee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a366a0 0x55e713a45860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f097bbee740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 0x55e713a45da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f096bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a366a0 0x55e713a45860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f096bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a366a0 0x55e713a45860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41084/0 (socket says 172.21.15.70:41084) 2024-04-01T01:45:32.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f096bfff640 1 -- 172.21.15.70:0/1593027270 learned_addr learned my addr 172.21.15.70:0/1593027270 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:32.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f097bbee740 1 --2- 172.21.15.70:0/1593027270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a3c200 0x55e713a46db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f097bbee740 1 -- 172.21.15.70:0/1593027270 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0954011110 con 0x55e713a37090 2024-04-01T01:45:32.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f097bbee740 1 -- 172.21.15.70:0/1593027270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0954010a40 con 0x55e713a366a0 2024-04-01T01:45:32.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f097bbee740 1 -- 172.21.15.70:0/1593027270 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f095400dc80 con 0x55e713a3c200 2024-04-01T01:45:32.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f097bbee740 10 client.0 _ll_register_callbacks cb 0x7f095c009bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:32.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f096bfff640 1 --2- 172.21.15.70:0/1593027270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a3c200 0x55e713a46db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f0972d9d640 1 --2- 172.21.15.70:0/1593027270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 0x55e713a45da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f096bfff640 1 -- 172.21.15.70:0/1593027270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a3c200 msgr2=0x55e713a46db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:32.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f096bfff640 1 --2- 172.21.15.70:0/1593027270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a3c200 0x55e713a46db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f096bfff640 1 -- 172.21.15.70:0/1593027270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 msgr2=0x55e713a45da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:32.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f096bfff640 1 --2- 172.21.15.70:0/1593027270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 0x55e713a45da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f096bfff640 1 -- 172.21.15.70:0/1593027270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e713a473a0 con 0x55e713a366a0 2024-04-01T01:45:32.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.885+0000 7f0972d9d640 1 --2- 172.21.15.70:0/1593027270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 0x55e713a45da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:45:32.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.886+0000 7f096bfff640 1 --2- 172.21.15.70:0/1593027270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a366a0 0x55e713a45860 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f09640022f0 tx=0x7f096401c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0edd36617f244d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:32.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.886+0000 7f095a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:45:32.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.886+0000 7f095a7fc640 1 -- 172.21.15.70:0/1593027270 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0964023040 con 0x55e713a366a0 2024-04-01T01:45:32.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.886+0000 7f095a7fc640 1 -- 172.21.15.70:0/1593027270 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f0964013570 con 0x55e713a366a0 2024-04-01T01:45:32.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.886+0000 7f097bbee740 1 -- 172.21.15.70:0/1593027270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f095c0044d0 con 0x55e713a366a0 2024-04-01T01:45:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.886+0000 7f095a7fc640 1 -- 172.21.15.70:0/1593027270 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f096402d400 con 0x55e713a366a0 2024-04-01T01:45:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.886+0000 7f097bbee740 10 client.29009 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.886+0000 7f097bbee740 1 -- 172.21.15.70:0/1593027270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e713ab2f10 con 0x55e713a366a0 2024-04-01T01:45:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.887+0000 7f095a7fc640 1 -- 172.21.15.70:0/1593027270 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f096402d5a0 con 0x55e713a366a0 2024-04-01T01:45:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.887+0000 7f095a7fc640 1 -- 172.21.15.70:0/1593027270 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f0964007530 con 0x55e713a366a0 2024-04-01T01:45:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.887+0000 7f097bbee740 10 client.29009 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:32.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.887+0000 7f097bbee740 10 client.29009 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:45:32.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.887+0000 7f097bbee740 20 client.29009 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:32.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.887+0000 7f097bbee740 1 --2- 172.21.15.70:0/1593027270 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55e713ab3780 0x55e713ab5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:32.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.887+0000 7f097bbee740 4 client.29009 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:45:32.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.887+0000 7f097bbee740 1 -- 172.21.15.70:0/1593027270 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f095c0044d0 con 0x55e713ab3780 2024-04-01T01:45:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.888+0000 7f0972d9d640 1 --2- 172.21.15.70:0/1593027270 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55e713ab3780 0x55e713ab5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.889+0000 7f0972d9d640 1 --2- 172.21.15.70:0/1593027270 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55e713ab3780 0x55e713ab5ba0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=623a9e44e3f07a4d server_cookie=1fdee4a3456e241b in_seq=0 out_seq=0 2024-04-01T01:45:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.889+0000 7f095a7fc640 10 client.29009 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:45:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.890+0000 7f095a7fc640 1 -- 172.21.15.70:0/1593027270 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f095c0044d0 con 0x55e713ab3780 2024-04-01T01:45:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.890+0000 7f095a7fc640 10 client.29009 handle_command_reply: tid=0 2024-04-01T01:45:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.890+0000 7f097bbee740 1 client.29009 shutdown 2024-04-01T01:45:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.890+0000 7f097bbee740 1 -- 172.21.15.70:0/1593027270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a366a0 msgr2=0x55e713a45860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.890+0000 7f097bbee740 1 --2- 172.21.15.70:0/1593027270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a366a0 0x55e713a45860 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f09640022f0 tx=0x7f096401c040 comp rx=0 tx=0).stop 2024-04-01T01:45:32.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.890+0000 7f097bbee740 1 -- 172.21.15.70:0/1593027270 shutdown_connections 2024-04-01T01:45:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.890+0000 7f097bbee740 1 --2- 172.21.15.70:0/1593027270 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55e713ab3780 0x55e713ab5ba0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.890+0000 7f097bbee740 1 --2- 172.21.15.70:0/1593027270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e713a3c200 0x55e713a46db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.890+0000 7f097bbee740 1 --2- 172.21.15.70:0/1593027270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e713a37090 0x55e713a45da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.890+0000 7f097bbee740 1 --2- 172.21.15.70:0/1593027270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e713a366a0 0x55e713a45860 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.890+0000 7f097bbee740 1 -- 172.21.15.70:0/1593027270 >> 172.21.15.70:0/1593027270 conn(0x55e713a31c10 msgr2=0x55e713a35700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.890+0000 7f095a7fc640 0 client.29009 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:45:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.890+0000 7f097bbee740 1 -- 172.21.15.70:0/1593027270 shutdown_connections 2024-04-01T01:45:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.890+0000 7f097bbee740 1 -- 172.21.15.70:0/1593027270 wait complete. 2024-04-01T01:45:32.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.891+0000 7f097bbee740 20 client.29009 trim_cache size 0 max 16384 2024-04-01T01:45:32.894 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:45:32.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.892+0000 7f0978806640 1 -- 172.21.15.70:0/1338910914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09600a25f0 msgr2=0x7f09600a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:32.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.892+0000 7f0978806640 1 --2- 172.21.15.70:0/1338910914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09600a25f0 0x7f09600a4ab0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f096c0c5e70 tx=0x7f096c02cea0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.892+0000 7f0978806640 1 -- 172.21.15.70:0/1338910914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0974101410 msgr2=0x7f09741a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:32.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.893+0000 7f0978806640 1 --2- 172.21.15.70:0/1338910914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0974101410 0x7f09741a5380 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f096c009340 tx=0x7f096c005010 comp rx=0 tx=0).stop 2024-04-01T01:45:32.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.893+0000 7f0978806640 1 -- 172.21.15.70:0/1338910914 shutdown_connections 2024-04-01T01:45:32.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.893+0000 7f0978806640 1 --2- 172.21.15.70:0/1338910914 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f09600a25f0 0x7f09600a4ab0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.893+0000 7f0978806640 1 --2- 172.21.15.70:0/1338910914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0974101410 0x7f09741a5380 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.893+0000 7f0978806640 1 --2- 172.21.15.70:0/1338910914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0974100a50 0x7f097419e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.893+0000 7f0978806640 1 --2- 172.21.15.70:0/1338910914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09740ff860 0x7f097419ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:32.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.893+0000 7f0978806640 1 -- 172.21.15.70:0/1338910914 >> 172.21.15.70:0/1338910914 conn(0x7f09740fadd0 msgr2=0x7f09741927a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:32.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.893+0000 7f0978806640 1 -- 172.21.15.70:0/1338910914 shutdown_connections 2024-04-01T01:45:32.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:32.893+0000 7f0978806640 1 -- 172.21.15.70:0/1338910914 wait complete. 2024-04-01T01:45:32.913 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:45:33.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.099+0000 7f72d663b640 1 -- 172.21.15.70:0/1495158088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c80a6710 msgr2=0x7f72c80aafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.099+0000 7f72d663b640 1 --2- 172.21.15.70:0/1495158088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c80a6710 0x7f72c80aafd0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f72c0009900 tx=0x7f72c002bc40 comp rx=0 tx=0).stop 2024-04-01T01:45:33.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.099+0000 7f72d663b640 1 -- 172.21.15.70:0/1495158088 shutdown_connections 2024-04-01T01:45:33.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.099+0000 7f72d663b640 1 --2- 172.21.15.70:0/1495158088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c80a6710 0x7f72c80aafd0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.099+0000 7f72d663b640 1 --2- 172.21.15.70:0/1495158088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72c80a5d70 0x7f72c80a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.099+0000 7f72d663b640 1 --2- 172.21.15.70:0/1495158088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72c80a4c40 0x7f72c80a5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.099+0000 7f72d663b640 1 -- 172.21.15.70:0/1495158088 >> 172.21.15.70:0/1495158088 conn(0x7f72c80a00d0 msgr2=0x7f72c80a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.099+0000 7f72d663b640 1 -- 172.21.15.70:0/1495158088 shutdown_connections 2024-04-01T01:45:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.099+0000 7f72d663b640 1 -- 172.21.15.70:0/1495158088 wait complete. 2024-04-01T01:45:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.100+0000 7f72d663b640 1 Processor -- start 2024-04-01T01:45:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.100+0000 7f72d663b640 1 -- start start 2024-04-01T01:45:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.100+0000 7f72d663b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c80a4c40 0x7f72c8143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.100+0000 7f72d663b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72c80a5d70 0x7f72c8143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.101+0000 7f72d663b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72c80a6710 0x7f72c814abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.101+0000 7f72d663b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f72c80b8f30 con 0x7f72c80a4c40 2024-04-01T01:45:33.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.101+0000 7f72d663b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f72c80b8db0 con 0x7f72c80a5d70 2024-04-01T01:45:33.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.101+0000 7f72d663b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f72c80b90b0 con 0x7f72c80a6710 2024-04-01T01:45:33.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.101+0000 7f72d4e38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72c80a5d70 0x7f72c8143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.101+0000 7f72d5639640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c80a4c40 0x7f72c8143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.101+0000 7f72d4e38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72c80a5d70 0x7f72c8143b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41104/0 (socket says 172.21.15.70:41104) 2024-04-01T01:45:33.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.101+0000 7f72d4e38640 1 -- 172.21.15.70:0/81010033 learned_addr learned my addr 172.21.15.70:0/81010033 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.102+0000 7f72d4e38640 1 -- 172.21.15.70:0/81010033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72c80a6710 msgr2=0x7f72c814abb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:45:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.102+0000 7f72d4e38640 1 --2- 172.21.15.70:0/81010033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72c80a6710 0x7f72c814abb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.102+0000 7f72d4e38640 1 -- 172.21.15.70:0/81010033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c80a4c40 msgr2=0x7f72c8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.102+0000 7f72d4e38640 1 --2- 172.21.15.70:0/81010033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c80a4c40 0x7f72c8143600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.102+0000 7f72d4e38640 1 -- 172.21.15.70:0/81010033 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c80ac990 con 0x7f72c80a5d70 2024-04-01T01:45:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.102+0000 7f72d4e38640 1 --2- 172.21.15.70:0/81010033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72c80a5d70 0x7f72c8143b40 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f72d0067290 tx=0x7f72d006f770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.102+0000 7f72c67fc640 1 -- 172.21.15.70:0/81010033 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0078070 con 0x7f72c80a5d70 2024-04-01T01:45:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.102+0000 7f72c67fc640 1 -- 172.21.15.70:0/81010033 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72d0069580 con 0x7f72c80a5d70 2024-04-01T01:45:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.102+0000 7f72c67fc640 1 -- 172.21.15.70:0/81010033 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d00748f0 con 0x7f72c80a5d70 2024-04-01T01:45:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.102+0000 7f72d663b640 1 -- 172.21.15.70:0/81010033 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72c80acc80 con 0x7f72c80a5d70 2024-04-01T01:45:33.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.102+0000 7f72d663b640 1 -- 172.21.15.70:0/81010033 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c80ad150 con 0x7f72c80a5d70 2024-04-01T01:45:33.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.108+0000 7f72da483740 1 Processor -- start 2024-04-01T01:45:33.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.108+0000 7f72da483740 1 -- start start 2024-04-01T01:45:33.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.108+0000 7f72da483740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd918646a0 0x55fd91864ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.109+0000 7f72da483740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd91865090 0x55fd91869b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.109+0000 7f72da483740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd9186a200 0x55fd9186c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.109+0000 7f72da483740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fd917c91b0 con 0x55fd9186a200 2024-04-01T01:45:33.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.109+0000 7f72da483740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55fd91771c00 con 0x55fd91865090 2024-04-01T01:45:33.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.109+0000 7f72da483740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fd917c5250 con 0x55fd918646a0 2024-04-01T01:45:33.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.109+0000 7f72d5639640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd91865090 0x55fd91869b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.109+0000 7f72d5639640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd91865090 0x55fd91869b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41120/0 (socket says 172.21.15.70:41120) 2024-04-01T01:45:33.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.109+0000 7f72d5639640 1 -- 172.21.15.70:0/3747285961 learned_addr learned my addr 172.21.15.70:0/3747285961 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.109+0000 7f72d5e3a640 1 --2- 172.21.15.70:0/3747285961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd918646a0 0x55fd91864ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.109+0000 7f72d5639640 1 -- 172.21.15.70:0/3747285961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd918646a0 msgr2=0x55fd91864ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.109+0000 7f72d5639640 1 --2- 172.21.15.70:0/3747285961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd918646a0 0x55fd91864ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.109+0000 7f72d5639640 1 -- 172.21.15.70:0/3747285961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd9186a200 msgr2=0x55fd9186c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.109+0000 7f72d5639640 1 --2- 172.21.15.70:0/3747285961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd9186a200 0x55fd9186c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.109+0000 7f72d5639640 1 -- 172.21.15.70:0/3747285961 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd917913d0 con 0x55fd91865090 2024-04-01T01:45:33.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.110+0000 7f72d5e3a640 1 --2- 172.21.15.70:0/3747285961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd918646a0 0x55fd91864ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:45:33.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.110+0000 7f72d5639640 1 --2- 172.21.15.70:0/3747285961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd91865090 0x55fd91869b90 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f72cc00bd10 tx=0x7f72cc00f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=913f2576cc6f19ee server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.110+0000 7f72c67fc640 1 -- 172.21.15.70:0/81010033 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f72d007b070 con 0x7f72c80a5d70 2024-04-01T01:45:33.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.110+0000 7f72bb7fe640 1 -- 172.21.15.70:0/3747285961 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc015070 con 0x55fd91865090 2024-04-01T01:45:33.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.110+0000 7f72bb7fe640 1 -- 172.21.15.70:0/3747285961 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72cc01a910 con 0x55fd91865090 2024-04-01T01:45:33.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.110+0000 7f72bb7fe640 1 -- 172.21.15.70:0/3747285961 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc011990 con 0x55fd91865090 2024-04-01T01:45:33.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.110+0000 7f72c67fc640 1 --2- 172.21.15.70:0/81010033 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72bc0a26b0 0x7f72bc0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.111+0000 7f72c67fc640 1 -- 172.21.15.70:0/81010033 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f72d01320e0 con 0x7f72c80a5d70 2024-04-01T01:45:33.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.111+0000 7f72da483740 1 -- 172.21.15.70:0/3747285961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd91865090 msgr2=0x55fd91869b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.111+0000 7f72da483740 1 --2- 172.21.15.70:0/3747285961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd91865090 0x55fd91869b90 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f72cc00bd10 tx=0x7f72cc00f730 comp rx=0 tx=0).stop 2024-04-01T01:45:33.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.111+0000 7f72d5e3a640 1 --2- 172.21.15.70:0/81010033 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72bc0a26b0 0x7f72bc0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.111+0000 7f72da483740 1 -- 172.21.15.70:0/3747285961 shutdown_connections 2024-04-01T01:45:33.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.111+0000 7f72da483740 1 --2- 172.21.15.70:0/3747285961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd9186a200 0x55fd9186c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.111+0000 7f72da483740 1 --2- 172.21.15.70:0/3747285961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd91865090 0x55fd91869b90 secure :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f72cc00bd10 tx=0x7f72cc00f730 comp rx=0 tx=0).stop 2024-04-01T01:45:33.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.111+0000 7f72da483740 1 --2- 172.21.15.70:0/3747285961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd918646a0 0x55fd91864ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.111+0000 7f72da483740 1 -- 172.21.15.70:0/3747285961 >> 172.21.15.70:0/3747285961 conn(0x55fd9185fc10 msgr2=0x55fd91860010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.112+0000 7f72da483740 1 -- 172.21.15.70:0/3747285961 shutdown_connections 2024-04-01T01:45:33.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.116+0000 7f72da483740 1 -- 172.21.15.70:0/3747285961 wait complete. 2024-04-01T01:45:33.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.116+0000 7f72da483740 1 -- start start 2024-04-01T01:45:33.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.116+0000 7f72da483740 1 Processor -- start 2024-04-01T01:45:33.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.117+0000 7f72d5e3a640 1 --2- 172.21.15.70:0/81010033 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72bc0a26b0 0x7f72bc0a4b70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f72c0006010 tx=0x7f72c0005d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.118+0000 7f72da483740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd918646a0 0x55fd9186edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.118+0000 7f72da483740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9186a200 0x55fd9186f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.118+0000 7f72da483740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd91874320 0x55fd91876710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.118+0000 7f72da483740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fd9186d1b0 con 0x55fd91874320 2024-04-01T01:45:33.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.118+0000 7f72da483740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55fd91767470 con 0x55fd918646a0 2024-04-01T01:45:33.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.119+0000 7f72d4e38640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9186a200 0x55fd9186f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.119+0000 7f72d4e38640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9186a200 0x55fd9186f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58400/0 (socket says 172.21.15.70:58400) 2024-04-01T01:45:33.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.119+0000 7f72d4e38640 1 -- 172.21.15.70:0/1875799360 learned_addr learned my addr 172.21.15.70:0/1875799360 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.119+0000 7f72d5639640 1 --2- 172.21.15.70:0/1875799360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd918646a0 0x55fd9186edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.119+0000 7f72da483740 1 -- 172.21.15.70:0/1875799360 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fd9186d330 con 0x55fd9186a200 2024-04-01T01:45:33.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.119+0000 7f72d5e3a640 1 --2- 172.21.15.70:0/1875799360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd91874320 0x55fd91876710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.119+0000 7f72da483740 10 client.0 _ll_register_callbacks cb 0x55fd91784cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:33.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.119+0000 7f72d4e38640 1 -- 172.21.15.70:0/1875799360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd918646a0 msgr2=0x55fd9186edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.119+0000 7f72d4e38640 1 --2- 172.21.15.70:0/1875799360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd918646a0 0x55fd9186edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.119+0000 7f72d4e38640 1 -- 172.21.15.70:0/1875799360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd91874320 msgr2=0x55fd91876710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.119+0000 7f72d4e38640 1 --2- 172.21.15.70:0/1875799360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd91874320 0x55fd91876710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.119+0000 7f72d4e38640 1 -- 172.21.15.70:0/1875799360 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd914dd3f0 con 0x55fd9186a200 2024-04-01T01:45:33.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.120+0000 7f72da483740 1 -- 172.21.15.70:0/1875799360 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72cc01a910 con 0x55fd9186a200 2024-04-01T01:45:33.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.120+0000 7f72d4e38640 1 --2- 172.21.15.70:0/1875799360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9186a200 0x55fd9186f330 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f72d0106c50 tx=0x7f72d00fe2b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69a724c09220d5c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.120+0000 7f72b8ff9640 10 client.28813 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:45:33.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.120+0000 7f72b8ff9640 1 -- 172.21.15.70:0/1875799360 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0082040 con 0x55fd9186a200 2024-04-01T01:45:33.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.120+0000 7f72b8ff9640 1 -- 172.21.15.70:0/1875799360 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f72d00fade0 con 0x55fd9186a200 2024-04-01T01:45:33.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.120+0000 7f72b8ff9640 1 -- 172.21.15.70:0/1875799360 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f72d00fe770 con 0x55fd9186a200 2024-04-01T01:45:33.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.120+0000 7f72da483740 10 client.28813 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:33.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.120+0000 7f72da483740 1 -- 172.21.15.70:0/1875799360 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55fd915a77d0 con 0x55fd9186a200 2024-04-01T01:45:33.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.121+0000 7f72b8ff9640 1 -- 172.21.15.70:0/1875799360 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f72d01340f0 con 0x55fd9186a200 2024-04-01T01:45:33.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.121+0000 7f72da483740 10 client.28813 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:33.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.121+0000 7f72da483740 10 client.28813 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:45:33.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.121+0000 7f72da483740 20 client.28813 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:33.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.121+0000 7f72da483740 1 --2- 172.21.15.70:0/1875799360 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55fd9187a9d0 0x55fd9187ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.121+0000 7f72da483740 4 client.28813 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:45:33.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.121+0000 7f72da483740 1 -- 172.21.15.70:0/1875799360 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd917d66f0 con 0x55fd9187a9d0 2024-04-01T01:45:33.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.121+0000 7f72d5639640 1 --2- 172.21.15.70:0/1875799360 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55fd9187a9d0 0x55fd9187ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.122+0000 7f72d5639640 1 --2- 172.21.15.70:0/1875799360 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55fd9187a9d0 0x55fd9187ce40 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e5ec301852e5ecea server_cookie=11c3f0fc66f5325d in_seq=0 out_seq=0 2024-04-01T01:45:33.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.122+0000 7f72b8ff9640 10 client.28813 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:45:33.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.123+0000 7f72b8ff9640 1 -- 172.21.15.70:0/1875799360 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fd917d66f0 con 0x55fd9187a9d0 2024-04-01T01:45:33.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.123+0000 7f72b8ff9640 10 client.28813 handle_command_reply: tid=0 2024-04-01T01:45:33.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.123+0000 7f72da483740 1 client.28813 shutdown 2024-04-01T01:45:33.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.123+0000 7f72da483740 1 -- 172.21.15.70:0/1875799360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9186a200 msgr2=0x55fd9186f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.123+0000 7f72da483740 1 --2- 172.21.15.70:0/1875799360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9186a200 0x55fd9186f330 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f72d0106c50 tx=0x7f72d00fe2b0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.123+0000 7f72da483740 1 -- 172.21.15.70:0/1875799360 shutdown_connections 2024-04-01T01:45:33.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.123+0000 7f72da483740 1 --2- 172.21.15.70:0/1875799360 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55fd9187a9d0 0x55fd9187ce40 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.123+0000 7f72da483740 1 --2- 172.21.15.70:0/1875799360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd91874320 0x55fd91876710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.123+0000 7f72da483740 1 --2- 172.21.15.70:0/1875799360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9186a200 0x55fd9186f330 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.123+0000 7f72da483740 1 --2- 172.21.15.70:0/1875799360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd918646a0 0x55fd9186edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.123+0000 7f72da483740 1 -- 172.21.15.70:0/1875799360 >> 172.21.15.70:0/1875799360 conn(0x55fd9185fc10 msgr2=0x55fd9185ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.123+0000 7f72b8ff9640 0 client.28813 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:45:33.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.125+0000 7f72da483740 1 -- 172.21.15.70:0/1875799360 shutdown_connections 2024-04-01T01:45:33.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.125+0000 7f72da483740 1 -- 172.21.15.70:0/1875799360 wait complete. 2024-04-01T01:45:33.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.125+0000 7f72da483740 20 client.28813 trim_cache size 0 max 16384 2024-04-01T01:45:33.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.149+0000 7f72da483740 1 Processor -- start 2024-04-01T01:45:33.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.149+0000 7f72da483740 1 -- start start 2024-04-01T01:45:33.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.149+0000 7f72da483740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd918646a0 0x55fd9186c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.149+0000 7f72da483740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd9186a200 0x55fd9186edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.149+0000 7f72d5639640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd918646a0 0x55fd9186c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.149+0000 7f72d5639640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd918646a0 0x55fd9186c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58402/0 (socket says 172.21.15.70:58402) 2024-04-01T01:45:33.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.149+0000 7f72d4e38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd9186a200 0x55fd9186edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.149+0000 7f72d4e38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd9186a200 0x55fd9186edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41128/0 (socket says 172.21.15.70:41128) 2024-04-01T01:45:33.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.149+0000 7f72da483740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd91874320 0x55fd9186f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.149+0000 7f72d5639640 1 -- 172.21.15.70:0/1090865345 learned_addr learned my addr 172.21.15.70:0/1090865345 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.149+0000 7f72da483740 1 -- 172.21.15.70:0/1090865345 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f72c815e830 con 0x55fd91874320 2024-04-01T01:45:33.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.149+0000 7f72da483740 1 -- 172.21.15.70:0/1090865345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f72c8160370 con 0x55fd9186a200 2024-04-01T01:45:33.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.149+0000 7f72da483740 1 -- 172.21.15.70:0/1090865345 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f72c815ef00 con 0x55fd918646a0 2024-04-01T01:45:33.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.150+0000 7f72d5e3a640 1 --2- 172.21.15.70:0/1090865345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd91874320 0x55fd9186f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.150+0000 7f72d4e38640 1 -- 172.21.15.70:0/1090865345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd918646a0 msgr2=0x55fd9186c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.150+0000 7f72d4e38640 1 --2- 172.21.15.70:0/1090865345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd918646a0 0x55fd9186c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.150+0000 7f72d4e38640 1 -- 172.21.15.70:0/1090865345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd91874320 msgr2=0x55fd9186f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.150+0000 7f72d4e38640 1 --2- 172.21.15.70:0/1090865345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd91874320 0x55fd9186f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.150+0000 7f72d4e38640 1 -- 172.21.15.70:0/1090865345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd91865920 con 0x55fd9186a200 2024-04-01T01:45:33.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.150+0000 7f72d5639640 1 --2- 172.21.15.70:0/1090865345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd918646a0 0x55fd9186c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:45:33.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.150+0000 7f72d4e38640 1 --2- 172.21.15.70:0/1090865345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd9186a200 0x55fd9186edc0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f72d0050040 tx=0x7f72d0069000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e796fec736093a36 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.150+0000 7f72baffd640 1 -- 172.21.15.70:0/1090865345 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0082040 con 0x55fd9186a200 2024-04-01T01:45:33.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.151+0000 7f72baffd640 1 -- 172.21.15.70:0/1090865345 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72cc01a910 con 0x55fd9186a200 2024-04-01T01:45:33.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.151+0000 7f72baffd640 1 -- 172.21.15.70:0/1090865345 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d00fa4c0 con 0x55fd9186a200 2024-04-01T01:45:33.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.152+0000 7f72da483740 1 -- 172.21.15.70:0/1090865345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd9186a200 msgr2=0x55fd9186edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.152+0000 7f72da483740 1 --2- 172.21.15.70:0/1090865345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd9186a200 0x55fd9186edc0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f72d0050040 tx=0x7f72d0069000 comp rx=0 tx=0).stop 2024-04-01T01:45:33.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.152+0000 7f72da483740 1 -- 172.21.15.70:0/1090865345 shutdown_connections 2024-04-01T01:45:33.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.152+0000 7f72da483740 1 --2- 172.21.15.70:0/1090865345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd91874320 0x55fd9186f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.152+0000 7f72da483740 1 --2- 172.21.15.70:0/1090865345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd9186a200 0x55fd9186edc0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.152+0000 7f72da483740 1 --2- 172.21.15.70:0/1090865345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd918646a0 0x55fd9186c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.152+0000 7f72da483740 1 -- 172.21.15.70:0/1090865345 >> 172.21.15.70:0/1090865345 conn(0x55fd9185fc10 msgr2=0x55fd91866770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.152+0000 7f72da483740 1 -- 172.21.15.70:0/1090865345 shutdown_connections 2024-04-01T01:45:33.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.152+0000 7f72da483740 1 -- 172.21.15.70:0/1090865345 wait complete. 2024-04-01T01:45:33.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.153+0000 7f72da483740 1 -- start start 2024-04-01T01:45:33.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.153+0000 7f72da483740 1 Processor -- start 2024-04-01T01:45:33.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.153+0000 7f72da483740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd918646a0 0x55fd9188b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.153+0000 7f72da483740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd9186a200 0x55fd9188bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.153+0000 7f72da483740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd91874320 0x55fd9188cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.153+0000 7f72da483740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f72c815fca0 con 0x55fd918646a0 2024-04-01T01:45:33.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.153+0000 7f72da483740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f72c815f5d0 con 0x55fd9186a200 2024-04-01T01:45:33.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.153+0000 7f72da483740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f72c80b4e00 con 0x55fd91874320 2024-04-01T01:45:33.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.153+0000 7f72da483740 10 client.0 _ll_register_callbacks cb 0x7f72d00fa3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:33.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.153+0000 7f72d5639640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd918646a0 0x55fd9188b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.153+0000 7f72d4e38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd9186a200 0x55fd9188bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.154+0000 7f72d4e38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd9186a200 0x55fd9188bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41140/0 (socket says 172.21.15.70:41140) 2024-04-01T01:45:33.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.154+0000 7f72d4e38640 1 -- 172.21.15.70:0/4131904269 learned_addr learned my addr 172.21.15.70:0/4131904269 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.154+0000 7f72d5e3a640 1 --2- 172.21.15.70:0/4131904269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd91874320 0x55fd9188cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.154+0000 7f72d5e3a640 1 -- 172.21.15.70:0/4131904269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd9186a200 msgr2=0x55fd9188bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.154+0000 7f72d5e3a640 1 --2- 172.21.15.70:0/4131904269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd9186a200 0x55fd9188bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.154+0000 7f72d5e3a640 1 -- 172.21.15.70:0/4131904269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd918646a0 msgr2=0x55fd9188b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.154+0000 7f72d5e3a640 1 --2- 172.21.15.70:0/4131904269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd918646a0 0x55fd9188b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.154+0000 7f72d5e3a640 1 -- 172.21.15.70:0/4131904269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd9188d270 con 0x55fd91874320 2024-04-01T01:45:33.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.155+0000 7f72d5e3a640 1 --2- 172.21.15.70:0/4131904269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd91874320 0x55fd9188cc80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f72c0009850 tx=0x7f72c0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baadfc52fe6c35ab server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.155+0000 7f72b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:45:33.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.155+0000 7f72b8ff9640 1 -- 172.21.15.70:0/4131904269 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c0050030 con 0x55fd91874320 2024-04-01T01:45:33.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.155+0000 7f72b8ff9640 1 -- 172.21.15.70:0/4131904269 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f72c003a040 con 0x55fd91874320 2024-04-01T01:45:33.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.155+0000 7f72da483740 1 -- 172.21.15.70:0/4131904269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72cc01a910 con 0x55fd91874320 2024-04-01T01:45:33.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.156+0000 7f72b8ff9640 1 -- 172.21.15.70:0/4131904269 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f72c0046040 con 0x55fd91874320 2024-04-01T01:45:33.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.156+0000 7f72da483740 10 client.28819 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:33.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.156+0000 7f72da483740 1 -- 172.21.15.70:0/4131904269 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fd918e0870 con 0x55fd91874320 2024-04-01T01:45:33.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.156+0000 7f72b8ff9640 1 -- 172.21.15.70:0/4131904269 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72cc01a910 con 0x55fd91874320 2024-04-01T01:45:33.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.156+0000 7f72b8ff9640 1 -- 172.21.15.70:0/4131904269 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f72c0003680 con 0x55fd91874320 2024-04-01T01:45:33.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.157+0000 7f72da483740 10 client.28819 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:33.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.157+0000 7f72da483740 10 client.28819 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:45:33.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.157+0000 7f72da483740 20 client.28819 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:33.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.157+0000 7f72da483740 1 --2- 172.21.15.70:0/4131904269 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55fd918e1210 0x55fd918e3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.157+0000 7f72da483740 4 client.28819 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:45:33.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.157+0000 7f72da483740 1 -- 172.21.15.70:0/4131904269 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fd917d0a40 con 0x55fd918e1210 2024-04-01T01:45:33.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.157+0000 7f72d5639640 1 --2- 172.21.15.70:0/4131904269 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55fd918e1210 0x55fd918e3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.157+0000 7f72b8ff9640 10 client.28819 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:45:33.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.157+0000 7f72d5639640 1 --2- 172.21.15.70:0/4131904269 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55fd918e1210 0x55fd918e3600 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8e87b09e229c456a server_cookie=a748b54d8fcdc9af in_seq=0 out_seq=0 2024-04-01T01:45:33.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.158+0000 7f72b8ff9640 1 -- 172.21.15.70:0/4131904269 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fd917d0a40 con 0x55fd918e1210 2024-04-01T01:45:33.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.158+0000 7f72b8ff9640 10 client.28819 handle_command_reply: tid=0 2024-04-01T01:45:33.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.158+0000 7f72da483740 1 client.28819 shutdown 2024-04-01T01:45:33.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.158+0000 7f72da483740 1 -- 172.21.15.70:0/4131904269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd91874320 msgr2=0x55fd9188cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.158+0000 7f72da483740 1 --2- 172.21.15.70:0/4131904269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd91874320 0x55fd9188cc80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f72c0009850 tx=0x7f72c0009260 comp rx=0 tx=0).stop 2024-04-01T01:45:33.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.158+0000 7f72da483740 1 -- 172.21.15.70:0/4131904269 shutdown_connections 2024-04-01T01:45:33.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.158+0000 7f72da483740 1 --2- 172.21.15.70:0/4131904269 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55fd918e1210 0x55fd918e3600 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.158+0000 7f72da483740 1 --2- 172.21.15.70:0/4131904269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd91874320 0x55fd9188cc80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.158+0000 7f72da483740 1 --2- 172.21.15.70:0/4131904269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fd9186a200 0x55fd9188bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.158+0000 7f72da483740 1 --2- 172.21.15.70:0/4131904269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fd918646a0 0x55fd9188b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.158+0000 7f72da483740 1 -- 172.21.15.70:0/4131904269 >> 172.21.15.70:0/4131904269 conn(0x55fd9185fc10 msgr2=0x55fd91866770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.158+0000 7f72b8ff9640 0 client.28819 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:45:33.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.159+0000 7f72da483740 1 -- 172.21.15.70:0/4131904269 shutdown_connections 2024-04-01T01:45:33.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.159+0000 7f72da483740 1 -- 172.21.15.70:0/4131904269 wait complete. 2024-04-01T01:45:33.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.159+0000 7f72da483740 20 client.28819 trim_cache size 0 max 16384 2024-04-01T01:45:33.160 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:45:33.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.160+0000 7f72d663b640 1 -- 172.21.15.70:0/81010033 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72bc0a26b0 msgr2=0x7f72bc0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.160+0000 7f72d663b640 1 --2- 172.21.15.70:0/81010033 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72bc0a26b0 0x7f72bc0a4b70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f72c0006010 tx=0x7f72c0005d50 comp rx=0 tx=0).stop 2024-04-01T01:45:33.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.160+0000 7f72d663b640 1 -- 172.21.15.70:0/81010033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72c80a5d70 msgr2=0x7f72c8143b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.160+0000 7f72d663b640 1 --2- 172.21.15.70:0/81010033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72c80a5d70 0x7f72c8143b40 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f72d0067290 tx=0x7f72d006f770 comp rx=0 tx=0).stop 2024-04-01T01:45:33.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.161+0000 7f72d663b640 1 -- 172.21.15.70:0/81010033 shutdown_connections 2024-04-01T01:45:33.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.161+0000 7f72d663b640 1 --2- 172.21.15.70:0/81010033 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72bc0a26b0 0x7f72bc0a4b70 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.161+0000 7f72d663b640 1 --2- 172.21.15.70:0/81010033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72c80a6710 0x7f72c814abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.161+0000 7f72d663b640 1 --2- 172.21.15.70:0/81010033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72c80a5d70 0x7f72c8143b40 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.161+0000 7f72d663b640 1 --2- 172.21.15.70:0/81010033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72c80a4c40 0x7f72c8143600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.161+0000 7f72d663b640 1 -- 172.21.15.70:0/81010033 >> 172.21.15.70:0/81010033 conn(0x7f72c80a00d0 msgr2=0x7f72c80a0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.162+0000 7f72d663b640 1 -- 172.21.15.70:0/81010033 shutdown_connections 2024-04-01T01:45:33.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.162+0000 7f72d663b640 1 -- 172.21.15.70:0/81010033 wait complete. 2024-04-01T01:45:33.183 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:32 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2757727089' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:45:33.185 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:45:33.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:32 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2757727089' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:45:33.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:32 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2757727089' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:45:33.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.370+0000 7f9bd6036640 1 -- 172.21.15.70:0/4190567677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc8095760 msgr2=0x7f9bc809a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.370+0000 7f9bd6036640 1 --2- 172.21.15.70:0/4190567677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc8095760 0x7f9bc809a000 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0067420 tx=0x7f9bd0097840 comp rx=0 tx=0).stop 2024-04-01T01:45:33.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.370+0000 7f9bd6036640 1 -- 172.21.15.70:0/4190567677 shutdown_connections 2024-04-01T01:45:33.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.370+0000 7f9bd6036640 1 --2- 172.21.15.70:0/4190567677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc8095760 0x7f9bc809a000 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.370+0000 7f9bd6036640 1 --2- 172.21.15.70:0/4190567677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bc8094da0 0x7f9bc8095220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.370+0000 7f9bd6036640 1 --2- 172.21.15.70:0/4190567677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9bc8094040 0x7f9bc8094460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.370+0000 7f9bd6036640 1 -- 172.21.15.70:0/4190567677 >> 172.21.15.70:0/4190567677 conn(0x7f9bc808f950 msgr2=0x7f9bc8091db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.370+0000 7f9bd6036640 1 -- 172.21.15.70:0/4190567677 shutdown_connections 2024-04-01T01:45:33.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.370+0000 7f9bd6036640 1 -- 172.21.15.70:0/4190567677 wait complete. 2024-04-01T01:45:33.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.371+0000 7f9bd6036640 1 Processor -- start 2024-04-01T01:45:33.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.371+0000 7f9bd6036640 1 -- start start 2024-04-01T01:45:33.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.371+0000 7f9bd6036640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9bc8094040 0x7f9bc80a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.371+0000 7f9bd6036640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bc8094da0 0x7f9bc80a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.371+0000 7f9bd6036640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc8095760 0x7f9bc8142510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.371+0000 7f9bd6036640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc80a7e40 con 0x7f9bc8094040 2024-04-01T01:45:33.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.371+0000 7f9bd6036640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc80a7cc0 con 0x7f9bc8094da0 2024-04-01T01:45:33.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.371+0000 7f9bd6036640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc80a7fc0 con 0x7f9bc8095760 2024-04-01T01:45:33.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.372+0000 7f9bd5034640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9bc8094040 0x7f9bc80a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.372+0000 7f9bd5034640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9bc8094040 0x7f9bc80a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36172/0 (socket says 172.21.15.70:36172) 2024-04-01T01:45:33.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.372+0000 7f9bd5835640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc8095760 0x7f9bc8142510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.372+0000 7f9bd5034640 1 -- 172.21.15.70:0/3634429393 learned_addr learned my addr 172.21.15.70:0/3634429393 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.372+0000 7f9bd4833640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bc8094da0 0x7f9bc80a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.372+0000 7f9bd5034640 1 -- 172.21.15.70:0/3634429393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc8095760 msgr2=0x7f9bc8142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.372+0000 7f9bd5034640 1 --2- 172.21.15.70:0/3634429393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc8095760 0x7f9bc8142510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.372+0000 7f9bd5034640 1 -- 172.21.15.70:0/3634429393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bc8094da0 msgr2=0x7f9bc80a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.372+0000 7f9bd5034640 1 --2- 172.21.15.70:0/3634429393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bc8094da0 0x7f9bc80a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.372+0000 7f9bd5034640 1 -- 172.21.15.70:0/3634429393 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bc8142a50 con 0x7f9bc8094040 2024-04-01T01:45:33.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.372+0000 7f9bd5034640 1 --2- 172.21.15.70:0/3634429393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9bc8094040 0x7f9bc80a03e0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00d950 tx=0x7f9bcc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.373+0000 7f9bc67fc640 1 -- 172.21.15.70:0/3634429393 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc017070 con 0x7f9bc8094040 2024-04-01T01:45:33.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.373+0000 7f9bd6036640 1 -- 172.21.15.70:0/3634429393 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bc8142c20 con 0x7f9bc8094040 2024-04-01T01:45:33.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.373+0000 7f9bd6036640 1 -- 172.21.15.70:0/3634429393 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bc8143160 con 0x7f9bc8094040 2024-04-01T01:45:33.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.373+0000 7f9bc67fc640 1 -- 172.21.15.70:0/3634429393 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bcc009c30 con 0x7f9bc8094040 2024-04-01T01:45:33.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.373+0000 7f9bc67fc640 1 -- 172.21.15.70:0/3634429393 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc020060 con 0x7f9bc8094040 2024-04-01T01:45:33.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.376+0000 7f9bd9e71740 1 Processor -- start 2024-04-01T01:45:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.376+0000 7f9bd9e71740 1 -- start start 2024-04-01T01:45:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.377+0000 7f9bd9e71740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 0x5598756c4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.377+0000 7f9bd9e71740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756c5090 0x5598756c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.377+0000 7f9bd9e71740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756ca200 0x5598756cc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.377+0000 7f9bd9e71740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5598756291b0 con 0x5598756c5090 2024-04-01T01:45:33.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.377+0000 7f9bd9e71740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5598755d1c00 con 0x5598756c46a0 2024-04-01T01:45:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.377+0000 7f9bd9e71740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559875625250 con 0x5598756ca200 2024-04-01T01:45:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.377+0000 7f9bd5835640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 0x5598756c4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.377+0000 7f9bd5835640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 0x5598756c4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41170/0 (socket says 172.21.15.70:41170) 2024-04-01T01:45:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.377+0000 7f9bd5835640 1 -- 172.21.15.70:0/2021093518 learned_addr learned my addr 172.21.15.70:0/2021093518 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.377+0000 7f9bc67fc640 1 -- 172.21.15.70:0/3634429393 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f9bcc010460 con 0x7f9bc8094040 2024-04-01T01:45:33.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.377+0000 7f9bd5034640 1 --2- 172.21.15.70:0/2021093518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756c5090 0x5598756c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.378+0000 7f9bd5835640 1 -- 172.21.15.70:0/2021093518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756ca200 msgr2=0x5598756cc690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:45:33.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.378+0000 7f9bd5835640 1 --2- 172.21.15.70:0/2021093518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756ca200 0x5598756cc690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.378+0000 7f9bd5835640 1 -- 172.21.15.70:0/2021093518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756c5090 msgr2=0x5598756c9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.378+0000 7f9bd5835640 1 --2- 172.21.15.70:0/2021093518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756c5090 0x5598756c9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.378+0000 7f9bd5835640 1 -- 172.21.15.70:0/2021093518 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598755f13d0 con 0x5598756c46a0 2024-04-01T01:45:33.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.378+0000 7f9bc67fc640 1 --2- 172.21.15.70:0/3634429393 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9bbc0a2520 0x7f9bbc0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.378+0000 7f9bd5835640 1 --2- 172.21.15.70:0/2021093518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 0x5598756c4ac0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0075bd0 tx=0x7f9bd00675f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed21c4dcd383c442 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.379+0000 7f9bd5034640 1 --2- 172.21.15.70:0/3634429393 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9bbc0a2520 0x7f9bbc0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.379+0000 7f9bc67fc640 1 -- 172.21.15.70:0/3634429393 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f9bcc0ce6f0 con 0x7f9bc8094040 2024-04-01T01:45:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.379+0000 7f9bbb7fe640 1 -- 172.21.15.70:0/2021093518 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd00ae070 con 0x5598756c46a0 2024-04-01T01:45:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.379+0000 7f9bbb7fe640 1 -- 172.21.15.70:0/2021093518 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bd00690d0 con 0x5598756c46a0 2024-04-01T01:45:33.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.379+0000 7f9bbb7fe640 1 -- 172.21.15.70:0/2021093518 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd00a3040 con 0x5598756c46a0 2024-04-01T01:45:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.380+0000 7f9bd9e71740 1 -- 172.21.15.70:0/2021093518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 msgr2=0x5598756c4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.380+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/2021093518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 0x5598756c4ac0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0075bd0 tx=0x7f9bd00675f0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.380+0000 7f9bd5034640 1 --2- 172.21.15.70:0/3634429393 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9bbc0a2520 0x7f9bbc0a49e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc09ab30 tx=0x7f9bcc005880 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.380+0000 7f9bd9e71740 1 -- 172.21.15.70:0/2021093518 shutdown_connections 2024-04-01T01:45:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.380+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/2021093518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756ca200 0x5598756cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.380+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/2021093518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756c5090 0x5598756c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.380+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/2021093518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 0x5598756c4ac0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.380+0000 7f9bd9e71740 1 -- 172.21.15.70:0/2021093518 >> 172.21.15.70:0/2021093518 conn(0x5598756bfc10 msgr2=0x5598756c0010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.380+0000 7f9bd9e71740 1 -- 172.21.15.70:0/2021093518 shutdown_connections 2024-04-01T01:45:33.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.380+0000 7f9bd9e71740 1 -- 172.21.15.70:0/2021093518 wait complete. 2024-04-01T01:45:33.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.381+0000 7f9bd9e71740 1 -- start start 2024-04-01T01:45:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.381+0000 7f9bd9e71740 1 Processor -- start 2024-04-01T01:45:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.381+0000 7f9bd9e71740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c46a0 0x5598756cebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.381+0000 7f9bd9e71740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756c5090 0x5598756cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.381+0000 7f9bd9e71740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756ca200 0x5598756d6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.381+0000 7f9bd9e71740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5598756cd150 con 0x5598756c5090 2024-04-01T01:45:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.381+0000 7f9bd9e71740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5598755c7470 con 0x5598756ca200 2024-04-01T01:45:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.381+0000 7f9bd9e71740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5598756cd2d0 con 0x5598756c46a0 2024-04-01T01:45:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.381+0000 7f9bd9e71740 10 client.0 _ll_register_callbacks cb 0x5598755e4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.382+0000 7f9bd5835640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c46a0 0x5598756cebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.382+0000 7f9bd4833640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756c5090 0x5598756cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.382+0000 7f9bd4833640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756c5090 0x5598756cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36198/0 (socket says 172.21.15.70:36198) 2024-04-01T01:45:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.382+0000 7f9bd4833640 1 -- 172.21.15.70:0/641302977 learned_addr learned my addr 172.21.15.70:0/641302977 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.382+0000 7f9bd5835640 1 --2- 172.21.15.70:0/641302977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756ca200 0x5598756d6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.382+0000 7f9bd4833640 1 -- 172.21.15.70:0/641302977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c46a0 msgr2=0x5598756cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.382+0000 7f9bd4833640 1 --2- 172.21.15.70:0/641302977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c46a0 0x5598756cebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.382+0000 7f9bd4833640 1 -- 172.21.15.70:0/641302977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756ca200 msgr2=0x5598756d6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.382+0000 7f9bd4833640 1 --2- 172.21.15.70:0/641302977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756ca200 0x5598756d6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.382+0000 7f9bd4833640 1 -- 172.21.15.70:0/641302977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55987533d3f0 con 0x5598756c5090 2024-04-01T01:45:33.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.383+0000 7f9bd4833640 1 --2- 172.21.15.70:0/641302977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756c5090 0x5598756cf130 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000be00 tx=0x7f9bc00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f4dd49e8074816 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.383+0000 7f9bb8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:45:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.386+0000 7f9bb8ff9640 1 -- 172.21.15.70:0/641302977 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0004220 con 0x5598756c5090 2024-04-01T01:45:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.386+0000 7f9bd9e71740 1 -- 172.21.15.70:0/641302977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bd00690d0 con 0x5598756c5090 2024-04-01T01:45:33.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.386+0000 7f9bb8ff9640 1 -- 172.21.15.70:0/641302977 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f9bc00043c0 con 0x5598756c5090 2024-04-01T01:45:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.387+0000 7f9bb8ff9640 1 -- 172.21.15.70:0/641302977 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9bc0024050 con 0x5598756c5090 2024-04-01T01:45:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.387+0000 7f9bd9e71740 10 client.18966 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:33.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.387+0000 7f9bd9e71740 1 -- 172.21.15.70:0/641302977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5598754077d0 con 0x5598756c5090 2024-04-01T01:45:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.387+0000 7f9bb8ff9640 1 -- 172.21.15.70:0/641302977 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bd00690d0 con 0x5598756c5090 2024-04-01T01:45:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.387+0000 7f9bb8ff9640 1 -- 172.21.15.70:0/641302977 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f9bc0003800 con 0x5598756c5090 2024-04-01T01:45:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.387+0000 7f9bd9e71740 10 client.18966 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.387+0000 7f9bd9e71740 10 client.18966 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:45:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.387+0000 7f9bd9e71740 20 client.18966 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.387+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/641302977 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5598756da820 0x5598756dcc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.387+0000 7f9bd9e71740 4 client.18966 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:45:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.387+0000 7f9bd9e71740 1 -- 172.21.15.70:0/641302977 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598756366f0 con 0x5598756da820 2024-04-01T01:45:33.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.388+0000 7f9bd5835640 1 --2- 172.21.15.70:0/641302977 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5598756da820 0x5598756dcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.389+0000 7f9bd5835640 1 --2- 172.21.15.70:0/641302977 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5598756da820 0x5598756dcc80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b9db2c55610d5d0c server_cookie=6347d85a767c87be in_seq=0 out_seq=0 2024-04-01T01:45:33.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.389+0000 7f9bb8ff9640 10 client.18966 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:45:33.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.390+0000 7f9bb8ff9640 1 -- 172.21.15.70:0/641302977 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5598756366f0 con 0x5598756da820 2024-04-01T01:45:33.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.391+0000 7f9bb8ff9640 10 client.18966 handle_command_reply: tid=0 2024-04-01T01:45:33.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.391+0000 7f9bd9e71740 1 client.18966 shutdown 2024-04-01T01:45:33.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.391+0000 7f9bd9e71740 1 -- 172.21.15.70:0/641302977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756c5090 msgr2=0x5598756cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.391+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/641302977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756c5090 0x5598756cf130 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000be00 tx=0x7f9bc00107a0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.391+0000 7f9bd9e71740 1 -- 172.21.15.70:0/641302977 shutdown_connections 2024-04-01T01:45:33.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.391+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/641302977 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5598756da820 0x5598756dcc80 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.391+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/641302977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756ca200 0x5598756d6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.391+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/641302977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756c5090 0x5598756cf130 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.391+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/641302977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c46a0 0x5598756cebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.391+0000 7f9bd9e71740 1 -- 172.21.15.70:0/641302977 >> 172.21.15.70:0/641302977 conn(0x5598756bfc10 msgr2=0x5598756bef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.393+0000 7f9bb8ff9640 0 client.18966 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:45:33.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.393+0000 7f9bd9e71740 1 -- 172.21.15.70:0/641302977 shutdown_connections 2024-04-01T01:45:33.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.393+0000 7f9bd9e71740 1 -- 172.21.15.70:0/641302977 wait complete. 2024-04-01T01:45:33.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.393+0000 7f9bd9e71740 20 client.18966 trim_cache size 0 max 16384 2024-04-01T01:45:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.419+0000 7f9bd9e71740 1 Processor -- start 2024-04-01T01:45:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.419+0000 7f9bd9e71740 1 -- start start 2024-04-01T01:45:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.420+0000 7f9bd9e71740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 0x5598756ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.420+0000 7f9bd9e71740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c5090 0x5598756d6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.420+0000 7f9bd9e71740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756ca200 0x5598756be700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.420+0000 7f9bd9e71740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb400fda0 con 0x5598756ca200 2024-04-01T01:45:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.420+0000 7f9bd9e71740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb40118e0 con 0x5598756c46a0 2024-04-01T01:45:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.420+0000 7f9bd9e71740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb4010470 con 0x5598756c5090 2024-04-01T01:45:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.420+0000 7f9bd5835640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 0x5598756ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.420+0000 7f9bd5835640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 0x5598756ce5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41174/0 (socket says 172.21.15.70:41174) 2024-04-01T01:45:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.420+0000 7f9bd5835640 1 -- 172.21.15.70:0/156374445 learned_addr learned my addr 172.21.15.70:0/156374445 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.420+0000 7f9bd4833640 1 --2- 172.21.15.70:0/156374445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c5090 0x5598756d6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.420+0000 7f9bd5835640 1 --2- 172.21.15.70:0/156374445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756ca200 0x5598756be700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.420+0000 7f9bd4833640 1 -- 172.21.15.70:0/156374445 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 msgr2=0x5598756ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.420+0000 7f9bd4833640 1 --2- 172.21.15.70:0/156374445 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 0x5598756ce5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.420+0000 7f9bd4833640 1 -- 172.21.15.70:0/156374445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756ca200 msgr2=0x5598756be700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.420+0000 7f9bd4833640 1 --2- 172.21.15.70:0/156374445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756ca200 0x5598756be700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.420+0000 7f9bd4833640 1 -- 172.21.15.70:0/156374445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598756b0e60 con 0x5598756c5090 2024-04-01T01:45:33.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.421+0000 7f9bd4833640 1 --2- 172.21.15.70:0/156374445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c5090 0x5598756d6130 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0001eb0 tx=0x7f9bc0003e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61402df47412d087 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.421+0000 7f9bbaffd640 1 -- 172.21.15.70:0/156374445 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0010e90 con 0x5598756c5090 2024-04-01T01:45:33.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.421+0000 7f9bbaffd640 1 -- 172.21.15.70:0/156374445 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bc002c070 con 0x5598756c5090 2024-04-01T01:45:33.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.421+0000 7f9bbaffd640 1 -- 172.21.15.70:0/156374445 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0005600 con 0x5598756c5090 2024-04-01T01:45:33.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.422+0000 7f9bd9e71740 1 -- 172.21.15.70:0/156374445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c5090 msgr2=0x5598756d6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.422+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/156374445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c5090 0x5598756d6130 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0001eb0 tx=0x7f9bc0003e40 comp rx=0 tx=0).stop 2024-04-01T01:45:33.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.422+0000 7f9bd9e71740 1 -- 172.21.15.70:0/156374445 shutdown_connections 2024-04-01T01:45:33.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.422+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/156374445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756ca200 0x5598756be700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.422+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/156374445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c5090 0x5598756d6130 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.422+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/156374445 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 0x5598756ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.422+0000 7f9bd9e71740 1 -- 172.21.15.70:0/156374445 >> 172.21.15.70:0/156374445 conn(0x5598756bfc10 msgr2=0x5598756cdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.423+0000 7f9bd9e71740 1 -- 172.21.15.70:0/156374445 shutdown_connections 2024-04-01T01:45:33.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.423+0000 7f9bd9e71740 1 -- 172.21.15.70:0/156374445 wait complete. 2024-04-01T01:45:33.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.423+0000 7f9bd9e71740 1 -- start start 2024-04-01T01:45:33.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.424+0000 7f9bd9e71740 1 Processor -- start 2024-04-01T01:45:33.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.424+0000 7f9bd9e71740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 0x5598756d30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.424+0000 7f9bd9e71740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c5090 0x5598756d3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.424+0000 7f9bd9e71740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756ca200 0x55987573ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.424+0000 7f9bd9e71740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb4011210 con 0x5598756ca200 2024-04-01T01:45:33.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.424+0000 7f9bd4833640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c5090 0x5598756d3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.424+0000 7f9bd9e71740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb4010b40 con 0x5598756c46a0 2024-04-01T01:45:33.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.424+0000 7f9bd9e71740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9bb400dd80 con 0x5598756c5090 2024-04-01T01:45:33.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.424+0000 7f9bd4833640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c5090 0x5598756d3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58462/0 (socket says 172.21.15.70:58462) 2024-04-01T01:45:33.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.424+0000 7f9bd4833640 1 -- 172.21.15.70:0/256125860 learned_addr learned my addr 172.21.15.70:0/256125860 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.424+0000 7f9bd9e71740 10 client.0 _ll_register_callbacks cb 0x7f9bc000ba10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:33.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.425+0000 7f9bd4833640 1 -- 172.21.15.70:0/256125860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 msgr2=0x5598756d30e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:45:33.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.425+0000 7f9bd4833640 1 --2- 172.21.15.70:0/256125860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 0x5598756d30e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.425+0000 7f9bd4833640 1 -- 172.21.15.70:0/256125860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756ca200 msgr2=0x55987573ebb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:45:33.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.425+0000 7f9bd4833640 1 --2- 172.21.15.70:0/256125860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756ca200 0x55987573ebb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.425+0000 7f9bd4833640 1 -- 172.21.15.70:0/256125860 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55987573f170 con 0x5598756c5090 2024-04-01T01:45:33.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.425+0000 7f9bd4833640 1 --2- 172.21.15.70:0/256125860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c5090 0x5598756d3620 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000bf90 tx=0x7f9bc0005630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e355f558b368f068 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.425+0000 7f9bb8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:45:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.425+0000 7f9bb8ff9640 1 -- 172.21.15.70:0/256125860 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0010e90 con 0x5598756c5090 2024-04-01T01:45:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.425+0000 7f9bb8ff9640 1 -- 172.21.15.70:0/256125860 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f9bc0003800 con 0x5598756c5090 2024-04-01T01:45:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.426+0000 7f9bd9e71740 1 -- 172.21.15.70:0/256125860 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bc002c070 con 0x5598756c5090 2024-04-01T01:45:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.426+0000 7f9bb8ff9640 1 -- 172.21.15.70:0/256125860 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9bc0040050 con 0x5598756c5090 2024-04-01T01:45:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.426+0000 7f9bd9e71740 10 client.28834 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.427+0000 7f9bd9e71740 1 -- 172.21.15.70:0/256125860 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559875740450 con 0x5598756c5090 2024-04-01T01:45:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.427+0000 7f9bb8ff9640 1 -- 172.21.15.70:0/256125860 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9bc0011b00 con 0x5598756c5090 2024-04-01T01:45:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.427+0000 7f9bb8ff9640 1 -- 172.21.15.70:0/256125860 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f9bc00314f0 con 0x5598756c5090 2024-04-01T01:45:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.427+0000 7f9bd9e71740 10 client.28834 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.427+0000 7f9bd9e71740 10 client.28834 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:45:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.427+0000 7f9bd9e71740 20 client.28834 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.427+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/256125860 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x559875740d40 0x559875743160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.428+0000 7f9bd9e71740 4 client.28834 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:45:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.428+0000 7f9bd9e71740 1 -- 172.21.15.70:0/256125860 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f9bc002c070 con 0x559875740d40 2024-04-01T01:45:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.428+0000 7f9bd5835640 1 --2- 172.21.15.70:0/256125860 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x559875740d40 0x559875743160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.429+0000 7f9bd5835640 1 --2- 172.21.15.70:0/256125860 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x559875740d40 0x559875743160 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=34ef2f59b25f779d server_cookie=98d814fd4136321a in_seq=0 out_seq=0 2024-04-01T01:45:33.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.429+0000 7f9bb8ff9640 10 client.28834 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:45:33.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.429+0000 7f9bb8ff9640 1 -- 172.21.15.70:0/256125860 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f9bc002c070 con 0x559875740d40 2024-04-01T01:45:33.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.429+0000 7f9bb8ff9640 10 client.28834 handle_command_reply: tid=0 2024-04-01T01:45:33.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.429+0000 7f9bd9e71740 1 client.28834 shutdown 2024-04-01T01:45:33.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.430+0000 7f9bd9e71740 1 -- 172.21.15.70:0/256125860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c5090 msgr2=0x5598756d3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.430+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/256125860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c5090 0x5598756d3620 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000bf90 tx=0x7f9bc0005630 comp rx=0 tx=0).stop 2024-04-01T01:45:33.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.431+0000 7f9bd9e71740 1 -- 172.21.15.70:0/256125860 shutdown_connections 2024-04-01T01:45:33.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.431+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/256125860 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x559875740d40 0x559875743160 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.431+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/256125860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5598756ca200 0x55987573ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.431+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/256125860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5598756c5090 0x5598756d3620 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.431+0000 7f9bd9e71740 1 --2- 172.21.15.70:0/256125860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5598756c46a0 0x5598756d30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.431+0000 7f9bd9e71740 1 -- 172.21.15.70:0/256125860 >> 172.21.15.70:0/256125860 conn(0x5598756bfc10 msgr2=0x5598756c2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.431+0000 7f9bb8ff9640 0 client.28834 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:45:33.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.431+0000 7f9bd9e71740 1 -- 172.21.15.70:0/256125860 shutdown_connections 2024-04-01T01:45:33.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.431+0000 7f9bd9e71740 1 -- 172.21.15.70:0/256125860 wait complete. 2024-04-01T01:45:33.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.432+0000 7f9bd9e71740 20 client.28834 trim_cache size 0 max 16384 2024-04-01T01:45:33.434 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:45:33.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.436+0000 7f9bd6036640 1 -- 172.21.15.70:0/3634429393 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9bbc0a2520 msgr2=0x7f9bbc0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.436+0000 7f9bd6036640 1 --2- 172.21.15.70:0/3634429393 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9bbc0a2520 0x7f9bbc0a49e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc09ab30 tx=0x7f9bcc005880 comp rx=0 tx=0).stop 2024-04-01T01:45:33.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.437+0000 7f9bd6036640 1 -- 172.21.15.70:0/3634429393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9bc8094040 msgr2=0x7f9bc80a03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.437+0000 7f9bd6036640 1 --2- 172.21.15.70:0/3634429393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9bc8094040 0x7f9bc80a03e0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00d950 tx=0x7f9bcc00de10 comp rx=0 tx=0).stop 2024-04-01T01:45:33.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.437+0000 7f9bd6036640 1 -- 172.21.15.70:0/3634429393 shutdown_connections 2024-04-01T01:45:33.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.437+0000 7f9bd6036640 1 --2- 172.21.15.70:0/3634429393 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9bbc0a2520 0x7f9bbc0a49e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.437+0000 7f9bd6036640 1 --2- 172.21.15.70:0/3634429393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc8095760 0x7f9bc8142510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.437+0000 7f9bd6036640 1 --2- 172.21.15.70:0/3634429393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bc8094da0 0x7f9bc80a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.437+0000 7f9bd6036640 1 --2- 172.21.15.70:0/3634429393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9bc8094040 0x7f9bc80a03e0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.437+0000 7f9bd6036640 1 -- 172.21.15.70:0/3634429393 >> 172.21.15.70:0/3634429393 conn(0x7f9bc808f950 msgr2=0x7f9bc8095fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.437+0000 7f9bd6036640 1 -- 172.21.15.70:0/3634429393 shutdown_connections 2024-04-01T01:45:33.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.437+0000 7f9bd6036640 1 -- 172.21.15.70:0/3634429393 wait complete. 2024-04-01T01:45:33.454 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:45:33.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.638+0000 7f331709d640 1 -- 172.21.15.70:0/1418734595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f331010b0c0 msgr2=0x7f331010d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.638+0000 7f331709d640 1 --2- 172.21.15.70:0/1418734595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f331010b0c0 0x7f331010d530 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f3300009900 tx=0x7f330002bc60 comp rx=0 tx=0).stop 2024-04-01T01:45:33.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.642+0000 7f331709d640 1 -- 172.21.15.70:0/1418734595 shutdown_connections 2024-04-01T01:45:33.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.642+0000 7f331709d640 1 --2- 172.21.15.70:0/1418734595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f331010da70 0x7f331010def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.642+0000 7f331709d640 1 --2- 172.21.15.70:0/1418734595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f331010b0c0 0x7f331010d530 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.642+0000 7f331709d640 1 --2- 172.21.15.70:0/1418734595 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3310108750 0x7f331010ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.642+0000 7f331709d640 1 -- 172.21.15.70:0/1418734595 >> 172.21.15.70:0/1418734595 conn(0x7f331006e5b0 msgr2=0x7f33100709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.644+0000 7f331709d640 1 -- 172.21.15.70:0/1418734595 shutdown_connections 2024-04-01T01:45:33.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.644+0000 7f331709d640 1 -- 172.21.15.70:0/1418734595 wait complete. 2024-04-01T01:45:33.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.645+0000 7f331709d640 1 Processor -- start 2024-04-01T01:45:33.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.645+0000 7f331709d640 1 -- start start 2024-04-01T01:45:33.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.646+0000 7f331709d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3310108750 0x7f33101a67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.646+0000 7f331709d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f331010b0c0 0x7f33101a6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.646+0000 7f331709d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f331010da70 0x7f33101add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.646+0000 7f331709d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f331011c140 con 0x7f331010b0c0 2024-04-01T01:45:33.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.646+0000 7f331709d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f331011bfc0 con 0x7f3310108750 2024-04-01T01:45:33.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.646+0000 7f331709d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f331011c2c0 con 0x7f331010da70 2024-04-01T01:45:33.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.646+0000 7f3314e39640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3310108750 0x7f33101a67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.646+0000 7f3314e39640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3310108750 0x7f33101a67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41204/0 (socket says 172.21.15.70:41204) 2024-04-01T01:45:33.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.646+0000 7f3314e39640 1 -- 172.21.15.70:0/2642656966 learned_addr learned my addr 172.21.15.70:0/2642656966 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.646+0000 7f330ffff640 1 --2- 172.21.15.70:0/2642656966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f331010b0c0 0x7f33101a6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.646+0000 7f331563a640 1 --2- 172.21.15.70:0/2642656966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f331010da70 0x7f33101add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.647+0000 7f3314e39640 1 -- 172.21.15.70:0/2642656966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f331010da70 msgr2=0x7f33101add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.647+0000 7f3314e39640 1 --2- 172.21.15.70:0/2642656966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f331010da70 0x7f33101add90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.647+0000 7f3314e39640 1 -- 172.21.15.70:0/2642656966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f331010b0c0 msgr2=0x7f33101a6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.647+0000 7f3314e39640 1 --2- 172.21.15.70:0/2642656966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f331010b0c0 0x7f33101a6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.647+0000 7f3314e39640 1 -- 172.21.15.70:0/2642656966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f331010ffb0 con 0x7f3310108750 2024-04-01T01:45:33.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.647+0000 7f330ffff640 1 --2- 172.21.15.70:0/2642656966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f331010b0c0 0x7f33101a6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:45:33.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.647+0000 7f3314e39640 1 --2- 172.21.15.70:0/2642656966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3310108750 0x7f33101a67e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f330400d950 tx=0x7f330400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.647+0000 7f330dffb640 1 -- 172.21.15.70:0/2642656966 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3304017070 con 0x7f3310108750 2024-04-01T01:45:33.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.647+0000 7f330dffb640 1 -- 172.21.15.70:0/2642656966 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3304009c30 con 0x7f3310108750 2024-04-01T01:45:33.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.647+0000 7f331709d640 1 -- 172.21.15.70:0/2642656966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33101102a0 con 0x7f3310108750 2024-04-01T01:45:33.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.647+0000 7f330dffb640 1 -- 172.21.15.70:0/2642656966 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f330400f620 con 0x7f3310108750 2024-04-01T01:45:33.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.647+0000 7f331709d640 1 -- 172.21.15.70:0/2642656966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3310110810 con 0x7f3310108750 2024-04-01T01:45:33.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.651+0000 7f330dffb640 1 -- 172.21.15.70:0/2642656966 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f330400f7c0 con 0x7f3310108750 2024-04-01T01:45:33.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.651+0000 7f3319c82740 1 Processor -- start 2024-04-01T01:45:33.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.651+0000 7f3319c82740 1 -- start start 2024-04-01T01:45:33.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.651+0000 7f3319c82740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4490 0x5594195b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.651+0000 7f3319c82740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b4e80 0x5594195b9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.651+0000 7f3319c82740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 0x5594195bc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.651+0000 7f3319c82740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5594195191b0 con 0x5594195b4490 2024-04-01T01:45:33.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.651+0000 7f3319c82740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5594194c1c00 con 0x5594195b9ff0 2024-04-01T01:45:33.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.651+0000 7f3319c82740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559419515250 con 0x5594195b4e80 2024-04-01T01:45:33.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.651+0000 7f330ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 0x5594195bc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.652+0000 7f331563a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4490 0x5594195b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.652+0000 7f330ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 0x5594195bc480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41212/0 (socket says 172.21.15.70:41212) 2024-04-01T01:45:33.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.652+0000 7f330ffff640 1 -- 172.21.15.70:0/1204781419 learned_addr learned my addr 172.21.15.70:0/1204781419 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.652+0000 7f3314e39640 1 --2- 172.21.15.70:0/1204781419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b4e80 0x5594195b9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.652+0000 7f330ffff640 1 -- 172.21.15.70:0/1204781419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b4e80 msgr2=0x5594195b9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.652+0000 7f330ffff640 1 --2- 172.21.15.70:0/1204781419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b4e80 0x5594195b9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.652+0000 7f330ffff640 1 -- 172.21.15.70:0/1204781419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4490 msgr2=0x5594195b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.652+0000 7f330ffff640 1 --2- 172.21.15.70:0/1204781419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4490 0x5594195b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.652+0000 7f330ffff640 1 -- 172.21.15.70:0/1204781419 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594194e13d0 con 0x5594195b9ff0 2024-04-01T01:45:33.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.652+0000 7f330ffff640 1 --2- 172.21.15.70:0/1204781419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 0x5594195bc480 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f330002c270 tx=0x7f330002ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e11d6ecc22382404 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.653+0000 7f32f6ffd640 1 -- 172.21.15.70:0/1204781419 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3300045020 con 0x5594195b9ff0 2024-04-01T01:45:33.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.653+0000 7f32f6ffd640 1 -- 172.21.15.70:0/1204781419 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3300005040 con 0x5594195b9ff0 2024-04-01T01:45:33.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.653+0000 7f330dffb640 1 --2- 172.21.15.70:0/2642656966 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32f80a2570 0x7f32f80a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.653+0000 7f331563a640 1 --2- 172.21.15.70:0/2642656966 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32f80a2570 0x7f32f80a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.653+0000 7f32f6ffd640 1 -- 172.21.15.70:0/1204781419 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3300004ae0 con 0x5594195b9ff0 2024-04-01T01:45:33.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.654+0000 7f3319c82740 1 -- 172.21.15.70:0/1204781419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 msgr2=0x5594195bc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.654+0000 7f3319c82740 1 --2- 172.21.15.70:0/1204781419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 0x5594195bc480 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f330002c270 tx=0x7f330002ccc0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.654+0000 7f3319c82740 1 -- 172.21.15.70:0/1204781419 shutdown_connections 2024-04-01T01:45:33.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.654+0000 7f3319c82740 1 --2- 172.21.15.70:0/1204781419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 0x5594195bc480 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.654+0000 7f3319c82740 1 --2- 172.21.15.70:0/1204781419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b4e80 0x5594195b9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.654+0000 7f3319c82740 1 --2- 172.21.15.70:0/1204781419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4490 0x5594195b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.654+0000 7f3319c82740 1 -- 172.21.15.70:0/1204781419 >> 172.21.15.70:0/1204781419 conn(0x5594195afa00 msgr2=0x5594195afe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.654+0000 7f330dffb640 1 -- 172.21.15.70:0/2642656966 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f33040ce6c0 con 0x7f3310108750 2024-04-01T01:45:33.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.655+0000 7f331563a640 1 --2- 172.21.15.70:0/2642656966 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32f80a2570 0x7f32f80a4a30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x5594195b56e0 tx=0x7f33080085f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.656+0000 7f3319c82740 1 -- 172.21.15.70:0/1204781419 shutdown_connections 2024-04-01T01:45:33.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.656+0000 7f3319c82740 1 -- 172.21.15.70:0/1204781419 wait complete. 2024-04-01T01:45:33.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.656+0000 7f3319c82740 1 -- start start 2024-04-01T01:45:33.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.656+0000 7f3319c82740 1 Processor -- start 2024-04-01T01:45:33.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.657+0000 7f3319c82740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b4490 0x5594195be8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.657+0000 7f3319c82740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4e80 0x5594195bee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.657+0000 7f3319c82740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 0x5594195c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.657+0000 7f3319c82740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5594195bcf90 con 0x5594195b4e80 2024-04-01T01:45:33.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.657+0000 7f3319c82740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5594194b7470 con 0x5594195b9ff0 2024-04-01T01:45:33.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.657+0000 7f3319c82740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5594195bd110 con 0x5594195b4490 2024-04-01T01:45:33.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.657+0000 7f3319c82740 10 client.0 _ll_register_callbacks cb 0x5594194d4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:33.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.657+0000 7f331563a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 0x5594195c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.657+0000 7f3314e39640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b4490 0x5594195be8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.657+0000 7f331563a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 0x5594195c5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41218/0 (socket says 172.21.15.70:41218) 2024-04-01T01:45:33.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.657+0000 7f331563a640 1 -- 172.21.15.70:0/1502400653 learned_addr learned my addr 172.21.15.70:0/1502400653 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.657+0000 7f330ffff640 1 --2- 172.21.15.70:0/1502400653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4e80 0x5594195bee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.658+0000 7f330ffff640 1 -- 172.21.15.70:0/1502400653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b4490 msgr2=0x5594195be8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.658+0000 7f330ffff640 1 --2- 172.21.15.70:0/1502400653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b4490 0x5594195be8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.658+0000 7f330ffff640 1 -- 172.21.15.70:0/1502400653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 msgr2=0x5594195c5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.658+0000 7f330ffff640 1 --2- 172.21.15.70:0/1502400653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 0x5594195c5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.658+0000 7f330ffff640 1 -- 172.21.15.70:0/1502400653 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55941922d3f0 con 0x5594195b4e80 2024-04-01T01:45:33.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.658+0000 7f330ffff640 1 --2- 172.21.15.70:0/1502400653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4e80 0x5594195bee30 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f3300005ab0 tx=0x7f3300004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e390790a35b6d673 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.658+0000 7f32effff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:45:33.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.659+0000 7f32effff640 1 -- 172.21.15.70:0/1502400653 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3300045040 con 0x5594195b4e80 2024-04-01T01:45:33.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.659+0000 7f3319c82740 1 -- 172.21.15.70:0/1502400653 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3300005040 con 0x5594195b4e80 2024-04-01T01:45:33.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.659+0000 7f32effff640 1 -- 172.21.15.70:0/1502400653 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3300041680 con 0x5594195b4e80 2024-04-01T01:45:33.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.659+0000 7f32effff640 1 -- 172.21.15.70:0/1502400653 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f33000418a0 con 0x5594195b4e80 2024-04-01T01:45:33.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.659+0000 7f3319c82740 10 client.18981 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:33.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.659+0000 7f3319c82740 1 -- 172.21.15.70:0/1502400653 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5594192f77d0 con 0x5594195b4e80 2024-04-01T01:45:33.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.660+0000 7f32effff640 1 -- 172.21.15.70:0/1502400653 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3300005040 con 0x5594195b4e80 2024-04-01T01:45:33.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.660+0000 7f32effff640 1 -- 172.21.15.70:0/1502400653 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f330004a980 con 0x5594195b4e80 2024-04-01T01:45:33.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.660+0000 7f3319c82740 10 client.18981 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:33.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.660+0000 7f3319c82740 10 client.18981 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:45:33.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.660+0000 7f3319c82740 20 client.18981 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:33.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.660+0000 7f3319c82740 1 --2- 172.21.15.70:0/1502400653 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5594195ca4f0 0x5594195cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.660+0000 7f3319c82740 4 client.18981 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:45:33.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.660+0000 7f3319c82740 1 -- 172.21.15.70:0/1502400653 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594195266f0 con 0x5594195ca4f0 2024-04-01T01:45:33.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.661+0000 7f3314e39640 1 --2- 172.21.15.70:0/1502400653 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5594195ca4f0 0x5594195cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.661+0000 7f3314e39640 1 --2- 172.21.15.70:0/1502400653 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5594195ca4f0 0x5594195cc950 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e6addd0fa66493bc server_cookie=1a198f281fa577a2 in_seq=0 out_seq=0 2024-04-01T01:45:33.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.662+0000 7f32effff640 10 client.18981 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:45:33.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.663+0000 7f32effff640 1 -- 172.21.15.70:0/1502400653 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5594195266f0 con 0x5594195ca4f0 2024-04-01T01:45:33.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.663+0000 7f32effff640 10 client.18981 handle_command_reply: tid=0 2024-04-01T01:45:33.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.663+0000 7f3319c82740 1 client.18981 shutdown 2024-04-01T01:45:33.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.664+0000 7f3319c82740 1 -- 172.21.15.70:0/1502400653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4e80 msgr2=0x5594195bee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.664+0000 7f3319c82740 1 --2- 172.21.15.70:0/1502400653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4e80 0x5594195bee30 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f3300005ab0 tx=0x7f3300004a90 comp rx=0 tx=0).stop 2024-04-01T01:45:33.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.668+0000 7f3319c82740 1 -- 172.21.15.70:0/1502400653 shutdown_connections 2024-04-01T01:45:33.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.668+0000 7f3319c82740 1 --2- 172.21.15.70:0/1502400653 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5594195ca4f0 0x5594195cc950 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.668+0000 7f3319c82740 1 --2- 172.21.15.70:0/1502400653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 0x5594195c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.668+0000 7f3319c82740 1 --2- 172.21.15.70:0/1502400653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4e80 0x5594195bee30 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.668+0000 7f3319c82740 1 --2- 172.21.15.70:0/1502400653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b4490 0x5594195be8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.668+0000 7f3319c82740 1 -- 172.21.15.70:0/1502400653 >> 172.21.15.70:0/1502400653 conn(0x5594195afa00 msgr2=0x5594195aed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.668+0000 7f32effff640 0 client.18981 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:45:33.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.668+0000 7f3319c82740 1 -- 172.21.15.70:0/1502400653 shutdown_connections 2024-04-01T01:45:33.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.668+0000 7f3319c82740 1 -- 172.21.15.70:0/1502400653 wait complete. 2024-04-01T01:45:33.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.668+0000 7f3319c82740 20 client.18981 trim_cache size 0 max 16384 2024-04-01T01:45:33.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.698+0000 7f3319c82740 1 Processor -- start 2024-04-01T01:45:33.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.700+0000 7f3319c82740 1 -- start start 2024-04-01T01:45:33.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.700+0000 7f3319c82740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b4490 0x5594195b98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.700+0000 7f3319c82740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4e80 0x5594195c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.700+0000 7f3319c82740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b9ff0 0x5594195afde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.700+0000 7f3319c82740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33101c1a10 con 0x5594195b4e80 2024-04-01T01:45:33.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.700+0000 7f3319c82740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f33101c3550 con 0x5594195b4490 2024-04-01T01:45:33.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.700+0000 7f3319c82740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f33101c20e0 con 0x5594195b9ff0 2024-04-01T01:45:33.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.700+0000 7f3314e39640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b4490 0x5594195b98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.700+0000 7f3314e39640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b4490 0x5594195b98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41230/0 (socket says 172.21.15.70:41230) 2024-04-01T01:45:33.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.701+0000 7f3314e39640 1 -- 172.21.15.70:0/304971639 learned_addr learned my addr 172.21.15.70:0/304971639 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.701+0000 7f3314e39640 1 -- 172.21.15.70:0/304971639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b9ff0 msgr2=0x5594195afde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:45:33.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.701+0000 7f3314e39640 1 --2- 172.21.15.70:0/304971639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b9ff0 0x5594195afde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.701+0000 7f3314e39640 1 -- 172.21.15.70:0/304971639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4e80 msgr2=0x5594195c5e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:45:33.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.701+0000 7f3314e39640 1 --2- 172.21.15.70:0/304971639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4e80 0x5594195c5e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.701+0000 7f3314e39640 1 -- 172.21.15.70:0/304971639 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594195b5710 con 0x5594195b4490 2024-04-01T01:45:33.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.701+0000 7f3314e39640 1 --2- 172.21.15.70:0/304971639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b4490 0x5594195b98b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f330409ba40 tx=0x7f33040cbd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7318fd308ecabfd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.701+0000 7f32f67fc640 1 -- 172.21.15.70:0/304971639 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3304020040 con 0x5594195b4490 2024-04-01T01:45:33.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.702+0000 7f32f67fc640 1 -- 172.21.15.70:0/304971639 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f330409fe90 con 0x5594195b4490 2024-04-01T01:45:33.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.702+0000 7f32f67fc640 1 -- 172.21.15.70:0/304971639 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33040ce4e0 con 0x5594195b4490 2024-04-01T01:45:33.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.703+0000 7f3319c82740 1 -- 172.21.15.70:0/304971639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b4490 msgr2=0x5594195b98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.703+0000 7f3319c82740 1 --2- 172.21.15.70:0/304971639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b4490 0x5594195b98b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f330409ba40 tx=0x7f33040cbd40 comp rx=0 tx=0).stop 2024-04-01T01:45:33.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.703+0000 7f3319c82740 1 -- 172.21.15.70:0/304971639 shutdown_connections 2024-04-01T01:45:33.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.703+0000 7f3319c82740 1 --2- 172.21.15.70:0/304971639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b9ff0 0x5594195afde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.703+0000 7f3319c82740 1 --2- 172.21.15.70:0/304971639 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4e80 0x5594195c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.703+0000 7f3319c82740 1 --2- 172.21.15.70:0/304971639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b4490 0x5594195b98b0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.703+0000 7f3319c82740 1 -- 172.21.15.70:0/304971639 >> 172.21.15.70:0/304971639 conn(0x5594195afa00 msgr2=0x5594195aabd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.704+0000 7f3319c82740 1 -- 172.21.15.70:0/304971639 shutdown_connections 2024-04-01T01:45:33.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.704+0000 7f3319c82740 1 -- 172.21.15.70:0/304971639 wait complete. 2024-04-01T01:45:33.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.704+0000 7f3319c82740 1 -- start start 2024-04-01T01:45:33.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.705+0000 7f3319c82740 1 Processor -- start 2024-04-01T01:45:33.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.705+0000 7f3319c82740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4490 0x5594195c2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.705+0000 7f3319c82740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b4e80 0x5594195c31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.705+0000 7f3319c82740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 0x55941962e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.705+0000 7f3319c82740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33101c2e80 con 0x5594195b4490 2024-04-01T01:45:33.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.705+0000 7f3319c82740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f33101c27b0 con 0x5594195b9ff0 2024-04-01T01:45:33.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.705+0000 7f3319c82740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3310118480 con 0x5594195b4e80 2024-04-01T01:45:33.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.705+0000 7f3319c82740 10 client.0 _ll_register_callbacks cb 0x7f33000055b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:33.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.705+0000 7f3314e39640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4490 0x5594195c2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.705+0000 7f3314e39640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4490 0x5594195c2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36226/0 (socket says 172.21.15.70:36226) 2024-04-01T01:45:33.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.705+0000 7f3314e39640 1 -- 172.21.15.70:0/2567460651 learned_addr learned my addr 172.21.15.70:0/2567460651 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.706+0000 7f3314e39640 1 -- 172.21.15.70:0/2567460651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b4e80 msgr2=0x5594195c31b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:45:33.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.706+0000 7f3314e39640 1 --2- 172.21.15.70:0/2567460651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b4e80 0x5594195c31b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.706+0000 7f3314e39640 1 -- 172.21.15.70:0/2567460651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 msgr2=0x55941962e880 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:45:33.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.706+0000 7f3314e39640 1 --2- 172.21.15.70:0/2567460651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 0x55941962e880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.706+0000 7f3314e39640 1 -- 172.21.15.70:0/2567460651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55941962edc0 con 0x5594195b4490 2024-04-01T01:45:33.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.706+0000 7f3314e39640 1 --2- 172.21.15.70:0/2567460651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4490 0x5594195c2c70 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f330409a290 tx=0x7f330400bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f88df1f07beb424 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.706+0000 7f32effff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:45:33.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.707+0000 7f32effff640 1 -- 172.21.15.70:0/2567460651 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3304020040 con 0x5594195b4490 2024-04-01T01:45:33.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.707+0000 7f32effff640 1 -- 172.21.15.70:0/2567460651 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f33040d0b90 con 0x5594195b4490 2024-04-01T01:45:33.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.709+0000 7f3319c82740 1 -- 172.21.15.70:0/2567460651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f330409fe90 con 0x5594195b4490 2024-04-01T01:45:33.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.713+0000 7f32effff640 1 -- 172.21.15.70:0/2567460651 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f330409a040 con 0x5594195b4490 2024-04-01T01:45:33.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.714+0000 7f3319c82740 10 client.18984 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:33.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.714+0000 7f3319c82740 1 -- 172.21.15.70:0/2567460651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55941962ff60 con 0x5594195b4490 2024-04-01T01:45:33.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.714+0000 7f32effff640 1 -- 172.21.15.70:0/2567460651 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f330409fe90 con 0x5594195b4490 2024-04-01T01:45:33.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.714+0000 7f32effff640 1 -- 172.21.15.70:0/2567460651 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f33040b4ce0 con 0x5594195b4490 2024-04-01T01:45:33.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.715+0000 7f3319c82740 10 client.18984 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:33.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.715+0000 7f3319c82740 10 client.18984 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:45:33.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.715+0000 7f3319c82740 20 client.18984 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:33.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.715+0000 7f3319c82740 1 --2- 172.21.15.70:0/2567460651 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x559419630890 0x559419632cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.715+0000 7f3319c82740 4 client.18984 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:45:33.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.715+0000 7f3319c82740 1 -- 172.21.15.70:0/2567460651 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559419520a40 con 0x559419630890 2024-04-01T01:45:33.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.716+0000 7f330ffff640 1 --2- 172.21.15.70:0/2567460651 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x559419630890 0x559419632cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.719+0000 7f330ffff640 1 --2- 172.21.15.70:0/2567460651 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x559419630890 0x559419632cb0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=991c32392fdc59b8 server_cookie=95e184e43a33bcd6 in_seq=0 out_seq=0 2024-04-01T01:45:33.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.721+0000 7f32effff640 10 client.18984 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:45:33.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.721+0000 7f32effff640 1 -- 172.21.15.70:0/2567460651 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559419520a40 con 0x559419630890 2024-04-01T01:45:33.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.721+0000 7f32effff640 10 client.18984 handle_command_reply: tid=0 2024-04-01T01:45:33.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.721+0000 7f3319c82740 1 client.18984 shutdown 2024-04-01T01:45:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.722+0000 7f3319c82740 1 -- 172.21.15.70:0/2567460651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4490 msgr2=0x5594195c2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.722+0000 7f3319c82740 1 --2- 172.21.15.70:0/2567460651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4490 0x5594195c2c70 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f330409a290 tx=0x7f330400bf60 comp rx=0 tx=0).stop 2024-04-01T01:45:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.723+0000 7f3319c82740 1 -- 172.21.15.70:0/2567460651 shutdown_connections 2024-04-01T01:45:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.723+0000 7f3319c82740 1 --2- 172.21.15.70:0/2567460651 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x559419630890 0x559419632cb0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.723+0000 7f3319c82740 1 --2- 172.21.15.70:0/2567460651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5594195b9ff0 0x55941962e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.723+0000 7f3319c82740 1 --2- 172.21.15.70:0/2567460651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594195b4e80 0x5594195c31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.723+0000 7f3319c82740 1 --2- 172.21.15.70:0/2567460651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594195b4490 0x5594195c2c70 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.723+0000 7f3319c82740 1 -- 172.21.15.70:0/2567460651 >> 172.21.15.70:0/2567460651 conn(0x5594195afa00 msgr2=0x5594195afde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.723+0000 7f32effff640 0 client.18984 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:45:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.723+0000 7f3319c82740 1 -- 172.21.15.70:0/2567460651 shutdown_connections 2024-04-01T01:45:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.723+0000 7f3319c82740 1 -- 172.21.15.70:0/2567460651 wait complete. 2024-04-01T01:45:33.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.723+0000 7f3319c82740 20 client.18984 trim_cache size 0 max 16384 2024-04-01T01:45:33.724 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:45:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.724+0000 7f331709d640 1 -- 172.21.15.70:0/2642656966 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32f80a2570 msgr2=0x7f32f80a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.725+0000 7f331709d640 1 --2- 172.21.15.70:0/2642656966 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32f80a2570 0x7f32f80a4a30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x5594195b56e0 tx=0x7f33080085f0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.725+0000 7f331709d640 1 -- 172.21.15.70:0/2642656966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3310108750 msgr2=0x7f33101a67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.725+0000 7f331709d640 1 --2- 172.21.15.70:0/2642656966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3310108750 0x7f33101a67e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f330400d950 tx=0x7f330400de10 comp rx=0 tx=0).stop 2024-04-01T01:45:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.725+0000 7f331709d640 1 -- 172.21.15.70:0/2642656966 shutdown_connections 2024-04-01T01:45:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.725+0000 7f331709d640 1 --2- 172.21.15.70:0/2642656966 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f32f80a2570 0x7f32f80a4a30 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.725+0000 7f331709d640 1 --2- 172.21.15.70:0/2642656966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f331010da70 0x7f33101add90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.725+0000 7f331709d640 1 --2- 172.21.15.70:0/2642656966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f331010b0c0 0x7f33101a6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.725+0000 7f331709d640 1 --2- 172.21.15.70:0/2642656966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3310108750 0x7f33101a67e0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.725+0000 7f331709d640 1 -- 172.21.15.70:0/2642656966 >> 172.21.15.70:0/2642656966 conn(0x7f331006e5b0 msgr2=0x7f3310109d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.725+0000 7f331709d640 1 -- 172.21.15.70:0/2642656966 shutdown_connections 2024-04-01T01:45:33.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.726+0000 7f331709d640 1 -- 172.21.15.70:0/2642656966 wait complete. 2024-04-01T01:45:33.744 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:45:33.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.929+0000 7f70d3e55640 1 -- 172.21.15.70:0/2678979434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70cc108750 msgr2=0x7f70cc10ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.929+0000 7f70d3e55640 1 --2- 172.21.15.70:0/2678979434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70cc108750 0x7f70cc10ab80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f70c4009be0 tx=0x7f70c402bef0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.930+0000 7f70d3e55640 1 -- 172.21.15.70:0/2678979434 shutdown_connections 2024-04-01T01:45:33.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.930+0000 7f70d3e55640 1 --2- 172.21.15.70:0/2678979434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70cc10da70 0x7f70cc10def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.930+0000 7f70d3e55640 1 --2- 172.21.15.70:0/2678979434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70cc10b0c0 0x7f70cc10d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.930+0000 7f70d3e55640 1 --2- 172.21.15.70:0/2678979434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70cc108750 0x7f70cc10ab80 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.930+0000 7f70d3e55640 1 -- 172.21.15.70:0/2678979434 >> 172.21.15.70:0/2678979434 conn(0x7f70cc06e5b0 msgr2=0x7f70cc0709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.931+0000 7f70d3e55640 1 -- 172.21.15.70:0/2678979434 shutdown_connections 2024-04-01T01:45:33.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.932+0000 7f70d3e55640 1 -- 172.21.15.70:0/2678979434 wait complete. 2024-04-01T01:45:33.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.933+0000 7f70d3e55640 1 Processor -- start 2024-04-01T01:45:33.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.933+0000 7f70d3e55640 1 -- start start 2024-04-01T01:45:33.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.933+0000 7f70d3e55640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70cc108750 0x7f70cc1a6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.933+0000 7f70d3e55640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70cc10b0c0 0x7f70cc1a6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.933+0000 7f70d3e55640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70cc10da70 0x7f70cc1addc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.933+0000 7f70d3e55640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f70cc11c100 con 0x7f70cc10da70 2024-04-01T01:45:33.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.933+0000 7f70d3e55640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f70cc11bf80 con 0x7f70cc108750 2024-04-01T01:45:33.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.933+0000 7f70d3e55640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f70cc11c280 con 0x7f70cc10b0c0 2024-04-01T01:45:33.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.934+0000 7f70d23f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70cc10da70 0x7f70cc1addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.934+0000 7f70d23f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70cc10da70 0x7f70cc1addc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36246/0 (socket says 172.21.15.70:36246) 2024-04-01T01:45:33.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.934+0000 7f70d23f2640 1 -- 172.21.15.70:0/2249184020 learned_addr learned my addr 172.21.15.70:0/2249184020 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.935+0000 7f70d13f0640 1 --2- 172.21.15.70:0/2249184020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70cc10b0c0 0x7f70cc1a6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.935+0000 7f70d23f2640 1 -- 172.21.15.70:0/2249184020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70cc10b0c0 msgr2=0x7f70cc1a6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.935+0000 7f70d1bf1640 1 --2- 172.21.15.70:0/2249184020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70cc108750 0x7f70cc1a6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.935+0000 7f70d23f2640 1 --2- 172.21.15.70:0/2249184020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70cc10b0c0 0x7f70cc1a6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.935+0000 7f70d23f2640 1 -- 172.21.15.70:0/2249184020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70cc108750 msgr2=0x7f70cc1a6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.935+0000 7f70d23f2640 1 --2- 172.21.15.70:0/2249184020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70cc108750 0x7f70cc1a6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.935+0000 7f70d23f2640 1 -- 172.21.15.70:0/2249184020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70cc1ae430 con 0x7f70cc10da70 2024-04-01T01:45:33.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.935+0000 7f70d13f0640 1 --2- 172.21.15.70:0/2249184020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70cc10b0c0 0x7f70cc1a6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:45:33.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.935+0000 7f70d1bf1640 1 --2- 172.21.15.70:0/2249184020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70cc108750 0x7f70cc1a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:45:33.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.935+0000 7f70d23f2640 1 --2- 172.21.15.70:0/2249184020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70cc10da70 0x7f70cc1addc0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00e970 tx=0x7f70bc00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.936+0000 7f70c2ffd640 1 -- 172.21.15.70:0/2249184020 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc00ccc0 con 0x7f70cc10da70 2024-04-01T01:45:33.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.936+0000 7f70c2ffd640 1 -- 172.21.15.70:0/2249184020 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70bc0054d0 con 0x7f70cc10da70 2024-04-01T01:45:33.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.936+0000 7f70c2ffd640 1 -- 172.21.15.70:0/2249184020 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc010430 con 0x7f70cc10da70 2024-04-01T01:45:33.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.936+0000 7f70d3e55640 1 -- 172.21.15.70:0/2249184020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70cc1ae720 con 0x7f70cc10da70 2024-04-01T01:45:33.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.936+0000 7f70d3e55640 1 -- 172.21.15.70:0/2249184020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70cc1aec20 con 0x7f70cc10da70 2024-04-01T01:45:33.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.940+0000 7f70d6a39740 1 Processor -- start 2024-04-01T01:45:33.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.940+0000 7f70d6a39740 1 -- start start 2024-04-01T01:45:33.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.940+0000 7f70d6a39740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b977d490 0x55a0b977d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.940+0000 7f70d6a39740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b977de80 0x55a0b9782980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.940+0000 7f70d6a39740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b9782ff0 0x55a0b9785480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.940+0000 7f70d6a39740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a0b96e21b0 con 0x55a0b977d490 2024-04-01T01:45:33.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.940+0000 7f70d6a39740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a0b968ac00 con 0x55a0b9782ff0 2024-04-01T01:45:33.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.940+0000 7f70d6a39740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a0b96de250 con 0x55a0b977de80 2024-04-01T01:45:33.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.940+0000 7f70d13f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b977d490 0x55a0b977d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.940+0000 7f70d1bf1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b977de80 0x55a0b9782980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.940+0000 7f70d13f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b977d490 0x55a0b977d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36254/0 (socket says 172.21.15.70:36254) 2024-04-01T01:45:33.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.940+0000 7f70d13f0640 1 -- 172.21.15.70:0/3382333629 learned_addr learned my addr 172.21.15.70:0/3382333629 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.940+0000 7f70d13f0640 1 --2- 172.21.15.70:0/3382333629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b9782ff0 0x55a0b9785480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.941+0000 7f70d13f0640 1 -- 172.21.15.70:0/3382333629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b977de80 msgr2=0x55a0b9782980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.941+0000 7f70d13f0640 1 --2- 172.21.15.70:0/3382333629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b977de80 0x55a0b9782980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.941+0000 7f70d13f0640 1 -- 172.21.15.70:0/3382333629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b9782ff0 msgr2=0x55a0b9785480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.941+0000 7f70d13f0640 1 --2- 172.21.15.70:0/3382333629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b9782ff0 0x55a0b9785480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.941+0000 7f70d13f0640 1 -- 172.21.15.70:0/3382333629 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0b96aa3d0 con 0x55a0b977d490 2024-04-01T01:45:33.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.941+0000 7f70d13f0640 1 --2- 172.21.15.70:0/3382333629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b977d490 0x55a0b977d8b0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f70c80158e0 tx=0x7f70c8009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a61244bd5a774d2b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.942+0000 7f70b7fff640 1 -- 172.21.15.70:0/3382333629 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c8004030 con 0x55a0b977d490 2024-04-01T01:45:33.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.942+0000 7f70b7fff640 1 -- 172.21.15.70:0/3382333629 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70c80041d0 con 0x55a0b977d490 2024-04-01T01:45:33.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.942+0000 7f70c2ffd640 1 -- 172.21.15.70:0/2249184020 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f70bc0046c0 con 0x7f70cc10da70 2024-04-01T01:45:33.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.942+0000 7f70b7fff640 1 -- 172.21.15.70:0/3382333629 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c80035f0 con 0x55a0b977d490 2024-04-01T01:45:33.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.943+0000 7f70c2ffd640 1 --2- 172.21.15.70:0/2249184020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f70b80a2620 0x7f70b80a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.943+0000 7f70d6a39740 1 -- 172.21.15.70:0/3382333629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b977d490 msgr2=0x55a0b977d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.943+0000 7f70d6a39740 1 --2- 172.21.15.70:0/3382333629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b977d490 0x55a0b977d8b0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f70c80158e0 tx=0x7f70c8009600 comp rx=0 tx=0).stop 2024-04-01T01:45:33.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.943+0000 7f70d6a39740 1 -- 172.21.15.70:0/3382333629 shutdown_connections 2024-04-01T01:45:33.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.943+0000 7f70d6a39740 1 --2- 172.21.15.70:0/3382333629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b9782ff0 0x55a0b9785480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.943+0000 7f70d6a39740 1 --2- 172.21.15.70:0/3382333629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b977de80 0x55a0b9782980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.943+0000 7f70d6a39740 1 --2- 172.21.15.70:0/3382333629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b977d490 0x55a0b977d8b0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.943+0000 7f70d6a39740 1 -- 172.21.15.70:0/3382333629 >> 172.21.15.70:0/3382333629 conn(0x55a0b9778a00 msgr2=0x55a0b9778e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.943+0000 7f70d6a39740 1 -- 172.21.15.70:0/3382333629 shutdown_connections 2024-04-01T01:45:33.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.943+0000 7f70c2ffd640 1 -- 172.21.15.70:0/2249184020 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f70bc0cedd0 con 0x7f70cc10da70 2024-04-01T01:45:33.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.943+0000 7f70d6a39740 1 -- 172.21.15.70:0/3382333629 wait complete. 2024-04-01T01:45:33.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.944+0000 7f70d6a39740 1 -- start start 2024-04-01T01:45:33.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.944+0000 7f70d6a39740 1 Processor -- start 2024-04-01T01:45:33.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.944+0000 7f70d6a39740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b977d490 0x55a0b97989d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.945+0000 7f70d6a39740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b977de80 0x55a0b9798f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.945+0000 7f70d1bf1640 1 --2- 172.21.15.70:0/2249184020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f70b80a2620 0x7f70b80a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.945+0000 7f70d6a39740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b9782ff0 0x55a0b979ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.945+0000 7f70d6a39740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a0b9796fa0 con 0x55a0b9782ff0 2024-04-01T01:45:33.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.945+0000 7f70d23f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b9782ff0 0x55a0b979ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.945+0000 7f70d23f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b9782ff0 0x55a0b979ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36264/0 (socket says 172.21.15.70:36264) 2024-04-01T01:45:33.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.945+0000 7f70d23f2640 1 -- 172.21.15.70:0/9300958 learned_addr learned my addr 172.21.15.70:0/9300958 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.945+0000 7f70d6a39740 1 -- 172.21.15.70:0/9300958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a0b9680470 con 0x55a0b977de80 2024-04-01T01:45:33.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.945+0000 7f70d6a39740 1 -- 172.21.15.70:0/9300958 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a0b9797120 con 0x55a0b977d490 2024-04-01T01:45:33.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.945+0000 7f70d6a39740 10 client.0 _ll_register_callbacks cb 0x55a0b969dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:33.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.946+0000 7f70d23f2640 1 -- 172.21.15.70:0/9300958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b977d490 msgr2=0x55a0b97989d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:45:33.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.946+0000 7f70d23f2640 1 --2- 172.21.15.70:0/9300958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b977d490 0x55a0b97989d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.946+0000 7f70d23f2640 1 -- 172.21.15.70:0/9300958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b977de80 msgr2=0x55a0b9798f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:45:33.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.946+0000 7f70d23f2640 1 --2- 172.21.15.70:0/9300958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b977de80 0x55a0b9798f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.946+0000 7f70d23f2640 1 -- 172.21.15.70:0/9300958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0b93f63f0 con 0x55a0b9782ff0 2024-04-01T01:45:33.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.946+0000 7f70d23f2640 1 --2- 172.21.15.70:0/9300958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b9782ff0 0x55a0b979ff50 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009380 tx=0x7f70bc005680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5557b17a8b66914 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.946+0000 7f70b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:45:33.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.946+0000 7f70b57fa640 1 -- 172.21.15.70:0/9300958 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc09f610 con 0x55a0b9782ff0 2024-04-01T01:45:33.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.946+0000 7f70b57fa640 1 -- 172.21.15.70:0/9300958 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f70bc0b76a0 con 0x55a0b9782ff0 2024-04-01T01:45:33.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.946+0000 7f70d1bf1640 1 --2- 172.21.15.70:0/2249184020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f70b80a2620 0x7f70b80a4ae0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f70c4000f80 tx=0x7f70c4034f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.946+0000 7f70d6a39740 1 -- 172.21.15.70:0/9300958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70c80041d0 con 0x55a0b9782ff0 2024-04-01T01:45:33.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.947+0000 7f70b57fa640 1 -- 172.21.15.70:0/9300958 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f70bc0a3ed0 con 0x55a0b9782ff0 2024-04-01T01:45:33.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.947+0000 7f70d6a39740 10 client.18996 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:33.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.947+0000 7f70d6a39740 1 -- 172.21.15.70:0/9300958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a0b94c07d0 con 0x55a0b9782ff0 2024-04-01T01:45:33.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.947+0000 7f70b57fa640 1 -- 172.21.15.70:0/9300958 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70c80041d0 con 0x55a0b9782ff0 2024-04-01T01:45:33.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.948+0000 7f70b57fa640 1 -- 172.21.15.70:0/9300958 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f70bc0a3a40 con 0x55a0b9782ff0 2024-04-01T01:45:33.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.948+0000 7f70d6a39740 10 client.18996 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:33.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.948+0000 7f70d6a39740 10 client.18996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:33.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.948+0000 7f70d6a39740 20 client.18996 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:33.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.948+0000 7f70d6a39740 1 --2- 172.21.15.70:0/9300958 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a0b97a44e0 0x55a0b97a6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.948+0000 7f70d6a39740 4 client.18996 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:45:33.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.948+0000 7f70d6a39740 1 -- 172.21.15.70:0/9300958 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0b96ef6f0 con 0x55a0b97a44e0 2024-04-01T01:45:33.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.949+0000 7f70d1bf1640 1 --2- 172.21.15.70:0/9300958 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a0b97a44e0 0x55a0b97a6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.950+0000 7f70d1bf1640 1 --2- 172.21.15.70:0/9300958 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a0b97a44e0 0x55a0b97a6960 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=958b2e6c72ff55cd server_cookie=55cf7e347b90b74b in_seq=0 out_seq=0 2024-04-01T01:45:33.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.950+0000 7f70b57fa640 10 client.18996 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:33.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.951+0000 7f70b57fa640 1 -- 172.21.15.70:0/9300958 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a0b96ef6f0 con 0x55a0b97a44e0 2024-04-01T01:45:33.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.951+0000 7f70b57fa640 10 client.18996 handle_command_reply: tid=0 2024-04-01T01:45:33.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.951+0000 7f70d6a39740 1 client.18996 shutdown 2024-04-01T01:45:33.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.951+0000 7f70d6a39740 1 -- 172.21.15.70:0/9300958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b9782ff0 msgr2=0x55a0b979ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.951+0000 7f70d6a39740 1 --2- 172.21.15.70:0/9300958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b9782ff0 0x55a0b979ff50 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009380 tx=0x7f70bc005680 comp rx=0 tx=0).stop 2024-04-01T01:45:33.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.952+0000 7f70d6a39740 1 -- 172.21.15.70:0/9300958 shutdown_connections 2024-04-01T01:45:33.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.952+0000 7f70d6a39740 1 --2- 172.21.15.70:0/9300958 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a0b97a44e0 0x55a0b97a6960 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.952+0000 7f70d6a39740 1 --2- 172.21.15.70:0/9300958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b9782ff0 0x55a0b979ff50 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.952+0000 7f70d6a39740 1 --2- 172.21.15.70:0/9300958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b977de80 0x55a0b9798f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.952+0000 7f70b57fa640 0 client.18996 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:33.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.953+0000 7f70d6a39740 1 --2- 172.21.15.70:0/9300958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b977d490 0x55a0b97989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.953+0000 7f70d6a39740 1 -- 172.21.15.70:0/9300958 >> 172.21.15.70:0/9300958 conn(0x55a0b9778a00 msgr2=0x55a0b9779c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.953+0000 7f70d6a39740 1 -- 172.21.15.70:0/9300958 shutdown_connections 2024-04-01T01:45:33.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.953+0000 7f70d6a39740 1 -- 172.21.15.70:0/9300958 wait complete. 2024-04-01T01:45:33.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.953+0000 7f70d6a39740 20 client.18996 trim_cache size 0 max 16384 2024-04-01T01:45:33.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.984+0000 7f70d6a39740 1 Processor -- start 2024-04-01T01:45:33.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.984+0000 7f70d6a39740 1 -- start start 2024-04-01T01:45:33.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.984+0000 7f70d6a39740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b977d490 0x55a0b97983b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.985+0000 7f70d6a39740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b977de80 0x55a0b979ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.985+0000 7f70d6a39740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b9782ff0 0x55a0b9773bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.985+0000 7f70d1bf1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b977d490 0x55a0b97983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.985+0000 7f70d1bf1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b977d490 0x55a0b97983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41278/0 (socket says 172.21.15.70:41278) 2024-04-01T01:45:33.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.985+0000 7f70d1bf1640 1 -- 172.21.15.70:0/3314519215 learned_addr learned my addr 172.21.15.70:0/3314519215 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.985+0000 7f70d23f2640 1 --2- 172.21.15.70:0/3314519215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b9782ff0 0x55a0b9773bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.985+0000 7f70d6a39740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f70b000fbe0 con 0x55a0b977de80 2024-04-01T01:45:33.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.985+0000 7f70d6a39740 1 -- 172.21.15.70:0/3314519215 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f70b0011720 con 0x55a0b977d490 2024-04-01T01:45:33.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.985+0000 7f70d6a39740 1 -- 172.21.15.70:0/3314519215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f70b00102b0 con 0x55a0b9782ff0 2024-04-01T01:45:33.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.985+0000 7f70d13f0640 1 --2- 172.21.15.70:0/3314519215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b977de80 0x55a0b979ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.985+0000 7f70d23f2640 1 -- 172.21.15.70:0/3314519215 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b977d490 msgr2=0x55a0b97983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.985+0000 7f70d23f2640 1 --2- 172.21.15.70:0/3314519215 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b977d490 0x55a0b97983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.985+0000 7f70d23f2640 1 -- 172.21.15.70:0/3314519215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b977de80 msgr2=0x55a0b979ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.985+0000 7f70d23f2640 1 --2- 172.21.15.70:0/3314519215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b977de80 0x55a0b979ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.985+0000 7f70d23f2640 1 -- 172.21.15.70:0/3314519215 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0b9799260 con 0x55a0b9782ff0 2024-04-01T01:45:33.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.985+0000 7f70d23f2640 1 --2- 172.21.15.70:0/3314519215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b9782ff0 0x55a0b9773bd0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f70bc097e40 tx=0x7f70bc09b3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31a1989289c9c80e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.986+0000 7f70b77fe640 1 -- 172.21.15.70:0/3314519215 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc09f610 con 0x55a0b9782ff0 2024-04-01T01:45:33.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.986+0000 7f70b77fe640 1 -- 172.21.15.70:0/3314519215 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70bc016df0 con 0x55a0b9782ff0 2024-04-01T01:45:33.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.986+0000 7f70b77fe640 1 -- 172.21.15.70:0/3314519215 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc0bb0b0 con 0x55a0b9782ff0 2024-04-01T01:45:33.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.987+0000 7f70d6a39740 1 -- 172.21.15.70:0/3314519215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b9782ff0 msgr2=0x55a0b9773bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.987+0000 7f70d6a39740 1 --2- 172.21.15.70:0/3314519215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b9782ff0 0x55a0b9773bd0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f70bc097e40 tx=0x7f70bc09b3c0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.987+0000 7f70d6a39740 1 -- 172.21.15.70:0/3314519215 shutdown_connections 2024-04-01T01:45:33.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.987+0000 7f70d6a39740 1 --2- 172.21.15.70:0/3314519215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b9782ff0 0x55a0b9773bd0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.987+0000 7f70d6a39740 1 --2- 172.21.15.70:0/3314519215 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b977de80 0x55a0b979ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.987+0000 7f70d6a39740 1 --2- 172.21.15.70:0/3314519215 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b977d490 0x55a0b97983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.987+0000 7f70d6a39740 1 -- 172.21.15.70:0/3314519215 >> 172.21.15.70:0/3314519215 conn(0x55a0b9778a00 msgr2=0x55a0b9797ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.988+0000 7f70d6a39740 1 -- 172.21.15.70:0/3314519215 shutdown_connections 2024-04-01T01:45:33.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.988+0000 7f70d6a39740 1 -- 172.21.15.70:0/3314519215 wait complete. 2024-04-01T01:45:33.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.989+0000 7f70d6a39740 1 -- start start 2024-04-01T01:45:33.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.989+0000 7f70d6a39740 1 Processor -- start 2024-04-01T01:45:33.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.989+0000 7f70d6a39740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b977d490 0x55a0b979fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.989+0000 7f70d6a39740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b977de80 0x55a0b979e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.989+0000 7f70d6a39740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b9782ff0 0x55a0b979e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.989+0000 7f70d6a39740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f70b0011050 con 0x55a0b9782ff0 2024-04-01T01:45:33.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.989+0000 7f70d6a39740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f70b0010980 con 0x55a0b977d490 2024-04-01T01:45:33.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.989+0000 7f70d6a39740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f70b000dbc0 con 0x55a0b977de80 2024-04-01T01:45:33.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.989+0000 7f70d6a39740 10 client.0 _ll_register_callbacks cb 0x7f70bc09ae20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:33.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.990+0000 7f70d23f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b9782ff0 0x55a0b979e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.990+0000 7f70d1bf1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b977d490 0x55a0b979fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.990+0000 7f70d23f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b9782ff0 0x55a0b979e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36282/0 (socket says 172.21.15.70:36282) 2024-04-01T01:45:33.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.990+0000 7f70d23f2640 1 -- 172.21.15.70:0/2897561732 learned_addr learned my addr 172.21.15.70:0/2897561732 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:33.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.990+0000 7f70d13f0640 1 --2- 172.21.15.70:0/2897561732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b977de80 0x55a0b979e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.990+0000 7f70d23f2640 1 -- 172.21.15.70:0/2897561732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b977de80 msgr2=0x55a0b979e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.990+0000 7f70d23f2640 1 --2- 172.21.15.70:0/2897561732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b977de80 0x55a0b979e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.990+0000 7f70d23f2640 1 -- 172.21.15.70:0/2897561732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b977d490 msgr2=0x55a0b979fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:33.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.990+0000 7f70d23f2640 1 --2- 172.21.15.70:0/2897561732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b977d490 0x55a0b979fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.990+0000 7f70d23f2640 1 -- 172.21.15.70:0/2897561732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0b979eb50 con 0x55a0b9782ff0 2024-04-01T01:45:33.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.990+0000 7f70d1bf1640 1 --2- 172.21.15.70:0/2897561732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b977d490 0x55a0b979fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:45:33.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.990+0000 7f70d23f2640 1 --2- 172.21.15.70:0/2897561732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b9782ff0 0x55a0b979e610 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f70bc0a39a0 tx=0x7f70bc0b4370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15f06e5e6037a35 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:33.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.990+0000 7f70b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:45:33.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.991+0000 7f70b57fa640 1 -- 172.21.15.70:0/2897561732 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc0a3b90 con 0x55a0b9782ff0 2024-04-01T01:45:33.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.991+0000 7f70d6a39740 1 -- 172.21.15.70:0/2897561732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70bc016df0 con 0x55a0b9782ff0 2024-04-01T01:45:33.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.991+0000 7f70b57fa640 1 -- 172.21.15.70:0/2897561732 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f70bc0b7360 con 0x55a0b9782ff0 2024-04-01T01:45:33.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.991+0000 7f70b57fa640 1 -- 172.21.15.70:0/2897561732 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f70bc09f610 con 0x55a0b9782ff0 2024-04-01T01:45:33.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.991+0000 7f70d6a39740 10 client.19002 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:33.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.991+0000 7f70d6a39740 1 -- 172.21.15.70:0/2897561732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a0b980b4e0 con 0x55a0b9782ff0 2024-04-01T01:45:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.992+0000 7f70b57fa640 1 -- 172.21.15.70:0/2897561732 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70bc016df0 con 0x55a0b9782ff0 2024-04-01T01:45:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.992+0000 7f70b57fa640 1 -- 172.21.15.70:0/2897561732 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f70bc09a380 con 0x55a0b9782ff0 2024-04-01T01:45:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.992+0000 7f70d6a39740 10 client.19002 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.992+0000 7f70d6a39740 10 client.19002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.992+0000 7f70d6a39740 20 client.19002 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.992+0000 7f70d6a39740 1 --2- 172.21.15.70:0/2897561732 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a0b97a44e0 0x55a0b980dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.992+0000 7f70d6a39740 4 client.19002 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:45:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.992+0000 7f70d6a39740 1 -- 172.21.15.70:0/2897561732 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a0b96e9a40 con 0x55a0b97a44e0 2024-04-01T01:45:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.992+0000 7f70d1bf1640 1 --2- 172.21.15.70:0/2897561732 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a0b97a44e0 0x55a0b980dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.993+0000 7f70d1bf1640 1 --2- 172.21.15.70:0/2897561732 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a0b97a44e0 0x55a0b980dc90 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d46f1cfac3f63bf3 server_cookie=36bd1cad18578c74 in_seq=0 out_seq=0 2024-04-01T01:45:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.993+0000 7f70b57fa640 10 client.19002 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.993+0000 7f70b57fa640 1 -- 172.21.15.70:0/2897561732 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a0b96e9a40 con 0x55a0b97a44e0 2024-04-01T01:45:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.993+0000 7f70b57fa640 10 client.19002 handle_command_reply: tid=0 2024-04-01T01:45:33.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.993+0000 7f70d6a39740 1 client.19002 shutdown 2024-04-01T01:45:33.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.994+0000 7f70d6a39740 1 -- 172.21.15.70:0/2897561732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b9782ff0 msgr2=0x55a0b979e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.994+0000 7f70d6a39740 1 --2- 172.21.15.70:0/2897561732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b9782ff0 0x55a0b979e610 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f70bc0a39a0 tx=0x7f70bc0b4370 comp rx=0 tx=0).stop 2024-04-01T01:45:33.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.994+0000 7f70d6a39740 1 -- 172.21.15.70:0/2897561732 shutdown_connections 2024-04-01T01:45:33.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.994+0000 7f70d6a39740 1 --2- 172.21.15.70:0/2897561732 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a0b97a44e0 0x55a0b980dc90 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.994+0000 7f70d6a39740 1 --2- 172.21.15.70:0/2897561732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0b9782ff0 0x55a0b979e610 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.994+0000 7f70d6a39740 1 --2- 172.21.15.70:0/2897561732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0b977de80 0x55a0b979e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.994+0000 7f70d6a39740 1 --2- 172.21.15.70:0/2897561732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0b977d490 0x55a0b979fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.994+0000 7f70b57fa640 0 client.19002 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:33.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.994+0000 7f70d6a39740 1 -- 172.21.15.70:0/2897561732 >> 172.21.15.70:0/2897561732 conn(0x55a0b9778a00 msgr2=0x55a0b97774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:33.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.994+0000 7f70d6a39740 1 -- 172.21.15.70:0/2897561732 shutdown_connections 2024-04-01T01:45:33.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.995+0000 7f70d6a39740 1 -- 172.21.15.70:0/2897561732 wait complete. 2024-04-01T01:45:33.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.995+0000 7f70d6a39740 20 client.19002 trim_cache size 0 max 16384 2024-04-01T01:45:33.998 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:45:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.996+0000 7f70d3e55640 1 -- 172.21.15.70:0/2249184020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f70b80a2620 msgr2=0x7f70b80a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.996+0000 7f70d3e55640 1 --2- 172.21.15.70:0/2249184020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f70b80a2620 0x7f70b80a4ae0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f70c4000f80 tx=0x7f70c4034f90 comp rx=0 tx=0).stop 2024-04-01T01:45:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.996+0000 7f70d3e55640 1 -- 172.21.15.70:0/2249184020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70cc10da70 msgr2=0x7f70cc1addc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.996+0000 7f70d3e55640 1 --2- 172.21.15.70:0/2249184020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70cc10da70 0x7f70cc1addc0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f70bc00e970 tx=0x7f70bc00ee30 comp rx=0 tx=0).stop 2024-04-01T01:45:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.996+0000 7f70d3e55640 1 -- 172.21.15.70:0/2249184020 shutdown_connections 2024-04-01T01:45:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.996+0000 7f70d3e55640 1 --2- 172.21.15.70:0/2249184020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70cc10da70 0x7f70cc1addc0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.996+0000 7f70d3e55640 1 --2- 172.21.15.70:0/2249184020 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f70b80a2620 0x7f70b80a4ae0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.996+0000 7f70d3e55640 1 --2- 172.21.15.70:0/2249184020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70cc10b0c0 0x7f70cc1a6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:33.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.996+0000 7f70d3e55640 1 --2- 172.21.15.70:0/2249184020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70cc108750 0x7f70cc1a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:34.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.996+0000 7f70d3e55640 1 -- 172.21.15.70:0/2249184020 >> 172.21.15.70:0/2249184020 conn(0x7f70cc06e5b0 msgr2=0x7f70cc10b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:34.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.996+0000 7f70d3e55640 1 -- 172.21.15.70:0/2249184020 shutdown_connections 2024-04-01T01:45:34.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:33.996+0000 7f70d3e55640 1 -- 172.21.15.70:0/2249184020 wait complete. 2024-04-01T01:45:34.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:33 smithi070 ceph-mon[32179]: pgmap v721: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 9.7 MiB/s wr, 798 op/s 2024-04-01T01:45:34.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:33 smithi189 ceph-mon[35753]: pgmap v721: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 9.7 MiB/s wr, 798 op/s 2024-04-01T01:45:34.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:33 smithi081 ceph-mon[35142]: pgmap v721: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 9.7 MiB/s wr, 798 op/s 2024-04-01T01:45:35.018 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:45:35.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:45:35.019 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:45:35.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.200+0000 7f563728b640 1 -- 172.21.15.70:0/2306183859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5630107c70 msgr2=0x7f5630108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.200+0000 7f563728b640 1 --2- 172.21.15.70:0/2306183859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5630107c70 0x7f5630108090 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5628009860 tx=0x7f562802bb70 comp rx=0 tx=0).stop 2024-04-01T01:45:35.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.200+0000 7f563728b640 1 -- 172.21.15.70:0/2306183859 shutdown_connections 2024-04-01T01:45:35.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.200+0000 7f563728b640 1 --2- 172.21.15.70:0/2306183859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5630109820 0x7f56301100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.200+0000 7f563728b640 1 --2- 172.21.15.70:0/2306183859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5630108e60 0x7f56301092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.200+0000 7f563728b640 1 --2- 172.21.15.70:0/2306183859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5630107c70 0x7f5630108090 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.200+0000 7f563728b640 1 -- 172.21.15.70:0/2306183859 >> 172.21.15.70:0/2306183859 conn(0x7f5630076c20 msgr2=0x7f5630079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:35.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.201+0000 7f563728b640 1 -- 172.21.15.70:0/2306183859 shutdown_connections 2024-04-01T01:45:35.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.202+0000 7f563728b640 1 -- 172.21.15.70:0/2306183859 wait complete. 2024-04-01T01:45:35.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.202+0000 7f563728b640 1 Processor -- start 2024-04-01T01:45:35.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.202+0000 7f563728b640 1 -- start start 2024-04-01T01:45:35.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.203+0000 7f563728b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5630107c70 0x7f563007eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.203+0000 7f563728b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5630108e60 0x7f563007f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.203+0000 7f563728b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5630109820 0x7f56301aded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.203+0000 7f5635a88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5630108e60 0x7f563007f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.203+0000 7f5635a88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5630108e60 0x7f563007f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36312/0 (socket says 172.21.15.70:36312) 2024-04-01T01:45:35.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.203+0000 7f563728b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f56301137c0 con 0x7f5630108e60 2024-04-01T01:45:35.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.203+0000 7f563728b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5630113640 con 0x7f5630107c70 2024-04-01T01:45:35.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.203+0000 7f563728b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5630113940 con 0x7f5630109820 2024-04-01T01:45:35.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.203+0000 7f5636a8a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5630109820 0x7f56301aded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.203+0000 7f5636289640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5630107c70 0x7f563007eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.203+0000 7f5636a8a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5630109820 0x7f56301aded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58564/0 (socket says 172.21.15.70:58564) 2024-04-01T01:45:35.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.203+0000 7f5636a8a640 1 -- 172.21.15.70:0/3288748433 learned_addr learned my addr 172.21.15.70:0/3288748433 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:35.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.204+0000 7f5636289640 1 -- 172.21.15.70:0/3288748433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5630109820 msgr2=0x7f56301aded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.204+0000 7f5636289640 1 --2- 172.21.15.70:0/3288748433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5630109820 0x7f56301aded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.204+0000 7f5636289640 1 -- 172.21.15.70:0/3288748433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5630108e60 msgr2=0x7f563007f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.204+0000 7f5636289640 1 --2- 172.21.15.70:0/3288748433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5630108e60 0x7f563007f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.204+0000 7f5636289640 1 -- 172.21.15.70:0/3288748433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5630082540 con 0x7f5630107c70 2024-04-01T01:45:35.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.205+0000 7f5636289640 1 --2- 172.21.15.70:0/3288748433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5630107c70 0x7f563007eed0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5628009780 tx=0x7f5628035bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:35.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.205+0000 7f562f7fe640 1 -- 172.21.15.70:0/3288748433 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628041020 con 0x7f5630107c70 2024-04-01T01:45:35.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.205+0000 7f562f7fe640 1 -- 172.21.15.70:0/3288748433 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56280035f0 con 0x7f5630107c70 2024-04-01T01:45:35.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.205+0000 7f562f7fe640 1 -- 172.21.15.70:0/3288748433 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628039040 con 0x7f5630107c70 2024-04-01T01:45:35.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.206+0000 7f563728b640 1 -- 172.21.15.70:0/3288748433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56301ae470 con 0x7f5630107c70 2024-04-01T01:45:35.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.206+0000 7f563728b640 1 -- 172.21.15.70:0/3288748433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56301ae820 con 0x7f5630107c70 2024-04-01T01:45:35.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.209+0000 7f562f7fe640 1 -- 172.21.15.70:0/3288748433 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f5628044020 con 0x7f5630107c70 2024-04-01T01:45:35.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.210+0000 7f563b1a9740 1 Processor -- start 2024-04-01T01:45:35.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.210+0000 7f563b1a9740 1 -- start start 2024-04-01T01:45:35.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.210+0000 7f563b1a9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37bcbf0 0x55abb37bd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.210+0000 7f563b1a9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37bd5e0 0x55abb37c20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.210+0000 7f563b1a9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37c2750 0x55abb37c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.210+0000 7f563b1a9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55abb36be0b0 con 0x55abb37bcbf0 2024-04-01T01:45:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.210+0000 7f563b1a9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55abb3720ea0 con 0x55abb37bd5e0 2024-04-01T01:45:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.210+0000 7f563b1a9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55abb36c9c00 con 0x55abb37c2750 2024-04-01T01:45:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.210+0000 7f5636a8a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37bcbf0 0x55abb37bd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.210+0000 7f5636a8a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37bcbf0 0x55abb37bd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36328/0 (socket says 172.21.15.70:36328) 2024-04-01T01:45:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.210+0000 7f5636a8a640 1 -- 172.21.15.70:0/2578213452 learned_addr learned my addr 172.21.15.70:0/2578213452 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.211+0000 7f5635a88640 1 --2- 172.21.15.70:0/2578213452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37c2750 0x55abb37c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.211+0000 7f5636a8a640 1 -- 172.21.15.70:0/2578213452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37c2750 msgr2=0x55abb37c4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.211+0000 7f5636a8a640 1 --2- 172.21.15.70:0/2578213452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37c2750 0x55abb37c4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.211+0000 7f5636a8a640 1 -- 172.21.15.70:0/2578213452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37bd5e0 msgr2=0x55abb37c20e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:45:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.211+0000 7f5636a8a640 1 --2- 172.21.15.70:0/2578213452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37bd5e0 0x55abb37c20e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.211+0000 7f5636a8a640 1 -- 172.21.15.70:0/2578213452 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abb34fd7e0 con 0x55abb37bcbf0 2024-04-01T01:45:35.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.211+0000 7f562f7fe640 1 --2- 172.21.15.70:0/3288748433 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56180a26a0 0x7f56180a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.212+0000 7f5636a8a640 1 --2- 172.21.15.70:0/2578213452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37bcbf0 0x55abb37bd010 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f562000e550 tx=0x7f56200095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca86d50f8a03e536 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:35.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.212+0000 7f562f7fe640 1 -- 172.21.15.70:0/3288748433 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f56280f0490 con 0x7f5630107c70 2024-04-01T01:45:35.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.212+0000 7f562cbfe640 1 -- 172.21.15.70:0/2578213452 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562000d990 con 0x55abb37bcbf0 2024-04-01T01:45:35.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.212+0000 7f562cbfe640 1 -- 172.21.15.70:0/2578213452 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f562001c070 con 0x55abb37bcbf0 2024-04-01T01:45:35.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.213+0000 7f5636289640 1 --2- 172.21.15.70:0/3288748433 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56180a26a0 0x7f56180a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.213+0000 7f563b1a9740 1 -- 172.21.15.70:0/2578213452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37bcbf0 msgr2=0x55abb37bd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.213+0000 7f563b1a9740 1 --2- 172.21.15.70:0/2578213452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37bcbf0 0x55abb37bd010 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f562000e550 tx=0x7f56200095c0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.214+0000 7f563b1a9740 1 -- 172.21.15.70:0/2578213452 shutdown_connections 2024-04-01T01:45:35.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.214+0000 7f563b1a9740 1 --2- 172.21.15.70:0/2578213452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37c2750 0x55abb37c4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.214+0000 7f563b1a9740 1 --2- 172.21.15.70:0/2578213452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37bd5e0 0x55abb37c20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.214+0000 7f563b1a9740 1 --2- 172.21.15.70:0/2578213452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37bcbf0 0x55abb37bd010 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.214+0000 7f5636289640 1 --2- 172.21.15.70:0/3288748433 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56180a26a0 0x7f56180a4b60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f56280b94f0 tx=0x7f5628003aa0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:35.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.214+0000 7f563b1a9740 1 -- 172.21.15.70:0/2578213452 >> 172.21.15.70:0/2578213452 conn(0x55abb37b8160 msgr2=0x55abb37b8560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:35.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.215+0000 7f563b1a9740 1 -- 172.21.15.70:0/2578213452 shutdown_connections 2024-04-01T01:45:35.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.215+0000 7f563b1a9740 1 -- 172.21.15.70:0/2578213452 wait complete. 2024-04-01T01:45:35.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.215+0000 7f563b1a9740 1 -- start start 2024-04-01T01:45:35.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.215+0000 7f563b1a9740 1 Processor -- start 2024-04-01T01:45:35.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.216+0000 7f563b1a9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37bd5e0 0x55abb37d7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.216+0000 7f563b1a9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37c2750 0x55abb37d8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.216+0000 7f563b1a9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37dd4b0 0x55abb37df8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.216+0000 7f563b1a9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55abb36bf3d0 con 0x55abb37bd5e0 2024-04-01T01:45:35.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.216+0000 7f563b1a9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55abb371cf40 con 0x55abb37dd4b0 2024-04-01T01:45:35.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.216+0000 7f563b1a9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55abb37d6760 con 0x55abb37c2750 2024-04-01T01:45:35.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.216+0000 7f563b1a9740 10 client.0 _ll_register_callbacks cb 0x55abb36dcb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:35.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.216+0000 7f5636a8a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37bd5e0 0x55abb37d7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.216+0000 7f5636a8a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37bd5e0 0x55abb37d7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36330/0 (socket says 172.21.15.70:36330) 2024-04-01T01:45:35.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.216+0000 7f5636a8a640 1 -- 172.21.15.70:0/3762281832 learned_addr learned my addr 172.21.15.70:0/3762281832 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:35.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.216+0000 7f5635a88640 1 --2- 172.21.15.70:0/3762281832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37c2750 0x55abb37d8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.216+0000 7f5636a8a640 1 --2- 172.21.15.70:0/3762281832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37dd4b0 0x55abb37df8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.217+0000 7f5636a8a640 1 -- 172.21.15.70:0/3762281832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37c2750 msgr2=0x55abb37d8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:35.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.217+0000 7f5636a8a640 1 --2- 172.21.15.70:0/3762281832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37c2750 0x55abb37d8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.217+0000 7f5636a8a640 1 -- 172.21.15.70:0/3762281832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37dd4b0 msgr2=0x55abb37df8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:35.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.217+0000 7f5636a8a640 1 --2- 172.21.15.70:0/3762281832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37dd4b0 0x55abb37df8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.217+0000 7f5636a8a640 1 -- 172.21.15.70:0/3762281832 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abb355df70 con 0x55abb37bd5e0 2024-04-01T01:45:35.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.217+0000 7f5636a8a640 1 --2- 172.21.15.70:0/3762281832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37bd5e0 0x55abb37d7f40 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5620009c00 tx=0x7f5620018e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688fad8bbf584052 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:35.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.217+0000 7f5615ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:45:35.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.218+0000 7f5615ffb640 1 -- 172.21.15.70:0/3762281832 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5620004a00 con 0x55abb37bd5e0 2024-04-01T01:45:35.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.218+0000 7f563b1a9740 1 -- 172.21.15.70:0/3762281832 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55abb372e3e0 con 0x55abb37bd5e0 2024-04-01T01:45:35.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.218+0000 7f5615ffb640 1 -- 172.21.15.70:0/3762281832 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5620004ba0 con 0x55abb37bd5e0 2024-04-01T01:45:35.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.218+0000 7f5615ffb640 1 -- 172.21.15.70:0/3762281832 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5620004da0 con 0x55abb37bd5e0 2024-04-01T01:45:35.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.218+0000 7f563b1a9740 10 client.19014 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:35.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.218+0000 7f563b1a9740 1 -- 172.21.15.70:0/3762281832 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55abb33f5170 con 0x55abb37bd5e0 2024-04-01T01:45:35.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.219+0000 7f5615ffb640 1 -- 172.21.15.70:0/3762281832 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55abb372e3e0 con 0x55abb37bd5e0 2024-04-01T01:45:35.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.219+0000 7f5615ffb640 1 -- 172.21.15.70:0/3762281832 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f562000b040 con 0x55abb37bd5e0 2024-04-01T01:45:35.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.220+0000 7f563b1a9740 10 client.19014 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:35.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.220+0000 7f563b1a9740 10 client.19014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:35.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.220+0000 7f563b1a9740 20 client.19014 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:35.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.220+0000 7f563b1a9740 1 --2- 172.21.15.70:0/3762281832 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55abb37e3c50 0x55abb37e6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.220+0000 7f563b1a9740 4 client.19014 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:45:35.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.220+0000 7f563b1a9740 1 -- 172.21.15.70:0/3762281832 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55abb3728730 con 0x55abb37e3c50 2024-04-01T01:45:35.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.221+0000 7f5635a88640 1 --2- 172.21.15.70:0/3762281832 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55abb37e3c50 0x55abb37e6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.222+0000 7f5635a88640 1 --2- 172.21.15.70:0/3762281832 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55abb37e3c50 0x55abb37e6080 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf52415b653b8863 server_cookie=1f509949cbbd8f2a in_seq=0 out_seq=0 2024-04-01T01:45:35.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.222+0000 7f5615ffb640 10 client.19014 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:35.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.223+0000 7f5615ffb640 1 -- 172.21.15.70:0/3762281832 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55abb3728730 con 0x55abb37e3c50 2024-04-01T01:45:35.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.223+0000 7f5615ffb640 10 client.19014 handle_command_reply: tid=0 2024-04-01T01:45:35.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.223+0000 7f563b1a9740 1 client.19014 shutdown 2024-04-01T01:45:35.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.224+0000 7f563b1a9740 1 -- 172.21.15.70:0/3762281832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37bd5e0 msgr2=0x55abb37d7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.224+0000 7f563b1a9740 1 --2- 172.21.15.70:0/3762281832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37bd5e0 0x55abb37d7f40 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5620009c00 tx=0x7f5620018e70 comp rx=0 tx=0).stop 2024-04-01T01:45:35.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.224+0000 7f563b1a9740 1 -- 172.21.15.70:0/3762281832 shutdown_connections 2024-04-01T01:45:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.224+0000 7f563b1a9740 1 --2- 172.21.15.70:0/3762281832 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55abb37e3c50 0x55abb37e6080 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.225+0000 7f5615ffb640 0 client.19014 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.224+0000 7f563b1a9740 1 --2- 172.21.15.70:0/3762281832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37dd4b0 0x55abb37df8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.225+0000 7f563b1a9740 1 --2- 172.21.15.70:0/3762281832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37c2750 0x55abb37d8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.225+0000 7f563b1a9740 1 --2- 172.21.15.70:0/3762281832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37bd5e0 0x55abb37d7f40 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.225+0000 7f563b1a9740 1 -- 172.21.15.70:0/3762281832 >> 172.21.15.70:0/3762281832 conn(0x55abb37b8160 msgr2=0x55abb37bce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.225+0000 7f563b1a9740 1 -- 172.21.15.70:0/3762281832 shutdown_connections 2024-04-01T01:45:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.225+0000 7f563b1a9740 1 -- 172.21.15.70:0/3762281832 wait complete. 2024-04-01T01:45:35.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.225+0000 7f563b1a9740 20 client.19014 trim_cache size 0 max 16384 2024-04-01T01:45:35.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.256+0000 7f563b1a9740 1 Processor -- start 2024-04-01T01:45:35.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.256+0000 7f563b1a9740 1 -- start start 2024-04-01T01:45:35.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.256+0000 7f563b1a9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37bd5e0 0x55abb37eb950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.256+0000 7f563b1a9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37c2750 0x55abb37bb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.256+0000 7f563b1a9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37dd4b0 0x55abb37bb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.256+0000 7f563b1a9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f56301b05a0 con 0x55abb37c2750 2024-04-01T01:45:35.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.256+0000 7f563b1a9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f56301b20e0 con 0x55abb37bd5e0 2024-04-01T01:45:35.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.256+0000 7f563b1a9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f56301b0c70 con 0x55abb37dd4b0 2024-04-01T01:45:35.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.256+0000 7f5635a88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37c2750 0x55abb37bb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.257+0000 7f5635a88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37c2750 0x55abb37bb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36338/0 (socket says 172.21.15.70:36338) 2024-04-01T01:45:35.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.257+0000 7f5635a88640 1 -- 172.21.15.70:0/2889802670 learned_addr learned my addr 172.21.15.70:0/2889802670 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:35.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.257+0000 7f5636a8a640 1 --2- 172.21.15.70:0/2889802670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37bd5e0 0x55abb37eb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.257+0000 7f5636a8a640 1 --2- 172.21.15.70:0/2889802670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37dd4b0 0x55abb37bb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.257+0000 7f5635a88640 1 -- 172.21.15.70:0/2889802670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37dd4b0 msgr2=0x55abb37bb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:35.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.257+0000 7f5635a88640 1 --2- 172.21.15.70:0/2889802670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37dd4b0 0x55abb37bb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.258+0000 7f5635a88640 1 -- 172.21.15.70:0/2889802670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37bd5e0 msgr2=0x55abb37eb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:35.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.258+0000 7f5635a88640 1 --2- 172.21.15.70:0/2889802670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37bd5e0 0x55abb37eb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.258+0000 7f5635a88640 1 -- 172.21.15.70:0/2889802670 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abb37bc830 con 0x55abb37c2750 2024-04-01T01:45:35.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.258+0000 7f5636a8a640 1 --2- 172.21.15.70:0/2889802670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37dd4b0 0x55abb37bb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:45:35.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.259+0000 7f5635a88640 1 --2- 172.21.15.70:0/2889802670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37c2750 0x55abb37bb3c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f5624000c00 tx=0x7f56240097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89385441fd687960 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:35.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.259+0000 7f5617fff640 1 -- 172.21.15.70:0/2889802670 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624024070 con 0x55abb37c2750 2024-04-01T01:45:35.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.259+0000 7f5617fff640 1 -- 172.21.15.70:0/2889802670 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f562401ec50 con 0x55abb37c2750 2024-04-01T01:45:35.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.259+0000 7f5617fff640 1 -- 172.21.15.70:0/2889802670 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624003850 con 0x55abb37c2750 2024-04-01T01:45:35.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.260+0000 7f563b1a9740 1 -- 172.21.15.70:0/2889802670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37c2750 msgr2=0x55abb37bb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.260+0000 7f563b1a9740 1 --2- 172.21.15.70:0/2889802670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37c2750 0x55abb37bb3c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f5624000c00 tx=0x7f56240097b0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.261+0000 7f563b1a9740 1 -- 172.21.15.70:0/2889802670 shutdown_connections 2024-04-01T01:45:35.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.261+0000 7f563b1a9740 1 --2- 172.21.15.70:0/2889802670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37dd4b0 0x55abb37bb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.261+0000 7f563b1a9740 1 --2- 172.21.15.70:0/2889802670 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37c2750 0x55abb37bb3c0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.261+0000 7f563b1a9740 1 --2- 172.21.15.70:0/2889802670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37bd5e0 0x55abb37eb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.261+0000 7f563b1a9740 1 -- 172.21.15.70:0/2889802670 >> 172.21.15.70:0/2889802670 conn(0x55abb37b8160 msgr2=0x55abb37b8540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:35.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.261+0000 7f563b1a9740 1 -- 172.21.15.70:0/2889802670 shutdown_connections 2024-04-01T01:45:35.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.261+0000 7f563b1a9740 1 -- 172.21.15.70:0/2889802670 wait complete. 2024-04-01T01:45:35.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.262+0000 7f563b1a9740 1 -- start start 2024-04-01T01:45:35.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.262+0000 7f563b1a9740 1 Processor -- start 2024-04-01T01:45:35.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.262+0000 7f563b1a9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37bd5e0 0x55abb37f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.262+0000 7f563b1a9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37c2750 0x55abb37f7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.263+0000 7f563b1a9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37dd4b0 0x55abb37f5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.263+0000 7f563b1a9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f56301b1a10 con 0x55abb37c2750 2024-04-01T01:45:35.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.263+0000 7f563b1a9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f56301b1340 con 0x55abb37dd4b0 2024-04-01T01:45:35.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.263+0000 7f563b1a9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5630073440 con 0x55abb37bd5e0 2024-04-01T01:45:35.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.263+0000 7f5636a8a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37bd5e0 0x55abb37f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.263+0000 7f5636a8a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37bd5e0 0x55abb37f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58594/0 (socket says 172.21.15.70:58594) 2024-04-01T01:45:35.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.263+0000 7f5636a8a640 1 -- 172.21.15.70:0/2795531711 learned_addr learned my addr 172.21.15.70:0/2795531711 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:35.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.263+0000 7f563b1a9740 10 client.0 _ll_register_callbacks cb 0x7f56200175c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:35.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.263+0000 7f5635a88640 1 --2- 172.21.15.70:0/2795531711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37c2750 0x55abb37f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.263+0000 7f5636a8a640 1 --2- 172.21.15.70:0/2795531711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37dd4b0 0x55abb37f5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.264+0000 7f5636a8a640 1 -- 172.21.15.70:0/2795531711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37dd4b0 msgr2=0x55abb37f5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:35.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.264+0000 7f5636a8a640 1 --2- 172.21.15.70:0/2795531711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37dd4b0 0x55abb37f5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.264+0000 7f5636a8a640 1 -- 172.21.15.70:0/2795531711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37c2750 msgr2=0x55abb37f7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:35.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.264+0000 7f5636a8a640 1 --2- 172.21.15.70:0/2795531711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37c2750 0x55abb37f7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.264+0000 7f5636a8a640 1 -- 172.21.15.70:0/2795531711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abb37f5cb0 con 0x55abb37bd5e0 2024-04-01T01:45:35.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.264+0000 7f5636a8a640 1 --2- 172.21.15.70:0/2795531711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37bd5e0 0x55abb37f6c10 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5620009620 tx=0x7f5620009d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87797a185299e971 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:35.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.264+0000 7f5615ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:45:35.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.267+0000 7f5615ffb640 1 -- 172.21.15.70:0/2795531711 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5620005750 con 0x55abb37bd5e0 2024-04-01T01:45:35.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.267+0000 7f5615ffb640 1 -- 172.21.15.70:0/2795531711 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f562002c040 con 0x55abb37bd5e0 2024-04-01T01:45:35.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.268+0000 7f563b1a9740 1 -- 172.21.15.70:0/2795531711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f562401ec50 con 0x55abb37bd5e0 2024-04-01T01:45:35.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.271+0000 7f5615ffb640 1 -- 172.21.15.70:0/2795531711 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5620003970 con 0x55abb37bd5e0 2024-04-01T01:45:35.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.273+0000 7f563b1a9740 10 client.28870 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:35.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.273+0000 7f563b1a9740 1 -- 172.21.15.70:0/2795531711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55abb384a590 con 0x55abb37bd5e0 2024-04-01T01:45:35.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.277+0000 7f5615ffb640 1 -- 172.21.15.70:0/2795531711 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f562001c050 con 0x55abb37bd5e0 2024-04-01T01:45:35.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.278+0000 7f5615ffb640 1 -- 172.21.15.70:0/2795531711 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f562000b040 con 0x55abb37bd5e0 2024-04-01T01:45:35.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.278+0000 7f563b1a9740 10 client.28870 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:35.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.278+0000 7f563b1a9740 10 client.28870 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:35.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.278+0000 7f563b1a9740 20 client.28870 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:35.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.278+0000 7f563b1a9740 1 --2- 172.21.15.70:0/2795531711 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55abb37e3c50 0x55abb384cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.278+0000 7f563b1a9740 4 client.28870 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:45:35.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.278+0000 7f563b1a9740 1 -- 172.21.15.70:0/2795531711 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f562401ec50 con 0x55abb37e3c50 2024-04-01T01:45:35.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.280+0000 7f5635a88640 1 --2- 172.21.15.70:0/2795531711 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55abb37e3c50 0x55abb384cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.289+0000 7f5635a88640 1 --2- 172.21.15.70:0/2795531711 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55abb37e3c50 0x55abb384cf50 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d71b0e6b064be39 server_cookie=4e3ae0a0bdf866bc in_seq=0 out_seq=0 2024-04-01T01:45:35.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.289+0000 7f5615ffb640 10 client.28870 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:35.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.295+0000 7f5615ffb640 1 -- 172.21.15.70:0/2795531711 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f562401ec50 con 0x55abb37e3c50 2024-04-01T01:45:35.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.295+0000 7f5615ffb640 10 client.28870 handle_command_reply: tid=0 2024-04-01T01:45:35.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.295+0000 7f563b1a9740 1 client.28870 shutdown 2024-04-01T01:45:35.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.296+0000 7f563b1a9740 1 -- 172.21.15.70:0/2795531711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37bd5e0 msgr2=0x55abb37f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.296+0000 7f563b1a9740 1 --2- 172.21.15.70:0/2795531711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37bd5e0 0x55abb37f6c10 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5620009620 tx=0x7f5620009d50 comp rx=0 tx=0).stop 2024-04-01T01:45:35.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.299+0000 7f563b1a9740 1 -- 172.21.15.70:0/2795531711 shutdown_connections 2024-04-01T01:45:35.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.299+0000 7f563b1a9740 1 --2- 172.21.15.70:0/2795531711 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55abb37e3c50 0x55abb384cf50 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.300+0000 7f5615ffb640 0 client.28870 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:35.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.300+0000 7f563b1a9740 1 --2- 172.21.15.70:0/2795531711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55abb37dd4b0 0x55abb37f5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.300+0000 7f563b1a9740 1 --2- 172.21.15.70:0/2795531711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55abb37c2750 0x55abb37f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.300+0000 7f563b1a9740 1 --2- 172.21.15.70:0/2795531711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55abb37bd5e0 0x55abb37f6c10 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.300+0000 7f563b1a9740 1 -- 172.21.15.70:0/2795531711 >> 172.21.15.70:0/2795531711 conn(0x55abb37b8160 msgr2=0x55abb37bb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:35.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.302+0000 7f563b1a9740 1 -- 172.21.15.70:0/2795531711 shutdown_connections 2024-04-01T01:45:35.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.302+0000 7f563b1a9740 1 -- 172.21.15.70:0/2795531711 wait complete. 2024-04-01T01:45:35.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.302+0000 7f563b1a9740 20 client.28870 trim_cache size 0 max 16384 2024-04-01T01:45:35.303 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:45:35.303 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:45:35.303 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "86d76efc-7d58-41e5-856a-d48aec2aa2e9", 2024-04-01T01:45:35.303 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:45:35.303 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:45:35.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.303+0000 7f563728b640 1 -- 172.21.15.70:0/3288748433 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56180a26a0 msgr2=0x7f56180a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.303+0000 7f563728b640 1 --2- 172.21.15.70:0/3288748433 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56180a26a0 0x7f56180a4b60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f56280b94f0 tx=0x7f5628003aa0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.303+0000 7f563728b640 1 -- 172.21.15.70:0/3288748433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5630107c70 msgr2=0x7f563007eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.303+0000 7f563728b640 1 --2- 172.21.15.70:0/3288748433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5630107c70 0x7f563007eed0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f5628009780 tx=0x7f5628035bc0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.303+0000 7f563728b640 1 -- 172.21.15.70:0/3288748433 shutdown_connections 2024-04-01T01:45:35.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.303+0000 7f563728b640 1 --2- 172.21.15.70:0/3288748433 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56180a26a0 0x7f56180a4b60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.303+0000 7f563728b640 1 --2- 172.21.15.70:0/3288748433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5630109820 0x7f56301aded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.303+0000 7f563728b640 1 --2- 172.21.15.70:0/3288748433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5630108e60 0x7f563007f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.303+0000 7f563728b640 1 --2- 172.21.15.70:0/3288748433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5630107c70 0x7f563007eed0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.303+0000 7f563728b640 1 -- 172.21.15.70:0/3288748433 >> 172.21.15.70:0/3288748433 conn(0x7f5630076c20 msgr2=0x7f563010c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:35.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.303+0000 7f563728b640 1 -- 172.21.15.70:0/3288748433 shutdown_connections 2024-04-01T01:45:35.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.303+0000 7f563728b640 1 -- 172.21.15.70:0/3288748433 wait complete. 2024-04-01T01:45:35.316 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:45:35.316 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:45:35.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.469+0000 7fa9cc455640 1 -- 172.21.15.70:0/1608410992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9c4100870 msgr2=0x7fa9c4100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.469+0000 7fa9cc455640 1 --2- 172.21.15.70:0/1608410992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9c4100870 0x7fa9c4100cf0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4009900 tx=0x7fa9b402bc30 comp rx=0 tx=0).stop 2024-04-01T01:45:35.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.469+0000 7fa9cc455640 1 -- 172.21.15.70:0/1608410992 shutdown_connections 2024-04-01T01:45:35.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.469+0000 7fa9cc455640 1 --2- 172.21.15.70:0/1608410992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9c4101230 0x7fa9c4108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.469+0000 7fa9cc455640 1 --2- 172.21.15.70:0/1608410992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9c4100870 0x7fa9c4100cf0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.469+0000 7fa9cc455640 1 --2- 172.21.15.70:0/1608410992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c4101c20 0x7fa9c4102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.469+0000 7fa9cc455640 1 -- 172.21.15.70:0/1608410992 >> 172.21.15.70:0/1608410992 conn(0x7fa9c40fb340 msgr2=0x7fa9c40fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:35.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.470+0000 7fa9cc455640 1 -- 172.21.15.70:0/1608410992 shutdown_connections 2024-04-01T01:45:35.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.470+0000 7fa9cc455640 1 -- 172.21.15.70:0/1608410992 wait complete. 2024-04-01T01:45:35.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.470+0000 7fa9cc455640 1 Processor -- start 2024-04-01T01:45:35.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9cc455640 1 -- start start 2024-04-01T01:45:35.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9cc455640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9c4100870 0x7fa9c41002d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9cc455640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c4101230 0x7fa9c419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9ca1f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9c4100870 0x7fa9c41002d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9ca1f1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9c4100870 0x7fa9c41002d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36366/0 (socket says 172.21.15.70:36366) 2024-04-01T01:45:35.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9cc455640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9c4101c20 0x7fa9c419ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9cc455640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c410c190 con 0x7fa9c4100870 2024-04-01T01:45:35.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9cc455640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c410c010 con 0x7fa9c4101c20 2024-04-01T01:45:35.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9cc455640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c410c310 con 0x7fa9c4101230 2024-04-01T01:45:35.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9ca1f1640 1 -- 172.21.15.70:0/1165988607 learned_addr learned my addr 172.21.15.70:0/1165988607 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:35.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9c99f0640 1 --2- 172.21.15.70:0/1165988607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c4101230 0x7fa9c419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9ca9f2640 1 --2- 172.21.15.70:0/1165988607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9c4101c20 0x7fa9c419ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9ca1f1640 1 -- 172.21.15.70:0/1165988607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c4101230 msgr2=0x7fa9c419c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9ca1f1640 1 --2- 172.21.15.70:0/1165988607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c4101230 0x7fa9c419c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9ca1f1640 1 -- 172.21.15.70:0/1165988607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9c4101c20 msgr2=0x7fa9c419ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9ca1f1640 1 --2- 172.21.15.70:0/1165988607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9c4101c20 0x7fa9c419ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.471+0000 7fa9ca1f1640 1 -- 172.21.15.70:0/1165988607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9c419f640 con 0x7fa9c4100870 2024-04-01T01:45:35.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.472+0000 7fa9ca1f1640 1 --2- 172.21.15.70:0/1165988607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9c4100870 0x7fa9c41002d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fa9c000e9c0 tx=0x7fa9c000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:35.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.472+0000 7fa9bb7fe640 1 -- 172.21.15.70:0/1165988607 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c000cd80 con 0x7fa9c4100870 2024-04-01T01:45:35.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.472+0000 7fa9bb7fe640 1 -- 172.21.15.70:0/1165988607 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9c00035f0 con 0x7fa9c4100870 2024-04-01T01:45:35.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.472+0000 7fa9bb7fe640 1 -- 172.21.15.70:0/1165988607 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c0010690 con 0x7fa9c4100870 2024-04-01T01:45:35.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.472+0000 7fa9cc455640 1 -- 172.21.15.70:0/1165988607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9c419f8d0 con 0x7fa9c4100870 2024-04-01T01:45:35.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.473+0000 7fa9cc455640 1 -- 172.21.15.70:0/1165988607 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9c419fda0 con 0x7fa9c4100870 2024-04-01T01:45:35.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.476+0000 7fa9bb7fe640 1 -- 172.21.15.70:0/1165988607 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fa9c00037a0 con 0x7fa9c4100870 2024-04-01T01:45:35.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.477+0000 7fa9cf036740 1 Processor -- start 2024-04-01T01:45:35.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.477+0000 7fa9cf036740 1 -- start start 2024-04-01T01:45:35.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.477+0000 7fa9bb7fe640 1 --2- 172.21.15.70:0/1165988607 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa9b00a25a0 0x7fa9b00a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.477+0000 7fa9cf036740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c450d0 0x563a63c454f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.477+0000 7fa9cf036740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c45ac0 0x563a63c4a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.477+0000 7fa9cf036740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 0x563a63c4cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.477+0000 7fa9cf036740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563a63ba1180 con 0x563a63c4a680 2024-04-01T01:45:35.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.477+0000 7fa9cf036740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563a63b49b70 con 0x563a63c45ac0 2024-04-01T01:45:35.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.477+0000 7fa9cf036740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563a63b9d220 con 0x563a63c450d0 2024-04-01T01:45:35.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.477+0000 7fa9bb7fe640 1 -- 172.21.15.70:0/1165988607 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa9c00ceec0 con 0x7fa9c4100870 2024-04-01T01:45:35.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.477+0000 7fa9c99f0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c45ac0 0x563a63c4a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.477+0000 7fa9c99f0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c45ac0 0x563a63c4a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41356/0 (socket says 172.21.15.70:41356) 2024-04-01T01:45:35.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.477+0000 7fa9c99f0640 1 -- 172.21.15.70:0/3168510284 learned_addr learned my addr 172.21.15.70:0/3168510284 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:35.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.477+0000 7fa9ca1f1640 1 --2- 172.21.15.70:0/3168510284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c450d0 0x563a63c454f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.477+0000 7fa9ca9f2640 1 --2- 172.21.15.70:0/1165988607 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa9b00a25a0 0x7fa9b00a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.478+0000 7fa9ca9f2640 1 --2- 172.21.15.70:0/3168510284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 0x563a63c4cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.478+0000 7fa9c99f0640 1 -- 172.21.15.70:0/3168510284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c450d0 msgr2=0x563a63c454f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:35.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.478+0000 7fa9c99f0640 1 --2- 172.21.15.70:0/3168510284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c450d0 0x563a63c454f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.479+0000 7fa9c99f0640 1 -- 172.21.15.70:0/3168510284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 msgr2=0x563a63c4cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:35.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.479+0000 7fa9c99f0640 1 --2- 172.21.15.70:0/3168510284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 0x563a63c4cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.479+0000 7fa9c99f0640 1 -- 172.21.15.70:0/3168510284 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a63b5d7c0 con 0x563a63c45ac0 2024-04-01T01:45:35.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.479+0000 7fa9ca9f2640 1 --2- 172.21.15.70:0/3168510284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 0x563a63c4cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:45:35.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.479+0000 7fa9c99f0640 1 --2- 172.21.15.70:0/3168510284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c45ac0 0x563a63c4a010 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4009900 tx=0x7fa9b4009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edc5130613b757c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:35.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.479+0000 7fa9b8bfe640 1 -- 172.21.15.70:0/3168510284 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4045020 con 0x563a63c45ac0 2024-04-01T01:45:35.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.479+0000 7fa9b8bfe640 1 -- 172.21.15.70:0/3168510284 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9b4004040 con 0x563a63c45ac0 2024-04-01T01:45:35.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.479+0000 7fa9b8bfe640 1 -- 172.21.15.70:0/3168510284 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b40055e0 con 0x563a63c45ac0 2024-04-01T01:45:35.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.480+0000 7fa9cf036740 1 -- 172.21.15.70:0/3168510284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c45ac0 msgr2=0x563a63c4a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.480+0000 7fa9cf036740 1 --2- 172.21.15.70:0/3168510284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c45ac0 0x563a63c4a010 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4009900 tx=0x7fa9b4009560 comp rx=0 tx=0).stop 2024-04-01T01:45:35.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.480+0000 7fa9ca9f2640 1 --2- 172.21.15.70:0/1165988607 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa9b00a25a0 0x7fa9b00a4a60 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc005fd0 tx=0x7fa9bc009bf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:35.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.480+0000 7fa9cf036740 1 -- 172.21.15.70:0/3168510284 shutdown_connections 2024-04-01T01:45:35.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.480+0000 7fa9cf036740 1 --2- 172.21.15.70:0/3168510284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 0x563a63c4cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.480+0000 7fa9cf036740 1 --2- 172.21.15.70:0/3168510284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c45ac0 0x563a63c4a010 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.480+0000 7fa9cf036740 1 --2- 172.21.15.70:0/3168510284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c450d0 0x563a63c454f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.480+0000 7fa9cf036740 1 -- 172.21.15.70:0/3168510284 >> 172.21.15.70:0/3168510284 conn(0x563a63c377a0 msgr2=0x563a63c37ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:35.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.481+0000 7fa9cf036740 1 -- 172.21.15.70:0/3168510284 shutdown_connections 2024-04-01T01:45:35.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.481+0000 7fa9cf036740 1 -- 172.21.15.70:0/3168510284 wait complete. 2024-04-01T01:45:35.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.481+0000 7fa9cf036740 1 -- start start 2024-04-01T01:45:35.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.481+0000 7fa9cf036740 1 Processor -- start 2024-04-01T01:45:35.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.482+0000 7fa9cf036740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c450d0 0x563a63c4ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.482+0000 7fa9cf036740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c45ac0 0x563a63c4f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.482+0000 7fa9cf036740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c4a680 0x563a63c562e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.482+0000 7fa9cf036740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563a63c4d5d0 con 0x563a63c45ac0 2024-04-01T01:45:35.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.482+0000 7fa9cf036740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563a63b3f4a0 con 0x563a63c450d0 2024-04-01T01:45:35.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.482+0000 7fa9cf036740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563a63c4d750 con 0x563a63c4a680 2024-04-01T01:45:35.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.482+0000 7fa9cf036740 10 client.0 _ll_register_callbacks cb 0x563a63b42d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:35.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.482+0000 7fa9c99f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c45ac0 0x563a63c4f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.482+0000 7fa9ca1f1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c450d0 0x563a63c4ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.482+0000 7fa9c99f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c45ac0 0x563a63c4f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36386/0 (socket says 172.21.15.70:36386) 2024-04-01T01:45:35.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.482+0000 7fa9c99f0640 1 -- 172.21.15.70:0/258612896 learned_addr learned my addr 172.21.15.70:0/258612896 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:35.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.482+0000 7fa9ca9f2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c4a680 0x563a63c562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.482+0000 7fa9ca9f2640 1 -- 172.21.15.70:0/258612896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c450d0 msgr2=0x563a63c4ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:35.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.482+0000 7fa9ca9f2640 1 --2- 172.21.15.70:0/258612896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c450d0 0x563a63c4ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.482+0000 7fa9ca9f2640 1 -- 172.21.15.70:0/258612896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c45ac0 msgr2=0x563a63c4f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:35.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.482+0000 7fa9ca9f2640 1 --2- 172.21.15.70:0/258612896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c45ac0 0x563a63c4f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.483+0000 7fa9ca9f2640 1 -- 172.21.15.70:0/258612896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a6397f7d0 con 0x563a63c4a680 2024-04-01T01:45:35.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.483+0000 7fa9ca9f2640 1 --2- 172.21.15.70:0/258612896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c4a680 0x563a63c562e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00ff90 tx=0x7fa9bc004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb1e08981517a4c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:35.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.483+0000 7fa9adffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:45:35.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.483+0000 7fa9adffb640 1 -- 172.21.15.70:0/258612896 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc01fe60 con 0x563a63c4a680 2024-04-01T01:45:35.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.483+0000 7fa9adffb640 1 -- 172.21.15.70:0/258612896 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa9bc004b20 con 0x563a63c4a680 2024-04-01T01:45:35.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.483+0000 7fa9cf036740 1 -- 172.21.15.70:0/258612896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9b4004040 con 0x563a63c4a680 2024-04-01T01:45:35.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.484+0000 7fa9adffb640 1 -- 172.21.15.70:0/258612896 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa9bc004d20 con 0x563a63c4a680 2024-04-01T01:45:35.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.484+0000 7fa9cf036740 10 client.28882 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:35.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.484+0000 7fa9cf036740 1 -- 172.21.15.70:0/258612896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563a638b53f0 con 0x563a63c4a680 2024-04-01T01:45:35.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.484+0000 7fa9adffb640 1 -- 172.21.15.70:0/258612896 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9bc01dac0 con 0x563a63c4a680 2024-04-01T01:45:35.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.485+0000 7fa9adffb640 1 -- 172.21.15.70:0/258612896 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fa9bc005510 con 0x563a63c4a680 2024-04-01T01:45:35.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.485+0000 7fa9cf036740 10 client.28882 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:35.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.485+0000 7fa9cf036740 10 client.28882 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:35.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.485+0000 7fa9cf036740 20 client.28882 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:35.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.485+0000 7fa9cf036740 1 --2- 172.21.15.70:0/258612896 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563a63c5adf0 0x563a63c5d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.485+0000 7fa9cf036740 4 client.28882 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:45:35.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.485+0000 7fa9cf036740 1 -- 172.21.15.70:0/258612896 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa9b4004040 con 0x563a63c5adf0 2024-04-01T01:45:35.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.485+0000 7fa9ca1f1640 1 --2- 172.21.15.70:0/258612896 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563a63c5adf0 0x563a63c5d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.486+0000 7fa9ca1f1640 1 --2- 172.21.15.70:0/258612896 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563a63c5adf0 0x563a63c5d210 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29f46aab333d04c3 server_cookie=a34266082b6a4af7 in_seq=0 out_seq=0 2024-04-01T01:45:35.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.486+0000 7fa9adffb640 10 client.28882 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:35.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.487+0000 7fa9adffb640 1 -- 172.21.15.70:0/258612896 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fa9b4004040 con 0x563a63c5adf0 2024-04-01T01:45:35.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.487+0000 7fa9adffb640 10 client.28882 handle_command_reply: tid=0 2024-04-01T01:45:35.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.487+0000 7fa9cf036740 1 client.28882 shutdown 2024-04-01T01:45:35.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.488+0000 7fa9cf036740 1 -- 172.21.15.70:0/258612896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c4a680 msgr2=0x563a63c562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.488+0000 7fa9cf036740 1 --2- 172.21.15.70:0/258612896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c4a680 0x563a63c562e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00ff90 tx=0x7fa9bc004ab0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.488+0000 7fa9cf036740 1 -- 172.21.15.70:0/258612896 shutdown_connections 2024-04-01T01:45:35.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.488+0000 7fa9cf036740 1 --2- 172.21.15.70:0/258612896 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563a63c5adf0 0x563a63c5d210 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.488+0000 7fa9adffb640 0 client.28882 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:35.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.488+0000 7fa9cf036740 1 --2- 172.21.15.70:0/258612896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c4a680 0x563a63c562e0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.488+0000 7fa9cf036740 1 --2- 172.21.15.70:0/258612896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c45ac0 0x563a63c4f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.488+0000 7fa9cf036740 1 --2- 172.21.15.70:0/258612896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c450d0 0x563a63c4ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.488+0000 7fa9cf036740 1 -- 172.21.15.70:0/258612896 >> 172.21.15.70:0/258612896 conn(0x563a63c377a0 msgr2=0x563a63c43ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:35.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.488+0000 7fa9cf036740 1 -- 172.21.15.70:0/258612896 shutdown_connections 2024-04-01T01:45:35.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.488+0000 7fa9cf036740 1 -- 172.21.15.70:0/258612896 wait complete. 2024-04-01T01:45:35.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.489+0000 7fa9cf036740 20 client.28882 trim_cache size 0 max 16384 2024-04-01T01:45:35.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.521+0000 7fa9cf036740 1 Processor -- start 2024-04-01T01:45:35.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.521+0000 7fa9cf036740 1 -- start start 2024-04-01T01:45:35.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.522+0000 7fa9cf036740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c450d0 0x563a63c36390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.522+0000 7fa9cf036740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c45ac0 0x563a63c562a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.522+0000 7fa9cf036740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 0x563a63c368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.522+0000 7fa9cf036740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c41a8160 con 0x563a63c4a680 2024-04-01T01:45:35.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.522+0000 7fa9cf036740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c41a9ca0 con 0x563a63c450d0 2024-04-01T01:45:35.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.522+0000 7fa9cf036740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c41a8830 con 0x563a63c45ac0 2024-04-01T01:45:35.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.522+0000 7fa9ca9f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 0x563a63c368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.522+0000 7fa9ca9f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 0x563a63c368d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36392/0 (socket says 172.21.15.70:36392) 2024-04-01T01:45:35.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.522+0000 7fa9ca9f2640 1 -- 172.21.15.70:0/1729632122 learned_addr learned my addr 172.21.15.70:0/1729632122 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:35.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.522+0000 7fa9ca1f1640 1 --2- 172.21.15.70:0/1729632122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c450d0 0x563a63c36390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.523+0000 7fa9c99f0640 1 --2- 172.21.15.70:0/1729632122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c45ac0 0x563a63c562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.523+0000 7fa9ca9f2640 1 -- 172.21.15.70:0/1729632122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c45ac0 msgr2=0x563a63c562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:35.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.523+0000 7fa9ca9f2640 1 --2- 172.21.15.70:0/1729632122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c45ac0 0x563a63c562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.523+0000 7fa9ca9f2640 1 -- 172.21.15.70:0/1729632122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c450d0 msgr2=0x563a63c36390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:35.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.523+0000 7fa9ca9f2640 1 --2- 172.21.15.70:0/1729632122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c450d0 0x563a63c36390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.523+0000 7fa9ca9f2640 1 -- 172.21.15.70:0/1729632122 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a63c5d580 con 0x563a63c4a680 2024-04-01T01:45:35.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.523+0000 7fa9ca9f2640 1 --2- 172.21.15.70:0/1729632122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 0x563a63c368d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc011680 tx=0x7fa9bc0117f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef3df1b1dabacfb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:35.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.524+0000 7fa9affff640 1 -- 172.21.15.70:0/1729632122 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc01fe60 con 0x563a63c4a680 2024-04-01T01:45:35.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.524+0000 7fa9affff640 1 -- 172.21.15.70:0/1729632122 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9bc0296e0 con 0x563a63c4a680 2024-04-01T01:45:35.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.524+0000 7fa9affff640 1 -- 172.21.15.70:0/1729632122 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc01db20 con 0x563a63c4a680 2024-04-01T01:45:35.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.525+0000 7fa9cf036740 1 -- 172.21.15.70:0/1729632122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 msgr2=0x563a63c368d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.525+0000 7fa9cf036740 1 --2- 172.21.15.70:0/1729632122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 0x563a63c368d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc011680 tx=0x7fa9bc0117f0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.526+0000 7fa9cf036740 1 -- 172.21.15.70:0/1729632122 shutdown_connections 2024-04-01T01:45:35.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.526+0000 7fa9cf036740 1 --2- 172.21.15.70:0/1729632122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 0x563a63c368d0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.526+0000 7fa9cf036740 1 --2- 172.21.15.70:0/1729632122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c45ac0 0x563a63c562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.526+0000 7fa9cf036740 1 --2- 172.21.15.70:0/1729632122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c450d0 0x563a63c36390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.526+0000 7fa9cf036740 1 -- 172.21.15.70:0/1729632122 >> 172.21.15.70:0/1729632122 conn(0x563a63c377a0 msgr2=0x563a63c46810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:35.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.528+0000 7fa9cf036740 1 -- 172.21.15.70:0/1729632122 shutdown_connections 2024-04-01T01:45:35.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.532+0000 7fa9cf036740 1 -- 172.21.15.70:0/1729632122 wait complete. 2024-04-01T01:45:35.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.532+0000 7fa9cf036740 1 -- start start 2024-04-01T01:45:35.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.533+0000 7fa9cf036740 1 Processor -- start 2024-04-01T01:45:35.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.533+0000 7fa9cf036740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c450d0 0x563a63c53bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.533+0000 7fa9cf036740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c45ac0 0x563a63c54110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.533+0000 7fa9cf036740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 0x563a63c550b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.533+0000 7fa9c99f0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c45ac0 0x563a63c54110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.534+0000 7fa9c99f0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c45ac0 0x563a63c54110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58662/0 (socket says 172.21.15.70:58662) 2024-04-01T01:45:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.534+0000 7fa9c99f0640 1 -- 172.21.15.70:0/2816378368 learned_addr learned my addr 172.21.15.70:0/2816378368 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.534+0000 7fa9ca1f1640 1 --2- 172.21.15.70:0/2816378368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c450d0 0x563a63c53bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.534+0000 7fa9cf036740 1 -- 172.21.15.70:0/2816378368 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c41a95d0 con 0x563a63c4a680 2024-04-01T01:45:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.534+0000 7fa9ca9f2640 1 --2- 172.21.15.70:0/2816378368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 0x563a63c550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.534+0000 7fa9cf036740 1 -- 172.21.15.70:0/2816378368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c41a8f00 con 0x563a63c450d0 2024-04-01T01:45:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.534+0000 7fa9cf036740 1 -- 172.21.15.70:0/2816378368 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c41a6140 con 0x563a63c45ac0 2024-04-01T01:45:35.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.534+0000 7fa9cf036740 10 client.0 _ll_register_callbacks cb 0x7fa9bc00fc40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.537+0000 7fa9ca9f2640 1 -- 172.21.15.70:0/2816378368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c45ac0 msgr2=0x563a63c54110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.537+0000 7fa9ca9f2640 1 --2- 172.21.15.70:0/2816378368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c45ac0 0x563a63c54110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.537+0000 7fa9ca9f2640 1 -- 172.21.15.70:0/2816378368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c450d0 msgr2=0x563a63c53bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:35.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.537+0000 7fa9ca9f2640 1 --2- 172.21.15.70:0/2816378368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c450d0 0x563a63c53bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.537+0000 7fa9ca9f2640 1 -- 172.21.15.70:0/2816378368 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a638750f0 con 0x563a63c4a680 2024-04-01T01:45:35.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.539+0000 7fa9ca9f2640 1 --2- 172.21.15.70:0/2816378368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 0x563a63c550b0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00f790 tx=0x7fa9bc01da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a4967cf897ae1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:35.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.540+0000 7fa9adffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:45:35.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.541+0000 7fa9adffb640 1 -- 172.21.15.70:0/2816378368 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc03d070 con 0x563a63c4a680 2024-04-01T01:45:35.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.541+0000 7fa9adffb640 1 -- 172.21.15.70:0/2816378368 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fa9bc004ec0 con 0x563a63c4a680 2024-04-01T01:45:35.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.543+0000 7fa9cf036740 1 -- 172.21.15.70:0/2816378368 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563a63bae6c0 con 0x563a63c4a680 2024-04-01T01:45:35.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.545+0000 7fa9adffb640 1 -- 172.21.15.70:0/2816378368 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa9bc01fe60 con 0x563a63c4a680 2024-04-01T01:45:35.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.546+0000 7fa9cf036740 10 client.19038 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:35.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.546+0000 7fa9cf036740 1 -- 172.21.15.70:0/2816378368 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563a63cc1150 con 0x563a63c4a680 2024-04-01T01:45:35.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.548+0000 7fa9adffb640 1 -- 172.21.15.70:0/2816378368 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa9bc005570 con 0x563a63c4a680 2024-04-01T01:45:35.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.548+0000 7fa9adffb640 1 -- 172.21.15.70:0/2816378368 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fa9bc04b640 con 0x563a63c4a680 2024-04-01T01:45:35.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.548+0000 7fa9cf036740 10 client.19038 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:35.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.548+0000 7fa9cf036740 10 client.19038 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:35.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.548+0000 7fa9cf036740 20 client.19038 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:35.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.548+0000 7fa9cf036740 1 --2- 172.21.15.70:0/2816378368 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563a63c5adf0 0x563a63cc3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:35.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.548+0000 7fa9cf036740 4 client.19038 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:45:35.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.548+0000 7fa9cf036740 1 -- 172.21.15.70:0/2816378368 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563a63bae6c0 con 0x563a63c5adf0 2024-04-01T01:45:35.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.548+0000 7fa9ca1f1640 1 --2- 172.21.15.70:0/2816378368 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563a63c5adf0 0x563a63cc3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:35.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.549+0000 7fa9ca1f1640 1 --2- 172.21.15.70:0/2816378368 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563a63c5adf0 0x563a63cc3aa0 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af546ef6e333396f server_cookie=adc8a21466f216e0 in_seq=0 out_seq=0 2024-04-01T01:45:35.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.550+0000 7fa9adffb640 10 client.19038 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:35.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.552+0000 7fa9adffb640 1 -- 172.21.15.70:0/2816378368 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x563a63bae6c0 con 0x563a63c5adf0 2024-04-01T01:45:35.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.552+0000 7fa9adffb640 10 client.19038 handle_command_reply: tid=0 2024-04-01T01:45:35.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.552+0000 7fa9cf036740 1 client.19038 shutdown 2024-04-01T01:45:35.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.552+0000 7fa9cf036740 1 -- 172.21.15.70:0/2816378368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 msgr2=0x563a63c550b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.552+0000 7fa9cf036740 1 --2- 172.21.15.70:0/2816378368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 0x563a63c550b0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00f790 tx=0x7fa9bc01da50 comp rx=0 tx=0).stop 2024-04-01T01:45:35.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.553+0000 7fa9cf036740 1 -- 172.21.15.70:0/2816378368 shutdown_connections 2024-04-01T01:45:35.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.553+0000 7fa9cf036740 1 --2- 172.21.15.70:0/2816378368 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563a63c5adf0 0x563a63cc3aa0 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.553+0000 7fa9cf036740 1 --2- 172.21.15.70:0/2816378368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a63c4a680 0x563a63c550b0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.553+0000 7fa9adffb640 0 client.19038 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:35.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.553+0000 7fa9cf036740 1 --2- 172.21.15.70:0/2816378368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563a63c45ac0 0x563a63c54110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.553+0000 7fa9cf036740 1 --2- 172.21.15.70:0/2816378368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563a63c450d0 0x563a63c53bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.553+0000 7fa9cf036740 1 -- 172.21.15.70:0/2816378368 >> 172.21.15.70:0/2816378368 conn(0x563a63c377a0 msgr2=0x563a63c4ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:35.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.554+0000 7fa9cf036740 1 -- 172.21.15.70:0/2816378368 shutdown_connections 2024-04-01T01:45:35.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.554+0000 7fa9cf036740 1 -- 172.21.15.70:0/2816378368 wait complete. 2024-04-01T01:45:35.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.555+0000 7fa9cf036740 20 client.19038 trim_cache size 0 max 16384 2024-04-01T01:45:35.558 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:45:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "scrub active (189 inodes in the stack)", 2024-04-01T01:45:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:45:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: "86d76efc-7d58-41e5-856a-d48aec2aa2e9": { 2024-04-01T01:45:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:45:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "86d76efc-7d58-41e5-856a-d48aec2aa2e9", 2024-04-01T01:45:35.559 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:45:35.560 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:45:35.560 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:45:35.560 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:45:35.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.559+0000 7fa9cc455640 1 -- 172.21.15.70:0/1165988607 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa9b00a25a0 msgr2=0x7fa9b00a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.559+0000 7fa9cc455640 1 --2- 172.21.15.70:0/1165988607 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa9b00a25a0 0x7fa9b00a4a60 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc005fd0 tx=0x7fa9bc009bf0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.559+0000 7fa9cc455640 1 -- 172.21.15.70:0/1165988607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9c4100870 msgr2=0x7fa9c41002d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:35.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.559+0000 7fa9cc455640 1 --2- 172.21.15.70:0/1165988607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9c4100870 0x7fa9c41002d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fa9c000e9c0 tx=0x7fa9c000ee80 comp rx=0 tx=0).stop 2024-04-01T01:45:35.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.561+0000 7fa9cc455640 1 -- 172.21.15.70:0/1165988607 shutdown_connections 2024-04-01T01:45:35.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.561+0000 7fa9cc455640 1 --2- 172.21.15.70:0/1165988607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9c4101c20 0x7fa9c419ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.561+0000 7fa9cc455640 1 --2- 172.21.15.70:0/1165988607 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa9b00a25a0 0x7fa9b00a4a60 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.561+0000 7fa9cc455640 1 --2- 172.21.15.70:0/1165988607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9c4101230 0x7fa9c419c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.561+0000 7fa9cc455640 1 --2- 172.21.15.70:0/1165988607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9c4100870 0x7fa9c41002d0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:35.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.561+0000 7fa9cc455640 1 -- 172.21.15.70:0/1165988607 >> 172.21.15.70:0/1165988607 conn(0x7fa9c40fb340 msgr2=0x7fa9c4104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:35.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.562+0000 7fa9cc455640 1 -- 172.21.15.70:0/1165988607 shutdown_connections 2024-04-01T01:45:35.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:45:35.563+0000 7fa9cc455640 1 -- 172.21.15.70:0/1165988607 wait complete. 2024-04-01T01:45:35.580 INFO:tasks.cephfs.filesystem:scrub status for tag:86d76efc-7d58-41e5-856a-d48aec2aa2e9 - {'path': '/', 'tag': '86d76efc-7d58-41e5-856a-d48aec2aa2e9', 'options': 'recursive,force'} 2024-04-01T01:45:36.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:35 smithi189 ceph-mon[35753]: pgmap v722: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 9.1 MiB/s wr, 741 op/s 2024-04-01T01:45:36.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:35 smithi070 ceph-mon[32179]: pgmap v722: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 9.1 MiB/s wr, 741 op/s 2024-04-01T01:45:36.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:35 smithi081 ceph-mon[35142]: pgmap v722: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 9.1 MiB/s wr, 741 op/s 2024-04-01T01:45:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.719+0000 7f615bfff640 1 -- 172.21.15.70:0/3875646160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c100cf0 msgr2=0x7f615c1030e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.719+0000 7f615bfff640 1 --2- 172.21.15.70:0/3875646160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c100cf0 0x7f615c1030e0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f614c009900 tx=0x7f614c02bc40 comp rx=0 tx=0).stop 2024-04-01T01:45:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.720+0000 7f615bfff640 1 -- 172.21.15.70:0/3875646160 shutdown_connections 2024-04-01T01:45:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.720+0000 7f615bfff640 1 --2- 172.21.15.70:0/3875646160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f615c106130 0x7f615c1085f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.720+0000 7f615bfff640 1 --2- 172.21.15.70:0/3875646160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f615c1036b0 0x7f615c105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.720+0000 7f615bfff640 1 --2- 172.21.15.70:0/3875646160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c100cf0 0x7f615c1030e0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.720+0000 7f615bfff640 1 -- 172.21.15.70:0/3875646160 >> 172.21.15.70:0/3875646160 conn(0x7f615c0fad60 msgr2=0x7f615c0fd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.724+0000 7f615bfff640 1 -- 172.21.15.70:0/3875646160 shutdown_connections 2024-04-01T01:45:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.726+0000 7f615bfff640 1 -- 172.21.15.70:0/3875646160 wait complete. 2024-04-01T01:45:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.727+0000 7f615bfff640 1 Processor -- start 2024-04-01T01:45:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.727+0000 7f615bfff640 1 -- start start 2024-04-01T01:45:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.727+0000 7f615bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f615c100cf0 0x7f615c19de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.727+0000 7f615bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c1036b0 0x7f615c19e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.727+0000 7f615bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f615c106130 0x7f615c1a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.727+0000 7f615bfff640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f615c10af00 con 0x7f615c1036b0 2024-04-01T01:45:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.727+0000 7f615bfff640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f615c10ad80 con 0x7f615c106130 2024-04-01T01:45:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.727+0000 7f615bfff640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f615c10b080 con 0x7f615c100cf0 2024-04-01T01:45:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.727+0000 7f615a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c1036b0 0x7f615c19e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.728+0000 7f615b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f615c106130 0x7f615c1a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.728+0000 7f615b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f615c106130 0x7f615c1a5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38622/0 (socket says 172.21.15.70:38622) 2024-04-01T01:45:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.728+0000 7f615b7fe640 1 -- 172.21.15.70:0/2376541708 learned_addr learned my addr 172.21.15.70:0/2376541708 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.728+0000 7f615affd640 1 --2- 172.21.15.70:0/2376541708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f615c100cf0 0x7f615c19de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.728+0000 7f615b7fe640 1 -- 172.21.15.70:0/2376541708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f615c100cf0 msgr2=0x7f615c19de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.728+0000 7f615b7fe640 1 --2- 172.21.15.70:0/2376541708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f615c100cf0 0x7f615c19de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.728+0000 7f615b7fe640 1 -- 172.21.15.70:0/2376541708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c1036b0 msgr2=0x7f615c19e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.728+0000 7f615b7fe640 1 --2- 172.21.15.70:0/2376541708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c1036b0 0x7f615c19e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.728+0000 7f615b7fe640 1 -- 172.21.15.70:0/2376541708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615c1a5ab0 con 0x7f615c106130 2024-04-01T01:45:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.728+0000 7f615b7fe640 1 --2- 172.21.15.70:0/2376541708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f615c106130 0x7f615c1a5440 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f615400da90 tx=0x7f615400df50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:36.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.729+0000 7f616083e640 1 -- 172.21.15.70:0/2376541708 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615401b020 con 0x7f615c106130 2024-04-01T01:45:36.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.729+0000 7f616083e640 1 -- 172.21.15.70:0/2376541708 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f615400bc30 con 0x7f615c106130 2024-04-01T01:45:36.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.729+0000 7f616083e640 1 -- 172.21.15.70:0/2376541708 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61540059d0 con 0x7f615c106130 2024-04-01T01:45:36.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.731+0000 7f615bfff640 1 -- 172.21.15.70:0/2376541708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f615c1a5da0 con 0x7f615c106130 2024-04-01T01:45:36.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.731+0000 7f615bfff640 1 -- 172.21.15.70:0/2376541708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f615c1a61b0 con 0x7f615c106130 2024-04-01T01:45:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.735+0000 7f6164425740 1 Processor -- start 2024-04-01T01:45:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.736+0000 7f6164425740 1 -- start start 2024-04-01T01:45:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.736+0000 7f6164425740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d805ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.736+0000 7f6164425740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 0x55e4d8063fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.736+0000 7f6164425740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 0x55e4d8066ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.736+0000 7f6164425740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e4d7fc28f0 con 0x55e4d8064640 2024-04-01T01:45:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.736+0000 7f6164425740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e4d7f6b8d0 con 0x55e4d805f4d0 2024-04-01T01:45:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.736+0000 7f6164425740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e4d7fbe990 con 0x55e4d805eae0 2024-04-01T01:45:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.736+0000 7f615a7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d805ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.736+0000 7f615a7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d805ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:48856/0 (socket says 172.21.15.70:48856) 2024-04-01T01:45:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.736+0000 7f615a7fc640 1 -- 172.21.15.70:0/3549401679 learned_addr learned my addr 172.21.15.70:0/3549401679 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.736+0000 7f615affd640 1 --2- 172.21.15.70:0/3549401679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 0x55e4d8063fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.736+0000 7f615a7fc640 1 --2- 172.21.15.70:0/3549401679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 0x55e4d8066ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.737+0000 7f615affd640 1 -- 172.21.15.70:0/3549401679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 msgr2=0x55e4d805ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.737+0000 7f615affd640 1 --2- 172.21.15.70:0/3549401679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d805ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.737+0000 7f615affd640 1 -- 172.21.15.70:0/3549401679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 msgr2=0x55e4d8066ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.737+0000 7f615affd640 1 --2- 172.21.15.70:0/3549401679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 0x55e4d8066ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.737+0000 7f615affd640 1 -- 172.21.15.70:0/3549401679 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4d7cd7210 con 0x55e4d805f4d0 2024-04-01T01:45:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.737+0000 7f615a7fc640 1 --2- 172.21.15.70:0/3549401679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 0x55e4d8066ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:45:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.737+0000 7f615affd640 1 --2- 172.21.15.70:0/3549401679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 0x55e4d8063fd0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f614c02c180 tx=0x7f614c005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3159c6b534e050e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.737+0000 7f616083e640 1 -- 172.21.15.70:0/2376541708 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f615401e070 con 0x7f615c106130 2024-04-01T01:45:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.738+0000 7f6141ffb640 1 -- 172.21.15.70:0/3549401679 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c043030 con 0x55e4d805f4d0 2024-04-01T01:45:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.738+0000 7f6141ffb640 1 -- 172.21.15.70:0/3549401679 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f614c004040 con 0x55e4d805f4d0 2024-04-01T01:45:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.738+0000 7f6141ffb640 1 -- 172.21.15.70:0/3549401679 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c0037e0 con 0x55e4d805f4d0 2024-04-01T01:45:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.738+0000 7f616083e640 1 --2- 172.21.15.70:0/2376541708 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61440a2660 0x7f61440a4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.738+0000 7f616083e640 1 -- 172.21.15.70:0/2376541708 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f61540ce110 con 0x7f615c106130 2024-04-01T01:45:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.738+0000 7f615a7fc640 1 --2- 172.21.15.70:0/2376541708 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61440a2660 0x7f61440a4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.739+0000 7f6164425740 1 -- 172.21.15.70:0/3549401679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 msgr2=0x55e4d8063fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.739+0000 7f6164425740 1 --2- 172.21.15.70:0/3549401679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 0x55e4d8063fd0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f614c02c180 tx=0x7f614c005be0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.739+0000 7f615a7fc640 1 --2- 172.21.15.70:0/2376541708 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61440a2660 0x7f61440a4b20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x55e4d805fd30 tx=0x7f6150013040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.739+0000 7f6164425740 1 -- 172.21.15.70:0/3549401679 shutdown_connections 2024-04-01T01:45:36.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.739+0000 7f6164425740 1 --2- 172.21.15.70:0/3549401679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 0x55e4d8066ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.739+0000 7f6164425740 1 --2- 172.21.15.70:0/3549401679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 0x55e4d8063fd0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.739+0000 7f6164425740 1 --2- 172.21.15.70:0/3549401679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d805ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.739+0000 7f6164425740 1 -- 172.21.15.70:0/3549401679 >> 172.21.15.70:0/3549401679 conn(0x55e4d805a050 msgr2=0x55e4d805a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:36.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.739+0000 7f6164425740 1 -- 172.21.15.70:0/3549401679 shutdown_connections 2024-04-01T01:45:36.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.739+0000 7f6164425740 1 -- 172.21.15.70:0/3549401679 wait complete. 2024-04-01T01:45:36.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.739+0000 7f6164425740 1 -- start start 2024-04-01T01:45:36.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.739+0000 7f6164425740 1 Processor -- start 2024-04-01T01:45:36.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f6164425740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d8079dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f6164425740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d805f4d0 0x55e4d807a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f6164425740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d8064640 0x55e4d8081340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f6164425740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e4d80785a0 con 0x55e4d805f4d0 2024-04-01T01:45:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f6164425740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e4d7f60f80 con 0x55e4d8064640 2024-04-01T01:45:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f6164425740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e4d8078720 con 0x55e4d805eae0 2024-04-01T01:45:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f6164425740 10 client.0 _ll_register_callbacks cb 0x55e4d7fb9290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f615b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d8064640 0x55e4d8081340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f615b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d8064640 0x55e4d8081340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38644/0 (socket says 172.21.15.70:38644) 2024-04-01T01:45:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f615b7fe640 1 -- 172.21.15.70:0/4281160995 learned_addr learned my addr 172.21.15.70:0/4281160995 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f615a7fc640 1 --2- 172.21.15.70:0/4281160995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d805f4d0 0x55e4d807a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f615affd640 1 --2- 172.21.15.70:0/4281160995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d8079dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f615b7fe640 1 -- 172.21.15.70:0/4281160995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 msgr2=0x55e4d8079dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f615b7fe640 1 --2- 172.21.15.70:0/4281160995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d8079dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f615b7fe640 1 -- 172.21.15.70:0/4281160995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d805f4d0 msgr2=0x55e4d807a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f615b7fe640 1 --2- 172.21.15.70:0/4281160995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d805f4d0 0x55e4d807a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f615b7fe640 1 -- 172.21.15.70:0/4281160995 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4d7da3620 con 0x55e4d8064640 2024-04-01T01:45:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.740+0000 7f615a7fc640 1 --2- 172.21.15.70:0/4281160995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d805f4d0 0x55e4d807a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:45:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.741+0000 7f615b7fe640 1 --2- 172.21.15.70:0/4281160995 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d8064640 0x55e4d8081340 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f61540a2c80 tx=0x7f615400bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=beb0fc86b064e4d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.741+0000 7f613affd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:45:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.741+0000 7f613affd640 1 -- 172.21.15.70:0/4281160995 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61540a31b0 con 0x55e4d8064640 2024-04-01T01:45:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.741+0000 7f613affd640 1 -- 172.21.15.70:0/4281160995 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6154005140 con 0x55e4d8064640 2024-04-01T01:45:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.741+0000 7f6164425740 1 -- 172.21.15.70:0/4281160995 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f614c004040 con 0x55e4d8064640 2024-04-01T01:45:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.741+0000 7f613affd640 1 -- 172.21.15.70:0/4281160995 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f61540b8050 con 0x55e4d8064640 2024-04-01T01:45:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.741+0000 7f6164425740 10 client.29111 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:36.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.741+0000 7f6164425740 1 -- 172.21.15.70:0/4281160995 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e4d80843e0 con 0x55e4d8064640 2024-04-01T01:45:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.742+0000 7f613affd640 1 -- 172.21.15.70:0/4281160995 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f614c004040 con 0x55e4d8064640 2024-04-01T01:45:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.742+0000 7f613affd640 1 -- 172.21.15.70:0/4281160995 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6154016070 con 0x55e4d8064640 2024-04-01T01:45:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.742+0000 7f6164425740 10 client.29111 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.742+0000 7f6164425740 10 client.29111 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.742+0000 7f6164425740 20 client.29111 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.742+0000 7f6164425740 1 --2- 172.21.15.70:0/4281160995 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e4d8085ed0 0x55e4d8088390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.742+0000 7f6164425740 4 client.29111 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:45:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.742+0000 7f6164425740 1 -- 172.21.15.70:0/4281160995 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4d7fd6120 con 0x55e4d8085ed0 2024-04-01T01:45:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.742+0000 7f615affd640 1 --2- 172.21.15.70:0/4281160995 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e4d8085ed0 0x55e4d8088390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.743+0000 7f615affd640 1 --2- 172.21.15.70:0/4281160995 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e4d8085ed0 0x55e4d8088390 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e0baddb69595d7c server_cookie=eecd35d489e11403 in_seq=0 out_seq=0 2024-04-01T01:45:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.743+0000 7f613affd640 10 client.29111 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.744+0000 7f613affd640 1 -- 172.21.15.70:0/4281160995 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e4d7fd6120 con 0x55e4d8085ed0 2024-04-01T01:45:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.744+0000 7f613affd640 10 client.29111 handle_command_reply: tid=0 2024-04-01T01:45:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.744+0000 7f6164425740 1 client.29111 shutdown 2024-04-01T01:45:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.746+0000 7f6164425740 1 -- 172.21.15.70:0/4281160995 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d8064640 msgr2=0x55e4d8081340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.746+0000 7f6164425740 1 --2- 172.21.15.70:0/4281160995 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d8064640 0x55e4d8081340 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f61540a2c80 tx=0x7f615400bf20 comp rx=0 tx=0).stop 2024-04-01T01:45:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.746+0000 7f6164425740 1 -- 172.21.15.70:0/4281160995 shutdown_connections 2024-04-01T01:45:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.746+0000 7f6164425740 1 --2- 172.21.15.70:0/4281160995 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e4d8085ed0 0x55e4d8088390 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.746+0000 7f6164425740 1 --2- 172.21.15.70:0/4281160995 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d8064640 0x55e4d8081340 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.746+0000 7f6164425740 1 --2- 172.21.15.70:0/4281160995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d805f4d0 0x55e4d807a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.746+0000 7f6164425740 1 --2- 172.21.15.70:0/4281160995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d8079dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.746+0000 7f6164425740 1 -- 172.21.15.70:0/4281160995 >> 172.21.15.70:0/4281160995 conn(0x55e4d805a050 msgr2=0x55e4d8059190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.746+0000 7f613affd640 0 client.29111 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.747+0000 7f6164425740 1 -- 172.21.15.70:0/4281160995 shutdown_connections 2024-04-01T01:45:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.747+0000 7f6164425740 1 -- 172.21.15.70:0/4281160995 wait complete. 2024-04-01T01:45:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.747+0000 7f6164425740 20 client.29111 trim_cache size 0 max 16384 2024-04-01T01:45:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.775+0000 7f6164425740 1 Processor -- start 2024-04-01T01:45:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.775+0000 7f6164425740 1 -- start start 2024-04-01T01:45:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.775+0000 7f6164425740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d8079dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.775+0000 7f6164425740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 0x55e4d807a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.776+0000 7f6164425740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 0x55e4d8081300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.776+0000 7f615affd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d8079dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.776+0000 7f615affd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d8079dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:48872/0 (socket says 172.21.15.70:48872) 2024-04-01T01:45:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.776+0000 7f615a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 0x55e4d807a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.776+0000 7f615a7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 0x55e4d807a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38658/0 (socket says 172.21.15.70:38658) 2024-04-01T01:45:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.776+0000 7f615b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 0x55e4d8081300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.776+0000 7f615b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 0x55e4d8081300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56950/0 (socket says 172.21.15.70:56950) 2024-04-01T01:45:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.776+0000 7f615affd640 1 -- 172.21.15.70:0/1993981885 learned_addr learned my addr 172.21.15.70:0/1993981885 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.776+0000 7f6164425740 1 -- 172.21.15.70:0/1993981885 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f615c1a7e90 con 0x55e4d8064640 2024-04-01T01:45:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.776+0000 7f6164425740 1 -- 172.21.15.70:0/1993981885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f615c1a99d0 con 0x55e4d805f4d0 2024-04-01T01:45:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.776+0000 7f6164425740 1 -- 172.21.15.70:0/1993981885 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f615c1a8560 con 0x55e4d805eae0 2024-04-01T01:45:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.777+0000 7f615affd640 1 -- 172.21.15.70:0/1993981885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 msgr2=0x55e4d807a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.777+0000 7f615affd640 1 --2- 172.21.15.70:0/1993981885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 0x55e4d807a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.777+0000 7f615affd640 1 -- 172.21.15.70:0/1993981885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 msgr2=0x55e4d8081300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.777+0000 7f615affd640 1 --2- 172.21.15.70:0/1993981885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 0x55e4d8081300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.777+0000 7f615affd640 1 -- 172.21.15.70:0/1993981885 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4d8063a80 con 0x55e4d805eae0 2024-04-01T01:45:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.777+0000 7f615a7fc640 1 --2- 172.21.15.70:0/1993981885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 0x55e4d807a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:45:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.777+0000 7f615b7fe640 1 --2- 172.21.15.70:0/1993981885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 0x55e4d8081300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:45:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.777+0000 7f615affd640 1 --2- 172.21.15.70:0/1993981885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d8079dc0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f614c009900 tx=0x7f614c041040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1fc2f70df2efb5d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.778+0000 7f61417fa640 1 -- 172.21.15.70:0/1993981885 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c043030 con 0x55e4d805eae0 2024-04-01T01:45:36.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.778+0000 7f61417fa640 1 -- 172.21.15.70:0/1993981885 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f614c044040 con 0x55e4d805eae0 2024-04-01T01:45:36.780 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.778+0000 7f61417fa640 1 -- 172.21.15.70:0/1993981885 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c0504e0 con 0x55e4d805eae0 2024-04-01T01:45:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.779+0000 7f6164425740 1 -- 172.21.15.70:0/1993981885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 msgr2=0x55e4d8079dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.779+0000 7f6164425740 1 --2- 172.21.15.70:0/1993981885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d8079dc0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f614c009900 tx=0x7f614c041040 comp rx=0 tx=0).stop 2024-04-01T01:45:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.779+0000 7f6164425740 1 -- 172.21.15.70:0/1993981885 shutdown_connections 2024-04-01T01:45:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.779+0000 7f6164425740 1 --2- 172.21.15.70:0/1993981885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 0x55e4d8081300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.779+0000 7f6164425740 1 --2- 172.21.15.70:0/1993981885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 0x55e4d807a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.779+0000 7f6164425740 1 --2- 172.21.15.70:0/1993981885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d8079dc0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.779+0000 7f6164425740 1 -- 172.21.15.70:0/1993981885 >> 172.21.15.70:0/1993981885 conn(0x55e4d805a050 msgr2=0x55e4d805a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.779+0000 7f6164425740 1 -- 172.21.15.70:0/1993981885 shutdown_connections 2024-04-01T01:45:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.779+0000 7f6164425740 1 -- 172.21.15.70:0/1993981885 wait complete. 2024-04-01T01:45:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.780+0000 7f6164425740 1 -- start start 2024-04-01T01:45:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.780+0000 7f6164425740 1 Processor -- start 2024-04-01T01:45:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.781+0000 7f6164425740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d8081040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.781+0000 7f6164425740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 0x55e4d807dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.781+0000 7f6164425740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 0x55e4d807e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.781+0000 7f615affd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d8081040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.781+0000 7f615affd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d8081040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:48884/0 (socket says 172.21.15.70:48884) 2024-04-01T01:45:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.781+0000 7f615affd640 1 -- 172.21.15.70:0/1271286798 learned_addr learned my addr 172.21.15.70:0/1271286798 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.781+0000 7f6164425740 1 -- 172.21.15.70:0/1271286798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f615c1a9300 con 0x55e4d8064640 2024-04-01T01:45:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.781+0000 7f6164425740 1 -- 172.21.15.70:0/1271286798 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f615c1a8c30 con 0x55e4d805f4d0 2024-04-01T01:45:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.781+0000 7f6164425740 1 -- 172.21.15.70:0/1271286798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f615c0733b0 con 0x55e4d805eae0 2024-04-01T01:45:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.781+0000 7f615b7fe640 1 --2- 172.21.15.70:0/1271286798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 0x55e4d807e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.782+0000 7f615a7fc640 1 --2- 172.21.15.70:0/1271286798 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 0x55e4d807dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.782+0000 7f6164425740 10 client.0 _ll_register_callbacks cb 0x7f6154096430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.782+0000 7f615affd640 1 -- 172.21.15.70:0/1271286798 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 msgr2=0x55e4d807dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.782+0000 7f615affd640 1 --2- 172.21.15.70:0/1271286798 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 0x55e4d807dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.782+0000 7f615affd640 1 -- 172.21.15.70:0/1271286798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 msgr2=0x55e4d807e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.782+0000 7f615affd640 1 --2- 172.21.15.70:0/1271286798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 0x55e4d807e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.782+0000 7f615affd640 1 -- 172.21.15.70:0/1271286798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4d807e750 con 0x55e4d805eae0 2024-04-01T01:45:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.782+0000 7f615affd640 1 --2- 172.21.15.70:0/1271286798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d8081040 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f614c02c960 tx=0x7f614c005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ba11f124f7b8bf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.782+0000 7f613affd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:45:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.782+0000 7f613affd640 1 -- 172.21.15.70:0/1271286798 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c058060 con 0x55e4d805eae0 2024-04-01T01:45:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.782+0000 7f613affd640 1 -- 172.21.15.70:0/1271286798 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f614c02ca30 con 0x55e4d805eae0 2024-04-01T01:45:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.783+0000 7f6164425740 1 -- 172.21.15.70:0/1271286798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f614c044040 con 0x55e4d805eae0 2024-04-01T01:45:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.783+0000 7f613affd640 1 -- 172.21.15.70:0/1271286798 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f614c043040 con 0x55e4d805eae0 2024-04-01T01:45:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.783+0000 7f6164425740 10 client.28906 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:36.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.783+0000 7f6164425740 1 -- 172.21.15.70:0/1271286798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e4d80eb3b0 con 0x55e4d805eae0 2024-04-01T01:45:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.783+0000 7f613affd640 1 -- 172.21.15.70:0/1271286798 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f614c03f8f0 con 0x55e4d805eae0 2024-04-01T01:45:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.783+0000 7f613affd640 1 -- 172.21.15.70:0/1271286798 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f614c052070 con 0x55e4d805eae0 2024-04-01T01:45:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.784+0000 7f6164425740 10 client.28906 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.784+0000 7f6164425740 10 client.28906 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.784+0000 7f6164425740 20 client.28906 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:36.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.784+0000 7f6164425740 1 --2- 172.21.15.70:0/1271286798 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e4d80eb850 0x55e4d80edc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.784+0000 7f6164425740 4 client.28906 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:45:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.784+0000 7f6164425740 1 -- 172.21.15.70:0/1271286798 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f614c044040 con 0x55e4d80eb850 2024-04-01T01:45:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.784+0000 7f615a7fc640 1 --2- 172.21.15.70:0/1271286798 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e4d80eb850 0x55e4d80edc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.785+0000 7f615a7fc640 1 --2- 172.21.15.70:0/1271286798 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e4d80eb850 0x55e4d80edc40 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f38a55932fcac170 server_cookie=7f67b206e414d5f4 in_seq=0 out_seq=0 2024-04-01T01:45:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.785+0000 7f613affd640 10 client.28906 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:36.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 650.0 s, 859.6 tps, lat 114.448 ms stddev 87.915 2024-04-01T01:45:36.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.961+0000 7f613affd640 1 -- 172.21.15.70:0/1271286798 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f614c044040 con 0x55e4d80eb850 2024-04-01T01:45:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.961+0000 7f613affd640 10 client.28906 handle_command_reply: tid=0 2024-04-01T01:45:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.961+0000 7f6164425740 1 client.28906 shutdown 2024-04-01T01:45:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.961+0000 7f6164425740 1 -- 172.21.15.70:0/1271286798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 msgr2=0x55e4d8081040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.961+0000 7f6164425740 1 --2- 172.21.15.70:0/1271286798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d8081040 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f614c02c960 tx=0x7f614c005be0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.961+0000 7f6164425740 1 -- 172.21.15.70:0/1271286798 shutdown_connections 2024-04-01T01:45:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.961+0000 7f6164425740 1 --2- 172.21.15.70:0/1271286798 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e4d80eb850 0x55e4d80edc40 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.961+0000 7f6164425740 1 --2- 172.21.15.70:0/1271286798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e4d8064640 0x55e4d807e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.961+0000 7f613affd640 0 client.28906 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.961+0000 7f6164425740 1 --2- 172.21.15.70:0/1271286798 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4d805f4d0 0x55e4d807dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.961+0000 7f6164425740 1 --2- 172.21.15.70:0/1271286798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4d805eae0 0x55e4d8081040 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.961+0000 7f6164425740 1 -- 172.21.15.70:0/1271286798 >> 172.21.15.70:0/1271286798 conn(0x55e4d805a050 msgr2=0x55e4d8060790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.961+0000 7f6164425740 1 -- 172.21.15.70:0/1271286798 shutdown_connections 2024-04-01T01:45:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.961+0000 7f6164425740 1 -- 172.21.15.70:0/1271286798 wait complete. 2024-04-01T01:45:36.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.961+0000 7f6164425740 20 client.28906 trim_cache size 0 max 16384 2024-04-01T01:45:36.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:45:36.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:45:36.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:45:36.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:45:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.962+0000 7f615bfff640 1 -- 172.21.15.70:0/2376541708 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61440a2660 msgr2=0x7f61440a4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.962+0000 7f615bfff640 1 --2- 172.21.15.70:0/2376541708 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61440a2660 0x7f61440a4b20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x55e4d805fd30 tx=0x7f6150013040 comp rx=0 tx=0).stop 2024-04-01T01:45:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.962+0000 7f615bfff640 1 -- 172.21.15.70:0/2376541708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f615c106130 msgr2=0x7f615c1a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:36.965 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.962+0000 7f615bfff640 1 --2- 172.21.15.70:0/2376541708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f615c106130 0x7f615c1a5440 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f615400da90 tx=0x7f615400df50 comp rx=0 tx=0).stop 2024-04-01T01:45:36.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.962+0000 7f615bfff640 1 -- 172.21.15.70:0/2376541708 shutdown_connections 2024-04-01T01:45:36.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.962+0000 7f615bfff640 1 --2- 172.21.15.70:0/2376541708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f615c106130 0x7f615c1a5440 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.962+0000 7f615bfff640 1 --2- 172.21.15.70:0/2376541708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f615c1036b0 0x7f615c19e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.962+0000 7f615bfff640 1 --2- 172.21.15.70:0/2376541708 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f61440a2660 0x7f61440a4b20 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.962+0000 7f615bfff640 1 --2- 172.21.15.70:0/2376541708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f615c100cf0 0x7f615c19de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:36.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.962+0000 7f615bfff640 1 -- 172.21.15.70:0/2376541708 >> 172.21.15.70:0/2376541708 conn(0x7f615c0fad60 msgr2=0x7f615c102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:36.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.962+0000 7f615bfff640 1 -- 172.21.15.70:0/2376541708 shutdown_connections 2024-04-01T01:45:36.966 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:36.962+0000 7f615bfff640 1 -- 172.21.15.70:0/2376541708 wait complete. 2024-04-01T01:45:37.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:36 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:45:37.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:36 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:45:37.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:36 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:45:37.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:36 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:45:37.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:36 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:45:37.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:36 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:45:37.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:36 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:45:37.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:36 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:45:37.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:36 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:45:37.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:36 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:45:37.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:36 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:45:37.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:36 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:45:37.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:36 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:45:37.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:36 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:45:37.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:36 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:45:38.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:37 smithi189 ceph-mon[35753]: pgmap v723: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 9.8 MiB/s wr, 849 op/s 2024-04-01T01:45:38.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:37 smithi070 ceph-mon[32179]: pgmap v723: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 9.8 MiB/s wr, 849 op/s 2024-04-01T01:45:38.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:37 smithi081 ceph-mon[35142]: pgmap v723: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 9.8 MiB/s wr, 849 op/s 2024-04-01T01:45:40.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:40 smithi189 ceph-mon[35753]: pgmap v724: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 9.7 MiB/s wr, 892 op/s 2024-04-01T01:45:41.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:40 smithi070 ceph-mon[32179]: pgmap v724: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 9.7 MiB/s wr, 892 op/s 2024-04-01T01:45:41.030 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:40 smithi081 ceph-mon[35142]: pgmap v724: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 9.7 MiB/s wr, 892 op/s 2024-04-01T01:45:41.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 655.0 s, 829.0 tps, lat 122.389 ms stddev 127.263 2024-04-01T01:45:42.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:42 smithi189 ceph-mon[35753]: pgmap v725: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 8.1 MiB/s wr, 871 op/s 2024-04-01T01:45:43.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:42 smithi070 ceph-mon[32179]: pgmap v725: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 8.1 MiB/s wr, 871 op/s 2024-04-01T01:45:43.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:42 smithi081 ceph-mon[35142]: pgmap v725: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 8.1 MiB/s wr, 871 op/s 2024-04-01T01:45:44.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:44 smithi189 ceph-mon[35753]: pgmap v726: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 9.0 MiB/s wr, 946 op/s 2024-04-01T01:45:45.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:44 smithi070 ceph-mon[32179]: pgmap v726: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 9.0 MiB/s wr, 946 op/s 2024-04-01T01:45:45.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:44 smithi081 ceph-mon[35142]: pgmap v726: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 9.0 MiB/s wr, 946 op/s 2024-04-01T01:45:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 660.0 s, 852.0 tps, lat 117.124 ms stddev 104.275 2024-04-01T01:45:46.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:46 smithi189 ceph-mon[35753]: pgmap v727: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 8.4 MiB/s wr, 872 op/s 2024-04-01T01:45:47.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:46 smithi070 ceph-mon[32179]: pgmap v727: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 8.4 MiB/s wr, 872 op/s 2024-04-01T01:45:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:46 smithi081 ceph-mon[35142]: pgmap v727: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 8.4 MiB/s wr, 872 op/s 2024-04-01T01:45:48.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:48 smithi189 ceph-mon[35753]: pgmap v728: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 8.5 MiB/s wr, 934 op/s 2024-04-01T01:45:49.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:48 smithi070 ceph-mon[32179]: pgmap v728: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 8.5 MiB/s wr, 934 op/s 2024-04-01T01:45:49.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:48 smithi081 ceph-mon[35142]: pgmap v728: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 8.5 MiB/s wr, 934 op/s 2024-04-01T01:45:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.109+0000 7f5ba3a49640 1 -- 172.21.15.70:0/2032347160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b9c109960 msgr2=0x7f5b9c1101f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.109+0000 7f5ba3a49640 1 --2- 172.21.15.70:0/2032347160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b9c109960 0x7f5b9c1101f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f5b98009900 tx=0x7f5b9802bc80 comp rx=0 tx=0).stop 2024-04-01T01:45:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.112+0000 7f5ba3a49640 1 -- 172.21.15.70:0/2032347160 shutdown_connections 2024-04-01T01:45:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.112+0000 7f5ba3a49640 1 --2- 172.21.15.70:0/2032347160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b9c109960 0x7f5b9c1101f0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.112+0000 7f5ba3a49640 1 --2- 172.21.15.70:0/2032347160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b9c108fa0 0x7f5b9c109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.112+0000 7f5ba3a49640 1 --2- 172.21.15.70:0/2032347160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9c107db0 0x7f5b9c1081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.112+0000 7f5ba3a49640 1 -- 172.21.15.70:0/2032347160 >> 172.21.15.70:0/2032347160 conn(0x7f5b9c076d60 msgr2=0x7f5b9c079180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.114+0000 7f5ba3a49640 1 -- 172.21.15.70:0/2032347160 shutdown_connections 2024-04-01T01:45:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.114+0000 7f5ba3a49640 1 -- 172.21.15.70:0/2032347160 wait complete. 2024-04-01T01:45:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.115+0000 7f5ba3a49640 1 Processor -- start 2024-04-01T01:45:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.115+0000 7f5ba3a49640 1 -- start start 2024-04-01T01:45:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.115+0000 7f5ba3a49640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b9c107db0 0x7f5b9c0700a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.115+0000 7f5ba3a49640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b9c108fa0 0x7f5b9c0705e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.115+0000 7f5ba3a49640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9c109960 0x7f5b9c1ae050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.115+0000 7f5ba3a49640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9c113940 con 0x7f5b9c108fa0 2024-04-01T01:45:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.116+0000 7f5ba3a49640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9c1137c0 con 0x7f5b9c107db0 2024-04-01T01:45:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.116+0000 7f5ba3a49640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9c113ac0 con 0x7f5b9c109960 2024-04-01T01:45:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.116+0000 7f5ba0fe4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b9c108fa0 0x7f5b9c0705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.116+0000 7f5ba1fe6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9c109960 0x7f5b9c1ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.116+0000 7f5ba0fe4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b9c108fa0 0x7f5b9c0705e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60482/0 (socket says 172.21.15.70:60482) 2024-04-01T01:45:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.116+0000 7f5ba0fe4640 1 -- 172.21.15.70:0/761762903 learned_addr learned my addr 172.21.15.70:0/761762903 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.116+0000 7f5ba17e5640 1 --2- 172.21.15.70:0/761762903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b9c107db0 0x7f5b9c0700a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.117+0000 7f5ba0fe4640 1 -- 172.21.15.70:0/761762903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9c109960 msgr2=0x7f5b9c1ae050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.117+0000 7f5ba0fe4640 1 --2- 172.21.15.70:0/761762903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9c109960 0x7f5b9c1ae050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.117+0000 7f5ba0fe4640 1 -- 172.21.15.70:0/761762903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b9c107db0 msgr2=0x7f5b9c0700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.117+0000 7f5ba0fe4640 1 --2- 172.21.15.70:0/761762903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b9c107db0 0x7f5b9c0700a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.117+0000 7f5ba0fe4640 1 -- 172.21.15.70:0/761762903 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b9c070c80 con 0x7f5b9c108fa0 2024-04-01T01:45:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.117+0000 7f5ba17e5640 1 --2- 172.21.15.70:0/761762903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b9c107db0 0x7f5b9c0700a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:45:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.118+0000 7f5ba0fe4640 1 --2- 172.21.15.70:0/761762903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b9c108fa0 0x7f5b9c0705e0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f5b9400ba50 tx=0x7f5b9400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.118+0000 7f5b927fc640 1 -- 172.21.15.70:0/761762903 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b94019070 con 0x7f5b9c108fa0 2024-04-01T01:45:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.118+0000 7f5b927fc640 1 -- 172.21.15.70:0/761762903 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b9400cbe0 con 0x7f5b9c108fa0 2024-04-01T01:45:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.118+0000 7f5ba3a49640 1 -- 172.21.15.70:0/761762903 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b9c07a0c0 con 0x7f5b9c108fa0 2024-04-01T01:45:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.118+0000 7f5ba3a49640 1 -- 172.21.15.70:0/761762903 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b9c07a690 con 0x7f5b9c108fa0 2024-04-01T01:45:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.118+0000 7f5b927fc640 1 -- 172.21.15.70:0/761762903 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b940075d0 con 0x7f5b9c108fa0 2024-04-01T01:45:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.121+0000 7f5b927fc640 1 -- 172.21.15.70:0/761762903 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f5b9401c070 con 0x7f5b9c108fa0 2024-04-01T01:45:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.121+0000 7f5ba6622740 1 Processor -- start 2024-04-01T01:45:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.122+0000 7f5ba6622740 1 -- start start 2024-04-01T01:45:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.122+0000 7f5ba6622740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3648d0 0x55720c364cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.122+0000 7f5ba6622740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3652c0 0x55720c369dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.122+0000 7f5ba6622740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c36a430 0x55720c36c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.122+0000 7f5ba6622740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55720c2c88f0 con 0x55720c36a430 2024-04-01T01:45:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.122+0000 7f5ba6622740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55720c2718d0 con 0x55720c3648d0 2024-04-01T01:45:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.122+0000 7f5ba6622740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55720c2c4990 con 0x55720c3652c0 2024-04-01T01:45:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.122+0000 7f5ba17e5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3652c0 0x55720c369dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.122+0000 7f5ba17e5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3652c0 0x55720c369dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40412/0 (socket says 172.21.15.70:40412) 2024-04-01T01:45:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.122+0000 7f5ba17e5640 1 -- 172.21.15.70:0/2221849824 learned_addr learned my addr 172.21.15.70:0/2221849824 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.122+0000 7f5ba0fe4640 1 --2- 172.21.15.70:0/2221849824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c36a430 0x55720c36c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.122+0000 7f5ba1fe6640 1 --2- 172.21.15.70:0/2221849824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3648d0 0x55720c364cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.123+0000 7f5ba17e5640 1 -- 172.21.15.70:0/2221849824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3648d0 msgr2=0x55720c364cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.123+0000 7f5ba17e5640 1 --2- 172.21.15.70:0/2221849824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3648d0 0x55720c364cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.123+0000 7f5ba17e5640 1 -- 172.21.15.70:0/2221849824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c36a430 msgr2=0x55720c36c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.123+0000 7f5ba17e5640 1 --2- 172.21.15.70:0/2221849824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c36a430 0x55720c36c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.123+0000 7f5ba17e5640 1 -- 172.21.15.70:0/2221849824 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55720bfdd210 con 0x55720c3652c0 2024-04-01T01:45:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.123+0000 7f5ba17e5640 1 --2- 172.21.15.70:0/2221849824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3652c0 0x55720c369dc0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c009c30 tx=0x7f5b8c00e6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=629cd39b419f2933 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.123+0000 7f5b927fc640 1 --2- 172.21.15.70:0/761762903 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5b880a2610 0x7f5b880a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.124+0000 7f5b877fe640 1 -- 172.21.15.70:0/2221849824 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c016070 con 0x55720c3652c0 2024-04-01T01:45:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.124+0000 7f5b877fe640 1 -- 172.21.15.70:0/2221849824 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b8c0054d0 con 0x55720c3652c0 2024-04-01T01:45:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.124+0000 7f5b877fe640 1 -- 172.21.15.70:0/2221849824 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c011470 con 0x55720c3652c0 2024-04-01T01:45:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.124+0000 7f5ba1fe6640 1 --2- 172.21.15.70:0/761762903 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5b880a2610 0x7f5b880a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.124+0000 7f5b927fc640 1 -- 172.21.15.70:0/761762903 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5b940ce190 con 0x7f5b9c108fa0 2024-04-01T01:45:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.125+0000 7f5ba6622740 1 -- 172.21.15.70:0/2221849824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3652c0 msgr2=0x55720c369dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.125+0000 7f5ba6622740 1 --2- 172.21.15.70:0/2221849824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3652c0 0x55720c369dc0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c009c30 tx=0x7f5b8c00e6c0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.126+0000 7f5ba1fe6640 1 --2- 172.21.15.70:0/761762903 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5b880a2610 0x7f5b880a4ad0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5b98005ab0 tx=0x7f5b98004be0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.126+0000 7f5ba6622740 1 -- 172.21.15.70:0/2221849824 shutdown_connections 2024-04-01T01:45:50.126 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.126+0000 7f5ba6622740 1 --2- 172.21.15.70:0/2221849824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c36a430 0x55720c36c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.126+0000 7f5ba6622740 1 --2- 172.21.15.70:0/2221849824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3652c0 0x55720c369dc0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.126+0000 7f5ba6622740 1 --2- 172.21.15.70:0/2221849824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3648d0 0x55720c364cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.126+0000 7f5ba6622740 1 -- 172.21.15.70:0/2221849824 >> 172.21.15.70:0/2221849824 conn(0x55720c35fe40 msgr2=0x55720c360240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:50.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.127+0000 7f5ba6622740 1 -- 172.21.15.70:0/2221849824 shutdown_connections 2024-04-01T01:45:50.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.127+0000 7f5ba6622740 1 -- 172.21.15.70:0/2221849824 wait complete. 2024-04-01T01:45:50.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.128+0000 7f5ba6622740 1 -- start start 2024-04-01T01:45:50.128 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.128+0000 7f5ba6622740 1 Processor -- start 2024-04-01T01:45:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.129+0000 7f5ba6622740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c3648d0 0x55720c36ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.129+0000 7f5ba6622740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3652c0 0x55720c36f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.129+0000 7f5ba6622740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c36a430 0x55720c376150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.129+0000 7f5ba6622740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55720c36d3e0 con 0x55720c3648d0 2024-04-01T01:45:50.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.129+0000 7f5ba6622740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55720c266f80 con 0x55720c36a430 2024-04-01T01:45:50.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.129+0000 7f5ba6622740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55720c36d560 con 0x55720c3652c0 2024-04-01T01:45:50.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.129+0000 7f5ba0fe4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3652c0 0x55720c36f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:50.130 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.129+0000 7f5ba0fe4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3652c0 0x55720c36f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40420/0 (socket says 172.21.15.70:40420) 2024-04-01T01:45:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.129+0000 7f5ba6622740 10 client.0 _ll_register_callbacks cb 0x55720c2bf290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.129+0000 7f5ba0fe4640 1 -- 172.21.15.70:0/3862073335 learned_addr learned my addr 172.21.15.70:0/3862073335 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.129+0000 7f5ba1fe6640 1 --2- 172.21.15.70:0/3862073335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c36a430 0x55720c376150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.130+0000 7f5ba0fe4640 1 -- 172.21.15.70:0/3862073335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c36a430 msgr2=0x55720c376150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.130+0000 7f5ba0fe4640 1 --2- 172.21.15.70:0/3862073335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c36a430 0x55720c376150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.130+0000 7f5ba0fe4640 1 -- 172.21.15.70:0/3862073335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c3648d0 msgr2=0x55720c36ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.130+0000 7f5ba0fe4640 1 --2- 172.21.15.70:0/3862073335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c3648d0 0x55720c36ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.130+0000 7f5ba0fe4640 1 -- 172.21.15.70:0/3862073335 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55720c0a9620 con 0x55720c3652c0 2024-04-01T01:45:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.130+0000 7f5ba1fe6640 1 --2- 172.21.15.70:0/3862073335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c36a430 0x55720c376150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:45:50.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.130+0000 7f5ba0fe4640 1 --2- 172.21.15.70:0/3862073335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3652c0 0x55720c36f110 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5b94097210 tx=0x7f5b9409a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69b810a3bfba2e72 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:50.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.130+0000 7f5b84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:45:50.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.131+0000 7f5b84ff9640 1 -- 172.21.15.70:0/3862073335 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b94098a10 con 0x55720c3652c0 2024-04-01T01:45:50.132 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.132+0000 7f5ba6622740 1 -- 172.21.15.70:0/3862073335 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b8c0054d0 con 0x55720c3652c0 2024-04-01T01:45:50.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.132+0000 7f5b84ff9640 1 -- 172.21.15.70:0/3862073335 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5b940850e0 con 0x55720c3652c0 2024-04-01T01:45:50.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.133+0000 7f5b84ff9640 1 -- 172.21.15.70:0/3862073335 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5b9409a8d0 con 0x55720c3652c0 2024-04-01T01:45:50.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.133+0000 7f5ba6622740 10 client.28918 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:50.133 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.133+0000 7f5ba6622740 1 -- 172.21.15.70:0/3862073335 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55720c3791f0 con 0x55720c3652c0 2024-04-01T01:45:50.136 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.136+0000 7f5b84ff9640 1 -- 172.21.15.70:0/3862073335 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b9409e970 con 0x55720c3652c0 2024-04-01T01:45:50.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.137+0000 7f5b84ff9640 1 -- 172.21.15.70:0/3862073335 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5b940d0110 con 0x55720c3652c0 2024-04-01T01:45:50.138 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.137+0000 7f5ba6622740 10 client.28918 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.137+0000 7f5ba6622740 10 client.28918 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.137+0000 7f5ba6622740 20 client.28918 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.138+0000 7f5ba6622740 1 --2- 172.21.15.70:0/3862073335 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55720c37ad70 0x55720c37d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.138+0000 7f5ba6622740 4 client.28918 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:45:50.139 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.138+0000 7f5ba17e5640 1 --2- 172.21.15.70:0/3862073335 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55720c37ad70 0x55720c37d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.138+0000 7f5ba6622740 1 -- 172.21.15.70:0/3862073335 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5b8c0054d0 con 0x55720c37ad70 2024-04-01T01:45:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.139+0000 7f5ba17e5640 1 --2- 172.21.15.70:0/3862073335 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55720c37ad70 0x55720c37d210 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d536d291af9cff66 server_cookie=df9363da0cb7227c in_seq=0 out_seq=0 2024-04-01T01:45:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.139+0000 7f5b84ff9640 10 client.28918 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.139+0000 7f5b84ff9640 1 -- 172.21.15.70:0/3862073335 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f5b8c0054d0 con 0x55720c37ad70 2024-04-01T01:45:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.139+0000 7f5b84ff9640 10 client.28918 handle_command_reply: tid=0 2024-04-01T01:45:50.140 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.140+0000 7f5ba6622740 1 client.28918 shutdown 2024-04-01T01:45:50.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.140+0000 7f5ba6622740 1 -- 172.21.15.70:0/3862073335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3652c0 msgr2=0x55720c36f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:50.141 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.140+0000 7f5ba6622740 1 --2- 172.21.15.70:0/3862073335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3652c0 0x55720c36f110 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5b94097210 tx=0x7f5b9409a6a0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.142+0000 7f5ba6622740 1 -- 172.21.15.70:0/3862073335 shutdown_connections 2024-04-01T01:45:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.142+0000 7f5ba6622740 1 --2- 172.21.15.70:0/3862073335 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55720c37ad70 0x55720c37d210 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.142+0000 7f5ba6622740 1 --2- 172.21.15.70:0/3862073335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c36a430 0x55720c376150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.142+0000 7f5ba6622740 1 --2- 172.21.15.70:0/3862073335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3652c0 0x55720c36f110 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.142+0000 7f5ba6622740 1 --2- 172.21.15.70:0/3862073335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c3648d0 0x55720c36ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.142+0000 7f5ba6622740 1 -- 172.21.15.70:0/3862073335 >> 172.21.15.70:0/3862073335 conn(0x55720c35fe40 msgr2=0x55720c35ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.142+0000 7f5b84ff9640 0 client.28918 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:50.143 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.143+0000 7f5ba6622740 1 -- 172.21.15.70:0/3862073335 shutdown_connections 2024-04-01T01:45:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.143+0000 7f5ba6622740 1 -- 172.21.15.70:0/3862073335 wait complete. 2024-04-01T01:45:50.144 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.143+0000 7f5ba6622740 20 client.28918 trim_cache size 0 max 16384 2024-04-01T01:45:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.176+0000 7f5ba6622740 1 Processor -- start 2024-04-01T01:45:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.176+0000 7f5ba6622740 1 -- start start 2024-04-01T01:45:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.177+0000 7f5ba6622740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c35fe40 0x55720c36c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.177+0000 7f5ba6622740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3648d0 0x55720c36ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.177+0000 7f5ba6622740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3652c0 0x55720c36f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.178+0000 7f5ba6622740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9c1c1940 con 0x55720c35fe40 2024-04-01T01:45:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.178+0000 7f5ba6622740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9c1c3480 con 0x55720c3652c0 2024-04-01T01:45:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.178+0000 7f5ba6622740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9c1c2010 con 0x55720c3648d0 2024-04-01T01:45:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.178+0000 7f5ba0fe4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3648d0 0x55720c36ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.178+0000 7f5ba1fe6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3652c0 0x55720c36f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.178+0000 7f5ba1fe6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3652c0 0x55720c36f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47326/0 (socket says 172.21.15.70:47326) 2024-04-01T01:45:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.178+0000 7f5ba1fe6640 1 -- 172.21.15.70:0/2838679087 learned_addr learned my addr 172.21.15.70:0/2838679087 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:50.180 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.179+0000 7f5ba1fe6640 1 -- 172.21.15.70:0/2838679087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3648d0 msgr2=0x55720c36ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.179+0000 7f5ba1fe6640 1 --2- 172.21.15.70:0/2838679087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3648d0 0x55720c36ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.179+0000 7f5ba1fe6640 1 -- 172.21.15.70:0/2838679087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c35fe40 msgr2=0x55720c36c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.179+0000 7f5ba1fe6640 1 --2- 172.21.15.70:0/2838679087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c35fe40 0x55720c36c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.179+0000 7f5ba1fe6640 1 -- 172.21.15.70:0/2838679087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55720c369810 con 0x55720c3652c0 2024-04-01T01:45:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.179+0000 7f5ba1fe6640 1 --2- 172.21.15.70:0/2838679087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3652c0 0x55720c36f110 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5b98009900 tx=0x7f5b98041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57496c2ccd95ac24 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.179+0000 7f5b86ffd640 1 -- 172.21.15.70:0/2838679087 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b98052070 con 0x55720c3652c0 2024-04-01T01:45:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.179+0000 7f5b86ffd640 1 -- 172.21.15.70:0/2838679087 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b9802cd80 con 0x55720c3652c0 2024-04-01T01:45:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.180+0000 7f5b86ffd640 1 -- 172.21.15.70:0/2838679087 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9804ede0 con 0x55720c3652c0 2024-04-01T01:45:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.181+0000 7f5ba6622740 1 -- 172.21.15.70:0/2838679087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3652c0 msgr2=0x55720c36f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.181+0000 7f5ba6622740 1 --2- 172.21.15.70:0/2838679087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3652c0 0x55720c36f110 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5b98009900 tx=0x7f5b98041000 comp rx=0 tx=0).stop 2024-04-01T01:45:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.181+0000 7f5ba6622740 1 -- 172.21.15.70:0/2838679087 shutdown_connections 2024-04-01T01:45:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.181+0000 7f5ba6622740 1 --2- 172.21.15.70:0/2838679087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3652c0 0x55720c36f110 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.181+0000 7f5ba6622740 1 --2- 172.21.15.70:0/2838679087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c3648d0 0x55720c36ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.181+0000 7f5ba6622740 1 --2- 172.21.15.70:0/2838679087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c35fe40 0x55720c36c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.181+0000 7f5ba6622740 1 -- 172.21.15.70:0/2838679087 >> 172.21.15.70:0/2838679087 conn(0x55720c369f00 msgr2=0x55720c360220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.181+0000 7f5ba6622740 1 -- 172.21.15.70:0/2838679087 shutdown_connections 2024-04-01T01:45:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.181+0000 7f5ba6622740 1 -- 172.21.15.70:0/2838679087 wait complete. 2024-04-01T01:45:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.182+0000 7f5ba6622740 1 -- start start 2024-04-01T01:45:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.182+0000 7f5ba6622740 1 Processor -- start 2024-04-01T01:45:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.182+0000 7f5ba6622740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c35fe40 0x55720c381650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.182+0000 7f5ba6622740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3648d0 0x55720c381b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.183+0000 7f5ba6622740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c3652c0 0x55720c3820d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.183+0000 7f5ba17e5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c35fe40 0x55720c381650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.183+0000 7f5ba17e5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c35fe40 0x55720c381650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:40442/0 (socket says 172.21.15.70:40442) 2024-04-01T01:45:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.183+0000 7f5ba17e5640 1 -- 172.21.15.70:0/3355364287 learned_addr learned my addr 172.21.15.70:0/3355364287 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:45:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.183+0000 7f5ba0fe4640 1 --2- 172.21.15.70:0/3355364287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3648d0 0x55720c381b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.183+0000 7f5ba6622740 1 -- 172.21.15.70:0/3355364287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9c1c2db0 con 0x55720c3652c0 2024-04-01T01:45:50.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.183+0000 7f5ba6622740 1 -- 172.21.15.70:0/3355364287 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9c1c26e0 con 0x55720c3648d0 2024-04-01T01:45:50.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.183+0000 7f5ba6622740 1 -- 172.21.15.70:0/3355364287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9c1bf9d0 con 0x55720c35fe40 2024-04-01T01:45:50.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.183+0000 7f5ba6622740 10 client.0 _ll_register_callbacks cb 0x7f5b94009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:45:50.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.183+0000 7f5ba1fe6640 1 --2- 172.21.15.70:0/3355364287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c3652c0 0x55720c3820d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:50.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.183+0000 7f5ba0fe4640 1 -- 172.21.15.70:0/3355364287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c35fe40 msgr2=0x55720c381650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:50.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.183+0000 7f5ba0fe4640 1 --2- 172.21.15.70:0/3355364287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c35fe40 0x55720c381650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.183+0000 7f5ba0fe4640 1 -- 172.21.15.70:0/3355364287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c3652c0 msgr2=0x55720c3820d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:45:50.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.183+0000 7f5ba0fe4640 1 --2- 172.21.15.70:0/3355364287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c3652c0 0x55720c3820d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.183+0000 7f5ba0fe4640 1 -- 172.21.15.70:0/3355364287 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55720c3df070 con 0x55720c3648d0 2024-04-01T01:45:50.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.183+0000 7f5ba17e5640 1 --2- 172.21.15.70:0/3355364287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c35fe40 0x55720c381650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:45:50.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.184+0000 7f5ba1fe6640 1 --2- 172.21.15.70:0/3355364287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c3652c0 0x55720c3820d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:45:50.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.184+0000 7f5ba0fe4640 1 --2- 172.21.15.70:0/3355364287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3648d0 0x55720c381b90 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x55720c36fa70 tx=0x7f5b940d0340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39044aa62d71856a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:45:50.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.184+0000 7f5b84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:45:50.187 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.184+0000 7f5b84ff9640 1 -- 172.21.15.70:0/3355364287 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b9409e970 con 0x55720c3648d0 2024-04-01T01:45:50.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.184+0000 7f5b84ff9640 1 -- 172.21.15.70:0/3355364287 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5b940aeb30 con 0x55720c3648d0 2024-04-01T01:45:50.188 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.185+0000 7f5ba6622740 1 -- 172.21.15.70:0/3355364287 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b9802cd80 con 0x55720c3648d0 2024-04-01T01:45:50.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.185+0000 7f5b84ff9640 1 -- 172.21.15.70:0/3355364287 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5b940aed70 con 0x55720c3648d0 2024-04-01T01:45:50.189 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.185+0000 7f5ba6622740 10 client.29132 fetch_fsmap learned FSMap version 31 2024-04-01T01:45:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.185+0000 7f5ba6622740 1 -- 172.21.15.70:0/3355364287 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55720c3e0330 con 0x55720c3648d0 2024-04-01T01:45:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.185+0000 7f5b84ff9640 1 -- 172.21.15.70:0/3355364287 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5b940b2050 con 0x55720c3648d0 2024-04-01T01:45:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.186+0000 7f5b84ff9640 1 -- 172.21.15.70:0/3355364287 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5b940b3250 con 0x55720c3648d0 2024-04-01T01:45:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.186+0000 7f5ba6622740 10 client.29132 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:45:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.186+0000 7f5ba6622740 10 client.29132 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:45:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.186+0000 7f5ba6622740 20 client.29132 populate_metadata read hostname 'smithi070' 2024-04-01T01:45:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.186+0000 7f5ba6622740 1 --2- 172.21.15.70:0/3355364287 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55720c36a430 0x55720c3e2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:45:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.186+0000 7f5ba6622740 4 client.29132 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:45:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.186+0000 7f5ba6622740 1 -- 172.21.15.70:0/3355364287 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f5b9802cd80 con 0x55720c36a430 2024-04-01T01:45:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.186+0000 7f5ba17e5640 1 --2- 172.21.15.70:0/3355364287 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55720c36a430 0x55720c3e2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:45:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.189+0000 7f5ba17e5640 1 --2- 172.21.15.70:0/3355364287 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55720c36a430 0x55720c3e2ca0 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=259d9b0905e5157c server_cookie=e8bdaff757e40344 in_seq=0 out_seq=0 2024-04-01T01:45:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.189+0000 7f5b84ff9640 10 client.29132 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:50.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.223+0000 7f5b84ff9640 1 -- 172.21.15.70:0/3355364287 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f5b9802cd80 con 0x55720c36a430 2024-04-01T01:45:50.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.223+0000 7f5b84ff9640 10 client.29132 handle_command_reply: tid=0 2024-04-01T01:45:50.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.223+0000 7f5ba6622740 1 client.29132 shutdown 2024-04-01T01:45:50.224 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.224+0000 7f5ba6622740 1 -- 172.21.15.70:0/3355364287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3648d0 msgr2=0x55720c381b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.224+0000 7f5ba6622740 1 --2- 172.21.15.70:0/3355364287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3648d0 0x55720c381b90 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x55720c36fa70 tx=0x7f5b940d0340 comp rx=0 tx=0).stop 2024-04-01T01:45:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.226+0000 7f5ba6622740 1 -- 172.21.15.70:0/3355364287 shutdown_connections 2024-04-01T01:45:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.226+0000 7f5ba6622740 1 --2- 172.21.15.70:0/3355364287 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55720c36a430 0x55720c3e2ca0 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.226+0000 7f5ba6622740 1 --2- 172.21.15.70:0/3355364287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55720c3652c0 0x55720c3820d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.226+0000 7f5ba6622740 1 --2- 172.21.15.70:0/3355364287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55720c3648d0 0x55720c381b90 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.226+0000 7f5ba6622740 1 --2- 172.21.15.70:0/3355364287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55720c35fe40 0x55720c381650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.226+0000 7f5ba6622740 1 -- 172.21.15.70:0/3355364287 >> 172.21.15.70:0/3355364287 conn(0x55720c369f00 msgr2=0x55720c36ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.227+0000 7f5b84ff9640 0 client.29132 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:45:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.227+0000 7f5ba6622740 1 -- 172.21.15.70:0/3355364287 shutdown_connections 2024-04-01T01:45:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.228+0000 7f5ba6622740 1 -- 172.21.15.70:0/3355364287 wait complete. 2024-04-01T01:45:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.228+0000 7f5ba6622740 20 client.29132 trim_cache size 0 max 16384 2024-04-01T01:45:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:45:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:45:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:45:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:45:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.229+0000 7f5ba3a49640 1 -- 172.21.15.70:0/761762903 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5b880a2610 msgr2=0x7f5b880a4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:50.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.230+0000 7f5ba3a49640 1 --2- 172.21.15.70:0/761762903 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5b880a2610 0x7f5b880a4ad0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5b98005ab0 tx=0x7f5b98004be0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.230+0000 7f5ba3a49640 1 -- 172.21.15.70:0/761762903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b9c108fa0 msgr2=0x7f5b9c0705e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:45:50.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.230+0000 7f5ba3a49640 1 --2- 172.21.15.70:0/761762903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b9c108fa0 0x7f5b9c0705e0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f5b9400ba50 tx=0x7f5b9400bf10 comp rx=0 tx=0).stop 2024-04-01T01:45:50.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.233+0000 7f5ba3a49640 1 -- 172.21.15.70:0/761762903 shutdown_connections 2024-04-01T01:45:50.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.233+0000 7f5ba3a49640 1 --2- 172.21.15.70:0/761762903 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5b880a2610 0x7f5b880a4ad0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.233+0000 7f5ba3a49640 1 --2- 172.21.15.70:0/761762903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b9c109960 0x7f5b9c1ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.233+0000 7f5ba3a49640 1 --2- 172.21.15.70:0/761762903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5b9c108fa0 0x7f5b9c0705e0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.233+0000 7f5ba3a49640 1 --2- 172.21.15.70:0/761762903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b9c107db0 0x7f5b9c0700a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:45:50.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.233+0000 7f5ba3a49640 1 -- 172.21.15.70:0/761762903 >> 172.21.15.70:0/761762903 conn(0x7f5b9c076d60 msgr2=0x7f5b9c10c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:45:50.236 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.235+0000 7f5ba3a49640 1 -- 172.21.15.70:0/761762903 shutdown_connections 2024-04-01T01:45:50.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:45:50.238+0000 7f5ba3a49640 1 -- 172.21.15.70:0/761762903 wait complete. 2024-04-01T01:45:50.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:50 smithi189 ceph-mon[35753]: pgmap v729: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 8.3 MiB/s wr, 905 op/s 2024-04-01T01:45:51.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:50 smithi070 ceph-mon[32179]: pgmap v729: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 8.3 MiB/s wr, 905 op/s 2024-04-01T01:45:51.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:50 smithi081 ceph-mon[35142]: pgmap v729: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 8.3 MiB/s wr, 905 op/s 2024-04-01T01:45:51.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 665.0 s, 780.6 tps, lat 128.520 ms stddev 128.292 2024-04-01T01:45:52.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:52 smithi189 ceph-mon[35753]: pgmap v730: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 8.5 MiB/s wr, 815 op/s 2024-04-01T01:45:53.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:52 smithi070 ceph-mon[32179]: pgmap v730: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 8.5 MiB/s wr, 815 op/s 2024-04-01T01:45:53.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:52 smithi081 ceph-mon[35142]: pgmap v730: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 8.5 MiB/s wr, 815 op/s 2024-04-01T01:45:54.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:54 smithi189 ceph-mon[35753]: pgmap v731: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.0 MiB/s wr, 798 op/s 2024-04-01T01:45:55.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:54 smithi070 ceph-mon[32179]: pgmap v731: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.0 MiB/s wr, 798 op/s 2024-04-01T01:45:55.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:54 smithi081 ceph-mon[35142]: pgmap v731: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.0 MiB/s wr, 798 op/s 2024-04-01T01:45:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 670.0 s, 852.4 tps, lat 117.450 ms stddev 121.731 2024-04-01T01:45:56.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:56 smithi189 ceph-mon[35753]: pgmap v732: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.7 MiB/s wr, 698 op/s 2024-04-01T01:45:57.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:56 smithi070 ceph-mon[32179]: pgmap v732: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.7 MiB/s wr, 698 op/s 2024-04-01T01:45:57.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:56 smithi081 ceph-mon[35142]: pgmap v732: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.7 MiB/s wr, 698 op/s 2024-04-01T01:45:58.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:45:58 smithi189 ceph-mon[35753]: pgmap v733: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 7.9 MiB/s wr, 793 op/s 2024-04-01T01:45:59.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:45:58 smithi070 ceph-mon[32179]: pgmap v733: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 7.9 MiB/s wr, 793 op/s 2024-04-01T01:45:59.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:45:58 smithi081 ceph-mon[35142]: pgmap v733: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 7.9 MiB/s wr, 793 op/s 2024-04-01T01:46:00.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:00 smithi070 ceph-mon[32179]: pgmap v734: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 8.5 MiB/s wr, 817 op/s 2024-04-01T01:46:00.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:00 smithi189 ceph-mon[35753]: pgmap v734: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 8.5 MiB/s wr, 817 op/s 2024-04-01T01:46:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:00 smithi081 ceph-mon[35142]: pgmap v734: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 8.5 MiB/s wr, 817 op/s 2024-04-01T01:46:01.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 675.0 s, 776.0 tps, lat 125.972 ms stddev 151.130 2024-04-01T01:46:02.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:02 smithi189 ceph-mon[35753]: pgmap v735: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.7 MiB/s wr, 763 op/s 2024-04-01T01:46:03.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:02 smithi070 ceph-mon[32179]: pgmap v735: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.7 MiB/s wr, 763 op/s 2024-04-01T01:46:03.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:02 smithi081 ceph-mon[35142]: pgmap v735: 129 pgs: 129 active+clean; 1.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.7 MiB/s wr, 763 op/s 2024-04-01T01:46:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.372+0000 7fb8f2ad6640 1 -- 172.21.15.70:0/4014350726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8ec1017a0 msgr2=0x7fb8ec101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.372+0000 7fb8f2ad6640 1 --2- 172.21.15.70:0/4014350726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8ec1017a0 0x7fb8ec101bc0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8009900 tx=0x7fb8e802bc30 comp rx=0 tx=0).stop 2024-04-01T01:46:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.373+0000 7fb8f2ad6640 1 -- 172.21.15.70:0/4014350726 shutdown_connections 2024-04-01T01:46:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.373+0000 7fb8f2ad6640 1 --2- 172.21.15.70:0/4014350726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8ec103350 0x7fb8ec109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.373+0000 7fb8f2ad6640 1 --2- 172.21.15.70:0/4014350726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec102990 0x7fb8ec102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.373+0000 7fb8f2ad6640 1 --2- 172.21.15.70:0/4014350726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8ec1017a0 0x7fb8ec101bc0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.373+0000 7fb8f2ad6640 1 -- 172.21.15.70:0/4014350726 >> 172.21.15.70:0/4014350726 conn(0x7fb8ec0fcd10 msgr2=0x7fb8ec0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.373+0000 7fb8f2ad6640 1 -- 172.21.15.70:0/4014350726 shutdown_connections 2024-04-01T01:46:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.373+0000 7fb8f2ad6640 1 -- 172.21.15.70:0/4014350726 wait complete. 2024-04-01T01:46:03.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.374+0000 7fb8f2ad6640 1 Processor -- start 2024-04-01T01:46:03.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.374+0000 7fb8f2ad6640 1 -- start start 2024-04-01T01:46:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.374+0000 7fb8f2ad6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8ec1017a0 0x7fb8ec19df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.374+0000 7fb8f2ad6640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8ec102990 0x7fb8ec19e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.374+0000 7fb8f2ad6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec103350 0x7fb8ec1a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.374+0000 7fb8f2ad6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec075890 con 0x7fb8ec1017a0 2024-04-01T01:46:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.374+0000 7fb8f2ad6640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec075710 con 0x7fb8ec103350 2024-04-01T01:46:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.374+0000 7fb8f2ad6640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec075a10 con 0x7fb8ec102990 2024-04-01T01:46:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.374+0000 7fb8f0872640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8ec1017a0 0x7fb8ec19df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.375+0000 7fb8e3fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8ec102990 0x7fb8ec19e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.375+0000 7fb8f1073640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec103350 0x7fb8ec1a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.375+0000 7fb8f0872640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8ec1017a0 0x7fb8ec19df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42842/0 (socket says 172.21.15.70:42842) 2024-04-01T01:46:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.375+0000 7fb8e3fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8ec102990 0x7fb8ec19e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36550/0 (socket says 172.21.15.70:36550) 2024-04-01T01:46:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.375+0000 7fb8f0872640 1 -- 172.21.15.70:0/310571329 learned_addr learned my addr 172.21.15.70:0/310571329 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.375+0000 7fb8f1073640 1 -- 172.21.15.70:0/310571329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8ec102990 msgr2=0x7fb8ec19e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.375+0000 7fb8f1073640 1 --2- 172.21.15.70:0/310571329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8ec102990 0x7fb8ec19e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.375+0000 7fb8f1073640 1 -- 172.21.15.70:0/310571329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8ec1017a0 msgr2=0x7fb8ec19df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.375+0000 7fb8f1073640 1 --2- 172.21.15.70:0/310571329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8ec1017a0 0x7fb8ec19df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.375+0000 7fb8f1073640 1 -- 172.21.15.70:0/310571329 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ec1a5b70 con 0x7fb8ec103350 2024-04-01T01:46:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.375+0000 7fb8f1073640 1 --2- 172.21.15.70:0/310571329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec103350 0x7fb8ec1a5500 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4009fd0 tx=0x7fb8e400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.376+0000 7fb8e1ffb640 1 -- 172.21.15.70:0/310571329 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e400ce20 con 0x7fb8ec103350 2024-04-01T01:46:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.376+0000 7fb8e1ffb640 1 -- 172.21.15.70:0/310571329 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8e4005020 con 0x7fb8ec103350 2024-04-01T01:46:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.376+0000 7fb8f2ad6640 1 -- 172.21.15.70:0/310571329 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8ec1a5e60 con 0x7fb8ec103350 2024-04-01T01:46:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.376+0000 7fb8e1ffb640 1 -- 172.21.15.70:0/310571329 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4010690 con 0x7fb8ec103350 2024-04-01T01:46:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.376+0000 7fb8f2ad6640 1 -- 172.21.15.70:0/310571329 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8ec1a6320 con 0x7fb8ec103350 2024-04-01T01:46:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.378+0000 7fb8e1ffb640 1 -- 172.21.15.70:0/310571329 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb8e40051d0 con 0x7fb8ec103350 2024-04-01T01:46:03.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.379+0000 7fb8e1ffb640 1 --2- 172.21.15.70:0/310571329 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8d40a25c0 0x7fb8d40a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.379+0000 7fb8f578a740 1 Processor -- start 2024-04-01T01:46:03.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.379+0000 7fb8f578a740 1 -- start start 2024-04-01T01:46:03.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.379+0000 7fb8f0872640 1 --2- 172.21.15.70:0/310571329 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8d40a25c0 0x7fb8d40a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.379+0000 7fb8e1ffb640 1 -- 172.21.15.70:0/310571329 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb8e40cf160 con 0x7fb8ec103350 2024-04-01T01:46:03.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.379+0000 7fb8f578a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db08d0 0x5581b2db0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.380 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8f578a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db12c0 0x5581b2db5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8f578a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db6430 0x5581b2db88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8f578a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5581b2d148f0 con 0x5581b2db12c0 2024-04-01T01:46:03.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8f578a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5581b2cbd8d0 con 0x5581b2db6430 2024-04-01T01:46:03.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8f578a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5581b2d10990 con 0x5581b2db08d0 2024-04-01T01:46:03.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8e3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db12c0 0x5581b2db5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8f1073640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db6430 0x5581b2db88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8e3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db12c0 0x5581b2db5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42848/0 (socket says 172.21.15.70:42848) 2024-04-01T01:46:03.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8e3fff640 1 -- 172.21.15.70:0/1871194947 learned_addr learned my addr 172.21.15.70:0/1871194947 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:03.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8f0872640 1 --2- 172.21.15.70:0/1871194947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db08d0 0x5581b2db0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8f1073640 1 -- 172.21.15.70:0/1871194947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db08d0 msgr2=0x5581b2db0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:03.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8f1073640 1 --2- 172.21.15.70:0/1871194947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db08d0 0x5581b2db0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8f1073640 1 -- 172.21.15.70:0/1871194947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db12c0 msgr2=0x5581b2db5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:03.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8f1073640 1 --2- 172.21.15.70:0/1871194947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db12c0 0x5581b2db5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8f1073640 1 -- 172.21.15.70:0/1871194947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581b2a29210 con 0x5581b2db6430 2024-04-01T01:46:03.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8e3fff640 1 --2- 172.21.15.70:0/1871194947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db12c0 0x5581b2db5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:46:03.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8f0872640 1 --2- 172.21.15.70:0/1871194947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db08d0 0x5581b2db0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:46:03.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8f0872640 1 --2- 172.21.15.70:0/310571329 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8d40a25c0 0x7fb8d40a4a80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8006010 tx=0x7fb8e8009660 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:03.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8f1073640 1 --2- 172.21.15.70:0/1871194947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db6430 0x5581b2db88c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40ccc00 tx=0x7fb8e4017490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29a9bc7ee68627bf server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:03.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8d2ffd640 1 -- 172.21.15.70:0/1871194947 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e40a4260 con 0x5581b2db6430 2024-04-01T01:46:03.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8d2ffd640 1 -- 172.21.15.70:0/1871194947 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8e4099b50 con 0x5581b2db6430 2024-04-01T01:46:03.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.380+0000 7fb8d2ffd640 1 -- 172.21.15.70:0/1871194947 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e409bcb0 con 0x5581b2db6430 2024-04-01T01:46:03.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.382+0000 7fb8f578a740 1 -- 172.21.15.70:0/1871194947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db6430 msgr2=0x5581b2db88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:03.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.382+0000 7fb8f578a740 1 --2- 172.21.15.70:0/1871194947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db6430 0x5581b2db88c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40ccc00 tx=0x7fb8e4017490 comp rx=0 tx=0).stop 2024-04-01T01:46:03.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.382+0000 7fb8f578a740 1 -- 172.21.15.70:0/1871194947 shutdown_connections 2024-04-01T01:46:03.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.382+0000 7fb8f578a740 1 --2- 172.21.15.70:0/1871194947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db6430 0x5581b2db88c0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.382+0000 7fb8f578a740 1 --2- 172.21.15.70:0/1871194947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db12c0 0x5581b2db5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.382+0000 7fb8f578a740 1 --2- 172.21.15.70:0/1871194947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db08d0 0x5581b2db0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.382+0000 7fb8f578a740 1 -- 172.21.15.70:0/1871194947 >> 172.21.15.70:0/1871194947 conn(0x5581b2dabe40 msgr2=0x5581b2dac240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:03.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.382+0000 7fb8f578a740 1 -- 172.21.15.70:0/1871194947 shutdown_connections 2024-04-01T01:46:03.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.382+0000 7fb8f578a740 1 -- 172.21.15.70:0/1871194947 wait complete. 2024-04-01T01:46:03.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.383+0000 7fb8f578a740 1 -- start start 2024-04-01T01:46:03.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.383+0000 7fb8f578a740 1 Processor -- start 2024-04-01T01:46:03.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.384+0000 7fb8f578a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db08d0 0x5581b2dbac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.384+0000 7fb8f578a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db12c0 0x5581b2dbb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.384+0000 7fb8f578a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db6430 0x5581b2dc2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.384+0000 7fb8f0872640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db08d0 0x5581b2dbac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.384+0000 7fb8f0872640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db08d0 0x5581b2dbac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:36570/0 (socket says 172.21.15.70:36570) 2024-04-01T01:46:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.384+0000 7fb8e3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db12c0 0x5581b2dbb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.384+0000 7fb8e3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db12c0 0x5581b2dbb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:52656/0 (socket says 172.21.15.70:52656) 2024-04-01T01:46:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.384+0000 7fb8f1073640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db6430 0x5581b2dc2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.384+0000 7fb8f1073640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db6430 0x5581b2dc2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42852/0 (socket says 172.21.15.70:42852) 2024-04-01T01:46:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.384+0000 7fb8f0872640 1 -- 172.21.15.70:0/2709265159 learned_addr learned my addr 172.21.15.70:0/2709265159 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.384+0000 7fb8f578a740 1 -- 172.21.15.70:0/2709265159 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5581b2db93e0 con 0x5581b2db6430 2024-04-01T01:46:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.384+0000 7fb8f578a740 1 -- 172.21.15.70:0/2709265159 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5581b2cb2f80 con 0x5581b2db12c0 2024-04-01T01:46:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.384+0000 7fb8f578a740 1 -- 172.21.15.70:0/2709265159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5581b2db9560 con 0x5581b2db08d0 2024-04-01T01:46:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.385+0000 7fb8f578a740 10 client.0 _ll_register_callbacks cb 0x5581b2d0b290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.385+0000 7fb8f0872640 1 -- 172.21.15.70:0/2709265159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db12c0 msgr2=0x5581b2dbb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.385+0000 7fb8f0872640 1 --2- 172.21.15.70:0/2709265159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db12c0 0x5581b2dbb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.387 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.385+0000 7fb8f0872640 1 -- 172.21.15.70:0/2709265159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db6430 msgr2=0x5581b2dc2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.385+0000 7fb8f0872640 1 --2- 172.21.15.70:0/2709265159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db6430 0x5581b2dc2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.385+0000 7fb8f0872640 1 -- 172.21.15.70:0/2709265159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581b2af5620 con 0x5581b2db08d0 2024-04-01T01:46:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.385+0000 7fb8e3fff640 1 --2- 172.21.15.70:0/2709265159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db12c0 0x5581b2dbb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:46:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.385+0000 7fb8f1073640 1 --2- 172.21.15.70:0/2709265159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db6430 0x5581b2dc2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:46:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.385+0000 7fb8f0872640 1 --2- 172.21.15.70:0/2709265159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db08d0 0x5581b2dbac00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8036040 tx=0x7fb8e802bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b5bce20e304d641 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.385+0000 7fb8cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:46:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.385+0000 7fb8cbfff640 1 -- 172.21.15.70:0/2709265159 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8050020 con 0x5581b2db08d0 2024-04-01T01:46:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.386+0000 7fb8cbfff640 1 -- 172.21.15.70:0/2709265159 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb8e8004380 con 0x5581b2db08d0 2024-04-01T01:46:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.386+0000 7fb8f578a740 1 -- 172.21.15.70:0/2709265159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8e4099b50 con 0x5581b2db08d0 2024-04-01T01:46:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.386+0000 7fb8cbfff640 1 -- 172.21.15.70:0/2709265159 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb8e805b050 con 0x5581b2db08d0 2024-04-01T01:46:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.386+0000 7fb8f578a740 10 client.28930 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.386+0000 7fb8f578a740 1 -- 172.21.15.70:0/2709265159 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5581b2dc5100 con 0x5581b2db08d0 2024-04-01T01:46:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.386+0000 7fb8cbfff640 1 -- 172.21.15.70:0/2709265159 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8e8047070 con 0x5581b2db08d0 2024-04-01T01:46:03.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.386+0000 7fb8cbfff640 1 -- 172.21.15.70:0/2709265159 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb8e8003880 con 0x5581b2db08d0 2024-04-01T01:46:03.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.386+0000 7fb8f578a740 10 client.28930 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:03.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.386+0000 7fb8f578a740 10 client.28930 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:03.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.386+0000 7fb8f578a740 20 client.28930 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:03.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.386+0000 7fb8f578a740 1 --2- 172.21.15.70:0/2709265159 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5581b2dc6890 0x5581b2dc8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.387+0000 7fb8f578a740 4 client.28930 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:03.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.387+0000 7fb8f578a740 1 -- 172.21.15.70:0/2709265159 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb8e4099b50 con 0x5581b2dc6890 2024-04-01T01:46:03.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.387+0000 7fb8e3fff640 1 --2- 172.21.15.70:0/2709265159 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5581b2dc6890 0x5581b2dc8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.387+0000 7fb8cbfff640 10 client.28930 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:03.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.387+0000 7fb8e3fff640 1 --2- 172.21.15.70:0/2709265159 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5581b2dc6890 0x5581b2dc8d00 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=302ebbe857aaff6e server_cookie=6863907f1fba7508 in_seq=0 out_seq=0 2024-04-01T01:46:03.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.388+0000 7fb8cbfff640 1 -- 172.21.15.70:0/2709265159 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fb8e4099b50 con 0x5581b2dc6890 2024-04-01T01:46:03.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.388+0000 7fb8cbfff640 10 client.28930 handle_command_reply: tid=0 2024-04-01T01:46:03.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.388+0000 7fb8f578a740 1 client.28930 shutdown 2024-04-01T01:46:03.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.388+0000 7fb8f578a740 1 -- 172.21.15.70:0/2709265159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db08d0 msgr2=0x5581b2dbac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:03.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.388+0000 7fb8f578a740 1 --2- 172.21.15.70:0/2709265159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db08d0 0x5581b2dbac00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8036040 tx=0x7fb8e802bc30 comp rx=0 tx=0).stop 2024-04-01T01:46:03.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.388+0000 7fb8f578a740 1 -- 172.21.15.70:0/2709265159 shutdown_connections 2024-04-01T01:46:03.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.388+0000 7fb8f578a740 1 --2- 172.21.15.70:0/2709265159 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5581b2dc6890 0x5581b2dc8d00 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.388+0000 7fb8cbfff640 0 client.28930 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:03.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.388+0000 7fb8f578a740 1 --2- 172.21.15.70:0/2709265159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db6430 0x5581b2dc2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.388+0000 7fb8f578a740 1 --2- 172.21.15.70:0/2709265159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db12c0 0x5581b2dbb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.388+0000 7fb8f578a740 1 --2- 172.21.15.70:0/2709265159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db08d0 0x5581b2dbac00 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.388+0000 7fb8f578a740 1 -- 172.21.15.70:0/2709265159 >> 172.21.15.70:0/2709265159 conn(0x5581b2dabe40 msgr2=0x5581b2daaf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:03.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.388+0000 7fb8f578a740 1 -- 172.21.15.70:0/2709265159 shutdown_connections 2024-04-01T01:46:03.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.388+0000 7fb8f578a740 1 -- 172.21.15.70:0/2709265159 wait complete. 2024-04-01T01:46:03.392 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.388+0000 7fb8f578a740 20 client.28930 trim_cache size 0 max 16384 2024-04-01T01:46:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.410+0000 7fb8f578a740 1 Processor -- start 2024-04-01T01:46:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.410+0000 7fb8f578a740 1 -- start start 2024-04-01T01:46:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.410+0000 7fb8f578a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db08d0 0x5581b2db1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.410+0000 7fb8f578a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db12c0 0x5581b2dc2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.410+0000 7fb8f578a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db6430 0x5581b2dce610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.410+0000 7fb8f578a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec1b08b0 con 0x5581b2db08d0 2024-04-01T01:46:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.410+0000 7fb8f578a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec1b23f0 con 0x5581b2db12c0 2024-04-01T01:46:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.410+0000 7fb8f578a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec1b0f80 con 0x5581b2db6430 2024-04-01T01:46:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.410+0000 7fb8f0872640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db08d0 0x5581b2db1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.410+0000 7fb8f0872640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db08d0 0x5581b2db1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42866/0 (socket says 172.21.15.70:42866) 2024-04-01T01:46:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.410+0000 7fb8e3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db12c0 0x5581b2dc2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.410+0000 7fb8f0872640 1 -- 172.21.15.70:0/1861357834 learned_addr learned my addr 172.21.15.70:0/1861357834 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.410+0000 7fb8f1073640 1 --2- 172.21.15.70:0/1861357834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db6430 0x5581b2dce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.411+0000 7fb8f0872640 1 -- 172.21.15.70:0/1861357834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db6430 msgr2=0x5581b2dce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.411+0000 7fb8f0872640 1 --2- 172.21.15.70:0/1861357834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db6430 0x5581b2dce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.411+0000 7fb8f0872640 1 -- 172.21.15.70:0/1861357834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db12c0 msgr2=0x5581b2dc2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.411+0000 7fb8f0872640 1 --2- 172.21.15.70:0/1861357834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db12c0 0x5581b2dc2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.411+0000 7fb8f0872640 1 -- 172.21.15.70:0/1861357834 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581b2da6170 con 0x5581b2db08d0 2024-04-01T01:46:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.411+0000 7fb8f0872640 1 --2- 172.21.15.70:0/1861357834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db08d0 0x5581b2db1d60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fb8e802c240 tx=0x7fb8e802c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56dfd59a27032ddf server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.411+0000 7fb8d27fc640 1 -- 172.21.15.70:0/1861357834 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8050040 con 0x5581b2db08d0 2024-04-01T01:46:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.411+0000 7fb8d27fc640 1 -- 172.21.15.70:0/1861357834 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8e8064070 con 0x5581b2db08d0 2024-04-01T01:46:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.411+0000 7fb8d27fc640 1 -- 172.21.15.70:0/1861357834 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8034a70 con 0x5581b2db08d0 2024-04-01T01:46:03.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.412+0000 7fb8f578a740 1 -- 172.21.15.70:0/1861357834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db08d0 msgr2=0x5581b2db1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.412+0000 7fb8f578a740 1 --2- 172.21.15.70:0/1861357834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db08d0 0x5581b2db1d60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fb8e802c240 tx=0x7fb8e802c7e0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.412+0000 7fb8f578a740 1 -- 172.21.15.70:0/1861357834 shutdown_connections 2024-04-01T01:46:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.413+0000 7fb8f578a740 1 --2- 172.21.15.70:0/1861357834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db6430 0x5581b2dce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.413+0000 7fb8f578a740 1 --2- 172.21.15.70:0/1861357834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db12c0 0x5581b2dc2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.413+0000 7fb8f578a740 1 --2- 172.21.15.70:0/1861357834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db08d0 0x5581b2db1d60 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.413+0000 7fb8f578a740 1 -- 172.21.15.70:0/1861357834 >> 172.21.15.70:0/1861357834 conn(0x5581b2dabe40 msgr2=0x5581b2dac220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.413+0000 7fb8f578a740 1 -- 172.21.15.70:0/1861357834 shutdown_connections 2024-04-01T01:46:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.413+0000 7fb8f578a740 1 -- 172.21.15.70:0/1861357834 wait complete. 2024-04-01T01:46:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.413+0000 7fb8f578a740 1 -- start start 2024-04-01T01:46:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.413+0000 7fb8f578a740 1 Processor -- start 2024-04-01T01:46:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f578a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db08d0 0x5581b2dbf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f578a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db12c0 0x5581b2dbff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f578a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db6430 0x5581b2dc0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f578a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec1b1d20 con 0x5581b2db12c0 2024-04-01T01:46:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f578a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec1b1650 con 0x5581b2db08d0 2024-04-01T01:46:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f578a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec1ae890 con 0x5581b2db6430 2024-04-01T01:46:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f0872640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db08d0 0x5581b2dbf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f578a740 10 client.0 _ll_register_callbacks cb 0x7fb8e40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f0872640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db08d0 0x5581b2dbf9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:52674/0 (socket says 172.21.15.70:52674) 2024-04-01T01:46:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f0872640 1 -- 172.21.15.70:0/695282087 learned_addr learned my addr 172.21.15.70:0/695282087 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8e3fff640 1 --2- 172.21.15.70:0/695282087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db12c0 0x5581b2dbff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f1073640 1 --2- 172.21.15.70:0/695282087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db6430 0x5581b2dc0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f0872640 1 -- 172.21.15.70:0/695282087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db6430 msgr2=0x5581b2dc0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f0872640 1 --2- 172.21.15.70:0/695282087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db6430 0x5581b2dc0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f0872640 1 -- 172.21.15.70:0/695282087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db12c0 msgr2=0x5581b2dbff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f0872640 1 --2- 172.21.15.70:0/695282087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db12c0 0x5581b2dbff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f0872640 1 -- 172.21.15.70:0/695282087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581b2dc1530 con 0x5581b2db08d0 2024-04-01T01:46:03.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.414+0000 7fb8f1073640 1 --2- 172.21.15.70:0/695282087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db6430 0x5581b2dc0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:46:03.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.415+0000 7fb8f0872640 1 --2- 172.21.15.70:0/695282087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db08d0 0x5581b2dbf9f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8003d70 tx=0x7fb8e8003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d76c4f7eb20899f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:03.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.415+0000 7fb8cbfff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:03.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.415+0000 7fb8cbfff640 1 -- 172.21.15.70:0/695282087 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e806c030 con 0x5581b2db08d0 2024-04-01T01:46:03.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.415+0000 7fb8cbfff640 1 -- 172.21.15.70:0/695282087 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb8e803a040 con 0x5581b2db08d0 2024-04-01T01:46:03.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.415+0000 7fb8f578a740 1 -- 172.21.15.70:0/695282087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8e8064070 con 0x5581b2db08d0 2024-04-01T01:46:03.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.415+0000 7fb8cbfff640 1 -- 172.21.15.70:0/695282087 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb8e8079050 con 0x5581b2db08d0 2024-04-01T01:46:03.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.415+0000 7fb8f578a740 10 client.29150 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:03.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.415+0000 7fb8f578a740 1 -- 172.21.15.70:0/695282087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5581b2e2cd60 con 0x5581b2db08d0 2024-04-01T01:46:03.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.416+0000 7fb8cbfff640 1 -- 172.21.15.70:0/695282087 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8e8047050 con 0x5581b2db08d0 2024-04-01T01:46:03.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.416+0000 7fb8cbfff640 1 -- 172.21.15.70:0/695282087 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb8e80046c0 con 0x5581b2db08d0 2024-04-01T01:46:03.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.416+0000 7fb8f578a740 10 client.29150 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:03.418 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.416+0000 7fb8f578a740 10 client.29150 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:03.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.416+0000 7fb8f578a740 20 client.29150 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:03.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.416+0000 7fb8f578a740 1 --2- 172.21.15.70:0/695282087 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5581b2e2d650 0x5581b2e2fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:03.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.416+0000 7fb8f578a740 4 client.29150 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:46:03.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.416+0000 7fb8f578a740 1 -- 172.21.15.70:0/695282087 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fb8e8064070 con 0x5581b2e2d650 2024-04-01T01:46:03.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.416+0000 7fb8e3fff640 1 --2- 172.21.15.70:0/695282087 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5581b2e2d650 0x5581b2e2fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:03.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.417+0000 7fb8cbfff640 10 client.29150 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:03.419 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.417+0000 7fb8e3fff640 1 --2- 172.21.15.70:0/695282087 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5581b2e2d650 0x5581b2e2fa70 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3da26b3dedd47c71 server_cookie=cef7776475a3907 in_seq=0 out_seq=0 2024-04-01T01:46:03.505 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.504+0000 7fb8cbfff640 1 -- 172.21.15.70:0/695282087 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fb8e8064070 con 0x5581b2e2d650 2024-04-01T01:46:03.505 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.504+0000 7fb8cbfff640 10 client.29150 handle_command_reply: tid=0 2024-04-01T01:46:03.505 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.505+0000 7fb8f578a740 1 client.29150 shutdown 2024-04-01T01:46:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.512+0000 7fb8f578a740 1 -- 172.21.15.70:0/695282087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db08d0 msgr2=0x5581b2dbf9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.512+0000 7fb8f578a740 1 --2- 172.21.15.70:0/695282087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db08d0 0x5581b2dbf9f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8003d70 tx=0x7fb8e8003820 comp rx=0 tx=0).stop 2024-04-01T01:46:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.516+0000 7fb8f578a740 1 -- 172.21.15.70:0/695282087 shutdown_connections 2024-04-01T01:46:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.516+0000 7fb8f578a740 1 --2- 172.21.15.70:0/695282087 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5581b2e2d650 0x5581b2e2fa70 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.516+0000 7fb8cbfff640 0 client.29150 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:03.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.516+0000 7fb8f578a740 1 --2- 172.21.15.70:0/695282087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b2db6430 0x5581b2dc0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.516+0000 7fb8f578a740 1 --2- 172.21.15.70:0/695282087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5581b2db12c0 0x5581b2dbff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.516+0000 7fb8f578a740 1 --2- 172.21.15.70:0/695282087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5581b2db08d0 0x5581b2dbf9f0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.516 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.516+0000 7fb8f578a740 1 -- 172.21.15.70:0/695282087 >> 172.21.15.70:0/695282087 conn(0x5581b2dabe40 msgr2=0x5581b2dba7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.517+0000 7fb8f578a740 1 -- 172.21.15.70:0/695282087 shutdown_connections 2024-04-01T01:46:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.517+0000 7fb8f578a740 1 -- 172.21.15.70:0/695282087 wait complete. 2024-04-01T01:46:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.518+0000 7fb8f578a740 20 client.29150 trim_cache size 0 max 16384 2024-04-01T01:46:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:46:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:46:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:46:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:46:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.522+0000 7fb8f2ad6640 1 -- 172.21.15.70:0/310571329 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8d40a25c0 msgr2=0x7fb8d40a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.522+0000 7fb8f2ad6640 1 --2- 172.21.15.70:0/310571329 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8d40a25c0 0x7fb8d40a4a80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8006010 tx=0x7fb8e8009660 comp rx=0 tx=0).stop 2024-04-01T01:46:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.522+0000 7fb8f2ad6640 1 -- 172.21.15.70:0/310571329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec103350 msgr2=0x7fb8ec1a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.522+0000 7fb8f2ad6640 1 --2- 172.21.15.70:0/310571329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec103350 0x7fb8ec1a5500 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4009fd0 tx=0x7fb8e400ef10 comp rx=0 tx=0).stop 2024-04-01T01:46:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.523+0000 7fb8f2ad6640 1 -- 172.21.15.70:0/310571329 shutdown_connections 2024-04-01T01:46:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.523+0000 7fb8f2ad6640 1 --2- 172.21.15.70:0/310571329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8ec103350 0x7fb8ec1a5500 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.523+0000 7fb8f2ad6640 1 --2- 172.21.15.70:0/310571329 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8d40a25c0 0x7fb8d40a4a80 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.523+0000 7fb8f2ad6640 1 --2- 172.21.15.70:0/310571329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8ec102990 0x7fb8ec19e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.523+0000 7fb8f2ad6640 1 --2- 172.21.15.70:0/310571329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8ec1017a0 0x7fb8ec19df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.523+0000 7fb8f2ad6640 1 -- 172.21.15.70:0/310571329 >> 172.21.15.70:0/310571329 conn(0x7fb8ec0fcd10 msgr2=0x7fb8ec10db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:03.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.524+0000 7fb8f2ad6640 1 -- 172.21.15.70:0/310571329 shutdown_connections 2024-04-01T01:46:03.524 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:03.524+0000 7fb8f2ad6640 1 -- 172.21.15.70:0/310571329 wait complete. 2024-04-01T01:46:04.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:04 smithi189 ceph-mon[35753]: pgmap v736: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.0 MiB/s wr, 804 op/s 2024-04-01T01:46:05.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:04 smithi070 ceph-mon[32179]: pgmap v736: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.0 MiB/s wr, 804 op/s 2024-04-01T01:46:05.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:04 smithi081 ceph-mon[35142]: pgmap v736: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.0 MiB/s wr, 804 op/s 2024-04-01T01:46:05.580 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:46:05.581 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:46:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.759+0000 7fcdcac3b640 1 -- 172.21.15.70:0/843278516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdc4108e40 msgr2=0x7fcdc41092c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.759+0000 7fcdcac3b640 1 --2- 172.21.15.70:0/843278516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdc4108e40 0x7fcdc41092c0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4009860 tx=0x7fcdb402bb90 comp rx=0 tx=0).stop 2024-04-01T01:46:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.759+0000 7fcdcac3b640 1 -- 172.21.15.70:0/843278516 shutdown_connections 2024-04-01T01:46:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.759+0000 7fcdcac3b640 1 --2- 172.21.15.70:0/843278516 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc4109800 0x7fcdc4110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.759+0000 7fcdcac3b640 1 --2- 172.21.15.70:0/843278516 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdc4108e40 0x7fcdc41092c0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.759+0000 7fcdcac3b640 1 --2- 172.21.15.70:0/843278516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4107c50 0x7fcdc4108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.759+0000 7fcdcac3b640 1 -- 172.21.15.70:0/843278516 >> 172.21.15.70:0/843278516 conn(0x7fcdc4076bf0 msgr2=0x7fcdc4079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.760+0000 7fcdcac3b640 1 -- 172.21.15.70:0/843278516 shutdown_connections 2024-04-01T01:46:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.760+0000 7fcdcac3b640 1 -- 172.21.15.70:0/843278516 wait complete. 2024-04-01T01:46:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.760+0000 7fcdcac3b640 1 Processor -- start 2024-04-01T01:46:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.761+0000 7fcdcac3b640 1 -- start start 2024-04-01T01:46:05.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.761+0000 7fcdcac3b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdc4107c50 0x7fcdc4070150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.761+0000 7fcdcac3b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4108e40 0x7fcdc4070690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.761+0000 7fcdc9c39640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdc4107c50 0x7fcdc4070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.761+0000 7fcdc9c39640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdc4107c50 0x7fcdc4070150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:52692/0 (socket says 172.21.15.70:52692) 2024-04-01T01:46:05.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.762+0000 7fcdc9c39640 1 -- 172.21.15.70:0/2060425297 learned_addr learned my addr 172.21.15.70:0/2060425297 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:05.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.762+0000 7fcdcac3b640 1 --2- 172.21.15.70:0/2060425297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc4109800 0x7fcdc41adee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.762+0000 7fcdc9438640 1 --2- 172.21.15.70:0/2060425297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4108e40 0x7fcdc4070690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.762+0000 7fcdcac3b640 1 -- 172.21.15.70:0/2060425297 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc41137d0 con 0x7fcdc4109800 2024-04-01T01:46:05.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.762+0000 7fcdcac3b640 1 -- 172.21.15.70:0/2060425297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc4113650 con 0x7fcdc4107c50 2024-04-01T01:46:05.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.762+0000 7fcdca43a640 1 --2- 172.21.15.70:0/2060425297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc4109800 0x7fcdc41adee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.762+0000 7fcdcac3b640 1 -- 172.21.15.70:0/2060425297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc4113950 con 0x7fcdc4108e40 2024-04-01T01:46:05.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.763+0000 7fcdca43a640 1 -- 172.21.15.70:0/2060425297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4108e40 msgr2=0x7fcdc4070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:05.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.763+0000 7fcdca43a640 1 --2- 172.21.15.70:0/2060425297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4108e40 0x7fcdc4070690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.763+0000 7fcdca43a640 1 -- 172.21.15.70:0/2060425297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdc4107c50 msgr2=0x7fcdc4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:05.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.763+0000 7fcdca43a640 1 --2- 172.21.15.70:0/2060425297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdc4107c50 0x7fcdc4070150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.763+0000 7fcdca43a640 1 -- 172.21.15.70:0/2060425297 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc41ae420 con 0x7fcdc4109800 2024-04-01T01:46:05.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.763+0000 7fcdca43a640 1 --2- 172.21.15.70:0/2060425297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc4109800 0x7fcdc41adee0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009fd0 tx=0x7fcdbc00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:05.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.763+0000 7fcdbaffd640 1 -- 172.21.15.70:0/2060425297 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc00ce20 con 0x7fcdc4109800 2024-04-01T01:46:05.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.763+0000 7fcdbaffd640 1 -- 172.21.15.70:0/2060425297 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdbc005020 con 0x7fcdc4109800 2024-04-01T01:46:05.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.764+0000 7fcdbaffd640 1 -- 172.21.15.70:0/2060425297 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc010690 con 0x7fcdc4109800 2024-04-01T01:46:05.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.764+0000 7fcdcac3b640 1 -- 172.21.15.70:0/2060425297 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc41ae5f0 con 0x7fcdc4109800 2024-04-01T01:46:05.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.764+0000 7fcdcac3b640 1 -- 172.21.15.70:0/2060425297 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc41aeb60 con 0x7fcdc4109800 2024-04-01T01:46:05.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.767+0000 7fcdbaffd640 1 -- 172.21.15.70:0/2060425297 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fcdbc0051d0 con 0x7fcdc4109800 2024-04-01T01:46:05.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.767+0000 7fcdcea84740 1 Processor -- start 2024-04-01T01:46:05.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.768+0000 7fcdcea84740 1 -- start start 2024-04-01T01:46:05.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.768+0000 7fcdcea84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cdc490 0x557e61cdc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.768+0000 7fcdcea84740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61cdce80 0x557e61ce1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.768+0000 7fcdcea84740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61ce1ff0 0x557e61ce4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.768+0000 7fcdcea84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557e61c41180 con 0x557e61cdc490 2024-04-01T01:46:05.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.768+0000 7fcdcea84740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x557e61be9b70 con 0x557e61ce1ff0 2024-04-01T01:46:05.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.768+0000 7fcdcea84740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557e61c3d220 con 0x557e61cdce80 2024-04-01T01:46:05.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.768+0000 7fcdbaffd640 1 --2- 172.21.15.70:0/2060425297 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcdb00a25f0 0x7fcdb00a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.769+0000 7fcdc9438640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61ce1ff0 0x557e61ce4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.769+0000 7fcdc9c39640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61cdce80 0x557e61ce1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.769+0000 7fcdc9438640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61ce1ff0 0x557e61ce4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:52704/0 (socket says 172.21.15.70:52704) 2024-04-01T01:46:05.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.769+0000 7fcdc9438640 1 -- 172.21.15.70:0/2617935619 learned_addr learned my addr 172.21.15.70:0/2617935619 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:05.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.769+0000 7fcdca43a640 1 --2- 172.21.15.70:0/2060425297 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcdb00a25f0 0x7fcdb00a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.769+0000 7fcdc9438640 1 --2- 172.21.15.70:0/2617935619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cdc490 0x557e61cdc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.769+0000 7fcdc9438640 1 -- 172.21.15.70:0/2617935619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61cdce80 msgr2=0x557e61ce1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:05.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.769+0000 7fcdc9438640 1 --2- 172.21.15.70:0/2617935619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61cdce80 0x557e61ce1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.769+0000 7fcdc9438640 1 -- 172.21.15.70:0/2617935619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cdc490 msgr2=0x557e61cdc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:05.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.769+0000 7fcdc9438640 1 --2- 172.21.15.70:0/2617935619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cdc490 0x557e61cdc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.769+0000 7fcdc9438640 1 -- 172.21.15.70:0/2617935619 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e61bfd7c0 con 0x557e61ce1ff0 2024-04-01T01:46:05.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.770+0000 7fcdc9438640 1 --2- 172.21.15.70:0/2617935619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61ce1ff0 0x557e61ce4480 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4005b60 tx=0x7fcdb4005a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ef8add834d86b81 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:05.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.770+0000 7fcdbaffd640 1 -- 172.21.15.70:0/2060425297 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fcdbc0cf170 con 0x7fcdc4109800 2024-04-01T01:46:05.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.770+0000 7fcdaffff640 1 -- 172.21.15.70:0/2617935619 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4047070 con 0x557e61ce1ff0 2024-04-01T01:46:05.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.770+0000 7fcdaffff640 1 -- 172.21.15.70:0/2617935619 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdb4035d80 con 0x557e61ce1ff0 2024-04-01T01:46:05.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.770+0000 7fcdaffff640 1 -- 172.21.15.70:0/2617935619 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4039040 con 0x557e61ce1ff0 2024-04-01T01:46:05.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.771+0000 7fcdcea84740 1 -- 172.21.15.70:0/2617935619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61ce1ff0 msgr2=0x557e61ce4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:05.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.771+0000 7fcdcea84740 1 --2- 172.21.15.70:0/2617935619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61ce1ff0 0x557e61ce4480 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4005b60 tx=0x7fcdb4005a90 comp rx=0 tx=0).stop 2024-04-01T01:46:05.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.772+0000 7fcdcea84740 1 -- 172.21.15.70:0/2617935619 shutdown_connections 2024-04-01T01:46:05.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.772+0000 7fcdcea84740 1 --2- 172.21.15.70:0/2617935619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61ce1ff0 0x557e61ce4480 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.772+0000 7fcdcea84740 1 --2- 172.21.15.70:0/2617935619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61cdce80 0x557e61ce1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.772+0000 7fcdcea84740 1 --2- 172.21.15.70:0/2617935619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cdc490 0x557e61cdc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.772+0000 7fcdcea84740 1 -- 172.21.15.70:0/2617935619 >> 172.21.15.70:0/2617935619 conn(0x557e61cd7a00 msgr2=0x557e61cd7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:05.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.772+0000 7fcdca43a640 1 --2- 172.21.15.70:0/2060425297 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcdb00a25f0 0x7fcdb00a4ab0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc00c760 tx=0x7fcdbc005f70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:05.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.772+0000 7fcdcea84740 1 -- 172.21.15.70:0/2617935619 shutdown_connections 2024-04-01T01:46:05.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.772+0000 7fcdcea84740 1 -- 172.21.15.70:0/2617935619 wait complete. 2024-04-01T01:46:05.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.773+0000 7fcdcea84740 1 -- start start 2024-04-01T01:46:05.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.773+0000 7fcdcea84740 1 Processor -- start 2024-04-01T01:46:05.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.773+0000 7fcdcea84740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61cdc490 0x557e61cf79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.774+0000 7fcdcea84740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61cdce80 0x557e61cf7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.774+0000 7fcdcea84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61ce1ff0 0x557e61cfef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.774+0000 7fcdc9438640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61cdc490 0x557e61cf79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.774+0000 7fcdc9438640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61cdc490 0x557e61cf79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:52714/0 (socket says 172.21.15.70:52714) 2024-04-01T01:46:05.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.774+0000 7fcdc9c39640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61cdce80 0x557e61cf7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.774+0000 7fcdc9438640 1 -- 172.21.15.70:0/3416885928 learned_addr learned my addr 172.21.15.70:0/3416885928 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:05.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.774+0000 7fcdcea84740 1 -- 172.21.15.70:0/3416885928 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557e61cf5f50 con 0x557e61ce1ff0 2024-04-01T01:46:05.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.774+0000 7fcdcea84740 1 -- 172.21.15.70:0/3416885928 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x557e61bdf4a0 con 0x557e61cdc490 2024-04-01T01:46:05.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.774+0000 7fcdcea84740 1 -- 172.21.15.70:0/3416885928 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557e61cf60d0 con 0x557e61cdce80 2024-04-01T01:46:05.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.774+0000 7fcdcea84740 10 client.0 _ll_register_callbacks cb 0x557e61be2d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.774+0000 7fcdc9438640 1 --2- 172.21.15.70:0/3416885928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61ce1ff0 0x557e61cfef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.775+0000 7fcdc9438640 1 -- 172.21.15.70:0/3416885928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61cdce80 msgr2=0x557e61cf7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.775+0000 7fcdc9438640 1 --2- 172.21.15.70:0/3416885928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61cdce80 0x557e61cf7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.775+0000 7fcdc9438640 1 -- 172.21.15.70:0/3416885928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61ce1ff0 msgr2=0x557e61cfef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.775+0000 7fcdc9438640 1 --2- 172.21.15.70:0/3416885928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61ce1ff0 0x557e61cfef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.775+0000 7fcdc9438640 1 -- 172.21.15.70:0/3416885928 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e61a1f7d0 con 0x557e61cdc490 2024-04-01T01:46:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.775+0000 7fcdc9438640 1 --2- 172.21.15.70:0/3416885928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61cdc490 0x557e61cf79f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4034920 tx=0x7fcdb402bb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76fc9e6a2a180c95 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.775+0000 7fcdad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.776+0000 7fcdad7fa640 1 -- 172.21.15.70:0/3416885928 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4039070 con 0x557e61cdc490 2024-04-01T01:46:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.776+0000 7fcdad7fa640 1 -- 172.21.15.70:0/3416885928 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fcdb402c730 con 0x557e61cdc490 2024-04-01T01:46:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.776+0000 7fcdcea84740 1 -- 172.21.15.70:0/3416885928 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdb4035d80 con 0x557e61cdc490 2024-04-01T01:46:05.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.776+0000 7fcdad7fa640 1 -- 172.21.15.70:0/3416885928 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcdb4047040 con 0x557e61cdc490 2024-04-01T01:46:05.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.777+0000 7fcdcea84740 10 client.29162 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:05.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.777+0000 7fcdcea84740 1 -- 172.21.15.70:0/3416885928 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557e619553f0 con 0x557e61cdc490 2024-04-01T01:46:05.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.777+0000 7fcdad7fa640 1 -- 172.21.15.70:0/3416885928 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdb4035d80 con 0x557e61cdc490 2024-04-01T01:46:05.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.777+0000 7fcdad7fa640 1 -- 172.21.15.70:0/3416885928 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fcdb404d070 con 0x557e61cdc490 2024-04-01T01:46:05.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.777+0000 7fcdcea84740 10 client.29162 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:05.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.777+0000 7fcdcea84740 10 client.29162 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:05.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.777+0000 7fcdcea84740 20 client.29162 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:05.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.777+0000 7fcdcea84740 1 --2- 172.21.15.70:0/3416885928 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557e61d03830 0x557e61d05cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.777+0000 7fcdcea84740 4 client.29162 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:05.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.777+0000 7fcdcea84740 1 -- 172.21.15.70:0/3416885928 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e61c4e6c0 con 0x557e61d03830 2024-04-01T01:46:05.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.778+0000 7fcdc9c39640 1 --2- 172.21.15.70:0/3416885928 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557e61d03830 0x557e61d05cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.778+0000 7fcdc9c39640 1 --2- 172.21.15.70:0/3416885928 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557e61d03830 0x557e61d05cd0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2da869a860c07657 server_cookie=de931d541e8c2cb9 in_seq=0 out_seq=0 2024-04-01T01:46:05.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.778+0000 7fcdad7fa640 10 client.29162 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:05.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.779+0000 7fcdad7fa640 1 -- 172.21.15.70:0/3416885928 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557e61c4e6c0 con 0x557e61d03830 2024-04-01T01:46:05.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.779+0000 7fcdad7fa640 10 client.29162 handle_command_reply: tid=0 2024-04-01T01:46:05.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.779+0000 7fcdcea84740 1 client.29162 shutdown 2024-04-01T01:46:05.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.780+0000 7fcdcea84740 1 -- 172.21.15.70:0/3416885928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61cdc490 msgr2=0x557e61cf79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:05.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.780+0000 7fcdcea84740 1 --2- 172.21.15.70:0/3416885928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61cdc490 0x557e61cf79f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4034920 tx=0x7fcdb402bb90 comp rx=0 tx=0).stop 2024-04-01T01:46:05.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.784+0000 7fcdcea84740 1 -- 172.21.15.70:0/3416885928 shutdown_connections 2024-04-01T01:46:05.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.784+0000 7fcdcea84740 1 --2- 172.21.15.70:0/3416885928 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557e61d03830 0x557e61d05cd0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.784+0000 7fcdcea84740 1 --2- 172.21.15.70:0/3416885928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61ce1ff0 0x557e61cfef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.784+0000 7fcdcea84740 1 --2- 172.21.15.70:0/3416885928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61cdce80 0x557e61cf7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.784+0000 7fcdcea84740 1 --2- 172.21.15.70:0/3416885928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61cdc490 0x557e61cf79f0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.784+0000 7fcdcea84740 1 -- 172.21.15.70:0/3416885928 >> 172.21.15.70:0/3416885928 conn(0x557e61cd7a00 msgr2=0x557e61cd6c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:05.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.786+0000 7fcdad7fa640 0 client.29162 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:05.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.786+0000 7fcdcea84740 1 -- 172.21.15.70:0/3416885928 shutdown_connections 2024-04-01T01:46:05.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.788+0000 7fcdcea84740 1 -- 172.21.15.70:0/3416885928 wait complete. 2024-04-01T01:46:05.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.788+0000 7fcdcea84740 20 client.29162 trim_cache size 0 max 16384 2024-04-01T01:46:05.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.824+0000 7fcdcea84740 1 Processor -- start 2024-04-01T01:46:05.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.824+0000 7fcdcea84740 1 -- start start 2024-04-01T01:46:05.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.824+0000 7fcdcea84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cdc490 0x557e61cd7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.824+0000 7fcdcea84740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61cdce80 0x557e61cfef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.824+0000 7fcdcea84740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61ce1ff0 0x557e61cf7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.824+0000 7fcdcea84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc41b07f0 con 0x557e61cdc490 2024-04-01T01:46:05.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.825+0000 7fcdcea84740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc41b2330 con 0x557e61ce1ff0 2024-04-01T01:46:05.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.825+0000 7fcdcea84740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc41b0ec0 con 0x557e61cdce80 2024-04-01T01:46:05.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.825+0000 7fcdc9438640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61ce1ff0 0x557e61cf7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.825+0000 7fcdc9438640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cdc490 0x557e61cd7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.825+0000 7fcdc9438640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cdc490 0x557e61cd7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42916/0 (socket says 172.21.15.70:42916) 2024-04-01T01:46:05.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.825+0000 7fcdc9438640 1 -- 172.21.15.70:0/993965480 learned_addr learned my addr 172.21.15.70:0/993965480 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:05.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.825+0000 7fcdc9438640 1 -- 172.21.15.70:0/993965480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61cdce80 msgr2=0x557e61cfef00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:05.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.825+0000 7fcdc9438640 1 --2- 172.21.15.70:0/993965480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61cdce80 0x557e61cfef00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.825+0000 7fcdc9438640 1 -- 172.21.15.70:0/993965480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61ce1ff0 msgr2=0x557e61cf7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:05.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.825+0000 7fcdc9438640 1 --2- 172.21.15.70:0/993965480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61ce1ff0 0x557e61cf7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.825+0000 7fcdc9438640 1 -- 172.21.15.70:0/993965480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e619150f0 con 0x557e61cdc490 2024-04-01T01:46:05.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.826+0000 7fcdc9438640 1 --2- 172.21.15.70:0/993965480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cdc490 0x557e61cd7de0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fcdb404fd30 tx=0x7fcdb4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557e4379d3084d8f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:05.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.826+0000 7fcdaf7fe640 1 -- 172.21.15.70:0/993965480 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4064070 con 0x557e61cdc490 2024-04-01T01:46:05.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.826+0000 7fcdaf7fe640 1 -- 172.21.15.70:0/993965480 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdb4034cf0 con 0x557e61cdc490 2024-04-01T01:46:05.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.827+0000 7fcdcea84740 1 -- 172.21.15.70:0/993965480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cdc490 msgr2=0x557e61cd7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:05.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.827+0000 7fcdcea84740 1 --2- 172.21.15.70:0/993965480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cdc490 0x557e61cd7de0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fcdb404fd30 tx=0x7fcdb4000bc0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.827+0000 7fcdcea84740 1 -- 172.21.15.70:0/993965480 shutdown_connections 2024-04-01T01:46:05.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.827+0000 7fcdcea84740 1 --2- 172.21.15.70:0/993965480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61ce1ff0 0x557e61cf7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.827+0000 7fcdcea84740 1 --2- 172.21.15.70:0/993965480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61cdce80 0x557e61cfef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.827+0000 7fcdcea84740 1 --2- 172.21.15.70:0/993965480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cdc490 0x557e61cd7de0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.827+0000 7fcdcea84740 1 -- 172.21.15.70:0/993965480 >> 172.21.15.70:0/993965480 conn(0x557e61cd7a00 msgr2=0x557e61cf7600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:05.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.830+0000 7fcdcea84740 1 -- 172.21.15.70:0/993965480 shutdown_connections 2024-04-01T01:46:05.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.831+0000 7fcdcea84740 1 -- 172.21.15.70:0/993965480 wait complete. 2024-04-01T01:46:05.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.832+0000 7fcdcea84740 1 -- start start 2024-04-01T01:46:05.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.832+0000 7fcdcea84740 1 Processor -- start 2024-04-01T01:46:05.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.832+0000 7fcdcea84740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61ce1ff0 0x557e61cfed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.832+0000 7fcdcea84740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61cfb7b0 0x557e61cfbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.832+0000 7fcdcea84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cfc170 0x557e61d67c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.832+0000 7fcdcea84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc41b1c60 con 0x557e61cfc170 2024-04-01T01:46:05.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.832+0000 7fcdcea84740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc41b1590 con 0x557e61cfb7b0 2024-04-01T01:46:05.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.832+0000 7fcdcea84740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc4082190 con 0x557e61ce1ff0 2024-04-01T01:46:05.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.832+0000 7fcdcea84740 10 client.0 _ll_register_callbacks cb 0x7fcdb4045070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:05.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.833+0000 7fcdc9c39640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61cfb7b0 0x557e61cfbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.833+0000 7fcdc9c39640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61cfb7b0 0x557e61cfbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:52722/0 (socket says 172.21.15.70:52722) 2024-04-01T01:46:05.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.833+0000 7fcdc9c39640 1 -- 172.21.15.70:0/4173144134 learned_addr learned my addr 172.21.15.70:0/4173144134 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:05.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.833+0000 7fcdc9438640 1 --2- 172.21.15.70:0/4173144134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61ce1ff0 0x557e61cfed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.833+0000 7fcdc9438640 1 --2- 172.21.15.70:0/4173144134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cfc170 0x557e61d67c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.833+0000 7fcdc9c39640 1 -- 172.21.15.70:0/4173144134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61ce1ff0 msgr2=0x557e61cfed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:05.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.833+0000 7fcdc9c39640 1 --2- 172.21.15.70:0/4173144134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61ce1ff0 0x557e61cfed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.833+0000 7fcdc9c39640 1 -- 172.21.15.70:0/4173144134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cfc170 msgr2=0x557e61d67c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:05.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.833+0000 7fcdc9c39640 1 --2- 172.21.15.70:0/4173144134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cfc170 0x557e61d67c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.833+0000 7fcdc9c39640 1 -- 172.21.15.70:0/4173144134 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e61d68270 con 0x557e61cfb7b0 2024-04-01T01:46:05.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.834+0000 7fcdc9c39640 1 --2- 172.21.15.70:0/4173144134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61cfb7b0 0x557e61cfbc30 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x557e61cd8970 tx=0x7fcdc0012820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3852c716a40672d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:05.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.834+0000 7fcdad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:05.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.835+0000 7fcdad7fa640 1 -- 172.21.15.70:0/4173144134 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc0004020 con 0x557e61cfb7b0 2024-04-01T01:46:05.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.835+0000 7fcdcea84740 1 -- 172.21.15.70:0/4173144134 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557e61c48a10 con 0x557e61cfb7b0 2024-04-01T01:46:05.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.835+0000 7fcdad7fa640 1 -- 172.21.15.70:0/4173144134 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fcdc00041c0 con 0x557e61cfb7b0 2024-04-01T01:46:05.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.835+0000 7fcdad7fa640 1 -- 172.21.15.70:0/4173144134 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcdc002d050 con 0x557e61cfb7b0 2024-04-01T01:46:05.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.835+0000 7fcdcea84740 10 client.29168 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:05.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.835+0000 7fcdcea84740 1 -- 172.21.15.70:0/4173144134 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557e61d696e0 con 0x557e61cfb7b0 2024-04-01T01:46:05.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.836+0000 7fcdad7fa640 1 -- 172.21.15.70:0/4173144134 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x557e61c48a10 con 0x557e61cfb7b0 2024-04-01T01:46:05.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.836+0000 7fcdad7fa640 1 -- 172.21.15.70:0/4173144134 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fcdc00035f0 con 0x557e61cfb7b0 2024-04-01T01:46:05.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.836+0000 7fcdcea84740 10 client.29168 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:05.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.836+0000 7fcdcea84740 10 client.29168 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:05.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.836+0000 7fcdcea84740 20 client.29168 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:05.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.836+0000 7fcdcea84740 1 --2- 172.21.15.70:0/4173144134 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557e61d6a0c0 0x557e61d6c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:05.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.836+0000 7fcdcea84740 4 client.29168 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:46:05.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.836+0000 7fcdcea84740 1 -- 172.21.15.70:0/4173144134 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557e61bc6540 con 0x557e61d6a0c0 2024-04-01T01:46:05.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.837+0000 7fcdc9438640 1 --2- 172.21.15.70:0/4173144134 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557e61d6a0c0 0x557e61d6c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:05.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.839+0000 7fcdad7fa640 10 client.29168 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:05.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.839+0000 7fcdc9438640 1 --2- 172.21.15.70:0/4173144134 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557e61d6a0c0 0x557e61d6c4e0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a47b1f0f8c7e9ee3 server_cookie=b5943bbf576f0b4d in_seq=0 out_seq=0 2024-04-01T01:46:05.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.841+0000 7fcdad7fa640 1 -- 172.21.15.70:0/4173144134 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557e61bc6540 con 0x557e61d6a0c0 2024-04-01T01:46:05.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.841+0000 7fcdad7fa640 10 client.29168 handle_command_reply: tid=0 2024-04-01T01:46:05.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.841+0000 7fcdcea84740 1 client.29168 shutdown 2024-04-01T01:46:05.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.841+0000 7fcdcea84740 1 -- 172.21.15.70:0/4173144134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61cfb7b0 msgr2=0x557e61cfbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:05.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.841+0000 7fcdcea84740 1 --2- 172.21.15.70:0/4173144134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61cfb7b0 0x557e61cfbc30 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x557e61cd8970 tx=0x7fcdc0012820 comp rx=0 tx=0).stop 2024-04-01T01:46:05.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.843+0000 7fcdcea84740 1 -- 172.21.15.70:0/4173144134 shutdown_connections 2024-04-01T01:46:05.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.843+0000 7fcdcea84740 1 --2- 172.21.15.70:0/4173144134 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x557e61d6a0c0 0x557e61d6c4e0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.843+0000 7fcdcea84740 1 --2- 172.21.15.70:0/4173144134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557e61cfc170 0x557e61d67c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.843+0000 7fcdcea84740 1 --2- 172.21.15.70:0/4173144134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557e61cfb7b0 0x557e61cfbc30 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.843+0000 7fcdcea84740 1 --2- 172.21.15.70:0/4173144134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557e61ce1ff0 0x557e61cfed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.843+0000 7fcdcea84740 1 -- 172.21.15.70:0/4173144134 >> 172.21.15.70:0/4173144134 conn(0x557e61cd7a00 msgr2=0x557e61cdac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:05.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.843+0000 7fcdad7fa640 0 client.29168 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:05.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.844+0000 7fcdcea84740 1 -- 172.21.15.70:0/4173144134 shutdown_connections 2024-04-01T01:46:05.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.844+0000 7fcdcea84740 1 -- 172.21.15.70:0/4173144134 wait complete. 2024-04-01T01:46:05.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.844+0000 7fcdcea84740 20 client.29168 trim_cache size 0 max 16384 2024-04-01T01:46:05.844 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:46:05.844 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:46:05.844 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:46:05.844 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:46:05.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.845+0000 7fcdcac3b640 1 -- 172.21.15.70:0/2060425297 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcdb00a25f0 msgr2=0x7fcdb00a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:05.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.845+0000 7fcdcac3b640 1 --2- 172.21.15.70:0/2060425297 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcdb00a25f0 0x7fcdb00a4ab0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc00c760 tx=0x7fcdbc005f70 comp rx=0 tx=0).stop 2024-04-01T01:46:05.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.845+0000 7fcdcac3b640 1 -- 172.21.15.70:0/2060425297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc4109800 msgr2=0x7fcdc41adee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:05.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.845+0000 7fcdcac3b640 1 --2- 172.21.15.70:0/2060425297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc4109800 0x7fcdc41adee0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009fd0 tx=0x7fcdbc00ef10 comp rx=0 tx=0).stop 2024-04-01T01:46:05.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.846+0000 7fcdcac3b640 1 -- 172.21.15.70:0/2060425297 shutdown_connections 2024-04-01T01:46:05.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.846+0000 7fcdcac3b640 1 --2- 172.21.15.70:0/2060425297 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdc4109800 0x7fcdc41adee0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.846+0000 7fcdcac3b640 1 --2- 172.21.15.70:0/2060425297 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcdb00a25f0 0x7fcdb00a4ab0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.846+0000 7fcdcac3b640 1 --2- 172.21.15.70:0/2060425297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdc4108e40 0x7fcdc4070690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.846+0000 7fcdcac3b640 1 --2- 172.21.15.70:0/2060425297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdc4107c50 0x7fcdc4070150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:05.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.846+0000 7fcdcac3b640 1 -- 172.21.15.70:0/2060425297 >> 172.21.15.70:0/2060425297 conn(0x7fcdc4076bf0 msgr2=0x7fcdc410c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:05.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.846+0000 7fcdcac3b640 1 -- 172.21.15.70:0/2060425297 shutdown_connections 2024-04-01T01:46:05.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:05.847+0000 7fcdcac3b640 1 -- 172.21.15.70:0/2060425297 wait complete. 2024-04-01T01:46:05.867 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:46:05.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:46:06.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.039+0000 7fd5d2ffd640 1 -- 172.21.15.70:0/2299858654 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8004c00 con 0x7fd5cc099aa0 2024-04-01T01:46:06.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.040+0000 7fd5d91ba640 1 -- 172.21.15.70:0/2299858654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5cc099aa0 msgr2=0x7fd5cc0a03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.040+0000 7fd5d91ba640 1 --2- 172.21.15.70:0/2299858654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5cc099aa0 0x7fd5cc0a03c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd5c801a910 tx=0x7fd5c803cc80 comp rx=0 tx=0).stop 2024-04-01T01:46:06.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.040+0000 7fd5d91ba640 1 -- 172.21.15.70:0/2299858654 shutdown_connections 2024-04-01T01:46:06.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.040+0000 7fd5d91ba640 1 --2- 172.21.15.70:0/2299858654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5cc099aa0 0x7fd5cc0a03c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.040+0000 7fd5d91ba640 1 --2- 172.21.15.70:0/2299858654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5cc0990e0 0x7fd5cc099560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.040+0000 7fd5d91ba640 1 --2- 172.21.15.70:0/2299858654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5cc097f10 0x7fd5cc098350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.040+0000 7fd5d91ba640 1 -- 172.21.15.70:0/2299858654 >> 172.21.15.70:0/2299858654 conn(0x7fd5cc0934d0 msgr2=0x7fd5cc095910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:06.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.040+0000 7fd5d91ba640 1 -- 172.21.15.70:0/2299858654 shutdown_connections 2024-04-01T01:46:06.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.040+0000 7fd5d91ba640 1 -- 172.21.15.70:0/2299858654 wait complete. 2024-04-01T01:46:06.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.041+0000 7fd5d91ba640 1 Processor -- start 2024-04-01T01:46:06.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.041+0000 7fd5d91ba640 1 -- start start 2024-04-01T01:46:06.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.041+0000 7fd5d91ba640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5cc097f10 0x7fd5cc12e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.041+0000 7fd5d91ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5cc0990e0 0x7fd5cc12e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.041+0000 7fd5d91ba640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5cc099aa0 0x7fd5cc157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.041+0000 7fd5d91ba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc0a3b50 con 0x7fd5cc0990e0 2024-04-01T01:46:06.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.041+0000 7fd5d91ba640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc0a39d0 con 0x7fd5cc097f10 2024-04-01T01:46:06.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.041+0000 7fd5d91ba640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd5cc0a3cd0 con 0x7fd5cc099aa0 2024-04-01T01:46:06.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.042+0000 7fd5d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5cc097f10 0x7fd5cc12e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.042+0000 7fd5d89b9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5cc099aa0 0x7fd5cc157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.042+0000 7fd5d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5cc097f10 0x7fd5cc12e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:52734/0 (socket says 172.21.15.70:52734) 2024-04-01T01:46:06.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.042+0000 7fd5d3fff640 1 -- 172.21.15.70:0/1605232859 learned_addr learned my addr 172.21.15.70:0/1605232859 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:06.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.042+0000 7fd5d37fe640 1 --2- 172.21.15.70:0/1605232859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5cc0990e0 0x7fd5cc12e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.042+0000 7fd5d3fff640 1 -- 172.21.15.70:0/1605232859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5cc099aa0 msgr2=0x7fd5cc157740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.042+0000 7fd5d3fff640 1 --2- 172.21.15.70:0/1605232859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5cc099aa0 0x7fd5cc157740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.042+0000 7fd5d3fff640 1 -- 172.21.15.70:0/1605232859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5cc0990e0 msgr2=0x7fd5cc12e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.042+0000 7fd5d3fff640 1 --2- 172.21.15.70:0/1605232859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5cc0990e0 0x7fd5cc12e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.042+0000 7fd5d3fff640 1 -- 172.21.15.70:0/1605232859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5cc157de0 con 0x7fd5cc097f10 2024-04-01T01:46:06.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.042+0000 7fd5d3fff640 1 --2- 172.21.15.70:0/1605232859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5cc097f10 0x7fd5cc12e140 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd5d404ff30 tx=0x7fd5d406d430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:06.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.043+0000 7fd5d17fa640 1 -- 172.21.15.70:0/1605232859 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4085070 con 0x7fd5cc097f10 2024-04-01T01:46:06.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.043+0000 7fd5d17fa640 1 -- 172.21.15.70:0/1605232859 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5d406de40 con 0x7fd5cc097f10 2024-04-01T01:46:06.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.043+0000 7fd5d17fa640 1 -- 172.21.15.70:0/1605232859 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4080d70 con 0x7fd5cc097f10 2024-04-01T01:46:06.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.043+0000 7fd5d91ba640 1 -- 172.21.15.70:0/1605232859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5cc009c10 con 0x7fd5cc097f10 2024-04-01T01:46:06.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.043+0000 7fd5d91ba640 1 -- 172.21.15.70:0/1605232859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5cc00a110 con 0x7fd5cc097f10 2024-04-01T01:46:06.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.044+0000 7fd5d91ba640 1 -- 172.21.15.70:0/1605232859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5cc09f2a0 con 0x7fd5cc097f10 2024-04-01T01:46:06.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.046+0000 7fd5d17fa640 1 -- 172.21.15.70:0/1605232859 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fd5d4088070 con 0x7fd5cc097f10 2024-04-01T01:46:06.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.046+0000 7fd5d17fa640 1 --2- 172.21.15.70:0/1605232859 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5c40a25f0 0x7fd5c40a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.046+0000 7fd5d17fa640 1 -- 172.21.15.70:0/1605232859 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd5d4139ea0 con 0x7fd5cc097f10 2024-04-01T01:46:06.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.047+0000 7fd5d37fe640 1 --2- 172.21.15.70:0/1605232859 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5c40a25f0 0x7fd5c40a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.049+0000 7fd5d37fe640 1 --2- 172.21.15.70:0/1605232859 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5c40a25f0 0x7fd5c40a4ab0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc12f260 tx=0x7fd5c0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:06.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.052+0000 7fd5d17fa640 1 -- 172.21.15.70:0/1605232859 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd5d408f070 con 0x7fd5cc097f10 2024-04-01T01:46:06.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.450+0000 7fd5d91ba640 1 -- 172.21.15.70:0/1605232859 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5cc0022f0 con 0x7fd5cc097f10 2024-04-01T01:46:06.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.453+0000 7fd5d17fa640 1 -- 172.21.15.70:0/1605232859 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7fd5d41065e0 con 0x7fd5cc097f10 2024-04-01T01:46:06.455 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:46:06.455 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:46:06.455 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:46:06.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.459+0000 7fd5d91ba640 1 -- 172.21.15.70:0/1605232859 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5c40a25f0 msgr2=0x7fd5c40a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.459+0000 7fd5d91ba640 1 --2- 172.21.15.70:0/1605232859 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5c40a25f0 0x7fd5c40a4ab0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc12f260 tx=0x7fd5c0005f90 comp rx=0 tx=0).stop 2024-04-01T01:46:06.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.459+0000 7fd5d91ba640 1 -- 172.21.15.70:0/1605232859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5cc097f10 msgr2=0x7fd5cc12e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.459+0000 7fd5d91ba640 1 --2- 172.21.15.70:0/1605232859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5cc097f10 0x7fd5cc12e140 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd5d404ff30 tx=0x7fd5d406d430 comp rx=0 tx=0).stop 2024-04-01T01:46:06.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.459+0000 7fd5d91ba640 1 -- 172.21.15.70:0/1605232859 shutdown_connections 2024-04-01T01:46:06.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.459+0000 7fd5d91ba640 1 --2- 172.21.15.70:0/1605232859 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd5c40a25f0 0x7fd5c40a4ab0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.459+0000 7fd5d91ba640 1 --2- 172.21.15.70:0/1605232859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5cc099aa0 0x7fd5cc157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.459+0000 7fd5d91ba640 1 --2- 172.21.15.70:0/1605232859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5cc0990e0 0x7fd5cc12e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.459+0000 7fd5d91ba640 1 --2- 172.21.15.70:0/1605232859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5cc097f10 0x7fd5cc12e140 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.459+0000 7fd5d91ba640 1 -- 172.21.15.70:0/1605232859 >> 172.21.15.70:0/1605232859 conn(0x7fd5cc0934d0 msgr2=0x7fd5cc09c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:06.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.459+0000 7fd5d91ba640 1 -- 172.21.15.70:0/1605232859 shutdown_connections 2024-04-01T01:46:06.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.460+0000 7fd5d91ba640 1 -- 172.21.15.70:0/1605232859 wait complete. 2024-04-01T01:46:06.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:46:06.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.646+0000 7efda27ac640 1 -- 172.21.15.70:0/866044432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd9c06b710 msgr2=0x7efd9c06bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.646+0000 7efda27ac640 1 --2- 172.21.15.70:0/866044432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd9c06b710 0x7efd9c06bb30 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7efd90009900 tx=0x7efd9004dc40 comp rx=0 tx=0).stop 2024-04-01T01:46:06.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.646+0000 7efda27ac640 1 -- 172.21.15.70:0/866044432 shutdown_connections 2024-04-01T01:46:06.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.646+0000 7efda27ac640 1 --2- 172.21.15.70:0/866044432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd9c06d2f0 0x7efd9c071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.647+0000 7efda27ac640 1 --2- 172.21.15.70:0/866044432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd9c06c950 0x7efd9c06cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.647+0000 7efda27ac640 1 --2- 172.21.15.70:0/866044432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd9c06b710 0x7efd9c06bb30 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.647+0000 7efda27ac640 1 -- 172.21.15.70:0/866044432 >> 172.21.15.70:0/866044432 conn(0x7efd9c0fbbe0 msgr2=0x7efd9c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:06.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.647+0000 7efda27ac640 1 -- 172.21.15.70:0/866044432 shutdown_connections 2024-04-01T01:46:06.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.647+0000 7efda27ac640 1 -- 172.21.15.70:0/866044432 wait complete. 2024-04-01T01:46:06.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.647+0000 7efda27ac640 1 Processor -- start 2024-04-01T01:46:06.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.648+0000 7efda27ac640 1 -- start start 2024-04-01T01:46:06.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.648+0000 7efda27ac640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd9c06b710 0x7efd9c18cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.648+0000 7efda27ac640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd9c06c950 0x7efd9c18d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.648+0000 7efda27ac640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd9c06d2f0 0x7efd9c194570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.648+0000 7efda27ac640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efd9c100890 con 0x7efd9c06b710 2024-04-01T01:46:06.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.648+0000 7efda27ac640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efd9c100710 con 0x7efd9c06c950 2024-04-01T01:46:06.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.648+0000 7efda27ac640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efd9c100a10 con 0x7efd9c06d2f0 2024-04-01T01:46:06.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.648+0000 7efda0d49640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd9c06d2f0 0x7efd9c194570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.648+0000 7efda0d49640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd9c06d2f0 0x7efd9c194570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:35448/0 (socket says 172.21.15.70:35448) 2024-04-01T01:46:06.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.648+0000 7efda0d49640 1 -- 172.21.15.70:0/1165960385 learned_addr learned my addr 172.21.15.70:0/1165960385 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:06.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.648+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/1165960385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd9c06c950 0x7efd9c18d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.649+0000 7efda0d49640 1 -- 172.21.15.70:0/1165960385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd9c06c950 msgr2=0x7efd9c18d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.649+0000 7efda0d49640 1 --2- 172.21.15.70:0/1165960385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd9c06c950 0x7efd9c18d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.649+0000 7efda0d49640 1 -- 172.21.15.70:0/1165960385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd9c06b710 msgr2=0x7efd9c18cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.649+0000 7efda0d49640 1 --2- 172.21.15.70:0/1165960385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd9c06b710 0x7efd9c18cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.649+0000 7efda0d49640 1 -- 172.21.15.70:0/1165960385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd9c194be0 con 0x7efd9c06d2f0 2024-04-01T01:46:06.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.649+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/1165960385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd9c06c950 0x7efd9c18d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:46:06.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.649+0000 7efda0d49640 1 --2- 172.21.15.70:0/1165960385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd9c06d2f0 0x7efd9c194570 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7efd8c00d750 tx=0x7efd8c00dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:06.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.649+0000 7efd997fa640 1 -- 172.21.15.70:0/1165960385 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8c017070 con 0x7efd9c06d2f0 2024-04-01T01:46:06.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.650+0000 7efd997fa640 1 -- 172.21.15.70:0/1165960385 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd8c0054d0 con 0x7efd9c06d2f0 2024-04-01T01:46:06.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.650+0000 7efd997fa640 1 -- 172.21.15.70:0/1165960385 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8c004c30 con 0x7efd9c06d2f0 2024-04-01T01:46:06.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.650+0000 7efda27ac640 1 -- 172.21.15.70:0/1165960385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd9c194ed0 con 0x7efd9c06d2f0 2024-04-01T01:46:06.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.650+0000 7efda27ac640 1 -- 172.21.15.70:0/1165960385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd9c19db80 con 0x7efd9c06d2f0 2024-04-01T01:46:06.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.655+0000 7efda538f740 1 Processor -- start 2024-04-01T01:46:06.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.655+0000 7efd997fa640 1 -- 172.21.15.70:0/1165960385 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7efd8c004dd0 con 0x7efd9c06d2f0 2024-04-01T01:46:06.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.656+0000 7efda538f740 1 -- start start 2024-04-01T01:46:06.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.656+0000 7efda538f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453cc490 0x556f453cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.656+0000 7efda538f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cce80 0x556f453d1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.656+0000 7efda538f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453d1ff0 0x556f453d4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.656+0000 7efda538f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556f453311b0 con 0x556f453cce80 2024-04-01T01:46:06.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.656+0000 7efda538f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x556f452d9c00 con 0x556f453d1ff0 2024-04-01T01:46:06.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.656+0000 7efda538f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556f4532d250 con 0x556f453cc490 2024-04-01T01:46:06.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.656+0000 7efd9b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453d1ff0 0x556f453d4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.656+0000 7efd9b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453d1ff0 0x556f453d4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:45030/0 (socket says 172.21.15.70:45030) 2024-04-01T01:46:06.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.656+0000 7efd9b7fe640 1 -- 172.21.15.70:0/2241142252 learned_addr learned my addr 172.21.15.70:0/2241142252 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:06.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.656+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/2241142252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453cc490 0x556f453cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.657+0000 7efd997fa640 1 --2- 172.21.15.70:0/1165960385 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efd840a2620 0x7efd840a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.657+0000 7efd9b7fe640 1 -- 172.21.15.70:0/2241142252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453cc490 msgr2=0x556f453cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:06.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.657+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/2241142252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453cc490 0x556f453cc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.657+0000 7efd9b7fe640 1 -- 172.21.15.70:0/2241142252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cce80 msgr2=0x556f453d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:06.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.657+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/2241142252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cce80 0x556f453d1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.657+0000 7efd9b7fe640 1 -- 172.21.15.70:0/2241142252 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f452f93d0 con 0x556f453d1ff0 2024-04-01T01:46:06.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.657+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/2241142252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453d1ff0 0x556f453d4480 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7efd940128d0 tx=0x7efd94009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24f9f612192d006e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:06.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.657+0000 7efd997fa640 1 -- 172.21.15.70:0/1165960385 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7efd8c0971e0 con 0x7efd9c06d2f0 2024-04-01T01:46:06.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.658+0000 7efda0d49640 1 --2- 172.21.15.70:0/1165960385 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efd840a2620 0x7efd840a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.658+0000 7efd827fc640 1 -- 172.21.15.70:0/2241142252 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd94004d20 con 0x556f453d1ff0 2024-04-01T01:46:06.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.658+0000 7efd827fc640 1 -- 172.21.15.70:0/2241142252 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd94010d20 con 0x556f453d1ff0 2024-04-01T01:46:06.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.658+0000 7efd827fc640 1 -- 172.21.15.70:0/2241142252 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd940057d0 con 0x556f453d1ff0 2024-04-01T01:46:06.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.659+0000 7efda0d49640 1 --2- 172.21.15.70:0/1165960385 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efd840a2620 0x7efd840a4ae0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efd8c0cbbb0 tx=0x7efd8c004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:06.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.663+0000 7efda538f740 1 -- 172.21.15.70:0/2241142252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453d1ff0 msgr2=0x556f453d4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.663+0000 7efda538f740 1 --2- 172.21.15.70:0/2241142252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453d1ff0 0x556f453d4480 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7efd940128d0 tx=0x7efd94009820 comp rx=0 tx=0).stop 2024-04-01T01:46:06.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.664+0000 7efda538f740 1 -- 172.21.15.70:0/2241142252 shutdown_connections 2024-04-01T01:46:06.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.664+0000 7efda538f740 1 --2- 172.21.15.70:0/2241142252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453d1ff0 0x556f453d4480 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.664+0000 7efda538f740 1 --2- 172.21.15.70:0/2241142252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cce80 0x556f453d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.664+0000 7efda538f740 1 --2- 172.21.15.70:0/2241142252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453cc490 0x556f453cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.664+0000 7efda538f740 1 -- 172.21.15.70:0/2241142252 >> 172.21.15.70:0/2241142252 conn(0x556f453c7a00 msgr2=0x556f453c7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:06.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.664+0000 7efda538f740 1 -- 172.21.15.70:0/2241142252 shutdown_connections 2024-04-01T01:46:06.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.664+0000 7efda538f740 1 -- 172.21.15.70:0/2241142252 wait complete. 2024-04-01T01:46:06.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.664+0000 7efda538f740 1 -- start start 2024-04-01T01:46:06.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.664+0000 7efda538f740 1 Processor -- start 2024-04-01T01:46:06.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.664+0000 7efda538f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cc490 0x556f453d6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.664+0000 7efda538f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 0x556f453d6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.664+0000 7efda538f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453d1ff0 0x556f453de010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.664+0000 7efda538f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556f453d4fa0 con 0x556f453cc490 2024-04-01T01:46:06.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.664+0000 7efda538f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x556f452cf470 con 0x556f453cce80 2024-04-01T01:46:06.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.664+0000 7efda538f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556f453d5120 con 0x556f453d1ff0 2024-04-01T01:46:06.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.665+0000 7efda538f740 10 client.0 _ll_register_callbacks cb 0x556f452eccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:06.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.665+0000 7efd9b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cc490 0x556f453d6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.665+0000 7efd9b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cc490 0x556f453d6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56670/0 (socket says 172.21.15.70:56670) 2024-04-01T01:46:06.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.665+0000 7efd9b7fe640 1 -- 172.21.15.70:0/2252976980 learned_addr learned my addr 172.21.15.70:0/2252976980 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:06.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.665+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/2252976980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453d1ff0 0x556f453de010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.665+0000 7efd9bfff640 1 --2- 172.21.15.70:0/2252976980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 0x556f453d6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.666+0000 7efd9b7fe640 1 -- 172.21.15.70:0/2252976980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 msgr2=0x556f453d6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:06.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.666+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/2252976980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 0x556f453d6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.666+0000 7efd9b7fe640 1 -- 172.21.15.70:0/2252976980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cc490 msgr2=0x556f453d6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:06.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.666+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/2252976980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cc490 0x556f453d6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.666+0000 7efd9b7fe640 1 -- 172.21.15.70:0/2252976980 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f450453f0 con 0x556f453d1ff0 2024-04-01T01:46:06.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.666+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/2252976980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453d1ff0 0x556f453de010 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7efd94011010 tx=0x7efd94005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60c0157b4285d20a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:06.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.666+0000 7efd7b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:46:06.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.666+0000 7efd7b7fe640 1 -- 172.21.15.70:0/2252976980 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd94004a40 con 0x556f453d1ff0 2024-04-01T01:46:06.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.666+0000 7efda538f740 1 -- 172.21.15.70:0/2252976980 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd94010d20 con 0x556f453d1ff0 2024-04-01T01:46:06.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.667+0000 7efd7b7fe640 1 -- 172.21.15.70:0/2252976980 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7efd94004be0 con 0x556f453d1ff0 2024-04-01T01:46:06.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.667+0000 7efd7b7fe640 1 -- 172.21.15.70:0/2252976980 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efd94004de0 con 0x556f453d1ff0 2024-04-01T01:46:06.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.667+0000 7efda538f740 10 client.28966 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:06.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.667+0000 7efda538f740 1 -- 172.21.15.70:0/2252976980 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556f4510f7d0 con 0x556f453d1ff0 2024-04-01T01:46:06.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.669+0000 7efd7b7fe640 1 -- 172.21.15.70:0/2252976980 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd94010d20 con 0x556f453d1ff0 2024-04-01T01:46:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.670+0000 7efd7b7fe640 1 -- 172.21.15.70:0/2252976980 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7efd9401c030 con 0x556f453d1ff0 2024-04-01T01:46:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.670+0000 7efda538f740 10 client.28966 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.670+0000 7efda538f740 10 client.28966 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:46:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.670+0000 7efda538f740 20 client.28966 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.670+0000 7efda538f740 1 --2- 172.21.15.70:0/2252976980 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x556f453e22f0 0x556f453e46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.670+0000 7efda538f740 4 client.28966 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.670+0000 7efda538f740 1 -- 172.21.15.70:0/2252976980 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f4533e6f0 con 0x556f453e22f0 2024-04-01T01:46:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.671+0000 7efd9bfff640 1 --2- 172.21.15.70:0/2252976980 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x556f453e22f0 0x556f453e46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.672+0000 7efd9bfff640 1 --2- 172.21.15.70:0/2252976980 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x556f453e22f0 0x556f453e46e0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ba6306c00d16ffe0 server_cookie=8edee62edc3af4ae in_seq=0 out_seq=0 2024-04-01T01:46:06.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.672+0000 7efd7b7fe640 10 client.28966 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:46:06.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.673+0000 7efd7b7fe640 1 -- 172.21.15.70:0/2252976980 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556f4533e6f0 con 0x556f453e22f0 2024-04-01T01:46:06.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.673+0000 7efd7b7fe640 10 client.28966 handle_command_reply: tid=0 2024-04-01T01:46:06.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.673+0000 7efda538f740 1 client.28966 shutdown 2024-04-01T01:46:06.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.674+0000 7efda538f740 1 -- 172.21.15.70:0/2252976980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453d1ff0 msgr2=0x556f453de010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.674+0000 7efda538f740 1 --2- 172.21.15.70:0/2252976980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453d1ff0 0x556f453de010 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7efd94011010 tx=0x7efd94005fd0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.674+0000 7efda538f740 1 -- 172.21.15.70:0/2252976980 shutdown_connections 2024-04-01T01:46:06.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.674+0000 7efda538f740 1 --2- 172.21.15.70:0/2252976980 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x556f453e22f0 0x556f453e46e0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.674+0000 7efda538f740 1 --2- 172.21.15.70:0/2252976980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453d1ff0 0x556f453de010 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.674+0000 7efda538f740 1 --2- 172.21.15.70:0/2252976980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 0x556f453d6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.674+0000 7efda538f740 1 --2- 172.21.15.70:0/2252976980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cc490 0x556f453d6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.674+0000 7efda538f740 1 -- 172.21.15.70:0/2252976980 >> 172.21.15.70:0/2252976980 conn(0x556f453c7a00 msgr2=0x556f453c6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:06.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.674+0000 7efd7b7fe640 0 client.28966 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:46:06.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.674+0000 7efda538f740 1 -- 172.21.15.70:0/2252976980 shutdown_connections 2024-04-01T01:46:06.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.675+0000 7efda538f740 1 -- 172.21.15.70:0/2252976980 wait complete. 2024-04-01T01:46:06.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.676+0000 7efda538f740 20 client.28966 trim_cache size 0 max 16384 2024-04-01T01:46:06.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.702+0000 7efda538f740 1 Processor -- start 2024-04-01T01:46:06.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.702+0000 7efda538f740 1 -- start start 2024-04-01T01:46:06.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.703+0000 7efda538f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cc490 0x556f453ce450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.703+0000 7efda538f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 0x556f453ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.703+0000 7efda538f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453d1ff0 0x556f453ddfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.703+0000 7efda538f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efd9c1a8120 con 0x556f453cc490 2024-04-01T01:46:06.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.703+0000 7efda538f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efd9c1a9c60 con 0x556f453cce80 2024-04-01T01:46:06.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.703+0000 7efda538f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efd9c1a87f0 con 0x556f453d1ff0 2024-04-01T01:46:06.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.703+0000 7efd9b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cc490 0x556f453ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.703+0000 7efd9b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cc490 0x556f453ce450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56672/0 (socket says 172.21.15.70:56672) 2024-04-01T01:46:06.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.703+0000 7efd9b7fe640 1 -- 172.21.15.70:0/2775447427 learned_addr learned my addr 172.21.15.70:0/2775447427 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:06.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.703+0000 7efd9bfff640 1 --2- 172.21.15.70:0/2775447427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 0x556f453ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.703+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/2775447427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453d1ff0 0x556f453ddfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.704+0000 7efd9b7fe640 1 -- 172.21.15.70:0/2775447427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453d1ff0 msgr2=0x556f453ddfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:06.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.704+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/2775447427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453d1ff0 0x556f453ddfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.704+0000 7efd9b7fe640 1 -- 172.21.15.70:0/2775447427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 msgr2=0x556f453ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:06.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.704+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/2775447427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 0x556f453ce990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.704+0000 7efd9b7fe640 1 -- 172.21.15.70:0/2775447427 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f453d7c10 con 0x556f453cc490 2024-04-01T01:46:06.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.704+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/2775447427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cc490 0x556f453ce450 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x556f453cef70 tx=0x7efd94000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4232e316aceb7281 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:06.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.705+0000 7efd81ffb640 1 -- 172.21.15.70:0/2775447427 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd94036c20 con 0x556f453cc490 2024-04-01T01:46:06.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.705+0000 7efd81ffb640 1 -- 172.21.15.70:0/2775447427 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd9403c040 con 0x556f453cc490 2024-04-01T01:46:06.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.705+0000 7efd81ffb640 1 -- 172.21.15.70:0/2775447427 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd94010a80 con 0x556f453cc490 2024-04-01T01:46:06.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.706+0000 7efda538f740 1 -- 172.21.15.70:0/2775447427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cc490 msgr2=0x556f453ce450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.706+0000 7efda538f740 1 --2- 172.21.15.70:0/2775447427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cc490 0x556f453ce450 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x556f453cef70 tx=0x7efd94000f40 comp rx=0 tx=0).stop 2024-04-01T01:46:06.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.707+0000 7efda538f740 1 -- 172.21.15.70:0/2775447427 shutdown_connections 2024-04-01T01:46:06.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.707+0000 7efda538f740 1 --2- 172.21.15.70:0/2775447427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453d1ff0 0x556f453ddfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.707+0000 7efda538f740 1 --2- 172.21.15.70:0/2775447427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 0x556f453ce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.707+0000 7efda538f740 1 --2- 172.21.15.70:0/2775447427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453cc490 0x556f453ce450 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.707+0000 7efda538f740 1 -- 172.21.15.70:0/2775447427 >> 172.21.15.70:0/2775447427 conn(0x556f453c7a00 msgr2=0x556f453c7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:06.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.707+0000 7efda538f740 1 -- 172.21.15.70:0/2775447427 shutdown_connections 2024-04-01T01:46:06.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.709+0000 7efda538f740 1 -- 172.21.15.70:0/2775447427 wait complete. 2024-04-01T01:46:06.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.710+0000 7efda538f740 1 -- start start 2024-04-01T01:46:06.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.710+0000 7efda538f740 1 Processor -- start 2024-04-01T01:46:06.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.710+0000 7efda538f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453cc490 0x556f453db060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.710+0000 7efda538f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 0x556f453db5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.710+0000 7efda538f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453d1ff0 0x556f45446580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.710+0000 7efda538f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efd9c1a9590 con 0x556f453d1ff0 2024-04-01T01:46:06.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.710+0000 7efda538f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efd9c1a8ec0 con 0x556f453cce80 2024-04-01T01:46:06.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.710+0000 7efda538f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efd9c1a61b0 con 0x556f453cc490 2024-04-01T01:46:06.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.711+0000 7efda538f740 10 client.0 _ll_register_callbacks cb 0x7efd94012500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:06.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.711+0000 7efd9bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 0x556f453db5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.711+0000 7efd9bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 0x556f453db5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:45054/0 (socket says 172.21.15.70:45054) 2024-04-01T01:46:06.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.711+0000 7efd9bfff640 1 -- 172.21.15.70:0/3948190636 learned_addr learned my addr 172.21.15.70:0/3948190636 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:06.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.711+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/3948190636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453cc490 0x556f453db060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.711+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/3948190636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453d1ff0 0x556f45446580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.711+0000 7efd9bfff640 1 -- 172.21.15.70:0/3948190636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453cc490 msgr2=0x556f453db060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:06.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.711+0000 7efd9bfff640 1 --2- 172.21.15.70:0/3948190636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453cc490 0x556f453db060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.711+0000 7efd9bfff640 1 -- 172.21.15.70:0/3948190636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453d1ff0 msgr2=0x556f45446580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:06.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.711+0000 7efd9bfff640 1 --2- 172.21.15.70:0/3948190636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453d1ff0 0x556f45446580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.711+0000 7efd9bfff640 1 -- 172.21.15.70:0/3948190636 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f45457ad0 con 0x556f453cce80 2024-04-01T01:46:06.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.712+0000 7efd9bfff640 1 --2- 172.21.15.70:0/3948190636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 0x556f453db5a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7efd90058040 tx=0x7efd90005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78d166a614106228 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:06.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.712+0000 7efd7b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:06.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.712+0000 7efd7b7fe640 1 -- 172.21.15.70:0/3948190636 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd90004240 con 0x556f453cce80 2024-04-01T01:46:06.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.712+0000 7efd7b7fe640 1 -- 172.21.15.70:0/3948190636 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7efd900043e0 con 0x556f453cce80 2024-04-01T01:46:06.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.712+0000 7efda538f740 1 -- 172.21.15.70:0/3948190636 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd9403c040 con 0x556f453cce80 2024-04-01T01:46:06.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.713+0000 7efd7b7fe640 1 -- 172.21.15.70:0/3948190636 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efd9004eca0 con 0x556f453cce80 2024-04-01T01:46:06.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.713+0000 7efda538f740 10 client.29192 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:06.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.713+0000 7efda538f740 1 -- 172.21.15.70:0/3948190636 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556f45458e10 con 0x556f453cce80 2024-04-01T01:46:06.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.716+0000 7efd7b7fe640 1 -- 172.21.15.70:0/3948190636 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd9403c040 con 0x556f453cce80 2024-04-01T01:46:06.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.716+0000 7efd7b7fe640 1 -- 172.21.15.70:0/3948190636 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7efd9006d030 con 0x556f453cce80 2024-04-01T01:46:06.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.716+0000 7efda538f740 10 client.29192 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:06.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.716+0000 7efda538f740 10 client.29192 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:46:06.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.716+0000 7efda538f740 20 client.29192 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:06.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.716+0000 7efda538f740 1 --2- 172.21.15.70:0/3948190636 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x556f454597c0 0x556f4545bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.716+0000 7efda538f740 4 client.29192 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:46:06.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.716+0000 7efda538f740 1 -- 172.21.15.70:0/3948190636 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556f45338a40 con 0x556f454597c0 2024-04-01T01:46:06.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.717+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/3948190636 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x556f454597c0 0x556f4545bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.719+0000 7efd9b7fe640 1 --2- 172.21.15.70:0/3948190636 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x556f454597c0 0x556f4545bbe0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=233c648996e813b0 server_cookie=c7a8bf09908c7289 in_seq=0 out_seq=0 2024-04-01T01:46:06.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.719+0000 7efd7b7fe640 10 client.29192 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:46:06.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.719+0000 7efd7b7fe640 1 -- 172.21.15.70:0/3948190636 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556f45338a40 con 0x556f454597c0 2024-04-01T01:46:06.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.719+0000 7efd7b7fe640 10 client.29192 handle_command_reply: tid=0 2024-04-01T01:46:06.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.719+0000 7efda538f740 1 client.29192 shutdown 2024-04-01T01:46:06.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.725+0000 7efda538f740 1 -- 172.21.15.70:0/3948190636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 msgr2=0x556f453db5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.725+0000 7efda538f740 1 --2- 172.21.15.70:0/3948190636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 0x556f453db5a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7efd90058040 tx=0x7efd90005e00 comp rx=0 tx=0).stop 2024-04-01T01:46:06.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.729+0000 7efda538f740 1 -- 172.21.15.70:0/3948190636 shutdown_connections 2024-04-01T01:46:06.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.729+0000 7efda538f740 1 --2- 172.21.15.70:0/3948190636 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x556f454597c0 0x556f4545bbe0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.729+0000 7efda538f740 1 --2- 172.21.15.70:0/3948190636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556f453d1ff0 0x556f45446580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.729+0000 7efda538f740 1 --2- 172.21.15.70:0/3948190636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556f453cce80 0x556f453db5a0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.729+0000 7efda538f740 1 --2- 172.21.15.70:0/3948190636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556f453cc490 0x556f453db060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.729+0000 7efda538f740 1 -- 172.21.15.70:0/3948190636 >> 172.21.15.70:0/3948190636 conn(0x556f453c7a00 msgr2=0x556f453d6a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:06.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.729+0000 7efd7b7fe640 0 client.29192 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:46:06.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.730+0000 7efda538f740 1 -- 172.21.15.70:0/3948190636 shutdown_connections 2024-04-01T01:46:06.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.731+0000 7efda538f740 1 -- 172.21.15.70:0/3948190636 wait complete. 2024-04-01T01:46:06.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.731+0000 7efda538f740 20 client.29192 trim_cache size 0 max 16384 2024-04-01T01:46:06.737 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:46:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.737+0000 7efda27ac640 1 -- 172.21.15.70:0/1165960385 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efd840a2620 msgr2=0x7efd840a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.737+0000 7efda27ac640 1 --2- 172.21.15.70:0/1165960385 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efd840a2620 0x7efd840a4ae0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efd8c0cbbb0 tx=0x7efd8c004730 comp rx=0 tx=0).stop 2024-04-01T01:46:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.737+0000 7efda27ac640 1 -- 172.21.15.70:0/1165960385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd9c06d2f0 msgr2=0x7efd9c194570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.737+0000 7efda27ac640 1 --2- 172.21.15.70:0/1165960385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd9c06d2f0 0x7efd9c194570 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7efd8c00d750 tx=0x7efd8c00dc10 comp rx=0 tx=0).stop 2024-04-01T01:46:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.738+0000 7efda27ac640 1 -- 172.21.15.70:0/1165960385 shutdown_connections 2024-04-01T01:46:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.738+0000 7efda27ac640 1 --2- 172.21.15.70:0/1165960385 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efd840a2620 0x7efd840a4ae0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.738+0000 7efda27ac640 1 --2- 172.21.15.70:0/1165960385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd9c06d2f0 0x7efd9c194570 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.738+0000 7efda27ac640 1 --2- 172.21.15.70:0/1165960385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd9c06c950 0x7efd9c18d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.738+0000 7efda27ac640 1 --2- 172.21.15.70:0/1165960385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd9c06b710 0x7efd9c18cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.738+0000 7efda27ac640 1 -- 172.21.15.70:0/1165960385 >> 172.21.15.70:0/1165960385 conn(0x7efd9c0fbbe0 msgr2=0x7efd9c06fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:06.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.740+0000 7efda27ac640 1 -- 172.21.15.70:0/1165960385 shutdown_connections 2024-04-01T01:46:06.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.740+0000 7efda27ac640 1 -- 172.21.15.70:0/1165960385 wait complete. 2024-04-01T01:46:06.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:06 smithi070 ceph-mon[32179]: pgmap v737: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.4 MiB/s wr, 733 op/s 2024-04-01T01:46:06.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:06 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1605232859' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:46:06.769 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:46:06.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 680.0 s, 839.8 tps, lat 121.572 ms stddev 122.110 2024-04-01T01:46:06.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.956+0000 7f8073422640 1 -- 172.21.15.70:0/3409218048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f806c1020e0 msgr2=0x7f806c108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.956+0000 7f8073422640 1 --2- 172.21.15.70:0/3409218048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f806c1020e0 0x7f806c108970 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8064009ca0 tx=0x7f806402c140 comp rx=0 tx=0).stop 2024-04-01T01:46:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.957+0000 7f8073422640 1 -- 172.21.15.70:0/3409218048 shutdown_connections 2024-04-01T01:46:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.957+0000 7f8073422640 1 --2- 172.21.15.70:0/3409218048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f806c1020e0 0x7f806c108970 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.957+0000 7f8073422640 1 --2- 172.21.15.70:0/3409218048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f806c101720 0x7f806c101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.957+0000 7f8073422640 1 --2- 172.21.15.70:0/3409218048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c100530 0x7f806c100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.957+0000 7f8073422640 1 -- 172.21.15.70:0/3409218048 >> 172.21.15.70:0/3409218048 conn(0x7f806c0fbac0 msgr2=0x7f806c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:06.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.960+0000 7f8073422640 1 -- 172.21.15.70:0/3409218048 shutdown_connections 2024-04-01T01:46:06.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.962+0000 7f8073422640 1 -- 172.21.15.70:0/3409218048 wait complete. 2024-04-01T01:46:06.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.963+0000 7f8073422640 1 Processor -- start 2024-04-01T01:46:06.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.963+0000 7f8073422640 1 -- start start 2024-04-01T01:46:06.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.963+0000 7f8073422640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f806c100530 0x7f806c19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.963+0000 7f8073422640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f806c101720 0x7f806c19e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.963+0000 7f8073422640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c1020e0 0x7f806c1a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.964+0000 7f8073422640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f806c10a8a0 con 0x7f806c100530 2024-04-01T01:46:06.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.964+0000 7f8073422640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f806c10a720 con 0x7f806c1020e0 2024-04-01T01:46:06.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.964+0000 7f8072c21640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c1020e0 0x7f806c1a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.964+0000 7f8072c21640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c1020e0 0x7f806c1a5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:45074/0 (socket says 172.21.15.70:45074) 2024-04-01T01:46:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.964+0000 7f8072c21640 1 -- 172.21.15.70:0/698675838 learned_addr learned my addr 172.21.15.70:0/698675838 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.964+0000 7f8073422640 1 -- 172.21.15.70:0/698675838 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f806c10aa20 con 0x7f806c101720 2024-04-01T01:46:06.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.964+0000 7f8072420640 1 --2- 172.21.15.70:0/698675838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f806c100530 0x7f806c19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.965+0000 7f8072c21640 1 -- 172.21.15.70:0/698675838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f806c101720 msgr2=0x7f806c19e4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:46:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.965+0000 7f8072c21640 1 --2- 172.21.15.70:0/698675838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f806c101720 0x7f806c19e4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.965+0000 7f8072c21640 1 -- 172.21.15.70:0/698675838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f806c100530 msgr2=0x7f806c19dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.965+0000 7f8072c21640 1 --2- 172.21.15.70:0/698675838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f806c100530 0x7f806c19dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.965+0000 7f8072c21640 1 -- 172.21.15.70:0/698675838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806c1a5bc0 con 0x7f806c1020e0 2024-04-01T01:46:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.965+0000 7f8072420640 1 --2- 172.21.15.70:0/698675838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f806c100530 0x7f806c19dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:46:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.965+0000 7f8072c21640 1 --2- 172.21.15.70:0/698675838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c1020e0 0x7f806c1a5550 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f80640325a0 tx=0x7f8064032870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.965+0000 7f80637fe640 1 -- 172.21.15.70:0/698675838 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8064039040 con 0x7f806c1020e0 2024-04-01T01:46:06.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.965+0000 7f80637fe640 1 -- 172.21.15.70:0/698675838 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8064044070 con 0x7f806c1020e0 2024-04-01T01:46:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.966+0000 7f80637fe640 1 -- 172.21.15.70:0/698675838 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8064004750 con 0x7f806c1020e0 2024-04-01T01:46:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.966+0000 7f8073422640 1 -- 172.21.15.70:0/698675838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f806c1a5e50 con 0x7f806c1020e0 2024-04-01T01:46:06.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.966+0000 7f8073422640 1 -- 172.21.15.70:0/698675838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f806c1a6200 con 0x7f806c1020e0 2024-04-01T01:46:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.970+0000 7f8077269740 1 Processor -- start 2024-04-01T01:46:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.970+0000 7f80637fe640 1 -- 172.21.15.70:0/698675838 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f8064004040 con 0x7f806c1020e0 2024-04-01T01:46:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.970+0000 7f8077269740 1 -- start start 2024-04-01T01:46:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.970+0000 7f8077269740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac556a0 0x55adeac55ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.970+0000 7f8077269740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac56090 0x55adeac5ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.970+0000 7f8077269740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 0x55adeac5d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.970+0000 7f8077269740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55adeabba1b0 con 0x55adeac5b200 2024-04-01T01:46:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.970+0000 7f8077269740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55adeab62c00 con 0x55adeac56090 2024-04-01T01:46:06.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.970+0000 7f8077269740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55adeabb6250 con 0x55adeac556a0 2024-04-01T01:46:06.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.971+0000 7f8071c1f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 0x55adeac5d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.971+0000 7f8071c1f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac556a0 0x55adeac55ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.971+0000 7f8071c1f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 0x55adeac5d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56700/0 (socket says 172.21.15.70:56700) 2024-04-01T01:46:06.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.971+0000 7f8071c1f640 1 -- 172.21.15.70:0/4037055163 learned_addr learned my addr 172.21.15.70:0/4037055163 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:06.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.972+0000 7f8071c1f640 1 -- 172.21.15.70:0/4037055163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac556a0 msgr2=0x55adeac55ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:06.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.972+0000 7f8071c1f640 1 --2- 172.21.15.70:0/4037055163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac556a0 0x55adeac55ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.972+0000 7f8071c1f640 1 -- 172.21.15.70:0/4037055163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac56090 msgr2=0x55adeac5ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:06.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.972+0000 7f8071c1f640 1 --2- 172.21.15.70:0/4037055163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac56090 0x55adeac5ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.972+0000 7f8071c1f640 1 -- 172.21.15.70:0/4037055163 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adeab823d0 con 0x55adeac5b200 2024-04-01T01:46:06.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.972+0000 7f80637fe640 1 --2- 172.21.15.70:0/698675838 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f80580a2480 0x7f80580a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.972+0000 7f8071c1f640 1 --2- 172.21.15.70:0/4037055163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 0x55adeac5d690 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f8068014420 tx=0x7f8068006be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3cc2a3a12c4927 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:06.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.973+0000 7f80637fe640 1 -- 172.21.15.70:0/698675838 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f80640f0bd0 con 0x7f806c1020e0 2024-04-01T01:46:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.973+0000 7f8060ff9640 1 -- 172.21.15.70:0/4037055163 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806801c070 con 0x55adeac5b200 2024-04-01T01:46:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.973+0000 7f8060ff9640 1 -- 172.21.15.70:0/4037055163 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f806800d070 con 0x55adeac5b200 2024-04-01T01:46:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.973+0000 7f8060ff9640 1 -- 172.21.15.70:0/4037055163 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806800f040 con 0x55adeac5b200 2024-04-01T01:46:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.974+0000 7f8072420640 1 --2- 172.21.15.70:0/698675838 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f80580a2480 0x7f80580a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.974+0000 7f8077269740 1 -- 172.21.15.70:0/4037055163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 msgr2=0x55adeac5d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.974+0000 7f8077269740 1 --2- 172.21.15.70:0/4037055163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 0x55adeac5d690 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f8068014420 tx=0x7f8068006be0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.974+0000 7f8077269740 1 -- 172.21.15.70:0/4037055163 shutdown_connections 2024-04-01T01:46:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.974+0000 7f8077269740 1 --2- 172.21.15.70:0/4037055163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 0x55adeac5d690 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.974+0000 7f8077269740 1 --2- 172.21.15.70:0/4037055163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac56090 0x55adeac5ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.974+0000 7f8077269740 1 --2- 172.21.15.70:0/4037055163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac556a0 0x55adeac55ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.974+0000 7f8077269740 1 -- 172.21.15.70:0/4037055163 >> 172.21.15.70:0/4037055163 conn(0x55adeac50c10 msgr2=0x55adeac51010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:06.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.975+0000 7f8077269740 1 -- 172.21.15.70:0/4037055163 shutdown_connections 2024-04-01T01:46:06.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.975+0000 7f8072420640 1 --2- 172.21.15.70:0/698675838 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f80580a2480 0x7f80580a4940 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x55adeac56490 tx=0x7f805c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:06.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.975+0000 7f8077269740 1 -- 172.21.15.70:0/4037055163 wait complete. 2024-04-01T01:46:06.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.975+0000 7f8077269740 1 -- start start 2024-04-01T01:46:06.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.975+0000 7f8077269740 1 Processor -- start 2024-04-01T01:46:06.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.976+0000 7f8077269740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac556a0 0x55adeac5fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.976+0000 7f8077269740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac56090 0x55adeac60230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.976+0000 7f8077269740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 0x55adeac67260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.976+0000 7f8077269740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55adeac5e200 con 0x55adeac5b200 2024-04-01T01:46:06.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.976+0000 7f8077269740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55adeab58470 con 0x55adeac556a0 2024-04-01T01:46:06.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.976+0000 7f8077269740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55adeac5e380 con 0x55adeac56090 2024-04-01T01:46:06.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.976+0000 7f8077269740 10 client.0 _ll_register_callbacks cb 0x55adeab75cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:06.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.976+0000 7f8072420640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac556a0 0x55adeac5fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.976+0000 7f8072420640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac556a0 0x55adeac5fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:45092/0 (socket says 172.21.15.70:45092) 2024-04-01T01:46:06.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.976+0000 7f8072420640 1 -- 172.21.15.70:0/2035317813 learned_addr learned my addr 172.21.15.70:0/2035317813 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.976+0000 7f8071c1f640 1 --2- 172.21.15.70:0/2035317813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac56090 0x55adeac60230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.976+0000 7f8072c21640 1 --2- 172.21.15.70:0/2035317813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 0x55adeac67260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.977+0000 7f8071c1f640 1 -- 172.21.15.70:0/2035317813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac556a0 msgr2=0x55adeac5fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.977+0000 7f8071c1f640 1 --2- 172.21.15.70:0/2035317813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac556a0 0x55adeac5fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.977+0000 7f8071c1f640 1 -- 172.21.15.70:0/2035317813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 msgr2=0x55adeac67260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.977+0000 7f8071c1f640 1 --2- 172.21.15.70:0/2035317813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 0x55adeac67260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.977+0000 7f8071c1f640 1 -- 172.21.15.70:0/2035317813 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adea8ce3f0 con 0x55adeac56090 2024-04-01T01:46:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.977+0000 7f8071c1f640 1 --2- 172.21.15.70:0/2035317813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac56090 0x55adeac60230 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f806801a040 tx=0x7f8068017a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fd915948acd4775 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:06.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.978+0000 7f8055ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:46:06.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.978+0000 7f8055ffb640 1 -- 172.21.15.70:0/2035317813 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806801c050 con 0x55adeac56090 2024-04-01T01:46:06.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.978+0000 7f8077269740 1 -- 172.21.15.70:0/2035317813 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f806800d070 con 0x55adeac56090 2024-04-01T01:46:06.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.978+0000 7f8055ffb640 1 -- 172.21.15.70:0/2035317813 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8068021810 con 0x55adeac56090 2024-04-01T01:46:06.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.978+0000 7f8055ffb640 1 -- 172.21.15.70:0/2035317813 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8068031050 con 0x55adeac56090 2024-04-01T01:46:06.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.978+0000 7f8077269740 10 client.28978 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:06.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.978+0000 7f8077269740 1 -- 172.21.15.70:0/2035317813 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55adea9987d0 con 0x55adeac56090 2024-04-01T01:46:06.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.979+0000 7f8055ffb640 1 -- 172.21.15.70:0/2035317813 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f806800d070 con 0x55adeac56090 2024-04-01T01:46:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.980+0000 7f8055ffb640 1 -- 172.21.15.70:0/2035317813 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8068023550 con 0x55adeac56090 2024-04-01T01:46:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.980+0000 7f8077269740 10 client.28978 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.980+0000 7f8077269740 10 client.28978 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:46:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.980+0000 7f8077269740 20 client.28978 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.980+0000 7f8077269740 1 --2- 172.21.15.70:0/2035317813 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55adeac6b3c0 0x55adeac6d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.980+0000 7f8077269740 4 client.28978 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.980+0000 7f8077269740 1 -- 172.21.15.70:0/2035317813 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adeabc76f0 con 0x55adeac6b3c0 2024-04-01T01:46:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.980+0000 7f8072420640 1 --2- 172.21.15.70:0/2035317813 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55adeac6b3c0 0x55adeac6d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.981+0000 7f8072420640 1 --2- 172.21.15.70:0/2035317813 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55adeac6b3c0 0x55adeac6d7b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=171069be10db243b server_cookie=60426fdd527417dd in_seq=0 out_seq=0 2024-04-01T01:46:06.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.981+0000 7f8055ffb640 10 client.28978 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:46:06.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.982+0000 7f8055ffb640 1 -- 172.21.15.70:0/2035317813 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55adeabc76f0 con 0x55adeac6b3c0 2024-04-01T01:46:06.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.982+0000 7f8055ffb640 10 client.28978 handle_command_reply: tid=0 2024-04-01T01:46:06.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.982+0000 7f8077269740 1 client.28978 shutdown 2024-04-01T01:46:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.983+0000 7f8077269740 1 -- 172.21.15.70:0/2035317813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac56090 msgr2=0x55adeac60230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.983+0000 7f8077269740 1 --2- 172.21.15.70:0/2035317813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac56090 0x55adeac60230 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f806801a040 tx=0x7f8068017a80 comp rx=0 tx=0).stop 2024-04-01T01:46:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.983+0000 7f8077269740 1 -- 172.21.15.70:0/2035317813 shutdown_connections 2024-04-01T01:46:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.983+0000 7f8077269740 1 --2- 172.21.15.70:0/2035317813 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55adeac6b3c0 0x55adeac6d7b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.983+0000 7f8077269740 1 --2- 172.21.15.70:0/2035317813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 0x55adeac67260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.983+0000 7f8077269740 1 --2- 172.21.15.70:0/2035317813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac56090 0x55adeac60230 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.983+0000 7f8077269740 1 --2- 172.21.15.70:0/2035317813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac556a0 0x55adeac5fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.983+0000 7f8077269740 1 -- 172.21.15.70:0/2035317813 >> 172.21.15.70:0/2035317813 conn(0x55adeac50c10 msgr2=0x55adeac4ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.983+0000 7f8055ffb640 0 client.28978 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:46:06.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.983+0000 7f8077269740 1 -- 172.21.15.70:0/2035317813 shutdown_connections 2024-04-01T01:46:06.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.983+0000 7f8077269740 1 -- 172.21.15.70:0/2035317813 wait complete. 2024-04-01T01:46:06.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:06.984+0000 7f8077269740 20 client.28978 trim_cache size 0 max 16384 2024-04-01T01:46:06.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:06 smithi189 ceph-mon[35753]: pgmap v737: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.4 MiB/s wr, 733 op/s 2024-04-01T01:46:06.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:06 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1605232859' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:46:07.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.011+0000 7f8077269740 1 Processor -- start 2024-04-01T01:46:07.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.011+0000 7f8077269740 1 -- start start 2024-04-01T01:46:07.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.011+0000 7f8077269740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac556a0 0x55adeac5e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.011+0000 7f8077269740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac56090 0x55adeac67220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.011+0000 7f8077269740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 0x55adeac5ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.011+0000 7f8077269740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f806c1a7f70 con 0x55adeac5b200 2024-04-01T01:46:07.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.011+0000 7f8077269740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f806c1a9ab0 con 0x55adeac556a0 2024-04-01T01:46:07.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.011+0000 7f8077269740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f806c1a8640 con 0x55adeac56090 2024-04-01T01:46:07.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.011+0000 7f8072c21640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 0x55adeac5ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.011+0000 7f8072c21640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 0x55adeac5ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56716/0 (socket says 172.21.15.70:56716) 2024-04-01T01:46:07.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.011+0000 7f8072c21640 1 -- 172.21.15.70:0/2898984272 learned_addr learned my addr 172.21.15.70:0/2898984272 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.011+0000 7f8071c1f640 1 --2- 172.21.15.70:0/2898984272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac56090 0x55adeac67220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.011+0000 7f8072420640 1 --2- 172.21.15.70:0/2898984272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac556a0 0x55adeac5e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.012+0000 7f8072c21640 1 -- 172.21.15.70:0/2898984272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac56090 msgr2=0x55adeac67220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.012+0000 7f8072c21640 1 --2- 172.21.15.70:0/2898984272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac56090 0x55adeac67220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.012+0000 7f8072c21640 1 -- 172.21.15.70:0/2898984272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac556a0 msgr2=0x55adeac5e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.012+0000 7f8072c21640 1 --2- 172.21.15.70:0/2898984272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac556a0 0x55adeac5e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.012+0000 7f8072c21640 1 -- 172.21.15.70:0/2898984272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adeac6dc80 con 0x55adeac5b200 2024-04-01T01:46:07.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.012+0000 7f8072c21640 1 --2- 172.21.15.70:0/2898984272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 0x55adeac5ec90 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f806402c750 tx=0x7f80640ee170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a097eb2a29432a0f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.012+0000 7f8057fff640 1 -- 172.21.15.70:0/2898984272 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80640b7e60 con 0x55adeac5b200 2024-04-01T01:46:07.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.012+0000 7f8057fff640 1 -- 172.21.15.70:0/2898984272 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8064004330 con 0x55adeac5b200 2024-04-01T01:46:07.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.013+0000 7f8057fff640 1 -- 172.21.15.70:0/2898984272 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80640bd2f0 con 0x55adeac5b200 2024-04-01T01:46:07.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.014+0000 7f8077269740 1 -- 172.21.15.70:0/2898984272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 msgr2=0x55adeac5ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.014+0000 7f8077269740 1 --2- 172.21.15.70:0/2898984272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 0x55adeac5ec90 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f806402c750 tx=0x7f80640ee170 comp rx=0 tx=0).stop 2024-04-01T01:46:07.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.014+0000 7f8077269740 1 -- 172.21.15.70:0/2898984272 shutdown_connections 2024-04-01T01:46:07.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.014+0000 7f8077269740 1 --2- 172.21.15.70:0/2898984272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac5b200 0x55adeac5ec90 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.014+0000 7f8077269740 1 --2- 172.21.15.70:0/2898984272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac56090 0x55adeac67220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.014+0000 7f8077269740 1 --2- 172.21.15.70:0/2898984272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac556a0 0x55adeac5e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.014+0000 7f8077269740 1 -- 172.21.15.70:0/2898984272 >> 172.21.15.70:0/2898984272 conn(0x55adeac50c10 msgr2=0x55adeac57620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.015+0000 7f8077269740 1 -- 172.21.15.70:0/2898984272 shutdown_connections 2024-04-01T01:46:07.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.015+0000 7f8077269740 1 -- 172.21.15.70:0/2898984272 wait complete. 2024-04-01T01:46:07.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.015+0000 7f8077269740 1 -- start start 2024-04-01T01:46:07.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.015+0000 7f8077269740 1 Processor -- start 2024-04-01T01:46:07.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.016+0000 7f8077269740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac556a0 0x55adeac64540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.016+0000 7f8077269740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac56090 0x55adeac64a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.016+0000 7f8077269740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac5b200 0x55adeac659f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.016+0000 7f8077269740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f806c1a93e0 con 0x55adeac556a0 2024-04-01T01:46:07.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.016+0000 7f8077269740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f806c1a8d10 con 0x55adeac56090 2024-04-01T01:46:07.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.016+0000 7f8077269740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f806c073460 con 0x55adeac5b200 2024-04-01T01:46:07.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.016+0000 7f8077269740 10 client.0 _ll_register_callbacks cb 0x7f8068014320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:07.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.016+0000 7f8072c21640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac5b200 0x55adeac659f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.016+0000 7f8072c21640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac5b200 0x55adeac659f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:35532/0 (socket says 172.21.15.70:35532) 2024-04-01T01:46:07.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.016+0000 7f8072c21640 1 -- 172.21.15.70:0/1751552038 learned_addr learned my addr 172.21.15.70:0/1751552038 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.016+0000 7f8071c1f640 1 --2- 172.21.15.70:0/1751552038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac56090 0x55adeac64a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.017+0000 7f8072420640 1 --2- 172.21.15.70:0/1751552038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac556a0 0x55adeac64540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.017+0000 7f8071c1f640 1 -- 172.21.15.70:0/1751552038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac5b200 msgr2=0x55adeac659f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.017+0000 7f8071c1f640 1 --2- 172.21.15.70:0/1751552038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac5b200 0x55adeac659f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.017+0000 7f8071c1f640 1 -- 172.21.15.70:0/1751552038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac556a0 msgr2=0x55adeac64540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.017+0000 7f8071c1f640 1 --2- 172.21.15.70:0/1751552038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac556a0 0x55adeac64540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.017+0000 7f8071c1f640 1 -- 172.21.15.70:0/1751552038 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adeac65fe0 con 0x55adeac56090 2024-04-01T01:46:07.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.017+0000 7f8071c1f640 1 --2- 172.21.15.70:0/1751552038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac56090 0x55adeac64a80 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8068023ae0 tx=0x7f806800ab60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=775deae7905f33f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.017+0000 7f8055ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:07.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.017+0000 7f8055ffb640 1 -- 172.21.15.70:0/1751552038 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068039070 con 0x55adeac56090 2024-04-01T01:46:07.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.018+0000 7f8055ffb640 1 -- 172.21.15.70:0/1751552038 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8068023b60 con 0x55adeac56090 2024-04-01T01:46:07.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.018+0000 7f8077269740 1 -- 172.21.15.70:0/1751552038 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55adeabc1a40 con 0x55adeac56090 2024-04-01T01:46:07.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.018+0000 7f8055ffb640 1 -- 172.21.15.70:0/1751552038 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8068045050 con 0x55adeac56090 2024-04-01T01:46:07.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.018+0000 7f8077269740 10 client.29204 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:07.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.018+0000 7f8077269740 1 -- 172.21.15.70:0/1751552038 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55adeac672e0 con 0x55adeac56090 2024-04-01T01:46:07.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.018+0000 7f8055ffb640 1 -- 172.21.15.70:0/1751552038 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55adeabc1a40 con 0x55adeac56090 2024-04-01T01:46:07.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.019+0000 7f8055ffb640 1 -- 172.21.15.70:0/1751552038 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8068004040 con 0x55adeac56090 2024-04-01T01:46:07.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.019+0000 7f8077269740 10 client.29204 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:07.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.019+0000 7f8077269740 10 client.29204 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:46:07.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.019+0000 7f8077269740 20 client.29204 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:07.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.019+0000 7f8077269740 1 --2- 172.21.15.70:0/1751552038 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55adeacd19b0 0x55adeacd3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.019+0000 7f8077269740 4 client.29204 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:46:07.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.019+0000 7f8077269740 1 -- 172.21.15.70:0/1751552038 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55adeab3f6a0 con 0x55adeacd19b0 2024-04-01T01:46:07.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.019+0000 7f8072420640 1 --2- 172.21.15.70:0/1751552038 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55adeacd19b0 0x55adeacd3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.020+0000 7f8072420640 1 --2- 172.21.15.70:0/1751552038 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55adeacd19b0 0x55adeacd3dd0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=463f3fe5abfc377e server_cookie=5ac5c2daefe1c803 in_seq=0 out_seq=0 2024-04-01T01:46:07.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.020+0000 7f8055ffb640 10 client.29204 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:46:07.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.021+0000 7f8055ffb640 1 -- 172.21.15.70:0/1751552038 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55adeab3f6a0 con 0x55adeacd19b0 2024-04-01T01:46:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.021+0000 7f8055ffb640 10 client.29204 handle_command_reply: tid=0 2024-04-01T01:46:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.021+0000 7f8077269740 1 client.29204 shutdown 2024-04-01T01:46:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.021+0000 7f8077269740 1 -- 172.21.15.70:0/1751552038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac56090 msgr2=0x55adeac64a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.021+0000 7f8077269740 1 --2- 172.21.15.70:0/1751552038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac56090 0x55adeac64a80 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8068023ae0 tx=0x7f806800ab60 comp rx=0 tx=0).stop 2024-04-01T01:46:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.022+0000 7f8077269740 1 -- 172.21.15.70:0/1751552038 shutdown_connections 2024-04-01T01:46:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.022+0000 7f8077269740 1 --2- 172.21.15.70:0/1751552038 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55adeacd19b0 0x55adeacd3dd0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.022+0000 7f8077269740 1 --2- 172.21.15.70:0/1751552038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55adeac5b200 0x55adeac659f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.022+0000 7f8077269740 1 --2- 172.21.15.70:0/1751552038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55adeac56090 0x55adeac64a80 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.022+0000 7f8077269740 1 --2- 172.21.15.70:0/1751552038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55adeac556a0 0x55adeac64540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.022+0000 7f8077269740 1 -- 172.21.15.70:0/1751552038 >> 172.21.15.70:0/1751552038 conn(0x55adeac50c10 msgr2=0x55adeac57620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.022+0000 7f8055ffb640 0 client.29204 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:46:07.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.026+0000 7f8077269740 1 -- 172.21.15.70:0/1751552038 shutdown_connections 2024-04-01T01:46:07.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.027+0000 7f8077269740 1 -- 172.21.15.70:0/1751552038 wait complete. 2024-04-01T01:46:07.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.028+0000 7f8077269740 20 client.29204 trim_cache size 0 max 16384 2024-04-01T01:46:07.027 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:46:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.029+0000 7f8073422640 1 -- 172.21.15.70:0/698675838 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f80580a2480 msgr2=0x7f80580a4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.029+0000 7f8073422640 1 --2- 172.21.15.70:0/698675838 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f80580a2480 0x7f80580a4940 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x55adeac56490 tx=0x7f805c008040 comp rx=0 tx=0).stop 2024-04-01T01:46:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.029+0000 7f8073422640 1 -- 172.21.15.70:0/698675838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c1020e0 msgr2=0x7f806c1a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.029+0000 7f8073422640 1 --2- 172.21.15.70:0/698675838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c1020e0 0x7f806c1a5550 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f80640325a0 tx=0x7f8064032870 comp rx=0 tx=0).stop 2024-04-01T01:46:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.030+0000 7f8073422640 1 -- 172.21.15.70:0/698675838 shutdown_connections 2024-04-01T01:46:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.030+0000 7f8073422640 1 --2- 172.21.15.70:0/698675838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c1020e0 0x7f806c1a5550 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.030+0000 7f8073422640 1 --2- 172.21.15.70:0/698675838 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f80580a2480 0x7f80580a4940 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.030+0000 7f8073422640 1 --2- 172.21.15.70:0/698675838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f806c101720 0x7f806c19e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.030+0000 7f8073422640 1 --2- 172.21.15.70:0/698675838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f806c100530 0x7f806c19dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.030+0000 7f8073422640 1 -- 172.21.15.70:0/698675838 >> 172.21.15.70:0/698675838 conn(0x7f806c0fbac0 msgr2=0x7f806c10dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.031+0000 7f8073422640 1 -- 172.21.15.70:0/698675838 shutdown_connections 2024-04-01T01:46:07.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.031+0000 7f8073422640 1 -- 172.21.15.70:0/698675838 wait complete. 2024-04-01T01:46:07.049 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:46:07.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:06 smithi081 ceph-mon[35142]: pgmap v737: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.4 MiB/s wr, 733 op/s 2024-04-01T01:46:07.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:06 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1605232859' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:46:07.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.226+0000 7fcf9a13f640 1 -- 172.21.15.70:0/2469555694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf94109ab0 msgr2=0x7fcf94109ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.226+0000 7fcf9a13f640 1 --2- 172.21.15.70:0/2469555694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf94109ab0 0x7fcf94109ed0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fcf90009930 tx=0x7fcf9002bc50 comp rx=0 tx=0).stop 2024-04-01T01:46:07.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.226+0000 7fcf9a13f640 1 -- 172.21.15.70:0/2469555694 shutdown_connections 2024-04-01T01:46:07.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.226+0000 7fcf9a13f640 1 --2- 172.21.15.70:0/2469555694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf94109110 0x7fcf941109b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.226+0000 7fcf9a13f640 1 --2- 172.21.15.70:0/2469555694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf94108750 0x7fcf94108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.226+0000 7fcf9a13f640 1 --2- 172.21.15.70:0/2469555694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf94109ab0 0x7fcf94109ed0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.226+0000 7fcf9a13f640 1 -- 172.21.15.70:0/2469555694 >> 172.21.15.70:0/2469555694 conn(0x7fcf9406e630 msgr2=0x7fcf94070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.226+0000 7fcf9a13f640 1 -- 172.21.15.70:0/2469555694 shutdown_connections 2024-04-01T01:46:07.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.226+0000 7fcf9a13f640 1 -- 172.21.15.70:0/2469555694 wait complete. 2024-04-01T01:46:07.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.227+0000 7fcf9a13f640 1 Processor -- start 2024-04-01T01:46:07.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.227+0000 7fcf9a13f640 1 -- start start 2024-04-01T01:46:07.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.227+0000 7fcf9a13f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf94108750 0x7fcf94115550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.227+0000 7fcf9a13f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf94109110 0x7fcf941a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.227+0000 7fcf9a13f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf94109ab0 0x7fcf941ada80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.227+0000 7fcf9a13f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcf94114080 con 0x7fcf94109ab0 2024-04-01T01:46:07.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.227+0000 7fcf9a13f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcf94113f00 con 0x7fcf94108750 2024-04-01T01:46:07.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.227+0000 7fcf9993e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf94109ab0 0x7fcf941ada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.227+0000 7fcf9993e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf94109ab0 0x7fcf941ada80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56742/0 (socket says 172.21.15.70:56742) 2024-04-01T01:46:07.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.227+0000 7fcf9993e640 1 -- 172.21.15.70:0/2860959838 learned_addr learned my addr 172.21.15.70:0/2860959838 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.227+0000 7fcf9913d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf94108750 0x7fcf94115550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.227+0000 7fcf9a13f640 1 -- 172.21.15.70:0/2860959838 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcf94114200 con 0x7fcf94109110 2024-04-01T01:46:07.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.227+0000 7fcf9893c640 1 --2- 172.21.15.70:0/2860959838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf94109110 0x7fcf941a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.228+0000 7fcf9913d640 1 -- 172.21.15.70:0/2860959838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf94109110 msgr2=0x7fcf941a2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.228+0000 7fcf9913d640 1 --2- 172.21.15.70:0/2860959838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf94109110 0x7fcf941a2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.228+0000 7fcf9913d640 1 -- 172.21.15.70:0/2860959838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf94109ab0 msgr2=0x7fcf941ada80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.228+0000 7fcf9913d640 1 --2- 172.21.15.70:0/2860959838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf94109ab0 0x7fcf941ada80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.228+0000 7fcf9913d640 1 -- 172.21.15.70:0/2860959838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf941ae340 con 0x7fcf94108750 2024-04-01T01:46:07.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.228+0000 7fcf9913d640 1 --2- 172.21.15.70:0/2860959838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf94108750 0x7fcf94115550 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fcf90036040 tx=0x7fcf90005e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.228+0000 7fcf8a7fc640 1 -- 172.21.15.70:0/2860959838 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf90043020 con 0x7fcf94108750 2024-04-01T01:46:07.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.228+0000 7fcf8a7fc640 1 -- 172.21.15.70:0/2860959838 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf900043d0 con 0x7fcf94108750 2024-04-01T01:46:07.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.228+0000 7fcf8a7fc640 1 -- 172.21.15.70:0/2860959838 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf90003a90 con 0x7fcf94108750 2024-04-01T01:46:07.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.229+0000 7fcf9a13f640 1 -- 172.21.15.70:0/2860959838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf941ae5d0 con 0x7fcf94108750 2024-04-01T01:46:07.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.229+0000 7fcf9a13f640 1 -- 172.21.15.70:0/2860959838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf941ae9e0 con 0x7fcf94108750 2024-04-01T01:46:07.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.232+0000 7fcf9df84740 1 Processor -- start 2024-04-01T01:46:07.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.232+0000 7fcf8a7fc640 1 -- 172.21.15.70:0/2860959838 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fcf90046020 con 0x7fcf94108750 2024-04-01T01:46:07.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.233+0000 7fcf9df84740 1 -- start start 2024-04-01T01:46:07.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.233+0000 7fcf9df84740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 0x5643b0710ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.233+0000 7fcf9df84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b0711090 0x5643b0715b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.233+0000 7fcf9df84740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b0716200 0x5643b0718690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.233+0000 7fcf9df84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5643b06751b0 con 0x5643b0711090 2024-04-01T01:46:07.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.234+0000 7fcf9df84740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5643b061dc00 con 0x5643b07106a0 2024-04-01T01:46:07.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.234+0000 7fcf9df84740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5643b0671250 con 0x5643b0716200 2024-04-01T01:46:07.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.234+0000 7fcf8a7fc640 1 --2- 172.21.15.70:0/2860959838 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcf800a2520 0x7fcf800a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.234+0000 7fcf8a7fc640 1 -- 172.21.15.70:0/2860959838 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fcf900f0b00 con 0x7fcf94108750 2024-04-01T01:46:07.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.235+0000 7fcf9913d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b0711090 0x5643b0715b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.235+0000 7fcf9913d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b0711090 0x5643b0715b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56746/0 (socket says 172.21.15.70:56746) 2024-04-01T01:46:07.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.235+0000 7fcf9913d640 1 -- 172.21.15.70:0/1054654892 learned_addr learned my addr 172.21.15.70:0/1054654892 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.235+0000 7fcf9993e640 1 --2- 172.21.15.70:0/1054654892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 0x5643b0710ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.238+0000 7fcf9893c640 1 --2- 172.21.15.70:0/1054654892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b0716200 0x5643b0718690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.238+0000 7fcf9913d640 1 -- 172.21.15.70:0/1054654892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b0716200 msgr2=0x5643b0718690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.238+0000 7fcf9913d640 1 --2- 172.21.15.70:0/1054654892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b0716200 0x5643b0718690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.238+0000 7fcf9913d640 1 -- 172.21.15.70:0/1054654892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 msgr2=0x5643b0710ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.238+0000 7fcf9913d640 1 --2- 172.21.15.70:0/1054654892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 0x5643b0710ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.238+0000 7fcf9913d640 1 -- 172.21.15.70:0/1054654892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643b063d3d0 con 0x5643b0711090 2024-04-01T01:46:07.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.239+0000 7fcf9993e640 1 --2- 172.21.15.70:0/2860959838 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcf800a2520 0x7fcf800a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.239+0000 7fcf9913d640 1 --2- 172.21.15.70:0/1054654892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b0711090 0x5643b0715b90 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fcf900b9bc0 tx=0x7fcf90005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9584575670f4e76a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.239+0000 7fcf7f7fe640 1 -- 172.21.15.70:0/1054654892 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf900c1120 con 0x5643b0711090 2024-04-01T01:46:07.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.239+0000 7fcf7f7fe640 1 -- 172.21.15.70:0/1054654892 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf900bd8f0 con 0x5643b0711090 2024-04-01T01:46:07.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.240+0000 7fcf9df84740 1 -- 172.21.15.70:0/1054654892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b0711090 msgr2=0x5643b0715b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.240+0000 7fcf9df84740 1 --2- 172.21.15.70:0/1054654892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b0711090 0x5643b0715b90 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fcf900b9bc0 tx=0x7fcf90005850 comp rx=0 tx=0).stop 2024-04-01T01:46:07.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.240+0000 7fcf9df84740 1 -- 172.21.15.70:0/1054654892 shutdown_connections 2024-04-01T01:46:07.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.240+0000 7fcf9df84740 1 --2- 172.21.15.70:0/1054654892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b0716200 0x5643b0718690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.240+0000 7fcf9df84740 1 --2- 172.21.15.70:0/1054654892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b0711090 0x5643b0715b90 secure :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fcf900b9bc0 tx=0x7fcf90005850 comp rx=0 tx=0).stop 2024-04-01T01:46:07.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.240+0000 7fcf9df84740 1 --2- 172.21.15.70:0/1054654892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 0x5643b0710ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.240+0000 7fcf9df84740 1 -- 172.21.15.70:0/1054654892 >> 172.21.15.70:0/1054654892 conn(0x5643b070bc10 msgr2=0x5643b070c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.241+0000 7fcf9993e640 1 --2- 172.21.15.70:0/2860959838 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcf800a2520 0x7fcf800a49e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009270 tx=0x7fcf8c004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.242+0000 7fcf9df84740 1 -- 172.21.15.70:0/1054654892 shutdown_connections 2024-04-01T01:46:07.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.242+0000 7fcf9df84740 1 -- 172.21.15.70:0/1054654892 wait complete. 2024-04-01T01:46:07.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.243+0000 7fcf9df84740 1 -- start start 2024-04-01T01:46:07.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.243+0000 7fcf9df84740 1 Processor -- start 2024-04-01T01:46:07.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.243+0000 7fcf9df84740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 0x5643b071a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.243+0000 7fcf9df84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b0716200 0x5643b071aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.243+0000 7fcf9df84740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b071ff00 0x5643b07222f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.244+0000 7fcf9df84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5643b0718f70 con 0x5643b0716200 2024-04-01T01:46:07.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.244+0000 7fcf9913d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 0x5643b071a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.244+0000 7fcf9913d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 0x5643b071a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:45140/0 (socket says 172.21.15.70:45140) 2024-04-01T01:46:07.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.244+0000 7fcf9913d640 1 -- 172.21.15.70:0/3690714038 learned_addr learned my addr 172.21.15.70:0/3690714038 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.244+0000 7fcf9993e640 1 --2- 172.21.15.70:0/3690714038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b071ff00 0x5643b07222f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.244+0000 7fcf9893c640 1 --2- 172.21.15.70:0/3690714038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b0716200 0x5643b071aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.244+0000 7fcf9df84740 1 -- 172.21.15.70:0/3690714038 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5643b0613470 con 0x5643b07106a0 2024-04-01T01:46:07.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.244+0000 7fcf9df84740 1 -- 172.21.15.70:0/3690714038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5643b07190f0 con 0x5643b071ff00 2024-04-01T01:46:07.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.245+0000 7fcf9df84740 10 client.0 _ll_register_callbacks cb 0x5643b0630cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:07.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.245+0000 7fcf9913d640 1 -- 172.21.15.70:0/3690714038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b071ff00 msgr2=0x5643b07222f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.245+0000 7fcf9913d640 1 --2- 172.21.15.70:0/3690714038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b071ff00 0x5643b07222f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.245+0000 7fcf9913d640 1 -- 172.21.15.70:0/3690714038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b0716200 msgr2=0x5643b071aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.245+0000 7fcf9913d640 1 --2- 172.21.15.70:0/3690714038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b0716200 0x5643b071aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.245+0000 7fcf9913d640 1 -- 172.21.15.70:0/3690714038 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643b03893f0 con 0x5643b07106a0 2024-04-01T01:46:07.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.246+0000 7fcf9913d640 1 --2- 172.21.15.70:0/3690714038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 0x5643b071a990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fcf900b9bc0 tx=0x7fcf900b9b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5634a675dc991bb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.246+0000 7fcf9893c640 1 --2- 172.21.15.70:0/3690714038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b0716200 0x5643b071aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:46:07.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.247+0000 7fcf7cff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:07.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.247+0000 7fcf7cff9640 1 -- 172.21.15.70:0/3690714038 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf900d5020 con 0x5643b07106a0 2024-04-01T01:46:07.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.247+0000 7fcf7cff9640 1 -- 172.21.15.70:0/3690714038 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fcf900bc640 con 0x5643b07106a0 2024-04-01T01:46:07.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.249+0000 7fcf9df84740 1 -- 172.21.15.70:0/3690714038 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf900bd8f0 con 0x5643b07106a0 2024-04-01T01:46:07.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.252+0000 7fcf7cff9640 1 -- 172.21.15.70:0/3690714038 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcf900bd1f0 con 0x5643b07106a0 2024-04-01T01:46:07.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.253+0000 7fcf9df84740 10 client.29216 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:07.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.253+0000 7fcf9df84740 1 -- 172.21.15.70:0/3690714038 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5643b04537d0 con 0x5643b07106a0 2024-04-01T01:46:07.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.253+0000 7fcf7cff9640 1 -- 172.21.15.70:0/3690714038 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf900bd8f0 con 0x5643b07106a0 2024-04-01T01:46:07.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.253+0000 7fcf7cff9640 1 -- 172.21.15.70:0/3690714038 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fcf900bc0b0 con 0x5643b07106a0 2024-04-01T01:46:07.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.254+0000 7fcf9df84740 10 client.29216 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:07.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.254+0000 7fcf9df84740 10 client.29216 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:46:07.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.254+0000 7fcf9df84740 20 client.29216 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:07.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.254+0000 7fcf9df84740 1 --2- 172.21.15.70:0/3690714038 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5643b0726450 0x5643b0728840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.254+0000 7fcf9df84740 4 client.29216 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:07.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.254+0000 7fcf9df84740 1 -- 172.21.15.70:0/3690714038 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643b06826f0 con 0x5643b0726450 2024-04-01T01:46:07.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.254+0000 7fcf9893c640 1 --2- 172.21.15.70:0/3690714038 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5643b0726450 0x5643b0728840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.255+0000 7fcf9893c640 1 --2- 172.21.15.70:0/3690714038 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5643b0726450 0x5643b0728840 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=69e779f81057b0c server_cookie=147adced4a91bbd0 in_seq=0 out_seq=0 2024-04-01T01:46:07.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.255+0000 7fcf7cff9640 10 client.29216 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:46:07.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.261+0000 7fcf7cff9640 1 -- 172.21.15.70:0/3690714038 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5643b06826f0 con 0x5643b0726450 2024-04-01T01:46:07.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.261+0000 7fcf7cff9640 10 client.29216 handle_command_reply: tid=0 2024-04-01T01:46:07.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.261+0000 7fcf9df84740 1 client.29216 shutdown 2024-04-01T01:46:07.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.263+0000 7fcf9df84740 1 -- 172.21.15.70:0/3690714038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 msgr2=0x5643b071a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.263+0000 7fcf9df84740 1 --2- 172.21.15.70:0/3690714038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 0x5643b071a990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fcf900b9bc0 tx=0x7fcf900b9b00 comp rx=0 tx=0).stop 2024-04-01T01:46:07.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.263+0000 7fcf9df84740 1 -- 172.21.15.70:0/3690714038 shutdown_connections 2024-04-01T01:46:07.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.263+0000 7fcf9df84740 1 --2- 172.21.15.70:0/3690714038 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5643b0726450 0x5643b0728840 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.263+0000 7fcf9df84740 1 --2- 172.21.15.70:0/3690714038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b071ff00 0x5643b07222f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.263+0000 7fcf7cff9640 0 client.29216 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:46:07.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.263+0000 7fcf9df84740 1 --2- 172.21.15.70:0/3690714038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b0716200 0x5643b071aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.263+0000 7fcf9df84740 1 --2- 172.21.15.70:0/3690714038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 0x5643b071a990 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.263+0000 7fcf9df84740 1 -- 172.21.15.70:0/3690714038 >> 172.21.15.70:0/3690714038 conn(0x5643b070bc10 msgr2=0x5643b070af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.264+0000 7fcf9df84740 1 -- 172.21.15.70:0/3690714038 shutdown_connections 2024-04-01T01:46:07.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.264+0000 7fcf9df84740 1 -- 172.21.15.70:0/3690714038 wait complete. 2024-04-01T01:46:07.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.264+0000 7fcf9df84740 20 client.29216 trim_cache size 0 max 16384 2024-04-01T01:46:07.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.292+0000 7fcf9df84740 1 Processor -- start 2024-04-01T01:46:07.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.296+0000 7fcf9df84740 1 -- start start 2024-04-01T01:46:07.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.296+0000 7fcf9df84740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 0x5643b071ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.297+0000 7fcf9df84740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b0716200 0x5643b0706e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.297+0000 7fcf9df84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b071ff00 0x5643b070bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.297+0000 7fcf9df84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcf941b91c0 con 0x5643b071ff00 2024-04-01T01:46:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.297+0000 7fcf9df84740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcf941bad00 con 0x5643b07106a0 2024-04-01T01:46:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.297+0000 7fcf9df84740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcf941b9890 con 0x5643b0716200 2024-04-01T01:46:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.297+0000 7fcf9893c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b0716200 0x5643b0706e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.297+0000 7fcf9893c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b0716200 0x5643b0706e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:35586/0 (socket says 172.21.15.70:35586) 2024-04-01T01:46:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.297+0000 7fcf9893c640 1 -- 172.21.15.70:0/3844641734 learned_addr learned my addr 172.21.15.70:0/3844641734 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.297+0000 7fcf9913d640 1 --2- 172.21.15.70:0/3844641734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 0x5643b071ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.298+0000 7fcf9893c640 1 -- 172.21.15.70:0/3844641734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 msgr2=0x5643b071ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.298+0000 7fcf9893c640 1 --2- 172.21.15.70:0/3844641734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 0x5643b071ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.298+0000 7fcf9893c640 1 -- 172.21.15.70:0/3844641734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b071ff00 msgr2=0x5643b070bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.298+0000 7fcf9893c640 1 --2- 172.21.15.70:0/3844641734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b071ff00 0x5643b070bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.298+0000 7fcf9893c640 1 -- 172.21.15.70:0/3844641734 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643b0728bb0 con 0x5643b0716200 2024-04-01T01:46:07.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.298+0000 7fcf9893c640 1 --2- 172.21.15.70:0/3844641734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b0716200 0x5643b0706e20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fcf840058c0 tx=0x7fcf84005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cdeb9e57183e4c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.298+0000 7fcf7effd640 1 -- 172.21.15.70:0/3844641734 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf8401f480 con 0x5643b0716200 2024-04-01T01:46:07.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.298+0000 7fcf7effd640 1 -- 172.21.15.70:0/3844641734 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf8401f620 con 0x5643b0716200 2024-04-01T01:46:07.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.298+0000 7fcf7effd640 1 -- 172.21.15.70:0/3844641734 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf84004a00 con 0x5643b0716200 2024-04-01T01:46:07.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.300+0000 7fcf9df84740 1 -- 172.21.15.70:0/3844641734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b0716200 msgr2=0x5643b0706e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.300+0000 7fcf9df84740 1 --2- 172.21.15.70:0/3844641734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b0716200 0x5643b0706e20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fcf840058c0 tx=0x7fcf84005e50 comp rx=0 tx=0).stop 2024-04-01T01:46:07.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.300+0000 7fcf9df84740 1 -- 172.21.15.70:0/3844641734 shutdown_connections 2024-04-01T01:46:07.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.300+0000 7fcf9df84740 1 --2- 172.21.15.70:0/3844641734 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b071ff00 0x5643b070bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.300+0000 7fcf9df84740 1 --2- 172.21.15.70:0/3844641734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b0716200 0x5643b0706e20 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.300+0000 7fcf9df84740 1 --2- 172.21.15.70:0/3844641734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b07106a0 0x5643b071ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.300+0000 7fcf9df84740 1 -- 172.21.15.70:0/3844641734 >> 172.21.15.70:0/3844641734 conn(0x5643b070bc10 msgr2=0x5643b0711ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.302+0000 7fcf9df84740 1 -- 172.21.15.70:0/3844641734 shutdown_connections 2024-04-01T01:46:07.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.302+0000 7fcf9df84740 1 -- 172.21.15.70:0/3844641734 wait complete. 2024-04-01T01:46:07.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.303+0000 7fcf9df84740 1 -- start start 2024-04-01T01:46:07.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.303+0000 7fcf9df84740 1 Processor -- start 2024-04-01T01:46:07.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.304+0000 7fcf9df84740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b07106a0 0x5643b07892f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.304+0000 7fcf9df84740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b0716200 0x5643b0789830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.304+0000 7fcf9df84740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b071ff00 0x5643b0738ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.304+0000 7fcf9df84740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcf941ba630 con 0x5643b07106a0 2024-04-01T01:46:07.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.304+0000 7fcf9df84740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcf941b9f60 con 0x5643b0716200 2024-04-01T01:46:07.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.304+0000 7fcf9df84740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcf941b71a0 con 0x5643b071ff00 2024-04-01T01:46:07.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.304+0000 7fcf9993e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b071ff00 0x5643b0738ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.304+0000 7fcf9993e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b071ff00 0x5643b0738ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:35598/0 (socket says 172.21.15.70:35598) 2024-04-01T01:46:07.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.304+0000 7fcf9993e640 1 -- 172.21.15.70:0/1574838416 learned_addr learned my addr 172.21.15.70:0/1574838416 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.305+0000 7fcf9df84740 10 client.0 _ll_register_callbacks cb 0x7fcf900dbf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:07.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.305+0000 7fcf9993e640 1 -- 172.21.15.70:0/1574838416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b0716200 msgr2=0x5643b0789830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:46:07.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.305+0000 7fcf9993e640 1 --2- 172.21.15.70:0/1574838416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b0716200 0x5643b0789830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.305+0000 7fcf9993e640 1 -- 172.21.15.70:0/1574838416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b07106a0 msgr2=0x5643b07892f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:46:07.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.305+0000 7fcf9993e640 1 --2- 172.21.15.70:0/1574838416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b07106a0 0x5643b07892f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.305+0000 7fcf9993e640 1 -- 172.21.15.70:0/1574838416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643b0739190 con 0x5643b071ff00 2024-04-01T01:46:07.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.306+0000 7fcf9993e640 1 --2- 172.21.15.70:0/1574838416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b071ff00 0x5643b0738ba0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c013620 tx=0x7fcf8c009c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=604b8b427e1ab218 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.306+0000 7fcf7cff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:46:07.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.306+0000 7fcf7cff9640 1 -- 172.21.15.70:0/1574838416 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c019070 con 0x5643b071ff00 2024-04-01T01:46:07.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.306+0000 7fcf9df84740 1 -- 172.21.15.70:0/1574838416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf8401f620 con 0x5643b071ff00 2024-04-01T01:46:07.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.306+0000 7fcf7cff9640 1 -- 172.21.15.70:0/1574838416 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fcf8c004b20 con 0x5643b071ff00 2024-04-01T01:46:07.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.307+0000 7fcf7cff9640 1 -- 172.21.15.70:0/1574838416 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcf8c004d20 con 0x5643b071ff00 2024-04-01T01:46:07.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.308+0000 7fcf9df84740 10 client.28999 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:07.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.308+0000 7fcf9df84740 1 -- 172.21.15.70:0/1574838416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5643b079d870 con 0x5643b071ff00 2024-04-01T01:46:07.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.308+0000 7fcf7cff9640 1 -- 172.21.15.70:0/1574838416 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf8c0054d0 con 0x5643b071ff00 2024-04-01T01:46:07.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.309+0000 7fcf7cff9640 1 -- 172.21.15.70:0/1574838416 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fcf8c027030 con 0x5643b071ff00 2024-04-01T01:46:07.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.309+0000 7fcf9df84740 10 client.28999 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:07.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.309+0000 7fcf9df84740 10 client.28999 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:46:07.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.309+0000 7fcf9df84740 20 client.28999 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:07.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.309+0000 7fcf9df84740 1 --2- 172.21.15.70:0/1574838416 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5643b079e280 0x5643b07a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.309+0000 7fcf9df84740 4 client.28999 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:46:07.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.309+0000 7fcf9df84740 1 -- 172.21.15.70:0/1574838416 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fcf8401f620 con 0x5643b079e280 2024-04-01T01:46:07.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.310+0000 7fcf9913d640 1 --2- 172.21.15.70:0/1574838416 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5643b079e280 0x5643b07a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.310+0000 7fcf9913d640 1 --2- 172.21.15.70:0/1574838416 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5643b079e280 0x5643b07a06a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8f845bda374579c5 server_cookie=e630b6b8f7993abb in_seq=0 out_seq=0 2024-04-01T01:46:07.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.310+0000 7fcf7cff9640 10 client.28999 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:46:07.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.310+0000 7fcf7cff9640 1 -- 172.21.15.70:0/1574838416 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fcf8401f620 con 0x5643b079e280 2024-04-01T01:46:07.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.311+0000 7fcf7cff9640 10 client.28999 handle_command_reply: tid=0 2024-04-01T01:46:07.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.311+0000 7fcf9df84740 1 client.28999 shutdown 2024-04-01T01:46:07.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.312+0000 7fcf9df84740 1 -- 172.21.15.70:0/1574838416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b071ff00 msgr2=0x5643b0738ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.312+0000 7fcf9df84740 1 --2- 172.21.15.70:0/1574838416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b071ff00 0x5643b0738ba0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c013620 tx=0x7fcf8c009c70 comp rx=0 tx=0).stop 2024-04-01T01:46:07.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.312+0000 7fcf9df84740 1 -- 172.21.15.70:0/1574838416 shutdown_connections 2024-04-01T01:46:07.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.312+0000 7fcf9df84740 1 --2- 172.21.15.70:0/1574838416 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5643b079e280 0x5643b07a06a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.312+0000 7fcf9df84740 1 --2- 172.21.15.70:0/1574838416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5643b071ff00 0x5643b0738ba0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.312+0000 7fcf9df84740 1 --2- 172.21.15.70:0/1574838416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5643b0716200 0x5643b0789830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.312+0000 7fcf9df84740 1 --2- 172.21.15.70:0/1574838416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643b07106a0 0x5643b07892f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.312+0000 7fcf9df84740 1 -- 172.21.15.70:0/1574838416 >> 172.21.15.70:0/1574838416 conn(0x5643b070bc10 msgr2=0x5643b0706e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.316+0000 7fcf7cff9640 0 client.28999 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:46:07.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.316+0000 7fcf9df84740 1 -- 172.21.15.70:0/1574838416 shutdown_connections 2024-04-01T01:46:07.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.317+0000 7fcf9df84740 1 -- 172.21.15.70:0/1574838416 wait complete. 2024-04-01T01:46:07.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.317+0000 7fcf9df84740 20 client.28999 trim_cache size 0 max 16384 2024-04-01T01:46:07.317 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:46:07.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.319+0000 7fcf9a13f640 1 -- 172.21.15.70:0/2860959838 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcf800a2520 msgr2=0x7fcf800a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.319+0000 7fcf9a13f640 1 --2- 172.21.15.70:0/2860959838 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcf800a2520 0x7fcf800a49e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009270 tx=0x7fcf8c004ab0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.319+0000 7fcf9a13f640 1 -- 172.21.15.70:0/2860959838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf94108750 msgr2=0x7fcf94115550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.319+0000 7fcf9a13f640 1 --2- 172.21.15.70:0/2860959838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf94108750 0x7fcf94115550 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fcf90036040 tx=0x7fcf90005e60 comp rx=0 tx=0).stop 2024-04-01T01:46:07.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.319+0000 7fcf9a13f640 1 -- 172.21.15.70:0/2860959838 shutdown_connections 2024-04-01T01:46:07.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.319+0000 7fcf9a13f640 1 --2- 172.21.15.70:0/2860959838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcf94109ab0 0x7fcf941ada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.319+0000 7fcf9a13f640 1 --2- 172.21.15.70:0/2860959838 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcf800a2520 0x7fcf800a49e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.319+0000 7fcf9a13f640 1 --2- 172.21.15.70:0/2860959838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf94109110 0x7fcf941a2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.319+0000 7fcf9a13f640 1 --2- 172.21.15.70:0/2860959838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf94108750 0x7fcf94115550 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.319+0000 7fcf9a13f640 1 -- 172.21.15.70:0/2860959838 >> 172.21.15.70:0/2860959838 conn(0x7fcf9406e630 msgr2=0x7fcf94070a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.322+0000 7fcf9a13f640 1 -- 172.21.15.70:0/2860959838 shutdown_connections 2024-04-01T01:46:07.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.322+0000 7fcf9a13f640 1 -- 172.21.15.70:0/2860959838 wait complete. 2024-04-01T01:46:07.340 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:46:07.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.517+0000 7f3b61fe5640 1 -- 172.21.15.70:0/3948503930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b5c0ffdd0 msgr2=0x7f3b5c1001f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.517+0000 7f3b61fe5640 1 --2- 172.21.15.70:0/3948503930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b5c0ffdd0 0x7f3b5c1001f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c009860 tx=0x7f3b4c02bb70 comp rx=0 tx=0).stop 2024-04-01T01:46:07.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.518+0000 7f3b61fe5640 1 -- 172.21.15.70:0/3948503930 shutdown_connections 2024-04-01T01:46:07.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.518+0000 7f3b61fe5640 1 --2- 172.21.15.70:0/3948503930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b5c101980 0x7f3b5c108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.518+0000 7f3b61fe5640 1 --2- 172.21.15.70:0/3948503930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b5c100fc0 0x7f3b5c101440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.518+0000 7f3b61fe5640 1 --2- 172.21.15.70:0/3948503930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b5c0ffdd0 0x7f3b5c1001f0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.518+0000 7f3b61fe5640 1 -- 172.21.15.70:0/3948503930 >> 172.21.15.70:0/3948503930 conn(0x7f3b5c0fb340 msgr2=0x7f3b5c0fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.521+0000 7f3b61fe5640 1 -- 172.21.15.70:0/3948503930 shutdown_connections 2024-04-01T01:46:07.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.521+0000 7f3b61fe5640 1 -- 172.21.15.70:0/3948503930 wait complete. 2024-04-01T01:46:07.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.522+0000 7f3b61fe5640 1 Processor -- start 2024-04-01T01:46:07.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.522+0000 7f3b61fe5640 1 -- start start 2024-04-01T01:46:07.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.522+0000 7f3b61fe5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b5c0ffdd0 0x7f3b5c110f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.522+0000 7f3b61fe5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b5c100fc0 0x7f3b5c111440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.522+0000 7f3b61fe5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b5c101980 0x7f3b5c10f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.522+0000 7f3b61fe5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5c109d60 con 0x7f3b5c0ffdd0 2024-04-01T01:46:07.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.522+0000 7f3b61fe5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5c109be0 con 0x7f3b5c101980 2024-04-01T01:46:07.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.522+0000 7f3b61fe5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5c109ee0 con 0x7f3b5c100fc0 2024-04-01T01:46:07.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.523+0000 7f3b5b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b5c0ffdd0 0x7f3b5c110f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.523+0000 7f3b5b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b5c0ffdd0 0x7f3b5c110f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56776/0 (socket says 172.21.15.70:56776) 2024-04-01T01:46:07.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.523+0000 7f3b5b7fe640 1 -- 172.21.15.70:0/271269652 learned_addr learned my addr 172.21.15.70:0/271269652 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.523+0000 7f3b5b7fe640 1 -- 172.21.15.70:0/271269652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b5c100fc0 msgr2=0x7f3b5c111440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:46:07.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.523+0000 7f3b5b7fe640 1 --2- 172.21.15.70:0/271269652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b5c100fc0 0x7f3b5c111440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.523+0000 7f3b5b7fe640 1 -- 172.21.15.70:0/271269652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b5c101980 msgr2=0x7f3b5c10f580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:46:07.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.523+0000 7f3b5b7fe640 1 --2- 172.21.15.70:0/271269652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b5c101980 0x7f3b5c10f580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.523+0000 7f3b5b7fe640 1 -- 172.21.15.70:0/271269652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b5c10fc80 con 0x7f3b5c0ffdd0 2024-04-01T01:46:07.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.524+0000 7f3b5b7fe640 1 --2- 172.21.15.70:0/271269652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b5c0ffdd0 0x7f3b5c110f00 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c000c00 tx=0x7f3b4c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.524+0000 7f3b58ff9640 1 -- 172.21.15.70:0/271269652 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c039040 con 0x7f3b5c0ffdd0 2024-04-01T01:46:07.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.524+0000 7f3b58ff9640 1 -- 172.21.15.70:0/271269652 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b4c042070 con 0x7f3b5c0ffdd0 2024-04-01T01:46:07.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.525+0000 7f3b58ff9640 1 -- 172.21.15.70:0/271269652 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c004920 con 0x7f3b5c0ffdd0 2024-04-01T01:46:07.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.525+0000 7f3b61fe5640 1 -- 172.21.15.70:0/271269652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b5c10ff10 con 0x7f3b5c0ffdd0 2024-04-01T01:46:07.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.525+0000 7f3b61fe5640 1 -- 172.21.15.70:0/271269652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b5c0711a0 con 0x7f3b5c0ffdd0 2024-04-01T01:46:07.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.529+0000 7f3b64bc6740 1 Processor -- start 2024-04-01T01:46:07.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.529+0000 7f3b64bc6740 1 -- start start 2024-04-01T01:46:07.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.529+0000 7f3b64bc6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11490 0x562a7df118b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.529+0000 7f3b64bc6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df11e80 0x562a7df16980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.529+0000 7f3b64bc6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df16ff0 0x562a7df19480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.529+0000 7f3b64bc6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562a7de761b0 con 0x562a7df11490 2024-04-01T01:46:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.529+0000 7f3b64bc6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562a7de1ec00 con 0x562a7df11e80 2024-04-01T01:46:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.529+0000 7f3b64bc6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562a7de72250 con 0x562a7df16ff0 2024-04-01T01:46:07.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.529+0000 7f3b5bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11490 0x562a7df118b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.529+0000 7f3b5bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11490 0x562a7df118b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56788/0 (socket says 172.21.15.70:56788) 2024-04-01T01:46:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.529+0000 7f3b5bfff640 1 -- 172.21.15.70:0/3543603739 learned_addr learned my addr 172.21.15.70:0/3543603739 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.530+0000 7f3b58ff9640 1 -- 172.21.15.70:0/271269652 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f3b4c005030 con 0x7f3b5c0ffdd0 2024-04-01T01:46:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.530+0000 7f3b5bfff640 1 -- 172.21.15.70:0/3543603739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df16ff0 msgr2=0x562a7df19480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.530+0000 7f3b5bfff640 1 --2- 172.21.15.70:0/3543603739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df16ff0 0x562a7df19480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.530+0000 7f3b5bfff640 1 -- 172.21.15.70:0/3543603739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df11e80 msgr2=0x562a7df16980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.530+0000 7f3b5bfff640 1 --2- 172.21.15.70:0/3543603739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df11e80 0x562a7df16980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.530+0000 7f3b5bfff640 1 -- 172.21.15.70:0/3543603739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a7de3e3d0 con 0x562a7df11490 2024-04-01T01:46:07.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.530+0000 7f3b5bfff640 1 --2- 172.21.15.70:0/3543603739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11490 0x562a7df118b0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f3b54009f80 tx=0x7f3b5400e880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b912e8722e30ae server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.531+0000 7f3b58ff9640 1 --2- 172.21.15.70:0/271269652 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3b440a23c0 0x7f3b440a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.531+0000 7f3b41ffb640 1 -- 172.21.15.70:0/3543603739 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b54004d20 con 0x562a7df11490 2024-04-01T01:46:07.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.531+0000 7f3b41ffb640 1 -- 172.21.15.70:0/3543603739 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b54003e20 con 0x562a7df11490 2024-04-01T01:46:07.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.531+0000 7f3b5b7fe640 1 --2- 172.21.15.70:0/271269652 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3b440a23c0 0x7f3b440a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.531+0000 7f3b41ffb640 1 -- 172.21.15.70:0/3543603739 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5400c8a0 con 0x562a7df11490 2024-04-01T01:46:07.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.531+0000 7f3b58ff9640 1 -- 172.21.15.70:0/271269652 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3b4c0ef910 con 0x7f3b5c0ffdd0 2024-04-01T01:46:07.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.532+0000 7f3b64bc6740 1 -- 172.21.15.70:0/3543603739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11490 msgr2=0x562a7df118b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.532+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/3543603739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11490 0x562a7df118b0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f3b54009f80 tx=0x7f3b5400e880 comp rx=0 tx=0).stop 2024-04-01T01:46:07.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.533+0000 7f3b64bc6740 1 -- 172.21.15.70:0/3543603739 shutdown_connections 2024-04-01T01:46:07.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.533+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/3543603739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df16ff0 0x562a7df19480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.533+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/3543603739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df11e80 0x562a7df16980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.533+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/3543603739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11490 0x562a7df118b0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.533+0000 7f3b64bc6740 1 -- 172.21.15.70:0/3543603739 >> 172.21.15.70:0/3543603739 conn(0x562a7df0ca00 msgr2=0x562a7df0ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.533+0000 7f3b5b7fe640 1 --2- 172.21.15.70:0/271269652 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3b440a23c0 0x7f3b440a4880 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c047000 tx=0x7f3b4c0e68d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.533+0000 7f3b64bc6740 1 -- 172.21.15.70:0/3543603739 shutdown_connections 2024-04-01T01:46:07.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.534+0000 7f3b64bc6740 1 -- 172.21.15.70:0/3543603739 wait complete. 2024-04-01T01:46:07.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.534+0000 7f3b64bc6740 1 -- start start 2024-04-01T01:46:07.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.534+0000 7f3b64bc6740 1 Processor -- start 2024-04-01T01:46:07.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.535+0000 7f3b64bc6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df11490 0x562a7df1ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.535+0000 7f3b64bc6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11e80 0x562a7df1bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.535+0000 7f3b64bc6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df16ff0 0x562a7df22fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.535+0000 7f3b64bc6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562a7df19ff0 con 0x562a7df11e80 2024-04-01T01:46:07.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.535+0000 7f3b64bc6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562a7de14470 con 0x562a7df16ff0 2024-04-01T01:46:07.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.535+0000 7f3b64bc6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562a7df1a170 con 0x562a7df11490 2024-04-01T01:46:07.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.535+0000 7f3b64bc6740 10 client.0 _ll_register_callbacks cb 0x562a7de31cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:07.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.536+0000 7f3b5affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11e80 0x562a7df1bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.536+0000 7f3b5affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11e80 0x562a7df1bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56790/0 (socket says 172.21.15.70:56790) 2024-04-01T01:46:07.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.536+0000 7f3b5affd640 1 -- 172.21.15.70:0/1346388267 learned_addr learned my addr 172.21.15.70:0/1346388267 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.536+0000 7f3b5affd640 1 -- 172.21.15.70:0/1346388267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df11490 msgr2=0x562a7df1ba30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:07.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.536+0000 7f3b5affd640 1 --2- 172.21.15.70:0/1346388267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df11490 0x562a7df1ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.537+0000 7f3b5affd640 1 -- 172.21.15.70:0/1346388267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df16ff0 msgr2=0x562a7df22fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:07.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.537+0000 7f3b5affd640 1 --2- 172.21.15.70:0/1346388267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df16ff0 0x562a7df22fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.537+0000 7f3b5affd640 1 -- 172.21.15.70:0/1346388267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a7db8a3f0 con 0x562a7df11e80 2024-04-01T01:46:07.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.537+0000 7f3b5affd640 1 --2- 172.21.15.70:0/1346388267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11e80 0x562a7df1bf70 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f3b500058c0 tx=0x7f3b5000d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ba32b1b1c39f6dd server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.538+0000 7f3b3affd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:46:07.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.538+0000 7f3b3affd640 1 -- 172.21.15.70:0/1346388267 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b50016070 con 0x562a7df11e80 2024-04-01T01:46:07.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.538+0000 7f3b64bc6740 1 -- 172.21.15.70:0/1346388267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b54003e20 con 0x562a7df11e80 2024-04-01T01:46:07.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.538+0000 7f3b3affd640 1 -- 172.21.15.70:0/1346388267 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3b50010660 con 0x562a7df11e80 2024-04-01T01:46:07.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.538+0000 7f3b3affd640 1 -- 172.21.15.70:0/1346388267 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3b50022050 con 0x562a7df11e80 2024-04-01T01:46:07.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.538+0000 7f3b64bc6740 10 client.19155 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:07.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.538+0000 7f3b64bc6740 1 -- 172.21.15.70:0/1346388267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562a7dc547d0 con 0x562a7df11e80 2024-04-01T01:46:07.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.539+0000 7f3b3affd640 1 -- 172.21.15.70:0/1346388267 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b54003e20 con 0x562a7df11e80 2024-04-01T01:46:07.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.539+0000 7f3b3affd640 1 -- 172.21.15.70:0/1346388267 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f3b5000f7b0 con 0x562a7df11e80 2024-04-01T01:46:07.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.539+0000 7f3b64bc6740 10 client.19155 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:07.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.539+0000 7f3b64bc6740 10 client.19155 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:46:07.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.539+0000 7f3b64bc6740 20 client.19155 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:07.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.539+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/1346388267 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562a7df27520 0x562a7df299c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.539+0000 7f3b64bc6740 4 client.19155 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:07.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.539+0000 7f3b64bc6740 1 -- 172.21.15.70:0/1346388267 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a7de836f0 con 0x562a7df27520 2024-04-01T01:46:07.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.539+0000 7f3b5bfff640 1 --2- 172.21.15.70:0/1346388267 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562a7df27520 0x562a7df299c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.540+0000 7f3b3affd640 10 client.19155 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:46:07.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.540+0000 7f3b5bfff640 1 --2- 172.21.15.70:0/1346388267 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562a7df27520 0x562a7df299c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=87c639972ec74176 server_cookie=848dbf23d9cabe45 in_seq=0 out_seq=0 2024-04-01T01:46:07.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.541+0000 7f3b3affd640 1 -- 172.21.15.70:0/1346388267 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562a7de836f0 con 0x562a7df27520 2024-04-01T01:46:07.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.541+0000 7f3b3affd640 10 client.19155 handle_command_reply: tid=0 2024-04-01T01:46:07.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.541+0000 7f3b64bc6740 1 client.19155 shutdown 2024-04-01T01:46:07.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.542+0000 7f3b64bc6740 1 -- 172.21.15.70:0/1346388267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11e80 msgr2=0x562a7df1bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.542+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/1346388267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11e80 0x562a7df1bf70 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f3b500058c0 tx=0x7f3b5000d6b0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.542+0000 7f3b64bc6740 1 -- 172.21.15.70:0/1346388267 shutdown_connections 2024-04-01T01:46:07.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.542+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/1346388267 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562a7df27520 0x562a7df299c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.542+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/1346388267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df16ff0 0x562a7df22fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.542+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/1346388267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11e80 0x562a7df1bf70 secure :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f3b500058c0 tx=0x7f3b5000d6b0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.542+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/1346388267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df11490 0x562a7df1ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.542+0000 7f3b3affd640 0 client.19155 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:46:07.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.542+0000 7f3b64bc6740 1 -- 172.21.15.70:0/1346388267 >> 172.21.15.70:0/1346388267 conn(0x562a7df0ca00 msgr2=0x562a7df0bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.542+0000 7f3b64bc6740 1 -- 172.21.15.70:0/1346388267 shutdown_connections 2024-04-01T01:46:07.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.542+0000 7f3b64bc6740 1 -- 172.21.15.70:0/1346388267 wait complete. 2024-04-01T01:46:07.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.542+0000 7f3b64bc6740 20 client.19155 trim_cache size 0 max 16384 2024-04-01T01:46:07.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.573+0000 7f3b64bc6740 1 Processor -- start 2024-04-01T01:46:07.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.573+0000 7f3b64bc6740 1 -- start start 2024-04-01T01:46:07.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.573+0000 7f3b64bc6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11490 0x562a7df13340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.573+0000 7f3b64bc6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df16ff0 0x562a7df22f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.573+0000 7f3b64bc6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df13880 0x562a7df0b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.574+0000 7f3b64bc6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5c1b09e0 con 0x562a7df11490 2024-04-01T01:46:07.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.574+0000 7f3b5bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11490 0x562a7df13340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.574+0000 7f3b5bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11490 0x562a7df13340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56806/0 (socket says 172.21.15.70:56806) 2024-04-01T01:46:07.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.574+0000 7f3b5bfff640 1 -- 172.21.15.70:0/2044549594 learned_addr learned my addr 172.21.15.70:0/2044549594 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.574+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2044549594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5c1b2520 con 0x562a7df13880 2024-04-01T01:46:07.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.574+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2044549594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5c1b10b0 con 0x562a7df16ff0 2024-04-01T01:46:07.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.574+0000 7f3b5bfff640 1 --2- 172.21.15.70:0/2044549594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df13880 0x562a7df0b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.574+0000 7f3b5bfff640 1 -- 172.21.15.70:0/2044549594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df16ff0 msgr2=0x562a7df22f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:46:07.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.574+0000 7f3b5bfff640 1 --2- 172.21.15.70:0/2044549594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df16ff0 0x562a7df22f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.574+0000 7f3b5bfff640 1 -- 172.21.15.70:0/2044549594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11490 msgr2=0x562a7df13340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.574+0000 7f3b5bfff640 1 --2- 172.21.15.70:0/2044549594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11490 0x562a7df13340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.574+0000 7f3b5bfff640 1 -- 172.21.15.70:0/2044549594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a7df1c8d0 con 0x562a7df13880 2024-04-01T01:46:07.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.575+0000 7f3b5bfff640 1 --2- 172.21.15.70:0/2044549594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df13880 0x562a7df0b4b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x562a7df1a7c0 tx=0x7f3b54004990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9bec9b1725f813d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.576+0000 7f3b417fa640 1 -- 172.21.15.70:0/2044549594 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5401a040 con 0x562a7df13880 2024-04-01T01:46:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.576+0000 7f3b417fa640 1 -- 172.21.15.70:0/2044549594 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b5401b040 con 0x562a7df13880 2024-04-01T01:46:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.576+0000 7f3b417fa640 1 -- 172.21.15.70:0/2044549594 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5400ccb0 con 0x562a7df13880 2024-04-01T01:46:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.577+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2044549594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df13880 msgr2=0x562a7df0b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.577+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/2044549594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df13880 0x562a7df0b4b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x562a7df1a7c0 tx=0x7f3b54004990 comp rx=0 tx=0).stop 2024-04-01T01:46:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.577+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2044549594 shutdown_connections 2024-04-01T01:46:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.577+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/2044549594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df13880 0x562a7df0b4b0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.577+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/2044549594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df16ff0 0x562a7df22f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.577+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/2044549594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df11490 0x562a7df13340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.577+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2044549594 >> 172.21.15.70:0/2044549594 conn(0x562a7df0ca00 msgr2=0x562a7df07bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.577+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2044549594 shutdown_connections 2024-04-01T01:46:07.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.577+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2044549594 wait complete. 2024-04-01T01:46:07.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.578+0000 7f3b64bc6740 1 -- start start 2024-04-01T01:46:07.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.578+0000 7f3b64bc6740 1 Processor -- start 2024-04-01T01:46:07.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.579+0000 7f3b64bc6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df11490 0x562a7df228b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.579+0000 7f3b64bc6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df13880 0x562a7df22df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.579+0000 7f3b5bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df11490 0x562a7df228b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.579+0000 7f3b5bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df11490 0x562a7df228b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:45218/0 (socket says 172.21.15.70:45218) 2024-04-01T01:46:07.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.579+0000 7f3b5bfff640 1 -- 172.21.15.70:0/2840237828 learned_addr learned my addr 172.21.15.70:0/2840237828 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.579+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/2840237828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df16ff0 0x562a7df20f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.579+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2840237828 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5c1b1e50 con 0x562a7df16ff0 2024-04-01T01:46:07.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.579+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2840237828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5c1b1780 con 0x562a7df11490 2024-04-01T01:46:07.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.579+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2840237828 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5c1ae9c0 con 0x562a7df13880 2024-04-01T01:46:07.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.579+0000 7f3b64bc6740 10 client.0 _ll_register_callbacks cb 0x7f3b5000b830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:07.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.579+0000 7f3b5affd640 1 --2- 172.21.15.70:0/2840237828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df13880 0x562a7df22df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.579+0000 7f3b5bfff640 1 --2- 172.21.15.70:0/2840237828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df16ff0 0x562a7df20f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.580+0000 7f3b5bfff640 1 -- 172.21.15.70:0/2840237828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df13880 msgr2=0x562a7df22df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.580+0000 7f3b5bfff640 1 --2- 172.21.15.70:0/2840237828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df13880 0x562a7df22df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.580+0000 7f3b5bfff640 1 -- 172.21.15.70:0/2840237828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df16ff0 msgr2=0x562a7df20f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.580+0000 7f3b5bfff640 1 --2- 172.21.15.70:0/2840237828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df16ff0 0x562a7df20f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.580+0000 7f3b5bfff640 1 -- 172.21.15.70:0/2840237828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a7df21760 con 0x562a7df11490 2024-04-01T01:46:07.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.580+0000 7f3b5affd640 1 --2- 172.21.15.70:0/2840237828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df13880 0x562a7df22df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:46:07.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.580+0000 7f3b5bfff640 1 --2- 172.21.15.70:0/2840237828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df11490 0x562a7df228b0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f3b54009620 tx=0x7f3b54009360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7146897656bd1c31 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.580+0000 7f3b3affd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:07.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.580+0000 7f3b3affd640 1 -- 172.21.15.70:0/2840237828 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b54039040 con 0x562a7df11490 2024-04-01T01:46:07.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.580+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2840237828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b5401b040 con 0x562a7df11490 2024-04-01T01:46:07.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.580+0000 7f3b3affd640 1 -- 172.21.15.70:0/2840237828 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3b5400ca30 con 0x562a7df11490 2024-04-01T01:46:07.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.580+0000 7f3b3affd640 1 -- 172.21.15.70:0/2840237828 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3b54047050 con 0x562a7df11490 2024-04-01T01:46:07.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.580+0000 7f3b64bc6740 10 client.29225 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:07.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.580+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2840237828 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562a7df8e380 con 0x562a7df11490 2024-04-01T01:46:07.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.581+0000 7f3b3affd640 1 -- 172.21.15.70:0/2840237828 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3b5401b040 con 0x562a7df11490 2024-04-01T01:46:07.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.581+0000 7f3b3affd640 1 -- 172.21.15.70:0/2840237828 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f3b5400b040 con 0x562a7df11490 2024-04-01T01:46:07.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.581+0000 7f3b64bc6740 10 client.29225 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:07.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.581+0000 7f3b64bc6740 10 client.29225 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:46:07.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.581+0000 7f3b64bc6740 20 client.29225 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:07.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.581+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/2840237828 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562a7df8ea40 0x562a7df90e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.581+0000 7f3b64bc6740 4 client.29225 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:46:07.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.581+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2840237828 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a7de7da40 con 0x562a7df8ea40 2024-04-01T01:46:07.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.582+0000 7f3b5affd640 1 --2- 172.21.15.70:0/2840237828 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562a7df8ea40 0x562a7df90e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.582+0000 7f3b5affd640 1 --2- 172.21.15.70:0/2840237828 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562a7df8ea40 0x562a7df90e60 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=90435941aa26b0f4 server_cookie=c19ec10d34b0cbda in_seq=0 out_seq=0 2024-04-01T01:46:07.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.582+0000 7f3b3affd640 10 client.29225 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:46:07.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.583+0000 7f3b3affd640 1 -- 172.21.15.70:0/2840237828 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a7de7da40 con 0x562a7df8ea40 2024-04-01T01:46:07.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.583+0000 7f3b3affd640 10 client.29225 handle_command_reply: tid=0 2024-04-01T01:46:07.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.583+0000 7f3b64bc6740 1 client.29225 shutdown 2024-04-01T01:46:07.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.583+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2840237828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df11490 msgr2=0x562a7df228b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.583+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/2840237828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df11490 0x562a7df228b0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f3b54009620 tx=0x7f3b54009360 comp rx=0 tx=0).stop 2024-04-01T01:46:07.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.583+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2840237828 shutdown_connections 2024-04-01T01:46:07.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.583+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/2840237828 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x562a7df8ea40 0x562a7df90e60 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.583+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/2840237828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562a7df16ff0 0x562a7df20f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.583+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/2840237828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a7df13880 0x562a7df22df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.583+0000 7f3b64bc6740 1 --2- 172.21.15.70:0/2840237828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a7df11490 0x562a7df228b0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.583+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2840237828 >> 172.21.15.70:0/2840237828 conn(0x562a7df0ca00 msgr2=0x562a7df0fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.583+0000 7f3b3affd640 0 client.29225 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:46:07.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.583+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2840237828 shutdown_connections 2024-04-01T01:46:07.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.584+0000 7f3b64bc6740 1 -- 172.21.15.70:0/2840237828 wait complete. 2024-04-01T01:46:07.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.584+0000 7f3b64bc6740 20 client.29225 trim_cache size 0 max 16384 2024-04-01T01:46:07.586 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:46:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.585+0000 7f3b61fe5640 1 -- 172.21.15.70:0/271269652 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3b440a23c0 msgr2=0x7f3b440a4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.585+0000 7f3b61fe5640 1 --2- 172.21.15.70:0/271269652 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3b440a23c0 0x7f3b440a4880 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c047000 tx=0x7f3b4c0e68d0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.585+0000 7f3b61fe5640 1 -- 172.21.15.70:0/271269652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b5c0ffdd0 msgr2=0x7f3b5c110f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.585+0000 7f3b61fe5640 1 --2- 172.21.15.70:0/271269652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b5c0ffdd0 0x7f3b5c110f00 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c000c00 tx=0x7f3b4c004270 comp rx=0 tx=0).stop 2024-04-01T01:46:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.585+0000 7f3b61fe5640 1 -- 172.21.15.70:0/271269652 shutdown_connections 2024-04-01T01:46:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.585+0000 7f3b61fe5640 1 --2- 172.21.15.70:0/271269652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b5c101980 0x7f3b5c10f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.586+0000 7f3b61fe5640 1 --2- 172.21.15.70:0/271269652 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3b440a23c0 0x7f3b440a4880 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.586+0000 7f3b61fe5640 1 --2- 172.21.15.70:0/271269652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b5c100fc0 0x7f3b5c111440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.586+0000 7f3b61fe5640 1 --2- 172.21.15.70:0/271269652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b5c0ffdd0 0x7f3b5c110f00 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.586+0000 7f3b61fe5640 1 -- 172.21.15.70:0/271269652 >> 172.21.15.70:0/271269652 conn(0x7f3b5c0fb340 msgr2=0x7f3b5c104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.586+0000 7f3b61fe5640 1 -- 172.21.15.70:0/271269652 shutdown_connections 2024-04-01T01:46:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.586+0000 7f3b61fe5640 1 -- 172.21.15.70:0/271269652 wait complete. 2024-04-01T01:46:07.607 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:46:07.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.794+0000 7fbd93eb5640 1 -- 172.21.15.70:0/350877227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd8c108f60 msgr2=0x7fbd8c1093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.794+0000 7fbd93eb5640 1 --2- 172.21.15.70:0/350877227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd8c108f60 0x7fbd8c1093e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fbd8401a910 tx=0x7fbd8403cca0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.794+0000 7fbd93eb5640 1 -- 172.21.15.70:0/350877227 shutdown_connections 2024-04-01T01:46:07.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.794+0000 7fbd93eb5640 1 --2- 172.21.15.70:0/350877227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd8c109920 0x7fbd8c1101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.794+0000 7fbd93eb5640 1 --2- 172.21.15.70:0/350877227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd8c108f60 0x7fbd8c1093e0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.794+0000 7fbd93eb5640 1 --2- 172.21.15.70:0/350877227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd8c107d70 0x7fbd8c108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.794+0000 7fbd93eb5640 1 -- 172.21.15.70:0/350877227 >> 172.21.15.70:0/350877227 conn(0x7fbd8c076d10 msgr2=0x7fbd8c079150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.795+0000 7fbd93eb5640 1 -- 172.21.15.70:0/350877227 shutdown_connections 2024-04-01T01:46:07.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.797+0000 7fbd93eb5640 1 -- 172.21.15.70:0/350877227 wait complete. 2024-04-01T01:46:07.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.797+0000 7fbd93eb5640 1 Processor -- start 2024-04-01T01:46:07.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.797+0000 7fbd93eb5640 1 -- start start 2024-04-01T01:46:07.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.798+0000 7fbd93eb5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd8c107d70 0x7fbd8c07a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.798+0000 7fbd93eb5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd8c108f60 0x7fbd8c07a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.798+0000 7fbd93eb5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd8c109920 0x7fbd8c081250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.798+0000 7fbd93eb5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c06dcd0 con 0x7fbd8c108f60 2024-04-01T01:46:07.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.798+0000 7fbd93eb5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c06db50 con 0x7fbd8c109920 2024-04-01T01:46:07.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.798+0000 7fbd93eb5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c06de50 con 0x7fbd8c107d70 2024-04-01T01:46:07.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.798+0000 7fbd91450640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd8c108f60 0x7fbd8c07a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.798+0000 7fbd91450640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd8c108f60 0x7fbd8c07a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56824/0 (socket says 172.21.15.70:56824) 2024-04-01T01:46:07.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.798+0000 7fbd91450640 1 -- 172.21.15.70:0/3440658772 learned_addr learned my addr 172.21.15.70:0/3440658772 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.798+0000 7fbd92452640 1 --2- 172.21.15.70:0/3440658772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd8c109920 0x7fbd8c081250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.798+0000 7fbd91c51640 1 --2- 172.21.15.70:0/3440658772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd8c107d70 0x7fbd8c07a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.799+0000 7fbd92452640 1 -- 172.21.15.70:0/3440658772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd8c107d70 msgr2=0x7fbd8c07a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.799+0000 7fbd92452640 1 --2- 172.21.15.70:0/3440658772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd8c107d70 0x7fbd8c07a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.799+0000 7fbd92452640 1 -- 172.21.15.70:0/3440658772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd8c108f60 msgr2=0x7fbd8c07a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.799+0000 7fbd92452640 1 --2- 172.21.15.70:0/3440658772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd8c108f60 0x7fbd8c07a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.799+0000 7fbd92452640 1 -- 172.21.15.70:0/3440658772 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd8c0818c0 con 0x7fbd8c109920 2024-04-01T01:46:07.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.800+0000 7fbd92452640 1 --2- 172.21.15.70:0/3440658772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd8c109920 0x7fbd8c081250 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c00d680 tx=0x7fbd7c00db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.802+0000 7fbd82ffd640 1 -- 172.21.15.70:0/3440658772 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c017070 con 0x7fbd8c109920 2024-04-01T01:46:07.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.802+0000 7fbd82ffd640 1 -- 172.21.15.70:0/3440658772 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd7c009820 con 0x7fbd8c109920 2024-04-01T01:46:07.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.802+0000 7fbd82ffd640 1 -- 172.21.15.70:0/3440658772 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c004cf0 con 0x7fbd8c109920 2024-04-01T01:46:07.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.802+0000 7fbd93eb5640 1 -- 172.21.15.70:0/3440658772 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd8c081bb0 con 0x7fbd8c109920 2024-04-01T01:46:07.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.802+0000 7fbd93eb5640 1 -- 172.21.15.70:0/3440658772 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd8c082130 con 0x7fbd8c109920 2024-04-01T01:46:07.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.806+0000 7fbd96b74740 1 Processor -- start 2024-04-01T01:46:07.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.807+0000 7fbd82ffd640 1 -- 172.21.15.70:0/3440658772 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fbd7c0099d0 con 0x7fbd8c109920 2024-04-01T01:46:07.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.808+0000 7fbd96b74740 1 -- start start 2024-04-01T01:46:07.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.808+0000 7fbd96b74740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a8e490 0x55cc11a8e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.808+0000 7fbd96b74740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a8ee80 0x55cc11a93980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.808+0000 7fbd96b74740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a93ff0 0x55cc11a96480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.808+0000 7fbd96b74740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cc119f31b0 con 0x55cc11a93ff0 2024-04-01T01:46:07.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.808+0000 7fbd96b74740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55cc1199bc00 con 0x55cc11a8e490 2024-04-01T01:46:07.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.808+0000 7fbd96b74740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cc119ef250 con 0x55cc11a8ee80 2024-04-01T01:46:07.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.809+0000 7fbd91450640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a8e490 0x55cc11a8e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.809+0000 7fbd91450640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a8e490 0x55cc11a8e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:45242/0 (socket says 172.21.15.70:45242) 2024-04-01T01:46:07.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.809+0000 7fbd91450640 1 -- 172.21.15.70:0/1855539881 learned_addr learned my addr 172.21.15.70:0/1855539881 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.809+0000 7fbd91450640 1 --2- 172.21.15.70:0/1855539881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a93ff0 0x55cc11a96480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.809+0000 7fbd91450640 1 -- 172.21.15.70:0/1855539881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a8ee80 msgr2=0x55cc11a93980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:46:07.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.809+0000 7fbd91450640 1 --2- 172.21.15.70:0/1855539881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a8ee80 0x55cc11a93980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.809+0000 7fbd91450640 1 -- 172.21.15.70:0/1855539881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a93ff0 msgr2=0x55cc11a96480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.809+0000 7fbd91450640 1 --2- 172.21.15.70:0/1855539881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a93ff0 0x55cc11a96480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.809+0000 7fbd91450640 1 -- 172.21.15.70:0/1855539881 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc119bb3d0 con 0x55cc11a8e490 2024-04-01T01:46:07.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.810+0000 7fbd82ffd640 1 --2- 172.21.15.70:0/3440658772 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbd780a25f0 0x7fbd780a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.810+0000 7fbd91450640 1 --2- 172.21.15.70:0/1855539881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a8e490 0x55cc11a8e8b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbd84005a90 tx=0x7fbd84005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb9ca595e4066cfe server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.810+0000 7fbd82ffd640 1 -- 172.21.15.70:0/3440658772 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fbd7c0ce3d0 con 0x7fbd8c109920 2024-04-01T01:46:07.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.810+0000 7fbd77fff640 1 -- 172.21.15.70:0/1855539881 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd84057070 con 0x55cc11a8e490 2024-04-01T01:46:07.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.810+0000 7fbd77fff640 1 -- 172.21.15.70:0/1855539881 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd8403d850 con 0x55cc11a8e490 2024-04-01T01:46:07.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.811+0000 7fbd77fff640 1 -- 172.21.15.70:0/1855539881 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd84005380 con 0x55cc11a8e490 2024-04-01T01:46:07.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.811+0000 7fbd96b74740 1 -- 172.21.15.70:0/1855539881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a8e490 msgr2=0x55cc11a8e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.811+0000 7fbd96b74740 1 --2- 172.21.15.70:0/1855539881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a8e490 0x55cc11a8e8b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbd84005a90 tx=0x7fbd84005a20 comp rx=0 tx=0).stop 2024-04-01T01:46:07.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.812+0000 7fbd96b74740 1 -- 172.21.15.70:0/1855539881 shutdown_connections 2024-04-01T01:46:07.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.812+0000 7fbd96b74740 1 --2- 172.21.15.70:0/1855539881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a93ff0 0x55cc11a96480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.812+0000 7fbd96b74740 1 --2- 172.21.15.70:0/1855539881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a8ee80 0x55cc11a93980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.812+0000 7fbd96b74740 1 --2- 172.21.15.70:0/1855539881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a8e490 0x55cc11a8e8b0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.812+0000 7fbd96b74740 1 -- 172.21.15.70:0/1855539881 >> 172.21.15.70:0/1855539881 conn(0x55cc11a89a00 msgr2=0x55cc11a89e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.812+0000 7fbd91c51640 1 --2- 172.21.15.70:0/3440658772 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbd780a25f0 0x7fbd780a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.813+0000 7fbd91c51640 1 --2- 172.21.15.70:0/3440658772 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbd780a25f0 0x7fbd780a4ab0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fbd880070b0 tx=0x7fbd88007040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.813+0000 7fbd96b74740 1 -- 172.21.15.70:0/1855539881 shutdown_connections 2024-04-01T01:46:07.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.813+0000 7fbd96b74740 1 -- 172.21.15.70:0/1855539881 wait complete. 2024-04-01T01:46:07.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.813+0000 7fbd96b74740 1 -- start start 2024-04-01T01:46:07.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.814+0000 7fbd96b74740 1 Processor -- start 2024-04-01T01:46:07.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.814+0000 7fbd96b74740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a8e490 0x55cc11a98b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.814+0000 7fbd96b74740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a93ff0 0x55cc11a99070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.814+0000 7fbd96b74740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a9e0a0 0x55cc11aa0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.814+0000 7fbd96b74740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cc11a96f90 con 0x55cc11a8e490 2024-04-01T01:46:07.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.814+0000 7fbd96b74740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55cc11991470 con 0x55cc11a9e0a0 2024-04-01T01:46:07.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.814+0000 7fbd96b74740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cc11a97110 con 0x55cc11a93ff0 2024-04-01T01:46:07.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.814+0000 7fbd96b74740 10 client.0 _ll_register_callbacks cb 0x55cc119aecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:07.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.815+0000 7fbd91c51640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a8e490 0x55cc11a98b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.815+0000 7fbd91c51640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a8e490 0x55cc11a98b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56840/0 (socket says 172.21.15.70:56840) 2024-04-01T01:46:07.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.815+0000 7fbd91c51640 1 -- 172.21.15.70:0/2351723702 learned_addr learned my addr 172.21.15.70:0/2351723702 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.815+0000 7fbd91c51640 1 -- 172.21.15.70:0/2351723702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a93ff0 msgr2=0x55cc11a99070 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:07.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.815+0000 7fbd91c51640 1 --2- 172.21.15.70:0/2351723702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a93ff0 0x55cc11a99070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.815+0000 7fbd91c51640 1 -- 172.21.15.70:0/2351723702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a9e0a0 msgr2=0x55cc11aa0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.815+0000 7fbd91c51640 1 --2- 172.21.15.70:0/2351723702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a9e0a0 0x55cc11aa0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.815+0000 7fbd91c51640 1 -- 172.21.15.70:0/2351723702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc117073f0 con 0x55cc11a8e490 2024-04-01T01:46:07.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.815+0000 7fbd91c51640 1 --2- 172.21.15.70:0/2351723702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a8e490 0x55cc11a98b30 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x55cc11a8f8f0 tx=0x7fbd88004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc818e7a09ab7c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.816+0000 7fbd757fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:46:07.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.816+0000 7fbd757fa640 1 -- 172.21.15.70:0/2351723702 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd88027060 con 0x55cc11a8e490 2024-04-01T01:46:07.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.816+0000 7fbd96b74740 1 -- 172.21.15.70:0/2351723702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd8403d850 con 0x55cc11a8e490 2024-04-01T01:46:07.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.816+0000 7fbd757fa640 1 -- 172.21.15.70:0/2351723702 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fbd8801d030 con 0x55cc11a8e490 2024-04-01T01:46:07.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.818+0000 7fbd757fa640 1 -- 172.21.15.70:0/2351723702 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbd88027060 con 0x55cc11a8e490 2024-04-01T01:46:07.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.819+0000 7fbd96b74740 10 client.19170 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:07.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.819+0000 7fbd96b74740 1 -- 172.21.15.70:0/2351723702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cc117d17d0 con 0x55cc11a8e490 2024-04-01T01:46:07.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.821+0000 7fbd757fa640 1 -- 172.21.15.70:0/2351723702 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd8403d850 con 0x55cc11a8e490 2024-04-01T01:46:07.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.822+0000 7fbd757fa640 1 -- 172.21.15.70:0/2351723702 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fbd88016870 con 0x55cc11a8e490 2024-04-01T01:46:07.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.822+0000 7fbd96b74740 10 client.19170 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:07.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.822+0000 7fbd96b74740 10 client.19170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:07.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.822+0000 7fbd96b74740 20 client.19170 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:07.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.822+0000 7fbd96b74740 1 --2- 172.21.15.70:0/2351723702 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cc11aa4f10 0x55cc11aa73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.822+0000 7fbd96b74740 4 client.19170 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:07.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.822+0000 7fbd96b74740 1 -- 172.21.15.70:0/2351723702 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc11a006f0 con 0x55cc11aa4f10 2024-04-01T01:46:07.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.823+0000 7fbd91450640 1 --2- 172.21.15.70:0/2351723702 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cc11aa4f10 0x55cc11aa73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.824+0000 7fbd91450640 1 --2- 172.21.15.70:0/2351723702 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cc11aa4f10 0x55cc11aa73d0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35e48b0a87374b1d server_cookie=14c23eaae637f560 in_seq=0 out_seq=0 2024-04-01T01:46:07.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.824+0000 7fbd757fa640 10 client.19170 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:07.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.825+0000 7fbd757fa640 1 -- 172.21.15.70:0/2351723702 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cc11a006f0 con 0x55cc11aa4f10 2024-04-01T01:46:07.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.825+0000 7fbd757fa640 10 client.19170 handle_command_reply: tid=0 2024-04-01T01:46:07.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.825+0000 7fbd96b74740 1 client.19170 shutdown 2024-04-01T01:46:07.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.827+0000 7fbd96b74740 1 -- 172.21.15.70:0/2351723702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a8e490 msgr2=0x55cc11a98b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.827+0000 7fbd96b74740 1 --2- 172.21.15.70:0/2351723702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a8e490 0x55cc11a98b30 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x55cc11a8f8f0 tx=0x7fbd88004af0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.828+0000 7fbd96b74740 1 -- 172.21.15.70:0/2351723702 shutdown_connections 2024-04-01T01:46:07.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.828+0000 7fbd96b74740 1 --2- 172.21.15.70:0/2351723702 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cc11aa4f10 0x55cc11aa73d0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.828+0000 7fbd96b74740 1 --2- 172.21.15.70:0/2351723702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a9e0a0 0x55cc11aa0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.828+0000 7fbd96b74740 1 --2- 172.21.15.70:0/2351723702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a93ff0 0x55cc11a99070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.828+0000 7fbd96b74740 1 --2- 172.21.15.70:0/2351723702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a8e490 0x55cc11a98b30 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.828+0000 7fbd96b74740 1 -- 172.21.15.70:0/2351723702 >> 172.21.15.70:0/2351723702 conn(0x55cc11a89a00 msgr2=0x55cc11a88d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.829+0000 7fbd757fa640 0 client.19170 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:07.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.829+0000 7fbd96b74740 1 -- 172.21.15.70:0/2351723702 shutdown_connections 2024-04-01T01:46:07.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.829+0000 7fbd96b74740 1 -- 172.21.15.70:0/2351723702 wait complete. 2024-04-01T01:46:07.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.830+0000 7fbd96b74740 20 client.19170 trim_cache size 0 max 16384 2024-04-01T01:46:07.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.854+0000 7fbd96b74740 1 Processor -- start 2024-04-01T01:46:07.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.854+0000 7fbd96b74740 1 -- start start 2024-04-01T01:46:07.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.854+0000 7fbd96b74740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a8e490 0x55cc11a894c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.854+0000 7fbd96b74740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a93ff0 0x55cc11a8d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.854+0000 7fbd96b74740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a9e0a0 0x55cc11a97e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.854+0000 7fbd96b74740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c1b0cc0 con 0x55cc11a93ff0 2024-04-01T01:46:07.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.854+0000 7fbd96b74740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c1b2800 con 0x55cc11a8e490 2024-04-01T01:46:07.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.854+0000 7fbd96b74740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c1b1390 con 0x55cc11a9e0a0 2024-04-01T01:46:07.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.854+0000 7fbd91450640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a93ff0 0x55cc11a8d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.854+0000 7fbd91450640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a93ff0 0x55cc11a8d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56842/0 (socket says 172.21.15.70:56842) 2024-04-01T01:46:07.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.855+0000 7fbd91450640 1 -- 172.21.15.70:0/2786212492 learned_addr learned my addr 172.21.15.70:0/2786212492 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.855+0000 7fbd92452640 1 --2- 172.21.15.70:0/2786212492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a9e0a0 0x55cc11a97e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.855+0000 7fbd92452640 1 -- 172.21.15.70:0/2786212492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a8e490 msgr2=0x55cc11a894c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:07.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.855+0000 7fbd92452640 1 --2- 172.21.15.70:0/2786212492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a8e490 0x55cc11a894c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.855+0000 7fbd92452640 1 -- 172.21.15.70:0/2786212492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a93ff0 msgr2=0x55cc11a8d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.855+0000 7fbd92452640 1 --2- 172.21.15.70:0/2786212492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a93ff0 0x55cc11a8d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.855+0000 7fbd92452640 1 -- 172.21.15.70:0/2786212492 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc11a8fff0 con 0x55cc11a9e0a0 2024-04-01T01:46:07.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.856+0000 7fbd92452640 1 --2- 172.21.15.70:0/2786212492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a9e0a0 0x55cc11a97e00 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c009610 tx=0x7fbd7c096e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb4914d268cb7fd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.856+0000 7fbd777fe640 1 -- 172.21.15.70:0/2786212492 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c020040 con 0x55cc11a9e0a0 2024-04-01T01:46:07.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.856+0000 7fbd777fe640 1 -- 172.21.15.70:0/2786212492 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd7c09aa60 con 0x55cc11a9e0a0 2024-04-01T01:46:07.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.856+0000 7fbd777fe640 1 -- 172.21.15.70:0/2786212492 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c096a10 con 0x55cc11a9e0a0 2024-04-01T01:46:07.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.859+0000 7fbd96b74740 1 -- 172.21.15.70:0/2786212492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a9e0a0 msgr2=0x55cc11a97e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.859+0000 7fbd96b74740 1 --2- 172.21.15.70:0/2786212492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a9e0a0 0x55cc11a97e00 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c009610 tx=0x7fbd7c096e30 comp rx=0 tx=0).stop 2024-04-01T01:46:07.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.859+0000 7fbd96b74740 1 -- 172.21.15.70:0/2786212492 shutdown_connections 2024-04-01T01:46:07.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.859+0000 7fbd96b74740 1 --2- 172.21.15.70:0/2786212492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a9e0a0 0x55cc11a97e00 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.859+0000 7fbd96b74740 1 --2- 172.21.15.70:0/2786212492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a93ff0 0x55cc11a8d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.859+0000 7fbd96b74740 1 --2- 172.21.15.70:0/2786212492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a8e490 0x55cc11a894c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.859+0000 7fbd96b74740 1 -- 172.21.15.70:0/2786212492 >> 172.21.15.70:0/2786212492 conn(0x55cc11a89a00 msgr2=0x55cc11a8cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.862+0000 7fbd96b74740 1 -- 172.21.15.70:0/2786212492 shutdown_connections 2024-04-01T01:46:07.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.863+0000 7fbd96b74740 1 -- 172.21.15.70:0/2786212492 wait complete. 2024-04-01T01:46:07.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.863+0000 7fbd96b74740 1 -- start start 2024-04-01T01:46:07.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.863+0000 7fbd96b74740 1 Processor -- start 2024-04-01T01:46:07.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.864+0000 7fbd96b74740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a8e490 0x55cc11ab5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.864+0000 7fbd96b74740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a93ff0 0x55cc11ab62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.864+0000 7fbd96b74740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a9e0a0 0x55cc11ab7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.864+0000 7fbd96b74740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c1b2130 con 0x55cc11a8e490 2024-04-01T01:46:07.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.864+0000 7fbd96b74740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c1b1a60 con 0x55cc11a93ff0 2024-04-01T01:46:07.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.864+0000 7fbd96b74740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c1aec40 con 0x55cc11a9e0a0 2024-04-01T01:46:07.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.864+0000 7fbd91450640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a93ff0 0x55cc11ab62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.864+0000 7fbd91450640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a93ff0 0x55cc11ab62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:45274/0 (socket says 172.21.15.70:45274) 2024-04-01T01:46:07.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.864+0000 7fbd91450640 1 -- 172.21.15.70:0/3452754119 learned_addr learned my addr 172.21.15.70:0/3452754119 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:07.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.864+0000 7fbd96b74740 10 client.0 _ll_register_callbacks cb 0x7fbd88017880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:07.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.864+0000 7fbd92452640 1 --2- 172.21.15.70:0/3452754119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a9e0a0 0x55cc11ab7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.864+0000 7fbd91c51640 1 --2- 172.21.15.70:0/3452754119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a8e490 0x55cc11ab5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.865+0000 7fbd91450640 1 -- 172.21.15.70:0/3452754119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a9e0a0 msgr2=0x55cc11ab7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.865+0000 7fbd91450640 1 --2- 172.21.15.70:0/3452754119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a9e0a0 0x55cc11ab7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.865+0000 7fbd91450640 1 -- 172.21.15.70:0/3452754119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a8e490 msgr2=0x55cc11ab5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:07.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.865+0000 7fbd91450640 1 --2- 172.21.15.70:0/3452754119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a8e490 0x55cc11ab5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.865+0000 7fbd91450640 1 -- 172.21.15.70:0/3452754119 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc11ab7860 con 0x55cc11a93ff0 2024-04-01T01:46:07.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.865+0000 7fbd91c51640 1 --2- 172.21.15.70:0/3452754119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a8e490 0x55cc11ab5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:46:07.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.865+0000 7fbd91450640 1 --2- 172.21.15.70:0/3452754119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a93ff0 0x55cc11ab62b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbd8401a440 tx=0x7fbd84000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdb8cc4667f5efad server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:07.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.865+0000 7fbd757fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:07.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.865+0000 7fbd757fa640 1 -- 172.21.15.70:0/3452754119 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd84061070 con 0x55cc11a93ff0 2024-04-01T01:46:07.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.866+0000 7fbd757fa640 1 -- 172.21.15.70:0/3452754119 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fbd8405dbd0 con 0x55cc11a93ff0 2024-04-01T01:46:07.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.866+0000 7fbd96b74740 1 -- 172.21.15.70:0/3452754119 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd7c09aa60 con 0x55cc11a93ff0 2024-04-01T01:46:07.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.871+0000 7fbd757fa640 1 -- 172.21.15.70:0/3452754119 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbd8406d050 con 0x55cc11a93ff0 2024-04-01T01:46:07.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.871+0000 7fbd96b74740 10 client.29237 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:07.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.871+0000 7fbd96b74740 1 -- 172.21.15.70:0/3452754119 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cc11b0adc0 con 0x55cc11a93ff0 2024-04-01T01:46:07.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.871+0000 7fbd757fa640 1 -- 172.21.15.70:0/3452754119 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd7c09aa60 con 0x55cc11a93ff0 2024-04-01T01:46:07.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.872+0000 7fbd757fa640 1 -- 172.21.15.70:0/3452754119 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fbd84005020 con 0x55cc11a93ff0 2024-04-01T01:46:07.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.872+0000 7fbd96b74740 10 client.29237 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:07.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.872+0000 7fbd96b74740 10 client.29237 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:07.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.872+0000 7fbd96b74740 20 client.29237 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:07.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.872+0000 7fbd96b74740 1 --2- 172.21.15.70:0/3452754119 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cc11b0b260 0x55cc11b0d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:07.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.872+0000 7fbd96b74740 4 client.29237 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:46:07.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.872+0000 7fbd96b74740 1 -- 172.21.15.70:0/3452754119 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cc119faa40 con 0x55cc11b0b260 2024-04-01T01:46:07.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.873+0000 7fbd91c51640 1 --2- 172.21.15.70:0/3452754119 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cc11b0b260 0x55cc11b0d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:07.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.874+0000 7fbd91c51640 1 --2- 172.21.15.70:0/3452754119 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cc11b0b260 0x55cc11b0d650 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e163dd47411d04da server_cookie=c70a5217d1011df2 in_seq=0 out_seq=0 2024-04-01T01:46:07.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.874+0000 7fbd757fa640 10 client.29237 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:07.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.874+0000 7fbd757fa640 1 -- 172.21.15.70:0/3452754119 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cc119faa40 con 0x55cc11b0b260 2024-04-01T01:46:07.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.874+0000 7fbd757fa640 10 client.29237 handle_command_reply: tid=0 2024-04-01T01:46:07.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.874+0000 7fbd96b74740 1 client.29237 shutdown 2024-04-01T01:46:07.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.875+0000 7fbd96b74740 1 -- 172.21.15.70:0/3452754119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a93ff0 msgr2=0x55cc11ab62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.875+0000 7fbd96b74740 1 --2- 172.21.15.70:0/3452754119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a93ff0 0x55cc11ab62b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbd8401a440 tx=0x7fbd84000bc0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.875+0000 7fbd96b74740 1 -- 172.21.15.70:0/3452754119 shutdown_connections 2024-04-01T01:46:07.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.875+0000 7fbd96b74740 1 --2- 172.21.15.70:0/3452754119 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cc11b0b260 0x55cc11b0d650 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.875+0000 7fbd757fa640 0 client.29237 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:07.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.875+0000 7fbd96b74740 1 --2- 172.21.15.70:0/3452754119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cc11a9e0a0 0x55cc11ab7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.875+0000 7fbd96b74740 1 --2- 172.21.15.70:0/3452754119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cc11a93ff0 0x55cc11ab62b0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.875+0000 7fbd96b74740 1 --2- 172.21.15.70:0/3452754119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cc11a8e490 0x55cc11ab5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.875+0000 7fbd96b74740 1 -- 172.21.15.70:0/3452754119 >> 172.21.15.70:0/3452754119 conn(0x55cc11a89a00 msgr2=0x55cc11a8d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.876+0000 7fbd96b74740 1 -- 172.21.15.70:0/3452754119 shutdown_connections 2024-04-01T01:46:07.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.876+0000 7fbd96b74740 1 -- 172.21.15.70:0/3452754119 wait complete. 2024-04-01T01:46:07.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.876+0000 7fbd96b74740 20 client.29237 trim_cache size 0 max 16384 2024-04-01T01:46:07.877 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:46:07.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.877+0000 7fbd93eb5640 1 -- 172.21.15.70:0/3440658772 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbd780a25f0 msgr2=0x7fbd780a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.877+0000 7fbd93eb5640 1 --2- 172.21.15.70:0/3440658772 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbd780a25f0 0x7fbd780a4ab0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fbd880070b0 tx=0x7fbd88007040 comp rx=0 tx=0).stop 2024-04-01T01:46:07.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.877+0000 7fbd93eb5640 1 -- 172.21.15.70:0/3440658772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd8c109920 msgr2=0x7fbd8c081250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:07.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.877+0000 7fbd93eb5640 1 --2- 172.21.15.70:0/3440658772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd8c109920 0x7fbd8c081250 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c00d680 tx=0x7fbd7c00db40 comp rx=0 tx=0).stop 2024-04-01T01:46:07.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.877+0000 7fbd93eb5640 1 -- 172.21.15.70:0/3440658772 shutdown_connections 2024-04-01T01:46:07.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.877+0000 7fbd93eb5640 1 --2- 172.21.15.70:0/3440658772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd8c109920 0x7fbd8c081250 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.877+0000 7fbd93eb5640 1 --2- 172.21.15.70:0/3440658772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbd8c108f60 0x7fbd8c07a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.877+0000 7fbd93eb5640 1 --2- 172.21.15.70:0/3440658772 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbd780a25f0 0x7fbd780a4ab0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.877+0000 7fbd93eb5640 1 --2- 172.21.15.70:0/3440658772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd8c107d70 0x7fbd8c07a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:07.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.877+0000 7fbd93eb5640 1 -- 172.21.15.70:0/3440658772 >> 172.21.15.70:0/3440658772 conn(0x7fbd8c076d10 msgr2=0x7fbd8c0707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:07.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.878+0000 7fbd93eb5640 1 -- 172.21.15.70:0/3440658772 shutdown_connections 2024-04-01T01:46:07.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:07.878+0000 7fbd93eb5640 1 -- 172.21.15.70:0/3440658772 wait complete. 2024-04-01T01:46:08.903 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:46:08.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:46:08.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:46:08.945 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:08 smithi070 ceph-mon[32179]: pgmap v738: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 8.7 MiB/s wr, 833 op/s 2024-04-01T01:46:08.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:08 smithi189 ceph-mon[35753]: pgmap v738: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 8.7 MiB/s wr, 833 op/s 2024-04-01T01:46:09.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:08 smithi081 ceph-mon[35142]: pgmap v738: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 8.7 MiB/s wr, 833 op/s 2024-04-01T01:46:09.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.090+0000 7f304759e640 1 -- 172.21.15.70:0/1570266605 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3048072b60 msgr2=0x7f3048072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.090+0000 7f304759e640 1 --2- 172.21.15.70:0/1570266605 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3048072b60 0x7f3048072f80 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f3038009900 tx=0x7f303802bc30 comp rx=0 tx=0).stop 2024-04-01T01:46:09.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.090+0000 7f304759e640 1 -- 172.21.15.70:0/1570266605 shutdown_connections 2024-04-01T01:46:09.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.090+0000 7f304759e640 1 --2- 172.21.15.70:0/1570266605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30481097e0 0x7f304810e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.091+0000 7f304759e640 1 --2- 172.21.15.70:0/1570266605 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3048108e20 0x7f30481092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.091+0000 7f304759e640 1 --2- 172.21.15.70:0/1570266605 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3048072b60 0x7f3048072f80 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.091+0000 7f304759e640 1 -- 172.21.15.70:0/1570266605 >> 172.21.15.70:0/1570266605 conn(0x7f304806e630 msgr2=0x7f3048070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:09.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.091+0000 7f304759e640 1 -- 172.21.15.70:0/1570266605 shutdown_connections 2024-04-01T01:46:09.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.091+0000 7f304759e640 1 -- 172.21.15.70:0/1570266605 wait complete. 2024-04-01T01:46:09.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.092+0000 7f304759e640 1 Processor -- start 2024-04-01T01:46:09.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.093+0000 7f304759e640 1 -- start start 2024-04-01T01:46:09.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.093+0000 7f304759e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3048072b60 0x7f30481a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.093+0000 7f304759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3048108e20 0x7f30481a69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.093+0000 7f304759e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30481097e0 0x7f30481ada50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.093+0000 7f304759e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f304811bfa0 con 0x7f3048108e20 2024-04-01T01:46:09.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.093+0000 7f304759e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f304811be20 con 0x7f30481097e0 2024-04-01T01:46:09.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.093+0000 7f304759e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f304811c120 con 0x7f3048072b60 2024-04-01T01:46:09.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.093+0000 7f3045d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3048108e20 0x7f30481a69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.093+0000 7f304659c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3048072b60 0x7f30481a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.093+0000 7f304659c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3048072b60 0x7f30481a64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:35716/0 (socket says 172.21.15.70:35716) 2024-04-01T01:46:09.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.093+0000 7f304659c640 1 -- 172.21.15.70:0/3107565076 learned_addr learned my addr 172.21.15.70:0/3107565076 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:09.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.093+0000 7f3045d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3048108e20 0x7f30481a69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56878/0 (socket says 172.21.15.70:56878) 2024-04-01T01:46:09.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.094+0000 7f304659c640 1 -- 172.21.15.70:0/3107565076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30481097e0 msgr2=0x7f30481ada50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:46:09.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.094+0000 7f304659c640 1 --2- 172.21.15.70:0/3107565076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30481097e0 0x7f30481ada50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.094+0000 7f304659c640 1 -- 172.21.15.70:0/3107565076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3048108e20 msgr2=0x7f30481a69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.094+0000 7f304659c640 1 --2- 172.21.15.70:0/3107565076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3048108e20 0x7f30481a69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.094+0000 7f304659c640 1 -- 172.21.15.70:0/3107565076 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30481ae0c0 con 0x7f3048072b60 2024-04-01T01:46:09.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.098+0000 7f304659c640 1 --2- 172.21.15.70:0/3107565076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3048072b60 0x7f30481a64a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f3038009850 tx=0x7f30380042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:09.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.098+0000 7f30377fe640 1 -- 172.21.15.70:0/3107565076 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3038043020 con 0x7f3048072b60 2024-04-01T01:46:09.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.098+0000 7f30377fe640 1 -- 172.21.15.70:0/3107565076 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30380054c0 con 0x7f3048072b60 2024-04-01T01:46:09.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.098+0000 7f304759e640 1 -- 172.21.15.70:0/3107565076 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30481ae350 con 0x7f3048072b60 2024-04-01T01:46:09.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.098+0000 7f30377fe640 1 -- 172.21.15.70:0/3107565076 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3038003d80 con 0x7f3048072b60 2024-04-01T01:46:09.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.098+0000 7f304759e640 1 -- 172.21.15.70:0/3107565076 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30481ae7e0 con 0x7f3048072b60 2024-04-01T01:46:09.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.102+0000 7f30377fe640 1 -- 172.21.15.70:0/3107565076 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f3038046020 con 0x7f3048072b60 2024-04-01T01:46:09.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.102+0000 7f30377fe640 1 --2- 172.21.15.70:0/3107565076 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30300a25a0 0x7f30300a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.103+0000 7f30377fe640 1 -- 172.21.15.70:0/3107565076 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f30380f0e60 con 0x7f3048072b60 2024-04-01T01:46:09.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.104+0000 7f3045d9b640 1 --2- 172.21.15.70:0/3107565076 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30300a25a0 0x7f30300a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.106+0000 7f3045d9b640 1 --2- 172.21.15.70:0/3107565076 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30300a25a0 0x7f30300a4a60 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f30481a79b0 tx=0x7f303c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:09.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.109+0000 7f304f7de740 1 Processor -- start 2024-04-01T01:46:09.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.109+0000 7f304f7de740 1 -- start start 2024-04-01T01:46:09.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.109+0000 7f304f7de740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e0cbf0 0x5592e7e0d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.109+0000 7f304f7de740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0d5e0 0x5592e7e120e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.109+0000 7f304f7de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e12750 0x5592e7e14be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.109+0000 7f304f7de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5592e7d0e0b0 con 0x5592e7e12750 2024-04-01T01:46:09.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.109+0000 7f304f7de740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5592e7d70ea0 con 0x5592e7e0d5e0 2024-04-01T01:46:09.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.109+0000 7f304f7de740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5592e7d19c00 con 0x5592e7e0cbf0 2024-04-01T01:46:09.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.110+0000 7f3046d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e12750 0x5592e7e14be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.110+0000 7f3046d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e12750 0x5592e7e14be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56884/0 (socket says 172.21.15.70:56884) 2024-04-01T01:46:09.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.110+0000 7f3046d9d640 1 -- 172.21.15.70:0/2854118131 learned_addr learned my addr 172.21.15.70:0/2854118131 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:09.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.110+0000 7f3046d9d640 1 -- 172.21.15.70:0/2854118131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e0cbf0 msgr2=0x5592e7e0d010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:09.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.110+0000 7f3046d9d640 1 --2- 172.21.15.70:0/2854118131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e0cbf0 0x5592e7e0d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.110+0000 7f3046d9d640 1 -- 172.21.15.70:0/2854118131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0d5e0 msgr2=0x5592e7e120e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:46:09.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.110+0000 7f3046d9d640 1 --2- 172.21.15.70:0/2854118131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0d5e0 0x5592e7e120e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.110+0000 7f3046d9d640 1 -- 172.21.15.70:0/2854118131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592e7b4d7e0 con 0x5592e7e12750 2024-04-01T01:46:09.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.111+0000 7f3046d9d640 1 --2- 172.21.15.70:0/2854118131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e12750 0x5592e7e14be0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f3040009e30 tx=0x7f304000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0ef9f57040444b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:09.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.111+0000 7f3034ff9640 1 -- 172.21.15.70:0/2854118131 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040003b70 con 0x5592e7e12750 2024-04-01T01:46:09.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.112+0000 7f3034ff9640 1 -- 172.21.15.70:0/2854118131 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3040003d10 con 0x5592e7e12750 2024-04-01T01:46:09.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.112+0000 7f3034ff9640 1 -- 172.21.15.70:0/2854118131 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040010830 con 0x5592e7e12750 2024-04-01T01:46:09.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.113+0000 7f304f7de740 1 -- 172.21.15.70:0/2854118131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e12750 msgr2=0x5592e7e14be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.113+0000 7f304f7de740 1 --2- 172.21.15.70:0/2854118131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e12750 0x5592e7e14be0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f3040009e30 tx=0x7f304000e720 comp rx=0 tx=0).stop 2024-04-01T01:46:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.113+0000 7f304f7de740 1 -- 172.21.15.70:0/2854118131 shutdown_connections 2024-04-01T01:46:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.113+0000 7f304f7de740 1 --2- 172.21.15.70:0/2854118131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e12750 0x5592e7e14be0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.113+0000 7f304f7de740 1 --2- 172.21.15.70:0/2854118131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0d5e0 0x5592e7e120e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.113+0000 7f304f7de740 1 --2- 172.21.15.70:0/2854118131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e0cbf0 0x5592e7e0d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.113+0000 7f304f7de740 1 -- 172.21.15.70:0/2854118131 >> 172.21.15.70:0/2854118131 conn(0x5592e7e08160 msgr2=0x5592e7e08560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.113+0000 7f304f7de740 1 -- 172.21.15.70:0/2854118131 shutdown_connections 2024-04-01T01:46:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.113+0000 7f304f7de740 1 -- 172.21.15.70:0/2854118131 wait complete. 2024-04-01T01:46:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.114+0000 7f304f7de740 1 -- start start 2024-04-01T01:46:09.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.114+0000 7f304f7de740 1 Processor -- start 2024-04-01T01:46:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.114+0000 7f304f7de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e0cbf0 0x5592e7e28050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.114+0000 7f304f7de740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0d5e0 0x5592e7e28590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.114+0000 7f304f7de740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e12750 0x5592e7e2f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.114+0000 7f304659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e0cbf0 0x5592e7e28050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.114+0000 7f304659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e0cbf0 0x5592e7e28050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56886/0 (socket says 172.21.15.70:56886) 2024-04-01T01:46:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.114+0000 7f304659c640 1 -- 172.21.15.70:0/3378381169 learned_addr learned my addr 172.21.15.70:0/3378381169 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.114+0000 7f304f7de740 1 -- 172.21.15.70:0/3378381169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5592e7d0f3d0 con 0x5592e7e0cbf0 2024-04-01T01:46:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.114+0000 7f304f7de740 1 -- 172.21.15.70:0/3378381169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5592e7d6cf40 con 0x5592e7e0d5e0 2024-04-01T01:46:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.114+0000 7f304f7de740 1 -- 172.21.15.70:0/3378381169 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5592e7e26760 con 0x5592e7e12750 2024-04-01T01:46:09.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.114+0000 7f304f7de740 10 client.0 _ll_register_callbacks cb 0x5592e7d2cb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:09.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.116+0000 7f3045d9b640 1 --2- 172.21.15.70:0/3378381169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0d5e0 0x5592e7e28590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.116+0000 7f3046d9d640 1 --2- 172.21.15.70:0/3378381169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e12750 0x5592e7e2f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.116+0000 7f3045d9b640 1 -- 172.21.15.70:0/3378381169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e12750 msgr2=0x5592e7e2f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:09.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.116+0000 7f3045d9b640 1 --2- 172.21.15.70:0/3378381169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e12750 0x5592e7e2f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.116+0000 7f3045d9b640 1 -- 172.21.15.70:0/3378381169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e0cbf0 msgr2=0x5592e7e28050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:09.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.116+0000 7f3045d9b640 1 --2- 172.21.15.70:0/3378381169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e0cbf0 0x5592e7e28050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.116+0000 7f3045d9b640 1 -- 172.21.15.70:0/3378381169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592e7badf70 con 0x5592e7e0d5e0 2024-04-01T01:46:09.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.117+0000 7f3045d9b640 1 --2- 172.21.15.70:0/3378381169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0d5e0 0x5592e7e28590 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x5592e7e0e050 tx=0x7f303c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=838514ed9e2fe79a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:09.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.117+0000 7f302dffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:09.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.117+0000 7f302dffb640 1 -- 172.21.15.70:0/3378381169 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c01ca70 con 0x5592e7e0d5e0 2024-04-01T01:46:09.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.117+0000 7f302dffb640 1 -- 172.21.15.70:0/3378381169 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f303c01cc10 con 0x5592e7e0d5e0 2024-04-01T01:46:09.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.118+0000 7f304659c640 1 --2- 172.21.15.70:0/3378381169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e0cbf0 0x5592e7e28050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:46:09.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.118+0000 7f304f7de740 1 -- 172.21.15.70:0/3378381169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3040003d10 con 0x5592e7e0d5e0 2024-04-01T01:46:09.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.118+0000 7f302dffb640 1 -- 172.21.15.70:0/3378381169 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f303c01ce10 con 0x5592e7e0d5e0 2024-04-01T01:46:09.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.118+0000 7f304f7de740 10 client.29243 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:09.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.118+0000 7f304f7de740 1 -- 172.21.15.70:0/3378381169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5592e7a45170 con 0x5592e7e0d5e0 2024-04-01T01:46:09.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.118+0000 7f302dffb640 1 -- 172.21.15.70:0/3378381169 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f303c028930 con 0x5592e7e0d5e0 2024-04-01T01:46:09.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.118+0000 7f302dffb640 1 -- 172.21.15.70:0/3378381169 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f303c0328f0 con 0x5592e7e0d5e0 2024-04-01T01:46:09.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.118+0000 7f304f7de740 10 client.29243 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:09.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.118+0000 7f304f7de740 10 client.29243 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:09.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.118+0000 7f304f7de740 20 client.29243 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:09.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.118+0000 7f304f7de740 1 --2- 172.21.15.70:0/3378381169 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5592e7e33a50 0x5592e7e35eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.118+0000 7f304f7de740 4 client.29243 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:09.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.119+0000 7f304f7de740 1 -- 172.21.15.70:0/3378381169 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3040003d10 con 0x5592e7e33a50 2024-04-01T01:46:09.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.120+0000 7f304659c640 1 --2- 172.21.15.70:0/3378381169 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5592e7e33a50 0x5592e7e35eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.121+0000 7f304659c640 1 --2- 172.21.15.70:0/3378381169 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5592e7e33a50 0x5592e7e35eb0 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f44580342cbffb5 server_cookie=276f5ce5e2bb383c in_seq=0 out_seq=0 2024-04-01T01:46:09.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.121+0000 7f302dffb640 10 client.29243 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:09.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.121+0000 7f302dffb640 1 -- 172.21.15.70:0/3378381169 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f3040003d10 con 0x5592e7e33a50 2024-04-01T01:46:09.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.121+0000 7f302dffb640 10 client.29243 handle_command_reply: tid=0 2024-04-01T01:46:09.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.121+0000 7f304f7de740 1 client.29243 shutdown 2024-04-01T01:46:09.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.122+0000 7f304f7de740 1 -- 172.21.15.70:0/3378381169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0d5e0 msgr2=0x5592e7e28590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.122+0000 7f304f7de740 1 --2- 172.21.15.70:0/3378381169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0d5e0 0x5592e7e28590 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x5592e7e0e050 tx=0x7f303c004730 comp rx=0 tx=0).stop 2024-04-01T01:46:09.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.122+0000 7f304f7de740 1 -- 172.21.15.70:0/3378381169 shutdown_connections 2024-04-01T01:46:09.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.122+0000 7f304f7de740 1 --2- 172.21.15.70:0/3378381169 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5592e7e33a50 0x5592e7e35eb0 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.122+0000 7f304f7de740 1 --2- 172.21.15.70:0/3378381169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e12750 0x5592e7e2f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.122+0000 7f304f7de740 1 --2- 172.21.15.70:0/3378381169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0d5e0 0x5592e7e28590 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.123+0000 7f304f7de740 1 --2- 172.21.15.70:0/3378381169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e0cbf0 0x5592e7e28050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.123+0000 7f304f7de740 1 -- 172.21.15.70:0/3378381169 >> 172.21.15.70:0/3378381169 conn(0x5592e7e08160 msgr2=0x5592e7e072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:09.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.123+0000 7f302dffb640 0 client.29243 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:09.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.123+0000 7f304f7de740 1 -- 172.21.15.70:0/3378381169 shutdown_connections 2024-04-01T01:46:09.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.123+0000 7f304f7de740 1 -- 172.21.15.70:0/3378381169 wait complete. 2024-04-01T01:46:09.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.124+0000 7f304f7de740 20 client.29243 trim_cache size 0 max 16384 2024-04-01T01:46:09.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.158+0000 7f304f7de740 1 Processor -- start 2024-04-01T01:46:09.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.159+0000 7f304f7de740 1 -- start start 2024-04-01T01:46:09.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.159+0000 7f304f7de740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0cbf0 0x5592e7e0eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.159+0000 7f304f7de740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e0d5e0 0x5592e7e0f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.159+0000 7f304f7de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e12750 0x5592e7e2f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.159+0000 7f304f7de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f30481b06f0 con 0x5592e7e12750 2024-04-01T01:46:09.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.159+0000 7f304f7de740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f30481b2230 con 0x5592e7e0cbf0 2024-04-01T01:46:09.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.159+0000 7f304f7de740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f30481b0dc0 con 0x5592e7e0d5e0 2024-04-01T01:46:09.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.160+0000 7f3046d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e12750 0x5592e7e2f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.160+0000 7f3046d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e12750 0x5592e7e2f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56902/0 (socket says 172.21.15.70:56902) 2024-04-01T01:46:09.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.160+0000 7f3046d9d640 1 -- 172.21.15.70:0/260712018 learned_addr learned my addr 172.21.15.70:0/260712018 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:09.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.160+0000 7f304659c640 1 --2- 172.21.15.70:0/260712018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0cbf0 0x5592e7e0eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.161+0000 7f3046d9d640 1 -- 172.21.15.70:0/260712018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e0d5e0 msgr2=0x5592e7e0f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:09.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.161+0000 7f3046d9d640 1 --2- 172.21.15.70:0/260712018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e0d5e0 0x5592e7e0f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.161+0000 7f3046d9d640 1 -- 172.21.15.70:0/260712018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0cbf0 msgr2=0x5592e7e0eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:09.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.161+0000 7f3046d9d640 1 --2- 172.21.15.70:0/260712018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0cbf0 0x5592e7e0eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.161+0000 7f3046d9d640 1 -- 172.21.15.70:0/260712018 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592e7e0d9c0 con 0x5592e7e12750 2024-04-01T01:46:09.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.161+0000 7f3046d9d640 1 --2- 172.21.15.70:0/260712018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e12750 0x5592e7e2f590 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f3040009970 tx=0x7f3040005a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e357de8dae4c78bc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:09.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.161+0000 7f302ffff640 1 -- 172.21.15.70:0/260712018 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040027020 con 0x5592e7e12750 2024-04-01T01:46:09.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.162+0000 7f302ffff640 1 -- 172.21.15.70:0/260712018 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3040009af0 con 0x5592e7e12750 2024-04-01T01:46:09.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.162+0000 7f302ffff640 1 -- 172.21.15.70:0/260712018 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f304001d040 con 0x5592e7e12750 2024-04-01T01:46:09.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.163+0000 7f304f7de740 1 -- 172.21.15.70:0/260712018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e12750 msgr2=0x5592e7e2f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.163+0000 7f304f7de740 1 --2- 172.21.15.70:0/260712018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e12750 0x5592e7e2f590 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f3040009970 tx=0x7f3040005a10 comp rx=0 tx=0).stop 2024-04-01T01:46:09.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.164+0000 7f304f7de740 1 -- 172.21.15.70:0/260712018 shutdown_connections 2024-04-01T01:46:09.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.164+0000 7f304f7de740 1 --2- 172.21.15.70:0/260712018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e12750 0x5592e7e2f590 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.164+0000 7f304f7de740 1 --2- 172.21.15.70:0/260712018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e0d5e0 0x5592e7e0f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.164+0000 7f304f7de740 1 --2- 172.21.15.70:0/260712018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0cbf0 0x5592e7e0eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.164+0000 7f304f7de740 1 -- 172.21.15.70:0/260712018 >> 172.21.15.70:0/260712018 conn(0x5592e7e08160 msgr2=0x5592e7e08540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:09.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.164+0000 7f304f7de740 1 -- 172.21.15.70:0/260712018 shutdown_connections 2024-04-01T01:46:09.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.165+0000 7f304f7de740 1 -- 172.21.15.70:0/260712018 wait complete. 2024-04-01T01:46:09.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.165+0000 7f304f7de740 1 -- start start 2024-04-01T01:46:09.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.165+0000 7f304f7de740 1 Processor -- start 2024-04-01T01:46:09.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.166+0000 7f304f7de740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0cbf0 0x5592e7e2d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.166+0000 7f304f7de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e0d5e0 0x5592e7e2d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.166+0000 7f304f7de740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e12750 0x5592e7e2e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.166+0000 7f304f7de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f30481b1b60 con 0x5592e7e0d5e0 2024-04-01T01:46:09.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.166+0000 7f304f7de740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f30481b1490 con 0x5592e7e0cbf0 2024-04-01T01:46:09.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.166+0000 7f304f7de740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3048118320 con 0x5592e7e12750 2024-04-01T01:46:09.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.166+0000 7f304f7de740 10 client.0 _ll_register_callbacks cb 0x7f303c00e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:09.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.166+0000 7f3045d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e0d5e0 0x5592e7e2d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.166+0000 7f3045d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e0d5e0 0x5592e7e2d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56916/0 (socket says 172.21.15.70:56916) 2024-04-01T01:46:09.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.166+0000 7f3045d9b640 1 -- 172.21.15.70:0/2132120724 learned_addr learned my addr 172.21.15.70:0/2132120724 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:09.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.166+0000 7f304659c640 1 --2- 172.21.15.70:0/2132120724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0cbf0 0x5592e7e2d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.167+0000 7f3046d9d640 1 --2- 172.21.15.70:0/2132120724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e12750 0x5592e7e2e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.167+0000 7f3045d9b640 1 -- 172.21.15.70:0/2132120724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e12750 msgr2=0x5592e7e2e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:09.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.167+0000 7f3045d9b640 1 --2- 172.21.15.70:0/2132120724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e12750 0x5592e7e2e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.167+0000 7f3045d9b640 1 -- 172.21.15.70:0/2132120724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0cbf0 msgr2=0x5592e7e2d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:09.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.167+0000 7f3045d9b640 1 --2- 172.21.15.70:0/2132120724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0cbf0 0x5592e7e2d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.167+0000 7f3045d9b640 1 -- 172.21.15.70:0/2132120724 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592e7e2ed70 con 0x5592e7e0d5e0 2024-04-01T01:46:09.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.167+0000 7f3045d9b640 1 --2- 172.21.15.70:0/2132120724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e0d5e0 0x5592e7e2d770 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f303c009730 tx=0x7f303c032650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a50086bafa0490 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:09.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.168+0000 7f302dffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:46:09.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.168+0000 7f302dffb640 1 -- 172.21.15.70:0/2132120724 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c01d410 con 0x5592e7e0d5e0 2024-04-01T01:46:09.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.168+0000 7f304f7de740 1 -- 172.21.15.70:0/2132120724 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3040009af0 con 0x5592e7e0d5e0 2024-04-01T01:46:09.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.168+0000 7f302dffb640 1 -- 172.21.15.70:0/2132120724 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f303c032a50 con 0x5592e7e0d5e0 2024-04-01T01:46:09.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.168+0000 7f302dffb640 1 -- 172.21.15.70:0/2132120724 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f303c045050 con 0x5592e7e0d5e0 2024-04-01T01:46:09.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.168+0000 7f304f7de740 10 client.19188 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:09.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.168+0000 7f304f7de740 1 -- 172.21.15.70:0/2132120724 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5592e7e9a430 con 0x5592e7e0d5e0 2024-04-01T01:46:09.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.169+0000 7f302dffb640 1 -- 172.21.15.70:0/2132120724 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3040009af0 con 0x5592e7e0d5e0 2024-04-01T01:46:09.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.169+0000 7f302dffb640 1 -- 172.21.15.70:0/2132120724 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f303c01dae0 con 0x5592e7e0d5e0 2024-04-01T01:46:09.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.169+0000 7f304f7de740 10 client.19188 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:09.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.169+0000 7f304f7de740 10 client.19188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:09.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.169+0000 7f304f7de740 20 client.19188 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:09.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.170+0000 7f304f7de740 1 --2- 172.21.15.70:0/2132120724 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5592e7e9adb0 0x5592e7e9d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.170+0000 7f304f7de740 4 client.19188 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:46:09.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.170+0000 7f304f7de740 1 -- 172.21.15.70:0/2132120724 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5592e7d7e3e0 con 0x5592e7e9adb0 2024-04-01T01:46:09.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.171+0000 7f304659c640 1 --2- 172.21.15.70:0/2132120724 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5592e7e9adb0 0x5592e7e9d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.171+0000 7f302dffb640 10 client.19188 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:09.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.171+0000 7f304659c640 1 --2- 172.21.15.70:0/2132120724 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5592e7e9adb0 0x5592e7e9d1d0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7172cf288b682b7 server_cookie=1b2fc8d2cf946388 in_seq=0 out_seq=0 2024-04-01T01:46:09.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.173+0000 7f302dffb640 1 -- 172.21.15.70:0/2132120724 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5592e7d7e3e0 con 0x5592e7e9adb0 2024-04-01T01:46:09.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.173+0000 7f302dffb640 10 client.19188 handle_command_reply: tid=0 2024-04-01T01:46:09.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.173+0000 7f304f7de740 1 client.19188 shutdown 2024-04-01T01:46:09.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.174+0000 7f304f7de740 1 -- 172.21.15.70:0/2132120724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e0d5e0 msgr2=0x5592e7e2d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.174+0000 7f304f7de740 1 --2- 172.21.15.70:0/2132120724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e0d5e0 0x5592e7e2d770 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f303c009730 tx=0x7f303c032650 comp rx=0 tx=0).stop 2024-04-01T01:46:09.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.174+0000 7f304f7de740 1 -- 172.21.15.70:0/2132120724 shutdown_connections 2024-04-01T01:46:09.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.174+0000 7f304f7de740 1 --2- 172.21.15.70:0/2132120724 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5592e7e9adb0 0x5592e7e9d1d0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.174+0000 7f304f7de740 1 --2- 172.21.15.70:0/2132120724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592e7e12750 0x5592e7e2e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.174+0000 7f304f7de740 1 --2- 172.21.15.70:0/2132120724 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5592e7e0d5e0 0x5592e7e2d770 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.174+0000 7f304f7de740 1 --2- 172.21.15.70:0/2132120724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5592e7e0cbf0 0x5592e7e2d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.174+0000 7f304f7de740 1 -- 172.21.15.70:0/2132120724 >> 172.21.15.70:0/2132120724 conn(0x5592e7e08160 msgr2=0x5592e7e0eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:09.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.174+0000 7f302dffb640 0 client.19188 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:09.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.175+0000 7f304f7de740 1 -- 172.21.15.70:0/2132120724 shutdown_connections 2024-04-01T01:46:09.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.175+0000 7f304f7de740 1 -- 172.21.15.70:0/2132120724 wait complete. 2024-04-01T01:46:09.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.175+0000 7f304f7de740 20 client.19188 trim_cache size 0 max 16384 2024-04-01T01:46:09.176 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:46:09.176 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:46:09.176 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "9b7827a3-286a-4656-b244-e395a0697248", 2024-04-01T01:46:09.176 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:46:09.176 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:46:09.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.177+0000 7f304759e640 1 -- 172.21.15.70:0/3107565076 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30300a25a0 msgr2=0x7f30300a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.177+0000 7f304759e640 1 --2- 172.21.15.70:0/3107565076 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30300a25a0 0x7f30300a4a60 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f30481a79b0 tx=0x7f303c005f90 comp rx=0 tx=0).stop 2024-04-01T01:46:09.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.177+0000 7f304759e640 1 -- 172.21.15.70:0/3107565076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3048072b60 msgr2=0x7f30481a64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.177+0000 7f304759e640 1 --2- 172.21.15.70:0/3107565076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3048072b60 0x7f30481a64a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f3038009850 tx=0x7f30380042f0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.177+0000 7f304759e640 1 -- 172.21.15.70:0/3107565076 shutdown_connections 2024-04-01T01:46:09.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.177+0000 7f304759e640 1 --2- 172.21.15.70:0/3107565076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f30481097e0 0x7f30481ada50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.177+0000 7f304759e640 1 --2- 172.21.15.70:0/3107565076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3048108e20 0x7f30481a69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.177+0000 7f304759e640 1 --2- 172.21.15.70:0/3107565076 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f30300a25a0 0x7f30300a4a60 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.177+0000 7f304759e640 1 --2- 172.21.15.70:0/3107565076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3048072b60 0x7f30481a64a0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.177+0000 7f304759e640 1 -- 172.21.15.70:0/3107565076 >> 172.21.15.70:0/3107565076 conn(0x7f304806e630 msgr2=0x7f304810a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:09.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.177+0000 7f304759e640 1 -- 172.21.15.70:0/3107565076 shutdown_connections 2024-04-01T01:46:09.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.177+0000 7f304759e640 1 -- 172.21.15.70:0/3107565076 wait complete. 2024-04-01T01:46:09.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:46:09.200 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:46:09.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.457+0000 7fd92dce1640 1 -- 172.21.15.70:0/3935369582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd928103200 msgr2=0x7fd928109a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.457+0000 7fd92dce1640 1 --2- 172.21.15.70:0/3935369582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd928103200 0x7fd928109a90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fd92001acb0 tx=0x7fd92003d150 comp rx=0 tx=0).stop 2024-04-01T01:46:09.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.461+0000 7fd92dce1640 1 -- 172.21.15.70:0/3935369582 shutdown_connections 2024-04-01T01:46:09.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.461+0000 7fd92dce1640 1 --2- 172.21.15.70:0/3935369582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd928103200 0x7fd928109a90 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.461+0000 7fd92dce1640 1 --2- 172.21.15.70:0/3935369582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd928102840 0x7fd928102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.461+0000 7fd92dce1640 1 --2- 172.21.15.70:0/3935369582 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928101650 0x7fd928101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.461+0000 7fd92dce1640 1 -- 172.21.15.70:0/3935369582 >> 172.21.15.70:0/3935369582 conn(0x7fd9280fcbc0 msgr2=0x7fd9280ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.462+0000 7fd92dce1640 1 -- 172.21.15.70:0/3935369582 shutdown_connections 2024-04-01T01:46:09.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.463+0000 7fd92dce1640 1 -- 172.21.15.70:0/3935369582 wait complete. 2024-04-01T01:46:09.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.463+0000 7fd92dce1640 1 Processor -- start 2024-04-01T01:46:09.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.464+0000 7fd92dce1640 1 -- start start 2024-04-01T01:46:09.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.464+0000 7fd92dce1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd928101650 0x7fd928107b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.464+0000 7fd92dce1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928102840 0x7fd9281080b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.464+0000 7fd92dce1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd928103200 0x7fd928108f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.464+0000 7fd92dce1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd928072130 con 0x7fd928103200 2024-04-01T01:46:09.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.464+0000 7fd92dce1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd928071fb0 con 0x7fd928102840 2024-04-01T01:46:09.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.464+0000 7fd92dce1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd9280722b0 con 0x7fd928101650 2024-04-01T01:46:09.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.465+0000 7fd92ccdf640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd928101650 0x7fd928107b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.465+0000 7fd927fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928102840 0x7fd9281080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.465+0000 7fd927fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928102840 0x7fd9281080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:45360/0 (socket says 172.21.15.70:45360) 2024-04-01T01:46:09.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.465+0000 7fd927fff640 1 -- 172.21.15.70:0/1924960044 learned_addr learned my addr 172.21.15.70:0/1924960044 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:09.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.465+0000 7fd927fff640 1 -- 172.21.15.70:0/1924960044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd928101650 msgr2=0x7fd928107b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.465+0000 7fd927fff640 1 --2- 172.21.15.70:0/1924960044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd928101650 0x7fd928107b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.465+0000 7fd927fff640 1 -- 172.21.15.70:0/1924960044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd928103200 msgr2=0x7fd928108f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.465+0000 7fd927fff640 1 --2- 172.21.15.70:0/1924960044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd928103200 0x7fd928108f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.465+0000 7fd927fff640 1 -- 172.21.15.70:0/1924960044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9281096b0 con 0x7fd928102840 2024-04-01T01:46:09.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.465+0000 7fd927fff640 1 --2- 172.21.15.70:0/1924960044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928102840 0x7fd9281080b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd91800b450 tx=0x7fd91800b850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:09.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.466+0000 7fd925ffb640 1 -- 172.21.15.70:0/1924960044 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd918015020 con 0x7fd928102840 2024-04-01T01:46:09.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.466+0000 7fd925ffb640 1 -- 172.21.15.70:0/1924960044 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd918011ce0 con 0x7fd928102840 2024-04-01T01:46:09.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.466+0000 7fd925ffb640 1 -- 172.21.15.70:0/1924960044 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91800f2e0 con 0x7fd928102840 2024-04-01T01:46:09.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.466+0000 7fd92dce1640 1 -- 172.21.15.70:0/1924960044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9281099a0 con 0x7fd928102840 2024-04-01T01:46:09.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.466+0000 7fd92dce1640 1 -- 172.21.15.70:0/1924960044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd92819dc30 con 0x7fd928102840 2024-04-01T01:46:09.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.470+0000 7fd925ffb640 1 -- 172.21.15.70:0/1924960044 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fd918018020 con 0x7fd928102840 2024-04-01T01:46:09.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.470+0000 7fd931bf3740 1 Processor -- start 2024-04-01T01:46:09.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.470+0000 7fd931bf3740 1 -- start start 2024-04-01T01:46:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.470+0000 7fd931bf3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87490 0x55786ca878b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.470+0000 7fd931bf3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87e80 0x55786ca8c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.470+0000 7fd931bf3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca8cff0 0x55786ca8f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.470+0000 7fd931bf3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55786c9ec180 con 0x55786ca87e80 2024-04-01T01:46:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.470+0000 7fd931bf3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55786c994b70 con 0x55786ca8cff0 2024-04-01T01:46:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.470+0000 7fd931bf3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55786c9e8220 con 0x55786ca87490 2024-04-01T01:46:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.471+0000 7fd925ffb640 1 --2- 172.21.15.70:0/1924960044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9100a2670 0x7fd9100a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.471+0000 7fd927fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca8cff0 0x55786ca8f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.471+0000 7fd927fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca8cff0 0x55786ca8f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:45376/0 (socket says 172.21.15.70:45376) 2024-04-01T01:46:09.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.471+0000 7fd927fff640 1 -- 172.21.15.70:0/1544017366 learned_addr learned my addr 172.21.15.70:0/1544017366 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:09.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.471+0000 7fd92d4e0640 1 --2- 172.21.15.70:0/1544017366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87490 0x55786ca878b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.471+0000 7fd92d4e0640 1 --2- 172.21.15.70:0/1924960044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9100a2670 0x7fd9100a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.471+0000 7fd925ffb640 1 -- 172.21.15.70:0/1924960044 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd9180ce350 con 0x7fd928102840 2024-04-01T01:46:09.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.471+0000 7fd92ccdf640 1 --2- 172.21.15.70:0/1544017366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87e80 0x55786ca8c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.471+0000 7fd92d4e0640 1 -- 172.21.15.70:0/1544017366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca8cff0 msgr2=0x55786ca8f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:09.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.471+0000 7fd92d4e0640 1 --2- 172.21.15.70:0/1544017366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca8cff0 0x55786ca8f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.471+0000 7fd92d4e0640 1 -- 172.21.15.70:0/1544017366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87e80 msgr2=0x55786ca8c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:09.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.471+0000 7fd92d4e0640 1 --2- 172.21.15.70:0/1544017366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87e80 0x55786ca8c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.471+0000 7fd92d4e0640 1 -- 172.21.15.70:0/1544017366 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55786c9a87c0 con 0x55786ca87490 2024-04-01T01:46:09.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.471+0000 7fd927fff640 1 --2- 172.21.15.70:0/1544017366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca8cff0 0x55786ca8f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:46:09.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.471+0000 7fd92d4e0640 1 --2- 172.21.15.70:0/1544017366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87490 0x55786ca878b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd920048040 tx=0x7fd920043aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe6ef555d3a1c3fb server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:09.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.472+0000 7fd92d4e0640 1 --2- 172.21.15.70:0/1924960044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9100a2670 0x7fd9100a4b30 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd920045f50 tx=0x7fd920050000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:09.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.472+0000 7fd90effd640 1 -- 172.21.15.70:0/1544017366 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92004a040 con 0x55786ca87490 2024-04-01T01:46:09.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.472+0000 7fd90effd640 1 -- 172.21.15.70:0/1544017366 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9200435c0 con 0x55786ca87490 2024-04-01T01:46:09.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.472+0000 7fd90effd640 1 -- 172.21.15.70:0/1544017366 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9200043a0 con 0x55786ca87490 2024-04-01T01:46:09.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.473+0000 7fd931bf3740 1 -- 172.21.15.70:0/1544017366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87490 msgr2=0x55786ca878b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.473+0000 7fd931bf3740 1 --2- 172.21.15.70:0/1544017366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87490 0x55786ca878b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd920048040 tx=0x7fd920043aa0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.473+0000 7fd931bf3740 1 -- 172.21.15.70:0/1544017366 shutdown_connections 2024-04-01T01:46:09.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.473+0000 7fd931bf3740 1 --2- 172.21.15.70:0/1544017366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca8cff0 0x55786ca8f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.473+0000 7fd931bf3740 1 --2- 172.21.15.70:0/1544017366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87e80 0x55786ca8c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.473+0000 7fd931bf3740 1 --2- 172.21.15.70:0/1544017366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87490 0x55786ca878b0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.473+0000 7fd931bf3740 1 -- 172.21.15.70:0/1544017366 >> 172.21.15.70:0/1544017366 conn(0x55786ca82a00 msgr2=0x55786ca82e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:09.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.473+0000 7fd931bf3740 1 -- 172.21.15.70:0/1544017366 shutdown_connections 2024-04-01T01:46:09.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.473+0000 7fd931bf3740 1 -- 172.21.15.70:0/1544017366 wait complete. 2024-04-01T01:46:09.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.474+0000 7fd931bf3740 1 -- start start 2024-04-01T01:46:09.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.474+0000 7fd931bf3740 1 Processor -- start 2024-04-01T01:46:09.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd931bf3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca87490 0x55786ca919e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd931bf3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87e80 0x55786ca91f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd931bf3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca8cff0 0x55786ca98f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd931bf3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55786ca8ff30 con 0x55786ca8cff0 2024-04-01T01:46:09.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd931bf3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55786c98a4a0 con 0x55786ca87490 2024-04-01T01:46:09.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd931bf3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55786ca900b0 con 0x55786ca87e80 2024-04-01T01:46:09.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd931bf3740 10 client.0 _ll_register_callbacks cb 0x55786c98dd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:09.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd92d4e0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca8cff0 0x55786ca98f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd927fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87e80 0x55786ca91f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd92d4e0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca8cff0 0x55786ca98f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56948/0 (socket says 172.21.15.70:56948) 2024-04-01T01:46:09.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd927fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87e80 0x55786ca91f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:35796/0 (socket says 172.21.15.70:35796) 2024-04-01T01:46:09.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd92d4e0640 1 -- 172.21.15.70:0/2194554963 learned_addr learned my addr 172.21.15.70:0/2194554963 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:09.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd92ccdf640 1 --2- 172.21.15.70:0/2194554963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca87490 0x55786ca919e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd92d4e0640 1 -- 172.21.15.70:0/2194554963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87e80 msgr2=0x55786ca91f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:09.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd92d4e0640 1 --2- 172.21.15.70:0/2194554963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87e80 0x55786ca91f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd92d4e0640 1 -- 172.21.15.70:0/2194554963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca87490 msgr2=0x55786ca919e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:09.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd92d4e0640 1 --2- 172.21.15.70:0/2194554963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca87490 0x55786ca919e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd92d4e0640 1 -- 172.21.15.70:0/2194554963 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55786c7ca7d0 con 0x55786ca8cff0 2024-04-01T01:46:09.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd92d4e0640 1 --2- 172.21.15.70:0/2194554963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca8cff0 0x55786ca98f60 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fd92001a440 tx=0x7fd92001a8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7a803af5090b02 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:09.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.475+0000 7fd907fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:46:09.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.476+0000 7fd907fff640 1 -- 172.21.15.70:0/2194554963 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92004a040 con 0x55786ca8cff0 2024-04-01T01:46:09.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.476+0000 7fd907fff640 1 -- 172.21.15.70:0/2194554963 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd9200047a0 con 0x55786ca8cff0 2024-04-01T01:46:09.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.476+0000 7fd931bf3740 1 -- 172.21.15.70:0/2194554963 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9200435c0 con 0x55786ca8cff0 2024-04-01T01:46:09.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.476+0000 7fd907fff640 1 -- 172.21.15.70:0/2194554963 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd920004b40 con 0x55786ca8cff0 2024-04-01T01:46:09.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.476+0000 7fd931bf3740 10 client.19194 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:09.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.476+0000 7fd931bf3740 1 -- 172.21.15.70:0/2194554963 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55786c7003f0 con 0x55786ca8cff0 2024-04-01T01:46:09.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.477+0000 7fd907fff640 1 -- 172.21.15.70:0/2194554963 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd920004ce0 con 0x55786ca8cff0 2024-04-01T01:46:09.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.477+0000 7fd907fff640 1 -- 172.21.15.70:0/2194554963 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd920050070 con 0x55786ca8cff0 2024-04-01T01:46:09.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.477+0000 7fd931bf3740 10 client.19194 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:09.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.477+0000 7fd931bf3740 10 client.19194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:09.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.477+0000 7fd931bf3740 20 client.19194 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:09.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.477+0000 7fd931bf3740 1 --2- 172.21.15.70:0/2194554963 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55786ca9d450 0x55786ca9f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.477+0000 7fd931bf3740 4 client.19194 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:09.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.477+0000 7fd931bf3740 1 -- 172.21.15.70:0/2194554963 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd9200435c0 con 0x55786ca9d450 2024-04-01T01:46:09.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.477+0000 7fd92ccdf640 1 --2- 172.21.15.70:0/2194554963 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55786ca9d450 0x55786ca9f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.478+0000 7fd907fff640 10 client.19194 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:09.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.478+0000 7fd92ccdf640 1 --2- 172.21.15.70:0/2194554963 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55786ca9d450 0x55786ca9f840 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ba7450776ae1ede server_cookie=1a2aaeece7da3400 in_seq=0 out_seq=0 2024-04-01T01:46:09.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.479+0000 7fd907fff640 1 -- 172.21.15.70:0/2194554963 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fd9200435c0 con 0x55786ca9d450 2024-04-01T01:46:09.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.479+0000 7fd907fff640 10 client.19194 handle_command_reply: tid=0 2024-04-01T01:46:09.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.479+0000 7fd931bf3740 1 client.19194 shutdown 2024-04-01T01:46:09.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.479+0000 7fd931bf3740 1 -- 172.21.15.70:0/2194554963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca8cff0 msgr2=0x55786ca98f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.479+0000 7fd931bf3740 1 --2- 172.21.15.70:0/2194554963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca8cff0 0x55786ca98f60 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fd92001a440 tx=0x7fd92001a8f0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.479+0000 7fd931bf3740 1 -- 172.21.15.70:0/2194554963 shutdown_connections 2024-04-01T01:46:09.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.479+0000 7fd931bf3740 1 --2- 172.21.15.70:0/2194554963 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55786ca9d450 0x55786ca9f840 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.479+0000 7fd907fff640 0 client.19194 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:09.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.479+0000 7fd931bf3740 1 --2- 172.21.15.70:0/2194554963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca8cff0 0x55786ca98f60 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.479+0000 7fd931bf3740 1 --2- 172.21.15.70:0/2194554963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87e80 0x55786ca91f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.479+0000 7fd931bf3740 1 --2- 172.21.15.70:0/2194554963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca87490 0x55786ca919e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.479+0000 7fd931bf3740 1 -- 172.21.15.70:0/2194554963 >> 172.21.15.70:0/2194554963 conn(0x55786ca82a00 msgr2=0x55786ca81c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:09.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.479+0000 7fd931bf3740 1 -- 172.21.15.70:0/2194554963 shutdown_connections 2024-04-01T01:46:09.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.479+0000 7fd931bf3740 1 -- 172.21.15.70:0/2194554963 wait complete. 2024-04-01T01:46:09.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.479+0000 7fd931bf3740 20 client.19194 trim_cache size 0 max 16384 2024-04-01T01:46:09.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.496+0000 7fd931bf3740 1 Processor -- start 2024-04-01T01:46:09.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.496+0000 7fd931bf3740 1 -- start start 2024-04-01T01:46:09.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd931bf3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87490 0x55786ca81390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd931bf3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87e80 0x55786ca98f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd931bf3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca8cff0 0x55786ca818d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd931bf3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd9281a8070 con 0x55786ca87490 2024-04-01T01:46:09.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd931bf3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd9281a9bb0 con 0x55786ca8cff0 2024-04-01T01:46:09.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd931bf3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd9281a8740 con 0x55786ca87e80 2024-04-01T01:46:09.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd92ccdf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87490 0x55786ca81390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd92ccdf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87490 0x55786ca81390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56956/0 (socket says 172.21.15.70:56956) 2024-04-01T01:46:09.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd92ccdf640 1 -- 172.21.15.70:0/3442603054 learned_addr learned my addr 172.21.15.70:0/3442603054 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:09.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd927fff640 1 --2- 172.21.15.70:0/3442603054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87e80 0x55786ca98f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd92d4e0640 1 --2- 172.21.15.70:0/3442603054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca8cff0 0x55786ca818d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd92ccdf640 1 -- 172.21.15.70:0/3442603054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87e80 msgr2=0x55786ca98f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:09.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd92ccdf640 1 --2- 172.21.15.70:0/3442603054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87e80 0x55786ca98f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd92ccdf640 1 -- 172.21.15.70:0/3442603054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca8cff0 msgr2=0x55786ca818d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:09.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd92ccdf640 1 --2- 172.21.15.70:0/3442603054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca8cff0 0x55786ca818d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd92ccdf640 1 -- 172.21.15.70:0/3442603054 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55786c6c00f0 con 0x55786ca87490 2024-04-01T01:46:09.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd92ccdf640 1 --2- 172.21.15.70:0/3442603054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87490 0x55786ca81390 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x55786ca888f0 tx=0x7fd91c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d73f167280bc81e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:09.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd90e7fc640 1 -- 172.21.15.70:0/3442603054 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c0178f0 con 0x55786ca87490 2024-04-01T01:46:09.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd90e7fc640 1 -- 172.21.15.70:0/3442603054 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd91c021030 con 0x55786ca87490 2024-04-01T01:46:09.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.497+0000 7fd90e7fc640 1 -- 172.21.15.70:0/3442603054 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c0253d0 con 0x55786ca87490 2024-04-01T01:46:09.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.498+0000 7fd931bf3740 1 -- 172.21.15.70:0/3442603054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87490 msgr2=0x55786ca81390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.498+0000 7fd931bf3740 1 --2- 172.21.15.70:0/3442603054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87490 0x55786ca81390 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x55786ca888f0 tx=0x7fd91c008040 comp rx=0 tx=0).stop 2024-04-01T01:46:09.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.498+0000 7fd931bf3740 1 -- 172.21.15.70:0/3442603054 shutdown_connections 2024-04-01T01:46:09.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.498+0000 7fd931bf3740 1 --2- 172.21.15.70:0/3442603054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca8cff0 0x55786ca818d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.498+0000 7fd931bf3740 1 --2- 172.21.15.70:0/3442603054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca87e80 0x55786ca98f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.498+0000 7fd931bf3740 1 --2- 172.21.15.70:0/3442603054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87490 0x55786ca81390 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.498+0000 7fd931bf3740 1 -- 172.21.15.70:0/3442603054 >> 172.21.15.70:0/3442603054 conn(0x55786ca82a00 msgr2=0x55786ca89410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:09.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.498+0000 7fd931bf3740 1 -- 172.21.15.70:0/3442603054 shutdown_connections 2024-04-01T01:46:09.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.498+0000 7fd931bf3740 1 -- 172.21.15.70:0/3442603054 wait complete. 2024-04-01T01:46:09.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd931bf3740 1 -- start start 2024-04-01T01:46:09.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd931bf3740 1 Processor -- start 2024-04-01T01:46:09.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd931bf3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87490 0x55786ca96cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd931bf3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca87e80 0x55786ca971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd931bf3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca8cff0 0x55786ca98200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd931bf3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd9281a94e0 con 0x55786ca87490 2024-04-01T01:46:09.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd931bf3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd9281a8e10 con 0x55786ca87e80 2024-04-01T01:46:09.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd931bf3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd9281a6050 con 0x55786ca8cff0 2024-04-01T01:46:09.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd92ccdf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87490 0x55786ca96cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd92ccdf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87490 0x55786ca96cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56966/0 (socket says 172.21.15.70:56966) 2024-04-01T01:46:09.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd92ccdf640 1 -- 172.21.15.70:0/1112478393 learned_addr learned my addr 172.21.15.70:0/1112478393 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:09.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd927fff640 1 --2- 172.21.15.70:0/1112478393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca87e80 0x55786ca971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd931bf3740 10 client.0 _ll_register_callbacks cb 0x7fd92001ab30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:09.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd92d4e0640 1 --2- 172.21.15.70:0/1112478393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca8cff0 0x55786ca98200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd92ccdf640 1 -- 172.21.15.70:0/1112478393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca8cff0 msgr2=0x55786ca98200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:09.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd92ccdf640 1 --2- 172.21.15.70:0/1112478393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca8cff0 0x55786ca98200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd92ccdf640 1 -- 172.21.15.70:0/1112478393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca87e80 msgr2=0x55786ca971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:09.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd92ccdf640 1 --2- 172.21.15.70:0/1112478393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca87e80 0x55786ca971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd92ccdf640 1 -- 172.21.15.70:0/1112478393 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55786ca987f0 con 0x55786ca87490 2024-04-01T01:46:09.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd92ccdf640 1 --2- 172.21.15.70:0/1112478393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87490 0x55786ca96cb0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fd91c006f00 tx=0x7fd91c01d8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa59a41e1ce78699 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:09.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd907fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:46:09.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd907fff640 1 -- 172.21.15.70:0/1112478393 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c01d520 con 0x55786ca87490 2024-04-01T01:46:09.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd907fff640 1 -- 172.21.15.70:0/1112478393 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd91c0178f0 con 0x55786ca87490 2024-04-01T01:46:09.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.499+0000 7fd931bf3740 1 -- 172.21.15.70:0/1112478393 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd91c021030 con 0x55786ca87490 2024-04-01T01:46:09.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.500+0000 7fd907fff640 1 -- 172.21.15.70:0/1112478393 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd91c037050 con 0x55786ca87490 2024-04-01T01:46:09.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.500+0000 7fd931bf3740 10 client.19200 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:09.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.500+0000 7fd931bf3740 1 -- 172.21.15.70:0/1112478393 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55786cb03ce0 con 0x55786ca87490 2024-04-01T01:46:09.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.500+0000 7fd907fff640 1 -- 172.21.15.70:0/1112478393 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd91c021030 con 0x55786ca87490 2024-04-01T01:46:09.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.500+0000 7fd907fff640 1 -- 172.21.15.70:0/1112478393 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd91c02e400 con 0x55786ca87490 2024-04-01T01:46:09.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.500+0000 7fd931bf3740 10 client.19200 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:09.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.500+0000 7fd931bf3740 10 client.19200 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:09.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.500+0000 7fd931bf3740 20 client.19200 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:09.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.500+0000 7fd931bf3740 1 --2- 172.21.15.70:0/1112478393 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55786ca9d450 0x55786cb06640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:09.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.500+0000 7fd931bf3740 4 client.19200 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:46:09.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.500+0000 7fd931bf3740 1 -- 172.21.15.70:0/1112478393 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55786c9f96c0 con 0x55786ca9d450 2024-04-01T01:46:09.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.500+0000 7fd927fff640 1 --2- 172.21.15.70:0/1112478393 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55786ca9d450 0x55786cb06640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:09.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd907fff640 10 client.19200 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:09.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd927fff640 1 --2- 172.21.15.70:0/1112478393 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55786ca9d450 0x55786cb06640 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5b6e7cc7f806461 server_cookie=8b41e8dafb37bab1 in_seq=0 out_seq=0 2024-04-01T01:46:09.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd907fff640 1 -- 172.21.15.70:0/1112478393 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55786c9f96c0 con 0x55786ca9d450 2024-04-01T01:46:09.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd907fff640 10 client.19200 handle_command_reply: tid=0 2024-04-01T01:46:09.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd931bf3740 1 client.19200 shutdown 2024-04-01T01:46:09.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd931bf3740 1 -- 172.21.15.70:0/1112478393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87490 msgr2=0x55786ca96cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd931bf3740 1 --2- 172.21.15.70:0/1112478393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87490 0x55786ca96cb0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fd91c006f00 tx=0x7fd91c01d8c0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd931bf3740 1 -- 172.21.15.70:0/1112478393 shutdown_connections 2024-04-01T01:46:09.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd931bf3740 1 --2- 172.21.15.70:0/1112478393 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55786ca9d450 0x55786cb06640 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd931bf3740 1 --2- 172.21.15.70:0/1112478393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55786ca8cff0 0x55786ca98200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd907fff640 0 client.19200 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:09.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd931bf3740 1 --2- 172.21.15.70:0/1112478393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55786ca87e80 0x55786ca971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd931bf3740 1 --2- 172.21.15.70:0/1112478393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55786ca87490 0x55786ca96cb0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd931bf3740 1 -- 172.21.15.70:0/1112478393 >> 172.21.15.70:0/1112478393 conn(0x55786ca82a00 msgr2=0x55786ca85c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:09.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd931bf3740 1 -- 172.21.15.70:0/1112478393 shutdown_connections 2024-04-01T01:46:09.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd931bf3740 1 -- 172.21.15.70:0/1112478393 wait complete. 2024-04-01T01:46:09.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.501+0000 7fd931bf3740 20 client.19200 trim_cache size 0 max 16384 2024-04-01T01:46:09.509 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:46:09.509 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:46:09.509 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:46:09.509 INFO:teuthology.orchestra.run.smithi070.stdout: "9b7827a3-286a-4656-b244-e395a0697248": { 2024-04-01T01:46:09.509 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:46:09.509 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "9b7827a3-286a-4656-b244-e395a0697248", 2024-04-01T01:46:09.509 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:46:09.509 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:46:09.509 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:46:09.509 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:46:09.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.502+0000 7fd92dce1640 1 -- 172.21.15.70:0/1924960044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9100a2670 msgr2=0x7fd9100a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.502+0000 7fd92dce1640 1 --2- 172.21.15.70:0/1924960044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9100a2670 0x7fd9100a4b30 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd920045f50 tx=0x7fd920050000 comp rx=0 tx=0).stop 2024-04-01T01:46:09.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.502+0000 7fd92dce1640 1 -- 172.21.15.70:0/1924960044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928102840 msgr2=0x7fd9281080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:09.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.502+0000 7fd92dce1640 1 --2- 172.21.15.70:0/1924960044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928102840 0x7fd9281080b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd91800b450 tx=0x7fd91800b850 comp rx=0 tx=0).stop 2024-04-01T01:46:09.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.502+0000 7fd92dce1640 1 -- 172.21.15.70:0/1924960044 shutdown_connections 2024-04-01T01:46:09.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.502+0000 7fd92dce1640 1 --2- 172.21.15.70:0/1924960044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd928103200 0x7fd928108f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.502+0000 7fd92dce1640 1 --2- 172.21.15.70:0/1924960044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd928102840 0x7fd9281080b0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.502+0000 7fd92dce1640 1 --2- 172.21.15.70:0/1924960044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9100a2670 0x7fd9100a4b30 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.502+0000 7fd92dce1640 1 --2- 172.21.15.70:0/1924960044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd928101650 0x7fd928107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:09.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.502+0000 7fd92dce1640 1 -- 172.21.15.70:0/1924960044 >> 172.21.15.70:0/1924960044 conn(0x7fd9280fcbc0 msgr2=0x7fd928074d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:09.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.502+0000 7fd92dce1640 1 -- 172.21.15.70:0/1924960044 shutdown_connections 2024-04-01T01:46:09.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:09.502+0000 7fd92dce1640 1 -- 172.21.15.70:0/1924960044 wait complete. 2024-04-01T01:46:09.512 INFO:tasks.cephfs.filesystem:scrub status for tag:9b7827a3-286a-4656-b244-e395a0697248 - {'path': '/', 'tag': '9b7827a3-286a-4656-b244-e395a0697248', 'options': 'recursive,force'} 2024-04-01T01:46:10.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:10 smithi189 ceph-mon[35753]: pgmap v739: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 8.1 MiB/s wr, 807 op/s 2024-04-01T01:46:10.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:10 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:46:10.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:10 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:46:10.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:10 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:46:10.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:10 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:46:11.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:10 smithi070 ceph-mon[32179]: pgmap v739: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 8.1 MiB/s wr, 807 op/s 2024-04-01T01:46:11.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:10 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:46:11.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:10 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:46:11.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:10 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:46:11.007 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:10 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:46:11.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:10 smithi081 ceph-mon[35142]: pgmap v739: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 8.1 MiB/s wr, 807 op/s 2024-04-01T01:46:11.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:10 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:46:11.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:10 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:46:11.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:10 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:46:11.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:10 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:46:11.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 685.0 s, 807.4 tps, lat 123.881 ms stddev 81.464 2024-04-01T01:46:11.978 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:11 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:46:11.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:11 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:46:12.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:11 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:46:12.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:12 smithi189 ceph-mon[35753]: pgmap v740: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 7.4 MiB/s wr, 768 op/s 2024-04-01T01:46:13.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:12 smithi070 ceph-mon[32179]: pgmap v740: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 7.4 MiB/s wr, 768 op/s 2024-04-01T01:46:13.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:12 smithi081 ceph-mon[35142]: pgmap v740: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 7.4 MiB/s wr, 768 op/s 2024-04-01T01:46:14.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:14 smithi189 ceph-mon[35753]: pgmap v741: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 8.0 MiB/s wr, 863 op/s 2024-04-01T01:46:15.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:14 smithi070 ceph-mon[32179]: pgmap v741: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 8.0 MiB/s wr, 863 op/s 2024-04-01T01:46:15.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:14 smithi081 ceph-mon[35142]: pgmap v741: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 8.0 MiB/s wr, 863 op/s 2024-04-01T01:46:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.668+0000 7faadd9e7640 1 -- 172.21.15.70:0/3527205428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad8072a00 msgr2=0x7faad8072e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.669+0000 7faadd9e7640 1 --2- 172.21.15.70:0/3527205428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad8072a00 0x7faad8072e20 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7faad0009900 tx=0x7faad002bc40 comp rx=0 tx=0).stop 2024-04-01T01:46:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.669+0000 7faadd9e7640 1 -- 172.21.15.70:0/3527205428 shutdown_connections 2024-04-01T01:46:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.669+0000 7faadd9e7640 1 --2- 172.21.15.70:0/3527205428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faad8109440 0x7faad810e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.669+0000 7faadd9e7640 1 --2- 172.21.15.70:0/3527205428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faad8073360 0x7faad8108d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.669+0000 7faadd9e7640 1 --2- 172.21.15.70:0/3527205428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad8072a00 0x7faad8072e20 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.669+0000 7faadd9e7640 1 -- 172.21.15.70:0/3527205428 >> 172.21.15.70:0/3527205428 conn(0x7faad806e5c0 msgr2=0x7faad8070a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:16.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.669+0000 7faadd9e7640 1 -- 172.21.15.70:0/3527205428 shutdown_connections 2024-04-01T01:46:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.669+0000 7faadd9e7640 1 -- 172.21.15.70:0/3527205428 wait complete. 2024-04-01T01:46:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.670+0000 7faadd9e7640 1 Processor -- start 2024-04-01T01:46:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.670+0000 7faadd9e7640 1 -- start start 2024-04-01T01:46:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.670+0000 7faadd9e7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faad8072a00 0x7faad8114e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.670+0000 7faadd9e7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faad8073360 0x7faad81153b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.670+0000 7faadd9e7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad8109440 0x7faad81b6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.670+0000 7faadd9e7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faad811bf30 con 0x7faad8072a00 2024-04-01T01:46:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.670+0000 7faadd9e7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faad811bdb0 con 0x7faad8109440 2024-04-01T01:46:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.670+0000 7faadd9e7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faad811c0b0 con 0x7faad8073360 2024-04-01T01:46:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.671+0000 7faadd1e6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad8109440 0x7faad81b6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.671+0000 7faadd1e6640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad8109440 0x7faad81b6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:45308/0 (socket says 172.21.15.70:45308) 2024-04-01T01:46:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.671+0000 7faadd1e6640 1 -- 172.21.15.70:0/2874145173 learned_addr learned my addr 172.21.15.70:0/2874145173 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.671+0000 7faadc9e5640 1 --2- 172.21.15.70:0/2874145173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faad8072a00 0x7faad8114e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.672+0000 7faadd1e6640 1 -- 172.21.15.70:0/2874145173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faad8073360 msgr2=0x7faad81153b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.672+0000 7faadd1e6640 1 --2- 172.21.15.70:0/2874145173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faad8073360 0x7faad81153b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.672+0000 7faadd1e6640 1 -- 172.21.15.70:0/2874145173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faad8072a00 msgr2=0x7faad8114e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.672+0000 7faadd1e6640 1 --2- 172.21.15.70:0/2874145173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faad8072a00 0x7faad8114e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.672+0000 7faadd1e6640 1 -- 172.21.15.70:0/2874145173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad81b6b60 con 0x7faad8109440 2024-04-01T01:46:16.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.672+0000 7faadd1e6640 1 --2- 172.21.15.70:0/2874145173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad8109440 0x7faad81b6620 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7faac800eaa0 tx=0x7faac800ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:16.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.672+0000 7faad5ffb640 1 -- 172.21.15.70:0/2874145173 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac800cdb0 con 0x7faad8109440 2024-04-01T01:46:16.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.672+0000 7faad5ffb640 1 -- 172.21.15.70:0/2874145173 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faac8005020 con 0x7faad8109440 2024-04-01T01:46:16.674 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.672+0000 7faad5ffb640 1 -- 172.21.15.70:0/2874145173 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac80116d0 con 0x7faad8109440 2024-04-01T01:46:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.673+0000 7faadd9e7640 1 -- 172.21.15.70:0/2874145173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faad81b6d30 con 0x7faad8109440 2024-04-01T01:46:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.673+0000 7faadd9e7640 1 -- 172.21.15.70:0/2874145173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faad81b7200 con 0x7faad8109440 2024-04-01T01:46:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.676+0000 7faae181f740 1 Processor -- start 2024-04-01T01:46:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.676+0000 7faae181f740 1 -- start start 2024-04-01T01:46:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.677+0000 7faae181f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b63f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.677+0000 7faae181f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b644d0 0x55e1c4b68fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.677+0000 7faae181f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b69640 0x55e1c4b6bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.677+0000 7faae181f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e1c4ac78f0 con 0x55e1c4b63ae0 2024-04-01T01:46:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.677+0000 7faae181f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e1c4a708d0 con 0x55e1c4b69640 2024-04-01T01:46:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.677+0000 7faae181f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e1c4ac3990 con 0x55e1c4b644d0 2024-04-01T01:46:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.680+0000 7faad5ffb640 1 -- 172.21.15.70:0/2874145173 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7faac80051d0 con 0x7faad8109440 2024-04-01T01:46:16.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.680+0000 7faad5ffb640 1 --2- 172.21.15.70:0/2874145173 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faac40a25d0 0x7faac40a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.681+0000 7faadc9e5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b644d0 0x55e1c4b68fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.681+0000 7faadc9e5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b644d0 0x55e1c4b68fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41886/0 (socket says 172.21.15.70:41886) 2024-04-01T01:46:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.681+0000 7faadc9e5640 1 -- 172.21.15.70:0/1871180981 learned_addr learned my addr 172.21.15.70:0/1871180981 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.681+0000 7faadd1e6640 1 --2- 172.21.15.70:0/2874145173 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faac40a25d0 0x7faac40a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.681+0000 7faad7fff640 1 --2- 172.21.15.70:0/1871180981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b69640 0x55e1c4b6bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.681+0000 7faad7fff640 1 --2- 172.21.15.70:0/1871180981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b63f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.681+0000 7faadc9e5640 1 -- 172.21.15.70:0/1871180981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b69640 msgr2=0x55e1c4b6bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.681+0000 7faadc9e5640 1 --2- 172.21.15.70:0/1871180981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b69640 0x55e1c4b6bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.681+0000 7faadc9e5640 1 -- 172.21.15.70:0/1871180981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 msgr2=0x55e1c4b63f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.681+0000 7faadc9e5640 1 --2- 172.21.15.70:0/1871180981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b63f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.681+0000 7faadc9e5640 1 -- 172.21.15.70:0/1871180981 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1c47dc210 con 0x55e1c4b644d0 2024-04-01T01:46:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.681+0000 7faad7fff640 1 --2- 172.21.15.70:0/1871180981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b63f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:46:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.682+0000 7faadc9e5640 1 --2- 172.21.15.70:0/1871180981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b644d0 0x55e1c4b68fd0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7faad002c850 tx=0x7faad002c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7deb4f81cfe017b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.682+0000 7faaceffd640 1 -- 172.21.15.70:0/1871180981 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad0045020 con 0x55e1c4b644d0 2024-04-01T01:46:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.682+0000 7faad5ffb640 1 -- 172.21.15.70:0/2874145173 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7faac80cefa0 con 0x7faad8109440 2024-04-01T01:46:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.682+0000 7faadd1e6640 1 --2- 172.21.15.70:0/2874145173 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faac40a25d0 0x7faac40a4a90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7faac80a3b10 tx=0x7faac800cf50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.683+0000 7faaceffd640 1 -- 172.21.15.70:0/1871180981 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faad002ca60 con 0x55e1c4b644d0 2024-04-01T01:46:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.684+0000 7faaceffd640 1 -- 172.21.15.70:0/1871180981 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad00046c0 con 0x55e1c4b644d0 2024-04-01T01:46:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.684+0000 7faae181f740 1 -- 172.21.15.70:0/1871180981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b644d0 msgr2=0x55e1c4b68fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:16.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.684+0000 7faae181f740 1 --2- 172.21.15.70:0/1871180981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b644d0 0x55e1c4b68fd0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7faad002c850 tx=0x7faad002c8f0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.684+0000 7faae181f740 1 -- 172.21.15.70:0/1871180981 shutdown_connections 2024-04-01T01:46:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.684+0000 7faae181f740 1 --2- 172.21.15.70:0/1871180981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b69640 0x55e1c4b6bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.684+0000 7faae181f740 1 --2- 172.21.15.70:0/1871180981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b644d0 0x55e1c4b68fd0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.684+0000 7faae181f740 1 --2- 172.21.15.70:0/1871180981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b63f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.684+0000 7faae181f740 1 -- 172.21.15.70:0/1871180981 >> 172.21.15.70:0/1871180981 conn(0x55e1c4b5f050 msgr2=0x55e1c4b5f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.684+0000 7faae181f740 1 -- 172.21.15.70:0/1871180981 shutdown_connections 2024-04-01T01:46:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.684+0000 7faae181f740 1 -- 172.21.15.70:0/1871180981 wait complete. 2024-04-01T01:46:16.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.685+0000 7faae181f740 1 -- start start 2024-04-01T01:46:16.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.685+0000 7faae181f740 1 Processor -- start 2024-04-01T01:46:16.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.685+0000 7faae181f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b6de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.685+0000 7faae181f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b644d0 0x55e1c4b6e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.685+0000 7faae181f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b69640 0x55e1c4b75360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.685+0000 7faae181f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e1c4b6c5e0 con 0x55e1c4b69640 2024-04-01T01:46:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.685+0000 7faae181f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e1c4a65f80 con 0x55e1c4b63ae0 2024-04-01T01:46:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.685+0000 7faae181f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e1c4b6c760 con 0x55e1c4b644d0 2024-04-01T01:46:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.685+0000 7faae181f740 10 client.0 _ll_register_callbacks cb 0x55e1c4abe290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.686+0000 7faad7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b6de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.686+0000 7faad7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b6de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:45322/0 (socket says 172.21.15.70:45322) 2024-04-01T01:46:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.686+0000 7faad7fff640 1 -- 172.21.15.70:0/2744737568 learned_addr learned my addr 172.21.15.70:0/2744737568 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.686+0000 7faadc9e5640 1 --2- 172.21.15.70:0/2744737568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b644d0 0x55e1c4b6e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.686+0000 7faad7fff640 1 --2- 172.21.15.70:0/2744737568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b69640 0x55e1c4b75360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.686+0000 7faadc9e5640 1 -- 172.21.15.70:0/2744737568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b63ae0 msgr2=0x55e1c4b6de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.686+0000 7faadc9e5640 1 --2- 172.21.15.70:0/2744737568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b6de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.686+0000 7faadc9e5640 1 -- 172.21.15.70:0/2744737568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b69640 msgr2=0x55e1c4b75360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.686+0000 7faadc9e5640 1 --2- 172.21.15.70:0/2744737568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b69640 0x55e1c4b75360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.686+0000 7faadc9e5640 1 -- 172.21.15.70:0/2744737568 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1c48a8620 con 0x55e1c4b644d0 2024-04-01T01:46:16.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.687+0000 7faadc9e5640 1 --2- 172.21.15.70:0/2744737568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b644d0 0x55e1c4b6e390 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x55e1c4b5ffc0 tx=0x7faad0004340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2468d2e42a39d48 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.687+0000 7faab7fff640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:46:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.687+0000 7faab7fff640 1 -- 172.21.15.70:0/2744737568 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad0050070 con 0x55e1c4b644d0 2024-04-01T01:46:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.687+0000 7faab7fff640 1 -- 172.21.15.70:0/2744737568 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7faad0003a00 con 0x55e1c4b644d0 2024-04-01T01:46:16.689 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.688+0000 7faae181f740 1 -- 172.21.15.70:0/2744737568 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faad002ca60 con 0x55e1c4b644d0 2024-04-01T01:46:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.696+0000 7faab7fff640 1 -- 172.21.15.70:0/2744737568 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faad0045040 con 0x55e1c4b644d0 2024-04-01T01:46:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.696+0000 7faae181f740 10 client.29056 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:16.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.696+0000 7faae181f740 1 -- 172.21.15.70:0/2744737568 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e1c4b6c5e0 con 0x55e1c4b644d0 2024-04-01T01:46:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.697+0000 7faab7fff640 1 -- 172.21.15.70:0/2744737568 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faad002ca60 con 0x55e1c4b644d0 2024-04-01T01:46:16.697 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.697+0000 7faab7fff640 1 -- 172.21.15.70:0/2744737568 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7faad0053020 con 0x55e1c4b644d0 2024-04-01T01:46:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.697+0000 7faae181f740 10 client.29056 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.697+0000 7faae181f740 10 client.29056 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.697+0000 7faae181f740 20 client.29056 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.697+0000 7faae181f740 1 --2- 172.21.15.70:0/2744737568 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e1c4b795b0 0x55e1c4b7b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.697+0000 7faae181f740 4 client.29056 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.697+0000 7faae181f740 1 -- 172.21.15.70:0/2744737568 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1c4adb120 con 0x55e1c4b795b0 2024-04-01T01:46:16.698 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.698+0000 7faad7fff640 1 --2- 172.21.15.70:0/2744737568 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e1c4b795b0 0x55e1c4b7b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.698+0000 7faad7fff640 1 --2- 172.21.15.70:0/2744737568 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e1c4b795b0 0x55e1c4b7b9a0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64b75ac37073f691 server_cookie=e39aa582ed695ad2 in_seq=0 out_seq=0 2024-04-01T01:46:16.699 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.698+0000 7faab7fff640 10 client.29056 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.699+0000 7faab7fff640 1 -- 172.21.15.70:0/2744737568 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e1c4adb120 con 0x55e1c4b795b0 2024-04-01T01:46:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.700+0000 7faab7fff640 10 client.29056 handle_command_reply: tid=0 2024-04-01T01:46:16.700 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.700+0000 7faae181f740 1 client.29056 shutdown 2024-04-01T01:46:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.700+0000 7faae181f740 1 -- 172.21.15.70:0/2744737568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b644d0 msgr2=0x55e1c4b6e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.700+0000 7faae181f740 1 --2- 172.21.15.70:0/2744737568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b644d0 0x55e1c4b6e390 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x55e1c4b5ffc0 tx=0x7faad0004340 comp rx=0 tx=0).stop 2024-04-01T01:46:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.701+0000 7faae181f740 1 -- 172.21.15.70:0/2744737568 shutdown_connections 2024-04-01T01:46:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.701+0000 7faae181f740 1 --2- 172.21.15.70:0/2744737568 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e1c4b795b0 0x55e1c4b7b9a0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.701+0000 7faae181f740 1 --2- 172.21.15.70:0/2744737568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b69640 0x55e1c4b75360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.701+0000 7faae181f740 1 --2- 172.21.15.70:0/2744737568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b644d0 0x55e1c4b6e390 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.701+0000 7faae181f740 1 --2- 172.21.15.70:0/2744737568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b6de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.701+0000 7faae181f740 1 -- 172.21.15.70:0/2744737568 >> 172.21.15.70:0/2744737568 conn(0x55e1c4b5f050 msgr2=0x55e1c4b5e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:16.701 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.701+0000 7faab7fff640 0 client.29056 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:16.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.701+0000 7faae181f740 1 -- 172.21.15.70:0/2744737568 shutdown_connections 2024-04-01T01:46:16.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.701+0000 7faae181f740 1 -- 172.21.15.70:0/2744737568 wait complete. 2024-04-01T01:46:16.702 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.701+0000 7faae181f740 20 client.29056 trim_cache size 0 max 16384 2024-04-01T01:46:16.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.732+0000 7faae181f740 1 Processor -- start 2024-04-01T01:46:16.733 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.733+0000 7faae181f740 1 -- start start 2024-04-01T01:46:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.733+0000 7faae181f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b5d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.733+0000 7faae181f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b644d0 0x55e1c4b75320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.733+0000 7faae181f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b69640 0x55e1c4b5de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.733+0000 7faae181f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faad81c1840 con 0x55e1c4b63ae0 2024-04-01T01:46:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.733+0000 7faae181f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faad81c3380 con 0x55e1c4b644d0 2024-04-01T01:46:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.733+0000 7faae181f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faad81c1f10 con 0x55e1c4b69640 2024-04-01T01:46:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.733+0000 7faadc9e5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b644d0 0x55e1c4b75320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.733+0000 7faad7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b5d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.733+0000 7faad7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b5d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60530/0 (socket says 172.21.15.70:60530) 2024-04-01T01:46:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.733+0000 7faad7fff640 1 -- 172.21.15.70:0/3980143415 learned_addr learned my addr 172.21.15.70:0/3980143415 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:16.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.733+0000 7faad7fff640 1 --2- 172.21.15.70:0/3980143415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b69640 0x55e1c4b5de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.734+0000 7faad7fff640 1 -- 172.21.15.70:0/3980143415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b69640 msgr2=0x55e1c4b5de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:16.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.734+0000 7faad7fff640 1 --2- 172.21.15.70:0/3980143415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b69640 0x55e1c4b5de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.734+0000 7faad7fff640 1 -- 172.21.15.70:0/3980143415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b644d0 msgr2=0x55e1c4b75320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:16.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.734+0000 7faad7fff640 1 --2- 172.21.15.70:0/3980143415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b644d0 0x55e1c4b75320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.734+0000 7faad7fff640 1 -- 172.21.15.70:0/3980143415 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1c4b5fdc0 con 0x55e1c4b63ae0 2024-04-01T01:46:16.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.734+0000 7faad7fff640 1 --2- 172.21.15.70:0/3980143415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b5d900 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7faac0000c00 tx=0x7faac0005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b803a6108f0b3c36 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:16.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.735+0000 7faace7fc640 1 -- 172.21.15.70:0/3980143415 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac001e350 con 0x55e1c4b63ae0 2024-04-01T01:46:16.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.735+0000 7faace7fc640 1 -- 172.21.15.70:0/3980143415 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faac0005020 con 0x55e1c4b63ae0 2024-04-01T01:46:16.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.736+0000 7faace7fc640 1 -- 172.21.15.70:0/3980143415 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac00049a0 con 0x55e1c4b63ae0 2024-04-01T01:46:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.736+0000 7faae181f740 1 -- 172.21.15.70:0/3980143415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 msgr2=0x55e1c4b5d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.736+0000 7faae181f740 1 --2- 172.21.15.70:0/3980143415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b5d900 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7faac0000c00 tx=0x7faac0005850 comp rx=0 tx=0).stop 2024-04-01T01:46:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.736+0000 7faae181f740 1 -- 172.21.15.70:0/3980143415 shutdown_connections 2024-04-01T01:46:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.736+0000 7faae181f740 1 --2- 172.21.15.70:0/3980143415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b69640 0x55e1c4b5de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.736+0000 7faae181f740 1 --2- 172.21.15.70:0/3980143415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b644d0 0x55e1c4b75320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.736+0000 7faae181f740 1 --2- 172.21.15.70:0/3980143415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b5d900 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.736+0000 7faae181f740 1 -- 172.21.15.70:0/3980143415 >> 172.21.15.70:0/3980143415 conn(0x55e1c4b5f050 msgr2=0x55e1c4b657f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.738+0000 7faae181f740 1 -- 172.21.15.70:0/3980143415 shutdown_connections 2024-04-01T01:46:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.738+0000 7faae181f740 1 -- 172.21.15.70:0/3980143415 wait complete. 2024-04-01T01:46:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.738+0000 7faae181f740 1 -- start start 2024-04-01T01:46:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.738+0000 7faae181f740 1 Processor -- start 2024-04-01T01:46:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.739+0000 7faae181f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b728e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.739+0000 7faae181f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b644d0 0x55e1c4b72e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.739+0000 7faae181f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b69640 0x55e1c4b73de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.739+0000 7faae181f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faad81c2cb0 con 0x55e1c4b63ae0 2024-04-01T01:46:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.739+0000 7faae181f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faad81c25e0 con 0x55e1c4b644d0 2024-04-01T01:46:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.739+0000 7faae181f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faad81bf820 con 0x55e1c4b69640 2024-04-01T01:46:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.739+0000 7faae181f740 10 client.0 _ll_register_callbacks cb 0x7faad002cc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.739+0000 7faadc9e5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b644d0 0x55e1c4b72e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.739+0000 7faadc9e5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b644d0 0x55e1c4b72e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:45336/0 (socket says 172.21.15.70:45336) 2024-04-01T01:46:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.739+0000 7faadc9e5640 1 -- 172.21.15.70:0/824267991 learned_addr learned my addr 172.21.15.70:0/824267991 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.739+0000 7faad7fff640 1 --2- 172.21.15.70:0/824267991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.739+0000 7faad7fff640 1 --2- 172.21.15.70:0/824267991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b69640 0x55e1c4b73de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.740+0000 7faad7fff640 1 -- 172.21.15.70:0/824267991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b69640 msgr2=0x55e1c4b73de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.740+0000 7faad7fff640 1 --2- 172.21.15.70:0/824267991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b69640 0x55e1c4b73de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.740+0000 7faad7fff640 1 -- 172.21.15.70:0/824267991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b644d0 msgr2=0x55e1c4b72e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.740+0000 7faad7fff640 1 --2- 172.21.15.70:0/824267991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b644d0 0x55e1c4b72e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.740+0000 7faad7fff640 1 -- 172.21.15.70:0/824267991 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1c4b59420 con 0x55e1c4b63ae0 2024-04-01T01:46:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.740+0000 7faad7fff640 1 --2- 172.21.15.70:0/824267991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b728e0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7faac0000c00 tx=0x7faac000def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e7c94e1905c59b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.740+0000 7faab7fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:46:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.741+0000 7faab7fff640 1 -- 172.21.15.70:0/824267991 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac0015020 con 0x55e1c4b63ae0 2024-04-01T01:46:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.741+0000 7faab7fff640 1 -- 172.21.15.70:0/824267991 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7faac001e350 con 0x55e1c4b63ae0 2024-04-01T01:46:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.741+0000 7faae181f740 1 -- 172.21.15.70:0/824267991 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faac0005020 con 0x55e1c4b63ae0 2024-04-01T01:46:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.742+0000 7faab7fff640 1 -- 172.21.15.70:0/824267991 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faac001e550 con 0x55e1c4b63ae0 2024-04-01T01:46:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.742+0000 7faae181f740 10 client.19215 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.742+0000 7faae181f740 1 -- 172.21.15.70:0/824267991 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e1c4b75510 con 0x55e1c4b63ae0 2024-04-01T01:46:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.743+0000 7faab7fff640 1 -- 172.21.15.70:0/824267991 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faac0005020 con 0x55e1c4b63ae0 2024-04-01T01:46:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.743+0000 7faab7fff640 1 -- 172.21.15.70:0/824267991 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7faad00054d0 con 0x55e1c4b63ae0 2024-04-01T01:46:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.743+0000 7faae181f740 10 client.19215 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.743+0000 7faae181f740 10 client.19215 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.743+0000 7faae181f740 20 client.19215 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.743+0000 7faae181f740 1 --2- 172.21.15.70:0/824267991 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e1c4bdfd30 0x55e1c4be2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.743+0000 7faae181f740 4 client.19215 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:46:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.743+0000 7faae181f740 1 -- 172.21.15.70:0/824267991 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e1c4ad4e30 con 0x55e1c4bdfd30 2024-04-01T01:46:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.744+0000 7faadc9e5640 1 --2- 172.21.15.70:0/824267991 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e1c4bdfd30 0x55e1c4be2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.745+0000 7faab7fff640 10 client.19215 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.745+0000 7faadc9e5640 1 --2- 172.21.15.70:0/824267991 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e1c4bdfd30 0x55e1c4be2150 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8715d87a3a19575f server_cookie=158046e0db59f7bf in_seq=0 out_seq=0 2024-04-01T01:46:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.780+0000 7faab7fff640 1 -- 172.21.15.70:0/824267991 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e1c4ad4e30 con 0x55e1c4bdfd30 2024-04-01T01:46:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.780+0000 7faab7fff640 10 client.19215 handle_command_reply: tid=0 2024-04-01T01:46:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.780+0000 7faae181f740 1 client.19215 shutdown 2024-04-01T01:46:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.781+0000 7faae181f740 1 -- 172.21.15.70:0/824267991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 msgr2=0x55e1c4b728e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.781+0000 7faae181f740 1 --2- 172.21.15.70:0/824267991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b728e0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7faac0000c00 tx=0x7faac000def0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.781+0000 7faae181f740 1 -- 172.21.15.70:0/824267991 shutdown_connections 2024-04-01T01:46:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.781+0000 7faae181f740 1 --2- 172.21.15.70:0/824267991 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e1c4bdfd30 0x55e1c4be2150 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.781+0000 7faae181f740 1 --2- 172.21.15.70:0/824267991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e1c4b69640 0x55e1c4b73de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.781+0000 7faae181f740 1 --2- 172.21.15.70:0/824267991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e1c4b644d0 0x55e1c4b72e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.781+0000 7faae181f740 1 --2- 172.21.15.70:0/824267991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e1c4b63ae0 0x55e1c4b728e0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.781+0000 7faae181f740 1 -- 172.21.15.70:0/824267991 >> 172.21.15.70:0/824267991 conn(0x55e1c4b5f050 msgr2=0x55e1c4b5d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.781+0000 7faab7fff640 0 client.19215 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.781+0000 7faae181f740 1 -- 172.21.15.70:0/824267991 shutdown_connections 2024-04-01T01:46:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.781+0000 7faae181f740 1 -- 172.21.15.70:0/824267991 wait complete. 2024-04-01T01:46:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.781+0000 7faae181f740 20 client.19215 trim_cache size 0 max 16384 2024-04-01T01:46:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:46:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:46:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:46:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:46:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.782+0000 7faadd9e7640 1 -- 172.21.15.70:0/2874145173 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faac40a25d0 msgr2=0x7faac40a4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.782+0000 7faadd9e7640 1 --2- 172.21.15.70:0/2874145173 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faac40a25d0 0x7faac40a4a90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7faac80a3b10 tx=0x7faac800cf50 comp rx=0 tx=0).stop 2024-04-01T01:46:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.782+0000 7faadd9e7640 1 -- 172.21.15.70:0/2874145173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad8109440 msgr2=0x7faad81b6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.782+0000 7faadd9e7640 1 --2- 172.21.15.70:0/2874145173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad8109440 0x7faad81b6620 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7faac800eaa0 tx=0x7faac800ef60 comp rx=0 tx=0).stop 2024-04-01T01:46:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.782+0000 7faadd9e7640 1 -- 172.21.15.70:0/2874145173 shutdown_connections 2024-04-01T01:46:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.782+0000 7faadd9e7640 1 --2- 172.21.15.70:0/2874145173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad8109440 0x7faad81b6620 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.782+0000 7faadd9e7640 1 --2- 172.21.15.70:0/2874145173 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faac40a25d0 0x7faac40a4a90 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.782+0000 7faadd9e7640 1 --2- 172.21.15.70:0/2874145173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faad8073360 0x7faad81153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.782+0000 7faadd9e7640 1 --2- 172.21.15.70:0/2874145173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faad8072a00 0x7faad8114e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.782+0000 7faadd9e7640 1 -- 172.21.15.70:0/2874145173 >> 172.21.15.70:0/2874145173 conn(0x7faad806e5c0 msgr2=0x7faad80702d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.783+0000 7faadd9e7640 1 -- 172.21.15.70:0/2874145173 shutdown_connections 2024-04-01T01:46:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:16.783+0000 7faadd9e7640 1 -- 172.21.15.70:0/2874145173 wait complete. 2024-04-01T01:46:16.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 690.0 s, 841.4 tps, lat 118.671 ms stddev 107.317 2024-04-01T01:46:16.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:16 smithi189 ceph-mon[35753]: pgmap v742: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 6.8 MiB/s wr, 798 op/s 2024-04-01T01:46:17.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:16 smithi070 ceph-mon[32179]: pgmap v742: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 6.8 MiB/s wr, 798 op/s 2024-04-01T01:46:17.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:16 smithi081 ceph-mon[35142]: pgmap v742: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 6.8 MiB/s wr, 798 op/s 2024-04-01T01:46:17.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:17 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:46:18.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:17 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:46:18.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:17 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:46:18.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:18 smithi189 ceph-mon[35753]: pgmap v743: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 7.7 MiB/s wr, 891 op/s 2024-04-01T01:46:18.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:46:18.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:46:18.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:46:19.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:18 smithi070 ceph-mon[32179]: pgmap v743: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 7.7 MiB/s wr, 891 op/s 2024-04-01T01:46:19.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:46:19.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:46:19.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:46:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:18 smithi081 ceph-mon[35142]: pgmap v743: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 7.7 MiB/s wr, 891 op/s 2024-04-01T01:46:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:46:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:46:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:46:20.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:20 smithi189 ceph-mon[35753]: pgmap v744: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 7.4 MiB/s wr, 882 op/s 2024-04-01T01:46:21.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:20 smithi070 ceph-mon[32179]: pgmap v744: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 7.4 MiB/s wr, 882 op/s 2024-04-01T01:46:21.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:20 smithi081 ceph-mon[35142]: pgmap v744: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 7.4 MiB/s wr, 882 op/s 2024-04-01T01:46:21.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 695.0 s, 784.0 tps, lat 125.867 ms stddev 113.298 2024-04-01T01:46:22.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:22 smithi189 ceph-mon[35753]: pgmap v745: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 7.3 MiB/s wr, 844 op/s 2024-04-01T01:46:23.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:22 smithi070 ceph-mon[32179]: pgmap v745: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 7.3 MiB/s wr, 844 op/s 2024-04-01T01:46:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:22 smithi081 ceph-mon[35142]: pgmap v745: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 7.3 MiB/s wr, 844 op/s 2024-04-01T01:46:24.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:24 smithi189 ceph-mon[35753]: pgmap v746: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 7.7 MiB/s wr, 870 op/s 2024-04-01T01:46:25.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:24 smithi070 ceph-mon[32179]: pgmap v746: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 7.7 MiB/s wr, 870 op/s 2024-04-01T01:46:25.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:24 smithi081 ceph-mon[35142]: pgmap v746: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 7.7 MiB/s wr, 870 op/s 2024-04-01T01:46:26.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 700.0 s, 778.4 tps, lat 127.687 ms stddev 128.443 2024-04-01T01:46:26.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:26 smithi189 ceph-mon[35753]: pgmap v747: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 6.7 MiB/s wr, 736 op/s 2024-04-01T01:46:27.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:26 smithi070 ceph-mon[32179]: pgmap v747: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 6.7 MiB/s wr, 736 op/s 2024-04-01T01:46:27.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:26 smithi081 ceph-mon[35142]: pgmap v747: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 6.7 MiB/s wr, 736 op/s 2024-04-01T01:46:28.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:28 smithi189 ceph-mon[35753]: pgmap v748: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.5 MiB/s wr, 787 op/s 2024-04-01T01:46:29.006 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:28 smithi070 ceph-mon[32179]: pgmap v748: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.5 MiB/s wr, 787 op/s 2024-04-01T01:46:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:28 smithi081 ceph-mon[35142]: pgmap v748: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.5 MiB/s wr, 787 op/s 2024-04-01T01:46:29.924 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.923+0000 7f07ed48f640 1 -- 172.21.15.70:0/3222574967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07e8108750 msgr2=0x7f07e810ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.923+0000 7f07ed48f640 1 --2- 172.21.15.70:0/3222574967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07e8108750 0x7f07e810ab80 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f07e0009ca0 tx=0x7f07e002c100 comp rx=0 tx=0).stop 2024-04-01T01:46:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.923+0000 7f07ed48f640 1 -- 172.21.15.70:0/3222574967 shutdown_connections 2024-04-01T01:46:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.923+0000 7f07ed48f640 1 --2- 172.21.15.70:0/3222574967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07e810da70 0x7f07e810def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.923+0000 7f07ed48f640 1 --2- 172.21.15.70:0/3222574967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07e810b0c0 0x7f07e810d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.923+0000 7f07ed48f640 1 --2- 172.21.15.70:0/3222574967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07e8108750 0x7f07e810ab80 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.923+0000 7f07ed48f640 1 -- 172.21.15.70:0/3222574967 >> 172.21.15.70:0/3222574967 conn(0x7f07e806e790 msgr2=0x7f07e8070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.924+0000 7f07ed48f640 1 -- 172.21.15.70:0/3222574967 shutdown_connections 2024-04-01T01:46:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.924+0000 7f07ed48f640 1 -- 172.21.15.70:0/3222574967 wait complete. 2024-04-01T01:46:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.924+0000 7f07ed48f640 1 Processor -- start 2024-04-01T01:46:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.925+0000 7f07ed48f640 1 -- start start 2024-04-01T01:46:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.925+0000 7f07ed48f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07e8108750 0x7f07e81a6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.925+0000 7f07ed48f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07e810b0c0 0x7f07e81a6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.925+0000 7f07ed48f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07e810da70 0x7f07e81adc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.925+0000 7f07ed48f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f07e811c0a0 con 0x7f07e810b0c0 2024-04-01T01:46:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.925+0000 7f07ed48f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f07e811bf20 con 0x7f07e810da70 2024-04-01T01:46:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.925+0000 7f07ed48f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f07e811c220 con 0x7f07e8108750 2024-04-01T01:46:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.925+0000 7f07e6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07e8108750 0x7f07e81a6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.925+0000 7f07e6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07e8108750 0x7f07e81a6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43700/0 (socket says 172.21.15.70:43700) 2024-04-01T01:46:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.925+0000 7f07e6ffd640 1 -- 172.21.15.70:0/3563124332 learned_addr learned my addr 172.21.15.70:0/3563124332 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.925+0000 7f07e77fe640 1 --2- 172.21.15.70:0/3563124332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07e810da70 0x7f07e81adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.925+0000 7f07e67fc640 1 --2- 172.21.15.70:0/3563124332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07e810b0c0 0x7f07e81a6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.926+0000 7f07e6ffd640 1 -- 172.21.15.70:0/3563124332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07e810da70 msgr2=0x7f07e81adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.926+0000 7f07e6ffd640 1 --2- 172.21.15.70:0/3563124332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07e810da70 0x7f07e81adc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.926+0000 7f07e6ffd640 1 -- 172.21.15.70:0/3563124332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07e810b0c0 msgr2=0x7f07e81a6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.926+0000 7f07e6ffd640 1 --2- 172.21.15.70:0/3563124332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07e810b0c0 0x7f07e81a6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.926+0000 7f07e6ffd640 1 -- 172.21.15.70:0/3563124332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e81ae2d0 con 0x7f07e8108750 2024-04-01T01:46:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.926+0000 7f07e6ffd640 1 --2- 172.21.15.70:0/3563124332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07e8108750 0x7f07e81a6700 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f07e0032380 tx=0x7f07e0035ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.926+0000 7f07cffff640 1 -- 172.21.15.70:0/3563124332 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e0039040 con 0x7f07e8108750 2024-04-01T01:46:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.926+0000 7f07cffff640 1 -- 172.21.15.70:0/3563124332 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07e0044070 con 0x7f07e8108750 2024-04-01T01:46:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.926+0000 7f07cffff640 1 -- 172.21.15.70:0/3563124332 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e00043c0 con 0x7f07e8108750 2024-04-01T01:46:29.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.926+0000 7f07ed48f640 1 -- 172.21.15.70:0/3563124332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07e81ae560 con 0x7f07e8108750 2024-04-01T01:46:29.930 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.926+0000 7f07ed48f640 1 -- 172.21.15.70:0/3563124332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07e81aea00 con 0x7f07e8108750 2024-04-01T01:46:29.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.930+0000 7f07f007c740 1 Processor -- start 2024-04-01T01:46:29.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.930+0000 7f07f007c740 1 -- start start 2024-04-01T01:46:29.932 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.930+0000 7f07cffff640 1 -- 172.21.15.70:0/3563124332 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f07e0004560 con 0x7f07e8108750 2024-04-01T01:46:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.930+0000 7f07f007c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 0x55a040203cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.930+0000 7f07f007c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0402042c0 0x55a040208dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.930+0000 7f07f007c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a040209430 0x55a04020b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.930+0000 7f07f007c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a0401678f0 con 0x55a040209430 2024-04-01T01:46:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.930+0000 7f07f007c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a0401108d0 con 0x55a0402038d0 2024-04-01T01:46:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.930+0000 7f07f007c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a040163990 con 0x55a0402042c0 2024-04-01T01:46:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.931+0000 7f07e67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a040209430 0x55a04020b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.931+0000 7f07e67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a040209430 0x55a04020b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45368/0 (socket says 172.21.15.70:45368) 2024-04-01T01:46:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.931+0000 7f07e67fc640 1 -- 172.21.15.70:0/4014339751 learned_addr learned my addr 172.21.15.70:0/4014339751 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.931+0000 7f07e77fe640 1 --2- 172.21.15.70:0/4014339751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 0x55a040203cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.931+0000 7f07e6ffd640 1 --2- 172.21.15.70:0/4014339751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0402042c0 0x55a040208dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.931+0000 7f07cffff640 1 --2- 172.21.15.70:0/3563124332 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07d00a26b0 0x7f07d00a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.931+0000 7f07e77fe640 1 -- 172.21.15.70:0/4014339751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0402042c0 msgr2=0x55a040208dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.931+0000 7f07e77fe640 1 --2- 172.21.15.70:0/4014339751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0402042c0 0x55a040208dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.931+0000 7f07e77fe640 1 -- 172.21.15.70:0/4014339751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a040209430 msgr2=0x55a04020b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.931+0000 7f07e77fe640 1 --2- 172.21.15.70:0/4014339751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a040209430 0x55a04020b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.931+0000 7f07e77fe640 1 -- 172.21.15.70:0/4014339751 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a03fe7c210 con 0x55a0402038d0 2024-04-01T01:46:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.931+0000 7f07cffff640 1 -- 172.21.15.70:0/3563124332 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f07e00f0930 con 0x7f07e8108750 2024-04-01T01:46:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.931+0000 7f07e6ffd640 1 --2- 172.21.15.70:0/4014339751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0402042c0 0x55a040208dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:46:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.931+0000 7f07e77fe640 1 --2- 172.21.15.70:0/4014339751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 0x55a040203cf0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f07dc004640 tx=0x7f07dc00ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87a1eb12901ac184 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.931+0000 7f07e77fe640 1 --2- 172.21.15.70:0/3563124332 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07d00a26b0 0x7f07d00a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.932+0000 7f07cd7fa640 1 -- 172.21.15.70:0/4014339751 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc003c10 con 0x55a0402038d0 2024-04-01T01:46:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.932+0000 7f07cd7fa640 1 -- 172.21.15.70:0/4014339751 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07dc015070 con 0x55a0402038d0 2024-04-01T01:46:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.932+0000 7f07cd7fa640 1 -- 172.21.15.70:0/4014339751 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc011840 con 0x55a0402038d0 2024-04-01T01:46:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.933+0000 7f07f007c740 1 -- 172.21.15.70:0/4014339751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 msgr2=0x55a040203cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.933+0000 7f07f007c740 1 --2- 172.21.15.70:0/4014339751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 0x55a040203cf0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f07dc004640 tx=0x7f07dc00ba90 comp rx=0 tx=0).stop 2024-04-01T01:46:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.933+0000 7f07e77fe640 1 --2- 172.21.15.70:0/3563124332 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07d00a26b0 0x7f07d00a4b70 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f07dc011fd0 tx=0x7f07dc011ac0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.933+0000 7f07f007c740 1 -- 172.21.15.70:0/4014339751 shutdown_connections 2024-04-01T01:46:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.933+0000 7f07f007c740 1 --2- 172.21.15.70:0/4014339751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a040209430 0x55a04020b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.933+0000 7f07f007c740 1 --2- 172.21.15.70:0/4014339751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0402042c0 0x55a040208dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.933+0000 7f07f007c740 1 --2- 172.21.15.70:0/4014339751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 0x55a040203cf0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.933+0000 7f07f007c740 1 -- 172.21.15.70:0/4014339751 >> 172.21.15.70:0/4014339751 conn(0x55a0401fee40 msgr2=0x55a0401ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.933+0000 7f07f007c740 1 -- 172.21.15.70:0/4014339751 shutdown_connections 2024-04-01T01:46:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.934+0000 7f07f007c740 1 -- 172.21.15.70:0/4014339751 wait complete. 2024-04-01T01:46:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.934+0000 7f07f007c740 1 -- start start 2024-04-01T01:46:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.934+0000 7f07f007c740 1 Processor -- start 2024-04-01T01:46:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.934+0000 7f07f007c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0402038d0 0x55a04021eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.935+0000 7f07f007c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0402042c0 0x55a04021f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.935+0000 7f07f007c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a040209430 0x55a040226110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.935+0000 7f07f007c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a04021d3f0 con 0x55a0402042c0 2024-04-01T01:46:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.935+0000 7f07f007c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a040105f80 con 0x55a040209430 2024-04-01T01:46:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.935+0000 7f07f007c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a04021d570 con 0x55a0402038d0 2024-04-01T01:46:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.935+0000 7f07f007c740 10 client.0 _ll_register_callbacks cb 0x55a04015e290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.935+0000 7f07e77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a040209430 0x55a040226110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.935+0000 7f07e77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a040209430 0x55a040226110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38020/0 (socket says 172.21.15.70:38020) 2024-04-01T01:46:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.935+0000 7f07e77fe640 1 -- 172.21.15.70:0/955103256 learned_addr learned my addr 172.21.15.70:0/955103256 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.935+0000 7f07e77fe640 1 -- 172.21.15.70:0/955103256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0402038d0 msgr2=0x55a04021eb90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.935+0000 7f07e77fe640 1 --2- 172.21.15.70:0/955103256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0402038d0 0x55a04021eb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.935+0000 7f07e77fe640 1 -- 172.21.15.70:0/955103256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0402042c0 msgr2=0x55a04021f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.935+0000 7f07e77fe640 1 --2- 172.21.15.70:0/955103256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0402042c0 0x55a04021f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.935+0000 7f07e77fe640 1 -- 172.21.15.70:0/955103256 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a03ff48620 con 0x55a040209430 2024-04-01T01:46:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.936+0000 7f07e77fe640 1 --2- 172.21.15.70:0/955103256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a040209430 0x55a040226110 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f07dc011f60 tx=0x7f07dc023000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8eac8208046a0c34 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:29.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.936+0000 7f07caff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:29.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.939+0000 7f07caff5640 1 -- 172.21.15.70:0/955103256 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc004210 con 0x55a040209430 2024-04-01T01:46:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.939+0000 7f07caff5640 1 -- 172.21.15.70:0/955103256 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f07dc011750 con 0x55a040209430 2024-04-01T01:46:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.939+0000 7f07f007c740 1 -- 172.21.15.70:0/955103256 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07dc015070 con 0x55a040209430 2024-04-01T01:46:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.940+0000 7f07caff5640 1 -- 172.21.15.70:0/955103256 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f07dc0039b0 con 0x55a040209430 2024-04-01T01:46:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.940+0000 7f07f007c740 10 client.29294 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:29.940 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.940+0000 7f07f007c740 1 -- 172.21.15.70:0/955103256 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a040229190 con 0x55a040209430 2024-04-01T01:46:29.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.940+0000 7f07caff5640 1 -- 172.21.15.70:0/955103256 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07dc015070 con 0x55a040209430 2024-04-01T01:46:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.941+0000 7f07caff5640 1 -- 172.21.15.70:0/955103256 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f07dc01b510 con 0x55a040209430 2024-04-01T01:46:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.941+0000 7f07f007c740 10 client.29294 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.941+0000 7f07f007c740 10 client.29294 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.941+0000 7f07f007c740 20 client.29294 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.941+0000 7f07f007c740 1 --2- 172.21.15.70:0/955103256 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a04022a6c0 0x55a04022cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.941+0000 7f07f007c740 4 client.29294 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.941+0000 7f07f007c740 1 -- 172.21.15.70:0/955103256 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a04017b120 con 0x55a04022a6c0 2024-04-01T01:46:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.941+0000 7f07e6ffd640 1 --2- 172.21.15.70:0/955103256 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a04022a6c0 0x55a04022cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.944+0000 7f07e6ffd640 1 --2- 172.21.15.70:0/955103256 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a04022a6c0 0x55a04022cb60 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf6222cd0a300968 server_cookie=f4cd26b53e1fde35 in_seq=0 out_seq=0 2024-04-01T01:46:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.944+0000 7f07caff5640 10 client.29294 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.946+0000 7f07caff5640 1 -- 172.21.15.70:0/955103256 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a04017b120 con 0x55a04022a6c0 2024-04-01T01:46:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.946+0000 7f07caff5640 10 client.29294 handle_command_reply: tid=0 2024-04-01T01:46:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.946+0000 7f07f007c740 1 client.29294 shutdown 2024-04-01T01:46:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.947+0000 7f07f007c740 1 -- 172.21.15.70:0/955103256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a040209430 msgr2=0x55a040226110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.947+0000 7f07f007c740 1 --2- 172.21.15.70:0/955103256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a040209430 0x55a040226110 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f07dc011f60 tx=0x7f07dc023000 comp rx=0 tx=0).stop 2024-04-01T01:46:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.948+0000 7f07f007c740 1 -- 172.21.15.70:0/955103256 shutdown_connections 2024-04-01T01:46:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.948+0000 7f07f007c740 1 --2- 172.21.15.70:0/955103256 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a04022a6c0 0x55a04022cb60 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.948+0000 7f07f007c740 1 --2- 172.21.15.70:0/955103256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a040209430 0x55a040226110 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.948+0000 7f07f007c740 1 --2- 172.21.15.70:0/955103256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0402042c0 0x55a04021f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.949+0000 7f07f007c740 1 --2- 172.21.15.70:0/955103256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0402038d0 0x55a04021eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.949+0000 7f07f007c740 1 -- 172.21.15.70:0/955103256 >> 172.21.15.70:0/955103256 conn(0x55a0401fee40 msgr2=0x55a0401fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:29.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.949+0000 7f07caff5640 0 client.29294 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:29.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.949+0000 7f07f007c740 1 -- 172.21.15.70:0/955103256 shutdown_connections 2024-04-01T01:46:29.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.949+0000 7f07f007c740 1 -- 172.21.15.70:0/955103256 wait complete. 2024-04-01T01:46:29.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.950+0000 7f07f007c740 20 client.29294 trim_cache size 0 max 16384 2024-04-01T01:46:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.982+0000 7f07f007c740 1 Processor -- start 2024-04-01T01:46:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.982+0000 7f07f007c740 1 -- start start 2024-04-01T01:46:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.982+0000 7f07f007c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 0x55a04020b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.982+0000 7f07f007c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0402042c0 0x55a0402260d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.982+0000 7f07f007c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a040209430 0x55a040208b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.983+0000 7f07f007c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f07d400fd90 con 0x55a0402042c0 2024-04-01T01:46:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.983+0000 7f07f007c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f07d40118d0 con 0x55a0402038d0 2024-04-01T01:46:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.983+0000 7f07f007c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f07d4010460 con 0x55a040209430 2024-04-01T01:46:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.983+0000 7f07e67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0402042c0 0x55a0402260d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.983+0000 7f07e67fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0402042c0 0x55a0402260d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45380/0 (socket says 172.21.15.70:45380) 2024-04-01T01:46:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.983+0000 7f07e67fc640 1 -- 172.21.15.70:0/1554348636 learned_addr learned my addr 172.21.15.70:0/1554348636 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.983+0000 7f07e6ffd640 1 --2- 172.21.15.70:0/1554348636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 0x55a04020b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.983+0000 7f07e77fe640 1 --2- 172.21.15.70:0/1554348636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a040209430 0x55a040208b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.983+0000 7f07e67fc640 1 -- 172.21.15.70:0/1554348636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a040209430 msgr2=0x55a040208b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.983+0000 7f07e67fc640 1 --2- 172.21.15.70:0/1554348636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a040209430 0x55a040208b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.983+0000 7f07e67fc640 1 -- 172.21.15.70:0/1554348636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 msgr2=0x55a04020b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.983+0000 7f07e67fc640 1 --2- 172.21.15.70:0/1554348636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 0x55a04020b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.983+0000 7f07e67fc640 1 -- 172.21.15.70:0/1554348636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0402046a0 con 0x55a0402042c0 2024-04-01T01:46:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.983+0000 7f07e67fc640 1 --2- 172.21.15.70:0/1554348636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0402042c0 0x55a0402260d0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f07d8009cc0 tx=0x7f07d8012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a16a2147f7ba36d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.984+0000 7f07ccff9640 1 -- 172.21.15.70:0/1554348636 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d8009db0 con 0x55a0402042c0 2024-04-01T01:46:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.984+0000 7f07ccff9640 1 -- 172.21.15.70:0/1554348636 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07d80120d0 con 0x55a0402042c0 2024-04-01T01:46:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.984+0000 7f07ccff9640 1 -- 172.21.15.70:0/1554348636 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d801b620 con 0x55a0402042c0 2024-04-01T01:46:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.985+0000 7f07f007c740 1 -- 172.21.15.70:0/1554348636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0402042c0 msgr2=0x55a0402260d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.985+0000 7f07f007c740 1 --2- 172.21.15.70:0/1554348636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0402042c0 0x55a0402260d0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f07d8009cc0 tx=0x7f07d8012820 comp rx=0 tx=0).stop 2024-04-01T01:46:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.985+0000 7f07f007c740 1 -- 172.21.15.70:0/1554348636 shutdown_connections 2024-04-01T01:46:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.985+0000 7f07f007c740 1 --2- 172.21.15.70:0/1554348636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a040209430 0x55a040208b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.985+0000 7f07f007c740 1 --2- 172.21.15.70:0/1554348636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0402042c0 0x55a0402260d0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.985+0000 7f07f007c740 1 --2- 172.21.15.70:0/1554348636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 0x55a04020b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.985+0000 7f07f007c740 1 -- 172.21.15.70:0/1554348636 >> 172.21.15.70:0/1554348636 conn(0x55a0401fee40 msgr2=0x55a0401ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.987+0000 7f07f007c740 1 -- 172.21.15.70:0/1554348636 shutdown_connections 2024-04-01T01:46:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.987+0000 7f07f007c740 1 -- 172.21.15.70:0/1554348636 wait complete. 2024-04-01T01:46:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.987+0000 7f07f007c740 1 -- start start 2024-04-01T01:46:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.987+0000 7f07f007c740 1 Processor -- start 2024-04-01T01:46:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.988+0000 7f07f007c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 0x55a040222f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.988 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.988+0000 7f07f007c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0402042c0 0x55a040223490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.988+0000 7f07e6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 0x55a040222f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.988+0000 7f07e6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 0x55a040222f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:38044/0 (socket says 172.21.15.70:38044) 2024-04-01T01:46:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.988+0000 7f07e6ffd640 1 -- 172.21.15.70:0/1232893108 learned_addr learned my addr 172.21.15.70:0/1232893108 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.988+0000 7f07e67fc640 1 --2- 172.21.15.70:0/1232893108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0402042c0 0x55a040223490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.989+0000 7f07f007c740 1 --2- 172.21.15.70:0/1232893108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a040209430 0x55a04028ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.989+0000 7f07f007c740 1 -- 172.21.15.70:0/1232893108 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f07d4011200 con 0x55a040209430 2024-04-01T01:46:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.989+0000 7f07f007c740 1 -- 172.21.15.70:0/1232893108 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f07d4010b30 con 0x55a0402038d0 2024-04-01T01:46:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.989+0000 7f07f007c740 1 -- 172.21.15.70:0/1232893108 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f07d400dd70 con 0x55a0402042c0 2024-04-01T01:46:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.989+0000 7f07f007c740 10 client.0 _ll_register_callbacks cb 0x7f07d80097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:29.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.990+0000 7f07e6ffd640 1 -- 172.21.15.70:0/1232893108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0402042c0 msgr2=0x55a040223490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:29.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.990+0000 7f07e6ffd640 1 --2- 172.21.15.70:0/1232893108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0402042c0 0x55a040223490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.990+0000 7f07e6ffd640 1 -- 172.21.15.70:0/1232893108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a040209430 msgr2=0x55a04028ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:29.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.990+0000 7f07e6ffd640 1 --2- 172.21.15.70:0/1232893108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a040209430 0x55a04028ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:29.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.990+0000 7f07e6ffd640 1 -- 172.21.15.70:0/1232893108 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a04028efd0 con 0x55a0402038d0 2024-04-01T01:46:29.991 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.990+0000 7f07e6ffd640 1 --2- 172.21.15.70:0/1232893108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 0x55a040222f50 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f07e00bcfb0 tx=0x7f07e002c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1eebcd449a340651 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:29.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.990+0000 7f07caff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:29.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.990+0000 7f07caff5640 1 -- 172.21.15.70:0/1232893108 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07e00bd720 con 0x55a0402038d0 2024-04-01T01:46:29.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.991+0000 7f07e67fc640 1 --2- 172.21.15.70:0/1232893108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0402042c0 0x55a040223490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:46:29.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.991+0000 7f07caff5640 1 -- 172.21.15.70:0/1232893108 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f07e00bb320 con 0x55a0402038d0 2024-04-01T01:46:29.992 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.991+0000 7f07f007c740 1 -- 172.21.15.70:0/1232893108 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07d80120d0 con 0x55a0402038d0 2024-04-01T01:46:29.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.991+0000 7f07caff5640 1 -- 172.21.15.70:0/1232893108 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f07e00bd020 con 0x55a0402038d0 2024-04-01T01:46:29.993 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.991+0000 7f07f007c740 10 client.29300 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.991+0000 7f07f007c740 1 -- 172.21.15.70:0/1232893108 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a0402902b0 con 0x55a0402038d0 2024-04-01T01:46:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.992+0000 7f07caff5640 1 -- 172.21.15.70:0/1232893108 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f07e00c1170 con 0x55a0402038d0 2024-04-01T01:46:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.992+0000 7f07caff5640 1 -- 172.21.15.70:0/1232893108 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f07e003c030 con 0x55a0402038d0 2024-04-01T01:46:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.992+0000 7f07f007c740 10 client.29300 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.992+0000 7f07f007c740 10 client.29300 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.992+0000 7f07f007c740 20 client.29300 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.992+0000 7f07f007c740 1 --2- 172.21.15.70:0/1232893108 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a040290cb0 0x55a0402930d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.992+0000 7f07f007c740 4 client.29300 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:46:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.993+0000 7f07f007c740 1 -- 172.21.15.70:0/1232893108 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f07d80120d0 con 0x55a040290cb0 2024-04-01T01:46:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.993+0000 7f07e67fc640 1 --2- 172.21.15.70:0/1232893108 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a040290cb0 0x55a0402930d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.993+0000 7f07e67fc640 1 --2- 172.21.15.70:0/1232893108 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a040290cb0 0x55a0402930d0 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6b7f35192b26662 server_cookie=a0919d6beda83577 in_seq=0 out_seq=0 2024-04-01T01:46:29.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:29.993+0000 7f07caff5640 10 client.29300 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:30.049 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.049+0000 7f07caff5640 1 -- 172.21.15.70:0/1232893108 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f07d80120d0 con 0x55a040290cb0 2024-04-01T01:46:30.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.049+0000 7f07caff5640 10 client.29300 handle_command_reply: tid=0 2024-04-01T01:46:30.050 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.049+0000 7f07f007c740 1 client.29300 shutdown 2024-04-01T01:46:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.050+0000 7f07f007c740 1 -- 172.21.15.70:0/1232893108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 msgr2=0x55a040222f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.050+0000 7f07f007c740 1 --2- 172.21.15.70:0/1232893108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 0x55a040222f50 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f07e00bcfb0 tx=0x7f07e002c100 comp rx=0 tx=0).stop 2024-04-01T01:46:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.050+0000 7f07f007c740 1 -- 172.21.15.70:0/1232893108 shutdown_connections 2024-04-01T01:46:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.050+0000 7f07f007c740 1 --2- 172.21.15.70:0/1232893108 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a040290cb0 0x55a0402930d0 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.050+0000 7f07f007c740 1 --2- 172.21.15.70:0/1232893108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a040209430 0x55a04028ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:30.051 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.050+0000 7f07caff5640 0 client.29300 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.050+0000 7f07f007c740 1 --2- 172.21.15.70:0/1232893108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a0402042c0 0x55a040223490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.050+0000 7f07f007c740 1 --2- 172.21.15.70:0/1232893108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a0402038d0 0x55a040222f50 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.050+0000 7f07f007c740 1 -- 172.21.15.70:0/1232893108 >> 172.21.15.70:0/1232893108 conn(0x55a0401fee40 msgr2=0x55a0401ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.050+0000 7f07f007c740 1 -- 172.21.15.70:0/1232893108 shutdown_connections 2024-04-01T01:46:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.050+0000 7f07f007c740 1 -- 172.21.15.70:0/1232893108 wait complete. 2024-04-01T01:46:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.051+0000 7f07f007c740 20 client.29300 trim_cache size 0 max 16384 2024-04-01T01:46:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:46:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:46:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:46:30.052 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:46:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.051+0000 7f07ed48f640 1 -- 172.21.15.70:0/3563124332 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07d00a26b0 msgr2=0x7f07d00a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:30.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.051+0000 7f07ed48f640 1 --2- 172.21.15.70:0/3563124332 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07d00a26b0 0x7f07d00a4b70 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f07dc011fd0 tx=0x7f07dc011ac0 comp rx=0 tx=0).stop 2024-04-01T01:46:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.051+0000 7f07ed48f640 1 -- 172.21.15.70:0/3563124332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07e8108750 msgr2=0x7f07e81a6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.051+0000 7f07ed48f640 1 --2- 172.21.15.70:0/3563124332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07e8108750 0x7f07e81a6700 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f07e0032380 tx=0x7f07e0035ce0 comp rx=0 tx=0).stop 2024-04-01T01:46:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.052+0000 7f07ed48f640 1 -- 172.21.15.70:0/3563124332 shutdown_connections 2024-04-01T01:46:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.052+0000 7f07ed48f640 1 --2- 172.21.15.70:0/3563124332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07e810da70 0x7f07e81adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.052+0000 7f07ed48f640 1 --2- 172.21.15.70:0/3563124332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07e810b0c0 0x7f07e81a6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.052+0000 7f07ed48f640 1 --2- 172.21.15.70:0/3563124332 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07d00a26b0 0x7f07d00a4b70 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.052+0000 7f07ed48f640 1 --2- 172.21.15.70:0/3563124332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07e8108750 0x7f07e81a6700 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.052+0000 7f07ed48f640 1 -- 172.21.15.70:0/3563124332 >> 172.21.15.70:0/3563124332 conn(0x7f07e806e790 msgr2=0x7f07e810b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.052+0000 7f07ed48f640 1 -- 172.21.15.70:0/3563124332 shutdown_connections 2024-04-01T01:46:30.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:30.052+0000 7f07ed48f640 1 -- 172.21.15.70:0/3563124332 wait complete. 2024-04-01T01:46:30.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:30 smithi189 ceph-mon[35753]: pgmap v749: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 7.6 MiB/s wr, 776 op/s 2024-04-01T01:46:31.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:30 smithi070 ceph-mon[32179]: pgmap v749: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 7.6 MiB/s wr, 776 op/s 2024-04-01T01:46:31.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:30 smithi081 ceph-mon[35142]: pgmap v749: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 7.6 MiB/s wr, 776 op/s 2024-04-01T01:46:31.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 705.0 s, 770.0 tps, lat 131.599 ms stddev 125.275 2024-04-01T01:46:32.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:32 smithi189 ceph-mon[35753]: pgmap v750: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.8 MiB/s wr, 704 op/s 2024-04-01T01:46:33.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:32 smithi070 ceph-mon[32179]: pgmap v750: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.8 MiB/s wr, 704 op/s 2024-04-01T01:46:33.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:32 smithi081 ceph-mon[35142]: pgmap v750: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.8 MiB/s wr, 704 op/s 2024-04-01T01:46:34.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:34 smithi189 ceph-mon[35753]: pgmap v751: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 750 op/s 2024-04-01T01:46:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:34 smithi070 ceph-mon[32179]: pgmap v751: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 750 op/s 2024-04-01T01:46:35.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:34 smithi081 ceph-mon[35142]: pgmap v751: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 750 op/s 2024-04-01T01:46:36.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 710.0 s, 779.2 tps, lat 127.864 ms stddev 122.733 2024-04-01T01:46:36.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:36 smithi189 ceph-mon[35753]: pgmap v752: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 6.2 MiB/s wr, 668 op/s 2024-04-01T01:46:37.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:36 smithi070 ceph-mon[32179]: pgmap v752: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 6.2 MiB/s wr, 668 op/s 2024-04-01T01:46:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:36 smithi081 ceph-mon[35142]: pgmap v752: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 6.2 MiB/s wr, 668 op/s 2024-04-01T01:46:38.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:38 smithi189 ceph-mon[35753]: pgmap v753: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.0 MiB/s wr, 747 op/s 2024-04-01T01:46:39.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:38 smithi070 ceph-mon[32179]: pgmap v753: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.0 MiB/s wr, 747 op/s 2024-04-01T01:46:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:38 smithi081 ceph-mon[35142]: pgmap v753: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 7.0 MiB/s wr, 747 op/s 2024-04-01T01:46:39.513 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:46:39.513 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:46:39.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.691+0000 7f6f9ec9d640 1 -- 172.21.15.70:0/1156383507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f9810de00 msgr2=0x7f6f981102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:39.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.691+0000 7f6f9ec9d640 1 --2- 172.21.15.70:0/1156383507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f9810de00 0x7f6f981102c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f6f90009ca0 tx=0x7f6f9002c120 comp rx=0 tx=0).stop 2024-04-01T01:46:39.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.691+0000 7f6f9ec9d640 1 -- 172.21.15.70:0/1156383507 shutdown_connections 2024-04-01T01:46:39.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.691+0000 7f6f9ec9d640 1 --2- 172.21.15.70:0/1156383507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f9810de00 0x7f6f981102c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.691+0000 7f6f9ec9d640 1 --2- 172.21.15.70:0/1156383507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f9810b380 0x7f6f9810d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.691+0000 7f6f9ec9d640 1 --2- 172.21.15.70:0/1156383507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f981089c0 0x7f6f9810adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.691+0000 7f6f9ec9d640 1 -- 172.21.15.70:0/1156383507 >> 172.21.15.70:0/1156383507 conn(0x7f6f9806e950 msgr2=0x7f6f98070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:39.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.692+0000 7f6f9ec9d640 1 -- 172.21.15.70:0/1156383507 shutdown_connections 2024-04-01T01:46:39.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.692+0000 7f6f9ec9d640 1 -- 172.21.15.70:0/1156383507 wait complete. 2024-04-01T01:46:39.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.693+0000 7f6f9ec9d640 1 Processor -- start 2024-04-01T01:46:39.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.693+0000 7f6f9ec9d640 1 -- start start 2024-04-01T01:46:39.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.693+0000 7f6f9ec9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f981089c0 0x7f6f98119680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.693+0000 7f6f9ec9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f9810b380 0x7f6f98119bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.693+0000 7f6f9ec9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f9810de00 0x7f6f98117cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.693+0000 7f6f9ec9d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6f98112510 con 0x7f6f9810b380 2024-04-01T01:46:39.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.693+0000 7f6f9ec9d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f98112390 con 0x7f6f9810de00 2024-04-01T01:46:39.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.693+0000 7f6f9ec9d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6f98112690 con 0x7f6f981089c0 2024-04-01T01:46:39.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.693+0000 7f6f9ca39640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f981089c0 0x7f6f98119680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.693+0000 7f6f9ca39640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f981089c0 0x7f6f98119680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57652/0 (socket says 172.21.15.70:57652) 2024-04-01T01:46:39.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.693+0000 7f6f9ca39640 1 -- 172.21.15.70:0/2050083510 learned_addr learned my addr 172.21.15.70:0/2050083510 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:39.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.694+0000 7f6f97fff640 1 --2- 172.21.15.70:0/2050083510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f9810b380 0x7f6f98119bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.694+0000 7f6f9d23a640 1 --2- 172.21.15.70:0/2050083510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f9810de00 0x7f6f98117cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.694+0000 7f6f9ca39640 1 -- 172.21.15.70:0/2050083510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f9810de00 msgr2=0x7f6f98117cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:39.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.694+0000 7f6f9ca39640 1 --2- 172.21.15.70:0/2050083510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f9810de00 0x7f6f98117cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.694+0000 7f6f9ca39640 1 -- 172.21.15.70:0/2050083510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f9810b380 msgr2=0x7f6f98119bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:39.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.694+0000 7f6f9ca39640 1 --2- 172.21.15.70:0/2050083510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f9810b380 0x7f6f98119bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.694+0000 7f6f9ca39640 1 -- 172.21.15.70:0/2050083510 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f98118360 con 0x7f6f981089c0 2024-04-01T01:46:39.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.694+0000 7f6f97fff640 1 --2- 172.21.15.70:0/2050083510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f9810b380 0x7f6f98119bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:46:39.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.694+0000 7f6f9ca39640 1 --2- 172.21.15.70:0/2050083510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f981089c0 0x7f6f98119680 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6f8800c890 tx=0x7f6f8800cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:39.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.695+0000 7f6f95ffb640 1 -- 172.21.15.70:0/2050083510 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f88004220 con 0x7f6f981089c0 2024-04-01T01:46:39.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.695+0000 7f6f95ffb640 1 -- 172.21.15.70:0/2050083510 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f880043c0 con 0x7f6f981089c0 2024-04-01T01:46:39.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.695+0000 7f6f95ffb640 1 -- 172.21.15.70:0/2050083510 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f88004a00 con 0x7f6f981089c0 2024-04-01T01:46:39.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.695+0000 7f6f9ec9d640 1 -- 172.21.15.70:0/2050083510 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f98118650 con 0x7f6f981089c0 2024-04-01T01:46:39.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.695+0000 7f6f9ec9d640 1 -- 172.21.15.70:0/2050083510 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f980799b0 con 0x7f6f981089c0 2024-04-01T01:46:39.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.699+0000 7f6fa1881740 1 Processor -- start 2024-04-01T01:46:39.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.699+0000 7f6fa1881740 1 -- start start 2024-04-01T01:46:39.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.699+0000 7f6fa1881740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316a86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.699+0000 7f6fa1881740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316a8ca0 0x5603316ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.699+0000 7f6fa1881740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316ade10 0x5603316b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.699+0000 7f6fa1881740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56033160d180 con 0x5603316ade10 2024-04-01T01:46:39.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.699+0000 7f6fa1881740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5603315b5b70 con 0x5603316a82b0 2024-04-01T01:46:39.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.699+0000 7f6fa1881740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560331609220 con 0x5603316a8ca0 2024-04-01T01:46:39.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.699+0000 7f6f97fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316ade10 0x5603316b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.700+0000 7f6f97fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316ade10 0x5603316b02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39100/0 (socket says 172.21.15.70:39100) 2024-04-01T01:46:39.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.700+0000 7f6f97fff640 1 -- 172.21.15.70:0/218886033 learned_addr learned my addr 172.21.15.70:0/218886033 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:39.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.700+0000 7f6f95ffb640 1 -- 172.21.15.70:0/2050083510 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f6f88004ba0 con 0x7f6f981089c0 2024-04-01T01:46:39.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.700+0000 7f6f97fff640 1 -- 172.21.15.70:0/218886033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316a8ca0 msgr2=0x5603316ad7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:39.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.700+0000 7f6f97fff640 1 --2- 172.21.15.70:0/218886033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316a8ca0 0x5603316ad7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.700+0000 7f6f97fff640 1 -- 172.21.15.70:0/218886033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 msgr2=0x5603316a86d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:39.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.700+0000 7f6f97fff640 1 --2- 172.21.15.70:0/218886033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316a86d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.700+0000 7f6f97fff640 1 -- 172.21.15.70:0/218886033 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603315c97c0 con 0x5603316ade10 2024-04-01T01:46:39.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.701+0000 7f6f95ffb640 1 --2- 172.21.15.70:0/2050083510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f800a2550 0x7f6f800a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.701+0000 7f6f97fff640 1 --2- 172.21.15.70:0/218886033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316ade10 0x5603316b02a0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c00bce0 tx=0x7f6f8c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee6527f8acf309a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:39.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.701+0000 7f6f95ffb640 1 -- 172.21.15.70:0/2050083510 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6f880ce5d0 con 0x7f6f981089c0 2024-04-01T01:46:39.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.701+0000 7f6f7effd640 1 -- 172.21.15.70:0/218886033 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c015070 con 0x5603316ade10 2024-04-01T01:46:39.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.702+0000 7f6f7effd640 1 -- 172.21.15.70:0/218886033 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f8c01a910 con 0x5603316ade10 2024-04-01T01:46:39.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.702+0000 7f6fa1881740 1 -- 172.21.15.70:0/218886033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316ade10 msgr2=0x5603316b02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:39.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.702+0000 7f6fa1881740 1 --2- 172.21.15.70:0/218886033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316ade10 0x5603316b02a0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c00bce0 tx=0x7f6f8c00f730 comp rx=0 tx=0).stop 2024-04-01T01:46:39.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.703+0000 7f6fa1881740 1 -- 172.21.15.70:0/218886033 shutdown_connections 2024-04-01T01:46:39.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.703+0000 7f6fa1881740 1 --2- 172.21.15.70:0/218886033 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316ade10 0x5603316b02a0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.703+0000 7f6fa1881740 1 --2- 172.21.15.70:0/218886033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316a8ca0 0x5603316ad7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.703+0000 7f6fa1881740 1 --2- 172.21.15.70:0/218886033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316a86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.703+0000 7f6fa1881740 1 -- 172.21.15.70:0/218886033 >> 172.21.15.70:0/218886033 conn(0x5603316a3820 msgr2=0x5603316a3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.703+0000 7f6fa1881740 1 -- 172.21.15.70:0/218886033 shutdown_connections 2024-04-01T01:46:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.703+0000 7f6fa1881740 1 -- 172.21.15.70:0/218886033 wait complete. 2024-04-01T01:46:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.703+0000 7f6f9d23a640 1 --2- 172.21.15.70:0/2050083510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f800a2550 0x7f6f800a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.703+0000 7f6fa1881740 1 -- start start 2024-04-01T01:46:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.703+0000 7f6fa1881740 1 Processor -- start 2024-04-01T01:46:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.704+0000 7f6fa1881740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316b27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.704+0000 7f6fa1881740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 0x5603316b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.704+0000 7f6f9ca39640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316b27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.704+0000 7f6fa1881740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316ade10 0x5603316b9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.704+0000 7f6fa1881740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5603316b0e10 con 0x5603316a8ca0 2024-04-01T01:46:39.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.704+0000 7f6fa1881740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5603315ab4a0 con 0x5603316a82b0 2024-04-01T01:46:39.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.704+0000 7f6fa1881740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5603316b0f90 con 0x5603316ade10 2024-04-01T01:46:39.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.704+0000 7f6fa1881740 10 client.0 _ll_register_callbacks cb 0x5603315aed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:39.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.704+0000 7f6f97fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 0x5603316b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.704+0000 7f6f97fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 0x5603316b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39102/0 (socket says 172.21.15.70:39102) 2024-04-01T01:46:39.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.704+0000 7f6f97fff640 1 -- 172.21.15.70:0/1955247713 learned_addr learned my addr 172.21.15.70:0/1955247713 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:39.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.705+0000 7f6f9ca39640 1 -- 172.21.15.70:0/1955247713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316ade10 msgr2=0x5603316b9d30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:39.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.705+0000 7f6f9ca39640 1 --2- 172.21.15.70:0/1955247713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316ade10 0x5603316b9d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.705+0000 7f6f9ca39640 1 -- 172.21.15.70:0/1955247713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 msgr2=0x5603316b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:39.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.705+0000 7f6f9ca39640 1 --2- 172.21.15.70:0/1955247713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 0x5603316b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.705+0000 7f6f9ca39640 1 -- 172.21.15.70:0/1955247713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603313eb7d0 con 0x5603316a82b0 2024-04-01T01:46:39.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.706+0000 7f6f9ca39640 1 --2- 172.21.15.70:0/1955247713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316b27b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x5603316a9710 tx=0x7f6f8809ab40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c0dda66f2c497ac server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:39.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.706+0000 7f6f77fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:39.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.706+0000 7f6f77fff640 1 -- 172.21.15.70:0/1955247713 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f880b3060 con 0x5603316a82b0 2024-04-01T01:46:39.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.706+0000 7f6f77fff640 1 -- 172.21.15.70:0/1955247713 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6f88004e90 con 0x5603316a82b0 2024-04-01T01:46:39.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.707+0000 7f6f9d23a640 1 --2- 172.21.15.70:0/2050083510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f800a2550 0x7f6f800a4a10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6f90006010 tx=0x7f6f90005ef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:39.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.708+0000 7f6fa1881740 1 -- 172.21.15.70:0/1955247713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f8c01a910 con 0x5603316a82b0 2024-04-01T01:46:39.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.708+0000 7f6f77fff640 1 -- 172.21.15.70:0/1955247713 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6f880b8050 con 0x5603316a82b0 2024-04-01T01:46:39.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.708+0000 7f6fa1881740 10 client.29306 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:39.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.708+0000 7f6fa1881740 1 -- 172.21.15.70:0/1955247713 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5603313213f0 con 0x5603316a82b0 2024-04-01T01:46:39.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.709+0000 7f6f77fff640 1 -- 172.21.15.70:0/1955247713 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f8809ac60 con 0x5603316a82b0 2024-04-01T01:46:39.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.709+0000 7f6f77fff640 1 -- 172.21.15.70:0/1955247713 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6f88020060 con 0x5603316a82b0 2024-04-01T01:46:39.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.709+0000 7f6fa1881740 10 client.29306 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:39.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.709+0000 7f6fa1881740 10 client.29306 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:39.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.709+0000 7f6fa1881740 20 client.29306 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:39.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.709+0000 7f6fa1881740 1 --2- 172.21.15.70:0/1955247713 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5603316be7e0 0x5603316c0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.709+0000 7f6fa1881740 4 client.29306 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:39.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.709+0000 7f6fa1881740 1 -- 172.21.15.70:0/1955247713 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6f8c01a910 con 0x5603316be7e0 2024-04-01T01:46:39.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.710+0000 7f6f97fff640 1 --2- 172.21.15.70:0/1955247713 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5603316be7e0 0x5603316c0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.710+0000 7f6f97fff640 1 --2- 172.21.15.70:0/1955247713 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5603316be7e0 0x5603316c0bd0 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9eb6a889d90dc43c server_cookie=453d7c987a895b34 in_seq=0 out_seq=0 2024-04-01T01:46:39.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.710+0000 7f6f77fff640 10 client.29306 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:39.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.711+0000 7f6f77fff640 1 -- 172.21.15.70:0/1955247713 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f6f8c01a910 con 0x5603316be7e0 2024-04-01T01:46:39.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.711+0000 7f6f77fff640 10 client.29306 handle_command_reply: tid=0 2024-04-01T01:46:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.711+0000 7f6fa1881740 1 client.29306 shutdown 2024-04-01T01:46:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.712+0000 7f6fa1881740 1 -- 172.21.15.70:0/1955247713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 msgr2=0x5603316b27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.712+0000 7f6fa1881740 1 --2- 172.21.15.70:0/1955247713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316b27b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x5603316a9710 tx=0x7f6f8809ab40 comp rx=0 tx=0).stop 2024-04-01T01:46:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.712+0000 7f6fa1881740 1 -- 172.21.15.70:0/1955247713 shutdown_connections 2024-04-01T01:46:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.712+0000 7f6fa1881740 1 --2- 172.21.15.70:0/1955247713 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5603316be7e0 0x5603316c0bd0 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.712+0000 7f6fa1881740 1 --2- 172.21.15.70:0/1955247713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316ade10 0x5603316b9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.712+0000 7f6fa1881740 1 --2- 172.21.15.70:0/1955247713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 0x5603316b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.712+0000 7f6fa1881740 1 --2- 172.21.15.70:0/1955247713 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316b27b0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.712+0000 7f6fa1881740 1 -- 172.21.15.70:0/1955247713 >> 172.21.15.70:0/1955247713 conn(0x5603316a3820 msgr2=0x5603316a2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.712+0000 7f6f77fff640 0 client.29306 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.712+0000 7f6fa1881740 1 -- 172.21.15.70:0/1955247713 shutdown_connections 2024-04-01T01:46:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.712+0000 7f6fa1881740 1 -- 172.21.15.70:0/1955247713 wait complete. 2024-04-01T01:46:39.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.713+0000 7f6fa1881740 20 client.29306 trim_cache size 0 max 16384 2024-04-01T01:46:39.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.748+0000 7f6fa1881740 1 Processor -- start 2024-04-01T01:46:39.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.749+0000 7f6fa1881740 1 -- start start 2024-04-01T01:46:39.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.749+0000 7f6fa1881740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316a3820 0x5603316b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.749+0000 7f6fa1881740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316a2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.749+0000 7f6fa1881740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 0x5603316a28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.749+0000 7f6f97fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.749+0000 7f6f97fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316a2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42760/0 (socket says 172.21.15.70:42760) 2024-04-01T01:46:39.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.749+0000 7f6f9ca39640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316a3820 0x5603316b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.749+0000 7f6f9ca39640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316a3820 0x5603316b2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57680/0 (socket says 172.21.15.70:57680) 2024-04-01T01:46:39.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.749+0000 7f6f97fff640 1 -- 172.21.15.70:0/676174922 learned_addr learned my addr 172.21.15.70:0/676174922 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:39.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.749+0000 7f6fa1881740 1 -- 172.21.15.70:0/676174922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6f98083fa0 con 0x5603316a8ca0 2024-04-01T01:46:39.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.749+0000 7f6fa1881740 1 -- 172.21.15.70:0/676174922 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f981d4670 con 0x5603316a82b0 2024-04-01T01:46:39.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.749+0000 7f6fa1881740 1 -- 172.21.15.70:0/676174922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6f98084670 con 0x5603316a3820 2024-04-01T01:46:39.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.750+0000 7f6f9d23a640 1 --2- 172.21.15.70:0/676174922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 0x5603316a28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.750+0000 7f6f97fff640 1 -- 172.21.15.70:0/676174922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316a3820 msgr2=0x5603316b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:39.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.750+0000 7f6f97fff640 1 --2- 172.21.15.70:0/676174922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316a3820 0x5603316b2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.750+0000 7f6f97fff640 1 -- 172.21.15.70:0/676174922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 msgr2=0x5603316a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:39.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.750+0000 7f6f97fff640 1 --2- 172.21.15.70:0/676174922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 0x5603316a28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.750+0000 7f6f97fff640 1 -- 172.21.15.70:0/676174922 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603312e10f0 con 0x5603316a82b0 2024-04-01T01:46:39.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.751+0000 7f6f97fff640 1 --2- 172.21.15.70:0/676174922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316a2390 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x5603316c1200 tx=0x7f6f8c00b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a4da23f5478c815 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:39.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.752+0000 7f6f7e7fc640 1 -- 172.21.15.70:0/676174922 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c015040 con 0x5603316a82b0 2024-04-01T01:46:39.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.752+0000 7f6f7e7fc640 1 -- 172.21.15.70:0/676174922 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f8c00b3a0 con 0x5603316a82b0 2024-04-01T01:46:39.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.752+0000 7f6f7e7fc640 1 -- 172.21.15.70:0/676174922 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c005280 con 0x5603316a82b0 2024-04-01T01:46:39.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.753+0000 7f6fa1881740 1 -- 172.21.15.70:0/676174922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 msgr2=0x5603316a2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:39.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.753+0000 7f6fa1881740 1 --2- 172.21.15.70:0/676174922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316a2390 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x5603316c1200 tx=0x7f6f8c00b820 comp rx=0 tx=0).stop 2024-04-01T01:46:39.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.754+0000 7f6fa1881740 1 -- 172.21.15.70:0/676174922 shutdown_connections 2024-04-01T01:46:39.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.754+0000 7f6fa1881740 1 --2- 172.21.15.70:0/676174922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 0x5603316a28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.754+0000 7f6fa1881740 1 --2- 172.21.15.70:0/676174922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316a2390 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.754+0000 7f6fa1881740 1 --2- 172.21.15.70:0/676174922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316a3820 0x5603316b2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.754+0000 7f6fa1881740 1 -- 172.21.15.70:0/676174922 >> 172.21.15.70:0/676174922 conn(0x5603316a3f70 msgr2=0x5603316b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:39.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.754+0000 7f6fa1881740 1 -- 172.21.15.70:0/676174922 shutdown_connections 2024-04-01T01:46:39.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.754+0000 7f6fa1881740 1 -- 172.21.15.70:0/676174922 wait complete. 2024-04-01T01:46:39.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.755+0000 7f6fa1881740 1 -- start start 2024-04-01T01:46:39.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.755+0000 7f6fa1881740 1 Processor -- start 2024-04-01T01:46:39.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.755+0000 7f6fa1881740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316a3820 0x5603316b93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.755+0000 7f6fa1881740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316b9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.755+0000 7f6fa1881740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 0x5603316b7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.755+0000 7f6fa1881740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6f981d3fa0 con 0x5603316a8ca0 2024-04-01T01:46:39.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.756+0000 7f6fa1881740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f981d38d0 con 0x5603316a82b0 2024-04-01T01:46:39.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.756+0000 7f6fa1881740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6f98081fa0 con 0x5603316a3820 2024-04-01T01:46:39.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.756+0000 7f6fa1881740 10 client.0 _ll_register_callbacks cb 0x7f6f88004570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:39.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.756+0000 7f6f97fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316b9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.756+0000 7f6f9ca39640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316a3820 0x5603316b93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.756+0000 7f6f9d23a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 0x5603316b7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.756+0000 7f6f97fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316b9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42774/0 (socket says 172.21.15.70:42774) 2024-04-01T01:46:39.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.756+0000 7f6f97fff640 1 -- 172.21.15.70:0/3144188828 learned_addr learned my addr 172.21.15.70:0/3144188828 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:39.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.756+0000 7f6f9d23a640 1 -- 172.21.15.70:0/3144188828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316a3820 msgr2=0x5603316b93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:39.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.756+0000 7f6f9d23a640 1 --2- 172.21.15.70:0/3144188828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316a3820 0x5603316b93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.756+0000 7f6f9d23a640 1 -- 172.21.15.70:0/3144188828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 msgr2=0x5603316b9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:39.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.756+0000 7f6f9d23a640 1 --2- 172.21.15.70:0/3144188828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316b9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.756+0000 7f6f9d23a640 1 -- 172.21.15.70:0/3144188828 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603316b8290 con 0x5603316a8ca0 2024-04-01T01:46:39.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.756+0000 7f6f97fff640 1 --2- 172.21.15.70:0/3144188828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316b9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:46:39.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.756+0000 7f6f9d23a640 1 --2- 172.21.15.70:0/3144188828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 0x5603316b7a60 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f6f90037040 tx=0x7f6f9002c120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d1658b177da2ff3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:39.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.756+0000 7f6f77fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:46:39.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.757+0000 7f6f77fff640 1 -- 172.21.15.70:0/3144188828 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f90046050 con 0x5603316a8ca0 2024-04-01T01:46:39.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.757+0000 7f6f77fff640 1 -- 172.21.15.70:0/3144188828 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6f90004040 con 0x5603316a8ca0 2024-04-01T01:46:39.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.757+0000 7f6fa1881740 1 -- 172.21.15.70:0/3144188828 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f8c00b3a0 con 0x5603316a8ca0 2024-04-01T01:46:39.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.757+0000 7f6f77fff640 1 -- 172.21.15.70:0/3144188828 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6f90039040 con 0x5603316a8ca0 2024-04-01T01:46:39.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.757+0000 7f6fa1881740 10 client.19245 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:39.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.757+0000 7f6fa1881740 1 -- 172.21.15.70:0/3144188828 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560331725430 con 0x5603316a8ca0 2024-04-01T01:46:39.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.758+0000 7f6f77fff640 1 -- 172.21.15.70:0/3144188828 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f8c00b3a0 con 0x5603316a8ca0 2024-04-01T01:46:39.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.758+0000 7f6f77fff640 1 -- 172.21.15.70:0/3144188828 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6f9003fad0 con 0x5603316a8ca0 2024-04-01T01:46:39.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.758+0000 7f6fa1881740 10 client.19245 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:39.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.758+0000 7f6fa1881740 10 client.19245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:39.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.758+0000 7f6fa1881740 20 client.19245 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:39.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.758+0000 7f6fa1881740 1 --2- 172.21.15.70:0/3144188828 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5603316ade10 0x560331727e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.758+0000 7f6fa1881740 4 client.19245 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:46:39.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.758+0000 7f6fa1881740 1 -- 172.21.15.70:0/3144188828 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56033161a6c0 con 0x5603316ade10 2024-04-01T01:46:39.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.759+0000 7f6f9ca39640 1 --2- 172.21.15.70:0/3144188828 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5603316ade10 0x560331727e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.764+0000 7f6f77fff640 10 client.19245 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:39.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.764+0000 7f6f9ca39640 1 --2- 172.21.15.70:0/3144188828 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5603316ade10 0x560331727e50 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25df55543b39506d server_cookie=584538dc701ca661 in_seq=0 out_seq=0 2024-04-01T01:46:39.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.766+0000 7f6f77fff640 1 -- 172.21.15.70:0/3144188828 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56033161a6c0 con 0x5603316ade10 2024-04-01T01:46:39.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.766+0000 7f6f77fff640 10 client.19245 handle_command_reply: tid=0 2024-04-01T01:46:39.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.766+0000 7f6fa1881740 1 client.19245 shutdown 2024-04-01T01:46:39.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.766+0000 7f6fa1881740 1 -- 172.21.15.70:0/3144188828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 msgr2=0x5603316b7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:39.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.766+0000 7f6fa1881740 1 --2- 172.21.15.70:0/3144188828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 0x5603316b7a60 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f6f90037040 tx=0x7f6f9002c120 comp rx=0 tx=0).stop 2024-04-01T01:46:39.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.767+0000 7f6fa1881740 1 -- 172.21.15.70:0/3144188828 shutdown_connections 2024-04-01T01:46:39.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.767+0000 7f6fa1881740 1 --2- 172.21.15.70:0/3144188828 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5603316ade10 0x560331727e50 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.767+0000 7f6fa1881740 1 --2- 172.21.15.70:0/3144188828 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603316a8ca0 0x5603316b7a60 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.767+0000 7f6fa1881740 1 --2- 172.21.15.70:0/3144188828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5603316a82b0 0x5603316b9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.767+0000 7f6fa1881740 1 --2- 172.21.15.70:0/3144188828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5603316a3820 0x5603316b93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.767+0000 7f6fa1881740 1 -- 172.21.15.70:0/3144188828 >> 172.21.15.70:0/3144188828 conn(0x5603316a3f70 msgr2=0x5603316a2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:39.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.767+0000 7f6f77fff640 0 client.19245 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:39.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.767+0000 7f6fa1881740 1 -- 172.21.15.70:0/3144188828 shutdown_connections 2024-04-01T01:46:39.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.767+0000 7f6fa1881740 1 -- 172.21.15.70:0/3144188828 wait complete. 2024-04-01T01:46:39.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.768+0000 7f6fa1881740 20 client.19245 trim_cache size 0 max 16384 2024-04-01T01:46:39.768 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:46:39.768 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:46:39.768 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:46:39.769 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:46:39.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.768+0000 7f6f9ec9d640 1 -- 172.21.15.70:0/2050083510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f800a2550 msgr2=0x7f6f800a4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:39.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.768+0000 7f6f9ec9d640 1 --2- 172.21.15.70:0/2050083510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f800a2550 0x7f6f800a4a10 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6f90006010 tx=0x7f6f90005ef0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.768+0000 7f6f9ec9d640 1 -- 172.21.15.70:0/2050083510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f981089c0 msgr2=0x7f6f98119680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:39.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.768+0000 7f6f9ec9d640 1 --2- 172.21.15.70:0/2050083510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f981089c0 0x7f6f98119680 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6f8800c890 tx=0x7f6f8800cd50 comp rx=0 tx=0).stop 2024-04-01T01:46:39.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.769+0000 7f6f9ec9d640 1 -- 172.21.15.70:0/2050083510 shutdown_connections 2024-04-01T01:46:39.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.769+0000 7f6f9ec9d640 1 --2- 172.21.15.70:0/2050083510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f9810de00 0x7f6f98117cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.769+0000 7f6f9ec9d640 1 --2- 172.21.15.70:0/2050083510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f9810b380 0x7f6f98119bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.769+0000 7f6f9ec9d640 1 --2- 172.21.15.70:0/2050083510 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f800a2550 0x7f6f800a4a10 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.769+0000 7f6f9ec9d640 1 --2- 172.21.15.70:0/2050083510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f981089c0 0x7f6f98119680 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.769+0000 7f6f9ec9d640 1 -- 172.21.15.70:0/2050083510 >> 172.21.15.70:0/2050083510 conn(0x7f6f9806e950 msgr2=0x7f6f981093c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:39.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.769+0000 7f6f9ec9d640 1 -- 172.21.15.70:0/2050083510 shutdown_connections 2024-04-01T01:46:39.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.769+0000 7f6f9ec9d640 1 -- 172.21.15.70:0/2050083510 wait complete. 2024-04-01T01:46:39.788 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:46:39.789 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:46:39.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.982+0000 7f7909fe2640 1 -- 172.21.15.70:0/843673963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f790410da90 msgr2=0x7f790410df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:39.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.982+0000 7f7909fe2640 1 --2- 172.21.15.70:0/843673963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f790410da90 0x7f790410df40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f78f8009860 tx=0x7f78f802bb90 comp rx=0 tx=0).stop 2024-04-01T01:46:39.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.983+0000 7f7909fe2640 1 -- 172.21.15.70:0/843673963 shutdown_connections 2024-04-01T01:46:39.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.983+0000 7f7909fe2640 1 --2- 172.21.15.70:0/843673963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f790410da90 0x7f790410df40 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.983+0000 7f7909fe2640 1 --2- 172.21.15.70:0/843673963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f790410b0c0 0x7f790410d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.983+0000 7f7909fe2640 1 --2- 172.21.15.70:0/843673963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7904108750 0x7f790410ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.983+0000 7f7909fe2640 1 -- 172.21.15.70:0/843673963 >> 172.21.15.70:0/843673963 conn(0x7f790406e5c0 msgr2=0x7f7904070a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:39.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.983+0000 7f7909fe2640 1 -- 172.21.15.70:0/843673963 shutdown_connections 2024-04-01T01:46:39.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.983+0000 7f7909fe2640 1 -- 172.21.15.70:0/843673963 wait complete. 2024-04-01T01:46:39.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.984+0000 7f7909fe2640 1 Processor -- start 2024-04-01T01:46:39.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.984+0000 7f7909fe2640 1 -- start start 2024-04-01T01:46:39.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.984+0000 7f7909fe2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7904108750 0x7f790419df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.984+0000 7f7909fe2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f790410b0c0 0x7f790419e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.984+0000 7f7909fe2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f790410da90 0x7f79041a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.984+0000 7f7909fe2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f79041118b0 con 0x7f790410b0c0 2024-04-01T01:46:39.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.984+0000 7f7909fe2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7904111730 con 0x7f790410da90 2024-04-01T01:46:39.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.984+0000 7f7909fe2640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7904111a30 con 0x7f7904108750 2024-04-01T01:46:39.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.985+0000 7f79037fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7904108750 0x7f790419df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.985+0000 7f79037fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7904108750 0x7f790419df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57700/0 (socket says 172.21.15.70:57700) 2024-04-01T01:46:39.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.985+0000 7f79037fe640 1 -- 172.21.15.70:0/3998934384 learned_addr learned my addr 172.21.15.70:0/3998934384 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:39.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.985+0000 7f7903fff640 1 --2- 172.21.15.70:0/3998934384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f790410da90 0x7f79041a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.985+0000 7f79037fe640 1 -- 172.21.15.70:0/3998934384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f790410da90 msgr2=0x7f79041a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:39.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.985+0000 7f79037fe640 1 --2- 172.21.15.70:0/3998934384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f790410da90 0x7f79041a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.985+0000 7f79037fe640 1 -- 172.21.15.70:0/3998934384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f790410b0c0 msgr2=0x7f790419e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:39.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.985+0000 7f79037fe640 1 --2- 172.21.15.70:0/3998934384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f790410b0c0 0x7f790419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:39.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.985+0000 7f79037fe640 1 -- 172.21.15.70:0/3998934384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79041a5b50 con 0x7f7904108750 2024-04-01T01:46:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.986+0000 7f79037fe640 1 --2- 172.21.15.70:0/3998934384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7904108750 0x7f790419df30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f78f401ea00 tx=0x7f78f401eec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.987+0000 7f7900ff9640 1 -- 172.21.15.70:0/3998934384 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f4028020 con 0x7f7904108750 2024-04-01T01:46:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.987+0000 7f7909fe2640 1 -- 172.21.15.70:0/3998934384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79041a5e40 con 0x7f7904108750 2024-04-01T01:46:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.987+0000 7f7909fe2640 1 -- 172.21.15.70:0/3998934384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79041a6350 con 0x7f7904108750 2024-04-01T01:46:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.987+0000 7f7900ff9640 1 -- 172.21.15.70:0/3998934384 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78f401ada0 con 0x7f7904108750 2024-04-01T01:46:39.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.987+0000 7f7900ff9640 1 -- 172.21.15.70:0/3998934384 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f4003cc0 con 0x7f7904108750 2024-04-01T01:46:39.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.988+0000 7f7909fe2640 1 -- 172.21.15.70:0/3998934384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f00052c0 con 0x7f7904108750 2024-04-01T01:46:39.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.990+0000 7f7900ff9640 1 -- 172.21.15.70:0/3998934384 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f78f402b020 con 0x7f7904108750 2024-04-01T01:46:39.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.991+0000 7f7900ff9640 1 --2- 172.21.15.70:0/3998934384 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f78ec0a2520 0x7f78ec0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:39.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.991+0000 7f7900ff9640 1 -- 172.21.15.70:0/3998934384 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f78f40dfb20 con 0x7f7904108750 2024-04-01T01:46:39.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.992+0000 7f7902ffd640 1 --2- 172.21.15.70:0/3998934384 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f78ec0a2520 0x7f78ec0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:39.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.994+0000 7f7902ffd640 1 --2- 172.21.15.70:0/3998934384 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f78ec0a2520 0x7f78ec0a49e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f790419f440 tx=0x7f78fc009710 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:39.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:39.998+0000 7f7900ff9640 1 -- 172.21.15.70:0/3998934384 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f78f4032020 con 0x7f7904108750 2024-04-01T01:46:40.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.424+0000 7f7909fe2640 1 -- 172.21.15.70:0/3998934384 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78f00055b0 con 0x7f7904108750 2024-04-01T01:46:40.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.425+0000 7f7900ff9640 1 -- 172.21.15.70:0/3998934384 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7f78f40ac8b0 con 0x7f7904108750 2024-04-01T01:46:40.425 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:46:40.425 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:46:40.426 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:46:40.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.428+0000 7f7909fe2640 1 -- 172.21.15.70:0/3998934384 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f78ec0a2520 msgr2=0x7f78ec0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.428+0000 7f7909fe2640 1 --2- 172.21.15.70:0/3998934384 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f78ec0a2520 0x7f78ec0a49e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f790419f440 tx=0x7f78fc009710 comp rx=0 tx=0).stop 2024-04-01T01:46:40.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.428+0000 7f7909fe2640 1 -- 172.21.15.70:0/3998934384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7904108750 msgr2=0x7f790419df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.428+0000 7f7909fe2640 1 --2- 172.21.15.70:0/3998934384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7904108750 0x7f790419df30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f78f401ea00 tx=0x7f78f401eec0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.428+0000 7f7909fe2640 1 -- 172.21.15.70:0/3998934384 shutdown_connections 2024-04-01T01:46:40.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.428+0000 7f7909fe2640 1 --2- 172.21.15.70:0/3998934384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f790410da90 0x7f79041a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.428+0000 7f7909fe2640 1 --2- 172.21.15.70:0/3998934384 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f790410b0c0 0x7f790419e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.428+0000 7f7909fe2640 1 --2- 172.21.15.70:0/3998934384 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f78ec0a2520 0x7f78ec0a49e0 secure :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f790419f440 tx=0x7f78fc009710 comp rx=0 tx=0).stop 2024-04-01T01:46:40.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.428+0000 7f7909fe2640 1 --2- 172.21.15.70:0/3998934384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7904108750 0x7f790419df30 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.428+0000 7f7909fe2640 1 -- 172.21.15.70:0/3998934384 >> 172.21.15.70:0/3998934384 conn(0x7f790406e5c0 msgr2=0x7f790410b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:40.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.429+0000 7f7909fe2640 1 -- 172.21.15.70:0/3998934384 shutdown_connections 2024-04-01T01:46:40.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.429+0000 7f7909fe2640 1 -- 172.21.15.70:0/3998934384 wait complete. 2024-04-01T01:46:40.447 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:46:40.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.631+0000 7f1d41d80640 1 -- 172.21.15.70:0/3121942180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d34094c80 msgr2=0x7f1d340950a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.631+0000 7f1d41d80640 1 --2- 172.21.15.70:0/3121942180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d34094c80 0x7f1d340950a0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009860 tx=0x7f1d2c02bb70 comp rx=0 tx=0).stop 2024-04-01T01:46:40.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.631+0000 7f1d41d80640 1 -- 172.21.15.70:0/3121942180 shutdown_connections 2024-04-01T01:46:40.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.631+0000 7f1d41d80640 1 --2- 172.21.15.70:0/3121942180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d34096830 0x7f1d3409d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.631+0000 7f1d41d80640 1 --2- 172.21.15.70:0/3121942180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d34095e70 0x7f1d340962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.631+0000 7f1d41d80640 1 --2- 172.21.15.70:0/3121942180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d34094c80 0x7f1d340950a0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.631+0000 7f1d41d80640 1 -- 172.21.15.70:0/3121942180 >> 172.21.15.70:0/3121942180 conn(0x7f1d340901d0 msgr2=0x7f1d34092630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:40.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.631+0000 7f1d41d80640 1 -- 172.21.15.70:0/3121942180 shutdown_connections 2024-04-01T01:46:40.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.631+0000 7f1d41d80640 1 -- 172.21.15.70:0/3121942180 wait complete. 2024-04-01T01:46:40.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.632+0000 7f1d41d80640 1 Processor -- start 2024-04-01T01:46:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.632+0000 7f1d41d80640 1 -- start start 2024-04-01T01:46:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.632+0000 7f1d41d80640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d34094c80 0x7f1d340052b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.632+0000 7f1d41d80640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d34095e70 0x7f1d34130d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.632+0000 7f1d41d80640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d34096830 0x7f1d34133330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.632+0000 7f1d41d80640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1d340a0780 con 0x7f1d34094c80 2024-04-01T01:46:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.632+0000 7f1d41d80640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1d340a0600 con 0x7f1d34096830 2024-04-01T01:46:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.632+0000 7f1d41d80640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1d340a0900 con 0x7f1d34095e70 2024-04-01T01:46:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.632+0000 7f1d40d7e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d34094c80 0x7f1d340052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.632+0000 7f1d3bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d34095e70 0x7f1d34130d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.632+0000 7f1d3bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d34095e70 0x7f1d34130d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57720/0 (socket says 172.21.15.70:57720) 2024-04-01T01:46:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.632+0000 7f1d3bfff640 1 -- 172.21.15.70:0/605491124 learned_addr learned my addr 172.21.15.70:0/605491124 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.632+0000 7f1d4157f640 1 --2- 172.21.15.70:0/605491124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d34096830 0x7f1d34133330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.633+0000 7f1d40d7e640 1 -- 172.21.15.70:0/605491124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d34095e70 msgr2=0x7f1d34130d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.633+0000 7f1d40d7e640 1 --2- 172.21.15.70:0/605491124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d34095e70 0x7f1d34130d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.633+0000 7f1d40d7e640 1 -- 172.21.15.70:0/605491124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d34096830 msgr2=0x7f1d34133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.633+0000 7f1d40d7e640 1 --2- 172.21.15.70:0/605491124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d34096830 0x7f1d34133330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.633+0000 7f1d40d7e640 1 -- 172.21.15.70:0/605491124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d34133c40 con 0x7f1d34094c80 2024-04-01T01:46:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.633+0000 7f1d4157f640 1 --2- 172.21.15.70:0/605491124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d34096830 0x7f1d34133330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:46:40.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.633+0000 7f1d40d7e640 1 --2- 172.21.15.70:0/605491124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d34094c80 0x7f1d340052b0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c037040 tx=0x7f1d2c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:40.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.634+0000 7f1d39ffb640 1 -- 172.21.15.70:0/605491124 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c041020 con 0x7f1d34094c80 2024-04-01T01:46:40.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.634+0000 7f1d39ffb640 1 -- 172.21.15.70:0/605491124 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d2c0035f0 con 0x7f1d34094c80 2024-04-01T01:46:40.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.634+0000 7f1d39ffb640 1 -- 172.21.15.70:0/605491124 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c039040 con 0x7f1d34094c80 2024-04-01T01:46:40.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.634+0000 7f1d41d80640 1 -- 172.21.15.70:0/605491124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d3412aae0 con 0x7f1d34094c80 2024-04-01T01:46:40.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.634+0000 7f1d41d80640 1 -- 172.21.15.70:0/605491124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d3412af80 con 0x7f1d34094c80 2024-04-01T01:46:40.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.638+0000 7f1d45bbf740 1 Processor -- start 2024-04-01T01:46:40.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.639+0000 7f1d45bbf740 1 -- start start 2024-04-01T01:46:40.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.639+0000 7f1d45bbf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa466a0 0x5578efa46ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.639+0000 7f1d45bbf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa47090 0x5578efa4bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.639+0000 7f1d45bbf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa4c200 0x5578efa4e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.639+0000 7f1d45bbf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5578ef9ab1b0 con 0x5578efa47090 2024-04-01T01:46:40.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.639+0000 7f1d45bbf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5578ef953c00 con 0x5578efa466a0 2024-04-01T01:46:40.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.639+0000 7f1d45bbf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5578ef9a7250 con 0x5578efa4c200 2024-04-01T01:46:40.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.639+0000 7f1d3bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa4c200 0x5578efa4e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.639+0000 7f1d3bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa4c200 0x5578efa4e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57734/0 (socket says 172.21.15.70:57734) 2024-04-01T01:46:40.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.639+0000 7f1d3bfff640 1 -- 172.21.15.70:0/2424985422 learned_addr learned my addr 172.21.15.70:0/2424985422 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:40.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.640+0000 7f1d3bfff640 1 -- 172.21.15.70:0/2424985422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa466a0 msgr2=0x5578efa46ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:40.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.640+0000 7f1d3bfff640 1 --2- 172.21.15.70:0/2424985422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa466a0 0x5578efa46ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.640+0000 7f1d3bfff640 1 -- 172.21.15.70:0/2424985422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa47090 msgr2=0x5578efa4bb90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:46:40.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.640+0000 7f1d3bfff640 1 --2- 172.21.15.70:0/2424985422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa47090 0x5578efa4bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.640+0000 7f1d3bfff640 1 -- 172.21.15.70:0/2424985422 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578ef9733d0 con 0x5578efa4c200 2024-04-01T01:46:40.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.640+0000 7f1d39ffb640 1 -- 172.21.15.70:0/605491124 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f1d2c044020 con 0x7f1d34094c80 2024-04-01T01:46:40.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.640+0000 7f1d3bfff640 1 --2- 172.21.15.70:0/2424985422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa4c200 0x5578efa4e690 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1d3000bd60 tx=0x7f1d300107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c370edfa486f113a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:40.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.640+0000 7f1d22ffd640 1 -- 172.21.15.70:0/2424985422 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d30011640 con 0x5578efa4c200 2024-04-01T01:46:40.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.640+0000 7f1d22ffd640 1 -- 172.21.15.70:0/2424985422 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d30011c90 con 0x5578efa4c200 2024-04-01T01:46:40.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.640+0000 7f1d22ffd640 1 -- 172.21.15.70:0/2424985422 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3001bc40 con 0x5578efa4c200 2024-04-01T01:46:40.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.641+0000 7f1d39ffb640 1 --2- 172.21.15.70:0/605491124 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1d240a25a0 0x7f1d240a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.641+0000 7f1d39ffb640 1 -- 172.21.15.70:0/605491124 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f1d2c0f04a0 con 0x7f1d34094c80 2024-04-01T01:46:40.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.641+0000 7f1d4157f640 1 --2- 172.21.15.70:0/605491124 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1d240a25a0 0x7f1d240a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.641+0000 7f1d45bbf740 1 -- 172.21.15.70:0/2424985422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa4c200 msgr2=0x5578efa4e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.641+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/2424985422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa4c200 0x5578efa4e690 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1d3000bd60 tx=0x7f1d300107a0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.642+0000 7f1d45bbf740 1 -- 172.21.15.70:0/2424985422 shutdown_connections 2024-04-01T01:46:40.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.642+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/2424985422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa4c200 0x5578efa4e690 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.642+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/2424985422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa47090 0x5578efa4bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.642+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/2424985422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa466a0 0x5578efa46ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.642+0000 7f1d45bbf740 1 -- 172.21.15.70:0/2424985422 >> 172.21.15.70:0/2424985422 conn(0x5578efa41c10 msgr2=0x5578efa42010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.642+0000 7f1d4157f640 1 --2- 172.21.15.70:0/605491124 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1d240a25a0 0x7f1d240a4a60 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c067290 tx=0x7f1d3c077040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.642+0000 7f1d45bbf740 1 -- 172.21.15.70:0/2424985422 shutdown_connections 2024-04-01T01:46:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.642+0000 7f1d45bbf740 1 -- 172.21.15.70:0/2424985422 wait complete. 2024-04-01T01:46:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.642+0000 7f1d45bbf740 1 -- start start 2024-04-01T01:46:40.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.642+0000 7f1d45bbf740 1 Processor -- start 2024-04-01T01:46:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.643+0000 7f1d45bbf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa466a0 0x5578efa50c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.643+0000 7f1d45bbf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa47090 0x5578efa511a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.643+0000 7f1d45bbf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa4c200 0x5578efa581a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.643+0000 7f1d45bbf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5578efa4f1b0 con 0x5578efa466a0 2024-04-01T01:46:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.643+0000 7f1d45bbf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5578ef949470 con 0x5578efa4c200 2024-04-01T01:46:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.643+0000 7f1d45bbf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5578efa4f330 con 0x5578efa47090 2024-04-01T01:46:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.643+0000 7f1d45bbf740 10 client.0 _ll_register_callbacks cb 0x5578ef966cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.644+0000 7f1d3bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa47090 0x5578efa511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.644+0000 7f1d3bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa47090 0x5578efa511a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57746/0 (socket says 172.21.15.70:57746) 2024-04-01T01:46:40.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.644+0000 7f1d3bfff640 1 -- 172.21.15.70:0/2522007738 learned_addr learned my addr 172.21.15.70:0/2522007738 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:40.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.644+0000 7f1d40d7e640 1 --2- 172.21.15.70:0/2522007738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa466a0 0x5578efa50c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.644+0000 7f1d4157f640 1 --2- 172.21.15.70:0/2522007738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa4c200 0x5578efa581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.644+0000 7f1d40d7e640 1 -- 172.21.15.70:0/2522007738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa47090 msgr2=0x5578efa511a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:40.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.644+0000 7f1d40d7e640 1 --2- 172.21.15.70:0/2522007738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa47090 0x5578efa511a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.644+0000 7f1d40d7e640 1 -- 172.21.15.70:0/2522007738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa4c200 msgr2=0x5578efa581a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:40.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.644+0000 7f1d40d7e640 1 --2- 172.21.15.70:0/2522007738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa4c200 0x5578efa581a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.644+0000 7f1d40d7e640 1 -- 172.21.15.70:0/2522007738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578ef6bf3f0 con 0x5578efa466a0 2024-04-01T01:46:40.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.645+0000 7f1d40d7e640 1 --2- 172.21.15.70:0/2522007738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa466a0 0x5578efa50c60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c0edb40 tx=0x7f1d2c0bc670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31939d2fde1bb607 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:40.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.645+0000 7f1d1bfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:46:40.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.645+0000 7f1d1bfff640 1 -- 172.21.15.70:0/2522007738 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c0d5060 con 0x5578efa466a0 2024-04-01T01:46:40.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.645+0000 7f1d1bfff640 1 -- 172.21.15.70:0/2522007738 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f1d2c0f0130 con 0x5578efa466a0 2024-04-01T01:46:40.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.645+0000 7f1d45bbf740 1 -- 172.21.15.70:0/2522007738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d30011c90 con 0x5578efa466a0 2024-04-01T01:46:40.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.646+0000 7f1d1bfff640 1 -- 172.21.15.70:0/2522007738 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1d2c0bde60 con 0x5578efa466a0 2024-04-01T01:46:40.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.646+0000 7f1d45bbf740 10 client.19257 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:40.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.646+0000 7f1d45bbf740 1 -- 172.21.15.70:0/2522007738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5578ef7897d0 con 0x5578efa466a0 2024-04-01T01:46:40.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.646+0000 7f1d1bfff640 1 -- 172.21.15.70:0/2522007738 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d30011c90 con 0x5578efa466a0 2024-04-01T01:46:40.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.646+0000 7f1d1bfff640 1 -- 172.21.15.70:0/2522007738 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f1d2c03c030 con 0x5578efa466a0 2024-04-01T01:46:40.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.646+0000 7f1d45bbf740 10 client.19257 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:40.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.646+0000 7f1d45bbf740 10 client.19257 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:46:40.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.646+0000 7f1d45bbf740 20 client.19257 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:40.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.646+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/2522007738 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5578efa6d5b0 0x5578efa6fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.647+0000 7f1d45bbf740 4 client.19257 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:40.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.647+0000 7f1d45bbf740 1 -- 172.21.15.70:0/2522007738 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578ef9b86f0 con 0x5578efa6d5b0 2024-04-01T01:46:40.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.647+0000 7f1d3bfff640 1 --2- 172.21.15.70:0/2522007738 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5578efa6d5b0 0x5578efa6fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.648+0000 7f1d3bfff640 1 --2- 172.21.15.70:0/2522007738 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5578efa6d5b0 0x5578efa6fa50 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5f97370382de5bcf server_cookie=f8a2416d6665c353 in_seq=0 out_seq=0 2024-04-01T01:46:40.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.648+0000 7f1d1bfff640 10 client.19257 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:46:40.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.654+0000 7f1d1bfff640 1 -- 172.21.15.70:0/2522007738 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5578ef9b86f0 con 0x5578efa6d5b0 2024-04-01T01:46:40.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.654+0000 7f1d1bfff640 10 client.19257 handle_command_reply: tid=0 2024-04-01T01:46:40.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.654+0000 7f1d45bbf740 1 client.19257 shutdown 2024-04-01T01:46:40.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.654+0000 7f1d45bbf740 1 -- 172.21.15.70:0/2522007738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa466a0 msgr2=0x5578efa50c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.654+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/2522007738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa466a0 0x5578efa50c60 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c0edb40 tx=0x7f1d2c0bc670 comp rx=0 tx=0).stop 2024-04-01T01:46:40.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.655+0000 7f1d45bbf740 1 -- 172.21.15.70:0/2522007738 shutdown_connections 2024-04-01T01:46:40.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.655+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/2522007738 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5578efa6d5b0 0x5578efa6fa50 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.655+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/2522007738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa4c200 0x5578efa581a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.655+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/2522007738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa47090 0x5578efa511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.655+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/2522007738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa466a0 0x5578efa50c60 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.655+0000 7f1d45bbf740 1 -- 172.21.15.70:0/2522007738 >> 172.21.15.70:0/2522007738 conn(0x5578efa41c10 msgr2=0x5578efa40f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:40.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.655+0000 7f1d1bfff640 0 client.19257 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:46:40.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.655+0000 7f1d45bbf740 1 -- 172.21.15.70:0/2522007738 shutdown_connections 2024-04-01T01:46:40.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.655+0000 7f1d45bbf740 1 -- 172.21.15.70:0/2522007738 wait complete. 2024-04-01T01:46:40.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.655+0000 7f1d45bbf740 20 client.19257 trim_cache size 0 max 16384 2024-04-01T01:46:40.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.679+0000 7f1d45bbf740 1 Processor -- start 2024-04-01T01:46:40.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.679+0000 7f1d45bbf740 1 -- start start 2024-04-01T01:46:40.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.679+0000 7f1d45bbf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa466a0 0x5578efa50c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.679+0000 7f1d45bbf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa47090 0x5578efa58160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.679+0000 7f1d45bbf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa4c200 0x5578efa51140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.679+0000 7f1d3bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa47090 0x5578efa58160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.680+0000 7f1d3bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa47090 0x5578efa58160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57754/0 (socket says 172.21.15.70:57754) 2024-04-01T01:46:40.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.680+0000 7f1d3bfff640 1 -- 172.21.15.70:0/344928168 learned_addr learned my addr 172.21.15.70:0/344928168 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:40.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.680+0000 7f1d45bbf740 1 -- 172.21.15.70:0/344928168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3413c780 con 0x5578efa4c200 2024-04-01T01:46:40.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.680+0000 7f1d45bbf740 1 -- 172.21.15.70:0/344928168 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3413e2c0 con 0x5578efa466a0 2024-04-01T01:46:40.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.680+0000 7f1d45bbf740 1 -- 172.21.15.70:0/344928168 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3413ce50 con 0x5578efa47090 2024-04-01T01:46:40.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.680+0000 7f1d4157f640 1 --2- 172.21.15.70:0/344928168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa4c200 0x5578efa51140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.680+0000 7f1d40d7e640 1 --2- 172.21.15.70:0/344928168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa466a0 0x5578efa50c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.681+0000 7f1d4157f640 1 -- 172.21.15.70:0/344928168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa47090 msgr2=0x5578efa58160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.681+0000 7f1d4157f640 1 --2- 172.21.15.70:0/344928168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa47090 0x5578efa58160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.681+0000 7f1d4157f640 1 -- 172.21.15.70:0/344928168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa466a0 msgr2=0x5578efa50c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.681+0000 7f1d4157f640 1 --2- 172.21.15.70:0/344928168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa466a0 0x5578efa50c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.681+0000 7f1d4157f640 1 -- 172.21.15.70:0/344928168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578efa42b80 con 0x5578efa4c200 2024-04-01T01:46:40.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.682+0000 7f1d4157f640 1 --2- 172.21.15.70:0/344928168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa4c200 0x5578efa51140 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c067010 tx=0x7f1d3c06b050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581c44d56f668731 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:40.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.684+0000 7f1d227fc640 1 -- 172.21.15.70:0/344928168 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c069930 con 0x5578efa4c200 2024-04-01T01:46:40.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.684+0000 7f1d227fc640 1 -- 172.21.15.70:0/344928168 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d3c08a8a0 con 0x5578efa4c200 2024-04-01T01:46:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.684+0000 7f1d227fc640 1 -- 172.21.15.70:0/344928168 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c0927a0 con 0x5578efa4c200 2024-04-01T01:46:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.685+0000 7f1d45bbf740 1 -- 172.21.15.70:0/344928168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa4c200 msgr2=0x5578efa51140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.685+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/344928168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa4c200 0x5578efa51140 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c067010 tx=0x7f1d3c06b050 comp rx=0 tx=0).stop 2024-04-01T01:46:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.685+0000 7f1d45bbf740 1 -- 172.21.15.70:0/344928168 shutdown_connections 2024-04-01T01:46:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.685+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/344928168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa4c200 0x5578efa51140 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.685+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/344928168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa47090 0x5578efa58160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.685+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/344928168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa466a0 0x5578efa50c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.685+0000 7f1d45bbf740 1 -- 172.21.15.70:0/344928168 >> 172.21.15.70:0/344928168 conn(0x5578efa41c10 msgr2=0x5578efa48590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:40.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.685+0000 7f1d45bbf740 1 -- 172.21.15.70:0/344928168 shutdown_connections 2024-04-01T01:46:40.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.685+0000 7f1d45bbf740 1 -- 172.21.15.70:0/344928168 wait complete. 2024-04-01T01:46:40.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.686+0000 7f1d45bbf740 1 -- start start 2024-04-01T01:46:40.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.686+0000 7f1d45bbf740 1 Processor -- start 2024-04-01T01:46:40.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.689+0000 7f1d45bbf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa466a0 0x5578efa55900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.689+0000 7f1d45bbf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa47090 0x5578efa55e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.689+0000 7f1d45bbf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa4c200 0x5578efa56d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.689+0000 7f1d45bbf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3413dbf0 con 0x5578efa4c200 2024-04-01T01:46:40.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.689+0000 7f1d45bbf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3413d520 con 0x5578efa466a0 2024-04-01T01:46:40.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.689+0000 7f1d45bbf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3413a760 con 0x5578efa47090 2024-04-01T01:46:40.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.689+0000 7f1d45bbf740 10 client.0 _ll_register_callbacks cb 0x7f1d2c0b8dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:40.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.691+0000 7f1d40d7e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa466a0 0x5578efa55900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.691+0000 7f1d40d7e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa466a0 0x5578efa55900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42834/0 (socket says 172.21.15.70:42834) 2024-04-01T01:46:40.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.691+0000 7f1d40d7e640 1 -- 172.21.15.70:0/1488020118 learned_addr learned my addr 172.21.15.70:0/1488020118 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:40.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.691+0000 7f1d40d7e640 1 -- 172.21.15.70:0/1488020118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa47090 msgr2=0x5578efa55e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:40.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.691+0000 7f1d40d7e640 1 --2- 172.21.15.70:0/1488020118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa47090 0x5578efa55e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.691+0000 7f1d40d7e640 1 -- 172.21.15.70:0/1488020118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa4c200 msgr2=0x5578efa56d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:40.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.691+0000 7f1d40d7e640 1 --2- 172.21.15.70:0/1488020118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa4c200 0x5578efa56d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.691+0000 7f1d40d7e640 1 -- 172.21.15.70:0/1488020118 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578efa57330 con 0x5578efa466a0 2024-04-01T01:46:40.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.692+0000 7f1d40d7e640 1 --2- 172.21.15.70:0/1488020118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa466a0 0x5578efa55900 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c0db270 tx=0x7f1d2c035f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c52f7a4ccb51da5d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:40.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.692+0000 7f1d1bfff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:40.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.692+0000 7f1d1bfff640 1 -- 172.21.15.70:0/1488020118 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c0c52b0 con 0x5578efa466a0 2024-04-01T01:46:40.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.692+0000 7f1d1bfff640 1 -- 172.21.15.70:0/1488020118 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f1d2c0bbf80 con 0x5578efa466a0 2024-04-01T01:46:40.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.693+0000 7f1d45bbf740 1 -- 172.21.15.70:0/1488020118 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d3c08a8a0 con 0x5578efa466a0 2024-04-01T01:46:40.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.693+0000 7f1d1bfff640 1 -- 172.21.15.70:0/1488020118 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1d2c0d5070 con 0x5578efa466a0 2024-04-01T01:46:40.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.695+0000 7f1d45bbf740 10 client.29333 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:40.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.695+0000 7f1d45bbf740 1 -- 172.21.15.70:0/1488020118 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5578efae4960 con 0x5578efa466a0 2024-04-01T01:46:40.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.695+0000 7f1d1bfff640 1 -- 172.21.15.70:0/1488020118 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1d3c08a8a0 con 0x5578efa466a0 2024-04-01T01:46:40.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.695+0000 7f1d1bfff640 1 -- 172.21.15.70:0/1488020118 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f1d2c03c040 con 0x5578efa466a0 2024-04-01T01:46:40.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.695+0000 7f1d45bbf740 10 client.29333 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:40.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.695+0000 7f1d45bbf740 10 client.29333 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:46:40.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.695+0000 7f1d45bbf740 20 client.29333 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:40.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.695+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/1488020118 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5578efae5100 0x5578efae7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.695+0000 7f1d45bbf740 4 client.29333 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:46:40.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.695+0000 7f1d45bbf740 1 -- 172.21.15.70:0/1488020118 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5578ef9b2a40 con 0x5578efae5100 2024-04-01T01:46:40.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.697+0000 7f1d3bfff640 1 --2- 172.21.15.70:0/1488020118 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5578efae5100 0x5578efae7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.702+0000 7f1d3bfff640 1 --2- 172.21.15.70:0/1488020118 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5578efae5100 0x5578efae7560 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fdaa409972490f34 server_cookie=aa963746adb7ca0c in_seq=0 out_seq=0 2024-04-01T01:46:40.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.702+0000 7f1d1bfff640 10 client.29333 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:46:40.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.702+0000 7f1d1bfff640 1 -- 172.21.15.70:0/1488020118 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5578ef9b2a40 con 0x5578efae5100 2024-04-01T01:46:40.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.702+0000 7f1d1bfff640 10 client.29333 handle_command_reply: tid=0 2024-04-01T01:46:40.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.702+0000 7f1d45bbf740 1 client.29333 shutdown 2024-04-01T01:46:40.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.703+0000 7f1d45bbf740 1 -- 172.21.15.70:0/1488020118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa466a0 msgr2=0x5578efa55900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.703+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/1488020118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa466a0 0x5578efa55900 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c0db270 tx=0x7f1d2c035f70 comp rx=0 tx=0).stop 2024-04-01T01:46:40.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.703+0000 7f1d45bbf740 1 -- 172.21.15.70:0/1488020118 shutdown_connections 2024-04-01T01:46:40.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.703+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/1488020118 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5578efae5100 0x5578efae7560 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.703+0000 7f1d1bfff640 0 client.29333 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:46:40.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.703+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/1488020118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5578efa4c200 0x5578efa56d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.703+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/1488020118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578efa47090 0x5578efa55e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.703+0000 7f1d45bbf740 1 --2- 172.21.15.70:0/1488020118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5578efa466a0 0x5578efa55900 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.703+0000 7f1d45bbf740 1 -- 172.21.15.70:0/1488020118 >> 172.21.15.70:0/1488020118 conn(0x5578efa41c10 msgr2=0x5578efa50be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:40.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.703+0000 7f1d45bbf740 1 -- 172.21.15.70:0/1488020118 shutdown_connections 2024-04-01T01:46:40.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.703+0000 7f1d45bbf740 1 -- 172.21.15.70:0/1488020118 wait complete. 2024-04-01T01:46:40.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.704+0000 7f1d45bbf740 20 client.29333 trim_cache size 0 max 16384 2024-04-01T01:46:40.704 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:46:40.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.705+0000 7f1d41d80640 1 -- 172.21.15.70:0/605491124 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1d240a25a0 msgr2=0x7f1d240a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.705+0000 7f1d41d80640 1 --2- 172.21.15.70:0/605491124 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1d240a25a0 0x7f1d240a4a60 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c067290 tx=0x7f1d3c077040 comp rx=0 tx=0).stop 2024-04-01T01:46:40.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.705+0000 7f1d41d80640 1 -- 172.21.15.70:0/605491124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d34094c80 msgr2=0x7f1d340052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.705+0000 7f1d41d80640 1 --2- 172.21.15.70:0/605491124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d34094c80 0x7f1d340052b0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c037040 tx=0x7f1d2c0042f0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.706+0000 7f1d41d80640 1 -- 172.21.15.70:0/605491124 shutdown_connections 2024-04-01T01:46:40.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.706+0000 7f1d41d80640 1 --2- 172.21.15.70:0/605491124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d34096830 0x7f1d34133330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.706+0000 7f1d41d80640 1 --2- 172.21.15.70:0/605491124 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f1d240a25a0 0x7f1d240a4a60 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.706+0000 7f1d41d80640 1 --2- 172.21.15.70:0/605491124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d34095e70 0x7f1d34130d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.706+0000 7f1d41d80640 1 --2- 172.21.15.70:0/605491124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d34094c80 0x7f1d340052b0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.706+0000 7f1d41d80640 1 -- 172.21.15.70:0/605491124 >> 172.21.15.70:0/605491124 conn(0x7f1d340901d0 msgr2=0x7f1d34099290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:40.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.707+0000 7f1d41d80640 1 -- 172.21.15.70:0/605491124 shutdown_connections 2024-04-01T01:46:40.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.707+0000 7f1d41d80640 1 -- 172.21.15.70:0/605491124 wait complete. 2024-04-01T01:46:40.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:46:40.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.907+0000 7f6f42d1e640 1 -- 172.21.15.70:0/2393130972 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f3c108750 msgr2=0x7f6f3c108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.907+0000 7f6f42d1e640 1 --2- 172.21.15.70:0/2393130972 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f3c108750 0x7f6f3c108b70 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6f34009ca0 tx=0x7f6f3402c100 comp rx=0 tx=0).stop 2024-04-01T01:46:40.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.907+0000 7f6f42d1e640 1 -- 172.21.15.70:0/2393130972 shutdown_connections 2024-04-01T01:46:40.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.907+0000 7f6f42d1e640 1 --2- 172.21.15.70:0/2393130972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3c10ffd0 0x7f6f3c1123c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.907+0000 7f6f42d1e640 1 --2- 172.21.15.70:0/2393130972 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f3c1090b0 0x7f6f3c10fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.907+0000 7f6f42d1e640 1 --2- 172.21.15.70:0/2393130972 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f3c108750 0x7f6f3c108b70 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.908+0000 7f6f42d1e640 1 -- 172.21.15.70:0/2393130972 >> 172.21.15.70:0/2393130972 conn(0x7f6f3c06e950 msgr2=0x7f6f3c070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:40.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.908+0000 7f6f42d1e640 1 -- 172.21.15.70:0/2393130972 shutdown_connections 2024-04-01T01:46:40.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.908+0000 7f6f42d1e640 1 -- 172.21.15.70:0/2393130972 wait complete. 2024-04-01T01:46:40.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.909+0000 7f6f42d1e640 1 Processor -- start 2024-04-01T01:46:40.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.909+0000 7f6f42d1e640 1 -- start start 2024-04-01T01:46:40.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.909+0000 7f6f42d1e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3c108750 0x7f6f3c1a0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.909+0000 7f6f42d1e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f3c1090b0 0x7f6f3c1a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.910+0000 7f6f42d1e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f3c10ffd0 0x7f6f3c1a7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.910+0000 7f6f40aba640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3c108750 0x7f6f3c1a0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.910+0000 7f6f40aba640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3c108750 0x7f6f3c1a0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57778/0 (socket says 172.21.15.70:57778) 2024-04-01T01:46:40.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.910+0000 7f6f40aba640 1 -- 172.21.15.70:0/2122934977 learned_addr learned my addr 172.21.15.70:0/2122934977 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:40.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.910+0000 7f6f42d1e640 1 -- 172.21.15.70:0/2122934977 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3c115ae0 con 0x7f6f3c10ffd0 2024-04-01T01:46:40.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.910+0000 7f6f412bb640 1 --2- 172.21.15.70:0/2122934977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f3c10ffd0 0x7f6f3c1a7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.910+0000 7f6f42d1e640 1 -- 172.21.15.70:0/2122934977 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3c115960 con 0x7f6f3c1090b0 2024-04-01T01:46:40.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.910+0000 7f6f42d1e640 1 -- 172.21.15.70:0/2122934977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3c115c60 con 0x7f6f3c108750 2024-04-01T01:46:40.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.911+0000 7f6f40aba640 1 -- 172.21.15.70:0/2122934977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f3c1090b0 msgr2=0x7f6f3c1a0750 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:46:40.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.911+0000 7f6f40aba640 1 --2- 172.21.15.70:0/2122934977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f3c1090b0 0x7f6f3c1a0750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.911+0000 7f6f40aba640 1 -- 172.21.15.70:0/2122934977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f3c10ffd0 msgr2=0x7f6f3c1a7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.911+0000 7f6f40aba640 1 --2- 172.21.15.70:0/2122934977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f3c10ffd0 0x7f6f3c1a7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.911+0000 7f6f40aba640 1 -- 172.21.15.70:0/2122934977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f3c1a7de0 con 0x7f6f3c108750 2024-04-01T01:46:40.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.911+0000 7f6f40aba640 1 --2- 172.21.15.70:0/2122934977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3c108750 0x7f6f3c1a0210 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6f34000f80 tx=0x7f6f34035d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:40.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.911+0000 7f6f39ffb640 1 -- 172.21.15.70:0/2122934977 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f34039040 con 0x7f6f3c108750 2024-04-01T01:46:40.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.912+0000 7f6f39ffb640 1 -- 172.21.15.70:0/2122934977 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f34003e20 con 0x7f6f3c108750 2024-04-01T01:46:40.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.912+0000 7f6f39ffb640 1 -- 172.21.15.70:0/2122934977 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f3403d5f0 con 0x7f6f3c108750 2024-04-01T01:46:40.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.914+0000 7f6f42d1e640 1 -- 172.21.15.70:0/2122934977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f3c109860 con 0x7f6f3c108750 2024-04-01T01:46:40.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.914+0000 7f6f42d1e640 1 -- 172.21.15.70:0/2122934977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f3c109d50 con 0x7f6f3c108750 2024-04-01T01:46:40.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.917+0000 7f6f459d9740 1 Processor -- start 2024-04-01T01:46:40.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.920+0000 7f6f39ffb640 1 -- 172.21.15.70:0/2122934977 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f6f340041b0 con 0x7f6f3c108750 2024-04-01T01:46:40.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.921+0000 7f6f39ffb640 1 --2- 172.21.15.70:0/2122934977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f240a25a0 0x7f6f240a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.921+0000 7f6f39ffb640 1 -- 172.21.15.70:0/2122934977 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6f340f0cf0 con 0x7f6f3c108750 2024-04-01T01:46:40.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.921+0000 7f6f412bb640 1 --2- 172.21.15.70:0/2122934977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f240a25a0 0x7f6f240a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.922+0000 7f6f412bb640 1 --2- 172.21.15.70:0/2122934977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f240a25a0 0x7f6f240a4a60 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f6f30009900 tx=0x7f6f300090e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:40.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.922+0000 7f6f459d9740 1 -- start start 2024-04-01T01:46:40.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.923+0000 7f6f459d9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f509fca0 0x5634f50a00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.923+0000 7f6f459d9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a0690 0x5634f50a5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.923+0000 7f6f459d9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 0x5634f50a7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.923+0000 7f6f459d9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5634f4ffc1b0 con 0x5634f509fca0 2024-04-01T01:46:40.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.923+0000 7f6f459d9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5634f4fa4c00 con 0x5634f50a0690 2024-04-01T01:46:40.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.923+0000 7f6f459d9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5634f4ff8250 con 0x5634f50a5800 2024-04-01T01:46:40.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.923+0000 7f6f3bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a0690 0x5634f50a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.923+0000 7f6f3bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a0690 0x5634f50a5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42866/0 (socket says 172.21.15.70:42866) 2024-04-01T01:46:40.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.923+0000 7f6f3bfff640 1 -- 172.21.15.70:0/3105775190 learned_addr learned my addr 172.21.15.70:0/3105775190 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:40.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.923+0000 7f6f412bb640 1 --2- 172.21.15.70:0/3105775190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 0x5634f50a7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.924+0000 7f6f40aba640 1 --2- 172.21.15.70:0/3105775190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f509fca0 0x5634f50a00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.924+0000 7f6f412bb640 1 -- 172.21.15.70:0/3105775190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a0690 msgr2=0x5634f50a5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:40.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.924+0000 7f6f412bb640 1 --2- 172.21.15.70:0/3105775190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a0690 0x5634f50a5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.924+0000 7f6f412bb640 1 -- 172.21.15.70:0/3105775190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f509fca0 msgr2=0x5634f50a00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:40.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.924+0000 7f6f412bb640 1 --2- 172.21.15.70:0/3105775190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f509fca0 0x5634f50a00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.924+0000 7f6f412bb640 1 -- 172.21.15.70:0/3105775190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634f4fc43d0 con 0x5634f50a5800 2024-04-01T01:46:40.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.924+0000 7f6f412bb640 1 --2- 172.21.15.70:0/3105775190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 0x5634f50a7c90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6f3000eb70 tx=0x7f6f30011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d445a5437fb6ba3d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:40.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.927+0000 7f6f22ffd640 1 -- 172.21.15.70:0/3105775190 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f3001b040 con 0x5634f50a5800 2024-04-01T01:46:40.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.927+0000 7f6f22ffd640 1 -- 172.21.15.70:0/3105775190 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f30024070 con 0x5634f50a5800 2024-04-01T01:46:40.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.927+0000 7f6f22ffd640 1 -- 172.21.15.70:0/3105775190 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f30013df0 con 0x5634f50a5800 2024-04-01T01:46:40.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.928+0000 7f6f459d9740 1 -- 172.21.15.70:0/3105775190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 msgr2=0x5634f50a7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.928+0000 7f6f459d9740 1 --2- 172.21.15.70:0/3105775190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 0x5634f50a7c90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6f3000eb70 tx=0x7f6f30011000 comp rx=0 tx=0).stop 2024-04-01T01:46:40.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.929+0000 7f6f459d9740 1 -- 172.21.15.70:0/3105775190 shutdown_connections 2024-04-01T01:46:40.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.929+0000 7f6f459d9740 1 --2- 172.21.15.70:0/3105775190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 0x5634f50a7c90 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.929+0000 7f6f459d9740 1 --2- 172.21.15.70:0/3105775190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a0690 0x5634f50a5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.929+0000 7f6f459d9740 1 --2- 172.21.15.70:0/3105775190 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f509fca0 0x5634f50a00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.929+0000 7f6f459d9740 1 -- 172.21.15.70:0/3105775190 >> 172.21.15.70:0/3105775190 conn(0x5634f509b210 msgr2=0x5634f509b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:40.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.931+0000 7f6f459d9740 1 -- 172.21.15.70:0/3105775190 shutdown_connections 2024-04-01T01:46:40.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.931+0000 7f6f459d9740 1 -- 172.21.15.70:0/3105775190 wait complete. 2024-04-01T01:46:40.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.931+0000 7f6f459d9740 1 -- start start 2024-04-01T01:46:40.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.931+0000 7f6f459d9740 1 Processor -- start 2024-04-01T01:46:40.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.932+0000 7f6f459d9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f509fca0 0x5634f50aa060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.932+0000 7f6f459d9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a0690 0x5634f50aa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.932+0000 7f6f459d9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 0x5634f50b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.932+0000 7f6f459d9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5634f50a87b0 con 0x5634f509fca0 2024-04-01T01:46:40.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.932+0000 7f6f459d9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5634f4f9a470 con 0x5634f50a0690 2024-04-01T01:46:40.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.932+0000 7f6f459d9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5634f50a8930 con 0x5634f50a5800 2024-04-01T01:46:40.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.932+0000 7f6f459d9740 10 client.0 _ll_register_callbacks cb 0x5634f4fb7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.932+0000 7f6f3bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a0690 0x5634f50aa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.932+0000 7f6f412bb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 0x5634f50b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.932+0000 7f6f412bb640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 0x5634f50b15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57784/0 (socket says 172.21.15.70:57784) 2024-04-01T01:46:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.932+0000 7f6f412bb640 1 -- 172.21.15.70:0/1656162380 learned_addr learned my addr 172.21.15.70:0/1656162380 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.932+0000 7f6f40aba640 1 --2- 172.21.15.70:0/1656162380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f509fca0 0x5634f50aa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.933+0000 7f6f412bb640 1 -- 172.21.15.70:0/1656162380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a0690 msgr2=0x5634f50aa5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.933+0000 7f6f412bb640 1 --2- 172.21.15.70:0/1656162380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a0690 0x5634f50aa5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.933+0000 7f6f412bb640 1 -- 172.21.15.70:0/1656162380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f509fca0 msgr2=0x5634f50aa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.933+0000 7f6f412bb640 1 --2- 172.21.15.70:0/1656162380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f509fca0 0x5634f50aa060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.933+0000 7f6f412bb640 1 -- 172.21.15.70:0/1656162380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634f4d103f0 con 0x5634f50a5800 2024-04-01T01:46:40.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.934+0000 7f6f412bb640 1 --2- 172.21.15.70:0/1656162380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 0x5634f50b15e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6f30013d60 tx=0x7f6f30013d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e15677db62576f94 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:40.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.934+0000 7f6f1bfff640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:46:40.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.938+0000 7f6f1bfff640 1 -- 172.21.15.70:0/1656162380 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f3002d070 con 0x5634f50a5800 2024-04-01T01:46:40.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.938+0000 7f6f1bfff640 1 -- 172.21.15.70:0/1656162380 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6f30029400 con 0x5634f50a5800 2024-04-01T01:46:40.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.938+0000 7f6f459d9740 1 -- 172.21.15.70:0/1656162380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f30024070 con 0x5634f50a5800 2024-04-01T01:46:40.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.939+0000 7f6f1bfff640 1 -- 172.21.15.70:0/1656162380 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6f3000e7f0 con 0x5634f50a5800 2024-04-01T01:46:40.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.939+0000 7f6f459d9740 10 client.29116 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:40.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.939+0000 7f6f459d9740 1 -- 172.21.15.70:0/1656162380 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5634f4dda7d0 con 0x5634f50a5800 2024-04-01T01:46:40.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.940+0000 7f6f1bfff640 1 -- 172.21.15.70:0/1656162380 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f30029ab0 con 0x5634f50a5800 2024-04-01T01:46:40.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.941+0000 7f6f1bfff640 1 -- 172.21.15.70:0/1656162380 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6f3001e030 con 0x5634f50a5800 2024-04-01T01:46:40.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.941+0000 7f6f459d9740 10 client.29116 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:40.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.941+0000 7f6f459d9740 10 client.29116 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:46:40.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.941+0000 7f6f459d9740 20 client.29116 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:40.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.941+0000 7f6f459d9740 1 --2- 172.21.15.70:0/1656162380 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5634f50b5d30 0x5634f50b8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.941+0000 7f6f459d9740 4 client.29116 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:40.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.941+0000 7f6f459d9740 1 -- 172.21.15.70:0/1656162380 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6f30024070 con 0x5634f50b5d30 2024-04-01T01:46:40.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.942+0000 7f6f40aba640 1 --2- 172.21.15.70:0/1656162380 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5634f50b5d30 0x5634f50b8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.942+0000 7f6f1bfff640 10 client.29116 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:46:40.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.942+0000 7f6f40aba640 1 --2- 172.21.15.70:0/1656162380 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5634f50b5d30 0x5634f50b8150 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=87c3c6496d11448d server_cookie=1203ccb526db67c in_seq=0 out_seq=0 2024-04-01T01:46:40.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.943+0000 7f6f1bfff640 1 -- 172.21.15.70:0/1656162380 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f6f30024070 con 0x5634f50b5d30 2024-04-01T01:46:40.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.943+0000 7f6f1bfff640 10 client.29116 handle_command_reply: tid=0 2024-04-01T01:46:40.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.943+0000 7f6f459d9740 1 client.29116 shutdown 2024-04-01T01:46:40.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.945+0000 7f6f459d9740 1 -- 172.21.15.70:0/1656162380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 msgr2=0x5634f50b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.945+0000 7f6f459d9740 1 --2- 172.21.15.70:0/1656162380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 0x5634f50b15e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6f30013d60 tx=0x7f6f30013d90 comp rx=0 tx=0).stop 2024-04-01T01:46:40.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.945+0000 7f6f459d9740 1 -- 172.21.15.70:0/1656162380 shutdown_connections 2024-04-01T01:46:40.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.945+0000 7f6f459d9740 1 --2- 172.21.15.70:0/1656162380 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5634f50b5d30 0x5634f50b8150 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.946+0000 7f6f459d9740 1 --2- 172.21.15.70:0/1656162380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 0x5634f50b15e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.946+0000 7f6f459d9740 1 --2- 172.21.15.70:0/1656162380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a0690 0x5634f50aa5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.946+0000 7f6f459d9740 1 --2- 172.21.15.70:0/1656162380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f509fca0 0x5634f50aa060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.946+0000 7f6f459d9740 1 -- 172.21.15.70:0/1656162380 >> 172.21.15.70:0/1656162380 conn(0x5634f509b210 msgr2=0x5634f5091d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:40.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.946+0000 7f6f1bfff640 0 client.29116 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:46:40.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.946+0000 7f6f459d9740 1 -- 172.21.15.70:0/1656162380 shutdown_connections 2024-04-01T01:46:40.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.946+0000 7f6f459d9740 1 -- 172.21.15.70:0/1656162380 wait complete. 2024-04-01T01:46:40.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.947+0000 7f6f459d9740 20 client.29116 trim_cache size 0 max 16384 2024-04-01T01:46:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.973+0000 7f6f459d9740 1 Processor -- start 2024-04-01T01:46:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.973+0000 7f6f459d9740 1 -- start start 2024-04-01T01:46:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.974+0000 7f6f459d9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f509fca0 0x5634f50914b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.974+0000 7f6f459d9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f50a0690 0x5634f50b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.974+0000 7f6f40aba640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f509fca0 0x5634f50914b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.974+0000 7f6f3bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f50a0690 0x5634f50b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.974+0000 7f6f3bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f50a0690 0x5634f50b15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39214/0 (socket says 172.21.15.70:39214) 2024-04-01T01:46:40.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.974+0000 7f6f40aba640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f509fca0 0x5634f50914b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42884/0 (socket says 172.21.15.70:42884) 2024-04-01T01:46:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.974+0000 7f6f3bfff640 1 -- 172.21.15.70:0/506768371 learned_addr learned my addr 172.21.15.70:0/506768371 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.974+0000 7f6f459d9740 1 --2- 172.21.15.70:0/506768371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 0x5634f50919f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.975+0000 7f6f459d9740 1 -- 172.21.15.70:0/506768371 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3c1b0940 con 0x5634f50a0690 2024-04-01T01:46:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.975+0000 7f6f459d9740 1 -- 172.21.15.70:0/506768371 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3c1b2480 con 0x5634f509fca0 2024-04-01T01:46:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.975+0000 7f6f412bb640 1 --2- 172.21.15.70:0/506768371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 0x5634f50919f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.975+0000 7f6f459d9740 1 -- 172.21.15.70:0/506768371 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3c1b1010 con 0x5634f50a5800 2024-04-01T01:46:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.976+0000 7f6f3bfff640 1 -- 172.21.15.70:0/506768371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 msgr2=0x5634f50919f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.976+0000 7f6f3bfff640 1 --2- 172.21.15.70:0/506768371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 0x5634f50919f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.976+0000 7f6f3bfff640 1 -- 172.21.15.70:0/506768371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f509fca0 msgr2=0x5634f50914b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.976+0000 7f6f3bfff640 1 --2- 172.21.15.70:0/506768371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f509fca0 0x5634f50914b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.976+0000 7f6f3bfff640 1 -- 172.21.15.70:0/506768371 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634f508e460 con 0x5634f50a0690 2024-04-01T01:46:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.976+0000 7f6f3bfff640 1 --2- 172.21.15.70:0/506768371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f50a0690 0x5634f50b15a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c00eb50 tx=0x7f6f2c00ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=282b77a9daccffff server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.977+0000 7f6f227fc640 1 -- 172.21.15.70:0/506768371 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c004020 con 0x5634f50a0690 2024-04-01T01:46:40.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.977+0000 7f6f227fc640 1 -- 172.21.15.70:0/506768371 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f2c0041c0 con 0x5634f50a0690 2024-04-01T01:46:40.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.977+0000 7f6f227fc640 1 -- 172.21.15.70:0/506768371 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c011660 con 0x5634f50a0690 2024-04-01T01:46:40.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.978+0000 7f6f459d9740 1 -- 172.21.15.70:0/506768371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f50a0690 msgr2=0x5634f50b15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.978+0000 7f6f459d9740 1 --2- 172.21.15.70:0/506768371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f50a0690 0x5634f50b15a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c00eb50 tx=0x7f6f2c00ef50 comp rx=0 tx=0).stop 2024-04-01T01:46:40.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.978+0000 7f6f459d9740 1 -- 172.21.15.70:0/506768371 shutdown_connections 2024-04-01T01:46:40.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.978+0000 7f6f459d9740 1 --2- 172.21.15.70:0/506768371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f50a5800 0x5634f50919f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.978+0000 7f6f459d9740 1 --2- 172.21.15.70:0/506768371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f50a0690 0x5634f50b15a0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.979+0000 7f6f459d9740 1 --2- 172.21.15.70:0/506768371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f509fca0 0x5634f50914b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.979+0000 7f6f459d9740 1 -- 172.21.15.70:0/506768371 >> 172.21.15.70:0/506768371 conn(0x5634f509b210 msgr2=0x5634f50a1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:40.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.979+0000 7f6f459d9740 1 -- 172.21.15.70:0/506768371 shutdown_connections 2024-04-01T01:46:40.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.979+0000 7f6f459d9740 1 -- 172.21.15.70:0/506768371 wait complete. 2024-04-01T01:46:40.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.979+0000 7f6f459d9740 1 -- start start 2024-04-01T01:46:40.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.979+0000 7f6f459d9740 1 Processor -- start 2024-04-01T01:46:40.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.980+0000 7f6f459d9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f509fca0 0x5634f50af220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.980+0000 7f6f459d9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f50a0690 0x5634f50af760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.980+0000 7f6f40aba640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f509fca0 0x5634f50af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.980+0000 7f6f40aba640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f509fca0 0x5634f50af220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57810/0 (socket says 172.21.15.70:57810) 2024-04-01T01:46:40.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.980+0000 7f6f40aba640 1 -- 172.21.15.70:0/3680948475 learned_addr learned my addr 172.21.15.70:0/3680948475 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:40.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.981+0000 7f6f459d9740 1 --2- 172.21.15.70:0/3680948475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a5800 0x5634f50b0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.981+0000 7f6f459d9740 1 -- 172.21.15.70:0/3680948475 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3c1b1db0 con 0x5634f50a0690 2024-04-01T01:46:40.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.981+0000 7f6f412bb640 1 --2- 172.21.15.70:0/3680948475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a5800 0x5634f50b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.981+0000 7f6f459d9740 1 -- 172.21.15.70:0/3680948475 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3c1b16e0 con 0x5634f50a5800 2024-04-01T01:46:40.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.981+0000 7f6f459d9740 1 -- 172.21.15.70:0/3680948475 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3c1ae9d0 con 0x5634f509fca0 2024-04-01T01:46:40.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.981+0000 7f6f459d9740 10 client.0 _ll_register_callbacks cb 0x7f6f30013810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:40.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.981+0000 7f6f3bfff640 1 --2- 172.21.15.70:0/3680948475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f50a0690 0x5634f50af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.982+0000 7f6f412bb640 1 -- 172.21.15.70:0/3680948475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f509fca0 msgr2=0x5634f50af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:40.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.982+0000 7f6f412bb640 1 --2- 172.21.15.70:0/3680948475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f509fca0 0x5634f50af220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.982+0000 7f6f412bb640 1 -- 172.21.15.70:0/3680948475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f50a0690 msgr2=0x5634f50af760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:40.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.982+0000 7f6f412bb640 1 --2- 172.21.15.70:0/3680948475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f50a0690 0x5634f50af760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.982+0000 7f6f412bb640 1 -- 172.21.15.70:0/3680948475 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634f50b0cf0 con 0x5634f50a5800 2024-04-01T01:46:40.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.982+0000 7f6f412bb640 1 --2- 172.21.15.70:0/3680948475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a5800 0x5634f50b0700 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f6f3000e9b0 tx=0x7f6f30015980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7cf4d2396683015 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:40.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.982+0000 7f6f1bfff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:40.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.983+0000 7f6f1bfff640 1 -- 172.21.15.70:0/3680948475 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f3002d050 con 0x5634f50a5800 2024-04-01T01:46:40.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.983+0000 7f6f1bfff640 1 -- 172.21.15.70:0/3680948475 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6f30038020 con 0x5634f50a5800 2024-04-01T01:46:40.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.983+0000 7f6f459d9740 1 -- 172.21.15.70:0/3680948475 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f2c0041c0 con 0x5634f50a5800 2024-04-01T01:46:40.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.983+0000 7f6f1bfff640 1 -- 172.21.15.70:0/3680948475 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6f3003f480 con 0x5634f50a5800 2024-04-01T01:46:40.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.984+0000 7f6f459d9740 10 client.29348 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:40.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.984+0000 7f6f459d9740 1 -- 172.21.15.70:0/3680948475 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5634f511c560 con 0x5634f50a5800 2024-04-01T01:46:40.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.989+0000 7f6f1bfff640 1 -- 172.21.15.70:0/3680948475 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f30024040 con 0x5634f50a5800 2024-04-01T01:46:40.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.989+0000 7f6f1bfff640 1 -- 172.21.15.70:0/3680948475 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6f3001e040 con 0x5634f50a5800 2024-04-01T01:46:40.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.991+0000 7f6f459d9740 10 client.29348 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:40.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.991+0000 7f6f459d9740 10 client.29348 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:46:40.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.991+0000 7f6f459d9740 20 client.29348 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:40.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.991+0000 7f6f459d9740 1 --2- 172.21.15.70:0/3680948475 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5634f511ceb0 0x5634f511f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:40.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.991+0000 7f6f459d9740 4 client.29348 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:46:40.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.991+0000 7f6f459d9740 1 -- 172.21.15.70:0/3680948475 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f6f2c0041c0 con 0x5634f511ceb0 2024-04-01T01:46:40.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.991+0000 7f6f40aba640 1 --2- 172.21.15.70:0/3680948475 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5634f511ceb0 0x5634f511f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:40.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.992+0000 7f6f40aba640 1 --2- 172.21.15.70:0/3680948475 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5634f511ceb0 0x5634f511f2d0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9fcbb439ee1e3638 server_cookie=e37e8dad0518367f in_seq=0 out_seq=0 2024-04-01T01:46:40.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.992+0000 7f6f1bfff640 10 client.29348 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:46:40.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.993+0000 7f6f1bfff640 1 -- 172.21.15.70:0/3680948475 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f6f2c0041c0 con 0x5634f511ceb0 2024-04-01T01:46:40.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.993+0000 7f6f1bfff640 10 client.29348 handle_command_reply: tid=0 2024-04-01T01:46:40.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.993+0000 7f6f459d9740 1 client.29348 shutdown 2024-04-01T01:46:40.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.996+0000 7f6f459d9740 1 -- 172.21.15.70:0/3680948475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a5800 msgr2=0x5634f50b0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.996+0000 7f6f459d9740 1 --2- 172.21.15.70:0/3680948475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a5800 0x5634f50b0700 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f6f3000e9b0 tx=0x7f6f30015980 comp rx=0 tx=0).stop 2024-04-01T01:46:40.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.996+0000 7f6f459d9740 1 -- 172.21.15.70:0/3680948475 shutdown_connections 2024-04-01T01:46:40.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.996+0000 7f6f459d9740 1 --2- 172.21.15.70:0/3680948475 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5634f511ceb0 0x5634f511f2d0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.996+0000 7f6f459d9740 1 --2- 172.21.15.70:0/3680948475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634f50a5800 0x5634f50b0700 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.996+0000 7f6f459d9740 1 --2- 172.21.15.70:0/3680948475 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5634f50a0690 0x5634f50af760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.996+0000 7f6f1bfff640 0 client.29348 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:46:40.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.996+0000 7f6f459d9740 1 --2- 172.21.15.70:0/3680948475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634f509fca0 0x5634f50af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.996+0000 7f6f459d9740 1 -- 172.21.15.70:0/3680948475 >> 172.21.15.70:0/3680948475 conn(0x5634f509b210 msgr2=0x5634f50914b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:40.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.997+0000 7f6f459d9740 1 -- 172.21.15.70:0/3680948475 shutdown_connections 2024-04-01T01:46:40.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.997+0000 7f6f459d9740 1 -- 172.21.15.70:0/3680948475 wait complete. 2024-04-01T01:46:40.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:40.997+0000 7f6f459d9740 20 client.29348 trim_cache size 0 max 16384 2024-04-01T01:46:40.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:40 smithi189 ceph-mon[35753]: pgmap v754: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.0 MiB/s wr, 755 op/s 2024-04-01T01:46:40.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:40 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3998934384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:46:40.999 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:46:40.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.000+0000 7f6f42d1e640 1 -- 172.21.15.70:0/2122934977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f240a25a0 msgr2=0x7f6f240a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:40.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.000+0000 7f6f42d1e640 1 --2- 172.21.15.70:0/2122934977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f240a25a0 0x7f6f240a4a60 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f6f30009900 tx=0x7f6f300090e0 comp rx=0 tx=0).stop 2024-04-01T01:46:40.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.000+0000 7f6f42d1e640 1 -- 172.21.15.70:0/2122934977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3c108750 msgr2=0x7f6f3c1a0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.000+0000 7f6f42d1e640 1 --2- 172.21.15.70:0/2122934977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3c108750 0x7f6f3c1a0210 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6f34000f80 tx=0x7f6f34035d60 comp rx=0 tx=0).stop 2024-04-01T01:46:41.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.000+0000 7f6f42d1e640 1 -- 172.21.15.70:0/2122934977 shutdown_connections 2024-04-01T01:46:41.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.000+0000 7f6f42d1e640 1 --2- 172.21.15.70:0/2122934977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6f3c10ffd0 0x7f6f3c1a7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.000+0000 7f6f42d1e640 1 --2- 172.21.15.70:0/2122934977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f3c1090b0 0x7f6f3c1a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.000+0000 7f6f42d1e640 1 --2- 172.21.15.70:0/2122934977 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6f240a25a0 0x7f6f240a4a60 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.000+0000 7f6f42d1e640 1 --2- 172.21.15.70:0/2122934977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6f3c108750 0x7f6f3c1a0210 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.000+0000 7f6f42d1e640 1 -- 172.21.15.70:0/2122934977 >> 172.21.15.70:0/2122934977 conn(0x7f6f3c06e950 msgr2=0x7f6f3c10ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.001+0000 7f6f42d1e640 1 -- 172.21.15.70:0/2122934977 shutdown_connections 2024-04-01T01:46:41.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.001+0000 7f6f42d1e640 1 -- 172.21.15.70:0/2122934977 wait complete. 2024-04-01T01:46:41.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:40 smithi070 ceph-mon[32179]: pgmap v754: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.0 MiB/s wr, 755 op/s 2024-04-01T01:46:41.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:40 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3998934384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:46:41.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:46:41.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:40 smithi081 ceph-mon[35142]: pgmap v754: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 7.0 MiB/s wr, 755 op/s 2024-04-01T01:46:41.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:40 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3998934384' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:46:41.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.196+0000 7fee5a11f640 1 -- 172.21.15.70:0/436295072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee541097e0 msgr2=0x7fee5410e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.196+0000 7fee5a11f640 1 --2- 172.21.15.70:0/436295072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee541097e0 0x7fee5410e080 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fee4c009ca0 tx=0x7fee4c02c100 comp rx=0 tx=0).stop 2024-04-01T01:46:41.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.196+0000 7fee5a11f640 1 -- 172.21.15.70:0/436295072 shutdown_connections 2024-04-01T01:46:41.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.196+0000 7fee5a11f640 1 --2- 172.21.15.70:0/436295072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee541097e0 0x7fee5410e080 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.196+0000 7fee5a11f640 1 --2- 172.21.15.70:0/436295072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee54108e20 0x7fee541092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.196+0000 7fee5a11f640 1 --2- 172.21.15.70:0/436295072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee54072b60 0x7fee54072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.196+0000 7fee5a11f640 1 -- 172.21.15.70:0/436295072 >> 172.21.15.70:0/436295072 conn(0x7fee5406e630 msgr2=0x7fee54070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.197+0000 7fee5a11f640 1 -- 172.21.15.70:0/436295072 shutdown_connections 2024-04-01T01:46:41.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.200+0000 7fee5a11f640 1 -- 172.21.15.70:0/436295072 wait complete. 2024-04-01T01:46:41.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.200+0000 7fee5a11f640 1 Processor -- start 2024-04-01T01:46:41.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.203+0000 7fee5a11f640 1 -- start start 2024-04-01T01:46:41.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.204+0000 7fee5a11f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee54072b60 0x7fee54114e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.204+0000 7fee5a11f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee54108e20 0x7fee541153a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.204+0000 7fee5a11f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee541097e0 0x7fee541b66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.204+0000 7fee5a11f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fee5411bfd0 con 0x7fee54108e20 2024-04-01T01:46:41.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.204+0000 7fee5a11f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fee5411be50 con 0x7fee54072b60 2024-04-01T01:46:41.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.204+0000 7fee5a11f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fee5411c150 con 0x7fee541097e0 2024-04-01T01:46:41.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.206+0000 7fee5991e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee541097e0 0x7fee541b66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.206+0000 7fee5891c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee54108e20 0x7fee541153a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.206+0000 7fee5991e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee541097e0 0x7fee541b66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57826/0 (socket says 172.21.15.70:57826) 2024-04-01T01:46:41.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.206+0000 7fee5991e640 1 -- 172.21.15.70:0/4068648295 learned_addr learned my addr 172.21.15.70:0/4068648295 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:41.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.206+0000 7fee5911d640 1 --2- 172.21.15.70:0/4068648295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee54072b60 0x7fee54114e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.206+0000 7fee5991e640 1 -- 172.21.15.70:0/4068648295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee54072b60 msgr2=0x7fee54114e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.206+0000 7fee5991e640 1 --2- 172.21.15.70:0/4068648295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee54072b60 0x7fee54114e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.206+0000 7fee5991e640 1 -- 172.21.15.70:0/4068648295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee54108e20 msgr2=0x7fee541153a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.206+0000 7fee5991e640 1 --2- 172.21.15.70:0/4068648295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee54108e20 0x7fee541153a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.206+0000 7fee5991e640 1 -- 172.21.15.70:0/4068648295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee541b6c20 con 0x7fee541097e0 2024-04-01T01:46:41.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.207+0000 7fee5911d640 1 --2- 172.21.15.70:0/4068648295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee54072b60 0x7fee54114e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:46:41.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.208+0000 7fee5991e640 1 --2- 172.21.15.70:0/4068648295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee541097e0 0x7fee541b66e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0098d0 tx=0x7fee4c035ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.209+0000 7fee467fc640 1 -- 172.21.15.70:0/4068648295 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c039040 con 0x7fee541097e0 2024-04-01T01:46:41.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.210+0000 7fee467fc640 1 -- 172.21.15.70:0/4068648295 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee4c044070 con 0x7fee541097e0 2024-04-01T01:46:41.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.210+0000 7fee467fc640 1 -- 172.21.15.70:0/4068648295 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c004460 con 0x7fee541097e0 2024-04-01T01:46:41.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.210+0000 7fee5a11f640 1 -- 172.21.15.70:0/4068648295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee541b6df0 con 0x7fee541097e0 2024-04-01T01:46:41.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.210+0000 7fee5a11f640 1 -- 172.21.15.70:0/4068648295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee541b7270 con 0x7fee541097e0 2024-04-01T01:46:41.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.214+0000 7fee5df65740 1 Processor -- start 2024-04-01T01:46:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.214+0000 7fee5df65740 1 -- start start 2024-04-01T01:46:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.214+0000 7fee5df65740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc156a0 0x55f6cdc15ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.214+0000 7fee5df65740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc16090 0x55f6cdc1ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.214+0000 7fee5df65740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 0x55f6cdc1d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.214+0000 7fee5df65740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f6cdb7a1b0 con 0x55f6cdc156a0 2024-04-01T01:46:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.214+0000 7fee5df65740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f6cdb22c00 con 0x55f6cdc1b200 2024-04-01T01:46:41.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.214+0000 7fee5df65740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f6cdb76250 con 0x55f6cdc16090 2024-04-01T01:46:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.216+0000 7fee467fc640 1 -- 172.21.15.70:0/4068648295 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fee4c004600 con 0x7fee541097e0 2024-04-01T01:46:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.216+0000 7fee5911d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc16090 0x55f6cdc1ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.216+0000 7fee5891c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc156a0 0x55f6cdc15ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.216+0000 7fee5891c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc156a0 0x55f6cdc15ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39262/0 (socket says 172.21.15.70:39262) 2024-04-01T01:46:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.216+0000 7fee5891c640 1 -- 172.21.15.70:0/1803033111 learned_addr learned my addr 172.21.15.70:0/1803033111 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:41.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.216+0000 7fee5891c640 1 --2- 172.21.15.70:0/1803033111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 0x55f6cdc1d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.217+0000 7fee467fc640 1 --2- 172.21.15.70:0/4068648295 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee400a26a0 0x7fee400a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.217+0000 7fee5891c640 1 -- 172.21.15.70:0/1803033111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc16090 msgr2=0x55f6cdc1ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.217+0000 7fee5891c640 1 --2- 172.21.15.70:0/1803033111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc16090 0x55f6cdc1ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.217+0000 7fee5891c640 1 -- 172.21.15.70:0/1803033111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc156a0 msgr2=0x55f6cdc15ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.217+0000 7fee5891c640 1 --2- 172.21.15.70:0/1803033111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc156a0 0x55f6cdc15ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.217+0000 7fee5891c640 1 -- 172.21.15.70:0/1803033111 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6cdb423d0 con 0x55f6cdc1b200 2024-04-01T01:46:41.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.217+0000 7fee467fc640 1 -- 172.21.15.70:0/4068648295 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fee4c0f0b20 con 0x7fee541097e0 2024-04-01T01:46:41.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.217+0000 7fee5891c640 1 --2- 172.21.15.70:0/1803033111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 0x55f6cdc1d690 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fee50005fd0 tx=0x7fee50005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ade8863bc1e9ff46 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.218+0000 7fee3f7fe640 1 -- 172.21.15.70:0/1803033111 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee500142d0 con 0x55f6cdc1b200 2024-04-01T01:46:41.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.218+0000 7fee5991e640 1 --2- 172.21.15.70:0/4068648295 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee400a26a0 0x7fee400a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.218+0000 7fee3f7fe640 1 -- 172.21.15.70:0/1803033111 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee500117e0 con 0x55f6cdc1b200 2024-04-01T01:46:41.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.218+0000 7fee3f7fe640 1 -- 172.21.15.70:0/1803033111 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee50003850 con 0x55f6cdc1b200 2024-04-01T01:46:41.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.219+0000 7fee5991e640 1 --2- 172.21.15.70:0/4068648295 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee400a26a0 0x7fee400a4b60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0b9b80 tx=0x7fee4c02c100 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.219+0000 7fee5df65740 1 -- 172.21.15.70:0/1803033111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 msgr2=0x55f6cdc1d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.219+0000 7fee5df65740 1 --2- 172.21.15.70:0/1803033111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 0x55f6cdc1d690 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fee50005fd0 tx=0x7fee50005850 comp rx=0 tx=0).stop 2024-04-01T01:46:41.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.219+0000 7fee5df65740 1 -- 172.21.15.70:0/1803033111 shutdown_connections 2024-04-01T01:46:41.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.219+0000 7fee5df65740 1 --2- 172.21.15.70:0/1803033111 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 0x55f6cdc1d690 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.219+0000 7fee5df65740 1 --2- 172.21.15.70:0/1803033111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc16090 0x55f6cdc1ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.219+0000 7fee5df65740 1 --2- 172.21.15.70:0/1803033111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc156a0 0x55f6cdc15ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.219+0000 7fee5df65740 1 -- 172.21.15.70:0/1803033111 >> 172.21.15.70:0/1803033111 conn(0x55f6cdc10c10 msgr2=0x55f6cdc11010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.220+0000 7fee5df65740 1 -- 172.21.15.70:0/1803033111 shutdown_connections 2024-04-01T01:46:41.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.220+0000 7fee5df65740 1 -- 172.21.15.70:0/1803033111 wait complete. 2024-04-01T01:46:41.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.221+0000 7fee5df65740 1 -- start start 2024-04-01T01:46:41.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.221+0000 7fee5df65740 1 Processor -- start 2024-04-01T01:46:41.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.221+0000 7fee5df65740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc1fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.222+0000 7fee5df65740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc16090 0x55f6cdc201e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.222+0000 7fee5df65740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc1b200 0x55f6cdc271e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.222+0000 7fee5891c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc1fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.222+0000 7fee5891c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc1fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57844/0 (socket says 172.21.15.70:57844) 2024-04-01T01:46:41.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.222+0000 7fee5911d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc16090 0x55f6cdc201e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.222+0000 7fee5911d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc16090 0x55f6cdc201e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42932/0 (socket says 172.21.15.70:42932) 2024-04-01T01:46:41.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.222+0000 7fee5891c640 1 -- 172.21.15.70:0/2597198694 learned_addr learned my addr 172.21.15.70:0/2597198694 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:41.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.222+0000 7fee5df65740 1 -- 172.21.15.70:0/2597198694 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f6cdc1e1b0 con 0x55f6cdc1b200 2024-04-01T01:46:41.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.222+0000 7fee5df65740 1 -- 172.21.15.70:0/2597198694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f6cdb18470 con 0x55f6cdc16090 2024-04-01T01:46:41.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.222+0000 7fee5df65740 1 -- 172.21.15.70:0/2597198694 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f6cdc1e330 con 0x55f6cdc156a0 2024-04-01T01:46:41.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.223+0000 7fee5df65740 10 client.0 _ll_register_callbacks cb 0x55f6cdb35cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:41.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.223+0000 7fee5891c640 1 -- 172.21.15.70:0/2597198694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc16090 msgr2=0x55f6cdc201e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.223+0000 7fee5891c640 1 --2- 172.21.15.70:0/2597198694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc16090 0x55f6cdc201e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.223+0000 7fee5891c640 1 -- 172.21.15.70:0/2597198694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc1b200 msgr2=0x55f6cdc271e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.223+0000 7fee5891c640 1 --2- 172.21.15.70:0/2597198694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc1b200 0x55f6cdc271e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.223+0000 7fee5891c640 1 -- 172.21.15.70:0/2597198694 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6cd88e3f0 con 0x55f6cdc156a0 2024-04-01T01:46:41.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.223+0000 7fee3cff9640 10 client.29134 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:46:41.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.223+0000 7fee5891c640 1 --2- 172.21.15.70:0/2597198694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc1fca0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fee50014440 tx=0x7fee50005f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b4a58436eaa78d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.223+0000 7fee5df65740 1 -- 172.21.15.70:0/2597198694 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee500117e0 con 0x55f6cdc156a0 2024-04-01T01:46:41.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.223+0000 7fee3cff9640 1 -- 172.21.15.70:0/2597198694 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee50030050 con 0x55f6cdc156a0 2024-04-01T01:46:41.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.224+0000 7fee3cff9640 1 -- 172.21.15.70:0/2597198694 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fee50011ce0 con 0x55f6cdc156a0 2024-04-01T01:46:41.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.224+0000 7fee3cff9640 1 -- 172.21.15.70:0/2597198694 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fee50030050 con 0x55f6cdc156a0 2024-04-01T01:46:41.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.224+0000 7fee5df65740 10 client.29134 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:41.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.224+0000 7fee5df65740 1 -- 172.21.15.70:0/2597198694 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55f6cd9587d0 con 0x55f6cdc156a0 2024-04-01T01:46:41.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.225+0000 7fee3cff9640 1 -- 172.21.15.70:0/2597198694 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fee50005040 con 0x55f6cdc156a0 2024-04-01T01:46:41.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.225+0000 7fee5df65740 10 client.29134 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:41.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.225+0000 7fee5df65740 10 client.29134 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:46:41.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.225+0000 7fee5df65740 20 client.29134 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:41.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.225+0000 7fee5df65740 1 --2- 172.21.15.70:0/2597198694 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55f6cdc2b230 0x55f6cdc2d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.225+0000 7fee5df65740 4 client.29134 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:41.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.225+0000 7fee5df65740 1 -- 172.21.15.70:0/2597198694 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6cdb876f0 con 0x55f6cdc2b230 2024-04-01T01:46:41.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.226+0000 7fee5911d640 1 --2- 172.21.15.70:0/2597198694 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55f6cdc2b230 0x55f6cdc2d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.227+0000 7fee5911d640 1 --2- 172.21.15.70:0/2597198694 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55f6cdc2b230 0x55f6cdc2d6d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b3038cc0900ff5f5 server_cookie=1a2f3f06bee92989 in_seq=0 out_seq=0 2024-04-01T01:46:41.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.227+0000 7fee3cff9640 10 client.29134 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:46:41.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.228+0000 7fee3cff9640 1 -- 172.21.15.70:0/2597198694 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f6cdb876f0 con 0x55f6cdc2b230 2024-04-01T01:46:41.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.228+0000 7fee3cff9640 10 client.29134 handle_command_reply: tid=0 2024-04-01T01:46:41.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.228+0000 7fee5df65740 1 client.29134 shutdown 2024-04-01T01:46:41.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.236+0000 7fee5df65740 1 -- 172.21.15.70:0/2597198694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 msgr2=0x55f6cdc1fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.236+0000 7fee5df65740 1 --2- 172.21.15.70:0/2597198694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc1fca0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fee50014440 tx=0x7fee50005f30 comp rx=0 tx=0).stop 2024-04-01T01:46:41.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.236+0000 7fee5df65740 1 -- 172.21.15.70:0/2597198694 shutdown_connections 2024-04-01T01:46:41.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.236+0000 7fee5df65740 1 --2- 172.21.15.70:0/2597198694 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55f6cdc2b230 0x55f6cdc2d6d0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.237+0000 7fee3cff9640 0 client.29134 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:46:41.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.237+0000 7fee5df65740 1 --2- 172.21.15.70:0/2597198694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc1b200 0x55f6cdc271e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.237+0000 7fee5df65740 1 --2- 172.21.15.70:0/2597198694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc16090 0x55f6cdc201e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.237+0000 7fee5df65740 1 --2- 172.21.15.70:0/2597198694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc1fca0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.237+0000 7fee5df65740 1 -- 172.21.15.70:0/2597198694 >> 172.21.15.70:0/2597198694 conn(0x55f6cdc10c10 msgr2=0x55f6cdc0ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.238+0000 7fee5df65740 1 -- 172.21.15.70:0/2597198694 shutdown_connections 2024-04-01T01:46:41.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.239+0000 7fee5df65740 1 -- 172.21.15.70:0/2597198694 wait complete. 2024-04-01T01:46:41.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.239+0000 7fee5df65740 20 client.29134 trim_cache size 0 max 16384 2024-04-01T01:46:41.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.265+0000 7fee5df65740 1 Processor -- start 2024-04-01T01:46:41.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.265+0000 7fee5df65740 1 -- start start 2024-04-01T01:46:41.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.265+0000 7fee5df65740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc20140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.265+0000 7fee5df65740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc16090 0x55f6cdc270b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.265+0000 7fee5df65740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 0x55f6cdc0f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.265+0000 7fee5df65740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fee3800ff20 con 0x55f6cdc16090 2024-04-01T01:46:41.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.265+0000 7fee5df65740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fee38011a60 con 0x55f6cdc1b200 2024-04-01T01:46:41.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.265+0000 7fee5df65740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fee380105f0 con 0x55f6cdc156a0 2024-04-01T01:46:41.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.266+0000 7fee5891c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc20140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.266+0000 7fee5891c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 0x55f6cdc0f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.266+0000 7fee5891c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc20140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57854/0 (socket says 172.21.15.70:57854) 2024-04-01T01:46:41.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.266+0000 7fee5891c640 1 -- 172.21.15.70:0/3688461114 learned_addr learned my addr 172.21.15.70:0/3688461114 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:41.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.266+0000 7fee5891c640 1 -- 172.21.15.70:0/3688461114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 msgr2=0x55f6cdc0f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.266+0000 7fee5891c640 1 --2- 172.21.15.70:0/3688461114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 0x55f6cdc0f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.266+0000 7fee5891c640 1 -- 172.21.15.70:0/3688461114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc16090 msgr2=0x55f6cdc270b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.266+0000 7fee5891c640 1 --2- 172.21.15.70:0/3688461114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc16090 0x55f6cdc270b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.266+0000 7fee5891c640 1 -- 172.21.15.70:0/3688461114 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6cdc01e60 con 0x55f6cdc156a0 2024-04-01T01:46:41.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.267+0000 7fee5891c640 1 --2- 172.21.15.70:0/3688461114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc20140 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fee5001e590 tx=0x7fee50017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d98c1f59168f0a5d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.268+0000 7fee3effd640 1 -- 172.21.15.70:0/3688461114 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5003b070 con 0x55f6cdc156a0 2024-04-01T01:46:41.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.268+0000 7fee3effd640 1 -- 172.21.15.70:0/3688461114 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee500117e0 con 0x55f6cdc156a0 2024-04-01T01:46:41.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.269+0000 7fee3effd640 1 -- 172.21.15.70:0/3688461114 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5003e040 con 0x55f6cdc156a0 2024-04-01T01:46:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.269+0000 7fee5df65740 1 -- 172.21.15.70:0/3688461114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 msgr2=0x55f6cdc20140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.269+0000 7fee5df65740 1 --2- 172.21.15.70:0/3688461114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc20140 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fee5001e590 tx=0x7fee50017000 comp rx=0 tx=0).stop 2024-04-01T01:46:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.269+0000 7fee5df65740 1 -- 172.21.15.70:0/3688461114 shutdown_connections 2024-04-01T01:46:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.269+0000 7fee5df65740 1 --2- 172.21.15.70:0/3688461114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 0x55f6cdc0f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.269+0000 7fee5df65740 1 --2- 172.21.15.70:0/3688461114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc16090 0x55f6cdc270b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.269+0000 7fee5df65740 1 --2- 172.21.15.70:0/3688461114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc20140 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.269+0000 7fee5df65740 1 -- 172.21.15.70:0/3688461114 >> 172.21.15.70:0/3688461114 conn(0x55f6cdc10c10 msgr2=0x55f6cdc1fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.269+0000 7fee5df65740 1 -- 172.21.15.70:0/3688461114 shutdown_connections 2024-04-01T01:46:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.269+0000 7fee5df65740 1 -- 172.21.15.70:0/3688461114 wait complete. 2024-04-01T01:46:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.270+0000 7fee5df65740 1 -- start start 2024-04-01T01:46:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.270+0000 7fee5df65740 1 Processor -- start 2024-04-01T01:46:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.270+0000 7fee5df65740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc249b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.270+0000 7fee5df65740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc16090 0x55f6cdc24ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.270+0000 7fee5df65740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 0x55f6cdc25eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.270+0000 7fee5df65740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fee38011390 con 0x55f6cdc16090 2024-04-01T01:46:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.270+0000 7fee5df65740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fee38010cc0 con 0x55f6cdc1b200 2024-04-01T01:46:41.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.270+0000 7fee5df65740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fee3800dea0 con 0x55f6cdc156a0 2024-04-01T01:46:41.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.270+0000 7fee5df65740 10 client.0 _ll_register_callbacks cb 0x7fee500148a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:41.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.270+0000 7fee5891c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.271+0000 7fee5891c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc249b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57868/0 (socket says 172.21.15.70:57868) 2024-04-01T01:46:41.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.271+0000 7fee5891c640 1 -- 172.21.15.70:0/2592077592 learned_addr learned my addr 172.21.15.70:0/2592077592 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:41.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.271+0000 7fee5891c640 1 --2- 172.21.15.70:0/2592077592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 0x55f6cdc25eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.271+0000 7fee5911d640 1 --2- 172.21.15.70:0/2592077592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc16090 0x55f6cdc24ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.271+0000 7fee5891c640 1 -- 172.21.15.70:0/2592077592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 msgr2=0x55f6cdc249b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.271+0000 7fee5891c640 1 --2- 172.21.15.70:0/2592077592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc249b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.271+0000 7fee5891c640 1 -- 172.21.15.70:0/2592077592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc16090 msgr2=0x55f6cdc24ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.271+0000 7fee5891c640 1 --2- 172.21.15.70:0/2592077592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc16090 0x55f6cdc24ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.271+0000 7fee5891c640 1 -- 172.21.15.70:0/2592077592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6cdc264a0 con 0x55f6cdc1b200 2024-04-01T01:46:41.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.271+0000 7fee5911d640 1 --2- 172.21.15.70:0/2592077592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc16090 0x55f6cdc24ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:46:41.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.271+0000 7fee5891c640 1 --2- 172.21.15.70:0/2592077592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 0x55f6cdc25eb0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fee5001e4c0 tx=0x7fee5000c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8c1fcb0ec650cfe server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.271+0000 7fee3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:41.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.272+0000 7fee3cff9640 1 -- 172.21.15.70:0/2592077592 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5003b070 con 0x55f6cdc1b200 2024-04-01T01:46:41.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.272+0000 7fee5df65740 1 -- 172.21.15.70:0/2592077592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee500117e0 con 0x55f6cdc1b200 2024-04-01T01:46:41.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.272+0000 7fee3cff9640 1 -- 172.21.15.70:0/2592077592 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fee50005560 con 0x55f6cdc1b200 2024-04-01T01:46:41.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.272+0000 7fee3cff9640 1 -- 172.21.15.70:0/2592077592 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fee50030050 con 0x55f6cdc1b200 2024-04-01T01:46:41.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.272+0000 7fee5df65740 10 client.29363 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:41.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.272+0000 7fee5df65740 1 -- 172.21.15.70:0/2592077592 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f6cdc917d0 con 0x55f6cdc1b200 2024-04-01T01:46:41.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.273+0000 7fee3cff9640 1 -- 172.21.15.70:0/2592077592 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee500117e0 con 0x55f6cdc1b200 2024-04-01T01:46:41.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.273+0000 7fee3cff9640 1 -- 172.21.15.70:0/2592077592 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fee50044030 con 0x55f6cdc1b200 2024-04-01T01:46:41.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.273+0000 7fee5df65740 10 client.29363 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:41.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.273+0000 7fee5df65740 10 client.29363 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:46:41.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.273+0000 7fee5df65740 20 client.29363 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:41.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.273+0000 7fee5df65740 1 --2- 172.21.15.70:0/2592077592 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55f6cdc2b230 0x55f6cdc94030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.273+0000 7fee5df65740 4 client.29363 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:46:41.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.273+0000 7fee5df65740 1 -- 172.21.15.70:0/2592077592 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f6cdb81a40 con 0x55f6cdc2b230 2024-04-01T01:46:41.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.273+0000 7fee5911d640 1 --2- 172.21.15.70:0/2592077592 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55f6cdc2b230 0x55f6cdc94030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.274+0000 7fee5911d640 1 --2- 172.21.15.70:0/2592077592 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55f6cdc2b230 0x55f6cdc94030 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b85162eeed29b057 server_cookie=228af98819dd600b in_seq=0 out_seq=0 2024-04-01T01:46:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.274+0000 7fee3cff9640 10 client.29363 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:46:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.274+0000 7fee3cff9640 1 -- 172.21.15.70:0/2592077592 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f6cdb81a40 con 0x55f6cdc2b230 2024-04-01T01:46:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.274+0000 7fee3cff9640 10 client.29363 handle_command_reply: tid=0 2024-04-01T01:46:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.274+0000 7fee5df65740 1 client.29363 shutdown 2024-04-01T01:46:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.275+0000 7fee5df65740 1 -- 172.21.15.70:0/2592077592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 msgr2=0x55f6cdc25eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.275+0000 7fee5df65740 1 --2- 172.21.15.70:0/2592077592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 0x55f6cdc25eb0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fee5001e4c0 tx=0x7fee5000c4d0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.275+0000 7fee5df65740 1 -- 172.21.15.70:0/2592077592 shutdown_connections 2024-04-01T01:46:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.275+0000 7fee5df65740 1 --2- 172.21.15.70:0/2592077592 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55f6cdc2b230 0x55f6cdc94030 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.275+0000 7fee5df65740 1 --2- 172.21.15.70:0/2592077592 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f6cdc1b200 0x55f6cdc25eb0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.275+0000 7fee5df65740 1 --2- 172.21.15.70:0/2592077592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6cdc16090 0x55f6cdc24ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.275+0000 7fee5df65740 1 --2- 172.21.15.70:0/2592077592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f6cdc156a0 0x55f6cdc249b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.275+0000 7fee5df65740 1 -- 172.21.15.70:0/2592077592 >> 172.21.15.70:0/2592077592 conn(0x55f6cdc10c10 msgr2=0x55f6cdc13e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.275+0000 7fee3cff9640 0 client.29363 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:46:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.275+0000 7fee5df65740 1 -- 172.21.15.70:0/2592077592 shutdown_connections 2024-04-01T01:46:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.275+0000 7fee5df65740 1 -- 172.21.15.70:0/2592077592 wait complete. 2024-04-01T01:46:41.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.275+0000 7fee5df65740 20 client.29363 trim_cache size 0 max 16384 2024-04-01T01:46:41.277 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:46:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.276+0000 7fee5a11f640 1 -- 172.21.15.70:0/4068648295 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee400a26a0 msgr2=0x7fee400a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.276+0000 7fee5a11f640 1 --2- 172.21.15.70:0/4068648295 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee400a26a0 0x7fee400a4b60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0b9b80 tx=0x7fee4c02c100 comp rx=0 tx=0).stop 2024-04-01T01:46:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.276+0000 7fee5a11f640 1 -- 172.21.15.70:0/4068648295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee541097e0 msgr2=0x7fee541b66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.276+0000 7fee5a11f640 1 --2- 172.21.15.70:0/4068648295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee541097e0 0x7fee541b66e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0098d0 tx=0x7fee4c035ce0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.276+0000 7fee5a11f640 1 -- 172.21.15.70:0/4068648295 shutdown_connections 2024-04-01T01:46:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.276+0000 7fee5a11f640 1 --2- 172.21.15.70:0/4068648295 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fee400a26a0 0x7fee400a4b60 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.276+0000 7fee5a11f640 1 --2- 172.21.15.70:0/4068648295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee541097e0 0x7fee541b66e0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.276+0000 7fee5a11f640 1 --2- 172.21.15.70:0/4068648295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee54108e20 0x7fee541153a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.276+0000 7fee5a11f640 1 --2- 172.21.15.70:0/4068648295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee54072b60 0x7fee54114e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.276+0000 7fee5a11f640 1 -- 172.21.15.70:0/4068648295 >> 172.21.15.70:0/4068648295 conn(0x7fee5406e630 msgr2=0x7fee54070260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.277+0000 7fee5a11f640 1 -- 172.21.15.70:0/4068648295 shutdown_connections 2024-04-01T01:46:41.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.277+0000 7fee5a11f640 1 -- 172.21.15.70:0/4068648295 wait complete. 2024-04-01T01:46:41.296 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:46:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.500+0000 7faba31c7640 1 -- 172.21.15.70:0/3750723633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab9c108f60 msgr2=0x7fab9c1093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.500+0000 7faba31c7640 1 --2- 172.21.15.70:0/3750723633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab9c108f60 0x7fab9c1093e0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fab90009900 tx=0x7fab9002bc30 comp rx=0 tx=0).stop 2024-04-01T01:46:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.500+0000 7faba31c7640 1 -- 172.21.15.70:0/3750723633 shutdown_connections 2024-04-01T01:46:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.500+0000 7faba31c7640 1 --2- 172.21.15.70:0/3750723633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab9c109920 0x7fab9c1101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.500+0000 7faba31c7640 1 --2- 172.21.15.70:0/3750723633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab9c108f60 0x7fab9c1093e0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.500+0000 7faba31c7640 1 --2- 172.21.15.70:0/3750723633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c107d70 0x7fab9c108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.500+0000 7faba31c7640 1 -- 172.21.15.70:0/3750723633 >> 172.21.15.70:0/3750723633 conn(0x7fab9c076d10 msgr2=0x7fab9c079150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.500+0000 7faba31c7640 1 -- 172.21.15.70:0/3750723633 shutdown_connections 2024-04-01T01:46:41.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.500+0000 7faba31c7640 1 -- 172.21.15.70:0/3750723633 wait complete. 2024-04-01T01:46:41.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.501+0000 7faba31c7640 1 Processor -- start 2024-04-01T01:46:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.501+0000 7faba31c7640 1 -- start start 2024-04-01T01:46:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.501+0000 7faba31c7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c107d70 0x7fab9c0819f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.501+0000 7faba31c7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab9c108f60 0x7fab9c081f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.502+0000 7faba31c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab9c109920 0x7fab9c080030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.502+0000 7faba31c7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c07a8a0 con 0x7fab9c107d70 2024-04-01T01:46:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.502+0000 7faba31c7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c07a720 con 0x7fab9c109920 2024-04-01T01:46:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.502+0000 7faba0f63640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c107d70 0x7fab9c0819f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.502+0000 7faba1764640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab9c109920 0x7fab9c080030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.502+0000 7faba1764640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab9c109920 0x7fab9c080030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42958/0 (socket says 172.21.15.70:42958) 2024-04-01T01:46:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.502+0000 7faba1764640 1 -- 172.21.15.70:0/236710032 learned_addr learned my addr 172.21.15.70:0/236710032 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:41.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.502+0000 7faba31c7640 1 -- 172.21.15.70:0/236710032 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c07aa20 con 0x7fab9c108f60 2024-04-01T01:46:41.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.503+0000 7fab9bfff640 1 --2- 172.21.15.70:0/236710032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab9c108f60 0x7fab9c081f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.505+0000 7faba1764640 1 -- 172.21.15.70:0/236710032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab9c108f60 msgr2=0x7fab9c081f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.505+0000 7faba1764640 1 --2- 172.21.15.70:0/236710032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab9c108f60 0x7fab9c081f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.505+0000 7faba1764640 1 -- 172.21.15.70:0/236710032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c107d70 msgr2=0x7fab9c0819f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.505+0000 7faba1764640 1 --2- 172.21.15.70:0/236710032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c107d70 0x7fab9c0819f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.505+0000 7faba1764640 1 -- 172.21.15.70:0/236710032 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9c0806d0 con 0x7fab9c109920 2024-04-01T01:46:41.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.506+0000 7faba1764640 1 --2- 172.21.15.70:0/236710032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab9c109920 0x7fab9c080030 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fab9400eb10 tx=0x7fab9400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.507+0000 7fab99ffb640 1 -- 172.21.15.70:0/236710032 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94004b20 con 0x7fab9c109920 2024-04-01T01:46:41.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.507+0000 7fab99ffb640 1 -- 172.21.15.70:0/236710032 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab94004cc0 con 0x7fab9c109920 2024-04-01T01:46:41.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.507+0000 7faba31c7640 1 -- 172.21.15.70:0/236710032 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9c0809c0 con 0x7fab9c109920 2024-04-01T01:46:41.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.507+0000 7fab99ffb640 1 -- 172.21.15.70:0/236710032 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94010720 con 0x7fab9c109920 2024-04-01T01:46:41.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.507+0000 7faba31c7640 1 -- 172.21.15.70:0/236710032 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab9c1aed20 con 0x7fab9c109920 2024-04-01T01:46:41.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.510+0000 7faba5da5740 1 Processor -- start 2024-04-01T01:46:41.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.511+0000 7faba5da5740 1 -- start start 2024-04-01T01:46:41.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.511+0000 7faba5da5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed0124b490 0x55ed0124b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.511+0000 7faba5da5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0124be80 0x55ed01250980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.511+0000 7faba5da5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed01250ff0 0x55ed01253480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.511+0000 7faba5da5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ed011b01b0 con 0x55ed0124be80 2024-04-01T01:46:41.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.511+0000 7faba5da5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ed01158c00 con 0x55ed0124b490 2024-04-01T01:46:41.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.511+0000 7faba5da5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ed011ac250 con 0x55ed01250ff0 2024-04-01T01:46:41.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.512+0000 7fab9bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed0124b490 0x55ed0124b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.512+0000 7fab9bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed01250ff0 0x55ed01253480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.512+0000 7fab9bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed0124b490 0x55ed0124b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42960/0 (socket says 172.21.15.70:42960) 2024-04-01T01:46:41.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.512+0000 7fab9bfff640 1 -- 172.21.15.70:0/2065341481 learned_addr learned my addr 172.21.15.70:0/2065341481 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:41.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.512+0000 7fab9bfff640 1 -- 172.21.15.70:0/2065341481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed01250ff0 msgr2=0x55ed01253480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.512+0000 7fab9bfff640 1 --2- 172.21.15.70:0/2065341481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed01250ff0 0x55ed01253480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.512+0000 7fab9bfff640 1 -- 172.21.15.70:0/2065341481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0124be80 msgr2=0x55ed01250980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:46:41.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.512+0000 7fab9bfff640 1 --2- 172.21.15.70:0/2065341481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0124be80 0x55ed01250980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.512+0000 7fab9bfff640 1 -- 172.21.15.70:0/2065341481 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed011783d0 con 0x55ed0124b490 2024-04-01T01:46:41.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.513+0000 7fab9bfff640 1 --2- 172.21.15.70:0/2065341481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed0124b490 0x55ed0124b8b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fab9002c840 tx=0x7fab90000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab290f5a7cbc7c6b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.513+0000 7fab82ffd640 1 -- 172.21.15.70:0/2065341481 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab90049070 con 0x55ed0124b490 2024-04-01T01:46:41.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.513+0000 7fab82ffd640 1 -- 172.21.15.70:0/2065341481 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab90005050 con 0x55ed0124b490 2024-04-01T01:46:41.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.513+0000 7fab82ffd640 1 -- 172.21.15.70:0/2065341481 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab900044e0 con 0x55ed0124b490 2024-04-01T01:46:41.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.514+0000 7faba5da5740 1 -- 172.21.15.70:0/2065341481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed0124b490 msgr2=0x55ed0124b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.514+0000 7faba5da5740 1 --2- 172.21.15.70:0/2065341481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed0124b490 0x55ed0124b8b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fab9002c840 tx=0x7fab90000bc0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.515+0000 7faba5da5740 1 -- 172.21.15.70:0/2065341481 shutdown_connections 2024-04-01T01:46:41.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.515+0000 7fab99ffb640 1 -- 172.21.15.70:0/236710032 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fab94005020 con 0x7fab9c109920 2024-04-01T01:46:41.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.515+0000 7faba5da5740 1 --2- 172.21.15.70:0/2065341481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed01250ff0 0x55ed01253480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.515+0000 7faba5da5740 1 --2- 172.21.15.70:0/2065341481 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0124be80 0x55ed01250980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.515+0000 7faba5da5740 1 --2- 172.21.15.70:0/2065341481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed0124b490 0x55ed0124b8b0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.515+0000 7faba5da5740 1 -- 172.21.15.70:0/2065341481 >> 172.21.15.70:0/2065341481 conn(0x55ed01246a00 msgr2=0x55ed01246e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.516+0000 7fab99ffb640 1 --2- 172.21.15.70:0/236710032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fab840a26a0 0x7fab840a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.516+0000 7fab99ffb640 1 -- 172.21.15.70:0/236710032 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fab940cf000 con 0x7fab9c109920 2024-04-01T01:46:41.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.518+0000 7faba0f63640 1 --2- 172.21.15.70:0/236710032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fab840a26a0 0x7fab840a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.518+0000 7faba5da5740 1 -- 172.21.15.70:0/2065341481 shutdown_connections 2024-04-01T01:46:41.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.518+0000 7faba5da5740 1 -- 172.21.15.70:0/2065341481 wait complete. 2024-04-01T01:46:41.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.519+0000 7faba0f63640 1 --2- 172.21.15.70:0/236710032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fab840a26a0 0x7fab840a4b60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x55ed01247970 tx=0x7fab8c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.519+0000 7faba5da5740 1 -- start start 2024-04-01T01:46:41.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.520+0000 7faba5da5740 1 Processor -- start 2024-04-01T01:46:41.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.520+0000 7faba5da5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0124b490 0x55ed01255a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.521+0000 7faba5da5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0124be80 0x55ed01255f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.521+0000 7faba5da5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed01250ff0 0x55ed0125cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.521+0000 7faba5da5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ed01253f40 con 0x55ed0124be80 2024-04-01T01:46:41.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.521+0000 7faba5da5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ed0114e470 con 0x55ed01250ff0 2024-04-01T01:46:41.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.521+0000 7faba5da5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ed012540c0 con 0x55ed0124b490 2024-04-01T01:46:41.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.521+0000 7faba5da5740 10 client.0 _ll_register_callbacks cb 0x55ed0116bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:41.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.521+0000 7fab9bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0124be80 0x55ed01255f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.521+0000 7fab9bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0124be80 0x55ed01255f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39308/0 (socket says 172.21.15.70:39308) 2024-04-01T01:46:41.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.521+0000 7fab9bfff640 1 -- 172.21.15.70:0/3621488216 learned_addr learned my addr 172.21.15.70:0/3621488216 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:41.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.521+0000 7faba1764640 1 --2- 172.21.15.70:0/3621488216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed01250ff0 0x55ed0125cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.522+0000 7fab9bfff640 1 -- 172.21.15.70:0/3621488216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0124b490 msgr2=0x55ed01255a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.522+0000 7fab9bfff640 1 --2- 172.21.15.70:0/3621488216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0124b490 0x55ed01255a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.522+0000 7fab9bfff640 1 -- 172.21.15.70:0/3621488216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed01250ff0 msgr2=0x55ed0125cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.522+0000 7fab9bfff640 1 --2- 172.21.15.70:0/3621488216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed01250ff0 0x55ed0125cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.523+0000 7fab9bfff640 1 -- 172.21.15.70:0/3621488216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed00ec43f0 con 0x55ed0124be80 2024-04-01T01:46:41.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.523+0000 7faba1764640 1 --2- 172.21.15.70:0/3621488216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed01250ff0 0x55ed0125cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:46:41.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.523+0000 7fab9bfff640 1 --2- 172.21.15.70:0/3621488216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0124be80 0x55ed01255f40 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fab90005ab0 tx=0x7fab90004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6dba0a4deb3f984 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.524+0000 7fab7bfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:46:41.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.524+0000 7fab7bfff640 1 -- 172.21.15.70:0/3621488216 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab90051020 con 0x55ed0124be80 2024-04-01T01:46:41.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.524+0000 7fab7bfff640 1 -- 172.21.15.70:0/3621488216 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fab900046c0 con 0x55ed0124be80 2024-04-01T01:46:41.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.524+0000 7faba5da5740 1 -- 172.21.15.70:0/3621488216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab90005050 con 0x55ed0124be80 2024-04-01T01:46:41.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.524+0000 7fab7bfff640 1 -- 172.21.15.70:0/3621488216 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fab90049040 con 0x55ed0124be80 2024-04-01T01:46:41.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.524+0000 7faba5da5740 10 client.19293 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:41.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.524+0000 7faba5da5740 1 -- 172.21.15.70:0/3621488216 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ed00f8e7d0 con 0x55ed0124be80 2024-04-01T01:46:41.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.525+0000 7fab7bfff640 1 -- 172.21.15.70:0/3621488216 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab90005050 con 0x55ed0124be80 2024-04-01T01:46:41.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.525+0000 7fab7bfff640 1 -- 172.21.15.70:0/3621488216 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fab9004c030 con 0x55ed0124be80 2024-04-01T01:46:41.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.525+0000 7faba5da5740 10 client.19293 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:41.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.525+0000 7faba5da5740 10 client.19293 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:46:41.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.525+0000 7faba5da5740 20 client.19293 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:41.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.525+0000 7faba5da5740 1 --2- 172.21.15.70:0/3621488216 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55ed012610e0 0x55ed012634d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.525+0000 7faba5da5740 4 client.19293 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:41.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.525+0000 7faba5da5740 1 -- 172.21.15.70:0/3621488216 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed011bd6f0 con 0x55ed012610e0 2024-04-01T01:46:41.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.526+0000 7faba0f63640 1 --2- 172.21.15.70:0/3621488216 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55ed012610e0 0x55ed012634d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.527+0000 7faba0f63640 1 --2- 172.21.15.70:0/3621488216 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55ed012610e0 0x55ed012634d0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=162c1c977b6222b0 server_cookie=9f74a779496e04ba in_seq=0 out_seq=0 2024-04-01T01:46:41.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.527+0000 7fab7bfff640 10 client.19293 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:46:41.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.529+0000 7fab7bfff640 1 -- 172.21.15.70:0/3621488216 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ed011bd6f0 con 0x55ed012610e0 2024-04-01T01:46:41.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.529+0000 7fab7bfff640 10 client.19293 handle_command_reply: tid=0 2024-04-01T01:46:41.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.529+0000 7faba5da5740 1 client.19293 shutdown 2024-04-01T01:46:41.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.529+0000 7faba5da5740 1 -- 172.21.15.70:0/3621488216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0124be80 msgr2=0x55ed01255f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.529+0000 7faba5da5740 1 --2- 172.21.15.70:0/3621488216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0124be80 0x55ed01255f40 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fab90005ab0 tx=0x7fab90004270 comp rx=0 tx=0).stop 2024-04-01T01:46:41.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.529+0000 7faba5da5740 1 -- 172.21.15.70:0/3621488216 shutdown_connections 2024-04-01T01:46:41.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.529+0000 7faba5da5740 1 --2- 172.21.15.70:0/3621488216 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55ed012610e0 0x55ed012634d0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.529+0000 7faba5da5740 1 --2- 172.21.15.70:0/3621488216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed01250ff0 0x55ed0125cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.529+0000 7faba5da5740 1 --2- 172.21.15.70:0/3621488216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0124be80 0x55ed01255f40 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.529+0000 7faba5da5740 1 --2- 172.21.15.70:0/3621488216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0124b490 0x55ed01255a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.529+0000 7faba5da5740 1 -- 172.21.15.70:0/3621488216 >> 172.21.15.70:0/3621488216 conn(0x55ed01246a00 msgr2=0x55ed01245d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.529+0000 7fab7bfff640 0 client.19293 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:46:41.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.530+0000 7faba5da5740 1 -- 172.21.15.70:0/3621488216 shutdown_connections 2024-04-01T01:46:41.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.530+0000 7faba5da5740 1 -- 172.21.15.70:0/3621488216 wait complete. 2024-04-01T01:46:41.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.530+0000 7faba5da5740 20 client.19293 trim_cache size 0 max 16384 2024-04-01T01:46:41.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.555+0000 7faba5da5740 1 Processor -- start 2024-04-01T01:46:41.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.555+0000 7faba5da5740 1 -- start start 2024-04-01T01:46:41.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.556+0000 7faba5da5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0124b490 0x55ed012544f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.556+0000 7faba5da5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0124be80 0x55ed0125cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.556+0000 7faba5da5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed01250ff0 0x55ed01254a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.556+0000 7faba5da5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab8800fdf0 con 0x55ed0124be80 2024-04-01T01:46:41.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.556+0000 7faba5da5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fab88011930 con 0x55ed01250ff0 2024-04-01T01:46:41.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.556+0000 7faba5da5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fab880104c0 con 0x55ed0124b490 2024-04-01T01:46:41.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.556+0000 7faba1764640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed01250ff0 0x55ed01254a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.556+0000 7faba1764640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed01250ff0 0x55ed01254a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:42972/0 (socket says 172.21.15.70:42972) 2024-04-01T01:46:41.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.556+0000 7faba1764640 1 -- 172.21.15.70:0/2644311537 learned_addr learned my addr 172.21.15.70:0/2644311537 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:41.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.556+0000 7fab9bfff640 1 --2- 172.21.15.70:0/2644311537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0124be80 0x55ed0125cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.556+0000 7faba0f63640 1 --2- 172.21.15.70:0/2644311537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0124b490 0x55ed012544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.556+0000 7faba1764640 1 -- 172.21.15.70:0/2644311537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0124b490 msgr2=0x55ed012544f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.556+0000 7faba1764640 1 --2- 172.21.15.70:0/2644311537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0124b490 0x55ed012544f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.557+0000 7faba1764640 1 -- 172.21.15.70:0/2644311537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0124be80 msgr2=0x55ed0125cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.557+0000 7faba1764640 1 --2- 172.21.15.70:0/2644311537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0124be80 0x55ed0125cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.557+0000 7faba1764640 1 -- 172.21.15.70:0/2644311537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed012466a0 con 0x55ed01250ff0 2024-04-01T01:46:41.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.557+0000 7faba1764640 1 --2- 172.21.15.70:0/2644311537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed01250ff0 0x55ed01254a30 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fab94004af0 tx=0x7fab9409b520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e2a0a763b841789 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.558+0000 7fab827fc640 1 -- 172.21.15.70:0/2644311537 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab940b9070 con 0x55ed01250ff0 2024-04-01T01:46:41.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.558+0000 7fab827fc640 1 -- 172.21.15.70:0/2644311537 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab9409b690 con 0x55ed01250ff0 2024-04-01T01:46:41.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.558+0000 7fab827fc640 1 -- 172.21.15.70:0/2644311537 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab940b3030 con 0x55ed01250ff0 2024-04-01T01:46:41.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.559+0000 7faba5da5740 1 -- 172.21.15.70:0/2644311537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed01250ff0 msgr2=0x55ed01254a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.559+0000 7faba5da5740 1 --2- 172.21.15.70:0/2644311537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed01250ff0 0x55ed01254a30 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fab94004af0 tx=0x7fab9409b520 comp rx=0 tx=0).stop 2024-04-01T01:46:41.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.559+0000 7faba5da5740 1 -- 172.21.15.70:0/2644311537 shutdown_connections 2024-04-01T01:46:41.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.559+0000 7faba5da5740 1 --2- 172.21.15.70:0/2644311537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed01250ff0 0x55ed01254a30 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.559+0000 7faba5da5740 1 --2- 172.21.15.70:0/2644311537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed0124be80 0x55ed0125cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.559+0000 7faba5da5740 1 --2- 172.21.15.70:0/2644311537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0124b490 0x55ed012544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.559+0000 7faba5da5740 1 -- 172.21.15.70:0/2644311537 >> 172.21.15.70:0/2644311537 conn(0x55ed01246a00 msgr2=0x55ed01245d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.563+0000 7faba5da5740 1 -- 172.21.15.70:0/2644311537 shutdown_connections 2024-04-01T01:46:41.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.567+0000 7faba5da5740 1 -- 172.21.15.70:0/2644311537 wait complete. 2024-04-01T01:46:41.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.567+0000 7faba5da5740 1 -- start start 2024-04-01T01:46:41.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.567+0000 7faba5da5740 1 Processor -- start 2024-04-01T01:46:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.568+0000 7faba5da5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0124b490 0x55ed0125a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.568+0000 7faba5da5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed0124be80 0x55ed0125acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.568+0000 7faba5da5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed01250ff0 0x55ed0125bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.568+0000 7faba5da5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab88011260 con 0x55ed01250ff0 2024-04-01T01:46:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.568+0000 7faba5da5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fab88010b90 con 0x55ed0124be80 2024-04-01T01:46:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.568+0000 7faba5da5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fab8800ddd0 con 0x55ed0124b490 2024-04-01T01:46:41.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.568+0000 7faba5da5740 10 client.0 _ll_register_callbacks cb 0x7fab90009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:41.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.569+0000 7faba1764640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed01250ff0 0x55ed0125bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.569+0000 7faba1764640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed01250ff0 0x55ed0125bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39326/0 (socket says 172.21.15.70:39326) 2024-04-01T01:46:41.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.569+0000 7faba1764640 1 -- 172.21.15.70:0/4221686073 learned_addr learned my addr 172.21.15.70:0/4221686073 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:41.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.569+0000 7faba1764640 1 -- 172.21.15.70:0/4221686073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0124b490 msgr2=0x55ed0125a790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:46:41.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.569+0000 7faba1764640 1 --2- 172.21.15.70:0/4221686073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0124b490 0x55ed0125a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.569+0000 7faba1764640 1 -- 172.21.15.70:0/4221686073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed0124be80 msgr2=0x55ed0125acd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:41.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.569+0000 7faba1764640 1 --2- 172.21.15.70:0/4221686073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed0124be80 0x55ed0125acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.569+0000 7faba1764640 1 -- 172.21.15.70:0/4221686073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed0125c260 con 0x55ed01250ff0 2024-04-01T01:46:41.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.570+0000 7faba1764640 1 --2- 172.21.15.70:0/4221686073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed01250ff0 0x55ed0125bc70 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fab94098060 tx=0x7fab9409b590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9741bde55d2fbb89 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.570+0000 7fab7bfff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:46:41.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.570+0000 7fab7bfff640 1 -- 172.21.15.70:0/4221686073 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab940b3070 con 0x55ed01250ff0 2024-04-01T01:46:41.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.570+0000 7fab7bfff640 1 -- 172.21.15.70:0/4221686073 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fab940b42a0 con 0x55ed01250ff0 2024-04-01T01:46:41.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.571+0000 7faba5da5740 1 -- 172.21.15.70:0/4221686073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab9409b690 con 0x55ed01250ff0 2024-04-01T01:46:41.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.571+0000 7fab7bfff640 1 -- 172.21.15.70:0/4221686073 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fab940b9040 con 0x55ed01250ff0 2024-04-01T01:46:41.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.571+0000 7faba5da5740 10 client.19299 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:41.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.571+0000 7faba5da5740 1 -- 172.21.15.70:0/4221686073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ed012c73a0 con 0x55ed01250ff0 2024-04-01T01:46:41.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.572+0000 7fab7bfff640 1 -- 172.21.15.70:0/4221686073 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab9409b690 con 0x55ed01250ff0 2024-04-01T01:46:41.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.573+0000 7fab7bfff640 1 -- 172.21.15.70:0/4221686073 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fab940a5020 con 0x55ed01250ff0 2024-04-01T01:46:41.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.573+0000 7faba5da5740 10 client.19299 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:41.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.573+0000 7faba5da5740 10 client.19299 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:46:41.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.573+0000 7faba5da5740 20 client.19299 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:41.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.573+0000 7faba5da5740 1 --2- 172.21.15.70:0/4221686073 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55ed012c7bd0 0x55ed012c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.573+0000 7faba5da5740 4 client.19299 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:46:41.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.573+0000 7faba5da5740 1 -- 172.21.15.70:0/4221686073 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ed011b7a40 con 0x55ed012c7bd0 2024-04-01T01:46:41.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.573+0000 7faba0f63640 1 --2- 172.21.15.70:0/4221686073 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55ed012c7bd0 0x55ed012c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.574+0000 7faba0f63640 1 --2- 172.21.15.70:0/4221686073 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55ed012c7bd0 0x55ed012c9ff0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3003434a8c614795 server_cookie=dd4142a759fc6e5c in_seq=0 out_seq=0 2024-04-01T01:46:41.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.574+0000 7fab7bfff640 10 client.19299 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:46:41.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.574+0000 7fab7bfff640 1 -- 172.21.15.70:0/4221686073 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ed011b7a40 con 0x55ed012c7bd0 2024-04-01T01:46:41.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.574+0000 7fab7bfff640 10 client.19299 handle_command_reply: tid=0 2024-04-01T01:46:41.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.575+0000 7faba5da5740 1 client.19299 shutdown 2024-04-01T01:46:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.575+0000 7faba5da5740 1 -- 172.21.15.70:0/4221686073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed01250ff0 msgr2=0x55ed0125bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.575+0000 7faba5da5740 1 --2- 172.21.15.70:0/4221686073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed01250ff0 0x55ed0125bc70 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fab94098060 tx=0x7fab9409b590 comp rx=0 tx=0).stop 2024-04-01T01:46:41.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.576+0000 7faba5da5740 1 -- 172.21.15.70:0/4221686073 shutdown_connections 2024-04-01T01:46:41.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.576+0000 7faba5da5740 1 --2- 172.21.15.70:0/4221686073 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55ed012c7bd0 0x55ed012c9ff0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.576+0000 7faba5da5740 1 --2- 172.21.15.70:0/4221686073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ed01250ff0 0x55ed0125bc70 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.576+0000 7faba5da5740 1 --2- 172.21.15.70:0/4221686073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ed0124be80 0x55ed0125acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.576+0000 7faba5da5740 1 --2- 172.21.15.70:0/4221686073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0124b490 0x55ed0125a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.576+0000 7faba5da5740 1 -- 172.21.15.70:0/4221686073 >> 172.21.15.70:0/4221686073 conn(0x55ed01246a00 msgr2=0x55ed01249c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.576+0000 7fab7bfff640 0 client.19299 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:46:41.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.579+0000 7faba5da5740 1 -- 172.21.15.70:0/4221686073 shutdown_connections 2024-04-01T01:46:41.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.579+0000 7faba5da5740 1 -- 172.21.15.70:0/4221686073 wait complete. 2024-04-01T01:46:41.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.579+0000 7faba5da5740 20 client.19299 trim_cache size 0 max 16384 2024-04-01T01:46:41.579 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:46:41.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.580+0000 7faba31c7640 1 -- 172.21.15.70:0/236710032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fab840a26a0 msgr2=0x7fab840a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.580+0000 7faba31c7640 1 --2- 172.21.15.70:0/236710032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fab840a26a0 0x7fab840a4b60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x55ed01247970 tx=0x7fab8c005f90 comp rx=0 tx=0).stop 2024-04-01T01:46:41.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.580+0000 7faba31c7640 1 -- 172.21.15.70:0/236710032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab9c109920 msgr2=0x7fab9c080030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.580+0000 7faba31c7640 1 --2- 172.21.15.70:0/236710032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab9c109920 0x7fab9c080030 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fab9400eb10 tx=0x7fab9400c2d0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.580+0000 7faba31c7640 1 -- 172.21.15.70:0/236710032 shutdown_connections 2024-04-01T01:46:41.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.580+0000 7faba31c7640 1 --2- 172.21.15.70:0/236710032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab9c109920 0x7fab9c080030 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.580+0000 7faba31c7640 1 --2- 172.21.15.70:0/236710032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fab840a26a0 0x7fab840a4b60 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.580+0000 7faba31c7640 1 --2- 172.21.15.70:0/236710032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab9c108f60 0x7fab9c081f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.581+0000 7faba31c7640 1 --2- 172.21.15.70:0/236710032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c107d70 0x7fab9c0819f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.581+0000 7faba31c7640 1 -- 172.21.15.70:0/236710032 >> 172.21.15.70:0/236710032 conn(0x7fab9c076d10 msgr2=0x7fab9c10c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.581+0000 7faba31c7640 1 -- 172.21.15.70:0/236710032 shutdown_connections 2024-04-01T01:46:41.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.581+0000 7faba31c7640 1 -- 172.21.15.70:0/236710032 wait complete. 2024-04-01T01:46:41.602 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:46:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.784+0000 7fbb31587640 1 -- 172.21.15.70:0/1156201569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb2c109820 msgr2=0x7fbb2c1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.784+0000 7fbb31587640 1 --2- 172.21.15.70:0/1156201569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb2c109820 0x7fbb2c1100b0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009ca0 tx=0x7fbb2402c140 comp rx=0 tx=0).stop 2024-04-01T01:46:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.784+0000 7fbb31587640 1 -- 172.21.15.70:0/1156201569 shutdown_connections 2024-04-01T01:46:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.784+0000 7fbb31587640 1 --2- 172.21.15.70:0/1156201569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb2c109820 0x7fbb2c1100b0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.784+0000 7fbb31587640 1 --2- 172.21.15.70:0/1156201569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb2c108e60 0x7fbb2c1092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.784+0000 7fbb31587640 1 --2- 172.21.15.70:0/1156201569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb2c107c70 0x7fbb2c108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.784+0000 7fbb31587640 1 -- 172.21.15.70:0/1156201569 >> 172.21.15.70:0/1156201569 conn(0x7fbb2c076c20 msgr2=0x7fbb2c079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.784+0000 7fbb31587640 1 -- 172.21.15.70:0/1156201569 shutdown_connections 2024-04-01T01:46:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.784+0000 7fbb31587640 1 -- 172.21.15.70:0/1156201569 wait complete. 2024-04-01T01:46:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.785+0000 7fbb31587640 1 Processor -- start 2024-04-01T01:46:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.785+0000 7fbb31587640 1 -- start start 2024-04-01T01:46:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.786+0000 7fbb31587640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb2c107c70 0x7fbb2c06b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.786+0000 7fbb31587640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb2c108e60 0x7fbb2c06ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.786+0000 7fbb31587640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb2c109820 0x7fbb2c072b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.786+0000 7fbb31587640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c1138b0 con 0x7fbb2c107c70 2024-04-01T01:46:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.786+0000 7fbb31587640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c113730 con 0x7fbb2c108e60 2024-04-01T01:46:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.786+0000 7fbb31587640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c113a30 con 0x7fbb2c109820 2024-04-01T01:46:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.786+0000 7fbb30d86640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb2c109820 0x7fbb2c072b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.786+0000 7fbb30d86640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb2c109820 0x7fbb2c072b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57918/0 (socket says 172.21.15.70:57918) 2024-04-01T01:46:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.786+0000 7fbb30d86640 1 -- 172.21.15.70:0/590511438 learned_addr learned my addr 172.21.15.70:0/590511438 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.786+0000 7fbb2bfff640 1 --2- 172.21.15.70:0/590511438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb2c107c70 0x7fbb2c06b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.787+0000 7fbb2b7fe640 1 --2- 172.21.15.70:0/590511438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb2c108e60 0x7fbb2c06ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.787+0000 7fbb30d86640 1 -- 172.21.15.70:0/590511438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb2c108e60 msgr2=0x7fbb2c06ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.787+0000 7fbb30d86640 1 --2- 172.21.15.70:0/590511438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb2c108e60 0x7fbb2c06ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.787+0000 7fbb30d86640 1 -- 172.21.15.70:0/590511438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb2c107c70 msgr2=0x7fbb2c06b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.787+0000 7fbb30d86640 1 --2- 172.21.15.70:0/590511438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb2c107c70 0x7fbb2c06b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.787+0000 7fbb30d86640 1 -- 172.21.15.70:0/590511438 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb2c073170 con 0x7fbb2c109820 2024-04-01T01:46:41.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.788+0000 7fbb30d86640 1 --2- 172.21.15.70:0/590511438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb2c109820 0x7fbb2c072b00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009bf0 tx=0x7fbb24032890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.788+0000 7fbb297fa640 1 -- 172.21.15.70:0/590511438 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb24039040 con 0x7fbb2c109820 2024-04-01T01:46:41.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.788+0000 7fbb297fa640 1 -- 172.21.15.70:0/590511438 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb24044070 con 0x7fbb2c109820 2024-04-01T01:46:41.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.789+0000 7fbb31587640 1 -- 172.21.15.70:0/590511438 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb2c073400 con 0x7fbb2c109820 2024-04-01T01:46:41.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.789+0000 7fbb31587640 1 -- 172.21.15.70:0/590511438 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb2c1aec40 con 0x7fbb2c109820 2024-04-01T01:46:41.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.789+0000 7fbb297fa640 1 -- 172.21.15.70:0/590511438 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb24039040 con 0x7fbb2c109820 2024-04-01T01:46:41.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.792+0000 7fbb353d1740 1 Processor -- start 2024-04-01T01:46:41.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.793+0000 7fbb353d1740 1 -- start start 2024-04-01T01:46:41.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.794+0000 7fbb297fa640 1 -- 172.21.15.70:0/590511438 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fbb240043b0 con 0x7fbb2c109820 2024-04-01T01:46:41.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.794+0000 7fbb353d1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03ba86a0 0x560d03ba8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.794+0000 7fbb353d1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03ba9090 0x560d03badb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.794+0000 7fbb2b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03ba86a0 0x560d03ba8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.794+0000 7fbb2b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03ba86a0 0x560d03ba8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57926/0 (socket says 172.21.15.70:57926) 2024-04-01T01:46:41.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.795+0000 7fbb353d1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 0x560d03bb0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.795+0000 7fbb353d1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560d03b0d1b0 con 0x560d03ba9090 2024-04-01T01:46:41.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.795+0000 7fbb353d1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560d03ab5c00 con 0x560d03bae200 2024-04-01T01:46:41.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.795+0000 7fbb353d1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560d03b09250 con 0x560d03ba86a0 2024-04-01T01:46:41.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.795+0000 7fbb2bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03ba9090 0x560d03badb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.795+0000 7fbb2bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03ba9090 0x560d03badb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39348/0 (socket says 172.21.15.70:39348) 2024-04-01T01:46:41.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.795+0000 7fbb2bfff640 1 -- 172.21.15.70:0/3581765920 learned_addr learned my addr 172.21.15.70:0/3581765920 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:41.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.795+0000 7fbb2b7fe640 1 --2- 172.21.15.70:0/3581765920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 0x560d03bb0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.796+0000 7fbb297fa640 1 --2- 172.21.15.70:0/590511438 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbb140a2960 0x7fbb140a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.796+0000 7fbb2bfff640 1 -- 172.21.15.70:0/3581765920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03ba86a0 msgr2=0x560d03ba8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.796+0000 7fbb2bfff640 1 --2- 172.21.15.70:0/3581765920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03ba86a0 0x560d03ba8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.796+0000 7fbb2bfff640 1 -- 172.21.15.70:0/3581765920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 msgr2=0x560d03bb0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.796+0000 7fbb2bfff640 1 --2- 172.21.15.70:0/3581765920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 0x560d03bb0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.796+0000 7fbb2bfff640 1 -- 172.21.15.70:0/3581765920 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d03ad53d0 con 0x560d03ba9090 2024-04-01T01:46:41.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.796+0000 7fbb297fa640 1 -- 172.21.15.70:0/590511438 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fbb24032980 con 0x7fbb2c109820 2024-04-01T01:46:41.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.796+0000 7fbb30d86640 1 --2- 172.21.15.70:0/590511438 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbb140a2960 0x7fbb140a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.797+0000 7fbb2bfff640 1 --2- 172.21.15.70:0/3581765920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03ba9090 0x560d03badb90 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fbb2000b7f0 tx=0x7fbb2000bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea11f72ddab1bf88 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.797+0000 7fbb30d86640 1 --2- 172.21.15.70:0/590511438 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbb140a2960 0x7fbb140a4e20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fbb240c4e90 tx=0x7fbb2402c140 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.798+0000 7fbb127fc640 1 -- 172.21.15.70:0/3581765920 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2000d950 con 0x560d03ba9090 2024-04-01T01:46:41.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.798+0000 7fbb127fc640 1 -- 172.21.15.70:0/3581765920 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb20018070 con 0x560d03ba9090 2024-04-01T01:46:41.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.798+0000 7fbb127fc640 1 -- 172.21.15.70:0/3581765920 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb200123e0 con 0x560d03ba9090 2024-04-01T01:46:41.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.799+0000 7fbb353d1740 1 -- 172.21.15.70:0/3581765920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03ba9090 msgr2=0x560d03badb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.799+0000 7fbb353d1740 1 --2- 172.21.15.70:0/3581765920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03ba9090 0x560d03badb90 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fbb2000b7f0 tx=0x7fbb2000bcb0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.799+0000 7fbb353d1740 1 -- 172.21.15.70:0/3581765920 shutdown_connections 2024-04-01T01:46:41.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.799+0000 7fbb353d1740 1 --2- 172.21.15.70:0/3581765920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 0x560d03bb0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.799+0000 7fbb353d1740 1 --2- 172.21.15.70:0/3581765920 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03ba9090 0x560d03badb90 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.799+0000 7fbb353d1740 1 --2- 172.21.15.70:0/3581765920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03ba86a0 0x560d03ba8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.799+0000 7fbb353d1740 1 -- 172.21.15.70:0/3581765920 >> 172.21.15.70:0/3581765920 conn(0x560d03ba3c10 msgr2=0x560d03ba4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.799+0000 7fbb353d1740 1 -- 172.21.15.70:0/3581765920 shutdown_connections 2024-04-01T01:46:41.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.799+0000 7fbb353d1740 1 -- 172.21.15.70:0/3581765920 wait complete. 2024-04-01T01:46:41.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.799+0000 7fbb353d1740 1 -- start start 2024-04-01T01:46:41.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.800+0000 7fbb353d1740 1 Processor -- start 2024-04-01T01:46:41.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.800+0000 7fbb353d1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03ba86a0 0x560d03bb2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.800+0000 7fbb353d1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03ba9090 0x560d03bb3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.800+0000 7fbb353d1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 0x560d03bba170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.800+0000 7fbb353d1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560d03bb11b0 con 0x560d03ba86a0 2024-04-01T01:46:41.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.800+0000 7fbb353d1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560d03aab470 con 0x560d03bae200 2024-04-01T01:46:41.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.800+0000 7fbb353d1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560d03bb1330 con 0x560d03ba9090 2024-04-01T01:46:41.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.800+0000 7fbb353d1740 10 client.0 _ll_register_callbacks cb 0x560d03ac8cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:41.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.801+0000 7fbb2b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 0x560d03bba170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.801+0000 7fbb2b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03ba86a0 0x560d03bb2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.801+0000 7fbb2b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03ba86a0 0x560d03bb2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39356/0 (socket says 172.21.15.70:39356) 2024-04-01T01:46:41.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.801+0000 7fbb2b7fe640 1 -- 172.21.15.70:0/4217544143 learned_addr learned my addr 172.21.15.70:0/4217544143 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:41.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.801+0000 7fbb2bfff640 1 --2- 172.21.15.70:0/4217544143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03ba9090 0x560d03bb3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.802+0000 7fbb2bfff640 1 -- 172.21.15.70:0/4217544143 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 msgr2=0x560d03bba170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.802+0000 7fbb2bfff640 1 --2- 172.21.15.70:0/4217544143 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 0x560d03bba170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.802+0000 7fbb2bfff640 1 -- 172.21.15.70:0/4217544143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03ba86a0 msgr2=0x560d03bb2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.802+0000 7fbb2bfff640 1 --2- 172.21.15.70:0/4217544143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03ba86a0 0x560d03bb2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.802+0000 7fbb2bfff640 1 -- 172.21.15.70:0/4217544143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d038213f0 con 0x560d03ba9090 2024-04-01T01:46:41.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.804+0000 7fbb2bfff640 1 --2- 172.21.15.70:0/4217544143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03ba9090 0x560d03bb3130 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x560d03ba4b80 tx=0x7fbb20014ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb007234e895830a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.804+0000 7fbb0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:46:41.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.807+0000 7fbb0b7fe640 1 -- 172.21.15.70:0/4217544143 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2000d950 con 0x560d03ba9090 2024-04-01T01:46:41.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.807+0000 7fbb0b7fe640 1 -- 172.21.15.70:0/4217544143 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fbb2000daf0 con 0x560d03ba9090 2024-04-01T01:46:41.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.808+0000 7fbb353d1740 1 -- 172.21.15.70:0/4217544143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb20018070 con 0x560d03ba9090 2024-04-01T01:46:41.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.809+0000 7fbb0b7fe640 1 -- 172.21.15.70:0/4217544143 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbb2000dd90 con 0x560d03ba9090 2024-04-01T01:46:41.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.809+0000 7fbb353d1740 10 client.29161 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:41.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.809+0000 7fbb353d1740 1 -- 172.21.15.70:0/4217544143 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560d038eb7d0 con 0x560d03ba9090 2024-04-01T01:46:41.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.809+0000 7fbb0b7fe640 1 -- 172.21.15.70:0/4217544143 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb20018070 con 0x560d03ba9090 2024-04-01T01:46:41.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.809+0000 7fbb0b7fe640 1 -- 172.21.15.70:0/4217544143 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fbb2001c3f0 con 0x560d03ba9090 2024-04-01T01:46:41.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.810+0000 7fbb353d1740 10 client.29161 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:41.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.810+0000 7fbb353d1740 10 client.29161 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:41.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.810+0000 7fbb353d1740 20 client.29161 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:41.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.810+0000 7fbb353d1740 1 --2- 172.21.15.70:0/4217544143 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560d03bbe2d0 0x560d03bc06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.810+0000 7fbb353d1740 4 client.29161 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:41.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.810+0000 7fbb353d1740 1 -- 172.21.15.70:0/4217544143 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d03b1a6f0 con 0x560d03bbe2d0 2024-04-01T01:46:41.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.810+0000 7fbb2b7fe640 1 --2- 172.21.15.70:0/4217544143 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560d03bbe2d0 0x560d03bc06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.811+0000 7fbb2b7fe640 1 --2- 172.21.15.70:0/4217544143 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560d03bbe2d0 0x560d03bc06c0 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41b61797acb6a24b server_cookie=2ea2c76a4719d76c in_seq=0 out_seq=0 2024-04-01T01:46:41.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.811+0000 7fbb0b7fe640 10 client.29161 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:41.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.813+0000 7fbb0b7fe640 1 -- 172.21.15.70:0/4217544143 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560d03b1a6f0 con 0x560d03bbe2d0 2024-04-01T01:46:41.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.813+0000 7fbb0b7fe640 10 client.29161 handle_command_reply: tid=0 2024-04-01T01:46:41.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.814+0000 7fbb353d1740 1 client.29161 shutdown 2024-04-01T01:46:41.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.815+0000 7fbb353d1740 1 -- 172.21.15.70:0/4217544143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03ba9090 msgr2=0x560d03bb3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.815+0000 7fbb353d1740 1 --2- 172.21.15.70:0/4217544143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03ba9090 0x560d03bb3130 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x560d03ba4b80 tx=0x7fbb20014ef0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.815+0000 7fbb353d1740 1 -- 172.21.15.70:0/4217544143 shutdown_connections 2024-04-01T01:46:41.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.815+0000 7fbb353d1740 1 --2- 172.21.15.70:0/4217544143 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560d03bbe2d0 0x560d03bc06c0 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.815+0000 7fbb353d1740 1 --2- 172.21.15.70:0/4217544143 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 0x560d03bba170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.815+0000 7fbb353d1740 1 --2- 172.21.15.70:0/4217544143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03ba9090 0x560d03bb3130 secure :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0x560d03ba4b80 tx=0x7fbb20014ef0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.815+0000 7fbb353d1740 1 --2- 172.21.15.70:0/4217544143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03ba86a0 0x560d03bb2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.815+0000 7fbb353d1740 1 -- 172.21.15.70:0/4217544143 >> 172.21.15.70:0/4217544143 conn(0x560d03ba3c10 msgr2=0x560d03ba2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.815+0000 7fbb0b7fe640 0 client.29161 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:41.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.815+0000 7fbb353d1740 1 -- 172.21.15.70:0/4217544143 shutdown_connections 2024-04-01T01:46:41.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.815+0000 7fbb353d1740 1 -- 172.21.15.70:0/4217544143 wait complete. 2024-04-01T01:46:41.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.815+0000 7fbb353d1740 20 client.29161 trim_cache size 0 max 16384 2024-04-01T01:46:41.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.840+0000 7fbb353d1740 1 Processor -- start 2024-04-01T01:46:41.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.840+0000 7fbb353d1740 1 -- start start 2024-04-01T01:46:41.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.840+0000 7fbb353d1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03ba86a0 0x560d03ba2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.840+0000 7fbb353d1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03bae200 0x560d03bba130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.840+0000 7fbb353d1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03baad30 0x560d03ba2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.840+0000 7fbb353d1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c1b9140 con 0x560d03baad30 2024-04-01T01:46:41.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.840+0000 7fbb353d1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c1bac80 con 0x560d03ba86a0 2024-04-01T01:46:41.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.840+0000 7fbb353d1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c1b9810 con 0x560d03bae200 2024-04-01T01:46:41.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.840+0000 7fbb2bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03bae200 0x560d03bba130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.840+0000 7fbb2b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03ba86a0 0x560d03ba2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.840+0000 7fbb2bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03bae200 0x560d03bba130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57938/0 (socket says 172.21.15.70:57938) 2024-04-01T01:46:41.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.840+0000 7fbb2b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03ba86a0 0x560d03ba2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:43000/0 (socket says 172.21.15.70:43000) 2024-04-01T01:46:41.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.840+0000 7fbb2b7fe640 1 -- 172.21.15.70:0/3670501523 learned_addr learned my addr 172.21.15.70:0/3670501523 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:41.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.841+0000 7fbb2b7fe640 1 --2- 172.21.15.70:0/3670501523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03baad30 0x560d03ba2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.841+0000 7fbb2bfff640 1 -- 172.21.15.70:0/3670501523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03ba86a0 msgr2=0x560d03ba2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.841+0000 7fbb2bfff640 1 --2- 172.21.15.70:0/3670501523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03ba86a0 0x560d03ba2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.841+0000 7fbb2bfff640 1 -- 172.21.15.70:0/3670501523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03baad30 msgr2=0x560d03ba2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.841+0000 7fbb2bfff640 1 --2- 172.21.15.70:0/3670501523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03baad30 0x560d03ba2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.841+0000 7fbb2bfff640 1 -- 172.21.15.70:0/3670501523 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d03b94e60 con 0x560d03bae200 2024-04-01T01:46:41.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.841+0000 7fbb2b7fe640 1 --2- 172.21.15.70:0/3670501523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03ba86a0 0x560d03ba2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:46:41.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.842+0000 7fbb2bfff640 1 --2- 172.21.15.70:0/3670501523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03bae200 0x560d03bba130 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x560d03bb3a10 tx=0x7fbb2000b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b6d99492320608a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.844+0000 7fbb11ffb640 1 -- 172.21.15.70:0/3670501523 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb20033070 con 0x560d03bae200 2024-04-01T01:46:41.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.844+0000 7fbb11ffb640 1 -- 172.21.15.70:0/3670501523 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb20018070 con 0x560d03bae200 2024-04-01T01:46:41.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.844+0000 7fbb11ffb640 1 -- 172.21.15.70:0/3670501523 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2000d950 con 0x560d03bae200 2024-04-01T01:46:41.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.845+0000 7fbb353d1740 1 -- 172.21.15.70:0/3670501523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03bae200 msgr2=0x560d03bba130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.845+0000 7fbb353d1740 1 --2- 172.21.15.70:0/3670501523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03bae200 0x560d03bba130 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x560d03bb3a10 tx=0x7fbb2000b670 comp rx=0 tx=0).stop 2024-04-01T01:46:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.846+0000 7fbb353d1740 1 -- 172.21.15.70:0/3670501523 shutdown_connections 2024-04-01T01:46:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.846+0000 7fbb353d1740 1 --2- 172.21.15.70:0/3670501523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03baad30 0x560d03ba2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.846+0000 7fbb353d1740 1 --2- 172.21.15.70:0/3670501523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03bae200 0x560d03bba130 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.847+0000 7fbb353d1740 1 --2- 172.21.15.70:0/3670501523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03ba86a0 0x560d03ba2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.847+0000 7fbb353d1740 1 -- 172.21.15.70:0/3670501523 >> 172.21.15.70:0/3670501523 conn(0x560d03ba3c10 msgr2=0x560d03baa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.847+0000 7fbb353d1740 1 -- 172.21.15.70:0/3670501523 shutdown_connections 2024-04-01T01:46:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.847+0000 7fbb353d1740 1 -- 172.21.15.70:0/3670501523 wait complete. 2024-04-01T01:46:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.848+0000 7fbb353d1740 1 -- start start 2024-04-01T01:46:41.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.848+0000 7fbb353d1740 1 Processor -- start 2024-04-01T01:46:41.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.849+0000 7fbb353d1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03ba86a0 0x560d03bb7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.849+0000 7fbb353d1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 0x560d03bb8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.849+0000 7fbb353d1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03bb9190 0x560d03bb9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.849+0000 7fbb353d1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c1ba5b0 con 0x560d03bb9190 2024-04-01T01:46:41.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.849+0000 7fbb353d1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c1b9ee0 con 0x560d03bae200 2024-04-01T01:46:41.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.849+0000 7fbb353d1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbb2c1b7120 con 0x560d03ba86a0 2024-04-01T01:46:41.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.849+0000 7fbb353d1740 10 client.0 _ll_register_callbacks cb 0x7fbb20013970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:41.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.850+0000 7fbb2bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 0x560d03bb8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.850+0000 7fbb2bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 0x560d03bb8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:43016/0 (socket says 172.21.15.70:43016) 2024-04-01T01:46:41.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.850+0000 7fbb2bfff640 1 -- 172.21.15.70:0/1223411412 learned_addr learned my addr 172.21.15.70:0/1223411412 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:41.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.850+0000 7fbb2bfff640 1 -- 172.21.15.70:0/1223411412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03ba86a0 msgr2=0x560d03bb7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.850+0000 7fbb2bfff640 1 --2- 172.21.15.70:0/1223411412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03ba86a0 0x560d03bb7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.850+0000 7fbb2bfff640 1 -- 172.21.15.70:0/1223411412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03bb9190 msgr2=0x560d03bb9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.850+0000 7fbb2bfff640 1 --2- 172.21.15.70:0/1223411412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03bb9190 0x560d03bb9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.850+0000 7fbb2bfff640 1 -- 172.21.15.70:0/1223411412 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d03bb9b50 con 0x560d03bae200 2024-04-01T01:46:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.851+0000 7fbb2bfff640 1 --2- 172.21.15.70:0/1223411412 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 0x560d03bb8180 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fbb20010040 tx=0x7fbb20014fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3374c845223deb06 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.851+0000 7fbb0b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.851+0000 7fbb0b7fe640 1 -- 172.21.15.70:0/1223411412 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2003c060 con 0x560d03bae200 2024-04-01T01:46:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.851+0000 7fbb0b7fe640 1 -- 172.21.15.70:0/1223411412 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fbb2000d950 con 0x560d03bae200 2024-04-01T01:46:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.851+0000 7fbb353d1740 1 -- 172.21.15.70:0/1223411412 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb20018070 con 0x560d03bae200 2024-04-01T01:46:41.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.851+0000 7fbb0b7fe640 1 -- 172.21.15.70:0/1223411412 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbb20033070 con 0x560d03bae200 2024-04-01T01:46:41.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.851+0000 7fbb353d1740 10 client.29390 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:41.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.851+0000 7fbb353d1740 1 -- 172.21.15.70:0/1223411412 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560d03c24d80 con 0x560d03bae200 2024-04-01T01:46:41.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.852+0000 7fbb0b7fe640 1 -- 172.21.15.70:0/1223411412 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb20018070 con 0x560d03bae200 2024-04-01T01:46:41.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.853+0000 7fbb0b7fe640 1 -- 172.21.15.70:0/1223411412 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fbb20004220 con 0x560d03bae200 2024-04-01T01:46:41.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.853+0000 7fbb353d1740 10 client.29390 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:41.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.853+0000 7fbb353d1740 10 client.29390 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:41.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.853+0000 7fbb353d1740 20 client.29390 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:41.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.853+0000 7fbb353d1740 1 --2- 172.21.15.70:0/1223411412 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560d03c25680 0x560d03c27aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:41.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.853+0000 7fbb353d1740 4 client.29390 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:46:41.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.853+0000 7fbb353d1740 1 -- 172.21.15.70:0/1223411412 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d03b14a40 con 0x560d03c25680 2024-04-01T01:46:41.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.855+0000 7fbb2b7fe640 1 --2- 172.21.15.70:0/1223411412 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560d03c25680 0x560d03c27aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:41.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.857+0000 7fbb2b7fe640 1 --2- 172.21.15.70:0/1223411412 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560d03c25680 0x560d03c27aa0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=336b3c8e3cfd24bc server_cookie=10410b737ba9bc36 in_seq=0 out_seq=0 2024-04-01T01:46:41.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.857+0000 7fbb0b7fe640 10 client.29390 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:41.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.857+0000 7fbb0b7fe640 1 -- 172.21.15.70:0/1223411412 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d03b14a40 con 0x560d03c25680 2024-04-01T01:46:41.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.857+0000 7fbb0b7fe640 10 client.29390 handle_command_reply: tid=0 2024-04-01T01:46:41.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.857+0000 7fbb353d1740 1 client.29390 shutdown 2024-04-01T01:46:41.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.857+0000 7fbb353d1740 1 -- 172.21.15.70:0/1223411412 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 msgr2=0x560d03bb8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.857+0000 7fbb353d1740 1 --2- 172.21.15.70:0/1223411412 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 0x560d03bb8180 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fbb20010040 tx=0x7fbb20014fe0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.858+0000 7fbb353d1740 1 -- 172.21.15.70:0/1223411412 shutdown_connections 2024-04-01T01:46:41.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.858+0000 7fbb353d1740 1 --2- 172.21.15.70:0/1223411412 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560d03c25680 0x560d03c27aa0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.858+0000 7fbb353d1740 1 --2- 172.21.15.70:0/1223411412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560d03bb9190 0x560d03bb9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.858+0000 7fbb353d1740 1 --2- 172.21.15.70:0/1223411412 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560d03bae200 0x560d03bb8180 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.858+0000 7fbb353d1740 1 --2- 172.21.15.70:0/1223411412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d03ba86a0 0x560d03bb7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.858+0000 7fbb353d1740 1 -- 172.21.15.70:0/1223411412 >> 172.21.15.70:0/1223411412 conn(0x560d03ba3c10 msgr2=0x560d03ba6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.858+0000 7fbb0b7fe640 0 client.29390 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:41.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.858+0000 7fbb353d1740 1 -- 172.21.15.70:0/1223411412 shutdown_connections 2024-04-01T01:46:41.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.858+0000 7fbb353d1740 1 -- 172.21.15.70:0/1223411412 wait complete. 2024-04-01T01:46:41.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.859+0000 7fbb353d1740 20 client.29390 trim_cache size 0 max 16384 2024-04-01T01:46:41.860 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:46:41.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.860+0000 7fbb31587640 1 -- 172.21.15.70:0/590511438 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbb140a2960 msgr2=0x7fbb140a4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.860+0000 7fbb31587640 1 --2- 172.21.15.70:0/590511438 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbb140a2960 0x7fbb140a4e20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fbb240c4e90 tx=0x7fbb2402c140 comp rx=0 tx=0).stop 2024-04-01T01:46:41.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.860+0000 7fbb31587640 1 -- 172.21.15.70:0/590511438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb2c109820 msgr2=0x7fbb2c072b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:41.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.860+0000 7fbb31587640 1 --2- 172.21.15.70:0/590511438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb2c109820 0x7fbb2c072b00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009bf0 tx=0x7fbb24032890 comp rx=0 tx=0).stop 2024-04-01T01:46:41.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.861+0000 7fbb31587640 1 -- 172.21.15.70:0/590511438 shutdown_connections 2024-04-01T01:46:41.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.861+0000 7fbb31587640 1 --2- 172.21.15.70:0/590511438 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbb140a2960 0x7fbb140a4e20 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.861+0000 7fbb31587640 1 --2- 172.21.15.70:0/590511438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb2c109820 0x7fbb2c072b00 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.861+0000 7fbb31587640 1 --2- 172.21.15.70:0/590511438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb2c108e60 0x7fbb2c06ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.861+0000 7fbb31587640 1 --2- 172.21.15.70:0/590511438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb2c107c70 0x7fbb2c06b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:41.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.861+0000 7fbb31587640 1 -- 172.21.15.70:0/590511438 >> 172.21.15.70:0/590511438 conn(0x7fbb2c076c20 msgr2=0x7fbb2c10c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:41.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.862+0000 7fbb31587640 1 -- 172.21.15.70:0/590511438 shutdown_connections 2024-04-01T01:46:41.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:41.862+0000 7fbb31587640 1 -- 172.21.15.70:0/590511438 wait complete. 2024-04-01T01:46:41.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 715.0 s, 711.7 tps, lat 141.780 ms stddev 117.218 2024-04-01T01:46:42.881 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:46:42.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:46:42.882 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:46:43.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:42 smithi081 ceph-mon[35142]: pgmap v755: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.3 MiB/s wr, 698 op/s 2024-04-01T01:46:43.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.077+0000 7f81329bc640 1 -- 172.21.15.70:0/3135125361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f812c108750 msgr2=0x7f812c10ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.077+0000 7f81329bc640 1 --2- 172.21.15.70:0/3135125361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f812c108750 0x7f812c10ab80 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f8120009860 tx=0x7f812002bbb0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.078+0000 7f81329bc640 1 -- 172.21.15.70:0/3135125361 shutdown_connections 2024-04-01T01:46:43.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.078+0000 7f81329bc640 1 --2- 172.21.15.70:0/3135125361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f812c10da70 0x7f812c10def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.078+0000 7f81329bc640 1 --2- 172.21.15.70:0/3135125361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f812c10b0c0 0x7f812c10d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.078+0000 7f81329bc640 1 --2- 172.21.15.70:0/3135125361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f812c108750 0x7f812c10ab80 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.078+0000 7f81329bc640 1 -- 172.21.15.70:0/3135125361 >> 172.21.15.70:0/3135125361 conn(0x7f812c06e5b0 msgr2=0x7f812c0709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.080+0000 7f81329bc640 1 -- 172.21.15.70:0/3135125361 shutdown_connections 2024-04-01T01:46:43.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.083+0000 7f81329bc640 1 -- 172.21.15.70:0/3135125361 wait complete. 2024-04-01T01:46:43.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.084+0000 7f81329bc640 1 Processor -- start 2024-04-01T01:46:43.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.084+0000 7f81329bc640 1 -- start start 2024-04-01T01:46:43.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.084+0000 7f81329bc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f812c108750 0x7f812c1a6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.084+0000 7f81329bc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f812c10b0c0 0x7f812c1a6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.084+0000 7f81329bc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f812c10da70 0x7f812c1ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.084+0000 7f81329bc640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f812c11c070 con 0x7f812c10b0c0 2024-04-01T01:46:43.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.084+0000 7f81329bc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f812c11bef0 con 0x7f812c10da70 2024-04-01T01:46:43.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.084+0000 7f81329bc640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f812c11c1f0 con 0x7f812c108750 2024-04-01T01:46:43.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.085+0000 7f8130f59640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f812c10da70 0x7f812c1ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.085+0000 7f8130f59640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f812c10da70 0x7f812c1ade10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:43042/0 (socket says 172.21.15.70:43042) 2024-04-01T01:46:43.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.085+0000 7f8130f59640 1 -- 172.21.15.70:0/1981049422 learned_addr learned my addr 172.21.15.70:0/1981049422 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:43.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.085+0000 7f812b7fe640 1 --2- 172.21.15.70:0/1981049422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f812c10b0c0 0x7f812c1a6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.085+0000 7f812bfff640 1 --2- 172.21.15.70:0/1981049422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f812c108750 0x7f812c1a6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.085+0000 7f8130f59640 1 -- 172.21.15.70:0/1981049422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f812c108750 msgr2=0x7f812c1a6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.085+0000 7f8130f59640 1 --2- 172.21.15.70:0/1981049422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f812c108750 0x7f812c1a6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.085+0000 7f8130f59640 1 -- 172.21.15.70:0/1981049422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f812c10b0c0 msgr2=0x7f812c1a6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.085+0000 7f8130f59640 1 --2- 172.21.15.70:0/1981049422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f812c10b0c0 0x7f812c1a6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.085+0000 7f8130f59640 1 -- 172.21.15.70:0/1981049422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f812c1ae480 con 0x7f812c10da70 2024-04-01T01:46:43.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.086+0000 7f8130f59640 1 --2- 172.21.15.70:0/1981049422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f812c10da70 0x7f812c1ade10 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f812400cab0 tx=0x7f812400cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.086+0000 7f81297fa640 1 -- 172.21.15.70:0/1981049422 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812400ae20 con 0x7f812c10da70 2024-04-01T01:46:43.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.086+0000 7f81297fa640 1 -- 172.21.15.70:0/1981049422 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f812400f900 con 0x7f812c10da70 2024-04-01T01:46:43.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.086+0000 7f81329bc640 1 -- 172.21.15.70:0/1981049422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f812c1ae770 con 0x7f812c10da70 2024-04-01T01:46:43.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.086+0000 7f81297fa640 1 -- 172.21.15.70:0/1981049422 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81240189f0 con 0x7f812c10da70 2024-04-01T01:46:43.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.086+0000 7f81329bc640 1 -- 172.21.15.70:0/1981049422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f812c1aecc0 con 0x7f812c10da70 2024-04-01T01:46:43.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.090+0000 7f813559f740 1 Processor -- start 2024-04-01T01:46:43.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.091+0000 7f81297fa640 1 -- 172.21.15.70:0/1981049422 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f812400f450 con 0x7f812c10da70 2024-04-01T01:46:43.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.091+0000 7f813559f740 1 -- start start 2024-04-01T01:46:43.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.091+0000 7f813559f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054ca10 0x564ac054ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.091+0000 7f813559f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac054d400 0x564ac0551f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.091+0000 7f813559f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac0552570 0x564ac0554a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.091+0000 7f813559f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564ac044e0b0 con 0x564ac054d400 2024-04-01T01:46:43.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.091+0000 7f813559f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x564ac04b0ea0 con 0x564ac054ca10 2024-04-01T01:46:43.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.091+0000 7f813559f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564ac0459c00 con 0x564ac0552570 2024-04-01T01:46:43.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.092+0000 7f812b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054ca10 0x564ac054ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.092+0000 7f812b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054ca10 0x564ac054ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:43046/0 (socket says 172.21.15.70:43046) 2024-04-01T01:46:43.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.092+0000 7f812b7fe640 1 -- 172.21.15.70:0/3351658697 learned_addr learned my addr 172.21.15.70:0/3351658697 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:43.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.092+0000 7f812bfff640 1 --2- 172.21.15.70:0/3351658697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac054d400 0x564ac0551f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.092+0000 7f81297fa640 1 --2- 172.21.15.70:0/1981049422 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81140a2620 0x7f81140a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.092+0000 7f81297fa640 1 -- 172.21.15.70:0/1981049422 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f81240cf170 con 0x7f812c10da70 2024-04-01T01:46:43.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.093+0000 7f812bfff640 1 -- 172.21.15.70:0/3351658697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac0552570 msgr2=0x564ac0554a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.093+0000 7f812bfff640 1 --2- 172.21.15.70:0/3351658697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac0552570 0x564ac0554a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.093+0000 7f812bfff640 1 -- 172.21.15.70:0/3351658697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054ca10 msgr2=0x564ac054ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.093+0000 7f812bfff640 1 --2- 172.21.15.70:0/3351658697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054ca10 0x564ac054ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.093+0000 7f812bfff640 1 -- 172.21.15.70:0/3351658697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ac028d7e0 con 0x564ac054d400 2024-04-01T01:46:43.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.093+0000 7f812bfff640 1 --2- 172.21.15.70:0/3351658697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac054d400 0x564ac0551f00 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f8120009780 tx=0x7f8120035b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06c1337cd16fef0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.093+0000 7f81127fc640 1 -- 172.21.15.70:0/3351658697 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8120039040 con 0x564ac054d400 2024-04-01T01:46:43.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.093+0000 7f81127fc640 1 -- 172.21.15.70:0/3351658697 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8120035e10 con 0x564ac054d400 2024-04-01T01:46:43.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.093+0000 7f81127fc640 1 -- 172.21.15.70:0/3351658697 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8120005630 con 0x564ac054d400 2024-04-01T01:46:43.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.094+0000 7f813559f740 1 -- 172.21.15.70:0/3351658697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac054d400 msgr2=0x564ac0551f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.094+0000 7f813559f740 1 --2- 172.21.15.70:0/3351658697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac054d400 0x564ac0551f00 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f8120009780 tx=0x7f8120035b60 comp rx=0 tx=0).stop 2024-04-01T01:46:43.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.096+0000 7f8130f59640 1 --2- 172.21.15.70:0/1981049422 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81140a2620 0x7f81140a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.096+0000 7f813559f740 1 -- 172.21.15.70:0/3351658697 shutdown_connections 2024-04-01T01:46:43.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.096+0000 7f813559f740 1 --2- 172.21.15.70:0/3351658697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac0552570 0x564ac0554a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.096+0000 7f813559f740 1 --2- 172.21.15.70:0/3351658697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac054d400 0x564ac0551f00 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.096+0000 7f813559f740 1 --2- 172.21.15.70:0/3351658697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054ca10 0x564ac054ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.096+0000 7f813559f740 1 -- 172.21.15.70:0/3351658697 >> 172.21.15.70:0/3351658697 conn(0x564ac0547f80 msgr2=0x564ac0548380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.098+0000 7f813559f740 1 -- 172.21.15.70:0/3351658697 shutdown_connections 2024-04-01T01:46:43.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.098+0000 7f813559f740 1 -- 172.21.15.70:0/3351658697 wait complete. 2024-04-01T01:46:43.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.098+0000 7f813559f740 1 -- start start 2024-04-01T01:46:43.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.099+0000 7f813559f740 1 Processor -- start 2024-04-01T01:46:43.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.099+0000 7f813559f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac054ca10 0x564ac0567d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.099+0000 7f813559f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac054d400 0x564ac05682b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.099+0000 7f813559f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac0552570 0x564ac056f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.099+0000 7f813559f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564ac044f3d0 con 0x564ac054ca10 2024-04-01T01:46:43.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.099+0000 7f813559f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x564ac04acf40 con 0x564ac0552570 2024-04-01T01:46:43.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.099+0000 7f813559f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564ac05554c0 con 0x564ac054d400 2024-04-01T01:46:43.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.099+0000 7f813559f740 10 client.0 _ll_register_callbacks cb 0x564ac046cb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:43.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.099+0000 7f8130f59640 1 --2- 172.21.15.70:0/1981049422 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81140a2620 0x7f81140a4ae0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f812409b560 tx=0x7f81240071c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.100+0000 7f812b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac0552570 0x564ac056f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.100+0000 7f812b7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac0552570 0x564ac056f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:43056/0 (socket says 172.21.15.70:43056) 2024-04-01T01:46:43.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.100+0000 7f812b7fe640 1 -- 172.21.15.70:0/264697985 learned_addr learned my addr 172.21.15.70:0/264697985 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:43.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.100+0000 7f812bfff640 1 --2- 172.21.15.70:0/264697985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac054d400 0x564ac05682b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.101+0000 7f812b7fe640 1 -- 172.21.15.70:0/264697985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac054d400 msgr2=0x564ac05682b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.101+0000 7f812b7fe640 1 --2- 172.21.15.70:0/264697985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac054d400 0x564ac05682b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.101+0000 7f812b7fe640 1 -- 172.21.15.70:0/264697985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac054ca10 msgr2=0x564ac0567d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.101+0000 7f812b7fe640 1 --2- 172.21.15.70:0/264697985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac054ca10 0x564ac0567d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.101+0000 7f812b7fe640 1 -- 172.21.15.70:0/264697985 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ac02edf70 con 0x564ac0552570 2024-04-01T01:46:43.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.105+0000 7f812b7fe640 1 --2- 172.21.15.70:0/264697985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac0552570 0x564ac056f2f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f811c0136e0 tx=0x7f811c013ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=666222114fce3d2d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.106+0000 7f810b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:43.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.106+0000 7f810b7fe640 1 -- 172.21.15.70:0/264697985 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811c011bc0 con 0x564ac0552570 2024-04-01T01:46:43.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.106+0000 7f810b7fe640 1 -- 172.21.15.70:0/264697985 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f811c011d60 con 0x564ac0552570 2024-04-01T01:46:43.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.106+0000 7f813559f740 1 -- 172.21.15.70:0/264697985 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8120035e10 con 0x564ac0552570 2024-04-01T01:46:43.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.106+0000 7f810b7fe640 1 -- 172.21.15.70:0/264697985 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f811c01c440 con 0x564ac0552570 2024-04-01T01:46:43.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.107+0000 7f813559f740 10 client.29399 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:43.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.107+0000 7f813559f740 1 -- 172.21.15.70:0/264697985 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564ac0185170 con 0x564ac0552570 2024-04-01T01:46:43.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.107+0000 7f810b7fe640 1 -- 172.21.15.70:0/264697985 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8120035e10 con 0x564ac0552570 2024-04-01T01:46:43.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.107+0000 7f810b7fe640 1 -- 172.21.15.70:0/264697985 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f811c01a070 con 0x564ac0552570 2024-04-01T01:46:43.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.107+0000 7f813559f740 10 client.29399 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:43.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.107+0000 7f813559f740 10 client.29399 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:43.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.107+0000 7f813559f740 20 client.29399 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:43.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.107+0000 7f813559f740 1 --2- 172.21.15.70:0/264697985 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564ac0559780 0x564ac055bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.107+0000 7f813559f740 4 client.29399 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:43.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.107+0000 7f813559f740 1 -- 172.21.15.70:0/264697985 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ac04be3e0 con 0x564ac0559780 2024-04-01T01:46:43.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.109+0000 7f812bfff640 1 --2- 172.21.15.70:0/264697985 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564ac0559780 0x564ac055bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.109+0000 7f812bfff640 1 --2- 172.21.15.70:0/264697985 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564ac0559780 0x564ac055bbe0 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bee6eb3cd92307a4 server_cookie=b708c0d6e3388c16 in_seq=0 out_seq=0 2024-04-01T01:46:43.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.109+0000 7f810b7fe640 10 client.29399 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:43.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.110+0000 7f810b7fe640 1 -- 172.21.15.70:0/264697985 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564ac04be3e0 con 0x564ac0559780 2024-04-01T01:46:43.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.110+0000 7f810b7fe640 10 client.29399 handle_command_reply: tid=0 2024-04-01T01:46:43.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.111+0000 7f813559f740 1 client.29399 shutdown 2024-04-01T01:46:43.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.111+0000 7f813559f740 1 -- 172.21.15.70:0/264697985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac0552570 msgr2=0x564ac056f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.111+0000 7f813559f740 1 --2- 172.21.15.70:0/264697985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac0552570 0x564ac056f2f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f811c0136e0 tx=0x7f811c013ba0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.111+0000 7f813559f740 1 -- 172.21.15.70:0/264697985 shutdown_connections 2024-04-01T01:46:43.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.111+0000 7f813559f740 1 --2- 172.21.15.70:0/264697985 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564ac0559780 0x564ac055bbe0 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.111+0000 7f813559f740 1 --2- 172.21.15.70:0/264697985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac0552570 0x564ac056f2f0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.111+0000 7f813559f740 1 --2- 172.21.15.70:0/264697985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac054d400 0x564ac05682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.111+0000 7f813559f740 1 --2- 172.21.15.70:0/264697985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac054ca10 0x564ac0567d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.111+0000 7f813559f740 1 -- 172.21.15.70:0/264697985 >> 172.21.15.70:0/264697985 conn(0x564ac0547f80 msgr2=0x564ac0547180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.111+0000 7f810b7fe640 0 client.29399 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:43.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.111+0000 7f813559f740 1 -- 172.21.15.70:0/264697985 shutdown_connections 2024-04-01T01:46:43.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.111+0000 7f813559f740 1 -- 172.21.15.70:0/264697985 wait complete. 2024-04-01T01:46:43.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.111+0000 7f813559f740 20 client.29399 trim_cache size 0 max 16384 2024-04-01T01:46:43.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.149+0000 7f813559f740 1 Processor -- start 2024-04-01T01:46:43.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.149+0000 7f813559f740 1 -- start start 2024-04-01T01:46:43.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.149+0000 7f813559f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac054ca10 0x564ac0548360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.149+0000 7f813559f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054d400 0x564ac056f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.150+0000 7f813559f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac0552570 0x564ac0567d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.150+0000 7f812bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054d400 0x564ac056f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.150+0000 7f812bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054d400 0x564ac056f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:43072/0 (socket says 172.21.15.70:43072) 2024-04-01T01:46:43.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.150+0000 7f812bfff640 1 -- 172.21.15.70:0/1646937536 learned_addr learned my addr 172.21.15.70:0/1646937536 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:43.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.150+0000 7f813559f740 1 -- 172.21.15.70:0/1646937536 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f811800fd90 con 0x564ac054ca10 2024-04-01T01:46:43.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.150+0000 7f813559f740 1 -- 172.21.15.70:0/1646937536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f81180118d0 con 0x564ac054d400 2024-04-01T01:46:43.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.150+0000 7f813559f740 1 -- 172.21.15.70:0/1646937536 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8118010460 con 0x564ac0552570 2024-04-01T01:46:43.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.153+0000 7f812b7fe640 1 --2- 172.21.15.70:0/1646937536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac0552570 0x564ac0567d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.153+0000 7f812b7fe640 1 --2- 172.21.15.70:0/1646937536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac054ca10 0x564ac0548360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.153+0000 7f812bfff640 1 -- 172.21.15.70:0/1646937536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac0552570 msgr2=0x564ac0567d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.153+0000 7f812bfff640 1 --2- 172.21.15.70:0/1646937536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac0552570 0x564ac0567d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.153+0000 7f812bfff640 1 -- 172.21.15.70:0/1646937536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac054ca10 msgr2=0x564ac0548360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.153+0000 7f812bfff640 1 --2- 172.21.15.70:0/1646937536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac054ca10 0x564ac0548360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.153+0000 7f812bfff640 1 -- 172.21.15.70:0/1646937536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ac054f220 con 0x564ac054d400 2024-04-01T01:46:43.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.153+0000 7f812bfff640 1 --2- 172.21.15.70:0/1646937536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054d400 0x564ac056f280 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f812002bbf0 tx=0x7f8120005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50207193c380e992 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.155+0000 7f8111ffb640 1 -- 172.21.15.70:0/1646937536 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8120039040 con 0x564ac054d400 2024-04-01T01:46:43.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.155+0000 7f8111ffb640 1 -- 172.21.15.70:0/1646937536 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8120005410 con 0x564ac054d400 2024-04-01T01:46:43.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.156+0000 7f813559f740 1 -- 172.21.15.70:0/1646937536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054d400 msgr2=0x564ac056f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.156+0000 7f813559f740 1 --2- 172.21.15.70:0/1646937536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054d400 0x564ac056f280 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f812002bbf0 tx=0x7f8120005b20 comp rx=0 tx=0).stop 2024-04-01T01:46:43.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.160+0000 7f813559f740 1 -- 172.21.15.70:0/1646937536 shutdown_connections 2024-04-01T01:46:43.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.161+0000 7f813559f740 1 --2- 172.21.15.70:0/1646937536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac0552570 0x564ac0567d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.161+0000 7f813559f740 1 --2- 172.21.15.70:0/1646937536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054d400 0x564ac056f280 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.161+0000 7f813559f740 1 --2- 172.21.15.70:0/1646937536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac054ca10 0x564ac0548360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.161+0000 7f813559f740 1 -- 172.21.15.70:0/1646937536 >> 172.21.15.70:0/1646937536 conn(0x564ac0547f80 msgr2=0x564ac0542d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.166+0000 7f813559f740 1 -- 172.21.15.70:0/1646937536 shutdown_connections 2024-04-01T01:46:43.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.168+0000 7f813559f740 1 -- 172.21.15.70:0/1646937536 wait complete. 2024-04-01T01:46:43.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.169+0000 7f813559f740 1 -- start start 2024-04-01T01:46:43.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.170+0000 7f813559f740 1 Processor -- start 2024-04-01T01:46:43.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.170+0000 7f813559f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac054ca10 0x564ac056ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.170+0000 7f813559f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054d400 0x564ac056d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.170+0000 7f812b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac054ca10 0x564ac056ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.170+0000 7f812b7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac054ca10 0x564ac056ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58004/0 (socket says 172.21.15.70:58004) 2024-04-01T01:46:43.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.171+0000 7f812b7fe640 1 -- 172.21.15.70:0/2031303984 learned_addr learned my addr 172.21.15.70:0/2031303984 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:43.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.171+0000 7f813559f740 1 --2- 172.21.15.70:0/2031303984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac0552570 0x564ac056e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.171+0000 7f813559f740 1 -- 172.21.15.70:0/2031303984 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8118011200 con 0x564ac0552570 2024-04-01T01:46:43.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.171+0000 7f813559f740 1 -- 172.21.15.70:0/2031303984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8118010b30 con 0x564ac054d400 2024-04-01T01:46:43.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.171+0000 7f813559f740 1 -- 172.21.15.70:0/2031303984 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f811800dd70 con 0x564ac054ca10 2024-04-01T01:46:43.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.171+0000 7f813559f740 10 client.0 _ll_register_callbacks cb 0x7f811c00ceb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:43.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.172+0000 7f812bfff640 1 --2- 172.21.15.70:0/2031303984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054d400 0x564ac056d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.172+0000 7f812bfff640 1 -- 172.21.15.70:0/2031303984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac054ca10 msgr2=0x564ac056ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.172+0000 7f812bfff640 1 --2- 172.21.15.70:0/2031303984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac054ca10 0x564ac056ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.172+0000 7f812bfff640 1 -- 172.21.15.70:0/2031303984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac0552570 msgr2=0x564ac056e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.172+0000 7f812bfff640 1 --2- 172.21.15.70:0/2031303984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac0552570 0x564ac056e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.172+0000 7f812bfff640 1 -- 172.21.15.70:0/2031303984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ac056e9a0 con 0x564ac054d400 2024-04-01T01:46:43.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.173+0000 7f812bfff640 1 --2- 172.21.15.70:0/2031303984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054d400 0x564ac056d410 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f812002c1c0 tx=0x7f81200517d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d21dc7dbfa0a325 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.174+0000 7f810b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:43.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.174+0000 7f810b7fe640 1 -- 172.21.15.70:0/2031303984 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8120039030 con 0x564ac054d400 2024-04-01T01:46:43.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.174+0000 7f810b7fe640 1 -- 172.21.15.70:0/2031303984 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8120034b50 con 0x564ac054d400 2024-04-01T01:46:43.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.177+0000 7f813559f740 1 -- 172.21.15.70:0/2031303984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564ac04b8730 con 0x564ac054d400 2024-04-01T01:46:43.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.179+0000 7f810b7fe640 1 -- 172.21.15.70:0/2031303984 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8120034d50 con 0x564ac054d400 2024-04-01T01:46:43.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.181+0000 7f813559f740 10 client.29405 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:43.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.181+0000 7f813559f740 1 -- 172.21.15.70:0/2031303984 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564ac05eb210 con 0x564ac054d400 2024-04-01T01:46:43.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.181+0000 7f810b7fe640 1 -- 172.21.15.70:0/2031303984 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x564ac04b8730 con 0x564ac054d400 2024-04-01T01:46:43.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.181+0000 7f810b7fe640 1 -- 172.21.15.70:0/2031303984 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8120003c70 con 0x564ac054d400 2024-04-01T01:46:43.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.181+0000 7f813559f740 10 client.29405 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:43.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.181+0000 7f813559f740 10 client.29405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:43.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.181+0000 7f813559f740 20 client.29405 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:43.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.181+0000 7f813559f740 1 --2- 172.21.15.70:0/2031303984 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564ac05eb650 0x564ac05eda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.181+0000 7f813559f740 4 client.29405 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:46:43.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.181+0000 7f813559f740 1 -- 172.21.15.70:0/2031303984 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564ac0436be0 con 0x564ac05eb650 2024-04-01T01:46:43.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.182+0000 7f812b7fe640 1 --2- 172.21.15.70:0/2031303984 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564ac05eb650 0x564ac05eda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.186+0000 7f810b7fe640 10 client.29405 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:43.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.186+0000 7f812b7fe640 1 --2- 172.21.15.70:0/2031303984 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564ac05eb650 0x564ac05eda40 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=528f2d46a18f8529 server_cookie=ed8d3f5bf243658 in_seq=0 out_seq=0 2024-04-01T01:46:43.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.187+0000 7f810b7fe640 1 -- 172.21.15.70:0/2031303984 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564ac0436be0 con 0x564ac05eb650 2024-04-01T01:46:43.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.187+0000 7f810b7fe640 10 client.29405 handle_command_reply: tid=0 2024-04-01T01:46:43.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.187+0000 7f813559f740 1 client.29405 shutdown 2024-04-01T01:46:43.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.189+0000 7f813559f740 1 -- 172.21.15.70:0/2031303984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054d400 msgr2=0x564ac056d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.189+0000 7f813559f740 1 --2- 172.21.15.70:0/2031303984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054d400 0x564ac056d410 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f812002c1c0 tx=0x7f81200517d0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.189+0000 7f813559f740 1 -- 172.21.15.70:0/2031303984 shutdown_connections 2024-04-01T01:46:43.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.189+0000 7f813559f740 1 --2- 172.21.15.70:0/2031303984 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564ac05eb650 0x564ac05eda40 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.189+0000 7f813559f740 1 --2- 172.21.15.70:0/2031303984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564ac0552570 0x564ac056e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.189+0000 7f813559f740 1 --2- 172.21.15.70:0/2031303984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564ac054d400 0x564ac056d410 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.189+0000 7f813559f740 1 --2- 172.21.15.70:0/2031303984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ac054ca10 0x564ac056ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.189+0000 7f810b7fe640 0 client.29405 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:43.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.189+0000 7f813559f740 1 -- 172.21.15.70:0/2031303984 >> 172.21.15.70:0/2031303984 conn(0x564ac0547f80 msgr2=0x564ac0548360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.189+0000 7f813559f740 1 -- 172.21.15.70:0/2031303984 shutdown_connections 2024-04-01T01:46:43.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.189+0000 7f813559f740 1 -- 172.21.15.70:0/2031303984 wait complete. 2024-04-01T01:46:43.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.190+0000 7f813559f740 20 client.29405 trim_cache size 0 max 16384 2024-04-01T01:46:43.191 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:46:43.191 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:46:43.191 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "88fc7b5f-e717-4c0c-a949-9ad1ddc9acf2", 2024-04-01T01:46:43.191 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:46:43.191 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:46:43.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.190+0000 7f81329bc640 1 -- 172.21.15.70:0/1981049422 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81140a2620 msgr2=0x7f81140a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.191+0000 7f81329bc640 1 --2- 172.21.15.70:0/1981049422 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81140a2620 0x7f81140a4ae0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f812409b560 tx=0x7f81240071c0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.191+0000 7f81329bc640 1 -- 172.21.15.70:0/1981049422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f812c10da70 msgr2=0x7f812c1ade10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.191+0000 7f81329bc640 1 --2- 172.21.15.70:0/1981049422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f812c10da70 0x7f812c1ade10 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f812400cab0 tx=0x7f812400cf70 comp rx=0 tx=0).stop 2024-04-01T01:46:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.191+0000 7f81329bc640 1 -- 172.21.15.70:0/1981049422 shutdown_connections 2024-04-01T01:46:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.191+0000 7f81329bc640 1 --2- 172.21.15.70:0/1981049422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f812c10da70 0x7f812c1ade10 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.191+0000 7f81329bc640 1 --2- 172.21.15.70:0/1981049422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f812c10b0c0 0x7f812c1a6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.191+0000 7f81329bc640 1 --2- 172.21.15.70:0/1981049422 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81140a2620 0x7f81140a4ae0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.191+0000 7f81329bc640 1 --2- 172.21.15.70:0/1981049422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f812c108750 0x7f812c1a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.191+0000 7f81329bc640 1 -- 172.21.15.70:0/1981049422 >> 172.21.15.70:0/1981049422 conn(0x7f812c06e5b0 msgr2=0x7f812c10c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.191+0000 7f81329bc640 1 -- 172.21.15.70:0/1981049422 shutdown_connections 2024-04-01T01:46:43.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.191+0000 7f81329bc640 1 -- 172.21.15.70:0/1981049422 wait complete. 2024-04-01T01:46:43.215 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:42 smithi070 ceph-mon[32179]: pgmap v755: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.3 MiB/s wr, 698 op/s 2024-04-01T01:46:43.216 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:46:43.217 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:46:43.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.219+0000 7fe67ea59640 1 -- 172.21.15.70:0/2721676402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe678109440 msgr2=0x7fe67810e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.219+0000 7fe67ea59640 1 --2- 172.21.15.70:0/2721676402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe678109440 0x7fe67810e0b0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fe67401acb0 tx=0x7fe67403d130 comp rx=0 tx=0).stop 2024-04-01T01:46:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.220+0000 7fe67ea59640 1 -- 172.21.15.70:0/2721676402 shutdown_connections 2024-04-01T01:46:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.220+0000 7fe67ea59640 1 --2- 172.21.15.70:0/2721676402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe678109440 0x7fe67810e0b0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.220+0000 7fe67ea59640 1 --2- 172.21.15.70:0/2721676402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe678073360 0x7fe678108d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.220+0000 7fe67ea59640 1 --2- 172.21.15.70:0/2721676402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe678072a00 0x7fe678072e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.220+0000 7fe67ea59640 1 -- 172.21.15.70:0/2721676402 >> 172.21.15.70:0/2721676402 conn(0x7fe67806e5c0 msgr2=0x7fe678070a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.220+0000 7fe67ea59640 1 -- 172.21.15.70:0/2721676402 shutdown_connections 2024-04-01T01:46:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.221+0000 7fe67ea59640 1 -- 172.21.15.70:0/2721676402 wait complete. 2024-04-01T01:46:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.222+0000 7fe67ea59640 1 Processor -- start 2024-04-01T01:46:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.222+0000 7fe67ea59640 1 -- start start 2024-04-01T01:46:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.222+0000 7fe67ea59640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe678072a00 0x7fe67819dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.222+0000 7fe67ea59640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe678073360 0x7fe67819e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.222+0000 7fe67ea59640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe678109440 0x7fe6781c7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.222+0000 7fe67ea59640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe678111770 con 0x7fe678073360 2024-04-01T01:46:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.222+0000 7fe67ea59640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe6781115f0 con 0x7fe678072a00 2024-04-01T01:46:43.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.222+0000 7fe67ea59640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe6781118f0 con 0x7fe678109440 2024-04-01T01:46:43.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.225+0000 7fe67da57640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe678072a00 0x7fe67819dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.225+0000 7fe67da57640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe678072a00 0x7fe67819dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:43102/0 (socket says 172.21.15.70:43102) 2024-04-01T01:46:43.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.225+0000 7fe67da57640 1 -- 172.21.15.70:0/713330151 learned_addr learned my addr 172.21.15.70:0/713330151 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:43.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.225+0000 7fe67e258640 1 --2- 172.21.15.70:0/713330151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe678109440 0x7fe6781c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.225+0000 7fe67d256640 1 --2- 172.21.15.70:0/713330151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe678073360 0x7fe67819e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.225+0000 7fe67d256640 1 -- 172.21.15.70:0/713330151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe678109440 msgr2=0x7fe6781c7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.226+0000 7fe67d256640 1 --2- 172.21.15.70:0/713330151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe678109440 0x7fe6781c7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.226+0000 7fe67d256640 1 -- 172.21.15.70:0/713330151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe678072a00 msgr2=0x7fe67819dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.226+0000 7fe67d256640 1 --2- 172.21.15.70:0/713330151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe678072a00 0x7fe67819dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.226+0000 7fe67d256640 1 -- 172.21.15.70:0/713330151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6781c79b0 con 0x7fe678073360 2024-04-01T01:46:43.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.226+0000 7fe67d256640 1 --2- 172.21.15.70:0/713330151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe678073360 0x7fe67819e250 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fe66800d6a0 tx=0x7fe66800db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.230 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.227+0000 7fe66effd640 1 -- 172.21.15.70:0/713330151 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe668017070 con 0x7fe678073360 2024-04-01T01:46:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.227+0000 7fe67ea59640 1 -- 172.21.15.70:0/713330151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6781c7ca0 con 0x7fe678073360 2024-04-01T01:46:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.227+0000 7fe67ea59640 1 -- 172.21.15.70:0/713330151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6781c81a0 con 0x7fe678073360 2024-04-01T01:46:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.227+0000 7fe66effd640 1 -- 172.21.15.70:0/713330151 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe668009820 con 0x7fe678073360 2024-04-01T01:46:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.228+0000 7fe66effd640 1 -- 172.21.15.70:0/713330151 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe668004c40 con 0x7fe678073360 2024-04-01T01:46:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.233+0000 7fe66effd640 1 -- 172.21.15.70:0/713330151 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fe668010460 con 0x7fe678073360 2024-04-01T01:46:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.234+0000 7fe66effd640 1 --2- 172.21.15.70:0/713330151 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6640a26b0 0x7fe6640a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.234+0000 7fe66effd640 1 -- 172.21.15.70:0/713330151 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe6680cf290 con 0x7fe678073360 2024-04-01T01:46:43.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.235+0000 7fe67da57640 1 --2- 172.21.15.70:0/713330151 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6640a26b0 0x7fe6640a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.235+0000 7fe67da57640 1 --2- 172.21.15.70:0/713330151 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6640a26b0 0x7fe6640a4b70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe67819f260 tx=0x7fe6700091c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.236+0000 7fe682970740 1 Processor -- start 2024-04-01T01:46:43.237 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.236+0000 7fe682970740 1 -- start start 2024-04-01T01:46:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.237+0000 7fe682970740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b25cae0 0x55a07b25cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.237+0000 7fe682970740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25d4d0 0x55a07b261fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.237+0000 7fe682970740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b262640 0x55a07b264ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.237+0000 7fe682970740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a07b1c08f0 con 0x55a07b25d4d0 2024-04-01T01:46:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.237+0000 7fe682970740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a07b1698d0 con 0x55a07b262640 2024-04-01T01:46:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.237+0000 7fe682970740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a07b1bc990 con 0x55a07b25cae0 2024-04-01T01:46:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.237+0000 7fe67e258640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b262640 0x55a07b264ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.237+0000 7fe67e258640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b262640 0x55a07b264ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:43106/0 (socket says 172.21.15.70:43106) 2024-04-01T01:46:43.238 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.237+0000 7fe67e258640 1 -- 172.21.15.70:0/391277731 learned_addr learned my addr 172.21.15.70:0/391277731 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.237+0000 7fe67d256640 1 --2- 172.21.15.70:0/391277731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25d4d0 0x55a07b261fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.238+0000 7fe67e258640 1 -- 172.21.15.70:0/391277731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b25cae0 msgr2=0x55a07b25cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.238+0000 7fe67e258640 1 --2- 172.21.15.70:0/391277731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b25cae0 0x55a07b25cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.238+0000 7fe67e258640 1 -- 172.21.15.70:0/391277731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25d4d0 msgr2=0x55a07b261fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.238+0000 7fe67e258640 1 --2- 172.21.15.70:0/391277731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25d4d0 0x55a07b261fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.238+0000 7fe67e258640 1 -- 172.21.15.70:0/391277731 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a07aed5210 con 0x55a07b262640 2024-04-01T01:46:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.238+0000 7fe67e258640 1 --2- 172.21.15.70:0/391277731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b262640 0x55a07b264ad0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fe674047000 tx=0x7fe67403dce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43227c198d7a3755 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.238+0000 7fe663fff640 1 -- 172.21.15.70:0/391277731 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674013020 con 0x55a07b262640 2024-04-01T01:46:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.239+0000 7fe663fff640 1 -- 172.21.15.70:0/391277731 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe674043440 con 0x55a07b262640 2024-04-01T01:46:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.239+0000 7fe663fff640 1 -- 172.21.15.70:0/391277731 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6740042a0 con 0x55a07b262640 2024-04-01T01:46:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.240+0000 7fe682970740 1 -- 172.21.15.70:0/391277731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b262640 msgr2=0x55a07b264ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.240+0000 7fe682970740 1 --2- 172.21.15.70:0/391277731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b262640 0x55a07b264ad0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fe674047000 tx=0x7fe67403dce0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.242+0000 7fe682970740 1 -- 172.21.15.70:0/391277731 shutdown_connections 2024-04-01T01:46:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.242+0000 7fe682970740 1 --2- 172.21.15.70:0/391277731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b262640 0x55a07b264ad0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.242+0000 7fe682970740 1 --2- 172.21.15.70:0/391277731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25d4d0 0x55a07b261fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.242+0000 7fe682970740 1 --2- 172.21.15.70:0/391277731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b25cae0 0x55a07b25cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.242+0000 7fe682970740 1 -- 172.21.15.70:0/391277731 >> 172.21.15.70:0/391277731 conn(0x55a07b258050 msgr2=0x55a07b258450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.244+0000 7fe682970740 1 -- 172.21.15.70:0/391277731 shutdown_connections 2024-04-01T01:46:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.245+0000 7fe682970740 1 -- 172.21.15.70:0/391277731 wait complete. 2024-04-01T01:46:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.245+0000 7fe682970740 1 -- start start 2024-04-01T01:46:43.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.245+0000 7fe682970740 1 Processor -- start 2024-04-01T01:46:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.246+0000 7fe682970740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b25cae0 0x55a07b266d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.246+0000 7fe682970740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25d4d0 0x55a07b2672c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.246+0000 7fe682970740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b262640 0x55a07b26e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.246+0000 7fe682970740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a07b265590 con 0x55a07b25d4d0 2024-04-01T01:46:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.246+0000 7fe682970740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a07b15ef80 con 0x55a07b25cae0 2024-04-01T01:46:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.246+0000 7fe682970740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a07b265710 con 0x55a07b262640 2024-04-01T01:46:43.246 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.246+0000 7fe682970740 10 client.0 _ll_register_callbacks cb 0x55a07b1b7290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.247+0000 7fe67e258640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b262640 0x55a07b26e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.247+0000 7fe67e258640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b262640 0x55a07b26e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58048/0 (socket says 172.21.15.70:58048) 2024-04-01T01:46:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.247+0000 7fe67e258640 1 -- 172.21.15.70:0/4036835038 learned_addr learned my addr 172.21.15.70:0/4036835038 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.247+0000 7fe67d256640 1 --2- 172.21.15.70:0/4036835038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25d4d0 0x55a07b2672c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.247+0000 7fe67d256640 1 -- 172.21.15.70:0/4036835038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b262640 msgr2=0x55a07b26e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.247+0000 7fe67d256640 1 --2- 172.21.15.70:0/4036835038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b262640 0x55a07b26e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.247+0000 7fe67d256640 1 -- 172.21.15.70:0/4036835038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b25cae0 msgr2=0x55a07b266d80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:43.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.247+0000 7fe67d256640 1 --2- 172.21.15.70:0/4036835038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b25cae0 0x55a07b266d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.248+0000 7fe67d256640 1 -- 172.21.15.70:0/4036835038 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a07afa1620 con 0x55a07b25d4d0 2024-04-01T01:46:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.248+0000 7fe67e258640 1 --2- 172.21.15.70:0/4036835038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b262640 0x55a07b26e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:46:43.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:42 smithi189 ceph-mon[35753]: pgmap v755: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.3 MiB/s wr, 698 op/s 2024-04-01T01:46:43.249 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.248+0000 7fe67d256640 1 --2- 172.21.15.70:0/4036835038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25d4d0 0x55a07b2672c0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe66809b8b0 tx=0x7fe66809b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a603e27ffa11e54 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.250+0000 7fe6617fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:46:43.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.253+0000 7fe6617fa640 1 -- 172.21.15.70:0/4036835038 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6680b4060 con 0x55a07b25d4d0 2024-04-01T01:46:43.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.253+0000 7fe682970740 1 -- 172.21.15.70:0/4036835038 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe674043440 con 0x55a07b25d4d0 2024-04-01T01:46:43.253 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.253+0000 7fe6617fa640 1 -- 172.21.15.70:0/4036835038 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe668097f90 con 0x55a07b25d4d0 2024-04-01T01:46:43.255 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.255+0000 7fe6617fa640 1 -- 172.21.15.70:0/4036835038 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe668020040 con 0x55a07b25d4d0 2024-04-01T01:46:43.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.257+0000 7fe682970740 10 client.19335 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:43.261 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.257+0000 7fe682970740 1 -- 172.21.15.70:0/4036835038 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a07b2713b0 con 0x55a07b25d4d0 2024-04-01T01:46:43.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.268+0000 7fe6617fa640 1 -- 172.21.15.70:0/4036835038 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe674043440 con 0x55a07b25d4d0 2024-04-01T01:46:43.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.268+0000 7fe6617fa640 1 -- 172.21.15.70:0/4036835038 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe66809a7e0 con 0x55a07b25d4d0 2024-04-01T01:46:43.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.268+0000 7fe682970740 10 client.19335 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:43.268 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.268+0000 7fe682970740 10 client.19335 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.268+0000 7fe682970740 20 client.19335 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.268+0000 7fe682970740 1 --2- 172.21.15.70:0/4036835038 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a07b272a30 0x55a07b274e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.268+0000 7fe682970740 4 client.19335 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.268+0000 7fe682970740 1 -- 172.21.15.70:0/4036835038 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a07b1d4120 con 0x55a07b272a30 2024-04-01T01:46:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.269+0000 7fe67da57640 1 --2- 172.21.15.70:0/4036835038 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a07b272a30 0x55a07b274e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.269+0000 7fe67da57640 1 --2- 172.21.15.70:0/4036835038 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a07b272a30 0x55a07b274e80 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42a0f2c59ec28888 server_cookie=d446e083a1639abe in_seq=0 out_seq=0 2024-04-01T01:46:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.273+0000 7fe6617fa640 10 client.19335 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.274+0000 7fe6617fa640 1 -- 172.21.15.70:0/4036835038 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a07b1d4120 con 0x55a07b272a30 2024-04-01T01:46:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.274+0000 7fe6617fa640 10 client.19335 handle_command_reply: tid=0 2024-04-01T01:46:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.274+0000 7fe682970740 1 client.19335 shutdown 2024-04-01T01:46:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.275+0000 7fe682970740 1 -- 172.21.15.70:0/4036835038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25d4d0 msgr2=0x55a07b2672c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.275+0000 7fe682970740 1 --2- 172.21.15.70:0/4036835038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25d4d0 0x55a07b2672c0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fe66809b8b0 tx=0x7fe66809b9e0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.275+0000 7fe682970740 1 -- 172.21.15.70:0/4036835038 shutdown_connections 2024-04-01T01:46:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.275+0000 7fe682970740 1 --2- 172.21.15.70:0/4036835038 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a07b272a30 0x55a07b274e80 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.275+0000 7fe682970740 1 --2- 172.21.15.70:0/4036835038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b262640 0x55a07b26e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.275+0000 7fe682970740 1 --2- 172.21.15.70:0/4036835038 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25d4d0 0x55a07b2672c0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.275+0000 7fe682970740 1 --2- 172.21.15.70:0/4036835038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b25cae0 0x55a07b266d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.275+0000 7fe682970740 1 -- 172.21.15.70:0/4036835038 >> 172.21.15.70:0/4036835038 conn(0x55a07b258050 msgr2=0x55a07b257190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.277+0000 7fe6617fa640 0 client.19335 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.279+0000 7fe682970740 1 -- 172.21.15.70:0/4036835038 shutdown_connections 2024-04-01T01:46:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.280+0000 7fe682970740 1 -- 172.21.15.70:0/4036835038 wait complete. 2024-04-01T01:46:43.280 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.280+0000 7fe682970740 20 client.19335 trim_cache size 0 max 16384 2024-04-01T01:46:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.313+0000 7fe682970740 1 Processor -- start 2024-04-01T01:46:43.315 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.315+0000 7fe682970740 1 -- start start 2024-04-01T01:46:43.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.316+0000 7fe682970740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b258050 0x55a07b25b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.316+0000 7fe682970740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25cae0 0x55a07b26e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.316+0000 7fe682970740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b25d4d0 0x55a07b25b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.316+0000 7fe682970740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c00fde0 con 0x55a07b25cae0 2024-04-01T01:46:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.316+0000 7fe67da57640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b258050 0x55a07b25b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.316+0000 7fe67da57640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b258050 0x55a07b25b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:43130/0 (socket says 172.21.15.70:43130) 2024-04-01T01:46:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.316+0000 7fe67da57640 1 -- 172.21.15.70:0/530325278 learned_addr learned my addr 172.21.15.70:0/530325278 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.316+0000 7fe682970740 1 -- 172.21.15.70:0/530325278 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c011920 con 0x55a07b258050 2024-04-01T01:46:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.316+0000 7fe67d256640 1 --2- 172.21.15.70:0/530325278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25cae0 0x55a07b26e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.316+0000 7fe682970740 1 -- 172.21.15.70:0/530325278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c0104b0 con 0x55a07b25d4d0 2024-04-01T01:46:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.316+0000 7fe67e258640 1 --2- 172.21.15.70:0/530325278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b25d4d0 0x55a07b25b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.317+0000 7fe67da57640 1 -- 172.21.15.70:0/530325278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b25d4d0 msgr2=0x55a07b25b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.317+0000 7fe67da57640 1 --2- 172.21.15.70:0/530325278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b25d4d0 0x55a07b25b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.317+0000 7fe67da57640 1 -- 172.21.15.70:0/530325278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25cae0 msgr2=0x55a07b26e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.317+0000 7fe67da57640 1 --2- 172.21.15.70:0/530325278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25cae0 0x55a07b26e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.317+0000 7fe67da57640 1 -- 172.21.15.70:0/530325278 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a07b267610 con 0x55a07b258050 2024-04-01T01:46:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.320+0000 7fe67da57640 1 --2- 172.21.15.70:0/530325278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b258050 0x55a07b25b2b0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe67000ec20 tx=0x7fe670011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2eac1ae38fbc131 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.322 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.322+0000 7fe6637fe640 1 -- 172.21.15.70:0/530325278 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe670016b90 con 0x55a07b258050 2024-04-01T01:46:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.322+0000 7fe6637fe640 1 -- 172.21.15.70:0/530325278 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe670027030 con 0x55a07b258050 2024-04-01T01:46:43.323 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.322+0000 7fe6637fe640 1 -- 172.21.15.70:0/530325278 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe670021840 con 0x55a07b258050 2024-04-01T01:46:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.325+0000 7fe682970740 1 -- 172.21.15.70:0/530325278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b258050 msgr2=0x55a07b25b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.326 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.325+0000 7fe682970740 1 --2- 172.21.15.70:0/530325278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b258050 0x55a07b25b2b0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe67000ec20 tx=0x7fe670011000 comp rx=0 tx=0).stop 2024-04-01T01:46:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.325+0000 7fe682970740 1 -- 172.21.15.70:0/530325278 shutdown_connections 2024-04-01T01:46:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.325+0000 7fe682970740 1 --2- 172.21.15.70:0/530325278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b25d4d0 0x55a07b25b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.326+0000 7fe682970740 1 --2- 172.21.15.70:0/530325278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25cae0 0x55a07b26e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.326+0000 7fe682970740 1 --2- 172.21.15.70:0/530325278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b258050 0x55a07b25b2b0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.326+0000 7fe682970740 1 -- 172.21.15.70:0/530325278 >> 172.21.15.70:0/530325278 conn(0x55a07b262140 msgr2=0x55a07b25e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.326+0000 7fe682970740 1 -- 172.21.15.70:0/530325278 shutdown_connections 2024-04-01T01:46:43.327 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.326+0000 7fe682970740 1 -- 172.21.15.70:0/530325278 wait complete. 2024-04-01T01:46:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.327+0000 7fe682970740 1 -- start start 2024-04-01T01:46:43.328 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.327+0000 7fe682970740 1 Processor -- start 2024-04-01T01:46:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.327+0000 7fe682970740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b258050 0x55a07b26b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.327+0000 7fe682970740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b25cae0 0x55a07b26ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.328+0000 7fe682970740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25d4d0 0x55a07b26ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.328+0000 7fe67da57640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b258050 0x55a07b26b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.328+0000 7fe67da57640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b258050 0x55a07b26b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58072/0 (socket says 172.21.15.70:58072) 2024-04-01T01:46:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.328+0000 7fe682970740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c011250 con 0x55a07b25d4d0 2024-04-01T01:46:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.328+0000 7fe682970740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c010b80 con 0x55a07b25cae0 2024-04-01T01:46:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.328+0000 7fe682970740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c00ddc0 con 0x55a07b258050 2024-04-01T01:46:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.328+0000 7fe67da57640 1 -- 172.21.15.70:0/1564423328 learned_addr learned my addr 172.21.15.70:0/1564423328 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.328+0000 7fe682970740 10 client.0 _ll_register_callbacks cb 0x7fe66809b280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.328+0000 7fe67da57640 1 -- 172.21.15.70:0/1564423328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b25cae0 msgr2=0x55a07b26ba40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.328+0000 7fe67da57640 1 --2- 172.21.15.70:0/1564423328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b25cae0 0x55a07b26ba40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.328+0000 7fe67da57640 1 -- 172.21.15.70:0/1564423328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25d4d0 msgr2=0x55a07b26ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.329 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.328+0000 7fe67da57640 1 --2- 172.21.15.70:0/1564423328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25d4d0 0x55a07b26ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.328+0000 7fe67da57640 1 -- 172.21.15.70:0/1564423328 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a07b26d040 con 0x55a07b258050 2024-04-01T01:46:43.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.328+0000 7fe67da57640 1 --2- 172.21.15.70:0/1564423328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b258050 0x55a07b26b500 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe670016dc0 tx=0x7fe6700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3eb076b26bf604 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.330 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.329+0000 7fe6617fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:46:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.330+0000 7fe6617fa640 1 -- 172.21.15.70:0/1564423328 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe670032070 con 0x55a07b258050 2024-04-01T01:46:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.330+0000 7fe6617fa640 1 -- 172.21.15.70:0/1564423328 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe6700234e0 con 0x55a07b258050 2024-04-01T01:46:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.330+0000 7fe682970740 1 -- 172.21.15.70:0/1564423328 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe670027030 con 0x55a07b258050 2024-04-01T01:46:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.331+0000 7fe6617fa640 1 -- 172.21.15.70:0/1564423328 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe670016e20 con 0x55a07b258050 2024-04-01T01:46:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.331+0000 7fe682970740 10 client.29194 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.331+0000 7fe682970740 1 -- 172.21.15.70:0/1564423328 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a07b26e360 con 0x55a07b258050 2024-04-01T01:46:43.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.331+0000 7fe6617fa640 1 -- 172.21.15.70:0/1564423328 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe670027030 con 0x55a07b258050 2024-04-01T01:46:43.332 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.331+0000 7fe6617fa640 1 -- 172.21.15.70:0/1564423328 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe670040620 con 0x55a07b258050 2024-04-01T01:46:43.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.332+0000 7fe682970740 10 client.29194 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:43.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.332+0000 7fe682970740 10 client.29194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:43.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.332+0000 7fe682970740 20 client.29194 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:43.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.332+0000 7fe682970740 1 --2- 172.21.15.70:0/1564423328 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a07b262640 0x55a07b2db170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.332+0000 7fe682970740 4 client.29194 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:46:43.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.332+0000 7fe682970740 1 -- 172.21.15.70:0/1564423328 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a07b1cde30 con 0x55a07b262640 2024-04-01T01:46:43.333 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.332+0000 7fe67d256640 1 --2- 172.21.15.70:0/1564423328 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a07b262640 0x55a07b2db170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.338+0000 7fe6617fa640 10 client.29194 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:43.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.338+0000 7fe67d256640 1 --2- 172.21.15.70:0/1564423328 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a07b262640 0x55a07b2db170 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b1e9aeca02183fc server_cookie=4c00345992da42ed in_seq=0 out_seq=0 2024-04-01T01:46:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.369+0000 7fe6617fa640 1 -- 172.21.15.70:0/1564423328 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a07b1cde30 con 0x55a07b262640 2024-04-01T01:46:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.369+0000 7fe6617fa640 10 client.29194 handle_command_reply: tid=0 2024-04-01T01:46:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.369+0000 7fe682970740 1 client.29194 shutdown 2024-04-01T01:46:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.369+0000 7fe682970740 1 -- 172.21.15.70:0/1564423328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b258050 msgr2=0x55a07b26b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.369+0000 7fe682970740 1 --2- 172.21.15.70:0/1564423328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b258050 0x55a07b26b500 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe670016dc0 tx=0x7fe6700046c0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.369+0000 7fe682970740 1 -- 172.21.15.70:0/1564423328 shutdown_connections 2024-04-01T01:46:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.369+0000 7fe682970740 1 --2- 172.21.15.70:0/1564423328 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a07b262640 0x55a07b2db170 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.370+0000 7fe682970740 1 --2- 172.21.15.70:0/1564423328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a07b25d4d0 0x55a07b26ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.370+0000 7fe682970740 1 --2- 172.21.15.70:0/1564423328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a07b25cae0 0x55a07b26ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.370+0000 7fe682970740 1 --2- 172.21.15.70:0/1564423328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a07b258050 0x55a07b26b500 secure :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe670016dc0 tx=0x7fe6700046c0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.370+0000 7fe682970740 1 -- 172.21.15.70:0/1564423328 >> 172.21.15.70:0/1564423328 conn(0x55a07b262140 msgr2=0x55a07b256900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.370+0000 7fe6617fa640 0 client.29194 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.370+0000 7fe682970740 1 -- 172.21.15.70:0/1564423328 shutdown_connections 2024-04-01T01:46:43.371 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.370+0000 7fe682970740 1 -- 172.21.15.70:0/1564423328 wait complete. 2024-04-01T01:46:43.372 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.371+0000 7fe682970740 20 client.29194 trim_cache size 0 max 16384 2024-04-01T01:46:43.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:46:43.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:46:43.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:46:43.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:46:43.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.373+0000 7fe67ea59640 1 -- 172.21.15.70:0/713330151 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6640a26b0 msgr2=0x7fe6640a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.373+0000 7fe67ea59640 1 --2- 172.21.15.70:0/713330151 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6640a26b0 0x7fe6640a4b70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe67819f260 tx=0x7fe6700091c0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.373+0000 7fe67ea59640 1 -- 172.21.15.70:0/713330151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe678073360 msgr2=0x7fe67819e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.373+0000 7fe67ea59640 1 --2- 172.21.15.70:0/713330151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe678073360 0x7fe67819e250 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fe66800d6a0 tx=0x7fe66800db60 comp rx=0 tx=0).stop 2024-04-01T01:46:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.377+0000 7fe67ea59640 1 -- 172.21.15.70:0/713330151 shutdown_connections 2024-04-01T01:46:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.377+0000 7fe67ea59640 1 --2- 172.21.15.70:0/713330151 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe6640a26b0 0x7fe6640a4b70 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.377+0000 7fe67ea59640 1 --2- 172.21.15.70:0/713330151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe678109440 0x7fe6781c7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.377+0000 7fe67ea59640 1 --2- 172.21.15.70:0/713330151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe678073360 0x7fe67819e250 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.377+0000 7fe67ea59640 1 --2- 172.21.15.70:0/713330151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe678072a00 0x7fe67819dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.378 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.377+0000 7fe67ea59640 1 -- 172.21.15.70:0/713330151 >> 172.21.15.70:0/713330151 conn(0x7fe67806e5c0 msgr2=0x7fe67810c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.378+0000 7fe67ea59640 1 -- 172.21.15.70:0/713330151 shutdown_connections 2024-04-01T01:46:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:43.379+0000 7fe67ea59640 1 -- 172.21.15.70:0/713330151 wait complete. 2024-04-01T01:46:43.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.431+0000 7f24f6fc9640 1 -- 172.21.15.70:0/199757247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24f0104b80 msgr2=0x7f24f0105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.431+0000 7f24f6fc9640 1 --2- 172.21.15.70:0/199757247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24f0104b80 0x7f24f0105000 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009860 tx=0x7f24e402bb70 comp rx=0 tx=0).stop 2024-04-01T01:46:43.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.431+0000 7f24f6fc9640 1 -- 172.21.15.70:0/199757247 shutdown_connections 2024-04-01T01:46:43.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.431+0000 7f24f6fc9640 1 --2- 172.21.15.70:0/199757247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24f0105540 0x7f24f010bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.431+0000 7f24f6fc9640 1 --2- 172.21.15.70:0/199757247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24f0104b80 0x7f24f0105000 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.431+0000 7f24f6fc9640 1 --2- 172.21.15.70:0/199757247 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f0103990 0x7f24f0103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.431+0000 7f24f6fc9640 1 -- 172.21.15.70:0/199757247 >> 172.21.15.70:0/199757247 conn(0x7f24f00fef00 msgr2=0x7f24f0101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.431+0000 7f24f6fc9640 1 -- 172.21.15.70:0/199757247 shutdown_connections 2024-04-01T01:46:43.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.431+0000 7f24f6fc9640 1 -- 172.21.15.70:0/199757247 wait complete. 2024-04-01T01:46:43.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.432+0000 7f24f6fc9640 1 Processor -- start 2024-04-01T01:46:43.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.432+0000 7f24f6fc9640 1 -- start start 2024-04-01T01:46:43.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.432+0000 7f24f6fc9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24f0103990 0x7f24f019e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.432+0000 7f24f6fc9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24f0104b80 0x7f24f019e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.433+0000 7f24f6fc9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f0105540 0x7f24f01a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.433+0000 7f24f6fc9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f24f0077840 con 0x7f24f0103990 2024-04-01T01:46:43.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.433+0000 7f24f6fc9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f24f00776c0 con 0x7f24f0105540 2024-04-01T01:46:43.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.433+0000 7f24f6fc9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f24f00779c0 con 0x7f24f0104b80 2024-04-01T01:46:43.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.433+0000 7f24f4d65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24f0103990 0x7f24f019e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.433+0000 7f24f5566640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f0105540 0x7f24f01a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.433+0000 7f24f4d65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24f0103990 0x7f24f019e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39504/0 (socket says 172.21.15.70:39504) 2024-04-01T01:46:43.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.433+0000 7f24f5566640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f0105540 0x7f24f01a55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:43158/0 (socket says 172.21.15.70:43158) 2024-04-01T01:46:43.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.433+0000 7f24f4d65640 1 -- 172.21.15.70:0/1007206753 learned_addr learned my addr 172.21.15.70:0/1007206753 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:43.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.433+0000 7f24effff640 1 --2- 172.21.15.70:0/1007206753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24f0104b80 0x7f24f019e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.433+0000 7f24f5566640 1 -- 172.21.15.70:0/1007206753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24f0104b80 msgr2=0x7f24f019e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.433+0000 7f24f5566640 1 --2- 172.21.15.70:0/1007206753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24f0104b80 0x7f24f019e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.433+0000 7f24f5566640 1 -- 172.21.15.70:0/1007206753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24f0103990 msgr2=0x7f24f019e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.433+0000 7f24f5566640 1 --2- 172.21.15.70:0/1007206753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24f0103990 0x7f24f019e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.433+0000 7f24f5566640 1 -- 172.21.15.70:0/1007206753 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f01a5c10 con 0x7f24f0105540 2024-04-01T01:46:43.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.434+0000 7f24f5566640 1 --2- 172.21.15.70:0/1007206753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f0105540 0x7f24f01a55a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f24e8009fd0 tx=0x7f24e800eba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.434+0000 7f24edffb640 1 -- 172.21.15.70:0/1007206753 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8004220 con 0x7f24f0105540 2024-04-01T01:46:43.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.435+0000 7f24edffb640 1 -- 172.21.15.70:0/1007206753 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24e80043c0 con 0x7f24f0105540 2024-04-01T01:46:43.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.435+0000 7f24edffb640 1 -- 172.21.15.70:0/1007206753 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8005650 con 0x7f24f0105540 2024-04-01T01:46:43.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.435+0000 7f24f6fc9640 1 -- 172.21.15.70:0/1007206753 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f01a5f00 con 0x7f24f0105540 2024-04-01T01:46:43.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.436+0000 7f24f6fc9640 1 -- 172.21.15.70:0/1007206753 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f01a6370 con 0x7f24f0105540 2024-04-01T01:46:43.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.440+0000 7f24f9bac740 1 Processor -- start 2024-04-01T01:46:43.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.440+0000 7f24edffb640 1 -- 172.21.15.70:0/1007206753 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f24e8004a00 con 0x7f24f0105540 2024-04-01T01:46:43.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.440+0000 7f24f9bac740 1 -- start start 2024-04-01T01:46:43.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.440+0000 7f24f9bac740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abf72b0 0x55cf1abf76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.440+0000 7f24f9bac740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf7ca0 0x55cf1abfc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.440+0000 7f24f9bac740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abfce10 0x55cf1abff2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.440+0000 7f24f9bac740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cf1ab5c180 con 0x55cf1abfce10 2024-04-01T01:46:43.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.440+0000 7f24f9bac740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55cf1ab04b70 con 0x55cf1abf7ca0 2024-04-01T01:46:43.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.440+0000 7f24f9bac740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cf1ab58220 con 0x55cf1abf72b0 2024-04-01T01:46:43.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.440+0000 7f24f4d65640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf7ca0 0x55cf1abfc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.440+0000 7f24f4d65640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf7ca0 0x55cf1abfc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:43160/0 (socket says 172.21.15.70:43160) 2024-04-01T01:46:43.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.440+0000 7f24f4d65640 1 -- 172.21.15.70:0/471859236 learned_addr learned my addr 172.21.15.70:0/471859236 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:43.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.441+0000 7f24effff640 1 --2- 172.21.15.70:0/471859236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abf72b0 0x55cf1abf76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.441+0000 7f24edffb640 1 --2- 172.21.15.70:0/1007206753 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24d80a26a0 0x7f24d80a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.441+0000 7f24effff640 1 --2- 172.21.15.70:0/471859236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abfce10 0x55cf1abff2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.441+0000 7f24effff640 1 -- 172.21.15.70:0/471859236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf7ca0 msgr2=0x55cf1abfc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.441+0000 7f24effff640 1 --2- 172.21.15.70:0/471859236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf7ca0 0x55cf1abfc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.441+0000 7f24effff640 1 -- 172.21.15.70:0/471859236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abfce10 msgr2=0x55cf1abff2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.441+0000 7f24effff640 1 --2- 172.21.15.70:0/471859236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abfce10 0x55cf1abff2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.441+0000 7f24effff640 1 -- 172.21.15.70:0/471859236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf1ab187c0 con 0x55cf1abf72b0 2024-04-01T01:46:43.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.441+0000 7f24f4d65640 1 --2- 172.21.15.70:0/471859236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf7ca0 0x55cf1abfc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:46:43.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.441+0000 7f24f5566640 1 --2- 172.21.15.70:0/1007206753 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24d80a26a0 0x7f24d80a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.442+0000 7f24effff640 1 --2- 172.21.15.70:0/471859236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abf72b0 0x55cf1abf76d0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009830 tx=0x7f24e40056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9be3db2aa938c86 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.442+0000 7f24edffb640 1 -- 172.21.15.70:0/1007206753 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f24e80cf6f0 con 0x7f24f0105540 2024-04-01T01:46:43.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.442+0000 7f24f5566640 1 --2- 172.21.15.70:0/1007206753 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24d80a26a0 0x7f24d80a4b60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f24e809bd10 tx=0x7f24e8004570 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.442+0000 7f24d6ffd640 1 -- 172.21.15.70:0/471859236 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e4039040 con 0x55cf1abf72b0 2024-04-01T01:46:43.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.442+0000 7f24d6ffd640 1 -- 172.21.15.70:0/471859236 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24e4047070 con 0x55cf1abf72b0 2024-04-01T01:46:43.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.442+0000 7f24d6ffd640 1 -- 172.21.15.70:0/471859236 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e402ca40 con 0x55cf1abf72b0 2024-04-01T01:46:43.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.443+0000 7f24f9bac740 1 -- 172.21.15.70:0/471859236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abf72b0 msgr2=0x55cf1abf76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.443+0000 7f24f9bac740 1 --2- 172.21.15.70:0/471859236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abf72b0 0x55cf1abf76d0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009830 tx=0x7f24e40056d0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.443+0000 7f24f9bac740 1 -- 172.21.15.70:0/471859236 shutdown_connections 2024-04-01T01:46:43.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.443+0000 7f24f9bac740 1 --2- 172.21.15.70:0/471859236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abfce10 0x55cf1abff2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.443+0000 7f24f9bac740 1 --2- 172.21.15.70:0/471859236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf7ca0 0x55cf1abfc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.443+0000 7f24f9bac740 1 --2- 172.21.15.70:0/471859236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abf72b0 0x55cf1abf76d0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.443+0000 7f24f9bac740 1 -- 172.21.15.70:0/471859236 >> 172.21.15.70:0/471859236 conn(0x55cf1abf2820 msgr2=0x55cf1abf2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.444+0000 7f24f9bac740 1 -- 172.21.15.70:0/471859236 shutdown_connections 2024-04-01T01:46:43.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.444+0000 7f24f9bac740 1 -- 172.21.15.70:0/471859236 wait complete. 2024-04-01T01:46:43.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.444+0000 7f24f9bac740 1 -- start start 2024-04-01T01:46:43.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.444+0000 7f24f9bac740 1 Processor -- start 2024-04-01T01:46:43.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.445+0000 7f24f9bac740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf72b0 0x55cf1ac12820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.445+0000 7f24f9bac740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abf7ca0 0x55cf1ac12d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.445+0000 7f24f9bac740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abfce10 0x55cf1ac19da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.445+0000 7f24effff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf72b0 0x55cf1ac12820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.445+0000 7f24effff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf72b0 0x55cf1ac12820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:43162/0 (socket says 172.21.15.70:43162) 2024-04-01T01:46:43.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.445+0000 7f24effff640 1 -- 172.21.15.70:0/3777949804 learned_addr learned my addr 172.21.15.70:0/3777949804 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:43.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.445+0000 7f24f9bac740 1 -- 172.21.15.70:0/3777949804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cf1ac10e10 con 0x55cf1abf7ca0 2024-04-01T01:46:43.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.445+0000 7f24f9bac740 1 -- 172.21.15.70:0/3777949804 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55cf1aafa4a0 con 0x55cf1abf72b0 2024-04-01T01:46:43.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.445+0000 7f24f9bac740 1 -- 172.21.15.70:0/3777949804 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55cf1ac10f90 con 0x55cf1abfce10 2024-04-01T01:46:43.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.445+0000 7f24f9bac740 10 client.0 _ll_register_callbacks cb 0x55cf1aafdd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:43.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.445+0000 7f24f4d65640 1 --2- 172.21.15.70:0/3777949804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abf7ca0 0x55cf1ac12d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.446+0000 7f24f4d65640 1 -- 172.21.15.70:0/3777949804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abfce10 msgr2=0x55cf1ac19da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:46:43.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.446+0000 7f24f4d65640 1 --2- 172.21.15.70:0/3777949804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abfce10 0x55cf1ac19da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.446+0000 7f24f4d65640 1 -- 172.21.15.70:0/3777949804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf72b0 msgr2=0x55cf1ac12820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.446+0000 7f24f4d65640 1 --2- 172.21.15.70:0/3777949804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf72b0 0x55cf1ac12820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.446+0000 7f24f4d65640 1 -- 172.21.15.70:0/3777949804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf1a93a7d0 con 0x55cf1abf7ca0 2024-04-01T01:46:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.446+0000 7f24effff640 1 --2- 172.21.15.70:0/3777949804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf72b0 0x55cf1ac12820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:46:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.446+0000 7f24f4d65640 1 --2- 172.21.15.70:0/3777949804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abf7ca0 0x55cf1ac12d60 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x55cf1abf80a0 tx=0x7f24e0013bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84fcbc09c593ea server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.446+0000 7f24cffff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:46:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.447+0000 7f24cffff640 1 -- 172.21.15.70:0/3777949804 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e0011e00 con 0x55cf1abf7ca0 2024-04-01T01:46:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.447+0000 7f24cffff640 1 -- 172.21.15.70:0/3777949804 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f24e0004b20 con 0x55cf1abf7ca0 2024-04-01T01:46:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.447+0000 7f24f9bac740 1 -- 172.21.15.70:0/3777949804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24e4047070 con 0x55cf1abf7ca0 2024-04-01T01:46:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.447+0000 7f24cffff640 1 -- 172.21.15.70:0/3777949804 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f24e0026050 con 0x55cf1abf7ca0 2024-04-01T01:46:43.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.447+0000 7f24f9bac740 10 client.19350 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:43.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.447+0000 7f24f9bac740 1 -- 172.21.15.70:0/3777949804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cf1a8703f0 con 0x55cf1abf7ca0 2024-04-01T01:46:43.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.448+0000 7f24cffff640 1 -- 172.21.15.70:0/3777949804 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24e0004d20 con 0x55cf1abf7ca0 2024-04-01T01:46:43.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.448+0000 7f24cffff640 1 -- 172.21.15.70:0/3777949804 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f24e001a030 con 0x55cf1abf7ca0 2024-04-01T01:46:43.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.448+0000 7f24f9bac740 10 client.19350 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:43.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.448+0000 7f24f9bac740 10 client.19350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:43.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.448+0000 7f24f9bac740 20 client.19350 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:43.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.448+0000 7f24f9bac740 1 --2- 172.21.15.70:0/3777949804 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cf1ac1e7a0 0x55cf1ac20c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.448+0000 7f24f9bac740 4 client.19350 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:43.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.448+0000 7f24f9bac740 1 -- 172.21.15.70:0/3777949804 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f24e4047070 con 0x55cf1ac1e7a0 2024-04-01T01:46:43.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.448+0000 7f24effff640 1 --2- 172.21.15.70:0/3777949804 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cf1ac1e7a0 0x55cf1ac20c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.449+0000 7f24effff640 1 --2- 172.21.15.70:0/3777949804 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cf1ac1e7a0 0x55cf1ac20c20 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da56473943667392 server_cookie=862f91bc2155a987 in_seq=0 out_seq=0 2024-04-01T01:46:43.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.449+0000 7f24cffff640 10 client.19350 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:43.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.450+0000 7f24cffff640 1 -- 172.21.15.70:0/3777949804 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f24e4047070 con 0x55cf1ac1e7a0 2024-04-01T01:46:43.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.451+0000 7f24cffff640 10 client.19350 handle_command_reply: tid=0 2024-04-01T01:46:43.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.451+0000 7f24f9bac740 1 client.19350 shutdown 2024-04-01T01:46:43.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.452+0000 7f24f9bac740 1 -- 172.21.15.70:0/3777949804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abf7ca0 msgr2=0x55cf1ac12d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.452+0000 7f24f9bac740 1 --2- 172.21.15.70:0/3777949804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abf7ca0 0x55cf1ac12d60 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x55cf1abf80a0 tx=0x7f24e0013bf0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.452+0000 7f24f9bac740 1 -- 172.21.15.70:0/3777949804 shutdown_connections 2024-04-01T01:46:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.452+0000 7f24f9bac740 1 --2- 172.21.15.70:0/3777949804 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cf1ac1e7a0 0x55cf1ac20c20 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.452+0000 7f24f9bac740 1 --2- 172.21.15.70:0/3777949804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abfce10 0x55cf1ac19da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.452+0000 7f24f9bac740 1 --2- 172.21.15.70:0/3777949804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abf7ca0 0x55cf1ac12d60 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.452+0000 7f24f9bac740 1 --2- 172.21.15.70:0/3777949804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf72b0 0x55cf1ac12820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.452+0000 7f24f9bac740 1 -- 172.21.15.70:0/3777949804 >> 172.21.15.70:0/3777949804 conn(0x55cf1abf2820 msgr2=0x55cf1abf1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.452+0000 7f24cffff640 0 client.19350 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:43.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.453+0000 7f24f9bac740 1 -- 172.21.15.70:0/3777949804 shutdown_connections 2024-04-01T01:46:43.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.453+0000 7f24f9bac740 1 -- 172.21.15.70:0/3777949804 wait complete. 2024-04-01T01:46:43.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.453+0000 7f24f9bac740 20 client.19350 trim_cache size 0 max 16384 2024-04-01T01:46:43.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.494+0000 7f24f9bac740 1 Processor -- start 2024-04-01T01:46:43.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.494+0000 7f24f9bac740 1 -- start start 2024-04-01T01:46:43.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.494+0000 7f24f9bac740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf72b0 0x55cf1ac12820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.494+0000 7f24f9bac740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abf7ca0 0x55cf1ac19d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.494+0000 7f24f9bac740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abfce10 0x55cf1ac12d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.494+0000 7f24f9bac740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f24f01a8120 con 0x55cf1abfce10 2024-04-01T01:46:43.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.494+0000 7f24f9bac740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f24f01a9c60 con 0x55cf1abf72b0 2024-04-01T01:46:43.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.494+0000 7f24f9bac740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f24f01a87f0 con 0x55cf1abf7ca0 2024-04-01T01:46:43.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.494+0000 7f24effff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf72b0 0x55cf1ac12820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.494+0000 7f24effff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf72b0 0x55cf1ac12820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:43172/0 (socket says 172.21.15.70:43172) 2024-04-01T01:46:43.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.494+0000 7f24effff640 1 -- 172.21.15.70:0/3958327886 learned_addr learned my addr 172.21.15.70:0/3958327886 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:43.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.494+0000 7f24f4d65640 1 --2- 172.21.15.70:0/3958327886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abf7ca0 0x55cf1ac19d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.495+0000 7f24effff640 1 -- 172.21.15.70:0/3958327886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abf7ca0 msgr2=0x55cf1ac19d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.495+0000 7f24effff640 1 --2- 172.21.15.70:0/3958327886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abf7ca0 0x55cf1ac19d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.495+0000 7f24effff640 1 -- 172.21.15.70:0/3958327886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abfce10 msgr2=0x55cf1ac12d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:43.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.495+0000 7f24effff640 1 --2- 172.21.15.70:0/3958327886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abfce10 0x55cf1ac12d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.495+0000 7f24effff640 1 -- 172.21.15.70:0/3958327886 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf1a8300f0 con 0x55cf1abf72b0 2024-04-01T01:46:43.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.495+0000 7f24effff640 1 --2- 172.21.15.70:0/3958327886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf72b0 0x55cf1ac12820 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009780 tx=0x7f24e4005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2ae63e5038a6761 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.496+0000 7f24d67fc640 1 -- 172.21.15.70:0/3958327886 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e4039040 con 0x55cf1abf72b0 2024-04-01T01:46:43.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.496+0000 7f24d67fc640 1 -- 172.21.15.70:0/3958327886 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24e4051070 con 0x55cf1abf72b0 2024-04-01T01:46:43.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.496+0000 7f24d67fc640 1 -- 172.21.15.70:0/3958327886 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e4052480 con 0x55cf1abf72b0 2024-04-01T01:46:43.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.497+0000 7f24f9bac740 1 -- 172.21.15.70:0/3958327886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf72b0 msgr2=0x55cf1ac12820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.497+0000 7f24f9bac740 1 --2- 172.21.15.70:0/3958327886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf72b0 0x55cf1ac12820 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009780 tx=0x7f24e4005850 comp rx=0 tx=0).stop 2024-04-01T01:46:43.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.497+0000 7f24f9bac740 1 -- 172.21.15.70:0/3958327886 shutdown_connections 2024-04-01T01:46:43.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.497+0000 7f24f9bac740 1 --2- 172.21.15.70:0/3958327886 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abfce10 0x55cf1ac12d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.497+0000 7f24f9bac740 1 --2- 172.21.15.70:0/3958327886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abf7ca0 0x55cf1ac19d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.497+0000 7f24f9bac740 1 --2- 172.21.15.70:0/3958327886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abf72b0 0x55cf1ac12820 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.497+0000 7f24f9bac740 1 -- 172.21.15.70:0/3958327886 >> 172.21.15.70:0/3958327886 conn(0x55cf1abf2820 msgr2=0x55cf1abf2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.499+0000 7f24f9bac740 1 -- 172.21.15.70:0/3958327886 shutdown_connections 2024-04-01T01:46:43.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.499+0000 7f24f9bac740 1 -- 172.21.15.70:0/3958327886 wait complete. 2024-04-01T01:46:43.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.500+0000 7f24f9bac740 1 -- start start 2024-04-01T01:46:43.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.500+0000 7f24f9bac740 1 Processor -- start 2024-04-01T01:46:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.500+0000 7f24f9bac740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abf72b0 0x55cf1ac17560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.500+0000 7f24f9bac740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abf7ca0 0x55cf1ac17aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.500+0000 7f24f9bac740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abfce10 0x55cf1ac18a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.500+0000 7f24f9bac740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f24f01a9590 con 0x55cf1abf7ca0 2024-04-01T01:46:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.500+0000 7f24f9bac740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f24f01a8ec0 con 0x55cf1abfce10 2024-04-01T01:46:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.500+0000 7f24f9bac740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f24f0115b70 con 0x55cf1abf72b0 2024-04-01T01:46:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.500+0000 7f24f9bac740 10 client.0 _ll_register_callbacks cb 0x7f24e0005bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:43.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.501+0000 7f24f4d65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abf7ca0 0x55cf1ac17aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.501+0000 7f24f4d65640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abf7ca0 0x55cf1ac17aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39548/0 (socket says 172.21.15.70:39548) 2024-04-01T01:46:43.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.501+0000 7f24f4d65640 1 -- 172.21.15.70:0/2835827160 learned_addr learned my addr 172.21.15.70:0/2835827160 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:43.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.501+0000 7f24f4d65640 1 -- 172.21.15.70:0/2835827160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abf72b0 msgr2=0x55cf1ac17560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:46:43.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.501+0000 7f24f4d65640 1 --2- 172.21.15.70:0/2835827160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abf72b0 0x55cf1ac17560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.501+0000 7f24f4d65640 1 -- 172.21.15.70:0/2835827160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abfce10 msgr2=0x55cf1ac18a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:46:43.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.501+0000 7f24f4d65640 1 --2- 172.21.15.70:0/2835827160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abfce10 0x55cf1ac18a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.501+0000 7f24f4d65640 1 -- 172.21.15.70:0/2835827160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf1ac19000 con 0x55cf1abf7ca0 2024-04-01T01:46:43.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.502+0000 7f24f4d65640 1 --2- 172.21.15.70:0/2835827160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abf7ca0 0x55cf1ac17aa0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f24e0004fd0 tx=0x7f24e0011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ead16d530b89dce server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:43.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.502+0000 7f24cffff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:46:43.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.502+0000 7f24cffff640 1 -- 172.21.15.70:0/2835827160 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e0004be0 con 0x55cf1abf7ca0 2024-04-01T01:46:43.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.502+0000 7f24f9bac740 1 -- 172.21.15.70:0/2835827160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24e4051070 con 0x55cf1abf7ca0 2024-04-01T01:46:43.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.502+0000 7f24cffff640 1 -- 172.21.15.70:0/2835827160 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f24e0004d80 con 0x55cf1abf7ca0 2024-04-01T01:46:43.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.508+0000 7f24cffff640 1 -- 172.21.15.70:0/2835827160 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f24e003d050 con 0x55cf1abf7ca0 2024-04-01T01:46:43.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.508+0000 7f24f9bac740 10 client.19353 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:43.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.508+0000 7f24f9bac740 1 -- 172.21.15.70:0/2835827160 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cf1ac84c00 con 0x55cf1abf7ca0 2024-04-01T01:46:43.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.508+0000 7f24cffff640 1 -- 172.21.15.70:0/2835827160 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24e4051070 con 0x55cf1abf7ca0 2024-04-01T01:46:43.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.508+0000 7f24cffff640 1 -- 172.21.15.70:0/2835827160 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f24e001a040 con 0x55cf1abf7ca0 2024-04-01T01:46:43.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.508+0000 7f24f9bac740 10 client.19353 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:43.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.508+0000 7f24f9bac740 10 client.19353 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:43.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.508+0000 7f24f9bac740 20 client.19353 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:43.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.508+0000 7f24f9bac740 1 --2- 172.21.15.70:0/2835827160 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cf1ac855b0 0x55cf1ac879d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:43.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.508+0000 7f24f9bac740 4 client.19353 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:46:43.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.508+0000 7f24f9bac740 1 -- 172.21.15.70:0/2835827160 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cf1ab696c0 con 0x55cf1ac855b0 2024-04-01T01:46:43.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.510+0000 7f24effff640 1 --2- 172.21.15.70:0/2835827160 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cf1ac855b0 0x55cf1ac879d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:43.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.510+0000 7f24effff640 1 --2- 172.21.15.70:0/2835827160 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cf1ac855b0 0x55cf1ac879d0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=305410807cbf3b84 server_cookie=f07ccbbbe057e129 in_seq=0 out_seq=0 2024-04-01T01:46:43.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.510+0000 7f24cffff640 10 client.19353 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:43.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.511+0000 7f24cffff640 1 -- 172.21.15.70:0/2835827160 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55cf1ab696c0 con 0x55cf1ac855b0 2024-04-01T01:46:43.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.511+0000 7f24cffff640 10 client.19353 handle_command_reply: tid=0 2024-04-01T01:46:43.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.511+0000 7f24f9bac740 1 client.19353 shutdown 2024-04-01T01:46:43.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.517+0000 7f24f9bac740 1 -- 172.21.15.70:0/2835827160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abf7ca0 msgr2=0x55cf1ac17aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.517+0000 7f24f9bac740 1 --2- 172.21.15.70:0/2835827160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abf7ca0 0x55cf1ac17aa0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f24e0004fd0 tx=0x7f24e0011e30 comp rx=0 tx=0).stop 2024-04-01T01:46:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.518+0000 7f24f9bac740 1 -- 172.21.15.70:0/2835827160 shutdown_connections 2024-04-01T01:46:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.518+0000 7f24f9bac740 1 --2- 172.21.15.70:0/2835827160 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55cf1ac855b0 0x55cf1ac879d0 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.518+0000 7f24f9bac740 1 --2- 172.21.15.70:0/2835827160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cf1abfce10 0x55cf1ac18a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.518+0000 7f24f9bac740 1 --2- 172.21.15.70:0/2835827160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf1abf7ca0 0x55cf1ac17aa0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.518+0000 7f24f9bac740 1 --2- 172.21.15.70:0/2835827160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf1abf72b0 0x55cf1ac17560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.518+0000 7f24f9bac740 1 -- 172.21.15.70:0/2835827160 >> 172.21.15.70:0/2835827160 conn(0x55cf1abf2820 msgr2=0x55cf1abf9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.519+0000 7f24cffff640 0 client.19353 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:43.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.521+0000 7f24f9bac740 1 -- 172.21.15.70:0/2835827160 shutdown_connections 2024-04-01T01:46:43.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.521+0000 7f24f9bac740 1 -- 172.21.15.70:0/2835827160 wait complete. 2024-04-01T01:46:43.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.522+0000 7f24f9bac740 20 client.19353 trim_cache size 0 max 16384 2024-04-01T01:46:43.523 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:46:43.523 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:46:43.523 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:46:43.523 INFO:teuthology.orchestra.run.smithi070.stdout: "88fc7b5f-e717-4c0c-a949-9ad1ddc9acf2": { 2024-04-01T01:46:43.523 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:46:43.523 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "88fc7b5f-e717-4c0c-a949-9ad1ddc9acf2", 2024-04-01T01:46:43.524 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:46:43.524 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:46:43.524 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:46:43.524 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:46:43.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.524+0000 7f24f6fc9640 1 -- 172.21.15.70:0/1007206753 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24d80a26a0 msgr2=0x7f24d80a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.524+0000 7f24f6fc9640 1 --2- 172.21.15.70:0/1007206753 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24d80a26a0 0x7f24d80a4b60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f24e809bd10 tx=0x7f24e8004570 comp rx=0 tx=0).stop 2024-04-01T01:46:43.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.524+0000 7f24f6fc9640 1 -- 172.21.15.70:0/1007206753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f0105540 msgr2=0x7f24f01a55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:43.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.524+0000 7f24f6fc9640 1 --2- 172.21.15.70:0/1007206753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f0105540 0x7f24f01a55a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f24e8009fd0 tx=0x7f24e800eba0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.525+0000 7f24f6fc9640 1 -- 172.21.15.70:0/1007206753 shutdown_connections 2024-04-01T01:46:43.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.525+0000 7f24f6fc9640 1 --2- 172.21.15.70:0/1007206753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24f0105540 0x7f24f01a55a0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.525+0000 7f24f6fc9640 1 --2- 172.21.15.70:0/1007206753 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f24d80a26a0 0x7f24d80a4b60 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.525+0000 7f24f6fc9640 1 --2- 172.21.15.70:0/1007206753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24f0104b80 0x7f24f019e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.526+0000 7f24f6fc9640 1 --2- 172.21.15.70:0/1007206753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24f0103990 0x7f24f019e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:43.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.526+0000 7f24f6fc9640 1 -- 172.21.15.70:0/1007206753 >> 172.21.15.70:0/1007206753 conn(0x7f24f00fef00 msgr2=0x7f24f00ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:43.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.526+0000 7f24f6fc9640 1 -- 172.21.15.70:0/1007206753 shutdown_connections 2024-04-01T01:46:43.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:46:43.526+0000 7f24f6fc9640 1 -- 172.21.15.70:0/1007206753 wait complete. 2024-04-01T01:46:43.545 INFO:tasks.cephfs.filesystem:scrub status for tag:88fc7b5f-e717-4c0c-a949-9ad1ddc9acf2 - {'path': '/', 'tag': '88fc7b5f-e717-4c0c-a949-9ad1ddc9acf2', 'options': 'recursive,force'} 2024-04-01T01:46:45.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:44 smithi081 ceph-mon[35142]: pgmap v756: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.8 MiB/s wr, 751 op/s 2024-04-01T01:46:45.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:44 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:46:45.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:44 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:46:45.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:44 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:46:45.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:44 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:46:45.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:44 smithi189 ceph-mon[35753]: pgmap v756: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.8 MiB/s wr, 751 op/s 2024-04-01T01:46:45.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:44 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:46:45.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:44 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:46:45.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:44 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:46:45.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:44 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:46:45.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:44 smithi070 ceph-mon[32179]: pgmap v756: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.8 MiB/s wr, 751 op/s 2024-04-01T01:46:45.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:44 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:46:45.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:44 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:46:45.256 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:44 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:46:45.256 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:44 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:46:46.897 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 720.0 s, 834.1 tps, lat 120.672 ms stddev 83.479 2024-04-01T01:46:47.055 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:46 smithi070 ceph-mon[32179]: pgmap v757: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.3 MiB/s wr, 698 op/s 2024-04-01T01:46:47.056 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:46 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:46:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:46 smithi081 ceph-mon[35142]: pgmap v757: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.3 MiB/s wr, 698 op/s 2024-04-01T01:46:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:46 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:46:47.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:46 smithi189 ceph-mon[35753]: pgmap v757: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.3 MiB/s wr, 698 op/s 2024-04-01T01:46:47.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:46 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:46:49.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:48 smithi081 ceph-mon[35142]: pgmap v758: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 7.3 MiB/s wr, 829 op/s 2024-04-01T01:46:49.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:48 smithi189 ceph-mon[35753]: pgmap v758: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 7.3 MiB/s wr, 829 op/s 2024-04-01T01:46:49.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:48 smithi070 ceph-mon[32179]: pgmap v758: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 7.3 MiB/s wr, 829 op/s 2024-04-01T01:46:51.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:50 smithi081 ceph-mon[35142]: pgmap v759: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 7.0 MiB/s wr, 850 op/s 2024-04-01T01:46:51.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:50 smithi189 ceph-mon[35753]: pgmap v759: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 7.0 MiB/s wr, 850 op/s 2024-04-01T01:46:51.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:50 smithi070 ceph-mon[32179]: pgmap v759: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 7.0 MiB/s wr, 850 op/s 2024-04-01T01:46:51.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 725.0 s, 799.8 tps, lat 124.574 ms stddev 101.346 2024-04-01T01:46:53.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:52 smithi081 ceph-mon[35142]: pgmap v760: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 6.3 MiB/s wr, 796 op/s 2024-04-01T01:46:53.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:52 smithi189 ceph-mon[35753]: pgmap v760: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 6.3 MiB/s wr, 796 op/s 2024-04-01T01:46:53.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:52 smithi070 ceph-mon[32179]: pgmap v760: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 6.3 MiB/s wr, 796 op/s 2024-04-01T01:46:55.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:54 smithi081 ceph-mon[35142]: pgmap v761: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 7.4 MiB/s wr, 883 op/s 2024-04-01T01:46:55.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:54 smithi189 ceph-mon[35753]: pgmap v761: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 7.4 MiB/s wr, 883 op/s 2024-04-01T01:46:55.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:54 smithi070 ceph-mon[32179]: pgmap v761: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 7.4 MiB/s wr, 883 op/s 2024-04-01T01:46:56.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.541+0000 7f8c40d6e640 1 -- 172.21.15.70:0/516303419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c340a4b10 msgr2=0x7f8c340a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:56.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.541+0000 7f8c40d6e640 1 --2- 172.21.15.70:0/516303419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c340a4b10 0x7f8c340a4f30 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f8c28013f10 tx=0x7f8c280341b0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.542+0000 7f8c40d6e640 1 -- 172.21.15.70:0/516303419 shutdown_connections 2024-04-01T01:46:56.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.542+0000 7f8c40d6e640 1 --2- 172.21.15.70:0/516303419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c340aca00 0x7f8c340aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.542 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.542+0000 7f8c40d6e640 1 --2- 172.21.15.70:0/516303419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c340a5470 0x7f8c340ac470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.542+0000 7f8c40d6e640 1 --2- 172.21.15.70:0/516303419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c340a4b10 0x7f8c340a4f30 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.542+0000 7f8c40d6e640 1 -- 172.21.15.70:0/516303419 >> 172.21.15.70:0/516303419 conn(0x7f8c340a00d0 msgr2=0x7f8c340a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.542+0000 7f8c40d6e640 1 -- 172.21.15.70:0/516303419 shutdown_connections 2024-04-01T01:46:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.543+0000 7f8c40d6e640 1 -- 172.21.15.70:0/516303419 wait complete. 2024-04-01T01:46:56.543 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.543+0000 7f8c40d6e640 1 Processor -- start 2024-04-01T01:46:56.544 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.543+0000 7f8c40d6e640 1 -- start start 2024-04-01T01:46:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.544+0000 7f8c40d6e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c340a4b10 0x7f8c3414bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.545 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.544+0000 7f8c40d6e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c340a5470 0x7f8c3414c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.544+0000 7f8c3b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c340a4b10 0x7f8c3414bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.545+0000 7f8c3b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c340a4b10 0x7f8c3414bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54292/0 (socket says 172.21.15.70:54292) 2024-04-01T01:46:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.545+0000 7f8c3affd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c340a5470 0x7f8c3414c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.550 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.545+0000 7f8c3affd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c340a5470 0x7f8c3414c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:56204/0 (socket says 172.21.15.70:56204) 2024-04-01T01:46:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.546+0000 7f8c40d6e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c340aca00 0x7f8c34153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.546+0000 7f8c40d6e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8c340b0730 con 0x7f8c340a4b10 2024-04-01T01:46:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.546+0000 7f8c40d6e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8c340b05b0 con 0x7f8c340aca00 2024-04-01T01:46:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.546+0000 7f8c40d6e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8c340b08b0 con 0x7f8c340a5470 2024-04-01T01:46:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.546+0000 7f8c3bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c340aca00 0x7f8c34153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.546+0000 7f8c3bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c340aca00 0x7f8c34153270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:45986/0 (socket says 172.21.15.70:45986) 2024-04-01T01:46:56.551 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.546+0000 7f8c3bfff640 1 -- 172.21.15.70:0/947089793 learned_addr learned my addr 172.21.15.70:0/947089793 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.546+0000 7f8c3bfff640 1 -- 172.21.15.70:0/947089793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c340a5470 msgr2=0x7f8c3414c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.546+0000 7f8c3bfff640 1 --2- 172.21.15.70:0/947089793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c340a5470 0x7f8c3414c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.546+0000 7f8c3bfff640 1 -- 172.21.15.70:0/947089793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c340a4b10 msgr2=0x7f8c3414bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.546+0000 7f8c3bfff640 1 --2- 172.21.15.70:0/947089793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c340a4b10 0x7f8c3414bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.546+0000 7f8c3bfff640 1 -- 172.21.15.70:0/947089793 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c34153910 con 0x7f8c340aca00 2024-04-01T01:46:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.547+0000 7f8c3b7fe640 1 --2- 172.21.15.70:0/947089793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c340a4b10 0x7f8c3414bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-01T01:46:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.547+0000 7f8c3affd640 1 --2- 172.21.15.70:0/947089793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c340a5470 0x7f8c3414c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-01T01:46:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.547+0000 7f8c3bfff640 1 --2- 172.21.15.70:0/947089793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c340aca00 0x7f8c34153270 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8c3000c970 tx=0x7f8c3000ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.547+0000 7f8c38ff9640 1 -- 172.21.15.70:0/947089793 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c30007bb0 con 0x7f8c340aca00 2024-04-01T01:46:56.552 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.548+0000 7f8c38ff9640 1 -- 172.21.15.70:0/947089793 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c30007d50 con 0x7f8c340aca00 2024-04-01T01:46:56.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.548+0000 7f8c38ff9640 1 -- 172.21.15.70:0/947089793 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c30018730 con 0x7f8c340aca00 2024-04-01T01:46:56.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.548+0000 7f8c40d6e640 1 -- 172.21.15.70:0/947089793 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c34153c00 con 0x7f8c340aca00 2024-04-01T01:46:56.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.548+0000 7f8c40d6e640 1 -- 172.21.15.70:0/947089793 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c34154100 con 0x7f8c340aca00 2024-04-01T01:46:56.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.551+0000 7f8c44bbc740 1 Processor -- start 2024-04-01T01:46:56.553 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.551+0000 7f8c38ff9640 1 -- 172.21.15.70:0/947089793 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f8c30018910 con 0x7f8c340aca00 2024-04-01T01:46:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.551+0000 7f8c44bbc740 1 -- start start 2024-04-01T01:46:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.552+0000 7f8c44bbc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f6ae0 0x55a3803f6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.552+0000 7f8c44bbc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f74d0 0x55a3803fbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.552+0000 7f8c44bbc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803fc640 0x55a3803fead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.552+0000 7f8c44bbc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a38035a8f0 con 0x55a3803fc640 2024-04-01T01:46:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.552+0000 7f8c44bbc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a3803038d0 con 0x55a3803f74d0 2024-04-01T01:46:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.552+0000 7f8c44bbc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a380356990 con 0x55a3803f6ae0 2024-04-01T01:46:56.554 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.552+0000 7f8c3affd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f6ae0 0x55a3803f6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.552+0000 7f8c3affd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f6ae0 0x55a3803f6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:56212/0 (socket says 172.21.15.70:56212) 2024-04-01T01:46:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.552+0000 7f8c3affd640 1 -- 172.21.15.70:0/1429502667 learned_addr learned my addr 172.21.15.70:0/1429502667 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.552+0000 7f8c3affd640 1 --2- 172.21.15.70:0/1429502667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803fc640 0x55a3803fead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.552+0000 7f8c3b7fe640 1 --2- 172.21.15.70:0/1429502667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f74d0 0x55a3803fbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.552+0000 7f8c38ff9640 1 --2- 172.21.15.70:0/947089793 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c2c0a26a0 0x7f8c2c0a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.553+0000 7f8c38ff9640 1 -- 172.21.15.70:0/947089793 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8c300cf050 con 0x7f8c340aca00 2024-04-01T01:46:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.553+0000 7f8c3b7fe640 1 -- 172.21.15.70:0/1429502667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f6ae0 msgr2=0x55a3803f6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.553+0000 7f8c3b7fe640 1 --2- 172.21.15.70:0/1429502667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f6ae0 0x55a3803f6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.553+0000 7f8c3b7fe640 1 -- 172.21.15.70:0/1429502667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803fc640 msgr2=0x55a3803fead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.553+0000 7f8c3b7fe640 1 --2- 172.21.15.70:0/1429502667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803fc640 0x55a3803fead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.553+0000 7f8c3b7fe640 1 -- 172.21.15.70:0/1429502667 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a38006f210 con 0x55a3803f74d0 2024-04-01T01:46:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.553+0000 7f8c3affd640 1 --2- 172.21.15.70:0/1429502667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f6ae0 0x55a3803f6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:46:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.553+0000 7f8c3b7fe640 1 --2- 172.21.15.70:0/1429502667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f74d0 0x55a3803fbfd0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8c28000c00 tx=0x7f8c28013c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ac7dc22836db3c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:56.555 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.553+0000 7f8c3bfff640 1 --2- 172.21.15.70:0/947089793 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c2c0a26a0 0x7f8c2c0a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.554+0000 7f8c25ffb640 1 -- 172.21.15.70:0/1429502667 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2803c640 con 0x55a3803f74d0 2024-04-01T01:46:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.554+0000 7f8c25ffb640 1 -- 172.21.15.70:0/1429502667 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c28034e10 con 0x55a3803f74d0 2024-04-01T01:46:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.554+0000 7f8c25ffb640 1 -- 172.21.15.70:0/1429502667 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2803cc90 con 0x55a3803f74d0 2024-04-01T01:46:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.555+0000 7f8c44bbc740 1 -- 172.21.15.70:0/1429502667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f74d0 msgr2=0x55a3803fbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:56.557 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.555+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/1429502667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f74d0 0x55a3803fbfd0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8c28000c00 tx=0x7f8c28013c90 comp rx=0 tx=0).stop 2024-04-01T01:46:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.555+0000 7f8c3bfff640 1 --2- 172.21.15.70:0/947089793 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c2c0a26a0 0x7f8c2c0a4b60 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8c3009bf10 tx=0x7f8c3009a220 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.555+0000 7f8c44bbc740 1 -- 172.21.15.70:0/1429502667 shutdown_connections 2024-04-01T01:46:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.555+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/1429502667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803fc640 0x55a3803fead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.555+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/1429502667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f74d0 0x55a3803fbfd0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.555+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/1429502667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f6ae0 0x55a3803f6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.555+0000 7f8c44bbc740 1 -- 172.21.15.70:0/1429502667 >> 172.21.15.70:0/1429502667 conn(0x55a3803f2050 msgr2=0x55a3803f2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.555+0000 7f8c44bbc740 1 -- 172.21.15.70:0/1429502667 shutdown_connections 2024-04-01T01:46:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.555+0000 7f8c44bbc740 1 -- 172.21.15.70:0/1429502667 wait complete. 2024-04-01T01:46:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.555+0000 7f8c44bbc740 1 -- start start 2024-04-01T01:46:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.556+0000 7f8c44bbc740 1 Processor -- start 2024-04-01T01:46:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.556+0000 7f8c44bbc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f6ae0 0x55a380400dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.556+0000 7f8c44bbc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803f74d0 0x55a380401310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.556+0000 7f8c44bbc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803fc640 0x55a380408310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.558 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.556+0000 7f8c3affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f6ae0 0x55a380400dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.556+0000 7f8c3affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f6ae0 0x55a380400dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46006/0 (socket says 172.21.15.70:46006) 2024-04-01T01:46:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.556+0000 7f8c3affd640 1 -- 172.21.15.70:0/4172806529 learned_addr learned my addr 172.21.15.70:0/4172806529 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.556+0000 7f8c44bbc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a3803ff590 con 0x55a3803f74d0 2024-04-01T01:46:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.556+0000 7f8c44bbc740 1 -- 172.21.15.70:0/4172806529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a3802f8f80 con 0x55a3803f6ae0 2024-04-01T01:46:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.556+0000 7f8c44bbc740 1 -- 172.21.15.70:0/4172806529 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a3803ff710 con 0x55a3803fc640 2024-04-01T01:46:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.556+0000 7f8c3affd640 1 --2- 172.21.15.70:0/4172806529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803fc640 0x55a380408310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.557+0000 7f8c3b7fe640 1 --2- 172.21.15.70:0/4172806529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803f74d0 0x55a380401310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.557+0000 7f8c44bbc740 10 client.0 _ll_register_callbacks cb 0x55a380351290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.557+0000 7f8c3affd640 1 -- 172.21.15.70:0/4172806529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803fc640 msgr2=0x55a380408310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.557+0000 7f8c3affd640 1 --2- 172.21.15.70:0/4172806529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803fc640 0x55a380408310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.557+0000 7f8c3affd640 1 -- 172.21.15.70:0/4172806529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803f74d0 msgr2=0x55a380401310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.557+0000 7f8c3affd640 1 --2- 172.21.15.70:0/4172806529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803f74d0 0x55a380401310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.557+0000 7f8c3affd640 1 -- 172.21.15.70:0/4172806529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a38013b620 con 0x55a3803f6ae0 2024-04-01T01:46:56.559 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.557+0000 7f8c3b7fe640 1 --2- 172.21.15.70:0/4172806529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803f74d0 0x55a380401310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:46:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.557+0000 7f8c3affd640 1 --2- 172.21.15.70:0/4172806529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f6ae0 0x55a380400dd0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c067430 tx=0x7f8c3c062ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5273a87484a3aa59 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.557+0000 7f8c1affd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:46:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.557+0000 7f8c1affd640 1 -- 172.21.15.70:0/4172806529 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c0817d0 con 0x55a3803f6ae0 2024-04-01T01:46:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.557+0000 7f8c1affd640 1 -- 172.21.15.70:0/4172806529 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8c3c081970 con 0x55a3803f6ae0 2024-04-01T01:46:56.560 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.558+0000 7f8c44bbc740 1 -- 172.21.15.70:0/4172806529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c28034e10 con 0x55a3803f6ae0 2024-04-01T01:46:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.562+0000 7f8c1affd640 1 -- 172.21.15.70:0/4172806529 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8c3c072080 con 0x55a3803f6ae0 2024-04-01T01:46:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.562+0000 7f8c44bbc740 10 client.29438 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:56.562 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.562+0000 7f8c44bbc740 1 -- 172.21.15.70:0/4172806529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a38040b3c0 con 0x55a3803f6ae0 2024-04-01T01:46:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.563+0000 7f8c1affd640 1 -- 172.21.15.70:0/4172806529 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c28034e10 con 0x55a3803f6ae0 2024-04-01T01:46:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.563+0000 7f8c1affd640 1 -- 172.21.15.70:0/4172806529 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8c3c083920 con 0x55a3803f6ae0 2024-04-01T01:46:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.563+0000 7f8c44bbc740 10 client.29438 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.563+0000 7f8c44bbc740 10 client.29438 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.563+0000 7f8c44bbc740 20 client.29438 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.563+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/4172806529 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a38040ca40 0x55a38040ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.563+0000 7f8c44bbc740 4 client.29438 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:46:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.563+0000 7f8c44bbc740 1 -- 172.21.15.70:0/4172806529 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a38036e120 con 0x55a38040ca40 2024-04-01T01:46:56.564 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.564+0000 7f8c3b7fe640 1 --2- 172.21.15.70:0/4172806529 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a38040ca40 0x55a38040ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.564+0000 7f8c1affd640 10 client.29438 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:56.565 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.565+0000 7f8c3b7fe640 1 --2- 172.21.15.70:0/4172806529 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a38040ca40 0x55a38040ef00 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=459940a78a63b4e8 server_cookie=bb2fff89aba64917 in_seq=0 out_seq=0 2024-04-01T01:46:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.566+0000 7f8c1affd640 1 -- 172.21.15.70:0/4172806529 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a38036e120 con 0x55a38040ca40 2024-04-01T01:46:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.566+0000 7f8c1affd640 10 client.29438 handle_command_reply: tid=0 2024-04-01T01:46:56.566 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.566+0000 7f8c44bbc740 1 client.29438 shutdown 2024-04-01T01:46:56.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.568+0000 7f8c44bbc740 1 -- 172.21.15.70:0/4172806529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f6ae0 msgr2=0x55a380400dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:56.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.568+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/4172806529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f6ae0 0x55a380400dd0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c067430 tx=0x7f8c3c062ba0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.569+0000 7f8c44bbc740 1 -- 172.21.15.70:0/4172806529 shutdown_connections 2024-04-01T01:46:56.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.569+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/4172806529 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a38040ca40 0x55a38040ef00 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.569+0000 7f8c1affd640 0 client.29438 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:56.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.569+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/4172806529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803fc640 0x55a380408310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.569+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/4172806529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803f74d0 0x55a380401310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.569+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/4172806529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f6ae0 0x55a380400dd0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.569+0000 7f8c44bbc740 1 -- 172.21.15.70:0/4172806529 >> 172.21.15.70:0/4172806529 conn(0x55a3803f2050 msgr2=0x55a3803f1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:56.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.569+0000 7f8c44bbc740 1 -- 172.21.15.70:0/4172806529 shutdown_connections 2024-04-01T01:46:56.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.569+0000 7f8c44bbc740 1 -- 172.21.15.70:0/4172806529 wait complete. 2024-04-01T01:46:56.570 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.569+0000 7f8c44bbc740 20 client.29438 trim_cache size 0 max 16384 2024-04-01T01:46:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.602+0000 7f8c44bbc740 1 Processor -- start 2024-04-01T01:46:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.602+0000 7f8c44bbc740 1 -- start start 2024-04-01T01:46:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.602+0000 7f8c44bbc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f6ae0 0x55a3803f0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.602+0000 7f8c44bbc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f74d0 0x55a3804082d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.602+0000 7f8c44bbc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803fc640 0x55a3803f0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.603+0000 7f8c3affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f6ae0 0x55a3803f0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.603+0000 7f8c3affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f6ae0 0x55a3803f0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:46020/0 (socket says 172.21.15.70:46020) 2024-04-01T01:46:56.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.603+0000 7f8c3affd640 1 -- 172.21.15.70:0/1150551506 learned_addr learned my addr 172.21.15.70:0/1150551506 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:56.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.603+0000 7f8c44bbc740 1 -- 172.21.15.70:0/1150551506 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2000fd50 con 0x55a3803fc640 2024-04-01T01:46:56.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.603+0000 7f8c44bbc740 1 -- 172.21.15.70:0/1150551506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8c20011890 con 0x55a3803f6ae0 2024-04-01T01:46:56.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.603+0000 7f8c44bbc740 1 -- 172.21.15.70:0/1150551506 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8c20010420 con 0x55a3803f74d0 2024-04-01T01:46:56.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.603+0000 7f8c3affd640 1 --2- 172.21.15.70:0/1150551506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803fc640 0x55a3803f0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.603+0000 7f8c3affd640 1 -- 172.21.15.70:0/1150551506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f74d0 msgr2=0x55a3804082d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.603+0000 7f8c3affd640 1 --2- 172.21.15.70:0/1150551506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f74d0 0x55a3804082d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.603+0000 7f8c3affd640 1 -- 172.21.15.70:0/1150551506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803fc640 msgr2=0x55a3803f0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.603+0000 7f8c3affd640 1 --2- 172.21.15.70:0/1150551506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803fc640 0x55a3803f0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.603+0000 7f8c3affd640 1 -- 172.21.15.70:0/1150551506 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3803f2dc0 con 0x55a3803f6ae0 2024-04-01T01:46:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.604+0000 7f8c3affd640 1 --2- 172.21.15.70:0/1150551506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f6ae0 0x55a3803f0900 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c000b60 tx=0x7f8c3c074040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fb0b3b3550a979c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.604+0000 7f8c257fa640 1 -- 172.21.15.70:0/1150551506 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c083470 con 0x55a3803f6ae0 2024-04-01T01:46:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.604+0000 7f8c257fa640 1 -- 172.21.15.70:0/1150551506 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c3c072590 con 0x55a3803f6ae0 2024-04-01T01:46:56.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.604+0000 7f8c257fa640 1 -- 172.21.15.70:0/1150551506 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c081cf0 con 0x55a3803f6ae0 2024-04-01T01:46:56.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.605+0000 7f8c44bbc740 1 -- 172.21.15.70:0/1150551506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f6ae0 msgr2=0x55a3803f0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:56.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.605+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/1150551506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f6ae0 0x55a3803f0900 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c000b60 tx=0x7f8c3c074040 comp rx=0 tx=0).stop 2024-04-01T01:46:56.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.605+0000 7f8c44bbc740 1 -- 172.21.15.70:0/1150551506 shutdown_connections 2024-04-01T01:46:56.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.605+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/1150551506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803fc640 0x55a3803f0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.605+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/1150551506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f74d0 0x55a3804082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.606+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/1150551506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803f6ae0 0x55a3803f0900 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.606+0000 7f8c44bbc740 1 -- 172.21.15.70:0/1150551506 >> 172.21.15.70:0/1150551506 conn(0x55a3803f2050 msgr2=0x55a3803f2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:56.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.606+0000 7f8c44bbc740 1 -- 172.21.15.70:0/1150551506 shutdown_connections 2024-04-01T01:46:56.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.606+0000 7f8c44bbc740 1 -- 172.21.15.70:0/1150551506 wait complete. 2024-04-01T01:46:56.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.606+0000 7f8c44bbc740 1 -- start start 2024-04-01T01:46:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.607+0000 7f8c44bbc740 1 Processor -- start 2024-04-01T01:46:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.607+0000 7f8c44bbc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f6ae0 0x55a3804054c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.608+0000 7f8c44bbc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803f74d0 0x55a380405a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.608+0000 7f8c3affd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f6ae0 0x55a3804054c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.608+0000 7f8c3affd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f6ae0 0x55a3804054c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:56240/0 (socket says 172.21.15.70:56240) 2024-04-01T01:46:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.608+0000 7f8c3b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803f74d0 0x55a380405a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.608+0000 7f8c3b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803f74d0 0x55a380405a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54330/0 (socket says 172.21.15.70:54330) 2024-04-01T01:46:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.608+0000 7f8c3affd640 1 -- 172.21.15.70:0/3516516035 learned_addr learned my addr 172.21.15.70:0/3516516035 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:46:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.608+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/3516516035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803fc640 0x55a380406a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.608+0000 7f8c44bbc740 1 -- 172.21.15.70:0/3516516035 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8c200111c0 con 0x55a3803f74d0 2024-04-01T01:46:56.609 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.608+0000 7f8c44bbc740 1 -- 172.21.15.70:0/3516516035 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8c20010af0 con 0x55a3803fc640 2024-04-01T01:46:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.608+0000 7f8c44bbc740 1 -- 172.21.15.70:0/3516516035 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2000dd30 con 0x55a3803f6ae0 2024-04-01T01:46:56.610 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.609+0000 7f8c3affd640 1 --2- 172.21.15.70:0/3516516035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803fc640 0x55a380406a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.609+0000 7f8c3affd640 1 -- 172.21.15.70:0/3516516035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803fc640 msgr2=0x55a380406a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.609+0000 7f8c3affd640 1 --2- 172.21.15.70:0/3516516035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803fc640 0x55a380406a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.609+0000 7f8c3affd640 1 -- 172.21.15.70:0/3516516035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803f74d0 msgr2=0x55a380405a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:46:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.609+0000 7f8c3affd640 1 --2- 172.21.15.70:0/3516516035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803f74d0 0x55a380405a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.609+0000 7f8c3affd640 1 -- 172.21.15.70:0/3516516035 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a380407000 con 0x55a3803f6ae0 2024-04-01T01:46:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.609+0000 7f8c3affd640 1 --2- 172.21.15.70:0/3516516035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f6ae0 0x55a3804054c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c04f780 tx=0x7f8c3c094040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f54ee001457152b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:46:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.609+0000 7f8c1affd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:46:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.609+0000 7f8c44bbc740 10 client.0 _ll_register_callbacks cb 0x7f8c3c072df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:46:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.609+0000 7f8c44bbc740 1 -- 172.21.15.70:0/3516516035 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c3c072590 con 0x55a3803f6ae0 2024-04-01T01:46:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.609+0000 7f8c1affd640 1 -- 172.21.15.70:0/3516516035 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c083680 con 0x55a3803f6ae0 2024-04-01T01:46:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.609+0000 7f8c1affd640 1 -- 172.21.15.70:0/3516516035 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8c3c083820 con 0x55a3803f6ae0 2024-04-01T01:46:56.611 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.609+0000 7f8c1affd640 1 -- 172.21.15.70:0/3516516035 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8c3c083a40 con 0x55a3803f6ae0 2024-04-01T01:46:56.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.612+0000 7f8c44bbc740 10 client.29215 fetch_fsmap learned FSMap version 31 2024-04-01T01:46:56.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.612+0000 7f8c44bbc740 1 -- 172.21.15.70:0/3516516035 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55a3804081b0 con 0x55a3803f6ae0 2024-04-01T01:46:56.612 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.612+0000 7f8c1affd640 1 -- 172.21.15.70:0/3516516035 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8c3c0704d0 con 0x55a3803f6ae0 2024-04-01T01:46:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.613+0000 7f8c44bbc740 10 client.29215 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:46:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.613+0000 7f8c44bbc740 10 client.29215 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:46:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.614+0000 7f8c44bbc740 20 client.29215 populate_metadata read hostname 'smithi070' 2024-04-01T01:46:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.614+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/3516516035 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a3804730c0 0x55a3804754e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:46:56.614 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.614+0000 7f8c44bbc740 4 client.29215 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:46:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.614+0000 7f8c44bbc740 1 -- 172.21.15.70:0/3516516035 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a380367e30 con 0x55a3804730c0 2024-04-01T01:46:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.614+0000 7f8c3b7fe640 1 --2- 172.21.15.70:0/3516516035 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a3804730c0 0x55a3804754e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:46:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.615+0000 7f8c3b7fe640 1 --2- 172.21.15.70:0/3516516035 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a3804730c0 0x55a3804754e0 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe011bae7d80229f server_cookie=e755a267b0dfa7d8 in_seq=0 out_seq=0 2024-04-01T01:46:56.615 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.615+0000 7f8c1affd640 10 client.29215 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.643+0000 7f8c1affd640 1 -- 172.21.15.70:0/3516516035 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a380367e30 con 0x55a3804730c0 2024-04-01T01:46:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.643+0000 7f8c1affd640 10 client.29215 handle_command_reply: tid=0 2024-04-01T01:46:56.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.644+0000 7f8c44bbc740 1 client.29215 shutdown 2024-04-01T01:46:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.647+0000 7f8c44bbc740 1 -- 172.21.15.70:0/3516516035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f6ae0 msgr2=0x55a3804054c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:56.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.647+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/3516516035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f6ae0 0x55a3804054c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c04f780 tx=0x7f8c3c094040 comp rx=0 tx=0).stop 2024-04-01T01:46:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.648+0000 7f8c44bbc740 1 -- 172.21.15.70:0/3516516035 shutdown_connections 2024-04-01T01:46:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.648+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/3516516035 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a3804730c0 0x55a3804754e0 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.648+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/3516516035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a3803fc640 0x55a380406a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.648+0000 7f8c1affd640 0 client.29215 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:46:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.648+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/3516516035 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a3803f74d0 0x55a380405a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.648+0000 7f8c44bbc740 1 --2- 172.21.15.70:0/3516516035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a3803f6ae0 0x55a3804054c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.648+0000 7f8c44bbc740 1 -- 172.21.15.70:0/3516516035 >> 172.21.15.70:0/3516516035 conn(0x55a3803f2050 msgr2=0x55a3803f0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:56.651 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.651+0000 7f8c44bbc740 1 -- 172.21.15.70:0/3516516035 shutdown_connections 2024-04-01T01:46:56.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.651+0000 7f8c44bbc740 1 -- 172.21.15.70:0/3516516035 wait complete. 2024-04-01T01:46:56.652 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.652+0000 7f8c44bbc740 20 client.29215 trim_cache size 0 max 16384 2024-04-01T01:46:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:46:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:46:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:46:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:46:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.656+0000 7f8c40d6e640 1 -- 172.21.15.70:0/947089793 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c2c0a26a0 msgr2=0x7f8c2c0a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.657+0000 7f8c40d6e640 1 --2- 172.21.15.70:0/947089793 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c2c0a26a0 0x7f8c2c0a4b60 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8c3009bf10 tx=0x7f8c3009a220 comp rx=0 tx=0).stop 2024-04-01T01:46:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.657+0000 7f8c40d6e640 1 -- 172.21.15.70:0/947089793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c340aca00 msgr2=0x7f8c34153270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:46:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.657+0000 7f8c40d6e640 1 --2- 172.21.15.70:0/947089793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c340aca00 0x7f8c34153270 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8c3000c970 tx=0x7f8c3000ce30 comp rx=0 tx=0).stop 2024-04-01T01:46:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.657+0000 7f8c40d6e640 1 -- 172.21.15.70:0/947089793 shutdown_connections 2024-04-01T01:46:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.657+0000 7f8c40d6e640 1 --2- 172.21.15.70:0/947089793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c340aca00 0x7f8c34153270 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.657+0000 7f8c40d6e640 1 --2- 172.21.15.70:0/947089793 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8c2c0a26a0 0x7f8c2c0a4b60 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.658 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.657+0000 7f8c40d6e640 1 --2- 172.21.15.70:0/947089793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c340a5470 0x7f8c3414c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.658+0000 7f8c40d6e640 1 --2- 172.21.15.70:0/947089793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c340a4b10 0x7f8c3414bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:46:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.658+0000 7f8c40d6e640 1 -- 172.21.15.70:0/947089793 >> 172.21.15.70:0/947089793 conn(0x7f8c340a00d0 msgr2=0x7f8c340a0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:46:56.659 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.658+0000 7f8c40d6e640 1 -- 172.21.15.70:0/947089793 shutdown_connections 2024-04-01T01:46:56.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:46:56.659+0000 7f8c40d6e640 1 -- 172.21.15.70:0/947089793 wait complete. 2024-04-01T01:46:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 730.0 s, 808.0 tps, lat 122.026 ms stddev 76.457 2024-04-01T01:46:57.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:56 smithi081 ceph-mon[35142]: pgmap v762: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 6.7 MiB/s wr, 813 op/s 2024-04-01T01:46:57.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:56 smithi189 ceph-mon[35753]: pgmap v762: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 6.7 MiB/s wr, 813 op/s 2024-04-01T01:46:57.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:56 smithi070 ceph-mon[32179]: pgmap v762: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 6.7 MiB/s wr, 813 op/s 2024-04-01T01:46:59.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:46:58 smithi081 ceph-mon[35142]: pgmap v763: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.5 MiB/s wr, 889 op/s 2024-04-01T01:46:59.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:46:58 smithi189 ceph-mon[35753]: pgmap v763: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.5 MiB/s wr, 889 op/s 2024-04-01T01:46:59.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:46:58 smithi070 ceph-mon[32179]: pgmap v763: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.5 MiB/s wr, 889 op/s 2024-04-01T01:47:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:00 smithi081 ceph-mon[35142]: pgmap v764: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.2 MiB/s wr, 831 op/s 2024-04-01T01:47:01.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:00 smithi189 ceph-mon[35753]: pgmap v764: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.2 MiB/s wr, 831 op/s 2024-04-01T01:47:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:00 smithi070 ceph-mon[32179]: pgmap v764: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.2 MiB/s wr, 831 op/s 2024-04-01T01:47:01.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 735.0 s, 758.6 tps, lat 132.478 ms stddev 135.489 2024-04-01T01:47:03.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:02 smithi081 ceph-mon[35142]: pgmap v765: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.7 MiB/s wr, 740 op/s 2024-04-01T01:47:03.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:02 smithi189 ceph-mon[35753]: pgmap v765: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.7 MiB/s wr, 740 op/s 2024-04-01T01:47:03.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:02 smithi070 ceph-mon[32179]: pgmap v765: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.7 MiB/s wr, 740 op/s 2024-04-01T01:47:05.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:04 smithi081 ceph-mon[35142]: pgmap v766: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.9 MiB/s wr, 768 op/s 2024-04-01T01:47:05.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:04 smithi189 ceph-mon[35753]: pgmap v766: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.9 MiB/s wr, 768 op/s 2024-04-01T01:47:05.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:04 smithi070 ceph-mon[32179]: pgmap v766: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.9 MiB/s wr, 768 op/s 2024-04-01T01:47:06.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 740.0 s, 772.4 tps, lat 129.619 ms stddev 131.446 2024-04-01T01:47:07.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:06 smithi081 ceph-mon[35142]: pgmap v767: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.0 MiB/s wr, 676 op/s 2024-04-01T01:47:07.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:06 smithi189 ceph-mon[35753]: pgmap v767: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.0 MiB/s wr, 676 op/s 2024-04-01T01:47:07.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:06 smithi070 ceph-mon[32179]: pgmap v767: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.0 MiB/s wr, 676 op/s 2024-04-01T01:47:09.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:08 smithi081 ceph-mon[35142]: pgmap v768: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 749 op/s 2024-04-01T01:47:09.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:08 smithi189 ceph-mon[35753]: pgmap v768: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 749 op/s 2024-04-01T01:47:09.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:08 smithi070 ceph-mon[32179]: pgmap v768: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 749 op/s 2024-04-01T01:47:09.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.815+0000 7f9f8c62e640 1 -- 172.21.15.70:0/1545976184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f84101860 msgr2=0x7f9f84101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:09.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.815+0000 7f9f8c62e640 1 --2- 172.21.15.70:0/1545976184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f84101860 0x7f9f84101ce0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009900 tx=0x7f9f7402bc30 comp rx=0 tx=0).stop 2024-04-01T01:47:09.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.815+0000 7f9f8c62e640 1 -- 172.21.15.70:0/1545976184 shutdown_connections 2024-04-01T01:47:09.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.815+0000 7f9f8c62e640 1 --2- 172.21.15.70:0/1545976184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f84102220 0x7f9f84108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.815+0000 7f9f8c62e640 1 --2- 172.21.15.70:0/1545976184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f84101860 0x7f9f84101ce0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.815+0000 7f9f8c62e640 1 --2- 172.21.15.70:0/1545976184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f84100670 0x7f9f84100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.815 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.815+0000 7f9f8c62e640 1 -- 172.21.15.70:0/1545976184 >> 172.21.15.70:0/1545976184 conn(0x7f9f840fbbe0 msgr2=0x7f9f840fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:09.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.815+0000 7f9f8c62e640 1 -- 172.21.15.70:0/1545976184 shutdown_connections 2024-04-01T01:47:09.816 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.815+0000 7f9f8c62e640 1 -- 172.21.15.70:0/1545976184 wait complete. 2024-04-01T01:47:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.816+0000 7f9f8c62e640 1 Processor -- start 2024-04-01T01:47:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.816+0000 7f9f8c62e640 1 -- start start 2024-04-01T01:47:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.816+0000 7f9f8c62e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f84100670 0x7f9f840699a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.816+0000 7f9f8c62e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f84101860 0x7f9f8419a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.816+0000 7f9f8c62e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f84102220 0x7f9f8419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.817 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.817+0000 7f9f89bc9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f84101860 0x7f9f8419a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:09.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.817+0000 7f9f89bc9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f84101860 0x7f9f8419a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44566/0 (socket says 172.21.15.70:44566) 2024-04-01T01:47:09.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.817+0000 7f9f8a3ca640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f84100670 0x7f9f840699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:09.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.817+0000 7f9f8a3ca640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f84100670 0x7f9f840699a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57056/0 (socket says 172.21.15.70:57056) 2024-04-01T01:47:09.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.817+0000 7f9f89bc9640 1 -- 172.21.15.70:0/607927151 learned_addr learned my addr 172.21.15.70:0/607927151 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:09.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.817+0000 7f9f8c62e640 1 -- 172.21.15.70:0/607927151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8410c1d0 con 0x7f9f84101860 2024-04-01T01:47:09.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.817+0000 7f9f8abcb640 1 --2- 172.21.15.70:0/607927151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f84102220 0x7f9f8419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:09.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.817+0000 7f9f8c62e640 1 -- 172.21.15.70:0/607927151 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8410c050 con 0x7f9f84102220 2024-04-01T01:47:09.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.817+0000 7f9f8c62e640 1 -- 172.21.15.70:0/607927151 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8410c350 con 0x7f9f84100670 2024-04-01T01:47:09.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.817+0000 7f9f8abcb640 1 -- 172.21.15.70:0/607927151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f84100670 msgr2=0x7f9f840699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:09.818 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.817+0000 7f9f8abcb640 1 --2- 172.21.15.70:0/607927151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f84100670 0x7f9f840699a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.817+0000 7f9f8abcb640 1 -- 172.21.15.70:0/607927151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f84101860 msgr2=0x7f9f8419a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:09.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.817+0000 7f9f8abcb640 1 --2- 172.21.15.70:0/607927151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f84101860 0x7f9f8419a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.817+0000 7f9f8abcb640 1 -- 172.21.15.70:0/607927151 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8419b2e0 con 0x7f9f84102220 2024-04-01T01:47:09.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.817+0000 7f9f8a3ca640 1 --2- 172.21.15.70:0/607927151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f84100670 0x7f9f840699a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:47:09.819 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.817+0000 7f9f8abcb640 1 --2- 172.21.15.70:0/607927151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f84102220 0x7f9f8419aa20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c009fd0 tx=0x7f9f7c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.819+0000 7f9f7b7fe640 1 -- 172.21.15.70:0/607927151 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c00ce20 con 0x7f9f84102220 2024-04-01T01:47:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.819+0000 7f9f7b7fe640 1 -- 172.21.15.70:0/607927151 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f7c005020 con 0x7f9f84102220 2024-04-01T01:47:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.819+0000 7f9f7b7fe640 1 -- 172.21.15.70:0/607927151 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c010690 con 0x7f9f84102220 2024-04-01T01:47:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.819+0000 7f9f8c62e640 1 -- 172.21.15.70:0/607927151 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f8419b5d0 con 0x7f9f84102220 2024-04-01T01:47:09.820 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.819+0000 7f9f8c62e640 1 -- 172.21.15.70:0/607927151 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f8419bad0 con 0x7f9f84102220 2024-04-01T01:47:09.821 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.822+0000 7f9f7b7fe640 1 -- 172.21.15.70:0/607927151 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f9f7c0051d0 con 0x7f9f84102220 2024-04-01T01:47:09.822 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.823+0000 7f9f8f214740 1 Processor -- start 2024-04-01T01:47:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.823+0000 7f9f8f214740 1 -- start start 2024-04-01T01:47:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.823+0000 7f9f7b7fe640 1 --2- 172.21.15.70:0/607927151 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9f700a26b0 0x7f9f700a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f8f214740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 0x561d0b893cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f8f214740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b898dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.823 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f8f214740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 0x561d0b89b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f7b7fe640 1 -- 172.21.15.70:0/607927151 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f9f7c0cf170 con 0x7f9f84102220 2024-04-01T01:47:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f89bc9640 1 --2- 172.21.15.70:0/607927151 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9f700a26b0 0x7f9f700a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f8a3ca640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 0x561d0b893cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f8a3ca640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 0x561d0b893cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47702/0 (socket says 172.21.15.70:47702) 2024-04-01T01:47:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f8a3ca640 1 -- 172.21.15.70:0/524125570 learned_addr learned my addr 172.21.15.70:0/524125570 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f8f214740 1 -- 172.21.15.70:0/524125570 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561d0b7f78f0 con 0x561d0b899430 2024-04-01T01:47:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f8f214740 1 -- 172.21.15.70:0/524125570 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561d0b7a08d0 con 0x561d0b8938d0 2024-04-01T01:47:09.824 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f8f214740 1 -- 172.21.15.70:0/524125570 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561d0b7f3990 con 0x561d0b8942c0 2024-04-01T01:47:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f8abcb640 1 --2- 172.21.15.70:0/524125570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 0x561d0b89b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f8a3ca640 1 -- 172.21.15.70:0/524125570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 msgr2=0x561d0b898dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f8a3ca640 1 --2- 172.21.15.70:0/524125570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b898dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f8a3ca640 1 -- 172.21.15.70:0/524125570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 msgr2=0x561d0b89b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f8a3ca640 1 --2- 172.21.15.70:0/524125570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 0x561d0b89b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.824+0000 7f9f8a3ca640 1 -- 172.21.15.70:0/524125570 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d0b50c210 con 0x561d0b8938d0 2024-04-01T01:47:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.825+0000 7f9f8a3ca640 1 --2- 172.21.15.70:0/524125570 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 0x561d0b893cf0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009cd0 tx=0x7f9f800117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5df99de5e62d219 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:09.825 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.825+0000 7f9f89bc9640 1 --2- 172.21.15.70:0/607927151 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9f700a26b0 0x7f9f700a4b70 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9f740096c0 tx=0x7f9f74005b70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:09.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.827+0000 7f9f78bfe640 1 -- 172.21.15.70:0/524125570 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f80005020 con 0x561d0b8938d0 2024-04-01T01:47:09.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.827+0000 7f9f78bfe640 1 -- 172.21.15.70:0/524125570 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f800047a0 con 0x561d0b8938d0 2024-04-01T01:47:09.826 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.827+0000 7f9f78bfe640 1 -- 172.21.15.70:0/524125570 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f80012740 con 0x561d0b8938d0 2024-04-01T01:47:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.829+0000 7f9f8f214740 1 -- 172.21.15.70:0/524125570 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 msgr2=0x561d0b893cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:09.828 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.829+0000 7f9f8f214740 1 --2- 172.21.15.70:0/524125570 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 0x561d0b893cf0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009cd0 tx=0x7f9f800117b0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.829+0000 7f9f8f214740 1 -- 172.21.15.70:0/524125570 shutdown_connections 2024-04-01T01:47:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.829+0000 7f9f8f214740 1 --2- 172.21.15.70:0/524125570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 0x561d0b89b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.829+0000 7f9f8f214740 1 --2- 172.21.15.70:0/524125570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b898dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.829+0000 7f9f8f214740 1 --2- 172.21.15.70:0/524125570 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 0x561d0b893cf0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.829+0000 7f9f8f214740 1 -- 172.21.15.70:0/524125570 >> 172.21.15.70:0/524125570 conn(0x561d0b88ee40 msgr2=0x561d0b88f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:09.829 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.830+0000 7f9f8f214740 1 -- 172.21.15.70:0/524125570 shutdown_connections 2024-04-01T01:47:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.830+0000 7f9f8f214740 1 -- 172.21.15.70:0/524125570 wait complete. 2024-04-01T01:47:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.830+0000 7f9f8f214740 1 -- start start 2024-04-01T01:47:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.830+0000 7f9f8f214740 1 Processor -- start 2024-04-01T01:47:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.831+0000 7f9f8f214740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 0x561d0b89dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.831+0000 7f9f8f214740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b89e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.831 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.831+0000 7f9f8f214740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 0x561d0b8a5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.831+0000 7f9f8f214740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561d0b89c3e0 con 0x561d0b899430 2024-04-01T01:47:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.831+0000 7f9f8f214740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561d0b795f80 con 0x561d0b8938d0 2024-04-01T01:47:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.831+0000 7f9f8f214740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561d0b89c560 con 0x561d0b8942c0 2024-04-01T01:47:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.831+0000 7f9f89bc9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b89e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.831+0000 7f9f89bc9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b89e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57074/0 (socket says 172.21.15.70:57074) 2024-04-01T01:47:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.831+0000 7f9f89bc9640 1 -- 172.21.15.70:0/4131091696 learned_addr learned my addr 172.21.15.70:0/4131091696 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.831+0000 7f9f8abcb640 1 --2- 172.21.15.70:0/4131091696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 0x561d0b8a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.831+0000 7f9f8f214740 10 client.0 _ll_register_callbacks cb 0x561d0b7ee290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.831+0000 7f9f8abcb640 1 -- 172.21.15.70:0/4131091696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 msgr2=0x561d0b89e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.831+0000 7f9f8abcb640 1 --2- 172.21.15.70:0/4131091696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b89e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.831+0000 7f9f8abcb640 1 -- 172.21.15.70:0/4131091696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 msgr2=0x561d0b89dc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:47:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.831+0000 7f9f8abcb640 1 --2- 172.21.15.70:0/4131091696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 0x561d0b89dc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.831+0000 7f9f8abcb640 1 -- 172.21.15.70:0/4131091696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d0b5d8620 con 0x561d0b899430 2024-04-01T01:47:09.832 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.832+0000 7f9f8abcb640 1 --2- 172.21.15.70:0/4131091696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 0x561d0b8a5190 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0cc870 tx=0x7f9f7c09c020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d2cd1e73e5f3788 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.832+0000 7f9f6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:47:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.832+0000 7f9f6dffb640 1 -- 172.21.15.70:0/4131091696 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c0b3070 con 0x561d0b899430 2024-04-01T01:47:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.833+0000 7f9f6dffb640 1 -- 172.21.15.70:0/4131091696 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f9f7c0a3de0 con 0x561d0b899430 2024-04-01T01:47:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.833+0000 7f9f8f214740 1 -- 172.21.15.70:0/4131091696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f800047a0 con 0x561d0b899430 2024-04-01T01:47:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.833+0000 7f9f6dffb640 1 -- 172.21.15.70:0/4131091696 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9f7c0a4270 con 0x561d0b899430 2024-04-01T01:47:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.833+0000 7f9f8f214740 10 client.19380 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:09.833 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.833+0000 7f9f8f214740 1 -- 172.21.15.70:0/4131091696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561d0b8b91c0 con 0x561d0b899430 2024-04-01T01:47:09.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.834+0000 7f9f6dffb640 1 -- 172.21.15.70:0/4131091696 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f800047a0 con 0x561d0b899430 2024-04-01T01:47:09.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.834+0000 7f9f6dffb640 1 -- 172.21.15.70:0/4131091696 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f9f7c0afb20 con 0x561d0b899430 2024-04-01T01:47:09.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.835+0000 7f9f8f214740 10 client.19380 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:09.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.835+0000 7f9f8f214740 10 client.19380 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:09.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.835+0000 7f9f8f214740 20 client.19380 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:09.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.835+0000 7f9f8f214740 1 --2- 172.21.15.70:0/4131091696 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561d0b8ba930 0x561d0b8bcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.835+0000 7f9f8f214740 4 client.19380 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:09.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.835+0000 7f9f8f214740 1 -- 172.21.15.70:0/4131091696 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d0b80b120 con 0x561d0b8ba930 2024-04-01T01:47:09.835 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.835+0000 7f9f8a3ca640 1 --2- 172.21.15.70:0/4131091696 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561d0b8ba930 0x561d0b8bcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:09.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.836+0000 7f9f8a3ca640 1 --2- 172.21.15.70:0/4131091696 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561d0b8ba930 0x561d0b8bcd80 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15405b5b135a50c1 server_cookie=8b6f190527ed2503 in_seq=0 out_seq=0 2024-04-01T01:47:09.836 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.836+0000 7f9f6dffb640 10 client.19380 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.838+0000 7f9f6dffb640 1 -- 172.21.15.70:0/4131091696 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561d0b80b120 con 0x561d0b8ba930 2024-04-01T01:47:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.838+0000 7f9f6dffb640 10 client.19380 handle_command_reply: tid=0 2024-04-01T01:47:09.837 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.838+0000 7f9f8f214740 1 client.19380 shutdown 2024-04-01T01:47:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.842+0000 7f9f8f214740 1 -- 172.21.15.70:0/4131091696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 msgr2=0x561d0b8a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.843+0000 7f9f8f214740 1 --2- 172.21.15.70:0/4131091696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 0x561d0b8a5190 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0cc870 tx=0x7f9f7c09c020 comp rx=0 tx=0).stop 2024-04-01T01:47:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.843+0000 7f9f8f214740 1 -- 172.21.15.70:0/4131091696 shutdown_connections 2024-04-01T01:47:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.843+0000 7f9f8f214740 1 --2- 172.21.15.70:0/4131091696 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561d0b8ba930 0x561d0b8bcd80 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.843+0000 7f9f8f214740 1 --2- 172.21.15.70:0/4131091696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 0x561d0b8a5190 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.843+0000 7f9f8f214740 1 --2- 172.21.15.70:0/4131091696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b89e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.843+0000 7f9f8f214740 1 --2- 172.21.15.70:0/4131091696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 0x561d0b89dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.843+0000 7f9f8f214740 1 -- 172.21.15.70:0/4131091696 >> 172.21.15.70:0/4131091696 conn(0x561d0b88ee40 msgr2=0x561d0b88df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.843+0000 7f9f6dffb640 0 client.19380 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.843+0000 7f9f8f214740 1 -- 172.21.15.70:0/4131091696 shutdown_connections 2024-04-01T01:47:09.843 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.843+0000 7f9f8f214740 1 -- 172.21.15.70:0/4131091696 wait complete. 2024-04-01T01:47:09.844 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.844+0000 7f9f8f214740 20 client.19380 trim_cache size 0 max 16384 2024-04-01T01:47:09.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.871+0000 7f9f8f214740 1 Processor -- start 2024-04-01T01:47:09.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.871+0000 7f9f8f214740 1 -- start start 2024-04-01T01:47:09.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.872+0000 7f9f8f214740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 0x561d0b889e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.872+0000 7f9f8f214740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b8a5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.872+0000 7f9f8f214740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 0x561d0b895620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.872+0000 7f9f8f214740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9f841a8130 con 0x561d0b899430 2024-04-01T01:47:09.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.872+0000 7f9f8f214740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9f841a9c70 con 0x561d0b8938d0 2024-04-01T01:47:09.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.872+0000 7f9f8f214740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9f841a8800 con 0x561d0b8942c0 2024-04-01T01:47:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.872+0000 7f9f8abcb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 0x561d0b895620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.872+0000 7f9f8abcb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 0x561d0b895620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44590/0 (socket says 172.21.15.70:44590) 2024-04-01T01:47:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.872+0000 7f9f8abcb640 1 -- 172.21.15.70:0/3616510653 learned_addr learned my addr 172.21.15.70:0/3616510653 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.872+0000 7f9f8a3ca640 1 --2- 172.21.15.70:0/3616510653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 0x561d0b889e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.873+0000 7f9f89bc9640 1 --2- 172.21.15.70:0/3616510653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b8a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.873+0000 7f9f8abcb640 1 -- 172.21.15.70:0/3616510653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 msgr2=0x561d0b8a5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.873+0000 7f9f8abcb640 1 --2- 172.21.15.70:0/3616510653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b8a5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.873+0000 7f9f8abcb640 1 -- 172.21.15.70:0/3616510653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 msgr2=0x561d0b889e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.873+0000 7f9f8abcb640 1 --2- 172.21.15.70:0/3616510653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 0x561d0b889e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.873+0000 7f9f8abcb640 1 -- 172.21.15.70:0/3616510653 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d0b89d7f0 con 0x561d0b899430 2024-04-01T01:47:09.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.873+0000 7f9f8a3ca640 1 --2- 172.21.15.70:0/3616510653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 0x561d0b889e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:47:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.873+0000 7f9f8abcb640 1 --2- 172.21.15.70:0/3616510653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 0x561d0b895620 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c09b560 tx=0x7f9f7c09b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65aaab69b90e3986 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:09.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.874+0000 7f9f6ffff640 1 -- 172.21.15.70:0/3616510653 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c0b3050 con 0x561d0b899430 2024-04-01T01:47:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.874+0000 7f9f6ffff640 1 -- 172.21.15.70:0/3616510653 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f7c0afb20 con 0x561d0b899430 2024-04-01T01:47:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.874+0000 7f9f6ffff640 1 -- 172.21.15.70:0/3616510653 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c09cdf0 con 0x561d0b899430 2024-04-01T01:47:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.875+0000 7f9f8f214740 1 -- 172.21.15.70:0/3616510653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 msgr2=0x561d0b895620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:09.875 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.875+0000 7f9f8f214740 1 --2- 172.21.15.70:0/3616510653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 0x561d0b895620 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c09b560 tx=0x7f9f7c09b9e0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.875+0000 7f9f8f214740 1 -- 172.21.15.70:0/3616510653 shutdown_connections 2024-04-01T01:47:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.875+0000 7f9f8f214740 1 --2- 172.21.15.70:0/3616510653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b899430 0x561d0b895620 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.875+0000 7f9f8f214740 1 --2- 172.21.15.70:0/3616510653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b8a5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.875+0000 7f9f8f214740 1 --2- 172.21.15.70:0/3616510653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b8938d0 0x561d0b889e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.875+0000 7f9f8f214740 1 -- 172.21.15.70:0/3616510653 >> 172.21.15.70:0/3616510653 conn(0x561d0b88ee40 msgr2=0x561d0b88f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.875+0000 7f9f8f214740 1 -- 172.21.15.70:0/3616510653 shutdown_connections 2024-04-01T01:47:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.875+0000 7f9f8f214740 1 -- 172.21.15.70:0/3616510653 wait complete. 2024-04-01T01:47:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.876+0000 7f9f8f214740 1 -- start start 2024-04-01T01:47:09.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.876+0000 7f9f8f214740 1 Processor -- start 2024-04-01T01:47:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.877+0000 7f9f8f214740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b8938d0 0x561d0b8a2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.877+0000 7f9f8f214740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b8a27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.877+0000 7f9f8f214740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b899430 0x561d0b8a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.877+0000 7f9f8f214740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9f841a95a0 con 0x561d0b8938d0 2024-04-01T01:47:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.877+0000 7f9f8f214740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9f841a8ed0 con 0x561d0b899430 2024-04-01T01:47:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.877+0000 7f9f8f214740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9f841a4100 con 0x561d0b8942c0 2024-04-01T01:47:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.877+0000 7f9f8f214740 10 client.0 _ll_register_callbacks cb 0x7f9f7c097440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.877+0000 7f9f89bc9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b8a27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.877+0000 7f9f8abcb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b899430 0x561d0b8a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.877+0000 7f9f89bc9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b8a27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57104/0 (socket says 172.21.15.70:57104) 2024-04-01T01:47:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.877+0000 7f9f89bc9640 1 -- 172.21.15.70:0/4016173388 learned_addr learned my addr 172.21.15.70:0/4016173388 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:09.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.877+0000 7f9f8a3ca640 1 --2- 172.21.15.70:0/4016173388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b8938d0 0x561d0b8a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.878+0000 7f9f8abcb640 1 -- 172.21.15.70:0/4016173388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 msgr2=0x561d0b8a27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.878+0000 7f9f8abcb640 1 --2- 172.21.15.70:0/4016173388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b8a27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.878+0000 7f9f8abcb640 1 -- 172.21.15.70:0/4016173388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b8938d0 msgr2=0x561d0b8a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.878+0000 7f9f8abcb640 1 --2- 172.21.15.70:0/4016173388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b8938d0 0x561d0b8a2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.878+0000 7f9f8abcb640 1 -- 172.21.15.70:0/4016173388 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d0b8a3d30 con 0x561d0b899430 2024-04-01T01:47:09.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.879+0000 7f9f8abcb640 1 --2- 172.21.15.70:0/4016173388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b899430 0x561d0b8a3740 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c097cc0 tx=0x7f9f7c09cda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43fd77995b792990 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.879+0000 7f9f6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.879+0000 7f9f6dffb640 1 -- 172.21.15.70:0/4016173388 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c0b3030 con 0x561d0b899430 2024-04-01T01:47:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.879+0000 7f9f6dffb640 1 -- 172.21.15.70:0/4016173388 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f9f7c0b4260 con 0x561d0b899430 2024-04-01T01:47:09.879 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.879+0000 7f9f8f214740 1 -- 172.21.15.70:0/4016173388 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f7c0afb20 con 0x561d0b899430 2024-04-01T01:47:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.879+0000 7f9f6dffb640 1 -- 172.21.15.70:0/4016173388 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9f7c0afcd0 con 0x561d0b899430 2024-04-01T01:47:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.879+0000 7f9f8f214740 10 client.29459 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.879+0000 7f9f8f214740 1 -- 172.21.15.70:0/4016173388 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561d0b8a5040 con 0x561d0b899430 2024-04-01T01:47:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.879+0000 7f9f6dffb640 1 -- 172.21.15.70:0/4016173388 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f7c0afb20 con 0x561d0b899430 2024-04-01T01:47:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.880+0000 7f9f6dffb640 1 -- 172.21.15.70:0/4016173388 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f9f7c0ab030 con 0x561d0b899430 2024-04-01T01:47:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.880+0000 7f9f8f214740 10 client.29459 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.880+0000 7f9f8f214740 10 client.29459 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.880+0000 7f9f8f214740 20 client.29459 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.880+0000 7f9f8f214740 1 --2- 172.21.15.70:0/4016173388 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561d0b920d80 0x561d0b923170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.880+0000 7f9f8f214740 4 client.29459 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:47:09.880 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.880+0000 7f9f8f214740 1 -- 172.21.15.70:0/4016173388 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561d0b804e30 con 0x561d0b920d80 2024-04-01T01:47:09.881 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.882+0000 7f9f8a3ca640 1 --2- 172.21.15.70:0/4016173388 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561d0b920d80 0x561d0b923170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.882+0000 7f9f8a3ca640 1 --2- 172.21.15.70:0/4016173388 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561d0b920d80 0x561d0b923170 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3e7d2d501e8a5e3 server_cookie=fc5999424b5729b5 in_seq=0 out_seq=0 2024-04-01T01:47:09.882 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.882+0000 7f9f6dffb640 10 client.29459 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.942+0000 7f9f6dffb640 1 -- 172.21.15.70:0/4016173388 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561d0b804e30 con 0x561d0b920d80 2024-04-01T01:47:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.942+0000 7f9f6dffb640 10 client.29459 handle_command_reply: tid=0 2024-04-01T01:47:09.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.942+0000 7f9f8f214740 1 client.29459 shutdown 2024-04-01T01:47:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.945+0000 7f9f8f214740 1 -- 172.21.15.70:0/4016173388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b899430 msgr2=0x561d0b8a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.945+0000 7f9f8f214740 1 --2- 172.21.15.70:0/4016173388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b899430 0x561d0b8a3740 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c097cc0 tx=0x7f9f7c09cda0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.945+0000 7f9f8f214740 1 -- 172.21.15.70:0/4016173388 shutdown_connections 2024-04-01T01:47:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.945+0000 7f9f8f214740 1 --2- 172.21.15.70:0/4016173388 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x561d0b920d80 0x561d0b923170 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.945+0000 7f9f8f214740 1 --2- 172.21.15.70:0/4016173388 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561d0b899430 0x561d0b8a3740 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.945+0000 7f9f8f214740 1 --2- 172.21.15.70:0/4016173388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d0b8942c0 0x561d0b8a27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.945+0000 7f9f8f214740 1 --2- 172.21.15.70:0/4016173388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d0b8938d0 0x561d0b8a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.945+0000 7f9f8f214740 1 -- 172.21.15.70:0/4016173388 >> 172.21.15.70:0/4016173388 conn(0x561d0b88ee40 msgr2=0x561d0b89d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.946+0000 7f9f6dffb640 0 client.29459 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:09.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.949+0000 7f9f8f214740 1 -- 172.21.15.70:0/4016173388 shutdown_connections 2024-04-01T01:47:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.950+0000 7f9f8f214740 1 -- 172.21.15.70:0/4016173388 wait complete. 2024-04-01T01:47:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.951+0000 7f9f8f214740 20 client.29459 trim_cache size 0 max 16384 2024-04-01T01:47:09.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:47:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:47:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:47:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:47:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.952+0000 7f9f8c62e640 1 -- 172.21.15.70:0/607927151 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9f700a26b0 msgr2=0x7f9f700a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.952+0000 7f9f8c62e640 1 --2- 172.21.15.70:0/607927151 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9f700a26b0 0x7f9f700a4b70 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9f740096c0 tx=0x7f9f74005b70 comp rx=0 tx=0).stop 2024-04-01T01:47:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.952+0000 7f9f8c62e640 1 -- 172.21.15.70:0/607927151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f84102220 msgr2=0x7f9f8419aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:09.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.952+0000 7f9f8c62e640 1 --2- 172.21.15.70:0/607927151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f84102220 0x7f9f8419aa20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c009fd0 tx=0x7f9f7c00ef10 comp rx=0 tx=0).stop 2024-04-01T01:47:09.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.953+0000 7f9f8c62e640 1 -- 172.21.15.70:0/607927151 shutdown_connections 2024-04-01T01:47:09.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.953+0000 7f9f8c62e640 1 --2- 172.21.15.70:0/607927151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f84102220 0x7f9f8419aa20 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.953+0000 7f9f8c62e640 1 --2- 172.21.15.70:0/607927151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f84101860 0x7f9f8419a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.953+0000 7f9f8c62e640 1 --2- 172.21.15.70:0/607927151 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9f700a26b0 0x7f9f700a4b70 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.953+0000 7f9f8c62e640 1 --2- 172.21.15.70:0/607927151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f84100670 0x7f9f840699a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:09.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.953+0000 7f9f8c62e640 1 -- 172.21.15.70:0/607927151 >> 172.21.15.70:0/607927151 conn(0x7f9f840fbbe0 msgr2=0x7f9f84104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:09.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.953+0000 7f9f8c62e640 1 -- 172.21.15.70:0/607927151 shutdown_connections 2024-04-01T01:47:09.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:09.953+0000 7f9f8c62e640 1 -- 172.21.15.70:0/607927151 wait complete. 2024-04-01T01:47:11.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:10 smithi081 ceph-mon[35142]: pgmap v769: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.6 MiB/s wr, 747 op/s 2024-04-01T01:47:11.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:10 smithi189 ceph-mon[35753]: pgmap v769: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.6 MiB/s wr, 747 op/s 2024-04-01T01:47:11.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:10 smithi070 ceph-mon[32179]: pgmap v769: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.6 MiB/s wr, 747 op/s 2024-04-01T01:47:11.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 745.0 s, 674.6 tps, lat 147.069 ms stddev 151.748 2024-04-01T01:47:13.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:12 smithi081 ceph-mon[35142]: pgmap v770: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.0 MiB/s wr, 688 op/s 2024-04-01T01:47:13.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:12 smithi189 ceph-mon[35753]: pgmap v770: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.0 MiB/s wr, 688 op/s 2024-04-01T01:47:13.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:12 smithi070 ceph-mon[32179]: pgmap v770: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.0 MiB/s wr, 688 op/s 2024-04-01T01:47:13.546 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:47:13.546 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:47:13.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.737+0000 7f6076ffd640 1 -- 172.21.15.70:0/3265218476 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60680328c0 con 0x7f6078108e60 2024-04-01T01:47:13.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.737+0000 7f607d03b640 1 -- 172.21.15.70:0/3265218476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6078108e60 msgr2=0x7f60781092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:13.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.737+0000 7f607d03b640 1 --2- 172.21.15.70:0/3265218476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6078108e60 0x7f60781092e0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f6068009ca0 tx=0x7f606802c100 comp rx=0 tx=0).stop 2024-04-01T01:47:13.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.738+0000 7f607d03b640 1 -- 172.21.15.70:0/3265218476 shutdown_connections 2024-04-01T01:47:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.738+0000 7f607d03b640 1 --2- 172.21.15.70:0/3265218476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6078109820 0x7f60781100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.738+0000 7f607d03b640 1 --2- 172.21.15.70:0/3265218476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6078108e60 0x7f60781092e0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.738+0000 7f607d03b640 1 --2- 172.21.15.70:0/3265218476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6078107c70 0x7f6078108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.738+0000 7f607d03b640 1 -- 172.21.15.70:0/3265218476 >> 172.21.15.70:0/3265218476 conn(0x7f6078076c20 msgr2=0x7f6078079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.739+0000 7f607d03b640 1 -- 172.21.15.70:0/3265218476 shutdown_connections 2024-04-01T01:47:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.739+0000 7f607d03b640 1 -- 172.21.15.70:0/3265218476 wait complete. 2024-04-01T01:47:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.739+0000 7f607d03b640 1 Processor -- start 2024-04-01T01:47:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.739+0000 7f607d03b640 1 -- start start 2024-04-01T01:47:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.740+0000 7f607d03b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6078107c70 0x7f607807fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.740+0000 7f607d03b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6078108e60 0x7f60780803a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.740+0000 7f607d03b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6078109820 0x7f60780813e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.740+0000 7f607d03b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f607807a470 con 0x7f6078109820 2024-04-01T01:47:13.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.740+0000 7f607d03b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f607807a2f0 con 0x7f6078108e60 2024-04-01T01:47:13.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.740+0000 7f607d03b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f607807a5f0 con 0x7f6078107c70 2024-04-01T01:47:13.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.740+0000 7f60777fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6078108e60 0x7f60780803a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:13.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.740+0000 7f60777fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6078108e60 0x7f60780803a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47754/0 (socket says 172.21.15.70:47754) 2024-04-01T01:47:13.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.740+0000 7f60777fe640 1 -- 172.21.15.70:0/180269933 learned_addr learned my addr 172.21.15.70:0/180269933 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:13.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.740+0000 7f6077fff640 1 --2- 172.21.15.70:0/180269933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6078107c70 0x7f607807fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:13.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.741+0000 7f60777fe640 1 -- 172.21.15.70:0/180269933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6078107c70 msgr2=0x7f607807fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:13.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.741+0000 7f60777fe640 1 --2- 172.21.15.70:0/180269933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6078107c70 0x7f607807fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.741+0000 7f60777fe640 1 -- 172.21.15.70:0/180269933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6078109820 msgr2=0x7f60780813e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:13.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.741+0000 7f60777fe640 1 --2- 172.21.15.70:0/180269933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6078109820 0x7f60780813e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.741+0000 7f60777fe640 1 -- 172.21.15.70:0/180269933 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6078081a50 con 0x7f6078108e60 2024-04-01T01:47:13.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.741+0000 7f60777fe640 1 --2- 172.21.15.70:0/180269933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6078108e60 0x7f60780803a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f606802c710 tx=0x7f6068035ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:13.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.743+0000 7f60757fa640 1 -- 172.21.15.70:0/180269933 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6068039040 con 0x7f6078108e60 2024-04-01T01:47:13.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.744+0000 7f60757fa640 1 -- 172.21.15.70:0/180269933 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6068035e50 con 0x7f6078108e60 2024-04-01T01:47:13.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.744+0000 7f607d03b640 1 -- 172.21.15.70:0/180269933 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6078081ce0 con 0x7f6078108e60 2024-04-01T01:47:13.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.744+0000 7f607d03b640 1 -- 172.21.15.70:0/180269933 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60780821d0 con 0x7f6078108e60 2024-04-01T01:47:13.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.745+0000 7f60757fa640 1 -- 172.21.15.70:0/180269933 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60680043c0 con 0x7f6078108e60 2024-04-01T01:47:13.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.747+0000 7f6080e81740 1 Processor -- start 2024-04-01T01:47:13.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.747+0000 7f6080e81740 1 -- start start 2024-04-01T01:47:13.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.747+0000 7f6080e81740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf382490 0x563faf3828b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.747+0000 7f6080e81740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf382e80 0x563faf387980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.747+0000 7f6080e81740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf387ff0 0x563faf38a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.747+0000 7f6080e81740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563faf2e7180 con 0x563faf382e80 2024-04-01T01:47:13.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.747+0000 7f6080e81740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563faf28fb70 con 0x563faf382490 2024-04-01T01:47:13.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.747+0000 7f6080e81740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563faf2e3220 con 0x563faf387ff0 2024-04-01T01:47:13.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.748+0000 7f607c83a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf382490 0x563faf3828b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:13.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.748+0000 7f607c83a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf382490 0x563faf3828b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47768/0 (socket says 172.21.15.70:47768) 2024-04-01T01:47:13.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.748+0000 7f607c83a640 1 -- 172.21.15.70:0/1499036204 learned_addr learned my addr 172.21.15.70:0/1499036204 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:13.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.748+0000 7f6077fff640 1 --2- 172.21.15.70:0/1499036204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf382e80 0x563faf387980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:13.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.748+0000 7f60757fa640 1 -- 172.21.15.70:0/180269933 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f6068004560 con 0x7f6078108e60 2024-04-01T01:47:13.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.748+0000 7f607c83a640 1 -- 172.21.15.70:0/1499036204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf387ff0 msgr2=0x563faf38a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:13.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.748+0000 7f607c83a640 1 --2- 172.21.15.70:0/1499036204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf387ff0 0x563faf38a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.748+0000 7f607c83a640 1 -- 172.21.15.70:0/1499036204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf382e80 msgr2=0x563faf387980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:13.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.748+0000 7f607c83a640 1 --2- 172.21.15.70:0/1499036204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf382e80 0x563faf387980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.748+0000 7f607c83a640 1 -- 172.21.15.70:0/1499036204 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563faf2a37c0 con 0x563faf382490 2024-04-01T01:47:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.748+0000 7f60757fa640 1 --2- 172.21.15.70:0/180269933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60600a2620 0x7f60600a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.749+0000 7f60757fa640 1 -- 172.21.15.70:0/180269933 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f60680f0630 con 0x7f6078108e60 2024-04-01T01:47:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.749+0000 7f6077fff640 1 --2- 172.21.15.70:0/180269933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60600a2620 0x7f60600a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.749+0000 7f607c83a640 1 --2- 172.21.15.70:0/1499036204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf382490 0x563faf3828b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6070009c40 tx=0x7f607000e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e16ed0dddeed71e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.750+0000 7f605e7fc640 1 -- 172.21.15.70:0/1499036204 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070005020 con 0x563faf382490 2024-04-01T01:47:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.750+0000 7f605e7fc640 1 -- 172.21.15.70:0/1499036204 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60700047a0 con 0x563faf382490 2024-04-01T01:47:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.750+0000 7f605e7fc640 1 -- 172.21.15.70:0/1499036204 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070010650 con 0x563faf382490 2024-04-01T01:47:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.750+0000 7f6077fff640 1 --2- 172.21.15.70:0/180269933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60600a2620 0x7f60600a4ae0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x563faf383280 tx=0x7f606c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.750+0000 7f6080e81740 1 -- 172.21.15.70:0/1499036204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf382490 msgr2=0x563faf3828b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:13.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.751+0000 7f6080e81740 1 --2- 172.21.15.70:0/1499036204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf382490 0x563faf3828b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6070009c40 tx=0x7f607000e720 comp rx=0 tx=0).stop 2024-04-01T01:47:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.751+0000 7f6080e81740 1 -- 172.21.15.70:0/1499036204 shutdown_connections 2024-04-01T01:47:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.751+0000 7f6080e81740 1 --2- 172.21.15.70:0/1499036204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf387ff0 0x563faf38a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.751+0000 7f6080e81740 1 --2- 172.21.15.70:0/1499036204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf382e80 0x563faf387980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.751+0000 7f6080e81740 1 --2- 172.21.15.70:0/1499036204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf382490 0x563faf3828b0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.751+0000 7f6080e81740 1 -- 172.21.15.70:0/1499036204 >> 172.21.15.70:0/1499036204 conn(0x563faf37da00 msgr2=0x563faf37de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.751+0000 7f6080e81740 1 -- 172.21.15.70:0/1499036204 shutdown_connections 2024-04-01T01:47:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.751+0000 7f6080e81740 1 -- 172.21.15.70:0/1499036204 wait complete. 2024-04-01T01:47:13.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.752+0000 7f6080e81740 1 -- start start 2024-04-01T01:47:13.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.752+0000 7f6080e81740 1 Processor -- start 2024-04-01T01:47:13.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.752+0000 7f6080e81740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf382490 0x563faf38caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.752+0000 7f6080e81740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf382e80 0x563faf38cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.752+0000 7f6080e81740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf387ff0 0x563faf394020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.752+0000 7f6080e81740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563faf38afa0 con 0x563faf382490 2024-04-01T01:47:13.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.752+0000 7f6080e81740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563faf2854a0 con 0x563faf387ff0 2024-04-01T01:47:13.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.752+0000 7f6080e81740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563faf38b120 con 0x563faf382e80 2024-04-01T01:47:13.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.752+0000 7f6080e81740 10 client.0 _ll_register_callbacks cb 0x563faf288d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:13.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.753+0000 7f60777fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf382e80 0x563faf38cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:13.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.753+0000 7f60777fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf382e80 0x563faf38cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57130/0 (socket says 172.21.15.70:57130) 2024-04-01T01:47:13.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.753+0000 7f60777fe640 1 -- 172.21.15.70:0/3518917366 learned_addr learned my addr 172.21.15.70:0/3518917366 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:13.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.753+0000 7f6077fff640 1 --2- 172.21.15.70:0/3518917366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf382490 0x563faf38caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:13.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.753+0000 7f607c83a640 1 --2- 172.21.15.70:0/3518917366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf387ff0 0x563faf394020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:13.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.753+0000 7f60777fe640 1 -- 172.21.15.70:0/3518917366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf387ff0 msgr2=0x563faf394020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:13.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.753+0000 7f60777fe640 1 --2- 172.21.15.70:0/3518917366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf387ff0 0x563faf394020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.753+0000 7f60777fe640 1 -- 172.21.15.70:0/3518917366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf382490 msgr2=0x563faf38caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:13.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.753+0000 7f60777fe640 1 --2- 172.21.15.70:0/3518917366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf382490 0x563faf38caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.753+0000 7f60777fe640 1 -- 172.21.15.70:0/3518917366 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563faf0c57d0 con 0x563faf382e80 2024-04-01T01:47:13.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.754+0000 7f60777fe640 1 --2- 172.21.15.70:0/3518917366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf382e80 0x563faf38cfe0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6068006010 tx=0x7f60680bc920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b8f5ef11507c6b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:13.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.754+0000 7f60577fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:47:13.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.756+0000 7f60577fe640 1 -- 172.21.15.70:0/3518917366 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60680d5060 con 0x563faf382e80 2024-04-01T01:47:13.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.756+0000 7f6080e81740 1 -- 172.21.15.70:0/3518917366 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60700047a0 con 0x563faf382e80 2024-04-01T01:47:13.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.756+0000 7f60577fe640 1 -- 172.21.15.70:0/3518917366 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f60680f0360 con 0x563faf382e80 2024-04-01T01:47:13.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.758+0000 7f60577fe640 1 -- 172.21.15.70:0/3518917366 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f60680da050 con 0x563faf382e80 2024-04-01T01:47:13.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.762+0000 7f6080e81740 10 client.29239 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:13.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.762+0000 7f6080e81740 1 -- 172.21.15.70:0/3518917366 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563faeffb3f0 con 0x563faf382e80 2024-04-01T01:47:13.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.764+0000 7f60577fe640 1 -- 172.21.15.70:0/3518917366 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60700047a0 con 0x563faf382e80 2024-04-01T01:47:13.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.765+0000 7f60577fe640 1 -- 172.21.15.70:0/3518917366 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f606803c030 con 0x563faf382e80 2024-04-01T01:47:13.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.765+0000 7f6080e81740 10 client.29239 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:13.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.765+0000 7f6080e81740 10 client.29239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:13.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.765+0000 7f6080e81740 20 client.29239 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:13.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.765+0000 7f6080e81740 1 --2- 172.21.15.70:0/3518917366 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563faf3a9ba0 0x563faf3abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.765+0000 7f6080e81740 4 client.29239 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:13.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.765+0000 7f6080e81740 1 -- 172.21.15.70:0/3518917366 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563faf2f46c0 con 0x563faf3a9ba0 2024-04-01T01:47:13.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.769+0000 7f6077fff640 1 --2- 172.21.15.70:0/3518917366 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563faf3a9ba0 0x563faf3abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:13.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.769+0000 7f6077fff640 1 --2- 172.21.15.70:0/3518917366 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563faf3a9ba0 0x563faf3abfc0 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bda6af551dff048 server_cookie=ac149dd7cabda47e in_seq=0 out_seq=0 2024-04-01T01:47:13.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.769+0000 7f60577fe640 10 client.29239 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:13.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.772+0000 7f60577fe640 1 -- 172.21.15.70:0/3518917366 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563faf2f46c0 con 0x563faf3a9ba0 2024-04-01T01:47:13.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.772+0000 7f60577fe640 10 client.29239 handle_command_reply: tid=0 2024-04-01T01:47:13.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.772+0000 7f6080e81740 1 client.29239 shutdown 2024-04-01T01:47:13.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.772+0000 7f6080e81740 1 -- 172.21.15.70:0/3518917366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf382e80 msgr2=0x563faf38cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:13.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.772+0000 7f6080e81740 1 --2- 172.21.15.70:0/3518917366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf382e80 0x563faf38cfe0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6068006010 tx=0x7f60680bc920 comp rx=0 tx=0).stop 2024-04-01T01:47:13.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.772+0000 7f6080e81740 1 -- 172.21.15.70:0/3518917366 shutdown_connections 2024-04-01T01:47:13.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.772+0000 7f6080e81740 1 --2- 172.21.15.70:0/3518917366 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563faf3a9ba0 0x563faf3abfc0 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.772+0000 7f6080e81740 1 --2- 172.21.15.70:0/3518917366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf387ff0 0x563faf394020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.772+0000 7f6080e81740 1 --2- 172.21.15.70:0/3518917366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf382e80 0x563faf38cfe0 secure :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f6068006010 tx=0x7f60680bc920 comp rx=0 tx=0).stop 2024-04-01T01:47:13.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.772+0000 7f6080e81740 1 --2- 172.21.15.70:0/3518917366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf382490 0x563faf38caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.772+0000 7f6080e81740 1 -- 172.21.15.70:0/3518917366 >> 172.21.15.70:0/3518917366 conn(0x563faf37da00 msgr2=0x563faf37cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:13.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.773+0000 7f60577fe640 0 client.29239 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:13.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.774+0000 7f6080e81740 1 -- 172.21.15.70:0/3518917366 shutdown_connections 2024-04-01T01:47:13.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.775+0000 7f6080e81740 1 -- 172.21.15.70:0/3518917366 wait complete. 2024-04-01T01:47:13.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.775+0000 7f6080e81740 20 client.29239 trim_cache size 0 max 16384 2024-04-01T01:47:13.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.808+0000 7f6080e81740 1 Processor -- start 2024-04-01T01:47:13.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.808+0000 7f6080e81740 1 -- start start 2024-04-01T01:47:13.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.808+0000 7f6080e81740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf382490 0x563faf37c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.808+0000 7f6080e81740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf387ff0 0x563faf393fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.808+0000 7f6080e81740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf37e240 0x563faf37c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.808+0000 7f6080e81740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f606400fd50 con 0x563faf382490 2024-04-01T01:47:13.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.808+0000 7f6080e81740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6064011890 con 0x563faf387ff0 2024-04-01T01:47:13.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.808+0000 7f6080e81740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6064010420 con 0x563faf37e240 2024-04-01T01:47:13.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.808+0000 7f607c83a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf37e240 0x563faf37c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:13.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.808+0000 7f607c83a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf37e240 0x563faf37c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57144/0 (socket says 172.21.15.70:57144) 2024-04-01T01:47:13.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.808+0000 7f607c83a640 1 -- 172.21.15.70:0/1819588016 learned_addr learned my addr 172.21.15.70:0/1819588016 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:13.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.808+0000 7f6077fff640 1 --2- 172.21.15.70:0/1819588016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf382490 0x563faf37c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:13.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.809+0000 7f60777fe640 1 --2- 172.21.15.70:0/1819588016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf387ff0 0x563faf393fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:13.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.809+0000 7f60777fe640 1 -- 172.21.15.70:0/1819588016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf37e240 msgr2=0x563faf37c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:13.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.809+0000 7f60777fe640 1 --2- 172.21.15.70:0/1819588016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf37e240 0x563faf37c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.809+0000 7f60777fe640 1 -- 172.21.15.70:0/1819588016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf382490 msgr2=0x563faf37c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:13.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.809+0000 7f60777fe640 1 --2- 172.21.15.70:0/1819588016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf382490 0x563faf37c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.809+0000 7f60777fe640 1 -- 172.21.15.70:0/1819588016 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563faefbb0f0 con 0x563faf387ff0 2024-04-01T01:47:13.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.809+0000 7f60777fe640 1 --2- 172.21.15.70:0/1819588016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf387ff0 0x563faf393fe0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6068032380 tx=0x7f60680bb230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb963bb779ffed55 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:13.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.809+0000 7f605dffb640 1 -- 172.21.15.70:0/1819588016 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6068003970 con 0x563faf387ff0 2024-04-01T01:47:13.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.809+0000 7f605dffb640 1 -- 172.21.15.70:0/1819588016 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60680c0e70 con 0x563faf387ff0 2024-04-01T01:47:13.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.810+0000 7f605dffb640 1 -- 172.21.15.70:0/1819588016 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60680b8fc0 con 0x563faf387ff0 2024-04-01T01:47:13.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.811+0000 7f6080e81740 1 -- 172.21.15.70:0/1819588016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf387ff0 msgr2=0x563faf393fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:13.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.811+0000 7f6080e81740 1 --2- 172.21.15.70:0/1819588016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf387ff0 0x563faf393fe0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6068032380 tx=0x7f60680bb230 comp rx=0 tx=0).stop 2024-04-01T01:47:13.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.811+0000 7f6080e81740 1 -- 172.21.15.70:0/1819588016 shutdown_connections 2024-04-01T01:47:13.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.811+0000 7f6080e81740 1 --2- 172.21.15.70:0/1819588016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf37e240 0x563faf37c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.811+0000 7f6080e81740 1 --2- 172.21.15.70:0/1819588016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf387ff0 0x563faf393fe0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.811+0000 7f6080e81740 1 --2- 172.21.15.70:0/1819588016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf382490 0x563faf37c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.811+0000 7f6080e81740 1 -- 172.21.15.70:0/1819588016 >> 172.21.15.70:0/1819588016 conn(0x563faf37da00 msgr2=0x563faf384410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:13.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.812+0000 7f6080e81740 1 -- 172.21.15.70:0/1819588016 shutdown_connections 2024-04-01T01:47:13.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.812+0000 7f6080e81740 1 -- 172.21.15.70:0/1819588016 wait complete. 2024-04-01T01:47:13.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.812+0000 7f6080e81740 1 -- start start 2024-04-01T01:47:13.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.812+0000 7f6080e81740 1 Processor -- start 2024-04-01T01:47:13.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.813+0000 7f6080e81740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf37e240 0x563faf3917c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.813+0000 7f6080e81740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf382490 0x563faf391d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.813+0000 7f6080e81740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf387ff0 0x563faf392d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.813+0000 7f60777fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf382490 0x563faf391d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:13.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.813+0000 7f60777fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf382490 0x563faf391d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47790/0 (socket says 172.21.15.70:47790) 2024-04-01T01:47:13.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.813+0000 7f6077fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf37e240 0x563faf3917c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:13.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.813+0000 7f60777fe640 1 -- 172.21.15.70:0/3346357940 learned_addr learned my addr 172.21.15.70:0/3346357940 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:13.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.813+0000 7f6080e81740 1 -- 172.21.15.70:0/3346357940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f60640111c0 con 0x563faf387ff0 2024-04-01T01:47:13.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.813+0000 7f6080e81740 1 -- 172.21.15.70:0/3346357940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6064010af0 con 0x563faf382490 2024-04-01T01:47:13.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.813+0000 7f607c83a640 1 --2- 172.21.15.70:0/3346357940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf387ff0 0x563faf392d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:13.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.813+0000 7f6080e81740 1 -- 172.21.15.70:0/3346357940 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f606400dd30 con 0x563faf37e240 2024-04-01T01:47:13.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.814+0000 7f6080e81740 10 client.0 _ll_register_callbacks cb 0x7f60680bcaf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:13.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.814+0000 7f60777fe640 1 -- 172.21.15.70:0/3346357940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf37e240 msgr2=0x563faf3917c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:13.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.814+0000 7f60777fe640 1 --2- 172.21.15.70:0/3346357940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf37e240 0x563faf3917c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.814+0000 7f60777fe640 1 -- 172.21.15.70:0/3346357940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf387ff0 msgr2=0x563faf392d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:13.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.814+0000 7f60777fe640 1 --2- 172.21.15.70:0/3346357940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf387ff0 0x563faf392d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.814+0000 7f60777fe640 1 -- 172.21.15.70:0/3346357940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563faf393300 con 0x563faf382490 2024-04-01T01:47:13.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.814+0000 7f60777fe640 1 --2- 172.21.15.70:0/3346357940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf382490 0x563faf391d00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f60680bcee0 tx=0x7f60680db4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c241186a7652a462 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:13.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.815+0000 7f60577fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:13.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.815+0000 7f60577fe640 1 -- 172.21.15.70:0/3346357940 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60680d5070 con 0x563faf382490 2024-04-01T01:47:13.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.815+0000 7f60577fe640 1 -- 172.21.15.70:0/3346357940 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6068009890 con 0x563faf382490 2024-04-01T01:47:13.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.815+0000 7f6080e81740 1 -- 172.21.15.70:0/3346357940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60680c0e70 con 0x563faf382490 2024-04-01T01:47:13.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.815+0000 7f60577fe640 1 -- 172.21.15.70:0/3346357940 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f60680a0050 con 0x563faf382490 2024-04-01T01:47:13.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.815+0000 7f6080e81740 10 client.29477 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:13.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.815+0000 7f6080e81740 1 -- 172.21.15.70:0/3346357940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563faf393300 con 0x563faf382490 2024-04-01T01:47:13.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.816+0000 7f60577fe640 1 -- 172.21.15.70:0/3346357940 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6068003970 con 0x563faf382490 2024-04-01T01:47:13.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.816+0000 7f60577fe640 1 -- 172.21.15.70:0/3346357940 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f60680325c0 con 0x563faf382490 2024-04-01T01:47:13.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.816+0000 7f6080e81740 10 client.29477 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:13.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.816+0000 7f6080e81740 10 client.29477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:13.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.816+0000 7f6080e81740 20 client.29477 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:13.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.816+0000 7f6080e81740 1 --2- 172.21.15.70:0/3346357940 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563faf4104b0 0x563faf4128d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:13.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.816+0000 7f6080e81740 4 client.29477 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:47:13.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.816+0000 7f6080e81740 1 -- 172.21.15.70:0/3346357940 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f60680c0e70 con 0x563faf4104b0 2024-04-01T01:47:13.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.816+0000 7f6077fff640 1 --2- 172.21.15.70:0/3346357940 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563faf4104b0 0x563faf4128d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:13.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.817+0000 7f6077fff640 1 --2- 172.21.15.70:0/3346357940 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563faf4104b0 0x563faf4128d0 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9a406f778fe6bd6 server_cookie=e1782d472bec24ef in_seq=0 out_seq=0 2024-04-01T01:47:13.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.817+0000 7f60577fe640 10 client.29477 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:13.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.817+0000 7f60577fe640 1 -- 172.21.15.70:0/3346357940 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f60680c0e70 con 0x563faf4104b0 2024-04-01T01:47:13.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.817+0000 7f60577fe640 10 client.29477 handle_command_reply: tid=0 2024-04-01T01:47:13.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.817+0000 7f6080e81740 1 client.29477 shutdown 2024-04-01T01:47:13.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.820+0000 7f6080e81740 1 -- 172.21.15.70:0/3346357940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf382490 msgr2=0x563faf391d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:13.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.820+0000 7f6080e81740 1 --2- 172.21.15.70:0/3346357940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf382490 0x563faf391d00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f60680bcee0 tx=0x7f60680db4f0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.820+0000 7f6080e81740 1 -- 172.21.15.70:0/3346357940 shutdown_connections 2024-04-01T01:47:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.820+0000 7f6080e81740 1 --2- 172.21.15.70:0/3346357940 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563faf4104b0 0x563faf4128d0 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.820+0000 7f6080e81740 1 --2- 172.21.15.70:0/3346357940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563faf387ff0 0x563faf392d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.820+0000 7f6080e81740 1 --2- 172.21.15.70:0/3346357940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563faf382490 0x563faf391d00 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.820+0000 7f6080e81740 1 --2- 172.21.15.70:0/3346357940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563faf37e240 0x563faf3917c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.820+0000 7f6080e81740 1 -- 172.21.15.70:0/3346357940 >> 172.21.15.70:0/3346357940 conn(0x563faf37da00 msgr2=0x563faf380c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.820+0000 7f60577fe640 0 client.29477 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.821+0000 7f6080e81740 1 -- 172.21.15.70:0/3346357940 shutdown_connections 2024-04-01T01:47:13.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.821+0000 7f6080e81740 1 -- 172.21.15.70:0/3346357940 wait complete. 2024-04-01T01:47:13.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.821+0000 7f6080e81740 20 client.29477 trim_cache size 0 max 16384 2024-04-01T01:47:13.825 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:47:13.825 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:47:13.825 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:47:13.825 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:47:13.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.822+0000 7f607d03b640 1 -- 172.21.15.70:0/180269933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60600a2620 msgr2=0x7f60600a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:13.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.822+0000 7f607d03b640 1 --2- 172.21.15.70:0/180269933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60600a2620 0x7f60600a4ae0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x563faf383280 tx=0x7f606c005f90 comp rx=0 tx=0).stop 2024-04-01T01:47:13.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.822+0000 7f607d03b640 1 -- 172.21.15.70:0/180269933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6078108e60 msgr2=0x7f60780803a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:13.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.822+0000 7f607d03b640 1 --2- 172.21.15.70:0/180269933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6078108e60 0x7f60780803a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f606802c710 tx=0x7f6068035ce0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.822+0000 7f607d03b640 1 -- 172.21.15.70:0/180269933 shutdown_connections 2024-04-01T01:47:13.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.822+0000 7f607d03b640 1 --2- 172.21.15.70:0/180269933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6078109820 0x7f60780813e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.822+0000 7f607d03b640 1 --2- 172.21.15.70:0/180269933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6078108e60 0x7f60780803a0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.822+0000 7f607d03b640 1 --2- 172.21.15.70:0/180269933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60600a2620 0x7f60600a4ae0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.822+0000 7f607d03b640 1 --2- 172.21.15.70:0/180269933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6078107c70 0x7f607807fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:13.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.822+0000 7f607d03b640 1 -- 172.21.15.70:0/180269933 >> 172.21.15.70:0/180269933 conn(0x7f6078076c20 msgr2=0x7f607810c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:13.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.822+0000 7f607d03b640 1 -- 172.21.15.70:0/180269933 shutdown_connections 2024-04-01T01:47:13.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:13.822+0000 7f607d03b640 1 -- 172.21.15.70:0/180269933 wait complete. 2024-04-01T01:47:13.841 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:47:13.841 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:47:14.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.008+0000 7f4f95324640 1 -- 172.21.15.70:0/1442157353 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f880328c0 con 0x7f4f90102230 2024-04-01T01:47:14.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.009+0000 7f4f9858a640 1 -- 172.21.15.70:0/1442157353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f90102230 msgr2=0x7f4f90108ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.009+0000 7f4f9858a640 1 --2- 172.21.15.70:0/1442157353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f90102230 0x7f4f90108ac0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f4f88009ca0 tx=0x7f4f8802c100 comp rx=0 tx=0).stop 2024-04-01T01:47:14.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.009+0000 7f4f9858a640 1 -- 172.21.15.70:0/1442157353 shutdown_connections 2024-04-01T01:47:14.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.009+0000 7f4f9858a640 1 --2- 172.21.15.70:0/1442157353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f90102230 0x7f4f90108ac0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.009+0000 7f4f9858a640 1 --2- 172.21.15.70:0/1442157353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f90101870 0x7f4f90101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.009+0000 7f4f9858a640 1 --2- 172.21.15.70:0/1442157353 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f90100680 0x7f4f90100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.009+0000 7f4f9858a640 1 -- 172.21.15.70:0/1442157353 >> 172.21.15.70:0/1442157353 conn(0x7f4f900fbc30 msgr2=0x7f4f900fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:14.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.009+0000 7f4f9858a640 1 -- 172.21.15.70:0/1442157353 shutdown_connections 2024-04-01T01:47:14.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.009+0000 7f4f9858a640 1 -- 172.21.15.70:0/1442157353 wait complete. 2024-04-01T01:47:14.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f9858a640 1 Processor -- start 2024-04-01T01:47:14.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f9858a640 1 -- start start 2024-04-01T01:47:14.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f9858a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f90100680 0x7f4f9019ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f9858a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f90101870 0x7f4f9019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f9858a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f90102230 0x7f4f901a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f9858a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9010f480 con 0x7f4f90100680 2024-04-01T01:47:14.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f9858a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9010f300 con 0x7f4f90101870 2024-04-01T01:47:14.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f9858a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9010f600 con 0x7f4f90102230 2024-04-01T01:47:14.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f96326640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f90100680 0x7f4f9019ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f96326640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f90100680 0x7f4f9019ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44652/0 (socket says 172.21.15.70:44652) 2024-04-01T01:47:14.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f95b25640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f90101870 0x7f4f9019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f96326640 1 -- 172.21.15.70:0/3942499383 learned_addr learned my addr 172.21.15.70:0/3942499383 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f96b27640 1 --2- 172.21.15.70:0/3942499383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f90102230 0x7f4f901a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f96326640 1 -- 172.21.15.70:0/3942499383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f90102230 msgr2=0x7f4f901a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f96326640 1 --2- 172.21.15.70:0/3942499383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f90102230 0x7f4f901a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f96326640 1 -- 172.21.15.70:0/3942499383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f90101870 msgr2=0x7f4f9019e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f96326640 1 --2- 172.21.15.70:0/3942499383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f90101870 0x7f4f9019e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f96326640 1 -- 172.21.15.70:0/3942499383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f901a5af0 con 0x7f4f90100680 2024-04-01T01:47:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f96b27640 1 --2- 172.21.15.70:0/3942499383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f90102230 0x7f4f901a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:47:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.010+0000 7f4f95b25640 1 --2- 172.21.15.70:0/3942499383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f90101870 0x7f4f9019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:47:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.011+0000 7f4f96326640 1 --2- 172.21.15.70:0/3942499383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f90100680 0x7f4f9019ded0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00e9c0 tx=0x7f4f8c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.011+0000 7f4f877fe640 1 -- 172.21.15.70:0/3942499383 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c00ccc0 con 0x7f4f90100680 2024-04-01T01:47:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.011+0000 7f4f9858a640 1 -- 172.21.15.70:0/3942499383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f901a5de0 con 0x7f4f90100680 2024-04-01T01:47:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.011+0000 7f4f877fe640 1 -- 172.21.15.70:0/3942499383 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4f8c0035f0 con 0x7f4f90100680 2024-04-01T01:47:14.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.011+0000 7f4f877fe640 1 -- 172.21.15.70:0/3942499383 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c00ce60 con 0x7f4f90100680 2024-04-01T01:47:14.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.011+0000 7f4f9858a640 1 -- 172.21.15.70:0/3942499383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f901a6240 con 0x7f4f90100680 2024-04-01T01:47:14.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.012+0000 7f4f9858a640 1 -- 172.21.15.70:0/3942499383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f901a6550 con 0x7f4f90100680 2024-04-01T01:47:14.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.013+0000 7f4f877fe640 1 -- 172.21.15.70:0/3942499383 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f4f8c0037a0 con 0x7f4f90100680 2024-04-01T01:47:14.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.014+0000 7f4f877fe640 1 --2- 172.21.15.70:0/3942499383 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4f7c0a24a0 0x7f4f7c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.015+0000 7f4f95b25640 1 --2- 172.21.15.70:0/3942499383 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4f7c0a24a0 0x7f4f7c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.015+0000 7f4f877fe640 1 -- 172.21.15.70:0/3942499383 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f4f8c0cf070 con 0x7f4f90100680 2024-04-01T01:47:14.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.015+0000 7f4f95b25640 1 --2- 172.21.15.70:0/3942499383 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4f7c0a24a0 0x7f4f7c0a4960 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4f9019f3e0 tx=0x7f4f8000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:14.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.022+0000 7f4f877fe640 1 -- 172.21.15.70:0/3942499383 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4f8c09be90 con 0x7f4f90100680 2024-04-01T01:47:14.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.388+0000 7f4f9858a640 1 -- 172.21.15.70:0/3942499383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f901a6720 con 0x7f4f90100680 2024-04-01T01:47:14.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.389+0000 7f4f877fe640 1 -- 172.21.15.70:0/3942499383 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7f4f8c09b790 con 0x7f4f90100680 2024-04-01T01:47:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:47:14.388 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:47:14.389 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:47:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.392+0000 7f4f9858a640 1 -- 172.21.15.70:0/3942499383 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4f7c0a24a0 msgr2=0x7f4f7c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.392+0000 7f4f9858a640 1 --2- 172.21.15.70:0/3942499383 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4f7c0a24a0 0x7f4f7c0a4960 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f4f9019f3e0 tx=0x7f4f8000a2d0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.392+0000 7f4f9858a640 1 -- 172.21.15.70:0/3942499383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f90100680 msgr2=0x7f4f9019ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.392+0000 7f4f9858a640 1 --2- 172.21.15.70:0/3942499383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f90100680 0x7f4f9019ded0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00e9c0 tx=0x7f4f8c00edc0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.392+0000 7f4f9858a640 1 -- 172.21.15.70:0/3942499383 shutdown_connections 2024-04-01T01:47:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.392+0000 7f4f9858a640 1 --2- 172.21.15.70:0/3942499383 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4f7c0a24a0 0x7f4f7c0a4960 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.392+0000 7f4f9858a640 1 --2- 172.21.15.70:0/3942499383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f90102230 0x7f4f901a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.392+0000 7f4f9858a640 1 --2- 172.21.15.70:0/3942499383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f90101870 0x7f4f9019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.392+0000 7f4f9858a640 1 --2- 172.21.15.70:0/3942499383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f90100680 0x7f4f9019ded0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.392+0000 7f4f9858a640 1 -- 172.21.15.70:0/3942499383 >> 172.21.15.70:0/3942499383 conn(0x7f4f900fbc30 msgr2=0x7f4f9010b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.392+0000 7f4f9858a640 1 -- 172.21.15.70:0/3942499383 shutdown_connections 2024-04-01T01:47:14.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.392+0000 7f4f9858a640 1 -- 172.21.15.70:0/3942499383 wait complete. 2024-04-01T01:47:14.407 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:47:14.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.579+0000 7fd2d34ed640 1 -- 172.21.15.70:0/1844084773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2cc108750 msgr2=0x7fd2cc10ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.579+0000 7fd2d34ed640 1 --2- 172.21.15.70:0/1844084773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2cc108750 0x7fd2cc10ab80 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009860 tx=0x7fd2c802bb70 comp rx=0 tx=0).stop 2024-04-01T01:47:14.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.579+0000 7fd2d34ed640 1 -- 172.21.15.70:0/1844084773 shutdown_connections 2024-04-01T01:47:14.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.579+0000 7fd2d34ed640 1 --2- 172.21.15.70:0/1844084773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2cc10da70 0x7fd2cc10def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.579+0000 7fd2d34ed640 1 --2- 172.21.15.70:0/1844084773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2cc10b0c0 0x7fd2cc10d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.579+0000 7fd2d34ed640 1 --2- 172.21.15.70:0/1844084773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2cc108750 0x7fd2cc10ab80 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.579+0000 7fd2d34ed640 1 -- 172.21.15.70:0/1844084773 >> 172.21.15.70:0/1844084773 conn(0x7fd2cc06e5b0 msgr2=0x7fd2cc0709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:14.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.579+0000 7fd2d34ed640 1 -- 172.21.15.70:0/1844084773 shutdown_connections 2024-04-01T01:47:14.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.579+0000 7fd2d34ed640 1 -- 172.21.15.70:0/1844084773 wait complete. 2024-04-01T01:47:14.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.580+0000 7fd2d34ed640 1 Processor -- start 2024-04-01T01:47:14.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.580+0000 7fd2d34ed640 1 -- start start 2024-04-01T01:47:14.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.580+0000 7fd2d34ed640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2cc108750 0x7fd2cc117ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.580+0000 7fd2d34ed640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2cc10b0c0 0x7fd2cc118410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.580+0000 7fd2d34ed640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2cc10da70 0x7fd2cc119450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.580+0000 7fd2d34ed640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc110430 con 0x7fd2cc108750 2024-04-01T01:47:14.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.580+0000 7fd2d34ed640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc1102b0 con 0x7fd2cc10da70 2024-04-01T01:47:14.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.580+0000 7fd2d34ed640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc1105b0 con 0x7fd2cc10b0c0 2024-04-01T01:47:14.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.580+0000 7fd2d1289640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2cc108750 0x7fd2cc117ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.581+0000 7fd2d1a8a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2cc10da70 0x7fd2cc119450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.581+0000 7fd2d1289640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2cc108750 0x7fd2cc117ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44672/0 (socket says 172.21.15.70:44672) 2024-04-01T01:47:14.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.581+0000 7fd2d1289640 1 -- 172.21.15.70:0/3176440138 learned_addr learned my addr 172.21.15.70:0/3176440138 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:14.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.581+0000 7fd2d0a88640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2cc10b0c0 0x7fd2cc118410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.581+0000 7fd2d1289640 1 -- 172.21.15.70:0/3176440138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2cc10b0c0 msgr2=0x7fd2cc118410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.581+0000 7fd2d1289640 1 --2- 172.21.15.70:0/3176440138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2cc10b0c0 0x7fd2cc118410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.581+0000 7fd2d1289640 1 -- 172.21.15.70:0/3176440138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2cc10da70 msgr2=0x7fd2cc119450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.581+0000 7fd2d1289640 1 --2- 172.21.15.70:0/3176440138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2cc10da70 0x7fd2cc119450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.581+0000 7fd2d1289640 1 -- 172.21.15.70:0/3176440138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2cc119ac0 con 0x7fd2cc108750 2024-04-01T01:47:14.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.581+0000 7fd2d1a8a640 1 --2- 172.21.15.70:0/3176440138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2cc10da70 0x7fd2cc119450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:47:14.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.581+0000 7fd2d1289640 1 --2- 172.21.15.70:0/3176440138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2cc108750 0x7fd2cc117ed0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80057c0 tx=0x7fd2c80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:14.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.581+0000 7fd2c27fc640 1 -- 172.21.15.70:0/3176440138 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8041030 con 0x7fd2cc108750 2024-04-01T01:47:14.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.581+0000 7fd2d34ed640 1 -- 172.21.15.70:0/3176440138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2cc119d50 con 0x7fd2cc108750 2024-04-01T01:47:14.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.581+0000 7fd2c27fc640 1 -- 172.21.15.70:0/3176440138 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2c802c9c0 con 0x7fd2cc108750 2024-04-01T01:47:14.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.581+0000 7fd2c27fc640 1 -- 172.21.15.70:0/3176440138 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8039040 con 0x7fd2cc108750 2024-04-01T01:47:14.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.581+0000 7fd2d34ed640 1 -- 172.21.15.70:0/3176440138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2cc11a290 con 0x7fd2cc108750 2024-04-01T01:47:14.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.583+0000 7fd2c27fc640 1 -- 172.21.15.70:0/3176440138 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fd2c8044070 con 0x7fd2cc108750 2024-04-01T01:47:14.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.584+0000 7fd2c27fc640 1 --2- 172.21.15.70:0/3176440138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd2b80a2520 0x7fd2b80a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.584+0000 7fd2d0a88640 1 --2- 172.21.15.70:0/3176440138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd2b80a2520 0x7fd2b80a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.585+0000 7fd2c27fc640 1 -- 172.21.15.70:0/3176440138 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd2c80f0090 con 0x7fd2cc108750 2024-04-01T01:47:14.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.585+0000 7fd2d61ac740 1 Processor -- start 2024-04-01T01:47:14.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.585+0000 7fd2d0a88640 1 --2- 172.21.15.70:0/3176440138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd2b80a2520 0x7fd2b80a49e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc1193e0 tx=0x7fd2bc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:14.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.585+0000 7fd2d61ac740 1 -- start start 2024-04-01T01:47:14.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.585+0000 7fd2d61ac740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 0x5653b1d028b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.585+0000 7fd2d61ac740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 0x5653b1d07980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.585+0000 7fd2d61ac740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d07ff0 0x5653b1d0a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.585+0000 7fd2d61ac740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5653b1c671b0 con 0x5653b1d02e80 2024-04-01T01:47:14.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.585+0000 7fd2d61ac740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5653b1c0fc00 con 0x5653b1d07ff0 2024-04-01T01:47:14.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.585+0000 7fd2d61ac740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5653b1c63250 con 0x5653b1d02490 2024-04-01T01:47:14.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.585+0000 7fd2d0a88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 0x5653b1d07980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.585+0000 7fd2d1289640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 0x5653b1d028b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.585+0000 7fd2d1a8a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d07ff0 0x5653b1d0a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.586+0000 7fd2d1289640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 0x5653b1d028b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57182/0 (socket says 172.21.15.70:57182) 2024-04-01T01:47:14.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.586+0000 7fd2d1a8a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d07ff0 0x5653b1d0a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47846/0 (socket says 172.21.15.70:47846) 2024-04-01T01:47:14.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.586+0000 7fd2d1289640 1 -- 172.21.15.70:0/3013007964 learned_addr learned my addr 172.21.15.70:0/3013007964 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:14.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.586+0000 7fd2d0a88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 0x5653b1d07980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44688/0 (socket says 172.21.15.70:44688) 2024-04-01T01:47:14.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.586+0000 7fd2d1289640 1 -- 172.21.15.70:0/3013007964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d07ff0 msgr2=0x5653b1d0a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:14.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.586+0000 7fd2d1289640 1 --2- 172.21.15.70:0/3013007964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d07ff0 0x5653b1d0a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.586+0000 7fd2d1289640 1 -- 172.21.15.70:0/3013007964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 msgr2=0x5653b1d07980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:14.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.586+0000 7fd2d1289640 1 --2- 172.21.15.70:0/3013007964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 0x5653b1d07980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.586+0000 7fd2d1289640 1 -- 172.21.15.70:0/3013007964 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653b1c2f3d0 con 0x5653b1d02490 2024-04-01T01:47:14.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.587+0000 7fd2d1289640 1 --2- 172.21.15.70:0/3013007964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 0x5653b1d028b0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80c4c60 tx=0x7fd2c8009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bb3d80e498ca03b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:14.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.587+0000 7fd2b77fe640 1 -- 172.21.15.70:0/3013007964 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c80bc0e0 con 0x5653b1d02490 2024-04-01T01:47:14.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.587+0000 7fd2b77fe640 1 -- 172.21.15.70:0/3013007964 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2c80bce80 con 0x5653b1d02490 2024-04-01T01:47:14.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.588+0000 7fd2b77fe640 1 -- 172.21.15.70:0/3013007964 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c80bab10 con 0x5653b1d02490 2024-04-01T01:47:14.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.588+0000 7fd2d61ac740 1 -- 172.21.15.70:0/3013007964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 msgr2=0x5653b1d028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.588+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/3013007964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 0x5653b1d028b0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80c4c60 tx=0x7fd2c8009560 comp rx=0 tx=0).stop 2024-04-01T01:47:14.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.589+0000 7fd2d61ac740 1 -- 172.21.15.70:0/3013007964 shutdown_connections 2024-04-01T01:47:14.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.589+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/3013007964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d07ff0 0x5653b1d0a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.589+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/3013007964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 0x5653b1d07980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.589+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/3013007964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 0x5653b1d028b0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.589+0000 7fd2d61ac740 1 -- 172.21.15.70:0/3013007964 >> 172.21.15.70:0/3013007964 conn(0x5653b1cfda00 msgr2=0x5653b1cfde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:14.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.589+0000 7fd2d61ac740 1 -- 172.21.15.70:0/3013007964 shutdown_connections 2024-04-01T01:47:14.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.589+0000 7fd2d61ac740 1 -- 172.21.15.70:0/3013007964 wait complete. 2024-04-01T01:47:14.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.589+0000 7fd2d61ac740 1 -- start start 2024-04-01T01:47:14.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.589+0000 7fd2d61ac740 1 Processor -- start 2024-04-01T01:47:14.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.590+0000 7fd2d61ac740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 0x5653b1d0c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.590+0000 7fd2d61ac740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d02e80 0x5653b1d0cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.590+0000 7fd2d61ac740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d07ff0 0x5653b1d13b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.590+0000 7fd2d61ac740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5653b1c63250 con 0x5653b1d07ff0 2024-04-01T01:47:14.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.590+0000 7fd2d61ac740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5653b1c05470 con 0x5653b1d02e80 2024-04-01T01:47:14.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.590+0000 7fd2d61ac740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5653b1d0ad60 con 0x5653b1d02490 2024-04-01T01:47:14.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.590+0000 7fd2d1289640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 0x5653b1d0c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.590+0000 7fd2d1289640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 0x5653b1d0c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57186/0 (socket says 172.21.15.70:57186) 2024-04-01T01:47:14.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.590+0000 7fd2d1289640 1 -- 172.21.15.70:0/3942665226 learned_addr learned my addr 172.21.15.70:0/3942665226 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:14.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.590+0000 7fd2d61ac740 10 client.0 _ll_register_callbacks cb 0x5653b1c22cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:14.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.590+0000 7fd2d1a8a640 1 --2- 172.21.15.70:0/3942665226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d07ff0 0x5653b1d13b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.590+0000 7fd2d0a88640 1 --2- 172.21.15.70:0/3942665226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d02e80 0x5653b1d0cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.590+0000 7fd2d1289640 1 -- 172.21.15.70:0/3942665226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d02e80 msgr2=0x5653b1d0cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:14.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.590+0000 7fd2d1289640 1 --2- 172.21.15.70:0/3942665226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d02e80 0x5653b1d0cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.591+0000 7fd2d1289640 1 -- 172.21.15.70:0/3942665226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d07ff0 msgr2=0x5653b1d13b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:14.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.591+0000 7fd2d1289640 1 --2- 172.21.15.70:0/3942665226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d07ff0 0x5653b1d13b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.591+0000 7fd2d1289640 1 -- 172.21.15.70:0/3942665226 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653b197b3f0 con 0x5653b1d02490 2024-04-01T01:47:14.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.591+0000 7fd2d1289640 1 --2- 172.21.15.70:0/3942665226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 0x5653b1d0c590 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80edb30 tx=0x7fd2c80ba840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ab3edf3cac8082b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:14.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.591+0000 7fd2b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:47:14.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.591+0000 7fd2b4ff9640 1 -- 172.21.15.70:0/3942665226 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c80c5190 con 0x5653b1d02490 2024-04-01T01:47:14.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.591+0000 7fd2d61ac740 1 -- 172.21.15.70:0/3942665226 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2c80bce80 con 0x5653b1d02490 2024-04-01T01:47:14.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.591+0000 7fd2b4ff9640 1 -- 172.21.15.70:0/3942665226 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd2c80c4d00 con 0x5653b1d02490 2024-04-01T01:47:14.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.592+0000 7fd2b4ff9640 1 -- 172.21.15.70:0/3942665226 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd2c80bc780 con 0x5653b1d02490 2024-04-01T01:47:14.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.592+0000 7fd2d61ac740 10 client.29260 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:14.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.592+0000 7fd2d61ac740 1 -- 172.21.15.70:0/3942665226 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5653b1a457d0 con 0x5653b1d02490 2024-04-01T01:47:14.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.592+0000 7fd2b4ff9640 1 -- 172.21.15.70:0/3942665226 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2c80bce80 con 0x5653b1d02490 2024-04-01T01:47:14.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.593+0000 7fd2b4ff9640 1 -- 172.21.15.70:0/3942665226 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd2c80b8040 con 0x5653b1d02490 2024-04-01T01:47:14.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.593+0000 7fd2d61ac740 10 client.29260 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:14.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.593+0000 7fd2d61ac740 10 client.29260 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:47:14.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.593+0000 7fd2d61ac740 20 client.29260 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:14.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.593+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/3942665226 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5653b1d17d20 0x5653b1d1a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.593+0000 7fd2d61ac740 4 client.29260 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.593+0000 7fd2d61ac740 1 -- 172.21.15.70:0/3942665226 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653b1c746f0 con 0x5653b1d17d20 2024-04-01T01:47:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.593+0000 7fd2d0a88640 1 --2- 172.21.15.70:0/3942665226 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5653b1d17d20 0x5653b1d1a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.593+0000 7fd2d0a88640 1 --2- 172.21.15.70:0/3942665226 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5653b1d17d20 0x5653b1d1a190 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f39e775e02e76733 server_cookie=7da1deb9a5c558e1 in_seq=0 out_seq=0 2024-04-01T01:47:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.593+0000 7fd2b4ff9640 10 client.29260 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:47:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.595+0000 7fd2b4ff9640 1 -- 172.21.15.70:0/3942665226 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5653b1c746f0 con 0x5653b1d17d20 2024-04-01T01:47:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.595+0000 7fd2b4ff9640 10 client.29260 handle_command_reply: tid=0 2024-04-01T01:47:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.595+0000 7fd2d61ac740 1 client.29260 shutdown 2024-04-01T01:47:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.595+0000 7fd2d61ac740 1 -- 172.21.15.70:0/3942665226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 msgr2=0x5653b1d0c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.595+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/3942665226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 0x5653b1d0c590 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80edb30 tx=0x7fd2c80ba840 comp rx=0 tx=0).stop 2024-04-01T01:47:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.595+0000 7fd2d61ac740 1 -- 172.21.15.70:0/3942665226 shutdown_connections 2024-04-01T01:47:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.595+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/3942665226 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5653b1d17d20 0x5653b1d1a190 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.595+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/3942665226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d07ff0 0x5653b1d13b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.595+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/3942665226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d02e80 0x5653b1d0cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.595+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/3942665226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 0x5653b1d0c590 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.595+0000 7fd2d61ac740 1 -- 172.21.15.70:0/3942665226 >> 172.21.15.70:0/3942665226 conn(0x5653b1cfda00 msgr2=0x5653b1cfcd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:14.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.595+0000 7fd2b4ff9640 0 client.29260 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:47:14.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.595+0000 7fd2d61ac740 1 -- 172.21.15.70:0/3942665226 shutdown_connections 2024-04-01T01:47:14.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.595+0000 7fd2d61ac740 1 -- 172.21.15.70:0/3942665226 wait complete. 2024-04-01T01:47:14.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.595+0000 7fd2d61ac740 20 client.29260 trim_cache size 0 max 16384 2024-04-01T01:47:14.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.612+0000 7fd2d61ac740 1 Processor -- start 2024-04-01T01:47:14.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.612+0000 7fd2d61ac740 1 -- start start 2024-04-01T01:47:14.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.612+0000 7fd2d61ac740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 0x5653b1d0c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.612+0000 7fd2d61ac740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 0x5653b1d13aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.612+0000 7fd2d61ac740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d07ff0 0x5653b1d0c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.612+0000 7fd2d61ac740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc1c1a30 con 0x5653b1d02e80 2024-04-01T01:47:14.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.612+0000 7fd2d61ac740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc1c3570 con 0x5653b1d07ff0 2024-04-01T01:47:14.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.612+0000 7fd2d61ac740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc1c2100 con 0x5653b1d02490 2024-04-01T01:47:14.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.612+0000 7fd2d1289640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 0x5653b1d0c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.612+0000 7fd2d0a88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 0x5653b1d13aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.612+0000 7fd2d0a88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 0x5653b1d13aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44706/0 (socket says 172.21.15.70:44706) 2024-04-01T01:47:14.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.612+0000 7fd2d0a88640 1 -- 172.21.15.70:0/2431096510 learned_addr learned my addr 172.21.15.70:0/2431096510 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:14.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.612+0000 7fd2d1a8a640 1 --2- 172.21.15.70:0/2431096510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d07ff0 0x5653b1d0c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.613+0000 7fd2d1a8a640 1 -- 172.21.15.70:0/2431096510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 msgr2=0x5653b1d0c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:14.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.613+0000 7fd2d1a8a640 1 --2- 172.21.15.70:0/2431096510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 0x5653b1d0c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.613+0000 7fd2d1a8a640 1 -- 172.21.15.70:0/2431096510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 msgr2=0x5653b1d13aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:14.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.613+0000 7fd2d1a8a640 1 --2- 172.21.15.70:0/2431096510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 0x5653b1d13aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.613+0000 7fd2d1a8a640 1 -- 172.21.15.70:0/2431096510 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653b1a457d0 con 0x5653b1d07ff0 2024-04-01T01:47:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.613+0000 7fd2d1a8a640 1 --2- 172.21.15.70:0/2431096510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d07ff0 0x5653b1d0c6e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc06b440 tx=0x7fd2c4013d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56fb310baeb6b14a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.613+0000 7fd2b6ffd640 1 -- 172.21.15.70:0/2431096510 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4004b20 con 0x5653b1d07ff0 2024-04-01T01:47:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.613+0000 7fd2b6ffd640 1 -- 172.21.15.70:0/2431096510 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2c4011e00 con 0x5653b1d07ff0 2024-04-01T01:47:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.613+0000 7fd2b6ffd640 1 -- 172.21.15.70:0/2431096510 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4005840 con 0x5653b1d07ff0 2024-04-01T01:47:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.614+0000 7fd2d61ac740 1 -- 172.21.15.70:0/2431096510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d07ff0 msgr2=0x5653b1d0c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.614+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/2431096510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d07ff0 0x5653b1d0c6e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc06b440 tx=0x7fd2c4013d00 comp rx=0 tx=0).stop 2024-04-01T01:47:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.614+0000 7fd2d61ac740 1 -- 172.21.15.70:0/2431096510 shutdown_connections 2024-04-01T01:47:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.615+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/2431096510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d07ff0 0x5653b1d0c6e0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.615+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/2431096510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 0x5653b1d13aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.615+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/2431096510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d02490 0x5653b1d0c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.615+0000 7fd2d61ac740 1 -- 172.21.15.70:0/2431096510 >> 172.21.15.70:0/2431096510 conn(0x5653b1cfda00 msgr2=0x5653b1cfdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.615+0000 7fd2d61ac740 1 -- 172.21.15.70:0/2431096510 shutdown_connections 2024-04-01T01:47:14.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.615+0000 7fd2d61ac740 1 -- 172.21.15.70:0/2431096510 wait complete. 2024-04-01T01:47:14.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.615+0000 7fd2d61ac740 1 -- start start 2024-04-01T01:47:14.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.615+0000 7fd2d61ac740 1 Processor -- start 2024-04-01T01:47:14.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d61ac740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d02490 0x5653b1d10dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d61ac740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 0x5653b1d11300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d61ac740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d07ff0 0x5653b1d122a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d61ac740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc1c2ea0 con 0x5653b1d02e80 2024-04-01T01:47:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d61ac740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc1c27d0 con 0x5653b1d02490 2024-04-01T01:47:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d61ac740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd2cc1bfa10 con 0x5653b1d07ff0 2024-04-01T01:47:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d61ac740 10 client.0 _ll_register_callbacks cb 0x7fd2c8005630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d1289640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d02490 0x5653b1d10dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d0a88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 0x5653b1d11300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d1289640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d02490 0x5653b1d10dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47884/0 (socket says 172.21.15.70:47884) 2024-04-01T01:47:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d1289640 1 -- 172.21.15.70:0/2560098771 learned_addr learned my addr 172.21.15.70:0/2560098771 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d1a8a640 1 --2- 172.21.15.70:0/2560098771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d07ff0 0x5653b1d122a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d1289640 1 -- 172.21.15.70:0/2560098771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d07ff0 msgr2=0x5653b1d122a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:14.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d1289640 1 --2- 172.21.15.70:0/2560098771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d07ff0 0x5653b1d122a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d1289640 1 -- 172.21.15.70:0/2560098771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 msgr2=0x5653b1d11300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d1289640 1 --2- 172.21.15.70:0/2560098771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 0x5653b1d11300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d1289640 1 -- 172.21.15.70:0/2560098771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653b1d12890 con 0x5653b1d02490 2024-04-01T01:47:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.616+0000 7fd2d0a88640 1 --2- 172.21.15.70:0/2560098771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 0x5653b1d11300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:47:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2d1289640 1 --2- 172.21.15.70:0/2560098771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d02490 0x5653b1d10dc0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80056e0 tx=0x7fd2c80b83f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dade6371112f02a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2b4ff9640 1 -- 172.21.15.70:0/2560098771 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c80ba8b0 con 0x5653b1d02490 2024-04-01T01:47:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2d61ac740 1 -- 172.21.15.70:0/2560098771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2c4011e00 con 0x5653b1d02490 2024-04-01T01:47:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2b4ff9640 1 -- 172.21.15.70:0/2560098771 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd2c80c4f10 con 0x5653b1d02490 2024-04-01T01:47:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2b4ff9640 1 -- 172.21.15.70:0/2560098771 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd2c80bce80 con 0x5653b1d02490 2024-04-01T01:47:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2d61ac740 10 client.29501 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2d61ac740 1 -- 172.21.15.70:0/2560098771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5653b1d12890 con 0x5653b1d02490 2024-04-01T01:47:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2b4ff9640 1 -- 172.21.15.70:0/2560098771 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2c4011e00 con 0x5653b1d02490 2024-04-01T01:47:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2b4ff9640 1 -- 172.21.15.70:0/2560098771 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd2c80b8040 con 0x5653b1d02490 2024-04-01T01:47:14.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2d61ac740 10 client.29501 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2d61ac740 10 client.29501 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:47:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2d61ac740 20 client.29501 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/2560098771 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5653b1d7e200 0x5653b1d805f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2d61ac740 4 client.29501 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:47:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2d61ac740 1 -- 172.21.15.70:0/2560098771 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5653b1c6ea40 con 0x5653b1d7e200 2024-04-01T01:47:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.617+0000 7fd2d0a88640 1 --2- 172.21.15.70:0/2560098771 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5653b1d7e200 0x5653b1d805f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.618+0000 7fd2b4ff9640 10 client.29501 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:47:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.618+0000 7fd2d0a88640 1 --2- 172.21.15.70:0/2560098771 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5653b1d7e200 0x5653b1d805f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c0545486bfa49826 server_cookie=b96f259c7a62c023 in_seq=0 out_seq=0 2024-04-01T01:47:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.618+0000 7fd2b4ff9640 1 -- 172.21.15.70:0/2560098771 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5653b1c6ea40 con 0x5653b1d7e200 2024-04-01T01:47:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.618+0000 7fd2b4ff9640 10 client.29501 handle_command_reply: tid=0 2024-04-01T01:47:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.618+0000 7fd2d61ac740 1 client.29501 shutdown 2024-04-01T01:47:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.618+0000 7fd2d61ac740 1 -- 172.21.15.70:0/2560098771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d02490 msgr2=0x5653b1d10dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.618+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/2560098771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d02490 0x5653b1d10dc0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80056e0 tx=0x7fd2c80b83f0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.618+0000 7fd2d61ac740 1 -- 172.21.15.70:0/2560098771 shutdown_connections 2024-04-01T01:47:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.618+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/2560098771 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5653b1d7e200 0x5653b1d805f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.618+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/2560098771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653b1d07ff0 0x5653b1d122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.618+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/2560098771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5653b1d02e80 0x5653b1d11300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.618+0000 7fd2d61ac740 1 --2- 172.21.15.70:0/2560098771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653b1d02490 0x5653b1d10dc0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.618+0000 7fd2d61ac740 1 -- 172.21.15.70:0/2560098771 >> 172.21.15.70:0/2560098771 conn(0x5653b1cfda00 msgr2=0x5653b1d0c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.618+0000 7fd2b4ff9640 0 client.29501 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:47:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.618+0000 7fd2d61ac740 1 -- 172.21.15.70:0/2560098771 shutdown_connections 2024-04-01T01:47:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.618+0000 7fd2d61ac740 1 -- 172.21.15.70:0/2560098771 wait complete. 2024-04-01T01:47:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.619+0000 7fd2d61ac740 20 client.29501 trim_cache size 0 max 16384 2024-04-01T01:47:14.623 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:47:14.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.619+0000 7fd2d34ed640 1 -- 172.21.15.70:0/3176440138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd2b80a2520 msgr2=0x7fd2b80a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.619+0000 7fd2d34ed640 1 --2- 172.21.15.70:0/3176440138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd2b80a2520 0x7fd2b80a49e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc1193e0 tx=0x7fd2bc009260 comp rx=0 tx=0).stop 2024-04-01T01:47:14.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.619+0000 7fd2d34ed640 1 -- 172.21.15.70:0/3176440138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2cc108750 msgr2=0x7fd2cc117ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.619+0000 7fd2d34ed640 1 --2- 172.21.15.70:0/3176440138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2cc108750 0x7fd2cc117ed0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80057c0 tx=0x7fd2c80042f0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.619+0000 7fd2d34ed640 1 -- 172.21.15.70:0/3176440138 shutdown_connections 2024-04-01T01:47:14.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.619+0000 7fd2d34ed640 1 --2- 172.21.15.70:0/3176440138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2cc10da70 0x7fd2cc119450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.619+0000 7fd2d34ed640 1 --2- 172.21.15.70:0/3176440138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd2b80a2520 0x7fd2b80a49e0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.619+0000 7fd2d34ed640 1 --2- 172.21.15.70:0/3176440138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2cc10b0c0 0x7fd2cc118410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.619+0000 7fd2d34ed640 1 --2- 172.21.15.70:0/3176440138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2cc108750 0x7fd2cc117ed0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.619+0000 7fd2d34ed640 1 -- 172.21.15.70:0/3176440138 >> 172.21.15.70:0/3176440138 conn(0x7fd2cc06e5b0 msgr2=0x7fd2cc109030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:14.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.619+0000 7fd2d34ed640 1 -- 172.21.15.70:0/3176440138 shutdown_connections 2024-04-01T01:47:14.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.619+0000 7fd2d34ed640 1 -- 172.21.15.70:0/3176440138 wait complete. 2024-04-01T01:47:14.632 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:47:14.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.800+0000 7fc1e128f640 1 -- 172.21.15.70:0/3653737367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1dc0fffc0 msgr2=0x7fc1dc1023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.800+0000 7fc1e128f640 1 --2- 172.21.15.70:0/3653737367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1dc0fffc0 0x7fc1dc1023f0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009900 tx=0x7fc1cc02bc80 comp rx=0 tx=0).stop 2024-04-01T01:47:14.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.801+0000 7fc1e128f640 1 -- 172.21.15.70:0/3653737367 shutdown_connections 2024-04-01T01:47:14.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.801+0000 7fc1e128f640 1 --2- 172.21.15.70:0/3653737367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1dc1052e0 0x7fc1dc105760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.801+0000 7fc1e128f640 1 --2- 172.21.15.70:0/3653737367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1dc102930 0x7fc1dc104da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.801+0000 7fc1e128f640 1 --2- 172.21.15.70:0/3653737367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1dc0fffc0 0x7fc1dc1023f0 secure :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009900 tx=0x7fc1cc02bc80 comp rx=0 tx=0).stop 2024-04-01T01:47:14.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.801+0000 7fc1e128f640 1 -- 172.21.15.70:0/3653737367 >> 172.21.15.70:0/3653737367 conn(0x7fc1dc0fbbe0 msgr2=0x7fc1dc0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:14.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.802+0000 7fc1e128f640 1 -- 172.21.15.70:0/3653737367 shutdown_connections 2024-04-01T01:47:14.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.802+0000 7fc1e128f640 1 -- 172.21.15.70:0/3653737367 wait complete. 2024-04-01T01:47:14.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.803+0000 7fc1e128f640 1 Processor -- start 2024-04-01T01:47:14.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.803+0000 7fc1e128f640 1 -- start start 2024-04-01T01:47:14.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.804+0000 7fc1e128f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1dc102930 0x7fc1dc19e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.804+0000 7fc1e128f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1dc1052e0 0x7fc1dc19e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.804+0000 7fc1e128f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1dc1a3970 0x7fc1dc1a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.804+0000 7fc1e128f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc1139d0 con 0x7fc1dc1052e0 2024-04-01T01:47:14.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.805+0000 7fc1db7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1dc1a3970 0x7fc1dc1a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.805+0000 7fc1db7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1dc1a3970 0x7fc1dc1a5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47896/0 (socket says 172.21.15.70:47896) 2024-04-01T01:47:14.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.805+0000 7fc1db7fe640 1 -- 172.21.15.70:0/1781537097 learned_addr learned my addr 172.21.15.70:0/1781537097 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:14.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.805+0000 7fc1e128f640 1 -- 172.21.15.70:0/1781537097 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc113850 con 0x7fc1dc1a3970 2024-04-01T01:47:14.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.805+0000 7fc1e128f640 1 -- 172.21.15.70:0/1781537097 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc1dc113b50 con 0x7fc1dc102930 2024-04-01T01:47:14.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.805+0000 7fc1da7fc640 1 --2- 172.21.15.70:0/1781537097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1dc1052e0 0x7fc1dc19e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.805+0000 7fc1db7fe640 1 -- 172.21.15.70:0/1781537097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1dc102930 msgr2=0x7fc1dc19e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.805+0000 7fc1db7fe640 1 --2- 172.21.15.70:0/1781537097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1dc102930 0x7fc1dc19e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.805+0000 7fc1db7fe640 1 -- 172.21.15.70:0/1781537097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1dc1052e0 msgr2=0x7fc1dc19e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.805+0000 7fc1db7fe640 1 --2- 172.21.15.70:0/1781537097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1dc1052e0 0x7fc1dc19e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.805+0000 7fc1db7fe640 1 -- 172.21.15.70:0/1781537097 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1dc1a63d0 con 0x7fc1dc1a3970 2024-04-01T01:47:14.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.805+0000 7fc1da7fc640 1 --2- 172.21.15.70:0/1781537097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1dc1052e0 0x7fc1dc19e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:47:14.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.805+0000 7fc1db7fe640 1 --2- 172.21.15.70:0/1781537097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1dc1a3970 0x7fc1dc1a5d60 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc1d400b990 tx=0x7fc1d400be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:14.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.807+0000 7fc1c3fff640 1 -- 172.21.15.70:0/1781537097 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4019070 con 0x7fc1dc1a3970 2024-04-01T01:47:14.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.807+0000 7fc1c3fff640 1 -- 172.21.15.70:0/1781537097 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc1d400cba0 con 0x7fc1dc1a3970 2024-04-01T01:47:14.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.807+0000 7fc1c3fff640 1 -- 172.21.15.70:0/1781537097 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d4007590 con 0x7fc1dc1a3970 2024-04-01T01:47:14.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.807+0000 7fc1e128f640 1 -- 172.21.15.70:0/1781537097 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1dc1a66c0 con 0x7fc1dc1a3970 2024-04-01T01:47:14.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.807+0000 7fc1e128f640 1 -- 172.21.15.70:0/1781537097 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1dc1a6c30 con 0x7fc1dc1a3970 2024-04-01T01:47:14.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.813+0000 7fc1c3fff640 1 -- 172.21.15.70:0/1781537097 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fc1d401c070 con 0x7fc1dc1a3970 2024-04-01T01:47:14.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.813+0000 7fc1e3e7c740 1 Processor -- start 2024-04-01T01:47:14.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.813+0000 7fc1e3e7c740 1 -- start start 2024-04-01T01:47:14.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.814+0000 7fc1e3e7c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6032490 0x5566d60328b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.814+0000 7fc1e3e7c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d6037980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.814+0000 7fc1e3e7c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6037ff0 0x5566d603a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.814+0000 7fc1e3e7c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5566d5f971b0 con 0x5566d6032e80 2024-04-01T01:47:14.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.814+0000 7fc1e3e7c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5566d5f3fc00 con 0x5566d6032490 2024-04-01T01:47:14.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.814+0000 7fc1e3e7c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5566d5f93250 con 0x5566d6037ff0 2024-04-01T01:47:14.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.818+0000 7fc1daffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d6037980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.817+0000 7fc1da7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6032490 0x5566d60328b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.820+0000 7fc1daffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d6037980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44736/0 (socket says 172.21.15.70:44736) 2024-04-01T01:47:14.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.820+0000 7fc1daffd640 1 -- 172.21.15.70:0/3013971654 learned_addr learned my addr 172.21.15.70:0/3013971654 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:14.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.820+0000 7fc1da7fc640 1 --2- 172.21.15.70:0/3013971654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6037ff0 0x5566d603a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.820+0000 7fc1daffd640 1 -- 172.21.15.70:0/3013971654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6037ff0 msgr2=0x5566d603a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:14.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.820+0000 7fc1daffd640 1 --2- 172.21.15.70:0/3013971654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6037ff0 0x5566d603a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.820+0000 7fc1daffd640 1 -- 172.21.15.70:0/3013971654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6032490 msgr2=0x5566d60328b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:14.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.820+0000 7fc1daffd640 1 --2- 172.21.15.70:0/3013971654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6032490 0x5566d60328b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.820+0000 7fc1daffd640 1 -- 172.21.15.70:0/3013971654 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566d5f5f3d0 con 0x5566d6032e80 2024-04-01T01:47:14.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.821+0000 7fc1c3fff640 1 --2- 172.21.15.70:0/1781537097 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc1c40a2620 0x7fc1c40a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.821+0000 7fc1c3fff640 1 -- 172.21.15.70:0/1781537097 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc1d4014440 con 0x7fc1dc1a3970 2024-04-01T01:47:14.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.822+0000 7fc1daffd640 1 --2- 172.21.15.70:0/3013971654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d6037980 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009850 tx=0x7fc1cc009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf39f0468cd1db server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:14.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.822+0000 7fc1c17fa640 1 -- 172.21.15.70:0/3013971654 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc042020 con 0x5566d6032e80 2024-04-01T01:47:14.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.823+0000 7fc1c17fa640 1 -- 172.21.15.70:0/3013971654 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc1cc02cbb0 con 0x5566d6032e80 2024-04-01T01:47:14.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.823+0000 7fc1c17fa640 1 -- 172.21.15.70:0/3013971654 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc003d40 con 0x5566d6032e80 2024-04-01T01:47:14.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.824+0000 7fc1da7fc640 1 --2- 172.21.15.70:0/1781537097 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc1c40a2620 0x7fc1c40a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.824+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3013971654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 msgr2=0x5566d6037980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.824+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/3013971654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d6037980 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009850 tx=0x7fc1cc009610 comp rx=0 tx=0).stop 2024-04-01T01:47:14.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.824+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3013971654 shutdown_connections 2024-04-01T01:47:14.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.824+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/3013971654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6037ff0 0x5566d603a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.824+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/3013971654 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d6037980 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.824+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/3013971654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6032490 0x5566d60328b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.824+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3013971654 >> 172.21.15.70:0/3013971654 conn(0x5566d602da00 msgr2=0x5566d602de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:14.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.824+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3013971654 shutdown_connections 2024-04-01T01:47:14.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.824+0000 7fc1da7fc640 1 --2- 172.21.15.70:0/1781537097 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc1c40a2620 0x7fc1c40a4ae0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x5566d60336e0 tx=0x7fc1d000a1f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:14.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.825+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3013971654 wait complete. 2024-04-01T01:47:14.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.826+0000 7fc1e3e7c740 1 -- start start 2024-04-01T01:47:14.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.826+0000 7fc1e3e7c740 1 Processor -- start 2024-04-01T01:47:14.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.827+0000 7fc1e3e7c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6032490 0x5566d604d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.827+0000 7fc1e3e7c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d604df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.827+0000 7fc1e3e7c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6037ff0 0x5566d6054f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.827+0000 7fc1e3e7c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5566d604bf50 con 0x5566d6032e80 2024-04-01T01:47:14.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.827+0000 7fc1e3e7c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5566d5f35470 con 0x5566d6032490 2024-04-01T01:47:14.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.827+0000 7fc1e3e7c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5566d604c0d0 con 0x5566d6037ff0 2024-04-01T01:47:14.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.827+0000 7fc1e3e7c740 10 client.0 _ll_register_callbacks cb 0x5566d5f52cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:14.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.828+0000 7fc1daffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6032490 0x5566d604d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.828+0000 7fc1da7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d604df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.828+0000 7fc1da7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d604df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44752/0 (socket says 172.21.15.70:44752) 2024-04-01T01:47:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.828+0000 7fc1daffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6032490 0x5566d604d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47912/0 (socket says 172.21.15.70:47912) 2024-04-01T01:47:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.828+0000 7fc1daffd640 1 -- 172.21.15.70:0/421243011 learned_addr learned my addr 172.21.15.70:0/421243011 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.829+0000 7fc1da7fc640 1 -- 172.21.15.70:0/421243011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6037ff0 msgr2=0x5566d6054f50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:47:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.829+0000 7fc1da7fc640 1 --2- 172.21.15.70:0/421243011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6037ff0 0x5566d6054f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.829+0000 7fc1da7fc640 1 -- 172.21.15.70:0/421243011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6032490 msgr2=0x5566d604d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.829+0000 7fc1da7fc640 1 --2- 172.21.15.70:0/421243011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6032490 0x5566d604d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.829+0000 7fc1da7fc640 1 -- 172.21.15.70:0/421243011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566d5cab3f0 con 0x5566d6032e80 2024-04-01T01:47:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.829+0000 7fc1da7fc640 1 --2- 172.21.15.70:0/421243011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d604df10 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc19f8e0 tx=0x7fc1d00046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3cdf176dc454a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.829+0000 7fc1daffd640 1 --2- 172.21.15.70:0/421243011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6032490 0x5566d604d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:47:14.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.829+0000 7fc1beff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:47:14.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.834+0000 7fc1beff5640 1 -- 172.21.15.70:0/421243011 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d001b050 con 0x5566d6032e80 2024-04-01T01:47:14.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.834+0000 7fc1beff5640 1 -- 172.21.15.70:0/421243011 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc1d0004020 con 0x5566d6032e80 2024-04-01T01:47:14.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.838+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/421243011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1cc02cbb0 con 0x5566d6032e80 2024-04-01T01:47:14.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.839+0000 7fc1beff5640 1 -- 172.21.15.70:0/421243011 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc1d000f040 con 0x5566d6032e80 2024-04-01T01:47:14.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.842+0000 7fc1e3e7c740 10 client.19434 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:14.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.842+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/421243011 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5566d5d757d0 con 0x5566d6032e80 2024-04-01T01:47:14.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.842+0000 7fc1beff5640 1 -- 172.21.15.70:0/421243011 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc1cc02cbb0 con 0x5566d6032e80 2024-04-01T01:47:14.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.843+0000 7fc1beff5640 1 -- 172.21.15.70:0/421243011 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc1d00038d0 con 0x5566d6032e80 2024-04-01T01:47:14.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.843+0000 7fc1e3e7c740 10 client.19434 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:14.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.843+0000 7fc1e3e7c740 10 client.19434 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:47:14.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.843+0000 7fc1e3e7c740 20 client.19434 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:14.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.843+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/421243011 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5566d60595c0 0x5566d605ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.843+0000 7fc1e3e7c740 4 client.19434 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:14.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.843+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/421243011 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566d5fa46f0 con 0x5566d60595c0 2024-04-01T01:47:14.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.845+0000 7fc1daffd640 1 --2- 172.21.15.70:0/421243011 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5566d60595c0 0x5566d605ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.851+0000 7fc1daffd640 1 --2- 172.21.15.70:0/421243011 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5566d60595c0 0x5566d605ba00 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=30c70a93f3f50a1 server_cookie=43f8e6b58402dd49 in_seq=0 out_seq=0 2024-04-01T01:47:14.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.851+0000 7fc1beff5640 10 client.19434 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:47:14.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.852+0000 7fc1beff5640 1 -- 172.21.15.70:0/421243011 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5566d5fa46f0 con 0x5566d60595c0 2024-04-01T01:47:14.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.852+0000 7fc1beff5640 10 client.19434 handle_command_reply: tid=0 2024-04-01T01:47:14.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.852+0000 7fc1e3e7c740 1 client.19434 shutdown 2024-04-01T01:47:14.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.852+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/421243011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 msgr2=0x5566d604df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.852+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/421243011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d604df10 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc19f8e0 tx=0x7fc1d00046e0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.852+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/421243011 shutdown_connections 2024-04-01T01:47:14.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.852+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/421243011 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5566d60595c0 0x5566d605ba00 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.852+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/421243011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6037ff0 0x5566d6054f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.852+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/421243011 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d604df10 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.852+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/421243011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6032490 0x5566d604d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.852+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/421243011 >> 172.21.15.70:0/421243011 conn(0x5566d602da00 msgr2=0x5566d602cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:14.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.852+0000 7fc1beff5640 0 client.19434 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:47:14.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.853+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/421243011 shutdown_connections 2024-04-01T01:47:14.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.853+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/421243011 wait complete. 2024-04-01T01:47:14.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.853+0000 7fc1e3e7c740 20 client.19434 trim_cache size 0 max 16384 2024-04-01T01:47:14.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.873+0000 7fc1e3e7c740 1 Processor -- start 2024-04-01T01:47:14.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.873+0000 7fc1e3e7c740 1 -- start start 2024-04-01T01:47:14.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.873+0000 7fc1e3e7c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6032490 0x5566d604cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.874+0000 7fc1e3e7c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d6054f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.874+0000 7fc1e3e7c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6037ff0 0x5566d604d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.874+0000 7fc1e3e7c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c800fda0 con 0x5566d6032e80 2024-04-01T01:47:14.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.874+0000 7fc1daffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6032490 0x5566d604cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.874+0000 7fc1da7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d6054f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.874+0000 7fc1daffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6032490 0x5566d604cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57238/0 (socket says 172.21.15.70:57238) 2024-04-01T01:47:14.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.874+0000 7fc1daffd640 1 -- 172.21.15.70:0/3302508338 learned_addr learned my addr 172.21.15.70:0/3302508338 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:14.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.874+0000 7fc1db7fe640 1 --2- 172.21.15.70:0/3302508338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6037ff0 0x5566d604d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.874+0000 7fc1e3e7c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c80118e0 con 0x5566d6037ff0 2024-04-01T01:47:14.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.874+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3302508338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c8010470 con 0x5566d6032490 2024-04-01T01:47:14.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.874+0000 7fc1db7fe640 1 -- 172.21.15.70:0/3302508338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6032490 msgr2=0x5566d604cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:14.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.874+0000 7fc1db7fe640 1 --2- 172.21.15.70:0/3302508338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6032490 0x5566d604cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.875+0000 7fc1db7fe640 1 -- 172.21.15.70:0/3302508338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 msgr2=0x5566d6054f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:14.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.875+0000 7fc1db7fe640 1 --2- 172.21.15.70:0/3302508338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d6054f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.875+0000 7fc1db7fe640 1 -- 172.21.15.70:0/3302508338 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566d6033260 con 0x5566d6037ff0 2024-04-01T01:47:14.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.875+0000 7fc1daffd640 1 --2- 172.21.15.70:0/3302508338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6032490 0x5566d604cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:47:14.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.875+0000 7fc1da7fc640 1 --2- 172.21.15.70:0/3302508338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d6054f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:47:14.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.875+0000 7fc1db7fe640 1 --2- 172.21.15.70:0/3302508338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6037ff0 0x5566d604d220 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4098100 tx=0x7fc1d4004a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa4f6b025eb9f21c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:14.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.875+0000 7fc1c0ff9640 1 -- 172.21.15.70:0/3302508338 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d40b3070 con 0x5566d6037ff0 2024-04-01T01:47:14.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.875+0000 7fc1c0ff9640 1 -- 172.21.15.70:0/3302508338 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc1d409b730 con 0x5566d6037ff0 2024-04-01T01:47:14.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.875+0000 7fc1c0ff9640 1 -- 172.21.15.70:0/3302508338 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d40b4260 con 0x5566d6037ff0 2024-04-01T01:47:14.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.876+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3302508338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6037ff0 msgr2=0x5566d604d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.876+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/3302508338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6037ff0 0x5566d604d220 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4098100 tx=0x7fc1d4004a60 comp rx=0 tx=0).stop 2024-04-01T01:47:14.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.876+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3302508338 shutdown_connections 2024-04-01T01:47:14.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.876+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/3302508338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6037ff0 0x5566d604d220 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.876+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/3302508338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032e80 0x5566d6054f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.876+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/3302508338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6032490 0x5566d604cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.877+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3302508338 >> 172.21.15.70:0/3302508338 conn(0x5566d602da00 msgr2=0x5566d6034410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:14.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.877+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3302508338 shutdown_connections 2024-04-01T01:47:14.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.877+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3302508338 wait complete. 2024-04-01T01:47:14.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.877+0000 7fc1e3e7c740 1 -- start start 2024-04-01T01:47:14.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.877+0000 7fc1e3e7c740 1 Processor -- start 2024-04-01T01:47:14.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.878+0000 7fc1e3e7c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032490 0x5566d60527d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.878+0000 7fc1e3e7c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6032e80 0x5566d6052d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.878+0000 7fc1e3e7c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6037ff0 0x5566d6053d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.878+0000 7fc1daffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032490 0x5566d60527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.878+0000 7fc1daffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032490 0x5566d60527d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44774/0 (socket says 172.21.15.70:44774) 2024-04-01T01:47:14.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.878+0000 7fc1daffd640 1 -- 172.21.15.70:0/3214861284 learned_addr learned my addr 172.21.15.70:0/3214861284 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:14.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.878+0000 7fc1da7fc640 1 --2- 172.21.15.70:0/3214861284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6032e80 0x5566d6052d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.878+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3214861284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c8011210 con 0x5566d6032490 2024-04-01T01:47:14.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.878+0000 7fc1db7fe640 1 --2- 172.21.15.70:0/3214861284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6037ff0 0x5566d6053d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.878+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3214861284 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c8010b40 con 0x5566d6037ff0 2024-04-01T01:47:14.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.878+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3214861284 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c800dd80 con 0x5566d6032e80 2024-04-01T01:47:14.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.879+0000 7fc1e3e7c740 10 client.0 _ll_register_callbacks cb 0x7fc1d000fd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:14.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.879+0000 7fc1daffd640 1 -- 172.21.15.70:0/3214861284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6032e80 msgr2=0x5566d6052d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:14.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.879+0000 7fc1daffd640 1 --2- 172.21.15.70:0/3214861284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6032e80 0x5566d6052d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.879+0000 7fc1daffd640 1 -- 172.21.15.70:0/3214861284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6037ff0 msgr2=0x5566d6053d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:14.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.879+0000 7fc1daffd640 1 --2- 172.21.15.70:0/3214861284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6037ff0 0x5566d6053d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.879+0000 7fc1daffd640 1 -- 172.21.15.70:0/3214861284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566d6054310 con 0x5566d6032490 2024-04-01T01:47:14.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.879+0000 7fc1daffd640 1 --2- 172.21.15.70:0/3214861284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032490 0x5566d60527d0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc02e010 tx=0x7fc1cc005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5383d5881a61ad93 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:14.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.879+0000 7fc1beff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:47:14.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.879+0000 7fc1beff5640 1 -- 172.21.15.70:0/3214861284 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1cc042040 con 0x5566d6032490 2024-04-01T01:47:14.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.879+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3214861284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1d409b730 con 0x5566d6032490 2024-04-01T01:47:14.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.879+0000 7fc1beff5640 1 -- 172.21.15.70:0/3214861284 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc1cc052ab0 con 0x5566d6032490 2024-04-01T01:47:14.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.879+0000 7fc1beff5640 1 -- 172.21.15.70:0/3214861284 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc1cc052cf0 con 0x5566d6032490 2024-04-01T01:47:14.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.879+0000 7fc1e3e7c740 10 client.19440 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:14.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.879+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3214861284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5566d60bfb30 con 0x5566d6032490 2024-04-01T01:47:14.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.880+0000 7fc1beff5640 1 -- 172.21.15.70:0/3214861284 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc1d409b730 con 0x5566d6032490 2024-04-01T01:47:14.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.880+0000 7fc1beff5640 1 -- 172.21.15.70:0/3214861284 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc1cc052020 con 0x5566d6032490 2024-04-01T01:47:14.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.880+0000 7fc1e3e7c740 10 client.19440 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:14.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.880+0000 7fc1e3e7c740 10 client.19440 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:47:14.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.880+0000 7fc1e3e7c740 20 client.19440 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:14.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.880+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/3214861284 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5566d60c01e0 0x5566d60c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:14.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.880+0000 7fc1e3e7c740 4 client.19440 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:47:14.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.880+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3214861284 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5566d5f9ea40 con 0x5566d60c01e0 2024-04-01T01:47:14.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.880+0000 7fc1da7fc640 1 --2- 172.21.15.70:0/3214861284 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5566d60c01e0 0x5566d60c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:14.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1beff5640 10 client.19440 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:47:14.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1da7fc640 1 --2- 172.21.15.70:0/3214861284 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5566d60c01e0 0x5566d60c25d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a9d0dc29770b9be0 server_cookie=bca876f63d955576 in_seq=0 out_seq=0 2024-04-01T01:47:14.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1beff5640 1 -- 172.21.15.70:0/3214861284 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5566d5f9ea40 con 0x5566d60c01e0 2024-04-01T01:47:14.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1beff5640 10 client.19440 handle_command_reply: tid=0 2024-04-01T01:47:14.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1e3e7c740 1 client.19440 shutdown 2024-04-01T01:47:14.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3214861284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032490 msgr2=0x5566d60527d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/3214861284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032490 0x5566d60527d0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc02e010 tx=0x7fc1cc005bf0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3214861284 shutdown_connections 2024-04-01T01:47:14.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/3214861284 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5566d60c01e0 0x5566d60c25d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1beff5640 0 client.19440 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:47:14.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/3214861284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5566d6037ff0 0x5566d6053d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/3214861284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d6032e80 0x5566d6052d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1e3e7c740 1 --2- 172.21.15.70:0/3214861284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5566d6032490 0x5566d60527d0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3214861284 >> 172.21.15.70:0/3214861284 conn(0x5566d602da00 msgr2=0x5566d604cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:14.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3214861284 shutdown_connections 2024-04-01T01:47:14.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1e3e7c740 1 -- 172.21.15.70:0/3214861284 wait complete. 2024-04-01T01:47:14.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.881+0000 7fc1e3e7c740 20 client.19440 trim_cache size 0 max 16384 2024-04-01T01:47:14.885 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:47:14.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.882+0000 7fc1e128f640 1 -- 172.21.15.70:0/1781537097 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc1c40a2620 msgr2=0x7fc1c40a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.882+0000 7fc1e128f640 1 --2- 172.21.15.70:0/1781537097 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc1c40a2620 0x7fc1c40a4ae0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x5566d60336e0 tx=0x7fc1d000a1f0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.882+0000 7fc1e128f640 1 -- 172.21.15.70:0/1781537097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1dc1a3970 msgr2=0x7fc1dc1a5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:14.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.882+0000 7fc1e128f640 1 --2- 172.21.15.70:0/1781537097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1dc1a3970 0x7fc1dc1a5d60 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc1d400b990 tx=0x7fc1d400be50 comp rx=0 tx=0).stop 2024-04-01T01:47:14.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.882+0000 7fc1e128f640 1 -- 172.21.15.70:0/1781537097 shutdown_connections 2024-04-01T01:47:14.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.882+0000 7fc1e128f640 1 --2- 172.21.15.70:0/1781537097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1dc1a3970 0x7fc1dc1a5d60 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.882+0000 7fc1e128f640 1 --2- 172.21.15.70:0/1781537097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc1dc1052e0 0x7fc1dc19e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.882+0000 7fc1e128f640 1 --2- 172.21.15.70:0/1781537097 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc1c40a2620 0x7fc1c40a4ae0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.882+0000 7fc1e128f640 1 --2- 172.21.15.70:0/1781537097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1dc102930 0x7fc1dc19e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:14.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.882+0000 7fc1e128f640 1 -- 172.21.15.70:0/1781537097 >> 172.21.15.70:0/1781537097 conn(0x7fc1dc0fbbe0 msgr2=0x7fc1dc1048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:14.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.882+0000 7fc1e128f640 1 -- 172.21.15.70:0/1781537097 shutdown_connections 2024-04-01T01:47:14.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:14.882+0000 7fc1e128f640 1 -- 172.21.15.70:0/1781537097 wait complete. 2024-04-01T01:47:14.895 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:47:14.924 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:14 smithi070 ceph-mon[32179]: pgmap v771: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.8 MiB/s wr, 762 op/s 2024-04-01T01:47:14.924 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:14 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3942499383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:47:15.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.042+0000 7f12c5335640 1 -- 172.21.15.70:0/26796497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0102240 msgr2=0x7f12c0108ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.042+0000 7f12c5335640 1 --2- 172.21.15.70:0/26796497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0102240 0x7f12c0108ad0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009ca0 tx=0x7f12b802c0f0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.043+0000 7f12c5335640 1 -- 172.21.15.70:0/26796497 shutdown_connections 2024-04-01T01:47:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.043+0000 7f12c5335640 1 --2- 172.21.15.70:0/26796497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0102240 0x7f12c0108ad0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.043+0000 7f12c5335640 1 --2- 172.21.15.70:0/26796497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12c0101880 0x7f12c0101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.043+0000 7f12c5335640 1 --2- 172.21.15.70:0/26796497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12c0100690 0x7f12c0100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.043+0000 7f12c5335640 1 -- 172.21.15.70:0/26796497 >> 172.21.15.70:0/26796497 conn(0x7f12c00fbc00 msgr2=0x7f12c00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.043+0000 7f12c5335640 1 -- 172.21.15.70:0/26796497 shutdown_connections 2024-04-01T01:47:15.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.043+0000 7f12c5335640 1 -- 172.21.15.70:0/26796497 wait complete. 2024-04-01T01:47:15.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.044+0000 7f12c5335640 1 Processor -- start 2024-04-01T01:47:15.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.044+0000 7f12c5335640 1 -- start start 2024-04-01T01:47:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.045+0000 7f12c5335640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12c0100690 0x7f12c01966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.045+0000 7f12c5335640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12c0101880 0x7f12c0196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.045+0000 7f12c5335640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0102240 0x7f12c019dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.045+0000 7f12beffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12c0100690 0x7f12c01966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.045+0000 7f12c5335640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f12c010c1a0 con 0x7f12c0102240 2024-04-01T01:47:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.045+0000 7f12c5335640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f12c010c020 con 0x7f12c0101880 2024-04-01T01:47:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.045+0000 7f12c5335640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f12c010c320 con 0x7f12c0100690 2024-04-01T01:47:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.045+0000 7f12beffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12c0100690 0x7f12c01966c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57264/0 (socket says 172.21.15.70:57264) 2024-04-01T01:47:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.045+0000 7f12beffd640 1 -- 172.21.15.70:0/2610806068 learned_addr learned my addr 172.21.15.70:0/2610806068 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.045+0000 7f12be7fc640 1 --2- 172.21.15.70:0/2610806068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12c0101880 0x7f12c0196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.045+0000 7f12bf7fe640 1 --2- 172.21.15.70:0/2610806068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0102240 0x7f12c019dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.046+0000 7f12bf7fe640 1 -- 172.21.15.70:0/2610806068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12c0100690 msgr2=0x7f12c01966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.046+0000 7f12bf7fe640 1 --2- 172.21.15.70:0/2610806068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12c0100690 0x7f12c01966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.046+0000 7f12bf7fe640 1 -- 172.21.15.70:0/2610806068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12c0101880 msgr2=0x7f12c0196c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.046+0000 7f12bf7fe640 1 --2- 172.21.15.70:0/2610806068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12c0101880 0x7f12c0196c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.046+0000 7f12bf7fe640 1 -- 172.21.15.70:0/2610806068 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c0069b50 con 0x7f12c0102240 2024-04-01T01:47:15.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.046+0000 7f12bf7fe640 1 --2- 172.21.15.70:0/2610806068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0102240 0x7f12c019dc70 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009bf0 tx=0x7f12b8032790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.046+0000 7f12a7fff640 1 -- 172.21.15.70:0/2610806068 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b8039040 con 0x7f12c0102240 2024-04-01T01:47:15.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.047+0000 7f12c5335640 1 -- 172.21.15.70:0/2610806068 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12c0069de0 con 0x7f12c0102240 2024-04-01T01:47:15.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.047+0000 7f12c5335640 1 -- 172.21.15.70:0/2610806068 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12c006a230 con 0x7f12c0102240 2024-04-01T01:47:15.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.047+0000 7f12a7fff640 1 -- 172.21.15.70:0/2610806068 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12b8044070 con 0x7f12c0102240 2024-04-01T01:47:15.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.047+0000 7f12a7fff640 1 -- 172.21.15.70:0/2610806068 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b80044f0 con 0x7f12c0102240 2024-04-01T01:47:15.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.050+0000 7f12a7fff640 1 -- 172.21.15.70:0/2610806068 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f12b8004690 con 0x7f12c0102240 2024-04-01T01:47:15.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.050+0000 7f12c7ff1740 1 Processor -- start 2024-04-01T01:47:15.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.050+0000 7f12c7ff1740 1 -- start start 2024-04-01T01:47:15.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.050+0000 7f12c7ff1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf895ca0 0x5644bf8960c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.050+0000 7f12c7ff1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 0x5644bf89b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.050+0000 7f12c7ff1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf89b800 0x5644bf89dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.050+0000 7f12c7ff1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5644bf7f21b0 con 0x5644bf896690 2024-04-01T01:47:15.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.050+0000 7f12c7ff1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5644bf79ac00 con 0x5644bf895ca0 2024-04-01T01:47:15.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.050+0000 7f12c7ff1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5644bf7ee250 con 0x5644bf89b800 2024-04-01T01:47:15.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.051+0000 7f12beffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 0x5644bf89b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.051+0000 7f12beffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 0x5644bf89b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44796/0 (socket says 172.21.15.70:44796) 2024-04-01T01:47:15.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.051+0000 7f12beffd640 1 -- 172.21.15.70:0/884755666 learned_addr learned my addr 172.21.15.70:0/884755666 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:15.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.051+0000 7f12be7fc640 1 --2- 172.21.15.70:0/884755666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf89b800 0x5644bf89dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.051+0000 7f12be7fc640 1 --2- 172.21.15.70:0/884755666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf895ca0 0x5644bf8960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.051+0000 7f12beffd640 1 -- 172.21.15.70:0/884755666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf89b800 msgr2=0x5644bf89dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.051+0000 7f12beffd640 1 --2- 172.21.15.70:0/884755666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf89b800 0x5644bf89dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.051+0000 7f12beffd640 1 -- 172.21.15.70:0/884755666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf895ca0 msgr2=0x5644bf8960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.051+0000 7f12beffd640 1 --2- 172.21.15.70:0/884755666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf895ca0 0x5644bf8960c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.051+0000 7f12beffd640 1 -- 172.21.15.70:0/884755666 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644bf7ba3d0 con 0x5644bf896690 2024-04-01T01:47:15.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.051+0000 7f12a7fff640 1 --2- 172.21.15.70:0/2610806068 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12a80a2520 0x7f12a80a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.051+0000 7f12beffd640 1 --2- 172.21.15.70:0/884755666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 0x5644bf89b190 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f12b000bd90 tx=0x7f12b00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b5a972723bfd46 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.051+0000 7f12a7fff640 1 -- 172.21.15.70:0/2610806068 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f12b80f1240 con 0x7f12c0102240 2024-04-01T01:47:15.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.052+0000 7f12be7fc640 1 --2- 172.21.15.70:0/2610806068 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12a80a2520 0x7f12a80a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.052+0000 7f12a57fa640 1 -- 172.21.15.70:0/884755666 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b0004220 con 0x5644bf896690 2024-04-01T01:47:15.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.052+0000 7f12a57fa640 1 -- 172.21.15.70:0/884755666 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12b00043c0 con 0x5644bf896690 2024-04-01T01:47:15.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.052+0000 7f12a57fa640 1 -- 172.21.15.70:0/884755666 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b00116e0 con 0x5644bf896690 2024-04-01T01:47:15.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.052+0000 7f12c7ff1740 1 -- 172.21.15.70:0/884755666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 msgr2=0x5644bf89b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.052+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/884755666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 0x5644bf89b190 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f12b000bd90 tx=0x7f12b00107a0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.052+0000 7f12be7fc640 1 --2- 172.21.15.70:0/2610806068 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12a80a2520 0x7f12a80a49e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x5644bf896ef0 tx=0x7f12b4004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.053+0000 7f12c7ff1740 1 -- 172.21.15.70:0/884755666 shutdown_connections 2024-04-01T01:47:15.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.053+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/884755666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf89b800 0x5644bf89dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.053+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/884755666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 0x5644bf89b190 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.053+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/884755666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf895ca0 0x5644bf8960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.053+0000 7f12c7ff1740 1 -- 172.21.15.70:0/884755666 >> 172.21.15.70:0/884755666 conn(0x5644bf891210 msgr2=0x5644bf891610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.053+0000 7f12c7ff1740 1 -- 172.21.15.70:0/884755666 shutdown_connections 2024-04-01T01:47:15.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.053+0000 7f12c7ff1740 1 -- 172.21.15.70:0/884755666 wait complete. 2024-04-01T01:47:15.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.053+0000 7f12c7ff1740 1 -- start start 2024-04-01T01:47:15.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.053+0000 7f12c7ff1740 1 Processor -- start 2024-04-01T01:47:15.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.054+0000 7f12c7ff1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf895ca0 0x5644bf8a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.054+0000 7f12c7ff1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf896690 0x5644bf8a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.054+0000 7f12c7ff1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf89b800 0x5644bf8a7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.054+0000 7f12bf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf89b800 0x5644bf8a7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.054+0000 7f12c7ff1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5644bf89e7b0 con 0x5644bf89b800 2024-04-01T01:47:15.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.055+0000 7f12c7ff1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5644bf790470 con 0x5644bf896690 2024-04-01T01:47:15.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.055+0000 7f12c7ff1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5644bf89e930 con 0x5644bf895ca0 2024-04-01T01:47:15.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.055+0000 7f12bf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf89b800 0x5644bf8a7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44798/0 (socket says 172.21.15.70:44798) 2024-04-01T01:47:15.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.055+0000 7f12bf7fe640 1 -- 172.21.15.70:0/1584266059 learned_addr learned my addr 172.21.15.70:0/1584266059 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:15.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.055+0000 7f12c7ff1740 10 client.0 _ll_register_callbacks cb 0x5644bf7adcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:15.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.055+0000 7f12beffd640 1 --2- 172.21.15.70:0/1584266059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf895ca0 0x5644bf8a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.055+0000 7f12bf7fe640 1 -- 172.21.15.70:0/1584266059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf895ca0 msgr2=0x5644bf8a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.055+0000 7f12be7fc640 1 --2- 172.21.15.70:0/1584266059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf896690 0x5644bf8a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.055+0000 7f12bf7fe640 1 --2- 172.21.15.70:0/1584266059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf895ca0 0x5644bf8a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.055+0000 7f12bf7fe640 1 -- 172.21.15.70:0/1584266059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf896690 msgr2=0x5644bf8a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.055+0000 7f12bf7fe640 1 --2- 172.21.15.70:0/1584266059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf896690 0x5644bf8a06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.055+0000 7f12bf7fe640 1 -- 172.21.15.70:0/1584266059 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644bf5063f0 con 0x5644bf89b800 2024-04-01T01:47:15.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.056+0000 7f12bf7fe640 1 --2- 172.21.15.70:0/1584266059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf89b800 0x5644bf8a7720 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f12b80c5db0 tx=0x7f12b80b9c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec96b49b81ff8c44 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.056+0000 7f12a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:47:15.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.056+0000 7f12a2ff5640 1 -- 172.21.15.70:0/1584266059 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b80a6e60 con 0x5644bf89b800 2024-04-01T01:47:15.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.056+0000 7f12c7ff1740 1 -- 172.21.15.70:0/1584266059 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12b00043c0 con 0x5644bf89b800 2024-04-01T01:47:15.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.056+0000 7f12a2ff5640 1 -- 172.21.15.70:0/1584266059 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f12b80bdb30 con 0x5644bf89b800 2024-04-01T01:47:15.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.057+0000 7f12a2ff5640 1 -- 172.21.15.70:0/1584266059 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f12b80bdfd0 con 0x5644bf89b800 2024-04-01T01:47:15.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.057+0000 7f12c7ff1740 10 client.19452 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:15.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.057+0000 7f12c7ff1740 1 -- 172.21.15.70:0/1584266059 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5644bf5d07d0 con 0x5644bf89b800 2024-04-01T01:47:15.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.057+0000 7f12a2ff5640 1 -- 172.21.15.70:0/1584266059 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12b80a82e0 con 0x5644bf89b800 2024-04-01T01:47:15.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.057+0000 7f12a2ff5640 1 -- 172.21.15.70:0/1584266059 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f12b803c030 con 0x5644bf89b800 2024-04-01T01:47:15.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.057+0000 7f12c7ff1740 10 client.19452 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:15.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.057+0000 7f12c7ff1740 10 client.19452 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:47:15.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.057+0000 7f12c7ff1740 20 client.19452 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:15.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.058+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/1584266059 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5644bf8abb90 0x5644bf8adff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.058+0000 7f12c7ff1740 4 client.19452 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:15.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.058+0000 7f12c7ff1740 1 -- 172.21.15.70:0/1584266059 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f12b00043c0 con 0x5644bf8abb90 2024-04-01T01:47:15.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.058+0000 7f12beffd640 1 --2- 172.21.15.70:0/1584266059 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5644bf8abb90 0x5644bf8adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.060+0000 7f12beffd640 1 --2- 172.21.15.70:0/1584266059 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5644bf8abb90 0x5644bf8adff0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b0feea561b9c819a server_cookie=c27e5d97e26eec48 in_seq=0 out_seq=0 2024-04-01T01:47:15.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.060+0000 7f12a2ff5640 10 client.19452 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:47:15.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.063+0000 7f12a2ff5640 1 -- 172.21.15.70:0/1584266059 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f12b00043c0 con 0x5644bf8abb90 2024-04-01T01:47:15.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.063+0000 7f12a2ff5640 10 client.19452 handle_command_reply: tid=0 2024-04-01T01:47:15.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.063+0000 7f12c7ff1740 1 client.19452 shutdown 2024-04-01T01:47:15.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.066+0000 7f12c7ff1740 1 -- 172.21.15.70:0/1584266059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf89b800 msgr2=0x5644bf8a7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.066+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/1584266059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf89b800 0x5644bf8a7720 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f12b80c5db0 tx=0x7f12b80b9c30 comp rx=0 tx=0).stop 2024-04-01T01:47:15.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.068+0000 7f12c7ff1740 1 -- 172.21.15.70:0/1584266059 shutdown_connections 2024-04-01T01:47:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.068+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/1584266059 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5644bf8abb90 0x5644bf8adff0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.068+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/1584266059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf89b800 0x5644bf8a7720 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.068+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/1584266059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf896690 0x5644bf8a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.068+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/1584266059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf895ca0 0x5644bf8a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.068+0000 7f12c7ff1740 1 -- 172.21.15.70:0/1584266059 >> 172.21.15.70:0/1584266059 conn(0x5644bf891210 msgr2=0x5644bf887d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.068+0000 7f12a2ff5640 0 client.19452 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:47:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.070+0000 7f12c7ff1740 1 -- 172.21.15.70:0/1584266059 shutdown_connections 2024-04-01T01:47:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.070+0000 7f12c7ff1740 1 -- 172.21.15.70:0/1584266059 wait complete. 2024-04-01T01:47:15.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.070+0000 7f12c7ff1740 20 client.19452 trim_cache size 0 max 16384 2024-04-01T01:47:15.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:14 smithi081 ceph-mon[35142]: pgmap v771: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.8 MiB/s wr, 762 op/s 2024-04-01T01:47:15.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:14 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3942499383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:47:15.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.094+0000 7f12c7ff1740 1 Processor -- start 2024-04-01T01:47:15.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.094+0000 7f12c7ff1740 1 -- start start 2024-04-01T01:47:15.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.094+0000 7f12c7ff1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf895ca0 0x5644bf887bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.094+0000 7f12c7ff1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 0x5644bf8a76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.094+0000 7f12c7ff1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf89b800 0x5644bf8b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.094+0000 7f12c7ff1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac00fcd0 con 0x5644bf896690 2024-04-01T01:47:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.094+0000 7f12c7ff1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac011810 con 0x5644bf895ca0 2024-04-01T01:47:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.094+0000 7f12c7ff1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac0103a0 con 0x5644bf89b800 2024-04-01T01:47:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.094+0000 7f12beffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf895ca0 0x5644bf887bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.095+0000 7f12beffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf895ca0 0x5644bf887bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47982/0 (socket says 172.21.15.70:47982) 2024-04-01T01:47:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.095+0000 7f12beffd640 1 -- 172.21.15.70:0/949488162 learned_addr learned my addr 172.21.15.70:0/949488162 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.095+0000 7f12bf7fe640 1 --2- 172.21.15.70:0/949488162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf89b800 0x5644bf8b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.095+0000 7f12beffd640 1 -- 172.21.15.70:0/949488162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf89b800 msgr2=0x5644bf8b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.095+0000 7f12beffd640 1 --2- 172.21.15.70:0/949488162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf89b800 0x5644bf8b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.095+0000 7f12beffd640 1 -- 172.21.15.70:0/949488162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 msgr2=0x5644bf8a76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.095+0000 7f12beffd640 1 --2- 172.21.15.70:0/949488162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 0x5644bf8a76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.095+0000 7f12beffd640 1 -- 172.21.15.70:0/949488162 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644bf896f20 con 0x5644bf895ca0 2024-04-01T01:47:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.095+0000 7f12bf7fe640 1 --2- 172.21.15.70:0/949488162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf89b800 0x5644bf8b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:47:15.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.095+0000 7f12beffd640 1 --2- 172.21.15.70:0/949488162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf895ca0 0x5644bf887bc0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f12b0004610 tx=0x7f12b000bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=156daf80d28415ec server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.096+0000 7f12a4ff9640 1 -- 172.21.15.70:0/949488162 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b0005240 con 0x5644bf895ca0 2024-04-01T01:47:15.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.096+0000 7f12a4ff9640 1 -- 172.21.15.70:0/949488162 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12b00053e0 con 0x5644bf895ca0 2024-04-01T01:47:15.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.096+0000 7f12a4ff9640 1 -- 172.21.15.70:0/949488162 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b00048e0 con 0x5644bf895ca0 2024-04-01T01:47:15.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.097+0000 7f12c7ff1740 1 -- 172.21.15.70:0/949488162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf895ca0 msgr2=0x5644bf887bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.097+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/949488162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf895ca0 0x5644bf887bc0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f12b0004610 tx=0x7f12b000bea0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.097+0000 7f12c7ff1740 1 -- 172.21.15.70:0/949488162 shutdown_connections 2024-04-01T01:47:15.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.097+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/949488162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf89b800 0x5644bf8b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.097+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/949488162 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 0x5644bf8a76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.097+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/949488162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf895ca0 0x5644bf887bc0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.097+0000 7f12c7ff1740 1 -- 172.21.15.70:0/949488162 >> 172.21.15.70:0/949488162 conn(0x5644bf891210 msgr2=0x5644bf8874b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.097+0000 7f12c7ff1740 1 -- 172.21.15.70:0/949488162 shutdown_connections 2024-04-01T01:47:15.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.097+0000 7f12c7ff1740 1 -- 172.21.15.70:0/949488162 wait complete. 2024-04-01T01:47:15.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.097+0000 7f12c7ff1740 1 -- start start 2024-04-01T01:47:15.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.097+0000 7f12c7ff1740 1 Processor -- start 2024-04-01T01:47:15.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.098+0000 7f12c7ff1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf895ca0 0x5644bf8a6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.098+0000 7f12c7ff1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 0x5644bf8a74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.098+0000 7f12c7ff1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf89b800 0x5644bf8a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.098+0000 7f12c7ff1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac011140 con 0x5644bf896690 2024-04-01T01:47:15.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.098+0000 7f12c7ff1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac010a70 con 0x5644bf89b800 2024-04-01T01:47:15.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.098+0000 7f12c7ff1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac00dcb0 con 0x5644bf895ca0 2024-04-01T01:47:15.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.098+0000 7f12be7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 0x5644bf8a74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.098+0000 7f12c7ff1740 10 client.0 _ll_register_callbacks cb 0x7f12b8032980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:15.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.098+0000 7f12be7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 0x5644bf8a74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44824/0 (socket says 172.21.15.70:44824) 2024-04-01T01:47:15.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.098+0000 7f12be7fc640 1 -- 172.21.15.70:0/3796605080 learned_addr learned my addr 172.21.15.70:0/3796605080 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:15.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.098+0000 7f12beffd640 1 --2- 172.21.15.70:0/3796605080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf895ca0 0x5644bf8a6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.099+0000 7f12be7fc640 1 -- 172.21.15.70:0/3796605080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf895ca0 msgr2=0x5644bf8a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.099+0000 7f12be7fc640 1 --2- 172.21.15.70:0/3796605080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf895ca0 0x5644bf8a6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.099+0000 7f12be7fc640 1 -- 172.21.15.70:0/3796605080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf89b800 msgr2=0x5644bf8a55c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:47:15.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.099+0000 7f12be7fc640 1 --2- 172.21.15.70:0/3796605080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf89b800 0x5644bf8a55c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.099+0000 7f12be7fc640 1 -- 172.21.15.70:0/3796605080 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644bf8a5cb0 con 0x5644bf896690 2024-04-01T01:47:15.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.099+0000 7f12be7fc640 1 --2- 172.21.15.70:0/3796605080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 0x5644bf8a74f0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f12b4010c30 tx=0x7f12b4009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=140146d6ef9ed552 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.099+0000 7f12a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:47:15.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.099+0000 7f12a2ff5640 1 -- 172.21.15.70:0/3796605080 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12b402e060 con 0x5644bf896690 2024-04-01T01:47:15.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.099+0000 7f12c7ff1740 1 -- 172.21.15.70:0/3796605080 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12b00053e0 con 0x5644bf896690 2024-04-01T01:47:15.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.099+0000 7f12a2ff5640 1 -- 172.21.15.70:0/3796605080 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f12b401ea90 con 0x5644bf896690 2024-04-01T01:47:15.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.100+0000 7f12a2ff5640 1 -- 172.21.15.70:0/3796605080 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f12b4035050 con 0x5644bf896690 2024-04-01T01:47:15.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.100+0000 7f12c7ff1740 10 client.19455 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:15.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.100+0000 7f12c7ff1740 1 -- 172.21.15.70:0/3796605080 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5644bf912a60 con 0x5644bf896690 2024-04-01T01:47:15.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.101+0000 7f12a2ff5640 1 -- 172.21.15.70:0/3796605080 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12b00053e0 con 0x5644bf896690 2024-04-01T01:47:15.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.101+0000 7f12a2ff5640 1 -- 172.21.15.70:0/3796605080 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f12b4028c00 con 0x5644bf896690 2024-04-01T01:47:15.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.101+0000 7f12c7ff1740 10 client.19455 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:15.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.101+0000 7f12c7ff1740 10 client.19455 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:47:15.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.101+0000 7f12c7ff1740 20 client.19455 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:15.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.101+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/3796605080 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5644bf913420 0x5644bf915840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.101+0000 7f12c7ff1740 4 client.19455 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:47:15.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.101+0000 7f12c7ff1740 1 -- 172.21.15.70:0/3796605080 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5644bf7ff6f0 con 0x5644bf913420 2024-04-01T01:47:15.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.101+0000 7f12beffd640 1 --2- 172.21.15.70:0/3796605080 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5644bf913420 0x5644bf915840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.102+0000 7f12a2ff5640 10 client.19455 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:47:15.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.102+0000 7f12beffd640 1 --2- 172.21.15.70:0/3796605080 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5644bf913420 0x5644bf915840 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=71a84a23f897973c server_cookie=b1d7dd26de93a5e3 in_seq=0 out_seq=0 2024-04-01T01:47:15.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.102+0000 7f12a2ff5640 1 -- 172.21.15.70:0/3796605080 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5644bf7ff6f0 con 0x5644bf913420 2024-04-01T01:47:15.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.102+0000 7f12a2ff5640 10 client.19455 handle_command_reply: tid=0 2024-04-01T01:47:15.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.102+0000 7f12c7ff1740 1 client.19455 shutdown 2024-04-01T01:47:15.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.102+0000 7f12c7ff1740 1 -- 172.21.15.70:0/3796605080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 msgr2=0x5644bf8a74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.102+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/3796605080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 0x5644bf8a74f0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f12b4010c30 tx=0x7f12b4009b40 comp rx=0 tx=0).stop 2024-04-01T01:47:15.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.102+0000 7f12c7ff1740 1 -- 172.21.15.70:0/3796605080 shutdown_connections 2024-04-01T01:47:15.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.102+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/3796605080 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5644bf913420 0x5644bf915840 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.102+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/3796605080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5644bf89b800 0x5644bf8a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.102+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/3796605080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644bf896690 0x5644bf8a74f0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.102+0000 7f12a2ff5640 0 client.19455 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:47:15.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.102+0000 7f12c7ff1740 1 --2- 172.21.15.70:0/3796605080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5644bf895ca0 0x5644bf8a6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.102+0000 7f12c7ff1740 1 -- 172.21.15.70:0/3796605080 >> 172.21.15.70:0/3796605080 conn(0x5644bf891210 msgr2=0x5644bf894470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.103+0000 7f12c7ff1740 1 -- 172.21.15.70:0/3796605080 shutdown_connections 2024-04-01T01:47:15.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.103+0000 7f12c7ff1740 1 -- 172.21.15.70:0/3796605080 wait complete. 2024-04-01T01:47:15.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.103+0000 7f12c7ff1740 20 client.19455 trim_cache size 0 max 16384 2024-04-01T01:47:15.105 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:47:15.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.103+0000 7f12c5335640 1 -- 172.21.15.70:0/2610806068 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12a80a2520 msgr2=0x7f12a80a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.103+0000 7f12c5335640 1 --2- 172.21.15.70:0/2610806068 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12a80a2520 0x7f12a80a49e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x5644bf896ef0 tx=0x7f12b4004730 comp rx=0 tx=0).stop 2024-04-01T01:47:15.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.103+0000 7f12c5335640 1 -- 172.21.15.70:0/2610806068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0102240 msgr2=0x7f12c019dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.103+0000 7f12c5335640 1 --2- 172.21.15.70:0/2610806068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0102240 0x7f12c019dc70 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f12b8009bf0 tx=0x7f12b8032790 comp rx=0 tx=0).stop 2024-04-01T01:47:15.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.103+0000 7f12c5335640 1 -- 172.21.15.70:0/2610806068 shutdown_connections 2024-04-01T01:47:15.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.103+0000 7f12c5335640 1 --2- 172.21.15.70:0/2610806068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f12c0102240 0x7f12c019dc70 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.103+0000 7f12c5335640 1 --2- 172.21.15.70:0/2610806068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f12c0101880 0x7f12c0196c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.103+0000 7f12c5335640 1 --2- 172.21.15.70:0/2610806068 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12a80a2520 0x7f12a80a49e0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.103+0000 7f12c5335640 1 --2- 172.21.15.70:0/2610806068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12c0100690 0x7f12c01966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.103+0000 7f12c5335640 1 -- 172.21.15.70:0/2610806068 >> 172.21.15.70:0/2610806068 conn(0x7f12c00fbc00 msgr2=0x7f12c0104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.104+0000 7f12c5335640 1 -- 172.21.15.70:0/2610806068 shutdown_connections 2024-04-01T01:47:15.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.104+0000 7f12c5335640 1 -- 172.21.15.70:0/2610806068 wait complete. 2024-04-01T01:47:15.118 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:47:15.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:14 smithi189 ceph-mon[35753]: pgmap v771: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 6.8 MiB/s wr, 762 op/s 2024-04-01T01:47:15.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:14 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3942499383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:47:15.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.301+0000 7f54f74f4640 1 -- 172.21.15.70:0/3302803145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54f010de20 msgr2=0x7f54f01102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.301+0000 7f54f74f4640 1 --2- 172.21.15.70:0/3302803145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54f010de20 0x7f54f01102e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f54e8009ca0 tx=0x7f54e802c0f0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.302+0000 7f54f74f4640 1 -- 172.21.15.70:0/3302803145 shutdown_connections 2024-04-01T01:47:15.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.302+0000 7f54f74f4640 1 --2- 172.21.15.70:0/3302803145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54f010de20 0x7f54f01102e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.302+0000 7f54f74f4640 1 --2- 172.21.15.70:0/3302803145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54f010b380 0x7f54f010d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.302+0000 7f54f74f4640 1 --2- 172.21.15.70:0/3302803145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54f01089c0 0x7f54f010adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.302+0000 7f54f74f4640 1 -- 172.21.15.70:0/3302803145 >> 172.21.15.70:0/3302803145 conn(0x7f54f006e5b0 msgr2=0x7f54f00709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.302+0000 7f54f74f4640 1 -- 172.21.15.70:0/3302803145 shutdown_connections 2024-04-01T01:47:15.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.302+0000 7f54f74f4640 1 -- 172.21.15.70:0/3302803145 wait complete. 2024-04-01T01:47:15.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f74f4640 1 Processor -- start 2024-04-01T01:47:15.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f74f4640 1 -- start start 2024-04-01T01:47:15.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f74f4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54f01089c0 0x7f54f01a2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f74f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54f010b380 0x7f54f01a2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f74f4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54f010de20 0x7f54f01a99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f74f4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f54f0117d10 con 0x7f54f010b380 2024-04-01T01:47:15.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f74f4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f54f0117b90 con 0x7f54f010de20 2024-04-01T01:47:15.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f74f4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f54f0117e90 con 0x7f54f01089c0 2024-04-01T01:47:15.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f4a8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54f010b380 0x7f54f01a2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f5a91640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54f010de20 0x7f54f01a99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f5a91640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54f010de20 0x7f54f01a99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48004/0 (socket says 172.21.15.70:48004) 2024-04-01T01:47:15.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f5a91640 1 -- 172.21.15.70:0/3129377983 learned_addr learned my addr 172.21.15.70:0/3129377983 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:15.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f5290640 1 --2- 172.21.15.70:0/3129377983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54f01089c0 0x7f54f01a2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f5a91640 1 -- 172.21.15.70:0/3129377983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54f01089c0 msgr2=0x7f54f01a2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f5a91640 1 --2- 172.21.15.70:0/3129377983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54f01089c0 0x7f54f01a2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f5a91640 1 -- 172.21.15.70:0/3129377983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54f010b380 msgr2=0x7f54f01a2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f5a91640 1 --2- 172.21.15.70:0/3129377983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54f010b380 0x7f54f01a2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f5a91640 1 -- 172.21.15.70:0/3129377983 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54f01aa050 con 0x7f54f010de20 2024-04-01T01:47:15.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.303+0000 7f54f4a8f640 1 --2- 172.21.15.70:0/3129377983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54f010b380 0x7f54f01a2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:47:15.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.304+0000 7f54f5a91640 1 --2- 172.21.15.70:0/3129377983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54f010de20 0x7f54f01a99e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f54e8037040 tx=0x7f54e8032820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.304+0000 7f54e67fc640 1 -- 172.21.15.70:0/3129377983 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e8041020 con 0x7f54f010de20 2024-04-01T01:47:15.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.304+0000 7f54f74f4640 1 -- 172.21.15.70:0/3129377983 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54f01aa2e0 con 0x7f54f010de20 2024-04-01T01:47:15.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.304+0000 7f54e67fc640 1 -- 172.21.15.70:0/3129377983 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54e8034700 con 0x7f54f010de20 2024-04-01T01:47:15.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.304+0000 7f54f74f4640 1 -- 172.21.15.70:0/3129377983 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54f01aa820 con 0x7f54f010de20 2024-04-01T01:47:15.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.304+0000 7f54e67fc640 1 -- 172.21.15.70:0/3129377983 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e8039040 con 0x7f54f010de20 2024-04-01T01:47:15.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.307+0000 7f54e67fc640 1 -- 172.21.15.70:0/3129377983 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f54e8044020 con 0x7f54f010de20 2024-04-01T01:47:15.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.308+0000 7f54fa1b5740 1 Processor -- start 2024-04-01T01:47:15.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.308+0000 7f54e67fc640 1 --2- 172.21.15.70:0/3129377983 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54dc0a24f0 0x7f54dc0a49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.309+0000 7f54e67fc640 1 -- 172.21.15.70:0/3129377983 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f54e8009870 con 0x7f54f010de20 2024-04-01T01:47:15.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.309+0000 7f54fa1b5740 1 -- start start 2024-04-01T01:47:15.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.309+0000 7f54fa1b5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b1490 0x5572180b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.309+0000 7f54fa1b5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180b1e80 0x5572180b6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.309+0000 7f54fa1b5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b6ff0 0x5572180b9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.309+0000 7f54fa1b5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5572180161b0 con 0x5572180b1e80 2024-04-01T01:47:15.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.309+0000 7f54fa1b5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x557217fbec00 con 0x5572180b1490 2024-04-01T01:47:15.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.309+0000 7f54fa1b5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557218012250 con 0x5572180b6ff0 2024-04-01T01:47:15.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.310+0000 7f54f5290640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b1490 0x5572180b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.310+0000 7f54f5290640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b1490 0x5572180b18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48010/0 (socket says 172.21.15.70:48010) 2024-04-01T01:47:15.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.310+0000 7f54f5290640 1 -- 172.21.15.70:0/1543186832 learned_addr learned my addr 172.21.15.70:0/1543186832 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.310+0000 7f54f5290640 1 -- 172.21.15.70:0/1543186832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b6ff0 msgr2=0x5572180b9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.310+0000 7f54f5290640 1 --2- 172.21.15.70:0/1543186832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b6ff0 0x5572180b9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.310+0000 7f54f5290640 1 -- 172.21.15.70:0/1543186832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180b1e80 msgr2=0x5572180b6980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:47:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.310+0000 7f54f5290640 1 --2- 172.21.15.70:0/1543186832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180b1e80 0x5572180b6980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.310+0000 7f54f5290640 1 -- 172.21.15.70:0/1543186832 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557217fde3d0 con 0x5572180b1490 2024-04-01T01:47:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.311+0000 7f54f5290640 1 --2- 172.21.15.70:0/1543186832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b1490 0x5572180b18b0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f54e000bcf0 tx=0x7f54e0005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5ee579badbd40f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.311+0000 7f54f4a8f640 1 --2- 172.21.15.70:0/3129377983 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54dc0a24f0 0x7f54dc0a49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.312+0000 7f54f4a8f640 1 --2- 172.21.15.70:0/3129377983 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54dc0a24f0 0x7f54dc0a49b0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x5572180b2280 tx=0x7f54ec00c040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.314+0000 7f54db7fe640 1 -- 172.21.15.70:0/1543186832 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e0015400 con 0x5572180b1490 2024-04-01T01:47:15.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.314+0000 7f54db7fe640 1 -- 172.21.15.70:0/1543186832 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54e001a070 con 0x5572180b1490 2024-04-01T01:47:15.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.314+0000 7f54db7fe640 1 -- 172.21.15.70:0/1543186832 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e0014760 con 0x5572180b1490 2024-04-01T01:47:15.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.315+0000 7f54fa1b5740 1 -- 172.21.15.70:0/1543186832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b1490 msgr2=0x5572180b18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.315+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/1543186832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b1490 0x5572180b18b0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f54e000bcf0 tx=0x7f54e0005f90 comp rx=0 tx=0).stop 2024-04-01T01:47:15.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.316+0000 7f54fa1b5740 1 -- 172.21.15.70:0/1543186832 shutdown_connections 2024-04-01T01:47:15.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.316+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/1543186832 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b6ff0 0x5572180b9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.316+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/1543186832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180b1e80 0x5572180b6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.316+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/1543186832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b1490 0x5572180b18b0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.316+0000 7f54fa1b5740 1 -- 172.21.15.70:0/1543186832 >> 172.21.15.70:0/1543186832 conn(0x5572180aca00 msgr2=0x5572180ace00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.320+0000 7f54fa1b5740 1 -- 172.21.15.70:0/1543186832 shutdown_connections 2024-04-01T01:47:15.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.323+0000 7f54fa1b5740 1 -- 172.21.15.70:0/1543186832 wait complete. 2024-04-01T01:47:15.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.323+0000 7f54fa1b5740 1 -- start start 2024-04-01T01:47:15.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.323+0000 7f54fa1b5740 1 Processor -- start 2024-04-01T01:47:15.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.325+0000 7f54fa1b5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b1490 0x5572180e6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.325+0000 7f54fa1b5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180b1e80 0x5572180e67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.325+0000 7f54fa1b5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 0x5572180ed7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.325+0000 7f54fa1b5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5572180dc000 con 0x5572180b1e80 2024-04-01T01:47:15.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.325+0000 7f54fa1b5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x557217fb4470 con 0x5572180b6ff0 2024-04-01T01:47:15.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.325+0000 7f54fa1b5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5572180dc180 con 0x5572180b1490 2024-04-01T01:47:15.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.325+0000 7f54fa1b5740 10 client.0 _ll_register_callbacks cb 0x557217fd1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:15.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.326+0000 7f54f4a8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180b1e80 0x5572180e67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.326+0000 7f54f4a8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180b1e80 0x5572180e67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44846/0 (socket says 172.21.15.70:44846) 2024-04-01T01:47:15.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.326+0000 7f54f4a8f640 1 -- 172.21.15.70:0/3912754084 learned_addr learned my addr 172.21.15.70:0/3912754084 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:15.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.326+0000 7f54f4a8f640 1 -- 172.21.15.70:0/3912754084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b1490 msgr2=0x5572180e6270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:47:15.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.326+0000 7f54f4a8f640 1 --2- 172.21.15.70:0/3912754084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b1490 0x5572180e6270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.326+0000 7f54f5a91640 1 --2- 172.21.15.70:0/3912754084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 0x5572180ed7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.326+0000 7f54f4a8f640 1 -- 172.21.15.70:0/3912754084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 msgr2=0x5572180ed7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.327+0000 7f54f4a8f640 1 --2- 172.21.15.70:0/3912754084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 0x5572180ed7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.327+0000 7f54f4a8f640 1 -- 172.21.15.70:0/3912754084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557217d2a3f0 con 0x5572180b1e80 2024-04-01T01:47:15.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.327+0000 7f54f5a91640 1 --2- 172.21.15.70:0/3912754084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 0x5572180ed7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:47:15.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.327+0000 7f54f4a8f640 1 --2- 172.21.15.70:0/3912754084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180b1e80 0x5572180e67b0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f54ec003ba0 tx=0x7f54ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5eaa03385cf60c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.328+0000 7f54d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:47:15.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.328+0000 7f54d8ff9640 1 -- 172.21.15.70:0/3912754084 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ec01b070 con 0x5572180b1e80 2024-04-01T01:47:15.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.329+0000 7f54d8ff9640 1 -- 172.21.15.70:0/3912754084 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f54ec017930 con 0x5572180b1e80 2024-04-01T01:47:15.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.329+0000 7f54fa1b5740 1 -- 172.21.15.70:0/3912754084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54e001a070 con 0x5572180b1e80 2024-04-01T01:47:15.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.331+0000 7f54d8ff9640 1 -- 172.21.15.70:0/3912754084 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f54ec017b30 con 0x5572180b1e80 2024-04-01T01:47:15.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.331+0000 7f54fa1b5740 10 client.19464 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:15.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.331+0000 7f54fa1b5740 1 -- 172.21.15.70:0/3912754084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557217df47d0 con 0x5572180b1e80 2024-04-01T01:47:15.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.332+0000 7f54d8ff9640 1 -- 172.21.15.70:0/3912754084 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54e001a070 con 0x5572180b1e80 2024-04-01T01:47:15.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.332+0000 7f54d8ff9640 1 -- 172.21.15.70:0/3912754084 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f54ec015400 con 0x5572180b1e80 2024-04-01T01:47:15.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.332+0000 7f54fa1b5740 10 client.19464 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:15.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.332+0000 7f54fa1b5740 10 client.19464 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:47:15.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.332+0000 7f54fa1b5740 20 client.19464 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:15.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.332+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/3912754084 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5572180e1c70 0x5572180e4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.332+0000 7f54fa1b5740 4 client.19464 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:15.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.332+0000 7f54fa1b5740 1 -- 172.21.15.70:0/3912754084 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572180236f0 con 0x5572180e1c70 2024-04-01T01:47:15.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.333+0000 7f54f5290640 1 --2- 172.21.15.70:0/3912754084 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5572180e1c70 0x5572180e4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.336+0000 7f54f5290640 1 --2- 172.21.15.70:0/3912754084 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5572180e1c70 0x5572180e4130 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=74d83a8cec78d3db server_cookie=4d31b226be8846d in_seq=0 out_seq=0 2024-04-01T01:47:15.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.336+0000 7f54d8ff9640 10 client.19464 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:47:15.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.337+0000 7f54d8ff9640 1 -- 172.21.15.70:0/3912754084 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5572180236f0 con 0x5572180e1c70 2024-04-01T01:47:15.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.337+0000 7f54d8ff9640 10 client.19464 handle_command_reply: tid=0 2024-04-01T01:47:15.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.337+0000 7f54fa1b5740 1 client.19464 shutdown 2024-04-01T01:47:15.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.339+0000 7f54fa1b5740 1 -- 172.21.15.70:0/3912754084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180b1e80 msgr2=0x5572180e67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.339+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/3912754084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180b1e80 0x5572180e67b0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f54ec003ba0 tx=0x7f54ec0046c0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.339+0000 7f54fa1b5740 1 -- 172.21.15.70:0/3912754084 shutdown_connections 2024-04-01T01:47:15.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.339+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/3912754084 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5572180e1c70 0x5572180e4130 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.339+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/3912754084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 0x5572180ed7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.339+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/3912754084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180b1e80 0x5572180e67b0 secure :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f54ec003ba0 tx=0x7f54ec0046c0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.339+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/3912754084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b1490 0x5572180e6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.339+0000 7f54fa1b5740 1 -- 172.21.15.70:0/3912754084 >> 172.21.15.70:0/3912754084 conn(0x5572180aca00 msgr2=0x5572180abd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.339+0000 7f54d8ff9640 0 client.19464 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:47:15.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.342+0000 7f54fa1b5740 1 -- 172.21.15.70:0/3912754084 shutdown_connections 2024-04-01T01:47:15.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.342+0000 7f54fa1b5740 1 -- 172.21.15.70:0/3912754084 wait complete. 2024-04-01T01:47:15.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.342+0000 7f54fa1b5740 20 client.19464 trim_cache size 0 max 16384 2024-04-01T01:47:15.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.380+0000 7f54fa1b5740 1 Processor -- start 2024-04-01T01:47:15.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.380+0000 7f54fa1b5740 1 -- start start 2024-04-01T01:47:15.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.380+0000 7f54fa1b5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b1490 0x5572180e6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.381+0000 7f54fa1b5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 0x5572180ed7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.381+0000 7f54fa1b5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180e67b0 0x5572180b3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.381+0000 7f54fa1b5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f54f00799b0 con 0x5572180e67b0 2024-04-01T01:47:15.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.381+0000 7f54fa1b5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f54f007b4f0 con 0x5572180b6ff0 2024-04-01T01:47:15.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.381+0000 7f54fa1b5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f54f007a080 con 0x5572180b1490 2024-04-01T01:47:15.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.381+0000 7f54f5290640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b1490 0x5572180e6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.381+0000 7f54f5290640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b1490 0x5572180e6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57318/0 (socket says 172.21.15.70:57318) 2024-04-01T01:47:15.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.381+0000 7f54f5290640 1 -- 172.21.15.70:0/2741245803 learned_addr learned my addr 172.21.15.70:0/2741245803 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:15.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.381+0000 7f54f5a91640 1 --2- 172.21.15.70:0/2741245803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180e67b0 0x5572180b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.381+0000 7f54f4a8f640 1 --2- 172.21.15.70:0/2741245803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 0x5572180ed7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.382+0000 7f54f5a91640 1 -- 172.21.15.70:0/2741245803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b1490 msgr2=0x5572180e6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.382+0000 7f54f5a91640 1 --2- 172.21.15.70:0/2741245803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b1490 0x5572180e6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.382+0000 7f54f5a91640 1 -- 172.21.15.70:0/2741245803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 msgr2=0x5572180ed7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.382+0000 7f54f5a91640 1 --2- 172.21.15.70:0/2741245803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 0x5572180ed7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.382 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.382+0000 7f54f5a91640 1 -- 172.21.15.70:0/2741245803 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572180b3950 con 0x5572180e67b0 2024-04-01T01:47:15.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.382+0000 7f54f5a91640 1 --2- 172.21.15.70:0/2741245803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180e67b0 0x5572180b3410 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f54e80bd180 tx=0x7f54e80bcfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=123ba735cbcf4bf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.384+0000 7f54daffd640 1 -- 172.21.15.70:0/2741245803 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e80dd070 con 0x5572180e67b0 2024-04-01T01:47:15.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.384+0000 7f54daffd640 1 -- 172.21.15.70:0/2741245803 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54e80c1340 con 0x5572180e67b0 2024-04-01T01:47:15.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.385+0000 7f54fa1b5740 1 -- 172.21.15.70:0/2741245803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180e67b0 msgr2=0x5572180b3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.385+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/2741245803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180e67b0 0x5572180b3410 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f54e80bd180 tx=0x7f54e80bcfc0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.387+0000 7f54fa1b5740 1 -- 172.21.15.70:0/2741245803 shutdown_connections 2024-04-01T01:47:15.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.387+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/2741245803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180e67b0 0x5572180b3410 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.387+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/2741245803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 0x5572180ed7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.387+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/2741245803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b1490 0x5572180e6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.387+0000 7f54fa1b5740 1 -- 172.21.15.70:0/2741245803 >> 172.21.15.70:0/2741245803 conn(0x5572180aca00 msgr2=0x5572180acde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.387+0000 7f54fa1b5740 1 -- 172.21.15.70:0/2741245803 shutdown_connections 2024-04-01T01:47:15.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.387+0000 7f54fa1b5740 1 -- 172.21.15.70:0/2741245803 wait complete. 2024-04-01T01:47:15.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.388+0000 7f54fa1b5740 1 -- start start 2024-04-01T01:47:15.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.388+0000 7f54fa1b5740 1 Processor -- start 2024-04-01T01:47:15.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.388+0000 7f54fa1b5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b1490 0x5572180ed0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.388+0000 7f54fa1b5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 0x5572180ed620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.388+0000 7f54fa1b5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180e67b0 0x5572180e9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.388+0000 7f54fa1b5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f54f007ae20 con 0x5572180e67b0 2024-04-01T01:47:15.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.388+0000 7f54fa1b5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f54f007a750 con 0x5572180b6ff0 2024-04-01T01:47:15.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.388+0000 7f54fa1b5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f54f0077990 con 0x5572180b1490 2024-04-01T01:47:15.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.388+0000 7f54fa1b5740 10 client.0 _ll_register_callbacks cb 0x7f54ec003820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:15.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.389+0000 7f54f4a8f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 0x5572180ed620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.389+0000 7f54f4a8f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 0x5572180ed620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48050/0 (socket says 172.21.15.70:48050) 2024-04-01T01:47:15.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.389+0000 7f54f4a8f640 1 -- 172.21.15.70:0/1412970479 learned_addr learned my addr 172.21.15.70:0/1412970479 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:15.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.389+0000 7f54f5a91640 1 --2- 172.21.15.70:0/1412970479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180e67b0 0x5572180e9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.390+0000 7f54f5290640 1 --2- 172.21.15.70:0/1412970479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b1490 0x5572180ed0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.390+0000 7f54f4a8f640 1 -- 172.21.15.70:0/1412970479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b1490 msgr2=0x5572180ed0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.390+0000 7f54f4a8f640 1 --2- 172.21.15.70:0/1412970479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b1490 0x5572180ed0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.390+0000 7f54f4a8f640 1 -- 172.21.15.70:0/1412970479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180e67b0 msgr2=0x5572180e9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.390+0000 7f54f4a8f640 1 --2- 172.21.15.70:0/1412970479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180e67b0 0x5572180e9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.390+0000 7f54f4a8f640 1 -- 172.21.15.70:0/1412970479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572180ea360 con 0x5572180b6ff0 2024-04-01T01:47:15.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.390+0000 7f54f4a8f640 1 --2- 172.21.15.70:0/1412970479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 0x5572180ed620 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f54ec015a60 tx=0x7f54ec003d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45b4f0eb3d973307 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.391+0000 7f54d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:15.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.392+0000 7f54d8ff9640 1 -- 172.21.15.70:0/1412970479 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ec01b040 con 0x5572180b6ff0 2024-04-01T01:47:15.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.392+0000 7f54fa1b5740 1 -- 172.21.15.70:0/1412970479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54e80c1340 con 0x5572180b6ff0 2024-04-01T01:47:15.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.393+0000 7f54d8ff9640 1 -- 172.21.15.70:0/1412970479 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f54ec015600 con 0x5572180b6ff0 2024-04-01T01:47:15.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.393+0000 7f54d8ff9640 1 -- 172.21.15.70:0/1412970479 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f54ec043050 con 0x5572180b6ff0 2024-04-01T01:47:15.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.393+0000 7f54fa1b5740 10 client.29543 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:15.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.393+0000 7f54fa1b5740 1 -- 172.21.15.70:0/1412970479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557218161060 con 0x5572180b6ff0 2024-04-01T01:47:15.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.394+0000 7f54d8ff9640 1 -- 172.21.15.70:0/1412970479 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f54e80c1340 con 0x5572180b6ff0 2024-04-01T01:47:15.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.395+0000 7f54d8ff9640 1 -- 172.21.15.70:0/1412970479 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f54ec004a30 con 0x5572180b6ff0 2024-04-01T01:47:15.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.395+0000 7f54fa1b5740 10 client.29543 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:15.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.395+0000 7f54fa1b5740 10 client.29543 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:47:15.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.395+0000 7f54fa1b5740 20 client.29543 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:15.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.395+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/1412970479 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x557218161a40 0x557218163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.395+0000 7f54fa1b5740 4 client.29543 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:47:15.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.395+0000 7f54fa1b5740 1 -- 172.21.15.70:0/1412970479 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55721801da40 con 0x557218161a40 2024-04-01T01:47:15.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.398+0000 7f54f5290640 1 --2- 172.21.15.70:0/1412970479 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x557218161a40 0x557218163e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.402+0000 7f54f5290640 1 --2- 172.21.15.70:0/1412970479 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x557218161a40 0x557218163e60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6f49012e86b6dd56 server_cookie=f9c5f72300b42b75 in_seq=0 out_seq=0 2024-04-01T01:47:15.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.402+0000 7f54d8ff9640 10 client.29543 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:47:15.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.402+0000 7f54d8ff9640 1 -- 172.21.15.70:0/1412970479 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55721801da40 con 0x557218161a40 2024-04-01T01:47:15.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.402+0000 7f54d8ff9640 10 client.29543 handle_command_reply: tid=0 2024-04-01T01:47:15.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.402+0000 7f54fa1b5740 1 client.29543 shutdown 2024-04-01T01:47:15.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.403+0000 7f54fa1b5740 1 -- 172.21.15.70:0/1412970479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 msgr2=0x5572180ed620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.403+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/1412970479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 0x5572180ed620 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f54ec015a60 tx=0x7f54ec003d10 comp rx=0 tx=0).stop 2024-04-01T01:47:15.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.404+0000 7f54fa1b5740 1 -- 172.21.15.70:0/1412970479 shutdown_connections 2024-04-01T01:47:15.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.404+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/1412970479 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x557218161a40 0x557218163e60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.404+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/1412970479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5572180e67b0 0x5572180e9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.404+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/1412970479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5572180b6ff0 0x5572180ed620 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.404+0000 7f54fa1b5740 1 --2- 172.21.15.70:0/1412970479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572180b1490 0x5572180ed0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.404+0000 7f54fa1b5740 1 -- 172.21.15.70:0/1412970479 >> 172.21.15.70:0/1412970479 conn(0x5572180aca00 msgr2=0x5572180ab4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.404+0000 7f54d8ff9640 0 client.29543 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:47:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.404+0000 7f54fa1b5740 1 -- 172.21.15.70:0/1412970479 shutdown_connections 2024-04-01T01:47:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.405+0000 7f54fa1b5740 1 -- 172.21.15.70:0/1412970479 wait complete. 2024-04-01T01:47:15.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.405+0000 7f54fa1b5740 20 client.29543 trim_cache size 0 max 16384 2024-04-01T01:47:15.405 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:47:15.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.405+0000 7f54f74f4640 1 -- 172.21.15.70:0/3129377983 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54dc0a24f0 msgr2=0x7f54dc0a49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.405+0000 7f54f74f4640 1 --2- 172.21.15.70:0/3129377983 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54dc0a24f0 0x7f54dc0a49b0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x5572180b2280 tx=0x7f54ec00c040 comp rx=0 tx=0).stop 2024-04-01T01:47:15.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.405+0000 7f54f74f4640 1 -- 172.21.15.70:0/3129377983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54f010de20 msgr2=0x7f54f01a99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.405+0000 7f54f74f4640 1 --2- 172.21.15.70:0/3129377983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54f010de20 0x7f54f01a99e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f54e8037040 tx=0x7f54e8032820 comp rx=0 tx=0).stop 2024-04-01T01:47:15.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.406+0000 7f54f74f4640 1 -- 172.21.15.70:0/3129377983 shutdown_connections 2024-04-01T01:47:15.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.406+0000 7f54f74f4640 1 --2- 172.21.15.70:0/3129377983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54f010de20 0x7f54f01a99e0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.406+0000 7f54f74f4640 1 --2- 172.21.15.70:0/3129377983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54f010b380 0x7f54f01a2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.406+0000 7f54f74f4640 1 --2- 172.21.15.70:0/3129377983 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54dc0a24f0 0x7f54dc0a49b0 secure :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0x5572180b2280 tx=0x7f54ec00c040 comp rx=0 tx=0).stop 2024-04-01T01:47:15.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.406+0000 7f54f74f4640 1 --2- 172.21.15.70:0/3129377983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54f01089c0 0x7f54f01a2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.406+0000 7f54f74f4640 1 -- 172.21.15.70:0/3129377983 >> 172.21.15.70:0/3129377983 conn(0x7f54f006e5b0 msgr2=0x7f54f0109d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.406+0000 7f54f74f4640 1 -- 172.21.15.70:0/3129377983 shutdown_connections 2024-04-01T01:47:15.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.407+0000 7f54f74f4640 1 -- 172.21.15.70:0/3129377983 wait complete. 2024-04-01T01:47:15.421 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:47:15.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.575+0000 7f2ffea55640 1 -- 172.21.15.70:0/3320445818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff8102930 msgr2=0x7f2ff8104da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.575+0000 7f2ffea55640 1 --2- 172.21.15.70:0/3320445818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff8102930 0x7f2ff8104da0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009900 tx=0x7f2fe002bc30 comp rx=0 tx=0).stop 2024-04-01T01:47:15.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.575+0000 7f2ffea55640 1 -- 172.21.15.70:0/3320445818 shutdown_connections 2024-04-01T01:47:15.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.575+0000 7f2ffea55640 1 --2- 172.21.15.70:0/3320445818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff81052e0 0x7f2ff8105760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.575+0000 7f2ffea55640 1 --2- 172.21.15.70:0/3320445818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff8102930 0x7f2ff8104da0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.575+0000 7f2ffea55640 1 --2- 172.21.15.70:0/3320445818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ff80fffc0 0x7f2ff81023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.575+0000 7f2ffea55640 1 -- 172.21.15.70:0/3320445818 >> 172.21.15.70:0/3320445818 conn(0x7f2ff80fbbe0 msgr2=0x7f2ff80fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.575+0000 7f2ffea55640 1 -- 172.21.15.70:0/3320445818 shutdown_connections 2024-04-01T01:47:15.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.575+0000 7f2ffea55640 1 -- 172.21.15.70:0/3320445818 wait complete. 2024-04-01T01:47:15.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.575+0000 7f2ffea55640 1 Processor -- start 2024-04-01T01:47:15.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.575+0000 7f2ffea55640 1 -- start start 2024-04-01T01:47:15.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2ffea55640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff80fffc0 0x7f2ff819e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2ffea55640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8102930 0x7f2ff819e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2ffea55640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ff81052e0 0x7f2ff81a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2ffea55640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff8113870 con 0x7f2ff80fffc0 2024-04-01T01:47:15.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2ffea55640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff81136f0 con 0x7f2ff81052e0 2024-04-01T01:47:15.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2ffea55640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff81139f0 con 0x7f2ff8102930 2024-04-01T01:47:15.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff80fffc0 0x7f2ff819e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff80fffc0 0x7f2ff819e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44864/0 (socket says 172.21.15.70:44864) 2024-04-01T01:47:15.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2ff7fff640 1 -- 172.21.15.70:0/3355585259 learned_addr learned my addr 172.21.15.70:0/3355585259 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2feffff640 1 --2- 172.21.15.70:0/3355585259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8102930 0x7f2ff819e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2ffcff2640 1 --2- 172.21.15.70:0/3355585259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ff81052e0 0x7f2ff81a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2ff7fff640 1 -- 172.21.15.70:0/3355585259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8102930 msgr2=0x7f2ff819e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2ff7fff640 1 --2- 172.21.15.70:0/3355585259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8102930 0x7f2ff819e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2ff7fff640 1 -- 172.21.15.70:0/3355585259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ff81052e0 msgr2=0x7f2ff81a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2ff7fff640 1 --2- 172.21.15.70:0/3355585259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ff81052e0 0x7f2ff81a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2ff7fff640 1 -- 172.21.15.70:0/3355585259 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff81a5be0 con 0x7f2ff80fffc0 2024-04-01T01:47:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.576+0000 7f2ff7fff640 1 --2- 172.21.15.70:0/3355585259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff80fffc0 0x7f2ff819e010 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800e9f0 tx=0x7f2fe800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.577+0000 7f2ff5ffb640 1 -- 172.21.15.70:0/3355585259 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe800cd80 con 0x7f2ff80fffc0 2024-04-01T01:47:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.577+0000 7f2ff5ffb640 1 -- 172.21.15.70:0/3355585259 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2fe80108e0 con 0x7f2ff80fffc0 2024-04-01T01:47:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.577+0000 7f2ff5ffb640 1 -- 172.21.15.70:0/3355585259 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8018920 con 0x7f2ff80fffc0 2024-04-01T01:47:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.577+0000 7f2ffea55640 1 -- 172.21.15.70:0/3355585259 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff81a5ed0 con 0x7f2ff80fffc0 2024-04-01T01:47:15.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.577+0000 7f2ffea55640 1 -- 172.21.15.70:0/3355585259 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ff81a62e0 con 0x7f2ff80fffc0 2024-04-01T01:47:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.580+0000 7f300163c740 1 Processor -- start 2024-04-01T01:47:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.580+0000 7f300163c740 1 -- start start 2024-04-01T01:47:15.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.581+0000 7f300163c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 0x558f5a40f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.581+0000 7f300163c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a40fe80 0x558f5a414980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.581+0000 7f300163c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a414ff0 0x558f5a417480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.581+0000 7f300163c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558f5a3741b0 con 0x558f5a414ff0 2024-04-01T01:47:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.581+0000 7f300163c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x558f5a31cc00 con 0x558f5a40f490 2024-04-01T01:47:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.581+0000 7f300163c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558f5a370250 con 0x558f5a40fe80 2024-04-01T01:47:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.581+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a40fe80 0x558f5a414980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.581+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a40fe80 0x558f5a414980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57346/0 (socket says 172.21.15.70:57346) 2024-04-01T01:47:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.581+0000 7f2ff7fff640 1 -- 172.21.15.70:0/3607487852 learned_addr learned my addr 172.21.15.70:0/3607487852 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:15.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.581+0000 7f2ffcff2640 1 --2- 172.21.15.70:0/3607487852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 0x558f5a40f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.581+0000 7f2ffcff2640 1 -- 172.21.15.70:0/3607487852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a40fe80 msgr2=0x558f5a414980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.581+0000 7f2ffcff2640 1 --2- 172.21.15.70:0/3607487852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a40fe80 0x558f5a414980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.581+0000 7f2ffcff2640 1 -- 172.21.15.70:0/3607487852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a414ff0 msgr2=0x558f5a417480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.581+0000 7f2ffcff2640 1 --2- 172.21.15.70:0/3607487852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a414ff0 0x558f5a417480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.581+0000 7f2ffcff2640 1 -- 172.21.15.70:0/3607487852 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f5a33c3d0 con 0x558f5a40f490 2024-04-01T01:47:15.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.582+0000 7f2ffcff2640 1 --2- 172.21.15.70:0/3607487852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 0x558f5a40f8b0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009f30 tx=0x7f2ff0010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2da211b29fe82734 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.582+0000 7f2fee7fc640 1 -- 172.21.15.70:0/3607487852 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0016070 con 0x558f5a40f490 2024-04-01T01:47:15.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.582+0000 7f2fee7fc640 1 -- 172.21.15.70:0/3607487852 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ff0004d20 con 0x558f5a40f490 2024-04-01T01:47:15.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.583+0000 7f2fee7fc640 1 -- 172.21.15.70:0/3607487852 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0012780 con 0x558f5a40f490 2024-04-01T01:47:15.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.583+0000 7f300163c740 1 -- 172.21.15.70:0/3607487852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 msgr2=0x558f5a40f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.583+0000 7f300163c740 1 --2- 172.21.15.70:0/3607487852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 0x558f5a40f8b0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009f30 tx=0x7f2ff0010800 comp rx=0 tx=0).stop 2024-04-01T01:47:15.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.583+0000 7f300163c740 1 -- 172.21.15.70:0/3607487852 shutdown_connections 2024-04-01T01:47:15.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.583+0000 7f300163c740 1 --2- 172.21.15.70:0/3607487852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a414ff0 0x558f5a417480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.583+0000 7f300163c740 1 --2- 172.21.15.70:0/3607487852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a40fe80 0x558f5a414980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.583+0000 7f300163c740 1 --2- 172.21.15.70:0/3607487852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 0x558f5a40f8b0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.583+0000 7f300163c740 1 -- 172.21.15.70:0/3607487852 >> 172.21.15.70:0/3607487852 conn(0x558f5a40aa00 msgr2=0x558f5a40ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.584+0000 7f2ff5ffb640 1 -- 172.21.15.70:0/3355585259 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f2fe801f020 con 0x7f2ff80fffc0 2024-04-01T01:47:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.584+0000 7f300163c740 1 -- 172.21.15.70:0/3607487852 shutdown_connections 2024-04-01T01:47:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.585+0000 7f2ff5ffb640 1 --2- 172.21.15.70:0/3355585259 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2fe40a24b0 0x7f2fe40a4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.585+0000 7f300163c740 1 -- 172.21.15.70:0/3607487852 wait complete. 2024-04-01T01:47:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.585+0000 7f2ff5ffb640 1 -- 172.21.15.70:0/3355585259 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f2fe80cf020 con 0x7f2ff80fffc0 2024-04-01T01:47:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.586+0000 7f300163c740 1 -- start start 2024-04-01T01:47:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.586+0000 7f2feffff640 1 --2- 172.21.15.70:0/3355585259 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2fe40a24b0 0x7f2fe40a4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.586+0000 7f300163c740 1 Processor -- start 2024-04-01T01:47:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.586+0000 7f300163c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a40f490 0x558f5a42a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.586+0000 7f300163c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a40fe80 0x558f5a42aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.586+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a40f490 0x558f5a42a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.586+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a40f490 0x558f5a42a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:57354/0 (socket says 172.21.15.70:57354) 2024-04-01T01:47:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.586+0000 7f300163c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a414ff0 0x558f5a431f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.586+0000 7f300163c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558f5a429050 con 0x558f5a40fe80 2024-04-01T01:47:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.586+0000 7f300163c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x558f5a312470 con 0x558f5a414ff0 2024-04-01T01:47:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.586+0000 7f300163c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558f5a4291d0 con 0x558f5a40f490 2024-04-01T01:47:15.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.586+0000 7f300163c740 10 client.0 _ll_register_callbacks cb 0x558f5a32fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.587+0000 7f2ffcff2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a414ff0 0x558f5a431f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.587+0000 7f2ff7fff640 1 -- 172.21.15.70:0/984218345 learned_addr learned my addr 172.21.15.70:0/984218345 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.587+0000 7f2feffff640 1 --2- 172.21.15.70:0/984218345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a40fe80 0x558f5a42aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.587+0000 7f2feffff640 1 --2- 172.21.15.70:0/3355585259 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2fe40a24b0 0x7f2fe40a4970 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009230 tx=0x7f2fe0005b90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.587+0000 7f2ffcff2640 1 -- 172.21.15.70:0/984218345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a40f490 msgr2=0x558f5a42a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.587+0000 7f2ffcff2640 1 --2- 172.21.15.70:0/984218345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a40f490 0x558f5a42a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.587+0000 7f2ffcff2640 1 -- 172.21.15.70:0/984218345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a40fe80 msgr2=0x558f5a42aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.587+0000 7f2ffcff2640 1 --2- 172.21.15.70:0/984218345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a40fe80 0x558f5a42aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.587+0000 7f2ffcff2640 1 -- 172.21.15.70:0/984218345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f5a0883f0 con 0x558f5a414ff0 2024-04-01T01:47:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.587+0000 7f2ffcff2640 1 --2- 172.21.15.70:0/984218345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a414ff0 0x558f5a431f30 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0000f80 tx=0x7f2ff0005100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f42b469a39a744c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.587+0000 7f2fd77fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.588+0000 7f2fd77fe640 1 -- 172.21.15.70:0/984218345 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0016070 con 0x558f5a414ff0 2024-04-01T01:47:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.588+0000 7f2fd77fe640 1 -- 172.21.15.70:0/984218345 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f2ff0005700 con 0x558f5a414ff0 2024-04-01T01:47:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.588+0000 7f300163c740 1 -- 172.21.15.70:0/984218345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ff0004d20 con 0x558f5a414ff0 2024-04-01T01:47:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.588+0000 7f2fd77fe640 1 -- 172.21.15.70:0/984218345 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2ff00124c0 con 0x558f5a414ff0 2024-04-01T01:47:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.588+0000 7f300163c740 10 client.29555 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:15.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.588+0000 7f300163c740 1 -- 172.21.15.70:0/984218345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558f5a1527d0 con 0x558f5a414ff0 2024-04-01T01:47:15.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.588+0000 7f2fd77fe640 1 -- 172.21.15.70:0/984218345 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ff0004d20 con 0x558f5a414ff0 2024-04-01T01:47:15.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.588+0000 7f2fd77fe640 1 -- 172.21.15.70:0/984218345 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f2ff000f040 con 0x558f5a414ff0 2024-04-01T01:47:15.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.589+0000 7f300163c740 10 client.29555 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:15.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.589+0000 7f300163c740 10 client.29555 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:15.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.589+0000 7f300163c740 20 client.29555 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:15.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.589+0000 7f300163c740 1 --2- 172.21.15.70:0/984218345 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558f5a4363e0 0x558f5a438820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.589+0000 7f300163c740 4 client.29555 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:15.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.589+0000 7f300163c740 1 -- 172.21.15.70:0/984218345 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f5a3816f0 con 0x558f5a4363e0 2024-04-01T01:47:15.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.589+0000 7f2ff7fff640 1 --2- 172.21.15.70:0/984218345 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558f5a4363e0 0x558f5a438820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.590+0000 7f2fd77fe640 10 client.29555 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:15.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.590+0000 7f2ff7fff640 1 --2- 172.21.15.70:0/984218345 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558f5a4363e0 0x558f5a438820 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93980feefa1e4b06 server_cookie=195af81ae141e477 in_seq=0 out_seq=0 2024-04-01T01:47:15.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.591+0000 7f2fd77fe640 1 -- 172.21.15.70:0/984218345 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558f5a3816f0 con 0x558f5a4363e0 2024-04-01T01:47:15.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.591+0000 7f2fd77fe640 10 client.29555 handle_command_reply: tid=0 2024-04-01T01:47:15.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.592+0000 7f300163c740 1 client.29555 shutdown 2024-04-01T01:47:15.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.592+0000 7f300163c740 1 -- 172.21.15.70:0/984218345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a414ff0 msgr2=0x558f5a431f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.592+0000 7f300163c740 1 --2- 172.21.15.70:0/984218345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a414ff0 0x558f5a431f30 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0000f80 tx=0x7f2ff0005100 comp rx=0 tx=0).stop 2024-04-01T01:47:15.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.592+0000 7f300163c740 1 -- 172.21.15.70:0/984218345 shutdown_connections 2024-04-01T01:47:15.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.592+0000 7f300163c740 1 --2- 172.21.15.70:0/984218345 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558f5a4363e0 0x558f5a438820 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.592+0000 7f300163c740 1 --2- 172.21.15.70:0/984218345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a414ff0 0x558f5a431f30 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.592+0000 7f300163c740 1 --2- 172.21.15.70:0/984218345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a40fe80 0x558f5a42aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.592+0000 7f300163c740 1 --2- 172.21.15.70:0/984218345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a40f490 0x558f5a42a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.592+0000 7f300163c740 1 -- 172.21.15.70:0/984218345 >> 172.21.15.70:0/984218345 conn(0x558f5a40aa00 msgr2=0x558f5a409d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.592+0000 7f2fd77fe640 0 client.29555 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:15.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.596+0000 7f300163c740 1 -- 172.21.15.70:0/984218345 shutdown_connections 2024-04-01T01:47:15.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.596+0000 7f300163c740 1 -- 172.21.15.70:0/984218345 wait complete. 2024-04-01T01:47:15.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.597+0000 7f300163c740 20 client.29555 trim_cache size 0 max 16384 2024-04-01T01:47:15.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.623+0000 7f300163c740 1 Processor -- start 2024-04-01T01:47:15.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.624+0000 7f300163c740 1 -- start start 2024-04-01T01:47:15.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.625+0000 7f300163c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 0x558f5a4094b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.625+0000 7f300163c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a40fe80 0x558f5a431ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.625+0000 7f300163c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a414ff0 0x558f5a4099f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.625+0000 7f300163c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd800fcd0 con 0x558f5a40fe80 2024-04-01T01:47:15.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.625+0000 7f300163c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd8011810 con 0x558f5a40f490 2024-04-01T01:47:15.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.625+0000 7f300163c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd80103a0 con 0x558f5a414ff0 2024-04-01T01:47:15.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.625+0000 7f2feffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a40fe80 0x558f5a431ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.625+0000 7f2feffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a40fe80 0x558f5a431ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44896/0 (socket says 172.21.15.70:44896) 2024-04-01T01:47:15.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.625+0000 7f2feffff640 1 -- 172.21.15.70:0/2374070797 learned_addr learned my addr 172.21.15.70:0/2374070797 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:15.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.626+0000 7f2feffff640 1 -- 172.21.15.70:0/2374070797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a414ff0 msgr2=0x558f5a4099f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:47:15.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.626+0000 7f2feffff640 1 --2- 172.21.15.70:0/2374070797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a414ff0 0x558f5a4099f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.626+0000 7f2feffff640 1 -- 172.21.15.70:0/2374070797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 msgr2=0x558f5a4094b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:47:15.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.626+0000 7f2feffff640 1 --2- 172.21.15.70:0/2374070797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 0x558f5a4094b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.626+0000 7f2feffff640 1 -- 172.21.15.70:0/2374070797 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f5a40f0d0 con 0x558f5a40fe80 2024-04-01T01:47:15.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.626+0000 7f2feffff640 1 --2- 172.21.15.70:0/2374070797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a40fe80 0x558f5a431ef0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009900 tx=0x7f2fe003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7cec5e271434f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.627+0000 7f2fedffb640 1 -- 172.21.15.70:0/2374070797 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0052070 con 0x558f5a40fe80 2024-04-01T01:47:15.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.627+0000 7f2fedffb640 1 -- 172.21.15.70:0/2374070797 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2fe00046c0 con 0x558f5a40fe80 2024-04-01T01:47:15.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.627+0000 7f2fedffb640 1 -- 172.21.15.70:0/2374070797 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0047040 con 0x558f5a40fe80 2024-04-01T01:47:15.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.628+0000 7f300163c740 1 -- 172.21.15.70:0/2374070797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a40fe80 msgr2=0x558f5a431ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.628+0000 7f300163c740 1 --2- 172.21.15.70:0/2374070797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a40fe80 0x558f5a431ef0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009900 tx=0x7f2fe003e000 comp rx=0 tx=0).stop 2024-04-01T01:47:15.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.628+0000 7f300163c740 1 -- 172.21.15.70:0/2374070797 shutdown_connections 2024-04-01T01:47:15.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.628+0000 7f300163c740 1 --2- 172.21.15.70:0/2374070797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a414ff0 0x558f5a4099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.628+0000 7f300163c740 1 --2- 172.21.15.70:0/2374070797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a40fe80 0x558f5a431ef0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.628+0000 7f300163c740 1 --2- 172.21.15.70:0/2374070797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 0x558f5a4094b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.628+0000 7f300163c740 1 -- 172.21.15.70:0/2374070797 >> 172.21.15.70:0/2374070797 conn(0x558f5a40aa00 msgr2=0x558f5a42a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.628+0000 7f300163c740 1 -- 172.21.15.70:0/2374070797 shutdown_connections 2024-04-01T01:47:15.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.628+0000 7f300163c740 1 -- 172.21.15.70:0/2374070797 wait complete. 2024-04-01T01:47:15.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.629+0000 7f300163c740 1 -- start start 2024-04-01T01:47:15.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.629+0000 7f300163c740 1 Processor -- start 2024-04-01T01:47:15.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.629+0000 7f300163c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 0x558f5a42cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.629+0000 7f300163c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a40fe80 0x558f5a42d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.629+0000 7f300163c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a414ff0 0x558f5a4ab720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.629+0000 7f300163c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd8011140 con 0x558f5a414ff0 2024-04-01T01:47:15.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.629+0000 7f300163c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd8010a70 con 0x558f5a40f490 2024-04-01T01:47:15.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.629+0000 7f300163c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd800dcb0 con 0x558f5a40fe80 2024-04-01T01:47:15.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.629+0000 7f300163c740 10 client.0 _ll_register_callbacks cb 0x7f2ff0004a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:15.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.630+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 0x558f5a42cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.630+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 0x558f5a42cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:48094/0 (socket says 172.21.15.70:48094) 2024-04-01T01:47:15.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.630+0000 7f2ff7fff640 1 -- 172.21.15.70:0/4133801014 learned_addr learned my addr 172.21.15.70:0/4133801014 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:15.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.633+0000 7f2ff7fff640 1 -- 172.21.15.70:0/4133801014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a40fe80 msgr2=0x558f5a42d280 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:47:15.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.633+0000 7f2ff7fff640 1 --2- 172.21.15.70:0/4133801014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a40fe80 0x558f5a42d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.633+0000 7f2ff7fff640 1 -- 172.21.15.70:0/4133801014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a414ff0 msgr2=0x558f5a4ab720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:15.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.633+0000 7f2ff7fff640 1 --2- 172.21.15.70:0/4133801014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a414ff0 0x558f5a4ab720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.633+0000 7f2ff7fff640 1 -- 172.21.15.70:0/4133801014 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f5a4293f0 con 0x558f5a40f490 2024-04-01T01:47:15.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.634+0000 7f2ff7fff640 1 --2- 172.21.15.70:0/4133801014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 0x558f5a42cd40 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800c6a0 tx=0x7f2fe809b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e88e1d5d721f1635 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:15.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.634+0000 7f2fd77fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:15.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.634+0000 7f2fd77fe640 1 -- 172.21.15.70:0/4133801014 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8099a30 con 0x558f5a40f490 2024-04-01T01:47:15.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.634+0000 7f2fd77fe640 1 -- 172.21.15.70:0/4133801014 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f2fe80cc800 con 0x558f5a40f490 2024-04-01T01:47:15.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.634+0000 7f300163c740 1 -- 172.21.15.70:0/4133801014 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fe00046c0 con 0x558f5a40f490 2024-04-01T01:47:15.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.639+0000 7f2fd77fe640 1 -- 172.21.15.70:0/4133801014 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2fe8097da0 con 0x558f5a40f490 2024-04-01T01:47:15.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.639+0000 7f300163c740 10 client.29558 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:15.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.639+0000 7f300163c740 1 -- 172.21.15.70:0/4133801014 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558f5a4acd10 con 0x558f5a40f490 2024-04-01T01:47:15.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.640+0000 7f2fd77fe640 1 -- 172.21.15.70:0/4133801014 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2fe00046c0 con 0x558f5a40f490 2024-04-01T01:47:15.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.640+0000 7f2fd77fe640 1 -- 172.21.15.70:0/4133801014 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f2fe80cccc0 con 0x558f5a40f490 2024-04-01T01:47:15.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.640+0000 7f300163c740 10 client.29558 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:15.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.640+0000 7f300163c740 10 client.29558 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:15.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.640+0000 7f300163c740 20 client.29558 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:15.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.640+0000 7f300163c740 1 --2- 172.21.15.70:0/4133801014 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558f5a4ad6a0 0x558f5a4afac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:15.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.640+0000 7f300163c740 4 client.29558 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:47:15.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.640+0000 7f300163c740 1 -- 172.21.15.70:0/4133801014 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558f5a37ba40 con 0x558f5a4ad6a0 2024-04-01T01:47:15.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.641+0000 7f2feffff640 1 --2- 172.21.15.70:0/4133801014 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558f5a4ad6a0 0x558f5a4afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:15.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.642+0000 7f2feffff640 1 --2- 172.21.15.70:0/4133801014 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558f5a4ad6a0 0x558f5a4afac0 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=170e0dc65b1ce82e server_cookie=9b9fe5ab4ec19140 in_seq=0 out_seq=0 2024-04-01T01:47:15.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.642+0000 7f2fd77fe640 10 client.29558 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:15.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.644+0000 7f2fd77fe640 1 -- 172.21.15.70:0/4133801014 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558f5a37ba40 con 0x558f5a4ad6a0 2024-04-01T01:47:15.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.644+0000 7f2fd77fe640 10 client.29558 handle_command_reply: tid=0 2024-04-01T01:47:15.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.645+0000 7f300163c740 1 client.29558 shutdown 2024-04-01T01:47:15.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.647+0000 7f300163c740 1 -- 172.21.15.70:0/4133801014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 msgr2=0x558f5a42cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.647+0000 7f300163c740 1 --2- 172.21.15.70:0/4133801014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 0x558f5a42cd40 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800c6a0 tx=0x7f2fe809b330 comp rx=0 tx=0).stop 2024-04-01T01:47:15.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.647+0000 7f300163c740 1 -- 172.21.15.70:0/4133801014 shutdown_connections 2024-04-01T01:47:15.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.647+0000 7f300163c740 1 --2- 172.21.15.70:0/4133801014 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558f5a4ad6a0 0x558f5a4afac0 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.647+0000 7f300163c740 1 --2- 172.21.15.70:0/4133801014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558f5a414ff0 0x558f5a4ab720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.647+0000 7f300163c740 1 --2- 172.21.15.70:0/4133801014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558f5a40fe80 0x558f5a42d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.647+0000 7f300163c740 1 --2- 172.21.15.70:0/4133801014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558f5a40f490 0x558f5a42cd40 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.647+0000 7f300163c740 1 -- 172.21.15.70:0/4133801014 >> 172.21.15.70:0/4133801014 conn(0x558f5a40aa00 msgr2=0x558f5a40dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.647+0000 7f2fd77fe640 0 client.29558 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:15.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.648+0000 7f300163c740 1 -- 172.21.15.70:0/4133801014 shutdown_connections 2024-04-01T01:47:15.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.648+0000 7f300163c740 1 -- 172.21.15.70:0/4133801014 wait complete. 2024-04-01T01:47:15.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.648+0000 7f300163c740 20 client.29558 trim_cache size 0 max 16384 2024-04-01T01:47:15.649 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:47:15.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.649+0000 7f2ffea55640 1 -- 172.21.15.70:0/3355585259 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2fe40a24b0 msgr2=0x7f2fe40a4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.649+0000 7f2ffea55640 1 --2- 172.21.15.70:0/3355585259 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2fe40a24b0 0x7f2fe40a4970 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009230 tx=0x7f2fe0005b90 comp rx=0 tx=0).stop 2024-04-01T01:47:15.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.649+0000 7f2ffea55640 1 -- 172.21.15.70:0/3355585259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff80fffc0 msgr2=0x7f2ff819e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:15.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.649+0000 7f2ffea55640 1 --2- 172.21.15.70:0/3355585259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff80fffc0 0x7f2ff819e010 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800e9f0 tx=0x7f2fe800eeb0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.651+0000 7f2ffea55640 1 -- 172.21.15.70:0/3355585259 shutdown_connections 2024-04-01T01:47:15.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.651+0000 7f2ffea55640 1 --2- 172.21.15.70:0/3355585259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ff81052e0 0x7f2ff81a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.651+0000 7f2ffea55640 1 --2- 172.21.15.70:0/3355585259 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2fe40a24b0 0x7f2fe40a4970 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.651+0000 7f2ffea55640 1 --2- 172.21.15.70:0/3355585259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8102930 0x7f2ff819e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.651+0000 7f2ffea55640 1 --2- 172.21.15.70:0/3355585259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ff80fffc0 0x7f2ff819e010 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:15.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.651+0000 7f2ffea55640 1 -- 172.21.15.70:0/3355585259 >> 172.21.15.70:0/3355585259 conn(0x7f2ff80fbbe0 msgr2=0x7f2ff8100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:15.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.651+0000 7f2ffea55640 1 -- 172.21.15.70:0/3355585259 shutdown_connections 2024-04-01T01:47:15.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:15.651+0000 7f2ffea55640 1 -- 172.21.15.70:0/3355585259 wait complete. 2024-04-01T01:47:16.672 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:47:16.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:47:16.673 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:47:16.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.853+0000 7efe69be9640 1 -- 172.21.15.70:0/1089352302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe6410da70 msgr2=0x7efe6410def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:16.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.853+0000 7efe69be9640 1 --2- 172.21.15.70:0/1089352302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe6410da70 0x7efe6410def0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7efe5c009900 tx=0x7efe5c02bc80 comp rx=0 tx=0).stop 2024-04-01T01:47:16.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.854+0000 7efe69be9640 1 -- 172.21.15.70:0/1089352302 shutdown_connections 2024-04-01T01:47:16.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.854+0000 7efe69be9640 1 --2- 172.21.15.70:0/1089352302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe6410da70 0x7efe6410def0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.854+0000 7efe69be9640 1 --2- 172.21.15.70:0/1089352302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe6410b0c0 0x7efe6410d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.854+0000 7efe69be9640 1 --2- 172.21.15.70:0/1089352302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe64108750 0x7efe6410ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.854+0000 7efe69be9640 1 -- 172.21.15.70:0/1089352302 >> 172.21.15.70:0/1089352302 conn(0x7efe6406e5b0 msgr2=0x7efe640709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:16.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.856+0000 7efe69be9640 1 -- 172.21.15.70:0/1089352302 shutdown_connections 2024-04-01T01:47:16.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.856+0000 7efe69be9640 1 -- 172.21.15.70:0/1089352302 wait complete. 2024-04-01T01:47:16.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.857+0000 7efe69be9640 1 Processor -- start 2024-04-01T01:47:16.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.857+0000 7efe69be9640 1 -- start start 2024-04-01T01:47:16.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.858+0000 7efe69be9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe64108750 0x7efe64114f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.858+0000 7efe69be9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe6410b0c0 0x7efe641154a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.858+0000 7efe69be9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe6410da70 0x7efe641b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.858+0000 7efe69be9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efe6411c190 con 0x7efe64108750 2024-04-01T01:47:16.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.858+0000 7efe69be9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efe6411c010 con 0x7efe6410b0c0 2024-04-01T01:47:16.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.858+0000 7efe69be9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efe6411c310 con 0x7efe6410da70 2024-04-01T01:47:16.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.858+0000 7efe62ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe6410b0c0 0x7efe641154a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:16.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.858+0000 7efe63fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe6410da70 0x7efe641b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:16.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.858+0000 7efe62ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe6410b0c0 0x7efe641154a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:51410/0 (socket says 172.21.15.70:51410) 2024-04-01T01:47:16.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.858+0000 7efe62ffd640 1 -- 172.21.15.70:0/1867870923 learned_addr learned my addr 172.21.15.70:0/1867870923 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:16.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.858+0000 7efe637fe640 1 --2- 172.21.15.70:0/1867870923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe64108750 0x7efe64114f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:16.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.859+0000 7efe62ffd640 1 -- 172.21.15.70:0/1867870923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe6410da70 msgr2=0x7efe641b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:16.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.859+0000 7efe62ffd640 1 --2- 172.21.15.70:0/1867870923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe6410da70 0x7efe641b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.859+0000 7efe62ffd640 1 -- 172.21.15.70:0/1867870923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe64108750 msgr2=0x7efe64114f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:16.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.859+0000 7efe62ffd640 1 --2- 172.21.15.70:0/1867870923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe64108750 0x7efe64114f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.859+0000 7efe62ffd640 1 -- 172.21.15.70:0/1867870923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe641b6de0 con 0x7efe6410b0c0 2024-04-01T01:47:16.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.859+0000 7efe62ffd640 1 --2- 172.21.15.70:0/1867870923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe6410b0c0 0x7efe641154a0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7efe5800b450 tx=0x7efe5800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:16.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.860+0000 7efe60ff9640 1 -- 172.21.15.70:0/1867870923 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe58015020 con 0x7efe6410b0c0 2024-04-01T01:47:16.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.860+0000 7efe60ff9640 1 -- 172.21.15.70:0/1867870923 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe58011d40 con 0x7efe6410b0c0 2024-04-01T01:47:16.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.860+0000 7efe60ff9640 1 -- 172.21.15.70:0/1867870923 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5800f340 con 0x7efe6410b0c0 2024-04-01T01:47:16.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.860+0000 7efe69be9640 1 -- 172.21.15.70:0/1867870923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe641b7070 con 0x7efe6410b0c0 2024-04-01T01:47:16.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.860+0000 7efe69be9640 1 -- 172.21.15.70:0/1867870923 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe641b75c0 con 0x7efe6410b0c0 2024-04-01T01:47:16.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.864+0000 7efe60ff9640 1 -- 172.21.15.70:0/1867870923 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7efe58018020 con 0x7efe6410b0c0 2024-04-01T01:47:16.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.864+0000 7efe6c7d3740 1 Processor -- start 2024-04-01T01:47:16.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.865+0000 7efe60ff9640 1 --2- 172.21.15.70:0/1867870923 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efe4c0a2570 0x7efe4c0a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.865+0000 7efe60ff9640 1 -- 172.21.15.70:0/1867870923 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7efe580ce6e0 con 0x7efe6410b0c0 2024-04-01T01:47:16.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.866+0000 7efe6c7d3740 1 -- start start 2024-04-01T01:47:16.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.866+0000 7efe63fff640 1 --2- 172.21.15.70:0/1867870923 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efe4c0a2570 0x7efe4c0a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:16.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.866+0000 7efe6c7d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b6a10 0x55f8f29b6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.866+0000 7efe6c7d3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29b7400 0x55f8f29bbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.866+0000 7efe6c7d3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29bc570 0x55f8f29bea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.866+0000 7efe6c7d3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f8f28b80b0 con 0x55f8f29bc570 2024-04-01T01:47:16.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.866+0000 7efe6c7d3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f8f291aea0 con 0x55f8f29b7400 2024-04-01T01:47:16.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.866+0000 7efe6c7d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f8f28c3c00 con 0x55f8f29b6a10 2024-04-01T01:47:16.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.866+0000 7efe62ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29b7400 0x55f8f29bbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:16.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.866+0000 7efe62ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29b7400 0x55f8f29bbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:51420/0 (socket says 172.21.15.70:51420) 2024-04-01T01:47:16.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.866+0000 7efe62ffd640 1 -- 172.21.15.70:0/2934385870 learned_addr learned my addr 172.21.15.70:0/2934385870 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:16.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.866+0000 7efe637fe640 1 --2- 172.21.15.70:0/2934385870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b6a10 0x55f8f29b6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:16.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.867+0000 7efe62ffd640 1 -- 172.21.15.70:0/2934385870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b6a10 msgr2=0x55f8f29b6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:16.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.867+0000 7efe62ffd640 1 --2- 172.21.15.70:0/2934385870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b6a10 0x55f8f29b6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.867+0000 7efe62ffd640 1 -- 172.21.15.70:0/2934385870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29bc570 msgr2=0x55f8f29bea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:16.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.867+0000 7efe62ffd640 1 --2- 172.21.15.70:0/2934385870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29bc570 0x55f8f29bea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.867+0000 7efe62ffd640 1 -- 172.21.15.70:0/2934385870 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8f26f77e0 con 0x55f8f29b7400 2024-04-01T01:47:16.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.868+0000 7efe63fff640 1 --2- 172.21.15.70:0/1867870923 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efe4c0a2570 0x7efe4c0a4a30 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7efe5c036040 tx=0x7efe5c02cb60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:16.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.868+0000 7efe62ffd640 1 --2- 172.21.15.70:0/2934385870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29b7400 0x55f8f29bbf00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7efe580971d0 tx=0x7efe5809a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=794a98613b8bb558 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:16.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.869+0000 7efe49ffb640 1 -- 172.21.15.70:0/2934385870 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5809ad70 con 0x55f8f29b7400 2024-04-01T01:47:16.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.869+0000 7efe49ffb640 1 -- 172.21.15.70:0/2934385870 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe580b3030 con 0x55f8f29b7400 2024-04-01T01:47:16.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.869+0000 7efe49ffb640 1 -- 172.21.15.70:0/2934385870 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe580973e0 con 0x55f8f29b7400 2024-04-01T01:47:16.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.870+0000 7efe6c7d3740 1 -- 172.21.15.70:0/2934385870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29b7400 msgr2=0x55f8f29bbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:16.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.870+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/2934385870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29b7400 0x55f8f29bbf00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7efe580971d0 tx=0x7efe5809a6d0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.871+0000 7efe6c7d3740 1 -- 172.21.15.70:0/2934385870 shutdown_connections 2024-04-01T01:47:16.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.871+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/2934385870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29bc570 0x55f8f29bea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.871+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/2934385870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29b7400 0x55f8f29bbf00 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.871+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/2934385870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b6a10 0x55f8f29b6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.871+0000 7efe6c7d3740 1 -- 172.21.15.70:0/2934385870 >> 172.21.15.70:0/2934385870 conn(0x55f8f29b1f80 msgr2=0x55f8f29b2380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:16.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.871+0000 7efe6c7d3740 1 -- 172.21.15.70:0/2934385870 shutdown_connections 2024-04-01T01:47:16.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.874+0000 7efe6c7d3740 1 -- 172.21.15.70:0/2934385870 wait complete. 2024-04-01T01:47:16.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.874+0000 7efe6c7d3740 1 -- start start 2024-04-01T01:47:16.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.874+0000 7efe6c7d3740 1 Processor -- start 2024-04-01T01:47:16.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.875+0000 7efe6c7d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b6a10 0x55f8f29c0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.875+0000 7efe6c7d3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29b7400 0x55f8f29c11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.875+0000 7efe6c7d3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29bc570 0x55f8f29c81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.875+0000 7efe6c7d3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f8f28b93d0 con 0x55f8f29b7400 2024-04-01T01:47:16.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.875+0000 7efe6c7d3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f8f2916f40 con 0x55f8f29bc570 2024-04-01T01:47:16.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.875+0000 7efe6c7d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f8f29bf510 con 0x55f8f29b6a10 2024-04-01T01:47:16.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.875+0000 7efe6c7d3740 10 client.0 _ll_register_callbacks cb 0x55f8f28d6b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:16.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.876+0000 7efe62ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29b7400 0x55f8f29c11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:16.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.876+0000 7efe62ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29b7400 0x55f8f29c11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43222/0 (socket says 172.21.15.70:43222) 2024-04-01T01:47:16.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.876+0000 7efe62ffd640 1 -- 172.21.15.70:0/3128165628 learned_addr learned my addr 172.21.15.70:0/3128165628 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:16.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.876+0000 7efe62ffd640 1 -- 172.21.15.70:0/3128165628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b6a10 msgr2=0x55f8f29c0c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:47:16.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.876+0000 7efe62ffd640 1 --2- 172.21.15.70:0/3128165628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b6a10 0x55f8f29c0c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.876+0000 7efe62ffd640 1 -- 172.21.15.70:0/3128165628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29bc570 msgr2=0x55f8f29c81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:16.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.876+0000 7efe62ffd640 1 --2- 172.21.15.70:0/3128165628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29bc570 0x55f8f29c81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.876+0000 7efe62ffd640 1 -- 172.21.15.70:0/3128165628 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8f2757f70 con 0x55f8f29b7400 2024-04-01T01:47:16.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.876+0000 7efe62ffd640 1 --2- 172.21.15.70:0/3128165628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29b7400 0x55f8f29c11c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7efe58097740 tx=0x7efe580ce5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4f3063e8a75868c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:16.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.879+0000 7efe42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:47:16.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.879+0000 7efe42ffd640 1 -- 172.21.15.70:0/3128165628 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe580a3780 con 0x55f8f29b7400 2024-04-01T01:47:16.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.879+0000 7efe6c7d3740 1 -- 172.21.15.70:0/3128165628 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe580b3030 con 0x55f8f29b7400 2024-04-01T01:47:16.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.879+0000 7efe42ffd640 1 -- 172.21.15.70:0/3128165628 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7efe580b9300 con 0x55f8f29b7400 2024-04-01T01:47:16.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.880+0000 7efe42ffd640 1 -- 172.21.15.70:0/3128165628 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efe580a8050 con 0x55f8f29b7400 2024-04-01T01:47:16.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.881+0000 7efe6c7d3740 10 client.19488 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:16.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.881+0000 7efe6c7d3740 1 -- 172.21.15.70:0/3128165628 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f8f25ef170 con 0x55f8f29b7400 2024-04-01T01:47:16.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.881+0000 7efe42ffd640 1 -- 172.21.15.70:0/3128165628 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe580b9500 con 0x55f8f29b7400 2024-04-01T01:47:16.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.882+0000 7efe42ffd640 1 -- 172.21.15.70:0/3128165628 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7efe58096ac0 con 0x55f8f29b7400 2024-04-01T01:47:16.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.882+0000 7efe6c7d3740 10 client.19488 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:16.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.882+0000 7efe6c7d3740 10 client.19488 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:16.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.882+0000 7efe6c7d3740 20 client.19488 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:16.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.883+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/3128165628 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f8f29cc810 0x55f8f29cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.883+0000 7efe6c7d3740 4 client.19488 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:16.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.883+0000 7efe6c7d3740 1 -- 172.21.15.70:0/3128165628 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7efe580b3030 con 0x55f8f29cc810 2024-04-01T01:47:16.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.885+0000 7efe637fe640 1 --2- 172.21.15.70:0/3128165628 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f8f29cc810 0x55f8f29cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:16.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.886+0000 7efe637fe640 1 --2- 172.21.15.70:0/3128165628 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f8f29cc810 0x55f8f29cec50 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c0bfefa1ef3fd76 server_cookie=1e8b825b3808e1ef in_seq=0 out_seq=0 2024-04-01T01:47:16.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.887+0000 7efe42ffd640 10 client.19488 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:16.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.888+0000 7efe42ffd640 1 -- 172.21.15.70:0/3128165628 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7efe580b3030 con 0x55f8f29cc810 2024-04-01T01:47:16.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.888+0000 7efe42ffd640 10 client.19488 handle_command_reply: tid=0 2024-04-01T01:47:16.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.888+0000 7efe6c7d3740 1 client.19488 shutdown 2024-04-01T01:47:16.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.891+0000 7efe6c7d3740 1 -- 172.21.15.70:0/3128165628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29b7400 msgr2=0x55f8f29c11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:16.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.891+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/3128165628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29b7400 0x55f8f29c11c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7efe58097740 tx=0x7efe580ce5c0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.891+0000 7efe6c7d3740 1 -- 172.21.15.70:0/3128165628 shutdown_connections 2024-04-01T01:47:16.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.891+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/3128165628 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f8f29cc810 0x55f8f29cec50 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.891+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/3128165628 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29bc570 0x55f8f29c81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.891+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/3128165628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29b7400 0x55f8f29c11c0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.892+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/3128165628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b6a10 0x55f8f29c0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.892+0000 7efe6c7d3740 1 -- 172.21.15.70:0/3128165628 >> 172.21.15.70:0/3128165628 conn(0x55f8f29b1f80 msgr2=0x55f8f29b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:16.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.892+0000 7efe42ffd640 0 client.19488 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:16.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.893+0000 7efe6c7d3740 1 -- 172.21.15.70:0/3128165628 shutdown_connections 2024-04-01T01:47:16.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.893+0000 7efe6c7d3740 1 -- 172.21.15.70:0/3128165628 wait complete. 2024-04-01T01:47:16.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.894+0000 7efe6c7d3740 20 client.19488 trim_cache size 0 max 16384 2024-04-01T01:47:16.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 750.0 s, 784.2 tps, lat 127.900 ms stddev 139.641 2024-04-01T01:47:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.930+0000 7efe6c7d3740 1 Processor -- start 2024-04-01T01:47:16.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.930+0000 7efe6c7d3740 1 -- start start 2024-04-01T01:47:16.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.931+0000 7efe6c7d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b6a10 0x55f8f29b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.931+0000 7efe6c7d3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29b7400 0x55f8f29c8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.931+0000 7efe6c7d3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29bc570 0x55f8f29b8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.931+0000 7efe6c7d3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efe641c1af0 con 0x55f8f29b7400 2024-04-01T01:47:16.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.931+0000 7efe6c7d3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efe641c3630 con 0x55f8f29bc570 2024-04-01T01:47:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.931+0000 7efe6c7d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efe641c21c0 con 0x55f8f29b6a10 2024-04-01T01:47:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.931+0000 7efe637fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b6a10 0x55f8f29b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.931+0000 7efe637fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b6a10 0x55f8f29b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43646/0 (socket says 172.21.15.70:43646) 2024-04-01T01:47:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.931+0000 7efe637fe640 1 -- 172.21.15.70:0/204643728 learned_addr learned my addr 172.21.15.70:0/204643728 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.931+0000 7efe63fff640 1 --2- 172.21.15.70:0/204643728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29bc570 0x55f8f29b8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.932+0000 7efe637fe640 1 -- 172.21.15.70:0/204643728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29bc570 msgr2=0x55f8f29b8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.932+0000 7efe637fe640 1 --2- 172.21.15.70:0/204643728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29bc570 0x55f8f29b8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.932+0000 7efe637fe640 1 -- 172.21.15.70:0/204643728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29b7400 msgr2=0x55f8f29c8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.932+0000 7efe637fe640 1 --2- 172.21.15.70:0/204643728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29b7400 0x55f8f29c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.932+0000 7efe637fe640 1 -- 172.21.15.70:0/204643728 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8f29b2cf0 con 0x55f8f29b6a10 2024-04-01T01:47:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.932+0000 7efe637fe640 1 --2- 172.21.15.70:0/204643728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b6a10 0x55f8f29b2590 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x55f8f29b7c60 tx=0x7efe54015810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b31e34bd3bbafc33 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.932+0000 7efe497fa640 1 -- 172.21.15.70:0/204643728 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5400d040 con 0x55f8f29b6a10 2024-04-01T01:47:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.933+0000 7efe497fa640 1 -- 172.21.15.70:0/204643728 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe5401c030 con 0x55f8f29b6a10 2024-04-01T01:47:16.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.933+0000 7efe497fa640 1 -- 172.21.15.70:0/204643728 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe54013a90 con 0x55f8f29b6a10 2024-04-01T01:47:16.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.934+0000 7efe6c7d3740 1 -- 172.21.15.70:0/204643728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b6a10 msgr2=0x55f8f29b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:16.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.934+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/204643728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b6a10 0x55f8f29b2590 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x55f8f29b7c60 tx=0x7efe54015810 comp rx=0 tx=0).stop 2024-04-01T01:47:16.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.934+0000 7efe6c7d3740 1 -- 172.21.15.70:0/204643728 shutdown_connections 2024-04-01T01:47:16.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.934+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/204643728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29bc570 0x55f8f29b8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.934+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/204643728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29b7400 0x55f8f29c8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.934+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/204643728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b6a10 0x55f8f29b2590 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.934+0000 7efe6c7d3740 1 -- 172.21.15.70:0/204643728 >> 172.21.15.70:0/204643728 conn(0x55f8f29b1f80 msgr2=0x55f8f29c0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:16.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.934+0000 7efe6c7d3740 1 -- 172.21.15.70:0/204643728 shutdown_connections 2024-04-01T01:47:16.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.934+0000 7efe6c7d3740 1 -- 172.21.15.70:0/204643728 wait complete. 2024-04-01T01:47:16.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.935+0000 7efe6c7d3740 1 -- start start 2024-04-01T01:47:16.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.935+0000 7efe6c7d3740 1 Processor -- start 2024-04-01T01:47:16.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.936+0000 7efe6c7d3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29b6a10 0x55f8f29c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.936+0000 7efe6c7d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b7400 0x55f8f29c5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.936+0000 7efe6c7d3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29bc570 0x55f8f2a30b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.936+0000 7efe6c7d3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efe641c2f60 con 0x55f8f29bc570 2024-04-01T01:47:16.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.936+0000 7efe6c7d3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efe641c2890 con 0x55f8f29b6a10 2024-04-01T01:47:16.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.936+0000 7efe6c7d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efe641bfb80 con 0x55f8f29b7400 2024-04-01T01:47:16.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.936+0000 7efe6c7d3740 10 client.0 _ll_register_callbacks cb 0x7efe58010760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:16.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.936+0000 7efe62ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b7400 0x55f8f29c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:16.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.936+0000 7efe62ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b7400 0x55f8f29c5220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43662/0 (socket says 172.21.15.70:43662) 2024-04-01T01:47:16.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.936+0000 7efe62ffd640 1 -- 172.21.15.70:0/590616897 learned_addr learned my addr 172.21.15.70:0/590616897 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:16.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.936+0000 7efe63fff640 1 --2- 172.21.15.70:0/590616897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29bc570 0x55f8f2a30b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:16.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.936+0000 7efe637fe640 1 --2- 172.21.15.70:0/590616897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29b6a10 0x55f8f29c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:16.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.937+0000 7efe637fe640 1 -- 172.21.15.70:0/590616897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b7400 msgr2=0x55f8f29c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:16.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.937+0000 7efe637fe640 1 --2- 172.21.15.70:0/590616897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b7400 0x55f8f29c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.937+0000 7efe637fe640 1 -- 172.21.15.70:0/590616897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29bc570 msgr2=0x55f8f2a30b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:16.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.937+0000 7efe637fe640 1 --2- 172.21.15.70:0/590616897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29bc570 0x55f8f2a30b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.937+0000 7efe637fe640 1 -- 172.21.15.70:0/590616897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8f29c82f0 con 0x55f8f29b6a10 2024-04-01T01:47:16.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.937+0000 7efe63fff640 1 --2- 172.21.15.70:0/590616897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29bc570 0x55f8f2a30b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:47:16.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.937+0000 7efe637fe640 1 --2- 172.21.15.70:0/590616897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29b6a10 0x55f8f29c4ce0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x55f8f29b7dd0 tx=0x7efe54007ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f64731411ecc4e8b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:16.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.937+0000 7efe42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:16.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.938+0000 7efe42ffd640 1 -- 172.21.15.70:0/590616897 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe54013a20 con 0x55f8f29b6a10 2024-04-01T01:47:16.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.938+0000 7efe42ffd640 1 -- 172.21.15.70:0/590616897 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7efe54013bc0 con 0x55f8f29b6a10 2024-04-01T01:47:16.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.938+0000 7efe6c7d3740 1 -- 172.21.15.70:0/590616897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe5401c030 con 0x55f8f29b6a10 2024-04-01T01:47:16.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.938+0000 7efe42ffd640 1 -- 172.21.15.70:0/590616897 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efe54013de0 con 0x55f8f29b6a10 2024-04-01T01:47:16.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.938+0000 7efe6c7d3740 10 client.29570 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:16.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.938+0000 7efe6c7d3740 1 -- 172.21.15.70:0/590616897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f8f2a320f0 con 0x55f8f29b6a10 2024-04-01T01:47:16.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.939+0000 7efe42ffd640 1 -- 172.21.15.70:0/590616897 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe5401c030 con 0x55f8f29b6a10 2024-04-01T01:47:16.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.939+0000 7efe42ffd640 1 -- 172.21.15.70:0/590616897 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7efe54021770 con 0x55f8f29b6a10 2024-04-01T01:47:16.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.939+0000 7efe6c7d3740 10 client.29570 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:16.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.939+0000 7efe6c7d3740 10 client.29570 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:16.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.939+0000 7efe6c7d3740 20 client.29570 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:16.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.939+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/590616897 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f8f2a32a10 0x55f8f2a34e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:16.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.939+0000 7efe6c7d3740 4 client.29570 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:47:16.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.939+0000 7efe6c7d3740 1 -- 172.21.15.70:0/590616897 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f8f29283e0 con 0x55f8f2a32a10 2024-04-01T01:47:16.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.940+0000 7efe62ffd640 1 --2- 172.21.15.70:0/590616897 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f8f2a32a10 0x55f8f2a34e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:16.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.940+0000 7efe62ffd640 1 --2- 172.21.15.70:0/590616897 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f8f2a32a10 0x55f8f2a34e30 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1d0ccefdf3d2306 server_cookie=a6afc66fa15d22e6 in_seq=0 out_seq=0 2024-04-01T01:47:16.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.941+0000 7efe42ffd640 10 client.29570 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:16.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.942+0000 7efe42ffd640 1 -- 172.21.15.70:0/590616897 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f8f29283e0 con 0x55f8f2a32a10 2024-04-01T01:47:16.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.942+0000 7efe42ffd640 10 client.29570 handle_command_reply: tid=0 2024-04-01T01:47:16.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.942+0000 7efe6c7d3740 1 client.29570 shutdown 2024-04-01T01:47:16.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.944+0000 7efe6c7d3740 1 -- 172.21.15.70:0/590616897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29b6a10 msgr2=0x55f8f29c4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:16.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.944+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/590616897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29b6a10 0x55f8f29c4ce0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x55f8f29b7dd0 tx=0x7efe54007ce0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.946+0000 7efe6c7d3740 1 -- 172.21.15.70:0/590616897 shutdown_connections 2024-04-01T01:47:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.946+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/590616897 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f8f2a32a10 0x55f8f2a34e30 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.946+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/590616897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f8f29bc570 0x55f8f2a30b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.946+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/590616897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8f29b7400 0x55f8f29c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.946+0000 7efe6c7d3740 1 --2- 172.21.15.70:0/590616897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f8f29b6a10 0x55f8f29c4ce0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.946+0000 7efe6c7d3740 1 -- 172.21.15.70:0/590616897 >> 172.21.15.70:0/590616897 conn(0x55f8f29b1f80 msgr2=0x55f8f29b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:16.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.947+0000 7efe42ffd640 0 client.29570 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:16.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.947+0000 7efe6c7d3740 1 -- 172.21.15.70:0/590616897 shutdown_connections 2024-04-01T01:47:16.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.947+0000 7efe6c7d3740 1 -- 172.21.15.70:0/590616897 wait complete. 2024-04-01T01:47:16.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.947+0000 7efe6c7d3740 20 client.29570 trim_cache size 0 max 16384 2024-04-01T01:47:16.949 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:47:16.949 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:47:16.949 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "c08d52e8-71ca-494f-8396-bb5ef70715ac", 2024-04-01T01:47:16.949 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:47:16.949 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:47:16.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.948+0000 7efe69be9640 1 -- 172.21.15.70:0/1867870923 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efe4c0a2570 msgr2=0x7efe4c0a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:16.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.948+0000 7efe69be9640 1 --2- 172.21.15.70:0/1867870923 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efe4c0a2570 0x7efe4c0a4a30 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7efe5c036040 tx=0x7efe5c02cb60 comp rx=0 tx=0).stop 2024-04-01T01:47:16.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.948+0000 7efe69be9640 1 -- 172.21.15.70:0/1867870923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe6410b0c0 msgr2=0x7efe641154a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:16.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.948+0000 7efe69be9640 1 --2- 172.21.15.70:0/1867870923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe6410b0c0 0x7efe641154a0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7efe5800b450 tx=0x7efe5800b910 comp rx=0 tx=0).stop 2024-04-01T01:47:16.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.949+0000 7efe69be9640 1 -- 172.21.15.70:0/1867870923 shutdown_connections 2024-04-01T01:47:16.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.949+0000 7efe69be9640 1 --2- 172.21.15.70:0/1867870923 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7efe4c0a2570 0x7efe4c0a4a30 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.949+0000 7efe69be9640 1 --2- 172.21.15.70:0/1867870923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe6410da70 0x7efe641b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.949+0000 7efe69be9640 1 --2- 172.21.15.70:0/1867870923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe6410b0c0 0x7efe641154a0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.949+0000 7efe69be9640 1 --2- 172.21.15.70:0/1867870923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe64108750 0x7efe64114f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:16.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.949+0000 7efe69be9640 1 -- 172.21.15.70:0/1867870923 >> 172.21.15.70:0/1867870923 conn(0x7efe6406e5b0 msgr2=0x7efe64108b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:16.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.952+0000 7efe69be9640 1 -- 172.21.15.70:0/1867870923 shutdown_connections 2024-04-01T01:47:16.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:16.952+0000 7efe69be9640 1 -- 172.21.15.70:0/1867870923 wait complete. 2024-04-01T01:47:16.982 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:16 smithi070 ceph-mon[32179]: pgmap v772: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.3 MiB/s wr, 710 op/s 2024-04-01T01:47:16.984 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:47:16.984 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:47:17.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.161+0000 7f47a3d0f640 1 -- 172.21.15.70:0/2952967208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f479c101840 msgr2=0x7f479c101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:17.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.161+0000 7f47a3d0f640 1 --2- 172.21.15.70:0/2952967208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f479c101840 0x7f479c101cc0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f478c009be0 tx=0x7f478c02bef0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.162+0000 7f47a3d0f640 1 -- 172.21.15.70:0/2952967208 shutdown_connections 2024-04-01T01:47:17.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.162+0000 7f47a3d0f640 1 --2- 172.21.15.70:0/2952967208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f479c102200 0x7f479c108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.162+0000 7f47a3d0f640 1 --2- 172.21.15.70:0/2952967208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f479c101840 0x7f479c101cc0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.162+0000 7f47a3d0f640 1 --2- 172.21.15.70:0/2952967208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f479c100650 0x7f479c100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.162+0000 7f47a3d0f640 1 -- 172.21.15.70:0/2952967208 >> 172.21.15.70:0/2952967208 conn(0x7f479c0fbbe0 msgr2=0x7f479c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:17.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.166+0000 7f47a3d0f640 1 -- 172.21.15.70:0/2952967208 shutdown_connections 2024-04-01T01:47:17.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.166+0000 7f47a3d0f640 1 -- 172.21.15.70:0/2952967208 wait complete. 2024-04-01T01:47:17.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.167+0000 7f47a3d0f640 1 Processor -- start 2024-04-01T01:47:17.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.167+0000 7f47a3d0f640 1 -- start start 2024-04-01T01:47:17.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.167+0000 7f47a3d0f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f479c100650 0x7f479c0701b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.167+0000 7f47a3d0f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f479c101840 0x7f479c0706f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.167+0000 7f47a3d0f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f479c102200 0x7f479c1a79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.167+0000 7f47a3d0f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f479c10d2e0 con 0x7f479c102200 2024-04-01T01:47:17.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.167+0000 7f47a3d0f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f479c10d160 con 0x7f479c100650 2024-04-01T01:47:17.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.167+0000 7f47a3d0f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f479c10d460 con 0x7f479c101840 2024-04-01T01:47:17.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.167+0000 7f47a1aab640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f479c100650 0x7f479c0701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.167+0000 7f47a22ac640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f479c102200 0x7f479c1a79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.167+0000 7f47a1aab640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f479c100650 0x7f479c0701b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:51466/0 (socket says 172.21.15.70:51466) 2024-04-01T01:47:17.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.167+0000 7f47a1aab640 1 -- 172.21.15.70:0/784672783 learned_addr learned my addr 172.21.15.70:0/784672783 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:17.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.167+0000 7f47a12aa640 1 --2- 172.21.15.70:0/784672783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f479c101840 0x7f479c0706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.168+0000 7f47a12aa640 1 -- 172.21.15.70:0/784672783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f479c100650 msgr2=0x7f479c0701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:17.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.168+0000 7f47a12aa640 1 --2- 172.21.15.70:0/784672783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f479c100650 0x7f479c0701b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.168+0000 7f47a12aa640 1 -- 172.21.15.70:0/784672783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f479c102200 msgr2=0x7f479c1a79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:17.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.168+0000 7f47a12aa640 1 --2- 172.21.15.70:0/784672783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f479c102200 0x7f479c1a79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.168+0000 7f47a12aa640 1 -- 172.21.15.70:0/784672783 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f479c1092d0 con 0x7f479c101840 2024-04-01T01:47:17.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.168+0000 7f47a12aa640 1 --2- 172.21.15.70:0/784672783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f479c101840 0x7f479c0706f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f478c02c430 tx=0x7f478c033cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:17.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.168+0000 7f479affd640 1 -- 172.21.15.70:0/784672783 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c043020 con 0x7f479c101840 2024-04-01T01:47:17.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.169+0000 7f479affd640 1 -- 172.21.15.70:0/784672783 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f478c02ca80 con 0x7f479c101840 2024-04-01T01:47:17.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.169+0000 7f47a3d0f640 1 -- 172.21.15.70:0/784672783 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f479c109560 con 0x7f479c101840 2024-04-01T01:47:17.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.169+0000 7f479affd640 1 -- 172.21.15.70:0/784672783 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c034430 con 0x7f479c101840 2024-04-01T01:47:17.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.169+0000 7f47a3d0f640 1 -- 172.21.15.70:0/784672783 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f479c109ad0 con 0x7f479c101840 2024-04-01T01:47:17.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.172+0000 7f47a69c5740 1 Processor -- start 2024-04-01T01:47:17.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.172+0000 7f479affd640 1 -- 172.21.15.70:0/784672783 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f478c046020 con 0x7f479c101840 2024-04-01T01:47:17.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.172+0000 7f47a69c5740 1 -- start start 2024-04-01T01:47:17.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.173+0000 7f47a69c5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab8512b0 0x55f4ab8516d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.173+0000 7f47a69c5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab851ca0 0x55f4ab8567a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.173+0000 7f47a69c5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab856e10 0x55f4ab8592a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.173+0000 7f47a69c5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f4ab7b6180 con 0x55f4ab8512b0 2024-04-01T01:47:17.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.173+0000 7f47a69c5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f4ab75eb70 con 0x55f4ab851ca0 2024-04-01T01:47:17.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.173+0000 7f47a69c5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f4ab7b2220 con 0x55f4ab856e10 2024-04-01T01:47:17.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.173+0000 7f47a22ac640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab8512b0 0x55f4ab8516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.173+0000 7f47a22ac640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab8512b0 0x55f4ab8516d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43260/0 (socket says 172.21.15.70:43260) 2024-04-01T01:47:17.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.173+0000 7f47a22ac640 1 -- 172.21.15.70:0/1266145744 learned_addr learned my addr 172.21.15.70:0/1266145744 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:17.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.173+0000 7f479affd640 1 --2- 172.21.15.70:0/784672783 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f47880a2520 0x7f47880a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.174+0000 7f479affd640 1 -- 172.21.15.70:0/784672783 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f478c0f11d0 con 0x7f479c101840 2024-04-01T01:47:17.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.174+0000 7f47a1aab640 1 --2- 172.21.15.70:0/1266145744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab851ca0 0x55f4ab8567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.174+0000 7f47a22ac640 1 --2- 172.21.15.70:0/784672783 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f47880a2520 0x7f47880a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.175+0000 7f47a22ac640 1 -- 172.21.15.70:0/1266145744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab856e10 msgr2=0x55f4ab8592a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:47:17.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.175+0000 7f47a22ac640 1 --2- 172.21.15.70:0/1266145744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab856e10 0x55f4ab8592a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.175+0000 7f47a22ac640 1 -- 172.21.15.70:0/1266145744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab851ca0 msgr2=0x55f4ab8567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:17.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.175+0000 7f47a22ac640 1 --2- 172.21.15.70:0/1266145744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab851ca0 0x55f4ab8567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.175+0000 7f47a22ac640 1 -- 172.21.15.70:0/1266145744 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4ab7727c0 con 0x55f4ab8512b0 2024-04-01T01:47:17.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.175+0000 7f47a22ac640 1 --2- 172.21.15.70:0/784672783 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f47880a2520 0x7f47880a49e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f478400fc90 tx=0x7f478401f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:17.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.176+0000 7f47a22ac640 1 --2- 172.21.15.70:0/1266145744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab8512b0 0x55f4ab8516d0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f4784011cb0 tx=0x7f478400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38205e53c9a0fd27 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:17.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.176+0000 7f4793fff640 1 -- 172.21.15.70:0/1266145744 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4784011e60 con 0x55f4ab8512b0 2024-04-01T01:47:17.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.176+0000 7f4793fff640 1 -- 172.21.15.70:0/1266145744 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f478401b040 con 0x55f4ab8512b0 2024-04-01T01:47:17.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.177+0000 7f4793fff640 1 -- 172.21.15.70:0/1266145744 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4784011e60 con 0x55f4ab8512b0 2024-04-01T01:47:17.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.177+0000 7f47a69c5740 1 -- 172.21.15.70:0/1266145744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab8512b0 msgr2=0x55f4ab8516d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:17.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.177+0000 7f47a69c5740 1 --2- 172.21.15.70:0/1266145744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab8512b0 0x55f4ab8516d0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f4784011cb0 tx=0x7f478400f790 comp rx=0 tx=0).stop 2024-04-01T01:47:17.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.177+0000 7f47a69c5740 1 -- 172.21.15.70:0/1266145744 shutdown_connections 2024-04-01T01:47:17.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.177+0000 7f47a69c5740 1 --2- 172.21.15.70:0/1266145744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab856e10 0x55f4ab8592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.178+0000 7f47a69c5740 1 --2- 172.21.15.70:0/1266145744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab851ca0 0x55f4ab8567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.178+0000 7f47a69c5740 1 --2- 172.21.15.70:0/1266145744 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab8512b0 0x55f4ab8516d0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.178+0000 7f47a69c5740 1 -- 172.21.15.70:0/1266145744 >> 172.21.15.70:0/1266145744 conn(0x55f4ab84c820 msgr2=0x55f4ab84cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:17.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.178+0000 7f47a69c5740 1 -- 172.21.15.70:0/1266145744 shutdown_connections 2024-04-01T01:47:17.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.178+0000 7f47a69c5740 1 -- 172.21.15.70:0/1266145744 wait complete. 2024-04-01T01:47:17.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.178+0000 7f47a69c5740 1 -- start start 2024-04-01T01:47:17.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.178+0000 7f47a69c5740 1 Processor -- start 2024-04-01T01:47:17.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.179+0000 7f47a69c5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab8512b0 0x55f4ab85b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.179+0000 7f47a69c5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab851ca0 0x55f4ab85bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.179+0000 7f47a69c5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab856e10 0x55f4ab862dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.179+0000 7f47a69c5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f4ab859d60 con 0x55f4ab856e10 2024-04-01T01:47:17.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.179+0000 7f47a69c5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f4ab7544a0 con 0x55f4ab8512b0 2024-04-01T01:47:17.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.179+0000 7f47a69c5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f4ab859ee0 con 0x55f4ab851ca0 2024-04-01T01:47:17.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.179+0000 7f47a69c5740 10 client.0 _ll_register_callbacks cb 0x55f4ab757d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:17.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.179+0000 7f47a12aa640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab851ca0 0x55f4ab85bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.179+0000 7f47a1aab640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab8512b0 0x55f4ab85b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.179+0000 7f47a1aab640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab8512b0 0x55f4ab85b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:51472/0 (socket says 172.21.15.70:51472) 2024-04-01T01:47:17.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.179+0000 7f47a1aab640 1 -- 172.21.15.70:0/2807530207 learned_addr learned my addr 172.21.15.70:0/2807530207 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:17.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.179+0000 7f47a22ac640 1 --2- 172.21.15.70:0/2807530207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab856e10 0x55f4ab862dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.180+0000 7f47a12aa640 1 -- 172.21.15.70:0/2807530207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab8512b0 msgr2=0x55f4ab85b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:17.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.180+0000 7f47a12aa640 1 --2- 172.21.15.70:0/2807530207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab8512b0 0x55f4ab85b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.180+0000 7f47a12aa640 1 -- 172.21.15.70:0/2807530207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab856e10 msgr2=0x55f4ab862dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:17.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.180+0000 7f47a12aa640 1 --2- 172.21.15.70:0/2807530207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab856e10 0x55f4ab862dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.180+0000 7f47a12aa640 1 -- 172.21.15.70:0/2807530207 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4ab5947d0 con 0x55f4ab851ca0 2024-04-01T01:47:17.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.180+0000 7f47a22ac640 1 --2- 172.21.15.70:0/2807530207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab856e10 0x55f4ab862dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:47:17.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.180+0000 7f47a12aa640 1 --2- 172.21.15.70:0/2807530207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab851ca0 0x55f4ab85bd90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f478c0ee8a0 tx=0x7f478c0bd6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f623c2baf2ffc9ab server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:17.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.180+0000 7f47917fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:47:17.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.181+0000 7f47917fa640 1 -- 172.21.15.70:0/2807530207 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c0bd860 con 0x55f4ab851ca0 2024-04-01T01:47:17.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.181+0000 7f47a69c5740 1 -- 172.21.15.70:0/2807530207 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f478401b040 con 0x55f4ab851ca0 2024-04-01T01:47:17.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.181+0000 7f47917fa640 1 -- 172.21.15.70:0/2807530207 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f478c0d6560 con 0x55f4ab851ca0 2024-04-01T01:47:17.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.181+0000 7f47917fa640 1 -- 172.21.15.70:0/2807530207 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f478c0c19b0 con 0x55f4ab851ca0 2024-04-01T01:47:17.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.181+0000 7f47a69c5740 10 client.29335 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:17.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.181+0000 7f47a69c5740 1 -- 172.21.15.70:0/2807530207 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f4ab4ca3f0 con 0x55f4ab851ca0 2024-04-01T01:47:17.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.182+0000 7f47917fa640 1 -- 172.21.15.70:0/2807530207 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f478401b040 con 0x55f4ab851ca0 2024-04-01T01:47:17.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.182+0000 7f47917fa640 1 -- 172.21.15.70:0/2807530207 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f478c0c5de0 con 0x55f4ab851ca0 2024-04-01T01:47:17.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.182+0000 7f47a69c5740 10 client.29335 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:17.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.182+0000 7f47a69c5740 10 client.29335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:17.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.182+0000 7f47a69c5740 20 client.29335 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:17.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.182+0000 7f47a69c5740 1 --2- 172.21.15.70:0/2807530207 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f4ab8676a0 0x55f4ab869ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.182+0000 7f47a69c5740 4 client.29335 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:17.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.182+0000 7f47a69c5740 1 -- 172.21.15.70:0/2807530207 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4ab7c36c0 con 0x55f4ab8676a0 2024-04-01T01:47:17.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.184+0000 7f47a1aab640 1 --2- 172.21.15.70:0/2807530207 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f4ab8676a0 0x55f4ab869ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.184+0000 7f47a1aab640 1 --2- 172.21.15.70:0/2807530207 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f4ab8676a0 0x55f4ab869ac0 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cfa9e107f5577d4 server_cookie=ced8571cff206a0b in_seq=0 out_seq=0 2024-04-01T01:47:17.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.184+0000 7f47917fa640 10 client.29335 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:17.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.187+0000 7f47917fa640 1 -- 172.21.15.70:0/2807530207 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f4ab7c36c0 con 0x55f4ab8676a0 2024-04-01T01:47:17.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.187+0000 7f47917fa640 10 client.29335 handle_command_reply: tid=0 2024-04-01T01:47:17.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.187+0000 7f47a69c5740 1 client.29335 shutdown 2024-04-01T01:47:17.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.187+0000 7f47a69c5740 1 -- 172.21.15.70:0/2807530207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab851ca0 msgr2=0x55f4ab85bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:17.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.187+0000 7f47a69c5740 1 --2- 172.21.15.70:0/2807530207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab851ca0 0x55f4ab85bd90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f478c0ee8a0 tx=0x7f478c0bd6f0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.188+0000 7f47a69c5740 1 -- 172.21.15.70:0/2807530207 shutdown_connections 2024-04-01T01:47:17.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.188+0000 7f47a69c5740 1 --2- 172.21.15.70:0/2807530207 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f4ab8676a0 0x55f4ab869ac0 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.188+0000 7f47a69c5740 1 --2- 172.21.15.70:0/2807530207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab856e10 0x55f4ab862dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.188+0000 7f47a69c5740 1 --2- 172.21.15.70:0/2807530207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab851ca0 0x55f4ab85bd90 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.188+0000 7f47a69c5740 1 --2- 172.21.15.70:0/2807530207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab8512b0 0x55f4ab85b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.188+0000 7f47a69c5740 1 -- 172.21.15.70:0/2807530207 >> 172.21.15.70:0/2807530207 conn(0x55f4ab84c820 msgr2=0x55f4ab84bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:17.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.189+0000 7f47917fa640 0 client.29335 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:17.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.189+0000 7f47a69c5740 1 -- 172.21.15.70:0/2807530207 shutdown_connections 2024-04-01T01:47:17.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.189+0000 7f47a69c5740 1 -- 172.21.15.70:0/2807530207 wait complete. 2024-04-01T01:47:17.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.189+0000 7f47a69c5740 20 client.29335 trim_cache size 0 max 16384 2024-04-01T01:47:17.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.220+0000 7f47a69c5740 1 Processor -- start 2024-04-01T01:47:17.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.220+0000 7f47a69c5740 1 -- start start 2024-04-01T01:47:17.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.220+0000 7f47a69c5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab8512b0 0x55f4ab8532b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.220+0000 7f47a69c5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab851ca0 0x55f4ab862d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.220+0000 7f47a69c5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab856e10 0x55f4ab8537f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.220+0000 7f47a1aab640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab8512b0 0x55f4ab8532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.220+0000 7f47a1aab640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab8512b0 0x55f4ab8532b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:51478/0 (socket says 172.21.15.70:51478) 2024-04-01T01:47:17.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.220+0000 7f47a1aab640 1 -- 172.21.15.70:0/3361369110 learned_addr learned my addr 172.21.15.70:0/3361369110 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:17.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.220+0000 7f47a69c5740 1 -- 172.21.15.70:0/3361369110 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f479c1b4d70 con 0x55f4ab856e10 2024-04-01T01:47:17.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.220+0000 7f47a69c5740 1 -- 172.21.15.70:0/3361369110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f479c1b68b0 con 0x55f4ab8512b0 2024-04-01T01:47:17.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.220+0000 7f47a69c5740 1 -- 172.21.15.70:0/3361369110 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f479c1b5440 con 0x55f4ab851ca0 2024-04-01T01:47:17.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.220+0000 7f47a12aa640 1 --2- 172.21.15.70:0/3361369110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab851ca0 0x55f4ab862d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.222+0000 7f47a22ac640 1 --2- 172.21.15.70:0/3361369110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab856e10 0x55f4ab8537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.223+0000 7f47a22ac640 1 -- 172.21.15.70:0/3361369110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab851ca0 msgr2=0x55f4ab862d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:17.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.223+0000 7f47a22ac640 1 --2- 172.21.15.70:0/3361369110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab851ca0 0x55f4ab862d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.223+0000 7f47a22ac640 1 -- 172.21.15.70:0/3361369110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab8512b0 msgr2=0x55f4ab8532b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:17.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.223+0000 7f47a22ac640 1 --2- 172.21.15.70:0/3361369110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab8512b0 0x55f4ab8532b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.223+0000 7f47a22ac640 1 -- 172.21.15.70:0/3361369110 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4ab48a0f0 con 0x55f4ab856e10 2024-04-01T01:47:17.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.223+0000 7f47a22ac640 1 --2- 172.21.15.70:0/3361369110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab856e10 0x55f4ab8537f0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f47840099e0 tx=0x7f478400fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e099acc58fd8ba1c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:17.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.224+0000 7f47937fe640 1 -- 172.21.15.70:0/3361369110 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478400ee10 con 0x55f4ab856e10 2024-04-01T01:47:17.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.224+0000 7f47937fe640 1 -- 172.21.15.70:0/3361369110 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f478401b050 con 0x55f4ab856e10 2024-04-01T01:47:17.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.224+0000 7f47937fe640 1 -- 172.21.15.70:0/3361369110 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4784004880 con 0x55f4ab856e10 2024-04-01T01:47:17.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.227+0000 7f47a69c5740 1 -- 172.21.15.70:0/3361369110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab856e10 msgr2=0x55f4ab8537f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:17.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.227+0000 7f47a69c5740 1 --2- 172.21.15.70:0/3361369110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab856e10 0x55f4ab8537f0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f47840099e0 tx=0x7f478400fd40 comp rx=0 tx=0).stop 2024-04-01T01:47:17.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.227+0000 7f47a69c5740 1 -- 172.21.15.70:0/3361369110 shutdown_connections 2024-04-01T01:47:17.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.227+0000 7f47a69c5740 1 --2- 172.21.15.70:0/3361369110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab856e10 0x55f4ab8537f0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.227+0000 7f47a69c5740 1 --2- 172.21.15.70:0/3361369110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab851ca0 0x55f4ab862d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.227+0000 7f47a69c5740 1 --2- 172.21.15.70:0/3361369110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab8512b0 0x55f4ab8532b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.227+0000 7f47a69c5740 1 -- 172.21.15.70:0/3361369110 >> 172.21.15.70:0/3361369110 conn(0x55f4ab84c820 msgr2=0x55f4ab85b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:17.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.227+0000 7f47a69c5740 1 -- 172.21.15.70:0/3361369110 shutdown_connections 2024-04-01T01:47:17.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.227+0000 7f47a69c5740 1 -- 172.21.15.70:0/3361369110 wait complete. 2024-04-01T01:47:17.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.227+0000 7f47a69c5740 1 -- start start 2024-04-01T01:47:17.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.228+0000 7f47a69c5740 1 Processor -- start 2024-04-01T01:47:17.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.228+0000 7f47a69c5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab8512b0 0x55f4ab8602f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.228+0000 7f47a69c5740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab851ca0 0x55f4ab860830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.228+0000 7f47a69c5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab856e10 0x55f4ab861840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.228+0000 7f47a69c5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f479c1b61e0 con 0x55f4ab8512b0 2024-04-01T01:47:17.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.228+0000 7f47a69c5740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f479c1b5b10 con 0x55f4ab851ca0 2024-04-01T01:47:17.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.228+0000 7f47a69c5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f479c1b2d50 con 0x55f4ab856e10 2024-04-01T01:47:17.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.228+0000 7f47a69c5740 10 client.0 _ll_register_callbacks cb 0x7f478c0b9b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:17.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.228+0000 7f47a22ac640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab856e10 0x55f4ab861840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.229+0000 7f47a22ac640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab856e10 0x55f4ab861840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43716/0 (socket says 172.21.15.70:43716) 2024-04-01T01:47:17.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.229+0000 7f47a22ac640 1 -- 172.21.15.70:0/190578081 learned_addr learned my addr 172.21.15.70:0/190578081 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:17.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.229+0000 7f47a12aa640 1 --2- 172.21.15.70:0/190578081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab851ca0 0x55f4ab860830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.229+0000 7f47a1aab640 1 --2- 172.21.15.70:0/190578081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab8512b0 0x55f4ab8602f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.229+0000 7f47a22ac640 1 -- 172.21.15.70:0/190578081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab851ca0 msgr2=0x55f4ab860830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:17.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.229+0000 7f47a22ac640 1 --2- 172.21.15.70:0/190578081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab851ca0 0x55f4ab860830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.229+0000 7f47a22ac640 1 -- 172.21.15.70:0/190578081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab8512b0 msgr2=0x55f4ab8602f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:17.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.229+0000 7f47a22ac640 1 --2- 172.21.15.70:0/190578081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab8512b0 0x55f4ab8602f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.229+0000 7f47a22ac640 1 -- 172.21.15.70:0/190578081 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4ab861e30 con 0x55f4ab856e10 2024-04-01T01:47:17.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.229+0000 7f47a12aa640 1 --2- 172.21.15.70:0/190578081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab851ca0 0x55f4ab860830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:47:17.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.229+0000 7f47a22ac640 1 --2- 172.21.15.70:0/190578081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab856e10 0x55f4ab861840 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4784005ec0 tx=0x7f478400eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=872381793b86ae23 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:17.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.229+0000 7f47917fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:47:17.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.230+0000 7f47917fa640 1 -- 172.21.15.70:0/190578081 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4784011e60 con 0x55f4ab856e10 2024-04-01T01:47:17.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.230+0000 7f47a69c5740 1 -- 172.21.15.70:0/190578081 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f478401b050 con 0x55f4ab856e10 2024-04-01T01:47:17.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.230+0000 7f47917fa640 1 -- 172.21.15.70:0/190578081 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f4784027650 con 0x55f4ab856e10 2024-04-01T01:47:17.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.230+0000 7f47917fa640 1 -- 172.21.15.70:0/190578081 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f47840278b0 con 0x55f4ab856e10 2024-04-01T01:47:17.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.230+0000 7f47a69c5740 10 client.29341 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:17.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.230+0000 7f47a69c5740 1 -- 172.21.15.70:0/190578081 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f4ab8d61d0 con 0x55f4ab856e10 2024-04-01T01:47:17.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.231+0000 7f47917fa640 1 -- 172.21.15.70:0/190578081 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f478401b050 con 0x55f4ab856e10 2024-04-01T01:47:17.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.234+0000 7f47917fa640 1 -- 172.21.15.70:0/190578081 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f4784031b10 con 0x55f4ab856e10 2024-04-01T01:47:17.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.234+0000 7f47a69c5740 10 client.29341 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:17.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.234+0000 7f47a69c5740 10 client.29341 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:17.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.234+0000 7f47a69c5740 20 client.29341 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:17.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.234+0000 7f47a69c5740 1 --2- 172.21.15.70:0/190578081 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f4ab8d67e0 0x55f4ab8d8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:17.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.234+0000 7f47a69c5740 4 client.29341 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:47:17.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.234+0000 7f47a69c5740 1 -- 172.21.15.70:0/190578081 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f4ab7bda10 con 0x55f4ab8d67e0 2024-04-01T01:47:17.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.235+0000 7f47a1aab640 1 --2- 172.21.15.70:0/190578081 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f4ab8d67e0 0x55f4ab8d8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:17.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.236+0000 7f47a1aab640 1 --2- 172.21.15.70:0/190578081 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f4ab8d67e0 0x55f4ab8d8c00 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59b131579979ae0a server_cookie=5585c5c7ace1e223 in_seq=0 out_seq=0 2024-04-01T01:47:17.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.236+0000 7f47917fa640 10 client.29341 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:17.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.237+0000 7f47917fa640 1 -- 172.21.15.70:0/190578081 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55f4ab7bda10 con 0x55f4ab8d67e0 2024-04-01T01:47:17.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.237+0000 7f47917fa640 10 client.29341 handle_command_reply: tid=0 2024-04-01T01:47:17.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.237+0000 7f47a69c5740 1 client.29341 shutdown 2024-04-01T01:47:17.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.237+0000 7f47a69c5740 1 -- 172.21.15.70:0/190578081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab856e10 msgr2=0x55f4ab861840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:17.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.237+0000 7f47a69c5740 1 --2- 172.21.15.70:0/190578081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab856e10 0x55f4ab861840 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4784005ec0 tx=0x7f478400eea0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.237+0000 7f47a69c5740 1 -- 172.21.15.70:0/190578081 shutdown_connections 2024-04-01T01:47:17.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.237+0000 7f47a69c5740 1 --2- 172.21.15.70:0/190578081 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f4ab8d67e0 0x55f4ab8d8c00 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.237+0000 7f47a69c5740 1 --2- 172.21.15.70:0/190578081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4ab856e10 0x55f4ab861840 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.237+0000 7f47a69c5740 1 --2- 172.21.15.70:0/190578081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f4ab851ca0 0x55f4ab860830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.237+0000 7f47a69c5740 1 --2- 172.21.15.70:0/190578081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f4ab8512b0 0x55f4ab8602f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.237+0000 7f47a69c5740 1 -- 172.21.15.70:0/190578081 >> 172.21.15.70:0/190578081 conn(0x55f4ab84c820 msgr2=0x55f4ab85b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:17.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.237+0000 7f47917fa640 0 client.29341 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:17.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.237+0000 7f47a69c5740 1 -- 172.21.15.70:0/190578081 shutdown_connections 2024-04-01T01:47:17.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.237+0000 7f47a69c5740 1 -- 172.21.15.70:0/190578081 wait complete. 2024-04-01T01:47:17.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.237+0000 7f47a69c5740 20 client.29341 trim_cache size 0 max 16384 2024-04-01T01:47:17.239 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:47:17.239 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "scrub active (169 inodes in the stack)", 2024-04-01T01:47:17.239 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:47:17.240 INFO:teuthology.orchestra.run.smithi070.stdout: "c08d52e8-71ca-494f-8396-bb5ef70715ac": { 2024-04-01T01:47:17.240 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:47:17.240 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "c08d52e8-71ca-494f-8396-bb5ef70715ac", 2024-04-01T01:47:17.240 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:47:17.240 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:47:17.240 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:47:17.240 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:47:17.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.238+0000 7f47a3d0f640 1 -- 172.21.15.70:0/784672783 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f47880a2520 msgr2=0x7f47880a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:17.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.238+0000 7f47a3d0f640 1 --2- 172.21.15.70:0/784672783 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f47880a2520 0x7f47880a49e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f478400fc90 tx=0x7f478401f000 comp rx=0 tx=0).stop 2024-04-01T01:47:17.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.238+0000 7f47a3d0f640 1 -- 172.21.15.70:0/784672783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f479c101840 msgr2=0x7f479c0706f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:17.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.238+0000 7f47a3d0f640 1 --2- 172.21.15.70:0/784672783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f479c101840 0x7f479c0706f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f478c02c430 tx=0x7f478c033cc0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.238+0000 7f47a3d0f640 1 -- 172.21.15.70:0/784672783 shutdown_connections 2024-04-01T01:47:17.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.238+0000 7f47a3d0f640 1 --2- 172.21.15.70:0/784672783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f479c102200 0x7f479c1a79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.238+0000 7f47a3d0f640 1 --2- 172.21.15.70:0/784672783 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f47880a2520 0x7f47880a49e0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.238+0000 7f47a3d0f640 1 --2- 172.21.15.70:0/784672783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f479c101840 0x7f479c0706f0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.238+0000 7f47a3d0f640 1 --2- 172.21.15.70:0/784672783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f479c100650 0x7f479c0701b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:17.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.238+0000 7f47a3d0f640 1 -- 172.21.15.70:0/784672783 >> 172.21.15.70:0/784672783 conn(0x7f479c0fbbe0 msgr2=0x7f479c104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:17.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.238+0000 7f47a3d0f640 1 -- 172.21.15.70:0/784672783 shutdown_connections 2024-04-01T01:47:17.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:17.238+0000 7f47a3d0f640 1 -- 172.21.15.70:0/784672783 wait complete. 2024-04-01T01:47:17.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:16 smithi189 ceph-mon[35753]: pgmap v772: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.3 MiB/s wr, 710 op/s 2024-04-01T01:47:17.256 INFO:tasks.cephfs.filesystem:scrub status for tag:c08d52e8-71ca-494f-8396-bb5ef70715ac - {'path': '/', 'tag': 'c08d52e8-71ca-494f-8396-bb5ef70715ac', 'options': 'recursive,force'} 2024-04-01T01:47:17.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:16 smithi081 ceph-mon[35142]: pgmap v772: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 6.3 MiB/s wr, 710 op/s 2024-04-01T01:47:18.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:17 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:47:18.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:17 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:47:18.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:17 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:47:18.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:17 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:47:18.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:17 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:47:18.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:17 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:47:18.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:17 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:47:18.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:17 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:47:18.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:17 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:47:19.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:18 smithi189 ceph-mon[35753]: pgmap v773: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.2 MiB/s wr, 775 op/s 2024-04-01T01:47:19.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:18 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:47:19.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:18 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:47:19.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:18 smithi070 ceph-mon[32179]: pgmap v773: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.2 MiB/s wr, 775 op/s 2024-04-01T01:47:19.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:18 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:47:19.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:18 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:47:19.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:18 smithi081 ceph-mon[35142]: pgmap v773: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.2 MiB/s wr, 775 op/s 2024-04-01T01:47:19.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:18 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:47:19.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:18 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:47:20.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:47:20.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:47:20.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:19 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:47:20.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:47:20.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:47:20.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:19 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:47:20.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:47:20.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:47:20.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:19 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:47:21.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:20 smithi189 ceph-mon[35753]: pgmap v774: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 7.1 MiB/s wr, 792 op/s 2024-04-01T01:47:21.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:20 smithi070 ceph-mon[32179]: pgmap v774: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 7.1 MiB/s wr, 792 op/s 2024-04-01T01:47:21.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:20 smithi081 ceph-mon[35142]: pgmap v774: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 7.1 MiB/s wr, 792 op/s 2024-04-01T01:47:21.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 755.0 s, 707.8 tps, lat 140.253 ms stddev 141.490 2024-04-01T01:47:22.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:21 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:47:22.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:21 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:47:22.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:21 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:47:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.096+0000 7f62dbfff640 1 -- 172.21.15.70:0/1459626870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62dc073360 msgr2=0x7f62dc108d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.096+0000 7f62dbfff640 1 --2- 172.21.15.70:0/1459626870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62dc073360 0x7f62dc108d80 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009ca0 tx=0x7f62d402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.097+0000 7f62dbfff640 1 -- 172.21.15.70:0/1459626870 shutdown_connections 2024-04-01T01:47:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.097+0000 7f62dbfff640 1 --2- 172.21.15.70:0/1459626870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62dc109440 0x7f62dc10e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.097+0000 7f62dbfff640 1 --2- 172.21.15.70:0/1459626870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62dc073360 0x7f62dc108d80 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.097+0000 7f62dbfff640 1 --2- 172.21.15.70:0/1459626870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc072a00 0x7f62dc072e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.097+0000 7f62dbfff640 1 -- 172.21.15.70:0/1459626870 >> 172.21.15.70:0/1459626870 conn(0x7f62dc06e5c0 msgr2=0x7f62dc070a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:23.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.097+0000 7f62dbfff640 1 -- 172.21.15.70:0/1459626870 shutdown_connections 2024-04-01T01:47:23.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.097+0000 7f62dbfff640 1 -- 172.21.15.70:0/1459626870 wait complete. 2024-04-01T01:47:23.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.098+0000 7f62dbfff640 1 Processor -- start 2024-04-01T01:47:23.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.098+0000 7f62dbfff640 1 -- start start 2024-04-01T01:47:23.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.099+0000 7f62dbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62dc072a00 0x7f62dc1a6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.099+0000 7f62dbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62dc073360 0x7f62dc1a6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.099+0000 7f62dbfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc109440 0x7f62dc1adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.099+0000 7f62dbfff640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc11bef0 con 0x7f62dc072a00 2024-04-01T01:47:23.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.099+0000 7f62dbfff640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc11bd70 con 0x7f62dc073360 2024-04-01T01:47:23.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.099+0000 7f62dbfff640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc11c070 con 0x7f62dc109440 2024-04-01T01:47:23.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.099+0000 7f62db7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc109440 0x7f62dc1adbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:23.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.099+0000 7f62da7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62dc073360 0x7f62dc1a6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:23.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.099+0000 7f62db7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc109440 0x7f62dc1adbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43732/0 (socket says 172.21.15.70:43732) 2024-04-01T01:47:23.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.099+0000 7f62db7fe640 1 -- 172.21.15.70:0/218271583 learned_addr learned my addr 172.21.15.70:0/218271583 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:23.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.099+0000 7f62da7fc640 1 -- 172.21.15.70:0/218271583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc109440 msgr2=0x7f62dc1adbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:23.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.099+0000 7f62da7fc640 1 --2- 172.21.15.70:0/218271583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc109440 0x7f62dc1adbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.099+0000 7f62da7fc640 1 -- 172.21.15.70:0/218271583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62dc072a00 msgr2=0x7f62dc1a6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:23.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.099+0000 7f62da7fc640 1 --2- 172.21.15.70:0/218271583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62dc072a00 0x7f62dc1a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.099+0000 7f62da7fc640 1 -- 172.21.15.70:0/218271583 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62dc1ae260 con 0x7f62dc073360 2024-04-01T01:47:23.101 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.099+0000 7f62db7fe640 1 --2- 172.21.15.70:0/218271583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc109440 0x7f62dc1adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:47:23.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.100+0000 7f62da7fc640 1 --2- 172.21.15.70:0/218271583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62dc073360 0x7f62dc1a6b80 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f62d4037040 tx=0x7f62d4032820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:23.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.100+0000 7f62e087f640 1 -- 172.21.15.70:0/218271583 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d4041020 con 0x7f62dc073360 2024-04-01T01:47:23.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.100+0000 7f62e087f640 1 -- 172.21.15.70:0/218271583 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62d4003e20 con 0x7f62dc073360 2024-04-01T01:47:23.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.100+0000 7f62dbfff640 1 -- 172.21.15.70:0/218271583 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62dc1ae4f0 con 0x7f62dc073360 2024-04-01T01:47:23.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.100+0000 7f62e087f640 1 -- 172.21.15.70:0/218271583 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d4039040 con 0x7f62dc073360 2024-04-01T01:47:23.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.101+0000 7f62dbfff640 1 -- 172.21.15.70:0/218271583 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62dc1aea00 con 0x7f62dc073360 2024-04-01T01:47:23.106 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.106+0000 7f62e445e740 1 Processor -- start 2024-04-01T01:47:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.106+0000 7f62e087f640 1 -- 172.21.15.70:0/218271583 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f62d4044020 con 0x7f62dc073360 2024-04-01T01:47:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.107+0000 7f62e445e740 1 -- start start 2024-04-01T01:47:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.107+0000 7f62e445e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239769f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.107+0000 7f62e445e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976a4d0 0x56123976efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.107+0000 7f62e445e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976f640 0x561239771ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.107+0000 7f62e445e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5612396cd8f0 con 0x561239769ae0 2024-04-01T01:47:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.107+0000 7f62e445e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5612396768d0 con 0x56123976f640 2024-04-01T01:47:23.108 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.107+0000 7f62e445e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5612396c9990 con 0x56123976a4d0 2024-04-01T01:47:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.107+0000 7f62daffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976a4d0 0x56123976efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.107+0000 7f62daffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976a4d0 0x56123976efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43742/0 (socket says 172.21.15.70:43742) 2024-04-01T01:47:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.107+0000 7f62daffd640 1 -- 172.21.15.70:0/351739261 learned_addr learned my addr 172.21.15.70:0/351739261 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.107+0000 7f62da7fc640 1 --2- 172.21.15.70:0/351739261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976f640 0x561239771ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.107+0000 7f62e087f640 1 --2- 172.21.15.70:0/218271583 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62c40a25d0 0x7f62c40a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.108+0000 7f62da7fc640 1 -- 172.21.15.70:0/351739261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976a4d0 msgr2=0x56123976efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.108+0000 7f62da7fc640 1 --2- 172.21.15.70:0/351739261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976a4d0 0x56123976efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.108+0000 7f62da7fc640 1 -- 172.21.15.70:0/351739261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 msgr2=0x561239769f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.108+0000 7f62da7fc640 1 --2- 172.21.15.70:0/351739261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239769f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.108+0000 7f62da7fc640 1 -- 172.21.15.70:0/351739261 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612393e2210 con 0x56123976f640 2024-04-01T01:47:23.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.108+0000 7f62db7fe640 1 --2- 172.21.15.70:0/218271583 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62c40a25d0 0x7f62c40a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:23.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.108+0000 7f62e087f640 1 -- 172.21.15.70:0/218271583 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f62d40f0610 con 0x7f62dc073360 2024-04-01T01:47:23.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.108+0000 7f62da7fc640 1 --2- 172.21.15.70:0/351739261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976f640 0x561239771ad0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f62d40edcb0 tx=0x7f62d4087ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=916e54faff636246 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:23.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.108+0000 7f62c1ffb640 1 -- 172.21.15.70:0/351739261 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d40bc7e0 con 0x56123976f640 2024-04-01T01:47:23.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.109+0000 7f62c1ffb640 1 -- 172.21.15.70:0/351739261 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62d4087fe0 con 0x56123976f640 2024-04-01T01:47:23.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.109+0000 7f62c1ffb640 1 -- 172.21.15.70:0/351739261 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d40bc600 con 0x56123976f640 2024-04-01T01:47:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.110+0000 7f62e445e740 1 -- 172.21.15.70:0/351739261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976f640 msgr2=0x561239771ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.110+0000 7f62e445e740 1 --2- 172.21.15.70:0/351739261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976f640 0x561239771ad0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f62d40edcb0 tx=0x7f62d4087ad0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.110+0000 7f62db7fe640 1 --2- 172.21.15.70:0/218271583 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62c40a25d0 0x7f62c40a4a90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00b640 tx=0x7f62cc00b5d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.110+0000 7f62e445e740 1 -- 172.21.15.70:0/351739261 shutdown_connections 2024-04-01T01:47:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.110+0000 7f62e445e740 1 --2- 172.21.15.70:0/351739261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976f640 0x561239771ad0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.110+0000 7f62e445e740 1 --2- 172.21.15.70:0/351739261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976a4d0 0x56123976efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.110+0000 7f62e445e740 1 --2- 172.21.15.70:0/351739261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239769f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.110+0000 7f62e445e740 1 -- 172.21.15.70:0/351739261 >> 172.21.15.70:0/351739261 conn(0x561239765050 msgr2=0x561239765450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.110+0000 7f62e445e740 1 -- 172.21.15.70:0/351739261 shutdown_connections 2024-04-01T01:47:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.110+0000 7f62e445e740 1 -- 172.21.15.70:0/351739261 wait complete. 2024-04-01T01:47:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.111+0000 7f62e445e740 1 -- start start 2024-04-01T01:47:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.111+0000 7f62e445e740 1 Processor -- start 2024-04-01T01:47:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.111+0000 7f62e445e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239773a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.111+0000 7f62e445e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976a4d0 0x561239773f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.111+0000 7f62e445e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976f640 0x56123977afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.111+0000 7f62e445e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5612397723b0 con 0x561239769ae0 2024-04-01T01:47:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.111+0000 7f62e445e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56123966bf80 con 0x56123976a4d0 2024-04-01T01:47:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.111+0000 7f62e445e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561239772530 con 0x56123976f640 2024-04-01T01:47:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.111+0000 7f62e445e740 10 client.0 _ll_register_callbacks cb 0x5612396c4290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.112+0000 7f62da7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976a4d0 0x561239773f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.112+0000 7f62da7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976a4d0 0x561239773f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:51532/0 (socket says 172.21.15.70:51532) 2024-04-01T01:47:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.112+0000 7f62da7fc640 1 -- 172.21.15.70:0/172860979 learned_addr learned my addr 172.21.15.70:0/172860979 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.112+0000 7f62daffd640 1 --2- 172.21.15.70:0/172860979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239773a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:23.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.112+0000 7f62db7fe640 1 --2- 172.21.15.70:0/172860979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976f640 0x56123977afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.112+0000 7f62da7fc640 1 -- 172.21.15.70:0/172860979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976f640 msgr2=0x56123977afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.112+0000 7f62da7fc640 1 --2- 172.21.15.70:0/172860979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976f640 0x56123977afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.112+0000 7f62da7fc640 1 -- 172.21.15.70:0/172860979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 msgr2=0x561239773a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.112+0000 7f62da7fc640 1 --2- 172.21.15.70:0/172860979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239773a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.112+0000 7f62da7fc640 1 -- 172.21.15.70:0/172860979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612394ae620 con 0x56123976a4d0 2024-04-01T01:47:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.112+0000 7f62daffd640 1 --2- 172.21.15.70:0/172860979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239773a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:47:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.113+0000 7f62da7fc640 1 --2- 172.21.15.70:0/172860979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976a4d0 0x561239773f70 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f62d4005e30 tx=0x7f62d40095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86620c2684918c04 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:23.114 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.113+0000 7f62baffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.113+0000 7f62baffd640 1 -- 172.21.15.70:0/172860979 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d40bcca0 con 0x56123976a4d0 2024-04-01T01:47:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.113+0000 7f62e445e740 1 -- 172.21.15.70:0/172860979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62d4087fe0 con 0x56123976a4d0 2024-04-01T01:47:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.113+0000 7f62baffd640 1 -- 172.21.15.70:0/172860979 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f62d40f3c60 con 0x56123976a4d0 2024-04-01T01:47:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.113+0000 7f62baffd640 1 -- 172.21.15.70:0/172860979 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f62d40c0df0 con 0x56123976a4d0 2024-04-01T01:47:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.115+0000 7f62e445e740 10 client.29600 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:23.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.115+0000 7f62e445e740 1 -- 172.21.15.70:0/172860979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5612397723b0 con 0x56123976a4d0 2024-04-01T01:47:23.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.116+0000 7f62baffd640 1 -- 172.21.15.70:0/172860979 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62d4087fe0 con 0x56123976a4d0 2024-04-01T01:47:23.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.116+0000 7f62baffd640 1 -- 172.21.15.70:0/172860979 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f62d403c030 con 0x56123976a4d0 2024-04-01T01:47:23.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.116+0000 7f62e445e740 10 client.29600 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:23.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.116+0000 7f62e445e740 10 client.29600 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.116+0000 7f62e445e740 20 client.29600 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.116+0000 7f62e445e740 1 --2- 172.21.15.70:0/172860979 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5612397905f0 0x561239792a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.116+0000 7f62e445e740 4 client.29600 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.116+0000 7f62e445e740 1 -- 172.21.15.70:0/172860979 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612396e1120 con 0x5612397905f0 2024-04-01T01:47:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.116+0000 7f62daffd640 1 --2- 172.21.15.70:0/172860979 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5612397905f0 0x561239792a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.117+0000 7f62daffd640 1 --2- 172.21.15.70:0/172860979 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5612397905f0 0x561239792a10 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3b654a760094683 server_cookie=434d6d4d13514ae4 in_seq=0 out_seq=0 2024-04-01T01:47:23.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.117+0000 7f62baffd640 10 client.29600 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.118+0000 7f62baffd640 1 -- 172.21.15.70:0/172860979 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5612396e1120 con 0x5612397905f0 2024-04-01T01:47:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.118+0000 7f62baffd640 10 client.29600 handle_command_reply: tid=0 2024-04-01T01:47:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.118+0000 7f62e445e740 1 client.29600 shutdown 2024-04-01T01:47:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.118+0000 7f62e445e740 1 -- 172.21.15.70:0/172860979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976a4d0 msgr2=0x561239773f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.118+0000 7f62e445e740 1 --2- 172.21.15.70:0/172860979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976a4d0 0x561239773f70 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f62d4005e30 tx=0x7f62d40095c0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.118+0000 7f62e445e740 1 -- 172.21.15.70:0/172860979 shutdown_connections 2024-04-01T01:47:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.118+0000 7f62e445e740 1 --2- 172.21.15.70:0/172860979 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5612397905f0 0x561239792a10 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.118+0000 7f62e445e740 1 --2- 172.21.15.70:0/172860979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976f640 0x56123977afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.118+0000 7f62e445e740 1 --2- 172.21.15.70:0/172860979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976a4d0 0x561239773f70 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.118+0000 7f62baffd640 0 client.29600 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.118+0000 7f62e445e740 1 --2- 172.21.15.70:0/172860979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239773a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.118+0000 7f62e445e740 1 -- 172.21.15.70:0/172860979 >> 172.21.15.70:0/172860979 conn(0x561239765050 msgr2=0x561239764190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.119+0000 7f62e445e740 1 -- 172.21.15.70:0/172860979 shutdown_connections 2024-04-01T01:47:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.119+0000 7f62e445e740 1 -- 172.21.15.70:0/172860979 wait complete. 2024-04-01T01:47:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.119+0000 7f62e445e740 20 client.29600 trim_cache size 0 max 16384 2024-04-01T01:47:23.149 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.148+0000 7f62e445e740 1 Processor -- start 2024-04-01T01:47:23.149 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.149+0000 7f62e445e740 1 -- start start 2024-04-01T01:47:23.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.149+0000 7f62e445e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239765430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.149+0000 7f62e445e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976a4d0 0x56123977af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.149+0000 7f62e445e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976f640 0x5612397982e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.149+0000 7f62e445e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc1b0890 con 0x561239769ae0 2024-04-01T01:47:23.150 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.149+0000 7f62e445e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc1b23d0 con 0x56123976f640 2024-04-01T01:47:23.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.149+0000 7f62e445e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc1b0f60 con 0x56123976a4d0 2024-04-01T01:47:23.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.150+0000 7f62da7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976a4d0 0x56123977af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:23.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.150+0000 7f62da7fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976a4d0 0x56123977af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:43756/0 (socket says 172.21.15.70:43756) 2024-04-01T01:47:23.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.150+0000 7f62da7fc640 1 -- 172.21.15.70:0/2573999553 learned_addr learned my addr 172.21.15.70:0/2573999553 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:23.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.150+0000 7f62db7fe640 1 --2- 172.21.15.70:0/2573999553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976f640 0x5612397982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:23.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.150+0000 7f62daffd640 1 --2- 172.21.15.70:0/2573999553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239765430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:23.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.150+0000 7f62db7fe640 1 -- 172.21.15.70:0/2573999553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976a4d0 msgr2=0x56123977af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:23.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.150+0000 7f62db7fe640 1 --2- 172.21.15.70:0/2573999553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976a4d0 0x56123977af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.150+0000 7f62db7fe640 1 -- 172.21.15.70:0/2573999553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 msgr2=0x561239765430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:23.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.150+0000 7f62db7fe640 1 --2- 172.21.15.70:0/2573999553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239765430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.151 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.150+0000 7f62db7fe640 1 -- 172.21.15.70:0/2573999553 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56123976b2e0 con 0x56123976f640 2024-04-01T01:47:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.153+0000 7f62db7fe640 1 --2- 172.21.15.70:0/2573999553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976f640 0x5612397982e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00bed0 tx=0x7f62cc013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e45d5e664a4f37dd server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.153+0000 7f62c17fa640 1 -- 172.21.15.70:0/2573999553 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc01fb80 con 0x56123976f640 2024-04-01T01:47:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.153+0000 7f62c17fa640 1 -- 172.21.15.70:0/2573999553 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62cc01fd20 con 0x56123976f640 2024-04-01T01:47:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.153+0000 7f62c17fa640 1 -- 172.21.15.70:0/2573999553 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc01a600 con 0x56123976f640 2024-04-01T01:47:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.154+0000 7f62e445e740 1 -- 172.21.15.70:0/2573999553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976f640 msgr2=0x5612397982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.154+0000 7f62e445e740 1 --2- 172.21.15.70:0/2573999553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976f640 0x5612397982e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00bed0 tx=0x7f62cc013000 comp rx=0 tx=0).stop 2024-04-01T01:47:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.154+0000 7f62e445e740 1 -- 172.21.15.70:0/2573999553 shutdown_connections 2024-04-01T01:47:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.154+0000 7f62e445e740 1 --2- 172.21.15.70:0/2573999553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976f640 0x5612397982e0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.154+0000 7f62e445e740 1 --2- 172.21.15.70:0/2573999553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976a4d0 0x56123977af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.154+0000 7f62e445e740 1 --2- 172.21.15.70:0/2573999553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239765430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.155+0000 7f62e445e740 1 -- 172.21.15.70:0/2573999553 >> 172.21.15.70:0/2573999553 conn(0x561239765050 msgr2=0x56123976edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.155+0000 7f62e445e740 1 -- 172.21.15.70:0/2573999553 shutdown_connections 2024-04-01T01:47:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.155+0000 7f62e445e740 1 -- 172.21.15.70:0/2573999553 wait complete. 2024-04-01T01:47:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.155+0000 7f62e445e740 1 -- start start 2024-04-01T01:47:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.155+0000 7f62e445e740 1 Processor -- start 2024-04-01T01:47:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.156+0000 7f62e445e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239778580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.156+0000 7f62e445e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976a4d0 0x561239778ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.156+0000 7f62daffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239778580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.156+0000 7f62daffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239778580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43350/0 (socket says 172.21.15.70:43350) 2024-04-01T01:47:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.156+0000 7f62da7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976a4d0 0x561239778ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.156+0000 7f62da7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976a4d0 0x561239778ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:51556/0 (socket says 172.21.15.70:51556) 2024-04-01T01:47:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.156+0000 7f62e445e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976f640 0x5612397799c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.156+0000 7f62da7fc640 1 -- 172.21.15.70:0/4286295760 learned_addr learned my addr 172.21.15.70:0/4286295760 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.156+0000 7f62e445e740 1 -- 172.21.15.70:0/4286295760 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc1b1d00 con 0x561239769ae0 2024-04-01T01:47:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.156+0000 7f62e445e740 1 -- 172.21.15.70:0/4286295760 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc1b1630 con 0x56123976a4d0 2024-04-01T01:47:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.156+0000 7f62e445e740 1 -- 172.21.15.70:0/4286295760 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc1181d0 con 0x56123976f640 2024-04-01T01:47:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.157+0000 7f62e445e740 10 client.0 _ll_register_callbacks cb 0x7f62d00094f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.157+0000 7f62db7fe640 1 --2- 172.21.15.70:0/4286295760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976f640 0x5612397799c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.157+0000 7f62da7fc640 1 -- 172.21.15.70:0/4286295760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976f640 msgr2=0x5612397799c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.157+0000 7f62da7fc640 1 --2- 172.21.15.70:0/4286295760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976f640 0x5612397799c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.157+0000 7f62da7fc640 1 -- 172.21.15.70:0/4286295760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 msgr2=0x561239778580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.157+0000 7f62da7fc640 1 --2- 172.21.15.70:0/4286295760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239778580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.157+0000 7f62da7fc640 1 -- 172.21.15.70:0/4286295760 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561239779fb0 con 0x56123976a4d0 2024-04-01T01:47:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.157+0000 7f62da7fc640 1 --2- 172.21.15.70:0/4286295760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976a4d0 0x561239778ac0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f62d40b9620 tx=0x7f62d402c0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af0c14e6c8330085 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.157+0000 7f62baffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:23.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.157+0000 7f62baffd640 1 -- 172.21.15.70:0/4286295760 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d40bc7e0 con 0x56123976a4d0 2024-04-01T01:47:23.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.158+0000 7f62e445e740 1 -- 172.21.15.70:0/4286295760 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62cc01fd20 con 0x56123976a4d0 2024-04-01T01:47:23.160 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.158+0000 7f62baffd640 1 -- 172.21.15.70:0/4286295760 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f62d40c52c0 con 0x56123976a4d0 2024-04-01T01:47:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.158+0000 7f62baffd640 1 -- 172.21.15.70:0/4286295760 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f62d40bd3a0 con 0x56123976a4d0 2024-04-01T01:47:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.158+0000 7f62e445e740 10 client.29606 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.158+0000 7f62e445e740 1 -- 172.21.15.70:0/4286295760 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561239779fb0 con 0x56123976a4d0 2024-04-01T01:47:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.158+0000 7f62baffd640 1 -- 172.21.15.70:0/4286295760 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62cc01fd20 con 0x56123976a4d0 2024-04-01T01:47:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.158+0000 7f62baffd640 1 -- 172.21.15.70:0/4286295760 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f62d4004b40 con 0x56123976a4d0 2024-04-01T01:47:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.158+0000 7f62e445e740 10 client.29606 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.158+0000 7f62e445e740 10 client.29606 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.158+0000 7f62e445e740 20 client.29606 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.158+0000 7f62e445e740 1 --2- 172.21.15.70:0/4286295760 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5612397f6d20 0x5612397f9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.158+0000 7f62e445e740 4 client.29606 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:47:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.158+0000 7f62e445e740 1 -- 172.21.15.70:0/4286295760 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5612396dae30 con 0x5612397f6d20 2024-04-01T01:47:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.159+0000 7f62daffd640 1 --2- 172.21.15.70:0/4286295760 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5612397f6d20 0x5612397f9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.161+0000 7f62daffd640 1 --2- 172.21.15.70:0/4286295760 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5612397f6d20 0x5612397f9140 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e7a124bc6a89c12 server_cookie=3bab99e80f5cf591 in_seq=0 out_seq=0 2024-04-01T01:47:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.161+0000 7f62baffd640 10 client.29606 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:23.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:22 smithi189 ceph-mon[35753]: pgmap v775: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 6.1 MiB/s wr, 729 op/s 2024-04-01T01:47:23.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:22 smithi070 ceph-mon[32179]: pgmap v775: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 6.1 MiB/s wr, 729 op/s 2024-04-01T01:47:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.284+0000 7f62baffd640 1 -- 172.21.15.70:0/4286295760 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5612396dae30 con 0x5612397f6d20 2024-04-01T01:47:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.284+0000 7f62baffd640 10 client.29606 handle_command_reply: tid=0 2024-04-01T01:47:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.284+0000 7f62e445e740 1 client.29606 shutdown 2024-04-01T01:47:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.285+0000 7f62e445e740 1 -- 172.21.15.70:0/4286295760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976a4d0 msgr2=0x561239778ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.285+0000 7f62e445e740 1 --2- 172.21.15.70:0/4286295760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976a4d0 0x561239778ac0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f62d40b9620 tx=0x7f62d402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.285+0000 7f62e445e740 1 -- 172.21.15.70:0/4286295760 shutdown_connections 2024-04-01T01:47:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.285+0000 7f62e445e740 1 --2- 172.21.15.70:0/4286295760 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5612397f6d20 0x5612397f9140 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.285+0000 7f62e445e740 1 --2- 172.21.15.70:0/4286295760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56123976f640 0x5612397799c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.285+0000 7f62e445e740 1 --2- 172.21.15.70:0/4286295760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56123976a4d0 0x561239778ac0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.285+0000 7f62baffd640 0 client.29606 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.285+0000 7f62e445e740 1 --2- 172.21.15.70:0/4286295760 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561239769ae0 0x561239778580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.285+0000 7f62e445e740 1 -- 172.21.15.70:0/4286295760 >> 172.21.15.70:0/4286295760 conn(0x561239765050 msgr2=0x56123976e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.285+0000 7f62e445e740 1 -- 172.21.15.70:0/4286295760 shutdown_connections 2024-04-01T01:47:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.285+0000 7f62e445e740 1 -- 172.21.15.70:0/4286295760 wait complete. 2024-04-01T01:47:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.286+0000 7f62e445e740 20 client.29606 trim_cache size 0 max 16384 2024-04-01T01:47:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:47:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:47:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:47:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:47:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.286+0000 7f62dbfff640 1 -- 172.21.15.70:0/218271583 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62c40a25d0 msgr2=0x7f62c40a4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.286+0000 7f62dbfff640 1 --2- 172.21.15.70:0/218271583 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62c40a25d0 0x7f62c40a4a90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00b640 tx=0x7f62cc00b5d0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.286+0000 7f62dbfff640 1 -- 172.21.15.70:0/218271583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62dc073360 msgr2=0x7f62dc1a6b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.286+0000 7f62dbfff640 1 --2- 172.21.15.70:0/218271583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62dc073360 0x7f62dc1a6b80 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f62d4037040 tx=0x7f62d4032820 comp rx=0 tx=0).stop 2024-04-01T01:47:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.287+0000 7f62dbfff640 1 -- 172.21.15.70:0/218271583 shutdown_connections 2024-04-01T01:47:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.287+0000 7f62dbfff640 1 --2- 172.21.15.70:0/218271583 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f62c40a25d0 0x7f62c40a4a90 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.287+0000 7f62dbfff640 1 --2- 172.21.15.70:0/218271583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc109440 0x7f62dc1adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.287+0000 7f62dbfff640 1 --2- 172.21.15.70:0/218271583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62dc073360 0x7f62dc1a6b80 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.287+0000 7f62dbfff640 1 --2- 172.21.15.70:0/218271583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62dc072a00 0x7f62dc1a6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.287+0000 7f62dbfff640 1 -- 172.21.15.70:0/218271583 >> 172.21.15.70:0/218271583 conn(0x7f62dc06e5c0 msgr2=0x7f62dc06f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.287+0000 7f62dbfff640 1 -- 172.21.15.70:0/218271583 shutdown_connections 2024-04-01T01:47:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:23.287+0000 7f62dbfff640 1 -- 172.21.15.70:0/218271583 wait complete. 2024-04-01T01:47:23.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:22 smithi081 ceph-mon[35142]: pgmap v775: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 6.1 MiB/s wr, 729 op/s 2024-04-01T01:47:25.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:24 smithi189 ceph-mon[35753]: pgmap v776: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 6.6 MiB/s wr, 793 op/s 2024-04-01T01:47:25.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:24 smithi070 ceph-mon[32179]: pgmap v776: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 6.6 MiB/s wr, 793 op/s 2024-04-01T01:47:25.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:24 smithi081 ceph-mon[35142]: pgmap v776: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 6.6 MiB/s wr, 793 op/s 2024-04-01T01:47:26.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 760.0 s, 748.6 tps, lat 134.176 ms stddev 152.593 2024-04-01T01:47:27.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:26 smithi189 ceph-mon[35753]: pgmap v777: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 5.7 MiB/s wr, 707 op/s 2024-04-01T01:47:27.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:26 smithi070 ceph-mon[32179]: pgmap v777: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 5.7 MiB/s wr, 707 op/s 2024-04-01T01:47:27.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:26 smithi081 ceph-mon[35142]: pgmap v777: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 5.7 MiB/s wr, 707 op/s 2024-04-01T01:47:29.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:28 smithi189 ceph-mon[35753]: pgmap v778: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 6.8 MiB/s wr, 799 op/s 2024-04-01T01:47:29.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:28 smithi070 ceph-mon[32179]: pgmap v778: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 6.8 MiB/s wr, 799 op/s 2024-04-01T01:47:29.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:28 smithi081 ceph-mon[35142]: pgmap v778: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 6.8 MiB/s wr, 799 op/s 2024-04-01T01:47:31.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:30 smithi189 ceph-mon[35753]: pgmap v779: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 6.0 MiB/s wr, 780 op/s 2024-04-01T01:47:31.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:30 smithi070 ceph-mon[32179]: pgmap v779: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 6.0 MiB/s wr, 780 op/s 2024-04-01T01:47:31.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:30 smithi081 ceph-mon[35142]: pgmap v779: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 6.0 MiB/s wr, 780 op/s 2024-04-01T01:47:31.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 765.0 s, 730.6 tps, lat 136.519 ms stddev 144.298 2024-04-01T01:47:33.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:32 smithi189 ceph-mon[35753]: pgmap v780: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 5.7 MiB/s wr, 694 op/s 2024-04-01T01:47:33.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:32 smithi070 ceph-mon[32179]: pgmap v780: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 5.7 MiB/s wr, 694 op/s 2024-04-01T01:47:33.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:32 smithi081 ceph-mon[35142]: pgmap v780: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 5.7 MiB/s wr, 694 op/s 2024-04-01T01:47:35.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:34 smithi189 ceph-mon[35753]: pgmap v781: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.4 MiB/s wr, 733 op/s 2024-04-01T01:47:35.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:34 smithi070 ceph-mon[32179]: pgmap v781: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.4 MiB/s wr, 733 op/s 2024-04-01T01:47:35.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:34 smithi081 ceph-mon[35142]: pgmap v781: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 6.4 MiB/s wr, 733 op/s 2024-04-01T01:47:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.408+0000 7f6dde302640 1 -- 172.21.15.70:0/1594905506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dd8102990 msgr2=0x7f6dd8102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.408+0000 7f6dde302640 1 --2- 172.21.15.70:0/1594905506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dd8102990 0x7f6dd8102e10 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009900 tx=0x7f6dcc02bc30 comp rx=0 tx=0).stop 2024-04-01T01:47:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.408+0000 7f6dde302640 1 -- 172.21.15.70:0/1594905506 shutdown_connections 2024-04-01T01:47:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.408+0000 7f6dde302640 1 --2- 172.21.15.70:0/1594905506 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dd8103350 0x7f6dd8109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.408+0000 7f6dde302640 1 --2- 172.21.15.70:0/1594905506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dd8102990 0x7f6dd8102e10 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.408+0000 7f6dde302640 1 --2- 172.21.15.70:0/1594905506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dd81017a0 0x7f6dd8101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.408+0000 7f6dde302640 1 -- 172.21.15.70:0/1594905506 >> 172.21.15.70:0/1594905506 conn(0x7f6dd80fcd10 msgr2=0x7f6dd80ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.408+0000 7f6dde302640 1 -- 172.21.15.70:0/1594905506 shutdown_connections 2024-04-01T01:47:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.408+0000 7f6dde302640 1 -- 172.21.15.70:0/1594905506 wait complete. 2024-04-01T01:47:36.410 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.409+0000 7f6dde302640 1 Processor -- start 2024-04-01T01:47:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.409+0000 7f6dde302640 1 -- start start 2024-04-01T01:47:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.409+0000 7f6dde302640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dd81017a0 0x7f6dd8075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.409+0000 7f6dde302640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dd8102990 0x7f6dd8073b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.409+0000 7f6dde302640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dd8103350 0x7f6dd8074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.410+0000 7f6dde302640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd810d2d0 con 0x7f6dd8102990 2024-04-01T01:47:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.410+0000 7f6dde302640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd810d150 con 0x7f6dd81017a0 2024-04-01T01:47:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.410+0000 7f6dde302640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd810d450 con 0x7f6dd8103350 2024-04-01T01:47:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.410+0000 7f6dd77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dd8102990 0x7f6dd8073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.410+0000 7f6dd7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dd81017a0 0x7f6dd8075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.410+0000 7f6dd77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dd8102990 0x7f6dd8073b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55940/0 (socket says 172.21.15.70:55940) 2024-04-01T01:47:36.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.410+0000 7f6dd77fe640 1 -- 172.21.15.70:0/3001500687 learned_addr learned my addr 172.21.15.70:0/3001500687 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.410+0000 7f6ddc89f640 1 --2- 172.21.15.70:0/3001500687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dd8103350 0x7f6dd8074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.410+0000 7f6dd77fe640 1 -- 172.21.15.70:0/3001500687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dd8103350 msgr2=0x7f6dd8074080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.410+0000 7f6dd77fe640 1 --2- 172.21.15.70:0/3001500687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dd8103350 0x7f6dd8074080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.410+0000 7f6dd77fe640 1 -- 172.21.15.70:0/3001500687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dd81017a0 msgr2=0x7f6dd8075500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.410+0000 7f6dd77fe640 1 --2- 172.21.15.70:0/3001500687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dd81017a0 0x7f6dd8075500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.410+0000 7f6dd77fe640 1 -- 172.21.15.70:0/3001500687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dd8074910 con 0x7f6dd8102990 2024-04-01T01:47:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.410+0000 7f6dd7fff640 1 --2- 172.21.15.70:0/3001500687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dd81017a0 0x7f6dd8075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:47:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.410+0000 7f6dd77fe640 1 --2- 172.21.15.70:0/3001500687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dd8102990 0x7f6dd8073b40 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc006010 tx=0x7f6dcc005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.411+0000 7f6dd57fa640 1 -- 172.21.15.70:0/3001500687 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc043020 con 0x7f6dd8102990 2024-04-01T01:47:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.411+0000 7f6dde302640 1 -- 172.21.15.70:0/3001500687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dd81a1a00 con 0x7f6dd8102990 2024-04-01T01:47:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.411+0000 7f6dd57fa640 1 -- 172.21.15.70:0/3001500687 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dcc0043d0 con 0x7f6dd8102990 2024-04-01T01:47:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.411+0000 7f6dd57fa640 1 -- 172.21.15.70:0/3001500687 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc003b80 con 0x7f6dd8102990 2024-04-01T01:47:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.411+0000 7f6dde302640 1 -- 172.21.15.70:0/3001500687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dd81a1fa0 con 0x7f6dd8102990 2024-04-01T01:47:36.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.414+0000 7f6dd57fa640 1 -- 172.21.15.70:0/3001500687 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f6dcc046020 con 0x7f6dd8102990 2024-04-01T01:47:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.414+0000 7f6dd57fa640 1 --2- 172.21.15.70:0/3001500687 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6dc00a25c0 0x7f6dc00a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.414+0000 7f6de0fc3740 1 Processor -- start 2024-04-01T01:47:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.414+0000 7f6de0fc3740 1 -- start start 2024-04-01T01:47:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.414+0000 7f6dd57fa640 1 -- 172.21.15.70:0/3001500687 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6dcc0f0c20 con 0x7f6dd8102990 2024-04-01T01:47:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.414+0000 7f6dd7fff640 1 --2- 172.21.15.70:0/3001500687 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6dc00a25c0 0x7f6dc00a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6de0fc3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974a68d0 0x5645974a6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6de0fc3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a72c0 0x5645974abdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6de0fc3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974ac430 0x5645974ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6de0fc3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56459740a8f0 con 0x5645974ac430 2024-04-01T01:47:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6de0fc3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5645973b38d0 con 0x5645974a72c0 2024-04-01T01:47:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6de0fc3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564597406990 con 0x5645974a68d0 2024-04-01T01:47:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6ddc89f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974ac430 0x5645974ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6ddc89f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974ac430 0x5645974ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55944/0 (socket says 172.21.15.70:55944) 2024-04-01T01:47:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6ddc89f640 1 -- 172.21.15.70:0/1238976906 learned_addr learned my addr 172.21.15.70:0/1238976906 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6dd77fe640 1 --2- 172.21.15.70:0/1238976906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a72c0 0x5645974abdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6dd7fff640 1 --2- 172.21.15.70:0/1238976906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974a68d0 0x5645974a6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6dd7fff640 1 --2- 172.21.15.70:0/3001500687 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6dc00a25c0 0x7f6dc00a4a80 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f6dd81027f0 tx=0x7f6dc8011810 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6ddc89f640 1 -- 172.21.15.70:0/1238976906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974a68d0 msgr2=0x5645974a6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6ddc89f640 1 --2- 172.21.15.70:0/1238976906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974a68d0 0x5645974a6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.424 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6ddc89f640 1 -- 172.21.15.70:0/1238976906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a72c0 msgr2=0x5645974abdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6ddc89f640 1 --2- 172.21.15.70:0/1238976906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a72c0 0x5645974abdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6ddc89f640 1 -- 172.21.15.70:0/1238976906 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56459711f210 con 0x5645974ac430 2024-04-01T01:47:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6dd7fff640 1 --2- 172.21.15.70:0/1238976906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974a68d0 0x5645974a6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:47:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6ddc89f640 1 --2- 172.21.15.70:0/1238976906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974ac430 0x5645974ae8c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0009fd0 tx=0x7f6dd00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ad8a88600fa0c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6dbe7fc640 1 -- 172.21.15.70:0/1238976906 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd0004d20 con 0x5645974ac430 2024-04-01T01:47:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6dbe7fc640 1 -- 172.21.15.70:0/1238976906 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dd0004050 con 0x5645974ac430 2024-04-01T01:47:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.415+0000 7f6dbe7fc640 1 -- 172.21.15.70:0/1238976906 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd0005280 con 0x5645974ac430 2024-04-01T01:47:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.416+0000 7f6de0fc3740 1 -- 172.21.15.70:0/1238976906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974ac430 msgr2=0x5645974ae8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.416+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/1238976906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974ac430 0x5645974ae8c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0009fd0 tx=0x7f6dd00118d0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.416+0000 7f6de0fc3740 1 -- 172.21.15.70:0/1238976906 shutdown_connections 2024-04-01T01:47:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.416+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/1238976906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974ac430 0x5645974ae8c0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.416+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/1238976906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a72c0 0x5645974abdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.416+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/1238976906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974a68d0 0x5645974a6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.416+0000 7f6de0fc3740 1 -- 172.21.15.70:0/1238976906 >> 172.21.15.70:0/1238976906 conn(0x5645974a1e40 msgr2=0x5645974a2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.416+0000 7f6de0fc3740 1 -- 172.21.15.70:0/1238976906 shutdown_connections 2024-04-01T01:47:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.416+0000 7f6de0fc3740 1 -- 172.21.15.70:0/1238976906 wait complete. 2024-04-01T01:47:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6de0fc3740 1 -- start start 2024-04-01T01:47:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6de0fc3740 1 Processor -- start 2024-04-01T01:47:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6de0fc3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974a68d0 0x5645974b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6de0fc3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a72c0 0x5645974b1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6de0fc3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974ac430 0x5645974b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6de0fc3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5645974af3e0 con 0x5645974a68d0 2024-04-01T01:47:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6de0fc3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5645973a8f80 con 0x5645974a72c0 2024-04-01T01:47:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6de0fc3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5645974af560 con 0x5645974ac430 2024-04-01T01:47:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6de0fc3740 10 client.0 _ll_register_callbacks cb 0x564597401290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6dd7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974a68d0 0x5645974b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6dd7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974a68d0 0x5645974b0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55958/0 (socket says 172.21.15.70:55958) 2024-04-01T01:47:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6dd7fff640 1 -- 172.21.15.70:0/776449328 learned_addr learned my addr 172.21.15.70:0/776449328 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6dd77fe640 1 --2- 172.21.15.70:0/776449328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a72c0 0x5645974b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6ddc89f640 1 --2- 172.21.15.70:0/776449328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974ac430 0x5645974b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6dd7fff640 1 -- 172.21.15.70:0/776449328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974ac430 msgr2=0x5645974b80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6dd7fff640 1 --2- 172.21.15.70:0/776449328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974ac430 0x5645974b80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6dd7fff640 1 -- 172.21.15.70:0/776449328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a72c0 msgr2=0x5645974b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6dd7fff640 1 --2- 172.21.15.70:0/776449328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a72c0 0x5645974b1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6dd7fff640 1 -- 172.21.15.70:0/776449328 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645971eb620 con 0x5645974a68d0 2024-04-01T01:47:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.417+0000 7f6dd77fe640 1 --2- 172.21.15.70:0/776449328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a72c0 0x5645974b1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:47:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.418+0000 7f6dd7fff640 1 --2- 172.21.15.70:0/776449328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974a68d0 0x5645974b0b20 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800fb30 tx=0x7f6dc8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdbdfb449081c0c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.418+0000 7f6db77fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:47:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.418+0000 7f6db77fe640 1 -- 172.21.15.70:0/776449328 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc802b060 con 0x5645974a68d0 2024-04-01T01:47:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.418+0000 7f6db77fe640 1 -- 172.21.15.70:0/776449328 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6dc801eb90 con 0x5645974a68d0 2024-04-01T01:47:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.418+0000 7f6de0fc3740 1 -- 172.21.15.70:0/776449328 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6dd0004050 con 0x5645974a68d0 2024-04-01T01:47:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.418+0000 7f6db77fe640 1 -- 172.21.15.70:0/776449328 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6dc8017040 con 0x5645974a68d0 2024-04-01T01:47:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.418+0000 7f6de0fc3740 10 client.19530 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.418+0000 7f6de0fc3740 1 -- 172.21.15.70:0/776449328 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5645974bb150 con 0x5645974a68d0 2024-04-01T01:47:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.418+0000 7f6db77fe640 1 -- 172.21.15.70:0/776449328 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dd0004050 con 0x5645974a68d0 2024-04-01T01:47:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.419+0000 7f6db77fe640 1 -- 172.21.15.70:0/776449328 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6dc8025030 con 0x5645974a68d0 2024-04-01T01:47:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.419+0000 7f6de0fc3740 10 client.19530 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.419+0000 7f6de0fc3740 10 client.19530 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.419+0000 7f6de0fc3740 20 client.19530 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.419+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/776449328 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5645974bc890 0x5645974bed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.419+0000 7f6de0fc3740 4 client.19530 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.419+0000 7f6de0fc3740 1 -- 172.21.15.70:0/776449328 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56459741e120 con 0x5645974bc890 2024-04-01T01:47:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.419+0000 7f6dd77fe640 1 --2- 172.21.15.70:0/776449328 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5645974bc890 0x5645974bed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.419+0000 7f6db77fe640 10 client.19530 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.419+0000 7f6dd77fe640 1 --2- 172.21.15.70:0/776449328 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5645974bc890 0x5645974bed50 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8d141bce1247895 server_cookie=728355d64ee55136 in_seq=0 out_seq=0 2024-04-01T01:47:36.428 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.420+0000 7f6db77fe640 1 -- 172.21.15.70:0/776449328 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f6dcc0a29c0 con 0x5645974bc890 2024-04-01T01:47:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.420+0000 7f6db77fe640 10 client.19530 handle_command_reply: tid=0 2024-04-01T01:47:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.420+0000 7f6de0fc3740 1 client.19530 shutdown 2024-04-01T01:47:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.420+0000 7f6de0fc3740 1 -- 172.21.15.70:0/776449328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974a68d0 msgr2=0x5645974b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.420+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/776449328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974a68d0 0x5645974b0b20 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800fb30 tx=0x7f6dc8004730 comp rx=0 tx=0).stop 2024-04-01T01:47:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.420+0000 7f6de0fc3740 1 -- 172.21.15.70:0/776449328 shutdown_connections 2024-04-01T01:47:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.420+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/776449328 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5645974bc890 0x5645974bed50 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.420+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/776449328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974ac430 0x5645974b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.420+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/776449328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a72c0 0x5645974b1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.420+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/776449328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974a68d0 0x5645974b0b20 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.420+0000 7f6de0fc3740 1 -- 172.21.15.70:0/776449328 >> 172.21.15.70:0/776449328 conn(0x5645974a1e40 msgr2=0x5645974a0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.420+0000 7f6db77fe640 0 client.19530 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.421+0000 7f6de0fc3740 1 -- 172.21.15.70:0/776449328 shutdown_connections 2024-04-01T01:47:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.421+0000 7f6de0fc3740 1 -- 172.21.15.70:0/776449328 wait complete. 2024-04-01T01:47:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.421+0000 7f6de0fc3740 20 client.19530 trim_cache size 0 max 16384 2024-04-01T01:47:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6de0fc3740 1 Processor -- start 2024-04-01T01:47:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6de0fc3740 1 -- start start 2024-04-01T01:47:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6de0fc3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974a68d0 0x5645974a2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6de0fc3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a72c0 0x5645974b8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6de0fc3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974ac430 0x5645974c4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6de0fc3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd81ac4b0 con 0x5645974a68d0 2024-04-01T01:47:36.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6de0fc3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd81adff0 con 0x5645974a72c0 2024-04-01T01:47:36.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6de0fc3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd81acb80 con 0x5645974ac430 2024-04-01T01:47:36.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6dd7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974a68d0 0x5645974a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6dd7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974a68d0 0x5645974a2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55966/0 (socket says 172.21.15.70:55966) 2024-04-01T01:47:36.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6dd7fff640 1 -- 172.21.15.70:0/2448875238 learned_addr learned my addr 172.21.15.70:0/2448875238 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:36.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6dd77fe640 1 --2- 172.21.15.70:0/2448875238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a72c0 0x5645974b8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6ddc89f640 1 --2- 172.21.15.70:0/2448875238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974ac430 0x5645974c4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6dd7fff640 1 -- 172.21.15.70:0/2448875238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974ac430 msgr2=0x5645974c4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:36.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6dd7fff640 1 --2- 172.21.15.70:0/2448875238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974ac430 0x5645974c4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6dd7fff640 1 -- 172.21.15.70:0/2448875238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a72c0 msgr2=0x5645974b8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:36.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6dd7fff640 1 --2- 172.21.15.70:0/2448875238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a72c0 0x5645974b8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6dd7fff640 1 -- 172.21.15.70:0/2448875238 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56459743f240 con 0x5645974a68d0 2024-04-01T01:47:36.446 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6ddc89f640 1 --2- 172.21.15.70:0/2448875238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974ac430 0x5645974c4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:47:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6dd7fff640 1 --2- 172.21.15.70:0/2448875238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974a68d0 0x5645974a2220 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800f940 tx=0x7f6dc800f970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa33378b688436 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.440+0000 7f6dbdffb640 1 -- 172.21.15.70:0/2448875238 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc803b070 con 0x5645974a68d0 2024-04-01T01:47:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.441+0000 7f6dbdffb640 1 -- 172.21.15.70:0/2448875238 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dc8019050 con 0x5645974a68d0 2024-04-01T01:47:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.441+0000 7f6dbdffb640 1 -- 172.21.15.70:0/2448875238 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc8004480 con 0x5645974a68d0 2024-04-01T01:47:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.441+0000 7f6de0fc3740 1 -- 172.21.15.70:0/2448875238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974a68d0 msgr2=0x5645974a2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.441+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/2448875238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974a68d0 0x5645974a2220 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800f940 tx=0x7f6dc800f970 comp rx=0 tx=0).stop 2024-04-01T01:47:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.441+0000 7f6de0fc3740 1 -- 172.21.15.70:0/2448875238 shutdown_connections 2024-04-01T01:47:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.441+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/2448875238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974ac430 0x5645974c4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.441+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/2448875238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a72c0 0x5645974b8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.441+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/2448875238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974a68d0 0x5645974a2220 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.441+0000 7f6de0fc3740 1 -- 172.21.15.70:0/2448875238 >> 172.21.15.70:0/2448875238 conn(0x5645974a1e40 msgr2=0x5645974abb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.441+0000 7f6de0fc3740 1 -- 172.21.15.70:0/2448875238 shutdown_connections 2024-04-01T01:47:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.441+0000 7f6de0fc3740 1 -- 172.21.15.70:0/2448875238 wait complete. 2024-04-01T01:47:36.447 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6de0fc3740 1 -- start start 2024-04-01T01:47:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6de0fc3740 1 Processor -- start 2024-04-01T01:47:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6de0fc3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a68d0 0x5645974b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6de0fc3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974a72c0 0x5645974b55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6de0fc3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974ac430 0x564597520c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6de0fc3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd81ad920 con 0x5645974ac430 2024-04-01T01:47:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6de0fc3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd81ad250 con 0x5645974a68d0 2024-04-01T01:47:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6de0fc3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6dd81aa490 con 0x5645974a72c0 2024-04-01T01:47:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6de0fc3740 10 client.0 _ll_register_callbacks cb 0x7f6dc800f560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6ddc89f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974ac430 0x564597520c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6dd7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a68d0 0x5645974b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6ddc89f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974ac430 0x564597520c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55970/0 (socket says 172.21.15.70:55970) 2024-04-01T01:47:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6ddc89f640 1 -- 172.21.15.70:0/1387176010 learned_addr learned my addr 172.21.15.70:0/1387176010 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6dd7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a68d0 0x5645974b5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41548/0 (socket says 172.21.15.70:41548) 2024-04-01T01:47:36.448 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6dd77fe640 1 --2- 172.21.15.70:0/1387176010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974a72c0 0x5645974b55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6ddc89f640 1 -- 172.21.15.70:0/1387176010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974a72c0 msgr2=0x5645974b55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6ddc89f640 1 --2- 172.21.15.70:0/1387176010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974a72c0 0x5645974b55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6ddc89f640 1 -- 172.21.15.70:0/1387176010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a68d0 msgr2=0x5645974b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6ddc89f640 1 --2- 172.21.15.70:0/1387176010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a68d0 0x5645974b5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6ddc89f640 1 -- 172.21.15.70:0/1387176010 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645975211c0 con 0x5645974ac430 2024-04-01T01:47:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6ddc89f640 1 --2- 172.21.15.70:0/1387176010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974ac430 0x564597520c80 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0006010 tx=0x7f6dd0009d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f751db4061062c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6db77fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:47:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.442+0000 7f6db77fe640 1 -- 172.21.15.70:0/1387176010 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd00042e0 con 0x5645974ac430 2024-04-01T01:47:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.443+0000 7f6db77fe640 1 -- 172.21.15.70:0/1387176010 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6dd0004480 con 0x5645974ac430 2024-04-01T01:47:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.443+0000 7f6de0fc3740 1 -- 172.21.15.70:0/1387176010 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6dc8019050 con 0x5645974ac430 2024-04-01T01:47:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.443+0000 7f6db77fe640 1 -- 172.21.15.70:0/1387176010 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6dd0004740 con 0x5645974ac430 2024-04-01T01:47:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.443+0000 7f6de0fc3740 10 client.19536 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.443+0000 7f6de0fc3740 1 -- 172.21.15.70:0/1387176010 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5645975224c0 con 0x5645974ac430 2024-04-01T01:47:36.449 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.443+0000 7f6db77fe640 1 -- 172.21.15.70:0/1387176010 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6dc8019050 con 0x5645974ac430 2024-04-01T01:47:36.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.443+0000 7f6db77fe640 1 -- 172.21.15.70:0/1387176010 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6dd0005540 con 0x5645974ac430 2024-04-01T01:47:36.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.443+0000 7f6de0fc3740 10 client.19536 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:36.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.443+0000 7f6de0fc3740 10 client.19536 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:36.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.443+0000 7f6de0fc3740 20 client.19536 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:36.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.443+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/1387176010 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564597522cd0 0x5645975250c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:36.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.443+0000 7f6de0fc3740 4 client.19536 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:47:36.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.443+0000 7f6de0fc3740 1 -- 172.21.15.70:0/1387176010 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564597417e30 con 0x564597522cd0 2024-04-01T01:47:36.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.443+0000 7f6dd7fff640 1 --2- 172.21.15.70:0/1387176010 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564597522cd0 0x5645975250c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:36.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.444+0000 7f6db77fe640 10 client.19536 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:36.450 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.444+0000 7f6dd7fff640 1 --2- 172.21.15.70:0/1387176010 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564597522cd0 0x5645975250c0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=350a9788c09ba24f server_cookie=e3a763b4766beb54 in_seq=0 out_seq=0 2024-04-01T01:47:36.465 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.464+0000 7f6db77fe640 1 -- 172.21.15.70:0/1387176010 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564597417e30 con 0x564597522cd0 2024-04-01T01:47:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.464+0000 7f6db77fe640 10 client.19536 handle_command_reply: tid=0 2024-04-01T01:47:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.464+0000 7f6de0fc3740 1 client.19536 shutdown 2024-04-01T01:47:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.464+0000 7f6de0fc3740 1 -- 172.21.15.70:0/1387176010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974ac430 msgr2=0x564597520c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.464+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/1387176010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974ac430 0x564597520c80 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0006010 tx=0x7f6dd0009d60 comp rx=0 tx=0).stop 2024-04-01T01:47:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.464+0000 7f6de0fc3740 1 -- 172.21.15.70:0/1387176010 shutdown_connections 2024-04-01T01:47:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.464+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/1387176010 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564597522cd0 0x5645975250c0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.464+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/1387176010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5645974ac430 0x564597520c80 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.465+0000 7f6db77fe640 0 client.19536 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.464+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/1387176010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645974a72c0 0x5645974b55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.465+0000 7f6de0fc3740 1 --2- 172.21.15.70:0/1387176010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5645974a68d0 0x5645974b5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.466 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.465+0000 7f6de0fc3740 1 -- 172.21.15.70:0/1387176010 >> 172.21.15.70:0/1387176010 conn(0x5645974a1e40 msgr2=0x5645974a84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.465+0000 7f6de0fc3740 1 -- 172.21.15.70:0/1387176010 shutdown_connections 2024-04-01T01:47:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.465+0000 7f6de0fc3740 1 -- 172.21.15.70:0/1387176010 wait complete. 2024-04-01T01:47:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.465+0000 7f6de0fc3740 20 client.19536 trim_cache size 0 max 16384 2024-04-01T01:47:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:47:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:47:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:47:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:47:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.465+0000 7f6dde302640 1 -- 172.21.15.70:0/3001500687 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6dc00a25c0 msgr2=0x7f6dc00a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.465+0000 7f6dde302640 1 --2- 172.21.15.70:0/3001500687 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6dc00a25c0 0x7f6dc00a4a80 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f6dd81027f0 tx=0x7f6dc8011810 comp rx=0 tx=0).stop 2024-04-01T01:47:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.465+0000 7f6dde302640 1 -- 172.21.15.70:0/3001500687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dd8102990 msgr2=0x7f6dd8073b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.465+0000 7f6dde302640 1 --2- 172.21.15.70:0/3001500687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dd8102990 0x7f6dd8073b40 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc006010 tx=0x7f6dcc005e40 comp rx=0 tx=0).stop 2024-04-01T01:47:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.466+0000 7f6dde302640 1 -- 172.21.15.70:0/3001500687 shutdown_connections 2024-04-01T01:47:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.466+0000 7f6dde302640 1 --2- 172.21.15.70:0/3001500687 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6dc00a25c0 0x7f6dc00a4a80 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.466+0000 7f6dde302640 1 --2- 172.21.15.70:0/3001500687 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dd8103350 0x7f6dd8074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.466+0000 7f6dde302640 1 --2- 172.21.15.70:0/3001500687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6dd8102990 0x7f6dd8073b40 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.466+0000 7f6dde302640 1 --2- 172.21.15.70:0/3001500687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dd81017a0 0x7f6dd8075500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.466+0000 7f6dde302640 1 -- 172.21.15.70:0/3001500687 >> 172.21.15.70:0/3001500687 conn(0x7f6dd80fcd10 msgr2=0x7f6dd8105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.466+0000 7f6dde302640 1 -- 172.21.15.70:0/3001500687 shutdown_connections 2024-04-01T01:47:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:36.466+0000 7f6dde302640 1 -- 172.21.15.70:0/3001500687 wait complete. 2024-04-01T01:47:36.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 770.0 s, 740.2 tps, lat 136.763 ms stddev 139.606 2024-04-01T01:47:37.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:36 smithi189 ceph-mon[35753]: pgmap v782: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.0 MiB/s wr, 658 op/s 2024-04-01T01:47:37.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:36 smithi070 ceph-mon[32179]: pgmap v782: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.0 MiB/s wr, 658 op/s 2024-04-01T01:47:37.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:36 smithi081 ceph-mon[35142]: pgmap v782: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 6.0 MiB/s wr, 658 op/s 2024-04-01T01:47:39.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:38 smithi189 ceph-mon[35753]: pgmap v783: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.5 MiB/s wr, 719 op/s 2024-04-01T01:47:39.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:38 smithi070 ceph-mon[32179]: pgmap v783: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.5 MiB/s wr, 719 op/s 2024-04-01T01:47:39.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:38 smithi081 ceph-mon[35142]: pgmap v783: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 6.5 MiB/s wr, 719 op/s 2024-04-01T01:47:41.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:40 smithi189 ceph-mon[35753]: pgmap v784: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.2 MiB/s wr, 708 op/s 2024-04-01T01:47:41.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:40 smithi070 ceph-mon[32179]: pgmap v784: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.2 MiB/s wr, 708 op/s 2024-04-01T01:47:41.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:40 smithi081 ceph-mon[35142]: pgmap v784: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 6.2 MiB/s wr, 708 op/s 2024-04-01T01:47:41.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 775.0 s, 749.8 tps, lat 133.074 ms stddev 191.332 2024-04-01T01:47:43.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:42 smithi189 ceph-mon[35753]: pgmap v785: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.2 MiB/s wr, 684 op/s 2024-04-01T01:47:43.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:42 smithi070 ceph-mon[32179]: pgmap v785: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.2 MiB/s wr, 684 op/s 2024-04-01T01:47:43.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:42 smithi081 ceph-mon[35142]: pgmap v785: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 6.2 MiB/s wr, 684 op/s 2024-04-01T01:47:45.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:44 smithi189 ceph-mon[35753]: pgmap v786: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.7 MiB/s wr, 744 op/s 2024-04-01T01:47:45.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:44 smithi070 ceph-mon[32179]: pgmap v786: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.7 MiB/s wr, 744 op/s 2024-04-01T01:47:45.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:44 smithi081 ceph-mon[35142]: pgmap v786: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.7 MiB/s wr, 744 op/s 2024-04-01T01:47:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 780.0 s, 864.8 tps, lat 117.248 ms stddev 118.074 2024-04-01T01:47:47.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:46 smithi189 ceph-mon[35753]: pgmap v787: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.8 MiB/s wr, 664 op/s 2024-04-01T01:47:47.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:46 smithi070 ceph-mon[32179]: pgmap v787: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.8 MiB/s wr, 664 op/s 2024-04-01T01:47:47.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:47:47.257 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:47:47.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:46 smithi081 ceph-mon[35142]: pgmap v787: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 5.8 MiB/s wr, 664 op/s 2024-04-01T01:47:47.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.436+0000 7f36b0d74640 1 -- 172.21.15.70:0/1365716487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36ac100650 msgr2=0x7f36ac100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:47.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.436+0000 7f36b0d74640 1 --2- 172.21.15.70:0/1365716487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36ac100650 0x7f36ac100ad0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009860 tx=0x7f36a002bb90 comp rx=0 tx=0).stop 2024-04-01T01:47:47.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.436+0000 7f36b0d74640 1 -- 172.21.15.70:0/1365716487 shutdown_connections 2024-04-01T01:47:47.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.436+0000 7f36b0d74640 1 --2- 172.21.15.70:0/1365716487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36ac101010 0x7f36ac1058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.436+0000 7f36b0d74640 1 --2- 172.21.15.70:0/1365716487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36ac100650 0x7f36ac100ad0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.436+0000 7f36b0d74640 1 --2- 172.21.15.70:0/1365716487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36ac0ff8f0 0x7f36ac0ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.436+0000 7f36b0d74640 1 -- 172.21.15.70:0/1365716487 >> 172.21.15.70:0/1365716487 conn(0x7f36ac0fb240 msgr2=0x7f36ac0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:47.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.436+0000 7f36b0d74640 1 -- 172.21.15.70:0/1365716487 shutdown_connections 2024-04-01T01:47:47.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.437+0000 7f36b0d74640 1 -- 172.21.15.70:0/1365716487 wait complete. 2024-04-01T01:47:47.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.437+0000 7f36b0d74640 1 Processor -- start 2024-04-01T01:47:47.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.438+0000 7f36b0d74640 1 -- start start 2024-04-01T01:47:47.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.438+0000 7f36b0d74640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36ac0ff8f0 0x7f36ac195770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.438+0000 7f36b0d74640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36ac100650 0x7f36ac195cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.438+0000 7f36b0d74640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36ac101010 0x7f36ac19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.438+0000 7f36b0d74640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac109000 con 0x7f36ac0ff8f0 2024-04-01T01:47:47.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.438+0000 7f36b0d74640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac108e80 con 0x7f36ac101010 2024-04-01T01:47:47.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.438+0000 7f36b0d74640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac109180 con 0x7f36ac100650 2024-04-01T01:47:47.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.438+0000 7f36abfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36ac101010 0x7f36ac19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.438+0000 7f36abfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36ac101010 0x7f36ac19cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33698/0 (socket says 172.21.15.70:33698) 2024-04-01T01:47:47.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.438+0000 7f36abfff640 1 -- 172.21.15.70:0/3549945044 learned_addr learned my addr 172.21.15.70:0/3549945044 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:47.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.439+0000 7f36abfff640 1 -- 172.21.15.70:0/3549945044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36ac100650 msgr2=0x7f36ac195cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:47:47.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.439+0000 7f36abfff640 1 --2- 172.21.15.70:0/3549945044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36ac100650 0x7f36ac195cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.439+0000 7f36abfff640 1 -- 172.21.15.70:0/3549945044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36ac0ff8f0 msgr2=0x7f36ac195770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:47:47.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.439+0000 7f36abfff640 1 --2- 172.21.15.70:0/3549945044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36ac0ff8f0 0x7f36ac195770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.439+0000 7f36abfff640 1 -- 172.21.15.70:0/3549945044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36ac06b2a0 con 0x7f36ac101010 2024-04-01T01:47:47.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.440+0000 7f36abfff640 1 --2- 172.21.15.70:0/3549945044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36ac101010 0x7f36ac19cd20 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f369c01e920 tx=0x7f369c01ede0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:47.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.440+0000 7f36a8ff9640 1 -- 172.21.15.70:0/3549945044 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369c028070 con 0x7f36ac101010 2024-04-01T01:47:47.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.440+0000 7f36b0d74640 1 -- 172.21.15.70:0/3549945044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36ac06b590 con 0x7f36ac101010 2024-04-01T01:47:47.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.440+0000 7f36a8ff9640 1 -- 172.21.15.70:0/3549945044 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f369c005020 con 0x7f36ac101010 2024-04-01T01:47:47.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.440+0000 7f36a8ff9640 1 -- 172.21.15.70:0/3549945044 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369c004c50 con 0x7f36ac101010 2024-04-01T01:47:47.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.440+0000 7f36b0d74640 1 -- 172.21.15.70:0/3549945044 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36ac06bb60 con 0x7f36ac101010 2024-04-01T01:47:47.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.444+0000 7f36b4bb0740 1 Processor -- start 2024-04-01T01:47:47.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.444+0000 7f36a8ff9640 1 -- 172.21.15.70:0/3549945044 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f369c021470 con 0x7f36ac101010 2024-04-01T01:47:47.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.444+0000 7f36b4bb0740 1 -- start start 2024-04-01T01:47:47.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.444+0000 7f36b4bb0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.444+0000 7f36b4bb0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 0x556a7d4d6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.444+0000 7f36b4bb0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d6ff0 0x556a7d4d9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.444+0000 7f36b4bb0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556a7d436180 con 0x556a7d4d1e80 2024-04-01T01:47:47.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.444+0000 7f36b4bb0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x556a7d3deb70 con 0x556a7d4d1490 2024-04-01T01:47:47.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.444+0000 7f36b4bb0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556a7d432220 con 0x556a7d4d6ff0 2024-04-01T01:47:47.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.445+0000 7f36aaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.445+0000 7f36aaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33708/0 (socket says 172.21.15.70:33708) 2024-04-01T01:47:47.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.445+0000 7f36aaffd640 1 -- 172.21.15.70:0/1095801926 learned_addr learned my addr 172.21.15.70:0/1095801926 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:47.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.445+0000 7f36aaffd640 1 --2- 172.21.15.70:0/1095801926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d6ff0 0x556a7d4d9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.445+0000 7f36ab7fe640 1 --2- 172.21.15.70:0/1095801926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 0x556a7d4d6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.445+0000 7f36a8ff9640 1 --2- 172.21.15.70:0/3549945044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36940a2570 0x7f36940a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.445+0000 7f36a8ff9640 1 -- 172.21.15.70:0/3549945044 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f369c0df540 con 0x7f36ac101010 2024-04-01T01:47:47.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.445+0000 7f36abfff640 1 --2- 172.21.15.70:0/3549945044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36940a2570 0x7f36940a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.445+0000 7f36aaffd640 1 -- 172.21.15.70:0/1095801926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d6ff0 msgr2=0x556a7d4d9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:47.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.445+0000 7f36aaffd640 1 --2- 172.21.15.70:0/1095801926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d6ff0 0x556a7d4d9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.445+0000 7f36aaffd640 1 -- 172.21.15.70:0/1095801926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 msgr2=0x556a7d4d6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:47.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.445+0000 7f36aaffd640 1 --2- 172.21.15.70:0/1095801926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 0x556a7d4d6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.445+0000 7f36aaffd640 1 -- 172.21.15.70:0/1095801926 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a7d3f27c0 con 0x556a7d4d1490 2024-04-01T01:47:47.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.446+0000 7f36ab7fe640 1 --2- 172.21.15.70:0/1095801926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 0x556a7d4d6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:47:47.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.447+0000 7f36abfff640 1 --2- 172.21.15.70:0/3549945044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36940a2570 0x7f36940a4a30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f369c01efb0 tx=0x7f369c005f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:47.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.454+0000 7f36aaffd640 1 --2- 172.21.15.70:0/1095801926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4d18b0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009780 tx=0x7f36a00349f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91e3c9853bf9f248 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:47.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.457+0000 7f3691ffb640 1 -- 172.21.15.70:0/1095801926 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a0039070 con 0x556a7d4d1490 2024-04-01T01:47:47.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.458+0000 7f3691ffb640 1 -- 172.21.15.70:0/1095801926 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36a003a040 con 0x556a7d4d1490 2024-04-01T01:47:47.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.459+0000 7f36b4bb0740 1 -- 172.21.15.70:0/1095801926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 msgr2=0x556a7d4d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:47.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.459+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/1095801926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4d18b0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009780 tx=0x7f36a00349f0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.462+0000 7f36b4bb0740 1 -- 172.21.15.70:0/1095801926 shutdown_connections 2024-04-01T01:47:47.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.462+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/1095801926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d6ff0 0x556a7d4d9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.462+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/1095801926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 0x556a7d4d6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.462+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/1095801926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4d18b0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.462+0000 7f36b4bb0740 1 -- 172.21.15.70:0/1095801926 >> 172.21.15.70:0/1095801926 conn(0x556a7d4cca00 msgr2=0x556a7d4cce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:47.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.463+0000 7f36b4bb0740 1 -- 172.21.15.70:0/1095801926 shutdown_connections 2024-04-01T01:47:47.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.463+0000 7f36b4bb0740 1 -- 172.21.15.70:0/1095801926 wait complete. 2024-04-01T01:47:47.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.464+0000 7f36b4bb0740 1 -- start start 2024-04-01T01:47:47.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.464+0000 7f36b4bb0740 1 Processor -- start 2024-04-01T01:47:47.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.464+0000 7f36b4bb0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4db930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.464+0000 7f36b4bb0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 0x556a7d4dbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.464+0000 7f36b4bb0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d6ff0 0x556a7d4e2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.465+0000 7f36b4bb0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556a7d4d9f30 con 0x556a7d4d1e80 2024-04-01T01:47:47.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.465+0000 7f36b4bb0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x556a7d3d44a0 con 0x556a7d4d1490 2024-04-01T01:47:47.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.465+0000 7f36b4bb0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556a7d4da0b0 con 0x556a7d4d6ff0 2024-04-01T01:47:47.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.465+0000 7f36b4bb0740 10 client.0 _ll_register_callbacks cb 0x556a7d3d7d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:47.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.466+0000 7f36aaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4db930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.466+0000 7f36aaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4db930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33714/0 (socket says 172.21.15.70:33714) 2024-04-01T01:47:47.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.466+0000 7f36aaffd640 1 -- 172.21.15.70:0/1672231447 learned_addr learned my addr 172.21.15.70:0/1672231447 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:47.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.466+0000 7f36aaffd640 1 --2- 172.21.15.70:0/1672231447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d6ff0 0x556a7d4e2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.467+0000 7f36ab7fe640 1 --2- 172.21.15.70:0/1672231447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 0x556a7d4dbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.468+0000 7f36ab7fe640 1 -- 172.21.15.70:0/1672231447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d6ff0 msgr2=0x556a7d4e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.468+0000 7f36ab7fe640 1 --2- 172.21.15.70:0/1672231447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d6ff0 0x556a7d4e2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.468+0000 7f36ab7fe640 1 -- 172.21.15.70:0/1672231447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 msgr2=0x556a7d4db930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.468+0000 7f36ab7fe640 1 --2- 172.21.15.70:0/1672231447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4db930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.468+0000 7f36ab7fe640 1 -- 172.21.15.70:0/1672231447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a7d2147d0 con 0x556a7d4d1e80 2024-04-01T01:47:47.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.468+0000 7f36ab7fe640 1 --2- 172.21.15.70:0/1672231447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 0x556a7d4dbe70 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x556a7d4d2280 tx=0x7f36a400be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce286b9046c5e3a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:47.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.470+0000 7f368affd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:47:47.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.470+0000 7f368affd640 1 -- 172.21.15.70:0/1672231447 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a401b060 con 0x556a7d4d1e80 2024-04-01T01:47:47.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.470+0000 7f36b4bb0740 1 -- 172.21.15.70:0/1672231447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36a003a040 con 0x556a7d4d1e80 2024-04-01T01:47:47.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.470+0000 7f368affd640 1 -- 172.21.15.70:0/1672231447 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f36a40078c0 con 0x556a7d4d1e80 2024-04-01T01:47:47.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.470+0000 7f368affd640 1 -- 172.21.15.70:0/1672231447 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f36a4022050 con 0x556a7d4d1e80 2024-04-01T01:47:47.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.470+0000 7f36b4bb0740 10 client.19542 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:47.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.470+0000 7f36b4bb0740 1 -- 172.21.15.70:0/1672231447 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556a7d14a3f0 con 0x556a7d4d1e80 2024-04-01T01:47:47.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.471+0000 7f368affd640 1 -- 172.21.15.70:0/1672231447 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36a003a040 con 0x556a7d4d1e80 2024-04-01T01:47:47.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.471+0000 7f368affd640 1 -- 172.21.15.70:0/1672231447 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f36a4014b70 con 0x556a7d4d1e80 2024-04-01T01:47:47.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.472+0000 7f36b4bb0740 10 client.19542 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:47.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.472+0000 7f36b4bb0740 10 client.19542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:47.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.472+0000 7f36b4bb0740 20 client.19542 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:47.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.472+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/1672231447 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556a7d4e7870 0x556a7d4e9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.472+0000 7f36b4bb0740 4 client.19542 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.472+0000 7f36b4bb0740 1 -- 172.21.15.70:0/1672231447 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a7d4436c0 con 0x556a7d4e7870 2024-04-01T01:47:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.473+0000 7f36aaffd640 1 --2- 172.21.15.70:0/1672231447 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556a7d4e7870 0x556a7d4e9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.473+0000 7f36aaffd640 1 --2- 172.21.15.70:0/1672231447 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556a7d4e7870 0x556a7d4e9cb0 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6c31d8aac887a6f server_cookie=65deb82eeaa2fb53 in_seq=0 out_seq=0 2024-04-01T01:47:47.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.473+0000 7f368affd640 10 client.19542 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:47.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.477+0000 7f368affd640 1 -- 172.21.15.70:0/1672231447 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556a7d4436c0 con 0x556a7d4e7870 2024-04-01T01:47:47.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.477+0000 7f368affd640 10 client.19542 handle_command_reply: tid=0 2024-04-01T01:47:47.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.478+0000 7f36b4bb0740 1 client.19542 shutdown 2024-04-01T01:47:47.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.479+0000 7f36b4bb0740 1 -- 172.21.15.70:0/1672231447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 msgr2=0x556a7d4dbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:47.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.479+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/1672231447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 0x556a7d4dbe70 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x556a7d4d2280 tx=0x7f36a400be90 comp rx=0 tx=0).stop 2024-04-01T01:47:47.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.480+0000 7f36b4bb0740 1 -- 172.21.15.70:0/1672231447 shutdown_connections 2024-04-01T01:47:47.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.480+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/1672231447 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556a7d4e7870 0x556a7d4e9cb0 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.480+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/1672231447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d6ff0 0x556a7d4e2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.480+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/1672231447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 0x556a7d4dbe70 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.480+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/1672231447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4db930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.480+0000 7f36b4bb0740 1 -- 172.21.15.70:0/1672231447 >> 172.21.15.70:0/1672231447 conn(0x556a7d4cca00 msgr2=0x556a7d4cbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:47.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.480+0000 7f368affd640 0 client.19542 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:47.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.480+0000 7f36b4bb0740 1 -- 172.21.15.70:0/1672231447 shutdown_connections 2024-04-01T01:47:47.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.480+0000 7f36b4bb0740 1 -- 172.21.15.70:0/1672231447 wait complete. 2024-04-01T01:47:47.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.481+0000 7f36b4bb0740 20 client.19542 trim_cache size 0 max 16384 2024-04-01T01:47:47.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.513+0000 7f36b4bb0740 1 Processor -- start 2024-04-01T01:47:47.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.513+0000 7f36b4bb0740 1 -- start start 2024-04-01T01:47:47.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.514+0000 7f36b4bb0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4d3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.514+0000 7f36b4bb0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 0x556a7d4d38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.514+0000 7f36b4bb0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d6ff0 0x556a7d4e2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.514+0000 7f36b4bb0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac1b0aa0 con 0x556a7d4d1e80 2024-04-01T01:47:47.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.514+0000 7f36b4bb0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac1b25e0 con 0x556a7d4d1490 2024-04-01T01:47:47.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.514+0000 7f36b4bb0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac1b1170 con 0x556a7d4d6ff0 2024-04-01T01:47:47.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.514+0000 7f36aaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4d3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.514+0000 7f36ab7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 0x556a7d4d38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.514+0000 7f36aaffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4d3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33724/0 (socket says 172.21.15.70:33724) 2024-04-01T01:47:47.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.514+0000 7f36aaffd640 1 -- 172.21.15.70:0/2208226649 learned_addr learned my addr 172.21.15.70:0/2208226649 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:47.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.514+0000 7f36aaffd640 1 --2- 172.21.15.70:0/2208226649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d6ff0 0x556a7d4e2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.515+0000 7f36aaffd640 1 -- 172.21.15.70:0/2208226649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d6ff0 msgr2=0x556a7d4e2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:47.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.515+0000 7f36aaffd640 1 --2- 172.21.15.70:0/2208226649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d6ff0 0x556a7d4e2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.515+0000 7f36aaffd640 1 -- 172.21.15.70:0/2208226649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 msgr2=0x556a7d4d38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:47.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.515+0000 7f36aaffd640 1 --2- 172.21.15.70:0/2208226649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 0x556a7d4d38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.515+0000 7f36aaffd640 1 -- 172.21.15.70:0/2208226649 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a7d4c8280 con 0x556a7d4d1490 2024-04-01T01:47:47.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.515+0000 7f36aaffd640 1 --2- 172.21.15.70:0/2208226649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4d3380 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f36a0034920 tx=0x7f36a0005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a47adb3669cc5a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:47.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.515+0000 7f36917fa640 1 -- 172.21.15.70:0/2208226649 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a0039040 con 0x556a7d4d1490 2024-04-01T01:47:47.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.515+0000 7f36917fa640 1 -- 172.21.15.70:0/2208226649 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36a0034680 con 0x556a7d4d1490 2024-04-01T01:47:47.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.515+0000 7f36917fa640 1 -- 172.21.15.70:0/2208226649 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a0004b50 con 0x556a7d4d1490 2024-04-01T01:47:47.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.517+0000 7f36b4bb0740 1 -- 172.21.15.70:0/2208226649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 msgr2=0x556a7d4d3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:47.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.517+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/2208226649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4d3380 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f36a0034920 tx=0x7f36a0005f90 comp rx=0 tx=0).stop 2024-04-01T01:47:47.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.517+0000 7f36b4bb0740 1 -- 172.21.15.70:0/2208226649 shutdown_connections 2024-04-01T01:47:47.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.517+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/2208226649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d6ff0 0x556a7d4e2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.517+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/2208226649 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d1e80 0x556a7d4d38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.517+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/2208226649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1490 0x556a7d4d3380 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.517+0000 7f36b4bb0740 1 -- 172.21.15.70:0/2208226649 >> 172.21.15.70:0/2208226649 conn(0x556a7d4cca00 msgr2=0x556a7d4db910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:47.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.517+0000 7f36b4bb0740 1 -- 172.21.15.70:0/2208226649 shutdown_connections 2024-04-01T01:47:47.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.517+0000 7f36b4bb0740 1 -- 172.21.15.70:0/2208226649 wait complete. 2024-04-01T01:47:47.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.517+0000 7f36b4bb0740 1 -- start start 2024-04-01T01:47:47.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.517+0000 7f36b4bb0740 1 Processor -- start 2024-04-01T01:47:47.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.518+0000 7f36b4bb0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d1490 0x556a7d4dfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.518+0000 7f36b4bb0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1e80 0x556a7d4e0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.518+0000 7f36b4bb0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d6ff0 0x556a7d4e14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.518+0000 7f36aaffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d1490 0x556a7d4dfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.518+0000 7f36aaffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d1490 0x556a7d4dfff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42400/0 (socket says 172.21.15.70:42400) 2024-04-01T01:47:47.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.518+0000 7f36ab7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1e80 0x556a7d4e0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.518+0000 7f36ab7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1e80 0x556a7d4e0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33740/0 (socket says 172.21.15.70:33740) 2024-04-01T01:47:47.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.519+0000 7f36aaffd640 1 -- 172.21.15.70:0/3799672015 learned_addr learned my addr 172.21.15.70:0/3799672015 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:47.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.519+0000 7f36b4bb0740 1 -- 172.21.15.70:0/3799672015 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac1b1f10 con 0x556a7d4d6ff0 2024-04-01T01:47:47.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.519+0000 7f36b4bb0740 1 -- 172.21.15.70:0/3799672015 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac1b1840 con 0x556a7d4d1e80 2024-04-01T01:47:47.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.519+0000 7f36b4bb0740 1 -- 172.21.15.70:0/3799672015 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac1aeb80 con 0x556a7d4d1490 2024-04-01T01:47:47.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.519+0000 7f36aaffd640 1 --2- 172.21.15.70:0/3799672015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d6ff0 0x556a7d4e14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.519+0000 7f36aaffd640 1 -- 172.21.15.70:0/3799672015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1e80 msgr2=0x556a7d4e0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:47.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.519+0000 7f36aaffd640 1 --2- 172.21.15.70:0/3799672015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1e80 0x556a7d4e0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.519+0000 7f36aaffd640 1 -- 172.21.15.70:0/3799672015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d6ff0 msgr2=0x556a7d4e14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:47.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.519+0000 7f36aaffd640 1 --2- 172.21.15.70:0/3799672015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d6ff0 0x556a7d4e14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.519+0000 7f36aaffd640 1 -- 172.21.15.70:0/3799672015 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a7d10a0f0 con 0x556a7d4d1490 2024-04-01T01:47:47.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.520+0000 7f36b4bb0740 10 client.0 _ll_register_callbacks cb 0x7f36a40095b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:47.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.520+0000 7f36aaffd640 1 --2- 172.21.15.70:0/3799672015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d1490 0x556a7d4dfff0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f36a002f020 tx=0x7f36a002bb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=237611332aeeb96a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:47.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.520+0000 7f368affd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:47:47.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.520+0000 7f36b4bb0740 1 -- 172.21.15.70:0/3799672015 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36a0034680 con 0x556a7d4d1490 2024-04-01T01:47:47.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.520+0000 7f368affd640 1 -- 172.21.15.70:0/3799672015 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a0064050 con 0x556a7d4d1490 2024-04-01T01:47:47.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.520+0000 7f368affd640 1 -- 172.21.15.70:0/3799672015 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f36a002c530 con 0x556a7d4d1490 2024-04-01T01:47:47.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.520+0000 7f368affd640 1 -- 172.21.15.70:0/3799672015 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f36a0039030 con 0x556a7d4d1490 2024-04-01T01:47:47.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.520+0000 7f36b4bb0740 10 client.29392 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:47.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.521+0000 7f36b4bb0740 1 -- 172.21.15.70:0/3799672015 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=31}) -- 0x556a7d4e28b0 con 0x556a7d4d1490 2024-04-01T01:47:47.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.521+0000 7f368affd640 1 -- 172.21.15.70:0/3799672015 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f36a0067020 con 0x556a7d4d1490 2024-04-01T01:47:47.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.521+0000 7f36b4bb0740 10 client.29392 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:47.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.521+0000 7f36b4bb0740 10 client.29392 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:47.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.521+0000 7f36b4bb0740 20 client.29392 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:47.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.521+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/3799672015 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556a7d4e2d00 0x556a7d54fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.521+0000 7f36b4bb0740 4 client.29392 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:47:47.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.521+0000 7f36b4bb0740 1 -- 172.21.15.70:0/3799672015 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f36a0034680 con 0x556a7d4e2d00 2024-04-01T01:47:47.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.522+0000 7f36ab7fe640 1 --2- 172.21.15.70:0/3799672015 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556a7d4e2d00 0x556a7d54fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.522+0000 7f36ab7fe640 1 --2- 172.21.15.70:0/3799672015 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556a7d4e2d00 0x556a7d54fc00 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=def47b978def811 server_cookie=e844ecb669139de in_seq=0 out_seq=0 2024-04-01T01:47:47.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.522+0000 7f368affd640 10 client.29392 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:47.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.523+0000 7f368affd640 1 -- 172.21.15.70:0/3799672015 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f36a0034680 con 0x556a7d4e2d00 2024-04-01T01:47:47.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.523+0000 7f368affd640 10 client.29392 handle_command_reply: tid=0 2024-04-01T01:47:47.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.523+0000 7f36b4bb0740 1 client.29392 shutdown 2024-04-01T01:47:47.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.525+0000 7f36b4bb0740 1 -- 172.21.15.70:0/3799672015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d1490 msgr2=0x556a7d4dfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:47.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.525+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/3799672015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d1490 0x556a7d4dfff0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f36a002f020 tx=0x7f36a002bb90 comp rx=0 tx=0).stop 2024-04-01T01:47:47.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.525+0000 7f36b4bb0740 1 -- 172.21.15.70:0/3799672015 shutdown_connections 2024-04-01T01:47:47.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.525+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/3799672015 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x556a7d4e2d00 0x556a7d54fc00 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.525+0000 7f368affd640 0 client.29392 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:47.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.525+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/3799672015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556a7d4d6ff0 0x556a7d4e14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.525+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/3799672015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556a7d4d1e80 0x556a7d4e0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.525+0000 7f36b4bb0740 1 --2- 172.21.15.70:0/3799672015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a7d4d1490 0x556a7d4dfff0 secure :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f36a002f020 tx=0x7f36a002bb90 comp rx=0 tx=0).stop 2024-04-01T01:47:47.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.525+0000 7f36b4bb0740 1 -- 172.21.15.70:0/3799672015 >> 172.21.15.70:0/3799672015 conn(0x556a7d4cca00 msgr2=0x556a7d4db850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:47.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.525+0000 7f36b4bb0740 1 -- 172.21.15.70:0/3799672015 shutdown_connections 2024-04-01T01:47:47.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.526+0000 7f36b4bb0740 1 -- 172.21.15.70:0/3799672015 wait complete. 2024-04-01T01:47:47.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.526+0000 7f36b4bb0740 20 client.29392 trim_cache size 0 max 16384 2024-04-01T01:47:47.526 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:47:47.526 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:47:47.527 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:47:47.527 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:47:47.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.527+0000 7f36b0d74640 1 -- 172.21.15.70:0/3549945044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36940a2570 msgr2=0x7f36940a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:47.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.527+0000 7f36b0d74640 1 --2- 172.21.15.70:0/3549945044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36940a2570 0x7f36940a4a30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f369c01efb0 tx=0x7f369c005f10 comp rx=0 tx=0).stop 2024-04-01T01:47:47.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.527+0000 7f36b0d74640 1 -- 172.21.15.70:0/3549945044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36ac101010 msgr2=0x7f36ac19cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:47.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.527+0000 7f36b0d74640 1 --2- 172.21.15.70:0/3549945044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36ac101010 0x7f36ac19cd20 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f369c01e920 tx=0x7f369c01ede0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.527+0000 7f36b0d74640 1 -- 172.21.15.70:0/3549945044 shutdown_connections 2024-04-01T01:47:47.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.527+0000 7f36b0d74640 1 --2- 172.21.15.70:0/3549945044 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36ac101010 0x7f36ac19cd20 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.528+0000 7f36b0d74640 1 --2- 172.21.15.70:0/3549945044 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f36940a2570 0x7f36940a4a30 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.528+0000 7f36b0d74640 1 --2- 172.21.15.70:0/3549945044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36ac100650 0x7f36ac195cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.528+0000 7f36b0d74640 1 --2- 172.21.15.70:0/3549945044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36ac0ff8f0 0x7f36ac195770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.528+0000 7f36b0d74640 1 -- 172.21.15.70:0/3549945044 >> 172.21.15.70:0/3549945044 conn(0x7f36ac0fb240 msgr2=0x7f36ac103a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:47.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.528+0000 7f36b0d74640 1 -- 172.21.15.70:0/3549945044 shutdown_connections 2024-04-01T01:47:47.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.528+0000 7f36b0d74640 1 -- 172.21.15.70:0/3549945044 wait complete. 2024-04-01T01:47:47.547 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:47:47.547 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:47:47.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.758+0000 7f58b45bf640 1 -- 172.21.15.70:0/48282525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ac10da90 msgr2=0x7f58ac10df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:47.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.758+0000 7f58b45bf640 1 --2- 172.21.15.70:0/48282525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ac10da90 0x7f58ac10df40 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f589c009900 tx=0x7f589c02bc80 comp rx=0 tx=0).stop 2024-04-01T01:47:47.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.759+0000 7f58b45bf640 1 -- 172.21.15.70:0/48282525 shutdown_connections 2024-04-01T01:47:47.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.759+0000 7f58b45bf640 1 --2- 172.21.15.70:0/48282525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ac10da90 0x7f58ac10df40 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.759+0000 7f58b45bf640 1 --2- 172.21.15.70:0/48282525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ac10b0c0 0x7f58ac10d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.759+0000 7f58b45bf640 1 --2- 172.21.15.70:0/48282525 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ac108750 0x7f58ac10ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.759+0000 7f58b45bf640 1 -- 172.21.15.70:0/48282525 >> 172.21.15.70:0/48282525 conn(0x7f58ac06e5c0 msgr2=0x7f58ac070a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:47.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.759+0000 7f58b45bf640 1 -- 172.21.15.70:0/48282525 shutdown_connections 2024-04-01T01:47:47.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.763+0000 7f58b45bf640 1 -- 172.21.15.70:0/48282525 wait complete. 2024-04-01T01:47:47.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.763+0000 7f58b45bf640 1 Processor -- start 2024-04-01T01:47:47.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.765+0000 7f58b45bf640 1 -- start start 2024-04-01T01:47:47.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.765+0000 7f58b45bf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ac108750 0x7f58ac1a6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.765+0000 7f58b45bf640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ac10b0c0 0x7f58ac1a6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.765+0000 7f58b45bf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ac10da90 0x7f58ac1adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.765+0000 7f58b45bf640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f58ac11c220 con 0x7f58ac108750 2024-04-01T01:47:47.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.765+0000 7f58b45bf640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f58ac11c0a0 con 0x7f58ac10da90 2024-04-01T01:47:47.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.765+0000 7f58b45bf640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f58ac11c3a0 con 0x7f58ac10b0c0 2024-04-01T01:47:47.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.766+0000 7f58b2b5c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ac10da90 0x7f58ac1adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.766+0000 7f58b2b5c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ac10da90 0x7f58ac1adde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33746/0 (socket says 172.21.15.70:33746) 2024-04-01T01:47:47.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.766+0000 7f58b2b5c640 1 -- 172.21.15.70:0/2798088582 learned_addr learned my addr 172.21.15.70:0/2798088582 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:47.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.766+0000 7f58b1b5a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ac10b0c0 0x7f58ac1a6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.766+0000 7f58b235b640 1 --2- 172.21.15.70:0/2798088582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ac108750 0x7f58ac1a6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.766+0000 7f58b2b5c640 1 -- 172.21.15.70:0/2798088582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ac10b0c0 msgr2=0x7f58ac1a6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:47.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.766+0000 7f58b2b5c640 1 --2- 172.21.15.70:0/2798088582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ac10b0c0 0x7f58ac1a6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.766+0000 7f58b2b5c640 1 -- 172.21.15.70:0/2798088582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ac108750 msgr2=0x7f58ac1a6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:47.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.766+0000 7f58b2b5c640 1 --2- 172.21.15.70:0/2798088582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ac108750 0x7f58ac1a6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:47.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.766+0000 7f58b2b5c640 1 -- 172.21.15.70:0/2798088582 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58ac1ae450 con 0x7f58ac10da90 2024-04-01T01:47:47.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.768+0000 7f58b2b5c640 1 --2- 172.21.15.70:0/2798088582 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ac10da90 0x7f58ac1adde0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f589c036040 tx=0x7f589c0045c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:47.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.769+0000 7f58a37fe640 1 -- 172.21.15.70:0/2798088582 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c042020 con 0x7f58ac10da90 2024-04-01T01:47:47.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.769+0000 7f58a37fe640 1 -- 172.21.15.70:0/2798088582 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f589c005350 con 0x7f58ac10da90 2024-04-01T01:47:47.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.769+0000 7f58a37fe640 1 -- 172.21.15.70:0/2798088582 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c03e820 con 0x7f58ac10da90 2024-04-01T01:47:47.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.769+0000 7f58b45bf640 1 -- 172.21.15.70:0/2798088582 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58ac1ae6e0 con 0x7f58ac10da90 2024-04-01T01:47:47.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.769+0000 7f58b45bf640 1 -- 172.21.15.70:0/2798088582 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58ac1aec00 con 0x7f58ac10da90 2024-04-01T01:47:47.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.771+0000 7f58b45bf640 1 -- 172.21.15.70:0/2798088582 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58ac0710c0 con 0x7f58ac10da90 2024-04-01T01:47:47.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.774+0000 7f58a37fe640 1 -- 172.21.15.70:0/2798088582 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f589c045020 con 0x7f58ac10da90 2024-04-01T01:47:47.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.775+0000 7f58a37fe640 1 --2- 172.21.15.70:0/2798088582 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f58980a26b0 0x7f58980a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:47.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.775+0000 7f58a37fe640 1 -- 172.21.15.70:0/2798088582 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f589c0f10a0 con 0x7f58ac10da90 2024-04-01T01:47:47.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.775+0000 7f58b235b640 1 --2- 172.21.15.70:0/2798088582 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f58980a26b0 0x7f58980a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:47.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.776+0000 7f58b235b640 1 --2- 172.21.15.70:0/2798088582 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f58980a26b0 0x7f58980a4b70 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f58ac072840 tx=0x7f58a400d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:47.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:47.778+0000 7f58a37fe640 1 -- 172.21.15.70:0/2798088582 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f589c04c020 con 0x7f58ac10da90 2024-04-01T01:47:48.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.194+0000 7f58b45bf640 1 -- 172.21.15.70:0/2798088582 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58ac02d170 con 0x7f58ac10da90 2024-04-01T01:47:48.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.198+0000 7f58a37fe640 1 -- 172.21.15.70:0/2798088582 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7f589c04c020 con 0x7f58ac10da90 2024-04-01T01:47:48.198 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:47:48.198 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:47:48.199 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:47:48.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.203+0000 7f58b45bf640 1 -- 172.21.15.70:0/2798088582 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f58980a26b0 msgr2=0x7f58980a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.203+0000 7f58b45bf640 1 --2- 172.21.15.70:0/2798088582 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f58980a26b0 0x7f58980a4b70 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f58ac072840 tx=0x7f58a400d040 comp rx=0 tx=0).stop 2024-04-01T01:47:48.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.203+0000 7f58b45bf640 1 -- 172.21.15.70:0/2798088582 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ac10da90 msgr2=0x7f58ac1adde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.203+0000 7f58b45bf640 1 --2- 172.21.15.70:0/2798088582 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ac10da90 0x7f58ac1adde0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f589c036040 tx=0x7f589c0045c0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.203+0000 7f58b45bf640 1 -- 172.21.15.70:0/2798088582 shutdown_connections 2024-04-01T01:47:48.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.203+0000 7f58b45bf640 1 --2- 172.21.15.70:0/2798088582 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ac10da90 0x7f58ac1adde0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.203+0000 7f58b45bf640 1 --2- 172.21.15.70:0/2798088582 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f58980a26b0 0x7f58980a4b70 secure :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f58ac072840 tx=0x7f58a400d040 comp rx=0 tx=0).stop 2024-04-01T01:47:48.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.203+0000 7f58b45bf640 1 --2- 172.21.15.70:0/2798088582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58ac10b0c0 0x7f58ac1a6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.203+0000 7f58b45bf640 1 --2- 172.21.15.70:0/2798088582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58ac108750 0x7f58ac1a6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.203+0000 7f58b45bf640 1 -- 172.21.15.70:0/2798088582 >> 172.21.15.70:0/2798088582 conn(0x7f58ac06e5c0 msgr2=0x7f58ac10a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:48.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.204+0000 7f58b45bf640 1 -- 172.21.15.70:0/2798088582 shutdown_connections 2024-04-01T01:47:48.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.204+0000 7f58b45bf640 1 -- 172.21.15.70:0/2798088582 wait complete. 2024-04-01T01:47:48.221 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:47:48.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.409+0000 7fb2639bf640 1 -- 172.21.15.70:0/3822133032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb25c108b70 msgr2=0x7fb25c10e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.409+0000 7fb2639bf640 1 --2- 172.21.15.70:0/3822133032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb25c108b70 0x7fb25c10e650 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb250009900 tx=0x7fb25002bc50 comp rx=0 tx=0).stop 2024-04-01T01:47:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.409+0000 7fb2639bf640 1 -- 172.21.15.70:0/3822133032 shutdown_connections 2024-04-01T01:47:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.409+0000 7fb2639bf640 1 --2- 172.21.15.70:0/3822133032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb25c10eb90 0x7fb25c110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.409+0000 7fb2639bf640 1 --2- 172.21.15.70:0/3822133032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb25c108b70 0x7fb25c10e650 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.409+0000 7fb2639bf640 1 --2- 172.21.15.70:0/3822133032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb25c108750 0x7fb25c10c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.409+0000 7fb2639bf640 1 -- 172.21.15.70:0/3822133032 >> 172.21.15.70:0/3822133032 conn(0x7fb25c06e950 msgr2=0x7fb25c070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.410+0000 7fb2639bf640 1 -- 172.21.15.70:0/3822133032 shutdown_connections 2024-04-01T01:47:48.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.410+0000 7fb2639bf640 1 -- 172.21.15.70:0/3822133032 wait complete. 2024-04-01T01:47:48.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.411+0000 7fb2639bf640 1 Processor -- start 2024-04-01T01:47:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.411+0000 7fb2639bf640 1 -- start start 2024-04-01T01:47:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.411+0000 7fb2639bf640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb25c108750 0x7fb25c1a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.411+0000 7fb2639bf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb25c108b70 0x7fb25c1a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.411+0000 7fb2639bf640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb25c10eb90 0x7fb25c1a7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.412+0000 7fb2639bf640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb25c114a70 con 0x7fb25c108b70 2024-04-01T01:47:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.412+0000 7fb2639bf640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb25c1148f0 con 0x7fb25c108750 2024-04-01T01:47:48.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.412+0000 7fb2639bf640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb25c114bf0 con 0x7fb25c10eb90 2024-04-01T01:47:48.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.413+0000 7fb26175b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb25c108750 0x7fb25c1a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.413+0000 7fb26175b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb25c108750 0x7fb25c1a0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33774/0 (socket says 172.21.15.70:33774) 2024-04-01T01:47:48.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.413+0000 7fb26175b640 1 -- 172.21.15.70:0/57883476 learned_addr learned my addr 172.21.15.70:0/57883476 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:48.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.413+0000 7fb26175b640 1 -- 172.21.15.70:0/57883476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb25c10eb90 msgr2=0x7fb25c1a7880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:47:48.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.413+0000 7fb26175b640 1 --2- 172.21.15.70:0/57883476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb25c10eb90 0x7fb25c1a7880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.413+0000 7fb26175b640 1 -- 172.21.15.70:0/57883476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb25c108b70 msgr2=0x7fb25c1a0860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:47:48.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.413+0000 7fb26175b640 1 --2- 172.21.15.70:0/57883476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb25c108b70 0x7fb25c1a0860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.413+0000 7fb26175b640 1 -- 172.21.15.70:0/57883476 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb25c1a7ef0 con 0x7fb25c108750 2024-04-01T01:47:48.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.414+0000 7fb26175b640 1 --2- 172.21.15.70:0/57883476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb25c108750 0x7fb25c1a0320 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb25400ea80 tx=0x7fb25400ef40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:48.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.417+0000 7fb24e7fc640 1 -- 172.21.15.70:0/57883476 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25400ce40 con 0x7fb25c108750 2024-04-01T01:47:48.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.417+0000 7fb24e7fc640 1 -- 172.21.15.70:0/57883476 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb254005020 con 0x7fb25c108750 2024-04-01T01:47:48.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.417+0000 7fb2639bf640 1 -- 172.21.15.70:0/57883476 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb25c1a81e0 con 0x7fb25c108750 2024-04-01T01:47:48.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.417+0000 7fb2639bf640 1 -- 172.21.15.70:0/57883476 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb25c1a86d0 con 0x7fb25c108750 2024-04-01T01:47:48.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.417+0000 7fb24e7fc640 1 -- 172.21.15.70:0/57883476 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2540116b0 con 0x7fb25c108750 2024-04-01T01:47:48.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.420+0000 7fb24e7fc640 1 -- 172.21.15.70:0/57883476 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb254011910 con 0x7fb25c108750 2024-04-01T01:47:48.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.421+0000 7fb24e7fc640 1 --2- 172.21.15.70:0/57883476 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2480a26a0 0x7fb2480a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.422+0000 7fb24e7fc640 1 -- 172.21.15.70:0/57883476 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb2540cfd00 con 0x7fb25c108750 2024-04-01T01:47:48.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.422+0000 7fb266595740 1 Processor -- start 2024-04-01T01:47:48.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.422+0000 7fb266595740 1 -- start start 2024-04-01T01:47:48.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.423+0000 7fb266595740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa4490 0x55609bfa48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.423+0000 7fb266595740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa4e80 0x55609bfa9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.423+0000 7fb266595740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa9ff0 0x55609bfac480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.423+0000 7fb266595740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55609bf091b0 con 0x55609bfa9ff0 2024-04-01T01:47:48.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.423+0000 7fb266595740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55609beb1c00 con 0x55609bfa4e80 2024-04-01T01:47:48.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.423+0000 7fb266595740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55609bf05250 con 0x55609bfa4490 2024-04-01T01:47:48.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.423+0000 7fb260f5a640 1 --2- 172.21.15.70:0/57883476 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2480a26a0 0x7fb2480a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.423+0000 7fb26175b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa4490 0x55609bfa48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.423+0000 7fb26175b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa4490 0x55609bfa48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42428/0 (socket says 172.21.15.70:42428) 2024-04-01T01:47:48.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.423+0000 7fb26175b640 1 -- 172.21.15.70:0/1994062361 learned_addr learned my addr 172.21.15.70:0/1994062361 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:48.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.424+0000 7fb261f5c640 1 --2- 172.21.15.70:0/1994062361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa9ff0 0x55609bfac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.424+0000 7fb26175b640 1 -- 172.21.15.70:0/1994062361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa4e80 msgr2=0x55609bfa9980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:47:48.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.424+0000 7fb26175b640 1 --2- 172.21.15.70:0/1994062361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa4e80 0x55609bfa9980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.424+0000 7fb26175b640 1 -- 172.21.15.70:0/1994062361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa9ff0 msgr2=0x55609bfac480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:48.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.424+0000 7fb26175b640 1 --2- 172.21.15.70:0/1994062361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa9ff0 0x55609bfac480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.424+0000 7fb26175b640 1 -- 172.21.15.70:0/1994062361 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55609bed13d0 con 0x55609bfa4490 2024-04-01T01:47:48.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.424+0000 7fb261f5c640 1 --2- 172.21.15.70:0/1994062361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa9ff0 0x55609bfac480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:47:48.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.428+0000 7fb26175b640 1 --2- 172.21.15.70:0/1994062361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa4490 0x55609bfa48b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb25409c090 tx=0x7fb25409c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ac5301a249fe176 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:48.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.428+0000 7fb2477fe640 1 -- 172.21.15.70:0/1994062361 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25401f040 con 0x55609bfa4490 2024-04-01T01:47:48.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.428+0000 7fb2477fe640 1 -- 172.21.15.70:0/1994062361 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2540bc0c0 con 0x55609bfa4490 2024-04-01T01:47:48.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.430+0000 7fb266595740 1 -- 172.21.15.70:0/1994062361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa4490 msgr2=0x55609bfa48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.430+0000 7fb266595740 1 --2- 172.21.15.70:0/1994062361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa4490 0x55609bfa48b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fb25409c090 tx=0x7fb25409c4d0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.431+0000 7fb266595740 1 -- 172.21.15.70:0/1994062361 shutdown_connections 2024-04-01T01:47:48.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.431+0000 7fb266595740 1 --2- 172.21.15.70:0/1994062361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa9ff0 0x55609bfac480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.431+0000 7fb266595740 1 --2- 172.21.15.70:0/1994062361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa4e80 0x55609bfa9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.431+0000 7fb266595740 1 --2- 172.21.15.70:0/1994062361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa4490 0x55609bfa48b0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.431+0000 7fb266595740 1 -- 172.21.15.70:0/1994062361 >> 172.21.15.70:0/1994062361 conn(0x55609bf9fa00 msgr2=0x55609bf9fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:48.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.432+0000 7fb266595740 1 -- 172.21.15.70:0/1994062361 shutdown_connections 2024-04-01T01:47:48.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.434+0000 7fb266595740 1 -- 172.21.15.70:0/1994062361 wait complete. 2024-04-01T01:47:48.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.434+0000 7fb260f5a640 1 --2- 172.21.15.70:0/57883476 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2480a26a0 0x7fb2480a4b60 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb250009230 tx=0x7fb250005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:48.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.434+0000 7fb266595740 1 -- start start 2024-04-01T01:47:48.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.434+0000 7fb266595740 1 Processor -- start 2024-04-01T01:47:48.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.434+0000 7fb266595740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa4490 0x55609bfae920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb266595740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa4e80 0x55609bfaee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb266595740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa9ff0 0x55609bfb5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb266595740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55609bfacff0 con 0x55609bfa4490 2024-04-01T01:47:48.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb266595740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55609bea7470 con 0x55609bfa4e80 2024-04-01T01:47:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb266595740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55609bfad170 con 0x55609bfa9ff0 2024-04-01T01:47:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb266595740 10 client.0 _ll_register_callbacks cb 0x55609bec4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb261f5c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa9ff0 0x55609bfb5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb260f5a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa4e80 0x55609bfaee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb261f5c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa9ff0 0x55609bfb5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42430/0 (socket says 172.21.15.70:42430) 2024-04-01T01:47:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb261f5c640 1 -- 172.21.15.70:0/3159284323 learned_addr learned my addr 172.21.15.70:0/3159284323 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb26175b640 1 --2- 172.21.15.70:0/3159284323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa4490 0x55609bfae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb260f5a640 1 -- 172.21.15.70:0/3159284323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa9ff0 msgr2=0x55609bfb5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb260f5a640 1 --2- 172.21.15.70:0/3159284323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa9ff0 0x55609bfb5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb260f5a640 1 -- 172.21.15.70:0/3159284323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa4490 msgr2=0x55609bfae920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb260f5a640 1 --2- 172.21.15.70:0/3159284323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa4490 0x55609bfae920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb260f5a640 1 -- 172.21.15.70:0/3159284323 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55609bc1d3f0 con 0x55609bfa4e80 2024-04-01T01:47:48.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.435+0000 7fb261f5c640 1 --2- 172.21.15.70:0/3159284323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa9ff0 0x55609bfb5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:47:48.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.436+0000 7fb260f5a640 1 --2- 172.21.15.70:0/3159284323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa4e80 0x55609bfaee60 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb25002c800 tx=0x7fb25002c5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1563b51a34b4146 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:48.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.436+0000 7fb244ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:48.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.436+0000 7fb244ff9640 1 -- 172.21.15.70:0/3159284323 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb250003800 con 0x55609bfa4e80 2024-04-01T01:47:48.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.436+0000 7fb266595740 1 -- 172.21.15.70:0/3159284323 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2540bc0c0 con 0x55609bfa4e80 2024-04-01T01:47:48.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.436+0000 7fb244ff9640 1 -- 172.21.15.70:0/3159284323 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb2500039a0 con 0x55609bfa4e80 2024-04-01T01:47:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.436+0000 7fb244ff9640 1 -- 172.21.15.70:0/3159284323 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb250055050 con 0x55609bfa4e80 2024-04-01T01:47:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.436+0000 7fb266595740 10 client.29654 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.436+0000 7fb266595740 1 -- 172.21.15.70:0/3159284323 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55609bce77d0 con 0x55609bfa4e80 2024-04-01T01:47:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.437+0000 7fb244ff9640 1 -- 172.21.15.70:0/3159284323 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2540bc0c0 con 0x55609bfa4e80 2024-04-01T01:47:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.438+0000 7fb244ff9640 1 -- 172.21.15.70:0/3159284323 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb250034440 con 0x55609bfa4e80 2024-04-01T01:47:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.438+0000 7fb266595740 10 client.29654 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.438+0000 7fb266595740 10 client.29654 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:47:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.438+0000 7fb266595740 20 client.29654 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.438+0000 7fb266595740 1 --2- 172.21.15.70:0/3159284323 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55609bfbeb50 0x55609bfc0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.438+0000 7fb266595740 4 client.29654 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.438+0000 7fb266595740 1 -- 172.21.15.70:0/3159284323 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55609bf166f0 con 0x55609bfbeb50 2024-04-01T01:47:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.438+0000 7fb26175b640 1 --2- 172.21.15.70:0/3159284323 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55609bfbeb50 0x55609bfc0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.442+0000 7fb26175b640 1 --2- 172.21.15.70:0/3159284323 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55609bfbeb50 0x55609bfc0fd0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=47ddec6d56236ed2 server_cookie=7e57ae91e3aef60b in_seq=0 out_seq=0 2024-04-01T01:47:48.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.443+0000 7fb244ff9640 10 client.29654 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:47:48.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.444+0000 7fb244ff9640 1 -- 172.21.15.70:0/3159284323 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55609bf166f0 con 0x55609bfbeb50 2024-04-01T01:47:48.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.444+0000 7fb244ff9640 10 client.29654 handle_command_reply: tid=0 2024-04-01T01:47:48.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.445+0000 7fb266595740 1 client.29654 shutdown 2024-04-01T01:47:48.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.446+0000 7fb266595740 1 -- 172.21.15.70:0/3159284323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa4e80 msgr2=0x55609bfaee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.446+0000 7fb266595740 1 --2- 172.21.15.70:0/3159284323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa4e80 0x55609bfaee60 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb25002c800 tx=0x7fb25002c5f0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.446+0000 7fb266595740 1 -- 172.21.15.70:0/3159284323 shutdown_connections 2024-04-01T01:47:48.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.446+0000 7fb266595740 1 --2- 172.21.15.70:0/3159284323 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55609bfbeb50 0x55609bfc0fd0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.446+0000 7fb266595740 1 --2- 172.21.15.70:0/3159284323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa9ff0 0x55609bfb5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.446+0000 7fb266595740 1 --2- 172.21.15.70:0/3159284323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa4e80 0x55609bfaee60 secure :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb25002c800 tx=0x7fb25002c5f0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.446+0000 7fb266595740 1 --2- 172.21.15.70:0/3159284323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa4490 0x55609bfae920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.446+0000 7fb266595740 1 -- 172.21.15.70:0/3159284323 >> 172.21.15.70:0/3159284323 conn(0x55609bf9fa00 msgr2=0x55609bf9ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:48.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.447+0000 7fb244ff9640 0 client.29654 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:47:48.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.449+0000 7fb266595740 1 -- 172.21.15.70:0/3159284323 shutdown_connections 2024-04-01T01:47:48.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.450+0000 7fb266595740 1 -- 172.21.15.70:0/3159284323 wait complete. 2024-04-01T01:47:48.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.453+0000 7fb266595740 20 client.29654 trim_cache size 0 max 16384 2024-04-01T01:47:48.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.494+0000 7fb266595740 1 Processor -- start 2024-04-01T01:47:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.494+0000 7fb266595740 1 -- start start 2024-04-01T01:47:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.494+0000 7fb266595740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa4490 0x55609bfbb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.494+0000 7fb266595740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa9ff0 0x55609bfa5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.494+0000 7fb266595740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa62a0 0x55609bfa6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.494+0000 7fb266595740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb24000fd00 con 0x55609bfa4490 2024-04-01T01:47:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.494+0000 7fb266595740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb240011840 con 0x55609bfa9ff0 2024-04-01T01:47:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.494+0000 7fb266595740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb2400103d0 con 0x55609bfa62a0 2024-04-01T01:47:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.495+0000 7fb26175b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa4490 0x55609bfbb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.495+0000 7fb260f5a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa9ff0 0x55609bfa5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.495+0000 7fb26175b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa4490 0x55609bfbb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41306/0 (socket says 172.21.15.70:41306) 2024-04-01T01:47:48.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.495+0000 7fb26175b640 1 -- 172.21.15.70:0/1115223611 learned_addr learned my addr 172.21.15.70:0/1115223611 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:48.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.495+0000 7fb261f5c640 1 --2- 172.21.15.70:0/1115223611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa62a0 0x55609bfa6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.496+0000 7fb26175b640 1 -- 172.21.15.70:0/1115223611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa62a0 msgr2=0x55609bfa6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:48.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.496+0000 7fb26175b640 1 --2- 172.21.15.70:0/1115223611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa62a0 0x55609bfa6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.496+0000 7fb26175b640 1 -- 172.21.15.70:0/1115223611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa9ff0 msgr2=0x55609bfa5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:48.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.496+0000 7fb26175b640 1 --2- 172.21.15.70:0/1115223611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa9ff0 0x55609bfa5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.496+0000 7fb26175b640 1 -- 172.21.15.70:0/1115223611 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55609bfad2e0 con 0x55609bfa4490 2024-04-01T01:47:48.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.496+0000 7fb26175b640 1 --2- 172.21.15.70:0/1115223611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa4490 0x55609bfbb020 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fb254005960 tx=0x7fb25409a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f0da1acf3aca483 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:48.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.497+0000 7fb246ffd640 1 -- 172.21.15.70:0/1115223611 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25401f040 con 0x55609bfa4490 2024-04-01T01:47:48.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.497+0000 7fb246ffd640 1 -- 172.21.15.70:0/1115223611 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2540bc0c0 con 0x55609bfa4490 2024-04-01T01:47:48.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.498+0000 7fb246ffd640 1 -- 172.21.15.70:0/1115223611 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2540a4b70 con 0x55609bfa4490 2024-04-01T01:47:48.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.498+0000 7fb266595740 1 -- 172.21.15.70:0/1115223611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa4490 msgr2=0x55609bfbb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.499+0000 7fb266595740 1 --2- 172.21.15.70:0/1115223611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa4490 0x55609bfbb020 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fb254005960 tx=0x7fb25409a5d0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.499+0000 7fb266595740 1 -- 172.21.15.70:0/1115223611 shutdown_connections 2024-04-01T01:47:48.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.499+0000 7fb266595740 1 --2- 172.21.15.70:0/1115223611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa62a0 0x55609bfa6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.499+0000 7fb266595740 1 --2- 172.21.15.70:0/1115223611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa9ff0 0x55609bfa5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.499+0000 7fb266595740 1 --2- 172.21.15.70:0/1115223611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa4490 0x55609bfbb020 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.499+0000 7fb266595740 1 -- 172.21.15.70:0/1115223611 >> 172.21.15.70:0/1115223611 conn(0x55609bf9fa00 msgr2=0x55609bf9fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:48.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.501+0000 7fb266595740 1 -- 172.21.15.70:0/1115223611 shutdown_connections 2024-04-01T01:47:48.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.501+0000 7fb266595740 1 -- 172.21.15.70:0/1115223611 wait complete. 2024-04-01T01:47:48.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.501+0000 7fb266595740 1 -- start start 2024-04-01T01:47:48.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.501+0000 7fb266595740 1 Processor -- start 2024-04-01T01:47:48.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.502+0000 7fb266595740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa4490 0x55609bfb1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.502+0000 7fb266595740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa62a0 0x55609bfb4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.502+0000 7fb266595740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa9ff0 0x55609bfb48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.502+0000 7fb266595740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb240011170 con 0x55609bfa4490 2024-04-01T01:47:48.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.502+0000 7fb266595740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb240010aa0 con 0x55609bfa9ff0 2024-04-01T01:47:48.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.502+0000 7fb266595740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb24000dce0 con 0x55609bfa62a0 2024-04-01T01:47:48.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.502+0000 7fb266595740 10 client.0 _ll_register_callbacks cb 0x7fb2500095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:48.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.502+0000 7fb260f5a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa62a0 0x55609bfb4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.502+0000 7fb261f5c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa9ff0 0x55609bfb48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.502+0000 7fb261f5c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa9ff0 0x55609bfb48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33810/0 (socket says 172.21.15.70:33810) 2024-04-01T01:47:48.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.502+0000 7fb261f5c640 1 -- 172.21.15.70:0/185315422 learned_addr learned my addr 172.21.15.70:0/185315422 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:48.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.503+0000 7fb261f5c640 1 -- 172.21.15.70:0/185315422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa62a0 msgr2=0x55609bfb4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:48.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.503+0000 7fb261f5c640 1 --2- 172.21.15.70:0/185315422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa62a0 0x55609bfb4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.503+0000 7fb261f5c640 1 -- 172.21.15.70:0/185315422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa4490 msgr2=0x55609bfb1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:48.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.503+0000 7fb261f5c640 1 --2- 172.21.15.70:0/185315422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa4490 0x55609bfb1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.503+0000 7fb261f5c640 1 -- 172.21.15.70:0/185315422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55609bfb4e60 con 0x55609bfa9ff0 2024-04-01T01:47:48.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.503+0000 7fb261f5c640 1 --2- 172.21.15.70:0/185315422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa9ff0 0x55609bfb48d0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb258035970 tx=0x7fb258035e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=930be72a18bdb284 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:48.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.503+0000 7fb244ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:48.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.503+0000 7fb244ff9640 1 -- 172.21.15.70:0/185315422 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb258033c40 con 0x55609bfa9ff0 2024-04-01T01:47:48.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.503+0000 7fb244ff9640 1 -- 172.21.15.70:0/185315422 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fb258033de0 con 0x55609bfa9ff0 2024-04-01T01:47:48.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.504+0000 7fb266595740 1 -- 172.21.15.70:0/185315422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2540bc0c0 con 0x55609bfa9ff0 2024-04-01T01:47:48.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.504+0000 7fb244ff9640 1 -- 172.21.15.70:0/185315422 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb258037430 con 0x55609bfa9ff0 2024-04-01T01:47:48.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.504+0000 7fb266595740 10 client.29660 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:48.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.504+0000 7fb266595740 1 -- 172.21.15.70:0/185315422 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55609bfb6230 con 0x55609bfa9ff0 2024-04-01T01:47:48.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.505+0000 7fb244ff9640 1 -- 172.21.15.70:0/185315422 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb2540bc0c0 con 0x55609bfa9ff0 2024-04-01T01:47:48.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.505+0000 7fb244ff9640 1 -- 172.21.15.70:0/185315422 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fb25804b750 con 0x55609bfa9ff0 2024-04-01T01:47:48.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.505+0000 7fb266595740 10 client.29660 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:48.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.505+0000 7fb266595740 10 client.29660 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:47:48.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.505+0000 7fb266595740 20 client.29660 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:48.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.505+0000 7fb266595740 1 --2- 172.21.15.70:0/185315422 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55609c029eb0 0x55609c02c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.505+0000 7fb266595740 4 client.29660 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:47:48.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.505+0000 7fb266595740 1 -- 172.21.15.70:0/185315422 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55609bf10a40 con 0x55609c029eb0 2024-04-01T01:47:48.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.505+0000 7fb26175b640 1 --2- 172.21.15.70:0/185315422 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55609c029eb0 0x55609c02c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.506+0000 7fb244ff9640 10 client.29660 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:47:48.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.506+0000 7fb26175b640 1 --2- 172.21.15.70:0/185315422 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55609c029eb0 0x55609c02c2d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=75abe6bc7ac4e8ef server_cookie=5d3d50c5540fcfd4 in_seq=0 out_seq=0 2024-04-01T01:47:48.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.506+0000 7fb244ff9640 1 -- 172.21.15.70:0/185315422 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55609bf10a40 con 0x55609c029eb0 2024-04-01T01:47:48.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.506+0000 7fb244ff9640 10 client.29660 handle_command_reply: tid=0 2024-04-01T01:47:48.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.507+0000 7fb266595740 1 client.29660 shutdown 2024-04-01T01:47:48.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.507+0000 7fb266595740 1 -- 172.21.15.70:0/185315422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa9ff0 msgr2=0x55609bfb48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.507+0000 7fb266595740 1 --2- 172.21.15.70:0/185315422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa9ff0 0x55609bfb48d0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb258035970 tx=0x7fb258035e30 comp rx=0 tx=0).stop 2024-04-01T01:47:48.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.507+0000 7fb266595740 1 -- 172.21.15.70:0/185315422 shutdown_connections 2024-04-01T01:47:48.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.507+0000 7fb266595740 1 --2- 172.21.15.70:0/185315422 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55609c029eb0 0x55609c02c2d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.507+0000 7fb266595740 1 --2- 172.21.15.70:0/185315422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55609bfa9ff0 0x55609bfb48d0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.507+0000 7fb266595740 1 --2- 172.21.15.70:0/185315422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55609bfa62a0 0x55609bfb4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.507+0000 7fb266595740 1 --2- 172.21.15.70:0/185315422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55609bfa4490 0x55609bfb1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.507+0000 7fb266595740 1 -- 172.21.15.70:0/185315422 >> 172.21.15.70:0/185315422 conn(0x55609bf9fa00 msgr2=0x55609bfa2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:48.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.507+0000 7fb244ff9640 0 client.29660 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:47:48.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.507+0000 7fb266595740 1 -- 172.21.15.70:0/185315422 shutdown_connections 2024-04-01T01:47:48.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.508+0000 7fb266595740 1 -- 172.21.15.70:0/185315422 wait complete. 2024-04-01T01:47:48.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.508+0000 7fb266595740 20 client.29660 trim_cache size 0 max 16384 2024-04-01T01:47:48.509 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:47:48.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.509+0000 7fb2639bf640 1 -- 172.21.15.70:0/57883476 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2480a26a0 msgr2=0x7fb2480a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.509+0000 7fb2639bf640 1 --2- 172.21.15.70:0/57883476 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2480a26a0 0x7fb2480a4b60 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb250009230 tx=0x7fb250005fd0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.509+0000 7fb2639bf640 1 -- 172.21.15.70:0/57883476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb25c108750 msgr2=0x7fb25c1a0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.509+0000 7fb2639bf640 1 --2- 172.21.15.70:0/57883476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb25c108750 0x7fb25c1a0320 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb25400ea80 tx=0x7fb25400ef40 comp rx=0 tx=0).stop 2024-04-01T01:47:48.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.509+0000 7fb2639bf640 1 -- 172.21.15.70:0/57883476 shutdown_connections 2024-04-01T01:47:48.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.509+0000 7fb2639bf640 1 --2- 172.21.15.70:0/57883476 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb2480a26a0 0x7fb2480a4b60 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.509+0000 7fb2639bf640 1 --2- 172.21.15.70:0/57883476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb25c10eb90 0x7fb25c1a7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.509+0000 7fb2639bf640 1 --2- 172.21.15.70:0/57883476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb25c108b70 0x7fb25c1a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.509+0000 7fb2639bf640 1 --2- 172.21.15.70:0/57883476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb25c108750 0x7fb25c1a0320 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.509+0000 7fb2639bf640 1 -- 172.21.15.70:0/57883476 >> 172.21.15.70:0/57883476 conn(0x7fb25c06e950 msgr2=0x7fb25c10b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:48.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.510+0000 7fb2639bf640 1 -- 172.21.15.70:0/57883476 shutdown_connections 2024-04-01T01:47:48.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.510+0000 7fb2639bf640 1 -- 172.21.15.70:0/57883476 wait complete. 2024-04-01T01:47:48.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:47:48.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.735+0000 7f0896ffd640 1 -- 172.21.15.70:0/1627173971 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884004de0 con 0x7f0890094c80 2024-04-01T01:47:48.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.736+0000 7f089e5b5640 1 -- 172.21.15.70:0/1627173971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890094c80 msgr2=0x7f08900950a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.736+0000 7f089e5b5640 1 --2- 172.21.15.70:0/1627173971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890094c80 0x7f08900950a0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f088401a910 tx=0x7f088403ee70 comp rx=0 tx=0).stop 2024-04-01T01:47:48.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.736+0000 7f089e5b5640 1 -- 172.21.15.70:0/1627173971 shutdown_connections 2024-04-01T01:47:48.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.736+0000 7f089e5b5640 1 --2- 172.21.15.70:0/1627173971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0890096830 0x7f089009d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.736+0000 7f089e5b5640 1 --2- 172.21.15.70:0/1627173971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0890095e70 0x7f08900962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.736+0000 7f089e5b5640 1 --2- 172.21.15.70:0/1627173971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890094c80 0x7f08900950a0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.736+0000 7f089e5b5640 1 -- 172.21.15.70:0/1627173971 >> 172.21.15.70:0/1627173971 conn(0x7f08900901d0 msgr2=0x7f0890092630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:48.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.737+0000 7f089e5b5640 1 -- 172.21.15.70:0/1627173971 shutdown_connections 2024-04-01T01:47:48.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.737+0000 7f089e5b5640 1 -- 172.21.15.70:0/1627173971 wait complete. 2024-04-01T01:47:48.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.737+0000 7f089e5b5640 1 Processor -- start 2024-04-01T01:47:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.738+0000 7f089e5b5640 1 -- start start 2024-04-01T01:47:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.738+0000 7f089e5b5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0890094c80 0x7f089012e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.738+0000 7f089e5b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890095e70 0x7f089012e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.738+0000 7f089e5b5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0890096830 0x7f08901357a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.738+0000 7f089e5b5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f089009f760 con 0x7f0890095e70 2024-04-01T01:47:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.738+0000 7f089e5b5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f089009f5e0 con 0x7f0890094c80 2024-04-01T01:47:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.738+0000 7f089e5b5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f089009f8e0 con 0x7f0890096830 2024-04-01T01:47:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.738+0000 7f0897fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0890094c80 0x7f089012e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.738+0000 7f0897fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0890094c80 0x7f089012e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33840/0 (socket says 172.21.15.70:33840) 2024-04-01T01:47:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.738+0000 7f0897fff640 1 -- 172.21.15.70:0/195839170 learned_addr learned my addr 172.21.15.70:0/195839170 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.738+0000 7f08977fe640 1 --2- 172.21.15.70:0/195839170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890095e70 0x7f089012e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.739+0000 7f08977fe640 1 -- 172.21.15.70:0/195839170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0890096830 msgr2=0x7f08901357a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.739+0000 7f08977fe640 1 --2- 172.21.15.70:0/195839170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0890096830 0x7f08901357a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.739+0000 7f08977fe640 1 -- 172.21.15.70:0/195839170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0890094c80 msgr2=0x7f089012e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.739+0000 7f08977fe640 1 --2- 172.21.15.70:0/195839170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0890094c80 0x7f089012e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.739+0000 7f08977fe640 1 -- 172.21.15.70:0/195839170 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0890135e10 con 0x7f0890095e70 2024-04-01T01:47:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.739+0000 7f08977fe640 1 --2- 172.21.15.70:0/195839170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890095e70 0x7f089012e730 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f088c00c970 tx=0x7f088c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.740+0000 7f08957fa640 1 -- 172.21.15.70:0/195839170 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c007bb0 con 0x7f0890095e70 2024-04-01T01:47:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.740+0000 7f08957fa640 1 -- 172.21.15.70:0/195839170 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f088c007d50 con 0x7f0890095e70 2024-04-01T01:47:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.740+0000 7f08957fa640 1 -- 172.21.15.70:0/195839170 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c00e760 con 0x7f0890095e70 2024-04-01T01:47:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.740+0000 7f089e5b5640 1 -- 172.21.15.70:0/195839170 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0890136100 con 0x7f0890095e70 2024-04-01T01:47:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.740+0000 7f089e5b5640 1 -- 172.21.15.70:0/195839170 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0890136600 con 0x7f0890095e70 2024-04-01T01:47:48.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.743+0000 7f08a1199740 1 Processor -- start 2024-04-01T01:47:48.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.744+0000 7f08957fa640 1 -- 172.21.15.70:0/195839170 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f088c00f450 con 0x7f0890095e70 2024-04-01T01:47:48.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.744+0000 7f08a1199740 1 -- start start 2024-04-01T01:47:48.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.744+0000 7f08a1199740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce97eb0 0x5639cce982d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.744+0000 7f08a1199740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce988a0 0x5639cce9d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.744+0000 7f08a1199740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9da10 0x5639cce9fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.744+0000 7f08a1199740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5639ccdf41b0 con 0x5639cce9da10 2024-04-01T01:47:48.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.744+0000 7f08a1199740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5639ccd9cc00 con 0x5639cce988a0 2024-04-01T01:47:48.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.744+0000 7f08a1199740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5639ccdf0250 con 0x5639cce97eb0 2024-04-01T01:47:48.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.745+0000 7f0897fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce988a0 0x5639cce9d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.745+0000 7f0897fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce988a0 0x5639cce9d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33844/0 (socket says 172.21.15.70:33844) 2024-04-01T01:47:48.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.745+0000 7f0897fff640 1 -- 172.21.15.70:0/760327566 learned_addr learned my addr 172.21.15.70:0/760327566 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:48.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.745+0000 7f08957fa640 1 --2- 172.21.15.70:0/195839170 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08880a2620 0x7f08880a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.745+0000 7f08957fa640 1 -- 172.21.15.70:0/195839170 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f088c0ceb50 con 0x7f0890095e70 2024-04-01T01:47:48.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.746+0000 7f089cb52640 1 --2- 172.21.15.70:0/760327566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce97eb0 0x5639cce982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.746+0000 7f089cb52640 1 --2- 172.21.15.70:0/195839170 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08880a2620 0x7f08880a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.746+0000 7f08977fe640 1 --2- 172.21.15.70:0/760327566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9da10 0x5639cce9fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.746+0000 7f0897fff640 1 -- 172.21.15.70:0/760327566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce97eb0 msgr2=0x5639cce982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:48.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.746+0000 7f0897fff640 1 --2- 172.21.15.70:0/760327566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce97eb0 0x5639cce982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.746+0000 7f0897fff640 1 -- 172.21.15.70:0/760327566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9da10 msgr2=0x5639cce9fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:48.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.746+0000 7f0897fff640 1 --2- 172.21.15.70:0/760327566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9da10 0x5639cce9fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.746+0000 7f0897fff640 1 -- 172.21.15.70:0/760327566 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639ccdbc3d0 con 0x5639cce988a0 2024-04-01T01:47:48.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.746+0000 7f089cb52640 1 --2- 172.21.15.70:0/195839170 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08880a2620 0x7f08880a4ae0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f0898050260 tx=0x7f0898062c10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:48.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.746+0000 7f0897fff640 1 --2- 172.21.15.70:0/760327566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce988a0 0x5639cce9d3a0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f0884005ab0 tx=0x7f088401a570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c372000a54fc79db server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:48.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.748+0000 7f08827fc640 1 -- 172.21.15.70:0/760327566 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884004740 con 0x5639cce988a0 2024-04-01T01:47:48.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.748+0000 7f08827fc640 1 -- 172.21.15.70:0/760327566 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0884055070 con 0x5639cce988a0 2024-04-01T01:47:48.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.748+0000 7f08827fc640 1 -- 172.21.15.70:0/760327566 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884003810 con 0x5639cce988a0 2024-04-01T01:47:48.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.749+0000 7f08a1199740 1 -- 172.21.15.70:0/760327566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce988a0 msgr2=0x5639cce9d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.749+0000 7f08a1199740 1 --2- 172.21.15.70:0/760327566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce988a0 0x5639cce9d3a0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f0884005ab0 tx=0x7f088401a570 comp rx=0 tx=0).stop 2024-04-01T01:47:48.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.750+0000 7f08a1199740 1 -- 172.21.15.70:0/760327566 shutdown_connections 2024-04-01T01:47:48.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.750+0000 7f08a1199740 1 --2- 172.21.15.70:0/760327566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9da10 0x5639cce9fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.750+0000 7f08a1199740 1 --2- 172.21.15.70:0/760327566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce988a0 0x5639cce9d3a0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.750+0000 7f08a1199740 1 --2- 172.21.15.70:0/760327566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce97eb0 0x5639cce982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.750+0000 7f08a1199740 1 -- 172.21.15.70:0/760327566 >> 172.21.15.70:0/760327566 conn(0x5639cce93420 msgr2=0x5639cce93820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:48.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.750+0000 7f08a1199740 1 -- 172.21.15.70:0/760327566 shutdown_connections 2024-04-01T01:47:48.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.750+0000 7f08a1199740 1 -- 172.21.15.70:0/760327566 wait complete. 2024-04-01T01:47:48.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.751+0000 7f08a1199740 1 -- start start 2024-04-01T01:47:48.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.751+0000 7f08a1199740 1 Processor -- start 2024-04-01T01:47:48.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.751+0000 7f08a1199740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce97eb0 0x5639ccea2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f08a1199740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce988a0 0x5639ccea2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f08a1199740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9da10 0x5639ccea9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f08a1199740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5639ccea0960 con 0x5639cce9da10 2024-04-01T01:47:48.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f08a1199740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5639ccd92470 con 0x5639cce988a0 2024-04-01T01:47:48.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f08a1199740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5639ccea0ae0 con 0x5639cce97eb0 2024-04-01T01:47:48.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f08a1199740 10 client.0 _ll_register_callbacks cb 0x5639ccdafcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:48.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f0897fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce97eb0 0x5639ccea2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f0897fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce97eb0 0x5639ccea2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42478/0 (socket says 172.21.15.70:42478) 2024-04-01T01:47:48.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f0897fff640 1 -- 172.21.15.70:0/293707273 learned_addr learned my addr 172.21.15.70:0/293707273 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:48.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f08977fe640 1 --2- 172.21.15.70:0/293707273 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce988a0 0x5639ccea2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f089cb52640 1 --2- 172.21.15.70:0/293707273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9da10 0x5639ccea9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f0897fff640 1 -- 172.21.15.70:0/293707273 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce988a0 msgr2=0x5639ccea2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:48.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f0897fff640 1 --2- 172.21.15.70:0/293707273 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce988a0 0x5639ccea2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f0897fff640 1 -- 172.21.15.70:0/293707273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9da10 msgr2=0x5639ccea9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:48.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f0897fff640 1 --2- 172.21.15.70:0/293707273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9da10 0x5639ccea9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f0897fff640 1 -- 172.21.15.70:0/293707273 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639ccb083f0 con 0x5639cce97eb0 2024-04-01T01:47:48.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.752+0000 7f089cb52640 1 --2- 172.21.15.70:0/293707273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9da10 0x5639ccea9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:47:48.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.753+0000 7f0897fff640 1 --2- 172.21.15.70:0/293707273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce97eb0 0x5639ccea2210 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f088401a860 tx=0x7f088403ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9442391e3e6181f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:48.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.753+0000 7f08777fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:47:48.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.753+0000 7f08777fe640 1 -- 172.21.15.70:0/293707273 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884004740 con 0x5639cce97eb0 2024-04-01T01:47:48.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.753+0000 7f08a1199740 1 -- 172.21.15.70:0/293707273 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0884055070 con 0x5639cce97eb0 2024-04-01T01:47:48.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.754+0000 7f08777fe640 1 -- 172.21.15.70:0/293707273 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f08840476f0 con 0x5639cce97eb0 2024-04-01T01:47:48.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.754+0000 7f08777fe640 1 -- 172.21.15.70:0/293707273 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f08840478f0 con 0x5639cce97eb0 2024-04-01T01:47:48.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.754+0000 7f08a1199740 10 client.29416 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:48.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.754+0000 7f08a1199740 1 -- 172.21.15.70:0/293707273 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5639ccbd27d0 con 0x5639cce97eb0 2024-04-01T01:47:48.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.754+0000 7f08777fe640 1 -- 172.21.15.70:0/293707273 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0884004b20 con 0x5639cce97eb0 2024-04-01T01:47:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.755+0000 7f08777fe640 1 -- 172.21.15.70:0/293707273 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f0884050480 con 0x5639cce97eb0 2024-04-01T01:47:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.755+0000 7f08a1199740 10 client.29416 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.755+0000 7f08a1199740 10 client.29416 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:47:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.755+0000 7f08a1199740 20 client.29416 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.755+0000 7f08a1199740 1 --2- 172.21.15.70:0/293707273 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5639cceadce0 0x5639cceb0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.755+0000 7f08a1199740 4 client.29416 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.755+0000 7f08a1199740 1 -- 172.21.15.70:0/293707273 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0884055070 con 0x5639cceadce0 2024-04-01T01:47:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.755+0000 7f08977fe640 1 --2- 172.21.15.70:0/293707273 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5639cceadce0 0x5639cceb0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.757+0000 7f08977fe640 1 --2- 172.21.15.70:0/293707273 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5639cceadce0 0x5639cceb0120 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4e2a3aef345eacdc server_cookie=6f0ec2dda7fd66b0 in_seq=0 out_seq=0 2024-04-01T01:47:48.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.757+0000 7f08777fe640 10 client.29416 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:47:48.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.758+0000 7f08777fe640 1 -- 172.21.15.70:0/293707273 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f0884055070 con 0x5639cceadce0 2024-04-01T01:47:48.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.758+0000 7f08777fe640 10 client.29416 handle_command_reply: tid=0 2024-04-01T01:47:48.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.758+0000 7f08a1199740 1 client.29416 shutdown 2024-04-01T01:47:48.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.759+0000 7f08a1199740 1 -- 172.21.15.70:0/293707273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce97eb0 msgr2=0x5639ccea2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.759+0000 7f08a1199740 1 --2- 172.21.15.70:0/293707273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce97eb0 0x5639ccea2210 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f088401a860 tx=0x7f088403ee70 comp rx=0 tx=0).stop 2024-04-01T01:47:48.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.759+0000 7f08a1199740 1 -- 172.21.15.70:0/293707273 shutdown_connections 2024-04-01T01:47:48.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.759+0000 7f08a1199740 1 --2- 172.21.15.70:0/293707273 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5639cceadce0 0x5639cceb0120 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.759+0000 7f08a1199740 1 --2- 172.21.15.70:0/293707273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9da10 0x5639ccea9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.759+0000 7f08a1199740 1 --2- 172.21.15.70:0/293707273 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce988a0 0x5639ccea2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.759+0000 7f08a1199740 1 --2- 172.21.15.70:0/293707273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce97eb0 0x5639ccea2210 secure :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f088401a860 tx=0x7f088403ee70 comp rx=0 tx=0).stop 2024-04-01T01:47:48.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.759+0000 7f08a1199740 1 -- 172.21.15.70:0/293707273 >> 172.21.15.70:0/293707273 conn(0x5639cce93420 msgr2=0x5639cce94670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:48.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.759+0000 7f08777fe640 0 client.29416 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:47:48.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.759+0000 7f08a1199740 1 -- 172.21.15.70:0/293707273 shutdown_connections 2024-04-01T01:47:48.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.759+0000 7f08a1199740 1 -- 172.21.15.70:0/293707273 wait complete. 2024-04-01T01:47:48.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.759+0000 7f08a1199740 20 client.29416 trim_cache size 0 max 16384 2024-04-01T01:47:48.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.791+0000 7f08a1199740 1 Processor -- start 2024-04-01T01:47:48.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.791+0000 7f08a1199740 1 -- start start 2024-04-01T01:47:48.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.792+0000 7f08a1199740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce988a0 0x5639cce89700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.792+0000 7f08a1199740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9da10 0x5639ccea96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.792+0000 7f08a1199740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce9a2d0 0x5639cce89c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.792+0000 7f08a1199740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f087c00fcc0 con 0x5639cce9da10 2024-04-01T01:47:48.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.792+0000 7f08a1199740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f087c011800 con 0x5639cce9a2d0 2024-04-01T01:47:48.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.792+0000 7f08a1199740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f087c010390 con 0x5639cce988a0 2024-04-01T01:47:48.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.792+0000 7f089cb52640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce9a2d0 0x5639cce89c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.792+0000 7f089cb52640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce9a2d0 0x5639cce89c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33856/0 (socket says 172.21.15.70:33856) 2024-04-01T01:47:48.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.792+0000 7f089cb52640 1 -- 172.21.15.70:0/4263464732 learned_addr learned my addr 172.21.15.70:0/4263464732 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:48.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.792+0000 7f08977fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9da10 0x5639ccea96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.792+0000 7f0897fff640 1 --2- 172.21.15.70:0/4263464732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce988a0 0x5639cce89700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.793+0000 7f089cb52640 1 -- 172.21.15.70:0/4263464732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce988a0 msgr2=0x5639cce89700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:48.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.793+0000 7f089cb52640 1 --2- 172.21.15.70:0/4263464732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce988a0 0x5639cce89700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.793+0000 7f089cb52640 1 -- 172.21.15.70:0/4263464732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9da10 msgr2=0x5639ccea96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:48.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.793+0000 7f089cb52640 1 --2- 172.21.15.70:0/4263464732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9da10 0x5639ccea96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.793+0000 7f089cb52640 1 -- 172.21.15.70:0/4263464732 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639cce93e70 con 0x5639cce9a2d0 2024-04-01T01:47:48.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.793+0000 7f089cb52640 1 --2- 172.21.15.70:0/4263464732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce9a2d0 0x5639cce89c40 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f0898067560 tx=0x7f08980503a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d5212249903a46e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:48.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.793+0000 7f0881ffb640 1 -- 172.21.15.70:0/4263464732 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0898067e10 con 0x5639cce9a2d0 2024-04-01T01:47:48.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.793+0000 7f0881ffb640 1 -- 172.21.15.70:0/4263464732 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0898088070 con 0x5639cce9a2d0 2024-04-01T01:47:48.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.793+0000 7f0881ffb640 1 -- 172.21.15.70:0/4263464732 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0898083b90 con 0x5639cce9a2d0 2024-04-01T01:47:48.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.794+0000 7f08a1199740 1 -- 172.21.15.70:0/4263464732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce9a2d0 msgr2=0x5639cce89c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.794+0000 7f08a1199740 1 --2- 172.21.15.70:0/4263464732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce9a2d0 0x5639cce89c40 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f0898067560 tx=0x7f08980503a0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.801+0000 7f08a1199740 1 -- 172.21.15.70:0/4263464732 shutdown_connections 2024-04-01T01:47:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.801+0000 7f08a1199740 1 --2- 172.21.15.70:0/4263464732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce9a2d0 0x5639cce89c40 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.801+0000 7f08a1199740 1 --2- 172.21.15.70:0/4263464732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9da10 0x5639ccea96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.801+0000 7f08a1199740 1 --2- 172.21.15.70:0/4263464732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce988a0 0x5639cce89700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.801+0000 7f08a1199740 1 -- 172.21.15.70:0/4263464732 >> 172.21.15.70:0/4263464732 conn(0x5639cce93420 msgr2=0x5639cce99bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:48.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.802+0000 7f08a1199740 1 -- 172.21.15.70:0/4263464732 shutdown_connections 2024-04-01T01:47:48.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.802+0000 7f08a1199740 1 -- 172.21.15.70:0/4263464732 wait complete. 2024-04-01T01:47:48.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.803+0000 7f08a1199740 1 -- start start 2024-04-01T01:47:48.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.803+0000 7f08a1199740 1 Processor -- start 2024-04-01T01:47:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.804+0000 7f08a1199740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce988a0 0x5639ccea6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.804+0000 7f08a1199740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9a2d0 0x5639ccea7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.804+0000 7f08a1199740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce9da10 0x5639ccea8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.804+0000 7f08a1199740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f087c011130 con 0x5639cce9a2d0 2024-04-01T01:47:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.805+0000 7f08a1199740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f087c010a60 con 0x5639cce9da10 2024-04-01T01:47:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.805+0000 7f089cb52640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce9da10 0x5639ccea8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.805+0000 7f089cb52640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce9da10 0x5639ccea8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33862/0 (socket says 172.21.15.70:33862) 2024-04-01T01:47:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.805+0000 7f089cb52640 1 -- 172.21.15.70:0/2626812787 learned_addr learned my addr 172.21.15.70:0/2626812787 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.805+0000 7f08a1199740 1 -- 172.21.15.70:0/2626812787 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f087c00dca0 con 0x5639cce988a0 2024-04-01T01:47:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.805+0000 7f08a1199740 10 client.0 _ll_register_callbacks cb 0x7f088403fc50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:48.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.806+0000 7f08977fe640 1 --2- 172.21.15.70:0/2626812787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9a2d0 0x5639ccea7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.808+0000 7f0897fff640 1 --2- 172.21.15.70:0/2626812787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce988a0 0x5639ccea6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.809+0000 7f089cb52640 1 -- 172.21.15.70:0/2626812787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce988a0 msgr2=0x5639ccea6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.809+0000 7f089cb52640 1 --2- 172.21.15.70:0/2626812787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce988a0 0x5639ccea6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.809+0000 7f089cb52640 1 -- 172.21.15.70:0/2626812787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9a2d0 msgr2=0x5639ccea7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.809+0000 7f089cb52640 1 --2- 172.21.15.70:0/2626812787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9a2d0 0x5639ccea7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.809+0000 7f089cb52640 1 -- 172.21.15.70:0/2626812787 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639ccea8a50 con 0x5639cce9da10 2024-04-01T01:47:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.809+0000 7f089cb52640 1 --2- 172.21.15.70:0/2626812787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce9da10 0x5639ccea8460 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f089804f780 tx=0x7f089804f9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b2b233e2d82f227 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.810+0000 7f08777fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.810+0000 7f08777fe640 1 -- 172.21.15.70:0/2626812787 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0898069820 con 0x5639cce9da10 2024-04-01T01:47:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.810+0000 7f08777fe640 1 -- 172.21.15.70:0/2626812787 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f08980699c0 con 0x5639cce9da10 2024-04-01T01:47:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.810+0000 7f08a1199740 1 -- 172.21.15.70:0/2626812787 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0898088070 con 0x5639cce9da10 2024-04-01T01:47:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.810+0000 7f08777fe640 1 -- 172.21.15.70:0/2626812787 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f089809d050 con 0x5639cce9da10 2024-04-01T01:47:48.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.810+0000 7f08a1199740 10 client.29675 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:48.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.810+0000 7f08a1199740 1 -- 172.21.15.70:0/2626812787 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5639ccf14060 con 0x5639cce9da10 2024-04-01T01:47:48.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.811+0000 7f08777fe640 1 -- 172.21.15.70:0/2626812787 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0898093070 con 0x5639cce9da10 2024-04-01T01:47:48.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.811+0000 7f08777fe640 1 -- 172.21.15.70:0/2626812787 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f0898073040 con 0x5639cce9da10 2024-04-01T01:47:48.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.811+0000 7f08a1199740 10 client.29675 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:48.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.811+0000 7f08a1199740 10 client.29675 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:47:48.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.811+0000 7f08a1199740 20 client.29675 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:48.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.811+0000 7f08a1199740 1 --2- 172.21.15.70:0/2626812787 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5639ccf149b0 0x5639ccf16dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:48.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.811+0000 7f08a1199740 4 client.29675 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:47:48.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.811+0000 7f08a1199740 1 -- 172.21.15.70:0/2626812787 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f0898088070 con 0x5639ccf149b0 2024-04-01T01:47:48.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.813+0000 7f0897fff640 1 --2- 172.21.15.70:0/2626812787 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5639ccf149b0 0x5639ccf16dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:48.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.814+0000 7f0897fff640 1 --2- 172.21.15.70:0/2626812787 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5639ccf149b0 0x5639ccf16dd0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=31d9af8ccdc29c31 server_cookie=b07c1f1607cc0094 in_seq=0 out_seq=0 2024-04-01T01:47:48.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.814+0000 7f08777fe640 10 client.29675 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:47:48.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.815+0000 7f08777fe640 1 -- 172.21.15.70:0/2626812787 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f0898088070 con 0x5639ccf149b0 2024-04-01T01:47:48.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.815+0000 7f08777fe640 10 client.29675 handle_command_reply: tid=0 2024-04-01T01:47:48.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.815+0000 7f08a1199740 1 client.29675 shutdown 2024-04-01T01:47:48.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.817+0000 7f08a1199740 1 -- 172.21.15.70:0/2626812787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce9da10 msgr2=0x5639ccea8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.817+0000 7f08a1199740 1 --2- 172.21.15.70:0/2626812787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce9da10 0x5639ccea8460 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f089804f780 tx=0x7f089804f9d0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.817+0000 7f08a1199740 1 -- 172.21.15.70:0/2626812787 shutdown_connections 2024-04-01T01:47:48.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.817+0000 7f08a1199740 1 --2- 172.21.15.70:0/2626812787 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5639ccf149b0 0x5639ccf16dd0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.817+0000 7f08a1199740 1 --2- 172.21.15.70:0/2626812787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5639cce9da10 0x5639ccea8460 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.817+0000 7f08a1199740 1 --2- 172.21.15.70:0/2626812787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5639cce9a2d0 0x5639ccea7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.817+0000 7f08a1199740 1 --2- 172.21.15.70:0/2626812787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5639cce988a0 0x5639ccea6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.817+0000 7f08a1199740 1 -- 172.21.15.70:0/2626812787 >> 172.21.15.70:0/2626812787 conn(0x5639cce93420 msgr2=0x5639cce89700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:48.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.817+0000 7f08777fe640 0 client.29675 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:47:48.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.818+0000 7f08a1199740 1 -- 172.21.15.70:0/2626812787 shutdown_connections 2024-04-01T01:47:48.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.818+0000 7f08a1199740 1 -- 172.21.15.70:0/2626812787 wait complete. 2024-04-01T01:47:48.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.818+0000 7f08a1199740 20 client.29675 trim_cache size 0 max 16384 2024-04-01T01:47:48.819 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:47:48.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.820+0000 7f089e5b5640 1 -- 172.21.15.70:0/195839170 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08880a2620 msgr2=0x7f08880a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.820+0000 7f089e5b5640 1 --2- 172.21.15.70:0/195839170 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08880a2620 0x7f08880a4ae0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f0898050260 tx=0x7f0898062c10 comp rx=0 tx=0).stop 2024-04-01T01:47:48.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.820+0000 7f089e5b5640 1 -- 172.21.15.70:0/195839170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890095e70 msgr2=0x7f089012e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:48.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.820+0000 7f089e5b5640 1 --2- 172.21.15.70:0/195839170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890095e70 0x7f089012e730 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f088c00c970 tx=0x7f088c00ce30 comp rx=0 tx=0).stop 2024-04-01T01:47:48.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.821+0000 7f089e5b5640 1 -- 172.21.15.70:0/195839170 shutdown_connections 2024-04-01T01:47:48.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.821+0000 7f089e5b5640 1 --2- 172.21.15.70:0/195839170 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08880a2620 0x7f08880a4ae0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.821+0000 7f089e5b5640 1 --2- 172.21.15.70:0/195839170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0890096830 0x7f08901357a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.821+0000 7f089e5b5640 1 --2- 172.21.15.70:0/195839170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0890095e70 0x7f089012e730 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.821+0000 7f089e5b5640 1 --2- 172.21.15.70:0/195839170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0890094c80 0x7f089012e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:48.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.821+0000 7f089e5b5640 1 -- 172.21.15.70:0/195839170 >> 172.21.15.70:0/195839170 conn(0x7f08900901d0 msgr2=0x7f0890124e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:48.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.822+0000 7f089e5b5640 1 -- 172.21.15.70:0/195839170 shutdown_connections 2024-04-01T01:47:48.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:48.822+0000 7f089e5b5640 1 -- 172.21.15.70:0/195839170 wait complete. 2024-04-01T01:47:48.838 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:47:49.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.035+0000 7f8f49f48640 1 -- 172.21.15.70:0/3589999803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f44100650 msgr2=0x7f8f44100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.035+0000 7f8f49f48640 1 --2- 172.21.15.70:0/3589999803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f44100650 0x7f8f44100a70 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c009ca0 tx=0x7f8f3c02c120 comp rx=0 tx=0).stop 2024-04-01T01:47:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.036+0000 7f8f49f48640 1 -- 172.21.15.70:0/3589999803 shutdown_connections 2024-04-01T01:47:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.036+0000 7f8f49f48640 1 --2- 172.21.15.70:0/3589999803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f44102200 0x7f8f44108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.036+0000 7f8f49f48640 1 --2- 172.21.15.70:0/3589999803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f44101840 0x7f8f44101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.036+0000 7f8f49f48640 1 --2- 172.21.15.70:0/3589999803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f44100650 0x7f8f44100a70 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.036+0000 7f8f49f48640 1 -- 172.21.15.70:0/3589999803 >> 172.21.15.70:0/3589999803 conn(0x7f8f440fbbe0 msgr2=0x7f8f440fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.037+0000 7f8f49f48640 1 -- 172.21.15.70:0/3589999803 shutdown_connections 2024-04-01T01:47:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.037+0000 7f8f49f48640 1 -- 172.21.15.70:0/3589999803 wait complete. 2024-04-01T01:47:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.037+0000 7f8f49f48640 1 Processor -- start 2024-04-01T01:47:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.037+0000 7f8f49f48640 1 -- start start 2024-04-01T01:47:49.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.038+0000 7f8f49f48640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f44100650 0x7f8f4410fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.038+0000 7f8f49f48640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f44101840 0x7f8f44110000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.038+0000 7f8f49f48640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f44102200 0x7f8f44110eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.038+0000 7f8f49f48640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8f44109fa0 con 0x7f8f44102200 2024-04-01T01:47:49.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.038+0000 7f8f49f48640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8f44109e20 con 0x7f8f44101840 2024-04-01T01:47:49.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.038+0000 7f8f49f48640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4410a120 con 0x7f8f44100650 2024-04-01T01:47:49.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.038+0000 7f8f437fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f44100650 0x7f8f4410fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.038+0000 7f8f43fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f44102200 0x7f8f44110eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.038+0000 7f8f43fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f44102200 0x7f8f44110eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41388/0 (socket says 172.21.15.70:41388) 2024-04-01T01:47:49.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.038+0000 7f8f437fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f44100650 0x7f8f4410fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42516/0 (socket says 172.21.15.70:42516) 2024-04-01T01:47:49.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.038+0000 7f8f43fff640 1 -- 172.21.15.70:0/3300002815 learned_addr learned my addr 172.21.15.70:0/3300002815 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.038+0000 7f8f42ffd640 1 --2- 172.21.15.70:0/3300002815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f44101840 0x7f8f44110000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.038+0000 7f8f43fff640 1 -- 172.21.15.70:0/3300002815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f44100650 msgr2=0x7f8f4410fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.038+0000 7f8f43fff640 1 --2- 172.21.15.70:0/3300002815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f44100650 0x7f8f4410fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.038+0000 7f8f43fff640 1 -- 172.21.15.70:0/3300002815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f44101840 msgr2=0x7f8f44110000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.038+0000 7f8f43fff640 1 --2- 172.21.15.70:0/3300002815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f44101840 0x7f8f44110000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.039+0000 7f8f43fff640 1 -- 172.21.15.70:0/3300002815 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f44111630 con 0x7f8f44102200 2024-04-01T01:47:49.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.039+0000 7f8f43fff640 1 --2- 172.21.15.70:0/3300002815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f44102200 0x7f8f44110eb0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f8f3400c970 tx=0x7f8f3400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.040+0000 7f8f40ff9640 1 -- 172.21.15.70:0/3300002815 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f34007bb0 con 0x7f8f44102200 2024-04-01T01:47:49.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.040+0000 7f8f40ff9640 1 -- 172.21.15.70:0/3300002815 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f34007d50 con 0x7f8f44102200 2024-04-01T01:47:49.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.041+0000 7f8f40ff9640 1 -- 172.21.15.70:0/3300002815 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f34004e10 con 0x7f8f44102200 2024-04-01T01:47:49.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.041+0000 7f8f49f48640 1 -- 172.21.15.70:0/3300002815 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f44111920 con 0x7f8f44102200 2024-04-01T01:47:49.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.041+0000 7f8f49f48640 1 -- 172.21.15.70:0/3300002815 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f441aed20 con 0x7f8f44102200 2024-04-01T01:47:49.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.044+0000 7f8f4cc09740 1 Processor -- start 2024-04-01T01:47:49.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.045+0000 7f8f40ff9640 1 -- 172.21.15.70:0/3300002815 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f8f34004fb0 con 0x7f8f44102200 2024-04-01T01:47:49.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.045+0000 7f8f4cc09740 1 -- start start 2024-04-01T01:47:49.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.045+0000 7f8f4cc09740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd238b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.045+0000 7f8f4cc09740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd23e80 0x5620fbd28980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.046+0000 7f8f4cc09740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd28ff0 0x5620fbd2b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.046+0000 7f8f4cc09740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5620fbc881b0 con 0x5620fbd23490 2024-04-01T01:47:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.046+0000 7f8f4cc09740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5620fbc30c00 con 0x5620fbd28ff0 2024-04-01T01:47:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.046+0000 7f8f4cc09740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5620fbc84250 con 0x5620fbd23e80 2024-04-01T01:47:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.046+0000 7f8f437fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd23e80 0x5620fbd28980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.046+0000 7f8f437fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd23e80 0x5620fbd28980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42532/0 (socket says 172.21.15.70:42532) 2024-04-01T01:47:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.046+0000 7f8f437fe640 1 -- 172.21.15.70:0/1612123476 learned_addr learned my addr 172.21.15.70:0/1612123476 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.046+0000 7f8f40ff9640 1 --2- 172.21.15.70:0/3300002815 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f2c0a2550 0x7f8f2c0a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.046+0000 7f8f437fe640 1 -- 172.21.15.70:0/1612123476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd28ff0 msgr2=0x5620fbd2b480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:47:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.046+0000 7f8f437fe640 1 --2- 172.21.15.70:0/1612123476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd28ff0 0x5620fbd2b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.046+0000 7f8f437fe640 1 -- 172.21.15.70:0/1612123476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 msgr2=0x5620fbd238b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.046+0000 7f8f437fe640 1 --2- 172.21.15.70:0/1612123476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd238b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.046+0000 7f8f437fe640 1 -- 172.21.15.70:0/1612123476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620fbc503d0 con 0x5620fbd23e80 2024-04-01T01:47:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.046+0000 7f8f40ff9640 1 -- 172.21.15.70:0/3300002815 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8f340ce860 con 0x7f8f44102200 2024-04-01T01:47:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.047+0000 7f8f43fff640 1 --2- 172.21.15.70:0/3300002815 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f2c0a2550 0x7f8f2c0a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.047+0000 7f8f437fe640 1 --2- 172.21.15.70:0/1612123476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd23e80 0x5620fbd28980 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c009ca0 tx=0x7f8f3c009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9a13085a4ac2c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.047+0000 7f8f29ffb640 1 -- 172.21.15.70:0/1612123476 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c039040 con 0x5620fbd23e80 2024-04-01T01:47:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.047+0000 7f8f29ffb640 1 -- 172.21.15.70:0/1612123476 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f3c032de0 con 0x5620fbd23e80 2024-04-01T01:47:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.047+0000 7f8f29ffb640 1 -- 172.21.15.70:0/1612123476 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c004280 con 0x5620fbd23e80 2024-04-01T01:47:49.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.048+0000 7f8f4cc09740 1 -- 172.21.15.70:0/1612123476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd23e80 msgr2=0x5620fbd28980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.048+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/1612123476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd23e80 0x5620fbd28980 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c009ca0 tx=0x7f8f3c009800 comp rx=0 tx=0).stop 2024-04-01T01:47:49.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.048+0000 7f8f43fff640 1 --2- 172.21.15.70:0/3300002815 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f2c0a2550 0x7f8f2c0a4a10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f8f340973d0 tx=0x7f8f3400af90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.048+0000 7f8f4cc09740 1 -- 172.21.15.70:0/1612123476 shutdown_connections 2024-04-01T01:47:49.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.048+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/1612123476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd28ff0 0x5620fbd2b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.048+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/1612123476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd23e80 0x5620fbd28980 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.048+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/1612123476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd238b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.048+0000 7f8f4cc09740 1 -- 172.21.15.70:0/1612123476 >> 172.21.15.70:0/1612123476 conn(0x5620fbd1ea00 msgr2=0x5620fbd1ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.049+0000 7f8f4cc09740 1 -- 172.21.15.70:0/1612123476 shutdown_connections 2024-04-01T01:47:49.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.049+0000 7f8f4cc09740 1 -- 172.21.15.70:0/1612123476 wait complete. 2024-04-01T01:47:49.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.049+0000 7f8f4cc09740 1 -- start start 2024-04-01T01:47:49.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.049+0000 7f8f4cc09740 1 Processor -- start 2024-04-01T01:47:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.050+0000 7f8f4cc09740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd2db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.050+0000 7f8f4cc09740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 0x5620fbd2e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.050+0000 7f8f4cc09740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd28ff0 0x5620fbd34fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.050+0000 7f8f4cc09740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5620fbd2bf40 con 0x5620fbd23490 2024-04-01T01:47:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.050+0000 7f8f4cc09740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5620fbc26470 con 0x5620fbd23e80 2024-04-01T01:47:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.050+0000 7f8f4cc09740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5620fbd2c0c0 con 0x5620fbd28ff0 2024-04-01T01:47:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.050+0000 7f8f4cc09740 10 client.0 _ll_register_callbacks cb 0x5620fbc43cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.050+0000 7f8f42ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd28ff0 0x5620fbd34fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.050+0000 7f8f437fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 0x5620fbd2e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.050+0000 7f8f437fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 0x5620fbd2e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33892/0 (socket says 172.21.15.70:33892) 2024-04-01T01:47:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.050+0000 7f8f437fe640 1 -- 172.21.15.70:0/503243368 learned_addr learned my addr 172.21.15.70:0/503243368 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.050+0000 7f8f42ffd640 1 --2- 172.21.15.70:0/503243368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd2db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.051+0000 7f8f437fe640 1 -- 172.21.15.70:0/503243368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd28ff0 msgr2=0x5620fbd34fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.051+0000 7f8f437fe640 1 --2- 172.21.15.70:0/503243368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd28ff0 0x5620fbd34fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.051+0000 7f8f437fe640 1 -- 172.21.15.70:0/503243368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 msgr2=0x5620fbd2db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.051+0000 7f8f437fe640 1 --2- 172.21.15.70:0/503243368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd2db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.051+0000 7f8f437fe640 1 -- 172.21.15.70:0/503243368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620fb99c3f0 con 0x5620fbd23e80 2024-04-01T01:47:49.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.051+0000 7f8f42ffd640 1 --2- 172.21.15.70:0/503243368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd2db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:47:49.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.051+0000 7f8f437fe640 1 --2- 172.21.15.70:0/503243368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 0x5620fbd2e070 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c000f80 tx=0x7f8f3c032f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79a650e5457a97de server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.051+0000 7f8f22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:49.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.051+0000 7f8f22ffd640 1 -- 172.21.15.70:0/503243368 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c04c070 con 0x5620fbd23e80 2024-04-01T01:47:49.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.051+0000 7f8f4cc09740 1 -- 172.21.15.70:0/503243368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f3c032de0 con 0x5620fbd23e80 2024-04-01T01:47:49.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.051+0000 7f8f22ffd640 1 -- 172.21.15.70:0/503243368 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8f3c003e60 con 0x5620fbd23e80 2024-04-01T01:47:49.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.052+0000 7f8f22ffd640 1 -- 172.21.15.70:0/503243368 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8f3c039070 con 0x5620fbd23e80 2024-04-01T01:47:49.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.052+0000 7f8f4cc09740 10 client.29684 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:49.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.052+0000 7f8f4cc09740 1 -- 172.21.15.70:0/503243368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5620fba667d0 con 0x5620fbd23e80 2024-04-01T01:47:49.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.052+0000 7f8f22ffd640 1 -- 172.21.15.70:0/503243368 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f3c032de0 con 0x5620fbd23e80 2024-04-01T01:47:49.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.052+0000 7f8f22ffd640 1 -- 172.21.15.70:0/503243368 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8f3c051840 con 0x5620fbd23e80 2024-04-01T01:47:49.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.053+0000 7f8f4cc09740 10 client.29684 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:49.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.053+0000 7f8f4cc09740 10 client.29684 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:47:49.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.053+0000 7f8f4cc09740 20 client.29684 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:49.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.053+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/503243368 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5620fbd39130 0x5620fbd3b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.053+0000 7f8f4cc09740 4 client.29684 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.053+0000 7f8f4cc09740 1 -- 172.21.15.70:0/503243368 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620fbc956f0 con 0x5620fbd39130 2024-04-01T01:47:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.053+0000 7f8f42ffd640 1 --2- 172.21.15.70:0/503243368 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5620fbd39130 0x5620fbd3b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.054+0000 7f8f42ffd640 1 --2- 172.21.15.70:0/503243368 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5620fbd39130 0x5620fbd3b520 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=18022b928cecea20 server_cookie=67054d88fe373685 in_seq=0 out_seq=0 2024-04-01T01:47:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.054+0000 7f8f22ffd640 10 client.29684 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:47:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.054+0000 7f8f22ffd640 1 -- 172.21.15.70:0/503243368 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5620fbc956f0 con 0x5620fbd39130 2024-04-01T01:47:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.054+0000 7f8f22ffd640 10 client.29684 handle_command_reply: tid=0 2024-04-01T01:47:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.054+0000 7f8f4cc09740 1 client.29684 shutdown 2024-04-01T01:47:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.055+0000 7f8f4cc09740 1 -- 172.21.15.70:0/503243368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 msgr2=0x5620fbd2e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.055+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/503243368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 0x5620fbd2e070 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c000f80 tx=0x7f8f3c032f90 comp rx=0 tx=0).stop 2024-04-01T01:47:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.055+0000 7f8f4cc09740 1 -- 172.21.15.70:0/503243368 shutdown_connections 2024-04-01T01:47:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.055+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/503243368 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5620fbd39130 0x5620fbd3b520 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.055+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/503243368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd28ff0 0x5620fbd34fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.055+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/503243368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 0x5620fbd2e070 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.055+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/503243368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd2db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.055+0000 7f8f4cc09740 1 -- 172.21.15.70:0/503243368 >> 172.21.15.70:0/503243368 conn(0x5620fbd1ea00 msgr2=0x5620fbd1dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.055+0000 7f8f22ffd640 0 client.29684 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:47:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.055+0000 7f8f4cc09740 1 -- 172.21.15.70:0/503243368 shutdown_connections 2024-04-01T01:47:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.055+0000 7f8f4cc09740 1 -- 172.21.15.70:0/503243368 wait complete. 2024-04-01T01:47:49.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.055+0000 7f8f4cc09740 20 client.29684 trim_cache size 0 max 16384 2024-04-01T01:47:49.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.080+0000 7f8f4cc09740 1 Processor -- start 2024-04-01T01:47:49.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.080+0000 7f8f4cc09740 1 -- start start 2024-04-01T01:47:49.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.080+0000 7f8f4cc09740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd22370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.080+0000 7f8f4cc09740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 0x5620fbd34f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.080+0000 7f8f4cc09740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd28ff0 0x5620fbd3fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.081+0000 7f8f4cc09740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8f441b90c0 con 0x5620fbd23490 2024-04-01T01:47:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.081+0000 7f8f4cc09740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8f441bac00 con 0x5620fbd23e80 2024-04-01T01:47:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.081+0000 7f8f4cc09740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8f441b9790 con 0x5620fbd28ff0 2024-04-01T01:47:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.081+0000 7f8f42ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd22370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.081+0000 7f8f42ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd22370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41414/0 (socket says 172.21.15.70:41414) 2024-04-01T01:47:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.081+0000 7f8f42ffd640 1 -- 172.21.15.70:0/1950978449 learned_addr learned my addr 172.21.15.70:0/1950978449 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.081+0000 7f8f42ffd640 1 --2- 172.21.15.70:0/1950978449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd28ff0 0x5620fbd3fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.081+0000 7f8f437fe640 1 --2- 172.21.15.70:0/1950978449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 0x5620fbd34f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.081+0000 7f8f437fe640 1 -- 172.21.15.70:0/1950978449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd28ff0 msgr2=0x5620fbd3fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.081+0000 7f8f437fe640 1 --2- 172.21.15.70:0/1950978449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd28ff0 0x5620fbd3fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.081+0000 7f8f437fe640 1 -- 172.21.15.70:0/1950978449 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 msgr2=0x5620fbd22370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.081+0000 7f8f437fe640 1 --2- 172.21.15.70:0/1950978449 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd22370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.081+0000 7f8f437fe640 1 -- 172.21.15.70:0/1950978449 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620fbd1f970 con 0x5620fbd23e80 2024-04-01T01:47:49.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.082+0000 7f8f437fe640 1 --2- 172.21.15.70:0/1950978449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 0x5620fbd34f90 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c051dd0 tx=0x7f8f3c009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae7aaabb34d64e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.082+0000 7f8f297fa640 1 -- 172.21.15.70:0/1950978449 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c04c070 con 0x5620fbd23e80 2024-04-01T01:47:49.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.082+0000 7f8f297fa640 1 -- 172.21.15.70:0/1950978449 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f3c051e50 con 0x5620fbd23e80 2024-04-01T01:47:49.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.082+0000 7f8f297fa640 1 -- 172.21.15.70:0/1950978449 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c03f6a0 con 0x5620fbd23e80 2024-04-01T01:47:49.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.083+0000 7f8f4cc09740 1 -- 172.21.15.70:0/1950978449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 msgr2=0x5620fbd34f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.083+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/1950978449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 0x5620fbd34f90 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c051dd0 tx=0x7f8f3c009b20 comp rx=0 tx=0).stop 2024-04-01T01:47:49.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.083+0000 7f8f4cc09740 1 -- 172.21.15.70:0/1950978449 shutdown_connections 2024-04-01T01:47:49.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.083+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/1950978449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd28ff0 0x5620fbd3fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.083+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/1950978449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 0x5620fbd34f90 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.083+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/1950978449 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd22370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.083+0000 7f8f4cc09740 1 -- 172.21.15.70:0/1950978449 >> 172.21.15.70:0/1950978449 conn(0x5620fbd1ea00 msgr2=0x5620fbd21c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.084+0000 7f8f4cc09740 1 -- 172.21.15.70:0/1950978449 shutdown_connections 2024-04-01T01:47:49.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.084+0000 7f8f4cc09740 1 -- 172.21.15.70:0/1950978449 wait complete. 2024-04-01T01:47:49.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.084+0000 7f8f4cc09740 1 -- start start 2024-04-01T01:47:49.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.085+0000 7f8f4cc09740 1 Processor -- start 2024-04-01T01:47:49.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.086+0000 7f8f4cc09740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd32c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.086+0000 7f8f4cc09740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 0x5620fbd331b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.086+0000 7f8f4cc09740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd28ff0 0x5620fbd34170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.086+0000 7f8f4cc09740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8f441ba530 con 0x5620fbd23490 2024-04-01T01:47:49.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.086+0000 7f8f4cc09740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8f441b9e60 con 0x5620fbd23e80 2024-04-01T01:47:49.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.086+0000 7f8f4cc09740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8f441b70a0 con 0x5620fbd28ff0 2024-04-01T01:47:49.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.086+0000 7f8f42ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd32c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.086+0000 7f8f42ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd32c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41422/0 (socket says 172.21.15.70:41422) 2024-04-01T01:47:49.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.086+0000 7f8f42ffd640 1 -- 172.21.15.70:0/2288503231 learned_addr learned my addr 172.21.15.70:0/2288503231 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.086+0000 7f8f4cc09740 10 client.0 _ll_register_callbacks cb 0x7f8f3c032cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:49.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.087+0000 7f8f42ffd640 1 --2- 172.21.15.70:0/2288503231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd28ff0 0x5620fbd34170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.087+0000 7f8f437fe640 1 --2- 172.21.15.70:0/2288503231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 0x5620fbd331b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.087+0000 7f8f42ffd640 1 -- 172.21.15.70:0/2288503231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd28ff0 msgr2=0x5620fbd34170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.087+0000 7f8f42ffd640 1 --2- 172.21.15.70:0/2288503231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd28ff0 0x5620fbd34170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.087+0000 7f8f42ffd640 1 -- 172.21.15.70:0/2288503231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 msgr2=0x5620fbd331b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.087+0000 7f8f42ffd640 1 --2- 172.21.15.70:0/2288503231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 0x5620fbd331b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.087+0000 7f8f42ffd640 1 -- 172.21.15.70:0/2288503231 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620fbd34760 con 0x5620fbd23490 2024-04-01T01:47:49.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.088+0000 7f8f42ffd640 1 --2- 172.21.15.70:0/2288503231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd32c70 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f8f38005fd0 tx=0x7f8f3800b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c09b531d288bdee server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.088+0000 7f8f22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:47:49.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.088+0000 7f8f22ffd640 1 -- 172.21.15.70:0/2288503231 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f38004220 con 0x5620fbd23490 2024-04-01T01:47:49.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.088+0000 7f8f22ffd640 1 -- 172.21.15.70:0/2288503231 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8f380043c0 con 0x5620fbd23490 2024-04-01T01:47:49.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.089+0000 7f8f4cc09740 1 -- 172.21.15.70:0/2288503231 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f3c051e50 con 0x5620fbd23490 2024-04-01T01:47:49.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.089+0000 7f8f22ffd640 1 -- 172.21.15.70:0/2288503231 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8f38030050 con 0x5620fbd23490 2024-04-01T01:47:49.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.090+0000 7f8f4cc09740 10 client.19593 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:49.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.090+0000 7f8f4cc09740 1 -- 172.21.15.70:0/2288503231 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5620fbd9f8d0 con 0x5620fbd23490 2024-04-01T01:47:49.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.090+0000 7f8f437fe640 1 --2- 172.21.15.70:0/2288503231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 0x5620fbd331b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:47:49.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.091+0000 7f8f22ffd640 1 -- 172.21.15.70:0/2288503231 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f3c051e50 con 0x5620fbd23490 2024-04-01T01:47:49.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.091+0000 7f8f22ffd640 1 -- 172.21.15.70:0/2288503231 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8f38004cb0 con 0x5620fbd23490 2024-04-01T01:47:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.092+0000 7f8f4cc09740 10 client.19593 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.092+0000 7f8f4cc09740 10 client.19593 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:47:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.092+0000 7f8f4cc09740 20 client.19593 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.092+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/2288503231 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5620fbda0200 0x5620fbda2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.092+0000 7f8f4cc09740 4 client.19593 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:47:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.092+0000 7f8f4cc09740 1 -- 172.21.15.70:0/2288503231 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5620fbc8fa40 con 0x5620fbda0200 2024-04-01T01:47:49.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.093+0000 7f8f437fe640 1 --2- 172.21.15.70:0/2288503231 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5620fbda0200 0x5620fbda2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.094+0000 7f8f437fe640 1 --2- 172.21.15.70:0/2288503231 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5620fbda0200 0x5620fbda2620 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ee3ca1b5644a6579 server_cookie=3028c419c822b237 in_seq=0 out_seq=0 2024-04-01T01:47:49.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.094+0000 7f8f22ffd640 10 client.19593 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:47:49.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.094+0000 7f8f22ffd640 1 -- 172.21.15.70:0/2288503231 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5620fbc8fa40 con 0x5620fbda0200 2024-04-01T01:47:49.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.094+0000 7f8f22ffd640 10 client.19593 handle_command_reply: tid=0 2024-04-01T01:47:49.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.094+0000 7f8f4cc09740 1 client.19593 shutdown 2024-04-01T01:47:49.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.096+0000 7f8f4cc09740 1 -- 172.21.15.70:0/2288503231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 msgr2=0x5620fbd32c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.096+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/2288503231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd32c70 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f8f38005fd0 tx=0x7f8f3800b410 comp rx=0 tx=0).stop 2024-04-01T01:47:49.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.097+0000 7f8f4cc09740 1 -- 172.21.15.70:0/2288503231 shutdown_connections 2024-04-01T01:47:49.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.097+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/2288503231 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x5620fbda0200 0x5620fbda2620 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.097+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/2288503231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5620fbd28ff0 0x5620fbd34170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.097+0000 7f8f22ffd640 0 client.19593 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:47:49.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.097+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/2288503231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5620fbd23e80 0x5620fbd331b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.097+0000 7f8f4cc09740 1 --2- 172.21.15.70:0/2288503231 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620fbd23490 0x5620fbd32c70 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.097+0000 7f8f4cc09740 1 -- 172.21.15.70:0/2288503231 >> 172.21.15.70:0/2288503231 conn(0x5620fbd1ea00 msgr2=0x5620fbd224f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.100+0000 7f8f4cc09740 1 -- 172.21.15.70:0/2288503231 shutdown_connections 2024-04-01T01:47:49.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.100+0000 7f8f4cc09740 1 -- 172.21.15.70:0/2288503231 wait complete. 2024-04-01T01:47:49.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.102+0000 7f8f4cc09740 20 client.19593 trim_cache size 0 max 16384 2024-04-01T01:47:49.102 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:47:49.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.104+0000 7f8f49f48640 1 -- 172.21.15.70:0/3300002815 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f2c0a2550 msgr2=0x7f8f2c0a4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.104+0000 7f8f49f48640 1 --2- 172.21.15.70:0/3300002815 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f2c0a2550 0x7f8f2c0a4a10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f8f340973d0 tx=0x7f8f3400af90 comp rx=0 tx=0).stop 2024-04-01T01:47:49.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.104+0000 7f8f49f48640 1 -- 172.21.15.70:0/3300002815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f44102200 msgr2=0x7f8f44110eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.104+0000 7f8f49f48640 1 --2- 172.21.15.70:0/3300002815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f44102200 0x7f8f44110eb0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f8f3400c970 tx=0x7f8f3400ce30 comp rx=0 tx=0).stop 2024-04-01T01:47:49.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.104+0000 7f8f49f48640 1 -- 172.21.15.70:0/3300002815 shutdown_connections 2024-04-01T01:47:49.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.104+0000 7f8f49f48640 1 --2- 172.21.15.70:0/3300002815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f44102200 0x7f8f44110eb0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.104+0000 7f8f49f48640 1 --2- 172.21.15.70:0/3300002815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f44101840 0x7f8f44110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.104+0000 7f8f49f48640 1 --2- 172.21.15.70:0/3300002815 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8f2c0a2550 0x7f8f2c0a4a10 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.104+0000 7f8f49f48640 1 --2- 172.21.15.70:0/3300002815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f44100650 0x7f8f4410fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.105+0000 7f8f49f48640 1 -- 172.21.15.70:0/3300002815 >> 172.21.15.70:0/3300002815 conn(0x7f8f440fbbe0 msgr2=0x7f8f44104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.105+0000 7f8f49f48640 1 -- 172.21.15.70:0/3300002815 shutdown_connections 2024-04-01T01:47:49.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.106+0000 7f8f49f48640 1 -- 172.21.15.70:0/3300002815 wait complete. 2024-04-01T01:47:49.130 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:47:49.131 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:48 smithi070 ceph-mon[32179]: pgmap v788: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.5 MiB/s wr, 750 op/s 2024-04-01T01:47:49.131 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:48 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2798088582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:47:49.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:48 smithi189 ceph-mon[35753]: pgmap v788: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.5 MiB/s wr, 750 op/s 2024-04-01T01:47:49.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:48 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2798088582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:47:49.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.320+0000 7f465830d640 1 -- 172.21.15.70:0/526088678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f465010b380 msgr2=0x7f465010d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.320+0000 7f465830d640 1 --2- 172.21.15.70:0/526088678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f465010b380 0x7f465010d770 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f4644007380 tx=0x7f464402c840 comp rx=0 tx=0).stop 2024-04-01T01:47:49.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.320+0000 7f465830d640 1 -- 172.21.15.70:0/526088678 shutdown_connections 2024-04-01T01:47:49.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.320+0000 7f465830d640 1 --2- 172.21.15.70:0/526088678 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f465010de00 0x7f46501102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.320+0000 7f465830d640 1 --2- 172.21.15.70:0/526088678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f465010b380 0x7f465010d770 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.320+0000 7f465830d640 1 --2- 172.21.15.70:0/526088678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46501089c0 0x7f465010adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.321+0000 7f465830d640 1 -- 172.21.15.70:0/526088678 >> 172.21.15.70:0/526088678 conn(0x7f465006e950 msgr2=0x7f4650070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.321+0000 7f465830d640 1 -- 172.21.15.70:0/526088678 shutdown_connections 2024-04-01T01:47:49.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.322+0000 7f465830d640 1 -- 172.21.15.70:0/526088678 wait complete. 2024-04-01T01:47:49.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.323+0000 7f465830d640 1 Processor -- start 2024-04-01T01:47:49.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.323+0000 7f465830d640 1 -- start start 2024-04-01T01:47:49.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.323+0000 7f465830d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46501089c0 0x7f465019ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.323+0000 7f465830d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f465010b380 0x7f465019e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.324+0000 7f465830d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f465010de00 0x7f46501a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.324+0000 7f465830d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4650113940 con 0x7f465010b380 2024-04-01T01:47:49.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.324+0000 7f465830d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46501137c0 con 0x7f46501089c0 2024-04-01T01:47:49.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.324+0000 7f46558a8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f465010b380 0x7f465019e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.324+0000 7f465830d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4650113ac0 con 0x7f465010de00 2024-04-01T01:47:49.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.324+0000 7f46558a8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f465010b380 0x7f465019e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41444/0 (socket says 172.21.15.70:41444) 2024-04-01T01:47:49.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.324+0000 7f46558a8640 1 -- 172.21.15.70:0/1472972268 learned_addr learned my addr 172.21.15.70:0/1472972268 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.324+0000 7f46560a9640 1 --2- 172.21.15.70:0/1472972268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46501089c0 0x7f465019ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.324+0000 7f46568aa640 1 --2- 172.21.15.70:0/1472972268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f465010de00 0x7f46501a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.324+0000 7f46558a8640 1 -- 172.21.15.70:0/1472972268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f465010de00 msgr2=0x7f46501a53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.324+0000 7f46558a8640 1 --2- 172.21.15.70:0/1472972268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f465010de00 0x7f46501a53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.324+0000 7f46558a8640 1 -- 172.21.15.70:0/1472972268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46501089c0 msgr2=0x7f465019ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.324+0000 7f46558a8640 1 --2- 172.21.15.70:0/1472972268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46501089c0 0x7f465019ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.324+0000 7f46558a8640 1 -- 172.21.15.70:0/1472972268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46501a5a10 con 0x7f465010b380 2024-04-01T01:47:49.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.325+0000 7f46558a8640 1 --2- 172.21.15.70:0/1472972268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f465010b380 0x7f465019e330 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f464402cdd0 tx=0x7f464400a440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.325+0000 7f464f7fe640 1 -- 172.21.15.70:0/1472972268 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4644044030 con 0x7f465010b380 2024-04-01T01:47:49.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.325+0000 7f464f7fe640 1 -- 172.21.15.70:0/1472972268 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4644005580 con 0x7f465010b380 2024-04-01T01:47:49.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.325+0000 7f464f7fe640 1 -- 172.21.15.70:0/1472972268 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4644033950 con 0x7f465010b380 2024-04-01T01:47:49.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.325+0000 7f465830d640 1 -- 172.21.15.70:0/1472972268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46501a5ca0 con 0x7f465010b380 2024-04-01T01:47:49.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.325+0000 7f465830d640 1 -- 172.21.15.70:0/1472972268 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46501a6080 con 0x7f465010b380 2024-04-01T01:47:49.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:48 smithi081 ceph-mon[35142]: pgmap v788: 129 pgs: 129 active+clean; 1.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.5 MiB/s wr, 750 op/s 2024-04-01T01:47:49.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:48 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2798088582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:47:49.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.329+0000 7f465afca740 1 Processor -- start 2024-04-01T01:47:49.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.329+0000 7f464f7fe640 1 -- 172.21.15.70:0/1472972268 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f4644040940 con 0x7f465010b380 2024-04-01T01:47:49.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.330+0000 7f464f7fe640 1 --2- 172.21.15.70:0/1472972268 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46380a26a0 0x7f46380a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.330+0000 7f464f7fe640 1 -- 172.21.15.70:0/1472972268 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f46440f4970 con 0x7f465010b380 2024-04-01T01:47:49.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.330+0000 7f46568aa640 1 --2- 172.21.15.70:0/1472972268 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46380a26a0 0x7f46380a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.330+0000 7f465afca740 1 -- start start 2024-04-01T01:47:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.330+0000 7f465afca740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823a490 0x55fb1823a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.330+0000 7f465afca740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823ae80 0x55fb1823f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.330+0000 7f465afca740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb1823fff0 0x55fb18242480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.330+0000 7f465afca740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fb1819f1b0 con 0x55fb1823a490 2024-04-01T01:47:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.330+0000 7f465afca740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55fb18147c00 con 0x55fb1823ae80 2024-04-01T01:47:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.330+0000 7f465afca740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fb1819b250 con 0x55fb1823fff0 2024-04-01T01:47:49.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.331+0000 7f46558a8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823ae80 0x55fb1823f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.331+0000 7f46558a8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823ae80 0x55fb1823f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33942/0 (socket says 172.21.15.70:33942) 2024-04-01T01:47:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.331+0000 7f46558a8640 1 -- 172.21.15.70:0/2152637032 learned_addr learned my addr 172.21.15.70:0/2152637032 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.331+0000 7f46560a9640 1 --2- 172.21.15.70:0/2152637032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823a490 0x55fb1823a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.331+0000 7f46568aa640 1 --2- 172.21.15.70:0/2152637032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb1823fff0 0x55fb18242480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.331+0000 7f46568aa640 1 --2- 172.21.15.70:0/1472972268 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46380a26a0 0x7f46380a4b60 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4648009b60 tx=0x7f4648005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.331+0000 7f46558a8640 1 -- 172.21.15.70:0/2152637032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb1823fff0 msgr2=0x55fb18242480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.331+0000 7f46558a8640 1 --2- 172.21.15.70:0/2152637032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb1823fff0 0x55fb18242480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.331+0000 7f46558a8640 1 -- 172.21.15.70:0/2152637032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823a490 msgr2=0x55fb1823a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.331+0000 7f46558a8640 1 --2- 172.21.15.70:0/2152637032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823a490 0x55fb1823a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.331+0000 7f46558a8640 1 -- 172.21.15.70:0/2152637032 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb181673d0 con 0x55fb1823ae80 2024-04-01T01:47:49.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.332+0000 7f46558a8640 1 --2- 172.21.15.70:0/2152637032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823ae80 0x55fb1823f980 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f46440c0f90 tx=0x7f46440c0c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf2cf57ee4a258b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.332+0000 7f464cbfe640 1 -- 172.21.15.70:0/2152637032 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464404e040 con 0x55fb1823ae80 2024-04-01T01:47:49.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.332+0000 7f464cbfe640 1 -- 172.21.15.70:0/2152637032 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46440c5150 con 0x55fb1823ae80 2024-04-01T01:47:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.333+0000 7f465afca740 1 -- 172.21.15.70:0/2152637032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823ae80 msgr2=0x55fb1823f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.333+0000 7f465afca740 1 --2- 172.21.15.70:0/2152637032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823ae80 0x55fb1823f980 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f46440c0f90 tx=0x7f46440c0c00 comp rx=0 tx=0).stop 2024-04-01T01:47:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.333+0000 7f465afca740 1 -- 172.21.15.70:0/2152637032 shutdown_connections 2024-04-01T01:47:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.333+0000 7f465afca740 1 --2- 172.21.15.70:0/2152637032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb1823fff0 0x55fb18242480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.333+0000 7f465afca740 1 --2- 172.21.15.70:0/2152637032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823ae80 0x55fb1823f980 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.333+0000 7f465afca740 1 --2- 172.21.15.70:0/2152637032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823a490 0x55fb1823a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.333+0000 7f465afca740 1 -- 172.21.15.70:0/2152637032 >> 172.21.15.70:0/2152637032 conn(0x55fb18235a00 msgr2=0x55fb18235e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.333+0000 7f465afca740 1 -- 172.21.15.70:0/2152637032 shutdown_connections 2024-04-01T01:47:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.333+0000 7f465afca740 1 -- 172.21.15.70:0/2152637032 wait complete. 2024-04-01T01:47:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.334+0000 7f465afca740 1 -- start start 2024-04-01T01:47:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.334+0000 7f465afca740 1 Processor -- start 2024-04-01T01:47:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.334+0000 7f465afca740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823a490 0x55fb1824cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.334+0000 7f465afca740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823fff0 0x55fb1824d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.334+0000 7f465afca740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb18252360 0x55fb18254750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.334+0000 7f465afca740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fb18242d60 con 0x55fb1823a490 2024-04-01T01:47:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.334+0000 7f465afca740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55fb1813d470 con 0x55fb1823fff0 2024-04-01T01:47:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.334+0000 7f465afca740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fb18242ee0 con 0x55fb18252360 2024-04-01T01:47:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.334+0000 7f465afca740 10 client.0 _ll_register_callbacks cb 0x55fb1815acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.334+0000 7f46558a8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823fff0 0x55fb1824d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.334+0000 7f46558a8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823fff0 0x55fb1824d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33948/0 (socket says 172.21.15.70:33948) 2024-04-01T01:47:49.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.334+0000 7f46558a8640 1 -- 172.21.15.70:0/1287820441 learned_addr learned my addr 172.21.15.70:0/1287820441 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.334+0000 7f46568aa640 1 --2- 172.21.15.70:0/1287820441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb18252360 0x55fb18254750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.335+0000 7f46568aa640 1 -- 172.21.15.70:0/1287820441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823fff0 msgr2=0x55fb1824d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.335+0000 7f46568aa640 1 --2- 172.21.15.70:0/1287820441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823fff0 0x55fb1824d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.335+0000 7f46568aa640 1 -- 172.21.15.70:0/1287820441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823a490 msgr2=0x55fb1824cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.335+0000 7f46568aa640 1 --2- 172.21.15.70:0/1287820441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823a490 0x55fb1824cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.335+0000 7f46568aa640 1 -- 172.21.15.70:0/1287820441 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb17eb33f0 con 0x55fb18252360 2024-04-01T01:47:49.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.336+0000 7f46568aa640 1 --2- 172.21.15.70:0/1287820441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb18252360 0x55fb18254750 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4648012900 tx=0x7f4648004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96fa58753fa418e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.336+0000 7f4635ffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:47:49.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.337+0000 7f4635ffb640 1 -- 172.21.15.70:0/1287820441 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464801edc0 con 0x55fb18252360 2024-04-01T01:47:49.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.337+0000 7f4635ffb640 1 -- 172.21.15.70:0/1287820441 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f46480263e0 con 0x55fb18252360 2024-04-01T01:47:49.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.337+0000 7f465afca740 1 -- 172.21.15.70:0/1287820441 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fb181ac6f0 con 0x55fb18252360 2024-04-01T01:47:49.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.337+0000 7f4635ffb640 1 -- 172.21.15.70:0/1287820441 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f46480265e0 con 0x55fb18252360 2024-04-01T01:47:49.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.337+0000 7f465afca740 10 client.29440 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:49.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.337+0000 7f465afca740 1 -- 172.21.15.70:0/1287820441 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fb17f7d7d0 con 0x55fb18252360 2024-04-01T01:47:49.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.337+0000 7f4635ffb640 1 -- 172.21.15.70:0/1287820441 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55fb181ac6f0 con 0x55fb18252360 2024-04-01T01:47:49.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.338+0000 7f4635ffb640 1 -- 172.21.15.70:0/1287820441 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f4648025070 con 0x55fb18252360 2024-04-01T01:47:49.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.338+0000 7f465afca740 10 client.29440 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:49.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.338+0000 7f465afca740 10 client.29440 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:47:49.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.338+0000 7f465afca740 20 client.29440 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:49.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.338+0000 7f465afca740 1 --2- 172.21.15.70:0/1287820441 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55fb18248560 0x55fb1824aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.338+0000 7f465afca740 4 client.29440 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:49.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.338+0000 7f465afca740 1 -- 172.21.15.70:0/1287820441 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb181a6a40 con 0x55fb18248560 2024-04-01T01:47:49.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.338+0000 7f46560a9640 1 --2- 172.21.15.70:0/1287820441 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55fb18248560 0x55fb1824aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.339+0000 7f46560a9640 1 --2- 172.21.15.70:0/1287820441 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55fb18248560 0x55fb1824aa00 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4d3c92add292c61f server_cookie=158c1b046c53748e in_seq=0 out_seq=0 2024-04-01T01:47:49.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.339+0000 7f4635ffb640 10 client.29440 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:47:49.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.340+0000 7f4635ffb640 1 -- 172.21.15.70:0/1287820441 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fb181a6a40 con 0x55fb18248560 2024-04-01T01:47:49.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.340+0000 7f4635ffb640 10 client.29440 handle_command_reply: tid=0 2024-04-01T01:47:49.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.340+0000 7f465afca740 1 client.29440 shutdown 2024-04-01T01:47:49.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.340+0000 7f465afca740 1 -- 172.21.15.70:0/1287820441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb18252360 msgr2=0x55fb18254750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.340+0000 7f465afca740 1 --2- 172.21.15.70:0/1287820441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb18252360 0x55fb18254750 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4648012900 tx=0x7f4648004730 comp rx=0 tx=0).stop 2024-04-01T01:47:49.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.340+0000 7f465afca740 1 -- 172.21.15.70:0/1287820441 shutdown_connections 2024-04-01T01:47:49.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.340+0000 7f465afca740 1 --2- 172.21.15.70:0/1287820441 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55fb18248560 0x55fb1824aa00 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.340+0000 7f465afca740 1 --2- 172.21.15.70:0/1287820441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb18252360 0x55fb18254750 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.340+0000 7f465afca740 1 --2- 172.21.15.70:0/1287820441 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823fff0 0x55fb1824d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.340+0000 7f4635ffb640 0 client.29440 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:47:49.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.340+0000 7f465afca740 1 --2- 172.21.15.70:0/1287820441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823a490 0x55fb1824cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.340+0000 7f465afca740 1 -- 172.21.15.70:0/1287820441 >> 172.21.15.70:0/1287820441 conn(0x55fb18235a00 msgr2=0x55fb18234d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.340+0000 7f465afca740 1 -- 172.21.15.70:0/1287820441 shutdown_connections 2024-04-01T01:47:49.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.340+0000 7f465afca740 1 -- 172.21.15.70:0/1287820441 wait complete. 2024-04-01T01:47:49.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.341+0000 7f465afca740 20 client.29440 trim_cache size 0 max 16384 2024-04-01T01:47:49.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.387+0000 7f465afca740 1 Processor -- start 2024-04-01T01:47:49.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.389+0000 7f465afca740 1 -- start start 2024-04-01T01:47:49.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.391+0000 7f465afca740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823a490 0x55fb1823c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.391+0000 7f46560a9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823a490 0x55fb1823c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.391+0000 7f465afca740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823fff0 0x55fb1823c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.392+0000 7f465afca740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb18252360 0x55fb182344b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.392+0000 7f46560a9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823a490 0x55fb1823c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41472/0 (socket says 172.21.15.70:41472) 2024-04-01T01:47:49.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.392+0000 7f46558a8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823fff0 0x55fb1823c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.392+0000 7f46558a8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823fff0 0x55fb1823c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33950/0 (socket says 172.21.15.70:33950) 2024-04-01T01:47:49.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.392+0000 7f46560a9640 1 -- 172.21.15.70:0/3592053525 learned_addr learned my addr 172.21.15.70:0/3592053525 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.392+0000 7f465afca740 1 -- 172.21.15.70:0/3592053525 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f46501b08f0 con 0x55fb1823a490 2024-04-01T01:47:49.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.392+0000 7f46568aa640 1 --2- 172.21.15.70:0/3592053525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb18252360 0x55fb182344b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.392+0000 7f465afca740 1 -- 172.21.15.70:0/3592053525 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46501b2430 con 0x55fb1823fff0 2024-04-01T01:47:49.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.392+0000 7f465afca740 1 -- 172.21.15.70:0/3592053525 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f46501b0fc0 con 0x55fb18252360 2024-04-01T01:47:49.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.395+0000 7f46558a8640 1 -- 172.21.15.70:0/3592053525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb18252360 msgr2=0x55fb182344b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.395+0000 7f46558a8640 1 --2- 172.21.15.70:0/3592053525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb18252360 0x55fb182344b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.395+0000 7f46558a8640 1 -- 172.21.15.70:0/3592053525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823a490 msgr2=0x55fb1823c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.395+0000 7f46558a8640 1 --2- 172.21.15.70:0/3592053525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823a490 0x55fb1823c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.395+0000 7f46558a8640 1 -- 172.21.15.70:0/3592053525 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb18231460 con 0x55fb1823fff0 2024-04-01T01:47:49.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.395+0000 7f46558a8640 1 --2- 172.21.15.70:0/3592053525 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823fff0 0x55fb1823c720 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f46440bf2b0 tx=0x7f46440c0a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eadf4e53a9a521b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.397+0000 7f4637fff640 1 -- 172.21.15.70:0/3592053525 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46440dd030 con 0x55fb1823fff0 2024-04-01T01:47:49.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.397+0000 7f4637fff640 1 -- 172.21.15.70:0/3592053525 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46440c1000 con 0x55fb1823fff0 2024-04-01T01:47:49.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.397+0000 7f4637fff640 1 -- 172.21.15.70:0/3592053525 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464404e040 con 0x55fb1823fff0 2024-04-01T01:47:49.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.399+0000 7f465afca740 1 -- 172.21.15.70:0/3592053525 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823fff0 msgr2=0x55fb1823c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.399+0000 7f465afca740 1 --2- 172.21.15.70:0/3592053525 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823fff0 0x55fb1823c720 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f46440bf2b0 tx=0x7f46440c0a20 comp rx=0 tx=0).stop 2024-04-01T01:47:49.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.405+0000 7f465afca740 1 -- 172.21.15.70:0/3592053525 shutdown_connections 2024-04-01T01:47:49.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.405+0000 7f465afca740 1 --2- 172.21.15.70:0/3592053525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb18252360 0x55fb182344b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.405+0000 7f465afca740 1 --2- 172.21.15.70:0/3592053525 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823fff0 0x55fb1823c720 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.405+0000 7f465afca740 1 --2- 172.21.15.70:0/3592053525 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823a490 0x55fb1823c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.405+0000 7f465afca740 1 -- 172.21.15.70:0/3592053525 >> 172.21.15.70:0/3592053525 conn(0x55fb18235a00 msgr2=0x55fb18235de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.406+0000 7f465afca740 1 -- 172.21.15.70:0/3592053525 shutdown_connections 2024-04-01T01:47:49.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.406+0000 7f465afca740 1 -- 172.21.15.70:0/3592053525 wait complete. 2024-04-01T01:47:49.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.407+0000 7f465afca740 1 -- start start 2024-04-01T01:47:49.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.407+0000 7f465afca740 1 Processor -- start 2024-04-01T01:47:49.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.407+0000 7f465afca740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823a490 0x55fb18272620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.407+0000 7f465afca740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823fff0 0x55fb18272b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.407+0000 7f465afca740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb18252360 0x55fb18275b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.407+0000 7f465afca740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f46501b1d60 con 0x55fb1823fff0 2024-04-01T01:47:49.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.407+0000 7f465afca740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f46501b1690 con 0x55fb1823a490 2024-04-01T01:47:49.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.407+0000 7f465afca740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f46501ae8d0 con 0x55fb18252360 2024-04-01T01:47:49.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.408+0000 7f465afca740 10 client.0 _ll_register_callbacks cb 0x7f46480124e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:49.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.408+0000 7f46560a9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823a490 0x55fb18272620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.408+0000 7f46560a9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823a490 0x55fb18272620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33956/0 (socket says 172.21.15.70:33956) 2024-04-01T01:47:49.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.408+0000 7f46560a9640 1 -- 172.21.15.70:0/1788024694 learned_addr learned my addr 172.21.15.70:0/1788024694 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.408+0000 7f46558a8640 1 --2- 172.21.15.70:0/1788024694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823fff0 0x55fb18272b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.408+0000 7f46560a9640 1 -- 172.21.15.70:0/1788024694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb18252360 msgr2=0x55fb18275b10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:47:49.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.408+0000 7f46560a9640 1 --2- 172.21.15.70:0/1788024694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb18252360 0x55fb18275b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.408+0000 7f46560a9640 1 -- 172.21.15.70:0/1788024694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823fff0 msgr2=0x55fb18272b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.408+0000 7f46560a9640 1 --2- 172.21.15.70:0/1788024694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823fff0 0x55fb18272b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.408+0000 7f46560a9640 1 -- 172.21.15.70:0/1788024694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb18276100 con 0x55fb1823a490 2024-04-01T01:47:49.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.408+0000 7f46560a9640 1 --2- 172.21.15.70:0/1788024694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823a490 0x55fb18272620 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x55fb1823b6e0 tx=0x7f4640003d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6750241edfa7be88 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.408+0000 7f4635ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:49.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.409+0000 7f4635ffb640 1 -- 172.21.15.70:0/1788024694 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4640020020 con 0x55fb1823a490 2024-04-01T01:47:49.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.409+0000 7f4635ffb640 1 -- 172.21.15.70:0/1788024694 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f464000fa90 con 0x55fb1823a490 2024-04-01T01:47:49.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.409+0000 7f465afca740 1 -- 172.21.15.70:0/1788024694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46440c1000 con 0x55fb1823a490 2024-04-01T01:47:49.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.409+0000 7f4635ffb640 1 -- 172.21.15.70:0/1788024694 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f464000fc90 con 0x55fb1823a490 2024-04-01T01:47:49.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.410+0000 7f465afca740 10 client.29705 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:49.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.410+0000 7f465afca740 1 -- 172.21.15.70:0/1788024694 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fb182c7730 con 0x55fb1823a490 2024-04-01T01:47:49.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.412+0000 7f4635ffb640 1 -- 172.21.15.70:0/1788024694 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46440c1000 con 0x55fb1823a490 2024-04-01T01:47:49.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.412+0000 7f4635ffb640 1 -- 172.21.15.70:0/1788024694 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f464002d020 con 0x55fb1823a490 2024-04-01T01:47:49.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.412+0000 7f465afca740 10 client.29705 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:49.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.412+0000 7f465afca740 10 client.29705 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:47:49.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.412+0000 7f465afca740 20 client.29705 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:49.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.412+0000 7f465afca740 1 --2- 172.21.15.70:0/1788024694 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55fb182c8140 0x55fb182ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.412+0000 7f465afca740 4 client.29705 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:47:49.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.412+0000 7f465afca740 1 -- 172.21.15.70:0/1788024694 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fb181246a0 con 0x55fb182c8140 2024-04-01T01:47:49.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.413+0000 7f46558a8640 1 --2- 172.21.15.70:0/1788024694 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55fb182c8140 0x55fb182ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.413+0000 7f46558a8640 1 --2- 172.21.15.70:0/1788024694 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55fb182c8140 0x55fb182ca560 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b304b89c29d7f5e9 server_cookie=e6c542e1264236e3 in_seq=0 out_seq=0 2024-04-01T01:47:49.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.413+0000 7f4635ffb640 10 client.29705 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:47:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.416+0000 7f4635ffb640 1 -- 172.21.15.70:0/1788024694 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fb181246a0 con 0x55fb182c8140 2024-04-01T01:47:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.416+0000 7f4635ffb640 10 client.29705 handle_command_reply: tid=0 2024-04-01T01:47:49.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.416+0000 7f465afca740 1 client.29705 shutdown 2024-04-01T01:47:49.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.416+0000 7f465afca740 1 -- 172.21.15.70:0/1788024694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823a490 msgr2=0x55fb18272620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.416+0000 7f465afca740 1 --2- 172.21.15.70:0/1788024694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823a490 0x55fb18272620 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x55fb1823b6e0 tx=0x7f4640003d40 comp rx=0 tx=0).stop 2024-04-01T01:47:49.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.416+0000 7f465afca740 1 -- 172.21.15.70:0/1788024694 shutdown_connections 2024-04-01T01:47:49.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.416+0000 7f465afca740 1 --2- 172.21.15.70:0/1788024694 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55fb182c8140 0x55fb182ca560 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.416+0000 7f4635ffb640 0 client.29705 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:47:49.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.417+0000 7f465afca740 1 --2- 172.21.15.70:0/1788024694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb18252360 0x55fb18275b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.417+0000 7f465afca740 1 --2- 172.21.15.70:0/1788024694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb1823fff0 0x55fb18272b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.417+0000 7f465afca740 1 --2- 172.21.15.70:0/1788024694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb1823a490 0x55fb18272620 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.417+0000 7f465afca740 1 -- 172.21.15.70:0/1788024694 >> 172.21.15.70:0/1788024694 conn(0x55fb18235a00 msgr2=0x55fb18235de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.417+0000 7f465afca740 1 -- 172.21.15.70:0/1788024694 shutdown_connections 2024-04-01T01:47:49.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.417+0000 7f465afca740 1 -- 172.21.15.70:0/1788024694 wait complete. 2024-04-01T01:47:49.419 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:47:49.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.418+0000 7f465afca740 20 client.29705 trim_cache size 0 max 16384 2024-04-01T01:47:49.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.422+0000 7f465830d640 1 -- 172.21.15.70:0/1472972268 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46380a26a0 msgr2=0x7f46380a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.422+0000 7f465830d640 1 --2- 172.21.15.70:0/1472972268 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46380a26a0 0x7f46380a4b60 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4648009b60 tx=0x7f4648005f90 comp rx=0 tx=0).stop 2024-04-01T01:47:49.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.422+0000 7f465830d640 1 -- 172.21.15.70:0/1472972268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f465010b380 msgr2=0x7f465019e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.422+0000 7f465830d640 1 --2- 172.21.15.70:0/1472972268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f465010b380 0x7f465019e330 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f464402cdd0 tx=0x7f464400a440 comp rx=0 tx=0).stop 2024-04-01T01:47:49.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.422+0000 7f465830d640 1 -- 172.21.15.70:0/1472972268 shutdown_connections 2024-04-01T01:47:49.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.422+0000 7f465830d640 1 --2- 172.21.15.70:0/1472972268 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f46380a26a0 0x7f46380a4b60 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.423+0000 7f465830d640 1 --2- 172.21.15.70:0/1472972268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f465010de00 0x7f46501a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.423+0000 7f465830d640 1 --2- 172.21.15.70:0/1472972268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f465010b380 0x7f465019e330 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.423+0000 7f465830d640 1 --2- 172.21.15.70:0/1472972268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46501089c0 0x7f465019ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.423+0000 7f465830d640 1 -- 172.21.15.70:0/1472972268 >> 172.21.15.70:0/1472972268 conn(0x7f465006e950 msgr2=0x7f465010bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.423+0000 7f465830d640 1 -- 172.21.15.70:0/1472972268 shutdown_connections 2024-04-01T01:47:49.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.423+0000 7f465830d640 1 -- 172.21.15.70:0/1472972268 wait complete. 2024-04-01T01:47:49.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:47:49.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.643+0000 7f9028e30640 1 -- 172.21.15.70:0/540928821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9024108fa0 msgr2=0x7f9024109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.644+0000 7f9028e30640 1 --2- 172.21.15.70:0/540928821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9024108fa0 0x7f9024109420 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f9014009900 tx=0x7f901402bc80 comp rx=0 tx=0).stop 2024-04-01T01:47:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.644+0000 7f9028e30640 1 -- 172.21.15.70:0/540928821 shutdown_connections 2024-04-01T01:47:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.644+0000 7f9028e30640 1 --2- 172.21.15.70:0/540928821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9024109960 0x7f90241101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.644+0000 7f9028e30640 1 --2- 172.21.15.70:0/540928821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9024108fa0 0x7f9024109420 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.644+0000 7f9028e30640 1 --2- 172.21.15.70:0/540928821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9024107db0 0x7f90241081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.644+0000 7f9028e30640 1 -- 172.21.15.70:0/540928821 >> 172.21.15.70:0/540928821 conn(0x7f9024076d60 msgr2=0x7f9024079180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.645+0000 7f9028e30640 1 -- 172.21.15.70:0/540928821 shutdown_connections 2024-04-01T01:47:49.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.645+0000 7f9028e30640 1 -- 172.21.15.70:0/540928821 wait complete. 2024-04-01T01:47:49.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.646+0000 7f9028e30640 1 Processor -- start 2024-04-01T01:47:49.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.649+0000 7f9028e30640 1 -- start start 2024-04-01T01:47:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.649+0000 7f9028e30640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9024107db0 0x7f902407ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.649+0000 7f9028e30640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9024108fa0 0x7f9024080500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.649+0000 7f9028e30640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9024109960 0x7f9024081540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.649+0000 7f9028e30640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f902407a8a0 con 0x7f9024108fa0 2024-04-01T01:47:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.649+0000 7f9028e30640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f902407a720 con 0x7f9024109960 2024-04-01T01:47:49.650 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.649+0000 7f9028e30640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f902407aa20 con 0x7f9024107db0 2024-04-01T01:47:49.651 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.650+0000 7f902259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9024107db0 0x7f902407ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.651 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.650+0000 7f902259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9024107db0 0x7f902407ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42624/0 (socket says 172.21.15.70:42624) 2024-04-01T01:47:49.651 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.650+0000 7f902259c640 1 -- 172.21.15.70:0/1449159988 learned_addr learned my addr 172.21.15.70:0/1449159988 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.653+0000 7f902259c640 1 -- 172.21.15.70:0/1449159988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9024109960 msgr2=0x7f9024081540 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:47:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.653+0000 7f902259c640 1 --2- 172.21.15.70:0/1449159988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9024109960 0x7f9024081540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.653+0000 7f902259c640 1 -- 172.21.15.70:0/1449159988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9024108fa0 msgr2=0x7f9024080500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.654+0000 7f902259c640 1 --2- 172.21.15.70:0/1449159988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9024108fa0 0x7f9024080500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.654+0000 7f902259c640 1 -- 172.21.15.70:0/1449159988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9024081bb0 con 0x7f9024107db0 2024-04-01T01:47:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.654+0000 7f902259c640 1 --2- 172.21.15.70:0/1449159988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9024107db0 0x7f902407ffc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f901c009d20 tx=0x7f901c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.655+0000 7f90137fe640 1 -- 172.21.15.70:0/1449159988 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c010630 con 0x7f9024107db0 2024-04-01T01:47:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.655+0000 7f9028e30640 1 -- 172.21.15.70:0/1449159988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9024081ea0 con 0x7f9024107db0 2024-04-01T01:47:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.655+0000 7f9028e30640 1 -- 172.21.15.70:0/1449159988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90240823f0 con 0x7f9024107db0 2024-04-01T01:47:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.655+0000 7f90137fe640 1 -- 172.21.15.70:0/1449159988 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f901c010c80 con 0x7f9024107db0 2024-04-01T01:47:49.656 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.655+0000 7f90137fe640 1 -- 172.21.15.70:0/1449159988 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c018d50 con 0x7f9024107db0 2024-04-01T01:47:49.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.657+0000 7f088f59e640 1 -- 172.21.15.70:0/186926841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08880a4fe0 msgr2=0x7f08880a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.657+0000 7f088f59e640 1 --2- 172.21.15.70:0/186926841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08880a4fe0 0x7f08880a5400 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f0880008570 tx=0x7f088000ac00 comp rx=0 tx=0).stop 2024-04-01T01:47:49.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.660+0000 7f088f59e640 1 -- 172.21.15.70:0/186926841 shutdown_connections 2024-04-01T01:47:49.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.660+0000 7f088f59e640 1 --2- 172.21.15.70:0/186926841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08880a6b90 0x7f08880ad420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.660+0000 7f088f59e640 1 --2- 172.21.15.70:0/186926841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08880a61d0 0x7f08880a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.660+0000 7f088f59e640 1 --2- 172.21.15.70:0/186926841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08880a4fe0 0x7f08880a5400 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.660+0000 7f088f59e640 1 -- 172.21.15.70:0/186926841 >> 172.21.15.70:0/186926841 conn(0x7f08880a0530 msgr2=0x7f08880a2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.661+0000 7f90137fe640 1 -- 172.21.15.70:0/1449159988 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f901c01f020 con 0x7f9024107db0 2024-04-01T01:47:49.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.662+0000 7f90137fe640 1 --2- 172.21.15.70:0/1449159988 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f900c0a2780 0x7f900c0a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.663+0000 7f90137fe640 1 -- 172.21.15.70:0/1449159988 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f901c0cf0b0 con 0x7f9024107db0 2024-04-01T01:47:49.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.663+0000 7f9021d9b640 1 --2- 172.21.15.70:0/1449159988 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f900c0a2780 0x7f900c0a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.663+0000 7f902ba0c740 1 Processor -- start 2024-04-01T01:47:49.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.664+0000 7f088f59e640 1 -- 172.21.15.70:0/186926841 shutdown_connections 2024-04-01T01:47:49.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.664+0000 7f902ba0c740 1 -- start start 2024-04-01T01:47:49.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.664+0000 7f088f59e640 1 -- 172.21.15.70:0/186926841 wait complete. 2024-04-01T01:47:49.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.664+0000 7f902ba0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d918d0 0x55aa47d91cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.664+0000 7f902ba0c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 0x55aa47d96dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.664+0000 7f9021d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 0x55aa47d96dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.664+0000 7f9021d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 0x55aa47d96dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33998/0 (socket says 172.21.15.70:33998) 2024-04-01T01:47:49.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.664+0000 7f9021d9b640 1 -- 172.21.15.70:0/2434543658 learned_addr learned my addr 172.21.15.70:0/2434543658 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.664+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2434543658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d97430 0x55aa47d998c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.665+0000 7f902ba0c740 1 -- 172.21.15.70:0/2434543658 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55aa47cf58f0 con 0x55aa47d918d0 2024-04-01T01:47:49.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.665+0000 7f902ba0c740 1 -- 172.21.15.70:0/2434543658 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55aa47c9e8d0 con 0x55aa47d922c0 2024-04-01T01:47:49.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.665+0000 7f902ba0c740 1 -- 172.21.15.70:0/2434543658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55aa47cf1990 con 0x55aa47d97430 2024-04-01T01:47:49.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.665+0000 7f902259c640 1 --2- 172.21.15.70:0/2434543658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d918d0 0x55aa47d91cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.665+0000 7f088f59e640 1 Processor -- start 2024-04-01T01:47:49.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.665+0000 7f088f59e640 1 -- start start 2024-04-01T01:47:49.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.665+0000 7f9021d9b640 1 -- 172.21.15.70:0/2434543658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d97430 msgr2=0x55aa47d998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.665+0000 7f9021d9b640 1 --2- 172.21.15.70:0/2434543658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d97430 0x55aa47d998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.665+0000 7f9021d9b640 1 -- 172.21.15.70:0/2434543658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d918d0 msgr2=0x55aa47d91cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.665+0000 7f9021d9b640 1 --2- 172.21.15.70:0/2434543658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d918d0 0x55aa47d91cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.667 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.665+0000 7f9021d9b640 1 -- 172.21.15.70:0/2434543658 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa47a0a210 con 0x55aa47d922c0 2024-04-01T01:47:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f088f59e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08880a4fe0 0x7f0888143360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f088f59e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08880a61d0 0x7f08881438a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f088f59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08880a6b90 0x7f088814a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f088f59e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08880afca0 con 0x7f08880a6b90 2024-04-01T01:47:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f088f59e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f08880afb20 con 0x7f08880a61d0 2024-04-01T01:47:49.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f088f59e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f08880afe20 con 0x7f08880a4fe0 2024-04-01T01:47:49.669 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f9021d9b640 1 --2- 172.21.15.70:0/1449159988 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f900c0a2780 0x7f900c0a4c40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9014036040 tx=0x7f9014005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f088dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08880a61d0 0x7f08881438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f088dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08880a61d0 0x7f08881438a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34010/0 (socket says 172.21.15.70:34010) 2024-04-01T01:47:49.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f088dd9b640 1 -- 172.21.15.70:0/2453882625 learned_addr learned my addr 172.21.15.70:0/2453882625 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.670 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f9021d9b640 1 --2- 172.21.15.70:0/2434543658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 0x55aa47d96dc0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f90240814d0 tx=0x7f9014045000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5207ea41432a962d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f088e59c640 1 --2- 172.21.15.70:0/2453882625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08880a4fe0 0x7f0888143360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f088ed9d640 1 --2- 172.21.15.70:0/2453882625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08880a6b90 0x7f088814a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f088dd9b640 1 -- 172.21.15.70:0/2453882625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08880a4fe0 msgr2=0x7f0888143360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f088dd9b640 1 --2- 172.21.15.70:0/2453882625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08880a4fe0 0x7f0888143360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f088dd9b640 1 -- 172.21.15.70:0/2453882625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08880a6b90 msgr2=0x7f088814a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f088dd9b640 1 --2- 172.21.15.70:0/2453882625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08880a6b90 0x7f088814a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.666+0000 7f088dd9b640 1 -- 172.21.15.70:0/2453882625 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f088814af80 con 0x7f08880a61d0 2024-04-01T01:47:49.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.667+0000 7f088dd9b640 1 --2- 172.21.15.70:0/2453882625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08880a61d0 0x7f08881438a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f088400e9f0 tx=0x7f088400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.669+0000 7f9010ff9640 1 -- 172.21.15.70:0/2434543658 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90140043d0 con 0x55aa47d922c0 2024-04-01T01:47:49.672 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.669+0000 7f9010ff9640 1 -- 172.21.15.70:0/2434543658 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f901404e070 con 0x55aa47d922c0 2024-04-01T01:47:49.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.669+0000 7f087f7fe640 1 -- 172.21.15.70:0/2453882625 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088400cd80 con 0x7f08880a61d0 2024-04-01T01:47:49.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.669+0000 7f088f59e640 1 -- 172.21.15.70:0/2453882625 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f088814b270 con 0x7f08880a61d0 2024-04-01T01:47:49.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.669+0000 7f088f59e640 1 -- 172.21.15.70:0/2453882625 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f088814b7c0 con 0x7f08880a61d0 2024-04-01T01:47:49.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.669+0000 7f087f7fe640 1 -- 172.21.15.70:0/2453882625 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08840108e0 con 0x7f08880a61d0 2024-04-01T01:47:49.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.669+0000 7f087f7fe640 1 -- 172.21.15.70:0/2453882625 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884018920 con 0x7f08880a61d0 2024-04-01T01:47:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.671+0000 7f902ba0c740 1 -- 172.21.15.70:0/2434543658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 msgr2=0x55aa47d96dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.671+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2434543658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 0x55aa47d96dc0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f90240814d0 tx=0x7f9014045000 comp rx=0 tx=0).stop 2024-04-01T01:47:49.673 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.671+0000 7f902ba0c740 1 -- 172.21.15.70:0/2434543658 shutdown_connections 2024-04-01T01:47:49.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.674+0000 7f087f7fe640 1 -- 172.21.15.70:0/2453882625 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f0884010430 con 0x7f08880a61d0 2024-04-01T01:47:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.671+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2434543658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d97430 0x55aa47d998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.674+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2434543658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 0x55aa47d96dc0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.674+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2434543658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d918d0 0x55aa47d91cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.675 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.674+0000 7f902ba0c740 1 -- 172.21.15.70:0/2434543658 >> 172.21.15.70:0/2434543658 conn(0x55aa47d8ce40 msgr2=0x55aa47d8d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.676 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.676+0000 7f902ba0c740 1 -- 172.21.15.70:0/2434543658 shutdown_connections 2024-04-01T01:47:49.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.676+0000 7f0897973740 1 Processor -- start 2024-04-01T01:47:49.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.676+0000 7f0897973740 1 -- start start 2024-04-01T01:47:49.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.676+0000 7f0897973740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a3378c6a0 0x558a3378cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.676+0000 7f0897973740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 0x558a33791b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.676+0000 7f0897973740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a33792200 0x558a33794690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.676+0000 7f0897973740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558a336f11b0 con 0x558a3378c6a0 2024-04-01T01:47:49.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.676+0000 7f0897973740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x558a33699c00 con 0x558a3378d090 2024-04-01T01:47:49.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.676+0000 7f0897973740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558a336ed250 con 0x558a33792200 2024-04-01T01:47:49.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.676+0000 7f088e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 0x558a33791b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.676+0000 7f088e59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 0x558a33791b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34022/0 (socket says 172.21.15.70:34022) 2024-04-01T01:47:49.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.676+0000 7f088e59c640 1 -- 172.21.15.70:0/1043160412 learned_addr learned my addr 172.21.15.70:0/1043160412 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.676+0000 7f088dd9b640 1 --2- 172.21.15.70:0/1043160412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a33792200 0x558a33794690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.677+0000 7f088ed9d640 1 --2- 172.21.15.70:0/1043160412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a3378c6a0 0x558a3378cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.677+0000 7f087f7fe640 1 --2- 172.21.15.70:0/2453882625 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08780a26a0 0x7f08780a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.677+0000 7f088e59c640 1 -- 172.21.15.70:0/1043160412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a33792200 msgr2=0x558a33794690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.677+0000 7f088e59c640 1 --2- 172.21.15.70:0/1043160412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a33792200 0x558a33794690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.677+0000 7f088e59c640 1 -- 172.21.15.70:0/1043160412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a3378c6a0 msgr2=0x558a3378cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.677+0000 7f088e59c640 1 --2- 172.21.15.70:0/1043160412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a3378c6a0 0x558a3378cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.677+0000 7f088e59c640 1 -- 172.21.15.70:0/1043160412 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a336b93d0 con 0x558a3378d090 2024-04-01T01:47:49.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.678+0000 7f087f7fe640 1 -- 172.21.15.70:0/2453882625 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f08840ced00 con 0x7f08880a61d0 2024-04-01T01:47:49.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.678+0000 7f902ba0c740 1 -- 172.21.15.70:0/2434543658 wait complete. 2024-04-01T01:47:49.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.678+0000 7f902ba0c740 1 -- start start 2024-04-01T01:47:49.679 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.678+0000 7f902ba0c740 1 Processor -- start 2024-04-01T01:47:49.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.678+0000 7f088e59c640 1 --2- 172.21.15.70:0/1043160412 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 0x558a33791b90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f0880034000 tx=0x7f0880005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12668db54c292b7e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.678+0000 7f088ed9d640 1 --2- 172.21.15.70:0/2453882625 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08780a26a0 0x7f08780a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.678+0000 7f087cff9640 1 -- 172.21.15.70:0/1043160412 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0880042070 con 0x558a3378d090 2024-04-01T01:47:49.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.678+0000 7f902ba0c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d918d0 0x55aa47d9bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.678+0000 7f902ba0c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 0x55aa47d9c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.678+0000 7f902ba0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d97430 0x55aa47da3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.678+0000 7f902ba0c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55aa47d9a3d0 con 0x55aa47d97430 2024-04-01T01:47:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.678+0000 7f902ba0c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55aa47c93f80 con 0x55aa47d922c0 2024-04-01T01:47:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.678+0000 7f902ba0c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55aa47d9a550 con 0x55aa47d918d0 2024-04-01T01:47:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.678+0000 7f902ba0c740 10 client.0 _ll_register_callbacks cb 0x55aa47cec290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:49.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f902259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d918d0 0x55aa47d9bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.678+0000 7f087cff9640 1 -- 172.21.15.70:0/1043160412 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f088000a630 con 0x558a3378d090 2024-04-01T01:47:49.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.678+0000 7f087cff9640 1 -- 172.21.15.70:0/1043160412 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08800035f0 con 0x558a3378d090 2024-04-01T01:47:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f902259c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d918d0 0x55aa47d9bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42670/0 (socket says 172.21.15.70:42670) 2024-04-01T01:47:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f902259c640 1 -- 172.21.15.70:0/2515857193 learned_addr learned my addr 172.21.15.70:0/2515857193 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f9021d9b640 1 --2- 172.21.15.70:0/2515857193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 0x55aa47d9c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f9022d9d640 1 --2- 172.21.15.70:0/2515857193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d97430 0x55aa47da3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f9021d9b640 1 -- 172.21.15.70:0/2515857193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d918d0 msgr2=0x55aa47d9bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f9021d9b640 1 --2- 172.21.15.70:0/2515857193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d918d0 0x55aa47d9bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f9021d9b640 1 -- 172.21.15.70:0/2515857193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d97430 msgr2=0x55aa47da3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f9021d9b640 1 --2- 172.21.15.70:0/2515857193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d97430 0x55aa47da3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.683 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f9021d9b640 1 -- 172.21.15.70:0/2515857193 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa47ad6620 con 0x55aa47d922c0 2024-04-01T01:47:49.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f0897973740 1 -- 172.21.15.70:0/1043160412 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 msgr2=0x558a33791b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f0897973740 1 --2- 172.21.15.70:0/1043160412 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 0x558a33791b90 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f0880034000 tx=0x7f0880005b50 comp rx=0 tx=0).stop 2024-04-01T01:47:49.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f0897973740 1 -- 172.21.15.70:0/1043160412 shutdown_connections 2024-04-01T01:47:49.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f0897973740 1 --2- 172.21.15.70:0/1043160412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a33792200 0x558a33794690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f0897973740 1 --2- 172.21.15.70:0/1043160412 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 0x558a33791b90 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f0897973740 1 --2- 172.21.15.70:0/1043160412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a3378c6a0 0x558a3378cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.679+0000 7f0897973740 1 -- 172.21.15.70:0/1043160412 >> 172.21.15.70:0/1043160412 conn(0x558a33787c10 msgr2=0x558a33788010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.680+0000 7f088ed9d640 1 --2- 172.21.15.70:0/2453882625 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08780a26a0 0x7f08780a4b60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0890066e80 tx=0x7f08900675f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.680+0000 7f9021d9b640 1 --2- 172.21.15.70:0/2515857193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 0x55aa47d9c0d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f901402c290 tx=0x7f901402cdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5379f8ef8ba9357 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.684 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.680+0000 7f9009ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:49.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.680+0000 7f0897973740 1 -- 172.21.15.70:0/1043160412 shutdown_connections 2024-04-01T01:47:49.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.680+0000 7f0897973740 1 -- 172.21.15.70:0/1043160412 wait complete. 2024-04-01T01:47:49.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.680+0000 7f9009ffb640 1 -- 172.21.15.70:0/2515857193 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9014059070 con 0x55aa47d922c0 2024-04-01T01:47:49.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.680+0000 7f902ba0c740 1 -- 172.21.15.70:0/2515857193 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f901404e070 con 0x55aa47d922c0 2024-04-01T01:47:49.685 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.680+0000 7f9009ffb640 1 -- 172.21.15.70:0/2515857193 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f9014049650 con 0x55aa47d922c0 2024-04-01T01:47:49.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.680+0000 7f0897973740 1 -- start start 2024-04-01T01:47:49.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.680+0000 7f0897973740 1 Processor -- start 2024-04-01T01:47:49.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.681+0000 7f0897973740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3378c6a0 0x558a33796bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.681+0000 7f0897973740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 0x558a33797100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.681+0000 7f0897973740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a33792200 0x558a3379e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.681+0000 7f0897973740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558a33795150 con 0x558a33792200 2024-04-01T01:47:49.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.681+0000 7f0897973740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x558a3368f470 con 0x558a3378d090 2024-04-01T01:47:49.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.681+0000 7f0897973740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558a337952d0 con 0x558a3378c6a0 2024-04-01T01:47:49.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.681+0000 7f0897973740 10 client.0 _ll_register_callbacks cb 0x558a336accb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:49.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.682+0000 7f9009ffb640 1 -- 172.21.15.70:0/2515857193 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9014063050 con 0x55aa47d922c0 2024-04-01T01:47:49.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.681+0000 7f088ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a33792200 0x558a3379e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.681+0000 7f088ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a33792200 0x558a3379e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41542/0 (socket says 172.21.15.70:41542) 2024-04-01T01:47:49.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.681+0000 7f088ed9d640 1 -- 172.21.15.70:0/880287282 learned_addr learned my addr 172.21.15.70:0/880287282 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.681+0000 7f088e59c640 1 --2- 172.21.15.70:0/880287282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3378c6a0 0x558a33796bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.682+0000 7f088ed9d640 1 -- 172.21.15.70:0/880287282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3378c6a0 msgr2=0x558a33796bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.682+0000 7f088ed9d640 1 --2- 172.21.15.70:0/880287282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3378c6a0 0x558a33796bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.682+0000 7f088ed9d640 1 -- 172.21.15.70:0/880287282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 msgr2=0x558a33797100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.682+0000 7f088ed9d640 1 --2- 172.21.15.70:0/880287282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 0x558a33797100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.682+0000 7f088ed9d640 1 -- 172.21.15.70:0/880287282 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a334053f0 con 0x558a33792200 2024-04-01T01:47:49.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.682+0000 7f088ed9d640 1 --2- 172.21.15.70:0/880287282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a33792200 0x558a3379e140 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0890067290 tx=0x7f0890050530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a8b14e418131ab server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.682+0000 7f0875ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:47:49.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.683+0000 7f902ba0c740 10 client.29720 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:49.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.683+0000 7f902ba0c740 1 -- 172.21.15.70:0/2515857193 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55aa47da5ee0 con 0x55aa47d922c0 2024-04-01T01:47:49.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.683+0000 7f0875ffb640 1 -- 172.21.15.70:0/880287282 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089007a7d0 con 0x558a33792200 2024-04-01T01:47:49.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.683+0000 7f0875ffb640 1 -- 172.21.15.70:0/880287282 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f089007a970 con 0x558a33792200 2024-04-01T01:47:49.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.683+0000 7f0897973740 1 -- 172.21.15.70:0/880287282 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f088000a630 con 0x558a33792200 2024-04-01T01:47:49.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.684+0000 7f9009ffb640 1 -- 172.21.15.70:0/2515857193 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f901404e070 con 0x55aa47d922c0 2024-04-01T01:47:49.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.684+0000 7f0875ffb640 1 -- 172.21.15.70:0/880287282 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0890094050 con 0x558a33792200 2024-04-01T01:47:49.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.684+0000 7f0897973740 10 client.19629 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:49.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.684+0000 7f0897973740 1 -- 172.21.15.70:0/880287282 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558a334cf7d0 con 0x558a33792200 2024-04-01T01:47:49.689 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.684+0000 7f9009ffb640 1 -- 172.21.15.70:0/2515857193 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f901405e890 con 0x55aa47d922c0 2024-04-01T01:47:49.689 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.684+0000 7f902ba0c740 10 client.29720 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:49.689 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.684+0000 7f902ba0c740 10 client.29720 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.684+0000 7f902ba0c740 20 client.29720 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.684+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2515857193 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55aa47da7810 0x55aa47da9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.684+0000 7f902ba0c740 4 client.29720 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.684+0000 7f902ba0c740 1 -- 172.21.15.70:0/2515857193 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa47d09120 con 0x55aa47da7810 2024-04-01T01:47:49.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.684+0000 7f0875ffb640 1 -- 172.21.15.70:0/880287282 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f088000a630 con 0x558a33792200 2024-04-01T01:47:49.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.684+0000 7f0875ffb640 1 -- 172.21.15.70:0/880287282 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f0890087b80 con 0x558a33792200 2024-04-01T01:47:49.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.685+0000 7f0897973740 10 client.19629 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:49.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.685+0000 7f0897973740 10 client.19629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:49.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.685+0000 7f0897973740 20 client.19629 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:49.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.685+0000 7f0897973740 1 --2- 172.21.15.70:0/880287282 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558a337a2c80 0x558a337a5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.685+0000 7f0897973740 4 client.19629 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:49.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.685+0000 7f0897973740 1 -- 172.21.15.70:0/880287282 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a336fe6f0 con 0x558a337a2c80 2024-04-01T01:47:49.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.685+0000 7f088e59c640 1 --2- 172.21.15.70:0/880287282 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558a337a2c80 0x558a337a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.685+0000 7f902259c640 1 --2- 172.21.15.70:0/2515857193 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55aa47da7810 0x55aa47da9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.687+0000 7f902259c640 1 --2- 172.21.15.70:0/2515857193 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55aa47da7810 0x55aa47da9c30 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99739d1584cfd1a9 server_cookie=11e3eac74d06482c in_seq=0 out_seq=0 2024-04-01T01:47:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.687+0000 7f9009ffb640 10 client.29720 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:49.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.688+0000 7f088e59c640 1 --2- 172.21.15.70:0/880287282 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558a337a2c80 0x558a337a5140 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db25b8ff2695a9a3 server_cookie=3f2f8a23976f1e85 in_seq=0 out_seq=0 2024-04-01T01:47:49.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.688+0000 7f0875ffb640 10 client.19629 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.688+0000 7f9009ffb640 1 -- 172.21.15.70:0/2515857193 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55aa47d09120 con 0x55aa47da7810 2024-04-01T01:47:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.688+0000 7f9009ffb640 10 client.29720 handle_command_reply: tid=0 2024-04-01T01:47:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.688+0000 7f902ba0c740 1 client.29720 shutdown 2024-04-01T01:47:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.689+0000 7f902ba0c740 1 -- 172.21.15.70:0/2515857193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 msgr2=0x55aa47d9c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.689+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2515857193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 0x55aa47d9c0d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f901402c290 tx=0x7f901402cdb0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.691+0000 7f0875ffb640 1 -- 172.21.15.70:0/880287282 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558a336fe6f0 con 0x558a337a2c80 2024-04-01T01:47:49.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.691+0000 7f0875ffb640 10 client.19629 handle_command_reply: tid=0 2024-04-01T01:47:49.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.691+0000 7f0897973740 1 client.19629 shutdown 2024-04-01T01:47:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f902ba0c740 1 -- 172.21.15.70:0/2515857193 shutdown_connections 2024-04-01T01:47:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2515857193 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55aa47da7810 0x55aa47da9c30 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2515857193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d97430 0x55aa47da3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2515857193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 0x55aa47d9c0d0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2515857193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d918d0 0x55aa47d9bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f902ba0c740 1 -- 172.21.15.70:0/2515857193 >> 172.21.15.70:0/2515857193 conn(0x55aa47d8ce40 msgr2=0x55aa47d8bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f0897973740 1 -- 172.21.15.70:0/880287282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a33792200 msgr2=0x558a3379e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f0897973740 1 --2- 172.21.15.70:0/880287282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a33792200 0x558a3379e140 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0890067290 tx=0x7f0890050530 comp rx=0 tx=0).stop 2024-04-01T01:47:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f9009ffb640 0 client.29720 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f902ba0c740 1 -- 172.21.15.70:0/2515857193 shutdown_connections 2024-04-01T01:47:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f902ba0c740 1 -- 172.21.15.70:0/2515857193 wait complete. 2024-04-01T01:47:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f902ba0c740 20 client.29720 trim_cache size 0 max 16384 2024-04-01T01:47:49.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f0897973740 1 -- 172.21.15.70:0/880287282 shutdown_connections 2024-04-01T01:47:49.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f0897973740 1 --2- 172.21.15.70:0/880287282 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558a337a2c80 0x558a337a5140 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f0897973740 1 --2- 172.21.15.70:0/880287282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a33792200 0x558a3379e140 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f0897973740 1 --2- 172.21.15.70:0/880287282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 0x558a33797100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f0897973740 1 --2- 172.21.15.70:0/880287282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a3378c6a0 0x558a33796bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f0897973740 1 -- 172.21.15.70:0/880287282 >> 172.21.15.70:0/880287282 conn(0x558a33787c10 msgr2=0x558a33786f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.692+0000 7f0875ffb640 0 client.19629 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:49.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.693+0000 7f0897973740 1 -- 172.21.15.70:0/880287282 shutdown_connections 2024-04-01T01:47:49.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.693+0000 7f0897973740 1 -- 172.21.15.70:0/880287282 wait complete. 2024-04-01T01:47:49.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.693+0000 7f0897973740 20 client.19629 trim_cache size 0 max 16384 2024-04-01T01:47:49.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.726+0000 7f0897973740 1 Processor -- start 2024-04-01T01:47:49.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.730+0000 7f0897973740 1 -- start start 2024-04-01T01:47:49.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.730+0000 7f0897973740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a3378c6a0 0x558a33796bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.728+0000 7f902ba0c740 1 Processor -- start 2024-04-01T01:47:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.728+0000 7f902ba0c740 1 -- start start 2024-04-01T01:47:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.728+0000 7f902ba0c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d918d0 0x55aa47d8d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.729+0000 7f902ba0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d922c0 0x55aa47da30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.729+0000 7f902259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d918d0 0x55aa47d8d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.729+0000 7f902ba0c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d97430 0x55aa47d9bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.729+0000 7f902259c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d918d0 0x55aa47d8d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34048/0 (socket says 172.21.15.70:34048) 2024-04-01T01:47:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.729+0000 7f902259c640 1 -- 172.21.15.70:0/1340501471 learned_addr learned my addr 172.21.15.70:0/1340501471 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.729+0000 7f902ba0c740 1 -- 172.21.15.70:0/1340501471 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f900400fe30 con 0x55aa47d922c0 2024-04-01T01:47:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.729+0000 7f902ba0c740 1 -- 172.21.15.70:0/1340501471 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9004011970 con 0x55aa47d918d0 2024-04-01T01:47:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.729+0000 7f902ba0c740 1 -- 172.21.15.70:0/1340501471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9004010500 con 0x55aa47d97430 2024-04-01T01:47:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.730+0000 7f9022d9d640 1 --2- 172.21.15.70:0/1340501471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d97430 0x55aa47d9bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.730+0000 7f9021d9b640 1 --2- 172.21.15.70:0/1340501471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d922c0 0x55aa47da30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.730+0000 7f902259c640 1 -- 172.21.15.70:0/1340501471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d97430 msgr2=0x55aa47d9bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.730+0000 7f902259c640 1 --2- 172.21.15.70:0/1340501471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d97430 0x55aa47d9bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.730+0000 7f902259c640 1 -- 172.21.15.70:0/1340501471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d922c0 msgr2=0x55aa47da30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.730+0000 7f902259c640 1 --2- 172.21.15.70:0/1340501471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d922c0 0x55aa47da30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.730+0000 7f902259c640 1 -- 172.21.15.70:0/1340501471 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa47d8dbb0 con 0x55aa47d918d0 2024-04-01T01:47:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.730+0000 7f902259c640 1 --2- 172.21.15.70:0/1340501471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d918d0 0x55aa47d8d220 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f901c097420 tx=0x7f901c012040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6249879689a4e211 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.731+0000 7f0897973740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 0x558a33797100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.731+0000 7f0897973740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a33792200 0x558a3379e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.731+0000 7f0897973740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f087000fda0 con 0x558a3378c6a0 2024-04-01T01:47:49.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.731+0000 7f0897973740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f08700118e0 con 0x558a3378d090 2024-04-01T01:47:49.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.731+0000 7f0897973740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0870010470 con 0x558a33792200 2024-04-01T01:47:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.731+0000 7f900bfff640 1 -- 172.21.15.70:0/1340501471 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c0b9070 con 0x55aa47d918d0 2024-04-01T01:47:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.731+0000 7f900bfff640 1 -- 172.21.15.70:0/1340501471 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f901c0a4150 con 0x55aa47d918d0 2024-04-01T01:47:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.731+0000 7f900bfff640 1 -- 172.21.15.70:0/1340501471 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c0b3030 con 0x55aa47d918d0 2024-04-01T01:47:49.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.732+0000 7f088dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 0x558a33797100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.732+0000 7f088dd9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 0x558a33797100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34064/0 (socket says 172.21.15.70:34064) 2024-04-01T01:47:49.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.732+0000 7f088dd9b640 1 -- 172.21.15.70:0/1354711068 learned_addr learned my addr 172.21.15.70:0/1354711068 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.732+0000 7f088ed9d640 1 --2- 172.21.15.70:0/1354711068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a33792200 0x558a3379e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.733+0000 7f902ba0c740 1 -- 172.21.15.70:0/1340501471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d918d0 msgr2=0x55aa47d8d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.733+0000 7f902ba0c740 1 --2- 172.21.15.70:0/1340501471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d918d0 0x55aa47d8d220 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f901c097420 tx=0x7f901c012040 comp rx=0 tx=0).stop 2024-04-01T01:47:49.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.733+0000 7f088dd9b640 1 -- 172.21.15.70:0/1354711068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a33792200 msgr2=0x558a3379e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.733+0000 7f088dd9b640 1 --2- 172.21.15.70:0/1354711068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a33792200 0x558a3379e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.733+0000 7f088dd9b640 1 -- 172.21.15.70:0/1354711068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a3378c6a0 msgr2=0x558a33796bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.733+0000 7f088dd9b640 1 --2- 172.21.15.70:0/1354711068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a3378c6a0 0x558a33796bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.733+0000 7f088dd9b640 1 -- 172.21.15.70:0/1354711068 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a33788980 con 0x558a3378d090 2024-04-01T01:47:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.734+0000 7f902ba0c740 1 -- 172.21.15.70:0/1340501471 shutdown_connections 2024-04-01T01:47:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.734+0000 7f902ba0c740 1 --2- 172.21.15.70:0/1340501471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d97430 0x55aa47d9bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.734+0000 7f902ba0c740 1 --2- 172.21.15.70:0/1340501471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d922c0 0x55aa47da30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.734+0000 7f902ba0c740 1 --2- 172.21.15.70:0/1340501471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d918d0 0x55aa47d8d220 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.734+0000 7f088dd9b640 1 --2- 172.21.15.70:0/1354711068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 0x558a33797100 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f088409b3d0 tx=0x7f08840d2640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=871c433804936c52 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.734+0000 7f902ba0c740 1 -- 172.21.15.70:0/1340501471 >> 172.21.15.70:0/1340501471 conn(0x55aa47d8ce40 msgr2=0x55aa47d87bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.735+0000 7f0877fff640 1 -- 172.21.15.70:0/1354711068 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088409af80 con 0x558a3378d090 2024-04-01T01:47:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.735+0000 7f902ba0c740 1 -- 172.21.15.70:0/1340501471 shutdown_connections 2024-04-01T01:47:49.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.735+0000 7f0877fff640 1 -- 172.21.15.70:0/1354711068 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0884010720 con 0x558a3378d090 2024-04-01T01:47:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.735+0000 7f902ba0c740 1 -- 172.21.15.70:0/1340501471 wait complete. 2024-04-01T01:47:49.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.735+0000 7f0877fff640 1 -- 172.21.15.70:0/1354711068 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088409ada0 con 0x558a3378d090 2024-04-01T01:47:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.735+0000 7f902ba0c740 1 -- start start 2024-04-01T01:47:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.736+0000 7f902ba0c740 1 Processor -- start 2024-04-01T01:47:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.736+0000 7f902ba0c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d918d0 0x55aa47d9fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.736+0000 7f902ba0c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 0x55aa47da0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.736+0000 7f902ba0c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d97430 0x55aa47e0bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.737+0000 7f902ba0c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f90040112a0 con 0x55aa47d918d0 2024-04-01T01:47:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.737+0000 7f902ba0c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9004010bd0 con 0x55aa47d922c0 2024-04-01T01:47:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.737+0000 7f902ba0c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f900400de10 con 0x55aa47d97430 2024-04-01T01:47:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.737+0000 7f902ba0c740 10 client.0 _ll_register_callbacks cb 0x7f901c010e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:49.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.737+0000 7f0897973740 1 -- 172.21.15.70:0/1354711068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 msgr2=0x558a33797100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.737+0000 7f0897973740 1 --2- 172.21.15.70:0/1354711068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 0x558a33797100 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f088409b3d0 tx=0x7f08840d2640 comp rx=0 tx=0).stop 2024-04-01T01:47:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.737+0000 7f902259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d918d0 0x55aa47d9fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.737+0000 7f902259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d918d0 0x55aa47d9fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41566/0 (socket says 172.21.15.70:41566) 2024-04-01T01:47:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.737+0000 7f902259c640 1 -- 172.21.15.70:0/2929681072 learned_addr learned my addr 172.21.15.70:0/2929681072 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.737+0000 7f0897973740 1 -- 172.21.15.70:0/1354711068 shutdown_connections 2024-04-01T01:47:49.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.737+0000 7f0897973740 1 --2- 172.21.15.70:0/1354711068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a33792200 0x558a3379e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.737+0000 7f0897973740 1 --2- 172.21.15.70:0/1354711068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378d090 0x558a33797100 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.737+0000 7f0897973740 1 --2- 172.21.15.70:0/1354711068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a3378c6a0 0x558a33796bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.737+0000 7f0897973740 1 -- 172.21.15.70:0/1354711068 >> 172.21.15.70:0/1354711068 conn(0x558a33787c10 msgr2=0x558a33782e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.737+0000 7f0897973740 1 -- 172.21.15.70:0/1354711068 shutdown_connections 2024-04-01T01:47:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f902259c640 1 -- 172.21.15.70:0/2929681072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d97430 msgr2=0x55aa47e0bb20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:47:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f902259c640 1 --2- 172.21.15.70:0/2929681072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d97430 0x55aa47e0bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f902259c640 1 -- 172.21.15.70:0/2929681072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 msgr2=0x55aa47da0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f902259c640 1 --2- 172.21.15.70:0/2929681072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 0x55aa47da0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f902259c640 1 -- 172.21.15.70:0/2929681072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa47da31f0 con 0x55aa47d918d0 2024-04-01T01:47:49.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f0897973740 1 -- 172.21.15.70:0/1354711068 wait complete. 2024-04-01T01:47:49.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f0897973740 1 -- start start 2024-04-01T01:47:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f902259c640 1 --2- 172.21.15.70:0/2929681072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d918d0 0x55aa47d9fbe0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f901c09b6d0 tx=0x7f901c01ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=260592a1d7a5b17b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f9009ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:47:49.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f0897973740 1 Processor -- start 2024-04-01T01:47:49.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f0897973740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378c6a0 0x558a3379b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f0897973740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a3378d090 0x558a3379bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f0897973740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a33792200 0x558a3379cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f0897973740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0870011210 con 0x558a3378d090 2024-04-01T01:47:49.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f0897973740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0870010b40 con 0x558a3378c6a0 2024-04-01T01:47:49.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f0897973740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f087000dd80 con 0x558a33792200 2024-04-01T01:47:49.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.738+0000 7f0897973740 10 client.0 _ll_register_callbacks cb 0x7f089006fbc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.739+0000 7f9009ffb640 1 -- 172.21.15.70:0/2929681072 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c0b3070 con 0x55aa47d918d0 2024-04-01T01:47:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.739+0000 7f9009ffb640 1 -- 172.21.15.70:0/2929681072 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f901c0b4260 con 0x55aa47d918d0 2024-04-01T01:47:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.739+0000 7f902ba0c740 1 -- 172.21.15.70:0/2929681072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f901c0a4150 con 0x55aa47d918d0 2024-04-01T01:47:49.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.739+0000 7f088dd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a3378d090 0x558a3379bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.739+0000 7f088dd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a3378d090 0x558a3379bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41574/0 (socket says 172.21.15.70:41574) 2024-04-01T01:47:49.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.739+0000 7f088dd9b640 1 -- 172.21.15.70:0/1515391227 learned_addr learned my addr 172.21.15.70:0/1515391227 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:49.746 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.739+0000 7f9009ffb640 1 -- 172.21.15.70:0/2929681072 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f901c0b9040 con 0x55aa47d918d0 2024-04-01T01:47:49.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.740+0000 7f088e59c640 1 --2- 172.21.15.70:0/1515391227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378c6a0 0x558a3379b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.740+0000 7f088dd9b640 1 -- 172.21.15.70:0/1515391227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a33792200 msgr2=0x558a3379cb70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:47:49.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.740+0000 7f088dd9b640 1 --2- 172.21.15.70:0/1515391227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a33792200 0x558a3379cb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.740+0000 7f088dd9b640 1 -- 172.21.15.70:0/1515391227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378c6a0 msgr2=0x558a3379b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:49.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.740+0000 7f088dd9b640 1 --2- 172.21.15.70:0/1515391227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378c6a0 0x558a3379b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.740+0000 7f088dd9b640 1 -- 172.21.15.70:0/1515391227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a3379d160 con 0x558a3378d090 2024-04-01T01:47:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.740+0000 7f902ba0c740 10 client.19635 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.740+0000 7f902ba0c740 1 -- 172.21.15.70:0/2929681072 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55aa47e0cf00 con 0x55aa47d918d0 2024-04-01T01:47:49.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.740+0000 7f088dd9b640 1 --2- 172.21.15.70:0/1515391227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a3378d090 0x558a3379bb60 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0884099990 tx=0x7f08840bb270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=124f9c73081266ad server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:49.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.740+0000 7f0875ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:47:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.740+0000 7f9009ffb640 1 -- 172.21.15.70:0/2929681072 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f901c0a4150 con 0x55aa47d918d0 2024-04-01T01:47:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.740+0000 7f9009ffb640 1 -- 172.21.15.70:0/2929681072 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f901c0a3cc0 con 0x55aa47d918d0 2024-04-01T01:47:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.741+0000 7f902ba0c740 10 client.19635 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.741+0000 7f902ba0c740 10 client.19635 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.741+0000 7f902ba0c740 20 client.19635 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.741+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2929681072 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55aa47e0d6f0 0x55aa47e0fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.741+0000 7f902ba0c740 4 client.19635 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:47:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.741+0000 7f902ba0c740 1 -- 172.21.15.70:0/2929681072 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aa47d02e30 con 0x55aa47e0d6f0 2024-04-01T01:47:49.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.741+0000 7f0875ffb640 1 -- 172.21.15.70:0/1515391227 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088409af80 con 0x558a3378d090 2024-04-01T01:47:49.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.741+0000 7f0875ffb640 1 -- 172.21.15.70:0/1515391227 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f088409a890 con 0x558a3378d090 2024-04-01T01:47:49.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.741+0000 7f0897973740 1 -- 172.21.15.70:0/1515391227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558a336f8a40 con 0x558a3378d090 2024-04-01T01:47:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.741+0000 7f9021d9b640 1 --2- 172.21.15.70:0/2929681072 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55aa47e0d6f0 0x55aa47e0fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.741+0000 7f0875ffb640 1 -- 172.21.15.70:0/1515391227 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f088409b440 con 0x558a3378d090 2024-04-01T01:47:49.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.741+0000 7f0897973740 10 client.19638 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:49.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.741+0000 7f0897973740 1 -- 172.21.15.70:0/1515391227 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558a3379e4d0 con 0x558a3378d090 2024-04-01T01:47:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.742+0000 7f9021d9b640 1 --2- 172.21.15.70:0/2929681072 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55aa47e0d6f0 0x55aa47e0fb10 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aee05edb5d687e85 server_cookie=2e6cff9f3250f3e1 in_seq=0 out_seq=0 2024-04-01T01:47:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.742+0000 7f9009ffb640 10 client.19635 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:49.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.742+0000 7f0875ffb640 1 -- 172.21.15.70:0/1515391227 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x558a336f8a40 con 0x558a3378d090 2024-04-01T01:47:49.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.742+0000 7f0875ffb640 1 -- 172.21.15.70:0/1515391227 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f08840d7070 con 0x558a3378d090 2024-04-01T01:47:49.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.742+0000 7f0897973740 10 client.19638 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:49.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.742+0000 7f0897973740 10 client.19638 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:49.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.742+0000 7f0897973740 20 client.19638 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:49.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.742+0000 7f0897973740 1 --2- 172.21.15.70:0/1515391227 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558a33809840 0x558a3380bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:49.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.743+0000 7f0897973740 4 client.19638 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:47:49.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.743+0000 7f0897973740 1 -- 172.21.15.70:0/1515391227 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558a336766a0 con 0x558a33809840 2024-04-01T01:47:49.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.743+0000 7f088e59c640 1 --2- 172.21.15.70:0/1515391227 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558a33809840 0x558a3380bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:49.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.745+0000 7f088e59c640 1 --2- 172.21.15.70:0/1515391227 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558a33809840 0x558a3380bc60 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3171d6a98bfcbc5 server_cookie=504e4840c59b7ec3 in_seq=0 out_seq=0 2024-04-01T01:47:49.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.745+0000 7f0875ffb640 10 client.19638 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:49.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.782+0000 7f0875ffb640 1 -- 172.21.15.70:0/1515391227 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558a336766a0 con 0x558a33809840 2024-04-01T01:47:49.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.782+0000 7f0875ffb640 10 client.19638 handle_command_reply: tid=0 2024-04-01T01:47:49.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.782+0000 7f0897973740 1 client.19638 shutdown 2024-04-01T01:47:49.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.783+0000 7f0897973740 1 -- 172.21.15.70:0/1515391227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a3378d090 msgr2=0x558a3379bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.783+0000 7f0897973740 1 --2- 172.21.15.70:0/1515391227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a3378d090 0x558a3379bb60 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0884099990 tx=0x7f08840bb270 comp rx=0 tx=0).stop 2024-04-01T01:47:49.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.783+0000 7f0897973740 1 -- 172.21.15.70:0/1515391227 shutdown_connections 2024-04-01T01:47:49.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.783+0000 7f0897973740 1 --2- 172.21.15.70:0/1515391227 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x558a33809840 0x558a3380bc60 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.783+0000 7f0897973740 1 --2- 172.21.15.70:0/1515391227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a33792200 0x558a3379cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.783+0000 7f0897973740 1 --2- 172.21.15.70:0/1515391227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558a3378d090 0x558a3379bb60 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.783+0000 7f0897973740 1 --2- 172.21.15.70:0/1515391227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558a3378c6a0 0x558a3379b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.783+0000 7f0897973740 1 -- 172.21.15.70:0/1515391227 >> 172.21.15.70:0/1515391227 conn(0x558a33787c10 msgr2=0x558a33796b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.783+0000 7f0875ffb640 0 client.19638 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:49.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.783+0000 7f0897973740 1 -- 172.21.15.70:0/1515391227 shutdown_connections 2024-04-01T01:47:49.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.783+0000 7f0897973740 1 -- 172.21.15.70:0/1515391227 wait complete. 2024-04-01T01:47:49.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.783+0000 7f0897973740 20 client.19638 trim_cache size 0 max 16384 2024-04-01T01:47:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.784+0000 7f9009ffb640 1 -- 172.21.15.70:0/2929681072 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55aa47d02e30 con 0x55aa47e0d6f0 2024-04-01T01:47:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.784+0000 7f9009ffb640 10 client.19635 handle_command_reply: tid=0 2024-04-01T01:47:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.784+0000 7f902ba0c740 1 client.19635 shutdown 2024-04-01T01:47:49.785 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:47:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.785+0000 7f902ba0c740 1 -- 172.21.15.70:0/2929681072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d918d0 msgr2=0x55aa47d9fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.785+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2929681072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d918d0 0x55aa47d9fbe0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f901c09b6d0 tx=0x7f901c01ee50 comp rx=0 tx=0).stop 2024-04-01T01:47:49.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.785+0000 7f088f59e640 1 -- 172.21.15.70:0/2453882625 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08780a26a0 msgr2=0x7f08780a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.785+0000 7f088f59e640 1 --2- 172.21.15.70:0/2453882625 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08780a26a0 0x7f08780a4b60 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0890066e80 tx=0x7f08900675f0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.785+0000 7f088f59e640 1 -- 172.21.15.70:0/2453882625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08880a61d0 msgr2=0x7f08881438a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.785+0000 7f088f59e640 1 --2- 172.21.15.70:0/2453882625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08880a61d0 0x7f08881438a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f088400e9f0 tx=0x7f088400eeb0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.786+0000 7f902ba0c740 1 -- 172.21.15.70:0/2929681072 shutdown_connections 2024-04-01T01:47:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.786+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2929681072 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55aa47e0d6f0 0x55aa47e0fb10 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.786+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2929681072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55aa47d97430 0x55aa47e0bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.786+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2929681072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa47d922c0 0x55aa47da0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.786+0000 7f902ba0c740 1 --2- 172.21.15.70:0/2929681072 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55aa47d918d0 0x55aa47d9fbe0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.786+0000 7f902ba0c740 1 -- 172.21.15.70:0/2929681072 >> 172.21.15.70:0/2929681072 conn(0x55aa47d8ce40 msgr2=0x55aa47d8d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.786+0000 7f088f59e640 1 -- 172.21.15.70:0/2453882625 shutdown_connections 2024-04-01T01:47:49.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.786+0000 7f088f59e640 1 --2- 172.21.15.70:0/2453882625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08880a6b90 0x7f088814a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.786+0000 7f088f59e640 1 --2- 172.21.15.70:0/2453882625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08880a61d0 0x7f08881438a0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.786+0000 7f088f59e640 1 --2- 172.21.15.70:0/2453882625 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f08780a26a0 0x7f08780a4b60 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.786+0000 7f088f59e640 1 --2- 172.21.15.70:0/2453882625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08880a4fe0 0x7f0888143360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.786+0000 7f088f59e640 1 -- 172.21.15.70:0/2453882625 >> 172.21.15.70:0/2453882625 conn(0x7f08880a0530 msgr2=0x7f08880b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.787+0000 7f088f59e640 1 -- 172.21.15.70:0/2453882625 shutdown_connections 2024-04-01T01:47:49.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:49.787+0000 7f088f59e640 1 -- 172.21.15.70:0/2453882625 wait complete. 2024-04-01T01:47:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.788+0000 7f9009ffb640 0 client.19635 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.788+0000 7f902ba0c740 1 -- 172.21.15.70:0/2929681072 shutdown_connections 2024-04-01T01:47:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.788+0000 7f902ba0c740 1 -- 172.21.15.70:0/2929681072 wait complete. 2024-04-01T01:47:49.789 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.789+0000 7f902ba0c740 20 client.19635 trim_cache size 0 max 16384 2024-04-01T01:47:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:47:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:47:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:47:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:47:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.791+0000 7f9028e30640 1 -- 172.21.15.70:0/1449159988 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f900c0a2780 msgr2=0x7f900c0a4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.791+0000 7f9028e30640 1 --2- 172.21.15.70:0/1449159988 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f900c0a2780 0x7f900c0a4c40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9014036040 tx=0x7f9014005850 comp rx=0 tx=0).stop 2024-04-01T01:47:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.791+0000 7f9028e30640 1 -- 172.21.15.70:0/1449159988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9024107db0 msgr2=0x7f902407ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.792+0000 7f9028e30640 1 --2- 172.21.15.70:0/1449159988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9024107db0 0x7f902407ffc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f901c009d20 tx=0x7f901c00c6a0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.793+0000 7f9028e30640 1 -- 172.21.15.70:0/1449159988 shutdown_connections 2024-04-01T01:47:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.793+0000 7f9028e30640 1 --2- 172.21.15.70:0/1449159988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9024109960 0x7f9024081540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.793+0000 7f9028e30640 1 --2- 172.21.15.70:0/1449159988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9024108fa0 0x7f9024080500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.793+0000 7f9028e30640 1 --2- 172.21.15.70:0/1449159988 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f900c0a2780 0x7f900c0a4c40 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.793+0000 7f9028e30640 1 --2- 172.21.15.70:0/1449159988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9024107db0 0x7f902407ffc0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:49.794 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.793+0000 7f9028e30640 1 -- 172.21.15.70:0/1449159988 >> 172.21.15.70:0/1449159988 conn(0x7f9024076d60 msgr2=0x7f902410c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.794+0000 7f9028e30640 1 -- 172.21.15.70:0/1449159988 shutdown_connections 2024-04-01T01:47:49.795 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:47:49.794+0000 7f9028e30640 1 -- 172.21.15.70:0/1449159988 wait complete. 2024-04-01T01:47:50.813 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:47:50.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:47:50.814 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:47:51.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.004+0000 7fe3aab3e640 1 -- 172.21.15.70:0/1048671397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3a4108750 msgr2=0x7fe3a4108b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.004+0000 7fe3aab3e640 1 --2- 172.21.15.70:0/1048671397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3a4108750 0x7fe3a4108b70 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fe398009900 tx=0x7fe39802bc30 comp rx=0 tx=0).stop 2024-04-01T01:47:51.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.007+0000 7fe3aab3e640 1 -- 172.21.15.70:0/1048671397 shutdown_connections 2024-04-01T01:47:51.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.007+0000 7fe3aab3e640 1 --2- 172.21.15.70:0/1048671397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3a410ffd0 0x7fe3a41123c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.007+0000 7fe3aab3e640 1 --2- 172.21.15.70:0/1048671397 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3a41090b0 0x7fe3a410fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.007+0000 7fe3aab3e640 1 --2- 172.21.15.70:0/1048671397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3a4108750 0x7fe3a4108b70 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.007+0000 7fe3aab3e640 1 -- 172.21.15.70:0/1048671397 >> 172.21.15.70:0/1048671397 conn(0x7fe3a406e5b0 msgr2=0x7fe3a40709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:51.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.007+0000 7fe3aab3e640 1 -- 172.21.15.70:0/1048671397 shutdown_connections 2024-04-01T01:47:51.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.008+0000 7fe3aab3e640 1 -- 172.21.15.70:0/1048671397 wait complete. 2024-04-01T01:47:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.008+0000 7fe3aab3e640 1 Processor -- start 2024-04-01T01:47:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.008+0000 7fe3aab3e640 1 -- start start 2024-04-01T01:47:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.009+0000 7fe3aab3e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3a4108750 0x7fe3a41a2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.009+0000 7fe3aab3e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3a41090b0 0x7fe3a41a2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.009+0000 7fe3aab3e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3a410ffd0 0x7fe3a41a99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.009+0000 7fe3aab3e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a4117cf0 con 0x7fe3a4108750 2024-04-01T01:47:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.009+0000 7fe3aab3e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a4117b70 con 0x7fe3a410ffd0 2024-04-01T01:47:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.009+0000 7fe3aab3e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a4117e70 con 0x7fe3a41090b0 2024-04-01T01:47:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.009+0000 7fe3a90db640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3a410ffd0 0x7fe3a41a99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.009+0000 7fe3a90db640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3a410ffd0 0x7fe3a41a99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34106/0 (socket says 172.21.15.70:34106) 2024-04-01T01:47:51.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.009+0000 7fe3a90db640 1 -- 172.21.15.70:0/4260675021 learned_addr learned my addr 172.21.15.70:0/4260675021 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.009+0000 7fe3a3fff640 1 --2- 172.21.15.70:0/4260675021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3a41090b0 0x7fe3a41a2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.009+0000 7fe3a88da640 1 --2- 172.21.15.70:0/4260675021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3a4108750 0x7fe3a41a2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.009+0000 7fe3a90db640 1 -- 172.21.15.70:0/4260675021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3a41090b0 msgr2=0x7fe3a41a2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.009+0000 7fe3a90db640 1 --2- 172.21.15.70:0/4260675021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3a41090b0 0x7fe3a41a2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.009+0000 7fe3a90db640 1 -- 172.21.15.70:0/4260675021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3a4108750 msgr2=0x7fe3a41a2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.009+0000 7fe3a90db640 1 --2- 172.21.15.70:0/4260675021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3a4108750 0x7fe3a41a2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.009+0000 7fe3a90db640 1 -- 172.21.15.70:0/4260675021 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a41aa020 con 0x7fe3a410ffd0 2024-04-01T01:47:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.010+0000 7fe3a90db640 1 --2- 172.21.15.70:0/4260675021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3a410ffd0 0x7fe3a41a99b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fe39400d6a0 tx=0x7fe39400db60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.010+0000 7fe3a1ffb640 1 -- 172.21.15.70:0/4260675021 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394017070 con 0x7fe3a410ffd0 2024-04-01T01:47:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.010+0000 7fe3aab3e640 1 -- 172.21.15.70:0/4260675021 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3a41aa310 con 0x7fe3a410ffd0 2024-04-01T01:47:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.010+0000 7fe3a1ffb640 1 -- 172.21.15.70:0/4260675021 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe394009820 con 0x7fe3a410ffd0 2024-04-01T01:47:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.010+0000 7fe3a1ffb640 1 -- 172.21.15.70:0/4260675021 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394004c40 con 0x7fe3a410ffd0 2024-04-01T01:47:51.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.010+0000 7fe3aab3e640 1 -- 172.21.15.70:0/4260675021 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3a41aa860 con 0x7fe3a410ffd0 2024-04-01T01:47:51.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.014+0000 7fe3ad7fe740 1 Processor -- start 2024-04-01T01:47:51.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.014+0000 7fe3a1ffb640 1 -- 172.21.15.70:0/4260675021 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fe394010460 con 0x7fe3a410ffd0 2024-04-01T01:47:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.014+0000 7fe3ad7fe740 1 -- start start 2024-04-01T01:47:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.014+0000 7fe3ad7fe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90bea10 0x55f1e90bee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3ad7fe740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90bf400 0x55f1e90c3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3ad7fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90c4570 0x55f1e90c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3a1ffb640 1 --2- 172.21.15.70:0/4260675021 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe38c0a2620 0x7fe38c0a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3ad7fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f1e8fc00b0 con 0x55f1e90c4570 2024-04-01T01:47:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3ad7fe740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f1e9022ea0 con 0x55f1e90bf400 2024-04-01T01:47:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3ad7fe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f1e8fcbc00 con 0x55f1e90bea10 2024-04-01T01:47:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3a88da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90bf400 0x55f1e90c3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3a3fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90bea10 0x55f1e90bee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3a88da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90bf400 0x55f1e90c3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34118/0 (socket says 172.21.15.70:34118) 2024-04-01T01:47:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3a88da640 1 -- 172.21.15.70:0/1438306447 learned_addr learned my addr 172.21.15.70:0/1438306447 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3a1ffb640 1 -- 172.21.15.70:0/4260675021 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe3940ce270 con 0x7fe3a410ffd0 2024-04-01T01:47:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3a90db640 1 --2- 172.21.15.70:0/4260675021 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe38c0a2620 0x7fe38c0a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3a3fff640 1 --2- 172.21.15.70:0/1438306447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90c4570 0x55f1e90c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3a88da640 1 -- 172.21.15.70:0/1438306447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90bea10 msgr2=0x55f1e90bee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3a88da640 1 --2- 172.21.15.70:0/1438306447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90bea10 0x55f1e90bee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3a88da640 1 -- 172.21.15.70:0/1438306447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90c4570 msgr2=0x55f1e90c6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3a88da640 1 --2- 172.21.15.70:0/1438306447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90c4570 0x55f1e90c6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3a88da640 1 -- 172.21.15.70:0/1438306447 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1e8dff7e0 con 0x55f1e90bf400 2024-04-01T01:47:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.015+0000 7fe3a3fff640 1 --2- 172.21.15.70:0/1438306447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90c4570 0x55f1e90c6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:47:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.016+0000 7fe3a88da640 1 --2- 172.21.15.70:0/1438306447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90bf400 0x55f1e90c3f00 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fe398009900 tx=0x7fe398005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa14a13458784955 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.016+0000 7fe3a90db640 1 --2- 172.21.15.70:0/4260675021 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe38c0a2620 0x7fe38c0a4ae0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe394000c00 tx=0x7fe3940046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.016+0000 7fe38affd640 1 -- 172.21.15.70:0/1438306447 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe398044020 con 0x55f1e90bf400 2024-04-01T01:47:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.016+0000 7fe38affd640 1 -- 172.21.15.70:0/1438306447 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe39802cb60 con 0x55f1e90bf400 2024-04-01T01:47:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.016+0000 7fe38affd640 1 -- 172.21.15.70:0/1438306447 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe398004920 con 0x55f1e90bf400 2024-04-01T01:47:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.017+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1438306447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90bf400 msgr2=0x55f1e90c3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.017+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/1438306447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90bf400 0x55f1e90c3f00 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fe398009900 tx=0x7fe398005c00 comp rx=0 tx=0).stop 2024-04-01T01:47:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.017+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1438306447 shutdown_connections 2024-04-01T01:47:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.017+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/1438306447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90c4570 0x55f1e90c6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.017+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/1438306447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90bf400 0x55f1e90c3f00 secure :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fe398009900 tx=0x7fe398005c00 comp rx=0 tx=0).stop 2024-04-01T01:47:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.017+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/1438306447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90bea10 0x55f1e90bee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.017+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1438306447 >> 172.21.15.70:0/1438306447 conn(0x55f1e90b9f80 msgr2=0x55f1e90ba380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.017+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1438306447 shutdown_connections 2024-04-01T01:47:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.017+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1438306447 wait complete. 2024-04-01T01:47:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.018+0000 7fe3ad7fe740 1 -- start start 2024-04-01T01:47:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.018+0000 7fe3ad7fe740 1 Processor -- start 2024-04-01T01:47:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.018+0000 7fe3ad7fe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90bea10 0x55f1e90c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.018+0000 7fe3ad7fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90c4570 0x55f1e90c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.019+0000 7fe3ad7fe740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90ce400 0x55f1e90d07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.019+0000 7fe3ad7fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f1e8fc13d0 con 0x55f1e90c4570 2024-04-01T01:47:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.019+0000 7fe3ad7fe740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f1e901ef40 con 0x55f1e90ce400 2024-04-01T01:47:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.019+0000 7fe3ad7fe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f1e90c74c0 con 0x55f1e90bea10 2024-04-01T01:47:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.019+0000 7fe3a3fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90bea10 0x55f1e90c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.019+0000 7fe3ad7fe740 10 client.0 _ll_register_callbacks cb 0x55f1e8fdeb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.019+0000 7fe3a3fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90bea10 0x55f1e90c8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42746/0 (socket says 172.21.15.70:42746) 2024-04-01T01:47:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.019+0000 7fe3a3fff640 1 -- 172.21.15.70:0/1598963210 learned_addr learned my addr 172.21.15.70:0/1598963210 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.019+0000 7fe3a88da640 1 --2- 172.21.15.70:0/1598963210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90c4570 0x55f1e90c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.019+0000 7fe3a88da640 1 -- 172.21.15.70:0/1598963210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90bea10 msgr2=0x55f1e90c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.019+0000 7fe3a88da640 1 --2- 172.21.15.70:0/1598963210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90bea10 0x55f1e90c8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.019+0000 7fe3a88da640 1 -- 172.21.15.70:0/1598963210 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90ce400 msgr2=0x55f1e90d07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.019+0000 7fe3a88da640 1 --2- 172.21.15.70:0/1598963210 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90ce400 0x55f1e90d07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.019+0000 7fe3a88da640 1 -- 172.21.15.70:0/1598963210 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1e8e5ff70 con 0x55f1e90c4570 2024-04-01T01:47:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.020+0000 7fe3a88da640 1 --2- 172.21.15.70:0/1598963210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90c4570 0x55f1e90c93d0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x55f1e90bf8c0 tx=0x7fe3980038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78740fde35b0ccc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.020+0000 7fe383fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:47:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.020+0000 7fe383fff640 1 -- 172.21.15.70:0/1598963210 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39804d030 con 0x55f1e90c4570 2024-04-01T01:47:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.020+0000 7fe383fff640 1 -- 172.21.15.70:0/1598963210 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe39802c960 con 0x55f1e90c4570 2024-04-01T01:47:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.020+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1598963210 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe39802cb60 con 0x55f1e90c4570 2024-04-01T01:47:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.021+0000 7fe383fff640 1 -- 172.21.15.70:0/1598963210 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe398057050 con 0x55f1e90c4570 2024-04-01T01:47:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.021+0000 7fe3ad7fe740 10 client.19647 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.021+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1598963210 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f1e8cf7170 con 0x55f1e90c4570 2024-04-01T01:47:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.021+0000 7fe383fff640 1 -- 172.21.15.70:0/1598963210 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe398003ca0 con 0x55f1e90c4570 2024-04-01T01:47:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.021+0000 7fe383fff640 1 -- 172.21.15.70:0/1598963210 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe398040ce0 con 0x55f1e90c4570 2024-04-01T01:47:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.021+0000 7fe3ad7fe740 10 client.19647 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.021+0000 7fe3ad7fe740 10 client.19647 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.021+0000 7fe3ad7fe740 20 client.19647 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.021+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/1598963210 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f1e90d48c0 0x55f1e90d6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.021+0000 7fe3ad7fe740 4 client.19647 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.021+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1598963210 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe39802cb60 con 0x55f1e90d48c0 2024-04-01T01:47:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.022+0000 7fe3a3fff640 1 --2- 172.21.15.70:0/1598963210 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f1e90d48c0 0x55f1e90d6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.022+0000 7fe383fff640 10 client.19647 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.022+0000 7fe3a3fff640 1 --2- 172.21.15.70:0/1598963210 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f1e90d48c0 0x55f1e90d6cb0 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b6ed63405a7db01 server_cookie=b4988cd9442ec9cf in_seq=0 out_seq=0 2024-04-01T01:47:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.023+0000 7fe383fff640 1 -- 172.21.15.70:0/1598963210 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fe39802cb60 con 0x55f1e90d48c0 2024-04-01T01:47:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.023+0000 7fe383fff640 10 client.19647 handle_command_reply: tid=0 2024-04-01T01:47:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.023+0000 7fe3ad7fe740 1 client.19647 shutdown 2024-04-01T01:47:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.024+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1598963210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90c4570 msgr2=0x55f1e90c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.024+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/1598963210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90c4570 0x55f1e90c93d0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x55f1e90bf8c0 tx=0x7fe3980038a0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.024+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1598963210 shutdown_connections 2024-04-01T01:47:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.024+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/1598963210 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f1e90d48c0 0x55f1e90d6cb0 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.024+0000 7fe383fff640 0 client.19647 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.024+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/1598963210 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90ce400 0x55f1e90d07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.024+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/1598963210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90c4570 0x55f1e90c93d0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.024+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/1598963210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90bea10 0x55f1e90c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.024+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1598963210 >> 172.21.15.70:0/1598963210 conn(0x55f1e90b9f80 msgr2=0x55f1e90b9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.024+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1598963210 shutdown_connections 2024-04-01T01:47:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.024+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1598963210 wait complete. 2024-04-01T01:47:51.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.024+0000 7fe3ad7fe740 20 client.19647 trim_cache size 0 max 16384 2024-04-01T01:47:51.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.047+0000 7fe3ad7fe740 1 Processor -- start 2024-04-01T01:47:51.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.047+0000 7fe3ad7fe740 1 -- start start 2024-04-01T01:47:51.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.048+0000 7fe3ad7fe740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90bea10 0x55f1e90b88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.048+0000 7fe3ad7fe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90c4570 0x55f1e90b8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.048+0000 7fe3ad7fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90ce400 0x55f1e90c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.048+0000 7fe3ad7fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe39000fd90 con 0x55f1e90ce400 2024-04-01T01:47:51.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.048+0000 7fe3ad7fe740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe3900118d0 con 0x55f1e90bea10 2024-04-01T01:47:51.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.048+0000 7fe3ad7fe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe390010460 con 0x55f1e90c4570 2024-04-01T01:47:51.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.049+0000 7fe3a3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90ce400 0x55f1e90c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.049+0000 7fe3a88da640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90c4570 0x55f1e90b8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.049+0000 7fe3a88da640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90c4570 0x55f1e90b8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42760/0 (socket says 172.21.15.70:42760) 2024-04-01T01:47:51.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.049+0000 7fe3a88da640 1 -- 172.21.15.70:0/266042371 learned_addr learned my addr 172.21.15.70:0/266042371 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:51.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.049+0000 7fe3a3fff640 1 --2- 172.21.15.70:0/266042371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90bea10 0x55f1e90b88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.049+0000 7fe3a3fff640 1 -- 172.21.15.70:0/266042371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90c4570 msgr2=0x55f1e90b8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:51.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.049+0000 7fe3a3fff640 1 --2- 172.21.15.70:0/266042371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90c4570 0x55f1e90b8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.049+0000 7fe3a3fff640 1 -- 172.21.15.70:0/266042371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90bea10 msgr2=0x55f1e90b88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:51.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.049+0000 7fe3a3fff640 1 --2- 172.21.15.70:0/266042371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90bea10 0x55f1e90b88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.049+0000 7fe3a3fff640 1 -- 172.21.15.70:0/266042371 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1e90c3c80 con 0x55f1e90ce400 2024-04-01T01:47:51.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.049+0000 7fe3a3fff640 1 --2- 172.21.15.70:0/266042371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90ce400 0x55f1e90c78f0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x55f1e90c7610 tx=0x7fe39c015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d5c24cfe3aeaa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:51.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.049+0000 7fe38a7fc640 1 -- 172.21.15.70:0/266042371 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c01c430 con 0x55f1e90ce400 2024-04-01T01:47:51.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.050+0000 7fe38a7fc640 1 -- 172.21.15.70:0/266042371 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe39c010cb0 con 0x55f1e90ce400 2024-04-01T01:47:51.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.050+0000 7fe38a7fc640 1 -- 172.21.15.70:0/266042371 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c011640 con 0x55f1e90ce400 2024-04-01T01:47:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.051+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/266042371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90ce400 msgr2=0x55f1e90c78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.051+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/266042371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90ce400 0x55f1e90c78f0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x55f1e90c7610 tx=0x7fe39c015040 comp rx=0 tx=0).stop 2024-04-01T01:47:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.051+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/266042371 shutdown_connections 2024-04-01T01:47:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.051+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/266042371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90ce400 0x55f1e90c78f0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.051+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/266042371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90c4570 0x55f1e90b8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.051+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/266042371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90bea10 0x55f1e90b88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.051+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/266042371 >> 172.21.15.70:0/266042371 conn(0x55f1e90b9f80 msgr2=0x55f1e90bfaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.051+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/266042371 shutdown_connections 2024-04-01T01:47:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.051+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/266042371 wait complete. 2024-04-01T01:47:51.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.051+0000 7fe3ad7fe740 1 -- start start 2024-04-01T01:47:51.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.051+0000 7fe3ad7fe740 1 Processor -- start 2024-04-01T01:47:51.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3ad7fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90bea10 0x55f1e90e7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3ad7fe740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90c4570 0x55f1e90e7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3ad7fe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90ce400 0x55f1e90e5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3ad7fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe390011200 con 0x55f1e90bea10 2024-04-01T01:47:51.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3ad7fe740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe390010b30 con 0x55f1e90c4570 2024-04-01T01:47:51.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3ad7fe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe39000dd70 con 0x55f1e90ce400 2024-04-01T01:47:51.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3ad7fe740 10 client.0 _ll_register_callbacks cb 0x7fe398004460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:51.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3a3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90bea10 0x55f1e90e7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3a88da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90c4570 0x55f1e90e7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3a88da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90c4570 0x55f1e90e7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34154/0 (socket says 172.21.15.70:34154) 2024-04-01T01:47:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3a88da640 1 -- 172.21.15.70:0/1409903904 learned_addr learned my addr 172.21.15.70:0/1409903904 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3a3fff640 1 --2- 172.21.15.70:0/1409903904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90ce400 0x55f1e90e5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3a88da640 1 -- 172.21.15.70:0/1409903904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90ce400 msgr2=0x55f1e90e5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3a88da640 1 --2- 172.21.15.70:0/1409903904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90ce400 0x55f1e90e5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3a88da640 1 -- 172.21.15.70:0/1409903904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90bea10 msgr2=0x55f1e90e7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3a88da640 1 --2- 172.21.15.70:0/1409903904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90bea10 0x55f1e90e7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.052+0000 7fe3a88da640 1 -- 172.21.15.70:0/1409903904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1e90e6170 con 0x55f1e90c4570 2024-04-01T01:47:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.053+0000 7fe3a88da640 1 --2- 172.21.15.70:0/1409903904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90c4570 0x55f1e90e7a60 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe398000c00 tx=0x7fe3980498f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=643f6b5a6e45c516 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.053+0000 7fe383fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.053+0000 7fe383fff640 1 -- 172.21.15.70:0/1409903904 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39804d070 con 0x55f1e90c4570 2024-04-01T01:47:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.053+0000 7fe383fff640 1 -- 172.21.15.70:0/1409903904 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe39802c8e0 con 0x55f1e90c4570 2024-04-01T01:47:51.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.053+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1409903904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe39c010cb0 con 0x55f1e90c4570 2024-04-01T01:47:51.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.053+0000 7fe383fff640 1 -- 172.21.15.70:0/1409903904 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe398009780 con 0x55f1e90c4570 2024-04-01T01:47:51.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.053+0000 7fe3ad7fe740 10 client.29744 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:51.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.053+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1409903904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f1e913af40 con 0x55f1e90c4570 2024-04-01T01:47:51.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.054+0000 7fe383fff640 1 -- 172.21.15.70:0/1409903904 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe39c010cb0 con 0x55f1e90c4570 2024-04-01T01:47:51.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.054+0000 7fe383fff640 1 -- 172.21.15.70:0/1409903904 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe3980046c0 con 0x55f1e90c4570 2024-04-01T01:47:51.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.054+0000 7fe3ad7fe740 10 client.29744 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:51.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.054+0000 7fe3ad7fe740 10 client.29744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:51.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.054+0000 7fe3ad7fe740 20 client.29744 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:51.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.054+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/1409903904 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f1e913b920 0x55f1e913dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.054+0000 7fe3ad7fe740 4 client.29744 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:47:51.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.054+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1409903904 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f1e90303e0 con 0x55f1e913b920 2024-04-01T01:47:51.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.054+0000 7fe3a3fff640 1 --2- 172.21.15.70:0/1409903904 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f1e913b920 0x55f1e913dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.055+0000 7fe3a3fff640 1 --2- 172.21.15.70:0/1409903904 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f1e913b920 0x55f1e913dd40 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=494eed39943650ea server_cookie=1ec049604921f897 in_seq=0 out_seq=0 2024-04-01T01:47:51.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.055+0000 7fe383fff640 10 client.29744 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:51.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.056+0000 7fe383fff640 1 -- 172.21.15.70:0/1409903904 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f1e90303e0 con 0x55f1e913b920 2024-04-01T01:47:51.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.056+0000 7fe383fff640 10 client.29744 handle_command_reply: tid=0 2024-04-01T01:47:51.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.056+0000 7fe3ad7fe740 1 client.29744 shutdown 2024-04-01T01:47:51.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.056+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1409903904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90c4570 msgr2=0x55f1e90e7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.056+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/1409903904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90c4570 0x55f1e90e7a60 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe398000c00 tx=0x7fe3980498f0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.056+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1409903904 shutdown_connections 2024-04-01T01:47:51.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.056+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/1409903904 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f1e913b920 0x55f1e913dd40 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.056+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/1409903904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1e90ce400 0x55f1e90e5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.056+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/1409903904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f1e90c4570 0x55f1e90e7a60 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.056+0000 7fe3ad7fe740 1 --2- 172.21.15.70:0/1409903904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f1e90bea10 0x55f1e90e7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.056+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1409903904 >> 172.21.15.70:0/1409903904 conn(0x55f1e90b9f80 msgr2=0x55f1e90b88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:51.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.056+0000 7fe383fff640 0 client.29744 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:51.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.056+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1409903904 shutdown_connections 2024-04-01T01:47:51.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.056+0000 7fe3ad7fe740 1 -- 172.21.15.70:0/1409903904 wait complete. 2024-04-01T01:47:51.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.056+0000 7fe3ad7fe740 20 client.29744 trim_cache size 0 max 16384 2024-04-01T01:47:51.062 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:47:51.062 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:47:51.062 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "1155fbee-5b0c-4291-a6ce-12c5a9e3250d", 2024-04-01T01:47:51.062 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:47:51.062 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:47:51.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.057+0000 7fe3aab3e640 1 -- 172.21.15.70:0/4260675021 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe38c0a2620 msgr2=0x7fe38c0a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.057+0000 7fe3aab3e640 1 --2- 172.21.15.70:0/4260675021 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe38c0a2620 0x7fe38c0a4ae0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe394000c00 tx=0x7fe3940046c0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.057+0000 7fe3aab3e640 1 -- 172.21.15.70:0/4260675021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3a410ffd0 msgr2=0x7fe3a41a99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.057+0000 7fe3aab3e640 1 --2- 172.21.15.70:0/4260675021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3a410ffd0 0x7fe3a41a99b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fe39400d6a0 tx=0x7fe39400db60 comp rx=0 tx=0).stop 2024-04-01T01:47:51.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.057+0000 7fe3aab3e640 1 -- 172.21.15.70:0/4260675021 shutdown_connections 2024-04-01T01:47:51.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.057+0000 7fe3aab3e640 1 --2- 172.21.15.70:0/4260675021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3a410ffd0 0x7fe3a41a99b0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.057+0000 7fe3aab3e640 1 --2- 172.21.15.70:0/4260675021 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe38c0a2620 0x7fe38c0a4ae0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.057+0000 7fe3aab3e640 1 --2- 172.21.15.70:0/4260675021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3a41090b0 0x7fe3a41a2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.057+0000 7fe3aab3e640 1 --2- 172.21.15.70:0/4260675021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3a4108750 0x7fe3a41a2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.057+0000 7fe3aab3e640 1 -- 172.21.15.70:0/4260675021 >> 172.21.15.70:0/4260675021 conn(0x7fe3a406e5b0 msgr2=0x7fe3a4070340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:51.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.057+0000 7fe3aab3e640 1 -- 172.21.15.70:0/4260675021 shutdown_connections 2024-04-01T01:47:51.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.057+0000 7fe3aab3e640 1 -- 172.21.15.70:0/4260675021 wait complete. 2024-04-01T01:47:51.071 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:47:51.071 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:47:51.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.243+0000 7fe333026640 1 -- 172.21.15.70:0/3054931349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe32c072b60 msgr2=0x7fe32c072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.243+0000 7fe333026640 1 --2- 172.21.15.70:0/3054931349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe32c072b60 0x7fe32c072f80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fe324009be0 tx=0x7fe32402bef0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.244+0000 7fe333026640 1 -- 172.21.15.70:0/3054931349 shutdown_connections 2024-04-01T01:47:51.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.244+0000 7fe333026640 1 --2- 172.21.15.70:0/3054931349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe32c1097e0 0x7fe32c10e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.244+0000 7fe333026640 1 --2- 172.21.15.70:0/3054931349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe32c108e20 0x7fe32c1092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.244+0000 7fe333026640 1 --2- 172.21.15.70:0/3054931349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe32c072b60 0x7fe32c072f80 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.244+0000 7fe333026640 1 -- 172.21.15.70:0/3054931349 >> 172.21.15.70:0/3054931349 conn(0x7fe32c06e630 msgr2=0x7fe32c070a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:51.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.244+0000 7fe333026640 1 -- 172.21.15.70:0/3054931349 shutdown_connections 2024-04-01T01:47:51.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.244+0000 7fe333026640 1 -- 172.21.15.70:0/3054931349 wait complete. 2024-04-01T01:47:51.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.245+0000 7fe333026640 1 Processor -- start 2024-04-01T01:47:51.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.245+0000 7fe333026640 1 -- start start 2024-04-01T01:47:51.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.245+0000 7fe333026640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe32c072b60 0x7fe32c1a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.245+0000 7fe333026640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe32c108e20 0x7fe32c1a6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.245+0000 7fe333026640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe32c1097e0 0x7fe32c1adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.245+0000 7fe333026640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c11bfa0 con 0x7fe32c1097e0 2024-04-01T01:47:51.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.245+0000 7fe333026640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c11be20 con 0x7fe32c108e20 2024-04-01T01:47:51.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.245+0000 7fe333026640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe32c11c120 con 0x7fe32c072b60 2024-04-01T01:47:51.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.246+0000 7fe331823640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe32c108e20 0x7fe32c1a6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.246+0000 7fe331823640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe32c108e20 0x7fe32c1a6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34176/0 (socket says 172.21.15.70:34176) 2024-04-01T01:47:51.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.246+0000 7fe331823640 1 -- 172.21.15.70:0/381368692 learned_addr learned my addr 172.21.15.70:0/381368692 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:51.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.246+0000 7fe332825640 1 --2- 172.21.15.70:0/381368692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe32c1097e0 0x7fe32c1adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.247+0000 7fe332024640 1 --2- 172.21.15.70:0/381368692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe32c072b60 0x7fe32c1a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.247+0000 7fe331823640 1 -- 172.21.15.70:0/381368692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe32c072b60 msgr2=0x7fe32c1a6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.247+0000 7fe331823640 1 --2- 172.21.15.70:0/381368692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe32c072b60 0x7fe32c1a6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.247+0000 7fe331823640 1 -- 172.21.15.70:0/381368692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe32c1097e0 msgr2=0x7fe32c1adc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.247+0000 7fe331823640 1 --2- 172.21.15.70:0/381368692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe32c1097e0 0x7fe32c1adc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.247+0000 7fe331823640 1 -- 172.21.15.70:0/381368692 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32c1ae270 con 0x7fe32c108e20 2024-04-01T01:47:51.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.247+0000 7fe331823640 1 --2- 172.21.15.70:0/381368692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe32c108e20 0x7fe32c1a6b90 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00e9c0 tx=0x7fe31c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:51.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.247+0000 7fe3237fe640 1 -- 172.21.15.70:0/381368692 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c00ccc0 con 0x7fe32c108e20 2024-04-01T01:47:51.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.248+0000 7fe333026640 1 -- 172.21.15.70:0/381368692 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe32c1ae560 con 0x7fe32c108e20 2024-04-01T01:47:51.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.248+0000 7fe3237fe640 1 -- 172.21.15.70:0/381368692 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe31c0054d0 con 0x7fe32c108e20 2024-04-01T01:47:51.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.248+0000 7fe3237fe640 1 -- 172.21.15.70:0/381368692 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c010430 con 0x7fe32c108e20 2024-04-01T01:47:51.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.248+0000 7fe333026640 1 -- 172.21.15.70:0/381368692 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe32c1aeb00 con 0x7fe32c108e20 2024-04-01T01:47:51.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:50 smithi189 ceph-mon[35753]: pgmap v789: 129 pgs: 129 active+clean; 1.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.1 MiB/s wr, 794 op/s 2024-04-01T01:47:51.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.251+0000 7fe336e61740 1 Processor -- start 2024-04-01T01:47:51.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.252+0000 7fe336e61740 1 -- start start 2024-04-01T01:47:51.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:50 smithi070 ceph-mon[32179]: pgmap v789: 129 pgs: 129 active+clean; 1.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.1 MiB/s wr, 794 op/s 2024-04-01T01:47:51.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.252+0000 7fe336e61740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2a5490 0x56506c2a58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.252+0000 7fe336e61740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2a5e80 0x56506c2aa980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.252+0000 7fe336e61740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2aaff0 0x56506c2ad480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.252+0000 7fe336e61740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56506c20a180 con 0x56506c2a5e80 2024-04-01T01:47:51.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.252+0000 7fe336e61740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56506c1b2b70 con 0x56506c2aaff0 2024-04-01T01:47:51.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.252+0000 7fe336e61740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56506c206220 con 0x56506c2a5490 2024-04-01T01:47:51.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.254+0000 7fe331823640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2aaff0 0x56506c2ad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.254+0000 7fe331823640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2aaff0 0x56506c2ad480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34180/0 (socket says 172.21.15.70:34180) 2024-04-01T01:47:51.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.254+0000 7fe331823640 1 -- 172.21.15.70:0/1296488105 learned_addr learned my addr 172.21.15.70:0/1296488105 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:51.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.255+0000 7fe3237fe640 1 -- 172.21.15.70:0/381368692 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fe31c0046c0 con 0x7fe32c108e20 2024-04-01T01:47:51.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.255+0000 7fe332825640 1 --2- 172.21.15.70:0/1296488105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2a5490 0x56506c2a58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.256+0000 7fe3237fe640 1 --2- 172.21.15.70:0/381368692 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe3180a25f0 0x7fe3180a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.256+0000 7fe3237fe640 1 -- 172.21.15.70:0/381368692 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe31c0ced80 con 0x7fe32c108e20 2024-04-01T01:47:51.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.257+0000 7fe331823640 1 -- 172.21.15.70:0/1296488105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2a5490 msgr2=0x56506c2a58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:51.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.257+0000 7fe331823640 1 --2- 172.21.15.70:0/1296488105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2a5490 0x56506c2a58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.257+0000 7fe331823640 1 -- 172.21.15.70:0/1296488105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2a5e80 msgr2=0x56506c2aa980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:51.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.257+0000 7fe331823640 1 --2- 172.21.15.70:0/1296488105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2a5e80 0x56506c2aa980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.257+0000 7fe331823640 1 -- 172.21.15.70:0/1296488105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56506c1c67c0 con 0x56506c2aaff0 2024-04-01T01:47:51.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.257+0000 7fe331823640 1 --2- 172.21.15.70:0/1296488105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2aaff0 0x56506c2ad480 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe31c097de0 tx=0x7fe31c009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c92d95847ab364a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:51.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.258+0000 7fe320bfe640 1 -- 172.21.15.70:0/1296488105 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c00ce60 con 0x56506c2aaff0 2024-04-01T01:47:51.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.258+0000 7fe320bfe640 1 -- 172.21.15.70:0/1296488105 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe31c005680 con 0x56506c2aaff0 2024-04-01T01:47:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.258+0000 7fe320bfe640 1 -- 172.21.15.70:0/1296488105 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c099710 con 0x56506c2aaff0 2024-04-01T01:47:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.258+0000 7fe332825640 1 --2- 172.21.15.70:0/381368692 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe3180a25f0 0x7fe3180a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.259+0000 7fe336e61740 1 -- 172.21.15.70:0/1296488105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2aaff0 msgr2=0x56506c2ad480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.259+0000 7fe336e61740 1 --2- 172.21.15.70:0/1296488105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2aaff0 0x56506c2ad480 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe31c097de0 tx=0x7fe31c009f90 comp rx=0 tx=0).stop 2024-04-01T01:47:51.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.259+0000 7fe332825640 1 --2- 172.21.15.70:0/381368692 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe3180a25f0 0x7fe3180a4ab0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x56506c2a66e0 tx=0x7fe3280046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.260+0000 7fe336e61740 1 -- 172.21.15.70:0/1296488105 shutdown_connections 2024-04-01T01:47:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.260+0000 7fe336e61740 1 --2- 172.21.15.70:0/1296488105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2aaff0 0x56506c2ad480 secure :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe31c097de0 tx=0x7fe31c009f90 comp rx=0 tx=0).stop 2024-04-01T01:47:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.260+0000 7fe336e61740 1 --2- 172.21.15.70:0/1296488105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2a5e80 0x56506c2aa980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.260+0000 7fe336e61740 1 --2- 172.21.15.70:0/1296488105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2a5490 0x56506c2a58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.260+0000 7fe336e61740 1 -- 172.21.15.70:0/1296488105 >> 172.21.15.70:0/1296488105 conn(0x56506c2a0a00 msgr2=0x56506c2a0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:51.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.262+0000 7fe336e61740 1 -- 172.21.15.70:0/1296488105 shutdown_connections 2024-04-01T01:47:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.262+0000 7fe336e61740 1 -- 172.21.15.70:0/1296488105 wait complete. 2024-04-01T01:47:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.262+0000 7fe336e61740 1 -- start start 2024-04-01T01:47:51.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.263+0000 7fe336e61740 1 Processor -- start 2024-04-01T01:47:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.264+0000 7fe336e61740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5490 0x56506c2c0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.264+0000 7fe336e61740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2a5e80 0x56506c2c1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.264+0000 7fe336e61740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2c60b0 0x56506c2c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.264+0000 7fe336e61740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56506c2bef50 con 0x56506c2c60b0 2024-04-01T01:47:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.264+0000 7fe336e61740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56506c1a84a0 con 0x56506c2a5490 2024-04-01T01:47:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.264+0000 7fe336e61740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56506c2bf0d0 con 0x56506c2a5e80 2024-04-01T01:47:51.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.264+0000 7fe336e61740 10 client.0 _ll_register_callbacks cb 0x56506c1abd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.264+0000 7fe332825640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2c60b0 0x56506c2c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.264+0000 7fe332825640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2c60b0 0x56506c2c84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41668/0 (socket says 172.21.15.70:41668) 2024-04-01T01:47:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.264+0000 7fe332825640 1 -- 172.21.15.70:0/3794434260 learned_addr learned my addr 172.21.15.70:0/3794434260 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.265+0000 7fe331823640 1 --2- 172.21.15.70:0/3794434260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2a5e80 0x56506c2c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.264+0000 7fe332024640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5490 0x56506c2c0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.265+0000 7fe332825640 1 -- 172.21.15.70:0/3794434260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2a5e80 msgr2=0x56506c2c1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.265+0000 7fe332825640 1 --2- 172.21.15.70:0/3794434260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2a5e80 0x56506c2c1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.265+0000 7fe332825640 1 -- 172.21.15.70:0/3794434260 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5490 msgr2=0x56506c2c0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.265+0000 7fe332825640 1 --2- 172.21.15.70:0/3794434260 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5490 0x56506c2c0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.265+0000 7fe332825640 1 -- 172.21.15.70:0/3794434260 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56506bfe87d0 con 0x56506c2c60b0 2024-04-01T01:47:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.265+0000 7fe332024640 1 --2- 172.21.15.70:0/3794434260 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5490 0x56506c2c0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:47:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.265+0000 7fe332825640 1 --2- 172.21.15.70:0/3794434260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2c60b0 0x56506c2c84a0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fe32800bb10 tx=0x7fe328004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d665728a62f367f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.265+0000 7fe315ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:47:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.266+0000 7fe315ffb640 1 -- 172.21.15.70:0/3794434260 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32802b060 con 0x56506c2c60b0 2024-04-01T01:47:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.266+0000 7fe315ffb640 1 -- 172.21.15.70:0/3794434260 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe32801c9a0 con 0x56506c2c60b0 2024-04-01T01:47:51.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.266+0000 7fe336e61740 1 -- 172.21.15.70:0/3794434260 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe31c005680 con 0x56506c2c60b0 2024-04-01T01:47:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.266+0000 7fe315ffb640 1 -- 172.21.15.70:0/3794434260 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe328032050 con 0x56506c2c60b0 2024-04-01T01:47:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.266+0000 7fe336e61740 10 client.19662 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.266+0000 7fe336e61740 1 -- 172.21.15.70:0/3794434260 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56506bf1e3f0 con 0x56506c2c60b0 2024-04-01T01:47:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.267+0000 7fe315ffb640 1 -- 172.21.15.70:0/3794434260 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe31c005680 con 0x56506c2c60b0 2024-04-01T01:47:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.267+0000 7fe315ffb640 1 -- 172.21.15.70:0/3794434260 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe3280103d0 con 0x56506c2c60b0 2024-04-01T01:47:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.267+0000 7fe336e61740 10 client.19662 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.267+0000 7fe336e61740 10 client.19662 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.267+0000 7fe336e61740 20 client.19662 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:51.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.267+0000 7fe336e61740 1 --2- 172.21.15.70:0/3794434260 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56506c2cd250 0x56506c2cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.267+0000 7fe336e61740 4 client.19662 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:47:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.267+0000 7fe336e61740 1 -- 172.21.15.70:0/3794434260 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56506c2176c0 con 0x56506c2cd250 2024-04-01T01:47:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.267+0000 7fe332024640 1 --2- 172.21.15.70:0/3794434260 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56506c2cd250 0x56506c2cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.268+0000 7fe332024640 1 --2- 172.21.15.70:0/3794434260 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56506c2cd250 0x56506c2cf6d0 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f94a2feb06209f2 server_cookie=eddeb6aa39dd9145 in_seq=0 out_seq=0 2024-04-01T01:47:51.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.268+0000 7fe315ffb640 10 client.19662 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.269+0000 7fe315ffb640 1 -- 172.21.15.70:0/3794434260 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56506c2176c0 con 0x56506c2cd250 2024-04-01T01:47:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.269+0000 7fe315ffb640 10 client.19662 handle_command_reply: tid=0 2024-04-01T01:47:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.269+0000 7fe336e61740 1 client.19662 shutdown 2024-04-01T01:47:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.270+0000 7fe336e61740 1 -- 172.21.15.70:0/3794434260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2c60b0 msgr2=0x56506c2c84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.270+0000 7fe336e61740 1 --2- 172.21.15.70:0/3794434260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2c60b0 0x56506c2c84a0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fe32800bb10 tx=0x7fe328004730 comp rx=0 tx=0).stop 2024-04-01T01:47:51.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.270+0000 7fe336e61740 1 -- 172.21.15.70:0/3794434260 shutdown_connections 2024-04-01T01:47:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.270+0000 7fe336e61740 1 --2- 172.21.15.70:0/3794434260 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56506c2cd250 0x56506c2cf6d0 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.270+0000 7fe336e61740 1 --2- 172.21.15.70:0/3794434260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2c60b0 0x56506c2c84a0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.270+0000 7fe336e61740 1 --2- 172.21.15.70:0/3794434260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2a5e80 0x56506c2c1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.270+0000 7fe336e61740 1 --2- 172.21.15.70:0/3794434260 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5490 0x56506c2c0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.270+0000 7fe336e61740 1 -- 172.21.15.70:0/3794434260 >> 172.21.15.70:0/3794434260 conn(0x56506c2a0a00 msgr2=0x56506c29fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.270+0000 7fe315ffb640 0 client.19662 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.270+0000 7fe336e61740 1 -- 172.21.15.70:0/3794434260 shutdown_connections 2024-04-01T01:47:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.270+0000 7fe336e61740 1 -- 172.21.15.70:0/3794434260 wait complete. 2024-04-01T01:47:51.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.270+0000 7fe336e61740 20 client.19662 trim_cache size 0 max 16384 2024-04-01T01:47:51.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.307+0000 7fe336e61740 1 Processor -- start 2024-04-01T01:47:51.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.307+0000 7fe336e61740 1 -- start start 2024-04-01T01:47:51.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.308+0000 7fe336e61740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5490 0x56506c2c0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.308+0000 7fe336e61740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2a5e80 0x56506c2c1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.308+0000 7fe336e61740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2c60b0 0x56506c2a7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.308+0000 7fe336e61740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe31000fda0 con 0x56506c2a5e80 2024-04-01T01:47:51.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.308+0000 7fe336e61740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe3100118e0 con 0x56506c2a5490 2024-04-01T01:47:51.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.308+0000 7fe336e61740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe310010470 con 0x56506c2c60b0 2024-04-01T01:47:51.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.308+0000 7fe332825640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2c60b0 0x56506c2a7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.308+0000 7fe332024640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5490 0x56506c2c0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.308+0000 7fe332825640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2c60b0 0x56506c2a7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42844/0 (socket says 172.21.15.70:42844) 2024-04-01T01:47:51.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.308+0000 7fe332825640 1 -- 172.21.15.70:0/2665226169 learned_addr learned my addr 172.21.15.70:0/2665226169 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:51.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.308+0000 7fe331823640 1 --2- 172.21.15.70:0/2665226169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2a5e80 0x56506c2c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.308+0000 7fe332024640 1 -- 172.21.15.70:0/2665226169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2c60b0 msgr2=0x56506c2a7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:51.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.308+0000 7fe332024640 1 --2- 172.21.15.70:0/2665226169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2c60b0 0x56506c2a7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.308+0000 7fe332024640 1 -- 172.21.15.70:0/2665226169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2a5e80 msgr2=0x56506c2c1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:51.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.308+0000 7fe332024640 1 --2- 172.21.15.70:0/2665226169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2a5e80 0x56506c2c1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.309+0000 7fe332024640 1 -- 172.21.15.70:0/2665226169 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56506bede0f0 con 0x56506c2a5490 2024-04-01T01:47:51.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.309+0000 7fe332024640 1 --2- 172.21.15.70:0/2665226169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5490 0x56506c2c0b40 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe324036040 tx=0x7fe324034f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3c6bcdd0a1a6411 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:51.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.309+0000 7fe317fff640 1 -- 172.21.15.70:0/2665226169 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe324033ad0 con 0x56506c2a5490 2024-04-01T01:47:51.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.309+0000 7fe317fff640 1 -- 172.21.15.70:0/2665226169 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe324003e20 con 0x56506c2a5490 2024-04-01T01:47:51.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.309+0000 7fe317fff640 1 -- 172.21.15.70:0/2665226169 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32402cdb0 con 0x56506c2a5490 2024-04-01T01:47:51.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.310+0000 7fe336e61740 1 -- 172.21.15.70:0/2665226169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5490 msgr2=0x56506c2c0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.310+0000 7fe336e61740 1 --2- 172.21.15.70:0/2665226169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5490 0x56506c2c0b40 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe324036040 tx=0x7fe324034f90 comp rx=0 tx=0).stop 2024-04-01T01:47:51.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.310+0000 7fe336e61740 1 -- 172.21.15.70:0/2665226169 shutdown_connections 2024-04-01T01:47:51.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.310+0000 7fe336e61740 1 --2- 172.21.15.70:0/2665226169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2c60b0 0x56506c2a7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.310+0000 7fe336e61740 1 --2- 172.21.15.70:0/2665226169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2a5e80 0x56506c2c1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.310+0000 7fe336e61740 1 --2- 172.21.15.70:0/2665226169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5490 0x56506c2c0b40 secure :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fe324036040 tx=0x7fe324034f90 comp rx=0 tx=0).stop 2024-04-01T01:47:51.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.310+0000 7fe336e61740 1 -- 172.21.15.70:0/2665226169 >> 172.21.15.70:0/2665226169 conn(0x56506c2a0a00 msgr2=0x56506c2cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:51.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.311+0000 7fe336e61740 1 -- 172.21.15.70:0/2665226169 shutdown_connections 2024-04-01T01:47:51.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.311+0000 7fe336e61740 1 -- 172.21.15.70:0/2665226169 wait complete. 2024-04-01T01:47:51.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.312+0000 7fe336e61740 1 -- start start 2024-04-01T01:47:51.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.312+0000 7fe336e61740 1 Processor -- start 2024-04-01T01:47:51.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.312+0000 7fe336e61740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5e80 0x56506c2de390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.312+0000 7fe336e61740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2c60b0 0x56506c2de8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.312+0000 7fe336e61740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2df7d0 0x56506c2dfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.312+0000 7fe336e61740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe310011210 con 0x56506c2df7d0 2024-04-01T01:47:51.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.312+0000 7fe336e61740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe310010b40 con 0x56506c2a5e80 2024-04-01T01:47:51.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.312+0000 7fe336e61740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe31000dd80 con 0x56506c2c60b0 2024-04-01T01:47:51.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.312+0000 7fe336e61740 10 client.0 _ll_register_callbacks cb 0x7fe32800b9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:47:51.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.312+0000 7fe332024640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5e80 0x56506c2de390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.312+0000 7fe332024640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5e80 0x56506c2de390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:34194/0 (socket says 172.21.15.70:34194) 2024-04-01T01:47:51.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.312+0000 7fe332024640 1 -- 172.21.15.70:0/1848214257 learned_addr learned my addr 172.21.15.70:0/1848214257 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:47:51.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.313+0000 7fe331823640 1 --2- 172.21.15.70:0/1848214257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2c60b0 0x56506c2de8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.313+0000 7fe332024640 1 -- 172.21.15.70:0/1848214257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2c60b0 msgr2=0x56506c2de8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:51.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.313+0000 7fe332024640 1 --2- 172.21.15.70:0/1848214257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2c60b0 0x56506c2de8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.313+0000 7fe332024640 1 -- 172.21.15.70:0/1848214257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2df7d0 msgr2=0x56506c2dfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:47:51.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.313+0000 7fe332024640 1 --2- 172.21.15.70:0/1848214257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2df7d0 0x56506c2dfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.313+0000 7fe332024640 1 -- 172.21.15.70:0/1848214257 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56506c2e0190 con 0x56506c2a5e80 2024-04-01T01:47:51.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.313+0000 7fe332024640 1 --2- 172.21.15.70:0/1848214257 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5e80 0x56506c2de390 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fe324034ec0 tx=0x7fe32402c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdf2f3635ac9a901 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:47:51.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.313+0000 7fe315ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:47:51.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.313+0000 7fe315ffb640 1 -- 172.21.15.70:0/1848214257 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe324003970 con 0x56506c2a5e80 2024-04-01T01:47:51.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.314+0000 7fe315ffb640 1 -- 172.21.15.70:0/1848214257 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe324003b10 con 0x56506c2a5e80 2024-04-01T01:47:51.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.314+0000 7fe336e61740 1 -- 172.21.15.70:0/1848214257 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe324003e20 con 0x56506c2a5e80 2024-04-01T01:47:51.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.314+0000 7fe315ffb640 1 -- 172.21.15.70:0/1848214257 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe324033ad0 con 0x56506c2a5e80 2024-04-01T01:47:51.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.315+0000 7fe336e61740 10 client.29762 fetch_fsmap learned FSMap version 31 2024-04-01T01:47:51.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.315+0000 7fe336e61740 1 -- 172.21.15.70:0/1848214257 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56506c355750 con 0x56506c2a5e80 2024-04-01T01:47:51.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.315+0000 7fe315ffb640 1 -- 172.21.15.70:0/1848214257 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe324003e20 con 0x56506c2a5e80 2024-04-01T01:47:51.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.315+0000 7fe315ffb640 1 -- 172.21.15.70:0/1848214257 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe324047040 con 0x56506c2a5e80 2024-04-01T01:47:51.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.315+0000 7fe336e61740 10 client.29762 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:47:51.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.315+0000 7fe336e61740 10 client.29762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:47:51.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.315+0000 7fe336e61740 20 client.29762 populate_metadata read hostname 'smithi070' 2024-04-01T01:47:51.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.315+0000 7fe336e61740 1 --2- 172.21.15.70:0/1848214257 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56506c356010 0x56506c358430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:47:51.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.315+0000 7fe336e61740 4 client.29762 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:47:51.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.315+0000 7fe336e61740 1 -- 172.21.15.70:0/1848214257 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56506c211a10 con 0x56506c356010 2024-04-01T01:47:51.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.316+0000 7fe331823640 1 --2- 172.21.15.70:0/1848214257 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56506c356010 0x56506c358430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:47:51.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.317+0000 7fe331823640 1 --2- 172.21.15.70:0/1848214257 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56506c356010 0x56506c358430 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f08fb8c09519b209 server_cookie=dfb7a36123b394a7 in_seq=0 out_seq=0 2024-04-01T01:47:51.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.317+0000 7fe315ffb640 10 client.29762 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:51.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.317+0000 7fe315ffb640 1 -- 172.21.15.70:0/1848214257 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x56506c211a10 con 0x56506c356010 2024-04-01T01:47:51.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.317+0000 7fe315ffb640 10 client.29762 handle_command_reply: tid=0 2024-04-01T01:47:51.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.317+0000 7fe336e61740 1 client.29762 shutdown 2024-04-01T01:47:51.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.317+0000 7fe336e61740 1 -- 172.21.15.70:0/1848214257 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5e80 msgr2=0x56506c2de390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.318+0000 7fe336e61740 1 --2- 172.21.15.70:0/1848214257 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5e80 0x56506c2de390 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fe324034ec0 tx=0x7fe32402c920 comp rx=0 tx=0).stop 2024-04-01T01:47:51.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.318+0000 7fe336e61740 1 -- 172.21.15.70:0/1848214257 shutdown_connections 2024-04-01T01:47:51.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.318+0000 7fe336e61740 1 --2- 172.21.15.70:0/1848214257 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56506c356010 0x56506c358430 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.318+0000 7fe336e61740 1 --2- 172.21.15.70:0/1848214257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56506c2df7d0 0x56506c2dfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.318+0000 7fe336e61740 1 --2- 172.21.15.70:0/1848214257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56506c2c60b0 0x56506c2de8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.318+0000 7fe336e61740 1 --2- 172.21.15.70:0/1848214257 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56506c2a5e80 0x56506c2de390 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.318+0000 7fe336e61740 1 -- 172.21.15.70:0/1848214257 >> 172.21.15.70:0/1848214257 conn(0x56506c2a0a00 msgr2=0x56506c2a7560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:51.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.318+0000 7fe315ffb640 0 client.29762 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:47:51.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.318+0000 7fe336e61740 1 -- 172.21.15.70:0/1848214257 shutdown_connections 2024-04-01T01:47:51.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.318+0000 7fe336e61740 1 -- 172.21.15.70:0/1848214257 wait complete. 2024-04-01T01:47:51.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.318+0000 7fe336e61740 20 client.29762 trim_cache size 0 max 16384 2024-04-01T01:47:51.323 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:47:51.323 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "scrub active (321 inodes in the stack)", 2024-04-01T01:47:51.323 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:47:51.323 INFO:teuthology.orchestra.run.smithi070.stdout: "1155fbee-5b0c-4291-a6ce-12c5a9e3250d": { 2024-04-01T01:47:51.323 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:47:51.323 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "1155fbee-5b0c-4291-a6ce-12c5a9e3250d", 2024-04-01T01:47:51.323 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:47:51.323 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:47:51.323 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:47:51.323 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:47:51.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.323+0000 7fe333026640 1 -- 172.21.15.70:0/381368692 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe3180a25f0 msgr2=0x7fe3180a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.323+0000 7fe333026640 1 --2- 172.21.15.70:0/381368692 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe3180a25f0 0x7fe3180a4ab0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x56506c2a66e0 tx=0x7fe3280046c0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.323+0000 7fe333026640 1 -- 172.21.15.70:0/381368692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe32c108e20 msgr2=0x7fe32c1a6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:47:51.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.323+0000 7fe333026640 1 --2- 172.21.15.70:0/381368692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe32c108e20 0x7fe32c1a6b90 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00e9c0 tx=0x7fe31c00ee80 comp rx=0 tx=0).stop 2024-04-01T01:47:51.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.325+0000 7fe333026640 1 -- 172.21.15.70:0/381368692 shutdown_connections 2024-04-01T01:47:51.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.325+0000 7fe333026640 1 --2- 172.21.15.70:0/381368692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe32c1097e0 0x7fe32c1adc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.325+0000 7fe333026640 1 --2- 172.21.15.70:0/381368692 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe32c108e20 0x7fe32c1a6b90 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.325+0000 7fe333026640 1 --2- 172.21.15.70:0/381368692 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe3180a25f0 0x7fe3180a4ab0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.325+0000 7fe333026640 1 --2- 172.21.15.70:0/381368692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe32c072b60 0x7fe32c1a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:47:51.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.325+0000 7fe333026640 1 -- 172.21.15.70:0/381368692 >> 172.21.15.70:0/381368692 conn(0x7fe32c06e630 msgr2=0x7fe32c0702b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:47:51.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.325+0000 7fe333026640 1 -- 172.21.15.70:0/381368692 shutdown_connections 2024-04-01T01:47:51.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:47:51.325+0000 7fe333026640 1 -- 172.21.15.70:0/381368692 wait complete. 2024-04-01T01:47:51.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:50 smithi081 ceph-mon[35142]: pgmap v789: 129 pgs: 129 active+clean; 1.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.1 MiB/s wr, 794 op/s 2024-04-01T01:47:51.347 INFO:tasks.cephfs.filesystem:scrub status for tag:1155fbee-5b0c-4291-a6ce-12c5a9e3250d - {'path': '/', 'tag': '1155fbee-5b0c-4291-a6ce-12c5a9e3250d', 'options': 'recursive,force'} 2024-04-01T01:47:51.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 785.0 s, 781.4 tps, lat 125.501 ms stddev 59.925 2024-04-01T01:47:52.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:51 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:47:52.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:51 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:47:52.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:51 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:47:52.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:51 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:47:52.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:51 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:47:52.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:51 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:47:52.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:51 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:47:52.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:51 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:47:52.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:51 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:47:53.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:52 smithi189 ceph-mon[35753]: pgmap v790: 129 pgs: 129 active+clean; 1.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.4 MiB/s wr, 738 op/s 2024-04-01T01:47:53.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:52 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:47:53.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:52 smithi070 ceph-mon[32179]: pgmap v790: 129 pgs: 129 active+clean; 1.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.4 MiB/s wr, 738 op/s 2024-04-01T01:47:53.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:52 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:47:53.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:52 smithi081 ceph-mon[35142]: pgmap v790: 129 pgs: 129 active+clean; 1.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.4 MiB/s wr, 738 op/s 2024-04-01T01:47:53.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:52 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:47:55.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:54 smithi189 ceph-mon[35753]: pgmap v791: 129 pgs: 129 active+clean; 1.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 6.9 MiB/s wr, 849 op/s 2024-04-01T01:47:55.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:54 smithi070 ceph-mon[32179]: pgmap v791: 129 pgs: 129 active+clean; 1.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 6.9 MiB/s wr, 849 op/s 2024-04-01T01:47:55.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:54 smithi081 ceph-mon[35142]: pgmap v791: 129 pgs: 129 active+clean; 1.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 6.9 MiB/s wr, 849 op/s 2024-04-01T01:47:56.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:55 smithi189 ceph-mon[35753]: pgmap v792: 129 pgs: 129 active+clean; 1.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 6.4 MiB/s wr, 799 op/s 2024-04-01T01:47:56.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:55 smithi070 ceph-mon[32179]: pgmap v792: 129 pgs: 129 active+clean; 1.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 6.4 MiB/s wr, 799 op/s 2024-04-01T01:47:56.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:55 smithi081 ceph-mon[35142]: pgmap v792: 129 pgs: 129 active+clean; 1.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 6.4 MiB/s wr, 799 op/s 2024-04-01T01:47:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 790.0 s, 738.4 tps, lat 135.966 ms stddev 143.203 2024-04-01T01:47:57.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:56 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:47:57.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:56 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:47:57.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:56 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:47:58.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:47:57 smithi189 ceph-mon[35753]: pgmap v793: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 7.4 MiB/s wr, 914 op/s 2024-04-01T01:47:58.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:47:57 smithi070 ceph-mon[32179]: pgmap v793: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 7.4 MiB/s wr, 914 op/s 2024-04-01T01:47:58.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:47:57 smithi081 ceph-mon[35142]: pgmap v793: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 7.4 MiB/s wr, 914 op/s 2024-04-01T01:48:00.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:00 smithi081 ceph-mon[35142]: pgmap v794: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 7.0 MiB/s wr, 876 op/s 2024-04-01T01:48:00.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:00 smithi189 ceph-mon[35753]: pgmap v794: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 7.0 MiB/s wr, 876 op/s 2024-04-01T01:48:00.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:00 smithi070 ceph-mon[32179]: pgmap v794: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 7.0 MiB/s wr, 876 op/s 2024-04-01T01:48:01.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 795.0 s, 736.4 tps, lat 135.401 ms stddev 146.709 2024-04-01T01:48:02.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:02 smithi081 ceph-mon[35142]: pgmap v795: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 6.2 MiB/s wr, 791 op/s 2024-04-01T01:48:02.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:02 smithi189 ceph-mon[35753]: pgmap v795: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 6.2 MiB/s wr, 791 op/s 2024-04-01T01:48:02.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:02 smithi070 ceph-mon[32179]: pgmap v795: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 6.2 MiB/s wr, 791 op/s 2024-04-01T01:48:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.933+0000 7f74879c7640 1 -- 172.21.15.70:0/2176313173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74801018a0 msgr2=0x7f7480101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.933+0000 7f74879c7640 1 --2- 172.21.15.70:0/2176313173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74801018a0 0x7f7480101d20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7470009860 tx=0x7f747002bb70 comp rx=0 tx=0).stop 2024-04-01T01:48:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.933+0000 7f74879c7640 1 -- 172.21.15.70:0/2176313173 shutdown_connections 2024-04-01T01:48:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.933+0000 7f74879c7640 1 --2- 172.21.15.70:0/2176313173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7480102260 0x7f7480108af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.933+0000 7f74879c7640 1 --2- 172.21.15.70:0/2176313173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74801018a0 0x7f7480101d20 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.933+0000 7f74879c7640 1 --2- 172.21.15.70:0/2176313173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74801006b0 0x7f7480100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.933+0000 7f74879c7640 1 -- 172.21.15.70:0/2176313173 >> 172.21.15.70:0/2176313173 conn(0x7f74800fbc20 msgr2=0x7f74800fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.934+0000 7f74879c7640 1 -- 172.21.15.70:0/2176313173 shutdown_connections 2024-04-01T01:48:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.934+0000 7f74879c7640 1 -- 172.21.15.70:0/2176313173 wait complete. 2024-04-01T01:48:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.935+0000 7f74879c7640 1 Processor -- start 2024-04-01T01:48:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.935+0000 7f74879c7640 1 -- start start 2024-04-01T01:48:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.935+0000 7f74879c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74801006b0 0x7f748010ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.935+0000 7f74879c7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74801018a0 0x7f74801091f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.935+0000 7f74879c7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7480102260 0x7f7480109730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.935+0000 7f74879c7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f748010d300 con 0x7f7480102260 2024-04-01T01:48:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.935+0000 7f74879c7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f748010d180 con 0x7f74801006b0 2024-04-01T01:48:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.935+0000 7f74879c7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f748010d480 con 0x7f74801018a0 2024-04-01T01:48:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.936+0000 7f7484f62640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74801018a0 0x7f74801091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.936+0000 7f7484f62640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74801018a0 0x7f74801091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37186/0 (socket says 172.21.15.70:37186) 2024-04-01T01:48:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.936+0000 7f7484f62640 1 -- 172.21.15.70:0/996821070 learned_addr learned my addr 172.21.15.70:0/996821070 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.936+0000 7f7485f64640 1 --2- 172.21.15.70:0/996821070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7480102260 0x7f7480109730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.936+0000 7f7485f64640 1 -- 172.21.15.70:0/996821070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74801018a0 msgr2=0x7f74801091f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.936+0000 7f7485f64640 1 --2- 172.21.15.70:0/996821070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74801018a0 0x7f74801091f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.936+0000 7f7485f64640 1 -- 172.21.15.70:0/996821070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74801006b0 msgr2=0x7f748010ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.936+0000 7f7485f64640 1 --2- 172.21.15.70:0/996821070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74801006b0 0x7f748010ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.936+0000 7f7485f64640 1 -- 172.21.15.70:0/996821070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7480109ff0 con 0x7f7480102260 2024-04-01T01:48:02.938 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.937+0000 7f7485f64640 1 --2- 172.21.15.70:0/996821070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7480102260 0x7f7480109730 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f7478009fd0 tx=0x7f747800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:02.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.937+0000 7f74767fc640 1 -- 172.21.15.70:0/996821070 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747800ce20 con 0x7f7480102260 2024-04-01T01:48:02.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.937+0000 7f74767fc640 1 -- 172.21.15.70:0/996821070 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7478005020 con 0x7f7480102260 2024-04-01T01:48:02.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.937+0000 7f74767fc640 1 -- 172.21.15.70:0/996821070 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7478010690 con 0x7f7480102260 2024-04-01T01:48:02.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.937+0000 7f74879c7640 1 -- 172.21.15.70:0/996821070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74801a1a80 con 0x7f7480102260 2024-04-01T01:48:02.939 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.937+0000 7f74879c7640 1 -- 172.21.15.70:0/996821070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74801a1f50 con 0x7f7480102260 2024-04-01T01:48:02.941 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.941+0000 7f748a5a6740 1 Processor -- start 2024-04-01T01:48:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.942+0000 7f74767fc640 1 -- 172.21.15.70:0/996821070 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f74780051d0 con 0x7f7480102260 2024-04-01T01:48:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.943+0000 7f74767fc640 1 --2- 172.21.15.70:0/996821070 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f746c0a2640 0x7f746c0a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.943+0000 7f74767fc640 1 -- 172.21.15.70:0/996821070 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f74780cf250 con 0x7f7480102260 2024-04-01T01:48:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.943+0000 7f748a5a6740 1 -- start start 2024-04-01T01:48:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.944+0000 7f748a5a6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c38d0 0x559da03c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.944+0000 7f748a5a6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c42c0 0x559da03d15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.944+0000 7f748a5a6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 0x559da03d40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.944+0000 7f748a5a6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559da03278f0 con 0x559da03c38d0 2024-04-01T01:48:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.944+0000 7f748a5a6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x559da02d08d0 con 0x559da03d1c40 2024-04-01T01:48:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.944+0000 7f748a5a6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559da0323990 con 0x559da03c42c0 2024-04-01T01:48:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.945+0000 7f7485763640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c38d0 0x559da03c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.945+0000 7f7485763640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c38d0 0x559da03c3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39128/0 (socket says 172.21.15.70:39128) 2024-04-01T01:48:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.945+0000 7f7485763640 1 -- 172.21.15.70:0/1755785081 learned_addr learned my addr 172.21.15.70:0/1755785081 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.945+0000 7f7484f62640 1 --2- 172.21.15.70:0/996821070 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f746c0a2640 0x7f746c0a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.946+0000 7f7485763640 1 -- 172.21.15.70:0/1755785081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c42c0 msgr2=0x559da03d15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.946+0000 7f7485763640 1 --2- 172.21.15.70:0/1755785081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c42c0 0x559da03d15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.946+0000 7f7485763640 1 -- 172.21.15.70:0/1755785081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 msgr2=0x559da03d40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.946+0000 7f7485763640 1 --2- 172.21.15.70:0/1755785081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 0x559da03d40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.946+0000 7f7485763640 1 -- 172.21.15.70:0/1755785081 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559da003c210 con 0x559da03c38d0 2024-04-01T01:48:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.946+0000 7f7485763640 1 --2- 172.21.15.70:0/1755785081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c38d0 0x559da03c3cf0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f747c009b20 tx=0x7f747c00e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6bcf749652dd690 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.946+0000 7f7484f62640 1 --2- 172.21.15.70:0/996821070 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f746c0a2640 0x7f746c0a4b00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f748010a8d0 tx=0x7f74700094b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.947+0000 7f746b7fe640 1 -- 172.21.15.70:0/1755785081 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c00ed80 con 0x559da03c38d0 2024-04-01T01:48:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.947+0000 7f746b7fe640 1 -- 172.21.15.70:0/1755785081 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f747c0035f0 con 0x559da03c38d0 2024-04-01T01:48:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.947+0000 7f746b7fe640 1 -- 172.21.15.70:0/1755785081 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c010690 con 0x559da03c38d0 2024-04-01T01:48:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.948+0000 7f748a5a6740 1 -- 172.21.15.70:0/1755785081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c38d0 msgr2=0x559da03c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.948+0000 7f748a5a6740 1 --2- 172.21.15.70:0/1755785081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c38d0 0x559da03c3cf0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f747c009b20 tx=0x7f747c00e350 comp rx=0 tx=0).stop 2024-04-01T01:48:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.948+0000 7f748a5a6740 1 -- 172.21.15.70:0/1755785081 shutdown_connections 2024-04-01T01:48:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.948+0000 7f748a5a6740 1 --2- 172.21.15.70:0/1755785081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 0x559da03d40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.948+0000 7f748a5a6740 1 --2- 172.21.15.70:0/1755785081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c42c0 0x559da03d15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.948+0000 7f748a5a6740 1 --2- 172.21.15.70:0/1755785081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c38d0 0x559da03c3cf0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.949 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.948+0000 7f748a5a6740 1 -- 172.21.15.70:0/1755785081 >> 172.21.15.70:0/1755785081 conn(0x559da03bee40 msgr2=0x559da03bf240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.948+0000 7f748a5a6740 1 -- 172.21.15.70:0/1755785081 shutdown_connections 2024-04-01T01:48:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.948+0000 7f748a5a6740 1 -- 172.21.15.70:0/1755785081 wait complete. 2024-04-01T01:48:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.949+0000 7f748a5a6740 1 -- start start 2024-04-01T01:48:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.949+0000 7f748a5a6740 1 Processor -- start 2024-04-01T01:48:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.949+0000 7f748a5a6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c38d0 0x559da03c6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.949+0000 7f748a5a6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03c42c0 0x559da03c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.949+0000 7f748a5a6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03d1c40 0x559da03cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:02.950 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.949+0000 7f748a5a6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559da03c4fa0 con 0x559da03d1c40 2024-04-01T01:48:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.949+0000 7f748a5a6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x559da02c5f80 con 0x559da03c42c0 2024-04-01T01:48:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.949+0000 7f748a5a6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559da03c5120 con 0x559da03c38d0 2024-04-01T01:48:02.951 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.949+0000 7f748a5a6740 10 client.0 _ll_register_callbacks cb 0x559da031e290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.950+0000 7f7484f62640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03c42c0 0x559da03c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.950+0000 7f7484f62640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03c42c0 0x559da03c6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47950/0 (socket says 172.21.15.70:47950) 2024-04-01T01:48:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.950+0000 7f7484f62640 1 -- 172.21.15.70:0/1363932248 learned_addr learned my addr 172.21.15.70:0/1363932248 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.950+0000 7f7485f64640 1 --2- 172.21.15.70:0/1363932248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03d1c40 0x559da03cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.950+0000 7f7485763640 1 --2- 172.21.15.70:0/1363932248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c38d0 0x559da03c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.950+0000 7f7484f62640 1 -- 172.21.15.70:0/1363932248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c38d0 msgr2=0x559da03c6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.950+0000 7f7484f62640 1 --2- 172.21.15.70:0/1363932248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c38d0 0x559da03c6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.950+0000 7f7484f62640 1 -- 172.21.15.70:0/1363932248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03d1c40 msgr2=0x559da03cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.950+0000 7f7484f62640 1 --2- 172.21.15.70:0/1363932248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03d1c40 0x559da03cbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.950+0000 7f7484f62640 1 -- 172.21.15.70:0/1363932248 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559da0108620 con 0x559da03c42c0 2024-04-01T01:48:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.951+0000 7f7484f62640 1 --2- 172.21.15.70:0/1363932248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03c42c0 0x559da03c6da0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f74700057c0 tx=0x7f7470005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5089dfd6d567125c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.951+0000 7f7468ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.951+0000 7f7468ff9640 1 -- 172.21.15.70:0/1363932248 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747004e020 con 0x559da03c42c0 2024-04-01T01:48:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.951+0000 7f7468ff9640 1 -- 172.21.15.70:0/1363932248 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f74700043c0 con 0x559da03c42c0 2024-04-01T01:48:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.951+0000 7f748a5a6740 1 -- 172.21.15.70:0/1363932248 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f747c0035f0 con 0x559da03c42c0 2024-04-01T01:48:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.951+0000 7f7468ff9640 1 -- 172.21.15.70:0/1363932248 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7470059050 con 0x559da03c42c0 2024-04-01T01:48:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.952+0000 7f748a5a6740 10 client.29768 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.952+0000 7f748a5a6740 1 -- 172.21.15.70:0/1363932248 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559da03e9230 con 0x559da03c42c0 2024-04-01T01:48:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.952+0000 7f7468ff9640 1 -- 172.21.15.70:0/1363932248 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f747c0035f0 con 0x559da03c42c0 2024-04-01T01:48:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.952+0000 7f7468ff9640 1 -- 172.21.15.70:0/1363932248 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f7470003ab0 con 0x559da03c42c0 2024-04-01T01:48:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.952+0000 7f748a5a6740 10 client.29768 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.952+0000 7f748a5a6740 10 client.29768 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.952+0000 7f748a5a6740 20 client.29768 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.952+0000 7f748a5a6740 1 --2- 172.21.15.70:0/1363932248 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559da03ea8c0 0x559da03eccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.952+0000 7f748a5a6740 4 client.29768 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.952+0000 7f748a5a6740 1 -- 172.21.15.70:0/1363932248 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559da033b120 con 0x559da03ea8c0 2024-04-01T01:48:02.955 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.952+0000 7f7485763640 1 --2- 172.21.15.70:0/1363932248 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559da03ea8c0 0x559da03eccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.953+0000 7f7485763640 1 --2- 172.21.15.70:0/1363932248 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559da03ea8c0 0x559da03eccf0 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6428892f31ec6391 server_cookie=6bc2a1133eb0ff45 in_seq=0 out_seq=0 2024-04-01T01:48:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.953+0000 7f7468ff9640 10 client.29768 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.954+0000 7f7468ff9640 1 -- 172.21.15.70:0/1363932248 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559da033b120 con 0x559da03ea8c0 2024-04-01T01:48:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.954+0000 7f7468ff9640 10 client.29768 handle_command_reply: tid=0 2024-04-01T01:48:02.956 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.955+0000 7f748a5a6740 1 client.29768 shutdown 2024-04-01T01:48:02.957 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.957+0000 7f748a5a6740 1 -- 172.21.15.70:0/1363932248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03c42c0 msgr2=0x559da03c6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.957+0000 7f748a5a6740 1 --2- 172.21.15.70:0/1363932248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03c42c0 0x559da03c6da0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f74700057c0 tx=0x7f7470005f90 comp rx=0 tx=0).stop 2024-04-01T01:48:02.958 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.958+0000 7f748a5a6740 1 -- 172.21.15.70:0/1363932248 shutdown_connections 2024-04-01T01:48:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.958+0000 7f748a5a6740 1 --2- 172.21.15.70:0/1363932248 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559da03ea8c0 0x559da03eccf0 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.958+0000 7f748a5a6740 1 --2- 172.21.15.70:0/1363932248 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03d1c40 0x559da03cbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.958+0000 7f748a5a6740 1 --2- 172.21.15.70:0/1363932248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03c42c0 0x559da03c6da0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.958+0000 7f748a5a6740 1 --2- 172.21.15.70:0/1363932248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c38d0 0x559da03c6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.959 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.958+0000 7f748a5a6740 1 -- 172.21.15.70:0/1363932248 >> 172.21.15.70:0/1363932248 conn(0x559da03bee40 msgr2=0x559da03bdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.961+0000 7f7468ff9640 0 client.29768 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:02.961 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.961+0000 7f748a5a6740 1 -- 172.21.15.70:0/1363932248 shutdown_connections 2024-04-01T01:48:02.962 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.962+0000 7f748a5a6740 1 -- 172.21.15.70:0/1363932248 wait complete. 2024-04-01T01:48:02.964 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.962+0000 7f748a5a6740 20 client.29768 trim_cache size 0 max 16384 2024-04-01T01:48:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.993+0000 7f748a5a6740 1 Processor -- start 2024-04-01T01:48:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.993+0000 7f748a5a6740 1 -- start start 2024-04-01T01:48:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.993+0000 7f748a5a6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c38d0 0x559da03d1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.993+0000 7f748a5a6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c42c0 0x559da03ec6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:02.994 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.994+0000 7f748a5a6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 0x559da03ecbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.994+0000 7f748a5a6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f74801a8130 con 0x559da03c38d0 2024-04-01T01:48:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.994+0000 7f748a5a6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f74801a9c70 con 0x559da03d1c40 2024-04-01T01:48:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.994+0000 7f748a5a6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f74801a8800 con 0x559da03c42c0 2024-04-01T01:48:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.994+0000 7f7485763640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c38d0 0x559da03d1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.994+0000 7f7485f64640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 0x559da03ecbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:02.995 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.994+0000 7f7485f64640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 0x559da03ecbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:47966/0 (socket says 172.21.15.70:47966) 2024-04-01T01:48:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.994+0000 7f7485f64640 1 -- 172.21.15.70:0/838758624 learned_addr learned my addr 172.21.15.70:0/838758624 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.995+0000 7f7485f64640 1 -- 172.21.15.70:0/838758624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c42c0 msgr2=0x559da03ec6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.995+0000 7f7485f64640 1 --2- 172.21.15.70:0/838758624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c42c0 0x559da03ec6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.995+0000 7f7485f64640 1 -- 172.21.15.70:0/838758624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c38d0 msgr2=0x559da03d1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.995+0000 7f7485f64640 1 --2- 172.21.15.70:0/838758624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c38d0 0x559da03d1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:02.996 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.995+0000 7f7485f64640 1 -- 172.21.15.70:0/838758624 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559da03c3cb0 con 0x559da03d1c40 2024-04-01T01:48:02.999 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:02.997+0000 7f7485f64640 1 --2- 172.21.15.70:0/838758624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 0x559da03ecbe0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f74780058c0 tx=0x7f7478086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34d7297d606df047 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.000+0000 7f746affd640 1 -- 172.21.15.70:0/838758624 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74780b4070 con 0x559da03d1c40 2024-04-01T01:48:03.000 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.000+0000 7f746affd640 1 -- 172.21.15.70:0/838758624 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74780cca30 con 0x559da03d1c40 2024-04-01T01:48:03.001 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.000+0000 7f746affd640 1 -- 172.21.15.70:0/838758624 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747809bda0 con 0x559da03d1c40 2024-04-01T01:48:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.001+0000 7f748a5a6740 1 -- 172.21.15.70:0/838758624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 msgr2=0x559da03ecbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.001+0000 7f748a5a6740 1 --2- 172.21.15.70:0/838758624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 0x559da03ecbe0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f74780058c0 tx=0x7f7478086040 comp rx=0 tx=0).stop 2024-04-01T01:48:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.001+0000 7f748a5a6740 1 -- 172.21.15.70:0/838758624 shutdown_connections 2024-04-01T01:48:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.001+0000 7f748a5a6740 1 --2- 172.21.15.70:0/838758624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 0x559da03ecbe0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.001+0000 7f748a5a6740 1 --2- 172.21.15.70:0/838758624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c42c0 0x559da03ec6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.001+0000 7f748a5a6740 1 --2- 172.21.15.70:0/838758624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c38d0 0x559da03d1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.001+0000 7f748a5a6740 1 -- 172.21.15.70:0/838758624 >> 172.21.15.70:0/838758624 conn(0x559da03bee40 msgr2=0x559da03bf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.001+0000 7f748a5a6740 1 -- 172.21.15.70:0/838758624 shutdown_connections 2024-04-01T01:48:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.001+0000 7f748a5a6740 1 -- 172.21.15.70:0/838758624 wait complete. 2024-04-01T01:48:03.003 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.002+0000 7f748a5a6740 1 -- start start 2024-04-01T01:48:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.002+0000 7f748a5a6740 1 Processor -- start 2024-04-01T01:48:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.002+0000 7f748a5a6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c38d0 0x559da03fb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.002+0000 7f748a5a6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c42c0 0x559da03fbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.002+0000 7f748a5a6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 0x559da03fcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.002+0000 7f748a5a6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f74801a95a0 con 0x559da03c42c0 2024-04-01T01:48:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.002+0000 7f748a5a6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f74801a8ed0 con 0x559da03d1c40 2024-04-01T01:48:03.005 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.002+0000 7f748a5a6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f74801a6110 con 0x559da03c38d0 2024-04-01T01:48:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.002+0000 7f748a5a6740 10 client.0 _ll_register_callbacks cb 0x7f74780093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.003+0000 7f7485763640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c38d0 0x559da03fb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.003+0000 7f7484f62640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c42c0 0x559da03fbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.003+0000 7f7485763640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c38d0 0x559da03fb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:37216/0 (socket says 172.21.15.70:37216) 2024-04-01T01:48:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.003+0000 7f7485763640 1 -- 172.21.15.70:0/3908789802 learned_addr learned my addr 172.21.15.70:0/3908789802 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.003+0000 7f7485f64640 1 --2- 172.21.15.70:0/3908789802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 0x559da03fcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.003+0000 7f7485f64640 1 -- 172.21.15.70:0/3908789802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c38d0 msgr2=0x559da03fb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.003+0000 7f7485f64640 1 --2- 172.21.15.70:0/3908789802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c38d0 0x559da03fb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.003+0000 7f7485f64640 1 -- 172.21.15.70:0/3908789802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c42c0 msgr2=0x559da03fbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.003+0000 7f7485f64640 1 --2- 172.21.15.70:0/3908789802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c42c0 0x559da03fbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.003+0000 7f7485f64640 1 -- 172.21.15.70:0/3908789802 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559da03fd320 con 0x559da03d1c40 2024-04-01T01:48:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.003+0000 7f7485f64640 1 --2- 172.21.15.70:0/3908789802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 0x559da03fcd30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7478097d40 tx=0x7f747809b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bf7094dff2bf8e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:03.006 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.003+0000 7f7468ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.003+0000 7f7468ff9640 1 -- 172.21.15.70:0/3908789802 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74780b4040 con 0x559da03d1c40 2024-04-01T01:48:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.003+0000 7f748a5a6740 1 -- 172.21.15.70:0/3908789802 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74780cca30 con 0x559da03d1c40 2024-04-01T01:48:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.004+0000 7f7468ff9640 1 -- 172.21.15.70:0/3908789802 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f74780a4060 con 0x559da03d1c40 2024-04-01T01:48:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.004+0000 7f7468ff9640 1 -- 172.21.15.70:0/3908789802 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f74780afba0 con 0x559da03d1c40 2024-04-01T01:48:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.004+0000 7f748a5a6740 10 client.29774 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.004+0000 7f748a5a6740 1 -- 172.21.15.70:0/3908789802 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559da04508c0 con 0x559da03d1c40 2024-04-01T01:48:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.004+0000 7f7468ff9640 1 -- 172.21.15.70:0/3908789802 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74780cca30 con 0x559da03d1c40 2024-04-01T01:48:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.004+0000 7f7468ff9640 1 -- 172.21.15.70:0/3908789802 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f74780977c0 con 0x559da03d1c40 2024-04-01T01:48:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.004+0000 7f748a5a6740 10 client.29774 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.004+0000 7f748a5a6740 10 client.29774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.004+0000 7f748a5a6740 20 client.29774 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:03.007 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.004+0000 7f748a5a6740 1 --2- 172.21.15.70:0/3908789802 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559da0451140 0x559da0453560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.004+0000 7f748a5a6740 4 client.29774 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:48:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.004+0000 7f748a5a6740 1 -- 172.21.15.70:0/3908789802 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559da0334e30 con 0x559da0451140 2024-04-01T01:48:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.004+0000 7f7485763640 1 --2- 172.21.15.70:0/3908789802 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559da0451140 0x559da0453560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.005+0000 7f7485763640 1 --2- 172.21.15.70:0/3908789802 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559da0451140 0x559da0453560 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33b10c468a765833 server_cookie=640017cd7436b8b9 in_seq=0 out_seq=0 2024-04-01T01:48:03.008 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.005+0000 7f7468ff9640 10 client.29774 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.053+0000 7f7468ff9640 1 -- 172.21.15.70:0/3908789802 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559da0334e30 con 0x559da0451140 2024-04-01T01:48:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.053+0000 7f7468ff9640 10 client.29774 handle_command_reply: tid=0 2024-04-01T01:48:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.053+0000 7f748a5a6740 1 client.29774 shutdown 2024-04-01T01:48:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.053+0000 7f748a5a6740 1 -- 172.21.15.70:0/3908789802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 msgr2=0x559da03fcd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.053+0000 7f748a5a6740 1 --2- 172.21.15.70:0/3908789802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 0x559da03fcd30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7478097d40 tx=0x7f747809b7e0 comp rx=0 tx=0).stop 2024-04-01T01:48:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.053+0000 7f748a5a6740 1 -- 172.21.15.70:0/3908789802 shutdown_connections 2024-04-01T01:48:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.053+0000 7f748a5a6740 1 --2- 172.21.15.70:0/3908789802 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559da0451140 0x559da0453560 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f748a5a6740 1 --2- 172.21.15.70:0/3908789802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559da03d1c40 0x559da03fcd30 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f7468ff9640 0 client.29774 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f748a5a6740 1 --2- 172.21.15.70:0/3908789802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559da03c42c0 0x559da03fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f748a5a6740 1 --2- 172.21.15.70:0/3908789802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559da03c38d0 0x559da03fb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f748a5a6740 1 -- 172.21.15.70:0/3908789802 >> 172.21.15.70:0/3908789802 conn(0x559da03bee40 msgr2=0x559da03ec6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f748a5a6740 1 -- 172.21.15.70:0/3908789802 shutdown_connections 2024-04-01T01:48:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f748a5a6740 1 -- 172.21.15.70:0/3908789802 wait complete. 2024-04-01T01:48:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f748a5a6740 20 client.29774 trim_cache size 0 max 16384 2024-04-01T01:48:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:48:03.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:48:03.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:48:03.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:48:03.057 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f74879c7640 1 -- 172.21.15.70:0/996821070 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f746c0a2640 msgr2=0x7f746c0a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f74879c7640 1 --2- 172.21.15.70:0/996821070 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f746c0a2640 0x7f746c0a4b00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f748010a8d0 tx=0x7f74700094b0 comp rx=0 tx=0).stop 2024-04-01T01:48:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f74879c7640 1 -- 172.21.15.70:0/996821070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7480102260 msgr2=0x7f7480109730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f74879c7640 1 --2- 172.21.15.70:0/996821070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7480102260 0x7f7480109730 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f7478009fd0 tx=0x7f747800ef10 comp rx=0 tx=0).stop 2024-04-01T01:48:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f74879c7640 1 -- 172.21.15.70:0/996821070 shutdown_connections 2024-04-01T01:48:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f74879c7640 1 --2- 172.21.15.70:0/996821070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7480102260 0x7f7480109730 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f74879c7640 1 --2- 172.21.15.70:0/996821070 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f746c0a2640 0x7f746c0a4b00 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f74879c7640 1 --2- 172.21.15.70:0/996821070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74801018a0 0x7f74801091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.054+0000 7f74879c7640 1 --2- 172.21.15.70:0/996821070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74801006b0 0x7f748010ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.055+0000 7f74879c7640 1 -- 172.21.15.70:0/996821070 >> 172.21.15.70:0/996821070 conn(0x7f74800fbc20 msgr2=0x7f7480104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.055+0000 7f74879c7640 1 -- 172.21.15.70:0/996821070 shutdown_connections 2024-04-01T01:48:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:03.055+0000 7f74879c7640 1 -- 172.21.15.70:0/996821070 wait complete. 2024-04-01T01:48:04.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:04 smithi081 ceph-mon[35142]: pgmap v796: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 6.7 MiB/s wr, 839 op/s 2024-04-01T01:48:04.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:04 smithi189 ceph-mon[35753]: pgmap v796: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 6.7 MiB/s wr, 839 op/s 2024-04-01T01:48:04.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:04 smithi070 ceph-mon[32179]: pgmap v796: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 6.7 MiB/s wr, 839 op/s 2024-04-01T01:48:06.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:06 smithi081 ceph-mon[35142]: pgmap v797: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 5.8 MiB/s wr, 686 op/s 2024-04-01T01:48:06.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:06 smithi189 ceph-mon[35753]: pgmap v797: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 5.8 MiB/s wr, 686 op/s 2024-04-01T01:48:06.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:06 smithi070 ceph-mon[32179]: pgmap v797: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 5.8 MiB/s wr, 686 op/s 2024-04-01T01:48:06.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 800.0 s, 508.6 tps, lat 198.333 ms stddev 269.773 2024-04-01T01:48:08.299 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:08 smithi189 ceph-mon[35753]: pgmap v798: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 6.8 MiB/s wr, 772 op/s 2024-04-01T01:48:08.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:08 smithi081 ceph-mon[35142]: pgmap v798: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 6.8 MiB/s wr, 772 op/s 2024-04-01T01:48:08.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:08 smithi070 ceph-mon[32179]: pgmap v798: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 6.8 MiB/s wr, 772 op/s 2024-04-01T01:48:10.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:10 smithi081 ceph-mon[35142]: pgmap v799: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.2 MiB/s wr, 688 op/s 2024-04-01T01:48:10.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:10 smithi189 ceph-mon[35753]: pgmap v799: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.2 MiB/s wr, 688 op/s 2024-04-01T01:48:10.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:10 smithi070 ceph-mon[32179]: pgmap v799: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 6.2 MiB/s wr, 688 op/s 2024-04-01T01:48:11.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 805.0 s, 228.8 tps, lat 271.250 ms stddev 317.877 2024-04-01T01:48:12.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:12 smithi081 ceph-mon[35142]: pgmap v800: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.7 MiB/s wr, 788 op/s 2024-04-01T01:48:12.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:12 smithi189 ceph-mon[35753]: pgmap v800: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.7 MiB/s wr, 788 op/s 2024-04-01T01:48:12.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:12 smithi070 ceph-mon[32179]: pgmap v800: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.7 MiB/s wr, 788 op/s 2024-04-01T01:48:14.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:14 smithi081 ceph-mon[35142]: pgmap v801: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.3 MiB/s wr, 891 op/s 2024-04-01T01:48:14.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:14 smithi189 ceph-mon[35753]: pgmap v801: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.3 MiB/s wr, 891 op/s 2024-04-01T01:48:14.503 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:14 smithi070 ceph-mon[32179]: pgmap v801: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 9.3 MiB/s wr, 891 op/s 2024-04-01T01:48:16.206 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.206+0000 7fd1fe272640 1 -- 172.21.15.70:0/2860020391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1f8107be0 msgr2=0x7fd1f8108000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.206+0000 7fd1fe272640 1 --2- 172.21.15.70:0/2860020391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1f8107be0 0x7fd1f8108000 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4009900 tx=0x7fd1f402bc60 comp rx=0 tx=0).stop 2024-04-01T01:48:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.206+0000 7fd1fe272640 1 -- 172.21.15.70:0/2860020391 shutdown_connections 2024-04-01T01:48:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.206+0000 7fd1fe272640 1 --2- 172.21.15.70:0/2860020391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1f8109790 0x7fd1f8110020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.206+0000 7fd1fe272640 1 --2- 172.21.15.70:0/2860020391 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1f8108dd0 0x7fd1f8109250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.206+0000 7fd1fe272640 1 --2- 172.21.15.70:0/2860020391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1f8107be0 0x7fd1f8108000 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.206+0000 7fd1fe272640 1 -- 172.21.15.70:0/2860020391 >> 172.21.15.70:0/2860020391 conn(0x7fd1f8076d70 msgr2=0x7fd1f80791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:16.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.210+0000 7fd1fe272640 1 -- 172.21.15.70:0/2860020391 shutdown_connections 2024-04-01T01:48:16.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.212+0000 7fd1fe272640 1 -- 172.21.15.70:0/2860020391 wait complete. 2024-04-01T01:48:16.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.213+0000 7fd1fe272640 1 Processor -- start 2024-04-01T01:48:16.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.213+0000 7fd1fe272640 1 -- start start 2024-04-01T01:48:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.213+0000 7fd1fe272640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1f8107be0 0x7fd1f8081a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.213+0000 7fd1fe272640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1f8108dd0 0x7fd1f8081f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.213+0000 7fd1fe272640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1f8109790 0x7fd1f8080050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.213+0000 7fd1fe272640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f807a8a0 con 0x7fd1f8108dd0 2024-04-01T01:48:16.214 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.213+0000 7fd1fe272640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f807a720 con 0x7fd1f8109790 2024-04-01T01:48:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.213+0000 7fd1fe272640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f807aa20 con 0x7fd1f8107be0 2024-04-01T01:48:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.214+0000 7fd1fda71640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1f8109790 0x7fd1f8080050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.214+0000 7fd1fd270640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1f8107be0 0x7fd1f8081a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.214+0000 7fd1fd270640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1f8107be0 0x7fd1f8081a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41700/0 (socket says 172.21.15.70:41700) 2024-04-01T01:48:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.214+0000 7fd1fd270640 1 -- 172.21.15.70:0/1630163308 learned_addr learned my addr 172.21.15.70:0/1630163308 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.214+0000 7fd1fda71640 1 -- 172.21.15.70:0/1630163308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1f8107be0 msgr2=0x7fd1f8081a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.214+0000 7fd1fda71640 1 --2- 172.21.15.70:0/1630163308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1f8107be0 0x7fd1f8081a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.214+0000 7fd1fda71640 1 -- 172.21.15.70:0/1630163308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1f8108dd0 msgr2=0x7fd1f8081f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.214+0000 7fd1fda71640 1 --2- 172.21.15.70:0/1630163308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1f8108dd0 0x7fd1f8081f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.214+0000 7fd1fda71640 1 -- 172.21.15.70:0/1630163308 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f8080780 con 0x7fd1f8109790 2024-04-01T01:48:16.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.214+0000 7fd1fd270640 1 --2- 172.21.15.70:0/1630163308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1f8107be0 0x7fd1f8081a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.215+0000 7fd1fda71640 1 --2- 172.21.15.70:0/1630163308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1f8109790 0x7fd1f8080050 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fd1f000be20 tx=0x7fd1f000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.216+0000 7fd1ee7fc640 1 -- 172.21.15.70:0/1630163308 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f0019070 con 0x7fd1f8109790 2024-04-01T01:48:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.216+0000 7fd1ee7fc640 1 -- 172.21.15.70:0/1630163308 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1f00148f0 con 0x7fd1f8109790 2024-04-01T01:48:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.216+0000 7fd1ee7fc640 1 -- 172.21.15.70:0/1630163308 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f0012860 con 0x7fd1f8109790 2024-04-01T01:48:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.216+0000 7fd1fe272640 1 -- 172.21.15.70:0/1630163308 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1f8080a70 con 0x7fd1f8109790 2024-04-01T01:48:16.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.217+0000 7fd1fe272640 1 -- 172.21.15.70:0/1630163308 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1f81aec20 con 0x7fd1f8109790 2024-04-01T01:48:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.222+0000 7fd20218e740 1 Processor -- start 2024-04-01T01:48:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.222+0000 7fd20218e740 1 -- start start 2024-04-01T01:48:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.222+0000 7fd20218e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c5cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.222+0000 7fd20218e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c61fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.222+0000 7fd20218e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c62640 0x55f7e4c64ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.222+0000 7fd20218e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f7e4bc08f0 con 0x55f7e4c5d4d0 2024-04-01T01:48:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.222+0000 7fd20218e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f7e4b698d0 con 0x55f7e4c62640 2024-04-01T01:48:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.222+0000 7fd20218e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f7e4bbc990 con 0x55f7e4c5cae0 2024-04-01T01:48:16.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.223+0000 7fd1fd270640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c61fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:16.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.224+0000 7fd1fd270640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c61fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53500/0 (socket says 172.21.15.70:53500) 2024-04-01T01:48:16.223 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.224+0000 7fd1fd270640 1 -- 172.21.15.70:0/2525628594 learned_addr learned my addr 172.21.15.70:0/2525628594 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:16.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.225+0000 7fd1ee7fc640 1 -- 172.21.15.70:0/1630163308 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fd1f001c070 con 0x7fd1f8109790 2024-04-01T01:48:16.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.225+0000 7fd1fca6f640 1 --2- 172.21.15.70:0/2525628594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c62640 0x55f7e4c64ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:16.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.225+0000 7fd1fca6f640 1 --2- 172.21.15.70:0/2525628594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c5cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:16.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.225+0000 7fd1fd270640 1 -- 172.21.15.70:0/2525628594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5cae0 msgr2=0x55f7e4c5cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:16.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.225+0000 7fd1fd270640 1 --2- 172.21.15.70:0/2525628594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c5cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.225+0000 7fd1fd270640 1 -- 172.21.15.70:0/2525628594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c62640 msgr2=0x55f7e4c64ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:16.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.225+0000 7fd1fd270640 1 --2- 172.21.15.70:0/2525628594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c62640 0x55f7e4c64ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.225+0000 7fd1fd270640 1 -- 172.21.15.70:0/2525628594 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7e48d5210 con 0x55f7e4c5d4d0 2024-04-01T01:48:16.225 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.225+0000 7fd1fca6f640 1 --2- 172.21.15.70:0/2525628594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c5cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:48:16.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.226+0000 7fd1fd270640 1 --2- 172.21.15.70:0/2525628594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c61fd0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4036040 tx=0x7fd1f402ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f38bb6bae673ff server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:16.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.226+0000 7fd1ee7fc640 1 --2- 172.21.15.70:0/1630163308 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1e40a26a0 0x7fd1e40a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.226 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.227+0000 7fd1ee7fc640 1 -- 172.21.15.70:0/1630163308 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd1f00ce360 con 0x7fd1f8109790 2024-04-01T01:48:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.227+0000 7fd1e37fe640 1 -- 172.21.15.70:0/2525628594 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4043070 con 0x55f7e4c5d4d0 2024-04-01T01:48:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.227+0000 7fd1e37fe640 1 -- 172.21.15.70:0/2525628594 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1f4005050 con 0x55f7e4c5d4d0 2024-04-01T01:48:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.227+0000 7fd1e37fe640 1 -- 172.21.15.70:0/2525628594 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4004c60 con 0x55f7e4c5d4d0 2024-04-01T01:48:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.228+0000 7fd1fca6f640 1 --2- 172.21.15.70:0/1630163308 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1e40a26a0 0x7fd1e40a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.230+0000 7fd1fca6f640 1 --2- 172.21.15.70:0/1630163308 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1e40a26a0 0x7fd1e40a4b60 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x55f7e4c5dd30 tx=0x7fd1e8005bf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.231+0000 7fd20218e740 1 -- 172.21.15.70:0/2525628594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c5d4d0 msgr2=0x55f7e4c61fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.231+0000 7fd20218e740 1 --2- 172.21.15.70:0/2525628594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c61fd0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4036040 tx=0x7fd1f402ce00 comp rx=0 tx=0).stop 2024-04-01T01:48:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.232+0000 7fd20218e740 1 -- 172.21.15.70:0/2525628594 shutdown_connections 2024-04-01T01:48:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.232+0000 7fd20218e740 1 --2- 172.21.15.70:0/2525628594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c62640 0x55f7e4c64ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.232+0000 7fd20218e740 1 --2- 172.21.15.70:0/2525628594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c61fd0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.232+0000 7fd20218e740 1 --2- 172.21.15.70:0/2525628594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c5cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.232+0000 7fd20218e740 1 -- 172.21.15.70:0/2525628594 >> 172.21.15.70:0/2525628594 conn(0x55f7e4c58050 msgr2=0x55f7e4c58450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.232+0000 7fd20218e740 1 -- 172.21.15.70:0/2525628594 shutdown_connections 2024-04-01T01:48:16.232 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.232+0000 7fd20218e740 1 -- 172.21.15.70:0/2525628594 wait complete. 2024-04-01T01:48:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.232+0000 7fd20218e740 1 -- start start 2024-04-01T01:48:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.232+0000 7fd20218e740 1 Processor -- start 2024-04-01T01:48:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.233+0000 7fd20218e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c77d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.233+0000 7fd20218e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c78270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.233+0000 7fd20218e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c62640 0x55f7e4c7f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.233+0000 7fd20218e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f7e4c765a0 con 0x55f7e4c62640 2024-04-01T01:48:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.233+0000 7fd20218e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f7e4b5ef80 con 0x55f7e4c5cae0 2024-04-01T01:48:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.233+0000 7fd20218e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f7e4c76720 con 0x55f7e4c5d4d0 2024-04-01T01:48:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.233+0000 7fd20218e740 10 client.0 _ll_register_callbacks cb 0x55f7e4bb7290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.234+0000 7fd1fd270640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c77d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.234+0000 7fd1fca6f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c78270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:16.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.234+0000 7fd1fd270640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c77d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37806/0 (socket says 172.21.15.70:37806) 2024-04-01T01:48:16.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.234+0000 7fd1fd270640 1 -- 172.21.15.70:0/2582779064 learned_addr learned my addr 172.21.15.70:0/2582779064 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:16.234 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.234+0000 7fd1fda71640 1 --2- 172.21.15.70:0/2582779064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c62640 0x55f7e4c7f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.237+0000 7fd1fca6f640 1 -- 172.21.15.70:0/2582779064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c5cae0 msgr2=0x55f7e4c77d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.237+0000 7fd1fca6f640 1 --2- 172.21.15.70:0/2582779064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c77d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.237+0000 7fd1fca6f640 1 -- 172.21.15.70:0/2582779064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c62640 msgr2=0x55f7e4c7f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.237+0000 7fd1fca6f640 1 --2- 172.21.15.70:0/2582779064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c62640 0x55f7e4c7f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.237+0000 7fd1fca6f640 1 -- 172.21.15.70:0/2582779064 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7e49a1620 con 0x55f7e4c5d4d0 2024-04-01T01:48:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.238+0000 7fd1fca6f640 1 --2- 172.21.15.70:0/2582779064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c78270 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8081780 tx=0x7fd1e8005ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8509bcb0f061722c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.239+0000 7fd1e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:48:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.239+0000 7fd1e0ff9640 1 -- 172.21.15.70:0/2582779064 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e8021070 con 0x55f7e4c5d4d0 2024-04-01T01:48:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.240+0000 7fd1e0ff9640 1 -- 172.21.15.70:0/2582779064 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd1e800f690 con 0x55f7e4c5d4d0 2024-04-01T01:48:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.240+0000 7fd20218e740 1 -- 172.21.15.70:0/2582779064 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1f4005050 con 0x55f7e4c5d4d0 2024-04-01T01:48:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.242+0000 7fd1e0ff9640 1 -- 172.21.15.70:0/2582779064 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd1e800f890 con 0x55f7e4c5d4d0 2024-04-01T01:48:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.243+0000 7fd20218e740 10 client.29524 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.243+0000 7fd20218e740 1 -- 172.21.15.70:0/2582779064 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f7e4c82320 con 0x55f7e4c5d4d0 2024-04-01T01:48:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.244+0000 7fd1e0ff9640 1 -- 172.21.15.70:0/2582779064 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1f4005050 con 0x55f7e4c5d4d0 2024-04-01T01:48:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.245+0000 7fd1e0ff9640 1 -- 172.21.15.70:0/2582779064 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd1e802e020 con 0x55f7e4c5d4d0 2024-04-01T01:48:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.245+0000 7fd20218e740 10 client.29524 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.245+0000 7fd20218e740 10 client.29524 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.245+0000 7fd20218e740 20 client.29524 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.245+0000 7fd20218e740 1 --2- 172.21.15.70:0/2582779064 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f7e4c83e50 0x55f7e4c86310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.245+0000 7fd20218e740 4 client.29524 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.245+0000 7fd20218e740 1 -- 172.21.15.70:0/2582779064 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7e4bd4120 con 0x55f7e4c83e50 2024-04-01T01:48:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.247+0000 7fd1fd270640 1 --2- 172.21.15.70:0/2582779064 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f7e4c83e50 0x55f7e4c86310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:16.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.252+0000 7fd1fd270640 1 --2- 172.21.15.70:0/2582779064 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f7e4c83e50 0x55f7e4c86310 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2faf044357c67a39 server_cookie=a553805304d047c4 in_seq=0 out_seq=0 2024-04-01T01:48:16.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.253+0000 7fd1e0ff9640 10 client.29524 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:16.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.255+0000 7fd1e0ff9640 1 -- 172.21.15.70:0/2582779064 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f7e4bd4120 con 0x55f7e4c83e50 2024-04-01T01:48:16.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.255+0000 7fd1e0ff9640 10 client.29524 handle_command_reply: tid=0 2024-04-01T01:48:16.256 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.255+0000 7fd20218e740 1 client.29524 shutdown 2024-04-01T01:48:16.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.258+0000 7fd20218e740 1 -- 172.21.15.70:0/2582779064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5d4d0 msgr2=0x55f7e4c78270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:16.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.258+0000 7fd20218e740 1 --2- 172.21.15.70:0/2582779064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c78270 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8081780 tx=0x7fd1e8005ac0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.258+0000 7fd20218e740 1 -- 172.21.15.70:0/2582779064 shutdown_connections 2024-04-01T01:48:16.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.258+0000 7fd20218e740 1 --2- 172.21.15.70:0/2582779064 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f7e4c83e50 0x55f7e4c86310 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.258+0000 7fd20218e740 1 --2- 172.21.15.70:0/2582779064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c62640 0x55f7e4c7f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.258+0000 7fd20218e740 1 --2- 172.21.15.70:0/2582779064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c78270 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.258+0000 7fd20218e740 1 --2- 172.21.15.70:0/2582779064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c77d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.258+0000 7fd20218e740 1 -- 172.21.15.70:0/2582779064 >> 172.21.15.70:0/2582779064 conn(0x55f7e4c58050 msgr2=0x55f7e4c57190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.259+0000 7fd1e0ff9640 0 client.29524 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.262+0000 7fd20218e740 1 -- 172.21.15.70:0/2582779064 shutdown_connections 2024-04-01T01:48:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.262+0000 7fd20218e740 1 -- 172.21.15.70:0/2582779064 wait complete. 2024-04-01T01:48:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.263+0000 7fd20218e740 20 client.29524 trim_cache size 0 max 16384 2024-04-01T01:48:16.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.293+0000 7fd20218e740 1 Processor -- start 2024-04-01T01:48:16.292 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.293+0000 7fd20218e740 1 -- start start 2024-04-01T01:48:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.293+0000 7fd20218e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c61d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.294+0000 7fd20218e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c7f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.294+0000 7fd1fd270640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c61d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.294+0000 7fd1fd270640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c61d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:37818/0 (socket says 172.21.15.70:37818) 2024-04-01T01:48:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.294+0000 7fd20218e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c62640 0x55f7e4c5e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.294+0000 7fd1fca6f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c7f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.294+0000 7fd1fca6f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c7f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:41722/0 (socket says 172.21.15.70:41722) 2024-04-01T01:48:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.294+0000 7fd1fd270640 1 -- 172.21.15.70:0/3222616624 learned_addr learned my addr 172.21.15.70:0/3222616624 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.294+0000 7fd20218e740 1 -- 172.21.15.70:0/3222616624 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f81b08e0 con 0x55f7e4c62640 2024-04-01T01:48:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.294+0000 7fd20218e740 1 -- 172.21.15.70:0/3222616624 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f81b2420 con 0x55f7e4c5cae0 2024-04-01T01:48:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.294+0000 7fd20218e740 1 -- 172.21.15.70:0/3222616624 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f81b0fb0 con 0x55f7e4c5d4d0 2024-04-01T01:48:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.294+0000 7fd1fda71640 1 --2- 172.21.15.70:0/3222616624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c62640 0x55f7e4c5e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.296+0000 7fd1fda71640 1 -- 172.21.15.70:0/3222616624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5d4d0 msgr2=0x55f7e4c7f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.296+0000 7fd1fda71640 1 --2- 172.21.15.70:0/3222616624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c7f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.296+0000 7fd1fda71640 1 -- 172.21.15.70:0/3222616624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c5cae0 msgr2=0x55f7e4c61d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.296+0000 7fd1fda71640 1 --2- 172.21.15.70:0/3222616624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c61d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.296+0000 7fd1fda71640 1 -- 172.21.15.70:0/3222616624 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7e4c5d8b0 con 0x55f7e4c62640 2024-04-01T01:48:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.296+0000 7fd1fda71640 1 --2- 172.21.15.70:0/3222616624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c62640 0x55f7e4c5e7b0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fd1f009aa30 tx=0x7fd1f000e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22299f3eac85a9e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.297+0000 7fd1e2ffd640 1 -- 172.21.15.70:0/3222616624 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f00b2020 con 0x55f7e4c62640 2024-04-01T01:48:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.297+0000 7fd1e2ffd640 1 -- 172.21.15.70:0/3222616624 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1f00a3400 con 0x55f7e4c62640 2024-04-01T01:48:16.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.297+0000 7fd1e2ffd640 1 -- 172.21.15.70:0/3222616624 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f00a2f70 con 0x55f7e4c62640 2024-04-01T01:48:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.299+0000 7fd20218e740 1 -- 172.21.15.70:0/3222616624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c62640 msgr2=0x55f7e4c5e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.299+0000 7fd20218e740 1 --2- 172.21.15.70:0/3222616624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c62640 0x55f7e4c5e7b0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fd1f009aa30 tx=0x7fd1f000e040 comp rx=0 tx=0).stop 2024-04-01T01:48:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.300+0000 7fd20218e740 1 -- 172.21.15.70:0/3222616624 shutdown_connections 2024-04-01T01:48:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.300+0000 7fd20218e740 1 --2- 172.21.15.70:0/3222616624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c62640 0x55f7e4c5e7b0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.300+0000 7fd20218e740 1 --2- 172.21.15.70:0/3222616624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c7f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.300+0000 7fd20218e740 1 --2- 172.21.15.70:0/3222616624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c61d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.300+0000 7fd20218e740 1 -- 172.21.15.70:0/3222616624 >> 172.21.15.70:0/3222616624 conn(0x55f7e4c58050 msgr2=0x55f7e4c58430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.302+0000 7fd20218e740 1 -- 172.21.15.70:0/3222616624 shutdown_connections 2024-04-01T01:48:16.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.302+0000 7fd20218e740 1 -- 172.21.15.70:0/3222616624 wait complete. 2024-04-01T01:48:16.302 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.303+0000 7fd20218e740 1 -- start start 2024-04-01T01:48:16.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.305+0000 7fd20218e740 1 Processor -- start 2024-04-01T01:48:16.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.307+0000 7fd20218e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c7f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.307+0000 7fd20218e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c7bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.308+0000 7fd1fd270640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c7f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:16.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.308+0000 7fd1fd270640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c7f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53512/0 (socket says 172.21.15.70:53512) 2024-04-01T01:48:16.307 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.308+0000 7fd1fd270640 1 -- 172.21.15.70:0/4101942313 learned_addr learned my addr 172.21.15.70:0/4101942313 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:16.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.308+0000 7fd20218e740 1 --2- 172.21.15.70:0/4101942313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c62640 0x55f7e4c7c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.308+0000 7fd20218e740 1 -- 172.21.15.70:0/4101942313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f81b1d50 con 0x55f7e4c5cae0 2024-04-01T01:48:16.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.309+0000 7fd20218e740 1 -- 172.21.15.70:0/4101942313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f81b1680 con 0x55f7e4c62640 2024-04-01T01:48:16.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.309+0000 7fd1fda71640 1 --2- 172.21.15.70:0/4101942313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c62640 0x55f7e4c7c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:16.308 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.309+0000 7fd20218e740 1 -- 172.21.15.70:0/4101942313 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f8073390 con 0x55f7e4c5d4d0 2024-04-01T01:48:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.309+0000 7fd20218e740 10 client.0 _ll_register_callbacks cb 0x7fd1e800ef50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.309+0000 7fd1fd270640 1 -- 172.21.15.70:0/4101942313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5d4d0 msgr2=0x55f7e4c7bcf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:48:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.309+0000 7fd1fd270640 1 --2- 172.21.15.70:0/4101942313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c7bcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.309+0000 7fd1fd270640 1 -- 172.21.15.70:0/4101942313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c62640 msgr2=0x55f7e4c7c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.309+0000 7fd1fd270640 1 --2- 172.21.15.70:0/4101942313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c62640 0x55f7e4c7c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.309+0000 7fd1fd270640 1 -- 172.21.15.70:0/4101942313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7e4ce82f0 con 0x55f7e4c5cae0 2024-04-01T01:48:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.310+0000 7fd1fd270640 1 --2- 172.21.15.70:0/4101942313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c7f200 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fd1f402c870 tx=0x7fd1f4009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0060227350d43e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:16.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.314+0000 7fd1e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:48:16.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.314+0000 7fd1e0ff9640 1 -- 172.21.15.70:0/4101942313 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f405a070 con 0x55f7e4c5cae0 2024-04-01T01:48:16.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.315+0000 7fd20218e740 1 -- 172.21.15.70:0/4101942313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1f00a3400 con 0x55f7e4c5cae0 2024-04-01T01:48:16.314 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.315+0000 7fd1e0ff9640 1 -- 172.21.15.70:0/4101942313 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fd1f4005280 con 0x55f7e4c5cae0 2024-04-01T01:48:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.316+0000 7fd1e0ff9640 1 -- 172.21.15.70:0/4101942313 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fd1f4066050 con 0x55f7e4c5cae0 2024-04-01T01:48:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.317+0000 7fd20218e740 10 client.19698 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:16.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.317+0000 7fd20218e740 1 -- 172.21.15.70:0/4101942313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f7e4ce9580 con 0x55f7e4c5cae0 2024-04-01T01:48:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.317+0000 7fd1e0ff9640 1 -- 172.21.15.70:0/4101942313 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1f00a3400 con 0x55f7e4c5cae0 2024-04-01T01:48:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.317+0000 7fd1e0ff9640 1 -- 172.21.15.70:0/4101942313 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fd1f4052040 con 0x55f7e4c5cae0 2024-04-01T01:48:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.318+0000 7fd20218e740 10 client.19698 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.318+0000 7fd20218e740 10 client.19698 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.318+0000 7fd20218e740 20 client.19698 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.318+0000 7fd20218e740 1 --2- 172.21.15.70:0/4101942313 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f7e4ce9f00 0x55f7e4cec320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.318+0000 7fd20218e740 4 client.19698 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:48:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.318+0000 7fd20218e740 1 -- 172.21.15.70:0/4101942313 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f7e4bcde30 con 0x55f7e4ce9f00 2024-04-01T01:48:16.320 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.321+0000 7fd1fca6f640 1 --2- 172.21.15.70:0/4101942313 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f7e4ce9f00 0x55f7e4cec320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:16.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:16 smithi081 ceph-mon[35142]: pgmap v802: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 9.3 MiB/s wr, 847 op/s 2024-04-01T01:48:16.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.339+0000 7fd1fca6f640 1 --2- 172.21.15.70:0/4101942313 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f7e4ce9f00 0x55f7e4cec320 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81a96320a8b29970 server_cookie=991f1a9e2d9f35ed in_seq=0 out_seq=0 2024-04-01T01:48:16.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:16.339+0000 7fd1e0ff9640 10 client.19698 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:16.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:16 smithi189 ceph-mon[35753]: pgmap v802: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 9.3 MiB/s wr, 847 op/s 2024-04-01T01:48:16.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:16 smithi070 ceph-mon[32179]: pgmap v802: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 9.3 MiB/s wr, 847 op/s 2024-04-01T01:48:16.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 810.0 s, 155.8 tps, lat 862.319 ms stddev 1422.801 2024-04-01T01:48:18.305 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:18 smithi070 ceph-mon[32179]: pgmap v803: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 968 op/s 2024-04-01T01:48:18.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:18 smithi081 ceph-mon[35142]: pgmap v803: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 968 op/s 2024-04-01T01:48:18.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:18 smithi189 ceph-mon[35753]: pgmap v803: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 968 op/s 2024-04-01T01:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.499+0000 7fd1e0ff9640 1 -- 172.21.15.70:0/4101942313 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f7e4bcde30 con 0x55f7e4ce9f00 2024-04-01T01:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.499+0000 7fd1e0ff9640 10 client.19698 handle_command_reply: tid=0 2024-04-01T01:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.499+0000 7fd20218e740 1 client.19698 shutdown 2024-04-01T01:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.499+0000 7fd20218e740 1 -- 172.21.15.70:0/4101942313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c5cae0 msgr2=0x55f7e4c7f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.499+0000 7fd20218e740 1 --2- 172.21.15.70:0/4101942313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c7f200 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fd1f402c870 tx=0x7fd1f4009880 comp rx=0 tx=0).stop 2024-04-01T01:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.499+0000 7fd20218e740 1 -- 172.21.15.70:0/4101942313 shutdown_connections 2024-04-01T01:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.499+0000 7fd20218e740 1 --2- 172.21.15.70:0/4101942313 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f7e4ce9f00 0x55f7e4cec320 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.499+0000 7fd20218e740 1 --2- 172.21.15.70:0/4101942313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f7e4c62640 0x55f7e4c7c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.499+0000 7fd20218e740 1 --2- 172.21.15.70:0/4101942313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f7e4c5d4d0 0x55f7e4c7bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.499+0000 7fd20218e740 1 --2- 172.21.15.70:0/4101942313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f7e4c5cae0 0x55f7e4c7f200 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.499+0000 7fd20218e740 1 -- 172.21.15.70:0/4101942313 >> 172.21.15.70:0/4101942313 conn(0x55f7e4c58050 msgr2=0x55f7e4c61c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.499+0000 7fd1e0ff9640 0 client.19698 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.500+0000 7fd20218e740 1 -- 172.21.15.70:0/4101942313 shutdown_connections 2024-04-01T01:48:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.500+0000 7fd20218e740 1 -- 172.21.15.70:0/4101942313 wait complete. 2024-04-01T01:48:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.500+0000 7fd20218e740 20 client.19698 trim_cache size 0 max 16384 2024-04-01T01:48:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:48:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:48:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:48:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:48:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.501+0000 7fd1fe272640 1 -- 172.21.15.70:0/1630163308 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1e40a26a0 msgr2=0x7fd1e40a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.501+0000 7fd1fe272640 1 --2- 172.21.15.70:0/1630163308 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1e40a26a0 0x7fd1e40a4b60 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x55f7e4c5dd30 tx=0x7fd1e8005bf0 comp rx=0 tx=0).stop 2024-04-01T01:48:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.501+0000 7fd1fe272640 1 -- 172.21.15.70:0/1630163308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1f8109790 msgr2=0x7fd1f8080050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.501+0000 7fd1fe272640 1 --2- 172.21.15.70:0/1630163308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1f8109790 0x7fd1f8080050 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fd1f000be20 tx=0x7fd1f000bf10 comp rx=0 tx=0).stop 2024-04-01T01:48:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.501+0000 7fd1fe272640 1 -- 172.21.15.70:0/1630163308 shutdown_connections 2024-04-01T01:48:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.501+0000 7fd1fe272640 1 --2- 172.21.15.70:0/1630163308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1f8109790 0x7fd1f8080050 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.501+0000 7fd1fe272640 1 --2- 172.21.15.70:0/1630163308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1f8108dd0 0x7fd1f8081f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.501+0000 7fd1fe272640 1 --2- 172.21.15.70:0/1630163308 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1e40a26a0 0x7fd1e40a4b60 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.501+0000 7fd1fe272640 1 --2- 172.21.15.70:0/1630163308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1f8107be0 0x7fd1f8081a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.501+0000 7fd1fe272640 1 -- 172.21.15.70:0/1630163308 >> 172.21.15.70:0/1630163308 conn(0x7fd1f8076d70 msgr2=0x7fd1f810c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.501+0000 7fd1fe272640 1 -- 172.21.15.70:0/1630163308 shutdown_connections 2024-04-01T01:48:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:18.501+0000 7fd1fe272640 1 -- 172.21.15.70:0/1630163308 wait complete. 2024-04-01T01:48:20.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:20 smithi081 ceph-mon[35142]: pgmap v804: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 12 MiB/s wr, 929 op/s 2024-04-01T01:48:20.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:20 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:48:20.498 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:20 smithi189 ceph-mon[35753]: pgmap v804: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 12 MiB/s wr, 929 op/s 2024-04-01T01:48:20.499 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:20 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:48:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:20 smithi070 ceph-mon[32179]: pgmap v804: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 12 MiB/s wr, 929 op/s 2024-04-01T01:48:20.504 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:20 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:48:21.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:48:21.330 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:48:21.330 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:48:21.348 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:48:21.349 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:48:21.376 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:48:21.376 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:48:21.376 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:48:21.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.481+0000 7fbdf2fb5640 1 -- 172.21.15.70:0/3144844189 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdec102930 msgr2=0x7fbdec104da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:21.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.481+0000 7fbdf2fb5640 1 --2- 172.21.15.70:0/3144844189 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdec102930 0x7fbdec104da0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009860 tx=0x7fbde002bb70 comp rx=0 tx=0).stop 2024-04-01T01:48:21.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.482+0000 7fbdf2fb5640 1 -- 172.21.15.70:0/3144844189 shutdown_connections 2024-04-01T01:48:21.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.482+0000 7fbdf2fb5640 1 --2- 172.21.15.70:0/3144844189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbdec1052e0 0x7fbdec105760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.482+0000 7fbdf2fb5640 1 --2- 172.21.15.70:0/3144844189 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdec102930 0x7fbdec104da0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.482+0000 7fbdf2fb5640 1 --2- 172.21.15.70:0/3144844189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdec0fffc0 0x7fbdec1023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.482+0000 7fbdf2fb5640 1 -- 172.21.15.70:0/3144844189 >> 172.21.15.70:0/3144844189 conn(0x7fbdec0fbbe0 msgr2=0x7fbdec0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:21.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.482+0000 7fbdf2fb5640 1 -- 172.21.15.70:0/3144844189 shutdown_connections 2024-04-01T01:48:21.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.482+0000 7fbdf2fb5640 1 -- 172.21.15.70:0/3144844189 wait complete. 2024-04-01T01:48:21.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.483+0000 7fbdf2fb5640 1 Processor -- start 2024-04-01T01:48:21.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.483+0000 7fbdf2fb5640 1 -- start start 2024-04-01T01:48:21.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.484+0000 7fbdf2fb5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbdec0fffc0 0x7fbdec19df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.484+0000 7fbdf2fb5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdec102930 0x7fbdec19e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.484+0000 7fbdf2fb5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdec1052e0 0x7fbdec1a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.484+0000 7fbdf2fb5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec113850 con 0x7fbdec0fffc0 2024-04-01T01:48:21.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.484+0000 7fbdf2fb5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec1136d0 con 0x7fbdec102930 2024-04-01T01:48:21.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.484+0000 7fbdf2fb5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec1139d0 con 0x7fbdec1052e0 2024-04-01T01:48:21.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.484+0000 7fbdf1552640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdec1052e0 0x7fbdec1a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.484+0000 7fbdf1552640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdec1052e0 0x7fbdec1a5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42040/0 (socket says 172.21.15.70:42040) 2024-04-01T01:48:21.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.484+0000 7fbdf1552640 1 -- 172.21.15.70:0/748275948 learned_addr learned my addr 172.21.15.70:0/748275948 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:21.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.484+0000 7fbdebfff640 1 --2- 172.21.15.70:0/748275948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdec102930 0x7fbdec19e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.484+0000 7fbdf0d51640 1 --2- 172.21.15.70:0/748275948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbdec0fffc0 0x7fbdec19df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.484+0000 7fbdf1552640 1 -- 172.21.15.70:0/748275948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdec102930 msgr2=0x7fbdec19e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:21.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.484+0000 7fbdf1552640 1 --2- 172.21.15.70:0/748275948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdec102930 0x7fbdec19e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.485+0000 7fbdf1552640 1 -- 172.21.15.70:0/748275948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbdec0fffc0 msgr2=0x7fbdec19df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:21.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.485+0000 7fbdf1552640 1 --2- 172.21.15.70:0/748275948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbdec0fffc0 0x7fbdec19df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.485+0000 7fbdf1552640 1 -- 172.21.15.70:0/748275948 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdec06b2a0 con 0x7fbdec1052e0 2024-04-01T01:48:21.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.485+0000 7fbdf1552640 1 --2- 172.21.15.70:0/748275948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdec1052e0 0x7fbdec1a5510 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbde4009fd0 tx=0x7fbde400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:21.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.485+0000 7fbde9ffb640 1 -- 172.21.15.70:0/748275948 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde4004b20 con 0x7fbdec1052e0 2024-04-01T01:48:21.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.485+0000 7fbdf2fb5640 1 -- 172.21.15.70:0/748275948 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdec06b590 con 0x7fbdec1052e0 2024-04-01T01:48:21.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.485+0000 7fbdf2fb5640 1 -- 172.21.15.70:0/748275948 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdec06bab0 con 0x7fbdec1052e0 2024-04-01T01:48:21.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.485+0000 7fbde9ffb640 1 -- 172.21.15.70:0/748275948 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbde4003e20 con 0x7fbdec1052e0 2024-04-01T01:48:21.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.486+0000 7fbde9ffb640 1 -- 172.21.15.70:0/748275948 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde401a020 con 0x7fbdec1052e0 2024-04-01T01:48:21.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.489+0000 7fbde9ffb640 1 -- 172.21.15.70:0/748275948 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fbde401d020 con 0x7fbdec1052e0 2024-04-01T01:48:21.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.489+0000 7fbde9ffb640 1 --2- 172.21.15.70:0/748275948 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbdd40a24a0 0x7fbdd40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.489+0000 7fbdf5b9a740 1 Processor -- start 2024-04-01T01:48:21.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.489+0000 7fbdf5b9a740 1 -- start start 2024-04-01T01:48:21.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.489+0000 7fbde9ffb640 1 -- 172.21.15.70:0/748275948 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fbde40cea00 con 0x7fbdec1052e0 2024-04-01T01:48:21.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.489+0000 7fbdf0d51640 1 --2- 172.21.15.70:0/748275948 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbdd40a24a0 0x7fbdd40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdf5b9a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a2962b0 0x55640a2966d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdf5b9a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a296ca0 0x55640a29b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdf5b9a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a29be10 0x55640a29e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdebfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a296ca0 0x55640a29b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdf0d51640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a2962b0 0x55640a2966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdebfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a296ca0 0x55640a29b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41196/0 (socket says 172.21.15.70:41196) 2024-04-01T01:48:21.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdf0d51640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a2962b0 0x55640a2966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42046/0 (socket says 172.21.15.70:42046) 2024-04-01T01:48:21.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdebfff640 1 -- 172.21.15.70:0/2759238246 learned_addr learned my addr 172.21.15.70:0/2759238246 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:21.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/2759238246 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55640a1fb180 con 0x55640a29be10 2024-04-01T01:48:21.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/2759238246 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55640a1a3b70 con 0x55640a296ca0 2024-04-01T01:48:21.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/2759238246 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55640a1f7220 con 0x55640a2962b0 2024-04-01T01:48:21.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdf1552640 1 --2- 172.21.15.70:0/2759238246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a29be10 0x55640a29e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdebfff640 1 -- 172.21.15.70:0/2759238246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a2962b0 msgr2=0x55640a2966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:21.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdebfff640 1 --2- 172.21.15.70:0/2759238246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a2962b0 0x55640a2966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdebfff640 1 -- 172.21.15.70:0/2759238246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a29be10 msgr2=0x55640a29e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:21.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdebfff640 1 --2- 172.21.15.70:0/2759238246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a29be10 0x55640a29e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.490+0000 7fbdebfff640 1 -- 172.21.15.70:0/2759238246 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55640a1b77c0 con 0x55640a296ca0 2024-04-01T01:48:21.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.491+0000 7fbdf0d51640 1 --2- 172.21.15.70:0/748275948 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbdd40a24a0 0x7fbdd40a4960 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fbddc009c20 tx=0x7fbddc011810 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:21.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.491+0000 7fbdebfff640 1 --2- 172.21.15.70:0/2759238246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a296ca0 0x55640a29b7a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009860 tx=0x7fbde0009490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=556eab5c53af3c78 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:21.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.491+0000 7fbdd2ffd640 1 -- 172.21.15.70:0/2759238246 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0043070 con 0x55640a296ca0 2024-04-01T01:48:21.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.491+0000 7fbdd2ffd640 1 -- 172.21.15.70:0/2759238246 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbde00054f0 con 0x55640a296ca0 2024-04-01T01:48:21.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.491+0000 7fbdd2ffd640 1 -- 172.21.15.70:0/2759238246 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0039070 con 0x55640a296ca0 2024-04-01T01:48:21.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.492+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/2759238246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a296ca0 msgr2=0x55640a29b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:21.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.492+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/2759238246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a296ca0 0x55640a29b7a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009860 tx=0x7fbde0009490 comp rx=0 tx=0).stop 2024-04-01T01:48:21.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.492+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/2759238246 shutdown_connections 2024-04-01T01:48:21.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.492+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/2759238246 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a29be10 0x55640a29e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.492+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/2759238246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a296ca0 0x55640a29b7a0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.492+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/2759238246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a2962b0 0x55640a2966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.492+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/2759238246 >> 172.21.15.70:0/2759238246 conn(0x55640a291820 msgr2=0x55640a291c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.493+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/2759238246 shutdown_connections 2024-04-01T01:48:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.493+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/2759238246 wait complete. 2024-04-01T01:48:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.493+0000 7fbdf5b9a740 1 -- start start 2024-04-01T01:48:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.493+0000 7fbdf5b9a740 1 Processor -- start 2024-04-01T01:48:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.494+0000 7fbdf5b9a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a2962b0 0x55640a2a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.494+0000 7fbdf5b9a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a296ca0 0x55640a2a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.494+0000 7fbdf5b9a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a29be10 0x55640a2a7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.494+0000 7fbdf0d51640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a2962b0 0x55640a2a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.494+0000 7fbdf0d51640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a2962b0 0x55640a2a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41200/0 (socket says 172.21.15.70:41200) 2024-04-01T01:48:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.494+0000 7fbdebfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a296ca0 0x55640a2a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.494+0000 7fbdebfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a296ca0 0x55640a2a0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42062/0 (socket says 172.21.15.70:42062) 2024-04-01T01:48:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.494+0000 7fbdf5b9a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55640a29ed60 con 0x55640a29be10 2024-04-01T01:48:21.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.494+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/3323463993 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55640a1994a0 con 0x55640a2962b0 2024-04-01T01:48:21.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.494+0000 7fbdf0d51640 1 -- 172.21.15.70:0/3323463993 learned_addr learned my addr 172.21.15.70:0/3323463993 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:21.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.494+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/3323463993 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55640a29eee0 con 0x55640a296ca0 2024-04-01T01:48:21.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.494+0000 7fbdf1552640 1 --2- 172.21.15.70:0/3323463993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a29be10 0x55640a2a7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.495+0000 7fbdf0d51640 1 -- 172.21.15.70:0/3323463993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a296ca0 msgr2=0x55640a2a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:21.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.495+0000 7fbdf0d51640 1 --2- 172.21.15.70:0/3323463993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a296ca0 0x55640a2a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.495+0000 7fbdf0d51640 1 -- 172.21.15.70:0/3323463993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a29be10 msgr2=0x55640a2a7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:21.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.495+0000 7fbdf0d51640 1 --2- 172.21.15.70:0/3323463993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a29be10 0x55640a2a7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.495+0000 7fbdf0d51640 1 -- 172.21.15.70:0/3323463993 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556409fd97d0 con 0x55640a2962b0 2024-04-01T01:48:21.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.495+0000 7fbdf0d51640 1 --2- 172.21.15.70:0/3323463993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a2962b0 0x55640a2a05a0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fbddc01e9b0 tx=0x7fbddc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1535983e0e661354 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:21.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.495+0000 7fbdcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:21.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.495+0000 7fbdf5b9a740 10 client.0 _ll_register_callbacks cb 0x55640a19cd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:21.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.495+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/3323463993 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbde00054f0 con 0x55640a2962b0 2024-04-01T01:48:21.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.495+0000 7fbdcbfff640 1 -- 172.21.15.70:0/3323463993 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbddc017040 con 0x55640a2962b0 2024-04-01T01:48:21.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.495+0000 7fbdcbfff640 1 -- 172.21.15.70:0/3323463993 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fbddc009040 con 0x55640a2962b0 2024-04-01T01:48:21.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.495+0000 7fbdcbfff640 1 -- 172.21.15.70:0/3323463993 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbddc030050 con 0x55640a2962b0 2024-04-01T01:48:21.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.495+0000 7fbdf5b9a740 10 client.29798 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:21.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.495+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/3323463993 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=31}) -- 0x556409f0f3f0 con 0x55640a2962b0 2024-04-01T01:48:21.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.496+0000 7fbdcbfff640 1 -- 172.21.15.70:0/3323463993 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fbddc01e030 con 0x55640a2962b0 2024-04-01T01:48:21.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.496+0000 7fbdf5b9a740 10 client.29798 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:21.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.496+0000 7fbdf5b9a740 10 client.29798 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:21.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.496+0000 7fbdf5b9a740 20 client.29798 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:21.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.496+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/3323463993 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55640a2ac110 0x55640a2ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.496+0000 7fbdf5b9a740 4 client.29798 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:21.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.496+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/3323463993 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55640a2086c0 con 0x55640a2ac110 2024-04-01T01:48:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.496+0000 7fbdebfff640 1 --2- 172.21.15.70:0/3323463993 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55640a2ac110 0x55640a2ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.497+0000 7fbdcbfff640 10 client.29798 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.497+0000 7fbdebfff640 1 --2- 172.21.15.70:0/3323463993 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55640a2ac110 0x55640a2ae5d0 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c5c140f70e26bc8 server_cookie=eec378828b2f6445 in_seq=0 out_seq=0 2024-04-01T01:48:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.497+0000 7fbdcbfff640 1 -- 172.21.15.70:0/3323463993 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55640a2086c0 con 0x55640a2ac110 2024-04-01T01:48:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.497+0000 7fbdcbfff640 10 client.29798 handle_command_reply: tid=0 2024-04-01T01:48:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.497+0000 7fbdf5b9a740 1 client.29798 shutdown 2024-04-01T01:48:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.497+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/3323463993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a2962b0 msgr2=0x55640a2a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.497+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/3323463993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a2962b0 0x55640a2a05a0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fbddc01e9b0 tx=0x7fbddc004730 comp rx=0 tx=0).stop 2024-04-01T01:48:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.497+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/3323463993 shutdown_connections 2024-04-01T01:48:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.497+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/3323463993 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55640a2ac110 0x55640a2ae5d0 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.497+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/3323463993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a29be10 0x55640a2a7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.497+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/3323463993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a296ca0 0x55640a2a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.497+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/3323463993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a2962b0 0x55640a2a05a0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.497+0000 7fbdcbfff640 0 client.29798 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:21.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.497+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/3323463993 >> 172.21.15.70:0/3323463993 conn(0x55640a291820 msgr2=0x55640a290c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:21.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.498+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/3323463993 shutdown_connections 2024-04-01T01:48:21.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.498+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/3323463993 wait complete. 2024-04-01T01:48:21.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.498+0000 7fbdf5b9a740 20 client.29798 trim_cache size 0 max 16384 2024-04-01T01:48:21.503 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:48:21.503 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:48:21.503 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:48:21.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.521+0000 7fbdf5b9a740 1 Processor -- start 2024-04-01T01:48:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.521+0000 7fbdf5b9a740 1 -- start start 2024-04-01T01:48:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.521+0000 7fbdf5b9a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a2962b0 0x55640a2a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.521+0000 7fbdf5b9a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a296ca0 0x55640a2a7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.521+0000 7fbdf5b9a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a29be10 0x55640a290390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.521+0000 7fbdf5b9a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd800fd00 con 0x55640a2962b0 2024-04-01T01:48:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.521+0000 7fbdf5b9a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd8011840 con 0x55640a29be10 2024-04-01T01:48:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.521+0000 7fbdf5b9a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd80103d0 con 0x55640a296ca0 2024-04-01T01:48:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.521+0000 7fbdf0d51640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a2962b0 0x55640a2a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.521+0000 7fbdf0d51640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a2962b0 0x55640a2a0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34814/0 (socket says 172.21.15.70:34814) 2024-04-01T01:48:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.521+0000 7fbdf0d51640 1 -- 172.21.15.70:0/4177112115 learned_addr learned my addr 172.21.15.70:0/4177112115 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.521+0000 7fbdf1552640 1 --2- 172.21.15.70:0/4177112115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a29be10 0x55640a290390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.522+0000 7fbdf0d51640 1 -- 172.21.15.70:0/4177112115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a296ca0 msgr2=0x55640a2a7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:21.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.522+0000 7fbdf0d51640 1 --2- 172.21.15.70:0/4177112115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a296ca0 0x55640a2a7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.522+0000 7fbdf0d51640 1 -- 172.21.15.70:0/4177112115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a29be10 msgr2=0x55640a290390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:21.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.522+0000 7fbdf0d51640 1 --2- 172.21.15.70:0/4177112115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a29be10 0x55640a290390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.522+0000 7fbdf0d51640 1 -- 172.21.15.70:0/4177112115 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556409ecf0f0 con 0x55640a2962b0 2024-04-01T01:48:21.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.522+0000 7fbdf0d51640 1 --2- 172.21.15.70:0/4177112115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a2962b0 0x55640a2a0a40 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fbddc009f20 tx=0x7fbddc020040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1a005e01b4605ba server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:21.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.522+0000 7fbdd27fc640 1 -- 172.21.15.70:0/4177112115 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbddc017070 con 0x55640a2962b0 2024-04-01T01:48:21.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.522+0000 7fbdd27fc640 1 -- 172.21.15.70:0/4177112115 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbde00054f0 con 0x55640a2962b0 2024-04-01T01:48:21.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.522+0000 7fbdd27fc640 1 -- 172.21.15.70:0/4177112115 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbddc004990 con 0x55640a2962b0 2024-04-01T01:48:21.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.523+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/4177112115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a2962b0 msgr2=0x55640a2a0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:21.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.523+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/4177112115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a2962b0 0x55640a2a0a40 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fbddc009f20 tx=0x7fbddc020040 comp rx=0 tx=0).stop 2024-04-01T01:48:21.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.523+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/4177112115 shutdown_connections 2024-04-01T01:48:21.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.523+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/4177112115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a29be10 0x55640a290390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.523+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/4177112115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a296ca0 0x55640a2a7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.523+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/4177112115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a2962b0 0x55640a2a0a40 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.523+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/4177112115 >> 172.21.15.70:0/4177112115 conn(0x55640a291820 msgr2=0x55640a29f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:21.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.523+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/4177112115 shutdown_connections 2024-04-01T01:48:21.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.523+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/4177112115 wait complete. 2024-04-01T01:48:21.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.524+0000 7fbdf5b9a740 1 -- start start 2024-04-01T01:48:21.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.524+0000 7fbdf5b9a740 1 Processor -- start 2024-04-01T01:48:21.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.524+0000 7fbdf5b9a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a2962b0 0x55640a2a5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.524+0000 7fbdf5b9a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a296ca0 0x55640a2a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.524+0000 7fbdf5b9a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a29be10 0x55640a2a66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.524+0000 7fbdf5b9a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd8011170 con 0x55640a29be10 2024-04-01T01:48:21.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.524+0000 7fbdf5b9a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd8010aa0 con 0x55640a296ca0 2024-04-01T01:48:21.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.524+0000 7fbdf5b9a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbdd800dce0 con 0x55640a2962b0 2024-04-01T01:48:21.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.524+0000 7fbdf5b9a740 10 client.0 _ll_register_callbacks cb 0x7fbddc009f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:21.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.524+0000 7fbdebfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a296ca0 0x55640a2a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.524+0000 7fbdebfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a296ca0 0x55640a2a56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41220/0 (socket says 172.21.15.70:41220) 2024-04-01T01:48:21.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.524+0000 7fbdebfff640 1 -- 172.21.15.70:0/4032855632 learned_addr learned my addr 172.21.15.70:0/4032855632 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:21.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.525+0000 7fbdebfff640 1 -- 172.21.15.70:0/4032855632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a2962b0 msgr2=0x55640a2a5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:21.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.525+0000 7fbdebfff640 1 --2- 172.21.15.70:0/4032855632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a2962b0 0x55640a2a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.525+0000 7fbdebfff640 1 -- 172.21.15.70:0/4032855632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a29be10 msgr2=0x55640a2a66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:21.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.525+0000 7fbdebfff640 1 --2- 172.21.15.70:0/4032855632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a29be10 0x55640a2a66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.525+0000 7fbdebfff640 1 -- 172.21.15.70:0/4032855632 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55640a2a6cd0 con 0x55640a296ca0 2024-04-01T01:48:21.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.525+0000 7fbdebfff640 1 --2- 172.21.15.70:0/4032855632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a296ca0 0x55640a2a56d0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fbde0034c40 tx=0x7fbde0034670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e11f3e638ff8ac51 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:21.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.525+0000 7fbdcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:21.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.525+0000 7fbdcbfff640 1 -- 172.21.15.70:0/4032855632 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0039040 con 0x55640a296ca0 2024-04-01T01:48:21.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.525+0000 7fbdcbfff640 1 -- 172.21.15.70:0/4032855632 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fbde0004450 con 0x55640a296ca0 2024-04-01T01:48:21.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.525+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/4032855632 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbde00054f0 con 0x55640a296ca0 2024-04-01T01:48:21.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.526+0000 7fbdcbfff640 1 -- 172.21.15.70:0/4032855632 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbde0043040 con 0x55640a296ca0 2024-04-01T01:48:21.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.526+0000 7fbdf5b9a740 10 client.29804 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:21.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.526+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/4032855632 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55640a312510 con 0x55640a296ca0 2024-04-01T01:48:21.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.526+0000 7fbdcbfff640 1 -- 172.21.15.70:0/4032855632 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbde00054f0 con 0x55640a296ca0 2024-04-01T01:48:21.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.526+0000 7fbdcbfff640 1 -- 172.21.15.70:0/4032855632 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fbde0053b60 con 0x55640a296ca0 2024-04-01T01:48:21.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.526+0000 7fbdf5b9a740 10 client.29804 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:21.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.526+0000 7fbdf5b9a740 10 client.29804 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:21.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.526+0000 7fbdf5b9a740 20 client.29804 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:21.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.526+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/4032855632 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55640a312980 0x55640a314d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.526+0000 7fbdf5b9a740 4 client.29804 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:48:21.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.526+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/4032855632 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55640a202a10 con 0x55640a312980 2024-04-01T01:48:21.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.526+0000 7fbdf0d51640 1 --2- 172.21.15.70:0/4032855632 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55640a312980 0x55640a314d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.527+0000 7fbdcbfff640 10 client.29804 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:21.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.527+0000 7fbdf0d51640 1 --2- 172.21.15.70:0/4032855632 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55640a312980 0x55640a314d70 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4b61fb1eee10c9a server_cookie=d64dc38a3d8dd1d9 in_seq=0 out_seq=0 2024-04-01T01:48:21.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.528+0000 7fbdcbfff640 1 -- 172.21.15.70:0/4032855632 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55640a202a10 con 0x55640a312980 2024-04-01T01:48:21.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.528+0000 7fbdcbfff640 10 client.29804 handle_command_reply: tid=0 2024-04-01T01:48:21.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.528+0000 7fbdf5b9a740 1 client.29804 shutdown 2024-04-01T01:48:21.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.528+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/4032855632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a296ca0 msgr2=0x55640a2a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:21.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.528+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/4032855632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a296ca0 0x55640a2a56d0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fbde0034c40 tx=0x7fbde0034670 comp rx=0 tx=0).stop 2024-04-01T01:48:21.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.528+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/4032855632 shutdown_connections 2024-04-01T01:48:21.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.528+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/4032855632 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55640a312980 0x55640a314d70 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.528+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/4032855632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55640a29be10 0x55640a2a66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.528+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/4032855632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55640a296ca0 0x55640a2a56d0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.528+0000 7fbdf5b9a740 1 --2- 172.21.15.70:0/4032855632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55640a2962b0 0x55640a2a5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.528+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/4032855632 >> 172.21.15.70:0/4032855632 conn(0x55640a291820 msgr2=0x55640a294a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:21.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.528+0000 7fbdcbfff640 0 client.29804 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:21.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.528+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/4032855632 shutdown_connections 2024-04-01T01:48:21.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.528+0000 7fbdf5b9a740 1 -- 172.21.15.70:0/4032855632 wait complete. 2024-04-01T01:48:21.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.528+0000 7fbdf5b9a740 20 client.29804 trim_cache size 0 max 16384 2024-04-01T01:48:21.532 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:48:21.532 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:48:21.533 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:48:21.533 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:48:21.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.529+0000 7fbdf2fb5640 1 -- 172.21.15.70:0/748275948 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbdd40a24a0 msgr2=0x7fbdd40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:21.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.529+0000 7fbdf2fb5640 1 --2- 172.21.15.70:0/748275948 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbdd40a24a0 0x7fbdd40a4960 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fbddc009c20 tx=0x7fbddc011810 comp rx=0 tx=0).stop 2024-04-01T01:48:21.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.529+0000 7fbdf2fb5640 1 -- 172.21.15.70:0/748275948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdec1052e0 msgr2=0x7fbdec1a5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:21.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.529+0000 7fbdf2fb5640 1 --2- 172.21.15.70:0/748275948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdec1052e0 0x7fbdec1a5510 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbde4009fd0 tx=0x7fbde400c800 comp rx=0 tx=0).stop 2024-04-01T01:48:21.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.529+0000 7fbdf2fb5640 1 -- 172.21.15.70:0/748275948 shutdown_connections 2024-04-01T01:48:21.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.529+0000 7fbdf2fb5640 1 --2- 172.21.15.70:0/748275948 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbdd40a24a0 0x7fbdd40a4960 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.529+0000 7fbdf2fb5640 1 --2- 172.21.15.70:0/748275948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbdec1052e0 0x7fbdec1a5510 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.529+0000 7fbdf2fb5640 1 --2- 172.21.15.70:0/748275948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbdec102930 0x7fbdec19e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.529+0000 7fbdf2fb5640 1 --2- 172.21.15.70:0/748275948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbdec0fffc0 0x7fbdec19df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.529+0000 7fbdf2fb5640 1 -- 172.21.15.70:0/748275948 >> 172.21.15.70:0/748275948 conn(0x7fbdec0fbbe0 msgr2=0x7fbdec100940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:21.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.529+0000 7fbdf2fb5640 1 -- 172.21.15.70:0/748275948 shutdown_connections 2024-04-01T01:48:21.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.529+0000 7fbdf2fb5640 1 -- 172.21.15.70:0/748275948 wait complete. 2024-04-01T01:48:21.543 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:48:21.543 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:48:21.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.748+0000 7f129af56640 1 -- 172.21.15.70:0/1060817456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1294105540 msgr2=0x7f129410bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:21.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.748+0000 7f129af56640 1 --2- 172.21.15.70:0/1060817456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1294105540 0x7f129410bdd0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f128c009ca0 tx=0x7f128c02c100 comp rx=0 tx=0).stop 2024-04-01T01:48:21.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.748+0000 7f129af56640 1 -- 172.21.15.70:0/1060817456 shutdown_connections 2024-04-01T01:48:21.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.748+0000 7f129af56640 1 --2- 172.21.15.70:0/1060817456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1294105540 0x7f129410bdd0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.748+0000 7f129af56640 1 --2- 172.21.15.70:0/1060817456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1294104b80 0x7f1294105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.748+0000 7f129af56640 1 --2- 172.21.15.70:0/1060817456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1294103990 0x7f1294103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.748+0000 7f129af56640 1 -- 172.21.15.70:0/1060817456 >> 172.21.15.70:0/1060817456 conn(0x7f12940fef00 msgr2=0x7f1294101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:21.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.749+0000 7f129af56640 1 -- 172.21.15.70:0/1060817456 shutdown_connections 2024-04-01T01:48:21.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.749+0000 7f129af56640 1 -- 172.21.15.70:0/1060817456 wait complete. 2024-04-01T01:48:21.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.750+0000 7f129af56640 1 Processor -- start 2024-04-01T01:48:21.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.750+0000 7f129af56640 1 -- start start 2024-04-01T01:48:21.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.751+0000 7f129af56640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1294103990 0x7f1294199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.751+0000 7f129af56640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1294104b80 0x7f1294199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.751+0000 7f129af56640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1294105540 0x7f12941a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.751+0000 7f1298cf2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1294103990 0x7f1294199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.751+0000 7f1298cf2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1294103990 0x7f1294199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41242/0 (socket says 172.21.15.70:41242) 2024-04-01T01:48:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.751+0000 7f1293fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1294104b80 0x7f1294199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.751+0000 7f1293fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1294104b80 0x7f1294199e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42088/0 (socket says 172.21.15.70:42088) 2024-04-01T01:48:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.751+0000 7f1298cf2640 1 -- 172.21.15.70:0/160937287 learned_addr learned my addr 172.21.15.70:0/160937287 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.751+0000 7f129af56640 1 -- 172.21.15.70:0/160937287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f129410f4a0 con 0x7f1294105540 2024-04-01T01:48:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.751+0000 7f129af56640 1 -- 172.21.15.70:0/160937287 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f129410f320 con 0x7f1294103990 2024-04-01T01:48:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.751+0000 7f129af56640 1 -- 172.21.15.70:0/160937287 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f129410f620 con 0x7f1294104b80 2024-04-01T01:48:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.751+0000 7f12994f3640 1 --2- 172.21.15.70:0/160937287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1294105540 0x7f12941a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.752+0000 7f1298cf2640 1 -- 172.21.15.70:0/160937287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1294104b80 msgr2=0x7f1294199e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.752+0000 7f1298cf2640 1 --2- 172.21.15.70:0/160937287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1294104b80 0x7f1294199e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.752+0000 7f1298cf2640 1 -- 172.21.15.70:0/160937287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1294105540 msgr2=0x7f12941a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.752+0000 7f1298cf2640 1 --2- 172.21.15.70:0/160937287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1294105540 0x7f12941a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.752+0000 7f1298cf2640 1 -- 172.21.15.70:0/160937287 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12941a1530 con 0x7f1294103990 2024-04-01T01:48:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.752+0000 7f1298cf2640 1 --2- 172.21.15.70:0/160937287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1294103990 0x7f1294199910 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f128800d950 tx=0x7f128800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.752+0000 7f1291ffb640 1 -- 172.21.15.70:0/160937287 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288017020 con 0x7f1294103990 2024-04-01T01:48:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.752+0000 7f1291ffb640 1 -- 172.21.15.70:0/160937287 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1288009c30 con 0x7f1294103990 2024-04-01T01:48:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.752+0000 7f1291ffb640 1 -- 172.21.15.70:0/160937287 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128800f6b0 con 0x7f1294103990 2024-04-01T01:48:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.753+0000 7f129af56640 1 -- 172.21.15.70:0/160937287 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12941a1820 con 0x7f1294103990 2024-04-01T01:48:21.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.753+0000 7f129af56640 1 -- 172.21.15.70:0/160937287 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12941a1d20 con 0x7f1294103990 2024-04-01T01:48:21.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.755+0000 7f129af56640 1 -- 172.21.15.70:0/160937287 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f129410abe0 con 0x7f1294103990 2024-04-01T01:48:21.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.756+0000 7f1291ffb640 1 -- 172.21.15.70:0/160937287 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f128801a020 con 0x7f1294103990 2024-04-01T01:48:21.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.756+0000 7f1291ffb640 1 --2- 172.21.15.70:0/160937287 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12840a2520 0x7f12840a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:21.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.757+0000 7f1291ffb640 1 -- 172.21.15.70:0/160937287 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f12880ce400 con 0x7f1294103990 2024-04-01T01:48:21.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.757+0000 7f1293fff640 1 --2- 172.21.15.70:0/160937287 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12840a2520 0x7f12840a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:21.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.758+0000 7f1293fff640 1 --2- 172.21.15.70:0/160937287 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12840a2520 0x7f12840a49e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f129419ae20 tx=0x7f128000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:21.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:21.765+0000 7f1291ffb640 1 -- 172.21.15.70:0/160937287 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1288021070 con 0x7f1294103990 2024-04-01T01:48:21.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 815.0 s, 121.2 tps, lat 560.379 ms stddev 712.124 2024-04-01T01:48:22.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.189+0000 7f129af56640 1 -- 172.21.15.70:0/160937287 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f129402d170 con 0x7f1294103990 2024-04-01T01:48:22.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.190+0000 7f1291ffb640 1 -- 172.21.15.70:0/160937287 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7f129402d170 con 0x7f1294103990 2024-04-01T01:48:22.190 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:48:22.190 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:48:22.190 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:48:22.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.197+0000 7f129af56640 1 -- 172.21.15.70:0/160937287 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12840a2520 msgr2=0x7f12840a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.197+0000 7f129af56640 1 --2- 172.21.15.70:0/160937287 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12840a2520 0x7f12840a49e0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f129419ae20 tx=0x7f128000a2d0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.197+0000 7f129af56640 1 -- 172.21.15.70:0/160937287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1294103990 msgr2=0x7f1294199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.197+0000 7f129af56640 1 --2- 172.21.15.70:0/160937287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1294103990 0x7f1294199910 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f128800d950 tx=0x7f128800de10 comp rx=0 tx=0).stop 2024-04-01T01:48:22.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.198+0000 7f129af56640 1 -- 172.21.15.70:0/160937287 shutdown_connections 2024-04-01T01:48:22.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.198+0000 7f129af56640 1 --2- 172.21.15.70:0/160937287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1294105540 0x7f12941a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.198+0000 7f129af56640 1 --2- 172.21.15.70:0/160937287 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f12840a2520 0x7f12840a49e0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.198+0000 7f129af56640 1 --2- 172.21.15.70:0/160937287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1294104b80 0x7f1294199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.198+0000 7f129af56640 1 --2- 172.21.15.70:0/160937287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1294103990 0x7f1294199910 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.198+0000 7f129af56640 1 -- 172.21.15.70:0/160937287 >> 172.21.15.70:0/160937287 conn(0x7f12940fef00 msgr2=0x7f1294107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.198+0000 7f129af56640 1 -- 172.21.15.70:0/160937287 shutdown_connections 2024-04-01T01:48:22.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.198+0000 7f129af56640 1 -- 172.21.15.70:0/160937287 wait complete. 2024-04-01T01:48:22.220 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:48:22.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.410+0000 7f563369d640 1 -- 172.21.15.70:0/2387073642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f562c10b0c0 msgr2=0x7f562c10d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.410+0000 7f563369d640 1 --2- 172.21.15.70:0/2387073642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f562c10b0c0 0x7f562c10d530 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f5624009be0 tx=0x7f562402bf10 comp rx=0 tx=0).stop 2024-04-01T01:48:22.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.410+0000 7f563369d640 1 -- 172.21.15.70:0/2387073642 shutdown_connections 2024-04-01T01:48:22.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.410+0000 7f563369d640 1 --2- 172.21.15.70:0/2387073642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f562c10da70 0x7f562c10def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.410+0000 7f563369d640 1 --2- 172.21.15.70:0/2387073642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f562c10b0c0 0x7f562c10d530 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.410+0000 7f563369d640 1 --2- 172.21.15.70:0/2387073642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f562c108750 0x7f562c10ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.410+0000 7f563369d640 1 -- 172.21.15.70:0/2387073642 >> 172.21.15.70:0/2387073642 conn(0x7f562c06e5b0 msgr2=0x7f562c0709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.410+0000 7f563369d640 1 -- 172.21.15.70:0/2387073642 shutdown_connections 2024-04-01T01:48:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.410+0000 7f563369d640 1 -- 172.21.15.70:0/2387073642 wait complete. 2024-04-01T01:48:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.411+0000 7f563369d640 1 Processor -- start 2024-04-01T01:48:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.411+0000 7f563369d640 1 -- start start 2024-04-01T01:48:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.411+0000 7f563369d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f562c108750 0x7f562c1a6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.411+0000 7f563369d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f562c10b0c0 0x7f562c1a6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.411+0000 7f563369d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f562c10da70 0x7f562c1ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.411+0000 7f563369d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f562c11c140 con 0x7f562c10b0c0 2024-04-01T01:48:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.411+0000 7f563369d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f562c11bfc0 con 0x7f562c108750 2024-04-01T01:48:22.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.411+0000 7f563369d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f562c11c2c0 con 0x7f562c10da70 2024-04-01T01:48:22.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.415+0000 7f5631c3a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f562c10da70 0x7f562c1ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.415+0000 7f5631c3a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f562c10da70 0x7f562c1ade20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42102/0 (socket says 172.21.15.70:42102) 2024-04-01T01:48:22.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.415+0000 7f5631c3a640 1 -- 172.21.15.70:0/3936443765 learned_addr learned my addr 172.21.15.70:0/3936443765 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.415+0000 7f5631439640 1 --2- 172.21.15.70:0/3936443765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f562c108750 0x7f562c1a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.415+0000 7f5630c38640 1 --2- 172.21.15.70:0/3936443765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f562c10b0c0 0x7f562c1a6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.415+0000 7f5631c3a640 1 -- 172.21.15.70:0/3936443765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f562c108750 msgr2=0x7f562c1a6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.415+0000 7f5631c3a640 1 --2- 172.21.15.70:0/3936443765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f562c108750 0x7f562c1a6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.416+0000 7f5631c3a640 1 -- 172.21.15.70:0/3936443765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f562c10b0c0 msgr2=0x7f562c1a6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.416+0000 7f5631c3a640 1 --2- 172.21.15.70:0/3936443765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f562c10b0c0 0x7f562c1a6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.416+0000 7f5631c3a640 1 -- 172.21.15.70:0/3936443765 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f562c1ae490 con 0x7f562c10da70 2024-04-01T01:48:22.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.416+0000 7f5630c38640 1 --2- 172.21.15.70:0/3936443765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f562c10b0c0 0x7f562c1a6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:48:22.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.416+0000 7f5631c3a640 1 --2- 172.21.15.70:0/3936443765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f562c10da70 0x7f562c1ade20 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f562800e940 tx=0x7f562800ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.417+0000 7f56227fc640 1 -- 172.21.15.70:0/3936443765 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562800ccc0 con 0x7f562c10da70 2024-04-01T01:48:22.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.417+0000 7f56227fc640 1 -- 172.21.15.70:0/3936443765 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f56280054d0 con 0x7f562c10da70 2024-04-01T01:48:22.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.417+0000 7f56227fc640 1 -- 172.21.15.70:0/3936443765 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628010430 con 0x7f562c10da70 2024-04-01T01:48:22.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.417+0000 7f563369d640 1 -- 172.21.15.70:0/3936443765 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f562c1ae780 con 0x7f562c10da70 2024-04-01T01:48:22.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.417+0000 7f563369d640 1 -- 172.21.15.70:0/3936443765 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f562c1aecd0 con 0x7f562c10da70 2024-04-01T01:48:22.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.421+0000 7f56227fc640 1 -- 172.21.15.70:0/3936443765 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f56280105d0 con 0x7f562c10da70 2024-04-01T01:48:22.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.422+0000 7f56227fc640 1 --2- 172.21.15.70:0/3936443765 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56180a2670 0x7f56180a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.423+0000 7f56227fc640 1 -- 172.21.15.70:0/3936443765 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f56280cfe50 con 0x7f562c10da70 2024-04-01T01:48:22.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.423+0000 7f5631439640 1 --2- 172.21.15.70:0/3936443765 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56180a2670 0x7f56180a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.425+0000 7f5635f22740 1 Processor -- start 2024-04-01T01:48:22.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.425+0000 7f5635f22740 1 -- start start 2024-04-01T01:48:22.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.425+0000 7f5635f22740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 0x55b4962d98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.426+0000 7f5635f22740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d9e80 0x55b4962de980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.426+0000 7f5635f22740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962deff0 0x55b4962e1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.426+0000 7f5635f22740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b49623e1b0 con 0x55b4962deff0 2024-04-01T01:48:22.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.426+0000 7f5631c3a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962deff0 0x55b4962e1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.426+0000 7f5631c3a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962deff0 0x55b4962e1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34852/0 (socket says 172.21.15.70:34852) 2024-04-01T01:48:22.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.426+0000 7f5631c3a640 1 -- 172.21.15.70:0/399106354 learned_addr learned my addr 172.21.15.70:0/399106354 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.426+0000 7f5635f22740 1 -- 172.21.15.70:0/399106354 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b4961e6c00 con 0x55b4962d9490 2024-04-01T01:48:22.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.426+0000 7f5635f22740 1 -- 172.21.15.70:0/399106354 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b49623a250 con 0x55b4962d9e80 2024-04-01T01:48:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.428+0000 7f5631439640 1 --2- 172.21.15.70:0/399106354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 0x55b4962d98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.429+0000 7f5631c3a640 1 -- 172.21.15.70:0/399106354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d9e80 msgr2=0x55b4962de980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:48:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.429+0000 7f5631c3a640 1 --2- 172.21.15.70:0/399106354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d9e80 0x55b4962de980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.429+0000 7f5631c3a640 1 -- 172.21.15.70:0/399106354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 msgr2=0x55b4962d98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.429+0000 7f5631c3a640 1 --2- 172.21.15.70:0/399106354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 0x55b4962d98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.429+0000 7f5631c3a640 1 -- 172.21.15.70:0/399106354 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4962063d0 con 0x55b4962deff0 2024-04-01T01:48:22.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.429+0000 7f5631439640 1 --2- 172.21.15.70:0/399106354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 0x55b4962d98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:48:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.429+0000 7f5631439640 1 --2- 172.21.15.70:0/3936443765 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56180a2670 0x7f56180a4b30 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f561c006fd0 tx=0x7f561c0090e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.430+0000 7f5631c3a640 1 --2- 172.21.15.70:0/399106354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962deff0 0x55b4962e1480 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f5628098eb0 tx=0x7f5628098890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=949471c56923578a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.431+0000 7f56177fe640 1 -- 172.21.15.70:0/399106354 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562809cbe0 con 0x55b4962deff0 2024-04-01T01:48:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.431+0000 7f56177fe640 1 -- 172.21.15.70:0/399106354 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f562809c4e0 con 0x55b4962deff0 2024-04-01T01:48:22.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.431+0000 7f56177fe640 1 -- 172.21.15.70:0/399106354 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628098ce0 con 0x55b4962deff0 2024-04-01T01:48:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.432+0000 7f5635f22740 1 -- 172.21.15.70:0/399106354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962deff0 msgr2=0x55b4962e1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.432+0000 7f5635f22740 1 --2- 172.21.15.70:0/399106354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962deff0 0x55b4962e1480 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f5628098eb0 tx=0x7f5628098890 comp rx=0 tx=0).stop 2024-04-01T01:48:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.432+0000 7f5635f22740 1 -- 172.21.15.70:0/399106354 shutdown_connections 2024-04-01T01:48:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.432+0000 7f5635f22740 1 --2- 172.21.15.70:0/399106354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962deff0 0x55b4962e1480 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.432+0000 7f5635f22740 1 --2- 172.21.15.70:0/399106354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d9e80 0x55b4962de980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.432+0000 7f5635f22740 1 --2- 172.21.15.70:0/399106354 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 0x55b4962d98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.432+0000 7f5635f22740 1 -- 172.21.15.70:0/399106354 >> 172.21.15.70:0/399106354 conn(0x55b4962d4a00 msgr2=0x55b4962d4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.433+0000 7f5635f22740 1 -- 172.21.15.70:0/399106354 shutdown_connections 2024-04-01T01:48:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.433+0000 7f5635f22740 1 -- 172.21.15.70:0/399106354 wait complete. 2024-04-01T01:48:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.433+0000 7f5635f22740 1 -- start start 2024-04-01T01:48:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.433+0000 7f5635f22740 1 Processor -- start 2024-04-01T01:48:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.433+0000 7f5635f22740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d9490 0x55b4962f4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.433+0000 7f5635f22740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 0x55b4962f4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.434+0000 7f5635f22740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962deff0 0x55b4962fbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.434+0000 7f5635f22740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b4962e1f40 con 0x55b4962d9e80 2024-04-01T01:48:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.434+0000 7f5635f22740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55b4961dc470 con 0x55b4962deff0 2024-04-01T01:48:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.434+0000 7f5635f22740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b4962e20c0 con 0x55b4962d9490 2024-04-01T01:48:22.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.434+0000 7f5635f22740 10 client.0 _ll_register_callbacks cb 0x55b4961f9cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.434+0000 7f5630c38640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 0x55b4962f4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.434+0000 7f5630c38640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 0x55b4962f4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34854/0 (socket says 172.21.15.70:34854) 2024-04-01T01:48:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.434+0000 7f5630c38640 1 -- 172.21.15.70:0/3992855264 learned_addr learned my addr 172.21.15.70:0/3992855264 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.435+0000 7f5630c38640 1 -- 172.21.15.70:0/3992855264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d9490 msgr2=0x55b4962f4980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:48:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.435+0000 7f5630c38640 1 --2- 172.21.15.70:0/3992855264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d9490 0x55b4962f4980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.435+0000 7f5630c38640 1 -- 172.21.15.70:0/3992855264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962deff0 msgr2=0x55b4962fbf00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:48:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.435+0000 7f5630c38640 1 --2- 172.21.15.70:0/3992855264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962deff0 0x55b4962fbf00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.435+0000 7f5630c38640 1 -- 172.21.15.70:0/3992855264 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b495f523f0 con 0x55b4962d9e80 2024-04-01T01:48:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.435+0000 7f5630c38640 1 --2- 172.21.15.70:0/3992855264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 0x55b4962f4ec0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f5624006010 tx=0x7f56240336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b4fa4fa867333c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.436+0000 7f5614ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:48:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.436+0000 7f5614ff9640 1 -- 172.21.15.70:0/3992855264 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624043070 con 0x55b4962d9e80 2024-04-01T01:48:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.436+0000 7f5614ff9640 1 -- 172.21.15.70:0/3992855264 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5624003b80 con 0x55b4962d9e80 2024-04-01T01:48:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.436+0000 7f5635f22740 1 -- 172.21.15.70:0/3992855264 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f562809c4e0 con 0x55b4962d9e80 2024-04-01T01:48:22.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.436+0000 7f5614ff9640 1 -- 172.21.15.70:0/3992855264 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f562404d050 con 0x55b4962d9e80 2024-04-01T01:48:22.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.439+0000 7f5635f22740 10 client.19725 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:22.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.439+0000 7f5635f22740 1 -- 172.21.15.70:0/3992855264 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b49601c7d0 con 0x55b4962d9e80 2024-04-01T01:48:22.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.440+0000 7f5614ff9640 1 -- 172.21.15.70:0/3992855264 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f562809c4e0 con 0x55b4962d9e80 2024-04-01T01:48:22.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.440+0000 7f5614ff9640 1 -- 172.21.15.70:0/3992855264 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5624004350 con 0x55b4962d9e80 2024-04-01T01:48:22.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.440+0000 7f5635f22740 10 client.19725 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:22.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.440+0000 7f5635f22740 10 client.19725 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:48:22.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.440+0000 7f5635f22740 20 client.19725 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:22.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.440+0000 7f5635f22740 1 --2- 172.21.15.70:0/3992855264 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b4962e71b0 0x55b4962e9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.440+0000 7f5635f22740 4 client.19725 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:22.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.440+0000 7f5635f22740 1 -- 172.21.15.70:0/3992855264 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b49624b6f0 con 0x55b4962e71b0 2024-04-01T01:48:22.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.443+0000 7f5631439640 1 --2- 172.21.15.70:0/3992855264 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b4962e71b0 0x55b4962e9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.447+0000 7f5631439640 1 --2- 172.21.15.70:0/3992855264 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b4962e71b0 0x55b4962e9610 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e3319e0190362c1d server_cookie=d3bdebd7e4991e1f in_seq=0 out_seq=0 2024-04-01T01:48:22.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.447+0000 7f5614ff9640 10 client.19725 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:48:22.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.448+0000 7f5614ff9640 1 -- 172.21.15.70:0/3992855264 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b49624b6f0 con 0x55b4962e71b0 2024-04-01T01:48:22.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.448+0000 7f5614ff9640 10 client.19725 handle_command_reply: tid=0 2024-04-01T01:48:22.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.448+0000 7f5635f22740 1 client.19725 shutdown 2024-04-01T01:48:22.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.448+0000 7f5635f22740 1 -- 172.21.15.70:0/3992855264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 msgr2=0x55b4962f4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.448+0000 7f5635f22740 1 --2- 172.21.15.70:0/3992855264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 0x55b4962f4ec0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f5624006010 tx=0x7f56240336f0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.448+0000 7f5635f22740 1 -- 172.21.15.70:0/3992855264 shutdown_connections 2024-04-01T01:48:22.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.448+0000 7f5635f22740 1 --2- 172.21.15.70:0/3992855264 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b4962e71b0 0x55b4962e9610 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.448+0000 7f5635f22740 1 --2- 172.21.15.70:0/3992855264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962deff0 0x55b4962fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.448+0000 7f5635f22740 1 --2- 172.21.15.70:0/3992855264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 0x55b4962f4ec0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.448+0000 7f5635f22740 1 --2- 172.21.15.70:0/3992855264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d9490 0x55b4962f4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.448+0000 7f5635f22740 1 -- 172.21.15.70:0/3992855264 >> 172.21.15.70:0/3992855264 conn(0x55b4962d4a00 msgr2=0x55b4962d3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.448+0000 7f5614ff9640 0 client.19725 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:48:22.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.449+0000 7f5635f22740 1 -- 172.21.15.70:0/3992855264 shutdown_connections 2024-04-01T01:48:22.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.449+0000 7f5635f22740 1 -- 172.21.15.70:0/3992855264 wait complete. 2024-04-01T01:48:22.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.449+0000 7f5635f22740 20 client.19725 trim_cache size 0 max 16384 2024-04-01T01:48:22.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.468+0000 7f5635f22740 1 Processor -- start 2024-04-01T01:48:22.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.468+0000 7f5635f22740 1 -- start start 2024-04-01T01:48:22.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.468+0000 7f5635f22740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d4a00 0x55b4962e2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.468+0000 7f5635f22740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 0x55b4962fbec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.468+0000 7f5635f22740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 0x55b4962d34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.468+0000 7f5635f22740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f561000fda0 con 0x55b4962d9e80 2024-04-01T01:48:22.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.468+0000 7f5635f22740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f56100118e0 con 0x55b4962d9490 2024-04-01T01:48:22.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.468+0000 7f5635f22740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5610010470 con 0x55b4962d4a00 2024-04-01T01:48:22.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.468+0000 7f5630c38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 0x55b4962fbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.468+0000 7f5631439640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d4a00 0x55b4962e2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.468+0000 7f5630c38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 0x55b4962fbec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41282/0 (socket says 172.21.15.70:41282) 2024-04-01T01:48:22.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.468+0000 7f5631c3a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 0x55b4962d34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.468+0000 7f5630c38640 1 -- 172.21.15.70:0/851281952 learned_addr learned my addr 172.21.15.70:0/851281952 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.469+0000 7f5631c3a640 1 -- 172.21.15.70:0/851281952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d4a00 msgr2=0x55b4962e2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.469+0000 7f5631c3a640 1 --2- 172.21.15.70:0/851281952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d4a00 0x55b4962e2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.469+0000 7f5631c3a640 1 -- 172.21.15.70:0/851281952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 msgr2=0x55b4962fbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.469+0000 7f5631c3a640 1 --2- 172.21.15.70:0/851281952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 0x55b4962fbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.469+0000 7f5631c3a640 1 -- 172.21.15.70:0/851281952 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4962e9c90 con 0x55b4962d9e80 2024-04-01T01:48:22.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.469+0000 7f5630c38640 1 --2- 172.21.15.70:0/851281952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 0x55b4962fbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:22.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.469+0000 7f5631c3a640 1 --2- 172.21.15.70:0/851281952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 0x55b4962d34b0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x55b4962e1b70 tx=0x7f562809c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c7cd6441dddba server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.469+0000 7f5616ffd640 1 -- 172.21.15.70:0/851281952 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562809cbe0 con 0x55b4962d9e80 2024-04-01T01:48:22.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.469+0000 7f5616ffd640 1 -- 172.21.15.70:0/851281952 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f562809c4e0 con 0x55b4962d9e80 2024-04-01T01:48:22.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.469+0000 7f5616ffd640 1 -- 172.21.15.70:0/851281952 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56280046c0 con 0x55b4962d9e80 2024-04-01T01:48:22.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.470+0000 7f5635f22740 1 -- 172.21.15.70:0/851281952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 msgr2=0x55b4962d34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.470+0000 7f5635f22740 1 --2- 172.21.15.70:0/851281952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 0x55b4962d34b0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x55b4962e1b70 tx=0x7f562809c360 comp rx=0 tx=0).stop 2024-04-01T01:48:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.470+0000 7f5635f22740 1 -- 172.21.15.70:0/851281952 shutdown_connections 2024-04-01T01:48:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.470+0000 7f5635f22740 1 --2- 172.21.15.70:0/851281952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 0x55b4962d34b0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.470+0000 7f5635f22740 1 --2- 172.21.15.70:0/851281952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 0x55b4962fbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.470+0000 7f5635f22740 1 --2- 172.21.15.70:0/851281952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d4a00 0x55b4962e2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.470+0000 7f5635f22740 1 -- 172.21.15.70:0/851281952 >> 172.21.15.70:0/851281952 conn(0x55b4962dac60 msgr2=0x55b4962e2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.470+0000 7f5635f22740 1 -- 172.21.15.70:0/851281952 shutdown_connections 2024-04-01T01:48:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.470+0000 7f5635f22740 1 -- 172.21.15.70:0/851281952 wait complete. 2024-04-01T01:48:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.471+0000 7f5635f22740 1 -- start start 2024-04-01T01:48:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.471+0000 7f5635f22740 1 Processor -- start 2024-04-01T01:48:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.471+0000 7f5635f22740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d4a00 0x55b4962faf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.471+0000 7f5635f22740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 0x55b4962fb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.471+0000 7f5635f22740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 0x55b496376150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.471+0000 7f5635f22740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5610011210 con 0x55b4962d9e80 2024-04-01T01:48:22.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.471+0000 7f5635f22740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5610010b40 con 0x55b4962d9490 2024-04-01T01:48:22.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.471+0000 7f5635f22740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f561000dd80 con 0x55b4962d4a00 2024-04-01T01:48:22.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.471+0000 7f5635f22740 10 client.0 _ll_register_callbacks cb 0x7f5624033f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:22.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.471+0000 7f5631439640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d4a00 0x55b4962faf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.471+0000 7f5630c38640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 0x55b4962fb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.471+0000 7f5631c3a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 0x55b496376150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.471+0000 7f5631439640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d4a00 0x55b4962faf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42136/0 (socket says 172.21.15.70:42136) 2024-04-01T01:48:22.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.471+0000 7f5631439640 1 -- 172.21.15.70:0/2380627720 learned_addr learned my addr 172.21.15.70:0/2380627720 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.471+0000 7f5631c3a640 1 -- 172.21.15.70:0/2380627720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d4a00 msgr2=0x55b4962faf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.471+0000 7f5631c3a640 1 --2- 172.21.15.70:0/2380627720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d4a00 0x55b4962faf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5631c3a640 1 -- 172.21.15.70:0/2380627720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 msgr2=0x55b4962fb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5631c3a640 1 --2- 172.21.15.70:0/2380627720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 0x55b4962fb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5631c3a640 1 -- 172.21.15.70:0/2380627720 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b496376710 con 0x55b4962d9e80 2024-04-01T01:48:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5630c38640 1 --2- 172.21.15.70:0/2380627720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 0x55b4962fb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5631439640 1 --2- 172.21.15.70:0/2380627720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d4a00 0x55b4962faf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5631c3a640 1 --2- 172.21.15.70:0/2380627720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 0x55b496376150 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f56280981c0 tx=0x7f5628098890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c7da43d075e161a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5614ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:48:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5614ff9640 1 -- 172.21.15.70:0/2380627720 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56280a4ef0 con 0x55b4962d9e80 2024-04-01T01:48:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5614ff9640 1 -- 172.21.15.70:0/2380627720 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f56280a0170 con 0x55b4962d9e80 2024-04-01T01:48:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5635f22740 1 -- 172.21.15.70:0/2380627720 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f562809c4e0 con 0x55b4962d9e80 2024-04-01T01:48:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5614ff9640 1 -- 172.21.15.70:0/2380627720 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f562809cbe0 con 0x55b4962d9e80 2024-04-01T01:48:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5635f22740 10 client.19731 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5635f22740 1 -- 172.21.15.70:0/2380627720 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b496377a10 con 0x55b4962d9e80 2024-04-01T01:48:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5614ff9640 1 -- 172.21.15.70:0/2380627720 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f562809c4e0 con 0x55b4962d9e80 2024-04-01T01:48:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5614ff9640 1 -- 172.21.15.70:0/2380627720 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f5628098230 con 0x55b4962d9e80 2024-04-01T01:48:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5635f22740 10 client.19731 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:22.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5635f22740 10 client.19731 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:48:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5635f22740 20 client.19731 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5635f22740 1 --2- 172.21.15.70:0/2380627720 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b4962deff0 0x55b496379e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5635f22740 4 client.19731 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:48:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.472+0000 7f5635f22740 1 -- 172.21.15.70:0/2380627720 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b496245a40 con 0x55b4962deff0 2024-04-01T01:48:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.473+0000 7f5631439640 1 --2- 172.21.15.70:0/2380627720 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b4962deff0 0x55b496379e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.473+0000 7f5631439640 1 --2- 172.21.15.70:0/2380627720 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b4962deff0 0x55b496379e10 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=de05160047be11df server_cookie=9428e4c840e71efa in_seq=0 out_seq=0 2024-04-01T01:48:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.473+0000 7f5614ff9640 10 client.19731 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:48:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.473+0000 7f5614ff9640 1 -- 172.21.15.70:0/2380627720 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b496245a40 con 0x55b4962deff0 2024-04-01T01:48:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.473+0000 7f5614ff9640 10 client.19731 handle_command_reply: tid=0 2024-04-01T01:48:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.473+0000 7f5635f22740 1 client.19731 shutdown 2024-04-01T01:48:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.474+0000 7f5635f22740 1 -- 172.21.15.70:0/2380627720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 msgr2=0x55b496376150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.474+0000 7f5635f22740 1 --2- 172.21.15.70:0/2380627720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 0x55b496376150 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f56280981c0 tx=0x7f5628098890 comp rx=0 tx=0).stop 2024-04-01T01:48:22.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.474+0000 7f5635f22740 1 -- 172.21.15.70:0/2380627720 shutdown_connections 2024-04-01T01:48:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.474+0000 7f5635f22740 1 --2- 172.21.15.70:0/2380627720 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55b4962deff0 0x55b496379e10 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.474+0000 7f5635f22740 1 --2- 172.21.15.70:0/2380627720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b4962d9e80 0x55b496376150 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.474+0000 7f5614ff9640 0 client.19731 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:48:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.474+0000 7f5635f22740 1 --2- 172.21.15.70:0/2380627720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b4962d9490 0x55b4962fb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.474+0000 7f5635f22740 1 --2- 172.21.15.70:0/2380627720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4962d4a00 0x55b4962faf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.474+0000 7f5635f22740 1 -- 172.21.15.70:0/2380627720 >> 172.21.15.70:0/2380627720 conn(0x55b4962dac60 msgr2=0x55b4962d34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.474+0000 7f5635f22740 1 -- 172.21.15.70:0/2380627720 shutdown_connections 2024-04-01T01:48:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.474+0000 7f5635f22740 1 -- 172.21.15.70:0/2380627720 wait complete. 2024-04-01T01:48:22.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.474+0000 7f5635f22740 20 client.19731 trim_cache size 0 max 16384 2024-04-01T01:48:22.479 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:48:22.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.474+0000 7f563369d640 1 -- 172.21.15.70:0/3936443765 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56180a2670 msgr2=0x7f56180a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.474+0000 7f563369d640 1 --2- 172.21.15.70:0/3936443765 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56180a2670 0x7f56180a4b30 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f561c006fd0 tx=0x7f561c0090e0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.474+0000 7f563369d640 1 -- 172.21.15.70:0/3936443765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f562c10da70 msgr2=0x7f562c1ade20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.474+0000 7f563369d640 1 --2- 172.21.15.70:0/3936443765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f562c10da70 0x7f562c1ade20 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f562800e940 tx=0x7f562800ee00 comp rx=0 tx=0).stop 2024-04-01T01:48:22.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.475+0000 7f563369d640 1 -- 172.21.15.70:0/3936443765 shutdown_connections 2024-04-01T01:48:22.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.475+0000 7f563369d640 1 --2- 172.21.15.70:0/3936443765 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f56180a2670 0x7f56180a4b30 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.475+0000 7f563369d640 1 --2- 172.21.15.70:0/3936443765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f562c10da70 0x7f562c1ade20 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.475+0000 7f563369d640 1 --2- 172.21.15.70:0/3936443765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f562c10b0c0 0x7f562c1a6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.475+0000 7f563369d640 1 --2- 172.21.15.70:0/3936443765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f562c108750 0x7f562c1a6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.475+0000 7f563369d640 1 -- 172.21.15.70:0/3936443765 >> 172.21.15.70:0/3936443765 conn(0x7f562c06e5b0 msgr2=0x7f562c109a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.475+0000 7f563369d640 1 -- 172.21.15.70:0/3936443765 shutdown_connections 2024-04-01T01:48:22.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.475+0000 7f563369d640 1 -- 172.21.15.70:0/3936443765 wait complete. 2024-04-01T01:48:22.486 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:48:22.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.623+0000 7f933dc2e640 1 -- 172.21.15.70:0/655610757 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93381021d0 msgr2=0x7f93381045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.623+0000 7f933dc2e640 1 --2- 172.21.15.70:0/655610757 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93381021d0 0x7f93381045c0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f9328009900 tx=0x7f932802bc30 comp rx=0 tx=0).stop 2024-04-01T01:48:22.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.623+0000 7f933dc2e640 1 -- 172.21.15.70:0/655610757 shutdown_connections 2024-04-01T01:48:22.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.623+0000 7f933dc2e640 1 --2- 172.21.15.70:0/655610757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9338107430 0x7f9338109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.623+0000 7f933dc2e640 1 --2- 172.21.15.70:0/655610757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9338104b00 0x7f9338106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.623+0000 7f933dc2e640 1 --2- 172.21.15.70:0/655610757 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f93381021d0 0x7f93381045c0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.623+0000 7f933dc2e640 1 -- 172.21.15.70:0/655610757 >> 172.21.15.70:0/655610757 conn(0x7f93380fbbc0 msgr2=0x7f93380fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.623+0000 7f933dc2e640 1 -- 172.21.15.70:0/655610757 shutdown_connections 2024-04-01T01:48:22.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.623+0000 7f933dc2e640 1 -- 172.21.15.70:0/655610757 wait complete. 2024-04-01T01:48:22.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.624+0000 7f933dc2e640 1 Processor -- start 2024-04-01T01:48:22.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.624+0000 7f933dc2e640 1 -- start start 2024-04-01T01:48:22.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.624+0000 7f933dc2e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93381021d0 0x7f93381957d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.624+0000 7f933dc2e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9338104b00 0x7f9338195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.624+0000 7f933dc2e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9338107430 0x7f93381a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.624+0000 7f933dc2e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f933810d2d0 con 0x7f9338107430 2024-04-01T01:48:22.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.624+0000 7f933dc2e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f933810d150 con 0x7f9338104b00 2024-04-01T01:48:22.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.624+0000 7f933dc2e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f933810d450 con 0x7f93381021d0 2024-04-01T01:48:22.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.624+0000 7f9336ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9338104b00 0x7f9338195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.624+0000 7f9336ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9338104b00 0x7f9338195d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41328/0 (socket says 172.21.15.70:41328) 2024-04-01T01:48:22.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.624+0000 7f9336ffd640 1 -- 172.21.15.70:0/1630061447 learned_addr learned my addr 172.21.15.70:0/1630061447 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.624+0000 7f9337fff640 1 --2- 172.21.15.70:0/1630061447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9338107430 0x7f93381a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.625+0000 7f93377fe640 1 --2- 172.21.15.70:0/1630061447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93381021d0 0x7f93381957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.625+0000 7f9336ffd640 1 -- 172.21.15.70:0/1630061447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93381021d0 msgr2=0x7f93381957d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.625+0000 7f9336ffd640 1 --2- 172.21.15.70:0/1630061447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93381021d0 0x7f93381957d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.625+0000 7f9336ffd640 1 -- 172.21.15.70:0/1630061447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9338107430 msgr2=0x7f93381a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.625+0000 7f9336ffd640 1 --2- 172.21.15.70:0/1630061447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9338107430 0x7f93381a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.625+0000 7f9336ffd640 1 -- 172.21.15.70:0/1630061447 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93381a5c70 con 0x7f9338104b00 2024-04-01T01:48:22.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.625+0000 7f9336ffd640 1 --2- 172.21.15.70:0/1630061447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9338104b00 0x7f9338195d10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f932c005fd0 tx=0x7f932c00dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.625+0000 7f9334ff9640 1 -- 172.21.15.70:0/1630061447 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932c017070 con 0x7f9338104b00 2024-04-01T01:48:22.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.626+0000 7f9334ff9640 1 -- 172.21.15.70:0/1630061447 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f932c009c30 con 0x7f9338104b00 2024-04-01T01:48:22.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.626+0000 7f933dc2e640 1 -- 172.21.15.70:0/1630061447 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93381a5f00 con 0x7f9338104b00 2024-04-01T01:48:22.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.626+0000 7f9334ff9640 1 -- 172.21.15.70:0/1630061447 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932c003d40 con 0x7f9338104b00 2024-04-01T01:48:22.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.626+0000 7f933dc2e640 1 -- 172.21.15.70:0/1630061447 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93381a6410 con 0x7f9338104b00 2024-04-01T01:48:22.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.628+0000 7f9334ff9640 1 -- 172.21.15.70:0/1630061447 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f932c003ee0 con 0x7f9338104b00 2024-04-01T01:48:22.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.629+0000 7f9340812740 1 Processor -- start 2024-04-01T01:48:22.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.629+0000 7f9334ff9640 1 --2- 172.21.15.70:0/1630061447 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93200a25a0 0x7f93200a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.630+0000 7f9340812740 1 -- start start 2024-04-01T01:48:22.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.630+0000 7f9334ff9640 1 -- 172.21.15.70:0/1630061447 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f932c0ce5f0 con 0x7f9338104b00 2024-04-01T01:48:22.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.630+0000 7f9340812740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cf490 0x55ff7b9cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.630+0000 7f9337fff640 1 --2- 172.21.15.70:0/1630061447 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93200a25a0 0x7f93200a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.630+0000 7f9340812740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9d4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.630+0000 7f93377fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cf490 0x55ff7b9cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.630+0000 7f93377fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cf490 0x55ff7b9cf8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34902/0 (socket says 172.21.15.70:34902) 2024-04-01T01:48:22.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.630+0000 7f93377fe640 1 -- 172.21.15.70:0/2982725347 learned_addr learned my addr 172.21.15.70:0/2982725347 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.630+0000 7f9336ffd640 1 --2- 172.21.15.70:0/2982725347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9d4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.630+0000 7f9340812740 1 --2- 172.21.15.70:0/2982725347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9d7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.630+0000 7f9340812740 1 -- 172.21.15.70:0/2982725347 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ff7b9341b0 con 0x55ff7b9cf490 2024-04-01T01:48:22.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.630+0000 7f9340812740 1 -- 172.21.15.70:0/2982725347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ff7b8dcc00 con 0x55ff7b9cfe80 2024-04-01T01:48:22.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.630+0000 7f9340812740 1 -- 172.21.15.70:0/2982725347 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ff7b930250 con 0x55ff7b9d4ff0 2024-04-01T01:48:22.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.630+0000 7f9337fff640 1 --2- 172.21.15.70:0/1630061447 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93200a25a0 0x7f93200a4a60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9330005fd0 tx=0x7f9330009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.631+0000 7f9337fff640 1 --2- 172.21.15.70:0/2982725347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9d7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.631+0000 7f93377fe640 1 -- 172.21.15.70:0/2982725347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9d4ff0 msgr2=0x55ff7b9d7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.631+0000 7f93377fe640 1 --2- 172.21.15.70:0/2982725347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9d7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.631+0000 7f93377fe640 1 -- 172.21.15.70:0/2982725347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 msgr2=0x55ff7b9d4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.631+0000 7f93377fe640 1 --2- 172.21.15.70:0/2982725347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9d4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.631+0000 7f93377fe640 1 -- 172.21.15.70:0/2982725347 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff7b8fc3d0 con 0x55ff7b9cf490 2024-04-01T01:48:22.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.631+0000 7f9336ffd640 1 --2- 172.21.15.70:0/2982725347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9d4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:22.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.631+0000 7f9337fff640 1 --2- 172.21.15.70:0/2982725347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9d7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:48:22.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.631+0000 7f93377fe640 1 --2- 172.21.15.70:0/2982725347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cf490 0x55ff7b9cf8b0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f9328005ab0 tx=0x7f9328009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e6901c2864501b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.631+0000 7f931dffb640 1 -- 172.21.15.70:0/2982725347 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9328045020 con 0x55ff7b9cf490 2024-04-01T01:48:22.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.631+0000 7f931dffb640 1 -- 172.21.15.70:0/2982725347 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f932802ca20 con 0x55ff7b9cf490 2024-04-01T01:48:22.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.631+0000 7f931dffb640 1 -- 172.21.15.70:0/2982725347 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93280046c0 con 0x55ff7b9cf490 2024-04-01T01:48:22.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.632+0000 7f9340812740 1 -- 172.21.15.70:0/2982725347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cf490 msgr2=0x55ff7b9cf8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.632+0000 7f9340812740 1 --2- 172.21.15.70:0/2982725347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cf490 0x55ff7b9cf8b0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f9328005ab0 tx=0x7f9328009560 comp rx=0 tx=0).stop 2024-04-01T01:48:22.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.632+0000 7f9340812740 1 -- 172.21.15.70:0/2982725347 shutdown_connections 2024-04-01T01:48:22.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.632+0000 7f9340812740 1 --2- 172.21.15.70:0/2982725347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9d7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.632+0000 7f9340812740 1 --2- 172.21.15.70:0/2982725347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9d4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.632+0000 7f9340812740 1 --2- 172.21.15.70:0/2982725347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cf490 0x55ff7b9cf8b0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.632+0000 7f9340812740 1 -- 172.21.15.70:0/2982725347 >> 172.21.15.70:0/2982725347 conn(0x55ff7b9caa00 msgr2=0x55ff7b9cae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.632+0000 7f9340812740 1 -- 172.21.15.70:0/2982725347 shutdown_connections 2024-04-01T01:48:22.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.632+0000 7f9340812740 1 -- 172.21.15.70:0/2982725347 wait complete. 2024-04-01T01:48:22.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.632+0000 7f9340812740 1 -- start start 2024-04-01T01:48:22.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f9340812740 1 Processor -- start 2024-04-01T01:48:22.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f9340812740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cf490 0x55ff7b9d98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f9340812740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9d9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f9340812740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9e0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f9340812740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ff7b9d7f40 con 0x55ff7b9cfe80 2024-04-01T01:48:22.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f9340812740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ff7b8d2470 con 0x55ff7b9cf490 2024-04-01T01:48:22.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f9340812740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ff7b9d80c0 con 0x55ff7b9d4ff0 2024-04-01T01:48:22.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f9340812740 10 client.0 _ll_register_callbacks cb 0x55ff7b8efcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:22.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f9336ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9d9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f9336ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9d9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34904/0 (socket says 172.21.15.70:34904) 2024-04-01T01:48:22.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f93377fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cf490 0x55ff7b9d98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f9336ffd640 1 -- 172.21.15.70:0/3232279542 learned_addr learned my addr 172.21.15.70:0/3232279542 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f9337fff640 1 --2- 172.21.15.70:0/3232279542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9e0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f93377fe640 1 -- 172.21.15.70:0/3232279542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9d4ff0 msgr2=0x55ff7b9e0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f93377fe640 1 --2- 172.21.15.70:0/3232279542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9e0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f93377fe640 1 -- 172.21.15.70:0/3232279542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cfe80 msgr2=0x55ff7b9d9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f93377fe640 1 --2- 172.21.15.70:0/3232279542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9d9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f93377fe640 1 -- 172.21.15.70:0/3232279542 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff7b6483f0 con 0x55ff7b9cf490 2024-04-01T01:48:22.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.633+0000 7f9336ffd640 1 --2- 172.21.15.70:0/3232279542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9d9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:22.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.634+0000 7f93377fe640 1 --2- 172.21.15.70:0/3232279542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cf490 0x55ff7b9d98f0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f9328006010 tx=0x7f93280040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=556f402f2001f8d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.634+0000 7f9316ffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:22.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.634+0000 7f9316ffd640 1 -- 172.21.15.70:0/3232279542 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9328050070 con 0x55ff7b9cf490 2024-04-01T01:48:22.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.634+0000 7f9316ffd640 1 -- 172.21.15.70:0/3232279542 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f93280037f0 con 0x55ff7b9cf490 2024-04-01T01:48:22.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.634+0000 7f9340812740 1 -- 172.21.15.70:0/3232279542 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f932802ca20 con 0x55ff7b9cf490 2024-04-01T01:48:22.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.634+0000 7f9316ffd640 1 -- 172.21.15.70:0/3232279542 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9328059050 con 0x55ff7b9cf490 2024-04-01T01:48:22.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.634+0000 7f9340812740 10 client.29837 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:22.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.634+0000 7f9340812740 1 -- 172.21.15.70:0/3232279542 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ff7b7127d0 con 0x55ff7b9cf490 2024-04-01T01:48:22.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.634+0000 7f9316ffd640 1 -- 172.21.15.70:0/3232279542 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f932802ca20 con 0x55ff7b9cf490 2024-04-01T01:48:22.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.634+0000 7f9316ffd640 1 -- 172.21.15.70:0/3232279542 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f9328053020 con 0x55ff7b9cf490 2024-04-01T01:48:22.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.634+0000 7f9340812740 10 client.29837 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:22.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.634+0000 7f9340812740 10 client.29837 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:48:22.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.635+0000 7f9340812740 20 client.29837 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:22.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.635+0000 7f9340812740 1 --2- 172.21.15.70:0/3232279542 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55ff7b9e4f40 0x55ff7b9e7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.635+0000 7f9340812740 4 client.29837 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:22.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.635+0000 7f9340812740 1 -- 172.21.15.70:0/3232279542 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff7b9416f0 con 0x55ff7b9e4f40 2024-04-01T01:48:22.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.635+0000 7f9336ffd640 1 --2- 172.21.15.70:0/3232279542 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55ff7b9e4f40 0x55ff7b9e7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.635+0000 7f9316ffd640 10 client.29837 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:48:22.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.635+0000 7f9336ffd640 1 --2- 172.21.15.70:0/3232279542 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55ff7b9e4f40 0x55ff7b9e7330 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f4620763ce221bb8 server_cookie=1ca2531f577584b in_seq=0 out_seq=0 2024-04-01T01:48:22.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.636+0000 7f9316ffd640 1 -- 172.21.15.70:0/3232279542 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ff7b9416f0 con 0x55ff7b9e4f40 2024-04-01T01:48:22.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.636+0000 7f9316ffd640 10 client.29837 handle_command_reply: tid=0 2024-04-01T01:48:22.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.636+0000 7f9340812740 1 client.29837 shutdown 2024-04-01T01:48:22.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.637+0000 7f9340812740 1 -- 172.21.15.70:0/3232279542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cf490 msgr2=0x55ff7b9d98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.637+0000 7f9340812740 1 --2- 172.21.15.70:0/3232279542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cf490 0x55ff7b9d98f0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f9328006010 tx=0x7f93280040c0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.637+0000 7f9340812740 1 -- 172.21.15.70:0/3232279542 shutdown_connections 2024-04-01T01:48:22.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.637+0000 7f9340812740 1 --2- 172.21.15.70:0/3232279542 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55ff7b9e4f40 0x55ff7b9e7330 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.637+0000 7f9316ffd640 0 client.29837 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:48:22.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.637+0000 7f9340812740 1 --2- 172.21.15.70:0/3232279542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9e0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.637+0000 7f9340812740 1 --2- 172.21.15.70:0/3232279542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9d9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.637+0000 7f9340812740 1 --2- 172.21.15.70:0/3232279542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cf490 0x55ff7b9d98f0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.637+0000 7f9340812740 1 -- 172.21.15.70:0/3232279542 >> 172.21.15.70:0/3232279542 conn(0x55ff7b9caa00 msgr2=0x55ff7b9c9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.637+0000 7f9340812740 1 -- 172.21.15.70:0/3232279542 shutdown_connections 2024-04-01T01:48:22.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.637+0000 7f9340812740 1 -- 172.21.15.70:0/3232279542 wait complete. 2024-04-01T01:48:22.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.637+0000 7f9340812740 20 client.29837 trim_cache size 0 max 16384 2024-04-01T01:48:22.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.656+0000 7f9340812740 1 Processor -- start 2024-04-01T01:48:22.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9340812740 1 -- start start 2024-04-01T01:48:22.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9340812740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9cf490 0x55ff7b9c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9340812740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9e0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9340812740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9ecc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9340812740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f932400fcc0 con 0x55ff7b9d4ff0 2024-04-01T01:48:22.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9340812740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9324011800 con 0x55ff7b9cfe80 2024-04-01T01:48:22.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9340812740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9324010390 con 0x55ff7b9cf490 2024-04-01T01:48:22.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9337fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9ecc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9337fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9ecc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34906/0 (socket says 172.21.15.70:34906) 2024-04-01T01:48:22.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9337fff640 1 -- 172.21.15.70:0/1578192501 learned_addr learned my addr 172.21.15.70:0/1578192501 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9336ffd640 1 --2- 172.21.15.70:0/1578192501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9e0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f93377fe640 1 --2- 172.21.15.70:0/1578192501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9cf490 0x55ff7b9c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9337fff640 1 -- 172.21.15.70:0/1578192501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9cf490 msgr2=0x55ff7b9c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9337fff640 1 --2- 172.21.15.70:0/1578192501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9cf490 0x55ff7b9c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9337fff640 1 -- 172.21.15.70:0/1578192501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 msgr2=0x55ff7b9e0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9337fff640 1 --2- 172.21.15.70:0/1578192501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9e0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9337fff640 1 -- 172.21.15.70:0/1578192501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff7b9d0260 con 0x55ff7b9d4ff0 2024-04-01T01:48:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9336ffd640 1 --2- 172.21.15.70:0/1578192501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9e0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.657+0000 7f9337fff640 1 --2- 172.21.15.70:0/1578192501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9ecc00 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f9330005f40 tx=0x7f9330014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07b14c3bf37b2ff server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.658+0000 7f931d7fa640 1 -- 172.21.15.70:0/1578192501 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9330004d20 con 0x55ff7b9d4ff0 2024-04-01T01:48:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.658+0000 7f931d7fa640 1 -- 172.21.15.70:0/1578192501 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9330003e10 con 0x55ff7b9d4ff0 2024-04-01T01:48:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.658+0000 7f931d7fa640 1 -- 172.21.15.70:0/1578192501 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933001e790 con 0x55ff7b9d4ff0 2024-04-01T01:48:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.658+0000 7f9340812740 1 -- 172.21.15.70:0/1578192501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9d4ff0 msgr2=0x55ff7b9ecc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.658+0000 7f9340812740 1 --2- 172.21.15.70:0/1578192501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9ecc00 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f9330005f40 tx=0x7f9330014000 comp rx=0 tx=0).stop 2024-04-01T01:48:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.658+0000 7f9340812740 1 -- 172.21.15.70:0/1578192501 shutdown_connections 2024-04-01T01:48:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.658+0000 7f9340812740 1 --2- 172.21.15.70:0/1578192501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9ecc00 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.658+0000 7f9340812740 1 --2- 172.21.15.70:0/1578192501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9e0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.658+0000 7f9340812740 1 --2- 172.21.15.70:0/1578192501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9cf490 0x55ff7b9c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.658+0000 7f9340812740 1 -- 172.21.15.70:0/1578192501 >> 172.21.15.70:0/1578192501 conn(0x55ff7b9caa00 msgr2=0x55ff7b9c94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.659+0000 7f9340812740 1 -- 172.21.15.70:0/1578192501 shutdown_connections 2024-04-01T01:48:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.659+0000 7f9340812740 1 -- 172.21.15.70:0/1578192501 wait complete. 2024-04-01T01:48:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.659+0000 7f9340812740 1 -- start start 2024-04-01T01:48:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.659+0000 7f9340812740 1 Processor -- start 2024-04-01T01:48:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.659+0000 7f9340812740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cf490 0x55ff7b9de810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.659+0000 7f9340812740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9ded50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.659+0000 7f9340812740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9dfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.659+0000 7f9340812740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9324011130 con 0x55ff7b9cf490 2024-04-01T01:48:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.659+0000 7f9340812740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9324010a60 con 0x55ff7b9cfe80 2024-04-01T01:48:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.659+0000 7f9340812740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f932400dca0 con 0x55ff7b9d4ff0 2024-04-01T01:48:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.659+0000 7f9340812740 10 client.0 _ll_register_callbacks cb 0x7f9328005870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.659+0000 7f93377fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cf490 0x55ff7b9de810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.659+0000 7f93377fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cf490 0x55ff7b9de810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34920/0 (socket says 172.21.15.70:34920) 2024-04-01T01:48:22.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.659+0000 7f93377fe640 1 -- 172.21.15.70:0/1343070186 learned_addr learned my addr 172.21.15.70:0/1343070186 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.659+0000 7f9336ffd640 1 --2- 172.21.15.70:0/1343070186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9ded50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.660+0000 7f9337fff640 1 --2- 172.21.15.70:0/1343070186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9dfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.660+0000 7f93377fe640 1 -- 172.21.15.70:0/1343070186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9d4ff0 msgr2=0x55ff7b9dfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.660+0000 7f93377fe640 1 --2- 172.21.15.70:0/1343070186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9dfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.660+0000 7f93377fe640 1 -- 172.21.15.70:0/1343070186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 msgr2=0x55ff7b9ded50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.660+0000 7f93377fe640 1 --2- 172.21.15.70:0/1343070186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9ded50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.660+0000 7f93377fe640 1 -- 172.21.15.70:0/1343070186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff7b9e0350 con 0x55ff7b9cf490 2024-04-01T01:48:22.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.660+0000 7f93377fe640 1 --2- 172.21.15.70:0/1343070186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cf490 0x55ff7b9de810 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f9328041b80 tx=0x7f932802caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2772dbb2ada084dc server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.660+0000 7f9316ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:48:22.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.660+0000 7f9316ffd640 1 -- 172.21.15.70:0/1343070186 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9328050040 con 0x55ff7b9cf490 2024-04-01T01:48:22.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.660+0000 7f9316ffd640 1 -- 172.21.15.70:0/1343070186 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f9328041690 con 0x55ff7b9cf490 2024-04-01T01:48:22.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.660+0000 7f9340812740 1 -- 172.21.15.70:0/1343070186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9330003e10 con 0x55ff7b9cf490 2024-04-01T01:48:22.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.660+0000 7f9316ffd640 1 -- 172.21.15.70:0/1343070186 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f932806a050 con 0x55ff7b9cf490 2024-04-01T01:48:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.660+0000 7f9340812740 10 client.19749 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.660+0000 7f9340812740 1 -- 172.21.15.70:0/1343070186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ff7ba4b530 con 0x55ff7b9cf490 2024-04-01T01:48:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.661+0000 7f9316ffd640 1 -- 172.21.15.70:0/1343070186 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9330003e10 con 0x55ff7b9cf490 2024-04-01T01:48:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.661+0000 7f9316ffd640 1 -- 172.21.15.70:0/1343070186 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f9328053040 con 0x55ff7b9cf490 2024-04-01T01:48:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.661+0000 7f9340812740 10 client.19749 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.661+0000 7f9340812740 10 client.19749 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:48:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.661+0000 7f9340812740 20 client.19749 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.661+0000 7f9340812740 1 --2- 172.21.15.70:0/1343070186 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55ff7ba4bf50 0x55ff7ba4e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.661+0000 7f9340812740 4 client.19749 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:48:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.661+0000 7f9340812740 1 -- 172.21.15.70:0/1343070186 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ff7b93ba40 con 0x55ff7ba4bf50 2024-04-01T01:48:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.661+0000 7f9336ffd640 1 --2- 172.21.15.70:0/1343070186 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55ff7ba4bf50 0x55ff7ba4e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.661+0000 7f9316ffd640 10 client.19749 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:48:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.661+0000 7f9336ffd640 1 --2- 172.21.15.70:0/1343070186 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55ff7ba4bf50 0x55ff7ba4e370 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=952af86a1b8c7131 server_cookie=90c3b714ff450b1b in_seq=0 out_seq=0 2024-04-01T01:48:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.662+0000 7f9316ffd640 1 -- 172.21.15.70:0/1343070186 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ff7b93ba40 con 0x55ff7ba4bf50 2024-04-01T01:48:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.662+0000 7f9316ffd640 10 client.19749 handle_command_reply: tid=0 2024-04-01T01:48:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.662+0000 7f9340812740 1 client.19749 shutdown 2024-04-01T01:48:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.662+0000 7f9340812740 1 -- 172.21.15.70:0/1343070186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cf490 msgr2=0x55ff7b9de810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.662+0000 7f9340812740 1 --2- 172.21.15.70:0/1343070186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cf490 0x55ff7b9de810 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f9328041b80 tx=0x7f932802caf0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.662+0000 7f9340812740 1 -- 172.21.15.70:0/1343070186 shutdown_connections 2024-04-01T01:48:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.662+0000 7f9340812740 1 --2- 172.21.15.70:0/1343070186 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55ff7ba4bf50 0x55ff7ba4e370 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.662+0000 7f9340812740 1 --2- 172.21.15.70:0/1343070186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ff7b9d4ff0 0x55ff7b9dfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.662+0000 7f9316ffd640 0 client.19749 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:48:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.662+0000 7f9340812740 1 --2- 172.21.15.70:0/1343070186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ff7b9cfe80 0x55ff7b9ded50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.662+0000 7f9340812740 1 --2- 172.21.15.70:0/1343070186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ff7b9cf490 0x55ff7b9de810 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.662+0000 7f9340812740 1 -- 172.21.15.70:0/1343070186 >> 172.21.15.70:0/1343070186 conn(0x55ff7b9caa00 msgr2=0x55ff7b9c9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.662+0000 7f9340812740 1 -- 172.21.15.70:0/1343070186 shutdown_connections 2024-04-01T01:48:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.662+0000 7f9340812740 1 -- 172.21.15.70:0/1343070186 wait complete. 2024-04-01T01:48:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.662+0000 7f9340812740 20 client.19749 trim_cache size 0 max 16384 2024-04-01T01:48:22.669 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:48:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.663+0000 7f933dc2e640 1 -- 172.21.15.70:0/1630061447 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93200a25a0 msgr2=0x7f93200a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.663+0000 7f933dc2e640 1 --2- 172.21.15.70:0/1630061447 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93200a25a0 0x7f93200a4a60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9330005fd0 tx=0x7f9330009500 comp rx=0 tx=0).stop 2024-04-01T01:48:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.663+0000 7f933dc2e640 1 -- 172.21.15.70:0/1630061447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9338104b00 msgr2=0x7f9338195d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.663+0000 7f933dc2e640 1 --2- 172.21.15.70:0/1630061447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9338104b00 0x7f9338195d10 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f932c005fd0 tx=0x7f932c00dde0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.663+0000 7f933dc2e640 1 -- 172.21.15.70:0/1630061447 shutdown_connections 2024-04-01T01:48:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.663+0000 7f933dc2e640 1 --2- 172.21.15.70:0/1630061447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9338107430 0x7f93381a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.663+0000 7f933dc2e640 1 --2- 172.21.15.70:0/1630061447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9338104b00 0x7f9338195d10 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.663+0000 7f933dc2e640 1 --2- 172.21.15.70:0/1630061447 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f93200a25a0 0x7f93200a4a60 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.663+0000 7f933dc2e640 1 --2- 172.21.15.70:0/1630061447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93381021d0 0x7f93381957d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.663+0000 7f933dc2e640 1 -- 172.21.15.70:0/1630061447 >> 172.21.15.70:0/1630061447 conn(0x7f93380fbbc0 msgr2=0x7f9338101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.663+0000 7f933dc2e640 1 -- 172.21.15.70:0/1630061447 shutdown_connections 2024-04-01T01:48:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.663+0000 7f933dc2e640 1 -- 172.21.15.70:0/1630061447 wait complete. 2024-04-01T01:48:22.675 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:48:22.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:22 smithi070 ceph-mon[32179]: pgmap v805: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 13 MiB/s wr, 1.02k op/s 2024-04-01T01:48:22.754 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:22 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/160937287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:48:22.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 -- 172.21.15.70:0/103581800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60c8102970 msgr2=0x7f60c8102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 --2- 172.21.15.70:0/103581800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60c8102970 0x7f60c8102df0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f60bc009900 tx=0x7f60bc02bc30 comp rx=0 tx=0).stop 2024-04-01T01:48:22.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 -- 172.21.15.70:0/103581800 shutdown_connections 2024-04-01T01:48:22.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 --2- 172.21.15.70:0/103581800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60c8103330 0x7f60c8109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 --2- 172.21.15.70:0/103581800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60c8102970 0x7f60c8102df0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 --2- 172.21.15.70:0/103581800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c8101780 0x7f60c8101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 -- 172.21.15.70:0/103581800 >> 172.21.15.70:0/103581800 conn(0x7f60c80fccf0 msgr2=0x7f60c80ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 -- 172.21.15.70:0/103581800 shutdown_connections 2024-04-01T01:48:22.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 -- 172.21.15.70:0/103581800 wait complete. 2024-04-01T01:48:22.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 Processor -- start 2024-04-01T01:48:22.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 -- start start 2024-04-01T01:48:22.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c8101780 0x7f60c8197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60c8102970 0x7f60c8197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60c8103330 0x7f60c819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f60c810d2b0 con 0x7f60c8103330 2024-04-01T01:48:22.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f60c810d130 con 0x7f60c8102970 2024-04-01T01:48:22.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.792+0000 7f60cfe9d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f60c810d430 con 0x7f60c8101780 2024-04-01T01:48:22.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.793+0000 7f60ce43a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60c8103330 0x7f60c819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.793+0000 7f60cd438640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60c8102970 0x7f60c8197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.793+0000 7f60ce43a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60c8103330 0x7f60c819efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34944/0 (socket says 172.21.15.70:34944) 2024-04-01T01:48:22.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.793+0000 7f60cd438640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60c8102970 0x7f60c8197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41386/0 (socket says 172.21.15.70:41386) 2024-04-01T01:48:22.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.793+0000 7f60ce43a640 1 -- 172.21.15.70:0/3831254676 learned_addr learned my addr 172.21.15.70:0/3831254676 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.793+0000 7f60ce43a640 1 -- 172.21.15.70:0/3831254676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c8101780 msgr2=0x7f60c8197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.793+0000 7f60ce43a640 1 --2- 172.21.15.70:0/3831254676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c8101780 0x7f60c8197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.793+0000 7f60ce43a640 1 -- 172.21.15.70:0/3831254676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60c8102970 msgr2=0x7f60c8197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.793+0000 7f60ce43a640 1 --2- 172.21.15.70:0/3831254676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60c8102970 0x7f60c8197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.793+0000 7f60ce43a640 1 -- 172.21.15.70:0/3831254676 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c819f620 con 0x7f60c8103330 2024-04-01T01:48:22.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.793+0000 7f60ce43a640 1 --2- 172.21.15.70:0/3831254676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60c8103330 0x7f60c819efb0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f60c400eb10 tx=0x7f60c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.793+0000 7f60baffd640 1 -- 172.21.15.70:0/3831254676 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c4004b20 con 0x7f60c8103330 2024-04-01T01:48:22.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.793+0000 7f60baffd640 1 -- 172.21.15.70:0/3831254676 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60c4004cc0 con 0x7f60c8103330 2024-04-01T01:48:22.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.793+0000 7f60cfe9d640 1 -- 172.21.15.70:0/3831254676 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c819f910 con 0x7f60c8103330 2024-04-01T01:48:22.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.793+0000 7f60baffd640 1 -- 172.21.15.70:0/3831254676 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c4010690 con 0x7f60c8103330 2024-04-01T01:48:22.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.793+0000 7f60cfe9d640 1 -- 172.21.15.70:0/3831254676 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c819fde0 con 0x7f60c8103330 2024-04-01T01:48:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.796+0000 7f60baffd640 1 -- 172.21.15.70:0/3831254676 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f60c4005020 con 0x7f60c8103330 2024-04-01T01:48:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.796+0000 7f60d2722740 1 Processor -- start 2024-04-01T01:48:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.796+0000 7f60d2722740 1 -- start start 2024-04-01T01:48:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.796+0000 7f60d2722740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d4f490 0x561b96d4f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.796+0000 7f60d2722740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4fe80 0x561b96d54980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.796+0000 7f60d2722740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d54ff0 0x561b96d57480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.796+0000 7f60d2722740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561b96cb41b0 con 0x561b96d54ff0 2024-04-01T01:48:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.796+0000 7f60d2722740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561b96c5cc00 con 0x561b96d4fe80 2024-04-01T01:48:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.796+0000 7f60d2722740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561b96cb0250 con 0x561b96d4f490 2024-04-01T01:48:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60cdc39640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4fe80 0x561b96d54980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60cd438640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d54ff0 0x561b96d57480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60baffd640 1 --2- 172.21.15.70:0/3831254676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60b40a24a0 0x7f60b40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60cdc39640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4fe80 0x561b96d54980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41402/0 (socket says 172.21.15.70:41402) 2024-04-01T01:48:22.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60cd438640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d54ff0 0x561b96d57480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34956/0 (socket says 172.21.15.70:34956) 2024-04-01T01:48:22.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60cdc39640 1 -- 172.21.15.70:0/3608863403 learned_addr learned my addr 172.21.15.70:0/3608863403 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60cd438640 1 --2- 172.21.15.70:0/3608863403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d4f490 0x561b96d4f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60ce43a640 1 --2- 172.21.15.70:0/3831254676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60b40a24a0 0x7f60b40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60cdc39640 1 -- 172.21.15.70:0/3608863403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d4f490 msgr2=0x561b96d4f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60cdc39640 1 --2- 172.21.15.70:0/3608863403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d4f490 0x561b96d4f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60cdc39640 1 -- 172.21.15.70:0/3608863403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d54ff0 msgr2=0x561b96d57480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60cdc39640 1 --2- 172.21.15.70:0/3608863403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d54ff0 0x561b96d57480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60cdc39640 1 -- 172.21.15.70:0/3608863403 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b96c7c3d0 con 0x561b96d4fe80 2024-04-01T01:48:22.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60baffd640 1 -- 172.21.15.70:0/3831254676 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f60c40cf190 con 0x7f60c8103330 2024-04-01T01:48:22.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60cdc39640 1 --2- 172.21.15.70:0/3608863403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4fe80 0x561b96d54980 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f60c0009bf0 tx=0x7f60c000e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=939b724e366f9925 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60b3fff640 1 -- 172.21.15.70:0/3608863403 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c0005020 con 0x561b96d4fe80 2024-04-01T01:48:22.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60ce43a640 1 --2- 172.21.15.70:0/3831254676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60b40a24a0 0x7f60b40a4960 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f60c4097500 tx=0x7f60c409f040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60b3fff640 1 -- 172.21.15.70:0/3608863403 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60c00047a0 con 0x561b96d4fe80 2024-04-01T01:48:22.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.797+0000 7f60b3fff640 1 -- 172.21.15.70:0/3608863403 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c0010690 con 0x561b96d4fe80 2024-04-01T01:48:22.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.798+0000 7f60d2722740 1 -- 172.21.15.70:0/3608863403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4fe80 msgr2=0x561b96d54980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.798+0000 7f60d2722740 1 --2- 172.21.15.70:0/3608863403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4fe80 0x561b96d54980 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f60c0009bf0 tx=0x7f60c000e720 comp rx=0 tx=0).stop 2024-04-01T01:48:22.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.798+0000 7f60d2722740 1 -- 172.21.15.70:0/3608863403 shutdown_connections 2024-04-01T01:48:22.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.798+0000 7f60d2722740 1 --2- 172.21.15.70:0/3608863403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d54ff0 0x561b96d57480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.798+0000 7f60d2722740 1 --2- 172.21.15.70:0/3608863403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4fe80 0x561b96d54980 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.798+0000 7f60d2722740 1 --2- 172.21.15.70:0/3608863403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d4f490 0x561b96d4f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.798+0000 7f60d2722740 1 -- 172.21.15.70:0/3608863403 >> 172.21.15.70:0/3608863403 conn(0x561b96d4aa00 msgr2=0x561b96d4ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.798+0000 7f60d2722740 1 -- 172.21.15.70:0/3608863403 shutdown_connections 2024-04-01T01:48:22.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.798+0000 7f60d2722740 1 -- 172.21.15.70:0/3608863403 wait complete. 2024-04-01T01:48:22.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60d2722740 1 -- start start 2024-04-01T01:48:22.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60d2722740 1 Processor -- start 2024-04-01T01:48:22.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60d2722740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4f490 0x561b96d599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60d2722740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d4fe80 0x561b96d59ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60d2722740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d54ff0 0x561b96d60f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60d2722740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561b96d57fa0 con 0x561b96d4f490 2024-04-01T01:48:22.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60d2722740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x561b96c52470 con 0x561b96d54ff0 2024-04-01T01:48:22.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60d2722740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561b96d58120 con 0x561b96d4fe80 2024-04-01T01:48:22.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60d2722740 10 client.0 _ll_register_callbacks cb 0x561b96c6fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:22.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60cd438640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4f490 0x561b96d599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60cd438640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4f490 0x561b96d599a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34966/0 (socket says 172.21.15.70:34966) 2024-04-01T01:48:22.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60cd438640 1 -- 172.21.15.70:0/2774535755 learned_addr learned my addr 172.21.15.70:0/2774535755 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60cd438640 1 --2- 172.21.15.70:0/2774535755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d54ff0 0x561b96d60f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60cdc39640 1 --2- 172.21.15.70:0/2774535755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d4fe80 0x561b96d59ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60cd438640 1 -- 172.21.15.70:0/2774535755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d4fe80 msgr2=0x561b96d59ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60cd438640 1 --2- 172.21.15.70:0/2774535755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d4fe80 0x561b96d59ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60cd438640 1 -- 172.21.15.70:0/2774535755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d54ff0 msgr2=0x561b96d60f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60cd438640 1 --2- 172.21.15.70:0/2774535755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d54ff0 0x561b96d60f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.799+0000 7f60cd438640 1 -- 172.21.15.70:0/2774535755 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b969c83f0 con 0x561b96d4f490 2024-04-01T01:48:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.800+0000 7f60cd438640 1 --2- 172.21.15.70:0/2774535755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4f490 0x561b96d599a0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0096c0 tx=0x7f60bc005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75804a911d3f81af server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.800+0000 7f60b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:48:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.800+0000 7f60b17fa640 1 -- 172.21.15.70:0/2774535755 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc0054f0 con 0x561b96d4f490 2024-04-01T01:48:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.800+0000 7f60b17fa640 1 -- 172.21.15.70:0/2774535755 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f60bc0046c0 con 0x561b96d4f490 2024-04-01T01:48:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.800+0000 7f60d2722740 1 -- 172.21.15.70:0/2774535755 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60c00047a0 con 0x561b96d4f490 2024-04-01T01:48:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.800+0000 7f60b17fa640 1 -- 172.21.15.70:0/2774535755 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f60bc056050 con 0x561b96d4f490 2024-04-01T01:48:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.800+0000 7f60d2722740 10 client.19761 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.800+0000 7f60d2722740 1 -- 172.21.15.70:0/2774535755 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561b96a927d0 con 0x561b96d4f490 2024-04-01T01:48:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.800+0000 7f60b17fa640 1 -- 172.21.15.70:0/2774535755 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60c00047a0 con 0x561b96d4f490 2024-04-01T01:48:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.800+0000 7f60b17fa640 1 -- 172.21.15.70:0/2774535755 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f60bc0038f0 con 0x561b96d4f490 2024-04-01T01:48:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.800+0000 7f60d2722740 10 client.19761 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.800+0000 7f60d2722740 10 client.19761 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:48:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.800+0000 7f60d2722740 20 client.19761 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.801+0000 7f60d2722740 1 --2- 172.21.15.70:0/2774535755 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x561b96d653c0 0x561b96d67830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.801+0000 7f60d2722740 4 client.19761 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.801+0000 7f60d2722740 1 -- 172.21.15.70:0/2774535755 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b96cc16f0 con 0x561b96d653c0 2024-04-01T01:48:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.801+0000 7f60cdc39640 1 --2- 172.21.15.70:0/2774535755 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x561b96d653c0 0x561b96d67830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.801+0000 7f60cdc39640 1 --2- 172.21.15.70:0/2774535755 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x561b96d653c0 0x561b96d67830 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f7b964c97a2e0efc server_cookie=e7b63f83bd326653 in_seq=0 out_seq=0 2024-04-01T01:48:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.801+0000 7f60b17fa640 10 client.19761 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:48:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.802+0000 7f60b17fa640 1 -- 172.21.15.70:0/2774535755 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561b96cc16f0 con 0x561b96d653c0 2024-04-01T01:48:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.802+0000 7f60b17fa640 10 client.19761 handle_command_reply: tid=0 2024-04-01T01:48:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.802+0000 7f60d2722740 1 client.19761 shutdown 2024-04-01T01:48:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.802+0000 7f60d2722740 1 -- 172.21.15.70:0/2774535755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4f490 msgr2=0x561b96d599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.802+0000 7f60d2722740 1 --2- 172.21.15.70:0/2774535755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4f490 0x561b96d599a0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f60bc0096c0 tx=0x7f60bc005d50 comp rx=0 tx=0).stop 2024-04-01T01:48:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.802+0000 7f60d2722740 1 -- 172.21.15.70:0/2774535755 shutdown_connections 2024-04-01T01:48:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.802+0000 7f60d2722740 1 --2- 172.21.15.70:0/2774535755 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x561b96d653c0 0x561b96d67830 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.802+0000 7f60b17fa640 0 client.19761 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:48:22.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.802+0000 7f60d2722740 1 --2- 172.21.15.70:0/2774535755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d54ff0 0x561b96d60f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.802+0000 7f60d2722740 1 --2- 172.21.15.70:0/2774535755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d4fe80 0x561b96d59ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.802+0000 7f60d2722740 1 --2- 172.21.15.70:0/2774535755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4f490 0x561b96d599a0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.802+0000 7f60d2722740 1 -- 172.21.15.70:0/2774535755 >> 172.21.15.70:0/2774535755 conn(0x561b96d4aa00 msgr2=0x561b96d49d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.803+0000 7f60d2722740 1 -- 172.21.15.70:0/2774535755 shutdown_connections 2024-04-01T01:48:22.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.803+0000 7f60d2722740 1 -- 172.21.15.70:0/2774535755 wait complete. 2024-04-01T01:48:22.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.803+0000 7f60d2722740 20 client.19761 trim_cache size 0 max 16384 2024-04-01T01:48:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60d2722740 1 Processor -- start 2024-04-01T01:48:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60d2722740 1 -- start start 2024-04-01T01:48:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60d2722740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4f490 0x561b96d59c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60d2722740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4fe80 0x561b96d60ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60d2722740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d54ff0 0x561b96d6c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60d2722740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f60c81aa3b0 con 0x561b96d4fe80 2024-04-01T01:48:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60d2722740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f60c81abef0 con 0x561b96d4f490 2024-04-01T01:48:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60d2722740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f60c81aaa80 con 0x561b96d54ff0 2024-04-01T01:48:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60cdc39640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4fe80 0x561b96d60ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60cdc39640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4fe80 0x561b96d60ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34968/0 (socket says 172.21.15.70:34968) 2024-04-01T01:48:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60cdc39640 1 -- 172.21.15.70:0/3837029788 learned_addr learned my addr 172.21.15.70:0/3837029788 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60cd438640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4f490 0x561b96d59c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60cd438640 1 --2- 172.21.15.70:0/3837029788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d54ff0 0x561b96d6c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60cdc39640 1 -- 172.21.15.70:0/3837029788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d54ff0 msgr2=0x561b96d6c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60cdc39640 1 --2- 172.21.15.70:0/3837029788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d54ff0 0x561b96d6c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60cdc39640 1 -- 172.21.15.70:0/3837029788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4f490 msgr2=0x561b96d59c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60cdc39640 1 --2- 172.21.15.70:0/3837029788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4f490 0x561b96d59c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.817+0000 7f60cdc39640 1 -- 172.21.15.70:0/3837029788 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b96d54d40 con 0x561b96d4fe80 2024-04-01T01:48:22.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.818+0000 7f60cdc39640 1 --2- 172.21.15.70:0/3837029788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4fe80 0x561b96d60ee0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f60c0009bf0 tx=0x7f60c0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc6d9dc70f93eac server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.818+0000 7f60b37fe640 1 -- 172.21.15.70:0/3837029788 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c0009d60 con 0x561b96d4fe80 2024-04-01T01:48:22.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.818+0000 7f60b37fe640 1 -- 172.21.15.70:0/3837029788 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60c0009530 con 0x561b96d4fe80 2024-04-01T01:48:22.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.818+0000 7f60b37fe640 1 -- 172.21.15.70:0/3837029788 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c0010430 con 0x561b96d4fe80 2024-04-01T01:48:22.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.819+0000 7f60d2722740 1 -- 172.21.15.70:0/3837029788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4fe80 msgr2=0x561b96d60ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.819+0000 7f60d2722740 1 --2- 172.21.15.70:0/3837029788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4fe80 0x561b96d60ee0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f60c0009bf0 tx=0x7f60c0005f90 comp rx=0 tx=0).stop 2024-04-01T01:48:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.819+0000 7f60d2722740 1 -- 172.21.15.70:0/3837029788 shutdown_connections 2024-04-01T01:48:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.819+0000 7f60d2722740 1 --2- 172.21.15.70:0/3837029788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d54ff0 0x561b96d6c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.819+0000 7f60d2722740 1 --2- 172.21.15.70:0/3837029788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4fe80 0x561b96d60ee0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.819+0000 7f60d2722740 1 --2- 172.21.15.70:0/3837029788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4f490 0x561b96d59c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.819+0000 7f60d2722740 1 -- 172.21.15.70:0/3837029788 >> 172.21.15.70:0/3837029788 conn(0x561b96d4aa00 msgr2=0x561b96d4e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.819+0000 7f60d2722740 1 -- 172.21.15.70:0/3837029788 shutdown_connections 2024-04-01T01:48:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.819+0000 7f60d2722740 1 -- 172.21.15.70:0/3837029788 wait complete. 2024-04-01T01:48:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.819+0000 7f60d2722740 1 -- start start 2024-04-01T01:48:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.819+0000 7f60d2722740 1 Processor -- start 2024-04-01T01:48:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60d2722740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4f490 0x561b96d5ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60d2722740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4fe80 0x561b96d5f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60d2722740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d54ff0 0x561b96d60210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60d2722740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f60c81ab820 con 0x561b96d4f490 2024-04-01T01:48:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60d2722740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f60c81ab150 con 0x561b96d4fe80 2024-04-01T01:48:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60d2722740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f60c81a8390 con 0x561b96d54ff0 2024-04-01T01:48:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60d2722740 10 client.0 _ll_register_callbacks cb 0x7f60bc004580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60cd438640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4f490 0x561b96d5ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60cd438640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4f490 0x561b96d5ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:34978/0 (socket says 172.21.15.70:34978) 2024-04-01T01:48:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60cd438640 1 -- 172.21.15.70:0/4175659091 learned_addr learned my addr 172.21.15.70:0/4175659091 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60cdc39640 1 --2- 172.21.15.70:0/4175659091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4fe80 0x561b96d5f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60cd438640 1 --2- 172.21.15.70:0/4175659091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d54ff0 0x561b96d60210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60cd438640 1 -- 172.21.15.70:0/4175659091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d54ff0 msgr2=0x561b96d60210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60cd438640 1 --2- 172.21.15.70:0/4175659091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d54ff0 0x561b96d60210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60cd438640 1 -- 172.21.15.70:0/4175659091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4fe80 msgr2=0x561b96d5f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60cd438640 1 --2- 172.21.15.70:0/4175659091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4fe80 0x561b96d5f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60cd438640 1 -- 172.21.15.70:0/4175659091 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b96d60800 con 0x561b96d4f490 2024-04-01T01:48:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60cdc39640 1 --2- 172.21.15.70:0/4175659091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4fe80 0x561b96d5f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60cd438640 1 --2- 172.21.15.70:0/4175659091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4f490 0x561b96d5ecc0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f60bc009900 tx=0x7f60bc052040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f518881ee6c30381 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:48:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60b17fa640 1 -- 172.21.15.70:0/4175659091 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc060030 con 0x561b96d4f490 2024-04-01T01:48:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60b17fa640 1 -- 172.21.15.70:0/4175659091 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f60bc003800 con 0x561b96d4f490 2024-04-01T01:48:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.820+0000 7f60d2722740 1 -- 172.21.15.70:0/4175659091 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60c0009530 con 0x561b96d4f490 2024-04-01T01:48:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.821+0000 7f60b17fa640 1 -- 172.21.15.70:0/4175659091 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f60bc06a050 con 0x561b96d4f490 2024-04-01T01:48:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.821+0000 7f60d2722740 10 client.19767 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.821+0000 7f60d2722740 1 -- 172.21.15.70:0/4175659091 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561b96dcbe10 con 0x561b96d4f490 2024-04-01T01:48:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.821+0000 7f60b17fa640 1 -- 172.21.15.70:0/4175659091 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f60c0009530 con 0x561b96d4f490 2024-04-01T01:48:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.821+0000 7f60b17fa640 1 -- 172.21.15.70:0/4175659091 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f60bc043450 con 0x561b96d4f490 2024-04-01T01:48:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.821+0000 7f60d2722740 10 client.19767 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.821+0000 7f60d2722740 10 client.19767 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:48:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.821+0000 7f60d2722740 20 client.19767 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.821+0000 7f60d2722740 1 --2- 172.21.15.70:0/4175659091 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x561b96dcc700 0x561b96dceb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.821+0000 7f60d2722740 4 client.19767 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:48:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.821+0000 7f60d2722740 1 -- 172.21.15.70:0/4175659091 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561b96cbba40 con 0x561b96dcc700 2024-04-01T01:48:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.821+0000 7f60cdc39640 1 --2- 172.21.15.70:0/4175659091 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x561b96dcc700 0x561b96dceb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60b17fa640 10 client.19767 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:48:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60cdc39640 1 --2- 172.21.15.70:0/4175659091 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x561b96dcc700 0x561b96dceb20 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=671f0510ae909a4a server_cookie=faa304425e5115f3 in_seq=0 out_seq=0 2024-04-01T01:48:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60b17fa640 1 -- 172.21.15.70:0/4175659091 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561b96cbba40 con 0x561b96dcc700 2024-04-01T01:48:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60b17fa640 10 client.19767 handle_command_reply: tid=0 2024-04-01T01:48:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60d2722740 1 client.19767 shutdown 2024-04-01T01:48:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60d2722740 1 -- 172.21.15.70:0/4175659091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4f490 msgr2=0x561b96d5ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60d2722740 1 --2- 172.21.15.70:0/4175659091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4f490 0x561b96d5ecc0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f60bc009900 tx=0x7f60bc052040 comp rx=0 tx=0).stop 2024-04-01T01:48:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60d2722740 1 -- 172.21.15.70:0/4175659091 shutdown_connections 2024-04-01T01:48:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60d2722740 1 --2- 172.21.15.70:0/4175659091 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x561b96dcc700 0x561b96dceb20 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60d2722740 1 --2- 172.21.15.70:0/4175659091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561b96d54ff0 0x561b96d60210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60b17fa640 0 client.19767 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:48:22.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60d2722740 1 --2- 172.21.15.70:0/4175659091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561b96d4fe80 0x561b96d5f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60d2722740 1 --2- 172.21.15.70:0/4175659091 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561b96d4f490 0x561b96d5ecc0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60d2722740 1 -- 172.21.15.70:0/4175659091 >> 172.21.15.70:0/4175659091 conn(0x561b96d4aa00 msgr2=0x561b96d4e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60d2722740 1 -- 172.21.15.70:0/4175659091 shutdown_connections 2024-04-01T01:48:22.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60d2722740 1 -- 172.21.15.70:0/4175659091 wait complete. 2024-04-01T01:48:22.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.822+0000 7f60d2722740 20 client.19767 trim_cache size 0 max 16384 2024-04-01T01:48:22.828 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:48:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.823+0000 7f60cfe9d640 1 -- 172.21.15.70:0/3831254676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60b40a24a0 msgr2=0x7f60b40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.823+0000 7f60cfe9d640 1 --2- 172.21.15.70:0/3831254676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60b40a24a0 0x7f60b40a4960 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f60c4097500 tx=0x7f60c409f040 comp rx=0 tx=0).stop 2024-04-01T01:48:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.823+0000 7f60cfe9d640 1 -- 172.21.15.70:0/3831254676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60c8103330 msgr2=0x7f60c819efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.823+0000 7f60cfe9d640 1 --2- 172.21.15.70:0/3831254676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60c8103330 0x7f60c819efb0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f60c400eb10 tx=0x7f60c400c2d0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.823+0000 7f60cfe9d640 1 -- 172.21.15.70:0/3831254676 shutdown_connections 2024-04-01T01:48:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.823+0000 7f60cfe9d640 1 --2- 172.21.15.70:0/3831254676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60c8103330 0x7f60c819efb0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.823+0000 7f60cfe9d640 1 --2- 172.21.15.70:0/3831254676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60c8102970 0x7f60c8197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.823+0000 7f60cfe9d640 1 --2- 172.21.15.70:0/3831254676 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f60b40a24a0 0x7f60b40a4960 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.823+0000 7f60cfe9d640 1 --2- 172.21.15.70:0/3831254676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60c8101780 0x7f60c8197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.823+0000 7f60cfe9d640 1 -- 172.21.15.70:0/3831254676 >> 172.21.15.70:0/3831254676 conn(0x7f60c80fccf0 msgr2=0x7f60c8105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.823+0000 7f60cfe9d640 1 -- 172.21.15.70:0/3831254676 shutdown_connections 2024-04-01T01:48:22.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.823+0000 7f60cfe9d640 1 -- 172.21.15.70:0/3831254676 wait complete. 2024-04-01T01:48:22.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:22 smithi081 ceph-mon[35142]: pgmap v805: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 13 MiB/s wr, 1.02k op/s 2024-04-01T01:48:22.830 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:22 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/160937287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:48:22.835 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:48:22.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.954+0000 7f0587a7c640 1 -- 172.21.15.70:0/3199673681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05800ffa10 msgr2=0x7f05800ffe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.954+0000 7f0587a7c640 1 --2- 172.21.15.70:0/3199673681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05800ffa10 0x7f05800ffe30 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f057c009930 tx=0x7f057c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:48:22.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.954+0000 7f0587a7c640 1 -- 172.21.15.70:0/3199673681 shutdown_connections 2024-04-01T01:48:22.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.954+0000 7f0587a7c640 1 --2- 172.21.15.70:0/3199673681 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05801033a0 0x7f0580109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.954+0000 7f0587a7c640 1 --2- 172.21.15.70:0/3199673681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05801029e0 0x7f0580102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.954+0000 7f0587a7c640 1 --2- 172.21.15.70:0/3199673681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05800ffa10 0x7f05800ffe30 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.954+0000 7f0587a7c640 1 -- 172.21.15.70:0/3199673681 >> 172.21.15.70:0/3199673681 conn(0x7f05800fb340 msgr2=0x7f05800fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.954+0000 7f0587a7c640 1 -- 172.21.15.70:0/3199673681 shutdown_connections 2024-04-01T01:48:22.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.954+0000 7f0587a7c640 1 -- 172.21.15.70:0/3199673681 wait complete. 2024-04-01T01:48:22.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0587a7c640 1 Processor -- start 2024-04-01T01:48:22.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0587a7c640 1 -- start start 2024-04-01T01:48:22.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0587a7c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800ffa10 0x7f05801979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0587a7c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05801029e0 0x7f0580197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0587a7c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05801033a0 0x7f058019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0587a7c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f058010d270 con 0x7f05801033a0 2024-04-01T01:48:22.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0587a7c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f058010d0f0 con 0x7f05800ffa10 2024-04-01T01:48:22.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0587a7c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f058010d3f0 con 0x7f05801029e0 2024-04-01T01:48:22.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0585818640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800ffa10 0x7f05801979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0586019640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05801033a0 0x7f058019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0585818640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800ffa10 0x7f05801979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41454/0 (socket says 172.21.15.70:41454) 2024-04-01T01:48:22.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0585818640 1 -- 172.21.15.70:0/104141560 learned_addr learned my addr 172.21.15.70:0/104141560 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0585017640 1 --2- 172.21.15.70:0/104141560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05801029e0 0x7f0580197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0585818640 1 -- 172.21.15.70:0/104141560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05801029e0 msgr2=0x7f0580197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0585818640 1 --2- 172.21.15.70:0/104141560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05801029e0 0x7f0580197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0585818640 1 -- 172.21.15.70:0/104141560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05801033a0 msgr2=0x7f058019ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0585818640 1 --2- 172.21.15.70:0/104141560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05801033a0 0x7f058019ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0585818640 1 -- 172.21.15.70:0/104141560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058019f600 con 0x7f05800ffa10 2024-04-01T01:48:22.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.955+0000 7f0586019640 1 --2- 172.21.15.70:0/104141560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05801033a0 0x7f058019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:22.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.956+0000 7f0585818640 1 --2- 172.21.15.70:0/104141560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800ffa10 0x7f05801979e0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f057c000c00 tx=0x7f057c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.956+0000 7f0576ffd640 1 -- 172.21.15.70:0/104141560 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c043020 con 0x7f05800ffa10 2024-04-01T01:48:22.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.956+0000 7f0587a7c640 1 -- 172.21.15.70:0/104141560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f058019f890 con 0x7f05800ffa10 2024-04-01T01:48:22.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.956+0000 7f0576ffd640 1 -- 172.21.15.70:0/104141560 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f057c004aa0 con 0x7f05800ffa10 2024-04-01T01:48:22.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.956+0000 7f0576ffd640 1 -- 172.21.15.70:0/104141560 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c003c30 con 0x7f05800ffa10 2024-04-01T01:48:22.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.956+0000 7f0587a7c640 1 -- 172.21.15.70:0/104141560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f058019fde0 con 0x7f05800ffa10 2024-04-01T01:48:22.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.958+0000 7f0576ffd640 1 -- 172.21.15.70:0/104141560 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f057c046020 con 0x7f05800ffa10 2024-04-01T01:48:22.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f058a65c740 1 Processor -- start 2024-04-01T01:48:22.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f058a65c740 1 -- start start 2024-04-01T01:48:22.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f058a65c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1a490 0x55c3cbe1a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f058a65c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe1f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f058a65c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe22480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f058a65c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c3cbd7f1b0 con 0x55c3cbe1a490 2024-04-01T01:48:22.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f058a65c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c3cbd27c00 con 0x55c3cbe1ae80 2024-04-01T01:48:22.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f058a65c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c3cbd7b250 con 0x55c3cbe1fff0 2024-04-01T01:48:22.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f0586019640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1a490 0x55c3cbe1a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f0586019640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1a490 0x55c3cbe1a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35000/0 (socket says 172.21.15.70:35000) 2024-04-01T01:48:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f0586019640 1 -- 172.21.15.70:0/196766278 learned_addr learned my addr 172.21.15.70:0/196766278 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f0585818640 1 --2- 172.21.15.70:0/196766278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe1f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f0585017640 1 --2- 172.21.15.70:0/196766278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe22480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f0576ffd640 1 --2- 172.21.15.70:0/104141560 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f056c0a24a0 0x7f056c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f0585818640 1 -- 172.21.15.70:0/196766278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1fff0 msgr2=0x55c3cbe22480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f0585818640 1 --2- 172.21.15.70:0/196766278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe22480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f0585818640 1 -- 172.21.15.70:0/196766278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1a490 msgr2=0x55c3cbe1a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f0585818640 1 --2- 172.21.15.70:0/196766278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1a490 0x55c3cbe1a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f0585818640 1 -- 172.21.15.70:0/196766278 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3cbd473d0 con 0x55c3cbe1ae80 2024-04-01T01:48:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f0586019640 1 --2- 172.21.15.70:0/196766278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1a490 0x55c3cbe1a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:22.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f0585017640 1 --2- 172.21.15.70:0/196766278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe22480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:48:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.959+0000 7f0586019640 1 --2- 172.21.15.70:0/104141560 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f056c0a24a0 0x7f056c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.960+0000 7f0576ffd640 1 -- 172.21.15.70:0/104141560 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f057c0f0f90 con 0x7f05800ffa10 2024-04-01T01:48:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.960+0000 7f0585818640 1 --2- 172.21.15.70:0/196766278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe1f980 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f057c0ee690 tx=0x7f057c07d950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b03686c0d6dddc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.960+0000 7f0586019640 1 --2- 172.21.15.70:0/104141560 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f056c0a24a0 0x7f056c0a4960 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f0578009fd0 tx=0x7f0578004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.960+0000 7f056bfff640 1 -- 172.21.15.70:0/196766278 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c0bd450 con 0x55c3cbe1ae80 2024-04-01T01:48:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.960+0000 7f056bfff640 1 -- 172.21.15.70:0/196766278 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f057c0055b0 con 0x55c3cbe1ae80 2024-04-01T01:48:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.960+0000 7f056bfff640 1 -- 172.21.15.70:0/196766278 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c0a80d0 con 0x55c3cbe1ae80 2024-04-01T01:48:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 -- 172.21.15.70:0/196766278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1ae80 msgr2=0x55c3cbe1f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 --2- 172.21.15.70:0/196766278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe1f980 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f057c0ee690 tx=0x7f057c07d950 comp rx=0 tx=0).stop 2024-04-01T01:48:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 -- 172.21.15.70:0/196766278 shutdown_connections 2024-04-01T01:48:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 --2- 172.21.15.70:0/196766278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe22480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 --2- 172.21.15.70:0/196766278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe1f980 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 --2- 172.21.15.70:0/196766278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1a490 0x55c3cbe1a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 -- 172.21.15.70:0/196766278 >> 172.21.15.70:0/196766278 conn(0x55c3cbe15a00 msgr2=0x55c3cbe15e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 -- 172.21.15.70:0/196766278 shutdown_connections 2024-04-01T01:48:22.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 -- 172.21.15.70:0/196766278 wait complete. 2024-04-01T01:48:22.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 -- start start 2024-04-01T01:48:22.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 Processor -- start 2024-04-01T01:48:22.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1a490 0x55c3cbe245f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe24b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe2bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c3cbd27c00 con 0x55c3cbe1fff0 2024-04-01T01:48:22.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c3cbd1d470 con 0x55c3cbe1a490 2024-04-01T01:48:22.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c3cbe22d60 con 0x55c3cbe1ae80 2024-04-01T01:48:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f0586019640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe2bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f058a65c740 10 client.0 _ll_register_callbacks cb 0x55c3cbd3acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f0586019640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe2bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35014/0 (socket says 172.21.15.70:35014) 2024-04-01T01:48:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f0586019640 1 -- 172.21.15.70:0/2377867565 learned_addr learned my addr 172.21.15.70:0/2377867565 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.961+0000 7f0585818640 1 --2- 172.21.15.70:0/2377867565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1a490 0x55c3cbe245f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.962+0000 7f0585017640 1 --2- 172.21.15.70:0/2377867565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe24b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.962+0000 7f0585818640 1 -- 172.21.15.70:0/2377867565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1ae80 msgr2=0x55c3cbe24b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.962+0000 7f0585818640 1 --2- 172.21.15.70:0/2377867565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe24b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.962+0000 7f0585818640 1 -- 172.21.15.70:0/2377867565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1fff0 msgr2=0x55c3cbe2bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.962+0000 7f0585818640 1 --2- 172.21.15.70:0/2377867565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe2bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.962+0000 7f0585818640 1 -- 172.21.15.70:0/2377867565 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3cba933f0 con 0x55c3cbe1a490 2024-04-01T01:48:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.962+0000 7f0586019640 1 --2- 172.21.15.70:0/2377867565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe2bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.962+0000 7f0585818640 1 --2- 172.21.15.70:0/2377867565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1a490 0x55c3cbe245f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f057c0ba050 tx=0x7f057c0092e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4826960f19f25df1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.962+0000 7f05697fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.962+0000 7f05697fa640 1 -- 172.21.15.70:0/2377867565 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c0bcfe0 con 0x55c3cbe1a490 2024-04-01T01:48:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.962+0000 7f05697fa640 1 -- 172.21.15.70:0/2377867565 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f057c0d1580 con 0x55c3cbe1a490 2024-04-01T01:48:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.962+0000 7f058a65c740 1 -- 172.21.15.70:0/2377867565 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c3cbd8c6f0 con 0x55c3cbe1a490 2024-04-01T01:48:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.962+0000 7f05697fa640 1 -- 172.21.15.70:0/2377867565 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f057c0bd1c0 con 0x55c3cbe1a490 2024-04-01T01:48:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.962+0000 7f058a65c740 10 client.29873 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.962+0000 7f058a65c740 1 -- 172.21.15.70:0/2377867565 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c3cbb5d7d0 con 0x55c3cbe1a490 2024-04-01T01:48:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.963+0000 7f05697fa640 1 -- 172.21.15.70:0/2377867565 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55c3cbd8c6f0 con 0x55c3cbe1a490 2024-04-01T01:48:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.963+0000 7f05697fa640 1 -- 172.21.15.70:0/2377867565 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f057c09c0e0 con 0x55c3cbe1a490 2024-04-01T01:48:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.963+0000 7f058a65c740 10 client.29873 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.963+0000 7f058a65c740 10 client.29873 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:48:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.963+0000 7f058a65c740 20 client.29873 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.963+0000 7f058a65c740 1 --2- 172.21.15.70:0/2377867565 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55c3cbe2fea0 0x55c3cbe32290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.963+0000 7f058a65c740 4 client.29873 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.963+0000 7f058a65c740 1 -- 172.21.15.70:0/2377867565 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3cbd86a40 con 0x55c3cbe2fea0 2024-04-01T01:48:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.963+0000 7f0585017640 1 --2- 172.21.15.70:0/2377867565 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55c3cbe2fea0 0x55c3cbe32290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.964+0000 7f0585017640 1 --2- 172.21.15.70:0/2377867565 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55c3cbe2fea0 0x55c3cbe32290 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=618d65f40e0d00d7 server_cookie=974d2b31b96b7932 in_seq=0 out_seq=0 2024-04-01T01:48:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.964+0000 7f05697fa640 10 client.29873 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:48:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.965+0000 7f05697fa640 1 -- 172.21.15.70:0/2377867565 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c3cbd86a40 con 0x55c3cbe2fea0 2024-04-01T01:48:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.965+0000 7f05697fa640 10 client.29873 handle_command_reply: tid=0 2024-04-01T01:48:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.965+0000 7f058a65c740 1 client.29873 shutdown 2024-04-01T01:48:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.965+0000 7f058a65c740 1 -- 172.21.15.70:0/2377867565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1a490 msgr2=0x55c3cbe245f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.965+0000 7f058a65c740 1 --2- 172.21.15.70:0/2377867565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1a490 0x55c3cbe245f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f057c0ba050 tx=0x7f057c0092e0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.965+0000 7f058a65c740 1 -- 172.21.15.70:0/2377867565 shutdown_connections 2024-04-01T01:48:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.965+0000 7f058a65c740 1 --2- 172.21.15.70:0/2377867565 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55c3cbe2fea0 0x55c3cbe32290 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.965+0000 7f058a65c740 1 --2- 172.21.15.70:0/2377867565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe2bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.965+0000 7f058a65c740 1 --2- 172.21.15.70:0/2377867565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe24b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.965+0000 7f058a65c740 1 --2- 172.21.15.70:0/2377867565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1a490 0x55c3cbe245f0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.965+0000 7f05697fa640 0 client.29873 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:48:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.965+0000 7f058a65c740 1 -- 172.21.15.70:0/2377867565 >> 172.21.15.70:0/2377867565 conn(0x55c3cbe15a00 msgr2=0x55c3cbe14d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.965+0000 7f058a65c740 1 -- 172.21.15.70:0/2377867565 shutdown_connections 2024-04-01T01:48:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.965+0000 7f058a65c740 1 -- 172.21.15.70:0/2377867565 wait complete. 2024-04-01T01:48:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.965+0000 7f058a65c740 20 client.29873 trim_cache size 0 max 16384 2024-04-01T01:48:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.979+0000 7f058a65c740 1 Processor -- start 2024-04-01T01:48:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.979+0000 7f058a65c740 1 -- start start 2024-04-01T01:48:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.979+0000 7f058a65c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1a490 0x55c3cbe24910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.979+0000 7f058a65c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe2bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.979+0000 7f058a65c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe144b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.979+0000 7f058a65c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f05801aa470 con 0x55c3cbe1a490 2024-04-01T01:48:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.979+0000 7f058a65c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f05801abfb0 con 0x55c3cbe1fff0 2024-04-01T01:48:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.979+0000 7f058a65c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f05801aab40 con 0x55c3cbe1ae80 2024-04-01T01:48:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.979+0000 7f0585818640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1a490 0x55c3cbe24910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.979+0000 7f0586019640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe144b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.979+0000 7f0585818640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1a490 0x55c3cbe24910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35016/0 (socket says 172.21.15.70:35016) 2024-04-01T01:48:22.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.979+0000 7f0586019640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe144b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41492/0 (socket says 172.21.15.70:41492) 2024-04-01T01:48:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.979+0000 7f0585818640 1 -- 172.21.15.70:0/70632665 learned_addr learned my addr 172.21.15.70:0/70632665 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.979+0000 7f0585017640 1 --2- 172.21.15.70:0/70632665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe2bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.980+0000 7f0586019640 1 -- 172.21.15.70:0/70632665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1ae80 msgr2=0x55c3cbe2bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.980+0000 7f0586019640 1 --2- 172.21.15.70:0/70632665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe2bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.980+0000 7f0586019640 1 -- 172.21.15.70:0/70632665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1a490 msgr2=0x55c3cbe24910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.980+0000 7f0586019640 1 --2- 172.21.15.70:0/70632665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1a490 0x55c3cbe24910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.980+0000 7f0586019640 1 -- 172.21.15.70:0/70632665 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3cbe328f0 con 0x55c3cbe1fff0 2024-04-01T01:48:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.980+0000 7f0585818640 1 --2- 172.21.15.70:0/70632665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1a490 0x55c3cbe24910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.980+0000 7f0586019640 1 --2- 172.21.15.70:0/70632665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe144b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f05780049c0 tx=0x7f0578011f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=194cacea8ca656e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.980+0000 7f056b7fe640 1 -- 172.21.15.70:0/70632665 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578004b20 con 0x55c3cbe1fff0 2024-04-01T01:48:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.980+0000 7f056b7fe640 1 -- 172.21.15.70:0/70632665 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0578004cc0 con 0x55c3cbe1fff0 2024-04-01T01:48:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.980+0000 7f056b7fe640 1 -- 172.21.15.70:0/70632665 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05780284b0 con 0x55c3cbe1fff0 2024-04-01T01:48:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.981+0000 7f058a65c740 1 -- 172.21.15.70:0/70632665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1fff0 msgr2=0x55c3cbe144b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.981+0000 7f058a65c740 1 --2- 172.21.15.70:0/70632665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe144b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f05780049c0 tx=0x7f0578011f90 comp rx=0 tx=0).stop 2024-04-01T01:48:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.981+0000 7f058a65c740 1 -- 172.21.15.70:0/70632665 shutdown_connections 2024-04-01T01:48:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.981+0000 7f058a65c740 1 --2- 172.21.15.70:0/70632665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe144b0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.981+0000 7f058a65c740 1 --2- 172.21.15.70:0/70632665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe2bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.981+0000 7f058a65c740 1 --2- 172.21.15.70:0/70632665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1a490 0x55c3cbe24910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.981+0000 7f058a65c740 1 -- 172.21.15.70:0/70632665 >> 172.21.15.70:0/70632665 conn(0x55c3cbe15a00 msgr2=0x55c3cbe24200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.981+0000 7f058a65c740 1 -- 172.21.15.70:0/70632665 shutdown_connections 2024-04-01T01:48:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.981+0000 7f058a65c740 1 -- 172.21.15.70:0/70632665 wait complete. 2024-04-01T01:48:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.981+0000 7f058a65c740 1 -- start start 2024-04-01T01:48:22.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.981+0000 7f058a65c740 1 Processor -- start 2024-04-01T01:48:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f058a65c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1a490 0x55c3cbe29490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f058a65c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe299d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f058a65c740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe2a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f0585017640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe299d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f058a65c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f05801ab8e0 con 0x55c3cbe1ae80 2024-04-01T01:48:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f058a65c740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f05801ab210 con 0x55c3cbe1fff0 2024-04-01T01:48:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f0585017640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe299d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35024/0 (socket says 172.21.15.70:35024) 2024-04-01T01:48:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f058a65c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f05801a8450 con 0x55c3cbe1a490 2024-04-01T01:48:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f0585017640 1 -- 172.21.15.70:0/83248551 learned_addr learned my addr 172.21.15.70:0/83248551 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f058a65c740 10 client.0 _ll_register_callbacks cb 0x7f057c0035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f0586019640 1 --2- 172.21.15.70:0/83248551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe2a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f0585818640 1 --2- 172.21.15.70:0/83248551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1a490 0x55c3cbe29490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f0585017640 1 -- 172.21.15.70:0/83248551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1a490 msgr2=0x55c3cbe29490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f0585017640 1 --2- 172.21.15.70:0/83248551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1a490 0x55c3cbe29490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f0585017640 1 -- 172.21.15.70:0/83248551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1fff0 msgr2=0x55c3cbe2a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f0585017640 1 --2- 172.21.15.70:0/83248551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe2a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.982+0000 7f0585017640 1 -- 172.21.15.70:0/83248551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3cbe2af80 con 0x55c3cbe1ae80 2024-04-01T01:48:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.983+0000 7f0585017640 1 --2- 172.21.15.70:0/83248551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe299d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f0580198ef0 tx=0x7f057001be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f9c7f93f88d4de server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.983+0000 7f05697fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:48:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.983+0000 7f05697fa640 1 -- 172.21.15.70:0/83248551 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0570004440 con 0x55c3cbe1ae80 2024-04-01T01:48:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.983+0000 7f05697fa640 1 -- 172.21.15.70:0/83248551 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f05700035f0 con 0x55c3cbe1ae80 2024-04-01T01:48:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.983+0000 7f058a65c740 1 -- 172.21.15.70:0/83248551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0578004cc0 con 0x55c3cbe1ae80 2024-04-01T01:48:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.983+0000 7f05697fa640 1 -- 172.21.15.70:0/83248551 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f057001b800 con 0x55c3cbe1ae80 2024-04-01T01:48:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.983+0000 7f058a65c740 10 client.19785 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.983+0000 7f058a65c740 1 -- 172.21.15.70:0/83248551 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c3cbe96460 con 0x55c3cbe1ae80 2024-04-01T01:48:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.983+0000 7f05697fa640 1 -- 172.21.15.70:0/83248551 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0578004cc0 con 0x55c3cbe1ae80 2024-04-01T01:48:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.983+0000 7f05697fa640 1 -- 172.21.15.70:0/83248551 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f0570022070 con 0x55c3cbe1ae80 2024-04-01T01:48:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.983+0000 7f058a65c740 10 client.19785 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.983+0000 7f058a65c740 10 client.19785 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:48:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.983+0000 7f058a65c740 20 client.19785 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.983+0000 7f058a65c740 1 --2- 172.21.15.70:0/83248551 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55c3cbe96f10 0x55c3cbe99300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:22.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.983+0000 7f058a65c740 4 client.19785 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:48:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.983+0000 7f058a65c740 1 -- 172.21.15.70:0/83248551 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c3cbd046a0 con 0x55c3cbe96f10 2024-04-01T01:48:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.984+0000 7f0585818640 1 --2- 172.21.15.70:0/83248551 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55c3cbe96f10 0x55c3cbe99300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.986+0000 7f05697fa640 10 client.19785 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:48:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.986+0000 7f0585818640 1 --2- 172.21.15.70:0/83248551 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55c3cbe96f10 0x55c3cbe99300 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1963799fbdfe63f2 server_cookie=8aec7310273cccab in_seq=0 out_seq=0 2024-04-01T01:48:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.987+0000 7f05697fa640 1 -- 172.21.15.70:0/83248551 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c3cbd046a0 con 0x55c3cbe96f10 2024-04-01T01:48:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.987+0000 7f05697fa640 10 client.19785 handle_command_reply: tid=0 2024-04-01T01:48:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.987+0000 7f058a65c740 1 client.19785 shutdown 2024-04-01T01:48:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.987+0000 7f058a65c740 1 -- 172.21.15.70:0/83248551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1ae80 msgr2=0x55c3cbe299d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.987+0000 7f058a65c740 1 --2- 172.21.15.70:0/83248551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe299d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f0580198ef0 tx=0x7f057001be70 comp rx=0 tx=0).stop 2024-04-01T01:48:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.987+0000 7f058a65c740 1 -- 172.21.15.70:0/83248551 shutdown_connections 2024-04-01T01:48:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.987+0000 7f058a65c740 1 --2- 172.21.15.70:0/83248551 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x55c3cbe96f10 0x55c3cbe99300 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.987+0000 7f05697fa640 0 client.19785 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:48:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.987+0000 7f058a65c740 1 --2- 172.21.15.70:0/83248551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c3cbe1fff0 0x55c3cbe2a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.987+0000 7f058a65c740 1 --2- 172.21.15.70:0/83248551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c3cbe1ae80 0x55c3cbe299d0 secure :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f0580198ef0 tx=0x7f057001be70 comp rx=0 tx=0).stop 2024-04-01T01:48:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.987+0000 7f058a65c740 1 --2- 172.21.15.70:0/83248551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3cbe1a490 0x55c3cbe29490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.987+0000 7f058a65c740 1 -- 172.21.15.70:0/83248551 >> 172.21.15.70:0/83248551 conn(0x55c3cbe15a00 msgr2=0x55c3cbe18c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.988+0000 7f058a65c740 1 -- 172.21.15.70:0/83248551 shutdown_connections 2024-04-01T01:48:22.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.988+0000 7f058a65c740 1 -- 172.21.15.70:0/83248551 wait complete. 2024-04-01T01:48:22.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.988+0000 7f058a65c740 20 client.19785 trim_cache size 0 max 16384 2024-04-01T01:48:22.992 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:48:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.990+0000 7f0587a7c640 1 -- 172.21.15.70:0/104141560 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f056c0a24a0 msgr2=0x7f056c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.991+0000 7f0587a7c640 1 --2- 172.21.15.70:0/104141560 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f056c0a24a0 0x7f056c0a4960 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f0578009fd0 tx=0x7f0578004a40 comp rx=0 tx=0).stop 2024-04-01T01:48:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.991+0000 7f0587a7c640 1 -- 172.21.15.70:0/104141560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800ffa10 msgr2=0x7f05801979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.991+0000 7f0587a7c640 1 --2- 172.21.15.70:0/104141560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800ffa10 0x7f05801979e0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f057c000c00 tx=0x7f057c004270 comp rx=0 tx=0).stop 2024-04-01T01:48:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.991+0000 7f0587a7c640 1 -- 172.21.15.70:0/104141560 shutdown_connections 2024-04-01T01:48:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.991+0000 7f0587a7c640 1 --2- 172.21.15.70:0/104141560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05801033a0 0x7f058019ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.991+0000 7f0587a7c640 1 --2- 172.21.15.70:0/104141560 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f056c0a24a0 0x7f056c0a4960 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.991+0000 7f0587a7c640 1 --2- 172.21.15.70:0/104141560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05801029e0 0x7f0580197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.991+0000 7f0587a7c640 1 --2- 172.21.15.70:0/104141560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05800ffa10 0x7f05801979e0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.991+0000 7f0587a7c640 1 -- 172.21.15.70:0/104141560 >> 172.21.15.70:0/104141560 conn(0x7f05800fb340 msgr2=0x7f0580105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.992+0000 7f0587a7c640 1 -- 172.21.15.70:0/104141560 shutdown_connections 2024-04-01T01:48:22.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:22.992+0000 7f0587a7c640 1 -- 172.21.15.70:0/104141560 wait complete. 2024-04-01T01:48:22.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:22 smithi189 ceph-mon[35753]: pgmap v805: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 13 MiB/s wr, 1.02k op/s 2024-04-01T01:48:22.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:22 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/160937287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:48:23.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:48:23.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.173+0000 7ff94bfff640 1 -- 172.21.15.70:0/2262349659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff944096f70 msgr2=0x7ff9440973f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:23.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.173+0000 7ff94bfff640 1 --2- 172.21.15.70:0/2262349659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff944096f70 0x7ff9440973f0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7ff940009900 tx=0x7ff94002bc30 comp rx=0 tx=0).stop 2024-04-01T01:48:23.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.173+0000 7ff94bfff640 1 -- 172.21.15.70:0/2262349659 shutdown_connections 2024-04-01T01:48:23.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.173+0000 7ff94bfff640 1 --2- 172.21.15.70:0/2262349659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff944097930 0x7ff94409e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.173+0000 7ff94bfff640 1 --2- 172.21.15.70:0/2262349659 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff944096f70 0x7ff9440973f0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.173+0000 7ff94bfff640 1 --2- 172.21.15.70:0/2262349659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff944095d80 0x7ff9440961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.173+0000 7ff94bfff640 1 -- 172.21.15.70:0/2262349659 >> 172.21.15.70:0/2262349659 conn(0x7ff9440912d0 msgr2=0x7ff944093730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:23.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.173+0000 7ff94bfff640 1 -- 172.21.15.70:0/2262349659 shutdown_connections 2024-04-01T01:48:23.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.173+0000 7ff94bfff640 1 -- 172.21.15.70:0/2262349659 wait complete. 2024-04-01T01:48:23.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.174+0000 7ff94bfff640 1 Processor -- start 2024-04-01T01:48:23.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.174+0000 7ff94bfff640 1 -- start start 2024-04-01T01:48:23.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.174+0000 7ff94bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff944095d80 0x7ff94412bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.174+0000 7ff94bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff944096f70 0x7ff94412c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.174+0000 7ff94bfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff944097930 0x7ff9441335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.174+0000 7ff94affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff944095d80 0x7ff94412bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.174+0000 7ff94affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff944095d80 0x7ff94412bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41522/0 (socket says 172.21.15.70:41522) 2024-04-01T01:48:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.174+0000 7ff94affd640 1 -- 172.21.15.70:0/3162999938 learned_addr learned my addr 172.21.15.70:0/3162999938 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.174+0000 7ff94a7fc640 1 --2- 172.21.15.70:0/3162999938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff944096f70 0x7ff94412c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.174+0000 7ff94bfff640 1 -- 172.21.15.70:0/3162999938 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9440a18b0 con 0x7ff944096f70 2024-04-01T01:48:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.174+0000 7ff94b7fe640 1 --2- 172.21.15.70:0/3162999938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff944097930 0x7ff9441335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.174+0000 7ff94bfff640 1 -- 172.21.15.70:0/3162999938 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff9440a1730 con 0x7ff944095d80 2024-04-01T01:48:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.174+0000 7ff94bfff640 1 -- 172.21.15.70:0/3162999938 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff9440a1a30 con 0x7ff944097930 2024-04-01T01:48:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.175+0000 7ff94a7fc640 1 -- 172.21.15.70:0/3162999938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff944097930 msgr2=0x7ff9441335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.175+0000 7ff94a7fc640 1 --2- 172.21.15.70:0/3162999938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff944097930 0x7ff9441335a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.175+0000 7ff94a7fc640 1 -- 172.21.15.70:0/3162999938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff944095d80 msgr2=0x7ff94412bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.175+0000 7ff94a7fc640 1 --2- 172.21.15.70:0/3162999938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff944095d80 0x7ff94412bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.175+0000 7ff94a7fc640 1 -- 172.21.15.70:0/3162999938 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff944133c10 con 0x7ff944096f70 2024-04-01T01:48:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.175+0000 7ff94a7fc640 1 --2- 172.21.15.70:0/3162999938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff944096f70 0x7ff94412c530 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7ff940006010 tx=0x7ff94002de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.175+0000 7ff933fff640 1 -- 172.21.15.70:0/3162999938 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff940043020 con 0x7ff944096f70 2024-04-01T01:48:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.175+0000 7ff94bfff640 1 -- 172.21.15.70:0/3162999938 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff944133ea0 con 0x7ff944096f70 2024-04-01T01:48:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.175+0000 7ff933fff640 1 -- 172.21.15.70:0/3162999938 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff9400048d0 con 0x7ff944096f70 2024-04-01T01:48:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.175+0000 7ff933fff640 1 -- 172.21.15.70:0/3162999938 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff940005220 con 0x7ff944096f70 2024-04-01T01:48:23.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.176+0000 7ff94bfff640 1 -- 172.21.15.70:0/3162999938 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9441343b0 con 0x7ff944096f70 2024-04-01T01:48:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.179+0000 7ff933fff640 1 -- 172.21.15.70:0/3162999938 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7ff940046020 con 0x7ff944096f70 2024-04-01T01:48:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.179+0000 7ff953fb7740 1 Processor -- start 2024-04-01T01:48:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.179+0000 7ff953fb7740 1 -- start start 2024-04-01T01:48:23.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.179+0000 7ff953fb7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453ce620 0x5561453cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff953fb7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453cf010 0x5561453d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff933fff640 1 --2- 172.21.15.70:0/3162999938 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff9340a24a0 0x7ff9340a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff953fb7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453d2170 0x5561453defe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff953fb7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5561453331b0 con 0x5561453ce620 2024-04-01T01:48:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff953fb7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5561452dbc00 con 0x5561453cf010 2024-04-01T01:48:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff953fb7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55614532f250 con 0x5561453d2170 2024-04-01T01:48:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff94b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453ce620 0x5561453cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff94b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453ce620 0x5561453cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35042/0 (socket says 172.21.15.70:35042) 2024-04-01T01:48:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff94b7fe640 1 -- 172.21.15.70:0/933523105 learned_addr learned my addr 172.21.15.70:0/933523105 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff94affd640 1 --2- 172.21.15.70:0/933523105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453cf010 0x5561453d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff933fff640 1 -- 172.21.15.70:0/3162999938 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff9400f0c90 con 0x7ff944096f70 2024-04-01T01:48:23.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff94a7fc640 1 --2- 172.21.15.70:0/3162999938 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff9340a24a0 0x7ff9340a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff94affd640 1 -- 172.21.15.70:0/933523105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453d2170 msgr2=0x5561453defe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff94b7fe640 1 --2- 172.21.15.70:0/933523105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453d2170 0x5561453defe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff94affd640 1 --2- 172.21.15.70:0/933523105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453d2170 0x5561453defe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff94affd640 1 -- 172.21.15.70:0/933523105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453ce620 msgr2=0x5561453cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff94affd640 1 --2- 172.21.15.70:0/933523105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453ce620 0x5561453cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff94affd640 1 -- 172.21.15.70:0/933523105 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561452fb3d0 con 0x5561453cf010 2024-04-01T01:48:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.180+0000 7ff94b7fe640 1 --2- 172.21.15.70:0/933523105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453d2170 0x5561453defe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:48:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.181+0000 7ff94affd640 1 --2- 172.21.15.70:0/933523105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453cf010 0x5561453d1b00 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff93c0058c0 tx=0x7ff93c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8d471122347f1a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.181+0000 7ff94a7fc640 1 --2- 172.21.15.70:0/3162999938 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff9340a24a0 0x7ff9340a4960 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff9400b9cc0 tx=0x7ff94002c750 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.181+0000 7ff9317fa640 1 -- 172.21.15.70:0/933523105 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93c004220 con 0x5561453cf010 2024-04-01T01:48:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.181+0000 7ff9317fa640 1 -- 172.21.15.70:0/933523105 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff93c009e20 con 0x5561453cf010 2024-04-01T01:48:23.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.181+0000 7ff9317fa640 1 -- 172.21.15.70:0/933523105 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93c0197d0 con 0x5561453cf010 2024-04-01T01:48:23.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.182+0000 7ff953fb7740 1 -- 172.21.15.70:0/933523105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453cf010 msgr2=0x5561453d1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:23.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.182+0000 7ff953fb7740 1 --2- 172.21.15.70:0/933523105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453cf010 0x5561453d1b00 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff93c0058c0 tx=0x7ff93c011810 comp rx=0 tx=0).stop 2024-04-01T01:48:23.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.182+0000 7ff953fb7740 1 -- 172.21.15.70:0/933523105 shutdown_connections 2024-04-01T01:48:23.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.182+0000 7ff953fb7740 1 --2- 172.21.15.70:0/933523105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453d2170 0x5561453defe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.182+0000 7ff953fb7740 1 --2- 172.21.15.70:0/933523105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453cf010 0x5561453d1b00 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.182+0000 7ff953fb7740 1 --2- 172.21.15.70:0/933523105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453ce620 0x5561453cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.182+0000 7ff953fb7740 1 -- 172.21.15.70:0/933523105 >> 172.21.15.70:0/933523105 conn(0x5561453c9c10 msgr2=0x5561453ca010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:23.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.182+0000 7ff953fb7740 1 -- 172.21.15.70:0/933523105 shutdown_connections 2024-04-01T01:48:23.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.182+0000 7ff953fb7740 1 -- 172.21.15.70:0/933523105 wait complete. 2024-04-01T01:48:23.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.183+0000 7ff953fb7740 1 -- start start 2024-04-01T01:48:23.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.183+0000 7ff953fb7740 1 Processor -- start 2024-04-01T01:48:23.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.183+0000 7ff953fb7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453ce620 0x5561453e11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.183+0000 7ff953fb7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453cf010 0x5561453e1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.183+0000 7ff953fb7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453d2170 0x5561453e8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.183+0000 7ff953fb7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5561453df9e0 con 0x5561453ce620 2024-04-01T01:48:23.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.183+0000 7ff953fb7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5561452d1470 con 0x5561453cf010 2024-04-01T01:48:23.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.183+0000 7ff953fb7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5561453dfb60 con 0x5561453d2170 2024-04-01T01:48:23.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.183+0000 7ff953fb7740 10 client.0 _ll_register_callbacks cb 0x5561452eecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:23.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.183+0000 7ff94affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453cf010 0x5561453e1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.183+0000 7ff94b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453ce620 0x5561453e11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.183+0000 7ff94affd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453cf010 0x5561453e1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41538/0 (socket says 172.21.15.70:41538) 2024-04-01T01:48:23.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.183+0000 7ff94affd640 1 -- 172.21.15.70:0/4239088944 learned_addr learned my addr 172.21.15.70:0/4239088944 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:23.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.183+0000 7ff94b7fe640 1 --2- 172.21.15.70:0/4239088944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453d2170 0x5561453e8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.184+0000 7ff94b7fe640 1 -- 172.21.15.70:0/4239088944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453d2170 msgr2=0x5561453e8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:23.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.184+0000 7ff94b7fe640 1 --2- 172.21.15.70:0/4239088944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453d2170 0x5561453e8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.184+0000 7ff94b7fe640 1 -- 172.21.15.70:0/4239088944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453cf010 msgr2=0x5561453e1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:23.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.184+0000 7ff94b7fe640 1 --2- 172.21.15.70:0/4239088944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453cf010 0x5561453e1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.184+0000 7ff94b7fe640 1 -- 172.21.15.70:0/4239088944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561450473f0 con 0x5561453ce620 2024-04-01T01:48:23.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.184+0000 7ff94b7fe640 1 --2- 172.21.15.70:0/4239088944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453ce620 0x5561453e11e0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7ff94c04fbf0 tx=0x7ff94c077040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6604d0ab742dc4f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:23.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.184+0000 7ff92eff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:48:23.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.184+0000 7ff92eff5640 1 -- 172.21.15.70:0/4239088944 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c067dd0 con 0x5561453ce620 2024-04-01T01:48:23.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.184+0000 7ff92eff5640 1 -- 172.21.15.70:0/4239088944 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff94c069110 con 0x5561453ce620 2024-04-01T01:48:23.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.184+0000 7ff953fb7740 1 -- 172.21.15.70:0/4239088944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff93c009e20 con 0x5561453ce620 2024-04-01T01:48:23.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.184+0000 7ff92eff5640 1 -- 172.21.15.70:0/4239088944 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff94c074b00 con 0x5561453ce620 2024-04-01T01:48:23.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.184+0000 7ff953fb7740 10 client.19797 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:23.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.184+0000 7ff953fb7740 1 -- 172.21.15.70:0/4239088944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5561451117d0 con 0x5561453ce620 2024-04-01T01:48:23.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.184+0000 7ff92eff5640 1 -- 172.21.15.70:0/4239088944 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff93c009e20 con 0x5561453ce620 2024-04-01T01:48:23.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.185+0000 7ff92eff5640 1 -- 172.21.15.70:0/4239088944 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff94c086640 con 0x5561453ce620 2024-04-01T01:48:23.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.185+0000 7ff953fb7740 10 client.19797 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:23.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.185+0000 7ff953fb7740 10 client.19797 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:23.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.185+0000 7ff953fb7740 20 client.19797 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:23.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.185+0000 7ff953fb7740 1 --2- 172.21.15.70:0/4239088944 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5561453ecc00 0x5561453ef080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.185+0000 7ff953fb7740 4 client.19797 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:23.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.185+0000 7ff953fb7740 1 -- 172.21.15.70:0/4239088944 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561453406f0 con 0x5561453ecc00 2024-04-01T01:48:23.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.185+0000 7ff94affd640 1 --2- 172.21.15.70:0/4239088944 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5561453ecc00 0x5561453ef080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.185+0000 7ff94affd640 1 --2- 172.21.15.70:0/4239088944 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5561453ecc00 0x5561453ef080 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=505bf54c491c66a4 server_cookie=e4d185dadc133d81 in_seq=0 out_seq=0 2024-04-01T01:48:23.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.185+0000 7ff92eff5640 10 client.19797 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:23.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.186+0000 7ff92eff5640 1 -- 172.21.15.70:0/4239088944 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5561453406f0 con 0x5561453ecc00 2024-04-01T01:48:23.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.186+0000 7ff92eff5640 10 client.19797 handle_command_reply: tid=0 2024-04-01T01:48:23.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.186+0000 7ff953fb7740 1 client.19797 shutdown 2024-04-01T01:48:23.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.186+0000 7ff953fb7740 1 -- 172.21.15.70:0/4239088944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453ce620 msgr2=0x5561453e11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:23.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.186+0000 7ff953fb7740 1 --2- 172.21.15.70:0/4239088944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453ce620 0x5561453e11e0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7ff94c04fbf0 tx=0x7ff94c077040 comp rx=0 tx=0).stop 2024-04-01T01:48:23.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.186+0000 7ff953fb7740 1 -- 172.21.15.70:0/4239088944 shutdown_connections 2024-04-01T01:48:23.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.186+0000 7ff953fb7740 1 --2- 172.21.15.70:0/4239088944 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5561453ecc00 0x5561453ef080 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.186+0000 7ff92eff5640 0 client.19797 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:23.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.186+0000 7ff953fb7740 1 --2- 172.21.15.70:0/4239088944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453d2170 0x5561453e8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.186+0000 7ff953fb7740 1 --2- 172.21.15.70:0/4239088944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453cf010 0x5561453e1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.186+0000 7ff953fb7740 1 --2- 172.21.15.70:0/4239088944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453ce620 0x5561453e11e0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.186+0000 7ff953fb7740 1 -- 172.21.15.70:0/4239088944 >> 172.21.15.70:0/4239088944 conn(0x5561453c9c10 msgr2=0x5561453c8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:23.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.186+0000 7ff953fb7740 1 -- 172.21.15.70:0/4239088944 shutdown_connections 2024-04-01T01:48:23.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.186+0000 7ff953fb7740 1 -- 172.21.15.70:0/4239088944 wait complete. 2024-04-01T01:48:23.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.186+0000 7ff953fb7740 20 client.19797 trim_cache size 0 max 16384 2024-04-01T01:48:23.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.201+0000 7ff953fb7740 1 Processor -- start 2024-04-01T01:48:23.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.201+0000 7ff953fb7740 1 -- start start 2024-04-01T01:48:23.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.201+0000 7ff953fb7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453ce620 0x5561453e0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.201+0000 7ff953fb7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453cf010 0x5561453e8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.201+0000 7ff953fb7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453d2170 0x5561453c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.201+0000 7ff953fb7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff94413c7f0 con 0x5561453d2170 2024-04-01T01:48:23.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.201+0000 7ff953fb7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff94413e330 con 0x5561453ce620 2024-04-01T01:48:23.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.201+0000 7ff953fb7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff94413cec0 con 0x5561453cf010 2024-04-01T01:48:23.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.202+0000 7ff94b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453d2170 0x5561453c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.202+0000 7ff94b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453d2170 0x5561453c9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35066/0 (socket says 172.21.15.70:35066) 2024-04-01T01:48:23.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.202+0000 7ff94b7fe640 1 -- 172.21.15.70:0/643559163 learned_addr learned my addr 172.21.15.70:0/643559163 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:23.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.202+0000 7ff94affd640 1 --2- 172.21.15.70:0/643559163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453cf010 0x5561453e8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.202+0000 7ff94b7fe640 1 --2- 172.21.15.70:0/643559163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453ce620 0x5561453e0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.202+0000 7ff94b7fe640 1 -- 172.21.15.70:0/643559163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453cf010 msgr2=0x5561453e8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:23.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.202+0000 7ff94b7fe640 1 --2- 172.21.15.70:0/643559163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453cf010 0x5561453e8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.202+0000 7ff94b7fe640 1 -- 172.21.15.70:0/643559163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453ce620 msgr2=0x5561453e0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:23.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.202+0000 7ff94b7fe640 1 --2- 172.21.15.70:0/643559163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453ce620 0x5561453e0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.202+0000 7ff94b7fe640 1 -- 172.21.15.70:0/643559163 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561453bae60 con 0x5561453d2170 2024-04-01T01:48:23.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.202+0000 7ff94b7fe640 1 --2- 172.21.15.70:0/643559163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453d2170 0x5561453c9ff0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7ff94c050260 tx=0x7ff94c088040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24099e8dd6eae5aa server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:23.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.202+0000 7ff930ff9640 1 -- 172.21.15.70:0/643559163 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c086d70 con 0x5561453d2170 2024-04-01T01:48:23.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.202+0000 7ff930ff9640 1 -- 172.21.15.70:0/643559163 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff94c069a40 con 0x5561453d2170 2024-04-01T01:48:23.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.202+0000 7ff930ff9640 1 -- 172.21.15.70:0/643559163 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c0693c0 con 0x5561453d2170 2024-04-01T01:48:23.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.203+0000 7ff953fb7740 1 -- 172.21.15.70:0/643559163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453d2170 msgr2=0x5561453c9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:23.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.203+0000 7ff953fb7740 1 --2- 172.21.15.70:0/643559163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453d2170 0x5561453c9ff0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7ff94c050260 tx=0x7ff94c088040 comp rx=0 tx=0).stop 2024-04-01T01:48:23.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.203+0000 7ff953fb7740 1 -- 172.21.15.70:0/643559163 shutdown_connections 2024-04-01T01:48:23.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.203+0000 7ff953fb7740 1 --2- 172.21.15.70:0/643559163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453d2170 0x5561453c9ff0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.203+0000 7ff953fb7740 1 --2- 172.21.15.70:0/643559163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453cf010 0x5561453e8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.203+0000 7ff953fb7740 1 --2- 172.21.15.70:0/643559163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453ce620 0x5561453e0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.203+0000 7ff953fb7740 1 -- 172.21.15.70:0/643559163 >> 172.21.15.70:0/643559163 conn(0x5561453c9c10 msgr2=0x5561453c4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:23.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.203+0000 7ff953fb7740 1 -- 172.21.15.70:0/643559163 shutdown_connections 2024-04-01T01:48:23.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.203+0000 7ff953fb7740 1 -- 172.21.15.70:0/643559163 wait complete. 2024-04-01T01:48:23.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.203+0000 7ff953fb7740 1 -- start start 2024-04-01T01:48:23.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff953fb7740 1 Processor -- start 2024-04-01T01:48:23.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff953fb7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453ce620 0x5561453e6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff953fb7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453cf010 0x5561453e6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff953fb7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453d2170 0x5561453e7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff953fb7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff94413dc60 con 0x5561453cf010 2024-04-01T01:48:23.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff953fb7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff94413d590 con 0x5561453d2170 2024-04-01T01:48:23.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff953fb7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff94413a7d0 con 0x5561453ce620 2024-04-01T01:48:23.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff94affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453cf010 0x5561453e6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff953fb7740 10 client.0 _ll_register_callbacks cb 0x7ff94c0889b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:23.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff94affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453cf010 0x5561453e6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35068/0 (socket says 172.21.15.70:35068) 2024-04-01T01:48:23.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff94affd640 1 -- 172.21.15.70:0/2574769034 learned_addr learned my addr 172.21.15.70:0/2574769034 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:23.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff94b7fe640 1 --2- 172.21.15.70:0/2574769034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453ce620 0x5561453e6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff94b7fe640 1 --2- 172.21.15.70:0/2574769034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453d2170 0x5561453e7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff94affd640 1 -- 172.21.15.70:0/2574769034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453ce620 msgr2=0x5561453e6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:23.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff94affd640 1 --2- 172.21.15.70:0/2574769034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453ce620 0x5561453e6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff94affd640 1 -- 172.21.15.70:0/2574769034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453d2170 msgr2=0x5561453e7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:23.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff94affd640 1 --2- 172.21.15.70:0/2574769034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453d2170 0x5561453e7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff94affd640 1 -- 172.21.15.70:0/2574769034 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561453e7b40 con 0x5561453cf010 2024-04-01T01:48:23.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff94affd640 1 --2- 172.21.15.70:0/2574769034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453cf010 0x5561453e6540 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7ff93c01b040 tx=0x7ff93c005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77aad75e83995fcf server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:23.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff92eff5640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:48:23.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff92eff5640 1 -- 172.21.15.70:0/2574769034 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93c01d040 con 0x5561453cf010 2024-04-01T01:48:23.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff92eff5640 1 -- 172.21.15.70:0/2574769034 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7ff93c003800 con 0x5561453cf010 2024-04-01T01:48:23.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.204+0000 7ff953fb7740 1 -- 172.21.15.70:0/2574769034 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff94c069a40 con 0x5561453cf010 2024-04-01T01:48:23.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.205+0000 7ff92eff5640 1 -- 172.21.15.70:0/2574769034 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff93c03d050 con 0x5561453cf010 2024-04-01T01:48:23.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.205+0000 7ff953fb7740 10 client.19803 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:23.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.205+0000 7ff953fb7740 1 -- 172.21.15.70:0/2574769034 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556145453200 con 0x5561453cf010 2024-04-01T01:48:23.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.205+0000 7ff92eff5640 1 -- 172.21.15.70:0/2574769034 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff93c030030 con 0x5561453cf010 2024-04-01T01:48:23.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.205+0000 7ff92eff5640 1 -- 172.21.15.70:0/2574769034 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff93c026a30 con 0x5561453cf010 2024-04-01T01:48:23.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.205+0000 7ff953fb7740 10 client.19803 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:23.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.205+0000 7ff953fb7740 10 client.19803 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:23.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.205+0000 7ff953fb7740 20 client.19803 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:23.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.205+0000 7ff953fb7740 1 --2- 172.21.15.70:0/2574769034 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5561453ecc00 0x556145455a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:23.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.205+0000 7ff953fb7740 4 client.19803 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:48:23.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.205+0000 7ff953fb7740 1 -- 172.21.15.70:0/2574769034 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7ff94c069a40 con 0x5561453ecc00 2024-04-01T01:48:23.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.205+0000 7ff94b7fe640 1 --2- 172.21.15.70:0/2574769034 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5561453ecc00 0x556145455a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:23.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff92eff5640 10 client.19803 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:23.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff94b7fe640 1 --2- 172.21.15.70:0/2574769034 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5561453ecc00 0x556145455a80 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e50dc669c2871415 server_cookie=db9277cd6ba83b9e in_seq=0 out_seq=0 2024-04-01T01:48:23.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff92eff5640 1 -- 172.21.15.70:0/2574769034 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7ff94c069a40 con 0x5561453ecc00 2024-04-01T01:48:23.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff92eff5640 10 client.19803 handle_command_reply: tid=0 2024-04-01T01:48:23.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff953fb7740 1 client.19803 shutdown 2024-04-01T01:48:23.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff953fb7740 1 -- 172.21.15.70:0/2574769034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453cf010 msgr2=0x5561453e6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:23.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff953fb7740 1 --2- 172.21.15.70:0/2574769034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453cf010 0x5561453e6540 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7ff93c01b040 tx=0x7ff93c005750 comp rx=0 tx=0).stop 2024-04-01T01:48:23.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff953fb7740 1 -- 172.21.15.70:0/2574769034 shutdown_connections 2024-04-01T01:48:23.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff953fb7740 1 --2- 172.21.15.70:0/2574769034 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5561453ecc00 0x556145455a80 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff953fb7740 1 --2- 172.21.15.70:0/2574769034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5561453d2170 0x5561453e7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff953fb7740 1 --2- 172.21.15.70:0/2574769034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561453cf010 0x5561453e6540 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff953fb7740 1 --2- 172.21.15.70:0/2574769034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5561453ce620 0x5561453e6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff953fb7740 1 -- 172.21.15.70:0/2574769034 >> 172.21.15.70:0/2574769034 conn(0x5561453c9c10 msgr2=0x5561453c9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:23.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff92eff5640 0 client.19803 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:23.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff953fb7740 1 -- 172.21.15.70:0/2574769034 shutdown_connections 2024-04-01T01:48:23.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff953fb7740 1 -- 172.21.15.70:0/2574769034 wait complete. 2024-04-01T01:48:23.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.206+0000 7ff953fb7740 20 client.19803 trim_cache size 0 max 16384 2024-04-01T01:48:23.214 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:48:23.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.207+0000 7ff94bfff640 1 -- 172.21.15.70:0/3162999938 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff9340a24a0 msgr2=0x7ff9340a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:23.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.207+0000 7ff94bfff640 1 --2- 172.21.15.70:0/3162999938 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff9340a24a0 0x7ff9340a4960 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff9400b9cc0 tx=0x7ff94002c750 comp rx=0 tx=0).stop 2024-04-01T01:48:23.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.207+0000 7ff94bfff640 1 -- 172.21.15.70:0/3162999938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff944096f70 msgr2=0x7ff94412c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:23.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.207+0000 7ff94bfff640 1 --2- 172.21.15.70:0/3162999938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff944096f70 0x7ff94412c530 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7ff940006010 tx=0x7ff94002de10 comp rx=0 tx=0).stop 2024-04-01T01:48:23.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.207+0000 7ff94bfff640 1 -- 172.21.15.70:0/3162999938 shutdown_connections 2024-04-01T01:48:23.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.207+0000 7ff94bfff640 1 --2- 172.21.15.70:0/3162999938 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff9340a24a0 0x7ff9340a4960 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.207+0000 7ff94bfff640 1 --2- 172.21.15.70:0/3162999938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff944097930 0x7ff9441335a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.207+0000 7ff94bfff640 1 --2- 172.21.15.70:0/3162999938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff944096f70 0x7ff94412c530 secure :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7ff940006010 tx=0x7ff94002de10 comp rx=0 tx=0).stop 2024-04-01T01:48:23.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.207+0000 7ff94bfff640 1 --2- 172.21.15.70:0/3162999938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff944095d80 0x7ff94412bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:23.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.207+0000 7ff94bfff640 1 -- 172.21.15.70:0/3162999938 >> 172.21.15.70:0/3162999938 conn(0x7ff9440912d0 msgr2=0x7ff94409a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:23.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.207+0000 7ff94bfff640 1 -- 172.21.15.70:0/3162999938 shutdown_connections 2024-04-01T01:48:23.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:23.208+0000 7ff94bfff640 1 -- 172.21.15.70:0/3162999938 wait complete. 2024-04-01T01:48:24.224 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:48:24.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:48:24.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:48:24.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.388+0000 7f990606f640 1 -- 172.21.15.70:0/582532459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9900101550 msgr2=0x7f9900107de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.388+0000 7f990606f640 1 --2- 172.21.15.70:0/582532459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9900101550 0x7f9900107de0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f98f8009ca0 tx=0x7f98f802c100 comp rx=0 tx=0).stop 2024-04-01T01:48:24.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.388+0000 7f990606f640 1 -- 172.21.15.70:0/582532459 shutdown_connections 2024-04-01T01:48:24.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.388+0000 7f990606f640 1 --2- 172.21.15.70:0/582532459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9900101550 0x7f9900107de0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.388+0000 7f990606f640 1 --2- 172.21.15.70:0/582532459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9900100b90 0x7f9900101010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.388+0000 7f990606f640 1 --2- 172.21.15.70:0/582532459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99000ff9a0 0x7f99000ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.388+0000 7f990606f640 1 -- 172.21.15.70:0/582532459 >> 172.21.15.70:0/582532459 conn(0x7f99000faf10 msgr2=0x7f99000fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:24.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.388+0000 7f990606f640 1 -- 172.21.15.70:0/582532459 shutdown_connections 2024-04-01T01:48:24.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.388+0000 7f990606f640 1 -- 172.21.15.70:0/582532459 wait complete. 2024-04-01T01:48:24.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.389+0000 7f990606f640 1 Processor -- start 2024-04-01T01:48:24.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.389+0000 7f990606f640 1 -- start start 2024-04-01T01:48:24.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.389+0000 7f990606f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99000ff9a0 0x7f9900199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.389+0000 7f990606f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9900100b90 0x7f990019a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.389+0000 7f990606f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9900101550 0x7f99001a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.389+0000 7f990606f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f990010d280 con 0x7f99000ff9a0 2024-04-01T01:48:24.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.389+0000 7f990606f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f990010d100 con 0x7f9900101550 2024-04-01T01:48:24.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.389+0000 7f990606f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f990010d400 con 0x7f9900100b90 2024-04-01T01:48:24.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.389+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99000ff9a0 0x7f9900199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.389+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99000ff9a0 0x7f9900199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35088/0 (socket says 172.21.15.70:35088) 2024-04-01T01:48:24.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.389+0000 7f98ff7fe640 1 -- 172.21.15.70:0/755013375 learned_addr learned my addr 172.21.15.70:0/755013375 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:24.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.390+0000 7f98fffff640 1 --2- 172.21.15.70:0/755013375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9900101550 0x7f99001a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.390+0000 7f98f6ffd640 1 --2- 172.21.15.70:0/755013375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9900100b90 0x7f990019a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.390+0000 7f98fffff640 1 -- 172.21.15.70:0/755013375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9900100b90 msgr2=0x7f990019a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.390+0000 7f98fffff640 1 --2- 172.21.15.70:0/755013375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9900100b90 0x7f990019a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.390+0000 7f98fffff640 1 -- 172.21.15.70:0/755013375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99000ff9a0 msgr2=0x7f9900199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.390+0000 7f98fffff640 1 --2- 172.21.15.70:0/755013375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99000ff9a0 0x7f9900199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.390+0000 7f98fffff640 1 -- 172.21.15.70:0/755013375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99001a1700 con 0x7f9900101550 2024-04-01T01:48:24.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.390+0000 7f98ff7fe640 1 --2- 172.21.15.70:0/755013375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99000ff9a0 0x7f9900199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:24.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.390+0000 7f98fffff640 1 --2- 172.21.15.70:0/755013375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9900101550 0x7f99001a1090 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f98f8009bf0 tx=0x7f98f8035ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:24.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.391+0000 7f98fd7fa640 1 -- 172.21.15.70:0/755013375 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8039040 con 0x7f9900101550 2024-04-01T01:48:24.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.391+0000 7f98fd7fa640 1 -- 172.21.15.70:0/755013375 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98f8035e50 con 0x7f9900101550 2024-04-01T01:48:24.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.392+0000 7f98fd7fa640 1 -- 172.21.15.70:0/755013375 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8004430 con 0x7f9900101550 2024-04-01T01:48:24.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.392+0000 7f990606f640 1 -- 172.21.15.70:0/755013375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99001a1990 con 0x7f9900101550 2024-04-01T01:48:24.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.392+0000 7f990606f640 1 -- 172.21.15.70:0/755013375 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99001a1d40 con 0x7f9900101550 2024-04-01T01:48:24.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.397+0000 7f98fd7fa640 1 -- 172.21.15.70:0/755013375 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f98f8046030 con 0x7f9900101550 2024-04-01T01:48:24.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.398+0000 7f98fd7fa640 1 --2- 172.21.15.70:0/755013375 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f98ec0a2550 0x7f98ec0a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.398+0000 7f98fd7fa640 1 -- 172.21.15.70:0/755013375 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f98f80f04e0 con 0x7f9900101550 2024-04-01T01:48:24.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.399+0000 7f98ff7fe640 1 --2- 172.21.15.70:0/755013375 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f98ec0a2550 0x7f98ec0a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.399+0000 7f9908c51740 1 Processor -- start 2024-04-01T01:48:24.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.399+0000 7f9908c51740 1 -- start start 2024-04-01T01:48:24.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.399+0000 7f9908c51740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728251220 0x555728251640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.399+0000 7f9908c51740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728251c10 0x555728256710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.399+0000 7f9908c51740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728256d80 0x555728259210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.399+0000 7f9908c51740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55572814a0b0 con 0x555728251c10 2024-04-01T01:48:24.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.399+0000 7f9908c51740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5557281acea0 con 0x555728251220 2024-04-01T01:48:24.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.399+0000 7f9908c51740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555728155c00 con 0x555728256d80 2024-04-01T01:48:24.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.399+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728251220 0x555728251640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.399+0000 7f98fffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728256d80 0x555728259210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.399+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728251220 0x555728251640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41566/0 (socket says 172.21.15.70:41566) 2024-04-01T01:48:24.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.399+0000 7f98ff7fe640 1 -- 172.21.15.70:0/973803996 learned_addr learned my addr 172.21.15.70:0/973803996 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:24.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.399+0000 7f98ff7fe640 1 --2- 172.21.15.70:0/755013375 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f98ec0a2550 0x7f98ec0a4a10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f98f000f4d0 tx=0x7f98f000a3b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:24.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.400+0000 7f98ff7fe640 1 -- 172.21.15.70:0/973803996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728256d80 msgr2=0x555728259210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:24.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.400+0000 7f98ff7fe640 1 --2- 172.21.15.70:0/973803996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728256d80 0x555728259210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.400+0000 7f98ff7fe640 1 -- 172.21.15.70:0/973803996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728251c10 msgr2=0x555728256710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:24.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.400+0000 7f98ff7fe640 1 --2- 172.21.15.70:0/973803996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728251c10 0x555728256710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.400+0000 7f98ff7fe640 1 -- 172.21.15.70:0/973803996 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555727f897e0 con 0x555728251220 2024-04-01T01:48:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.400+0000 7f98ff7fe640 1 --2- 172.21.15.70:0/973803996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728251220 0x555728251640 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f98f00107e0 tx=0x7f98f0012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b1a3caea2a5302b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.400+0000 7f98f5ffb640 1 -- 172.21.15.70:0/973803996 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f0016070 con 0x555728251220 2024-04-01T01:48:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.401+0000 7f98f5ffb640 1 -- 172.21.15.70:0/973803996 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98f001c420 con 0x555728251220 2024-04-01T01:48:24.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.401+0000 7f98f5ffb640 1 -- 172.21.15.70:0/973803996 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f0019600 con 0x555728251220 2024-04-01T01:48:24.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.403+0000 7f9908c51740 1 -- 172.21.15.70:0/973803996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728251220 msgr2=0x555728251640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.403+0000 7f9908c51740 1 --2- 172.21.15.70:0/973803996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728251220 0x555728251640 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f98f00107e0 tx=0x7f98f0012000 comp rx=0 tx=0).stop 2024-04-01T01:48:24.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.403+0000 7f9908c51740 1 -- 172.21.15.70:0/973803996 shutdown_connections 2024-04-01T01:48:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.403+0000 7f9908c51740 1 --2- 172.21.15.70:0/973803996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728256d80 0x555728259210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.403+0000 7f9908c51740 1 --2- 172.21.15.70:0/973803996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728251c10 0x555728256710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.403+0000 7f9908c51740 1 --2- 172.21.15.70:0/973803996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728251220 0x555728251640 secure :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f98f00107e0 tx=0x7f98f0012000 comp rx=0 tx=0).stop 2024-04-01T01:48:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.403+0000 7f9908c51740 1 -- 172.21.15.70:0/973803996 >> 172.21.15.70:0/973803996 conn(0x555728243f80 msgr2=0x555728244380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.403+0000 7f9908c51740 1 -- 172.21.15.70:0/973803996 shutdown_connections 2024-04-01T01:48:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.403+0000 7f9908c51740 1 -- 172.21.15.70:0/973803996 wait complete. 2024-04-01T01:48:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.404+0000 7f9908c51740 1 -- start start 2024-04-01T01:48:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.404+0000 7f9908c51740 1 Processor -- start 2024-04-01T01:48:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.404+0000 7f9908c51740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728251c10 0x55572825b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.404+0000 7f9908c51740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728256d80 0x55572825bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.404+0000 7f9908c51740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728260d60 0x555728263150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.404+0000 7f9908c51740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55572814b3d0 con 0x555728256d80 2024-04-01T01:48:24.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.404+0000 7f9908c51740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5557281a8f40 con 0x555728251c10 2024-04-01T01:48:24.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.404+0000 7f9908c51740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555728259d30 con 0x555728260d60 2024-04-01T01:48:24.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.404+0000 7f98f6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728256d80 0x55572825bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.404+0000 7f9908c51740 10 client.0 _ll_register_callbacks cb 0x555728168b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:24.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.404+0000 7f98f6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728256d80 0x55572825bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35106/0 (socket says 172.21.15.70:35106) 2024-04-01T01:48:24.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.404+0000 7f98f6ffd640 1 -- 172.21.15.70:0/852090443 learned_addr learned my addr 172.21.15.70:0/852090443 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:24.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.405+0000 7f98fffff640 1 --2- 172.21.15.70:0/852090443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728260d60 0x555728263150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.405+0000 7f98ff7fe640 1 --2- 172.21.15.70:0/852090443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728251c10 0x55572825b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.405+0000 7f98f6ffd640 1 -- 172.21.15.70:0/852090443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728260d60 msgr2=0x555728263150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:24.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.405+0000 7f98f6ffd640 1 --2- 172.21.15.70:0/852090443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728260d60 0x555728263150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.405+0000 7f98f6ffd640 1 -- 172.21.15.70:0/852090443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728251c10 msgr2=0x55572825b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:24.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.405+0000 7f98f6ffd640 1 --2- 172.21.15.70:0/852090443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728251c10 0x55572825b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.405+0000 7f98f6ffd640 1 -- 172.21.15.70:0/852090443 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555727fe9f70 con 0x555728256d80 2024-04-01T01:48:24.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.405+0000 7f98ff7fe640 1 --2- 172.21.15.70:0/852090443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728251c10 0x55572825b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:48:24.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.405+0000 7f98f6ffd640 1 --2- 172.21.15.70:0/852090443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728256d80 0x55572825bd30 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f98e800bd10 tx=0x7f98e800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4fb489c46cd8783 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:24.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.405+0000 7f98deffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:48:24.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.406+0000 7f98deffd640 1 -- 172.21.15.70:0/852090443 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e8015070 con 0x555728256d80 2024-04-01T01:48:24.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.406+0000 7f98deffd640 1 -- 172.21.15.70:0/852090443 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f98e801a660 con 0x555728256d80 2024-04-01T01:48:24.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.406+0000 7f9908c51740 1 -- 172.21.15.70:0/852090443 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98f001c420 con 0x555728256d80 2024-04-01T01:48:24.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.407+0000 7f98deffd640 1 -- 172.21.15.70:0/852090443 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f98e801a860 con 0x555728256d80 2024-04-01T01:48:24.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.407+0000 7f9908c51740 10 client.19812 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:24.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.407+0000 7f9908c51740 1 -- 172.21.15.70:0/852090443 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555727e81170 con 0x555728256d80 2024-04-01T01:48:24.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.407+0000 7f98deffd640 1 -- 172.21.15.70:0/852090443 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98f001c420 con 0x555728256d80 2024-04-01T01:48:24.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.407+0000 7f98deffd640 1 -- 172.21.15.70:0/852090443 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f98e8011bf0 con 0x555728256d80 2024-04-01T01:48:24.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.407+0000 7f9908c51740 10 client.19812 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:24.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.407+0000 7f9908c51740 10 client.19812 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:24.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.407+0000 7f9908c51740 20 client.19812 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:24.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.407+0000 7f9908c51740 1 --2- 172.21.15.70:0/852090443 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5557282676f0 0x555728269b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.407+0000 7f9908c51740 4 client.19812 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:24.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.407+0000 7f9908c51740 1 -- 172.21.15.70:0/852090443 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557281ba3e0 con 0x5557282676f0 2024-04-01T01:48:24.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.409+0000 7f98ff7fe640 1 --2- 172.21.15.70:0/852090443 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5557282676f0 0x555728269b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.410+0000 7f98ff7fe640 1 --2- 172.21.15.70:0/852090443 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5557282676f0 0x555728269b30 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=565e53bf3c20930c server_cookie=450b6d33ce3038ce in_seq=0 out_seq=0 2024-04-01T01:48:24.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.410+0000 7f98deffd640 10 client.19812 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:24.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.411+0000 7f98deffd640 1 -- 172.21.15.70:0/852090443 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5557281ba3e0 con 0x5557282676f0 2024-04-01T01:48:24.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.411+0000 7f98deffd640 10 client.19812 handle_command_reply: tid=0 2024-04-01T01:48:24.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.411+0000 7f9908c51740 1 client.19812 shutdown 2024-04-01T01:48:24.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.411+0000 7f9908c51740 1 -- 172.21.15.70:0/852090443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728256d80 msgr2=0x55572825bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.412+0000 7f9908c51740 1 --2- 172.21.15.70:0/852090443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728256d80 0x55572825bd30 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f98e800bd10 tx=0x7f98e800f730 comp rx=0 tx=0).stop 2024-04-01T01:48:24.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.413+0000 7f9908c51740 1 -- 172.21.15.70:0/852090443 shutdown_connections 2024-04-01T01:48:24.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.413+0000 7f9908c51740 1 --2- 172.21.15.70:0/852090443 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5557282676f0 0x555728269b30 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.413+0000 7f9908c51740 1 --2- 172.21.15.70:0/852090443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728260d60 0x555728263150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.413+0000 7f9908c51740 1 --2- 172.21.15.70:0/852090443 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728256d80 0x55572825bd30 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.413+0000 7f9908c51740 1 --2- 172.21.15.70:0/852090443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728251c10 0x55572825b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.413+0000 7f98deffd640 0 client.19812 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:24.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.413+0000 7f9908c51740 1 -- 172.21.15.70:0/852090443 >> 172.21.15.70:0/852090443 conn(0x555728243f80 msgr2=0x555728243180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:24.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.413+0000 7f9908c51740 1 -- 172.21.15.70:0/852090443 shutdown_connections 2024-04-01T01:48:24.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.413+0000 7f9908c51740 1 -- 172.21.15.70:0/852090443 wait complete. 2024-04-01T01:48:24.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.413+0000 7f9908c51740 20 client.19812 trim_cache size 0 max 16384 2024-04-01T01:48:24.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.453+0000 7f9908c51740 1 Processor -- start 2024-04-01T01:48:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.453+0000 7f9908c51740 1 -- start start 2024-04-01T01:48:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.453+0000 7f9908c51740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728251c10 0x5557282428f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.453+0000 7f9908c51740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728256d80 0x555728242e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.453+0000 7f9908c51740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728260d60 0x555728243370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.453+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728251c10 0x5557282428f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.454+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728251c10 0x5557282428f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42374/0 (socket says 172.21.15.70:42374) 2024-04-01T01:48:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.454+0000 7f98f6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728256d80 0x555728242e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.454+0000 7f98f6ffd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728256d80 0x555728242e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41588/0 (socket says 172.21.15.70:41588) 2024-04-01T01:48:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.454+0000 7f98ff7fe640 1 -- 172.21.15.70:0/2901063336 learned_addr learned my addr 172.21.15.70:0/2901063336 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:24.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.454+0000 7f9908c51740 1 -- 172.21.15.70:0/2901063336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f99001a8060 con 0x555728260d60 2024-04-01T01:48:24.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.454+0000 7f9908c51740 1 -- 172.21.15.70:0/2901063336 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f99001a9ba0 con 0x555728256d80 2024-04-01T01:48:24.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.454+0000 7f9908c51740 1 -- 172.21.15.70:0/2901063336 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f99001a8730 con 0x555728251c10 2024-04-01T01:48:24.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.454+0000 7f98fffff640 1 --2- 172.21.15.70:0/2901063336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728260d60 0x555728243370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.455+0000 7f98ff7fe640 1 -- 172.21.15.70:0/2901063336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728256d80 msgr2=0x555728242e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:24.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.455+0000 7f98ff7fe640 1 --2- 172.21.15.70:0/2901063336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728256d80 0x555728242e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.455+0000 7f98ff7fe640 1 -- 172.21.15.70:0/2901063336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728260d60 msgr2=0x555728243370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:24.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.455+0000 7f98ff7fe640 1 --2- 172.21.15.70:0/2901063336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728260d60 0x555728243370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.455+0000 7f98ff7fe640 1 -- 172.21.15.70:0/2901063336 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555728259eb0 con 0x555728251c10 2024-04-01T01:48:24.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.455+0000 7f98ff7fe640 1 --2- 172.21.15.70:0/2901063336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728251c10 0x5557282428f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f98f001b780 tx=0x7f98f001e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80767885c1ed7667 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:24.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.456+0000 7f98f57fa640 1 -- 172.21.15.70:0/2901063336 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f0016040 con 0x555728251c10 2024-04-01T01:48:24.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.456+0000 7f98f57fa640 1 -- 172.21.15.70:0/2901063336 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98f001c810 con 0x555728251c10 2024-04-01T01:48:24.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.456+0000 7f98f57fa640 1 -- 172.21.15.70:0/2901063336 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f00334c0 con 0x555728251c10 2024-04-01T01:48:24.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.461+0000 7f9908c51740 1 -- 172.21.15.70:0/2901063336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728251c10 msgr2=0x5557282428f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.461+0000 7f9908c51740 1 --2- 172.21.15.70:0/2901063336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728251c10 0x5557282428f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f98f001b780 tx=0x7f98f001e040 comp rx=0 tx=0).stop 2024-04-01T01:48:24.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.462+0000 7f9908c51740 1 -- 172.21.15.70:0/2901063336 shutdown_connections 2024-04-01T01:48:24.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.462+0000 7f9908c51740 1 --2- 172.21.15.70:0/2901063336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728260d60 0x555728243370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.462+0000 7f9908c51740 1 --2- 172.21.15.70:0/2901063336 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728256d80 0x555728242e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.462+0000 7f9908c51740 1 --2- 172.21.15.70:0/2901063336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728251c10 0x5557282428f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.462+0000 7f9908c51740 1 -- 172.21.15.70:0/2901063336 >> 172.21.15.70:0/2901063336 conn(0x555728243f80 msgr2=0x5557282532f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:24.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.462+0000 7f9908c51740 1 -- 172.21.15.70:0/2901063336 shutdown_connections 2024-04-01T01:48:24.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.462+0000 7f9908c51740 1 -- 172.21.15.70:0/2901063336 wait complete. 2024-04-01T01:48:24.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.462+0000 7f9908c51740 1 -- start start 2024-04-01T01:48:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.462+0000 7f9908c51740 1 Processor -- start 2024-04-01T01:48:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.463+0000 7f9908c51740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728251c10 0x55572827aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.463+0000 7f9908c51740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728256d80 0x5557282790d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.463+0000 7f9908c51740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728260d60 0x555728279610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.463+0000 7f9908c51740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f99001a94d0 con 0x555728256d80 2024-04-01T01:48:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.463+0000 7f9908c51740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f99001a8e00 con 0x555728260d60 2024-04-01T01:48:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.463+0000 7f9908c51740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f99001a6040 con 0x555728251c10 2024-04-01T01:48:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.463+0000 7f9908c51740 10 client.0 _ll_register_callbacks cb 0x7f98e800b8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:24.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.463+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728251c10 0x55572827aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.463+0000 7f98f6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728256d80 0x5557282790d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.463+0000 7f98ff7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728251c10 0x55572827aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42384/0 (socket says 172.21.15.70:42384) 2024-04-01T01:48:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.463+0000 7f98ff7fe640 1 -- 172.21.15.70:0/323034616 learned_addr learned my addr 172.21.15.70:0/323034616 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.463+0000 7f98fffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728260d60 0x555728279610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.464+0000 7f98fffff640 1 -- 172.21.15.70:0/323034616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728251c10 msgr2=0x55572827aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.464+0000 7f98fffff640 1 --2- 172.21.15.70:0/323034616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728251c10 0x55572827aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.464+0000 7f98fffff640 1 -- 172.21.15.70:0/323034616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728256d80 msgr2=0x5557282790d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.464+0000 7f98fffff640 1 --2- 172.21.15.70:0/323034616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728256d80 0x5557282790d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.464+0000 7f98fffff640 1 -- 172.21.15.70:0/323034616 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555728279d50 con 0x555728260d60 2024-04-01T01:48:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.464+0000 7f98ff7fe640 1 --2- 172.21.15.70:0/323034616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728251c10 0x55572827aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:24.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.464+0000 7f98f6ffd640 1 --2- 172.21.15.70:0/323034616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728256d80 0x5557282790d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:24.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.464+0000 7f98fffff640 1 --2- 172.21.15.70:0/323034616 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728260d60 0x555728279610 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f98f80bc900 tx=0x7f98f8009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f9c35f412c51ed9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:24.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.464+0000 7f98deffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:24.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.465+0000 7f98deffd640 1 -- 172.21.15.70:0/323034616 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f80c0d50 con 0x555728260d60 2024-04-01T01:48:24.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.465+0000 7f98deffd640 1 -- 172.21.15.70:0/323034616 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f98f8003970 con 0x555728260d60 2024-04-01T01:48:24.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.465+0000 7f9908c51740 1 -- 172.21.15.70:0/323034616 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98f001c810 con 0x555728260d60 2024-04-01T01:48:24.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.465+0000 7f98deffd640 1 -- 172.21.15.70:0/323034616 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f98f80c5580 con 0x555728260d60 2024-04-01T01:48:24.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.465+0000 7f9908c51740 10 client.29915 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:24.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.465+0000 7f9908c51740 1 -- 172.21.15.70:0/323034616 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5557282ce450 con 0x555728260d60 2024-04-01T01:48:24.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.466+0000 7f98deffd640 1 -- 172.21.15.70:0/323034616 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98f001c810 con 0x555728260d60 2024-04-01T01:48:24.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.466+0000 7f98deffd640 1 -- 172.21.15.70:0/323034616 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f98f803c030 con 0x555728260d60 2024-04-01T01:48:24.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.466+0000 7f9908c51740 10 client.29915 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:24.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.466+0000 7f9908c51740 10 client.29915 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:24.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.466+0000 7f9908c51740 20 client.29915 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:24.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.466+0000 7f9908c51740 1 --2- 172.21.15.70:0/323034616 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5557282cece0 0x5557282d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.466+0000 7f9908c51740 4 client.29915 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:48:24.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.466+0000 7f9908c51740 1 -- 172.21.15.70:0/323034616 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55572825b5e0 con 0x5557282cece0 2024-04-01T01:48:24.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.466+0000 7f98ff7fe640 1 --2- 172.21.15.70:0/323034616 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5557282cece0 0x5557282d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.469+0000 7f98ff7fe640 1 --2- 172.21.15.70:0/323034616 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5557282cece0 0x5557282d1100 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea89a07f38ed246c server_cookie=979a9d3ff12dea7 in_seq=0 out_seq=0 2024-04-01T01:48:24.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.470+0000 7f98deffd640 10 client.29915 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:24.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.471+0000 7f98deffd640 1 -- 172.21.15.70:0/323034616 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55572825b5e0 con 0x5557282cece0 2024-04-01T01:48:24.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.471+0000 7f98deffd640 10 client.29915 handle_command_reply: tid=0 2024-04-01T01:48:24.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.471+0000 7f9908c51740 1 client.29915 shutdown 2024-04-01T01:48:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.471+0000 7f9908c51740 1 -- 172.21.15.70:0/323034616 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728260d60 msgr2=0x555728279610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.471+0000 7f9908c51740 1 --2- 172.21.15.70:0/323034616 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728260d60 0x555728279610 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f98f80bc900 tx=0x7f98f8009650 comp rx=0 tx=0).stop 2024-04-01T01:48:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.472+0000 7f9908c51740 1 -- 172.21.15.70:0/323034616 shutdown_connections 2024-04-01T01:48:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.472+0000 7f9908c51740 1 --2- 172.21.15.70:0/323034616 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5557282cece0 0x5557282d1100 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.472+0000 7f9908c51740 1 --2- 172.21.15.70:0/323034616 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555728260d60 0x555728279610 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.472+0000 7f9908c51740 1 --2- 172.21.15.70:0/323034616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555728256d80 0x5557282790d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.472+0000 7f9908c51740 1 --2- 172.21.15.70:0/323034616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555728251c10 0x55572827aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.472+0000 7f9908c51740 1 -- 172.21.15.70:0/323034616 >> 172.21.15.70:0/323034616 conn(0x555728243f80 msgr2=0x5557282471e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.472+0000 7f98deffd640 0 client.29915 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.472+0000 7f9908c51740 1 -- 172.21.15.70:0/323034616 shutdown_connections 2024-04-01T01:48:24.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.472+0000 7f9908c51740 1 -- 172.21.15.70:0/323034616 wait complete. 2024-04-01T01:48:24.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.472+0000 7f9908c51740 20 client.29915 trim_cache size 0 max 16384 2024-04-01T01:48:24.474 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:48:24.474 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:48:24.474 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "00c3f15a-6e7f-4512-a617-06c2ffd7d58c", 2024-04-01T01:48:24.474 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:48:24.474 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:48:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.473+0000 7f990606f640 1 -- 172.21.15.70:0/755013375 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f98ec0a2550 msgr2=0x7f98ec0a4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.473+0000 7f990606f640 1 --2- 172.21.15.70:0/755013375 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f98ec0a2550 0x7f98ec0a4a10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f98f000f4d0 tx=0x7f98f000a3b0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.473+0000 7f990606f640 1 -- 172.21.15.70:0/755013375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9900101550 msgr2=0x7f99001a1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.473+0000 7f990606f640 1 --2- 172.21.15.70:0/755013375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9900101550 0x7f99001a1090 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f98f8009bf0 tx=0x7f98f8035ce0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.474+0000 7f990606f640 1 -- 172.21.15.70:0/755013375 shutdown_connections 2024-04-01T01:48:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.474+0000 7f990606f640 1 --2- 172.21.15.70:0/755013375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9900101550 0x7f99001a1090 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.474+0000 7f990606f640 1 --2- 172.21.15.70:0/755013375 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f98ec0a2550 0x7f98ec0a4a10 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.474+0000 7f990606f640 1 --2- 172.21.15.70:0/755013375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9900100b90 0x7f990019a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.474+0000 7f990606f640 1 --2- 172.21.15.70:0/755013375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99000ff9a0 0x7f9900199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.474+0000 7f990606f640 1 -- 172.21.15.70:0/755013375 >> 172.21.15.70:0/755013375 conn(0x7f99000faf10 msgr2=0x7f9900109cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:24.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.475+0000 7f990606f640 1 -- 172.21.15.70:0/755013375 shutdown_connections 2024-04-01T01:48:24.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.475+0000 7f990606f640 1 -- 172.21.15.70:0/755013375 wait complete. 2024-04-01T01:48:24.494 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:48:24.494 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:48:24.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.664+0000 7f6ba77fe640 1 -- 172.21.15.70:0/2252443662 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b98005220 con 0x7f6ba0096860 2024-04-01T01:48:24.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.666+0000 7f6bad8d3640 1 -- 172.21.15.70:0/2252443662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ba0096860 msgr2=0x7f6ba0098c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.666+0000 7f6bad8d3640 1 --2- 172.21.15.70:0/2252443662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ba0096860 0x7f6ba0098c50 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009900 tx=0x7f6b9802bc30 comp rx=0 tx=0).stop 2024-04-01T01:48:24.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.666+0000 7f6bad8d3640 1 -- 172.21.15.70:0/2252443662 shutdown_connections 2024-04-01T01:48:24.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.666+0000 7f6bad8d3640 1 --2- 172.21.15.70:0/2252443662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ba009bac0 0x7f6ba009deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.666+0000 7f6bad8d3640 1 --2- 172.21.15.70:0/2252443662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba0099190 0x7f6ba009b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.666+0000 7f6bad8d3640 1 --2- 172.21.15.70:0/2252443662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ba0096860 0x7f6ba0098c50 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.666+0000 7f6bad8d3640 1 -- 172.21.15.70:0/2252443662 >> 172.21.15.70:0/2252443662 conn(0x7f6ba00901d0 msgr2=0x7f6ba0092630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:24.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.667+0000 7f6bad8d3640 1 -- 172.21.15.70:0/2252443662 shutdown_connections 2024-04-01T01:48:24.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.667+0000 7f6bad8d3640 1 -- 172.21.15.70:0/2252443662 wait complete. 2024-04-01T01:48:24.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.668+0000 7f6bad8d3640 1 Processor -- start 2024-04-01T01:48:24.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.668+0000 7f6bad8d3640 1 -- start start 2024-04-01T01:48:24.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.668+0000 7f6bad8d3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ba0096860 0x7f6ba00a4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.668+0000 7f6bad8d3640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba0099190 0x7f6ba00a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.668+0000 7f6bad8d3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ba009bac0 0x7f6ba00a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.668+0000 7f6bad8d3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba0095700 con 0x7f6ba0096860 2024-04-01T01:48:24.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.668+0000 7f6bad8d3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba0095580 con 0x7f6ba009bac0 2024-04-01T01:48:24.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.668+0000 7f6bad8d3640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba0095880 con 0x7f6ba0099190 2024-04-01T01:48:24.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.668+0000 7f6ba7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba0099190 0x7f6ba00a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.668+0000 7f6ba7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba0099190 0x7f6ba00a46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42396/0 (socket says 172.21.15.70:42396) 2024-04-01T01:48:24.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.668+0000 7f6ba7fff640 1 -- 172.21.15.70:0/897731032 learned_addr learned my addr 172.21.15.70:0/897731032 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:24.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.669+0000 7f6bad0d2640 1 --2- 172.21.15.70:0/897731032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ba009bac0 0x7f6ba00a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.669+0000 7f6bad0d2640 1 -- 172.21.15.70:0/897731032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba0099190 msgr2=0x7f6ba00a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.669+0000 7f6bad0d2640 1 --2- 172.21.15.70:0/897731032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba0099190 0x7f6ba00a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.669+0000 7f6bad0d2640 1 -- 172.21.15.70:0/897731032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ba0096860 msgr2=0x7f6ba00a4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.669+0000 7f6bad0d2640 1 --2- 172.21.15.70:0/897731032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ba0096860 0x7f6ba00a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.669+0000 7f6bad0d2640 1 -- 172.21.15.70:0/897731032 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba00a1590 con 0x7f6ba009bac0 2024-04-01T01:48:24.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.669+0000 7f6bad0d2640 1 --2- 172.21.15.70:0/897731032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ba009bac0 0x7f6ba00a0cd0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8067420 tx=0x7f6ba806b540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:24.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.673+0000 7f6ba5ffb640 1 -- 172.21.15.70:0/897731032 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba806c040 con 0x7f6ba009bac0 2024-04-01T01:48:24.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.673+0000 7f6ba5ffb640 1 -- 172.21.15.70:0/897731032 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6ba8061d90 con 0x7f6ba009bac0 2024-04-01T01:48:24.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.673+0000 7f6ba5ffb640 1 -- 172.21.15.70:0/897731032 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8069330 con 0x7f6ba009bac0 2024-04-01T01:48:24.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.674+0000 7f6bad8d3640 1 -- 172.21.15.70:0/897731032 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ba014b6c0 con 0x7f6ba009bac0 2024-04-01T01:48:24.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.674+0000 7f6bad8d3640 1 -- 172.21.15.70:0/897731032 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ba014ba70 con 0x7f6ba009bac0 2024-04-01T01:48:24.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.677+0000 7f6ba5ffb640 1 -- 172.21.15.70:0/897731032 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f6ba8073110 con 0x7f6ba009bac0 2024-04-01T01:48:24.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.677+0000 7f6bb17f0740 1 Processor -- start 2024-04-01T01:48:24.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.677+0000 7f6bb17f0740 1 -- start start 2024-04-01T01:48:24.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.678+0000 7f6bb17f0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e98490 0x55a2e0e988b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.678+0000 7f6bb17f0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 0x55a2e0e9d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.678+0000 7f6bb17f0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e9dff0 0x55a2e0ea0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.678+0000 7f6bb17f0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a2e0dfd180 con 0x55a2e0e98490 2024-04-01T01:48:24.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.678+0000 7f6bb17f0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a2e0da5b70 con 0x55a2e0e98e80 2024-04-01T01:48:24.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.678+0000 7f6bb17f0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a2e0df9220 con 0x55a2e0e9dff0 2024-04-01T01:48:24.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.678+0000 7f6ba7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e9dff0 0x55a2e0ea0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.678+0000 7f6ba7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e98490 0x55a2e0e988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.678+0000 7f6ba7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e98490 0x55a2e0e988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35138/0 (socket says 172.21.15.70:35138) 2024-04-01T01:48:24.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.678+0000 7f6ba7fff640 1 -- 172.21.15.70:0/2757851546 learned_addr learned my addr 172.21.15.70:0/2757851546 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:24.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.678+0000 7f6bac8d1640 1 --2- 172.21.15.70:0/2757851546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 0x55a2e0e9d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.679+0000 7f6ba5ffb640 1 --2- 172.21.15.70:0/897731032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6b900a2670 0x7f6b900a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.679+0000 7f6ba7fff640 1 -- 172.21.15.70:0/2757851546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e9dff0 msgr2=0x55a2e0ea0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:24.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.679+0000 7f6ba7fff640 1 --2- 172.21.15.70:0/2757851546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e9dff0 0x55a2e0ea0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.679+0000 7f6ba7fff640 1 -- 172.21.15.70:0/2757851546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 msgr2=0x55a2e0e9d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:24.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.679+0000 7f6ba7fff640 1 --2- 172.21.15.70:0/2757851546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 0x55a2e0e9d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.679+0000 7f6ba7fff640 1 -- 172.21.15.70:0/2757851546 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2e0db97c0 con 0x55a2e0e98490 2024-04-01T01:48:24.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.679+0000 7f6ba7fff640 1 --2- 172.21.15.70:0/2757851546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e98490 0x55a2e0e988b0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c009600 tx=0x7f6b9c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3dddf4d0769b8f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.679+0000 7f6ba5ffb640 1 -- 172.21.15.70:0/897731032 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6ba81318f0 con 0x7f6ba009bac0 2024-04-01T01:48:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.680+0000 7f6bad0d2640 1 --2- 172.21.15.70:0/897731032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6b900a2670 0x7f6b900a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.680+0000 7f6b8effd640 1 -- 172.21.15.70:0/2757851546 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c016030 con 0x55a2e0e98490 2024-04-01T01:48:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.680+0000 7f6b8effd640 1 -- 172.21.15.70:0/2757851546 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b9c00f7c0 con 0x55a2e0e98490 2024-04-01T01:48:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.680+0000 7f6b8effd640 1 -- 172.21.15.70:0/2757851546 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c019040 con 0x55a2e0e98490 2024-04-01T01:48:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.681+0000 7f6bb17f0740 1 -- 172.21.15.70:0/2757851546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e98490 msgr2=0x55a2e0e988b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.681+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/2757851546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e98490 0x55a2e0e988b0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c009600 tx=0x7f6b9c005f90 comp rx=0 tx=0).stop 2024-04-01T01:48:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.681+0000 7f6bad0d2640 1 --2- 172.21.15.70:0/897731032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6b900a2670 0x7f6b900a4b30 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8064c40 tx=0x7f6ba8069040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.681+0000 7f6bb17f0740 1 -- 172.21.15.70:0/2757851546 shutdown_connections 2024-04-01T01:48:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.681+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/2757851546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e9dff0 0x55a2e0ea0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.681+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/2757851546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 0x55a2e0e9d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.681+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/2757851546 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e98490 0x55a2e0e988b0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.681+0000 7f6bb17f0740 1 -- 172.21.15.70:0/2757851546 >> 172.21.15.70:0/2757851546 conn(0x55a2e0e93a00 msgr2=0x55a2e0e93e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.681+0000 7f6bb17f0740 1 -- 172.21.15.70:0/2757851546 shutdown_connections 2024-04-01T01:48:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.681+0000 7f6bb17f0740 1 -- 172.21.15.70:0/2757851546 wait complete. 2024-04-01T01:48:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.682+0000 7f6bb17f0740 1 -- start start 2024-04-01T01:48:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.682+0000 7f6bb17f0740 1 Processor -- start 2024-04-01T01:48:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.682+0000 7f6bb17f0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98490 0x55a2e0ea2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.682+0000 7f6bb17f0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 0x55a2e0ea2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.682+0000 7f6bb17f0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e9dff0 0x55a2e0ea9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.682+0000 7f6bb17f0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a2e0ea0fa0 con 0x55a2e0e9dff0 2024-04-01T01:48:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.682+0000 7f6bb17f0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55a2e0d9b4a0 con 0x55a2e0e98e80 2024-04-01T01:48:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.682+0000 7f6bb17f0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a2e0ea1120 con 0x55a2e0e98490 2024-04-01T01:48:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.682+0000 7f6bb17f0740 10 client.0 _ll_register_callbacks cb 0x55a2e0d9ed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.683+0000 7f6bac8d1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 0x55a2e0ea2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.683+0000 7f6bac8d1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 0x55a2e0ea2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41640/0 (socket says 172.21.15.70:41640) 2024-04-01T01:48:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.683+0000 7f6bac8d1640 1 -- 172.21.15.70:0/1486409252 learned_addr learned my addr 172.21.15.70:0/1486409252 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.683+0000 7f6ba7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e9dff0 0x55a2e0ea9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.683+0000 7f6ba7fff640 1 --2- 172.21.15.70:0/1486409252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98490 0x55a2e0ea2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.683+0000 7f6bac8d1640 1 -- 172.21.15.70:0/1486409252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98490 msgr2=0x55a2e0ea2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.683+0000 7f6bac8d1640 1 --2- 172.21.15.70:0/1486409252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98490 0x55a2e0ea2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.683+0000 7f6bac8d1640 1 -- 172.21.15.70:0/1486409252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e9dff0 msgr2=0x55a2e0ea9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.683+0000 7f6bac8d1640 1 --2- 172.21.15.70:0/1486409252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e9dff0 0x55a2e0ea9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.683+0000 7f6bac8d1640 1 -- 172.21.15.70:0/1486409252 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2e0bdb7d0 con 0x55a2e0e98e80 2024-04-01T01:48:24.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.684+0000 7f6b87fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:24.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.684+0000 7f6bac8d1640 1 --2- 172.21.15.70:0/1486409252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 0x55a2e0ea2f40 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f6b9802c170 tx=0x7f6b98004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85115f58dad22d1e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:24.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.685+0000 7f6b87fff640 1 -- 172.21.15.70:0/1486409252 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b98042070 con 0x55a2e0e98e80 2024-04-01T01:48:24.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.685+0000 7f6bb17f0740 1 -- 172.21.15.70:0/1486409252 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b9c00f7c0 con 0x55a2e0e98e80 2024-04-01T01:48:24.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.685+0000 7f6b87fff640 1 -- 172.21.15.70:0/1486409252 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6b98005240 con 0x55a2e0e98e80 2024-04-01T01:48:24.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.685+0000 7f6b87fff640 1 -- 172.21.15.70:0/1486409252 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6b9804e050 con 0x55a2e0e98e80 2024-04-01T01:48:24.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.685+0000 7f6bb17f0740 10 client.29927 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:24.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.685+0000 7f6bb17f0740 1 -- 172.21.15.70:0/1486409252 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a2e0b113f0 con 0x55a2e0e98e80 2024-04-01T01:48:24.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.686+0000 7f6b87fff640 1 -- 172.21.15.70:0/1486409252 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b9c00f7c0 con 0x55a2e0e98e80 2024-04-01T01:48:24.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.686+0000 7f6b87fff640 1 -- 172.21.15.70:0/1486409252 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6b98004b70 con 0x55a2e0e98e80 2024-04-01T01:48:24.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.686+0000 7f6bb17f0740 10 client.29927 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:24.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.686+0000 7f6bb17f0740 10 client.29927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.686+0000 7f6bb17f0740 20 client.29927 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.686+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/1486409252 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a2e0ebf910 0x55a2e0ec1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.686+0000 7f6bb17f0740 4 client.29927 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.686+0000 7f6bb17f0740 1 -- 172.21.15.70:0/1486409252 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2e0e0a6c0 con 0x55a2e0ebf910 2024-04-01T01:48:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.686+0000 7f6ba7fff640 1 --2- 172.21.15.70:0/1486409252 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a2e0ebf910 0x55a2e0ec1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.687+0000 7f6b87fff640 10 client.29927 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.687+0000 7f6ba7fff640 1 --2- 172.21.15.70:0/1486409252 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a2e0ebf910 0x55a2e0ec1d50 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff45601d4a40b902 server_cookie=2f2ab4aefcce7b44 in_seq=0 out_seq=0 2024-04-01T01:48:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.688+0000 7f6b87fff640 1 -- 172.21.15.70:0/1486409252 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a2e0e0a6c0 con 0x55a2e0ebf910 2024-04-01T01:48:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.688+0000 7f6b87fff640 10 client.29927 handle_command_reply: tid=0 2024-04-01T01:48:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.688+0000 7f6bb17f0740 1 client.29927 shutdown 2024-04-01T01:48:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.689+0000 7f6bb17f0740 1 -- 172.21.15.70:0/1486409252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 msgr2=0x55a2e0ea2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.689+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/1486409252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 0x55a2e0ea2f40 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f6b9802c170 tx=0x7f6b98004250 comp rx=0 tx=0).stop 2024-04-01T01:48:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.689+0000 7f6bb17f0740 1 -- 172.21.15.70:0/1486409252 shutdown_connections 2024-04-01T01:48:24.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.689+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/1486409252 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a2e0ebf910 0x55a2e0ec1d50 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.689+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/1486409252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e9dff0 0x55a2e0ea9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.689+0000 7f6b87fff640 0 client.29927 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.689+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/1486409252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 0x55a2e0ea2f40 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.689+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/1486409252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98490 0x55a2e0ea2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.689+0000 7f6bb17f0740 1 -- 172.21.15.70:0/1486409252 >> 172.21.15.70:0/1486409252 conn(0x55a2e0e93a00 msgr2=0x55a2e0e92c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.689+0000 7f6bb17f0740 1 -- 172.21.15.70:0/1486409252 shutdown_connections 2024-04-01T01:48:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.689+0000 7f6bb17f0740 1 -- 172.21.15.70:0/1486409252 wait complete. 2024-04-01T01:48:24.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.690+0000 7f6bb17f0740 20 client.29927 trim_cache size 0 max 16384 2024-04-01T01:48:24.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.718+0000 7f6bb17f0740 1 Processor -- start 2024-04-01T01:48:24.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.718+0000 7f6bb17f0740 1 -- start start 2024-04-01T01:48:24.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.718+0000 7f6bb17f0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e98490 0x55a2e0ec1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.718+0000 7f6bb17f0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98e80 0x55a2e0ea9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.718+0000 7f6bb17f0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e9dff0 0x55a2e0ea1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.718+0000 7f6bb17f0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba014d730 con 0x55a2e0e98490 2024-04-01T01:48:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.718+0000 7f6ba7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e98490 0x55a2e0ec1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.718+0000 7f6ba7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e98490 0x55a2e0ec1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:35156/0 (socket says 172.21.15.70:35156) 2024-04-01T01:48:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.718+0000 7f6ba7fff640 1 -- 172.21.15.70:0/2551409611 learned_addr learned my addr 172.21.15.70:0/2551409611 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.718+0000 7f6bac8d1640 1 --2- 172.21.15.70:0/2551409611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98e80 0x55a2e0ea9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.719+0000 7f6bb17f0740 1 -- 172.21.15.70:0/2551409611 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba014f270 con 0x55a2e0e9dff0 2024-04-01T01:48:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.719+0000 7f6bb17f0740 1 -- 172.21.15.70:0/2551409611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba014de00 con 0x55a2e0e98e80 2024-04-01T01:48:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.719+0000 7f6ba7fff640 1 --2- 172.21.15.70:0/2551409611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e9dff0 0x55a2e0ea1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.719+0000 7f6ba7fff640 1 -- 172.21.15.70:0/2551409611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98e80 msgr2=0x55a2e0ea9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.719+0000 7f6ba7fff640 1 --2- 172.21.15.70:0/2551409611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98e80 0x55a2e0ea9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.719+0000 7f6ba7fff640 1 -- 172.21.15.70:0/2551409611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e9dff0 msgr2=0x55a2e0ea1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.719+0000 7f6ba7fff640 1 --2- 172.21.15.70:0/2551409611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e9dff0 0x55a2e0ea1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.719+0000 7f6ba7fff640 1 -- 172.21.15.70:0/2551409611 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2e0ad10f0 con 0x55a2e0e98490 2024-04-01T01:48:24.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.719+0000 7f6ba7fff640 1 --2- 172.21.15.70:0/2551409611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e98490 0x55a2e0ec1d30 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c0058c0 tx=0x7f6b9c009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e29e3af879ff601 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:24.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.719+0000 7f6b8e7fc640 1 -- 172.21.15.70:0/2551409611 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c016040 con 0x55a2e0e98490 2024-04-01T01:48:24.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.719+0000 7f6b8e7fc640 1 -- 172.21.15.70:0/2551409611 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a2e0e0a6c0 con 0x55a2e0e98490 2024-04-01T01:48:24.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.719+0000 7f6b8e7fc640 1 -- 172.21.15.70:0/2551409611 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c029040 con 0x55a2e0e98490 2024-04-01T01:48:24.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.720+0000 7f6bb17f0740 1 -- 172.21.15.70:0/2551409611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e98490 msgr2=0x55a2e0ec1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.720+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/2551409611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e98490 0x55a2e0ec1d30 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c0058c0 tx=0x7f6b9c009530 comp rx=0 tx=0).stop 2024-04-01T01:48:24.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.722+0000 7f6bb17f0740 1 -- 172.21.15.70:0/2551409611 shutdown_connections 2024-04-01T01:48:24.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.722+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/2551409611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e9dff0 0x55a2e0ea1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.722+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/2551409611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98e80 0x55a2e0ea9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.722+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/2551409611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e98490 0x55a2e0ec1d30 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.722+0000 7f6bb17f0740 1 -- 172.21.15.70:0/2551409611 >> 172.21.15.70:0/2551409611 conn(0x55a2e0e93a00 msgr2=0x55a2e0e93de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:24.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.722+0000 7f6bb17f0740 1 -- 172.21.15.70:0/2551409611 shutdown_connections 2024-04-01T01:48:24.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.722+0000 7f6bb17f0740 1 -- 172.21.15.70:0/2551409611 wait complete. 2024-04-01T01:48:24.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.722+0000 7f6bb17f0740 1 -- start start 2024-04-01T01:48:24.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.723+0000 7f6bb17f0740 1 Processor -- start 2024-04-01T01:48:24.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.723+0000 7f6bb17f0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98490 0x55a2e0ea75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.723+0000 7f6bb17f0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 0x55a2e0ea7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.723+0000 7f6bb17f0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e9dff0 0x55a2e0ea8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.723+0000 7f6ba7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98490 0x55a2e0ea75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.723+0000 7f6ba7fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98490 0x55a2e0ea75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:42420/0 (socket says 172.21.15.70:42420) 2024-04-01T01:48:24.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.723+0000 7f6bac8d1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 0x55a2e0ea7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.723+0000 7f6bac8d1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 0x55a2e0ea7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41668/0 (socket says 172.21.15.70:41668) 2024-04-01T01:48:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.723+0000 7f6ba7fff640 1 -- 172.21.15.70:0/3468414093 learned_addr learned my addr 172.21.15.70:0/3468414093 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.724+0000 7f6bb17f0740 1 -- 172.21.15.70:0/3468414093 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba014eba0 con 0x55a2e0e9dff0 2024-04-01T01:48:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.724+0000 7f6bb17f0740 1 -- 172.21.15.70:0/3468414093 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba014e4d0 con 0x55a2e0e98e80 2024-04-01T01:48:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.724+0000 7f6bb17f0740 1 -- 172.21.15.70:0/3468414093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba01471e0 con 0x55a2e0e98490 2024-04-01T01:48:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.724+0000 7f6bb17f0740 10 client.0 _ll_register_callbacks cb 0x7f6ba8067a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.724+0000 7f6ba7fff640 1 -- 172.21.15.70:0/3468414093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 msgr2=0x55a2e0ea7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.724+0000 7f6ba7fff640 1 --2- 172.21.15.70:0/3468414093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 0x55a2e0ea7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.724+0000 7f6ba7fff640 1 -- 172.21.15.70:0/3468414093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e9dff0 msgr2=0x55a2e0ea8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.724+0000 7f6ba7fff640 1 --2- 172.21.15.70:0/3468414093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e9dff0 0x55a2e0ea8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.724+0000 7f6ba7fff640 1 -- 172.21.15.70:0/3468414093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2e0ea90f0 con 0x55a2e0e98490 2024-04-01T01:48:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.724+0000 7f6bb17f0740 1 -- 172.21.15.70:0/3468414093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a2e0e0a6c0 con 0x55a2e0e98490 2024-04-01T01:48:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.725+0000 7f6ba7fff640 1 --2- 172.21.15.70:0/3468414093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98490 0x55a2e0ea75b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c00fb20 tx=0x7f6b9c009c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de5630a65fd5071 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:24.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.725+0000 7f6b87fff640 10 client.29653 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:48:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.725+0000 7f6b87fff640 1 -- 172.21.15.70:0/3468414093 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c043050 con 0x55a2e0e98490 2024-04-01T01:48:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.725+0000 7f6b87fff640 1 -- 172.21.15.70:0/3468414093 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f6b9c00fe30 con 0x55a2e0e98490 2024-04-01T01:48:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.725+0000 7f6b87fff640 1 -- 172.21.15.70:0/3468414093 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6b9c016050 con 0x55a2e0e98490 2024-04-01T01:48:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.725+0000 7f6bb17f0740 10 client.29653 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.725+0000 7f6bb17f0740 1 -- 172.21.15.70:0/3468414093 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55a2e0eaa280 con 0x55a2e0e98490 2024-04-01T01:48:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.726+0000 7f6b87fff640 1 -- 172.21.15.70:0/3468414093 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f6b9c02f070 con 0x55a2e0e98490 2024-04-01T01:48:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.726+0000 7f6bb17f0740 10 client.29653 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.726+0000 7f6bb17f0740 10 client.29653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.726+0000 7f6bb17f0740 20 client.29653 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:24.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.726+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/3468414093 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a2e0ebf910 0x55a2e0f28020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:24.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.726+0000 7f6bb17f0740 4 client.29653 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:48:24.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.726+0000 7f6bb17f0740 1 -- 172.21.15.70:0/3468414093 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a2e0e04a10 con 0x55a2e0ebf910 2024-04-01T01:48:24.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.726+0000 7f6bac8d1640 1 --2- 172.21.15.70:0/3468414093 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a2e0ebf910 0x55a2e0f28020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:24.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.727+0000 7f6b87fff640 10 client.29653 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:24.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.727+0000 7f6bac8d1640 1 --2- 172.21.15.70:0/3468414093 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a2e0ebf910 0x55a2e0f28020 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2307c80e1b9fa547 server_cookie=ba0d1f6e7ee36d79 in_seq=0 out_seq=0 2024-04-01T01:48:24.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.728+0000 7f6b87fff640 1 -- 172.21.15.70:0/3468414093 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55a2e0e04a10 con 0x55a2e0ebf910 2024-04-01T01:48:24.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.728+0000 7f6b87fff640 10 client.29653 handle_command_reply: tid=0 2024-04-01T01:48:24.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.728+0000 7f6bb17f0740 1 client.29653 shutdown 2024-04-01T01:48:24.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.728+0000 7f6bb17f0740 1 -- 172.21.15.70:0/3468414093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98490 msgr2=0x55a2e0ea75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.728+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/3468414093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98490 0x55a2e0ea75b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c00fb20 tx=0x7f6b9c009c70 comp rx=0 tx=0).stop 2024-04-01T01:48:24.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.729+0000 7f6bb17f0740 1 -- 172.21.15.70:0/3468414093 shutdown_connections 2024-04-01T01:48:24.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.729+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/3468414093 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55a2e0ebf910 0x55a2e0f28020 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.729+0000 7f6b87fff640 0 client.29653 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:24.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.729+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/3468414093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a2e0e9dff0 0x55a2e0ea8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.729+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/3468414093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a2e0e98e80 0x55a2e0ea7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.729+0000 7f6bb17f0740 1 --2- 172.21.15.70:0/3468414093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2e0e98490 0x55a2e0ea75b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.729+0000 7f6bb17f0740 1 -- 172.21.15.70:0/3468414093 >> 172.21.15.70:0/3468414093 conn(0x55a2e0e93a00 msgr2=0x55a2e0ea1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:24.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.729+0000 7f6bb17f0740 1 -- 172.21.15.70:0/3468414093 shutdown_connections 2024-04-01T01:48:24.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.729+0000 7f6bb17f0740 1 -- 172.21.15.70:0/3468414093 wait complete. 2024-04-01T01:48:24.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.730+0000 7f6bb17f0740 20 client.29653 trim_cache size 0 max 16384 2024-04-01T01:48:24.732 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:48:24.732 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "scrub active (283 inodes in the stack)", 2024-04-01T01:48:24.732 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:48:24.732 INFO:teuthology.orchestra.run.smithi070.stdout: "00c3f15a-6e7f-4512-a617-06c2ffd7d58c": { 2024-04-01T01:48:24.732 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:48:24.732 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "00c3f15a-6e7f-4512-a617-06c2ffd7d58c", 2024-04-01T01:48:24.732 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:48:24.732 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:48:24.732 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:48:24.732 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:48:24.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.731+0000 7f6bad8d3640 1 -- 172.21.15.70:0/897731032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6b900a2670 msgr2=0x7f6b900a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.731+0000 7f6bad8d3640 1 --2- 172.21.15.70:0/897731032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6b900a2670 0x7f6b900a4b30 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8064c40 tx=0x7f6ba8069040 comp rx=0 tx=0).stop 2024-04-01T01:48:24.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.731+0000 7f6bad8d3640 1 -- 172.21.15.70:0/897731032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ba009bac0 msgr2=0x7f6ba00a0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:24.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.731+0000 7f6bad8d3640 1 --2- 172.21.15.70:0/897731032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ba009bac0 0x7f6ba00a0cd0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8067420 tx=0x7f6ba806b540 comp rx=0 tx=0).stop 2024-04-01T01:48:24.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.731+0000 7f6bad8d3640 1 -- 172.21.15.70:0/897731032 shutdown_connections 2024-04-01T01:48:24.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.731+0000 7f6bad8d3640 1 --2- 172.21.15.70:0/897731032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ba009bac0 0x7f6ba00a0cd0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.731+0000 7f6bad8d3640 1 --2- 172.21.15.70:0/897731032 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6b900a2670 0x7f6b900a4b30 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.731+0000 7f6bad8d3640 1 --2- 172.21.15.70:0/897731032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ba0099190 0x7f6ba00a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.731+0000 7f6bad8d3640 1 --2- 172.21.15.70:0/897731032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ba0096860 0x7f6ba00a4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:24.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.731+0000 7f6bad8d3640 1 -- 172.21.15.70:0/897731032 >> 172.21.15.70:0/897731032 conn(0x7f6ba00901d0 msgr2=0x7f6ba009bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:24.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.731+0000 7f6bad8d3640 1 -- 172.21.15.70:0/897731032 shutdown_connections 2024-04-01T01:48:24.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:24.731+0000 7f6bad8d3640 1 -- 172.21.15.70:0/897731032 wait complete. 2024-04-01T01:48:24.746 INFO:tasks.cephfs.filesystem:scrub status for tag:00c3f15a-6e7f-4512-a617-06c2ffd7d58c - {'path': '/', 'tag': '00c3f15a-6e7f-4512-a617-06c2ffd7d58c', 'options': 'recursive,force'} 2024-04-01T01:48:24.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:24 smithi081 ceph-mon[35142]: pgmap v806: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 932 op/s 2024-04-01T01:48:24.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:24 smithi189 ceph-mon[35753]: pgmap v806: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 932 op/s 2024-04-01T01:48:25.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:24 smithi070 ceph-mon[32179]: pgmap v806: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 932 op/s 2024-04-01T01:48:25.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:25 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:48:25.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:25 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:48:25.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:25 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:48:25.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:25 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:48:25.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:25 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:48:25.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:25 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:48:26.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:25 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:48:26.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:25 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:48:26.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:25 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:48:26.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:26 smithi081 ceph-mon[35142]: pgmap v807: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 872 op/s 2024-04-01T01:48:26.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:26 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:48:26.829 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:26 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:48:26.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 820.0 s, 561.4 tps, lat 241.364 ms stddev 596.222 2024-04-01T01:48:26.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:26 smithi189 ceph-mon[35753]: pgmap v807: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 872 op/s 2024-04-01T01:48:26.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:26 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:48:26.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:26 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:48:27.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:26 smithi070 ceph-mon[32179]: pgmap v807: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 872 op/s 2024-04-01T01:48:27.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:26 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:48:27.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:26 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:48:28.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:28 smithi081 ceph-mon[35142]: pgmap v808: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 11 MiB/s wr, 867 op/s 2024-04-01T01:48:28.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:28 smithi189 ceph-mon[35753]: pgmap v808: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 11 MiB/s wr, 867 op/s 2024-04-01T01:48:29.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:28 smithi070 ceph-mon[32179]: pgmap v808: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 11 MiB/s wr, 867 op/s 2024-04-01T01:48:30.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:30 smithi081 ceph-mon[35142]: pgmap v809: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 9.9 MiB/s wr, 834 op/s 2024-04-01T01:48:30.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:30 smithi189 ceph-mon[35753]: pgmap v809: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 9.9 MiB/s wr, 834 op/s 2024-04-01T01:48:31.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:30 smithi070 ceph-mon[32179]: pgmap v809: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 9.9 MiB/s wr, 834 op/s 2024-04-01T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.644+0000 7f81a24ca640 1 -- 172.21.15.70:0/1701306928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f819c101690 msgr2=0x7f819c101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.644+0000 7f81a24ca640 1 --2- 172.21.15.70:0/1701306928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f819c101690 0x7f819c101b10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f818c009900 tx=0x7f818c02bc60 comp rx=0 tx=0).stop 2024-04-01T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.644+0000 7f81a24ca640 1 -- 172.21.15.70:0/1701306928 shutdown_connections 2024-04-01T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.644+0000 7f81a24ca640 1 --2- 172.21.15.70:0/1701306928 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f819c102050 0x7f819c1088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.644+0000 7f81a24ca640 1 --2- 172.21.15.70:0/1701306928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f819c101690 0x7f819c101b10 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.644+0000 7f81a24ca640 1 --2- 172.21.15.70:0/1701306928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f819c1004a0 0x7f819c1008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.644+0000 7f81a24ca640 1 -- 172.21.15.70:0/1701306928 >> 172.21.15.70:0/1701306928 conn(0x7f819c0fba30 msgr2=0x7f819c0fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.645+0000 7f81a24ca640 1 -- 172.21.15.70:0/1701306928 shutdown_connections 2024-04-01T01:48:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.645+0000 7f81a24ca640 1 -- 172.21.15.70:0/1701306928 wait complete. 2024-04-01T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.645+0000 7f81a24ca640 1 Processor -- start 2024-04-01T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.645+0000 7f81a24ca640 1 -- start start 2024-04-01T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.646+0000 7f81a24ca640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f819c1004a0 0x7f819c06b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.646+0000 7f81a24ca640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f819c101690 0x7f819c06bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.646+0000 7f81a24ca640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f819c102050 0x7f819c072b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.646+0000 7f81a24ca640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f819c10af90 con 0x7f819c102050 2024-04-01T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.646+0000 7f81a24ca640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f819c10ae10 con 0x7f819c1004a0 2024-04-01T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.646+0000 7f81a24ca640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f819c10b110 con 0x7f819c101690 2024-04-01T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.646+0000 7f81a0cc7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f819c101690 0x7f819c06bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.646+0000 7f81a0cc7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f819c101690 0x7f819c06bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58948/0 (socket says 172.21.15.70:58948) 2024-04-01T01:48:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.646+0000 7f81a0cc7640 1 -- 172.21.15.70:0/2041076404 learned_addr learned my addr 172.21.15.70:0/2041076404 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.646+0000 7f81a1cc9640 1 --2- 172.21.15.70:0/2041076404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f819c102050 0x7f819c072b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.646+0000 7f81a0cc7640 1 -- 172.21.15.70:0/2041076404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f819c1004a0 msgr2=0x7f819c06b570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-01T01:48:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.646+0000 7f81a0cc7640 1 --2- 172.21.15.70:0/2041076404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f819c1004a0 0x7f819c06b570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.646+0000 7f81a0cc7640 1 -- 172.21.15.70:0/2041076404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f819c102050 msgr2=0x7f819c072b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.646+0000 7f81a0cc7640 1 --2- 172.21.15.70:0/2041076404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f819c102050 0x7f819c072b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.646+0000 7f81a0cc7640 1 -- 172.21.15.70:0/2041076404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f819c073190 con 0x7f819c101690 2024-04-01T01:48:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.647+0000 7f81a0cc7640 1 --2- 172.21.15.70:0/2041076404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f819c101690 0x7f819c06bab0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f818c000c00 tx=0x7f818c0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.647+0000 7f81927fc640 1 -- 172.21.15.70:0/2041076404 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818c043020 con 0x7f819c101690 2024-04-01T01:48:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.647+0000 7f81927fc640 1 -- 172.21.15.70:0/2041076404 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f818c004aa0 con 0x7f819c101690 2024-04-01T01:48:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.647+0000 7f81a24ca640 1 -- 172.21.15.70:0/2041076404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f819c073420 con 0x7f819c101690 2024-04-01T01:48:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.647+0000 7f81927fc640 1 -- 172.21.15.70:0/2041076404 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818c003d00 con 0x7f819c101690 2024-04-01T01:48:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.647+0000 7f81a24ca640 1 -- 172.21.15.70:0/2041076404 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f819c1aa860 con 0x7f819c101690 2024-04-01T01:48:31.651 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.650+0000 7f81927fc640 1 -- 172.21.15.70:0/2041076404 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f818c046020 con 0x7f819c101690 2024-04-01T01:48:31.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.651+0000 7f81a63ef740 1 Processor -- start 2024-04-01T01:48:31.653 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.651+0000 7f81a63ef740 1 -- start start 2024-04-01T01:48:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.651+0000 7f81a63ef740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687e5ae0 0x5630687e5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.651+0000 7f81a63ef740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e64d0 0x5630687eafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.651+0000 7f81a63ef740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687eb640 0x5630687edad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.651+0000 7f81a63ef740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5630687498f0 con 0x5630687e64d0 2024-04-01T01:48:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.651+0000 7f81a63ef740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5630686f28d0 con 0x5630687eb640 2024-04-01T01:48:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.651+0000 7f81a63ef740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563068745990 con 0x5630687e5ae0 2024-04-01T01:48:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.651+0000 7f81a0cc7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687eb640 0x5630687edad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.651+0000 7f81a0cc7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687eb640 0x5630687edad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:40474/0 (socket says 172.21.15.70:40474) 2024-04-01T01:48:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.651+0000 7f81a0cc7640 1 -- 172.21.15.70:0/4060070779 learned_addr learned my addr 172.21.15.70:0/4060070779 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.651+0000 7f81a14c8640 1 --2- 172.21.15.70:0/4060070779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e64d0 0x5630687eafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.651+0000 7f81927fc640 1 --2- 172.21.15.70:0/2041076404 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81880a2510 0x7f81880a49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.651+0000 7f81a1cc9640 1 --2- 172.21.15.70:0/4060070779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687e5ae0 0x5630687e5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.654 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.652+0000 7f81a0cc7640 1 -- 172.21.15.70:0/4060070779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687e5ae0 msgr2=0x5630687e5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:31.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.652+0000 7f81a0cc7640 1 --2- 172.21.15.70:0/4060070779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687e5ae0 0x5630687e5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.652+0000 7f81a0cc7640 1 -- 172.21.15.70:0/4060070779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e64d0 msgr2=0x5630687eafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:31.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.652+0000 7f81a0cc7640 1 --2- 172.21.15.70:0/4060070779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e64d0 0x5630687eafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.652+0000 7f81a0cc7640 1 -- 172.21.15.70:0/4060070779 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56306845e210 con 0x5630687eb640 2024-04-01T01:48:31.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.652+0000 7f81927fc640 1 -- 172.21.15.70:0/2041076404 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f818c0f0e20 con 0x7f819c101690 2024-04-01T01:48:31.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.652+0000 7f81a1cc9640 1 --2- 172.21.15.70:0/4060070779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687e5ae0 0x5630687e5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:48:31.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.652+0000 7f81a1cc9640 1 --2- 172.21.15.70:0/2041076404 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81880a2510 0x7f81880a49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.652+0000 7f81a0cc7640 1 --2- 172.21.15.70:0/4060070779 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687eb640 0x5630687edad0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f818c0b9e80 tx=0x7f818c02bc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfbe21bfb83dd35f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:31.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.653+0000 7f81a1cc9640 1 --2- 172.21.15.70:0/2041076404 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81880a2510 0x7f81880a49d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x5630687e6d30 tx=0x7f8194014040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:31.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.653+0000 7f81877fe640 1 -- 172.21.15.70:0/4060070779 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818c0bd280 con 0x5630687eb640 2024-04-01T01:48:31.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.653+0000 7f81877fe640 1 -- 172.21.15.70:0/4060070779 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f818c004c50 con 0x5630687eb640 2024-04-01T01:48:31.655 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.653+0000 7f81877fe640 1 -- 172.21.15.70:0/4060070779 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818c0044e0 con 0x5630687eb640 2024-04-01T01:48:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.654+0000 7f81a63ef740 1 -- 172.21.15.70:0/4060070779 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687eb640 msgr2=0x5630687edad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.654+0000 7f81a63ef740 1 --2- 172.21.15.70:0/4060070779 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687eb640 0x5630687edad0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f818c0b9e80 tx=0x7f818c02bc60 comp rx=0 tx=0).stop 2024-04-01T01:48:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.654+0000 7f81a63ef740 1 -- 172.21.15.70:0/4060070779 shutdown_connections 2024-04-01T01:48:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.654+0000 7f81a63ef740 1 --2- 172.21.15.70:0/4060070779 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687eb640 0x5630687edad0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.654+0000 7f81a63ef740 1 --2- 172.21.15.70:0/4060070779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e64d0 0x5630687eafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.654+0000 7f81a63ef740 1 --2- 172.21.15.70:0/4060070779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687e5ae0 0x5630687e5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.654+0000 7f81a63ef740 1 -- 172.21.15.70:0/4060070779 >> 172.21.15.70:0/4060070779 conn(0x5630687e1050 msgr2=0x5630687e1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.654+0000 7f81a63ef740 1 -- 172.21.15.70:0/4060070779 shutdown_connections 2024-04-01T01:48:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.654+0000 7f81a63ef740 1 -- 172.21.15.70:0/4060070779 wait complete. 2024-04-01T01:48:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.654+0000 7f81a63ef740 1 -- start start 2024-04-01T01:48:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.654+0000 7f81a63ef740 1 Processor -- start 2024-04-01T01:48:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.655+0000 7f81a63ef740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687e5ae0 0x5630687f8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.655+0000 7f81a63ef740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e64d0 0x5630687f8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.655+0000 7f81a14c8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687e5ae0 0x5630687f8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.655+0000 7f81a63ef740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687eb640 0x5630687ff8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.655+0000 7f81a14c8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687e5ae0 0x5630687f8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:40482/0 (socket says 172.21.15.70:40482) 2024-04-01T01:48:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.655+0000 7f81a0cc7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e64d0 0x5630687f8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.655+0000 7f81a0cc7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e64d0 0x5630687f8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39938/0 (socket says 172.21.15.70:39938) 2024-04-01T01:48:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.655+0000 7f81a14c8640 1 -- 172.21.15.70:0/24157882 learned_addr learned my addr 172.21.15.70:0/24157882 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.655+0000 7f81a63ef740 1 -- 172.21.15.70:0/24157882 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5630687ee3b0 con 0x5630687e64d0 2024-04-01T01:48:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.655+0000 7f81a63ef740 1 -- 172.21.15.70:0/24157882 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5630686e7f80 con 0x5630687e5ae0 2024-04-01T01:48:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.655+0000 7f81a63ef740 1 -- 172.21.15.70:0/24157882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5630687ee530 con 0x5630687eb640 2024-04-01T01:48:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.655+0000 7f81a1cc9640 1 --2- 172.21.15.70:0/24157882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687eb640 0x5630687ff8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.655+0000 7f81a63ef740 10 client.0 _ll_register_callbacks cb 0x563068740290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.656+0000 7f81a14c8640 1 -- 172.21.15.70:0/24157882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687eb640 msgr2=0x5630687ff8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.656+0000 7f81a14c8640 1 --2- 172.21.15.70:0/24157882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687eb640 0x5630687ff8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.656+0000 7f81a14c8640 1 -- 172.21.15.70:0/24157882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e64d0 msgr2=0x5630687f8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.656+0000 7f81a14c8640 1 --2- 172.21.15.70:0/24157882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e64d0 0x5630687f8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.656+0000 7f81a14c8640 1 -- 172.21.15.70:0/24157882 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56306852a620 con 0x5630687e5ae0 2024-04-01T01:48:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.656+0000 7f81a0cc7640 1 --2- 172.21.15.70:0/24157882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e64d0 0x5630687f8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:48:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.656+0000 7f81a14c8640 1 --2- 172.21.15.70:0/24157882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687e5ae0 0x5630687f8320 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f8198009fd0 tx=0x7f819800e660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46a60b4d63c8fa52 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.656+0000 7f8184ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.656+0000 7f8184ff9640 1 -- 172.21.15.70:0/24157882 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8198017070 con 0x5630687e5ae0 2024-04-01T01:48:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.656+0000 7f81a63ef740 1 -- 172.21.15.70:0/24157882 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f818c004c50 con 0x5630687e5ae0 2024-04-01T01:48:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.656+0000 7f8184ff9640 1 -- 172.21.15.70:0/24157882 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8198013680 con 0x5630687e5ae0 2024-04-01T01:48:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.656+0000 7f8184ff9640 1 -- 172.21.15.70:0/24157882 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8198023050 con 0x5630687e5ae0 2024-04-01T01:48:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.656+0000 7f81a63ef740 10 client.29939 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.656+0000 7f81a63ef740 1 -- 172.21.15.70:0/24157882 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563068802920 con 0x5630687e5ae0 2024-04-01T01:48:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.657+0000 7f8184ff9640 1 -- 172.21.15.70:0/24157882 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f818c004c50 con 0x5630687e5ae0 2024-04-01T01:48:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.657+0000 7f8184ff9640 1 -- 172.21.15.70:0/24157882 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8198012810 con 0x5630687e5ae0 2024-04-01T01:48:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.657+0000 7f81a63ef740 10 client.29939 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.657+0000 7f81a63ef740 10 client.29939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.657+0000 7f81a63ef740 20 client.29939 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.657+0000 7f81a63ef740 1 --2- 172.21.15.70:0/24157882 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563068804420 0x5630688068e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.657+0000 7f81a63ef740 4 client.29939 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.657+0000 7f81a63ef740 1 -- 172.21.15.70:0/24157882 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56306875d120 con 0x563068804420 2024-04-01T01:48:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.657+0000 7f81a0cc7640 1 --2- 172.21.15.70:0/24157882 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563068804420 0x5630688068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.658+0000 7f81a0cc7640 1 --2- 172.21.15.70:0/24157882 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563068804420 0x5630688068e0 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fad80d319137f83f server_cookie=fdaee79dd518a406 in_seq=0 out_seq=0 2024-04-01T01:48:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.658+0000 7f8184ff9640 10 client.29939 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.658+0000 7f8184ff9640 1 -- 172.21.15.70:0/24157882 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56306875d120 con 0x563068804420 2024-04-01T01:48:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.658+0000 7f8184ff9640 10 client.29939 handle_command_reply: tid=0 2024-04-01T01:48:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.658+0000 7f81a63ef740 1 client.29939 shutdown 2024-04-01T01:48:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.659+0000 7f81a63ef740 1 -- 172.21.15.70:0/24157882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687e5ae0 msgr2=0x5630687f8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:31.664 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.659+0000 7f81a63ef740 1 --2- 172.21.15.70:0/24157882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687e5ae0 0x5630687f8320 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f8198009fd0 tx=0x7f819800e660 comp rx=0 tx=0).stop 2024-04-01T01:48:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.659+0000 7f81a63ef740 1 -- 172.21.15.70:0/24157882 shutdown_connections 2024-04-01T01:48:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.659+0000 7f81a63ef740 1 --2- 172.21.15.70:0/24157882 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x563068804420 0x5630688068e0 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.659+0000 7f81a63ef740 1 --2- 172.21.15.70:0/24157882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687eb640 0x5630687ff8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.659+0000 7f81a63ef740 1 --2- 172.21.15.70:0/24157882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e64d0 0x5630687f8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.659+0000 7f81a63ef740 1 --2- 172.21.15.70:0/24157882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687e5ae0 0x5630687f8320 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.659+0000 7f81a63ef740 1 -- 172.21.15.70:0/24157882 >> 172.21.15.70:0/24157882 conn(0x5630687e1050 msgr2=0x5630687e0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.659+0000 7f8184ff9640 0 client.29939 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.659+0000 7f81a63ef740 1 -- 172.21.15.70:0/24157882 shutdown_connections 2024-04-01T01:48:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.659+0000 7f81a63ef740 1 -- 172.21.15.70:0/24157882 wait complete. 2024-04-01T01:48:31.666 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.659+0000 7f81a63ef740 20 client.29939 trim_cache size 0 max 16384 2024-04-01T01:48:31.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.679+0000 7f81a63ef740 1 Processor -- start 2024-04-01T01:48:31.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.679+0000 7f81a63ef740 1 -- start start 2024-04-01T01:48:31.680 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.679+0000 7f81a63ef740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687e5ae0 0x5630687df900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.679+0000 7f81a63ef740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687e64d0 0x5630687dfe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.679+0000 7f81a63ef740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687eb640 0x5630687f7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.679+0000 7f81a63ef740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f819c1b0850 con 0x5630687eb640 2024-04-01T01:48:31.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.679+0000 7f81a63ef740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f819c1b2390 con 0x5630687e64d0 2024-04-01T01:48:31.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.679+0000 7f81a63ef740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f819c1b0f20 con 0x5630687e5ae0 2024-04-01T01:48:31.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.679+0000 7f81a0cc7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687e64d0 0x5630687dfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.679+0000 7f81a1cc9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687eb640 0x5630687f7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.679+0000 7f81a14c8640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687e5ae0 0x5630687df900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.679+0000 7f81a0cc7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687e64d0 0x5630687dfe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:40488/0 (socket says 172.21.15.70:40488) 2024-04-01T01:48:31.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.679+0000 7f81a0cc7640 1 -- 172.21.15.70:0/2124580092 learned_addr learned my addr 172.21.15.70:0/2124580092 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:31.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.679+0000 7f81a0cc7640 1 -- 172.21.15.70:0/2124580092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687e5ae0 msgr2=0x5630687df900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:31.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.680+0000 7f81a0cc7640 1 --2- 172.21.15.70:0/2124580092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687e5ae0 0x5630687df900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.680+0000 7f81a0cc7640 1 -- 172.21.15.70:0/2124580092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687eb640 msgr2=0x5630687f7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:31.681 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.680+0000 7f81a0cc7640 1 --2- 172.21.15.70:0/2124580092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687eb640 0x5630687f7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.680+0000 7f81a0cc7640 1 -- 172.21.15.70:0/2124580092 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630687ffcc0 con 0x5630687e64d0 2024-04-01T01:48:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.680+0000 7f81a0cc7640 1 --2- 172.21.15.70:0/2124580092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687e64d0 0x5630687dfe40 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f818c0b9910 tx=0x7f818c041040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71510e8d3be259e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.680+0000 7f8186ffd640 1 -- 172.21.15.70:0/2124580092 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818c0d6020 con 0x5630687e64d0 2024-04-01T01:48:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.680+0000 7f8186ffd640 1 -- 172.21.15.70:0/2124580092 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f818c0bd4b0 con 0x5630687e64d0 2024-04-01T01:48:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.680+0000 7f8186ffd640 1 -- 172.21.15.70:0/2124580092 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818c03c350 con 0x5630687e64d0 2024-04-01T01:48:31.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.681+0000 7f81a63ef740 1 -- 172.21.15.70:0/2124580092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687e64d0 msgr2=0x5630687dfe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:31.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.681+0000 7f81a63ef740 1 --2- 172.21.15.70:0/2124580092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687e64d0 0x5630687dfe40 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f818c0b9910 tx=0x7f818c041040 comp rx=0 tx=0).stop 2024-04-01T01:48:31.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.681+0000 7f81a63ef740 1 -- 172.21.15.70:0/2124580092 shutdown_connections 2024-04-01T01:48:31.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.681+0000 7f81a63ef740 1 --2- 172.21.15.70:0/2124580092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687eb640 0x5630687f7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.681+0000 7f81a63ef740 1 --2- 172.21.15.70:0/2124580092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687e64d0 0x5630687dfe40 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.681+0000 7f81a63ef740 1 --2- 172.21.15.70:0/2124580092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687e5ae0 0x5630687df900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.681+0000 7f81a63ef740 1 -- 172.21.15.70:0/2124580092 >> 172.21.15.70:0/2124580092 conn(0x5630687e1050 msgr2=0x5630687e1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:31.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.681+0000 7f81a63ef740 1 -- 172.21.15.70:0/2124580092 shutdown_connections 2024-04-01T01:48:31.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.681+0000 7f81a63ef740 1 -- 172.21.15.70:0/2124580092 wait complete. 2024-04-01T01:48:31.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.681+0000 7f81a63ef740 1 -- start start 2024-04-01T01:48:31.686 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.681+0000 7f81a63ef740 1 Processor -- start 2024-04-01T01:48:31.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a63ef740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e5ae0 0x56306880b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a63ef740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687e64d0 0x56306880ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a63ef740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687eb640 0x563068868730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a63ef740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f819c1b1cc0 con 0x5630687e5ae0 2024-04-01T01:48:31.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a63ef740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f819c1b15f0 con 0x5630687eb640 2024-04-01T01:48:31.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a63ef740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f819c1ae830 con 0x5630687e64d0 2024-04-01T01:48:31.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a63ef740 10 client.0 _ll_register_callbacks cb 0x7f8198009310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:31.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a14c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e5ae0 0x56306880b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a1cc9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687eb640 0x563068868730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a14c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e5ae0 0x56306880b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:39948/0 (socket says 172.21.15.70:39948) 2024-04-01T01:48:31.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a14c8640 1 -- 172.21.15.70:0/786706638 learned_addr learned my addr 172.21.15.70:0/786706638 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:31.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a0cc7640 1 --2- 172.21.15.70:0/786706638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687e64d0 0x56306880ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a1cc9640 1 -- 172.21.15.70:0/786706638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687e64d0 msgr2=0x56306880ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:31.687 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a1cc9640 1 --2- 172.21.15.70:0/786706638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687e64d0 0x56306880ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a1cc9640 1 -- 172.21.15.70:0/786706638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e5ae0 msgr2=0x56306880b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a1cc9640 1 --2- 172.21.15.70:0/786706638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e5ae0 0x56306880b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a1cc9640 1 -- 172.21.15.70:0/786706638 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563068868cc0 con 0x5630687eb640 2024-04-01T01:48:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a14c8640 1 --2- 172.21.15.70:0/786706638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e5ae0 0x56306880b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f81a1cc9640 1 --2- 172.21.15.70:0/786706638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687eb640 0x563068868730 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f8194005fd0 tx=0x7f8194009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a961068c1c684366 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f8184ff9640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.682+0000 7f8184ff9640 1 -- 172.21.15.70:0/786706638 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f819401fc90 con 0x5630687eb640 2024-04-01T01:48:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.683+0000 7f81a63ef740 1 -- 172.21.15.70:0/786706638 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563068756e30 con 0x5630687eb640 2024-04-01T01:48:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.683+0000 7f8184ff9640 1 -- 172.21.15.70:0/786706638 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8194004b20 con 0x5630687eb640 2024-04-01T01:48:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.683+0000 7f8184ff9640 1 -- 172.21.15.70:0/786706638 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f819401fe30 con 0x5630687eb640 2024-04-01T01:48:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.683+0000 7f81a63ef740 10 client.29945 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.683+0000 7f81a63ef740 1 -- 172.21.15.70:0/786706638 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56306886a090 con 0x5630687eb640 2024-04-01T01:48:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.683+0000 7f8184ff9640 1 -- 172.21.15.70:0/786706638 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x563068756e30 con 0x5630687eb640 2024-04-01T01:48:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.684+0000 7f8184ff9640 1 -- 172.21.15.70:0/786706638 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8194005250 con 0x5630687eb640 2024-04-01T01:48:31.689 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.684+0000 7f81a63ef740 10 client.29945 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:31.689 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.684+0000 7f81a63ef740 10 client.29945 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:31.689 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.684+0000 7f81a63ef740 20 client.29945 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:31.689 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.684+0000 7f81a63ef740 1 --2- 172.21.15.70:0/786706638 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56306886aaa0 0x56306886cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:31.689 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.684+0000 7f81a63ef740 4 client.29945 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:48:31.689 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.684+0000 7f81a63ef740 1 -- 172.21.15.70:0/786706638 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563068751180 con 0x56306886aaa0 2024-04-01T01:48:31.689 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.684+0000 7f81a14c8640 1 --2- 172.21.15.70:0/786706638 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56306886aaa0 0x56306886cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:31.690 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.685+0000 7f81a14c8640 1 --2- 172.21.15.70:0/786706638 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56306886aaa0 0x56306886cec0 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f833f94657a1276 server_cookie=f273e7ec6532dab2 in_seq=0 out_seq=0 2024-04-01T01:48:31.690 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.685+0000 7f8184ff9640 10 client.29945 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.713+0000 7f8184ff9640 1 -- 172.21.15.70:0/786706638 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563068751180 con 0x56306886aaa0 2024-04-01T01:48:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.714+0000 7f8184ff9640 10 client.29945 handle_command_reply: tid=0 2024-04-01T01:48:31.715 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.715+0000 7f81a63ef740 1 client.29945 shutdown 2024-04-01T01:48:31.717 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.715+0000 7f81a63ef740 1 -- 172.21.15.70:0/786706638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687eb640 msgr2=0x563068868730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:31.717 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.715+0000 7f81a63ef740 1 --2- 172.21.15.70:0/786706638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687eb640 0x563068868730 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f8194005fd0 tx=0x7f8194009500 comp rx=0 tx=0).stop 2024-04-01T01:48:31.717 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.715+0000 7f81a63ef740 1 -- 172.21.15.70:0/786706638 shutdown_connections 2024-04-01T01:48:31.717 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.715+0000 7f81a63ef740 1 --2- 172.21.15.70:0/786706638 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56306886aaa0 0x56306886cec0 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.717 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.715+0000 7f81a63ef740 1 --2- 172.21.15.70:0/786706638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630687eb640 0x563068868730 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.717 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.715+0000 7f8184ff9640 0 client.29945 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:31.717 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.715+0000 7f81a63ef740 1 --2- 172.21.15.70:0/786706638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630687e64d0 0x56306880ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.717 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.716+0000 7f81a63ef740 1 --2- 172.21.15.70:0/786706638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630687e5ae0 0x56306880b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.717 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.716+0000 7f81a63ef740 1 -- 172.21.15.70:0/786706638 >> 172.21.15.70:0/786706638 conn(0x5630687e1050 msgr2=0x5630687df900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:31.717 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.716+0000 7f81a63ef740 1 -- 172.21.15.70:0/786706638 shutdown_connections 2024-04-01T01:48:31.717 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.716+0000 7f81a63ef740 1 -- 172.21.15.70:0/786706638 wait complete. 2024-04-01T01:48:31.717 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.716+0000 7f81a63ef740 20 client.29945 trim_cache size 0 max 16384 2024-04-01T01:48:31.718 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:48:31.718 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:48:31.718 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:48:31.718 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:48:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.716+0000 7f81a24ca640 1 -- 172.21.15.70:0/2041076404 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81880a2510 msgr2=0x7f81880a49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.716+0000 7f81a24ca640 1 --2- 172.21.15.70:0/2041076404 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81880a2510 0x7f81880a49d0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x5630687e6d30 tx=0x7f8194014040 comp rx=0 tx=0).stop 2024-04-01T01:48:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.716+0000 7f81a24ca640 1 -- 172.21.15.70:0/2041076404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f819c101690 msgr2=0x7f819c06bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.716+0000 7f81a24ca640 1 --2- 172.21.15.70:0/2041076404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f819c101690 0x7f819c06bab0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f818c000c00 tx=0x7f818c0042f0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.717+0000 7f81a24ca640 1 -- 172.21.15.70:0/2041076404 shutdown_connections 2024-04-01T01:48:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.717+0000 7f81a24ca640 1 --2- 172.21.15.70:0/2041076404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f819c102050 0x7f819c072b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.719 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.717+0000 7f81a24ca640 1 --2- 172.21.15.70:0/2041076404 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f81880a2510 0x7f81880a49d0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.717+0000 7f81a24ca640 1 --2- 172.21.15.70:0/2041076404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f819c101690 0x7f819c06bab0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.717+0000 7f81a24ca640 1 --2- 172.21.15.70:0/2041076404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f819c1004a0 0x7f819c06b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.717+0000 7f81a24ca640 1 -- 172.21.15.70:0/2041076404 >> 172.21.15.70:0/2041076404 conn(0x7f819c0fba30 msgr2=0x7f819c104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.717+0000 7f81a24ca640 1 -- 172.21.15.70:0/2041076404 shutdown_connections 2024-04-01T01:48:31.720 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:31.717+0000 7f81a24ca640 1 -- 172.21.15.70:0/2041076404 wait complete. 2024-04-01T01:48:31.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 825.0 s, 701.4 tps, lat 140.943 ms stddev 152.710 2024-04-01T01:48:32.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:32 smithi081 ceph-mon[35142]: pgmap v810: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 9.3 MiB/s wr, 857 op/s 2024-04-01T01:48:32.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:32 smithi189 ceph-mon[35753]: pgmap v810: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 9.3 MiB/s wr, 857 op/s 2024-04-01T01:48:33.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:32 smithi070 ceph-mon[32179]: pgmap v810: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 9.3 MiB/s wr, 857 op/s 2024-04-01T01:48:34.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:34 smithi081 ceph-mon[35142]: pgmap v811: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 7.9 MiB/s wr, 783 op/s 2024-04-01T01:48:34.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:34 smithi189 ceph-mon[35753]: pgmap v811: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 7.9 MiB/s wr, 783 op/s 2024-04-01T01:48:35.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:34 smithi070 ceph-mon[32179]: pgmap v811: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 7.9 MiB/s wr, 783 op/s 2024-04-01T01:48:36.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 830.0 s, 742.8 tps, lat 136.010 ms stddev 135.600 2024-04-01T01:48:36.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:36 smithi189 ceph-mon[35753]: pgmap v812: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 6.6 MiB/s wr, 735 op/s 2024-04-01T01:48:37.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:36 smithi070 ceph-mon[32179]: pgmap v812: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 6.6 MiB/s wr, 735 op/s 2024-04-01T01:48:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:36 smithi081 ceph-mon[35142]: pgmap v812: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 6.6 MiB/s wr, 735 op/s 2024-04-01T01:48:38.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:38 smithi189 ceph-mon[35753]: pgmap v813: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 5.6 MiB/s wr, 696 op/s 2024-04-01T01:48:39.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:38 smithi070 ceph-mon[32179]: pgmap v813: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 5.6 MiB/s wr, 696 op/s 2024-04-01T01:48:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:38 smithi081 ceph-mon[35142]: pgmap v813: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 5.6 MiB/s wr, 696 op/s 2024-04-01T01:48:40.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:40 smithi189 ceph-mon[35753]: pgmap v814: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 5.9 MiB/s wr, 721 op/s 2024-04-01T01:48:41.005 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:40 smithi070 ceph-mon[32179]: pgmap v814: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 5.9 MiB/s wr, 721 op/s 2024-04-01T01:48:41.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:40 smithi081 ceph-mon[35142]: pgmap v814: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 5.9 MiB/s wr, 721 op/s 2024-04-01T01:48:41.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 835.0 s, 715.4 tps, lat 138.004 ms stddev 149.934 2024-04-01T01:48:42.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:42 smithi189 ceph-mon[35753]: pgmap v815: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.2 MiB/s wr, 734 op/s 2024-04-01T01:48:43.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:42 smithi070 ceph-mon[32179]: pgmap v815: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.2 MiB/s wr, 734 op/s 2024-04-01T01:48:43.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:42 smithi081 ceph-mon[35142]: pgmap v815: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 6.2 MiB/s wr, 734 op/s 2024-04-01T01:48:44.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.858+0000 7fed660ed640 1 -- 172.21.15.70:0/1384626267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed60101860 msgr2=0x7fed60101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:44.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.858+0000 7fed660ed640 1 --2- 172.21.15.70:0/1384626267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed60101860 0x7fed60101ce0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fed54009900 tx=0x7fed5402bc30 comp rx=0 tx=0).stop 2024-04-01T01:48:44.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.858+0000 7fed660ed640 1 -- 172.21.15.70:0/1384626267 shutdown_connections 2024-04-01T01:48:44.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.858+0000 7fed660ed640 1 --2- 172.21.15.70:0/1384626267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed60102220 0x7fed60108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.858+0000 7fed660ed640 1 --2- 172.21.15.70:0/1384626267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed60101860 0x7fed60101ce0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.858+0000 7fed660ed640 1 --2- 172.21.15.70:0/1384626267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed60100670 0x7fed60100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.858+0000 7fed660ed640 1 -- 172.21.15.70:0/1384626267 >> 172.21.15.70:0/1384626267 conn(0x7fed600fbbe0 msgr2=0x7fed600fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:44.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.859+0000 7fed660ed640 1 -- 172.21.15.70:0/1384626267 shutdown_connections 2024-04-01T01:48:44.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.859+0000 7fed660ed640 1 -- 172.21.15.70:0/1384626267 wait complete. 2024-04-01T01:48:44.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.859+0000 7fed660ed640 1 Processor -- start 2024-04-01T01:48:44.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.860+0000 7fed660ed640 1 -- start start 2024-04-01T01:48:44.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.860+0000 7fed660ed640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed60100670 0x7fed60196830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.860+0000 7fed660ed640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed60101860 0x7fed60196d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.860+0000 7fed660ed640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed60102220 0x7fed6019dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.860+0000 7fed660ed640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fed6010c190 con 0x7fed60102220 2024-04-01T01:48:44.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.860+0000 7fed5effd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed60101860 0x7fed60196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.860+0000 7fed5f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed60100670 0x7fed60196830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.860+0000 7fed5effd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed60101860 0x7fed60196d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:47258/0 (socket says 172.21.15.70:47258) 2024-04-01T01:48:44.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.860+0000 7fed5f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed60100670 0x7fed60196830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33548/0 (socket says 172.21.15.70:33548) 2024-04-01T01:48:44.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.860+0000 7fed5effd640 1 -- 172.21.15.70:0/182601534 learned_addr learned my addr 172.21.15.70:0/182601534 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:44.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.860+0000 7fed5ffff640 1 --2- 172.21.15.70:0/182601534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed60102220 0x7fed6019dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.860+0000 7fed660ed640 1 -- 172.21.15.70:0/182601534 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fed6010c010 con 0x7fed60100670 2024-04-01T01:48:44.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.860+0000 7fed660ed640 1 -- 172.21.15.70:0/182601534 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fed6010c310 con 0x7fed60101860 2024-04-01T01:48:44.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.861+0000 7fed5f7fe640 1 -- 172.21.15.70:0/182601534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed60101860 msgr2=0x7fed60196d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:44.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.861+0000 7fed5f7fe640 1 --2- 172.21.15.70:0/182601534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed60101860 0x7fed60196d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.861+0000 7fed5f7fe640 1 -- 172.21.15.70:0/182601534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed60102220 msgr2=0x7fed6019dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:44.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.861+0000 7fed5f7fe640 1 --2- 172.21.15.70:0/182601534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed60102220 0x7fed6019dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.861+0000 7fed5f7fe640 1 -- 172.21.15.70:0/182601534 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed6019e400 con 0x7fed60100670 2024-04-01T01:48:44.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.861+0000 7fed5f7fe640 1 --2- 172.21.15.70:0/182601534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed60100670 0x7fed60196830 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fed601016c0 tx=0x7fed5000be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:44.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.861+0000 7fed5cff9640 1 -- 172.21.15.70:0/182601534 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5001e070 con 0x7fed60100670 2024-04-01T01:48:44.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.861+0000 7fed5cff9640 1 -- 172.21.15.70:0/182601534 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed50019420 con 0x7fed60100670 2024-04-01T01:48:44.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.861+0000 7fed660ed640 1 -- 172.21.15.70:0/182601534 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed60068e30 con 0x7fed60100670 2024-04-01T01:48:44.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.861+0000 7fed5cff9640 1 -- 172.21.15.70:0/182601534 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed50017580 con 0x7fed60100670 2024-04-01T01:48:44.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.861+0000 7fed660ed640 1 -- 172.21.15.70:0/182601534 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed600693d0 con 0x7fed60100670 2024-04-01T01:48:44.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.865+0000 7fed5cff9640 1 -- 172.21.15.70:0/182601534 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fed50017720 con 0x7fed60100670 2024-04-01T01:48:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.865+0000 7fed5cff9640 1 --2- 172.21.15.70:0/182601534 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fed480a25c0 0x7fed480a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.865+0000 7fed5cff9640 1 -- 172.21.15.70:0/182601534 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fed500ce850 con 0x7fed60100670 2024-04-01T01:48:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.866+0000 7fed5effd640 1 --2- 172.21.15.70:0/182601534 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fed480a25c0 0x7fed480a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.866 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.866+0000 7fed68dae740 1 Processor -- start 2024-04-01T01:48:44.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.866+0000 7fed68dae740 1 -- start start 2024-04-01T01:48:44.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.866+0000 7fed68dae740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e08d0 0x55c8d33e0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.866+0000 7fed68dae740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e12c0 0x55c8d33e5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.866+0000 7fed68dae740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e6430 0x55c8d33e88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.866+0000 7fed68dae740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c8d33448f0 con 0x55c8d33e6430 2024-04-01T01:48:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.866+0000 7fed68dae740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c8d32ed8d0 con 0x55c8d33e08d0 2024-04-01T01:48:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.866+0000 7fed68dae740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c8d3340990 con 0x55c8d33e12c0 2024-04-01T01:48:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.866+0000 7fed5effd640 1 --2- 172.21.15.70:0/182601534 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fed480a25c0 0x7fed480a4a80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fed60197cf0 tx=0x7fed5402c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.866+0000 7fed5effd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e12c0 0x55c8d33e5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.866+0000 7fed5f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e08d0 0x55c8d33e0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.866+0000 7fed5effd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e12c0 0x55c8d33e5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:47272/0 (socket says 172.21.15.70:47272) 2024-04-01T01:48:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.866+0000 7fed5effd640 1 -- 172.21.15.70:0/545023633 learned_addr learned my addr 172.21.15.70:0/545023633 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.867+0000 7fed5ffff640 1 --2- 172.21.15.70:0/545023633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e6430 0x55c8d33e88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.867+0000 7fed5effd640 1 -- 172.21.15.70:0/545023633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e08d0 msgr2=0x55c8d33e0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.867+0000 7fed5effd640 1 --2- 172.21.15.70:0/545023633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e08d0 0x55c8d33e0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.867+0000 7fed5effd640 1 -- 172.21.15.70:0/545023633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e6430 msgr2=0x55c8d33e88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.867+0000 7fed5effd640 1 --2- 172.21.15.70:0/545023633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e6430 0x55c8d33e88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.867+0000 7fed5effd640 1 -- 172.21.15.70:0/545023633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d3059210 con 0x55c8d33e12c0 2024-04-01T01:48:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.867+0000 7fed5ffff640 1 --2- 172.21.15.70:0/545023633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e6430 0x55c8d33e88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:48:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.867+0000 7fed5effd640 1 --2- 172.21.15.70:0/545023633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e12c0 0x55c8d33e5dc0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fed54009430 tx=0x7fed5403e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e3c5e4421b2799 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.867+0000 7fed45ffb640 1 -- 172.21.15.70:0/545023633 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5402cc70 con 0x55c8d33e12c0 2024-04-01T01:48:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.867+0000 7fed45ffb640 1 -- 172.21.15.70:0/545023633 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed54050070 con 0x55c8d33e12c0 2024-04-01T01:48:44.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.868+0000 7fed45ffb640 1 -- 172.21.15.70:0/545023633 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed540035f0 con 0x55c8d33e12c0 2024-04-01T01:48:44.871 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.868+0000 7fed68dae740 1 -- 172.21.15.70:0/545023633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e12c0 msgr2=0x55c8d33e5dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.868+0000 7fed68dae740 1 --2- 172.21.15.70:0/545023633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e12c0 0x55c8d33e5dc0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fed54009430 tx=0x7fed5403e000 comp rx=0 tx=0).stop 2024-04-01T01:48:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.869+0000 7fed68dae740 1 -- 172.21.15.70:0/545023633 shutdown_connections 2024-04-01T01:48:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.869+0000 7fed68dae740 1 --2- 172.21.15.70:0/545023633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e6430 0x55c8d33e88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.869+0000 7fed68dae740 1 --2- 172.21.15.70:0/545023633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e12c0 0x55c8d33e5dc0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.869+0000 7fed68dae740 1 --2- 172.21.15.70:0/545023633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e08d0 0x55c8d33e0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.869+0000 7fed68dae740 1 -- 172.21.15.70:0/545023633 >> 172.21.15.70:0/545023633 conn(0x55c8d33dbe40 msgr2=0x55c8d33dc240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.869+0000 7fed68dae740 1 -- 172.21.15.70:0/545023633 shutdown_connections 2024-04-01T01:48:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.869+0000 7fed68dae740 1 -- 172.21.15.70:0/545023633 wait complete. 2024-04-01T01:48:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.869+0000 7fed68dae740 1 -- start start 2024-04-01T01:48:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.869+0000 7fed68dae740 1 Processor -- start 2024-04-01T01:48:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed68dae740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e08d0 0x55c8d33fbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed68dae740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e12c0 0x55c8d33fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.872 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed68dae740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e6430 0x55c8d3403100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed5effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e12c0 0x55c8d33fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed5f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e08d0 0x55c8d33fbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed68dae740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c8d33fa3e0 con 0x55c8d33e6430 2024-04-01T01:48:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed68dae740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55c8d32e2f80 con 0x55c8d33e12c0 2024-04-01T01:48:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed5effd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e12c0 0x55c8d33fc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33566/0 (socket says 172.21.15.70:33566) 2024-04-01T01:48:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed5effd640 1 -- 172.21.15.70:0/511955880 learned_addr learned my addr 172.21.15.70:0/511955880 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed68dae740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c8d33fa560 con 0x55c8d33e08d0 2024-04-01T01:48:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed5ffff640 1 --2- 172.21.15.70:0/511955880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e6430 0x55c8d3403100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed68dae740 10 client.0 _ll_register_callbacks cb 0x55c8d333b290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed5effd640 1 -- 172.21.15.70:0/511955880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e08d0 msgr2=0x55c8d33fbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed5effd640 1 --2- 172.21.15.70:0/511955880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e08d0 0x55c8d33fbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed5effd640 1 -- 172.21.15.70:0/511955880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e6430 msgr2=0x55c8d3403100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:44.873 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed5effd640 1 --2- 172.21.15.70:0/511955880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e6430 0x55c8d3403100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.870+0000 7fed5effd640 1 -- 172.21.15.70:0/511955880 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d3125620 con 0x55c8d33e12c0 2024-04-01T01:48:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.871+0000 7fed5effd640 1 --2- 172.21.15.70:0/511955880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e12c0 0x55c8d33fc0c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fed54000c00 tx=0x7fed540048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b31c5386c4f0fef0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.871+0000 7fed3effd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.871+0000 7fed3effd640 1 -- 172.21.15.70:0/511955880 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed54005450 con 0x55c8d33e12c0 2024-04-01T01:48:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.871+0000 7fed3effd640 1 -- 172.21.15.70:0/511955880 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fed540035f0 con 0x55c8d33e12c0 2024-04-01T01:48:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.871+0000 7fed68dae740 1 -- 172.21.15.70:0/511955880 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed54050070 con 0x55c8d33e12c0 2024-04-01T01:48:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.871+0000 7fed3effd640 1 -- 172.21.15.70:0/511955880 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fed54066050 con 0x55c8d33e12c0 2024-04-01T01:48:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.871+0000 7fed68dae740 10 client.29957 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:44.874 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.871+0000 7fed68dae740 1 -- 172.21.15.70:0/511955880 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c8d3406010 con 0x55c8d33e12c0 2024-04-01T01:48:44.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.872+0000 7fed3effd640 1 -- 172.21.15.70:0/511955880 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed540055f0 con 0x55c8d33e12c0 2024-04-01T01:48:44.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.872+0000 7fed3effd640 1 -- 172.21.15.70:0/511955880 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fed540419b0 con 0x55c8d33e12c0 2024-04-01T01:48:44.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.872+0000 7fed68dae740 10 client.29957 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:44.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.872+0000 7fed68dae740 10 client.29957 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:44.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.872+0000 7fed68dae740 20 client.29957 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:44.876 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.872+0000 7fed68dae740 1 --2- 172.21.15.70:0/511955880 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c8d3407360 0x55c8d3409750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.872+0000 7fed68dae740 4 client.29957 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:44.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.872+0000 7fed68dae740 1 -- 172.21.15.70:0/511955880 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fed54050070 con 0x55c8d3407360 2024-04-01T01:48:44.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.873+0000 7fed5f7fe640 1 --2- 172.21.15.70:0/511955880 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c8d3407360 0x55c8d3409750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.873+0000 7fed3effd640 10 client.29957 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:44.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.873+0000 7fed5f7fe640 1 --2- 172.21.15.70:0/511955880 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c8d3407360 0x55c8d3409750 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0312e89f90b8858 server_cookie=a9470b30494586b3 in_seq=0 out_seq=0 2024-04-01T01:48:44.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.874+0000 7fed3effd640 1 -- 172.21.15.70:0/511955880 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fed54050070 con 0x55c8d3407360 2024-04-01T01:48:44.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.874+0000 7fed3effd640 10 client.29957 handle_command_reply: tid=0 2024-04-01T01:48:44.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.874+0000 7fed68dae740 1 client.29957 shutdown 2024-04-01T01:48:44.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.874+0000 7fed68dae740 1 -- 172.21.15.70:0/511955880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e12c0 msgr2=0x55c8d33fc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:44.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.874+0000 7fed68dae740 1 --2- 172.21.15.70:0/511955880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e12c0 0x55c8d33fc0c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fed54000c00 tx=0x7fed540048f0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.874+0000 7fed68dae740 1 -- 172.21.15.70:0/511955880 shutdown_connections 2024-04-01T01:48:44.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.874+0000 7fed68dae740 1 --2- 172.21.15.70:0/511955880 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c8d3407360 0x55c8d3409750 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.877 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.874+0000 7fed68dae740 1 --2- 172.21.15.70:0/511955880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e6430 0x55c8d3403100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.874+0000 7fed3effd640 0 client.29957 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:44.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.874+0000 7fed68dae740 1 --2- 172.21.15.70:0/511955880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e12c0 0x55c8d33fc0c0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.874+0000 7fed68dae740 1 --2- 172.21.15.70:0/511955880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e08d0 0x55c8d33fbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.874+0000 7fed68dae740 1 -- 172.21.15.70:0/511955880 >> 172.21.15.70:0/511955880 conn(0x55c8d33dbe40 msgr2=0x55c8d33daf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:44.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.874+0000 7fed68dae740 1 -- 172.21.15.70:0/511955880 shutdown_connections 2024-04-01T01:48:44.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.874+0000 7fed68dae740 1 -- 172.21.15.70:0/511955880 wait complete. 2024-04-01T01:48:44.878 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.874+0000 7fed68dae740 20 client.29957 trim_cache size 0 max 16384 2024-04-01T01:48:44.878 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:44 smithi081 ceph-mon[35142]: pgmap v816: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.0 MiB/s wr, 686 op/s 2024-04-01T01:48:44.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.899+0000 7fed68dae740 1 Processor -- start 2024-04-01T01:48:44.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.900+0000 7fed68dae740 1 -- start start 2024-04-01T01:48:44.902 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.900+0000 7fed68dae740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e08d0 0x55c8d33e8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.900+0000 7fed68dae740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e12c0 0x55c8d34030c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.900+0000 7fed68dae740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e6430 0x55c8d33e5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.900+0000 7fed68dae740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fed60070810 con 0x55c8d33e08d0 2024-04-01T01:48:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.900+0000 7fed68dae740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fed60072350 con 0x55c8d33e12c0 2024-04-01T01:48:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.900+0000 7fed5ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e6430 0x55c8d33e5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.900+0000 7fed5ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e6430 0x55c8d33e5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:47290/0 (socket says 172.21.15.70:47290) 2024-04-01T01:48:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.900+0000 7fed5ffff640 1 -- 172.21.15.70:0/313169958 learned_addr learned my addr 172.21.15.70:0/313169958 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.901+0000 7fed68dae740 1 -- 172.21.15.70:0/313169958 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fed60070ee0 con 0x55c8d33e6430 2024-04-01T01:48:44.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.901+0000 7fed5f7fe640 1 --2- 172.21.15.70:0/313169958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e08d0 0x55c8d33e8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.901+0000 7fed5ffff640 1 -- 172.21.15.70:0/313169958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e12c0 msgr2=0x55c8d34030c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:44.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.901+0000 7fed5ffff640 1 --2- 172.21.15.70:0/313169958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e12c0 0x55c8d34030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.901+0000 7fed5ffff640 1 -- 172.21.15.70:0/313169958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e08d0 msgr2=0x55c8d33e8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:44.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.901+0000 7fed5ffff640 1 --2- 172.21.15.70:0/313169958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e08d0 0x55c8d33e8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.901+0000 7fed5ffff640 1 -- 172.21.15.70:0/313169958 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8d3409ac0 con 0x55c8d33e6430 2024-04-01T01:48:44.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.901+0000 7fed5f7fe640 1 --2- 172.21.15.70:0/313169958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e08d0 0x55c8d33e8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:48:44.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.901+0000 7fed5ffff640 1 --2- 172.21.15.70:0/313169958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e6430 0x55c8d33e5b80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fed58003cc0 tx=0x7fed58004410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c0b1f7c0c555553 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:44.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.902+0000 7fed457fa640 1 -- 172.21.15.70:0/313169958 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed58010460 con 0x55c8d33e6430 2024-04-01T01:48:44.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.902+0000 7fed457fa640 1 -- 172.21.15.70:0/313169958 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed58010ab0 con 0x55c8d33e6430 2024-04-01T01:48:44.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.902+0000 7fed457fa640 1 -- 172.21.15.70:0/313169958 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed58011b10 con 0x55c8d33e6430 2024-04-01T01:48:44.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.904+0000 7fed68dae740 1 -- 172.21.15.70:0/313169958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e6430 msgr2=0x55c8d33e5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:44.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.904+0000 7fed68dae740 1 --2- 172.21.15.70:0/313169958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e6430 0x55c8d33e5b80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fed58003cc0 tx=0x7fed58004410 comp rx=0 tx=0).stop 2024-04-01T01:48:44.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.904+0000 7fed68dae740 1 -- 172.21.15.70:0/313169958 shutdown_connections 2024-04-01T01:48:44.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.904+0000 7fed68dae740 1 --2- 172.21.15.70:0/313169958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e6430 0x55c8d33e5b80 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.904+0000 7fed68dae740 1 --2- 172.21.15.70:0/313169958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e12c0 0x55c8d34030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.904+0000 7fed68dae740 1 --2- 172.21.15.70:0/313169958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e08d0 0x55c8d33e8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.904+0000 7fed68dae740 1 -- 172.21.15.70:0/313169958 >> 172.21.15.70:0/313169958 conn(0x55c8d33dbe40 msgr2=0x55c8d33dc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:44.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.905+0000 7fed68dae740 1 -- 172.21.15.70:0/313169958 shutdown_connections 2024-04-01T01:48:44.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.905+0000 7fed68dae740 1 -- 172.21.15.70:0/313169958 wait complete. 2024-04-01T01:48:44.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.906+0000 7fed68dae740 1 -- start start 2024-04-01T01:48:44.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.906+0000 7fed68dae740 1 Processor -- start 2024-04-01T01:48:44.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.906+0000 7fed68dae740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e08d0 0x55c8d3400350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.907+0000 7fed68dae740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e12c0 0x55c8d3400890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.907+0000 7fed5f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e08d0 0x55c8d3400350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.907+0000 7fed5f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e08d0 0x55c8d3400350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:33580/0 (socket says 172.21.15.70:33580) 2024-04-01T01:48:44.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.907+0000 7fed5f7fe640 1 -- 172.21.15.70:0/1036725853 learned_addr learned my addr 172.21.15.70:0/1036725853 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:44.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.907+0000 7fed5effd640 1 --2- 172.21.15.70:0/1036725853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e12c0 0x55c8d3400890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.907+0000 7fed68dae740 1 --2- 172.21.15.70:0/1036725853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e6430 0x55c8d34018a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.907+0000 7fed68dae740 1 -- 172.21.15.70:0/1036725853 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fed60071c80 con 0x55c8d33e6430 2024-04-01T01:48:44.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.907+0000 7fed68dae740 1 -- 172.21.15.70:0/1036725853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fed600715b0 con 0x55c8d33e08d0 2024-04-01T01:48:44.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.907+0000 7fed5ffff640 1 --2- 172.21.15.70:0/1036725853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e6430 0x55c8d34018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.907+0000 7fed68dae740 1 -- 172.21.15.70:0/1036725853 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fed6006e7f0 con 0x55c8d33e12c0 2024-04-01T01:48:44.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.908+0000 7fed5f7fe640 1 -- 172.21.15.70:0/1036725853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e12c0 msgr2=0x55c8d3400890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:44.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.908+0000 7fed5f7fe640 1 --2- 172.21.15.70:0/1036725853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e12c0 0x55c8d3400890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.908+0000 7fed5f7fe640 1 -- 172.21.15.70:0/1036725853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e6430 msgr2=0x55c8d34018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:44.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.908+0000 7fed5f7fe640 1 --2- 172.21.15.70:0/1036725853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e6430 0x55c8d34018a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.908+0000 7fed5f7fe640 1 -- 172.21.15.70:0/1036725853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8d3401e90 con 0x55c8d33e08d0 2024-04-01T01:48:44.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.908+0000 7fed5f7fe640 1 --2- 172.21.15.70:0/1036725853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e08d0 0x55c8d3400350 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fed5009bee0 tx=0x7fed5000f420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b482e962ab47a452 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:44.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.908+0000 7fed3effd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:44.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.908+0000 7fed68dae740 10 client.0 _ll_register_callbacks cb 0x7fed500046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:44.911 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.908+0000 7fed68dae740 1 -- 172.21.15.70:0/1036725853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed58010ab0 con 0x55c8d33e08d0 2024-04-01T01:48:44.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.910+0000 7fed3effd640 1 -- 172.21.15.70:0/1036725853 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5009a250 con 0x55c8d33e08d0 2024-04-01T01:48:44.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.910+0000 7fed3effd640 1 -- 172.21.15.70:0/1036725853 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fed5009e040 con 0x55c8d33e08d0 2024-04-01T01:48:44.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.910+0000 7fed3effd640 1 -- 172.21.15.70:0/1036725853 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fed500195d0 con 0x55c8d33e08d0 2024-04-01T01:48:44.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.910+0000 7fed68dae740 10 client.29960 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:44.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.911+0000 7fed68dae740 1 -- 172.21.15.70:0/1036725853 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55c8d3403080 con 0x55c8d33e08d0 2024-04-01T01:48:44.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.911+0000 7fed3effd640 1 -- 172.21.15.70:0/1036725853 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fed50021020 con 0x55c8d33e08d0 2024-04-01T01:48:44.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.911+0000 7fed68dae740 10 client.29960 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:44.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.911+0000 7fed68dae740 10 client.29960 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:44.912 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.911+0000 7fed68dae740 20 client.29960 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:44.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.911+0000 7fed68dae740 1 --2- 172.21.15.70:0/1036725853 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c8d3407360 0x55c8d346f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:44.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.911+0000 7fed68dae740 4 client.29960 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:48:44.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.911+0000 7fed68dae740 1 -- 172.21.15.70:0/1036725853 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fed58010ab0 con 0x55c8d3407360 2024-04-01T01:48:44.913 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.911+0000 7fed5effd640 1 --2- 172.21.15.70:0/1036725853 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c8d3407360 0x55c8d346f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:44.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.914+0000 7fed5effd640 1 --2- 172.21.15.70:0/1036725853 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c8d3407360 0x55c8d346f8a0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bf94db428d43206 server_cookie=fad39f9ccbdd4dce in_seq=0 out_seq=0 2024-04-01T01:48:44.914 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.914+0000 7fed3effd640 10 client.29960 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:44.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.942+0000 7fed3effd640 1 -- 172.21.15.70:0/1036725853 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fed58010ab0 con 0x55c8d3407360 2024-04-01T01:48:44.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.942+0000 7fed3effd640 10 client.29960 handle_command_reply: tid=0 2024-04-01T01:48:44.942 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.942+0000 7fed68dae740 1 client.29960 shutdown 2024-04-01T01:48:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.942+0000 7fed68dae740 1 -- 172.21.15.70:0/1036725853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e08d0 msgr2=0x55c8d3400350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.942+0000 7fed68dae740 1 --2- 172.21.15.70:0/1036725853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e08d0 0x55c8d3400350 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fed5009bee0 tx=0x7fed5000f420 comp rx=0 tx=0).stop 2024-04-01T01:48:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.943+0000 7fed68dae740 1 -- 172.21.15.70:0/1036725853 shutdown_connections 2024-04-01T01:48:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.943+0000 7fed68dae740 1 --2- 172.21.15.70:0/1036725853 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55c8d3407360 0x55c8d346f8a0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.943+0000 7fed68dae740 1 --2- 172.21.15.70:0/1036725853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c8d33e6430 0x55c8d34018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.943 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.943+0000 7fed68dae740 1 --2- 172.21.15.70:0/1036725853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8d33e12c0 0x55c8d3400890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.943+0000 7fed68dae740 1 --2- 172.21.15.70:0/1036725853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8d33e08d0 0x55c8d3400350 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.943+0000 7fed68dae740 1 -- 172.21.15.70:0/1036725853 >> 172.21.15.70:0/1036725853 conn(0x55c8d33dbe40 msgr2=0x55c8d33e8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.943+0000 7fed3effd640 0 client.29960 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.944+0000 7fed68dae740 1 -- 172.21.15.70:0/1036725853 shutdown_connections 2024-04-01T01:48:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.944+0000 7fed68dae740 1 -- 172.21.15.70:0/1036725853 wait complete. 2024-04-01T01:48:44.944 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.944+0000 7fed68dae740 20 client.29960 trim_cache size 0 max 16384 2024-04-01T01:48:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:48:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:48:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:48:44.946 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:48:44.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.946+0000 7fed660ed640 1 -- 172.21.15.70:0/182601534 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fed480a25c0 msgr2=0x7fed480a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:44.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.946+0000 7fed660ed640 1 --2- 172.21.15.70:0/182601534 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fed480a25c0 0x7fed480a4a80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fed60197cf0 tx=0x7fed5402c7e0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.946+0000 7fed660ed640 1 -- 172.21.15.70:0/182601534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed60100670 msgr2=0x7fed60196830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:44.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.946+0000 7fed660ed640 1 --2- 172.21.15.70:0/182601534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed60100670 0x7fed60196830 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fed601016c0 tx=0x7fed5000be10 comp rx=0 tx=0).stop 2024-04-01T01:48:44.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.947+0000 7fed660ed640 1 -- 172.21.15.70:0/182601534 shutdown_connections 2024-04-01T01:48:44.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.947+0000 7fed660ed640 1 --2- 172.21.15.70:0/182601534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed60102220 0x7fed6019dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.947+0000 7fed660ed640 1 --2- 172.21.15.70:0/182601534 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fed480a25c0 0x7fed480a4a80 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.947+0000 7fed660ed640 1 --2- 172.21.15.70:0/182601534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed60101860 0x7fed60196d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.947+0000 7fed660ed640 1 --2- 172.21.15.70:0/182601534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed60100670 0x7fed60196830 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:44.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.947+0000 7fed660ed640 1 -- 172.21.15.70:0/182601534 >> 172.21.15.70:0/182601534 conn(0x7fed600fbbe0 msgr2=0x7fed60104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:44.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.947+0000 7fed660ed640 1 -- 172.21.15.70:0/182601534 shutdown_connections 2024-04-01T01:48:44.948 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:44.947+0000 7fed660ed640 1 -- 172.21.15.70:0/182601534 wait complete. 2024-04-01T01:48:44.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:44 smithi189 ceph-mon[35753]: pgmap v816: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.0 MiB/s wr, 686 op/s 2024-04-01T01:48:45.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:44 smithi070 ceph-mon[32179]: pgmap v816: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 6.0 MiB/s wr, 686 op/s 2024-04-01T01:48:46.898 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 840.0 s, 749.0 tps, lat 135.050 ms stddev 133.969 2024-04-01T01:48:46.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:46 smithi189 ceph-mon[35753]: pgmap v817: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.9 MiB/s wr, 675 op/s 2024-04-01T01:48:47.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:46 smithi070 ceph-mon[32179]: pgmap v817: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.9 MiB/s wr, 675 op/s 2024-04-01T01:48:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:46 smithi081 ceph-mon[35142]: pgmap v817: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 5.9 MiB/s wr, 675 op/s 2024-04-01T01:48:48.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:48 smithi189 ceph-mon[35753]: pgmap v818: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 5.9 MiB/s wr, 679 op/s 2024-04-01T01:48:49.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:48 smithi070 ceph-mon[32179]: pgmap v818: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 5.9 MiB/s wr, 679 op/s 2024-04-01T01:48:49.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:48 smithi081 ceph-mon[35142]: pgmap v818: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 5.9 MiB/s wr, 679 op/s 2024-04-01T01:48:50.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:50 smithi189 ceph-mon[35753]: pgmap v819: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.4 MiB/s wr, 720 op/s 2024-04-01T01:48:51.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:50 smithi070 ceph-mon[32179]: pgmap v819: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.4 MiB/s wr, 720 op/s 2024-04-01T01:48:51.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:50 smithi081 ceph-mon[35142]: pgmap v819: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.4 MiB/s wr, 720 op/s 2024-04-01T01:48:51.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 845.0 s, 802.8 tps, lat 123.543 ms stddev 126.167 2024-04-01T01:48:52.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:52 smithi189 ceph-mon[35753]: pgmap v820: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 6.5 MiB/s wr, 753 op/s 2024-04-01T01:48:53.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:52 smithi070 ceph-mon[32179]: pgmap v820: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 6.5 MiB/s wr, 753 op/s 2024-04-01T01:48:53.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:52 smithi081 ceph-mon[35142]: pgmap v820: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 6.5 MiB/s wr, 753 op/s 2024-04-01T01:48:54.747 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:48:54.747 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:48:54.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.928+0000 7f07960da640 1 -- 172.21.15.70:0/448997548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0790100fc0 msgr2=0x7f0790101440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:54.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.928+0000 7f07960da640 1 --2- 172.21.15.70:0/448997548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0790100fc0 0x7f0790101440 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f0778009900 tx=0x7f077802bc30 comp rx=0 tx=0).stop 2024-04-01T01:48:54.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.928+0000 7f07960da640 1 -- 172.21.15.70:0/448997548 shutdown_connections 2024-04-01T01:48:54.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.928+0000 7f07960da640 1 --2- 172.21.15.70:0/448997548 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0790101980 0x7f0790108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.928+0000 7f07960da640 1 --2- 172.21.15.70:0/448997548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0790100fc0 0x7f0790101440 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.928+0000 7f07960da640 1 --2- 172.21.15.70:0/448997548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07900ffdd0 0x7f07901001f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.928+0000 7f07960da640 1 -- 172.21.15.70:0/448997548 >> 172.21.15.70:0/448997548 conn(0x7f07900fb340 msgr2=0x7f07900fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:54.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.928+0000 7f07960da640 1 -- 172.21.15.70:0/448997548 shutdown_connections 2024-04-01T01:48:54.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.929+0000 7f07960da640 1 -- 172.21.15.70:0/448997548 wait complete. 2024-04-01T01:48:54.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.929+0000 7f07960da640 1 Processor -- start 2024-04-01T01:48:54.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.929+0000 7f07960da640 1 -- start start 2024-04-01T01:48:54.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.930+0000 7f07960da640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07900ffdd0 0x7f079019dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.930+0000 7f07960da640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0790100fc0 0x7f079019e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.930+0000 7f07960da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0790101980 0x7f07901a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.930+0000 7f07960da640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f079010b0c0 con 0x7f0790100fc0 2024-04-01T01:48:54.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.930+0000 7f07960da640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f079010af40 con 0x7f0790101980 2024-04-01T01:48:54.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.930+0000 7f07960da640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f079010b240 con 0x7f07900ffdd0 2024-04-01T01:48:54.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.930+0000 7f078f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07900ffdd0 0x7f079019dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:54.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.930+0000 7f078f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07900ffdd0 0x7f079019dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:34164/0 (socket says 172.21.15.70:34164) 2024-04-01T01:48:54.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.930+0000 7f078f7fe640 1 -- 172.21.15.70:0/2268160249 learned_addr learned my addr 172.21.15.70:0/2268160249 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:54.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.931+0000 7f078ffff640 1 --2- 172.21.15.70:0/2268160249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0790101980 0x7f07901a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:54.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.931+0000 7f0786ffd640 1 --2- 172.21.15.70:0/2268160249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0790100fc0 0x7f079019e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:54.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.931+0000 7f078f7fe640 1 -- 172.21.15.70:0/2268160249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0790101980 msgr2=0x7f07901a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:54.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.931+0000 7f078f7fe640 1 --2- 172.21.15.70:0/2268160249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0790101980 0x7f07901a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.931+0000 7f078f7fe640 1 -- 172.21.15.70:0/2268160249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0790100fc0 msgr2=0x7f079019e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:54.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.931+0000 7f078f7fe640 1 --2- 172.21.15.70:0/2268160249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0790100fc0 0x7f079019e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.931+0000 7f078f7fe640 1 -- 172.21.15.70:0/2268160249 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0790195510 con 0x7f07900ffdd0 2024-04-01T01:48:54.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.931+0000 7f078f7fe640 1 --2- 172.21.15.70:0/2268160249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07900ffdd0 0x7f079019dfd0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f078000fad0 tx=0x7f078000ff90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:54.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.932+0000 7f078d7fa640 1 -- 172.21.15.70:0/2268160249 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078001c070 con 0x7f07900ffdd0 2024-04-01T01:48:54.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.932+0000 7f07960da640 1 -- 172.21.15.70:0/2268160249 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0790195800 con 0x7f07900ffdd0 2024-04-01T01:48:54.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.932+0000 7f078d7fa640 1 -- 172.21.15.70:0/2268160249 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f078000dbd0 con 0x7f07900ffdd0 2024-04-01T01:48:54.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.932+0000 7f078d7fa640 1 -- 172.21.15.70:0/2268160249 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0780003d70 con 0x7f07900ffdd0 2024-04-01T01:48:54.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.932+0000 7f07960da640 1 -- 172.21.15.70:0/2268160249 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0790195d20 con 0x7f07900ffdd0 2024-04-01T01:48:54.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.936+0000 7f078d7fa640 1 -- 172.21.15.70:0/2268160249 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f078001f070 con 0x7f07900ffdd0 2024-04-01T01:48:54.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.937+0000 7f078d7fa640 1 --2- 172.21.15.70:0/2268160249 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07700a26a0 0x7f07700a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.937+0000 7f078d7fa640 1 -- 172.21.15.70:0/2268160249 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f07800ce560 con 0x7f07900ffdd0 2024-04-01T01:48:54.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.937+0000 7f0786ffd640 1 --2- 172.21.15.70:0/2268160249 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07700a26a0 0x7f07700a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:54.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.938+0000 7f0798d95740 1 Processor -- start 2024-04-01T01:48:54.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.938+0000 7f0798d95740 1 -- start start 2024-04-01T01:48:54.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.938+0000 7f0798d95740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d6752b0 0x55847d6756d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.938+0000 7f0798d95740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d675ca0 0x55847d67a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.938+0000 7f0798d95740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d67ae10 0x55847d67d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.938+0000 7f0798d95740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55847d5da180 con 0x55847d67ae10 2024-04-01T01:48:54.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.938+0000 7f0798d95740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55847d582b70 con 0x55847d675ca0 2024-04-01T01:48:54.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.938+0000 7f0798d95740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55847d5d6220 con 0x55847d6752b0 2024-04-01T01:48:54.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.939+0000 7f078ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d67ae10 0x55847d67d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:54.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.939+0000 7f078ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d67ae10 0x55847d67d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49194/0 (socket says 172.21.15.70:49194) 2024-04-01T01:48:54.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.939+0000 7f078ffff640 1 -- 172.21.15.70:0/2618491802 learned_addr learned my addr 172.21.15.70:0/2618491802 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:54.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.939+0000 7f0786ffd640 1 --2- 172.21.15.70:0/2618491802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d675ca0 0x55847d67a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:54.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.940+0000 7f078ffff640 1 -- 172.21.15.70:0/2618491802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d6752b0 msgr2=0x55847d6756d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:48:54.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.940+0000 7f078ffff640 1 --2- 172.21.15.70:0/2618491802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d6752b0 0x55847d6756d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.940+0000 7f078ffff640 1 -- 172.21.15.70:0/2618491802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d675ca0 msgr2=0x55847d67a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:54.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.940+0000 7f078ffff640 1 --2- 172.21.15.70:0/2618491802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d675ca0 0x55847d67a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.940+0000 7f078ffff640 1 -- 172.21.15.70:0/2618491802 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55847d5967c0 con 0x55847d67ae10 2024-04-01T01:48:54.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.940+0000 7f078ffff640 1 --2- 172.21.15.70:0/2618491802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d67ae10 0x55847d67d2a0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f0788009ee0 tx=0x7f0788010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0fee34017fa333e server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:54.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.940+0000 7f0786ffd640 1 --2- 172.21.15.70:0/2268160249 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07700a26a0 0x7f07700a4b60 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0778006010 tx=0x7f0778009620 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:54.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.941+0000 7f0785ffb640 1 -- 172.21.15.70:0/2618491802 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0788016070 con 0x55847d67ae10 2024-04-01T01:48:54.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.941+0000 7f0785ffb640 1 -- 172.21.15.70:0/2618491802 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0788004d20 con 0x55847d67ae10 2024-04-01T01:48:54.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.941+0000 7f0785ffb640 1 -- 172.21.15.70:0/2618491802 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0788012740 con 0x55847d67ae10 2024-04-01T01:48:54.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.943+0000 7f0798d95740 1 -- 172.21.15.70:0/2618491802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d67ae10 msgr2=0x55847d67d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:54.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.943+0000 7f0798d95740 1 --2- 172.21.15.70:0/2618491802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d67ae10 0x55847d67d2a0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f0788009ee0 tx=0x7f0788010740 comp rx=0 tx=0).stop 2024-04-01T01:48:54.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.943+0000 7f0798d95740 1 -- 172.21.15.70:0/2618491802 shutdown_connections 2024-04-01T01:48:54.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.944+0000 7f0798d95740 1 --2- 172.21.15.70:0/2618491802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d67ae10 0x55847d67d2a0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.944+0000 7f0798d95740 1 --2- 172.21.15.70:0/2618491802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d675ca0 0x55847d67a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.944+0000 7f0798d95740 1 --2- 172.21.15.70:0/2618491802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d6752b0 0x55847d6756d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.944+0000 7f0798d95740 1 -- 172.21.15.70:0/2618491802 >> 172.21.15.70:0/2618491802 conn(0x55847d670820 msgr2=0x55847d670c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:54.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.944+0000 7f0798d95740 1 -- 172.21.15.70:0/2618491802 shutdown_connections 2024-04-01T01:48:54.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.944+0000 7f0798d95740 1 -- 172.21.15.70:0/2618491802 wait complete. 2024-04-01T01:48:54.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.944+0000 7f0798d95740 1 -- start start 2024-04-01T01:48:54.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.944+0000 7f0798d95740 1 Processor -- start 2024-04-01T01:48:54.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.945+0000 7f0798d95740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d6752b0 0x55847d6906f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.945+0000 7f0798d95740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d675ca0 0x55847d690c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.945+0000 7f0798d95740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d67ae10 0x55847d697c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.945+0000 7f0798d95740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55847d68edd0 con 0x55847d67ae10 2024-04-01T01:48:54.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.945+0000 7f0798d95740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55847d5784a0 con 0x55847d675ca0 2024-04-01T01:48:54.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.945+0000 7f0798d95740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55847d68ef50 con 0x55847d6752b0 2024-04-01T01:48:54.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.945+0000 7f0798d95740 10 client.0 _ll_register_callbacks cb 0x55847d57bd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:54.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.946+0000 7f078ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d67ae10 0x55847d697c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:54.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.946+0000 7f078ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d67ae10 0x55847d697c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49198/0 (socket says 172.21.15.70:49198) 2024-04-01T01:48:54.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.946+0000 7f078ffff640 1 -- 172.21.15.70:0/2441235029 learned_addr learned my addr 172.21.15.70:0/2441235029 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:54.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.946+0000 7f078f7fe640 1 --2- 172.21.15.70:0/2441235029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d6752b0 0x55847d6906f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:54.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.946+0000 7f078f7fe640 1 -- 172.21.15.70:0/2441235029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d675ca0 msgr2=0x55847d690c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:54.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.946+0000 7f078f7fe640 1 --2- 172.21.15.70:0/2441235029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d675ca0 0x55847d690c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.946+0000 7f078f7fe640 1 -- 172.21.15.70:0/2441235029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d67ae10 msgr2=0x55847d697c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:54.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.946+0000 7f078f7fe640 1 --2- 172.21.15.70:0/2441235029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d67ae10 0x55847d697c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.946+0000 7f078f7fe640 1 -- 172.21.15.70:0/2441235029 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55847d3b87d0 con 0x55847d6752b0 2024-04-01T01:48:54.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.948+0000 7f078f7fe640 1 --2- 172.21.15.70:0/2441235029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d6752b0 0x55847d6906f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x55847d676500 tx=0x7f078009a610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ee3d257a27e4730 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:54.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.948+0000 7f077effd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:48:54.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.948+0000 7f077effd640 1 -- 172.21.15.70:0/2441235029 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078009ed40 con 0x55847d6752b0 2024-04-01T01:48:54.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.948+0000 7f0798d95740 1 -- 172.21.15.70:0/2441235029 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0788004d20 con 0x55847d6752b0 2024-04-01T01:48:54.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.948+0000 7f077effd640 1 -- 172.21.15.70:0/2441235029 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f07800ce1d0 con 0x55847d6752b0 2024-04-01T01:48:54.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.948+0000 7f077effd640 1 -- 172.21.15.70:0/2441235029 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f07800a3600 con 0x55847d6752b0 2024-04-01T01:48:54.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.948+0000 7f0798d95740 10 client.29698 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:54.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.948+0000 7f0798d95740 1 -- 172.21.15.70:0/2441235029 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55847d2ee3f0 con 0x55847d6752b0 2024-04-01T01:48:54.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.949+0000 7f077effd640 1 -- 172.21.15.70:0/2441235029 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0788004d20 con 0x55847d6752b0 2024-04-01T01:48:54.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.949+0000 7f077effd640 1 -- 172.21.15.70:0/2441235029 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f0780096940 con 0x55847d6752b0 2024-04-01T01:48:54.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.949+0000 7f0798d95740 10 client.29698 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:54.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.949+0000 7f0798d95740 10 client.29698 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:54.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.949+0000 7f0798d95740 20 client.29698 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:54.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.949+0000 7f0798d95740 1 --2- 172.21.15.70:0/2441235029 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55847d69c5a0 0x55847d69e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.949+0000 7f0798d95740 4 client.29698 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:54.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.949+0000 7f0798d95740 1 -- 172.21.15.70:0/2441235029 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55847d5e76c0 con 0x55847d69c5a0 2024-04-01T01:48:54.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.950+0000 7f0786ffd640 1 --2- 172.21.15.70:0/2441235029 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55847d69c5a0 0x55847d69e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:54.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.951+0000 7f0786ffd640 1 --2- 172.21.15.70:0/2441235029 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55847d69c5a0 0x55847d69e9e0 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de48b8c7f797364f server_cookie=5753a3bb0af7653a in_seq=0 out_seq=0 2024-04-01T01:48:54.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.951+0000 7f077effd640 10 client.29698 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:54.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.952+0000 7f077effd640 1 -- 172.21.15.70:0/2441235029 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55847d5e76c0 con 0x55847d69c5a0 2024-04-01T01:48:54.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.952+0000 7f077effd640 10 client.29698 handle_command_reply: tid=0 2024-04-01T01:48:54.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.952+0000 7f0798d95740 1 client.29698 shutdown 2024-04-01T01:48:54.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.953+0000 7f0798d95740 1 -- 172.21.15.70:0/2441235029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d6752b0 msgr2=0x55847d6906f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:54.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.953+0000 7f0798d95740 1 --2- 172.21.15.70:0/2441235029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d6752b0 0x55847d6906f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x55847d676500 tx=0x7f078009a610 comp rx=0 tx=0).stop 2024-04-01T01:48:54.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.953+0000 7f0798d95740 1 -- 172.21.15.70:0/2441235029 shutdown_connections 2024-04-01T01:48:54.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.953+0000 7f0798d95740 1 --2- 172.21.15.70:0/2441235029 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55847d69c5a0 0x55847d69e9e0 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.953+0000 7f0798d95740 1 --2- 172.21.15.70:0/2441235029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d67ae10 0x55847d697c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.953+0000 7f0798d95740 1 --2- 172.21.15.70:0/2441235029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d675ca0 0x55847d690c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.953+0000 7f0798d95740 1 --2- 172.21.15.70:0/2441235029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d6752b0 0x55847d6906f0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.953+0000 7f0798d95740 1 -- 172.21.15.70:0/2441235029 >> 172.21.15.70:0/2441235029 conn(0x55847d670820 msgr2=0x55847d66fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:54.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.953+0000 7f077effd640 0 client.29698 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:54.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.953+0000 7f0798d95740 1 -- 172.21.15.70:0/2441235029 shutdown_connections 2024-04-01T01:48:54.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.953+0000 7f0798d95740 1 -- 172.21.15.70:0/2441235029 wait complete. 2024-04-01T01:48:54.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.953+0000 7f0798d95740 20 client.29698 trim_cache size 0 max 16384 2024-04-01T01:48:54.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.984+0000 7f0798d95740 1 Processor -- start 2024-04-01T01:48:54.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.984+0000 7f0798d95740 1 -- start start 2024-04-01T01:48:54.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.984+0000 7f0798d95740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d6752b0 0x55847d690a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.984+0000 7f0798d95740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d675ca0 0x55847d6a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.984+0000 7f0798d95740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d67ae10 0x55847d68f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.984+0000 7f0798d95740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f077400fda0 con 0x55847d675ca0 2024-04-01T01:48:54.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.984+0000 7f0798d95740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f07740118e0 con 0x55847d67ae10 2024-04-01T01:48:54.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.984+0000 7f0798d95740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0774010470 con 0x55847d6752b0 2024-04-01T01:48:54.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.984+0000 7f078ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d67ae10 0x55847d68f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:54.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.984+0000 7f078ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d67ae10 0x55847d68f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57222/0 (socket says 172.21.15.70:57222) 2024-04-01T01:48:54.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.984+0000 7f078ffff640 1 -- 172.21.15.70:0/659849685 learned_addr learned my addr 172.21.15.70:0/659849685 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:54.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.985+0000 7f078ffff640 1 -- 172.21.15.70:0/659849685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d6752b0 msgr2=0x55847d690a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:48:54.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.985+0000 7f078ffff640 1 --2- 172.21.15.70:0/659849685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d6752b0 0x55847d690a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.985+0000 7f078ffff640 1 -- 172.21.15.70:0/659849685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d675ca0 msgr2=0x55847d6a42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:54.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.985+0000 7f078ffff640 1 --2- 172.21.15.70:0/659849685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d675ca0 0x55847d6a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.985+0000 7f078ffff640 1 -- 172.21.15.70:0/659849685 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55847d2ae0f0 con 0x55847d67ae10 2024-04-01T01:48:54.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.985+0000 7f078ffff640 1 --2- 172.21.15.70:0/659849685 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d67ae10 0x55847d68f350 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0788009c60 tx=0x7f07880097e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8367cb501946ba6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:54.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.987+0000 7f07857fa640 1 -- 172.21.15.70:0/659849685 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0788016040 con 0x55847d67ae10 2024-04-01T01:48:54.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.987+0000 7f07857fa640 1 -- 172.21.15.70:0/659849685 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f078804a070 con 0x55847d67ae10 2024-04-01T01:48:54.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.988+0000 7f0798d95740 1 -- 172.21.15.70:0/659849685 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d67ae10 msgr2=0x55847d68f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:54.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.988+0000 7f0798d95740 1 --2- 172.21.15.70:0/659849685 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d67ae10 0x55847d68f350 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0788009c60 tx=0x7f07880097e0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.988+0000 7f0798d95740 1 -- 172.21.15.70:0/659849685 shutdown_connections 2024-04-01T01:48:54.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.988+0000 7f0798d95740 1 --2- 172.21.15.70:0/659849685 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d67ae10 0x55847d68f350 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.988+0000 7f0798d95740 1 --2- 172.21.15.70:0/659849685 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d675ca0 0x55847d6a42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.988+0000 7f0798d95740 1 --2- 172.21.15.70:0/659849685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d6752b0 0x55847d690a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.988+0000 7f0798d95740 1 -- 172.21.15.70:0/659849685 >> 172.21.15.70:0/659849685 conn(0x55847d670820 msgr2=0x55847d690300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:54.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.989+0000 7f0798d95740 1 -- 172.21.15.70:0/659849685 shutdown_connections 2024-04-01T01:48:54.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.989+0000 7f0798d95740 1 -- 172.21.15.70:0/659849685 wait complete. 2024-04-01T01:48:54.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.989+0000 7f0798d95740 1 -- start start 2024-04-01T01:48:54.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.989+0000 7f0798d95740 1 Processor -- start 2024-04-01T01:48:54.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.990+0000 7f0798d95740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d6752b0 0x55847d695200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.990+0000 7f0798d95740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d675ca0 0x55847d695740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.990+0000 7f0798d95740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d67ae10 0x55847d696750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.990+0000 7f0798d95740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0774011210 con 0x55847d6752b0 2024-04-01T01:48:54.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.990+0000 7f0798d95740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0774010b40 con 0x55847d67ae10 2024-04-01T01:48:54.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.990+0000 7f0798d95740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f077400dd80 con 0x55847d675ca0 2024-04-01T01:48:54.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.990+0000 7f0798d95740 10 client.0 _ll_register_callbacks cb 0x7f07800035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:54.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.990+0000 7f0786ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d675ca0 0x55847d695740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:54.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.990+0000 7f0786ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d675ca0 0x55847d695740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:34190/0 (socket says 172.21.15.70:34190) 2024-04-01T01:48:54.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.990+0000 7f0786ffd640 1 -- 172.21.15.70:0/2794057153 learned_addr learned my addr 172.21.15.70:0/2794057153 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:54.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.990+0000 7f078f7fe640 1 --2- 172.21.15.70:0/2794057153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d6752b0 0x55847d695200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:54.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.990+0000 7f078ffff640 1 --2- 172.21.15.70:0/2794057153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d67ae10 0x55847d696750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:54.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.991+0000 7f0786ffd640 1 -- 172.21.15.70:0/2794057153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d67ae10 msgr2=0x55847d696750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:54.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.991+0000 7f0786ffd640 1 --2- 172.21.15.70:0/2794057153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d67ae10 0x55847d696750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.991+0000 7f0786ffd640 1 -- 172.21.15.70:0/2794057153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d6752b0 msgr2=0x55847d695200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:54.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.991+0000 7f0786ffd640 1 --2- 172.21.15.70:0/2794057153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d6752b0 0x55847d695200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.991+0000 7f0786ffd640 1 -- 172.21.15.70:0/2794057153 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55847d696d40 con 0x55847d675ca0 2024-04-01T01:48:54.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.991+0000 7f0786ffd640 1 --2- 172.21.15.70:0/2794057153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d675ca0 0x55847d695740 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f0778009230 tx=0x7f077802c1d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7a8c1e692045d6b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:54.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.991+0000 7f077effd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:48:54.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.992+0000 7f077effd640 1 -- 172.21.15.70:0/2794057153 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778046070 con 0x55847d675ca0 2024-04-01T01:48:54.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.992+0000 7f077effd640 1 -- 172.21.15.70:0/2794057153 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f07780505f0 con 0x55847d675ca0 2024-04-01T01:48:54.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.992+0000 7f0798d95740 1 -- 172.21.15.70:0/2794057153 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55847d5e1a10 con 0x55847d675ca0 2024-04-01T01:48:54.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.993+0000 7f077effd640 1 -- 172.21.15.70:0/2794057153 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f07780507f0 con 0x55847d675ca0 2024-04-01T01:48:54.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.993+0000 7f0798d95740 10 client.29701 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.993+0000 7f0798d95740 1 -- 172.21.15.70:0/2794057153 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55847d702bc0 con 0x55847d675ca0 2024-04-01T01:48:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.993+0000 7f077effd640 1 -- 172.21.15.70:0/2794057153 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55847d5e1a10 con 0x55847d675ca0 2024-04-01T01:48:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.994+0000 7f077effd640 1 -- 172.21.15.70:0/2794057153 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f0778004b50 con 0x55847d675ca0 2024-04-01T01:48:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.995+0000 7f0798d95740 10 client.29701 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.995+0000 7f0798d95740 10 client.29701 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.995+0000 7f0798d95740 20 client.29701 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.995+0000 7f0798d95740 1 --2- 172.21.15.70:0/2794057153 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55847d7035a0 0x55847d7059c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.995+0000 7f0798d95740 4 client.29701 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:48:54.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.995+0000 7f0798d95740 1 -- 172.21.15.70:0/2794057153 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55847d55f540 con 0x55847d7035a0 2024-04-01T01:48:54.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.995+0000 7f078f7fe640 1 --2- 172.21.15.70:0/2794057153 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55847d7035a0 0x55847d7059c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:54.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.996+0000 7f078f7fe640 1 --2- 172.21.15.70:0/2794057153 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55847d7035a0 0x55847d7059c0 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=812269cb7017c5ae server_cookie=a5ffd17ee4290187 in_seq=0 out_seq=0 2024-04-01T01:48:54.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.996+0000 7f077effd640 10 client.29701 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:54.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.996+0000 7f077effd640 1 -- 172.21.15.70:0/2794057153 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55847d55f540 con 0x55847d7035a0 2024-04-01T01:48:54.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.996+0000 7f077effd640 10 client.29701 handle_command_reply: tid=0 2024-04-01T01:48:54.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.996+0000 7f0798d95740 1 client.29701 shutdown 2024-04-01T01:48:54.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.997+0000 7f0798d95740 1 -- 172.21.15.70:0/2794057153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d675ca0 msgr2=0x55847d695740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:54.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.997+0000 7f0798d95740 1 --2- 172.21.15.70:0/2794057153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d675ca0 0x55847d695740 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f0778009230 tx=0x7f077802c1d0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.997+0000 7f0798d95740 1 -- 172.21.15.70:0/2794057153 shutdown_connections 2024-04-01T01:48:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.997+0000 7f0798d95740 1 --2- 172.21.15.70:0/2794057153 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55847d7035a0 0x55847d7059c0 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.997+0000 7f0798d95740 1 --2- 172.21.15.70:0/2794057153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55847d67ae10 0x55847d696750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.997+0000 7f0798d95740 1 --2- 172.21.15.70:0/2794057153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55847d675ca0 0x55847d695740 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.997+0000 7f0798d95740 1 --2- 172.21.15.70:0/2794057153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55847d6752b0 0x55847d695200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.997+0000 7f0798d95740 1 -- 172.21.15.70:0/2794057153 >> 172.21.15.70:0/2794057153 conn(0x55847d670820 msgr2=0x55847d66fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.997+0000 7f077effd640 0 client.29701 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.997+0000 7f0798d95740 1 -- 172.21.15.70:0/2794057153 shutdown_connections 2024-04-01T01:48:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.998+0000 7f0798d95740 1 -- 172.21.15.70:0/2794057153 wait complete. 2024-04-01T01:48:54.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:54.998+0000 7f0798d95740 20 client.29701 trim_cache size 0 max 16384 2024-04-01T01:48:54.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:54 smithi189 ceph-mon[35753]: pgmap v821: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.4 MiB/s wr, 731 op/s 2024-04-01T01:48:54.999 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:48:54.999 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:48:54.999 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:48:54.999 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:48:55.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.000+0000 7f07960da640 1 -- 172.21.15.70:0/2268160249 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07700a26a0 msgr2=0x7f07700a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:55.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.000+0000 7f07960da640 1 --2- 172.21.15.70:0/2268160249 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07700a26a0 0x7f07700a4b60 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f0778006010 tx=0x7f0778009620 comp rx=0 tx=0).stop 2024-04-01T01:48:55.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.000+0000 7f07960da640 1 -- 172.21.15.70:0/2268160249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07900ffdd0 msgr2=0x7f079019dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:55.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.000+0000 7f07960da640 1 --2- 172.21.15.70:0/2268160249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07900ffdd0 0x7f079019dfd0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f078000fad0 tx=0x7f078000ff90 comp rx=0 tx=0).stop 2024-04-01T01:48:55.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.000+0000 7f07960da640 1 -- 172.21.15.70:0/2268160249 shutdown_connections 2024-04-01T01:48:55.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.000+0000 7f07960da640 1 --2- 172.21.15.70:0/2268160249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0790101980 0x7f07901a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.000+0000 7f07960da640 1 --2- 172.21.15.70:0/2268160249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0790100fc0 0x7f079019e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.000+0000 7f07960da640 1 --2- 172.21.15.70:0/2268160249 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f07700a26a0 0x7f07700a4b60 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.000+0000 7f07960da640 1 --2- 172.21.15.70:0/2268160249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07900ffdd0 0x7f079019dfd0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.000+0000 7f07960da640 1 -- 172.21.15.70:0/2268160249 >> 172.21.15.70:0/2268160249 conn(0x7f07900fb340 msgr2=0x7f0790104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:55.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.001+0000 7f07960da640 1 -- 172.21.15.70:0/2268160249 shutdown_connections 2024-04-01T01:48:55.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.001+0000 7f07960da640 1 -- 172.21.15.70:0/2268160249 wait complete. 2024-04-01T01:48:55.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:54 smithi070 ceph-mon[32179]: pgmap v821: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.4 MiB/s wr, 731 op/s 2024-04-01T01:48:55.019 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:48:55.019 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:48:55.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:54 smithi081 ceph-mon[35142]: pgmap v821: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.4 MiB/s wr, 731 op/s 2024-04-01T01:48:55.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.213+0000 7f8d92e8a640 1 -- 172.21.15.70:0/1021597631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d8c102220 msgr2=0x7f8d8c108ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:55.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.213+0000 7f8d92e8a640 1 --2- 172.21.15.70:0/1021597631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d8c102220 0x7f8d8c108ab0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f8d8401acb0 tx=0x7f8d8403d170 comp rx=0 tx=0).stop 2024-04-01T01:48:55.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.213+0000 7f8d92e8a640 1 -- 172.21.15.70:0/1021597631 shutdown_connections 2024-04-01T01:48:55.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.213+0000 7f8d92e8a640 1 --2- 172.21.15.70:0/1021597631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d8c102220 0x7f8d8c108ab0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.213+0000 7f8d92e8a640 1 --2- 172.21.15.70:0/1021597631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d8c101860 0x7f8d8c101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.213+0000 7f8d92e8a640 1 --2- 172.21.15.70:0/1021597631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c100670 0x7f8d8c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.213+0000 7f8d92e8a640 1 -- 172.21.15.70:0/1021597631 >> 172.21.15.70:0/1021597631 conn(0x7f8d8c0fbbe0 msgr2=0x7f8d8c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:55.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.213+0000 7f8d92e8a640 1 -- 172.21.15.70:0/1021597631 shutdown_connections 2024-04-01T01:48:55.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.213+0000 7f8d92e8a640 1 -- 172.21.15.70:0/1021597631 wait complete. 2024-04-01T01:48:55.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.215+0000 7f8d92e8a640 1 Processor -- start 2024-04-01T01:48:55.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.215+0000 7f8d92e8a640 1 -- start start 2024-04-01T01:48:55.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.215+0000 7f8d92e8a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c100670 0x7f8d8c195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.215+0000 7f8d92e8a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d8c101860 0x7f8d8c195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.215+0000 7f8d92e8a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d8c102220 0x7f8d8c19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.216+0000 7f8d8bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d8c101860 0x7f8d8c195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.216+0000 7f8d90c26640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c100670 0x7f8d8c195650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.216+0000 7f8d90c26640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c100670 0x7f8d8c195650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:34210/0 (socket says 172.21.15.70:34210) 2024-04-01T01:48:55.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.216+0000 7f8d8bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d8c101860 0x7f8d8c195b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57250/0 (socket says 172.21.15.70:57250) 2024-04-01T01:48:55.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.216+0000 7f8d91427640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d8c102220 0x7f8d8c19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.216+0000 7f8d92e8a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c10b100 con 0x7f8d8c102220 2024-04-01T01:48:55.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.216+0000 7f8d92e8a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c10af80 con 0x7f8d8c101860 2024-04-01T01:48:55.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.216+0000 7f8d91427640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d8c102220 0x7f8d8c19cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49220/0 (socket says 172.21.15.70:49220) 2024-04-01T01:48:55.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.216+0000 7f8d91427640 1 -- 172.21.15.70:0/2482373683 learned_addr learned my addr 172.21.15.70:0/2482373683 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:55.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.216+0000 7f8d92e8a640 1 -- 172.21.15.70:0/2482373683 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c10b280 con 0x7f8d8c100670 2024-04-01T01:48:55.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.217+0000 7f8d91427640 1 -- 172.21.15.70:0/2482373683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c100670 msgr2=0x7f8d8c195650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:55.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.217+0000 7f8d91427640 1 --2- 172.21.15.70:0/2482373683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c100670 0x7f8d8c195650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.217+0000 7f8d91427640 1 -- 172.21.15.70:0/2482373683 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d8c101860 msgr2=0x7f8d8c195b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:55.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.217+0000 7f8d91427640 1 --2- 172.21.15.70:0/2482373683 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d8c101860 0x7f8d8c195b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.217+0000 7f8d91427640 1 -- 172.21.15.70:0/2482373683 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8c19d270 con 0x7f8d8c102220 2024-04-01T01:48:55.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.217+0000 7f8d91427640 1 --2- 172.21.15.70:0/2482373683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d8c102220 0x7f8d8c19cc00 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f8d8403d700 tx=0x7f8d84045670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:55.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.218+0000 7f8d89ffb640 1 -- 172.21.15.70:0/2482373683 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84052020 con 0x7f8d8c102220 2024-04-01T01:48:55.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.218+0000 7f8d89ffb640 1 -- 172.21.15.70:0/2482373683 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d84045c90 con 0x7f8d8c102220 2024-04-01T01:48:55.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.218+0000 7f8d92e8a640 1 -- 172.21.15.70:0/2482373683 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d8c19d500 con 0x7f8d8c102220 2024-04-01T01:48:55.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.218+0000 7f8d89ffb640 1 -- 172.21.15.70:0/2482373683 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8404a040 con 0x7f8d8c102220 2024-04-01T01:48:55.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.218+0000 7f8d92e8a640 1 -- 172.21.15.70:0/2482373683 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d8c19d8e0 con 0x7f8d8c102220 2024-04-01T01:48:55.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.220+0000 7f8d92e8a640 1 -- 172.21.15.70:0/2482373683 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d8c1078c0 con 0x7f8d8c102220 2024-04-01T01:48:55.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.222+0000 7f8d89ffb640 1 -- 172.21.15.70:0/2482373683 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f8d84055020 con 0x7f8d8c102220 2024-04-01T01:48:55.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.222+0000 7f8d89ffb640 1 --2- 172.21.15.70:0/2482373683 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8d7c0a2570 0x7f8d7c0a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.223+0000 7f8d89ffb640 1 -- 172.21.15.70:0/2482373683 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f8d84101cf0 con 0x7f8d8c102220 2024-04-01T01:48:55.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.223+0000 7f8d90c26640 1 --2- 172.21.15.70:0/2482373683 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8d7c0a2570 0x7f8d7c0a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.224+0000 7f8d90c26640 1 --2- 172.21.15.70:0/2482373683 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8d7c0a2570 0x7f8d7c0a4a30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c1016c0 tx=0x7f8d80008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:55.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.235+0000 7f8d89ffb640 1 -- 172.21.15.70:0/2482373683 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8d8405c020 con 0x7f8d8c102220 2024-04-01T01:48:55.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.656+0000 7f8d92e8a640 1 -- 172.21.15.70:0/2482373683 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d8c062350 con 0x7f8d8c102220 2024-04-01T01:48:55.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.658+0000 7f8d89ffb640 1 -- 172.21.15.70:0/2482373683 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11926 (secure 0 0 0) 0x7f8d840cea80 con 0x7f8d8c102220 2024-04-01T01:48:55.658 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:48:55.658 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:39:44.821635+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27302":{"gid":27302,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/980664761","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":980664761},{"type":"v1","addr":"172.21.15.189:6837","nonce":980664761}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24404,24418,24379,24464,24451]},"id":1}]} 2024-04-01T01:48:55.658 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 31 2024-04-01T01:48:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.662+0000 7f8d92e8a640 1 -- 172.21.15.70:0/2482373683 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8d7c0a2570 msgr2=0x7f8d7c0a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.662+0000 7f8d92e8a640 1 --2- 172.21.15.70:0/2482373683 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8d7c0a2570 0x7f8d7c0a4a30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c1016c0 tx=0x7f8d80008040 comp rx=0 tx=0).stop 2024-04-01T01:48:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.662+0000 7f8d92e8a640 1 -- 172.21.15.70:0/2482373683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d8c102220 msgr2=0x7f8d8c19cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.662+0000 7f8d92e8a640 1 --2- 172.21.15.70:0/2482373683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d8c102220 0x7f8d8c19cc00 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f8d8403d700 tx=0x7f8d84045670 comp rx=0 tx=0).stop 2024-04-01T01:48:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.663+0000 7f8d92e8a640 1 -- 172.21.15.70:0/2482373683 shutdown_connections 2024-04-01T01:48:55.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.663+0000 7f8d92e8a640 1 --2- 172.21.15.70:0/2482373683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d8c102220 0x7f8d8c19cc00 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.663+0000 7f8d92e8a640 1 --2- 172.21.15.70:0/2482373683 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8d8c101860 0x7f8d8c195b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.663+0000 7f8d92e8a640 1 --2- 172.21.15.70:0/2482373683 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8d7c0a2570 0x7f8d7c0a4a30 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.663+0000 7f8d92e8a640 1 --2- 172.21.15.70:0/2482373683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c100670 0x7f8d8c195650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.663+0000 7f8d92e8a640 1 -- 172.21.15.70:0/2482373683 >> 172.21.15.70:0/2482373683 conn(0x7f8d8c0fbbe0 msgr2=0x7f8d8c104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:55.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.665+0000 7f8d92e8a640 1 -- 172.21.15.70:0/2482373683 shutdown_connections 2024-04-01T01:48:55.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.666+0000 7f8d92e8a640 1 -- 172.21.15.70:0/2482373683 wait complete. 2024-04-01T01:48:55.688 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:48:55.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.871+0000 7f28cdb35640 1 -- 172.21.15.70:0/3318157460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c810de00 msgr2=0x7f28c81102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:55.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.871+0000 7f28cdb35640 1 --2- 172.21.15.70:0/3318157460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c810de00 0x7f28c81102c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f28c000bd70 tx=0x7f28c002dc80 comp rx=0 tx=0).stop 2024-04-01T01:48:55.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.871+0000 7f28cdb35640 1 -- 172.21.15.70:0/3318157460 shutdown_connections 2024-04-01T01:48:55.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.871+0000 7f28cdb35640 1 --2- 172.21.15.70:0/3318157460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c810de00 0x7f28c81102c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.871+0000 7f28cdb35640 1 --2- 172.21.15.70:0/3318157460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28c810b380 0x7f28c810d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.871+0000 7f28cdb35640 1 --2- 172.21.15.70:0/3318157460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c81089c0 0x7f28c810adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.872+0000 7f28cdb35640 1 -- 172.21.15.70:0/3318157460 >> 172.21.15.70:0/3318157460 conn(0x7f28c806e950 msgr2=0x7f28c8070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:55.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.872+0000 7f28cdb35640 1 -- 172.21.15.70:0/3318157460 shutdown_connections 2024-04-01T01:48:55.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.872+0000 7f28cdb35640 1 -- 172.21.15.70:0/3318157460 wait complete. 2024-04-01T01:48:55.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.873+0000 7f28cdb35640 1 Processor -- start 2024-04-01T01:48:55.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.873+0000 7f28cdb35640 1 -- start start 2024-04-01T01:48:55.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.873+0000 7f28cdb35640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c81089c0 0x7f28c81196e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.873+0000 7f28cdb35640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28c810b380 0x7f28c8119c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.873+0000 7f28cdb35640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c810de00 0x7f28c8117d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.873+0000 7f28cdb35640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f28c8112550 con 0x7f28c810b380 2024-04-01T01:48:55.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.873+0000 7f28cdb35640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f28c81123d0 con 0x7f28c810de00 2024-04-01T01:48:55.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.873+0000 7f28cdb35640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f28c81126d0 con 0x7f28c81089c0 2024-04-01T01:48:55.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.874+0000 7f28c7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c810de00 0x7f28c8117d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.874+0000 7f28c7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c810de00 0x7f28c8117d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57270/0 (socket says 172.21.15.70:57270) 2024-04-01T01:48:55.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.874+0000 7f28c7fff640 1 -- 172.21.15.70:0/2937576866 learned_addr learned my addr 172.21.15.70:0/2937576866 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:55.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.874+0000 7f28c77fe640 1 --2- 172.21.15.70:0/2937576866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c81089c0 0x7f28c81196e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.874+0000 7f28c6ffd640 1 --2- 172.21.15.70:0/2937576866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28c810b380 0x7f28c8119c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.874+0000 7f28c7fff640 1 -- 172.21.15.70:0/2937576866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c81089c0 msgr2=0x7f28c81196e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:55.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.874+0000 7f28c7fff640 1 --2- 172.21.15.70:0/2937576866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c81089c0 0x7f28c81196e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.874+0000 7f28c7fff640 1 -- 172.21.15.70:0/2937576866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28c810b380 msgr2=0x7f28c8119c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:55.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.874+0000 7f28c7fff640 1 --2- 172.21.15.70:0/2937576866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28c810b380 0x7f28c8119c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.874+0000 7f28c7fff640 1 -- 172.21.15.70:0/2937576866 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c8118400 con 0x7f28c810de00 2024-04-01T01:48:55.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.874+0000 7f28c6ffd640 1 --2- 172.21.15.70:0/2937576866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28c810b380 0x7f28c8119c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:48:55.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.875+0000 7f28c7fff640 1 --2- 172.21.15.70:0/2937576866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c810de00 0x7f28c8117d20 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f28c000bd40 tx=0x7f28c0036660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:55.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.879+0000 7f28c4ff9640 1 -- 172.21.15.70:0/2937576866 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c000a630 con 0x7f28c810de00 2024-04-01T01:48:55.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.879+0000 7f28cdb35640 1 -- 172.21.15.70:0/2937576866 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28c8118690 con 0x7f28c810de00 2024-04-01T01:48:55.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.879+0000 7f28cdb35640 1 -- 172.21.15.70:0/2937576866 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28c8121830 con 0x7f28c810de00 2024-04-01T01:48:55.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.879+0000 7f28c4ff9640 1 -- 172.21.15.70:0/2937576866 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f28c0036d60 con 0x7f28c810de00 2024-04-01T01:48:55.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.879+0000 7f28c4ff9640 1 -- 172.21.15.70:0/2937576866 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c0040ab0 con 0x7f28c810de00 2024-04-01T01:48:55.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.883+0000 7f28d07f1740 1 Processor -- start 2024-04-01T01:48:55.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.885+0000 7f28c4ff9640 1 -- 172.21.15.70:0/2937576866 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f28c003d070 con 0x7f28c810de00 2024-04-01T01:48:55.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.885+0000 7f28d07f1740 1 -- start start 2024-04-01T01:48:55.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.885+0000 7f28d07f1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab52e490 0x55edab52e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.885+0000 7f28d07f1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52ee80 0x55edab533980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.885+0000 7f28d07f1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab533ff0 0x55edab536480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.885+0000 7f28d07f1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55edab4931b0 con 0x55edab52ee80 2024-04-01T01:48:55.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.885+0000 7f28d07f1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55edab43bc00 con 0x55edab533ff0 2024-04-01T01:48:55.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.885+0000 7f28d07f1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55edab48f250 con 0x55edab52e490 2024-04-01T01:48:55.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.886+0000 7f28c77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52ee80 0x55edab533980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.886+0000 7f28c77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52ee80 0x55edab533980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49242/0 (socket says 172.21.15.70:49242) 2024-04-01T01:48:55.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.886+0000 7f28c77fe640 1 -- 172.21.15.70:0/3206390184 learned_addr learned my addr 172.21.15.70:0/3206390184 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:55.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.886+0000 7f28c4ff9640 1 --2- 172.21.15.70:0/2937576866 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28b00a26b0 0x7f28b00a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.887+0000 7f28c6ffd640 1 --2- 172.21.15.70:0/3206390184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab52e490 0x55edab52e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.887+0000 7f28c4ff9640 1 -- 172.21.15.70:0/2937576866 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f28c00f04f0 con 0x7f28c810de00 2024-04-01T01:48:55.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.887+0000 7f28c6ffd640 1 --2- 172.21.15.70:0/3206390184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab533ff0 0x55edab536480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.887+0000 7f28c77fe640 1 -- 172.21.15.70:0/3206390184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab52e490 msgr2=0x55edab52e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:55.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.887+0000 7f28c77fe640 1 --2- 172.21.15.70:0/3206390184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab52e490 0x55edab52e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.887+0000 7f28c77fe640 1 -- 172.21.15.70:0/3206390184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab533ff0 msgr2=0x55edab536480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:55.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.887+0000 7f28c77fe640 1 --2- 172.21.15.70:0/3206390184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab533ff0 0x55edab536480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.887+0000 7f28c77fe640 1 -- 172.21.15.70:0/3206390184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edab45b3d0 con 0x55edab52ee80 2024-04-01T01:48:55.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.888+0000 7f28c77fe640 1 --2- 172.21.15.70:0/3206390184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52ee80 0x55edab533980 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00a7f0 tx=0x7f28bc00ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997cdea9c40bc420 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:55.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.888+0000 7f28adffb640 1 -- 172.21.15.70:0/3206390184 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28bc019070 con 0x55edab52ee80 2024-04-01T01:48:55.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.888+0000 7f28adffb640 1 -- 172.21.15.70:0/3206390184 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f28bc0054d0 con 0x55edab52ee80 2024-04-01T01:48:55.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.888+0000 7f28adffb640 1 -- 172.21.15.70:0/3206390184 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28bc004ba0 con 0x55edab52ee80 2024-04-01T01:48:55.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.890+0000 7f28d07f1740 1 -- 172.21.15.70:0/3206390184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52ee80 msgr2=0x55edab533980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:55.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.890+0000 7f28d07f1740 1 --2- 172.21.15.70:0/3206390184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52ee80 0x55edab533980 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00a7f0 tx=0x7f28bc00ac00 comp rx=0 tx=0).stop 2024-04-01T01:48:55.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.890+0000 7f28c7fff640 1 --2- 172.21.15.70:0/2937576866 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28b00a26b0 0x7f28b00a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.891+0000 7f28c7fff640 1 --2- 172.21.15.70:0/2937576866 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28b00a26b0 0x7f28b00a4b70 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f28c0006010 tx=0x7f28c002e1a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:55.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.891+0000 7f28d07f1740 1 -- 172.21.15.70:0/3206390184 shutdown_connections 2024-04-01T01:48:55.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.891+0000 7f28d07f1740 1 --2- 172.21.15.70:0/3206390184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab533ff0 0x55edab536480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.891+0000 7f28d07f1740 1 --2- 172.21.15.70:0/3206390184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52ee80 0x55edab533980 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.891+0000 7f28d07f1740 1 --2- 172.21.15.70:0/3206390184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab52e490 0x55edab52e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.891+0000 7f28d07f1740 1 -- 172.21.15.70:0/3206390184 >> 172.21.15.70:0/3206390184 conn(0x55edab529a00 msgr2=0x55edab529e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:55.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.892+0000 7f28d07f1740 1 -- 172.21.15.70:0/3206390184 shutdown_connections 2024-04-01T01:48:55.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.892+0000 7f28d07f1740 1 -- 172.21.15.70:0/3206390184 wait complete. 2024-04-01T01:48:55.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.893+0000 7f28d07f1740 1 -- start start 2024-04-01T01:48:55.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.893+0000 7f28d07f1740 1 Processor -- start 2024-04-01T01:48:55.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.894+0000 7f28d07f1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52e490 0x55edab538850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.894+0000 7f28d07f1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab52ee80 0x55edab538d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.894+0000 7f28d07f1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 0x55edab53fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.894+0000 7f28d07f1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55edab536fa0 con 0x55edab52e490 2024-04-01T01:48:55.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.894+0000 7f28d07f1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55edab431470 con 0x55edab52ee80 2024-04-01T01:48:55.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.894+0000 7f28c6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 0x55edab53fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.894+0000 7f28c6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 0x55edab53fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:34240/0 (socket says 172.21.15.70:34240) 2024-04-01T01:48:55.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.894+0000 7f28c6ffd640 1 -- 172.21.15.70:0/1664460068 learned_addr learned my addr 172.21.15.70:0/1664460068 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:55.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.894+0000 7f28d07f1740 1 -- 172.21.15.70:0/1664460068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55edab537120 con 0x55edab533ff0 2024-04-01T01:48:55.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.895+0000 7f28d07f1740 10 client.0 _ll_register_callbacks cb 0x55edab44ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:55.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.895+0000 7f28c6ffd640 1 -- 172.21.15.70:0/1664460068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab52ee80 msgr2=0x55edab538d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:48:55.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.895+0000 7f28c6ffd640 1 --2- 172.21.15.70:0/1664460068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab52ee80 0x55edab538d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.895+0000 7f28c6ffd640 1 -- 172.21.15.70:0/1664460068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52e490 msgr2=0x55edab538850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:55.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.895+0000 7f28c6ffd640 1 --2- 172.21.15.70:0/1664460068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52e490 0x55edab538850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.895+0000 7f28c6ffd640 1 -- 172.21.15.70:0/1664460068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edab1a73f0 con 0x55edab533ff0 2024-04-01T01:48:55.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.896+0000 7f28c6ffd640 1 --2- 172.21.15.70:0/1664460068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 0x55edab53fdd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f28b800ab70 tx=0x7f28b8016ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8c6f60eb9709ea8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:55.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.896+0000 7f28a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:48:55.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.896+0000 7f28a6ffd640 1 -- 172.21.15.70:0/1664460068 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b801c020 con 0x55edab533ff0 2024-04-01T01:48:55.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.896+0000 7f28d07f1740 1 -- 172.21.15.70:0/1664460068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28bc0054d0 con 0x55edab533ff0 2024-04-01T01:48:55.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.897+0000 7f28a6ffd640 1 -- 172.21.15.70:0/1664460068 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f28b8014a10 con 0x55edab533ff0 2024-04-01T01:48:55.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.897+0000 7f28a6ffd640 1 -- 172.21.15.70:0/1664460068 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f28b8027050 con 0x55edab533ff0 2024-04-01T01:48:55.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.897+0000 7f28d07f1740 10 client.29716 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:55.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.897+0000 7f28d07f1740 1 -- 172.21.15.70:0/1664460068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55edab2717d0 con 0x55edab533ff0 2024-04-01T01:48:55.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.898+0000 7f28a6ffd640 1 -- 172.21.15.70:0/1664460068 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f28bc0054d0 con 0x55edab533ff0 2024-04-01T01:48:55.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.899+0000 7f28a6ffd640 1 -- 172.21.15.70:0/1664460068 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f28b8015040 con 0x55edab533ff0 2024-04-01T01:48:55.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.899+0000 7f28d07f1740 10 client.29716 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:55.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.899+0000 7f28d07f1740 10 client.29716 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:48:55.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.899+0000 7f28d07f1740 20 client.29716 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:55.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.899+0000 7f28d07f1740 1 --2- 172.21.15.70:0/1664460068 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55edab5443d0 0x55edab546840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.899+0000 7f28d07f1740 4 client.29716 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:55.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.899+0000 7f28d07f1740 1 -- 172.21.15.70:0/1664460068 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edab4a06f0 con 0x55edab5443d0 2024-04-01T01:48:55.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.900+0000 7f28c77fe640 1 --2- 172.21.15.70:0/1664460068 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55edab5443d0 0x55edab546840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.901+0000 7f28c77fe640 1 --2- 172.21.15.70:0/1664460068 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55edab5443d0 0x55edab546840 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b275b9d95424c99e server_cookie=4e2a470cba9545e3 in_seq=0 out_seq=0 2024-04-01T01:48:55.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.901+0000 7f28a6ffd640 10 client.29716 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:48:55.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.902+0000 7f28a6ffd640 1 -- 172.21.15.70:0/1664460068 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55edab4a06f0 con 0x55edab5443d0 2024-04-01T01:48:55.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.902+0000 7f28a6ffd640 10 client.29716 handle_command_reply: tid=0 2024-04-01T01:48:55.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.902+0000 7f28d07f1740 1 client.29716 shutdown 2024-04-01T01:48:55.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.903+0000 7f28d07f1740 1 -- 172.21.15.70:0/1664460068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 msgr2=0x55edab53fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:55.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.903+0000 7f28d07f1740 1 --2- 172.21.15.70:0/1664460068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 0x55edab53fdd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f28b800ab70 tx=0x7f28b8016ee0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.903+0000 7f28d07f1740 1 -- 172.21.15.70:0/1664460068 shutdown_connections 2024-04-01T01:48:55.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.903+0000 7f28d07f1740 1 --2- 172.21.15.70:0/1664460068 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55edab5443d0 0x55edab546840 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.903+0000 7f28d07f1740 1 --2- 172.21.15.70:0/1664460068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 0x55edab53fdd0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.903+0000 7f28d07f1740 1 --2- 172.21.15.70:0/1664460068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab52ee80 0x55edab538d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.903+0000 7f28d07f1740 1 --2- 172.21.15.70:0/1664460068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52e490 0x55edab538850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.903+0000 7f28d07f1740 1 -- 172.21.15.70:0/1664460068 >> 172.21.15.70:0/1664460068 conn(0x55edab529a00 msgr2=0x55edab528d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:55.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.904+0000 7f28a6ffd640 0 client.29716 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:48:55.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.904+0000 7f28d07f1740 1 -- 172.21.15.70:0/1664460068 shutdown_connections 2024-04-01T01:48:55.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.904+0000 7f28d07f1740 1 -- 172.21.15.70:0/1664460068 wait complete. 2024-04-01T01:48:55.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.906+0000 7f28d07f1740 20 client.29716 trim_cache size 0 max 16384 2024-04-01T01:48:55.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.942+0000 7f28d07f1740 1 Processor -- start 2024-04-01T01:48:55.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.942+0000 7f28d07f1740 1 -- start start 2024-04-01T01:48:55.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.943+0000 7f28d07f1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52e490 0x55edab538850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.943+0000 7f28d07f1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab52ee80 0x55edab538d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.943+0000 7f28d07f1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 0x55edab53fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.943+0000 7f28d07f1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f28b400fd10 con 0x55edab52e490 2024-04-01T01:48:55.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.943+0000 7f28d07f1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f28b4011850 con 0x55edab52ee80 2024-04-01T01:48:55.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.943+0000 7f28d07f1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f28b40103e0 con 0x55edab533ff0 2024-04-01T01:48:55.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.946+0000 7f28c6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 0x55edab53fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.946+0000 7f28c6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52e490 0x55edab538850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.946+0000 7f28c6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52e490 0x55edab538850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49246/0 (socket says 172.21.15.70:49246) 2024-04-01T01:48:55.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.946+0000 7f28c6ffd640 1 -- 172.21.15.70:0/2590512954 learned_addr learned my addr 172.21.15.70:0/2590512954 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:55.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.946+0000 7f28c6ffd640 1 -- 172.21.15.70:0/2590512954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab52ee80 msgr2=0x55edab538d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:48:55.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.946+0000 7f28c6ffd640 1 --2- 172.21.15.70:0/2590512954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab52ee80 0x55edab538d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.946+0000 7f28c6ffd640 1 -- 172.21.15.70:0/2590512954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52e490 msgr2=0x55edab538850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:55.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.946+0000 7f28c6ffd640 1 --2- 172.21.15.70:0/2590512954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52e490 0x55edab538850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.946+0000 7f28c6ffd640 1 -- 172.21.15.70:0/2590512954 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edab52fc90 con 0x55edab533ff0 2024-04-01T01:48:55.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.947+0000 7f28c6ffd640 1 --2- 172.21.15.70:0/2590512954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 0x55edab53fd90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f28b800a730 tx=0x7f28b8016ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b3fdc676e4355b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:55.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.948+0000 7f28ad7fa640 1 -- 172.21.15.70:0/2590512954 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b801c040 con 0x55edab533ff0 2024-04-01T01:48:55.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.948+0000 7f28ad7fa640 1 -- 172.21.15.70:0/2590512954 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f28b8017010 con 0x55edab533ff0 2024-04-01T01:48:55.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.948+0000 7f28ad7fa640 1 -- 172.21.15.70:0/2590512954 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b8017950 con 0x55edab533ff0 2024-04-01T01:48:55.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.949+0000 7f28d07f1740 1 -- 172.21.15.70:0/2590512954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 msgr2=0x55edab53fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:55.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.949+0000 7f28d07f1740 1 --2- 172.21.15.70:0/2590512954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 0x55edab53fd90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f28b800a730 tx=0x7f28b8016ee0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.950+0000 7f28d07f1740 1 -- 172.21.15.70:0/2590512954 shutdown_connections 2024-04-01T01:48:55.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.950+0000 7f28d07f1740 1 --2- 172.21.15.70:0/2590512954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 0x55edab53fd90 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.950+0000 7f28d07f1740 1 --2- 172.21.15.70:0/2590512954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab52ee80 0x55edab538d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.950+0000 7f28d07f1740 1 --2- 172.21.15.70:0/2590512954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52e490 0x55edab538850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.950+0000 7f28d07f1740 1 -- 172.21.15.70:0/2590512954 >> 172.21.15.70:0/2590512954 conn(0x55edab529a00 msgr2=0x55edab529de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:55.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.950+0000 7f28d07f1740 1 -- 172.21.15.70:0/2590512954 shutdown_connections 2024-04-01T01:48:55.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.950+0000 7f28d07f1740 1 -- 172.21.15.70:0/2590512954 wait complete. 2024-04-01T01:48:55.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.951+0000 7f28d07f1740 1 -- start start 2024-04-01T01:48:55.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.951+0000 7f28d07f1740 1 Processor -- start 2024-04-01T01:48:55.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.952+0000 7f28d07f1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52e490 0x55edab53d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.952+0000 7f28d07f1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab52ee80 0x55edab53d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.952+0000 7f28d07f1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 0x55edab53e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.952+0000 7f28c77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab52ee80 0x55edab53d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.952+0000 7f28c77fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab52ee80 0x55edab53d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57280/0 (socket says 172.21.15.70:57280) 2024-04-01T01:48:55.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.952+0000 7f28c6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 0x55edab53e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.952+0000 7f28c6ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 0x55edab53e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:34270/0 (socket says 172.21.15.70:34270) 2024-04-01T01:48:55.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.953+0000 7f28c77fe640 1 -- 172.21.15.70:0/4178828700 learned_addr learned my addr 172.21.15.70:0/4178828700 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:55.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.953+0000 7f28d07f1740 1 -- 172.21.15.70:0/4178828700 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f28b4011180 con 0x55edab52e490 2024-04-01T01:48:55.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.953+0000 7f28d07f1740 1 -- 172.21.15.70:0/4178828700 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f28b4010ab0 con 0x55edab52ee80 2024-04-01T01:48:55.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.953+0000 7f28d07f1740 1 -- 172.21.15.70:0/4178828700 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f28b400dcf0 con 0x55edab533ff0 2024-04-01T01:48:55.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.953+0000 7f28d07f1740 10 client.0 _ll_register_callbacks cb 0x7f28b8016680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:55.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.953+0000 7f28c77fe640 1 -- 172.21.15.70:0/4178828700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 msgr2=0x55edab53e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:55.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.953+0000 7f28c77fe640 1 --2- 172.21.15.70:0/4178828700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 0x55edab53e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.953+0000 7f28c77fe640 1 -- 172.21.15.70:0/4178828700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52e490 msgr2=0x55edab53d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:55.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.953+0000 7f28c77fe640 1 --2- 172.21.15.70:0/4178828700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52e490 0x55edab53d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.953+0000 7f28c77fe640 1 -- 172.21.15.70:0/4178828700 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edab53eec0 con 0x55edab52ee80 2024-04-01T01:48:55.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.954+0000 7f28d07f1740 1 -- 172.21.15.70:0/4178828700 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28b8017010 con 0x55edab52ee80 2024-04-01T01:48:55.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.954+0000 7f28c77fe640 1 --2- 172.21.15.70:0/4178828700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab52ee80 0x55edab53d8c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00a710 tx=0x7f28bc005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3847d3cb064ee51 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:55.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.954+0000 7f28a6ffd640 10 client.29993 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:55.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.954+0000 7f28a6ffd640 1 -- 172.21.15.70:0/4178828700 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28bc019040 con 0x55edab52ee80 2024-04-01T01:48:55.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.954+0000 7f28a6ffd640 1 -- 172.21.15.70:0/4178828700 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f28bc029d20 con 0x55edab52ee80 2024-04-01T01:48:55.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.954+0000 7f28a6ffd640 1 -- 172.21.15.70:0/4178828700 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f28bc029420 con 0x55edab52ee80 2024-04-01T01:48:55.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.954+0000 7f28d07f1740 10 client.29993 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:55.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.954+0000 7f28d07f1740 1 -- 172.21.15.70:0/4178828700 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55edab5aa780 con 0x55edab52ee80 2024-04-01T01:48:55.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.955+0000 7f28a6ffd640 1 -- 172.21.15.70:0/4178828700 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f28bc0295c0 con 0x55edab52ee80 2024-04-01T01:48:55.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.955+0000 7f28d07f1740 10 client.29993 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:55.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.955+0000 7f28d07f1740 10 client.29993 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:48:55.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.955+0000 7f28d07f1740 20 client.29993 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:55.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.955+0000 7f28d07f1740 1 --2- 172.21.15.70:0/4178828700 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55edab5aaaf0 0x55edab5acee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:55.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.955+0000 7f28d07f1740 4 client.29993 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:48:55.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.955+0000 7f28d07f1740 1 -- 172.21.15.70:0/4178828700 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55edab49aa40 con 0x55edab5aaaf0 2024-04-01T01:48:55.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.955+0000 7f28c6ffd640 1 --2- 172.21.15.70:0/4178828700 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55edab5aaaf0 0x55edab5acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:55.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.956+0000 7f28a6ffd640 10 client.29993 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:48:55.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.956+0000 7f28c6ffd640 1 --2- 172.21.15.70:0/4178828700 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55edab5aaaf0 0x55edab5acee0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=deb6234e94bb16c6 server_cookie=20015893f0b7b943 in_seq=0 out_seq=0 2024-04-01T01:48:55.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.956+0000 7f28a6ffd640 1 -- 172.21.15.70:0/4178828700 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55edab49aa40 con 0x55edab5aaaf0 2024-04-01T01:48:55.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.956+0000 7f28a6ffd640 10 client.29993 handle_command_reply: tid=0 2024-04-01T01:48:55.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.956+0000 7f28d07f1740 1 client.29993 shutdown 2024-04-01T01:48:55.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.957+0000 7f28d07f1740 1 -- 172.21.15.70:0/4178828700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab52ee80 msgr2=0x55edab53d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:55.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.957+0000 7f28d07f1740 1 --2- 172.21.15.70:0/4178828700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab52ee80 0x55edab53d8c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00a710 tx=0x7f28bc005270 comp rx=0 tx=0).stop 2024-04-01T01:48:55.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.957+0000 7f28d07f1740 1 -- 172.21.15.70:0/4178828700 shutdown_connections 2024-04-01T01:48:55.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.957+0000 7f28d07f1740 1 --2- 172.21.15.70:0/4178828700 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55edab5aaaf0 0x55edab5acee0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.957+0000 7f28d07f1740 1 --2- 172.21.15.70:0/4178828700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55edab533ff0 0x55edab53e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.957+0000 7f28a6ffd640 0 client.29993 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:48:55.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.957+0000 7f28d07f1740 1 --2- 172.21.15.70:0/4178828700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55edab52ee80 0x55edab53d8c0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.957+0000 7f28d07f1740 1 --2- 172.21.15.70:0/4178828700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55edab52e490 0x55edab53d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.957+0000 7f28d07f1740 1 -- 172.21.15.70:0/4178828700 >> 172.21.15.70:0/4178828700 conn(0x55edab529a00 msgr2=0x55edab538850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:55.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.958+0000 7f28d07f1740 1 -- 172.21.15.70:0/4178828700 shutdown_connections 2024-04-01T01:48:55.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.958+0000 7f28d07f1740 1 -- 172.21.15.70:0/4178828700 wait complete. 2024-04-01T01:48:55.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.958+0000 7f28d07f1740 20 client.29993 trim_cache size 0 max 16384 2024-04-01T01:48:55.967 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:48:55.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.963+0000 7f28cdb35640 1 -- 172.21.15.70:0/2937576866 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28b00a26b0 msgr2=0x7f28b00a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:55.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.963+0000 7f28cdb35640 1 --2- 172.21.15.70:0/2937576866 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28b00a26b0 0x7f28b00a4b70 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f28c0006010 tx=0x7f28c002e1a0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.963+0000 7f28cdb35640 1 -- 172.21.15.70:0/2937576866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c810de00 msgr2=0x7f28c8117d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:55.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.963+0000 7f28cdb35640 1 --2- 172.21.15.70:0/2937576866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c810de00 0x7f28c8117d20 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f28c000bd40 tx=0x7f28c0036660 comp rx=0 tx=0).stop 2024-04-01T01:48:55.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.963+0000 7f28cdb35640 1 -- 172.21.15.70:0/2937576866 shutdown_connections 2024-04-01T01:48:55.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.963+0000 7f28cdb35640 1 --2- 172.21.15.70:0/2937576866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c810de00 0x7f28c8117d20 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.963+0000 7f28cdb35640 1 --2- 172.21.15.70:0/2937576866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f28c810b380 0x7f28c8119c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.963+0000 7f28cdb35640 1 --2- 172.21.15.70:0/2937576866 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f28b00a26b0 0x7f28b00a4b70 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.963+0000 7f28cdb35640 1 --2- 172.21.15.70:0/2937576866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c81089c0 0x7f28c81196e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:55.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.963+0000 7f28cdb35640 1 -- 172.21.15.70:0/2937576866 >> 172.21.15.70:0/2937576866 conn(0x7f28c806e950 msgr2=0x7f28c81093e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:55.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.964+0000 7f28cdb35640 1 -- 172.21.15.70:0/2937576866 shutdown_connections 2024-04-01T01:48:55.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:55.964+0000 7f28cdb35640 1 -- 172.21.15.70:0/2937576866 wait complete. 2024-04-01T01:48:55.991 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:48:56.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.198+0000 7f4971101640 1 -- 172.21.15.70:0/3491847 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496c1020e0 msgr2=0x7f496c108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.198+0000 7f4971101640 1 --2- 172.21.15.70:0/3491847 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496c1020e0 0x7f496c108970 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f4964009900 tx=0x7f496402bc30 comp rx=0 tx=0).stop 2024-04-01T01:48:56.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.199+0000 7f4971101640 1 -- 172.21.15.70:0/3491847 shutdown_connections 2024-04-01T01:48:56.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.200+0000 7f4971101640 1 --2- 172.21.15.70:0/3491847 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496c1020e0 0x7f496c108970 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.200+0000 7f4971101640 1 --2- 172.21.15.70:0/3491847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f496c101720 0x7f496c101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.200+0000 7f4971101640 1 --2- 172.21.15.70:0/3491847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f496c100530 0x7f496c100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.200+0000 7f4971101640 1 -- 172.21.15.70:0/3491847 >> 172.21.15.70:0/3491847 conn(0x7f496c0fbac0 msgr2=0x7f496c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.200+0000 7f4971101640 1 -- 172.21.15.70:0/3491847 shutdown_connections 2024-04-01T01:48:56.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.202+0000 7f4971101640 1 -- 172.21.15.70:0/3491847 wait complete. 2024-04-01T01:48:56.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.203+0000 7f4971101640 1 Processor -- start 2024-04-01T01:48:56.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.205+0000 7f4971101640 1 -- start start 2024-04-01T01:48:56.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.205+0000 7f4971101640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496c100530 0x7f496c111010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.205+0000 7f4971101640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f496c101720 0x7f496c111550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.205+0000 7f4971101640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f496c1020e0 0x7f496c10f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.205+0000 7f4971101640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f496c109ec0 con 0x7f496c101720 2024-04-01T01:48:56.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.205+0000 7f4971101640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f496c109d40 con 0x7f496c100530 2024-04-01T01:48:56.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.206+0000 7f4971101640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f496c10a040 con 0x7f496c1020e0 2024-04-01T01:48:56.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.206+0000 7f4970900640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f496c1020e0 0x7f496c10f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.206+0000 7f496bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496c100530 0x7f496c111010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.206+0000 7f496bfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496c100530 0x7f496c111010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57294/0 (socket says 172.21.15.70:57294) 2024-04-01T01:48:56.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.206+0000 7f496bfff640 1 -- 172.21.15.70:0/3766003180 learned_addr learned my addr 172.21.15.70:0/3766003180 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:56.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.207+0000 7f496bfff640 1 -- 172.21.15.70:0/3766003180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f496c1020e0 msgr2=0x7f496c10f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.207+0000 7f496bfff640 1 --2- 172.21.15.70:0/3766003180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f496c1020e0 0x7f496c10f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.207+0000 7f496bfff640 1 -- 172.21.15.70:0/3766003180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f496c101720 msgr2=0x7f496c111550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.207+0000 7f496bfff640 1 --2- 172.21.15.70:0/3766003180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f496c101720 0x7f496c111550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.207+0000 7f496bfff640 1 -- 172.21.15.70:0/3766003180 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496c10fcf0 con 0x7f496c100530 2024-04-01T01:48:56.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.207+0000 7f496bfff640 1 --2- 172.21.15.70:0/3766003180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496c100530 0x7f496c111010 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f495c00b9c0 tx=0x7f495c00be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.207+0000 7f4969ffb640 1 -- 172.21.15.70:0/3766003180 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c019070 con 0x7f496c100530 2024-04-01T01:48:56.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.207+0000 7f4969ffb640 1 -- 172.21.15.70:0/3766003180 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f495c00cba0 con 0x7f496c100530 2024-04-01T01:48:56.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.207+0000 7f4969ffb640 1 -- 172.21.15.70:0/3766003180 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c0125a0 con 0x7f496c100530 2024-04-01T01:48:56.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.208+0000 7f4971101640 1 -- 172.21.15.70:0/3766003180 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f496c10ffe0 con 0x7f496c100530 2024-04-01T01:48:56.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.209+0000 7f4971101640 1 -- 172.21.15.70:0/3766003180 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f496c110480 con 0x7f496c100530 2024-04-01T01:48:56.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.212+0000 7f4969ffb640 1 -- 172.21.15.70:0/3766003180 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f495c01c070 con 0x7f496c100530 2024-04-01T01:48:56.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.212+0000 7f4975017740 1 Processor -- start 2024-04-01T01:48:56.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.213+0000 7f4975017740 1 -- start start 2024-04-01T01:48:56.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.213+0000 7f4975017740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d6736a0 0x55f23d673ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.213+0000 7f4975017740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d674090 0x55f23d678b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.213+0000 7f4975017740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 0x55f23d67b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.213+0000 7f4975017740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f23d5d81b0 con 0x55f23d674090 2024-04-01T01:48:56.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.213+0000 7f4975017740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f23d580c00 con 0x55f23d679200 2024-04-01T01:48:56.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.213+0000 7f4975017740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f23d5d4250 con 0x55f23d6736a0 2024-04-01T01:48:56.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.213+0000 7f4969ffb640 1 --2- 172.21.15.70:0/3766003180 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f49580a2570 0x7f49580a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.214+0000 7f4969ffb640 1 -- 172.21.15.70:0/3766003180 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f495c0ce1e0 con 0x7f496c100530 2024-04-01T01:48:56.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.214+0000 7f496bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d674090 0x55f23d678b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.214+0000 7f496bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d674090 0x55f23d678b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49274/0 (socket says 172.21.15.70:49274) 2024-04-01T01:48:56.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.214+0000 7f496bfff640 1 -- 172.21.15.70:0/2512201259 learned_addr learned my addr 172.21.15.70:0/2512201259 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:56.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.214+0000 7f496bfff640 1 -- 172.21.15.70:0/2512201259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d6736a0 msgr2=0x55f23d673ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:48:56.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.214+0000 7f496bfff640 1 --2- 172.21.15.70:0/2512201259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d6736a0 0x55f23d673ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.214+0000 7f496bfff640 1 -- 172.21.15.70:0/2512201259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 msgr2=0x55f23d67b690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:48:56.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.214+0000 7f496bfff640 1 --2- 172.21.15.70:0/2512201259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 0x55f23d67b690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.214+0000 7f496bfff640 1 -- 172.21.15.70:0/2512201259 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f23d5a03d0 con 0x55f23d674090 2024-04-01T01:48:56.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.215+0000 7f496bfff640 1 --2- 172.21.15.70:0/2512201259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d674090 0x55f23d678b90 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f495c09a570 tx=0x7f495c09a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17a5ff7a47b7e2d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.216+0000 7f49627fc640 1 -- 172.21.15.70:0/2512201259 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c09a870 con 0x55f23d674090 2024-04-01T01:48:56.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.216+0000 7f49627fc640 1 -- 172.21.15.70:0/2512201259 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f495c09e9c0 con 0x55f23d674090 2024-04-01T01:48:56.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.217+0000 7f4970900640 1 --2- 172.21.15.70:0/3766003180 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f49580a2570 0x7f49580a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.217+0000 7f4975017740 1 -- 172.21.15.70:0/2512201259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d674090 msgr2=0x55f23d678b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.217+0000 7f4975017740 1 --2- 172.21.15.70:0/2512201259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d674090 0x55f23d678b90 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f495c09a570 tx=0x7f495c09a1d0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.219+0000 7f4975017740 1 -- 172.21.15.70:0/2512201259 shutdown_connections 2024-04-01T01:48:56.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.219+0000 7f4975017740 1 --2- 172.21.15.70:0/2512201259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 0x55f23d67b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.219+0000 7f4975017740 1 --2- 172.21.15.70:0/2512201259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d674090 0x55f23d678b90 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.219+0000 7f4975017740 1 --2- 172.21.15.70:0/2512201259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d6736a0 0x55f23d673ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.219+0000 7f4975017740 1 -- 172.21.15.70:0/2512201259 >> 172.21.15.70:0/2512201259 conn(0x55f23d66ec10 msgr2=0x55f23d66f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.220+0000 7f4975017740 1 -- 172.21.15.70:0/2512201259 shutdown_connections 2024-04-01T01:48:56.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.220+0000 7f4975017740 1 -- 172.21.15.70:0/2512201259 wait complete. 2024-04-01T01:48:56.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.220+0000 7f4970900640 1 --2- 172.21.15.70:0/3766003180 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f49580a2570 0x7f49580a4a30 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4964006010 tx=0x7f496402c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.220+0000 7f4975017740 1 -- start start 2024-04-01T01:48:56.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.220+0000 7f4975017740 1 Processor -- start 2024-04-01T01:48:56.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.220+0000 7f4975017740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 0x55f23d67dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.220+0000 7f4975017740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 0x55f23d67e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.221+0000 7f4975017740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 0x55f23d685140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.221+0000 7f4975017740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f23d67c1a0 con 0x55f23d6736a0 2024-04-01T01:48:56.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.221+0000 7f4975017740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f23d576470 con 0x55f23d679200 2024-04-01T01:48:56.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.221+0000 7f4975017740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f23d67c320 con 0x55f23d674090 2024-04-01T01:48:56.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.221+0000 7f4975017740 10 client.0 _ll_register_callbacks cb 0x55f23d593cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:56.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.221+0000 7f49637fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 0x55f23d67e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.221+0000 7f4970900640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 0x55f23d685140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.221+0000 7f49637fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 0x55f23d67e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:34286/0 (socket says 172.21.15.70:34286) 2024-04-01T01:48:56.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.221+0000 7f49637fe640 1 -- 172.21.15.70:0/3036122206 learned_addr learned my addr 172.21.15.70:0/3036122206 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:56.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.221+0000 7f496bfff640 1 --2- 172.21.15.70:0/3036122206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 0x55f23d67dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.221+0000 7f4970900640 1 -- 172.21.15.70:0/3036122206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 msgr2=0x55f23d67e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.221+0000 7f4970900640 1 --2- 172.21.15.70:0/3036122206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 0x55f23d67e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.221+0000 7f4970900640 1 -- 172.21.15.70:0/3036122206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 msgr2=0x55f23d67dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.221+0000 7f4970900640 1 --2- 172.21.15.70:0/3036122206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 0x55f23d67dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.221+0000 7f4970900640 1 -- 172.21.15.70:0/3036122206 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f23d2ec3f0 con 0x55f23d679200 2024-04-01T01:48:56.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.222+0000 7f4970900640 1 --2- 172.21.15.70:0/3036122206 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 0x55f23d685140 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4964036040 tx=0x7f4964009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b7c31281f0e5d77 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.222+0000 7f494b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:56.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.222+0000 7f494b7fe640 1 -- 172.21.15.70:0/3036122206 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496404f020 con 0x55f23d679200 2024-04-01T01:48:56.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.223+0000 7f4975017740 1 -- 172.21.15.70:0/3036122206 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f495c09e9c0 con 0x55f23d679200 2024-04-01T01:48:56.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.223+0000 7f494b7fe640 1 -- 172.21.15.70:0/3036122206 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f49640048d0 con 0x55f23d679200 2024-04-01T01:48:56.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.228+0000 7f494b7fe640 1 -- 172.21.15.70:0/3036122206 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f496405a050 con 0x55f23d679200 2024-04-01T01:48:56.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.228+0000 7f4975017740 10 client.30002 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:56.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.228+0000 7f4975017740 1 -- 172.21.15.70:0/3036122206 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f23d3b67d0 con 0x55f23d679200 2024-04-01T01:48:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.230+0000 7f494b7fe640 1 -- 172.21.15.70:0/3036122206 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f495c09e9c0 con 0x55f23d679200 2024-04-01T01:48:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.231+0000 7f494b7fe640 1 -- 172.21.15.70:0/3036122206 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f4964041460 con 0x55f23d679200 2024-04-01T01:48:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.231+0000 7f4975017740 10 client.30002 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.231+0000 7f4975017740 10 client.30002 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:48:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.231+0000 7f4975017740 20 client.30002 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.231+0000 7f4975017740 1 --2- 172.21.15.70:0/3036122206 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55f23d689b10 0x55f23d68bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.231+0000 7f4975017740 4 client.30002 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.231+0000 7f4975017740 1 -- 172.21.15.70:0/3036122206 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f23d5e56f0 con 0x55f23d689b10 2024-04-01T01:48:56.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.231+0000 7f496bfff640 1 --2- 172.21.15.70:0/3036122206 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55f23d689b10 0x55f23d68bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.233+0000 7f496bfff640 1 --2- 172.21.15.70:0/3036122206 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55f23d689b10 0x55f23d68bf80 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b4dea2e235b888b server_cookie=34d0ae628d8f8972 in_seq=0 out_seq=0 2024-04-01T01:48:56.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.233+0000 7f494b7fe640 10 client.30002 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:48:56.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.234+0000 7f494b7fe640 1 -- 172.21.15.70:0/3036122206 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f23d5e56f0 con 0x55f23d689b10 2024-04-01T01:48:56.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.234+0000 7f494b7fe640 10 client.30002 handle_command_reply: tid=0 2024-04-01T01:48:56.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.235+0000 7f4975017740 1 client.30002 shutdown 2024-04-01T01:48:56.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.235+0000 7f4975017740 1 -- 172.21.15.70:0/3036122206 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 msgr2=0x55f23d685140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.235+0000 7f4975017740 1 --2- 172.21.15.70:0/3036122206 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 0x55f23d685140 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4964036040 tx=0x7f4964009260 comp rx=0 tx=0).stop 2024-04-01T01:48:56.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.236+0000 7f4975017740 1 -- 172.21.15.70:0/3036122206 shutdown_connections 2024-04-01T01:48:56.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.236+0000 7f4975017740 1 --2- 172.21.15.70:0/3036122206 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55f23d689b10 0x55f23d68bf80 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.236+0000 7f4975017740 1 --2- 172.21.15.70:0/3036122206 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 0x55f23d685140 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.236+0000 7f4975017740 1 --2- 172.21.15.70:0/3036122206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 0x55f23d67e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.236+0000 7f4975017740 1 --2- 172.21.15.70:0/3036122206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 0x55f23d67dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.236+0000 7f4975017740 1 -- 172.21.15.70:0/3036122206 >> 172.21.15.70:0/3036122206 conn(0x55f23d66ec10 msgr2=0x55f23d66df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.236+0000 7f494b7fe640 0 client.30002 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:48:56.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.236+0000 7f4975017740 1 -- 172.21.15.70:0/3036122206 shutdown_connections 2024-04-01T01:48:56.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.236+0000 7f4975017740 1 -- 172.21.15.70:0/3036122206 wait complete. 2024-04-01T01:48:56.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.236+0000 7f4975017740 20 client.30002 trim_cache size 0 max 16384 2024-04-01T01:48:56.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.266+0000 7f4975017740 1 Processor -- start 2024-04-01T01:48:56.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.267+0000 7f4975017740 1 -- start start 2024-04-01T01:48:56.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.267+0000 7f4975017740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 0x55f23d675590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.267+0000 7f4975017740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 0x55f23d685100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.267+0000 7f4975017740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 0x55f23d675ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.267+0000 7f4975017740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f496c1b9010 con 0x55f23d6736a0 2024-04-01T01:48:56.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.267+0000 7f4975017740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f496c1bab50 con 0x55f23d679200 2024-04-01T01:48:56.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.267+0000 7f4975017740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f496c1b96e0 con 0x55f23d674090 2024-04-01T01:48:56.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.267+0000 7f496bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 0x55f23d675590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.267+0000 7f496bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 0x55f23d675590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49278/0 (socket says 172.21.15.70:49278) 2024-04-01T01:48:56.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.267+0000 7f496bfff640 1 -- 172.21.15.70:0/2479255083 learned_addr learned my addr 172.21.15.70:0/2479255083 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:56.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.267+0000 7f49637fe640 1 --2- 172.21.15.70:0/2479255083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 0x55f23d685100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.268+0000 7f496bfff640 1 -- 172.21.15.70:0/2479255083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 msgr2=0x55f23d685100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.268+0000 7f496bfff640 1 --2- 172.21.15.70:0/2479255083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 0x55f23d685100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.268+0000 7f496bfff640 1 -- 172.21.15.70:0/2479255083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 msgr2=0x55f23d675ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.268+0000 7f496bfff640 1 --2- 172.21.15.70:0/2479255083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 0x55f23d675ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.268+0000 7f496bfff640 1 -- 172.21.15.70:0/2479255083 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f23d6732d0 con 0x55f23d6736a0 2024-04-01T01:48:56.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.268+0000 7f496bfff640 1 --2- 172.21.15.70:0/2479255083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 0x55f23d675590 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f495c097240 tx=0x7f495c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6cf103a8d2c0d3d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.269+0000 7f4961ffb640 1 -- 172.21.15.70:0/2479255083 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c09e9c0 con 0x55f23d6736a0 2024-04-01T01:48:56.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.269+0000 7f4961ffb640 1 -- 172.21.15.70:0/2479255083 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f495c09a870 con 0x55f23d6736a0 2024-04-01T01:48:56.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.269+0000 7f4961ffb640 1 -- 172.21.15.70:0/2479255083 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c0a3050 con 0x55f23d6736a0 2024-04-01T01:48:56.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.270+0000 7f4975017740 1 -- 172.21.15.70:0/2479255083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 msgr2=0x55f23d675590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.270+0000 7f4975017740 1 --2- 172.21.15.70:0/2479255083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 0x55f23d675590 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f495c097240 tx=0x7f495c004a40 comp rx=0 tx=0).stop 2024-04-01T01:48:56.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.274+0000 7f4975017740 1 -- 172.21.15.70:0/2479255083 shutdown_connections 2024-04-01T01:48:56.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.274+0000 7f4975017740 1 --2- 172.21.15.70:0/2479255083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 0x55f23d675ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.274+0000 7f4975017740 1 --2- 172.21.15.70:0/2479255083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 0x55f23d685100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.274+0000 7f4975017740 1 --2- 172.21.15.70:0/2479255083 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 0x55f23d675590 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.274+0000 7f4975017740 1 -- 172.21.15.70:0/2479255083 >> 172.21.15.70:0/2479255083 conn(0x55f23d66ec10 msgr2=0x55f23d67dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.277+0000 7f4975017740 1 -- 172.21.15.70:0/2479255083 shutdown_connections 2024-04-01T01:48:56.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.279+0000 7f4975017740 1 -- 172.21.15.70:0/2479255083 wait complete. 2024-04-01T01:48:56.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.279+0000 7f4975017740 1 -- start start 2024-04-01T01:48:56.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.279+0000 7f4975017740 1 Processor -- start 2024-04-01T01:48:56.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.279+0000 7f4975017740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 0x55f23d682b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.279+0000 7f4975017740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 0x55f23d6830d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.279+0000 7f4975017740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 0x55f23d684090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.279+0000 7f4975017740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f496c1ba480 con 0x55f23d6736a0 2024-04-01T01:48:56.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.279+0000 7f4975017740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f496c1b9db0 con 0x55f23d679200 2024-04-01T01:48:56.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.279+0000 7f4975017740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f496c1b6ff0 con 0x55f23d674090 2024-04-01T01:48:56.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.280+0000 7f4975017740 10 client.0 _ll_register_callbacks cb 0x7f49640044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:56.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.280+0000 7f496bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 0x55f23d682b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.280+0000 7f496bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 0x55f23d682b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49284/0 (socket says 172.21.15.70:49284) 2024-04-01T01:48:56.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.280+0000 7f496bfff640 1 -- 172.21.15.70:0/3324812865 learned_addr learned my addr 172.21.15.70:0/3324812865 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:56.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.280+0000 7f4970900640 1 --2- 172.21.15.70:0/3324812865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 0x55f23d684090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.280+0000 7f49637fe640 1 --2- 172.21.15.70:0/3324812865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 0x55f23d6830d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.280+0000 7f496bfff640 1 -- 172.21.15.70:0/3324812865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 msgr2=0x55f23d6830d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.280+0000 7f496bfff640 1 --2- 172.21.15.70:0/3324812865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 0x55f23d6830d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.280+0000 7f496bfff640 1 -- 172.21.15.70:0/3324812865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 msgr2=0x55f23d684090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.280+0000 7f496bfff640 1 --2- 172.21.15.70:0/3324812865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 0x55f23d684090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.280+0000 7f496bfff640 1 -- 172.21.15.70:0/3324812865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f23d684680 con 0x55f23d6736a0 2024-04-01T01:48:56.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.280+0000 7f49637fe640 1 --2- 172.21.15.70:0/3324812865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 0x55f23d6830d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:48:56.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.281+0000 7f496bfff640 1 --2- 172.21.15.70:0/3324812865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 0x55f23d682b90 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f495c0850b0 tx=0x7f495c014a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2820910643a0aa31 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.281+0000 7f494b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:48:56.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.284+0000 7f494b7fe640 1 -- 172.21.15.70:0/3324812865 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f495c0a3280 con 0x55f23d6736a0 2024-04-01T01:48:56.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.284+0000 7f4975017740 1 -- 172.21.15.70:0/3324812865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f495c09a870 con 0x55f23d6736a0 2024-04-01T01:48:56.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.284+0000 7f494b7fe640 1 -- 172.21.15.70:0/3324812865 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f495c0a2df0 con 0x55f23d6736a0 2024-04-01T01:48:56.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.285+0000 7f494b7fe640 1 -- 172.21.15.70:0/3324812865 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f495c0a2ff0 con 0x55f23d6736a0 2024-04-01T01:48:56.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.287+0000 7f4975017740 10 client.19905 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:56.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.287+0000 7f4975017740 1 -- 172.21.15.70:0/3324812865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f23d6f01b0 con 0x55f23d6736a0 2024-04-01T01:48:56.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.288+0000 7f494b7fe640 1 -- 172.21.15.70:0/3324812865 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f495c09a870 con 0x55f23d6736a0 2024-04-01T01:48:56.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.288+0000 7f494b7fe640 1 -- 172.21.15.70:0/3324812865 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f495c0965c0 con 0x55f23d6736a0 2024-04-01T01:48:56.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.288+0000 7f4975017740 10 client.19905 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:56.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.288+0000 7f4975017740 10 client.19905 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:48:56.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.288+0000 7f4975017740 20 client.19905 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:56.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.288+0000 7f4975017740 1 --2- 172.21.15.70:0/3324812865 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55f23d6f0b90 0x55f23d6f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.288+0000 7f4975017740 4 client.19905 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:48:56.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.288+0000 7f4975017740 1 -- 172.21.15.70:0/3324812865 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f23d5dfa40 con 0x55f23d6f0b90 2024-04-01T01:48:56.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.288+0000 7f49637fe640 1 --2- 172.21.15.70:0/3324812865 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55f23d6f0b90 0x55f23d6f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.289+0000 7f49637fe640 1 --2- 172.21.15.70:0/3324812865 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55f23d6f0b90 0x55f23d6f2fb0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e09e09a0cc9fa26e server_cookie=1b1778ccd79a2fb3 in_seq=0 out_seq=0 2024-04-01T01:48:56.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.289+0000 7f494b7fe640 10 client.19905 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:48:56.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.291+0000 7f494b7fe640 1 -- 172.21.15.70:0/3324812865 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f23d5dfa40 con 0x55f23d6f0b90 2024-04-01T01:48:56.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.292+0000 7f494b7fe640 10 client.19905 handle_command_reply: tid=0 2024-04-01T01:48:56.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.292+0000 7f4975017740 1 client.19905 shutdown 2024-04-01T01:48:56.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.292+0000 7f4975017740 1 -- 172.21.15.70:0/3324812865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 msgr2=0x55f23d682b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.292+0000 7f4975017740 1 --2- 172.21.15.70:0/3324812865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 0x55f23d682b90 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f495c0850b0 tx=0x7f495c014a90 comp rx=0 tx=0).stop 2024-04-01T01:48:56.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.292+0000 7f4975017740 1 -- 172.21.15.70:0/3324812865 shutdown_connections 2024-04-01T01:48:56.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.292+0000 7f4975017740 1 --2- 172.21.15.70:0/3324812865 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x55f23d6f0b90 0x55f23d6f2fb0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.292+0000 7f4975017740 1 --2- 172.21.15.70:0/3324812865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f23d679200 0x55f23d684090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.292+0000 7f4975017740 1 --2- 172.21.15.70:0/3324812865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f23d674090 0x55f23d6830d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.292+0000 7f4975017740 1 --2- 172.21.15.70:0/3324812865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f23d6736a0 0x55f23d682b90 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.293+0000 7f4975017740 1 -- 172.21.15.70:0/3324812865 >> 172.21.15.70:0/3324812865 conn(0x55f23d66ec10 msgr2=0x55f23d67db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.293+0000 7f494b7fe640 0 client.19905 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:48:56.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.293+0000 7f4975017740 1 -- 172.21.15.70:0/3324812865 shutdown_connections 2024-04-01T01:48:56.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.293+0000 7f4975017740 1 -- 172.21.15.70:0/3324812865 wait complete. 2024-04-01T01:48:56.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.293+0000 7f4975017740 20 client.19905 trim_cache size 0 max 16384 2024-04-01T01:48:56.293 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:48:56.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.294+0000 7f4971101640 1 -- 172.21.15.70:0/3766003180 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f49580a2570 msgr2=0x7f49580a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.294+0000 7f4971101640 1 --2- 172.21.15.70:0/3766003180 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f49580a2570 0x7f49580a4a30 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4964006010 tx=0x7f496402c7e0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.294+0000 7f4971101640 1 -- 172.21.15.70:0/3766003180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496c100530 msgr2=0x7f496c111010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.294+0000 7f4971101640 1 --2- 172.21.15.70:0/3766003180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496c100530 0x7f496c111010 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f495c00b9c0 tx=0x7f495c00be80 comp rx=0 tx=0).stop 2024-04-01T01:48:56.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.294+0000 7f4971101640 1 -- 172.21.15.70:0/3766003180 shutdown_connections 2024-04-01T01:48:56.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.295+0000 7f4971101640 1 --2- 172.21.15.70:0/3766003180 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f49580a2570 0x7f49580a4a30 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.295+0000 7f4971101640 1 --2- 172.21.15.70:0/3766003180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f496c1020e0 0x7f496c10f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.295+0000 7f4971101640 1 --2- 172.21.15.70:0/3766003180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f496c101720 0x7f496c111550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.295+0000 7f4971101640 1 --2- 172.21.15.70:0/3766003180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f496c100530 0x7f496c111010 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.295+0000 7f4971101640 1 -- 172.21.15.70:0/3766003180 >> 172.21.15.70:0/3766003180 conn(0x7f496c0fbac0 msgr2=0x7f496c104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.295+0000 7f4971101640 1 -- 172.21.15.70:0/3766003180 shutdown_connections 2024-04-01T01:48:56.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.295+0000 7f4971101640 1 -- 172.21.15.70:0/3766003180 wait complete. 2024-04-01T01:48:56.318 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:48:56.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.525+0000 7f16b4b3e640 1 -- 172.21.15.70:0/1921072407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16b0101840 msgr2=0x7f16b0101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.525+0000 7f16b4b3e640 1 --2- 172.21.15.70:0/1921072407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16b0101840 0x7f16b0101cc0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f16a8009ca0 tx=0x7f16a802c100 comp rx=0 tx=0).stop 2024-04-01T01:48:56.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.527+0000 7f16b4b3e640 1 -- 172.21.15.70:0/1921072407 shutdown_connections 2024-04-01T01:48:56.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.527+0000 7f16b4b3e640 1 --2- 172.21.15.70:0/1921072407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b0102200 0x7f16b0108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.527+0000 7f16b4b3e640 1 --2- 172.21.15.70:0/1921072407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16b0101840 0x7f16b0101cc0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.527+0000 7f16b4b3e640 1 --2- 172.21.15.70:0/1921072407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16b0100650 0x7f16b0100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.527+0000 7f16b4b3e640 1 -- 172.21.15.70:0/1921072407 >> 172.21.15.70:0/1921072407 conn(0x7f16b00fbbe0 msgr2=0x7f16b00fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.527+0000 7f16b4b3e640 1 -- 172.21.15.70:0/1921072407 shutdown_connections 2024-04-01T01:48:56.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.527+0000 7f16b4b3e640 1 -- 172.21.15.70:0/1921072407 wait complete. 2024-04-01T01:48:56.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.528+0000 7f16b4b3e640 1 Processor -- start 2024-04-01T01:48:56.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.528+0000 7f16b4b3e640 1 -- start start 2024-04-01T01:48:56.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.528+0000 7f16b4b3e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16b0100650 0x7f16b019e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.528+0000 7f16b4b3e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16b0101840 0x7f16b019e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.528+0000 7f16b4b3e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b0102200 0x7f16b01a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.528+0000 7f16b4b3e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f16b010ac30 con 0x7f16b0102200 2024-04-01T01:48:56.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.528+0000 7f16b4b3e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f16b010aab0 con 0x7f16b0100650 2024-04-01T01:48:56.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.528+0000 7f16b4b3e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f16b010adb0 con 0x7f16b0101840 2024-04-01T01:48:56.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.528+0000 7f16ae59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16b0100650 0x7f16b019e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.528+0000 7f16ae59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16b0100650 0x7f16b019e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57336/0 (socket says 172.21.15.70:57336) 2024-04-01T01:48:56.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.528+0000 7f16ae59c640 1 -- 172.21.15.70:0/3307151363 learned_addr learned my addr 172.21.15.70:0/3307151363 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:56.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.528+0000 7f16aed9d640 1 --2- 172.21.15.70:0/3307151363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b0102200 0x7f16b01a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.529+0000 7f16add9b640 1 --2- 172.21.15.70:0/3307151363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16b0101840 0x7f16b019e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.529+0000 7f16ae59c640 1 -- 172.21.15.70:0/3307151363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16b0101840 msgr2=0x7f16b019e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.529+0000 7f16ae59c640 1 --2- 172.21.15.70:0/3307151363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16b0101840 0x7f16b019e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.529+0000 7f16ae59c640 1 -- 172.21.15.70:0/3307151363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b0102200 msgr2=0x7f16b01a55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.529+0000 7f16ae59c640 1 --2- 172.21.15.70:0/3307151363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b0102200 0x7f16b01a55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.529+0000 7f16ae59c640 1 -- 172.21.15.70:0/3307151363 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b01a5c40 con 0x7f16b0100650 2024-04-01T01:48:56.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.529+0000 7f16ae59c640 1 --2- 172.21.15.70:0/3307151363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16b0100650 0x7f16b019e020 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f16a000e9c0 tx=0x7f16a000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.530+0000 7f169f7fe640 1 -- 172.21.15.70:0/3307151363 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a000cd80 con 0x7f16b0100650 2024-04-01T01:48:56.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.530+0000 7f16b4b3e640 1 -- 172.21.15.70:0/3307151363 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b01a5f30 con 0x7f16b0100650 2024-04-01T01:48:56.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.530+0000 7f16b4b3e640 1 -- 172.21.15.70:0/3307151363 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b01a6410 con 0x7f16b0100650 2024-04-01T01:48:56.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.530+0000 7f169f7fe640 1 -- 172.21.15.70:0/3307151363 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16a00035f0 con 0x7f16b0100650 2024-04-01T01:48:56.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.531+0000 7f169f7fe640 1 -- 172.21.15.70:0/3307151363 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0010690 con 0x7f16b0100650 2024-04-01T01:48:56.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.533+0000 7f16b77fc740 1 Processor -- start 2024-04-01T01:48:56.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.533+0000 7f16b77fc740 1 -- start start 2024-04-01T01:48:56.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.533+0000 7f169f7fe640 1 -- 172.21.15.70:0/3307151363 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f16a0010830 con 0x7f16b0100650 2024-04-01T01:48:56.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.533+0000 7f16b77fc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 0x562c11be58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.533+0000 7f16b77fc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 0x562c11bea980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.533+0000 7f16b77fc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 0x562c11bed480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.533+0000 7f16b77fc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562c11b4a1b0 con 0x562c11be5e80 2024-04-01T01:48:56.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.533+0000 7f16b77fc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562c11af2c00 con 0x562c11be5490 2024-04-01T01:48:56.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.533+0000 7f16b77fc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562c11b46250 con 0x562c11beaff0 2024-04-01T01:48:56.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.534+0000 7f16ae59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 0x562c11bea980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.534+0000 7f16ae59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 0x562c11bea980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49320/0 (socket says 172.21.15.70:49320) 2024-04-01T01:48:56.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.534+0000 7f16ae59c640 1 -- 172.21.15.70:0/1160429895 learned_addr learned my addr 172.21.15.70:0/1160429895 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:56.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.534+0000 7f16add9b640 1 --2- 172.21.15.70:0/1160429895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 0x562c11bed480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.534+0000 7f16aed9d640 1 --2- 172.21.15.70:0/1160429895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 0x562c11be58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.534+0000 7f169f7fe640 1 --2- 172.21.15.70:0/3307151363 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f16980a25f0 0x7f16980a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.534+0000 7f16add9b640 1 -- 172.21.15.70:0/1160429895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 msgr2=0x562c11be58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.534+0000 7f16add9b640 1 --2- 172.21.15.70:0/1160429895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 0x562c11be58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.534+0000 7f16add9b640 1 -- 172.21.15.70:0/1160429895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 msgr2=0x562c11bea980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.534+0000 7f16add9b640 1 --2- 172.21.15.70:0/1160429895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 0x562c11bea980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.534+0000 7f16add9b640 1 -- 172.21.15.70:0/1160429895 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c11b123d0 con 0x562c11beaff0 2024-04-01T01:48:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.534+0000 7f16aed9d640 1 --2- 172.21.15.70:0/1160429895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 0x562c11be58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:48:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.534+0000 7f169f7fe640 1 -- 172.21.15.70:0/3307151363 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f16a00ceec0 con 0x7f16b0100650 2024-04-01T01:48:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.535+0000 7f16aed9d640 1 --2- 172.21.15.70:0/3307151363 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f16980a25f0 0x7f16980a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.535+0000 7f16add9b640 1 --2- 172.21.15.70:0/1160429895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 0x562c11bed480 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f16a802c640 tx=0x7f16a8009930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6051ec93c3ecaa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.535+0000 7f169cff9640 1 -- 172.21.15.70:0/1160429895 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a8043020 con 0x562c11beaff0 2024-04-01T01:48:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.535+0000 7f169cff9640 1 -- 172.21.15.70:0/1160429895 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16a8003970 con 0x562c11beaff0 2024-04-01T01:48:56.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.535+0000 7f169cff9640 1 -- 172.21.15.70:0/1160429895 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a8039040 con 0x562c11beaff0 2024-04-01T01:48:56.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.536+0000 7f16b77fc740 1 -- 172.21.15.70:0/1160429895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 msgr2=0x562c11bed480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.536+0000 7f16aed9d640 1 --2- 172.21.15.70:0/3307151363 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f16980a25f0 0x7f16980a4ab0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x562c11be66e0 tx=0x7f16a4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.536+0000 7f16b77fc740 1 --2- 172.21.15.70:0/1160429895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 0x562c11bed480 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f16a802c640 tx=0x7f16a8009930 comp rx=0 tx=0).stop 2024-04-01T01:48:56.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.541+0000 7f16b77fc740 1 -- 172.21.15.70:0/1160429895 shutdown_connections 2024-04-01T01:48:56.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.541+0000 7f16b77fc740 1 --2- 172.21.15.70:0/1160429895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 0x562c11bed480 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.541+0000 7f16b77fc740 1 --2- 172.21.15.70:0/1160429895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 0x562c11bea980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.542+0000 7f16b77fc740 1 --2- 172.21.15.70:0/1160429895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 0x562c11be58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.542+0000 7f16b77fc740 1 -- 172.21.15.70:0/1160429895 >> 172.21.15.70:0/1160429895 conn(0x562c11be0a00 msgr2=0x562c11be0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.542+0000 7f16b77fc740 1 -- 172.21.15.70:0/1160429895 shutdown_connections 2024-04-01T01:48:56.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.542+0000 7f16b77fc740 1 -- 172.21.15.70:0/1160429895 wait complete. 2024-04-01T01:48:56.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.542+0000 7f16b77fc740 1 -- start start 2024-04-01T01:48:56.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.542+0000 7f16b77fc740 1 Processor -- start 2024-04-01T01:48:56.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.543+0000 7f16b77fc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 0x562c11bef970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.543+0000 7f16b77fc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 0x562c11befeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.543+0000 7f16ae59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 0x562c11bef970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.543+0000 7f16ae59c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 0x562c11bef970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57356/0 (socket says 172.21.15.70:57356) 2024-04-01T01:48:56.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.543+0000 7f16ae59c640 1 -- 172.21.15.70:0/777697775 learned_addr learned my addr 172.21.15.70:0/777697775 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:56.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.543+0000 7f16b77fc740 1 --2- 172.21.15.70:0/777697775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 0x562c11bf6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.543+0000 7f16b77fc740 1 -- 172.21.15.70:0/777697775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562c11bedf40 con 0x562c11be5e80 2024-04-01T01:48:56.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.543+0000 7f16add9b640 1 --2- 172.21.15.70:0/777697775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 0x562c11befeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.544+0000 7f16b77fc740 1 -- 172.21.15.70:0/777697775 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562c11ae8470 con 0x562c11be5490 2024-04-01T01:48:56.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.544+0000 7f16aed9d640 1 --2- 172.21.15.70:0/777697775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 0x562c11bf6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.544+0000 7f16b77fc740 1 -- 172.21.15.70:0/777697775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562c11bee0c0 con 0x562c11beaff0 2024-04-01T01:48:56.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.544+0000 7f16b77fc740 10 client.0 _ll_register_callbacks cb 0x562c11b05cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:56.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.544+0000 7f16add9b640 1 -- 172.21.15.70:0/777697775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 msgr2=0x562c11bf6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.544+0000 7f16add9b640 1 --2- 172.21.15.70:0/777697775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 0x562c11bf6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.544+0000 7f16add9b640 1 -- 172.21.15.70:0/777697775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 msgr2=0x562c11bef970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.544+0000 7f16add9b640 1 --2- 172.21.15.70:0/777697775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 0x562c11bef970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.544+0000 7f16add9b640 1 -- 172.21.15.70:0/777697775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c1185e3f0 con 0x562c11be5e80 2024-04-01T01:48:56.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.545+0000 7f16add9b640 1 --2- 172.21.15.70:0/777697775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 0x562c11befeb0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f16a8000f80 tx=0x7f16a8005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998e6d71c52f6202 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.545+0000 7f1695ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:48:56.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.546+0000 7f1695ffb640 1 -- 172.21.15.70:0/777697775 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a804c070 con 0x562c11be5e80 2024-04-01T01:48:56.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.546+0000 7f1695ffb640 1 -- 172.21.15.70:0/777697775 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f16a8004140 con 0x562c11be5e80 2024-04-01T01:48:56.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.548+0000 7f16b77fc740 1 -- 172.21.15.70:0/777697775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16a8003970 con 0x562c11be5e80 2024-04-01T01:48:56.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.548+0000 7f1695ffb640 1 -- 172.21.15.70:0/777697775 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f16a8056050 con 0x562c11be5e80 2024-04-01T01:48:56.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.548+0000 7f16b77fc740 10 client.19914 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:56.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.548+0000 7f16b77fc740 1 -- 172.21.15.70:0/777697775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562c119287d0 con 0x562c11be5e80 2024-04-01T01:48:56.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.549+0000 7f1695ffb640 1 -- 172.21.15.70:0/777697775 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16a8003970 con 0x562c11be5e80 2024-04-01T01:48:56.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.549+0000 7f1695ffb640 1 -- 172.21.15.70:0/777697775 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f16a803f970 con 0x562c11be5e80 2024-04-01T01:48:56.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.550+0000 7f16b77fc740 10 client.19914 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:56.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.550+0000 7f16b77fc740 10 client.19914 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:48:56.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.550+0000 7f16b77fc740 20 client.19914 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:56.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.550+0000 7f16b77fc740 1 --2- 172.21.15.70:0/777697775 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x562c11bfb030 0x562c11bfd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.550+0000 7f16b77fc740 4 client.19914 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:56.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.550+0000 7f16b77fc740 1 -- 172.21.15.70:0/777697775 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c11b576f0 con 0x562c11bfb030 2024-04-01T01:48:56.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.550+0000 7f16ae59c640 1 --2- 172.21.15.70:0/777697775 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x562c11bfb030 0x562c11bfd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.551+0000 7f16ae59c640 1 --2- 172.21.15.70:0/777697775 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x562c11bfb030 0x562c11bfd420 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c012ae41949538f1 server_cookie=47e8ebc0f9f93926 in_seq=0 out_seq=0 2024-04-01T01:48:56.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.551+0000 7f1695ffb640 10 client.19914 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:48:56.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.555+0000 7f1695ffb640 1 -- 172.21.15.70:0/777697775 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562c11b576f0 con 0x562c11bfb030 2024-04-01T01:48:56.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.555+0000 7f1695ffb640 10 client.19914 handle_command_reply: tid=0 2024-04-01T01:48:56.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.556+0000 7f16b77fc740 1 client.19914 shutdown 2024-04-01T01:48:56.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.557+0000 7f16b77fc740 1 -- 172.21.15.70:0/777697775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 msgr2=0x562c11befeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.557+0000 7f16b77fc740 1 --2- 172.21.15.70:0/777697775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 0x562c11befeb0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f16a8000f80 tx=0x7f16a8005bf0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.558+0000 7f16b77fc740 1 -- 172.21.15.70:0/777697775 shutdown_connections 2024-04-01T01:48:56.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.559+0000 7f16b77fc740 1 --2- 172.21.15.70:0/777697775 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x562c11bfb030 0x562c11bfd420 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.559+0000 7f16b77fc740 1 --2- 172.21.15.70:0/777697775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 0x562c11bf6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.559+0000 7f16b77fc740 1 --2- 172.21.15.70:0/777697775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 0x562c11befeb0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.559+0000 7f16b77fc740 1 --2- 172.21.15.70:0/777697775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 0x562c11bef970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.559+0000 7f16b77fc740 1 -- 172.21.15.70:0/777697775 >> 172.21.15.70:0/777697775 conn(0x562c11be0a00 msgr2=0x562c11bdfd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.560+0000 7f1695ffb640 0 client.19914 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:48:56.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.560+0000 7f16b77fc740 1 -- 172.21.15.70:0/777697775 shutdown_connections 2024-04-01T01:48:56.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.561+0000 7f16b77fc740 1 -- 172.21.15.70:0/777697775 wait complete. 2024-04-01T01:48:56.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.561+0000 7f16b77fc740 20 client.19914 trim_cache size 0 max 16384 2024-04-01T01:48:56.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.592+0000 7f16b77fc740 1 Processor -- start 2024-04-01T01:48:56.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.593+0000 7f16b77fc740 1 -- start start 2024-04-01T01:48:56.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.593+0000 7f16b77fc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11be5490 0x562c11be4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.593+0000 7f16b77fc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5e80 0x562c11bf6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.593+0000 7f16b77fc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11beaff0 0x562c11c01c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.593+0000 7f16b77fc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f169000fde0 con 0x562c11beaff0 2024-04-01T01:48:56.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.593+0000 7f16b77fc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1690011920 con 0x562c11be5e80 2024-04-01T01:48:56.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.593+0000 7f16b77fc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f16900104b0 con 0x562c11be5490 2024-04-01T01:48:56.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.593+0000 7f16add9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5e80 0x562c11bf6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.593+0000 7f16add9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5e80 0x562c11bf6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:57368/0 (socket says 172.21.15.70:57368) 2024-04-01T01:48:56.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.593+0000 7f16add9b640 1 -- 172.21.15.70:0/677753420 learned_addr learned my addr 172.21.15.70:0/677753420 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:56.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.594+0000 7f16add9b640 1 -- 172.21.15.70:0/677753420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11be5490 msgr2=0x562c11be4370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:48:56.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.594+0000 7f16add9b640 1 --2- 172.21.15.70:0/677753420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11be5490 0x562c11be4370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.594+0000 7f16add9b640 1 -- 172.21.15.70:0/677753420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11beaff0 msgr2=0x562c11c01c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:48:56.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.594+0000 7f16add9b640 1 --2- 172.21.15.70:0/677753420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11beaff0 0x562c11c01c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.594+0000 7f16add9b640 1 -- 172.21.15.70:0/677753420 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c11be1970 con 0x562c11be5e80 2024-04-01T01:48:56.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.595+0000 7f16add9b640 1 --2- 172.21.15.70:0/677753420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5e80 0x562c11bf6eb0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f16a802c640 tx=0x7f16a8032aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdbf3e14560f0e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.596+0000 7f1697fff640 1 -- 172.21.15.70:0/677753420 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a8071070 con 0x562c11be5e80 2024-04-01T01:48:56.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.596+0000 7f1697fff640 1 -- 172.21.15.70:0/677753420 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16a8003970 con 0x562c11be5e80 2024-04-01T01:48:56.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.596+0000 7f1697fff640 1 -- 172.21.15.70:0/677753420 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a804c040 con 0x562c11be5e80 2024-04-01T01:48:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.597+0000 7f16b77fc740 1 -- 172.21.15.70:0/677753420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5e80 msgr2=0x562c11bf6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.597+0000 7f16b77fc740 1 --2- 172.21.15.70:0/677753420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5e80 0x562c11bf6eb0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f16a802c640 tx=0x7f16a8032aa0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.598+0000 7f16b77fc740 1 -- 172.21.15.70:0/677753420 shutdown_connections 2024-04-01T01:48:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.598+0000 7f16b77fc740 1 --2- 172.21.15.70:0/677753420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11beaff0 0x562c11c01c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.598+0000 7f16b77fc740 1 --2- 172.21.15.70:0/677753420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5e80 0x562c11bf6eb0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.598+0000 7f16b77fc740 1 --2- 172.21.15.70:0/677753420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11be5490 0x562c11be4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.598+0000 7f16b77fc740 1 -- 172.21.15.70:0/677753420 >> 172.21.15.70:0/677753420 conn(0x562c11be0a00 msgr2=0x562c11be3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.598+0000 7f16b77fc740 1 -- 172.21.15.70:0/677753420 shutdown_connections 2024-04-01T01:48:56.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.598+0000 7f16b77fc740 1 -- 172.21.15.70:0/677753420 wait complete. 2024-04-01T01:48:56.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.598+0000 7f16b77fc740 1 -- start start 2024-04-01T01:48:56.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.599+0000 7f16b77fc740 1 Processor -- start 2024-04-01T01:48:56.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.599+0000 7f16b77fc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 0x562c11bf4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.599+0000 7f16b77fc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 0x562c11bf4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.599+0000 7f16b77fc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 0x562c11bf5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.599+0000 7f16b77fc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1690011250 con 0x562c11be5e80 2024-04-01T01:48:56.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.599+0000 7f16b77fc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1690010b80 con 0x562c11be5490 2024-04-01T01:48:56.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.599+0000 7f16b77fc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f169000ddc0 con 0x562c11beaff0 2024-04-01T01:48:56.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.599+0000 7f16b77fc740 10 client.0 _ll_register_callbacks cb 0x7f16a8032940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:56.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.599+0000 7f16aed9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 0x562c11bf5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.599+0000 7f16aed9d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 0x562c11bf5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:34328/0 (socket says 172.21.15.70:34328) 2024-04-01T01:48:56.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.599+0000 7f16aed9d640 1 -- 172.21.15.70:0/1523448884 learned_addr learned my addr 172.21.15.70:0/1523448884 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:56.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.599+0000 7f16add9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 0x562c11bf4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.599+0000 7f16ae59c640 1 --2- 172.21.15.70:0/1523448884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 0x562c11bf4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.600+0000 7f16ae59c640 1 -- 172.21.15.70:0/1523448884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 msgr2=0x562c11bf5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.600+0000 7f16ae59c640 1 --2- 172.21.15.70:0/1523448884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 0x562c11bf5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.600+0000 7f16ae59c640 1 -- 172.21.15.70:0/1523448884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 msgr2=0x562c11bf4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.600+0000 7f16ae59c640 1 --2- 172.21.15.70:0/1523448884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 0x562c11bf4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.600+0000 7f16ae59c640 1 -- 172.21.15.70:0/1523448884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c11bf62a0 con 0x562c11be5490 2024-04-01T01:48:56.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.605+0000 7f16ae59c640 1 --2- 172.21.15.70:0/1523448884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 0x562c11bf4760 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f16a00a3a90 tx=0x7f16a00038f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=111e21f4de168308 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.605+0000 7f1695ffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.605+0000 7f1695ffb640 1 -- 172.21.15.70:0/1523448884 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a00a3fc0 con 0x562c11be5490 2024-04-01T01:48:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.605+0000 7f1695ffb640 1 -- 172.21.15.70:0/1523448884 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f16a0098ca0 con 0x562c11be5490 2024-04-01T01:48:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.605+0000 7f16b77fc740 1 -- 172.21.15.70:0/1523448884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16a8003970 con 0x562c11be5490 2024-04-01T01:48:56.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.606+0000 7f1695ffb640 1 -- 172.21.15.70:0/1523448884 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f16a00b40f0 con 0x562c11be5490 2024-04-01T01:48:56.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.606+0000 7f16b77fc740 10 client.30023 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:56.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.606+0000 7f16b77fc740 1 -- 172.21.15.70:0/1523448884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562c11c614d0 con 0x562c11be5490 2024-04-01T01:48:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.607+0000 7f1695ffb640 1 -- 172.21.15.70:0/1523448884 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16a8003970 con 0x562c11be5490 2024-04-01T01:48:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.607+0000 7f1695ffb640 1 -- 172.21.15.70:0/1523448884 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f16a00bb0b0 con 0x562c11be5490 2024-04-01T01:48:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.607+0000 7f16b77fc740 10 client.30023 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.607+0000 7f16b77fc740 10 client.30023 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:48:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.607+0000 7f16b77fc740 20 client.30023 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.607+0000 7f16b77fc740 1 --2- 172.21.15.70:0/1523448884 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x562c11c61df0 0x562c11c64210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.607+0000 7f16b77fc740 4 client.30023 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:48:56.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.607+0000 7f16b77fc740 1 -- 172.21.15.70:0/1523448884 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562c11b51a40 con 0x562c11c61df0 2024-04-01T01:48:56.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.608+0000 7f16add9b640 1 --2- 172.21.15.70:0/1523448884 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x562c11c61df0 0x562c11c64210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.609+0000 7f16add9b640 1 --2- 172.21.15.70:0/1523448884 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x562c11c61df0 0x562c11c64210 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bd9e5cc9d9450015 server_cookie=1191bd3927f53126 in_seq=0 out_seq=0 2024-04-01T01:48:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.609+0000 7f1695ffb640 10 client.30023 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:48:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.610+0000 7f1695ffb640 1 -- 172.21.15.70:0/1523448884 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562c11b51a40 con 0x562c11c61df0 2024-04-01T01:48:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.610+0000 7f1695ffb640 10 client.30023 handle_command_reply: tid=0 2024-04-01T01:48:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.610+0000 7f16b77fc740 1 client.30023 shutdown 2024-04-01T01:48:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.611+0000 7f16b77fc740 1 -- 172.21.15.70:0/1523448884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 msgr2=0x562c11bf4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.611+0000 7f16b77fc740 1 --2- 172.21.15.70:0/1523448884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 0x562c11bf4760 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f16a00a3a90 tx=0x7f16a00038f0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.612+0000 7f16b77fc740 1 -- 172.21.15.70:0/1523448884 shutdown_connections 2024-04-01T01:48:56.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.612+0000 7f16b77fc740 1 --2- 172.21.15.70:0/1523448884 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x562c11c61df0 0x562c11c64210 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.612+0000 7f16b77fc740 1 --2- 172.21.15.70:0/1523448884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562c11beaff0 0x562c11bf5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.612+0000 7f16b77fc740 1 --2- 172.21.15.70:0/1523448884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562c11be5e80 0x562c11bf4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.612+0000 7f16b77fc740 1 --2- 172.21.15.70:0/1523448884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562c11be5490 0x562c11bf4760 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.612+0000 7f16b77fc740 1 -- 172.21.15.70:0/1523448884 >> 172.21.15.70:0/1523448884 conn(0x562c11be0a00 msgr2=0x562c11be44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.612+0000 7f1695ffb640 0 client.30023 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:48:56.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.612+0000 7f16b77fc740 1 -- 172.21.15.70:0/1523448884 shutdown_connections 2024-04-01T01:48:56.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.613+0000 7f16b77fc740 1 -- 172.21.15.70:0/1523448884 wait complete. 2024-04-01T01:48:56.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.613+0000 7f16b77fc740 20 client.30023 trim_cache size 0 max 16384 2024-04-01T01:48:56.615 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:48:56.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.617+0000 7f16b4b3e640 1 -- 172.21.15.70:0/3307151363 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f16980a25f0 msgr2=0x7f16980a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.617+0000 7f16b4b3e640 1 --2- 172.21.15.70:0/3307151363 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f16980a25f0 0x7f16980a4ab0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x562c11be66e0 tx=0x7f16a4008040 comp rx=0 tx=0).stop 2024-04-01T01:48:56.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.617+0000 7f16b4b3e640 1 -- 172.21.15.70:0/3307151363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16b0100650 msgr2=0x7f16b019e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.617+0000 7f16b4b3e640 1 --2- 172.21.15.70:0/3307151363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16b0100650 0x7f16b019e020 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f16a000e9c0 tx=0x7f16a000ee80 comp rx=0 tx=0).stop 2024-04-01T01:48:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.617+0000 7f16b4b3e640 1 -- 172.21.15.70:0/3307151363 shutdown_connections 2024-04-01T01:48:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.617+0000 7f16b4b3e640 1 --2- 172.21.15.70:0/3307151363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f16b0102200 0x7f16b01a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.617+0000 7f16b4b3e640 1 --2- 172.21.15.70:0/3307151363 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f16980a25f0 0x7f16980a4ab0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.617+0000 7f16b4b3e640 1 --2- 172.21.15.70:0/3307151363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16b0101840 0x7f16b019e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.617+0000 7f16b4b3e640 1 --2- 172.21.15.70:0/3307151363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f16b0100650 0x7f16b019e020 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.617+0000 7f16b4b3e640 1 -- 172.21.15.70:0/3307151363 >> 172.21.15.70:0/3307151363 conn(0x7f16b00fbbe0 msgr2=0x7f16b010dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.622+0000 7f16b4b3e640 1 -- 172.21.15.70:0/3307151363 shutdown_connections 2024-04-01T01:48:56.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.622+0000 7f16b4b3e640 1 -- 172.21.15.70:0/3307151363 wait complete. 2024-04-01T01:48:56.643 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:48:56.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.838+0000 7fe09615a640 1 -- 172.21.15.70:0/786896679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0901011a0 msgr2=0x7fe090101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.838+0000 7fe09615a640 1 --2- 172.21.15.70:0/786896679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0901011a0 0x7fe090101620 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe08c009900 tx=0x7fe08c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:48:56.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.838+0000 7fe09615a640 1 -- 172.21.15.70:0/786896679 shutdown_connections 2024-04-01T01:48:56.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.838+0000 7fe09615a640 1 --2- 172.21.15.70:0/786896679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0901011a0 0x7fe090101620 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.838+0000 7fe09615a640 1 --2- 172.21.15.70:0/786896679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe090100800 0x7fe090100c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.838+0000 7fe09615a640 1 --2- 172.21.15.70:0/786896679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0900ffea0 0x7fe0901002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.838+0000 7fe09615a640 1 -- 172.21.15.70:0/786896679 >> 172.21.15.70:0/786896679 conn(0x7fe0900fbac0 msgr2=0x7fe0900fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.838+0000 7fe09615a640 1 -- 172.21.15.70:0/786896679 shutdown_connections 2024-04-01T01:48:56.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.838+0000 7fe09615a640 1 -- 172.21.15.70:0/786896679 wait complete. 2024-04-01T01:48:56.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.839+0000 7fe09615a640 1 Processor -- start 2024-04-01T01:48:56.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.839+0000 7fe09615a640 1 -- start start 2024-04-01T01:48:56.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.839+0000 7fe09615a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0900ffea0 0x7fe090105a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.839+0000 7fe09615a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe090100800 0x7fe090105f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.840+0000 7fe09615a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0901011a0 0x7fe090102590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.840+0000 7fe09615a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe09010f330 con 0x7fe090100800 2024-04-01T01:48:56.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.840+0000 7fe09615a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe09010f1b0 con 0x7fe0900ffea0 2024-04-01T01:48:56.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.840+0000 7fe09615a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe09010f4b0 con 0x7fe0901011a0 2024-04-01T01:48:56.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.840+0000 7fe094957640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe090100800 0x7fe090105f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.840+0000 7fe094957640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe090100800 0x7fe090105f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38256/0 (socket says 172.21.15.70:38256) 2024-04-01T01:48:56.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.840+0000 7fe094957640 1 -- 172.21.15.70:0/2720589651 learned_addr learned my addr 172.21.15.70:0/2720589651 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:56.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.841+0000 7fe095959640 1 --2- 172.21.15.70:0/2720589651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0901011a0 0x7fe090102590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.841+0000 7fe094957640 1 -- 172.21.15.70:0/2720589651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0901011a0 msgr2=0x7fe090102590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.841+0000 7fe094957640 1 --2- 172.21.15.70:0/2720589651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0901011a0 0x7fe090102590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.841+0000 7fe094957640 1 -- 172.21.15.70:0/2720589651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0900ffea0 msgr2=0x7fe090105a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.841+0000 7fe095158640 1 --2- 172.21.15.70:0/2720589651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0900ffea0 0x7fe090105a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.841+0000 7fe094957640 1 --2- 172.21.15.70:0/2720589651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0900ffea0 0x7fe090105a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.841+0000 7fe094957640 1 -- 172.21.15.70:0/2720589651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe090102e50 con 0x7fe090100800 2024-04-01T01:48:56.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.841+0000 7fe095959640 1 --2- 172.21.15.70:0/2720589651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0901011a0 0x7fe090102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:48:56.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.841+0000 7fe095158640 1 --2- 172.21.15.70:0/2720589651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0900ffea0 0x7fe090105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:48:56.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.843+0000 7fe094957640 1 --2- 172.21.15.70:0/2720589651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe090100800 0x7fe090105f60 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fe08000e9f0 tx=0x7fe08000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.844+0000 7fe0867fc640 1 -- 172.21.15.70:0/2720589651 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08000cd80 con 0x7fe090100800 2024-04-01T01:48:56.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.844+0000 7fe0867fc640 1 -- 172.21.15.70:0/2720589651 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe0800108e0 con 0x7fe090100800 2024-04-01T01:48:56.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.844+0000 7fe0867fc640 1 -- 172.21.15.70:0/2720589651 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe080018920 con 0x7fe090100800 2024-04-01T01:48:56.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.844+0000 7fe09615a640 1 -- 172.21.15.70:0/2720589651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe09019b610 con 0x7fe090100800 2024-04-01T01:48:56.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.845+0000 7fe09615a640 1 -- 172.21.15.70:0/2720589651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe09019bb10 con 0x7fe090100800 2024-04-01T01:48:56.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.849+0000 7fe099f9e740 1 Processor -- start 2024-04-01T01:48:56.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.849+0000 7fe099f9e740 1 -- start start 2024-04-01T01:48:56.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.849+0000 7fe099f9e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d666b0 0x555f61d66ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.849+0000 7fe099f9e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d670a0 0x555f61d6bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.849+0000 7fe099f9e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d6c210 0x555f61d6e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.849+0000 7fe099f9e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555f61cba1b0 con 0x555f61d6c210 2024-04-01T01:48:56.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.849+0000 7fe099f9e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555f61c62c00 con 0x555f61d670a0 2024-04-01T01:48:56.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.850+0000 7fe099f9e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555f61cb6250 con 0x555f61d666b0 2024-04-01T01:48:56.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.850+0000 7fe095158640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d670a0 0x555f61d6bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.850+0000 7fe0867fc640 1 -- 172.21.15.70:0/2720589651 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fe080010a90 con 0x7fe090100800 2024-04-01T01:48:56.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.850+0000 7fe095158640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d670a0 0x555f61d6bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41112/0 (socket says 172.21.15.70:41112) 2024-04-01T01:48:56.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.850+0000 7fe095158640 1 -- 172.21.15.70:0/919829054 learned_addr learned my addr 172.21.15.70:0/919829054 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:56.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.850+0000 7fe094957640 1 --2- 172.21.15.70:0/919829054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d6c210 0x555f61d6e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.851+0000 7fe094957640 1 -- 172.21.15.70:0/919829054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d666b0 msgr2=0x555f61d66ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:48:56.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.851+0000 7fe094957640 1 --2- 172.21.15.70:0/919829054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d666b0 0x555f61d66ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.851+0000 7fe094957640 1 -- 172.21.15.70:0/919829054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d670a0 msgr2=0x555f61d6bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.851+0000 7fe094957640 1 --2- 172.21.15.70:0/919829054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d670a0 0x555f61d6bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.851+0000 7fe094957640 1 -- 172.21.15.70:0/919829054 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f61c823d0 con 0x555f61d6c210 2024-04-01T01:48:56.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.851+0000 7fe095158640 1 --2- 172.21.15.70:0/919829054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d670a0 0x555f61d6bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:56.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.851+0000 7fe094957640 1 --2- 172.21.15.70:0/919829054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d6c210 0x555f61d6e6a0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fe0800cc520 tx=0x7fe0800d74b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aedc74655b707ee server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.852+0000 7fe0867fc640 1 --2- 172.21.15.70:0/2720589651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe07c0a2670 0x7fe07c0a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.852+0000 7fe07b7fe640 1 -- 172.21.15.70:0/919829054 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0800e3070 con 0x555f61d6c210 2024-04-01T01:48:56.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.852+0000 7fe095959640 1 --2- 172.21.15.70:0/2720589651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe07c0a2670 0x7fe07c0a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.852+0000 7fe0867fc640 1 -- 172.21.15.70:0/2720589651 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe0800ceeb0 con 0x7fe090100800 2024-04-01T01:48:56.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.852+0000 7fe07b7fe640 1 -- 172.21.15.70:0/919829054 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe080010de0 con 0x555f61d6c210 2024-04-01T01:48:56.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.852+0000 7fe07b7fe640 1 -- 172.21.15.70:0/919829054 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08009b310 con 0x555f61d6c210 2024-04-01T01:48:56.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.853+0000 7fe099f9e740 1 -- 172.21.15.70:0/919829054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d6c210 msgr2=0x555f61d6e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.853+0000 7fe099f9e740 1 --2- 172.21.15.70:0/919829054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d6c210 0x555f61d6e6a0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fe0800cc520 tx=0x7fe0800d74b0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.853+0000 7fe095959640 1 --2- 172.21.15.70:0/2720589651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe07c0a2670 0x7fe07c0a4b30 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fe08c036040 tx=0x7fe08c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.854+0000 7fe099f9e740 1 -- 172.21.15.70:0/919829054 shutdown_connections 2024-04-01T01:48:56.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.854+0000 7fe099f9e740 1 --2- 172.21.15.70:0/919829054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d6c210 0x555f61d6e6a0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.854+0000 7fe099f9e740 1 --2- 172.21.15.70:0/919829054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d670a0 0x555f61d6bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.854+0000 7fe099f9e740 1 --2- 172.21.15.70:0/919829054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d666b0 0x555f61d66ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.854+0000 7fe099f9e740 1 -- 172.21.15.70:0/919829054 >> 172.21.15.70:0/919829054 conn(0x555f61d50c10 msgr2=0x555f61d51010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.854+0000 7fe099f9e740 1 -- 172.21.15.70:0/919829054 shutdown_connections 2024-04-01T01:48:56.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.854+0000 7fe099f9e740 1 -- 172.21.15.70:0/919829054 wait complete. 2024-04-01T01:48:56.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.855+0000 7fe099f9e740 1 -- start start 2024-04-01T01:48:56.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.855+0000 7fe099f9e740 1 Processor -- start 2024-04-01T01:48:56.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.855+0000 7fe099f9e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d666b0 0x555f61d70c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.856+0000 7fe099f9e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d670a0 0x555f61d711a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.856+0000 7fe099f9e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d6c210 0x555f61d781e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.856+0000 7fe099f9e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555f61d6f1c0 con 0x555f61d666b0 2024-04-01T01:48:56.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.856+0000 7fe099f9e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555f61c58470 con 0x555f61d6c210 2024-04-01T01:48:56.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.856+0000 7fe099f9e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555f61d6f340 con 0x555f61d670a0 2024-04-01T01:48:56.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.856+0000 7fe095158640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d666b0 0x555f61d70c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.856+0000 7fe095158640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d666b0 0x555f61d70c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38270/0 (socket says 172.21.15.70:38270) 2024-04-01T01:48:56.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.856+0000 7fe095158640 1 -- 172.21.15.70:0/2434276037 learned_addr learned my addr 172.21.15.70:0/2434276037 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:56.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.856+0000 7fe099f9e740 10 client.0 _ll_register_callbacks cb 0x555f61c75cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:56.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.856+0000 7fe095959640 1 --2- 172.21.15.70:0/2434276037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d6c210 0x555f61d781e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.856+0000 7fe094957640 1 --2- 172.21.15.70:0/2434276037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d670a0 0x555f61d711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.856+0000 7fe095158640 1 -- 172.21.15.70:0/2434276037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d670a0 msgr2=0x555f61d711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.856+0000 7fe095158640 1 --2- 172.21.15.70:0/2434276037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d670a0 0x555f61d711a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.856+0000 7fe095158640 1 -- 172.21.15.70:0/2434276037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d6c210 msgr2=0x555f61d781e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.856+0000 7fe095158640 1 --2- 172.21.15.70:0/2434276037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d6c210 0x555f61d781e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.856+0000 7fe095158640 1 -- 172.21.15.70:0/2434276037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f619ce3f0 con 0x555f61d666b0 2024-04-01T01:48:56.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.857+0000 7fe095158640 1 --2- 172.21.15.70:0/2434276037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d666b0 0x555f61d70c60 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fe0880126d0 tx=0x7fe088012b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4510c4398b7e2c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.857+0000 7fe078ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:48:56.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.857+0000 7fe078ff9640 1 -- 172.21.15.70:0/2434276037 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe088003d80 con 0x555f61d666b0 2024-04-01T01:48:56.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.857+0000 7fe099f9e740 1 -- 172.21.15.70:0/2434276037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555f61cc76f0 con 0x555f61d666b0 2024-04-01T01:48:56.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.857+0000 7fe078ff9640 1 -- 172.21.15.70:0/2434276037 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe088010950 con 0x555f61d666b0 2024-04-01T01:48:56.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.857+0000 7fe078ff9640 1 -- 172.21.15.70:0/2434276037 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe088010b50 con 0x555f61d666b0 2024-04-01T01:48:56.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.857+0000 7fe099f9e740 10 client.19929 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:56.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.857+0000 7fe099f9e740 1 -- 172.21.15.70:0/2434276037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555f61a987d0 con 0x555f61d666b0 2024-04-01T01:48:56.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.858+0000 7fe078ff9640 1 -- 172.21.15.70:0/2434276037 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x555f61cc76f0 con 0x555f61d666b0 2024-04-01T01:48:56.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.858+0000 7fe078ff9640 1 -- 172.21.15.70:0/2434276037 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe088019020 con 0x555f61d666b0 2024-04-01T01:48:56.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.858+0000 7fe099f9e740 10 client.19929 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:56.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.858+0000 7fe099f9e740 10 client.19929 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:48:56.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.858+0000 7fe099f9e740 20 client.19929 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:56.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.858+0000 7fe099f9e740 1 --2- 172.21.15.70:0/2434276037 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x555f61d7c740 0x555f61d7ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.858+0000 7fe099f9e740 4 client.19929 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:56.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.858+0000 7fe099f9e740 1 -- 172.21.15.70:0/2434276037 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f61cc1a40 con 0x555f61d7c740 2024-04-01T01:48:56.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.861+0000 7fe094957640 1 --2- 172.21.15.70:0/2434276037 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x555f61d7c740 0x555f61d7ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.862+0000 7fe094957640 1 --2- 172.21.15.70:0/2434276037 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x555f61d7c740 0x555f61d7ebc0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e45356b9257b78d1 server_cookie=5ba7d579e0c287e1 in_seq=0 out_seq=0 2024-04-01T01:48:56.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.862+0000 7fe078ff9640 10 client.19929 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:48:56.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.866+0000 7fe078ff9640 1 -- 172.21.15.70:0/2434276037 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555f61cc1a40 con 0x555f61d7c740 2024-04-01T01:48:56.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.866+0000 7fe078ff9640 10 client.19929 handle_command_reply: tid=0 2024-04-01T01:48:56.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.867+0000 7fe099f9e740 1 client.19929 shutdown 2024-04-01T01:48:56.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.868+0000 7fe099f9e740 1 -- 172.21.15.70:0/2434276037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d666b0 msgr2=0x555f61d70c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.868+0000 7fe099f9e740 1 --2- 172.21.15.70:0/2434276037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d666b0 0x555f61d70c60 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fe0880126d0 tx=0x7fe088012b90 comp rx=0 tx=0).stop 2024-04-01T01:48:56.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.868+0000 7fe099f9e740 1 -- 172.21.15.70:0/2434276037 shutdown_connections 2024-04-01T01:48:56.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.868+0000 7fe099f9e740 1 --2- 172.21.15.70:0/2434276037 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x555f61d7c740 0x555f61d7ebc0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.868+0000 7fe099f9e740 1 --2- 172.21.15.70:0/2434276037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d6c210 0x555f61d781e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.868+0000 7fe099f9e740 1 --2- 172.21.15.70:0/2434276037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d670a0 0x555f61d711a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.868+0000 7fe099f9e740 1 --2- 172.21.15.70:0/2434276037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d666b0 0x555f61d70c60 secure :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fe0880126d0 tx=0x7fe088012b90 comp rx=0 tx=0).stop 2024-04-01T01:48:56.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.868+0000 7fe099f9e740 1 -- 172.21.15.70:0/2434276037 >> 172.21.15.70:0/2434276037 conn(0x555f61d50c10 msgr2=0x555f61d4ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.870+0000 7fe078ff9640 0 client.19929 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:48:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.870+0000 7fe099f9e740 1 -- 172.21.15.70:0/2434276037 shutdown_connections 2024-04-01T01:48:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.870+0000 7fe099f9e740 1 -- 172.21.15.70:0/2434276037 wait complete. 2024-04-01T01:48:56.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.871+0000 7fe099f9e740 20 client.19929 trim_cache size 0 max 16384 2024-04-01T01:48:56.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 850.0 s, 729.8 tps, lat 138.766 ms stddev 105.905 2024-04-01T01:48:56.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.897+0000 7fe099f9e740 1 Processor -- start 2024-04-01T01:48:56.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.899+0000 7fe099f9e740 1 -- start start 2024-04-01T01:48:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.900+0000 7fe099f9e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d670a0 0x555f61d4fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.900+0000 7fe099f9e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d6c210 0x555f61d781a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.900+0000 7fe099f9e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d702b0 0x555f61d70730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.900+0000 7fe099f9e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe09006f410 con 0x555f61d6c210 2024-04-01T01:48:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.900+0000 7fe099f9e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe090070f50 con 0x555f61d670a0 2024-04-01T01:48:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.900+0000 7fe099f9e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe09006fae0 con 0x555f61d702b0 2024-04-01T01:48:56.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.901+0000 7fe095959640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d702b0 0x555f61d70730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.901+0000 7fe095959640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d702b0 0x555f61d70730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46480/0 (socket says 172.21.15.70:46480) 2024-04-01T01:48:56.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.901+0000 7fe095959640 1 -- 172.21.15.70:0/2489681892 learned_addr learned my addr 172.21.15.70:0/2489681892 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:56.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.901+0000 7fe095158640 1 --2- 172.21.15.70:0/2489681892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d670a0 0x555f61d4fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.901+0000 7fe094957640 1 --2- 172.21.15.70:0/2489681892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d6c210 0x555f61d781a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.901+0000 7fe094957640 1 -- 172.21.15.70:0/2489681892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d702b0 msgr2=0x555f61d70730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.901+0000 7fe094957640 1 --2- 172.21.15.70:0/2489681892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d702b0 0x555f61d70730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.901+0000 7fe094957640 1 -- 172.21.15.70:0/2489681892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d670a0 msgr2=0x555f61d4fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.901+0000 7fe094957640 1 --2- 172.21.15.70:0/2489681892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d670a0 0x555f61d4fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.901+0000 7fe094957640 1 -- 172.21.15.70:0/2489681892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f61d67480 con 0x555f61d6c210 2024-04-01T01:48:56.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.902+0000 7fe094957640 1 --2- 172.21.15.70:0/2489681892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d6c210 0x555f61d781a0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fe0800cf1e0 tx=0x7fe0800d2870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b25d0b552bca978 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.903+0000 7fe07affd640 1 -- 172.21.15.70:0/2489681892 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe080066c50 con 0x555f61d6c210 2024-04-01T01:48:56.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.903+0000 7fe07affd640 1 -- 172.21.15.70:0/2489681892 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe08009bc40 con 0x555f61d6c210 2024-04-01T01:48:56.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.903+0000 7fe07affd640 1 -- 172.21.15.70:0/2489681892 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08009aea0 con 0x555f61d6c210 2024-04-01T01:48:56.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.904+0000 7fe099f9e740 1 -- 172.21.15.70:0/2489681892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d6c210 msgr2=0x555f61d781a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.904+0000 7fe099f9e740 1 --2- 172.21.15.70:0/2489681892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d6c210 0x555f61d781a0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fe0800cf1e0 tx=0x7fe0800d2870 comp rx=0 tx=0).stop 2024-04-01T01:48:56.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.904+0000 7fe099f9e740 1 -- 172.21.15.70:0/2489681892 shutdown_connections 2024-04-01T01:48:56.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.904+0000 7fe099f9e740 1 --2- 172.21.15.70:0/2489681892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d702b0 0x555f61d70730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.904+0000 7fe099f9e740 1 --2- 172.21.15.70:0/2489681892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d6c210 0x555f61d781a0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.904+0000 7fe099f9e740 1 --2- 172.21.15.70:0/2489681892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d670a0 0x555f61d4fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.904+0000 7fe099f9e740 1 -- 172.21.15.70:0/2489681892 >> 172.21.15.70:0/2489681892 conn(0x555f61d50c10 msgr2=0x555f61d4f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.909+0000 7fe099f9e740 1 -- 172.21.15.70:0/2489681892 shutdown_connections 2024-04-01T01:48:56.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.910+0000 7fe099f9e740 1 -- 172.21.15.70:0/2489681892 wait complete. 2024-04-01T01:48:56.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.910+0000 7fe099f9e740 1 -- start start 2024-04-01T01:48:56.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.911+0000 7fe099f9e740 1 Processor -- start 2024-04-01T01:48:56.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.911+0000 7fe099f9e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d670a0 0x555f61d75f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.911+0000 7fe099f9e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d6c210 0x555f61d76450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.911+0000 7fe099f9e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d702b0 0x555f61d77460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.911+0000 7fe094957640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d6c210 0x555f61d76450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.912+0000 7fe094957640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d6c210 0x555f61d76450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46496/0 (socket says 172.21.15.70:46496) 2024-04-01T01:48:56.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.912+0000 7fe094957640 1 -- 172.21.15.70:0/1838549611 learned_addr learned my addr 172.21.15.70:0/1838549611 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:56.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.912+0000 7fe095959640 1 --2- 172.21.15.70:0/1838549611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d702b0 0x555f61d77460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.912+0000 7fe095158640 1 --2- 172.21.15.70:0/1838549611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d670a0 0x555f61d75f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.912+0000 7fe099f9e740 1 -- 172.21.15.70:0/1838549611 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe090070880 con 0x555f61d670a0 2024-04-01T01:48:56.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.912+0000 7fe099f9e740 1 -- 172.21.15.70:0/1838549611 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe0900701b0 con 0x555f61d702b0 2024-04-01T01:48:56.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.912+0000 7fe099f9e740 1 -- 172.21.15.70:0/1838549611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe0901a1c60 con 0x555f61d6c210 2024-04-01T01:48:56.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.912+0000 7fe099f9e740 10 client.0 _ll_register_callbacks cb 0x7fe088009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:56.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.913+0000 7fe095959640 1 -- 172.21.15.70:0/1838549611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d6c210 msgr2=0x555f61d76450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.913+0000 7fe095959640 1 --2- 172.21.15.70:0/1838549611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d6c210 0x555f61d76450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.913+0000 7fe095959640 1 -- 172.21.15.70:0/1838549611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d670a0 msgr2=0x555f61d75f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:56.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.913+0000 7fe095959640 1 --2- 172.21.15.70:0/1838549611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d670a0 0x555f61d75f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.913+0000 7fe095959640 1 -- 172.21.15.70:0/1838549611 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f61d77a50 con 0x555f61d702b0 2024-04-01T01:48:56.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.913+0000 7fe099f9e740 1 -- 172.21.15.70:0/1838549611 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe08009bc40 con 0x555f61d702b0 2024-04-01T01:48:56.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.913+0000 7fe094957640 1 --2- 172.21.15.70:0/1838549611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d6c210 0x555f61d76450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:48:56.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.913+0000 7fe095959640 1 --2- 172.21.15.70:0/1838549611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d702b0 0x555f61d77460 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fe08c02c170 tx=0x7fe08c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=639bdfea8ea567b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:56.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.914+0000 7fe078ff9640 10 client.30038 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:56.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.914+0000 7fe078ff9640 1 -- 172.21.15.70:0/1838549611 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08c052060 con 0x555f61d702b0 2024-04-01T01:48:56.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.914+0000 7fe078ff9640 1 -- 172.21.15.70:0/1838549611 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fe08c004a50 con 0x555f61d702b0 2024-04-01T01:48:56.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.914+0000 7fe078ff9640 1 -- 172.21.15.70:0/1838549611 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fe08c005220 con 0x555f61d702b0 2024-04-01T01:48:56.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.916+0000 7fe099f9e740 10 client.30038 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:56.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.916+0000 7fe099f9e740 1 -- 172.21.15.70:0/1838549611 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=31}) -- 0x555f61de3030 con 0x555f61d702b0 2024-04-01T01:48:56.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.917+0000 7fe078ff9640 1 -- 172.21.15.70:0/1838549611 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fe08c004c50 con 0x555f61d702b0 2024-04-01T01:48:56.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.917+0000 7fe099f9e740 10 client.30038 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:56.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.917+0000 7fe099f9e740 10 client.30038 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:48:56.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.917+0000 7fe099f9e740 20 client.30038 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:56.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.917+0000 7fe099f9e740 1 --2- 172.21.15.70:0/1838549611 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x555f61de37f0 0x555f61de5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:56.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.917+0000 7fe099f9e740 4 client.30038 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:48:56.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.917+0000 7fe099f9e740 1 -- 172.21.15.70:0/1838549611 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555f61c3f6a0 con 0x555f61de37f0 2024-04-01T01:48:56.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.921+0000 7fe095158640 1 --2- 172.21.15.70:0/1838549611 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x555f61de37f0 0x555f61de5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:56.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.925+0000 7fe095158640 1 --2- 172.21.15.70:0/1838549611 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x555f61de37f0 0x555f61de5c10 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1efc931bb44a3b88 server_cookie=5598abeede1e5e7c in_seq=0 out_seq=0 2024-04-01T01:48:56.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.925+0000 7fe078ff9640 10 client.30038 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:48:56.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.925+0000 7fe078ff9640 1 -- 172.21.15.70:0/1838549611 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555f61c3f6a0 con 0x555f61de37f0 2024-04-01T01:48:56.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.925+0000 7fe078ff9640 10 client.30038 handle_command_reply: tid=0 2024-04-01T01:48:56.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.926+0000 7fe099f9e740 1 client.30038 shutdown 2024-04-01T01:48:56.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.927+0000 7fe099f9e740 1 -- 172.21.15.70:0/1838549611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d702b0 msgr2=0x555f61d77460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.927+0000 7fe099f9e740 1 --2- 172.21.15.70:0/1838549611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d702b0 0x555f61d77460 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fe08c02c170 tx=0x7fe08c009260 comp rx=0 tx=0).stop 2024-04-01T01:48:56.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.929+0000 7fe099f9e740 1 -- 172.21.15.70:0/1838549611 shutdown_connections 2024-04-01T01:48:56.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.929+0000 7fe099f9e740 1 --2- 172.21.15.70:0/1838549611 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x555f61de37f0 0x555f61de5c10 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.930+0000 7fe099f9e740 1 --2- 172.21.15.70:0/1838549611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555f61d702b0 0x555f61d77460 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.930+0000 7fe078ff9640 0 client.30038 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:48:56.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.930+0000 7fe099f9e740 1 --2- 172.21.15.70:0/1838549611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f61d6c210 0x555f61d76450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.930+0000 7fe099f9e740 1 --2- 172.21.15.70:0/1838549611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555f61d670a0 0x555f61d75f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.930+0000 7fe099f9e740 1 -- 172.21.15.70:0/1838549611 >> 172.21.15.70:0/1838549611 conn(0x555f61d50c10 msgr2=0x555f61d54700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.931+0000 7fe099f9e740 1 -- 172.21.15.70:0/1838549611 shutdown_connections 2024-04-01T01:48:56.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.931+0000 7fe099f9e740 1 -- 172.21.15.70:0/1838549611 wait complete. 2024-04-01T01:48:56.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.932+0000 7fe099f9e740 20 client.30038 trim_cache size 0 max 16384 2024-04-01T01:48:56.933 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:48:56.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.941+0000 7fe09615a640 1 -- 172.21.15.70:0/2720589651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe07c0a2670 msgr2=0x7fe07c0a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.941+0000 7fe09615a640 1 --2- 172.21.15.70:0/2720589651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe07c0a2670 0x7fe07c0a4b30 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fe08c036040 tx=0x7fe08c005fd0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.942+0000 7fe09615a640 1 -- 172.21.15.70:0/2720589651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe090100800 msgr2=0x7fe090105f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:56.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.942+0000 7fe09615a640 1 --2- 172.21.15.70:0/2720589651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe090100800 0x7fe090105f60 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fe08000e9f0 tx=0x7fe08000eeb0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.942+0000 7fe09615a640 1 -- 172.21.15.70:0/2720589651 shutdown_connections 2024-04-01T01:48:56.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.942+0000 7fe09615a640 1 --2- 172.21.15.70:0/2720589651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe07c0a2670 0x7fe07c0a4b30 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.942+0000 7fe09615a640 1 --2- 172.21.15.70:0/2720589651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0901011a0 0x7fe090102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.942+0000 7fe09615a640 1 --2- 172.21.15.70:0/2720589651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe090100800 0x7fe090105f60 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.942+0000 7fe09615a640 1 --2- 172.21.15.70:0/2720589651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0900ffea0 0x7fe090105a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:56.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.942+0000 7fe09615a640 1 -- 172.21.15.70:0/2720589651 >> 172.21.15.70:0/2720589651 conn(0x7fe0900fbac0 msgr2=0x7fe0900fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:56.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.943+0000 7fe09615a640 1 -- 172.21.15.70:0/2720589651 shutdown_connections 2024-04-01T01:48:56.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:56.943+0000 7fe09615a640 1 -- 172.21.15.70:0/2720589651 wait complete. 2024-04-01T01:48:56.970 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:56 smithi070 ceph-mon[32179]: pgmap v822: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.2 MiB/s wr, 714 op/s 2024-04-01T01:48:56.970 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:56 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2482373683' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:48:56.973 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:48:56.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:56 smithi189 ceph-mon[35753]: pgmap v822: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.2 MiB/s wr, 714 op/s 2024-04-01T01:48:56.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:56 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2482373683' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:48:57.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:56 smithi081 ceph-mon[35142]: pgmap v822: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.2 MiB/s wr, 714 op/s 2024-04-01T01:48:57.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:56 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2482373683' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:48:57.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.162+0000 7f3daeced640 1 -- 172.21.15.70:0/1554137944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3da0094c80 msgr2=0x7f3da00950a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:57.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.162+0000 7f3daeced640 1 --2- 172.21.15.70:0/1554137944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3da0094c80 0x7f3da00950a0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f3da8067420 tx=0x7f3da8097880 comp rx=0 tx=0).stop 2024-04-01T01:48:57.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.163+0000 7f3daeced640 1 -- 172.21.15.70:0/1554137944 shutdown_connections 2024-04-01T01:48:57.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.163+0000 7f3daeced640 1 --2- 172.21.15.70:0/1554137944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3da0096830 0x7f3da009d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.163+0000 7f3daeced640 1 --2- 172.21.15.70:0/1554137944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3da0095e70 0x7f3da00962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.163+0000 7f3daeced640 1 --2- 172.21.15.70:0/1554137944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3da0094c80 0x7f3da00950a0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.163+0000 7f3daeced640 1 -- 172.21.15.70:0/1554137944 >> 172.21.15.70:0/1554137944 conn(0x7f3da00901d0 msgr2=0x7f3da0092630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:57.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.164+0000 7f3daeced640 1 -- 172.21.15.70:0/1554137944 shutdown_connections 2024-04-01T01:48:57.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.164+0000 7f3daeced640 1 -- 172.21.15.70:0/1554137944 wait complete. 2024-04-01T01:48:57.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.166+0000 7f3daeced640 1 Processor -- start 2024-04-01T01:48:57.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.166+0000 7f3daeced640 1 -- start start 2024-04-01T01:48:57.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.166+0000 7f3daeced640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3da0094c80 0x7f3da012e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.166+0000 7f3daeced640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3da0095e70 0x7f3da012e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.166+0000 7f3daeced640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3da0096830 0x7f3da0135800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.166+0000 7f3daeced640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3da00a18d0 con 0x7f3da0095e70 2024-04-01T01:48:57.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.166+0000 7f3daeced640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3da00a1750 con 0x7f3da0096830 2024-04-01T01:48:57.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.166+0000 7f3daeced640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3da00a1a50 con 0x7f3da0094c80 2024-04-01T01:48:57.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.166+0000 7f3daca89640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3da0094c80 0x7f3da012e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.166+0000 7f3da7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3da0095e70 0x7f3da012e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.166+0000 7f3da7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3da0095e70 0x7f3da012e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38316/0 (socket says 172.21.15.70:38316) 2024-04-01T01:48:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.166+0000 7f3da7fff640 1 -- 172.21.15.70:0/2959263769 learned_addr learned my addr 172.21.15.70:0/2959263769 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.167+0000 7f3dad28a640 1 --2- 172.21.15.70:0/2959263769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3da0096830 0x7f3da0135800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.167+0000 7f3dad28a640 1 -- 172.21.15.70:0/2959263769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3da0094c80 msgr2=0x7f3da012e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.167+0000 7f3dad28a640 1 --2- 172.21.15.70:0/2959263769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3da0094c80 0x7f3da012e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.167+0000 7f3dad28a640 1 -- 172.21.15.70:0/2959263769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3da0095e70 msgr2=0x7f3da012e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.167+0000 7f3dad28a640 1 --2- 172.21.15.70:0/2959263769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3da0095e70 0x7f3da012e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.167+0000 7f3dad28a640 1 -- 172.21.15.70:0/2959263769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da0135e70 con 0x7f3da0096830 2024-04-01T01:48:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.167+0000 7f3da7fff640 1 --2- 172.21.15.70:0/2959263769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3da0095e70 0x7f3da012e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:48:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.167+0000 7f3daca89640 1 --2- 172.21.15.70:0/2959263769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3da0094c80 0x7f3da012e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:48:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.168+0000 7f3dad28a640 1 --2- 172.21.15.70:0/2959263769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3da0096830 0x7f3da0135800 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800b410 tx=0x7f3d9800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.168+0000 7f3da5ffb640 1 -- 172.21.15.70:0/2959263769 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d98015020 con 0x7f3da0096830 2024-04-01T01:48:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.168+0000 7f3daeced640 1 -- 172.21.15.70:0/2959263769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3da0136160 con 0x7f3da0096830 2024-04-01T01:48:57.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.168+0000 7f3daeced640 1 -- 172.21.15.70:0/2959263769 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3da0136730 con 0x7f3da0096830 2024-04-01T01:48:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.168+0000 7f3da5ffb640 1 -- 172.21.15.70:0/2959263769 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d98011ce0 con 0x7f3da0096830 2024-04-01T01:48:57.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.168+0000 7f3da5ffb640 1 -- 172.21.15.70:0/2959263769 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d98015020 con 0x7f3da0096830 2024-04-01T01:48:57.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.172+0000 7f3db19ae740 1 Processor -- start 2024-04-01T01:48:57.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.172+0000 7f3da5ffb640 1 -- 172.21.15.70:0/2959263769 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f3d98018020 con 0x7f3da0096830 2024-04-01T01:48:57.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.172+0000 7f3db19ae740 1 -- start start 2024-04-01T01:48:57.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.173+0000 7f3db19ae740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 0x5599d720e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.173+0000 7f3db19ae740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d720e8a0 0x5599d72133a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.173+0000 7f3db19ae740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d7213a10 0x5599d7215ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.173+0000 7f3db19ae740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5599d716a1b0 con 0x5599d720deb0 2024-04-01T01:48:57.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.173+0000 7f3db19ae740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5599d7112c00 con 0x5599d7213a10 2024-04-01T01:48:57.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.173+0000 7f3db19ae740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5599d7166250 con 0x5599d720e8a0 2024-04-01T01:48:57.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.173+0000 7f3da7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 0x5599d720e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:57.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.173+0000 7f3da7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 0x5599d720e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38328/0 (socket says 172.21.15.70:38328) 2024-04-01T01:48:57.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.173+0000 7f3da7fff640 1 -- 172.21.15.70:0/79432018 learned_addr learned my addr 172.21.15.70:0/79432018 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.173+0000 7f3da7fff640 1 --2- 172.21.15.70:0/79432018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d7213a10 0x5599d7215ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.173+0000 7f3da5ffb640 1 --2- 172.21.15.70:0/2959263769 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3d900a2760 0x7f3d900a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.173+0000 7f3da7fff640 1 -- 172.21.15.70:0/79432018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d720e8a0 msgr2=0x5599d72133a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:48:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.173+0000 7f3da7fff640 1 --2- 172.21.15.70:0/79432018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d720e8a0 0x5599d72133a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.173+0000 7f3da7fff640 1 -- 172.21.15.70:0/79432018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d7213a10 msgr2=0x5599d7215ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.173+0000 7f3da7fff640 1 --2- 172.21.15.70:0/79432018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d7213a10 0x5599d7215ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.173+0000 7f3da7fff640 1 -- 172.21.15.70:0/79432018 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599d71323d0 con 0x5599d720deb0 2024-04-01T01:48:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.174+0000 7f3da5ffb640 1 -- 172.21.15.70:0/2959263769 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3d980ce070 con 0x7f3da0096830 2024-04-01T01:48:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.174+0000 7f3da7fff640 1 --2- 172.21.15.70:0/79432018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 0x5599d720e2d0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0149e0 tx=0x7f3d9c009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e1fe5a7e7bfb7e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:57.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.174+0000 7f3d8effd640 1 -- 172.21.15.70:0/79432018 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c005020 con 0x5599d720deb0 2024-04-01T01:48:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.174+0000 7f3d8effd640 1 -- 172.21.15.70:0/79432018 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d9c0047a0 con 0x5599d720deb0 2024-04-01T01:48:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.175+0000 7f3dad28a640 1 --2- 172.21.15.70:0/2959263769 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3d900a2760 0x7f3d900a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.175+0000 7f3db19ae740 1 -- 172.21.15.70:0/79432018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 msgr2=0x5599d720e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:57.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.175+0000 7f3db19ae740 1 --2- 172.21.15.70:0/79432018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 0x5599d720e2d0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0149e0 tx=0x7f3d9c009710 comp rx=0 tx=0).stop 2024-04-01T01:48:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.177+0000 7f3db19ae740 1 -- 172.21.15.70:0/79432018 shutdown_connections 2024-04-01T01:48:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.177+0000 7f3db19ae740 1 --2- 172.21.15.70:0/79432018 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d7213a10 0x5599d7215ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.177+0000 7f3db19ae740 1 --2- 172.21.15.70:0/79432018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d720e8a0 0x5599d72133a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.177+0000 7f3db19ae740 1 --2- 172.21.15.70:0/79432018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 0x5599d720e2d0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.177+0000 7f3db19ae740 1 -- 172.21.15.70:0/79432018 >> 172.21.15.70:0/79432018 conn(0x5599d7200c10 msgr2=0x5599d7201010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:57.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.178+0000 7f3db19ae740 1 -- 172.21.15.70:0/79432018 shutdown_connections 2024-04-01T01:48:57.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.178+0000 7f3db19ae740 1 -- 172.21.15.70:0/79432018 wait complete. 2024-04-01T01:48:57.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.178+0000 7f3db19ae740 1 -- start start 2024-04-01T01:48:57.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.179+0000 7f3db19ae740 1 Processor -- start 2024-04-01T01:48:57.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.179+0000 7f3dad28a640 1 --2- 172.21.15.70:0/2959263769 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3d900a2760 0x7f3d900a4c20 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3d9809b5a0 tx=0x7f3d98011000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.179+0000 7f3db19ae740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d720deb0 0x5599d7218470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.179+0000 7f3db19ae740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720e8a0 0x5599d72189b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.179+0000 7f3db19ae740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d7213a10 0x5599d721f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.179+0000 7f3db19ae740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5599d7216a10 con 0x5599d720e8a0 2024-04-01T01:48:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.179+0000 7f3db19ae740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5599d7108470 con 0x5599d720deb0 2024-04-01T01:48:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.179+0000 7f3db19ae740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5599d7216b90 con 0x5599d7213a10 2024-04-01T01:48:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.179+0000 7f3db19ae740 10 client.0 _ll_register_callbacks cb 0x5599d7125cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.180+0000 7f3da7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d720deb0 0x5599d7218470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:57.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.180+0000 7f3da7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d720deb0 0x5599d7218470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41176/0 (socket says 172.21.15.70:41176) 2024-04-01T01:48:57.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.180+0000 7f3da7fff640 1 -- 172.21.15.70:0/1568999857 learned_addr learned my addr 172.21.15.70:0/1568999857 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:57.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.180+0000 7f3daca89640 1 --2- 172.21.15.70:0/1568999857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720e8a0 0x5599d72189b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:57.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.180+0000 7f3da7fff640 1 --2- 172.21.15.70:0/1568999857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d7213a10 0x5599d721f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.180+0000 7f3da7fff640 1 -- 172.21.15.70:0/1568999857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d7213a10 msgr2=0x5599d721f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.180+0000 7f3da7fff640 1 --2- 172.21.15.70:0/1568999857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d7213a10 0x5599d721f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.180+0000 7f3da7fff640 1 -- 172.21.15.70:0/1568999857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720e8a0 msgr2=0x5599d72189b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.180+0000 7f3da7fff640 1 --2- 172.21.15.70:0/1568999857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720e8a0 0x5599d72189b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.180+0000 7f3da7fff640 1 -- 172.21.15.70:0/1568999857 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599d6e7e3f0 con 0x5599d720deb0 2024-04-01T01:48:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.180+0000 7f3daca89640 1 --2- 172.21.15.70:0/1568999857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720e8a0 0x5599d72189b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.181+0000 7f3da7fff640 1 --2- 172.21.15.70:0/1568999857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d720deb0 0x5599d7218470 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c014440 tx=0x7f3d9c014f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cd98bb80962e09d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:57.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.181+0000 7f3d87fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.181+0000 7f3d87fff640 1 -- 172.21.15.70:0/1568999857 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c029040 con 0x5599d720deb0 2024-04-01T01:48:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.181+0000 7f3d87fff640 1 -- 172.21.15.70:0/1568999857 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3d9c0051c0 con 0x5599d720deb0 2024-04-01T01:48:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.181+0000 7f3db19ae740 1 -- 172.21.15.70:0/1568999857 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d9c0047a0 con 0x5599d720deb0 2024-04-01T01:48:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.182+0000 7f3d87fff640 1 -- 172.21.15.70:0/1568999857 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3d9c0053c0 con 0x5599d720deb0 2024-04-01T01:48:57.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.184+0000 7f3db19ae740 10 client.30050 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:57.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.184+0000 7f3db19ae740 1 -- 172.21.15.70:0/1568999857 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5599d6f487d0 con 0x5599d720deb0 2024-04-01T01:48:57.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.195+0000 7f3d87fff640 1 -- 172.21.15.70:0/1568999857 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d9c0047a0 con 0x5599d720deb0 2024-04-01T01:48:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.196+0000 7f3d87fff640 1 -- 172.21.15.70:0/1568999857 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f3d9c011440 con 0x5599d720deb0 2024-04-01T01:48:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.196+0000 7f3db19ae740 10 client.30050 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.196+0000 7f3db19ae740 10 client.30050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.196+0000 7f3db19ae740 20 client.30050 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:57.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.196+0000 7f3db19ae740 1 --2- 172.21.15.70:0/1568999857 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599d72240a0 0x5599d7226500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.196+0000 7f3db19ae740 4 client.30050 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.196+0000 7f3db19ae740 1 -- 172.21.15.70:0/1568999857 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599d71776f0 con 0x5599d72240a0 2024-04-01T01:48:57.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.197+0000 7f3daca89640 1 --2- 172.21.15.70:0/1568999857 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599d72240a0 0x5599d7226500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.200+0000 7f3daca89640 1 --2- 172.21.15.70:0/1568999857 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599d72240a0 0x5599d7226500 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdbcc71e727a00b9 server_cookie=8853d6a3cf140d24 in_seq=0 out_seq=0 2024-04-01T01:48:57.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.200+0000 7f3d87fff640 10 client.30050 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:57.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.201+0000 7f3d87fff640 1 -- 172.21.15.70:0/1568999857 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5599d71776f0 con 0x5599d72240a0 2024-04-01T01:48:57.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.201+0000 7f3d87fff640 10 client.30050 handle_command_reply: tid=0 2024-04-01T01:48:57.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.201+0000 7f3db19ae740 1 client.30050 shutdown 2024-04-01T01:48:57.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.202+0000 7f3db19ae740 1 -- 172.21.15.70:0/1568999857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d720deb0 msgr2=0x5599d7218470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:57.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.202+0000 7f3db19ae740 1 --2- 172.21.15.70:0/1568999857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d720deb0 0x5599d7218470 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c014440 tx=0x7f3d9c014f60 comp rx=0 tx=0).stop 2024-04-01T01:48:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.204+0000 7f3db19ae740 1 -- 172.21.15.70:0/1568999857 shutdown_connections 2024-04-01T01:48:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.204+0000 7f3db19ae740 1 --2- 172.21.15.70:0/1568999857 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599d72240a0 0x5599d7226500 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.204+0000 7f3db19ae740 1 --2- 172.21.15.70:0/1568999857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d7213a10 0x5599d721f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.204+0000 7f3db19ae740 1 --2- 172.21.15.70:0/1568999857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720e8a0 0x5599d72189b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.204+0000 7f3db19ae740 1 --2- 172.21.15.70:0/1568999857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d720deb0 0x5599d7218470 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.204+0000 7f3db19ae740 1 -- 172.21.15.70:0/1568999857 >> 172.21.15.70:0/1568999857 conn(0x5599d7200c10 msgr2=0x5599d71fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:57.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.204+0000 7f3d87fff640 0 client.30050 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:57.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.205+0000 7f3db19ae740 1 -- 172.21.15.70:0/1568999857 shutdown_connections 2024-04-01T01:48:57.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.206+0000 7f3db19ae740 1 -- 172.21.15.70:0/1568999857 wait complete. 2024-04-01T01:48:57.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.208+0000 7f3db19ae740 20 client.30050 trim_cache size 0 max 16384 2024-04-01T01:48:57.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.238+0000 7f3db19ae740 1 Processor -- start 2024-04-01T01:48:57.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.238+0000 7f3db19ae740 1 -- start start 2024-04-01T01:48:57.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.239+0000 7f3db19ae740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 0x5599d720fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.239+0000 7f3db19ae740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d720e8a0 0x5599d7210240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.240+0000 7f3db19ae740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d7213a10 0x5599d721f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.240+0000 7f3db19ae740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9400fc70 con 0x5599d720deb0 2024-04-01T01:48:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.240+0000 7f3db19ae740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3d940117b0 con 0x5599d7213a10 2024-04-01T01:48:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.240+0000 7f3db19ae740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3d94010340 con 0x5599d720e8a0 2024-04-01T01:48:57.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.240+0000 7f3da7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 0x5599d720fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:57.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.240+0000 7f3da7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 0x5599d720fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38352/0 (socket says 172.21.15.70:38352) 2024-04-01T01:48:57.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.240+0000 7f3da7fff640 1 -- 172.21.15.70:0/3830759293 learned_addr learned my addr 172.21.15.70:0/3830759293 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:57.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.241+0000 7f3daca89640 1 --2- 172.21.15.70:0/3830759293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d720e8a0 0x5599d7210240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:57.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.242+0000 7f3daca89640 1 -- 172.21.15.70:0/3830759293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d7213a10 msgr2=0x5599d721f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:57.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.242+0000 7f3daca89640 1 --2- 172.21.15.70:0/3830759293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d7213a10 0x5599d721f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.242+0000 7f3daca89640 1 -- 172.21.15.70:0/3830759293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 msgr2=0x5599d720fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:57.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.242+0000 7f3daca89640 1 --2- 172.21.15.70:0/3830759293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 0x5599d720fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.242+0000 7f3daca89640 1 -- 172.21.15.70:0/3830759293 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599d71f1e60 con 0x5599d720e8a0 2024-04-01T01:48:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.242+0000 7f3daca89640 1 --2- 172.21.15.70:0/3830759293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d720e8a0 0x5599d7210240 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x5599d720eca0 tx=0x7f3da80675f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bc2b70e09f6685b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.242+0000 7f3d8e7fc640 1 -- 172.21.15.70:0/3830759293 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da80a3040 con 0x5599d720e8a0 2024-04-01T01:48:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.242+0000 7f3d8e7fc640 1 -- 172.21.15.70:0/3830759293 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3da80b4030 con 0x5599d720e8a0 2024-04-01T01:48:57.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.243+0000 7f3d8e7fc640 1 -- 172.21.15.70:0/3830759293 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da80698a0 con 0x5599d720e8a0 2024-04-01T01:48:57.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.244+0000 7f3db19ae740 1 -- 172.21.15.70:0/3830759293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d720e8a0 msgr2=0x5599d7210240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.244+0000 7f3db19ae740 1 --2- 172.21.15.70:0/3830759293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d720e8a0 0x5599d7210240 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x5599d720eca0 tx=0x7f3da80675f0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.244+0000 7f3db19ae740 1 -- 172.21.15.70:0/3830759293 shutdown_connections 2024-04-01T01:48:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.244+0000 7f3db19ae740 1 --2- 172.21.15.70:0/3830759293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d7213a10 0x5599d721f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.244+0000 7f3db19ae740 1 --2- 172.21.15.70:0/3830759293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d720e8a0 0x5599d7210240 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.244+0000 7f3db19ae740 1 --2- 172.21.15.70:0/3830759293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 0x5599d720fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.244+0000 7f3db19ae740 1 -- 172.21.15.70:0/3830759293 >> 172.21.15.70:0/3830759293 conn(0x5599d7200c10 msgr2=0x5599d7200ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.244+0000 7f3db19ae740 1 -- 172.21.15.70:0/3830759293 shutdown_connections 2024-04-01T01:48:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.244+0000 7f3db19ae740 1 -- 172.21.15.70:0/3830759293 wait complete. 2024-04-01T01:48:57.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.245+0000 7f3db19ae740 1 -- start start 2024-04-01T01:48:57.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.245+0000 7f3db19ae740 1 Processor -- start 2024-04-01T01:48:57.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.246+0000 7f3db19ae740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 0x5599d721f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.246+0000 7f3db19ae740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d720e8a0 0x5599d721c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.246+0000 7f3db19ae740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d7213a10 0x5599d721c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.247+0000 7f3db19ae740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3d940110e0 con 0x5599d720deb0 2024-04-01T01:48:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.247+0000 7f3db19ae740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3d94010a10 con 0x5599d7213a10 2024-04-01T01:48:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.247+0000 7f3da7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d7213a10 0x5599d721c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.247+0000 7f3da7fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d7213a10 0x5599d721c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41202/0 (socket says 172.21.15.70:41202) 2024-04-01T01:48:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.247+0000 7f3da7fff640 1 -- 172.21.15.70:0/1132205253 learned_addr learned my addr 172.21.15.70:0/1132205253 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.247+0000 7f3db19ae740 1 -- 172.21.15.70:0/1132205253 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9400dd00 con 0x5599d720e8a0 2024-04-01T01:48:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.247+0000 7f3da7fff640 1 --2- 172.21.15.70:0/1132205253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 0x5599d721f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.247+0000 7f3daca89640 1 --2- 172.21.15.70:0/1132205253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d720e8a0 0x5599d721c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.247+0000 7f3da7fff640 1 -- 172.21.15.70:0/1132205253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d720e8a0 msgr2=0x5599d721c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.247+0000 7f3da7fff640 1 --2- 172.21.15.70:0/1132205253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d720e8a0 0x5599d721c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.247+0000 7f3da7fff640 1 -- 172.21.15.70:0/1132205253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 msgr2=0x5599d721f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.247+0000 7f3da7fff640 1 --2- 172.21.15.70:0/1132205253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 0x5599d721f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.247+0000 7f3da7fff640 1 -- 172.21.15.70:0/1132205253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599d721cba0 con 0x5599d7213a10 2024-04-01T01:48:57.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.248+0000 7f3db19ae740 10 client.0 _ll_register_callbacks cb 0x7f3d9c0049e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:57.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.248+0000 7f3da7fff640 1 --2- 172.21.15.70:0/1132205253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d7213a10 0x5599d721c660 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c011890 tx=0x7f3d9c004a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f58e2eb8bd1c49f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:57.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.248+0000 7f3db19ae740 1 -- 172.21.15.70:0/1132205253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3da80b4030 con 0x5599d7213a10 2024-04-01T01:48:57.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.250+0000 7f3d87fff640 10 client.30053 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:57.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.250+0000 7f3d87fff640 1 -- 172.21.15.70:0/1132205253 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c024020 con 0x5599d7213a10 2024-04-01T01:48:57.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.250+0000 7f3d87fff640 1 -- 172.21.15.70:0/1132205253 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3d9c011440 con 0x5599d7213a10 2024-04-01T01:48:57.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.250+0000 7f3d87fff640 1 -- 172.21.15.70:0/1132205253 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3d9c045050 con 0x5599d7213a10 2024-04-01T01:48:57.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.251+0000 7f3db19ae740 10 client.30053 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:57.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.251+0000 7f3db19ae740 1 -- 172.21.15.70:0/1132205253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=31}) -- 0x5599d72892d0 con 0x5599d7213a10 2024-04-01T01:48:57.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.252+0000 7f3d87fff640 1 -- 172.21.15.70:0/1132205253 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f3d9c011a30 con 0x5599d7213a10 2024-04-01T01:48:57.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.252+0000 7f3db19ae740 10 client.30053 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:57.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.253+0000 7f3db19ae740 10 client.30053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:57.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.253+0000 7f3db19ae740 20 client.30053 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:57.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.253+0000 7f3db19ae740 1 --2- 172.21.15.70:0/1132205253 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599d7289b20 0x5599d728bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:57.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.253+0000 7f3db19ae740 4 client.30053 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:48:57.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.253+0000 7f3db19ae740 1 -- 172.21.15.70:0/1132205253 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f3da80b4030 con 0x5599d7289b20 2024-04-01T01:48:57.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.253+0000 7f3daca89640 1 --2- 172.21.15.70:0/1132205253 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599d7289b20 0x5599d728bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:57.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.255+0000 7f3daca89640 1 --2- 172.21.15.70:0/1132205253 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599d7289b20 0x5599d728bf40 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20baac8b7c41a60c server_cookie=1baca880da231654 in_seq=0 out_seq=0 2024-04-01T01:48:57.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.255+0000 7f3d87fff640 10 client.30053 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:57.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.255+0000 7f3d87fff640 1 -- 172.21.15.70:0/1132205253 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f3da80b4030 con 0x5599d7289b20 2024-04-01T01:48:57.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.255+0000 7f3d87fff640 10 client.30053 handle_command_reply: tid=0 2024-04-01T01:48:57.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.256+0000 7f3db19ae740 1 client.30053 shutdown 2024-04-01T01:48:57.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.256+0000 7f3db19ae740 1 -- 172.21.15.70:0/1132205253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d7213a10 msgr2=0x5599d721c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:57.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.256+0000 7f3db19ae740 1 --2- 172.21.15.70:0/1132205253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d7213a10 0x5599d721c660 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c011890 tx=0x7f3d9c004a90 comp rx=0 tx=0).stop 2024-04-01T01:48:57.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.257+0000 7f3db19ae740 1 -- 172.21.15.70:0/1132205253 shutdown_connections 2024-04-01T01:48:57.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.257+0000 7f3db19ae740 1 --2- 172.21.15.70:0/1132205253 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599d7289b20 0x5599d728bf40 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.257+0000 7f3db19ae740 1 --2- 172.21.15.70:0/1132205253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599d7213a10 0x5599d721c660 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.257+0000 7f3db19ae740 1 --2- 172.21.15.70:0/1132205253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599d720e8a0 0x5599d721c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.257+0000 7f3d87fff640 0 client.30053 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:57.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.257+0000 7f3db19ae740 1 --2- 172.21.15.70:0/1132205253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599d720deb0 0x5599d721f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.257+0000 7f3db19ae740 1 -- 172.21.15.70:0/1132205253 >> 172.21.15.70:0/1132205253 conn(0x5599d7200c10 msgr2=0x5599d720fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:57.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.257+0000 7f3db19ae740 1 -- 172.21.15.70:0/1132205253 shutdown_connections 2024-04-01T01:48:57.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.257+0000 7f3db19ae740 1 -- 172.21.15.70:0/1132205253 wait complete. 2024-04-01T01:48:57.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.257+0000 7f3db19ae740 20 client.30053 trim_cache size 0 max 16384 2024-04-01T01:48:57.258 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:48:57.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.260+0000 7f3daeced640 1 -- 172.21.15.70:0/2959263769 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3d900a2760 msgr2=0x7f3d900a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:57.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.260+0000 7f3daeced640 1 --2- 172.21.15.70:0/2959263769 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3d900a2760 0x7f3d900a4c20 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3d9809b5a0 tx=0x7f3d98011000 comp rx=0 tx=0).stop 2024-04-01T01:48:57.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.260+0000 7f3daeced640 1 -- 172.21.15.70:0/2959263769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3da0096830 msgr2=0x7f3da0135800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:57.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.260+0000 7f3daeced640 1 --2- 172.21.15.70:0/2959263769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3da0096830 0x7f3da0135800 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800b410 tx=0x7f3d9800b8d0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.262+0000 7f3daeced640 1 -- 172.21.15.70:0/2959263769 shutdown_connections 2024-04-01T01:48:57.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.262+0000 7f3daeced640 1 --2- 172.21.15.70:0/2959263769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3da0096830 0x7f3da0135800 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.262+0000 7f3daeced640 1 --2- 172.21.15.70:0/2959263769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3da0095e70 0x7f3da012e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.262+0000 7f3daeced640 1 --2- 172.21.15.70:0/2959263769 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f3d900a2760 0x7f3d900a4c20 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.262+0000 7f3daeced640 1 --2- 172.21.15.70:0/2959263769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3da0094c80 0x7f3da012e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:57.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.262+0000 7f3daeced640 1 -- 172.21.15.70:0/2959263769 >> 172.21.15.70:0/2959263769 conn(0x7f3da00901d0 msgr2=0x7f3da009da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:57.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.264+0000 7f3daeced640 1 -- 172.21.15.70:0/2959263769 shutdown_connections 2024-04-01T01:48:57.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:57.265+0000 7f3daeced640 1 -- 172.21.15.70:0/2959263769 wait complete. 2024-04-01T01:48:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.094+0000 7fc05c78a640 1 -- 172.21.15.70:0/332742887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc054109960 msgr2=0x7fc0541101f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.094+0000 7fc05c78a640 1 --2- 172.21.15.70:0/332742887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc054109960 0x7fc0541101f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc04c009ca0 tx=0x7fc04c02c140 comp rx=0 tx=0).stop 2024-04-01T01:48:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.094+0000 7fc05c78a640 1 -- 172.21.15.70:0/332742887 shutdown_connections 2024-04-01T01:48:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.094+0000 7fc05c78a640 1 --2- 172.21.15.70:0/332742887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc054109960 0x7fc0541101f0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.094+0000 7fc05c78a640 1 --2- 172.21.15.70:0/332742887 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc054108fa0 0x7fc054109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.094+0000 7fc05c78a640 1 --2- 172.21.15.70:0/332742887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc054107db0 0x7fc0541081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.094+0000 7fc05c78a640 1 -- 172.21.15.70:0/332742887 >> 172.21.15.70:0/332742887 conn(0x7fc054076d60 msgr2=0x7fc054079180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.097+0000 7fc05c78a640 1 -- 172.21.15.70:0/332742887 shutdown_connections 2024-04-01T01:48:58.098 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.098+0000 7fc05c78a640 1 -- 172.21.15.70:0/332742887 wait complete. 2024-04-01T01:48:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.099+0000 7fc05c78a640 1 Processor -- start 2024-04-01T01:48:58.099 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.099+0000 7fc05c78a640 1 -- start start 2024-04-01T01:48:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.100+0000 7fc05c78a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc054107db0 0x7fc05419dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.100+0000 7fc05c78a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc054108fa0 0x7fc05419e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.100+0000 7fc05c78a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc054109960 0x7fc0541a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.100+0000 7fc05c78a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc054113960 con 0x7fc054109960 2024-04-01T01:48:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.100+0000 7fc05c78a640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc0541137e0 con 0x7fc054107db0 2024-04-01T01:48:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.100+0000 7fc05c78a640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc054113ae0 con 0x7fc054108fa0 2024-04-01T01:48:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.100+0000 7fc05a526640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc054107db0 0x7fc05419dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.100+0000 7fc05a526640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc054107db0 0x7fc05419dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41214/0 (socket says 172.21.15.70:41214) 2024-04-01T01:48:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.100+0000 7fc05a526640 1 -- 172.21.15.70:0/3257456409 learned_addr learned my addr 172.21.15.70:0/3257456409 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.101+0000 7fc05ad27640 1 --2- 172.21.15.70:0/3257456409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc054109960 0x7fc0541a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.101+0000 7fc059d25640 1 --2- 172.21.15.70:0/3257456409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc054108fa0 0x7fc05419e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.101+0000 7fc05a526640 1 -- 172.21.15.70:0/3257456409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc054108fa0 msgr2=0x7fc05419e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.101+0000 7fc05a526640 1 --2- 172.21.15.70:0/3257456409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc054108fa0 0x7fc05419e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.101+0000 7fc05a526640 1 -- 172.21.15.70:0/3257456409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc054109960 msgr2=0x7fc0541a5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.101+0000 7fc05a526640 1 --2- 172.21.15.70:0/3257456409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc054109960 0x7fc0541a5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.101+0000 7fc05a526640 1 -- 172.21.15.70:0/3257456409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0541a5900 con 0x7fc054107db0 2024-04-01T01:48:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.101+0000 7fc05a526640 1 --2- 172.21.15.70:0/3257456409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc054107db0 0x7fc05419dd40 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc04400e9c0 tx=0x7fc04400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.102+0000 7fc04b7fe640 1 -- 172.21.15.70:0/3257456409 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04400cd80 con 0x7fc054107db0 2024-04-01T01:48:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.102+0000 7fc04b7fe640 1 -- 172.21.15.70:0/3257456409 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc0440035f0 con 0x7fc054107db0 2024-04-01T01:48:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.102+0000 7fc04b7fe640 1 -- 172.21.15.70:0/3257456409 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc044010690 con 0x7fc054107db0 2024-04-01T01:48:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.102+0000 7fc05c78a640 1 -- 172.21.15.70:0/3257456409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0541a5bf0 con 0x7fc054107db0 2024-04-01T01:48:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.102+0000 7fc05c78a640 1 -- 172.21.15.70:0/3257456409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0541a6090 con 0x7fc054107db0 2024-04-01T01:48:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.105+0000 7fc04b7fe640 1 -- 172.21.15.70:0/3257456409 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fc0440037a0 con 0x7fc054107db0 2024-04-01T01:48:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.106+0000 7fc04b7fe640 1 --2- 172.21.15.70:0/3257456409 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc0400a25c0 0x7fc0400a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.106+0000 7fc04b7fe640 1 -- 172.21.15.70:0/3257456409 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc0440cf030 con 0x7fc054107db0 2024-04-01T01:48:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.107+0000 7fc05f368740 1 Processor -- start 2024-04-01T01:48:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.107+0000 7fc059d25640 1 --2- 172.21.15.70:0/3257456409 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc0400a25c0 0x7fc0400a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.107+0000 7fc05f368740 1 -- start start 2024-04-01T01:48:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.107+0000 7fc05f368740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 0x55e46ef33cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.107+0000 7fc05f368740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef342c0 0x55e46ef38dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.107+0000 7fc05f368740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef39430 0x55e46ef3b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.107+0000 7fc05f368740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e46ee978f0 con 0x55e46ef338d0 2024-04-01T01:48:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.108+0000 7fc05f368740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e46ee408d0 con 0x55e46ef39430 2024-04-01T01:48:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.108+0000 7fc05f368740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e46ee93990 con 0x55e46ef342c0 2024-04-01T01:48:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.108+0000 7fc05ad27640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef39430 0x55e46ef3b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.108+0000 7fc05ad27640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef39430 0x55e46ef3b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41222/0 (socket says 172.21.15.70:41222) 2024-04-01T01:48:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.108+0000 7fc05ad27640 1 -- 172.21.15.70:0/1575027805 learned_addr learned my addr 172.21.15.70:0/1575027805 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.108+0000 7fc059d25640 1 --2- 172.21.15.70:0/1575027805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef342c0 0x55e46ef38dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.108+0000 7fc05ad27640 1 -- 172.21.15.70:0/1575027805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef342c0 msgr2=0x55e46ef38dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.108+0000 7fc05ad27640 1 --2- 172.21.15.70:0/1575027805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef342c0 0x55e46ef38dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.108+0000 7fc05ad27640 1 -- 172.21.15.70:0/1575027805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 msgr2=0x55e46ef33cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.108+0000 7fc05ad27640 1 --2- 172.21.15.70:0/1575027805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 0x55e46ef33cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.108+0000 7fc05ad27640 1 -- 172.21.15.70:0/1575027805 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e46ebac210 con 0x55e46ef39430 2024-04-01T01:48:58.111 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.109+0000 7fc05ad27640 1 --2- 172.21.15.70:0/1575027805 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef39430 0x55e46ef3b8c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc04c02c750 tx=0x7fc04c005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3750db98ddbf8144 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.110+0000 7fc059d25640 1 --2- 172.21.15.70:0/3257456409 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc0400a25c0 0x7fc0400a4a80 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x55e46ef346c0 tx=0x7fc050009aa0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.110+0000 7fc048ff9640 1 -- 172.21.15.70:0/1575027805 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c043070 con 0x55e46ef39430 2024-04-01T01:48:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.110+0000 7fc048ff9640 1 -- 172.21.15.70:0/1575027805 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc04c032de0 con 0x55e46ef39430 2024-04-01T01:48:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.110+0000 7fc048ff9640 1 -- 172.21.15.70:0/1575027805 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c039040 con 0x55e46ef39430 2024-04-01T01:48:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.111+0000 7fc05f368740 1 -- 172.21.15.70:0/1575027805 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef39430 msgr2=0x55e46ef3b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.111+0000 7fc05f368740 1 --2- 172.21.15.70:0/1575027805 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef39430 0x55e46ef3b8c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc04c02c750 tx=0x7fc04c005ef0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.112 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.111+0000 7fc05f368740 1 -- 172.21.15.70:0/1575027805 shutdown_connections 2024-04-01T01:48:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.111+0000 7fc05f368740 1 --2- 172.21.15.70:0/1575027805 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef39430 0x55e46ef3b8c0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.111+0000 7fc05f368740 1 --2- 172.21.15.70:0/1575027805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef342c0 0x55e46ef38dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.111+0000 7fc05f368740 1 --2- 172.21.15.70:0/1575027805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 0x55e46ef33cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.111+0000 7fc05f368740 1 -- 172.21.15.70:0/1575027805 >> 172.21.15.70:0/1575027805 conn(0x55e46ef2ee40 msgr2=0x55e46ef2f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.111+0000 7fc05f368740 1 -- 172.21.15.70:0/1575027805 shutdown_connections 2024-04-01T01:48:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.111+0000 7fc05f368740 1 -- 172.21.15.70:0/1575027805 wait complete. 2024-04-01T01:48:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.112+0000 7fc05f368740 1 -- start start 2024-04-01T01:48:58.113 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.112+0000 7fc05f368740 1 Processor -- start 2024-04-01T01:48:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.112+0000 7fc05f368740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 0x55e46ef4ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.112+0000 7fc05f368740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef342c0 0x55e46ef4f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.112+0000 7fc05f368740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef39430 0x55e46ef56190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.112+0000 7fc05a526640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 0x55e46ef4ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.112+0000 7fc05f368740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e46ef4d390 con 0x55e46ef338d0 2024-04-01T01:48:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.112+0000 7fc05a526640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 0x55e46ef4ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38398/0 (socket says 172.21.15.70:38398) 2024-04-01T01:48:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.113+0000 7fc05a526640 1 -- 172.21.15.70:0/1661988523 learned_addr learned my addr 172.21.15.70:0/1661988523 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.113+0000 7fc05f368740 1 -- 172.21.15.70:0/1661988523 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e46ee35f80 con 0x55e46ef39430 2024-04-01T01:48:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.113+0000 7fc05f368740 1 -- 172.21.15.70:0/1661988523 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e46ef4d510 con 0x55e46ef342c0 2024-04-01T01:48:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.113+0000 7fc059d25640 1 --2- 172.21.15.70:0/1661988523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef342c0 0x55e46ef4f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.113+0000 7fc05f368740 10 client.0 _ll_register_callbacks cb 0x55e46ee8e290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.113+0000 7fc05ad27640 1 --2- 172.21.15.70:0/1661988523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef39430 0x55e46ef56190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.113+0000 7fc059d25640 1 -- 172.21.15.70:0/1661988523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef39430 msgr2=0x55e46ef56190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.113+0000 7fc059d25640 1 --2- 172.21.15.70:0/1661988523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef39430 0x55e46ef56190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.113+0000 7fc059d25640 1 -- 172.21.15.70:0/1661988523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 msgr2=0x55e46ef4ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.113+0000 7fc059d25640 1 --2- 172.21.15.70:0/1661988523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 0x55e46ef4ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.113+0000 7fc059d25640 1 -- 172.21.15.70:0/1661988523 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e46ec78620 con 0x55e46ef342c0 2024-04-01T01:48:58.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.113+0000 7fc059d25640 1 --2- 172.21.15.70:0/1661988523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef342c0 0x55e46ef4f150 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x55e46ef34d30 tx=0x7fc050005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3419053b5ce9ae70 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.113+0000 7fc03dffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:48:58.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.114+0000 7fc03dffb640 1 -- 172.21.15.70:0/1661988523 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc050028070 con 0x55e46ef342c0 2024-04-01T01:48:58.116 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.114+0000 7fc05f368740 1 -- 172.21.15.70:0/1661988523 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc04c032de0 con 0x55e46ef342c0 2024-04-01T01:48:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.114+0000 7fc03dffb640 1 -- 172.21.15.70:0/1661988523 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc050012db0 con 0x55e46ef342c0 2024-04-01T01:48:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.114+0000 7fc03dffb640 1 -- 172.21.15.70:0/1661988523 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc050030050 con 0x55e46ef342c0 2024-04-01T01:48:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.115+0000 7fc05f368740 10 client.29782 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.115+0000 7fc05f368740 1 -- 172.21.15.70:0/1661988523 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e46ef59230 con 0x55e46ef342c0 2024-04-01T01:48:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.116+0000 7fc03dffb640 1 -- 172.21.15.70:0/1661988523 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc04c032de0 con 0x55e46ef342c0 2024-04-01T01:48:58.117 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.116+0000 7fc03dffb640 1 -- 172.21.15.70:0/1661988523 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc05002b030 con 0x55e46ef342c0 2024-04-01T01:48:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.116+0000 7fc05f368740 10 client.29782 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.116+0000 7fc05f368740 10 client.29782 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.116+0000 7fc05f368740 20 client.29782 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.116+0000 7fc05f368740 1 --2- 172.21.15.70:0/1661988523 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e46ef5a990 0x55e46ef5cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.117+0000 7fc05f368740 4 client.29782 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.117+0000 7fc05f368740 1 -- 172.21.15.70:0/1661988523 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e46eeab120 con 0x55e46ef5a990 2024-04-01T01:48:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.117+0000 7fc05a526640 1 --2- 172.21.15.70:0/1661988523 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e46ef5a990 0x55e46ef5cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.118 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.118+0000 7fc05a526640 1 --2- 172.21.15.70:0/1661988523 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e46ef5a990 0x55e46ef5cd80 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49f1cbfc1b483d4a server_cookie=900d8c5a12beff52 in_seq=0 out_seq=0 2024-04-01T01:48:58.119 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.118+0000 7fc03dffb640 10 client.29782 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.119+0000 7fc03dffb640 1 -- 172.21.15.70:0/1661988523 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e46eeab120 con 0x55e46ef5a990 2024-04-01T01:48:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.119+0000 7fc03dffb640 10 client.29782 handle_command_reply: tid=0 2024-04-01T01:48:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.119+0000 7fc05f368740 1 client.29782 shutdown 2024-04-01T01:48:58.120 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.119+0000 7fc05f368740 1 -- 172.21.15.70:0/1661988523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef342c0 msgr2=0x55e46ef4f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.119+0000 7fc05f368740 1 --2- 172.21.15.70:0/1661988523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef342c0 0x55e46ef4f150 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x55e46ef34d30 tx=0x7fc050005f90 comp rx=0 tx=0).stop 2024-04-01T01:48:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.119+0000 7fc05f368740 1 -- 172.21.15.70:0/1661988523 shutdown_connections 2024-04-01T01:48:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.119+0000 7fc05f368740 1 --2- 172.21.15.70:0/1661988523 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e46ef5a990 0x55e46ef5cd80 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.119+0000 7fc05f368740 1 --2- 172.21.15.70:0/1661988523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef39430 0x55e46ef56190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.119+0000 7fc03dffb640 0 client.29782 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:58.121 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.119+0000 7fc05f368740 1 --2- 172.21.15.70:0/1661988523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef342c0 0x55e46ef4f150 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.119+0000 7fc05f368740 1 --2- 172.21.15.70:0/1661988523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 0x55e46ef4ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.119+0000 7fc05f368740 1 -- 172.21.15.70:0/1661988523 >> 172.21.15.70:0/1661988523 conn(0x55e46ef2ee40 msgr2=0x55e46ef2df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.120+0000 7fc05f368740 1 -- 172.21.15.70:0/1661988523 shutdown_connections 2024-04-01T01:48:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.120+0000 7fc05f368740 1 -- 172.21.15.70:0/1661988523 wait complete. 2024-04-01T01:48:58.122 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.120+0000 7fc05f368740 20 client.29782 trim_cache size 0 max 16384 2024-04-01T01:48:58.158 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.157+0000 7fc05f368740 1 Processor -- start 2024-04-01T01:48:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.161+0000 7fc05f368740 1 -- start start 2024-04-01T01:48:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.162+0000 7fc05f368740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 0x55e46ef62690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.162+0000 7fc05f368740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef342c0 0x55e46ef56150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.162+0000 7fc05f368740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef39430 0x55e46ef355e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.162+0000 7fc05f368740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc0541a7f10 con 0x55e46ef338d0 2024-04-01T01:48:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.162+0000 7fc05f368740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc0541a9a50 con 0x55e46ef39430 2024-04-01T01:48:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.162+0000 7fc05f368740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc0541a85e0 con 0x55e46ef342c0 2024-04-01T01:48:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.162+0000 7fc05a526640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 0x55e46ef62690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.162+0000 7fc05a526640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 0x55e46ef62690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38410/0 (socket says 172.21.15.70:38410) 2024-04-01T01:48:58.168 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.162+0000 7fc05a526640 1 -- 172.21.15.70:0/2349784879 learned_addr learned my addr 172.21.15.70:0/2349784879 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.162+0000 7fc059d25640 1 --2- 172.21.15.70:0/2349784879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef342c0 0x55e46ef56150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.162+0000 7fc05a526640 1 -- 172.21.15.70:0/2349784879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef342c0 msgr2=0x55e46ef56150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.162+0000 7fc05a526640 1 --2- 172.21.15.70:0/2349784879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef342c0 0x55e46ef56150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.163+0000 7fc05a526640 1 -- 172.21.15.70:0/2349784879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef39430 msgr2=0x55e46ef355e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:48:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.163+0000 7fc05a526640 1 --2- 172.21.15.70:0/2349784879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef39430 0x55e46ef355e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.163+0000 7fc05a526640 1 -- 172.21.15.70:0/2349784879 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e46eecc240 con 0x55e46ef338d0 2024-04-01T01:48:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.163+0000 7fc05a526640 1 --2- 172.21.15.70:0/2349784879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 0x55e46ef62690 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc044098000 tx=0x7fc04409b0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc284c63a2d0891 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.163+0000 7fc03ffff640 1 -- 172.21.15.70:0/2349784879 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0440b9070 con 0x55e46ef338d0 2024-04-01T01:48:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.164+0000 7fc03ffff640 1 -- 172.21.15.70:0/2349784879 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc044003b10 con 0x55e46ef338d0 2024-04-01T01:48:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.164+0000 7fc03ffff640 1 -- 172.21.15.70:0/2349784879 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc044097ca0 con 0x55e46ef338d0 2024-04-01T01:48:58.169 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.164+0000 7fc05f368740 1 -- 172.21.15.70:0/2349784879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 msgr2=0x55e46ef62690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.164+0000 7fc05f368740 1 --2- 172.21.15.70:0/2349784879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 0x55e46ef62690 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fc044098000 tx=0x7fc04409b0e0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.165+0000 7fc05f368740 1 -- 172.21.15.70:0/2349784879 shutdown_connections 2024-04-01T01:48:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.165+0000 7fc05f368740 1 --2- 172.21.15.70:0/2349784879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef39430 0x55e46ef355e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.165+0000 7fc05f368740 1 --2- 172.21.15.70:0/2349784879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef342c0 0x55e46ef56150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.165+0000 7fc05f368740 1 --2- 172.21.15.70:0/2349784879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef338d0 0x55e46ef62690 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.165+0000 7fc05f368740 1 -- 172.21.15.70:0/2349784879 >> 172.21.15.70:0/2349784879 conn(0x55e46ef2ee40 msgr2=0x55e46ef2f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.165+0000 7fc05f368740 1 -- 172.21.15.70:0/2349784879 shutdown_connections 2024-04-01T01:48:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.165+0000 7fc05f368740 1 -- 172.21.15.70:0/2349784879 wait complete. 2024-04-01T01:48:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.165+0000 7fc05f368740 1 -- start start 2024-04-01T01:48:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.165+0000 7fc05f368740 1 Processor -- start 2024-04-01T01:48:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.166+0000 7fc05f368740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef338d0 0x55e46ef51be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.166+0000 7fc05f368740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef342c0 0x55e46ef52120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.170 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.166+0000 7fc05f368740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef39430 0x55e46ef550f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.166+0000 7fc05f368740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc0541a9380 con 0x55e46ef342c0 2024-04-01T01:48:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.166+0000 7fc05f368740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc0541a8cb0 con 0x55e46ef338d0 2024-04-01T01:48:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.166+0000 7fc05f368740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc054073610 con 0x55e46ef39430 2024-04-01T01:48:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.166+0000 7fc05f368740 10 client.0 _ll_register_callbacks cb 0x7fc050009670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.166+0000 7fc059d25640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef342c0 0x55e46ef52120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.166+0000 7fc05ad27640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef39430 0x55e46ef550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.166+0000 7fc059d25640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef342c0 0x55e46ef52120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38418/0 (socket says 172.21.15.70:38418) 2024-04-01T01:48:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.166+0000 7fc059d25640 1 -- 172.21.15.70:0/2736224069 learned_addr learned my addr 172.21.15.70:0/2736224069 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.167+0000 7fc05ad27640 1 -- 172.21.15.70:0/2736224069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef338d0 msgr2=0x55e46ef51be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.167+0000 7fc05ad27640 1 --2- 172.21.15.70:0/2736224069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef338d0 0x55e46ef51be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.167+0000 7fc05ad27640 1 -- 172.21.15.70:0/2736224069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef342c0 msgr2=0x55e46ef52120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.167+0000 7fc05ad27640 1 --2- 172.21.15.70:0/2736224069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef342c0 0x55e46ef52120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.167+0000 7fc05ad27640 1 -- 172.21.15.70:0/2736224069 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e46ef556e0 con 0x55e46ef39430 2024-04-01T01:48:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.167+0000 7fc059d25640 1 --2- 172.21.15.70:0/2736224069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef342c0 0x55e46ef52120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:48:58.171 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.167+0000 7fc05ad27640 1 --2- 172.21.15.70:0/2736224069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef39430 0x55e46ef550f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc04c005e30 tx=0x7fc04c032e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbe55aa96db4638b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.167+0000 7fc03dffb640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:48:58.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.167+0000 7fc03dffb640 1 -- 172.21.15.70:0/2736224069 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c039030 con 0x55e46ef39430 2024-04-01T01:48:58.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.167+0000 7fc03dffb640 1 -- 172.21.15.70:0/2736224069 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7fc04c003970 con 0x55e46ef39430 2024-04-01T01:48:58.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.167+0000 7fc05f368740 1 -- 172.21.15.70:0/2736224069 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e46eea4e30 con 0x55e46ef39430 2024-04-01T01:48:58.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.168+0000 7fc03dffb640 1 -- 172.21.15.70:0/2736224069 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc04c056050 con 0x55e46ef39430 2024-04-01T01:48:58.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.168+0000 7fc05f368740 10 client.29785 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:58.172 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.168+0000 7fc05f368740 1 -- 172.21.15.70:0/2736224069 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e46efd20a0 con 0x55e46ef39430 2024-04-01T01:48:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.168+0000 7fc03dffb640 1 -- 172.21.15.70:0/2736224069 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55e46eea4e30 con 0x55e46ef39430 2024-04-01T01:48:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.168+0000 7fc03dffb640 1 -- 172.21.15.70:0/2736224069 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fc04c0044f0 con 0x55e46ef39430 2024-04-01T01:48:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.168+0000 7fc05f368740 10 client.29785 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.168+0000 7fc05f368740 10 client.29785 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.168+0000 7fc05f368740 20 client.29785 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.168+0000 7fc05f368740 1 --2- 172.21.15.70:0/2736224069 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e46ef5a990 0x55e46efd4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.168+0000 7fc05f368740 4 client.29785 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:48:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.168+0000 7fc05f368740 1 -- 172.21.15.70:0/2736224069 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e46ee9f180 con 0x55e46ef5a990 2024-04-01T01:48:58.173 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.169+0000 7fc05a526640 1 --2- 172.21.15.70:0/2736224069 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e46ef5a990 0x55e46efd4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.170+0000 7fc03dffb640 10 client.29785 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:58.174 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.170+0000 7fc05a526640 1 --2- 172.21.15.70:0/2736224069 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e46ef5a990 0x55e46efd4670 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a710f6da793be338 server_cookie=c97a2c67420ce9ed in_seq=0 out_seq=0 2024-04-01T01:48:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.207+0000 7fc03dffb640 1 -- 172.21.15.70:0/2736224069 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e46ee9f180 con 0x55e46ef5a990 2024-04-01T01:48:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.207+0000 7fc03dffb640 10 client.29785 handle_command_reply: tid=0 2024-04-01T01:48:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.207+0000 7fc05f368740 1 client.29785 shutdown 2024-04-01T01:48:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.207+0000 7fc05f368740 1 -- 172.21.15.70:0/2736224069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef39430 msgr2=0x55e46ef550f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.207+0000 7fc05f368740 1 --2- 172.21.15.70:0/2736224069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef39430 0x55e46ef550f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc04c005e30 tx=0x7fc04c032e10 comp rx=0 tx=0).stop 2024-04-01T01:48:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.209+0000 7fc05f368740 1 -- 172.21.15.70:0/2736224069 shutdown_connections 2024-04-01T01:48:58.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.209+0000 7fc05f368740 1 --2- 172.21.15.70:0/2736224069 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e46ef5a990 0x55e46efd4670 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.209+0000 7fc05f368740 1 --2- 172.21.15.70:0/2736224069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e46ef39430 0x55e46ef550f0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.209+0000 7fc05f368740 1 --2- 172.21.15.70:0/2736224069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e46ef342c0 0x55e46ef52120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.209+0000 7fc05f368740 1 --2- 172.21.15.70:0/2736224069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e46ef338d0 0x55e46ef51be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.210 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.209+0000 7fc05f368740 1 -- 172.21.15.70:0/2736224069 >> 172.21.15.70:0/2736224069 conn(0x55e46ef2ee40 msgr2=0x55e46ef355e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.211+0000 7fc03dffb640 0 client.29785 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:58.212 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.212+0000 7fc05f368740 1 -- 172.21.15.70:0/2736224069 shutdown_connections 2024-04-01T01:48:58.213 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.213+0000 7fc05f368740 1 -- 172.21.15.70:0/2736224069 wait complete. 2024-04-01T01:48:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.215+0000 7fc05f368740 20 client.29785 trim_cache size 0 max 16384 2024-04-01T01:48:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:48:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:48:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:48:58.217 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:48:58.218 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.217+0000 7fc05c78a640 1 -- 172.21.15.70:0/3257456409 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc0400a25c0 msgr2=0x7fc0400a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.217+0000 7fc05c78a640 1 --2- 172.21.15.70:0/3257456409 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc0400a25c0 0x7fc0400a4a80 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x55e46ef346c0 tx=0x7fc050009aa0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.217+0000 7fc05c78a640 1 -- 172.21.15.70:0/3257456409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc054107db0 msgr2=0x7fc05419dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.217+0000 7fc05c78a640 1 --2- 172.21.15.70:0/3257456409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc054107db0 0x7fc05419dd40 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc04400e9c0 tx=0x7fc04400ee80 comp rx=0 tx=0).stop 2024-04-01T01:48:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.217+0000 7fc05c78a640 1 -- 172.21.15.70:0/3257456409 shutdown_connections 2024-04-01T01:48:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.217+0000 7fc05c78a640 1 --2- 172.21.15.70:0/3257456409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc054109960 0x7fc0541a5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.217+0000 7fc05c78a640 1 --2- 172.21.15.70:0/3257456409 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fc0400a25c0 0x7fc0400a4a80 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.217+0000 7fc05c78a640 1 --2- 172.21.15.70:0/3257456409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc054108fa0 0x7fc05419e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.217+0000 7fc05c78a640 1 --2- 172.21.15.70:0/3257456409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc054107db0 0x7fc05419dd40 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.217+0000 7fc05c78a640 1 -- 172.21.15.70:0/3257456409 >> 172.21.15.70:0/3257456409 conn(0x7fc054076d60 msgr2=0x7fc05410c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.217+0000 7fc05c78a640 1 -- 172.21.15.70:0/3257456409 shutdown_connections 2024-04-01T01:48:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:48:58.217+0000 7fc05c78a640 1 -- 172.21.15.70:0/3257456409 wait complete. 2024-04-01T01:48:58.285 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:48:58.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:48:58.285 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:48:58.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.479+0000 7f5463fff640 1 -- 172.21.15.70:0/3839757955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f545c099170 msgr2=0x7f545c0995f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.479+0000 7f5463fff640 1 --2- 172.21.15.70:0/3839757955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f545c099170 0x7f545c0995f0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f5454009900 tx=0x7f545402bc40 comp rx=0 tx=0).stop 2024-04-01T01:48:58.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.479+0000 7f5463fff640 1 -- 172.21.15.70:0/3839757955 shutdown_connections 2024-04-01T01:48:58.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.479+0000 7f5463fff640 1 --2- 172.21.15.70:0/3839757955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f545c099b30 0x7f545c0a03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.479+0000 7f5463fff640 1 --2- 172.21.15.70:0/3839757955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f545c099170 0x7f545c0995f0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.479+0000 7f5463fff640 1 --2- 172.21.15.70:0/3839757955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f545c097f80 0x7f545c0983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.479+0000 7f5463fff640 1 -- 172.21.15.70:0/3839757955 >> 172.21.15.70:0/3839757955 conn(0x7f545c0934f0 msgr2=0x7f545c095930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.479+0000 7f5463fff640 1 -- 172.21.15.70:0/3839757955 shutdown_connections 2024-04-01T01:48:58.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.479+0000 7f5463fff640 1 -- 172.21.15.70:0/3839757955 wait complete. 2024-04-01T01:48:58.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.480+0000 7f5463fff640 1 Processor -- start 2024-04-01T01:48:58.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.480+0000 7f5463fff640 1 -- start start 2024-04-01T01:48:58.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.480+0000 7f5463fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f545c097f80 0x7f545c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.480+0000 7f5463fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f545c099170 0x7f545c0a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.480+0000 7f5463fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f545c099b30 0x7f545c142720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.480+0000 7f5463fff640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f545c00b140 con 0x7f545c099b30 2024-04-01T01:48:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.480+0000 7f5463fff640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f545c00afc0 con 0x7f545c099170 2024-04-01T01:48:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.480+0000 7f5463fff640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f545c00b2c0 con 0x7f545c097f80 2024-04-01T01:48:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.481+0000 7f5462ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f545c097f80 0x7f545c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.481+0000 7f5462ffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f545c097f80 0x7f545c0a50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46610/0 (socket says 172.21.15.70:46610) 2024-04-01T01:48:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.481+0000 7f5462ffd640 1 -- 172.21.15.70:0/898611682 learned_addr learned my addr 172.21.15.70:0/898611682 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.481+0000 7f5462ffd640 1 -- 172.21.15.70:0/898611682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f545c099170 msgr2=0x7f545c0a5630 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:48:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.481+0000 7f5462ffd640 1 --2- 172.21.15.70:0/898611682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f545c099170 0x7f545c0a5630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.481+0000 7f5462ffd640 1 -- 172.21.15.70:0/898611682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f545c099b30 msgr2=0x7f545c142720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.481+0000 7f5462ffd640 1 --2- 172.21.15.70:0/898611682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f545c099b30 0x7f545c142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.482+0000 7f5462ffd640 1 -- 172.21.15.70:0/898611682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f545c0aa770 con 0x7f545c097f80 2024-04-01T01:48:58.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.482+0000 7f5462ffd640 1 --2- 172.21.15.70:0/898611682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f545c097f80 0x7f545c0a50f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f544c00d770 tx=0x7f544c00dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.483+0000 7f545bfff640 1 -- 172.21.15.70:0/898611682 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f544c017070 con 0x7f545c097f80 2024-04-01T01:48:58.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.483+0000 7f545bfff640 1 -- 172.21.15.70:0/898611682 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f544c009c30 con 0x7f545c097f80 2024-04-01T01:48:58.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.483+0000 7f545bfff640 1 -- 172.21.15.70:0/898611682 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f544c003d00 con 0x7f545c097f80 2024-04-01T01:48:58.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.484+0000 7f5463fff640 1 -- 172.21.15.70:0/898611682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f545c153d30 con 0x7f545c097f80 2024-04-01T01:48:58.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.484+0000 7f5463fff640 1 -- 172.21.15.70:0/898611682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f545c1541a0 con 0x7f545c097f80 2024-04-01T01:48:58.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.487+0000 7f545bfff640 1 -- 172.21.15.70:0/898611682 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f544c004020 con 0x7f545c097f80 2024-04-01T01:48:58.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.487+0000 7f546c02b740 1 Processor -- start 2024-04-01T01:48:58.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.488+0000 7f546c02b740 1 -- start start 2024-04-01T01:48:58.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.488+0000 7f546c02b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd1dbf0 0x55f07dd1e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.488+0000 7f546c02b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 0x55f07dd230e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.488+0000 7f546c02b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd23750 0x55f07dd25be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.488+0000 7f546c02b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f07dc1f0b0 con 0x55f07dd1dbf0 2024-04-01T01:48:58.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.488+0000 7f546c02b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f07dc81ea0 con 0x55f07dd23750 2024-04-01T01:48:58.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.488+0000 7f546c02b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f07dc2ac00 con 0x55f07dd1e5e0 2024-04-01T01:48:58.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.488+0000 7f545bfff640 1 --2- 172.21.15.70:0/898611682 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54500a25f0 0x7f54500a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.488+0000 7f545bfff640 1 -- 172.21.15.70:0/898611682 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f544c0ce370 con 0x7f545c097f80 2024-04-01T01:48:58.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.489+0000 7f54637fe640 1 --2- 172.21.15.70:0/898611682 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54500a25f0 0x7f54500a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.489+0000 7f54637fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd1dbf0 0x55f07dd1e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.489+0000 7f54637fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd1dbf0 0x55f07dd1e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38448/0 (socket says 172.21.15.70:38448) 2024-04-01T01:48:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.489+0000 7f54637fe640 1 -- 172.21.15.70:0/617904823 learned_addr learned my addr 172.21.15.70:0/617904823 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:58.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.489+0000 7f5462ffd640 1 --2- 172.21.15.70:0/617904823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 0x55f07dd230e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.489+0000 7f54627fc640 1 --2- 172.21.15.70:0/617904823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd23750 0x55f07dd25be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.489+0000 7f54637fe640 1 -- 172.21.15.70:0/617904823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 msgr2=0x55f07dd230e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.489+0000 7f54637fe640 1 --2- 172.21.15.70:0/617904823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 0x55f07dd230e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.489+0000 7f54637fe640 1 -- 172.21.15.70:0/617904823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd23750 msgr2=0x55f07dd25be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.489+0000 7f54637fe640 1 --2- 172.21.15.70:0/617904823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd23750 0x55f07dd25be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.489+0000 7f54637fe640 1 -- 172.21.15.70:0/617904823 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f07da5e7e0 con 0x55f07dd1dbf0 2024-04-01T01:48:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.490+0000 7f54637fe640 1 --2- 172.21.15.70:0/898611682 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54500a25f0 0x7f54500a4ab0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f546404ff30 tx=0x7f5464077040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.490+0000 7f54637fe640 1 --2- 172.21.15.70:0/617904823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd1dbf0 0x55f07dd1e010 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f5464050720 tx=0x7f546407d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dab470918de8076 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.490+0000 7f54597fa640 1 -- 172.21.15.70:0/617904823 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464084030 con 0x55f07dd1dbf0 2024-04-01T01:48:58.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.490+0000 7f54597fa640 1 -- 172.21.15.70:0/617904823 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5464071040 con 0x55f07dd1dbf0 2024-04-01T01:48:58.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.491+0000 7f546c02b740 1 -- 172.21.15.70:0/617904823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd1dbf0 msgr2=0x55f07dd1e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.491+0000 7f546c02b740 1 --2- 172.21.15.70:0/617904823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd1dbf0 0x55f07dd1e010 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f5464050720 tx=0x7f546407d000 comp rx=0 tx=0).stop 2024-04-01T01:48:58.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.492+0000 7f546c02b740 1 -- 172.21.15.70:0/617904823 shutdown_connections 2024-04-01T01:48:58.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.492+0000 7f546c02b740 1 --2- 172.21.15.70:0/617904823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd23750 0x55f07dd25be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.492+0000 7f546c02b740 1 --2- 172.21.15.70:0/617904823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 0x55f07dd230e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.492+0000 7f546c02b740 1 --2- 172.21.15.70:0/617904823 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd1dbf0 0x55f07dd1e010 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.492+0000 7f546c02b740 1 -- 172.21.15.70:0/617904823 >> 172.21.15.70:0/617904823 conn(0x55f07dd19160 msgr2=0x55f07dd19560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.494+0000 7f546c02b740 1 -- 172.21.15.70:0/617904823 shutdown_connections 2024-04-01T01:48:58.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.495+0000 7f546c02b740 1 -- 172.21.15.70:0/617904823 wait complete. 2024-04-01T01:48:58.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.495+0000 7f546c02b740 1 -- start start 2024-04-01T01:48:58.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.495+0000 7f546c02b740 1 Processor -- start 2024-04-01T01:48:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.496+0000 7f546c02b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd1dbf0 0x55f07dd27f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.496+0000 7f546c02b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 0x55f07dd284d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.496+0000 7f546c02b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd23750 0x55f07dd2f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.496+0000 7f546c02b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f07dc203d0 con 0x55f07dd1dbf0 2024-04-01T01:48:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.496+0000 7f54627fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 0x55f07dd284d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.496+0000 7f54627fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 0x55f07dd284d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46624/0 (socket says 172.21.15.70:46624) 2024-04-01T01:48:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.496+0000 7f54627fc640 1 -- 172.21.15.70:0/671575608 learned_addr learned my addr 172.21.15.70:0/671575608 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.496+0000 7f546c02b740 1 -- 172.21.15.70:0/671575608 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55f07dc7df40 con 0x55f07dd23750 2024-04-01T01:48:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.496+0000 7f54637fe640 1 --2- 172.21.15.70:0/671575608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd23750 0x55f07dd2f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.497+0000 7f546c02b740 1 -- 172.21.15.70:0/671575608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f07dd266f0 con 0x55f07dd1e5e0 2024-04-01T01:48:58.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.497+0000 7f546c02b740 10 client.0 _ll_register_callbacks cb 0x55f07dc3db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:58.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.497+0000 7f5462ffd640 1 --2- 172.21.15.70:0/671575608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd1dbf0 0x55f07dd27f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.501+0000 7f54627fc640 1 -- 172.21.15.70:0/671575608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd23750 msgr2=0x55f07dd2f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.501+0000 7f54627fc640 1 --2- 172.21.15.70:0/671575608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd23750 0x55f07dd2f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.501+0000 7f54627fc640 1 -- 172.21.15.70:0/671575608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd1dbf0 msgr2=0x55f07dd27f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.501+0000 7f54627fc640 1 --2- 172.21.15.70:0/671575608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd1dbf0 0x55f07dd27f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.501+0000 7f54627fc640 1 -- 172.21.15.70:0/671575608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f07dabef70 con 0x55f07dd1e5e0 2024-04-01T01:48:58.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.502+0000 7f54627fc640 1 --2- 172.21.15.70:0/671575608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 0x55f07dd284d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5454035000 tx=0x7f5454004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6716856e52fc88b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.502+0000 7f544a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:48:58.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.502+0000 7f544a7fc640 1 -- 172.21.15.70:0/671575608 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5454043020 con 0x55f07dd1e5e0 2024-04-01T01:48:58.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.502+0000 7f546c02b740 1 -- 172.21.15.70:0/671575608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5464071040 con 0x55f07dd1e5e0 2024-04-01T01:48:58.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.502+0000 7f544a7fc640 1 -- 172.21.15.70:0/671575608 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5454005040 con 0x55f07dd1e5e0 2024-04-01T01:48:58.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.503+0000 7f544a7fc640 1 -- 172.21.15.70:0/671575608 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f545404e050 con 0x55f07dd1e5e0 2024-04-01T01:48:58.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.503+0000 7f546c02b740 10 client.29797 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:58.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.503+0000 7f546c02b740 1 -- 172.21.15.70:0/671575608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f07d956170 con 0x55f07dd1e5e0 2024-04-01T01:48:58.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.504+0000 7f544a7fc640 1 -- 172.21.15.70:0/671575608 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5454009520 con 0x55f07dd1e5e0 2024-04-01T01:48:58.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.504+0000 7f544a7fc640 1 -- 172.21.15.70:0/671575608 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f54540035f0 con 0x55f07dd1e5e0 2024-04-01T01:48:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.505+0000 7f546c02b740 10 client.29797 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.505+0000 7f546c02b740 10 client.29797 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.505+0000 7f546c02b740 20 client.29797 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.505+0000 7f546c02b740 1 --2- 172.21.15.70:0/671575608 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f07dd33ae0 0x55f07dd35fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.505+0000 7f546c02b740 4 client.29797 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.505+0000 7f546c02b740 1 -- 172.21.15.70:0/671575608 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5464071040 con 0x55f07dd33ae0 2024-04-01T01:48:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.506+0000 7f5462ffd640 1 --2- 172.21.15.70:0/671575608 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f07dd33ae0 0x55f07dd35fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.507+0000 7f5462ffd640 1 --2- 172.21.15.70:0/671575608 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f07dd33ae0 0x55f07dd35fa0 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ff9589e366dd04d server_cookie=77d8f597aa65891e in_seq=0 out_seq=0 2024-04-01T01:48:58.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.507+0000 7f544a7fc640 10 client.29797 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:58.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.508+0000 7f544a7fc640 1 -- 172.21.15.70:0/671575608 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f5464071040 con 0x55f07dd33ae0 2024-04-01T01:48:58.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.508+0000 7f544a7fc640 10 client.29797 handle_command_reply: tid=0 2024-04-01T01:48:58.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.508+0000 7f546c02b740 1 client.29797 shutdown 2024-04-01T01:48:58.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.508+0000 7f546c02b740 1 -- 172.21.15.70:0/671575608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 msgr2=0x55f07dd284d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.509+0000 7f546c02b740 1 --2- 172.21.15.70:0/671575608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 0x55f07dd284d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5454035000 tx=0x7f5454004270 comp rx=0 tx=0).stop 2024-04-01T01:48:58.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.509+0000 7f546c02b740 1 -- 172.21.15.70:0/671575608 shutdown_connections 2024-04-01T01:48:58.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.509+0000 7f546c02b740 1 --2- 172.21.15.70:0/671575608 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f07dd33ae0 0x55f07dd35fa0 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.509+0000 7f546c02b740 1 --2- 172.21.15.70:0/671575608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd23750 0x55f07dd2f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.509+0000 7f546c02b740 1 --2- 172.21.15.70:0/671575608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 0x55f07dd284d0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.509+0000 7f546c02b740 1 --2- 172.21.15.70:0/671575608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd1dbf0 0x55f07dd27f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.509+0000 7f546c02b740 1 -- 172.21.15.70:0/671575608 >> 172.21.15.70:0/671575608 conn(0x55f07dd19160 msgr2=0x55f07dd182b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.509+0000 7f544a7fc640 0 client.29797 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:58.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.509+0000 7f546c02b740 1 -- 172.21.15.70:0/671575608 shutdown_connections 2024-04-01T01:48:58.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.510+0000 7f546c02b740 1 -- 172.21.15.70:0/671575608 wait complete. 2024-04-01T01:48:58.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.510+0000 7f546c02b740 20 client.29797 trim_cache size 0 max 16384 2024-04-01T01:48:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.545+0000 7f546c02b740 1 Processor -- start 2024-04-01T01:48:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.545+0000 7f546c02b740 1 -- start start 2024-04-01T01:48:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.546+0000 7f546c02b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd1dbf0 0x55f07dd17a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.546+0000 7f546c02b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 0x55f07dd2f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.546+0000 7f546c02b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd23750 0x55f07dd17f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.546+0000 7f546c02b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f545c15ea00 con 0x55f07dd23750 2024-04-01T01:48:58.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.546+0000 7f546c02b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f545c160540 con 0x55f07dd1dbf0 2024-04-01T01:48:58.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.546+0000 7f546c02b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f545c15f0d0 con 0x55f07dd1e5e0 2024-04-01T01:48:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.549+0000 7f54627fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 0x55f07dd2f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.549+0000 7f54627fc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 0x55f07dd2f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46630/0 (socket says 172.21.15.70:46630) 2024-04-01T01:48:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.549+0000 7f54627fc640 1 -- 172.21.15.70:0/177577200 learned_addr learned my addr 172.21.15.70:0/177577200 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.549+0000 7f5462ffd640 1 --2- 172.21.15.70:0/177577200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd1dbf0 0x55f07dd17a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.549+0000 7f54627fc640 1 -- 172.21.15.70:0/177577200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd1dbf0 msgr2=0x55f07dd17a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.549+0000 7f54627fc640 1 --2- 172.21.15.70:0/177577200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd1dbf0 0x55f07dd17a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.549+0000 7f54627fc640 1 -- 172.21.15.70:0/177577200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd23750 msgr2=0x55f07dd17f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.549+0000 7f54627fc640 1 --2- 172.21.15.70:0/177577200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd23750 0x55f07dd17f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.549+0000 7f54627fc640 1 -- 172.21.15.70:0/177577200 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f07dd13440 con 0x55f07dd1e5e0 2024-04-01T01:48:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.549+0000 7f54627fc640 1 --2- 172.21.15.70:0/177577200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 0x55f07dd2f4d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5454003bb0 tx=0x7f5454009830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b0fe1ceb324f2b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.550+0000 7f5458ff9640 1 -- 172.21.15.70:0/177577200 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5454058030 con 0x55f07dd1e5e0 2024-04-01T01:48:58.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.551+0000 7f5458ff9640 1 -- 172.21.15.70:0/177577200 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5454009520 con 0x55f07dd1e5e0 2024-04-01T01:48:58.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.551+0000 7f5458ff9640 1 -- 172.21.15.70:0/177577200 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5454005040 con 0x55f07dd1e5e0 2024-04-01T01:48:58.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.552+0000 7f546c02b740 1 -- 172.21.15.70:0/177577200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 msgr2=0x55f07dd2f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.552+0000 7f546c02b740 1 --2- 172.21.15.70:0/177577200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 0x55f07dd2f4d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5454003bb0 tx=0x7f5454009830 comp rx=0 tx=0).stop 2024-04-01T01:48:58.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.557+0000 7f546c02b740 1 -- 172.21.15.70:0/177577200 shutdown_connections 2024-04-01T01:48:58.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.557+0000 7f546c02b740 1 --2- 172.21.15.70:0/177577200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd23750 0x55f07dd17f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.557+0000 7f546c02b740 1 --2- 172.21.15.70:0/177577200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd1e5e0 0x55f07dd2f4d0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.557+0000 7f546c02b740 1 --2- 172.21.15.70:0/177577200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd1dbf0 0x55f07dd17a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.557+0000 7f546c02b740 1 -- 172.21.15.70:0/177577200 >> 172.21.15.70:0/177577200 conn(0x55f07dd19160 msgr2=0x55f07dd1fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.559+0000 7f546c02b740 1 -- 172.21.15.70:0/177577200 shutdown_connections 2024-04-01T01:48:58.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.560+0000 7f546c02b740 1 -- 172.21.15.70:0/177577200 wait complete. 2024-04-01T01:48:58.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.561+0000 7f546c02b740 1 -- start start 2024-04-01T01:48:58.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.561+0000 7f546c02b740 1 Processor -- start 2024-04-01T01:48:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.561+0000 7f546c02b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd1dbf0 0x55f07dd2cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.561+0000 7f546c02b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd1e5e0 0x55f07dd2d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.561+0000 7f546c02b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd23750 0x55f07dd2e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.561+0000 7f546c02b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f545c15fe70 con 0x55f07dd1dbf0 2024-04-01T01:48:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.561+0000 7f546c02b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f545c15f7a0 con 0x55f07dd1e5e0 2024-04-01T01:48:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.561+0000 7f546c02b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f545c15c9e0 con 0x55f07dd23750 2024-04-01T01:48:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.561+0000 7f546c02b740 10 client.0 _ll_register_callbacks cb 0x7f545402cc90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.561+0000 7f54627fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd1e5e0 0x55f07dd2d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.561+0000 7f54627fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd1e5e0 0x55f07dd2d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41290/0 (socket says 172.21.15.70:41290) 2024-04-01T01:48:58.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.561+0000 7f54627fc640 1 -- 172.21.15.70:0/1331534262 learned_addr learned my addr 172.21.15.70:0/1331534262 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:58.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.562+0000 7f54627fc640 1 -- 172.21.15.70:0/1331534262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd23750 msgr2=0x55f07dd2e2b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:48:58.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.562+0000 7f54627fc640 1 --2- 172.21.15.70:0/1331534262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd23750 0x55f07dd2e2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.562+0000 7f54627fc640 1 -- 172.21.15.70:0/1331534262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd1dbf0 msgr2=0x55f07dd2cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.562+0000 7f54627fc640 1 --2- 172.21.15.70:0/1331534262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd1dbf0 0x55f07dd2cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.562+0000 7f54627fc640 1 -- 172.21.15.70:0/1331534262 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f07dd2e8a0 con 0x55f07dd1e5e0 2024-04-01T01:48:58.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.564+0000 7f54627fc640 1 --2- 172.21.15.70:0/1331534262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd1e5e0 0x55f07dd2d2a0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f5454003f60 tx=0x7f5454000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efdab716c0ed79d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.564+0000 7f544a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:48:58.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.564+0000 7f544a7fc640 1 -- 172.21.15.70:0/1331534262 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54540048d0 con 0x55f07dd1e5e0 2024-04-01T01:48:58.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.564+0000 7f546c02b740 1 -- 172.21.15.70:0/1331534262 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5454009520 con 0x55f07dd1e5e0 2024-04-01T01:48:58.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.564+0000 7f544a7fc640 1 -- 172.21.15.70:0/1331534262 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f5454004a70 con 0x55f07dd1e5e0 2024-04-01T01:48:58.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.565+0000 7f544a7fc640 1 -- 172.21.15.70:0/1331534262 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5454005040 con 0x55f07dd1e5e0 2024-04-01T01:48:58.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.566+0000 7f546c02b740 10 client.30074 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:58.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.566+0000 7f546c02b740 1 -- 172.21.15.70:0/1331534262 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f07dd9a080 con 0x55f07dd1e5e0 2024-04-01T01:48:58.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.567+0000 7f544a7fc640 1 -- 172.21.15.70:0/1331534262 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5454009520 con 0x55f07dd1e5e0 2024-04-01T01:48:58.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.567+0000 7f544a7fc640 1 -- 172.21.15.70:0/1331534262 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f545405d9d0 con 0x55f07dd1e5e0 2024-04-01T01:48:58.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.567+0000 7f546c02b740 10 client.30074 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:58.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.567+0000 7f546c02b740 10 client.30074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:58.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.567+0000 7f546c02b740 20 client.30074 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:58.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.567+0000 7f546c02b740 1 --2- 172.21.15.70:0/1331534262 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f07dd9a9d0 0x55f07dd9cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.567+0000 7f546c02b740 4 client.30074 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:48:58.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.567+0000 7f546c02b740 1 -- 172.21.15.70:0/1331534262 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f07dc8f3e0 con 0x55f07dd9a9d0 2024-04-01T01:48:58.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.569+0000 7f5462ffd640 1 --2- 172.21.15.70:0/1331534262 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f07dd9a9d0 0x55f07dd9cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.569+0000 7f544a7fc640 10 client.30074 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:58.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.570+0000 7f5462ffd640 1 --2- 172.21.15.70:0/1331534262 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f07dd9a9d0 0x55f07dd9cdf0 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8b5648c96e6b6f3 server_cookie=37ec302f86eef8e4 in_seq=0 out_seq=0 2024-04-01T01:48:58.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.571+0000 7f544a7fc640 1 -- 172.21.15.70:0/1331534262 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f07dc8f3e0 con 0x55f07dd9a9d0 2024-04-01T01:48:58.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.571+0000 7f544a7fc640 10 client.30074 handle_command_reply: tid=0 2024-04-01T01:48:58.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.573+0000 7f546c02b740 1 client.30074 shutdown 2024-04-01T01:48:58.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.573+0000 7f546c02b740 1 -- 172.21.15.70:0/1331534262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd1e5e0 msgr2=0x55f07dd2d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.573+0000 7f546c02b740 1 --2- 172.21.15.70:0/1331534262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd1e5e0 0x55f07dd2d2a0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f5454003f60 tx=0x7f5454000bc0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.573+0000 7f546c02b740 1 -- 172.21.15.70:0/1331534262 shutdown_connections 2024-04-01T01:48:58.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.573+0000 7f546c02b740 1 --2- 172.21.15.70:0/1331534262 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55f07dd9a9d0 0x55f07dd9cdf0 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.573+0000 7f546c02b740 1 --2- 172.21.15.70:0/1331534262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f07dd23750 0x55f07dd2e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.573+0000 7f546c02b740 1 --2- 172.21.15.70:0/1331534262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f07dd1e5e0 0x55f07dd2d2a0 secure :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f5454003f60 tx=0x7f5454000bc0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.573+0000 7f546c02b740 1 --2- 172.21.15.70:0/1331534262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f07dd1dbf0 0x55f07dd2cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.573+0000 7f546c02b740 1 -- 172.21.15.70:0/1331534262 >> 172.21.15.70:0/1331534262 conn(0x55f07dd19160 msgr2=0x55f07dd17a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.573+0000 7f544a7fc640 0 client.30074 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:58.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.574+0000 7f546c02b740 1 -- 172.21.15.70:0/1331534262 shutdown_connections 2024-04-01T01:48:58.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.574+0000 7f546c02b740 1 -- 172.21.15.70:0/1331534262 wait complete. 2024-04-01T01:48:58.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.575+0000 7f546c02b740 20 client.30074 trim_cache size 0 max 16384 2024-04-01T01:48:58.576 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:48:58.576 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:48:58.576 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "ba90a4c8-b63d-4268-b89f-2de4a1e342cb", 2024-04-01T01:48:58.576 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:48:58.576 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:48:58.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.578+0000 7f5463fff640 1 -- 172.21.15.70:0/898611682 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54500a25f0 msgr2=0x7f54500a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.578+0000 7f5463fff640 1 --2- 172.21.15.70:0/898611682 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54500a25f0 0x7f54500a4ab0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f546404ff30 tx=0x7f5464077040 comp rx=0 tx=0).stop 2024-04-01T01:48:58.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.578+0000 7f5463fff640 1 -- 172.21.15.70:0/898611682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f545c097f80 msgr2=0x7f545c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.578+0000 7f5463fff640 1 --2- 172.21.15.70:0/898611682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f545c097f80 0x7f545c0a50f0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f544c00d770 tx=0x7f544c00dc30 comp rx=0 tx=0).stop 2024-04-01T01:48:58.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.579+0000 7f5463fff640 1 -- 172.21.15.70:0/898611682 shutdown_connections 2024-04-01T01:48:58.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.579+0000 7f5463fff640 1 --2- 172.21.15.70:0/898611682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f545c099b30 0x7f545c142720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.579+0000 7f5463fff640 1 --2- 172.21.15.70:0/898611682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f545c099170 0x7f545c0a5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.579+0000 7f5463fff640 1 --2- 172.21.15.70:0/898611682 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f54500a25f0 0x7f54500a4ab0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.579+0000 7f5463fff640 1 --2- 172.21.15.70:0/898611682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f545c097f80 0x7f545c0a50f0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.579+0000 7f5463fff640 1 -- 172.21.15.70:0/898611682 >> 172.21.15.70:0/898611682 conn(0x7f545c0934f0 msgr2=0x7f545c09a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.579+0000 7f5463fff640 1 -- 172.21.15.70:0/898611682 shutdown_connections 2024-04-01T01:48:58.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.579+0000 7f5463fff640 1 -- 172.21.15.70:0/898611682 wait complete. 2024-04-01T01:48:58.604 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:48:58.605 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:48:58.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.791+0000 7f388756b640 1 -- 172.21.15.70:0/1086179903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38801052e0 msgr2=0x7f3880105760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.791+0000 7f388756b640 1 --2- 172.21.15.70:0/1086179903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38801052e0 0x7f3880105760 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3878009be0 tx=0x7f387802bef0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.791+0000 7f388756b640 1 -- 172.21.15.70:0/1086179903 shutdown_connections 2024-04-01T01:48:58.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.791+0000 7f388756b640 1 --2- 172.21.15.70:0/1086179903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38801052e0 0x7f3880105760 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.791+0000 7f388756b640 1 --2- 172.21.15.70:0/1086179903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3880102930 0x7f3880104da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.791+0000 7f388756b640 1 --2- 172.21.15.70:0/1086179903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38800fffc0 0x7f38801023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.791+0000 7f388756b640 1 -- 172.21.15.70:0/1086179903 >> 172.21.15.70:0/1086179903 conn(0x7f38800fbbe0 msgr2=0x7f38800fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.791+0000 7f388756b640 1 -- 172.21.15.70:0/1086179903 shutdown_connections 2024-04-01T01:48:58.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.792+0000 7f388756b640 1 -- 172.21.15.70:0/1086179903 wait complete. 2024-04-01T01:48:58.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.793+0000 7f388756b640 1 Processor -- start 2024-04-01T01:48:58.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.793+0000 7f388756b640 1 -- start start 2024-04-01T01:48:58.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.793+0000 7f388756b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38800fffc0 0x7f388010b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.793+0000 7f388756b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3880102930 0x7f388010bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.793+0000 7f388756b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38801052e0 0x7f38801ae050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.793+0000 7f388756b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3880113940 con 0x7f3880102930 2024-04-01T01:48:58.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.793+0000 7f388756b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f38801137c0 con 0x7f38800fffc0 2024-04-01T01:48:58.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.793+0000 7f388756b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3880113ac0 con 0x7f38801052e0 2024-04-01T01:48:58.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.794+0000 7f3885307640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38800fffc0 0x7f388010b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.794+0000 7f3885307640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38800fffc0 0x7f388010b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:41308/0 (socket says 172.21.15.70:41308) 2024-04-01T01:48:58.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.794+0000 7f3885307640 1 -- 172.21.15.70:0/2269620277 learned_addr learned my addr 172.21.15.70:0/2269620277 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:58.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.794+0000 7f3884b06640 1 --2- 172.21.15.70:0/2269620277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3880102930 0x7f388010bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.794+0000 7f3885307640 1 -- 172.21.15.70:0/2269620277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38801052e0 msgr2=0x7f38801ae050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.794+0000 7f3885307640 1 --2- 172.21.15.70:0/2269620277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38801052e0 0x7f38801ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.794+0000 7f3885307640 1 -- 172.21.15.70:0/2269620277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3880102930 msgr2=0x7f388010bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.794+0000 7f3885307640 1 --2- 172.21.15.70:0/2269620277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3880102930 0x7f388010bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.794+0000 7f3885307640 1 -- 172.21.15.70:0/2269620277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388010c170 con 0x7f38800fffc0 2024-04-01T01:48:58.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.799+0000 7f3885307640 1 --2- 172.21.15.70:0/2269620277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38800fffc0 0x7f388010b580 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f387c00b4a0 tx=0x7f387c00b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.804+0000 7f38767fc640 1 -- 172.21.15.70:0/2269620277 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c015020 con 0x7f38800fffc0 2024-04-01T01:48:58.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.804+0000 7f38767fc640 1 -- 172.21.15.70:0/2269620277 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f387c011ce0 con 0x7f38800fffc0 2024-04-01T01:48:58.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.804+0000 7f38767fc640 1 -- 172.21.15.70:0/2269620277 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c010e20 con 0x7f38800fffc0 2024-04-01T01:48:58.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.805+0000 7f388756b640 1 -- 172.21.15.70:0/2269620277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f388006b290 con 0x7f38800fffc0 2024-04-01T01:48:58.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.805+0000 7f388756b640 1 -- 172.21.15.70:0/2269620277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f388006b760 con 0x7f38800fffc0 2024-04-01T01:48:58.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.811+0000 7f388a220740 1 Processor -- start 2024-04-01T01:48:58.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.811+0000 7f38767fc640 1 -- 172.21.15.70:0/2269620277 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f387c018020 con 0x7f38800fffc0 2024-04-01T01:48:58.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.812+0000 7f388a220740 1 -- start start 2024-04-01T01:48:58.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.812+0000 7f388a220740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db0032b0 0x5559db0036d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.812+0000 7f388a220740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db003ca0 0x5559db0087a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.812+0000 7f388a220740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db008e10 0x5559db00b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.812+0000 7f388a220740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5559daf68180 con 0x5559db003ca0 2024-04-01T01:48:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.812+0000 7f388a220740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5559daf10b70 con 0x5559db008e10 2024-04-01T01:48:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.812+0000 7f388a220740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5559daf64220 con 0x5559db0032b0 2024-04-01T01:48:58.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.813+0000 7f38767fc640 1 --2- 172.21.15.70:0/2269620277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f386c0a26b0 0x7f386c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.813+0000 7f38767fc640 1 -- 172.21.15.70:0/2269620277 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f387c0ce200 con 0x7f38800fffc0 2024-04-01T01:48:58.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.815+0000 7f3885b08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db0032b0 0x5559db0036d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.815+0000 7f3885b08640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db0032b0 0x5559db0036d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46678/0 (socket says 172.21.15.70:46678) 2024-04-01T01:48:58.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.815+0000 7f3885b08640 1 -- 172.21.15.70:0/363260082 learned_addr learned my addr 172.21.15.70:0/363260082 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:58.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.815+0000 7f3885307640 1 --2- 172.21.15.70:0/363260082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db003ca0 0x5559db0087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.816+0000 7f3885307640 1 -- 172.21.15.70:0/363260082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db0032b0 msgr2=0x5559db0036d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.816+0000 7f3885307640 1 --2- 172.21.15.70:0/363260082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db0032b0 0x5559db0036d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.816+0000 7f3885307640 1 -- 172.21.15.70:0/363260082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db008e10 msgr2=0x5559db00b2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:48:58.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.816+0000 7f3885307640 1 --2- 172.21.15.70:0/363260082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db008e10 0x5559db00b2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.816+0000 7f3885307640 1 -- 172.21.15.70:0/363260082 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559daf247c0 con 0x5559db003ca0 2024-04-01T01:48:58.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.816+0000 7f3885307640 1 --2- 172.21.15.70:0/363260082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db003ca0 0x5559db0087a0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f387c00bda0 tx=0x7f387c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=914a3d607ce07ff8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.816+0000 7f3885b08640 1 --2- 172.21.15.70:0/2269620277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f386c0a26b0 0x7f386c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.817+0000 7f386b7fe640 1 -- 172.21.15.70:0/363260082 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c0a32a0 con 0x5559db003ca0 2024-04-01T01:48:58.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.817+0000 7f3885b08640 1 --2- 172.21.15.70:0/2269620277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f386c0a26b0 0x7f386c0a4b70 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f3878034ee0 tx=0x7f3878009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.817+0000 7f386b7fe640 1 -- 172.21.15.70:0/363260082 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f387c0cb9e0 con 0x5559db003ca0 2024-04-01T01:48:58.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.817+0000 7f386b7fe640 1 -- 172.21.15.70:0/363260082 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c09be20 con 0x5559db003ca0 2024-04-01T01:48:58.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.818+0000 7f388a220740 1 -- 172.21.15.70:0/363260082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db003ca0 msgr2=0x5559db0087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.818+0000 7f388a220740 1 --2- 172.21.15.70:0/363260082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db003ca0 0x5559db0087a0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f387c00bda0 tx=0x7f387c000bc0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.818+0000 7f388a220740 1 -- 172.21.15.70:0/363260082 shutdown_connections 2024-04-01T01:48:58.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.818+0000 7f388a220740 1 --2- 172.21.15.70:0/363260082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db008e10 0x5559db00b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.818+0000 7f388a220740 1 --2- 172.21.15.70:0/363260082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db003ca0 0x5559db0087a0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.818+0000 7f388a220740 1 --2- 172.21.15.70:0/363260082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db0032b0 0x5559db0036d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.818+0000 7f388a220740 1 -- 172.21.15.70:0/363260082 >> 172.21.15.70:0/363260082 conn(0x5559daffe820 msgr2=0x5559daffec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.818+0000 7f388a220740 1 -- 172.21.15.70:0/363260082 shutdown_connections 2024-04-01T01:48:58.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.818+0000 7f388a220740 1 -- 172.21.15.70:0/363260082 wait complete. 2024-04-01T01:48:58.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.819+0000 7f388a220740 1 -- start start 2024-04-01T01:48:58.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.819+0000 7f388a220740 1 Processor -- start 2024-04-01T01:48:58.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.819+0000 7f388a220740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db0032b0 0x5559db00d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.819+0000 7f388a220740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db003ca0 0x5559db00dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.819+0000 7f388a220740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db008e10 0x5559db014c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.819+0000 7f388a220740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5559db00bc40 con 0x5559db003ca0 2024-04-01T01:48:58.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.819+0000 7f388a220740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5559daf064a0 con 0x5559db008e10 2024-04-01T01:48:58.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.819+0000 7f388a220740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5559db00bdc0 con 0x5559db0032b0 2024-04-01T01:48:58.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.819+0000 7f388a220740 10 client.0 _ll_register_callbacks cb 0x5559daf09d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:58.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.820+0000 7f3885307640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db0032b0 0x5559db00d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.820+0000 7f3885307640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db0032b0 0x5559db00d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46686/0 (socket says 172.21.15.70:46686) 2024-04-01T01:48:58.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.820+0000 7f3885307640 1 -- 172.21.15.70:0/1598052050 learned_addr learned my addr 172.21.15.70:0/1598052050 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:58.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.820+0000 7f3885b08640 1 --2- 172.21.15.70:0/1598052050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db008e10 0x5559db014c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.820+0000 7f3884b06640 1 --2- 172.21.15.70:0/1598052050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db003ca0 0x5559db00dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.821+0000 7f3885307640 1 -- 172.21.15.70:0/1598052050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db008e10 msgr2=0x5559db014c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.821+0000 7f3885307640 1 --2- 172.21.15.70:0/1598052050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db008e10 0x5559db014c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.821+0000 7f3885307640 1 -- 172.21.15.70:0/1598052050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db003ca0 msgr2=0x5559db00dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.821+0000 7f3885307640 1 --2- 172.21.15.70:0/1598052050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db003ca0 0x5559db00dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.821+0000 7f3885307640 1 -- 172.21.15.70:0/1598052050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559dad467d0 con 0x5559db0032b0 2024-04-01T01:48:58.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.821+0000 7f3885307640 1 --2- 172.21.15.70:0/1598052050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db0032b0 0x5559db00d6d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f387c096cf0 tx=0x7f387c010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68e7489e2d058b85 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.821+0000 7f3868ff9640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:48:58.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.822+0000 7f3868ff9640 1 -- 172.21.15.70:0/1598052050 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c09bdd0 con 0x5559db0032b0 2024-04-01T01:48:58.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.822+0000 7f3868ff9640 1 -- 172.21.15.70:0/1598052050 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f387c0aeb50 con 0x5559db0032b0 2024-04-01T01:48:58.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.823+0000 7f388a220740 1 -- 172.21.15.70:0/1598052050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f387c0cb9e0 con 0x5559db0032b0 2024-04-01T01:48:58.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.823+0000 7f3868ff9640 1 -- 172.21.15.70:0/1598052050 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f387c0aed50 con 0x5559db0032b0 2024-04-01T01:48:58.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.823+0000 7f388a220740 10 client.29809 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:58.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.823+0000 7f388a220740 1 -- 172.21.15.70:0/1598052050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5559dac7c3f0 con 0x5559db0032b0 2024-04-01T01:48:58.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.824+0000 7f3868ff9640 1 -- 172.21.15.70:0/1598052050 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f387c0cb9e0 con 0x5559db0032b0 2024-04-01T01:48:58.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.824+0000 7f3868ff9640 1 -- 172.21.15.70:0/1598052050 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f387c022060 con 0x5559db0032b0 2024-04-01T01:48:58.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.824+0000 7f388a220740 10 client.29809 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:58.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.824+0000 7f388a220740 10 client.29809 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:58.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.824+0000 7f388a220740 20 client.29809 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:58.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.824+0000 7f388a220740 1 --2- 172.21.15.70:0/1598052050 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5559db019140 0x5559db01b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.824+0000 7f388a220740 4 client.29809 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:48:58.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.824+0000 7f388a220740 1 -- 172.21.15.70:0/1598052050 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559daf756c0 con 0x5559db019140 2024-04-01T01:48:58.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.824+0000 7f3884b06640 1 --2- 172.21.15.70:0/1598052050 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5559db019140 0x5559db01b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.825+0000 7f3884b06640 1 --2- 172.21.15.70:0/1598052050 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5559db019140 0x5559db01b530 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71ec1682bdc547f8 server_cookie=248252713ddc5069 in_seq=0 out_seq=0 2024-04-01T01:48:58.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.825+0000 7f3868ff9640 10 client.29809 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:58.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.827+0000 7f3868ff9640 1 -- 172.21.15.70:0/1598052050 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5559daf756c0 con 0x5559db019140 2024-04-01T01:48:58.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.827+0000 7f3868ff9640 10 client.29809 handle_command_reply: tid=0 2024-04-01T01:48:58.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.827+0000 7f388a220740 1 client.29809 shutdown 2024-04-01T01:48:58.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.827+0000 7f388a220740 1 -- 172.21.15.70:0/1598052050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db0032b0 msgr2=0x5559db00d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.827+0000 7f388a220740 1 --2- 172.21.15.70:0/1598052050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db0032b0 0x5559db00d6d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f387c096cf0 tx=0x7f387c010740 comp rx=0 tx=0).stop 2024-04-01T01:48:58.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.827+0000 7f388a220740 1 -- 172.21.15.70:0/1598052050 shutdown_connections 2024-04-01T01:48:58.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.827+0000 7f388a220740 1 --2- 172.21.15.70:0/1598052050 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5559db019140 0x5559db01b530 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.827+0000 7f388a220740 1 --2- 172.21.15.70:0/1598052050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db008e10 0x5559db014c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.827+0000 7f388a220740 1 --2- 172.21.15.70:0/1598052050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db003ca0 0x5559db00dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.827+0000 7f388a220740 1 --2- 172.21.15.70:0/1598052050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db0032b0 0x5559db00d6d0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.827+0000 7f388a220740 1 -- 172.21.15.70:0/1598052050 >> 172.21.15.70:0/1598052050 conn(0x5559daffe820 msgr2=0x5559daffdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.828+0000 7f3868ff9640 0 client.29809 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:58.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.829+0000 7f388a220740 1 -- 172.21.15.70:0/1598052050 shutdown_connections 2024-04-01T01:48:58.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.829+0000 7f388a220740 1 -- 172.21.15.70:0/1598052050 wait complete. 2024-04-01T01:48:58.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.829+0000 7f388a220740 20 client.29809 trim_cache size 0 max 16384 2024-04-01T01:48:58.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.869+0000 7f388a220740 1 Processor -- start 2024-04-01T01:48:58.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.869+0000 7f388a220740 1 -- start start 2024-04-01T01:48:58.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.869+0000 7f388a220740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db0032b0 0x5559db01b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.870+0000 7f388a220740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db003ca0 0x5559db014be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.870+0000 7f388a220740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db008e10 0x5559db00d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.870+0000 7f388a220740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f38801c1b40 con 0x5559db0032b0 2024-04-01T01:48:58.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.870+0000 7f3885307640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db0032b0 0x5559db01b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.870+0000 7f3885307640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db0032b0 0x5559db01b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38512/0 (socket says 172.21.15.70:38512) 2024-04-01T01:48:58.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.870+0000 7f3885307640 1 -- 172.21.15.70:0/4140015955 learned_addr learned my addr 172.21.15.70:0/4140015955 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:58.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.870+0000 7f3885b08640 1 --2- 172.21.15.70:0/4140015955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db008e10 0x5559db00d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.870+0000 7f388a220740 1 -- 172.21.15.70:0/4140015955 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f38801c3680 con 0x5559db003ca0 2024-04-01T01:48:58.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.870+0000 7f388a220740 1 -- 172.21.15.70:0/4140015955 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f38801c2210 con 0x5559db008e10 2024-04-01T01:48:58.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.871+0000 7f3884b06640 1 --2- 172.21.15.70:0/4140015955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db003ca0 0x5559db014be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.871+0000 7f3885307640 1 -- 172.21.15.70:0/4140015955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db008e10 msgr2=0x5559db00d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.871+0000 7f3885307640 1 --2- 172.21.15.70:0/4140015955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db008e10 0x5559db00d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.871+0000 7f3885307640 1 -- 172.21.15.70:0/4140015955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db003ca0 msgr2=0x5559db014be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:48:58.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.871+0000 7f3885307640 1 --2- 172.21.15.70:0/4140015955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db003ca0 0x5559db014be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.871+0000 7f3885307640 1 -- 172.21.15.70:0/4140015955 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559db00c840 con 0x5559db0032b0 2024-04-01T01:48:58.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.871+0000 7f3885b08640 1 --2- 172.21.15.70:0/4140015955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db008e10 0x5559db00d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:48:58.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.871+0000 7f3885307640 1 --2- 172.21.15.70:0/4140015955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db0032b0 0x5559db01b150 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f387c096790 tx=0x7f387c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f12cba4b9a5c53a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.872+0000 7f386affd640 1 -- 172.21.15.70:0/4140015955 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c099a70 con 0x5559db0032b0 2024-04-01T01:48:58.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.872+0000 7f386affd640 1 -- 172.21.15.70:0/4140015955 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f387c099750 con 0x5559db0032b0 2024-04-01T01:48:58.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.872+0000 7f386affd640 1 -- 172.21.15.70:0/4140015955 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c099ce0 con 0x5559db0032b0 2024-04-01T01:48:58.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.873+0000 7f388a220740 1 -- 172.21.15.70:0/4140015955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db0032b0 msgr2=0x5559db01b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.873+0000 7f388a220740 1 --2- 172.21.15.70:0/4140015955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db0032b0 0x5559db01b150 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f387c096790 tx=0x7f387c0046c0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.873+0000 7f388a220740 1 -- 172.21.15.70:0/4140015955 shutdown_connections 2024-04-01T01:48:58.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.873+0000 7f388a220740 1 --2- 172.21.15.70:0/4140015955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db008e10 0x5559db00d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.873+0000 7f388a220740 1 --2- 172.21.15.70:0/4140015955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db003ca0 0x5559db014be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.873+0000 7f388a220740 1 --2- 172.21.15.70:0/4140015955 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db0032b0 0x5559db01b150 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.873+0000 7f388a220740 1 -- 172.21.15.70:0/4140015955 >> 172.21.15.70:0/4140015955 conn(0x5559daffe820 msgr2=0x5559db00d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.873+0000 7f388a220740 1 -- 172.21.15.70:0/4140015955 shutdown_connections 2024-04-01T01:48:58.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.873+0000 7f388a220740 1 -- 172.21.15.70:0/4140015955 wait complete. 2024-04-01T01:48:58.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.874+0000 7f388a220740 1 -- start start 2024-04-01T01:48:58.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.874+0000 7f388a220740 1 Processor -- start 2024-04-01T01:48:58.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.874+0000 7f388a220740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db0032b0 0x5559db012400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.874+0000 7f388a220740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db003ca0 0x5559db012940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.874+0000 7f388a220740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db008e10 0x5559db013950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.874+0000 7f388a220740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f38801c2fb0 con 0x5559db008e10 2024-04-01T01:48:58.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.874+0000 7f388a220740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f38801c28e0 con 0x5559db0032b0 2024-04-01T01:48:58.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.874+0000 7f388a220740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f38800735e0 con 0x5559db003ca0 2024-04-01T01:48:58.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.874+0000 7f388a220740 10 client.0 _ll_register_callbacks cb 0x7f387c09a730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:48:58.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.874+0000 7f3885b08640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db008e10 0x5559db013950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.874+0000 7f3885b08640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db008e10 0x5559db013950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:38518/0 (socket says 172.21.15.70:38518) 2024-04-01T01:48:58.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.874+0000 7f3885b08640 1 -- 172.21.15.70:0/2751175514 learned_addr learned my addr 172.21.15.70:0/2751175514 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:48:58.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.875+0000 7f3885b08640 1 -- 172.21.15.70:0/2751175514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db003ca0 msgr2=0x5559db012940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:48:58.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.875+0000 7f3885b08640 1 --2- 172.21.15.70:0/2751175514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db003ca0 0x5559db012940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.875+0000 7f3885b08640 1 -- 172.21.15.70:0/2751175514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db0032b0 msgr2=0x5559db012400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:48:58.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.875+0000 7f3885b08640 1 --2- 172.21.15.70:0/2751175514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db0032b0 0x5559db012400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.875+0000 7f3885b08640 1 -- 172.21.15.70:0/2751175514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559dac3c0f0 con 0x5559db008e10 2024-04-01T01:48:58.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.875+0000 7f3885b08640 1 --2- 172.21.15.70:0/2751175514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db008e10 0x5559db013950 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f3878033440 tx=0x7f3878033a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9366441bf0a40c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:48:58.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.875+0000 7f3868ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:48:58.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.876+0000 7f3868ff9640 1 -- 172.21.15.70:0/2751175514 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38780344d0 con 0x5559db008e10 2024-04-01T01:48:58.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.876+0000 7f388a220740 1 -- 172.21.15.70:0/2751175514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f387c099750 con 0x5559db008e10 2024-04-01T01:48:58.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.876+0000 7f3868ff9640 1 -- 172.21.15.70:0/2751175514 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7f3878034670 con 0x5559db008e10 2024-04-01T01:48:58.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.876+0000 7f3868ff9640 1 -- 172.21.15.70:0/2751175514 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f387805d050 con 0x5559db008e10 2024-04-01T01:48:58.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.876+0000 7f388a220740 10 client.19986 fetch_fsmap learned FSMap version 31 2024-04-01T01:48:58.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.876+0000 7f388a220740 1 -- 172.21.15.70:0/2751175514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5559db07f3e0 con 0x5559db008e10 2024-04-01T01:48:58.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.877+0000 7f3868ff9640 1 -- 172.21.15.70:0/2751175514 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f387c099750 con 0x5559db008e10 2024-04-01T01:48:58.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.877+0000 7f3868ff9640 1 -- 172.21.15.70:0/2751175514 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f387803f450 con 0x5559db008e10 2024-04-01T01:48:58.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.877+0000 7f388a220740 10 client.19986 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:48:58.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.877+0000 7f388a220740 10 client.19986 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:48:58.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.877+0000 7f388a220740 20 client.19986 populate_metadata read hostname 'smithi070' 2024-04-01T01:48:58.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.877+0000 7f388a220740 1 --2- 172.21.15.70:0/2751175514 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5559db07fe70 0x5559db082330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:48:58.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.877+0000 7f388a220740 4 client.19986 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:48:58.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.877+0000 7f388a220740 1 -- 172.21.15.70:0/2751175514 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5559daf6fa10 con 0x5559db07fe70 2024-04-01T01:48:58.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.879+0000 7f3885307640 1 --2- 172.21.15.70:0/2751175514 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5559db07fe70 0x5559db082330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:48:58.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.884+0000 7f3885307640 1 --2- 172.21.15.70:0/2751175514 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5559db07fe70 0x5559db082330 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5011502701067900 server_cookie=1662c5eb03d4cf8b in_seq=0 out_seq=0 2024-04-01T01:48:58.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.884+0000 7f3868ff9640 10 client.19986 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:58.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.888+0000 7f3868ff9640 1 -- 172.21.15.70:0/2751175514 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5559daf6fa10 con 0x5559db07fe70 2024-04-01T01:48:58.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.888+0000 7f3868ff9640 10 client.19986 handle_command_reply: tid=0 2024-04-01T01:48:58.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.888+0000 7f388a220740 1 client.19986 shutdown 2024-04-01T01:48:58.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.888+0000 7f388a220740 1 -- 172.21.15.70:0/2751175514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db008e10 msgr2=0x5559db013950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.889+0000 7f388a220740 1 --2- 172.21.15.70:0/2751175514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db008e10 0x5559db013950 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f3878033440 tx=0x7f3878033a70 comp rx=0 tx=0).stop 2024-04-01T01:48:58.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.889+0000 7f388a220740 1 -- 172.21.15.70:0/2751175514 shutdown_connections 2024-04-01T01:48:58.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.889+0000 7f388a220740 1 --2- 172.21.15.70:0/2751175514 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5559db07fe70 0x5559db082330 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.889+0000 7f388a220740 1 --2- 172.21.15.70:0/2751175514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5559db008e10 0x5559db013950 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.889+0000 7f388a220740 1 --2- 172.21.15.70:0/2751175514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559db003ca0 0x5559db012940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.889+0000 7f388a220740 1 --2- 172.21.15.70:0/2751175514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5559db0032b0 0x5559db012400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.889+0000 7f388a220740 1 -- 172.21.15.70:0/2751175514 >> 172.21.15.70:0/2751175514 conn(0x5559daffe820 msgr2=0x5559daffdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.889+0000 7f3868ff9640 0 client.19986 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:48:58.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.889+0000 7f388a220740 1 -- 172.21.15.70:0/2751175514 shutdown_connections 2024-04-01T01:48:58.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.889+0000 7f388a220740 1 -- 172.21.15.70:0/2751175514 wait complete. 2024-04-01T01:48:58.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.889+0000 7f388a220740 20 client.19986 trim_cache size 0 max 16384 2024-04-01T01:48:58.891 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:48:58.891 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:48:58.891 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:48:58.891 INFO:teuthology.orchestra.run.smithi070.stdout: "ba90a4c8-b63d-4268-b89f-2de4a1e342cb": { 2024-04-01T01:48:58.891 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:48:58.891 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "ba90a4c8-b63d-4268-b89f-2de4a1e342cb", 2024-04-01T01:48:58.891 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:48:58.892 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:48:58.892 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:48:58.892 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:48:58.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.890+0000 7f388756b640 1 -- 172.21.15.70:0/2269620277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f386c0a26b0 msgr2=0x7f386c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.890+0000 7f388756b640 1 --2- 172.21.15.70:0/2269620277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f386c0a26b0 0x7f386c0a4b70 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f3878034ee0 tx=0x7f3878009500 comp rx=0 tx=0).stop 2024-04-01T01:48:58.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.890+0000 7f388756b640 1 -- 172.21.15.70:0/2269620277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38800fffc0 msgr2=0x7f388010b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:48:58.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.890+0000 7f388756b640 1 --2- 172.21.15.70:0/2269620277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38800fffc0 0x7f388010b580 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f387c00b4a0 tx=0x7f387c00b8a0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.891+0000 7f388756b640 1 -- 172.21.15.70:0/2269620277 shutdown_connections 2024-04-01T01:48:58.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.891+0000 7f388756b640 1 --2- 172.21.15.70:0/2269620277 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f386c0a26b0 0x7f386c0a4b70 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.891+0000 7f388756b640 1 --2- 172.21.15.70:0/2269620277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38801052e0 0x7f38801ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.891+0000 7f388756b640 1 --2- 172.21.15.70:0/2269620277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3880102930 0x7f388010bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.891+0000 7f388756b640 1 --2- 172.21.15.70:0/2269620277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38800fffc0 0x7f388010b580 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:48:58.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.891+0000 7f388756b640 1 -- 172.21.15.70:0/2269620277 >> 172.21.15.70:0/2269620277 conn(0x7f38800fbbe0 msgr2=0x7f3880100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:48:58.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.893+0000 7f388756b640 1 -- 172.21.15.70:0/2269620277 shutdown_connections 2024-04-01T01:48:58.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:48:58.893+0000 7f388756b640 1 -- 172.21.15.70:0/2269620277 wait complete. 2024-04-01T01:48:58.916 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:58 smithi070 ceph-mon[32179]: pgmap v823: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.9 MiB/s wr, 698 op/s 2024-04-01T01:48:58.917 INFO:tasks.cephfs.filesystem:scrub status for tag:ba90a4c8-b63d-4268-b89f-2de4a1e342cb - {'path': '/', 'tag': 'ba90a4c8-b63d-4268-b89f-2de4a1e342cb', 'options': 'recursive,force'} 2024-04-01T01:48:58.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:58 smithi189 ceph-mon[35753]: pgmap v823: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.9 MiB/s wr, 698 op/s 2024-04-01T01:48:59.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:58 smithi081 ceph-mon[35142]: pgmap v823: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.9 MiB/s wr, 698 op/s 2024-04-01T01:48:59.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:59 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:48:59.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:59 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:48:59.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:59 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:48:59.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:48:59 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:49:00.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:59 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:49:00.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:59 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:49:00.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:59 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:49:00.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:48:59 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:49:00.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:59 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:49:00.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:59 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:49:00.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:59 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:49:00.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:48:59 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:49:00.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:00 smithi189 ceph-mon[35753]: pgmap v824: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 MiB/s wr, 735 op/s 2024-04-01T01:49:01.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:00 smithi070 ceph-mon[32179]: pgmap v824: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 MiB/s wr, 735 op/s 2024-04-01T01:49:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:00 smithi081 ceph-mon[35142]: pgmap v824: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 MiB/s wr, 735 op/s 2024-04-01T01:49:01.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 855.0 s, 728.6 tps, lat 134.008 ms stddev 107.758 2024-04-01T01:49:01.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:01 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:49:02.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:01 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:49:02.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:01 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:49:02.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:02 smithi189 ceph-mon[35753]: pgmap v825: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 6.4 MiB/s wr, 788 op/s 2024-04-01T01:49:03.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:02 smithi070 ceph-mon[32179]: pgmap v825: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 6.4 MiB/s wr, 788 op/s 2024-04-01T01:49:03.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:02 smithi081 ceph-mon[35142]: pgmap v825: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 6.4 MiB/s wr, 788 op/s 2024-04-01T01:49:04.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:04 smithi189 ceph-mon[35753]: pgmap v826: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 6.3 MiB/s wr, 802 op/s 2024-04-01T01:49:05.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:04 smithi070 ceph-mon[32179]: pgmap v826: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 6.3 MiB/s wr, 802 op/s 2024-04-01T01:49:05.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:04 smithi081 ceph-mon[35142]: pgmap v826: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 6.3 MiB/s wr, 802 op/s 2024-04-01T01:49:06.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 860.0 s, 711.6 tps, lat 142.371 ms stddev 172.290 2024-04-01T01:49:06.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:06 smithi189 ceph-mon[35753]: pgmap v827: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 5.8 MiB/s wr, 769 op/s 2024-04-01T01:49:07.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:06 smithi070 ceph-mon[32179]: pgmap v827: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 5.8 MiB/s wr, 769 op/s 2024-04-01T01:49:07.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:06 smithi081 ceph-mon[35142]: pgmap v827: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 5.8 MiB/s wr, 769 op/s 2024-04-01T01:49:08.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:08 smithi189 ceph-mon[35753]: pgmap v828: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 6.4 MiB/s wr, 798 op/s 2024-04-01T01:49:09.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:08 smithi070 ceph-mon[32179]: pgmap v828: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 6.4 MiB/s wr, 798 op/s 2024-04-01T01:49:09.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:08 smithi081 ceph-mon[35142]: pgmap v828: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 6.4 MiB/s wr, 798 op/s 2024-04-01T01:49:10.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:10 smithi189 ceph-mon[35753]: pgmap v829: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 8.5 MiB/s wr, 831 op/s 2024-04-01T01:49:11.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:10 smithi070 ceph-mon[32179]: pgmap v829: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 8.5 MiB/s wr, 831 op/s 2024-04-01T01:49:11.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:10 smithi081 ceph-mon[35142]: pgmap v829: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 8.5 MiB/s wr, 831 op/s 2024-04-01T01:49:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.359+0000 7faf6946b640 1 -- 172.21.15.70:0/3479196154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf64100d00 msgr2=0x7faf641030f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.360+0000 7faf6946b640 1 --2- 172.21.15.70:0/3479196154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf64100d00 0x7faf641030f0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7faf54009900 tx=0x7faf5402bcd0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.360+0000 7faf6946b640 1 -- 172.21.15.70:0/3479196154 shutdown_connections 2024-04-01T01:49:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.360+0000 7faf6946b640 1 --2- 172.21.15.70:0/3479196154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf64106130 0x7faf641085f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.360+0000 7faf6946b640 1 --2- 172.21.15.70:0/3479196154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf64103630 0x7faf64105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.360+0000 7faf6946b640 1 --2- 172.21.15.70:0/3479196154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf64100d00 0x7faf641030f0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.360+0000 7faf6946b640 1 -- 172.21.15.70:0/3479196154 >> 172.21.15.70:0/3479196154 conn(0x7faf640fad60 msgr2=0x7faf640fd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.360+0000 7faf6946b640 1 -- 172.21.15.70:0/3479196154 shutdown_connections 2024-04-01T01:49:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.360+0000 7faf6946b640 1 -- 172.21.15.70:0/3479196154 wait complete. 2024-04-01T01:49:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.361+0000 7faf6946b640 1 Processor -- start 2024-04-01T01:49:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.362+0000 7faf6946b640 1 -- start start 2024-04-01T01:49:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.362+0000 7faf6946b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf64100d00 0x7faf64199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.362+0000 7faf6946b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf64103630 0x7faf64199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.362+0000 7faf6946b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf64106130 0x7faf641a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.362+0000 7faf6946b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faf6410aef0 con 0x7faf64100d00 2024-04-01T01:49:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.362+0000 7faf637fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf64103630 0x7faf64199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.363+0000 7faf68c6a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf64106130 0x7faf641a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.363+0000 7faf68c6a640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf64106130 0x7faf641a1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:39500/0 (socket says 172.21.15.70:39500) 2024-04-01T01:49:11.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.363+0000 7faf68c6a640 1 -- 172.21.15.70:0/4081226439 learned_addr learned my addr 172.21.15.70:0/4081226439 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:11.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.363+0000 7faf63fff640 1 --2- 172.21.15.70:0/4081226439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf64100d00 0x7faf64199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.363+0000 7faf6946b640 1 -- 172.21.15.70:0/4081226439 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faf6410ad70 con 0x7faf64106130 2024-04-01T01:49:11.364 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.363+0000 7faf6946b640 1 -- 172.21.15.70:0/4081226439 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faf6410b070 con 0x7faf64103630 2024-04-01T01:49:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.364+0000 7faf63fff640 1 -- 172.21.15.70:0/4081226439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf64103630 msgr2=0x7faf64199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.364+0000 7faf63fff640 1 --2- 172.21.15.70:0/4081226439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf64103630 0x7faf64199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.364+0000 7faf63fff640 1 -- 172.21.15.70:0/4081226439 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf64106130 msgr2=0x7faf641a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.364+0000 7faf63fff640 1 --2- 172.21.15.70:0/4081226439 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf64106130 0x7faf641a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.364+0000 7faf63fff640 1 -- 172.21.15.70:0/4081226439 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf641a1670 con 0x7faf64100d00 2024-04-01T01:49:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.364+0000 7faf68c6a640 1 --2- 172.21.15.70:0/4081226439 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf64106130 0x7faf641a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:49:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.364+0000 7faf63fff640 1 --2- 172.21.15.70:0/4081226439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf64100d00 0x7faf64199a50 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7faf54000c00 tx=0x7faf54004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.365+0000 7faf617fa640 1 -- 172.21.15.70:0/4081226439 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf54043020 con 0x7faf64100d00 2024-04-01T01:49:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.365+0000 7faf617fa640 1 -- 172.21.15.70:0/4081226439 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf5402dc20 con 0x7faf64100d00 2024-04-01T01:49:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.365+0000 7faf617fa640 1 -- 172.21.15.70:0/4081226439 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf54003ae0 con 0x7faf64100d00 2024-04-01T01:49:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.367+0000 7faf6946b640 1 -- 172.21.15.70:0/4081226439 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf641a1900 con 0x7faf64100d00 2024-04-01T01:49:11.368 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.367+0000 7faf6946b640 1 -- 172.21.15.70:0/4081226439 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf641a1dd0 con 0x7faf64100d00 2024-04-01T01:49:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.370+0000 7faf617fa640 1 -- 172.21.15.70:0/4081226439 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7faf54046020 con 0x7faf64100d00 2024-04-01T01:49:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.371+0000 7faf617fa640 1 --2- 172.21.15.70:0/4081226439 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faf4c0a25e0 0x7faf4c0a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.372+0000 7faf617fa640 1 -- 172.21.15.70:0/4081226439 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7faf5402ddd0 con 0x7faf64100d00 2024-04-01T01:49:11.373 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.373+0000 7faf6d381740 1 Processor -- start 2024-04-01T01:49:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.374+0000 7faf637fe640 1 --2- 172.21.15.70:0/4081226439 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faf4c0a25e0 0x7faf4c0a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.374+0000 7faf6d381740 1 -- start start 2024-04-01T01:49:11.374 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.374+0000 7faf6d381740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed5ae0 0x5599b1ed5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.374+0000 7faf6d381740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed64d0 0x5599b1edafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.374+0000 7faf6d381740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 0x5599b1eddad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.374+0000 7faf6d381740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5599b1e398f0 con 0x5599b1ed64d0 2024-04-01T01:49:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.374+0000 7faf6d381740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5599b1de28d0 con 0x5599b1ed5ae0 2024-04-01T01:49:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.374+0000 7faf6d381740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5599b1e35990 con 0x5599b1edb640 2024-04-01T01:49:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.374+0000 7faf68c6a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 0x5599b1eddad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.374+0000 7faf68c6a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 0x5599b1eddad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:35252/0 (socket says 172.21.15.70:35252) 2024-04-01T01:49:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.374+0000 7faf68c6a640 1 -- 172.21.15.70:0/524440893 learned_addr learned my addr 172.21.15.70:0/524440893 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:11.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.375+0000 7faf63fff640 1 --2- 172.21.15.70:0/524440893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed5ae0 0x5599b1ed5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.375+0000 7faf637fe640 1 --2- 172.21.15.70:0/4081226439 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faf4c0a25e0 0x7faf4c0a4aa0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7faf6419af60 tx=0x7faf58011440 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:11.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.375+0000 7faf637fe640 1 --2- 172.21.15.70:0/524440893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed64d0 0x5599b1edafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.376 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.375+0000 7faf68c6a640 1 -- 172.21.15.70:0/524440893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed5ae0 msgr2=0x5599b1ed5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:11.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.375+0000 7faf68c6a640 1 --2- 172.21.15.70:0/524440893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed5ae0 0x5599b1ed5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.375+0000 7faf68c6a640 1 -- 172.21.15.70:0/524440893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed64d0 msgr2=0x5599b1edafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:11.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.375+0000 7faf68c6a640 1 --2- 172.21.15.70:0/524440893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed64d0 0x5599b1edafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.375+0000 7faf68c6a640 1 -- 172.21.15.70:0/524440893 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599b1b4e210 con 0x5599b1edb640 2024-04-01T01:49:11.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.375+0000 7faf637fe640 1 --2- 172.21.15.70:0/524440893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed64d0 0x5599b1edafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:49:11.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.375+0000 7faf68c6a640 1 --2- 172.21.15.70:0/524440893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 0x5599b1eddad0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7faf5c0114b0 tx=0x7faf5c011970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2aecddc99e21bb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:11.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.376+0000 7faf4a7fc640 1 -- 172.21.15.70:0/524440893 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c01f070 con 0x5599b1edb640 2024-04-01T01:49:11.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.376+0000 7faf4a7fc640 1 -- 172.21.15.70:0/524440893 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf5c012c20 con 0x5599b1edb640 2024-04-01T01:49:11.377 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.376+0000 7faf4a7fc640 1 -- 172.21.15.70:0/524440893 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c01ad60 con 0x5599b1edb640 2024-04-01T01:49:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.377+0000 7faf6d381740 1 -- 172.21.15.70:0/524440893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 msgr2=0x5599b1eddad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.377+0000 7faf6d381740 1 --2- 172.21.15.70:0/524440893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 0x5599b1eddad0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7faf5c0114b0 tx=0x7faf5c011970 comp rx=0 tx=0).stop 2024-04-01T01:49:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.377+0000 7faf6d381740 1 -- 172.21.15.70:0/524440893 shutdown_connections 2024-04-01T01:49:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.377+0000 7faf6d381740 1 --2- 172.21.15.70:0/524440893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 0x5599b1eddad0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.377+0000 7faf6d381740 1 --2- 172.21.15.70:0/524440893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed64d0 0x5599b1edafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.377+0000 7faf6d381740 1 --2- 172.21.15.70:0/524440893 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed5ae0 0x5599b1ed5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.377+0000 7faf6d381740 1 -- 172.21.15.70:0/524440893 >> 172.21.15.70:0/524440893 conn(0x5599b1ed1050 msgr2=0x5599b1ed1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.377+0000 7faf6d381740 1 -- 172.21.15.70:0/524440893 shutdown_connections 2024-04-01T01:49:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.377+0000 7faf6d381740 1 -- 172.21.15.70:0/524440893 wait complete. 2024-04-01T01:49:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.377+0000 7faf6d381740 1 -- start start 2024-04-01T01:49:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.378+0000 7faf6d381740 1 Processor -- start 2024-04-01T01:49:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.378+0000 7faf6d381740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1ed5ae0 0x5599b1edfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.378+0000 7faf6d381740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed64d0 0x5599b1ee0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.378+0000 7faf6d381740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1edb640 0x5599b1ee7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.378+0000 7faf6d381740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5599b1ede5f0 con 0x5599b1edb640 2024-04-01T01:49:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.378+0000 7faf6d381740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5599b1dd7f80 con 0x5599b1ed64d0 2024-04-01T01:49:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.378+0000 7faf6d381740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5599b1ede770 con 0x5599b1ed5ae0 2024-04-01T01:49:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.378+0000 7faf6d381740 10 client.0 _ll_register_callbacks cb 0x5599b1e30290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.378+0000 7faf68c6a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1edb640 0x5599b1ee7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.378+0000 7faf68c6a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1edb640 0x5599b1ee7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41782/0 (socket says 172.21.15.70:41782) 2024-04-01T01:49:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.378+0000 7faf68c6a640 1 -- 172.21.15.70:0/749560783 learned_addr learned my addr 172.21.15.70:0/749560783 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.378+0000 7faf637fe640 1 --2- 172.21.15.70:0/749560783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed64d0 0x5599b1ee0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.378+0000 7faf63fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1ed5ae0 0x5599b1edfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.378+0000 7faf68c6a640 1 -- 172.21.15.70:0/749560783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1ed5ae0 msgr2=0x5599b1edfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.379+0000 7faf68c6a640 1 --2- 172.21.15.70:0/749560783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1ed5ae0 0x5599b1edfdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.379+0000 7faf68c6a640 1 -- 172.21.15.70:0/749560783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed64d0 msgr2=0x5599b1ee0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.379+0000 7faf68c6a640 1 --2- 172.21.15.70:0/749560783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed64d0 0x5599b1ee0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.379+0000 7faf68c6a640 1 -- 172.21.15.70:0/749560783 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599b1c1a620 con 0x5599b1edb640 2024-04-01T01:49:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.379+0000 7faf68c6a640 1 --2- 172.21.15.70:0/749560783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1edb640 0x5599b1ee7370 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7faf5c000f80 tx=0x7faf5c009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc44d30223e21488 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.379+0000 7faf437fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:49:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.379+0000 7faf437fe640 1 -- 172.21.15.70:0/749560783 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c029020 con 0x5599b1edb640 2024-04-01T01:49:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.379+0000 7faf437fe640 1 -- 172.21.15.70:0/749560783 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7faf5c004a40 con 0x5599b1edb640 2024-04-01T01:49:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.379+0000 7faf6d381740 1 -- 172.21.15.70:0/749560783 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf5c012c20 con 0x5599b1edb640 2024-04-01T01:49:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.379+0000 7faf437fe640 1 -- 172.21.15.70:0/749560783 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faf5c01f040 con 0x5599b1edb640 2024-04-01T01:49:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.379+0000 7faf6d381740 10 client.19995 fetch_fsmap learned FSMap version 31 2024-04-01T01:49:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.379+0000 7faf6d381740 1 -- 172.21.15.70:0/749560783 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5599b1eea2c0 con 0x5599b1edb640 2024-04-01T01:49:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.379+0000 7faf437fe640 1 -- 172.21.15.70:0/749560783 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf5c012c20 con 0x5599b1edb640 2024-04-01T01:49:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.380+0000 7faf437fe640 1 -- 172.21.15.70:0/749560783 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7faf5c036020 con 0x5599b1edb640 2024-04-01T01:49:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.380+0000 7faf6d381740 10 client.19995 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:49:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.380+0000 7faf6d381740 10 client.19995 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:49:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.380+0000 7faf6d381740 20 client.19995 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.380+0000 7faf6d381740 1 --2- 172.21.15.70:0/749560783 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599b1eeb6c0 0x5599b1eedab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.380+0000 7faf6d381740 4 client.19995 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:49:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.380+0000 7faf6d381740 1 -- 172.21.15.70:0/749560783 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599b1e4d120 con 0x5599b1eeb6c0 2024-04-01T01:49:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.380+0000 7faf63fff640 1 --2- 172.21.15.70:0/749560783 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599b1eeb6c0 0x5599b1eedab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.380+0000 7faf437fe640 10 client.19995 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.380+0000 7faf63fff640 1 --2- 172.21.15.70:0/749560783 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599b1eeb6c0 0x5599b1eedab0 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=197ddfa1c7d2e6a1 server_cookie=b2f6b27c78a81240 in_seq=0 out_seq=0 2024-04-01T01:49:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.381+0000 7faf437fe640 1 -- 172.21.15.70:0/749560783 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5599b1e4d120 con 0x5599b1eeb6c0 2024-04-01T01:49:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.381+0000 7faf437fe640 10 client.19995 handle_command_reply: tid=0 2024-04-01T01:49:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.381+0000 7faf6d381740 1 client.19995 shutdown 2024-04-01T01:49:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.381+0000 7faf6d381740 1 -- 172.21.15.70:0/749560783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1edb640 msgr2=0x5599b1ee7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.381+0000 7faf6d381740 1 --2- 172.21.15.70:0/749560783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1edb640 0x5599b1ee7370 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7faf5c000f80 tx=0x7faf5c009de0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.381+0000 7faf6d381740 1 -- 172.21.15.70:0/749560783 shutdown_connections 2024-04-01T01:49:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.381+0000 7faf6d381740 1 --2- 172.21.15.70:0/749560783 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599b1eeb6c0 0x5599b1eedab0 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.381+0000 7faf437fe640 0 client.19995 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.381+0000 7faf6d381740 1 --2- 172.21.15.70:0/749560783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1edb640 0x5599b1ee7370 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.381+0000 7faf6d381740 1 --2- 172.21.15.70:0/749560783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed64d0 0x5599b1ee0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.381+0000 7faf6d381740 1 --2- 172.21.15.70:0/749560783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1ed5ae0 0x5599b1edfdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.381+0000 7faf6d381740 1 -- 172.21.15.70:0/749560783 >> 172.21.15.70:0/749560783 conn(0x5599b1ed1050 msgr2=0x5599b1ed0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.382+0000 7faf6d381740 1 -- 172.21.15.70:0/749560783 shutdown_connections 2024-04-01T01:49:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.382+0000 7faf6d381740 1 -- 172.21.15.70:0/749560783 wait complete. 2024-04-01T01:49:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.382+0000 7faf6d381740 20 client.19995 trim_cache size 0 max 16384 2024-04-01T01:49:11.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.404+0000 7faf6d381740 1 Processor -- start 2024-04-01T01:49:11.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.404+0000 7faf6d381740 1 -- start start 2024-04-01T01:49:11.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.405+0000 7faf6d381740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed5ae0 0x5599b1ef3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.405+0000 7faf6d381740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed64d0 0x5599b1ee7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.405+0000 7faf6d381740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 0x5599b1edfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.405+0000 7faf6d381740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faf641ac530 con 0x5599b1ed5ae0 2024-04-01T01:49:11.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.405+0000 7faf6d381740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faf641ae070 con 0x5599b1ed64d0 2024-04-01T01:49:11.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.405+0000 7faf6d381740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faf641acc00 con 0x5599b1edb640 2024-04-01T01:49:11.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.405+0000 7faf63fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed5ae0 0x5599b1ef3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.405+0000 7faf63fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed5ae0 0x5599b1ef3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41788/0 (socket says 172.21.15.70:41788) 2024-04-01T01:49:11.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.405+0000 7faf63fff640 1 -- 172.21.15.70:0/2454634990 learned_addr learned my addr 172.21.15.70:0/2454634990 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:11.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.405+0000 7faf637fe640 1 --2- 172.21.15.70:0/2454634990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed64d0 0x5599b1ee7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.405+0000 7faf68c6a640 1 --2- 172.21.15.70:0/2454634990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 0x5599b1edfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.405+0000 7faf63fff640 1 -- 172.21.15.70:0/2454634990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 msgr2=0x5599b1edfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.405+0000 7faf63fff640 1 --2- 172.21.15.70:0/2454634990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 0x5599b1edfdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.405+0000 7faf63fff640 1 -- 172.21.15.70:0/2454634990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed64d0 msgr2=0x5599b1ee7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.405+0000 7faf63fff640 1 --2- 172.21.15.70:0/2454634990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed64d0 0x5599b1ee7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.405+0000 7faf63fff640 1 -- 172.21.15.70:0/2454634990 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599b1ed1dc0 con 0x5599b1ed5ae0 2024-04-01T01:49:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.406+0000 7faf637fe640 1 --2- 172.21.15.70:0/2454634990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed64d0 0x5599b1ee7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:49:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.406+0000 7faf63fff640 1 --2- 172.21.15.70:0/2454634990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed5ae0 0x5599b1ef3380 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7faf540b8d10 tx=0x7faf540f2000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e42afe50f494b298 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.406+0000 7faf49ffb640 1 -- 172.21.15.70:0/2454634990 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf540f4060 con 0x5599b1ed5ae0 2024-04-01T01:49:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.406+0000 7faf49ffb640 1 -- 172.21.15.70:0/2454634990 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf540c0490 con 0x5599b1ed5ae0 2024-04-01T01:49:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.406+0000 7faf49ffb640 1 -- 172.21.15.70:0/2454634990 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf540dc0b0 con 0x5599b1ed5ae0 2024-04-01T01:49:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.407+0000 7faf6d381740 1 -- 172.21.15.70:0/2454634990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed5ae0 msgr2=0x5599b1ef3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.407+0000 7faf6d381740 1 --2- 172.21.15.70:0/2454634990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed5ae0 0x5599b1ef3380 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7faf540b8d10 tx=0x7faf540f2000 comp rx=0 tx=0).stop 2024-04-01T01:49:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.407+0000 7faf6d381740 1 -- 172.21.15.70:0/2454634990 shutdown_connections 2024-04-01T01:49:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.407+0000 7faf6d381740 1 --2- 172.21.15.70:0/2454634990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 0x5599b1edfdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.407+0000 7faf6d381740 1 --2- 172.21.15.70:0/2454634990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed64d0 0x5599b1ee7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.407+0000 7faf6d381740 1 --2- 172.21.15.70:0/2454634990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed5ae0 0x5599b1ef3380 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.407+0000 7faf6d381740 1 -- 172.21.15.70:0/2454634990 >> 172.21.15.70:0/2454634990 conn(0x5599b1ed1050 msgr2=0x5599b1ed1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.408+0000 7faf6d381740 1 -- 172.21.15.70:0/2454634990 shutdown_connections 2024-04-01T01:49:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.408+0000 7faf6d381740 1 -- 172.21.15.70:0/2454634990 wait complete. 2024-04-01T01:49:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.409+0000 7faf6d381740 1 -- start start 2024-04-01T01:49:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.409+0000 7faf6d381740 1 Processor -- start 2024-04-01T01:49:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.409+0000 7faf6d381740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed5ae0 0x5599b1ee4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.410+0000 7faf6d381740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed64d0 0x5599b1ee5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.410+0000 7faf6d381740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 0x5599b1ee6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.410+0000 7faf6d381740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faf641ad9a0 con 0x5599b1ed5ae0 2024-04-01T01:49:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.410+0000 7faf6d381740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faf641ad2d0 con 0x5599b1ed64d0 2024-04-01T01:49:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.410+0000 7faf6d381740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faf641aa510 con 0x5599b1edb640 2024-04-01T01:49:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.410+0000 7faf6d381740 10 client.0 _ll_register_callbacks cb 0x7faf5c01d070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.410+0000 7faf68c6a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 0x5599b1ee6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.410+0000 7faf68c6a640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 0x5599b1ee6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:35290/0 (socket says 172.21.15.70:35290) 2024-04-01T01:49:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.410+0000 7faf68c6a640 1 -- 172.21.15.70:0/3472713015 learned_addr learned my addr 172.21.15.70:0/3472713015 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.410+0000 7faf63fff640 1 --2- 172.21.15.70:0/3472713015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed5ae0 0x5599b1ee4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.410+0000 7faf637fe640 1 --2- 172.21.15.70:0/3472713015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed64d0 0x5599b1ee5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.410+0000 7faf68c6a640 1 -- 172.21.15.70:0/3472713015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed64d0 msgr2=0x5599b1ee5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.410+0000 7faf68c6a640 1 --2- 172.21.15.70:0/3472713015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed64d0 0x5599b1ee5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.410+0000 7faf68c6a640 1 -- 172.21.15.70:0/3472713015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed5ae0 msgr2=0x5599b1ee4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.410+0000 7faf68c6a640 1 --2- 172.21.15.70:0/3472713015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed5ae0 0x5599b1ee4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.410+0000 7faf68c6a640 1 -- 172.21.15.70:0/3472713015 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599b1ee6a40 con 0x5599b1edb640 2024-04-01T01:49:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.411+0000 7faf63fff640 1 --2- 172.21.15.70:0/3472713015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed5ae0 0x5599b1ee4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:49:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.411+0000 7faf68c6a640 1 --2- 172.21.15.70:0/3472713015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 0x5599b1ee6450 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7faf5c0052d0 tx=0x7faf5c009af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f6bbce616d0ab24 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.411+0000 7faf437fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:49:11.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.413+0000 7faf437fe640 1 -- 172.21.15.70:0/3472713015 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5c03b070 con 0x5599b1edb640 2024-04-01T01:49:11.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.413+0000 7faf437fe640 1 -- 172.21.15.70:0/3472713015 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9208+0+0 (secure 0 0 0) 0x7faf5c004f60 con 0x5599b1edb640 2024-04-01T01:49:11.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.413+0000 7faf6d381740 1 -- 172.21.15.70:0/3472713015 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf540c0490 con 0x5599b1edb640 2024-04-01T01:49:11.414 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.414+0000 7faf437fe640 1 -- 172.21.15.70:0/3472713015 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7faf5c044050 con 0x5599b1edb640 2024-04-01T01:49:11.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.414+0000 7faf6d381740 10 client.29827 fetch_fsmap learned FSMap version 31 2024-04-01T01:49:11.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.414+0000 7faf6d381740 1 -- 172.21.15.70:0/3472713015 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5599b1f5a5c0 con 0x5599b1edb640 2024-04-01T01:49:11.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.415+0000 7faf437fe640 1 -- 172.21.15.70:0/3472713015 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faf5c01ace0 con 0x5599b1edb640 2024-04-01T01:49:11.416 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.416+0000 7faf437fe640 1 -- 172.21.15.70:0/3472713015 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7faf5c036040 con 0x5599b1edb640 2024-04-01T01:49:11.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.416+0000 7faf6d381740 10 client.29827 fetch_fsmap finished waiting for FSMap version 31 2024-04-01T01:49:11.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.416+0000 7faf6d381740 10 client.29827 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:49:11.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.416+0000 7faf6d381740 20 client.29827 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:11.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.416+0000 7faf6d381740 1 --2- 172.21.15.70:0/3472713015 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599b1f5af90 0x5599b1f5d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:11.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.416+0000 7faf6d381740 4 client.29827 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:49:11.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.416+0000 7faf6d381740 1 -- 172.21.15.70:0/3472713015 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7faf540c0490 con 0x5599b1f5af90 2024-04-01T01:49:11.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.416+0000 7faf63fff640 1 --2- 172.21.15.70:0/3472713015 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599b1f5af90 0x5599b1f5d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:11.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.417+0000 7faf437fe640 10 client.29827 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:11.417 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.417+0000 7faf63fff640 1 --2- 172.21.15.70:0/3472713015 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599b1f5af90 0x5599b1f5d3b0 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f790dda1314d8543 server_cookie=66d192957698916d in_seq=0 out_seq=0 2024-04-01T01:49:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.454+0000 7faf437fe640 1 -- 172.21.15.70:0/3472713015 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7faf540c0490 con 0x5599b1f5af90 2024-04-01T01:49:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.454+0000 7faf437fe640 10 client.29827 handle_command_reply: tid=0 2024-04-01T01:49:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.454+0000 7faf6d381740 1 client.29827 shutdown 2024-04-01T01:49:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.454+0000 7faf6d381740 1 -- 172.21.15.70:0/3472713015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 msgr2=0x5599b1ee6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.454+0000 7faf6d381740 1 --2- 172.21.15.70:0/3472713015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 0x5599b1ee6450 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7faf5c0052d0 tx=0x7faf5c009af0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.454+0000 7faf6d381740 1 -- 172.21.15.70:0/3472713015 shutdown_connections 2024-04-01T01:49:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.454+0000 7faf6d381740 1 --2- 172.21.15.70:0/3472713015 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5599b1f5af90 0x5599b1f5d3b0 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.455+0000 7faf6d381740 1 --2- 172.21.15.70:0/3472713015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5599b1edb640 0x5599b1ee6450 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.455+0000 7faf437fe640 0 client.29827 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.455+0000 7faf6d381740 1 --2- 172.21.15.70:0/3472713015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5599b1ed64d0 0x5599b1ee5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.455+0000 7faf6d381740 1 --2- 172.21.15.70:0/3472713015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599b1ed5ae0 0x5599b1ee4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.455+0000 7faf6d381740 1 -- 172.21.15.70:0/3472713015 >> 172.21.15.70:0/3472713015 conn(0x5599b1ed1050 msgr2=0x5599b1edfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.455+0000 7faf6d381740 1 -- 172.21.15.70:0/3472713015 shutdown_connections 2024-04-01T01:49:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.455+0000 7faf6d381740 1 -- 172.21.15.70:0/3472713015 wait complete. 2024-04-01T01:49:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.455+0000 7faf6d381740 20 client.29827 trim_cache size 0 max 16384 2024-04-01T01:49:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:49:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:49:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:49:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:49:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.456+0000 7faf6946b640 1 -- 172.21.15.70:0/4081226439 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faf4c0a25e0 msgr2=0x7faf4c0a4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.456+0000 7faf6946b640 1 --2- 172.21.15.70:0/4081226439 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faf4c0a25e0 0x7faf4c0a4aa0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7faf6419af60 tx=0x7faf58011440 comp rx=0 tx=0).stop 2024-04-01T01:49:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.456+0000 7faf6946b640 1 -- 172.21.15.70:0/4081226439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf64100d00 msgr2=0x7faf64199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.456+0000 7faf6946b640 1 --2- 172.21.15.70:0/4081226439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf64100d00 0x7faf64199a50 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7faf54000c00 tx=0x7faf54004600 comp rx=0 tx=0).stop 2024-04-01T01:49:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.459+0000 7faf6946b640 1 -- 172.21.15.70:0/4081226439 shutdown_connections 2024-04-01T01:49:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.459+0000 7faf6946b640 1 --2- 172.21.15.70:0/4081226439 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf64106130 0x7faf641a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.459+0000 7faf6946b640 1 --2- 172.21.15.70:0/4081226439 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7faf4c0a25e0 0x7faf4c0a4aa0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.459+0000 7faf6946b640 1 --2- 172.21.15.70:0/4081226439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf64103630 0x7faf64199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.459+0000 7faf6946b640 1 --2- 172.21.15.70:0/4081226439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf64100d00 0x7faf64199a50 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.459+0000 7faf6946b640 1 -- 172.21.15.70:0/4081226439 >> 172.21.15.70:0/4081226439 conn(0x7faf640fad60 msgr2=0x7faf641065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.460+0000 7faf6946b640 1 -- 172.21.15.70:0/4081226439 shutdown_connections 2024-04-01T01:49:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:11.460+0000 7faf6946b640 1 -- 172.21.15.70:0/4081226439 wait complete. 2024-04-01T01:49:11.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 865.0 s, 848.4 tps, lat 119.297 ms stddev 116.555 2024-04-01T01:49:12.499 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:49:12 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-c[66342]: ignoring --setuser ceph since I am not root 2024-04-01T01:49:12.499 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:49:12 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-c[66342]: ignoring --setgroup ceph since I am not root 2024-04-01T01:49:12.499 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:49:12 smithi189 ceph-mds[66346]: -- 172.21.15.189:0/401494001 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5636e7a07a40 con 0x5636e79db000 2024-04-01T01:49:12.499 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:49:12 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-c[66342]: starting mds.c at 2024-04-01T01:49:12.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:12 smithi189 ceph-mon[35753]: pgmap v830: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 11 MiB/s wr, 888 op/s 2024-04-01T01:49:12.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:12 smithi189 ceph-mon[35753]: daemon mds.c restarted 2024-04-01T01:49:13.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:12 smithi070 ceph-mon[32179]: pgmap v830: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 11 MiB/s wr, 888 op/s 2024-04-01T01:49:13.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:12 smithi070 ceph-mon[32179]: daemon mds.c restarted 2024-04-01T01:49:13.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:12 smithi081 ceph-mon[35142]: pgmap v830: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 11 MiB/s wr, 888 op/s 2024-04-01T01:49:13.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:12 smithi081 ceph-mon[35142]: daemon mds.c restarted 2024-04-01T01:49:13.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:13 smithi189 ceph-mon[35753]: osdmap e95: 12 total, 12 up, 12 in 2024-04-01T01:49:13.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:13 smithi189 ceph-mon[35753]: mds.? [v2:172.21.15.189:6836/921108936,v1:172.21.15.189:6837/921108936] up:boot 2024-04-01T01:49:13.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:13 smithi189 ceph-mon[35753]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-01T01:49:13.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:13 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-01T01:49:13.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:13 smithi189 ceph-mon[35753]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-01T01:49:14.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:13 smithi070 ceph-mon[32179]: osdmap e95: 12 total, 12 up, 12 in 2024-04-01T01:49:14.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:13 smithi070 ceph-mon[32179]: mds.? [v2:172.21.15.189:6836/921108936,v1:172.21.15.189:6837/921108936] up:boot 2024-04-01T01:49:14.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:13 smithi070 ceph-mon[32179]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-01T01:49:14.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:13 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-01T01:49:14.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:13 smithi070 ceph-mon[32179]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-01T01:49:14.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:13 smithi081 ceph-mon[35142]: osdmap e95: 12 total, 12 up, 12 in 2024-04-01T01:49:14.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:13 smithi081 ceph-mon[35142]: mds.? [v2:172.21.15.189:6836/921108936,v1:172.21.15.189:6837/921108936] up:boot 2024-04-01T01:49:14.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:13 smithi081 ceph-mon[35142]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-01T01:49:14.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:13 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-01T01:49:14.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:13 smithi081 ceph-mon[35142]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-01T01:49:14.979 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:14 smithi081 ceph-mon[35142]: pgmap v832: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 12 MiB/s wr, 790 op/s 2024-04-01T01:49:14.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:14 smithi189 ceph-mon[35753]: pgmap v832: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 12 MiB/s wr, 790 op/s 2024-04-01T01:49:15.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:14 smithi070 ceph-mon[32179]: pgmap v832: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 12 MiB/s wr, 790 op/s 2024-04-01T01:49:16.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 870.0 s, 855.4 tps, lat 117.530 ms stddev 99.278 2024-04-01T01:49:17.001 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:16 smithi189 ceph-mon[35753]: pgmap v833: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 12 MiB/s wr, 765 op/s 2024-04-01T01:49:17.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:16 smithi070 ceph-mon[32179]: pgmap v833: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 12 MiB/s wr, 765 op/s 2024-04-01T01:49:17.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:16 smithi081 ceph-mon[35142]: pgmap v833: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 12 MiB/s wr, 765 op/s 2024-04-01T01:49:18.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:18 smithi189 ceph-mon[35753]: pgmap v834: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 13 MiB/s wr, 767 op/s 2024-04-01T01:49:19.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:18 smithi070 ceph-mon[32179]: pgmap v834: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 13 MiB/s wr, 767 op/s 2024-04-01T01:49:19.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:18 smithi081 ceph-mon[35142]: pgmap v834: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 13 MiB/s wr, 767 op/s 2024-04-01T01:49:20.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:20 smithi189 ceph-mon[35753]: pgmap v835: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 13 MiB/s wr, 768 op/s 2024-04-01T01:49:21.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:20 smithi070 ceph-mon[32179]: pgmap v835: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 13 MiB/s wr, 768 op/s 2024-04-01T01:49:21.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:20 smithi081 ceph-mon[35142]: pgmap v835: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 13 MiB/s wr, 768 op/s 2024-04-01T01:49:21.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 875.0 s, 805.4 tps, lat 121.062 ms stddev 102.355 2024-04-01T01:49:21.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:21 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:49:22.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:21 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:49:22.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:21 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:49:22.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:22 smithi189 ceph-mon[35753]: pgmap v836: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 11 MiB/s wr, 737 op/s 2024-04-01T01:49:22.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:49:22.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:49:22.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:49:23.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:22 smithi070 ceph-mon[32179]: pgmap v836: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 11 MiB/s wr, 737 op/s 2024-04-01T01:49:23.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:49:23.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:49:23.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:49:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:22 smithi081 ceph-mon[35142]: pgmap v836: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 11 MiB/s wr, 737 op/s 2024-04-01T01:49:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:49:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:49:23.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:49:24.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.582+0000 7f644ebc7640 1 -- 172.21.15.70:0/1735570603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64480ffa30 msgr2=0x7f64480ffe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:24.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.582+0000 7f644ebc7640 1 --2- 172.21.15.70:0/1735570603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64480ffa30 0x7f64480ffe50 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f643c009930 tx=0x7f643c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:49:24.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.582+0000 7f644ebc7640 1 -- 172.21.15.70:0/1735570603 shutdown_connections 2024-04-01T01:49:24.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.582+0000 7f644ebc7640 1 --2- 172.21.15.70:0/1735570603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64481033c0 0x7f6448109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.582+0000 7f644ebc7640 1 --2- 172.21.15.70:0/1735570603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6448102a00 0x7f6448102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.582+0000 7f644ebc7640 1 --2- 172.21.15.70:0/1735570603 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64480ffa30 0x7f64480ffe50 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.582+0000 7f644ebc7640 1 -- 172.21.15.70:0/1735570603 >> 172.21.15.70:0/1735570603 conn(0x7f64480fb3a0 msgr2=0x7f64480fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:24.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.583+0000 7f644ebc7640 1 -- 172.21.15.70:0/1735570603 shutdown_connections 2024-04-01T01:49:24.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.583+0000 7f644ebc7640 1 -- 172.21.15.70:0/1735570603 wait complete. 2024-04-01T01:49:24.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.583+0000 7f644ebc7640 1 Processor -- start 2024-04-01T01:49:24.585 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.583+0000 7f644ebc7640 1 -- start start 2024-04-01T01:49:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f644ebc7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64480ffa30 0x7f64481979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f644ebc7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6448102a00 0x7f6448197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f644ebc7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64481033c0 0x7f644819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f644ebc7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f644810d2b0 con 0x7f64480ffa30 2024-04-01T01:49:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f644ebc7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f644810d130 con 0x7f6448102a00 2024-04-01T01:49:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f644ebc7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f644810d430 con 0x7f64481033c0 2024-04-01T01:49:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f6447fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6448102a00 0x7f6448197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f6447fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6448102a00 0x7f6448197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53824/0 (socket says 172.21.15.70:53824) 2024-04-01T01:49:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f6447fff640 1 -- 172.21.15.70:0/297235723 learned_addr learned my addr 172.21.15.70:0/297235723 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f644c963640 1 --2- 172.21.15.70:0/297235723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64480ffa30 0x7f64481979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f644d164640 1 --2- 172.21.15.70:0/297235723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64481033c0 0x7f644819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f6447fff640 1 -- 172.21.15.70:0/297235723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64481033c0 msgr2=0x7f644819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:24.586 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f6447fff640 1 --2- 172.21.15.70:0/297235723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64481033c0 0x7f644819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f6447fff640 1 -- 172.21.15.70:0/297235723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64480ffa30 msgr2=0x7f64481979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f6447fff640 1 --2- 172.21.15.70:0/297235723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64480ffa30 0x7f64481979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f6447fff640 1 -- 172.21.15.70:0/297235723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f644819f5d0 con 0x7f6448102a00 2024-04-01T01:49:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.584+0000 7f644c963640 1 --2- 172.21.15.70:0/297235723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64480ffa30 0x7f64481979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:49:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.585+0000 7f6447fff640 1 --2- 172.21.15.70:0/297235723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6448102a00 0x7f6448197ef0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f643400ea60 tx=0x7f643400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.585+0000 7f6445ffb640 1 -- 172.21.15.70:0/297235723 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643400cde0 con 0x7f6448102a00 2024-04-01T01:49:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.585+0000 7f6445ffb640 1 -- 172.21.15.70:0/297235723 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f64340035f0 con 0x7f6448102a00 2024-04-01T01:49:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.585+0000 7f6445ffb640 1 -- 172.21.15.70:0/297235723 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6434010690 con 0x7f6448102a00 2024-04-01T01:49:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.585+0000 7f644ebc7640 1 -- 172.21.15.70:0/297235723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6448100850 con 0x7f6448102a00 2024-04-01T01:49:24.587 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.585+0000 7f644ebc7640 1 -- 172.21.15.70:0/297235723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6448100d50 con 0x7f6448102a00 2024-04-01T01:49:24.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.589+0000 7f64517a6740 1 Processor -- start 2024-04-01T01:49:24.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.590+0000 7f64517a6740 1 -- start start 2024-04-01T01:49:24.589 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.590+0000 7f6445ffb640 1 -- 172.21.15.70:0/297235723 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f64340037a0 con 0x7f6448102a00 2024-04-01T01:49:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.591+0000 7f64517a6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb16620e0 0x559cb1662500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.591+0000 7f64517a6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb1662ad0 0x559cb16675d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.591+0000 7f64517a6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb1667c40 0x559cb166a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.591+0000 7f64517a6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559cb15bd8f0 con 0x559cb1667c40 2024-04-01T01:49:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.591+0000 7f64517a6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x559cb15668d0 con 0x559cb16620e0 2024-04-01T01:49:24.590 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.591+0000 7f64517a6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559cb15b9990 con 0x559cb1662ad0 2024-04-01T01:49:24.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.591+0000 7f6445ffb640 1 --2- 172.21.15.70:0/297235723 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64380a25c0 0x7f64380a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.591+0000 7f6445ffb640 1 -- 172.21.15.70:0/297235723 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f64340cedc0 con 0x7f6448102a00 2024-04-01T01:49:24.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.591+0000 7f6447fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb1667c40 0x559cb166a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:24.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.591+0000 7f644d164640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb16620e0 0x559cb1662500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:24.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.591+0000 7f6447fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb1667c40 0x559cb166a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41442/0 (socket says 172.21.15.70:41442) 2024-04-01T01:49:24.591 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.591+0000 7f6447fff640 1 -- 172.21.15.70:0/1529395337 learned_addr learned my addr 172.21.15.70:0/1529395337 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.592+0000 7f644d164640 1 --2- 172.21.15.70:0/297235723 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64380a25c0 0x7f64380a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.592+0000 7f6447fff640 1 -- 172.21.15.70:0/1529395337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb1662ad0 msgr2=0x559cb16675d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:49:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.592+0000 7f6447fff640 1 --2- 172.21.15.70:0/1529395337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb1662ad0 0x559cb16675d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.592+0000 7f6447fff640 1 -- 172.21.15.70:0/1529395337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb16620e0 msgr2=0x559cb1662500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.592+0000 7f6447fff640 1 --2- 172.21.15.70:0/1529395337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb16620e0 0x559cb1662500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.592 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.592+0000 7f6447fff640 1 -- 172.21.15.70:0/1529395337 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cb12d2210 con 0x559cb1667c40 2024-04-01T01:49:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.592+0000 7f6447fff640 1 --2- 172.21.15.70:0/1529395337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb1667c40 0x559cb166a0d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f64340a3930 tx=0x7f643409b2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a6b6b6fbb07cb8a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.592+0000 7f644d164640 1 --2- 172.21.15.70:0/297235723 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64380a25c0 0x7f64380a4a80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x559cb1663330 tx=0x7f6440004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:24.593 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.593+0000 7f6432ffd640 1 -- 172.21.15.70:0/1529395337 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64340b4070 con 0x559cb1667c40 2024-04-01T01:49:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.593+0000 7f6432ffd640 1 -- 172.21.15.70:0/1529395337 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f643409bb50 con 0x559cb1667c40 2024-04-01T01:49:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.593+0000 7f6432ffd640 1 -- 172.21.15.70:0/1529395337 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6434097ac0 con 0x559cb1667c40 2024-04-01T01:49:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.593+0000 7f64517a6740 1 -- 172.21.15.70:0/1529395337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb1667c40 msgr2=0x559cb166a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.593+0000 7f64517a6740 1 --2- 172.21.15.70:0/1529395337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb1667c40 0x559cb166a0d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f64340a3930 tx=0x7f643409b2e0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.593+0000 7f64517a6740 1 -- 172.21.15.70:0/1529395337 shutdown_connections 2024-04-01T01:49:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.593+0000 7f64517a6740 1 --2- 172.21.15.70:0/1529395337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb1667c40 0x559cb166a0d0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.593+0000 7f64517a6740 1 --2- 172.21.15.70:0/1529395337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb1662ad0 0x559cb16675d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.593+0000 7f64517a6740 1 --2- 172.21.15.70:0/1529395337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb16620e0 0x559cb1662500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.594+0000 7f64517a6740 1 -- 172.21.15.70:0/1529395337 >> 172.21.15.70:0/1529395337 conn(0x559cb165d650 msgr2=0x559cb165da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.594+0000 7f64517a6740 1 -- 172.21.15.70:0/1529395337 shutdown_connections 2024-04-01T01:49:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.594+0000 7f64517a6740 1 -- 172.21.15.70:0/1529395337 wait complete. 2024-04-01T01:49:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.594+0000 7f64517a6740 1 -- start start 2024-04-01T01:49:24.594 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.594+0000 7f64517a6740 1 Processor -- start 2024-04-01T01:49:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.595+0000 7f64517a6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb16620e0 0x559cb166c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.595+0000 7f64517a6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1662ad0 0x559cb166c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.595+0000 7f64517a6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb1667c40 0x559cb16739a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.595+0000 7f64517a6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559cb166ab90 con 0x559cb16620e0 2024-04-01T01:49:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.595+0000 7f64517a6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x559cb155bf80 con 0x559cb1662ad0 2024-04-01T01:49:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.595+0000 7f64517a6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559cb166ad10 con 0x559cb1667c40 2024-04-01T01:49:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.595+0000 7f64517a6740 10 client.0 _ll_register_callbacks cb 0x559cb15b4290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.595+0000 7f6447fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1662ad0 0x559cb166c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.595+0000 7f644d164640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb1667c40 0x559cb16739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:24.596 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.595+0000 7f644d164640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb1667c40 0x559cb16739a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:33462/0 (socket says 172.21.15.70:33462) 2024-04-01T01:49:24.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.595+0000 7f644d164640 1 -- 172.21.15.70:0/141321347 learned_addr learned my addr 172.21.15.70:0/141321347 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:24.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.595+0000 7f644c963640 1 --2- 172.21.15.70:0/141321347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb16620e0 0x559cb166c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:24.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.596+0000 7f6447fff640 1 -- 172.21.15.70:0/141321347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb1667c40 msgr2=0x559cb16739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:24.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.596+0000 7f6447fff640 1 --2- 172.21.15.70:0/141321347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb1667c40 0x559cb16739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.596+0000 7f6447fff640 1 -- 172.21.15.70:0/141321347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb16620e0 msgr2=0x559cb166c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:24.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.596+0000 7f6447fff640 1 --2- 172.21.15.70:0/141321347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb16620e0 0x559cb166c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.596+0000 7f6447fff640 1 -- 172.21.15.70:0/141321347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cb139e620 con 0x559cb1662ad0 2024-04-01T01:49:24.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.596+0000 7f6447fff640 1 --2- 172.21.15.70:0/141321347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1662ad0 0x559cb166c960 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f643409af20 tx=0x7f64340970d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b95681e9577ed54c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:24.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.596+0000 7f6427fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:49:24.597 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.597+0000 7f6427fff640 1 -- 172.21.15.70:0/141321347 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64340b8020 con 0x559cb1662ad0 2024-04-01T01:49:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.597+0000 7f6427fff640 1 -- 172.21.15.70:0/141321347 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f643409f170 con 0x559cb1662ad0 2024-04-01T01:49:24.598 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.598+0000 7f64517a6740 1 -- 172.21.15.70:0/141321347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f643409bb50 con 0x559cb1662ad0 2024-04-01T01:49:24.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.601+0000 7f6427fff640 1 -- 172.21.15.70:0/141321347 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f64340b4040 con 0x559cb1662ad0 2024-04-01T01:49:24.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.601+0000 7f64517a6740 10 client.30122 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:24.600 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.601+0000 7f64517a6740 1 -- 172.21.15.70:0/141321347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559cb1676980 con 0x559cb1662ad0 2024-04-01T01:49:24.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.602+0000 7f6427fff640 1 -- 172.21.15.70:0/141321347 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f643409bb50 con 0x559cb1662ad0 2024-04-01T01:49:24.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.602+0000 7f6427fff640 1 -- 172.21.15.70:0/141321347 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f643409a8a0 con 0x559cb1662ad0 2024-04-01T01:49:24.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.602+0000 7f64517a6740 10 client.30122 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:24.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.602+0000 7f64517a6740 10 client.30122 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:49:24.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.602+0000 7f64517a6740 20 client.30122 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:24.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.602+0000 7f64517a6740 1 --2- 172.21.15.70:0/141321347 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559cb1677b00 0x559cb1679ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.602+0000 7f64517a6740 4 client.30122 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:49:24.602 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.602+0000 7f64517a6740 1 -- 172.21.15.70:0/141321347 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559cb15d1120 con 0x559cb1677b00 2024-04-01T01:49:24.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.604+0000 7f644c963640 1 --2- 172.21.15.70:0/141321347 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559cb1677b00 0x559cb1679ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:24.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.604+0000 7f644c963640 1 --2- 172.21.15.70:0/141321347 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559cb1677b00 0x559cb1679ef0 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee097230242d582d server_cookie=8e7c1b3db1d2b5f9 in_seq=0 out_seq=0 2024-04-01T01:49:24.603 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.604+0000 7f6427fff640 10 client.30122 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:24.604 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.605+0000 7f6427fff640 1 -- 172.21.15.70:0/141321347 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559cb15d1120 con 0x559cb1677b00 2024-04-01T01:49:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.605+0000 7f6427fff640 10 client.30122 handle_command_reply: tid=0 2024-04-01T01:49:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.605+0000 7f64517a6740 1 client.30122 shutdown 2024-04-01T01:49:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.605+0000 7f64517a6740 1 -- 172.21.15.70:0/141321347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1662ad0 msgr2=0x559cb166c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.605+0000 7f64517a6740 1 --2- 172.21.15.70:0/141321347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1662ad0 0x559cb166c960 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f643409af20 tx=0x7f64340970d0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.605+0000 7f64517a6740 1 -- 172.21.15.70:0/141321347 shutdown_connections 2024-04-01T01:49:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.606+0000 7f64517a6740 1 --2- 172.21.15.70:0/141321347 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559cb1677b00 0x559cb1679ef0 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.606+0000 7f64517a6740 1 --2- 172.21.15.70:0/141321347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb1667c40 0x559cb16739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.605 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.606+0000 7f64517a6740 1 --2- 172.21.15.70:0/141321347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1662ad0 0x559cb166c960 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.606+0000 7f64517a6740 1 --2- 172.21.15.70:0/141321347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb16620e0 0x559cb166c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.606+0000 7f64517a6740 1 -- 172.21.15.70:0/141321347 >> 172.21.15.70:0/141321347 conn(0x559cb165d650 msgr2=0x559cb1653f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:24.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.606+0000 7f6427fff640 0 client.30122 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:24.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.606+0000 7f64517a6740 1 -- 172.21.15.70:0/141321347 shutdown_connections 2024-04-01T01:49:24.606 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.606+0000 7f64517a6740 1 -- 172.21.15.70:0/141321347 wait complete. 2024-04-01T01:49:24.607 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.606+0000 7f64517a6740 20 client.30122 trim_cache size 0 max 16384 2024-04-01T01:49:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.636+0000 7f64517a6740 1 Processor -- start 2024-04-01T01:49:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.636+0000 7f64517a6740 1 -- start start 2024-04-01T01:49:24.636 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.637+0000 7f64517a6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb16620e0 0x559cb1663e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.637+0000 7f64517a6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb1662ad0 0x559cb1673960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.637+0000 7f64517a6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1667c40 0x559cb16643d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.637+0000 7f64517a6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f64481a8280 con 0x559cb1662ad0 2024-04-01T01:49:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.637+0000 7f64517a6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f64481a9dc0 con 0x559cb1667c40 2024-04-01T01:49:24.637 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.637+0000 7f64517a6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f64481a8950 con 0x559cb16620e0 2024-04-01T01:49:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.638+0000 7f644d164640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1667c40 0x559cb16643d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.638+0000 7f644d164640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1667c40 0x559cb16643d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:53848/0 (socket says 172.21.15.70:53848) 2024-04-01T01:49:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.638+0000 7f644d164640 1 -- 172.21.15.70:0/2480224401 learned_addr learned my addr 172.21.15.70:0/2480224401 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:24.638 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.639+0000 7f644c963640 1 --2- 172.21.15.70:0/2480224401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb16620e0 0x559cb1663e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.639+0000 7f644d164640 1 -- 172.21.15.70:0/2480224401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb16620e0 msgr2=0x559cb1663e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.639+0000 7f644d164640 1 --2- 172.21.15.70:0/2480224401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb16620e0 0x559cb1663e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.639+0000 7f644d164640 1 -- 172.21.15.70:0/2480224401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb1662ad0 msgr2=0x559cb1673960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.639+0000 7f644d164640 1 --2- 172.21.15.70:0/2480224401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb1662ad0 0x559cb1673960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.639+0000 7f644d164640 1 -- 172.21.15.70:0/2480224401 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cb166ae50 con 0x559cb1667c40 2024-04-01T01:49:24.639 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.639+0000 7f644d164640 1 --2- 172.21.15.70:0/2480224401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1667c40 0x559cb16643d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f6440009f90 tx=0x7f6440005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c675eeaeb73da600 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.641+0000 7f64327fc640 1 -- 172.21.15.70:0/2480224401 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644000ecb0 con 0x559cb1667c40 2024-04-01T01:49:24.640 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.641+0000 7f64327fc640 1 -- 172.21.15.70:0/2480224401 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f644000ee50 con 0x559cb1667c40 2024-04-01T01:49:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.642+0000 7f64517a6740 1 -- 172.21.15.70:0/2480224401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1667c40 msgr2=0x559cb16643d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.642+0000 7f64517a6740 1 --2- 172.21.15.70:0/2480224401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1667c40 0x559cb16643d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f6440009f90 tx=0x7f6440005bd0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.642+0000 7f64517a6740 1 -- 172.21.15.70:0/2480224401 shutdown_connections 2024-04-01T01:49:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.642+0000 7f64517a6740 1 --2- 172.21.15.70:0/2480224401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1667c40 0x559cb16643d0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.642+0000 7f64517a6740 1 --2- 172.21.15.70:0/2480224401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb1662ad0 0x559cb1673960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.641 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.642+0000 7f64517a6740 1 --2- 172.21.15.70:0/2480224401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb16620e0 0x559cb1663e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.642+0000 7f64517a6740 1 -- 172.21.15.70:0/2480224401 >> 172.21.15.70:0/2480224401 conn(0x559cb165d650 msgr2=0x559cb165da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.642+0000 7f64517a6740 1 -- 172.21.15.70:0/2480224401 shutdown_connections 2024-04-01T01:49:24.642 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.642+0000 7f64517a6740 1 -- 172.21.15.70:0/2480224401 wait complete. 2024-04-01T01:49:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.642+0000 7f64517a6740 1 -- start start 2024-04-01T01:49:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.642+0000 7f64517a6740 1 Processor -- start 2024-04-01T01:49:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.643+0000 7f64517a6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb16620e0 0x559cb1671260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.643+0000 7f64517a6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1662ad0 0x559cb16717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.643+0000 7f64517a6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb1667c40 0x559cb16727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.643+0000 7f64517a6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f64481a96f0 con 0x559cb16620e0 2024-04-01T01:49:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.643+0000 7f64517a6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f64481a9020 con 0x559cb1662ad0 2024-04-01T01:49:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.643+0000 7f64517a6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f64481a6260 con 0x559cb1667c40 2024-04-01T01:49:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.643+0000 7f64517a6740 10 client.0 _ll_register_callbacks cb 0x7f64340b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.643+0000 7f644c963640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb16620e0 0x559cb1671260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:24.643 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.643+0000 7f644c963640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb16620e0 0x559cb1671260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41462/0 (socket says 172.21.15.70:41462) 2024-04-01T01:49:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.643+0000 7f644c963640 1 -- 172.21.15.70:0/118355956 learned_addr learned my addr 172.21.15.70:0/118355956 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:24.644 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.643+0000 7f6447fff640 1 --2- 172.21.15.70:0/118355956 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1662ad0 0x559cb16717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.643+0000 7f6447fff640 1 -- 172.21.15.70:0/118355956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb1667c40 msgr2=0x559cb16727b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:49:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.643+0000 7f6447fff640 1 --2- 172.21.15.70:0/118355956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb1667c40 0x559cb16727b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.643+0000 7f6447fff640 1 -- 172.21.15.70:0/118355956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb16620e0 msgr2=0x559cb1671260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.643+0000 7f6447fff640 1 --2- 172.21.15.70:0/118355956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb16620e0 0x559cb1671260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.643+0000 7f6447fff640 1 -- 172.21.15.70:0/118355956 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cb1672da0 con 0x559cb1662ad0 2024-04-01T01:49:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.644+0000 7f6447fff640 1 --2- 172.21.15.70:0/118355956 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1662ad0 0x559cb16717a0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6434003ee0 tx=0x7f6434097cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=245f99cdef578828 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.644+0000 7f6427fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:49:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.644+0000 7f6427fff640 1 -- 172.21.15.70:0/118355956 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64340b8020 con 0x559cb1662ad0 2024-04-01T01:49:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.644+0000 7f6427fff640 1 -- 172.21.15.70:0/118355956 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f6434004000 con 0x559cb1662ad0 2024-04-01T01:49:24.645 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.644+0000 7f64517a6740 1 -- 172.21.15.70:0/118355956 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559cb15cae30 con 0x559cb1662ad0 2024-04-01T01:49:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.644+0000 7f6427fff640 1 -- 172.21.15.70:0/118355956 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f6434097510 con 0x559cb1662ad0 2024-04-01T01:49:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.644+0000 7f64517a6740 10 client.30128 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.644+0000 7f64517a6740 1 -- 172.21.15.70:0/118355956 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559cb1672da0 con 0x559cb1662ad0 2024-04-01T01:49:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.645+0000 7f6427fff640 1 -- 172.21.15.70:0/118355956 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x559cb15cae30 con 0x559cb1662ad0 2024-04-01T01:49:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.645+0000 7f6427fff640 1 -- 172.21.15.70:0/118355956 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f64340a0730 con 0x559cb1662ad0 2024-04-01T01:49:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.646+0000 7f64517a6740 10 client.30128 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.646+0000 7f64517a6740 10 client.30128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:49:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.646+0000 7f64517a6740 20 client.30128 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.646+0000 7f64517a6740 1 --2- 172.21.15.70:0/118355956 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559cb16de4c0 0x559cb16e08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.646+0000 7f64517a6740 4 client.30128 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:49:24.646 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.646+0000 7f64517a6740 1 -- 172.21.15.70:0/118355956 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559cb15c5180 con 0x559cb16de4c0 2024-04-01T01:49:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.647+0000 7f644c963640 1 --2- 172.21.15.70:0/118355956 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559cb16de4c0 0x559cb16e08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.648+0000 7f644c963640 1 --2- 172.21.15.70:0/118355956 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559cb16de4c0 0x559cb16e08e0 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe02990dc1163ebc server_cookie=f0a3505aa15752a1 in_seq=0 out_seq=0 2024-04-01T01:49:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.648+0000 7f6427fff640 10 client.30128 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:24.692 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.692+0000 7f6427fff640 1 -- 172.21.15.70:0/118355956 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559cb15c5180 con 0x559cb16de4c0 2024-04-01T01:49:24.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.692+0000 7f6427fff640 10 client.30128 handle_command_reply: tid=0 2024-04-01T01:49:24.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.693+0000 7f64517a6740 1 client.30128 shutdown 2024-04-01T01:49:24.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.693+0000 7f64517a6740 1 -- 172.21.15.70:0/118355956 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1662ad0 msgr2=0x559cb16717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:24.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.693+0000 7f64517a6740 1 --2- 172.21.15.70:0/118355956 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1662ad0 0x559cb16717a0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6434003ee0 tx=0x7f6434097cf0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.693+0000 7f64517a6740 1 -- 172.21.15.70:0/118355956 shutdown_connections 2024-04-01T01:49:24.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.693+0000 7f64517a6740 1 --2- 172.21.15.70:0/118355956 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x559cb16de4c0 0x559cb16e08e0 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.693+0000 7f64517a6740 1 --2- 172.21.15.70:0/118355956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559cb1667c40 0x559cb16727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.693+0000 7f64517a6740 1 --2- 172.21.15.70:0/118355956 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559cb1662ad0 0x559cb16717a0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.693+0000 7f64517a6740 1 --2- 172.21.15.70:0/118355956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559cb16620e0 0x559cb1671260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.693+0000 7f64517a6740 1 -- 172.21.15.70:0/118355956 >> 172.21.15.70:0/118355956 conn(0x559cb165d650 msgr2=0x559cb16608b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:24.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.693+0000 7f6427fff640 0 client.30128 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:24.693 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.693+0000 7f64517a6740 1 -- 172.21.15.70:0/118355956 shutdown_connections 2024-04-01T01:49:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.693+0000 7f64517a6740 1 -- 172.21.15.70:0/118355956 wait complete. 2024-04-01T01:49:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.693+0000 7f64517a6740 20 client.30128 trim_cache size 0 max 16384 2024-04-01T01:49:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:49:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:49:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:49:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:49:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.694+0000 7f644ebc7640 1 -- 172.21.15.70:0/297235723 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64380a25c0 msgr2=0x7f64380a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.694+0000 7f644ebc7640 1 --2- 172.21.15.70:0/297235723 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64380a25c0 0x7f64380a4a80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x559cb1663330 tx=0x7f6440004ab0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.694+0000 7f644ebc7640 1 -- 172.21.15.70:0/297235723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6448102a00 msgr2=0x7f6448197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.694+0000 7f644ebc7640 1 --2- 172.21.15.70:0/297235723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6448102a00 0x7f6448197ef0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f643400ea60 tx=0x7f643400ef20 comp rx=0 tx=0).stop 2024-04-01T01:49:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.694+0000 7f644ebc7640 1 -- 172.21.15.70:0/297235723 shutdown_connections 2024-04-01T01:49:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.694+0000 7f644ebc7640 1 --2- 172.21.15.70:0/297235723 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f64380a25c0 0x7f64380a4a80 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.695 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.694+0000 7f644ebc7640 1 --2- 172.21.15.70:0/297235723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64481033c0 0x7f644819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.694+0000 7f644ebc7640 1 --2- 172.21.15.70:0/297235723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6448102a00 0x7f6448197ef0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.694+0000 7f644ebc7640 1 --2- 172.21.15.70:0/297235723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64480ffa30 0x7f64481979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.694+0000 7f644ebc7640 1 -- 172.21.15.70:0/297235723 >> 172.21.15.70:0/297235723 conn(0x7f64480fb3a0 msgr2=0x7f6448105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.694+0000 7f644ebc7640 1 -- 172.21.15.70:0/297235723 shutdown_connections 2024-04-01T01:49:24.696 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:24.694+0000 7f644ebc7640 1 -- 172.21.15.70:0/297235723 wait complete. 2024-04-01T01:49:24.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:24 smithi189 ceph-mon[35753]: pgmap v837: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 11 MiB/s wr, 728 op/s 2024-04-01T01:49:25.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:24 smithi070 ceph-mon[32179]: pgmap v837: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 11 MiB/s wr, 728 op/s 2024-04-01T01:49:25.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:24 smithi081 ceph-mon[35142]: pgmap v837: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 11 MiB/s wr, 728 op/s 2024-04-01T01:49:26.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 880.0 s, 822.8 tps, lat 123.934 ms stddev 99.889 2024-04-01T01:49:26.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:26 smithi189 ceph-mon[35753]: pgmap v838: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 11 MiB/s wr, 728 op/s 2024-04-01T01:49:27.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:26 smithi070 ceph-mon[32179]: pgmap v838: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 11 MiB/s wr, 728 op/s 2024-04-01T01:49:27.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:26 smithi081 ceph-mon[35142]: pgmap v838: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 11 MiB/s wr, 728 op/s 2024-04-01T01:49:28.918 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:49:28.918 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:49:29.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:28 smithi070 ceph-mon[32179]: pgmap v839: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 12 MiB/s wr, 755 op/s 2024-04-01T01:49:29.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.064+0000 7fb40eef4640 1 -- 172.21.15.70:0/1228149534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb408102200 msgr2=0x7fb408108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:29.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.064+0000 7fb40eef4640 1 --2- 172.21.15.70:0/1228149534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb408102200 0x7fb408108a90 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fb400009be0 tx=0x7fb40002bef0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.064+0000 7fb40eef4640 1 -- 172.21.15.70:0/1228149534 shutdown_connections 2024-04-01T01:49:29.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.064+0000 7fb40eef4640 1 --2- 172.21.15.70:0/1228149534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb408102200 0x7fb408108a90 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.064+0000 7fb40eef4640 1 --2- 172.21.15.70:0/1228149534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb408101840 0x7fb408101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.064+0000 7fb40eef4640 1 --2- 172.21.15.70:0/1228149534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb408100650 0x7fb408100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.064+0000 7fb40eef4640 1 -- 172.21.15.70:0/1228149534 >> 172.21.15.70:0/1228149534 conn(0x7fb4080fbbe0 msgr2=0x7fb4080fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:29.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.064+0000 7fb40eef4640 1 -- 172.21.15.70:0/1228149534 shutdown_connections 2024-04-01T01:49:29.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.064+0000 7fb40eef4640 1 -- 172.21.15.70:0/1228149534 wait complete. 2024-04-01T01:49:29.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.065+0000 7fb40eef4640 1 Processor -- start 2024-04-01T01:49:29.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.065+0000 7fb40eef4640 1 -- start start 2024-04-01T01:49:29.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.065+0000 7fb40eef4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb408100650 0x7fb408069440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.065+0000 7fb40eef4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb408101840 0x7fb40819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.065+0000 7fb407fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb408101840 0x7fb40819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.065+0000 7fb40eef4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb408102200 0x7fb40819ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.066+0000 7fb40eef4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb40810c190 con 0x7fb408101840 2024-04-01T01:49:29.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.066+0000 7fb40eef4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb40810c010 con 0x7fb408100650 2024-04-01T01:49:29.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.066+0000 7fb40eef4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb40810c310 con 0x7fb408102200 2024-04-01T01:49:29.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.066+0000 7fb407fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb408101840 0x7fb40819c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48934/0 (socket says 172.21.15.70:48934) 2024-04-01T01:49:29.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.066+0000 7fb407fff640 1 -- 172.21.15.70:0/3882903809 learned_addr learned my addr 172.21.15.70:0/3882903809 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:29.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.066+0000 7fb40d491640 1 --2- 172.21.15.70:0/3882903809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb408102200 0x7fb40819ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.066+0000 7fb407fff640 1 -- 172.21.15.70:0/3882903809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb408102200 msgr2=0x7fb40819ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:29.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.066+0000 7fb407fff640 1 --2- 172.21.15.70:0/3882903809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb408102200 0x7fb40819ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.066+0000 7fb407fff640 1 -- 172.21.15.70:0/3882903809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb408100650 msgr2=0x7fb408069440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:29.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.066+0000 7fb407fff640 1 --2- 172.21.15.70:0/3882903809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb408100650 0x7fb408069440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.066+0000 7fb407fff640 1 -- 172.21.15.70:0/3882903809 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40819f5f0 con 0x7fb408101840 2024-04-01T01:49:29.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.066+0000 7fb407fff640 1 --2- 172.21.15.70:0/3882903809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb408101840 0x7fb40819c6f0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fb3f400e9f0 tx=0x7fb3f400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:29.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.071+0000 7fb405ffb640 1 -- 172.21.15.70:0/3882903809 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f400cd80 con 0x7fb408101840 2024-04-01T01:49:29.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.071+0000 7fb405ffb640 1 -- 172.21.15.70:0/3882903809 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb3f40054d0 con 0x7fb408101840 2024-04-01T01:49:29.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.071+0000 7fb405ffb640 1 -- 172.21.15.70:0/3882903809 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4010430 con 0x7fb408101840 2024-04-01T01:49:29.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.072+0000 7fb40eef4640 1 -- 172.21.15.70:0/3882903809 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb40819f8e0 con 0x7fb408101840 2024-04-01T01:49:29.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.072+0000 7fb40eef4640 1 -- 172.21.15.70:0/3882903809 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb40819fde0 con 0x7fb408101840 2024-04-01T01:49:29.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.075+0000 7fb411bb1740 1 Processor -- start 2024-04-01T01:49:29.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.078+0000 7fb405ffb640 1 -- 172.21.15.70:0/3882903809 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb3f4005020 con 0x7fb408101840 2024-04-01T01:49:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:28 smithi081 ceph-mon[35142]: pgmap v839: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 12 MiB/s wr, 755 op/s 2024-04-01T01:49:29.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.079+0000 7fb405ffb640 1 --2- 172.21.15.70:0/3882903809 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3f80a2670 0x7fb3f80a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.079+0000 7fb405ffb640 1 -- 172.21.15.70:0/3882903809 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fb3f40ceee0 con 0x7fb408101840 2024-04-01T01:49:29.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.079+0000 7fb40d491640 1 --2- 172.21.15.70:0/3882903809 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3f80a2670 0x7fb3f80a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.079+0000 7fb411bb1740 1 -- start start 2024-04-01T01:49:29.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.079+0000 7fb411bb1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab3ac0 0x562e24ab3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.079+0000 7fb411bb1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24ab44b0 0x562e24ab8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.079+0000 7fb411bb1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab9620 0x562e24abbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.080+0000 7fb411bb1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562e24a10180 con 0x562e24ab3ac0 2024-04-01T01:49:29.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.080+0000 7fb411bb1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562e249b8b70 con 0x562e24ab44b0 2024-04-01T01:49:29.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.080+0000 7fb411bb1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562e24a0c220 con 0x562e24ab9620 2024-04-01T01:49:29.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.080+0000 7fb40d491640 1 --2- 172.21.15.70:0/3882903809 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3f80a2670 0x7fb3f80a4b30 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb400004af0 tx=0x7fb400005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:29.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.080+0000 7fb40d491640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab9620 0x562e24abbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.080+0000 7fb40d491640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab9620 0x562e24abbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:51344/0 (socket says 172.21.15.70:51344) 2024-04-01T01:49:29.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.080+0000 7fb40d491640 1 -- 172.21.15.70:0/3766753959 learned_addr learned my addr 172.21.15.70:0/3766753959 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:29.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.081+0000 7fb40d491640 1 -- 172.21.15.70:0/3766753959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24ab44b0 msgr2=0x562e24ab8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:29.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.081+0000 7fb407fff640 1 --2- 172.21.15.70:0/3766753959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24ab44b0 0x562e24ab8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.081+0000 7fb40d491640 1 --2- 172.21.15.70:0/3766753959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24ab44b0 0x562e24ab8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.081+0000 7fb40d491640 1 -- 172.21.15.70:0/3766753959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab3ac0 msgr2=0x562e24ab3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:29.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.081+0000 7fb40d491640 1 --2- 172.21.15.70:0/3766753959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab3ac0 0x562e24ab3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.081+0000 7fb40d491640 1 -- 172.21.15.70:0/3766753959 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e249cc7c0 con 0x562e24ab9620 2024-04-01T01:49:29.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.081+0000 7fb40d491640 1 --2- 172.21.15.70:0/3766753959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab9620 0x562e24abbab0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb400035000 tx=0x7fb40003f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92df46c6b5c4acb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:29.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.082+0000 7fb3f2ffd640 1 -- 172.21.15.70:0/3766753959 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb400003970 con 0x562e24ab9620 2024-04-01T01:49:29.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.082+0000 7fb3f2ffd640 1 -- 172.21.15.70:0/3766753959 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb400050070 con 0x562e24ab9620 2024-04-01T01:49:29.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.082+0000 7fb3f2ffd640 1 -- 172.21.15.70:0/3766753959 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4000042b0 con 0x562e24ab9620 2024-04-01T01:49:29.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.083+0000 7fb411bb1740 1 -- 172.21.15.70:0/3766753959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab9620 msgr2=0x562e24abbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:29.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.083+0000 7fb411bb1740 1 --2- 172.21.15.70:0/3766753959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab9620 0x562e24abbab0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb400035000 tx=0x7fb40003f000 comp rx=0 tx=0).stop 2024-04-01T01:49:29.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.083+0000 7fb411bb1740 1 -- 172.21.15.70:0/3766753959 shutdown_connections 2024-04-01T01:49:29.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.083+0000 7fb411bb1740 1 --2- 172.21.15.70:0/3766753959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab9620 0x562e24abbab0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.083+0000 7fb411bb1740 1 --2- 172.21.15.70:0/3766753959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24ab44b0 0x562e24ab8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.083+0000 7fb411bb1740 1 --2- 172.21.15.70:0/3766753959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab3ac0 0x562e24ab3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.083+0000 7fb411bb1740 1 -- 172.21.15.70:0/3766753959 >> 172.21.15.70:0/3766753959 conn(0x562e24aa6820 msgr2=0x562e24aa6c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:29.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.083+0000 7fb411bb1740 1 -- 172.21.15.70:0/3766753959 shutdown_connections 2024-04-01T01:49:29.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.083+0000 7fb411bb1740 1 -- 172.21.15.70:0/3766753959 wait complete. 2024-04-01T01:49:29.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.084+0000 7fb411bb1740 1 -- start start 2024-04-01T01:49:29.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.084+0000 7fb411bb1740 1 Processor -- start 2024-04-01T01:49:29.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.085+0000 7fb411bb1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24ab3ac0 0x562e24aaceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.085+0000 7fb411bb1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab44b0 0x562e24aad3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.085+0000 7fb411bb1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab9620 0x562e24ab2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.085+0000 7fb411bb1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562e24aab360 con 0x562e24ab9620 2024-04-01T01:49:29.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.085+0000 7fb40cc90640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24ab3ac0 0x562e24aaceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.085+0000 7fb411bb1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562e249ae4a0 con 0x562e24ab3ac0 2024-04-01T01:49:29.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.085+0000 7fb411bb1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562e24aab4e0 con 0x562e24ab44b0 2024-04-01T01:49:29.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.085+0000 7fb40cc90640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24ab3ac0 0x562e24aaceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49512/0 (socket says 172.21.15.70:49512) 2024-04-01T01:49:29.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.085+0000 7fb40cc90640 1 -- 172.21.15.70:0/2761621129 learned_addr learned my addr 172.21.15.70:0/2761621129 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:29.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.085+0000 7fb40d491640 1 --2- 172.21.15.70:0/2761621129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab9620 0x562e24ab2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.085+0000 7fb411bb1740 10 client.0 _ll_register_callbacks cb 0x562e249b1d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:29.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.085+0000 7fb407fff640 1 --2- 172.21.15.70:0/2761621129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab44b0 0x562e24aad3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.085+0000 7fb40cc90640 1 -- 172.21.15.70:0/2761621129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab44b0 msgr2=0x562e24aad3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:29.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.085+0000 7fb40cc90640 1 --2- 172.21.15.70:0/2761621129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab44b0 0x562e24aad3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.085+0000 7fb40cc90640 1 -- 172.21.15.70:0/2761621129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab9620 msgr2=0x562e24ab2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:29.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.086+0000 7fb40cc90640 1 --2- 172.21.15.70:0/2761621129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab9620 0x562e24ab2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.086+0000 7fb40cc90640 1 -- 172.21.15.70:0/2761621129 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e247ee7d0 con 0x562e24ab3ac0 2024-04-01T01:49:29.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.086+0000 7fb40cc90640 1 --2- 172.21.15.70:0/2761621129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24ab3ac0 0x562e24aaceb0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00e3f0 tx=0x7fb3fc00e8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc0fda6c395e1b15 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:29.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.086+0000 7fb3e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:49:29.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.086+0000 7fb3e7fff640 1 -- 172.21.15.70:0/2761621129 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc010840 con 0x562e24ab3ac0 2024-04-01T01:49:29.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.086+0000 7fb3e7fff640 1 -- 172.21.15.70:0/2761621129 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fb3fc0109e0 con 0x562e24ab3ac0 2024-04-01T01:49:29.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.086+0000 7fb411bb1740 1 -- 172.21.15.70:0/2761621129 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb400050070 con 0x562e24ab3ac0 2024-04-01T01:49:29.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.086+0000 7fb3e7fff640 1 -- 172.21.15.70:0/2761621129 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb3fc023050 con 0x562e24ab3ac0 2024-04-01T01:49:29.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.087+0000 7fb411bb1740 10 client.30134 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:29.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.087+0000 7fb411bb1740 1 -- 172.21.15.70:0/2761621129 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562e247243f0 con 0x562e24ab3ac0 2024-04-01T01:49:29.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.088+0000 7fb3e7fff640 1 -- 172.21.15.70:0/2761621129 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb3fc010be0 con 0x562e24ab3ac0 2024-04-01T01:49:29.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.088+0000 7fb3e7fff640 1 -- 172.21.15.70:0/2761621129 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fb3fc0173d0 con 0x562e24ab3ac0 2024-04-01T01:49:29.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.088+0000 7fb411bb1740 10 client.30134 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:29.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.088+0000 7fb411bb1740 10 client.30134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:49:29.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.088+0000 7fb411bb1740 20 client.30134 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:29.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.088+0000 7fb411bb1740 1 --2- 172.21.15.70:0/2761621129 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562e24ad25f0 0x562e24ad4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.088+0000 7fb411bb1740 4 client.30134 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:49:29.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.088+0000 7fb411bb1740 1 -- 172.21.15.70:0/2761621129 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb400050070 con 0x562e24ad25f0 2024-04-01T01:49:29.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.088+0000 7fb407fff640 1 --2- 172.21.15.70:0/2761621129 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562e24ad25f0 0x562e24ad4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.089+0000 7fb3e7fff640 10 client.30134 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:29.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.089+0000 7fb407fff640 1 --2- 172.21.15.70:0/2761621129 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562e24ad25f0 0x562e24ad4a90 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=422366eabbe709b9 server_cookie=9ee36c14ce0e6b98 in_seq=0 out_seq=0 2024-04-01T01:49:29.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.093+0000 7fb3e7fff640 1 -- 172.21.15.70:0/2761621129 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fb400050070 con 0x562e24ad25f0 2024-04-01T01:49:29.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.093+0000 7fb3e7fff640 10 client.30134 handle_command_reply: tid=0 2024-04-01T01:49:29.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.094+0000 7fb411bb1740 1 client.30134 shutdown 2024-04-01T01:49:29.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.100+0000 7fb411bb1740 1 -- 172.21.15.70:0/2761621129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24ab3ac0 msgr2=0x562e24aaceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:29.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.100+0000 7fb411bb1740 1 --2- 172.21.15.70:0/2761621129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24ab3ac0 0x562e24aaceb0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00e3f0 tx=0x7fb3fc00e8b0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.100+0000 7fb411bb1740 1 -- 172.21.15.70:0/2761621129 shutdown_connections 2024-04-01T01:49:29.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.100+0000 7fb411bb1740 1 --2- 172.21.15.70:0/2761621129 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562e24ad25f0 0x562e24ad4a90 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.100+0000 7fb411bb1740 1 --2- 172.21.15.70:0/2761621129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab9620 0x562e24ab2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.100+0000 7fb411bb1740 1 --2- 172.21.15.70:0/2761621129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab44b0 0x562e24aad3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.100+0000 7fb411bb1740 1 --2- 172.21.15.70:0/2761621129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24ab3ac0 0x562e24aaceb0 secure :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00e3f0 tx=0x7fb3fc00e8b0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.100+0000 7fb3e7fff640 0 client.30134 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:29.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.100+0000 7fb411bb1740 1 -- 172.21.15.70:0/2761621129 >> 172.21.15.70:0/2761621129 conn(0x562e24aa6820 msgr2=0x562e24aa5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:29.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.105+0000 7fb411bb1740 1 -- 172.21.15.70:0/2761621129 shutdown_connections 2024-04-01T01:49:29.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.108+0000 7fb411bb1740 1 -- 172.21.15.70:0/2761621129 wait complete. 2024-04-01T01:49:29.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.109+0000 7fb411bb1740 20 client.30134 trim_cache size 0 max 16384 2024-04-01T01:49:29.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.157+0000 7fb411bb1740 1 Processor -- start 2024-04-01T01:49:29.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.158+0000 7fb411bb1740 1 -- start start 2024-04-01T01:49:29.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.158+0000 7fb411bb1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab44b0 0x562e24aa6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.158+0000 7fb411bb1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab9620 0x562e24aac500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.158+0000 7fb411bb1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24aaca40 0x562e24aacec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.158+0000 7fb411bb1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081a8300 con 0x562e24ab44b0 2024-04-01T01:49:29.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.158+0000 7fb411bb1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081a9e40 con 0x562e24aaca40 2024-04-01T01:49:29.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.158+0000 7fb411bb1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081a89d0 con 0x562e24ab9620 2024-04-01T01:49:29.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.158+0000 7fb40d491640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24aaca40 0x562e24aacec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.158+0000 7fb40d491640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24aaca40 0x562e24aacec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49522/0 (socket says 172.21.15.70:49522) 2024-04-01T01:49:29.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.158+0000 7fb40d491640 1 -- 172.21.15.70:0/3536901009 learned_addr learned my addr 172.21.15.70:0/3536901009 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:29.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.159+0000 7fb40d491640 1 -- 172.21.15.70:0/3536901009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab9620 msgr2=0x562e24aac500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:49:29.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.159+0000 7fb40d491640 1 --2- 172.21.15.70:0/3536901009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab9620 0x562e24aac500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.159+0000 7fb40d491640 1 -- 172.21.15.70:0/3536901009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab44b0 msgr2=0x562e24aa6c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:49:29.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.159+0000 7fb40d491640 1 --2- 172.21.15.70:0/3536901009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab44b0 0x562e24aa6c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.159+0000 7fb40d491640 1 -- 172.21.15.70:0/3536901009 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e246e40f0 con 0x562e24aaca40 2024-04-01T01:49:29.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.160+0000 7fb40d491640 1 --2- 172.21.15.70:0/3536901009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24aaca40 0x562e24aacec0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fb40002c430 tx=0x7fb400009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b6b42292fdd6ecf server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:29.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.163+0000 7fb3f27fc640 1 -- 172.21.15.70:0/3536901009 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb400003c30 con 0x562e24aaca40 2024-04-01T01:49:29.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.163+0000 7fb3f27fc640 1 -- 172.21.15.70:0/3536901009 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb400050040 con 0x562e24aaca40 2024-04-01T01:49:29.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.163+0000 7fb3f27fc640 1 -- 172.21.15.70:0/3536901009 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb400034690 con 0x562e24aaca40 2024-04-01T01:49:29.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.164+0000 7fb411bb1740 1 -- 172.21.15.70:0/3536901009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24aaca40 msgr2=0x562e24aacec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:29.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.164+0000 7fb411bb1740 1 --2- 172.21.15.70:0/3536901009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24aaca40 0x562e24aacec0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fb40002c430 tx=0x7fb400009500 comp rx=0 tx=0).stop 2024-04-01T01:49:29.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.164+0000 7fb411bb1740 1 -- 172.21.15.70:0/3536901009 shutdown_connections 2024-04-01T01:49:29.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.164+0000 7fb411bb1740 1 --2- 172.21.15.70:0/3536901009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24aaca40 0x562e24aacec0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.164+0000 7fb411bb1740 1 --2- 172.21.15.70:0/3536901009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab9620 0x562e24aac500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.164+0000 7fb411bb1740 1 --2- 172.21.15.70:0/3536901009 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab44b0 0x562e24aa6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.164+0000 7fb411bb1740 1 -- 172.21.15.70:0/3536901009 >> 172.21.15.70:0/3536901009 conn(0x562e24aa6820 msgr2=0x562e24aa5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:29.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.165+0000 7fb411bb1740 1 -- 172.21.15.70:0/3536901009 shutdown_connections 2024-04-01T01:49:29.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.165+0000 7fb411bb1740 1 -- 172.21.15.70:0/3536901009 wait complete. 2024-04-01T01:49:29.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.165+0000 7fb411bb1740 1 -- start start 2024-04-01T01:49:29.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.165+0000 7fb411bb1740 1 Processor -- start 2024-04-01T01:49:29.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.165+0000 7fb411bb1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24aaca40 0x562e24ae54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.165+0000 7fb411bb1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab44b0 0x562e24ae59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.165+0000 7fb411bb1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab9620 0x562e24ae3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.165+0000 7fb411bb1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081a9770 con 0x562e24ab44b0 2024-04-01T01:49:29.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.165+0000 7fb411bb1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081a90a0 con 0x562e24aaca40 2024-04-01T01:49:29.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.165+0000 7fb411bb1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb4081a62e0 con 0x562e24ab9620 2024-04-01T01:49:29.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.165+0000 7fb411bb1740 10 client.0 _ll_register_callbacks cb 0x7fb3fc009cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:29.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.166+0000 7fb40cc90640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24aaca40 0x562e24ae54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.166+0000 7fb407fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab44b0 0x562e24ae59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.167+0000 7fb407fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab44b0 0x562e24ae59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48958/0 (socket says 172.21.15.70:48958) 2024-04-01T01:49:29.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.167+0000 7fb407fff640 1 -- 172.21.15.70:0/4174173891 learned_addr learned my addr 172.21.15.70:0/4174173891 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:29.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.167+0000 7fb40d491640 1 --2- 172.21.15.70:0/4174173891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab9620 0x562e24ae3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.167+0000 7fb407fff640 1 -- 172.21.15.70:0/4174173891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab9620 msgr2=0x562e24ae3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:29.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.167+0000 7fb407fff640 1 --2- 172.21.15.70:0/4174173891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab9620 0x562e24ae3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.167+0000 7fb407fff640 1 -- 172.21.15.70:0/4174173891 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24aaca40 msgr2=0x562e24ae54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:29.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.167+0000 7fb407fff640 1 --2- 172.21.15.70:0/4174173891 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24aaca40 0x562e24ae54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.167+0000 7fb407fff640 1 -- 172.21.15.70:0/4174173891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e24ae4370 con 0x562e24ab44b0 2024-04-01T01:49:29.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.167+0000 7fb407fff640 1 --2- 172.21.15.70:0/4174173891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab44b0 0x562e24ae59e0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fb3f409b270 tx=0x7fb3f409aed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675416f8e611c136 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:29.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.169+0000 7fb3e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:49:29.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.169+0000 7fb3e7fff640 1 -- 172.21.15.70:0/4174173891 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4005680 con 0x562e24ab44b0 2024-04-01T01:49:29.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.169+0000 7fb411bb1740 1 -- 172.21.15.70:0/4174173891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb400050040 con 0x562e24ab44b0 2024-04-01T01:49:29.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.169+0000 7fb3e7fff640 1 -- 172.21.15.70:0/4174173891 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fb3f40cc6c0 con 0x562e24ab44b0 2024-04-01T01:49:29.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.170+0000 7fb3e7fff640 1 -- 172.21.15.70:0/4174173891 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb3f409bc70 con 0x562e24ab44b0 2024-04-01T01:49:29.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.170+0000 7fb411bb1740 10 client.20031 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:29.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.170+0000 7fb411bb1740 1 -- 172.21.15.70:0/4174173891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562e24b38eb0 con 0x562e24ab44b0 2024-04-01T01:49:29.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.170+0000 7fb3e7fff640 1 -- 172.21.15.70:0/4174173891 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb400050040 con 0x562e24ab44b0 2024-04-01T01:49:29.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.171+0000 7fb3e7fff640 1 -- 172.21.15.70:0/4174173891 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fb3f40b4080 con 0x562e24ab44b0 2024-04-01T01:49:29.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.171+0000 7fb411bb1740 10 client.20031 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:29.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.171+0000 7fb411bb1740 10 client.20031 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:49:29.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.171+0000 7fb411bb1740 20 client.20031 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:29.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.171+0000 7fb411bb1740 1 --2- 172.21.15.70:0/4174173891 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562e24b39840 0x562e24b3bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.171+0000 7fb411bb1740 4 client.20031 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:49:29.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.171+0000 7fb411bb1740 1 -- 172.21.15.70:0/4174173891 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562e24a1d6c0 con 0x562e24b39840 2024-04-01T01:49:29.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.171+0000 7fb40cc90640 1 --2- 172.21.15.70:0/4174173891 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562e24b39840 0x562e24b3bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.174+0000 7fb3e7fff640 10 client.20031 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:29.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.175+0000 7fb40cc90640 1 --2- 172.21.15.70:0/4174173891 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562e24b39840 0x562e24b3bc60 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d9e97de178fd663 server_cookie=68f927885741b93b in_seq=0 out_seq=0 2024-04-01T01:49:29.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.175+0000 7fb3e7fff640 1 -- 172.21.15.70:0/4174173891 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562e24a1d6c0 con 0x562e24b39840 2024-04-01T01:49:29.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.175+0000 7fb3e7fff640 10 client.20031 handle_command_reply: tid=0 2024-04-01T01:49:29.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.176+0000 7fb411bb1740 1 client.20031 shutdown 2024-04-01T01:49:29.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.176+0000 7fb411bb1740 1 -- 172.21.15.70:0/4174173891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab44b0 msgr2=0x562e24ae59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:29.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.176+0000 7fb411bb1740 1 --2- 172.21.15.70:0/4174173891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab44b0 0x562e24ae59e0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fb3f409b270 tx=0x7fb3f409aed0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.181+0000 7fb411bb1740 1 -- 172.21.15.70:0/4174173891 shutdown_connections 2024-04-01T01:49:29.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.181+0000 7fb411bb1740 1 --2- 172.21.15.70:0/4174173891 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x562e24b39840 0x562e24b3bc60 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.181+0000 7fb411bb1740 1 --2- 172.21.15.70:0/4174173891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e24ab9620 0x562e24ae3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.181+0000 7fb411bb1740 1 --2- 172.21.15.70:0/4174173891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e24ab44b0 0x562e24ae59e0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.181+0000 7fb411bb1740 1 --2- 172.21.15.70:0/4174173891 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562e24aaca40 0x562e24ae54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.181+0000 7fb411bb1740 1 -- 172.21.15.70:0/4174173891 >> 172.21.15.70:0/4174173891 conn(0x562e24aa6820 msgr2=0x562e24aa5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:29.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.181+0000 7fb3e7fff640 0 client.20031 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:29.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.182+0000 7fb411bb1740 1 -- 172.21.15.70:0/4174173891 shutdown_connections 2024-04-01T01:49:29.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.183+0000 7fb411bb1740 1 -- 172.21.15.70:0/4174173891 wait complete. 2024-04-01T01:49:29.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.183+0000 7fb411bb1740 20 client.20031 trim_cache size 0 max 16384 2024-04-01T01:49:29.187 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:49:29.187 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:49:29.187 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:49:29.187 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:49:29.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.192+0000 7fb40eef4640 1 -- 172.21.15.70:0/3882903809 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3f80a2670 msgr2=0x7fb3f80a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:29.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.192+0000 7fb40eef4640 1 --2- 172.21.15.70:0/3882903809 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3f80a2670 0x7fb3f80a4b30 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb400004af0 tx=0x7fb400005fd0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.192+0000 7fb40eef4640 1 -- 172.21.15.70:0/3882903809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb408101840 msgr2=0x7fb40819c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:29.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.192+0000 7fb40eef4640 1 --2- 172.21.15.70:0/3882903809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb408101840 0x7fb40819c6f0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fb3f400e9f0 tx=0x7fb3f400edf0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.192+0000 7fb40eef4640 1 -- 172.21.15.70:0/3882903809 shutdown_connections 2024-04-01T01:49:29.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.192+0000 7fb40eef4640 1 --2- 172.21.15.70:0/3882903809 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb3f80a2670 0x7fb3f80a4b30 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.192+0000 7fb40eef4640 1 --2- 172.21.15.70:0/3882903809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb408102200 0x7fb40819ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.192+0000 7fb40eef4640 1 --2- 172.21.15.70:0/3882903809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb408101840 0x7fb40819c6f0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.192+0000 7fb40eef4640 1 --2- 172.21.15.70:0/3882903809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb408100650 0x7fb408069440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.192+0000 7fb40eef4640 1 -- 172.21.15.70:0/3882903809 >> 172.21.15.70:0/3882903809 conn(0x7fb4080fbbe0 msgr2=0x7fb408104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:29.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.192+0000 7fb40eef4640 1 -- 172.21.15.70:0/3882903809 shutdown_connections 2024-04-01T01:49:29.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.192+0000 7fb40eef4640 1 -- 172.21.15.70:0/3882903809 wait complete. 2024-04-01T01:49:29.207 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:49:29.207 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:49:29.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:28 smithi189 ceph-mon[35753]: pgmap v839: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 12 MiB/s wr, 755 op/s 2024-04-01T01:49:29.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.383+0000 7f06f98dc640 1 -- 172.21.15.70:0/115949822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06f41093b0 msgr2=0x7f06f4109830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:29.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.383+0000 7f06f98dc640 1 --2- 172.21.15.70:0/115949822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06f41093b0 0x7f06f4109830 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f06e800d990 tx=0x7f06e802dae0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.384+0000 7f06f98dc640 1 -- 172.21.15.70:0/115949822 shutdown_connections 2024-04-01T01:49:29.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.384+0000 7f06f98dc640 1 --2- 172.21.15.70:0/115949822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06f4109e50 0x7f06f41109b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.384+0000 7f06f98dc640 1 --2- 172.21.15.70:0/115949822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06f41093b0 0x7f06f4109830 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.384+0000 7f06f98dc640 1 --2- 172.21.15.70:0/115949822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06f41089c0 0x7f06f4108de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.384+0000 7f06f98dc640 1 -- 172.21.15.70:0/115949822 >> 172.21.15.70:0/115949822 conn(0x7f06f406e760 msgr2=0x7f06f4070ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:29.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.384+0000 7f06f98dc640 1 -- 172.21.15.70:0/115949822 shutdown_connections 2024-04-01T01:49:29.384 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.384+0000 7f06f98dc640 1 -- 172.21.15.70:0/115949822 wait complete. 2024-04-01T01:49:29.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.385+0000 7f06f98dc640 1 Processor -- start 2024-04-01T01:49:29.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.386+0000 7f06f98dc640 1 -- start start 2024-04-01T01:49:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.386+0000 7f06f98dc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06f41089c0 0x7f06f419de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.386+0000 7f06f98dc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06f41093b0 0x7f06f419e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.386+0000 7f06f98dc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06f4109e50 0x7f06f41a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.386+0000 7f06f98dc640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f06f4113720 con 0x7f06f4109e50 2024-04-01T01:49:29.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.386+0000 7f06f98dc640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f06f41135a0 con 0x7f06f41089c0 2024-04-01T01:49:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.386+0000 7f06f98dc640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f06f41138a0 con 0x7f06f41093b0 2024-04-01T01:49:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.386+0000 7f06f88da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06f41089c0 0x7f06f419de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.386+0000 7f06f88da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06f41089c0 0x7f06f419de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49538/0 (socket says 172.21.15.70:49538) 2024-04-01T01:49:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.386+0000 7f06f88da640 1 -- 172.21.15.70:0/3983928844 learned_addr learned my addr 172.21.15.70:0/3983928844 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.386+0000 7f06f90db640 1 --2- 172.21.15.70:0/3983928844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06f4109e50 0x7f06f41a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.386+0000 7f06f3fff640 1 --2- 172.21.15.70:0/3983928844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06f41093b0 0x7f06f419e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.387+0000 7f06f88da640 1 -- 172.21.15.70:0/3983928844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06f41093b0 msgr2=0x7f06f419e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.387+0000 7f06f88da640 1 --2- 172.21.15.70:0/3983928844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06f41093b0 0x7f06f419e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.387+0000 7f06f88da640 1 -- 172.21.15.70:0/3983928844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06f4109e50 msgr2=0x7f06f41a53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.387+0000 7f06f88da640 1 --2- 172.21.15.70:0/3983928844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06f4109e50 0x7f06f41a53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.387+0000 7f06f88da640 1 -- 172.21.15.70:0/3983928844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06f41a5a50 con 0x7f06f41089c0 2024-04-01T01:49:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.387+0000 7f06f88da640 1 --2- 172.21.15.70:0/3983928844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06f41089c0 0x7f06f419de30 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f06ec009650 tx=0x7f06ec007590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:29.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.387+0000 7f06f1ffb640 1 -- 172.21.15.70:0/3983928844 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ec014740 con 0x7f06f41089c0 2024-04-01T01:49:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.387+0000 7f06f1ffb640 1 -- 172.21.15.70:0/3983928844 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f06ec018070 con 0x7f06f41089c0 2024-04-01T01:49:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.387+0000 7f06f1ffb640 1 -- 172.21.15.70:0/3983928844 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ec01cb20 con 0x7f06f41089c0 2024-04-01T01:49:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.387+0000 7f06f98dc640 1 -- 172.21.15.70:0/3983928844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06f41a5d40 con 0x7f06f41089c0 2024-04-01T01:49:29.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.387+0000 7f06f98dc640 1 -- 172.21.15.70:0/3983928844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06f41a6150 con 0x7f06f41089c0 2024-04-01T01:49:29.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.390+0000 7f06f1ffb640 1 -- 172.21.15.70:0/3983928844 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f06ec01a030 con 0x7f06f41089c0 2024-04-01T01:49:29.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.391+0000 7f06f98dc640 1 -- 172.21.15.70:0/3983928844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06f410f950 con 0x7f06f41089c0 2024-04-01T01:49:29.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.391+0000 7f06f1ffb640 1 --2- 172.21.15.70:0/3983928844 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06dc0a25f0 0x7f06dc0a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:29.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.391+0000 7f06f1ffb640 1 -- 172.21.15.70:0/3983928844 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f06ec0d2920 con 0x7f06f41089c0 2024-04-01T01:49:29.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.392+0000 7f06f3fff640 1 --2- 172.21.15.70:0/3983928844 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06dc0a25f0 0x7f06dc0a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:29.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.392+0000 7f06f3fff640 1 --2- 172.21.15.70:0/3983928844 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06dc0a25f0 0x7f06dc0a4ab0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f06e802e0f0 tx=0x7f06e800d590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:29.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.398+0000 7f06f1ffb640 1 -- 172.21.15.70:0/3983928844 <== mon.2 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f06ec071510 con 0x7f06f41089c0 2024-04-01T01:49:29.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.817+0000 7f06f98dc640 1 -- 172.21.15.70:0/3983928844 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06f402d170 con 0x7f06f41089c0 2024-04-01T01:49:29.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.818+0000 7f06f1ffb640 1 -- 172.21.15.70:0/3983928844 <== mon.2 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11926 (secure 0 0 0) 0x7f06f402d170 con 0x7f06f41089c0 2024-04-01T01:49:29.818 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:29.818 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:49:12.699222+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_29833":{"gid":29833,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/921108936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":921108936},{"type":"v1","addr":"172.21.15.189:6837","nonce":921108936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24418,24404,24379,24451,24464]},"id":1}]} 2024-04-01T01:49:29.818 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 33 2024-04-01T01:49:29.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.821+0000 7f06f98dc640 1 -- 172.21.15.70:0/3983928844 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06dc0a25f0 msgr2=0x7f06dc0a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:29.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.821+0000 7f06f98dc640 1 --2- 172.21.15.70:0/3983928844 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06dc0a25f0 0x7f06dc0a4ab0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f06e802e0f0 tx=0x7f06e800d590 comp rx=0 tx=0).stop 2024-04-01T01:49:29.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.821+0000 7f06f98dc640 1 -- 172.21.15.70:0/3983928844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06f41089c0 msgr2=0x7f06f419de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:29.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.821+0000 7f06f98dc640 1 --2- 172.21.15.70:0/3983928844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06f41089c0 0x7f06f419de30 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f06ec009650 tx=0x7f06ec007590 comp rx=0 tx=0).stop 2024-04-01T01:49:29.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.822+0000 7f06f98dc640 1 -- 172.21.15.70:0/3983928844 shutdown_connections 2024-04-01T01:49:29.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.822+0000 7f06f98dc640 1 --2- 172.21.15.70:0/3983928844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06f4109e50 0x7f06f41a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.822+0000 7f06f98dc640 1 --2- 172.21.15.70:0/3983928844 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f06dc0a25f0 0x7f06dc0a4ab0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.822+0000 7f06f98dc640 1 --2- 172.21.15.70:0/3983928844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06f41093b0 0x7f06f419e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.822+0000 7f06f98dc640 1 --2- 172.21.15.70:0/3983928844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06f41089c0 0x7f06f419de30 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:29.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.822+0000 7f06f98dc640 1 -- 172.21.15.70:0/3983928844 >> 172.21.15.70:0/3983928844 conn(0x7f06f406e760 msgr2=0x7f06f410cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:29.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.822+0000 7f06f98dc640 1 -- 172.21.15.70:0/3983928844 shutdown_connections 2024-04-01T01:49:29.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:29.822+0000 7f06f98dc640 1 -- 172.21.15.70:0/3983928844 wait complete. 2024-04-01T01:49:29.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:49:30.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.023+0000 7f0dd5c1c640 1 -- 172.21.15.70:0/733277792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dc80a2330 msgr2=0x7f0dc80a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.023+0000 7f0dd5c1c640 1 --2- 172.21.15.70:0/733277792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dc80a2330 0x7f0dc80a4720 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f0dd004ff20 tx=0x7f0dd0097860 comp rx=0 tx=0).stop 2024-04-01T01:49:30.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.023+0000 7f0dd5c1c640 1 -- 172.21.15.70:0/733277792 shutdown_connections 2024-04-01T01:49:30.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.023+0000 7f0dd5c1c640 1 --2- 172.21.15.70:0/733277792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dc80a2330 0x7f0dc80a4720 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.023+0000 7f0dd5c1c640 1 --2- 172.21.15.70:0/733277792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc8008220 0x7f0dc80086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.023+0000 7f0dd5c1c640 1 --2- 172.21.15.70:0/733277792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc8009800 0x7f0dc8007c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.023+0000 7f0dd5c1c640 1 -- 172.21.15.70:0/733277792 >> 172.21.15.70:0/733277792 conn(0x7f0dc80912f0 msgr2=0x7f0dc8093710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.023+0000 7f0dd5c1c640 1 -- 172.21.15.70:0/733277792 shutdown_connections 2024-04-01T01:49:30.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.025+0000 7f0dd5c1c640 1 -- 172.21.15.70:0/733277792 wait complete. 2024-04-01T01:49:30.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.026+0000 7f0dd5c1c640 1 Processor -- start 2024-04-01T01:49:30.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.027+0000 7f0dd5c1c640 1 -- start start 2024-04-01T01:49:30.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.027+0000 7f0dd5c1c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dc8008220 0x7f0dc809ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.027+0000 7f0dd5c1c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc8009800 0x7f0dc809f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.027+0000 7f0dd5c1c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc80a2330 0x7f0dc8142500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.027+0000 7f0dd5c1c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc80a7df0 con 0x7f0dc80a2330 2024-04-01T01:49:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.027+0000 7f0dd5c1c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc80a7c70 con 0x7f0dc8008220 2024-04-01T01:49:30.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.027+0000 7f0dd5c1c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc80a7f70 con 0x7f0dc8009800 2024-04-01T01:49:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.028+0000 7f0dceffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc8009800 0x7f0dc809f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.028+0000 7f0dceffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc8009800 0x7f0dc809f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:51386/0 (socket says 172.21.15.70:51386) 2024-04-01T01:49:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.028+0000 7f0dceffd640 1 -- 172.21.15.70:0/987312845 learned_addr learned my addr 172.21.15.70:0/987312845 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.028+0000 7f0dcffff640 1 --2- 172.21.15.70:0/987312845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc80a2330 0x7f0dc8142500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.029+0000 7f0dceffd640 1 -- 172.21.15.70:0/987312845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dc8008220 msgr2=0x7f0dc809ebc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:49:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.029+0000 7f0dceffd640 1 --2- 172.21.15.70:0/987312845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dc8008220 0x7f0dc809ebc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.029+0000 7f0dceffd640 1 -- 172.21.15.70:0/987312845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc80a2330 msgr2=0x7f0dc8142500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.029+0000 7f0dceffd640 1 --2- 172.21.15.70:0/987312845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc80a2330 0x7f0dc8142500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.029+0000 7f0dceffd640 1 -- 172.21.15.70:0/987312845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc8153a50 con 0x7f0dc8009800 2024-04-01T01:49:30.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.029+0000 7f0dceffd640 1 --2- 172.21.15.70:0/987312845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc8009800 0x7f0dc809f100 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc00b4a0 tx=0x7f0dbc00b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.030+0000 7f0dccff9640 1 -- 172.21.15.70:0/987312845 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc014070 con 0x7f0dc8009800 2024-04-01T01:49:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.030+0000 7f0dccff9640 1 -- 172.21.15.70:0/987312845 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0dbc010ac0 con 0x7f0dc8009800 2024-04-01T01:49:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.030+0000 7f0dd5c1c640 1 -- 172.21.15.70:0/987312845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc8153ce0 con 0x7f0dc8009800 2024-04-01T01:49:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.030+0000 7f0dccff9640 1 -- 172.21.15.70:0/987312845 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc017c00 con 0x7f0dc8009800 2024-04-01T01:49:30.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.030+0000 7f0dd5c1c640 1 -- 172.21.15.70:0/987312845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc8154250 con 0x7f0dc8009800 2024-04-01T01:49:30.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.034+0000 7f0dd8801740 1 Processor -- start 2024-04-01T01:49:30.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.034+0000 7f0dd8801740 1 -- start start 2024-04-01T01:49:30.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.035+0000 7f0dd8801740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b3a6a0 0x5616a2b3aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.035+0000 7f0dd8801740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b3b090 0x5616a2b3fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.035+0000 7f0dd8801740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b40200 0x5616a2b42690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.035+0000 7f0dd8801740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5616a2a9f1b0 con 0x5616a2b3a6a0 2024-04-01T01:49:30.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.035+0000 7f0dd8801740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5616a2a47c00 con 0x5616a2b40200 2024-04-01T01:49:30.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.035+0000 7f0dd8801740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5616a2a9b250 con 0x5616a2b3b090 2024-04-01T01:49:30.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.035+0000 7f0dcffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b3a6a0 0x5616a2b3aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.035+0000 7f0dcffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b3a6a0 0x5616a2b3aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48990/0 (socket says 172.21.15.70:48990) 2024-04-01T01:49:30.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.035+0000 7f0dcffff640 1 -- 172.21.15.70:0/2286749540 learned_addr learned my addr 172.21.15.70:0/2286749540 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.036+0000 7f0dcffff640 1 -- 172.21.15.70:0/2286749540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b3b090 msgr2=0x5616a2b3fb90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:49:30.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.036+0000 7f0dcffff640 1 --2- 172.21.15.70:0/2286749540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b3b090 0x5616a2b3fb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.036+0000 7f0dcffff640 1 -- 172.21.15.70:0/2286749540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b40200 msgr2=0x5616a2b42690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:49:30.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.036+0000 7f0dcffff640 1 --2- 172.21.15.70:0/2286749540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b40200 0x5616a2b42690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.036+0000 7f0dcffff640 1 -- 172.21.15.70:0/2286749540 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616a2a673d0 con 0x5616a2b3a6a0 2024-04-01T01:49:30.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.036+0000 7f0dccff9640 1 -- 172.21.15.70:0/987312845 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f0dbc017070 con 0x7f0dc8009800 2024-04-01T01:49:30.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.037+0000 7f0dcffff640 1 --2- 172.21.15.70:0/2286749540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b3a6a0 0x5616a2b3aac0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0066e80 tx=0x7f0dd00679a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9cf4c08c9d76163 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.037+0000 7f0db9ffb640 1 -- 172.21.15.70:0/2286749540 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd00a3070 con 0x5616a2b3a6a0 2024-04-01T01:49:30.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.038+0000 7f0db9ffb640 1 -- 172.21.15.70:0/2286749540 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0dd009e710 con 0x5616a2b3a6a0 2024-04-01T01:49:30.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.038+0000 7f0dccff9640 1 --2- 172.21.15.70:0/987312845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0dc00a25f0 0x7f0dc00a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.038+0000 7f0dccff9640 1 -- 172.21.15.70:0/987312845 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f0dbc0cdec0 con 0x7f0dc8009800 2024-04-01T01:49:30.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.038+0000 7f0dcf7fe640 1 --2- 172.21.15.70:0/987312845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0dc00a25f0 0x7f0dc00a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.038+0000 7f0db9ffb640 1 -- 172.21.15.70:0/2286749540 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd006c040 con 0x5616a2b3a6a0 2024-04-01T01:49:30.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.039+0000 7f0dd8801740 1 -- 172.21.15.70:0/2286749540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b3a6a0 msgr2=0x5616a2b3aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.039+0000 7f0dd8801740 1 --2- 172.21.15.70:0/2286749540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b3a6a0 0x5616a2b3aac0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0066e80 tx=0x7f0dd00679a0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.039+0000 7f0dd8801740 1 -- 172.21.15.70:0/2286749540 shutdown_connections 2024-04-01T01:49:30.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.039+0000 7f0dd8801740 1 --2- 172.21.15.70:0/2286749540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b40200 0x5616a2b42690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.039+0000 7f0dcf7fe640 1 --2- 172.21.15.70:0/987312845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0dc00a25f0 0x7f0dc00a4ab0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x5616a2b3b490 tx=0x7f0dc400b370 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.039+0000 7f0dd8801740 1 --2- 172.21.15.70:0/2286749540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b3b090 0x5616a2b3fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.039+0000 7f0dd8801740 1 --2- 172.21.15.70:0/2286749540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b3a6a0 0x5616a2b3aac0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.039+0000 7f0dd8801740 1 -- 172.21.15.70:0/2286749540 >> 172.21.15.70:0/2286749540 conn(0x5616a2b35c10 msgr2=0x5616a2b36010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.039+0000 7f0dd8801740 1 -- 172.21.15.70:0/2286749540 shutdown_connections 2024-04-01T01:49:30.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.039+0000 7f0dd8801740 1 -- 172.21.15.70:0/2286749540 wait complete. 2024-04-01T01:49:30.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.039+0000 7f0dd8801740 1 -- start start 2024-04-01T01:49:30.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.040+0000 7f0dd8801740 1 Processor -- start 2024-04-01T01:49:30.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.040+0000 7f0dd8801740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3a6a0 0x5616a2b44c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.040+0000 7f0dd8801740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b3b090 0x5616a2b451c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.040+0000 7f0dd8801740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b40200 0x5616a2b4c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.040+0000 7f0dd8801740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5616a2b431a0 con 0x5616a2b3b090 2024-04-01T01:49:30.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.040+0000 7f0dd8801740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5616a2a3d470 con 0x5616a2b3a6a0 2024-04-01T01:49:30.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.040+0000 7f0dd8801740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5616a2b43320 con 0x5616a2b40200 2024-04-01T01:49:30.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.040+0000 7f0dd8801740 10 client.0 _ll_register_callbacks cb 0x5616a2a5acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:30.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.040+0000 7f0dceffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b3b090 0x5616a2b451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.040+0000 7f0dcf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3a6a0 0x5616a2b44c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.040+0000 7f0dceffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b3b090 0x5616a2b451c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49002/0 (socket says 172.21.15.70:49002) 2024-04-01T01:49:30.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.040+0000 7f0dceffd640 1 -- 172.21.15.70:0/1324920514 learned_addr learned my addr 172.21.15.70:0/1324920514 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.041+0000 7f0dcffff640 1 --2- 172.21.15.70:0/1324920514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b40200 0x5616a2b4c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.041+0000 7f0dcf7fe640 1 -- 172.21.15.70:0/1324920514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b40200 msgr2=0x5616a2b4c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.041+0000 7f0dcf7fe640 1 --2- 172.21.15.70:0/1324920514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b40200 0x5616a2b4c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.041+0000 7f0dcf7fe640 1 -- 172.21.15.70:0/1324920514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b3b090 msgr2=0x5616a2b451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.041+0000 7f0dcf7fe640 1 --2- 172.21.15.70:0/1324920514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b3b090 0x5616a2b451c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.041+0000 7f0dcf7fe640 1 -- 172.21.15.70:0/1324920514 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616a27b33f0 con 0x5616a2b3a6a0 2024-04-01T01:49:30.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.041+0000 7f0dcf7fe640 1 --2- 172.21.15.70:0/1324920514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3a6a0 0x5616a2b44c80 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4005fd0 tx=0x7f0dc4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b8fb50ea13171ac server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.041+0000 7f0daeffd640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:49:30.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.041+0000 7f0daeffd640 1 -- 172.21.15.70:0/1324920514 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc4021020 con 0x5616a2b3a6a0 2024-04-01T01:49:30.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.041+0000 7f0daeffd640 1 -- 172.21.15.70:0/1324920514 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f0dc401d440 con 0x5616a2b3a6a0 2024-04-01T01:49:30.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.042+0000 7f0dd8801740 1 -- 172.21.15.70:0/1324920514 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0dd009e710 con 0x5616a2b3a6a0 2024-04-01T01:49:30.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.042+0000 7f0daeffd640 1 -- 172.21.15.70:0/1324920514 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0dc400dea0 con 0x5616a2b3a6a0 2024-04-01T01:49:30.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.042+0000 7f0dd8801740 10 client.30149 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:30.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.042+0000 7f0dd8801740 1 -- 172.21.15.70:0/1324920514 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5616a287d7d0 con 0x5616a2b3a6a0 2024-04-01T01:49:30.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.042+0000 7f0daeffd640 1 -- 172.21.15.70:0/1324920514 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0dd009e710 con 0x5616a2b3a6a0 2024-04-01T01:49:30.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.042+0000 7f0daeffd640 1 -- 172.21.15.70:0/1324920514 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f0dc402e020 con 0x5616a2b3a6a0 2024-04-01T01:49:30.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.043+0000 7f0dd8801740 10 client.30149 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:30.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.043+0000 7f0dd8801740 10 client.30149 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:49:30.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.043+0000 7f0dd8801740 20 client.30149 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:30.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.043+0000 7f0dd8801740 1 --2- 172.21.15.70:0/1324920514 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5616a2b507b0 0x5616a2b52bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.043+0000 7f0dd8801740 4 client.30149 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:49:30.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.043+0000 7f0dd8801740 1 -- 172.21.15.70:0/1324920514 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616a2aac6f0 con 0x5616a2b507b0 2024-04-01T01:49:30.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.043+0000 7f0dceffd640 1 --2- 172.21.15.70:0/1324920514 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5616a2b507b0 0x5616a2b52bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.044+0000 7f0daeffd640 10 client.30149 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:49:30.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.044+0000 7f0dceffd640 1 --2- 172.21.15.70:0/1324920514 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5616a2b507b0 0x5616a2b52bf0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d14be0c9d3f9bf91 server_cookie=22fe4b73be1e9114 in_seq=0 out_seq=0 2024-04-01T01:49:30.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.048+0000 7f0daeffd640 1 -- 172.21.15.70:0/1324920514 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5616a2aac6f0 con 0x5616a2b507b0 2024-04-01T01:49:30.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.048+0000 7f0daeffd640 10 client.30149 handle_command_reply: tid=0 2024-04-01T01:49:30.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.048+0000 7f0dd8801740 1 client.30149 shutdown 2024-04-01T01:49:30.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.049+0000 7f0dd8801740 1 -- 172.21.15.70:0/1324920514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3a6a0 msgr2=0x5616a2b44c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.049+0000 7f0dd8801740 1 --2- 172.21.15.70:0/1324920514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3a6a0 0x5616a2b44c80 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4005fd0 tx=0x7f0dc4004730 comp rx=0 tx=0).stop 2024-04-01T01:49:30.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.049+0000 7f0dd8801740 1 -- 172.21.15.70:0/1324920514 shutdown_connections 2024-04-01T01:49:30.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.049+0000 7f0dd8801740 1 --2- 172.21.15.70:0/1324920514 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5616a2b507b0 0x5616a2b52bf0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.049+0000 7f0dd8801740 1 --2- 172.21.15.70:0/1324920514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b40200 0x5616a2b4c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.049+0000 7f0dd8801740 1 --2- 172.21.15.70:0/1324920514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b3b090 0x5616a2b451c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.049+0000 7f0dd8801740 1 --2- 172.21.15.70:0/1324920514 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3a6a0 0x5616a2b44c80 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.049+0000 7f0dd8801740 1 -- 172.21.15.70:0/1324920514 >> 172.21.15.70:0/1324920514 conn(0x5616a2b35c10 msgr2=0x5616a2b34f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.049+0000 7f0daeffd640 0 client.30149 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:49:30.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.049+0000 7f0dd8801740 1 -- 172.21.15.70:0/1324920514 shutdown_connections 2024-04-01T01:49:30.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.050+0000 7f0dd8801740 1 -- 172.21.15.70:0/1324920514 wait complete. 2024-04-01T01:49:30.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.050+0000 7f0dd8801740 20 client.30149 trim_cache size 0 max 16384 2024-04-01T01:49:30.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.075+0000 7f0dd8801740 1 Processor -- start 2024-04-01T01:49:30.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.075+0000 7f0dd8801740 1 -- start start 2024-04-01T01:49:30.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.075+0000 7f0dd8801740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3a6a0 0x5616a2b34700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.076+0000 7f0dd8801740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b3b090 0x5616a2b4c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.076+0000 7f0dcf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3a6a0 0x5616a2b34700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.076+0000 7f0dcf7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3a6a0 0x5616a2b34700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49572/0 (socket says 172.21.15.70:49572) 2024-04-01T01:49:30.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.076+0000 7f0dcf7fe640 1 -- 172.21.15.70:0/708985509 learned_addr learned my addr 172.21.15.70:0/708985509 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.076+0000 7f0dd8801740 1 --2- 172.21.15.70:0/708985509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b40200 0x5616a2b34c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.076+0000 7f0dd8801740 1 -- 172.21.15.70:0/708985509 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0db400fda0 con 0x5616a2b40200 2024-04-01T01:49:30.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.076+0000 7f0dd8801740 1 -- 172.21.15.70:0/708985509 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0db40118e0 con 0x5616a2b3a6a0 2024-04-01T01:49:30.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.076+0000 7f0dd8801740 1 -- 172.21.15.70:0/708985509 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0db4010470 con 0x5616a2b3b090 2024-04-01T01:49:30.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.077+0000 7f0dcffff640 1 --2- 172.21.15.70:0/708985509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b40200 0x5616a2b34c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.077+0000 7f0dcf7fe640 1 -- 172.21.15.70:0/708985509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b3b090 msgr2=0x5616a2b4c180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:49:30.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.077+0000 7f0dcf7fe640 1 --2- 172.21.15.70:0/708985509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b3b090 0x5616a2b4c180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.077+0000 7f0dcf7fe640 1 -- 172.21.15.70:0/708985509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b40200 msgr2=0x5616a2b34c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.077+0000 7f0dcf7fe640 1 --2- 172.21.15.70:0/708985509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b40200 0x5616a2b34c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.077+0000 7f0dcf7fe640 1 -- 172.21.15.70:0/708985509 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616a2b31670 con 0x5616a2b3a6a0 2024-04-01T01:49:30.077 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.078+0000 7f0dcf7fe640 1 --2- 172.21.15.70:0/708985509 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3a6a0 0x5616a2b34700 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4012040 tx=0x7f0dc40058b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f633103b4c35422 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.078+0000 7f0db97fa640 1 -- 172.21.15.70:0/708985509 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc4021040 con 0x5616a2b3a6a0 2024-04-01T01:49:30.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.078+0000 7f0db97fa640 1 -- 172.21.15.70:0/708985509 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0dc4016030 con 0x5616a2b3a6a0 2024-04-01T01:49:30.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.078+0000 7f0db97fa640 1 -- 172.21.15.70:0/708985509 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc401de00 con 0x5616a2b3a6a0 2024-04-01T01:49:30.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.079+0000 7f0dd8801740 1 -- 172.21.15.70:0/708985509 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3a6a0 msgr2=0x5616a2b34700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.079+0000 7f0dd8801740 1 --2- 172.21.15.70:0/708985509 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3a6a0 0x5616a2b34700 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4012040 tx=0x7f0dc40058b0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.080+0000 7f0dd8801740 1 -- 172.21.15.70:0/708985509 shutdown_connections 2024-04-01T01:49:30.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.080+0000 7f0dd8801740 1 --2- 172.21.15.70:0/708985509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b40200 0x5616a2b34c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.080+0000 7f0dd8801740 1 --2- 172.21.15.70:0/708985509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b3b090 0x5616a2b4c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.080+0000 7f0dd8801740 1 --2- 172.21.15.70:0/708985509 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3a6a0 0x5616a2b34700 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.080+0000 7f0dd8801740 1 -- 172.21.15.70:0/708985509 >> 172.21.15.70:0/708985509 conn(0x5616a2b35c10 msgr2=0x5616a2b44c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.083+0000 7f0dd8801740 1 -- 172.21.15.70:0/708985509 shutdown_connections 2024-04-01T01:49:30.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.083+0000 7f0dd8801740 1 -- 172.21.15.70:0/708985509 wait complete. 2024-04-01T01:49:30.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.083+0000 7f0dd8801740 1 -- start start 2024-04-01T01:49:30.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.083+0000 7f0dd8801740 1 Processor -- start 2024-04-01T01:49:30.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.083+0000 7f0dd8801740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b3a6a0 0x5616a2b49ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.083+0000 7f0dd8801740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3b090 0x5616a2b4a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.083+0000 7f0dd8801740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b40200 0x5616a2b4b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.083+0000 7f0dd8801740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0db4011210 con 0x5616a2b40200 2024-04-01T01:49:30.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.083+0000 7f0dd8801740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0db4010b40 con 0x5616a2b3b090 2024-04-01T01:49:30.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.083+0000 7f0dd8801740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0db400dd80 con 0x5616a2b3a6a0 2024-04-01T01:49:30.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.084+0000 7f0dd8801740 10 client.0 _ll_register_callbacks cb 0x7f0dc400da90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:30.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.084+0000 7f0dcffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b40200 0x5616a2b4b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.084+0000 7f0dcffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b40200 0x5616a2b4b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49016/0 (socket says 172.21.15.70:49016) 2024-04-01T01:49:30.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.084+0000 7f0dcffff640 1 -- 172.21.15.70:0/1103618637 learned_addr learned my addr 172.21.15.70:0/1103618637 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.084+0000 7f0dceffd640 1 --2- 172.21.15.70:0/1103618637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3b090 0x5616a2b4a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.085+0000 7f0dcffff640 1 -- 172.21.15.70:0/1103618637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b3a6a0 msgr2=0x5616a2b49ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.085+0000 7f0dcffff640 1 --2- 172.21.15.70:0/1103618637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b3a6a0 0x5616a2b49ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.085+0000 7f0dcffff640 1 -- 172.21.15.70:0/1103618637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3b090 msgr2=0x5616a2b4a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.085+0000 7f0dcffff640 1 --2- 172.21.15.70:0/1103618637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3b090 0x5616a2b4a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.085+0000 7f0dcffff640 1 -- 172.21.15.70:0/1103618637 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616a2b4b610 con 0x5616a2b40200 2024-04-01T01:49:30.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.085+0000 7f0dcffff640 1 --2- 172.21.15.70:0/1103618637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b40200 0x5616a2b4b020 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0064770 tx=0x7f0dd0050420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc695a4affb7256 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.085+0000 7f0daeffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:49:30.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.086+0000 7f0daeffd640 1 -- 172.21.15.70:0/1103618637 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd006c040 con 0x5616a2b40200 2024-04-01T01:49:30.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.086+0000 7f0dd8801740 1 -- 172.21.15.70:0/1103618637 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0dc4016030 con 0x5616a2b40200 2024-04-01T01:49:30.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.086+0000 7f0daeffd640 1 -- 172.21.15.70:0/1103618637 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f0dd0073400 con 0x5616a2b40200 2024-04-01T01:49:30.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.086+0000 7f0daeffd640 1 -- 172.21.15.70:0/1103618637 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0dd00a3040 con 0x5616a2b40200 2024-04-01T01:49:30.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.091+0000 7f0dd8801740 10 client.20055 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:30.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.091+0000 7f0dd8801740 1 -- 172.21.15.70:0/1103618637 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5616a2bb6db0 con 0x5616a2b40200 2024-04-01T01:49:30.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.091+0000 7f0daeffd640 1 -- 172.21.15.70:0/1103618637 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0dc4016030 con 0x5616a2b40200 2024-04-01T01:49:30.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.092+0000 7f0daeffd640 1 -- 172.21.15.70:0/1103618637 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f0dd00b24c0 con 0x5616a2b40200 2024-04-01T01:49:30.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.092+0000 7f0dd8801740 10 client.20055 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:30.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.092+0000 7f0dd8801740 10 client.20055 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:49:30.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.092+0000 7f0dd8801740 20 client.20055 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:30.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.092+0000 7f0dd8801740 1 --2- 172.21.15.70:0/1103618637 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5616a2bb7320 0x5616a2bb9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.092+0000 7f0dd8801740 4 client.20055 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:49:30.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.092+0000 7f0dd8801740 1 -- 172.21.15.70:0/1103618637 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5616a2aa6a40 con 0x5616a2bb7320 2024-04-01T01:49:30.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.093+0000 7f0dcf7fe640 1 --2- 172.21.15.70:0/1103618637 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5616a2bb7320 0x5616a2bb9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.094+0000 7f0dcf7fe640 1 --2- 172.21.15.70:0/1103618637 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5616a2bb7320 0x5616a2bb9710 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c3264a1e01021f56 server_cookie=fef8ab21737773db in_seq=0 out_seq=0 2024-04-01T01:49:30.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.094+0000 7f0daeffd640 10 client.20055 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:49:30.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.095+0000 7f0daeffd640 1 -- 172.21.15.70:0/1103618637 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5616a2aa6a40 con 0x5616a2bb7320 2024-04-01T01:49:30.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.095+0000 7f0daeffd640 10 client.20055 handle_command_reply: tid=0 2024-04-01T01:49:30.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.095+0000 7f0dd8801740 1 client.20055 shutdown 2024-04-01T01:49:30.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.096+0000 7f0dd8801740 1 -- 172.21.15.70:0/1103618637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b40200 msgr2=0x5616a2b4b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.096+0000 7f0dd8801740 1 --2- 172.21.15.70:0/1103618637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b40200 0x5616a2b4b020 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0064770 tx=0x7f0dd0050420 comp rx=0 tx=0).stop 2024-04-01T01:49:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.097+0000 7f0dd8801740 1 -- 172.21.15.70:0/1103618637 shutdown_connections 2024-04-01T01:49:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.097+0000 7f0dd8801740 1 --2- 172.21.15.70:0/1103618637 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x5616a2bb7320 0x5616a2bb9710 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.097+0000 7f0dd8801740 1 --2- 172.21.15.70:0/1103618637 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5616a2b40200 0x5616a2b4b020 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.097+0000 7f0dd8801740 1 --2- 172.21.15.70:0/1103618637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5616a2b3b090 0x5616a2b4a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.097+0000 7f0dd8801740 1 --2- 172.21.15.70:0/1103618637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5616a2b3a6a0 0x5616a2b49ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.097+0000 7f0dd8801740 1 -- 172.21.15.70:0/1103618637 >> 172.21.15.70:0/1103618637 conn(0x5616a2b35c10 msgr2=0x5616a2b38e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.097+0000 7f0daeffd640 0 client.20055 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:49:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.098+0000 7f0dd8801740 1 -- 172.21.15.70:0/1103618637 shutdown_connections 2024-04-01T01:49:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.098+0000 7f0dd8801740 1 -- 172.21.15.70:0/1103618637 wait complete. 2024-04-01T01:49:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.098+0000 7f0dd8801740 20 client.20055 trim_cache size 0 max 16384 2024-04-01T01:49:30.098 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:49:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.099+0000 7f0dd5c1c640 1 -- 172.21.15.70:0/987312845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0dc00a25f0 msgr2=0x7f0dc00a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.099+0000 7f0dd5c1c640 1 --2- 172.21.15.70:0/987312845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0dc00a25f0 0x7f0dc00a4ab0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x5616a2b3b490 tx=0x7f0dc400b370 comp rx=0 tx=0).stop 2024-04-01T01:49:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.099+0000 7f0dd5c1c640 1 -- 172.21.15.70:0/987312845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc8009800 msgr2=0x7f0dc809f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.099+0000 7f0dd5c1c640 1 --2- 172.21.15.70:0/987312845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc8009800 0x7f0dc809f100 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc00b4a0 tx=0x7f0dbc00b960 comp rx=0 tx=0).stop 2024-04-01T01:49:30.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.099+0000 7f0dd5c1c640 1 -- 172.21.15.70:0/987312845 shutdown_connections 2024-04-01T01:49:30.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.099+0000 7f0dd5c1c640 1 --2- 172.21.15.70:0/987312845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dc80a2330 0x7f0dc8142500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.100+0000 7f0dd5c1c640 1 --2- 172.21.15.70:0/987312845 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f0dc00a25f0 0x7f0dc00a4ab0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.100+0000 7f0dd5c1c640 1 --2- 172.21.15.70:0/987312845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc8009800 0x7f0dc809f100 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.100+0000 7f0dd5c1c640 1 --2- 172.21.15.70:0/987312845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dc8008220 0x7f0dc809ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.100+0000 7f0dd5c1c640 1 -- 172.21.15.70:0/987312845 >> 172.21.15.70:0/987312845 conn(0x7f0dc80912f0 msgr2=0x7f0dc80956d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.100+0000 7f0dd5c1c640 1 -- 172.21.15.70:0/987312845 shutdown_connections 2024-04-01T01:49:30.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.100+0000 7f0dd5c1c640 1 -- 172.21.15.70:0/987312845 wait complete. 2024-04-01T01:49:30.116 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:49:30.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.320+0000 7f4e1e8d4640 1 -- 172.21.15.70:0/783090114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e1810da70 msgr2=0x7f4e1810def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.320+0000 7f4e1e8d4640 1 --2- 172.21.15.70:0/783090114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e1810da70 0x7f4e1810def0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f4e1001acb0 tx=0x7f4e1003d150 comp rx=0 tx=0).stop 2024-04-01T01:49:30.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.321+0000 7f4e1e8d4640 1 -- 172.21.15.70:0/783090114 shutdown_connections 2024-04-01T01:49:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.321+0000 7f4e1e8d4640 1 --2- 172.21.15.70:0/783090114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e1810da70 0x7f4e1810def0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.321+0000 7f4e1e8d4640 1 --2- 172.21.15.70:0/783090114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e1810b0c0 0x7f4e1810d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.321+0000 7f4e1e8d4640 1 --2- 172.21.15.70:0/783090114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e18108750 0x7f4e1810ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.321+0000 7f4e1e8d4640 1 -- 172.21.15.70:0/783090114 >> 172.21.15.70:0/783090114 conn(0x7f4e1806e5b0 msgr2=0x7f4e180709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.321+0000 7f4e1e8d4640 1 -- 172.21.15.70:0/783090114 shutdown_connections 2024-04-01T01:49:30.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.322+0000 7f4e1e8d4640 1 -- 172.21.15.70:0/783090114 wait complete. 2024-04-01T01:49:30.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.323+0000 7f4e1e8d4640 1 Processor -- start 2024-04-01T01:49:30.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.323+0000 7f4e1e8d4640 1 -- start start 2024-04-01T01:49:30.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.323+0000 7f4e1e8d4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e18108750 0x7f4e1819e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.323+0000 7f4e1e8d4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e1810b0c0 0x7f4e1819e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.323+0000 7f4e1e8d4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e1810da70 0x7f4e181a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.323+0000 7f4e1e8d4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e181118d0 con 0x7f4e1810b0c0 2024-04-01T01:49:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.323+0000 7f4e1e8d4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4e18111750 con 0x7f4e18108750 2024-04-01T01:49:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.323+0000 7f4e1e8d4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4e18111a50 con 0x7f4e1810da70 2024-04-01T01:49:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.324+0000 7f4e177fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e1810b0c0 0x7f4e1819e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.324+0000 7f4e177fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e1810b0c0 0x7f4e1819e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49034/0 (socket says 172.21.15.70:49034) 2024-04-01T01:49:30.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.324+0000 7f4e177fe640 1 -- 172.21.15.70:0/303031375 learned_addr learned my addr 172.21.15.70:0/303031375 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.324+0000 7f4e1ce71640 1 --2- 172.21.15.70:0/303031375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e1810da70 0x7f4e181a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.324+0000 7f4e1ce71640 1 -- 172.21.15.70:0/303031375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e18108750 msgr2=0x7f4e1819e030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:49:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.324+0000 7f4e1ce71640 1 --2- 172.21.15.70:0/303031375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e18108750 0x7f4e1819e030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.324+0000 7f4e1ce71640 1 -- 172.21.15.70:0/303031375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e1810b0c0 msgr2=0x7f4e1819e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.324+0000 7f4e1ce71640 1 --2- 172.21.15.70:0/303031375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e1810b0c0 0x7f4e1819e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.324+0000 7f4e1ce71640 1 -- 172.21.15.70:0/303031375 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e181a5c50 con 0x7f4e1810da70 2024-04-01T01:49:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.325+0000 7f4e1ce71640 1 --2- 172.21.15.70:0/303031375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e1810da70 0x7f4e181a55e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4e10045f50 tx=0x7f4e10043f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.325+0000 7f4e157fa640 1 -- 172.21.15.70:0/303031375 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10052020 con 0x7f4e1810da70 2024-04-01T01:49:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.325+0000 7f4e157fa640 1 -- 172.21.15.70:0/303031375 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e10004040 con 0x7f4e1810da70 2024-04-01T01:49:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.325+0000 7f4e1e8d4640 1 -- 172.21.15.70:0/303031375 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e181a5ee0 con 0x7f4e1810da70 2024-04-01T01:49:30.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.325+0000 7f4e157fa640 1 -- 172.21.15.70:0/303031375 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1004a040 con 0x7f4e1810da70 2024-04-01T01:49:30.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.326+0000 7f4e1e8d4640 1 -- 172.21.15.70:0/303031375 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e181a62e0 con 0x7f4e1810da70 2024-04-01T01:49:30.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.329+0000 7f4e2158e740 1 Processor -- start 2024-04-01T01:49:30.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.330+0000 7f4e157fa640 1 -- 172.21.15.70:0/303031375 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f4e10055020 con 0x7f4e1810da70 2024-04-01T01:49:30.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.330+0000 7f4e2158e740 1 -- start start 2024-04-01T01:49:30.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.330+0000 7f4e2158e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddba490 0x559deddba8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.330+0000 7f4e2158e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbae80 0x559deddbf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.330+0000 7f4e2158e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbfff0 0x559deddc2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.330+0000 7f4e2158e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559dedd1f1b0 con 0x559deddba490 2024-04-01T01:49:30.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.330+0000 7f4e2158e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x559dedcc7c00 con 0x559deddbfff0 2024-04-01T01:49:30.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.330+0000 7f4e2158e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559dedd1b250 con 0x559deddbae80 2024-04-01T01:49:30.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.330+0000 7f4e177fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbfff0 0x559deddc2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.331+0000 7f4e177fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbfff0 0x559deddc2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49608/0 (socket says 172.21.15.70:49608) 2024-04-01T01:49:30.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.331+0000 7f4e177fe640 1 -- 172.21.15.70:0/1287627943 learned_addr learned my addr 172.21.15.70:0/1287627943 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.332+0000 7f4e177fe640 1 --2- 172.21.15.70:0/1287627943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddba490 0x559deddba8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.332+0000 7f4e17fff640 1 --2- 172.21.15.70:0/1287627943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbae80 0x559deddbf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.333+0000 7f4e177fe640 1 -- 172.21.15.70:0/1287627943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbae80 msgr2=0x559deddbf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.333+0000 7f4e177fe640 1 --2- 172.21.15.70:0/1287627943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbae80 0x559deddbf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.333+0000 7f4e177fe640 1 -- 172.21.15.70:0/1287627943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddba490 msgr2=0x559deddba8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.333+0000 7f4e177fe640 1 --2- 172.21.15.70:0/1287627943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddba490 0x559deddba8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.333+0000 7f4e177fe640 1 -- 172.21.15.70:0/1287627943 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dedce73d0 con 0x559deddbfff0 2024-04-01T01:49:30.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.333+0000 7f4e157fa640 1 --2- 172.21.15.70:0/303031375 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e000a2570 0x7f4e000a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.333+0000 7f4e177fe640 1 --2- 172.21.15.70:0/1287627943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbfff0 0x559deddc2480 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4e080118a0 tx=0x7f4e08009860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f3ec3f3a66be8d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.333+0000 7f4e157fa640 1 -- 172.21.15.70:0/303031375 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f4e10101990 con 0x7f4e1810da70 2024-04-01T01:49:30.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.334+0000 7f4dfe7fc640 1 -- 172.21.15.70:0/1287627943 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e08004030 con 0x559deddbfff0 2024-04-01T01:49:30.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.334+0000 7f4dfe7fc640 1 -- 172.21.15.70:0/1287627943 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e08005020 con 0x559deddbfff0 2024-04-01T01:49:30.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.334+0000 7f4dfe7fc640 1 -- 172.21.15.70:0/1287627943 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e08004b20 con 0x559deddbfff0 2024-04-01T01:49:30.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.334+0000 7f4e17fff640 1 --2- 172.21.15.70:0/303031375 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e000a2570 0x7f4e000a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.336+0000 7f4e2158e740 1 -- 172.21.15.70:0/1287627943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbfff0 msgr2=0x559deddc2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.336+0000 7f4e2158e740 1 --2- 172.21.15.70:0/1287627943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbfff0 0x559deddc2480 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f4e080118a0 tx=0x7f4e08009860 comp rx=0 tx=0).stop 2024-04-01T01:49:30.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.336+0000 7f4e17fff640 1 --2- 172.21.15.70:0/303031375 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e000a2570 0x7f4e000a4a30 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c006fd0 tx=0x7f4e0c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.336+0000 7f4e2158e740 1 -- 172.21.15.70:0/1287627943 shutdown_connections 2024-04-01T01:49:30.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.336+0000 7f4e2158e740 1 --2- 172.21.15.70:0/1287627943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbfff0 0x559deddc2480 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.336+0000 7f4e2158e740 1 --2- 172.21.15.70:0/1287627943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbae80 0x559deddbf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.336+0000 7f4e2158e740 1 --2- 172.21.15.70:0/1287627943 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddba490 0x559deddba8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.336+0000 7f4e2158e740 1 -- 172.21.15.70:0/1287627943 >> 172.21.15.70:0/1287627943 conn(0x559deddb5a00 msgr2=0x559deddb5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.337+0000 7f4e2158e740 1 -- 172.21.15.70:0/1287627943 shutdown_connections 2024-04-01T01:49:30.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.337+0000 7f4e2158e740 1 -- 172.21.15.70:0/1287627943 wait complete. 2024-04-01T01:49:30.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.338+0000 7f4e2158e740 1 -- start start 2024-04-01T01:49:30.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.338+0000 7f4e2158e740 1 Processor -- start 2024-04-01T01:49:30.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.339+0000 7f4e2158e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddba490 0x559deddc4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.339+0000 7f4e2158e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbae80 0x559deddc4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.339+0000 7f4e177fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbae80 0x559deddc4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.339+0000 7f4e177fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbae80 0x559deddc4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:51430/0 (socket says 172.21.15.70:51430) 2024-04-01T01:49:30.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.339+0000 7f4e177fe640 1 -- 172.21.15.70:0/2884044590 learned_addr learned my addr 172.21.15.70:0/2884044590 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.339+0000 7f4e17fff640 1 --2- 172.21.15.70:0/2884044590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddba490 0x559deddc4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.340+0000 7f4e2158e740 1 --2- 172.21.15.70:0/2884044590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbfff0 0x559deddcbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.340+0000 7f4e2158e740 1 -- 172.21.15.70:0/2884044590 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559deddc2fa0 con 0x559deddba490 2024-04-01T01:49:30.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.340+0000 7f4e2158e740 1 -- 172.21.15.70:0/2884044590 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x559dedcbd470 con 0x559deddbfff0 2024-04-01T01:49:30.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.340+0000 7f4e1ce71640 1 --2- 172.21.15.70:0/2884044590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbfff0 0x559deddcbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.340+0000 7f4e2158e740 1 -- 172.21.15.70:0/2884044590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559deddc3120 con 0x559deddbae80 2024-04-01T01:49:30.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.341+0000 7f4e177fe640 1 -- 172.21.15.70:0/2884044590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbfff0 msgr2=0x559deddcbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.341+0000 7f4e177fe640 1 --2- 172.21.15.70:0/2884044590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbfff0 0x559deddcbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.341+0000 7f4e177fe640 1 -- 172.21.15.70:0/2884044590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddba490 msgr2=0x559deddc4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.341+0000 7f4e177fe640 1 --2- 172.21.15.70:0/2884044590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddba490 0x559deddc4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.341+0000 7f4e177fe640 1 -- 172.21.15.70:0/2884044590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559deda333f0 con 0x559deddbae80 2024-04-01T01:49:30.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.341+0000 7f4e177fe640 1 --2- 172.21.15.70:0/2884044590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbae80 0x559deddc4fa0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4e08006010 tx=0x7f4e080051d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=308768c951c41bd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.341+0000 7f4e1ce71640 1 --2- 172.21.15.70:0/2884044590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbfff0 0x559deddcbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:49:30.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.341+0000 7f4df77fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:49:30.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.341+0000 7f4df77fe640 1 -- 172.21.15.70:0/2884044590 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e08004230 con 0x559deddbae80 2024-04-01T01:49:30.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.342+0000 7f4df77fe640 1 -- 172.21.15.70:0/2884044590 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f4e080043d0 con 0x559deddbae80 2024-04-01T01:49:30.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.342+0000 7f4df77fe640 10 client.0 inactive, discarding osd_map(95..95 src has 1..95) 2024-04-01T01:49:30.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.343+0000 7f4e2158e740 10 client.0 _ll_register_callbacks cb 0x559dedcdacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:30.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.343+0000 7f4e2158e740 1 -- 172.21.15.70:0/2884044590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e08005020 con 0x559deddbae80 2024-04-01T01:49:30.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.343+0000 7f4df77fe640 1 -- 172.21.15.70:0/2884044590 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4e0802d050 con 0x559deddbae80 2024-04-01T01:49:30.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.343+0000 7f4e2158e740 10 client.29878 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:30.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.343+0000 7f4e2158e740 1 -- 172.21.15.70:0/2884044590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=33}) -- 0x559dedafd7d0 con 0x559deddbae80 2024-04-01T01:49:30.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.344+0000 7f4df77fe640 1 -- 172.21.15.70:0/2884044590 <== mon.1 v2:172.21.15.189:3300/0 4 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f4e080048a0 con 0x559deddbae80 2024-04-01T01:49:30.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.344+0000 7f4e2158e740 10 client.29878 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:30.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.344+0000 7f4e2158e740 10 client.29878 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:49:30.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.344+0000 7f4e2158e740 20 client.29878 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:30.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.344+0000 7f4e2158e740 1 --2- 172.21.15.70:0/2884044590 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x559deddcff40 0x559deddd23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.344+0000 7f4e2158e740 4 client.29878 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:49:30.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.344+0000 7f4e2158e740 1 -- 172.21.15.70:0/2884044590 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4e08005020 con 0x559deddcff40 2024-04-01T01:49:30.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.345+0000 7f4e17fff640 1 --2- 172.21.15.70:0/2884044590 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x559deddcff40 0x559deddd23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.350+0000 7f4e17fff640 1 --2- 172.21.15.70:0/2884044590 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x559deddcff40 0x559deddd23e0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=57e7af6077b87429 server_cookie=fb6e72e832be04c0 in_seq=0 out_seq=0 2024-04-01T01:49:30.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.350+0000 7f4df77fe640 10 client.29878 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:49:30.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.352+0000 7f4df77fe640 1 -- 172.21.15.70:0/2884044590 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f4e08005020 con 0x559deddcff40 2024-04-01T01:49:30.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.352+0000 7f4df77fe640 10 client.29878 handle_command_reply: tid=0 2024-04-01T01:49:30.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.352+0000 7f4e2158e740 1 client.29878 shutdown 2024-04-01T01:49:30.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.353+0000 7f4e2158e740 1 -- 172.21.15.70:0/2884044590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbae80 msgr2=0x559deddc4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.353+0000 7f4e2158e740 1 --2- 172.21.15.70:0/2884044590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbae80 0x559deddc4fa0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4e08006010 tx=0x7f4e080051d0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.355+0000 7f4e2158e740 1 -- 172.21.15.70:0/2884044590 shutdown_connections 2024-04-01T01:49:30.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.355+0000 7f4e2158e740 1 --2- 172.21.15.70:0/2884044590 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x559deddcff40 0x559deddd23e0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.355+0000 7f4e2158e740 1 --2- 172.21.15.70:0/2884044590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbfff0 0x559deddcbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.355+0000 7f4e2158e740 1 --2- 172.21.15.70:0/2884044590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbae80 0x559deddc4fa0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.355+0000 7f4e2158e740 1 --2- 172.21.15.70:0/2884044590 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddba490 0x559deddc4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.355+0000 7f4e2158e740 1 -- 172.21.15.70:0/2884044590 >> 172.21.15.70:0/2884044590 conn(0x559deddb5a00 msgr2=0x559deddb4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.355+0000 7f4df77fe640 0 client.29878 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:49:30.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.356+0000 7f4e2158e740 1 -- 172.21.15.70:0/2884044590 shutdown_connections 2024-04-01T01:49:30.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.356+0000 7f4e2158e740 1 -- 172.21.15.70:0/2884044590 wait complete. 2024-04-01T01:49:30.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.356+0000 7f4e2158e740 20 client.29878 trim_cache size 0 max 16384 2024-04-01T01:49:30.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.396+0000 7f4e2158e740 1 Processor -- start 2024-04-01T01:49:30.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.396+0000 7f4e2158e740 1 -- start start 2024-04-01T01:49:30.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.398+0000 7f4e2158e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddba490 0x559deddb4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.398+0000 7f4e2158e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbae80 0x559deddcbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.398+0000 7f4e2158e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddbfff0 0x559deddc34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.398+0000 7f4e2158e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e181b0950 con 0x559deddbfff0 2024-04-01T01:49:30.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.398+0000 7f4e2158e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4e181b2490 con 0x559deddbae80 2024-04-01T01:49:30.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.398+0000 7f4e2158e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4e181b1020 con 0x559deddba490 2024-04-01T01:49:30.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.398+0000 7f4e1ce71640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddbfff0 0x559deddc34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.398+0000 7f4e1ce71640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddbfff0 0x559deddc34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49070/0 (socket says 172.21.15.70:49070) 2024-04-01T01:49:30.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.398+0000 7f4e1ce71640 1 -- 172.21.15.70:0/1207116341 learned_addr learned my addr 172.21.15.70:0/1207116341 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.398+0000 7f4e177fe640 1 --2- 172.21.15.70:0/1207116341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbae80 0x559deddcbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.399+0000 7f4e17fff640 1 --2- 172.21.15.70:0/1207116341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddba490 0x559deddb4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.401+0000 7f4e1ce71640 1 -- 172.21.15.70:0/1207116341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddba490 msgr2=0x559deddb4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.401+0000 7f4e1ce71640 1 --2- 172.21.15.70:0/1207116341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddba490 0x559deddb4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.401+0000 7f4e1ce71640 1 -- 172.21.15.70:0/1207116341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbae80 msgr2=0x559deddcbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.401+0000 7f4e1ce71640 1 --2- 172.21.15.70:0/1207116341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbae80 0x559deddcbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.401+0000 7f4e1ce71640 1 -- 172.21.15.70:0/1207116341 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559deddc5820 con 0x559deddbfff0 2024-04-01T01:49:30.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.405+0000 7f4e1ce71640 1 --2- 172.21.15.70:0/1207116341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddbfff0 0x559deddc34c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f4e1003da00 tx=0x7f4e100cda40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=547e3da306099762 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.406+0000 7f4dfdffb640 1 -- 172.21.15.70:0/1207116341 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e100d2170 con 0x559deddbfff0 2024-04-01T01:49:30.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.406+0000 7f4dfdffb640 1 -- 172.21.15.70:0/1207116341 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e100d6a30 con 0x559deddbfff0 2024-04-01T01:49:30.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.406+0000 7f4dfdffb640 1 -- 172.21.15.70:0/1207116341 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e100d65a0 con 0x559deddbfff0 2024-04-01T01:49:30.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.407+0000 7f4e2158e740 1 -- 172.21.15.70:0/1207116341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddbfff0 msgr2=0x559deddc34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.407+0000 7f4e2158e740 1 --2- 172.21.15.70:0/1207116341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddbfff0 0x559deddc34c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f4e1003da00 tx=0x7f4e100cda40 comp rx=0 tx=0).stop 2024-04-01T01:49:30.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.407+0000 7f4e2158e740 1 -- 172.21.15.70:0/1207116341 shutdown_connections 2024-04-01T01:49:30.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.407+0000 7f4e2158e740 1 --2- 172.21.15.70:0/1207116341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddbfff0 0x559deddc34c0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.407+0000 7f4e2158e740 1 --2- 172.21.15.70:0/1207116341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddbae80 0x559deddcbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.407+0000 7f4e2158e740 1 --2- 172.21.15.70:0/1207116341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddba490 0x559deddb4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.408+0000 7f4e2158e740 1 -- 172.21.15.70:0/1207116341 >> 172.21.15.70:0/1207116341 conn(0x559deddb5a00 msgr2=0x559deddb44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.408+0000 7f4e2158e740 1 -- 172.21.15.70:0/1207116341 shutdown_connections 2024-04-01T01:49:30.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.408+0000 7f4e2158e740 1 -- 172.21.15.70:0/1207116341 wait complete. 2024-04-01T01:49:30.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.409+0000 7f4e2158e740 1 -- start start 2024-04-01T01:49:30.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.409+0000 7f4e2158e740 1 Processor -- start 2024-04-01T01:49:30.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.409+0000 7f4e2158e740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddba490 0x559deddc99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.409+0000 7f4e2158e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddbae80 0x559deddc9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.410+0000 7f4e2158e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbfff0 0x559deddcae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.410+0000 7f4e2158e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e181b1dc0 con 0x559deddbae80 2024-04-01T01:49:30.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.410+0000 7f4e2158e740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4e181b16f0 con 0x559deddba490 2024-04-01T01:49:30.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.410+0000 7f4e2158e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4e181ae930 con 0x559deddbfff0 2024-04-01T01:49:30.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.410+0000 7f4e2158e740 10 client.0 _ll_register_callbacks cb 0x7f4e08011370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:30.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.410+0000 7f4e17fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddba490 0x559deddc99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.410+0000 7f4e17fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddba490 0x559deddc99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49630/0 (socket says 172.21.15.70:49630) 2024-04-01T01:49:30.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.410+0000 7f4e17fff640 1 -- 172.21.15.70:0/2699352499 learned_addr learned my addr 172.21.15.70:0/2699352499 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.410+0000 7f4e177fe640 1 --2- 172.21.15.70:0/2699352499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddbae80 0x559deddc9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.410+0000 7f4e1ce71640 1 --2- 172.21.15.70:0/2699352499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbfff0 0x559deddcae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.410+0000 7f4e17fff640 1 -- 172.21.15.70:0/2699352499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbfff0 msgr2=0x559deddcae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.410+0000 7f4e17fff640 1 --2- 172.21.15.70:0/2699352499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbfff0 0x559deddcae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.410+0000 7f4e17fff640 1 -- 172.21.15.70:0/2699352499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddbae80 msgr2=0x559deddc9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.410+0000 7f4e17fff640 1 --2- 172.21.15.70:0/2699352499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddbae80 0x559deddc9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.410+0000 7f4e17fff640 1 -- 172.21.15.70:0/2699352499 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559deddcb480 con 0x559deddba490 2024-04-01T01:49:30.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.411+0000 7f4e17fff640 1 --2- 172.21.15.70:0/2699352499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddba490 0x559deddc99b0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c00bfd0 tx=0x7f4e0c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ede94966a602714d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.411+0000 7f4df77fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:49:30.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.413+0000 7f4df77fe640 1 -- 172.21.15.70:0/2699352499 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c014e10 con 0x559deddba490 2024-04-01T01:49:30.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.413+0000 7f4df77fe640 1 -- 172.21.15.70:0/2699352499 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f4e0c00f040 con 0x559deddba490 2024-04-01T01:49:30.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.414+0000 7f4e2158e740 1 -- 172.21.15.70:0/2699352499 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e100d6a30 con 0x559deddba490 2024-04-01T01:49:30.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.414+0000 7f4df77fe640 1 -- 172.21.15.70:0/2699352499 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4e0c034050 con 0x559deddba490 2024-04-01T01:49:30.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.414+0000 7f4e2158e740 10 client.30164 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:30.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.414+0000 7f4e2158e740 1 -- 172.21.15.70:0/2699352499 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559dede365a0 con 0x559deddba490 2024-04-01T01:49:30.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.414+0000 7f4df77fe640 1 -- 172.21.15.70:0/2699352499 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e100d6a30 con 0x559deddba490 2024-04-01T01:49:30.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.417+0000 7f4df77fe640 1 -- 172.21.15.70:0/2699352499 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f4e0c013070 con 0x559deddba490 2024-04-01T01:49:30.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.417+0000 7f4e2158e740 10 client.30164 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:30.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.417+0000 7f4e2158e740 10 client.30164 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:49:30.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.417+0000 7f4e2158e740 20 client.30164 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:30.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.417+0000 7f4e2158e740 1 --2- 172.21.15.70:0/2699352499 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x559dede36f40 0x559dede39330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.417+0000 7f4e2158e740 4 client.30164 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:49:30.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.417+0000 7f4e2158e740 1 -- 172.21.15.70:0/2699352499 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559dedd2c6f0 con 0x559dede36f40 2024-04-01T01:49:30.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.418+0000 7f4e177fe640 1 --2- 172.21.15.70:0/2699352499 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x559dede36f40 0x559dede39330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.419+0000 7f4e177fe640 1 --2- 172.21.15.70:0/2699352499 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x559dede36f40 0x559dede39330 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9e658766bda546cb server_cookie=7bf4f55d2e5f8cec in_seq=0 out_seq=0 2024-04-01T01:49:30.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.419+0000 7f4df77fe640 10 client.30164 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:49:30.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.420+0000 7f4df77fe640 1 -- 172.21.15.70:0/2699352499 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559dedd2c6f0 con 0x559dede36f40 2024-04-01T01:49:30.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.420+0000 7f4df77fe640 10 client.30164 handle_command_reply: tid=0 2024-04-01T01:49:30.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.420+0000 7f4e2158e740 1 client.30164 shutdown 2024-04-01T01:49:30.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.420+0000 7f4e2158e740 1 -- 172.21.15.70:0/2699352499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddba490 msgr2=0x559deddc99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.420+0000 7f4e2158e740 1 --2- 172.21.15.70:0/2699352499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddba490 0x559deddc99b0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c00bfd0 tx=0x7f4e0c004750 comp rx=0 tx=0).stop 2024-04-01T01:49:30.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.421+0000 7f4e2158e740 1 -- 172.21.15.70:0/2699352499 shutdown_connections 2024-04-01T01:49:30.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.421+0000 7f4e2158e740 1 --2- 172.21.15.70:0/2699352499 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x559dede36f40 0x559dede39330 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.421+0000 7f4e2158e740 1 --2- 172.21.15.70:0/2699352499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559deddbfff0 0x559deddcae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.421+0000 7f4df77fe640 0 client.30164 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:49:30.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.421+0000 7f4e2158e740 1 --2- 172.21.15.70:0/2699352499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559deddbae80 0x559deddc9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.421+0000 7f4e2158e740 1 --2- 172.21.15.70:0/2699352499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559deddba490 0x559deddc99b0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.421+0000 7f4e2158e740 1 -- 172.21.15.70:0/2699352499 >> 172.21.15.70:0/2699352499 conn(0x559deddb5a00 msgr2=0x559deddb8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.421+0000 7f4e2158e740 1 -- 172.21.15.70:0/2699352499 shutdown_connections 2024-04-01T01:49:30.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.422+0000 7f4e2158e740 1 -- 172.21.15.70:0/2699352499 wait complete. 2024-04-01T01:49:30.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.422+0000 7f4e2158e740 20 client.30164 trim_cache size 0 max 16384 2024-04-01T01:49:30.422 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:49:30.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.423+0000 7f4e1e8d4640 1 -- 172.21.15.70:0/303031375 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e000a2570 msgr2=0x7f4e000a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.423+0000 7f4e1e8d4640 1 --2- 172.21.15.70:0/303031375 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e000a2570 0x7f4e000a4a30 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c006fd0 tx=0x7f4e0c008040 comp rx=0 tx=0).stop 2024-04-01T01:49:30.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.423+0000 7f4e1e8d4640 1 -- 172.21.15.70:0/303031375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e1810da70 msgr2=0x7f4e181a55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.423+0000 7f4e1e8d4640 1 --2- 172.21.15.70:0/303031375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e1810da70 0x7f4e181a55e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4e10045f50 tx=0x7f4e10043f70 comp rx=0 tx=0).stop 2024-04-01T01:49:30.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.423+0000 7f4e1e8d4640 1 -- 172.21.15.70:0/303031375 shutdown_connections 2024-04-01T01:49:30.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.423+0000 7f4e1e8d4640 1 --2- 172.21.15.70:0/303031375 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4e000a2570 0x7f4e000a4a30 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.423+0000 7f4e1e8d4640 1 --2- 172.21.15.70:0/303031375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e1810da70 0x7f4e181a55e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.423+0000 7f4e1e8d4640 1 --2- 172.21.15.70:0/303031375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e1810b0c0 0x7f4e1819e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.423+0000 7f4e1e8d4640 1 --2- 172.21.15.70:0/303031375 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e18108750 0x7f4e1819e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.423+0000 7f4e1e8d4640 1 -- 172.21.15.70:0/303031375 >> 172.21.15.70:0/303031375 conn(0x7f4e1806e5b0 msgr2=0x7f4e1810b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.423+0000 7f4e1e8d4640 1 -- 172.21.15.70:0/303031375 shutdown_connections 2024-04-01T01:49:30.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.423+0000 7f4e1e8d4640 1 -- 172.21.15.70:0/303031375 wait complete. 2024-04-01T01:49:30.442 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:49:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.620+0000 7fb47a640640 1 -- 172.21.15.70:0/1317477416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb474109920 msgr2=0x7fb4741101b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.620+0000 7fb47a640640 1 --2- 172.21.15.70:0/1317477416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb474109920 0x7fb4741101b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb46c009ca0 tx=0x7fb46c02c100 comp rx=0 tx=0).stop 2024-04-01T01:49:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.620+0000 7fb47a640640 1 -- 172.21.15.70:0/1317477416 shutdown_connections 2024-04-01T01:49:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.620+0000 7fb47a640640 1 --2- 172.21.15.70:0/1317477416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb474109920 0x7fb4741101b0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.620+0000 7fb47a640640 1 --2- 172.21.15.70:0/1317477416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb474108f60 0x7fb4741093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.620+0000 7fb47a640640 1 --2- 172.21.15.70:0/1317477416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb474107d70 0x7fb474108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.620+0000 7fb47a640640 1 -- 172.21.15.70:0/1317477416 >> 172.21.15.70:0/1317477416 conn(0x7fb474076d10 msgr2=0x7fb474079150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.620+0000 7fb47a640640 1 -- 172.21.15.70:0/1317477416 shutdown_connections 2024-04-01T01:49:30.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.620+0000 7fb47a640640 1 -- 172.21.15.70:0/1317477416 wait complete. 2024-04-01T01:49:30.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.621+0000 7fb47a640640 1 Processor -- start 2024-04-01T01:49:30.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.621+0000 7fb47a640640 1 -- start start 2024-04-01T01:49:30.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.621+0000 7fb47a640640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb474107d70 0x7fb47419df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.621+0000 7fb47a640640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb474108f60 0x7fb47419e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.621+0000 7fb47a640640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb474109920 0x7fb4741a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.621+0000 7fb47a640640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb474113880 con 0x7fb474108f60 2024-04-01T01:49:30.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.621+0000 7fb47a640640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb474113700 con 0x7fb474109920 2024-04-01T01:49:30.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.621+0000 7fb47a640640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb474113a00 con 0x7fb474107d70 2024-04-01T01:49:30.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.622+0000 7fb4737fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb474108f60 0x7fb47419e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.622+0000 7fb4737fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb474108f60 0x7fb47419e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49086/0 (socket says 172.21.15.70:49086) 2024-04-01T01:49:30.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.622+0000 7fb4737fe640 1 -- 172.21.15.70:0/1863431461 learned_addr learned my addr 172.21.15.70:0/1863431461 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.622+0000 7fb478bdd640 1 --2- 172.21.15.70:0/1863431461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb474109920 0x7fb4741a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.622+0000 7fb473fff640 1 --2- 172.21.15.70:0/1863431461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb474107d70 0x7fb47419df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.622+0000 7fb478bdd640 1 -- 172.21.15.70:0/1863431461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb474107d70 msgr2=0x7fb47419df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.622+0000 7fb478bdd640 1 --2- 172.21.15.70:0/1863431461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb474107d70 0x7fb47419df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.622+0000 7fb478bdd640 1 -- 172.21.15.70:0/1863431461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb474108f60 msgr2=0x7fb47419e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.622+0000 7fb478bdd640 1 --2- 172.21.15.70:0/1863431461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb474108f60 0x7fb47419e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.622+0000 7fb478bdd640 1 -- 172.21.15.70:0/1863431461 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4741a5b40 con 0x7fb474109920 2024-04-01T01:49:30.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.622+0000 7fb4737fe640 1 --2- 172.21.15.70:0/1863431461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb474108f60 0x7fb47419e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:49:30.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.622+0000 7fb473fff640 1 --2- 172.21.15.70:0/1863431461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb474107d70 0x7fb47419df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:49:30.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.623+0000 7fb478bdd640 1 --2- 172.21.15.70:0/1863431461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb474109920 0x7fb4741a54d0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb46c009bf0 tx=0x7fb46c035ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.623+0000 7fb4717fa640 1 -- 172.21.15.70:0/1863431461 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c039040 con 0x7fb474109920 2024-04-01T01:49:30.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.623+0000 7fb4717fa640 1 -- 172.21.15.70:0/1863431461 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb46c035e50 con 0x7fb474109920 2024-04-01T01:49:30.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.623+0000 7fb4717fa640 1 -- 172.21.15.70:0/1863431461 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c0044d0 con 0x7fb474109920 2024-04-01T01:49:30.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.623+0000 7fb47a640640 1 -- 172.21.15.70:0/1863431461 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4741a5dd0 con 0x7fb474109920 2024-04-01T01:49:30.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.623+0000 7fb47a640640 1 -- 172.21.15.70:0/1863431461 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4741a6230 con 0x7fb474109920 2024-04-01T01:49:30.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.626+0000 7fb47d223740 1 Processor -- start 2024-04-01T01:49:30.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.626+0000 7fb4717fa640 1 -- 172.21.15.70:0/1863431461 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb46c046030 con 0x7fb474109920 2024-04-01T01:49:30.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.626+0000 7fb47d223740 1 -- start start 2024-04-01T01:49:30.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.626+0000 7fb47d223740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 0x55df9a2dc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb47d223740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2dce80 0x55df9a2e1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb47d223740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2e1ff0 0x55df9a2e4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb47d223740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55df9a2411b0 con 0x55df9a2dce80 2024-04-01T01:49:30.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb47d223740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55df9a1e9c00 con 0x55df9a2e1ff0 2024-04-01T01:49:30.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb47d223740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55df9a23d250 con 0x55df9a2dc490 2024-04-01T01:49:30.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb473fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2dce80 0x55df9a2e1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb473fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2dce80 0x55df9a2e1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49100/0 (socket says 172.21.15.70:49100) 2024-04-01T01:49:30.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb473fff640 1 -- 172.21.15.70:0/2256529293 learned_addr learned my addr 172.21.15.70:0/2256529293 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb4737fe640 1 --2- 172.21.15.70:0/2256529293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 0x55df9a2dc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb4737fe640 1 --2- 172.21.15.70:0/2256529293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2e1ff0 0x55df9a2e4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb4717fa640 1 --2- 172.21.15.70:0/1863431461 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb45c0a2520 0x7fb45c0a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb473fff640 1 -- 172.21.15.70:0/2256529293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 msgr2=0x55df9a2dc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb473fff640 1 --2- 172.21.15.70:0/2256529293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 0x55df9a2dc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb473fff640 1 -- 172.21.15.70:0/2256529293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2e1ff0 msgr2=0x55df9a2e4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb473fff640 1 --2- 172.21.15.70:0/2256529293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2e1ff0 0x55df9a2e4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb473fff640 1 -- 172.21.15.70:0/2256529293 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df9a2093d0 con 0x55df9a2dce80 2024-04-01T01:49:30.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb4737fe640 1 --2- 172.21.15.70:0/2256529293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 0x55df9a2dc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:49:30.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb478bdd640 1 --2- 172.21.15.70:0/1863431461 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb45c0a2520 0x7fb45c0a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.627+0000 7fb473fff640 1 --2- 172.21.15.70:0/2256529293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2dce80 0x55df9a2e1980 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fb464010450 tx=0x7fb464010910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e5eb1b7dc5240a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.628+0000 7fb4717fa640 1 -- 172.21.15.70:0/1863431461 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fb46c0f0590 con 0x7fb474109920 2024-04-01T01:49:30.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.628+0000 7fb45a7fc640 1 -- 172.21.15.70:0/2256529293 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb464011760 con 0x55df9a2dce80 2024-04-01T01:49:30.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.628+0000 7fb45a7fc640 1 -- 172.21.15.70:0/2256529293 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb464011db0 con 0x55df9a2dce80 2024-04-01T01:49:30.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.629+0000 7fb45a7fc640 1 -- 172.21.15.70:0/2256529293 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46401a380 con 0x55df9a2dce80 2024-04-01T01:49:30.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.629+0000 7fb47d223740 1 -- 172.21.15.70:0/2256529293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2dce80 msgr2=0x55df9a2e1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.629+0000 7fb47d223740 1 --2- 172.21.15.70:0/2256529293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2dce80 0x55df9a2e1980 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fb464010450 tx=0x7fb464010910 comp rx=0 tx=0).stop 2024-04-01T01:49:30.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.629+0000 7fb478bdd640 1 --2- 172.21.15.70:0/1863431461 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb45c0a2520 0x7fb45c0a49e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb46c02c640 tx=0x7fb46c087b90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.629+0000 7fb47d223740 1 -- 172.21.15.70:0/2256529293 shutdown_connections 2024-04-01T01:49:30.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.629+0000 7fb47d223740 1 --2- 172.21.15.70:0/2256529293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2e1ff0 0x55df9a2e4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.629+0000 7fb47d223740 1 --2- 172.21.15.70:0/2256529293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2dce80 0x55df9a2e1980 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.629+0000 7fb47d223740 1 --2- 172.21.15.70:0/2256529293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 0x55df9a2dc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.629+0000 7fb47d223740 1 -- 172.21.15.70:0/2256529293 >> 172.21.15.70:0/2256529293 conn(0x55df9a2d7a00 msgr2=0x55df9a2d7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.630+0000 7fb47d223740 1 -- 172.21.15.70:0/2256529293 shutdown_connections 2024-04-01T01:49:30.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.630+0000 7fb47d223740 1 -- 172.21.15.70:0/2256529293 wait complete. 2024-04-01T01:49:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.630+0000 7fb47d223740 1 -- start start 2024-04-01T01:49:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.630+0000 7fb47d223740 1 Processor -- start 2024-04-01T01:49:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.631+0000 7fb47d223740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 0x55df9a2e69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.631+0000 7fb47d223740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2dce80 0x55df9a2e6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.631+0000 7fb47d223740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2e1ff0 0x55df9a2edf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.631+0000 7fb47d223740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55df9a2e4ff0 con 0x55df9a2dce80 2024-04-01T01:49:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.631+0000 7fb47d223740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55df9a1df470 con 0x55df9a2e1ff0 2024-04-01T01:49:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.631+0000 7fb47d223740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55df9a2e5170 con 0x55df9a2dc490 2024-04-01T01:49:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.631+0000 7fb47d223740 10 client.0 _ll_register_callbacks cb 0x55df9a1fccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.631+0000 7fb4737fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 0x55df9a2e69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.631+0000 7fb4737fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 0x55df9a2e69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:51486/0 (socket says 172.21.15.70:51486) 2024-04-01T01:49:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.631+0000 7fb4737fe640 1 -- 172.21.15.70:0/2556704446 learned_addr learned my addr 172.21.15.70:0/2556704446 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.631+0000 7fb4737fe640 1 --2- 172.21.15.70:0/2556704446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2e1ff0 0x55df9a2edf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.632+0000 7fb4737fe640 1 -- 172.21.15.70:0/2556704446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2e1ff0 msgr2=0x55df9a2edf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.632+0000 7fb4737fe640 1 --2- 172.21.15.70:0/2556704446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2e1ff0 0x55df9a2edf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.632+0000 7fb4737fe640 1 -- 172.21.15.70:0/2556704446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2dce80 msgr2=0x55df9a2e6f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:49:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.632+0000 7fb473fff640 1 --2- 172.21.15.70:0/2556704446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2dce80 0x55df9a2e6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.632+0000 7fb4737fe640 1 --2- 172.21.15.70:0/2556704446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2dce80 0x55df9a2e6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.632+0000 7fb4737fe640 1 -- 172.21.15.70:0/2556704446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df99f553f0 con 0x55df9a2dc490 2024-04-01T01:49:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.632+0000 7fb473fff640 1 --2- 172.21.15.70:0/2556704446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2dce80 0x55df9a2e6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:49:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.632+0000 7fb4737fe640 1 --2- 172.21.15.70:0/2556704446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 0x55df9a2e69d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb468012e80 tx=0x7fb468014040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2f6be5f1ec0fcef server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.632+0000 7fb4537fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:49:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.632+0000 7fb4537fe640 1 -- 172.21.15.70:0/2556704446 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46800dea0 con 0x55df9a2dc490 2024-04-01T01:49:30.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.633+0000 7fb47d223740 1 -- 172.21.15.70:0/2556704446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb464011db0 con 0x55df9a2dc490 2024-04-01T01:49:30.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.633+0000 7fb4537fe640 1 -- 172.21.15.70:0/2556704446 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fb468021420 con 0x55df9a2dc490 2024-04-01T01:49:30.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.633+0000 7fb4537fe640 1 -- 172.21.15.70:0/2556704446 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb468029050 con 0x55df9a2dc490 2024-04-01T01:49:30.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.633+0000 7fb47d223740 10 client.29896 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:30.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.633+0000 7fb47d223740 1 -- 172.21.15.70:0/2556704446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55df9a01f7d0 con 0x55df9a2dc490 2024-04-01T01:49:30.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.633+0000 7fb4537fe640 1 -- 172.21.15.70:0/2556704446 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb464011db0 con 0x55df9a2dc490 2024-04-01T01:49:30.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.634+0000 7fb4537fe640 1 -- 172.21.15.70:0/2556704446 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fb4680236e0 con 0x55df9a2dc490 2024-04-01T01:49:30.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.634+0000 7fb47d223740 10 client.29896 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.634+0000 7fb47d223740 10 client.29896 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:49:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.634+0000 7fb47d223740 20 client.29896 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.634+0000 7fb47d223740 1 --2- 172.21.15.70:0/2556704446 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55df9a2f28a0 0x55df9a2f4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.634+0000 7fb47d223740 4 client.29896 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:49:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.634+0000 7fb47d223740 1 -- 172.21.15.70:0/2556704446 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df9a24e6f0 con 0x55df9a2f28a0 2024-04-01T01:49:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.634+0000 7fb473fff640 1 --2- 172.21.15.70:0/2556704446 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55df9a2f28a0 0x55df9a2f4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.635+0000 7fb473fff640 1 --2- 172.21.15.70:0/2556704446 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55df9a2f28a0 0x55df9a2f4d60 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bd64dac9db1f6bda server_cookie=710d0bbb5ac812e8 in_seq=0 out_seq=0 2024-04-01T01:49:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.635+0000 7fb4537fe640 10 client.29896 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:49:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.636+0000 7fb4537fe640 1 -- 172.21.15.70:0/2556704446 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55df9a24e6f0 con 0x55df9a2f28a0 2024-04-01T01:49:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.636+0000 7fb4537fe640 10 client.29896 handle_command_reply: tid=0 2024-04-01T01:49:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.636+0000 7fb47d223740 1 client.29896 shutdown 2024-04-01T01:49:30.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.636+0000 7fb47d223740 1 -- 172.21.15.70:0/2556704446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 msgr2=0x55df9a2e69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.636+0000 7fb47d223740 1 --2- 172.21.15.70:0/2556704446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 0x55df9a2e69d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fb468012e80 tx=0x7fb468014040 comp rx=0 tx=0).stop 2024-04-01T01:49:30.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.636+0000 7fb47d223740 1 -- 172.21.15.70:0/2556704446 shutdown_connections 2024-04-01T01:49:30.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.636+0000 7fb47d223740 1 --2- 172.21.15.70:0/2556704446 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55df9a2f28a0 0x55df9a2f4d60 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.636+0000 7fb47d223740 1 --2- 172.21.15.70:0/2556704446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2e1ff0 0x55df9a2edf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.636+0000 7fb47d223740 1 --2- 172.21.15.70:0/2556704446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2dce80 0x55df9a2e6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.636+0000 7fb47d223740 1 --2- 172.21.15.70:0/2556704446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 0x55df9a2e69d0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.636+0000 7fb47d223740 1 -- 172.21.15.70:0/2556704446 >> 172.21.15.70:0/2556704446 conn(0x55df9a2d7a00 msgr2=0x55df9a2d6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.636+0000 7fb4537fe640 0 client.29896 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:49:30.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.636+0000 7fb47d223740 1 -- 172.21.15.70:0/2556704446 shutdown_connections 2024-04-01T01:49:30.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.637+0000 7fb47d223740 1 -- 172.21.15.70:0/2556704446 wait complete. 2024-04-01T01:49:30.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.637+0000 7fb47d223740 20 client.29896 trim_cache size 0 max 16384 2024-04-01T01:49:30.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.661+0000 7fb47d223740 1 Processor -- start 2024-04-01T01:49:30.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.661+0000 7fb47d223740 1 -- start start 2024-04-01T01:49:30.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.661+0000 7fb47d223740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2dc490 0x55df9a2e6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.661+0000 7fb47d223740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dce80 0x55df9a2edee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.661+0000 7fb47d223740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2e1ff0 0x55df9a2d2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.661+0000 7fb47d223740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb4741b09d0 con 0x55df9a2e1ff0 2024-04-01T01:49:30.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.661+0000 7fb47d223740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb4741b2510 con 0x55df9a2dc490 2024-04-01T01:49:30.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.661+0000 7fb47d223740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb4741b10a0 con 0x55df9a2dce80 2024-04-01T01:49:30.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.661+0000 7fb4737fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2dc490 0x55df9a2e6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.661+0000 7fb473fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dce80 0x55df9a2edee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.662+0000 7fb473fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dce80 0x55df9a2edee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:51500/0 (socket says 172.21.15.70:51500) 2024-04-01T01:49:30.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.662+0000 7fb4737fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2dc490 0x55df9a2e6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49660/0 (socket says 172.21.15.70:49660) 2024-04-01T01:49:30.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.662+0000 7fb473fff640 1 -- 172.21.15.70:0/833366853 learned_addr learned my addr 172.21.15.70:0/833366853 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.662+0000 7fb4737fe640 1 --2- 172.21.15.70:0/833366853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2e1ff0 0x55df9a2d2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.662+0000 7fb473fff640 1 -- 172.21.15.70:0/833366853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2dc490 msgr2=0x55df9a2e6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.662+0000 7fb473fff640 1 --2- 172.21.15.70:0/833366853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2dc490 0x55df9a2e6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.662+0000 7fb473fff640 1 -- 172.21.15.70:0/833366853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2e1ff0 msgr2=0x55df9a2d2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.662+0000 7fb473fff640 1 --2- 172.21.15.70:0/833366853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2e1ff0 0x55df9a2d2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.662+0000 7fb473fff640 1 -- 172.21.15.70:0/833366853 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df9a2f0e70 con 0x55df9a2dce80 2024-04-01T01:49:30.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.662+0000 7fb4737fe640 1 --2- 172.21.15.70:0/833366853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2e1ff0 0x55df9a2d2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:49:30.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.662+0000 7fb473fff640 1 --2- 172.21.15.70:0/833366853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dce80 0x55df9a2edee0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb464010e10 tx=0x7fb464005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b6a913e152165c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.663+0000 7fb459ffb640 1 -- 172.21.15.70:0/833366853 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4640050b0 con 0x55df9a2dce80 2024-04-01T01:49:30.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.663+0000 7fb459ffb640 1 -- 172.21.15.70:0/833366853 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb46401c840 con 0x55df9a2dce80 2024-04-01T01:49:30.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.663+0000 7fb459ffb640 1 -- 172.21.15.70:0/833366853 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4640284a0 con 0x55df9a2dce80 2024-04-01T01:49:30.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.664+0000 7fb47d223740 1 -- 172.21.15.70:0/833366853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dce80 msgr2=0x55df9a2edee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.664+0000 7fb47d223740 1 --2- 172.21.15.70:0/833366853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dce80 0x55df9a2edee0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb464010e10 tx=0x7fb464005040 comp rx=0 tx=0).stop 2024-04-01T01:49:30.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.664+0000 7fb47d223740 1 -- 172.21.15.70:0/833366853 shutdown_connections 2024-04-01T01:49:30.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.664+0000 7fb47d223740 1 --2- 172.21.15.70:0/833366853 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2e1ff0 0x55df9a2d2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.664+0000 7fb47d223740 1 --2- 172.21.15.70:0/833366853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dce80 0x55df9a2edee0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.664+0000 7fb47d223740 1 --2- 172.21.15.70:0/833366853 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2dc490 0x55df9a2e6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.664+0000 7fb47d223740 1 -- 172.21.15.70:0/833366853 >> 172.21.15.70:0/833366853 conn(0x55df9a2d7a00 msgr2=0x55df9a2d7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.665+0000 7fb47d223740 1 -- 172.21.15.70:0/833366853 shutdown_connections 2024-04-01T01:49:30.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.665+0000 7fb47d223740 1 -- 172.21.15.70:0/833366853 wait complete. 2024-04-01T01:49:30.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.665+0000 7fb47d223740 1 -- start start 2024-04-01T01:49:30.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.665+0000 7fb47d223740 1 Processor -- start 2024-04-01T01:49:30.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb47d223740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 0x55df9a2eb580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb47d223740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2dce80 0x55df9a2ebac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb47d223740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2e1ff0 0x55df9a2ecad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb4737fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 0x55df9a2eb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb4737fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 0x55df9a2eb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:51506/0 (socket says 172.21.15.70:51506) 2024-04-01T01:49:30.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb4737fe640 1 -- 172.21.15.70:0/3532302877 learned_addr learned my addr 172.21.15.70:0/3532302877 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb47d223740 1 -- 172.21.15.70:0/3532302877 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb4741b1e40 con 0x55df9a2e1ff0 2024-04-01T01:49:30.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb47d223740 1 -- 172.21.15.70:0/3532302877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb4741b1770 con 0x55df9a2dce80 2024-04-01T01:49:30.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb473fff640 1 --2- 172.21.15.70:0/3532302877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2dce80 0x55df9a2ebac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb47d223740 1 -- 172.21.15.70:0/3532302877 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb4741ae9b0 con 0x55df9a2dc490 2024-04-01T01:49:30.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb47d223740 10 client.0 _ll_register_callbacks cb 0x7fb468012cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:30.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb4737fe640 1 --2- 172.21.15.70:0/3532302877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2e1ff0 0x55df9a2ecad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb473fff640 1 -- 172.21.15.70:0/3532302877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 msgr2=0x55df9a2eb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb473fff640 1 --2- 172.21.15.70:0/3532302877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 0x55df9a2eb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb473fff640 1 -- 172.21.15.70:0/3532302877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2e1ff0 msgr2=0x55df9a2ecad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb473fff640 1 --2- 172.21.15.70:0/3532302877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2e1ff0 0x55df9a2ecad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.666+0000 7fb473fff640 1 -- 172.21.15.70:0/3532302877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df9a2ed0c0 con 0x55df9a2dce80 2024-04-01T01:49:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.667+0000 7fb473fff640 1 --2- 172.21.15.70:0/3532302877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2dce80 0x55df9a2ebac0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fb464028050 tx=0x7fb4640058d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfecc22c21cfb86c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.667+0000 7fb4537fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:49:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.667+0000 7fb4537fe640 1 -- 172.21.15.70:0/3532302877 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46401cab0 con 0x55df9a2dce80 2024-04-01T01:49:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.667+0000 7fb4537fe640 1 -- 172.21.15.70:0/3532302877 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fb46401cc50 con 0x55df9a2dce80 2024-04-01T01:49:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.667+0000 7fb47d223740 1 -- 172.21.15.70:0/3532302877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb46401c840 con 0x55df9a2dce80 2024-04-01T01:49:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.667+0000 7fb4537fe640 1 -- 172.21.15.70:0/3532302877 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb46401ce50 con 0x55df9a2dce80 2024-04-01T01:49:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.667+0000 7fb47d223740 10 client.30182 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.667+0000 7fb47d223740 1 -- 172.21.15.70:0/3532302877 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55df9a358ca0 con 0x55df9a2dce80 2024-04-01T01:49:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.668+0000 7fb4537fe640 1 -- 172.21.15.70:0/3532302877 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb464005560 con 0x55df9a2dce80 2024-04-01T01:49:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.668+0000 7fb4537fe640 1 -- 172.21.15.70:0/3532302877 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fb464003680 con 0x55df9a2dce80 2024-04-01T01:49:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.668+0000 7fb47d223740 10 client.30182 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.668+0000 7fb47d223740 10 client.30182 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:49:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.668+0000 7fb47d223740 20 client.30182 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.668+0000 7fb47d223740 1 --2- 172.21.15.70:0/3532302877 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55df9a359300 0x55df9a35b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.668+0000 7fb47d223740 4 client.30182 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:49:30.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.668+0000 7fb47d223740 1 -- 172.21.15.70:0/3532302877 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fb46401c840 con 0x55df9a359300 2024-04-01T01:49:30.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.668+0000 7fb4737fe640 1 --2- 172.21.15.70:0/3532302877 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55df9a359300 0x55df9a35b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.669+0000 7fb4737fe640 1 --2- 172.21.15.70:0/3532302877 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55df9a359300 0x55df9a35b720 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=28e460812f669588 server_cookie=92309e28a2fed2b4 in_seq=0 out_seq=0 2024-04-01T01:49:30.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.669+0000 7fb4537fe640 10 client.30182 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:49:30.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.669+0000 7fb4537fe640 1 -- 172.21.15.70:0/3532302877 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fb46401c840 con 0x55df9a359300 2024-04-01T01:49:30.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.669+0000 7fb4537fe640 10 client.30182 handle_command_reply: tid=0 2024-04-01T01:49:30.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.669+0000 7fb47d223740 1 client.30182 shutdown 2024-04-01T01:49:30.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.670+0000 7fb47d223740 1 -- 172.21.15.70:0/3532302877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2dce80 msgr2=0x55df9a2ebac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.670+0000 7fb47d223740 1 --2- 172.21.15.70:0/3532302877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2dce80 0x55df9a2ebac0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fb464028050 tx=0x7fb4640058d0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.670+0000 7fb47d223740 1 -- 172.21.15.70:0/3532302877 shutdown_connections 2024-04-01T01:49:30.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.670+0000 7fb47d223740 1 --2- 172.21.15.70:0/3532302877 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x55df9a359300 0x55df9a35b720 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.670+0000 7fb47d223740 1 --2- 172.21.15.70:0/3532302877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55df9a2e1ff0 0x55df9a2ecad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.670+0000 7fb47d223740 1 --2- 172.21.15.70:0/3532302877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55df9a2dce80 0x55df9a2ebac0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.670+0000 7fb4537fe640 0 client.30182 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:49:30.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.670+0000 7fb47d223740 1 --2- 172.21.15.70:0/3532302877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df9a2dc490 0x55df9a2eb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.670+0000 7fb47d223740 1 -- 172.21.15.70:0/3532302877 >> 172.21.15.70:0/3532302877 conn(0x55df9a2d7a00 msgr2=0x55df9a2d2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.670+0000 7fb47d223740 1 -- 172.21.15.70:0/3532302877 shutdown_connections 2024-04-01T01:49:30.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.670+0000 7fb47d223740 1 -- 172.21.15.70:0/3532302877 wait complete. 2024-04-01T01:49:30.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.670+0000 7fb47d223740 20 client.30182 trim_cache size 0 max 16384 2024-04-01T01:49:30.674 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:49:30.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.672+0000 7fb47a640640 1 -- 172.21.15.70:0/1863431461 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb45c0a2520 msgr2=0x7fb45c0a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.672+0000 7fb47a640640 1 --2- 172.21.15.70:0/1863431461 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb45c0a2520 0x7fb45c0a49e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb46c02c640 tx=0x7fb46c087b90 comp rx=0 tx=0).stop 2024-04-01T01:49:30.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.672+0000 7fb47a640640 1 -- 172.21.15.70:0/1863431461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb474109920 msgr2=0x7fb4741a54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.672+0000 7fb47a640640 1 --2- 172.21.15.70:0/1863431461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb474109920 0x7fb4741a54d0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb46c009bf0 tx=0x7fb46c035ce0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.672+0000 7fb47a640640 1 -- 172.21.15.70:0/1863431461 shutdown_connections 2024-04-01T01:49:30.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.672+0000 7fb47a640640 1 --2- 172.21.15.70:0/1863431461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb474109920 0x7fb4741a54d0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.672+0000 7fb47a640640 1 --2- 172.21.15.70:0/1863431461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb474108f60 0x7fb47419e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.672+0000 7fb47a640640 1 --2- 172.21.15.70:0/1863431461 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb45c0a2520 0x7fb45c0a49e0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.672+0000 7fb47a640640 1 --2- 172.21.15.70:0/1863431461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb474107d70 0x7fb47419df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.672+0000 7fb47a640640 1 -- 172.21.15.70:0/1863431461 >> 172.21.15.70:0/1863431461 conn(0x7fb474076d10 msgr2=0x7fb47410c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.673+0000 7fb47a640640 1 -- 172.21.15.70:0/1863431461 shutdown_connections 2024-04-01T01:49:30.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.673+0000 7fb47a640640 1 -- 172.21.15.70:0/1863431461 wait complete. 2024-04-01T01:49:30.691 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:49:30.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.896+0000 7ff3d58dc640 1 -- 172.21.15.70:0/2989599802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3d0101720 msgr2=0x7ff3d0101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.896+0000 7ff3d58dc640 1 --2- 172.21.15.70:0/2989599802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3d0101720 0x7ff3d0101ba0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009860 tx=0x7ff3bc02bb70 comp rx=0 tx=0).stop 2024-04-01T01:49:30.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.897+0000 7ff3d58dc640 1 -- 172.21.15.70:0/2989599802 shutdown_connections 2024-04-01T01:49:30.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.897+0000 7ff3d58dc640 1 --2- 172.21.15.70:0/2989599802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3d01020e0 0x7ff3d0108970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.897+0000 7ff3d58dc640 1 --2- 172.21.15.70:0/2989599802 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3d0101720 0x7ff3d0101ba0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.897+0000 7ff3d58dc640 1 --2- 172.21.15.70:0/2989599802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3d0100530 0x7ff3d0100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.897+0000 7ff3d58dc640 1 -- 172.21.15.70:0/2989599802 >> 172.21.15.70:0/2989599802 conn(0x7ff3d00fbac0 msgr2=0x7ff3d00fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.897+0000 7ff3d58dc640 1 -- 172.21.15.70:0/2989599802 shutdown_connections 2024-04-01T01:49:30.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.897+0000 7ff3d58dc640 1 -- 172.21.15.70:0/2989599802 wait complete. 2024-04-01T01:49:30.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.898+0000 7ff3d58dc640 1 Processor -- start 2024-04-01T01:49:30.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.898+0000 7ff3d58dc640 1 -- start start 2024-04-01T01:49:30.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.899+0000 7ff3d58dc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3d0100530 0x7ff3d01967a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.899+0000 7ff3d58dc640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3d0101720 0x7ff3d0196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.899+0000 7ff3d58dc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3d01020e0 0x7ff3d019dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.899+0000 7ff3d48da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3d0100530 0x7ff3d01967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.899+0000 7ff3d48da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3d0100530 0x7ff3d01967a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49690/0 (socket says 172.21.15.70:49690) 2024-04-01T01:49:30.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.899+0000 7ff3cffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3d0101720 0x7ff3d0196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.899+0000 7ff3cffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3d0101720 0x7ff3d0196ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:51518/0 (socket says 172.21.15.70:51518) 2024-04-01T01:49:30.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.899+0000 7ff3d48da640 1 -- 172.21.15.70:0/2294294907 learned_addr learned my addr 172.21.15.70:0/2294294907 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.899+0000 7ff3d58dc640 1 -- 172.21.15.70:0/2294294907 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d010c0d0 con 0x7ff3d01020e0 2024-04-01T01:49:30.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.899+0000 7ff3d58dc640 1 -- 172.21.15.70:0/2294294907 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d010bf50 con 0x7ff3d0100530 2024-04-01T01:49:30.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.899+0000 7ff3d58dc640 1 -- 172.21.15.70:0/2294294907 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d010c250 con 0x7ff3d0101720 2024-04-01T01:49:30.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.900+0000 7ff3d48da640 1 -- 172.21.15.70:0/2294294907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3d0101720 msgr2=0x7ff3d0196ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.900+0000 7ff3d50db640 1 --2- 172.21.15.70:0/2294294907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3d01020e0 0x7ff3d019dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.900+0000 7ff3d48da640 1 --2- 172.21.15.70:0/2294294907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3d0101720 0x7ff3d0196ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.900+0000 7ff3d48da640 1 -- 172.21.15.70:0/2294294907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3d01020e0 msgr2=0x7ff3d019dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.900+0000 7ff3d48da640 1 --2- 172.21.15.70:0/2294294907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3d01020e0 0x7ff3d019dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.900+0000 7ff3d48da640 1 -- 172.21.15.70:0/2294294907 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d019e3c0 con 0x7ff3d0100530 2024-04-01T01:49:30.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.900+0000 7ff3d50db640 1 --2- 172.21.15.70:0/2294294907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3d01020e0 0x7ff3d019dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:49:30.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.900+0000 7ff3d48da640 1 --2- 172.21.15.70:0/2294294907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3d0100530 0x7ff3d01967a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4009eb0 tx=0x7ff3c400ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.900+0000 7ff3cdffb640 1 -- 172.21.15.70:0/2294294907 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c400cc60 con 0x7ff3d0100530 2024-04-01T01:49:30.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.900+0000 7ff3cdffb640 1 -- 172.21.15.70:0/2294294907 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff3c400ce00 con 0x7ff3d0100530 2024-04-01T01:49:30.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.900+0000 7ff3cdffb640 1 -- 172.21.15.70:0/2294294907 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c40115d0 con 0x7ff3d0100530 2024-04-01T01:49:30.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.901+0000 7ff3d58dc640 1 -- 172.21.15.70:0/2294294907 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3d0068910 con 0x7ff3d0100530 2024-04-01T01:49:30.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.901+0000 7ff3d58dc640 1 -- 172.21.15.70:0/2294294907 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3d0068e60 con 0x7ff3d0100530 2024-04-01T01:49:30.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.904+0000 7ff3cdffb640 1 -- 172.21.15.70:0/2294294907 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7ff3c4011030 con 0x7ff3d0100530 2024-04-01T01:49:30.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.904+0000 7ff3cdffb640 1 --2- 172.21.15.70:0/2294294907 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff3c00a6a80 0x7ff3c00a8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.905+0000 7ff3d97fe740 1 Processor -- start 2024-04-01T01:49:30.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.905+0000 7ff3cdffb640 1 -- 172.21.15.70:0/2294294907 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7ff3c40cefd0 con 0x7ff3d0100530 2024-04-01T01:49:30.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.905+0000 7ff3d97fe740 1 -- start start 2024-04-01T01:49:30.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.905+0000 7ff3d97fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc416a0 0x5630bcc41ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.905+0000 7ff3cffff640 1 --2- 172.21.15.70:0/2294294907 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff3c00a6a80 0x7ff3c00a8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.905+0000 7ff3d97fe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc42090 0x5630bcc46b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.905+0000 7ff3d97fe740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc47200 0x5630bcc49690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.905+0000 7ff3d48da640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc416a0 0x5630bcc41ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.905+0000 7ff3d48da640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc416a0 0x5630bcc41ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49150/0 (socket says 172.21.15.70:49150) 2024-04-01T01:49:30.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.905+0000 7ff3d48da640 1 -- 172.21.15.70:0/1711555080 learned_addr learned my addr 172.21.15.70:0/1711555080 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.905+0000 7ff3d97fe740 1 -- 172.21.15.70:0/1711555080 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5630bcba61b0 con 0x5630bcc416a0 2024-04-01T01:49:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.905+0000 7ff3d97fe740 1 -- 172.21.15.70:0/1711555080 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5630bcb4ec00 con 0x5630bcc47200 2024-04-01T01:49:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.905+0000 7ff3d97fe740 1 -- 172.21.15.70:0/1711555080 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5630bcba2250 con 0x5630bcc42090 2024-04-01T01:49:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.905+0000 7ff3d50db640 1 --2- 172.21.15.70:0/1711555080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc47200 0x5630bcc49690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.906+0000 7ff3d50db640 1 -- 172.21.15.70:0/1711555080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc42090 msgr2=0x5630bcc46b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:49:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.906+0000 7ff3d50db640 1 --2- 172.21.15.70:0/1711555080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc42090 0x5630bcc46b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.906+0000 7ff3d50db640 1 -- 172.21.15.70:0/1711555080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc416a0 msgr2=0x5630bcc41ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.906+0000 7ff3d50db640 1 --2- 172.21.15.70:0/1711555080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc416a0 0x5630bcc41ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.906+0000 7ff3d50db640 1 -- 172.21.15.70:0/1711555080 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630bcb6e3d0 con 0x5630bcc47200 2024-04-01T01:49:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.906+0000 7ff3d48da640 1 --2- 172.21.15.70:0/1711555080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc416a0 0x5630bcc41ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:49:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.906+0000 7ff3d50db640 1 --2- 172.21.15.70:0/1711555080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc47200 0x5630bcc49690 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8007e70 tx=0x7ff3c800c690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30d43c62b842e04 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.906+0000 7ff3baffd640 1 -- 172.21.15.70:0/1711555080 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8014070 con 0x5630bcc47200 2024-04-01T01:49:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.906+0000 7ff3baffd640 1 -- 172.21.15.70:0/1711555080 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff3c800fd00 con 0x5630bcc47200 2024-04-01T01:49:30.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.906+0000 7ff3baffd640 1 -- 172.21.15.70:0/1711555080 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c800eb60 con 0x5630bcc47200 2024-04-01T01:49:30.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.907+0000 7ff3cffff640 1 --2- 172.21.15.70:0/2294294907 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff3c00a6a80 0x7ff3c00a8f40 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc02c180 tx=0x7ff3bc0095b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.907+0000 7ff3d97fe740 1 -- 172.21.15.70:0/1711555080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc47200 msgr2=0x5630bcc49690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.908+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/1711555080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc47200 0x5630bcc49690 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8007e70 tx=0x7ff3c800c690 comp rx=0 tx=0).stop 2024-04-01T01:49:30.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.908+0000 7ff3d97fe740 1 -- 172.21.15.70:0/1711555080 shutdown_connections 2024-04-01T01:49:30.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.908+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/1711555080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc47200 0x5630bcc49690 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.908+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/1711555080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc42090 0x5630bcc46b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.908+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/1711555080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc416a0 0x5630bcc41ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.908+0000 7ff3d97fe740 1 -- 172.21.15.70:0/1711555080 >> 172.21.15.70:0/1711555080 conn(0x5630bcc3cc10 msgr2=0x5630bcc3d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.908+0000 7ff3d97fe740 1 -- 172.21.15.70:0/1711555080 shutdown_connections 2024-04-01T01:49:30.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.908+0000 7ff3d97fe740 1 -- 172.21.15.70:0/1711555080 wait complete. 2024-04-01T01:49:30.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.908+0000 7ff3d97fe740 1 -- start start 2024-04-01T01:49:30.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.909+0000 7ff3d97fe740 1 Processor -- start 2024-04-01T01:49:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.909+0000 7ff3d97fe740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc416a0 0x5630bcc4bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.910+0000 7ff3d97fe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc42090 0x5630bcc4c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.910+0000 7ff3d48da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc416a0 0x5630bcc4bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.910+0000 7ff3d48da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc416a0 0x5630bcc4bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49698/0 (socket says 172.21.15.70:49698) 2024-04-01T01:49:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.910+0000 7ff3d48da640 1 -- 172.21.15.70:0/2934079705 learned_addr learned my addr 172.21.15.70:0/2934079705 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.910+0000 7ff3cffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc42090 0x5630bcc4c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.910+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/2934079705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc47200 0x5630bcc53170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.910+0000 7ff3d97fe740 1 -- 172.21.15.70:0/2934079705 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5630bcc4a1b0 con 0x5630bcc47200 2024-04-01T01:49:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.910+0000 7ff3d97fe740 1 -- 172.21.15.70:0/2934079705 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5630bcb44470 con 0x5630bcc416a0 2024-04-01T01:49:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.910+0000 7ff3d97fe740 1 -- 172.21.15.70:0/2934079705 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5630bcc4a330 con 0x5630bcc42090 2024-04-01T01:49:30.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.910+0000 7ff3d50db640 1 --2- 172.21.15.70:0/2934079705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc47200 0x5630bcc53170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.911+0000 7ff3d48da640 1 -- 172.21.15.70:0/2934079705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc42090 msgr2=0x5630bcc4c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.911+0000 7ff3d48da640 1 --2- 172.21.15.70:0/2934079705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc42090 0x5630bcc4c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.911+0000 7ff3d48da640 1 -- 172.21.15.70:0/2934079705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc47200 msgr2=0x5630bcc53170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.911+0000 7ff3d48da640 1 --2- 172.21.15.70:0/2934079705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc47200 0x5630bcc53170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.911+0000 7ff3d48da640 1 -- 172.21.15.70:0/2934079705 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630bc8ba3f0 con 0x5630bcc416a0 2024-04-01T01:49:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.911+0000 7ff3d48da640 1 --2- 172.21.15.70:0/2934079705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc416a0 0x5630bcc4bbf0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7ff3c40039a0 tx=0x7ff3c409bd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c784fe50bab0f1d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.911+0000 7ff3d97fe740 10 client.0 _ll_register_callbacks cb 0x5630bcb61cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.911+0000 7ff3d97fe740 1 -- 172.21.15.70:0/2934079705 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3c800fd00 con 0x5630bcc416a0 2024-04-01T01:49:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.911+0000 7ff3affff640 10 client.30194 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:49:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.911+0000 7ff3affff640 1 -- 172.21.15.70:0/2934079705 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c409f7b0 con 0x5630bcc416a0 2024-04-01T01:49:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.911+0000 7ff3affff640 1 -- 172.21.15.70:0/2934079705 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7ff3c4011320 con 0x5630bcc416a0 2024-04-01T01:49:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.912+0000 7ff3affff640 1 -- 172.21.15.70:0/2934079705 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff3c40a4070 con 0x5630bcc416a0 2024-04-01T01:49:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.912+0000 7ff3d97fe740 10 client.30194 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.912+0000 7ff3d97fe740 1 -- 172.21.15.70:0/2934079705 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({fsmap=33}) -- 0x5630bc9847d0 con 0x5630bcc416a0 2024-04-01T01:49:30.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.912+0000 7ff3affff640 1 -- 172.21.15.70:0/2934079705 <== mon.2 v2:172.21.15.81:3300/0 4 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ff3c40b4200 con 0x5630bcc416a0 2024-04-01T01:49:30.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.912+0000 7ff3d97fe740 10 client.30194 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:30.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.912+0000 7ff3d97fe740 10 client.30194 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:49:30.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.912+0000 7ff3d97fe740 20 client.30194 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:30.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.912+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/2934079705 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5630bcc573e0 0x5630bcc59870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.912+0000 7ff3d97fe740 4 client.30194 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:49:30.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.912+0000 7ff3d97fe740 1 -- 172.21.15.70:0/2934079705 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff3c800fd00 con 0x5630bcc573e0 2024-04-01T01:49:30.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.913+0000 7ff3cffff640 1 --2- 172.21.15.70:0/2934079705 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5630bcc573e0 0x5630bcc59870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.913+0000 7ff3cffff640 1 --2- 172.21.15.70:0/2934079705 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5630bcc573e0 0x5630bcc59870 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7736f00692b677e8 server_cookie=2be40018dbf5828c in_seq=0 out_seq=0 2024-04-01T01:49:30.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.914+0000 7ff3affff640 10 client.30194 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:49:30.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.914+0000 7ff3affff640 1 -- 172.21.15.70:0/2934079705 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7ff3c800fd00 con 0x5630bcc573e0 2024-04-01T01:49:30.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.914+0000 7ff3affff640 10 client.30194 handle_command_reply: tid=0 2024-04-01T01:49:30.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.915+0000 7ff3d97fe740 1 client.30194 shutdown 2024-04-01T01:49:30.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.915+0000 7ff3d97fe740 1 -- 172.21.15.70:0/2934079705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc416a0 msgr2=0x5630bcc4bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.916+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/2934079705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc416a0 0x5630bcc4bbf0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7ff3c40039a0 tx=0x7ff3c409bd60 comp rx=0 tx=0).stop 2024-04-01T01:49:30.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.916+0000 7ff3d97fe740 1 -- 172.21.15.70:0/2934079705 shutdown_connections 2024-04-01T01:49:30.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.916+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/2934079705 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5630bcc573e0 0x5630bcc59870 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.916+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/2934079705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc47200 0x5630bcc53170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.916+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/2934079705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc42090 0x5630bcc4c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.916+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/2934079705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc416a0 0x5630bcc4bbf0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.916+0000 7ff3affff640 0 client.30194 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:49:30.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.916+0000 7ff3d97fe740 1 -- 172.21.15.70:0/2934079705 >> 172.21.15.70:0/2934079705 conn(0x5630bcc3cc10 msgr2=0x5630bcc3bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.917+0000 7ff3d97fe740 1 -- 172.21.15.70:0/2934079705 shutdown_connections 2024-04-01T01:49:30.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.917+0000 7ff3d97fe740 1 -- 172.21.15.70:0/2934079705 wait complete. 2024-04-01T01:49:30.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.917+0000 7ff3d97fe740 20 client.30194 trim_cache size 0 max 16384 2024-04-01T01:49:30.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.945+0000 7ff3d97fe740 1 Processor -- start 2024-04-01T01:49:30.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.945+0000 7ff3d97fe740 1 -- start start 2024-04-01T01:49:30.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.945+0000 7ff3d97fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc416a0 0x5630bcc3cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.945+0000 7ff3d97fe740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc42090 0x5630bcc4b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.945+0000 7ff3d97fe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc47200 0x5630bcc4bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.945+0000 7ff3d97fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d01a8070 con 0x5630bcc416a0 2024-04-01T01:49:30.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.945+0000 7ff3d97fe740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d01a9bb0 con 0x5630bcc42090 2024-04-01T01:49:30.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.945+0000 7ff3d97fe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d01a8740 con 0x5630bcc47200 2024-04-01T01:49:30.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.945+0000 7ff3d48da640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc416a0 0x5630bcc3cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.945+0000 7ff3d48da640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc416a0 0x5630bcc3cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49154/0 (socket says 172.21.15.70:49154) 2024-04-01T01:49:30.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.945+0000 7ff3d48da640 1 -- 172.21.15.70:0/566846636 learned_addr learned my addr 172.21.15.70:0/566846636 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.945+0000 7ff3cffff640 1 --2- 172.21.15.70:0/566846636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc42090 0x5630bcc4b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.945+0000 7ff3d50db640 1 --2- 172.21.15.70:0/566846636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc47200 0x5630bcc4bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.946+0000 7ff3d48da640 1 -- 172.21.15.70:0/566846636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc47200 msgr2=0x5630bcc4bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.946+0000 7ff3d48da640 1 --2- 172.21.15.70:0/566846636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc47200 0x5630bcc4bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.946+0000 7ff3d48da640 1 -- 172.21.15.70:0/566846636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc42090 msgr2=0x5630bcc4b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.946+0000 7ff3d48da640 1 --2- 172.21.15.70:0/566846636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc42090 0x5630bcc4b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.946+0000 7ff3d48da640 1 -- 172.21.15.70:0/566846636 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630bcc3d980 con 0x5630bcc416a0 2024-04-01T01:49:30.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.946+0000 7ff3d48da640 1 --2- 172.21.15.70:0/566846636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc416a0 0x5630bcc3cff0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7ff3c409b5f0 tx=0x7ff3c40bb050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb78bb80d8e9077 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.946+0000 7ff3ba7fc640 1 -- 172.21.15.70:0/566846636 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c40cc5e0 con 0x5630bcc416a0 2024-04-01T01:49:30.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.946+0000 7ff3ba7fc640 1 -- 172.21.15.70:0/566846636 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff3c409f7b0 con 0x5630bcc416a0 2024-04-01T01:49:30.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.946+0000 7ff3ba7fc640 1 -- 172.21.15.70:0/566846636 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4097610 con 0x5630bcc416a0 2024-04-01T01:49:30.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.947+0000 7ff3d97fe740 1 -- 172.21.15.70:0/566846636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc416a0 msgr2=0x5630bcc3cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.947+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/566846636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc416a0 0x5630bcc3cff0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7ff3c409b5f0 tx=0x7ff3c40bb050 comp rx=0 tx=0).stop 2024-04-01T01:49:30.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.948+0000 7ff3d97fe740 1 -- 172.21.15.70:0/566846636 shutdown_connections 2024-04-01T01:49:30.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.948+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/566846636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc47200 0x5630bcc4bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.948+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/566846636 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc42090 0x5630bcc4b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.948+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/566846636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc416a0 0x5630bcc3cff0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.948+0000 7ff3d97fe740 1 -- 172.21.15.70:0/566846636 >> 172.21.15.70:0/566846636 conn(0x5630bcc3cc10 msgr2=0x5630bcc42a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.948+0000 7ff3d97fe740 1 -- 172.21.15.70:0/566846636 shutdown_connections 2024-04-01T01:49:30.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.948+0000 7ff3d97fe740 1 -- 172.21.15.70:0/566846636 wait complete. 2024-04-01T01:49:30.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.948+0000 7ff3d97fe740 1 -- start start 2024-04-01T01:49:30.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.949+0000 7ff3d97fe740 1 Processor -- start 2024-04-01T01:49:30.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.949+0000 7ff3d97fe740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc416a0 0x5630bcc50350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.949+0000 7ff3d97fe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc42090 0x5630bcc50890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.949+0000 7ff3d97fe740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc47200 0x5630bcc518a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.949+0000 7ff3d97fe740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d01a94e0 con 0x5630bcc47200 2024-04-01T01:49:30.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.949+0000 7ff3d97fe740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d01a8e10 con 0x5630bcc416a0 2024-04-01T01:49:30.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.949+0000 7ff3d97fe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d01a6050 con 0x5630bcc42090 2024-04-01T01:49:30.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.949+0000 7ff3d97fe740 10 client.0 _ll_register_callbacks cb 0x7ff3c409b030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:30.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.950+0000 7ff3d48da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc416a0 0x5630bcc50350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.950+0000 7ff3d48da640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc416a0 0x5630bcc50350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49716/0 (socket says 172.21.15.70:49716) 2024-04-01T01:49:30.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.950+0000 7ff3d48da640 1 -- 172.21.15.70:0/3786303068 learned_addr learned my addr 172.21.15.70:0/3786303068 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:30.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.950+0000 7ff3d50db640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc47200 0x5630bcc518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.950+0000 7ff3d48da640 1 -- 172.21.15.70:0/3786303068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc42090 msgr2=0x5630bcc50890 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:49:30.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.950+0000 7ff3d48da640 1 --2- 172.21.15.70:0/3786303068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc42090 0x5630bcc50890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.950+0000 7ff3d48da640 1 -- 172.21.15.70:0/3786303068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc47200 msgr2=0x5630bcc518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:30.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.950+0000 7ff3d48da640 1 --2- 172.21.15.70:0/3786303068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc47200 0x5630bcc518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.950+0000 7ff3d48da640 1 -- 172.21.15.70:0/3786303068 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630bcc51e90 con 0x5630bcc416a0 2024-04-01T01:49:30.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.951+0000 7ff3d48da640 1 --2- 172.21.15.70:0/3786303068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc416a0 0x5630bcc50350 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7ff3c409b360 tx=0x7ff3c409b430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=668932b0589a684b server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:30.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.951+0000 7ff3affff640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:49:30.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.951+0000 7ff3affff640 1 -- 172.21.15.70:0/3786303068 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c409ade0 con 0x5630bcc416a0 2024-04-01T01:49:30.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.951+0000 7ff3affff640 1 -- 172.21.15.70:0/3786303068 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7ff3c4011320 con 0x5630bcc416a0 2024-04-01T01:49:30.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.951+0000 7ff3d97fe740 1 -- 172.21.15.70:0/3786303068 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3c409f7b0 con 0x5630bcc416a0 2024-04-01T01:49:30.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.956+0000 7ff3affff640 1 -- 172.21.15.70:0/3786303068 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff3c409a730 con 0x5630bcc416a0 2024-04-01T01:49:30.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.956+0000 7ff3d97fe740 10 client.30200 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:30.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.956+0000 7ff3d97fe740 1 -- 172.21.15.70:0/3786303068 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5630bcc53190 con 0x5630bcc416a0 2024-04-01T01:49:30.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.958+0000 7ff3affff640 1 -- 172.21.15.70:0/3786303068 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff3c40a0940 con 0x5630bcc416a0 2024-04-01T01:49:30.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.958+0000 7ff3affff640 1 -- 172.21.15.70:0/3786303068 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ff3c409f1b0 con 0x5630bcc416a0 2024-04-01T01:49:30.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.959+0000 7ff3d97fe740 10 client.30200 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:30.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.959+0000 7ff3d97fe740 10 client.30200 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:49:30.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.959+0000 7ff3d97fe740 20 client.30200 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:30.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.959+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/3786303068 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5630bccc66e0 0x5630bccc8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:30.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.959+0000 7ff3d97fe740 4 client.30200 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:49:30.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.959+0000 7ff3d97fe740 1 -- 172.21.15.70:0/3786303068 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7ff3c409f7b0 con 0x5630bccc66e0 2024-04-01T01:49:30.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.959+0000 7ff3cffff640 1 --2- 172.21.15.70:0/3786303068 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5630bccc66e0 0x5630bccc8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:30.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.960+0000 7ff3cffff640 1 --2- 172.21.15.70:0/3786303068 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5630bccc66e0 0x5630bccc8ad0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c9c7d8251762c929 server_cookie=ca36cf700001a649 in_seq=0 out_seq=0 2024-04-01T01:49:30.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.960+0000 7ff3affff640 10 client.30200 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:49:30.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.961+0000 7ff3affff640 1 -- 172.21.15.70:0/3786303068 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7ff3c409f7b0 con 0x5630bccc66e0 2024-04-01T01:49:30.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.961+0000 7ff3affff640 10 client.30200 handle_command_reply: tid=0 2024-04-01T01:49:30.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.961+0000 7ff3d97fe740 1 client.30200 shutdown 2024-04-01T01:49:30.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.961+0000 7ff3d97fe740 1 -- 172.21.15.70:0/3786303068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc416a0 msgr2=0x5630bcc50350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.961+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/3786303068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc416a0 0x5630bcc50350 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7ff3c409b360 tx=0x7ff3c409b430 comp rx=0 tx=0).stop 2024-04-01T01:49:30.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.961+0000 7ff3d97fe740 1 -- 172.21.15.70:0/3786303068 shutdown_connections 2024-04-01T01:49:30.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.961+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/3786303068 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x5630bccc66e0 0x5630bccc8ad0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.961+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/3786303068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5630bcc47200 0x5630bcc518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.961+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/3786303068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630bcc42090 0x5630bcc50890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.961+0000 7ff3d97fe740 1 --2- 172.21.15.70:0/3786303068 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630bcc416a0 0x5630bcc50350 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.961+0000 7ff3d97fe740 1 -- 172.21.15.70:0/3786303068 >> 172.21.15.70:0/3786303068 conn(0x5630bcc3cc10 msgr2=0x5630bcc3cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.961+0000 7ff3affff640 0 client.30200 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:49:30.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.962+0000 7ff3d97fe740 1 -- 172.21.15.70:0/3786303068 shutdown_connections 2024-04-01T01:49:30.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.962+0000 7ff3d97fe740 1 -- 172.21.15.70:0/3786303068 wait complete. 2024-04-01T01:49:30.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.962+0000 7ff3d97fe740 20 client.30200 trim_cache size 0 max 16384 2024-04-01T01:49:30.964 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:49:30.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.963+0000 7ff3d58dc640 1 -- 172.21.15.70:0/2294294907 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff3c00a6a80 msgr2=0x7ff3c00a8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.963+0000 7ff3d58dc640 1 --2- 172.21.15.70:0/2294294907 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff3c00a6a80 0x7ff3c00a8f40 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc02c180 tx=0x7ff3bc0095b0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.963+0000 7ff3d58dc640 1 -- 172.21.15.70:0/2294294907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3d0100530 msgr2=0x7ff3d01967a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:30.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.963+0000 7ff3d58dc640 1 --2- 172.21.15.70:0/2294294907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3d0100530 0x7ff3d01967a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4009eb0 tx=0x7ff3c400ed10 comp rx=0 tx=0).stop 2024-04-01T01:49:30.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.963+0000 7ff3d58dc640 1 -- 172.21.15.70:0/2294294907 shutdown_connections 2024-04-01T01:49:30.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.963+0000 7ff3d58dc640 1 --2- 172.21.15.70:0/2294294907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3d01020e0 0x7ff3d019dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.963+0000 7ff3d58dc640 1 --2- 172.21.15.70:0/2294294907 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7ff3c00a6a80 0x7ff3c00a8f40 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.963+0000 7ff3d58dc640 1 --2- 172.21.15.70:0/2294294907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3d0101720 0x7ff3d0196ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.963+0000 7ff3d58dc640 1 --2- 172.21.15.70:0/2294294907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3d0100530 0x7ff3d01967a0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:30.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.963+0000 7ff3d58dc640 1 -- 172.21.15.70:0/2294294907 >> 172.21.15.70:0/2294294907 conn(0x7ff3d00fbac0 msgr2=0x7ff3d0104af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:30.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.964+0000 7ff3d58dc640 1 -- 172.21.15.70:0/2294294907 shutdown_connections 2024-04-01T01:49:30.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:30.964+0000 7ff3d58dc640 1 -- 172.21.15.70:0/2294294907 wait complete. 2024-04-01T01:49:30.990 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:49:31.003 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:30 smithi070 ceph-mon[32179]: pgmap v840: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 12 MiB/s wr, 779 op/s 2024-04-01T01:49:31.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:30 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3983928844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:49:31.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:30 smithi081 ceph-mon[35142]: pgmap v840: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 12 MiB/s wr, 779 op/s 2024-04-01T01:49:31.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:30 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3983928844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:49:31.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.174+0000 7f766553e640 1 -- 172.21.15.70:0/3257893999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7660108750 msgr2=0x7f766010ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:31.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.174+0000 7f766553e640 1 --2- 172.21.15.70:0/3257893999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7660108750 0x7f766010ab80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f765401a910 tx=0x7f765403cc70 comp rx=0 tx=0).stop 2024-04-01T01:49:31.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.174+0000 7f766553e640 1 -- 172.21.15.70:0/3257893999 shutdown_connections 2024-04-01T01:49:31.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.174+0000 7f766553e640 1 --2- 172.21.15.70:0/3257893999 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f766010da70 0x7f766010def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.174+0000 7f766553e640 1 --2- 172.21.15.70:0/3257893999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f766010b0c0 0x7f766010d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.174+0000 7f766553e640 1 --2- 172.21.15.70:0/3257893999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7660108750 0x7f766010ab80 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.174+0000 7f766553e640 1 -- 172.21.15.70:0/3257893999 >> 172.21.15.70:0/3257893999 conn(0x7f766006e5b0 msgr2=0x7f76600709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:31.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.175+0000 7f766553e640 1 -- 172.21.15.70:0/3257893999 shutdown_connections 2024-04-01T01:49:31.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.175+0000 7f766553e640 1 -- 172.21.15.70:0/3257893999 wait complete. 2024-04-01T01:49:31.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.176+0000 7f766553e640 1 Processor -- start 2024-04-01T01:49:31.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.176+0000 7f766553e640 1 -- start start 2024-04-01T01:49:31.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.176+0000 7f766553e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7660108750 0x7f766019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.177+0000 7f766553e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f766010b0c0 0x7f766019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.177+0000 7f766553e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f766010da70 0x7f76601a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.177+0000 7f766553e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f76601118c0 con 0x7f7660108750 2024-04-01T01:49:31.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.177+0000 7f766553e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7660111740 con 0x7f766010b0c0 2024-04-01T01:49:31.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.177+0000 7f766553e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7660111a40 con 0x7f766010da70 2024-04-01T01:49:31.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.177+0000 7f765e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f766010b0c0 0x7f766019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:31.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.177+0000 7f765e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f766010b0c0 0x7f766019e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49724/0 (socket says 172.21.15.70:49724) 2024-04-01T01:49:31.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.177+0000 7f765e7fc640 1 -- 172.21.15.70:0/3751262408 learned_addr learned my addr 172.21.15.70:0/3751262408 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:31.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.177+0000 7f765f7fe640 1 --2- 172.21.15.70:0/3751262408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f766010da70 0x7f76601a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:31.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.177+0000 7f765effd640 1 --2- 172.21.15.70:0/3751262408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7660108750 0x7f766019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:31.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.177+0000 7f765e7fc640 1 -- 172.21.15.70:0/3751262408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f766010da70 msgr2=0x7f76601a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:31.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.177+0000 7f765e7fc640 1 --2- 172.21.15.70:0/3751262408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f766010da70 0x7f76601a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.177+0000 7f765e7fc640 1 -- 172.21.15.70:0/3751262408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7660108750 msgr2=0x7f766019e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:31.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.177+0000 7f765e7fc640 1 --2- 172.21.15.70:0/3751262408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7660108750 0x7f766019e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.177+0000 7f765e7fc640 1 -- 172.21.15.70:0/3751262408 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76601a5bd0 con 0x7f766010b0c0 2024-04-01T01:49:31.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.178+0000 7f765e7fc640 1 --2- 172.21.15.70:0/3751262408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f766010b0c0 0x7f766019e540 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f765000e9f0 tx=0x7f765000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:31.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.178+0000 7f7647fff640 1 -- 172.21.15.70:0/3751262408 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765000cd80 con 0x7f766010b0c0 2024-04-01T01:49:31.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.178+0000 7f7647fff640 1 -- 172.21.15.70:0/3751262408 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76500108e0 con 0x7f766010b0c0 2024-04-01T01:49:31.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.178+0000 7f7647fff640 1 -- 172.21.15.70:0/3751262408 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650018920 con 0x7f766010b0c0 2024-04-01T01:49:31.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.178+0000 7f766553e640 1 -- 172.21.15.70:0/3751262408 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76601a5ec0 con 0x7f766010b0c0 2024-04-01T01:49:31.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.178+0000 7f766553e640 1 -- 172.21.15.70:0/3751262408 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76601a6330 con 0x7f766010b0c0 2024-04-01T01:49:31.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.182+0000 7f7668201740 1 Processor -- start 2024-04-01T01:49:31.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.182+0000 7f7647fff640 1 -- 172.21.15.70:0/3751262408 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f7650010a90 con 0x7f766010b0c0 2024-04-01T01:49:31.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.183+0000 7f7668201740 1 -- start start 2024-04-01T01:49:31.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.183+0000 7f7668201740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562203765490 0x5622037658b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.183+0000 7f7668201740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765e80 0x56220376a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.183+0000 7f7668201740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56220376aff0 0x56220376d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.183+0000 7f7668201740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5622036ca1b0 con 0x56220376aff0 2024-04-01T01:49:31.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.183+0000 7f7668201740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562203672c00 con 0x562203765490 2024-04-01T01:49:31.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.183+0000 7f7668201740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5622036c6250 con 0x562203765e80 2024-04-01T01:49:31.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.183+0000 7f7647fff640 1 --2- 172.21.15.70:0/3751262408 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f76480a2520 0x7f76480a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.183+0000 7f765effd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765e80 0x56220376a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:31.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.183+0000 7f7647fff640 1 -- 172.21.15.70:0/3751262408 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f76500cedd0 con 0x7f766010b0c0 2024-04-01T01:49:31.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.183+0000 7f765effd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765e80 0x56220376a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:51594/0 (socket says 172.21.15.70:51594) 2024-04-01T01:49:31.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.183+0000 7f765effd640 1 -- 172.21.15.70:0/911538875 learned_addr learned my addr 172.21.15.70:0/911538875 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:31.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.184+0000 7f765e7fc640 1 --2- 172.21.15.70:0/911538875 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56220376aff0 0x56220376d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.184+0000 7f765effd640 1 -- 172.21.15.70:0/911538875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562203765490 msgr2=0x5622037658b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:49:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.184+0000 7f765effd640 1 --2- 172.21.15.70:0/911538875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562203765490 0x5622037658b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.184+0000 7f765effd640 1 -- 172.21.15.70:0/911538875 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56220376aff0 msgr2=0x56220376d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.184+0000 7f765effd640 1 --2- 172.21.15.70:0/911538875 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56220376aff0 0x56220376d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.184+0000 7f765effd640 1 -- 172.21.15.70:0/911538875 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622036923d0 con 0x562203765e80 2024-04-01T01:49:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.184+0000 7f765e7fc640 1 --2- 172.21.15.70:0/911538875 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56220376aff0 0x56220376d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:49:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.184+0000 7f765effd640 1 --2- 172.21.15.70:0/911538875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765e80 0x56220376a980 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f76540048a0 tx=0x7f7654005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3665c569cd9a94d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.185+0000 7f76457fa640 1 -- 172.21.15.70:0/911538875 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654004940 con 0x562203765e80 2024-04-01T01:49:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.185+0000 7f76457fa640 1 -- 172.21.15.70:0/911538875 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7654004ae0 con 0x562203765e80 2024-04-01T01:49:31.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.185+0000 7f76457fa640 1 -- 172.21.15.70:0/911538875 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765403d820 con 0x562203765e80 2024-04-01T01:49:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.186+0000 7f7668201740 1 -- 172.21.15.70:0/911538875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765e80 msgr2=0x56220376a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.186+0000 7f7668201740 1 --2- 172.21.15.70:0/911538875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765e80 0x56220376a980 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f76540048a0 tx=0x7f7654005be0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.187+0000 7f765f7fe640 1 --2- 172.21.15.70:0/3751262408 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f76480a2520 0x7f76480a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.187+0000 7f7668201740 1 -- 172.21.15.70:0/911538875 shutdown_connections 2024-04-01T01:49:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.187+0000 7f7668201740 1 --2- 172.21.15.70:0/911538875 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56220376aff0 0x56220376d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.187+0000 7f7668201740 1 --2- 172.21.15.70:0/911538875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765e80 0x56220376a980 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.187+0000 7f7668201740 1 --2- 172.21.15.70:0/911538875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562203765490 0x5622037658b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.187+0000 7f7668201740 1 -- 172.21.15.70:0/911538875 >> 172.21.15.70:0/911538875 conn(0x562203760a00 msgr2=0x562203760e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.187+0000 7f7668201740 1 -- 172.21.15.70:0/911538875 shutdown_connections 2024-04-01T01:49:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.187+0000 7f7668201740 1 -- 172.21.15.70:0/911538875 wait complete. 2024-04-01T01:49:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.188+0000 7f7668201740 1 -- start start 2024-04-01T01:49:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.188+0000 7f7668201740 1 Processor -- start 2024-04-01T01:49:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.189+0000 7f7668201740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562203765490 0x56220376f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.189+0000 7f7668201740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562203765e80 0x56220376feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.189+0000 7f7668201740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56220376aff0 0x562203776ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.189+0000 7f7668201740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56220376df40 con 0x562203765e80 2024-04-01T01:49:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.189+0000 7f7668201740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x562203668470 con 0x562203765490 2024-04-01T01:49:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.189+0000 7f7668201740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56220376e0c0 con 0x56220376aff0 2024-04-01T01:49:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.189+0000 7f765f7fe640 1 --2- 172.21.15.70:0/3751262408 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f76480a2520 0x7f76480a49e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x5622037666e0 tx=0x7f7658009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.189+0000 7f7668201740 10 client.0 _ll_register_callbacks cb 0x562203685cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.189+0000 7f765f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56220376aff0 0x562203776ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.189+0000 7f765f7fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56220376aff0 0x562203776ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:51606/0 (socket says 172.21.15.70:51606) 2024-04-01T01:49:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.189+0000 7f765f7fe640 1 -- 172.21.15.70:0/7716535 learned_addr learned my addr 172.21.15.70:0/7716535 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.189+0000 7f765e7fc640 1 --2- 172.21.15.70:0/7716535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562203765e80 0x56220376feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.190+0000 7f765f7fe640 1 -- 172.21.15.70:0/7716535 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562203765490 msgr2=0x56220376f970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:49:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.190+0000 7f765f7fe640 1 --2- 172.21.15.70:0/7716535 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562203765490 0x56220376f970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.190+0000 7f765f7fe640 1 -- 172.21.15.70:0/7716535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562203765e80 msgr2=0x56220376feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.190+0000 7f765f7fe640 1 --2- 172.21.15.70:0/7716535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562203765e80 0x56220376feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.190+0000 7f765f7fe640 1 -- 172.21.15.70:0/7716535 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622033de3f0 con 0x56220376aff0 2024-04-01T01:49:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.190+0000 7f765f7fe640 1 --2- 172.21.15.70:0/7716535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56220376aff0 0x562203776ef0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f76580099c0 tx=0x7f7658004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9ee0155297e4130 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.190+0000 7f7642ff5640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:49:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.190+0000 7f7642ff5640 1 -- 172.21.15.70:0/7716535 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7658029060 con 0x56220376aff0 2024-04-01T01:49:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.190+0000 7f7668201740 1 -- 172.21.15.70:0/7716535 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7654004ae0 con 0x56220376aff0 2024-04-01T01:49:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.190+0000 7f7642ff5640 1 -- 172.21.15.70:0/7716535 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f765801cb60 con 0x56220376aff0 2024-04-01T01:49:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.191+0000 7f7642ff5640 1 -- 172.21.15.70:0/7716535 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7658030050 con 0x56220376aff0 2024-04-01T01:49:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.191+0000 7f7668201740 10 client.29926 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.191+0000 7f7668201740 1 -- 172.21.15.70:0/7716535 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5622034a87d0 con 0x56220376aff0 2024-04-01T01:49:31.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.191+0000 7f7642ff5640 1 -- 172.21.15.70:0/7716535 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7654004ae0 con 0x56220376aff0 2024-04-01T01:49:31.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.192+0000 7f7642ff5640 1 -- 172.21.15.70:0/7716535 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f7658018040 con 0x56220376aff0 2024-04-01T01:49:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.192+0000 7f7668201740 10 client.29926 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.192+0000 7f7668201740 10 client.29926 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:49:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.192+0000 7f7668201740 20 client.29926 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.192+0000 7f7668201740 1 --2- 172.21.15.70:0/7716535 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56220377b4e0 0x56220377d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.192+0000 7f7668201740 4 client.29926 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:49:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.192+0000 7f7668201740 1 -- 172.21.15.70:0/7716535 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622036d76f0 con 0x56220377b4e0 2024-04-01T01:49:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.193+0000 7f765effd640 1 --2- 172.21.15.70:0/7716535 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56220377b4e0 0x56220377d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.194+0000 7f765effd640 1 --2- 172.21.15.70:0/7716535 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56220377b4e0 0x56220377d9a0 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f45fefefbb86f82e server_cookie=25169205453f6dc9 in_seq=0 out_seq=0 2024-04-01T01:49:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.194+0000 7f7642ff5640 10 client.29926 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.195+0000 7f7642ff5640 1 -- 172.21.15.70:0/7716535 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5622036d76f0 con 0x56220377b4e0 2024-04-01T01:49:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.195+0000 7f7642ff5640 10 client.29926 handle_command_reply: tid=0 2024-04-01T01:49:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.195+0000 7f7668201740 1 client.29926 shutdown 2024-04-01T01:49:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.196+0000 7f7668201740 1 -- 172.21.15.70:0/7716535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56220376aff0 msgr2=0x562203776ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:31.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.196+0000 7f7668201740 1 --2- 172.21.15.70:0/7716535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56220376aff0 0x562203776ef0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f76580099c0 tx=0x7f7658004ab0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.196+0000 7f7668201740 1 -- 172.21.15.70:0/7716535 shutdown_connections 2024-04-01T01:49:31.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.196+0000 7f7668201740 1 --2- 172.21.15.70:0/7716535 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56220377b4e0 0x56220377d9a0 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.196+0000 7f7668201740 1 --2- 172.21.15.70:0/7716535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56220376aff0 0x562203776ef0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.196+0000 7f7668201740 1 --2- 172.21.15.70:0/7716535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562203765e80 0x56220376feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.196+0000 7f7668201740 1 --2- 172.21.15.70:0/7716535 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562203765490 0x56220376f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.196+0000 7f7668201740 1 -- 172.21.15.70:0/7716535 >> 172.21.15.70:0/7716535 conn(0x562203760a00 msgr2=0x56220375fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:31.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.196+0000 7f7642ff5640 0 client.29926 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:31.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.196+0000 7f7668201740 1 -- 172.21.15.70:0/7716535 shutdown_connections 2024-04-01T01:49:31.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.199+0000 7f7668201740 1 -- 172.21.15.70:0/7716535 wait complete. 2024-04-01T01:49:31.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.200+0000 7f7668201740 20 client.29926 trim_cache size 0 max 16384 2024-04-01T01:49:31.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.226+0000 7f7668201740 1 Processor -- start 2024-04-01T01:49:31.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.226+0000 7f7668201740 1 -- start start 2024-04-01T01:49:31.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.227+0000 7f7668201740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765490 0x562203767410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.227+0000 7f7668201740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562203765e80 0x562203767950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.227+0000 7f7668201740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56220376aff0 0x562203776eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.227+0000 7f7668201740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f76601b0ae0 con 0x56220376aff0 2024-04-01T01:49:31.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.227+0000 7f7668201740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f76601b2620 con 0x562203765e80 2024-04-01T01:49:31.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.227+0000 7f7668201740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f76601b11b0 con 0x562203765490 2024-04-01T01:49:31.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.227+0000 7f765e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562203765e80 0x562203767950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:31.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.227+0000 7f765e7fc640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562203765e80 0x562203767950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49750/0 (socket says 172.21.15.70:49750) 2024-04-01T01:49:31.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.227+0000 7f765e7fc640 1 -- 172.21.15.70:0/197643800 learned_addr learned my addr 172.21.15.70:0/197643800 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:31.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.228+0000 7f765f7fe640 1 --2- 172.21.15.70:0/197643800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56220376aff0 0x562203776eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:31.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.228+0000 7f765effd640 1 --2- 172.21.15.70:0/197643800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765490 0x562203767410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:31.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.228+0000 7f765effd640 1 -- 172.21.15.70:0/197643800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562203765e80 msgr2=0x562203767950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:31.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.228+0000 7f765effd640 1 --2- 172.21.15.70:0/197643800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562203765e80 0x562203767950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.228+0000 7f765effd640 1 -- 172.21.15.70:0/197643800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56220376aff0 msgr2=0x562203776eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:31.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.228+0000 7f765effd640 1 --2- 172.21.15.70:0/197643800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56220376aff0 0x562203776eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.228+0000 7f765effd640 1 -- 172.21.15.70:0/197643800 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56220376a690 con 0x562203765490 2024-04-01T01:49:31.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.229+0000 7f765effd640 1 --2- 172.21.15.70:0/197643800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765490 0x562203767410 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x562203761970 tx=0x7f7654005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c8d9978167819cf server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:31.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.229+0000 7f7644ff9640 1 -- 172.21.15.70:0/197643800 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654064070 con 0x562203765490 2024-04-01T01:49:31.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.229+0000 7f7644ff9640 1 -- 172.21.15.70:0/197643800 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7654004a60 con 0x562203765490 2024-04-01T01:49:31.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.229+0000 7f7644ff9640 1 -- 172.21.15.70:0/197643800 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654004f40 con 0x562203765490 2024-04-01T01:49:31.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.231+0000 7f7668201740 1 -- 172.21.15.70:0/197643800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765490 msgr2=0x562203767410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:31.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.231+0000 7f7668201740 1 --2- 172.21.15.70:0/197643800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765490 0x562203767410 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x562203761970 tx=0x7f7654005be0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.232+0000 7f7668201740 1 -- 172.21.15.70:0/197643800 shutdown_connections 2024-04-01T01:49:31.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.232+0000 7f7668201740 1 --2- 172.21.15.70:0/197643800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56220376aff0 0x562203776eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.232+0000 7f7668201740 1 --2- 172.21.15.70:0/197643800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562203765e80 0x562203767950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.232+0000 7f7668201740 1 --2- 172.21.15.70:0/197643800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765490 0x562203767410 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.232+0000 7f7668201740 1 -- 172.21.15.70:0/197643800 >> 172.21.15.70:0/197643800 conn(0x562203760a00 msgr2=0x56220376e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:31.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.232+0000 7f7668201740 1 -- 172.21.15.70:0/197643800 shutdown_connections 2024-04-01T01:49:31.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.232+0000 7f7668201740 1 -- 172.21.15.70:0/197643800 wait complete. 2024-04-01T01:49:31.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.233+0000 7f7668201740 1 -- start start 2024-04-01T01:49:31.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.233+0000 7f7668201740 1 Processor -- start 2024-04-01T01:49:31.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.234+0000 7f7668201740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765490 0x562203773e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.234+0000 7f7668201740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562203765e80 0x562203774390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.234+0000 7f7668201740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56220376aff0 0x5622037df890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.234+0000 7f7668201740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f76601b1f50 con 0x562203765e80 2024-04-01T01:49:31.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.234+0000 7f7668201740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f76601b1880 con 0x56220376aff0 2024-04-01T01:49:31.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.234+0000 7f7668201740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f76601aeac0 con 0x562203765490 2024-04-01T01:49:31.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.234+0000 7f7668201740 10 client.0 _ll_register_callbacks cb 0x7f7658003f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:31.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.235+0000 7f765f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56220376aff0 0x5622037df890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:31.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.235+0000 7f765f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56220376aff0 0x5622037df890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49754/0 (socket says 172.21.15.70:49754) 2024-04-01T01:49:31.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.235+0000 7f765effd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765490 0x562203773e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:31.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.235+0000 7f765f7fe640 1 -- 172.21.15.70:0/2960097498 learned_addr learned my addr 172.21.15.70:0/2960097498 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:31.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.235+0000 7f765f7fe640 1 -- 172.21.15.70:0/2960097498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765490 msgr2=0x562203773e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:31.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.235+0000 7f765f7fe640 1 --2- 172.21.15.70:0/2960097498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765490 0x562203773e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.235+0000 7f765f7fe640 1 -- 172.21.15.70:0/2960097498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562203765e80 msgr2=0x562203774390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-01T01:49:31.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.235+0000 7f765f7fe640 1 --2- 172.21.15.70:0/2960097498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562203765e80 0x562203774390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.235+0000 7f765f7fe640 1 -- 172.21.15.70:0/2960097498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622037dfdd0 con 0x56220376aff0 2024-04-01T01:49:31.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.235+0000 7f765f7fe640 1 --2- 172.21.15.70:0/2960097498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56220376aff0 0x5622037df890 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f765801aed0 tx=0x7f7658009fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b6a92e735fbadaa server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:31.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.235+0000 7f7642ff5640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:49:31.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.236+0000 7f7642ff5640 1 -- 172.21.15.70:0/2960097498 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7658038070 con 0x56220376aff0 2024-04-01T01:49:31.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.236+0000 7f7668201740 1 -- 172.21.15.70:0/2960097498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7654004a60 con 0x56220376aff0 2024-04-01T01:49:31.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.236+0000 7f7642ff5640 1 -- 172.21.15.70:0/2960097498 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f7658004d20 con 0x56220376aff0 2024-04-01T01:49:31.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.236+0000 7f7642ff5640 1 -- 172.21.15.70:0/2960097498 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7658041050 con 0x56220376aff0 2024-04-01T01:49:31.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.236+0000 7f7668201740 10 client.30212 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:31.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.236+0000 7f7668201740 1 -- 172.21.15.70:0/2960097498 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5622037e1040 con 0x56220376aff0 2024-04-01T01:49:31.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.236+0000 7f7642ff5640 1 -- 172.21.15.70:0/2960097498 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7654004a60 con 0x56220376aff0 2024-04-01T01:49:31.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.236+0000 7f7642ff5640 1 -- 172.21.15.70:0/2960097498 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f765803b020 con 0x56220376aff0 2024-04-01T01:49:31.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.236+0000 7f7668201740 10 client.30212 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:31.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.236+0000 7f7668201740 10 client.30212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:49:31.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.236+0000 7f7668201740 20 client.30212 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:31.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.236+0000 7f7668201740 1 --2- 172.21.15.70:0/2960097498 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5622037e1aa0 0x5622037e3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:31.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.236+0000 7f7668201740 4 client.30212 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:49:31.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.236+0000 7f7668201740 1 -- 172.21.15.70:0/2960097498 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5622036d1a40 con 0x5622037e1aa0 2024-04-01T01:49:31.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.237+0000 7f765effd640 1 --2- 172.21.15.70:0/2960097498 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5622037e1aa0 0x5622037e3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:31.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.237+0000 7f765effd640 1 --2- 172.21.15.70:0/2960097498 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5622037e1aa0 0x5622037e3ec0 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24f0de5d6f6ac8b8 server_cookie=56c0d66c4178e073 in_seq=0 out_seq=0 2024-04-01T01:49:31.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.237+0000 7f7642ff5640 10 client.30212 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:31.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.238+0000 7f7642ff5640 1 -- 172.21.15.70:0/2960097498 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5622036d1a40 con 0x5622037e1aa0 2024-04-01T01:49:31.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.238+0000 7f7642ff5640 10 client.30212 handle_command_reply: tid=0 2024-04-01T01:49:31.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.238+0000 7f7668201740 1 client.30212 shutdown 2024-04-01T01:49:31.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.238+0000 7f7668201740 1 -- 172.21.15.70:0/2960097498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56220376aff0 msgr2=0x5622037df890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:31.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.238+0000 7f7668201740 1 --2- 172.21.15.70:0/2960097498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56220376aff0 0x5622037df890 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f765801aed0 tx=0x7f7658009fd0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.238+0000 7f7668201740 1 -- 172.21.15.70:0/2960097498 shutdown_connections 2024-04-01T01:49:31.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.238+0000 7f7668201740 1 --2- 172.21.15.70:0/2960097498 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5622037e1aa0 0x5622037e3ec0 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.238+0000 7f7668201740 1 --2- 172.21.15.70:0/2960097498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56220376aff0 0x5622037df890 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.238+0000 7f7668201740 1 --2- 172.21.15.70:0/2960097498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562203765e80 0x562203774390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.238+0000 7f7668201740 1 --2- 172.21.15.70:0/2960097498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562203765490 0x562203773e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.238+0000 7f7668201740 1 -- 172.21.15.70:0/2960097498 >> 172.21.15.70:0/2960097498 conn(0x562203760a00 msgr2=0x56220376f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:31.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.238+0000 7f7642ff5640 0 client.30212 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:31.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.238+0000 7f7668201740 1 -- 172.21.15.70:0/2960097498 shutdown_connections 2024-04-01T01:49:31.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.238+0000 7f7668201740 1 -- 172.21.15.70:0/2960097498 wait complete. 2024-04-01T01:49:31.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.239+0000 7f7668201740 20 client.30212 trim_cache size 0 max 16384 2024-04-01T01:49:31.242 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:49:31.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.240+0000 7f766553e640 1 -- 172.21.15.70:0/3751262408 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f76480a2520 msgr2=0x7f76480a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:31.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.240+0000 7f766553e640 1 --2- 172.21.15.70:0/3751262408 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f76480a2520 0x7f76480a49e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x5622037666e0 tx=0x7f7658009500 comp rx=0 tx=0).stop 2024-04-01T01:49:31.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.240+0000 7f766553e640 1 -- 172.21.15.70:0/3751262408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f766010b0c0 msgr2=0x7f766019e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:31.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.240+0000 7f766553e640 1 --2- 172.21.15.70:0/3751262408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f766010b0c0 0x7f766019e540 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f765000e9f0 tx=0x7f765000eeb0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.240+0000 7f766553e640 1 -- 172.21.15.70:0/3751262408 shutdown_connections 2024-04-01T01:49:31.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.240+0000 7f766553e640 1 --2- 172.21.15.70:0/3751262408 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f76480a2520 0x7f76480a49e0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.240+0000 7f766553e640 1 --2- 172.21.15.70:0/3751262408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f766010da70 0x7f76601a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.240+0000 7f766553e640 1 --2- 172.21.15.70:0/3751262408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f766010b0c0 0x7f766019e540 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.240+0000 7f766553e640 1 --2- 172.21.15.70:0/3751262408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7660108750 0x7f766019e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:31.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.241+0000 7f766553e640 1 -- 172.21.15.70:0/3751262408 >> 172.21.15.70:0/3751262408 conn(0x7f766006e5b0 msgr2=0x7f7660109030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:31.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.241+0000 7f766553e640 1 -- 172.21.15.70:0/3751262408 shutdown_connections 2024-04-01T01:49:31.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:31.241+0000 7f766553e640 1 -- 172.21.15.70:0/3751262408 wait complete. 2024-04-01T01:49:31.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:30 smithi189 ceph-mon[35753]: pgmap v840: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 12 MiB/s wr, 779 op/s 2024-04-01T01:49:31.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:30 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3983928844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:49:31.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 885.0 s, 835.0 tps, lat 119.077 ms stddev 94.905 2024-04-01T01:49:32.263 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-01T01:49:32.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:49:32.263 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-01T01:49:32.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.455+0000 7f72ecbfd640 1 -- 172.21.15.70:0/1280424860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72e8101980 msgr2=0x7f72e8108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.455+0000 7f72ecbfd640 1 --2- 172.21.15.70:0/1280424860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72e8101980 0x7f72e8108210 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009860 tx=0x7f72d802bb70 comp rx=0 tx=0).stop 2024-04-01T01:49:32.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.455+0000 7f72ecbfd640 1 -- 172.21.15.70:0/1280424860 shutdown_connections 2024-04-01T01:49:32.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.455+0000 7f72ecbfd640 1 --2- 172.21.15.70:0/1280424860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72e8101980 0x7f72e8108210 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.455+0000 7f72ecbfd640 1 --2- 172.21.15.70:0/1280424860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72e8100fc0 0x7f72e8101440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.455+0000 7f72ecbfd640 1 --2- 172.21.15.70:0/1280424860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72e80ffdd0 0x7f72e81001f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.455+0000 7f72ecbfd640 1 -- 172.21.15.70:0/1280424860 >> 172.21.15.70:0/1280424860 conn(0x7f72e80fb340 msgr2=0x7f72e80fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:32.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.455+0000 7f72ecbfd640 1 -- 172.21.15.70:0/1280424860 shutdown_connections 2024-04-01T01:49:32.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.456+0000 7f72ecbfd640 1 -- 172.21.15.70:0/1280424860 wait complete. 2024-04-01T01:49:32.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.457+0000 7f72ecbfd640 1 Processor -- start 2024-04-01T01:49:32.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.458+0000 7f72ecbfd640 1 -- start start 2024-04-01T01:49:32.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.458+0000 7f72ecbfd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72e80ffdd0 0x7f72e8199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.458+0000 7f72ecbfd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72e8100fc0 0x7f72e819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.458+0000 7f72ecbfd640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72e8101980 0x7f72e81a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.458+0000 7f72ecbfd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f72e810ab30 con 0x7f72e8100fc0 2024-04-01T01:49:32.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.458+0000 7f72ecbfd640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f72e810a9b0 con 0x7f72e8101980 2024-04-01T01:49:32.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.458+0000 7f72ecbfd640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f72e810acb0 con 0x7f72e80ffdd0 2024-04-01T01:49:32.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.458+0000 7f72e5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72e8100fc0 0x7f72e819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.458+0000 7f72e5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72e8100fc0 0x7f72e819a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49250/0 (socket says 172.21.15.70:49250) 2024-04-01T01:49:32.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.458+0000 7f72e5d9b640 1 -- 172.21.15.70:0/1283321320 learned_addr learned my addr 172.21.15.70:0/1283321320 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:32.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.459+0000 7f72e6d9d640 1 --2- 172.21.15.70:0/1283321320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72e8101980 0x7f72e81a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.459+0000 7f72e659c640 1 --2- 172.21.15.70:0/1283321320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72e80ffdd0 0x7f72e8199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.459+0000 7f72e5d9b640 1 -- 172.21.15.70:0/1283321320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72e80ffdd0 msgr2=0x7f72e8199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.459+0000 7f72e5d9b640 1 --2- 172.21.15.70:0/1283321320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72e80ffdd0 0x7f72e8199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.459+0000 7f72e5d9b640 1 -- 172.21.15.70:0/1283321320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72e8101980 msgr2=0x7f72e81a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.459+0000 7f72e5d9b640 1 --2- 172.21.15.70:0/1283321320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72e8101980 0x7f72e81a10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.459+0000 7f72e5d9b640 1 -- 172.21.15.70:0/1283321320 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e81a1740 con 0x7f72e8100fc0 2024-04-01T01:49:32.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.459+0000 7f72e5d9b640 1 --2- 172.21.15.70:0/1283321320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72e8100fc0 0x7f72e819a060 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f72e000de60 tx=0x7f72e000b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:32.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.460+0000 7f72d77fe640 1 -- 172.21.15.70:0/1283321320 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e001b6b0 con 0x7f72e8100fc0 2024-04-01T01:49:32.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.460+0000 7f72d77fe640 1 -- 172.21.15.70:0/1283321320 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72e001bd00 con 0x7f72e8100fc0 2024-04-01T01:49:32.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.460+0000 7f72d77fe640 1 -- 172.21.15.70:0/1283321320 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e001ed80 con 0x7f72e8100fc0 2024-04-01T01:49:32.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.462+0000 7f72ecbfd640 1 -- 172.21.15.70:0/1283321320 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e810c6b0 con 0x7f72e8100fc0 2024-04-01T01:49:32.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.462+0000 7f72ecbfd640 1 -- 172.21.15.70:0/1283321320 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e810cb80 con 0x7f72e8100fc0 2024-04-01T01:49:32.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.466+0000 7f72d77fe640 1 -- 172.21.15.70:0/1283321320 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f72e001b850 con 0x7f72e8100fc0 2024-04-01T01:49:32.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.466+0000 7f72ef7db740 1 Processor -- start 2024-04-01T01:49:32.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.467+0000 7f72ef7db740 1 -- start start 2024-04-01T01:49:32.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.467+0000 7f72ef7db740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8131a10 0x55e5d8131e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.467+0000 7f72ef7db740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8132400 0x55e5d8136f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.467+0000 7f72ef7db740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8137570 0x55e5d8139a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.467+0000 7f72ef7db740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e5d80330b0 con 0x55e5d8132400 2024-04-01T01:49:32.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.467+0000 7f72ef7db740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e5d8095ea0 con 0x55e5d8131a10 2024-04-01T01:49:32.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.467+0000 7f72ef7db740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e5d803ec00 con 0x55e5d8137570 2024-04-01T01:49:32.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.467+0000 7f72e6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8131a10 0x55e5d8131e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.467+0000 7f72e6d9d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8131a10 0x55e5d8131e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49772/0 (socket says 172.21.15.70:49772) 2024-04-01T01:49:32.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.467+0000 7f72e6d9d640 1 -- 172.21.15.70:0/28335116 learned_addr learned my addr 172.21.15.70:0/28335116 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:32.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.467+0000 7f72e5d9b640 1 --2- 172.21.15.70:0/28335116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8137570 0x55e5d8139a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.467+0000 7f72e659c640 1 --2- 172.21.15.70:0/28335116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8132400 0x55e5d8136f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.468+0000 7f72e6d9d640 1 -- 172.21.15.70:0/28335116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8137570 msgr2=0x55e5d8139a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:32.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.468+0000 7f72e6d9d640 1 --2- 172.21.15.70:0/28335116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8137570 0x55e5d8139a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.468+0000 7f72e6d9d640 1 -- 172.21.15.70:0/28335116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8132400 msgr2=0x55e5d8136f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:32.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.468+0000 7f72e6d9d640 1 --2- 172.21.15.70:0/28335116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8132400 0x55e5d8136f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.468+0000 7f72e6d9d640 1 -- 172.21.15.70:0/28335116 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5d7e727e0 con 0x55e5d8131a10 2024-04-01T01:49:32.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.468+0000 7f72d77fe640 1 --2- 172.21.15.70:0/1283321320 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72d00a2670 0x7f72d00a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.468+0000 7f72e6d9d640 1 --2- 172.21.15.70:0/28335116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8131a10 0x55e5d8131e30 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009780 tx=0x7f72d8005d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcc34646fd6c7aee server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:32.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.468+0000 7f72d77fe640 1 -- 172.21.15.70:0/1283321320 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f72e00d7cb0 con 0x7f72e8100fc0 2024-04-01T01:49:32.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.468+0000 7f72e659c640 1 --2- 172.21.15.70:0/1283321320 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72d00a2670 0x7f72d00a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.468+0000 7f72d4ff9640 1 -- 172.21.15.70:0/28335116 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d8039070 con 0x55e5d8131a10 2024-04-01T01:49:32.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.468+0000 7f72d4ff9640 1 -- 172.21.15.70:0/28335116 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72d803a040 con 0x55e5d8131a10 2024-04-01T01:49:32.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.468+0000 7f72d4ff9640 1 -- 172.21.15.70:0/28335116 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d8005440 con 0x55e5d8131a10 2024-04-01T01:49:32.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.469+0000 7f72ef7db740 1 -- 172.21.15.70:0/28335116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8131a10 msgr2=0x55e5d8131e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.469+0000 7f72e659c640 1 --2- 172.21.15.70:0/1283321320 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72d00a2670 0x7f72d00a4b30 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x55e5d8132800 tx=0x7f72dc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:32.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.469+0000 7f72ef7db740 1 --2- 172.21.15.70:0/28335116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8131a10 0x55e5d8131e30 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009780 tx=0x7f72d8005d40 comp rx=0 tx=0).stop 2024-04-01T01:49:32.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.469+0000 7f72ef7db740 1 -- 172.21.15.70:0/28335116 shutdown_connections 2024-04-01T01:49:32.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.469+0000 7f72ef7db740 1 --2- 172.21.15.70:0/28335116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8137570 0x55e5d8139a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.469+0000 7f72ef7db740 1 --2- 172.21.15.70:0/28335116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8132400 0x55e5d8136f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.469+0000 7f72ef7db740 1 --2- 172.21.15.70:0/28335116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8131a10 0x55e5d8131e30 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.469+0000 7f72ef7db740 1 -- 172.21.15.70:0/28335116 >> 172.21.15.70:0/28335116 conn(0x55e5d812cf80 msgr2=0x55e5d812d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:32.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.469+0000 7f72ef7db740 1 -- 172.21.15.70:0/28335116 shutdown_connections 2024-04-01T01:49:32.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.469+0000 7f72ef7db740 1 -- 172.21.15.70:0/28335116 wait complete. 2024-04-01T01:49:32.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.470+0000 7f72ef7db740 1 -- start start 2024-04-01T01:49:32.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.470+0000 7f72ef7db740 1 Processor -- start 2024-04-01T01:49:32.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.470+0000 7f72ef7db740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8131a10 0x55e5d813be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.470+0000 7f72ef7db740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8132400 0x55e5d813c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72ef7db740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8137570 0x55e5d8143390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72e659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8131a10 0x55e5d813be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72e659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8131a10 0x55e5d813be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49272/0 (socket says 172.21.15.70:49272) 2024-04-01T01:49:32.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72ef7db740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e5d80343d0 con 0x55e5d8131a10 2024-04-01T01:49:32.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72e659c640 1 -- 172.21.15.70:0/1452959594 learned_addr learned my addr 172.21.15.70:0/1452959594 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:32.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72ef7db740 1 -- 172.21.15.70:0/1452959594 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55e5d8091f40 con 0x55e5d8137570 2024-04-01T01:49:32.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72ef7db740 1 -- 172.21.15.70:0/1452959594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e5d813a4c0 con 0x55e5d8132400 2024-04-01T01:49:32.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72e6d9d640 1 --2- 172.21.15.70:0/1452959594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8137570 0x55e5d8143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72ef7db740 10 client.0 _ll_register_callbacks cb 0x55e5d8051b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:32.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72e5d9b640 1 --2- 172.21.15.70:0/1452959594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8132400 0x55e5d813c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72e659c640 1 -- 172.21.15.70:0/1452959594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8132400 msgr2=0x55e5d813c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:32.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72e659c640 1 --2- 172.21.15.70:0/1452959594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8132400 0x55e5d813c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72e659c640 1 -- 172.21.15.70:0/1452959594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8137570 msgr2=0x55e5d8143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:32.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72e659c640 1 --2- 172.21.15.70:0/1452959594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8137570 0x55e5d8143390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72e659c640 1 -- 172.21.15.70:0/1452959594 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5d7ed2f70 con 0x55e5d8131a10 2024-04-01T01:49:32.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72e6d9d640 1 --2- 172.21.15.70:0/1452959594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8137570 0x55e5d8143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:49:32.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.471+0000 7f72ef7db740 1 -- 172.21.15.70:0/1452959594 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72d803a040 con 0x55e5d8131a10 2024-04-01T01:49:32.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.472+0000 7f72e659c640 1 --2- 172.21.15.70:0/1452959594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8131a10 0x55e5d813be10 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f72dc01d820 tx=0x7f72dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c7b366d28a0557 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:32.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.476+0000 7f72cdffb640 10 client.20124 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:49:32.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.476+0000 7f72cdffb640 1 -- 172.21.15.70:0/1452959594 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc02a060 con 0x55e5d8131a10 2024-04-01T01:49:32.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.476+0000 7f72cdffb640 1 -- 172.21.15.70:0/1452959594 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f72dc010900 con 0x55e5d8131a10 2024-04-01T01:49:32.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.476+0000 7f72cdffb640 1 -- 172.21.15.70:0/1452959594 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f72dc010b60 con 0x55e5d8131a10 2024-04-01T01:49:32.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.476+0000 7f72ef7db740 10 client.20124 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:32.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.476+0000 7f72ef7db740 1 -- 172.21.15.70:0/1452959594 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({fsmap=33}) -- 0x55e5d7d6a170 con 0x55e5d8131a10 2024-04-01T01:49:32.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.479+0000 7f72cdffb640 1 -- 172.21.15.70:0/1452959594 <== mon.0 v2:172.21.15.70:3300/0 4 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f72dc019040 con 0x55e5d8131a10 2024-04-01T01:49:32.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.479+0000 7f72ef7db740 10 client.20124 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:32.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.480+0000 7f72ef7db740 10 client.20124 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:49:32.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.480+0000 7f72ef7db740 20 client.20124 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:32.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.480+0000 7f72ef7db740 1 --2- 172.21.15.70:0/1452959594 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e5d8147720 0x55e5d8149b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.480+0000 7f72ef7db740 4 client.20124 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:49:32.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.480+0000 7f72ef7db740 1 -- 172.21.15.70:0/1452959594 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5d80a33e0 con 0x55e5d8147720 2024-04-01T01:49:32.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.481+0000 7f72e5d9b640 1 --2- 172.21.15.70:0/1452959594 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e5d8147720 0x55e5d8149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.482+0000 7f72e5d9b640 1 --2- 172.21.15.70:0/1452959594 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e5d8147720 0x55e5d8149b90 crc :-1 s=READY pgs=400 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e28b4f40c5ed35c server_cookie=54129f53d4eb671e in_seq=0 out_seq=0 2024-04-01T01:49:32.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.482+0000 7f72cdffb640 10 client.20124 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:32.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.483+0000 7f72cdffb640 1 -- 172.21.15.70:0/1452959594 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e5d80a33e0 con 0x55e5d8147720 2024-04-01T01:49:32.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.483+0000 7f72cdffb640 10 client.20124 handle_command_reply: tid=0 2024-04-01T01:49:32.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.483+0000 7f72ef7db740 1 client.20124 shutdown 2024-04-01T01:49:32.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.484+0000 7f72ef7db740 1 -- 172.21.15.70:0/1452959594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8131a10 msgr2=0x55e5d813be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.484+0000 7f72ef7db740 1 --2- 172.21.15.70:0/1452959594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8131a10 0x55e5d813be10 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f72dc01d820 tx=0x7f72dc0046c0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.485+0000 7f72ef7db740 1 -- 172.21.15.70:0/1452959594 shutdown_connections 2024-04-01T01:49:32.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.485+0000 7f72ef7db740 1 --2- 172.21.15.70:0/1452959594 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e5d8147720 0x55e5d8149b90 crc :-1 s=READY pgs=400 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.485+0000 7f72ef7db740 1 --2- 172.21.15.70:0/1452959594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8137570 0x55e5d8143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.485+0000 7f72ef7db740 1 --2- 172.21.15.70:0/1452959594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8132400 0x55e5d813c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.485+0000 7f72ef7db740 1 --2- 172.21.15.70:0/1452959594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8131a10 0x55e5d813be10 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.485+0000 7f72ef7db740 1 -- 172.21.15.70:0/1452959594 >> 172.21.15.70:0/1452959594 conn(0x55e5d812cf80 msgr2=0x55e5d812c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:32.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.485+0000 7f72cdffb640 0 client.20124 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:32.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.486+0000 7f72ef7db740 1 -- 172.21.15.70:0/1452959594 shutdown_connections 2024-04-01T01:49:32.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.492+0000 7f72ef7db740 1 -- 172.21.15.70:0/1452959594 wait complete. 2024-04-01T01:49:32.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.494+0000 7f72ef7db740 20 client.20124 trim_cache size 0 max 16384 2024-04-01T01:49:32.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.530+0000 7f72ef7db740 1 Processor -- start 2024-04-01T01:49:32.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.531+0000 7f72ef7db740 1 -- start start 2024-04-01T01:49:32.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.531+0000 7f72ef7db740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8131a10 0x55e5d813c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.531+0000 7f72ef7db740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8132400 0x55e5d8143350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.531+0000 7f72e659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8131a10 0x55e5d813c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.531+0000 7f72e659c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8131a10 0x55e5d813c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49788/0 (socket says 172.21.15.70:49788) 2024-04-01T01:49:32.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.532+0000 7f72e659c640 1 -- 172.21.15.70:0/228594031 learned_addr learned my addr 172.21.15.70:0/228594031 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:32.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.532+0000 7f72ef7db740 1 --2- 172.21.15.70:0/228594031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8137570 0x55e5d812b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.532+0000 7f72e5d9b640 1 --2- 172.21.15.70:0/228594031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8132400 0x55e5d8143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.532+0000 7f72ef7db740 1 -- 172.21.15.70:0/228594031 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f72e81a81d0 con 0x55e5d8137570 2024-04-01T01:49:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.532+0000 7f72ef7db740 1 -- 172.21.15.70:0/228594031 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f72e81a9d10 con 0x55e5d8131a10 2024-04-01T01:49:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.532+0000 7f72ef7db740 1 -- 172.21.15.70:0/228594031 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f72e81a88a0 con 0x55e5d8132400 2024-04-01T01:49:32.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.533+0000 7f72e6d9d640 1 --2- 172.21.15.70:0/228594031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8137570 0x55e5d812b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.533+0000 7f72e659c640 1 -- 172.21.15.70:0/228594031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8132400 msgr2=0x55e5d8143350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.533+0000 7f72e659c640 1 --2- 172.21.15.70:0/228594031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8132400 0x55e5d8143350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.533+0000 7f72e659c640 1 -- 172.21.15.70:0/228594031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8137570 msgr2=0x55e5d812b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.533+0000 7f72e659c640 1 --2- 172.21.15.70:0/228594031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8137570 0x55e5d812b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.533+0000 7f72e659c640 1 -- 172.21.15.70:0/228594031 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5d8136c70 con 0x55e5d8131a10 2024-04-01T01:49:32.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.533+0000 7f72e659c640 1 --2- 172.21.15.70:0/228594031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8131a10 0x55e5d813c130 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f72dc0058c0 tx=0x7f72dc01f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9605e6f06317823 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:32.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.534+0000 7f72cffff640 1 -- 172.21.15.70:0/228594031 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc01d230 con 0x55e5d8131a10 2024-04-01T01:49:32.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.534+0000 7f72cffff640 1 -- 172.21.15.70:0/228594031 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72d803a040 con 0x55e5d8131a10 2024-04-01T01:49:32.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.534+0000 7f72cffff640 1 -- 172.21.15.70:0/228594031 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc004af0 con 0x55e5d8131a10 2024-04-01T01:49:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.535+0000 7f72ef7db740 1 -- 172.21.15.70:0/228594031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8131a10 msgr2=0x55e5d813c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.535+0000 7f72ef7db740 1 --2- 172.21.15.70:0/228594031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8131a10 0x55e5d813c130 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f72dc0058c0 tx=0x7f72dc01f040 comp rx=0 tx=0).stop 2024-04-01T01:49:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.535+0000 7f72ef7db740 1 -- 172.21.15.70:0/228594031 shutdown_connections 2024-04-01T01:49:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.535+0000 7f72ef7db740 1 --2- 172.21.15.70:0/228594031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8137570 0x55e5d812b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.536+0000 7f72ef7db740 1 --2- 172.21.15.70:0/228594031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8132400 0x55e5d8143350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.536+0000 7f72ef7db740 1 --2- 172.21.15.70:0/228594031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8131a10 0x55e5d813c130 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.536+0000 7f72ef7db740 1 -- 172.21.15.70:0/228594031 >> 172.21.15.70:0/228594031 conn(0x55e5d812cf80 msgr2=0x55e5d813ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:32.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.536+0000 7f72ef7db740 1 -- 172.21.15.70:0/228594031 shutdown_connections 2024-04-01T01:49:32.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.536+0000 7f72ef7db740 1 -- 172.21.15.70:0/228594031 wait complete. 2024-04-01T01:49:32.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.537+0000 7f72ef7db740 1 -- start start 2024-04-01T01:49:32.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.537+0000 7f72ef7db740 1 Processor -- start 2024-04-01T01:49:32.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.537+0000 7f72ef7db740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8131a10 0x55e5d8140bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.537+0000 7f72ef7db740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8132400 0x55e5d8141110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.537+0000 7f72ef7db740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8137570 0x55e5d8142080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.537+0000 7f72ef7db740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f72e81a9640 con 0x55e5d8137570 2024-04-01T01:49:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.537+0000 7f72ef7db740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f72e81a8f70 con 0x55e5d8132400 2024-04-01T01:49:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.537+0000 7f72ef7db740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f72e81a61b0 con 0x55e5d8131a10 2024-04-01T01:49:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.537+0000 7f72ef7db740 10 client.0 _ll_register_callbacks cb 0x7f72dc01d4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.538+0000 7f72e5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8132400 0x55e5d8141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.538+0000 7f72e5d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8132400 0x55e5d8141110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49790/0 (socket says 172.21.15.70:49790) 2024-04-01T01:49:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.538+0000 7f72e5d9b640 1 -- 172.21.15.70:0/4154104771 learned_addr learned my addr 172.21.15.70:0/4154104771 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.538+0000 7f72e6d9d640 1 --2- 172.21.15.70:0/4154104771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8137570 0x55e5d8142080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.538+0000 7f72e659c640 1 --2- 172.21.15.70:0/4154104771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8131a10 0x55e5d8140bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.538+0000 7f72e5d9b640 1 -- 172.21.15.70:0/4154104771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8131a10 msgr2=0x55e5d8140bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.538+0000 7f72e5d9b640 1 --2- 172.21.15.70:0/4154104771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8131a10 0x55e5d8140bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.538+0000 7f72e5d9b640 1 -- 172.21.15.70:0/4154104771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8137570 msgr2=0x55e5d8142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.538+0000 7f72e5d9b640 1 --2- 172.21.15.70:0/4154104771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8137570 0x55e5d8142080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.538+0000 7f72e5d9b640 1 -- 172.21.15.70:0/4154104771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5d8142670 con 0x55e5d8132400 2024-04-01T01:49:32.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.538+0000 7f72e5d9b640 1 --2- 172.21.15.70:0/4154104771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8132400 0x55e5d8141110 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f72e000bc30 tx=0x7f72e001e430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=736902de98d46e71 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:32.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.539+0000 7f72cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:49:32.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.539+0000 7f72cdffb640 1 -- 172.21.15.70:0/4154104771 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e001e4c0 con 0x55e5d8132400 2024-04-01T01:49:32.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.539+0000 7f72cdffb640 1 -- 172.21.15.70:0/4154104771 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f72e00a8170 con 0x55e5d8132400 2024-04-01T01:49:32.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.539+0000 7f72ef7db740 1 -- 172.21.15.70:0/4154104771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72d803a040 con 0x55e5d8132400 2024-04-01T01:49:32.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.539+0000 7f72cdffb640 1 -- 172.21.15.70:0/4154104771 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f72e00a4a40 con 0x55e5d8132400 2024-04-01T01:49:32.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.539+0000 7f72ef7db740 10 client.30227 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:32.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.539+0000 7f72ef7db740 1 -- 172.21.15.70:0/4154104771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e5d81adc20 con 0x55e5d8132400 2024-04-01T01:49:32.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.540+0000 7f72cdffb640 1 -- 172.21.15.70:0/4154104771 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72d803a040 con 0x55e5d8132400 2024-04-01T01:49:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.540+0000 7f72cdffb640 1 -- 172.21.15.70:0/4154104771 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f72e000b040 con 0x55e5d8132400 2024-04-01T01:49:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.540+0000 7f72ef7db740 10 client.30227 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.540+0000 7f72ef7db740 10 client.30227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:49:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.540+0000 7f72ef7db740 20 client.30227 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.540+0000 7f72ef7db740 1 --2- 172.21.15.70:0/4154104771 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e5d81ae510 0x55e5d81b0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.540+0000 7f72ef7db740 4 client.30227 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-01T01:49:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.540+0000 7f72ef7db740 1 -- 172.21.15.70:0/4154104771 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e5d809d730 con 0x55e5d81ae510 2024-04-01T01:49:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.540+0000 7f72e659c640 1 --2- 172.21.15.70:0/4154104771 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e5d81ae510 0x55e5d81b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.541+0000 7f72cdffb640 10 client.30227 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:32.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.541+0000 7f72e659c640 1 --2- 172.21.15.70:0/4154104771 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e5d81ae510 0x55e5d81b0900 crc :-1 s=READY pgs=401 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3ebb178bdf54fc9 server_cookie=28baaa93cb3f7636 in_seq=0 out_seq=0 2024-04-01T01:49:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.542+0000 7f72cdffb640 1 -- 172.21.15.70:0/4154104771 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e5d809d730 con 0x55e5d81ae510 2024-04-01T01:49:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.542+0000 7f72cdffb640 10 client.30227 handle_command_reply: tid=0 2024-04-01T01:49:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.542+0000 7f72ef7db740 1 client.30227 shutdown 2024-04-01T01:49:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.542+0000 7f72ef7db740 1 -- 172.21.15.70:0/4154104771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8132400 msgr2=0x55e5d8141110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.543 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.543+0000 7f72ef7db740 1 --2- 172.21.15.70:0/4154104771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8132400 0x55e5d8141110 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f72e000bc30 tx=0x7f72e001e430 comp rx=0 tx=0).stop 2024-04-01T01:49:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.543+0000 7f72ef7db740 1 -- 172.21.15.70:0/4154104771 shutdown_connections 2024-04-01T01:49:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.543+0000 7f72ef7db740 1 --2- 172.21.15.70:0/4154104771 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55e5d81ae510 0x55e5d81b0900 crc :-1 s=READY pgs=401 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.543+0000 7f72ef7db740 1 --2- 172.21.15.70:0/4154104771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e5d8137570 0x55e5d8142080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.543+0000 7f72ef7db740 1 --2- 172.21.15.70:0/4154104771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e5d8132400 0x55e5d8141110 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.543+0000 7f72ef7db740 1 --2- 172.21.15.70:0/4154104771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5d8131a10 0x55e5d8140bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.543+0000 7f72ef7db740 1 -- 172.21.15.70:0/4154104771 >> 172.21.15.70:0/4154104771 conn(0x55e5d812cf80 msgr2=0x55e5d81301e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:32.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.543+0000 7f72cdffb640 0 client.30227 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:32.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.543+0000 7f72ef7db740 1 -- 172.21.15.70:0/4154104771 shutdown_connections 2024-04-01T01:49:32.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.543+0000 7f72ef7db740 1 -- 172.21.15.70:0/4154104771 wait complete. 2024-04-01T01:49:32.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.544+0000 7f72ef7db740 20 client.30227 trim_cache size 0 max 16384 2024-04-01T01:49:32.545 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:49:32.545 INFO:teuthology.orchestra.run.smithi070.stdout: "return_code": 0, 2024-04-01T01:49:32.545 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_tag": "e2921bc4-396e-4958-b226-9a64e1ee273d", 2024-04-01T01:49:32.545 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": "asynchronous" 2024-04-01T01:49:32.545 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:49:32.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.544+0000 7f72ecbfd640 1 -- 172.21.15.70:0/1283321320 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72d00a2670 msgr2=0x7f72d00a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.544+0000 7f72ecbfd640 1 --2- 172.21.15.70:0/1283321320 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72d00a2670 0x7f72d00a4b30 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x55e5d8132800 tx=0x7f72dc009260 comp rx=0 tx=0).stop 2024-04-01T01:49:32.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.544+0000 7f72ecbfd640 1 -- 172.21.15.70:0/1283321320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72e8100fc0 msgr2=0x7f72e819a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.544+0000 7f72ecbfd640 1 --2- 172.21.15.70:0/1283321320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72e8100fc0 0x7f72e819a060 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f72e000de60 tx=0x7f72e000b730 comp rx=0 tx=0).stop 2024-04-01T01:49:32.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.544+0000 7f72ecbfd640 1 -- 172.21.15.70:0/1283321320 shutdown_connections 2024-04-01T01:49:32.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.544+0000 7f72ecbfd640 1 --2- 172.21.15.70:0/1283321320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72e8101980 0x7f72e81a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.544+0000 7f72ecbfd640 1 --2- 172.21.15.70:0/1283321320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f72e8100fc0 0x7f72e819a060 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.544+0000 7f72ecbfd640 1 --2- 172.21.15.70:0/1283321320 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72d00a2670 0x7f72d00a4b30 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.544+0000 7f72ecbfd640 1 --2- 172.21.15.70:0/1283321320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72e80ffdd0 0x7f72e8199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.545+0000 7f72ecbfd640 1 -- 172.21.15.70:0/1283321320 >> 172.21.15.70:0/1283321320 conn(0x7f72e80fb340 msgr2=0x7f72e80fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:32.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.545+0000 7f72ecbfd640 1 -- 172.21.15.70:0/1283321320 shutdown_connections 2024-04-01T01:49:32.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.545+0000 7f72ecbfd640 1 -- 172.21.15.70:0/1283321320 wait complete. 2024-04-01T01:49:32.564 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:49:32.564 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:49:32.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.749+0000 7f39569c1640 1 -- 172.21.15.70:0/1189243662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3948099910 msgr2=0x7f3948099d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.749+0000 7f39569c1640 1 --2- 172.21.15.70:0/1189243662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3948099910 0x7f3948099d90 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3940009900 tx=0x7f394002bc80 comp rx=0 tx=0).stop 2024-04-01T01:49:32.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.750+0000 7f39569c1640 1 -- 172.21.15.70:0/1189243662 shutdown_connections 2024-04-01T01:49:32.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.750+0000 7f39569c1640 1 --2- 172.21.15.70:0/1189243662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3948099910 0x7f3948099d90 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.750+0000 7f39569c1640 1 --2- 172.21.15.70:0/1189243662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3948096f60 0x7f39480993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.750+0000 7f39569c1640 1 --2- 172.21.15.70:0/1189243662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39480945f0 0x7f3948096a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.750+0000 7f39569c1640 1 -- 172.21.15.70:0/1189243662 >> 172.21.15.70:0/1189243662 conn(0x7f39480901d0 msgr2=0x7f3948092630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:32.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.750+0000 7f39569c1640 1 -- 172.21.15.70:0/1189243662 shutdown_connections 2024-04-01T01:49:32.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.751+0000 7f39569c1640 1 -- 172.21.15.70:0/1189243662 wait complete. 2024-04-01T01:49:32.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.751+0000 7f39569c1640 1 Processor -- start 2024-04-01T01:49:32.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.752+0000 7f39569c1640 1 -- start start 2024-04-01T01:49:32.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.752+0000 7f39569c1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39480945f0 0x7f39481326b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.752+0000 7f39569c1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3948096f60 0x7f3948132bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.752+0000 7f39569c1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3948099910 0x7f3948139c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.752+0000 7f39569c1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f39480a7fb0 con 0x7f39480945f0 2024-04-01T01:49:32.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.752+0000 7f39569c1640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f39480a7e30 con 0x7f3948096f60 2024-04-01T01:49:32.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.752+0000 7f39569c1640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f39480a8130 con 0x7f3948099910 2024-04-01T01:49:32.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.752+0000 7f39559bf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39480945f0 0x7f39481326b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.752+0000 7f39559bf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39480945f0 0x7f39481326b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49312/0 (socket says 172.21.15.70:49312) 2024-04-01T01:49:32.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.752+0000 7f39559bf640 1 -- 172.21.15.70:0/128486881 learned_addr learned my addr 172.21.15.70:0/128486881 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:32.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.752+0000 7f39551be640 1 --2- 172.21.15.70:0/128486881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3948096f60 0x7f3948132bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.752+0000 7f39561c0640 1 --2- 172.21.15.70:0/128486881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3948099910 0x7f3948139c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.753+0000 7f39551be640 1 -- 172.21.15.70:0/128486881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3948099910 msgr2=0x7f3948139c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.753+0000 7f39551be640 1 --2- 172.21.15.70:0/128486881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3948099910 0x7f3948139c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.753+0000 7f39551be640 1 -- 172.21.15.70:0/128486881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39480945f0 msgr2=0x7f39481326b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.753+0000 7f39551be640 1 --2- 172.21.15.70:0/128486881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39480945f0 0x7f39481326b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.753+0000 7f39551be640 1 -- 172.21.15.70:0/128486881 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f394814b2e0 con 0x7f3948096f60 2024-04-01T01:49:32.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.753+0000 7f39559bf640 1 --2- 172.21.15.70:0/128486881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39480945f0 0x7f39481326b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:49:32.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.753+0000 7f39551be640 1 --2- 172.21.15.70:0/128486881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3948096f60 0x7f3948132bf0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f394c009fd0 tx=0x7f394c00b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:32.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.754+0000 7f3946ffd640 1 -- 172.21.15.70:0/128486881 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c015020 con 0x7f3948096f60 2024-04-01T01:49:32.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.754+0000 7f39569c1640 1 -- 172.21.15.70:0/128486881 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f394814b5d0 con 0x7f3948096f60 2024-04-01T01:49:32.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.754+0000 7f3946ffd640 1 -- 172.21.15.70:0/128486881 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f394c011ce0 con 0x7f3948096f60 2024-04-01T01:49:32.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.754+0000 7f3946ffd640 1 -- 172.21.15.70:0/128486881 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c010dd0 con 0x7f3948096f60 2024-04-01T01:49:32.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.754+0000 7f39569c1640 1 -- 172.21.15.70:0/128486881 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f394814bba0 con 0x7f3948096f60 2024-04-01T01:49:32.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.758+0000 7f395a808740 1 Processor -- start 2024-04-01T01:49:32.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.758+0000 7f395a808740 1 -- start start 2024-04-01T01:49:32.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.758+0000 7f395a808740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7f490 0x55888da7f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.758+0000 7f395a808740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7fe80 0x55888da84980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.758+0000 7f395a808740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da84ff0 0x55888da87480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.759+0000 7f395a808740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55888d9e4180 con 0x55888da7fe80 2024-04-01T01:49:32.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.759+0000 7f395a808740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55888d98cb70 con 0x55888da7f490 2024-04-01T01:49:32.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.759+0000 7f395a808740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55888d9e0220 con 0x55888da84ff0 2024-04-01T01:49:32.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.759+0000 7f3946ffd640 1 -- 172.21.15.70:0/128486881 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f394c018020 con 0x7f3948096f60 2024-04-01T01:49:32.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.759+0000 7f39559bf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7fe80 0x55888da84980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.759+0000 7f39561c0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7f490 0x55888da7f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.759+0000 7f39559bf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7fe80 0x55888da84980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49318/0 (socket says 172.21.15.70:49318) 2024-04-01T01:49:32.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.759+0000 7f39559bf640 1 -- 172.21.15.70:0/3507227173 learned_addr learned my addr 172.21.15.70:0/3507227173 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:32.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.760+0000 7f39559bf640 1 -- 172.21.15.70:0/3507227173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da84ff0 msgr2=0x55888da87480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-01T01:49:32.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.760+0000 7f39559bf640 1 --2- 172.21.15.70:0/3507227173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da84ff0 0x55888da87480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.760+0000 7f39559bf640 1 -- 172.21.15.70:0/3507227173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7f490 msgr2=0x55888da7f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:32.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.760+0000 7f39559bf640 1 --2- 172.21.15.70:0/3507227173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7f490 0x55888da7f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.760+0000 7f39559bf640 1 -- 172.21.15.70:0/3507227173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55888d9a07c0 con 0x55888da7fe80 2024-04-01T01:49:32.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.760+0000 7f3946ffd640 1 --2- 172.21.15.70:0/128486881 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f393c0a25f0 0x7f393c0a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.760+0000 7f3946ffd640 1 -- 172.21.15.70:0/128486881 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f394c0ce1d0 con 0x7f3948096f60 2024-04-01T01:49:32.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.760+0000 7f39559bf640 1 --2- 172.21.15.70:0/3507227173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7fe80 0x55888da84980 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f395004fbf0 tx=0x7f39500504d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e8b5233df64369 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:32.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.760+0000 7f39561c0640 1 --2- 172.21.15.70:0/128486881 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f393c0a25f0 0x7f393c0a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.761+0000 7f39561c0640 1 --2- 172.21.15.70:0/128486881 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f393c0a25f0 0x7f393c0a4ab0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3940006010 tx=0x7f394002c990 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:32.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.768+0000 7f393bfff640 1 -- 172.21.15.70:0/3507227173 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395007c020 con 0x55888da7fe80 2024-04-01T01:49:32.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.768+0000 7f393bfff640 1 -- 172.21.15.70:0/3507227173 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3950072e40 con 0x55888da7fe80 2024-04-01T01:49:32.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.768+0000 7f393bfff640 1 -- 172.21.15.70:0/3507227173 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3950077d90 con 0x55888da7fe80 2024-04-01T01:49:32.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.769+0000 7f395a808740 1 -- 172.21.15.70:0/3507227173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7fe80 msgr2=0x55888da84980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.769+0000 7f395a808740 1 --2- 172.21.15.70:0/3507227173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7fe80 0x55888da84980 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f395004fbf0 tx=0x7f39500504d0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.770+0000 7f395a808740 1 -- 172.21.15.70:0/3507227173 shutdown_connections 2024-04-01T01:49:32.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.770+0000 7f395a808740 1 --2- 172.21.15.70:0/3507227173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da84ff0 0x55888da87480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.770+0000 7f395a808740 1 --2- 172.21.15.70:0/3507227173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7fe80 0x55888da84980 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.770+0000 7f395a808740 1 --2- 172.21.15.70:0/3507227173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7f490 0x55888da7f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.770+0000 7f395a808740 1 -- 172.21.15.70:0/3507227173 >> 172.21.15.70:0/3507227173 conn(0x55888da7aa00 msgr2=0x55888da7ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:32.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.770+0000 7f395a808740 1 -- 172.21.15.70:0/3507227173 shutdown_connections 2024-04-01T01:49:32.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.770+0000 7f395a808740 1 -- 172.21.15.70:0/3507227173 wait complete. 2024-04-01T01:49:32.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.771+0000 7f395a808740 1 -- start start 2024-04-01T01:49:32.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.771+0000 7f395a808740 1 Processor -- start 2024-04-01T01:49:32.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.771+0000 7f395a808740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7f490 0x55888da899a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.771+0000 7f395a808740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7fe80 0x55888da89ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.772+0000 7f395a808740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da84ff0 0x55888da90f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.772+0000 7f395a808740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55888da87fa0 con 0x55888da7f490 2024-04-01T01:49:32.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.772+0000 7f395a808740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55888d9824a0 con 0x55888da7fe80 2024-04-01T01:49:32.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.772+0000 7f395a808740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55888da88120 con 0x55888da84ff0 2024-04-01T01:49:32.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.772+0000 7f395a808740 10 client.0 _ll_register_callbacks cb 0x55888d985d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:32.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.772+0000 7f39551be640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7fe80 0x55888da89ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.774+0000 7f39561c0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da84ff0 0x55888da90f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.775+0000 7f39551be640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7fe80 0x55888da89ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49804/0 (socket says 172.21.15.70:49804) 2024-04-01T01:49:32.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.775+0000 7f39559bf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7f490 0x55888da899a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.775+0000 7f39551be640 1 -- 172.21.15.70:0/1165858664 learned_addr learned my addr 172.21.15.70:0/1165858664 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:32.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.776+0000 7f39559bf640 1 -- 172.21.15.70:0/1165858664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da84ff0 msgr2=0x55888da90f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:32.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.776+0000 7f39559bf640 1 --2- 172.21.15.70:0/1165858664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da84ff0 0x55888da90f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.776+0000 7f39559bf640 1 -- 172.21.15.70:0/1165858664 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7fe80 msgr2=0x55888da89ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:32.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.776+0000 7f39559bf640 1 --2- 172.21.15.70:0/1165858664 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7fe80 0x55888da89ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.776+0000 7f39559bf640 1 -- 172.21.15.70:0/1165858664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55888d7c27d0 con 0x55888da7f490 2024-04-01T01:49:32.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.777+0000 7f39559bf640 1 --2- 172.21.15.70:0/1165858664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7f490 0x55888da899a0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x55888da7b970 tx=0x7f3950072bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9dce716841ebff7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:32.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.777+0000 7f39397fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:49:32.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.779+0000 7f39397fa640 1 -- 172.21.15.70:0/1165858664 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3950090060 con 0x55888da7f490 2024-04-01T01:49:32.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.779+0000 7f39397fa640 1 -- 172.21.15.70:0/1165858664 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f3950077cd0 con 0x55888da7f490 2024-04-01T01:49:32.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.779+0000 7f395a808740 1 -- 172.21.15.70:0/1165858664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3950072e40 con 0x55888da7f490 2024-04-01T01:49:32.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.779+0000 7f39397fa640 1 -- 172.21.15.70:0/1165858664 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3950097050 con 0x55888da7f490 2024-04-01T01:49:32.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.779+0000 7f395a808740 10 client.20139 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:32.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.779+0000 7f395a808740 1 -- 172.21.15.70:0/1165858664 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55888d6f83f0 con 0x55888da7f490 2024-04-01T01:49:32.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.780+0000 7f39397fa640 1 -- 172.21.15.70:0/1165858664 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3950072e40 con 0x55888da7f490 2024-04-01T01:49:32.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.780+0000 7f39397fa640 1 -- 172.21.15.70:0/1165858664 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f3950081420 con 0x55888da7f490 2024-04-01T01:49:32.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.780+0000 7f395a808740 10 client.20139 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:32.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.780+0000 7f395a808740 10 client.20139 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:49:32.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.780+0000 7f395a808740 20 client.20139 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:32.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.780+0000 7f395a808740 1 --2- 172.21.15.70:0/1165858664 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55888da95410 0x55888da97800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.780+0000 7f395a808740 4 client.20139 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:49:32.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.780+0000 7f395a808740 1 -- 172.21.15.70:0/1165858664 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55888d9f16c0 con 0x55888da95410 2024-04-01T01:49:32.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.781+0000 7f39551be640 1 --2- 172.21.15.70:0/1165858664 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55888da95410 0x55888da97800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.783+0000 7f39551be640 1 --2- 172.21.15.70:0/1165858664 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55888da95410 0x55888da97800 crc :-1 s=READY pgs=402 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f891c81a738d3844 server_cookie=2fc44fae4386083d in_seq=0 out_seq=0 2024-04-01T01:49:32.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.783+0000 7f39397fa640 10 client.20139 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:32.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.785+0000 7f39397fa640 1 -- 172.21.15.70:0/1165858664 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55888d9f16c0 con 0x55888da95410 2024-04-01T01:49:32.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.785+0000 7f39397fa640 10 client.20139 handle_command_reply: tid=0 2024-04-01T01:49:32.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.786+0000 7f395a808740 1 client.20139 shutdown 2024-04-01T01:49:32.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.790+0000 7f395a808740 1 -- 172.21.15.70:0/1165858664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7f490 msgr2=0x55888da899a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.790+0000 7f395a808740 1 --2- 172.21.15.70:0/1165858664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7f490 0x55888da899a0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x55888da7b970 tx=0x7f3950072bc0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.791+0000 7f395a808740 1 -- 172.21.15.70:0/1165858664 shutdown_connections 2024-04-01T01:49:32.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.791+0000 7f395a808740 1 --2- 172.21.15.70:0/1165858664 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55888da95410 0x55888da97800 crc :-1 s=READY pgs=402 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.791+0000 7f395a808740 1 --2- 172.21.15.70:0/1165858664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da84ff0 0x55888da90f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.791+0000 7f395a808740 1 --2- 172.21.15.70:0/1165858664 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7fe80 0x55888da89ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.791+0000 7f395a808740 1 --2- 172.21.15.70:0/1165858664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7f490 0x55888da899a0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.791+0000 7f395a808740 1 -- 172.21.15.70:0/1165858664 >> 172.21.15.70:0/1165858664 conn(0x55888da7aa00 msgr2=0x55888da79c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:32.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.791+0000 7f39397fa640 0 client.20139 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:32.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.792+0000 7f395a808740 1 -- 172.21.15.70:0/1165858664 shutdown_connections 2024-04-01T01:49:32.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.793+0000 7f395a808740 1 -- 172.21.15.70:0/1165858664 wait complete. 2024-04-01T01:49:32.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.795+0000 7f395a808740 20 client.20139 trim_cache size 0 max 16384 2024-04-01T01:49:32.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.829+0000 7f395a808740 1 Processor -- start 2024-04-01T01:49:32.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.829+0000 7f395a808740 1 -- start start 2024-04-01T01:49:32.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.829+0000 7f395a808740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7f490 0x55888da79a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.830+0000 7f395a808740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7fe80 0x55888da90ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.830+0000 7f395a808740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da84ff0 0x55888da81340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.830+0000 7f395a808740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f393400fdf0 con 0x55888da7f490 2024-04-01T01:49:32.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.830+0000 7f395a808740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3934011930 con 0x55888da7fe80 2024-04-01T01:49:32.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.830+0000 7f395a808740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f39340104c0 con 0x55888da84ff0 2024-04-01T01:49:32.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.830+0000 7f39551be640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7fe80 0x55888da90ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.830+0000 7f39551be640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7fe80 0x55888da90ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49810/0 (socket says 172.21.15.70:49810) 2024-04-01T01:49:32.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.830+0000 7f39551be640 1 -- 172.21.15.70:0/3532917876 learned_addr learned my addr 172.21.15.70:0/3532917876 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:32.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.830+0000 7f39561c0640 1 --2- 172.21.15.70:0/3532917876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da84ff0 0x55888da81340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.830+0000 7f39551be640 1 -- 172.21.15.70:0/3532917876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da84ff0 msgr2=0x55888da81340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:32.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.830+0000 7f39551be640 1 --2- 172.21.15.70:0/3532917876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da84ff0 0x55888da81340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.830+0000 7f39551be640 1 -- 172.21.15.70:0/3532917876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7f490 msgr2=0x55888da79a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:32.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.830+0000 7f39551be640 1 --2- 172.21.15.70:0/3532917876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7f490 0x55888da79a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.830+0000 7f39551be640 1 -- 172.21.15.70:0/3532917876 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55888d6b80f0 con 0x55888da7fe80 2024-04-01T01:49:32.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.831+0000 7f39551be640 1 --2- 172.21.15.70:0/3532917876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7fe80 0x55888da90ee0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f394c096c70 tx=0x7f394c085040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e0c6a6d2a0d3d0a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:32.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.831+0000 7f393b7fe640 1 -- 172.21.15.70:0/3532917876 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c09a860 con 0x55888da7fe80 2024-04-01T01:49:32.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.831+0000 7f393b7fe640 1 -- 172.21.15.70:0/3532917876 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f394c09e9b0 con 0x55888da7fe80 2024-04-01T01:49:32.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.831+0000 7f393b7fe640 1 -- 172.21.15.70:0/3532917876 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c0850b0 con 0x55888da7fe80 2024-04-01T01:49:32.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.832+0000 7f395a808740 1 -- 172.21.15.70:0/3532917876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7fe80 msgr2=0x55888da90ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.832+0000 7f395a808740 1 --2- 172.21.15.70:0/3532917876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7fe80 0x55888da90ee0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f394c096c70 tx=0x7f394c085040 comp rx=0 tx=0).stop 2024-04-01T01:49:32.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.832+0000 7f395a808740 1 -- 172.21.15.70:0/3532917876 shutdown_connections 2024-04-01T01:49:32.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.832+0000 7f395a808740 1 --2- 172.21.15.70:0/3532917876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da84ff0 0x55888da81340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.832+0000 7f395a808740 1 --2- 172.21.15.70:0/3532917876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da7fe80 0x55888da90ee0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.832+0000 7f395a808740 1 --2- 172.21.15.70:0/3532917876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7f490 0x55888da79a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.832+0000 7f395a808740 1 -- 172.21.15.70:0/3532917876 >> 172.21.15.70:0/3532917876 conn(0x55888da7aa00 msgr2=0x55888da7e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:32.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.834+0000 7f395a808740 1 -- 172.21.15.70:0/3532917876 shutdown_connections 2024-04-01T01:49:32.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.834+0000 7f395a808740 1 -- 172.21.15.70:0/3532917876 wait complete. 2024-04-01T01:49:32.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.834+0000 7f395a808740 1 -- start start 2024-04-01T01:49:32.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.834+0000 7f395a808740 1 Processor -- start 2024-04-01T01:49:32.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.835+0000 7f395a808740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da7f490 0x55888da90930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.835+0000 7f395a808740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7fe80 0x55888da90e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.835+0000 7f395a808740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da84ff0 0x55888da8efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.835+0000 7f395a808740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3934011260 con 0x55888da7fe80 2024-04-01T01:49:32.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.835+0000 7f395a808740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3934010b90 con 0x55888da84ff0 2024-04-01T01:49:32.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.835+0000 7f395a808740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f393400ddd0 con 0x55888da7f490 2024-04-01T01:49:32.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.835+0000 7f395a808740 10 client.0 _ll_register_callbacks cb 0x7f395006f6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:32.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.835+0000 7f39561c0640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da84ff0 0x55888da8efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.835+0000 7f39559bf640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da7f490 0x55888da90930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.835+0000 7f39551be640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7fe80 0x55888da90e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.835+0000 7f39559bf640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da7f490 0x55888da90930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:51754/0 (socket says 172.21.15.70:51754) 2024-04-01T01:49:32.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.835+0000 7f39559bf640 1 -- 172.21.15.70:0/1014389257 learned_addr learned my addr 172.21.15.70:0/1014389257 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:32.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.836+0000 7f39561c0640 1 -- 172.21.15.70:0/1014389257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da7f490 msgr2=0x55888da90930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.836+0000 7f39561c0640 1 --2- 172.21.15.70:0/1014389257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da7f490 0x55888da90930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.836+0000 7f39561c0640 1 -- 172.21.15.70:0/1014389257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7fe80 msgr2=0x55888da90e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.836+0000 7f39561c0640 1 --2- 172.21.15.70:0/1014389257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7fe80 0x55888da90e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.836+0000 7f39561c0640 1 -- 172.21.15.70:0/1014389257 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55888da8f7f0 con 0x55888da84ff0 2024-04-01T01:49:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.836+0000 7f39559bf640 1 --2- 172.21.15.70:0/1014389257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da7f490 0x55888da90930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:49:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.836+0000 7f39561c0640 1 --2- 172.21.15.70:0/1014389257 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da84ff0 0x55888da8efc0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f3940009850 tx=0x7f394002bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=812df123d9af2ecf server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.837+0000 7f39397fa640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:49:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.837+0000 7f39397fa640 1 -- 172.21.15.70:0/1014389257 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39400055c0 con 0x55888da84ff0 2024-04-01T01:49:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.837+0000 7f39397fa640 1 -- 172.21.15.70:0/1014389257 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f39400046c0 con 0x55888da84ff0 2024-04-01T01:49:32.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.837+0000 7f395a808740 1 -- 172.21.15.70:0/1014389257 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f394c09e9b0 con 0x55888da84ff0 2024-04-01T01:49:32.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.838+0000 7f39397fa640 1 -- 172.21.15.70:0/1014389257 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f39400048c0 con 0x55888da84ff0 2024-04-01T01:49:32.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.839+0000 7f395a808740 10 client.30242 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:32.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.839+0000 7f395a808740 1 -- 172.21.15.70:0/1014389257 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55888dafc260 con 0x55888da84ff0 2024-04-01T01:49:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.843+0000 7f39397fa640 1 -- 172.21.15.70:0/1014389257 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f394c09e9b0 con 0x55888da84ff0 2024-04-01T01:49:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.843+0000 7f39397fa640 1 -- 172.21.15.70:0/1014389257 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f394004d440 con 0x55888da84ff0 2024-04-01T01:49:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.843+0000 7f395a808740 10 client.30242 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.843+0000 7f395a808740 10 client.30242 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:49:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.843+0000 7f395a808740 20 client.30242 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.843+0000 7f395a808740 1 --2- 172.21.15.70:0/1014389257 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55888dafcc10 0x55888daff030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:32.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.843+0000 7f395a808740 4 client.30242 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:49:32.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.843+0000 7f395a808740 1 -- 172.21.15.70:0/1014389257 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55888d9eba10 con 0x55888dafcc10 2024-04-01T01:49:32.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.846+0000 7f39559bf640 1 --2- 172.21.15.70:0/1014389257 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55888dafcc10 0x55888daff030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:32.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.846+0000 7f39559bf640 1 --2- 172.21.15.70:0/1014389257 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55888dafcc10 0x55888daff030 crc :-1 s=READY pgs=403 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dbaf5c463cf4a82 server_cookie=4d7dbcafbde7af50 in_seq=0 out_seq=0 2024-04-01T01:49:32.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.847+0000 7f39397fa640 10 client.30242 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:32.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.848+0000 7f39397fa640 1 -- 172.21.15.70:0/1014389257 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55888d9eba10 con 0x55888dafcc10 2024-04-01T01:49:32.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.848+0000 7f39397fa640 10 client.30242 handle_command_reply: tid=0 2024-04-01T01:49:32.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.848+0000 7f395a808740 1 client.30242 shutdown 2024-04-01T01:49:32.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.849+0000 7f395a808740 1 -- 172.21.15.70:0/1014389257 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da84ff0 msgr2=0x55888da8efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.849+0000 7f395a808740 1 --2- 172.21.15.70:0/1014389257 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da84ff0 0x55888da8efc0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f3940009850 tx=0x7f394002bc80 comp rx=0 tx=0).stop 2024-04-01T01:49:32.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.849+0000 7f395a808740 1 -- 172.21.15.70:0/1014389257 shutdown_connections 2024-04-01T01:49:32.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.849+0000 7f395a808740 1 --2- 172.21.15.70:0/1014389257 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x55888dafcc10 0x55888daff030 crc :-1 s=READY pgs=403 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.849+0000 7f395a808740 1 --2- 172.21.15.70:0/1014389257 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55888da84ff0 0x55888da8efc0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.849+0000 7f395a808740 1 --2- 172.21.15.70:0/1014389257 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55888da7fe80 0x55888da90e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.849+0000 7f395a808740 1 --2- 172.21.15.70:0/1014389257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55888da7f490 0x55888da90930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.849+0000 7f39397fa640 0 client.30242 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:32.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.849+0000 7f395a808740 1 -- 172.21.15.70:0/1014389257 >> 172.21.15.70:0/1014389257 conn(0x55888da7aa00 msgr2=0x55888da88fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:32.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.849+0000 7f395a808740 1 -- 172.21.15.70:0/1014389257 shutdown_connections 2024-04-01T01:49:32.850 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.849+0000 7f395a808740 1 -- 172.21.15.70:0/1014389257 wait complete. 2024-04-01T01:49:32.851 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.850+0000 7f395a808740 20 client.30242 trim_cache size 0 max 16384 2024-04-01T01:49:32.853 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:49:32.854 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-01T01:49:32.854 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": { 2024-04-01T01:49:32.854 INFO:teuthology.orchestra.run.smithi070.stdout: "e2921bc4-396e-4958-b226-9a64e1ee273d": { 2024-04-01T01:49:32.854 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/", 2024-04-01T01:49:32.854 INFO:teuthology.orchestra.run.smithi070.stdout: "tag": "e2921bc4-396e-4958-b226-9a64e1ee273d", 2024-04-01T01:49:32.854 INFO:teuthology.orchestra.run.smithi070.stdout: "options": "recursive,force" 2024-04-01T01:49:32.854 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:49:32.854 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:49:32.854 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:49:32.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.854+0000 7f39569c1640 1 -- 172.21.15.70:0/128486881 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f393c0a25f0 msgr2=0x7f393c0a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.854+0000 7f39569c1640 1 --2- 172.21.15.70:0/128486881 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f393c0a25f0 0x7f393c0a4ab0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3940006010 tx=0x7f394002c990 comp rx=0 tx=0).stop 2024-04-01T01:49:32.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.854+0000 7f39569c1640 1 -- 172.21.15.70:0/128486881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3948096f60 msgr2=0x7f3948132bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:32.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.854+0000 7f39569c1640 1 --2- 172.21.15.70:0/128486881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3948096f60 0x7f3948132bf0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f394c009fd0 tx=0x7f394c00b870 comp rx=0 tx=0).stop 2024-04-01T01:49:32.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.854+0000 7f39569c1640 1 -- 172.21.15.70:0/128486881 shutdown_connections 2024-04-01T01:49:32.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.854+0000 7f39569c1640 1 --2- 172.21.15.70:0/128486881 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f393c0a25f0 0x7f393c0a4ab0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.854+0000 7f39569c1640 1 --2- 172.21.15.70:0/128486881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3948099910 0x7f3948139c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.854+0000 7f39569c1640 1 --2- 172.21.15.70:0/128486881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3948096f60 0x7f3948132bf0 secure :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f394c009fd0 tx=0x7f394c00b870 comp rx=0 tx=0).stop 2024-04-01T01:49:32.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.854+0000 7f39569c1640 1 --2- 172.21.15.70:0/128486881 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f39480945f0 0x7f39481326b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:32.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.854+0000 7f39569c1640 1 -- 172.21.15.70:0/128486881 >> 172.21.15.70:0/128486881 conn(0x7f39480901d0 msgr2=0x7f3948095380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:32.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.855+0000 7f39569c1640 1 -- 172.21.15.70:0/128486881 shutdown_connections 2024-04-01T01:49:32.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:49:32.856+0000 7f39569c1640 1 -- 172.21.15.70:0/128486881 wait complete. 2024-04-01T01:49:32.882 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:32 smithi070 ceph-mon[32179]: pgmap v841: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 823 op/s 2024-04-01T01:49:32.883 INFO:tasks.cephfs.filesystem:scrub status for tag:e2921bc4-396e-4958-b226-9a64e1ee273d - {'path': '/', 'tag': 'e2921bc4-396e-4958-b226-9a64e1ee273d', 'options': 'recursive,force'} 2024-04-01T01:49:33.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:32 smithi081 ceph-mon[35142]: pgmap v841: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 823 op/s 2024-04-01T01:49:33.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:32 smithi189 ceph-mon[35753]: pgmap v841: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 823 op/s 2024-04-01T01:49:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:33 smithi081 ceph-mon[35142]: scrub queued for path: / 2024-04-01T01:49:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:33 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:49:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:33 smithi081 ceph-mon[35142]: scrub summary: active paths [/] 2024-04-01T01:49:34.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:33 smithi081 ceph-mon[35142]: scrub summary: idle+waiting paths [/] 2024-04-01T01:49:34.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:33 smithi189 ceph-mon[35753]: scrub queued for path: / 2024-04-01T01:49:34.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:33 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:49:34.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:33 smithi189 ceph-mon[35753]: scrub summary: active paths [/] 2024-04-01T01:49:34.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:33 smithi189 ceph-mon[35753]: scrub summary: idle+waiting paths [/] 2024-04-01T01:49:34.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:33 smithi070 ceph-mon[32179]: scrub queued for path: / 2024-04-01T01:49:34.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:33 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:49:34.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:33 smithi070 ceph-mon[32179]: scrub summary: active paths [/] 2024-04-01T01:49:34.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:33 smithi070 ceph-mon[32179]: scrub summary: idle+waiting paths [/] 2024-04-01T01:49:35.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:34 smithi081 ceph-mon[35142]: pgmap v842: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 11 MiB/s wr, 779 op/s 2024-04-01T01:49:35.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:34 smithi189 ceph-mon[35753]: pgmap v842: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 11 MiB/s wr, 779 op/s 2024-04-01T01:49:35.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:34 smithi070 ceph-mon[32179]: pgmap v842: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 11 MiB/s wr, 779 op/s 2024-04-01T01:49:36.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 890.0 s, 850.0 tps, lat 117.044 ms stddev 91.315 2024-04-01T01:49:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:36 smithi081 ceph-mon[35142]: pgmap v843: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 11 MiB/s wr, 785 op/s 2024-04-01T01:49:37.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:36 smithi081 ceph-mon[35142]: scrub summary: idle 2024-04-01T01:49:37.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:36 smithi189 ceph-mon[35753]: pgmap v843: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 11 MiB/s wr, 785 op/s 2024-04-01T01:49:37.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:36 smithi189 ceph-mon[35753]: scrub summary: idle 2024-04-01T01:49:37.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:36 smithi070 ceph-mon[32179]: pgmap v843: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 11 MiB/s wr, 785 op/s 2024-04-01T01:49:37.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:36 smithi070 ceph-mon[32179]: scrub summary: idle 2024-04-01T01:49:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.848+0000 7f8e75387640 1 -- 172.21.15.70:0/1852792967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e680a6b80 msgr2=0x7f8e680ad410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:37.849 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.848+0000 7f8e75387640 1 --2- 172.21.15.70:0/1852792967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e680a6b80 0x7f8e680ad410 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f8e70066e90 tx=0x7f8e7008ff00 comp rx=0 tx=0).stop 2024-04-01T01:49:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.848+0000 7f8e75387640 1 -- 172.21.15.70:0/1852792967 shutdown_connections 2024-04-01T01:49:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.848+0000 7f8e75387640 1 --2- 172.21.15.70:0/1852792967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e680a6b80 0x7f8e680ad410 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.848+0000 7f8e75387640 1 --2- 172.21.15.70:0/1852792967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e680a61c0 0x7f8e680a6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.848+0000 7f8e75387640 1 --2- 172.21.15.70:0/1852792967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e680a4fd0 0x7f8e680a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.848+0000 7f8e75387640 1 -- 172.21.15.70:0/1852792967 >> 172.21.15.70:0/1852792967 conn(0x7f8e680a00d0 msgr2=0x7f8e680a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.848+0000 7f8e75387640 1 -- 172.21.15.70:0/1852792967 shutdown_connections 2024-04-01T01:49:37.850 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.849+0000 7f8e75387640 1 -- 172.21.15.70:0/1852792967 wait complete. 2024-04-01T01:49:37.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.849+0000 7f8e75387640 1 Processor -- start 2024-04-01T01:49:37.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.849+0000 7f8e75387640 1 -- start start 2024-04-01T01:49:37.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.849+0000 7f8e75387640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e680a4fd0 0x7f8e6813ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.849+0000 7f8e75387640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e680a61c0 0x7f8e6813b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.849+0000 7f8e75387640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e680a6b80 0x7f8e68142310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.852 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.850+0000 7f8e6f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e680a61c0 0x7f8e6813b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.850+0000 7f8e6f7fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e680a61c0 0x7f8e6813b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60878/0 (socket says 172.21.15.70:60878) 2024-04-01T01:49:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.850+0000 7f8e75387640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e680b06b0 con 0x7f8e680a6b80 2024-04-01T01:49:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.850+0000 7f8e75387640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8e680b0530 con 0x7f8e680a61c0 2024-04-01T01:49:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.850+0000 7f8e75387640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8e680b0830 con 0x7f8e680a4fd0 2024-04-01T01:49:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.850+0000 7f8e6f7fe640 1 -- 172.21.15.70:0/1076940106 learned_addr learned my addr 172.21.15.70:0/1076940106 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.850+0000 7f8e74b86640 1 --2- 172.21.15.70:0/1076940106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e680a6b80 0x7f8e68142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.850+0000 7f8e6ffff640 1 --2- 172.21.15.70:0/1076940106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e680a4fd0 0x7f8e6813ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.850+0000 7f8e6f7fe640 1 -- 172.21.15.70:0/1076940106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e680a4fd0 msgr2=0x7f8e6813ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.850+0000 7f8e6f7fe640 1 --2- 172.21.15.70:0/1076940106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e680a4fd0 0x7f8e6813ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.850+0000 7f8e6f7fe640 1 -- 172.21.15.70:0/1076940106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e680a6b80 msgr2=0x7f8e68142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.850+0000 7f8e6f7fe640 1 --2- 172.21.15.70:0/1076940106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e680a6b80 0x7f8e68142310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.853 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.850+0000 7f8e6f7fe640 1 -- 172.21.15.70:0/1076940106 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e68144bc0 con 0x7f8e680a61c0 2024-04-01T01:49:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.850+0000 7f8e6f7fe640 1 --2- 172.21.15.70:0/1076940106 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e680a61c0 0x7f8e6813b2e0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f8e6400b450 tx=0x7f8e6400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.851+0000 7f8e6d7fa640 1 -- 172.21.15.70:0/1076940106 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e64015020 con 0x7f8e680a61c0 2024-04-01T01:49:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.851+0000 7f8e6d7fa640 1 -- 172.21.15.70:0/1076940106 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e64011d40 con 0x7f8e680a61c0 2024-04-01T01:49:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.851+0000 7f8e6d7fa640 1 -- 172.21.15.70:0/1076940106 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6400f340 con 0x7f8e680a61c0 2024-04-01T01:49:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.851+0000 7f8e75387640 1 -- 172.21.15.70:0/1076940106 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e68142910 con 0x7f8e680a61c0 2024-04-01T01:49:37.854 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.851+0000 7f8e75387640 1 -- 172.21.15.70:0/1076940106 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e68142ee0 con 0x7f8e680a61c0 2024-04-01T01:49:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.855+0000 7f8e791cf740 1 Processor -- start 2024-04-01T01:49:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.855+0000 7f8e791cf740 1 -- start start 2024-04-01T01:49:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.855+0000 7f8e791cf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64c9ae0 0x5604d64c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.855+0000 7f8e791cf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64ca4d0 0x5604d64cefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.856 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.855+0000 7f8e791cf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64cf640 0x5604d64d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.855+0000 7f8e6d7fa640 1 -- 172.21.15.70:0/1076940106 <== mon.2 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f8e64018020 con 0x7f8e680a61c0 2024-04-01T01:49:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.855+0000 7f8e791cf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5604d642d8f0 con 0x5604d64c9ae0 2024-04-01T01:49:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.855+0000 7f8e791cf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5604d63d68d0 con 0x5604d64cf640 2024-04-01T01:49:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.855+0000 7f8e791cf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5604d6429990 con 0x5604d64ca4d0 2024-04-01T01:49:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.855+0000 7f8e6ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64ca4d0 0x5604d64cefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.855+0000 7f8e74b86640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64c9ae0 0x5604d64c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.856+0000 7f8e74b86640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64c9ae0 0x5604d64c9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33886/0 (socket says 172.21.15.70:33886) 2024-04-01T01:49:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.856+0000 7f8e74b86640 1 -- 172.21.15.70:0/1259713553 learned_addr learned my addr 172.21.15.70:0/1259713553 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.856+0000 7f8e6f7fe640 1 --2- 172.21.15.70:0/1259713553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64cf640 0x5604d64d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.856+0000 7f8e6ffff640 1 -- 172.21.15.70:0/1259713553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64cf640 msgr2=0x5604d64d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.856+0000 7f8e6ffff640 1 --2- 172.21.15.70:0/1259713553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64cf640 0x5604d64d1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.856+0000 7f8e6ffff640 1 -- 172.21.15.70:0/1259713553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64c9ae0 msgr2=0x5604d64c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.856+0000 7f8e6d7fa640 1 --2- 172.21.15.70:0/1076940106 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e600a2520 0x7f8e600a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.856+0000 7f8e6ffff640 1 --2- 172.21.15.70:0/1259713553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64c9ae0 0x5604d64c9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.856+0000 7f8e6d7fa640 1 -- 172.21.15.70:0/1076940106 <== mon.2 v2:172.21.15.81:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f8e640ce590 con 0x7f8e680a61c0 2024-04-01T01:49:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.856+0000 7f8e6ffff640 1 -- 172.21.15.70:0/1259713553 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604d6142210 con 0x5604d64ca4d0 2024-04-01T01:49:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.856+0000 7f8e6f7fe640 1 --2- 172.21.15.70:0/1076940106 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e600a2520 0x7f8e600a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.857+0000 7f8e6ffff640 1 --2- 172.21.15.70:0/1259713553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64ca4d0 0x5604d64cefd0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8e680a6020 tx=0x7f8e5c0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=824b9d5f5b656ba7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.857+0000 7f8e6f7fe640 1 --2- 172.21.15.70:0/1076940106 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e600a2520 0x7f8e600a49e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8e64097050 tx=0x7f8e64006df0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.857+0000 7f8e5a7fc640 1 -- 172.21.15.70:0/1259713553 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c004020 con 0x5604d64ca4d0 2024-04-01T01:49:37.859 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.858+0000 7f8e5a7fc640 1 -- 172.21.15.70:0/1259713553 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e5c0041c0 con 0x5604d64ca4d0 2024-04-01T01:49:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.858+0000 7f8e5a7fc640 1 -- 172.21.15.70:0/1259713553 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c0116a0 con 0x5604d64ca4d0 2024-04-01T01:49:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.858+0000 7f8e791cf740 1 -- 172.21.15.70:0/1259713553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64ca4d0 msgr2=0x5604d64cefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.858+0000 7f8e791cf740 1 --2- 172.21.15.70:0/1259713553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64ca4d0 0x5604d64cefd0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8e680a6020 tx=0x7f8e5c0107a0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.858+0000 7f8e791cf740 1 -- 172.21.15.70:0/1259713553 shutdown_connections 2024-04-01T01:49:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.858+0000 7f8e791cf740 1 --2- 172.21.15.70:0/1259713553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64cf640 0x5604d64d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.858+0000 7f8e791cf740 1 --2- 172.21.15.70:0/1259713553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64ca4d0 0x5604d64cefd0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.858+0000 7f8e791cf740 1 --2- 172.21.15.70:0/1259713553 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64c9ae0 0x5604d64c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.858+0000 7f8e791cf740 1 -- 172.21.15.70:0/1259713553 >> 172.21.15.70:0/1259713553 conn(0x5604d64c5050 msgr2=0x5604d64c5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.858+0000 7f8e791cf740 1 -- 172.21.15.70:0/1259713553 shutdown_connections 2024-04-01T01:49:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.858+0000 7f8e791cf740 1 -- 172.21.15.70:0/1259713553 wait complete. 2024-04-01T01:49:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.859+0000 7f8e791cf740 1 -- start start 2024-04-01T01:49:37.860 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.859+0000 7f8e791cf740 1 Processor -- start 2024-04-01T01:49:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.859+0000 7f8e791cf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64c9ae0 0x5604d64d3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.859+0000 7f8e791cf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64ca4d0 0x5604d64d4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.859+0000 7f8e791cf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64cf640 0x5604d64db390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.859+0000 7f8e791cf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5604d64d25f0 con 0x5604d64ca4d0 2024-04-01T01:49:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.859+0000 7f8e791cf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5604d63cbf80 con 0x5604d64c9ae0 2024-04-01T01:49:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.859+0000 7f8e791cf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5604d64d2770 con 0x5604d64cf640 2024-04-01T01:49:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.859+0000 7f8e791cf740 10 client.0 _ll_register_callbacks cb 0x5604d6424290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.859+0000 7f8e6ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64ca4d0 0x5604d64d4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.862 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.859+0000 7f8e6ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64ca4d0 0x5604d64d4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33888/0 (socket says 172.21.15.70:33888) 2024-04-01T01:49:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.859+0000 7f8e6ffff640 1 -- 172.21.15.70:0/1667212948 learned_addr learned my addr 172.21.15.70:0/1667212948 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.860+0000 7f8e74b86640 1 --2- 172.21.15.70:0/1667212948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64c9ae0 0x5604d64d3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.860+0000 7f8e74b86640 1 --2- 172.21.15.70:0/1667212948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64cf640 0x5604d64db390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.861+0000 7f8e74b86640 1 -- 172.21.15.70:0/1667212948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64cf640 msgr2=0x5604d64db390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.861+0000 7f8e74b86640 1 --2- 172.21.15.70:0/1667212948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64cf640 0x5604d64db390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.861+0000 7f8e74b86640 1 -- 172.21.15.70:0/1667212948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64ca4d0 msgr2=0x5604d64d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.861+0000 7f8e74b86640 1 --2- 172.21.15.70:0/1667212948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64ca4d0 0x5604d64d4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.861+0000 7f8e74b86640 1 -- 172.21.15.70:0/1667212948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604d620e620 con 0x5604d64c9ae0 2024-04-01T01:49:37.863 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.861+0000 7f8e6ffff640 1 --2- 172.21.15.70:0/1667212948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64ca4d0 0x5604d64d4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:49:37.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.862+0000 7f8e4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-01T01:49:37.864 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.862+0000 7f8e74b86640 1 --2- 172.21.15.70:0/1667212948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64c9ae0 0x5604d64d3e10 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f8e70090510 tx=0x7f8e70067870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2910f00989f001f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.862+0000 7f8e4f7fe640 1 -- 172.21.15.70:0/1667212948 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e70098610 con 0x5604d64c9ae0 2024-04-01T01:49:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.862+0000 7f8e791cf740 1 -- 172.21.15.70:0/1667212948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e5c0041c0 con 0x5604d64c9ae0 2024-04-01T01:49:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.862+0000 7f8e4f7fe640 1 -- 172.21.15.70:0/1667212948 <== mon.2 v2:172.21.15.81:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f8e70098410 con 0x5604d64c9ae0 2024-04-01T01:49:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.863+0000 7f8e4f7fe640 1 -- 172.21.15.70:0/1667212948 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8e70098610 con 0x5604d64c9ae0 2024-04-01T01:49:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.863+0000 7f8e791cf740 10 client.30254 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.863+0000 7f8e791cf740 1 -- 172.21.15.70:0/1667212948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5604d64de430 con 0x5604d64c9ae0 2024-04-01T01:49:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.864+0000 7f8e4f7fe640 1 -- 172.21.15.70:0/1667212948 <== mon.2 v2:172.21.15.81:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e5c0041c0 con 0x5604d64c9ae0 2024-04-01T01:49:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.864+0000 7f8e4f7fe640 1 -- 172.21.15.70:0/1667212948 <== mon.2 v2:172.21.15.81:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f8e700a85b0 con 0x5604d64c9ae0 2024-04-01T01:49:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.864+0000 7f8e791cf740 10 client.30254 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.864+0000 7f8e791cf740 10 client.30254 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:49:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.864+0000 7f8e791cf740 20 client.30254 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.864+0000 7f8e791cf740 1 --2- 172.21.15.70:0/1667212948 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5604d64dfe30 0x5604d64e22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.864+0000 7f8e791cf740 4 client.30254 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:49:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.864+0000 7f8e791cf740 1 -- 172.21.15.70:0/1667212948 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604d6441120 con 0x5604d64dfe30 2024-04-01T01:49:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.864+0000 7f8e6ffff640 1 --2- 172.21.15.70:0/1667212948 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5604d64dfe30 0x5604d64e22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.865+0000 7f8e6ffff640 1 --2- 172.21.15.70:0/1667212948 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5604d64dfe30 0x5604d64e22c0 crc :-1 s=READY pgs=404 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ef5aa10a90d92d server_cookie=57cc34605fd5e60d in_seq=0 out_seq=0 2024-04-01T01:49:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.865+0000 7f8e4f7fe640 10 client.30254 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.866+0000 7f8e4f7fe640 1 -- 172.21.15.70:0/1667212948 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5604d6441120 con 0x5604d64dfe30 2024-04-01T01:49:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.866+0000 7f8e4f7fe640 10 client.30254 handle_command_reply: tid=0 2024-04-01T01:49:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.866+0000 7f8e791cf740 1 client.30254 shutdown 2024-04-01T01:49:37.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.866+0000 7f8e791cf740 1 -- 172.21.15.70:0/1667212948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64c9ae0 msgr2=0x5604d64d3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:37.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.866+0000 7f8e791cf740 1 --2- 172.21.15.70:0/1667212948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64c9ae0 0x5604d64d3e10 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f8e70090510 tx=0x7f8e70067870 comp rx=0 tx=0).stop 2024-04-01T01:49:37.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.866+0000 7f8e791cf740 1 -- 172.21.15.70:0/1667212948 shutdown_connections 2024-04-01T01:49:37.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.866+0000 7f8e791cf740 1 --2- 172.21.15.70:0/1667212948 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5604d64dfe30 0x5604d64e22c0 crc :-1 s=READY pgs=404 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.866+0000 7f8e791cf740 1 --2- 172.21.15.70:0/1667212948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64cf640 0x5604d64db390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.866+0000 7f8e4f7fe640 0 client.30254 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:37.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.866+0000 7f8e791cf740 1 --2- 172.21.15.70:0/1667212948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64ca4d0 0x5604d64d4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.866+0000 7f8e791cf740 1 --2- 172.21.15.70:0/1667212948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64c9ae0 0x5604d64d3e10 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.866+0000 7f8e791cf740 1 -- 172.21.15.70:0/1667212948 >> 172.21.15.70:0/1667212948 conn(0x5604d64c5050 msgr2=0x5604d64c4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:37.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.866+0000 7f8e791cf740 1 -- 172.21.15.70:0/1667212948 shutdown_connections 2024-04-01T01:49:37.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.867+0000 7f8e791cf740 1 -- 172.21.15.70:0/1667212948 wait complete. 2024-04-01T01:49:37.868 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.867+0000 7f8e791cf740 20 client.30254 trim_cache size 0 max 16384 2024-04-01T01:49:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.890+0000 7f8e791cf740 1 Processor -- start 2024-04-01T01:49:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.890+0000 7f8e791cf740 1 -- start start 2024-04-01T01:49:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.890+0000 7f8e791cf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64c9ae0 0x5604d64c82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.891+0000 7f8e791cf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64ca4d0 0x5604d64c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.891+0000 7f8e791cf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64cf640 0x5604d64c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.891+0000 7f8e791cf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e68151eb0 con 0x5604d64cf640 2024-04-01T01:49:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.891+0000 7f8e791cf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8e681539f0 con 0x5604d64ca4d0 2024-04-01T01:49:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.891+0000 7f8e791cf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8e68152580 con 0x5604d64c9ae0 2024-04-01T01:49:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.891+0000 7f8e74b86640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64c9ae0 0x5604d64c82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.891+0000 7f8e6ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64ca4d0 0x5604d64c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.891+0000 7f8e6ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64ca4d0 0x5604d64c87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60908/0 (socket says 172.21.15.70:60908) 2024-04-01T01:49:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.891+0000 7f8e74b86640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64c9ae0 0x5604d64c82b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46412/0 (socket says 172.21.15.70:46412) 2024-04-01T01:49:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.891+0000 7f8e6ffff640 1 -- 172.21.15.70:0/2497086830 learned_addr learned my addr 172.21.15.70:0/2497086830 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.891+0000 7f8e74b86640 1 --2- 172.21.15.70:0/2497086830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64cf640 0x5604d64c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.891+0000 7f8e6ffff640 1 -- 172.21.15.70:0/2497086830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64c9ae0 msgr2=0x5604d64c82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.891+0000 7f8e6ffff640 1 --2- 172.21.15.70:0/2497086830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64c9ae0 0x5604d64c82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.891+0000 7f8e6ffff640 1 -- 172.21.15.70:0/2497086830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64cf640 msgr2=0x5604d64c3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.891+0000 7f8e6ffff640 1 --2- 172.21.15.70:0/2497086830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64cf640 0x5604d64c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.891+0000 7f8e6ffff640 1 -- 172.21.15.70:0/2497086830 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604d64ca8b0 con 0x5604d64ca4d0 2024-04-01T01:49:37.894 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.894+0000 7f8e6ffff640 1 --2- 172.21.15.70:0/2497086830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64ca4d0 0x5604d64c87f0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0058c0 tx=0x7f8e5c013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6061cb83676a87f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.895+0000 7f8e59ffb640 1 -- 172.21.15.70:0/2497086830 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c005450 con 0x5604d64ca4d0 2024-04-01T01:49:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.897+0000 7f8e59ffb640 1 -- 172.21.15.70:0/2497086830 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e5c030070 con 0x5604d64ca4d0 2024-04-01T01:49:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.897+0000 7f8e59ffb640 1 -- 172.21.15.70:0/2497086830 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c0039e0 con 0x5604d64ca4d0 2024-04-01T01:49:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.900+0000 7f8e791cf740 1 -- 172.21.15.70:0/2497086830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64ca4d0 msgr2=0x5604d64c87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.900+0000 7f8e791cf740 1 --2- 172.21.15.70:0/2497086830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64ca4d0 0x5604d64c87f0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0058c0 tx=0x7f8e5c013040 comp rx=0 tx=0).stop 2024-04-01T01:49:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.902+0000 7f8e791cf740 1 -- 172.21.15.70:0/2497086830 shutdown_connections 2024-04-01T01:49:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.902+0000 7f8e791cf740 1 --2- 172.21.15.70:0/2497086830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64cf640 0x5604d64c3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.902+0000 7f8e791cf740 1 --2- 172.21.15.70:0/2497086830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64ca4d0 0x5604d64c87f0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.902+0000 7f8e791cf740 1 --2- 172.21.15.70:0/2497086830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64c9ae0 0x5604d64c82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.902+0000 7f8e791cf740 1 -- 172.21.15.70:0/2497086830 >> 172.21.15.70:0/2497086830 conn(0x5604d64c5050 msgr2=0x5604d64c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.903+0000 7f8e791cf740 1 -- 172.21.15.70:0/2497086830 shutdown_connections 2024-04-01T01:49:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.903+0000 7f8e791cf740 1 -- 172.21.15.70:0/2497086830 wait complete. 2024-04-01T01:49:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.904+0000 7f8e791cf740 1 -- start start 2024-04-01T01:49:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.904+0000 7f8e791cf740 1 Processor -- start 2024-04-01T01:49:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.904+0000 7f8e791cf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64c9ae0 0x5604d64e7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.905+0000 7f8e791cf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64ca4d0 0x5604d64e75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.905+0000 7f8e791cf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64cf640 0x5604d6544210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.905+0000 7f8e791cf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e68153320 con 0x5604d64c9ae0 2024-04-01T01:49:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.905+0000 7f8e791cf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8e68152c50 con 0x5604d64cf640 2024-04-01T01:49:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.905+0000 7f8e791cf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8e6814fe30 con 0x5604d64ca4d0 2024-04-01T01:49:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.905+0000 7f8e791cf740 10 client.0 _ll_register_callbacks cb 0x7f8e7004fb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.905+0000 7f8e74b86640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64c9ae0 0x5604d64e7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.905+0000 7f8e6ffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64ca4d0 0x5604d64e75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.905+0000 7f8e74b86640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64c9ae0 0x5604d64e7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33904/0 (socket says 172.21.15.70:33904) 2024-04-01T01:49:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.905+0000 7f8e74b86640 1 -- 172.21.15.70:0/924414217 learned_addr learned my addr 172.21.15.70:0/924414217 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.905+0000 7f8e74b86640 1 --2- 172.21.15.70:0/924414217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64cf640 0x5604d6544210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.905+0000 7f8e6ffff640 1 -- 172.21.15.70:0/924414217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64cf640 msgr2=0x5604d6544210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.905+0000 7f8e6ffff640 1 --2- 172.21.15.70:0/924414217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64cf640 0x5604d6544210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.906+0000 7f8e6ffff640 1 -- 172.21.15.70:0/924414217 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64c9ae0 msgr2=0x5604d64e7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.906+0000 7f8e6ffff640 1 --2- 172.21.15.70:0/924414217 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64c9ae0 0x5604d64e7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.906+0000 7f8e6ffff640 1 -- 172.21.15.70:0/924414217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604d6544820 con 0x5604d64ca4d0 2024-04-01T01:49:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.906+0000 7f8e6ffff640 1 --2- 172.21.15.70:0/924414217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64ca4d0 0x5604d64e75a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x5604d64d4dd0 tx=0x7f8e5c00bd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa094b280829042c server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.906+0000 7f8e4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:49:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.906+0000 7f8e4f7fe640 1 -- 172.21.15.70:0/924414217 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c003760 con 0x5604d64ca4d0 2024-04-01T01:49:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.906+0000 7f8e4f7fe640 1 -- 172.21.15.70:0/924414217 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f8e5c003900 con 0x5604d64ca4d0 2024-04-01T01:49:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.907+0000 7f8e791cf740 1 -- 172.21.15.70:0/924414217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e5c030070 con 0x5604d64ca4d0 2024-04-01T01:49:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.907+0000 7f8e4f7fe640 1 -- 172.21.15.70:0/924414217 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8e5c003b20 con 0x5604d64ca4d0 2024-04-01T01:49:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.907+0000 7f8e791cf740 10 client.29977 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.907+0000 7f8e791cf740 1 -- 172.21.15.70:0/924414217 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5604d654e340 con 0x5604d64ca4d0 2024-04-01T01:49:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.907+0000 7f8e4f7fe640 1 -- 172.21.15.70:0/924414217 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e5c039070 con 0x5604d64ca4d0 2024-04-01T01:49:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.908+0000 7f8e4f7fe640 1 -- 172.21.15.70:0/924414217 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f8e5c0117b0 con 0x5604d64ca4d0 2024-04-01T01:49:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.908+0000 7f8e791cf740 10 client.29977 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.908+0000 7f8e791cf740 10 client.29977 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:49:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.908+0000 7f8e791cf740 20 client.29977 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.908+0000 7f8e791cf740 1 --2- 172.21.15.70:0/924414217 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5604d64dfe30 0x5604d6550a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.908+0000 7f8e791cf740 4 client.29977 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:49:37.909 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.908+0000 7f8e791cf740 1 -- 172.21.15.70:0/924414217 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f8e5c030070 con 0x5604d64dfe30 2024-04-01T01:49:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.908+0000 7f8e74b86640 1 --2- 172.21.15.70:0/924414217 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5604d64dfe30 0x5604d6550a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.909+0000 7f8e4f7fe640 10 client.29977 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:37.910 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:37.909+0000 7f8e74b86640 1 --2- 172.21.15.70:0/924414217 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5604d64dfe30 0x5604d6550a10 crc :-1 s=READY pgs=405 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48b87256e7d4e357 server_cookie=c40b3944e98ff4e1 in_seq=0 out_seq=0 2024-04-01T01:49:39.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:38 smithi189 ceph-mon[35753]: pgmap v844: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 11 MiB/s wr, 822 op/s 2024-04-01T01:49:39.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:38 smithi081 ceph-mon[35142]: pgmap v844: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 11 MiB/s wr, 822 op/s 2024-04-01T01:49:39.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:38 smithi070 ceph-mon[32179]: pgmap v844: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 11 MiB/s wr, 822 op/s 2024-04-01T01:49:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.062+0000 7f8e4f7fe640 1 -- 172.21.15.70:0/924414217 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f8e5c030070 con 0x5604d64dfe30 2024-04-01T01:49:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.062+0000 7f8e4f7fe640 10 client.29977 handle_command_reply: tid=0 2024-04-01T01:49:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.062+0000 7f8e791cf740 1 client.29977 shutdown 2024-04-01T01:49:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.062+0000 7f8e791cf740 1 -- 172.21.15.70:0/924414217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64ca4d0 msgr2=0x5604d64e75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.062+0000 7f8e791cf740 1 --2- 172.21.15.70:0/924414217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64ca4d0 0x5604d64e75a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x5604d64d4dd0 tx=0x7f8e5c00bd10 comp rx=0 tx=0).stop 2024-04-01T01:49:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.062+0000 7f8e791cf740 1 -- 172.21.15.70:0/924414217 shutdown_connections 2024-04-01T01:49:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.062+0000 7f8e791cf740 1 --2- 172.21.15.70:0/924414217 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x5604d64dfe30 0x5604d6550a10 crc :-1 s=READY pgs=405 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.062+0000 7f8e791cf740 1 --2- 172.21.15.70:0/924414217 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5604d64cf640 0x5604d6544210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.062+0000 7f8e791cf740 1 --2- 172.21.15.70:0/924414217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604d64ca4d0 0x5604d64e75a0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.062+0000 7f8e791cf740 1 --2- 172.21.15.70:0/924414217 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5604d64c9ae0 0x5604d64e7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.062+0000 7f8e791cf740 1 -- 172.21.15.70:0/924414217 >> 172.21.15.70:0/924414217 conn(0x5604d64c5050 msgr2=0x5604d64c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.062+0000 7f8e4f7fe640 0 client.29977 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.063+0000 7f8e791cf740 1 -- 172.21.15.70:0/924414217 shutdown_connections 2024-04-01T01:49:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.063+0000 7f8e791cf740 1 -- 172.21.15.70:0/924414217 wait complete. 2024-04-01T01:49:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.063+0000 7f8e791cf740 20 client.29977 trim_cache size 0 max 16384 2024-04-01T01:49:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:49:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:49:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:49:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:49:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.064+0000 7f8e75387640 1 -- 172.21.15.70:0/1076940106 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e600a2520 msgr2=0x7f8e600a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.064+0000 7f8e75387640 1 --2- 172.21.15.70:0/1076940106 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e600a2520 0x7f8e600a49e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8e64097050 tx=0x7f8e64006df0 comp rx=0 tx=0).stop 2024-04-01T01:49:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.064+0000 7f8e75387640 1 -- 172.21.15.70:0/1076940106 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e680a61c0 msgr2=0x7f8e6813b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.064+0000 7f8e75387640 1 --2- 172.21.15.70:0/1076940106 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e680a61c0 0x7f8e6813b2e0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f8e6400b450 tx=0x7f8e6400b910 comp rx=0 tx=0).stop 2024-04-01T01:49:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.064+0000 7f8e75387640 1 -- 172.21.15.70:0/1076940106 shutdown_connections 2024-04-01T01:49:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.064+0000 7f8e75387640 1 --2- 172.21.15.70:0/1076940106 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e680a6b80 0x7f8e68142310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.064+0000 7f8e75387640 1 --2- 172.21.15.70:0/1076940106 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e680a61c0 0x7f8e6813b2e0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.064+0000 7f8e75387640 1 --2- 172.21.15.70:0/1076940106 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f8e600a2520 0x7f8e600a49e0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.064+0000 7f8e75387640 1 --2- 172.21.15.70:0/1076940106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e680a4fd0 0x7f8e6813ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.065+0000 7f8e75387640 1 -- 172.21.15.70:0/1076940106 >> 172.21.15.70:0/1076940106 conn(0x7f8e680a00d0 msgr2=0x7f8e680a94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.065+0000 7f8e75387640 1 -- 172.21.15.70:0/1076940106 shutdown_connections 2024-04-01T01:49:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:40.065+0000 7f8e75387640 1 -- 172.21.15.70:0/1076940106 wait complete. 2024-04-01T01:49:41.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:40 smithi081 ceph-mon[35142]: pgmap v845: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 12 MiB/s wr, 868 op/s 2024-04-01T01:49:41.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:40 smithi189 ceph-mon[35753]: pgmap v845: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 12 MiB/s wr, 868 op/s 2024-04-01T01:49:41.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:40 smithi070 ceph-mon[32179]: pgmap v845: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 12 MiB/s wr, 868 op/s 2024-04-01T01:49:41.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 895.0 s, 787.1 tps, lat 126.921 ms stddev 105.484 2024-04-01T01:49:43.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:42 smithi081 ceph-mon[35142]: pgmap v846: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 11 MiB/s wr, 899 op/s 2024-04-01T01:49:43.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:42 smithi189 ceph-mon[35753]: pgmap v846: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 11 MiB/s wr, 899 op/s 2024-04-01T01:49:43.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:42 smithi070 ceph-mon[32179]: pgmap v846: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 11 MiB/s wr, 899 op/s 2024-04-01T01:49:45.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:44 smithi081 ceph-mon[35142]: pgmap v847: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 11 MiB/s wr, 846 op/s 2024-04-01T01:49:45.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:44 smithi189 ceph-mon[35753]: pgmap v847: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 11 MiB/s wr, 846 op/s 2024-04-01T01:49:45.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:44 smithi070 ceph-mon[32179]: pgmap v847: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 11 MiB/s wr, 846 op/s 2024-04-01T01:49:46.896 INFO:teuthology.orchestra.run.smithi070.stderr:progress: 900.0 s, 779.3 tps, lat 128.086 ms stddev 115.165 2024-04-01T01:49:47.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:46 smithi081 ceph-mon[35142]: pgmap v848: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 10 MiB/s wr, 830 op/s 2024-04-01T01:49:47.156 INFO:teuthology.orchestra.run.smithi070.stdout:transaction type: 2024-04-01T01:49:47.156 INFO:teuthology.orchestra.run.smithi070.stdout:scaling factor: 32 2024-04-01T01:49:47.156 INFO:teuthology.orchestra.run.smithi070.stdout:query mode: simple 2024-04-01T01:49:47.156 INFO:teuthology.orchestra.run.smithi070.stdout:number of clients: 100 2024-04-01T01:49:47.156 INFO:teuthology.orchestra.run.smithi070.stdout:number of threads: 4 2024-04-01T01:49:47.156 INFO:teuthology.orchestra.run.smithi070.stdout:duration: 900 s 2024-04-01T01:49:47.156 INFO:teuthology.orchestra.run.smithi070.stdout:number of transactions actually processed: 803167 2024-04-01T01:49:47.156 INFO:teuthology.orchestra.run.smithi070.stdout:latency average = 112.024 ms 2024-04-01T01:49:47.156 INFO:teuthology.orchestra.run.smithi070.stdout:latency stddev = 136.932 ms 2024-04-01T01:49:47.156 INFO:teuthology.orchestra.run.smithi070.stdout:tps = 892.150095 (including connections establishing) 2024-04-01T01:49:47.156 INFO:teuthology.orchestra.run.smithi070.stdout:tps = 892.160647 (excluding connections establishing) 2024-04-01T01:49:47.157 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:46 smithi070 ceph-mon[32179]: pgmap v848: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 10 MiB/s wr, 830 op/s 2024-04-01T01:49:47.159 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl stop postgresql' 2024-04-01T01:49:47.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:46 smithi189 ceph-mon[35753]: pgmap v848: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 10 MiB/s wr, 830 op/s 2024-04-01T01:49:49.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:48 smithi081 ceph-mon[35142]: pgmap v849: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 9.8 MiB/s wr, 807 op/s 2024-04-01T01:49:49.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:48 smithi189 ceph-mon[35753]: pgmap v849: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 9.8 MiB/s wr, 807 op/s 2024-04-01T01:49:49.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:48 smithi070 ceph-mon[32179]: pgmap v849: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 9.8 MiB/s wr, 807 op/s 2024-04-01T01:49:51.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:50 smithi081 ceph-mon[35142]: pgmap v850: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 9.9 MiB/s wr, 784 op/s 2024-04-01T01:49:51.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:50 smithi189 ceph-mon[35753]: pgmap v850: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 9.9 MiB/s wr, 784 op/s 2024-04-01T01:49:51.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:50 smithi070 ceph-mon[32179]: pgmap v850: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 9.9 MiB/s wr, 784 op/s 2024-04-01T01:49:53.079 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:52 smithi081 ceph-mon[35142]: pgmap v851: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 9.6 MiB/s wr, 780 op/s 2024-04-01T01:49:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.241+0000 7f6e86bf3640 1 -- 172.21.15.70:0/833270508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e801016d0 msgr2=0x7f6e80101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:53.242 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.241+0000 7f6e86bf3640 1 --2- 172.21.15.70:0/833270508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e801016d0 0x7f6e80101b50 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6e70009900 tx=0x7f6e7002bc60 comp rx=0 tx=0).stop 2024-04-01T01:49:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.248+0000 7f6e86bf3640 1 -- 172.21.15.70:0/833270508 shutdown_connections 2024-04-01T01:49:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.248+0000 7f6e86bf3640 1 --2- 172.21.15.70:0/833270508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e80102090 0x7f6e80108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.248+0000 7f6e86bf3640 1 --2- 172.21.15.70:0/833270508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e801016d0 0x7f6e80101b50 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.248+0000 7f6e86bf3640 1 --2- 172.21.15.70:0/833270508 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e801004e0 0x7f6e80100900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.248 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.248+0000 7f6e86bf3640 1 -- 172.21.15.70:0/833270508 >> 172.21.15.70:0/833270508 conn(0x7f6e800fba50 msgr2=0x7f6e800fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:53.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:52 smithi189 ceph-mon[35753]: pgmap v851: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 9.6 MiB/s wr, 780 op/s 2024-04-01T01:49:53.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:52 smithi070 ceph-mon[32179]: pgmap v851: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 9.6 MiB/s wr, 780 op/s 2024-04-01T01:49:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.253+0000 7f6e86bf3640 1 -- 172.21.15.70:0/833270508 shutdown_connections 2024-04-01T01:49:53.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.257+0000 7f6e86bf3640 1 -- 172.21.15.70:0/833270508 wait complete. 2024-04-01T01:49:53.258 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.257+0000 7f6e86bf3640 1 Processor -- start 2024-04-01T01:49:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.262+0000 7f6e86bf3640 1 -- start start 2024-04-01T01:49:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.262+0000 7f6e86bf3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e801004e0 0x7f6e801955e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.262+0000 7f6e86bf3640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e801016d0 0x7f6e80195b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.262+0000 7f6e86bf3640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e80102090 0x7f6e801ade40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.262+0000 7f6e86bf3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8010af60 con 0x7f6e801004e0 2024-04-01T01:49:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.262+0000 7f6e86bf3640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8010ade0 con 0x7f6e80102090 2024-04-01T01:49:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.262+0000 7f6e86bf3640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8010b0e0 con 0x7f6e801016d0 2024-04-01T01:49:53.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.265+0000 7f6e853f0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e801016d0 0x7f6e80195b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:53.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.265+0000 7f6e853f0640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e801016d0 0x7f6e80195b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58214/0 (socket says 172.21.15.70:58214) 2024-04-01T01:49:53.266 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.265+0000 7f6e853f0640 1 -- 172.21.15.70:0/2965855177 learned_addr learned my addr 172.21.15.70:0/2965855177 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:53.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.268+0000 7f6e85bf1640 1 --2- 172.21.15.70:0/2965855177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e801004e0 0x7f6e801955e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:53.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.268+0000 7f6e853f0640 1 -- 172.21.15.70:0/2965855177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e80102090 msgr2=0x7f6e801ade40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-01T01:49:53.269 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.268+0000 7f6e853f0640 1 --2- 172.21.15.70:0/2965855177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e80102090 0x7f6e801ade40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.268+0000 7f6e853f0640 1 -- 172.21.15.70:0/2965855177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e801004e0 msgr2=0x7f6e801955e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:53.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.268+0000 7f6e853f0640 1 --2- 172.21.15.70:0/2965855177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e801004e0 0x7f6e801955e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.268+0000 7f6e853f0640 1 -- 172.21.15.70:0/2965855177 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e8006b170 con 0x7f6e801016d0 2024-04-01T01:49:53.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.269+0000 7f6e853f0640 1 --2- 172.21.15.70:0/2965855177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e801016d0 0x7f6e80195b20 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6e7002efd0 tx=0x7f6e700045c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:53.270 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.269+0000 7f6e85bf1640 1 --2- 172.21.15.70:0/2965855177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e801004e0 0x7f6e801955e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-01T01:49:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.271+0000 7f6e76ffd640 1 -- 172.21.15.70:0/2965855177 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e70042020 con 0x7f6e801016d0 2024-04-01T01:49:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.271+0000 7f6e76ffd640 1 -- 172.21.15.70:0/2965855177 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e700035f0 con 0x7f6e801016d0 2024-04-01T01:49:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.271+0000 7f6e76ffd640 1 -- 172.21.15.70:0/2965855177 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7003e6c0 con 0x7f6e801016d0 2024-04-01T01:49:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.272+0000 7f6e86bf3640 1 -- 172.21.15.70:0/2965855177 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e8006b400 con 0x7f6e801016d0 2024-04-01T01:49:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.272+0000 7f6e86bf3640 1 -- 172.21.15.70:0/2965855177 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e8006b8a0 con 0x7f6e801016d0 2024-04-01T01:49:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.276+0000 7f6e8aa36740 1 Processor -- start 2024-04-01T01:49:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.278+0000 7f6e8aa36740 1 -- start start 2024-04-01T01:49:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.278+0000 7f6e8aa36740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f5bae0 0x564bd0f5bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.278+0000 7f6e8aa36740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5c4d0 0x564bd0f60fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.278+0000 7f6e8aa36740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f61640 0x564bd0f63ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.278+0000 7f6e8aa36740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564bd0ebf8f0 con 0x564bd0f5c4d0 2024-04-01T01:49:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.278+0000 7f6e8aa36740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x564bd0e688d0 con 0x564bd0f61640 2024-04-01T01:49:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.278+0000 7f6e8aa36740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564bd0ebb990 con 0x564bd0f5bae0 2024-04-01T01:49:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.280+0000 7f6e863f2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f5bae0 0x564bd0f5bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.282+0000 7f6e863f2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f5bae0 0x564bd0f5bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58224/0 (socket says 172.21.15.70:58224) 2024-04-01T01:49:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.282+0000 7f6e863f2640 1 -- 172.21.15.70:0/2010283171 learned_addr learned my addr 172.21.15.70:0/2010283171 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.282+0000 7f6e863f2640 1 -- 172.21.15.70:0/2010283171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f61640 msgr2=0x564bd0f63ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.282+0000 7f6e863f2640 1 --2- 172.21.15.70:0/2010283171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f61640 0x564bd0f63ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.282+0000 7f6e863f2640 1 -- 172.21.15.70:0/2010283171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5c4d0 msgr2=0x564bd0f60fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.282+0000 7f6e863f2640 1 --2- 172.21.15.70:0/2010283171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5c4d0 0x564bd0f60fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.283+0000 7f6e863f2640 1 -- 172.21.15.70:0/2010283171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bd0bd4210 con 0x564bd0f5bae0 2024-04-01T01:49:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.283+0000 7f6e863f2640 1 --2- 172.21.15.70:0/2010283171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f5bae0 0x564bd0f5bf00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800b580 tx=0x7f6e7800d910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a992e72130c1fe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.283+0000 7f6e76ffd640 1 -- 172.21.15.70:0/2965855177 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f6e70045020 con 0x7f6e801016d0 2024-04-01T01:49:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.283+0000 7f6e6bfff640 1 -- 172.21.15.70:0/2010283171 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e78004b20 con 0x564bd0f5bae0 2024-04-01T01:49:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.283+0000 7f6e6bfff640 1 -- 172.21.15.70:0/2010283171 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e78004cc0 con 0x564bd0f5bae0 2024-04-01T01:49:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.283+0000 7f6e6bfff640 1 -- 172.21.15.70:0/2010283171 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e78017770 con 0x564bd0f5bae0 2024-04-01T01:49:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.284+0000 7f6e76ffd640 1 --2- 172.21.15.70:0/2965855177 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6e6c0a26a0 0x7f6e6c0a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.284+0000 7f6e76ffd640 1 -- 172.21.15.70:0/2965855177 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f6e700f0bb0 con 0x7f6e801016d0 2024-04-01T01:49:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.286+0000 7f6e8aa36740 1 -- 172.21.15.70:0/2010283171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f5bae0 msgr2=0x564bd0f5bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.286+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/2010283171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f5bae0 0x564bd0f5bf00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800b580 tx=0x7f6e7800d910 comp rx=0 tx=0).stop 2024-04-01T01:49:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.286+0000 7f6e8aa36740 1 -- 172.21.15.70:0/2010283171 shutdown_connections 2024-04-01T01:49:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.286+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/2010283171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f61640 0x564bd0f63ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.286+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/2010283171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5c4d0 0x564bd0f60fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.286+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/2010283171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f5bae0 0x564bd0f5bf00 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.286+0000 7f6e8aa36740 1 -- 172.21.15.70:0/2010283171 >> 172.21.15.70:0/2010283171 conn(0x564bd0f57050 msgr2=0x564bd0f57450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.288+0000 7f6e85bf1640 1 --2- 172.21.15.70:0/2965855177 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6e6c0a26a0 0x7f6e6c0a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.291+0000 7f6e8aa36740 1 -- 172.21.15.70:0/2010283171 shutdown_connections 2024-04-01T01:49:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.294+0000 7f6e8aa36740 1 -- 172.21.15.70:0/2010283171 wait complete. 2024-04-01T01:49:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.294+0000 7f6e8aa36740 1 -- start start 2024-04-01T01:49:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.294+0000 7f6e8aa36740 1 Processor -- start 2024-04-01T01:49:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.295+0000 7f6e8aa36740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5bae0 0x564bd0f65e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.295+0000 7f6e8aa36740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f5c4d0 0x564bd0f66360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.295+0000 7f6e8aa36740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f61640 0x564bd0f6d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.295+0000 7f6e8aa36740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564bd0f645f0 con 0x564bd0f5bae0 2024-04-01T01:49:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.295+0000 7f6e8aa36740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x564bd0e5df80 con 0x564bd0f5c4d0 2024-04-01T01:49:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.295+0000 7f6e8aa36740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564bd0f64770 con 0x564bd0f61640 2024-04-01T01:49:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.295+0000 7f6e8aa36740 10 client.0 _ll_register_callbacks cb 0x564bd0eb6290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.295+0000 7f6e863f2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f61640 0x564bd0f6d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.295+0000 7f6e863f2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f61640 0x564bd0f6d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58232/0 (socket says 172.21.15.70:58232) 2024-04-01T01:49:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.295+0000 7f6e863f2640 1 -- 172.21.15.70:0/1177322618 learned_addr learned my addr 172.21.15.70:0/1177322618 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:53.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.299+0000 7f6e863f2640 1 -- 172.21.15.70:0/1177322618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f5c4d0 msgr2=0x564bd0f66360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:53.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.299+0000 7f6e863f2640 1 --2- 172.21.15.70:0/1177322618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f5c4d0 0x564bd0f66360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.299+0000 7f6e863f2640 1 -- 172.21.15.70:0/1177322618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5bae0 msgr2=0x564bd0f65e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:53.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.299+0000 7f6e863f2640 1 --2- 172.21.15.70:0/1177322618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5bae0 0x564bd0f65e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.299 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.299+0000 7f6e863f2640 1 -- 172.21.15.70:0/1177322618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bd0ca0620 con 0x564bd0f61640 2024-04-01T01:49:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.299+0000 7f6e863f2640 1 --2- 172.21.15.70:0/1177322618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f61640 0x564bd0f6d3a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6e78003c20 tx=0x7f6e78003c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c191f6ffa59c4d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.299+0000 7f6e697fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:49:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.299+0000 7f6e697fa640 1 -- 172.21.15.70:0/1177322618 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e78025020 con 0x564bd0f61640 2024-04-01T01:49:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.300+0000 7f6e8aa36740 1 -- 172.21.15.70:0/1177322618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e78004cc0 con 0x564bd0f61640 2024-04-01T01:49:53.300 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.300+0000 7f6e697fa640 1 -- 172.21.15.70:0/1177322618 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f6e78005230 con 0x564bd0f61640 2024-04-01T01:49:53.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.300+0000 7f6e697fa640 1 -- 172.21.15.70:0/1177322618 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f6e78030050 con 0x564bd0f61640 2024-04-01T01:49:53.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.301+0000 7f6e8aa36740 10 client.29989 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:53.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.301+0000 7f6e8aa36740 1 -- 172.21.15.70:0/1177322618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564bd0f703d0 con 0x564bd0f61640 2024-04-01T01:49:53.301 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.301+0000 7f6e85bf1640 1 --2- 172.21.15.70:0/2965855177 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6e6c0a26a0 0x7f6e6c0a4b60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x564bd0f57fc0 tx=0x7f6e7c00d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:53.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.305+0000 7f6e697fa640 1 -- 172.21.15.70:0/1177322618 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e78004cc0 con 0x564bd0f61640 2024-04-01T01:49:53.305 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.305+0000 7f6e697fa640 1 -- 172.21.15.70:0/1177322618 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f6e780174d0 con 0x564bd0f61640 2024-04-01T01:49:53.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.306+0000 7f6e8aa36740 10 client.29989 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:53.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.306+0000 7f6e8aa36740 10 client.29989 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:49:53.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.306+0000 7f6e8aa36740 20 client.29989 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:53.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.306+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/1177322618 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564bd0f71830 0x564bd0f73c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.306+0000 7f6e8aa36740 4 client.29989 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:49:53.309 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.306+0000 7f6e8aa36740 1 -- 172.21.15.70:0/1177322618 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564bd0ed3120 con 0x564bd0f71830 2024-04-01T01:49:53.316 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.315+0000 7f6e85bf1640 1 --2- 172.21.15.70:0/1177322618 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564bd0f71830 0x564bd0f73c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.333+0000 7f6e85bf1640 1 --2- 172.21.15.70:0/1177322618 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564bd0f71830 0x564bd0f73c20 crc :-1 s=READY pgs=406 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb9199c759caeb78 server_cookie=cfad2d06a61cbce4 in_seq=0 out_seq=0 2024-04-01T01:49:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.334+0000 7f6e697fa640 10 client.29989 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.337+0000 7f6e697fa640 1 -- 172.21.15.70:0/1177322618 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564bd0ed3120 con 0x564bd0f71830 2024-04-01T01:49:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.337+0000 7f6e697fa640 10 client.29989 handle_command_reply: tid=0 2024-04-01T01:49:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.337+0000 7f6e8aa36740 1 client.29989 shutdown 2024-04-01T01:49:53.343 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.343+0000 7f6e8aa36740 1 -- 172.21.15.70:0/1177322618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f61640 msgr2=0x564bd0f6d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:53.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.343+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/1177322618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f61640 0x564bd0f6d3a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6e78003c20 tx=0x7f6e78003c50 comp rx=0 tx=0).stop 2024-04-01T01:49:53.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.343+0000 7f6e8aa36740 1 -- 172.21.15.70:0/1177322618 shutdown_connections 2024-04-01T01:49:53.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.343+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/1177322618 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564bd0f71830 0x564bd0f73c20 crc :-1 s=READY pgs=406 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.343+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/1177322618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f61640 0x564bd0f6d3a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.343+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/1177322618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f5c4d0 0x564bd0f66360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.343+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/1177322618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5bae0 0x564bd0f65e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.344 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.343+0000 7f6e8aa36740 1 -- 172.21.15.70:0/1177322618 >> 172.21.15.70:0/1177322618 conn(0x564bd0f57050 msgr2=0x564bd0f56190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:53.345 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.343+0000 7f6e697fa640 0 client.29989 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.346+0000 7f6e8aa36740 1 -- 172.21.15.70:0/1177322618 shutdown_connections 2024-04-01T01:49:53.347 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.347+0000 7f6e8aa36740 1 -- 172.21.15.70:0/1177322618 wait complete. 2024-04-01T01:49:53.348 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.348+0000 7f6e8aa36740 20 client.29989 trim_cache size 0 max 16384 2024-04-01T01:49:53.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.385+0000 7f6e8aa36740 1 Processor -- start 2024-04-01T01:49:53.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.385+0000 7f6e8aa36740 1 -- start start 2024-04-01T01:49:53.388 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.385+0000 7f6e8aa36740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f5bae0 0x564bd0f55900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.386+0000 7f6e8aa36740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5c4d0 0x564bd0f6d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.386+0000 7f6e8aa36740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f61640 0x564bd0f55e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.386+0000 7f6e85bf1640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f5bae0 0x564bd0f55900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.386+0000 7f6e8aa36740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8019b320 con 0x564bd0f5c4d0 2024-04-01T01:49:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.386+0000 7f6e863f2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f61640 0x564bd0f55e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.386+0000 7f6e863f2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f61640 0x564bd0f55e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58246/0 (socket says 172.21.15.70:58246) 2024-04-01T01:49:53.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.386+0000 7f6e863f2640 1 -- 172.21.15.70:0/2890050658 learned_addr learned my addr 172.21.15.70:0/2890050658 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:53.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.386+0000 7f6e8aa36740 1 -- 172.21.15.70:0/2890050658 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8019ce60 con 0x564bd0f5bae0 2024-04-01T01:49:53.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.386+0000 7f6e8aa36740 1 -- 172.21.15.70:0/2890050658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8019b9f0 con 0x564bd0f61640 2024-04-01T01:49:53.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.387+0000 7f6e853f0640 1 --2- 172.21.15.70:0/2890050658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5c4d0 0x564bd0f6d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:53.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.387+0000 7f6e853f0640 1 -- 172.21.15.70:0/2890050658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f61640 msgr2=0x564bd0f55e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:53.390 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.387+0000 7f6e853f0640 1 --2- 172.21.15.70:0/2890050658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f61640 0x564bd0f55e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.387+0000 7f6e853f0640 1 -- 172.21.15.70:0/2890050658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f5bae0 msgr2=0x564bd0f55900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:53.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.387+0000 7f6e853f0640 1 --2- 172.21.15.70:0/2890050658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f5bae0 0x564bd0f55900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.391 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.387+0000 7f6e853f0640 1 -- 172.21.15.70:0/2890050658 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bd0f5bec0 con 0x564bd0f5c4d0 2024-04-01T01:49:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.388+0000 7f6e853f0640 1 --2- 172.21.15.70:0/2890050658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5c4d0 0x564bd0f6d360 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f6e700ee250 tx=0x7f6e700b8ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de38f96d641e9696 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.388+0000 7f6e6b7fe640 1 -- 172.21.15.70:0/2890050658 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e700bcba0 con 0x564bd0f5c4d0 2024-04-01T01:49:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.388+0000 7f6e6b7fe640 1 -- 172.21.15.70:0/2890050658 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e70004c50 con 0x564bd0f5c4d0 2024-04-01T01:49:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.388+0000 7f6e6b7fe640 1 -- 172.21.15.70:0/2890050658 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e700c5c50 con 0x564bd0f5c4d0 2024-04-01T01:49:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.391+0000 7f6e8aa36740 1 -- 172.21.15.70:0/2890050658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5c4d0 msgr2=0x564bd0f6d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.391+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/2890050658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5c4d0 0x564bd0f6d360 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f6e700ee250 tx=0x7f6e700b8ee0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.391+0000 7f6e8aa36740 1 -- 172.21.15.70:0/2890050658 shutdown_connections 2024-04-01T01:49:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.391+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/2890050658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f61640 0x564bd0f55e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.391+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/2890050658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5c4d0 0x564bd0f6d360 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.391+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/2890050658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f5bae0 0x564bd0f55900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.391+0000 7f6e8aa36740 1 -- 172.21.15.70:0/2890050658 >> 172.21.15.70:0/2890050658 conn(0x564bd0f57050 msgr2=0x564bd0f57430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.391+0000 7f6e8aa36740 1 -- 172.21.15.70:0/2890050658 shutdown_connections 2024-04-01T01:49:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.391+0000 7f6e8aa36740 1 -- 172.21.15.70:0/2890050658 wait complete. 2024-04-01T01:49:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.391+0000 7f6e8aa36740 1 -- start start 2024-04-01T01:49:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.391+0000 7f6e8aa36740 1 Processor -- start 2024-04-01T01:49:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.392+0000 7f6e8aa36740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f5bae0 0x564bd0f6a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.392+0000 7f6e8aa36740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5c4d0 0x564bd0f6adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.392+0000 7f6e8aa36740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f61640 0x564bd0f6bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.392+0000 7f6e8aa36740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8019c790 con 0x564bd0f5c4d0 2024-04-01T01:49:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.392+0000 7f6e8aa36740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8019c0c0 con 0x564bd0f61640 2024-04-01T01:49:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.392+0000 7f6e8aa36740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6e80199300 con 0x564bd0f5bae0 2024-04-01T01:49:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.392+0000 7f6e8aa36740 10 client.0 _ll_register_callbacks cb 0x7f6e7800b480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:49:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.392+0000 7f6e85bf1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f5bae0 0x564bd0f6a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.392+0000 7f6e85bf1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f5bae0 0x564bd0f6a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:58252/0 (socket says 172.21.15.70:58252) 2024-04-01T01:49:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.392+0000 7f6e85bf1640 1 -- 172.21.15.70:0/566888332 learned_addr learned my addr 172.21.15.70:0/566888332 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:49:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.396+0000 7f6e853f0640 1 --2- 172.21.15.70:0/566888332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5c4d0 0x564bd0f6adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.396+0000 7f6e863f2640 1 --2- 172.21.15.70:0/566888332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f61640 0x564bd0f6bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.396+0000 7f6e85bf1640 1 -- 172.21.15.70:0/566888332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f61640 msgr2=0x564bd0f6bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.396+0000 7f6e85bf1640 1 --2- 172.21.15.70:0/566888332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f61640 0x564bd0f6bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.396+0000 7f6e85bf1640 1 -- 172.21.15.70:0/566888332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5c4d0 msgr2=0x564bd0f6adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:49:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.396+0000 7f6e85bf1640 1 --2- 172.21.15.70:0/566888332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5c4d0 0x564bd0f6adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:53.399 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.396+0000 7f6e85bf1640 1 -- 172.21.15.70:0/566888332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bd0f6c3b0 con 0x564bd0f5bae0 2024-04-01T01:49:53.402 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.402+0000 7f6e85bf1640 1 --2- 172.21.15.70:0/566888332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f5bae0 0x564bd0f6a870 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x564bd0f5cf40 tx=0x7f6e7c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac160d29071e868f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:49:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.403+0000 7f6e697fa640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-01T01:49:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.403+0000 7f6e697fa640 1 -- 172.21.15.70:0/566888332 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c0168d0 con 0x564bd0f5bae0 2024-04-01T01:49:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.403+0000 7f6e697fa640 1 -- 172.21.15.70:0/566888332 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f6e7c016a70 con 0x564bd0f5bae0 2024-04-01T01:49:53.404 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.403+0000 7f6e8aa36740 1 -- 172.21.15.70:0/566888332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e70004c50 con 0x564bd0f5bae0 2024-04-01T01:49:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.404+0000 7f6e697fa640 1 -- 172.21.15.70:0/566888332 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f6e7c031050 con 0x564bd0f5bae0 2024-04-01T01:49:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.404+0000 7f6e8aa36740 10 client.29995 fetch_fsmap learned FSMap version 33 2024-04-01T01:49:53.405 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.405+0000 7f6e8aa36740 1 -- 172.21.15.70:0/566888332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564bd0fd7b20 con 0x564bd0f5bae0 2024-04-01T01:49:53.406 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.405+0000 7f6e697fa640 1 -- 172.21.15.70:0/566888332 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e70004c50 con 0x564bd0f5bae0 2024-04-01T01:49:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.406+0000 7f6e697fa640 1 -- 172.21.15.70:0/566888332 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f6e7c015070 con 0x564bd0f5bae0 2024-04-01T01:49:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.406+0000 7f6e8aa36740 10 client.29995 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:49:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.406+0000 7f6e8aa36740 10 client.29995 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-01T01:49:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.406+0000 7f6e8aa36740 20 client.29995 populate_metadata read hostname 'smithi070' 2024-04-01T01:49:53.407 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.406+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/566888332 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564bd0fd8460 0x564bd0fda880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:49:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.406+0000 7f6e8aa36740 4 client.29995 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-01T01:49:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.406+0000 7f6e8aa36740 1 -- 172.21.15.70:0/566888332 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564bd0ecce30 con 0x564bd0fd8460 2024-04-01T01:49:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.412+0000 7f6e853f0640 1 --2- 172.21.15.70:0/566888332 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564bd0fd8460 0x564bd0fda880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:49:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.414+0000 7f6e853f0640 1 --2- 172.21.15.70:0/566888332 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564bd0fd8460 0x564bd0fda880 crc :-1 s=READY pgs=407 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cea77ca8f137e851 server_cookie=1cfc424cf6d8f0a4 in_seq=0 out_seq=0 2024-04-01T01:49:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:53.415+0000 7f6e697fa640 10 client.29995 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:55.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:54 smithi081 ceph-mon[35142]: pgmap v852: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.3 MiB/s wr, 819 op/s 2024-04-01T01:49:55.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:54 smithi189 ceph-mon[35753]: pgmap v852: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.3 MiB/s wr, 819 op/s 2024-04-01T01:49:55.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:54 smithi070 ceph-mon[32179]: pgmap v852: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.3 MiB/s wr, 819 op/s 2024-04-01T01:49:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.338+0000 7f6e697fa640 1 -- 172.21.15.70:0/566888332 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564bd0ecce30 con 0x564bd0fd8460 2024-04-01T01:49:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.338+0000 7f6e697fa640 10 client.29995 handle_command_reply: tid=0 2024-04-01T01:49:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.338+0000 7f6e8aa36740 1 client.29995 shutdown 2024-04-01T01:49:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.338+0000 7f6e8aa36740 1 -- 172.21.15.70:0/566888332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f5bae0 msgr2=0x564bd0f6a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.338+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/566888332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f5bae0 0x564bd0f6a870 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x564bd0f5cf40 tx=0x7f6e7c004730 comp rx=0 tx=0).stop 2024-04-01T01:49:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.338+0000 7f6e8aa36740 1 -- 172.21.15.70:0/566888332 shutdown_connections 2024-04-01T01:49:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.338+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/566888332 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x564bd0fd8460 0x564bd0fda880 crc :-1 s=READY pgs=407 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.338+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/566888332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x564bd0f61640 0x564bd0f6bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.338+0000 7f6e697fa640 0 client.29995 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:49:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.338+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/566888332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564bd0f5c4d0 0x564bd0f6adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.338+0000 7f6e8aa36740 1 --2- 172.21.15.70:0/566888332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564bd0f5bae0 0x564bd0f6a870 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.338+0000 7f6e8aa36740 1 -- 172.21.15.70:0/566888332 >> 172.21.15.70:0/566888332 conn(0x564bd0f57050 msgr2=0x564bd0f65dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:56.339 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.338+0000 7f6e8aa36740 1 -- 172.21.15.70:0/566888332 shutdown_connections 2024-04-01T01:49:56.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.339+0000 7f6e8aa36740 1 -- 172.21.15.70:0/566888332 wait complete. 2024-04-01T01:49:56.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.339+0000 7f6e8aa36740 20 client.29995 trim_cache size 0 max 16384 2024-04-01T01:49:56.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:{ 2024-04-01T01:49:56.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "message": "", 2024-04-01T01:49:56.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout: "return_code": 0 2024-04-01T01:49:56.340 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stdout:} 2024-04-01T01:49:56.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.339+0000 7f6e86bf3640 1 -- 172.21.15.70:0/2965855177 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6e6c0a26a0 msgr2=0x7f6e6c0a4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:56.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.339+0000 7f6e86bf3640 1 --2- 172.21.15.70:0/2965855177 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6e6c0a26a0 0x7f6e6c0a4b60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x564bd0f57fc0 tx=0x7f6e7c00d040 comp rx=0 tx=0).stop 2024-04-01T01:49:56.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.339+0000 7f6e86bf3640 1 -- 172.21.15.70:0/2965855177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e801016d0 msgr2=0x7f6e80195b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:49:56.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.339+0000 7f6e86bf3640 1 --2- 172.21.15.70:0/2965855177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e801016d0 0x7f6e80195b20 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6e7002efd0 tx=0x7f6e700045c0 comp rx=0 tx=0).stop 2024-04-01T01:49:56.341 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.339+0000 7f6e86bf3640 1 -- 172.21.15.70:0/2965855177 shutdown_connections 2024-04-01T01:49:56.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.339+0000 7f6e86bf3640 1 --2- 172.21.15.70:0/2965855177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e80102090 0x7f6e801ade40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:56.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.339+0000 7f6e86bf3640 1 --2- 172.21.15.70:0/2965855177 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6e6c0a26a0 0x7f6e6c0a4b60 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:56.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.339+0000 7f6e86bf3640 1 --2- 172.21.15.70:0/2965855177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e801016d0 0x7f6e80195b20 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:56.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.339+0000 7f6e86bf3640 1 --2- 172.21.15.70:0/2965855177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e801004e0 0x7f6e801955e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:49:56.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.339+0000 7f6e86bf3640 1 -- 172.21.15.70:0/2965855177 >> 172.21.15.70:0/2965855177 conn(0x7f6e800fba50 msgr2=0x7f6e80104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:49:56.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.339+0000 7f6e86bf3640 1 -- 172.21.15.70:0/2965855177 shutdown_connections 2024-04-01T01:49:56.342 INFO:teuthology.task.background_exec.ubuntu@smithi070.front.sepia.ceph.com.smithi070.stderr:2024-04-01T01:49:56.339+0000 7f6e86bf3640 1 -- 172.21.15.70:0/2965855177 wait complete. 2024-04-01T01:49:57.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:56 smithi081 ceph-mon[35142]: pgmap v853: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.1 MiB/s wr, 824 op/s 2024-04-01T01:49:57.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:56 smithi189 ceph-mon[35753]: pgmap v853: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.1 MiB/s wr, 824 op/s 2024-04-01T01:49:57.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:56 smithi070 ceph-mon[32179]: pgmap v853: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 9.1 MiB/s wr, 824 op/s 2024-04-01T01:49:57.823 DEBUG:teuthology.orchestra.run.smithi070:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-04-01T01:49:58.106 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/: 2024-04-01T01:49:58.106 INFO:teuthology.orchestra.run.smithi070.stdout:total 2 2024-04-01T01:49:58.106 INFO:teuthology.orchestra.run.smithi070.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1557632143 Apr 1 01:33 . 2024-04-01T01:49:58.106 INFO:teuthology.orchestra.run.smithi070.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1557632143 Apr 1 01:33 .. 2024-04-01T01:49:58.106 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 data 2024-04-01T01:49:58.106 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.106 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data: 2024-04-01T01:49:58.106 INFO:teuthology.orchestra.run.smithi070.stdout:total 46 2024-04-01T01:49:58.107 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 . 2024-04-01T01:49:58.107 INFO:teuthology.orchestra.run.smithi070.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1557632143 Apr 1 01:33 .. 2024-04-01T01:49:58.107 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 583636533 Apr 1 01:34 base 2024-04-01T01:49:58.107 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 30 Apr 1 01:34 current_logfiles 2024-04-01T01:49:58.107 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 572520 Apr 1 01:34 global 2024-04-01T01:49:58.107 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1047 Apr 1 01:34 log 2024-04-01T01:49:58.107 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_commit_ts 2024-04-01T01:49:58.107 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_dynshmem 2024-04-01T01:49:58.107 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Apr 1 01:33 pg_hba.conf 2024-04-01T01:49:58.107 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Apr 1 01:33 pg_ident.conf 2024-04-01T01:49:58.107 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 1 01:49 pg_logical 2024-04-01T01:49:58.107 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 pg_multixact 2024-04-01T01:49:58.108 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_notify 2024-04-01T01:49:58.108 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_replslot 2024-04-01T01:49:58.108 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_serial 2024-04-01T01:49:58.108 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_snapshots 2024-04-01T01:49:58.108 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 14082 Apr 1 01:49 pg_stat 2024-04-01T01:49:58.108 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:49 pg_stat_tmp 2024-04-01T01:49:58.108 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 73728 Apr 1 01:49 pg_subtrans 2024-04-01T01:49:58.108 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_tblspc 2024-04-01T01:49:58.108 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 pg_twophase 2024-04-01T01:49:58.108 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 1 01:33 PG_VERSION 2024-04-01T01:49:58.108 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 939524096 Apr 1 01:49 pg_wal 2024-04-01T01:49:58.108 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 204800 Apr 1 01:33 pg_xact 2024-04-01T01:49:58.108 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Apr 1 01:33 postgresql.auto.conf 2024-04-01T01:49:58.108 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Apr 1 01:33 postgresql.conf 2024-04-01T01:49:58.109 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 51 Apr 1 01:34 postmaster.opts 2024-04-01T01:49:58.109 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.109 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/base: 2024-04-01T01:49:58.109 INFO:teuthology.orchestra.run.smithi070.stdout:total 3 2024-04-01T01:49:58.109 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 583636533 Apr 1 01:34 . 2024-04-01T01:49:58.109 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.109 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Apr 1 01:33 1 2024-04-01T01:49:58.109 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Apr 1 01:33 13493 2024-04-01T01:49:58.109 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 567710255 Apr 1 01:40 13494 2024-04-01T01:49:58.109 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:34 pgsql_tmp 2024-04-01T01:49:58.109 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.110 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/base/1: 2024-04-01T01:49:58.110 INFO:teuthology.orchestra.run.smithi070.stdout:total 7778 2024-04-01T01:49:58.110 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Apr 1 01:33 . 2024-04-01T01:49:58.110 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 583636533 Apr 1 01:34 .. 2024-04-01T01:49:58.110 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 112 2024-04-01T01:49:58.110 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 113 2024-04-01T01:49:58.111 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 1 01:33 1247 2024-04-01T01:49:58.111 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1247_fsm 2024-04-01T01:49:58.111 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1247_vm 2024-04-01T01:49:58.111 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 1 01:33 1249 2024-04-01T01:49:58.111 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1249_fsm 2024-04-01T01:49:58.111 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1249_vm 2024-04-01T01:49:58.112 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 1 01:33 1255 2024-04-01T01:49:58.112 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1255_fsm 2024-04-01T01:49:58.112 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1255_vm 2024-04-01T01:49:58.112 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 1 01:33 1259 2024-04-01T01:49:58.112 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1259_fsm 2024-04-01T01:49:58.113 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1259_vm 2024-04-01T01:49:58.113 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 1 01:33 13344 2024-04-01T01:49:58.113 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13344_fsm 2024-04-01T01:49:58.113 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13344_vm 2024-04-01T01:49:58.113 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13346 2024-04-01T01:49:58.113 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13348 2024-04-01T01:49:58.114 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13349 2024-04-01T01:49:58.114 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13349_fsm 2024-04-01T01:49:58.114 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13349_vm 2024-04-01T01:49:58.114 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13351 2024-04-01T01:49:58.114 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13353 2024-04-01T01:49:58.114 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13354 2024-04-01T01:49:58.115 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13354_fsm 2024-04-01T01:49:58.115 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13354_vm 2024-04-01T01:49:58.115 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13356 2024-04-01T01:49:58.115 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13358 2024-04-01T01:49:58.115 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13359 2024-04-01T01:49:58.115 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13359_fsm 2024-04-01T01:49:58.116 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13359_vm 2024-04-01T01:49:58.116 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13361 2024-04-01T01:49:58.116 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13363 2024-04-01T01:49:58.116 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 1417 2024-04-01T01:49:58.116 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 1418 2024-04-01T01:49:58.116 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 174 2024-04-01T01:49:58.117 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 175 2024-04-01T01:49:58.117 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2187 2024-04-01T01:49:58.117 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2224 2024-04-01T01:49:58.117 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2328 2024-04-01T01:49:58.117 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2336 2024-04-01T01:49:58.118 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2337 2024-04-01T01:49:58.118 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2579 2024-04-01T01:49:58.118 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2600 2024-04-01T01:49:58.118 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2600_fsm 2024-04-01T01:49:58.118 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2600_vm 2024-04-01T01:49:58.118 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2601 2024-04-01T01:49:58.119 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2601_fsm 2024-04-01T01:49:58.119 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2601_vm 2024-04-01T01:49:58.119 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2602 2024-04-01T01:49:58.119 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2602_fsm 2024-04-01T01:49:58.119 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2602_vm 2024-04-01T01:49:58.119 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2603 2024-04-01T01:49:58.119 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2603_fsm 2024-04-01T01:49:58.119 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2603_vm 2024-04-01T01:49:58.119 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2604 2024-04-01T01:49:58.119 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2605 2024-04-01T01:49:58.119 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2605_fsm 2024-04-01T01:49:58.120 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2605_vm 2024-04-01T01:49:58.120 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2606 2024-04-01T01:49:58.120 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2606_fsm 2024-04-01T01:49:58.120 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2606_vm 2024-04-01T01:49:58.120 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2607 2024-04-01T01:49:58.120 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2607_fsm 2024-04-01T01:49:58.120 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2607_vm 2024-04-01T01:49:58.120 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 1 01:33 2608 2024-04-01T01:49:58.120 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2608_fsm 2024-04-01T01:49:58.120 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2608_vm 2024-04-01T01:49:58.120 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 1 01:33 2609 2024-04-01T01:49:58.120 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2609_fsm 2024-04-01T01:49:58.120 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2609_vm 2024-04-01T01:49:58.121 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2610 2024-04-01T01:49:58.121 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2610_fsm 2024-04-01T01:49:58.121 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2610_vm 2024-04-01T01:49:58.121 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2611 2024-04-01T01:49:58.121 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2612 2024-04-01T01:49:58.121 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2612_fsm 2024-04-01T01:49:58.121 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2612_vm 2024-04-01T01:49:58.121 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2613 2024-04-01T01:49:58.121 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2615 2024-04-01T01:49:58.121 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2615_fsm 2024-04-01T01:49:58.121 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2615_vm 2024-04-01T01:49:58.121 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2616 2024-04-01T01:49:58.121 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2616_fsm 2024-04-01T01:49:58.122 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2616_vm 2024-04-01T01:49:58.122 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 1 01:33 2617 2024-04-01T01:49:58.122 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2617_fsm 2024-04-01T01:49:58.122 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2617_vm 2024-04-01T01:49:58.122 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 1 01:33 2618 2024-04-01T01:49:58.122 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2618_fsm 2024-04-01T01:49:58.122 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2618_vm 2024-04-01T01:49:58.122 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 1 01:33 2619 2024-04-01T01:49:58.122 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2619_fsm 2024-04-01T01:49:58.122 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2619_vm 2024-04-01T01:49:58.123 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2620 2024-04-01T01:49:58.123 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2650 2024-04-01T01:49:58.123 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2651 2024-04-01T01:49:58.123 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2652 2024-04-01T01:49:58.123 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2653 2024-04-01T01:49:58.123 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2654 2024-04-01T01:49:58.123 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2655 2024-04-01T01:49:58.123 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2656 2024-04-01T01:49:58.123 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2657 2024-04-01T01:49:58.123 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 1 01:33 2658 2024-04-01T01:49:58.123 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 1 01:33 2659 2024-04-01T01:49:58.123 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2660 2024-04-01T01:49:58.123 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2661 2024-04-01T01:49:58.123 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2662 2024-04-01T01:49:58.124 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2663 2024-04-01T01:49:58.124 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2664 2024-04-01T01:49:58.124 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2665 2024-04-01T01:49:58.124 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2666 2024-04-01T01:49:58.124 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2667 2024-04-01T01:49:58.124 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2668 2024-04-01T01:49:58.124 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2669 2024-04-01T01:49:58.124 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2670 2024-04-01T01:49:58.124 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 1 01:33 2673 2024-04-01T01:49:58.124 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 1 01:33 2674 2024-04-01T01:49:58.124 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 1 01:33 2675 2024-04-01T01:49:58.124 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2678 2024-04-01T01:49:58.124 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2679 2024-04-01T01:49:58.124 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2680 2024-04-01T01:49:58.125 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2681 2024-04-01T01:49:58.125 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2682 2024-04-01T01:49:58.125 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2683 2024-04-01T01:49:58.125 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2684 2024-04-01T01:49:58.125 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2685 2024-04-01T01:49:58.125 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2686 2024-04-01T01:49:58.125 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2687 2024-04-01T01:49:58.125 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2688 2024-04-01T01:49:58.125 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2689 2024-04-01T01:49:58.125 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 1 01:33 2690 2024-04-01T01:49:58.125 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 1 01:33 2691 2024-04-01T01:49:58.125 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2692 2024-04-01T01:49:58.125 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2693 2024-04-01T01:49:58.126 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2696 2024-04-01T01:49:58.126 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2699 2024-04-01T01:49:58.126 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2701 2024-04-01T01:49:58.126 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2702 2024-04-01T01:49:58.126 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2703 2024-04-01T01:49:58.126 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2704 2024-04-01T01:49:58.126 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2753 2024-04-01T01:49:58.126 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2753_fsm 2024-04-01T01:49:58.126 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2753_vm 2024-04-01T01:49:58.126 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2754 2024-04-01T01:49:58.126 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2755 2024-04-01T01:49:58.127 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2756 2024-04-01T01:49:58.127 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2757 2024-04-01T01:49:58.127 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2830 2024-04-01T01:49:58.127 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2831 2024-04-01T01:49:58.127 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2832 2024-04-01T01:49:58.127 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2833 2024-04-01T01:49:58.127 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2834 2024-04-01T01:49:58.127 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2835 2024-04-01T01:49:58.127 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2836 2024-04-01T01:49:58.127 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2837 2024-04-01T01:49:58.127 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 1 01:33 2838 2024-04-01T01:49:58.127 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2838_fsm 2024-04-01T01:49:58.127 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2838_vm 2024-04-01T01:49:58.127 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2839 2024-04-01T01:49:58.128 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2840 2024-04-01T01:49:58.128 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2840_fsm 2024-04-01T01:49:58.128 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2840_vm 2024-04-01T01:49:58.128 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2841 2024-04-01T01:49:58.128 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2995 2024-04-01T01:49:58.128 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2996 2024-04-01T01:49:58.128 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3079 2024-04-01T01:49:58.128 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3079_fsm 2024-04-01T01:49:58.128 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3079_vm 2024-04-01T01:49:58.128 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3080 2024-04-01T01:49:58.128 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3081 2024-04-01T01:49:58.128 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 3085 2024-04-01T01:49:58.128 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3118 2024-04-01T01:49:58.128 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3119 2024-04-01T01:49:58.129 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 1 01:33 3164 2024-04-01T01:49:58.129 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3256 2024-04-01T01:49:58.129 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3257 2024-04-01T01:49:58.129 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3258 2024-04-01T01:49:58.129 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3350 2024-04-01T01:49:58.129 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3351 2024-04-01T01:49:58.129 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3379 2024-04-01T01:49:58.129 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3380 2024-04-01T01:49:58.129 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3381 2024-04-01T01:49:58.129 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3394 2024-04-01T01:49:58.129 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3394_fsm 2024-04-01T01:49:58.129 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3394_vm 2024-04-01T01:49:58.129 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3395 2024-04-01T01:49:58.129 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3429 2024-04-01T01:49:58.129 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3430 2024-04-01T01:49:58.130 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3431 2024-04-01T01:49:58.130 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3433 2024-04-01T01:49:58.130 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3439 2024-04-01T01:49:58.130 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3440 2024-04-01T01:49:58.130 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 3455 2024-04-01T01:49:58.130 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 1 01:33 3456 2024-04-01T01:49:58.130 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3456_fsm 2024-04-01T01:49:58.130 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3456_vm 2024-04-01T01:49:58.130 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3466 2024-04-01T01:49:58.130 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3467 2024-04-01T01:49:58.130 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3468 2024-04-01T01:49:58.130 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3501 2024-04-01T01:49:58.130 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3502 2024-04-01T01:49:58.130 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3503 2024-04-01T01:49:58.131 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3534 2024-04-01T01:49:58.131 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3541 2024-04-01T01:49:58.131 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3541_fsm 2024-04-01T01:49:58.131 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3541_vm 2024-04-01T01:49:58.131 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3542 2024-04-01T01:49:58.131 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3574 2024-04-01T01:49:58.131 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3575 2024-04-01T01:49:58.131 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3576 2024-04-01T01:49:58.131 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3596 2024-04-01T01:49:58.131 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3597 2024-04-01T01:49:58.131 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3598 2024-04-01T01:49:58.131 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3599 2024-04-01T01:49:58.131 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3600 2024-04-01T01:49:58.131 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3600_fsm 2024-04-01T01:49:58.132 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3600_vm 2024-04-01T01:49:58.132 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3601 2024-04-01T01:49:58.132 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3601_fsm 2024-04-01T01:49:58.132 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3601_vm 2024-04-01T01:49:58.132 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3602 2024-04-01T01:49:58.132 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3602_fsm 2024-04-01T01:49:58.132 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3602_vm 2024-04-01T01:49:58.132 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3603 2024-04-01T01:49:58.132 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3603_fsm 2024-04-01T01:49:58.132 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3603_vm 2024-04-01T01:49:58.132 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3604 2024-04-01T01:49:58.132 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3605 2024-04-01T01:49:58.132 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3606 2024-04-01T01:49:58.132 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3607 2024-04-01T01:49:58.132 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3608 2024-04-01T01:49:58.133 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 3609 2024-04-01T01:49:58.133 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3712 2024-04-01T01:49:58.133 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3764 2024-04-01T01:49:58.133 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3764_fsm 2024-04-01T01:49:58.133 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3764_vm 2024-04-01T01:49:58.133 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3766 2024-04-01T01:49:58.133 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3767 2024-04-01T01:49:58.133 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3997 2024-04-01T01:49:58.133 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4143 2024-04-01T01:49:58.133 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4144 2024-04-01T01:49:58.133 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4145 2024-04-01T01:49:58.133 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4146 2024-04-01T01:49:58.133 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4147 2024-04-01T01:49:58.133 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4148 2024-04-01T01:49:58.133 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4149 2024-04-01T01:49:58.134 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4150 2024-04-01T01:49:58.134 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4151 2024-04-01T01:49:58.313 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4152 2024-04-01T01:49:58.313 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4153 2024-04-01T01:49:58.313 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4154 2024-04-01T01:49:58.313 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4155 2024-04-01T01:49:58.314 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4156 2024-04-01T01:49:58.314 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4157 2024-04-01T01:49:58.314 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4158 2024-04-01T01:49:58.314 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4159 2024-04-01T01:49:58.314 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4160 2024-04-01T01:49:58.314 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4161 2024-04-01T01:49:58.314 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4162 2024-04-01T01:49:58.314 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4163 2024-04-01T01:49:58.314 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4164 2024-04-01T01:49:58.314 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4165 2024-04-01T01:49:58.314 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4166 2024-04-01T01:49:58.314 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4167 2024-04-01T01:49:58.314 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4168 2024-04-01T01:49:58.314 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4169 2024-04-01T01:49:58.314 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4170 2024-04-01T01:49:58.315 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4171 2024-04-01T01:49:58.315 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4172 2024-04-01T01:49:58.315 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4173 2024-04-01T01:49:58.315 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4174 2024-04-01T01:49:58.315 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 5002 2024-04-01T01:49:58.315 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 548 2024-04-01T01:49:58.315 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 549 2024-04-01T01:49:58.315 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6102 2024-04-01T01:49:58.315 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6104 2024-04-01T01:49:58.315 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6106 2024-04-01T01:49:58.315 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6110 2024-04-01T01:49:58.315 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6111 2024-04-01T01:49:58.315 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6112 2024-04-01T01:49:58.315 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6113 2024-04-01T01:49:58.316 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6117 2024-04-01T01:49:58.316 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 826 2024-04-01T01:49:58.316 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 827 2024-04-01T01:49:58.316 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 828 2024-04-01T01:49:58.316 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 1 01:33 pg_filenode.map 2024-04-01T01:49:58.316 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 1 01:33 PG_VERSION 2024-04-01T01:49:58.316 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.316 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-04-01T01:49:58.316 INFO:teuthology.orchestra.run.smithi070.stdout:total 7778 2024-04-01T01:49:58.316 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 7963139 Apr 1 01:33 . 2024-04-01T01:49:58.316 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 583636533 Apr 1 01:34 .. 2024-04-01T01:49:58.316 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 112 2024-04-01T01:49:58.316 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 113 2024-04-01T01:49:58.316 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 1 01:33 1247 2024-04-01T01:49:58.316 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1247_fsm 2024-04-01T01:49:58.316 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1247_vm 2024-04-01T01:49:58.317 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 1 01:33 1249 2024-04-01T01:49:58.317 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1249_fsm 2024-04-01T01:49:58.317 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1249_vm 2024-04-01T01:49:58.317 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 1 01:33 1255 2024-04-01T01:49:58.317 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1255_fsm 2024-04-01T01:49:58.317 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1255_vm 2024-04-01T01:49:58.317 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 1 01:33 1259 2024-04-01T01:49:58.317 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1259_fsm 2024-04-01T01:49:58.317 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1259_vm 2024-04-01T01:49:58.317 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 1 01:33 13344 2024-04-01T01:49:58.317 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13344_fsm 2024-04-01T01:49:58.317 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13344_vm 2024-04-01T01:49:58.318 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13346 2024-04-01T01:49:58.318 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13348 2024-04-01T01:49:58.318 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13349 2024-04-01T01:49:58.318 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13349_fsm 2024-04-01T01:49:58.318 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13349_vm 2024-04-01T01:49:58.318 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13351 2024-04-01T01:49:58.318 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13353 2024-04-01T01:49:58.318 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13354 2024-04-01T01:49:58.318 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13354_fsm 2024-04-01T01:49:58.318 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13354_vm 2024-04-01T01:49:58.318 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13356 2024-04-01T01:49:58.318 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13358 2024-04-01T01:49:58.318 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13359 2024-04-01T01:49:58.318 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13359_fsm 2024-04-01T01:49:58.318 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13359_vm 2024-04-01T01:49:58.319 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13361 2024-04-01T01:49:58.319 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13363 2024-04-01T01:49:58.319 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 1417 2024-04-01T01:49:58.319 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 1418 2024-04-01T01:49:58.319 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 174 2024-04-01T01:49:58.319 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 175 2024-04-01T01:49:58.319 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2187 2024-04-01T01:49:58.319 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2224 2024-04-01T01:49:58.319 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2328 2024-04-01T01:49:58.319 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2336 2024-04-01T01:49:58.319 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2337 2024-04-01T01:49:58.319 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2579 2024-04-01T01:49:58.319 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2600 2024-04-01T01:49:58.319 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2600_fsm 2024-04-01T01:49:58.319 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2600_vm 2024-04-01T01:49:58.320 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2601 2024-04-01T01:49:58.320 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2601_fsm 2024-04-01T01:49:58.320 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2601_vm 2024-04-01T01:49:58.320 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2602 2024-04-01T01:49:58.320 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2602_fsm 2024-04-01T01:49:58.320 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2602_vm 2024-04-01T01:49:58.320 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2603 2024-04-01T01:49:58.320 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2603_fsm 2024-04-01T01:49:58.320 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2603_vm 2024-04-01T01:49:58.320 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2604 2024-04-01T01:49:58.320 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2605 2024-04-01T01:49:58.320 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2605_fsm 2024-04-01T01:49:58.320 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2605_vm 2024-04-01T01:49:58.320 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2606 2024-04-01T01:49:58.320 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2606_fsm 2024-04-01T01:49:58.321 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2606_vm 2024-04-01T01:49:58.321 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2607 2024-04-01T01:49:58.321 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2607_fsm 2024-04-01T01:49:58.321 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2607_vm 2024-04-01T01:49:58.321 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 1 01:33 2608 2024-04-01T01:49:58.321 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2608_fsm 2024-04-01T01:49:58.321 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2608_vm 2024-04-01T01:49:58.321 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 1 01:33 2609 2024-04-01T01:49:58.321 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2609_fsm 2024-04-01T01:49:58.321 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2609_vm 2024-04-01T01:49:58.322 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2610 2024-04-01T01:49:58.322 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2610_fsm 2024-04-01T01:49:58.322 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2610_vm 2024-04-01T01:49:58.322 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2611 2024-04-01T01:49:58.322 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2612 2024-04-01T01:49:58.322 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2612_fsm 2024-04-01T01:49:58.322 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2612_vm 2024-04-01T01:49:58.322 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2613 2024-04-01T01:49:58.322 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2615 2024-04-01T01:49:58.322 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2615_fsm 2024-04-01T01:49:58.322 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2615_vm 2024-04-01T01:49:58.322 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2616 2024-04-01T01:49:58.322 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2616_fsm 2024-04-01T01:49:58.322 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2616_vm 2024-04-01T01:49:58.322 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 1 01:33 2617 2024-04-01T01:49:58.323 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2617_fsm 2024-04-01T01:49:58.323 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2617_vm 2024-04-01T01:49:58.323 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 1 01:33 2618 2024-04-01T01:49:58.323 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2618_fsm 2024-04-01T01:49:58.323 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2618_vm 2024-04-01T01:49:58.323 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Apr 1 01:33 2619 2024-04-01T01:49:58.323 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2619_fsm 2024-04-01T01:49:58.323 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2619_vm 2024-04-01T01:49:58.323 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2620 2024-04-01T01:49:58.323 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2650 2024-04-01T01:49:58.323 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2651 2024-04-01T01:49:58.323 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2652 2024-04-01T01:49:58.324 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2653 2024-04-01T01:49:58.324 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2654 2024-04-01T01:49:58.324 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2655 2024-04-01T01:49:58.324 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2656 2024-04-01T01:49:58.324 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2657 2024-04-01T01:49:58.324 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 1 01:33 2658 2024-04-01T01:49:58.324 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 1 01:33 2659 2024-04-01T01:49:58.324 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2660 2024-04-01T01:49:58.324 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2661 2024-04-01T01:49:58.324 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2662 2024-04-01T01:49:58.324 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2663 2024-04-01T01:49:58.324 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2664 2024-04-01T01:49:58.324 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2665 2024-04-01T01:49:58.324 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2666 2024-04-01T01:49:58.324 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2667 2024-04-01T01:49:58.325 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2668 2024-04-01T01:49:58.325 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2669 2024-04-01T01:49:58.325 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2670 2024-04-01T01:49:58.325 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 1 01:33 2673 2024-04-01T01:49:58.325 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 1 01:33 2674 2024-04-01T01:49:58.325 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 1 01:33 2675 2024-04-01T01:49:58.325 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2678 2024-04-01T01:49:58.325 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2679 2024-04-01T01:49:58.325 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2680 2024-04-01T01:49:58.325 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2681 2024-04-01T01:49:58.325 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2682 2024-04-01T01:49:58.325 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2683 2024-04-01T01:49:58.325 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2684 2024-04-01T01:49:58.325 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2685 2024-04-01T01:49:58.325 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2686 2024-04-01T01:49:58.326 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2687 2024-04-01T01:49:58.326 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2688 2024-04-01T01:49:58.326 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2689 2024-04-01T01:49:58.326 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 1 01:33 2690 2024-04-01T01:49:58.326 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 1 01:33 2691 2024-04-01T01:49:58.326 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2692 2024-04-01T01:49:58.326 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2693 2024-04-01T01:49:58.326 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2696 2024-04-01T01:49:58.326 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2699 2024-04-01T01:49:58.326 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2701 2024-04-01T01:49:58.326 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2702 2024-04-01T01:49:58.326 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2703 2024-04-01T01:49:58.326 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2704 2024-04-01T01:49:58.326 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2753 2024-04-01T01:49:58.327 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2753_fsm 2024-04-01T01:49:58.327 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2753_vm 2024-04-01T01:49:58.327 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2754 2024-04-01T01:49:58.327 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2755 2024-04-01T01:49:58.327 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2756 2024-04-01T01:49:58.327 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2757 2024-04-01T01:49:58.327 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2830 2024-04-01T01:49:58.327 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2831 2024-04-01T01:49:58.327 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2832 2024-04-01T01:49:58.327 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2833 2024-04-01T01:49:58.327 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2834 2024-04-01T01:49:58.327 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2835 2024-04-01T01:49:58.327 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2836 2024-04-01T01:49:58.327 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2837 2024-04-01T01:49:58.327 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 1 01:33 2838 2024-04-01T01:49:58.328 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2838_fsm 2024-04-01T01:49:58.328 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2838_vm 2024-04-01T01:49:58.328 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2839 2024-04-01T01:49:58.328 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2840 2024-04-01T01:49:58.328 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2840_fsm 2024-04-01T01:49:58.328 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2840_vm 2024-04-01T01:49:58.328 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2841 2024-04-01T01:49:58.328 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2995 2024-04-01T01:49:58.328 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2996 2024-04-01T01:49:58.328 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3079 2024-04-01T01:49:58.328 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3079_fsm 2024-04-01T01:49:58.328 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3079_vm 2024-04-01T01:49:58.328 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3080 2024-04-01T01:49:58.328 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3081 2024-04-01T01:49:58.329 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 3085 2024-04-01T01:49:58.329 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3118 2024-04-01T01:49:58.329 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3119 2024-04-01T01:49:58.329 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 1 01:33 3164 2024-04-01T01:49:58.329 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3256 2024-04-01T01:49:58.329 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3257 2024-04-01T01:49:58.329 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3258 2024-04-01T01:49:58.329 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3350 2024-04-01T01:49:58.329 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3351 2024-04-01T01:49:58.329 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3379 2024-04-01T01:49:58.329 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3380 2024-04-01T01:49:58.330 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3381 2024-04-01T01:49:58.330 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3394 2024-04-01T01:49:58.330 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3394_fsm 2024-04-01T01:49:58.330 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3394_vm 2024-04-01T01:49:58.330 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3395 2024-04-01T01:49:58.330 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3429 2024-04-01T01:49:58.330 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3430 2024-04-01T01:49:58.330 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3431 2024-04-01T01:49:58.330 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3433 2024-04-01T01:49:58.330 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3439 2024-04-01T01:49:58.330 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3440 2024-04-01T01:49:58.330 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 3455 2024-04-01T01:49:58.330 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 1 01:33 3456 2024-04-01T01:49:58.330 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3456_fsm 2024-04-01T01:49:58.330 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3456_vm 2024-04-01T01:49:58.331 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3466 2024-04-01T01:49:58.331 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3467 2024-04-01T01:49:58.331 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3468 2024-04-01T01:49:58.331 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3501 2024-04-01T01:49:58.331 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3502 2024-04-01T01:49:58.331 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3503 2024-04-01T01:49:58.331 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3534 2024-04-01T01:49:58.331 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3541 2024-04-01T01:49:58.331 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3541_fsm 2024-04-01T01:49:58.331 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3541_vm 2024-04-01T01:49:58.331 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3542 2024-04-01T01:49:58.331 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3574 2024-04-01T01:49:58.332 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3575 2024-04-01T01:49:58.332 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3576 2024-04-01T01:49:58.332 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3596 2024-04-01T01:49:58.332 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3597 2024-04-01T01:49:58.332 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3598 2024-04-01T01:49:58.332 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3599 2024-04-01T01:49:58.332 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3600 2024-04-01T01:49:58.332 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3600_fsm 2024-04-01T01:49:58.332 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3600_vm 2024-04-01T01:49:58.332 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3601 2024-04-01T01:49:58.332 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3601_fsm 2024-04-01T01:49:58.332 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3601_vm 2024-04-01T01:49:58.332 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3602 2024-04-01T01:49:58.333 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3602_fsm 2024-04-01T01:49:58.333 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3602_vm 2024-04-01T01:49:58.333 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3603 2024-04-01T01:49:58.333 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3603_fsm 2024-04-01T01:49:58.333 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3603_vm 2024-04-01T01:49:58.333 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3604 2024-04-01T01:49:58.333 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3605 2024-04-01T01:49:58.333 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3606 2024-04-01T01:49:58.333 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3607 2024-04-01T01:49:58.333 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3608 2024-04-01T01:49:58.333 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 3609 2024-04-01T01:49:58.333 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3712 2024-04-01T01:49:58.333 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3764 2024-04-01T01:49:58.333 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3764_fsm 2024-04-01T01:49:58.333 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3764_vm 2024-04-01T01:49:58.334 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3766 2024-04-01T01:49:58.334 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3767 2024-04-01T01:49:58.334 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3997 2024-04-01T01:49:58.334 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4143 2024-04-01T01:49:58.334 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4144 2024-04-01T01:49:58.334 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4145 2024-04-01T01:49:58.334 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4146 2024-04-01T01:49:58.334 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4147 2024-04-01T01:49:58.334 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4148 2024-04-01T01:49:58.334 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4149 2024-04-01T01:49:58.334 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4150 2024-04-01T01:49:58.334 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4151 2024-04-01T01:49:58.334 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4152 2024-04-01T01:49:58.334 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4153 2024-04-01T01:49:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4154 2024-04-01T01:49:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4155 2024-04-01T01:49:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4156 2024-04-01T01:49:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4157 2024-04-01T01:49:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4158 2024-04-01T01:49:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4159 2024-04-01T01:49:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4160 2024-04-01T01:49:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4161 2024-04-01T01:49:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4162 2024-04-01T01:49:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4163 2024-04-01T01:49:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4164 2024-04-01T01:49:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4165 2024-04-01T01:49:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4166 2024-04-01T01:49:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4167 2024-04-01T01:49:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4168 2024-04-01T01:49:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4169 2024-04-01T01:49:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4170 2024-04-01T01:49:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4171 2024-04-01T01:49:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4172 2024-04-01T01:49:58.572 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4173 2024-04-01T01:49:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4174 2024-04-01T01:49:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 5002 2024-04-01T01:49:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 548 2024-04-01T01:49:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 549 2024-04-01T01:49:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6102 2024-04-01T01:49:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6104 2024-04-01T01:49:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6106 2024-04-01T01:49:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6110 2024-04-01T01:49:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6111 2024-04-01T01:49:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6112 2024-04-01T01:49:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6113 2024-04-01T01:49:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6117 2024-04-01T01:49:58.573 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 826 2024-04-01T01:49:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 827 2024-04-01T01:49:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 828 2024-04-01T01:49:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 1 01:33 pg_filenode.map 2024-04-01T01:49:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 1 01:33 PG_VERSION 2024-04-01T01:49:58.574 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-04-01T01:49:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:total 554407 2024-04-01T01:49:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 567710255 Apr 1 01:40 . 2024-04-01T01:49:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 583636533 Apr 1 01:34 .. 2024-04-01T01:49:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 112 2024-04-01T01:49:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 113 2024-04-01T01:49:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 1 01:35 1247 2024-04-01T01:49:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1247_fsm 2024-04-01T01:49:58.574 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:34 1247_vm 2024-04-01T01:49:58.575 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Apr 1 01:35 1249 2024-04-01T01:49:58.575 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:34 1249_fsm 2024-04-01T01:49:58.575 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:34 1249_vm 2024-04-01T01:49:58.575 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Apr 1 01:33 1255 2024-04-01T01:49:58.575 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1255_fsm 2024-04-01T01:49:58.575 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1255_vm 2024-04-01T01:49:58.575 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Apr 1 01:49 1259 2024-04-01T01:49:58.575 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1259_fsm 2024-04-01T01:49:58.575 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:34 1259_vm 2024-04-01T01:49:58.575 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Apr 1 01:33 13344 2024-04-01T01:49:58.575 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13344_fsm 2024-04-01T01:49:58.575 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13344_vm 2024-04-01T01:49:58.575 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13346 2024-04-01T01:49:58.575 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13348 2024-04-01T01:49:58.575 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13349 2024-04-01T01:49:58.576 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13349_fsm 2024-04-01T01:49:58.576 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13349_vm 2024-04-01T01:49:58.576 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13351 2024-04-01T01:49:58.576 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13353 2024-04-01T01:49:58.576 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13354 2024-04-01T01:49:58.576 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13354_fsm 2024-04-01T01:49:58.576 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13354_vm 2024-04-01T01:49:58.576 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13356 2024-04-01T01:49:58.576 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13358 2024-04-01T01:49:58.576 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13359 2024-04-01T01:49:58.576 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 13359_fsm 2024-04-01T01:49:58.576 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13359_vm 2024-04-01T01:49:58.576 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 13361 2024-04-01T01:49:58.576 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 13363 2024-04-01T01:49:58.577 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 1417 2024-04-01T01:49:58.577 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 1418 2024-04-01T01:49:58.577 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 439402496 Apr 1 01:49 16396 2024-04-01T01:49:58.577 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Apr 1 01:49 16396_fsm 2024-04-01T01:49:58.577 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:47 16396_vm 2024-04-01T01:49:58.577 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1507328 Apr 1 01:49 16397 2024-04-01T01:49:58.577 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:49 16397_fsm 2024-04-01T01:49:58.577 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:49 16397_vm 2024-04-01T01:49:58.577 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4153344 Apr 1 01:49 16399 2024-04-01T01:49:58.577 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:49 16399_fsm 2024-04-01T01:49:58.577 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:48 16399_vm 2024-04-01T01:49:58.577 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:49 16400 2024-04-01T01:49:58.577 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:49 16402 2024-04-01T01:49:58.577 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 71901184 Apr 1 01:49 16404 2024-04-01T01:49:58.578 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 42287104 Apr 1 01:49 16406 2024-04-01T01:49:58.578 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:49 16406_fsm 2024-04-01T01:49:58.578 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:49 16406_vm 2024-04-01T01:49:58.578 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 174 2024-04-01T01:49:58.578 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 175 2024-04-01T01:49:58.578 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2187 2024-04-01T01:49:58.578 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2224 2024-04-01T01:49:58.578 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2328 2024-04-01T01:49:58.578 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2336 2024-04-01T01:49:58.578 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2337 2024-04-01T01:49:58.578 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:34 2579 2024-04-01T01:49:58.578 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2600 2024-04-01T01:49:58.578 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2600_fsm 2024-04-01T01:49:58.578 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2600_vm 2024-04-01T01:49:58.579 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2601 2024-04-01T01:49:58.579 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2601_fsm 2024-04-01T01:49:58.579 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2601_vm 2024-04-01T01:49:58.579 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2602 2024-04-01T01:49:58.579 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2602_fsm 2024-04-01T01:49:58.579 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2602_vm 2024-04-01T01:49:58.579 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2603 2024-04-01T01:49:58.579 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2603_fsm 2024-04-01T01:49:58.579 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2603_vm 2024-04-01T01:49:58.579 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2604 2024-04-01T01:49:58.579 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2605 2024-04-01T01:49:58.579 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2605_fsm 2024-04-01T01:49:58.579 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2605_vm 2024-04-01T01:49:58.579 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:34 2606 2024-04-01T01:49:58.579 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2606_fsm 2024-04-01T01:49:58.580 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:34 2606_vm 2024-04-01T01:49:58.580 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2607 2024-04-01T01:49:58.580 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2607_fsm 2024-04-01T01:49:58.580 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2607_vm 2024-04-01T01:49:58.580 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Apr 1 01:35 2608 2024-04-01T01:49:58.580 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2608_fsm 2024-04-01T01:49:58.580 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:34 2608_vm 2024-04-01T01:49:58.580 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Apr 1 01:33 2609 2024-04-01T01:49:58.580 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2609_fsm 2024-04-01T01:49:58.580 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2609_vm 2024-04-01T01:49:58.580 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:35 2610 2024-04-01T01:49:58.580 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2610_fsm 2024-04-01T01:49:58.580 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:34 2610_vm 2024-04-01T01:49:58.580 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2611 2024-04-01T01:49:58.580 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2612 2024-04-01T01:49:58.581 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2612_fsm 2024-04-01T01:49:58.581 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2612_vm 2024-04-01T01:49:58.581 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2613 2024-04-01T01:49:58.581 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2615 2024-04-01T01:49:58.581 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2615_fsm 2024-04-01T01:49:58.581 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2615_vm 2024-04-01T01:49:58.581 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2616 2024-04-01T01:49:58.581 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2616_fsm 2024-04-01T01:49:58.581 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2616_vm 2024-04-01T01:49:58.581 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Apr 1 01:33 2617 2024-04-01T01:49:58.581 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2617_fsm 2024-04-01T01:49:58.581 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2617_vm 2024-04-01T01:49:58.581 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Apr 1 01:33 2618 2024-04-01T01:49:58.581 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2618_fsm 2024-04-01T01:49:58.581 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2618_vm 2024-04-01T01:49:58.582 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 163840 Apr 1 01:49 2619 2024-04-01T01:49:58.582 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:35 2619_fsm 2024-04-01T01:49:58.582 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:34 2619_vm 2024-04-01T01:49:58.582 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2620 2024-04-01T01:49:58.582 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2650 2024-04-01T01:49:58.582 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2651 2024-04-01T01:49:58.582 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2652 2024-04-01T01:49:58.582 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2653 2024-04-01T01:49:58.582 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2654 2024-04-01T01:49:58.582 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2655 2024-04-01T01:49:58.582 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2656 2024-04-01T01:49:58.582 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2657 2024-04-01T01:49:58.582 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Apr 1 01:35 2658 2024-04-01T01:49:58.582 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Apr 1 01:35 2659 2024-04-01T01:49:58.583 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2660 2024-04-01T01:49:58.583 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2661 2024-04-01T01:49:58.583 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:35 2662 2024-04-01T01:49:58.583 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:35 2663 2024-04-01T01:49:58.583 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:35 2664 2024-04-01T01:49:58.583 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:35 2665 2024-04-01T01:49:58.583 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:34 2666 2024-04-01T01:49:58.583 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:35 2667 2024-04-01T01:49:58.583 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2668 2024-04-01T01:49:58.583 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2669 2024-04-01T01:49:58.583 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2670 2024-04-01T01:49:58.583 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Apr 1 01:35 2673 2024-04-01T01:49:58.583 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Apr 1 01:35 2674 2024-04-01T01:49:58.583 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Apr 1 01:33 2675 2024-04-01T01:49:58.583 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:35 2678 2024-04-01T01:49:58.584 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:35 2679 2024-04-01T01:49:58.584 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2680 2024-04-01T01:49:58.584 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2681 2024-04-01T01:49:58.584 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2682 2024-04-01T01:49:58.584 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2683 2024-04-01T01:49:58.584 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2684 2024-04-01T01:49:58.584 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2685 2024-04-01T01:49:58.584 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2686 2024-04-01T01:49:58.584 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2687 2024-04-01T01:49:58.584 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 2688 2024-04-01T01:49:58.584 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Apr 1 01:33 2689 2024-04-01T01:49:58.584 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Apr 1 01:33 2690 2024-04-01T01:49:58.584 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Apr 1 01:33 2691 2024-04-01T01:49:58.584 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2692 2024-04-01T01:49:58.584 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2693 2024-04-01T01:49:58.585 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:37 2696 2024-04-01T01:49:58.585 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2699 2024-04-01T01:49:58.585 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2701 2024-04-01T01:49:58.585 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2702 2024-04-01T01:49:58.585 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:35 2703 2024-04-01T01:49:58.585 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:35 2704 2024-04-01T01:49:58.585 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2753 2024-04-01T01:49:58.585 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2753_fsm 2024-04-01T01:49:58.585 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2753_vm 2024-04-01T01:49:58.585 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2754 2024-04-01T01:49:58.585 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2755 2024-04-01T01:49:58.585 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2756 2024-04-01T01:49:58.585 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 2757 2024-04-01T01:49:58.585 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2830 2024-04-01T01:49:58.585 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2831 2024-04-01T01:49:58.586 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2832 2024-04-01T01:49:58.586 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2833 2024-04-01T01:49:58.586 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2834 2024-04-01T01:49:58.586 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2835 2024-04-01T01:49:58.586 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2836 2024-04-01T01:49:58.586 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2837 2024-04-01T01:49:58.586 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Apr 1 01:33 2838 2024-04-01T01:49:58.586 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2838_fsm 2024-04-01T01:49:58.586 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2838_vm 2024-04-01T01:49:58.586 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2839 2024-04-01T01:49:58.586 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2840 2024-04-01T01:49:58.586 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2840_fsm 2024-04-01T01:49:58.586 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2840_vm 2024-04-01T01:49:58.586 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2841 2024-04-01T01:49:58.586 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2995 2024-04-01T01:49:58.587 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2996 2024-04-01T01:49:58.587 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3079 2024-04-01T01:49:58.587 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3079_fsm 2024-04-01T01:49:58.587 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3079_vm 2024-04-01T01:49:58.587 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3080 2024-04-01T01:49:58.587 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3081 2024-04-01T01:49:58.587 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:33 3085 2024-04-01T01:49:58.587 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3118 2024-04-01T01:49:58.587 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3119 2024-04-01T01:49:58.587 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Apr 1 01:33 3164 2024-04-01T01:49:58.587 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3256 2024-04-01T01:49:58.587 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3257 2024-04-01T01:49:58.587 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3258 2024-04-01T01:49:58.587 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3350 2024-04-01T01:49:58.587 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3351 2024-04-01T01:49:58.587 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3379 2024-04-01T01:49:58.588 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3380 2024-04-01T01:49:58.588 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3381 2024-04-01T01:49:58.588 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3394 2024-04-01T01:49:58.588 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3394_fsm 2024-04-01T01:49:58.588 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3394_vm 2024-04-01T01:49:58.588 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3395 2024-04-01T01:49:58.588 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3429 2024-04-01T01:49:58.588 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3430 2024-04-01T01:49:58.588 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3431 2024-04-01T01:49:58.588 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3433 2024-04-01T01:49:58.588 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3439 2024-04-01T01:49:58.588 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3440 2024-04-01T01:49:58.588 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Apr 1 01:35 3455 2024-04-01T01:49:58.588 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Apr 1 01:33 3456 2024-04-01T01:49:58.588 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3456_fsm 2024-04-01T01:49:58.589 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3456_vm 2024-04-01T01:49:58.589 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3466 2024-04-01T01:49:58.589 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3467 2024-04-01T01:49:58.589 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3468 2024-04-01T01:49:58.589 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3501 2024-04-01T01:49:58.589 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3502 2024-04-01T01:49:58.589 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3503 2024-04-01T01:49:58.589 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3534 2024-04-01T01:49:58.589 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3541 2024-04-01T01:49:58.589 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3541_fsm 2024-04-01T01:49:58.589 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3541_vm 2024-04-01T01:49:58.589 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3542 2024-04-01T01:49:58.589 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3574 2024-04-01T01:49:58.589 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3575 2024-04-01T01:49:58.589 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3576 2024-04-01T01:49:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3596 2024-04-01T01:49:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3597 2024-04-01T01:49:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3598 2024-04-01T01:49:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3599 2024-04-01T01:49:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3600 2024-04-01T01:49:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3600_fsm 2024-04-01T01:49:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3600_vm 2024-04-01T01:49:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3601 2024-04-01T01:49:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3601_fsm 2024-04-01T01:49:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3601_vm 2024-04-01T01:49:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3602 2024-04-01T01:49:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3602_fsm 2024-04-01T01:49:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3602_vm 2024-04-01T01:49:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3603 2024-04-01T01:49:58.590 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3603_fsm 2024-04-01T01:49:58.591 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3603_vm 2024-04-01T01:49:58.591 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3604 2024-04-01T01:49:58.591 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3605 2024-04-01T01:49:58.591 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3606 2024-04-01T01:49:58.591 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3607 2024-04-01T01:49:58.591 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3608 2024-04-01T01:49:58.591 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Apr 1 01:33 3609 2024-04-01T01:49:58.591 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3712 2024-04-01T01:49:58.591 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3764 2024-04-01T01:49:58.591 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 3764_fsm 2024-04-01T01:49:58.591 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3764_vm 2024-04-01T01:49:58.591 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3766 2024-04-01T01:49:58.591 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 3767 2024-04-01T01:49:58.591 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3997 2024-04-01T01:49:58.591 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4143 2024-04-01T01:49:58.592 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4144 2024-04-01T01:49:58.592 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4145 2024-04-01T01:49:58.592 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4146 2024-04-01T01:49:58.592 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4147 2024-04-01T01:49:58.592 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4148 2024-04-01T01:49:58.592 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4149 2024-04-01T01:49:58.592 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4150 2024-04-01T01:49:58.592 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4151 2024-04-01T01:49:58.592 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4152 2024-04-01T01:49:58.592 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4153 2024-04-01T01:49:58.592 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4154 2024-04-01T01:49:58.592 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4155 2024-04-01T01:49:58.592 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4156 2024-04-01T01:49:58.592 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4157 2024-04-01T01:49:58.592 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4158 2024-04-01T01:49:58.593 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4159 2024-04-01T01:49:58.593 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4160 2024-04-01T01:49:58.593 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4161 2024-04-01T01:49:58.593 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4162 2024-04-01T01:49:58.593 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4163 2024-04-01T01:49:58.593 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4164 2024-04-01T01:49:58.593 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4165 2024-04-01T01:49:58.593 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4166 2024-04-01T01:49:58.593 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4167 2024-04-01T01:49:58.593 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4168 2024-04-01T01:49:58.593 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4169 2024-04-01T01:49:58.593 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4170 2024-04-01T01:49:58.593 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4171 2024-04-01T01:49:58.593 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4172 2024-04-01T01:49:58.593 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4173 2024-04-01T01:49:58.594 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4174 2024-04-01T01:49:58.594 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 5002 2024-04-01T01:49:58.594 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 548 2024-04-01T01:49:58.594 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 549 2024-04-01T01:49:58.594 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6102 2024-04-01T01:49:58.594 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6104 2024-04-01T01:49:58.594 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6106 2024-04-01T01:49:58.594 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6110 2024-04-01T01:49:58.611 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6111 2024-04-01T01:49:58.611 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6112 2024-04-01T01:49:58.611 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6113 2024-04-01T01:49:58.611 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6117 2024-04-01T01:49:58.611 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 826 2024-04-01T01:49:58.611 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 827 2024-04-01T01:49:58.612 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 828 2024-04-01T01:49:58.612 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 1 01:33 pg_filenode.map 2024-04-01T01:49:58.612 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 151596 Apr 1 01:34 pg_internal.init 2024-04-01T01:49:58.612 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Apr 1 01:33 PG_VERSION 2024-04-01T01:49:58.612 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.612 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/base/pgsql_tmp: 2024-04-01T01:49:58.612 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:49:58.612 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:34 . 2024-04-01T01:49:58.612 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 583636533 Apr 1 01:34 .. 2024-04-01T01:49:58.612 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.612 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/global: 2024-04-01T01:49:58.612 INFO:teuthology.orchestra.run.smithi070.stdout:total 561 2024-04-01T01:49:58.612 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 572520 Apr 1 01:34 . 2024-04-01T01:49:58.612 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.613 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1213 2024-04-01T01:49:58.613 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1213_fsm 2024-04-01T01:49:58.613 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1213_vm 2024-04-01T01:49:58.613 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1214 2024-04-01T01:49:58.613 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1214_fsm 2024-04-01T01:49:58.613 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1214_vm 2024-04-01T01:49:58.613 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 1232 2024-04-01T01:49:58.613 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 1233 2024-04-01T01:49:58.613 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1260 2024-04-01T01:49:58.613 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1260_fsm 2024-04-01T01:49:58.613 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1260_vm 2024-04-01T01:49:58.613 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1261 2024-04-01T01:49:58.613 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1261_fsm 2024-04-01T01:49:58.613 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1261_vm 2024-04-01T01:49:58.613 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1262 2024-04-01T01:49:58.614 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 1262_fsm 2024-04-01T01:49:58.614 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 1262_vm 2024-04-01T01:49:58.614 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2396 2024-04-01T01:49:58.614 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Apr 1 01:33 2396_fsm 2024-04-01T01:49:58.614 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2396_vm 2024-04-01T01:49:58.614 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2397 2024-04-01T01:49:58.614 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2671 2024-04-01T01:49:58.614 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2672 2024-04-01T01:49:58.614 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2676 2024-04-01T01:49:58.614 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2677 2024-04-01T01:49:58.614 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2694 2024-04-01T01:49:58.625 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2695 2024-04-01T01:49:58.625 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2697 2024-04-01T01:49:58.625 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 2698 2024-04-01T01:49:58.625 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2846 2024-04-01T01:49:58.625 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2847 2024-04-01T01:49:58.625 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2964 2024-04-01T01:49:58.625 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2965 2024-04-01T01:49:58.625 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 2966 2024-04-01T01:49:58.625 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 2967 2024-04-01T01:49:58.625 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 3592 2024-04-01T01:49:58.625 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 3593 2024-04-01T01:49:58.625 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4060 2024-04-01T01:49:58.625 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4061 2024-04-01T01:49:58.625 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4175 2024-04-01T01:49:58.625 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4176 2024-04-01T01:49:58.625 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4177 2024-04-01T01:49:58.626 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4178 2024-04-01T01:49:58.626 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4181 2024-04-01T01:49:58.626 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4182 2024-04-01T01:49:58.626 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4183 2024-04-01T01:49:58.626 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4184 2024-04-01T01:49:58.626 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 4185 2024-04-01T01:49:58.626 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 4186 2024-04-01T01:49:58.626 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6000 2024-04-01T01:49:58.626 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6001 2024-04-01T01:49:58.626 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6002 2024-04-01T01:49:58.626 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 6100 2024-04-01T01:49:58.626 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6114 2024-04-01T01:49:58.626 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 6115 2024-04-01T01:49:58.627 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:49 pg_control 2024-04-01T01:49:58.627 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Apr 1 01:33 pg_filenode.map 2024-04-01T01:49:58.627 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 23144 Apr 1 01:34 pg_internal.init 2024-04-01T01:49:58.627 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.627 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/log: 2024-04-01T01:49:58.627 INFO:teuthology.orchestra.run.smithi070.stdout:total 3 2024-04-01T01:49:58.627 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1047 Apr 1 01:34 . 2024-04-01T01:49:58.627 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.627 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1118 Apr 1 01:49 postgresql-Mon.log 2024-04-01T01:49:58.627 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.627 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-04-01T01:49:58.627 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:49:58.627 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:49:58.627 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.627 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.628 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-04-01T01:49:58.628 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:49:58.628 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:49:58.628 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.628 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.628 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-04-01T01:49:58.628 INFO:teuthology.orchestra.run.smithi070.stdout:total 3 2024-04-01T01:49:58.659 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 1 01:49 . 2024-04-01T01:49:58.659 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.659 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 mappings 2024-04-01T01:49:58.659 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 1 01:49 replorigin_checkpoint 2024-04-01T01:49:58.659 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 snapshots 2024-04-01T01:49:58.659 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.659 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-04-01T01:49:58.659 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:49:58.659 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:49:58.659 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 1 01:49 .. 2024-04-01T01:49:58.659 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.659 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-04-01T01:49:58.660 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:49:58.660 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:49:58.660 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Apr 1 01:49 .. 2024-04-01T01:49:58.660 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.660 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-04-01T01:49:58.660 INFO:teuthology.orchestra.run.smithi070.stdout:total 2 2024-04-01T01:49:58.660 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 . 2024-04-01T01:49:58.660 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.660 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 members 2024-04-01T01:49:58.660 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 offsets 2024-04-01T01:49:58.660 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.660 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-04-01T01:49:58.660 INFO:teuthology.orchestra.run.smithi070.stdout:total 9 2024-04-01T01:49:58.660 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 . 2024-04-01T01:49:58.661 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 .. 2024-04-01T01:49:58.661 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 0000 2024-04-01T01:49:58.661 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.661 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-04-01T01:49:58.661 INFO:teuthology.orchestra.run.smithi070.stdout:total 9 2024-04-01T01:49:58.661 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:33 . 2024-04-01T01:49:58.661 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Apr 1 01:33 .. 2024-04-01T01:49:58.661 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Apr 1 01:39 0000 2024-04-01T01:49:58.661 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.661 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-04-01T01:49:58.661 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:49:58.661 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:49:58.661 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.661 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.662 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-04-01T01:49:58.662 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:49:58.662 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:49:58.662 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.662 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.662 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-04-01T01:49:58.662 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:49:58.662 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:49:58.662 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.662 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.662 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-04-01T01:49:58.662 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:49:58.662 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:49:58.662 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.663 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.663 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-04-01T01:49:58.663 INFO:teuthology.orchestra.run.smithi070.stdout:total 16 2024-04-01T01:49:58.663 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 14082 Apr 1 01:49 . 2024-04-01T01:49:58.663 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1952 Apr 1 01:49 db_0.stat 2024-04-01T01:49:58.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 10979 Apr 1 01:49 db_13494.stat 2024-04-01T01:49:58.663 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1151 Apr 1 01:49 global.stat 2024-04-01T01:49:58.663 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.663 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-04-01T01:49:58.663 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:49:58.663 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:49 . 2024-04-01T01:49:58.663 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.663 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.663 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-04-01T01:49:58.664 INFO:teuthology.orchestra.run.smithi070.stdout:total 73 2024-04-01T01:49:58.664 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 73728 Apr 1 01:49 . 2024-04-01T01:49:58.664 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.704 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 73728 Apr 1 01:49 000C 2024-04-01T01:49:58.704 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.704 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-04-01T01:49:58.705 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:49:58.705 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:49:58.705 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.705 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.705 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-04-01T01:49:58.705 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:49:58.705 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:49:58.705 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.705 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.705 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-04-01T01:49:58.705 INFO:teuthology.orchestra.run.smithi070.stdout:total 917506 2024-04-01T01:49:58.705 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 939524096 Apr 1 01:49 . 2024-04-01T01:49:58.705 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.705 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:49 000000010000000000000073 2024-04-01T01:49:58.706 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:40 000000010000000000000074 2024-04-01T01:49:58.706 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:39 000000010000000000000075 2024-04-01T01:49:58.706 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:43 000000010000000000000076 2024-04-01T01:49:58.706 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:39 000000010000000000000077 2024-04-01T01:49:58.706 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:40 000000010000000000000078 2024-04-01T01:49:58.706 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:40 000000010000000000000079 2024-04-01T01:49:58.706 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:39 00000001000000000000007A 2024-04-01T01:49:58.706 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:41 00000001000000000000007B 2024-04-01T01:49:58.706 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:40 00000001000000000000007C 2024-04-01T01:49:58.706 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:39 00000001000000000000007D 2024-04-01T01:49:58.706 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:39 00000001000000000000007E 2024-04-01T01:49:58.706 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 00000001000000000000007F 2024-04-01T01:49:58.706 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:45 000000010000000000000080 2024-04-01T01:49:58.706 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 000000010000000000000081 2024-04-01T01:49:58.707 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 000000010000000000000082 2024-04-01T01:49:58.707 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:45 000000010000000000000083 2024-04-01T01:49:58.707 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:45 000000010000000000000084 2024-04-01T01:49:58.707 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:47 000000010000000000000085 2024-04-01T01:49:58.707 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 000000010000000000000086 2024-04-01T01:49:58.707 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 000000010000000000000087 2024-04-01T01:49:58.707 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:47 000000010000000000000088 2024-04-01T01:49:58.707 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:45 000000010000000000000089 2024-04-01T01:49:58.707 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 00000001000000000000008A 2024-04-01T01:49:58.707 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:46 00000001000000000000008B 2024-04-01T01:49:58.707 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 00000001000000000000008C 2024-04-01T01:49:58.707 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:45 00000001000000000000008D 2024-04-01T01:49:58.707 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 00000001000000000000008E 2024-04-01T01:49:58.710 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:45 00000001000000000000008F 2024-04-01T01:49:58.710 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:48 000000010000000000000090 2024-04-01T01:49:58.710 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 000000010000000000000091 2024-04-01T01:49:58.710 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:45 000000010000000000000092 2024-04-01T01:49:58.710 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:47 000000010000000000000093 2024-04-01T01:49:58.710 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:45 000000010000000000000094 2024-04-01T01:49:58.710 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 000000010000000000000095 2024-04-01T01:49:58.711 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 000000010000000000000096 2024-04-01T01:49:58.711 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 000000010000000000000097 2024-04-01T01:49:58.711 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 000000010000000000000098 2024-04-01T01:49:58.711 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:46 000000010000000000000099 2024-04-01T01:49:58.711 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 00000001000000000000009A 2024-04-01T01:49:58.711 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 00000001000000000000009B 2024-04-01T01:49:58.711 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 00000001000000000000009C 2024-04-01T01:49:58.711 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:44 00000001000000000000009D 2024-04-01T01:49:58.711 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:46 00000001000000000000009E 2024-04-01T01:49:58.711 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:49 00000001000000000000009F 2024-04-01T01:49:58.711 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:49 0000000100000000000000A0 2024-04-01T01:49:58.711 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:49 0000000100000000000000A1 2024-04-01T01:49:58.711 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:49 0000000100000000000000A2 2024-04-01T01:49:58.711 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:49 0000000100000000000000A3 2024-04-01T01:49:58.711 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:49 0000000100000000000000A4 2024-04-01T01:49:58.712 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:49 0000000100000000000000A5 2024-04-01T01:49:58.712 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:49 0000000100000000000000A6 2024-04-01T01:49:58.712 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:49 0000000100000000000000A7 2024-04-01T01:49:58.712 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:49 0000000100000000000000A8 2024-04-01T01:49:58.712 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:49 0000000100000000000000A9 2024-04-01T01:49:58.712 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Apr 1 01:49 0000000100000000000000AA 2024-04-01T01:49:58.712 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 archive_status 2024-04-01T01:49:58.712 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.712 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-04-01T01:49:58.712 INFO:teuthology.orchestra.run.smithi070.stdout:total 1 2024-04-01T01:49:58.712 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Apr 1 01:33 . 2024-04-01T01:49:58.712 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 939524096 Apr 1 01:49 .. 2024-04-01T01:49:58.712 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:49:58.713 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-04-01T01:49:58.713 INFO:teuthology.orchestra.run.smithi070.stdout:total 201 2024-04-01T01:49:58.713 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 204800 Apr 1 01:33 . 2024-04-01T01:49:58.713 INFO:teuthology.orchestra.run.smithi070.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 1524077604 Apr 1 01:49 .. 2024-04-01T01:49:58.713 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 204800 Apr 1 01:49 0000 2024-04-01T01:49:58.714 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-01T01:49:58.725 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi070.front.sepia.ceph.com 2024-04-01T01:49:58.733 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-01T01:49:58.742 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-01T01:49:59.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:49:58 smithi070 ceph-mon[32179]: pgmap v854: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 9.4 MiB/s wr, 888 op/s 2024-04-01T01:49:59.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:49:58 smithi081 ceph-mon[35142]: pgmap v854: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 9.4 MiB/s wr, 888 op/s 2024-04-01T01:49:59.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:49:58 smithi189 ceph-mon[35753]: pgmap v854: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 9.4 MiB/s wr, 888 op/s 2024-04-01T01:50:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:00 smithi081 ceph-mon[35142]: pgmap v855: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 9.1 MiB/s wr, 901 op/s 2024-04-01T01:50:01.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:00 smithi081 ceph-mon[35142]: overall HEALTH_OK 2024-04-01T01:50:01.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:00 smithi189 ceph-mon[35753]: pgmap v855: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 9.1 MiB/s wr, 901 op/s 2024-04-01T01:50:01.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:00 smithi189 ceph-mon[35753]: overall HEALTH_OK 2024-04-01T01:50:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:00 smithi070 ceph-mon[32179]: pgmap v855: 129 pgs: 129 active+clean; 1.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 9.1 MiB/s wr, 901 op/s 2024-04-01T01:50:01.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:00 smithi070 ceph-mon[32179]: overall HEALTH_OK 2024-04-01T01:50:02.884 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-01T01:50:02.884 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-01T01:50:03.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.004+0000 7f4fdb556640 1 -- 172.21.15.70:0/1311290970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fd4102210 msgr2=0x7f4fd4108aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.004+0000 7f4fdb556640 1 --2- 172.21.15.70:0/1311290970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fd4102210 0x7f4fd4108aa0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009ca0 tx=0x7f4fcc02c0f0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.004+0000 7f4fdb556640 1 -- 172.21.15.70:0/1311290970 shutdown_connections 2024-04-01T01:50:03.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.004+0000 7f4fdb556640 1 --2- 172.21.15.70:0/1311290970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fd4102210 0x7f4fd4108aa0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.004+0000 7f4fdb556640 1 --2- 172.21.15.70:0/1311290970 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fd4101850 0x7f4fd4101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.004+0000 7f4fdb556640 1 --2- 172.21.15.70:0/1311290970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fd4100660 0x7f4fd4100a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.004+0000 7f4fdb556640 1 -- 172.21.15.70:0/1311290970 >> 172.21.15.70:0/1311290970 conn(0x7f4fd40fbbf0 msgr2=0x7f4fd40fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.004+0000 7f4fdb556640 1 -- 172.21.15.70:0/1311290970 shutdown_connections 2024-04-01T01:50:03.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.005+0000 7f4fdb556640 1 -- 172.21.15.70:0/1311290970 wait complete. 2024-04-01T01:50:03.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.005+0000 7f4fdb556640 1 Processor -- start 2024-04-01T01:50:03.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.005+0000 7f4fdb556640 1 -- start start 2024-04-01T01:50:03.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.005+0000 7f4fdb556640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fd4100660 0x7f4fd4197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.005+0000 7f4fdb556640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fd4101850 0x7f4fd4197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.005+0000 7f4fdb556640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fd4102210 0x7f4fd419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.005+0000 7f4fdb556640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd410c190 con 0x7f4fd4101850 2024-04-01T01:50:03.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.005+0000 7f4fdb556640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd410c010 con 0x7f4fd4102210 2024-04-01T01:50:03.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.005+0000 7f4fdb556640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd410c310 con 0x7f4fd4100660 2024-04-01T01:50:03.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.005+0000 7f4fd8af1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fd4101850 0x7f4fd4197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.005+0000 7f4fd92f2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fd4100660 0x7f4fd4197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.005+0000 7f4fd8af1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fd4101850 0x7f4fd4197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60156/0 (socket says 172.21.15.70:60156) 2024-04-01T01:50:03.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.005+0000 7f4fd8af1640 1 -- 172.21.15.70:0/1168677401 learned_addr learned my addr 172.21.15.70:0/1168677401 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.005+0000 7f4fd9af3640 1 --2- 172.21.15.70:0/1168677401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fd4102210 0x7f4fd419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.006+0000 7f4fd92f2640 1 -- 172.21.15.70:0/1168677401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fd4102210 msgr2=0x7f4fd419ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.006+0000 7f4fd92f2640 1 --2- 172.21.15.70:0/1168677401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fd4102210 0x7f4fd419ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.006+0000 7f4fd92f2640 1 -- 172.21.15.70:0/1168677401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fd4101850 msgr2=0x7f4fd4197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.006+0000 7f4fd92f2640 1 --2- 172.21.15.70:0/1168677401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fd4101850 0x7f4fd4197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.006+0000 7f4fd92f2640 1 -- 172.21.15.70:0/1168677401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd419f570 con 0x7f4fd4100660 2024-04-01T01:50:03.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.006+0000 7f4fd8af1640 1 --2- 172.21.15.70:0/1168677401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fd4101850 0x7f4fd4197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:03.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.006+0000 7f4fd92f2640 1 --2- 172.21.15.70:0/1168677401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fd4100660 0x7f4fd4197950 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000e9c0 tx=0x7f4fd000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.006+0000 7f4fca7fc640 1 -- 172.21.15.70:0/1168677401 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd000ccc0 con 0x7f4fd4100660 2024-04-01T01:50:03.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.006+0000 7f4fca7fc640 1 -- 172.21.15.70:0/1168677401 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4fd00035f0 con 0x7f4fd4100660 2024-04-01T01:50:03.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.006+0000 7f4fca7fc640 1 -- 172.21.15.70:0/1168677401 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd000ce60 con 0x7f4fd4100660 2024-04-01T01:50:03.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.006+0000 7f4fdb556640 1 -- 172.21.15.70:0/1168677401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd419f860 con 0x7f4fd4100660 2024-04-01T01:50:03.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.006+0000 7f4fdb556640 1 -- 172.21.15.70:0/1168677401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd419fd60 con 0x7f4fd4100660 2024-04-01T01:50:03.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.008+0000 7f4fca7fc640 1 -- 172.21.15.70:0/1168677401 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f4fd00037a0 con 0x7f4fd4100660 2024-04-01T01:50:03.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fde212740 1 Processor -- start 2024-04-01T01:50:03.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fde212740 1 -- start start 2024-04-01T01:50:03.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fde212740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d086d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fde212740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d08ca0 0x560406d0d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fde212740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d0de10 0x560406d102a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fde212740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560406c6d180 con 0x560406d0de10 2024-04-01T01:50:03.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fde212740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560406c15b70 con 0x560406d082b0 2024-04-01T01:50:03.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fde212740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560406c69220 con 0x560406d08ca0 2024-04-01T01:50:03.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fca7fc640 1 --2- 172.21.15.70:0/1168677401 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4fc00a24a0 0x7f4fc00a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fd92f2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d08ca0 0x560406d0d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fd8af1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d0de10 0x560406d102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fd8af1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d0de10 0x560406d102a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60170/0 (socket says 172.21.15.70:60170) 2024-04-01T01:50:03.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fd8af1640 1 -- 172.21.15.70:0/2984836615 learned_addr learned my addr 172.21.15.70:0/2984836615 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fd9af3640 1 --2- 172.21.15.70:0/2984836615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fd9af3640 1 --2- 172.21.15.70:0/1168677401 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4fc00a24a0 0x7f4fc00a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fca7fc640 1 -- 172.21.15.70:0/1168677401 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f4fd00cee40 con 0x7f4fd4100660 2024-04-01T01:50:03.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fd92f2640 1 -- 172.21.15.70:0/2984836615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 msgr2=0x560406d086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fd92f2640 1 --2- 172.21.15.70:0/2984836615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fd92f2640 1 -- 172.21.15.70:0/2984836615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d0de10 msgr2=0x560406d102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fd92f2640 1 --2- 172.21.15.70:0/2984836615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d0de10 0x560406d102a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fd92f2640 1 -- 172.21.15.70:0/2984836615 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560406c297c0 con 0x560406d08ca0 2024-04-01T01:50:03.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.009+0000 7f4fd8af1640 1 --2- 172.21.15.70:0/2984836615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d0de10 0x560406d102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:03.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.010+0000 7f4fd9af3640 1 --2- 172.21.15.70:0/1168677401 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4fc00a24a0 0x7f4fc00a4960 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc006010 tx=0x7f4fcc0324f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.010+0000 7f4fd92f2640 1 --2- 172.21.15.70:0/2984836615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d08ca0 0x560406d0d7a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0097990 tx=0x7f4fd006e5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4447c9ab78f45ab6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.010+0000 7f4fbf7fe640 1 -- 172.21.15.70:0/2984836615 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0003b10 con 0x560406d08ca0 2024-04-01T01:50:03.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.010+0000 7f4fbf7fe640 1 -- 172.21.15.70:0/2984836615 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4fd009bc30 con 0x560406d08ca0 2024-04-01T01:50:03.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.010+0000 7f4fbf7fe640 1 -- 172.21.15.70:0/2984836615 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0066480 con 0x560406d08ca0 2024-04-01T01:50:03.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 -- 172.21.15.70:0/2984836615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d08ca0 msgr2=0x560406d0d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 --2- 172.21.15.70:0/2984836615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d08ca0 0x560406d0d7a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0097990 tx=0x7f4fd006e5f0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 -- 172.21.15.70:0/2984836615 shutdown_connections 2024-04-01T01:50:03.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 --2- 172.21.15.70:0/2984836615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d0de10 0x560406d102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 --2- 172.21.15.70:0/2984836615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d08ca0 0x560406d0d7a0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 --2- 172.21.15.70:0/2984836615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 -- 172.21.15.70:0/2984836615 >> 172.21.15.70:0/2984836615 conn(0x560406d03820 msgr2=0x560406d03c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 -- 172.21.15.70:0/2984836615 shutdown_connections 2024-04-01T01:50:03.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 -- 172.21.15.70:0/2984836615 wait complete. 2024-04-01T01:50:03.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 -- start start 2024-04-01T01:50:03.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 Processor -- start 2024-04-01T01:50:03.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d12820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d08ca0 0x560406d12d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d0de10 0x560406d19da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560406d10dc0 con 0x560406d0de10 2024-04-01T01:50:03.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560406c0b4a0 con 0x560406d082b0 2024-04-01T01:50:03.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.011+0000 7f4fde212740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560406d10f40 con 0x560406d08ca0 2024-04-01T01:50:03.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fde212740 10 client.0 _ll_register_callbacks cb 0x560406c0ed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:50:03.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fd9af3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d0de10 0x560406d19da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fd8af1640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d08ca0 0x560406d12d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fd9af3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d0de10 0x560406d19da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60178/0 (socket says 172.21.15.70:60178) 2024-04-01T01:50:03.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fd9af3640 1 -- 172.21.15.70:0/1936898877 learned_addr learned my addr 172.21.15.70:0/1936898877 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fd92f2640 1 --2- 172.21.15.70:0/1936898877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d12820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fd9af3640 1 -- 172.21.15.70:0/1936898877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d08ca0 msgr2=0x560406d12d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fd9af3640 1 --2- 172.21.15.70:0/1936898877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d08ca0 0x560406d12d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fd9af3640 1 -- 172.21.15.70:0/1936898877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 msgr2=0x560406d12820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fd9af3640 1 --2- 172.21.15.70:0/1936898877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d12820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fd9af3640 1 -- 172.21.15.70:0/1936898877 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560406a4b7d0 con 0x560406d0de10 2024-04-01T01:50:03.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fd8af1640 1 --2- 172.21.15.70:0/1936898877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d08ca0 0x560406d12d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:03.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fd9af3640 1 --2- 172.21.15.70:0/1936898877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d0de10 0x560406d19da0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009a60 tx=0x7f4fcc02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54d37df8be9a3296 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fbcff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:50:03.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fbcff9640 1 -- 172.21.15.70:0/1936898877 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc054060 con 0x560406d0de10 2024-04-01T01:50:03.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fbcff9640 1 -- 172.21.15.70:0/1936898877 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f4fcc003e00 con 0x560406d0de10 2024-04-01T01:50:03.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fde212740 1 -- 172.21.15.70:0/1936898877 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fd009bc30 con 0x560406d0de10 2024-04-01T01:50:03.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fbcff9640 1 -- 172.21.15.70:0/1936898877 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4fcc039040 con 0x560406d0de10 2024-04-01T01:50:03.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fde212740 10 client.20178 fetch_fsmap learned FSMap version 33 2024-04-01T01:50:03.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.012+0000 7f4fde212740 1 -- 172.21.15.70:0/1936898877 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5604069813f0 con 0x560406d0de10 2024-04-01T01:50:03.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.013+0000 7f4fbcff9640 1 -- 172.21.15.70:0/1936898877 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4fd009bc30 con 0x560406d0de10 2024-04-01T01:50:03.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.013+0000 7f4fbcff9640 1 -- 172.21.15.70:0/1936898877 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f4fcc04fb70 con 0x560406d0de10 2024-04-01T01:50:03.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.013+0000 7f4fde212740 10 client.20178 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:50:03.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.013+0000 7f4fde212740 10 client.20178 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:50:03.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.013+0000 7f4fde212740 20 client.20178 populate_metadata read hostname 'smithi070' 2024-04-01T01:50:03.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.013+0000 7f4fde212740 1 --2- 172.21.15.70:0/1936898877 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560406d1e720 0x560406d20b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.013+0000 7f4fde212740 4 client.20178 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:50:03.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.013+0000 7f4fde212740 1 -- 172.21.15.70:0/1936898877 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560406c7a6c0 con 0x560406d1e720 2024-04-01T01:50:03.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.013+0000 7f4fd92f2640 1 --2- 172.21.15.70:0/1936898877 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560406d1e720 0x560406d20b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.013+0000 7f4fbcff9640 10 client.20178 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:50:03.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.013+0000 7f4fd92f2640 1 --2- 172.21.15.70:0/1936898877 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560406d1e720 0x560406d20b50 crc :-1 s=READY pgs=408 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cd7aab8f2b5ccc7 server_cookie=61c5d730804fb20e in_seq=0 out_seq=0 2024-04-01T01:50:03.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.014+0000 7f4fbcff9640 1 -- 172.21.15.70:0/1936898877 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560406c7a6c0 con 0x560406d1e720 2024-04-01T01:50:03.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.014+0000 7f4fbcff9640 10 client.20178 handle_command_reply: tid=0 2024-04-01T01:50:03.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.014+0000 7f4fde212740 1 client.20178 shutdown 2024-04-01T01:50:03.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.014+0000 7f4fde212740 1 -- 172.21.15.70:0/1936898877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d0de10 msgr2=0x560406d19da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.014+0000 7f4fde212740 1 --2- 172.21.15.70:0/1936898877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d0de10 0x560406d19da0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009a60 tx=0x7f4fcc02c0f0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.014+0000 7f4fde212740 1 -- 172.21.15.70:0/1936898877 shutdown_connections 2024-04-01T01:50:03.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.014+0000 7f4fde212740 1 --2- 172.21.15.70:0/1936898877 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560406d1e720 0x560406d20b50 crc :-1 s=READY pgs=408 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.014+0000 7f4fde212740 1 --2- 172.21.15.70:0/1936898877 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d0de10 0x560406d19da0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.014+0000 7f4fde212740 1 --2- 172.21.15.70:0/1936898877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d08ca0 0x560406d12d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.014+0000 7f4fbcff9640 0 client.20178 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:50:03.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.014+0000 7f4fde212740 1 --2- 172.21.15.70:0/1936898877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d12820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.014+0000 7f4fde212740 1 -- 172.21.15.70:0/1936898877 >> 172.21.15.70:0/1936898877 conn(0x560406d03820 msgr2=0x560406d02c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.014+0000 7f4fde212740 1 -- 172.21.15.70:0/1936898877 shutdown_connections 2024-04-01T01:50:03.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.014+0000 7f4fde212740 1 -- 172.21.15.70:0/1936898877 wait complete. 2024-04-01T01:50:03.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.014+0000 7f4fde212740 20 client.20178 trim_cache size 0 max 16384 2024-04-01T01:50:03.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fde212740 1 Processor -- start 2024-04-01T01:50:03.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fde212740 1 -- start start 2024-04-01T01:50:03.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fde212740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d0a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fde212740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d08ca0 0x560406d0a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fde212740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d0de10 0x560406d19d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fde212740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd41aa3c0 con 0x560406d08ca0 2024-04-01T01:50:03.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fde212740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd41abf00 con 0x560406d082b0 2024-04-01T01:50:03.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fde212740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd41aaa90 con 0x560406d0de10 2024-04-01T01:50:03.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fd8af1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d08ca0 0x560406d0a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fd8af1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d08ca0 0x560406d0a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60188/0 (socket says 172.21.15.70:60188) 2024-04-01T01:50:03.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fd8af1640 1 -- 172.21.15.70:0/1081413480 learned_addr learned my addr 172.21.15.70:0/1081413480 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fd92f2640 1 --2- 172.21.15.70:0/1081413480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d0a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fd9af3640 1 --2- 172.21.15.70:0/1081413480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d0de10 0x560406d19d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fd8af1640 1 -- 172.21.15.70:0/1081413480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d0de10 msgr2=0x560406d19d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fd8af1640 1 --2- 172.21.15.70:0/1081413480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d0de10 0x560406d19d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fd8af1640 1 -- 172.21.15.70:0/1081413480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 msgr2=0x560406d0a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fd8af1640 1 --2- 172.21.15.70:0/1081413480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d0a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fd8af1640 1 -- 172.21.15.70:0/1081413480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604069410f0 con 0x560406d08ca0 2024-04-01T01:50:03.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.032+0000 7f4fd8af1640 1 --2- 172.21.15.70:0/1081413480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d08ca0 0x560406d0a740 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4012b60 tx=0x7f4fc4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d965782dc5c28a server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.033+0000 7f4fbeffd640 1 -- 172.21.15.70:0/1081413480 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc40120d0 con 0x560406d08ca0 2024-04-01T01:50:03.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.033+0000 7f4fbeffd640 1 -- 172.21.15.70:0/1081413480 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4fc4012270 con 0x560406d08ca0 2024-04-01T01:50:03.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.033+0000 7f4fbeffd640 1 -- 172.21.15.70:0/1081413480 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4004ba0 con 0x560406d08ca0 2024-04-01T01:50:03.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.033+0000 7f4fde212740 1 -- 172.21.15.70:0/1081413480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d08ca0 msgr2=0x560406d0a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.033+0000 7f4fde212740 1 --2- 172.21.15.70:0/1081413480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d08ca0 0x560406d0a740 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4012b60 tx=0x7f4fc4005f90 comp rx=0 tx=0).stop 2024-04-01T01:50:03.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.033+0000 7f4fde212740 1 -- 172.21.15.70:0/1081413480 shutdown_connections 2024-04-01T01:50:03.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.033+0000 7f4fde212740 1 --2- 172.21.15.70:0/1081413480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d0de10 0x560406d19d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.033+0000 7f4fde212740 1 --2- 172.21.15.70:0/1081413480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d08ca0 0x560406d0a740 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.033+0000 7f4fde212740 1 --2- 172.21.15.70:0/1081413480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d0a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.033+0000 7f4fde212740 1 -- 172.21.15.70:0/1081413480 >> 172.21.15.70:0/1081413480 conn(0x560406d03820 msgr2=0x560406d12820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.033+0000 7f4fde212740 1 -- 172.21.15.70:0/1081413480 shutdown_connections 2024-04-01T01:50:03.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.033+0000 7f4fde212740 1 -- 172.21.15.70:0/1081413480 wait complete. 2024-04-01T01:50:03.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fde212740 1 -- start start 2024-04-01T01:50:03.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fde212740 1 Processor -- start 2024-04-01T01:50:03.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fde212740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d171c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fde212740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d08ca0 0x560406d17700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fde212740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d0de10 0x560406d186a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fde212740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd41ab830 con 0x560406d08ca0 2024-04-01T01:50:03.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fde212740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd41ab160 con 0x560406d082b0 2024-04-01T01:50:03.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fde212740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd41a83a0 con 0x560406d0de10 2024-04-01T01:50:03.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fde212740 10 client.0 _ll_register_callbacks cb 0x7f4fd0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:50:03.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fd8af1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d08ca0 0x560406d17700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fd8af1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d08ca0 0x560406d17700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60198/0 (socket says 172.21.15.70:60198) 2024-04-01T01:50:03.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fd8af1640 1 -- 172.21.15.70:0/3585854407 learned_addr learned my addr 172.21.15.70:0/3585854407 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fd92f2640 1 --2- 172.21.15.70:0/3585854407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d171c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fd9af3640 1 --2- 172.21.15.70:0/3585854407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d0de10 0x560406d186a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fd8af1640 1 -- 172.21.15.70:0/3585854407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d0de10 msgr2=0x560406d186a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fd8af1640 1 --2- 172.21.15.70:0/3585854407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d0de10 0x560406d186a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fd8af1640 1 -- 172.21.15.70:0/3585854407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 msgr2=0x560406d171c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fd8af1640 1 --2- 172.21.15.70:0/3585854407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d171c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fd8af1640 1 -- 172.21.15.70:0/3585854407 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560406d18c90 con 0x560406d08ca0 2024-04-01T01:50:03.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.034+0000 7f4fd92f2640 1 --2- 172.21.15.70:0/3585854407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d171c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:03.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fd8af1640 1 --2- 172.21.15.70:0/3585854407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d08ca0 0x560406d17700 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4000c00 tx=0x7f4fc4004f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99a0d057b345fa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fbcff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:50:03.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fbcff9640 1 -- 172.21.15.70:0/3585854407 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc40120d0 con 0x560406d08ca0 2024-04-01T01:50:03.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fbcff9640 1 -- 172.21.15.70:0/3585854407 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f4fc4004030 con 0x560406d08ca0 2024-04-01T01:50:03.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fde212740 1 -- 172.21.15.70:0/3585854407 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fc4012270 con 0x560406d08ca0 2024-04-01T01:50:03.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fbcff9640 1 -- 172.21.15.70:0/3585854407 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4fc4034050 con 0x560406d08ca0 2024-04-01T01:50:03.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fde212740 10 client.20184 fetch_fsmap learned FSMap version 33 2024-04-01T01:50:03.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fde212740 1 -- 172.21.15.70:0/3585854407 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x560406d19fd0 con 0x560406d08ca0 2024-04-01T01:50:03.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fbcff9640 1 -- 172.21.15.70:0/3585854407 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4fc4012270 con 0x560406d08ca0 2024-04-01T01:50:03.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fbcff9640 1 -- 172.21.15.70:0/3585854407 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f4fc40035f0 con 0x560406d08ca0 2024-04-01T01:50:03.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fde212740 10 client.20184 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:50:03.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fde212740 10 client.20184 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:50:03.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fde212740 20 client.20184 populate_metadata read hostname 'smithi070' 2024-04-01T01:50:03.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fde212740 1 --2- 172.21.15.70:0/3585854407 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560406d851f0 0x560406d87610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fde212740 4 client.20184 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-01T01:50:03.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fde212740 1 -- 172.21.15.70:0/3585854407 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560406c74a10 con 0x560406d851f0 2024-04-01T01:50:03.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.035+0000 7f4fd92f2640 1 --2- 172.21.15.70:0/3585854407 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560406d851f0 0x560406d87610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fbcff9640 10 client.20184 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:50:03.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fd92f2640 1 --2- 172.21.15.70:0/3585854407 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560406d851f0 0x560406d87610 crc :-1 s=READY pgs=409 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecea4c8e60ead1d8 server_cookie=c426d11b903cde24 in_seq=0 out_seq=0 2024-04-01T01:50:03.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fbcff9640 1 -- 172.21.15.70:0/3585854407 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560406c74a10 con 0x560406d851f0 2024-04-01T01:50:03.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fbcff9640 10 client.20184 handle_command_reply: tid=0 2024-04-01T01:50:03.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fde212740 1 client.20184 shutdown 2024-04-01T01:50:03.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fde212740 1 -- 172.21.15.70:0/3585854407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d08ca0 msgr2=0x560406d17700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fde212740 1 --2- 172.21.15.70:0/3585854407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d08ca0 0x560406d17700 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4000c00 tx=0x7f4fc4004f50 comp rx=0 tx=0).stop 2024-04-01T01:50:03.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fde212740 1 -- 172.21.15.70:0/3585854407 shutdown_connections 2024-04-01T01:50:03.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fde212740 1 --2- 172.21.15.70:0/3585854407 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x560406d851f0 0x560406d87610 crc :-1 s=READY pgs=409 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fde212740 1 --2- 172.21.15.70:0/3585854407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560406d0de10 0x560406d186a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fbcff9640 0 client.20184 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:50:03.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fde212740 1 --2- 172.21.15.70:0/3585854407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560406d08ca0 0x560406d17700 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fde212740 1 --2- 172.21.15.70:0/3585854407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560406d082b0 0x560406d171c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fde212740 1 -- 172.21.15.70:0/3585854407 >> 172.21.15.70:0/3585854407 conn(0x560406d03820 msgr2=0x560406d12820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fde212740 1 -- 172.21.15.70:0/3585854407 shutdown_connections 2024-04-01T01:50:03.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fde212740 1 -- 172.21.15.70:0/3585854407 wait complete. 2024-04-01T01:50:03.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.036+0000 7f4fde212740 20 client.20184 trim_cache size 0 max 16384 2024-04-01T01:50:03.044 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:50:03.044 INFO:teuthology.orchestra.run.smithi070.stdout: "status": "no active scrubs running", 2024-04-01T01:50:03.044 INFO:teuthology.orchestra.run.smithi070.stdout: "scrubs": {} 2024-04-01T01:50:03.044 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:50:03.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.037+0000 7f4fdb556640 1 -- 172.21.15.70:0/1168677401 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4fc00a24a0 msgr2=0x7f4fc00a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.037+0000 7f4fdb556640 1 --2- 172.21.15.70:0/1168677401 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4fc00a24a0 0x7f4fc00a4960 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc006010 tx=0x7f4fcc0324f0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.037+0000 7f4fdb556640 1 -- 172.21.15.70:0/1168677401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fd4100660 msgr2=0x7f4fd4197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.037+0000 7f4fdb556640 1 --2- 172.21.15.70:0/1168677401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fd4100660 0x7f4fd4197950 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000e9c0 tx=0x7f4fd000ee80 comp rx=0 tx=0).stop 2024-04-01T01:50:03.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.037+0000 7f4fdb556640 1 -- 172.21.15.70:0/1168677401 shutdown_connections 2024-04-01T01:50:03.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.037+0000 7f4fdb556640 1 --2- 172.21.15.70:0/1168677401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4fd4102210 0x7f4fd419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.037+0000 7f4fdb556640 1 --2- 172.21.15.70:0/1168677401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4fd4101850 0x7f4fd4197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.037+0000 7f4fdb556640 1 --2- 172.21.15.70:0/1168677401 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f4fc00a24a0 0x7f4fc00a4960 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.037+0000 7f4fdb556640 1 --2- 172.21.15.70:0/1168677401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fd4100660 0x7f4fd4197950 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.037+0000 7f4fdb556640 1 -- 172.21.15.70:0/1168677401 >> 172.21.15.70:0/1168677401 conn(0x7f4fd40fbbf0 msgr2=0x7f4fd4069d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.037+0000 7f4fdb556640 1 -- 172.21.15.70:0/1168677401 shutdown_connections 2024-04-01T01:50:03.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.037+0000 7f4fdb556640 1 -- 172.21.15.70:0/1168677401 wait complete. 2024-04-01T01:50:03.047 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-01T01:50:03.047 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-01T01:50:03.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:02 smithi081 ceph-mon[35142]: pgmap v856: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 8.4 MiB/s wr, 887 op/s 2024-04-01T01:50:03.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.150+0000 7f19b0c8f640 1 -- 172.21.15.70:0/2899830187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19ac102d60 msgr2=0x7f19ac103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.150+0000 7f19b0c8f640 1 --2- 172.21.15.70:0/2899830187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19ac102d60 0x7f19ac103180 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f19a4009ca0 tx=0x7f19a402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.150+0000 7f19b0c8f640 1 -- 172.21.15.70:0/2899830187 shutdown_connections 2024-04-01T01:50:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.150+0000 7f19b0c8f640 1 --2- 172.21.15.70:0/2899830187 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19ac100190 0x7f19ac109d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.150+0000 7f19b0c8f640 1 --2- 172.21.15.70:0/2899830187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19ac0ff7d0 0x7f19ac0ffc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.150+0000 7f19b0c8f640 1 --2- 172.21.15.70:0/2899830187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19ac102d60 0x7f19ac103180 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.150+0000 7f19b0c8f640 1 -- 172.21.15.70:0/2899830187 >> 172.21.15.70:0/2899830187 conn(0x7f19ac0fb330 msgr2=0x7f19ac0fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.150+0000 7f19b0c8f640 1 -- 172.21.15.70:0/2899830187 shutdown_connections 2024-04-01T01:50:03.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.150+0000 7f19b0c8f640 1 -- 172.21.15.70:0/2899830187 wait complete. 2024-04-01T01:50:03.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19b0c8f640 1 Processor -- start 2024-04-01T01:50:03.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19b0c8f640 1 -- start start 2024-04-01T01:50:03.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19b0c8f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19ac0ff7d0 0x7f19ac197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19b0c8f640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19ac100190 0x7f19ac197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19b0c8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19ac102d60 0x7f19ac19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19b0c8f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f19ac10d2b0 con 0x7f19ac102d60 2024-04-01T01:50:03.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19b0c8f640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f19ac10d130 con 0x7f19ac100190 2024-04-01T01:50:03.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19b0c8f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f19ac10d430 con 0x7f19ac0ff7d0 2024-04-01T01:50:03.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19aad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19ac102d60 0x7f19ac19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19aad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19ac102d60 0x7f19ac19ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60224/0 (socket says 172.21.15.70:60224) 2024-04-01T01:50:03.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19aad9d640 1 -- 172.21.15.70:0/1008847451 learned_addr learned my addr 172.21.15.70:0/1008847451 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19a9d9b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19ac100190 0x7f19ac197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19aa59c640 1 --2- 172.21.15.70:0/1008847451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19ac0ff7d0 0x7f19ac197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19aad9d640 1 -- 172.21.15.70:0/1008847451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19ac0ff7d0 msgr2=0x7f19ac197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19aad9d640 1 --2- 172.21.15.70:0/1008847451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19ac0ff7d0 0x7f19ac197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19aad9d640 1 -- 172.21.15.70:0/1008847451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19ac100190 msgr2=0x7f19ac197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19aad9d640 1 --2- 172.21.15.70:0/1008847451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19ac100190 0x7f19ac197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19aad9d640 1 -- 172.21.15.70:0/1008847451 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19ac19f5e0 con 0x7f19ac102d60 2024-04-01T01:50:03.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19aad9d640 1 --2- 172.21.15.70:0/1008847451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19ac102d60 0x7f19ac19ef70 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f199c00e970 tx=0x7f199c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.151+0000 7f19a37fe640 1 -- 172.21.15.70:0/1008847451 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199c00ccc0 con 0x7f19ac102d60 2024-04-01T01:50:03.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.152+0000 7f19a37fe640 1 -- 172.21.15.70:0/1008847451 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f199c00ce60 con 0x7f19ac102d60 2024-04-01T01:50:03.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.152+0000 7f19b0c8f640 1 -- 172.21.15.70:0/1008847451 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19ac19f8d0 con 0x7f19ac102d60 2024-04-01T01:50:03.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.152+0000 7f19a37fe640 1 -- 172.21.15.70:0/1008847451 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199c010430 con 0x7f19ac102d60 2024-04-01T01:50:03.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.152+0000 7f19b0c8f640 1 -- 172.21.15.70:0/1008847451 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19ac19fdd0 con 0x7f19ac102d60 2024-04-01T01:50:03.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.153+0000 7f19b0c8f640 1 -- 172.21.15.70:0/1008847451 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19ac108d90 con 0x7f19ac102d60 2024-04-01T01:50:03.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.153+0000 7f19a37fe640 1 -- 172.21.15.70:0/1008847451 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f199c004ff0 con 0x7f19ac102d60 2024-04-01T01:50:03.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.154+0000 7f19a37fe640 1 --2- 172.21.15.70:0/1008847451 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f19980a24a0 0x7f19980a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.154+0000 7f19aa59c640 1 --2- 172.21.15.70:0/1008847451 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f19980a24a0 0x7f19980a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.154+0000 7f19a37fe640 1 -- 172.21.15.70:0/1008847451 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f199c0ceee0 con 0x7f19ac102d60 2024-04-01T01:50:03.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.155+0000 7f19aa59c640 1 --2- 172.21.15.70:0/1008847451 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f19980a24a0 0x7f19980a4960 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f19a4032380 tx=0x7f19a4000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.159+0000 7f19a37fe640 1 -- 172.21.15.70:0/1008847451 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f199c09bcd0 con 0x7f19ac102d60 2024-04-01T01:50:03.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:02 smithi189 ceph-mon[35753]: pgmap v856: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 8.4 MiB/s wr, 887 op/s 2024-04-01T01:50:03.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:02 smithi070 ceph-mon[32179]: pgmap v856: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 8.4 MiB/s wr, 887 op/s 2024-04-01T01:50:03.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.371+0000 7f19b0c8f640 1 -- 172.21.15.70:0/1008847451 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19ac062350 con 0x7f19ac102d60 2024-04-01T01:50:03.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.372+0000 7f19a37fe640 1 -- 172.21.15.70:0/1008847451 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11926 (secure 0 0 0) 0x7f199c09b5d0 con 0x7f19ac102d60 2024-04-01T01:50:03.372 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:50:03.372 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/33736871","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":33736871},{"type":"v1","addr":"172.21.15.70:6835","nonce":33736871}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2674132555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":2674132555},{"type":"v1","addr":"172.21.15.70:6837","nonce":2674132555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-01T01:31:15.732761+0000","modified":"2024-04-01T01:49:12.699222+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24464,"mds_1":24451,"mds_2":24418,"mds_3":24404,"mds_4":24379},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6839/1626804615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1626804615},{"type":"v1","addr":"172.21.15.70:6839","nonce":1626804615}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6841/2380158677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6840","nonce":2380158677},{"type":"v1","addr":"172.21.15.70:6841","nonce":2380158677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24371":{"gid":24371,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6833/1246539452","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1246539452},{"type":"v1","addr":"172.21.15.81:6833","nonce":1246539452}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24379":{"gid":24379,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.189:6833/2161209704","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2161209704},{"type":"v1","addr":"172.21.15.189:6833","nonce":2161209704}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6835/970944957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":970944957},{"type":"v1","addr":"172.21.15.81:6835","nonce":970944957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.189:6835/3861912911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3861912911},{"type":"v1","addr":"172.21.15.189:6835","nonce":3861912911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6837/3527001697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3527001697},{"type":"v1","addr":"172.21.15.81:6837","nonce":3527001697}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6839/2833086571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":2833086571},{"type":"v1","addr":"172.21.15.189:6839","nonce":2833086571}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6839/2805044611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":2805044611},{"type":"v1","addr":"172.21.15.81:6839","nonce":2805044611}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_29833":{"gid":29833,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/921108936","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":921108936},{"type":"v1","addr":"172.21.15.189:6837","nonce":921108936}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24418,24404,24379,24451,24464]},"id":1}]} 2024-04-01T01:50:03.372 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 33 2024-04-01T01:50:03.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.374+0000 7f19b0c8f640 1 -- 172.21.15.70:0/1008847451 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f19980a24a0 msgr2=0x7f19980a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.374+0000 7f19b0c8f640 1 --2- 172.21.15.70:0/1008847451 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f19980a24a0 0x7f19980a4960 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f19a4032380 tx=0x7f19a4000f40 comp rx=0 tx=0).stop 2024-04-01T01:50:03.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.374+0000 7f19b0c8f640 1 -- 172.21.15.70:0/1008847451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19ac102d60 msgr2=0x7f19ac19ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.374+0000 7f19b0c8f640 1 --2- 172.21.15.70:0/1008847451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19ac102d60 0x7f19ac19ef70 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f199c00e970 tx=0x7f199c00ed70 comp rx=0 tx=0).stop 2024-04-01T01:50:03.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.374+0000 7f19b0c8f640 1 -- 172.21.15.70:0/1008847451 shutdown_connections 2024-04-01T01:50:03.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.374+0000 7f19b0c8f640 1 --2- 172.21.15.70:0/1008847451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19ac102d60 0x7f19ac19ef70 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.374+0000 7f19b0c8f640 1 --2- 172.21.15.70:0/1008847451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19ac100190 0x7f19ac197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.374+0000 7f19b0c8f640 1 --2- 172.21.15.70:0/1008847451 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f19980a24a0 0x7f19980a4960 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.374+0000 7f19b0c8f640 1 --2- 172.21.15.70:0/1008847451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19ac0ff7d0 0x7f19ac197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.374+0000 7f19b0c8f640 1 -- 172.21.15.70:0/1008847451 >> 172.21.15.70:0/1008847451 conn(0x7f19ac0fb330 msgr2=0x7f19ac105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.374+0000 7f19b0c8f640 1 -- 172.21.15.70:0/1008847451 shutdown_connections 2024-04-01T01:50:03.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.374+0000 7f19b0c8f640 1 -- 172.21.15.70:0/1008847451 wait complete. 2024-04-01T01:50:03.385 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-01T01:50:03.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.491+0000 7fceda09d640 1 -- 172.21.15.70:0/374815130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4100650 msgr2=0x7fced4100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.491+0000 7fceda09d640 1 --2- 172.21.15.70:0/374815130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4100650 0x7fced4100a70 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fcec4009900 tx=0x7fcec402bc30 comp rx=0 tx=0).stop 2024-04-01T01:50:03.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.491+0000 7fceda09d640 1 -- 172.21.15.70:0/374815130 shutdown_connections 2024-04-01T01:50:03.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.491+0000 7fceda09d640 1 --2- 172.21.15.70:0/374815130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4102200 0x7fced4108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.491+0000 7fceda09d640 1 --2- 172.21.15.70:0/374815130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fced4101840 0x7fced4101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.491+0000 7fceda09d640 1 --2- 172.21.15.70:0/374815130 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4100650 0x7fced4100a70 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.491+0000 7fceda09d640 1 -- 172.21.15.70:0/374815130 >> 172.21.15.70:0/374815130 conn(0x7fced40fbbe0 msgr2=0x7fced40fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.491+0000 7fceda09d640 1 -- 172.21.15.70:0/374815130 shutdown_connections 2024-04-01T01:50:03.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.491+0000 7fceda09d640 1 -- 172.21.15.70:0/374815130 wait complete. 2024-04-01T01:50:03.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.491+0000 7fceda09d640 1 Processor -- start 2024-04-01T01:50:03.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.491+0000 7fceda09d640 1 -- start start 2024-04-01T01:50:03.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fceda09d640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4100650 0x7fced4196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fceda09d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fced4101840 0x7fced4196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fceda09d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4102200 0x7fced419dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fceda09d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fced410c180 con 0x7fced4102200 2024-04-01T01:50:03.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fceda09d640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fced410c000 con 0x7fced4100650 2024-04-01T01:50:03.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fceda09d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fced410c300 con 0x7fced4101840 2024-04-01T01:50:03.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fced3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4102200 0x7fced419dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fced37fe640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4100650 0x7fced4196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fced3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4102200 0x7fced419dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60232/0 (socket says 172.21.15.70:60232) 2024-04-01T01:50:03.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fcecaffd640 1 --2- 172.21.15.70:0/3770046542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fced4101840 0x7fced4196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fced3fff640 1 -- 172.21.15.70:0/3770046542 learned_addr learned my addr 172.21.15.70:0/3770046542 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fced3fff640 1 -- 172.21.15.70:0/3770046542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fced4101840 msgr2=0x7fced4196e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fced3fff640 1 --2- 172.21.15.70:0/3770046542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fced4101840 0x7fced4196e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fced3fff640 1 -- 172.21.15.70:0/3770046542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4100650 msgr2=0x7fced4196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fced3fff640 1 --2- 172.21.15.70:0/3770046542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4100650 0x7fced4196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fced3fff640 1 -- 172.21.15.70:0/3770046542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced419e530 con 0x7fced4102200 2024-04-01T01:50:03.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fced37fe640 1 --2- 172.21.15.70:0/3770046542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4100650 0x7fced4196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:03.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fced3fff640 1 --2- 172.21.15.70:0/3770046542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4102200 0x7fced419dec0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00eb10 tx=0x7fcecc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fced17fa640 1 -- 172.21.15.70:0/3770046542 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcecc004b20 con 0x7fced4102200 2024-04-01T01:50:03.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fced17fa640 1 -- 172.21.15.70:0/3770046542 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcecc004cc0 con 0x7fced4102200 2024-04-01T01:50:03.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fceda09d640 1 -- 172.21.15.70:0/3770046542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fced419e820 con 0x7fced4102200 2024-04-01T01:50:03.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fced17fa640 1 -- 172.21.15.70:0/3770046542 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcecc010690 con 0x7fced4102200 2024-04-01T01:50:03.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.492+0000 7fceda09d640 1 -- 172.21.15.70:0/3770046542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fced419ed20 con 0x7fced4102200 2024-04-01T01:50:03.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.494+0000 7fced17fa640 1 -- 172.21.15.70:0/3770046542 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fcecc005020 con 0x7fced4102200 2024-04-01T01:50:03.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fcedc922740 1 Processor -- start 2024-04-01T01:50:03.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fced17fa640 1 --2- 172.21.15.70:0/3770046542 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcec00a24a0 0x7fcec00a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fcedc922740 1 -- start start 2024-04-01T01:50:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fcedc922740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fcecaffd640 1 --2- 172.21.15.70:0/3770046542 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcec00a24a0 0x7fcec00a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fcedc922740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a1090 0x55ca5b4a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fcedc922740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a6200 0x55ca5b4a8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fcedc922740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ca5b4031b0 con 0x55ca5b4a6200 2024-04-01T01:50:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fcedc922740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ca5b3abc00 con 0x55ca5b4a06a0 2024-04-01T01:50:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fcedc922740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ca5b3ff250 con 0x55ca5b4a1090 2024-04-01T01:50:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fced17fa640 1 -- 172.21.15.70:0/3770046542 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fcecc0cf080 con 0x7fced4102200 2024-04-01T01:50:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fced3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a6200 0x55ca5b4a8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fcecaffd640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a1090 0x55ca5b4a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fced3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a6200 0x55ca5b4a8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60236/0 (socket says 172.21.15.70:60236) 2024-04-01T01:50:03.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fced3fff640 1 -- 172.21.15.70:0/2727237000 learned_addr learned my addr 172.21.15.70:0/2727237000 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fced37fe640 1 --2- 172.21.15.70:0/2727237000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fced3fff640 1 -- 172.21.15.70:0/2727237000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a1090 msgr2=0x55ca5b4a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fced3fff640 1 --2- 172.21.15.70:0/2727237000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a1090 0x55ca5b4a5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fced3fff640 1 -- 172.21.15.70:0/2727237000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a06a0 msgr2=0x55ca5b4a0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fced3fff640 1 --2- 172.21.15.70:0/2727237000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4a0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.495+0000 7fced3fff640 1 -- 172.21.15.70:0/2727237000 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca5b3cb3d0 con 0x55ca5b4a6200 2024-04-01T01:50:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.496+0000 7fcecaffd640 1 --2- 172.21.15.70:0/3770046542 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcec00a24a0 0x7fcec00a4960 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fcebc004650 tx=0x7fcebc009ad0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.496+0000 7fced3fff640 1 --2- 172.21.15.70:0/2727237000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a6200 0x55ca5b4a8690 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fcecc0a3c50 tx=0x7fcecc09b600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=348e69f91165c62f server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.496+0000 7fcec9ffb640 1 -- 172.21.15.70:0/2727237000 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcecc0b4070 con 0x55ca5b4a6200 2024-04-01T01:50:03.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.496+0000 7fcec9ffb640 1 -- 172.21.15.70:0/2727237000 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcecc09b770 con 0x55ca5b4a6200 2024-04-01T01:50:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.496+0000 7fcec9ffb640 1 -- 172.21.15.70:0/2727237000 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcecc097de0 con 0x55ca5b4a6200 2024-04-01T01:50:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.496+0000 7fcedc922740 1 -- 172.21.15.70:0/2727237000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a6200 msgr2=0x55ca5b4a8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.496+0000 7fcedc922740 1 --2- 172.21.15.70:0/2727237000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a6200 0x55ca5b4a8690 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fcecc0a3c50 tx=0x7fcecc09b600 comp rx=0 tx=0).stop 2024-04-01T01:50:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.496+0000 7fcedc922740 1 -- 172.21.15.70:0/2727237000 shutdown_connections 2024-04-01T01:50:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.496+0000 7fcedc922740 1 --2- 172.21.15.70:0/2727237000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a6200 0x55ca5b4a8690 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.496+0000 7fcedc922740 1 --2- 172.21.15.70:0/2727237000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a1090 0x55ca5b4a5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.496+0000 7fcedc922740 1 --2- 172.21.15.70:0/2727237000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.496+0000 7fcedc922740 1 -- 172.21.15.70:0/2727237000 >> 172.21.15.70:0/2727237000 conn(0x55ca5b49bc10 msgr2=0x55ca5b49c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.497+0000 7fcedc922740 1 -- 172.21.15.70:0/2727237000 shutdown_connections 2024-04-01T01:50:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.497+0000 7fcedc922740 1 -- 172.21.15.70:0/2727237000 wait complete. 2024-04-01T01:50:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.497+0000 7fcedc922740 1 -- start start 2024-04-01T01:50:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.497+0000 7fcedc922740 1 Processor -- start 2024-04-01T01:50:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.497+0000 7fcedc922740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4aac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.497+0000 7fcedc922740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a1090 0x55ca5b4ab150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.497+0000 7fcedc922740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a6200 0x55ca5b4b2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.497+0000 7fcedc922740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ca5b4a91b0 con 0x55ca5b4a1090 2024-04-01T01:50:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.497+0000 7fcedc922740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x55ca5b3a1470 con 0x55ca5b4a6200 2024-04-01T01:50:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.497+0000 7fcedc922740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ca5b4a9330 con 0x55ca5b4a06a0 2024-04-01T01:50:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.497+0000 7fcedc922740 10 client.0 _ll_register_callbacks cb 0x55ca5b3becb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:50:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.497+0000 7fcecaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a1090 0x55ca5b4ab150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.497+0000 7fced37fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4aac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.497+0000 7fcecaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a1090 0x55ca5b4ab150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60248/0 (socket says 172.21.15.70:60248) 2024-04-01T01:50:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.497+0000 7fcecaffd640 1 -- 172.21.15.70:0/2420607502 learned_addr learned my addr 172.21.15.70:0/2420607502 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.497+0000 7fced3fff640 1 --2- 172.21.15.70:0/2420607502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a6200 0x55ca5b4b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.498+0000 7fcecaffd640 1 -- 172.21.15.70:0/2420607502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a06a0 msgr2=0x55ca5b4aac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.498+0000 7fcecaffd640 1 --2- 172.21.15.70:0/2420607502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4aac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.498+0000 7fcecaffd640 1 -- 172.21.15.70:0/2420607502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a6200 msgr2=0x55ca5b4b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.498+0000 7fcecaffd640 1 --2- 172.21.15.70:0/2420607502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a6200 0x55ca5b4b2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.498+0000 7fcecaffd640 1 -- 172.21.15.70:0/2420607502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca5b1173f0 con 0x55ca5b4a1090 2024-04-01T01:50:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.498+0000 7fced3fff640 1 --2- 172.21.15.70:0/2420607502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a6200 0x55ca5b4b2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.498+0000 7fcecaffd640 1 --2- 172.21.15.70:0/2420607502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a1090 0x55ca5b4ab150 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fcebc013250 tx=0x7fcebc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=426538ab7d436ec7 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.498+0000 7fceb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:50:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.498+0000 7fceb2ffd640 1 -- 172.21.15.70:0/2420607502 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc012070 con 0x55ca5b4a1090 2024-04-01T01:50:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.498+0000 7fceb2ffd640 1 -- 172.21.15.70:0/2420607502 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fcebc01c040 con 0x55ca5b4a1090 2024-04-01T01:50:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.498+0000 7fcedc922740 1 -- 172.21.15.70:0/2420607502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcecc09b770 con 0x55ca5b4a1090 2024-04-01T01:50:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.498+0000 7fceb2ffd640 1 -- 172.21.15.70:0/2420607502 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fcebc012210 con 0x55ca5b4a1090 2024-04-01T01:50:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.498+0000 7fcedc922740 10 client.20202 fetch_fsmap learned FSMap version 33 2024-04-01T01:50:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.498+0000 7fcedc922740 1 -- 172.21.15.70:0/2420607502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ca5b1e17d0 con 0x55ca5b4a1090 2024-04-01T01:50:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.498+0000 7fceb2ffd640 1 -- 172.21.15.70:0/2420607502 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcecc09b770 con 0x55ca5b4a1090 2024-04-01T01:50:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.499+0000 7fceb2ffd640 1 -- 172.21.15.70:0/2420607502 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fcebc020050 con 0x55ca5b4a1090 2024-04-01T01:50:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.499+0000 7fcedc922740 10 client.20202 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:50:03.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.499+0000 7fcedc922740 10 client.20202 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:50:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.499+0000 7fcedc922740 20 client.20202 populate_metadata read hostname 'smithi070' 2024-04-01T01:50:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.499+0000 7fcedc922740 1 --2- 172.21.15.70:0/2420607502 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55ca5b49b170 0x55ca5b4baea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.499+0000 7fcedc922740 4 client.20202 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:50:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.499+0000 7fcedc922740 1 -- 172.21.15.70:0/2420607502 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca5b4106f0 con 0x55ca5b49b170 2024-04-01T01:50:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.499+0000 7fced37fe640 1 --2- 172.21.15.70:0/2420607502 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55ca5b49b170 0x55ca5b4baea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.499+0000 7fceb2ffd640 10 client.20202 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:50:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.499+0000 7fced37fe640 1 --2- 172.21.15.70:0/2420607502 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55ca5b49b170 0x55ca5b4baea0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=945d597d27c938dc server_cookie=2d50a83f5f69d2a3 in_seq=0 out_seq=0 2024-04-01T01:50:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.500+0000 7fceb2ffd640 1 -- 172.21.15.70:0/2420607502 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ca5b4106f0 con 0x55ca5b49b170 2024-04-01T01:50:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.500+0000 7fceb2ffd640 10 client.20202 handle_command_reply: tid=0 2024-04-01T01:50:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.500+0000 7fcedc922740 1 client.20202 shutdown 2024-04-01T01:50:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.500+0000 7fcedc922740 1 -- 172.21.15.70:0/2420607502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a1090 msgr2=0x55ca5b4ab150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.500+0000 7fcedc922740 1 --2- 172.21.15.70:0/2420607502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a1090 0x55ca5b4ab150 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fcebc013250 tx=0x7fcebc004730 comp rx=0 tx=0).stop 2024-04-01T01:50:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.500+0000 7fcedc922740 1 -- 172.21.15.70:0/2420607502 shutdown_connections 2024-04-01T01:50:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.500+0000 7fcedc922740 1 --2- 172.21.15.70:0/2420607502 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55ca5b49b170 0x55ca5b4baea0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.500+0000 7fcedc922740 1 --2- 172.21.15.70:0/2420607502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a6200 0x55ca5b4b2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.500+0000 7fcedc922740 1 --2- 172.21.15.70:0/2420607502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a1090 0x55ca5b4ab150 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.500+0000 7fceb2ffd640 0 client.20202 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:50:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.500+0000 7fcedc922740 1 --2- 172.21.15.70:0/2420607502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4aac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.500+0000 7fcedc922740 1 -- 172.21.15.70:0/2420607502 >> 172.21.15.70:0/2420607502 conn(0x55ca5b49bc10 msgr2=0x55ca5b498d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.500+0000 7fcedc922740 1 -- 172.21.15.70:0/2420607502 shutdown_connections 2024-04-01T01:50:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.500+0000 7fcedc922740 1 -- 172.21.15.70:0/2420607502 wait complete. 2024-04-01T01:50:03.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.500+0000 7fcedc922740 20 client.20202 trim_cache size 0 max 16384 2024-04-01T01:50:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.513+0000 7fcedc922740 1 Processor -- start 2024-04-01T01:50:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.513+0000 7fcedc922740 1 -- start start 2024-04-01T01:50:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fcedc922740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4a96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fcedc922740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a1090 0x55ca5b4b2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fcedc922740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a6200 0x55ca5b4a25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fcedc922740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fced41a8270 con 0x55ca5b4a06a0 2024-04-01T01:50:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fcedc922740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fced41a9db0 con 0x55ca5b4a1090 2024-04-01T01:50:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fcedc922740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fced41a8940 con 0x55ca5b4a6200 2024-04-01T01:50:03.520 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fced37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4a96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fced37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4a96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60264/0 (socket says 172.21.15.70:60264) 2024-04-01T01:50:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fced37fe640 1 -- 172.21.15.70:0/262733264 learned_addr learned my addr 172.21.15.70:0/262733264 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fced3fff640 1 --2- 172.21.15.70:0/262733264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a6200 0x55ca5b4a25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fcecaffd640 1 --2- 172.21.15.70:0/262733264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a1090 0x55ca5b4b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fced37fe640 1 -- 172.21.15.70:0/262733264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a6200 msgr2=0x55ca5b4a25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fced37fe640 1 --2- 172.21.15.70:0/262733264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a6200 0x55ca5b4a25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fced37fe640 1 -- 172.21.15.70:0/262733264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a1090 msgr2=0x55ca5b4b2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fced37fe640 1 --2- 172.21.15.70:0/262733264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a1090 0x55ca5b4b2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fced37fe640 1 -- 172.21.15.70:0/262733264 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca5b4bb460 con 0x55ca5b4a06a0 2024-04-01T01:50:03.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fcecaffd640 1 --2- 172.21.15.70:0/262733264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a1090 0x55ca5b4b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:50:03.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fced37fe640 1 --2- 172.21.15.70:0/262733264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4a96d0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fced41016a0 tx=0x7fcec4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c46f82d1d8374b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fcec97fa640 1 -- 172.21.15.70:0/262733264 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec404f020 con 0x55ca5b4a06a0 2024-04-01T01:50:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fcec97fa640 1 -- 172.21.15.70:0/262733264 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcec404bab0 con 0x55ca5b4a06a0 2024-04-01T01:50:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.514+0000 7fcec97fa640 1 -- 172.21.15.70:0/262733264 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec4049070 con 0x55ca5b4a06a0 2024-04-01T01:50:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 -- 172.21.15.70:0/262733264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a06a0 msgr2=0x55ca5b4a96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 --2- 172.21.15.70:0/262733264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4a96d0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fced41016a0 tx=0x7fcec4005fd0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 -- 172.21.15.70:0/262733264 shutdown_connections 2024-04-01T01:50:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 --2- 172.21.15.70:0/262733264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a6200 0x55ca5b4a25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 --2- 172.21.15.70:0/262733264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a1090 0x55ca5b4b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 --2- 172.21.15.70:0/262733264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4a96d0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 -- 172.21.15.70:0/262733264 >> 172.21.15.70:0/262733264 conn(0x55ca5b49bc10 msgr2=0x55ca5b49bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 -- 172.21.15.70:0/262733264 shutdown_connections 2024-04-01T01:50:03.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 -- 172.21.15.70:0/262733264 wait complete. 2024-04-01T01:50:03.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 -- start start 2024-04-01T01:50:03.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 Processor -- start 2024-04-01T01:50:03.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a1090 0x55ca5b4afd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a6200 0x55ca5b4b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fced41a96e0 con 0x55ca5b4a1090 2024-04-01T01:50:03.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fced41a9010 con 0x55ca5b4a06a0 2024-04-01T01:50:03.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.515+0000 7fcedc922740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fced41a6250 con 0x55ca5b4a6200 2024-04-01T01:50:03.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fcedc922740 10 client.0 _ll_register_callbacks cb 0x7fcebc009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:50:03.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fcecaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a1090 0x55ca5b4afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fcecaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a1090 0x55ca5b4afd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60270/0 (socket says 172.21.15.70:60270) 2024-04-01T01:50:03.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fcecaffd640 1 -- 172.21.15.70:0/2816613896 learned_addr learned my addr 172.21.15.70:0/2816613896 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fced3fff640 1 --2- 172.21.15.70:0/2816613896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a6200 0x55ca5b4b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fced37fe640 1 --2- 172.21.15.70:0/2816613896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fcecaffd640 1 -- 172.21.15.70:0/2816613896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a6200 msgr2=0x55ca5b4b0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fcecaffd640 1 --2- 172.21.15.70:0/2816613896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a6200 0x55ca5b4b0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fcecaffd640 1 -- 172.21.15.70:0/2816613896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a06a0 msgr2=0x55ca5b4af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fcecaffd640 1 --2- 172.21.15.70:0/2816613896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fcecaffd640 1 -- 172.21.15.70:0/2816613896 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca5b4b1380 con 0x55ca5b4a1090 2024-04-01T01:50:03.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fcecaffd640 1 --2- 172.21.15.70:0/2816613896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a1090 0x55ca5b4afd80 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fcebc02ab70 tx=0x7fcebc012ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf2b9277ceb7b89 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fceb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:50:03.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fceb2ffd640 1 -- 172.21.15.70:0/2816613896 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc02a910 con 0x55ca5b4a1090 2024-04-01T01:50:03.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fceb2ffd640 1 -- 172.21.15.70:0/2816613896 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fcebc009040 con 0x55ca5b4a1090 2024-04-01T01:50:03.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fcedc922740 1 -- 172.21.15.70:0/2816613896 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcec404bab0 con 0x55ca5b4a1090 2024-04-01T01:50:03.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fceb2ffd640 1 -- 172.21.15.70:0/2816613896 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fcebc03f050 con 0x55ca5b4a1090 2024-04-01T01:50:03.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fcedc922740 10 client.20208 fetch_fsmap learned FSMap version 33 2024-04-01T01:50:03.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.516+0000 7fcedc922740 1 -- 172.21.15.70:0/2816613896 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ca5b51ecf0 con 0x55ca5b4a1090 2024-04-01T01:50:03.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.517+0000 7fceb2ffd640 1 -- 172.21.15.70:0/2816613896 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcec404bab0 con 0x55ca5b4a1090 2024-04-01T01:50:03.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.517+0000 7fceb2ffd640 1 -- 172.21.15.70:0/2816613896 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fcebc005040 con 0x55ca5b4a1090 2024-04-01T01:50:03.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.517+0000 7fcedc922740 10 client.20208 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:50:03.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.517+0000 7fcedc922740 10 client.20208 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-04-01T01:50:03.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.517+0000 7fcedc922740 20 client.20208 populate_metadata read hostname 'smithi070' 2024-04-01T01:50:03.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.517+0000 7fcedc922740 1 --2- 172.21.15.70:0/2816613896 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55ca5b51f720 0x55ca5b521b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.517+0000 7fcedc922740 4 client.20208 mds_command: new command op to 24379 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:50:03.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.517+0000 7fcedc922740 1 -- 172.21.15.70:0/2816613896 --> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ca5b40aa40 con 0x55ca5b51f720 2024-04-01T01:50:03.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.517+0000 7fced37fe640 1 --2- 172.21.15.70:0/2816613896 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55ca5b51f720 0x55ca5b521b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.517+0000 7fced37fe640 1 --2- 172.21.15.70:0/2816613896 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55ca5b51f720 0x55ca5b521b40 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=298ef53aaad08056 server_cookie=b2c2f95971f7a4ed in_seq=0 out_seq=0 2024-04-01T01:50:03.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.517+0000 7fceb2ffd640 10 client.20208 ms_handle_connect on v2:172.21.15.189:6832/2161209704 2024-04-01T01:50:03.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.517+0000 7fceb2ffd640 1 -- 172.21.15.70:0/2816613896 <== mds.4 v2:172.21.15.189:6832/2161209704 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ca5b40aa40 con 0x55ca5b51f720 2024-04-01T01:50:03.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.517+0000 7fceb2ffd640 10 client.20208 handle_command_reply: tid=0 2024-04-01T01:50:03.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.517+0000 7fcedc922740 1 client.20208 shutdown 2024-04-01T01:50:03.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fcedc922740 1 -- 172.21.15.70:0/2816613896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a1090 msgr2=0x55ca5b4afd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fcedc922740 1 --2- 172.21.15.70:0/2816613896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a1090 0x55ca5b4afd80 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fcebc02ab70 tx=0x7fcebc012ad0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fcedc922740 1 -- 172.21.15.70:0/2816613896 shutdown_connections 2024-04-01T01:50:03.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fcedc922740 1 --2- 172.21.15.70:0/2816613896 >> [v2:172.21.15.189:6832/2161209704,v1:172.21.15.189:6833/2161209704] conn(0x55ca5b51f720 0x55ca5b521b40 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fcedc922740 1 --2- 172.21.15.70:0/2816613896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ca5b4a6200 0x55ca5b4b0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fceb2ffd640 0 client.20208 ms_handle_reset on v2:172.21.15.189:6832/2161209704 2024-04-01T01:50:03.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fcedc922740 1 --2- 172.21.15.70:0/2816613896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ca5b4a1090 0x55ca5b4afd80 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fcedc922740 1 --2- 172.21.15.70:0/2816613896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ca5b4a06a0 0x55ca5b4af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fcedc922740 1 -- 172.21.15.70:0/2816613896 >> 172.21.15.70:0/2816613896 conn(0x55ca5b49bc10 msgr2=0x55ca5b4aab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fcedc922740 1 -- 172.21.15.70:0/2816613896 shutdown_connections 2024-04-01T01:50:03.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fcedc922740 1 -- 172.21.15.70:0/2816613896 wait complete. 2024-04-01T01:50:03.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fcedc922740 20 client.20208 trim_cache size 0 max 16384 2024-04-01T01:50:03.533 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:50:03.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fceda09d640 1 -- 172.21.15.70:0/3770046542 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcec00a24a0 msgr2=0x7fcec00a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fceda09d640 1 --2- 172.21.15.70:0/3770046542 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcec00a24a0 0x7fcec00a4960 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fcebc004650 tx=0x7fcebc009ad0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fceda09d640 1 -- 172.21.15.70:0/3770046542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4102200 msgr2=0x7fced419dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fceda09d640 1 --2- 172.21.15.70:0/3770046542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4102200 0x7fced419dec0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00eb10 tx=0x7fcecc00c2d0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fceda09d640 1 -- 172.21.15.70:0/3770046542 shutdown_connections 2024-04-01T01:50:03.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fceda09d640 1 --2- 172.21.15.70:0/3770046542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fced4102200 0x7fced419dec0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fceda09d640 1 --2- 172.21.15.70:0/3770046542 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fcec00a24a0 0x7fcec00a4960 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fceda09d640 1 --2- 172.21.15.70:0/3770046542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fced4101840 0x7fced4196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fceda09d640 1 --2- 172.21.15.70:0/3770046542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fced4100650 0x7fced4196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fceda09d640 1 -- 172.21.15.70:0/3770046542 >> 172.21.15.70:0/3770046542 conn(0x7fced40fbbe0 msgr2=0x7fced4104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.518+0000 7fceda09d640 1 -- 172.21.15.70:0/3770046542 shutdown_connections 2024-04-01T01:50:03.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.519+0000 7fceda09d640 1 -- 172.21.15.70:0/3770046542 wait complete. 2024-04-01T01:50:03.539 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-01T01:50:03.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.643+0000 7effc6525640 1 -- 172.21.15.70:0/3873296979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effc0100650 msgr2=0x7effc0100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.644+0000 7effc6525640 1 --2- 172.21.15.70:0/3873296979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effc0100650 0x7effc0100a70 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7effb0009900 tx=0x7effb002bc30 comp rx=0 tx=0).stop 2024-04-01T01:50:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.644+0000 7effc6525640 1 -- 172.21.15.70:0/3873296979 shutdown_connections 2024-04-01T01:50:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.644+0000 7effc6525640 1 --2- 172.21.15.70:0/3873296979 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effc0102200 0x7effc0108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.644+0000 7effc6525640 1 --2- 172.21.15.70:0/3873296979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effc0101840 0x7effc0101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.644+0000 7effc6525640 1 --2- 172.21.15.70:0/3873296979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effc0100650 0x7effc0100a70 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.644+0000 7effc6525640 1 -- 172.21.15.70:0/3873296979 >> 172.21.15.70:0/3873296979 conn(0x7effc00fbbe0 msgr2=0x7effc00fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.644+0000 7effc6525640 1 -- 172.21.15.70:0/3873296979 shutdown_connections 2024-04-01T01:50:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.644+0000 7effc6525640 1 -- 172.21.15.70:0/3873296979 wait complete. 2024-04-01T01:50:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.644+0000 7effc6525640 1 Processor -- start 2024-04-01T01:50:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.644+0000 7effc6525640 1 -- start start 2024-04-01T01:50:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.644+0000 7effc6525640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effc0100650 0x7effc0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.644+0000 7effc6525640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effc0101840 0x7effc019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.644+0000 7effc6525640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effc0102200 0x7effc01a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.644+0000 7effc6525640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7effc010c170 con 0x7effc0100650 2024-04-01T01:50:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.644+0000 7effc6525640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7effc010bff0 con 0x7effc0102200 2024-04-01T01:50:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.644+0000 7effc6525640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7effc010c2f0 con 0x7effc0101840 2024-04-01T01:50:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.645+0000 7effbffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effc0100650 0x7effc0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.645+0000 7effbffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effc0100650 0x7effc0069420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60284/0 (socket says 172.21.15.70:60284) 2024-04-01T01:50:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.645+0000 7effbffff640 1 -- 172.21.15.70:0/945459782 learned_addr learned my addr 172.21.15.70:0/945459782 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.645+0000 7effc4ac2640 1 --2- 172.21.15.70:0/945459782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effc0102200 0x7effc01a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.645+0000 7effbf7fe640 1 --2- 172.21.15.70:0/945459782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effc0101840 0x7effc019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.645+0000 7effbffff640 1 -- 172.21.15.70:0/945459782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effc0101840 msgr2=0x7effc019eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.645+0000 7effbffff640 1 --2- 172.21.15.70:0/945459782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effc0101840 0x7effc019eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.645+0000 7effbffff640 1 -- 172.21.15.70:0/945459782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effc0102200 msgr2=0x7effc01a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.645+0000 7effbffff640 1 --2- 172.21.15.70:0/945459782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effc0102200 0x7effc01a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.645+0000 7effbffff640 1 -- 172.21.15.70:0/945459782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc01a1890 con 0x7effc0100650 2024-04-01T01:50:03.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.645+0000 7effbffff640 1 --2- 172.21.15.70:0/945459782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effc0100650 0x7effc0069420 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7effb0000c00 tx=0x7effb0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.645+0000 7effbd7fa640 1 -- 172.21.15.70:0/945459782 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb0043020 con 0x7effc0100650 2024-04-01T01:50:03.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.645+0000 7effc6525640 1 -- 172.21.15.70:0/945459782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effc01a1ac0 con 0x7effc0100650 2024-04-01T01:50:03.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.645+0000 7effbd7fa640 1 -- 172.21.15.70:0/945459782 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effb0004350 con 0x7effc0100650 2024-04-01T01:50:03.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.645+0000 7effbd7fa640 1 -- 172.21.15.70:0/945459782 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb0003b80 con 0x7effc0100650 2024-04-01T01:50:03.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.645+0000 7effc6525640 1 -- 172.21.15.70:0/945459782 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effc01a2060 con 0x7effc0100650 2024-04-01T01:50:03.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.647+0000 7effbd7fa640 1 -- 172.21.15.70:0/945459782 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7effb0046020 con 0x7effc0100650 2024-04-01T01:50:03.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effc91e9740 1 Processor -- start 2024-04-01T01:50:03.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effbd7fa640 1 --2- 172.21.15.70:0/945459782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7effa80a24d0 0x7effa80a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effc91e9740 1 -- start start 2024-04-01T01:50:03.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effc4ac2640 1 --2- 172.21.15.70:0/945459782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7effa80a24d0 0x7effa80a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effc91e9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334a490 0x56045334a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effc91e9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334ae80 0x56045334f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effbd7fa640 1 -- 172.21.15.70:0/945459782 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7effb00f0c00 con 0x7effc0100650 2024-04-01T01:50:03.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effc91e9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334fff0 0x560453352480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effc91e9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5604532af1b0 con 0x56045334a490 2024-04-01T01:50:03.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effc91e9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x560453257c00 con 0x56045334fff0 2024-04-01T01:50:03.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effc91e9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5604532ab250 con 0x56045334ae80 2024-04-01T01:50:03.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effbffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334a490 0x56045334a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effbffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334a490 0x56045334a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60286/0 (socket says 172.21.15.70:60286) 2024-04-01T01:50:03.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effbffff640 1 -- 172.21.15.70:0/318490435 learned_addr learned my addr 172.21.15.70:0/318490435 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effbf7fe640 1 --2- 172.21.15.70:0/318490435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334ae80 0x56045334f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effc4ac2640 1 --2- 172.21.15.70:0/318490435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334fff0 0x560453352480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effbffff640 1 -- 172.21.15.70:0/318490435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334ae80 msgr2=0x56045334f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effbffff640 1 --2- 172.21.15.70:0/318490435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334ae80 0x56045334f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effbffff640 1 -- 172.21.15.70:0/318490435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334fff0 msgr2=0x560453352480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effbffff640 1 --2- 172.21.15.70:0/318490435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334fff0 0x560453352480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.648+0000 7effbffff640 1 -- 172.21.15.70:0/318490435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604532773d0 con 0x56045334a490 2024-04-01T01:50:03.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.649+0000 7effc4ac2640 1 --2- 172.21.15.70:0/945459782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7effa80a24d0 0x7effa80a4990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7effb8005fd0 tx=0x7effb8011bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.649+0000 7effbffff640 1 --2- 172.21.15.70:0/318490435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334a490 0x56045334a8b0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7effb00bd280 tx=0x7effb002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ea7416bb2eb053 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.649+0000 7effa67fc640 1 -- 172.21.15.70:0/318490435 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb00b7e60 con 0x56045334a490 2024-04-01T01:50:03.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.649+0000 7effa67fc640 1 -- 172.21.15.70:0/318490435 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effb00bd2f0 con 0x56045334a490 2024-04-01T01:50:03.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.649+0000 7effa67fc640 1 -- 172.21.15.70:0/318490435 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb00054d0 con 0x56045334a490 2024-04-01T01:50:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.649+0000 7effc91e9740 1 -- 172.21.15.70:0/318490435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334a490 msgr2=0x56045334a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.649+0000 7effc91e9740 1 --2- 172.21.15.70:0/318490435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334a490 0x56045334a8b0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7effb00bd280 tx=0x7effb002bc30 comp rx=0 tx=0).stop 2024-04-01T01:50:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.649+0000 7effc91e9740 1 -- 172.21.15.70:0/318490435 shutdown_connections 2024-04-01T01:50:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.649+0000 7effc91e9740 1 --2- 172.21.15.70:0/318490435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334fff0 0x560453352480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.649+0000 7effc91e9740 1 --2- 172.21.15.70:0/318490435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334ae80 0x56045334f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.649+0000 7effc91e9740 1 --2- 172.21.15.70:0/318490435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334a490 0x56045334a8b0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.649+0000 7effc91e9740 1 -- 172.21.15.70:0/318490435 >> 172.21.15.70:0/318490435 conn(0x560453345a00 msgr2=0x560453345e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.649+0000 7effc91e9740 1 -- 172.21.15.70:0/318490435 shutdown_connections 2024-04-01T01:50:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.649+0000 7effc91e9740 1 -- 172.21.15.70:0/318490435 wait complete. 2024-04-01T01:50:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.650+0000 7effc91e9740 1 -- start start 2024-04-01T01:50:03.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.650+0000 7effc91e9740 1 Processor -- start 2024-04-01T01:50:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.650+0000 7effc91e9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334a490 0x560453354700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.650+0000 7effc91e9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334ae80 0x560453354c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.650+0000 7effc91e9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334fff0 0x56045335bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.650+0000 7effc91e9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560453352d60 con 0x56045334ae80 2024-04-01T01:50:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.650+0000 7effc91e9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56045324d470 con 0x56045334a490 2024-04-01T01:50:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.650+0000 7effc91e9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560453352ee0 con 0x56045334fff0 2024-04-01T01:50:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.650+0000 7effc91e9740 10 client.0 _ll_register_callbacks cb 0x56045326acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:50:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.650+0000 7effbf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334ae80 0x560453354c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.650+0000 7effbffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334a490 0x560453354700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.650+0000 7effbf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334ae80 0x560453354c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60298/0 (socket says 172.21.15.70:60298) 2024-04-01T01:50:03.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.650+0000 7effbf7fe640 1 -- 172.21.15.70:0/473690566 learned_addr learned my addr 172.21.15.70:0/473690566 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.650+0000 7effbffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334a490 0x560453354700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60074/0 (socket says 172.21.15.70:60074) 2024-04-01T01:50:03.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.650+0000 7effc4ac2640 1 --2- 172.21.15.70:0/473690566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334fff0 0x56045335bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effbf7fe640 1 -- 172.21.15.70:0/473690566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334fff0 msgr2=0x56045335bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effbf7fe640 1 --2- 172.21.15.70:0/473690566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334fff0 0x56045335bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effbf7fe640 1 -- 172.21.15.70:0/473690566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334a490 msgr2=0x560453354700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effbf7fe640 1 --2- 172.21.15.70:0/473690566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334a490 0x560453354700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effbf7fe640 1 -- 172.21.15.70:0/473690566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560452fc33f0 con 0x56045334ae80 2024-04-01T01:50:03.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effbffff640 1 --2- 172.21.15.70:0/473690566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334a490 0x560453354700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:03.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effc4ac2640 1 --2- 172.21.15.70:0/473690566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334fff0 0x56045335bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:03.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effbf7fe640 1 --2- 172.21.15.70:0/473690566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334ae80 0x560453354c40 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7effb4012700 tx=0x7effb4012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b9ab7bd06c027c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7eff9f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:50:03.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7eff9f7fe640 1 -- 172.21.15.70:0/473690566 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb400f8c0 con 0x56045334ae80 2024-04-01T01:50:03.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7eff9f7fe640 1 -- 172.21.15.70:0/473690566 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7effb400fa60 con 0x56045334ae80 2024-04-01T01:50:03.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effc91e9740 1 -- 172.21.15.70:0/473690566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effb00bd2f0 con 0x56045334ae80 2024-04-01T01:50:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7eff9f7fe640 1 -- 172.21.15.70:0/473690566 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7effb400fc60 con 0x56045334ae80 2024-04-01T01:50:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effc91e9740 10 client.20220 fetch_fsmap learned FSMap version 33 2024-04-01T01:50:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effc91e9740 1 -- 172.21.15.70:0/473690566 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56045308d7d0 con 0x56045334ae80 2024-04-01T01:50:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7eff9f7fe640 1 -- 172.21.15.70:0/473690566 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effb00bd2f0 con 0x56045334ae80 2024-04-01T01:50:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7eff9f7fe640 1 -- 172.21.15.70:0/473690566 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7effb401c420 con 0x56045334ae80 2024-04-01T01:50:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effc91e9740 10 client.20220 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:50:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effc91e9740 10 client.20220 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:50:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effc91e9740 20 client.20220 populate_metadata read hostname 'smithi070' 2024-04-01T01:50:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effc91e9740 1 --2- 172.21.15.70:0/473690566 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5604533602b0 0x560453362710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effc91e9740 4 client.20220 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:50:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.651+0000 7effc91e9740 1 -- 172.21.15.70:0/473690566 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604532bc6f0 con 0x5604533602b0 2024-04-01T01:50:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.652+0000 7effbffff640 1 --2- 172.21.15.70:0/473690566 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5604533602b0 0x560453362710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.652+0000 7effbffff640 1 --2- 172.21.15.70:0/473690566 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5604533602b0 0x560453362710 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=74c8cea803cd7499 server_cookie=acca15fd5badd235 in_seq=0 out_seq=0 2024-04-01T01:50:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.652+0000 7eff9f7fe640 10 client.20220 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:50:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.653+0000 7eff9f7fe640 1 -- 172.21.15.70:0/473690566 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5604532bc6f0 con 0x5604533602b0 2024-04-01T01:50:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.653+0000 7eff9f7fe640 10 client.20220 handle_command_reply: tid=0 2024-04-01T01:50:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.653+0000 7effc91e9740 1 client.20220 shutdown 2024-04-01T01:50:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.653+0000 7effc91e9740 1 -- 172.21.15.70:0/473690566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334ae80 msgr2=0x560453354c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.653+0000 7effc91e9740 1 --2- 172.21.15.70:0/473690566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334ae80 0x560453354c40 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7effb4012700 tx=0x7effb4012bc0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.653+0000 7effc91e9740 1 -- 172.21.15.70:0/473690566 shutdown_connections 2024-04-01T01:50:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.653+0000 7effc91e9740 1 --2- 172.21.15.70:0/473690566 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5604533602b0 0x560453362710 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.653+0000 7effc91e9740 1 --2- 172.21.15.70:0/473690566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334fff0 0x56045335bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.653+0000 7eff9f7fe640 0 client.20220 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:50:03.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.653+0000 7effc91e9740 1 --2- 172.21.15.70:0/473690566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334ae80 0x560453354c40 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.653+0000 7effc91e9740 1 --2- 172.21.15.70:0/473690566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334a490 0x560453354700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.653+0000 7effc91e9740 1 -- 172.21.15.70:0/473690566 >> 172.21.15.70:0/473690566 conn(0x560453345a00 msgr2=0x560453344d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.653+0000 7effc91e9740 1 -- 172.21.15.70:0/473690566 shutdown_connections 2024-04-01T01:50:03.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.653+0000 7effc91e9740 1 -- 172.21.15.70:0/473690566 wait complete. 2024-04-01T01:50:03.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.653+0000 7effc91e9740 20 client.20220 trim_cache size 0 max 16384 2024-04-01T01:50:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.666+0000 7effc91e9740 1 Processor -- start 2024-04-01T01:50:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.666+0000 7effc91e9740 1 -- start start 2024-04-01T01:50:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.666+0000 7effc91e9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334a490 0x5604533444b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.666+0000 7effc91e9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334ae80 0x56045335bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.666+0000 7effc91e9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334fff0 0x5604533449f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.666+0000 7effc91e9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7effc01ac630 con 0x56045334fff0 2024-04-01T01:50:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.666+0000 7effc91e9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7effc01ae170 con 0x56045334ae80 2024-04-01T01:50:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.666+0000 7effc91e9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7effc01acd00 con 0x56045334a490 2024-04-01T01:50:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.666+0000 7effc4ac2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334fff0 0x5604533449f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.666+0000 7effc4ac2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334fff0 0x5604533449f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60304/0 (socket says 172.21.15.70:60304) 2024-04-01T01:50:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.666+0000 7effc4ac2640 1 -- 172.21.15.70:0/401846666 learned_addr learned my addr 172.21.15.70:0/401846666 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.666+0000 7effbffff640 1 --2- 172.21.15.70:0/401846666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334a490 0x5604533444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.666+0000 7effbf7fe640 1 --2- 172.21.15.70:0/401846666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334ae80 0x56045335bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.667+0000 7effc4ac2640 1 -- 172.21.15.70:0/401846666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334a490 msgr2=0x5604533444b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.667+0000 7effc4ac2640 1 --2- 172.21.15.70:0/401846666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334a490 0x5604533444b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.667+0000 7effc4ac2640 1 -- 172.21.15.70:0/401846666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334ae80 msgr2=0x56045335bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.667+0000 7effc4ac2640 1 --2- 172.21.15.70:0/401846666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334ae80 0x56045335bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.667+0000 7effc4ac2640 1 -- 172.21.15.70:0/401846666 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56045334f6d0 con 0x56045334fff0 2024-04-01T01:50:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.667+0000 7effc4ac2640 1 --2- 172.21.15.70:0/401846666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334fff0 0x5604533449f0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7effb800fe10 tx=0x7effb8009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1347b693452aa3d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.667+0000 7effa5ffb640 1 -- 172.21.15.70:0/401846666 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb801a040 con 0x56045334fff0 2024-04-01T01:50:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.667+0000 7effa5ffb640 1 -- 172.21.15.70:0/401846666 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effb8018040 con 0x56045334fff0 2024-04-01T01:50:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.667+0000 7effa5ffb640 1 -- 172.21.15.70:0/401846666 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb800e3b0 con 0x56045334fff0 2024-04-01T01:50:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.667+0000 7effc91e9740 1 -- 172.21.15.70:0/401846666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334fff0 msgr2=0x5604533449f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.667+0000 7effc91e9740 1 --2- 172.21.15.70:0/401846666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334fff0 0x5604533449f0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7effb800fe10 tx=0x7effb8009500 comp rx=0 tx=0).stop 2024-04-01T01:50:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.667+0000 7effc91e9740 1 -- 172.21.15.70:0/401846666 shutdown_connections 2024-04-01T01:50:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.667+0000 7effc91e9740 1 --2- 172.21.15.70:0/401846666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334fff0 0x5604533449f0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.667+0000 7effc91e9740 1 --2- 172.21.15.70:0/401846666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334ae80 0x56045335bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.667+0000 7effc91e9740 1 --2- 172.21.15.70:0/401846666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334a490 0x5604533444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.667+0000 7effc91e9740 1 -- 172.21.15.70:0/401846666 >> 172.21.15.70:0/401846666 conn(0x560453345a00 msgr2=0x56045334c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.668+0000 7effc91e9740 1 -- 172.21.15.70:0/401846666 shutdown_connections 2024-04-01T01:50:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.668+0000 7effc91e9740 1 -- 172.21.15.70:0/401846666 wait complete. 2024-04-01T01:50:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.668+0000 7effc91e9740 1 -- start start 2024-04-01T01:50:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.668+0000 7effc91e9740 1 Processor -- start 2024-04-01T01:50:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.668+0000 7effc91e9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334a490 0x560453359020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.668+0000 7effc91e9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334ae80 0x560453359560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.668+0000 7effc91e9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334fff0 0x56045335a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.668+0000 7effc91e9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7effc01adaa0 con 0x56045334fff0 2024-04-01T01:50:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.668+0000 7effc91e9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7effc01ad3d0 con 0x56045334ae80 2024-04-01T01:50:03.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.668+0000 7effc91e9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7effc01aa610 con 0x56045334a490 2024-04-01T01:50:03.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.668+0000 7effc91e9740 10 client.0 _ll_register_callbacks cb 0x7effb4009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:50:03.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.668+0000 7effc4ac2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334fff0 0x56045335a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.668+0000 7effbffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334a490 0x560453359020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.668+0000 7effc4ac2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334fff0 0x56045335a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60310/0 (socket says 172.21.15.70:60310) 2024-04-01T01:50:03.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.668+0000 7effc4ac2640 1 -- 172.21.15.70:0/391135335 learned_addr learned my addr 172.21.15.70:0/391135335 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.668+0000 7effbf7fe640 1 --2- 172.21.15.70:0/391135335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334ae80 0x560453359560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7effc4ac2640 1 -- 172.21.15.70:0/391135335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334a490 msgr2=0x560453359020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7effc4ac2640 1 --2- 172.21.15.70:0/391135335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334a490 0x560453359020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7effc4ac2640 1 -- 172.21.15.70:0/391135335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334ae80 msgr2=0x560453359560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7effc4ac2640 1 --2- 172.21.15.70:0/391135335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334ae80 0x560453359560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7effc4ac2640 1 -- 172.21.15.70:0/391135335 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56045335ab60 con 0x56045334fff0 2024-04-01T01:50:03.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7effc4ac2640 1 --2- 172.21.15.70:0/391135335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334fff0 0x56045335a570 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7effb8014010 tx=0x7effb800fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d4ccef42621ea6 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7eff9f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:50:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7eff9f7fe640 1 -- 172.21.15.70:0/391135335 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb801a050 con 0x56045334fff0 2024-04-01T01:50:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7eff9f7fe640 1 -- 172.21.15.70:0/391135335 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7effb8021dd0 con 0x56045334fff0 2024-04-01T01:50:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7effc91e9740 1 -- 172.21.15.70:0/391135335 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effb8018040 con 0x56045334fff0 2024-04-01T01:50:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7eff9f7fe640 1 -- 172.21.15.70:0/391135335 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7effb803f050 con 0x56045334fff0 2024-04-01T01:50:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7effc91e9740 10 client.20226 fetch_fsmap learned FSMap version 33 2024-04-01T01:50:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7effc91e9740 1 -- 172.21.15.70:0/391135335 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56045335bed0 con 0x56045334fff0 2024-04-01T01:50:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7eff9f7fe640 1 -- 172.21.15.70:0/391135335 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7effb8018040 con 0x56045334fff0 2024-04-01T01:50:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7eff9f7fe640 1 -- 172.21.15.70:0/391135335 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7effb8021750 con 0x56045334fff0 2024-04-01T01:50:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7effc91e9740 10 client.20226 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:50:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7effc91e9740 10 client.20226 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-01T01:50:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7effc91e9740 20 client.20226 populate_metadata read hostname 'smithi070' 2024-04-01T01:50:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7effc91e9740 1 --2- 172.21.15.70:0/391135335 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5604533c6da0 0x5604533c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7effc91e9740 4 client.20226 mds_command: new command op to 24404 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:50:03.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.669+0000 7effc91e9740 1 -- 172.21.15.70:0/391135335 --> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5604532b6a40 con 0x5604533c6da0 2024-04-01T01:50:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7effbffff640 1 --2- 172.21.15.70:0/391135335 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5604533c6da0 0x5604533c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7eff9f7fe640 10 client.20226 ms_handle_connect on v2:172.21.15.81:6834/970944957 2024-04-01T01:50:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7effbffff640 1 --2- 172.21.15.70:0/391135335 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5604533c6da0 0x5604533c91c0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b0abaf4c6c579406 server_cookie=bb7ac06e01f9cb2 in_seq=0 out_seq=0 2024-04-01T01:50:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7eff9f7fe640 1 -- 172.21.15.70:0/391135335 <== mds.3 v2:172.21.15.81:6834/970944957 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5604532b6a40 con 0x5604533c6da0 2024-04-01T01:50:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7eff9f7fe640 10 client.20226 handle_command_reply: tid=0 2024-04-01T01:50:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7effc91e9740 1 client.20226 shutdown 2024-04-01T01:50:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7effc91e9740 1 -- 172.21.15.70:0/391135335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334fff0 msgr2=0x56045335a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7effc91e9740 1 --2- 172.21.15.70:0/391135335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334fff0 0x56045335a570 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7effb8014010 tx=0x7effb800fab0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7effc91e9740 1 -- 172.21.15.70:0/391135335 shutdown_connections 2024-04-01T01:50:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7effc91e9740 1 --2- 172.21.15.70:0/391135335 >> [v2:172.21.15.81:6834/970944957,v1:172.21.15.81:6835/970944957] conn(0x5604533c6da0 0x5604533c91c0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7effc91e9740 1 --2- 172.21.15.70:0/391135335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56045334fff0 0x56045335a570 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7effc91e9740 1 --2- 172.21.15.70:0/391135335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56045334ae80 0x560453359560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7eff9f7fe640 0 client.20226 ms_handle_reset on v2:172.21.15.81:6834/970944957 2024-04-01T01:50:03.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7effc91e9740 1 --2- 172.21.15.70:0/391135335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56045334a490 0x560453359020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7effc91e9740 1 -- 172.21.15.70:0/391135335 >> 172.21.15.70:0/391135335 conn(0x560453345a00 msgr2=0x560453348c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7effc91e9740 1 -- 172.21.15.70:0/391135335 shutdown_connections 2024-04-01T01:50:03.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7effc91e9740 1 -- 172.21.15.70:0/391135335 wait complete. 2024-04-01T01:50:03.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.670+0000 7effc91e9740 20 client.20226 trim_cache size 0 max 16384 2024-04-01T01:50:03.678 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:50:03.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.671+0000 7effc6525640 1 -- 172.21.15.70:0/945459782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7effa80a24d0 msgr2=0x7effa80a4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.671+0000 7effc6525640 1 --2- 172.21.15.70:0/945459782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7effa80a24d0 0x7effa80a4990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7effb8005fd0 tx=0x7effb8011bc0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.671+0000 7effc6525640 1 -- 172.21.15.70:0/945459782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effc0100650 msgr2=0x7effc0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.671+0000 7effc6525640 1 --2- 172.21.15.70:0/945459782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effc0100650 0x7effc0069420 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7effb0000c00 tx=0x7effb0005dc0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.671+0000 7effc6525640 1 -- 172.21.15.70:0/945459782 shutdown_connections 2024-04-01T01:50:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.671+0000 7effc6525640 1 --2- 172.21.15.70:0/945459782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effc0102200 0x7effc01a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.671+0000 7effc6525640 1 --2- 172.21.15.70:0/945459782 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7effa80a24d0 0x7effa80a4990 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.671+0000 7effc6525640 1 --2- 172.21.15.70:0/945459782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effc0101840 0x7effc019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.671+0000 7effc6525640 1 --2- 172.21.15.70:0/945459782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7effc0100650 0x7effc0069420 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.671+0000 7effc6525640 1 -- 172.21.15.70:0/945459782 >> 172.21.15.70:0/945459782 conn(0x7effc00fbbe0 msgr2=0x7effc0104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.671+0000 7effc6525640 1 -- 172.21.15.70:0/945459782 shutdown_connections 2024-04-01T01:50:03.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.671+0000 7effc6525640 1 -- 172.21.15.70:0/945459782 wait complete. 2024-04-01T01:50:03.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-01T01:50:03.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 -- 172.21.15.70:0/2318285021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7230102200 msgr2=0x7f7230108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 --2- 172.21.15.70:0/2318285021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7230102200 0x7f7230108a90 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f7228009ca0 tx=0x7f722802c0f0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 -- 172.21.15.70:0/2318285021 shutdown_connections 2024-04-01T01:50:03.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 --2- 172.21.15.70:0/2318285021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7230102200 0x7f7230108a90 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 --2- 172.21.15.70:0/2318285021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7230101840 0x7f7230101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 --2- 172.21.15.70:0/2318285021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7230100650 0x7f7230100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 -- 172.21.15.70:0/2318285021 >> 172.21.15.70:0/2318285021 conn(0x7f72300fbbe0 msgr2=0x7f72300fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 -- 172.21.15.70:0/2318285021 shutdown_connections 2024-04-01T01:50:03.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 -- 172.21.15.70:0/2318285021 wait complete. 2024-04-01T01:50:03.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 Processor -- start 2024-04-01T01:50:03.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 -- start start 2024-04-01T01:50:03.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7230100650 0x7f7230069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7230101840 0x7f723019c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7230102200 0x7f723019eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f723010c160 con 0x7f7230101840 2024-04-01T01:50:03.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f723010bfe0 con 0x7f7230102200 2024-04-01T01:50:03.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f723713e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f723010c2e0 con 0x7f7230100650 2024-04-01T01:50:03.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f722ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7230101840 0x7f723019c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f722ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7230101840 0x7f723019c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60338/0 (socket says 172.21.15.70:60338) 2024-04-01T01:50:03.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f722ffff640 1 -- 172.21.15.70:0/3600409138 learned_addr learned my addr 172.21.15.70:0/3600409138 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.787+0000 7f7234eda640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7230100650 0x7f7230069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.788+0000 7f72356db640 1 --2- 172.21.15.70:0/3600409138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7230102200 0x7f723019eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.788+0000 7f722ffff640 1 -- 172.21.15.70:0/3600409138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7230100650 msgr2=0x7f7230069420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.788+0000 7f722ffff640 1 --2- 172.21.15.70:0/3600409138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7230100650 0x7f7230069420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.788+0000 7f722ffff640 1 -- 172.21.15.70:0/3600409138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7230102200 msgr2=0x7f723019eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.788+0000 7f722ffff640 1 --2- 172.21.15.70:0/3600409138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7230102200 0x7f723019eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.788+0000 7f722ffff640 1 -- 172.21.15.70:0/3600409138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f723019f5b0 con 0x7f7230101840 2024-04-01T01:50:03.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.788+0000 7f7234eda640 1 --2- 172.21.15.70:0/3600409138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7230100650 0x7f7230069420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:03.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.788+0000 7f722ffff640 1 --2- 172.21.15.70:0/3600409138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7230101840 0x7f723019c680 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f721c00e9f0 tx=0x7f721c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.788+0000 7f722dffb640 1 -- 172.21.15.70:0/3600409138 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721c00cd80 con 0x7f7230101840 2024-04-01T01:50:03.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.788+0000 7f722dffb640 1 -- 172.21.15.70:0/3600409138 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f721c0108e0 con 0x7f7230101840 2024-04-01T01:50:03.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.788+0000 7f723713e640 1 -- 172.21.15.70:0/3600409138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f723019f840 con 0x7f7230101840 2024-04-01T01:50:03.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.788+0000 7f722dffb640 1 -- 172.21.15.70:0/3600409138 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f721c018920 con 0x7f7230101840 2024-04-01T01:50:03.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.788+0000 7f723713e640 1 -- 172.21.15.70:0/3600409138 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f723019fd90 con 0x7f7230101840 2024-04-01T01:50:03.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.790+0000 7f722dffb640 1 -- 172.21.15.70:0/3600409138 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f721c010a90 con 0x7f7230101840 2024-04-01T01:50:03.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f7239df9740 1 Processor -- start 2024-04-01T01:50:03.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f7239df9740 1 -- start start 2024-04-01T01:50:03.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f7239df9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda633490 0x556bda6338b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f7239df9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda633e80 0x556bda638980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f7239df9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda638ff0 0x556bda63b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f7239df9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556bda5981b0 con 0x556bda633e80 2024-04-01T01:50:03.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f7239df9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x556bda540c00 con 0x556bda638ff0 2024-04-01T01:50:03.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f7239df9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556bda594250 con 0x556bda633490 2024-04-01T01:50:03.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f722dffb640 1 --2- 172.21.15.70:0/3600409138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72200a24a0 0x7f72200a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f7234eda640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda633e80 0x556bda638980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f7234eda640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda633e80 0x556bda638980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60346/0 (socket says 172.21.15.70:60346) 2024-04-01T01:50:03.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f7234eda640 1 -- 172.21.15.70:0/1669881814 learned_addr learned my addr 172.21.15.70:0/1669881814 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f72356db640 1 --2- 172.21.15.70:0/1669881814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda633490 0x556bda6338b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f722ffff640 1 --2- 172.21.15.70:0/1669881814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda638ff0 0x556bda63b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f72356db640 1 --2- 172.21.15.70:0/3600409138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72200a24a0 0x7f72200a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f7234eda640 1 -- 172.21.15.70:0/1669881814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda633490 msgr2=0x556bda6338b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f7234eda640 1 --2- 172.21.15.70:0/1669881814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda633490 0x556bda6338b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f7234eda640 1 -- 172.21.15.70:0/1669881814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda638ff0 msgr2=0x556bda63b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f722dffb640 1 -- 172.21.15.70:0/3600409138 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f721c0cedc0 con 0x7f7230101840 2024-04-01T01:50:03.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f7234eda640 1 --2- 172.21.15.70:0/1669881814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda638ff0 0x556bda63b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f7234eda640 1 -- 172.21.15.70:0/1669881814 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bda5603d0 con 0x556bda633e80 2024-04-01T01:50:03.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f722ffff640 1 --2- 172.21.15.70:0/1669881814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda638ff0 0x556bda63b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:03.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.791+0000 7f72356db640 1 --2- 172.21.15.70:0/3600409138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72200a24a0 0x7f72200a4960 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f722802c630 tx=0x7f7228009360 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.792+0000 7f7234eda640 1 --2- 172.21.15.70:0/1669881814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda633e80 0x556bda638980 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f72240114c0 tx=0x7f7224011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11852ff03807feff server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.792+0000 7f721affd640 1 -- 172.21.15.70:0/1669881814 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7224004020 con 0x556bda633e80 2024-04-01T01:50:03.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.792+0000 7f721affd640 1 -- 172.21.15.70:0/1669881814 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72240041c0 con 0x556bda633e80 2024-04-01T01:50:03.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.792+0000 7f721affd640 1 -- 172.21.15.70:0/1669881814 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722400f820 con 0x556bda633e80 2024-04-01T01:50:03.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.792+0000 7f7239df9740 1 -- 172.21.15.70:0/1669881814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda633e80 msgr2=0x556bda638980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.792+0000 7f7239df9740 1 --2- 172.21.15.70:0/1669881814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda633e80 0x556bda638980 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f72240114c0 tx=0x7f7224011980 comp rx=0 tx=0).stop 2024-04-01T01:50:03.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.792+0000 7f7239df9740 1 -- 172.21.15.70:0/1669881814 shutdown_connections 2024-04-01T01:50:03.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.792+0000 7f7239df9740 1 --2- 172.21.15.70:0/1669881814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda638ff0 0x556bda63b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.792+0000 7f7239df9740 1 --2- 172.21.15.70:0/1669881814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda633e80 0x556bda638980 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.792+0000 7f7239df9740 1 --2- 172.21.15.70:0/1669881814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda633490 0x556bda6338b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.792+0000 7f7239df9740 1 -- 172.21.15.70:0/1669881814 >> 172.21.15.70:0/1669881814 conn(0x556bda62ea00 msgr2=0x556bda62ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.792+0000 7f7239df9740 1 -- 172.21.15.70:0/1669881814 shutdown_connections 2024-04-01T01:50:03.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.793+0000 7f7239df9740 1 -- 172.21.15.70:0/1669881814 wait complete. 2024-04-01T01:50:03.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.793+0000 7f7239df9740 1 -- start start 2024-04-01T01:50:03.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.793+0000 7f7239df9740 1 Processor -- start 2024-04-01T01:50:03.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.793+0000 7f7239df9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda633490 0x556bda63d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.793+0000 7f7239df9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda633e80 0x556bda63dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.793+0000 7f7239df9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda638ff0 0x556bda644f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.793+0000 7f7239df9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556bda63bfa0 con 0x556bda638ff0 2024-04-01T01:50:03.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.793+0000 7f7239df9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x556bda536470 con 0x556bda633490 2024-04-01T01:50:03.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.793+0000 7f7239df9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556bda63c120 con 0x556bda633e80 2024-04-01T01:50:03.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.793+0000 7f7239df9740 10 client.0 _ll_register_callbacks cb 0x556bda553cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:50:03.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.793+0000 7f72356db640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda638ff0 0x556bda644f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.793+0000 7f7234eda640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda633490 0x556bda63d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.793+0000 7f72356db640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda638ff0 0x556bda644f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60362/0 (socket says 172.21.15.70:60362) 2024-04-01T01:50:03.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.793+0000 7f72356db640 1 -- 172.21.15.70:0/1120834918 learned_addr learned my addr 172.21.15.70:0/1120834918 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.793+0000 7f722ffff640 1 --2- 172.21.15.70:0/1120834918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda633e80 0x556bda63dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f72356db640 1 -- 172.21.15.70:0/1120834918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda633e80 msgr2=0x556bda63dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f72356db640 1 --2- 172.21.15.70:0/1120834918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda633e80 0x556bda63dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f72356db640 1 -- 172.21.15.70:0/1120834918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda633490 msgr2=0x556bda63d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f72356db640 1 --2- 172.21.15.70:0/1120834918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda633490 0x556bda63d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f72356db640 1 -- 172.21.15.70:0/1120834918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bda2ac3f0 con 0x556bda638ff0 2024-04-01T01:50:03.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f72356db640 1 --2- 172.21.15.70:0/1120834918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda638ff0 0x556bda644f00 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f7228009ca0 tx=0x7f72280095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea05b9bb93b664e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f720ffff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:50:03.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f720ffff640 1 -- 172.21.15.70:0/1120834918 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7228054060 con 0x556bda638ff0 2024-04-01T01:50:03.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f720ffff640 1 -- 172.21.15.70:0/1120834918 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f7228032850 con 0x556bda638ff0 2024-04-01T01:50:03.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f7239df9740 1 -- 172.21.15.70:0/1120834918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72240041c0 con 0x556bda638ff0 2024-04-01T01:50:03.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f720ffff640 1 -- 172.21.15.70:0/1120834918 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7228039040 con 0x556bda638ff0 2024-04-01T01:50:03.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f7239df9740 10 client.20238 fetch_fsmap learned FSMap version 33 2024-04-01T01:50:03.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f7239df9740 1 -- 172.21.15.70:0/1120834918 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556bda3767d0 con 0x556bda638ff0 2024-04-01T01:50:03.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f720ffff640 1 -- 172.21.15.70:0/1120834918 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72240041c0 con 0x556bda638ff0 2024-04-01T01:50:03.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f720ffff640 1 -- 172.21.15.70:0/1120834918 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f722804f830 con 0x556bda638ff0 2024-04-01T01:50:03.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f7239df9740 10 client.20238 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:50:03.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.794+0000 7f7239df9740 10 client.20238 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:50:03.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.795+0000 7f7239df9740 20 client.20238 populate_metadata read hostname 'smithi070' 2024-04-01T01:50:03.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.795+0000 7f7239df9740 1 --2- 172.21.15.70:0/1120834918 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x556bda649520 0x556bda64b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.795+0000 7f7239df9740 4 client.20238 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:50:03.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.795+0000 7f7239df9740 1 -- 172.21.15.70:0/1120834918 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556bda5a56f0 con 0x556bda649520 2024-04-01T01:50:03.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.795+0000 7f7234eda640 1 --2- 172.21.15.70:0/1120834918 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x556bda649520 0x556bda64b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.795+0000 7f720ffff640 10 client.20238 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:50:03.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.795+0000 7f7234eda640 1 --2- 172.21.15.70:0/1120834918 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x556bda649520 0x556bda64b9b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=abc1731f23d6e154 server_cookie=d565e0d251821fbb in_seq=0 out_seq=0 2024-04-01T01:50:03.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.796+0000 7f720ffff640 1 -- 172.21.15.70:0/1120834918 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556bda5a56f0 con 0x556bda649520 2024-04-01T01:50:03.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.796+0000 7f720ffff640 10 client.20238 handle_command_reply: tid=0 2024-04-01T01:50:03.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.796+0000 7f7239df9740 1 client.20238 shutdown 2024-04-01T01:50:03.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.796+0000 7f7239df9740 1 -- 172.21.15.70:0/1120834918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda638ff0 msgr2=0x556bda644f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.796+0000 7f7239df9740 1 --2- 172.21.15.70:0/1120834918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda638ff0 0x556bda644f00 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f7228009ca0 tx=0x7f72280095c0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.796+0000 7f7239df9740 1 -- 172.21.15.70:0/1120834918 shutdown_connections 2024-04-01T01:50:03.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.796+0000 7f7239df9740 1 --2- 172.21.15.70:0/1120834918 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x556bda649520 0x556bda64b9b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.796+0000 7f7239df9740 1 --2- 172.21.15.70:0/1120834918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda638ff0 0x556bda644f00 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.796+0000 7f720ffff640 0 client.20238 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:50:03.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.796+0000 7f7239df9740 1 --2- 172.21.15.70:0/1120834918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda633e80 0x556bda63dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.796+0000 7f7239df9740 1 --2- 172.21.15.70:0/1120834918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda633490 0x556bda63d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.796+0000 7f7239df9740 1 -- 172.21.15.70:0/1120834918 >> 172.21.15.70:0/1120834918 conn(0x556bda62ea00 msgr2=0x556bda62dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.796+0000 7f7239df9740 1 -- 172.21.15.70:0/1120834918 shutdown_connections 2024-04-01T01:50:03.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.796+0000 7f7239df9740 1 -- 172.21.15.70:0/1120834918 wait complete. 2024-04-01T01:50:03.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.796+0000 7f7239df9740 20 client.20238 trim_cache size 0 max 16384 2024-04-01T01:50:03.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.809+0000 7f7239df9740 1 Processor -- start 2024-04-01T01:50:03.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.809+0000 7f7239df9740 1 -- start start 2024-04-01T01:50:03.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f7239df9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda633490 0x556bda62d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f7239df9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda633e80 0x556bda644ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f7239df9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda638ff0 0x556bda62d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f7239df9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f72301aa410 con 0x556bda638ff0 2024-04-01T01:50:03.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f7239df9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f72301abf50 con 0x556bda633e80 2024-04-01T01:50:03.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f7239df9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f72301aaae0 con 0x556bda633490 2024-04-01T01:50:03.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f72356db640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda638ff0 0x556bda62d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f722ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda633e80 0x556bda644ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f72356db640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda638ff0 0x556bda62d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60364/0 (socket says 172.21.15.70:60364) 2024-04-01T01:50:03.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f72356db640 1 -- 172.21.15.70:0/557601673 learned_addr learned my addr 172.21.15.70:0/557601673 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f7234eda640 1 --2- 172.21.15.70:0/557601673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda633490 0x556bda62d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f72356db640 1 -- 172.21.15.70:0/557601673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda633490 msgr2=0x556bda62d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f72356db640 1 --2- 172.21.15.70:0/557601673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda633490 0x556bda62d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f72356db640 1 -- 172.21.15.70:0/557601673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda633e80 msgr2=0x556bda644ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f72356db640 1 --2- 172.21.15.70:0/557601673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda633e80 0x556bda644ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f72356db640 1 -- 172.21.15.70:0/557601673 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bda63ca60 con 0x556bda638ff0 2024-04-01T01:50:03.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f72356db640 1 --2- 172.21.15.70:0/557601673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda638ff0 0x556bda62d9f0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f722804fc70 tx=0x7f7228009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b9afcd98a445f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f721a7fc640 1 -- 172.21.15.70:0/557601673 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7228054070 con 0x556bda638ff0 2024-04-01T01:50:03.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f721a7fc640 1 -- 172.21.15.70:0/557601673 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72280328c0 con 0x556bda638ff0 2024-04-01T01:50:03.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.810+0000 7f721a7fc640 1 -- 172.21.15.70:0/557601673 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72280346a0 con 0x556bda638ff0 2024-04-01T01:50:03.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.811+0000 7f7239df9740 1 -- 172.21.15.70:0/557601673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda638ff0 msgr2=0x556bda62d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.811+0000 7f7239df9740 1 --2- 172.21.15.70:0/557601673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda638ff0 0x556bda62d9f0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f722804fc70 tx=0x7f7228009b40 comp rx=0 tx=0).stop 2024-04-01T01:50:03.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.811+0000 7f7239df9740 1 -- 172.21.15.70:0/557601673 shutdown_connections 2024-04-01T01:50:03.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.811+0000 7f7239df9740 1 --2- 172.21.15.70:0/557601673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda638ff0 0x556bda62d9f0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.811+0000 7f7239df9740 1 --2- 172.21.15.70:0/557601673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda633e80 0x556bda644ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.811+0000 7f7239df9740 1 --2- 172.21.15.70:0/557601673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda633490 0x556bda62d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.811+0000 7f7239df9740 1 -- 172.21.15.70:0/557601673 >> 172.21.15.70:0/557601673 conn(0x556bda62ea00 msgr2=0x556bda62ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.811+0000 7f7239df9740 1 -- 172.21.15.70:0/557601673 shutdown_connections 2024-04-01T01:50:03.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.811+0000 7f7239df9740 1 -- 172.21.15.70:0/557601673 wait complete. 2024-04-01T01:50:03.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.811+0000 7f7239df9740 1 -- start start 2024-04-01T01:50:03.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.811+0000 7f7239df9740 1 Processor -- start 2024-04-01T01:50:03.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7239df9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda633490 0x556bda644880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7239df9740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda633e80 0x556bda644dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7239df9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda638ff0 0x556bda642e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7239df9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f72301ab880 con 0x556bda633490 2024-04-01T01:50:03.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7239df9740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f72301ab1b0 con 0x556bda633e80 2024-04-01T01:50:03.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7239df9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f72301a83f0 con 0x556bda638ff0 2024-04-01T01:50:03.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7239df9740 10 client.0 _ll_register_callbacks cb 0x7f7228003ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:50:03.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7234eda640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda633490 0x556bda644880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7234eda640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda633490 0x556bda644880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60368/0 (socket says 172.21.15.70:60368) 2024-04-01T01:50:03.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f722ffff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda633e80 0x556bda644dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7234eda640 1 -- 172.21.15.70:0/1230382233 learned_addr learned my addr 172.21.15.70:0/1230382233 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f72356db640 1 --2- 172.21.15.70:0/1230382233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda638ff0 0x556bda642e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7234eda640 1 -- 172.21.15.70:0/1230382233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda638ff0 msgr2=0x556bda642e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7234eda640 1 --2- 172.21.15.70:0/1230382233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda638ff0 0x556bda642e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7234eda640 1 -- 172.21.15.70:0/1230382233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda633e80 msgr2=0x556bda644dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7234eda640 1 --2- 172.21.15.70:0/1230382233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda633e80 0x556bda644dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7234eda640 1 -- 172.21.15.70:0/1230382233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bda643620 con 0x556bda633490 2024-04-01T01:50:03.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f722ffff640 1 --2- 172.21.15.70:0/1230382233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda633e80 0x556bda644dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:03.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7234eda640 1 --2- 172.21.15.70:0/1230382233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda633490 0x556bda644880 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f7224009fd0 tx=0x7f7224005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a9d9294b3cd3b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f720ffff640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:50:03.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f720ffff640 1 -- 172.21.15.70:0/1230382233 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722401d040 con 0x556bda633490 2024-04-01T01:50:03.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f720ffff640 1 -- 172.21.15.70:0/1230382233 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f7224011040 con 0x556bda633490 2024-04-01T01:50:03.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7239df9740 1 -- 172.21.15.70:0/1230382233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72280328c0 con 0x556bda633490 2024-04-01T01:50:03.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f720ffff640 1 -- 172.21.15.70:0/1230382233 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7224039050 con 0x556bda633490 2024-04-01T01:50:03.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7239df9740 10 client.20244 fetch_fsmap learned FSMap version 33 2024-04-01T01:50:03.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.812+0000 7f7239df9740 1 -- 172.21.15.70:0/1230382233 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556bda6b03a0 con 0x556bda633490 2024-04-01T01:50:03.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.813+0000 7f720ffff640 1 -- 172.21.15.70:0/1230382233 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f72280328c0 con 0x556bda633490 2024-04-01T01:50:03.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.813+0000 7f720ffff640 1 -- 172.21.15.70:0/1230382233 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f722402b3b0 con 0x556bda633490 2024-04-01T01:50:03.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.813+0000 7f7239df9740 10 client.20244 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:50:03.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.813+0000 7f7239df9740 10 client.20244 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-04-01T01:50:03.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.813+0000 7f7239df9740 20 client.20244 populate_metadata read hostname 'smithi070' 2024-04-01T01:50:03.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.813+0000 7f7239df9740 1 --2- 172.21.15.70:0/1230382233 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x556bda6b0bb0 0x556bda6b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.813+0000 7f7239df9740 4 client.20244 mds_command: new command op to 24418 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:50:03.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.813+0000 7f7239df9740 1 -- 172.21.15.70:0/1230382233 --> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556bda59fa40 con 0x556bda6b0bb0 2024-04-01T01:50:03.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.813+0000 7f722ffff640 1 --2- 172.21.15.70:0/1230382233 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x556bda6b0bb0 0x556bda6b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.813+0000 7f720ffff640 10 client.20244 ms_handle_connect on v2:172.21.15.189:6834/3861912911 2024-04-01T01:50:03.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.813+0000 7f722ffff640 1 --2- 172.21.15.70:0/1230382233 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x556bda6b0bb0 0x556bda6b2fd0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d19363ed80c0d29d server_cookie=1b78120661ef6a9 in_seq=0 out_seq=0 2024-04-01T01:50:03.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f720ffff640 1 -- 172.21.15.70:0/1230382233 <== mds.2 v2:172.21.15.189:6834/3861912911 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556bda59fa40 con 0x556bda6b0bb0 2024-04-01T01:50:03.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f720ffff640 10 client.20244 handle_command_reply: tid=0 2024-04-01T01:50:03.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f7239df9740 1 client.20244 shutdown 2024-04-01T01:50:03.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f7239df9740 1 -- 172.21.15.70:0/1230382233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda633490 msgr2=0x556bda644880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f7239df9740 1 --2- 172.21.15.70:0/1230382233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda633490 0x556bda644880 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f7224009fd0 tx=0x7f7224005da0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f7239df9740 1 -- 172.21.15.70:0/1230382233 shutdown_connections 2024-04-01T01:50:03.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f7239df9740 1 --2- 172.21.15.70:0/1230382233 >> [v2:172.21.15.189:6834/3861912911,v1:172.21.15.189:6835/3861912911] conn(0x556bda6b0bb0 0x556bda6b2fd0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f7239df9740 1 --2- 172.21.15.70:0/1230382233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556bda638ff0 0x556bda642e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f7239df9740 1 --2- 172.21.15.70:0/1230382233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556bda633e80 0x556bda644dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f720ffff640 0 client.20244 ms_handle_reset on v2:172.21.15.189:6834/3861912911 2024-04-01T01:50:03.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f7239df9740 1 --2- 172.21.15.70:0/1230382233 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556bda633490 0x556bda644880 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f7239df9740 1 -- 172.21.15.70:0/1230382233 >> 172.21.15.70:0/1230382233 conn(0x556bda62ea00 msgr2=0x556bda631c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f7239df9740 1 -- 172.21.15.70:0/1230382233 shutdown_connections 2024-04-01T01:50:03.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f7239df9740 1 -- 172.21.15.70:0/1230382233 wait complete. 2024-04-01T01:50:03.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f7239df9740 20 client.20244 trim_cache size 0 max 16384 2024-04-01T01:50:03.823 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:50:03.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f723713e640 1 -- 172.21.15.70:0/3600409138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72200a24a0 msgr2=0x7f72200a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f723713e640 1 --2- 172.21.15.70:0/3600409138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72200a24a0 0x7f72200a4960 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f722802c630 tx=0x7f7228009360 comp rx=0 tx=0).stop 2024-04-01T01:50:03.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f723713e640 1 -- 172.21.15.70:0/3600409138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7230101840 msgr2=0x7f723019c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f723713e640 1 --2- 172.21.15.70:0/3600409138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7230101840 0x7f723019c680 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f721c00e9f0 tx=0x7f721c00eeb0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f723713e640 1 -- 172.21.15.70:0/3600409138 shutdown_connections 2024-04-01T01:50:03.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f723713e640 1 --2- 172.21.15.70:0/3600409138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7230102200 0x7f723019eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f723713e640 1 --2- 172.21.15.70:0/3600409138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7230101840 0x7f723019c680 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f723713e640 1 --2- 172.21.15.70:0/3600409138 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f72200a24a0 0x7f72200a4960 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f723713e640 1 --2- 172.21.15.70:0/3600409138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7230100650 0x7f7230069420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.814+0000 7f723713e640 1 -- 172.21.15.70:0/3600409138 >> 172.21.15.70:0/3600409138 conn(0x7f72300fbbe0 msgr2=0x7f7230104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.815+0000 7f723713e640 1 -- 172.21.15.70:0/3600409138 shutdown_connections 2024-04-01T01:50:03.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.815+0000 7f723713e640 1 -- 172.21.15.70:0/3600409138 wait complete. 2024-04-01T01:50:03.826 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-01T01:50:03.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 -- 172.21.15.70:0/1480299095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8f0102200 msgr2=0x7fb8f0108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 --2- 172.21.15.70:0/1480299095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8f0102200 0x7fb8f0108a90 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0009900 tx=0x7fb8e002bc30 comp rx=0 tx=0).stop 2024-04-01T01:50:03.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 -- 172.21.15.70:0/1480299095 shutdown_connections 2024-04-01T01:50:03.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 --2- 172.21.15.70:0/1480299095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8f0102200 0x7fb8f0108a90 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 --2- 172.21.15.70:0/1480299095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f0101840 0x7fb8f0101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 --2- 172.21.15.70:0/1480299095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8f0100650 0x7fb8f0100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 -- 172.21.15.70:0/1480299095 >> 172.21.15.70:0/1480299095 conn(0x7fb8f00fbbe0 msgr2=0x7fb8f00fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 -- 172.21.15.70:0/1480299095 shutdown_connections 2024-04-01T01:50:03.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 -- 172.21.15.70:0/1480299095 wait complete. 2024-04-01T01:50:03.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 Processor -- start 2024-04-01T01:50:03.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 -- start start 2024-04-01T01:50:03.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8f0100650 0x7fb8f01968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8f0101840 0x7fb8f0196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f0102200 0x7fb8f019de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f010c190 con 0x7fb8f0100650 2024-04-01T01:50:03.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f010c010 con 0x7fb8f0101840 2024-04-01T01:50:03.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f7fc7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f010c310 con 0x7fb8f0102200 2024-04-01T01:50:03.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.931+0000 7fb8f5d63640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8f0100650 0x7fb8f01968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8f5d63640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8f0100650 0x7fb8f01968c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60374/0 (socket says 172.21.15.70:60374) 2024-04-01T01:50:03.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8f5d63640 1 -- 172.21.15.70:0/2338307261 learned_addr learned my addr 172.21.15.70:0/2338307261 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8f5562640 1 --2- 172.21.15.70:0/2338307261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8f0101840 0x7fb8f0196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8f6564640 1 --2- 172.21.15.70:0/2338307261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f0102200 0x7fb8f019de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8f5d63640 1 -- 172.21.15.70:0/2338307261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f0102200 msgr2=0x7fb8f019de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8f5d63640 1 --2- 172.21.15.70:0/2338307261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f0102200 0x7fb8f019de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8f5d63640 1 -- 172.21.15.70:0/2338307261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8f0101840 msgr2=0x7fb8f0196e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8f5d63640 1 --2- 172.21.15.70:0/2338307261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8f0101840 0x7fb8f0196e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8f5d63640 1 -- 172.21.15.70:0/2338307261 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8f019e4e0 con 0x7fb8f0100650 2024-04-01T01:50:03.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8f5562640 1 --2- 172.21.15.70:0/2338307261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8f0101840 0x7fb8f0196e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:50:03.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8f6564640 1 --2- 172.21.15.70:0/2338307261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f0102200 0x7fb8f019de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:50:03.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8f5d63640 1 --2- 172.21.15.70:0/2338307261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8f0100650 0x7fb8f01968c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8009d20 tx=0x7fb8e800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8eeffd640 1 -- 172.21.15.70:0/2338307261 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8010630 con 0x7fb8f0100650 2024-04-01T01:50:03.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8eeffd640 1 -- 172.21.15.70:0/2338307261 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8e8010c80 con 0x7fb8f0100650 2024-04-01T01:50:03.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8eeffd640 1 -- 172.21.15.70:0/2338307261 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8018d70 con 0x7fb8f0100650 2024-04-01T01:50:03.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8f7fc7640 1 -- 172.21.15.70:0/2338307261 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8f019e7d0 con 0x7fb8f0100650 2024-04-01T01:50:03.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.932+0000 7fb8f7fc7640 1 -- 172.21.15.70:0/2338307261 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8f019ed70 con 0x7fb8f0100650 2024-04-01T01:50:03.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.934+0000 7fb8eeffd640 1 -- 172.21.15.70:0/2338307261 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb8e801f020 con 0x7fb8f0100650 2024-04-01T01:50:03.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.934+0000 7fb8faba1740 1 Processor -- start 2024-04-01T01:50:03.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8faba1740 1 -- start start 2024-04-01T01:50:03.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8eeffd640 1 --2- 172.21.15.70:0/2338307261 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8d80a2520 0x7fb8d80a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8faba1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12b490 0x56308f12b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8faba1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f12be80 0x56308f130980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8faba1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f130ff0 0x56308f133480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8faba1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56308f0901b0 con 0x56308f12b490 2024-04-01T01:50:03.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8faba1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56308f038c00 con 0x56308f12be80 2024-04-01T01:50:03.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8faba1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56308f08c250 con 0x56308f130ff0 2024-04-01T01:50:03.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8f6564640 1 --2- 172.21.15.70:0/2338307261 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8d80a2520 0x7fb8d80a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8eeffd640 1 -- 172.21.15.70:0/2338307261 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fb8e80cf060 con 0x7fb8f0100650 2024-04-01T01:50:03.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8f5d63640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12b490 0x56308f12b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8f5d63640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12b490 0x56308f12b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60386/0 (socket says 172.21.15.70:60386) 2024-04-01T01:50:03.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8f5d63640 1 -- 172.21.15.70:0/3233237750 learned_addr learned my addr 172.21.15.70:0/3233237750 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8f5562640 1 --2- 172.21.15.70:0/3233237750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f12be80 0x56308f130980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8f6564640 1 --2- 172.21.15.70:0/3233237750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f130ff0 0x56308f133480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8f5d63640 1 -- 172.21.15.70:0/3233237750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f130ff0 msgr2=0x56308f133480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8f5d63640 1 --2- 172.21.15.70:0/3233237750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f130ff0 0x56308f133480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8f5d63640 1 -- 172.21.15.70:0/3233237750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f12be80 msgr2=0x56308f130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8f5d63640 1 --2- 172.21.15.70:0/3233237750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f12be80 0x56308f130980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8f5d63640 1 -- 172.21.15.70:0/3233237750 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56308f0583d0 con 0x56308f12b490 2024-04-01T01:50:03.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.935+0000 7fb8f5562640 1 --2- 172.21.15.70:0/3233237750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f12be80 0x56308f130980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:03.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.936+0000 7fb8f6564640 1 --2- 172.21.15.70:0/2338307261 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8d80a2520 0x7fb8d80a49e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0005ab0 tx=0x7fb8e002c960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.936+0000 7fb8f5d63640 1 --2- 172.21.15.70:0/3233237750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12b490 0x56308f12b8b0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fb8e809b6e0 tx=0x7fb8e809b520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c397c96e35f6dcea server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.936+0000 7fb8d7fff640 1 -- 172.21.15.70:0/3233237750 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8010910 con 0x56308f12b490 2024-04-01T01:50:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.936+0000 7fb8d7fff640 1 -- 172.21.15.70:0/3233237750 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8e801e2c0 con 0x56308f12b490 2024-04-01T01:50:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.936+0000 7fb8d7fff640 1 -- 172.21.15.70:0/3233237750 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8084ce0 con 0x56308f12b490 2024-04-01T01:50:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.936+0000 7fb8faba1740 1 -- 172.21.15.70:0/3233237750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12b490 msgr2=0x56308f12b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.936+0000 7fb8faba1740 1 --2- 172.21.15.70:0/3233237750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12b490 0x56308f12b8b0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fb8e809b6e0 tx=0x7fb8e809b520 comp rx=0 tx=0).stop 2024-04-01T01:50:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.936+0000 7fb8faba1740 1 -- 172.21.15.70:0/3233237750 shutdown_connections 2024-04-01T01:50:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.936+0000 7fb8faba1740 1 --2- 172.21.15.70:0/3233237750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f130ff0 0x56308f133480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.936+0000 7fb8faba1740 1 --2- 172.21.15.70:0/3233237750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f12be80 0x56308f130980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.936+0000 7fb8faba1740 1 --2- 172.21.15.70:0/3233237750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12b490 0x56308f12b8b0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.936+0000 7fb8faba1740 1 -- 172.21.15.70:0/3233237750 >> 172.21.15.70:0/3233237750 conn(0x56308f126a00 msgr2=0x56308f126e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.937+0000 7fb8faba1740 1 -- 172.21.15.70:0/3233237750 shutdown_connections 2024-04-01T01:50:03.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.937+0000 7fb8faba1740 1 -- 172.21.15.70:0/3233237750 wait complete. 2024-04-01T01:50:03.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.937+0000 7fb8faba1740 1 -- start start 2024-04-01T01:50:03.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.937+0000 7fb8faba1740 1 Processor -- start 2024-04-01T01:50:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.937+0000 7fb8faba1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f12b490 0x56308f1359e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.937+0000 7fb8faba1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f12be80 0x56308f135f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.937+0000 7fb8faba1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f130ff0 0x56308f13cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.937+0000 7fb8faba1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56308f133fa0 con 0x56308f130ff0 2024-04-01T01:50:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.937+0000 7fb8faba1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56308f02e470 con 0x56308f12be80 2024-04-01T01:50:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.937+0000 7fb8faba1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56308f134120 con 0x56308f12b490 2024-04-01T01:50:03.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.937+0000 7fb8faba1740 10 client.0 _ll_register_callbacks cb 0x56308f04bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:50:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.937+0000 7fb8f6564640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f130ff0 0x56308f13cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.937+0000 7fb8f6564640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f130ff0 0x56308f13cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60402/0 (socket says 172.21.15.70:60402) 2024-04-01T01:50:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.937+0000 7fb8f6564640 1 -- 172.21.15.70:0/1192021629 learned_addr learned my addr 172.21.15.70:0/1192021629 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.937+0000 7fb8f5562640 1 --2- 172.21.15.70:0/1192021629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f12be80 0x56308f135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.937+0000 7fb8f5d63640 1 --2- 172.21.15.70:0/1192021629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f12b490 0x56308f1359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8f6564640 1 -- 172.21.15.70:0/1192021629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f12b490 msgr2=0x56308f1359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8f6564640 1 --2- 172.21.15.70:0/1192021629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f12b490 0x56308f1359e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8f6564640 1 -- 172.21.15.70:0/1192021629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f12be80 msgr2=0x56308f135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8f6564640 1 --2- 172.21.15.70:0/1192021629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f12be80 0x56308f135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8f6564640 1 -- 172.21.15.70:0/1192021629 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56308eda43f0 con 0x56308f130ff0 2024-04-01T01:50:03.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8f6564640 1 --2- 172.21.15.70:0/1192021629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f130ff0 0x56308f13cf60 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0035000 tx=0x7fb8e002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e45e3c3c3c2ec6d server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:50:03.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8d57fa640 1 -- 172.21.15.70:0/1192021629 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e0005520 con 0x56308f130ff0 2024-04-01T01:50:03.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8d57fa640 1 -- 172.21.15.70:0/1192021629 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fb8e0004c40 con 0x56308f130ff0 2024-04-01T01:50:03.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8faba1740 1 -- 172.21.15.70:0/1192021629 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8e801e2c0 con 0x56308f130ff0 2024-04-01T01:50:03.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8d57fa640 1 -- 172.21.15.70:0/1192021629 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb8e005c050 con 0x56308f130ff0 2024-04-01T01:50:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8faba1740 10 client.20256 fetch_fsmap learned FSMap version 33 2024-04-01T01:50:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8faba1740 1 -- 172.21.15.70:0/1192021629 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56308ee6e7d0 con 0x56308f130ff0 2024-04-01T01:50:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8d57fa640 1 -- 172.21.15.70:0/1192021629 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8e801e2c0 con 0x56308f130ff0 2024-04-01T01:50:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8d57fa640 1 -- 172.21.15.70:0/1192021629 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fb8e0040040 con 0x56308f130ff0 2024-04-01T01:50:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8faba1740 10 client.20256 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:50:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8faba1740 10 client.20256 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:50:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.938+0000 7fb8faba1740 20 client.20256 populate_metadata read hostname 'smithi070' 2024-04-01T01:50:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.939+0000 7fb8faba1740 1 --2- 172.21.15.70:0/1192021629 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x56308f141570 0x56308f143990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.939+0000 7fb8faba1740 4 client.20256 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:50:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.939+0000 7fb8faba1740 1 -- 172.21.15.70:0/1192021629 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56308f09d6f0 con 0x56308f141570 2024-04-01T01:50:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.939+0000 7fb8f5d63640 1 --2- 172.21.15.70:0/1192021629 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x56308f141570 0x56308f143990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.939+0000 7fb8d57fa640 10 client.20256 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:50:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.939+0000 7fb8f5d63640 1 --2- 172.21.15.70:0/1192021629 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x56308f141570 0x56308f143990 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8155e34356214a93 server_cookie=8146bd2e9143cb17 in_seq=0 out_seq=0 2024-04-01T01:50:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.940+0000 7fb8d57fa640 1 -- 172.21.15.70:0/1192021629 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56308f09d6f0 con 0x56308f141570 2024-04-01T01:50:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.940+0000 7fb8d57fa640 10 client.20256 handle_command_reply: tid=0 2024-04-01T01:50:03.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.940+0000 7fb8faba1740 1 client.20256 shutdown 2024-04-01T01:50:03.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.940+0000 7fb8faba1740 1 -- 172.21.15.70:0/1192021629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f130ff0 msgr2=0x56308f13cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.940+0000 7fb8faba1740 1 --2- 172.21.15.70:0/1192021629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f130ff0 0x56308f13cf60 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0035000 tx=0x7fb8e002bc30 comp rx=0 tx=0).stop 2024-04-01T01:50:03.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.940+0000 7fb8faba1740 1 -- 172.21.15.70:0/1192021629 shutdown_connections 2024-04-01T01:50:03.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.940+0000 7fb8faba1740 1 --2- 172.21.15.70:0/1192021629 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x56308f141570 0x56308f143990 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.940+0000 7fb8faba1740 1 --2- 172.21.15.70:0/1192021629 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f130ff0 0x56308f13cf60 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.940+0000 7fb8faba1740 1 --2- 172.21.15.70:0/1192021629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f12be80 0x56308f135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.940+0000 7fb8faba1740 1 --2- 172.21.15.70:0/1192021629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f12b490 0x56308f1359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.940+0000 7fb8d57fa640 0 client.20256 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:50:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.940+0000 7fb8faba1740 1 -- 172.21.15.70:0/1192021629 >> 172.21.15.70:0/1192021629 conn(0x56308f126a00 msgr2=0x56308f125d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.940+0000 7fb8faba1740 1 -- 172.21.15.70:0/1192021629 shutdown_connections 2024-04-01T01:50:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.940+0000 7fb8faba1740 1 -- 172.21.15.70:0/1192021629 wait complete. 2024-04-01T01:50:03.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.940+0000 7fb8faba1740 20 client.20256 trim_cache size 0 max 16384 2024-04-01T01:50:03.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8faba1740 1 Processor -- start 2024-04-01T01:50:03.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8faba1740 1 -- start start 2024-04-01T01:50:03.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8faba1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f12b490 0x56308f12d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8faba1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12be80 0x56308f12d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8faba1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f130ff0 0x56308f1254b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8faba1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f01a93b0 con 0x56308f12be80 2024-04-01T01:50:03.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8faba1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f01aaef0 con 0x56308f12b490 2024-04-01T01:50:03.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8faba1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f01a9a80 con 0x56308f130ff0 2024-04-01T01:50:03.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8f5562640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12be80 0x56308f12d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8f5562640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12be80 0x56308f12d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60418/0 (socket says 172.21.15.70:60418) 2024-04-01T01:50:03.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8f5562640 1 -- 172.21.15.70:0/619308617 learned_addr learned my addr 172.21.15.70:0/619308617 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8f5d63640 1 --2- 172.21.15.70:0/619308617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f12b490 0x56308f12d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8f6564640 1 --2- 172.21.15.70:0/619308617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f130ff0 0x56308f1254b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8f5562640 1 -- 172.21.15.70:0/619308617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f130ff0 msgr2=0x56308f1254b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8f5562640 1 --2- 172.21.15.70:0/619308617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f130ff0 0x56308f1254b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8f5562640 1 -- 172.21.15.70:0/619308617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f12b490 msgr2=0x56308f12d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8f5562640 1 --2- 172.21.15.70:0/619308617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f12b490 0x56308f12d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.953+0000 7fb8f5562640 1 -- 172.21.15.70:0/619308617 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56308f130690 con 0x56308f12be80 2024-04-01T01:50:03.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.954+0000 7fb8f6564640 1 --2- 172.21.15.70:0/619308617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f130ff0 0x56308f1254b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-01T01:50:03.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.954+0000 7fb8f5562640 1 --2- 172.21.15.70:0/619308617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12be80 0x56308f12d950 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4009e20 tx=0x7fb8e4013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1576a236e94d6091 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.954+0000 7fb8d77fe640 1 -- 172.21.15.70:0/619308617 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4020070 con 0x56308f12be80 2024-04-01T01:50:03.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.954+0000 7fb8d77fe640 1 -- 172.21.15.70:0/619308617 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8e40054d0 con 0x56308f12be80 2024-04-01T01:50:03.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.954+0000 7fb8d77fe640 1 -- 172.21.15.70:0/619308617 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4011560 con 0x56308f12be80 2024-04-01T01:50:03.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.954+0000 7fb8faba1740 1 -- 172.21.15.70:0/619308617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12be80 msgr2=0x56308f12d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.954+0000 7fb8faba1740 1 --2- 172.21.15.70:0/619308617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12be80 0x56308f12d950 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4009e20 tx=0x7fb8e4013830 comp rx=0 tx=0).stop 2024-04-01T01:50:03.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.954+0000 7fb8faba1740 1 -- 172.21.15.70:0/619308617 shutdown_connections 2024-04-01T01:50:03.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.954+0000 7fb8faba1740 1 --2- 172.21.15.70:0/619308617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f130ff0 0x56308f1254b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.954+0000 7fb8faba1740 1 --2- 172.21.15.70:0/619308617 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12be80 0x56308f12d950 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.954+0000 7fb8faba1740 1 --2- 172.21.15.70:0/619308617 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f12b490 0x56308f12d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.954+0000 7fb8faba1740 1 -- 172.21.15.70:0/619308617 >> 172.21.15.70:0/619308617 conn(0x56308f126a00 msgr2=0x56308f126de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.954+0000 7fb8faba1740 1 -- 172.21.15.70:0/619308617 shutdown_connections 2024-04-01T01:50:03.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.954+0000 7fb8faba1740 1 -- 172.21.15.70:0/619308617 wait complete. 2024-04-01T01:50:03.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.955+0000 7fb8faba1740 1 -- start start 2024-04-01T01:50:03.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.955+0000 7fb8faba1740 1 Processor -- start 2024-04-01T01:50:03.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.955+0000 7fb8faba1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12b490 0x56308f1a3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.955+0000 7fb8faba1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f12be80 0x56308f1a5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.955+0000 7fb8faba1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f130ff0 0x56308f1a8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.955+0000 7fb8faba1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f01aa820 con 0x56308f12b490 2024-04-01T01:50:03.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.955+0000 7fb8faba1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f01aa150 con 0x56308f130ff0 2024-04-01T01:50:03.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.955+0000 7fb8faba1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f01a7390 con 0x56308f12be80 2024-04-01T01:50:03.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.955+0000 7fb8faba1740 10 client.0 _ll_register_callbacks cb 0x7fb8e0005370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:50:03.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.955+0000 7fb8f5d63640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12b490 0x56308f1a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.955+0000 7fb8f5d63640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12b490 0x56308f1a3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60434/0 (socket says 172.21.15.70:60434) 2024-04-01T01:50:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.955+0000 7fb8f5d63640 1 -- 172.21.15.70:0/1615692824 learned_addr learned my addr 172.21.15.70:0/1615692824 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.955+0000 7fb8f5562640 1 --2- 172.21.15.70:0/1615692824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f12be80 0x56308f1a5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.955+0000 7fb8f6564640 1 --2- 172.21.15.70:0/1615692824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f130ff0 0x56308f1a8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8f5d63640 1 -- 172.21.15.70:0/1615692824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f12be80 msgr2=0x56308f1a5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8f5d63640 1 --2- 172.21.15.70:0/1615692824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f12be80 0x56308f1a5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8f5d63640 1 -- 172.21.15.70:0/1615692824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f130ff0 msgr2=0x56308f1a8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8f5d63640 1 --2- 172.21.15.70:0/1615692824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f130ff0 0x56308f1a8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8f5d63640 1 -- 172.21.15.70:0/1615692824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56308f1a8bc0 con 0x56308f12b490 2024-04-01T01:50:03.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8f5d63640 1 --2- 172.21.15.70:0/1615692824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12b490 0x56308f1a3870 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8097b60 tx=0x7fb8e8009200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0fb5a6f27f2a9ff server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:03.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:50:03.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8d57fa640 1 -- 172.21.15.70:0/1615692824 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e807e070 con 0x56308f12b490 2024-04-01T01:50:03.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8d57fa640 1 -- 172.21.15.70:0/1615692824 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fb8e8084ce0 con 0x56308f12b490 2024-04-01T01:50:03.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8faba1740 1 -- 172.21.15.70:0/1615692824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8e40054d0 con 0x56308f12b490 2024-04-01T01:50:03.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8d57fa640 1 -- 172.21.15.70:0/1615692824 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb8e809b290 con 0x56308f12b490 2024-04-01T01:50:03.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8faba1740 10 client.20262 fetch_fsmap learned FSMap version 33 2024-04-01T01:50:03.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8faba1740 1 -- 172.21.15.70:0/1615692824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56308f1a9f30 con 0x56308f12b490 2024-04-01T01:50:03.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8d57fa640 1 -- 172.21.15.70:0/1615692824 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb8e40054d0 con 0x56308f12b490 2024-04-01T01:50:03.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8d57fa640 1 -- 172.21.15.70:0/1615692824 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fb8e8081020 con 0x56308f12b490 2024-04-01T01:50:03.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8faba1740 10 client.20262 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:50:03.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8faba1740 10 client.20262 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-01T01:50:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8faba1740 20 client.20262 populate_metadata read hostname 'smithi070' 2024-04-01T01:50:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8faba1740 1 --2- 172.21.15.70:0/1615692824 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x56308f1aa560 0x56308f1ac950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8faba1740 4 client.20262 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:50:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.956+0000 7fb8faba1740 1 -- 172.21.15.70:0/1615692824 --> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56308f097a40 con 0x56308f1aa560 2024-04-01T01:50:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8f5562640 1 --2- 172.21.15.70:0/1615692824 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x56308f1aa560 0x56308f1ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8d57fa640 10 client.20262 ms_handle_connect on v2:172.21.15.189:6838/2833086571 2024-04-01T01:50:03.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8f5562640 1 --2- 172.21.15.70:0/1615692824 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x56308f1aa560 0x56308f1ac950 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2ed218160b0eebaf server_cookie=7c8086f9423a7893 in_seq=0 out_seq=0 2024-04-01T01:50:03.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8d57fa640 1 -- 172.21.15.70:0/1615692824 <== mds.1 v2:172.21.15.189:6838/2833086571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56308f097a40 con 0x56308f1aa560 2024-04-01T01:50:03.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8d57fa640 10 client.20262 handle_command_reply: tid=0 2024-04-01T01:50:03.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8faba1740 1 client.20262 shutdown 2024-04-01T01:50:03.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8faba1740 1 -- 172.21.15.70:0/1615692824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12b490 msgr2=0x56308f1a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8faba1740 1 --2- 172.21.15.70:0/1615692824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12b490 0x56308f1a3870 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8097b60 tx=0x7fb8e8009200 comp rx=0 tx=0).stop 2024-04-01T01:50:03.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8faba1740 1 -- 172.21.15.70:0/1615692824 shutdown_connections 2024-04-01T01:50:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8faba1740 1 --2- 172.21.15.70:0/1615692824 >> [v2:172.21.15.189:6838/2833086571,v1:172.21.15.189:6839/2833086571] conn(0x56308f1aa560 0x56308f1ac950 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8faba1740 1 --2- 172.21.15.70:0/1615692824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56308f130ff0 0x56308f1a8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8faba1740 1 --2- 172.21.15.70:0/1615692824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56308f12be80 0x56308f1a5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8d57fa640 0 client.20262 ms_handle_reset on v2:172.21.15.189:6838/2833086571 2024-04-01T01:50:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8faba1740 1 --2- 172.21.15.70:0/1615692824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56308f12b490 0x56308f1a3870 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8faba1740 1 -- 172.21.15.70:0/1615692824 >> 172.21.15.70:0/1615692824 conn(0x56308f126a00 msgr2=0x56308f1359e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8faba1740 1 -- 172.21.15.70:0/1615692824 shutdown_connections 2024-04-01T01:50:03.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8faba1740 1 -- 172.21.15.70:0/1615692824 wait complete. 2024-04-01T01:50:03.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.957+0000 7fb8faba1740 20 client.20262 trim_cache size 0 max 16384 2024-04-01T01:50:03.983 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:50:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.958+0000 7fb8f7fc7640 1 -- 172.21.15.70:0/2338307261 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8d80a2520 msgr2=0x7fb8d80a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.958+0000 7fb8f7fc7640 1 --2- 172.21.15.70:0/2338307261 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8d80a2520 0x7fb8d80a49e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0005ab0 tx=0x7fb8e002c960 comp rx=0 tx=0).stop 2024-04-01T01:50:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.958+0000 7fb8f7fc7640 1 -- 172.21.15.70:0/2338307261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8f0100650 msgr2=0x7fb8f01968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.958+0000 7fb8f7fc7640 1 --2- 172.21.15.70:0/2338307261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8f0100650 0x7fb8f01968c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8009d20 tx=0x7fb8e800c6a0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.958+0000 7fb8f7fc7640 1 -- 172.21.15.70:0/2338307261 shutdown_connections 2024-04-01T01:50:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.958+0000 7fb8f7fc7640 1 --2- 172.21.15.70:0/2338307261 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb8d80a2520 0x7fb8d80a49e0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.958+0000 7fb8f7fc7640 1 --2- 172.21.15.70:0/2338307261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f0102200 0x7fb8f019de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.958+0000 7fb8f7fc7640 1 --2- 172.21.15.70:0/2338307261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8f0101840 0x7fb8f0196e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.958+0000 7fb8f7fc7640 1 --2- 172.21.15.70:0/2338307261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8f0100650 0x7fb8f01968c0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.958+0000 7fb8f7fc7640 1 -- 172.21.15.70:0/2338307261 >> 172.21.15.70:0/2338307261 conn(0x7fb8f00fbbe0 msgr2=0x7fb8f0104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.958+0000 7fb8f7fc7640 1 -- 172.21.15.70:0/2338307261 shutdown_connections 2024-04-01T01:50:03.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:03.958+0000 7fb8f7fc7640 1 -- 172.21.15.70:0/2338307261 wait complete. 2024-04-01T01:50:03.987 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:03 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1008847451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:50:03.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-01T01:50:03.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:03 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1008847451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:50:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.093+0000 7fa12432c640 1 -- 172.21.15.70:0/3029311245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa11c101860 msgr2=0x7fa11c101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.093+0000 7fa12432c640 1 --2- 172.21.15.70:0/3029311245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa11c101860 0x7fa11c101ce0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fa10c009900 tx=0x7fa10c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:50:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.093+0000 7fa12432c640 1 -- 172.21.15.70:0/3029311245 shutdown_connections 2024-04-01T01:50:04.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.093+0000 7fa12432c640 1 --2- 172.21.15.70:0/3029311245 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa11c102220 0x7fa11c108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.093+0000 7fa12432c640 1 --2- 172.21.15.70:0/3029311245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa11c101860 0x7fa11c101ce0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.093+0000 7fa12432c640 1 --2- 172.21.15.70:0/3029311245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa11c100670 0x7fa11c100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.093+0000 7fa12432c640 1 -- 172.21.15.70:0/3029311245 >> 172.21.15.70:0/3029311245 conn(0x7fa11c0fbc00 msgr2=0x7fa11c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.093+0000 7fa12432c640 1 -- 172.21.15.70:0/3029311245 shutdown_connections 2024-04-01T01:50:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.093+0000 7fa12432c640 1 -- 172.21.15.70:0/3029311245 wait complete. 2024-04-01T01:50:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa12432c640 1 Processor -- start 2024-04-01T01:50:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa12432c640 1 -- start start 2024-04-01T01:50:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa12432c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa11c100670 0x7fa11c1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa12432c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa11c101860 0x7fa11c197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa12432c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa11c102220 0x7fa11c19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa12432c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c10c190 con 0x7fa11c102220 2024-04-01T01:50:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa12432c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c10c010 con 0x7fa11c101860 2024-04-01T01:50:04.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa12432c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c10c310 con 0x7fa11c100670 2024-04-01T01:50:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa1228c9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa11c102220 0x7fa11c19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa1228c9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa11c102220 0x7fa11c19ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60454/0 (socket says 172.21.15.70:60454) 2024-04-01T01:50:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa1228c9640 1 -- 172.21.15.70:0/4009485272 learned_addr learned my addr 172.21.15.70:0/4009485272 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa1220c8640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa11c100670 0x7fa11c1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa1218c7640 1 --2- 172.21.15.70:0/4009485272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa11c101860 0x7fa11c197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa1228c9640 1 -- 172.21.15.70:0/4009485272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa11c100670 msgr2=0x7fa11c1979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa1228c9640 1 --2- 172.21.15.70:0/4009485272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa11c100670 0x7fa11c1979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa1228c9640 1 -- 172.21.15.70:0/4009485272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa11c101860 msgr2=0x7fa11c197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa1228c9640 1 --2- 172.21.15.70:0/4009485272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa11c101860 0x7fa11c197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa1228c9640 1 -- 172.21.15.70:0/4009485272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa11c19f5d0 con 0x7fa11c102220 2024-04-01T01:50:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.094+0000 7fa1228c9640 1 --2- 172.21.15.70:0/4009485272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa11c102220 0x7fa11c19ef60 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fa11400efd0 tx=0x7fa11400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.095+0000 7fa1137fe640 1 -- 172.21.15.70:0/4009485272 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa114003b70 con 0x7fa11c102220 2024-04-01T01:50:04.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.095+0000 7fa1137fe640 1 -- 172.21.15.70:0/4009485272 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa114003d10 con 0x7fa11c102220 2024-04-01T01:50:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.095+0000 7fa12432c640 1 -- 172.21.15.70:0/4009485272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa11c19f8c0 con 0x7fa11c102220 2024-04-01T01:50:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.095+0000 7fa1137fe640 1 -- 172.21.15.70:0/4009485272 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa114010980 con 0x7fa11c102220 2024-04-01T01:50:04.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.095+0000 7fa12432c640 1 -- 172.21.15.70:0/4009485272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa11c19fd90 con 0x7fa11c102220 2024-04-01T01:50:04.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.097+0000 7fa1137fe640 1 -- 172.21.15.70:0/4009485272 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fa114004020 con 0x7fa11c102220 2024-04-01T01:50:04.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.097+0000 7fa126feb740 1 Processor -- start 2024-04-01T01:50:04.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa126feb740 1 -- start start 2024-04-01T01:50:04.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa1137fe640 1 --2- 172.21.15.70:0/4009485272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa1080a24a0 0x7fa1080a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa126feb740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3e490 0x56517dd3e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa126feb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3ee80 0x56517dd43980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa126feb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd43ff0 0x56517dd46480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa126feb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56517dca31b0 con 0x56517dd3ee80 2024-04-01T01:50:04.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa126feb740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56517dc4bc00 con 0x56517dd3e490 2024-04-01T01:50:04.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa126feb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56517dc9f250 con 0x56517dd43ff0 2024-04-01T01:50:04.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa1218c7640 1 --2- 172.21.15.70:0/4009485272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa1080a24a0 0x7fa1080a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa1137fe640 1 -- 172.21.15.70:0/4009485272 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fa1140cef30 con 0x7fa11c102220 2024-04-01T01:50:04.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa1218c7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3ee80 0x56517dd43980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa1220c8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3e490 0x56517dd3e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa1220c8640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3e490 0x56517dd3e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:60226/0 (socket says 172.21.15.70:60226) 2024-04-01T01:50:04.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa1218c7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3ee80 0x56517dd43980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60460/0 (socket says 172.21.15.70:60460) 2024-04-01T01:50:04.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa1220c8640 1 -- 172.21.15.70:0/3403427787 learned_addr learned my addr 172.21.15.70:0/3403427787 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:04.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa1228c9640 1 --2- 172.21.15.70:0/3403427787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd43ff0 0x56517dd46480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa1220c8640 1 -- 172.21.15.70:0/3403427787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd43ff0 msgr2=0x56517dd46480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:04.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa1220c8640 1 --2- 172.21.15.70:0/3403427787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd43ff0 0x56517dd46480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa1220c8640 1 -- 172.21.15.70:0/3403427787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3ee80 msgr2=0x56517dd43980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:04.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa1220c8640 1 --2- 172.21.15.70:0/3403427787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3ee80 0x56517dd43980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa1220c8640 1 -- 172.21.15.70:0/3403427787 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56517dc6b3d0 con 0x56517dd3e490 2024-04-01T01:50:04.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa1218c7640 1 --2- 172.21.15.70:0/3403427787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3ee80 0x56517dd43980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:04.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.098+0000 7fa1218c7640 1 --2- 172.21.15.70:0/4009485272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa1080a24a0 0x7fa1080a4960 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa10c009230 tx=0x7fa10c005e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.099+0000 7fa1220c8640 1 --2- 172.21.15.70:0/3403427787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3e490 0x56517dd3e8b0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fa118009ca0 tx=0x7fa118011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f4ccd78b9596645 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.099+0000 7fa110bfe640 1 -- 172.21.15.70:0/3403427787 <== mon.2 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa118009e50 con 0x56517dd3e490 2024-04-01T01:50:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.099+0000 7fa110bfe640 1 -- 172.21.15.70:0/3403427787 <== mon.2 v2:172.21.15.81:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1180035f0 con 0x56517dd3e490 2024-04-01T01:50:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.099+0000 7fa110bfe640 1 -- 172.21.15.70:0/3403427787 <== mon.2 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa118019610 con 0x56517dd3e490 2024-04-01T01:50:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.099+0000 7fa126feb740 1 -- 172.21.15.70:0/3403427787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3e490 msgr2=0x56517dd3e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.099+0000 7fa126feb740 1 --2- 172.21.15.70:0/3403427787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3e490 0x56517dd3e8b0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fa118009ca0 tx=0x7fa118011810 comp rx=0 tx=0).stop 2024-04-01T01:50:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.099+0000 7fa126feb740 1 -- 172.21.15.70:0/3403427787 shutdown_connections 2024-04-01T01:50:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.099+0000 7fa126feb740 1 --2- 172.21.15.70:0/3403427787 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd43ff0 0x56517dd46480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.099+0000 7fa126feb740 1 --2- 172.21.15.70:0/3403427787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3ee80 0x56517dd43980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.099+0000 7fa126feb740 1 --2- 172.21.15.70:0/3403427787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3e490 0x56517dd3e8b0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.099+0000 7fa126feb740 1 -- 172.21.15.70:0/3403427787 >> 172.21.15.70:0/3403427787 conn(0x56517dd39a00 msgr2=0x56517dd39e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.100+0000 7fa126feb740 1 -- 172.21.15.70:0/3403427787 shutdown_connections 2024-04-01T01:50:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.100+0000 7fa126feb740 1 -- 172.21.15.70:0/3403427787 wait complete. 2024-04-01T01:50:04.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.100+0000 7fa126feb740 1 -- start start 2024-04-01T01:50:04.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.100+0000 7fa126feb740 1 Processor -- start 2024-04-01T01:50:04.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.100+0000 7fa126feb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd3e490 0x56517dd48940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.100+0000 7fa126feb740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3ee80 0x56517dd48e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.100+0000 7fa126feb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd43ff0 0x56517dd4fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.100+0000 7fa126feb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56517dd46fa0 con 0x56517dd43ff0 2024-04-01T01:50:04.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.100+0000 7fa126feb740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56517dc41470 con 0x56517dd3ee80 2024-04-01T01:50:04.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.100+0000 7fa126feb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56517dd47120 con 0x56517dd3e490 2024-04-01T01:50:04.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.100+0000 7fa126feb740 10 client.0 _ll_register_callbacks cb 0x56517dc5ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:50:04.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.100+0000 7fa1228c9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd43ff0 0x56517dd4fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.100+0000 7fa1220c8640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd3e490 0x56517dd48940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.100+0000 7fa1218c7640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3ee80 0x56517dd48e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.100+0000 7fa1228c9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd43ff0 0x56517dd4fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60470/0 (socket says 172.21.15.70:60470) 2024-04-01T01:50:04.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.100+0000 7fa1228c9640 1 -- 172.21.15.70:0/637476124 learned_addr learned my addr 172.21.15.70:0/637476124 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa1228c9640 1 -- 172.21.15.70:0/637476124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd3e490 msgr2=0x56517dd48940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa1228c9640 1 --2- 172.21.15.70:0/637476124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd3e490 0x56517dd48940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa1228c9640 1 -- 172.21.15.70:0/637476124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3ee80 msgr2=0x56517dd48e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa1228c9640 1 --2- 172.21.15.70:0/637476124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3ee80 0x56517dd48e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa1228c9640 1 -- 172.21.15.70:0/637476124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56517d9b73f0 con 0x56517dd43ff0 2024-04-01T01:50:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa1220c8640 1 --2- 172.21.15.70:0/637476124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd3e490 0x56517dd48940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa1228c9640 1 --2- 172.21.15.70:0/637476124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd43ff0 0x56517dd4fec0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fa114003f90 tx=0x7fa11409b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a761a5d75a3d119 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa105ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:50:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa105ffb640 1 -- 172.21.15.70:0/637476124 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1140045f0 con 0x56517dd43ff0 2024-04-01T01:50:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa105ffb640 1 -- 172.21.15.70:0/637476124 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fa114084bb0 con 0x56517dd43ff0 2024-04-01T01:50:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa126feb740 1 -- 172.21.15.70:0/637476124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1180035f0 con 0x56517dd43ff0 2024-04-01T01:50:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa105ffb640 1 -- 172.21.15.70:0/637476124 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa11409bd20 con 0x56517dd43ff0 2024-04-01T01:50:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa126feb740 10 client.20274 fetch_fsmap learned FSMap version 33 2024-04-01T01:50:04.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa126feb740 1 -- 172.21.15.70:0/637476124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56517da817d0 con 0x56517dd43ff0 2024-04-01T01:50:04.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa105ffb640 1 -- 172.21.15.70:0/637476124 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa1180035f0 con 0x56517dd43ff0 2024-04-01T01:50:04.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa105ffb640 1 -- 172.21.15.70:0/637476124 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fa11401f020 con 0x56517dd43ff0 2024-04-01T01:50:04.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa126feb740 10 client.20274 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:50:04.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa126feb740 10 client.20274 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:50:04.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa126feb740 20 client.20274 populate_metadata read hostname 'smithi070' 2024-04-01T01:50:04.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa126feb740 1 --2- 172.21.15.70:0/637476124 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56517dd544a0 0x56517dd56910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa126feb740 4 client.20274 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-01T01:50:04.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.101+0000 7fa126feb740 1 -- 172.21.15.70:0/637476124 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56517dcb06f0 con 0x56517dd544a0 2024-04-01T01:50:04.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.102+0000 7fa1220c8640 1 --2- 172.21.15.70:0/637476124 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56517dd544a0 0x56517dd56910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.102+0000 7fa1220c8640 1 --2- 172.21.15.70:0/637476124 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56517dd544a0 0x56517dd56910 crc :-1 s=READY pgs=410 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccd72dab529558ae server_cookie=dbdb2fbeb6cdfc38 in_seq=0 out_seq=0 2024-04-01T01:50:04.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.102+0000 7fa105ffb640 10 client.20274 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:50:04.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.102+0000 7fa105ffb640 1 -- 172.21.15.70:0/637476124 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56517dcb06f0 con 0x56517dd544a0 2024-04-01T01:50:04.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.102+0000 7fa105ffb640 10 client.20274 handle_command_reply: tid=0 2024-04-01T01:50:04.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.103+0000 7fa126feb740 1 client.20274 shutdown 2024-04-01T01:50:04.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.103+0000 7fa126feb740 1 -- 172.21.15.70:0/637476124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd43ff0 msgr2=0x56517dd4fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:04.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.103+0000 7fa126feb740 1 --2- 172.21.15.70:0/637476124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd43ff0 0x56517dd4fec0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fa114003f90 tx=0x7fa11409b4b0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.103+0000 7fa126feb740 1 -- 172.21.15.70:0/637476124 shutdown_connections 2024-04-01T01:50:04.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.103+0000 7fa126feb740 1 --2- 172.21.15.70:0/637476124 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56517dd544a0 0x56517dd56910 crc :-1 s=READY pgs=410 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.103+0000 7fa126feb740 1 --2- 172.21.15.70:0/637476124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd43ff0 0x56517dd4fec0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.103+0000 7fa105ffb640 0 client.20274 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:50:04.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.103+0000 7fa126feb740 1 --2- 172.21.15.70:0/637476124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3ee80 0x56517dd48e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.103+0000 7fa126feb740 1 --2- 172.21.15.70:0/637476124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd3e490 0x56517dd48940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.103+0000 7fa126feb740 1 -- 172.21.15.70:0/637476124 >> 172.21.15.70:0/637476124 conn(0x56517dd39a00 msgr2=0x56517dd38d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:04.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.103+0000 7fa126feb740 1 -- 172.21.15.70:0/637476124 shutdown_connections 2024-04-01T01:50:04.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.103+0000 7fa126feb740 1 -- 172.21.15.70:0/637476124 wait complete. 2024-04-01T01:50:04.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.103+0000 7fa126feb740 20 client.20274 trim_cache size 0 max 16384 2024-04-01T01:50:04.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa126feb740 1 Processor -- start 2024-04-01T01:50:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa126feb740 1 -- start start 2024-04-01T01:50:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa126feb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3e490 0x56517dd402e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa126feb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd3ee80 0x56517dd40820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa126feb740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd43ff0 0x56517dd384b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa126feb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c1aa300 con 0x56517dd3e490 2024-04-01T01:50:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa126feb740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c1abe40 con 0x56517dd43ff0 2024-04-01T01:50:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa126feb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c1aa9d0 con 0x56517dd3ee80 2024-04-01T01:50:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa1220c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3e490 0x56517dd402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa1220c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3e490 0x56517dd402e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60482/0 (socket says 172.21.15.70:60482) 2024-04-01T01:50:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa1220c8640 1 -- 172.21.15.70:0/667986884 learned_addr learned my addr 172.21.15.70:0/667986884 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa1218c7640 1 --2- 172.21.15.70:0/667986884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd3ee80 0x56517dd40820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa1228c9640 1 --2- 172.21.15.70:0/667986884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd43ff0 0x56517dd384b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa1220c8640 1 -- 172.21.15.70:0/667986884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd3ee80 msgr2=0x56517dd40820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa1220c8640 1 --2- 172.21.15.70:0/667986884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd3ee80 0x56517dd40820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa1220c8640 1 -- 172.21.15.70:0/667986884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd43ff0 msgr2=0x56517dd384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa1220c8640 1 --2- 172.21.15.70:0/667986884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd43ff0 0x56517dd384b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa1220c8640 1 -- 172.21.15.70:0/667986884 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56517dd3f260 con 0x56517dd3e490 2024-04-01T01:50:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa1220c8640 1 --2- 172.21.15.70:0/667986884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3e490 0x56517dd402e0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fa11801b040 tx=0x7fa1180096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92010912bbf3b9cd server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.116+0000 7fa107fff640 1 -- 172.21.15.70:0/667986884 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1180035f0 con 0x56517dd3e490 2024-04-01T01:50:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.117+0000 7fa107fff640 1 -- 172.21.15.70:0/667986884 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa118003d70 con 0x56517dd3e490 2024-04-01T01:50:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.117+0000 7fa107fff640 1 -- 172.21.15.70:0/667986884 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa118026680 con 0x56517dd3e490 2024-04-01T01:50:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.117+0000 7fa126feb740 1 -- 172.21.15.70:0/667986884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3e490 msgr2=0x56517dd402e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.117+0000 7fa126feb740 1 --2- 172.21.15.70:0/667986884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3e490 0x56517dd402e0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fa11801b040 tx=0x7fa1180096a0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.117+0000 7fa126feb740 1 -- 172.21.15.70:0/667986884 shutdown_connections 2024-04-01T01:50:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.117+0000 7fa126feb740 1 --2- 172.21.15.70:0/667986884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd43ff0 0x56517dd384b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.117+0000 7fa126feb740 1 --2- 172.21.15.70:0/667986884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd3ee80 0x56517dd40820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.117+0000 7fa126feb740 1 --2- 172.21.15.70:0/667986884 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3e490 0x56517dd402e0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.117+0000 7fa126feb740 1 -- 172.21.15.70:0/667986884 >> 172.21.15.70:0/667986884 conn(0x56517dd39a00 msgr2=0x56517dd34bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.117+0000 7fa126feb740 1 -- 172.21.15.70:0/667986884 shutdown_connections 2024-04-01T01:50:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.117+0000 7fa126feb740 1 -- 172.21.15.70:0/667986884 wait complete. 2024-04-01T01:50:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.117+0000 7fa126feb740 1 -- start start 2024-04-01T01:50:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.117+0000 7fa126feb740 1 Processor -- start 2024-04-01T01:50:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa126feb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3e490 0x56517ddb5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa126feb740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3ee80 0x56517ddb62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa126feb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd43ff0 0x56517ddbd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa126feb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c1ab770 con 0x56517dd3e490 2024-04-01T01:50:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa126feb740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c1ab0a0 con 0x56517dd3ee80 2024-04-01T01:50:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa126feb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c1a82e0 con 0x56517dd43ff0 2024-04-01T01:50:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa126feb740 10 client.0 _ll_register_callbacks cb 0x7fa1140978c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-01T01:50:04.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa1220c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3e490 0x56517ddb5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa1220c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3e490 0x56517ddb5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60496/0 (socket says 172.21.15.70:60496) 2024-04-01T01:50:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa1220c8640 1 -- 172.21.15.70:0/2567672423 learned_addr learned my addr 172.21.15.70:0/2567672423 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa1218c7640 1 --2- 172.21.15.70:0/2567672423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3ee80 0x56517ddb62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa1228c9640 1 --2- 172.21.15.70:0/2567672423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd43ff0 0x56517ddbd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa1220c8640 1 -- 172.21.15.70:0/2567672423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd43ff0 msgr2=0x56517ddbd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa1220c8640 1 --2- 172.21.15.70:0/2567672423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd43ff0 0x56517ddbd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa1220c8640 1 -- 172.21.15.70:0/2567672423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3ee80 msgr2=0x56517ddb62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-01T01:50:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa1220c8640 1 --2- 172.21.15.70:0/2567672423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3ee80 0x56517ddb62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa1220c8640 1 -- 172.21.15.70:0/2567672423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56517ddbd8d0 con 0x56517dd3e490 2024-04-01T01:50:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa1220c8640 1 --2- 172.21.15.70:0/2567672423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3e490 0x56517ddb5d60 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fa118009250 tx=0x7fa1180098d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0b4e240244da27 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa105ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-01T01:50:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa105ffb640 1 -- 172.21.15.70:0/2567672423 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11803c060 con 0x56517dd3e490 2024-04-01T01:50:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa105ffb640 1 -- 172.21.15.70:0/2567672423 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fa118005360 con 0x56517dd3e490 2024-04-01T01:50:04.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa126feb740 1 -- 172.21.15.70:0/2567672423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa118003d70 con 0x56517dd3e490 2024-04-01T01:50:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa105ffb640 1 -- 172.21.15.70:0/2567672423 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa118043050 con 0x56517dd3e490 2024-04-01T01:50:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa126feb740 10 client.20280 fetch_fsmap learned FSMap version 33 2024-04-01T01:50:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.118+0000 7fa126feb740 1 -- 172.21.15.70:0/2567672423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56517ddbebd0 con 0x56517dd3e490 2024-04-01T01:50:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.119+0000 7fa105ffb640 1 -- 172.21.15.70:0/2567672423 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa118003d70 con 0x56517dd3e490 2024-04-01T01:50:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.119+0000 7fa105ffb640 1 -- 172.21.15.70:0/2567672423 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fa11802d070 con 0x56517dd3e490 2024-04-01T01:50:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.119+0000 7fa126feb740 10 client.20280 fetch_fsmap finished waiting for FSMap version 33 2024-04-01T01:50:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.119+0000 7fa126feb740 10 client.20280 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-01T01:50:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.119+0000 7fa126feb740 20 client.20280 populate_metadata read hostname 'smithi070' 2024-04-01T01:50:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.119+0000 7fa126feb740 1 --2- 172.21.15.70:0/2567672423 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56517ddbf380 0x56517ddc17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.119+0000 7fa126feb740 4 client.20280 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-01T01:50:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.119+0000 7fa126feb740 1 -- 172.21.15.70:0/2567672423 --> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56517dcaaa40 con 0x56517ddbf380 2024-04-01T01:50:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.119+0000 7fa1218c7640 1 --2- 172.21.15.70:0/2567672423 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56517ddbf380 0x56517ddc17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.119+0000 7fa105ffb640 10 client.20280 ms_handle_connect on v2:172.21.15.81:6838/2805044611 2024-04-01T01:50:04.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.119+0000 7fa1218c7640 1 --2- 172.21.15.70:0/2567672423 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56517ddbf380 0x56517ddc17a0 crc :-1 s=READY pgs=411 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e309ebe9b4c0e05 server_cookie=ec1f0d32a4be6c6f in_seq=0 out_seq=0 2024-04-01T01:50:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa105ffb640 1 -- 172.21.15.70:0/2567672423 <== mds.0 v2:172.21.15.81:6838/2805044611 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56517dcaaa40 con 0x56517ddbf380 2024-04-01T01:50:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa105ffb640 10 client.20280 handle_command_reply: tid=0 2024-04-01T01:50:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa126feb740 1 client.20280 shutdown 2024-04-01T01:50:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa126feb740 1 -- 172.21.15.70:0/2567672423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3e490 msgr2=0x56517ddb5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa126feb740 1 --2- 172.21.15.70:0/2567672423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3e490 0x56517ddb5d60 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fa118009250 tx=0x7fa1180098d0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa126feb740 1 -- 172.21.15.70:0/2567672423 shutdown_connections 2024-04-01T01:50:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa126feb740 1 --2- 172.21.15.70:0/2567672423 >> [v2:172.21.15.81:6838/2805044611,v1:172.21.15.81:6839/2805044611] conn(0x56517ddbf380 0x56517ddc17a0 crc :-1 s=READY pgs=411 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa126feb740 1 --2- 172.21.15.70:0/2567672423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56517dd43ff0 0x56517ddbd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa126feb740 1 --2- 172.21.15.70:0/2567672423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56517dd3ee80 0x56517ddb62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa126feb740 1 --2- 172.21.15.70:0/2567672423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56517dd3e490 0x56517ddb5d60 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa105ffb640 0 client.20280 ms_handle_reset on v2:172.21.15.81:6838/2805044611 2024-04-01T01:50:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa126feb740 1 -- 172.21.15.70:0/2567672423 >> 172.21.15.70:0/2567672423 conn(0x56517dd39a00 msgr2=0x56517dd48900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa126feb740 1 -- 172.21.15.70:0/2567672423 shutdown_connections 2024-04-01T01:50:04.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa126feb740 1 -- 172.21.15.70:0/2567672423 wait complete. 2024-04-01T01:50:04.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa126feb740 20 client.20280 trim_cache size 0 max 16384 2024-04-01T01:50:04.127 INFO:teuthology.orchestra.run.smithi070.stdout:[] 2024-04-01T01:50:04.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa12432c640 1 -- 172.21.15.70:0/4009485272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa1080a24a0 msgr2=0x7fa1080a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:04.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa12432c640 1 --2- 172.21.15.70:0/4009485272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa1080a24a0 0x7fa1080a4960 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa10c009230 tx=0x7fa10c005e40 comp rx=0 tx=0).stop 2024-04-01T01:50:04.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa12432c640 1 -- 172.21.15.70:0/4009485272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa11c102220 msgr2=0x7fa11c19ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:04.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa12432c640 1 --2- 172.21.15.70:0/4009485272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa11c102220 0x7fa11c19ef60 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fa11400efd0 tx=0x7fa11400c6a0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa12432c640 1 -- 172.21.15.70:0/4009485272 shutdown_connections 2024-04-01T01:50:04.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa12432c640 1 --2- 172.21.15.70:0/4009485272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa11c102220 0x7fa11c19ef60 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa12432c640 1 --2- 172.21.15.70:0/4009485272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa11c101860 0x7fa11c197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa12432c640 1 --2- 172.21.15.70:0/4009485272 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa1080a24a0 0x7fa1080a4960 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa12432c640 1 --2- 172.21.15.70:0/4009485272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa11c100670 0x7fa11c1979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:04.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa12432c640 1 -- 172.21.15.70:0/4009485272 >> 172.21.15.70:0/4009485272 conn(0x7fa11c0fbc00 msgr2=0x7fa11c0fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:04.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa12432c640 1 -- 172.21.15.70:0/4009485272 shutdown_connections 2024-04-01T01:50:04.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:04.120+0000 7fa12432c640 1 -- 172.21.15.70:0/4009485272 wait complete. 2024-04-01T01:50:04.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:03 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1008847451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-01T01:50:05.131 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-04-01T01:50:05.132 INFO:tasks.fwd_scrub:done joining 2024-04-01T01:50:05.132 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-01T01:50:05.154 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:50:05.180 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:04 smithi070 ceph-mon[32179]: pgmap v857: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 6.7 MiB/s wr, 739 op/s 2024-04-01T01:50:05.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:04 smithi189 ceph-mon[35753]: pgmap v857: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 6.7 MiB/s wr, 739 op/s 2024-04-01T01:50:05.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.260+0000 7fbc517b5640 1 -- 172.21.15.70:0/1038457561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc4c1021f0 msgr2=0x7fbc4c1045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:05.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.260+0000 7fbc517b5640 1 --2- 172.21.15.70:0/1038457561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc4c1021f0 0x7fbc4c1045e0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c009900 tx=0x7fbc3c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:50:05.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.260+0000 7fbc517b5640 1 -- 172.21.15.70:0/1038457561 shutdown_connections 2024-04-01T01:50:05.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.260+0000 7fbc517b5640 1 --2- 172.21.15.70:0/1038457561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc4c107450 0x7fbc4c1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.260+0000 7fbc517b5640 1 --2- 172.21.15.70:0/1038457561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c104b20 0x7fbc4c106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.260+0000 7fbc517b5640 1 --2- 172.21.15.70:0/1038457561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc4c1021f0 0x7fbc4c1045e0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.260+0000 7fbc517b5640 1 -- 172.21.15.70:0/1038457561 >> 172.21.15.70:0/1038457561 conn(0x7fbc4c0fbbe0 msgr2=0x7fbc4c0fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:05.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.260+0000 7fbc517b5640 1 -- 172.21.15.70:0/1038457561 shutdown_connections 2024-04-01T01:50:05.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.260+0000 7fbc517b5640 1 -- 172.21.15.70:0/1038457561 wait complete. 2024-04-01T01:50:05.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc517b5640 1 Processor -- start 2024-04-01T01:50:05.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc517b5640 1 -- start start 2024-04-01T01:50:05.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc517b5640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc4c1021f0 0x7fbc4c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:05.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc517b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc4c104b20 0x7fbc4c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:05.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc517b5640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c107450 0x7fbc4c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:05.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc517b5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c10d2f0 con 0x7fbc4c104b20 2024-04-01T01:50:05.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc517b5640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c10d170 con 0x7fbc4c1021f0 2024-04-01T01:50:05.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc517b5640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c10d470 con 0x7fbc4c107450 2024-04-01T01:50:05.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc4a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc4c104b20 0x7fbc4c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:05.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc4a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc4c104b20 0x7fbc4c197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60520/0 (socket says 172.21.15.70:60520) 2024-04-01T01:50:05.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc4a7fc640 1 -- 172.21.15.70:0/2549593005 learned_addr learned my addr 172.21.15.70:0/2549593005 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:05.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc4b7fe640 1 --2- 172.21.15.70:0/2549593005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c107450 0x7fbc4c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:05.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc4affd640 1 --2- 172.21.15.70:0/2549593005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc4c1021f0 0x7fbc4c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:05.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc4a7fc640 1 -- 172.21.15.70:0/2549593005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c107450 msgr2=0x7fbc4c19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:05.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc4a7fc640 1 --2- 172.21.15.70:0/2549593005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c107450 0x7fbc4c19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc4a7fc640 1 -- 172.21.15.70:0/2549593005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc4c1021f0 msgr2=0x7fbc4c197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:05.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc4a7fc640 1 --2- 172.21.15.70:0/2549593005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc4c1021f0 0x7fbc4c197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc4a7fc640 1 -- 172.21.15.70:0/2549593005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4c19f620 con 0x7fbc4c104b20 2024-04-01T01:50:05.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.261+0000 7fbc4affd640 1 --2- 172.21.15.70:0/2549593005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc4c1021f0 0x7fbc4c197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:05.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.262+0000 7fbc4a7fc640 1 --2- 172.21.15.70:0/2549593005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc4c104b20 0x7fbc4c197f40 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fbc4000e9c0 tx=0x7fbc4000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:05.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.262+0000 7fbc33fff640 1 -- 172.21.15.70:0/2549593005 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4000cd80 con 0x7fbc4c104b20 2024-04-01T01:50:05.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.262+0000 7fbc517b5640 1 -- 172.21.15.70:0/2549593005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4c19f910 con 0x7fbc4c104b20 2024-04-01T01:50:05.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.262+0000 7fbc33fff640 1 -- 172.21.15.70:0/2549593005 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbc400054d0 con 0x7fbc4c104b20 2024-04-01T01:50:05.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.262+0000 7fbc33fff640 1 -- 172.21.15.70:0/2549593005 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc40010430 con 0x7fbc4c104b20 2024-04-01T01:50:05.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.262+0000 7fbc517b5640 1 -- 172.21.15.70:0/2549593005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc4c19fe60 con 0x7fbc4c104b20 2024-04-01T01:50:05.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.263+0000 7fbc517b5640 1 -- 172.21.15.70:0/2549593005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc4c0fd6d0 con 0x7fbc4c104b20 2024-04-01T01:50:05.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.264+0000 7fbc33fff640 1 -- 172.21.15.70:0/2549593005 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fbc40004fd0 con 0x7fbc4c104b20 2024-04-01T01:50:05.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.264+0000 7fbc33fff640 1 --2- 172.21.15.70:0/2549593005 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbc340a24a0 0x7fbc340a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:05.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.264+0000 7fbc4affd640 1 --2- 172.21.15.70:0/2549593005 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbc340a24a0 0x7fbc340a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:05.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.264+0000 7fbc33fff640 1 -- 172.21.15.70:0/2549593005 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fbc400ceee0 con 0x7fbc4c104b20 2024-04-01T01:50:05.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.265+0000 7fbc4affd640 1 --2- 172.21.15.70:0/2549593005 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbc340a24a0 0x7fbc340a4960 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c000c00 tx=0x7fbc3c02c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:05.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.269+0000 7fbc33fff640 1 -- 172.21.15.70:0/2549593005 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbc4009bcd0 con 0x7fbc4c104b20 2024-04-01T01:50:05.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:04 smithi081 ceph-mon[35142]: pgmap v857: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 6.7 MiB/s wr, 739 op/s 2024-04-01T01:50:05.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.443+0000 7fbc517b5640 1 -- 172.21.15.70:0/2549593005 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc4c062350 con 0x7fbc4c104b20 2024-04-01T01:50:05.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.445+0000 7fbc33fff640 1 -- 172.21.15.70:0/2549593005 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24781 (secure 0 0 0) 0x7fbc4009b5d0 con 0x7fbc4c104b20 2024-04-01T01:50:05.446 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:50:05.446 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":95,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:49:12.151965+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:50:05.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.447+0000 7fbc517b5640 1 -- 172.21.15.70:0/2549593005 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbc340a24a0 msgr2=0x7fbc340a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:05.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.447+0000 7fbc517b5640 1 --2- 172.21.15.70:0/2549593005 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbc340a24a0 0x7fbc340a4960 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c000c00 tx=0x7fbc3c02c7e0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.447+0000 7fbc517b5640 1 -- 172.21.15.70:0/2549593005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc4c104b20 msgr2=0x7fbc4c197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:05.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.447+0000 7fbc517b5640 1 --2- 172.21.15.70:0/2549593005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc4c104b20 0x7fbc4c197f40 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fbc4000e9c0 tx=0x7fbc4000ee80 comp rx=0 tx=0).stop 2024-04-01T01:50:05.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.447+0000 7fbc517b5640 1 -- 172.21.15.70:0/2549593005 shutdown_connections 2024-04-01T01:50:05.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.447+0000 7fbc517b5640 1 --2- 172.21.15.70:0/2549593005 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fbc340a24a0 0x7fbc340a4960 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.447+0000 7fbc517b5640 1 --2- 172.21.15.70:0/2549593005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c107450 0x7fbc4c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.447+0000 7fbc517b5640 1 --2- 172.21.15.70:0/2549593005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc4c104b20 0x7fbc4c197f40 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.447+0000 7fbc517b5640 1 --2- 172.21.15.70:0/2549593005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc4c1021f0 0x7fbc4c197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.447+0000 7fbc517b5640 1 -- 172.21.15.70:0/2549593005 >> 172.21.15.70:0/2549593005 conn(0x7fbc4c0fbbe0 msgr2=0x7fbc4c102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:05.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.447+0000 7fbc517b5640 1 -- 172.21.15.70:0/2549593005 shutdown_connections 2024-04-01T01:50:05.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.447+0000 7fbc517b5640 1 -- 172.21.15.70:0/2549593005 wait complete. 2024-04-01T01:50:05.459 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-01T01:26:42.809779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-01T01:31:13.612369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-01T01:31:14.910162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-01T01:31:18.783179+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-01T01:50:05.460 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-01T01:50:05.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.604+0000 7f9e4dc55640 1 -- 172.21.15.70:0/1813920736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e48102960 msgr2=0x7f9e48102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:05.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.604+0000 7f9e4dc55640 1 --2- 172.21.15.70:0/1813920736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e48102960 0x7f9e48102de0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c009900 tx=0x7f9e3c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:50:05.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.604+0000 7f9e4dc55640 1 -- 172.21.15.70:0/1813920736 shutdown_connections 2024-04-01T01:50:05.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.604+0000 7f9e4dc55640 1 --2- 172.21.15.70:0/1813920736 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e48103320 0x7f9e48109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.604+0000 7f9e4dc55640 1 --2- 172.21.15.70:0/1813920736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e48102960 0x7f9e48102de0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.604+0000 7f9e4dc55640 1 --2- 172.21.15.70:0/1813920736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e48101770 0x7f9e48101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.604+0000 7f9e4dc55640 1 -- 172.21.15.70:0/1813920736 >> 172.21.15.70:0/1813920736 conn(0x7f9e480fcd00 msgr2=0x7f9e480ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:05.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.604+0000 7f9e4dc55640 1 -- 172.21.15.70:0/1813920736 shutdown_connections 2024-04-01T01:50:05.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.604+0000 7f9e4dc55640 1 -- 172.21.15.70:0/1813920736 wait complete. 2024-04-01T01:50:05.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.604+0000 7f9e4dc55640 1 Processor -- start 2024-04-01T01:50:05.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e4dc55640 1 -- start start 2024-04-01T01:50:05.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e4dc55640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e48101770 0x7f9e481979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:05.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e4dc55640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e48102960 0x7f9e48197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:05.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e4dc55640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e48103320 0x7f9e4819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:05.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e4dc55640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4810d2a0 con 0x7f9e48101770 2024-04-01T01:50:05.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e4dc55640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4810d120 con 0x7f9e48103320 2024-04-01T01:50:05.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e4dc55640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4810d420 con 0x7f9e48102960 2024-04-01T01:50:05.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e477fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e48101770 0x7f9e481979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e477fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e48101770 0x7f9e481979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60546/0 (socket says 172.21.15.70:60546) 2024-04-01T01:50:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e477fe640 1 -- 172.21.15.70:0/1921387651 learned_addr learned my addr 172.21.15.70:0/1921387651 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e46ffd640 1 --2- 172.21.15.70:0/1921387651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e48102960 0x7f9e48197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e47fff640 1 --2- 172.21.15.70:0/1921387651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e48103320 0x7f9e4819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e477fe640 1 -- 172.21.15.70:0/1921387651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e48102960 msgr2=0x7f9e48197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e477fe640 1 --2- 172.21.15.70:0/1921387651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e48102960 0x7f9e48197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e477fe640 1 -- 172.21.15.70:0/1921387651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e48103320 msgr2=0x7f9e4819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e477fe640 1 --2- 172.21.15.70:0/1921387651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e48103320 0x7f9e4819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e477fe640 1 -- 172.21.15.70:0/1921387651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e4819f5c0 con 0x7f9e48101770 2024-04-01T01:50:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e477fe640 1 --2- 172.21.15.70:0/1921387651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e48101770 0x7f9e481979a0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f9e3800e9c0 tx=0x7f9e3800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.605+0000 7f9e44ff9640 1 -- 172.21.15.70:0/1921387651 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3800cd80 con 0x7f9e48101770 2024-04-01T01:50:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.606+0000 7f9e44ff9640 1 -- 172.21.15.70:0/1921387651 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e380035f0 con 0x7f9e48101770 2024-04-01T01:50:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.606+0000 7f9e4dc55640 1 -- 172.21.15.70:0/1921387651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e4819f8b0 con 0x7f9e48101770 2024-04-01T01:50:05.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.606+0000 7f9e44ff9640 1 -- 172.21.15.70:0/1921387651 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e38010690 con 0x7f9e48101770 2024-04-01T01:50:05.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.606+0000 7f9e4dc55640 1 -- 172.21.15.70:0/1921387651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e4819fdd0 con 0x7f9e48101770 2024-04-01T01:50:05.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.606+0000 7f9e4dc55640 1 -- 172.21.15.70:0/1921387651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e48108b30 con 0x7f9e48101770 2024-04-01T01:50:05.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.608+0000 7f9e44ff9640 1 -- 172.21.15.70:0/1921387651 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f9e380037a0 con 0x7f9e48101770 2024-04-01T01:50:05.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.608+0000 7f9e44ff9640 1 --2- 172.21.15.70:0/1921387651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e300a2520 0x7f9e300a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:05.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.608+0000 7f9e46ffd640 1 --2- 172.21.15.70:0/1921387651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e300a2520 0x7f9e300a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:05.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.608+0000 7f9e44ff9640 1 -- 172.21.15.70:0/1921387651 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f9e380cf100 con 0x7f9e48101770 2024-04-01T01:50:05.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.609+0000 7f9e46ffd640 1 --2- 172.21.15.70:0/1921387651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e300a2520 0x7f9e300a49e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c009230 tx=0x7f9e3c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:05.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.612+0000 7f9e44ff9640 1 -- 172.21.15.70:0/1921387651 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9e3809bef0 con 0x7f9e48101770 2024-04-01T01:50:05.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.786+0000 7f9e4dc55640 1 -- 172.21.15.70:0/1921387651 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9e4802d300 con 0x7f9e48101770 2024-04-01T01:50:05.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.787+0000 7f9e44ff9640 1 -- 172.21.15.70:0/1921387651 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f9e3809b7f0 con 0x7f9e48101770 2024-04-01T01:50:05.787 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-01T01:50:05.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.789+0000 7f9e4dc55640 1 -- 172.21.15.70:0/1921387651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e300a2520 msgr2=0x7f9e300a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:05.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.789+0000 7f9e4dc55640 1 --2- 172.21.15.70:0/1921387651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e300a2520 0x7f9e300a49e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c009230 tx=0x7f9e3c005fd0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.789+0000 7f9e4dc55640 1 -- 172.21.15.70:0/1921387651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e48101770 msgr2=0x7f9e481979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:05.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.789+0000 7f9e4dc55640 1 --2- 172.21.15.70:0/1921387651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e48101770 0x7f9e481979a0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f9e3800e9c0 tx=0x7f9e3800ee80 comp rx=0 tx=0).stop 2024-04-01T01:50:05.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.789+0000 7f9e4dc55640 1 -- 172.21.15.70:0/1921387651 shutdown_connections 2024-04-01T01:50:05.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.789+0000 7f9e4dc55640 1 --2- 172.21.15.70:0/1921387651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e48103320 0x7f9e4819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.789+0000 7f9e4dc55640 1 --2- 172.21.15.70:0/1921387651 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f9e300a2520 0x7f9e300a49e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.789+0000 7f9e4dc55640 1 --2- 172.21.15.70:0/1921387651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e48102960 0x7f9e48197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.789+0000 7f9e4dc55640 1 --2- 172.21.15.70:0/1921387651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e48101770 0x7f9e481979a0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.789+0000 7f9e4dc55640 1 -- 172.21.15.70:0/1921387651 >> 172.21.15.70:0/1921387651 conn(0x7f9e480fcd00 msgr2=0x7f9e48105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:05.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.789+0000 7f9e4dc55640 1 -- 172.21.15.70:0/1921387651 shutdown_connections 2024-04-01T01:50:05.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.789+0000 7f9e4dc55640 1 -- 172.21.15.70:0/1921387651 wait complete. 2024-04-01T01:50:05.801 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-01T01:50:05.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.909+0000 7f860f688640 1 -- 172.21.15.70:0/502999377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8608100660 msgr2=0x7f8608100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:05.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.909+0000 7f860f688640 1 --2- 172.21.15.70:0/502999377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8608100660 0x7f8608100a80 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f8604009860 tx=0x7f860402bb70 comp rx=0 tx=0).stop 2024-04-01T01:50:05.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.909+0000 7f860f688640 1 -- 172.21.15.70:0/502999377 shutdown_connections 2024-04-01T01:50:05.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.909+0000 7f860f688640 1 --2- 172.21.15.70:0/502999377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8608102210 0x7f8608108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.909+0000 7f860f688640 1 --2- 172.21.15.70:0/502999377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8608101850 0x7f8608101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.909+0000 7f860f688640 1 --2- 172.21.15.70:0/502999377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8608100660 0x7f8608100a80 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.909+0000 7f860f688640 1 -- 172.21.15.70:0/502999377 >> 172.21.15.70:0/502999377 conn(0x7f86080fbbf0 msgr2=0x7f86080fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:05.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.909+0000 7f860f688640 1 -- 172.21.15.70:0/502999377 shutdown_connections 2024-04-01T01:50:05.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.909+0000 7f860f688640 1 -- 172.21.15.70:0/502999377 wait complete. 2024-04-01T01:50:05.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.909+0000 7f860f688640 1 Processor -- start 2024-04-01T01:50:05.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.909+0000 7f860f688640 1 -- start start 2024-04-01T01:50:05.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860f688640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8608100660 0x7f8608197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:05.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860f688640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8608101850 0x7f8608197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:05.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860f688640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8608102210 0x7f860819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:05.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860f688640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f860810c180 con 0x7f8608100660 2024-04-01T01:50:05.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860f688640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f860810c000 con 0x7f8608101850 2024-04-01T01:50:05.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860f688640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f860810c300 con 0x7f8608102210 2024-04-01T01:50:05.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860d424640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8608100660 0x7f8608197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:05.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860d424640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8608100660 0x7f8608197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60564/0 (socket says 172.21.15.70:60564) 2024-04-01T01:50:05.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860d424640 1 -- 172.21.15.70:0/1216293223 learned_addr learned my addr 172.21.15.70:0/1216293223 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:05.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860cc23640 1 --2- 172.21.15.70:0/1216293223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8608101850 0x7f8608197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:05.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860dc25640 1 --2- 172.21.15.70:0/1216293223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8608102210 0x7f860819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:05.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860d424640 1 -- 172.21.15.70:0/1216293223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8608102210 msgr2=0x7f860819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:05.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860d424640 1 --2- 172.21.15.70:0/1216293223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8608102210 0x7f860819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860d424640 1 -- 172.21.15.70:0/1216293223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8608101850 msgr2=0x7f8608197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:05.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860d424640 1 --2- 172.21.15.70:0/1216293223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8608101850 0x7f8608197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:05.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860d424640 1 -- 172.21.15.70:0/1216293223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860819f620 con 0x7f8608100660 2024-04-01T01:50:05.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860d424640 1 --2- 172.21.15.70:0/1216293223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8608100660 0x7f8608197a00 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f8604005940 tx=0x7f86040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:05.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f85fe7fc640 1 -- 172.21.15.70:0/1216293223 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604041020 con 0x7f8608100660 2024-04-01T01:50:05.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f85fe7fc640 1 -- 172.21.15.70:0/1216293223 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86040035f0 con 0x7f8608100660 2024-04-01T01:50:05.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860f688640 1 -- 172.21.15.70:0/1216293223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860819f8b0 con 0x7f8608100660 2024-04-01T01:50:05.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f85fe7fc640 1 -- 172.21.15.70:0/1216293223 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604039040 con 0x7f8608100660 2024-04-01T01:50:05.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.910+0000 7f860f688640 1 -- 172.21.15.70:0/1216293223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f860819fda0 con 0x7f8608100660 2024-04-01T01:50:05.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.911+0000 7f860f688640 1 -- 172.21.15.70:0/1216293223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f860819ffa0 con 0x7f8608100660 2024-04-01T01:50:05.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.912+0000 7f85fe7fc640 1 -- 172.21.15.70:0/1216293223 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f8604044020 con 0x7f8608100660 2024-04-01T01:50:05.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.913+0000 7f85fe7fc640 1 --2- 172.21.15.70:0/1216293223 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85f40a24a0 0x7f85f40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:05.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.913+0000 7f860cc23640 1 --2- 172.21.15.70:0/1216293223 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85f40a24a0 0x7f85f40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:05.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.913+0000 7f85fe7fc640 1 -- 172.21.15.70:0/1216293223 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f86040f03c0 con 0x7f8608100660 2024-04-01T01:50:05.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.913+0000 7f860cc23640 1 --2- 172.21.15.70:0/1216293223 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85f40a24a0 0x7f85f40a4960 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f8608198f10 tx=0x7f85f8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:05.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:05.917+0000 7f85fe7fc640 1 -- 172.21.15.70:0/1216293223 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f860404b020 con 0x7f8608100660 2024-04-01T01:50:06.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.091+0000 7f860f688640 1 -- 172.21.15.70:0/1216293223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f86081a0330 con 0x7f8608100660 2024-04-01T01:50:06.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.092+0000 7f85fe7fc640 1 -- 172.21.15.70:0/1216293223 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f86040bd250 con 0x7f8608100660 2024-04-01T01:50:06.092 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:50:06.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.094+0000 7f860f688640 1 -- 172.21.15.70:0/1216293223 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85f40a24a0 msgr2=0x7f85f40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:06.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.094+0000 7f860f688640 1 --2- 172.21.15.70:0/1216293223 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85f40a24a0 0x7f85f40a4960 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f8608198f10 tx=0x7f85f8009260 comp rx=0 tx=0).stop 2024-04-01T01:50:06.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.094+0000 7f860f688640 1 -- 172.21.15.70:0/1216293223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8608100660 msgr2=0x7f8608197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:06.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.094+0000 7f860f688640 1 --2- 172.21.15.70:0/1216293223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8608100660 0x7f8608197a00 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f8604005940 tx=0x7f86040042f0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.094+0000 7f860f688640 1 -- 172.21.15.70:0/1216293223 shutdown_connections 2024-04-01T01:50:06.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.094+0000 7f860f688640 1 --2- 172.21.15.70:0/1216293223 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f85f40a24a0 0x7f85f40a4960 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.094+0000 7f860f688640 1 --2- 172.21.15.70:0/1216293223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8608102210 0x7f860819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.094+0000 7f860f688640 1 --2- 172.21.15.70:0/1216293223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8608101850 0x7f8608197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.094+0000 7f860f688640 1 --2- 172.21.15.70:0/1216293223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8608100660 0x7f8608197a00 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.094+0000 7f860f688640 1 -- 172.21.15.70:0/1216293223 >> 172.21.15.70:0/1216293223 conn(0x7f86080fbbf0 msgr2=0x7f8608069d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:06.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.094+0000 7f860f688640 1 -- 172.21.15.70:0/1216293223 shutdown_connections 2024-04-01T01:50:06.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.094+0000 7f860f688640 1 -- 172.21.15.70:0/1216293223 wait complete. 2024-04-01T01:50:06.104 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:05 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2549593005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:50:06.104 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:05 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1921387651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:50:06.105 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-01T01:50:06.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.211+0000 7fefe634b640 1 -- 172.21.15.70:0/1149341037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe0100670 msgr2=0x7fefe0100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:06.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.211+0000 7fefe634b640 1 --2- 172.21.15.70:0/1149341037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe0100670 0x7fefe0100a90 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fefd0009900 tx=0x7fefd002bc30 comp rx=0 tx=0).stop 2024-04-01T01:50:06.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.211+0000 7fefe634b640 1 -- 172.21.15.70:0/1149341037 shutdown_connections 2024-04-01T01:50:06.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.211+0000 7fefe634b640 1 --2- 172.21.15.70:0/1149341037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefe0102220 0x7fefe0108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.211+0000 7fefe634b640 1 --2- 172.21.15.70:0/1149341037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe0101860 0x7fefe0101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.211+0000 7fefe634b640 1 --2- 172.21.15.70:0/1149341037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe0100670 0x7fefe0100a90 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.211+0000 7fefe634b640 1 -- 172.21.15.70:0/1149341037 >> 172.21.15.70:0/1149341037 conn(0x7fefe00fbc20 msgr2=0x7fefe00fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:06.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.211+0000 7fefe634b640 1 -- 172.21.15.70:0/1149341037 shutdown_connections 2024-04-01T01:50:06.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.211+0000 7fefe634b640 1 -- 172.21.15.70:0/1149341037 wait complete. 2024-04-01T01:50:06.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefe634b640 1 Processor -- start 2024-04-01T01:50:06.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefe634b640 1 -- start start 2024-04-01T01:50:06.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefe634b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe0100670 0x7fefe01979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:06.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefe634b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe0101860 0x7fefe0197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefe634b640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefe0102220 0x7fefe019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefe634b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fefe010c190 con 0x7fefe0101860 2024-04-01T01:50:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefe634b640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fefe010c010 con 0x7fefe0102220 2024-04-01T01:50:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefe634b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fefe010c310 con 0x7fefe0100670 2024-04-01T01:50:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefdf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe0101860 0x7fefe0197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefdf7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe0101860 0x7fefe0197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60576/0 (socket says 172.21.15.70:60576) 2024-04-01T01:50:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefdffff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe0100670 0x7fefe01979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefdf7fe640 1 -- 172.21.15.70:0/2186794165 learned_addr learned my addr 172.21.15.70:0/2186794165 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefe48e8640 1 --2- 172.21.15.70:0/2186794165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefe0102220 0x7fefe019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefdf7fe640 1 -- 172.21.15.70:0/2186794165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe0100670 msgr2=0x7fefe01979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefdf7fe640 1 --2- 172.21.15.70:0/2186794165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe0100670 0x7fefe01979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefdf7fe640 1 -- 172.21.15.70:0/2186794165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefe0102220 msgr2=0x7fefe019ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefdf7fe640 1 --2- 172.21.15.70:0/2186794165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefe0102220 0x7fefe019ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.212+0000 7fefdf7fe640 1 -- 172.21.15.70:0/2186794165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe019f5d0 con 0x7fefe0101860 2024-04-01T01:50:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.213+0000 7fefdf7fe640 1 --2- 172.21.15.70:0/2186794165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe0101860 0x7fefe0197ef0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fefd400ea60 tx=0x7fefd400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:06.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.213+0000 7fefdd7fa640 1 -- 172.21.15.70:0/2186794165 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd400cde0 con 0x7fefe0101860 2024-04-01T01:50:06.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.213+0000 7fefe634b640 1 -- 172.21.15.70:0/2186794165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe019f8c0 con 0x7fefe0101860 2024-04-01T01:50:06.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.213+0000 7fefdd7fa640 1 -- 172.21.15.70:0/2186794165 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fefd40035f0 con 0x7fefe0101860 2024-04-01T01:50:06.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.213+0000 7fefdd7fa640 1 -- 172.21.15.70:0/2186794165 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd4010690 con 0x7fefe0101860 2024-04-01T01:50:06.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.213+0000 7fefe634b640 1 -- 172.21.15.70:0/2186794165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe019fe10 con 0x7fefe0101860 2024-04-01T01:50:06.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.214+0000 7fefe634b640 1 -- 172.21.15.70:0/2186794165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefe01a00a0 con 0x7fefe0101860 2024-04-01T01:50:06.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.215+0000 7fefdd7fa640 1 -- 172.21.15.70:0/2186794165 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fefd40037a0 con 0x7fefe0101860 2024-04-01T01:50:06.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.215+0000 7fefdd7fa640 1 --2- 172.21.15.70:0/2186794165 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fefc80a24a0 0x7fefc80a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:06.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.215+0000 7fefdffff640 1 --2- 172.21.15.70:0/2186794165 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fefc80a24a0 0x7fefc80a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:06.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.215+0000 7fefdd7fa640 1 -- 172.21.15.70:0/2186794165 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fefd40cef40 con 0x7fefe0101860 2024-04-01T01:50:06.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.216+0000 7fefdffff640 1 --2- 172.21.15.70:0/2186794165 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fefc80a24a0 0x7fefc80a4960 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fefd0005ab0 tx=0x7fefd0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:06.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.219+0000 7fefdd7fa640 1 -- 172.21.15.70:0/2186794165 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fefd4053280 con 0x7fefe0101860 2024-04-01T01:50:06.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:05 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2549593005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:50:06.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:05 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1921387651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:50:06.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:05 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2549593005' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:50:06.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:05 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1921387651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:50:06.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.393+0000 7fefe634b640 1 -- 172.21.15.70:0/2186794165 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fefe01a0270 con 0x7fefe0101860 2024-04-01T01:50:06.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.393+0000 7fefdd7fa640 1 -- 172.21.15.70:0/2186794165 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7fefd409bd30 con 0x7fefe0101860 2024-04-01T01:50:06.394 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-01T01:50:06.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.395+0000 7fefe634b640 1 -- 172.21.15.70:0/2186794165 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fefc80a24a0 msgr2=0x7fefc80a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:06.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.395+0000 7fefe634b640 1 --2- 172.21.15.70:0/2186794165 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fefc80a24a0 0x7fefc80a4960 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fefd0005ab0 tx=0x7fefd0005fd0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.395+0000 7fefe634b640 1 -- 172.21.15.70:0/2186794165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe0101860 msgr2=0x7fefe0197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:06.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.395+0000 7fefe634b640 1 --2- 172.21.15.70:0/2186794165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe0101860 0x7fefe0197ef0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fefd400ea60 tx=0x7fefd400ef20 comp rx=0 tx=0).stop 2024-04-01T01:50:06.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.395+0000 7fefe634b640 1 -- 172.21.15.70:0/2186794165 shutdown_connections 2024-04-01T01:50:06.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.395+0000 7fefe634b640 1 --2- 172.21.15.70:0/2186794165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefe0102220 0x7fefe019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.395+0000 7fefe634b640 1 --2- 172.21.15.70:0/2186794165 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe0101860 0x7fefe0197ef0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.395+0000 7fefe634b640 1 --2- 172.21.15.70:0/2186794165 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fefc80a24a0 0x7fefc80a4960 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.395+0000 7fefe634b640 1 --2- 172.21.15.70:0/2186794165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe0100670 0x7fefe01979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.395+0000 7fefe634b640 1 -- 172.21.15.70:0/2186794165 >> 172.21.15.70:0/2186794165 conn(0x7fefe00fbc20 msgr2=0x7fefe00fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:06.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.395+0000 7fefe634b640 1 -- 172.21.15.70:0/2186794165 shutdown_connections 2024-04-01T01:50:06.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.395+0000 7fefe634b640 1 -- 172.21.15.70:0/2186794165 wait complete. 2024-04-01T01:50:06.406 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-01T01:50:06.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.510+0000 7fe9eb5e2640 1 -- 172.21.15.70:0/1159899034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9e40ffdf0 msgr2=0x7fe9e4100210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:06.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.510+0000 7fe9eb5e2640 1 --2- 172.21.15.70:0/1159899034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9e40ffdf0 0x7fe9e4100210 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0009900 tx=0x7fe9e002bc30 comp rx=0 tx=0).stop 2024-04-01T01:50:06.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.510+0000 7fe9eb5e2640 1 -- 172.21.15.70:0/1159899034 shutdown_connections 2024-04-01T01:50:06.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.510+0000 7fe9eb5e2640 1 --2- 172.21.15.70:0/1159899034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e41019a0 0x7fe9e4108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.510+0000 7fe9eb5e2640 1 --2- 172.21.15.70:0/1159899034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9e4100fe0 0x7fe9e4101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.510+0000 7fe9eb5e2640 1 --2- 172.21.15.70:0/1159899034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9e40ffdf0 0x7fe9e4100210 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.510+0000 7fe9eb5e2640 1 -- 172.21.15.70:0/1159899034 >> 172.21.15.70:0/1159899034 conn(0x7fe9e40fb380 msgr2=0x7fe9e40fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:06.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.510+0000 7fe9eb5e2640 1 -- 172.21.15.70:0/1159899034 shutdown_connections 2024-04-01T01:50:06.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.510+0000 7fe9eb5e2640 1 -- 172.21.15.70:0/1159899034 wait complete. 2024-04-01T01:50:06.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9eb5e2640 1 Processor -- start 2024-04-01T01:50:06.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9eb5e2640 1 -- start start 2024-04-01T01:50:06.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9eb5e2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9e40ffdf0 0x7fe9e41979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:06.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9eb5e2640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9e4100fe0 0x7fe9e4197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:06.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9eb5e2640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e41019a0 0x7fe9e419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:06.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9eb5e2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e410c1c0 con 0x7fe9e40ffdf0 2024-04-01T01:50:06.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9eb5e2640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e410c040 con 0x7fe9e4100fe0 2024-04-01T01:50:06.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9eb5e2640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e410c340 con 0x7fe9e41019a0 2024-04-01T01:50:06.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9e937e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9e40ffdf0 0x7fe9e41979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:06.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9e937e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9e40ffdf0 0x7fe9e41979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60602/0 (socket says 172.21.15.70:60602) 2024-04-01T01:50:06.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9e937e640 1 -- 172.21.15.70:0/3743546289 learned_addr learned my addr 172.21.15.70:0/3743546289 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:06.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9e8b7d640 1 --2- 172.21.15.70:0/3743546289 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9e4100fe0 0x7fe9e4197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:06.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9e9b7f640 1 --2- 172.21.15.70:0/3743546289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e41019a0 0x7fe9e419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:06.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9e937e640 1 -- 172.21.15.70:0/3743546289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e41019a0 msgr2=0x7fe9e419ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:06.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9e937e640 1 --2- 172.21.15.70:0/3743546289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e41019a0 0x7fe9e419ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9e937e640 1 -- 172.21.15.70:0/3743546289 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9e4100fe0 msgr2=0x7fe9e4197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:06.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.511+0000 7fe9e937e640 1 --2- 172.21.15.70:0/3743546289 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9e4100fe0 0x7fe9e4197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.512+0000 7fe9e937e640 1 -- 172.21.15.70:0/3743546289 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e419f600 con 0x7fe9e40ffdf0 2024-04-01T01:50:06.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.512+0000 7fe9e937e640 1 --2- 172.21.15.70:0/3743546289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9e40ffdf0 0x7fe9e41979e0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0000c00 tx=0x7fe9e0005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:06.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.512+0000 7fe9da7fc640 1 -- 172.21.15.70:0/3743546289 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0043020 con 0x7fe9e40ffdf0 2024-04-01T01:50:06.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.512+0000 7fe9eb5e2640 1 -- 172.21.15.70:0/3743546289 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e419f890 con 0x7fe9e40ffdf0 2024-04-01T01:50:06.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.512+0000 7fe9da7fc640 1 -- 172.21.15.70:0/3743546289 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9e00043d0 con 0x7fe9e40ffdf0 2024-04-01T01:50:06.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.512+0000 7fe9da7fc640 1 -- 172.21.15.70:0/3743546289 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0003b40 con 0x7fe9e40ffdf0 2024-04-01T01:50:06.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.512+0000 7fe9eb5e2640 1 -- 172.21.15.70:0/3743546289 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e419fe30 con 0x7fe9e40ffdf0 2024-04-01T01:50:06.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.513+0000 7fe9eb5e2640 1 -- 172.21.15.70:0/3743546289 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9e4068990 con 0x7fe9e40ffdf0 2024-04-01T01:50:06.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.514+0000 7fe9da7fc640 1 -- 172.21.15.70:0/3743546289 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fe9e0046020 con 0x7fe9e40ffdf0 2024-04-01T01:50:06.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.514+0000 7fe9da7fc640 1 --2- 172.21.15.70:0/3743546289 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe9d00a24a0 0x7fe9d00a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:06.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.514+0000 7fe9da7fc640 1 -- 172.21.15.70:0/3743546289 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fe9e00f0c20 con 0x7fe9e40ffdf0 2024-04-01T01:50:06.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.514+0000 7fe9e8b7d640 1 --2- 172.21.15.70:0/3743546289 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe9d00a24a0 0x7fe9d00a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:06.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.515+0000 7fe9e8b7d640 1 --2- 172.21.15.70:0/3743546289 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe9d00a24a0 0x7fe9d00a4960 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40095f0 tx=0x7fe9d4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:06.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.519+0000 7fe9da7fc640 1 -- 172.21.15.70:0/3743546289 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe9e004d020 con 0x7fe9e40ffdf0 2024-04-01T01:50:06.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.692+0000 7fe9eb5e2640 1 -- 172.21.15.70:0/3743546289 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe9e41a0700 con 0x7fe9e40ffdf0 2024-04-01T01:50:06.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.693+0000 7fe9da7fc640 1 -- 172.21.15.70:0/3743546289 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7fe9e00bda10 con 0x7fe9e40ffdf0 2024-04-01T01:50:06.693 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:50:06.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.695+0000 7fe9eb5e2640 1 -- 172.21.15.70:0/3743546289 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe9d00a24a0 msgr2=0x7fe9d00a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:06.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.695+0000 7fe9eb5e2640 1 --2- 172.21.15.70:0/3743546289 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe9d00a24a0 0x7fe9d00a4960 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40095f0 tx=0x7fe9d4005f90 comp rx=0 tx=0).stop 2024-04-01T01:50:06.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.695+0000 7fe9eb5e2640 1 -- 172.21.15.70:0/3743546289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9e40ffdf0 msgr2=0x7fe9e41979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:06.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.695+0000 7fe9eb5e2640 1 --2- 172.21.15.70:0/3743546289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9e40ffdf0 0x7fe9e41979e0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0000c00 tx=0x7fe9e0005e60 comp rx=0 tx=0).stop 2024-04-01T01:50:06.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.695+0000 7fe9eb5e2640 1 -- 172.21.15.70:0/3743546289 shutdown_connections 2024-04-01T01:50:06.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.695+0000 7fe9eb5e2640 1 --2- 172.21.15.70:0/3743546289 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe9d00a24a0 0x7fe9d00a4960 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.695+0000 7fe9eb5e2640 1 --2- 172.21.15.70:0/3743546289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e41019a0 0x7fe9e419ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.695+0000 7fe9eb5e2640 1 --2- 172.21.15.70:0/3743546289 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9e4100fe0 0x7fe9e4197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.695+0000 7fe9eb5e2640 1 --2- 172.21.15.70:0/3743546289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9e40ffdf0 0x7fe9e41979e0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.695+0000 7fe9eb5e2640 1 -- 172.21.15.70:0/3743546289 >> 172.21.15.70:0/3743546289 conn(0x7fe9e40fb380 msgr2=0x7fe9e40fce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:06.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.695+0000 7fe9eb5e2640 1 -- 172.21.15.70:0/3743546289 shutdown_connections 2024-04-01T01:50:06.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.695+0000 7fe9eb5e2640 1 -- 172.21.15.70:0/3743546289 wait complete. 2024-04-01T01:50:06.707 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-01T01:50:06.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.807+0000 7f2c2a4aa640 1 -- 172.21.15.70:0/3376115468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c24100680 msgr2=0x7f2c24100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:06.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.807+0000 7f2c2a4aa640 1 --2- 172.21.15.70:0/3376115468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c24100680 0x7f2c24100aa0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009900 tx=0x7f2c0c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:50:06.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.807+0000 7f2c2a4aa640 1 -- 172.21.15.70:0/3376115468 shutdown_connections 2024-04-01T01:50:06.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.807+0000 7f2c2a4aa640 1 --2- 172.21.15.70:0/3376115468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c24102230 0x7f2c24108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.807+0000 7f2c2a4aa640 1 --2- 172.21.15.70:0/3376115468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c24101870 0x7f2c24101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.807+0000 7f2c2a4aa640 1 --2- 172.21.15.70:0/3376115468 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c24100680 0x7f2c24100aa0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.807+0000 7f2c2a4aa640 1 -- 172.21.15.70:0/3376115468 >> 172.21.15.70:0/3376115468 conn(0x7f2c240fbc30 msgr2=0x7f2c240fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:06.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.807+0000 7f2c2a4aa640 1 -- 172.21.15.70:0/3376115468 shutdown_connections 2024-04-01T01:50:06.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.807+0000 7f2c2a4aa640 1 -- 172.21.15.70:0/3376115468 wait complete. 2024-04-01T01:50:06.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.807+0000 7f2c2a4aa640 1 Processor -- start 2024-04-01T01:50:06.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.807+0000 7f2c2a4aa640 1 -- start start 2024-04-01T01:50:06.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c2a4aa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c24100680 0x7f2c24197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:06.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c2a4aa640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c24101870 0x7f2c24197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:06.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c2a4aa640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c24102230 0x7f2c2419efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:06.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c2a4aa640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2410d2d0 con 0x7f2c24100680 2024-04-01T01:50:06.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c2a4aa640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2410d150 con 0x7f2c24101870 2024-04-01T01:50:06.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c2a4aa640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2410d450 con 0x7f2c24102230 2024-04-01T01:50:06.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c23fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c24100680 0x7f2c24197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:06.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c23fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c24100680 0x7f2c24197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43688/0 (socket says 172.21.15.70:43688) 2024-04-01T01:50:06.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c23fff640 1 -- 172.21.15.70:0/1501003947 learned_addr learned my addr 172.21.15.70:0/1501003947 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:06.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c237fe640 1 --2- 172.21.15.70:0/1501003947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c24101870 0x7f2c24197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:06.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c28a47640 1 --2- 172.21.15.70:0/1501003947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c24102230 0x7f2c2419efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:06.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c23fff640 1 -- 172.21.15.70:0/1501003947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c24102230 msgr2=0x7f2c2419efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:06.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c23fff640 1 --2- 172.21.15.70:0/1501003947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c24102230 0x7f2c2419efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c23fff640 1 -- 172.21.15.70:0/1501003947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c24101870 msgr2=0x7f2c24197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:06.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c23fff640 1 --2- 172.21.15.70:0/1501003947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c24101870 0x7f2c24197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:06.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c23fff640 1 -- 172.21.15.70:0/1501003947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c2419f650 con 0x7f2c24100680 2024-04-01T01:50:06.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c23fff640 1 --2- 172.21.15.70:0/1501003947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c24100680 0x7f2c24197a30 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c000c00 tx=0x7f2c0c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:06.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.808+0000 7f2c217fa640 1 -- 172.21.15.70:0/1501003947 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c043020 con 0x7f2c24100680 2024-04-01T01:50:06.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.809+0000 7f2c2a4aa640 1 -- 172.21.15.70:0/1501003947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c2419f8e0 con 0x7f2c24100680 2024-04-01T01:50:06.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.809+0000 7f2c217fa640 1 -- 172.21.15.70:0/1501003947 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c0c004350 con 0x7f2c24100680 2024-04-01T01:50:06.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.809+0000 7f2c217fa640 1 -- 172.21.15.70:0/1501003947 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c003b10 con 0x7f2c24100680 2024-04-01T01:50:06.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.809+0000 7f2c2a4aa640 1 -- 172.21.15.70:0/1501003947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c2419fe30 con 0x7f2c24100680 2024-04-01T01:50:06.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.809+0000 7f2c2a4aa640 1 -- 172.21.15.70:0/1501003947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c24107a60 con 0x7f2c24100680 2024-04-01T01:50:06.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.811+0000 7f2c217fa640 1 -- 172.21.15.70:0/1501003947 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f2c0c046020 con 0x7f2c24100680 2024-04-01T01:50:06.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.811+0000 7f2c217fa640 1 --2- 172.21.15.70:0/1501003947 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2c100a24a0 0x7f2c100a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:06.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.811+0000 7f2c237fe640 1 --2- 172.21.15.70:0/1501003947 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2c100a24a0 0x7f2c100a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:06.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.811+0000 7f2c217fa640 1 -- 172.21.15.70:0/1501003947 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f2c0c0f1000 con 0x7f2c24100680 2024-04-01T01:50:06.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.812+0000 7f2c237fe640 1 --2- 172.21.15.70:0/1501003947 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2c100a24a0 0x7f2c100a4960 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f2c24198f40 tx=0x7f2c14005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:06.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:06.815+0000 7f2c217fa640 1 -- 172.21.15.70:0/1501003947 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2c0c04d020 con 0x7f2c24100680 2024-04-01T01:50:07.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:06 smithi070 ceph-mon[32179]: pgmap v858: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 4.8 MiB/s wr, 528 op/s 2024-04-01T01:50:07.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:06 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1216293223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:50:07.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:06 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2186794165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:50:07.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:06 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3743546289' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:50:07.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:07.037+0000 7f2c2a4aa640 1 -- 172.21.15.70:0/1501003947 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f2c2402d340 con 0x7f2c24100680 2024-04-01T01:50:07.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:07.044+0000 7f2c217fa640 1 -- 172.21.15.70:0/1501003947 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) ==== 81+0+738410 (secure 0 0 0) 0x7f2c0c0bddf0 con 0x7f2c24100680 2024-04-01T01:50:07.058 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:50:07.059 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:50:07.059 INFO:teuthology.orchestra.run.smithi070.stdout: "epoch": 20, 2024-04-01T01:50:07.059 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.059 INFO:teuthology.orchestra.run.smithi070.stdout: "active_gid": 14150, 2024-04-01T01:50:07.059 INFO:teuthology.orchestra.run.smithi070.stdout: "active_name": "x", 2024-04-01T01:50:07.059 INFO:teuthology.orchestra.run.smithi070.stdout: "active_addrs": { 2024-04-01T01:50:07.059 INFO:teuthology.orchestra.run.smithi070.stdout: "addrvec": [ 2024-04-01T01:50:07.059 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.059 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v2", 2024-04-01T01:50:07.059 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "172.21.15.70:6800", 2024-04-01T01:50:07.059 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 2070166741 2024-04-01T01:50:07.059 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.060 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.060 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-01T01:50:07.060 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "172.21.15.70:6801", 2024-04-01T01:50:07.060 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 2070166741 2024-04-01T01:50:07.060 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.060 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-01T01:50:07.060 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.060 INFO:teuthology.orchestra.run.smithi070.stdout: "active_addr": "172.21.15.70:6801/2070166741", 2024-04-01T01:50:07.060 INFO:teuthology.orchestra.run.smithi070.stdout: "active_change": "2024-04-01T01:24:18.477720+0000", 2024-04-01T01:50:07.060 INFO:teuthology.orchestra.run.smithi070.stdout: "active_mgr_features": 4540701547738038271, 2024-04-01T01:50:07.060 INFO:teuthology.orchestra.run.smithi070.stdout: "available": true, 2024-04-01T01:50:07.060 INFO:teuthology.orchestra.run.smithi070.stdout: "standbys": [ 2024-04-01T01:50:07.060 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.060 INFO:teuthology.orchestra.run.smithi070.stdout: "gid": 24106, 2024-04-01T01:50:07.060 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "z", 2024-04-01T01:50:07.061 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_features": 4540701547738038271, 2024-04-01T01:50:07.061 INFO:teuthology.orchestra.run.smithi070.stdout: "available_modules": [ 2024-04-01T01:50:07.061 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.061 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "alerts", 2024-04-01T01:50:07.061 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.061 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.061 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.061 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-04-01T01:50:07.061 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-04-01T01:50:07.061 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.062 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.062 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.062 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-04-01T01:50:07.062 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.062 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.062 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.062 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How frequently to reexamine health status", 2024-04-01T01:50:07.062 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.062 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.062 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.062 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.062 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.063 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.063 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.063 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.063 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.063 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.063 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.063 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.063 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.063 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.063 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.063 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.063 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.063 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.063 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.063 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.064 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.064 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.064 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.064 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.064 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.064 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.064 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.064 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.064 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.064 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.064 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.064 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.064 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.065 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.065 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.065 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.065 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.065 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.065 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.065 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.065 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.065 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.065 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.065 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.066 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.066 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.066 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.066 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.066 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.066 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.066 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.066 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.067 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.067 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.067 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.067 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.067 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.067 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.067 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.067 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.067 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.067 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.067 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.067 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.068 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.068 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.068 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.068 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.068 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.068 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.068 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.068 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.068 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.069 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.069 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_destination": { 2024-04-01T01:50:07.069 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_destination", 2024-04-01T01:50:07.069 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.069 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.069 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.069 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.070 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.070 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.070 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.070 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Email address to send alerts to", 2024-04-01T01:50:07.070 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.070 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.070 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.070 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.070 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_from_name": { 2024-04-01T01:50:07.070 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_from_name", 2024-04-01T01:50:07.070 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.071 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.071 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.071 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "Ceph", 2024-04-01T01:50:07.071 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.071 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.071 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.071 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Email From: name", 2024-04-01T01:50:07.071 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.071 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.071 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.071 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.071 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_host": { 2024-04-01T01:50:07.071 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_host", 2024-04-01T01:50:07.071 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.072 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.072 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.072 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.072 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.072 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.072 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.072 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP server", 2024-04-01T01:50:07.072 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.072 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.072 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.072 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.072 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_password": { 2024-04-01T01:50:07.072 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_password", 2024-04-01T01:50:07.072 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.072 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.073 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.073 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.073 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.073 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.073 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.073 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Password to authenticate with", 2024-04-01T01:50:07.073 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.073 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.073 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.073 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.073 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_port": { 2024-04-01T01:50:07.073 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_port", 2024-04-01T01:50:07.073 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.073 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.073 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.074 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "465", 2024-04-01T01:50:07.074 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.074 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.074 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.074 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP port", 2024-04-01T01:50:07.074 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.074 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.074 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.074 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.074 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_sender": { 2024-04-01T01:50:07.074 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_sender", 2024-04-01T01:50:07.074 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.074 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.075 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.075 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.075 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.075 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.075 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.075 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP envelope sender", 2024-04-01T01:50:07.075 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.075 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.075 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.075 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.075 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_ssl": { 2024-04-01T01:50:07.075 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_ssl", 2024-04-01T01:50:07.075 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.076 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.076 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.076 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.076 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.076 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.076 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.076 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-01T01:50:07.076 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.076 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.076 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.076 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.076 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_user": { 2024-04-01T01:50:07.076 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_user", 2024-04-01T01:50:07.076 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.076 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.077 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.077 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.077 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.077 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.077 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.077 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "User to authenticate as", 2024-04-01T01:50:07.077 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.077 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.077 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.077 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.077 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.077 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.077 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.077 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "balancer", 2024-04-01T01:50:07.077 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.078 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.078 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.078 INFO:teuthology.orchestra.run.smithi070.stdout: "active": { 2024-04-01T01:50:07.078 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "active", 2024-04-01T01:50:07.078 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.078 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.078 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.078 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.078 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.078 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.078 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.078 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "automatically balance PGs across cluster", 2024-04-01T01:50:07.078 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.079 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.079 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.079 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.079 INFO:teuthology.orchestra.run.smithi070.stdout: "begin_time": { 2024-04-01T01:50:07.079 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "begin_time", 2024-04-01T01:50:07.079 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.079 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.079 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.079 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0000", 2024-04-01T01:50:07.079 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.079 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.079 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.079 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "beginning time of day to automatically balance", 2024-04-01T01:50:07.079 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-01T01:50:07.080 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.080 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.080 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.080 INFO:teuthology.orchestra.run.smithi070.stdout: "begin_weekday": { 2024-04-01T01:50:07.080 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "begin_weekday", 2024-04-01T01:50:07.080 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-04-01T01:50:07.080 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.080 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.080 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-04-01T01:50:07.080 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0", 2024-04-01T01:50:07.080 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "6", 2024-04-01T01:50:07.080 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.080 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-01T01:50:07.080 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-01T01:50:07.080 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.081 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.081 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.081 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_max_iterations": { 2024-04-01T01:50:07.081 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_max_iterations", 2024-04-01T01:50:07.081 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-04-01T01:50:07.081 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.081 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.081 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "25", 2024-04-01T01:50:07.081 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-04-01T01:50:07.081 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "250", 2024-04-01T01:50:07.081 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.081 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-01T01:50:07.081 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.081 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.082 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.082 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.082 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_metrics": { 2024-04-01T01:50:07.082 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_metrics", 2024-04-01T01:50:07.082 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.082 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.082 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.082 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "pgs,objects,bytes", 2024-04-01T01:50:07.082 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.082 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.082 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.082 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-01T01:50:07.082 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-01T01:50:07.082 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.083 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.083 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.083 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_step": { 2024-04-01T01:50:07.083 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_step", 2024-04-01T01:50:07.083 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.083 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.083 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.083 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0.5", 2024-04-01T01:50:07.083 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0.001", 2024-04-01T01:50:07.083 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "0.999", 2024-04-01T01:50:07.083 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.083 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "aggressiveness of optimization", 2024-04-01T01:50:07.083 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-01T01:50:07.083 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.083 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.084 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.084 INFO:teuthology.orchestra.run.smithi070.stdout: "end_time": { 2024-04-01T01:50:07.084 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "end_time", 2024-04-01T01:50:07.084 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.084 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.084 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.084 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "2359", 2024-04-01T01:50:07.084 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.084 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.084 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.084 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "ending time of day to automatically balance", 2024-04-01T01:50:07.084 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-01T01:50:07.084 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.084 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.085 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.085 INFO:teuthology.orchestra.run.smithi070.stdout: "end_weekday": { 2024-04-01T01:50:07.085 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "end_weekday", 2024-04-01T01:50:07.085 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-04-01T01:50:07.085 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.085 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.085 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-04-01T01:50:07.085 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0", 2024-04-01T01:50:07.085 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "6", 2024-04-01T01:50:07.085 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.085 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-01T01:50:07.085 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-01T01:50:07.085 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.085 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.085 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.086 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.086 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.086 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.086 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.086 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.086 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.086 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.086 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.086 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.086 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.086 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.087 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.087 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.087 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.087 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.087 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.087 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.087 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.087 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.087 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.087 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.087 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.087 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.087 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.087 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.087 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.088 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.088 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.088 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.088 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.088 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.088 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.088 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.088 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.088 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.088 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.088 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.088 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.089 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.089 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.089 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.089 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.089 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.089 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.089 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.089 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.089 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.089 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.089 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.089 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.089 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.089 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.090 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.090 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.090 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.090 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.090 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.090 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.090 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.090 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.090 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.090 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.090 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.090 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.091 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.091 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.091 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.091 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.091 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.091 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.091 INFO:teuthology.orchestra.run.smithi070.stdout: "min_score": { 2024-04-01T01:50:07.091 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "min_score", 2024-04-01T01:50:07.091 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.091 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.091 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.091 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-04-01T01:50:07.091 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.091 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.091 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.092 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-01T01:50:07.092 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.092 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.092 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.092 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.092 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": { 2024-04-01T01:50:07.092 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mode", 2024-04-01T01:50:07.092 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.092 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.092 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.092 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "upmap", 2024-04-01T01:50:07.092 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.092 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.092 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.093 INFO:teuthology.orchestra.run.smithi070.stdout: "crush-compat", 2024-04-01T01:50:07.093 INFO:teuthology.orchestra.run.smithi070.stdout: "none", 2024-04-01T01:50:07.093 INFO:teuthology.orchestra.run.smithi070.stdout: "read", 2024-04-01T01:50:07.093 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap", 2024-04-01T01:50:07.093 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap-read" 2024-04-01T01:50:07.093 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.093 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Balancer mode", 2024-04-01T01:50:07.093 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.093 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.093 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.093 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.093 INFO:teuthology.orchestra.run.smithi070.stdout: "pool_ids": { 2024-04-01T01:50:07.093 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pool_ids", 2024-04-01T01:50:07.093 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.094 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.094 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.094 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.094 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.094 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.094 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.094 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-01T01:50:07.094 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.094 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.094 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.094 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.094 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-04-01T01:50:07.095 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-04-01T01:50:07.095 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.095 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.095 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.095 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-04-01T01:50:07.095 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.095 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.095 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.095 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-01T01:50:07.095 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.095 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.095 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.095 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.095 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap_max_deviation": { 2024-04-01T01:50:07.095 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "upmap_max_deviation", 2024-04-01T01:50:07.096 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.096 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.096 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.096 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-04-01T01:50:07.096 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-04-01T01:50:07.096 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.096 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.096 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-01T01:50:07.097 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-01T01:50:07.097 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.097 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.097 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.097 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap_max_optimizations": { 2024-04-01T01:50:07.097 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "upmap_max_optimizations", 2024-04-01T01:50:07.097 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-04-01T01:50:07.097 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.097 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.097 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.097 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.097 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.097 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.098 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-01T01:50:07.098 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.098 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.098 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.098 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.098 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.098 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.098 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.099 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cephadm", 2024-04-01T01:50:07.099 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.099 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.099 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.099 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_down_multiplier": { 2024-04-01T01:50:07.099 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_down_multiplier", 2024-04-01T01:50:07.099 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.099 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.099 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.100 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3.0", 2024-04-01T01:50:07.100 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.100 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.100 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.100 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-01T01:50:07.100 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.101 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.101 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.101 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.101 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_refresh_rate": { 2024-04-01T01:50:07.101 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_refresh_rate", 2024-04-01T01:50:07.101 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.101 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.101 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.101 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "20", 2024-04-01T01:50:07.101 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.101 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.101 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.101 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-01T01:50:07.101 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.102 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.102 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.102 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.102 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_starting_port": { 2024-04-01T01:50:07.102 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_starting_port", 2024-04-01T01:50:07.102 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.102 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.102 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.102 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "4721", 2024-04-01T01:50:07.102 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.102 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.102 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.102 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-01T01:50:07.103 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.103 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.103 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.103 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.103 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_ptrace": { 2024-04-01T01:50:07.103 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_ptrace", 2024-04-01T01:50:07.103 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.103 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.103 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.103 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.103 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.103 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.103 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.104 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-01T01:50:07.104 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-01T01:50:07.104 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.104 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.104 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.104 INFO:teuthology.orchestra.run.smithi070.stdout: "autotune_interval": { 2024-04-01T01:50:07.104 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "autotune_interval", 2024-04-01T01:50:07.104 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.104 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.104 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.104 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-04-01T01:50:07.104 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.104 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.105 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.105 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-01T01:50:07.105 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.105 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.105 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.105 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.105 INFO:teuthology.orchestra.run.smithi070.stdout: "autotune_memory_target_ratio": { 2024-04-01T01:50:07.105 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "autotune_memory_target_ratio", 2024-04-01T01:50:07.105 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.105 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.105 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.105 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0.7", 2024-04-01T01:50:07.106 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.106 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.106 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.106 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-01T01:50:07.106 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.106 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.106 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.106 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.106 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm_log_destination": { 2024-04-01T01:50:07.106 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cephadm_log_destination", 2024-04-01T01:50:07.106 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.106 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.106 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.107 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.107 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.107 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.107 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.107 INFO:teuthology.orchestra.run.smithi070.stdout: "file", 2024-04-01T01:50:07.107 INFO:teuthology.orchestra.run.smithi070.stdout: "file,syslog", 2024-04-01T01:50:07.107 INFO:teuthology.orchestra.run.smithi070.stdout: "syslog" 2024-04-01T01:50:07.107 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.107 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-01T01:50:07.107 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.107 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.107 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.107 INFO:teuthology.orchestra.run.smithi070.stdout: "cgroups_split": { 2024-04-01T01:50:07.107 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cgroups_split", 2024-04-01T01:50:07.108 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.108 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.108 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.108 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.108 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.108 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.108 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.108 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-01T01:50:07.108 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.108 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.108 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.109 INFO:teuthology.orchestra.run.smithi070.stdout: "config_checks_enabled": { 2024-04-01T01:50:07.109 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_checks_enabled", 2024-04-01T01:50:07.109 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.109 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.109 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.109 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.109 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.109 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.109 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.109 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-01T01:50:07.109 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.109 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.110 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.110 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.110 INFO:teuthology.orchestra.run.smithi070.stdout: "config_dashboard": { 2024-04-01T01:50:07.110 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_dashboard", 2024-04-01T01:50:07.110 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.110 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.110 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.111 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.111 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.111 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.111 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.111 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-01T01:50:07.111 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.111 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.111 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.111 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.111 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_alertmanager": { 2024-04-01T01:50:07.111 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_alertmanager", 2024-04-01T01:50:07.111 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.111 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.111 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.111 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-01T01:50:07.112 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.112 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.112 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.112 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-04-01T01:50:07.112 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.112 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.112 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.112 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.112 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_base": { 2024-04-01T01:50:07.112 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_base", 2024-04-01T01:50:07.112 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.112 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.112 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.112 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-01T01:50:07.113 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.113 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.113 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.113 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Container image name, without the tag", 2024-04-01T01:50:07.113 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.113 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.113 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.113 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.113 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_elasticsearch": { 2024-04-01T01:50:07.113 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_elasticsearch", 2024-04-01T01:50:07.113 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.113 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.113 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.113 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-01T01:50:07.113 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.114 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.114 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.114 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "elasticsearch container image", 2024-04-01T01:50:07.114 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.114 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.114 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.114 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_grafana": { 2024-04-01T01:50:07.114 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_grafana", 2024-04-01T01:50:07.115 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.115 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.115 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.115 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-01T01:50:07.115 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.115 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.116 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.116 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-04-01T01:50:07.116 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.116 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.116 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.116 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.117 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_haproxy": { 2024-04-01T01:50:07.117 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_haproxy", 2024-04-01T01:50:07.117 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.117 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.117 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.117 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-01T01:50:07.117 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.117 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.117 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.118 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "HAproxy container image", 2024-04-01T01:50:07.118 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.118 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.118 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.118 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.118 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_agent": { 2024-04-01T01:50:07.118 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_agent", 2024-04-01T01:50:07.118 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.118 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.118 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.118 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-01T01:50:07.119 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.119 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.119 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.119 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger agent container image", 2024-04-01T01:50:07.119 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.119 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.119 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.119 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_collector": { 2024-04-01T01:50:07.119 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_collector", 2024-04-01T01:50:07.119 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.119 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.119 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.120 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-01T01:50:07.120 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.120 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.120 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.120 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger collector container image", 2024-04-01T01:50:07.120 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.120 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.120 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.120 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_query": { 2024-04-01T01:50:07.121 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_query", 2024-04-01T01:50:07.121 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.121 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.121 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.121 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-01T01:50:07.121 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.121 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.121 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.121 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger query container image", 2024-04-01T01:50:07.121 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.121 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.122 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.122 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.123 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_keepalived": { 2024-04-01T01:50:07.123 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_keepalived", 2024-04-01T01:50:07.123 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.123 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.123 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.124 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-01T01:50:07.124 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.124 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.124 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.124 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Keepalived container image", 2024-04-01T01:50:07.124 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.125 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.125 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.125 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_loki": { 2024-04-01T01:50:07.125 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_loki", 2024-04-01T01:50:07.125 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.125 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.125 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.125 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-01T01:50:07.126 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.126 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.126 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.126 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Loki container image", 2024-04-01T01:50:07.126 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.126 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.126 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.126 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.126 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_node_exporter": { 2024-04-01T01:50:07.126 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_node_exporter", 2024-04-01T01:50:07.126 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.126 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.126 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.127 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-01T01:50:07.127 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.127 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.127 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.127 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-04-01T01:50:07.127 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.127 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.127 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.128 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_nvmeof": { 2024-04-01T01:50:07.128 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_nvmeof", 2024-04-01T01:50:07.128 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.128 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.128 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.129 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-04-01T01:50:07.129 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.129 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.129 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.129 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Nvme-of container image", 2024-04-01T01:50:07.129 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.130 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.130 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.130 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.130 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_prometheus": { 2024-04-01T01:50:07.130 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_prometheus", 2024-04-01T01:50:07.131 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.131 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.131 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.131 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-01T01:50:07.131 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.131 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.132 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.132 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-04-01T01:50:07.132 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.132 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.132 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.133 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_promtail": { 2024-04-01T01:50:07.133 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_promtail", 2024-04-01T01:50:07.133 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.133 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.133 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.133 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-01T01:50:07.134 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.134 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.134 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.134 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Promtail container image", 2024-04-01T01:50:07.134 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.135 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.135 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.135 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.135 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_snmp_gateway": { 2024-04-01T01:50:07.135 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_snmp_gateway", 2024-04-01T01:50:07.135 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.136 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.136 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.136 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-01T01:50:07.136 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.136 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.136 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.137 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SNMP Gateway container image", 2024-04-01T01:50:07.137 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.137 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.137 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.137 INFO:teuthology.orchestra.run.smithi070.stdout: "container_init": { 2024-04-01T01:50:07.138 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_init", 2024-04-01T01:50:07.138 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.138 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.138 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.138 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.138 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.139 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.139 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.139 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Run podman/docker with `--init`", 2024-04-01T01:50:07.139 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.139 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.139 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.140 INFO:teuthology.orchestra.run.smithi070.stdout: "daemon_cache_timeout": { 2024-04-01T01:50:07.140 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "daemon_cache_timeout", 2024-04-01T01:50:07.140 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.140 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.140 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.141 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-04-01T01:50:07.141 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.141 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.141 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.141 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-01T01:50:07.142 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.142 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.142 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.142 INFO:teuthology.orchestra.run.smithi070.stdout: "default_cephadm_command_timeout": { 2024-04-01T01:50:07.142 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "default_cephadm_command_timeout", 2024-04-01T01:50:07.142 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.143 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.143 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.143 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "900", 2024-04-01T01:50:07.143 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.143 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.143 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.143 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-01T01:50:07.143 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.144 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.144 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.144 INFO:teuthology.orchestra.run.smithi070.stdout: "default_registry": { 2024-04-01T01:50:07.144 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "default_registry", 2024-04-01T01:50:07.144 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.144 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.144 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.145 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io", 2024-04-01T01:50:07.145 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.145 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.145 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.145 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-01T01:50:07.145 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.145 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.145 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.146 INFO:teuthology.orchestra.run.smithi070.stdout: "device_cache_timeout": { 2024-04-01T01:50:07.146 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_cache_timeout", 2024-04-01T01:50:07.146 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.146 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.146 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.147 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1800", 2024-04-01T01:50:07.147 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.147 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.147 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.147 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache device inventory", 2024-04-01T01:50:07.147 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.147 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.147 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.147 INFO:teuthology.orchestra.run.smithi070.stdout: "device_enhanced_scan": { 2024-04-01T01:50:07.147 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_enhanced_scan", 2024-04-01T01:50:07.147 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.147 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.147 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.148 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.148 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.148 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.148 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.148 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-01T01:50:07.148 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.148 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.148 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.149 INFO:teuthology.orchestra.run.smithi070.stdout: "facts_cache_timeout": { 2024-04-01T01:50:07.149 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "facts_cache_timeout", 2024-04-01T01:50:07.149 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.149 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.149 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.149 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-04-01T01:50:07.149 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.149 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.149 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.149 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache host facts data", 2024-04-01T01:50:07.149 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.149 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.150 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.150 INFO:teuthology.orchestra.run.smithi070.stdout: "grafana_dashboards_path": { 2024-04-01T01:50:07.150 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "grafana_dashboards_path", 2024-04-01T01:50:07.150 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.150 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.150 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.150 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-01T01:50:07.150 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.150 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.150 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.150 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-01T01:50:07.151 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.151 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.151 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.151 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.151 INFO:teuthology.orchestra.run.smithi070.stdout: "host_check_interval": { 2024-04-01T01:50:07.151 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "host_check_interval", 2024-04-01T01:50:07.151 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.151 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.151 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.151 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-04-01T01:50:07.151 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.151 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.151 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.151 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to perform a host check", 2024-04-01T01:50:07.152 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.152 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.152 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.152 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.152 INFO:teuthology.orchestra.run.smithi070.stdout: "hw_monitoring": { 2024-04-01T01:50:07.152 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "hw_monitoring", 2024-04-01T01:50:07.152 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.152 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.152 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.152 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.152 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.152 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.152 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.152 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-01T01:50:07.152 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.153 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.153 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.153 INFO:teuthology.orchestra.run.smithi070.stdout: "inventory_list_all": { 2024-04-01T01:50:07.153 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "inventory_list_all", 2024-04-01T01:50:07.153 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.153 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.153 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.153 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.153 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.153 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.154 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.154 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-01T01:50:07.154 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.154 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.154 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.154 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.154 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.154 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.154 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.155 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.155 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.155 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.155 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.155 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.155 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.155 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.155 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.155 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.155 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.155 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.156 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.157 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.157 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.158 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.158 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.158 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.158 INFO:teuthology.orchestra.run.smithi070.stdout: "log_refresh_metadata": { 2024-04-01T01:50:07.158 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_refresh_metadata", 2024-04-01T01:50:07.159 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.159 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.159 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.159 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.159 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.159 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.159 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.159 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-01T01:50:07.159 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.160 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.160 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.160 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.160 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.160 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.161 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.161 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.161 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.161 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.161 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.162 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.162 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-01T01:50:07.162 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.162 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.162 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.163 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.163 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.163 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.163 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.163 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.163 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.164 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.164 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.164 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.164 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.164 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.164 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.165 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.165 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.165 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.165 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.165 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.165 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.166 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.166 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.166 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.167 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.167 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.167 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.167 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.167 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.167 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.167 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.168 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.168 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.168 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.168 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.168 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.168 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.169 INFO:teuthology.orchestra.run.smithi070.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-01T01:50:07.169 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-01T01:50:07.169 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.169 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.169 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.169 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.170 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.170 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.170 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.170 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-01T01:50:07.170 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.170 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.171 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.171 INFO:teuthology.orchestra.run.smithi070.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-01T01:50:07.171 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-01T01:50:07.171 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.171 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.171 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.171 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "*", 2024-04-01T01:50:07.172 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.172 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.172 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.172 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-01T01:50:07.172 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.172 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.172 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.172 INFO:teuthology.orchestra.run.smithi070.stdout: "max_count_per_host": { 2024-04-01T01:50:07.172 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_count_per_host", 2024-04-01T01:50:07.172 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.172 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.172 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.173 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.173 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.173 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.173 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.173 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "max number of daemons per service per host", 2024-04-01T01:50:07.173 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.173 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.173 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.173 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.173 INFO:teuthology.orchestra.run.smithi070.stdout: "max_osd_draining_count": { 2024-04-01T01:50:07.173 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_osd_draining_count", 2024-04-01T01:50:07.173 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.173 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.173 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.173 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.174 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.174 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.174 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.174 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-01T01:50:07.174 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.174 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.174 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.174 INFO:teuthology.orchestra.run.smithi070.stdout: "migration_current": { 2024-04-01T01:50:07.174 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "migration_current", 2024-04-01T01:50:07.174 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.174 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.174 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.174 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.175 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.175 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.175 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.175 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "internal - do not modify", 2024-04-01T01:50:07.175 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.175 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.175 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.175 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.175 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": { 2024-04-01T01:50:07.175 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mode", 2024-04-01T01:50:07.175 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.175 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.175 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.175 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "root", 2024-04-01T01:50:07.176 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.176 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.176 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.176 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm-package", 2024-04-01T01:50:07.176 INFO:teuthology.orchestra.run.smithi070.stdout: "root" 2024-04-01T01:50:07.176 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.176 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "mode for remote execution of cephadm", 2024-04-01T01:50:07.176 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.176 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.176 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.176 INFO:teuthology.orchestra.run.smithi070.stdout: "oob_default_addr": { 2024-04-01T01:50:07.176 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "oob_default_addr", 2024-04-01T01:50:07.176 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "169.254.1.1", 2024-04-01T01:50:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-01T01:50:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: "prometheus_alerts_path": { 2024-04-01T01:50:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prometheus_alerts_path", 2024-04-01T01:50:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.177 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-01T01:50:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-01T01:50:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_insecure": { 2024-04-01T01:50:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_insecure", 2024-04-01T01:50:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.178 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.179 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.179 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.179 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.179 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.179 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.179 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-01T01:50:07.179 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.179 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.179 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.179 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.179 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_password": { 2024-04-01T01:50:07.179 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_password", 2024-04-01T01:50:07.180 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.180 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.180 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.180 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.180 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.180 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.180 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.180 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-01T01:50:07.180 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.180 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.180 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.180 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_url": { 2024-04-01T01:50:07.181 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_url", 2024-04-01T01:50:07.181 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.181 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.181 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.181 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.181 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.181 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.181 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.181 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-01T01:50:07.181 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.181 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.181 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.181 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_username": { 2024-04-01T01:50:07.182 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_username", 2024-04-01T01:50:07.182 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.182 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.182 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.182 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.182 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.182 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.182 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.182 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-01T01:50:07.182 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.182 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.182 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.182 INFO:teuthology.orchestra.run.smithi070.stdout: "secure_monitoring_stack": { 2024-04-01T01:50:07.183 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "secure_monitoring_stack", 2024-04-01T01:50:07.183 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.183 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.183 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.183 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.183 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.183 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.183 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.183 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-01T01:50:07.183 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.183 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.183 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.184 INFO:teuthology.orchestra.run.smithi070.stdout: "service_discovery_port": { 2024-04-01T01:50:07.184 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "service_discovery_port", 2024-04-01T01:50:07.184 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.184 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.184 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.184 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8765", 2024-04-01T01:50:07.184 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.184 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.184 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.184 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "cephadm service discovery port", 2024-04-01T01:50:07.184 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.184 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.184 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.185 INFO:teuthology.orchestra.run.smithi070.stdout: "ssh_config_file": { 2024-04-01T01:50:07.185 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssh_config_file", 2024-04-01T01:50:07.185 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.185 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.185 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.185 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.185 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.185 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.185 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.185 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-01T01:50:07.185 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.185 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.185 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.186 INFO:teuthology.orchestra.run.smithi070.stdout: "use_agent": { 2024-04-01T01:50:07.186 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "use_agent", 2024-04-01T01:50:07.186 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.186 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.186 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.186 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.186 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.186 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.186 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.186 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-01T01:50:07.186 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.186 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.186 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.187 INFO:teuthology.orchestra.run.smithi070.stdout: "use_repo_digest": { 2024-04-01T01:50:07.187 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "use_repo_digest", 2024-04-01T01:50:07.187 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.187 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.187 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.187 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.187 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.187 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.187 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.187 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-01T01:50:07.187 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.187 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.187 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.188 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_failed_host_check": { 2024-04-01T01:50:07.188 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_failed_host_check", 2024-04-01T01:50:07.188 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.188 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.188 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.188 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.188 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.188 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.188 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.188 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if the host check fails", 2024-04-01T01:50:07.188 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.188 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.188 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.189 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_stray_daemons": { 2024-04-01T01:50:07.189 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_stray_daemons", 2024-04-01T01:50:07.189 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.189 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.189 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.189 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.189 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.189 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.189 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.189 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-01T01:50:07.189 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.189 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.189 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.189 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_stray_hosts": { 2024-04-01T01:50:07.190 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_stray_hosts", 2024-04-01T01:50:07.190 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.190 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.190 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.190 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.190 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.190 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.190 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.190 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-01T01:50:07.190 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.190 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.190 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.190 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.190 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.191 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crash", 2024-04-01T01:50:07.191 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.191 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.191 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.191 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.191 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.191 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.191 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.191 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.191 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.191 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.192 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.192 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.192 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.192 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.192 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.192 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.192 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.192 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.192 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.192 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.192 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.192 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.192 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.193 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.193 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.193 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.193 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.193 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.193 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.193 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.193 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.193 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.193 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.193 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.193 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.193 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.194 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.194 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.194 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.194 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.194 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.194 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.194 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.194 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.194 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.194 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.194 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.194 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.194 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.195 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.195 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.195 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.195 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.195 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.195 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.195 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.195 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.195 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.195 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.195 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.195 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.195 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.196 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.196 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.196 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.196 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.196 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.196 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.196 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.196 INFO:teuthology.orchestra.run.smithi070.stdout: "retain_interval": { 2024-04-01T01:50:07.196 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "retain_interval", 2024-04-01T01:50:07.196 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.196 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.196 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.196 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "31536000", 2024-04-01T01:50:07.197 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.197 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.197 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.197 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-01T01:50:07.197 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.197 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.197 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.197 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_recent_interval": { 2024-04-01T01:50:07.197 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_recent_interval", 2024-04-01T01:50:07.197 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.197 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.197 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.197 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1209600", 2024-04-01T01:50:07.197 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.198 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.198 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.198 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-01T01:50:07.198 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.198 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.198 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.198 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.198 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.198 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "dashboard", 2024-04-01T01:50:07.198 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.198 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.199 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.199 INFO:teuthology.orchestra.run.smithi070.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-01T01:50:07.199 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-01T01:50:07.199 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.199 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.199 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.199 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.199 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.199 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.199 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.199 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.199 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.199 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.199 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.200 INFO:teuthology.orchestra.run.smithi070.stdout: "ALERTMANAGER_API_HOST": { 2024-04-01T01:50:07.200 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-01T01:50:07.200 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.200 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.200 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.200 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.200 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.200 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.200 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.200 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.200 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.200 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.200 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.201 INFO:teuthology.orchestra.run.smithi070.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-01T01:50:07.201 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-01T01:50:07.201 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.201 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.201 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.201 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.201 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.201 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.201 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.201 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.201 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.201 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.201 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.201 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: "AUDIT_API_ENABLED": { 2024-04-01T01:50:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "AUDIT_API_ENABLED", 2024-04-01T01:50:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.203 INFO:teuthology.orchestra.run.smithi070.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-01T01:50:07.203 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-01T01:50:07.203 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.203 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.203 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.203 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.203 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.203 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.203 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.203 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.203 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.203 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.203 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.203 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.204 INFO:teuthology.orchestra.run.smithi070.stdout: "ENABLE_BROWSABLE_API": { 2024-04-01T01:50:07.204 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-01T01:50:07.204 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.204 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.204 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.204 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.204 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.204 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.204 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.204 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.204 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.204 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.204 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.204 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-01T01:50:07.205 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-01T01:50:07.205 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.205 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.205 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.205 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.205 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.205 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.205 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.205 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.205 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.205 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.205 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.206 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.206 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-01T01:50:07.206 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-01T01:50:07.206 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.206 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.206 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.206 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.206 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.206 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.206 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.206 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.206 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.206 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.206 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.207 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-01T01:50:07.207 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-01T01:50:07.207 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.207 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.207 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.207 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.207 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.207 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.207 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.207 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.207 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.207 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.208 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.208 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-01T01:50:07.208 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-01T01:50:07.208 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.208 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.208 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.208 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.208 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.208 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.208 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.208 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.208 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.208 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.208 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.209 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-01T01:50:07.209 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-01T01:50:07.209 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.209 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.209 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.209 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.209 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.209 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.209 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.209 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.209 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.209 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.209 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.210 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.210 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-01T01:50:07.210 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-01T01:50:07.210 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.210 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.210 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.210 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.210 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.210 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.210 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.210 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.210 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.210 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.211 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.211 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-01T01:50:07.211 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-01T01:50:07.211 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.211 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.211 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.211 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.211 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.211 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.211 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.211 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.211 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.211 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.212 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.212 INFO:teuthology.orchestra.run.smithi070.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-01T01:50:07.212 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-01T01:50:07.212 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.212 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.212 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.212 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.212 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.212 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.212 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.212 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.212 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.212 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.213 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.213 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_PASSWORD": { 2024-04-01T01:50:07.213 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-01T01:50:07.213 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.213 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.213 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.213 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-04-01T01:50:07.213 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.213 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.213 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.213 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.213 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.213 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.214 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.214 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-01T01:50:07.214 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-01T01:50:07.214 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.214 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.214 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.214 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.214 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.214 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.214 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.214 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.214 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.214 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.215 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.215 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_URL": { 2024-04-01T01:50:07.215 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_URL", 2024-04-01T01:50:07.215 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.215 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.215 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.215 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.215 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.215 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.215 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.215 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.215 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.215 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.216 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.216 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.216 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_USERNAME": { 2024-04-01T01:50:07.216 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-01T01:50:07.216 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.216 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.216 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.216 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-04-01T01:50:07.216 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.216 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.216 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.216 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.216 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.216 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.217 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.217 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-01T01:50:07.217 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-01T01:50:07.217 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.217 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.217 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.217 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.217 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.217 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.217 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.217 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.217 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.217 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.217 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.218 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.218 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-01T01:50:07.218 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-01T01:50:07.218 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.218 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.218 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.218 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.218 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.218 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.218 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.218 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.218 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.218 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.218 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.219 INFO:teuthology.orchestra.run.smithi070.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-01T01:50:07.219 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-01T01:50:07.219 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.219 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.219 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.219 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.219 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.219 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.219 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.219 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.220 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.220 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.220 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.220 INFO:teuthology.orchestra.run.smithi070.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-01T01:50:07.220 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-01T01:50:07.220 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.220 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.220 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.220 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.220 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.220 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.220 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.220 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.221 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.221 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.221 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.221 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.221 INFO:teuthology.orchestra.run.smithi070.stdout: "PROMETHEUS_API_HOST": { 2024-04-01T01:50:07.221 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-01T01:50:07.221 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.221 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.221 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.221 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.221 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.221 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.221 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.221 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.222 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.222 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.222 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.222 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.222 INFO:teuthology.orchestra.run.smithi070.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-01T01:50:07.222 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-01T01:50:07.222 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.222 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.222 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.222 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.222 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.222 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.222 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.222 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.223 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.223 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.223 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.223 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.223 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-01T01:50:07.223 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-01T01:50:07.223 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.223 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.223 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.223 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.223 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.223 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.223 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.224 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.224 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.224 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.224 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.224 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-01T01:50:07.224 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-01T01:50:07.224 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.224 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.224 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.224 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.224 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.224 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.225 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.225 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.225 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.225 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.225 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.225 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-01T01:50:07.225 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-01T01:50:07.225 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.225 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.225 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.225 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.225 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.225 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.225 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.226 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.226 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.226 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.226 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.226 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.226 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-01T01:50:07.226 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-01T01:50:07.226 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.226 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.226 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.226 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.226 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.226 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.226 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.227 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.227 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.227 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.227 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.227 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.227 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-01T01:50:07.227 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-01T01:50:07.227 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.227 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.227 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.227 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.227 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.227 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.227 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.228 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.228 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.228 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.228 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.228 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.228 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-01T01:50:07.228 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-01T01:50:07.228 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.228 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.228 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.228 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.228 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.228 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.228 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.229 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.229 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.229 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.229 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.229 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.229 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-01T01:50:07.229 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-01T01:50:07.229 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.229 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.229 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.229 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.229 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.229 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.229 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.230 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.230 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.230 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.230 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.230 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_ENABLED": { 2024-04-01T01:50:07.230 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-01T01:50:07.230 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.230 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.230 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.230 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.230 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.230 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.230 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.231 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.231 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.231 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.231 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.231 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-01T01:50:07.231 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-01T01:50:07.231 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.231 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.231 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.231 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-01T01:50:07.231 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.231 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.231 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.232 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.232 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.232 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.232 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.232 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-01T01:50:07.232 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-01T01:50:07.232 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.232 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.232 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.232 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.232 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.233 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.233 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.233 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.233 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.233 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.233 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.233 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-01T01:50:07.233 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-01T01:50:07.233 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.233 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.234 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.234 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8", 2024-04-01T01:50:07.234 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.234 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.234 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.234 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.234 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.234 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.234 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.234 INFO:teuthology.orchestra.run.smithi070.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-01T01:50:07.234 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-01T01:50:07.234 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.234 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.235 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.235 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "45", 2024-04-01T01:50:07.235 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.235 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.235 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.235 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.235 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.235 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.235 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.235 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_ACCESS_KEY": { 2024-04-01T01:50:07.235 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-01T01:50:07.235 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.236 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.236 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.236 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.236 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.236 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.236 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.236 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.236 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.236 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.236 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.236 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-01T01:50:07.236 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-01T01:50:07.236 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.237 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.237 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.237 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-04-01T01:50:07.237 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.237 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.237 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.237 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.237 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.237 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.237 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.237 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.237 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_SECRET_KEY": { 2024-04-01T01:50:07.237 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-01T01:50:07.237 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.238 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.238 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.238 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.238 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.238 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.238 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.238 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.238 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.238 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.238 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.238 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.238 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_SSL_VERIFY": { 2024-04-01T01:50:07.238 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-01T01:50:07.238 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.239 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.239 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.239 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.239 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.239 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.239 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.239 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.239 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.239 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.239 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.239 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-01T01:50:07.239 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-01T01:50:07.239 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.240 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.240 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.240 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-04-01T01:50:07.240 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.240 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.240 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.240 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.240 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.240 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.240 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.240 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-01T01:50:07.240 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-01T01:50:07.240 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.241 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.241 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.241 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.241 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.241 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.241 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.241 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.241 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.241 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.241 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.241 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-01T01:50:07.241 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-01T01:50:07.241 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.241 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.242 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.242 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-04-01T01:50:07.242 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.242 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.242 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.242 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.242 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.242 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.242 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.242 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.242 INFO:teuthology.orchestra.run.smithi070.stdout: "cross_origin_url": { 2024-04-01T01:50:07.242 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cross_origin_url", 2024-04-01T01:50:07.242 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.242 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.243 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.243 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.243 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.243 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.243 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.243 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.243 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.243 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.243 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.243 INFO:teuthology.orchestra.run.smithi070.stdout: "crt_file": { 2024-04-01T01:50:07.243 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crt_file", 2024-04-01T01:50:07.243 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.243 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.244 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.244 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.244 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.244 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.244 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.244 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.244 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.244 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.244 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.244 INFO:teuthology.orchestra.run.smithi070.stdout: "debug": { 2024-04-01T01:50:07.244 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "debug", 2024-04-01T01:50:07.244 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.244 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.244 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.245 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.245 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.245 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.245 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.245 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable/disable debug options", 2024-04-01T01:50:07.245 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.245 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.245 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.245 INFO:teuthology.orchestra.run.smithi070.stdout: "jwt_token_ttl": { 2024-04-01T01:50:07.245 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "jwt_token_ttl", 2024-04-01T01:50:07.245 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.246 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.246 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.246 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "28800", 2024-04-01T01:50:07.246 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.246 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.246 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.246 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.246 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.246 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.246 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.246 INFO:teuthology.orchestra.run.smithi070.stdout: "key_file": { 2024-04-01T01:50:07.246 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "key_file", 2024-04-01T01:50:07.246 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.247 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.247 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.247 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.247 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.247 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.247 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.247 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.247 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.247 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.247 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.247 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.247 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.247 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.248 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.248 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.248 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.248 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.248 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.248 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.248 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.248 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.248 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.248 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.248 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.248 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.248 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.248 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.249 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.249 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.249 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.249 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.249 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.249 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.249 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.249 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.249 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.249 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.249 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.249 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.249 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.250 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.250 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.250 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.250 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.250 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.250 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.250 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.250 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.250 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.250 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.250 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.250 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.250 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.251 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.251 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.251 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.251 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.251 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.251 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.251 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.251 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.251 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.251 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.251 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.251 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.252 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.252 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.252 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.252 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.252 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.252 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.252 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.252 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.252 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.252 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.252 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.252 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.252 INFO:teuthology.orchestra.run.smithi070.stdout: "motd": { 2024-04-01T01:50:07.252 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "motd", 2024-04-01T01:50:07.253 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.253 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.253 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.253 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.253 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.253 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.253 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.253 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "The message of the day", 2024-04-01T01:50:07.253 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.253 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.253 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.253 INFO:teuthology.orchestra.run.smithi070.stdout: "redirect_resolve_ip_addr": { 2024-04-01T01:50:07.253 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "redirect_resolve_ip_addr", 2024-04-01T01:50:07.253 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.254 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.254 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.254 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.254 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.254 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.254 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.254 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.254 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.254 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.254 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.254 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-04-01T01:50:07.254 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-04-01T01:50:07.255 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.255 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.255 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.255 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "::", 2024-04-01T01:50:07.255 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.255 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.255 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.255 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.255 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.255 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.255 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.255 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-04-01T01:50:07.255 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-04-01T01:50:07.255 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.256 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.256 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.256 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8080", 2024-04-01T01:50:07.256 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.256 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.256 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.256 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.256 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.256 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.256 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.256 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl": { 2024-04-01T01:50:07.256 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl", 2024-04-01T01:50:07.256 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.257 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.257 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.257 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.257 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.257 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.257 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.257 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.257 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.257 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.257 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.257 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl_server_port": { 2024-04-01T01:50:07.257 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl_server_port", 2024-04-01T01:50:07.257 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.258 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.258 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.258 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8443", 2024-04-01T01:50:07.258 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.258 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.258 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.258 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.258 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.258 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.258 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.258 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_behaviour": { 2024-04-01T01:50:07.258 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_behaviour", 2024-04-01T01:50:07.258 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.259 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.259 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.259 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "redirect", 2024-04-01T01:50:07.259 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.259 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.259 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.259 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.259 INFO:teuthology.orchestra.run.smithi070.stdout: "redirect" 2024-04-01T01:50:07.259 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.259 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.259 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.259 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.260 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.260 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_error_status_code": { 2024-04-01T01:50:07.260 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_error_status_code", 2024-04-01T01:50:07.260 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.260 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.260 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.260 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "500", 2024-04-01T01:50:07.260 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "400", 2024-04-01T01:50:07.260 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "599", 2024-04-01T01:50:07.260 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.260 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.260 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.260 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.261 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.261 INFO:teuthology.orchestra.run.smithi070.stdout: "url_prefix": { 2024-04-01T01:50:07.261 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "url_prefix", 2024-04-01T01:50:07.261 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.261 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.261 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.261 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.261 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.261 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.261 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.261 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.261 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.261 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.262 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.262 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.262 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.262 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "devicehealth", 2024-04-01T01:50:07.262 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.262 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.262 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.262 INFO:teuthology.orchestra.run.smithi070.stdout: "enable_monitoring": { 2024-04-01T01:50:07.262 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enable_monitoring", 2024-04-01T01:50:07.262 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.262 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.262 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.263 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.263 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.263 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.263 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.263 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "monitor device health metrics", 2024-04-01T01:50:07.263 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.263 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.263 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.263 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.263 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.263 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.263 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.264 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.264 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.264 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.264 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.264 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.264 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.264 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.264 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.264 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.264 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.264 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.264 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.264 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.264 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.265 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.265 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.265 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.265 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.265 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.265 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.265 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.265 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.265 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.265 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.265 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.265 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.265 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.266 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.266 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.266 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.266 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.266 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.266 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.266 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.266 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.266 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.266 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.266 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.266 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.266 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.267 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.267 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.267 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.267 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.267 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.267 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.267 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.267 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.267 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.267 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.267 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.267 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.267 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.267 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.268 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.268 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.268 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.268 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.268 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.268 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.268 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.268 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.268 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.268 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.268 INFO:teuthology.orchestra.run.smithi070.stdout: "mark_out_threshold": { 2024-04-01T01:50:07.268 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mark_out_threshold", 2024-04-01T01:50:07.268 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.269 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.269 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.269 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "2419200", 2024-04-01T01:50:07.269 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.269 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.269 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.269 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-01T01:50:07.269 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.269 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.269 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.269 INFO:teuthology.orchestra.run.smithi070.stdout: "pool_name": { 2024-04-01T01:50:07.269 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pool_name", 2024-04-01T01:50:07.269 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.269 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.270 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.270 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "device_health_metrics", 2024-04-01T01:50:07.270 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.270 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.270 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.270 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-01T01:50:07.270 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.270 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.270 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.270 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.270 INFO:teuthology.orchestra.run.smithi070.stdout: "retention_period": { 2024-04-01T01:50:07.270 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "retention_period", 2024-04-01T01:50:07.270 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.270 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.271 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.271 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15552000", 2024-04-01T01:50:07.271 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.271 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.271 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.271 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long to retain device health metrics", 2024-04-01T01:50:07.271 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.271 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.271 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.271 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.271 INFO:teuthology.orchestra.run.smithi070.stdout: "scrape_frequency": { 2024-04-01T01:50:07.271 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "scrape_frequency", 2024-04-01T01:50:07.271 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.271 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.272 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.272 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "86400", 2024-04-01T01:50:07.272 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.272 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.272 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.272 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-01T01:50:07.272 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.272 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.272 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.272 INFO:teuthology.orchestra.run.smithi070.stdout: "self_heal": { 2024-04-01T01:50:07.272 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "self_heal", 2024-04-01T01:50:07.272 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.272 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.272 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.273 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.273 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.273 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.273 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.273 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-01T01:50:07.273 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.273 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.273 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.273 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.273 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-04-01T01:50:07.273 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-04-01T01:50:07.273 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.273 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.274 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.274 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-04-01T01:50:07.274 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.274 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.274 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.274 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to wake up and check device health", 2024-04-01T01:50:07.274 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.274 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.274 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.274 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.274 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_threshold": { 2024-04-01T01:50:07.274 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_threshold", 2024-04-01T01:50:07.274 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.275 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.275 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.275 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "7257600", 2024-04-01T01:50:07.275 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.275 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.275 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.275 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-01T01:50:07.275 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.275 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.275 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.275 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.275 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.275 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.275 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.275 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "diskprediction_local", 2024-04-01T01:50:07.276 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.276 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.276 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.276 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.276 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.276 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.276 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.276 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.276 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.276 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.276 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.276 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.276 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.277 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.277 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.277 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.277 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.277 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.277 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.277 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.277 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.277 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.277 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.277 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.277 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.277 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.277 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.278 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.278 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.278 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.278 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.278 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.278 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.278 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.278 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.278 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.278 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.278 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.278 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.279 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.279 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.279 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.279 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.279 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.279 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.279 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.279 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.279 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.279 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.279 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.279 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.279 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.279 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.279 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.280 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.280 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.280 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.280 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.280 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.280 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.280 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.280 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.280 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.280 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.280 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.280 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.280 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.280 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.280 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.280 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.281 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.281 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.281 INFO:teuthology.orchestra.run.smithi070.stdout: "predict_interval": { 2024-04-01T01:50:07.281 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "predict_interval", 2024-04-01T01:50:07.281 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.281 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.281 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.281 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "86400", 2024-04-01T01:50:07.281 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.281 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.281 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.281 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.281 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.282 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.282 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.282 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.282 INFO:teuthology.orchestra.run.smithi070.stdout: "predictor_model": { 2024-04-01T01:50:07.282 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "predictor_model", 2024-04-01T01:50:07.282 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.282 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.282 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.282 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "prophetstor", 2024-04-01T01:50:07.282 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.282 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.282 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.282 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.283 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.283 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.283 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.283 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.283 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-04-01T01:50:07.283 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-04-01T01:50:07.283 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.283 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.283 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.283 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-04-01T01:50:07.283 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.283 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.283 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.283 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.284 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.284 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.284 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.284 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.284 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.284 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.284 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.284 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "influx", 2024-04-01T01:50:07.284 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": false, 2024-04-01T01:50:07.284 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "influxdb python module not found", 2024-04-01T01:50:07.284 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.284 INFO:teuthology.orchestra.run.smithi070.stdout: "batch_size": { 2024-04-01T01:50:07.284 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "batch_size", 2024-04-01T01:50:07.284 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.285 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.285 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.285 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5000", 2024-04-01T01:50:07.285 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.285 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.285 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.285 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-01T01:50:07.285 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.285 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.285 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.285 INFO:teuthology.orchestra.run.smithi070.stdout: "database": { 2024-04-01T01:50:07.285 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "database", 2024-04-01T01:50:07.285 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.285 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.286 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.286 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "ceph", 2024-04-01T01:50:07.286 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.286 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.286 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.286 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-01T01:50:07.286 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.286 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.286 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.286 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": { 2024-04-01T01:50:07.286 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "hostname", 2024-04-01T01:50:07.287 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.287 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.287 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.287 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.287 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.287 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.287 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.287 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB server hostname", 2024-04-01T01:50:07.287 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.287 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.287 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.287 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.287 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-04-01T01:50:07.287 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-04-01T01:50:07.288 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.288 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.288 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.288 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "30", 2024-04-01T01:50:07.288 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "5", 2024-04-01T01:50:07.288 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.288 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.288 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-01T01:50:07.288 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.288 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.288 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.288 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.288 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.289 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.289 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.289 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.289 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.289 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.289 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.289 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.289 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.289 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.289 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.289 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.289 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.289 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.289 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.290 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.290 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.290 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.290 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.290 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.290 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.290 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.290 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.290 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.290 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.290 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.290 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.290 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.291 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.291 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.291 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.291 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.291 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.291 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.291 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.291 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.291 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.291 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.291 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.291 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.291 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.291 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.292 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.292 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.292 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.292 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.292 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.292 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.292 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.292 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.292 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.292 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.292 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.292 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.292 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.292 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.293 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.293 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.293 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.293 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.293 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.293 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.293 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.293 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.293 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.293 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.293 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.293 INFO:teuthology.orchestra.run.smithi070.stdout: "password": { 2024-04-01T01:50:07.293 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "password", 2024-04-01T01:50:07.294 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.294 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.294 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.294 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.294 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.294 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.294 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.294 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "password of InfluxDB server user", 2024-04-01T01:50:07.294 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.294 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.294 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.294 INFO:teuthology.orchestra.run.smithi070.stdout: "port": { 2024-04-01T01:50:07.294 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "port", 2024-04-01T01:50:07.295 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.295 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.295 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.295 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8086", 2024-04-01T01:50:07.295 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.295 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.295 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.295 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB server port", 2024-04-01T01:50:07.295 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.295 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.295 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.295 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.295 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl": { 2024-04-01T01:50:07.295 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl", 2024-04-01T01:50:07.296 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.296 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.296 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.296 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "false", 2024-04-01T01:50:07.296 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.296 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.296 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.296 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-01T01:50:07.296 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.296 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.296 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.296 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.296 INFO:teuthology.orchestra.run.smithi070.stdout: "threads": { 2024-04-01T01:50:07.296 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "threads", 2024-04-01T01:50:07.296 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.297 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.297 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.297 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-04-01T01:50:07.297 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-04-01T01:50:07.297 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "32", 2024-04-01T01:50:07.297 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.297 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-01T01:50:07.297 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.297 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.297 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.297 INFO:teuthology.orchestra.run.smithi070.stdout: "username": { 2024-04-01T01:50:07.297 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "username", 2024-04-01T01:50:07.297 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.297 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.298 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.298 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.298 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.298 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.298 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.298 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "username of InfluxDB server user", 2024-04-01T01:50:07.298 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.298 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.298 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.298 INFO:teuthology.orchestra.run.smithi070.stdout: "verify_ssl": { 2024-04-01T01:50:07.298 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "verify_ssl", 2024-04-01T01:50:07.298 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.298 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.299 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.299 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "true", 2024-04-01T01:50:07.299 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.299 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.299 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.299 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-01T01:50:07.299 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.299 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.299 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.299 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.299 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.299 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.299 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.299 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "insights", 2024-04-01T01:50:07.300 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.300 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.300 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.300 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.300 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.300 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.300 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.300 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.300 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.300 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.300 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.300 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.300 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.300 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.301 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.301 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.301 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.301 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.301 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.301 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.301 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.301 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.301 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.301 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.301 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.301 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.301 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.301 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.302 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.302 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.302 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.302 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.302 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.302 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.302 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.302 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.302 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.302 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.302 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.302 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.302 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.302 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.303 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.303 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.303 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.303 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.303 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.303 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.303 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.303 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.303 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.303 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.303 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.303 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.303 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.303 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.304 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.304 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.304 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.304 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.304 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.304 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.304 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.304 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.304 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.304 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.304 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.304 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.304 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.304 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.305 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.305 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.305 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.305 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.305 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.305 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.305 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "iostat", 2024-04-01T01:50:07.305 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.305 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.305 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.305 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.305 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.305 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.305 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.305 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.306 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.306 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.306 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.306 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.306 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.306 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.306 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.306 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.306 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.306 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.306 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.306 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.306 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.307 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.307 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.307 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.307 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.307 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.307 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.307 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.307 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.307 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.307 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.307 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.307 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.307 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.307 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.308 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.308 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.308 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.308 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.308 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.308 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.308 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.308 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.308 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.308 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.308 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.308 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.308 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.308 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.308 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.309 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.309 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.309 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.309 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.309 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.309 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.309 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.309 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.309 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.309 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.309 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.309 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.309 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.310 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.310 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.310 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.310 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.310 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.310 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.310 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.310 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.310 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.310 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.310 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "k8sevents", 2024-04-01T01:50:07.310 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.311 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.311 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.311 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_event_retention_days": { 2024-04-01T01:50:07.311 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ceph_event_retention_days", 2024-04-01T01:50:07.311 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.311 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.311 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.311 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "7", 2024-04-01T01:50:07.311 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.311 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.311 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.311 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-01T01:50:07.311 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.311 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.312 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.312 INFO:teuthology.orchestra.run.smithi070.stdout: "config_check_secs": { 2024-04-01T01:50:07.312 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_check_secs", 2024-04-01T01:50:07.312 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.312 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.312 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.312 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.312 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "10", 2024-04-01T01:50:07.312 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.312 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.312 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-01T01:50:07.312 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.312 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.313 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.313 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.313 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.313 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.313 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.313 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.313 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.313 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.313 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.313 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.313 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.313 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.314 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.314 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.314 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.314 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.314 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.314 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.314 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.314 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.314 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.314 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.314 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.314 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.315 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.315 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.315 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.315 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.315 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.315 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.315 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.315 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.315 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.315 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.315 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.315 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.315 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.316 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.316 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.316 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.316 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.316 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.316 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.316 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.316 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.316 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.316 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.316 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.316 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.316 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.316 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.317 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.317 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.317 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.317 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.317 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.317 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.317 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.317 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.317 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.317 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.317 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.317 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.317 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.317 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.318 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.318 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.318 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.318 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.318 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.318 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.318 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.318 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.318 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "localpool", 2024-04-01T01:50:07.318 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.318 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.318 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.319 INFO:teuthology.orchestra.run.smithi070.stdout: "failure_domain": { 2024-04-01T01:50:07.319 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "failure_domain", 2024-04-01T01:50:07.319 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.319 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.319 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.319 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "host", 2024-04-01T01:50:07.319 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.319 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.319 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.319 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "failure domain for any created local pool", 2024-04-01T01:50:07.319 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-01T01:50:07.319 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.319 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.320 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.320 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.320 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.320 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.320 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.320 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.320 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.320 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.320 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.320 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.320 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.320 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.321 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.321 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.321 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.321 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.321 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.321 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.321 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.321 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.321 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.321 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.321 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.321 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.322 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.322 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.322 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.322 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.322 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.322 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.322 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.322 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.322 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.322 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.322 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.322 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.323 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.323 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.323 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.323 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.323 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.323 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.323 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.323 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.323 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.323 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.323 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.323 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.323 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.323 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.324 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.324 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.324 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.324 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.324 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.324 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.324 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.324 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.324 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.324 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.324 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.324 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.324 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.324 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.325 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.325 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.325 INFO:teuthology.orchestra.run.smithi070.stdout: "min_size": { 2024-04-01T01:50:07.325 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "min_size", 2024-04-01T01:50:07.325 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.325 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.325 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.325 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.325 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.325 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.325 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.325 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default min_size for any created local pool", 2024-04-01T01:50:07.325 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-01T01:50:07.326 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.326 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.326 INFO:teuthology.orchestra.run.smithi070.stdout: "num_rep": { 2024-04-01T01:50:07.326 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "num_rep", 2024-04-01T01:50:07.326 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.326 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.326 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.326 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3", 2024-04-01T01:50:07.326 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.326 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.326 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.326 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default replica count for any created local pool", 2024-04-01T01:50:07.326 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.327 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.327 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.327 INFO:teuthology.orchestra.run.smithi070.stdout: "pg_num": { 2024-04-01T01:50:07.327 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pg_num", 2024-04-01T01:50:07.327 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.327 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.327 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.327 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "128", 2024-04-01T01:50:07.327 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.327 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.327 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.327 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default pg_num for any created local pool", 2024-04-01T01:50:07.328 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.328 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.328 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.328 INFO:teuthology.orchestra.run.smithi070.stdout: "prefix": { 2024-04-01T01:50:07.328 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prefix", 2024-04-01T01:50:07.328 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.328 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.328 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.328 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.328 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.328 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.328 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.328 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "name prefix for any created local pool", 2024-04-01T01:50:07.329 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.329 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.329 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.329 INFO:teuthology.orchestra.run.smithi070.stdout: "subtree": { 2024-04-01T01:50:07.329 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "subtree", 2024-04-01T01:50:07.329 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.329 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.329 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.329 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "rack", 2024-04-01T01:50:07.329 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.329 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.329 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.329 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-01T01:50:07.330 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-01T01:50:07.330 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.330 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.330 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.330 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.330 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mds_autoscaler", 2024-04-01T01:50:07.330 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.330 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.330 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.330 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.330 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.330 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.331 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.331 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.331 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.331 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.331 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.331 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.331 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.331 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.331 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.331 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.331 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.331 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.331 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.331 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.332 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.332 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.332 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.332 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.332 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.332 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.332 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.332 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.332 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.332 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.332 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.332 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.332 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.332 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.333 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.333 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.333 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.333 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.333 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.333 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.333 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.333 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.333 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.333 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.333 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.333 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.333 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.334 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.334 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.334 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.334 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.334 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.334 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.334 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.334 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.334 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.334 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.334 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.334 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.335 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.335 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.335 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.335 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.335 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.335 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.335 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.335 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.335 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.335 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.335 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.335 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.336 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mirroring", 2024-04-01T01:50:07.336 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.336 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.336 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.336 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.336 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.336 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.336 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.336 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.336 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.336 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.336 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.336 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.337 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.338 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.339 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.340 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "nfs", 2024-04-01T01:50:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.341 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.342 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.343 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "orchestrator", 2024-04-01T01:50:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "fail_fs": { 2024-04-01T01:50:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "fail_fs", 2024-04-01T01:50:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-01T01:50:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "orchestrator": { 2024-04-01T01:50:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "orchestrator", 2024-04-01T01:50:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm", 2024-04-01T01:50:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "rook", 2024-04-01T01:50:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "test_orchestrator" 2024-04-01T01:50:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Orchestrator backend", 2024-04-01T01:50:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "osd_perf_query", 2024-04-01T01:50:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "osd_support", 2024-04-01T01:50:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pg_autoscaler", 2024-04-01T01:50:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-04-01T01:50:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-04-01T01:50:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-04-01T01:50:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "threshold": { 2024-04-01T01:50:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "threshold", 2024-04-01T01:50:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3.0", 2024-04-01T01:50:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1.0", 2024-04-01T01:50:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "scaling threshold", 2024-04-01T01:50:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-01T01:50:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "progress", 2024-04-01T01:50:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_pg_recovery_event": { 2024-04-01T01:50:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_pg_recovery_event", 2024-04-01T01:50:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-01T01:50:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": { 2024-04-01T01:50:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enabled", 2024-04-01T01:50:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "max_completed_events": { 2024-04-01T01:50:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_completed_events", 2024-04-01T01:50:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "50", 2024-04-01T01:50:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "number of past completed events to remember", 2024-04-01T01:50:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-04-01T01:50:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-04-01T01:50:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-04-01T01:50:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long the module is going to sleep", 2024-04-01T01:50:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prometheus", 2024-04-01T01:50:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "cache": { 2024-04-01T01:50:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cache", 2024-04-01T01:50:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "exclude_perf_counters": { 2024-04-01T01:50:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "exclude_perf_counters", 2024-04-01T01:50:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-01T01:50:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-01T01:50:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_stats_pools": { 2024-04-01T01:50:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_stats_pools", 2024-04-01T01:50:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-01T01:50:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-01T01:50:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "300", 2024-04-01T01:50:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "scrape_interval": { 2024-04-01T01:50:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "scrape_interval", 2024-04-01T01:50:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15.0", 2024-04-01T01:50:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-04-01T01:50:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-04-01T01:50:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "::", 2024-04-01T01:50:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-01T01:50:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-04-01T01:50:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-04-01T01:50:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "9283", 2024-04-01T01:50:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-01T01:50:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "stale_cache_strategy": { 2024-04-01T01:50:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "stale_cache_strategy", 2024-04-01T01:50:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "log", 2024-04-01T01:50:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_behaviour": { 2024-04-01T01:50:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_behaviour", 2024-04-01T01:50:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "default", 2024-04-01T01:50:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "default", 2024-04-01T01:50:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "error" 2024-04-01T01:50:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_error_status_code": { 2024-04-01T01:50:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_error_status_code", 2024-04-01T01:50:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "500", 2024-04-01T01:50:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "400", 2024-04-01T01:50:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "599", 2024-04-01T01:50:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_support", 2024-04-01T01:50:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "max_concurrent_snap_create": { 2024-04-01T01:50:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_concurrent_snap_create", 2024-04-01T01:50:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "mirror_snapshot_schedule": { 2024-04-01T01:50:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mirror_snapshot_schedule", 2024-04-01T01:50:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "trash_purge_schedule": { 2024-04-01T01:50:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "trash_purge_schedule", 2024-04-01T01:50:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "restful", 2024-04-01T01:50:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "enable_auth": { 2024-04-01T01:50:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enable_auth", 2024-04-01T01:50:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "key_file": { 2024-04-01T01:50:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "key_file", 2024-04-01T01:50:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-04-01T01:50:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-04-01T01:50:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-04-01T01:50:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-04-01T01:50:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rgw", 2024-04-01T01:50:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rook", 2024-04-01T01:50:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "storage_class": { 2024-04-01T01:50:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "storage_class", 2024-04-01T01:50:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "local", 2024-04-01T01:50:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-01T01:50:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "selftest", 2024-04-01T01:50:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.435 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.435 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.435 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.435 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.435 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.435 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.435 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.435 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.435 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.435 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.435 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.435 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.436 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.436 INFO:teuthology.orchestra.run.smithi070.stdout: "roption1": { 2024-04-01T01:50:07.436 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "roption1", 2024-04-01T01:50:07.436 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.436 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.436 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.436 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.436 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.436 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.436 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.436 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.436 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.437 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.437 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.437 INFO:teuthology.orchestra.run.smithi070.stdout: "roption2": { 2024-04-01T01:50:07.437 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "roption2", 2024-04-01T01:50:07.437 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.437 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.437 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.437 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "xyz", 2024-04-01T01:50:07.437 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.437 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.437 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.437 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.438 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.438 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.438 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.438 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption1": { 2024-04-01T01:50:07.438 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption1", 2024-04-01T01:50:07.438 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.438 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.438 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.438 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.438 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.438 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.438 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.439 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.439 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.439 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.439 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.439 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption2": { 2024-04-01T01:50:07.439 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption2", 2024-04-01T01:50:07.439 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.439 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.439 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.439 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.439 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.439 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.439 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.439 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.440 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.440 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.440 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.440 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption3": { 2024-04-01T01:50:07.440 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption3", 2024-04-01T01:50:07.440 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.440 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.440 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.440 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.440 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.440 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.440 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.440 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.441 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.441 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.441 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.441 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption4": { 2024-04-01T01:50:07.441 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption4", 2024-04-01T01:50:07.441 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.441 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.441 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.441 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.441 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.441 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.441 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.442 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.442 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.442 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.442 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.442 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption5": { 2024-04-01T01:50:07.442 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption5", 2024-04-01T01:50:07.442 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.442 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.442 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.442 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.442 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.442 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.443 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.443 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.443 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.443 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.443 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.443 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.443 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption6": { 2024-04-01T01:50:07.443 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption6", 2024-04-01T01:50:07.443 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.443 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.443 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.443 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.443 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.444 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.444 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.444 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.444 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.444 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.444 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.444 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.444 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption7": { 2024-04-01T01:50:07.444 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption7", 2024-04-01T01:50:07.444 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.444 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.444 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.444 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.444 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-04-01T01:50:07.445 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "42", 2024-04-01T01:50:07.445 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.445 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.445 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.445 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.445 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.445 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.445 INFO:teuthology.orchestra.run.smithi070.stdout: "testkey": { 2024-04-01T01:50:07.445 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testkey", 2024-04-01T01:50:07.445 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.445 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.445 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.445 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.445 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.446 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.446 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.446 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.446 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.446 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.446 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.446 INFO:teuthology.orchestra.run.smithi070.stdout: "testlkey": { 2024-04-01T01:50:07.446 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testlkey", 2024-04-01T01:50:07.446 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.446 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.446 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.446 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.446 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.447 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.447 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.447 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.447 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.447 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.447 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.447 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.447 INFO:teuthology.orchestra.run.smithi070.stdout: "testnewline": { 2024-04-01T01:50:07.447 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testnewline", 2024-04-01T01:50:07.447 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.447 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.447 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.447 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.447 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.448 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.448 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.448 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.448 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.448 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.448 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.448 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.448 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.448 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.448 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snap_schedule", 2024-04-01T01:50:07.448 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.448 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.449 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.449 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_m_granularity": { 2024-04-01T01:50:07.449 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_m_granularity", 2024-04-01T01:50:07.449 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.449 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.449 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.449 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.449 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.449 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.449 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.449 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow minute scheduled snapshots", 2024-04-01T01:50:07.449 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.449 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.449 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.450 INFO:teuthology.orchestra.run.smithi070.stdout: "dump_on_update": { 2024-04-01T01:50:07.450 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "dump_on_update", 2024-04-01T01:50:07.450 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.450 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.450 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.450 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.450 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.450 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.450 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.450 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "dump database to debug log on update", 2024-04-01T01:50:07.450 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.450 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.450 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.451 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.452 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.452 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.452 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.452 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.452 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.452 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.452 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.452 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.452 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.452 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.452 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.452 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.452 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.452 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.453 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.453 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.453 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.453 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.453 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.453 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.453 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.453 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.453 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.453 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.453 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.453 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.453 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.453 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.454 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.454 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.454 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.454 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.454 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.454 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.454 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.454 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.454 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.454 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.454 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.454 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.454 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.455 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.455 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.455 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.455 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.455 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.455 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.455 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.455 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.455 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.455 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.455 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.455 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.455 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.456 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.456 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.456 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.456 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.456 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.456 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "stats", 2024-04-01T01:50:07.456 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.456 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.456 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.456 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.456 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.456 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.456 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.457 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.457 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.457 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.457 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.457 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.457 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.457 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.457 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.457 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.457 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.457 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.457 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.457 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.457 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.458 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.458 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.458 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.458 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.458 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.458 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.458 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.458 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.458 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.458 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.458 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.458 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.458 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.458 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.459 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.459 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.459 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.459 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.459 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.459 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.459 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.459 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.459 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.459 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.459 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.459 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.459 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.459 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.460 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.460 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.460 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.460 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.460 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.460 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.460 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.460 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.460 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.460 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.460 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.460 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.460 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.460 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.461 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.461 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.461 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.461 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.461 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.461 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.461 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.461 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.461 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.461 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.461 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.461 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.461 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.462 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "status", 2024-04-01T01:50:07.462 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.462 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.462 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.462 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.462 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.462 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.462 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.462 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.462 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.462 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.462 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.462 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.462 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.463 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.463 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.463 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.463 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.463 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.463 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.463 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.463 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.463 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.463 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.463 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.463 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.463 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.463 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.463 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.464 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.464 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.464 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.464 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.464 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.464 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.464 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.464 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.464 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.464 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.464 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.464 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.464 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.464 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.465 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.465 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.465 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.465 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.465 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.465 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.465 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.465 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.465 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.465 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.465 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.465 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.465 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.466 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.466 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.466 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.466 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.466 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.466 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.466 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.466 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.466 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.466 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.466 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.466 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.466 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.466 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.467 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.467 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.467 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.467 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.467 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.467 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.467 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.467 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "telegraf", 2024-04-01T01:50:07.467 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.467 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.467 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.467 INFO:teuthology.orchestra.run.smithi070.stdout: "address": { 2024-04-01T01:50:07.467 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "address", 2024-04-01T01:50:07.467 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.468 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.468 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.468 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-01T01:50:07.468 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.468 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.468 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.468 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.468 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.468 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.468 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.468 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.468 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-04-01T01:50:07.468 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-04-01T01:50:07.468 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.468 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.469 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.469 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15", 2024-04-01T01:50:07.469 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.469 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.469 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.469 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.469 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.469 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.469 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.469 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.469 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.469 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.469 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.470 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.470 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.470 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.470 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.470 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.470 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.470 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.470 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.470 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.470 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.470 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.470 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.470 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.471 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.472 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.472 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.472 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.472 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.472 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.472 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.472 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.472 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.472 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.472 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.472 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.472 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.472 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.472 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.473 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.473 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.473 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.473 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.473 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.473 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.473 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.473 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.473 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.473 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.473 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.473 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.474 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.474 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.474 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.474 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.474 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.474 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.474 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.474 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.474 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.474 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.474 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.474 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.474 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.474 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.475 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.475 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.475 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.475 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "telemetry", 2024-04-01T01:50:07.475 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.475 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.475 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.475 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_basic": { 2024-04-01T01:50:07.475 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_basic", 2024-04-01T01:50:07.475 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.475 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.475 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.475 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.476 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.476 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.476 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.476 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-01T01:50:07.476 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.476 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.476 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.476 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.476 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_crash": { 2024-04-01T01:50:07.476 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_crash", 2024-04-01T01:50:07.476 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.476 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.476 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.476 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.476 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.477 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.477 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.477 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-01T01:50:07.477 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.477 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.477 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.477 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.477 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_device": { 2024-04-01T01:50:07.477 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_device", 2024-04-01T01:50:07.477 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.477 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.477 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.477 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.477 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.478 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.478 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.478 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-01T01:50:07.478 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.478 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.478 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.478 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.478 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_ident": { 2024-04-01T01:50:07.478 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_ident", 2024-04-01T01:50:07.478 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.478 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.478 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.478 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.478 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.479 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.479 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.479 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-01T01:50:07.479 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.479 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.479 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.479 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.479 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_perf": { 2024-04-01T01:50:07.479 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_perf", 2024-04-01T01:50:07.479 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.479 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.479 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.479 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.480 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.480 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.480 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.480 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-01T01:50:07.480 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.480 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.480 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.480 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.480 INFO:teuthology.orchestra.run.smithi070.stdout: "contact": { 2024-04-01T01:50:07.480 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "contact", 2024-04-01T01:50:07.480 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.480 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.480 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.480 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.481 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.481 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.481 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.481 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.481 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.481 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.481 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.481 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.481 INFO:teuthology.orchestra.run.smithi070.stdout: "description": { 2024-04-01T01:50:07.481 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "description", 2024-04-01T01:50:07.481 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.481 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.481 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.481 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.482 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.482 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.482 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.482 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.482 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.482 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.482 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.482 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.482 INFO:teuthology.orchestra.run.smithi070.stdout: "device_url": { 2024-04-01T01:50:07.482 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_url", 2024-04-01T01:50:07.482 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.482 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.482 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.483 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-01T01:50:07.483 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.483 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.483 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.483 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.483 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.483 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.483 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.483 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.483 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": { 2024-04-01T01:50:07.483 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enabled", 2024-04-01T01:50:07.483 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.483 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.484 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.484 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.484 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.484 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.484 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.484 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.484 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.484 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.484 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.484 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.484 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-04-01T01:50:07.484 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-04-01T01:50:07.484 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.485 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.485 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.485 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "24", 2024-04-01T01:50:07.485 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "8", 2024-04-01T01:50:07.485 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.485 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.485 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.485 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.485 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.485 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.485 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.485 INFO:teuthology.orchestra.run.smithi070.stdout: "last_opt_revision": { 2024-04-01T01:50:07.485 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "last_opt_revision", 2024-04-01T01:50:07.485 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.486 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.486 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.486 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1", 2024-04-01T01:50:07.486 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.486 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.486 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.486 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.486 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.486 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.486 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.486 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.486 INFO:teuthology.orchestra.run.smithi070.stdout: "leaderboard": { 2024-04-01T01:50:07.486 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "leaderboard", 2024-04-01T01:50:07.487 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.487 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.487 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.487 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.487 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.487 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.487 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.487 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.487 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.487 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.487 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.487 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.487 INFO:teuthology.orchestra.run.smithi070.stdout: "leaderboard_description": { 2024-04-01T01:50:07.487 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "leaderboard_description", 2024-04-01T01:50:07.488 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.488 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.488 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.488 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.488 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.488 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.488 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.488 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.488 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.488 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.488 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.488 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.488 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.489 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.489 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.489 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.489 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.489 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.489 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.489 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.489 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.489 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.489 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.489 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.489 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.489 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.489 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.489 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.490 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.490 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.490 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.490 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.490 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.490 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.490 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.490 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.490 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.490 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.490 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.490 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.490 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.491 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.491 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.491 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.491 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.491 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.491 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.491 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.491 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.491 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.491 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.491 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.491 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.491 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.491 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.492 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.492 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.492 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.492 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.492 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.492 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.492 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.492 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.492 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.492 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.492 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.492 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.492 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.492 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.493 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.493 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.493 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.493 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.493 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.493 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.493 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.493 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.493 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.493 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.493 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.493 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.493 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.493 INFO:teuthology.orchestra.run.smithi070.stdout: "organization": { 2024-04-01T01:50:07.493 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "organization", 2024-04-01T01:50:07.494 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.494 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.494 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.494 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.494 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.494 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.494 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.494 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.494 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.494 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.494 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.494 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.494 INFO:teuthology.orchestra.run.smithi070.stdout: "proxy": { 2024-04-01T01:50:07.495 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "proxy", 2024-04-01T01:50:07.495 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.495 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.495 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.495 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.495 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.495 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.495 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.495 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.495 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.495 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.495 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.495 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.495 INFO:teuthology.orchestra.run.smithi070.stdout: "url": { 2024-04-01T01:50:07.496 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "url", 2024-04-01T01:50:07.496 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.496 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.496 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.496 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-01T01:50:07.496 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.496 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.496 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.496 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.496 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.496 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.496 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.496 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.496 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.497 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.497 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.497 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "test_orchestrator", 2024-04-01T01:50:07.497 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.497 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.497 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.497 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.497 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.497 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.497 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.497 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.497 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.498 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.498 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.498 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.498 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.498 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.498 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.498 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.498 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.498 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.498 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.498 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.498 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.498 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.499 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.499 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.499 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.499 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.499 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.499 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.499 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.499 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.499 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.499 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.499 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.499 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.499 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.500 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.500 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.500 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.500 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.500 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.500 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.500 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.500 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.500 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.500 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.500 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.500 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.500 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.500 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.501 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.501 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.501 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.501 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.501 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.501 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.501 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.501 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.501 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.501 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.501 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.501 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.501 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.501 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.502 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.502 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.502 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.502 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.502 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.502 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.502 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.502 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.502 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.502 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.502 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.502 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.502 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.503 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "volumes", 2024-04-01T01:50:07.503 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.503 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.503 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.503 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.503 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.503 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.503 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.503 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.503 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.503 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.503 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.503 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.504 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.504 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.504 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.504 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.504 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.504 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.504 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.504 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.504 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.504 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.504 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.504 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.504 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.504 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.505 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.505 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.505 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.505 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.505 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.505 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.505 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.505 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.505 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.505 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.505 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.505 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.505 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.505 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.506 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.506 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.506 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.506 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.506 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.506 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.506 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.506 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.506 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.506 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.506 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.506 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.506 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.506 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.507 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: "max_concurrent_clones": { 2024-04-01T01:50:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_concurrent_clones", 2024-04-01T01:50:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "4", 2024-04-01T01:50:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.508 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.509 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-01T01:50:07.509 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.509 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.509 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.509 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.509 INFO:teuthology.orchestra.run.smithi070.stdout: "periodic_async_work": { 2024-04-01T01:50:07.509 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "periodic_async_work", 2024-04-01T01:50:07.509 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.509 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.509 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.509 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.509 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.509 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.509 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.510 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Periodically check for async work", 2024-04-01T01:50:07.510 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.510 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.510 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.510 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.510 INFO:teuthology.orchestra.run.smithi070.stdout: "snapshot_clone_delay": { 2024-04-01T01:50:07.510 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snapshot_clone_delay", 2024-04-01T01:50:07.510 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.510 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.510 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.510 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-04-01T01:50:07.510 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.510 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.511 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.511 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-01T01:50:07.511 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.511 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.511 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.511 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.511 INFO:teuthology.orchestra.run.smithi070.stdout: "snapshot_clone_no_wait": { 2024-04-01T01:50:07.511 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snapshot_clone_no_wait", 2024-04-01T01:50:07.511 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.511 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.511 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.511 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.511 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.511 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.512 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.512 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-01T01:50:07.512 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.512 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.512 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.512 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.512 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.512 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.512 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.512 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix", 2024-04-01T01:50:07.512 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.512 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.512 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.512 INFO:teuthology.orchestra.run.smithi070.stdout: "discovery_interval": { 2024-04-01T01:50:07.513 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "discovery_interval", 2024-04-01T01:50:07.513 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-04-01T01:50:07.513 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.513 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.513 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "100", 2024-04-01T01:50:07.513 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.513 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.513 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.513 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.513 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.513 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.513 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.513 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.513 INFO:teuthology.orchestra.run.smithi070.stdout: "identifier": { 2024-04-01T01:50:07.513 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "identifier", 2024-04-01T01:50:07.514 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.514 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.514 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.514 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.514 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.514 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.514 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.514 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.514 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.514 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.514 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.514 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.514 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-04-01T01:50:07.515 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-04-01T01:50:07.515 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.515 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.515 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.515 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-04-01T01:50:07.515 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.515 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.515 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.515 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.515 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.515 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.515 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.515 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.515 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.516 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.516 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.516 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.516 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.516 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.516 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.516 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.516 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.516 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.516 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.516 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.516 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.516 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.516 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.517 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.517 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.517 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.517 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.517 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.517 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.517 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.517 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.517 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.517 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.517 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.517 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.517 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.517 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.517 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.518 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.518 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.518 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.518 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.518 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.518 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.518 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.518 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.518 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.518 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.518 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.518 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.518 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.519 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.519 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.519 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.519 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.519 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.519 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.519 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.519 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.519 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.519 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.519 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.519 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.519 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.519 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.520 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.520 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.520 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.520 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.520 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.520 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.520 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.520 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.520 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.520 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.520 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.520 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.520 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.520 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_host": { 2024-04-01T01:50:07.521 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_host", 2024-04-01T01:50:07.521 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.521 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.521 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.521 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.521 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.521 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.521 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.521 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.521 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.521 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.521 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.521 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.521 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_port": { 2024-04-01T01:50:07.521 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_port", 2024-04-01T01:50:07.522 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.522 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.522 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.522 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10051", 2024-04-01T01:50:07.522 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.522 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.522 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.522 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.522 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.522 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.522 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.522 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.522 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_sender": { 2024-04-01T01:50:07.522 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_sender", 2024-04-01T01:50:07.523 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.523 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.523 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.523 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-01T01:50:07.523 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.523 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.523 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.523 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.523 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.523 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.523 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.523 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.523 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.523 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.524 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-01T01:50:07.524 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.524 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.524 INFO:teuthology.orchestra.run.smithi070.stdout: "gid": 24107, 2024-04-01T01:50:07.524 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "y", 2024-04-01T01:50:07.524 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr_features": 4540701547738038271, 2024-04-01T01:50:07.524 INFO:teuthology.orchestra.run.smithi070.stdout: "available_modules": [ 2024-04-01T01:50:07.524 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.524 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "alerts", 2024-04-01T01:50:07.524 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.524 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.524 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.524 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-04-01T01:50:07.524 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-04-01T01:50:07.525 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.525 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.525 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.525 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-04-01T01:50:07.525 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.525 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.525 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.525 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How frequently to reexamine health status", 2024-04-01T01:50:07.525 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.525 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.525 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.525 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.525 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.526 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.526 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.526 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.526 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.526 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.526 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.526 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.526 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.526 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.526 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.526 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.526 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.526 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.527 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.527 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.527 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.527 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.527 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.527 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.527 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.527 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.527 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.527 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.527 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.527 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.527 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.527 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.528 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.528 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.528 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.528 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.528 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.528 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.528 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.528 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.528 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.528 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.528 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.528 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.528 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.529 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.529 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.529 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.529 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.529 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.529 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.529 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.529 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.529 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.529 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.529 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.529 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.529 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.530 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.530 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.530 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.530 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.530 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.530 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.530 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.530 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.530 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.530 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.530 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.530 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.530 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.530 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.531 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.531 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.531 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_destination": { 2024-04-01T01:50:07.531 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_destination", 2024-04-01T01:50:07.531 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.531 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.531 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.531 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.531 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.531 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.531 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.531 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Email address to send alerts to", 2024-04-01T01:50:07.531 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.531 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.532 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.532 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.532 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_from_name": { 2024-04-01T01:50:07.532 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_from_name", 2024-04-01T01:50:07.532 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.532 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.532 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.532 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "Ceph", 2024-04-01T01:50:07.532 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.532 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.532 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.532 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Email From: name", 2024-04-01T01:50:07.532 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.532 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.533 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.533 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.533 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_host": { 2024-04-01T01:50:07.533 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_host", 2024-04-01T01:50:07.533 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.533 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.533 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.533 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.533 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.533 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.533 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.533 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP server", 2024-04-01T01:50:07.533 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.533 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.533 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.534 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.534 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_password": { 2024-04-01T01:50:07.534 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_password", 2024-04-01T01:50:07.534 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.534 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.534 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.534 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.534 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.534 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.534 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.534 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Password to authenticate with", 2024-04-01T01:50:07.534 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.534 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.535 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.535 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.535 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_port": { 2024-04-01T01:50:07.535 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_port", 2024-04-01T01:50:07.535 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.535 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.535 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.535 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "465", 2024-04-01T01:50:07.535 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.535 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.535 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.535 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP port", 2024-04-01T01:50:07.535 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.535 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.536 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.536 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.536 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_sender": { 2024-04-01T01:50:07.536 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_sender", 2024-04-01T01:50:07.536 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.536 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.536 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.536 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.536 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.536 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.536 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.536 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP envelope sender", 2024-04-01T01:50:07.536 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.536 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.537 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.537 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.537 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_ssl": { 2024-04-01T01:50:07.537 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_ssl", 2024-04-01T01:50:07.537 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.537 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.537 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.537 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.537 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.537 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.537 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.537 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-01T01:50:07.537 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.537 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.537 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.538 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.538 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_user": { 2024-04-01T01:50:07.538 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_user", 2024-04-01T01:50:07.538 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.538 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.538 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.538 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.538 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.538 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.538 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.538 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "User to authenticate as", 2024-04-01T01:50:07.538 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.538 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.539 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.539 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.539 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.539 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.539 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.539 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "balancer", 2024-04-01T01:50:07.539 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.539 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.539 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.539 INFO:teuthology.orchestra.run.smithi070.stdout: "active": { 2024-04-01T01:50:07.539 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "active", 2024-04-01T01:50:07.539 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.539 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.540 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.540 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.540 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.540 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.540 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.540 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "automatically balance PGs across cluster", 2024-04-01T01:50:07.540 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.540 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.540 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.540 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.540 INFO:teuthology.orchestra.run.smithi070.stdout: "begin_time": { 2024-04-01T01:50:07.540 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "begin_time", 2024-04-01T01:50:07.540 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.540 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.541 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.541 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0000", 2024-04-01T01:50:07.541 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.541 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.541 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.541 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "beginning time of day to automatically balance", 2024-04-01T01:50:07.541 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-01T01:50:07.541 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.541 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.541 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.541 INFO:teuthology.orchestra.run.smithi070.stdout: "begin_weekday": { 2024-04-01T01:50:07.541 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "begin_weekday", 2024-04-01T01:50:07.541 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-04-01T01:50:07.542 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.542 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.542 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-04-01T01:50:07.542 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0", 2024-04-01T01:50:07.542 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "6", 2024-04-01T01:50:07.542 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.542 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-01T01:50:07.542 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-01T01:50:07.542 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.542 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.542 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.542 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_max_iterations": { 2024-04-01T01:50:07.542 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_max_iterations", 2024-04-01T01:50:07.543 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-04-01T01:50:07.543 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.543 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.543 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "25", 2024-04-01T01:50:07.543 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-04-01T01:50:07.543 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "250", 2024-04-01T01:50:07.543 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.543 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-01T01:50:07.543 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.543 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.543 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.543 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.543 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_metrics": { 2024-04-01T01:50:07.543 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_metrics", 2024-04-01T01:50:07.543 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.544 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.544 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.544 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "pgs,objects,bytes", 2024-04-01T01:50:07.544 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.544 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.544 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.544 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-01T01:50:07.544 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-01T01:50:07.544 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.544 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.544 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.544 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_step": { 2024-04-01T01:50:07.544 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_step", 2024-04-01T01:50:07.544 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.544 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.545 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.545 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0.5", 2024-04-01T01:50:07.545 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0.001", 2024-04-01T01:50:07.545 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "0.999", 2024-04-01T01:50:07.545 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.545 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "aggressiveness of optimization", 2024-04-01T01:50:07.545 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-01T01:50:07.545 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.545 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.545 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.545 INFO:teuthology.orchestra.run.smithi070.stdout: "end_time": { 2024-04-01T01:50:07.545 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "end_time", 2024-04-01T01:50:07.545 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.545 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.545 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.546 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "2359", 2024-04-01T01:50:07.546 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.546 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.546 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.546 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "ending time of day to automatically balance", 2024-04-01T01:50:07.546 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-01T01:50:07.546 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.546 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.546 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.546 INFO:teuthology.orchestra.run.smithi070.stdout: "end_weekday": { 2024-04-01T01:50:07.546 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "end_weekday", 2024-04-01T01:50:07.546 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-04-01T01:50:07.546 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.547 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.547 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-04-01T01:50:07.547 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0", 2024-04-01T01:50:07.547 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "6", 2024-04-01T01:50:07.547 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.547 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-01T01:50:07.547 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-01T01:50:07.547 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.547 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.547 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.547 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.547 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.547 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.547 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.547 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.547 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.548 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.548 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.548 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.548 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.548 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.548 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.548 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.548 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.548 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.548 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.548 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.548 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.548 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.549 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.549 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.549 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.549 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.549 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.549 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.549 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.549 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.549 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.549 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.549 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.549 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.549 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.549 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.550 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.550 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.550 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.550 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.550 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.550 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.550 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.550 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.550 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.550 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.550 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.550 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.550 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.550 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.550 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.551 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.551 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.551 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.551 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.551 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.551 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.551 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.551 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.551 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.551 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.551 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.551 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.551 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.552 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.552 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.552 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.552 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.552 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.552 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.552 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.552 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.552 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.552 INFO:teuthology.orchestra.run.smithi070.stdout: "min_score": { 2024-04-01T01:50:07.552 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "min_score", 2024-04-01T01:50:07.552 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.552 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.552 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.552 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-04-01T01:50:07.552 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.553 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.553 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.553 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-01T01:50:07.553 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.553 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.553 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.553 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.553 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": { 2024-04-01T01:50:07.553 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mode", 2024-04-01T01:50:07.553 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.553 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.553 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.553 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "upmap", 2024-04-01T01:50:07.553 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.553 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.554 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.554 INFO:teuthology.orchestra.run.smithi070.stdout: "crush-compat", 2024-04-01T01:50:07.554 INFO:teuthology.orchestra.run.smithi070.stdout: "none", 2024-04-01T01:50:07.554 INFO:teuthology.orchestra.run.smithi070.stdout: "read", 2024-04-01T01:50:07.554 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap", 2024-04-01T01:50:07.554 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap-read" 2024-04-01T01:50:07.554 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.554 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Balancer mode", 2024-04-01T01:50:07.554 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.554 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.554 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.554 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.554 INFO:teuthology.orchestra.run.smithi070.stdout: "pool_ids": { 2024-04-01T01:50:07.554 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pool_ids", 2024-04-01T01:50:07.555 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.555 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.555 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.555 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.555 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.555 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.555 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.555 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-01T01:50:07.555 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.555 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.555 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.555 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.555 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-04-01T01:50:07.555 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-04-01T01:50:07.555 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.556 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.556 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.556 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-04-01T01:50:07.556 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.556 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.556 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.556 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-01T01:50:07.556 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.556 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.556 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.556 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.556 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap_max_deviation": { 2024-04-01T01:50:07.556 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "upmap_max_deviation", 2024-04-01T01:50:07.556 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.557 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.557 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.557 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-04-01T01:50:07.557 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-04-01T01:50:07.557 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.557 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.557 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-01T01:50:07.557 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-01T01:50:07.557 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.557 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.557 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.557 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap_max_optimizations": { 2024-04-01T01:50:07.557 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "upmap_max_optimizations", 2024-04-01T01:50:07.557 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-04-01T01:50:07.558 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.558 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.558 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.558 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.558 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.558 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.558 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-01T01:50:07.558 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.558 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.558 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.558 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.558 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.558 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.558 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.559 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cephadm", 2024-04-01T01:50:07.559 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.559 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.559 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.559 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_down_multiplier": { 2024-04-01T01:50:07.559 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_down_multiplier", 2024-04-01T01:50:07.559 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.559 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.559 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.559 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3.0", 2024-04-01T01:50:07.559 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.559 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.559 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.559 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-01T01:50:07.560 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.560 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.560 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.560 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.560 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_refresh_rate": { 2024-04-01T01:50:07.560 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_refresh_rate", 2024-04-01T01:50:07.560 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.560 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.560 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.560 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "20", 2024-04-01T01:50:07.560 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.560 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.560 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.560 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-01T01:50:07.561 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.561 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.561 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.561 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.561 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_starting_port": { 2024-04-01T01:50:07.561 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_starting_port", 2024-04-01T01:50:07.561 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.561 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.561 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.561 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "4721", 2024-04-01T01:50:07.561 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.561 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.561 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.562 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-01T01:50:07.562 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.562 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.562 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.562 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.562 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_ptrace": { 2024-04-01T01:50:07.562 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_ptrace", 2024-04-01T01:50:07.562 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.562 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.562 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.562 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.562 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.562 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.563 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.563 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-01T01:50:07.563 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-01T01:50:07.563 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.563 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.563 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.563 INFO:teuthology.orchestra.run.smithi070.stdout: "autotune_interval": { 2024-04-01T01:50:07.563 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "autotune_interval", 2024-04-01T01:50:07.563 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.563 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.563 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.563 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-04-01T01:50:07.563 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.563 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-01T01:50:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "autotune_memory_target_ratio": { 2024-04-01T01:50:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "autotune_memory_target_ratio", 2024-04-01T01:50:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0.7", 2024-04-01T01:50:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.564 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-01T01:50:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm_log_destination": { 2024-04-01T01:50:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cephadm_log_destination", 2024-04-01T01:50:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.565 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: "file", 2024-04-01T01:50:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: "file,syslog", 2024-04-01T01:50:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: "syslog" 2024-04-01T01:50:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-01T01:50:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: "cgroups_split": { 2024-04-01T01:50:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cgroups_split", 2024-04-01T01:50:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.567 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.567 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.567 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.567 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.567 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.567 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-01T01:50:07.567 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.567 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.567 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.567 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.567 INFO:teuthology.orchestra.run.smithi070.stdout: "config_checks_enabled": { 2024-04-01T01:50:07.567 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_checks_enabled", 2024-04-01T01:50:07.568 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.568 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.568 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.568 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.568 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.568 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.568 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.568 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-01T01:50:07.569 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.569 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.569 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.569 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.569 INFO:teuthology.orchestra.run.smithi070.stdout: "config_dashboard": { 2024-04-01T01:50:07.570 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_dashboard", 2024-04-01T01:50:07.570 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.570 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.570 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.570 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.570 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.570 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.570 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.570 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-01T01:50:07.570 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.570 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.570 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.570 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.571 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_alertmanager": { 2024-04-01T01:50:07.571 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_alertmanager", 2024-04-01T01:50:07.571 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.571 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.571 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.571 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-01T01:50:07.571 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.571 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.571 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.571 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-04-01T01:50:07.571 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.571 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.571 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.571 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.572 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_base": { 2024-04-01T01:50:07.572 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_base", 2024-04-01T01:50:07.572 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.572 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.572 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.572 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-01T01:50:07.572 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.572 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.572 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.572 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Container image name, without the tag", 2024-04-01T01:50:07.572 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.572 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.572 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.572 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.573 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_elasticsearch": { 2024-04-01T01:50:07.573 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_elasticsearch", 2024-04-01T01:50:07.573 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.573 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.573 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.573 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-01T01:50:07.573 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.573 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.573 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.573 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "elasticsearch container image", 2024-04-01T01:50:07.573 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.573 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.573 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.573 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.574 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_grafana": { 2024-04-01T01:50:07.574 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_grafana", 2024-04-01T01:50:07.574 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.574 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.574 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.574 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-01T01:50:07.574 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.574 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.574 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.574 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-04-01T01:50:07.574 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.574 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.574 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.575 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.575 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_haproxy": { 2024-04-01T01:50:07.575 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_haproxy", 2024-04-01T01:50:07.575 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.575 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.575 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.575 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-01T01:50:07.575 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.575 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.575 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.575 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "HAproxy container image", 2024-04-01T01:50:07.575 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.575 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.575 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.575 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.576 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_agent": { 2024-04-01T01:50:07.576 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_agent", 2024-04-01T01:50:07.576 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.576 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.576 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.576 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-01T01:50:07.576 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.576 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.576 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.576 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger agent container image", 2024-04-01T01:50:07.576 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.576 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.576 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.576 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.576 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_collector": { 2024-04-01T01:50:07.577 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_collector", 2024-04-01T01:50:07.577 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.577 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.577 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.577 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-01T01:50:07.577 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.577 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.577 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.577 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger collector container image", 2024-04-01T01:50:07.577 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.577 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.577 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.578 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_query": { 2024-04-01T01:50:07.578 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_query", 2024-04-01T01:50:07.578 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.578 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.578 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.578 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-01T01:50:07.578 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.578 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.578 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.578 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger query container image", 2024-04-01T01:50:07.578 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.578 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.579 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.579 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_keepalived": { 2024-04-01T01:50:07.579 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_keepalived", 2024-04-01T01:50:07.579 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.579 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.579 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.579 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-01T01:50:07.579 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.579 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.579 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.579 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Keepalived container image", 2024-04-01T01:50:07.579 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.579 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.580 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.580 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_loki": { 2024-04-01T01:50:07.580 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_loki", 2024-04-01T01:50:07.580 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.580 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.580 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.580 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-01T01:50:07.580 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.580 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.580 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.580 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Loki container image", 2024-04-01T01:50:07.580 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.580 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.581 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.581 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_node_exporter": { 2024-04-01T01:50:07.581 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_node_exporter", 2024-04-01T01:50:07.581 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.581 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.581 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.581 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-01T01:50:07.581 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.581 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.581 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.581 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-04-01T01:50:07.581 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.582 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.582 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.582 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.582 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_nvmeof": { 2024-04-01T01:50:07.582 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_nvmeof", 2024-04-01T01:50:07.582 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.582 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.582 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.582 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-04-01T01:50:07.582 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.582 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.582 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.582 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Nvme-of container image", 2024-04-01T01:50:07.582 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.583 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.583 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.583 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_prometheus": { 2024-04-01T01:50:07.583 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_prometheus", 2024-04-01T01:50:07.583 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.583 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.583 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.583 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-01T01:50:07.583 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.583 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.583 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.583 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-04-01T01:50:07.583 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.584 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.584 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.584 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_promtail": { 2024-04-01T01:50:07.584 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_promtail", 2024-04-01T01:50:07.584 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.584 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.584 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.584 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-01T01:50:07.584 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.584 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.584 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.584 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Promtail container image", 2024-04-01T01:50:07.584 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.585 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.585 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.585 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_snmp_gateway": { 2024-04-01T01:50:07.585 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_snmp_gateway", 2024-04-01T01:50:07.585 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.585 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.585 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.585 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-01T01:50:07.585 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.585 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.585 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.585 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SNMP Gateway container image", 2024-04-01T01:50:07.585 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.585 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.586 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.586 INFO:teuthology.orchestra.run.smithi070.stdout: "container_init": { 2024-04-01T01:50:07.586 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_init", 2024-04-01T01:50:07.586 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.586 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.586 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.586 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.586 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.586 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.586 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.586 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Run podman/docker with `--init`", 2024-04-01T01:50:07.586 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.586 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.587 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.587 INFO:teuthology.orchestra.run.smithi070.stdout: "daemon_cache_timeout": { 2024-04-01T01:50:07.587 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "daemon_cache_timeout", 2024-04-01T01:50:07.587 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.587 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.587 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.587 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-04-01T01:50:07.587 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.587 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.587 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.587 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-01T01:50:07.587 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.587 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.588 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.588 INFO:teuthology.orchestra.run.smithi070.stdout: "default_cephadm_command_timeout": { 2024-04-01T01:50:07.588 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "default_cephadm_command_timeout", 2024-04-01T01:50:07.588 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.588 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.588 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.588 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "900", 2024-04-01T01:50:07.588 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.588 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.588 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.588 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-01T01:50:07.588 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.588 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.589 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.589 INFO:teuthology.orchestra.run.smithi070.stdout: "default_registry": { 2024-04-01T01:50:07.589 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "default_registry", 2024-04-01T01:50:07.589 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.589 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.589 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.589 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io", 2024-04-01T01:50:07.589 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.589 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.589 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.589 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-01T01:50:07.589 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.590 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.590 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.590 INFO:teuthology.orchestra.run.smithi070.stdout: "device_cache_timeout": { 2024-04-01T01:50:07.590 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_cache_timeout", 2024-04-01T01:50:07.590 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.590 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.590 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.590 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1800", 2024-04-01T01:50:07.590 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.590 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.590 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.590 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache device inventory", 2024-04-01T01:50:07.591 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.591 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.591 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.591 INFO:teuthology.orchestra.run.smithi070.stdout: "device_enhanced_scan": { 2024-04-01T01:50:07.591 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_enhanced_scan", 2024-04-01T01:50:07.591 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.591 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.591 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.591 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.591 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.591 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.591 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.591 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-01T01:50:07.592 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.592 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.592 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.592 INFO:teuthology.orchestra.run.smithi070.stdout: "facts_cache_timeout": { 2024-04-01T01:50:07.592 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "facts_cache_timeout", 2024-04-01T01:50:07.592 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.592 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.592 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.592 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-04-01T01:50:07.592 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.592 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.592 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.593 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache host facts data", 2024-04-01T01:50:07.593 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.593 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.593 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.593 INFO:teuthology.orchestra.run.smithi070.stdout: "grafana_dashboards_path": { 2024-04-01T01:50:07.593 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "grafana_dashboards_path", 2024-04-01T01:50:07.593 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.593 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.593 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.593 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-01T01:50:07.593 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.593 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.593 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.594 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-01T01:50:07.594 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.594 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.594 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.594 INFO:teuthology.orchestra.run.smithi070.stdout: "host_check_interval": { 2024-04-01T01:50:07.594 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "host_check_interval", 2024-04-01T01:50:07.594 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.594 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.594 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.594 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-04-01T01:50:07.594 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.594 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.594 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.595 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to perform a host check", 2024-04-01T01:50:07.595 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.595 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.595 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.595 INFO:teuthology.orchestra.run.smithi070.stdout: "hw_monitoring": { 2024-04-01T01:50:07.595 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "hw_monitoring", 2024-04-01T01:50:07.595 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.595 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.595 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.595 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.595 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.595 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.595 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.595 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-01T01:50:07.596 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.596 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.596 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.596 INFO:teuthology.orchestra.run.smithi070.stdout: "inventory_list_all": { 2024-04-01T01:50:07.596 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "inventory_list_all", 2024-04-01T01:50:07.596 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.596 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.596 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.596 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.596 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.596 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.596 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.597 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-01T01:50:07.597 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.597 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.597 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.597 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.597 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.597 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.597 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.597 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.597 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.597 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.597 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.597 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.598 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.598 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.598 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.598 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.598 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.598 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.598 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.598 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.598 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.598 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.598 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.598 INFO:teuthology.orchestra.run.smithi070.stdout: "log_refresh_metadata": { 2024-04-01T01:50:07.598 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_refresh_metadata", 2024-04-01T01:50:07.599 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.599 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.599 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.599 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.599 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.599 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.599 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.599 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-01T01:50:07.599 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.599 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.599 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.599 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.600 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.600 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.600 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.600 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.600 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.600 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.600 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.600 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.600 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-01T01:50:07.600 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.600 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.600 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.600 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.600 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.601 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.601 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.601 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.601 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.601 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.601 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.601 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.601 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.601 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.601 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.601 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.601 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.601 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.601 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.601 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.602 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.602 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.602 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.602 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.602 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.602 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.602 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.602 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.602 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.602 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.602 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.603 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.603 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.603 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.603 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.603 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.603 INFO:teuthology.orchestra.run.smithi070.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-01T01:50:07.603 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-01T01:50:07.603 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.603 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.603 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.603 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.603 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.603 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.604 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.604 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-01T01:50:07.604 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.604 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.604 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.604 INFO:teuthology.orchestra.run.smithi070.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-01T01:50:07.604 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-01T01:50:07.604 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.604 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.604 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.604 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "*", 2024-04-01T01:50:07.604 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.605 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.605 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.605 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-01T01:50:07.605 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.605 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.605 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.605 INFO:teuthology.orchestra.run.smithi070.stdout: "max_count_per_host": { 2024-04-01T01:50:07.605 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_count_per_host", 2024-04-01T01:50:07.605 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.605 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.605 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.605 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.606 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.606 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.606 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.606 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "max number of daemons per service per host", 2024-04-01T01:50:07.606 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.606 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.606 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.606 INFO:teuthology.orchestra.run.smithi070.stdout: "max_osd_draining_count": { 2024-04-01T01:50:07.606 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_osd_draining_count", 2024-04-01T01:50:07.606 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.606 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.606 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.607 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.607 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.607 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.607 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.607 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-01T01:50:07.607 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.607 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.607 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.607 INFO:teuthology.orchestra.run.smithi070.stdout: "migration_current": { 2024-04-01T01:50:07.607 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "migration_current", 2024-04-01T01:50:07.607 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.607 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.607 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.608 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.608 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.608 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.608 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.608 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "internal - do not modify", 2024-04-01T01:50:07.608 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.608 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.608 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.608 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": { 2024-04-01T01:50:07.608 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mode", 2024-04-01T01:50:07.608 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.608 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.608 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.609 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "root", 2024-04-01T01:50:07.609 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.609 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.609 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.609 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm-package", 2024-04-01T01:50:07.609 INFO:teuthology.orchestra.run.smithi070.stdout: "root" 2024-04-01T01:50:07.609 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.609 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "mode for remote execution of cephadm", 2024-04-01T01:50:07.609 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.609 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.609 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.609 INFO:teuthology.orchestra.run.smithi070.stdout: "oob_default_addr": { 2024-04-01T01:50:07.609 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "oob_default_addr", 2024-04-01T01:50:07.610 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.610 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.610 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.610 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "169.254.1.1", 2024-04-01T01:50:07.610 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.610 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.610 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.610 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-01T01:50:07.610 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.610 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.610 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.610 INFO:teuthology.orchestra.run.smithi070.stdout: "prometheus_alerts_path": { 2024-04-01T01:50:07.610 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prometheus_alerts_path", 2024-04-01T01:50:07.611 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.611 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.611 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.611 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-01T01:50:07.611 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.611 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.611 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.611 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-01T01:50:07.611 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.611 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.611 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.611 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_insecure": { 2024-04-01T01:50:07.612 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_insecure", 2024-04-01T01:50:07.612 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.612 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.612 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.612 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.612 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.612 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.612 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.612 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-01T01:50:07.612 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.612 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.612 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.612 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_password": { 2024-04-01T01:50:07.613 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_password", 2024-04-01T01:50:07.613 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.613 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.613 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.613 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.613 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.613 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.613 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.613 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-01T01:50:07.613 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.613 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.613 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.614 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_url": { 2024-04-01T01:50:07.614 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_url", 2024-04-01T01:50:07.614 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.614 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.614 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.614 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.614 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.614 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.614 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.614 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-01T01:50:07.614 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.614 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.614 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.615 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_username": { 2024-04-01T01:50:07.615 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_username", 2024-04-01T01:50:07.615 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.615 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.615 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.615 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.615 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.615 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.615 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.615 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-01T01:50:07.615 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.615 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.616 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.616 INFO:teuthology.orchestra.run.smithi070.stdout: "secure_monitoring_stack": { 2024-04-01T01:50:07.616 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "secure_monitoring_stack", 2024-04-01T01:50:07.616 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.616 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.616 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.616 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.616 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.616 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.616 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.616 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-01T01:50:07.616 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.616 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.616 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.617 INFO:teuthology.orchestra.run.smithi070.stdout: "service_discovery_port": { 2024-04-01T01:50:07.617 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "service_discovery_port", 2024-04-01T01:50:07.617 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.617 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.617 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.617 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8765", 2024-04-01T01:50:07.617 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.617 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.617 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.617 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "cephadm service discovery port", 2024-04-01T01:50:07.617 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.617 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.617 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.618 INFO:teuthology.orchestra.run.smithi070.stdout: "ssh_config_file": { 2024-04-01T01:50:07.618 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssh_config_file", 2024-04-01T01:50:07.618 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.618 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.618 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.618 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.618 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.618 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.618 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.618 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-01T01:50:07.618 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.618 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.618 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.619 INFO:teuthology.orchestra.run.smithi070.stdout: "use_agent": { 2024-04-01T01:50:07.619 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "use_agent", 2024-04-01T01:50:07.619 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.619 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.619 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.619 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.619 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.619 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.619 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.619 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-01T01:50:07.619 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.619 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.619 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.620 INFO:teuthology.orchestra.run.smithi070.stdout: "use_repo_digest": { 2024-04-01T01:50:07.620 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "use_repo_digest", 2024-04-01T01:50:07.620 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.620 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.620 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.620 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.620 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.620 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.620 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.620 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-01T01:50:07.620 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.620 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.620 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.620 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_failed_host_check": { 2024-04-01T01:50:07.620 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_failed_host_check", 2024-04-01T01:50:07.621 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.621 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.621 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.621 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.621 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.621 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.621 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.621 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if the host check fails", 2024-04-01T01:50:07.621 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.621 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.621 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.621 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_stray_daemons": { 2024-04-01T01:50:07.621 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_stray_daemons", 2024-04-01T01:50:07.622 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.622 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.622 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.622 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.622 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.622 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.622 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.622 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-01T01:50:07.622 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.622 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.622 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.622 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_stray_hosts": { 2024-04-01T01:50:07.622 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_stray_hosts", 2024-04-01T01:50:07.623 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.623 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.623 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.623 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.623 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.623 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.623 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.623 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-01T01:50:07.623 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.623 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.623 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.623 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.623 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.623 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crash", 2024-04-01T01:50:07.624 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.624 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.624 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.624 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.624 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.624 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.624 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.624 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.624 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.624 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.624 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.624 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.624 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.625 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.625 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.625 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.625 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.625 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.625 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.625 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.625 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.625 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.625 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.625 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.625 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.625 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.625 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.626 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.626 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.626 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.626 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.626 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.626 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.626 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.626 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.626 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.626 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.626 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.626 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.627 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.627 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.627 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.627 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.627 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.627 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.627 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.627 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.627 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.627 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.627 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.627 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.627 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.627 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.628 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.628 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.628 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.628 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.628 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.628 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.628 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.628 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.628 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.628 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.628 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.628 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.628 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.628 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.629 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.629 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.629 INFO:teuthology.orchestra.run.smithi070.stdout: "retain_interval": { 2024-04-01T01:50:07.629 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "retain_interval", 2024-04-01T01:50:07.629 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.629 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.629 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.629 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "31536000", 2024-04-01T01:50:07.629 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.629 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.629 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.629 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-01T01:50:07.630 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.630 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.630 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.630 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_recent_interval": { 2024-04-01T01:50:07.630 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_recent_interval", 2024-04-01T01:50:07.630 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.630 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.630 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.630 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1209600", 2024-04-01T01:50:07.630 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.630 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.630 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.630 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-01T01:50:07.631 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.631 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.631 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.631 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.631 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.631 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "dashboard", 2024-04-01T01:50:07.631 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.631 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.631 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.631 INFO:teuthology.orchestra.run.smithi070.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-01T01:50:07.631 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-01T01:50:07.631 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.631 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.632 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.632 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.632 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.632 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.632 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.632 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.632 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.632 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.632 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.632 INFO:teuthology.orchestra.run.smithi070.stdout: "ALERTMANAGER_API_HOST": { 2024-04-01T01:50:07.632 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-01T01:50:07.632 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.632 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.632 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.632 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.633 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.633 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.633 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.633 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.633 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.633 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.633 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.633 INFO:teuthology.orchestra.run.smithi070.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-01T01:50:07.633 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-01T01:50:07.633 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.633 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.633 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.633 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.634 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.634 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.634 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.634 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.634 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.634 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.634 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.634 INFO:teuthology.orchestra.run.smithi070.stdout: "AUDIT_API_ENABLED": { 2024-04-01T01:50:07.634 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "AUDIT_API_ENABLED", 2024-04-01T01:50:07.634 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.634 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.634 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.635 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.635 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.635 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.635 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.635 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.635 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.635 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.635 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.635 INFO:teuthology.orchestra.run.smithi070.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-01T01:50:07.635 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-01T01:50:07.635 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.635 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.635 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.636 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.636 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.636 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.636 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.636 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.636 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.636 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.636 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.636 INFO:teuthology.orchestra.run.smithi070.stdout: "ENABLE_BROWSABLE_API": { 2024-04-01T01:50:07.636 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-01T01:50:07.636 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.636 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.636 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.637 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.637 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.637 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.637 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.637 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.637 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.637 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.637 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.637 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-01T01:50:07.637 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-01T01:50:07.637 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.638 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.638 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.638 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.638 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.638 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.638 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.638 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.638 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.638 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.638 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.638 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-01T01:50:07.638 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-01T01:50:07.638 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.639 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.639 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.639 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.639 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.639 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.639 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.639 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.639 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.639 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.639 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.639 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-01T01:50:07.639 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-01T01:50:07.639 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.640 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.640 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.640 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.640 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.640 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.640 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.640 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.640 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.640 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.640 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.640 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-01T01:50:07.640 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-01T01:50:07.640 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.641 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.641 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.641 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.641 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.641 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.641 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.641 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.641 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.641 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.641 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.641 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-01T01:50:07.641 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-01T01:50:07.641 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.642 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.642 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.642 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.642 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.642 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.642 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.642 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.642 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.642 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.642 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.642 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-01T01:50:07.642 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-01T01:50:07.643 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.643 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.643 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.643 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.643 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.643 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.643 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.643 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.643 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.643 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.643 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.643 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-01T01:50:07.643 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-01T01:50:07.644 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.644 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.644 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.644 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.644 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.644 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.644 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.644 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.644 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.644 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.644 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.644 INFO:teuthology.orchestra.run.smithi070.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-01T01:50:07.644 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-01T01:50:07.645 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.645 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.645 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.645 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.645 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.645 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.645 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.645 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.645 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.645 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.645 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.645 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_PASSWORD": { 2024-04-01T01:50:07.645 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-01T01:50:07.646 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.646 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.646 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.646 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-04-01T01:50:07.646 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.646 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.646 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.646 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.646 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.646 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.646 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.646 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-01T01:50:07.646 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-01T01:50:07.647 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.647 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.647 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.647 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.647 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.647 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.647 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.647 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.647 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.647 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.647 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.647 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_URL": { 2024-04-01T01:50:07.648 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_URL", 2024-04-01T01:50:07.648 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.648 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.648 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.648 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.648 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.648 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.648 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.648 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.648 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.648 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.648 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.648 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_USERNAME": { 2024-04-01T01:50:07.649 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-01T01:50:07.649 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.649 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.649 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.649 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-04-01T01:50:07.649 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.649 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.649 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.649 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.649 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.649 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.649 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.649 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-01T01:50:07.649 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-01T01:50:07.650 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.650 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.650 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.650 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.650 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.650 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.650 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.650 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.650 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.650 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.650 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.650 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-01T01:50:07.651 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-01T01:50:07.651 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.651 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.651 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.651 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.651 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.651 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.651 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.651 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.651 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.651 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.651 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.651 INFO:teuthology.orchestra.run.smithi070.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-01T01:50:07.651 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-01T01:50:07.652 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.652 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.652 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.652 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.652 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.652 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.652 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.652 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.652 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.652 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.652 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.652 INFO:teuthology.orchestra.run.smithi070.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-01T01:50:07.652 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-01T01:50:07.653 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.653 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.653 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.653 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.653 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.653 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.653 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.653 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.653 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.653 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.653 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.653 INFO:teuthology.orchestra.run.smithi070.stdout: "PROMETHEUS_API_HOST": { 2024-04-01T01:50:07.653 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-01T01:50:07.654 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.654 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.654 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.654 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.654 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.654 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.654 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.654 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.654 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.654 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.654 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.654 INFO:teuthology.orchestra.run.smithi070.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-01T01:50:07.655 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-01T01:50:07.655 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.655 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.655 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.655 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.655 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.655 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.655 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.655 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.655 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.655 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.655 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.655 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-01T01:50:07.656 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-01T01:50:07.656 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.656 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.656 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.656 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.656 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.656 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.656 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.656 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.656 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.656 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.656 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.656 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-01T01:50:07.657 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-01T01:50:07.657 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.657 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.657 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.657 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.657 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.657 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.657 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.657 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.657 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.657 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.657 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.657 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-01T01:50:07.658 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-01T01:50:07.658 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.658 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.658 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.658 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.658 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.658 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.658 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.658 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.658 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.658 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.658 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.658 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-01T01:50:07.659 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-01T01:50:07.659 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.659 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.659 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.659 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.659 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.659 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.659 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.659 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.659 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.659 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.659 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.659 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-01T01:50:07.660 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-01T01:50:07.660 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.660 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.660 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.660 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.660 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.660 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.660 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.660 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.660 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.660 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.660 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.660 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-01T01:50:07.661 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-01T01:50:07.661 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.661 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.661 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.661 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.661 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.661 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.661 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.661 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.661 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.661 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.661 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.661 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-01T01:50:07.661 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-01T01:50:07.662 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.662 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.662 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.662 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.662 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.662 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.662 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.662 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.662 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.662 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.662 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.662 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_ENABLED": { 2024-04-01T01:50:07.662 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-01T01:50:07.663 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.663 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.663 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.663 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.663 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.663 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.663 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.663 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.663 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.663 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.663 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.663 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-01T01:50:07.663 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-01T01:50:07.664 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.664 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.664 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.664 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-01T01:50:07.664 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.664 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.664 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.664 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.664 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.664 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.664 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.664 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-01T01:50:07.664 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-01T01:50:07.664 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.665 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.665 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.665 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.665 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.665 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.665 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.665 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.665 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.665 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.665 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.665 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-01T01:50:07.665 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-01T01:50:07.665 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.666 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.666 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.666 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8", 2024-04-01T01:50:07.666 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.666 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.666 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.666 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.666 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.666 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.666 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.666 INFO:teuthology.orchestra.run.smithi070.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-01T01:50:07.666 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-01T01:50:07.667 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.667 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.667 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.667 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "45", 2024-04-01T01:50:07.667 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.667 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.667 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.667 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.667 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.667 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.667 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.667 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_ACCESS_KEY": { 2024-04-01T01:50:07.667 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-01T01:50:07.668 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.668 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.668 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.668 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.668 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.668 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.668 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.668 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.668 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.668 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.668 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.668 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-01T01:50:07.668 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-01T01:50:07.669 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.669 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.669 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.669 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-04-01T01:50:07.669 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.669 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.669 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.669 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.669 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.669 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.669 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.669 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_SECRET_KEY": { 2024-04-01T01:50:07.669 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-01T01:50:07.670 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.670 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.670 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.670 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.670 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.670 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.670 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.670 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.670 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.670 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.670 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.670 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_SSL_VERIFY": { 2024-04-01T01:50:07.671 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-01T01:50:07.671 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.671 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.671 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.671 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.671 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.671 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.671 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.671 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.671 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.671 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.671 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.671 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-01T01:50:07.672 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-01T01:50:07.672 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.672 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.672 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.672 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-04-01T01:50:07.672 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.672 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.672 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.672 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.672 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.672 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.672 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.673 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-01T01:50:07.673 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-01T01:50:07.673 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.673 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.673 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.673 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.673 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.673 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.673 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.673 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.673 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.673 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.673 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.673 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-01T01:50:07.674 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-01T01:50:07.674 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.674 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.674 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.674 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-04-01T01:50:07.674 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.674 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.674 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.674 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.674 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.674 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.674 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.674 INFO:teuthology.orchestra.run.smithi070.stdout: "cross_origin_url": { 2024-04-01T01:50:07.675 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cross_origin_url", 2024-04-01T01:50:07.675 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.675 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.675 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.675 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.675 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.675 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.675 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.675 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.675 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.675 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.675 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.675 INFO:teuthology.orchestra.run.smithi070.stdout: "crt_file": { 2024-04-01T01:50:07.676 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crt_file", 2024-04-01T01:50:07.676 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.676 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.676 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.676 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.676 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.676 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.676 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.676 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.676 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.676 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.676 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.676 INFO:teuthology.orchestra.run.smithi070.stdout: "debug": { 2024-04-01T01:50:07.677 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "debug", 2024-04-01T01:50:07.677 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.677 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.677 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.677 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.677 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.677 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.677 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.677 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable/disable debug options", 2024-04-01T01:50:07.677 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.677 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.677 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.678 INFO:teuthology.orchestra.run.smithi070.stdout: "jwt_token_ttl": { 2024-04-01T01:50:07.678 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "jwt_token_ttl", 2024-04-01T01:50:07.678 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.678 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.678 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.678 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "28800", 2024-04-01T01:50:07.678 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.678 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.678 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.678 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.678 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.678 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.678 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.679 INFO:teuthology.orchestra.run.smithi070.stdout: "key_file": { 2024-04-01T01:50:07.679 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "key_file", 2024-04-01T01:50:07.679 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.679 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.679 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.679 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.679 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.679 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.679 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.679 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.679 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.679 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.679 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.679 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.680 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.680 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.680 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.680 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.680 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.680 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.680 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.680 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.680 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.680 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.680 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.680 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.680 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.681 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.681 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.681 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.681 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.681 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.681 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.681 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.681 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.681 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.681 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.681 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.681 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.682 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.682 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.682 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.682 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.682 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.682 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.682 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.682 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.682 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.682 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.682 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.682 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.683 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.683 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.683 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.683 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.683 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.683 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.683 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.683 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.683 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.683 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.683 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.683 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.683 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.683 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.684 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.684 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.684 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.684 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.684 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.684 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.684 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.684 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.684 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.684 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.684 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.684 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.684 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.684 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.685 INFO:teuthology.orchestra.run.smithi070.stdout: "motd": { 2024-04-01T01:50:07.685 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "motd", 2024-04-01T01:50:07.685 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.685 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.685 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.685 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.685 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.685 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.685 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.685 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "The message of the day", 2024-04-01T01:50:07.685 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.685 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.685 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.686 INFO:teuthology.orchestra.run.smithi070.stdout: "redirect_resolve_ip_addr": { 2024-04-01T01:50:07.686 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "redirect_resolve_ip_addr", 2024-04-01T01:50:07.686 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.686 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.686 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.686 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.686 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.686 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.686 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.686 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.686 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.686 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.687 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.687 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-04-01T01:50:07.687 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-04-01T01:50:07.687 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.687 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.687 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.687 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "::", 2024-04-01T01:50:07.687 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.687 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.687 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.687 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.687 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.687 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.688 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.688 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-04-01T01:50:07.688 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-04-01T01:50:07.688 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.688 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.688 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.688 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8080", 2024-04-01T01:50:07.688 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.688 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.688 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.688 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.688 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.688 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.689 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.689 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl": { 2024-04-01T01:50:07.689 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl", 2024-04-01T01:50:07.689 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.689 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.689 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.689 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.689 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.689 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.689 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.689 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.689 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.689 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.689 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.690 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl_server_port": { 2024-04-01T01:50:07.690 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl_server_port", 2024-04-01T01:50:07.690 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.690 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.690 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.690 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8443", 2024-04-01T01:50:07.690 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.690 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.690 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.690 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.690 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.690 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.690 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.691 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_behaviour": { 2024-04-01T01:50:07.691 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_behaviour", 2024-04-01T01:50:07.691 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.691 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.691 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.691 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "redirect", 2024-04-01T01:50:07.691 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.691 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.691 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.691 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.691 INFO:teuthology.orchestra.run.smithi070.stdout: "redirect" 2024-04-01T01:50:07.691 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.692 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.692 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.692 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.692 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.692 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_error_status_code": { 2024-04-01T01:50:07.692 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_error_status_code", 2024-04-01T01:50:07.692 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.692 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.692 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.692 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "500", 2024-04-01T01:50:07.692 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "400", 2024-04-01T01:50:07.692 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "599", 2024-04-01T01:50:07.692 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.692 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.693 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.693 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.693 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.693 INFO:teuthology.orchestra.run.smithi070.stdout: "url_prefix": { 2024-04-01T01:50:07.693 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "url_prefix", 2024-04-01T01:50:07.693 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.693 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.693 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.693 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.693 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.693 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.693 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.693 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.694 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.694 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.694 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.694 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.694 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.694 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "devicehealth", 2024-04-01T01:50:07.694 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.694 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.694 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.694 INFO:teuthology.orchestra.run.smithi070.stdout: "enable_monitoring": { 2024-04-01T01:50:07.695 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enable_monitoring", 2024-04-01T01:50:07.695 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.695 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.695 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.695 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.695 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.695 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.695 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.695 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "monitor device health metrics", 2024-04-01T01:50:07.695 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.695 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.695 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.695 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.696 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.696 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.696 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.696 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.696 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.696 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.696 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.696 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.696 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.696 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.696 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.696 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.696 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.696 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.697 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.697 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.697 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.697 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.697 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.697 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.697 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.697 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.697 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.697 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.697 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.697 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.697 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.698 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.698 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.698 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.698 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.698 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.698 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.698 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.698 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.698 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.698 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.698 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.698 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.699 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.699 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.699 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.699 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.699 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.699 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.699 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.699 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.699 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.699 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.699 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.699 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.699 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.700 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.700 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.700 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.700 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.700 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.700 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.700 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.700 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.700 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.700 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.700 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.700 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.700 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.701 INFO:teuthology.orchestra.run.smithi070.stdout: "mark_out_threshold": { 2024-04-01T01:50:07.701 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mark_out_threshold", 2024-04-01T01:50:07.701 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.701 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.701 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.701 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "2419200", 2024-04-01T01:50:07.701 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.701 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.701 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.701 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-01T01:50:07.701 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.701 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.701 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.702 INFO:teuthology.orchestra.run.smithi070.stdout: "pool_name": { 2024-04-01T01:50:07.702 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pool_name", 2024-04-01T01:50:07.702 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.702 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.702 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.702 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "device_health_metrics", 2024-04-01T01:50:07.702 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.702 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.702 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.702 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-01T01:50:07.702 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.702 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.702 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.703 INFO:teuthology.orchestra.run.smithi070.stdout: "retention_period": { 2024-04-01T01:50:07.703 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "retention_period", 2024-04-01T01:50:07.703 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.703 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.703 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.703 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15552000", 2024-04-01T01:50:07.703 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.703 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.703 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.703 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long to retain device health metrics", 2024-04-01T01:50:07.703 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.703 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.703 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.704 INFO:teuthology.orchestra.run.smithi070.stdout: "scrape_frequency": { 2024-04-01T01:50:07.704 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "scrape_frequency", 2024-04-01T01:50:07.704 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.704 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.704 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.704 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "86400", 2024-04-01T01:50:07.704 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.704 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.704 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.704 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-01T01:50:07.704 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.704 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.704 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.705 INFO:teuthology.orchestra.run.smithi070.stdout: "self_heal": { 2024-04-01T01:50:07.705 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "self_heal", 2024-04-01T01:50:07.705 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.705 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.705 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.705 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.705 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.705 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.705 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.705 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-01T01:50:07.705 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.705 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.705 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.706 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-04-01T01:50:07.706 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-04-01T01:50:07.706 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.706 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.706 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.706 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-04-01T01:50:07.706 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.706 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.706 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.706 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to wake up and check device health", 2024-04-01T01:50:07.706 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.706 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.706 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.707 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_threshold": { 2024-04-01T01:50:07.707 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_threshold", 2024-04-01T01:50:07.707 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.707 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.707 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.707 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "7257600", 2024-04-01T01:50:07.707 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.707 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.707 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.707 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-01T01:50:07.707 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.707 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.708 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.708 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.708 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.708 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "diskprediction_local", 2024-04-01T01:50:07.708 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.708 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.708 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.708 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.708 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.708 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.708 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.709 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.709 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.709 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.709 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.709 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.709 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.709 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.709 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.709 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.709 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.709 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.709 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.709 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.710 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.710 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.710 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.710 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.710 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.710 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.710 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.710 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.710 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.710 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.710 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.710 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.710 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.711 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.711 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.711 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.711 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.711 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.711 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.711 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.711 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.711 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.711 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.711 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.711 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.711 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.712 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.712 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.712 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.712 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.712 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.712 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.712 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.712 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.712 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.712 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.712 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.712 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.713 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.713 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.713 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.713 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.713 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.713 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.713 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.713 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.713 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.713 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.713 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.713 INFO:teuthology.orchestra.run.smithi070.stdout: "predict_interval": { 2024-04-01T01:50:07.713 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "predict_interval", 2024-04-01T01:50:07.714 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.714 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.714 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.714 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "86400", 2024-04-01T01:50:07.714 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.714 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.714 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.714 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.714 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.714 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.714 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.714 INFO:teuthology.orchestra.run.smithi070.stdout: "predictor_model": { 2024-04-01T01:50:07.715 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "predictor_model", 2024-04-01T01:50:07.715 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.715 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.715 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.715 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "prophetstor", 2024-04-01T01:50:07.715 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.715 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.715 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.715 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.715 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.715 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.715 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.715 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-04-01T01:50:07.716 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-04-01T01:50:07.716 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.716 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.716 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.716 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-04-01T01:50:07.716 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.716 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.716 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.716 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.716 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.716 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.716 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.716 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.716 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.717 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "influx", 2024-04-01T01:50:07.717 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": false, 2024-04-01T01:50:07.717 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "influxdb python module not found", 2024-04-01T01:50:07.717 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.717 INFO:teuthology.orchestra.run.smithi070.stdout: "batch_size": { 2024-04-01T01:50:07.717 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "batch_size", 2024-04-01T01:50:07.717 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.717 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.717 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.717 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5000", 2024-04-01T01:50:07.717 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.717 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.718 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.718 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-01T01:50:07.718 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.718 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.718 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.718 INFO:teuthology.orchestra.run.smithi070.stdout: "database": { 2024-04-01T01:50:07.718 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "database", 2024-04-01T01:50:07.718 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.718 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.718 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.718 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "ceph", 2024-04-01T01:50:07.718 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.719 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.719 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.719 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-01T01:50:07.719 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.719 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.719 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.719 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": { 2024-04-01T01:50:07.719 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "hostname", 2024-04-01T01:50:07.719 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.719 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.719 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.719 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.720 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.720 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.720 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.720 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB server hostname", 2024-04-01T01:50:07.720 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.720 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.720 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.720 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-04-01T01:50:07.720 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-04-01T01:50:07.720 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.720 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.720 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.720 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "30", 2024-04-01T01:50:07.720 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "5", 2024-04-01T01:50:07.721 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.721 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.721 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-01T01:50:07.721 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.721 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.721 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.721 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.721 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.721 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.721 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.721 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.721 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.721 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.722 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.722 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.722 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.722 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.722 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.722 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.722 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.722 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.722 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.722 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.722 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.722 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.722 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.723 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.723 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.723 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.723 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.723 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.723 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.723 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.723 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.723 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.723 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.723 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.723 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.723 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.724 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.724 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.724 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.724 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.724 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.724 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.724 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.724 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.724 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.724 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.724 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.724 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.724 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.725 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.725 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.725 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.725 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.725 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.725 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.725 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.725 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.725 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.725 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.725 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.725 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.725 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.726 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.726 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.726 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.726 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.726 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.726 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.726 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.726 INFO:teuthology.orchestra.run.smithi070.stdout: "password": { 2024-04-01T01:50:07.726 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "password", 2024-04-01T01:50:07.726 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.726 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.726 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.727 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.727 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.727 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.727 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.727 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "password of InfluxDB server user", 2024-04-01T01:50:07.727 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.727 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.727 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.727 INFO:teuthology.orchestra.run.smithi070.stdout: "port": { 2024-04-01T01:50:07.727 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "port", 2024-04-01T01:50:07.727 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.727 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.728 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.728 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8086", 2024-04-01T01:50:07.728 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.728 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.728 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.728 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB server port", 2024-04-01T01:50:07.728 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.728 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.728 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.728 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl": { 2024-04-01T01:50:07.728 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl", 2024-04-01T01:50:07.728 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.728 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.729 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.729 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "false", 2024-04-01T01:50:07.729 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.729 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.729 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.729 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-01T01:50:07.729 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.729 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.729 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.729 INFO:teuthology.orchestra.run.smithi070.stdout: "threads": { 2024-04-01T01:50:07.729 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "threads", 2024-04-01T01:50:07.729 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.729 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.730 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.730 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-04-01T01:50:07.730 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-04-01T01:50:07.730 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "32", 2024-04-01T01:50:07.730 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.730 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-01T01:50:07.730 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.730 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.730 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.730 INFO:teuthology.orchestra.run.smithi070.stdout: "username": { 2024-04-01T01:50:07.730 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "username", 2024-04-01T01:50:07.730 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.730 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.731 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.731 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.731 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.731 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.731 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.731 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "username of InfluxDB server user", 2024-04-01T01:50:07.731 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.731 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.731 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.731 INFO:teuthology.orchestra.run.smithi070.stdout: "verify_ssl": { 2024-04-01T01:50:07.731 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "verify_ssl", 2024-04-01T01:50:07.731 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.731 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.732 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.732 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "true", 2024-04-01T01:50:07.732 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.732 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.732 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.732 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-01T01:50:07.732 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.732 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.732 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.732 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.732 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.733 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "insights", 2024-04-01T01:50:07.733 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.733 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.733 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.733 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.733 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.733 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.733 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.733 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.733 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.733 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.733 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.733 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.733 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.733 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.734 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.735 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.736 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.736 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.736 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.736 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.736 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.736 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.736 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.736 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.736 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.736 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.736 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.736 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.736 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.737 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.737 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.737 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.737 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.737 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.737 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.737 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.737 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.737 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.737 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.737 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.737 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.737 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.738 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.738 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.738 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.738 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.738 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.738 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.738 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.738 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.738 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.738 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "iostat", 2024-04-01T01:50:07.738 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.738 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.739 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.739 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.739 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.739 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.739 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.739 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.739 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.739 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.739 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.739 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.739 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.739 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.739 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.739 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.740 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.740 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.740 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.740 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.740 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.740 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.740 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.740 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.740 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.740 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.740 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.740 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.741 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.741 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.741 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.741 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.741 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.741 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.741 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.741 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.741 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.741 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.741 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.741 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.741 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.741 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.742 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.742 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.742 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.742 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.742 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.742 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.742 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.742 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.742 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.742 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.742 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.742 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.742 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.742 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.743 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.743 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.743 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.743 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.743 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.743 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.743 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.743 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.743 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.743 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.743 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.743 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.743 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.743 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.744 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.744 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.744 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "k8sevents", 2024-04-01T01:50:07.744 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.744 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.744 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.744 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_event_retention_days": { 2024-04-01T01:50:07.744 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ceph_event_retention_days", 2024-04-01T01:50:07.744 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.744 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.744 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.744 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "7", 2024-04-01T01:50:07.744 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-01T01:50:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: "config_check_secs": { 2024-04-01T01:50:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_check_secs", 2024-04-01T01:50:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "10", 2024-04-01T01:50:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.745 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.746 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-01T01:50:07.746 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.746 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.746 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.746 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.746 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.746 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.746 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.746 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.746 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.746 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.746 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.746 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.746 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.747 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.747 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.747 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.747 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.747 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.747 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.747 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.747 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.747 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.747 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.747 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.747 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.748 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.748 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.748 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.748 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.748 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.748 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.748 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.748 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.748 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.748 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.748 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.748 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.748 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.748 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.749 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.749 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.749 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.749 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.749 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.749 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.749 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.749 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.749 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.749 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.749 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.749 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.749 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.750 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.750 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.750 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.750 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.750 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.750 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.750 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.750 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.750 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.750 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.750 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.750 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.750 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.751 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.751 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.751 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.751 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.751 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.751 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.751 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.751 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "localpool", 2024-04-01T01:50:07.751 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.751 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.751 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.751 INFO:teuthology.orchestra.run.smithi070.stdout: "failure_domain": { 2024-04-01T01:50:07.751 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "failure_domain", 2024-04-01T01:50:07.752 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.752 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.752 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.752 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "host", 2024-04-01T01:50:07.752 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.752 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.752 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.752 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "failure domain for any created local pool", 2024-04-01T01:50:07.752 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-01T01:50:07.752 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.752 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.752 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.753 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.753 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.753 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.753 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.753 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.753 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.753 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.753 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.753 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.753 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.753 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.753 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.753 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.753 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.754 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.754 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.754 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.754 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.754 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.754 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.754 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.754 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.754 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.754 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.754 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.755 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.755 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.755 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.755 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.755 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.755 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.755 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.755 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.755 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.755 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.755 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.755 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.755 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.756 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.756 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.756 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.756 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.756 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.756 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.756 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.756 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.756 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.756 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.756 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.756 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.756 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.756 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.757 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.757 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.757 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.757 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.757 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.757 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.757 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.757 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.757 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.757 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.757 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.757 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.757 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.758 INFO:teuthology.orchestra.run.smithi070.stdout: "min_size": { 2024-04-01T01:50:07.758 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "min_size", 2024-04-01T01:50:07.758 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.758 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.758 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.758 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.758 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.758 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.758 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.758 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default min_size for any created local pool", 2024-04-01T01:50:07.758 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-01T01:50:07.758 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.758 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.759 INFO:teuthology.orchestra.run.smithi070.stdout: "num_rep": { 2024-04-01T01:50:07.759 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "num_rep", 2024-04-01T01:50:07.759 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.759 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.759 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.759 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3", 2024-04-01T01:50:07.759 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.759 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.759 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.759 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default replica count for any created local pool", 2024-04-01T01:50:07.759 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.759 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.759 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.760 INFO:teuthology.orchestra.run.smithi070.stdout: "pg_num": { 2024-04-01T01:50:07.760 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pg_num", 2024-04-01T01:50:07.760 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.760 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.760 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.760 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "128", 2024-04-01T01:50:07.760 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.760 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.760 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.760 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default pg_num for any created local pool", 2024-04-01T01:50:07.760 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.760 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.760 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.761 INFO:teuthology.orchestra.run.smithi070.stdout: "prefix": { 2024-04-01T01:50:07.761 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prefix", 2024-04-01T01:50:07.761 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.761 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.761 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.761 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.761 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.761 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.761 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.761 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "name prefix for any created local pool", 2024-04-01T01:50:07.761 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.761 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.761 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.762 INFO:teuthology.orchestra.run.smithi070.stdout: "subtree": { 2024-04-01T01:50:07.762 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "subtree", 2024-04-01T01:50:07.762 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.762 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.762 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.762 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "rack", 2024-04-01T01:50:07.762 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.762 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.762 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.763 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-01T01:50:07.763 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-01T01:50:07.763 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.763 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.763 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.763 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.763 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mds_autoscaler", 2024-04-01T01:50:07.763 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.763 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.763 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.763 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.763 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.764 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.764 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.764 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.764 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.764 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.764 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.764 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.764 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.764 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.764 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.764 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.764 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.764 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.764 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.764 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.765 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.765 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.765 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.765 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.765 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.765 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.765 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.765 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.765 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.765 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.765 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.765 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.765 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.766 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.766 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.766 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.766 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.766 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.766 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.766 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.766 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.766 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.766 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.766 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.766 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.767 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.767 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.767 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.767 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.767 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.767 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.767 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.767 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.767 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.767 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.767 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.767 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.767 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.767 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.768 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.768 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.768 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.768 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.768 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.768 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.768 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.768 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.768 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.768 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.768 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.768 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.768 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.768 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.769 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mirroring", 2024-04-01T01:50:07.769 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.769 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.769 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.769 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.769 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.769 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.769 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.769 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.769 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.769 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.769 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.769 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.770 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.770 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.770 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.770 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.770 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.770 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.770 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.770 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.770 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.770 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.770 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.770 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.771 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.771 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.771 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.771 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.771 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.771 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.771 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.771 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.771 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.771 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.771 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.771 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.773 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.773 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.773 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.773 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.773 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.773 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.773 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.773 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.773 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.773 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.773 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.773 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.773 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.773 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.773 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.774 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.774 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.774 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.774 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.774 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.774 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.774 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.774 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "nfs", 2024-04-01T01:50:07.774 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.774 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.774 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.774 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.775 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.775 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.775 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.775 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.775 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.775 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.775 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.775 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.775 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.775 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.775 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.775 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.775 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.775 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.776 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.776 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.776 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.776 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.776 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.776 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.776 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.776 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.776 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.776 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.776 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.776 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.776 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.776 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.776 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.777 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.777 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.777 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.777 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.777 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.777 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.777 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.777 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.777 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.777 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.777 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.777 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.778 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.779 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.780 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.780 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.780 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.780 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.780 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "orchestrator", 2024-04-01T01:50:07.780 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.780 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.780 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.780 INFO:teuthology.orchestra.run.smithi070.stdout: "fail_fs": { 2024-04-01T01:50:07.780 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "fail_fs", 2024-04-01T01:50:07.780 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.780 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.780 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.780 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.781 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.781 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.781 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.781 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-01T01:50:07.781 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.781 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.781 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.781 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.781 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.781 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.781 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.781 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.781 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.781 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.782 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.782 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.782 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.782 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.782 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.782 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.782 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.782 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.782 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.782 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.782 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.782 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.782 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.782 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.783 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.783 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.783 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.783 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.783 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.783 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.783 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.783 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.783 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.783 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.783 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.783 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.783 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.783 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.783 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.783 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.784 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.784 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.784 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.784 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.784 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.784 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.784 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.784 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.784 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.784 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.784 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.784 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.784 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.784 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.784 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.785 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.786 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.786 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.786 INFO:teuthology.orchestra.run.smithi070.stdout: "orchestrator": { 2024-04-01T01:50:07.786 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "orchestrator", 2024-04-01T01:50:07.786 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.786 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.786 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.786 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.786 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.786 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.786 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.786 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm", 2024-04-01T01:50:07.786 INFO:teuthology.orchestra.run.smithi070.stdout: "rook", 2024-04-01T01:50:07.786 INFO:teuthology.orchestra.run.smithi070.stdout: "test_orchestrator" 2024-04-01T01:50:07.787 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.787 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Orchestrator backend", 2024-04-01T01:50:07.787 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.787 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.787 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.787 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.787 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.787 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.787 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.787 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "osd_perf_query", 2024-04-01T01:50:07.787 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.787 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.787 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.787 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.787 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.788 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.788 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.788 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.788 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.788 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.788 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.788 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.788 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.788 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.788 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.788 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.788 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.788 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.788 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.788 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.789 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.789 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.789 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.789 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.789 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.789 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.789 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.789 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.789 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.789 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.789 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.789 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.789 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.789 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.789 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.790 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.790 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.790 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.790 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.790 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.790 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.790 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.790 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.790 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.790 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.790 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.790 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.790 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.790 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.791 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.791 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.791 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.791 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.791 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.791 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.791 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.791 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.791 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.791 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.791 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.791 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.791 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.791 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.792 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.792 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.792 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.792 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.792 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.792 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.792 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.792 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.792 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.792 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.792 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.792 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.792 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.792 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "osd_support", 2024-04-01T01:50:07.793 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.793 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.793 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.793 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.793 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.793 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.793 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.793 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.793 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.793 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.793 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.793 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.793 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.793 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.793 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.794 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.794 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.794 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.794 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.794 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.794 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.794 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.794 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.794 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.794 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.794 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.794 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.794 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.794 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.795 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.795 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.795 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.795 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.795 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.795 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.795 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.795 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.795 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.795 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.795 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.795 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.795 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.795 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.795 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.796 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.796 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.796 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.796 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.796 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.796 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.796 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.796 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.796 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.796 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.796 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.796 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.796 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.796 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.796 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.797 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.797 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.797 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.797 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.797 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.797 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.797 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.797 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.797 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.797 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.797 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.797 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.797 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.798 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.798 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.798 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.798 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.798 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pg_autoscaler", 2024-04-01T01:50:07.798 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.798 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.798 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.798 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.798 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.798 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.798 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.798 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.798 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.799 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.799 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.799 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.799 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.799 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.799 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.799 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.799 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.799 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.799 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.799 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.799 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.799 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.800 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.800 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.800 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.800 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.800 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.800 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.800 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.800 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.800 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.800 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.800 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.800 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.800 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.800 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.801 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.801 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.801 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.801 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.801 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.801 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.801 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.801 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.801 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.801 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.801 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.801 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.801 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.801 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.802 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.802 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.802 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.802 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.802 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.802 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.802 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.802 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.802 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.802 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.802 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.802 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.802 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.802 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.803 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.803 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.803 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.803 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.803 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.803 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.803 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.803 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.803 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.803 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-04-01T01:50:07.803 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-04-01T01:50:07.803 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.803 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.803 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.803 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-04-01T01:50:07.804 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.804 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.804 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.804 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.804 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.804 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.804 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.804 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.804 INFO:teuthology.orchestra.run.smithi070.stdout: "threshold": { 2024-04-01T01:50:07.804 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "threshold", 2024-04-01T01:50:07.804 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.804 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.804 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3.0", 2024-04-01T01:50:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1.0", 2024-04-01T01:50:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "scaling threshold", 2024-04-01T01:50:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-01T01:50:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "progress", 2024-04-01T01:50:07.805 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.806 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.806 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.806 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_pg_recovery_event": { 2024-04-01T01:50:07.806 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_pg_recovery_event", 2024-04-01T01:50:07.806 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.806 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.806 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.806 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.806 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.806 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.806 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.806 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-01T01:50:07.806 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.807 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.807 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.807 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.807 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": { 2024-04-01T01:50:07.807 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enabled", 2024-04-01T01:50:07.807 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.807 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.807 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.807 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.807 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.807 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.807 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.807 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.807 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.808 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.808 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.808 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.808 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.808 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.808 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.808 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.808 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.808 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.808 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.808 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.808 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.808 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.809 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.809 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.809 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.809 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.809 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.809 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.809 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.809 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.809 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.809 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.809 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.809 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.809 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.809 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.810 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.810 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.810 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.810 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.810 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.810 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.810 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.810 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.810 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.810 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.810 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.810 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.810 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.811 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.811 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.811 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.811 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.811 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.811 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.811 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.811 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.811 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.811 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.811 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.811 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.811 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.812 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.812 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.812 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.812 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.812 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.812 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.812 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.812 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.813 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.813 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.813 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.813 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.813 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.813 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.813 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.814 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.814 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.814 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.814 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.814 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.814 INFO:teuthology.orchestra.run.smithi070.stdout: "max_completed_events": { 2024-04-01T01:50:07.814 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_completed_events", 2024-04-01T01:50:07.814 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.815 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.815 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.815 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "50", 2024-04-01T01:50:07.815 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.815 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.815 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.815 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "number of past completed events to remember", 2024-04-01T01:50:07.815 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.816 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.816 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.816 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.816 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-04-01T01:50:07.816 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-04-01T01:50:07.816 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.816 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.816 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.816 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-04-01T01:50:07.816 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.816 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.816 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.817 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long the module is going to sleep", 2024-04-01T01:50:07.817 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.817 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.817 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.817 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.817 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.817 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.817 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.817 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prometheus", 2024-04-01T01:50:07.817 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.817 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.817 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.817 INFO:teuthology.orchestra.run.smithi070.stdout: "cache": { 2024-04-01T01:50:07.817 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cache", 2024-04-01T01:50:07.818 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.818 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.818 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.818 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.818 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.818 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.818 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.818 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.818 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.818 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.818 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.818 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.819 INFO:teuthology.orchestra.run.smithi070.stdout: "exclude_perf_counters": { 2024-04-01T01:50:07.819 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "exclude_perf_counters", 2024-04-01T01:50:07.819 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.819 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.819 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.819 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.819 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.819 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.819 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.819 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-01T01:50:07.819 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-01T01:50:07.819 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.819 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.820 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.820 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.820 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.820 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.820 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.820 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.820 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.820 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.820 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.820 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.820 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.820 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.820 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.820 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.821 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.821 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.821 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.821 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.821 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.821 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.821 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.821 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.821 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.821 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.821 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.821 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.821 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.821 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.822 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.822 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.822 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.822 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.822 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.822 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.822 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.822 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.822 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.822 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.822 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.822 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.822 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.823 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.823 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.823 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.823 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.823 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.823 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.823 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.823 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.823 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.823 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.823 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.823 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.823 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.823 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.824 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.824 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.824 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.824 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.824 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.824 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.824 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.824 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.824 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.824 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.824 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.824 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.824 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.825 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.825 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_stats_pools": { 2024-04-01T01:50:07.825 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_stats_pools", 2024-04-01T01:50:07.825 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.825 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.825 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.825 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.825 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.825 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.825 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.825 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.825 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.826 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.826 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.826 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.826 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-01T01:50:07.826 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-01T01:50:07.826 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.826 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.826 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.826 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "300", 2024-04-01T01:50:07.826 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.826 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.826 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.826 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.827 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.827 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.827 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.827 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.827 INFO:teuthology.orchestra.run.smithi070.stdout: "scrape_interval": { 2024-04-01T01:50:07.827 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "scrape_interval", 2024-04-01T01:50:07.827 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.827 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.827 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.827 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15.0", 2024-04-01T01:50:07.827 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.827 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.827 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.827 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.828 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.828 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.828 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.828 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.828 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-04-01T01:50:07.828 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-04-01T01:50:07.828 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.828 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.828 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.828 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "::", 2024-04-01T01:50:07.828 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.828 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.828 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.828 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-01T01:50:07.828 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.829 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.829 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.829 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.829 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-04-01T01:50:07.829 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-04-01T01:50:07.829 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.829 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.829 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.829 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "9283", 2024-04-01T01:50:07.829 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.829 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.829 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.829 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-01T01:50:07.830 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.830 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.830 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.830 INFO:teuthology.orchestra.run.smithi070.stdout: "stale_cache_strategy": { 2024-04-01T01:50:07.830 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "stale_cache_strategy", 2024-04-01T01:50:07.830 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.830 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.830 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.830 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "log", 2024-04-01T01:50:07.830 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.830 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.830 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.830 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.831 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.831 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.831 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.831 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.831 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_behaviour": { 2024-04-01T01:50:07.831 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_behaviour", 2024-04-01T01:50:07.831 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.831 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.831 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.831 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "default", 2024-04-01T01:50:07.831 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.831 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.831 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.831 INFO:teuthology.orchestra.run.smithi070.stdout: "default", 2024-04-01T01:50:07.831 INFO:teuthology.orchestra.run.smithi070.stdout: "error" 2024-04-01T01:50:07.831 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.832 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.832 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.832 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.832 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.832 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_error_status_code": { 2024-04-01T01:50:07.832 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_error_status_code", 2024-04-01T01:50:07.832 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.832 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.832 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.832 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "500", 2024-04-01T01:50:07.832 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "400", 2024-04-01T01:50:07.832 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "599", 2024-04-01T01:50:07.833 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.833 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.833 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.833 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.833 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.833 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.833 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.833 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.833 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.833 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_support", 2024-04-01T01:50:07.833 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.833 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.833 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.834 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.834 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.834 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.834 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.834 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.834 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.834 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.834 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.834 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.834 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.834 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.834 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.834 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.834 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.835 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.835 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.835 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.835 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.835 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.835 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.835 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.835 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.835 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.835 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.835 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.835 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.836 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.836 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.836 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.836 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.836 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.836 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.836 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.836 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.836 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.836 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.836 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.836 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.836 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.836 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.836 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.837 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.837 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.837 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.837 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.837 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.837 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.837 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.837 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.837 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.837 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.837 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.837 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.837 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.837 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.838 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.838 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.838 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.838 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.838 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.838 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.838 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.838 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.838 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.838 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.838 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.838 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.838 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.839 INFO:teuthology.orchestra.run.smithi070.stdout: "max_concurrent_snap_create": { 2024-04-01T01:50:07.839 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_concurrent_snap_create", 2024-04-01T01:50:07.839 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.839 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.839 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.839 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.839 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.839 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.839 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.839 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.839 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.839 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.839 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "mirror_snapshot_schedule": { 2024-04-01T01:50:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mirror_snapshot_schedule", 2024-04-01T01:50:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "trash_purge_schedule": { 2024-04-01T01:50:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "trash_purge_schedule", 2024-04-01T01:50:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "restful", 2024-04-01T01:50:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "enable_auth": { 2024-04-01T01:50:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enable_auth", 2024-04-01T01:50:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "key_file": { 2024-04-01T01:50:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "key_file", 2024-04-01T01:50:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-04-01T01:50:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-04-01T01:50:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-04-01T01:50:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-04-01T01:50:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rgw", 2024-04-01T01:50:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rook", 2024-04-01T01:50:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "storage_class": { 2024-04-01T01:50:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "storage_class", 2024-04-01T01:50:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "local", 2024-04-01T01:50:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-01T01:50:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "selftest", 2024-04-01T01:50:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "roption1": { 2024-04-01T01:50:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "roption1", 2024-04-01T01:50:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "roption2": { 2024-04-01T01:50:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "roption2", 2024-04-01T01:50:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "xyz", 2024-04-01T01:50:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption1": { 2024-04-01T01:50:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption1", 2024-04-01T01:50:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption2": { 2024-04-01T01:50:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption2", 2024-04-01T01:50:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption3": { 2024-04-01T01:50:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption3", 2024-04-01T01:50:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption4": { 2024-04-01T01:50:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption4", 2024-04-01T01:50:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption5": { 2024-04-01T01:50:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption5", 2024-04-01T01:50:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption6": { 2024-04-01T01:50:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption6", 2024-04-01T01:50:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption7": { 2024-04-01T01:50:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption7", 2024-04-01T01:50:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-04-01T01:50:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "42", 2024-04-01T01:50:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "testkey": { 2024-04-01T01:50:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testkey", 2024-04-01T01:50:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "testlkey": { 2024-04-01T01:50:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testlkey", 2024-04-01T01:50:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "testnewline": { 2024-04-01T01:50:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testnewline", 2024-04-01T01:50:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snap_schedule", 2024-04-01T01:50:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_m_granularity": { 2024-04-01T01:50:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_m_granularity", 2024-04-01T01:50:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow minute scheduled snapshots", 2024-04-01T01:50:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "dump_on_update": { 2024-04-01T01:50:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "dump_on_update", 2024-04-01T01:50:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "dump database to debug log on update", 2024-04-01T01:50:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "stats", 2024-04-01T01:50:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "status", 2024-04-01T01:50:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "telegraf", 2024-04-01T01:50:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "address": { 2024-04-01T01:50:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "address", 2024-04-01T01:50:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-01T01:50:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-04-01T01:50:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-04-01T01:50:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15", 2024-04-01T01:50:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "telemetry", 2024-04-01T01:50:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_basic": { 2024-04-01T01:50:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_basic", 2024-04-01T01:50:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-01T01:50:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_crash": { 2024-04-01T01:50:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_crash", 2024-04-01T01:50:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-01T01:50:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_device": { 2024-04-01T01:50:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_device", 2024-04-01T01:50:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-01T01:50:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_ident": { 2024-04-01T01:50:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_ident", 2024-04-01T01:50:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-01T01:50:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_perf": { 2024-04-01T01:50:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_perf", 2024-04-01T01:50:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-01T01:50:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "contact": { 2024-04-01T01:50:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "contact", 2024-04-01T01:50:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "description": { 2024-04-01T01:50:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "description", 2024-04-01T01:50:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "device_url": { 2024-04-01T01:50:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_url", 2024-04-01T01:50:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-01T01:50:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": { 2024-04-01T01:50:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enabled", 2024-04-01T01:50:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-04-01T01:50:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-04-01T01:50:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "24", 2024-04-01T01:50:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "8", 2024-04-01T01:50:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "last_opt_revision": { 2024-04-01T01:50:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "last_opt_revision", 2024-04-01T01:50:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1", 2024-04-01T01:50:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "leaderboard": { 2024-04-01T01:50:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "leaderboard", 2024-04-01T01:50:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "leaderboard_description": { 2024-04-01T01:50:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "leaderboard_description", 2024-04-01T01:50:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.925 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.925 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.925 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.925 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.925 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.925 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.925 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.925 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.925 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.925 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.925 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.925 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.925 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.926 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.926 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.926 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.926 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.926 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.926 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.926 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.926 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.926 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.926 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.926 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.926 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.927 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.927 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.927 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.927 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.927 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.927 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.927 INFO:teuthology.orchestra.run.smithi070.stdout: "organization": { 2024-04-01T01:50:07.927 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "organization", 2024-04-01T01:50:07.927 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.927 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.927 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.927 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.927 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.928 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.928 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.928 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.928 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.928 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.928 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.928 INFO:teuthology.orchestra.run.smithi070.stdout: "proxy": { 2024-04-01T01:50:07.928 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "proxy", 2024-04-01T01:50:07.928 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.928 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.928 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.928 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.928 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.928 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.929 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.929 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.929 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.929 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.929 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.929 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.929 INFO:teuthology.orchestra.run.smithi070.stdout: "url": { 2024-04-01T01:50:07.929 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "url", 2024-04-01T01:50:07.929 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.929 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.929 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.929 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-01T01:50:07.929 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.929 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.930 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.930 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.930 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.930 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.930 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.930 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.930 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.930 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.930 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "test_orchestrator", 2024-04-01T01:50:07.930 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.930 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.930 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.930 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.931 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.931 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.931 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.931 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.931 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.931 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.931 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.931 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.931 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.931 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.931 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.931 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.931 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.931 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.931 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.932 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.932 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.932 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.932 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.932 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.932 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.932 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.932 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.932 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.932 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.932 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.932 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.932 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.933 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.933 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.933 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.933 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.933 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.933 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.933 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.933 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.933 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.933 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.933 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.933 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.933 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.934 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.934 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.934 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.934 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.934 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.934 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.934 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.934 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.934 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.934 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.934 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.934 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.935 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.935 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.935 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.935 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.935 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.935 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.935 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.935 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.935 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.935 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.935 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.935 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.935 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.935 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.936 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.936 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "volumes", 2024-04-01T01:50:07.936 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.936 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.936 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.936 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.936 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.936 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.936 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.936 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.936 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.936 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.936 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.937 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.937 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.937 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.937 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.937 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.937 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.937 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.937 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.937 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.937 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.937 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.937 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.937 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.938 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.938 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.938 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.938 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.938 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.938 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.938 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.938 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.938 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.938 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.938 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.938 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.938 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.939 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.939 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.939 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.939 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.939 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.939 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.939 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.939 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.939 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.939 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.939 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.939 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.939 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.939 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.939 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.940 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.940 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.940 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.940 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.940 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.940 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.940 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.940 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.940 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.940 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.940 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.940 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.941 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.941 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.941 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.941 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.941 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.941 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.941 INFO:teuthology.orchestra.run.smithi070.stdout: "max_concurrent_clones": { 2024-04-01T01:50:07.941 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_concurrent_clones", 2024-04-01T01:50:07.941 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.941 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.941 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.941 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "4", 2024-04-01T01:50:07.941 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.941 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.942 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.942 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-01T01:50:07.942 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.942 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.942 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.942 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.942 INFO:teuthology.orchestra.run.smithi070.stdout: "periodic_async_work": { 2024-04-01T01:50:07.942 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "periodic_async_work", 2024-04-01T01:50:07.942 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.942 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.942 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.942 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.942 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.943 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.943 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.943 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Periodically check for async work", 2024-04-01T01:50:07.943 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.943 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.943 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.943 INFO:teuthology.orchestra.run.smithi070.stdout: "snapshot_clone_delay": { 2024-04-01T01:50:07.943 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snapshot_clone_delay", 2024-04-01T01:50:07.943 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.943 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.943 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.943 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-04-01T01:50:07.944 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.944 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.944 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.944 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-01T01:50:07.944 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.944 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.944 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.944 INFO:teuthology.orchestra.run.smithi070.stdout: "snapshot_clone_no_wait": { 2024-04-01T01:50:07.944 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snapshot_clone_no_wait", 2024-04-01T01:50:07.944 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.944 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.944 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.945 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.945 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.945 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.945 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.945 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-01T01:50:07.945 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.945 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.945 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.945 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.945 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.945 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.945 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix", 2024-04-01T01:50:07.945 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.945 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.946 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.946 INFO:teuthology.orchestra.run.smithi070.stdout: "discovery_interval": { 2024-04-01T01:50:07.946 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "discovery_interval", 2024-04-01T01:50:07.946 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-04-01T01:50:07.946 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.946 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.946 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "100", 2024-04-01T01:50:07.946 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.946 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.946 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.946 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.946 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.946 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.947 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.947 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.947 INFO:teuthology.orchestra.run.smithi070.stdout: "identifier": { 2024-04-01T01:50:07.947 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "identifier", 2024-04-01T01:50:07.947 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.947 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.947 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.947 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.947 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.947 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.947 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.947 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.947 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.947 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.948 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.948 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-04-01T01:50:07.948 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-04-01T01:50:07.948 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.948 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.948 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.948 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-04-01T01:50:07.948 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.948 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.948 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.948 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.948 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.949 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.949 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.949 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.949 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.949 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.949 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.949 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.949 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.949 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.949 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.949 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.949 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.949 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.949 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.950 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.950 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.950 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.950 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.950 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.950 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.950 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.950 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.950 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.950 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.950 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.950 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.950 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.950 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.951 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.951 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.951 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.951 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.951 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.951 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.951 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.951 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.951 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.951 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.951 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.951 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.951 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.951 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.952 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.952 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.952 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.952 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.952 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.952 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.952 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.952 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.952 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.952 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.952 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.952 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.952 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.952 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.952 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.953 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.953 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.953 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.953 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.953 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.953 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.953 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.953 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.953 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.953 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.953 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.953 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.953 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.954 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.954 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_host": { 2024-04-01T01:50:07.954 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_host", 2024-04-01T01:50:07.954 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.954 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.954 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.954 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.954 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.954 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.954 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.954 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.954 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.954 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.955 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.955 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.955 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_port": { 2024-04-01T01:50:07.955 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_port", 2024-04-01T01:50:07.955 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.955 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.955 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.955 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10051", 2024-04-01T01:50:07.955 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.955 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.955 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.955 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.955 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.955 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.956 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.956 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.956 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_sender": { 2024-04-01T01:50:07.956 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_sender", 2024-04-01T01:50:07.956 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.956 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.956 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.956 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-01T01:50:07.956 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.956 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.956 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.956 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.956 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.956 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.957 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.957 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.957 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.957 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.957 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-01T01:50:07.957 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.957 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.957 INFO:teuthology.orchestra.run.smithi070.stdout: "modules": [ 2024-04-01T01:50:07.957 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm", 2024-04-01T01:50:07.957 INFO:teuthology.orchestra.run.smithi070.stdout: "dashboard", 2024-04-01T01:50:07.957 INFO:teuthology.orchestra.run.smithi070.stdout: "iostat", 2024-04-01T01:50:07.957 INFO:teuthology.orchestra.run.smithi070.stdout: "nfs", 2024-04-01T01:50:07.957 INFO:teuthology.orchestra.run.smithi070.stdout: "restful" 2024-04-01T01:50:07.958 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.958 INFO:teuthology.orchestra.run.smithi070.stdout: "available_modules": [ 2024-04-01T01:50:07.958 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.958 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "alerts", 2024-04-01T01:50:07.958 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.958 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.958 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.958 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-04-01T01:50:07.958 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-04-01T01:50:07.958 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.958 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.958 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.958 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-04-01T01:50:07.958 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.959 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.959 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.959 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How frequently to reexamine health status", 2024-04-01T01:50:07.959 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.959 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.959 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.959 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.959 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.959 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.959 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.959 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.959 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.959 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.959 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.959 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.960 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.960 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.960 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.960 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.960 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.960 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.960 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.960 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.960 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.960 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.960 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.960 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.960 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.960 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.960 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.961 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.961 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.961 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.961 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.961 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.961 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.961 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.961 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.961 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.961 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.961 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.961 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.961 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.961 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.962 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.962 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.962 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.962 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.962 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.962 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.962 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.962 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.962 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.962 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.962 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.962 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.962 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.963 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.963 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.963 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.963 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.963 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.963 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.963 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.963 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.963 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.963 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.963 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.963 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.963 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.963 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.964 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.964 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.964 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.964 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.964 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.964 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.964 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_destination": { 2024-04-01T01:50:07.964 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_destination", 2024-04-01T01:50:07.964 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.964 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.964 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.964 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.964 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.964 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.965 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.965 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Email address to send alerts to", 2024-04-01T01:50:07.965 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.965 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.965 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.965 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.965 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_from_name": { 2024-04-01T01:50:07.965 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_from_name", 2024-04-01T01:50:07.965 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.965 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.965 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.965 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "Ceph", 2024-04-01T01:50:07.965 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.965 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.965 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.966 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Email From: name", 2024-04-01T01:50:07.966 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.966 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.966 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.966 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.966 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_host": { 2024-04-01T01:50:07.966 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_host", 2024-04-01T01:50:07.966 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.966 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.966 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.966 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.966 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.966 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.967 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.967 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP server", 2024-04-01T01:50:07.967 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.967 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.967 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.967 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.967 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_password": { 2024-04-01T01:50:07.967 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_password", 2024-04-01T01:50:07.967 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.967 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.967 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.967 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.967 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.967 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.968 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.968 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Password to authenticate with", 2024-04-01T01:50:07.968 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.968 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.968 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.968 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_port": { 2024-04-01T01:50:07.968 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_port", 2024-04-01T01:50:07.968 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.968 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.968 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.968 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "465", 2024-04-01T01:50:07.968 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.968 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.969 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.969 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP port", 2024-04-01T01:50:07.969 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.969 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.969 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.969 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_sender": { 2024-04-01T01:50:07.969 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_sender", 2024-04-01T01:50:07.969 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.969 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.969 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.969 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.969 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.969 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.969 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.970 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SMTP envelope sender", 2024-04-01T01:50:07.970 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.970 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.970 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.970 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.970 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_ssl": { 2024-04-01T01:50:07.970 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_ssl", 2024-04-01T01:50:07.970 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.970 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.970 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.970 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.970 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.970 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.971 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.971 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-01T01:50:07.971 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.971 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.971 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.971 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.971 INFO:teuthology.orchestra.run.smithi070.stdout: "smtp_user": { 2024-04-01T01:50:07.971 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "smtp_user", 2024-04-01T01:50:07.971 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.971 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.971 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.971 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.971 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.971 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.971 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.972 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "User to authenticate as", 2024-04-01T01:50:07.972 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.972 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.972 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.972 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.972 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.972 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.972 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "balancer", 2024-04-01T01:50:07.972 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.972 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.972 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.972 INFO:teuthology.orchestra.run.smithi070.stdout: "active": { 2024-04-01T01:50:07.972 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "active", 2024-04-01T01:50:07.972 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.973 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.973 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.973 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.973 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.973 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.973 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.973 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "automatically balance PGs across cluster", 2024-04-01T01:50:07.973 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.973 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.973 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.973 INFO:teuthology.orchestra.run.smithi070.stdout: "begin_time": { 2024-04-01T01:50:07.973 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "begin_time", 2024-04-01T01:50:07.973 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.974 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.974 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.974 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0000", 2024-04-01T01:50:07.974 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.974 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.974 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.974 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "beginning time of day to automatically balance", 2024-04-01T01:50:07.974 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-01T01:50:07.974 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.974 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.974 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.974 INFO:teuthology.orchestra.run.smithi070.stdout: "begin_weekday": { 2024-04-01T01:50:07.974 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "begin_weekday", 2024-04-01T01:50:07.974 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-04-01T01:50:07.974 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.975 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.975 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-04-01T01:50:07.975 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0", 2024-04-01T01:50:07.975 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "6", 2024-04-01T01:50:07.975 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.975 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-01T01:50:07.975 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-01T01:50:07.975 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.975 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.975 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.975 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_max_iterations": { 2024-04-01T01:50:07.975 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_max_iterations", 2024-04-01T01:50:07.975 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-04-01T01:50:07.975 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.975 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.976 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "25", 2024-04-01T01:50:07.976 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-04-01T01:50:07.976 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "250", 2024-04-01T01:50:07.976 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.976 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-01T01:50:07.976 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.976 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.976 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.976 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.976 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_metrics": { 2024-04-01T01:50:07.976 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_metrics", 2024-04-01T01:50:07.976 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.976 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.976 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.976 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "pgs,objects,bytes", 2024-04-01T01:50:07.976 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.977 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.977 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.977 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-01T01:50:07.977 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-01T01:50:07.977 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.977 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.977 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.977 INFO:teuthology.orchestra.run.smithi070.stdout: "crush_compat_step": { 2024-04-01T01:50:07.977 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crush_compat_step", 2024-04-01T01:50:07.977 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.977 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.977 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.977 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0.5", 2024-04-01T01:50:07.977 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0.001", 2024-04-01T01:50:07.978 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "0.999", 2024-04-01T01:50:07.978 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.978 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "aggressiveness of optimization", 2024-04-01T01:50:07.978 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-01T01:50:07.978 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.978 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.978 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.978 INFO:teuthology.orchestra.run.smithi070.stdout: "end_time": { 2024-04-01T01:50:07.978 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "end_time", 2024-04-01T01:50:07.978 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.978 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.978 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.978 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "2359", 2024-04-01T01:50:07.978 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.979 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.979 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.979 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "ending time of day to automatically balance", 2024-04-01T01:50:07.979 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-01T01:50:07.979 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.979 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.979 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.979 INFO:teuthology.orchestra.run.smithi070.stdout: "end_weekday": { 2024-04-01T01:50:07.979 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "end_weekday", 2024-04-01T01:50:07.979 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-04-01T01:50:07.979 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.979 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.979 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-04-01T01:50:07.979 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "0", 2024-04-01T01:50:07.979 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "6", 2024-04-01T01:50:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-01T01:50:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-01T01:50:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.980 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.981 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.981 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.981 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.981 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.981 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.981 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.981 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.981 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.981 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.981 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.981 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:07.981 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:07.981 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.981 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.981 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.982 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.982 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.982 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.982 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.982 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.982 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.982 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.982 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.982 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.982 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:07.982 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:07.982 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.982 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.982 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.983 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:07.983 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.983 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.983 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.983 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:07.983 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:07.983 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:07.983 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:07.983 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:07.983 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:07.983 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.983 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.983 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.983 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.983 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: "min_score": { 2024-04-01T01:50:07.984 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "min_score", 2024-04-01T01:50:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-04-01T01:50:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-01T01:50:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": { 2024-04-01T01:50:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mode", 2024-04-01T01:50:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.985 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.986 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.986 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "upmap", 2024-04-01T01:50:07.986 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.986 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.986 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.986 INFO:teuthology.orchestra.run.smithi070.stdout: "crush-compat", 2024-04-01T01:50:07.986 INFO:teuthology.orchestra.run.smithi070.stdout: "none", 2024-04-01T01:50:07.986 INFO:teuthology.orchestra.run.smithi070.stdout: "read", 2024-04-01T01:50:07.986 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap", 2024-04-01T01:50:07.986 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap-read" 2024-04-01T01:50:07.986 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.986 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Balancer mode", 2024-04-01T01:50:07.986 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.986 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.987 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.987 INFO:teuthology.orchestra.run.smithi070.stdout: "pool_ids": { 2024-04-01T01:50:07.987 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pool_ids", 2024-04-01T01:50:07.987 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.987 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.987 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.987 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.987 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.987 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.987 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.987 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-01T01:50:07.987 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.987 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.987 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.988 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-04-01T01:50:07.988 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-04-01T01:50:07.988 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.988 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.988 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.988 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-04-01T01:50:07.988 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.988 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.988 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.988 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-01T01:50:07.988 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.988 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.988 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.988 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.989 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap_max_deviation": { 2024-04-01T01:50:07.989 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "upmap_max_deviation", 2024-04-01T01:50:07.989 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.989 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.989 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.989 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-04-01T01:50:07.989 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-04-01T01:50:07.989 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.989 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.989 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-01T01:50:07.989 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-01T01:50:07.989 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.989 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.989 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.989 INFO:teuthology.orchestra.run.smithi070.stdout: "upmap_max_optimizations": { 2024-04-01T01:50:07.990 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "upmap_max_optimizations", 2024-04-01T01:50:07.990 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-04-01T01:50:07.990 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.990 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:07.990 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:07.990 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.990 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.990 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.990 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-01T01:50:07.990 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.990 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.990 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.990 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.990 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:07.991 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.991 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:07.991 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cephadm", 2024-04-01T01:50:07.991 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:07.991 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:07.991 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:07.991 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_down_multiplier": { 2024-04-01T01:50:07.991 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_down_multiplier", 2024-04-01T01:50:07.991 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.991 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.991 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.991 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3.0", 2024-04-01T01:50:07.991 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.991 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.991 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.992 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-01T01:50:07.992 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.992 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.992 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.992 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.992 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_refresh_rate": { 2024-04-01T01:50:07.992 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_refresh_rate", 2024-04-01T01:50:07.992 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.992 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.992 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.992 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "20", 2024-04-01T01:50:07.992 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.992 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.992 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.993 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-01T01:50:07.993 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.993 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.993 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.993 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.993 INFO:teuthology.orchestra.run.smithi070.stdout: "agent_starting_port": { 2024-04-01T01:50:07.993 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "agent_starting_port", 2024-04-01T01:50:07.993 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:07.993 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.993 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.993 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "4721", 2024-04-01T01:50:07.993 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.993 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.993 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.994 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-01T01:50:07.994 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.994 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.994 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.994 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.994 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_ptrace": { 2024-04-01T01:50:07.994 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_ptrace", 2024-04-01T01:50:07.994 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.994 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.994 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.994 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:07.994 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.994 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-01T01:50:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-01T01:50:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "autotune_interval": { 2024-04-01T01:50:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "autotune_interval", 2024-04-01T01:50:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-04-01T01:50:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.995 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-01T01:50:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "autotune_memory_target_ratio": { 2024-04-01T01:50:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "autotune_memory_target_ratio", 2024-04-01T01:50:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0.7", 2024-04-01T01:50:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.996 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-01T01:50:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm_log_destination": { 2024-04-01T01:50:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cephadm_log_destination", 2024-04-01T01:50:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:07.997 INFO:teuthology.orchestra.run.smithi070.stdout: "file", 2024-04-01T01:50:07.998 INFO:teuthology.orchestra.run.smithi070.stdout: "file,syslog", 2024-04-01T01:50:07.998 INFO:teuthology.orchestra.run.smithi070.stdout: "syslog" 2024-04-01T01:50:07.998 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:07.998 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-01T01:50:07.998 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.998 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.998 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.998 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.998 INFO:teuthology.orchestra.run.smithi070.stdout: "cgroups_split": { 2024-04-01T01:50:07.998 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cgroups_split", 2024-04-01T01:50:07.998 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.998 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.998 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:07.999 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:07.999 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:07.999 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:07.999 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:07.999 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-01T01:50:07.999 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:07.999 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:07.999 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:07.999 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:07.999 INFO:teuthology.orchestra.run.smithi070.stdout: "config_checks_enabled": { 2024-04-01T01:50:07.999 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_checks_enabled", 2024-04-01T01:50:07.999 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:07.999 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:07.999 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.000 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.000 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.000 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.000 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.000 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-01T01:50:08.000 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.000 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.000 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.000 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.000 INFO:teuthology.orchestra.run.smithi070.stdout: "config_dashboard": { 2024-04-01T01:50:08.000 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_dashboard", 2024-04-01T01:50:08.000 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.000 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.001 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.001 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.001 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.001 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.001 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.001 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-01T01:50:08.001 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.001 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.001 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.001 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.001 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_alertmanager": { 2024-04-01T01:50:08.001 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_alertmanager", 2024-04-01T01:50:08.001 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.001 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.002 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.002 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-01T01:50:08.002 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.002 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.002 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.002 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-04-01T01:50:08.002 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.002 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.002 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.002 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.002 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_base": { 2024-04-01T01:50:08.002 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_base", 2024-04-01T01:50:08.002 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.002 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.003 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.003 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-01T01:50:08.003 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.003 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.003 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.003 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Container image name, without the tag", 2024-04-01T01:50:08.003 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.003 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.003 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.003 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.003 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_elasticsearch": { 2024-04-01T01:50:08.003 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_elasticsearch", 2024-04-01T01:50:08.003 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.003 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.004 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.004 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-01T01:50:08.004 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.004 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.004 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.004 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "elasticsearch container image", 2024-04-01T01:50:08.004 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.004 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.004 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.004 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.004 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_grafana": { 2024-04-01T01:50:08.004 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_grafana", 2024-04-01T01:50:08.004 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.004 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.005 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.005 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-01T01:50:08.005 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.005 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.005 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.005 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-04-01T01:50:08.005 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.005 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.006 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.006 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.006 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_haproxy": { 2024-04-01T01:50:08.006 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_haproxy", 2024-04-01T01:50:08.006 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.006 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.006 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.006 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-01T01:50:08.007 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.007 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.007 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.007 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "HAproxy container image", 2024-04-01T01:50:08.007 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.007 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.007 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.007 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.007 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_agent": { 2024-04-01T01:50:08.007 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_agent", 2024-04-01T01:50:08.007 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.007 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.007 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.008 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-01T01:50:08.008 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.008 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.008 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.008 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger agent container image", 2024-04-01T01:50:08.008 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.008 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.008 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.008 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.008 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_collector": { 2024-04-01T01:50:08.008 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_collector", 2024-04-01T01:50:08.008 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.008 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.008 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-01T01:50:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger collector container image", 2024-04-01T01:50:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_jaeger_query": { 2024-04-01T01:50:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_jaeger_query", 2024-04-01T01:50:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.009 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-01T01:50:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Jaeger query container image", 2024-04-01T01:50:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_keepalived": { 2024-04-01T01:50:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_keepalived", 2024-04-01T01:50:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.010 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.011 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.011 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-01T01:50:08.011 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.011 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.011 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.011 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Keepalived container image", 2024-04-01T01:50:08.011 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.011 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.011 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.011 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.011 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_loki": { 2024-04-01T01:50:08.011 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_loki", 2024-04-01T01:50:08.011 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.011 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.012 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.012 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-01T01:50:08.012 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.012 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.012 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.012 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Loki container image", 2024-04-01T01:50:08.012 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.012 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.012 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.012 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.012 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_node_exporter": { 2024-04-01T01:50:08.012 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_node_exporter", 2024-04-01T01:50:08.012 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.012 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.013 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.013 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-01T01:50:08.013 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.013 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.013 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.013 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-04-01T01:50:08.013 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.013 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.013 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.013 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.013 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_nvmeof": { 2024-04-01T01:50:08.013 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_nvmeof", 2024-04-01T01:50:08.013 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.013 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.014 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.014 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-04-01T01:50:08.014 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.014 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.014 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.014 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Nvme-of container image", 2024-04-01T01:50:08.014 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.014 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.014 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.014 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.014 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_prometheus": { 2024-04-01T01:50:08.014 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_prometheus", 2024-04-01T01:50:08.014 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.014 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.015 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.015 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-01T01:50:08.015 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.015 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.015 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.015 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Prometheus container image", 2024-04-01T01:50:08.015 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.015 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.015 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.015 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.015 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_promtail": { 2024-04-01T01:50:08.015 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_promtail", 2024-04-01T01:50:08.015 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.016 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.016 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.016 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-01T01:50:08.016 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.016 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.016 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.016 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Promtail container image", 2024-04-01T01:50:08.016 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.016 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.016 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.016 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.016 INFO:teuthology.orchestra.run.smithi070.stdout: "container_image_snmp_gateway": { 2024-04-01T01:50:08.016 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_image_snmp_gateway", 2024-04-01T01:50:08.016 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.016 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.017 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.017 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-01T01:50:08.017 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.017 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.017 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.017 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "SNMP Gateway container image", 2024-04-01T01:50:08.017 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.017 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.017 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.017 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.017 INFO:teuthology.orchestra.run.smithi070.stdout: "container_init": { 2024-04-01T01:50:08.017 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "container_init", 2024-04-01T01:50:08.017 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.017 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.018 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.018 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.018 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.018 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.018 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.018 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Run podman/docker with `--init`", 2024-04-01T01:50:08.018 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.018 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.018 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.018 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.018 INFO:teuthology.orchestra.run.smithi070.stdout: "daemon_cache_timeout": { 2024-04-01T01:50:08.018 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "daemon_cache_timeout", 2024-04-01T01:50:08.018 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:08.019 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.019 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.019 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-04-01T01:50:08.019 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.019 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.019 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.019 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-01T01:50:08.019 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.019 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.019 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.019 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.019 INFO:teuthology.orchestra.run.smithi070.stdout: "default_cephadm_command_timeout": { 2024-04-01T01:50:08.019 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "default_cephadm_command_timeout", 2024-04-01T01:50:08.019 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.019 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.020 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.020 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "900", 2024-04-01T01:50:08.020 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.020 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.020 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.020 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-01T01:50:08.020 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.020 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.020 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.020 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.020 INFO:teuthology.orchestra.run.smithi070.stdout: "default_registry": { 2024-04-01T01:50:08.020 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "default_registry", 2024-04-01T01:50:08.020 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.021 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.021 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.021 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "docker.io", 2024-04-01T01:50:08.021 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.021 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.021 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.021 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-01T01:50:08.021 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.021 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.021 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.021 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.021 INFO:teuthology.orchestra.run.smithi070.stdout: "device_cache_timeout": { 2024-04-01T01:50:08.021 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_cache_timeout", 2024-04-01T01:50:08.021 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:08.022 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.022 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.022 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1800", 2024-04-01T01:50:08.022 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.022 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.022 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.022 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache device inventory", 2024-04-01T01:50:08.022 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.022 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.022 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.022 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.022 INFO:teuthology.orchestra.run.smithi070.stdout: "device_enhanced_scan": { 2024-04-01T01:50:08.022 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_enhanced_scan", 2024-04-01T01:50:08.023 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.023 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.023 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.023 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.023 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.023 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.023 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.023 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-01T01:50:08.023 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.023 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.023 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.023 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.023 INFO:teuthology.orchestra.run.smithi070.stdout: "facts_cache_timeout": { 2024-04-01T01:50:08.023 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "facts_cache_timeout", 2024-04-01T01:50:08.024 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:08.024 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.024 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.024 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-04-01T01:50:08.024 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.024 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.024 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.024 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "seconds to cache host facts data", 2024-04-01T01:50:08.024 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.024 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.024 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.024 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.024 INFO:teuthology.orchestra.run.smithi070.stdout: "grafana_dashboards_path": { 2024-04-01T01:50:08.024 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "grafana_dashboards_path", 2024-04-01T01:50:08.024 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.025 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.025 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.025 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-01T01:50:08.025 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.025 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.025 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.025 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-01T01:50:08.025 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.025 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.025 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.025 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.025 INFO:teuthology.orchestra.run.smithi070.stdout: "host_check_interval": { 2024-04-01T01:50:08.025 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "host_check_interval", 2024-04-01T01:50:08.026 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:08.026 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.026 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.026 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-04-01T01:50:08.026 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.026 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.026 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.026 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to perform a host check", 2024-04-01T01:50:08.026 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.026 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.026 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.026 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.026 INFO:teuthology.orchestra.run.smithi070.stdout: "hw_monitoring": { 2024-04-01T01:50:08.026 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "hw_monitoring", 2024-04-01T01:50:08.027 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.027 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.027 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.027 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.027 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.027 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.027 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.027 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-01T01:50:08.027 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.027 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.027 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.027 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.027 INFO:teuthology.orchestra.run.smithi070.stdout: "inventory_list_all": { 2024-04-01T01:50:08.028 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "inventory_list_all", 2024-04-01T01:50:08.028 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.028 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.028 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.028 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.028 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.028 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.028 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.028 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-01T01:50:08.028 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.028 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.028 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.028 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.028 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.029 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.029 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.029 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.029 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.029 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.029 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.029 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.029 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.029 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.029 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.029 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.029 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.029 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.029 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.029 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.030 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.030 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.030 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.030 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.030 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.030 INFO:teuthology.orchestra.run.smithi070.stdout: "log_refresh_metadata": { 2024-04-01T01:50:08.030 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_refresh_metadata", 2024-04-01T01:50:08.030 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.030 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.030 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.030 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.030 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.030 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.031 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.031 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-01T01:50:08.031 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.031 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.031 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.031 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.031 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.031 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.031 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.031 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.031 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.031 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.031 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.031 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.032 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.032 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-01T01:50:08.032 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.032 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.032 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.032 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.032 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.032 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.032 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.032 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.032 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.032 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.032 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.032 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.032 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.033 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.033 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.033 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.033 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.033 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.033 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.033 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.033 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.033 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.033 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.033 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.033 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.033 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.033 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.033 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.034 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.034 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.034 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.034 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.034 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.034 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.034 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.034 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.034 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.034 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.034 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.034 INFO:teuthology.orchestra.run.smithi070.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-01T01:50:08.034 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-01T01:50:08.035 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.035 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.035 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.035 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.035 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.035 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.035 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.035 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-01T01:50:08.035 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.035 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.035 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.035 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.035 INFO:teuthology.orchestra.run.smithi070.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-01T01:50:08.035 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-01T01:50:08.036 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.036 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.036 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.036 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "*", 2024-04-01T01:50:08.036 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.036 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.036 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.036 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-01T01:50:08.036 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.036 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.036 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.036 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.036 INFO:teuthology.orchestra.run.smithi070.stdout: "max_count_per_host": { 2024-04-01T01:50:08.036 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_count_per_host", 2024-04-01T01:50:08.037 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.037 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.037 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.037 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:08.037 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.037 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.037 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.037 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "max number of daemons per service per host", 2024-04-01T01:50:08.037 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.037 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.037 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.037 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.037 INFO:teuthology.orchestra.run.smithi070.stdout: "max_osd_draining_count": { 2024-04-01T01:50:08.037 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_osd_draining_count", 2024-04-01T01:50:08.038 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.038 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.038 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.038 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:08.038 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.038 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.038 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.038 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-01T01:50:08.038 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.038 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.038 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.038 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.038 INFO:teuthology.orchestra.run.smithi070.stdout: "migration_current": { 2024-04-01T01:50:08.038 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "migration_current", 2024-04-01T01:50:08.039 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.039 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.039 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.039 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.039 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.039 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.039 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.039 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "internal - do not modify", 2024-04-01T01:50:08.039 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.039 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.039 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.039 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.039 INFO:teuthology.orchestra.run.smithi070.stdout: "mode": { 2024-04-01T01:50:08.039 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mode", 2024-04-01T01:50:08.040 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.040 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.040 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.040 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "root", 2024-04-01T01:50:08.040 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.040 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.040 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.040 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm-package", 2024-04-01T01:50:08.040 INFO:teuthology.orchestra.run.smithi070.stdout: "root" 2024-04-01T01:50:08.040 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.040 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "mode for remote execution of cephadm", 2024-04-01T01:50:08.040 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.040 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.040 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.041 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.041 INFO:teuthology.orchestra.run.smithi070.stdout: "oob_default_addr": { 2024-04-01T01:50:08.041 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "oob_default_addr", 2024-04-01T01:50:08.041 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.041 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.041 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.041 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "169.254.1.1", 2024-04-01T01:50:08.041 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.041 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.041 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.041 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-01T01:50:08.041 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.041 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.041 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.042 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.042 INFO:teuthology.orchestra.run.smithi070.stdout: "prometheus_alerts_path": { 2024-04-01T01:50:08.042 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prometheus_alerts_path", 2024-04-01T01:50:08.042 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.042 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.042 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.042 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-01T01:50:08.042 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.042 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.042 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.042 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-01T01:50:08.042 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.042 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.042 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.043 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.043 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_insecure": { 2024-04-01T01:50:08.043 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_insecure", 2024-04-01T01:50:08.043 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.043 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.043 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.043 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.043 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.043 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.043 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.043 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-01T01:50:08.043 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.043 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.043 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_password": { 2024-04-01T01:50:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_password", 2024-04-01T01:50:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-01T01:50:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.044 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.045 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.045 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_url": { 2024-04-01T01:50:08.045 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_url", 2024-04-01T01:50:08.045 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.045 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.045 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.045 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.045 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.045 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.045 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.045 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-01T01:50:08.045 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.045 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.045 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.046 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.046 INFO:teuthology.orchestra.run.smithi070.stdout: "registry_username": { 2024-04-01T01:50:08.046 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "registry_username", 2024-04-01T01:50:08.046 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.046 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.046 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.046 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.046 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.046 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.046 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.046 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-01T01:50:08.046 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.046 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.047 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.047 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.047 INFO:teuthology.orchestra.run.smithi070.stdout: "secure_monitoring_stack": { 2024-04-01T01:50:08.047 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "secure_monitoring_stack", 2024-04-01T01:50:08.047 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.047 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.047 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.047 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.047 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.047 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.047 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.047 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-01T01:50:08.047 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.047 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.048 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.048 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.048 INFO:teuthology.orchestra.run.smithi070.stdout: "service_discovery_port": { 2024-04-01T01:50:08.048 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "service_discovery_port", 2024-04-01T01:50:08.048 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.048 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.048 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.048 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8765", 2024-04-01T01:50:08.048 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.048 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.048 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.048 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "cephadm service discovery port", 2024-04-01T01:50:08.048 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.048 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.049 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.049 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.049 INFO:teuthology.orchestra.run.smithi070.stdout: "ssh_config_file": { 2024-04-01T01:50:08.049 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssh_config_file", 2024-04-01T01:50:08.049 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.049 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.049 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.049 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.049 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.049 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.049 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.049 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-01T01:50:08.049 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.049 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.049 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.050 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.050 INFO:teuthology.orchestra.run.smithi070.stdout: "use_agent": { 2024-04-01T01:50:08.050 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "use_agent", 2024-04-01T01:50:08.050 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.050 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.050 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.050 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.050 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.050 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.050 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.050 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-01T01:50:08.050 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.050 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.050 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.051 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.051 INFO:teuthology.orchestra.run.smithi070.stdout: "use_repo_digest": { 2024-04-01T01:50:08.051 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "use_repo_digest", 2024-04-01T01:50:08.051 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.051 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.051 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.051 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.051 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.051 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.051 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.051 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-01T01:50:08.051 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.052 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.052 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.052 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.052 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_failed_host_check": { 2024-04-01T01:50:08.052 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_failed_host_check", 2024-04-01T01:50:08.052 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.052 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.052 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.052 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.052 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.052 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.052 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.052 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if the host check fails", 2024-04-01T01:50:08.052 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.053 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.053 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.053 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.053 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_stray_daemons": { 2024-04-01T01:50:08.053 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_stray_daemons", 2024-04-01T01:50:08.053 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.053 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.053 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.053 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.053 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.053 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.053 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.053 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-01T01:50:08.053 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.054 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.054 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.054 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.054 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_on_stray_hosts": { 2024-04-01T01:50:08.054 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_on_stray_hosts", 2024-04-01T01:50:08.054 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.054 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.054 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.054 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.054 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.054 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.054 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.054 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-01T01:50:08.055 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.055 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.055 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.055 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.055 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.055 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.055 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.055 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crash", 2024-04-01T01:50:08.055 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.055 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.055 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.055 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.055 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.055 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.056 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.056 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.056 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.056 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.056 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.056 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.056 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.056 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.056 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.056 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.056 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.056 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.056 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.056 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.057 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.057 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.057 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.057 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.057 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.057 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.057 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.057 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.057 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.057 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.057 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.057 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.057 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.057 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.058 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.058 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.058 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.058 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.058 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.058 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.058 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.058 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.058 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.058 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.058 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.058 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.058 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.058 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.059 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.059 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.059 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.059 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.059 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.059 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.059 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.059 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.059 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.059 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.059 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.059 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.059 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.060 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.060 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.060 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.060 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.060 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.060 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.060 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.060 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.060 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.060 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.060 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.060 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.060 INFO:teuthology.orchestra.run.smithi070.stdout: "retain_interval": { 2024-04-01T01:50:08.060 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "retain_interval", 2024-04-01T01:50:08.061 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:08.061 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.061 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.061 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "31536000", 2024-04-01T01:50:08.061 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.061 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.061 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.061 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-01T01:50:08.061 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.061 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.061 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.061 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.061 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_recent_interval": { 2024-04-01T01:50:08.061 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_recent_interval", 2024-04-01T01:50:08.062 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:08.062 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.062 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.062 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1209600", 2024-04-01T01:50:08.062 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.062 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.062 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.062 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-01T01:50:08.062 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.062 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.062 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.062 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.062 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.063 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.063 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.063 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "dashboard", 2024-04-01T01:50:08.063 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.063 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.063 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.063 INFO:teuthology.orchestra.run.smithi070.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-01T01:50:08.063 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-01T01:50:08.063 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.063 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.063 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.063 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:08.063 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.063 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.063 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.064 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.064 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.064 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.064 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.064 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.064 INFO:teuthology.orchestra.run.smithi070.stdout: "ALERTMANAGER_API_HOST": { 2024-04-01T01:50:08.064 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-01T01:50:08.064 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.064 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.064 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.064 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.064 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.064 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.064 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.064 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.065 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.065 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.065 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.065 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.065 INFO:teuthology.orchestra.run.smithi070.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-01T01:50:08.065 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-01T01:50:08.065 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.065 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.065 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.065 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.065 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.065 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.065 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.066 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.066 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.066 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.066 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.066 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.066 INFO:teuthology.orchestra.run.smithi070.stdout: "AUDIT_API_ENABLED": { 2024-04-01T01:50:08.066 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "AUDIT_API_ENABLED", 2024-04-01T01:50:08.066 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.066 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.066 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.066 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.066 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.066 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.067 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.067 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.067 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.067 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.067 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.067 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.067 INFO:teuthology.orchestra.run.smithi070.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-01T01:50:08.067 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-01T01:50:08.067 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.067 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.067 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.067 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.067 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.068 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.068 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.068 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.068 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.068 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.068 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.068 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.068 INFO:teuthology.orchestra.run.smithi070.stdout: "ENABLE_BROWSABLE_API": { 2024-04-01T01:50:08.068 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-01T01:50:08.068 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.068 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.068 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.068 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.068 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.069 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.069 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.069 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.069 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.069 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.069 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.069 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.069 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-01T01:50:08.069 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-01T01:50:08.069 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.069 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.069 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.069 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.069 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.070 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.070 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.070 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.070 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.070 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.070 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.070 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.070 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-01T01:50:08.070 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-01T01:50:08.070 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.070 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.070 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.071 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.071 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.071 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.071 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.071 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.071 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.071 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.071 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.071 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.071 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-01T01:50:08.071 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-01T01:50:08.071 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.071 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.071 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.072 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.072 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.072 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.072 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.072 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.072 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.072 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.072 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.072 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.072 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-01T01:50:08.072 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-01T01:50:08.072 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.072 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-01T01:50:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-01T01:50:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.073 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-01T01:50:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-01T01:50:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.074 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.075 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.075 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.075 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.075 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.075 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.075 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.075 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.075 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.075 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.075 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.075 INFO:teuthology.orchestra.run.smithi070.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-01T01:50:08.075 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-01T01:50:08.075 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.075 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.075 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.076 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.076 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.076 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.076 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.076 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.076 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.076 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.076 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.076 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.076 INFO:teuthology.orchestra.run.smithi070.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-01T01:50:08.076 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-01T01:50:08.076 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.076 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.076 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.077 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.077 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.077 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.077 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.077 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.077 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.077 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.077 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.077 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.077 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_PASSWORD": { 2024-04-01T01:50:08.077 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-01T01:50:08.077 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.077 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.077 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.078 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-04-01T01:50:08.078 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.078 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.078 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.078 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.078 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.078 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.078 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.078 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.078 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-01T01:50:08.078 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-01T01:50:08.078 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.078 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.078 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.079 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.079 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.079 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.079 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.079 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.079 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.079 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.079 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.079 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.079 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_URL": { 2024-04-01T01:50:08.079 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_URL", 2024-04-01T01:50:08.079 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.079 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.079 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.080 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.080 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.080 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.080 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.080 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.080 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.080 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.080 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.080 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.080 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_API_USERNAME": { 2024-04-01T01:50:08.080 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-01T01:50:08.080 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.080 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.081 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.081 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-04-01T01:50:08.081 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.081 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.081 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.081 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.081 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.081 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.081 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.081 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.081 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-01T01:50:08.081 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-01T01:50:08.081 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.081 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.082 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.082 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.082 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.082 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.082 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.082 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.082 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.082 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.082 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.082 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.082 INFO:teuthology.orchestra.run.smithi070.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-01T01:50:08.082 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-01T01:50:08.082 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.083 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.083 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.083 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.083 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.083 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.083 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.083 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.083 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.083 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.083 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.083 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.083 INFO:teuthology.orchestra.run.smithi070.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-01T01:50:08.083 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-01T01:50:08.083 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.083 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.084 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.084 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.084 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.084 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.084 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.084 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.084 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.084 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.084 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.084 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.084 INFO:teuthology.orchestra.run.smithi070.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-01T01:50:08.084 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-01T01:50:08.084 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.085 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.085 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.085 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.085 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.085 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.085 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.085 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.085 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.085 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.085 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.085 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.085 INFO:teuthology.orchestra.run.smithi070.stdout: "PROMETHEUS_API_HOST": { 2024-04-01T01:50:08.085 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-01T01:50:08.085 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.086 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.086 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.086 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.086 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.086 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.086 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.086 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.086 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.086 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.086 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.086 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.086 INFO:teuthology.orchestra.run.smithi070.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-01T01:50:08.086 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-01T01:50:08.086 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.087 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.087 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.087 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.087 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.087 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.087 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.087 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.087 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.087 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.087 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.087 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.087 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-01T01:50:08.087 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-01T01:50:08.087 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.088 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.088 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.088 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.088 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.088 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.088 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.088 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.088 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.088 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.088 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.088 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.088 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-01T01:50:08.088 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-01T01:50:08.088 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.089 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.089 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.089 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.089 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.089 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.089 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.089 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.089 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.089 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.089 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.089 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.089 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-01T01:50:08.089 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-01T01:50:08.089 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.090 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.090 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.090 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.090 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.090 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.090 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.090 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.090 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.090 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.090 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.090 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.090 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-01T01:50:08.090 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-01T01:50:08.090 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.091 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.091 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.091 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.091 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.091 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.091 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.091 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.091 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.091 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.091 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.091 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.091 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-01T01:50:08.091 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-01T01:50:08.092 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.092 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.092 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.092 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.092 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.092 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.092 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.092 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.092 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.092 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.092 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.092 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.092 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-01T01:50:08.092 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-01T01:50:08.092 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.093 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.093 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.093 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.093 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.093 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.093 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.093 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.093 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.093 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.093 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.093 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.093 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-01T01:50:08.093 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-01T01:50:08.093 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.094 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.094 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.094 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.094 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.094 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.094 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.094 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.094 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.094 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.094 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.094 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.094 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_ENABLED": { 2024-04-01T01:50:08.094 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-01T01:50:08.095 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.095 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.095 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.095 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.095 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.095 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.095 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.095 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.095 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.095 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.095 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.095 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.095 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-01T01:50:08.095 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-01T01:50:08.096 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.096 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.096 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.096 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-01T01:50:08.096 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.096 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.096 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.096 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.096 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.096 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.096 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.096 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.096 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-01T01:50:08.096 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-01T01:50:08.097 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.097 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.097 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.097 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:08.097 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.097 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.097 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.097 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.097 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.097 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.097 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.097 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.097 INFO:teuthology.orchestra.run.smithi070.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-01T01:50:08.097 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-01T01:50:08.098 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.098 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.098 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.098 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8", 2024-04-01T01:50:08.098 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.098 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.098 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.098 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.098 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.098 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.098 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.098 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.098 INFO:teuthology.orchestra.run.smithi070.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-01T01:50:08.098 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-01T01:50:08.099 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.099 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.099 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.099 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "45", 2024-04-01T01:50:08.099 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.099 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.099 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.099 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.099 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.099 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.099 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.099 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.099 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_ACCESS_KEY": { 2024-04-01T01:50:08.100 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-01T01:50:08.100 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.100 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.100 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.100 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.100 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.100 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.100 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.100 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.100 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.100 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.100 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.100 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.100 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-01T01:50:08.101 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-01T01:50:08.101 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.101 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.101 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.101 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "admin", 2024-04-01T01:50:08.101 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.101 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.101 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.101 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.101 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.101 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.101 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.101 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.101 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_SECRET_KEY": { 2024-04-01T01:50:08.102 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-01T01:50:08.102 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.102 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.102 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.102 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.102 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.102 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.102 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.102 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.102 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.102 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.102 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.102 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.102 INFO:teuthology.orchestra.run.smithi070.stdout: "RGW_API_SSL_VERIFY": { 2024-04-01T01:50:08.103 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-01T01:50:08.103 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.103 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.103 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.103 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.103 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.103 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.103 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.103 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.103 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.103 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.103 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.103 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.103 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-01T01:50:08.104 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-01T01:50:08.104 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.104 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.104 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.104 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-04-01T01:50:08.104 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.104 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.104 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.104 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.104 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.104 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.104 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.104 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.104 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-01T01:50:08.105 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-01T01:50:08.105 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.105 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.105 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.105 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:08.105 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.105 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.105 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.105 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.105 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.105 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.105 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.105 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.105 INFO:teuthology.orchestra.run.smithi070.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-01T01:50:08.106 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-01T01:50:08.106 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.106 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.106 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.106 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-04-01T01:50:08.106 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.106 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.106 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.106 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.106 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.106 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.106 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.106 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.106 INFO:teuthology.orchestra.run.smithi070.stdout: "cross_origin_url": { 2024-04-01T01:50:08.107 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cross_origin_url", 2024-04-01T01:50:08.107 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.107 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.107 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.107 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.107 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.107 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.107 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.107 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.107 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.107 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.107 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.108 INFO:teuthology.orchestra.run.smithi070.stdout: "crt_file": { 2024-04-01T01:50:08.108 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "crt_file", 2024-04-01T01:50:08.108 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.108 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.108 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.108 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.108 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.108 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.108 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.108 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.108 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.108 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.108 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.109 INFO:teuthology.orchestra.run.smithi070.stdout: "debug": { 2024-04-01T01:50:08.109 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "debug", 2024-04-01T01:50:08.109 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.109 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.109 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.109 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.109 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.109 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.109 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.109 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Enable/disable debug options", 2024-04-01T01:50:08.109 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.109 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.109 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.109 INFO:teuthology.orchestra.run.smithi070.stdout: "jwt_token_ttl": { 2024-04-01T01:50:08.110 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "jwt_token_ttl", 2024-04-01T01:50:08.110 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.110 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.110 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.110 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "28800", 2024-04-01T01:50:08.110 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.110 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.110 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.110 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.110 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.110 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.110 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.110 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.110 INFO:teuthology.orchestra.run.smithi070.stdout: "key_file": { 2024-04-01T01:50:08.111 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "key_file", 2024-04-01T01:50:08.111 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.111 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.111 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.111 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.111 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.111 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.111 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.111 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.111 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.111 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.111 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.111 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.111 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.112 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.112 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.112 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.112 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.112 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.112 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.112 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.112 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.112 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.112 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.112 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.112 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.112 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.112 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.113 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.113 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.113 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.113 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.113 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.113 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.113 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.113 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.113 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.113 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.113 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.113 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.113 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.113 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.113 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.114 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.114 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.114 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.114 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.114 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.114 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.114 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.114 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.114 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.114 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.114 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.114 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.114 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.115 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.115 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.115 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.115 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.115 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.115 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.115 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.115 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.115 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.115 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.115 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.115 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.115 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.115 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.116 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.116 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.116 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.116 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.116 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.116 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.116 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.116 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.116 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.116 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.116 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.116 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.116 INFO:teuthology.orchestra.run.smithi070.stdout: "motd": { 2024-04-01T01:50:08.117 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "motd", 2024-04-01T01:50:08.117 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.117 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.117 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.117 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.117 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.117 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.117 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.117 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "The message of the day", 2024-04-01T01:50:08.117 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.117 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.117 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.117 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.117 INFO:teuthology.orchestra.run.smithi070.stdout: "redirect_resolve_ip_addr": { 2024-04-01T01:50:08.118 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "redirect_resolve_ip_addr", 2024-04-01T01:50:08.118 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.118 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.118 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.118 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.118 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.118 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.118 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.118 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.118 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.118 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.118 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.118 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.119 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-04-01T01:50:08.119 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-04-01T01:50:08.119 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.119 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.119 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.119 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "::", 2024-04-01T01:50:08.119 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.119 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.119 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.119 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.119 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.119 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.119 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.120 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-04-01T01:50:08.120 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-04-01T01:50:08.120 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.120 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.120 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.120 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8080", 2024-04-01T01:50:08.120 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.120 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.120 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.120 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.120 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.120 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.120 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.121 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl": { 2024-04-01T01:50:08.121 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl", 2024-04-01T01:50:08.121 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.121 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.121 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.121 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.121 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.121 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.121 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.121 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.121 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.121 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.121 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.122 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.122 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl_server_port": { 2024-04-01T01:50:08.122 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl_server_port", 2024-04-01T01:50:08.122 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.122 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.122 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.122 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8443", 2024-04-01T01:50:08.122 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.122 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.122 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.122 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.122 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.122 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.122 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.123 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_behaviour": { 2024-04-01T01:50:08.123 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_behaviour", 2024-04-01T01:50:08.123 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.123 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.123 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.123 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "redirect", 2024-04-01T01:50:08.123 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.123 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.123 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.123 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.123 INFO:teuthology.orchestra.run.smithi070.stdout: "redirect" 2024-04-01T01:50:08.123 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.123 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.124 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.124 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.124 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.124 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_error_status_code": { 2024-04-01T01:50:08.124 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_error_status_code", 2024-04-01T01:50:08.124 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.124 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.124 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.124 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "500", 2024-04-01T01:50:08.124 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "400", 2024-04-01T01:50:08.124 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "599", 2024-04-01T01:50:08.124 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.124 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.125 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.125 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.125 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.125 INFO:teuthology.orchestra.run.smithi070.stdout: "url_prefix": { 2024-04-01T01:50:08.125 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "url_prefix", 2024-04-01T01:50:08.125 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.125 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.125 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.125 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.125 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.125 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.125 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.125 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.125 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.126 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.126 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.126 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.126 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.126 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.126 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.126 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "devicehealth", 2024-04-01T01:50:08.126 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.126 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.126 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.126 INFO:teuthology.orchestra.run.smithi070.stdout: "enable_monitoring": { 2024-04-01T01:50:08.126 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enable_monitoring", 2024-04-01T01:50:08.126 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.126 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.127 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.127 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.127 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.127 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.127 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.127 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "monitor device health metrics", 2024-04-01T01:50:08.127 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.127 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.127 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.127 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.127 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.127 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.127 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.128 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.128 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.128 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.128 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.128 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.128 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.128 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.128 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.128 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.128 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.128 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.128 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.128 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.128 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.129 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.129 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.129 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.129 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.129 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.129 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.129 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.129 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.129 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.129 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.129 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.129 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.129 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.130 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.130 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.130 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.130 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.130 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.130 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.130 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.130 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.130 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.130 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.130 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.130 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.130 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.131 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.131 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.131 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.131 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.131 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.131 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.131 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.131 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.131 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.131 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.131 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.131 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.131 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.132 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.132 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.132 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.132 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.132 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.132 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.132 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.132 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.132 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.132 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.132 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.132 INFO:teuthology.orchestra.run.smithi070.stdout: "mark_out_threshold": { 2024-04-01T01:50:08.132 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mark_out_threshold", 2024-04-01T01:50:08.133 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:08.133 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.133 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.133 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "2419200", 2024-04-01T01:50:08.133 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.133 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.133 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.133 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-01T01:50:08.133 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.133 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.133 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.133 INFO:teuthology.orchestra.run.smithi070.stdout: "pool_name": { 2024-04-01T01:50:08.133 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pool_name", 2024-04-01T01:50:08.134 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.134 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.134 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.134 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "device_health_metrics", 2024-04-01T01:50:08.134 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.134 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.134 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.134 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-01T01:50:08.134 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.134 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.134 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.134 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.134 INFO:teuthology.orchestra.run.smithi070.stdout: "retention_period": { 2024-04-01T01:50:08.135 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "retention_period", 2024-04-01T01:50:08.135 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:08.135 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.135 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.135 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15552000", 2024-04-01T01:50:08.135 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.135 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.135 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.135 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long to retain device health metrics", 2024-04-01T01:50:08.135 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.135 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.135 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.135 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.135 INFO:teuthology.orchestra.run.smithi070.stdout: "scrape_frequency": { 2024-04-01T01:50:08.136 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "scrape_frequency", 2024-04-01T01:50:08.136 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:08.136 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.136 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.136 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "86400", 2024-04-01T01:50:08.136 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.136 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.136 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.136 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-01T01:50:08.136 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.136 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.136 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.136 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.136 INFO:teuthology.orchestra.run.smithi070.stdout: "self_heal": { 2024-04-01T01:50:08.136 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "self_heal", 2024-04-01T01:50:08.137 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.137 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.137 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.137 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.137 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.137 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.137 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.137 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-01T01:50:08.137 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.137 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.137 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.137 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-04-01T01:50:08.137 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-04-01T01:50:08.137 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:08.138 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.138 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.138 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-04-01T01:50:08.138 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.138 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.138 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.138 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how frequently to wake up and check device health", 2024-04-01T01:50:08.138 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.138 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.138 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.138 INFO:teuthology.orchestra.run.smithi070.stdout: "warn_threshold": { 2024-04-01T01:50:08.138 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "warn_threshold", 2024-04-01T01:50:08.139 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:08.139 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.139 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.139 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "7257600", 2024-04-01T01:50:08.139 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.139 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.139 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.139 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-01T01:50:08.139 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.139 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.139 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.139 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.139 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.139 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "diskprediction_local", 2024-04-01T01:50:08.140 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.140 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.140 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.140 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.140 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.140 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.140 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.140 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.140 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.140 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.140 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.140 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.140 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.140 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.140 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.141 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.141 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.141 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.141 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.141 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.141 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.141 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.141 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.141 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.141 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.141 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.141 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.141 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.141 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.142 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.142 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.142 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.142 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.142 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.142 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.142 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.142 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.142 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.142 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.142 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.142 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.143 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.143 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.143 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.143 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.143 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.143 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.143 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.143 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.143 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.143 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.143 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.143 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.143 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.143 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.144 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.144 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.144 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.144 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.144 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.144 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.144 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.144 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.144 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.144 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.144 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.144 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.144 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.145 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.145 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.145 INFO:teuthology.orchestra.run.smithi070.stdout: "predict_interval": { 2024-04-01T01:50:08.145 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "predict_interval", 2024-04-01T01:50:08.145 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.145 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.145 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.145 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "86400", 2024-04-01T01:50:08.145 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.145 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.145 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.145 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.146 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.146 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.146 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.146 INFO:teuthology.orchestra.run.smithi070.stdout: "predictor_model": { 2024-04-01T01:50:08.146 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "predictor_model", 2024-04-01T01:50:08.146 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.146 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.146 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.146 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "prophetstor", 2024-04-01T01:50:08.146 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.146 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.146 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.146 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.147 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.147 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.147 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.147 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-04-01T01:50:08.147 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-04-01T01:50:08.147 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.147 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.147 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.147 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "600", 2024-04-01T01:50:08.147 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.147 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.147 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.148 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.148 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.148 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.148 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.148 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.148 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.148 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.148 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.148 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "influx", 2024-04-01T01:50:08.148 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": false, 2024-04-01T01:50:08.148 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "influxdb python module not found", 2024-04-01T01:50:08.148 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.148 INFO:teuthology.orchestra.run.smithi070.stdout: "batch_size": { 2024-04-01T01:50:08.148 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "batch_size", 2024-04-01T01:50:08.148 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.149 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.149 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.149 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5000", 2024-04-01T01:50:08.149 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.149 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.149 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.149 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-01T01:50:08.149 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.149 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.149 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.150 INFO:teuthology.orchestra.run.smithi070.stdout: "database": { 2024-04-01T01:50:08.150 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "database", 2024-04-01T01:50:08.150 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.150 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.150 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.150 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "ceph", 2024-04-01T01:50:08.150 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.150 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.150 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.150 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-01T01:50:08.150 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.150 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.151 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.151 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.151 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": { 2024-04-01T01:50:08.151 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "hostname", 2024-04-01T01:50:08.151 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.151 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.151 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.151 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.151 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.151 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.151 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.151 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB server hostname", 2024-04-01T01:50:08.151 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.151 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.152 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.152 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.152 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-04-01T01:50:08.152 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-04-01T01:50:08.152 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:08.152 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.152 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.152 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "30", 2024-04-01T01:50:08.152 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "5", 2024-04-01T01:50:08.152 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.152 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.152 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-01T01:50:08.152 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.152 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.153 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.153 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.153 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.153 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.153 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.153 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.153 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.153 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.153 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.153 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.153 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.153 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.153 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.153 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.153 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.154 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.154 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.154 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.154 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.154 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.154 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.154 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.154 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.154 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.154 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.154 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.154 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.154 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.155 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.155 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.155 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.155 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.155 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.155 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.155 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.155 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.155 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.155 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.155 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.155 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.155 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.155 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.155 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.156 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.156 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.156 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.156 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.156 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.156 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.156 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.156 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.156 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.156 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.156 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.156 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.156 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.156 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.156 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.157 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.157 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.157 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.157 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.157 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.157 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.157 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.157 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.157 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.157 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.157 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.157 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.157 INFO:teuthology.orchestra.run.smithi070.stdout: "password": { 2024-04-01T01:50:08.158 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "password", 2024-04-01T01:50:08.158 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.158 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.158 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.158 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.158 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.158 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.158 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.158 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "password of InfluxDB server user", 2024-04-01T01:50:08.158 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.158 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.158 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.159 INFO:teuthology.orchestra.run.smithi070.stdout: "port": { 2024-04-01T01:50:08.159 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "port", 2024-04-01T01:50:08.159 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.159 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.159 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.159 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "8086", 2024-04-01T01:50:08.159 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.159 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.159 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.159 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "InfluxDB server port", 2024-04-01T01:50:08.159 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.159 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.159 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.160 INFO:teuthology.orchestra.run.smithi070.stdout: "ssl": { 2024-04-01T01:50:08.160 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ssl", 2024-04-01T01:50:08.160 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.160 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.160 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.160 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "false", 2024-04-01T01:50:08.160 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.160 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.160 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.160 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-01T01:50:08.160 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.160 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.160 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.161 INFO:teuthology.orchestra.run.smithi070.stdout: "threads": { 2024-04-01T01:50:08.161 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "threads", 2024-04-01T01:50:08.161 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.161 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.161 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.161 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-04-01T01:50:08.161 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-04-01T01:50:08.161 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "32", 2024-04-01T01:50:08.161 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.161 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-01T01:50:08.161 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.161 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.161 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.162 INFO:teuthology.orchestra.run.smithi070.stdout: "username": { 2024-04-01T01:50:08.162 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "username", 2024-04-01T01:50:08.162 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.162 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.162 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.162 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.162 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.162 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.162 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.162 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "username of InfluxDB server user", 2024-04-01T01:50:08.162 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.162 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.162 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.163 INFO:teuthology.orchestra.run.smithi070.stdout: "verify_ssl": { 2024-04-01T01:50:08.163 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "verify_ssl", 2024-04-01T01:50:08.163 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.163 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.163 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.163 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "true", 2024-04-01T01:50:08.163 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.163 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.163 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.163 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-01T01:50:08.163 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.163 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.163 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.163 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.164 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.164 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "insights", 2024-04-01T01:50:08.164 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.164 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.164 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.164 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.164 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.164 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.164 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.164 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.164 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.164 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.165 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.165 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.165 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.165 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.165 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.165 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.165 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.165 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.165 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.165 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.165 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.165 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.165 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.165 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.166 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.166 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.166 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.166 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.166 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.166 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.166 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.166 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.166 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.166 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.166 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.166 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.167 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.167 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.167 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.167 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.167 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.167 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.167 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.167 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.167 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.167 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.167 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.167 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.167 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.167 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.168 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.168 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.168 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.168 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.168 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.168 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.168 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.168 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.168 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.168 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.168 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.168 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.168 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.168 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.169 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.169 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.169 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.169 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.169 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.169 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.169 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.169 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.169 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.169 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "iostat", 2024-04-01T01:50:08.169 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.169 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.170 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.170 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.170 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.170 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.170 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.170 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.170 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.170 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.170 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.170 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.170 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.170 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.170 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.170 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.171 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.171 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.171 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.171 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.171 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.171 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.171 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.171 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.171 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.171 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.171 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.171 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.172 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.172 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.172 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.172 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.172 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.172 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.172 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.172 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.172 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.172 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.172 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.172 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.172 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.173 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.173 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.173 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.173 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.173 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.173 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.173 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.173 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.173 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.173 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.173 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.173 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.173 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.174 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.174 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.174 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.174 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.174 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.174 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.174 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.174 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.174 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.174 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.174 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.174 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.174 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.175 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.175 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.175 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.175 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.175 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.175 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.175 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "k8sevents", 2024-04-01T01:50:08.175 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.175 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.175 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.175 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_event_retention_days": { 2024-04-01T01:50:08.175 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "ceph_event_retention_days", 2024-04-01T01:50:08.175 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.175 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.175 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.175 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "7", 2024-04-01T01:50:08.176 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.176 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.176 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.176 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-01T01:50:08.176 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.176 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.176 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.176 INFO:teuthology.orchestra.run.smithi070.stdout: "config_check_secs": { 2024-04-01T01:50:08.176 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "config_check_secs", 2024-04-01T01:50:08.176 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.176 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.176 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.176 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:08.176 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "10", 2024-04-01T01:50:08.176 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.177 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.177 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-01T01:50:08.177 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.177 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.177 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.177 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.177 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.177 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.177 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.177 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.177 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.177 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.178 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.178 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.178 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.178 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.178 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.178 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.178 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.178 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.178 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.178 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.178 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.178 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.178 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.178 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.179 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.179 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.179 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.179 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.179 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.179 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.179 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.179 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.179 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.179 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.179 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.179 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.179 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.179 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.179 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.179 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.180 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.180 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.180 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.181 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.181 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.181 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.181 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.181 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.181 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.182 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.182 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.182 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.182 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.182 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.182 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.182 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.182 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.182 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.182 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.182 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.182 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.182 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.183 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.183 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.183 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.183 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.183 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.183 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.183 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.183 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.183 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.183 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.183 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "localpool", 2024-04-01T01:50:08.183 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.183 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.184 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.184 INFO:teuthology.orchestra.run.smithi070.stdout: "failure_domain": { 2024-04-01T01:50:08.184 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "failure_domain", 2024-04-01T01:50:08.184 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.184 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.184 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.184 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "host", 2024-04-01T01:50:08.184 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.184 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.184 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.184 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "failure domain for any created local pool", 2024-04-01T01:50:08.184 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-01T01:50:08.184 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.184 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.185 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.185 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.185 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.185 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.185 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.185 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.185 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.185 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.185 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.185 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.185 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.185 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.185 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.185 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.186 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.186 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.186 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.186 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.186 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.186 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.186 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.186 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.186 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.186 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.186 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.186 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.186 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.187 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.187 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.187 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.187 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.187 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.187 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.187 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.187 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.187 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.187 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.187 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.187 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.187 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.187 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.188 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.188 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.188 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.188 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.188 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.188 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.188 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.188 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.188 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.188 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.188 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.188 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.188 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.188 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.189 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.189 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.189 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.189 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.189 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.189 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.189 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.189 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.189 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.189 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.189 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.189 INFO:teuthology.orchestra.run.smithi070.stdout: "min_size": { 2024-04-01T01:50:08.189 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "min_size", 2024-04-01T01:50:08.189 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.189 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.190 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.190 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.190 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.190 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.190 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.190 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default min_size for any created local pool", 2024-04-01T01:50:08.190 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-01T01:50:08.190 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.190 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.190 INFO:teuthology.orchestra.run.smithi070.stdout: "num_rep": { 2024-04-01T01:50:08.190 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "num_rep", 2024-04-01T01:50:08.190 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.190 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.191 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.191 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3", 2024-04-01T01:50:08.191 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.191 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.191 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.191 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default replica count for any created local pool", 2024-04-01T01:50:08.191 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.191 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.191 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.191 INFO:teuthology.orchestra.run.smithi070.stdout: "pg_num": { 2024-04-01T01:50:08.191 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pg_num", 2024-04-01T01:50:08.191 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.191 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.192 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.192 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "128", 2024-04-01T01:50:08.192 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.192 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.192 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.192 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "default pg_num for any created local pool", 2024-04-01T01:50:08.192 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.192 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.192 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.192 INFO:teuthology.orchestra.run.smithi070.stdout: "prefix": { 2024-04-01T01:50:08.192 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prefix", 2024-04-01T01:50:08.192 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.192 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.192 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.193 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.193 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.193 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.193 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.193 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "name prefix for any created local pool", 2024-04-01T01:50:08.193 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.193 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.193 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.193 INFO:teuthology.orchestra.run.smithi070.stdout: "subtree": { 2024-04-01T01:50:08.193 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "subtree", 2024-04-01T01:50:08.193 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.193 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.194 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.194 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "rack", 2024-04-01T01:50:08.194 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.194 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.194 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.194 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-01T01:50:08.194 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-01T01:50:08.194 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.194 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.194 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.194 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.194 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mds_autoscaler", 2024-04-01T01:50:08.195 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.195 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.195 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.195 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.195 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.195 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.195 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.195 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.195 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.195 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.195 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.195 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.195 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.195 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.195 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.195 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.196 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.196 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.196 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.196 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.196 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.196 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.196 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.196 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.196 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.196 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.196 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.196 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.196 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.196 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.197 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.197 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.197 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.197 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.197 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.197 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.197 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.197 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.197 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.197 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.197 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.197 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.197 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.197 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.198 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.198 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.198 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.198 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.198 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.198 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.198 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.198 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.198 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.198 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.198 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.198 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.198 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.199 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.199 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.199 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.199 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.199 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.199 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.199 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.199 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.199 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.199 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.199 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.199 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.199 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.199 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.200 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.200 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mirroring", 2024-04-01T01:50:08.200 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.200 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.200 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.200 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.200 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.200 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.200 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.200 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.200 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.200 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.200 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.201 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.201 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.201 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.201 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.201 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.201 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.201 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.201 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.201 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.201 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.201 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.201 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.201 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.201 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.202 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.202 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.202 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.202 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.202 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.202 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.202 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.202 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.202 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.202 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.202 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.202 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.202 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.202 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.203 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.203 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.203 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.203 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.203 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.203 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.203 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.203 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.203 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.203 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.203 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.203 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.203 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.203 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.204 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.204 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.204 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.204 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.204 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.204 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.204 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.204 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.204 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.204 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.204 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.204 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.204 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.204 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.205 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.205 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.205 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.205 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.205 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.205 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.205 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.205 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.205 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "nfs", 2024-04-01T01:50:08.205 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.205 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.205 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.205 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.205 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.205 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.206 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.206 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.206 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.206 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.206 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.206 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.206 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.206 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.206 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.206 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.206 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.206 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.206 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.207 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.207 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.207 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.207 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.207 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.207 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.207 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.207 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.207 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.207 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.207 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.207 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.208 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.208 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.208 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.208 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.208 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.208 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.208 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.208 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.208 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.208 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.208 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.208 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.208 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.209 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.209 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.209 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.209 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.209 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.209 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.209 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.209 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.209 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.209 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.209 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.209 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.209 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.210 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.210 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.210 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.210 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.210 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.210 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.210 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.210 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.210 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.210 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.210 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.210 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.210 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.211 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.211 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "orchestrator", 2024-04-01T01:50:08.211 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.211 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.211 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.211 INFO:teuthology.orchestra.run.smithi070.stdout: "fail_fs": { 2024-04-01T01:50:08.211 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "fail_fs", 2024-04-01T01:50:08.211 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.211 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.211 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.211 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.211 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.211 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.211 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.212 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-01T01:50:08.212 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.212 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.212 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.212 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.212 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.212 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.212 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.212 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.212 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.212 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.212 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.212 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.213 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.213 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.213 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.213 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.213 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.213 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.213 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.213 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.213 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.213 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.213 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.213 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.213 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.214 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.214 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.214 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.214 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.214 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.214 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.214 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.214 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.214 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.214 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.214 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.214 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.214 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.215 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.215 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.215 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.215 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.215 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.215 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.215 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.215 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.215 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.215 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.215 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.215 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.215 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.215 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.215 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.216 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.216 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.216 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.216 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.216 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.216 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.216 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.216 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.216 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.216 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.216 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.216 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.216 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.217 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.217 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.217 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.217 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.217 INFO:teuthology.orchestra.run.smithi070.stdout: "orchestrator": { 2024-04-01T01:50:08.217 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "orchestrator", 2024-04-01T01:50:08.217 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.217 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.217 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.217 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.217 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.217 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.217 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.218 INFO:teuthology.orchestra.run.smithi070.stdout: "cephadm", 2024-04-01T01:50:08.218 INFO:teuthology.orchestra.run.smithi070.stdout: "rook", 2024-04-01T01:50:08.218 INFO:teuthology.orchestra.run.smithi070.stdout: "test_orchestrator" 2024-04-01T01:50:08.218 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.218 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Orchestrator backend", 2024-04-01T01:50:08.218 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.218 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.218 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.218 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.218 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.218 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.218 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.218 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "osd_perf_query", 2024-04-01T01:50:08.219 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.219 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.219 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.219 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.219 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.219 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.219 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.219 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.219 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.219 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.219 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.219 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.219 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.219 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.220 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.220 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.220 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.220 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.220 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.220 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.220 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.220 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.220 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.220 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.220 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.220 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.220 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.221 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.221 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.221 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.221 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.221 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.221 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.221 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.221 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.221 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.221 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.221 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.221 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.221 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.221 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.222 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.222 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.222 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.222 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.222 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.222 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.222 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.222 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.222 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.222 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.222 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.222 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.222 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.223 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "osd_support", 2024-04-01T01:50:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.224 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.225 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.225 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.225 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.225 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.225 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.225 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.225 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.225 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.225 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.225 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.225 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.225 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.225 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.225 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.226 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.226 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.226 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.226 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.226 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.226 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.226 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.226 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.226 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.226 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.226 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.226 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.226 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.226 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.227 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.227 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.227 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.227 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.227 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.227 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.227 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.227 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.227 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.227 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.227 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.227 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.227 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.227 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "pg_autoscaler", 2024-04-01T01:50:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-04-01T01:50:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-04-01T01:50:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-04-01T01:50:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "threshold": { 2024-04-01T01:50:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "threshold", 2024-04-01T01:50:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "3.0", 2024-04-01T01:50:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1.0", 2024-04-01T01:50:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "scaling threshold", 2024-04-01T01:50:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-01T01:50:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "progress", 2024-04-01T01:50:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_pg_recovery_event": { 2024-04-01T01:50:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_pg_recovery_event", 2024-04-01T01:50:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-01T01:50:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": { 2024-04-01T01:50:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enabled", 2024-04-01T01:50:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "max_completed_events": { 2024-04-01T01:50:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_completed_events", 2024-04-01T01:50:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "50", 2024-04-01T01:50:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "number of past completed events to remember", 2024-04-01T01:50:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "sleep_interval": { 2024-04-01T01:50:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "sleep_interval", 2024-04-01T01:50:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "5", 2024-04-01T01:50:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "how long the module is going to sleep", 2024-04-01T01:50:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "prometheus", 2024-04-01T01:50:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "cache": { 2024-04-01T01:50:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "cache", 2024-04-01T01:50:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "exclude_perf_counters": { 2024-04-01T01:50:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "exclude_perf_counters", 2024-04-01T01:50:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-01T01:50:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-01T01:50:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_stats_pools": { 2024-04-01T01:50:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_stats_pools", 2024-04-01T01:50:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-01T01:50:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-01T01:50:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "300", 2024-04-01T01:50:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "scrape_interval": { 2024-04-01T01:50:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "scrape_interval", 2024-04-01T01:50:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15.0", 2024-04-01T01:50:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-04-01T01:50:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-04-01T01:50:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "::", 2024-04-01T01:50:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-01T01:50:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-04-01T01:50:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-04-01T01:50:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "9283", 2024-04-01T01:50:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-01T01:50:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "stale_cache_strategy": { 2024-04-01T01:50:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "stale_cache_strategy", 2024-04-01T01:50:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "log", 2024-04-01T01:50:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_behaviour": { 2024-04-01T01:50:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_behaviour", 2024-04-01T01:50:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "default", 2024-04-01T01:50:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "default", 2024-04-01T01:50:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "error" 2024-04-01T01:50:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "standby_error_status_code": { 2024-04-01T01:50:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "standby_error_status_code", 2024-04-01T01:50:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "500", 2024-04-01T01:50:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "400", 2024-04-01T01:50:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "599", 2024-04-01T01:50:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_support", 2024-04-01T01:50:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "max_concurrent_snap_create": { 2024-04-01T01:50:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_concurrent_snap_create", 2024-04-01T01:50:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10", 2024-04-01T01:50:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "mirror_snapshot_schedule": { 2024-04-01T01:50:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "mirror_snapshot_schedule", 2024-04-01T01:50:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "trash_purge_schedule": { 2024-04-01T01:50:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "trash_purge_schedule", 2024-04-01T01:50:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "restful", 2024-04-01T01:50:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "enable_auth": { 2024-04-01T01:50:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enable_auth", 2024-04-01T01:50:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "key_file": { 2024-04-01T01:50:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "key_file", 2024-04-01T01:50:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "server_addr": { 2024-04-01T01:50:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_addr", 2024-04-01T01:50:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "server_port": { 2024-04-01T01:50:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "server_port", 2024-04-01T01:50:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rgw", 2024-04-01T01:50:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rook", 2024-04-01T01:50:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "storage_class": { 2024-04-01T01:50:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "storage_class", 2024-04-01T01:50:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "local", 2024-04-01T01:50:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-01T01:50:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "selftest", 2024-04-01T01:50:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "roption1": { 2024-04-01T01:50:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "roption1", 2024-04-01T01:50:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "roption2": { 2024-04-01T01:50:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "roption2", 2024-04-01T01:50:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "xyz", 2024-04-01T01:50:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption1": { 2024-04-01T01:50:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption1", 2024-04-01T01:50:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption2": { 2024-04-01T01:50:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption2", 2024-04-01T01:50:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption3": { 2024-04-01T01:50:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption3", 2024-04-01T01:50:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "float", 2024-04-01T01:50:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption4": { 2024-04-01T01:50:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption4", 2024-04-01T01:50:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption5": { 2024-04-01T01:50:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption5", 2024-04-01T01:50:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption6": { 2024-04-01T01:50:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption6", 2024-04-01T01:50:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "rwoption7": { 2024-04-01T01:50:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rwoption7", 2024-04-01T01:50:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "1", 2024-04-01T01:50:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "42", 2024-04-01T01:50:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: "testkey": { 2024-04-01T01:50:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testkey", 2024-04-01T01:50:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "testlkey": { 2024-04-01T01:50:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testlkey", 2024-04-01T01:50:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "testnewline": { 2024-04-01T01:50:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "testnewline", 2024-04-01T01:50:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snap_schedule", 2024-04-01T01:50:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "allow_m_granularity": { 2024-04-01T01:50:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "allow_m_granularity", 2024-04-01T01:50:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "allow minute scheduled snapshots", 2024-04-01T01:50:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "dump_on_update": { 2024-04-01T01:50:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "dump_on_update", 2024-04-01T01:50:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "dump database to debug log on update", 2024-04-01T01:50:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "stats", 2024-04-01T01:50:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "status", 2024-04-01T01:50:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "telegraf", 2024-04-01T01:50:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "address": { 2024-04-01T01:50:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "address", 2024-04-01T01:50:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-01T01:50:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-04-01T01:50:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-04-01T01:50:08.331 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:08.331 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.331 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.331 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "15", 2024-04-01T01:50:08.331 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.331 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.331 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.331 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.331 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.331 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.331 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.331 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.332 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.332 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.332 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.332 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.332 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.332 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.332 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.332 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.332 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.332 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.332 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.332 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.332 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.332 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.332 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.333 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.333 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.333 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.333 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.333 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.333 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.333 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.333 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.333 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.333 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.333 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.333 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.333 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.334 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.334 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.334 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.334 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.334 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.334 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.334 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.334 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.334 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.334 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.334 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.334 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.335 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.335 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.335 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.335 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.335 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.335 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.335 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.335 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.335 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.335 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.335 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.335 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.336 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.336 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.336 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.336 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.336 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.336 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.336 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.336 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.336 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.336 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.336 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.336 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.336 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.337 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.337 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.337 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "telemetry", 2024-04-01T01:50:08.337 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.337 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.337 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.337 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_basic": { 2024-04-01T01:50:08.337 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_basic", 2024-04-01T01:50:08.337 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.337 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.337 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.337 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.338 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.338 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.338 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.338 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-01T01:50:08.338 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.338 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.338 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.338 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_crash": { 2024-04-01T01:50:08.338 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_crash", 2024-04-01T01:50:08.338 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.338 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.338 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.338 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.339 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.339 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.339 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.339 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-01T01:50:08.339 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.339 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.339 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.339 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_device": { 2024-04-01T01:50:08.339 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_device", 2024-04-01T01:50:08.339 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.339 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.339 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.339 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.340 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.340 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.340 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.340 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-01T01:50:08.340 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.340 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.340 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.340 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_ident": { 2024-04-01T01:50:08.340 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_ident", 2024-04-01T01:50:08.340 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.340 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.340 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.340 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.341 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.341 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.341 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.341 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-01T01:50:08.341 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.341 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.341 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.341 INFO:teuthology.orchestra.run.smithi070.stdout: "channel_perf": { 2024-04-01T01:50:08.341 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "channel_perf", 2024-04-01T01:50:08.341 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.341 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.341 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.341 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.342 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.342 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.342 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.342 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-01T01:50:08.342 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.342 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.342 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.342 INFO:teuthology.orchestra.run.smithi070.stdout: "contact": { 2024-04-01T01:50:08.342 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "contact", 2024-04-01T01:50:08.342 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.342 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.342 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.342 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.343 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.343 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.343 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.343 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.343 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.343 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.343 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.343 INFO:teuthology.orchestra.run.smithi070.stdout: "description": { 2024-04-01T01:50:08.343 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "description", 2024-04-01T01:50:08.343 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.343 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.343 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.343 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.344 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.344 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.344 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.344 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.344 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.344 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.344 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.344 INFO:teuthology.orchestra.run.smithi070.stdout: "device_url": { 2024-04-01T01:50:08.344 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "device_url", 2024-04-01T01:50:08.344 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.344 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.344 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.345 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-01T01:50:08.345 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.345 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.345 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.345 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.345 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.345 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.345 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.345 INFO:teuthology.orchestra.run.smithi070.stdout: "enabled": { 2024-04-01T01:50:08.345 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "enabled", 2024-04-01T01:50:08.345 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.345 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.345 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.345 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.346 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.346 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.346 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.346 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.346 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.346 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.346 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.346 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-04-01T01:50:08.346 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-04-01T01:50:08.346 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.346 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.346 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.347 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "24", 2024-04-01T01:50:08.347 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "8", 2024-04-01T01:50:08.347 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.347 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.347 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.347 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.347 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.347 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.347 INFO:teuthology.orchestra.run.smithi070.stdout: "last_opt_revision": { 2024-04-01T01:50:08.347 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "last_opt_revision", 2024-04-01T01:50:08.347 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.347 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.347 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.348 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "1", 2024-04-01T01:50:08.348 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.348 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.348 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.348 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.348 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.348 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.348 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.348 INFO:teuthology.orchestra.run.smithi070.stdout: "leaderboard": { 2024-04-01T01:50:08.348 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "leaderboard", 2024-04-01T01:50:08.348 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.348 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.348 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.349 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.349 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.349 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.349 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.349 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.349 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.349 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.349 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.349 INFO:teuthology.orchestra.run.smithi070.stdout: "leaderboard_description": { 2024-04-01T01:50:08.349 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "leaderboard_description", 2024-04-01T01:50:08.349 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.349 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.349 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.349 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.350 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.350 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.350 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.350 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.350 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.350 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.350 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.350 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.350 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.350 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.350 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.350 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.351 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.351 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.351 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.351 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.351 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.351 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.351 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.351 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.351 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.351 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.351 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.351 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.351 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.351 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.351 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.352 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.352 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.352 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.352 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.352 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.352 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.352 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.352 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.352 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.352 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.352 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.352 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.352 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.353 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.353 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.353 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.353 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.353 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.353 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.353 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.353 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.353 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.353 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.353 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.353 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.353 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.353 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.354 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.354 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.354 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.354 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.354 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.354 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.354 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.354 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.354 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.354 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.354 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.354 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.355 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.355 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.355 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.355 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.355 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.355 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.355 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.355 INFO:teuthology.orchestra.run.smithi070.stdout: "organization": { 2024-04-01T01:50:08.355 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "organization", 2024-04-01T01:50:08.355 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.355 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.355 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.355 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.356 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.356 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.356 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.356 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.356 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.356 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.356 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.356 INFO:teuthology.orchestra.run.smithi070.stdout: "proxy": { 2024-04-01T01:50:08.356 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "proxy", 2024-04-01T01:50:08.356 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.356 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.356 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.356 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.357 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.357 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.357 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.357 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.357 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.357 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.357 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.357 INFO:teuthology.orchestra.run.smithi070.stdout: "url": { 2024-04-01T01:50:08.357 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "url", 2024-04-01T01:50:08.357 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.357 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.357 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.357 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-01T01:50:08.358 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.358 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.358 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.358 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.358 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.358 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.358 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.358 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.358 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.358 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "test_orchestrator", 2024-04-01T01:50:08.358 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.359 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.359 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.359 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.359 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.359 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.359 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.359 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.359 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.359 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.359 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.359 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.359 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.359 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.360 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.360 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.360 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.360 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.360 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.360 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.360 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.360 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.360 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.360 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.360 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.360 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.360 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.361 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.361 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.361 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.361 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.361 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.361 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.361 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.361 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.361 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.361 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.361 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.361 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.361 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.362 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.362 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.362 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.362 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.362 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.362 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.362 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.362 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.362 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.362 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.362 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.363 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.363 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.363 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.363 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.363 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.363 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.363 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.363 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.363 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.363 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.363 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.363 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.364 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.364 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.364 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.364 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.364 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.364 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.364 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.364 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.364 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "volumes", 2024-04-01T01:50:08.364 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.364 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.364 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.365 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.365 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.365 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.365 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.365 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.365 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.365 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.365 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.365 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.365 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.365 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.365 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.365 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.366 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.366 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.366 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.366 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.366 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.366 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.366 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.366 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.366 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.366 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.366 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.366 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.367 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.367 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.367 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.367 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.367 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.367 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.367 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.367 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.367 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.367 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.367 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.367 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.367 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.367 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.368 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.368 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.368 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.368 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.368 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.368 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.368 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.368 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.368 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.368 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.368 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.368 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.368 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.368 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.370 INFO:teuthology.orchestra.run.smithi070.stdout: "max_concurrent_clones": { 2024-04-01T01:50:08.370 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "max_concurrent_clones", 2024-04-01T01:50:08.370 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.370 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.370 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.370 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "4", 2024-04-01T01:50:08.370 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.370 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.370 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.370 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-01T01:50:08.370 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.370 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.370 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.371 INFO:teuthology.orchestra.run.smithi070.stdout: "periodic_async_work": { 2024-04-01T01:50:08.371 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "periodic_async_work", 2024-04-01T01:50:08.371 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.371 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.371 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.371 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.371 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.371 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.371 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.371 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Periodically check for async work", 2024-04-01T01:50:08.371 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.371 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.371 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.371 INFO:teuthology.orchestra.run.smithi070.stdout: "snapshot_clone_delay": { 2024-04-01T01:50:08.372 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snapshot_clone_delay", 2024-04-01T01:50:08.372 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.372 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.372 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.372 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "0", 2024-04-01T01:50:08.372 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.372 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.372 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.372 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-01T01:50:08.372 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.372 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.372 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.372 INFO:teuthology.orchestra.run.smithi070.stdout: "snapshot_clone_no_wait": { 2024-04-01T01:50:08.372 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "snapshot_clone_no_wait", 2024-04-01T01:50:08.373 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.373 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.373 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.373 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "True", 2024-04-01T01:50:08.373 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.373 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.373 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.373 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-01T01:50:08.373 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.373 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.373 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.373 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.373 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.374 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix", 2024-04-01T01:50:08.374 INFO:teuthology.orchestra.run.smithi070.stdout: "can_run": true, 2024-04-01T01:50:08.374 INFO:teuthology.orchestra.run.smithi070.stdout: "error_string": "", 2024-04-01T01:50:08.374 INFO:teuthology.orchestra.run.smithi070.stdout: "module_options": { 2024-04-01T01:50:08.374 INFO:teuthology.orchestra.run.smithi070.stdout: "discovery_interval": { 2024-04-01T01:50:08.374 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "discovery_interval", 2024-04-01T01:50:08.374 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "uint", 2024-04-01T01:50:08.374 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.374 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.374 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "100", 2024-04-01T01:50:08.374 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.374 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.374 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.374 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.375 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.375 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.375 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.375 INFO:teuthology.orchestra.run.smithi070.stdout: "identifier": { 2024-04-01T01:50:08.375 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "identifier", 2024-04-01T01:50:08.375 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.375 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.375 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.375 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.375 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.375 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.375 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.375 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.375 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.375 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: "interval": { 2024-04-01T01:50:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "interval", 2024-04-01T01:50:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "secs", 2024-04-01T01:50:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "60", 2024-04-01T01:50:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.377 INFO:teuthology.orchestra.run.smithi070.stdout: "log_level": { 2024-04-01T01:50:08.377 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_level", 2024-04-01T01:50:08.377 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.377 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.377 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.377 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.377 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.377 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.377 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.377 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.377 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.377 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.377 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.377 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.378 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.378 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.378 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.378 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.378 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.378 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.378 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster": { 2024-04-01T01:50:08.378 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster", 2024-04-01T01:50:08.378 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.378 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.378 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.378 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.379 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.379 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.379 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.379 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.379 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.379 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.379 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.379 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_cluster_level": { 2024-04-01T01:50:08.379 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_cluster_level", 2024-04-01T01:50:08.379 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.379 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.379 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.379 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "info", 2024-04-01T01:50:08.380 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.380 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.380 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [ 2024-04-01T01:50:08.380 INFO:teuthology.orchestra.run.smithi070.stdout: "", 2024-04-01T01:50:08.380 INFO:teuthology.orchestra.run.smithi070.stdout: "critical", 2024-04-01T01:50:08.380 INFO:teuthology.orchestra.run.smithi070.stdout: "debug", 2024-04-01T01:50:08.380 INFO:teuthology.orchestra.run.smithi070.stdout: "error", 2024-04-01T01:50:08.380 INFO:teuthology.orchestra.run.smithi070.stdout: "info", 2024-04-01T01:50:08.380 INFO:teuthology.orchestra.run.smithi070.stdout: "warning" 2024-04-01T01:50:08.380 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.380 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.380 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.380 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.380 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.381 INFO:teuthology.orchestra.run.smithi070.stdout: "log_to_file": { 2024-04-01T01:50:08.381 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "log_to_file", 2024-04-01T01:50:08.381 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "bool", 2024-04-01T01:50:08.381 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.381 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 1, 2024-04-01T01:50:08.381 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "False", 2024-04-01T01:50:08.381 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.381 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.381 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.381 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.381 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.381 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.381 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.381 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_host": { 2024-04-01T01:50:08.382 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_host", 2024-04-01T01:50:08.382 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.382 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.382 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.382 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "", 2024-04-01T01:50:08.382 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.382 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.382 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.382 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.382 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.382 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.382 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.382 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_port": { 2024-04-01T01:50:08.383 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_port", 2024-04-01T01:50:08.383 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "int", 2024-04-01T01:50:08.383 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.383 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.383 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "10051", 2024-04-01T01:50:08.383 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.383 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.383 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.383 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.383 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.383 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.383 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.383 INFO:teuthology.orchestra.run.smithi070.stdout: "zabbix_sender": { 2024-04-01T01:50:08.384 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "zabbix_sender", 2024-04-01T01:50:08.384 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "str", 2024-04-01T01:50:08.384 INFO:teuthology.orchestra.run.smithi070.stdout: "level": "advanced", 2024-04-01T01:50:08.384 INFO:teuthology.orchestra.run.smithi070.stdout: "flags": 0, 2024-04-01T01:50:08.384 INFO:teuthology.orchestra.run.smithi070.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-01T01:50:08.384 INFO:teuthology.orchestra.run.smithi070.stdout: "min": "", 2024-04-01T01:50:08.384 INFO:teuthology.orchestra.run.smithi070.stdout: "max": "", 2024-04-01T01:50:08.384 INFO:teuthology.orchestra.run.smithi070.stdout: "enum_allowed": [], 2024-04-01T01:50:08.384 INFO:teuthology.orchestra.run.smithi070.stdout: "desc": "", 2024-04-01T01:50:08.384 INFO:teuthology.orchestra.run.smithi070.stdout: "long_desc": "", 2024-04-01T01:50:08.384 INFO:teuthology.orchestra.run.smithi070.stdout: "tags": [], 2024-04-01T01:50:08.384 INFO:teuthology.orchestra.run.smithi070.stdout: "see_also": [] 2024-04-01T01:50:08.384 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.384 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.384 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.385 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.385 INFO:teuthology.orchestra.run.smithi070.stdout: "services": { 2024-04-01T01:50:08.385 INFO:teuthology.orchestra.run.smithi070.stdout: "dashboard": "https://172.21.15.70:8443/" 2024-04-01T01:50:08.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.385 INFO:teuthology.orchestra.run.smithi070.stdout: "always_on_modules": { 2024-04-01T01:50:08.385 INFO:teuthology.orchestra.run.smithi070.stdout: "octopus": [ 2024-04-01T01:50:08.385 INFO:teuthology.orchestra.run.smithi070.stdout: "balancer", 2024-04-01T01:50:08.385 INFO:teuthology.orchestra.run.smithi070.stdout: "crash", 2024-04-01T01:50:08.385 INFO:teuthology.orchestra.run.smithi070.stdout: "devicehealth", 2024-04-01T01:50:08.385 INFO:teuthology.orchestra.run.smithi070.stdout: "orchestrator", 2024-04-01T01:50:08.385 INFO:teuthology.orchestra.run.smithi070.stdout: "pg_autoscaler", 2024-04-01T01:50:08.385 INFO:teuthology.orchestra.run.smithi070.stdout: "progress", 2024-04-01T01:50:08.385 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_support", 2024-04-01T01:50:08.385 INFO:teuthology.orchestra.run.smithi070.stdout: "status", 2024-04-01T01:50:08.386 INFO:teuthology.orchestra.run.smithi070.stdout: "telemetry", 2024-04-01T01:50:08.386 INFO:teuthology.orchestra.run.smithi070.stdout: "volumes" 2024-04-01T01:50:08.386 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.386 INFO:teuthology.orchestra.run.smithi070.stdout: "pacific": [ 2024-04-01T01:50:08.386 INFO:teuthology.orchestra.run.smithi070.stdout: "balancer", 2024-04-01T01:50:08.386 INFO:teuthology.orchestra.run.smithi070.stdout: "crash", 2024-04-01T01:50:08.386 INFO:teuthology.orchestra.run.smithi070.stdout: "devicehealth", 2024-04-01T01:50:08.386 INFO:teuthology.orchestra.run.smithi070.stdout: "orchestrator", 2024-04-01T01:50:08.386 INFO:teuthology.orchestra.run.smithi070.stdout: "pg_autoscaler", 2024-04-01T01:50:08.386 INFO:teuthology.orchestra.run.smithi070.stdout: "progress", 2024-04-01T01:50:08.386 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_support", 2024-04-01T01:50:08.386 INFO:teuthology.orchestra.run.smithi070.stdout: "status", 2024-04-01T01:50:08.386 INFO:teuthology.orchestra.run.smithi070.stdout: "telemetry", 2024-04-01T01:50:08.386 INFO:teuthology.orchestra.run.smithi070.stdout: "volumes" 2024-04-01T01:50:08.386 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.387 INFO:teuthology.orchestra.run.smithi070.stdout: "quincy": [ 2024-04-01T01:50:08.387 INFO:teuthology.orchestra.run.smithi070.stdout: "balancer", 2024-04-01T01:50:08.387 INFO:teuthology.orchestra.run.smithi070.stdout: "crash", 2024-04-01T01:50:08.387 INFO:teuthology.orchestra.run.smithi070.stdout: "devicehealth", 2024-04-01T01:50:08.387 INFO:teuthology.orchestra.run.smithi070.stdout: "orchestrator", 2024-04-01T01:50:08.387 INFO:teuthology.orchestra.run.smithi070.stdout: "pg_autoscaler", 2024-04-01T01:50:08.387 INFO:teuthology.orchestra.run.smithi070.stdout: "progress", 2024-04-01T01:50:08.387 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_support", 2024-04-01T01:50:08.387 INFO:teuthology.orchestra.run.smithi070.stdout: "status", 2024-04-01T01:50:08.387 INFO:teuthology.orchestra.run.smithi070.stdout: "telemetry", 2024-04-01T01:50:08.387 INFO:teuthology.orchestra.run.smithi070.stdout: "volumes" 2024-04-01T01:50:08.387 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.387 INFO:teuthology.orchestra.run.smithi070.stdout: "reef": [ 2024-04-01T01:50:08.388 INFO:teuthology.orchestra.run.smithi070.stdout: "balancer", 2024-04-01T01:50:08.388 INFO:teuthology.orchestra.run.smithi070.stdout: "crash", 2024-04-01T01:50:08.388 INFO:teuthology.orchestra.run.smithi070.stdout: "devicehealth", 2024-04-01T01:50:08.388 INFO:teuthology.orchestra.run.smithi070.stdout: "orchestrator", 2024-04-01T01:50:08.388 INFO:teuthology.orchestra.run.smithi070.stdout: "pg_autoscaler", 2024-04-01T01:50:08.388 INFO:teuthology.orchestra.run.smithi070.stdout: "progress", 2024-04-01T01:50:08.388 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_support", 2024-04-01T01:50:08.388 INFO:teuthology.orchestra.run.smithi070.stdout: "status", 2024-04-01T01:50:08.388 INFO:teuthology.orchestra.run.smithi070.stdout: "telemetry", 2024-04-01T01:50:08.388 INFO:teuthology.orchestra.run.smithi070.stdout: "volumes" 2024-04-01T01:50:08.388 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-01T01:50:08.388 INFO:teuthology.orchestra.run.smithi070.stdout: "squid": [ 2024-04-01T01:50:08.388 INFO:teuthology.orchestra.run.smithi070.stdout: "balancer", 2024-04-01T01:50:08.388 INFO:teuthology.orchestra.run.smithi070.stdout: "crash", 2024-04-01T01:50:08.388 INFO:teuthology.orchestra.run.smithi070.stdout: "devicehealth", 2024-04-01T01:50:08.389 INFO:teuthology.orchestra.run.smithi070.stdout: "orchestrator", 2024-04-01T01:50:08.389 INFO:teuthology.orchestra.run.smithi070.stdout: "pg_autoscaler", 2024-04-01T01:50:08.389 INFO:teuthology.orchestra.run.smithi070.stdout: "progress", 2024-04-01T01:50:08.389 INFO:teuthology.orchestra.run.smithi070.stdout: "rbd_support", 2024-04-01T01:50:08.389 INFO:teuthology.orchestra.run.smithi070.stdout: "status", 2024-04-01T01:50:08.389 INFO:teuthology.orchestra.run.smithi070.stdout: "telemetry", 2024-04-01T01:50:08.389 INFO:teuthology.orchestra.run.smithi070.stdout: "volumes" 2024-04-01T01:50:08.389 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-01T01:50:08.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.389 INFO:teuthology.orchestra.run.smithi070.stdout: "last_failure_osd_epoch": 3, 2024-04-01T01:50:08.389 INFO:teuthology.orchestra.run.smithi070.stdout: "active_clients": [ 2024-04-01T01:50:08.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.389 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "devicehealth", 2024-04-01T01:50:08.390 INFO:teuthology.orchestra.run.smithi070.stdout: "addrvec": [ 2024-04-01T01:50:08.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.390 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v2", 2024-04-01T01:50:08.390 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "172.21.15.70:0", 2024-04-01T01:50:08.390 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3629482083 2024-04-01T01:50:08.390 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.390 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-01T01:50:08.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.390 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "libcephsqlite", 2024-04-01T01:50:08.390 INFO:teuthology.orchestra.run.smithi070.stdout: "addrvec": [ 2024-04-01T01:50:08.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.390 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v2", 2024-04-01T01:50:08.390 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "172.21.15.70:0", 2024-04-01T01:50:08.391 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3255492857 2024-04-01T01:50:08.391 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.391 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-01T01:50:08.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.391 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "rbd_support", 2024-04-01T01:50:08.391 INFO:teuthology.orchestra.run.smithi070.stdout: "addrvec": [ 2024-04-01T01:50:08.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.391 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v2", 2024-04-01T01:50:08.391 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "172.21.15.70:0", 2024-04-01T01:50:08.391 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 2741070294 2024-04-01T01:50:08.391 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.391 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-01T01:50:08.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:08.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.392 INFO:teuthology.orchestra.run.smithi070.stdout: "name": "volumes", 2024-04-01T01:50:08.392 INFO:teuthology.orchestra.run.smithi070.stdout: "addrvec": [ 2024-04-01T01:50:08.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-01T01:50:08.392 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v2", 2024-04-01T01:50:08.392 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "172.21.15.70:0", 2024-04-01T01:50:08.392 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3639427196 2024-04-01T01:50:08.392 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.392 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-01T01:50:08.392 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:08.392 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-01T01:50:08.392 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:50:08.394 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:06 smithi081 ceph-mon[35142]: pgmap v858: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 4.8 MiB/s wr, 528 op/s 2024-04-01T01:50:08.394 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:06 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1216293223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:50:08.394 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:06 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2186794165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:50:08.394 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:06 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3743546289' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:50:08.394 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:07 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1501003947' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-01T01:50:08.395 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:06 smithi189 ceph-mon[35753]: pgmap v858: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 4.8 MiB/s wr, 528 op/s 2024-04-01T01:50:08.395 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:06 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1216293223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:50:08.395 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:06 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2186794165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:50:08.395 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:06 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3743546289' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:50:08.395 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:07 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1501003947' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-01T01:50:08.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:07.050+0000 7f2c2a4aa640 1 -- 172.21.15.70:0/1501003947 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2c100a24a0 msgr2=0x7f2c100a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:08.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:07.050+0000 7f2c2a4aa640 1 --2- 172.21.15.70:0/1501003947 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2c100a24a0 0x7f2c100a4960 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f2c24198f40 tx=0x7f2c14005f90 comp rx=0 tx=0).stop 2024-04-01T01:50:08.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:07.050+0000 7f2c2a4aa640 1 -- 172.21.15.70:0/1501003947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c24100680 msgr2=0x7f2c24197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:08.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:07.050+0000 7f2c2a4aa640 1 --2- 172.21.15.70:0/1501003947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c24100680 0x7f2c24197a30 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c000c00 tx=0x7f2c0c005dc0 comp rx=0 tx=0).stop 2024-04-01T01:50:08.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:07.050+0000 7f2c2a4aa640 1 -- 172.21.15.70:0/1501003947 shutdown_connections 2024-04-01T01:50:08.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:07.050+0000 7f2c2a4aa640 1 --2- 172.21.15.70:0/1501003947 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f2c100a24a0 0x7f2c100a4960 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:08.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:07.050+0000 7f2c2a4aa640 1 --2- 172.21.15.70:0/1501003947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c24102230 0x7f2c2419efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:08.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:07.050+0000 7f2c2a4aa640 1 --2- 172.21.15.70:0/1501003947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c24101870 0x7f2c24197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:08.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:07.050+0000 7f2c2a4aa640 1 --2- 172.21.15.70:0/1501003947 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c24100680 0x7f2c24197a30 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:08.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:07.050+0000 7f2c2a4aa640 1 -- 172.21.15.70:0/1501003947 >> 172.21.15.70:0/1501003947 conn(0x7f2c240fbc30 msgr2=0x7f2c24104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:08.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:07.050+0000 7f2c2a4aa640 1 -- 172.21.15.70:0/1501003947 shutdown_connections 2024-04-01T01:50:08.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:07.050+0000 7f2c2a4aa640 1 -- 172.21.15.70:0/1501003947 wait complete. 2024-04-01T01:50:08.402 DEBUG:tasks.check_counter:Getting stats from a 2024-04-01T01:50:08.402 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph daemon mds.a perf dump 2024-04-01T01:50:08.403 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:07 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1501003947' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-01T01:50:08.660 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:50:09.110 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:08 smithi070 ceph-mon[32179]: pgmap v859: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 3.6 MiB/s wr, 404 op/s 2024-04-01T01:50:09.150 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:08 smithi189 ceph-mon[35753]: pgmap v859: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 3.6 MiB/s wr, 404 op/s 2024-04-01T01:50:09.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:08 smithi081 ceph-mon[35142]: pgmap v859: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 3.6 MiB/s wr, 404 op/s 2024-04-01T01:50:10.891 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:50:10.892 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-0": { 2024-04-01T01:50:10.892 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 1426, 2024-04-01T01:50:10.892 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 1416, 2024-04-01T01:50:10.892 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 1198592, 2024-04-01T01:50:10.892 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 539768, 2024-04-01T01:50:10.893 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 5, 2024-04-01T01:50:10.893 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 1, 2024-04-01T01:50:10.893 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.179693105, 2024-04-01T01:50:10.893 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.061306863, 2024-04-01T01:50:10.893 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.056267996, 2024-04-01T01:50:10.893 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.052697961, 2024-04-01T01:50:10.894 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:10.894 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 1415, 2024-04-01T01:50:10.894 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.116521294, 2024-04-01T01:50:10.894 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000082347 2024-04-01T01:50:10.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.894 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:10.895 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.895 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.895 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.895 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 828992, 2024-04-01T01:50:10.895 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 189968 2024-04-01T01:50:10.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.896 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-1": { 2024-04-01T01:50:10.896 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 987, 2024-04-01T01:50:10.896 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 1218, 2024-04-01T01:50:10.896 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 243368, 2024-04-01T01:50:10.896 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 801696, 2024-04-01T01:50:10.897 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 6, 2024-04-01T01:50:10.897 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 4, 2024-04-01T01:50:10.897 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.110147674, 2024-04-01T01:50:10.897 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.035899002, 2024-04-01T01:50:10.897 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.020650147, 2024-04-01T01:50:10.897 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.038698812, 2024-04-01T01:50:10.898 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:10.898 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 1216, 2024-04-01T01:50:10.898 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.062705017, 2024-04-01T01:50:10.898 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000051566 2024-04-01T01:50:10.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.898 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:10.899 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.899 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.899 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.899 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-01T01:50:10.899 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 488080 2024-04-01T01:50:10.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.900 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-2": { 2024-04-01T01:50:10.900 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 1110, 2024-04-01T01:50:10.900 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 1110, 2024-04-01T01:50:10.900 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 294322, 2024-04-01T01:50:10.900 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 352157, 2024-04-01T01:50:10.900 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 5, 2024-04-01T01:50:10.900 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 3, 2024-04-01T01:50:10.900 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.096135644, 2024-04-01T01:50:10.900 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.027714867, 2024-04-01T01:50:10.900 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.024190997, 2024-04-01T01:50:10.900 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.031120083, 2024-04-01T01:50:10.900 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:10.901 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 1109, 2024-04-01T01:50:10.901 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.046603178, 2024-04-01T01:50:10.901 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000042022 2024-04-01T01:50:10.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.901 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:10.902 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.902 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.902 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.902 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-01T01:50:10.902 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 448 2024-04-01T01:50:10.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.902 INFO:teuthology.orchestra.run.smithi070.stdout: "cct": { 2024-04-01T01:50:10.902 INFO:teuthology.orchestra.run.smithi070.stdout: "total_workers": 1, 2024-04-01T01:50:10.902 INFO:teuthology.orchestra.run.smithi070.stdout: "unhealthy_workers": 0 2024-04-01T01:50:10.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.903 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-MDSRank": { 2024-04-01T01:50:10.903 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-04-01T01:50:10.903 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-04-01T01:50:10.903 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 5195, 2024-04-01T01:50:10.903 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.215323548, 2024-04-01T01:50:10.903 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000041448 2024-04-01T01:50:10.903 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:10.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.903 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-PurgeQueue": { 2024-04-01T01:50:10.903 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-04-01T01:50:10.903 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-04-01T01:50:10.903 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.903 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.903 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.904 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:10.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.904 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-04-01T01:50:10.904 INFO:teuthology.orchestra.run.smithi070.stdout: "request": 0, 2024-04-01T01:50:10.904 INFO:teuthology.orchestra.run.smithi070.stdout: "reply": 0, 2024-04-01T01:50:10.904 INFO:teuthology.orchestra.run.smithi070.stdout: "reply_latency": { 2024-04-01T01:50:10.904 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.904 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.904 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.904 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_reply": 0, 2024-04-01T01:50:10.904 INFO:teuthology.orchestra.run.smithi070.stdout: "forward": 0, 2024-04-01T01:50:10.904 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_complete": 0, 2024-04-01T01:50:10.904 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_keys": 0, 2024-04-01T01:50:10.905 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_commit": 0, 2024-04-01T01:50:10.905 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_split": 0, 2024-04-01T01:50:10.905 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_merge": 0, 2024-04-01T01:50:10.905 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes": 0, 2024-04-01T01:50:10.905 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_top": 0, 2024-04-01T01:50:10.905 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_bottom": 0, 2024-04-01T01:50:10.905 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pin_tail": 0, 2024-04-01T01:50:10.905 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pinned": 0, 2024-04-01T01:50:10.905 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_expired": 0, 2024-04-01T01:50:10.905 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_with_caps": 0, 2024-04-01T01:50:10.905 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": 0, 2024-04-01T01:50:10.905 INFO:teuthology.orchestra.run.smithi070.stdout: "subtrees": 1, 2024-04-01T01:50:10.905 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse": 0, 2024-04-01T01:50:10.905 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_hit": 0, 2024-04-01T01:50:10.905 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_forward": 0, 2024-04-01T01:50:10.906 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_discover": 0, 2024-04-01T01:50:10.906 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_dir_fetch": 0, 2024-04-01T01:50:10.906 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_remote_ino": 0, 2024-04-01T01:50:10.906 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_lock": 0, 2024-04-01T01:50:10.906 INFO:teuthology.orchestra.run.smithi070.stdout: "load_cent": 0, 2024-04-01T01:50:10.906 INFO:teuthology.orchestra.run.smithi070.stdout: "q": 0, 2024-04-01T01:50:10.906 INFO:teuthology.orchestra.run.smithi070.stdout: "exported": 0, 2024-04-01T01:50:10.906 INFO:teuthology.orchestra.run.smithi070.stdout: "exported_inodes": 0, 2024-04-01T01:50:10.906 INFO:teuthology.orchestra.run.smithi070.stdout: "imported": 0, 2024-04-01T01:50:10.906 INFO:teuthology.orchestra.run.smithi070.stdout: "imported_inodes": 0, 2024-04-01T01:50:10.906 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_dir_fetch": 0, 2024-04-01T01:50:10.906 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_backtrace_fetch": 0, 2024-04-01T01:50:10.906 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_peer_discover": 0, 2024-04-01T01:50:10.907 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rfiles": 0, 2024-04-01T01:50:10.907 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rbytes": 0, 2024-04-01T01:50:10.907 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rsnaps": 0, 2024-04-01T01:50:10.907 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_fetch": 0, 2024-04-01T01:50:10.907 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_set_tag": 0, 2024-04-01T01:50:10.907 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_repaired": 0, 2024-04-01T01:50:10.907 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_inotable_repaired": 0, 2024-04-01T01:50:10.907 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_inodes": 0, 2024-04-01T01:50:10.907 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_base_inodes": 0, 2024-04-01T01:50:10.907 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dirfrag_rstats": 0, 2024-04-01T01:50:10.907 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_file_inodes": 0, 2024-04-01T01:50:10.907 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_inode_file_caps": 0, 2024-04-01T01:50:10.907 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_revoke": 0, 2024-04-01T01:50:10.907 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_grant": 0, 2024-04-01T01:50:10.908 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_trunc": 0, 2024-04-01T01:50:10.908 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-01T01:50:10.908 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-01T01:50:10.908 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps": 0, 2024-04-01T01:50:10.908 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps_dirty": 0, 2024-04-01T01:50:10.908 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_cap_release": 0, 2024-04-01T01:50:10.908 INFO:teuthology.orchestra.run.smithi070.stdout: "process_request_cap_release": 0 2024-04-01T01:50:10.908 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.908 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache": { 2024-04-01T01:50:10.909 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update": 0, 2024-04-01T01:50:10.909 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update_receipt": 0, 2024-04-01T01:50:10.909 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_try_discover": 0, 2024-04-01T01:50:10.909 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_send_discover": 0, 2024-04-01T01:50:10.909 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_handle_discover": 0, 2024-04-01T01:50:10.909 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays": 0, 2024-04-01T01:50:10.909 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_delayed": 0, 2024-04-01T01:50:10.909 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_enqueuing": 0, 2024-04-01T01:50:10.909 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_created": 0, 2024-04-01T01:50:10.909 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_enqueued": 0, 2024-04-01T01:50:10.909 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_reintegrated": 0, 2024-04-01T01:50:10.909 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_migrated": 0, 2024-04-01T01:50:10.909 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_processing": 0, 2024-04-01T01:50:10.910 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_enqueued": 0, 2024-04-01T01:50:10.910 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_prioritized": 0, 2024-04-01T01:50:10.910 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_started": 0, 2024-04-01T01:50:10.910 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_completed": 0, 2024-04-01T01:50:10.910 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_quiesce_path": 0, 2024-04-01T01:50:10.910 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_quiesce_inode": 0, 2024-04-01T01:50:10.910 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_enqueue_scrub": 0, 2024-04-01T01:50:10.910 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_exportdir": 0, 2024-04-01T01:50:10.910 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_flush": 0, 2024-04-01T01:50:10.910 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragmentdir": 0, 2024-04-01T01:50:10.911 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragstats": 0, 2024-04-01T01:50:10.911 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_inodestats": 0 2024-04-01T01:50:10.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.911 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log": { 2024-04-01T01:50:10.911 INFO:teuthology.orchestra.run.smithi070.stdout: "evlrg": 0, 2024-04-01T01:50:10.911 INFO:teuthology.orchestra.run.smithi070.stdout: "evadd": 0, 2024-04-01T01:50:10.911 INFO:teuthology.orchestra.run.smithi070.stdout: "evex": 0, 2024-04-01T01:50:10.911 INFO:teuthology.orchestra.run.smithi070.stdout: "evtrm": 0, 2024-04-01T01:50:10.911 INFO:teuthology.orchestra.run.smithi070.stdout: "ev": 5, 2024-04-01T01:50:10.911 INFO:teuthology.orchestra.run.smithi070.stdout: "evexg": 0, 2024-04-01T01:50:10.911 INFO:teuthology.orchestra.run.smithi070.stdout: "evexd": 0, 2024-04-01T01:50:10.911 INFO:teuthology.orchestra.run.smithi070.stdout: "segadd": 0, 2024-04-01T01:50:10.911 INFO:teuthology.orchestra.run.smithi070.stdout: "segex": 0, 2024-04-01T01:50:10.912 INFO:teuthology.orchestra.run.smithi070.stdout: "segtrm": 0, 2024-04-01T01:50:10.912 INFO:teuthology.orchestra.run.smithi070.stdout: "seg": 2, 2024-04-01T01:50:10.912 INFO:teuthology.orchestra.run.smithi070.stdout: "segmjr": 2, 2024-04-01T01:50:10.912 INFO:teuthology.orchestra.run.smithi070.stdout: "segexg": 0, 2024-04-01T01:50:10.912 INFO:teuthology.orchestra.run.smithi070.stdout: "segexd": 0, 2024-04-01T01:50:10.912 INFO:teuthology.orchestra.run.smithi070.stdout: "expos": 4194304, 2024-04-01T01:50:10.912 INFO:teuthology.orchestra.run.smithi070.stdout: "wrpos": 4195901, 2024-04-01T01:50:10.912 INFO:teuthology.orchestra.run.smithi070.stdout: "rdpos": 4195760, 2024-04-01T01:50:10.912 INFO:teuthology.orchestra.run.smithi070.stdout: "jlat": { 2024-04-01T01:50:10.912 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.912 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.912 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.912 INFO:teuthology.orchestra.run.smithi070.stdout: "replayed": 5 2024-04-01T01:50:10.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.913 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_mem": { 2024-04-01T01:50:10.913 INFO:teuthology.orchestra.run.smithi070.stdout: "ino": 3, 2024-04-01T01:50:10.913 INFO:teuthology.orchestra.run.smithi070.stdout: "ino+": 3, 2024-04-01T01:50:10.913 INFO:teuthology.orchestra.run.smithi070.stdout: "ino-": 0, 2024-04-01T01:50:10.913 INFO:teuthology.orchestra.run.smithi070.stdout: "dir": 1, 2024-04-01T01:50:10.913 INFO:teuthology.orchestra.run.smithi070.stdout: "dir+": 1, 2024-04-01T01:50:10.913 INFO:teuthology.orchestra.run.smithi070.stdout: "dir-": 0, 2024-04-01T01:50:10.913 INFO:teuthology.orchestra.run.smithi070.stdout: "dn": 0, 2024-04-01T01:50:10.913 INFO:teuthology.orchestra.run.smithi070.stdout: "dn+": 0, 2024-04-01T01:50:10.913 INFO:teuthology.orchestra.run.smithi070.stdout: "dn-": 0, 2024-04-01T01:50:10.913 INFO:teuthology.orchestra.run.smithi070.stdout: "cap": 0, 2024-04-01T01:50:10.913 INFO:teuthology.orchestra.run.smithi070.stdout: "cap+": 0, 2024-04-01T01:50:10.913 INFO:teuthology.orchestra.run.smithi070.stdout: "cap-": 0, 2024-04-01T01:50:10.914 INFO:teuthology.orchestra.run.smithi070.stdout: "rss": 50600, 2024-04-01T01:50:10.914 INFO:teuthology.orchestra.run.smithi070.stdout: "heap": 215420 2024-04-01T01:50:10.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.914 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_server": { 2024-04-01T01:50:10.914 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_client_request": 0, 2024-04-01T01:50:10.914 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_server_request": 0, 2024-04-01T01:50:10.915 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_request": 0, 2024-04-01T01:50:10.915 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_session": 0, 2024-04-01T01:50:10.915 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_peer_request": 0, 2024-04-01T01:50:10.915 INFO:teuthology.orchestra.run.smithi070.stdout: "req_create_latency": { 2024-04-01T01:50:10.915 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.915 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.916 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.916 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getattr_latency": { 2024-04-01T01:50:10.916 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.916 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.916 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.916 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getfilelock_latency": { 2024-04-01T01:50:10.916 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.916 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.917 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.917 INFO:teuthology.orchestra.run.smithi070.stdout: "req_link_latency": { 2024-04-01T01:50:10.917 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.917 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.917 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.917 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookup_latency": { 2024-04-01T01:50:10.917 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.917 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.917 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.917 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookuphash_latency": { 2024-04-01T01:50:10.917 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.917 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.918 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.918 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupino_latency": { 2024-04-01T01:50:10.918 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.918 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.918 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.918 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupname_latency": { 2024-04-01T01:50:10.918 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.918 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.918 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.918 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupparent_latency": { 2024-04-01T01:50:10.918 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.919 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.919 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.919 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupsnap_latency": { 2024-04-01T01:50:10.919 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.919 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.919 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.919 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lssnap_latency": { 2024-04-01T01:50:10.919 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.919 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.919 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.919 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mkdir_latency": { 2024-04-01T01:50:10.919 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.920 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.920 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.920 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mknod_latency": { 2024-04-01T01:50:10.920 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.920 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.920 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.921 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mksnap_latency": { 2024-04-01T01:50:10.921 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.921 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.921 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.921 INFO:teuthology.orchestra.run.smithi070.stdout: "req_open_latency": { 2024-04-01T01:50:10.921 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.921 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.921 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.921 INFO:teuthology.orchestra.run.smithi070.stdout: "req_readdir_latency": { 2024-04-01T01:50:10.921 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.921 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.921 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.922 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rename_latency": { 2024-04-01T01:50:10.922 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.922 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.922 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.922 INFO:teuthology.orchestra.run.smithi070.stdout: "req_renamesnap_latency": { 2024-04-01T01:50:10.922 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.922 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.922 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.923 INFO:teuthology.orchestra.run.smithi070.stdout: "req_snapdiff_latency": { 2024-04-01T01:50:10.923 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.923 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.923 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.923 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmdir_latency": { 2024-04-01T01:50:10.923 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.923 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.923 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.923 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmsnap_latency": { 2024-04-01T01:50:10.923 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.923 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.924 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.924 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmxattr_latency": { 2024-04-01T01:50:10.924 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.924 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.924 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.924 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setattr_latency": { 2024-04-01T01:50:10.924 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.924 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.924 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.924 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setdirlayout_latency": { 2024-04-01T01:50:10.924 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.924 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.925 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.925 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.925 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setfilelock_latency": { 2024-04-01T01:50:10.925 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.925 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.925 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.925 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.925 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setlayout_latency": { 2024-04-01T01:50:10.925 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.925 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.925 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.925 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.925 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setxattr_latency": { 2024-04-01T01:50:10.925 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.926 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.926 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.926 INFO:teuthology.orchestra.run.smithi070.stdout: "req_symlink_latency": { 2024-04-01T01:50:10.926 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.926 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.926 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.926 INFO:teuthology.orchestra.run.smithi070.stdout: "req_unlink_latency": { 2024-04-01T01:50:10.926 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.926 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.926 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.926 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_revoke_eviction": 0, 2024-04-01T01:50:10.927 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_acquisition_throttle": 0, 2024-04-01T01:50:10.927 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getvxattr_latency": { 2024-04-01T01:50:10.927 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.927 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.927 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.927 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:10.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.927 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_sessions": { 2024-04-01T01:50:10.927 INFO:teuthology.orchestra.run.smithi070.stdout: "session_count": 1, 2024-04-01T01:50:10.927 INFO:teuthology.orchestra.run.smithi070.stdout: "session_add": 2, 2024-04-01T01:50:10.927 INFO:teuthology.orchestra.run.smithi070.stdout: "session_remove": 1, 2024-04-01T01:50:10.927 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_open": 2, 2024-04-01T01:50:10.927 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_stale": 0, 2024-04-01T01:50:10.928 INFO:teuthology.orchestra.run.smithi070.stdout: "total_load": 0, 2024-04-01T01:50:10.928 INFO:teuthology.orchestra.run.smithi070.stdout: "average_load": 0, 2024-04-01T01:50:10.928 INFO:teuthology.orchestra.run.smithi070.stdout: "avg_session_uptime": 3319, 2024-04-01T01:50:10.928 INFO:teuthology.orchestra.run.smithi070.stdout: "mdthresh_evicted": 0 2024-04-01T01:50:10.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.928 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool": { 2024-04-01T01:50:10.928 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_bytes": 0, 2024-04-01T01:50:10.928 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_items": 0, 2024-04-01T01:50:10.928 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_bytes": 0, 2024-04-01T01:50:10.928 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_items": 0, 2024-04-01T01:50:10.928 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_bytes": 0, 2024-04-01T01:50:10.928 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_items": 0, 2024-04-01T01:50:10.928 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-01T01:50:10.929 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_items": 0, 2024-04-01T01:50:10.929 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-01T01:50:10.929 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_items": 0, 2024-04-01T01:50:10.929 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_bytes": 0, 2024-04-01T01:50:10.929 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_items": 0, 2024-04-01T01:50:10.929 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-01T01:50:10.929 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_items": 0, 2024-04-01T01:50:10.929 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_bytes": 0, 2024-04-01T01:50:10.929 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_items": 0, 2024-04-01T01:50:10.929 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_bytes": 0, 2024-04-01T01:50:10.929 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_items": 0, 2024-04-01T01:50:10.929 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-01T01:50:10.929 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_items": 0, 2024-04-01T01:50:10.929 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-01T01:50:10.930 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_items": 0, 2024-04-01T01:50:10.930 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_bytes": 0, 2024-04-01T01:50:10.930 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_items": 0, 2024-04-01T01:50:10.930 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_bytes": 0, 2024-04-01T01:50:10.930 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_items": 0, 2024-04-01T01:50:10.930 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-01T01:50:10.930 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_items": 0, 2024-04-01T01:50:10.930 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_bytes": 0, 2024-04-01T01:50:10.930 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_items": 0, 2024-04-01T01:50:10.930 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_bytes": 0, 2024-04-01T01:50:10.930 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_items": 0, 2024-04-01T01:50:10.930 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_bytes": 0, 2024-04-01T01:50:10.930 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_items": 0, 2024-04-01T01:50:10.931 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_bytes": 0, 2024-04-01T01:50:10.931 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_items": 0, 2024-04-01T01:50:10.931 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_bytes": 190167, 2024-04-01T01:50:10.931 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_items": 52, 2024-04-01T01:50:10.931 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_bytes": 0, 2024-04-01T01:50:10.931 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_items": 0, 2024-04-01T01:50:10.931 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bytes": 0, 2024-04-01T01:50:10.931 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_items": 0, 2024-04-01T01:50:10.931 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_bytes": 0, 2024-04-01T01:50:10.931 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_items": 0, 2024-04-01T01:50:10.931 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_bytes": 0, 2024-04-01T01:50:10.931 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_items": 0, 2024-04-01T01:50:10.931 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_bytes": 9080, 2024-04-01T01:50:10.931 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_items": 174, 2024-04-01T01:50:10.932 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_bytes": 0, 2024-04-01T01:50:10.932 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_items": 0, 2024-04-01T01:50:10.932 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_bytes": 0, 2024-04-01T01:50:10.932 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_items": 0, 2024-04-01T01:50:10.932 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_bytes": 8464, 2024-04-01T01:50:10.932 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_items": 13, 2024-04-01T01:50:10.932 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_bytes": 0, 2024-04-01T01:50:10.932 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_items": 0, 2024-04-01T01:50:10.932 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_bytes": 0, 2024-04-01T01:50:10.932 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_items": 0 2024-04-01T01:50:10.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.932 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter": { 2024-04-01T01:50:10.933 INFO:teuthology.orchestra.run.smithi070.stdout: "op_active": 0, 2024-04-01T01:50:10.933 INFO:teuthology.orchestra.run.smithi070.stdout: "op_laggy": 0, 2024-04-01T01:50:10.933 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send": 3192, 2024-04-01T01:50:10.933 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send_bytes": 21, 2024-04-01T01:50:10.933 INFO:teuthology.orchestra.run.smithi070.stdout: "op_resend": 3, 2024-04-01T01:50:10.933 INFO:teuthology.orchestra.run.smithi070.stdout: "op_reply": 3192, 2024-04-01T01:50:10.933 INFO:teuthology.orchestra.run.smithi070.stdout: "op_latency": { 2024-04-01T01:50:10.933 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 3192, 2024-04-01T01:50:10.933 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 13.624594400, 2024-04-01T01:50:10.933 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.004268356 2024-04-01T01:50:10.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.933 INFO:teuthology.orchestra.run.smithi070.stdout: "op_inflight": 0, 2024-04-01T01:50:10.934 INFO:teuthology.orchestra.run.smithi070.stdout: "oplen_avg": { 2024-04-01T01:50:10.934 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 3192, 2024-04-01T01:50:10.934 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 3193 2024-04-01T01:50:10.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.934 INFO:teuthology.orchestra.run.smithi070.stdout: "op": 3192, 2024-04-01T01:50:10.934 INFO:teuthology.orchestra.run.smithi070.stdout: "op_r": 3192, 2024-04-01T01:50:10.934 INFO:teuthology.orchestra.run.smithi070.stdout: "op_w": 0, 2024-04-01T01:50:10.934 INFO:teuthology.orchestra.run.smithi070.stdout: "op_rmw": 0, 2024-04-01T01:50:10.934 INFO:teuthology.orchestra.run.smithi070.stdout: "op_pg": 0, 2024-04-01T01:50:10.934 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_stat": 2083, 2024-04-01T01:50:10.934 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_create": 0, 2024-04-01T01:50:10.934 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_read": 1107, 2024-04-01T01:50:10.934 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_write": 0, 2024-04-01T01:50:10.934 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writefull": 0, 2024-04-01T01:50:10.935 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writesame": 0, 2024-04-01T01:50:10.935 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_append": 0, 2024-04-01T01:50:10.935 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_zero": 0, 2024-04-01T01:50:10.935 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_truncate": 0, 2024-04-01T01:50:10.935 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_delete": 0, 2024-04-01T01:50:10.935 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_mapext": 0, 2024-04-01T01:50:10.935 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_sparse_read": 0, 2024-04-01T01:50:10.935 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_clonerange": 0, 2024-04-01T01:50:10.935 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_getxattr": 1, 2024-04-01T01:50:10.935 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_setxattr": 0, 2024-04-01T01:50:10.935 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_cmpxattr": 0, 2024-04-01T01:50:10.935 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_rmxattr": 0, 2024-04-01T01:50:10.935 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_resetxattrs": 0, 2024-04-01T01:50:10.935 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_call": 0, 2024-04-01T01:50:10.936 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_watch": 0, 2024-04-01T01:50:10.936 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_notify": 0, 2024-04-01T01:50:10.936 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_src_cmpxattr": 0, 2024-04-01T01:50:10.936 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls": 0, 2024-04-01T01:50:10.936 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls_filter": 0, 2024-04-01T01:50:10.936 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_other": 0, 2024-04-01T01:50:10.936 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_active": 0, 2024-04-01T01:50:10.936 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_send": 0, 2024-04-01T01:50:10.936 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_resend": 0, 2024-04-01T01:50:10.936 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_ping": 0, 2024-04-01T01:50:10.936 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_active": 0, 2024-04-01T01:50:10.936 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_send": 0, 2024-04-01T01:50:10.936 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_resend": 0, 2024-04-01T01:50:10.936 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_active": 0, 2024-04-01T01:50:10.937 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_send": 0, 2024-04-01T01:50:10.937 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_resend": 0, 2024-04-01T01:50:10.937 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_active": 0, 2024-04-01T01:50:10.937 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_send": 0, 2024-04-01T01:50:10.937 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_resend": 0, 2024-04-01T01:50:10.937 INFO:teuthology.orchestra.run.smithi070.stdout: "command_active": 0, 2024-04-01T01:50:10.937 INFO:teuthology.orchestra.run.smithi070.stdout: "command_send": 0, 2024-04-01T01:50:10.937 INFO:teuthology.orchestra.run.smithi070.stdout: "command_resend": 0, 2024-04-01T01:50:10.937 INFO:teuthology.orchestra.run.smithi070.stdout: "map_epoch": 95, 2024-04-01T01:50:10.937 INFO:teuthology.orchestra.run.smithi070.stdout: "map_full": 0, 2024-04-01T01:50:10.937 INFO:teuthology.orchestra.run.smithi070.stdout: "map_inc": 6, 2024-04-01T01:50:10.937 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_sessions": 5, 2024-04-01T01:50:10.937 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_open": 7, 2024-04-01T01:50:10.937 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_close": 2, 2024-04-01T01:50:10.937 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_laggy": 0, 2024-04-01T01:50:10.938 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_wr": 0, 2024-04-01T01:50:10.938 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_rd": 2, 2024-04-01T01:50:10.938 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_del": 0 2024-04-01T01:50:10.938 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.938 INFO:teuthology.orchestra.run.smithi070.stdout: "oft": { 2024-04-01T01:50:10.938 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_objs": 0, 2024-04-01T01:50:10.938 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_kv_pairs": 0, 2024-04-01T01:50:10.938 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_updates": 0, 2024-04-01T01:50:10.938 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_removes": 0 2024-04-01T01:50:10.938 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.938 INFO:teuthology.orchestra.run.smithi070.stdout: "osd-slow-ops": { 2024-04-01T01:50:10.938 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_ops_count": 0 2024-04-01T01:50:10.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.939 INFO:teuthology.orchestra.run.smithi070.stdout: "purge_queue": { 2024-04-01T01:50:10.939 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops": 0, 2024-04-01T01:50:10.939 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops_high_water": 0, 2024-04-01T01:50:10.939 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing": 0, 2024-04-01T01:50:10.939 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_high_water": 0, 2024-04-01T01:50:10.939 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed_ops": 0, 2024-04-01T01:50:10.939 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed": 0, 2024-04-01T01:50:10.939 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_item_in_journal": 0 2024-04-01T01:50:10.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.939 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-01T01:50:10.940 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-01T01:50:10.940 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-04-01T01:50:10.940 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-01T01:50:10.940 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 3519, 2024-04-01T01:50:10.940 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 1399874, 2024-04-01T01:50:10.940 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:10.940 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 3519, 2024-04-01T01:50:10.940 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-01T01:50:10.940 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-01T01:50:10.940 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 3519, 2024-04-01T01:50:10.940 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 1399874, 2024-04-01T01:50:10.940 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-01T01:50:10.941 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.941 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.941 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.941 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:10.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.941 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_bytes": { 2024-04-01T01:50:10.941 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-01T01:50:10.941 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-04-01T01:50:10.941 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-01T01:50:10.941 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-01T01:50:10.941 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-01T01:50:10.941 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:10.941 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:10.941 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 3192, 2024-04-01T01:50:10.941 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 1602, 2024-04-01T01:50:10.942 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 3192, 2024-04-01T01:50:10.942 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 1602, 2024-04-01T01:50:10.942 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-01T01:50:10.942 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.942 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.942 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.942 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:10.942 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.942 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_ops": { 2024-04-01T01:50:10.942 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-01T01:50:10.942 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 1024, 2024-04-01T01:50:10.942 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-01T01:50:10.942 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-01T01:50:10.943 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-01T01:50:10.943 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:10.943 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:10.943 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 3192, 2024-04-01T01:50:10.943 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 3192, 2024-04-01T01:50:10.943 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 3192, 2024-04-01T01:50:10.943 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 3192, 2024-04-01T01:50:10.943 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-01T01:50:10.943 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.943 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.943 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.943 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:10.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.944 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle": { 2024-04-01T01:50:10.944 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-01T01:50:10.944 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-04-01T01:50:10.944 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-01T01:50:10.944 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-01T01:50:10.944 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-01T01:50:10.944 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:10.945 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:10.945 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-01T01:50:10.945 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-01T01:50:10.945 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 0, 2024-04-01T01:50:10.945 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 0, 2024-04-01T01:50:10.945 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-01T01:50:10.945 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.945 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.945 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.945 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:10.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:10.945 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle-0x55a60ac8c6e0": { 2024-04-01T01:50:10.945 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-01T01:50:10.946 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-04-01T01:50:10.946 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-01T01:50:10.946 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-01T01:50:10.946 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-01T01:50:10.946 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:10.946 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:10.946 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-01T01:50:10.946 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-01T01:50:10.946 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 0, 2024-04-01T01:50:10.947 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 0, 2024-04-01T01:50:10.947 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-01T01:50:10.947 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:10.947 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:10.947 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:10.947 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:10.947 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:10.947 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:50:11.209 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:10 smithi189 ceph-mon[35753]: pgmap v860: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 2.2 MiB/s wr, 239 op/s 2024-04-01T01:50:11.213 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:10 smithi070 ceph-mon[32179]: pgmap v860: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 2.2 MiB/s wr, 239 op/s 2024-04-01T01:50:11.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:10 smithi081 ceph-mon[35142]: pgmap v860: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 2.2 MiB/s wr, 239 op/s 2024-04-01T01:50:11.680 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-04-01T01:50:11.680 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-04-01T01:50:11.680 DEBUG:tasks.check_counter:Getting stats from d 2024-04-01T01:50:11.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph daemon mds.d perf dump 2024-04-01T01:50:11.938 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:50:13.113 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:12 smithi070 ceph-mon[32179]: pgmap v861: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 987 KiB/s wr, 118 op/s 2024-04-01T01:50:13.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:12 smithi189 ceph-mon[35753]: pgmap v861: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 987 KiB/s wr, 118 op/s 2024-04-01T01:50:13.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:12 smithi081 ceph-mon[35142]: pgmap v861: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 987 KiB/s wr, 118 op/s 2024-04-01T01:50:14.691 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:50:14.691 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-0": { 2024-04-01T01:50:14.691 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 0, 2024-04-01T01:50:14.691 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 2, 2024-04-01T01:50:14.691 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 0, 2024-04-01T01:50:14.691 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 0, 2024-04-01T01:50:14.691 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 4, 2024-04-01T01:50:14.691 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 0, 2024-04-01T01:50:14.691 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.001036570, 2024-04-01T01:50:14.691 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.000000000, 2024-04-01T01:50:14.692 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.000183988, 2024-04-01T01:50:14.692 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-01T01:50:14.692 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:14.692 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:14.692 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:14.692 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:14.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:14.692 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:14.692 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:14.692 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:14.692 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:14.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:14.692 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-01T01:50:14.692 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 0 2024-04-01T01:50:14.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:14.693 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-1": { 2024-04-01T01:50:14.693 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 6, 2024-04-01T01:50:14.693 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 240, 2024-04-01T01:50:14.693 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 3312, 2024-04-01T01:50:14.693 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 197888, 2024-04-01T01:50:14.693 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 4, 2024-04-01T01:50:14.693 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 2, 2024-04-01T01:50:14.693 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.018097849, 2024-04-01T01:50:14.693 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.014593193, 2024-04-01T01:50:14.693 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.001922252, 2024-04-01T01:50:14.693 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-01T01:50:14.693 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:14.693 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 239, 2024-04-01T01:50:14.694 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.012879729, 2024-04-01T01:50:14.694 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000053890 2024-04-01T01:50:14.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:14.694 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:14.694 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:14.694 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:14.694 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:14.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:14.694 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-01T01:50:14.694 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 197888 2024-04-01T01:50:14.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:14.694 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-2": { 2024-04-01T01:50:14.694 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 316, 2024-04-01T01:50:14.695 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 311, 2024-04-01T01:50:14.695 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 778912, 2024-04-01T01:50:14.695 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 191920, 2024-04-01T01:50:14.695 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 2, 2024-04-01T01:50:14.695 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 1, 2024-04-01T01:50:14.695 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.050410586, 2024-04-01T01:50:14.695 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.022614025, 2024-04-01T01:50:14.695 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.021316899, 2024-04-01T01:50:14.695 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.007346678, 2024-04-01T01:50:14.695 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:14.695 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 310, 2024-04-01T01:50:14.695 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.017865540, 2024-04-01T01:50:14.695 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000057630 2024-04-01T01:50:14.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:14.695 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:14.696 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:14.696 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:14.696 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:14.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:14.696 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 778912, 2024-04-01T01:50:14.696 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 191920 2024-04-01T01:50:14.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:14.696 INFO:teuthology.orchestra.run.smithi070.stdout: "cct": { 2024-04-01T01:50:14.696 INFO:teuthology.orchestra.run.smithi070.stdout: "total_workers": 0, 2024-04-01T01:50:14.696 INFO:teuthology.orchestra.run.smithi070.stdout: "unhealthy_workers": 0 2024-04-01T01:50:14.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:14.696 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool": { 2024-04-01T01:50:14.696 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_bytes": 0, 2024-04-01T01:50:14.696 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_items": 0, 2024-04-01T01:50:14.697 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_bytes": 0, 2024-04-01T01:50:14.697 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_items": 0, 2024-04-01T01:50:14.697 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_bytes": 0, 2024-04-01T01:50:14.697 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_items": 0, 2024-04-01T01:50:14.697 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-01T01:50:14.697 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_items": 0, 2024-04-01T01:50:14.697 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-01T01:50:14.697 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_items": 0, 2024-04-01T01:50:14.697 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_bytes": 0, 2024-04-01T01:50:14.697 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_items": 0, 2024-04-01T01:50:14.697 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-01T01:50:14.697 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_items": 0, 2024-04-01T01:50:14.697 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_bytes": 0, 2024-04-01T01:50:14.697 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_items": 0, 2024-04-01T01:50:14.697 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_bytes": 0, 2024-04-01T01:50:14.698 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_items": 0, 2024-04-01T01:50:14.698 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-01T01:50:14.698 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_items": 0, 2024-04-01T01:50:14.698 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-01T01:50:14.698 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_items": 0, 2024-04-01T01:50:14.698 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_bytes": 0, 2024-04-01T01:50:14.698 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_items": 0, 2024-04-01T01:50:14.698 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_bytes": 0, 2024-04-01T01:50:14.698 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_items": 0, 2024-04-01T01:50:14.698 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-01T01:50:14.698 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_items": 0, 2024-04-01T01:50:14.698 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_bytes": 0, 2024-04-01T01:50:14.698 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_items": 0, 2024-04-01T01:50:14.699 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_bytes": 0, 2024-04-01T01:50:14.699 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_items": 0, 2024-04-01T01:50:14.699 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_bytes": 0, 2024-04-01T01:50:14.699 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_items": 0, 2024-04-01T01:50:14.699 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_bytes": 0, 2024-04-01T01:50:14.699 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_items": 0, 2024-04-01T01:50:14.699 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_bytes": 148704, 2024-04-01T01:50:14.699 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_items": 29, 2024-04-01T01:50:14.699 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_bytes": 0, 2024-04-01T01:50:14.699 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_items": 0, 2024-04-01T01:50:14.699 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bytes": 0, 2024-04-01T01:50:14.699 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_items": 0, 2024-04-01T01:50:14.699 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_bytes": 0, 2024-04-01T01:50:14.699 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_items": 0, 2024-04-01T01:50:14.699 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_bytes": 0, 2024-04-01T01:50:14.700 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_items": 0, 2024-04-01T01:50:14.700 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_bytes": 0, 2024-04-01T01:50:14.700 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_items": 0, 2024-04-01T01:50:14.700 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_bytes": 0, 2024-04-01T01:50:14.700 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_items": 0, 2024-04-01T01:50:14.700 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_bytes": 0, 2024-04-01T01:50:14.700 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_items": 0, 2024-04-01T01:50:14.700 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_bytes": 928, 2024-04-01T01:50:14.700 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_items": 2, 2024-04-01T01:50:14.700 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_bytes": 0, 2024-04-01T01:50:14.700 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_items": 0, 2024-04-01T01:50:14.700 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_bytes": 0, 2024-04-01T01:50:14.700 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_items": 0 2024-04-01T01:50:14.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:14.701 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-01T01:50:14.701 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-01T01:50:14.701 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-04-01T01:50:14.701 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-01T01:50:14.701 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 318, 2024-04-01T01:50:14.701 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 734267, 2024-04-01T01:50:14.701 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:14.701 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 318, 2024-04-01T01:50:14.701 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-01T01:50:14.701 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-01T01:50:14.701 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 318, 2024-04-01T01:50:14.701 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 734267, 2024-04-01T01:50:14.701 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-01T01:50:14.701 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:14.702 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:14.702 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:14.702 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:14.702 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:14.702 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:50:15.130 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:14 smithi081 ceph-mon[35142]: pgmap v862: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 17 op/s 2024-04-01T01:50:15.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:14 smithi189 ceph-mon[35753]: pgmap v862: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 17 op/s 2024-04-01T01:50:15.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:14 smithi070 ceph-mon[32179]: pgmap v862: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 17 op/s 2024-04-01T01:50:15.899 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-01T01:50:15.899 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-01T01:50:15.899 DEBUG:tasks.check_counter:Getting stats from g 2024-04-01T01:50:15.900 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph daemon mds.g perf dump 2024-04-01T01:50:16.155 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:50:17.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:16 smithi189 ceph-mon[35753]: pgmap v863: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.9 KiB/s wr, 10 op/s 2024-04-01T01:50:17.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:16 smithi070 ceph-mon[32179]: pgmap v863: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.9 KiB/s wr, 10 op/s 2024-04-01T01:50:17.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:16 smithi081 ceph-mon[35142]: pgmap v863: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.9 KiB/s wr, 10 op/s 2024-04-01T01:50:17.888 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:50:17.889 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-0": { 2024-04-01T01:50:17.889 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 1439, 2024-04-01T01:50:17.889 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 1429, 2024-04-01T01:50:17.889 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 1202768, 2024-04-01T01:50:17.889 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 544868, 2024-04-01T01:50:17.889 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 5, 2024-04-01T01:50:17.889 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 1, 2024-04-01T01:50:17.889 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.192136712, 2024-04-01T01:50:17.889 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.072176817, 2024-04-01T01:50:17.889 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.056201312, 2024-04-01T01:50:17.889 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.054857614, 2024-04-01T01:50:17.889 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:17.889 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 1428, 2024-04-01T01:50:17.889 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.112539191, 2024-04-01T01:50:17.890 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000078808 2024-04-01T01:50:17.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.890 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:17.890 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.890 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.890 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.890 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 829808, 2024-04-01T01:50:17.890 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 191888 2024-04-01T01:50:17.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.890 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-1": { 2024-04-01T01:50:17.890 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 2109, 2024-04-01T01:50:17.891 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 2342, 2024-04-01T01:50:17.891 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 538149, 2024-04-01T01:50:17.891 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 1160449, 2024-04-01T01:50:17.891 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 5, 2024-04-01T01:50:17.891 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 4, 2024-04-01T01:50:17.891 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.177617990, 2024-04-01T01:50:17.891 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.060310639, 2024-04-01T01:50:17.891 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.041781567, 2024-04-01T01:50:17.891 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.064761405, 2024-04-01T01:50:17.891 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:17.891 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2340, 2024-04-01T01:50:17.891 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.128865986, 2024-04-01T01:50:17.891 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000055070 2024-04-01T01:50:17.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.892 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:17.892 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.892 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.892 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.892 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-01T01:50:17.892 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 490048 2024-04-01T01:50:17.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.892 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-2": { 2024-04-01T01:50:17.892 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 8, 2024-04-01T01:50:17.892 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 8, 2024-04-01T01:50:17.892 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 4621, 2024-04-01T01:50:17.892 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 1732, 2024-04-01T01:50:17.893 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 7, 2024-04-01T01:50:17.893 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 4, 2024-04-01T01:50:17.893 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.002477001, 2024-04-01T01:50:17.893 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.000094334, 2024-04-01T01:50:17.893 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.002097442, 2024-04-01T01:50:17.893 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.000070057, 2024-04-01T01:50:17.893 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:17.893 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 7, 2024-04-01T01:50:17.893 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.002701115, 2024-04-01T01:50:17.893 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000385873 2024-04-01T01:50:17.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.893 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:17.893 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.894 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.894 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.894 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-01T01:50:17.894 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 448 2024-04-01T01:50:17.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.894 INFO:teuthology.orchestra.run.smithi070.stdout: "cct": { 2024-04-01T01:50:17.894 INFO:teuthology.orchestra.run.smithi070.stdout: "total_workers": 1, 2024-04-01T01:50:17.894 INFO:teuthology.orchestra.run.smithi070.stdout: "unhealthy_workers": 0 2024-04-01T01:50:17.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.894 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-MDSRank": { 2024-04-01T01:50:17.894 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-04-01T01:50:17.894 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-04-01T01:50:17.894 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 5165, 2024-04-01T01:50:17.895 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.162157741, 2024-04-01T01:50:17.895 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000031395 2024-04-01T01:50:17.895 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:17.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.895 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-PurgeQueue": { 2024-04-01T01:50:17.895 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-04-01T01:50:17.895 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-04-01T01:50:17.895 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.895 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.895 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.895 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:17.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.895 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-04-01T01:50:17.896 INFO:teuthology.orchestra.run.smithi070.stdout: "request": 0, 2024-04-01T01:50:17.896 INFO:teuthology.orchestra.run.smithi070.stdout: "reply": 0, 2024-04-01T01:50:17.896 INFO:teuthology.orchestra.run.smithi070.stdout: "reply_latency": { 2024-04-01T01:50:17.896 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.896 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.896 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.896 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_reply": 0, 2024-04-01T01:50:17.896 INFO:teuthology.orchestra.run.smithi070.stdout: "forward": 0, 2024-04-01T01:50:17.896 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_complete": 0, 2024-04-01T01:50:17.896 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_keys": 0, 2024-04-01T01:50:17.896 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_commit": 0, 2024-04-01T01:50:17.896 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_split": 0, 2024-04-01T01:50:17.897 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_merge": 0, 2024-04-01T01:50:17.897 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes": 0, 2024-04-01T01:50:17.897 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_top": 0, 2024-04-01T01:50:17.897 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_bottom": 0, 2024-04-01T01:50:17.897 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pin_tail": 0, 2024-04-01T01:50:17.897 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pinned": 0, 2024-04-01T01:50:17.897 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_expired": 0, 2024-04-01T01:50:17.897 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_with_caps": 0, 2024-04-01T01:50:17.897 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": 0, 2024-04-01T01:50:17.897 INFO:teuthology.orchestra.run.smithi070.stdout: "subtrees": 1, 2024-04-01T01:50:17.897 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse": 0, 2024-04-01T01:50:17.897 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_hit": 0, 2024-04-01T01:50:17.897 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_forward": 0, 2024-04-01T01:50:17.898 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_discover": 0, 2024-04-01T01:50:17.898 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_dir_fetch": 0, 2024-04-01T01:50:17.898 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_remote_ino": 0, 2024-04-01T01:50:17.898 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_lock": 0, 2024-04-01T01:50:17.898 INFO:teuthology.orchestra.run.smithi070.stdout: "load_cent": 0, 2024-04-01T01:50:17.898 INFO:teuthology.orchestra.run.smithi070.stdout: "q": 0, 2024-04-01T01:50:17.898 INFO:teuthology.orchestra.run.smithi070.stdout: "exported": 0, 2024-04-01T01:50:17.898 INFO:teuthology.orchestra.run.smithi070.stdout: "exported_inodes": 0, 2024-04-01T01:50:17.898 INFO:teuthology.orchestra.run.smithi070.stdout: "imported": 0, 2024-04-01T01:50:17.898 INFO:teuthology.orchestra.run.smithi070.stdout: "imported_inodes": 0, 2024-04-01T01:50:17.898 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_dir_fetch": 0, 2024-04-01T01:50:17.898 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_backtrace_fetch": 0, 2024-04-01T01:50:17.898 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_peer_discover": 0, 2024-04-01T01:50:17.898 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rfiles": 0, 2024-04-01T01:50:17.899 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rbytes": 0, 2024-04-01T01:50:17.899 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rsnaps": 0, 2024-04-01T01:50:17.899 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_fetch": 0, 2024-04-01T01:50:17.899 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_set_tag": 0, 2024-04-01T01:50:17.899 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_repaired": 0, 2024-04-01T01:50:17.899 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_inotable_repaired": 0, 2024-04-01T01:50:17.899 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_inodes": 0, 2024-04-01T01:50:17.899 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_base_inodes": 0, 2024-04-01T01:50:17.899 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dirfrag_rstats": 0, 2024-04-01T01:50:17.899 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_file_inodes": 0, 2024-04-01T01:50:17.899 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_inode_file_caps": 0, 2024-04-01T01:50:17.899 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_revoke": 0, 2024-04-01T01:50:17.899 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_grant": 0, 2024-04-01T01:50:17.899 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_trunc": 0, 2024-04-01T01:50:17.899 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-01T01:50:17.900 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-01T01:50:17.900 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps": 0, 2024-04-01T01:50:17.900 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps_dirty": 0, 2024-04-01T01:50:17.900 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_cap_release": 0, 2024-04-01T01:50:17.900 INFO:teuthology.orchestra.run.smithi070.stdout: "process_request_cap_release": 0 2024-04-01T01:50:17.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.900 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache": { 2024-04-01T01:50:17.900 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update": 0, 2024-04-01T01:50:17.900 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update_receipt": 0, 2024-04-01T01:50:17.900 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_try_discover": 0, 2024-04-01T01:50:17.900 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_send_discover": 0, 2024-04-01T01:50:17.900 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_handle_discover": 0, 2024-04-01T01:50:17.900 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays": 0, 2024-04-01T01:50:17.900 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_delayed": 0, 2024-04-01T01:50:17.900 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_enqueuing": 0, 2024-04-01T01:50:17.901 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_created": 0, 2024-04-01T01:50:17.901 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_enqueued": 0, 2024-04-01T01:50:17.901 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_reintegrated": 0, 2024-04-01T01:50:17.901 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_migrated": 0, 2024-04-01T01:50:17.901 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_processing": 0, 2024-04-01T01:50:17.901 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_enqueued": 0, 2024-04-01T01:50:17.901 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_prioritized": 0, 2024-04-01T01:50:17.901 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_started": 0, 2024-04-01T01:50:17.901 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_completed": 0, 2024-04-01T01:50:17.901 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_quiesce_path": 0, 2024-04-01T01:50:17.901 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_quiesce_inode": 0, 2024-04-01T01:50:17.901 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_enqueue_scrub": 0, 2024-04-01T01:50:17.901 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_exportdir": 0, 2024-04-01T01:50:17.901 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_flush": 0, 2024-04-01T01:50:17.901 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragmentdir": 0, 2024-04-01T01:50:17.902 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragstats": 0, 2024-04-01T01:50:17.902 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_inodestats": 0 2024-04-01T01:50:17.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.902 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log": { 2024-04-01T01:50:17.902 INFO:teuthology.orchestra.run.smithi070.stdout: "evlrg": 0, 2024-04-01T01:50:17.902 INFO:teuthology.orchestra.run.smithi070.stdout: "evadd": 0, 2024-04-01T01:50:17.902 INFO:teuthology.orchestra.run.smithi070.stdout: "evex": 0, 2024-04-01T01:50:17.902 INFO:teuthology.orchestra.run.smithi070.stdout: "evtrm": 0, 2024-04-01T01:50:17.902 INFO:teuthology.orchestra.run.smithi070.stdout: "ev": 5, 2024-04-01T01:50:17.902 INFO:teuthology.orchestra.run.smithi070.stdout: "evexg": 0, 2024-04-01T01:50:17.902 INFO:teuthology.orchestra.run.smithi070.stdout: "evexd": 0, 2024-04-01T01:50:17.902 INFO:teuthology.orchestra.run.smithi070.stdout: "segadd": 0, 2024-04-01T01:50:17.902 INFO:teuthology.orchestra.run.smithi070.stdout: "segex": 0, 2024-04-01T01:50:17.903 INFO:teuthology.orchestra.run.smithi070.stdout: "segtrm": 0, 2024-04-01T01:50:17.903 INFO:teuthology.orchestra.run.smithi070.stdout: "seg": 2, 2024-04-01T01:50:17.903 INFO:teuthology.orchestra.run.smithi070.stdout: "segmjr": 2, 2024-04-01T01:50:17.903 INFO:teuthology.orchestra.run.smithi070.stdout: "segexg": 0, 2024-04-01T01:50:17.903 INFO:teuthology.orchestra.run.smithi070.stdout: "segexd": 0, 2024-04-01T01:50:17.903 INFO:teuthology.orchestra.run.smithi070.stdout: "expos": 4194304, 2024-04-01T01:50:17.903 INFO:teuthology.orchestra.run.smithi070.stdout: "wrpos": 4195901, 2024-04-01T01:50:17.903 INFO:teuthology.orchestra.run.smithi070.stdout: "rdpos": 4195760, 2024-04-01T01:50:17.903 INFO:teuthology.orchestra.run.smithi070.stdout: "jlat": { 2024-04-01T01:50:17.903 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.904 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.904 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.904 INFO:teuthology.orchestra.run.smithi070.stdout: "replayed": 5 2024-04-01T01:50:17.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.904 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_mem": { 2024-04-01T01:50:17.904 INFO:teuthology.orchestra.run.smithi070.stdout: "ino": 3, 2024-04-01T01:50:17.904 INFO:teuthology.orchestra.run.smithi070.stdout: "ino+": 3, 2024-04-01T01:50:17.904 INFO:teuthology.orchestra.run.smithi070.stdout: "ino-": 0, 2024-04-01T01:50:17.904 INFO:teuthology.orchestra.run.smithi070.stdout: "dir": 1, 2024-04-01T01:50:17.904 INFO:teuthology.orchestra.run.smithi070.stdout: "dir+": 1, 2024-04-01T01:50:17.904 INFO:teuthology.orchestra.run.smithi070.stdout: "dir-": 0, 2024-04-01T01:50:17.904 INFO:teuthology.orchestra.run.smithi070.stdout: "dn": 0, 2024-04-01T01:50:17.904 INFO:teuthology.orchestra.run.smithi070.stdout: "dn+": 0, 2024-04-01T01:50:17.905 INFO:teuthology.orchestra.run.smithi070.stdout: "dn-": 0, 2024-04-01T01:50:17.905 INFO:teuthology.orchestra.run.smithi070.stdout: "cap": 0, 2024-04-01T01:50:17.905 INFO:teuthology.orchestra.run.smithi070.stdout: "cap+": 0, 2024-04-01T01:50:17.905 INFO:teuthology.orchestra.run.smithi070.stdout: "cap-": 0, 2024-04-01T01:50:17.905 INFO:teuthology.orchestra.run.smithi070.stdout: "rss": 52672, 2024-04-01T01:50:17.905 INFO:teuthology.orchestra.run.smithi070.stdout: "heap": 215420 2024-04-01T01:50:17.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.905 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_server": { 2024-04-01T01:50:17.905 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_client_request": 0, 2024-04-01T01:50:17.905 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_server_request": 0, 2024-04-01T01:50:17.905 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_request": 0, 2024-04-01T01:50:17.905 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_session": 0, 2024-04-01T01:50:17.905 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_peer_request": 0, 2024-04-01T01:50:17.905 INFO:teuthology.orchestra.run.smithi070.stdout: "req_create_latency": { 2024-04-01T01:50:17.905 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.906 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.906 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.906 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getattr_latency": { 2024-04-01T01:50:17.906 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.906 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.906 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.906 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getfilelock_latency": { 2024-04-01T01:50:17.906 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.906 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.906 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.907 INFO:teuthology.orchestra.run.smithi070.stdout: "req_link_latency": { 2024-04-01T01:50:17.907 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.907 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.907 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.907 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookup_latency": { 2024-04-01T01:50:17.907 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.907 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.907 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.908 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookuphash_latency": { 2024-04-01T01:50:17.908 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.908 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.908 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.908 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.908 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupino_latency": { 2024-04-01T01:50:17.908 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.908 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.908 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.908 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.909 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupname_latency": { 2024-04-01T01:50:17.909 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.909 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.909 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.909 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupparent_latency": { 2024-04-01T01:50:17.909 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.909 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.909 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.909 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupsnap_latency": { 2024-04-01T01:50:17.909 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.909 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.909 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.910 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lssnap_latency": { 2024-04-01T01:50:17.910 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.910 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.910 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.910 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mkdir_latency": { 2024-04-01T01:50:17.910 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.911 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.911 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.911 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mknod_latency": { 2024-04-01T01:50:17.911 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.911 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.911 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.911 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mksnap_latency": { 2024-04-01T01:50:17.911 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.911 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.912 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.912 INFO:teuthology.orchestra.run.smithi070.stdout: "req_open_latency": { 2024-04-01T01:50:17.912 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.912 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.912 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.912 INFO:teuthology.orchestra.run.smithi070.stdout: "req_readdir_latency": { 2024-04-01T01:50:17.912 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.912 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.912 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.912 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rename_latency": { 2024-04-01T01:50:17.912 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.913 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.913 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.913 INFO:teuthology.orchestra.run.smithi070.stdout: "req_renamesnap_latency": { 2024-04-01T01:50:17.913 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.913 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.913 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.913 INFO:teuthology.orchestra.run.smithi070.stdout: "req_snapdiff_latency": { 2024-04-01T01:50:17.913 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.913 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.913 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.914 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmdir_latency": { 2024-04-01T01:50:17.914 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.914 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.914 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.914 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmsnap_latency": { 2024-04-01T01:50:17.914 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.914 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.914 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.914 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmxattr_latency": { 2024-04-01T01:50:17.914 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.914 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.914 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setattr_latency": { 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setdirlayout_latency": { 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setfilelock_latency": { 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setlayout_latency": { 2024-04-01T01:50:17.915 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.916 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.916 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.916 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setxattr_latency": { 2024-04-01T01:50:17.916 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.916 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.916 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.916 INFO:teuthology.orchestra.run.smithi070.stdout: "req_symlink_latency": { 2024-04-01T01:50:17.916 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.916 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.916 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.916 INFO:teuthology.orchestra.run.smithi070.stdout: "req_unlink_latency": { 2024-04-01T01:50:17.916 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.917 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.917 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.917 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_revoke_eviction": 0, 2024-04-01T01:50:17.917 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_acquisition_throttle": 0, 2024-04-01T01:50:17.917 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getvxattr_latency": { 2024-04-01T01:50:17.917 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.917 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.917 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.917 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:17.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.917 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_sessions": { 2024-04-01T01:50:17.917 INFO:teuthology.orchestra.run.smithi070.stdout: "session_count": 1, 2024-04-01T01:50:17.917 INFO:teuthology.orchestra.run.smithi070.stdout: "session_add": 2, 2024-04-01T01:50:17.917 INFO:teuthology.orchestra.run.smithi070.stdout: "session_remove": 1, 2024-04-01T01:50:17.918 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_open": 2, 2024-04-01T01:50:17.918 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_stale": 0, 2024-04-01T01:50:17.918 INFO:teuthology.orchestra.run.smithi070.stdout: "total_load": 0, 2024-04-01T01:50:17.918 INFO:teuthology.orchestra.run.smithi070.stdout: "average_load": 0, 2024-04-01T01:50:17.918 INFO:teuthology.orchestra.run.smithi070.stdout: "avg_session_uptime": 3326, 2024-04-01T01:50:17.918 INFO:teuthology.orchestra.run.smithi070.stdout: "mdthresh_evicted": 0 2024-04-01T01:50:17.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.918 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool": { 2024-04-01T01:50:17.918 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_bytes": 0, 2024-04-01T01:50:17.918 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_items": 0, 2024-04-01T01:50:17.918 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_bytes": 0, 2024-04-01T01:50:17.918 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_items": 0, 2024-04-01T01:50:17.918 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_bytes": 0, 2024-04-01T01:50:17.918 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_items": 0, 2024-04-01T01:50:17.919 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-01T01:50:17.919 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_items": 0, 2024-04-01T01:50:17.919 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-01T01:50:17.919 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_items": 0, 2024-04-01T01:50:17.919 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_bytes": 0, 2024-04-01T01:50:17.919 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_items": 0, 2024-04-01T01:50:17.919 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-01T01:50:17.919 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_items": 0, 2024-04-01T01:50:17.919 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_bytes": 0, 2024-04-01T01:50:17.919 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_items": 0, 2024-04-01T01:50:17.919 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_bytes": 0, 2024-04-01T01:50:17.919 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_items": 0, 2024-04-01T01:50:17.919 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-01T01:50:17.920 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_items": 0, 2024-04-01T01:50:17.920 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-01T01:50:17.920 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_items": 0, 2024-04-01T01:50:17.920 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_bytes": 0, 2024-04-01T01:50:17.920 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_items": 0, 2024-04-01T01:50:17.920 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_bytes": 0, 2024-04-01T01:50:17.920 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_items": 0, 2024-04-01T01:50:17.920 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-01T01:50:17.920 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_items": 0, 2024-04-01T01:50:17.920 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_bytes": 0, 2024-04-01T01:50:17.920 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_items": 0, 2024-04-01T01:50:17.920 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_bytes": 0, 2024-04-01T01:50:17.920 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_items": 0, 2024-04-01T01:50:17.920 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_bytes": 0, 2024-04-01T01:50:17.920 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_items": 0, 2024-04-01T01:50:17.921 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_bytes": 0, 2024-04-01T01:50:17.921 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_items": 0, 2024-04-01T01:50:17.921 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_bytes": 198247, 2024-04-01T01:50:17.921 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_items": 56, 2024-04-01T01:50:17.921 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_bytes": 0, 2024-04-01T01:50:17.921 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_items": 0, 2024-04-01T01:50:17.921 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bytes": 0, 2024-04-01T01:50:17.921 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_items": 0, 2024-04-01T01:50:17.921 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_bytes": 0, 2024-04-01T01:50:17.921 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_items": 0, 2024-04-01T01:50:17.921 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_bytes": 0, 2024-04-01T01:50:17.921 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_items": 0, 2024-04-01T01:50:17.921 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_bytes": 9080, 2024-04-01T01:50:17.922 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_items": 174, 2024-04-01T01:50:17.922 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_bytes": 0, 2024-04-01T01:50:17.922 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_items": 0, 2024-04-01T01:50:17.922 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_bytes": 0, 2024-04-01T01:50:17.922 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_items": 0, 2024-04-01T01:50:17.922 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_bytes": 8464, 2024-04-01T01:50:17.922 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_items": 13, 2024-04-01T01:50:17.922 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_bytes": 0, 2024-04-01T01:50:17.922 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_items": 0, 2024-04-01T01:50:17.922 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_bytes": 0, 2024-04-01T01:50:17.922 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_items": 0 2024-04-01T01:50:17.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.922 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter": { 2024-04-01T01:50:17.922 INFO:teuthology.orchestra.run.smithi070.stdout: "op_active": 0, 2024-04-01T01:50:17.923 INFO:teuthology.orchestra.run.smithi070.stdout: "op_laggy": 0, 2024-04-01T01:50:17.923 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send": 3222, 2024-04-01T01:50:17.923 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send_bytes": 21, 2024-04-01T01:50:17.923 INFO:teuthology.orchestra.run.smithi070.stdout: "op_resend": 3, 2024-04-01T01:50:17.923 INFO:teuthology.orchestra.run.smithi070.stdout: "op_reply": 3222, 2024-04-01T01:50:17.923 INFO:teuthology.orchestra.run.smithi070.stdout: "op_latency": { 2024-04-01T01:50:17.923 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 3222, 2024-04-01T01:50:17.923 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 9.732709920, 2024-04-01T01:50:17.923 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.003020704 2024-04-01T01:50:17.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.923 INFO:teuthology.orchestra.run.smithi070.stdout: "op_inflight": 0, 2024-04-01T01:50:17.923 INFO:teuthology.orchestra.run.smithi070.stdout: "oplen_avg": { 2024-04-01T01:50:17.923 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 3222, 2024-04-01T01:50:17.924 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 3223 2024-04-01T01:50:17.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.924 INFO:teuthology.orchestra.run.smithi070.stdout: "op": 3222, 2024-04-01T01:50:17.924 INFO:teuthology.orchestra.run.smithi070.stdout: "op_r": 3222, 2024-04-01T01:50:17.924 INFO:teuthology.orchestra.run.smithi070.stdout: "op_w": 0, 2024-04-01T01:50:17.924 INFO:teuthology.orchestra.run.smithi070.stdout: "op_rmw": 0, 2024-04-01T01:50:17.924 INFO:teuthology.orchestra.run.smithi070.stdout: "op_pg": 0, 2024-04-01T01:50:17.924 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_stat": 2103, 2024-04-01T01:50:17.924 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_create": 0, 2024-04-01T01:50:17.924 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_read": 1117, 2024-04-01T01:50:17.924 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_write": 0, 2024-04-01T01:50:17.924 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writefull": 0, 2024-04-01T01:50:17.924 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writesame": 0, 2024-04-01T01:50:17.924 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_append": 0, 2024-04-01T01:50:17.925 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_zero": 0, 2024-04-01T01:50:17.925 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_truncate": 0, 2024-04-01T01:50:17.925 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_delete": 0, 2024-04-01T01:50:17.925 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_mapext": 0, 2024-04-01T01:50:17.925 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_sparse_read": 0, 2024-04-01T01:50:17.925 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_clonerange": 0, 2024-04-01T01:50:17.925 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_getxattr": 1, 2024-04-01T01:50:17.925 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_setxattr": 0, 2024-04-01T01:50:17.925 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_cmpxattr": 0, 2024-04-01T01:50:17.925 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_rmxattr": 0, 2024-04-01T01:50:17.925 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_resetxattrs": 0, 2024-04-01T01:50:17.925 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_call": 0, 2024-04-01T01:50:17.925 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_watch": 0, 2024-04-01T01:50:17.925 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_notify": 0, 2024-04-01T01:50:17.925 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_src_cmpxattr": 0, 2024-04-01T01:50:17.926 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls": 0, 2024-04-01T01:50:17.926 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls_filter": 0, 2024-04-01T01:50:17.926 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_other": 0, 2024-04-01T01:50:17.926 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_active": 0, 2024-04-01T01:50:17.926 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_send": 0, 2024-04-01T01:50:17.926 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_resend": 0, 2024-04-01T01:50:17.926 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_ping": 0, 2024-04-01T01:50:17.926 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_active": 0, 2024-04-01T01:50:17.926 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_send": 0, 2024-04-01T01:50:17.926 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_resend": 0, 2024-04-01T01:50:17.926 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_active": 0, 2024-04-01T01:50:17.926 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_send": 0, 2024-04-01T01:50:17.926 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_resend": 0, 2024-04-01T01:50:17.926 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_active": 0, 2024-04-01T01:50:17.927 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_send": 0, 2024-04-01T01:50:17.927 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_resend": 0, 2024-04-01T01:50:17.927 INFO:teuthology.orchestra.run.smithi070.stdout: "command_active": 0, 2024-04-01T01:50:17.927 INFO:teuthology.orchestra.run.smithi070.stdout: "command_send": 0, 2024-04-01T01:50:17.927 INFO:teuthology.orchestra.run.smithi070.stdout: "command_resend": 0, 2024-04-01T01:50:17.927 INFO:teuthology.orchestra.run.smithi070.stdout: "map_epoch": 95, 2024-04-01T01:50:17.927 INFO:teuthology.orchestra.run.smithi070.stdout: "map_full": 0, 2024-04-01T01:50:17.927 INFO:teuthology.orchestra.run.smithi070.stdout: "map_inc": 6, 2024-04-01T01:50:17.927 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_sessions": 6, 2024-04-01T01:50:17.927 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_open": 8, 2024-04-01T01:50:17.927 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_close": 2, 2024-04-01T01:50:17.927 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_laggy": 0, 2024-04-01T01:50:17.927 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_wr": 0, 2024-04-01T01:50:17.928 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_rd": 2, 2024-04-01T01:50:17.928 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_del": 0 2024-04-01T01:50:17.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.928 INFO:teuthology.orchestra.run.smithi070.stdout: "oft": { 2024-04-01T01:50:17.928 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_objs": 0, 2024-04-01T01:50:17.928 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_kv_pairs": 0, 2024-04-01T01:50:17.928 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_updates": 0, 2024-04-01T01:50:17.928 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_removes": 0 2024-04-01T01:50:17.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.928 INFO:teuthology.orchestra.run.smithi070.stdout: "osd-slow-ops": { 2024-04-01T01:50:17.928 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_ops_count": 0 2024-04-01T01:50:17.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.928 INFO:teuthology.orchestra.run.smithi070.stdout: "purge_queue": { 2024-04-01T01:50:17.928 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops": 0, 2024-04-01T01:50:17.928 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops_high_water": 0, 2024-04-01T01:50:17.929 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing": 0, 2024-04-01T01:50:17.929 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_high_water": 0, 2024-04-01T01:50:17.929 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed_ops": 0, 2024-04-01T01:50:17.929 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed": 0, 2024-04-01T01:50:17.929 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_item_in_journal": 0 2024-04-01T01:50:17.929 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.929 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-01T01:50:17.929 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-01T01:50:17.929 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-04-01T01:50:17.929 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-01T01:50:17.929 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 3552, 2024-04-01T01:50:17.929 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 1406004, 2024-04-01T01:50:17.929 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:17.929 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 3552, 2024-04-01T01:50:17.929 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-01T01:50:17.929 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-01T01:50:17.930 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 3552, 2024-04-01T01:50:17.930 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 1406004, 2024-04-01T01:50:17.930 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-01T01:50:17.930 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.930 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.930 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.930 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:17.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.930 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_bytes": { 2024-04-01T01:50:17.930 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-01T01:50:17.930 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-04-01T01:50:17.930 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-01T01:50:17.930 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-01T01:50:17.930 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-01T01:50:17.930 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:17.931 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:17.931 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 3222, 2024-04-01T01:50:17.931 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 1602, 2024-04-01T01:50:17.931 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 3222, 2024-04-01T01:50:17.931 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 1602, 2024-04-01T01:50:17.931 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-01T01:50:17.931 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.931 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.931 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.931 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:17.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.931 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_ops": { 2024-04-01T01:50:17.931 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-01T01:50:17.931 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 1024, 2024-04-01T01:50:17.931 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-01T01:50:17.932 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-01T01:50:17.932 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-01T01:50:17.932 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:17.932 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:17.932 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 3222, 2024-04-01T01:50:17.932 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 3222, 2024-04-01T01:50:17.932 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 3222, 2024-04-01T01:50:17.932 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 3222, 2024-04-01T01:50:17.932 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-01T01:50:17.932 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.932 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.932 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.932 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:17.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.933 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle": { 2024-04-01T01:50:17.933 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-01T01:50:17.933 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-04-01T01:50:17.933 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-01T01:50:17.933 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-01T01:50:17.933 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-01T01:50:17.933 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:17.933 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:17.933 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-01T01:50:17.933 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-01T01:50:17.933 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 0, 2024-04-01T01:50:17.933 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 0, 2024-04-01T01:50:17.933 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-01T01:50:17.933 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.934 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.934 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.934 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:17.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:17.934 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle-0x564b6ab2c6e0": { 2024-04-01T01:50:17.934 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-01T01:50:17.934 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-04-01T01:50:17.934 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-01T01:50:17.934 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-01T01:50:17.934 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-01T01:50:17.934 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:17.934 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:17.934 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-01T01:50:17.935 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-01T01:50:17.935 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 0, 2024-04-01T01:50:17.935 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 0, 2024-04-01T01:50:17.935 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-01T01:50:17.935 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:17.935 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:17.935 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:17.935 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:17.935 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:17.935 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:50:18.728 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-01T01:50:18.728 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-01T01:50:18.728 DEBUG:tasks.check_counter:Getting stats from j 2024-04-01T01:50:18.728 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph daemon mds.j perf dump 2024-04-01T01:50:18.973 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.a/config 2024-04-01T01:50:19.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:18 smithi070 ceph-mon[32179]: pgmap v864: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.8 KiB/s wr, 12 op/s 2024-04-01T01:50:19.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:18 smithi189 ceph-mon[35753]: pgmap v864: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.8 KiB/s wr, 12 op/s 2024-04-01T01:50:19.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:18 smithi081 ceph-mon[35142]: pgmap v864: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.8 KiB/s wr, 12 op/s 2024-04-01T01:50:20.930 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-01T01:50:20.930 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-0": { 2024-04-01T01:50:20.930 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 0, 2024-04-01T01:50:20.930 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 2, 2024-04-01T01:50:20.930 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 0, 2024-04-01T01:50:20.930 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 0, 2024-04-01T01:50:20.930 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 4, 2024-04-01T01:50:20.931 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 0, 2024-04-01T01:50:20.931 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.001132856, 2024-04-01T01:50:20.931 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.000000000, 2024-04-01T01:50:20.931 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.000296658, 2024-04-01T01:50:20.931 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-01T01:50:20.931 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:20.931 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:20.931 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:20.931 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:20.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:20.931 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:20.931 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:20.931 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:20.932 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:20.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:20.932 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-01T01:50:20.932 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 0 2024-04-01T01:50:20.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:20.932 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-1": { 2024-04-01T01:50:20.932 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 6, 2024-04-01T01:50:20.932 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 241, 2024-04-01T01:50:20.932 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 3312, 2024-04-01T01:50:20.932 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 198064, 2024-04-01T01:50:20.932 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 4, 2024-04-01T01:50:20.932 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 2, 2024-04-01T01:50:20.932 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.021552362, 2024-04-01T01:50:20.932 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.016763996, 2024-04-01T01:50:20.933 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.001995038, 2024-04-01T01:50:20.933 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-01T01:50:20.933 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:20.933 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 240, 2024-04-01T01:50:20.933 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.016066135, 2024-04-01T01:50:20.933 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000066942 2024-04-01T01:50:20.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:20.933 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:20.933 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:20.933 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:20.933 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:20.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:20.933 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-01T01:50:20.933 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 198064 2024-04-01T01:50:20.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:20.934 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-2": { 2024-04-01T01:50:20.934 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 318, 2024-04-01T01:50:20.934 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 313, 2024-04-01T01:50:20.934 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 779456, 2024-04-01T01:50:20.934 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 193184, 2024-04-01T01:50:20.934 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 2, 2024-04-01T01:50:20.934 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 1, 2024-04-01T01:50:20.934 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.048732940, 2024-04-01T01:50:20.934 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.019822597, 2024-04-01T01:50:20.934 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.022304386, 2024-04-01T01:50:20.934 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.007639501, 2024-04-01T01:50:20.935 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:20.935 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 312, 2024-04-01T01:50:20.935 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.020156263, 2024-04-01T01:50:20.935 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000064603 2024-04-01T01:50:20.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:20.935 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:20.935 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:20.935 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:20.935 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:20.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:20.935 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 779456, 2024-04-01T01:50:20.935 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 193184 2024-04-01T01:50:20.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:20.936 INFO:teuthology.orchestra.run.smithi070.stdout: "cct": { 2024-04-01T01:50:20.936 INFO:teuthology.orchestra.run.smithi070.stdout: "total_workers": 0, 2024-04-01T01:50:20.936 INFO:teuthology.orchestra.run.smithi070.stdout: "unhealthy_workers": 0 2024-04-01T01:50:20.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:20.936 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool": { 2024-04-01T01:50:20.936 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_bytes": 0, 2024-04-01T01:50:20.936 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_items": 0, 2024-04-01T01:50:20.936 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_bytes": 0, 2024-04-01T01:50:20.936 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_items": 0, 2024-04-01T01:50:20.936 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_bytes": 0, 2024-04-01T01:50:20.936 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_items": 0, 2024-04-01T01:50:20.937 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-01T01:50:20.937 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_items": 0, 2024-04-01T01:50:20.937 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-01T01:50:20.937 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_items": 0, 2024-04-01T01:50:20.937 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_bytes": 0, 2024-04-01T01:50:20.937 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_items": 0, 2024-04-01T01:50:20.937 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-01T01:50:20.937 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_items": 0, 2024-04-01T01:50:20.937 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_bytes": 0, 2024-04-01T01:50:20.937 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_items": 0, 2024-04-01T01:50:20.937 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_bytes": 0, 2024-04-01T01:50:20.937 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_items": 0, 2024-04-01T01:50:20.937 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-01T01:50:20.938 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_items": 0, 2024-04-01T01:50:20.938 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-01T01:50:20.938 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_items": 0, 2024-04-01T01:50:20.938 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_bytes": 0, 2024-04-01T01:50:20.938 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_items": 0, 2024-04-01T01:50:20.938 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_bytes": 0, 2024-04-01T01:50:20.938 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_items": 0, 2024-04-01T01:50:20.938 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-01T01:50:20.939 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_items": 0, 2024-04-01T01:50:20.939 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_bytes": 0, 2024-04-01T01:50:20.939 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_items": 0, 2024-04-01T01:50:20.939 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_bytes": 0, 2024-04-01T01:50:20.939 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_items": 0, 2024-04-01T01:50:20.939 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_bytes": 0, 2024-04-01T01:50:20.939 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_items": 0, 2024-04-01T01:50:20.939 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_bytes": 0, 2024-04-01T01:50:20.939 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_items": 0, 2024-04-01T01:50:20.939 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_bytes": 148704, 2024-04-01T01:50:20.939 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_items": 29, 2024-04-01T01:50:20.939 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_bytes": 0, 2024-04-01T01:50:20.939 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_items": 0, 2024-04-01T01:50:20.939 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bytes": 0, 2024-04-01T01:50:20.940 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_items": 0, 2024-04-01T01:50:20.940 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_bytes": 0, 2024-04-01T01:50:20.940 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_items": 0, 2024-04-01T01:50:20.940 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_bytes": 0, 2024-04-01T01:50:20.940 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_items": 0, 2024-04-01T01:50:20.940 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_bytes": 0, 2024-04-01T01:50:20.940 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_items": 0, 2024-04-01T01:50:20.940 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_bytes": 0, 2024-04-01T01:50:20.940 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_items": 0, 2024-04-01T01:50:20.940 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_bytes": 0, 2024-04-01T01:50:20.940 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_items": 0, 2024-04-01T01:50:20.940 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_bytes": 928, 2024-04-01T01:50:20.940 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_items": 2, 2024-04-01T01:50:20.940 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_bytes": 0, 2024-04-01T01:50:20.941 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_items": 0, 2024-04-01T01:50:20.941 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_bytes": 0, 2024-04-01T01:50:20.941 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_items": 0 2024-04-01T01:50:20.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-01T01:50:20.941 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-01T01:50:20.941 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-01T01:50:20.941 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-04-01T01:50:20.941 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-01T01:50:20.941 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 320, 2024-04-01T01:50:20.941 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 734527, 2024-04-01T01:50:20.941 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:20.941 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 320, 2024-04-01T01:50:20.941 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-01T01:50:20.941 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-01T01:50:20.941 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 320, 2024-04-01T01:50:20.942 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 734527, 2024-04-01T01:50:20.942 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-01T01:50:20.942 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-01T01:50:20.942 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-01T01:50:20.942 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-01T01:50:20.942 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:20.942 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-01T01:50:20.942 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-01T01:50:20.942 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:20 smithi070 ceph-mon[32179]: pgmap v865: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-04-01T01:50:21.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:20 smithi189 ceph-mon[35753]: pgmap v865: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-04-01T01:50:21.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:20 smithi081 ceph-mon[35142]: pgmap v865: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-04-01T01:50:21.625 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-01T01:50:21.625 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-01T01:50:21.625 DEBUG:tasks.check_counter:Getting stats from b 2024-04-01T01:50:21.625 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph daemon mds.b perf dump 2024-04-01T01:50:21.929 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-01T01:50:23.149 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:22 smithi081 ceph-mon[35142]: pgmap v866: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-04-01T01:50:23.149 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:50:23.149 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.149 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.149 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.150 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.150 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.150 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.150 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.150 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.150 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:50:23.150 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:50:23.150 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:22 smithi081 ceph-mon[35142]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:50:23.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:22 smithi189 ceph-mon[35753]: pgmap v866: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-04-01T01:50:23.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:50:23.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:50:23.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:50:23.250 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:22 smithi189 ceph-mon[35753]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:50:23.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:22 smithi070 ceph-mon[32179]: pgmap v866: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-04-01T01:50:23.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-01T01:50:23.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-01T01:50:23.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-01T01:50:23.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-01T01:50:23.255 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:22 smithi070 ceph-mon[32179]: from='mgr.14150 172.21.15.70:0/1690304786' entity='mgr.x' 2024-04-01T01:50:24.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:23 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-04-01T01:50:24.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:23 smithi189 ceph-mon[35753]: Unable to set osd_memory_target on smithi070 to 184772403: error parsing value: Value '184772403' is below minimum 939524096 2024-04-01T01:50:24.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:23 smithi189 ceph-mon[35753]: Adjusting osd_memory_target on smithi081 to 176.2M 2024-04-01T01:50:24.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:23 smithi189 ceph-mon[35753]: Unable to set osd_memory_target on smithi081 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-01T01:50:24.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:23 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-04-01T01:50:24.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:23 smithi070 ceph-mon[32179]: Unable to set osd_memory_target on smithi070 to 184772403: error parsing value: Value '184772403' is below minimum 939524096 2024-04-01T01:50:24.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:23 smithi070 ceph-mon[32179]: Adjusting osd_memory_target on smithi081 to 176.2M 2024-04-01T01:50:24.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:23 smithi070 ceph-mon[32179]: Unable to set osd_memory_target on smithi081 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-01T01:50:24.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:23 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-04-01T01:50:24.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:23 smithi081 ceph-mon[35142]: Unable to set osd_memory_target on smithi070 to 184772403: error parsing value: Value '184772403' is below minimum 939524096 2024-04-01T01:50:24.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:23 smithi081 ceph-mon[35142]: Adjusting osd_memory_target on smithi081 to 176.2M 2024-04-01T01:50:24.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:23 smithi081 ceph-mon[35142]: Unable to set osd_memory_target on smithi081 to 184786022: error parsing value: Value '184786022' is below minimum 939524096 2024-04-01T01:50:24.376 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-01T01:50:24.377 INFO:teuthology.orchestra.run.smithi081.stdout: "AsyncMessenger::Worker-0": { 2024-04-01T01:50:24.377 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_messages": 15129, 2024-04-01T01:50:24.377 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages": 15317, 2024-04-01T01:50:24.377 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_bytes": 8543006, 2024-04-01T01:50:24.377 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_bytes": 13369526, 2024-04-01T01:50:24.377 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_created_connections": 36, 2024-04-01T01:50:24.377 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_active_connections": 5, 2024-04-01T01:50:24.377 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_total_time": 0.993831993, 2024-04-01T01:50:24.377 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_send_time": 0.284342247, 2024-04-01T01:50:24.377 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_recv_time": 0.279859955, 2024-04-01T01:50:24.377 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_fast_dispatch_time": 0.318338056, 2024-04-01T01:50:24.377 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:24.378 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 15317, 2024-04-01T01:50:24.378 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 4.460165321, 2024-04-01T01:50:24.378 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000291190 2024-04-01T01:50:24.378 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.378 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:24.378 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 2050, 2024-04-01T01:50:24.378 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.001243638, 2024-04-01T01:50:24.378 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000606 2024-04-01T01:50:24.378 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.378 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_encrypted_bytes": 889712, 2024-04-01T01:50:24.378 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_encrypted_bytes": 231360 2024-04-01T01:50:24.379 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.379 INFO:teuthology.orchestra.run.smithi081.stdout: "AsyncMessenger::Worker-1": { 2024-04-01T01:50:24.379 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_messages": 11531, 2024-04-01T01:50:24.379 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages": 11594, 2024-04-01T01:50:24.379 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_bytes": 5505319, 2024-04-01T01:50:24.379 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_bytes": 22276483, 2024-04-01T01:50:24.379 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_created_connections": 9, 2024-04-01T01:50:24.379 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_active_connections": 5, 2024-04-01T01:50:24.379 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_total_time": 0.844973224, 2024-04-01T01:50:24.379 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_send_time": 0.273772794, 2024-04-01T01:50:24.379 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_recv_time": 0.225764041, 2024-04-01T01:50:24.379 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_fast_dispatch_time": 0.269602929, 2024-04-01T01:50:24.380 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:24.380 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 11598, 2024-04-01T01:50:24.380 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 3.452410983, 2024-04-01T01:50:24.380 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000297672 2024-04-01T01:50:24.380 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.380 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:24.380 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 2946, 2024-04-01T01:50:24.380 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000544553, 2024-04-01T01:50:24.380 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000184 2024-04-01T01:50:24.380 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.380 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-01T01:50:24.380 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_encrypted_bytes": 503056 2024-04-01T01:50:24.380 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.380 INFO:teuthology.orchestra.run.smithi081.stdout: "AsyncMessenger::Worker-2": { 2024-04-01T01:50:24.381 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_messages": 44193, 2024-04-01T01:50:24.381 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages": 44198, 2024-04-01T01:50:24.381 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_bytes": 17574991, 2024-04-01T01:50:24.381 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_bytes": 44934014, 2024-04-01T01:50:24.381 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_created_connections": 370, 2024-04-01T01:50:24.381 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_active_connections": 8, 2024-04-01T01:50:24.381 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_total_time": 3.087400312, 2024-04-01T01:50:24.381 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_send_time": 1.190034456, 2024-04-01T01:50:24.382 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_recv_time": 1.596844409, 2024-04-01T01:50:24.382 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_fast_dispatch_time": 0.411493790, 2024-04-01T01:50:24.382 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:24.382 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 44190, 2024-04-01T01:50:24.382 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 5.804369791, 2024-04-01T01:50:24.382 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000131350 2024-04-01T01:50:24.382 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.382 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:24.382 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 53969, 2024-04-01T01:50:24.383 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.010564024, 2024-04-01T01:50:24.383 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000195 2024-04-01T01:50:24.383 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.383 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_encrypted_bytes": 83088, 2024-04-01T01:50:24.383 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_encrypted_bytes": 31200 2024-04-01T01:50:24.383 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.383 INFO:teuthology.orchestra.run.smithi081.stdout: "cct": { 2024-04-01T01:50:24.383 INFO:teuthology.orchestra.run.smithi081.stdout: "total_workers": 1, 2024-04-01T01:50:24.383 INFO:teuthology.orchestra.run.smithi081.stdout: "unhealthy_workers": 0 2024-04-01T01:50:24.383 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: "finisher-MDSRank": { 2024-04-01T01:50:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: "queue_len": 0, 2024-04-01T01:50:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: "complete_latency": { 2024-04-01T01:50:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 30096, 2024-04-01T01:50:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 14.546541103, 2024-04-01T01:50:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000483338 2024-04-01T01:50:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: "finisher-PurgeQueue": { 2024-04-01T01:50:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: "queue_len": 0, 2024-04-01T01:50:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: "complete_latency": { 2024-04-01T01:50:24.384 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 909, 2024-04-01T01:50:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.057043829, 2024-04-01T01:50:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000062754 2024-04-01T01:50:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": { 2024-04-01T01:50:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: "request": 8496, 2024-04-01T01:50:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: "reply": 8496, 2024-04-01T01:50:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: "reply_latency": { 2024-04-01T01:50:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 8496, 2024-04-01T01:50:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 15.220782387, 2024-04-01T01:50:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.001791523 2024-04-01T01:50:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: "slow_reply": 0, 2024-04-01T01:50:24.385 INFO:teuthology.orchestra.run.smithi081.stdout: "forward": 0, 2024-04-01T01:50:24.386 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_fetch_complete": 28, 2024-04-01T01:50:24.386 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_fetch_keys": 28, 2024-04-01T01:50:24.386 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_commit": 1291, 2024-04-01T01:50:24.386 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_split": 3, 2024-04-01T01:50:24.386 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_merge": 0, 2024-04-01T01:50:24.386 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes": 1102, 2024-04-01T01:50:24.386 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_top": 7, 2024-04-01T01:50:24.386 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_bottom": 1095, 2024-04-01T01:50:24.386 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_pin_tail": 0, 2024-04-01T01:50:24.386 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_pinned": 1091, 2024-04-01T01:50:24.386 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_expired": 0, 2024-04-01T01:50:24.386 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_with_caps": 1044, 2024-04-01T01:50:24.386 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": 1044, 2024-04-01T01:50:24.387 INFO:teuthology.orchestra.run.smithi081.stdout: "subtrees": 3, 2024-04-01T01:50:24.387 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse": 13709, 2024-04-01T01:50:24.387 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_hit": 12533, 2024-04-01T01:50:24.387 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_forward": 0, 2024-04-01T01:50:24.387 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_discover": 0, 2024-04-01T01:50:24.387 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_dir_fetch": 0, 2024-04-01T01:50:24.387 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_remote_ino": 0, 2024-04-01T01:50:24.387 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_lock": 9, 2024-04-01T01:50:24.387 INFO:teuthology.orchestra.run.smithi081.stdout: "load_cent": 0, 2024-04-01T01:50:24.387 INFO:teuthology.orchestra.run.smithi081.stdout: "q": 0, 2024-04-01T01:50:24.387 INFO:teuthology.orchestra.run.smithi081.stdout: "exported": 0, 2024-04-01T01:50:24.387 INFO:teuthology.orchestra.run.smithi081.stdout: "exported_inodes": 0, 2024-04-01T01:50:24.387 INFO:teuthology.orchestra.run.smithi081.stdout: "imported": 0, 2024-04-01T01:50:24.388 INFO:teuthology.orchestra.run.smithi081.stdout: "imported_inodes": 0, 2024-04-01T01:50:24.388 INFO:teuthology.orchestra.run.smithi081.stdout: "openino_dir_fetch": 0, 2024-04-01T01:50:24.388 INFO:teuthology.orchestra.run.smithi081.stdout: "openino_backtrace_fetch": 29324, 2024-04-01T01:50:24.388 INFO:teuthology.orchestra.run.smithi081.stdout: "openino_peer_discover": 0, 2024-04-01T01:50:24.388 INFO:teuthology.orchestra.run.smithi081.stdout: "root_rfiles": 1017, 2024-04-01T01:50:24.388 INFO:teuthology.orchestra.run.smithi081.stdout: "root_rbytes": 1524077838, 2024-04-01T01:50:24.388 INFO:teuthology.orchestra.run.smithi081.stdout: "root_rsnaps": 0, 2024-04-01T01:50:24.388 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_backtrace_fetch": 29324, 2024-04-01T01:50:24.388 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_set_tag": 0, 2024-04-01T01:50:24.388 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_backtrace_repaired": 0, 2024-04-01T01:50:24.388 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_inotable_repaired": 0, 2024-04-01T01:50:24.388 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_dir_inodes": 1014, 2024-04-01T01:50:24.388 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_dir_base_inodes": 29, 2024-04-01T01:50:24.388 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_dirfrag_rstats": 985, 2024-04-01T01:50:24.388 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_file_inodes": 28310, 2024-04-01T01:50:24.389 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_inode_file_caps": 0, 2024-04-01T01:50:24.389 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_revoke": 1504, 2024-04-01T01:50:24.389 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_grant": 2835, 2024-04-01T01:50:24.389 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_trunc": 1, 2024-04-01T01:50:24.389 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-01T01:50:24.389 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-01T01:50:24.389 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_caps": 17296, 2024-04-01T01:50:24.389 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_caps_dirty": 11807, 2024-04-01T01:50:24.389 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_cap_release": 57, 2024-04-01T01:50:24.389 INFO:teuthology.orchestra.run.smithi081.stdout: "process_request_cap_release": 2749 2024-04-01T01:50:24.389 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.389 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache": { 2024-04-01T01:50:24.389 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_update": 0, 2024-04-01T01:50:24.389 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_update_receipt": 0, 2024-04-01T01:50:24.390 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_try_discover": 0, 2024-04-01T01:50:24.390 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_send_discover": 0, 2024-04-01T01:50:24.390 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_handle_discover": 7, 2024-04-01T01:50:24.390 INFO:teuthology.orchestra.run.smithi081.stdout: "num_strays": 0, 2024-04-01T01:50:24.390 INFO:teuthology.orchestra.run.smithi081.stdout: "num_strays_delayed": 0, 2024-04-01T01:50:24.390 INFO:teuthology.orchestra.run.smithi081.stdout: "num_strays_enqueuing": 0, 2024-04-01T01:50:24.390 INFO:teuthology.orchestra.run.smithi081.stdout: "strays_created": 425, 2024-04-01T01:50:24.390 INFO:teuthology.orchestra.run.smithi081.stdout: "strays_enqueued": 255, 2024-04-01T01:50:24.390 INFO:teuthology.orchestra.run.smithi081.stdout: "strays_reintegrated": 170, 2024-04-01T01:50:24.390 INFO:teuthology.orchestra.run.smithi081.stdout: "strays_migrated": 0, 2024-04-01T01:50:24.390 INFO:teuthology.orchestra.run.smithi081.stdout: "num_recovering_processing": 0, 2024-04-01T01:50:24.390 INFO:teuthology.orchestra.run.smithi081.stdout: "num_recovering_enqueued": 0, 2024-04-01T01:50:24.390 INFO:teuthology.orchestra.run.smithi081.stdout: "num_recovering_prioritized": 0, 2024-04-01T01:50:24.391 INFO:teuthology.orchestra.run.smithi081.stdout: "recovery_started": 0, 2024-04-01T01:50:24.391 INFO:teuthology.orchestra.run.smithi081.stdout: "recovery_completed": 0, 2024-04-01T01:50:24.391 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_quiesce_path": 0, 2024-04-01T01:50:24.391 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_quiesce_inode": 0, 2024-04-01T01:50:24.391 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_enqueue_scrub": 29, 2024-04-01T01:50:24.391 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_exportdir": 0, 2024-04-01T01:50:24.391 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_flush": 0, 2024-04-01T01:50:24.391 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_fragmentdir": 3, 2024-04-01T01:50:24.391 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_fragstats": 0, 2024-04-01T01:50:24.391 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_inodestats": 0 2024-04-01T01:50:24.391 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.391 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log": { 2024-04-01T01:50:24.391 INFO:teuthology.orchestra.run.smithi081.stdout: "evlrg": 0, 2024-04-01T01:50:24.391 INFO:teuthology.orchestra.run.smithi081.stdout: "evadd": 19954, 2024-04-01T01:50:24.391 INFO:teuthology.orchestra.run.smithi081.stdout: "evex": 19790, 2024-04-01T01:50:24.392 INFO:teuthology.orchestra.run.smithi081.stdout: "evtrm": 19790, 2024-04-01T01:50:24.392 INFO:teuthology.orchestra.run.smithi081.stdout: "ev": 164, 2024-04-01T01:50:24.392 INFO:teuthology.orchestra.run.smithi081.stdout: "evexg": 0, 2024-04-01T01:50:24.392 INFO:teuthology.orchestra.run.smithi081.stdout: "evexd": 525, 2024-04-01T01:50:24.392 INFO:teuthology.orchestra.run.smithi081.stdout: "segadd": 87, 2024-04-01T01:50:24.392 INFO:teuthology.orchestra.run.smithi081.stdout: "segex": 86, 2024-04-01T01:50:24.392 INFO:teuthology.orchestra.run.smithi081.stdout: "segtrm": 86, 2024-04-01T01:50:24.392 INFO:teuthology.orchestra.run.smithi081.stdout: "seg": 1, 2024-04-01T01:50:24.392 INFO:teuthology.orchestra.run.smithi081.stdout: "segmjr": 1, 2024-04-01T01:50:24.392 INFO:teuthology.orchestra.run.smithi081.stdout: "segexg": 0, 2024-04-01T01:50:24.392 INFO:teuthology.orchestra.run.smithi081.stdout: "segexd": 1, 2024-04-01T01:50:24.392 INFO:teuthology.orchestra.run.smithi081.stdout: "expos": 49476516, 2024-04-01T01:50:24.392 INFO:teuthology.orchestra.run.smithi081.stdout: "wrpos": 49900219, 2024-04-01T01:50:24.393 INFO:teuthology.orchestra.run.smithi081.stdout: "rdpos": 0, 2024-04-01T01:50:24.393 INFO:teuthology.orchestra.run.smithi081.stdout: "jlat": { 2024-04-01T01:50:24.393 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 3726, 2024-04-01T01:50:24.393 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 183.520139393, 2024-04-01T01:50:24.393 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.049253928 2024-04-01T01:50:24.393 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.393 INFO:teuthology.orchestra.run.smithi081.stdout: "replayed": 0 2024-04-01T01:50:24.393 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.393 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_mem": { 2024-04-01T01:50:24.393 INFO:teuthology.orchestra.run.smithi081.stdout: "ino": 1065, 2024-04-01T01:50:24.393 INFO:teuthology.orchestra.run.smithi081.stdout: "ino+": 1349, 2024-04-01T01:50:24.393 INFO:teuthology.orchestra.run.smithi081.stdout: "ino-": 284, 2024-04-01T01:50:24.393 INFO:teuthology.orchestra.run.smithi081.stdout: "dir": 67, 2024-04-01T01:50:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: "dir+": 101, 2024-04-01T01:50:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: "dir-": 34, 2024-04-01T01:50:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: "dn": 1102, 2024-04-01T01:50:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: "dn+": 1676, 2024-04-01T01:50:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: "dn-": 574, 2024-04-01T01:50:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: "cap": 1044, 2024-04-01T01:50:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: "cap+": 1312, 2024-04-01T01:50:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: "cap-": 268, 2024-04-01T01:50:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: "rss": 75504, 2024-04-01T01:50:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: "heap": 231804 2024-04-01T01:50:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_server": { 2024-04-01T01:50:24.394 INFO:teuthology.orchestra.run.smithi081.stdout: "dispatch_client_request": 9322, 2024-04-01T01:50:24.395 INFO:teuthology.orchestra.run.smithi081.stdout: "dispatch_server_request": 1, 2024-04-01T01:50:24.395 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_request": 8496, 2024-04-01T01:50:24.395 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_session": 346, 2024-04-01T01:50:24.395 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_peer_request": 2, 2024-04-01T01:50:24.395 INFO:teuthology.orchestra.run.smithi081.stdout: "req_create_latency": { 2024-04-01T01:50:24.395 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 1270, 2024-04-01T01:50:24.395 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 7.467812724, 2024-04-01T01:50:24.395 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.005880167 2024-04-01T01:50:24.395 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.395 INFO:teuthology.orchestra.run.smithi081.stdout: "req_getattr_latency": { 2024-04-01T01:50:24.395 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 465, 2024-04-01T01:50:24.395 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.127603476, 2024-04-01T01:50:24.395 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000274416 2024-04-01T01:50:24.395 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.395 INFO:teuthology.orchestra.run.smithi081.stdout: "req_getfilelock_latency": { 2024-04-01T01:50:24.396 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.396 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.396 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.396 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.396 INFO:teuthology.orchestra.run.smithi081.stdout: "req_link_latency": { 2024-04-01T01:50:24.396 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 170, 2024-04-01T01:50:24.396 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.381516834, 2024-04-01T01:50:24.396 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.002244216 2024-04-01T01:50:24.396 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.396 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookup_latency": { 2024-04-01T01:50:24.396 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 3912, 2024-04-01T01:50:24.396 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 2.413884354, 2024-04-01T01:50:24.397 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000617046 2024-04-01T01:50:24.397 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.397 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookuphash_latency": { 2024-04-01T01:50:24.397 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.397 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.397 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.397 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.397 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookupino_latency": { 2024-04-01T01:50:24.397 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.397 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.397 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.397 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.397 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookupname_latency": { 2024-04-01T01:50:24.397 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.397 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.398 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.398 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.398 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookupparent_latency": { 2024-04-01T01:50:24.398 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.398 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.398 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.398 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.398 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookupsnap_latency": { 2024-04-01T01:50:24.398 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.398 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.398 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.398 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.399 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lssnap_latency": { 2024-04-01T01:50:24.399 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.399 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.399 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.399 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.399 INFO:teuthology.orchestra.run.smithi081.stdout: "req_mkdir_latency": { 2024-04-01T01:50:24.400 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 37, 2024-04-01T01:50:24.400 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.050494032, 2024-04-01T01:50:24.400 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.001364703 2024-04-01T01:50:24.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.400 INFO:teuthology.orchestra.run.smithi081.stdout: "req_mknod_latency": { 2024-04-01T01:50:24.400 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.400 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.400 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.400 INFO:teuthology.orchestra.run.smithi081.stdout: "req_mksnap_latency": { 2024-04-01T01:50:24.400 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.400 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.400 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.400 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.401 INFO:teuthology.orchestra.run.smithi081.stdout: "req_open_latency": { 2024-04-01T01:50:24.401 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 1945, 2024-04-01T01:50:24.401 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 1.874678936, 2024-04-01T01:50:24.401 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000963845 2024-04-01T01:50:24.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.401 INFO:teuthology.orchestra.run.smithi081.stdout: "req_readdir_latency": { 2024-04-01T01:50:24.401 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 76, 2024-04-01T01:50:24.401 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.222389602, 2024-04-01T01:50:24.401 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.002926178 2024-04-01T01:50:24.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.401 INFO:teuthology.orchestra.run.smithi081.stdout: "req_rename_latency": { 2024-04-01T01:50:24.401 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 404, 2024-04-01T01:50:24.401 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 2.263372047, 2024-04-01T01:50:24.401 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.005602406 2024-04-01T01:50:24.401 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.402 INFO:teuthology.orchestra.run.smithi081.stdout: "req_renamesnap_latency": { 2024-04-01T01:50:24.402 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.402 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.402 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.402 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.402 INFO:teuthology.orchestra.run.smithi081.stdout: "req_snapdiff_latency": { 2024-04-01T01:50:24.402 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.402 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.402 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.402 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.402 INFO:teuthology.orchestra.run.smithi081.stdout: "req_rmdir_latency": { 2024-04-01T01:50:24.402 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 2, 2024-04-01T01:50:24.402 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.005636106, 2024-04-01T01:50:24.402 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.002818053 2024-04-01T01:50:24.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.403 INFO:teuthology.orchestra.run.smithi081.stdout: "req_rmsnap_latency": { 2024-04-01T01:50:24.403 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.403 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.403 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.403 INFO:teuthology.orchestra.run.smithi081.stdout: "req_rmxattr_latency": { 2024-04-01T01:50:24.403 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.403 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.403 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.403 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.403 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setattr_latency": { 2024-04-01T01:50:24.403 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 3, 2024-04-01T01:50:24.404 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.016332747, 2024-04-01T01:50:24.404 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.005444249 2024-04-01T01:50:24.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.404 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setdirlayout_latency": { 2024-04-01T01:50:24.404 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.404 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.404 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.404 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setfilelock_latency": { 2024-04-01T01:50:24.404 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.404 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.404 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.404 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.404 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setlayout_latency": { 2024-04-01T01:50:24.405 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.405 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.405 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.405 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setxattr_latency": { 2024-04-01T01:50:24.405 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 9, 2024-04-01T01:50:24.405 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.022649686, 2024-04-01T01:50:24.405 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.002516631 2024-04-01T01:50:24.405 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.405 INFO:teuthology.orchestra.run.smithi081.stdout: "req_symlink_latency": { 2024-04-01T01:50:24.405 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.405 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.406 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.406 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.406 INFO:teuthology.orchestra.run.smithi081.stdout: "req_unlink_latency": { 2024-04-01T01:50:24.406 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 203, 2024-04-01T01:50:24.406 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.374411843, 2024-04-01T01:50:24.406 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.001844393 2024-04-01T01:50:24.406 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.406 INFO:teuthology.orchestra.run.smithi081.stdout: "cap_revoke_eviction": 0, 2024-04-01T01:50:24.406 INFO:teuthology.orchestra.run.smithi081.stdout: "cap_acquisition_throttle": 0, 2024-04-01T01:50:24.406 INFO:teuthology.orchestra.run.smithi081.stdout: "req_getvxattr_latency": { 2024-04-01T01:50:24.406 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.406 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.406 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.406 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:24.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.407 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_sessions": { 2024-04-01T01:50:24.407 INFO:teuthology.orchestra.run.smithi081.stdout: "session_count": 1, 2024-04-01T01:50:24.407 INFO:teuthology.orchestra.run.smithi081.stdout: "session_add": 4, 2024-04-01T01:50:24.407 INFO:teuthology.orchestra.run.smithi081.stdout: "session_remove": 3, 2024-04-01T01:50:24.407 INFO:teuthology.orchestra.run.smithi081.stdout: "sessions_open": 1, 2024-04-01T01:50:24.407 INFO:teuthology.orchestra.run.smithi081.stdout: "sessions_stale": 0, 2024-04-01T01:50:24.407 INFO:teuthology.orchestra.run.smithi081.stdout: "total_load": 2050, 2024-04-01T01:50:24.407 INFO:teuthology.orchestra.run.smithi081.stdout: "average_load": 2050, 2024-04-01T01:50:24.407 INFO:teuthology.orchestra.run.smithi081.stdout: "avg_session_uptime": 1006, 2024-04-01T01:50:24.407 INFO:teuthology.orchestra.run.smithi081.stdout: "mdthresh_evicted": 0 2024-04-01T01:50:24.407 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.407 INFO:teuthology.orchestra.run.smithi081.stdout: "mempool": { 2024-04-01T01:50:24.407 INFO:teuthology.orchestra.run.smithi081.stdout: "bloom_filter_bytes": 0, 2024-04-01T01:50:24.408 INFO:teuthology.orchestra.run.smithi081.stdout: "bloom_filter_items": 0, 2024-04-01T01:50:24.408 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_alloc_bytes": 0, 2024-04-01T01:50:24.408 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_alloc_items": 0, 2024-04-01T01:50:24.408 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_data_bytes": 0, 2024-04-01T01:50:24.408 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_data_items": 0, 2024-04-01T01:50:24.408 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-01T01:50:24.408 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_onode_items": 0, 2024-04-01T01:50:24.408 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-01T01:50:24.408 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_meta_items": 0, 2024-04-01T01:50:24.408 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_other_bytes": 0, 2024-04-01T01:50:24.408 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_other_items": 0, 2024-04-01T01:50:24.408 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-01T01:50:24.408 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_buffer_items": 0, 2024-04-01T01:50:24.408 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_bytes": 0, 2024-04-01T01:50:24.409 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_items": 0, 2024-04-01T01:50:24.409 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_blob_bytes": 0, 2024-04-01T01:50:24.409 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_blob_items": 0, 2024-04-01T01:50:24.409 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-01T01:50:24.409 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_shared_blob_items": 0, 2024-04-01T01:50:24.409 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-01T01:50:24.409 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_inline_bl_items": 0, 2024-04-01T01:50:24.409 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_bytes": 0, 2024-04-01T01:50:24.409 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_items": 0, 2024-04-01T01:50:24.409 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_txc_bytes": 0, 2024-04-01T01:50:24.409 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_txc_items": 0, 2024-04-01T01:50:24.409 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-01T01:50:24.409 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_writing_deferred_items": 0, 2024-04-01T01:50:24.409 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_writing_bytes": 0, 2024-04-01T01:50:24.410 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_writing_items": 0, 2024-04-01T01:50:24.410 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_bytes": 0, 2024-04-01T01:50:24.410 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_items": 0, 2024-04-01T01:50:24.410 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_file_reader_bytes": 0, 2024-04-01T01:50:24.410 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_file_reader_items": 0, 2024-04-01T01:50:24.410 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_file_writer_bytes": 0, 2024-04-01T01:50:24.410 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_file_writer_items": 0, 2024-04-01T01:50:24.410 INFO:teuthology.orchestra.run.smithi081.stdout: "buffer_anon_bytes": 300336, 2024-04-01T01:50:24.410 INFO:teuthology.orchestra.run.smithi081.stdout: "buffer_anon_items": 119, 2024-04-01T01:50:24.410 INFO:teuthology.orchestra.run.smithi081.stdout: "buffer_meta_bytes": 0, 2024-04-01T01:50:24.410 INFO:teuthology.orchestra.run.smithi081.stdout: "buffer_meta_items": 0, 2024-04-01T01:50:24.410 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_bytes": 0, 2024-04-01T01:50:24.411 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_items": 0, 2024-04-01T01:50:24.411 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mapbl_bytes": 0, 2024-04-01T01:50:24.411 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mapbl_items": 0, 2024-04-01T01:50:24.411 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pglog_bytes": 0, 2024-04-01T01:50:24.411 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pglog_items": 0, 2024-04-01T01:50:24.411 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_bytes": 9080, 2024-04-01T01:50:24.411 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_items": 174, 2024-04-01T01:50:24.411 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_mapping_bytes": 0, 2024-04-01T01:50:24.411 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_mapping_items": 0, 2024-04-01T01:50:24.411 INFO:teuthology.orchestra.run.smithi081.stdout: "pgmap_bytes": 0, 2024-04-01T01:50:24.411 INFO:teuthology.orchestra.run.smithi081.stdout: "pgmap_items": 0, 2024-04-01T01:50:24.411 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_co_bytes": 3587465, 2024-04-01T01:50:24.411 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_co_items": 31159, 2024-04-01T01:50:24.412 INFO:teuthology.orchestra.run.smithi081.stdout: "unittest_1_bytes": 0, 2024-04-01T01:50:24.412 INFO:teuthology.orchestra.run.smithi081.stdout: "unittest_1_items": 0, 2024-04-01T01:50:24.412 INFO:teuthology.orchestra.run.smithi081.stdout: "unittest_2_bytes": 0, 2024-04-01T01:50:24.412 INFO:teuthology.orchestra.run.smithi081.stdout: "unittest_2_items": 0 2024-04-01T01:50:24.412 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.412 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter": { 2024-04-01T01:50:24.412 INFO:teuthology.orchestra.run.smithi081.stdout: "op_active": 0, 2024-04-01T01:50:24.412 INFO:teuthology.orchestra.run.smithi081.stdout: "op_laggy": 0, 2024-04-01T01:50:24.412 INFO:teuthology.orchestra.run.smithi081.stdout: "op_send": 39064, 2024-04-01T01:50:24.412 INFO:teuthology.orchestra.run.smithi081.stdout: "op_send_bytes": 49907241, 2024-04-01T01:50:24.412 INFO:teuthology.orchestra.run.smithi081.stdout: "op_resend": 32, 2024-04-01T01:50:24.412 INFO:teuthology.orchestra.run.smithi081.stdout: "op_reply": 39064, 2024-04-01T01:50:24.412 INFO:teuthology.orchestra.run.smithi081.stdout: "op_latency": { 2024-04-01T01:50:24.412 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 39064, 2024-04-01T01:50:24.413 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 394.903227330, 2024-04-01T01:50:24.413 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.010109134 2024-04-01T01:50:24.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.413 INFO:teuthology.orchestra.run.smithi081.stdout: "op_inflight": 0, 2024-04-01T01:50:24.413 INFO:teuthology.orchestra.run.smithi081.stdout: "oplen_avg": { 2024-04-01T01:50:24.413 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 39064, 2024-04-01T01:50:24.413 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 45971 2024-04-01T01:50:24.413 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.413 INFO:teuthology.orchestra.run.smithi081.stdout: "op": 39064, 2024-04-01T01:50:24.413 INFO:teuthology.orchestra.run.smithi081.stdout: "op_r": 29543, 2024-04-01T01:50:24.413 INFO:teuthology.orchestra.run.smithi081.stdout: "op_w": 9521, 2024-04-01T01:50:24.413 INFO:teuthology.orchestra.run.smithi081.stdout: "op_rmw": 0, 2024-04-01T01:50:24.413 INFO:teuthology.orchestra.run.smithi081.stdout: "op_pg": 0, 2024-04-01T01:50:24.414 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_stat": 1220, 2024-04-01T01:50:24.414 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_create": 2418, 2024-04-01T01:50:24.414 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_read": 134, 2024-04-01T01:50:24.414 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_write": 3831, 2024-04-01T01:50:24.414 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_writefull": 351, 2024-04-01T01:50:24.414 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_writesame": 0, 2024-04-01T01:50:24.414 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_append": 0, 2024-04-01T01:50:24.414 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_zero": 0, 2024-04-01T01:50:24.414 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_truncate": 3, 2024-04-01T01:50:24.414 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_delete": 1075, 2024-04-01T01:50:24.414 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_mapext": 0, 2024-04-01T01:50:24.414 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_sparse_read": 0, 2024-04-01T01:50:24.414 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_clonerange": 0, 2024-04-01T01:50:24.415 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_getxattr": 29409, 2024-04-01T01:50:24.415 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_setxattr": 3711, 2024-04-01T01:50:24.415 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_cmpxattr": 0, 2024-04-01T01:50:24.415 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_rmxattr": 0, 2024-04-01T01:50:24.415 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_resetxattrs": 0, 2024-04-01T01:50:24.415 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_call": 0, 2024-04-01T01:50:24.415 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_watch": 0, 2024-04-01T01:50:24.415 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_notify": 0, 2024-04-01T01:50:24.415 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_src_cmpxattr": 0, 2024-04-01T01:50:24.415 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_pgls": 0, 2024-04-01T01:50:24.415 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_pgls_filter": 0, 2024-04-01T01:50:24.415 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_other": 1, 2024-04-01T01:50:24.415 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_active": 0, 2024-04-01T01:50:24.415 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_send": 0, 2024-04-01T01:50:24.416 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_resend": 0, 2024-04-01T01:50:24.416 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_ping": 0, 2024-04-01T01:50:24.416 INFO:teuthology.orchestra.run.smithi081.stdout: "poolop_active": 0, 2024-04-01T01:50:24.416 INFO:teuthology.orchestra.run.smithi081.stdout: "poolop_send": 0, 2024-04-01T01:50:24.416 INFO:teuthology.orchestra.run.smithi081.stdout: "poolop_resend": 0, 2024-04-01T01:50:24.416 INFO:teuthology.orchestra.run.smithi081.stdout: "poolstat_active": 0, 2024-04-01T01:50:24.416 INFO:teuthology.orchestra.run.smithi081.stdout: "poolstat_send": 0, 2024-04-01T01:50:24.416 INFO:teuthology.orchestra.run.smithi081.stdout: "poolstat_resend": 0, 2024-04-01T01:50:24.416 INFO:teuthology.orchestra.run.smithi081.stdout: "statfs_active": 0, 2024-04-01T01:50:24.416 INFO:teuthology.orchestra.run.smithi081.stdout: "statfs_send": 0, 2024-04-01T01:50:24.416 INFO:teuthology.orchestra.run.smithi081.stdout: "statfs_resend": 0, 2024-04-01T01:50:24.416 INFO:teuthology.orchestra.run.smithi081.stdout: "command_active": 0, 2024-04-01T01:50:24.416 INFO:teuthology.orchestra.run.smithi081.stdout: "command_send": 0, 2024-04-01T01:50:24.416 INFO:teuthology.orchestra.run.smithi081.stdout: "command_resend": 0, 2024-04-01T01:50:24.417 INFO:teuthology.orchestra.run.smithi081.stdout: "map_epoch": 95, 2024-04-01T01:50:24.417 INFO:teuthology.orchestra.run.smithi081.stdout: "map_full": 0, 2024-04-01T01:50:24.417 INFO:teuthology.orchestra.run.smithi081.stdout: "map_inc": 11, 2024-04-01T01:50:24.417 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_sessions": 12, 2024-04-01T01:50:24.417 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_session_open": 12, 2024-04-01T01:50:24.417 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_session_close": 0, 2024-04-01T01:50:24.417 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_laggy": 0, 2024-04-01T01:50:24.417 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_wr": 3388, 2024-04-01T01:50:24.417 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_rd": 112, 2024-04-01T01:50:24.417 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_del": 318 2024-04-01T01:50:24.417 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.417 INFO:teuthology.orchestra.run.smithi081.stdout: "oft": { 2024-04-01T01:50:24.418 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_total_objs": 1, 2024-04-01T01:50:24.418 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_total_kv_pairs": 0, 2024-04-01T01:50:24.418 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_total_updates": 0, 2024-04-01T01:50:24.418 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_total_removes": 0 2024-04-01T01:50:24.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.418 INFO:teuthology.orchestra.run.smithi081.stdout: "osd-slow-ops": { 2024-04-01T01:50:24.418 INFO:teuthology.orchestra.run.smithi081.stdout: "slow_ops_count": 0 2024-04-01T01:50:24.418 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.418 INFO:teuthology.orchestra.run.smithi081.stdout: "purge_queue": { 2024-04-01T01:50:24.418 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executing_ops": 0, 2024-04-01T01:50:24.418 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executing_ops_high_water": 24, 2024-04-01T01:50:24.418 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executing": 0, 2024-04-01T01:50:24.418 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executing_high_water": 5, 2024-04-01T01:50:24.418 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executed_ops": 543, 2024-04-01T01:50:24.419 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executed": 255, 2024-04-01T01:50:24.419 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_item_in_journal": 0 2024-04-01T01:50:24.419 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.419 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-01T01:50:24.419 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:24.419 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 104857600, 2024-04-01T01:50:24.419 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:24.419 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 70849, 2024-04-01T01:50:24.419 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 25213462, 2024-04-01T01:50:24.419 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:24.419 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 70849, 2024-04-01T01:50:24.419 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 0, 2024-04-01T01:50:24.419 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 0, 2024-04-01T01:50:24.419 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 70849, 2024-04-01T01:50:24.420 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 25213462, 2024-04-01T01:50:24.420 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:24.420 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.420 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.420 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.420 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:24.420 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.420 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-objecter_bytes": { 2024-04-01T01:50:24.420 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:24.420 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 104857600, 2024-04-01T01:50:24.420 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:24.420 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 0, 2024-04-01T01:50:24.420 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 0, 2024-04-01T01:50:24.420 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:24.421 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:24.421 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 39064, 2024-04-01T01:50:24.421 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 49934460, 2024-04-01T01:50:24.421 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 39064, 2024-04-01T01:50:24.421 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 49934460, 2024-04-01T01:50:24.421 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:24.421 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.421 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.421 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.421 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:24.421 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.421 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-objecter_ops": { 2024-04-01T01:50:24.421 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:24.421 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 1024, 2024-04-01T01:50:24.422 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:24.422 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 0, 2024-04-01T01:50:24.422 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 0, 2024-04-01T01:50:24.422 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:24.422 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:24.422 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 39064, 2024-04-01T01:50:24.422 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 39064, 2024-04-01T01:50:24.422 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 39064, 2024-04-01T01:50:24.422 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 39064, 2024-04-01T01:50:24.422 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:24.422 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.422 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.422 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.423 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:24.423 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.423 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-write_buf_throttle": { 2024-04-01T01:50:24.423 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:24.423 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 3758096384, 2024-04-01T01:50:24.423 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:24.423 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 255, 2024-04-01T01:50:24.423 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 27779, 2024-04-01T01:50:24.423 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:24.423 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 255, 2024-04-01T01:50:24.423 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 0, 2024-04-01T01:50:24.423 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 0, 2024-04-01T01:50:24.423 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 105, 2024-04-01T01:50:24.423 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 27779, 2024-04-01T01:50:24.423 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:24.424 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.424 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.424 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.424 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:24.424 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:24.424 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-write_buf_throttle-0x55eca4be2500": { 2024-04-01T01:50:24.424 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:24.424 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 3758096384, 2024-04-01T01:50:24.424 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:24.424 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 19954, 2024-04-01T01:50:24.424 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 45705915, 2024-04-01T01:50:24.424 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:24.424 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 19954, 2024-04-01T01:50:24.424 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 0, 2024-04-01T01:50:24.424 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 0, 2024-04-01T01:50:24.425 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 3726, 2024-04-01T01:50:24.425 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 45705915, 2024-04-01T01:50:24.425 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:24.425 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:24.425 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:24.425 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:24.425 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:24.425 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:24.425 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-01T01:50:25.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:24 smithi081 ceph-mon[35142]: pgmap v867: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-04-01T01:50:25.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:24 smithi189 ceph-mon[35753]: pgmap v867: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-04-01T01:50:25.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:24 smithi070 ceph-mon[32179]: pgmap v867: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-04-01T01:50:25.379 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-04-01T01:50:25.379 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-01T01:50:25.379 DEBUG:tasks.check_counter:Getting stats from e 2024-04-01T01:50:25.380 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph daemon mds.e perf dump 2024-04-01T01:50:25.640 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-01T01:50:27.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:26 smithi081 ceph-mon[35142]: pgmap v868: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-01T01:50:27.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:26 smithi189 ceph-mon[35753]: pgmap v868: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-01T01:50:27.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:26 smithi070 ceph-mon[32179]: pgmap v868: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-01T01:50:27.550 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-01T01:50:27.550 INFO:teuthology.orchestra.run.smithi081.stdout: "AsyncMessenger::Worker-0": { 2024-04-01T01:50:27.550 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_messages": 2002, 2024-04-01T01:50:27.551 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages": 1243, 2024-04-01T01:50:27.551 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_bytes": 488517, 2024-04-01T01:50:27.551 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_bytes": 465866, 2024-04-01T01:50:27.551 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_created_connections": 11, 2024-04-01T01:50:27.551 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_active_connections": 2, 2024-04-01T01:50:27.551 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_total_time": 0.124515472, 2024-04-01T01:50:27.551 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_send_time": 0.052644357, 2024-04-01T01:50:27.551 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_recv_time": 0.091110101, 2024-04-01T01:50:27.551 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_fast_dispatch_time": 0.004951863, 2024-04-01T01:50:27.551 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:27.551 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 1246, 2024-04-01T01:50:27.551 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.092048577, 2024-04-01T01:50:27.551 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000073875 2024-04-01T01:50:27.551 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.552 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:27.552 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 2990, 2024-04-01T01:50:27.552 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000673921, 2024-04-01T01:50:27.552 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000225 2024-04-01T01:50:27.552 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.552 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-01T01:50:27.552 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_encrypted_bytes": 448 2024-04-01T01:50:27.552 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.552 INFO:teuthology.orchestra.run.smithi081.stdout: "AsyncMessenger::Worker-1": { 2024-04-01T01:50:27.552 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_messages": 384, 2024-04-01T01:50:27.552 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages": 370, 2024-04-01T01:50:27.552 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_bytes": 859159, 2024-04-01T01:50:27.552 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_bytes": 207858, 2024-04-01T01:50:27.552 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_created_connections": 9, 2024-04-01T01:50:27.553 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_active_connections": 5, 2024-04-01T01:50:27.553 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_total_time": 0.045551865, 2024-04-01T01:50:27.553 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_send_time": 0.014841618, 2024-04-01T01:50:27.553 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_recv_time": 0.026030500, 2024-04-01T01:50:27.553 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_fast_dispatch_time": 0.006800574, 2024-04-01T01:50:27.553 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:27.553 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 369, 2024-04-01T01:50:27.553 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.039627940, 2024-04-01T01:50:27.553 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000107392 2024-04-01T01:50:27.553 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.553 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:27.553 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.553 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.554 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.554 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_encrypted_bytes": 854992, 2024-04-01T01:50:27.554 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_encrypted_bytes": 200528 2024-04-01T01:50:27.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.554 INFO:teuthology.orchestra.run.smithi081.stdout: "AsyncMessenger::Worker-2": { 2024-04-01T01:50:27.554 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_messages": 93, 2024-04-01T01:50:27.554 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages": 335, 2024-04-01T01:50:27.554 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_bytes": 15978, 2024-04-01T01:50:27.554 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_bytes": 1291853, 2024-04-01T01:50:27.554 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_created_connections": 71, 2024-04-01T01:50:27.554 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_active_connections": 6, 2024-04-01T01:50:27.554 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_total_time": 0.036499836, 2024-04-01T01:50:27.555 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_send_time": 0.010900597, 2024-04-01T01:50:27.555 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_recv_time": 0.022880453, 2024-04-01T01:50:27.555 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_fast_dispatch_time": 0.000227494, 2024-04-01T01:50:27.555 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:27.555 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 329, 2024-04-01T01:50:27.555 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.015013840, 2024-04-01T01:50:27.555 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000045634 2024-04-01T01:50:27.555 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.555 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:27.555 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 109, 2024-04-01T01:50:27.555 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000015387, 2024-04-01T01:50:27.555 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000141 2024-04-01T01:50:27.555 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.555 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_encrypted_bytes": 2832, 2024-04-01T01:50:27.556 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_encrypted_bytes": 585040 2024-04-01T01:50:27.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.556 INFO:teuthology.orchestra.run.smithi081.stdout: "cct": { 2024-04-01T01:50:27.556 INFO:teuthology.orchestra.run.smithi081.stdout: "total_workers": 1, 2024-04-01T01:50:27.556 INFO:teuthology.orchestra.run.smithi081.stdout: "unhealthy_workers": 0 2024-04-01T01:50:27.556 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.556 INFO:teuthology.orchestra.run.smithi081.stdout: "finisher-MDSRank": { 2024-04-01T01:50:27.556 INFO:teuthology.orchestra.run.smithi081.stdout: "queue_len": 0, 2024-04-01T01:50:27.556 INFO:teuthology.orchestra.run.smithi081.stdout: "complete_latency": { 2024-04-01T01:50:27.556 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 56, 2024-04-01T01:50:27.556 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.001384271, 2024-04-01T01:50:27.556 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000024719 2024-04-01T01:50:27.556 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:27.557 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.557 INFO:teuthology.orchestra.run.smithi081.stdout: "finisher-PurgeQueue": { 2024-04-01T01:50:27.557 INFO:teuthology.orchestra.run.smithi081.stdout: "queue_len": 0, 2024-04-01T01:50:27.557 INFO:teuthology.orchestra.run.smithi081.stdout: "complete_latency": { 2024-04-01T01:50:27.557 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 2, 2024-04-01T01:50:27.557 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000006627, 2024-04-01T01:50:27.557 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000003313 2024-04-01T01:50:27.557 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:27.557 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.557 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": { 2024-04-01T01:50:27.557 INFO:teuthology.orchestra.run.smithi081.stdout: "request": 7, 2024-04-01T01:50:27.557 INFO:teuthology.orchestra.run.smithi081.stdout: "reply": 4, 2024-04-01T01:50:27.557 INFO:teuthology.orchestra.run.smithi081.stdout: "reply_latency": { 2024-04-01T01:50:27.557 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 4, 2024-04-01T01:50:27.557 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.001966876, 2024-04-01T01:50:27.558 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000491719 2024-04-01T01:50:27.558 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.558 INFO:teuthology.orchestra.run.smithi081.stdout: "slow_reply": 0, 2024-04-01T01:50:27.558 INFO:teuthology.orchestra.run.smithi081.stdout: "forward": 3, 2024-04-01T01:50:27.558 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_fetch_complete": 0, 2024-04-01T01:50:27.558 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_fetch_keys": 0, 2024-04-01T01:50:27.558 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_commit": 11, 2024-04-01T01:50:27.558 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_split": 0, 2024-04-01T01:50:27.558 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_merge": 0, 2024-04-01T01:50:27.558 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes": 14, 2024-04-01T01:50:27.558 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_top": 0, 2024-04-01T01:50:27.558 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_bottom": 14, 2024-04-01T01:50:27.558 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_pin_tail": 0, 2024-04-01T01:50:27.559 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_pinned": 14, 2024-04-01T01:50:27.559 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_expired": 0, 2024-04-01T01:50:27.559 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_with_caps": 1, 2024-04-01T01:50:27.559 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": 1, 2024-04-01T01:50:27.559 INFO:teuthology.orchestra.run.smithi081.stdout: "subtrees": 2, 2024-04-01T01:50:27.559 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse": 9, 2024-04-01T01:50:27.559 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_hit": 7, 2024-04-01T01:50:27.559 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_forward": 0, 2024-04-01T01:50:27.559 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_discover": 1, 2024-04-01T01:50:27.559 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_dir_fetch": 0, 2024-04-01T01:50:27.559 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_remote_ino": 0, 2024-04-01T01:50:27.559 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_lock": 0, 2024-04-01T01:50:27.559 INFO:teuthology.orchestra.run.smithi081.stdout: "load_cent": 0, 2024-04-01T01:50:27.559 INFO:teuthology.orchestra.run.smithi081.stdout: "q": 0, 2024-04-01T01:50:27.560 INFO:teuthology.orchestra.run.smithi081.stdout: "exported": 0, 2024-04-01T01:50:27.560 INFO:teuthology.orchestra.run.smithi081.stdout: "exported_inodes": 0, 2024-04-01T01:50:27.560 INFO:teuthology.orchestra.run.smithi081.stdout: "imported": 0, 2024-04-01T01:50:27.560 INFO:teuthology.orchestra.run.smithi081.stdout: "imported_inodes": 0, 2024-04-01T01:50:27.560 INFO:teuthology.orchestra.run.smithi081.stdout: "openino_dir_fetch": 0, 2024-04-01T01:50:27.560 INFO:teuthology.orchestra.run.smithi081.stdout: "openino_backtrace_fetch": 0, 2024-04-01T01:50:27.560 INFO:teuthology.orchestra.run.smithi081.stdout: "openino_peer_discover": 0, 2024-04-01T01:50:27.560 INFO:teuthology.orchestra.run.smithi081.stdout: "root_rfiles": 1022, 2024-04-01T01:50:27.560 INFO:teuthology.orchestra.run.smithi081.stdout: "root_rbytes": 1557107758, 2024-04-01T01:50:27.560 INFO:teuthology.orchestra.run.smithi081.stdout: "root_rsnaps": 0, 2024-04-01T01:50:27.560 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_backtrace_fetch": 0, 2024-04-01T01:50:27.560 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_set_tag": 0, 2024-04-01T01:50:27.560 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_backtrace_repaired": 0, 2024-04-01T01:50:27.560 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_inotable_repaired": 0, 2024-04-01T01:50:27.560 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_dir_inodes": 0, 2024-04-01T01:50:27.561 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_dir_base_inodes": 0, 2024-04-01T01:50:27.561 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_dirfrag_rstats": 0, 2024-04-01T01:50:27.561 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_file_inodes": 0, 2024-04-01T01:50:27.561 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_inode_file_caps": 0, 2024-04-01T01:50:27.561 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_revoke": 1, 2024-04-01T01:50:27.561 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_grant": 0, 2024-04-01T01:50:27.561 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_trunc": 0, 2024-04-01T01:50:27.561 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-01T01:50:27.561 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-01T01:50:27.561 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_caps": 1, 2024-04-01T01:50:27.561 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_caps_dirty": 0, 2024-04-01T01:50:27.561 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_cap_release": 0, 2024-04-01T01:50:27.561 INFO:teuthology.orchestra.run.smithi081.stdout: "process_request_cap_release": 1 2024-04-01T01:50:27.561 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.561 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache": { 2024-04-01T01:50:27.562 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_update": 0, 2024-04-01T01:50:27.562 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_update_receipt": 0, 2024-04-01T01:50:27.562 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_try_discover": 0, 2024-04-01T01:50:27.562 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_send_discover": 2, 2024-04-01T01:50:27.562 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_handle_discover": 0, 2024-04-01T01:50:27.562 INFO:teuthology.orchestra.run.smithi081.stdout: "num_strays": 0, 2024-04-01T01:50:27.562 INFO:teuthology.orchestra.run.smithi081.stdout: "num_strays_delayed": 0, 2024-04-01T01:50:27.562 INFO:teuthology.orchestra.run.smithi081.stdout: "num_strays_enqueuing": 0, 2024-04-01T01:50:27.562 INFO:teuthology.orchestra.run.smithi081.stdout: "strays_created": 0, 2024-04-01T01:50:27.562 INFO:teuthology.orchestra.run.smithi081.stdout: "strays_enqueued": 0, 2024-04-01T01:50:27.562 INFO:teuthology.orchestra.run.smithi081.stdout: "strays_reintegrated": 0, 2024-04-01T01:50:27.562 INFO:teuthology.orchestra.run.smithi081.stdout: "strays_migrated": 0, 2024-04-01T01:50:27.562 INFO:teuthology.orchestra.run.smithi081.stdout: "num_recovering_processing": 0, 2024-04-01T01:50:27.562 INFO:teuthology.orchestra.run.smithi081.stdout: "num_recovering_enqueued": 0, 2024-04-01T01:50:27.563 INFO:teuthology.orchestra.run.smithi081.stdout: "num_recovering_prioritized": 0, 2024-04-01T01:50:27.563 INFO:teuthology.orchestra.run.smithi081.stdout: "recovery_started": 0, 2024-04-01T01:50:27.563 INFO:teuthology.orchestra.run.smithi081.stdout: "recovery_completed": 0, 2024-04-01T01:50:27.563 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_quiesce_path": 0, 2024-04-01T01:50:27.563 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_quiesce_inode": 0, 2024-04-01T01:50:27.563 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_enqueue_scrub": 0, 2024-04-01T01:50:27.563 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_exportdir": 0, 2024-04-01T01:50:27.563 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_flush": 0, 2024-04-01T01:50:27.563 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_fragmentdir": 0, 2024-04-01T01:50:27.563 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_fragstats": 0, 2024-04-01T01:50:27.563 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_inodestats": 0 2024-04-01T01:50:27.563 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.563 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log": { 2024-04-01T01:50:27.563 INFO:teuthology.orchestra.run.smithi081.stdout: "evlrg": 0, 2024-04-01T01:50:27.564 INFO:teuthology.orchestra.run.smithi081.stdout: "evadd": 5, 2024-04-01T01:50:27.564 INFO:teuthology.orchestra.run.smithi081.stdout: "evex": 0, 2024-04-01T01:50:27.564 INFO:teuthology.orchestra.run.smithi081.stdout: "evtrm": 0, 2024-04-01T01:50:27.564 INFO:teuthology.orchestra.run.smithi081.stdout: "ev": 5, 2024-04-01T01:50:27.564 INFO:teuthology.orchestra.run.smithi081.stdout: "evexg": 0, 2024-04-01T01:50:27.564 INFO:teuthology.orchestra.run.smithi081.stdout: "evexd": 0, 2024-04-01T01:50:27.564 INFO:teuthology.orchestra.run.smithi081.stdout: "segadd": 2, 2024-04-01T01:50:27.564 INFO:teuthology.orchestra.run.smithi081.stdout: "segex": 0, 2024-04-01T01:50:27.564 INFO:teuthology.orchestra.run.smithi081.stdout: "segtrm": 0, 2024-04-01T01:50:27.564 INFO:teuthology.orchestra.run.smithi081.stdout: "seg": 2, 2024-04-01T01:50:27.564 INFO:teuthology.orchestra.run.smithi081.stdout: "segmjr": 2, 2024-04-01T01:50:27.564 INFO:teuthology.orchestra.run.smithi081.stdout: "segexg": 0, 2024-04-01T01:50:27.564 INFO:teuthology.orchestra.run.smithi081.stdout: "segexd": 0, 2024-04-01T01:50:27.564 INFO:teuthology.orchestra.run.smithi081.stdout: "expos": 4194304, 2024-04-01T01:50:27.564 INFO:teuthology.orchestra.run.smithi081.stdout: "wrpos": 4195901, 2024-04-01T01:50:27.565 INFO:teuthology.orchestra.run.smithi081.stdout: "rdpos": 0, 2024-04-01T01:50:27.565 INFO:teuthology.orchestra.run.smithi081.stdout: "jlat": { 2024-04-01T01:50:27.565 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 4, 2024-04-01T01:50:27.565 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.006899953, 2024-04-01T01:50:27.565 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.001724988 2024-04-01T01:50:27.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.565 INFO:teuthology.orchestra.run.smithi081.stdout: "replayed": 0 2024-04-01T01:50:27.565 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.565 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_mem": { 2024-04-01T01:50:27.565 INFO:teuthology.orchestra.run.smithi081.stdout: "ino": 17, 2024-04-01T01:50:27.565 INFO:teuthology.orchestra.run.smithi081.stdout: "ino+": 17, 2024-04-01T01:50:27.565 INFO:teuthology.orchestra.run.smithi081.stdout: "ino-": 0, 2024-04-01T01:50:27.565 INFO:teuthology.orchestra.run.smithi081.stdout: "dir": 15, 2024-04-01T01:50:27.565 INFO:teuthology.orchestra.run.smithi081.stdout: "dir+": 16, 2024-04-01T01:50:27.566 INFO:teuthology.orchestra.run.smithi081.stdout: "dir-": 1, 2024-04-01T01:50:27.566 INFO:teuthology.orchestra.run.smithi081.stdout: "dn": 14, 2024-04-01T01:50:27.566 INFO:teuthology.orchestra.run.smithi081.stdout: "dn+": 14, 2024-04-01T01:50:27.566 INFO:teuthology.orchestra.run.smithi081.stdout: "dn-": 0, 2024-04-01T01:50:27.566 INFO:teuthology.orchestra.run.smithi081.stdout: "cap": 1, 2024-04-01T01:50:27.566 INFO:teuthology.orchestra.run.smithi081.stdout: "cap+": 4, 2024-04-01T01:50:27.566 INFO:teuthology.orchestra.run.smithi081.stdout: "cap-": 3, 2024-04-01T01:50:27.566 INFO:teuthology.orchestra.run.smithi081.stdout: "rss": 52912, 2024-04-01T01:50:27.566 INFO:teuthology.orchestra.run.smithi081.stdout: "heap": 223612 2024-04-01T01:50:27.566 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.566 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_server": { 2024-04-01T01:50:27.566 INFO:teuthology.orchestra.run.smithi081.stdout: "dispatch_client_request": 8, 2024-04-01T01:50:27.566 INFO:teuthology.orchestra.run.smithi081.stdout: "dispatch_server_request": 0, 2024-04-01T01:50:27.566 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_request": 7, 2024-04-01T01:50:27.567 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_session": 17, 2024-04-01T01:50:27.567 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_peer_request": 0, 2024-04-01T01:50:27.567 INFO:teuthology.orchestra.run.smithi081.stdout: "req_create_latency": { 2024-04-01T01:50:27.567 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.567 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.567 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.567 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.567 INFO:teuthology.orchestra.run.smithi081.stdout: "req_getattr_latency": { 2024-04-01T01:50:27.567 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 2, 2024-04-01T01:50:27.567 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.001432855, 2024-04-01T01:50:27.567 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000716427 2024-04-01T01:50:27.567 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.567 INFO:teuthology.orchestra.run.smithi081.stdout: "req_getfilelock_latency": { 2024-04-01T01:50:27.567 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.568 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.568 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.568 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.568 INFO:teuthology.orchestra.run.smithi081.stdout: "req_link_latency": { 2024-04-01T01:50:27.568 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.568 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.568 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.568 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.568 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookup_latency": { 2024-04-01T01:50:27.568 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 2, 2024-04-01T01:50:27.568 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000534021, 2024-04-01T01:50:27.568 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000267010 2024-04-01T01:50:27.568 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.568 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookuphash_latency": { 2024-04-01T01:50:27.568 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.569 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.569 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.569 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.569 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookupino_latency": { 2024-04-01T01:50:27.569 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.569 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.569 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.569 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.569 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookupname_latency": { 2024-04-01T01:50:27.569 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.569 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.569 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.569 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.569 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookupparent_latency": { 2024-04-01T01:50:27.570 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.570 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.570 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.570 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookupsnap_latency": { 2024-04-01T01:50:27.570 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.570 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.570 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.570 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lssnap_latency": { 2024-04-01T01:50:27.570 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.570 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.570 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.570 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.571 INFO:teuthology.orchestra.run.smithi081.stdout: "req_mkdir_latency": { 2024-04-01T01:50:27.571 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.571 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.571 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.571 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.571 INFO:teuthology.orchestra.run.smithi081.stdout: "req_mknod_latency": { 2024-04-01T01:50:27.572 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.572 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.572 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.572 INFO:teuthology.orchestra.run.smithi081.stdout: "req_mksnap_latency": { 2024-04-01T01:50:27.572 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.572 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.572 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.572 INFO:teuthology.orchestra.run.smithi081.stdout: "req_open_latency": { 2024-04-01T01:50:27.572 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.572 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.572 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.572 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.573 INFO:teuthology.orchestra.run.smithi081.stdout: "req_readdir_latency": { 2024-04-01T01:50:27.573 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.573 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.573 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.573 INFO:teuthology.orchestra.run.smithi081.stdout: "req_rename_latency": { 2024-04-01T01:50:27.573 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.573 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.573 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.573 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.573 INFO:teuthology.orchestra.run.smithi081.stdout: "req_renamesnap_latency": { 2024-04-01T01:50:27.573 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.573 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.574 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.574 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.574 INFO:teuthology.orchestra.run.smithi081.stdout: "req_snapdiff_latency": { 2024-04-01T01:50:27.574 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.574 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.574 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.574 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.574 INFO:teuthology.orchestra.run.smithi081.stdout: "req_rmdir_latency": { 2024-04-01T01:50:27.574 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.574 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.574 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.574 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.574 INFO:teuthology.orchestra.run.smithi081.stdout: "req_rmsnap_latency": { 2024-04-01T01:50:27.575 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.575 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.575 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.575 INFO:teuthology.orchestra.run.smithi081.stdout: "req_rmxattr_latency": { 2024-04-01T01:50:27.575 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.575 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.575 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.575 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setattr_latency": { 2024-04-01T01:50:27.575 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.575 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.575 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.575 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.576 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setdirlayout_latency": { 2024-04-01T01:50:27.576 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.576 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.576 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.576 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.576 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setfilelock_latency": { 2024-04-01T01:50:27.576 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.576 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.576 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.576 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.576 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setlayout_latency": { 2024-04-01T01:50:27.576 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.576 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.576 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.576 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.577 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setxattr_latency": { 2024-04-01T01:50:27.577 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.577 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.577 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.577 INFO:teuthology.orchestra.run.smithi081.stdout: "req_symlink_latency": { 2024-04-01T01:50:27.577 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.577 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.577 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.577 INFO:teuthology.orchestra.run.smithi081.stdout: "req_unlink_latency": { 2024-04-01T01:50:27.577 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.577 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.577 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.577 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.578 INFO:teuthology.orchestra.run.smithi081.stdout: "cap_revoke_eviction": 0, 2024-04-01T01:50:27.578 INFO:teuthology.orchestra.run.smithi081.stdout: "cap_acquisition_throttle": 0, 2024-04-01T01:50:27.578 INFO:teuthology.orchestra.run.smithi081.stdout: "req_getvxattr_latency": { 2024-04-01T01:50:27.578 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.578 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.578 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.578 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:27.578 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.578 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_sessions": { 2024-04-01T01:50:27.578 INFO:teuthology.orchestra.run.smithi081.stdout: "session_count": 1, 2024-04-01T01:50:27.578 INFO:teuthology.orchestra.run.smithi081.stdout: "session_add": 2, 2024-04-01T01:50:27.578 INFO:teuthology.orchestra.run.smithi081.stdout: "session_remove": 1, 2024-04-01T01:50:27.578 INFO:teuthology.orchestra.run.smithi081.stdout: "sessions_open": 1, 2024-04-01T01:50:27.578 INFO:teuthology.orchestra.run.smithi081.stdout: "sessions_stale": 0, 2024-04-01T01:50:27.579 INFO:teuthology.orchestra.run.smithi081.stdout: "total_load": 3, 2024-04-01T01:50:27.579 INFO:teuthology.orchestra.run.smithi081.stdout: "average_load": 1, 2024-04-01T01:50:27.579 INFO:teuthology.orchestra.run.smithi081.stdout: "avg_session_uptime": 1011, 2024-04-01T01:50:27.579 INFO:teuthology.orchestra.run.smithi081.stdout: "mdthresh_evicted": 0 2024-04-01T01:50:27.579 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.579 INFO:teuthology.orchestra.run.smithi081.stdout: "mempool": { 2024-04-01T01:50:27.579 INFO:teuthology.orchestra.run.smithi081.stdout: "bloom_filter_bytes": 0, 2024-04-01T01:50:27.579 INFO:teuthology.orchestra.run.smithi081.stdout: "bloom_filter_items": 0, 2024-04-01T01:50:27.579 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_alloc_bytes": 0, 2024-04-01T01:50:27.579 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_alloc_items": 0, 2024-04-01T01:50:27.579 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_data_bytes": 0, 2024-04-01T01:50:27.579 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_data_items": 0, 2024-04-01T01:50:27.579 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-01T01:50:27.579 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_onode_items": 0, 2024-04-01T01:50:27.580 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-01T01:50:27.580 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_meta_items": 0, 2024-04-01T01:50:27.580 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_other_bytes": 0, 2024-04-01T01:50:27.580 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_other_items": 0, 2024-04-01T01:50:27.580 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-01T01:50:27.580 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_buffer_items": 0, 2024-04-01T01:50:27.580 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_bytes": 0, 2024-04-01T01:50:27.580 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_items": 0, 2024-04-01T01:50:27.580 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_blob_bytes": 0, 2024-04-01T01:50:27.580 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_blob_items": 0, 2024-04-01T01:50:27.580 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-01T01:50:27.580 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_shared_blob_items": 0, 2024-04-01T01:50:27.580 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-01T01:50:27.580 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_inline_bl_items": 0, 2024-04-01T01:50:27.581 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_bytes": 0, 2024-04-01T01:50:27.581 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_items": 0, 2024-04-01T01:50:27.581 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_txc_bytes": 0, 2024-04-01T01:50:27.581 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_txc_items": 0, 2024-04-01T01:50:27.581 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-01T01:50:27.581 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_writing_deferred_items": 0, 2024-04-01T01:50:27.581 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_writing_bytes": 0, 2024-04-01T01:50:27.581 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_writing_items": 0, 2024-04-01T01:50:27.581 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_bytes": 0, 2024-04-01T01:50:27.581 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_items": 0, 2024-04-01T01:50:27.581 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_file_reader_bytes": 0, 2024-04-01T01:50:27.581 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_file_reader_items": 0, 2024-04-01T01:50:27.581 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_file_writer_bytes": 0, 2024-04-01T01:50:27.581 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_file_writer_items": 0, 2024-04-01T01:50:27.582 INFO:teuthology.orchestra.run.smithi081.stdout: "buffer_anon_bytes": 248399, 2024-04-01T01:50:27.582 INFO:teuthology.orchestra.run.smithi081.stdout: "buffer_anon_items": 89, 2024-04-01T01:50:27.582 INFO:teuthology.orchestra.run.smithi081.stdout: "buffer_meta_bytes": 176, 2024-04-01T01:50:27.582 INFO:teuthology.orchestra.run.smithi081.stdout: "buffer_meta_items": 2, 2024-04-01T01:50:27.582 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_bytes": 0, 2024-04-01T01:50:27.582 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_items": 0, 2024-04-01T01:50:27.582 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mapbl_bytes": 0, 2024-04-01T01:50:27.582 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mapbl_items": 0, 2024-04-01T01:50:27.582 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pglog_bytes": 0, 2024-04-01T01:50:27.582 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pglog_items": 0, 2024-04-01T01:50:27.582 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_bytes": 9080, 2024-04-01T01:50:27.582 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_items": 174, 2024-04-01T01:50:27.582 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_mapping_bytes": 0, 2024-04-01T01:50:27.582 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_mapping_items": 0, 2024-04-01T01:50:27.583 INFO:teuthology.orchestra.run.smithi081.stdout: "pgmap_bytes": 0, 2024-04-01T01:50:27.583 INFO:teuthology.orchestra.run.smithi081.stdout: "pgmap_items": 0, 2024-04-01T01:50:27.583 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_co_bytes": 67829, 2024-04-01T01:50:27.583 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_co_items": 296, 2024-04-01T01:50:27.583 INFO:teuthology.orchestra.run.smithi081.stdout: "unittest_1_bytes": 0, 2024-04-01T01:50:27.583 INFO:teuthology.orchestra.run.smithi081.stdout: "unittest_1_items": 0, 2024-04-01T01:50:27.583 INFO:teuthology.orchestra.run.smithi081.stdout: "unittest_2_bytes": 0, 2024-04-01T01:50:27.583 INFO:teuthology.orchestra.run.smithi081.stdout: "unittest_2_items": 0 2024-04-01T01:50:27.583 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.583 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter": { 2024-04-01T01:50:27.583 INFO:teuthology.orchestra.run.smithi081.stdout: "op_active": 0, 2024-04-01T01:50:27.583 INFO:teuthology.orchestra.run.smithi081.stdout: "op_laggy": 0, 2024-04-01T01:50:27.583 INFO:teuthology.orchestra.run.smithi081.stdout: "op_send": 43, 2024-04-01T01:50:27.583 INFO:teuthology.orchestra.run.smithi081.stdout: "op_send_bytes": 12478, 2024-04-01T01:50:27.583 INFO:teuthology.orchestra.run.smithi081.stdout: "op_resend": 33, 2024-04-01T01:50:27.584 INFO:teuthology.orchestra.run.smithi081.stdout: "op_reply": 43, 2024-04-01T01:50:27.584 INFO:teuthology.orchestra.run.smithi081.stdout: "op_latency": { 2024-04-01T01:50:27.584 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 43, 2024-04-01T01:50:27.584 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.203995920, 2024-04-01T01:50:27.584 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.004744091 2024-04-01T01:50:27.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.584 INFO:teuthology.orchestra.run.smithi081.stdout: "op_inflight": 0, 2024-04-01T01:50:27.584 INFO:teuthology.orchestra.run.smithi081.stdout: "oplen_avg": { 2024-04-01T01:50:27.584 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 43, 2024-04-01T01:50:27.584 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 68 2024-04-01T01:50:27.584 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.584 INFO:teuthology.orchestra.run.smithi081.stdout: "op": 43, 2024-04-01T01:50:27.584 INFO:teuthology.orchestra.run.smithi081.stdout: "op_r": 0, 2024-04-01T01:50:27.584 INFO:teuthology.orchestra.run.smithi081.stdout: "op_w": 43, 2024-04-01T01:50:27.585 INFO:teuthology.orchestra.run.smithi081.stdout: "op_rmw": 0, 2024-04-01T01:50:27.585 INFO:teuthology.orchestra.run.smithi081.stdout: "op_pg": 0, 2024-04-01T01:50:27.585 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_stat": 0, 2024-04-01T01:50:27.585 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_create": 10, 2024-04-01T01:50:27.585 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_read": 0, 2024-04-01T01:50:27.585 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_write": 4, 2024-04-01T01:50:27.585 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_writefull": 7, 2024-04-01T01:50:27.585 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_writesame": 0, 2024-04-01T01:50:27.585 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_append": 0, 2024-04-01T01:50:27.585 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_zero": 0, 2024-04-01T01:50:27.585 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_truncate": 0, 2024-04-01T01:50:27.585 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_delete": 6, 2024-04-01T01:50:27.585 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_mapext": 0, 2024-04-01T01:50:27.585 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_sparse_read": 0, 2024-04-01T01:50:27.585 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_clonerange": 0, 2024-04-01T01:50:27.586 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_getxattr": 0, 2024-04-01T01:50:27.586 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_setxattr": 20, 2024-04-01T01:50:27.586 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_cmpxattr": 0, 2024-04-01T01:50:27.586 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_rmxattr": 0, 2024-04-01T01:50:27.586 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_resetxattrs": 0, 2024-04-01T01:50:27.586 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_call": 0, 2024-04-01T01:50:27.586 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_watch": 0, 2024-04-01T01:50:27.586 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_notify": 0, 2024-04-01T01:50:27.586 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_src_cmpxattr": 0, 2024-04-01T01:50:27.586 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_pgls": 0, 2024-04-01T01:50:27.586 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_pgls_filter": 0, 2024-04-01T01:50:27.586 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_other": 0, 2024-04-01T01:50:27.586 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_active": 0, 2024-04-01T01:50:27.586 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_send": 0, 2024-04-01T01:50:27.587 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_resend": 0, 2024-04-01T01:50:27.587 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_ping": 0, 2024-04-01T01:50:27.587 INFO:teuthology.orchestra.run.smithi081.stdout: "poolop_active": 0, 2024-04-01T01:50:27.587 INFO:teuthology.orchestra.run.smithi081.stdout: "poolop_send": 0, 2024-04-01T01:50:27.587 INFO:teuthology.orchestra.run.smithi081.stdout: "poolop_resend": 0, 2024-04-01T01:50:27.587 INFO:teuthology.orchestra.run.smithi081.stdout: "poolstat_active": 0, 2024-04-01T01:50:27.587 INFO:teuthology.orchestra.run.smithi081.stdout: "poolstat_send": 0, 2024-04-01T01:50:27.587 INFO:teuthology.orchestra.run.smithi081.stdout: "poolstat_resend": 0, 2024-04-01T01:50:27.587 INFO:teuthology.orchestra.run.smithi081.stdout: "statfs_active": 0, 2024-04-01T01:50:27.587 INFO:teuthology.orchestra.run.smithi081.stdout: "statfs_send": 0, 2024-04-01T01:50:27.587 INFO:teuthology.orchestra.run.smithi081.stdout: "statfs_resend": 0, 2024-04-01T01:50:27.587 INFO:teuthology.orchestra.run.smithi081.stdout: "command_active": 0, 2024-04-01T01:50:27.587 INFO:teuthology.orchestra.run.smithi081.stdout: "command_send": 0, 2024-04-01T01:50:27.587 INFO:teuthology.orchestra.run.smithi081.stdout: "command_resend": 0, 2024-04-01T01:50:27.588 INFO:teuthology.orchestra.run.smithi081.stdout: "map_epoch": 95, 2024-04-01T01:50:27.588 INFO:teuthology.orchestra.run.smithi081.stdout: "map_full": 0, 2024-04-01T01:50:27.588 INFO:teuthology.orchestra.run.smithi081.stdout: "map_inc": 6, 2024-04-01T01:50:27.588 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_sessions": 9, 2024-04-01T01:50:27.588 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_session_open": 15, 2024-04-01T01:50:27.588 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_session_close": 6, 2024-04-01T01:50:27.588 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_laggy": 0, 2024-04-01T01:50:27.588 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_wr": 20, 2024-04-01T01:50:27.588 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_rd": 0, 2024-04-01T01:50:27.588 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_del": 1 2024-04-01T01:50:27.588 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.588 INFO:teuthology.orchestra.run.smithi081.stdout: "oft": { 2024-04-01T01:50:27.588 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_total_objs": 1, 2024-04-01T01:50:27.588 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_total_kv_pairs": 0, 2024-04-01T01:50:27.588 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_total_updates": 0, 2024-04-01T01:50:27.589 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_total_removes": 0 2024-04-01T01:50:27.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.589 INFO:teuthology.orchestra.run.smithi081.stdout: "osd-slow-ops": { 2024-04-01T01:50:27.589 INFO:teuthology.orchestra.run.smithi081.stdout: "slow_ops_count": 0 2024-04-01T01:50:27.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.589 INFO:teuthology.orchestra.run.smithi081.stdout: "purge_queue": { 2024-04-01T01:50:27.589 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executing_ops": 0, 2024-04-01T01:50:27.589 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executing_ops_high_water": 0, 2024-04-01T01:50:27.589 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executing": 0, 2024-04-01T01:50:27.589 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executing_high_water": 0, 2024-04-01T01:50:27.589 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executed_ops": 0, 2024-04-01T01:50:27.589 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executed": 0, 2024-04-01T01:50:27.589 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_item_in_journal": 0 2024-04-01T01:50:27.589 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.589 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-01T01:50:27.590 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:27.590 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 104857600, 2024-04-01T01:50:27.590 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:27.590 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 2475, 2024-04-01T01:50:27.590 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 1118716, 2024-04-01T01:50:27.590 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:27.590 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 2475, 2024-04-01T01:50:27.590 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 0, 2024-04-01T01:50:27.590 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 0, 2024-04-01T01:50:27.590 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 2475, 2024-04-01T01:50:27.590 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 1118716, 2024-04-01T01:50:27.590 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:27.590 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.590 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.591 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.591 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:27.591 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.591 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-objecter_bytes": { 2024-04-01T01:50:27.591 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:27.591 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 104857600, 2024-04-01T01:50:27.591 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:27.591 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 0, 2024-04-01T01:50:27.591 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 0, 2024-04-01T01:50:27.591 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:27.591 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:27.591 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 43, 2024-04-01T01:50:27.591 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 12478, 2024-04-01T01:50:27.591 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 43, 2024-04-01T01:50:27.591 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 12478, 2024-04-01T01:50:27.592 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:27.592 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.592 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.592 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.592 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:27.592 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.592 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-objecter_ops": { 2024-04-01T01:50:27.592 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:27.592 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 1024, 2024-04-01T01:50:27.592 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:27.592 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 0, 2024-04-01T01:50:27.592 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 0, 2024-04-01T01:50:27.592 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:27.592 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:27.593 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 43, 2024-04-01T01:50:27.593 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 43, 2024-04-01T01:50:27.593 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 43, 2024-04-01T01:50:27.593 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 43, 2024-04-01T01:50:27.593 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:27.593 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.593 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.593 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.593 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:27.593 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.593 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-write_buf_throttle": { 2024-04-01T01:50:27.593 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:27.593 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 3758096384, 2024-04-01T01:50:27.593 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:27.594 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 0, 2024-04-01T01:50:27.594 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 0, 2024-04-01T01:50:27.594 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:27.594 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:27.594 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 0, 2024-04-01T01:50:27.594 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 0, 2024-04-01T01:50:27.594 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 0, 2024-04-01T01:50:27.594 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 0, 2024-04-01T01:50:27.594 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:27.594 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.594 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.595 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.595 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:27.595 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:27.595 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-write_buf_throttle-0x561261ac0500": { 2024-04-01T01:50:27.595 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:27.595 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 3758096384, 2024-04-01T01:50:27.595 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:27.595 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 5, 2024-04-01T01:50:27.595 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 1597, 2024-04-01T01:50:27.595 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:27.595 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 5, 2024-04-01T01:50:27.595 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 0, 2024-04-01T01:50:27.595 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 0, 2024-04-01T01:50:27.595 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 4, 2024-04-01T01:50:27.596 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 1597, 2024-04-01T01:50:27.596 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:27.596 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:27.596 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:27.596 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:27.596 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:27.596 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:27.596 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-01T01:50:28.402 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-01T01:50:28.402 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-01T01:50:28.402 DEBUG:tasks.check_counter:Getting stats from h 2024-04-01T01:50:28.402 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph daemon mds.h perf dump 2024-04-01T01:50:28.672 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-01T01:50:29.078 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:28 smithi081 ceph-mon[35142]: pgmap v869: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 853 B/s wr, 11 op/s 2024-04-01T01:50:29.174 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:28 smithi070 ceph-mon[32179]: pgmap v869: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 853 B/s wr, 11 op/s 2024-04-01T01:50:29.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:28 smithi189 ceph-mon[35753]: pgmap v869: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 853 B/s wr, 11 op/s 2024-04-01T01:50:30.512 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-01T01:50:30.512 INFO:teuthology.orchestra.run.smithi081.stdout: "AsyncMessenger::Worker-0": { 2024-04-01T01:50:30.512 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_messages": 1469, 2024-04-01T01:50:30.512 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages": 1452, 2024-04-01T01:50:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_bytes": 1239680, 2024-04-01T01:50:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_bytes": 553470, 2024-04-01T01:50:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_created_connections": 5, 2024-04-01T01:50:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_active_connections": 1, 2024-04-01T01:50:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_total_time": 0.141746262, 2024-04-01T01:50:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_send_time": 0.050278429, 2024-04-01T01:50:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_recv_time": 0.046142997, 2024-04-01T01:50:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_fast_dispatch_time": 0.039548585, 2024-04-01T01:50:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 1451, 2024-04-01T01:50:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.064642048, 2024-04-01T01:50:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000044549 2024-04-01T01:50:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.513 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_encrypted_bytes": 860336, 2024-04-01T01:50:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_encrypted_bytes": 194448 2024-04-01T01:50:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "AsyncMessenger::Worker-1": { 2024-04-01T01:50:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_messages": 8, 2024-04-01T01:50:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages": 243, 2024-04-01T01:50:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_bytes": 3868, 2024-04-01T01:50:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_bytes": 584154, 2024-04-01T01:50:30.514 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_created_connections": 5, 2024-04-01T01:50:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_active_connections": 3, 2024-04-01T01:50:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_total_time": 0.012180432, 2024-04-01T01:50:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_send_time": 0.008766150, 2024-04-01T01:50:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_recv_time": 0.001794415, 2024-04-01T01:50:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_fast_dispatch_time": 0.000039459, 2024-04-01T01:50:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 242, 2024-04-01T01:50:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.014759917, 2024-04-01T01:50:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000060991 2024-04-01T01:50:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.515 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-01T01:50:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_encrypted_bytes": 583456 2024-04-01T01:50:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "AsyncMessenger::Worker-2": { 2024-04-01T01:50:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_messages": 2193, 2024-04-01T01:50:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages": 2194, 2024-04-01T01:50:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_bytes": 559972, 2024-04-01T01:50:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_bytes": 697063, 2024-04-01T01:50:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_created_connections": 5, 2024-04-01T01:50:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_active_connections": 3, 2024-04-01T01:50:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_total_time": 0.119363939, 2024-04-01T01:50:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_send_time": 0.027352171, 2024-04-01T01:50:30.516 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_recv_time": 0.031916740, 2024-04-01T01:50:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_fast_dispatch_time": 0.053441814, 2024-04-01T01:50:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 2192, 2024-04-01T01:50:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.084185698, 2024-04-01T01:50:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000038405 2024-04-01T01:50:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-01T01:50:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_encrypted_bytes": 0 2024-04-01T01:50:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.517 INFO:teuthology.orchestra.run.smithi081.stdout: "cct": { 2024-04-01T01:50:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "total_workers": 1, 2024-04-01T01:50:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "unhealthy_workers": 0 2024-04-01T01:50:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "finisher-MDSRank": { 2024-04-01T01:50:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "queue_len": 0, 2024-04-01T01:50:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "complete_latency": { 2024-04-01T01:50:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 5531, 2024-04-01T01:50:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.123711225, 2024-04-01T01:50:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000022366 2024-04-01T01:50:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "finisher-PurgeQueue": { 2024-04-01T01:50:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "queue_len": 0, 2024-04-01T01:50:30.518 INFO:teuthology.orchestra.run.smithi081.stdout: "complete_latency": { 2024-04-01T01:50:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": { 2024-04-01T01:50:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "request": 0, 2024-04-01T01:50:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "reply": 0, 2024-04-01T01:50:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "reply_latency": { 2024-04-01T01:50:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "slow_reply": 0, 2024-04-01T01:50:30.519 INFO:teuthology.orchestra.run.smithi081.stdout: "forward": 0, 2024-04-01T01:50:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_fetch_complete": 0, 2024-04-01T01:50:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_fetch_keys": 0, 2024-04-01T01:50:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_commit": 0, 2024-04-01T01:50:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_split": 0, 2024-04-01T01:50:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_merge": 0, 2024-04-01T01:50:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes": 0, 2024-04-01T01:50:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_top": 0, 2024-04-01T01:50:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_bottom": 0, 2024-04-01T01:50:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_pin_tail": 0, 2024-04-01T01:50:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_pinned": 0, 2024-04-01T01:50:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_expired": 0, 2024-04-01T01:50:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_with_caps": 0, 2024-04-01T01:50:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": 0, 2024-04-01T01:50:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "subtrees": 1, 2024-04-01T01:50:30.520 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse": 0, 2024-04-01T01:50:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_hit": 0, 2024-04-01T01:50:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_forward": 0, 2024-04-01T01:50:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_discover": 0, 2024-04-01T01:50:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_dir_fetch": 0, 2024-04-01T01:50:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_remote_ino": 0, 2024-04-01T01:50:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_lock": 0, 2024-04-01T01:50:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "load_cent": 0, 2024-04-01T01:50:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "q": 0, 2024-04-01T01:50:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "exported": 0, 2024-04-01T01:50:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "exported_inodes": 0, 2024-04-01T01:50:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "imported": 0, 2024-04-01T01:50:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "imported_inodes": 0, 2024-04-01T01:50:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "openino_dir_fetch": 0, 2024-04-01T01:50:30.521 INFO:teuthology.orchestra.run.smithi081.stdout: "openino_backtrace_fetch": 0, 2024-04-01T01:50:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "openino_peer_discover": 0, 2024-04-01T01:50:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "root_rfiles": 0, 2024-04-01T01:50:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "root_rbytes": 0, 2024-04-01T01:50:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "root_rsnaps": 0, 2024-04-01T01:50:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_backtrace_fetch": 0, 2024-04-01T01:50:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_set_tag": 0, 2024-04-01T01:50:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_backtrace_repaired": 0, 2024-04-01T01:50:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_inotable_repaired": 0, 2024-04-01T01:50:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_dir_inodes": 0, 2024-04-01T01:50:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_dir_base_inodes": 0, 2024-04-01T01:50:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_dirfrag_rstats": 0, 2024-04-01T01:50:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_file_inodes": 0, 2024-04-01T01:50:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_inode_file_caps": 0, 2024-04-01T01:50:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_revoke": 0, 2024-04-01T01:50:30.522 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_grant": 0, 2024-04-01T01:50:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_trunc": 0, 2024-04-01T01:50:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-01T01:50:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-01T01:50:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_caps": 0, 2024-04-01T01:50:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_caps_dirty": 0, 2024-04-01T01:50:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_cap_release": 0, 2024-04-01T01:50:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "process_request_cap_release": 0 2024-04-01T01:50:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache": { 2024-04-01T01:50:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_update": 0, 2024-04-01T01:50:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_update_receipt": 0, 2024-04-01T01:50:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_try_discover": 0, 2024-04-01T01:50:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_send_discover": 0, 2024-04-01T01:50:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_handle_discover": 0, 2024-04-01T01:50:30.523 INFO:teuthology.orchestra.run.smithi081.stdout: "num_strays": 0, 2024-04-01T01:50:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "num_strays_delayed": 0, 2024-04-01T01:50:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "num_strays_enqueuing": 0, 2024-04-01T01:50:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "strays_created": 0, 2024-04-01T01:50:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "strays_enqueued": 0, 2024-04-01T01:50:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "strays_reintegrated": 0, 2024-04-01T01:50:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "strays_migrated": 0, 2024-04-01T01:50:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "num_recovering_processing": 0, 2024-04-01T01:50:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "num_recovering_enqueued": 0, 2024-04-01T01:50:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "num_recovering_prioritized": 0, 2024-04-01T01:50:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "recovery_started": 0, 2024-04-01T01:50:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "recovery_completed": 0, 2024-04-01T01:50:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_quiesce_path": 0, 2024-04-01T01:50:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_quiesce_inode": 0, 2024-04-01T01:50:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_enqueue_scrub": 0, 2024-04-01T01:50:30.524 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_exportdir": 0, 2024-04-01T01:50:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_flush": 0, 2024-04-01T01:50:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_fragmentdir": 0, 2024-04-01T01:50:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_fragstats": 0, 2024-04-01T01:50:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_inodestats": 0 2024-04-01T01:50:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log": { 2024-04-01T01:50:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "evlrg": 0, 2024-04-01T01:50:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "evadd": 0, 2024-04-01T01:50:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "evex": 0, 2024-04-01T01:50:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "evtrm": 0, 2024-04-01T01:50:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "ev": 5, 2024-04-01T01:50:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "evexg": 0, 2024-04-01T01:50:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "evexd": 0, 2024-04-01T01:50:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "segadd": 0, 2024-04-01T01:50:30.525 INFO:teuthology.orchestra.run.smithi081.stdout: "segex": 0, 2024-04-01T01:50:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "segtrm": 0, 2024-04-01T01:50:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "seg": 2, 2024-04-01T01:50:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "segmjr": 2, 2024-04-01T01:50:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "segexg": 0, 2024-04-01T01:50:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "segexd": 0, 2024-04-01T01:50:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "expos": 4194304, 2024-04-01T01:50:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "wrpos": 4195901, 2024-04-01T01:50:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "rdpos": 4195424, 2024-04-01T01:50:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "jlat": { 2024-04-01T01:50:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: "replayed": 5 2024-04-01T01:50:30.526 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_mem": { 2024-04-01T01:50:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "ino": 3, 2024-04-01T01:50:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "ino+": 3, 2024-04-01T01:50:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "ino-": 0, 2024-04-01T01:50:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "dir": 1, 2024-04-01T01:50:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "dir+": 1, 2024-04-01T01:50:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "dir-": 0, 2024-04-01T01:50:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "dn": 0, 2024-04-01T01:50:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "dn+": 0, 2024-04-01T01:50:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "dn-": 0, 2024-04-01T01:50:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "cap": 0, 2024-04-01T01:50:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "cap+": 0, 2024-04-01T01:50:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "cap-": 0, 2024-04-01T01:50:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "rss": 50284, 2024-04-01T01:50:30.527 INFO:teuthology.orchestra.run.smithi081.stdout: "heap": 215420 2024-04-01T01:50:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_server": { 2024-04-01T01:50:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "dispatch_client_request": 0, 2024-04-01T01:50:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "dispatch_server_request": 0, 2024-04-01T01:50:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_request": 0, 2024-04-01T01:50:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_session": 0, 2024-04-01T01:50:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_peer_request": 0, 2024-04-01T01:50:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "req_create_latency": { 2024-04-01T01:50:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "req_getattr_latency": { 2024-04-01T01:50:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.528 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "req_getfilelock_latency": { 2024-04-01T01:50:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "req_link_latency": { 2024-04-01T01:50:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookup_latency": { 2024-04-01T01:50:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.529 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookuphash_latency": { 2024-04-01T01:50:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookupino_latency": { 2024-04-01T01:50:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookupname_latency": { 2024-04-01T01:50:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.530 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookupparent_latency": { 2024-04-01T01:50:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookupsnap_latency": { 2024-04-01T01:50:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lssnap_latency": { 2024-04-01T01:50:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.531 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: "req_mkdir_latency": { 2024-04-01T01:50:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.532 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "req_mknod_latency": { 2024-04-01T01:50:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "req_mksnap_latency": { 2024-04-01T01:50:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "req_open_latency": { 2024-04-01T01:50:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.533 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "req_readdir_latency": { 2024-04-01T01:50:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "req_rename_latency": { 2024-04-01T01:50:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "req_renamesnap_latency": { 2024-04-01T01:50:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.534 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "req_snapdiff_latency": { 2024-04-01T01:50:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "req_rmdir_latency": { 2024-04-01T01:50:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "req_rmsnap_latency": { 2024-04-01T01:50:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.535 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "req_rmxattr_latency": { 2024-04-01T01:50:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setattr_latency": { 2024-04-01T01:50:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setdirlayout_latency": { 2024-04-01T01:50:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.536 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setfilelock_latency": { 2024-04-01T01:50:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setlayout_latency": { 2024-04-01T01:50:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setxattr_latency": { 2024-04-01T01:50:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.537 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "req_symlink_latency": { 2024-04-01T01:50:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "req_unlink_latency": { 2024-04-01T01:50:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "cap_revoke_eviction": 0, 2024-04-01T01:50:30.538 INFO:teuthology.orchestra.run.smithi081.stdout: "cap_acquisition_throttle": 0, 2024-04-01T01:50:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "req_getvxattr_latency": { 2024-04-01T01:50:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_sessions": { 2024-04-01T01:50:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "session_count": 1, 2024-04-01T01:50:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "session_add": 2, 2024-04-01T01:50:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "session_remove": 1, 2024-04-01T01:50:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "sessions_open": 1, 2024-04-01T01:50:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "sessions_stale": 0, 2024-04-01T01:50:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "total_load": 0, 2024-04-01T01:50:30.539 INFO:teuthology.orchestra.run.smithi081.stdout: "average_load": 0, 2024-04-01T01:50:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "avg_session_uptime": 2172, 2024-04-01T01:50:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "mdthresh_evicted": 0 2024-04-01T01:50:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "mempool": { 2024-04-01T01:50:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "bloom_filter_bytes": 0, 2024-04-01T01:50:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "bloom_filter_items": 0, 2024-04-01T01:50:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_alloc_bytes": 0, 2024-04-01T01:50:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_alloc_items": 0, 2024-04-01T01:50:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_data_bytes": 0, 2024-04-01T01:50:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_data_items": 0, 2024-04-01T01:50:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-01T01:50:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_onode_items": 0, 2024-04-01T01:50:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-01T01:50:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_meta_items": 0, 2024-04-01T01:50:30.540 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_other_bytes": 0, 2024-04-01T01:50:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_other_items": 0, 2024-04-01T01:50:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-01T01:50:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_buffer_items": 0, 2024-04-01T01:50:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_bytes": 0, 2024-04-01T01:50:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_items": 0, 2024-04-01T01:50:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_blob_bytes": 0, 2024-04-01T01:50:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_blob_items": 0, 2024-04-01T01:50:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-01T01:50:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_shared_blob_items": 0, 2024-04-01T01:50:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-01T01:50:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_inline_bl_items": 0, 2024-04-01T01:50:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_bytes": 0, 2024-04-01T01:50:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_items": 0, 2024-04-01T01:50:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_txc_bytes": 0, 2024-04-01T01:50:30.541 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_txc_items": 0, 2024-04-01T01:50:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-01T01:50:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_writing_deferred_items": 0, 2024-04-01T01:50:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_writing_bytes": 0, 2024-04-01T01:50:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_writing_items": 0, 2024-04-01T01:50:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_bytes": 0, 2024-04-01T01:50:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_items": 0, 2024-04-01T01:50:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_file_reader_bytes": 0, 2024-04-01T01:50:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_file_reader_items": 0, 2024-04-01T01:50:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_file_writer_bytes": 0, 2024-04-01T01:50:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_file_writer_items": 0, 2024-04-01T01:50:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "buffer_anon_bytes": 182105, 2024-04-01T01:50:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "buffer_anon_items": 49, 2024-04-01T01:50:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "buffer_meta_bytes": 0, 2024-04-01T01:50:30.542 INFO:teuthology.orchestra.run.smithi081.stdout: "buffer_meta_items": 0, 2024-04-01T01:50:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_bytes": 0, 2024-04-01T01:50:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_items": 0, 2024-04-01T01:50:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mapbl_bytes": 0, 2024-04-01T01:50:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mapbl_items": 0, 2024-04-01T01:50:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pglog_bytes": 0, 2024-04-01T01:50:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pglog_items": 0, 2024-04-01T01:50:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_bytes": 9080, 2024-04-01T01:50:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_items": 174, 2024-04-01T01:50:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_mapping_bytes": 0, 2024-04-01T01:50:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_mapping_items": 0, 2024-04-01T01:50:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "pgmap_bytes": 0, 2024-04-01T01:50:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "pgmap_items": 0, 2024-04-01T01:50:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_co_bytes": 8464, 2024-04-01T01:50:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_co_items": 13, 2024-04-01T01:50:30.543 INFO:teuthology.orchestra.run.smithi081.stdout: "unittest_1_bytes": 0, 2024-04-01T01:50:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "unittest_1_items": 0, 2024-04-01T01:50:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "unittest_2_bytes": 0, 2024-04-01T01:50:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "unittest_2_items": 0 2024-04-01T01:50:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter": { 2024-04-01T01:50:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "op_active": 0, 2024-04-01T01:50:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "op_laggy": 0, 2024-04-01T01:50:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "op_send": 3323, 2024-04-01T01:50:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "op_send_bytes": 21, 2024-04-01T01:50:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "op_resend": 3, 2024-04-01T01:50:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "op_reply": 3323, 2024-04-01T01:50:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "op_latency": { 2024-04-01T01:50:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 3323, 2024-04-01T01:50:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 9.618814358, 2024-04-01T01:50:30.544 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.002894617 2024-04-01T01:50:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "op_inflight": 0, 2024-04-01T01:50:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "oplen_avg": { 2024-04-01T01:50:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 3323, 2024-04-01T01:50:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 3324 2024-04-01T01:50:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "op": 3323, 2024-04-01T01:50:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "op_r": 3323, 2024-04-01T01:50:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "op_w": 0, 2024-04-01T01:50:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "op_rmw": 0, 2024-04-01T01:50:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "op_pg": 0, 2024-04-01T01:50:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_stat": 2185, 2024-04-01T01:50:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_create": 0, 2024-04-01T01:50:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_read": 1136, 2024-04-01T01:50:30.545 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_write": 0, 2024-04-01T01:50:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_writefull": 0, 2024-04-01T01:50:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_writesame": 0, 2024-04-01T01:50:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_append": 0, 2024-04-01T01:50:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_zero": 0, 2024-04-01T01:50:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_truncate": 0, 2024-04-01T01:50:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_delete": 0, 2024-04-01T01:50:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_mapext": 0, 2024-04-01T01:50:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_sparse_read": 0, 2024-04-01T01:50:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_clonerange": 0, 2024-04-01T01:50:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_getxattr": 1, 2024-04-01T01:50:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_setxattr": 0, 2024-04-01T01:50:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_cmpxattr": 0, 2024-04-01T01:50:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_rmxattr": 0, 2024-04-01T01:50:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_resetxattrs": 0, 2024-04-01T01:50:30.546 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_call": 0, 2024-04-01T01:50:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_watch": 0, 2024-04-01T01:50:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_notify": 0, 2024-04-01T01:50:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_src_cmpxattr": 0, 2024-04-01T01:50:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_pgls": 0, 2024-04-01T01:50:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_pgls_filter": 0, 2024-04-01T01:50:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_other": 0, 2024-04-01T01:50:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_active": 0, 2024-04-01T01:50:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_send": 0, 2024-04-01T01:50:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_resend": 0, 2024-04-01T01:50:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_ping": 0, 2024-04-01T01:50:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "poolop_active": 0, 2024-04-01T01:50:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "poolop_send": 0, 2024-04-01T01:50:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "poolop_resend": 0, 2024-04-01T01:50:30.547 INFO:teuthology.orchestra.run.smithi081.stdout: "poolstat_active": 0, 2024-04-01T01:50:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "poolstat_send": 0, 2024-04-01T01:50:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "poolstat_resend": 0, 2024-04-01T01:50:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "statfs_active": 0, 2024-04-01T01:50:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "statfs_send": 0, 2024-04-01T01:50:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "statfs_resend": 0, 2024-04-01T01:50:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "command_active": 0, 2024-04-01T01:50:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "command_send": 0, 2024-04-01T01:50:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "command_resend": 0, 2024-04-01T01:50:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "map_epoch": 95, 2024-04-01T01:50:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "map_full": 0, 2024-04-01T01:50:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "map_inc": 6, 2024-04-01T01:50:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_sessions": 4, 2024-04-01T01:50:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_session_open": 5, 2024-04-01T01:50:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_session_close": 1, 2024-04-01T01:50:30.548 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_laggy": 0, 2024-04-01T01:50:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_wr": 0, 2024-04-01T01:50:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_rd": 2, 2024-04-01T01:50:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_del": 0 2024-04-01T01:50:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "oft": { 2024-04-01T01:50:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_total_objs": 0, 2024-04-01T01:50:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_total_kv_pairs": 0, 2024-04-01T01:50:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_total_updates": 0, 2024-04-01T01:50:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_total_removes": 0 2024-04-01T01:50:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "osd-slow-ops": { 2024-04-01T01:50:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "slow_ops_count": 0 2024-04-01T01:50:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "purge_queue": { 2024-04-01T01:50:30.549 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executing_ops": 0, 2024-04-01T01:50:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executing_ops_high_water": 0, 2024-04-01T01:50:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executing": 0, 2024-04-01T01:50:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executing_high_water": 0, 2024-04-01T01:50:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executed_ops": 0, 2024-04-01T01:50:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executed": 0, 2024-04-01T01:50:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_item_in_journal": 0 2024-04-01T01:50:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-01T01:50:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 104857600, 2024-04-01T01:50:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 3666, 2024-04-01T01:50:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 1453158, 2024-04-01T01:50:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:30.550 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 3666, 2024-04-01T01:50:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 0, 2024-04-01T01:50:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 0, 2024-04-01T01:50:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 3666, 2024-04-01T01:50:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 1453158, 2024-04-01T01:50:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-objecter_bytes": { 2024-04-01T01:50:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 104857600, 2024-04-01T01:50:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:30.551 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 0, 2024-04-01T01:50:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 0, 2024-04-01T01:50:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 3323, 2024-04-01T01:50:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 1602, 2024-04-01T01:50:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 3323, 2024-04-01T01:50:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 1602, 2024-04-01T01:50:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-objecter_ops": { 2024-04-01T01:50:30.552 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:30.553 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 1024, 2024-04-01T01:50:30.553 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:30.553 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 0, 2024-04-01T01:50:30.553 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 0, 2024-04-01T01:50:30.553 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:30.553 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:30.553 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 3323, 2024-04-01T01:50:30.553 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 3323, 2024-04-01T01:50:30.553 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 3323, 2024-04-01T01:50:30.553 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 3323, 2024-04-01T01:50:30.553 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:30.553 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.553 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-write_buf_throttle": { 2024-04-01T01:50:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 3758096384, 2024-04-01T01:50:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 0, 2024-04-01T01:50:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 0, 2024-04-01T01:50:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 0, 2024-04-01T01:50:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 0, 2024-04-01T01:50:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 0, 2024-04-01T01:50:30.554 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 0, 2024-04-01T01:50:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-write_buf_throttle-0x55e773ea26e0": { 2024-04-01T01:50:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 3758096384, 2024-04-01T01:50:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 0, 2024-04-01T01:50:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 0, 2024-04-01T01:50:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:30.555 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 0, 2024-04-01T01:50:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 0, 2024-04-01T01:50:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 0, 2024-04-01T01:50:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 0, 2024-04-01T01:50:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:30.556 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:30.556 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-01T01:50:31.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:30 smithi189 ceph-mon[35753]: pgmap v870: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-01T01:50:31.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:30 smithi070 ceph-mon[32179]: pgmap v870: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-01T01:50:31.323 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:30 smithi081 ceph-mon[35142]: pgmap v870: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 853 B/s wr, 9 op/s 2024-04-01T01:50:31.354 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-01T01:50:31.354 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-01T01:50:31.354 DEBUG:tasks.check_counter:Getting stats from k 2024-04-01T01:50:31.354 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph daemon mds.k perf dump 2024-04-01T01:50:31.607 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.b/config 2024-04-01T01:50:33.007 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:32 smithi081 ceph-mon[35142]: pgmap v871: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 853 B/s wr, 11 op/s 2024-04-01T01:50:33.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:32 smithi189 ceph-mon[35753]: pgmap v871: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 853 B/s wr, 11 op/s 2024-04-01T01:50:33.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:32 smithi070 ceph-mon[32179]: pgmap v871: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 853 B/s wr, 11 op/s 2024-04-01T01:50:33.615 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-01T01:50:33.615 INFO:teuthology.orchestra.run.smithi081.stdout: "AsyncMessenger::Worker-0": { 2024-04-01T01:50:33.615 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_messages": 339, 2024-04-01T01:50:33.615 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages": 329, 2024-04-01T01:50:33.615 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_bytes": 833090, 2024-04-01T01:50:33.615 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_bytes": 197972, 2024-04-01T01:50:33.616 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_created_connections": 6, 2024-04-01T01:50:33.616 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_active_connections": 1, 2024-04-01T01:50:33.616 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_total_time": 0.043734439, 2024-04-01T01:50:33.616 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_send_time": 0.014623264, 2024-04-01T01:50:33.616 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_recv_time": 0.024129434, 2024-04-01T01:50:33.616 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_fast_dispatch_time": 0.006883059, 2024-04-01T01:50:33.616 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:33.616 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 328, 2024-04-01T01:50:33.616 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.011762270, 2024-04-01T01:50:33.616 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000035860 2024-04-01T01:50:33.616 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.616 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:33.617 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.617 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.617 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.617 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_encrypted_bytes": 832256, 2024-04-01T01:50:33.617 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_encrypted_bytes": 197648 2024-04-01T01:50:33.617 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.617 INFO:teuthology.orchestra.run.smithi081.stdout: "AsyncMessenger::Worker-1": { 2024-04-01T01:50:33.617 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_messages": 1066, 2024-04-01T01:50:33.617 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages": 1307, 2024-04-01T01:50:33.617 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_bytes": 262086, 2024-04-01T01:50:33.617 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_bytes": 922069, 2024-04-01T01:50:33.617 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_created_connections": 6, 2024-04-01T01:50:33.617 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_active_connections": 4, 2024-04-01T01:50:33.618 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_total_time": 0.086788580, 2024-04-01T01:50:33.618 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_send_time": 0.028630357, 2024-04-01T01:50:33.618 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_recv_time": 0.018104178, 2024-04-01T01:50:33.618 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_fast_dispatch_time": 0.032540189, 2024-04-01T01:50:33.618 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:33.618 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 1305, 2024-04-01T01:50:33.618 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.049827600, 2024-04-01T01:50:33.618 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000038182 2024-04-01T01:50:33.618 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.618 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:33.618 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.618 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.618 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.619 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-01T01:50:33.619 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_encrypted_bytes": 583712 2024-04-01T01:50:33.619 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.619 INFO:teuthology.orchestra.run.smithi081.stdout: "AsyncMessenger::Worker-2": { 2024-04-01T01:50:33.619 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_messages": 2273, 2024-04-01T01:50:33.619 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages": 2273, 2024-04-01T01:50:33.619 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_bytes": 682854, 2024-04-01T01:50:33.619 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_bytes": 722048, 2024-04-01T01:50:33.619 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_created_connections": 4, 2024-04-01T01:50:33.619 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_active_connections": 3, 2024-04-01T01:50:33.619 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_total_time": 0.163944296, 2024-04-01T01:50:33.619 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_send_time": 0.054381724, 2024-04-01T01:50:33.619 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_recv_time": 0.042362211, 2024-04-01T01:50:33.620 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_running_fast_dispatch_time": 0.054634306, 2024-04-01T01:50:33.620 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:33.620 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 2272, 2024-04-01T01:50:33.620 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.084413079, 2024-04-01T01:50:33.620 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000037153 2024-04-01T01:50:33.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.620 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:33.620 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.620 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.620 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.620 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-01T01:50:33.620 INFO:teuthology.orchestra.run.smithi081.stdout: "msgr_send_encrypted_bytes": 448 2024-04-01T01:50:33.620 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.621 INFO:teuthology.orchestra.run.smithi081.stdout: "cct": { 2024-04-01T01:50:33.621 INFO:teuthology.orchestra.run.smithi081.stdout: "total_workers": 1, 2024-04-01T01:50:33.621 INFO:teuthology.orchestra.run.smithi081.stdout: "unhealthy_workers": 0 2024-04-01T01:50:33.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.621 INFO:teuthology.orchestra.run.smithi081.stdout: "finisher-MDSRank": { 2024-04-01T01:50:33.621 INFO:teuthology.orchestra.run.smithi081.stdout: "queue_len": 0, 2024-04-01T01:50:33.621 INFO:teuthology.orchestra.run.smithi081.stdout: "complete_latency": { 2024-04-01T01:50:33.621 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 5524, 2024-04-01T01:50:33.621 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.128283032, 2024-04-01T01:50:33.621 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000023222 2024-04-01T01:50:33.621 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:33.621 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.621 INFO:teuthology.orchestra.run.smithi081.stdout: "finisher-PurgeQueue": { 2024-04-01T01:50:33.621 INFO:teuthology.orchestra.run.smithi081.stdout: "queue_len": 0, 2024-04-01T01:50:33.621 INFO:teuthology.orchestra.run.smithi081.stdout: "complete_latency": { 2024-04-01T01:50:33.622 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.622 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.622 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.622 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:33.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.622 INFO:teuthology.orchestra.run.smithi081.stdout: "mds": { 2024-04-01T01:50:33.622 INFO:teuthology.orchestra.run.smithi081.stdout: "request": 0, 2024-04-01T01:50:33.622 INFO:teuthology.orchestra.run.smithi081.stdout: "reply": 0, 2024-04-01T01:50:33.622 INFO:teuthology.orchestra.run.smithi081.stdout: "reply_latency": { 2024-04-01T01:50:33.622 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.622 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.622 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.622 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.622 INFO:teuthology.orchestra.run.smithi081.stdout: "slow_reply": 0, 2024-04-01T01:50:33.623 INFO:teuthology.orchestra.run.smithi081.stdout: "forward": 0, 2024-04-01T01:50:33.623 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_fetch_complete": 0, 2024-04-01T01:50:33.623 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_fetch_keys": 0, 2024-04-01T01:50:33.623 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_commit": 0, 2024-04-01T01:50:33.623 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_split": 0, 2024-04-01T01:50:33.623 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_merge": 0, 2024-04-01T01:50:33.623 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes": 0, 2024-04-01T01:50:33.623 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_top": 0, 2024-04-01T01:50:33.623 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_bottom": 0, 2024-04-01T01:50:33.623 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_pin_tail": 0, 2024-04-01T01:50:33.623 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_pinned": 0, 2024-04-01T01:50:33.623 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_expired": 0, 2024-04-01T01:50:33.623 INFO:teuthology.orchestra.run.smithi081.stdout: "inodes_with_caps": 0, 2024-04-01T01:50:33.623 INFO:teuthology.orchestra.run.smithi081.stdout: "caps": 0, 2024-04-01T01:50:33.624 INFO:teuthology.orchestra.run.smithi081.stdout: "subtrees": 1, 2024-04-01T01:50:33.624 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse": 0, 2024-04-01T01:50:33.624 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_hit": 0, 2024-04-01T01:50:33.624 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_forward": 0, 2024-04-01T01:50:33.624 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_discover": 0, 2024-04-01T01:50:33.624 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_dir_fetch": 0, 2024-04-01T01:50:33.624 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_remote_ino": 0, 2024-04-01T01:50:33.624 INFO:teuthology.orchestra.run.smithi081.stdout: "traverse_lock": 0, 2024-04-01T01:50:33.624 INFO:teuthology.orchestra.run.smithi081.stdout: "load_cent": 0, 2024-04-01T01:50:33.624 INFO:teuthology.orchestra.run.smithi081.stdout: "q": 0, 2024-04-01T01:50:33.624 INFO:teuthology.orchestra.run.smithi081.stdout: "exported": 0, 2024-04-01T01:50:33.624 INFO:teuthology.orchestra.run.smithi081.stdout: "exported_inodes": 0, 2024-04-01T01:50:33.624 INFO:teuthology.orchestra.run.smithi081.stdout: "imported": 0, 2024-04-01T01:50:33.624 INFO:teuthology.orchestra.run.smithi081.stdout: "imported_inodes": 0, 2024-04-01T01:50:33.625 INFO:teuthology.orchestra.run.smithi081.stdout: "openino_dir_fetch": 0, 2024-04-01T01:50:33.625 INFO:teuthology.orchestra.run.smithi081.stdout: "openino_backtrace_fetch": 0, 2024-04-01T01:50:33.625 INFO:teuthology.orchestra.run.smithi081.stdout: "openino_peer_discover": 0, 2024-04-01T01:50:33.625 INFO:teuthology.orchestra.run.smithi081.stdout: "root_rfiles": 0, 2024-04-01T01:50:33.625 INFO:teuthology.orchestra.run.smithi081.stdout: "root_rbytes": 0, 2024-04-01T01:50:33.625 INFO:teuthology.orchestra.run.smithi081.stdout: "root_rsnaps": 0, 2024-04-01T01:50:33.625 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_backtrace_fetch": 0, 2024-04-01T01:50:33.625 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_set_tag": 0, 2024-04-01T01:50:33.625 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_backtrace_repaired": 0, 2024-04-01T01:50:33.625 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_inotable_repaired": 0, 2024-04-01T01:50:33.625 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_dir_inodes": 0, 2024-04-01T01:50:33.625 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_dir_base_inodes": 0, 2024-04-01T01:50:33.625 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_dirfrag_rstats": 0, 2024-04-01T01:50:33.625 INFO:teuthology.orchestra.run.smithi081.stdout: "scrub_file_inodes": 0, 2024-04-01T01:50:33.626 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_inode_file_caps": 0, 2024-04-01T01:50:33.626 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_revoke": 0, 2024-04-01T01:50:33.626 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_grant": 0, 2024-04-01T01:50:33.626 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_trunc": 0, 2024-04-01T01:50:33.626 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-01T01:50:33.626 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-01T01:50:33.626 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_caps": 0, 2024-04-01T01:50:33.626 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_caps_dirty": 0, 2024-04-01T01:50:33.626 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_cap_release": 0, 2024-04-01T01:50:33.626 INFO:teuthology.orchestra.run.smithi081.stdout: "process_request_cap_release": 0 2024-04-01T01:50:33.626 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.626 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_cache": { 2024-04-01T01:50:33.626 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_update": 0, 2024-04-01T01:50:33.626 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_update_receipt": 0, 2024-04-01T01:50:33.627 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_try_discover": 0, 2024-04-01T01:50:33.627 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_send_discover": 0, 2024-04-01T01:50:33.627 INFO:teuthology.orchestra.run.smithi081.stdout: "dir_handle_discover": 0, 2024-04-01T01:50:33.627 INFO:teuthology.orchestra.run.smithi081.stdout: "num_strays": 0, 2024-04-01T01:50:33.627 INFO:teuthology.orchestra.run.smithi081.stdout: "num_strays_delayed": 0, 2024-04-01T01:50:33.627 INFO:teuthology.orchestra.run.smithi081.stdout: "num_strays_enqueuing": 0, 2024-04-01T01:50:33.627 INFO:teuthology.orchestra.run.smithi081.stdout: "strays_created": 0, 2024-04-01T01:50:33.627 INFO:teuthology.orchestra.run.smithi081.stdout: "strays_enqueued": 0, 2024-04-01T01:50:33.627 INFO:teuthology.orchestra.run.smithi081.stdout: "strays_reintegrated": 0, 2024-04-01T01:50:33.627 INFO:teuthology.orchestra.run.smithi081.stdout: "strays_migrated": 0, 2024-04-01T01:50:33.627 INFO:teuthology.orchestra.run.smithi081.stdout: "num_recovering_processing": 0, 2024-04-01T01:50:33.627 INFO:teuthology.orchestra.run.smithi081.stdout: "num_recovering_enqueued": 0, 2024-04-01T01:50:33.627 INFO:teuthology.orchestra.run.smithi081.stdout: "num_recovering_prioritized": 0, 2024-04-01T01:50:33.627 INFO:teuthology.orchestra.run.smithi081.stdout: "recovery_started": 0, 2024-04-01T01:50:33.628 INFO:teuthology.orchestra.run.smithi081.stdout: "recovery_completed": 0, 2024-04-01T01:50:33.628 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_quiesce_path": 0, 2024-04-01T01:50:33.628 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_quiesce_inode": 0, 2024-04-01T01:50:33.628 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_enqueue_scrub": 0, 2024-04-01T01:50:33.628 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_exportdir": 0, 2024-04-01T01:50:33.628 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_flush": 0, 2024-04-01T01:50:33.628 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_fragmentdir": 0, 2024-04-01T01:50:33.628 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_fragstats": 0, 2024-04-01T01:50:33.628 INFO:teuthology.orchestra.run.smithi081.stdout: "ireq_inodestats": 0 2024-04-01T01:50:33.628 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.628 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_log": { 2024-04-01T01:50:33.628 INFO:teuthology.orchestra.run.smithi081.stdout: "evlrg": 0, 2024-04-01T01:50:33.628 INFO:teuthology.orchestra.run.smithi081.stdout: "evadd": 0, 2024-04-01T01:50:33.628 INFO:teuthology.orchestra.run.smithi081.stdout: "evex": 0, 2024-04-01T01:50:33.629 INFO:teuthology.orchestra.run.smithi081.stdout: "evtrm": 0, 2024-04-01T01:50:33.629 INFO:teuthology.orchestra.run.smithi081.stdout: "ev": 5, 2024-04-01T01:50:33.629 INFO:teuthology.orchestra.run.smithi081.stdout: "evexg": 0, 2024-04-01T01:50:33.629 INFO:teuthology.orchestra.run.smithi081.stdout: "evexd": 0, 2024-04-01T01:50:33.629 INFO:teuthology.orchestra.run.smithi081.stdout: "segadd": 0, 2024-04-01T01:50:33.629 INFO:teuthology.orchestra.run.smithi081.stdout: "segex": 0, 2024-04-01T01:50:33.629 INFO:teuthology.orchestra.run.smithi081.stdout: "segtrm": 0, 2024-04-01T01:50:33.629 INFO:teuthology.orchestra.run.smithi081.stdout: "seg": 2, 2024-04-01T01:50:33.629 INFO:teuthology.orchestra.run.smithi081.stdout: "segmjr": 2, 2024-04-01T01:50:33.629 INFO:teuthology.orchestra.run.smithi081.stdout: "segexg": 0, 2024-04-01T01:50:33.629 INFO:teuthology.orchestra.run.smithi081.stdout: "segexd": 0, 2024-04-01T01:50:33.629 INFO:teuthology.orchestra.run.smithi081.stdout: "expos": 4194304, 2024-04-01T01:50:33.629 INFO:teuthology.orchestra.run.smithi081.stdout: "wrpos": 4195901, 2024-04-01T01:50:33.629 INFO:teuthology.orchestra.run.smithi081.stdout: "rdpos": 4195424, 2024-04-01T01:50:33.629 INFO:teuthology.orchestra.run.smithi081.stdout: "jlat": { 2024-04-01T01:50:33.630 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.630 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.630 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.630 INFO:teuthology.orchestra.run.smithi081.stdout: "replayed": 5 2024-04-01T01:50:33.630 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.630 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_mem": { 2024-04-01T01:50:33.630 INFO:teuthology.orchestra.run.smithi081.stdout: "ino": 3, 2024-04-01T01:50:33.630 INFO:teuthology.orchestra.run.smithi081.stdout: "ino+": 3, 2024-04-01T01:50:33.630 INFO:teuthology.orchestra.run.smithi081.stdout: "ino-": 0, 2024-04-01T01:50:33.630 INFO:teuthology.orchestra.run.smithi081.stdout: "dir": 1, 2024-04-01T01:50:33.630 INFO:teuthology.orchestra.run.smithi081.stdout: "dir+": 1, 2024-04-01T01:50:33.630 INFO:teuthology.orchestra.run.smithi081.stdout: "dir-": 0, 2024-04-01T01:50:33.631 INFO:teuthology.orchestra.run.smithi081.stdout: "dn": 0, 2024-04-01T01:50:33.631 INFO:teuthology.orchestra.run.smithi081.stdout: "dn+": 0, 2024-04-01T01:50:33.631 INFO:teuthology.orchestra.run.smithi081.stdout: "dn-": 0, 2024-04-01T01:50:33.631 INFO:teuthology.orchestra.run.smithi081.stdout: "cap": 0, 2024-04-01T01:50:33.631 INFO:teuthology.orchestra.run.smithi081.stdout: "cap+": 0, 2024-04-01T01:50:33.631 INFO:teuthology.orchestra.run.smithi081.stdout: "cap-": 0, 2024-04-01T01:50:33.631 INFO:teuthology.orchestra.run.smithi081.stdout: "rss": 50360, 2024-04-01T01:50:33.631 INFO:teuthology.orchestra.run.smithi081.stdout: "heap": 215420 2024-04-01T01:50:33.631 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.631 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_server": { 2024-04-01T01:50:33.631 INFO:teuthology.orchestra.run.smithi081.stdout: "dispatch_client_request": 0, 2024-04-01T01:50:33.631 INFO:teuthology.orchestra.run.smithi081.stdout: "dispatch_server_request": 0, 2024-04-01T01:50:33.631 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_request": 0, 2024-04-01T01:50:33.631 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_client_session": 0, 2024-04-01T01:50:33.632 INFO:teuthology.orchestra.run.smithi081.stdout: "handle_peer_request": 0, 2024-04-01T01:50:33.632 INFO:teuthology.orchestra.run.smithi081.stdout: "req_create_latency": { 2024-04-01T01:50:33.632 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.632 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.632 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.632 INFO:teuthology.orchestra.run.smithi081.stdout: "req_getattr_latency": { 2024-04-01T01:50:33.632 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.632 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.632 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.632 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.632 INFO:teuthology.orchestra.run.smithi081.stdout: "req_getfilelock_latency": { 2024-04-01T01:50:33.632 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.632 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.633 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.633 INFO:teuthology.orchestra.run.smithi081.stdout: "req_link_latency": { 2024-04-01T01:50:33.633 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.633 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.633 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.633 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookup_latency": { 2024-04-01T01:50:33.633 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.633 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.633 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.633 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.633 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookuphash_latency": { 2024-04-01T01:50:33.633 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.633 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.633 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.634 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.634 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookupino_latency": { 2024-04-01T01:50:33.634 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.634 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.634 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.634 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.634 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookupname_latency": { 2024-04-01T01:50:33.634 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.634 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.634 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.634 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.634 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookupparent_latency": { 2024-04-01T01:50:33.634 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.634 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.635 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.635 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lookupsnap_latency": { 2024-04-01T01:50:33.635 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.635 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.635 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.635 INFO:teuthology.orchestra.run.smithi081.stdout: "req_lssnap_latency": { 2024-04-01T01:50:33.635 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.635 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.635 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.635 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.635 INFO:teuthology.orchestra.run.smithi081.stdout: "req_mkdir_latency": { 2024-04-01T01:50:33.635 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.636 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.636 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.636 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.636 INFO:teuthology.orchestra.run.smithi081.stdout: "req_mknod_latency": { 2024-04-01T01:50:33.636 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.637 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.637 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.637 INFO:teuthology.orchestra.run.smithi081.stdout: "req_mksnap_latency": { 2024-04-01T01:50:33.637 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.637 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.637 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.637 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.637 INFO:teuthology.orchestra.run.smithi081.stdout: "req_open_latency": { 2024-04-01T01:50:33.637 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.637 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.637 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.638 INFO:teuthology.orchestra.run.smithi081.stdout: "req_readdir_latency": { 2024-04-01T01:50:33.638 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.638 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.638 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.638 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.638 INFO:teuthology.orchestra.run.smithi081.stdout: "req_rename_latency": { 2024-04-01T01:50:33.639 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.639 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.639 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.639 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.639 INFO:teuthology.orchestra.run.smithi081.stdout: "req_renamesnap_latency": { 2024-04-01T01:50:33.639 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.639 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.639 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.640 INFO:teuthology.orchestra.run.smithi081.stdout: "req_snapdiff_latency": { 2024-04-01T01:50:33.640 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.640 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.640 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.640 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.641 INFO:teuthology.orchestra.run.smithi081.stdout: "req_rmdir_latency": { 2024-04-01T01:50:33.641 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.641 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.641 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.641 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.641 INFO:teuthology.orchestra.run.smithi081.stdout: "req_rmsnap_latency": { 2024-04-01T01:50:33.642 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.642 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.642 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.642 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.642 INFO:teuthology.orchestra.run.smithi081.stdout: "req_rmxattr_latency": { 2024-04-01T01:50:33.642 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.643 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.643 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.643 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setattr_latency": { 2024-04-01T01:50:33.643 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.643 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.643 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.643 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.644 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setdirlayout_latency": { 2024-04-01T01:50:33.644 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.644 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.644 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.644 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.644 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setfilelock_latency": { 2024-04-01T01:50:33.644 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.645 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.645 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.645 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.645 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setlayout_latency": { 2024-04-01T01:50:33.645 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.645 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.646 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.646 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.646 INFO:teuthology.orchestra.run.smithi081.stdout: "req_setxattr_latency": { 2024-04-01T01:50:33.646 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.646 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.646 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.647 INFO:teuthology.orchestra.run.smithi081.stdout: "req_symlink_latency": { 2024-04-01T01:50:33.647 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.647 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.647 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.647 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.647 INFO:teuthology.orchestra.run.smithi081.stdout: "req_unlink_latency": { 2024-04-01T01:50:33.647 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.647 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.648 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.648 INFO:teuthology.orchestra.run.smithi081.stdout: "cap_revoke_eviction": 0, 2024-04-01T01:50:33.648 INFO:teuthology.orchestra.run.smithi081.stdout: "cap_acquisition_throttle": 0, 2024-04-01T01:50:33.648 INFO:teuthology.orchestra.run.smithi081.stdout: "req_getvxattr_latency": { 2024-04-01T01:50:33.648 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.648 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.648 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.648 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:33.648 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.648 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_sessions": { 2024-04-01T01:50:33.648 INFO:teuthology.orchestra.run.smithi081.stdout: "session_count": 1, 2024-04-01T01:50:33.648 INFO:teuthology.orchestra.run.smithi081.stdout: "session_add": 2, 2024-04-01T01:50:33.648 INFO:teuthology.orchestra.run.smithi081.stdout: "session_remove": 1, 2024-04-01T01:50:33.649 INFO:teuthology.orchestra.run.smithi081.stdout: "sessions_open": 1, 2024-04-01T01:50:33.649 INFO:teuthology.orchestra.run.smithi081.stdout: "sessions_stale": 0, 2024-04-01T01:50:33.649 INFO:teuthology.orchestra.run.smithi081.stdout: "total_load": 0, 2024-04-01T01:50:33.649 INFO:teuthology.orchestra.run.smithi081.stdout: "average_load": 0, 2024-04-01T01:50:33.649 INFO:teuthology.orchestra.run.smithi081.stdout: "avg_session_uptime": 2175, 2024-04-01T01:50:33.649 INFO:teuthology.orchestra.run.smithi081.stdout: "mdthresh_evicted": 0 2024-04-01T01:50:33.649 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.649 INFO:teuthology.orchestra.run.smithi081.stdout: "mempool": { 2024-04-01T01:50:33.649 INFO:teuthology.orchestra.run.smithi081.stdout: "bloom_filter_bytes": 0, 2024-04-01T01:50:33.649 INFO:teuthology.orchestra.run.smithi081.stdout: "bloom_filter_items": 0, 2024-04-01T01:50:33.649 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_alloc_bytes": 0, 2024-04-01T01:50:33.649 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_alloc_items": 0, 2024-04-01T01:50:33.649 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_data_bytes": 0, 2024-04-01T01:50:33.649 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_data_items": 0, 2024-04-01T01:50:33.650 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-01T01:50:33.650 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_onode_items": 0, 2024-04-01T01:50:33.650 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-01T01:50:33.650 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_meta_items": 0, 2024-04-01T01:50:33.650 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_other_bytes": 0, 2024-04-01T01:50:33.650 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_other_items": 0, 2024-04-01T01:50:33.650 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-01T01:50:33.650 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_cache_buffer_items": 0, 2024-04-01T01:50:33.650 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_bytes": 0, 2024-04-01T01:50:33.650 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_extent_items": 0, 2024-04-01T01:50:33.650 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_blob_bytes": 0, 2024-04-01T01:50:33.651 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_blob_items": 0, 2024-04-01T01:50:33.651 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-01T01:50:33.651 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_shared_blob_items": 0, 2024-04-01T01:50:33.651 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-01T01:50:33.651 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_inline_bl_items": 0, 2024-04-01T01:50:33.651 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_bytes": 0, 2024-04-01T01:50:33.651 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_fsck_items": 0, 2024-04-01T01:50:33.651 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_txc_bytes": 0, 2024-04-01T01:50:33.651 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_txc_items": 0, 2024-04-01T01:50:33.651 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-01T01:50:33.651 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_writing_deferred_items": 0, 2024-04-01T01:50:33.651 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_writing_bytes": 0, 2024-04-01T01:50:33.651 INFO:teuthology.orchestra.run.smithi081.stdout: "bluestore_writing_items": 0, 2024-04-01T01:50:33.651 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_bytes": 0, 2024-04-01T01:50:33.652 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_items": 0, 2024-04-01T01:50:33.652 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_file_reader_bytes": 0, 2024-04-01T01:50:33.652 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_file_reader_items": 0, 2024-04-01T01:50:33.652 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_file_writer_bytes": 0, 2024-04-01T01:50:33.652 INFO:teuthology.orchestra.run.smithi081.stdout: "bluefs_file_writer_items": 0, 2024-04-01T01:50:33.652 INFO:teuthology.orchestra.run.smithi081.stdout: "buffer_anon_bytes": 190167, 2024-04-01T01:50:33.652 INFO:teuthology.orchestra.run.smithi081.stdout: "buffer_anon_items": 52, 2024-04-01T01:50:33.652 INFO:teuthology.orchestra.run.smithi081.stdout: "buffer_meta_bytes": 0, 2024-04-01T01:50:33.652 INFO:teuthology.orchestra.run.smithi081.stdout: "buffer_meta_items": 0, 2024-04-01T01:50:33.652 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_bytes": 0, 2024-04-01T01:50:33.652 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_items": 0, 2024-04-01T01:50:33.652 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mapbl_bytes": 0, 2024-04-01T01:50:33.652 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_mapbl_items": 0, 2024-04-01T01:50:33.652 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pglog_bytes": 0, 2024-04-01T01:50:33.653 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_pglog_items": 0, 2024-04-01T01:50:33.653 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_bytes": 9080, 2024-04-01T01:50:33.653 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_items": 174, 2024-04-01T01:50:33.653 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_mapping_bytes": 0, 2024-04-01T01:50:33.653 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_mapping_items": 0, 2024-04-01T01:50:33.653 INFO:teuthology.orchestra.run.smithi081.stdout: "pgmap_bytes": 0, 2024-04-01T01:50:33.653 INFO:teuthology.orchestra.run.smithi081.stdout: "pgmap_items": 0, 2024-04-01T01:50:33.653 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_co_bytes": 8464, 2024-04-01T01:50:33.653 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_co_items": 13, 2024-04-01T01:50:33.653 INFO:teuthology.orchestra.run.smithi081.stdout: "unittest_1_bytes": 0, 2024-04-01T01:50:33.653 INFO:teuthology.orchestra.run.smithi081.stdout: "unittest_1_items": 0, 2024-04-01T01:50:33.653 INFO:teuthology.orchestra.run.smithi081.stdout: "unittest_2_bytes": 0, 2024-04-01T01:50:33.653 INFO:teuthology.orchestra.run.smithi081.stdout: "unittest_2_items": 0 2024-04-01T01:50:33.653 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.653 INFO:teuthology.orchestra.run.smithi081.stdout: "objecter": { 2024-04-01T01:50:33.654 INFO:teuthology.orchestra.run.smithi081.stdout: "op_active": 0, 2024-04-01T01:50:33.654 INFO:teuthology.orchestra.run.smithi081.stdout: "op_laggy": 0, 2024-04-01T01:50:33.654 INFO:teuthology.orchestra.run.smithi081.stdout: "op_send": 3334, 2024-04-01T01:50:33.654 INFO:teuthology.orchestra.run.smithi081.stdout: "op_send_bytes": 21, 2024-04-01T01:50:33.654 INFO:teuthology.orchestra.run.smithi081.stdout: "op_resend": 3, 2024-04-01T01:50:33.654 INFO:teuthology.orchestra.run.smithi081.stdout: "op_reply": 3334, 2024-04-01T01:50:33.654 INFO:teuthology.orchestra.run.smithi081.stdout: "op_latency": { 2024-04-01T01:50:33.654 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 3334, 2024-04-01T01:50:33.654 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 3.621929707, 2024-04-01T01:50:33.654 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.001086361 2024-04-01T01:50:33.654 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.654 INFO:teuthology.orchestra.run.smithi081.stdout: "op_inflight": 0, 2024-04-01T01:50:33.654 INFO:teuthology.orchestra.run.smithi081.stdout: "oplen_avg": { 2024-04-01T01:50:33.655 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 3334, 2024-04-01T01:50:33.655 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 3335 2024-04-01T01:50:33.655 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.655 INFO:teuthology.orchestra.run.smithi081.stdout: "op": 3334, 2024-04-01T01:50:33.655 INFO:teuthology.orchestra.run.smithi081.stdout: "op_r": 3334, 2024-04-01T01:50:33.655 INFO:teuthology.orchestra.run.smithi081.stdout: "op_w": 0, 2024-04-01T01:50:33.655 INFO:teuthology.orchestra.run.smithi081.stdout: "op_rmw": 0, 2024-04-01T01:50:33.655 INFO:teuthology.orchestra.run.smithi081.stdout: "op_pg": 0, 2024-04-01T01:50:33.655 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_stat": 2194, 2024-04-01T01:50:33.655 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_create": 0, 2024-04-01T01:50:33.655 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_read": 1138, 2024-04-01T01:50:33.655 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_write": 0, 2024-04-01T01:50:33.655 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_writefull": 0, 2024-04-01T01:50:33.656 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_writesame": 0, 2024-04-01T01:50:33.656 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_append": 0, 2024-04-01T01:50:33.656 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_zero": 0, 2024-04-01T01:50:33.656 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_truncate": 0, 2024-04-01T01:50:33.656 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_delete": 0, 2024-04-01T01:50:33.656 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_mapext": 0, 2024-04-01T01:50:33.656 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_sparse_read": 0, 2024-04-01T01:50:33.656 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_clonerange": 0, 2024-04-01T01:50:33.656 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_getxattr": 1, 2024-04-01T01:50:33.656 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_setxattr": 0, 2024-04-01T01:50:33.656 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_cmpxattr": 0, 2024-04-01T01:50:33.656 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_rmxattr": 0, 2024-04-01T01:50:33.656 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_resetxattrs": 0, 2024-04-01T01:50:33.657 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_call": 0, 2024-04-01T01:50:33.657 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_watch": 0, 2024-04-01T01:50:33.657 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_notify": 0, 2024-04-01T01:50:33.657 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_src_cmpxattr": 0, 2024-04-01T01:50:33.657 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_pgls": 0, 2024-04-01T01:50:33.657 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_pgls_filter": 0, 2024-04-01T01:50:33.657 INFO:teuthology.orchestra.run.smithi081.stdout: "osdop_other": 0, 2024-04-01T01:50:33.657 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_active": 0, 2024-04-01T01:50:33.657 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_send": 0, 2024-04-01T01:50:33.658 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_resend": 0, 2024-04-01T01:50:33.658 INFO:teuthology.orchestra.run.smithi081.stdout: "linger_ping": 0, 2024-04-01T01:50:33.658 INFO:teuthology.orchestra.run.smithi081.stdout: "poolop_active": 0, 2024-04-01T01:50:33.658 INFO:teuthology.orchestra.run.smithi081.stdout: "poolop_send": 0, 2024-04-01T01:50:33.658 INFO:teuthology.orchestra.run.smithi081.stdout: "poolop_resend": 0, 2024-04-01T01:50:33.658 INFO:teuthology.orchestra.run.smithi081.stdout: "poolstat_active": 0, 2024-04-01T01:50:33.658 INFO:teuthology.orchestra.run.smithi081.stdout: "poolstat_send": 0, 2024-04-01T01:50:33.658 INFO:teuthology.orchestra.run.smithi081.stdout: "poolstat_resend": 0, 2024-04-01T01:50:33.659 INFO:teuthology.orchestra.run.smithi081.stdout: "statfs_active": 0, 2024-04-01T01:50:33.659 INFO:teuthology.orchestra.run.smithi081.stdout: "statfs_send": 0, 2024-04-01T01:50:33.659 INFO:teuthology.orchestra.run.smithi081.stdout: "statfs_resend": 0, 2024-04-01T01:50:33.659 INFO:teuthology.orchestra.run.smithi081.stdout: "command_active": 0, 2024-04-01T01:50:33.659 INFO:teuthology.orchestra.run.smithi081.stdout: "command_send": 0, 2024-04-01T01:50:33.659 INFO:teuthology.orchestra.run.smithi081.stdout: "command_resend": 0, 2024-04-01T01:50:33.659 INFO:teuthology.orchestra.run.smithi081.stdout: "map_epoch": 95, 2024-04-01T01:50:33.659 INFO:teuthology.orchestra.run.smithi081.stdout: "map_full": 0, 2024-04-01T01:50:33.659 INFO:teuthology.orchestra.run.smithi081.stdout: "map_inc": 6, 2024-04-01T01:50:33.660 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_sessions": 5, 2024-04-01T01:50:33.660 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_session_open": 6, 2024-04-01T01:50:33.660 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_session_close": 1, 2024-04-01T01:50:33.660 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_laggy": 0, 2024-04-01T01:50:33.660 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_wr": 0, 2024-04-01T01:50:33.660 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_rd": 2, 2024-04-01T01:50:33.660 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_del": 0 2024-04-01T01:50:33.660 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.660 INFO:teuthology.orchestra.run.smithi081.stdout: "oft": { 2024-04-01T01:50:33.661 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_total_objs": 0, 2024-04-01T01:50:33.661 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_total_kv_pairs": 0, 2024-04-01T01:50:33.661 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_total_updates": 0, 2024-04-01T01:50:33.661 INFO:teuthology.orchestra.run.smithi081.stdout: "omap_total_removes": 0 2024-04-01T01:50:33.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.661 INFO:teuthology.orchestra.run.smithi081.stdout: "osd-slow-ops": { 2024-04-01T01:50:33.661 INFO:teuthology.orchestra.run.smithi081.stdout: "slow_ops_count": 0 2024-04-01T01:50:33.661 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.662 INFO:teuthology.orchestra.run.smithi081.stdout: "purge_queue": { 2024-04-01T01:50:33.662 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executing_ops": 0, 2024-04-01T01:50:33.662 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executing_ops_high_water": 0, 2024-04-01T01:50:33.662 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executing": 0, 2024-04-01T01:50:33.662 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executing_high_water": 0, 2024-04-01T01:50:33.662 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executed_ops": 0, 2024-04-01T01:50:33.662 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_executed": 0, 2024-04-01T01:50:33.662 INFO:teuthology.orchestra.run.smithi081.stdout: "pq_item_in_journal": 0 2024-04-01T01:50:33.663 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.663 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-01T01:50:33.663 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:33.663 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 104857600, 2024-04-01T01:50:33.663 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:33.663 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 3674, 2024-04-01T01:50:33.663 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 1427005, 2024-04-01T01:50:33.663 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:33.663 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 3674, 2024-04-01T01:50:33.664 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 0, 2024-04-01T01:50:33.664 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 0, 2024-04-01T01:50:33.664 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 3674, 2024-04-01T01:50:33.664 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 1427005, 2024-04-01T01:50:33.664 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:33.664 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.664 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.664 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.665 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:33.665 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.665 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-objecter_bytes": { 2024-04-01T01:50:33.665 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:33.665 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 104857600, 2024-04-01T01:50:33.665 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:33.665 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 0, 2024-04-01T01:50:33.665 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 0, 2024-04-01T01:50:33.665 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:33.666 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:33.666 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 3334, 2024-04-01T01:50:33.666 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 1602, 2024-04-01T01:50:33.666 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 3334, 2024-04-01T01:50:33.666 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 1602, 2024-04-01T01:50:33.666 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:33.666 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.666 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.667 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.667 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:33.667 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.667 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-objecter_ops": { 2024-04-01T01:50:33.667 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:33.667 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 1024, 2024-04-01T01:50:33.667 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:33.667 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 0, 2024-04-01T01:50:33.667 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 0, 2024-04-01T01:50:33.668 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:33.668 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:33.668 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 3334, 2024-04-01T01:50:33.668 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 3334, 2024-04-01T01:50:33.668 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 3334, 2024-04-01T01:50:33.668 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 3334, 2024-04-01T01:50:33.668 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:33.668 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.669 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.669 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.669 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:33.669 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.669 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-write_buf_throttle": { 2024-04-01T01:50:33.669 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:33.669 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 3758096384, 2024-04-01T01:50:33.669 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:33.669 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 0, 2024-04-01T01:50:33.670 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 0, 2024-04-01T01:50:33.670 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:33.670 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:33.670 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 0, 2024-04-01T01:50:33.670 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 0, 2024-04-01T01:50:33.670 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 0, 2024-04-01T01:50:33.670 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 0, 2024-04-01T01:50:33.670 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:33.670 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.671 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.671 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.671 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:33.671 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-01T01:50:33.671 INFO:teuthology.orchestra.run.smithi081.stdout: "throttle-write_buf_throttle-0x55c42e66e6e0": { 2024-04-01T01:50:33.671 INFO:teuthology.orchestra.run.smithi081.stdout: "val": 0, 2024-04-01T01:50:33.671 INFO:teuthology.orchestra.run.smithi081.stdout: "max": 3758096384, 2024-04-01T01:50:33.671 INFO:teuthology.orchestra.run.smithi081.stdout: "get_started": 0, 2024-04-01T01:50:33.671 INFO:teuthology.orchestra.run.smithi081.stdout: "get": 0, 2024-04-01T01:50:33.672 INFO:teuthology.orchestra.run.smithi081.stdout: "get_sum": 0, 2024-04-01T01:50:33.672 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:33.672 INFO:teuthology.orchestra.run.smithi081.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:33.672 INFO:teuthology.orchestra.run.smithi081.stdout: "take": 0, 2024-04-01T01:50:33.672 INFO:teuthology.orchestra.run.smithi081.stdout: "take_sum": 0, 2024-04-01T01:50:33.672 INFO:teuthology.orchestra.run.smithi081.stdout: "put": 0, 2024-04-01T01:50:33.672 INFO:teuthology.orchestra.run.smithi081.stdout: "put_sum": 0, 2024-04-01T01:50:33.672 INFO:teuthology.orchestra.run.smithi081.stdout: "wait": { 2024-04-01T01:50:33.672 INFO:teuthology.orchestra.run.smithi081.stdout: "avgcount": 0, 2024-04-01T01:50:33.672 INFO:teuthology.orchestra.run.smithi081.stdout: "sum": 0.000000000, 2024-04-01T01:50:33.672 INFO:teuthology.orchestra.run.smithi081.stdout: "avgtime": 0.000000000 2024-04-01T01:50:33.672 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:33.672 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-01T01:50:33.672 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-01T01:50:34.400 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-04-01T01:50:34.400 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-04-01T01:50:34.400 DEBUG:tasks.check_counter:Getting stats from c 2024-04-01T01:50:34.400 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph daemon mds.c perf dump 2024-04-01T01:50:34.675 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-01T01:50:35.112 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:34 smithi189 ceph-mon[35753]: pgmap v872: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-01T01:50:35.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:34 smithi070 ceph-mon[32179]: pgmap v872: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-01T01:50:35.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:34 smithi081 ceph-mon[35142]: pgmap v872: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-01T01:50:36.636 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-01T01:50:36.636 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-04-01T01:50:36.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 89, 2024-04-01T01:50:36.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 89, 2024-04-01T01:50:36.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 31903, 2024-04-01T01:50:36.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 27488, 2024-04-01T01:50:36.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 6, 2024-04-01T01:50:36.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 2, 2024-04-01T01:50:36.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.009504131, 2024-04-01T01:50:36.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.002220687, 2024-04-01T01:50:36.636 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.003060549, 2024-04-01T01:50:36.637 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.002525352, 2024-04-01T01:50:36.637 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:36.637 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 88, 2024-04-01T01:50:36.637 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.005082675, 2024-04-01T01:50:36.637 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000057757 2024-04-01T01:50:36.637 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.637 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:36.637 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.637 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.637 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.637 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.637 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-01T01:50:36.637 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 448 2024-04-01T01:50:36.637 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.637 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-04-01T01:50:36.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 199, 2024-04-01T01:50:36.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 218, 2024-04-01T01:50:36.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 3743918, 2024-04-01T01:50:36.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 180893, 2024-04-01T01:50:36.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 5, 2024-04-01T01:50:36.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 4, 2024-04-01T01:50:36.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.016498948, 2024-04-01T01:50:36.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.003368757, 2024-04-01T01:50:36.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.007706188, 2024-04-01T01:50:36.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.004538758, 2024-04-01T01:50:36.638 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:36.638 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 216, 2024-04-01T01:50:36.638 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.010607089, 2024-04-01T01:50:36.638 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000049106 2024-04-01T01:50:36.639 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.639 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:36.639 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.639 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.639 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.639 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.639 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-01T01:50:36.639 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 117872 2024-04-01T01:50:36.639 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.639 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-04-01T01:50:36.640 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 38, 2024-04-01T01:50:36.640 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 37, 2024-04-01T01:50:36.640 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 165875, 2024-04-01T01:50:36.640 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 17652, 2024-04-01T01:50:36.640 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 4, 2024-04-01T01:50:36.640 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 3, 2024-04-01T01:50:36.640 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.005773957, 2024-04-01T01:50:36.640 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.001337952, 2024-04-01T01:50:36.640 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.005057687, 2024-04-01T01:50:36.640 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.000568673, 2024-04-01T01:50:36.640 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:36.640 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 36, 2024-04-01T01:50:36.640 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.006172825, 2024-04-01T01:50:36.641 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000171467 2024-04-01T01:50:36.641 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.641 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:36.641 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.641 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.641 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.641 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.641 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 164480, 2024-04-01T01:50:36.641 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 16688 2024-04-01T01:50:36.641 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.641 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-04-01T01:50:36.641 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-04-01T01:50:36.642 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-04-01T01:50:36.642 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.642 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-04-01T01:50:36.642 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-04-01T01:50:36.642 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-04-01T01:50:36.642 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 466, 2024-04-01T01:50:36.642 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.012573949, 2024-04-01T01:50:36.642 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000026982 2024-04-01T01:50:36.642 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:36.642 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.642 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-04-01T01:50:36.642 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-04-01T01:50:36.643 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-04-01T01:50:36.643 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.643 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.643 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.643 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:36.643 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.643 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-04-01T01:50:36.643 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 0, 2024-04-01T01:50:36.643 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 0, 2024-04-01T01:50:36.643 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-04-01T01:50:36.643 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.644 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.644 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.644 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.644 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-04-01T01:50:36.644 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 0, 2024-04-01T01:50:36.644 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-04-01T01:50:36.644 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-04-01T01:50:36.644 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 0, 2024-04-01T01:50:36.644 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-04-01T01:50:36.644 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-04-01T01:50:36.644 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 220, 2024-04-01T01:50:36.644 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 93, 2024-04-01T01:50:36.644 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 127, 2024-04-01T01:50:36.644 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 0, 2024-04-01T01:50:36.645 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 91, 2024-04-01T01:50:36.645 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 0, 2024-04-01T01:50:36.645 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 0, 2024-04-01T01:50:36.645 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 0, 2024-04-01T01:50:36.645 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 2, 2024-04-01T01:50:36.645 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 0, 2024-04-01T01:50:36.645 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 0, 2024-04-01T01:50:36.645 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 0, 2024-04-01T01:50:36.645 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 0, 2024-04-01T01:50:36.645 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-04-01T01:50:36.645 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-04-01T01:50:36.645 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 0, 2024-04-01T01:50:36.645 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-04-01T01:50:36.645 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-04-01T01:50:36.646 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 0, 2024-04-01T01:50:36.646 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 0, 2024-04-01T01:50:36.646 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 0, 2024-04-01T01:50:36.646 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 0, 2024-04-01T01:50:36.646 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-04-01T01:50:36.646 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 0, 2024-04-01T01:50:36.646 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-04-01T01:50:36.646 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 1017, 2024-04-01T01:50:36.646 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 1524077909, 2024-04-01T01:50:36.646 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 0, 2024-04-01T01:50:36.646 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 0, 2024-04-01T01:50:36.647 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-04-01T01:50:36.647 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-04-01T01:50:36.647 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-04-01T01:50:36.647 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-04-01T01:50:36.647 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-04-01T01:50:36.647 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-04-01T01:50:36.647 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 0, 2024-04-01T01:50:36.647 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-04-01T01:50:36.647 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 0, 2024-04-01T01:50:36.647 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 0, 2024-04-01T01:50:36.647 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-04-01T01:50:36.647 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-01T01:50:36.647 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-01T01:50:36.647 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 0, 2024-04-01T01:50:36.647 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 0, 2024-04-01T01:50:36.648 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 0, 2024-04-01T01:50:36.648 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 0 2024-04-01T01:50:36.648 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.648 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-04-01T01:50:36.648 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 0, 2024-04-01T01:50:36.648 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 0, 2024-04-01T01:50:36.648 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 0, 2024-04-01T01:50:36.648 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 0, 2024-04-01T01:50:36.648 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 0, 2024-04-01T01:50:36.648 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-04-01T01:50:36.648 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-04-01T01:50:36.648 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-04-01T01:50:36.648 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 0, 2024-04-01T01:50:36.649 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 0, 2024-04-01T01:50:36.649 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-04-01T01:50:36.649 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-04-01T01:50:36.649 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-04-01T01:50:36.649 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-04-01T01:50:36.649 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-04-01T01:50:36.649 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-04-01T01:50:36.649 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-04-01T01:50:36.649 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_path": 0, 2024-04-01T01:50:36.649 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_inode": 0, 2024-04-01T01:50:36.649 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-04-01T01:50:36.649 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 0, 2024-04-01T01:50:36.649 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-04-01T01:50:36.649 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-04-01T01:50:36.649 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-04-01T01:50:36.650 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-04-01T01:50:36.650 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.650 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-04-01T01:50:36.650 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-04-01T01:50:36.650 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 0, 2024-04-01T01:50:36.650 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-04-01T01:50:36.650 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-04-01T01:50:36.650 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 1562, 2024-04-01T01:50:36.650 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-04-01T01:50:36.650 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-04-01T01:50:36.650 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 0, 2024-04-01T01:50:36.650 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-04-01T01:50:36.651 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-04-01T01:50:36.651 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 2, 2024-04-01T01:50:36.651 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 4, 2024-04-01T01:50:36.651 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-04-01T01:50:36.651 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-04-01T01:50:36.651 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 49476516, 2024-04-01T01:50:36.651 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 49904520, 2024-04-01T01:50:36.651 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 49900219, 2024-04-01T01:50:36.651 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-04-01T01:50:36.651 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.651 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.651 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.651 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.651 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 1562 2024-04-01T01:50:36.652 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.652 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-04-01T01:50:36.652 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 111, 2024-04-01T01:50:36.652 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 122, 2024-04-01T01:50:36.652 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 11, 2024-04-01T01:50:36.652 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 26, 2024-04-01T01:50:36.652 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 26, 2024-04-01T01:50:36.652 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 0, 2024-04-01T01:50:36.652 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 220, 2024-04-01T01:50:36.652 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 220, 2024-04-01T01:50:36.652 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 0, 2024-04-01T01:50:36.652 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 0, 2024-04-01T01:50:36.652 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 0, 2024-04-01T01:50:36.652 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 0, 2024-04-01T01:50:36.652 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 50360, 2024-04-01T01:50:36.653 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 215396 2024-04-01T01:50:36.653 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.653 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-04-01T01:50:36.653 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 0, 2024-04-01T01:50:36.653 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 0, 2024-04-01T01:50:36.653 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 0, 2024-04-01T01:50:36.653 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 0, 2024-04-01T01:50:36.653 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 0, 2024-04-01T01:50:36.653 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-04-01T01:50:36.653 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.653 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.653 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.653 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.654 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-04-01T01:50:36.654 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.654 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.654 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.654 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.654 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-04-01T01:50:36.654 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.654 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.654 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.654 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.654 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-04-01T01:50:36.654 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.654 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.654 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.655 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.655 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-04-01T01:50:36.655 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.655 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.655 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.655 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.655 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-04-01T01:50:36.655 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.655 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.655 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.655 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.655 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-04-01T01:50:36.655 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.655 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.656 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.656 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.656 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-04-01T01:50:36.656 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.656 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.656 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.656 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.656 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-04-01T01:50:36.656 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.656 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.656 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.657 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.657 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-04-01T01:50:36.657 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.657 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.657 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.657 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.657 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-04-01T01:50:36.657 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.657 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.657 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.657 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.657 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-04-01T01:50:36.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.658 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.658 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.658 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-04-01T01:50:36.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.658 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.659 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.659 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-04-01T01:50:36.659 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.659 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.659 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.659 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.659 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-04-01T01:50:36.659 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.659 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.659 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.659 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.659 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-04-01T01:50:36.659 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.659 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.660 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.660 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.660 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-04-01T01:50:36.660 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.660 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.660 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.660 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.660 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-04-01T01:50:36.660 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.660 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.660 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.660 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.660 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-04-01T01:50:36.660 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.661 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.661 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.661 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.661 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-04-01T01:50:36.661 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.661 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.661 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.661 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.661 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-04-01T01:50:36.661 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.661 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.661 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.661 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.661 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-04-01T01:50:36.662 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.662 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.662 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.662 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.662 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-04-01T01:50:36.662 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.662 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.662 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.662 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.662 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-04-01T01:50:36.662 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.662 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.662 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.663 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.663 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-04-01T01:50:36.663 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.663 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.663 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.663 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.663 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-04-01T01:50:36.663 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.663 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.663 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.663 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.664 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-04-01T01:50:36.664 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.664 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.664 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.664 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.664 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-04-01T01:50:36.665 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.665 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.665 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.665 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.665 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-04-01T01:50:36.665 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.666 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.666 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.666 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.666 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-04-01T01:50:36.666 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-04-01T01:50:36.666 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-04-01T01:50:36.667 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.667 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.667 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.667 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:36.667 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.667 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-04-01T01:50:36.668 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 1, 2024-04-01T01:50:36.668 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 1, 2024-04-01T01:50:36.668 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 0, 2024-04-01T01:50:36.668 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 0, 2024-04-01T01:50:36.668 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-04-01T01:50:36.669 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 0, 2024-04-01T01:50:36.669 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 0, 2024-04-01T01:50:36.669 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 2161, 2024-04-01T01:50:36.669 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-04-01T01:50:36.669 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.670 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-04-01T01:50:36.670 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-04-01T01:50:36.670 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-04-01T01:50:36.670 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-04-01T01:50:36.670 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-04-01T01:50:36.670 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-04-01T01:50:36.671 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-04-01T01:50:36.671 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-01T01:50:36.671 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-04-01T01:50:36.671 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-01T01:50:36.671 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-04-01T01:50:36.671 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-04-01T01:50:36.672 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-04-01T01:50:36.672 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-01T01:50:36.672 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-04-01T01:50:36.673 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-04-01T01:50:36.674 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-04-01T01:50:36.674 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-04-01T01:50:36.674 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-04-01T01:50:36.674 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-01T01:50:36.674 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-04-01T01:50:36.674 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-01T01:50:36.674 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-04-01T01:50:36.674 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-04-01T01:50:36.674 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-04-01T01:50:36.674 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-04-01T01:50:36.675 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-04-01T01:50:36.675 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-01T01:50:36.675 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-04-01T01:50:36.675 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-04-01T01:50:36.675 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-04-01T01:50:36.675 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-04-01T01:50:36.675 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-04-01T01:50:36.675 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-04-01T01:50:36.676 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-04-01T01:50:36.676 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-04-01T01:50:36.676 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-04-01T01:50:36.676 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 196497, 2024-04-01T01:50:36.676 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 49, 2024-04-01T01:50:36.676 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 0, 2024-04-01T01:50:36.676 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 0, 2024-04-01T01:50:36.676 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-04-01T01:50:36.676 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-04-01T01:50:36.676 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-04-01T01:50:36.677 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-04-01T01:50:36.677 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-04-01T01:50:36.677 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-04-01T01:50:36.677 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 9080, 2024-04-01T01:50:36.677 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 174, 2024-04-01T01:50:36.677 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-04-01T01:50:36.677 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-04-01T01:50:36.677 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-04-01T01:50:36.677 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-04-01T01:50:36.677 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 431095, 2024-04-01T01:50:36.677 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 8376, 2024-04-01T01:50:36.677 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-04-01T01:50:36.677 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-04-01T01:50:36.677 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-04-01T01:50:36.678 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-04-01T01:50:36.678 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.678 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-04-01T01:50:36.678 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-04-01T01:50:36.678 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-04-01T01:50:36.678 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 286, 2024-04-01T01:50:36.678 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 26, 2024-04-01T01:50:36.678 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 4, 2024-04-01T01:50:36.678 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 286, 2024-04-01T01:50:36.678 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-04-01T01:50:36.678 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 286, 2024-04-01T01:50:36.678 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 2.657951561, 2024-04-01T01:50:36.678 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.009293536 2024-04-01T01:50:36.678 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.679 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-04-01T01:50:36.679 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-04-01T01:50:36.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 286, 2024-04-01T01:50:36.679 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 287 2024-04-01T01:50:36.679 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.679 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 286, 2024-04-01T01:50:36.679 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 286, 2024-04-01T01:50:36.679 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 0, 2024-04-01T01:50:36.679 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-04-01T01:50:36.679 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-04-01T01:50:36.679 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 164, 2024-04-01T01:50:36.679 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 0, 2024-04-01T01:50:36.679 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 119, 2024-04-01T01:50:36.680 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 0, 2024-04-01T01:50:36.680 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 0, 2024-04-01T01:50:36.680 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-04-01T01:50:36.680 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-04-01T01:50:36.680 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-04-01T01:50:36.680 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-04-01T01:50:36.680 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 0, 2024-04-01T01:50:36.680 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-04-01T01:50:36.680 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-04-01T01:50:36.680 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-04-01T01:50:36.680 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 2, 2024-04-01T01:50:36.680 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 0, 2024-04-01T01:50:36.680 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-04-01T01:50:36.680 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-04-01T01:50:36.681 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-04-01T01:50:36.681 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-04-01T01:50:36.681 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-04-01T01:50:36.681 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-04-01T01:50:36.681 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-04-01T01:50:36.681 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-04-01T01:50:36.681 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-04-01T01:50:36.681 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-04-01T01:50:36.681 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-04-01T01:50:36.681 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-04-01T01:50:36.681 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-04-01T01:50:36.681 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-04-01T01:50:36.681 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-04-01T01:50:36.681 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-04-01T01:50:36.681 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-04-01T01:50:36.682 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-04-01T01:50:36.682 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-04-01T01:50:36.682 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-04-01T01:50:36.682 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-04-01T01:50:36.682 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-04-01T01:50:36.682 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-04-01T01:50:36.682 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-04-01T01:50:36.682 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-04-01T01:50:36.682 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-04-01T01:50:36.682 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 0, 2024-04-01T01:50:36.682 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-04-01T01:50:36.682 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 0, 2024-04-01T01:50:36.682 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 6, 2024-04-01T01:50:36.682 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 6, 2024-04-01T01:50:36.683 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 0, 2024-04-01T01:50:36.683 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-04-01T01:50:36.683 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 0, 2024-04-01T01:50:36.683 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 2, 2024-04-01T01:50:36.683 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 0 2024-04-01T01:50:36.683 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.683 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-04-01T01:50:36.683 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 0, 2024-04-01T01:50:36.683 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-04-01T01:50:36.683 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-04-01T01:50:36.683 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-04-01T01:50:36.683 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.683 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-04-01T01:50:36.683 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-04-01T01:50:36.684 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.684 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-04-01T01:50:36.684 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-04-01T01:50:36.684 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 0, 2024-04-01T01:50:36.684 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-04-01T01:50:36.684 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 0, 2024-04-01T01:50:36.684 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 0, 2024-04-01T01:50:36.684 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 0, 2024-04-01T01:50:36.684 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-04-01T01:50:36.684 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.684 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-01T01:50:36.684 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:36.684 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-04-01T01:50:36.684 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:36.685 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 322, 2024-04-01T01:50:36.685 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 3907953, 2024-04-01T01:50:36.685 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:36.685 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 322, 2024-04-01T01:50:36.685 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-01T01:50:36.685 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-01T01:50:36.685 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 322, 2024-04-01T01:50:36.685 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 3907953, 2024-04-01T01:50:36.685 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:36.685 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.685 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.685 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.685 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:36.685 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.686 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-04-01T01:50:36.686 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:36.686 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-04-01T01:50:36.686 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:36.686 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-01T01:50:36.686 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-01T01:50:36.686 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:36.686 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:36.686 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 286, 2024-04-01T01:50:36.686 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 3693023, 2024-04-01T01:50:36.686 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 286, 2024-04-01T01:50:36.686 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 3693023, 2024-04-01T01:50:36.686 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:36.687 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.687 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.687 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.687 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:36.687 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.687 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-04-01T01:50:36.687 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:36.687 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-04-01T01:50:36.687 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:36.687 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-01T01:50:36.687 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-01T01:50:36.687 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:36.687 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:36.687 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 286, 2024-04-01T01:50:36.688 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 286, 2024-04-01T01:50:36.688 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 286, 2024-04-01T01:50:36.688 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 286, 2024-04-01T01:50:36.688 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:36.688 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.688 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.688 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.688 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:36.688 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.688 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-04-01T01:50:36.688 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:36.688 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-04-01T01:50:36.688 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:36.688 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-01T01:50:36.689 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-01T01:50:36.689 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:36.689 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:36.689 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-01T01:50:36.689 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-01T01:50:36.689 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-04-01T01:50:36.689 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-04-01T01:50:36.689 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:36.689 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.689 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.689 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.689 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:36.690 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:36.690 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x5636e89366e0": { 2024-04-01T01:50:36.690 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:36.690 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-04-01T01:50:36.690 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:36.690 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-01T01:50:36.690 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-01T01:50:36.690 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:36.690 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:36.690 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-01T01:50:36.690 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-01T01:50:36.691 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-04-01T01:50:36.691 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-04-01T01:50:36.691 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:36.691 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:36.691 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:36.691 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:36.691 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:36.691 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:36.691 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-01T01:50:37.214 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:36 smithi189 ceph-mon[35753]: pgmap v873: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-01T01:50:37.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:36 smithi070 ceph-mon[32179]: pgmap v873: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-01T01:50:37.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:36 smithi081 ceph-mon[35142]: pgmap v873: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-01T01:50:37.540 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-01T01:50:37.540 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-01T01:50:37.540 DEBUG:tasks.check_counter:Getting stats from f 2024-04-01T01:50:37.541 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph daemon mds.f perf dump 2024-04-01T01:50:37.819 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-01T01:50:38.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:38 smithi189 ceph-mon[35753]: pgmap v874: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-01T01:50:39.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:38 smithi070 ceph-mon[32179]: pgmap v874: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-01T01:50:39.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:38 smithi081 ceph-mon[35142]: pgmap v874: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-01T01:50:39.654 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-01T01:50:39.655 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-04-01T01:50:39.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 413, 2024-04-01T01:50:39.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 397, 2024-04-01T01:50:39.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 871721, 2024-04-01T01:50:39.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 260617, 2024-04-01T01:50:39.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 13, 2024-04-01T01:50:39.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 2, 2024-04-01T01:50:39.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.053469549, 2024-04-01T01:50:39.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.016924411, 2024-04-01T01:50:39.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.027662972, 2024-04-01T01:50:39.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.007591278, 2024-04-01T01:50:39.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:39.655 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 396, 2024-04-01T01:50:39.656 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.030898703, 2024-04-01T01:50:39.656 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000078027 2024-04-01T01:50:39.656 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.656 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:39.656 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 45, 2024-04-01T01:50:39.656 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000007519, 2024-04-01T01:50:39.656 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000167 2024-04-01T01:50:39.656 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.656 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 865280, 2024-04-01T01:50:39.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 204032 2024-04-01T01:50:39.657 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.657 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-04-01T01:50:39.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 75, 2024-04-01T01:50:39.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 315, 2024-04-01T01:50:39.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 12783, 2024-04-01T01:50:39.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 1271111, 2024-04-01T01:50:39.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 67, 2024-04-01T01:50:39.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 5, 2024-04-01T01:50:39.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.037925601, 2024-04-01T01:50:39.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.011966290, 2024-04-01T01:50:39.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.022561192, 2024-04-01T01:50:39.658 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.000392994, 2024-04-01T01:50:39.658 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:39.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 313, 2024-04-01T01:50:39.658 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.038916545, 2024-04-01T01:50:39.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000124334 2024-04-01T01:50:39.658 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.658 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:39.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 58, 2024-04-01T01:50:39.658 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000008835, 2024-04-01T01:50:39.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000152 2024-04-01T01:50:39.658 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.658 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-01T01:50:39.658 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 586768 2024-04-01T01:50:39.659 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.659 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-04-01T01:50:39.659 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 908, 2024-04-01T01:50:39.659 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 885, 2024-04-01T01:50:39.659 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 221730, 2024-04-01T01:50:39.659 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 352438, 2024-04-01T01:50:39.659 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 8, 2024-04-01T01:50:39.659 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 4, 2024-04-01T01:50:39.659 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.075561594, 2024-04-01T01:50:39.659 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.027486910, 2024-04-01T01:50:39.659 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.048148044, 2024-04-01T01:50:39.659 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.005702314, 2024-04-01T01:50:39.659 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:39.659 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 884, 2024-04-01T01:50:39.659 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.077918843, 2024-04-01T01:50:39.660 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000088143 2024-04-01T01:50:39.660 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.660 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:39.660 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1724, 2024-04-01T01:50:39.660 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000399379, 2024-04-01T01:50:39.660 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000231 2024-04-01T01:50:39.660 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.660 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-01T01:50:39.660 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 448 2024-04-01T01:50:39.660 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.660 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-04-01T01:50:39.660 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-04-01T01:50:39.660 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-04-01T01:50:39.661 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.661 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-04-01T01:50:39.661 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-04-01T01:50:39.661 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-04-01T01:50:39.661 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 44, 2024-04-01T01:50:39.661 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.002320743, 2024-04-01T01:50:39.661 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000052744 2024-04-01T01:50:39.661 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:39.661 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.661 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-04-01T01:50:39.661 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-04-01T01:50:39.661 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-04-01T01:50:39.661 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2, 2024-04-01T01:50:39.661 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000006440, 2024-04-01T01:50:39.662 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000003220 2024-04-01T01:50:39.662 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:39.662 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.662 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-04-01T01:50:39.662 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 4, 2024-04-01T01:50:39.662 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 2, 2024-04-01T01:50:39.662 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-04-01T01:50:39.663 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2, 2024-04-01T01:50:39.663 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000482049, 2024-04-01T01:50:39.663 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000241024 2024-04-01T01:50:39.663 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.663 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-04-01T01:50:39.663 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 2, 2024-04-01T01:50:39.663 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-04-01T01:50:39.663 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-04-01T01:50:39.663 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 11, 2024-04-01T01:50:39.663 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-04-01T01:50:39.664 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-04-01T01:50:39.664 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 11, 2024-04-01T01:50:39.664 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 0, 2024-04-01T01:50:39.664 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 11, 2024-04-01T01:50:39.664 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 0, 2024-04-01T01:50:39.664 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 11, 2024-04-01T01:50:39.664 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 0, 2024-04-01T01:50:39.664 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 1, 2024-04-01T01:50:39.664 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 1, 2024-04-01T01:50:39.664 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 2, 2024-04-01T01:50:39.664 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 6, 2024-04-01T01:50:39.664 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 3, 2024-04-01T01:50:39.664 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 0, 2024-04-01T01:50:39.664 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 1, 2024-04-01T01:50:39.665 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-04-01T01:50:39.665 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-04-01T01:50:39.665 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 0, 2024-04-01T01:50:39.665 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-04-01T01:50:39.665 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-04-01T01:50:39.665 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 0, 2024-04-01T01:50:39.665 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 0, 2024-04-01T01:50:39.665 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 0, 2024-04-01T01:50:39.665 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 0, 2024-04-01T01:50:39.665 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-04-01T01:50:39.665 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 0, 2024-04-01T01:50:39.665 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-04-01T01:50:39.665 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 1022, 2024-04-01T01:50:39.665 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 1557107758, 2024-04-01T01:50:39.666 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 0, 2024-04-01T01:50:39.666 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 0, 2024-04-01T01:50:39.666 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-04-01T01:50:39.666 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-04-01T01:50:39.666 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-04-01T01:50:39.666 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-04-01T01:50:39.666 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-04-01T01:50:39.666 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-04-01T01:50:39.667 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 0, 2024-04-01T01:50:39.667 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-04-01T01:50:39.667 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 0, 2024-04-01T01:50:39.667 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 0, 2024-04-01T01:50:39.667 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-04-01T01:50:39.668 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-01T01:50:39.668 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-01T01:50:39.668 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 0, 2024-04-01T01:50:39.668 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 0, 2024-04-01T01:50:39.668 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 0, 2024-04-01T01:50:39.668 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 2 2024-04-01T01:50:39.669 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.669 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-04-01T01:50:39.669 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 0, 2024-04-01T01:50:39.669 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 0, 2024-04-01T01:50:39.669 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 0, 2024-04-01T01:50:39.669 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 2, 2024-04-01T01:50:39.669 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 0, 2024-04-01T01:50:39.669 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-04-01T01:50:39.669 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-04-01T01:50:39.669 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-04-01T01:50:39.669 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 0, 2024-04-01T01:50:39.669 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 0, 2024-04-01T01:50:39.669 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-04-01T01:50:39.669 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-04-01T01:50:39.670 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-04-01T01:50:39.670 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-04-01T01:50:39.670 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-04-01T01:50:39.670 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-04-01T01:50:39.670 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-04-01T01:50:39.670 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_path": 0, 2024-04-01T01:50:39.670 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_inode": 0, 2024-04-01T01:50:39.670 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-04-01T01:50:39.670 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 0, 2024-04-01T01:50:39.670 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-04-01T01:50:39.670 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-04-01T01:50:39.670 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-04-01T01:50:39.670 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-04-01T01:50:39.670 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.671 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-04-01T01:50:39.671 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-04-01T01:50:39.671 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 5, 2024-04-01T01:50:39.671 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-04-01T01:50:39.671 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-04-01T01:50:39.671 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 5, 2024-04-01T01:50:39.671 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-04-01T01:50:39.671 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-04-01T01:50:39.671 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 2, 2024-04-01T01:50:39.671 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-04-01T01:50:39.671 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-04-01T01:50:39.671 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 2, 2024-04-01T01:50:39.671 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 2, 2024-04-01T01:50:39.671 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-04-01T01:50:39.671 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-04-01T01:50:39.671 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 4194304, 2024-04-01T01:50:39.672 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 4195901, 2024-04-01T01:50:39.672 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 0, 2024-04-01T01:50:39.672 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-04-01T01:50:39.672 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 4, 2024-04-01T01:50:39.672 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.006619751, 2024-04-01T01:50:39.672 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001654937 2024-04-01T01:50:39.672 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.672 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 0 2024-04-01T01:50:39.672 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.672 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-04-01T01:50:39.672 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 14, 2024-04-01T01:50:39.672 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 14, 2024-04-01T01:50:39.672 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 0, 2024-04-01T01:50:39.672 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 12, 2024-04-01T01:50:39.672 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 13, 2024-04-01T01:50:39.672 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 1, 2024-04-01T01:50:39.673 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 11, 2024-04-01T01:50:39.673 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 11, 2024-04-01T01:50:39.673 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 0, 2024-04-01T01:50:39.673 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 1, 2024-04-01T01:50:39.673 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 2, 2024-04-01T01:50:39.673 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 1, 2024-04-01T01:50:39.673 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 52984, 2024-04-01T01:50:39.673 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 223612 2024-04-01T01:50:39.673 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.673 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-04-01T01:50:39.673 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 5, 2024-04-01T01:50:39.674 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 0, 2024-04-01T01:50:39.674 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 4, 2024-04-01T01:50:39.674 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 17, 2024-04-01T01:50:39.674 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 0, 2024-04-01T01:50:39.674 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-04-01T01:50:39.674 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.674 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.674 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.674 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.675 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-04-01T01:50:39.675 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2, 2024-04-01T01:50:39.675 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000482049, 2024-04-01T01:50:39.675 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000241024 2024-04-01T01:50:39.675 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.675 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-04-01T01:50:39.675 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.675 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.675 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.675 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.675 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-04-01T01:50:39.675 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.675 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.675 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.675 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.675 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-04-01T01:50:39.676 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.676 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.676 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.676 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.676 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-04-01T01:50:39.676 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.676 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.676 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.676 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.676 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-04-01T01:50:39.676 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.676 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.676 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.676 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.676 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-04-01T01:50:39.677 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.677 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.677 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.677 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.677 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-04-01T01:50:39.677 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.677 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.677 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.677 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.677 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-04-01T01:50:39.677 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.677 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.677 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.677 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.678 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-04-01T01:50:39.678 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.678 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.678 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.678 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.678 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-04-01T01:50:39.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.679 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.679 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.679 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-04-01T01:50:39.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.679 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.679 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.680 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-04-01T01:50:39.680 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.680 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.680 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.680 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.680 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-04-01T01:50:39.681 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.681 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.681 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.681 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.681 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-04-01T01:50:39.681 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.681 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.681 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.681 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.681 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-04-01T01:50:39.681 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.681 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.681 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.681 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.682 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-04-01T01:50:39.682 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.682 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.682 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.682 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.682 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-04-01T01:50:39.682 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.682 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.682 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.682 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.682 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-04-01T01:50:39.682 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.682 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.683 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.683 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.683 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-04-01T01:50:39.683 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.683 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.683 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.683 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.683 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-04-01T01:50:39.683 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.683 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.683 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.683 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.683 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-04-01T01:50:39.683 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.684 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.684 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.684 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.684 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-04-01T01:50:39.684 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.684 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.684 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.684 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.685 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-04-01T01:50:39.685 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.685 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.685 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.685 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.686 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-04-01T01:50:39.686 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.686 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.686 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.686 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.686 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-04-01T01:50:39.687 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.687 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.687 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.687 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.687 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-04-01T01:50:39.687 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.688 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.688 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.688 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.688 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-04-01T01:50:39.688 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.688 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.689 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.689 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.689 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-04-01T01:50:39.689 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-04-01T01:50:39.689 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-04-01T01:50:39.689 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.690 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.690 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.690 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:39.690 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.690 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-04-01T01:50:39.691 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 1, 2024-04-01T01:50:39.691 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 2, 2024-04-01T01:50:39.691 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 1, 2024-04-01T01:50:39.691 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 1, 2024-04-01T01:50:39.691 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-04-01T01:50:39.691 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 1, 2024-04-01T01:50:39.692 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 1, 2024-04-01T01:50:39.692 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 1024, 2024-04-01T01:50:39.692 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-04-01T01:50:39.692 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.692 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-04-01T01:50:39.692 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-04-01T01:50:39.692 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-04-01T01:50:39.692 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-04-01T01:50:39.692 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-04-01T01:50:39.692 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-04-01T01:50:39.692 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-04-01T01:50:39.693 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-01T01:50:39.693 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-04-01T01:50:39.693 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-01T01:50:39.693 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-04-01T01:50:39.693 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-04-01T01:50:39.693 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-04-01T01:50:39.693 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-01T01:50:39.693 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-04-01T01:50:39.693 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-04-01T01:50:39.693 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-04-01T01:50:39.694 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-04-01T01:50:39.694 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-04-01T01:50:39.694 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-01T01:50:39.694 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-04-01T01:50:39.694 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-01T01:50:39.694 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-04-01T01:50:39.694 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-04-01T01:50:39.694 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-04-01T01:50:39.694 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-04-01T01:50:39.694 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-04-01T01:50:39.694 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-01T01:50:39.695 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-04-01T01:50:39.695 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-04-01T01:50:39.695 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-04-01T01:50:39.695 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-04-01T01:50:39.695 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-04-01T01:50:39.695 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-04-01T01:50:39.695 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-04-01T01:50:39.695 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-04-01T01:50:39.695 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-04-01T01:50:39.695 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 239782, 2024-04-01T01:50:39.695 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 82, 2024-04-01T01:50:39.695 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 0, 2024-04-01T01:50:39.695 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 0, 2024-04-01T01:50:39.695 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-04-01T01:50:39.695 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-04-01T01:50:39.696 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-04-01T01:50:39.696 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-04-01T01:50:39.696 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-04-01T01:50:39.696 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-04-01T01:50:39.696 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 9080, 2024-04-01T01:50:39.696 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 174, 2024-04-01T01:50:39.696 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-04-01T01:50:39.696 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-04-01T01:50:39.696 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-04-01T01:50:39.696 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-04-01T01:50:39.696 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 55240, 2024-04-01T01:50:39.696 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 217, 2024-04-01T01:50:39.696 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-04-01T01:50:39.696 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-04-01T01:50:39.696 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-04-01T01:50:39.697 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-04-01T01:50:39.697 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.697 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-04-01T01:50:39.697 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-04-01T01:50:39.697 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-04-01T01:50:39.697 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 41, 2024-04-01T01:50:39.697 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 11993, 2024-04-01T01:50:39.697 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 33, 2024-04-01T01:50:39.697 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 41, 2024-04-01T01:50:39.697 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-04-01T01:50:39.697 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 41, 2024-04-01T01:50:39.697 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.221995623, 2024-04-01T01:50:39.697 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.005414527 2024-04-01T01:50:39.697 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.697 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-04-01T01:50:39.697 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-04-01T01:50:39.698 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 41, 2024-04-01T01:50:39.698 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 64 2024-04-01T01:50:39.698 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.698 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 41, 2024-04-01T01:50:39.698 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 0, 2024-04-01T01:50:39.698 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 41, 2024-04-01T01:50:39.698 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-04-01T01:50:39.698 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-04-01T01:50:39.698 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 0, 2024-04-01T01:50:39.698 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 10, 2024-04-01T01:50:39.698 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 0, 2024-04-01T01:50:39.698 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 4, 2024-04-01T01:50:39.698 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 7, 2024-04-01T01:50:39.698 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 6, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 0, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 20, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-04-01T01:50:39.699 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-04-01T01:50:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-04-01T01:50:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-04-01T01:50:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-04-01T01:50:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-04-01T01:50:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-04-01T01:50:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-04-01T01:50:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-04-01T01:50:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-04-01T01:50:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-04-01T01:50:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-04-01T01:50:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-04-01T01:50:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-04-01T01:50:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-04-01T01:50:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-04-01T01:50:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-04-01T01:50:39.701 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-04-01T01:50:39.701 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-04-01T01:50:39.701 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-04-01T01:50:39.701 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 95, 2024-04-01T01:50:39.701 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-04-01T01:50:39.701 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 6, 2024-04-01T01:50:39.701 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 8, 2024-04-01T01:50:39.701 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 12, 2024-04-01T01:50:39.701 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 4, 2024-04-01T01:50:39.701 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-04-01T01:50:39.701 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 16, 2024-04-01T01:50:39.702 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 0, 2024-04-01T01:50:39.702 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 1 2024-04-01T01:50:39.702 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.702 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-04-01T01:50:39.702 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 1, 2024-04-01T01:50:39.703 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-04-01T01:50:39.703 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-04-01T01:50:39.703 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-04-01T01:50:39.703 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.703 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-04-01T01:50:39.703 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-04-01T01:50:39.703 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.704 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-04-01T01:50:39.704 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-04-01T01:50:39.704 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 0, 2024-04-01T01:50:39.704 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-04-01T01:50:39.704 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 0, 2024-04-01T01:50:39.704 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 0, 2024-04-01T01:50:39.704 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 0, 2024-04-01T01:50:39.704 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-04-01T01:50:39.704 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.704 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-01T01:50:39.704 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:39.704 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-04-01T01:50:39.705 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:39.705 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 1392, 2024-04-01T01:50:39.705 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 958627, 2024-04-01T01:50:39.705 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:39.705 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 1392, 2024-04-01T01:50:39.705 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-01T01:50:39.705 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-01T01:50:39.705 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1392, 2024-04-01T01:50:39.705 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 958627, 2024-04-01T01:50:39.705 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:39.705 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.705 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.705 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.705 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:39.705 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.706 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-04-01T01:50:39.706 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:39.706 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-04-01T01:50:39.706 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:39.706 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-01T01:50:39.706 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-01T01:50:39.706 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:39.706 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:39.706 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 41, 2024-04-01T01:50:39.706 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 11993, 2024-04-01T01:50:39.706 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 41, 2024-04-01T01:50:39.706 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 11993, 2024-04-01T01:50:39.706 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:39.706 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.706 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.707 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.707 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:39.707 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.707 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-04-01T01:50:39.707 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:39.707 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-04-01T01:50:39.707 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:39.707 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-01T01:50:39.707 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-01T01:50:39.707 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:39.707 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:39.707 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 41, 2024-04-01T01:50:39.707 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 41, 2024-04-01T01:50:39.708 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 41, 2024-04-01T01:50:39.708 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 41, 2024-04-01T01:50:39.708 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:39.708 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.708 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.708 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.708 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:39.708 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.708 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-04-01T01:50:39.708 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:39.708 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-04-01T01:50:39.708 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:39.708 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-01T01:50:39.709 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-01T01:50:39.709 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:39.709 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:39.709 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-01T01:50:39.709 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-01T01:50:39.709 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-04-01T01:50:39.709 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-04-01T01:50:39.709 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:39.709 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.710 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.710 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.710 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:39.710 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:39.710 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x55b7b6c72500": { 2024-04-01T01:50:39.710 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:39.710 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-04-01T01:50:39.710 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:39.710 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 5, 2024-04-01T01:50:39.710 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 1597, 2024-04-01T01:50:39.710 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:39.710 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 5, 2024-04-01T01:50:39.710 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-01T01:50:39.710 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-01T01:50:39.710 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 4, 2024-04-01T01:50:39.711 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1597, 2024-04-01T01:50:39.711 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:39.711 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:39.711 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:39.711 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:39.711 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:39.711 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:39.711 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-01T01:50:40.508 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-01T01:50:40.508 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-01T01:50:40.508 DEBUG:tasks.check_counter:Getting stats from i 2024-04-01T01:50:40.509 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph daemon mds.i perf dump 2024-04-01T01:50:40.832 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-01T01:50:41.219 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:40 smithi189 ceph-mon[35753]: pgmap v875: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 8 op/s 2024-04-01T01:50:41.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:40 smithi070 ceph-mon[32179]: pgmap v875: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 8 op/s 2024-04-01T01:50:41.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:40 smithi081 ceph-mon[35142]: pgmap v875: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 8 op/s 2024-04-01T01:50:42.594 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-01T01:50:42.595 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-04-01T01:50:42.595 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 465, 2024-04-01T01:50:42.595 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 447, 2024-04-01T01:50:42.595 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 883312, 2024-04-01T01:50:42.595 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 930521, 2024-04-01T01:50:42.595 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 71, 2024-04-01T01:50:42.595 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 6, 2024-04-01T01:50:42.595 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.074580478, 2024-04-01T01:50:42.595 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.018296659, 2024-04-01T01:50:42.595 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.046948124, 2024-04-01T01:50:42.595 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.007607864, 2024-04-01T01:50:42.595 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:42.595 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 446, 2024-04-01T01:50:42.596 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.033843472, 2024-04-01T01:50:42.596 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000075882 2024-04-01T01:50:42.596 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.596 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:42.596 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 99, 2024-04-01T01:50:42.596 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000016894, 2024-04-01T01:50:42.596 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000170 2024-04-01T01:50:42.596 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.596 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 868064, 2024-04-01T01:50:42.596 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 201104 2024-04-01T01:50:42.596 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.596 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-04-01T01:50:42.596 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 1205, 2024-04-01T01:50:42.597 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 1234, 2024-04-01T01:50:42.597 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 293145, 2024-04-01T01:50:42.597 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 877445, 2024-04-01T01:50:42.597 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 10, 2024-04-01T01:50:42.597 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 5, 2024-04-01T01:50:42.597 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.102359936, 2024-04-01T01:50:42.597 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.039976065, 2024-04-01T01:50:42.597 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.066925143, 2024-04-01T01:50:42.597 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.005873311, 2024-04-01T01:50:42.597 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:42.597 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1233, 2024-04-01T01:50:42.597 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.091088028, 2024-04-01T01:50:42.597 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000073875 2024-04-01T01:50:42.597 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:42.598 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2089, 2024-04-01T01:50:42.598 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000597420, 2024-04-01T01:50:42.598 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000285 2024-04-01T01:50:42.598 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-01T01:50:42.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 499456 2024-04-01T01:50:42.598 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.598 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-04-01T01:50:42.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 16, 2024-04-01T01:50:42.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 18, 2024-04-01T01:50:42.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 4409, 2024-04-01T01:50:42.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 12978, 2024-04-01T01:50:42.598 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 10, 2024-04-01T01:50:42.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 5, 2024-04-01T01:50:42.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.004370238, 2024-04-01T01:50:42.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.000244584, 2024-04-01T01:50:42.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.003295890, 2024-04-01T01:50:42.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.000361984, 2024-04-01T01:50:42.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:42.599 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 16, 2024-04-01T01:50:42.599 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.015713174, 2024-04-01T01:50:42.599 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000982073 2024-04-01T01:50:42.599 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.599 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:42.599 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.599 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.599 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.600 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.600 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-01T01:50:42.600 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 0 2024-04-01T01:50:42.600 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.600 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-04-01T01:50:42.600 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-04-01T01:50:42.600 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-04-01T01:50:42.600 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.600 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-04-01T01:50:42.600 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-04-01T01:50:42.600 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-04-01T01:50:42.600 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 37, 2024-04-01T01:50:42.600 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.001555389, 2024-04-01T01:50:42.600 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000042037 2024-04-01T01:50:42.601 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:42.601 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.601 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-04-01T01:50:42.601 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-04-01T01:50:42.601 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-04-01T01:50:42.601 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2, 2024-04-01T01:50:42.601 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000007324, 2024-04-01T01:50:42.601 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000003662 2024-04-01T01:50:42.601 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:42.601 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.601 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-04-01T01:50:42.601 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 2, 2024-04-01T01:50:42.601 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 2, 2024-04-01T01:50:42.601 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-04-01T01:50:42.602 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2, 2024-04-01T01:50:42.602 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.002194359, 2024-04-01T01:50:42.602 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001097179 2024-04-01T01:50:42.602 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.602 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-04-01T01:50:42.602 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 0, 2024-04-01T01:50:42.602 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-04-01T01:50:42.602 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-04-01T01:50:42.602 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 11, 2024-04-01T01:50:42.602 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-04-01T01:50:42.602 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-04-01T01:50:42.602 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 12, 2024-04-01T01:50:42.602 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 0, 2024-04-01T01:50:42.603 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 12, 2024-04-01T01:50:42.603 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 0, 2024-04-01T01:50:42.603 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 12, 2024-04-01T01:50:42.603 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 0, 2024-04-01T01:50:42.603 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 2, 2024-04-01T01:50:42.603 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 2, 2024-04-01T01:50:42.603 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 2, 2024-04-01T01:50:42.603 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 4, 2024-04-01T01:50:42.603 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 3, 2024-04-01T01:50:42.603 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 0, 2024-04-01T01:50:42.603 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 1, 2024-04-01T01:50:42.603 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-04-01T01:50:42.603 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-04-01T01:50:42.603 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 0, 2024-04-01T01:50:42.604 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-04-01T01:50:42.604 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-04-01T01:50:42.604 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 0, 2024-04-01T01:50:42.604 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 0, 2024-04-01T01:50:42.604 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 0, 2024-04-01T01:50:42.604 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 0, 2024-04-01T01:50:42.604 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-04-01T01:50:42.604 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 0, 2024-04-01T01:50:42.604 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-04-01T01:50:42.604 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 1022, 2024-04-01T01:50:42.604 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 1557107758, 2024-04-01T01:50:42.604 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 0, 2024-04-01T01:50:42.604 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 0, 2024-04-01T01:50:42.604 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-04-01T01:50:42.605 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-04-01T01:50:42.605 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-04-01T01:50:42.605 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-04-01T01:50:42.605 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-04-01T01:50:42.605 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-04-01T01:50:42.605 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 0, 2024-04-01T01:50:42.605 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-04-01T01:50:42.605 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 0, 2024-04-01T01:50:42.605 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 0, 2024-04-01T01:50:42.605 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-04-01T01:50:42.605 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-01T01:50:42.605 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-01T01:50:42.605 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 0, 2024-04-01T01:50:42.605 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 0, 2024-04-01T01:50:42.606 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 0, 2024-04-01T01:50:42.606 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 0 2024-04-01T01:50:42.606 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.606 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-04-01T01:50:42.606 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 0, 2024-04-01T01:50:42.606 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 0, 2024-04-01T01:50:42.606 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 0, 2024-04-01T01:50:42.606 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 2, 2024-04-01T01:50:42.606 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 0, 2024-04-01T01:50:42.606 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-04-01T01:50:42.606 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-04-01T01:50:42.606 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-04-01T01:50:42.606 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 0, 2024-04-01T01:50:42.606 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 0, 2024-04-01T01:50:42.607 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-04-01T01:50:42.607 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-04-01T01:50:42.607 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-04-01T01:50:42.607 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-04-01T01:50:42.607 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-04-01T01:50:42.607 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-04-01T01:50:42.607 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-04-01T01:50:42.607 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_path": 0, 2024-04-01T01:50:42.607 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_inode": 0, 2024-04-01T01:50:42.607 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-04-01T01:50:42.607 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 0, 2024-04-01T01:50:42.607 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-04-01T01:50:42.607 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-04-01T01:50:42.607 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-04-01T01:50:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-04-01T01:50:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-04-01T01:50:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-04-01T01:50:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 5, 2024-04-01T01:50:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-04-01T01:50:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-04-01T01:50:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 5, 2024-04-01T01:50:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-04-01T01:50:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-04-01T01:50:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 2, 2024-04-01T01:50:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-04-01T01:50:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-04-01T01:50:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 2, 2024-04-01T01:50:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 2, 2024-04-01T01:50:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-04-01T01:50:42.609 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-04-01T01:50:42.609 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 4194304, 2024-04-01T01:50:42.609 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 4195901, 2024-04-01T01:50:42.609 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 0, 2024-04-01T01:50:42.609 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-04-01T01:50:42.609 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 4, 2024-04-01T01:50:42.609 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.006945956, 2024-04-01T01:50:42.609 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001736489 2024-04-01T01:50:42.609 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.609 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 0 2024-04-01T01:50:42.609 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.609 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-04-01T01:50:42.609 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 15, 2024-04-01T01:50:42.609 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 15, 2024-04-01T01:50:42.610 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 0, 2024-04-01T01:50:42.610 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 13, 2024-04-01T01:50:42.610 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 14, 2024-04-01T01:50:42.610 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 1, 2024-04-01T01:50:42.610 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 12, 2024-04-01T01:50:42.610 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 12, 2024-04-01T01:50:42.610 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 0, 2024-04-01T01:50:42.610 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 2, 2024-04-01T01:50:42.610 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 2, 2024-04-01T01:50:42.610 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 0, 2024-04-01T01:50:42.610 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 52928, 2024-04-01T01:50:42.610 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 223612 2024-04-01T01:50:42.610 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.610 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-04-01T01:50:42.611 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 6, 2024-04-01T01:50:42.611 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 0, 2024-04-01T01:50:42.611 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 2, 2024-04-01T01:50:42.611 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 17, 2024-04-01T01:50:42.611 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 1, 2024-04-01T01:50:42.611 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-04-01T01:50:42.611 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.611 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.611 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.611 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.611 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-04-01T01:50:42.611 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2, 2024-04-01T01:50:42.611 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.002194359, 2024-04-01T01:50:42.611 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001097179 2024-04-01T01:50:42.612 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.612 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-04-01T01:50:42.612 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.612 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.612 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.612 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.612 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-04-01T01:50:42.612 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.612 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.612 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.612 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.612 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-04-01T01:50:42.612 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.612 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.612 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.613 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.613 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-04-01T01:50:42.613 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.613 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.613 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.613 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.613 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-04-01T01:50:42.613 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.613 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.613 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.613 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.613 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-04-01T01:50:42.613 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.613 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.614 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.614 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.614 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-04-01T01:50:42.614 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.614 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.614 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.614 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.614 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-04-01T01:50:42.614 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.614 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.614 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.614 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.614 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-04-01T01:50:42.615 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.615 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.615 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.615 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.615 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-04-01T01:50:42.615 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.615 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.616 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.616 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.616 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-04-01T01:50:42.616 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.616 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.616 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.616 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.616 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-04-01T01:50:42.616 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.617 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.617 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.617 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.617 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-04-01T01:50:42.617 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.617 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.617 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.618 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.618 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-04-01T01:50:42.618 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.618 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.618 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.619 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.619 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-04-01T01:50:42.619 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.619 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.619 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.619 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.619 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-04-01T01:50:42.619 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.619 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.619 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.619 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.619 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-04-01T01:50:42.619 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.619 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.620 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.620 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.620 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-04-01T01:50:42.620 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.620 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.620 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.620 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.620 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-04-01T01:50:42.620 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.620 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.620 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.620 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.620 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-04-01T01:50:42.621 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.621 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.621 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.621 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.621 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-04-01T01:50:42.621 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.621 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.621 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.621 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.621 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-04-01T01:50:42.621 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.621 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.621 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.621 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.621 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-04-01T01:50:42.622 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.622 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.622 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.622 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.622 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-04-01T01:50:42.622 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.622 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.622 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.622 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.622 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-04-01T01:50:42.622 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.622 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.622 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.622 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.623 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-04-01T01:50:42.623 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.623 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.623 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.623 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.623 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-04-01T01:50:42.623 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.623 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.623 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.623 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.623 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-04-01T01:50:42.623 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-04-01T01:50:42.623 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-04-01T01:50:42.623 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.624 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.624 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.624 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:42.624 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.624 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-04-01T01:50:42.624 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 1, 2024-04-01T01:50:42.624 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 2, 2024-04-01T01:50:42.624 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 1, 2024-04-01T01:50:42.624 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 1, 2024-04-01T01:50:42.624 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-04-01T01:50:42.624 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 1, 2024-04-01T01:50:42.624 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 1, 2024-04-01T01:50:42.624 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 1027, 2024-04-01T01:50:42.624 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-04-01T01:50:42.625 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.625 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-04-01T01:50:42.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-04-01T01:50:42.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-04-01T01:50:42.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-04-01T01:50:42.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-04-01T01:50:42.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-04-01T01:50:42.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-04-01T01:50:42.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-01T01:50:42.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-04-01T01:50:42.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-01T01:50:42.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-04-01T01:50:42.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-04-01T01:50:42.625 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-04-01T01:50:42.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-01T01:50:42.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-04-01T01:50:42.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-04-01T01:50:42.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-04-01T01:50:42.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-04-01T01:50:42.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-04-01T01:50:42.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-01T01:50:42.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-04-01T01:50:42.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-01T01:50:42.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-04-01T01:50:42.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-04-01T01:50:42.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-04-01T01:50:42.626 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-04-01T01:50:42.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-04-01T01:50:42.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-01T01:50:42.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-04-01T01:50:42.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-04-01T01:50:42.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-04-01T01:50:42.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-04-01T01:50:42.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-04-01T01:50:42.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-04-01T01:50:42.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-04-01T01:50:42.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-04-01T01:50:42.627 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-04-01T01:50:42.627 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 255713, 2024-04-01T01:50:42.627 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 92, 2024-04-01T01:50:42.628 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 0, 2024-04-01T01:50:42.628 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 0, 2024-04-01T01:50:42.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-04-01T01:50:42.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-04-01T01:50:42.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-04-01T01:50:42.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-04-01T01:50:42.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-04-01T01:50:42.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-04-01T01:50:42.628 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 9080, 2024-04-01T01:50:42.629 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 174, 2024-04-01T01:50:42.629 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-04-01T01:50:42.629 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-04-01T01:50:42.629 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-04-01T01:50:42.629 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-04-01T01:50:42.629 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 59792, 2024-04-01T01:50:42.629 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 233, 2024-04-01T01:50:42.629 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-04-01T01:50:42.629 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-04-01T01:50:42.629 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-04-01T01:50:42.629 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-04-01T01:50:42.629 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.629 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-04-01T01:50:42.629 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-04-01T01:50:42.630 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-04-01T01:50:42.630 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 41, 2024-04-01T01:50:42.630 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 12054, 2024-04-01T01:50:42.630 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 28, 2024-04-01T01:50:42.630 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 41, 2024-04-01T01:50:42.630 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-04-01T01:50:42.630 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 41, 2024-04-01T01:50:42.630 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.222995607, 2024-04-01T01:50:42.630 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.005438917 2024-04-01T01:50:42.630 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.630 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-04-01T01:50:42.630 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-04-01T01:50:42.630 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 41, 2024-04-01T01:50:42.630 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 64 2024-04-01T01:50:42.631 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.631 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 41, 2024-04-01T01:50:42.631 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 0, 2024-04-01T01:50:42.631 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 41, 2024-04-01T01:50:42.631 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-04-01T01:50:42.631 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-04-01T01:50:42.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 0, 2024-04-01T01:50:42.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 10, 2024-04-01T01:50:42.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 0, 2024-04-01T01:50:42.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 4, 2024-04-01T01:50:42.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 7, 2024-04-01T01:50:42.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-04-01T01:50:42.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-04-01T01:50:42.631 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-04-01T01:50:42.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-04-01T01:50:42.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 6, 2024-04-01T01:50:42.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-04-01T01:50:42.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-04-01T01:50:42.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-04-01T01:50:42.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 0, 2024-04-01T01:50:42.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 20, 2024-04-01T01:50:42.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-04-01T01:50:42.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-04-01T01:50:42.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-04-01T01:50:42.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-04-01T01:50:42.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-04-01T01:50:42.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-04-01T01:50:42.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-04-01T01:50:42.632 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-04-01T01:50:42.633 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-04-01T01:50:42.633 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-04-01T01:50:42.633 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-04-01T01:50:42.633 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-04-01T01:50:42.633 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-04-01T01:50:42.633 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-04-01T01:50:42.633 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-04-01T01:50:42.633 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-04-01T01:50:42.633 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-04-01T01:50:42.633 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-04-01T01:50:42.633 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-04-01T01:50:42.633 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-04-01T01:50:42.633 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-04-01T01:50:42.633 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-04-01T01:50:42.634 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-04-01T01:50:42.634 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-04-01T01:50:42.634 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-04-01T01:50:42.634 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-04-01T01:50:42.634 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 95, 2024-04-01T01:50:42.634 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-04-01T01:50:42.634 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 6, 2024-04-01T01:50:42.634 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 10, 2024-04-01T01:50:42.634 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 16, 2024-04-01T01:50:42.634 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 6, 2024-04-01T01:50:42.634 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-04-01T01:50:42.634 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 16, 2024-04-01T01:50:42.634 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 0, 2024-04-01T01:50:42.634 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 1 2024-04-01T01:50:42.635 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.635 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-04-01T01:50:42.635 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 1, 2024-04-01T01:50:42.635 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-04-01T01:50:42.635 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-04-01T01:50:42.635 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-04-01T01:50:42.635 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.635 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-04-01T01:50:42.635 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-04-01T01:50:42.635 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.635 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-04-01T01:50:42.635 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-04-01T01:50:42.635 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 0, 2024-04-01T01:50:42.636 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-04-01T01:50:42.636 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 0, 2024-04-01T01:50:42.636 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 0, 2024-04-01T01:50:42.636 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 0, 2024-04-01T01:50:42.636 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-04-01T01:50:42.636 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.636 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-01T01:50:42.637 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:42.637 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-04-01T01:50:42.637 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:42.637 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 1682, 2024-04-01T01:50:42.637 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 1007588, 2024-04-01T01:50:42.637 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:42.637 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 1682, 2024-04-01T01:50:42.637 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-01T01:50:42.638 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-01T01:50:42.638 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1682, 2024-04-01T01:50:42.638 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1007588, 2024-04-01T01:50:42.638 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:42.638 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.638 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.639 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.639 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:42.639 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.639 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-04-01T01:50:42.639 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:42.639 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-04-01T01:50:42.639 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:42.640 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-01T01:50:42.640 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-01T01:50:42.640 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:42.640 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:42.640 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 41, 2024-04-01T01:50:42.640 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 12054, 2024-04-01T01:50:42.640 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 41, 2024-04-01T01:50:42.641 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 12054, 2024-04-01T01:50:42.641 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:42.641 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.641 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.641 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.641 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:42.641 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.642 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-04-01T01:50:42.642 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:42.642 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-04-01T01:50:42.642 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:42.642 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-01T01:50:42.642 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-01T01:50:42.642 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:42.642 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:42.643 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 41, 2024-04-01T01:50:42.643 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 41, 2024-04-01T01:50:42.643 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 41, 2024-04-01T01:50:42.643 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 41, 2024-04-01T01:50:42.643 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:42.643 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.644 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.644 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.644 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:42.644 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.644 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-04-01T01:50:42.644 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:42.644 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-04-01T01:50:42.645 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:42.645 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-01T01:50:42.645 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-01T01:50:42.645 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:42.645 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:42.645 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-01T01:50:42.645 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-01T01:50:42.645 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-04-01T01:50:42.645 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-04-01T01:50:42.646 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:42.646 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.646 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.646 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.646 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:42.646 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:42.646 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x55c451ac0500": { 2024-04-01T01:50:42.646 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:42.646 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-04-01T01:50:42.646 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:42.646 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 5, 2024-04-01T01:50:42.646 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 1597, 2024-04-01T01:50:42.646 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:42.647 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 5, 2024-04-01T01:50:42.647 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-01T01:50:42.647 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-01T01:50:42.647 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 4, 2024-04-01T01:50:42.647 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1597, 2024-04-01T01:50:42.647 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:42.647 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:42.647 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:42.647 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:42.647 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:42.647 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:42.647 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-01T01:50:43.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:42 smithi189 ceph-mon[35753]: pgmap v876: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-01T01:50:43.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:42 smithi070 ceph-mon[32179]: pgmap v876: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-01T01:50:43.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:42 smithi081 ceph-mon[35142]: pgmap v876: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-01T01:50:43.423 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-01T01:50:43.423 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-04-01T01:50:43.424 DEBUG:tasks.check_counter:Getting stats from l 2024-04-01T01:50:43.424 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583 shell --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 -- ceph daemon mds.l perf dump 2024-04-01T01:50:43.739 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/mon.c/config 2024-04-01T01:50:44.998 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:44 smithi189 ceph-mon[35753]: pgmap v877: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-01T01:50:45.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:44 smithi070 ceph-mon[32179]: pgmap v877: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-01T01:50:45.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:44 smithi081 ceph-mon[35142]: pgmap v877: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-01T01:50:45.653 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-01T01:50:45.653 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-04-01T01:50:45.653 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 562, 2024-04-01T01:50:45.653 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 763, 2024-04-01T01:50:45.653 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 132288, 2024-04-01T01:50:45.653 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 340783, 2024-04-01T01:50:45.653 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 10, 2024-04-01T01:50:45.653 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 3, 2024-04-01T01:50:45.653 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.062893404, 2024-04-01T01:50:45.653 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.021926278, 2024-04-01T01:50:45.653 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.030995173, 2024-04-01T01:50:45.653 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.005377902, 2024-04-01T01:50:45.654 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:45.654 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 761, 2024-04-01T01:50:45.654 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.069672250, 2024-04-01T01:50:45.654 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000091553 2024-04-01T01:50:45.654 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.654 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:45.654 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1276, 2024-04-01T01:50:45.654 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000324099, 2024-04-01T01:50:45.654 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000253 2024-04-01T01:50:45.654 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.654 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-01T01:50:45.654 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 0 2024-04-01T01:50:45.654 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.654 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-04-01T01:50:45.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 405, 2024-04-01T01:50:45.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 391, 2024-04-01T01:50:45.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 856377, 2024-04-01T01:50:45.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 264480, 2024-04-01T01:50:45.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 14, 2024-04-01T01:50:45.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 4, 2024-04-01T01:50:45.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.050321684, 2024-04-01T01:50:45.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.017151634, 2024-04-01T01:50:45.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.030464720, 2024-04-01T01:50:45.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.007929770, 2024-04-01T01:50:45.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:45.655 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 391, 2024-04-01T01:50:45.655 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.036656261, 2024-04-01T01:50:45.655 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000093750 2024-04-01T01:50:45.655 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.655 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:45.656 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 30, 2024-04-01T01:50:45.656 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000005704, 2024-04-01T01:50:45.656 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000190 2024-04-01T01:50:45.656 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.656 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 851392, 2024-04-01T01:50:45.656 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 206096 2024-04-01T01:50:45.656 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.656 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-04-01T01:50:45.656 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 77, 2024-04-01T01:50:45.656 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 321, 2024-04-01T01:50:45.656 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 13421, 2024-04-01T01:50:45.656 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 1245809, 2024-04-01T01:50:45.656 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 66, 2024-04-01T01:50:45.656 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 8, 2024-04-01T01:50:45.656 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.036192145, 2024-04-01T01:50:45.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.010994896, 2024-04-01T01:50:45.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.022174157, 2024-04-01T01:50:45.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.000340318, 2024-04-01T01:50:45.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-01T01:50:45.657 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 319, 2024-04-01T01:50:45.657 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.030320531, 2024-04-01T01:50:45.657 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000095048 2024-04-01T01:50:45.657 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-01T01:50:45.657 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 63, 2024-04-01T01:50:45.657 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000009071, 2024-04-01T01:50:45.657 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000143 2024-04-01T01:50:45.657 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.657 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 1248, 2024-04-01T01:50:45.658 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 589824 2024-04-01T01:50:45.658 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.658 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-04-01T01:50:45.658 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-04-01T01:50:45.658 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-04-01T01:50:45.658 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.658 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-04-01T01:50:45.658 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-04-01T01:50:45.658 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-04-01T01:50:45.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 50, 2024-04-01T01:50:45.658 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.001372581, 2024-04-01T01:50:45.658 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000027451 2024-04-01T01:50:45.658 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:45.658 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.659 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-04-01T01:50:45.659 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-04-01T01:50:45.659 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-04-01T01:50:45.659 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2, 2024-04-01T01:50:45.659 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000006174, 2024-04-01T01:50:45.659 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000003087 2024-04-01T01:50:45.659 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:45.659 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.659 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-04-01T01:50:45.659 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 0, 2024-04-01T01:50:45.659 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 0, 2024-04-01T01:50:45.659 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-04-01T01:50:45.659 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.659 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.660 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.660 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.660 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-04-01T01:50:45.660 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 0, 2024-04-01T01:50:45.660 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-04-01T01:50:45.660 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-04-01T01:50:45.660 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 11, 2024-04-01T01:50:45.660 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-04-01T01:50:45.660 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-04-01T01:50:45.660 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 10, 2024-04-01T01:50:45.660 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 0, 2024-04-01T01:50:45.660 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 10, 2024-04-01T01:50:45.660 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 0, 2024-04-01T01:50:45.660 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 10, 2024-04-01T01:50:45.660 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 0, 2024-04-01T01:50:45.661 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 0, 2024-04-01T01:50:45.661 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 0, 2024-04-01T01:50:45.661 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 1, 2024-04-01T01:50:45.661 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 0, 2024-04-01T01:50:45.661 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 0, 2024-04-01T01:50:45.661 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 0, 2024-04-01T01:50:45.661 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 0, 2024-04-01T01:50:45.661 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-04-01T01:50:45.661 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-04-01T01:50:45.661 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 0, 2024-04-01T01:50:45.661 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-04-01T01:50:45.661 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-04-01T01:50:45.661 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 0, 2024-04-01T01:50:45.661 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 0, 2024-04-01T01:50:45.661 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 0, 2024-04-01T01:50:45.662 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 0, 2024-04-01T01:50:45.662 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-04-01T01:50:45.662 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 0, 2024-04-01T01:50:45.662 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-04-01T01:50:45.662 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 1022, 2024-04-01T01:50:45.662 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 1557107758, 2024-04-01T01:50:45.662 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 0, 2024-04-01T01:50:45.662 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 0, 2024-04-01T01:50:45.662 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-04-01T01:50:45.662 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-04-01T01:50:45.662 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-04-01T01:50:45.662 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-04-01T01:50:45.662 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-04-01T01:50:45.663 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-04-01T01:50:45.663 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 0, 2024-04-01T01:50:45.663 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-04-01T01:50:45.663 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 0, 2024-04-01T01:50:45.663 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 0, 2024-04-01T01:50:45.663 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-04-01T01:50:45.663 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-01T01:50:45.663 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-01T01:50:45.663 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 0, 2024-04-01T01:50:45.663 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 0, 2024-04-01T01:50:45.663 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 0, 2024-04-01T01:50:45.663 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 0 2024-04-01T01:50:45.663 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.663 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-04-01T01:50:45.663 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 0, 2024-04-01T01:50:45.664 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 0, 2024-04-01T01:50:45.664 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 0, 2024-04-01T01:50:45.664 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 1, 2024-04-01T01:50:45.664 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 0, 2024-04-01T01:50:45.664 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-04-01T01:50:45.664 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-04-01T01:50:45.664 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-04-01T01:50:45.664 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 0, 2024-04-01T01:50:45.664 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 0, 2024-04-01T01:50:45.664 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-04-01T01:50:45.664 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-04-01T01:50:45.664 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-04-01T01:50:45.664 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-04-01T01:50:45.665 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-04-01T01:50:45.665 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-04-01T01:50:45.665 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-04-01T01:50:45.665 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_path": 0, 2024-04-01T01:50:45.665 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_inode": 0, 2024-04-01T01:50:45.665 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-04-01T01:50:45.665 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 0, 2024-04-01T01:50:45.665 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-04-01T01:50:45.665 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-04-01T01:50:45.665 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-04-01T01:50:45.665 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-04-01T01:50:45.665 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.665 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-04-01T01:50:45.666 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-04-01T01:50:45.666 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 5, 2024-04-01T01:50:45.666 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-04-01T01:50:45.666 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-04-01T01:50:45.666 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 5, 2024-04-01T01:50:45.666 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-04-01T01:50:45.666 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-04-01T01:50:45.666 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 2, 2024-04-01T01:50:45.666 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-04-01T01:50:45.666 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-04-01T01:50:45.666 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 2, 2024-04-01T01:50:45.666 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 2, 2024-04-01T01:50:45.666 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-04-01T01:50:45.666 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-04-01T01:50:45.667 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 4194304, 2024-04-01T01:50:45.667 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 4195901, 2024-04-01T01:50:45.667 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 0, 2024-04-01T01:50:45.667 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-04-01T01:50:45.667 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 4, 2024-04-01T01:50:45.667 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.006361768, 2024-04-01T01:50:45.667 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001590442 2024-04-01T01:50:45.667 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.667 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 0 2024-04-01T01:50:45.667 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.667 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-04-01T01:50:45.667 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 13, 2024-04-01T01:50:45.667 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 13, 2024-04-01T01:50:45.667 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 0, 2024-04-01T01:50:45.667 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 11, 2024-04-01T01:50:45.668 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 12, 2024-04-01T01:50:45.668 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 1, 2024-04-01T01:50:45.668 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 10, 2024-04-01T01:50:45.668 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 10, 2024-04-01T01:50:45.669 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 0, 2024-04-01T01:50:45.669 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 0, 2024-04-01T01:50:45.669 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 0, 2024-04-01T01:50:45.669 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 0, 2024-04-01T01:50:45.669 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 52568, 2024-04-01T01:50:45.669 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 223612 2024-04-01T01:50:45.669 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.669 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-04-01T01:50:45.670 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 0, 2024-04-01T01:50:45.670 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 0, 2024-04-01T01:50:45.670 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 0, 2024-04-01T01:50:45.670 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 17, 2024-04-01T01:50:45.670 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 0, 2024-04-01T01:50:45.670 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-04-01T01:50:45.670 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.670 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.670 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.670 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.670 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-04-01T01:50:45.670 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.670 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.670 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.671 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.671 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-04-01T01:50:45.671 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.671 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.671 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.671 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.671 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-04-01T01:50:45.671 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.671 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.671 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.671 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.671 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-04-01T01:50:45.671 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.671 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.671 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.672 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.672 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-04-01T01:50:45.672 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.672 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.672 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.672 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.672 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-04-01T01:50:45.672 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.672 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.672 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.672 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.672 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-04-01T01:50:45.672 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.672 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.673 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.673 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.673 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-04-01T01:50:45.673 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.673 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.673 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.673 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.673 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-04-01T01:50:45.673 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.673 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.673 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.673 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.673 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-04-01T01:50:45.673 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.673 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.674 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.674 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.674 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-04-01T01:50:45.674 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.675 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.675 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.675 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.675 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-04-01T01:50:45.675 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.675 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.675 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.675 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.675 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-04-01T01:50:45.675 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.675 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.675 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.675 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.676 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-04-01T01:50:45.676 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.676 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.676 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.676 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.676 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-04-01T01:50:45.676 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.676 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.676 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.676 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.676 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-04-01T01:50:45.676 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.676 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.676 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.676 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.677 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-04-01T01:50:45.677 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.677 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.677 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.677 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.677 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-04-01T01:50:45.677 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.677 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.677 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.677 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.677 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-04-01T01:50:45.677 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.677 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.677 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.678 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.678 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-04-01T01:50:45.678 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.678 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.678 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.678 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.678 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-04-01T01:50:45.678 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.678 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.678 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.678 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.678 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-04-01T01:50:45.678 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.678 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.679 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.679 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-04-01T01:50:45.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.679 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.679 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.679 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-04-01T01:50:45.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.679 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.679 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.679 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-04-01T01:50:45.679 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.679 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.680 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.680 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.680 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-04-01T01:50:45.680 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.680 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.680 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.680 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.680 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-04-01T01:50:45.680 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.680 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.680 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.680 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.680 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-04-01T01:50:45.680 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.681 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.681 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.681 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.681 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-04-01T01:50:45.681 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-04-01T01:50:45.681 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-04-01T01:50:45.681 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.681 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.681 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.681 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:45.681 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.681 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-04-01T01:50:45.681 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 1, 2024-04-01T01:50:45.681 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 2, 2024-04-01T01:50:45.682 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 1, 2024-04-01T01:50:45.682 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 1, 2024-04-01T01:50:45.682 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-04-01T01:50:45.682 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 0, 2024-04-01T01:50:45.682 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 0, 2024-04-01T01:50:45.682 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 1029, 2024-04-01T01:50:45.682 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-04-01T01:50:45.682 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.682 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-04-01T01:50:45.682 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-04-01T01:50:45.682 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-04-01T01:50:45.682 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-04-01T01:50:45.682 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-04-01T01:50:45.682 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-04-01T01:50:45.683 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-04-01T01:50:45.683 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-01T01:50:45.683 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-04-01T01:50:45.683 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-01T01:50:45.683 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-04-01T01:50:45.683 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-04-01T01:50:45.683 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-04-01T01:50:45.683 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-01T01:50:45.683 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-04-01T01:50:45.683 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-04-01T01:50:45.683 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-04-01T01:50:45.683 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-04-01T01:50:45.683 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-04-01T01:50:45.683 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-01T01:50:45.684 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-04-01T01:50:45.684 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-01T01:50:45.684 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-04-01T01:50:45.684 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-04-01T01:50:45.684 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-04-01T01:50:45.684 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-04-01T01:50:45.684 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-04-01T01:50:45.684 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-01T01:50:45.684 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-04-01T01:50:45.684 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-04-01T01:50:45.684 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-04-01T01:50:45.684 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-04-01T01:50:45.684 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-04-01T01:50:45.684 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-04-01T01:50:45.685 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-04-01T01:50:45.685 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-04-01T01:50:45.685 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-04-01T01:50:45.685 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 240039, 2024-04-01T01:50:45.685 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 85, 2024-04-01T01:50:45.685 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 0, 2024-04-01T01:50:45.685 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 0, 2024-04-01T01:50:45.685 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-04-01T01:50:45.685 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-04-01T01:50:45.685 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-04-01T01:50:45.685 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-04-01T01:50:45.685 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-04-01T01:50:45.685 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-04-01T01:50:45.685 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 9080, 2024-04-01T01:50:45.686 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 174, 2024-04-01T01:50:45.686 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-04-01T01:50:45.686 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-04-01T01:50:45.686 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-04-01T01:50:45.686 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-04-01T01:50:45.686 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 50664, 2024-04-01T01:50:45.686 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 198, 2024-04-01T01:50:45.686 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-04-01T01:50:45.686 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-04-01T01:50:45.686 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-04-01T01:50:45.686 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-04-01T01:50:45.686 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.686 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-04-01T01:50:45.686 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-04-01T01:50:45.687 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-04-01T01:50:45.687 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 40, 2024-04-01T01:50:45.687 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 11899, 2024-04-01T01:50:45.687 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 33, 2024-04-01T01:50:45.687 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 40, 2024-04-01T01:50:45.687 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-04-01T01:50:45.687 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 40, 2024-04-01T01:50:45.687 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.217995689, 2024-04-01T01:50:45.687 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.005449892 2024-04-01T01:50:45.687 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.687 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-04-01T01:50:45.687 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-04-01T01:50:45.687 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 40, 2024-04-01T01:50:45.687 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 62 2024-04-01T01:50:45.687 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.688 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 40, 2024-04-01T01:50:45.688 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 0, 2024-04-01T01:50:45.688 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 40, 2024-04-01T01:50:45.688 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-04-01T01:50:45.688 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-04-01T01:50:45.688 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 0, 2024-04-01T01:50:45.688 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 10, 2024-04-01T01:50:45.688 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 0, 2024-04-01T01:50:45.688 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 4, 2024-04-01T01:50:45.688 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 7, 2024-04-01T01:50:45.688 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-04-01T01:50:45.688 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-04-01T01:50:45.688 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-04-01T01:50:45.688 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-04-01T01:50:45.689 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 6, 2024-04-01T01:50:45.689 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-04-01T01:50:45.689 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-04-01T01:50:45.689 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-04-01T01:50:45.689 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 0, 2024-04-01T01:50:45.689 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 20, 2024-04-01T01:50:45.689 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-04-01T01:50:45.689 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-04-01T01:50:45.689 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-04-01T01:50:45.689 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-04-01T01:50:45.689 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-04-01T01:50:45.689 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-04-01T01:50:45.689 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-04-01T01:50:45.689 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-04-01T01:50:45.690 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-04-01T01:50:45.690 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-04-01T01:50:45.690 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-04-01T01:50:45.690 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-04-01T01:50:45.690 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-04-01T01:50:45.690 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-04-01T01:50:45.690 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-04-01T01:50:45.690 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-04-01T01:50:45.690 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-04-01T01:50:45.690 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-04-01T01:50:45.690 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-04-01T01:50:45.690 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-04-01T01:50:45.690 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-04-01T01:50:45.691 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-04-01T01:50:45.691 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-04-01T01:50:45.691 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-04-01T01:50:45.691 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-04-01T01:50:45.691 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-04-01T01:50:45.691 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 95, 2024-04-01T01:50:45.691 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-04-01T01:50:45.691 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 6, 2024-04-01T01:50:45.691 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 9, 2024-04-01T01:50:45.691 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 14, 2024-04-01T01:50:45.691 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 5, 2024-04-01T01:50:45.691 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-04-01T01:50:45.691 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 14, 2024-04-01T01:50:45.691 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 0, 2024-04-01T01:50:45.691 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 1 2024-04-01T01:50:45.692 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.692 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-04-01T01:50:45.692 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 1, 2024-04-01T01:50:45.692 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-04-01T01:50:45.692 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-04-01T01:50:45.692 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-04-01T01:50:45.692 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.692 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-04-01T01:50:45.692 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-04-01T01:50:45.692 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.692 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-04-01T01:50:45.692 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-04-01T01:50:45.692 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 0, 2024-04-01T01:50:45.692 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-04-01T01:50:45.692 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 0, 2024-04-01T01:50:45.693 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 0, 2024-04-01T01:50:45.693 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 0, 2024-04-01T01:50:45.693 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-04-01T01:50:45.693 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.693 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-01T01:50:45.693 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:45.693 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-04-01T01:50:45.693 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:45.693 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 1040, 2024-04-01T01:50:45.693 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 886221, 2024-04-01T01:50:45.693 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:45.693 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 1040, 2024-04-01T01:50:45.693 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-01T01:50:45.693 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-01T01:50:45.693 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1040, 2024-04-01T01:50:45.694 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 886221, 2024-04-01T01:50:45.694 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:45.694 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.694 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.694 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.694 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:45.694 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.694 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-04-01T01:50:45.694 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:45.694 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-04-01T01:50:45.694 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:45.694 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-01T01:50:45.694 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-01T01:50:45.695 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:45.695 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:45.695 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 40, 2024-04-01T01:50:45.695 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 11899, 2024-04-01T01:50:45.695 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 40, 2024-04-01T01:50:45.695 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 11899, 2024-04-01T01:50:45.695 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:45.695 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.695 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.695 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.695 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:45.695 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.695 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-04-01T01:50:45.695 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:45.696 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-04-01T01:50:45.696 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:45.696 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-01T01:50:45.696 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-01T01:50:45.696 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:45.696 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:45.696 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 40, 2024-04-01T01:50:45.696 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 40, 2024-04-01T01:50:45.696 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 40, 2024-04-01T01:50:45.696 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 40, 2024-04-01T01:50:45.696 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:45.696 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.696 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.697 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.697 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:45.697 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.697 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-04-01T01:50:45.697 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:45.697 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-04-01T01:50:45.697 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:45.697 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-01T01:50:45.697 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-01T01:50:45.697 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:45.697 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-01T01:50:45.697 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-01T01:50:45.697 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-01T01:50:45.697 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-04-01T01:50:45.697 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-04-01T01:50:45.698 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:45.698 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.698 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.698 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.698 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:45.698 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-01T01:50:45.698 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x55ffa08e2500": { 2024-04-01T01:50:45.698 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-01T01:50:45.698 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-04-01T01:50:45.698 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-01T01:50:45.698 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 5, 2024-04-01T01:50:45.698 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 1597, 2024-04-01T01:50:45.698 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-01T01:50:45.698 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 5, 2024-04-01T01:50:45.699 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-01T01:50:45.699 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-01T01:50:45.699 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 4, 2024-04-01T01:50:45.699 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1597, 2024-04-01T01:50:45.699 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-01T01:50:45.699 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-01T01:50:45.699 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-01T01:50:45.699 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-01T01:50:45.699 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:45.699 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-01T01:50:45.699 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-01T01:50:46.454 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-04-01T01:50:46.455 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-04-01T01:50:46.455 ERROR:teuthology.run_tasks:Manager failed: check-counter Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/check_counter.py", line 125, in end raise RuntimeError("The following counters failed to be set " RuntimeError: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} 2024-04-01T01:50:46.456 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-01T01:50:46.467 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-01T01:50:46.467 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:50:46.467 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-01T01:50:46.501 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:50:46.501 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-01T01:50:46.560 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:50:46.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.712+0000 7f6c2702e640 1 -- 172.21.15.70:0/2323426923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c201022a0 msgr2=0x7f6c20108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:46.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.712+0000 7f6c2702e640 1 --2- 172.21.15.70:0/2323426923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c201022a0 0x7f6c20108b30 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f6c18009be0 tx=0x7f6c1802bef0 comp rx=0 tx=0).stop 2024-04-01T01:50:46.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.712+0000 7f6c2702e640 1 -- 172.21.15.70:0/2323426923 shutdown_connections 2024-04-01T01:50:46.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.712+0000 7f6c2702e640 1 --2- 172.21.15.70:0/2323426923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c201022a0 0x7f6c20108b30 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:46.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.712+0000 7f6c2702e640 1 --2- 172.21.15.70:0/2323426923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c201018e0 0x7f6c20101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:46.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.712+0000 7f6c2702e640 1 --2- 172.21.15.70:0/2323426923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c201006f0 0x7f6c20100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:46.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.712+0000 7f6c2702e640 1 -- 172.21.15.70:0/2323426923 >> 172.21.15.70:0/2323426923 conn(0x7f6c200fbc80 msgr2=0x7f6c200fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:46.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.712+0000 7f6c2702e640 1 -- 172.21.15.70:0/2323426923 shutdown_connections 2024-04-01T01:50:46.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.712+0000 7f6c2702e640 1 -- 172.21.15.70:0/2323426923 wait complete. 2024-04-01T01:50:46.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.713+0000 7f6c2702e640 1 Processor -- start 2024-04-01T01:50:46.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.713+0000 7f6c2702e640 1 -- start start 2024-04-01T01:50:46.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.713+0000 7f6c2702e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c201006f0 0x7f6c20197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:46.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.713+0000 7f6c2702e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c201018e0 0x7f6c20197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:46.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.713+0000 7f6c2702e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c201022a0 0x7f6c2019eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:46.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.713+0000 7f6c2702e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2010c190 con 0x7f6c201018e0 2024-04-01T01:50:46.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.713+0000 7f6c2702e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2010c010 con 0x7f6c201022a0 2024-04-01T01:50:46.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.713+0000 7f6c2702e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2010c310 con 0x7f6c201006f0 2024-04-01T01:50:46.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.713+0000 7f6c17fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c201018e0 0x7f6c20197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:46.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.713+0000 7f6c17fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c201018e0 0x7f6c20197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44332/0 (socket says 172.21.15.70:44332) 2024-04-01T01:50:46.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.713+0000 7f6c24dca640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c201006f0 0x7f6c20197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:46.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.713+0000 7f6c255cb640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c201022a0 0x7f6c2019eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:46.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.713+0000 7f6c17fff640 1 -- 172.21.15.70:0/3137027797 learned_addr learned my addr 172.21.15.70:0/3137027797 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:46.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.714+0000 7f6c17fff640 1 -- 172.21.15.70:0/3137027797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c201006f0 msgr2=0x7f6c20197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:46.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.714+0000 7f6c17fff640 1 --2- 172.21.15.70:0/3137027797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c201006f0 0x7f6c20197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:46.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.714+0000 7f6c17fff640 1 -- 172.21.15.70:0/3137027797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c201022a0 msgr2=0x7f6c2019eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:46.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.714+0000 7f6c17fff640 1 --2- 172.21.15.70:0/3137027797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c201022a0 0x7f6c2019eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:46.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.714+0000 7f6c17fff640 1 -- 172.21.15.70:0/3137027797 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c2019f520 con 0x7f6c201018e0 2024-04-01T01:50:46.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.714+0000 7f6c255cb640 1 --2- 172.21.15.70:0/3137027797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c201022a0 0x7f6c2019eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:46.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.714+0000 7f6c17fff640 1 --2- 172.21.15.70:0/3137027797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c201018e0 0x7f6c20197e40 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000ea60 tx=0x7f6c1000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:46.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.714+0000 7f6c15ffb640 1 -- 172.21.15.70:0/3137027797 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1000cde0 con 0x7f6c201018e0 2024-04-01T01:50:46.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.714+0000 7f6c2702e640 1 -- 172.21.15.70:0/3137027797 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c2019f810 con 0x7f6c201018e0 2024-04-01T01:50:46.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.714+0000 7f6c15ffb640 1 -- 172.21.15.70:0/3137027797 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c100035f0 con 0x7f6c201018e0 2024-04-01T01:50:46.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.714+0000 7f6c15ffb640 1 -- 172.21.15.70:0/3137027797 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10010690 con 0x7f6c201018e0 2024-04-01T01:50:46.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.714+0000 7f6c2702e640 1 -- 172.21.15.70:0/3137027797 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c2019fd60 con 0x7f6c201018e0 2024-04-01T01:50:46.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.715+0000 7f6c2702e640 1 -- 172.21.15.70:0/3137027797 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c201a0020 con 0x7f6c201018e0 2024-04-01T01:50:46.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.716+0000 7f6c15ffb640 1 -- 172.21.15.70:0/3137027797 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f6c100037a0 con 0x7f6c201018e0 2024-04-01T01:50:46.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.717+0000 7f6c15ffb640 1 --2- 172.21.15.70:0/3137027797 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c080a24a0 0x7f6c080a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:46.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.717+0000 7f6c24dca640 1 --2- 172.21.15.70:0/3137027797 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c080a24a0 0x7f6c080a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:46.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.717+0000 7f6c15ffb640 1 -- 172.21.15.70:0/3137027797 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f6c100ced60 con 0x7f6c201018e0 2024-04-01T01:50:46.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.717+0000 7f6c24dca640 1 --2- 172.21.15.70:0/3137027797 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c080a24a0 0x7f6c080a4960 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c005fd0 tx=0x7f6c1c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:46.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.721+0000 7f6c15ffb640 1 -- 172.21.15.70:0/3137027797 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6c1009bb50 con 0x7f6c201018e0 2024-04-01T01:50:46.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.896+0000 7f6c2702e640 1 -- 172.21.15.70:0/3137027797 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c201a01f0 con 0x7f6c201018e0 2024-04-01T01:50:46.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.898+0000 7f6c15ffb640 1 -- 172.21.15.70:0/3137027797 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24781 (secure 0 0 0) 0x7f6c1009b450 con 0x7f6c201018e0 2024-04-01T01:50:46.899 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:50:46.899 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":95,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:49:12.151965+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:50:46.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.900+0000 7f6c2702e640 1 -- 172.21.15.70:0/3137027797 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c080a24a0 msgr2=0x7f6c080a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:46.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.900+0000 7f6c2702e640 1 --2- 172.21.15.70:0/3137027797 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c080a24a0 0x7f6c080a4960 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c005fd0 tx=0x7f6c1c00b390 comp rx=0 tx=0).stop 2024-04-01T01:50:46.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.900+0000 7f6c2702e640 1 -- 172.21.15.70:0/3137027797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c201018e0 msgr2=0x7f6c20197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:46.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.900+0000 7f6c2702e640 1 --2- 172.21.15.70:0/3137027797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c201018e0 0x7f6c20197e40 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000ea60 tx=0x7f6c1000ef20 comp rx=0 tx=0).stop 2024-04-01T01:50:46.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.900+0000 7f6c2702e640 1 -- 172.21.15.70:0/3137027797 shutdown_connections 2024-04-01T01:50:46.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.900+0000 7f6c2702e640 1 --2- 172.21.15.70:0/3137027797 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c201022a0 0x7f6c2019eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:46.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.900+0000 7f6c2702e640 1 --2- 172.21.15.70:0/3137027797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c201018e0 0x7f6c20197e40 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:46.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.900+0000 7f6c2702e640 1 --2- 172.21.15.70:0/3137027797 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c080a24a0 0x7f6c080a4960 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:46.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.900+0000 7f6c2702e640 1 --2- 172.21.15.70:0/3137027797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c201006f0 0x7f6c20197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:46.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.900+0000 7f6c2702e640 1 -- 172.21.15.70:0/3137027797 >> 172.21.15.70:0/3137027797 conn(0x7f6c200fbc80 msgr2=0x7f6c200fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:46.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.900+0000 7f6c2702e640 1 -- 172.21.15.70:0/3137027797 shutdown_connections 2024-04-01T01:50:46.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:46.900+0000 7f6c2702e640 1 -- 172.21.15.70:0/3137027797 wait complete. 2024-04-01T01:50:46.912 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-01T01:26:42.809779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-01T01:31:13.612369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-01T01:31:14.910162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-01T01:31:18.783179+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-01T01:50:46.913 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-01T01:50:47.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.017+0000 7f5f95255640 1 -- 172.21.15.70:0/3106813220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 msgr2=0x7f5f90108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.017+0000 7f5f95255640 1 --2- 172.21.15.70:0/3106813220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 0x7f5f90108a90 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f5f88009ca0 tx=0x7f5f8802c0f0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.017+0000 7f5f95255640 1 -- 172.21.15.70:0/3106813220 shutdown_connections 2024-04-01T01:50:47.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.017+0000 7f5f95255640 1 --2- 172.21.15.70:0/3106813220 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 0x7f5f90108a90 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.017+0000 7f5f95255640 1 --2- 172.21.15.70:0/3106813220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f90101840 0x7f5f90101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.017+0000 7f5f95255640 1 --2- 172.21.15.70:0/3106813220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f90100650 0x7f5f90100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.017+0000 7f5f95255640 1 -- 172.21.15.70:0/3106813220 >> 172.21.15.70:0/3106813220 conn(0x7f5f900fbbc0 msgr2=0x7f5f900fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:47.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.017+0000 7f5f95255640 1 -- 172.21.15.70:0/3106813220 shutdown_connections 2024-04-01T01:50:47.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.017+0000 7f5f95255640 1 -- 172.21.15.70:0/3106813220 wait complete. 2024-04-01T01:50:47.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.018+0000 7f5f95255640 1 Processor -- start 2024-04-01T01:50:47.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.018+0000 7f5f95255640 1 -- start start 2024-04-01T01:50:47.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.018+0000 7f5f95255640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f90100650 0x7f5f90197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:47.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.018+0000 7f5f95255640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f90101840 0x7f5f90197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:47.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.018+0000 7f5f95255640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 0x7f5f9019ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:47.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.018+0000 7f5f95255640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9010c190 con 0x7f5f90102200 2024-04-01T01:50:47.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.018+0000 7f5f95255640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9010c010 con 0x7f5f90100650 2024-04-01T01:50:47.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.018+0000 7f5f95255640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9010c310 con 0x7f5f90101840 2024-04-01T01:50:47.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.018+0000 7f5f8f59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 0x7f5f9019ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:47.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.018+0000 7f5f8f59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 0x7f5f9019ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44348/0 (socket says 172.21.15.70:44348) 2024-04-01T01:50:47.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.018+0000 7f5f8f59e640 1 -- 172.21.15.70:0/2261399232 learned_addr learned my addr 172.21.15.70:0/2261399232 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:47.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.018+0000 7f5f8e59c640 1 --2- 172.21.15.70:0/2261399232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f90101840 0x7f5f90197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:47.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.018+0000 7f5f8ed9d640 1 --2- 172.21.15.70:0/2261399232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f90100650 0x7f5f90197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:47.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.019+0000 7f5f8f59e640 1 -- 172.21.15.70:0/2261399232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f90101840 msgr2=0x7f5f90197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.019+0000 7f5f8f59e640 1 --2- 172.21.15.70:0/2261399232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f90101840 0x7f5f90197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.019+0000 7f5f8f59e640 1 -- 172.21.15.70:0/2261399232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f90100650 msgr2=0x7f5f90197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.019+0000 7f5f8f59e640 1 --2- 172.21.15.70:0/2261399232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f90100650 0x7f5f90197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.019+0000 7f5f8f59e640 1 -- 172.21.15.70:0/2261399232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f9019f570 con 0x7f5f90102200 2024-04-01T01:50:47.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.019+0000 7f5f8f59e640 1 --2- 172.21.15.70:0/2261399232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 0x7f5f9019ef00 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f5f8802c700 tx=0x7f5f880328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:47.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.019+0000 7f5f77fff640 1 -- 172.21.15.70:0/2261399232 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f88041020 con 0x7f5f90102200 2024-04-01T01:50:47.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.019+0000 7f5f77fff640 1 -- 172.21.15.70:0/2261399232 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f88003e20 con 0x7f5f90102200 2024-04-01T01:50:47.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.019+0000 7f5f95255640 1 -- 172.21.15.70:0/2261399232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f9019f800 con 0x7f5f90102200 2024-04-01T01:50:47.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.019+0000 7f5f77fff640 1 -- 172.21.15.70:0/2261399232 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f88039040 con 0x7f5f90102200 2024-04-01T01:50:47.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.019+0000 7f5f95255640 1 -- 172.21.15.70:0/2261399232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f9019fcf0 con 0x7f5f90102200 2024-04-01T01:50:47.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.020+0000 7f5f95255640 1 -- 172.21.15.70:0/2261399232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f9019ff50 con 0x7f5f90102200 2024-04-01T01:50:47.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.021+0000 7f5f77fff640 1 -- 172.21.15.70:0/2261399232 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f5f88044020 con 0x7f5f90102200 2024-04-01T01:50:47.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.022+0000 7f5f77fff640 1 --2- 172.21.15.70:0/2261399232 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f780a24a0 0x7f5f780a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:47.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.022+0000 7f5f8ed9d640 1 --2- 172.21.15.70:0/2261399232 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f780a24a0 0x7f5f780a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:47.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.022+0000 7f5f77fff640 1 -- 172.21.15.70:0/2261399232 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f5f880f1060 con 0x7f5f90102200 2024-04-01T01:50:47.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.022+0000 7f5f8ed9d640 1 --2- 172.21.15.70:0/2261399232 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f780a24a0 0x7f5f780a4960 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f5f901016a0 tx=0x7f5f84005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:47.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.025+0000 7f5f77fff640 1 -- 172.21.15.70:0/2261399232 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5f8804b020 con 0x7f5f90102200 2024-04-01T01:50:47.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.199+0000 7f5f95255640 1 -- 172.21.15.70:0/2261399232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5f901a02e0 con 0x7f5f90102200 2024-04-01T01:50:47.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.199+0000 7f5f77fff640 1 -- 172.21.15.70:0/2261399232 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f5f880bde50 con 0x7f5f90102200 2024-04-01T01:50:47.200 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-01T01:50:47.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.201+0000 7f5f95255640 1 -- 172.21.15.70:0/2261399232 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f780a24a0 msgr2=0x7f5f780a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.201+0000 7f5f95255640 1 --2- 172.21.15.70:0/2261399232 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f780a24a0 0x7f5f780a4960 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f5f901016a0 tx=0x7f5f84005f90 comp rx=0 tx=0).stop 2024-04-01T01:50:47.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.201+0000 7f5f95255640 1 -- 172.21.15.70:0/2261399232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 msgr2=0x7f5f9019ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.201+0000 7f5f95255640 1 --2- 172.21.15.70:0/2261399232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 0x7f5f9019ef00 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f5f8802c700 tx=0x7f5f880328a0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.201+0000 7f5f95255640 1 -- 172.21.15.70:0/2261399232 shutdown_connections 2024-04-01T01:50:47.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.201+0000 7f5f95255640 1 --2- 172.21.15.70:0/2261399232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f90102200 0x7f5f9019ef00 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.201+0000 7f5f95255640 1 --2- 172.21.15.70:0/2261399232 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f5f780a24a0 0x7f5f780a4960 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.201+0000 7f5f95255640 1 --2- 172.21.15.70:0/2261399232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f90101840 0x7f5f90197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.201+0000 7f5f95255640 1 --2- 172.21.15.70:0/2261399232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f90100650 0x7f5f90197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.201+0000 7f5f95255640 1 -- 172.21.15.70:0/2261399232 >> 172.21.15.70:0/2261399232 conn(0x7f5f900fbbc0 msgr2=0x7f5f900688b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:47.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.201+0000 7f5f95255640 1 -- 172.21.15.70:0/2261399232 shutdown_connections 2024-04-01T01:50:47.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.201+0000 7f5f95255640 1 -- 172.21.15.70:0/2261399232 wait complete. 2024-04-01T01:50:47.212 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:46 smithi070 ceph-mon[32179]: pgmap v878: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-01T01:50:47.212 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:46 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3137027797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:50:47.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-01T01:50:47.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:46 smithi189 ceph-mon[35753]: pgmap v878: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-01T01:50:47.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:46 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3137027797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:50:47.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.317+0000 7f010293e640 1 -- 172.21.15.70:0/1018748899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00fc0ffa40 msgr2=0x7f00fc0ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.317+0000 7f010293e640 1 --2- 172.21.15.70:0/1018748899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00fc0ffa40 0x7f00fc0ffe60 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f00ec009930 tx=0x7f00ec02bc30 comp rx=0 tx=0).stop 2024-04-01T01:50:47.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.317+0000 7f010293e640 1 -- 172.21.15.70:0/1018748899 shutdown_connections 2024-04-01T01:50:47.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.317+0000 7f010293e640 1 --2- 172.21.15.70:0/1018748899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc1033d0 0x7f00fc109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.317+0000 7f010293e640 1 --2- 172.21.15.70:0/1018748899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00fc102a10 0x7f00fc102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.317+0000 7f010293e640 1 --2- 172.21.15.70:0/1018748899 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00fc0ffa40 0x7f00fc0ffe60 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.317+0000 7f010293e640 1 -- 172.21.15.70:0/1018748899 >> 172.21.15.70:0/1018748899 conn(0x7f00fc0fb350 msgr2=0x7f00fc0fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:47.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.317+0000 7f010293e640 1 -- 172.21.15.70:0/1018748899 shutdown_connections 2024-04-01T01:50:47.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.317+0000 7f010293e640 1 -- 172.21.15.70:0/1018748899 wait complete. 2024-04-01T01:50:47.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.317+0000 7f010293e640 1 Processor -- start 2024-04-01T01:50:47.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f010293e640 1 -- start start 2024-04-01T01:50:47.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f010293e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00fc0ffa40 0x7f00fc1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:47.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f010293e640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00fc102a10 0x7f00fc197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:47.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f010293e640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc1033d0 0x7f00fc19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:47.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f010293e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f00fc10d2c0 con 0x7f00fc0ffa40 2024-04-01T01:50:47.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f010293e640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f00fc10d140 con 0x7f00fc102a10 2024-04-01T01:50:47.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f010293e640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f00fc10d440 con 0x7f00fc1033d0 2024-04-01T01:50:47.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f00fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00fc0ffa40 0x7f00fc1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:47.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f00fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00fc0ffa40 0x7f00fc1979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44370/0 (socket says 172.21.15.70:44370) 2024-04-01T01:50:47.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f00fbfff640 1 -- 172.21.15.70:0/3764693067 learned_addr learned my addr 172.21.15.70:0/3764693067 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:47.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f00fb7fe640 1 --2- 172.21.15.70:0/3764693067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00fc102a10 0x7f00fc197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:47.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f0100edb640 1 --2- 172.21.15.70:0/3764693067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc1033d0 0x7f00fc19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:47.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f00fbfff640 1 -- 172.21.15.70:0/3764693067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc1033d0 msgr2=0x7f00fc19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f00fbfff640 1 --2- 172.21.15.70:0/3764693067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc1033d0 0x7f00fc19ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f00fbfff640 1 -- 172.21.15.70:0/3764693067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00fc102a10 msgr2=0x7f00fc197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f00fbfff640 1 --2- 172.21.15.70:0/3764693067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00fc102a10 0x7f00fc197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f00fbfff640 1 -- 172.21.15.70:0/3764693067 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00fc19f5f0 con 0x7f00fc0ffa40 2024-04-01T01:50:47.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.318+0000 7f00fb7fe640 1 --2- 172.21.15.70:0/3764693067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00fc102a10 0x7f00fc197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:47.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.319+0000 7f00fbfff640 1 --2- 172.21.15.70:0/3764693067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00fc0ffa40 0x7f00fc1979d0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f00ec000c00 tx=0x7f00ec005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:47.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.319+0000 7f00f97fa640 1 -- 172.21.15.70:0/3764693067 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ec043020 con 0x7f00fc0ffa40 2024-04-01T01:50:47.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.319+0000 7f00f97fa640 1 -- 172.21.15.70:0/3764693067 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00ec0043d0 con 0x7f00fc0ffa40 2024-04-01T01:50:47.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.319+0000 7f010293e640 1 -- 172.21.15.70:0/3764693067 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00fc19f880 con 0x7f00fc0ffa40 2024-04-01T01:50:47.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.319+0000 7f00f97fa640 1 -- 172.21.15.70:0/3764693067 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ec003b50 con 0x7f00fc0ffa40 2024-04-01T01:50:47.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.319+0000 7f010293e640 1 -- 172.21.15.70:0/3764693067 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00fc19fdd0 con 0x7f00fc0ffa40 2024-04-01T01:50:47.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.320+0000 7f010293e640 1 -- 172.21.15.70:0/3764693067 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00fc108c30 con 0x7f00fc0ffa40 2024-04-01T01:50:47.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.321+0000 7f00f97fa640 1 -- 172.21.15.70:0/3764693067 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f00ec046020 con 0x7f00fc0ffa40 2024-04-01T01:50:47.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.321+0000 7f00f97fa640 1 --2- 172.21.15.70:0/3764693067 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00e40a24a0 0x7f00e40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:47.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.321+0000 7f00fb7fe640 1 --2- 172.21.15.70:0/3764693067 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00e40a24a0 0x7f00e40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:47.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.322+0000 7f00f97fa640 1 -- 172.21.15.70:0/3764693067 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f00ec0f0c20 con 0x7f00fc0ffa40 2024-04-01T01:50:47.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.322+0000 7f00fb7fe640 1 --2- 172.21.15.70:0/3764693067 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00e40a24a0 0x7f00e40a4960 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f00fc198ee0 tx=0x7f00f0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:47.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.326+0000 7f00f97fa640 1 -- 172.21.15.70:0/3764693067 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f00ec04d020 con 0x7f00fc0ffa40 2024-04-01T01:50:47.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:46 smithi081 ceph-mon[35142]: pgmap v878: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-01T01:50:47.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:46 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3137027797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:50:47.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.499+0000 7f010293e640 1 -- 172.21.15.70:0/3764693067 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f00fc062350 con 0x7f00fc0ffa40 2024-04-01T01:50:47.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.499+0000 7f00f97fa640 1 -- 172.21.15.70:0/3764693067 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f00ec0bda10 con 0x7f00fc0ffa40 2024-04-01T01:50:47.500 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:50:47.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.501+0000 7f010293e640 1 -- 172.21.15.70:0/3764693067 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00e40a24a0 msgr2=0x7f00e40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.501+0000 7f010293e640 1 --2- 172.21.15.70:0/3764693067 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00e40a24a0 0x7f00e40a4960 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f00fc198ee0 tx=0x7f00f0005f90 comp rx=0 tx=0).stop 2024-04-01T01:50:47.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.501+0000 7f010293e640 1 -- 172.21.15.70:0/3764693067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00fc0ffa40 msgr2=0x7f00fc1979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.501+0000 7f010293e640 1 --2- 172.21.15.70:0/3764693067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00fc0ffa40 0x7f00fc1979d0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f00ec000c00 tx=0x7f00ec005e70 comp rx=0 tx=0).stop 2024-04-01T01:50:47.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.501+0000 7f010293e640 1 -- 172.21.15.70:0/3764693067 shutdown_connections 2024-04-01T01:50:47.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.501+0000 7f010293e640 1 --2- 172.21.15.70:0/3764693067 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f00e40a24a0 0x7f00e40a4960 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.501+0000 7f010293e640 1 --2- 172.21.15.70:0/3764693067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f00fc1033d0 0x7f00fc19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.501+0000 7f010293e640 1 --2- 172.21.15.70:0/3764693067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00fc102a10 0x7f00fc197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.501+0000 7f010293e640 1 --2- 172.21.15.70:0/3764693067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f00fc0ffa40 0x7f00fc1979d0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.501+0000 7f010293e640 1 -- 172.21.15.70:0/3764693067 >> 172.21.15.70:0/3764693067 conn(0x7f00fc0fb350 msgr2=0x7f00fc105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:47.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.501+0000 7f010293e640 1 -- 172.21.15.70:0/3764693067 shutdown_connections 2024-04-01T01:50:47.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.501+0000 7f010293e640 1 -- 172.21.15.70:0/3764693067 wait complete. 2024-04-01T01:50:47.514 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-01T01:50:47.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 -- 172.21.15.70:0/1963793031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c90101840 msgr2=0x7f6c90101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 --2- 172.21.15.70:0/1963793031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c90101840 0x7f6c90101cc0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f6c78009900 tx=0x7f6c7802bc30 comp rx=0 tx=0).stop 2024-04-01T01:50:47.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 -- 172.21.15.70:0/1963793031 shutdown_connections 2024-04-01T01:50:47.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 --2- 172.21.15.70:0/1963793031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c90102200 0x7f6c90108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 --2- 172.21.15.70:0/1963793031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c90101840 0x7f6c90101cc0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 --2- 172.21.15.70:0/1963793031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c90100650 0x7f6c90100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 -- 172.21.15.70:0/1963793031 >> 172.21.15.70:0/1963793031 conn(0x7f6c900fbbe0 msgr2=0x7f6c900fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:47.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 -- 172.21.15.70:0/1963793031 shutdown_connections 2024-04-01T01:50:47.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 -- 172.21.15.70:0/1963793031 wait complete. 2024-04-01T01:50:47.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 Processor -- start 2024-04-01T01:50:47.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 -- start start 2024-04-01T01:50:47.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c90100650 0x7f6c90197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:47.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c90101840 0x7f6c90197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:47.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c90102200 0x7f6c9019ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:47.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9010c190 con 0x7f6c90101840 2024-04-01T01:50:47.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9010c010 con 0x7f6c90100650 2024-04-01T01:50:47.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.618+0000 7f6c954f4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9010c310 con 0x7f6c90102200 2024-04-01T01:50:47.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c87fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c90101840 0x7f6c90197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:47.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c87fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c90101840 0x7f6c90197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44390/0 (socket says 172.21.15.70:44390) 2024-04-01T01:50:47.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c87fff640 1 -- 172.21.15.70:0/1339769143 learned_addr learned my addr 172.21.15.70:0/1339769143 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:47.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c8effd640 1 --2- 172.21.15.70:0/1339769143 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c90100650 0x7f6c90197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:47.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c8f7fe640 1 --2- 172.21.15.70:0/1339769143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c90102200 0x7f6c9019ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:47.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c87fff640 1 -- 172.21.15.70:0/1339769143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c90102200 msgr2=0x7f6c9019ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c87fff640 1 --2- 172.21.15.70:0/1339769143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c90102200 0x7f6c9019ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c87fff640 1 -- 172.21.15.70:0/1339769143 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c90100650 msgr2=0x7f6c90197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c87fff640 1 --2- 172.21.15.70:0/1339769143 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c90100650 0x7f6c90197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c87fff640 1 -- 172.21.15.70:0/1339769143 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9019f570 con 0x7f6c90101840 2024-04-01T01:50:47.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c8effd640 1 --2- 172.21.15.70:0/1339769143 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c90100650 0x7f6c90197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:47.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c87fff640 1 --2- 172.21.15.70:0/1339769143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c90101840 0x7f6c90197e90 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f6c78006010 tx=0x7f6c78005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:47.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c8cff9640 1 -- 172.21.15.70:0/1339769143 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c78043020 con 0x7f6c90101840 2024-04-01T01:50:47.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c954f4640 1 -- 172.21.15.70:0/1339769143 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c9019f800 con 0x7f6c90101840 2024-04-01T01:50:47.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c8cff9640 1 -- 172.21.15.70:0/1339769143 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c780043d0 con 0x7f6c90101840 2024-04-01T01:50:47.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c8cff9640 1 -- 172.21.15.70:0/1339769143 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c78003b80 con 0x7f6c90101840 2024-04-01T01:50:47.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.619+0000 7f6c954f4640 1 -- 172.21.15.70:0/1339769143 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c9019fd50 con 0x7f6c90101840 2024-04-01T01:50:47.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.620+0000 7f6c954f4640 1 -- 172.21.15.70:0/1339769143 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c9019ff50 con 0x7f6c90101840 2024-04-01T01:50:47.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.621+0000 7f6c8cff9640 1 -- 172.21.15.70:0/1339769143 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f6c78046020 con 0x7f6c90101840 2024-04-01T01:50:47.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.622+0000 7f6c8cff9640 1 --2- 172.21.15.70:0/1339769143 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c7c0a24a0 0x7f6c7c0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:47.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.622+0000 7f6c8effd640 1 --2- 172.21.15.70:0/1339769143 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c7c0a24a0 0x7f6c7c0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:47.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.622+0000 7f6c8cff9640 1 -- 172.21.15.70:0/1339769143 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f6c780f0ca0 con 0x7f6c90101840 2024-04-01T01:50:47.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.622+0000 7f6c8effd640 1 --2- 172.21.15.70:0/1339769143 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c7c0a24a0 0x7f6c7c0a4960 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6c901016a0 tx=0x7f6c80005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:47.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.626+0000 7f6c8cff9640 1 -- 172.21.15.70:0/1339769143 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6c7804d020 con 0x7f6c90101840 2024-04-01T01:50:47.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.801+0000 7f6c954f4640 1 -- 172.21.15.70:0/1339769143 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6c901a0120 con 0x7f6c90101840 2024-04-01T01:50:47.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.802+0000 7f6c8cff9640 1 -- 172.21.15.70:0/1339769143 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f6c780bda90 con 0x7f6c90101840 2024-04-01T01:50:47.802 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-01T01:50:47.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.804+0000 7f6c954f4640 1 -- 172.21.15.70:0/1339769143 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c7c0a24a0 msgr2=0x7f6c7c0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.804+0000 7f6c954f4640 1 --2- 172.21.15.70:0/1339769143 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c7c0a24a0 0x7f6c7c0a4960 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6c901016a0 tx=0x7f6c80005f90 comp rx=0 tx=0).stop 2024-04-01T01:50:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.804+0000 7f6c954f4640 1 -- 172.21.15.70:0/1339769143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c90101840 msgr2=0x7f6c90197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.804+0000 7f6c954f4640 1 --2- 172.21.15.70:0/1339769143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c90101840 0x7f6c90197e90 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f6c78006010 tx=0x7f6c78005e40 comp rx=0 tx=0).stop 2024-04-01T01:50:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.804+0000 7f6c954f4640 1 -- 172.21.15.70:0/1339769143 shutdown_connections 2024-04-01T01:50:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.804+0000 7f6c954f4640 1 --2- 172.21.15.70:0/1339769143 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f6c7c0a24a0 0x7f6c7c0a4960 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.804+0000 7f6c954f4640 1 --2- 172.21.15.70:0/1339769143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c90102200 0x7f6c9019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.804+0000 7f6c954f4640 1 --2- 172.21.15.70:0/1339769143 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c90101840 0x7f6c90197e90 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.804+0000 7f6c954f4640 1 --2- 172.21.15.70:0/1339769143 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c90100650 0x7f6c90197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.804+0000 7f6c954f4640 1 -- 172.21.15.70:0/1339769143 >> 172.21.15.70:0/1339769143 conn(0x7f6c900fbbe0 msgr2=0x7f6c900fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.804+0000 7f6c954f4640 1 -- 172.21.15.70:0/1339769143 shutdown_connections 2024-04-01T01:50:47.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.804+0000 7f6c954f4640 1 -- 172.21.15.70:0/1339769143 wait complete. 2024-04-01T01:50:47.816 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-01T01:50:47.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.919+0000 7fa65e276640 1 -- 172.21.15.70:0/670209702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa658107490 msgr2=0x7fa658109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.919+0000 7fa65e276640 1 --2- 172.21.15.70:0/670209702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa658107490 0x7fa658109880 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fa650009be0 tx=0x7fa65002bef0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.919+0000 7fa65e276640 1 -- 172.21.15.70:0/670209702 shutdown_connections 2024-04-01T01:50:47.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.919+0000 7fa65e276640 1 --2- 172.21.15.70:0/670209702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa658107490 0x7fa658109880 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.919+0000 7fa65e276640 1 --2- 172.21.15.70:0/670209702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa658104b60 0x7fa658106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.919+0000 7fa65e276640 1 --2- 172.21.15.70:0/670209702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa658102230 0x7fa658104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.919+0000 7fa65e276640 1 -- 172.21.15.70:0/670209702 >> 172.21.15.70:0/670209702 conn(0x7fa6580fbbe0 msgr2=0x7fa6580fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:47.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.919+0000 7fa65e276640 1 -- 172.21.15.70:0/670209702 shutdown_connections 2024-04-01T01:50:47.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.919+0000 7fa65e276640 1 -- 172.21.15.70:0/670209702 wait complete. 2024-04-01T01:50:47.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.920+0000 7fa65e276640 1 Processor -- start 2024-04-01T01:50:47.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.920+0000 7fa65e276640 1 -- start start 2024-04-01T01:50:47.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.920+0000 7fa65e276640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa658102230 0x7fa658197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:47.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.920+0000 7fa65e276640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa658104b60 0x7fa658197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:47.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.920+0000 7fa65e276640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa658107490 0x7fa65819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:47.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.920+0000 7fa65e276640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa65810d2a0 con 0x7fa658104b60 2024-04-01T01:50:47.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.920+0000 7fa65e276640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa65810d120 con 0x7fa658107490 2024-04-01T01:50:47.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.920+0000 7fa65e276640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa65810d420 con 0x7fa658102230 2024-04-01T01:50:47.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.920+0000 7fa6577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa658104b60 0x7fa658197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:47.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.920+0000 7fa657fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa658102230 0x7fa658197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:47.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.920+0000 7fa657fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa658102230 0x7fa658197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46182/0 (socket says 172.21.15.70:46182) 2024-04-01T01:50:47.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.920+0000 7fa6577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa658104b60 0x7fa658197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44406/0 (socket says 172.21.15.70:44406) 2024-04-01T01:50:47.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.920+0000 7fa657fff640 1 -- 172.21.15.70:0/514263705 learned_addr learned my addr 172.21.15.70:0/514263705 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:47.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.920+0000 7fa65c813640 1 --2- 172.21.15.70:0/514263705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa658107490 0x7fa65819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:47.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.921+0000 7fa657fff640 1 -- 172.21.15.70:0/514263705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa658107490 msgr2=0x7fa65819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.921+0000 7fa657fff640 1 --2- 172.21.15.70:0/514263705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa658107490 0x7fa65819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.921+0000 7fa657fff640 1 -- 172.21.15.70:0/514263705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa658104b60 msgr2=0x7fa658197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:47.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.921+0000 7fa657fff640 1 --2- 172.21.15.70:0/514263705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa658104b60 0x7fa658197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:47.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.921+0000 7fa657fff640 1 -- 172.21.15.70:0/514263705 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65819f590 con 0x7fa658102230 2024-04-01T01:50:47.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.921+0000 7fa6577fe640 1 --2- 172.21.15.70:0/514263705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa658104b60 0x7fa658197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:47.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.921+0000 7fa657fff640 1 --2- 172.21.15.70:0/514263705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa658102230 0x7fa658197970 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fa64800e9f0 tx=0x7fa64800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:47.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.921+0000 7fa6557fa640 1 -- 172.21.15.70:0/514263705 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64800ccc0 con 0x7fa658102230 2024-04-01T01:50:47.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.921+0000 7fa6557fa640 1 -- 172.21.15.70:0/514263705 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa64800ce60 con 0x7fa658102230 2024-04-01T01:50:47.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.921+0000 7fa65e276640 1 -- 172.21.15.70:0/514263705 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa65819f880 con 0x7fa658102230 2024-04-01T01:50:47.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.921+0000 7fa6557fa640 1 -- 172.21.15.70:0/514263705 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa648010430 con 0x7fa658102230 2024-04-01T01:50:47.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.921+0000 7fa65e276640 1 -- 172.21.15.70:0/514263705 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa65819fd80 con 0x7fa658102230 2024-04-01T01:50:47.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.922+0000 7fa65e276640 1 -- 172.21.15.70:0/514263705 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6580fd6d0 con 0x7fa658102230 2024-04-01T01:50:47.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.923+0000 7fa6557fa640 1 -- 172.21.15.70:0/514263705 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fa648005000 con 0x7fa658102230 2024-04-01T01:50:47.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.924+0000 7fa6557fa640 1 --2- 172.21.15.70:0/514263705 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa6400a24a0 0x7fa6400a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:47.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.924+0000 7fa6577fe640 1 --2- 172.21.15.70:0/514263705 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa6400a24a0 0x7fa6400a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:47.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.924+0000 7fa6557fa640 1 -- 172.21.15.70:0/514263705 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fa648099800 con 0x7fa658102230 2024-04-01T01:50:47.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.924+0000 7fa6577fe640 1 --2- 172.21.15.70:0/514263705 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa6400a24a0 0x7fa6400a4960 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa658198e80 tx=0x7fa64c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:47.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:47.929+0000 7fa6557fa640 1 -- 172.21.15.70:0/514263705 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa648053280 con 0x7fa658102230 2024-04-01T01:50:48.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.104+0000 7fa65e276640 1 -- 172.21.15.70:0/514263705 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa65802d300 con 0x7fa658102230 2024-04-01T01:50:48.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.104+0000 7fa6557fa640 1 -- 172.21.15.70:0/514263705 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7fa64809be60 con 0x7fa658102230 2024-04-01T01:50:48.105 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:50:48.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.106+0000 7fa65e276640 1 -- 172.21.15.70:0/514263705 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa6400a24a0 msgr2=0x7fa6400a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:48.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.106+0000 7fa65e276640 1 --2- 172.21.15.70:0/514263705 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa6400a24a0 0x7fa6400a4960 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa658198e80 tx=0x7fa64c009260 comp rx=0 tx=0).stop 2024-04-01T01:50:48.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.106+0000 7fa65e276640 1 -- 172.21.15.70:0/514263705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa658102230 msgr2=0x7fa658197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:48.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.106+0000 7fa65e276640 1 --2- 172.21.15.70:0/514263705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa658102230 0x7fa658197970 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fa64800e9f0 tx=0x7fa64800edf0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.106+0000 7fa65e276640 1 -- 172.21.15.70:0/514263705 shutdown_connections 2024-04-01T01:50:48.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.106+0000 7fa65e276640 1 --2- 172.21.15.70:0/514263705 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa658107490 0x7fa65819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.106+0000 7fa65e276640 1 --2- 172.21.15.70:0/514263705 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa658104b60 0x7fa658197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.106+0000 7fa65e276640 1 --2- 172.21.15.70:0/514263705 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa6400a24a0 0x7fa6400a4960 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.106+0000 7fa65e276640 1 --2- 172.21.15.70:0/514263705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa658102230 0x7fa658197970 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.106+0000 7fa65e276640 1 -- 172.21.15.70:0/514263705 >> 172.21.15.70:0/514263705 conn(0x7fa6580fbbe0 msgr2=0x7fa658102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:48.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.106+0000 7fa65e276640 1 -- 172.21.15.70:0/514263705 shutdown_connections 2024-04-01T01:50:48.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.106+0000 7fa65e276640 1 -- 172.21.15.70:0/514263705 wait complete. 2024-04-01T01:50:48.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:47 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2261399232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:50:48.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:47 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3764693067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:50:48.117 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:47 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/1339769143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:50:48.118 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-01T01:50:48.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 -- 172.21.15.70:0/1534125775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9a4101770 msgr2=0x7fd9a4101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:48.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 --2- 172.21.15.70:0/1534125775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9a4101770 0x7fd9a4101b90 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fd99c009900 tx=0x7fd99c02bc30 comp rx=0 tx=0).stop 2024-04-01T01:50:48.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 -- 172.21.15.70:0/1534125775 shutdown_connections 2024-04-01T01:50:48.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 --2- 172.21.15.70:0/1534125775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9a4103320 0x7fd9a4109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 --2- 172.21.15.70:0/1534125775 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9a4102960 0x7fd9a4102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 --2- 172.21.15.70:0/1534125775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9a4101770 0x7fd9a4101b90 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 -- 172.21.15.70:0/1534125775 >> 172.21.15.70:0/1534125775 conn(0x7fd9a40fccc0 msgr2=0x7fd9a40ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:48.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 -- 172.21.15.70:0/1534125775 shutdown_connections 2024-04-01T01:50:48.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 -- 172.21.15.70:0/1534125775 wait complete. 2024-04-01T01:50:48.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 Processor -- start 2024-04-01T01:50:48.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 -- start start 2024-04-01T01:50:48.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9a4101770 0x7fd9a41979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:48.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9a4102960 0x7fd9a4197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:48.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9a4103320 0x7fd9a419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:48.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a410d2f0 con 0x7fd9a4103320 2024-04-01T01:50:48.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a410d170 con 0x7fd9a4101770 2024-04-01T01:50:48.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.221+0000 7fd9aab99640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a410d470 con 0x7fd9a4102960 2024-04-01T01:50:48.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.222+0000 7fd9a9136640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9a4103320 0x7fd9a419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:48.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.222+0000 7fd9a8935640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9a4101770 0x7fd9a41979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:48.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.222+0000 7fd9a3fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9a4102960 0x7fd9a4197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:48.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.222+0000 7fd9a9136640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9a4103320 0x7fd9a419ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44420/0 (socket says 172.21.15.70:44420) 2024-04-01T01:50:48.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.222+0000 7fd9a9136640 1 -- 172.21.15.70:0/2137539529 learned_addr learned my addr 172.21.15.70:0/2137539529 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:48.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.222+0000 7fd9a9136640 1 -- 172.21.15.70:0/2137539529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9a4102960 msgr2=0x7fd9a4197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:48.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.222+0000 7fd9a9136640 1 --2- 172.21.15.70:0/2137539529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9a4102960 0x7fd9a4197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.222+0000 7fd9a9136640 1 -- 172.21.15.70:0/2137539529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9a4101770 msgr2=0x7fd9a41979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:48.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.222+0000 7fd9a9136640 1 --2- 172.21.15.70:0/2137539529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9a4101770 0x7fd9a41979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.222+0000 7fd9a9136640 1 -- 172.21.15.70:0/2137539529 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a419f5e0 con 0x7fd9a4103320 2024-04-01T01:50:48.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.222+0000 7fd9a9136640 1 --2- 172.21.15.70:0/2137539529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9a4103320 0x7fd9a419ef70 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00e970 tx=0x7fd98c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:48.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.222+0000 7fd9a1ffb640 1 -- 172.21.15.70:0/2137539529 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c00ccc0 con 0x7fd9a4103320 2024-04-01T01:50:48.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.222+0000 7fd9a1ffb640 1 -- 172.21.15.70:0/2137539529 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd98c00ce60 con 0x7fd9a4103320 2024-04-01T01:50:48.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.222+0000 7fd9aab99640 1 -- 172.21.15.70:0/2137539529 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a419f8d0 con 0x7fd9a4103320 2024-04-01T01:50:48.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.222+0000 7fd9a1ffb640 1 -- 172.21.15.70:0/2137539529 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c010430 con 0x7fd9a4103320 2024-04-01T01:50:48.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.222+0000 7fd9aab99640 1 -- 172.21.15.70:0/2137539529 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a419fe20 con 0x7fd9a4103320 2024-04-01T01:50:48.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.223+0000 7fd9aab99640 1 -- 172.21.15.70:0/2137539529 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9a4108bb0 con 0x7fd9a4103320 2024-04-01T01:50:48.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.224+0000 7fd9a1ffb640 1 -- 172.21.15.70:0/2137539529 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fd98c004f90 con 0x7fd9a4103320 2024-04-01T01:50:48.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.225+0000 7fd9a1ffb640 1 --2- 172.21.15.70:0/2137539529 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9900a24a0 0x7fd9900a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:48.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.225+0000 7fd9a1ffb640 1 -- 172.21.15.70:0/2137539529 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fd98c0ceea0 con 0x7fd9a4103320 2024-04-01T01:50:48.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.225+0000 7fd9a8935640 1 --2- 172.21.15.70:0/2137539529 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9900a24a0 0x7fd9900a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:48.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.225+0000 7fd9a8935640 1 --2- 172.21.15.70:0/2137539529 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9900a24a0 0x7fd9900a4960 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd99c02c240 tx=0x7fd99c02c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:48.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.229+0000 7fd9a1ffb640 1 -- 172.21.15.70:0/2137539529 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd98c09bc90 con 0x7fd9a4103320 2024-04-01T01:50:48.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:47 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2261399232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:50:48.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:47 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3764693067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:50:48.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:47 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/1339769143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:50:48.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:47 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2261399232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:50:48.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:47 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3764693067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:50:48.329 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:47 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/1339769143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:50:48.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.416+0000 7fd9aab99640 1 -- 172.21.15.70:0/2137539529 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd9a402d170 con 0x7fd9a4103320 2024-04-01T01:50:48.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.416+0000 7fd9a1ffb640 1 -- 172.21.15.70:0/2137539529 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v95) ==== 81+0+590 (secure 0 0 0) 0x7fd98c09b590 con 0x7fd9a4103320 2024-04-01T01:50:48.417 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/950171900 2024-04-02T01:24:18.477610+0000 2024-04-01T01:50:48.417 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/2902831307 2024-04-02T01:24:18.477610+0000 2024-04-01T01:50:48.417 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6801/2836712828 2024-04-02T01:24:18.477610+0000 2024-04-01T01:50:48.417 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6800/2836712828 2024-04-02T01:24:18.477610+0000 2024-04-01T01:50:48.418 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/602656012 2024-04-02T01:23:52.091015+0000 2024-04-01T01:50:48.418 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6800/2121290979 2024-04-02T01:23:52.091015+0000 2024-04-01T01:50:48.418 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6801/2121290979 2024-04-02T01:23:52.091015+0000 2024-04-01T01:50:48.418 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/3734182136 2024-04-02T01:23:52.091015+0000 2024-04-01T01:50:48.418 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/2899206193 2024-04-02T01:23:52.091015+0000 2024-04-01T01:50:48.418 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/3948868469 2024-04-02T01:24:18.477610+0000 2024-04-01T01:50:48.418 INFO:teuthology.orchestra.run.smithi070.stderr:listed 10 entries 2024-04-01T01:50:48.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.418+0000 7fd9aab99640 1 -- 172.21.15.70:0/2137539529 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9900a24a0 msgr2=0x7fd9900a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:48.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.418+0000 7fd9aab99640 1 --2- 172.21.15.70:0/2137539529 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9900a24a0 0x7fd9900a4960 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd99c02c240 tx=0x7fd99c02c7e0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.418+0000 7fd9aab99640 1 -- 172.21.15.70:0/2137539529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9a4103320 msgr2=0x7fd9a419ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:48.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.418+0000 7fd9aab99640 1 --2- 172.21.15.70:0/2137539529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9a4103320 0x7fd9a419ef70 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00e970 tx=0x7fd98c00ed70 comp rx=0 tx=0).stop 2024-04-01T01:50:48.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.418+0000 7fd9aab99640 1 -- 172.21.15.70:0/2137539529 shutdown_connections 2024-04-01T01:50:48.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.418+0000 7fd9aab99640 1 --2- 172.21.15.70:0/2137539529 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9a4103320 0x7fd9a419ef70 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.418+0000 7fd9aab99640 1 --2- 172.21.15.70:0/2137539529 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd9900a24a0 0x7fd9900a4960 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.418+0000 7fd9aab99640 1 --2- 172.21.15.70:0/2137539529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9a4102960 0x7fd9a4197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.418+0000 7fd9aab99640 1 --2- 172.21.15.70:0/2137539529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9a4101770 0x7fd9a41979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.418+0000 7fd9aab99640 1 -- 172.21.15.70:0/2137539529 >> 172.21.15.70:0/2137539529 conn(0x7fd9a40fccc0 msgr2=0x7fd9a4105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:48.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.418+0000 7fd9aab99640 1 -- 172.21.15.70:0/2137539529 shutdown_connections 2024-04-01T01:50:48.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.418+0000 7fd9aab99640 1 -- 172.21.15.70:0/2137539529 wait complete. 2024-04-01T01:50:48.430 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-01T01:50:48.430 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-01T01:50:48.446 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-01T01:50:48.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.587+0000 7fd171e7c640 1 -- 172.21.15.70:0/3120110641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16c1022a0 msgr2=0x7fd16c108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:48.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.587+0000 7fd171e7c640 1 --2- 172.21.15.70:0/3120110641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16c1022a0 0x7fd16c108b30 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fd164009ca0 tx=0x7fd16402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.588+0000 7fd171e7c640 1 -- 172.21.15.70:0/3120110641 shutdown_connections 2024-04-01T01:50:48.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.588+0000 7fd171e7c640 1 --2- 172.21.15.70:0/3120110641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16c1022a0 0x7fd16c108b30 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.588+0000 7fd171e7c640 1 --2- 172.21.15.70:0/3120110641 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd16c1018e0 0x7fd16c101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.588+0000 7fd171e7c640 1 --2- 172.21.15.70:0/3120110641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd16c1006f0 0x7fd16c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.588+0000 7fd171e7c640 1 -- 172.21.15.70:0/3120110641 >> 172.21.15.70:0/3120110641 conn(0x7fd16c0fbc80 msgr2=0x7fd16c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:48.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.588+0000 7fd171e7c640 1 -- 172.21.15.70:0/3120110641 shutdown_connections 2024-04-01T01:50:48.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.588+0000 7fd171e7c640 1 -- 172.21.15.70:0/3120110641 wait complete. 2024-04-01T01:50:48.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.588+0000 7fd171e7c640 1 Processor -- start 2024-04-01T01:50:48.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.588+0000 7fd171e7c640 1 -- start start 2024-04-01T01:50:48.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.588+0000 7fd171e7c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16c1006f0 0x7fd16c197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:48.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.588+0000 7fd171e7c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd16c1018e0 0x7fd16c197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:48.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.588+0000 7fd171e7c640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd16c1022a0 0x7fd16c19eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:48.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd171e7c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c10c1b0 con 0x7fd16c1006f0 2024-04-01T01:50:48.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd171e7c640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c10c030 con 0x7fd16c1022a0 2024-04-01T01:50:48.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd171e7c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c10c330 con 0x7fd16c1018e0 2024-04-01T01:50:48.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd16b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16c1006f0 0x7fd16c197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:48.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd16b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16c1006f0 0x7fd16c197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44442/0 (socket says 172.21.15.70:44442) 2024-04-01T01:50:48.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd16b7fe640 1 -- 172.21.15.70:0/290100396 learned_addr learned my addr 172.21.15.70:0/290100396 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:48.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd16bfff640 1 --2- 172.21.15.70:0/290100396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd16c1022a0 0x7fd16c19eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:48.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd16affd640 1 --2- 172.21.15.70:0/290100396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd16c1018e0 0x7fd16c197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:48.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd16b7fe640 1 -- 172.21.15.70:0/290100396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd16c1018e0 msgr2=0x7fd16c197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:48.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd16b7fe640 1 --2- 172.21.15.70:0/290100396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd16c1018e0 0x7fd16c197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd16b7fe640 1 -- 172.21.15.70:0/290100396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd16c1022a0 msgr2=0x7fd16c19eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:48.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd16b7fe640 1 --2- 172.21.15.70:0/290100396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd16c1022a0 0x7fd16c19eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd16b7fe640 1 -- 172.21.15.70:0/290100396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16c19f560 con 0x7fd16c1006f0 2024-04-01T01:50:48.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd16b7fe640 1 --2- 172.21.15.70:0/290100396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16c1006f0 0x7fd16c197990 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00e9c0 tx=0x7fd15c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:48.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd168ff9640 1 -- 172.21.15.70:0/290100396 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c00cd80 con 0x7fd16c1006f0 2024-04-01T01:50:48.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd168ff9640 1 -- 172.21.15.70:0/290100396 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd15c0035f0 con 0x7fd16c1006f0 2024-04-01T01:50:48.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd171e7c640 1 -- 172.21.15.70:0/290100396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd16c19f850 con 0x7fd16c1006f0 2024-04-01T01:50:48.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd168ff9640 1 -- 172.21.15.70:0/290100396 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c010690 con 0x7fd16c1006f0 2024-04-01T01:50:48.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.589+0000 7fd171e7c640 1 -- 172.21.15.70:0/290100396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd16c19fda0 con 0x7fd16c1006f0 2024-04-01T01:50:48.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.590+0000 7fd171e7c640 1 -- 172.21.15.70:0/290100396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd16c1a0000 con 0x7fd16c1006f0 2024-04-01T01:50:48.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.591+0000 7fd168ff9640 1 -- 172.21.15.70:0/290100396 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fd15c0037a0 con 0x7fd16c1006f0 2024-04-01T01:50:48.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.592+0000 7fd168ff9640 1 --2- 172.21.15.70:0/290100396 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1540a2520 0x7fd1540a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:48.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.592+0000 7fd16affd640 1 --2- 172.21.15.70:0/290100396 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1540a2520 0x7fd1540a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:48.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.592+0000 7fd168ff9640 1 -- 172.21.15.70:0/290100396 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fd15c0cf100 con 0x7fd16c1006f0 2024-04-01T01:50:48.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.592+0000 7fd16affd640 1 --2- 172.21.15.70:0/290100396 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1540a2520 0x7fd1540a49e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd16c198e50 tx=0x7fd160005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:48.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.595+0000 7fd168ff9640 1 -- 172.21.15.70:0/290100396 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd15c09bef0 con 0x7fd16c1006f0 2024-04-01T01:50:48.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.774+0000 7fd171e7c640 1 -- 172.21.15.70:0/290100396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd16c1a01d0 con 0x7fd16c1006f0 2024-04-01T01:50:48.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.776+0000 7fd168ff9640 1 -- 172.21.15.70:0/290100396 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24781 (secure 0 0 0) 0x7fd15c09b7f0 con 0x7fd16c1006f0 2024-04-01T01:50:48.776 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-01T01:50:48.776 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":95,"fsid":"36577394-efc6-11ee-b647-cb9ed24678a4","created":"2024-04-01T01:23:20.696672+0000","modified":"2024-04-01T01:49:12.151965+0000","last_up_change":"2024-04-01T01:30:25.488576+0000","last_in_change":"2024-04-01T01:30:05.229248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-01T01:26:42.809779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-01T01:31:13.612369+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-01T01:31:14.910162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-01T01:31:18.783179+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a4e9041f-f0ad-4b4b-80f6-2726874d9231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6803","nonce":3697496043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6805","nonce":3697496043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6809","nonce":3697496043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":3697496043},{"type":"v1","addr":"172.21.15.70:6807","nonce":3697496043}]},"public_addr":"172.21.15.70:6803/3697496043","cluster_addr":"172.21.15.70:6805/3697496043","heartbeat_back_addr":"172.21.15.70:6809/3697496043","heartbeat_front_addr":"172.21.15.70:6807/3697496043","state":["exists","up"]},{"osd":1,"uuid":"9df24cdb-3f8d-427c-afd8-dd579612d782","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6801","nonce":4081087424}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6803","nonce":4081087424}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6807","nonce":4081087424}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4081087424},{"type":"v1","addr":"172.21.15.81:6805","nonce":4081087424}]},"public_addr":"172.21.15.81:6801/4081087424","cluster_addr":"172.21.15.81:6803/4081087424","heartbeat_back_addr":"172.21.15.81:6807/4081087424","heartbeat_front_addr":"172.21.15.81:6805/4081087424","state":["exists","up"]},{"osd":2,"uuid":"800721f9-d63a-4bea-8bbb-ae09c8c161be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6801","nonce":4191559460}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6803","nonce":4191559460}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6807","nonce":4191559460}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4191559460},{"type":"v1","addr":"172.21.15.189:6805","nonce":4191559460}]},"public_addr":"172.21.15.189:6801/4191559460","cluster_addr":"172.21.15.189:6803/4191559460","heartbeat_back_addr":"172.21.15.189:6807/4191559460","heartbeat_front_addr":"172.21.15.189:6805/4191559460","state":["exists","up"]},{"osd":3,"uuid":"835bbe5b-ecc9-4ebb-b084-9707dce28d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6811","nonce":627924617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6813","nonce":627924617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6817","nonce":627924617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":627924617},{"type":"v1","addr":"172.21.15.70:6815","nonce":627924617}]},"public_addr":"172.21.15.70:6811/627924617","cluster_addr":"172.21.15.70:6813/627924617","heartbeat_back_addr":"172.21.15.70:6817/627924617","heartbeat_front_addr":"172.21.15.70:6815/627924617","state":["exists","up"]},{"osd":4,"uuid":"ddf50f36-6795-440a-9890-2f20c63820fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6809","nonce":404027261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6811","nonce":404027261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6815","nonce":404027261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":404027261},{"type":"v1","addr":"172.21.15.81:6813","nonce":404027261}]},"public_addr":"172.21.15.81:6809/404027261","cluster_addr":"172.21.15.81:6811/404027261","heartbeat_back_addr":"172.21.15.81:6815/404027261","heartbeat_front_addr":"172.21.15.81:6813/404027261","state":["exists","up"]},{"osd":5,"uuid":"9cae9565-6f21-4686-a5e9-a8c08d5fe0bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6809","nonce":2305146377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6811","nonce":2305146377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6815","nonce":2305146377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":2305146377},{"type":"v1","addr":"172.21.15.189:6813","nonce":2305146377}]},"public_addr":"172.21.15.189:6809/2305146377","cluster_addr":"172.21.15.189:6811/2305146377","heartbeat_back_addr":"172.21.15.189:6815/2305146377","heartbeat_front_addr":"172.21.15.189:6813/2305146377","state":["exists","up"]},{"osd":6,"uuid":"0446322b-061f-4134-aa65-2f4639238ce5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6819","nonce":2689871673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6821","nonce":2689871673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6825","nonce":2689871673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2689871673},{"type":"v1","addr":"172.21.15.70:6823","nonce":2689871673}]},"public_addr":"172.21.15.70:6819/2689871673","cluster_addr":"172.21.15.70:6821/2689871673","heartbeat_back_addr":"172.21.15.70:6825/2689871673","heartbeat_front_addr":"172.21.15.70:6823/2689871673","state":["exists","up"]},{"osd":7,"uuid":"d654db8e-5fdc-47ff-99d9-db10263fa237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6817","nonce":2088069716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6819","nonce":2088069716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6823","nonce":2088069716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2088069716},{"type":"v1","addr":"172.21.15.81:6821","nonce":2088069716}]},"public_addr":"172.21.15.81:6817/2088069716","cluster_addr":"172.21.15.81:6819/2088069716","heartbeat_back_addr":"172.21.15.81:6823/2088069716","heartbeat_front_addr":"172.21.15.81:6821/2088069716","state":["exists","up"]},{"osd":8,"uuid":"073456b5-3f5d-491a-86a4-317161989697","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6817","nonce":2097527021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6819","nonce":2097527021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6823","nonce":2097527021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2097527021},{"type":"v1","addr":"172.21.15.189:6821","nonce":2097527021}]},"public_addr":"172.21.15.189:6817/2097527021","cluster_addr":"172.21.15.189:6819/2097527021","heartbeat_back_addr":"172.21.15.189:6823/2097527021","heartbeat_front_addr":"172.21.15.189:6821/2097527021","state":["exists","up"]},{"osd":9,"uuid":"c3e5db1b-9902-44a7-882f-1c4f1e0d7b9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6827","nonce":1192094091}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6829","nonce":1192094091}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6833","nonce":1192094091}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1192094091},{"type":"v1","addr":"172.21.15.70:6831","nonce":1192094091}]},"public_addr":"172.21.15.70:6827/1192094091","cluster_addr":"172.21.15.70:6829/1192094091","heartbeat_back_addr":"172.21.15.70:6833/1192094091","heartbeat_front_addr":"172.21.15.70:6831/1192094091","state":["exists","up"]},{"osd":10,"uuid":"f58e117c-8f4b-4ca4-bddc-9ae37024d996","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6825","nonce":1556495251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6827","nonce":1556495251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6831","nonce":1556495251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1556495251},{"type":"v1","addr":"172.21.15.81:6829","nonce":1556495251}]},"public_addr":"172.21.15.81:6825/1556495251","cluster_addr":"172.21.15.81:6827/1556495251","heartbeat_back_addr":"172.21.15.81:6831/1556495251","heartbeat_front_addr":"172.21.15.81:6829/1556495251","state":["exists","up"]},{"osd":11,"uuid":"365b3275-80d3-4888-8a4e-e00b571599ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6825","nonce":22739510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6827","nonce":22739510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6831","nonce":22739510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":22739510},{"type":"v1","addr":"172.21.15.189:6829","nonce":22739510}]},"public_addr":"172.21.15.189:6825/22739510","cluster_addr":"172.21.15.189:6827/22739510","heartbeat_back_addr":"172.21.15.189:6831/22739510","heartbeat_front_addr":"172.21.15.189:6829/22739510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:25:53.664820+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:17.419905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:26:40.660177+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:04.683001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:29.195890+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:27:53.242096+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:17.531865+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:28:42.323203+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:06.385713+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:32.201907+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:29:57.770604+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-01T01:30:23.216094+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.70:0/950171900":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/2902831307":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6801/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:6800/2836712828":"2024-04-02T01:24:18.477610+0000","172.21.15.70:0/602656012":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6800/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:6801/2121290979":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3734182136":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/2899206193":"2024-04-02T01:23:52.091015+0000","172.21.15.70:0/3948868469":"2024-04-02T01:24:18.477610+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-01T01:50:48.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.778+0000 7fd171e7c640 1 -- 172.21.15.70:0/290100396 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1540a2520 msgr2=0x7fd1540a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:48.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.778+0000 7fd171e7c640 1 --2- 172.21.15.70:0/290100396 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1540a2520 0x7fd1540a49e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd16c198e50 tx=0x7fd160005f90 comp rx=0 tx=0).stop 2024-04-01T01:50:48.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.778+0000 7fd171e7c640 1 -- 172.21.15.70:0/290100396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16c1006f0 msgr2=0x7fd16c197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:48.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.778+0000 7fd171e7c640 1 --2- 172.21.15.70:0/290100396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16c1006f0 0x7fd16c197990 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00e9c0 tx=0x7fd15c00ee80 comp rx=0 tx=0).stop 2024-04-01T01:50:48.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.778+0000 7fd171e7c640 1 -- 172.21.15.70:0/290100396 shutdown_connections 2024-04-01T01:50:48.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.778+0000 7fd171e7c640 1 --2- 172.21.15.70:0/290100396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd16c1022a0 0x7fd16c19eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.778+0000 7fd171e7c640 1 --2- 172.21.15.70:0/290100396 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fd1540a2520 0x7fd1540a49e0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.778+0000 7fd171e7c640 1 --2- 172.21.15.70:0/290100396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd16c1018e0 0x7fd16c197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.778+0000 7fd171e7c640 1 --2- 172.21.15.70:0/290100396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd16c1006f0 0x7fd16c197990 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.778+0000 7fd171e7c640 1 -- 172.21.15.70:0/290100396 >> 172.21.15.70:0/290100396 conn(0x7fd16c0fbc80 msgr2=0x7fd16c069d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:48.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.778+0000 7fd171e7c640 1 -- 172.21.15.70:0/290100396 shutdown_connections 2024-04-01T01:50:48.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.778+0000 7fd171e7c640 1 -- 172.21.15.70:0/290100396 wait complete. 2024-04-01T01:50:48.790 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-01T01:26:42.809779+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-01T01:31:13.612369+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-01T01:31:14.910162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-01T01:31:18.783179+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-01T01:50:48.790 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-01T01:50:48.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.894+0000 7fe11a4ed640 1 -- 172.21.15.70:0/2639508344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1141021d0 msgr2=0x7fe1141045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:48.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.894+0000 7fe11a4ed640 1 --2- 172.21.15.70:0/2639508344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1141021d0 0x7fe1141045c0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fe104009900 tx=0x7fe10402bc30 comp rx=0 tx=0).stop 2024-04-01T01:50:48.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.894+0000 7fe11a4ed640 1 -- 172.21.15.70:0/2639508344 shutdown_connections 2024-04-01T01:50:48.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.894+0000 7fe11a4ed640 1 --2- 172.21.15.70:0/2639508344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe114107430 0x7fe114109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.894+0000 7fe11a4ed640 1 --2- 172.21.15.70:0/2639508344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe114104b00 0x7fe114106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.894+0000 7fe11a4ed640 1 --2- 172.21.15.70:0/2639508344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1141021d0 0x7fe1141045c0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.894+0000 7fe11a4ed640 1 -- 172.21.15.70:0/2639508344 >> 172.21.15.70:0/2639508344 conn(0x7fe1140fbbc0 msgr2=0x7fe1140fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:48.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.894+0000 7fe11a4ed640 1 -- 172.21.15.70:0/2639508344 shutdown_connections 2024-04-01T01:50:48.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.894+0000 7fe11a4ed640 1 -- 172.21.15.70:0/2639508344 wait complete. 2024-04-01T01:50:48.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.895+0000 7fe11a4ed640 1 Processor -- start 2024-04-01T01:50:48.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.895+0000 7fe11a4ed640 1 -- start start 2024-04-01T01:50:48.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.895+0000 7fe11a4ed640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1141021d0 0x7fe1141979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:48.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.895+0000 7fe11a4ed640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe114104b00 0x7fe114197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:48.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.895+0000 7fe11a4ed640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe114107430 0x7fe11419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:48.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.895+0000 7fe11a4ed640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe11410d2d0 con 0x7fe114107430 2024-04-01T01:50:48.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.895+0000 7fe11a4ed640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe11410d150 con 0x7fe114104b00 2024-04-01T01:50:48.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.895+0000 7fe11a4ed640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe11410d450 con 0x7fe1141021d0 2024-04-01T01:50:48.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.895+0000 7fe113fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1141021d0 0x7fe1141979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:48.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.895+0000 7fe118a8a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe114107430 0x7fe11419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:48.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.895+0000 7fe113fff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1141021d0 0x7fe1141979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46250/0 (socket says 172.21.15.70:46250) 2024-04-01T01:50:48.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.895+0000 7fe113fff640 1 -- 172.21.15.70:0/2926260435 learned_addr learned my addr 172.21.15.70:0/2926260435 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:48.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.895+0000 7fe1137fe640 1 --2- 172.21.15.70:0/2926260435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe114104b00 0x7fe114197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:48.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.896+0000 7fe118a8a640 1 -- 172.21.15.70:0/2926260435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1141021d0 msgr2=0x7fe1141979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:48.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.896+0000 7fe118a8a640 1 --2- 172.21.15.70:0/2926260435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1141021d0 0x7fe1141979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.896+0000 7fe118a8a640 1 -- 172.21.15.70:0/2926260435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe114104b00 msgr2=0x7fe114197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:48.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.896+0000 7fe118a8a640 1 --2- 172.21.15.70:0/2926260435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe114104b00 0x7fe114197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:48.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.896+0000 7fe118a8a640 1 -- 172.21.15.70:0/2926260435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11419f5c0 con 0x7fe114107430 2024-04-01T01:50:48.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.896+0000 7fe113fff640 1 --2- 172.21.15.70:0/2926260435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1141021d0 0x7fe1141979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:48.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.896+0000 7fe1137fe640 1 --2- 172.21.15.70:0/2926260435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe114104b00 0x7fe114197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-01T01:50:48.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.896+0000 7fe118a8a640 1 --2- 172.21.15.70:0/2926260435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe114107430 0x7fe11419ef50 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fe10c009fd0 tx=0x7fe10c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:48.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.896+0000 7fe1117fa640 1 -- 172.21.15.70:0/2926260435 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c00ce20 con 0x7fe114107430 2024-04-01T01:50:48.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.896+0000 7fe1117fa640 1 -- 172.21.15.70:0/2926260435 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe10c005020 con 0x7fe114107430 2024-04-01T01:50:48.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.896+0000 7fe1117fa640 1 -- 172.21.15.70:0/2926260435 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c010690 con 0x7fe114107430 2024-04-01T01:50:48.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.896+0000 7fe11a4ed640 1 -- 172.21.15.70:0/2926260435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe11419f8b0 con 0x7fe114107430 2024-04-01T01:50:48.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.896+0000 7fe11a4ed640 1 -- 172.21.15.70:0/2926260435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe11419fdb0 con 0x7fe114107430 2024-04-01T01:50:48.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.897+0000 7fe11a4ed640 1 -- 172.21.15.70:0/2926260435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1140fd6b0 con 0x7fe114107430 2024-04-01T01:50:48.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.899+0000 7fe1117fa640 1 -- 172.21.15.70:0/2926260435 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fe10c0051d0 con 0x7fe114107430 2024-04-01T01:50:48.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.899+0000 7fe1117fa640 1 --2- 172.21.15.70:0/2926260435 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe0fc0a24a0 0x7fe0fc0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:48.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.900+0000 7fe113fff640 1 --2- 172.21.15.70:0/2926260435 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe0fc0a24a0 0x7fe0fc0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:48.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.900+0000 7fe1117fa640 1 -- 172.21.15.70:0/2926260435 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fe10c0cf210 con 0x7fe114107430 2024-04-01T01:50:48.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.900+0000 7fe113fff640 1 --2- 172.21.15.70:0/2926260435 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe0fc0a24a0 0x7fe0fc0a4960 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe1040096c0 tx=0x7fe10402c7e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:48.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:48.903+0000 7fe1117fa640 1 -- 172.21.15.70:0/2926260435 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe10c09bfa0 con 0x7fe114107430 2024-04-01T01:50:49.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.078+0000 7fe11a4ed640 1 -- 172.21.15.70:0/2926260435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe11402d170 con 0x7fe114107430 2024-04-01T01:50:49.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.078+0000 7fe1117fa640 1 -- 172.21.15.70:0/2926260435 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7fe10c09b8a0 con 0x7fe114107430 2024-04-01T01:50:49.079 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-01T01:50:49.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.080+0000 7fe11a4ed640 1 -- 172.21.15.70:0/2926260435 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe0fc0a24a0 msgr2=0x7fe0fc0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.080+0000 7fe11a4ed640 1 --2- 172.21.15.70:0/2926260435 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe0fc0a24a0 0x7fe0fc0a4960 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe1040096c0 tx=0x7fe10402c7e0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.080+0000 7fe11a4ed640 1 -- 172.21.15.70:0/2926260435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe114107430 msgr2=0x7fe11419ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.080+0000 7fe11a4ed640 1 --2- 172.21.15.70:0/2926260435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe114107430 0x7fe11419ef50 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fe10c009fd0 tx=0x7fe10c00ef10 comp rx=0 tx=0).stop 2024-04-01T01:50:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.080+0000 7fe11a4ed640 1 -- 172.21.15.70:0/2926260435 shutdown_connections 2024-04-01T01:50:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.080+0000 7fe11a4ed640 1 --2- 172.21.15.70:0/2926260435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe114107430 0x7fe11419ef50 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.080+0000 7fe11a4ed640 1 --2- 172.21.15.70:0/2926260435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe114104b00 0x7fe114197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.080+0000 7fe11a4ed640 1 --2- 172.21.15.70:0/2926260435 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fe0fc0a24a0 0x7fe0fc0a4960 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.080+0000 7fe11a4ed640 1 --2- 172.21.15.70:0/2926260435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1141021d0 0x7fe1141979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.080+0000 7fe11a4ed640 1 -- 172.21.15.70:0/2926260435 >> 172.21.15.70:0/2926260435 conn(0x7fe1140fbbc0 msgr2=0x7fe114102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.080+0000 7fe11a4ed640 1 -- 172.21.15.70:0/2926260435 shutdown_connections 2024-04-01T01:50:49.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.080+0000 7fe11a4ed640 1 -- 172.21.15.70:0/2926260435 wait complete. 2024-04-01T01:50:49.090 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:48 smithi070 ceph-mon[32179]: pgmap v879: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 11 op/s 2024-04-01T01:50:49.091 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:48 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/514263705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:50:49.091 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:48 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2137539529' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-01T01:50:49.091 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:48 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/290100396' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:50:49.092 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-01T01:50:49.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.197+0000 7f40fe840640 1 -- 172.21.15.70:0/3506897206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f8100670 msgr2=0x7f40f8100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.197+0000 7f40fe840640 1 --2- 172.21.15.70:0/3506897206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f8100670 0x7f40f8100a90 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f40e8009900 tx=0x7f40e802bc30 comp rx=0 tx=0).stop 2024-04-01T01:50:49.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.197+0000 7f40fe840640 1 -- 172.21.15.70:0/3506897206 shutdown_connections 2024-04-01T01:50:49.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.197+0000 7f40fe840640 1 --2- 172.21.15.70:0/3506897206 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40f8102220 0x7f40f8108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.197+0000 7f40fe840640 1 --2- 172.21.15.70:0/3506897206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40f8101860 0x7f40f8101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.197+0000 7f40fe840640 1 --2- 172.21.15.70:0/3506897206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f8100670 0x7f40f8100a90 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.197+0000 7f40fe840640 1 -- 172.21.15.70:0/3506897206 >> 172.21.15.70:0/3506897206 conn(0x7f40f80fbc00 msgr2=0x7f40f80fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:49.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.197+0000 7f40fe840640 1 -- 172.21.15.70:0/3506897206 shutdown_connections 2024-04-01T01:50:49.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.197+0000 7f40fe840640 1 -- 172.21.15.70:0/3506897206 wait complete. 2024-04-01T01:50:49.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40fe840640 1 Processor -- start 2024-04-01T01:50:49.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40fe840640 1 -- start start 2024-04-01T01:50:49.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40fe840640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f8100670 0x7f40f806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:49.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40fe840640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40f8101860 0x7f40f819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:49.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40fe840640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40f8102220 0x7f40f819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:49.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40fe840640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f40f810c1a0 con 0x7f40f8100670 2024-04-01T01:50:49.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40fe840640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f40f810c020 con 0x7f40f8102220 2024-04-01T01:50:49.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40fe840640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f40f810c320 con 0x7f40f8101860 2024-04-01T01:50:49.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40f7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f8100670 0x7f40f806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:49.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40f7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f8100670 0x7f40f806a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44488/0 (socket says 172.21.15.70:44488) 2024-04-01T01:50:49.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40f7fff640 1 -- 172.21.15.70:0/4102578490 learned_addr learned my addr 172.21.15.70:0/4102578490 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:49.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40f77fe640 1 --2- 172.21.15.70:0/4102578490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40f8101860 0x7f40f819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:49.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40fcddd640 1 --2- 172.21.15.70:0/4102578490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40f8102220 0x7f40f819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:49.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40f7fff640 1 -- 172.21.15.70:0/4102578490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40f8101860 msgr2=0x7f40f819c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40f7fff640 1 --2- 172.21.15.70:0/4102578490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40f8101860 0x7f40f819c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40f7fff640 1 -- 172.21.15.70:0/4102578490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40f8102220 msgr2=0x7f40f819ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40f7fff640 1 --2- 172.21.15.70:0/4102578490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40f8102220 0x7f40f819ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40f7fff640 1 -- 172.21.15.70:0/4102578490 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f819f670 con 0x7f40f8100670 2024-04-01T01:50:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.198+0000 7f40f7fff640 1 --2- 172.21.15.70:0/4102578490 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f8100670 0x7f40f806a890 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f40e8000c00 tx=0x7f40e8005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.199+0000 7f40f57fa640 1 -- 172.21.15.70:0/4102578490 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e8043020 con 0x7f40f8100670 2024-04-01T01:50:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.199+0000 7f40f57fa640 1 -- 172.21.15.70:0/4102578490 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f40e8004350 con 0x7f40f8100670 2024-04-01T01:50:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.199+0000 7f40fe840640 1 -- 172.21.15.70:0/4102578490 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40f819f8a0 con 0x7f40f8100670 2024-04-01T01:50:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.199+0000 7f40f57fa640 1 -- 172.21.15.70:0/4102578490 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e8003b80 con 0x7f40f8100670 2024-04-01T01:50:49.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.199+0000 7f40fe840640 1 -- 172.21.15.70:0/4102578490 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40f819fe40 con 0x7f40f8100670 2024-04-01T01:50:49.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.200+0000 7f40fe840640 1 -- 172.21.15.70:0/4102578490 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40f8107ab0 con 0x7f40f8100670 2024-04-01T01:50:49.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.201+0000 7f40f57fa640 1 -- 172.21.15.70:0/4102578490 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7f40e8046020 con 0x7f40f8100670 2024-04-01T01:50:49.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.201+0000 7f40f57fa640 1 --2- 172.21.15.70:0/4102578490 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f40e00a2520 0x7f40e00a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:49.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.201+0000 7f40f77fe640 1 --2- 172.21.15.70:0/4102578490 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f40e00a2520 0x7f40e00a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:49.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.201+0000 7f40f57fa640 1 -- 172.21.15.70:0/4102578490 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7f40e80f0b30 con 0x7f40f8100670 2024-04-01T01:50:49.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.202+0000 7f40f77fe640 1 --2- 172.21.15.70:0/4102578490 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f40e00a2520 0x7f40e00a49e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f40f806a600 tx=0x7f40ec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:49.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.206+0000 7f40f57fa640 1 -- 172.21.15.70:0/4102578490 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f40e804d020 con 0x7f40f8100670 2024-04-01T01:50:49.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:48 smithi189 ceph-mon[35753]: pgmap v879: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 11 op/s 2024-04-01T01:50:49.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:48 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/514263705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:50:49.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:48 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2137539529' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-01T01:50:49.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:48 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/290100396' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:50:49.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:48 smithi081 ceph-mon[35142]: pgmap v879: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 11 op/s 2024-04-01T01:50:49.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:48 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/514263705' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:50:49.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:48 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2137539529' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-01T01:50:49.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:48 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/290100396' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-01T01:50:49.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.380+0000 7f40fe840640 1 -- 172.21.15.70:0/4102578490 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f40f8062350 con 0x7f40f8100670 2024-04-01T01:50:49.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.380+0000 7f40f57fa640 1 -- 172.21.15.70:0/4102578490 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f40e80bd920 con 0x7f40f8100670 2024-04-01T01:50:49.381 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:50:49.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.382+0000 7f40fe840640 1 -- 172.21.15.70:0/4102578490 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f40e00a2520 msgr2=0x7f40e00a49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.382+0000 7f40fe840640 1 --2- 172.21.15.70:0/4102578490 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f40e00a2520 0x7f40e00a49e0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f40f806a600 tx=0x7f40ec005f90 comp rx=0 tx=0).stop 2024-04-01T01:50:49.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.382+0000 7f40fe840640 1 -- 172.21.15.70:0/4102578490 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f8100670 msgr2=0x7f40f806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.382+0000 7f40fe840640 1 --2- 172.21.15.70:0/4102578490 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f8100670 0x7f40f806a890 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f40e8000c00 tx=0x7f40e8005dc0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.382+0000 7f40fe840640 1 -- 172.21.15.70:0/4102578490 shutdown_connections 2024-04-01T01:50:49.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.382+0000 7f40fe840640 1 --2- 172.21.15.70:0/4102578490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40f8102220 0x7f40f819ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.382+0000 7f40fe840640 1 --2- 172.21.15.70:0/4102578490 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7f40e00a2520 0x7f40e00a49e0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.382+0000 7f40fe840640 1 --2- 172.21.15.70:0/4102578490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40f8101860 0x7f40f819c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.382+0000 7f40fe840640 1 --2- 172.21.15.70:0/4102578490 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f8100670 0x7f40f806a890 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.382+0000 7f40fe840640 1 -- 172.21.15.70:0/4102578490 >> 172.21.15.70:0/4102578490 conn(0x7f40f80fbc00 msgr2=0x7f40f8104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:49.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.382+0000 7f40fe840640 1 -- 172.21.15.70:0/4102578490 shutdown_connections 2024-04-01T01:50:49.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.382+0000 7f40fe840640 1 -- 172.21.15.70:0/4102578490 wait complete. 2024-04-01T01:50:49.394 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-01T01:50:49.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.498+0000 7fa0da3c9640 1 -- 172.21.15.70:0/1245383371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d4102200 msgr2=0x7fa0d4108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.498+0000 7fa0da3c9640 1 --2- 172.21.15.70:0/1245383371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d4102200 0x7fa0d4108a90 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc009ca0 tx=0x7fa0cc02c0f0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.498+0000 7fa0da3c9640 1 -- 172.21.15.70:0/1245383371 shutdown_connections 2024-04-01T01:50:49.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.498+0000 7fa0da3c9640 1 --2- 172.21.15.70:0/1245383371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d4102200 0x7fa0d4108a90 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.498+0000 7fa0da3c9640 1 --2- 172.21.15.70:0/1245383371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0d4101840 0x7fa0d4101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.498+0000 7fa0da3c9640 1 --2- 172.21.15.70:0/1245383371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d4100650 0x7fa0d4100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.498+0000 7fa0da3c9640 1 -- 172.21.15.70:0/1245383371 >> 172.21.15.70:0/1245383371 conn(0x7fa0d40fbbc0 msgr2=0x7fa0d40fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:49.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.498+0000 7fa0da3c9640 1 -- 172.21.15.70:0/1245383371 shutdown_connections 2024-04-01T01:50:49.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.498+0000 7fa0da3c9640 1 -- 172.21.15.70:0/1245383371 wait complete. 2024-04-01T01:50:49.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.499+0000 7fa0da3c9640 1 Processor -- start 2024-04-01T01:50:49.500 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.499+0000 7fa0da3c9640 1 -- start start 2024-04-01T01:50:49.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.499+0000 7fa0da3c9640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d4100650 0x7fa0d41979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:49.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.499+0000 7fa0da3c9640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0d4101840 0x7fa0d4197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:49.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.499+0000 7fa0da3c9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d4102200 0x7fa0d419ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:49.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.499+0000 7fa0da3c9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d410c190 con 0x7fa0d4102200 2024-04-01T01:50:49.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.499+0000 7fa0da3c9640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d410c010 con 0x7fa0d4100650 2024-04-01T01:50:49.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.499+0000 7fa0da3c9640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d410c310 con 0x7fa0d4101840 2024-04-01T01:50:49.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0d8966640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d4102200 0x7fa0d419ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:49.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d4100650 0x7fa0d41979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:49.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0d37fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0d4101840 0x7fa0d4197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:49.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0d3fff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d4100650 0x7fa0d41979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.70:49384/0 (socket says 172.21.15.70:49384) 2024-04-01T01:50:49.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0d8966640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d4102200 0x7fa0d419ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44502/0 (socket says 172.21.15.70:44502) 2024-04-01T01:50:49.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0d3fff640 1 -- 172.21.15.70:0/3270112915 learned_addr learned my addr 172.21.15.70:0/3270112915 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:49.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0d37fe640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0d4101840 0x7fa0d4197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.70:46278/0 (socket says 172.21.15.70:46278) 2024-04-01T01:50:49.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0d8966640 1 -- 172.21.15.70:0/3270112915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0d4101840 msgr2=0x7fa0d4197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0d8966640 1 --2- 172.21.15.70:0/3270112915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0d4101840 0x7fa0d4197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0d8966640 1 -- 172.21.15.70:0/3270112915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d4100650 msgr2=0x7fa0d41979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0d8966640 1 --2- 172.21.15.70:0/3270112915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d4100650 0x7fa0d41979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0d8966640 1 -- 172.21.15.70:0/3270112915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d419f590 con 0x7fa0d4102200 2024-04-01T01:50:49.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0d8966640 1 --2- 172.21.15.70:0/3270112915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d4102200 0x7fa0d419ef20 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc02c630 tx=0x7fa0cc0328a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:49.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0d17fa640 1 -- 172.21.15.70:0/3270112915 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc041020 con 0x7fa0d4102200 2024-04-01T01:50:49.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0da3c9640 1 -- 172.21.15.70:0/3270112915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d419f820 con 0x7fa0d4102200 2024-04-01T01:50:49.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0d17fa640 1 -- 172.21.15.70:0/3270112915 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0cc003e20 con 0x7fa0d4102200 2024-04-01T01:50:49.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0d17fa640 1 -- 172.21.15.70:0/3270112915 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc039040 con 0x7fa0d4102200 2024-04-01T01:50:49.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.500+0000 7fa0da3c9640 1 -- 172.21.15.70:0/3270112915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d419fd60 con 0x7fa0d4102200 2024-04-01T01:50:49.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.501+0000 7fa0da3c9640 1 -- 172.21.15.70:0/3270112915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d41a0020 con 0x7fa0d4102200 2024-04-01T01:50:49.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.502+0000 7fa0d17fa640 1 -- 172.21.15.70:0/3270112915 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fa0cc044020 con 0x7fa0d4102200 2024-04-01T01:50:49.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.503+0000 7fa0d17fa640 1 --2- 172.21.15.70:0/3270112915 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa0bc0a24a0 0x7fa0bc0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:49.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.503+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/3270112915 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa0bc0a24a0 0x7fa0bc0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:49.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.503+0000 7fa0d17fa640 1 -- 172.21.15.70:0/3270112915 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fa0cc0f1000 con 0x7fa0d4102200 2024-04-01T01:50:49.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.503+0000 7fa0d3fff640 1 --2- 172.21.15.70:0/3270112915 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa0bc0a24a0 0x7fa0bc0a4960 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fa0c40058c0 tx=0x7fa0c4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:49.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.506+0000 7fa0d17fa640 1 -- 172.21.15.70:0/3270112915 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa0cc04b020 con 0x7fa0d4102200 2024-04-01T01:50:49.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.680+0000 7fa0da3c9640 1 -- 172.21.15.70:0/3270112915 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa0d41a01f0 con 0x7fa0d4102200 2024-04-01T01:50:49.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.681+0000 7fa0d17fa640 1 -- 172.21.15.70:0/3270112915 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7fa0cc0bddf0 con 0x7fa0d4102200 2024-04-01T01:50:49.681 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-01T01:50:49.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.683+0000 7fa0da3c9640 1 -- 172.21.15.70:0/3270112915 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa0bc0a24a0 msgr2=0x7fa0bc0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.683+0000 7fa0da3c9640 1 --2- 172.21.15.70:0/3270112915 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa0bc0a24a0 0x7fa0bc0a4960 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fa0c40058c0 tx=0x7fa0c4005f90 comp rx=0 tx=0).stop 2024-04-01T01:50:49.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.683+0000 7fa0da3c9640 1 -- 172.21.15.70:0/3270112915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d4102200 msgr2=0x7fa0d419ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.683+0000 7fa0da3c9640 1 --2- 172.21.15.70:0/3270112915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d4102200 0x7fa0d419ef20 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc02c630 tx=0x7fa0cc0328a0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.683+0000 7fa0da3c9640 1 -- 172.21.15.70:0/3270112915 shutdown_connections 2024-04-01T01:50:49.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.683+0000 7fa0da3c9640 1 --2- 172.21.15.70:0/3270112915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa0d4102200 0x7fa0d419ef20 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.683+0000 7fa0da3c9640 1 --2- 172.21.15.70:0/3270112915 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fa0bc0a24a0 0x7fa0bc0a4960 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.683+0000 7fa0da3c9640 1 --2- 172.21.15.70:0/3270112915 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0d4101840 0x7fa0d4197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.683+0000 7fa0da3c9640 1 --2- 172.21.15.70:0/3270112915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d4100650 0x7fa0d41979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.683+0000 7fa0da3c9640 1 -- 172.21.15.70:0/3270112915 >> 172.21.15.70:0/3270112915 conn(0x7fa0d40fbbc0 msgr2=0x7fa0d40fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:49.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.683+0000 7fa0da3c9640 1 -- 172.21.15.70:0/3270112915 shutdown_connections 2024-04-01T01:50:49.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.683+0000 7fa0da3c9640 1 -- 172.21.15.70:0/3270112915 wait complete. 2024-04-01T01:50:49.694 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-01T01:50:49.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.798+0000 7fb9fc4f4640 1 -- 172.21.15.70:0/2393502048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f41022a0 msgr2=0x7fb9f4104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.798+0000 7fb9fc4f4640 1 --2- 172.21.15.70:0/2393502048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f41022a0 0x7fb9f4104690 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec009be0 tx=0x7fb9ec02bef0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9fc4f4640 1 -- 172.21.15.70:0/2393502048 shutdown_connections 2024-04-01T01:50:49.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9fc4f4640 1 --2- 172.21.15.70:0/2393502048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9f4107500 0x7fb9f41098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9fc4f4640 1 --2- 172.21.15.70:0/2393502048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f4104bd0 0x7fb9f4106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9fc4f4640 1 --2- 172.21.15.70:0/2393502048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f41022a0 0x7fb9f4104690 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9fc4f4640 1 -- 172.21.15.70:0/2393502048 >> 172.21.15.70:0/2393502048 conn(0x7fb9f40fbbf0 msgr2=0x7fb9f40fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:49.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9fc4f4640 1 -- 172.21.15.70:0/2393502048 shutdown_connections 2024-04-01T01:50:49.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9fc4f4640 1 -- 172.21.15.70:0/2393502048 wait complete. 2024-04-01T01:50:49.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9fc4f4640 1 Processor -- start 2024-04-01T01:50:49.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9fc4f4640 1 -- start start 2024-04-01T01:50:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9fc4f4640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f41022a0 0x7fb9f4197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9fc4f4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9f4104bd0 0x7fb9f4197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9fc4f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f4107500 0x7fb9f419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9fc4f4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f410d310 con 0x7fb9f4107500 2024-04-01T01:50:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9fc4f4640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f410d190 con 0x7fb9f41022a0 2024-04-01T01:50:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9fc4f4640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f410d490 con 0x7fb9f4104bd0 2024-04-01T01:50:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9faa91640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f4107500 0x7fb9f419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9faa91640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f4107500 0x7fb9f419efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44510/0 (socket says 172.21.15.70:44510) 2024-04-01T01:50:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9faa91640 1 -- 172.21.15.70:0/3955349933 learned_addr learned my addr 172.21.15.70:0/3955349933 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9f9a8f640 1 --2- 172.21.15.70:0/3955349933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9f4104bd0 0x7fb9f4197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.799+0000 7fb9fa290640 1 --2- 172.21.15.70:0/3955349933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f41022a0 0x7fb9f4197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.800+0000 7fb9faa91640 1 -- 172.21.15.70:0/3955349933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9f4104bd0 msgr2=0x7fb9f4197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.800+0000 7fb9faa91640 1 --2- 172.21.15.70:0/3955349933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9f4104bd0 0x7fb9f4197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.800+0000 7fb9faa91640 1 -- 172.21.15.70:0/3955349933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f41022a0 msgr2=0x7fb9f4197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.800+0000 7fb9faa91640 1 --2- 172.21.15.70:0/3955349933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f41022a0 0x7fb9f4197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.800+0000 7fb9faa91640 1 -- 172.21.15.70:0/3955349933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f419f640 con 0x7fb9f4107500 2024-04-01T01:50:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.800+0000 7fb9faa91640 1 --2- 172.21.15.70:0/3955349933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f4107500 0x7fb9f419efd0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fb9e400e970 tx=0x7fb9e400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.800+0000 7fb9f37fe640 1 -- 172.21.15.70:0/3955349933 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e400ccc0 con 0x7fb9f4107500 2024-04-01T01:50:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.800+0000 7fb9f37fe640 1 -- 172.21.15.70:0/3955349933 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb9e400ce60 con 0x7fb9f4107500 2024-04-01T01:50:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.800+0000 7fb9fc4f4640 1 -- 172.21.15.70:0/3955349933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9f419f930 con 0x7fb9f4107500 2024-04-01T01:50:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.800+0000 7fb9f37fe640 1 -- 172.21.15.70:0/3955349933 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e4010430 con 0x7fb9f4107500 2024-04-01T01:50:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.800+0000 7fb9fc4f4640 1 -- 172.21.15.70:0/3955349933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9f419fe80 con 0x7fb9f4107500 2024-04-01T01:50:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.801+0000 7fb9fc4f4640 1 -- 172.21.15.70:0/3955349933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9f40fd6e0 con 0x7fb9f4107500 2024-04-01T01:50:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.802+0000 7fb9f37fe640 1 -- 172.21.15.70:0/3955349933 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb9e4004f70 con 0x7fb9f4107500 2024-04-01T01:50:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.803+0000 7fb9f37fe640 1 --2- 172.21.15.70:0/3955349933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9dc0a24a0 0x7fb9dc0a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.803+0000 7fb9fa290640 1 --2- 172.21.15.70:0/3955349933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9dc0a24a0 0x7fb9dc0a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.803+0000 7fb9f37fe640 1 -- 172.21.15.70:0/3955349933 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fb9e40cf070 con 0x7fb9f4107500 2024-04-01T01:50:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.803+0000 7fb9fa290640 1 --2- 172.21.15.70:0/3955349933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9dc0a24a0 0x7fb9dc0a4960 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec000f80 tx=0x7fb9ec034f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:49.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.806+0000 7fb9f37fe640 1 -- 172.21.15.70:0/3955349933 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb9e4053280 con 0x7fb9f4107500 2024-04-01T01:50:49.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.981+0000 7fb9fc4f4640 1 -- 172.21.15.70:0/3955349933 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb9f402d3c0 con 0x7fb9f4107500 2024-04-01T01:50:49.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.981+0000 7fb9f37fe640 1 -- 172.21.15.70:0/3955349933 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7fb9e409be60 con 0x7fb9f4107500 2024-04-01T01:50:49.982 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-01T01:50:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.983+0000 7fb9fc4f4640 1 -- 172.21.15.70:0/3955349933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9dc0a24a0 msgr2=0x7fb9dc0a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.983+0000 7fb9fc4f4640 1 --2- 172.21.15.70:0/3955349933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9dc0a24a0 0x7fb9dc0a4960 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec000f80 tx=0x7fb9ec034f90 comp rx=0 tx=0).stop 2024-04-01T01:50:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.983+0000 7fb9fc4f4640 1 -- 172.21.15.70:0/3955349933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f4107500 msgr2=0x7fb9f419efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.983+0000 7fb9fc4f4640 1 --2- 172.21.15.70:0/3955349933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f4107500 0x7fb9f419efd0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fb9e400e970 tx=0x7fb9e400ed70 comp rx=0 tx=0).stop 2024-04-01T01:50:49.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.983+0000 7fb9fc4f4640 1 -- 172.21.15.70:0/3955349933 shutdown_connections 2024-04-01T01:50:49.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.983+0000 7fb9fc4f4640 1 --2- 172.21.15.70:0/3955349933 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9f4107500 0x7fb9f419efd0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.983+0000 7fb9fc4f4640 1 --2- 172.21.15.70:0/3955349933 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb9dc0a24a0 0x7fb9dc0a4960 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.983+0000 7fb9fc4f4640 1 --2- 172.21.15.70:0/3955349933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9f4104bd0 0x7fb9f4197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.983+0000 7fb9fc4f4640 1 --2- 172.21.15.70:0/3955349933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f41022a0 0x7fb9f4197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:49.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.983+0000 7fb9fc4f4640 1 -- 172.21.15.70:0/3955349933 >> 172.21.15.70:0/3955349933 conn(0x7fb9f40fbbf0 msgr2=0x7fb9f4102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:49.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.984+0000 7fb9fc4f4640 1 -- 172.21.15.70:0/3955349933 shutdown_connections 2024-04-01T01:50:49.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:49.984+0000 7fb9fc4f4640 1 -- 172.21.15.70:0/3955349933 wait complete. 2024-04-01T01:50:49.994 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:49 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/2926260435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:50:49.994 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:49 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/4102578490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:50:49.994 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:49 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3270112915' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:50:49.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-01T01:50:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.098+0000 7fb7fbfff640 1 -- 172.21.15.70:0/2332579368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7fc107330 msgr2=0x7fb7fc109780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.098+0000 7fb7fbfff640 1 --2- 172.21.15.70:0/2332579368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7fc107330 0x7fb7fc109780 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4009ca0 tx=0x7fb7f402c0f0 comp rx=0 tx=0).stop 2024-04-01T01:50:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.098+0000 7fb7fbfff640 1 -- 172.21.15.70:0/2332579368 shutdown_connections 2024-04-01T01:50:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.098+0000 7fb7fbfff640 1 --2- 172.21.15.70:0/2332579368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7fc107330 0x7fb7fc109780 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.098+0000 7fb7fbfff640 1 --2- 172.21.15.70:0/2332579368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7fc104a00 0x7fb7fc106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.098+0000 7fb7fbfff640 1 --2- 172.21.15.70:0/2332579368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7fc1020d0 0x7fb7fc1044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.098+0000 7fb7fbfff640 1 -- 172.21.15.70:0/2332579368 >> 172.21.15.70:0/2332579368 conn(0x7fb7fc0fbac0 msgr2=0x7fb7fc0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.098+0000 7fb7fbfff640 1 -- 172.21.15.70:0/2332579368 shutdown_connections 2024-04-01T01:50:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.098+0000 7fb7fbfff640 1 -- 172.21.15.70:0/2332579368 wait complete. 2024-04-01T01:50:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fbfff640 1 Processor -- start 2024-04-01T01:50:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fbfff640 1 -- start start 2024-04-01T01:50:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fbfff640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7fc1020d0 0x7fb7fc101a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7fc104a00 0x7fb7fc100080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fbfff640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7fc107330 0x7fb7fc1005c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fbfff640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc10d180 con 0x7fb7fc104a00 2024-04-01T01:50:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fbfff640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc10d000 con 0x7fb7fc1020d0 2024-04-01T01:50:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fbfff640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc10d300 con 0x7fb7fc107330 2024-04-01T01:50:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fa7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7fc104a00 0x7fb7fc100080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fa7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7fc104a00 0x7fb7fc100080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44528/0 (socket says 172.21.15.70:44528) 2024-04-01T01:50:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fa7fc640 1 -- 172.21.15.70:0/745335229 learned_addr learned my addr 172.21.15.70:0/745335229 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-01T01:50:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fb7fe640 1 --2- 172.21.15.70:0/745335229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7fc107330 0x7fb7fc1005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7faffd640 1 --2- 172.21.15.70:0/745335229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7fc1020d0 0x7fb7fc101a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fa7fc640 1 -- 172.21.15.70:0/745335229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7fc107330 msgr2=0x7fb7fc1005c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fa7fc640 1 --2- 172.21.15.70:0/745335229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7fc107330 0x7fb7fc1005c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:50.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fa7fc640 1 -- 172.21.15.70:0/745335229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7fc1020d0 msgr2=0x7fb7fc101a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:50.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fa7fc640 1 --2- 172.21.15.70:0/745335229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7fc1020d0 0x7fb7fc101a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:50.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.099+0000 7fb7fa7fc640 1 -- 172.21.15.70:0/745335229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7fc100e50 con 0x7fb7fc104a00 2024-04-01T01:50:50.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.100+0000 7fb7fa7fc640 1 --2- 172.21.15.70:0/745335229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7fc104a00 0x7fb7fc100080 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000e9c0 tx=0x7fb7f000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:50.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.100+0000 7fb7e3fff640 1 -- 172.21.15.70:0/745335229 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f000cd80 con 0x7fb7fc104a00 2024-04-01T01:50:50.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.100+0000 7fb7fbfff640 1 -- 172.21.15.70:0/745335229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7fc1a1940 con 0x7fb7fc104a00 2024-04-01T01:50:50.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.100+0000 7fb7e3fff640 1 -- 172.21.15.70:0/745335229 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb7f00054d0 con 0x7fb7fc104a00 2024-04-01T01:50:50.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.100+0000 7fb7e3fff640 1 -- 172.21.15.70:0/745335229 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0010430 con 0x7fb7fc104a00 2024-04-01T01:50:50.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.100+0000 7fb7fbfff640 1 -- 172.21.15.70:0/745335229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7fc1a1ee0 con 0x7fb7fc104a00 2024-04-01T01:50:50.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.101+0000 7fb7fbfff640 1 -- 172.21.15.70:0/745335229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7fc0fd5b0 con 0x7fb7fc104a00 2024-04-01T01:50:50.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.102+0000 7fb7e3fff640 1 -- 172.21.15.70:0/745335229 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 20) ==== 137320+0+0 (secure 0 0 0) 0x7fb7f0004fb0 con 0x7fb7fc104a00 2024-04-01T01:50:50.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.102+0000 7fb7e3fff640 1 --2- 172.21.15.70:0/745335229 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7e40a24a0 0x7fb7e40a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-01T01:50:50.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.103+0000 7fb7faffd640 1 --2- 172.21.15.70:0/745335229 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7e40a24a0 0x7fb7e40a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-01T01:50:50.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.103+0000 7fb7e3fff640 1 -- 172.21.15.70:0/745335229 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9208+0+0 (secure 0 0 0) 0x7fb7f00cf170 con 0x7fb7fc104a00 2024-04-01T01:50:50.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.103+0000 7fb7faffd640 1 --2- 172.21.15.70:0/745335229 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7e40a24a0 0x7fb7e40a4960 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0095f0 tx=0x7fb7ec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-01T01:50:50.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.106+0000 7fb7e3fff640 1 -- 172.21.15.70:0/745335229 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb7f0053280 con 0x7fb7fc104a00 2024-04-01T01:50:50.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:49 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/2926260435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:50:50.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:49 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/4102578490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:50:50.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:49 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3270112915' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:50:50.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.293+0000 7fb7fbfff640 1 -- 172.21.15.70:0/745335229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb7fc02d170 con 0x7fb7fc104a00 2024-04-01T01:50:50.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.294+0000 7fb7e3fff640 1 -- 172.21.15.70:0/745335229 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v95) ==== 81+0+590 (secure 0 0 0) 0x7fb7f009bf60 con 0x7fb7fc104a00 2024-04-01T01:50:50.295 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/950171900 2024-04-02T01:24:18.477610+0000 2024-04-01T01:50:50.295 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/2902831307 2024-04-02T01:24:18.477610+0000 2024-04-01T01:50:50.295 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6801/2836712828 2024-04-02T01:24:18.477610+0000 2024-04-01T01:50:50.295 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6800/2836712828 2024-04-02T01:24:18.477610+0000 2024-04-01T01:50:50.295 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/602656012 2024-04-02T01:23:52.091015+0000 2024-04-01T01:50:50.295 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6800/2121290979 2024-04-02T01:23:52.091015+0000 2024-04-01T01:50:50.295 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:6801/2121290979 2024-04-02T01:23:52.091015+0000 2024-04-01T01:50:50.295 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/3734182136 2024-04-02T01:23:52.091015+0000 2024-04-01T01:50:50.295 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/2899206193 2024-04-02T01:23:52.091015+0000 2024-04-01T01:50:50.295 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70:0/3948868469 2024-04-02T01:24:18.477610+0000 2024-04-01T01:50:50.295 INFO:teuthology.orchestra.run.smithi070.stderr:listed 10 entries 2024-04-01T01:50:50.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.296+0000 7fb7fbfff640 1 -- 172.21.15.70:0/745335229 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7e40a24a0 msgr2=0x7fb7e40a4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:50.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.296+0000 7fb7fbfff640 1 --2- 172.21.15.70:0/745335229 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7e40a24a0 0x7fb7e40a4960 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0095f0 tx=0x7fb7ec005f90 comp rx=0 tx=0).stop 2024-04-01T01:50:50.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.296+0000 7fb7fbfff640 1 -- 172.21.15.70:0/745335229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7fc104a00 msgr2=0x7fb7fc100080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-01T01:50:50.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.296+0000 7fb7fbfff640 1 --2- 172.21.15.70:0/745335229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7fc104a00 0x7fb7fc100080 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000e9c0 tx=0x7fb7f000ee80 comp rx=0 tx=0).stop 2024-04-01T01:50:50.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.296+0000 7fb7fbfff640 1 -- 172.21.15.70:0/745335229 shutdown_connections 2024-04-01T01:50:50.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.296+0000 7fb7fbfff640 1 --2- 172.21.15.70:0/745335229 >> [v2:172.21.15.70:6800/2070166741,v1:172.21.15.70:6801/2070166741] conn(0x7fb7e40a24a0 0x7fb7e40a4960 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:50.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.296+0000 7fb7fbfff640 1 --2- 172.21.15.70:0/745335229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7fc107330 0x7fb7fc1005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:50.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.296+0000 7fb7fbfff640 1 --2- 172.21.15.70:0/745335229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb7fc104a00 0x7fb7fc100080 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:50.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.296+0000 7fb7fbfff640 1 --2- 172.21.15.70:0/745335229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7fc1020d0 0x7fb7fc101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-01T01:50:50.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.296+0000 7fb7fbfff640 1 -- 172.21.15.70:0/745335229 >> 172.21.15.70:0/745335229 conn(0x7fb7fc0fbac0 msgr2=0x7fb7fc102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-01T01:50:50.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.296+0000 7fb7fbfff640 1 -- 172.21.15.70:0/745335229 shutdown_connections 2024-04-01T01:50:50.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-01T01:50:50.296+0000 7fb7fbfff640 1 -- 172.21.15.70:0/745335229 wait complete. 2024-04-01T01:50:50.307 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-01T01:50:50.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:50:50.308 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /tmp/cephfs 2024-04-01T01:50:50.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:49 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/2926260435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-01T01:50:50.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:49 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/4102578490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-01T01:50:50.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:49 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3270112915' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-01T01:50:50.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[86412]: fuse finished with error 0 and tester_r 0 2024-04-01T01:50:50.439 INFO:teuthology.orchestra.run:waiting for 300 2024-04-01T01:50:51.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:50 smithi070 ceph-mon[32179]: pgmap v880: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-01T01:50:51.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:50 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/3955349933' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:50:51.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:50 smithi070 ceph-mon[32179]: from='client.? 172.21.15.70:0/745335229' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-01T01:50:51.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:50 smithi189 ceph-mon[35753]: pgmap v880: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-01T01:50:51.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:50 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/3955349933' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:50:51.249 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:50 smithi189 ceph-mon[35753]: from='client.? 172.21.15.70:0/745335229' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-01T01:50:51.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:50 smithi081 ceph-mon[35142]: pgmap v880: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-01T01:50:51.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:50 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/3955349933' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-01T01:50:51.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:50 smithi081 ceph-mon[35142]: from='client.? 172.21.15.70:0/745335229' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-01T01:50:53.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:52 smithi189 ceph-mon[35753]: pgmap v881: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-01T01:50:53.253 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:52 smithi070 ceph-mon[32179]: pgmap v881: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-01T01:50:53.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:52 smithi081 ceph-mon[35142]: pgmap v881: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-01T01:50:55.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:54 smithi189 ceph-mon[35753]: pgmap v882: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-01T01:50:55.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:54 smithi070 ceph-mon[32179]: pgmap v882: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-01T01:50:55.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:54 smithi081 ceph-mon[35142]: pgmap v882: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-04-01T01:50:56.541 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-01T01:50:56.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:50:56.541 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /tmp/cephfs) 2024-04-01T01:50:56.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:50:56.570 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-01T01:50:56.651 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--tmp-cephfs (id: 0) 2024-04-01T01:50:56.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:50:56.651 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns delete ceph-ns--tmp-cephfs 2024-04-01T01:50:56.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-01T01:50:56.724 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link delete ceph-brx 2024-04-01T01:50:56.826 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-01T01:50:56.838 INFO:tasks.cephadm:Teardown begin 2024-04-01T01:50:56.838 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/check_counter.py", line 125, in end raise RuntimeError("The following counters failed to be set " RuntimeError: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} 2024-04-01T01:50:56.839 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-01T01:50:56.888 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-01T01:50:56.919 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-01T01:50:56.950 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-01T01:50:56.950 DEBUG:teuthology.orchestra.run.smithi070:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-01T01:50:56.966 DEBUG:teuthology.orchestra.run.smithi081:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-01T01:50:56.980 DEBUG:teuthology.orchestra.run.smithi189:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-01T01:50:57.004 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:56 smithi070 ceph-mon[32179]: pgmap v883: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-01T01:50:57.005 INFO:tasks.cephadm:Stopping all daemons... 2024-04-01T01:50:57.006 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-01T01:50:57.006 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mon.a 2024-04-01T01:50:57.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:56 smithi189 ceph-mon[35753]: pgmap v883: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-01T01:50:57.316 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:57 smithi070 systemd[1]: Stopping Ceph mon.a for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:50:57.328 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:56 smithi081 ceph-mon[35142]: pgmap v883: 129 pgs: 129 active+clean; 1.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-01T01:50:57.581 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:57 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-a[32175]: 2024-04-01T01:50:57.315+0000 7f7ab202a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:50:57.581 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:57 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-a[32175]: 2024-04-01T01:50:57.315+0000 7f7ab202a700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-01T01:50:57.841 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:57 smithi070 podman[117688]: 2024-04-01 01:50:57.581709761 +0000 UTC m=+0.380738972 container died 3284b805c9455080371237fc50d172cf40066feb8605cb7bb0fe88e233bfc5d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-01T01:50:58.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:57 smithi070 podman[117688]: 2024-04-01 01:50:57.941248247 +0000 UTC m=+0.740277464 container cleanup 3284b805c9455080371237fc50d172cf40066feb8605cb7bb0fe88e233bfc5d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-01T01:50:58.254 INFO:journalctl@ceph.mon.a.smithi070.stdout:Apr 01 01:50:57 smithi070 bash[117688]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-a 2024-04-01T01:50:58.401 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mon.a.service' 2024-04-01T01:50:58.455 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:50:58.455 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-01T01:50:58.456 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-01T01:50:58.456 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mon.b 2024-04-01T01:50:58.828 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:58 smithi081 systemd[1]: Stopping Ceph mon.b for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:50:59.308 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:58 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-b[35138]: 2024-04-01T01:50:58.856+0000 7f44b6959700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:50:59.308 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:58 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-b[35138]: 2024-04-01T01:50:58.856+0000 7f44b6959700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-01T01:50:59.308 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:59 smithi081 podman[70043]: 2024-04-01 01:50:59.050338369 +0000 UTC m=+0.338364226 container died be19c90532f555e6f26e81b53c206f5ac644a169cb29f88b084bc56b20d90531 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=) 2024-04-01T01:50:59.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:59 smithi081 podman[70043]: 2024-04-01 01:50:59.415614978 +0000 UTC m=+0.703640828 container cleanup be19c90532f555e6f26e81b53c206f5ac644a169cb29f88b084bc56b20d90531 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, org.label-schema.license=GPLv2) 2024-04-01T01:50:59.578 INFO:journalctl@ceph.mon.b.smithi081.stdout:Apr 01 01:50:59 smithi081 bash[70043]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-b 2024-04-01T01:50:59.883 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mon.b.service' 2024-04-01T01:50:59.943 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:50:59.943 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-01T01:50:59.943 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-01T01:50:59.943 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mon.c 2024-04-01T01:51:00.248 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:50:59 smithi189 systemd[1]: Stopping Ceph mon.c for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:51:00.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:51:00 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-c[35749]: 2024-04-01T01:51:00.334+0000 7f6d2a934700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:51:00.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:51:00 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-c[35749]: 2024-04-01T01:51:00.334+0000 7f6d2a934700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-01T01:51:00.749 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:51:00 smithi189 podman[72283]: 2024-04-01 01:51:00.558598058 +0000 UTC m=+0.336820863 container died 72e7bf301c270dc40b9e417b706fd5d50b20635662da27ad0b1df30e44ec847e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326) 2024-04-01T01:51:01.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:51:00 smithi189 podman[72283]: 2024-04-01 01:51:00.969543588 +0000 UTC m=+0.747766398 container cleanup 72e7bf301c270dc40b9e417b706fd5d50b20635662da27ad0b1df30e44ec847e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-c, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-01T01:51:01.221 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 01 01:51:00 smithi189 bash[72283]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mon-c 2024-04-01T01:51:01.355 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mon.c.service' 2024-04-01T01:51:01.410 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:51:01.411 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-01T01:51:01.411 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-01T01:51:01.411 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mgr.x 2024-04-01T01:51:01.737 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:51:01 smithi070 systemd[1]: Stopping Ceph mgr.x for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:51:02.004 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:51:01 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x[32412]: 2024-04-01T01:51:01.737+0000 7f383f9e7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:51:02.004 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:51:01 smithi070 podman[117823]: 2024-04-01 01:51:01.768288228 +0000 UTC m=+0.162135546 container died aed29d5f424085705770a436e499e640e113ad1e429d818dcfc91f5444981f25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_CLEAN=True, RELEASE=squid-384f31f, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-01T01:51:02.421 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:51:02 smithi070 podman[117823]: 2024-04-01 01:51:02.19597235 +0000 UTC m=+0.589819669 container cleanup aed29d5f424085705770a436e499e640e113ad1e429d818dcfc91f5444981f25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:384f31f3d3aee7f8ba3f60846b8844c049165583, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-384f31f) 2024-04-01T01:51:02.422 INFO:journalctl@ceph.mgr.x.smithi070.stdout:Apr 01 01:51:02 smithi070 bash[117823]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-x 2024-04-01T01:51:02.621 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mgr.x.service' 2024-04-01T01:51:02.661 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:51:02.661 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-01T01:51:02.661 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-01T01:51:02.662 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mgr.y 2024-04-01T01:51:03.008 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:51:02 smithi081 systemd[1]: Stopping Ceph mgr.y for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:51:03.297 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:51:03 smithi081 podman[70162]: 2024-04-01 01:51:03.00851677 +0000 UTC m=+0.141534103 container died 2e186bfb704ac6f29a42abb78454e9bf0ad41fd3f93f157f228c0a4789746f89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-01T01:51:03.578 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:51:03 smithi081 podman[70162]: 2024-04-01 01:51:03.380444449 +0000 UTC m=+0.513461764 container cleanup 2e186bfb704ac6f29a42abb78454e9bf0ad41fd3f93f157f228c0a4789746f89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-01T01:51:03.578 INFO:journalctl@ceph.mgr.y.smithi081.stdout:Apr 01 01:51:03 smithi081 bash[70162]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-y 2024-04-01T01:51:03.812 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mgr.y.service' 2024-04-01T01:51:03.851 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:51:03.851 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-01T01:51:03.852 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-01T01:51:03.852 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mgr.z 2024-04-01T01:51:04.249 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:51:03 smithi189 systemd[1]: Stopping Ceph mgr.z for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:51:04.593 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:51:04 smithi189 podman[72403]: 2024-04-01 01:51:04.298480138 +0000 UTC m=+0.213184776 container died d852bc6452591807f09150d4be370d925fdb3f45fc44459e916f8469ced5874e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f) 2024-04-01T01:51:04.978 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:51:04 smithi189 podman[72403]: 2024-04-01 01:51:04.701488747 +0000 UTC m=+0.616193373 container cleanup d852bc6452591807f09150d4be370d925fdb3f45fc44459e916f8469ced5874e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-01T01:51:04.979 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 01 01:51:04 smithi189 bash[72403]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mgr-z 2024-04-01T01:51:05.181 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mgr.z.service' 2024-04-01T01:51:05.219 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:51:05.219 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-01T01:51:05.219 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-01T01:51:05.219 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.0 2024-04-01T01:51:05.504 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:51:05 smithi070 systemd[1]: Stopping Ceph osd.0 for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:51:06.004 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:51:05 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-0[47150]: 2024-04-01T01:51:05.542+0000 7ffad127b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:51:06.004 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:51:05 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-0[47150]: 2024-04-01T01:51:05.542+0000 7ffad127b700 -1 osd.0 95 *** Got signal Terminated *** 2024-04-01T01:51:06.004 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:51:05 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-0[47150]: 2024-04-01T01:51:05.542+0000 7ffad127b700 -1 osd.0 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-01T01:51:11.575 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:51:11 smithi070 podman[117944]: 2024-04-01 01:51:11.256541212 +0000 UTC m=+5.843111082 container died 47d3ee1d77ea657b29cdf05bfd07811e7e3fb7639d6d4398c490f407df978c13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-04-01T01:51:11.933 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:51:11 smithi070 podman[117944]: 2024-04-01 01:51:11.682825947 +0000 UTC m=+6.269395818 container cleanup 47d3ee1d77ea657b29cdf05bfd07811e7e3fb7639d6d4398c490f407df978c13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-01T01:51:11.933 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:51:11 smithi070 bash[117944]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-0 2024-04-01T01:51:12.254 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:51:11 smithi070 podman[117962]: 2024-04-01 01:51:11.934229152 +0000 UTC m=+0.674007190 container remove 47d3ee1d77ea657b29cdf05bfd07811e7e3fb7639d6d4398c490f407df978c13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-04-01T01:51:12.637 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:51:12 smithi070 podman[118057]: 2024-04-01 01:51:12.254988032 +0000 UTC m=+0.018805373 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:51:12.637 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:51:12 smithi070 podman[118057]: 2024-04-01 01:51:12.368512275 +0000 UTC m=+0.132329586 container create 8c23361357f947c3bd87ebe5e07282080c1d38078d033e5c62f2657a679137c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-384f31f, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326) 2024-04-01T01:51:13.004 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:51:12 smithi070 podman[118057]: 2024-04-01 01:51:12.637855621 +0000 UTC m=+0.401672939 container init 8c23361357f947c3bd87ebe5e07282080c1d38078d033e5c62f2657a679137c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-0-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-01T01:51:13.004 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:51:12 smithi070 podman[118057]: 2024-04-01 01:51:12.641320501 +0000 UTC m=+0.405137820 container start 8c23361357f947c3bd87ebe5e07282080c1d38078d033e5c62f2657a679137c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f) 2024-04-01T01:51:13.004 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:51:12 smithi070 podman[118057]: 2024-04-01 01:51:12.737874735 +0000 UTC m=+0.501692050 container attach 8c23361357f947c3bd87ebe5e07282080c1d38078d033e5c62f2657a679137c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-0-deactivate, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-01T01:51:13.504 INFO:journalctl@ceph.osd.0.smithi070.stdout:Apr 01 01:51:13 smithi070 podman[118057]: 2024-04-01 01:51:13.163942156 +0000 UTC m=+0.927759483 container died 8c23361357f947c3bd87ebe5e07282080c1d38078d033e5c62f2657a679137c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-0-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-01T01:51:13.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.0.service' 2024-04-01T01:51:13.913 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:51:13.914 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-01T01:51:13.914 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-01T01:51:13.914 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.1 2024-04-01T01:51:14.311 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:51:13 smithi081 systemd[1]: Stopping Ceph osd.1 for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:51:14.578 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:51:14 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-1[39718]: 2024-04-01T01:51:14.311+0000 7f66edd89700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:51:14.578 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:51:14 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-1[39718]: 2024-04-01T01:51:14.311+0000 7f66edd89700 -1 osd.1 95 *** Got signal Terminated *** 2024-04-01T01:51:14.578 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:51:14 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-1[39718]: 2024-04-01T01:51:14.311+0000 7f66edd89700 -1 osd.1 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-01T01:51:20.328 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:51:20 smithi081 podman[70290]: 2024-04-01 01:51:20.077041464 +0000 UTC m=+5.951210866 container died 86cb72dad9a0fa16ae3be5d504f5afff6c65c3c7ec5f6da241f5aab4004b660b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-1, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-01T01:51:20.738 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:51:20 smithi081 podman[70290]: 2024-04-01 01:51:20.461669107 +0000 UTC m=+6.335838500 container cleanup 86cb72dad9a0fa16ae3be5d504f5afff6c65c3c7ec5f6da241f5aab4004b660b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-01T01:51:20.738 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:51:20 smithi081 bash[70290]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-1 2024-04-01T01:51:21.078 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:51:20 smithi081 podman[70309]: 2024-04-01 01:51:20.738696208 +0000 UTC m=+0.665632551 container remove 86cb72dad9a0fa16ae3be5d504f5afff6c65c3c7ec5f6da241f5aab4004b660b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-1, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-01T01:51:21.418 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:51:21 smithi081 podman[70396]: 2024-04-01 01:51:21.01919841 +0000 UTC m=+0.022856834 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:51:21.418 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:51:21 smithi081 podman[70396]: 2024-04-01 01:51:21.148215195 +0000 UTC m=+0.151873665 container create 11da34ac73098d2373abcc52b13f3fc41532d3740d7eb1c9fff521180ce631aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, GIT_CLEAN=True) 2024-04-01T01:51:21.418 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:51:21 smithi081 podman[70396]: 2024-04-01 01:51:21.417714486 +0000 UTC m=+0.421372910 container init 11da34ac73098d2373abcc52b13f3fc41532d3740d7eb1c9fff521180ce631aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-01T01:51:21.828 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:51:21 smithi081 podman[70396]: 2024-04-01 01:51:21.422350572 +0000 UTC m=+0.426009004 container start 11da34ac73098d2373abcc52b13f3fc41532d3740d7eb1c9fff521180ce631aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-01T01:51:21.828 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:51:21 smithi081 podman[70396]: 2024-04-01 01:51:21.517736024 +0000 UTC m=+0.521394463 container attach 11da34ac73098d2373abcc52b13f3fc41532d3740d7eb1c9fff521180ce631aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-01T01:51:22.246 INFO:journalctl@ceph.osd.1.smithi081.stdout:Apr 01 01:51:21 smithi081 podman[70396]: 2024-04-01 01:51:21.965450042 +0000 UTC m=+0.969108471 container died 11da34ac73098d2373abcc52b13f3fc41532d3740d7eb1c9fff521180ce631aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-1-deactivate, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-01T01:51:22.551 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.1.service' 2024-04-01T01:51:22.595 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:51:22.595 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-01T01:51:22.595 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-01T01:51:22.595 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.2 2024-04-01T01:51:22.932 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:51:22 smithi189 systemd[1]: Stopping Ceph osd.2 for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:51:23.248 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:51:22 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-2[40897]: 2024-04-01T01:51:22.931+0000 7fc32c234700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:51:23.249 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:51:22 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-2[40897]: 2024-04-01T01:51:22.931+0000 7fc32c234700 -1 osd.2 95 *** Got signal Terminated *** 2024-04-01T01:51:23.249 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:51:22 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-2[40897]: 2024-04-01T01:51:22.931+0000 7fc32c234700 -1 osd.2 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-01T01:51:28.958 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:51:28 smithi189 podman[72538]: 2024-04-01 01:51:28.658097394 +0000 UTC m=+5.846520811 container died aa94aee2b9ef1a08995b5777c6842435aaa803cd591f55196677e20c2f282e7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-04-01T01:51:29.249 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:51:29 smithi189 podman[72538]: 2024-04-01 01:51:29.057639065 +0000 UTC m=+6.246062481 container cleanup aa94aee2b9ef1a08995b5777c6842435aaa803cd591f55196677e20c2f282e7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-01T01:51:29.249 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:51:29 smithi189 bash[72538]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-2 2024-04-01T01:51:29.685 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:51:29 smithi189 podman[72556]: 2024-04-01 01:51:29.318029767 +0000 UTC m=+0.661961906 container remove aa94aee2b9ef1a08995b5777c6842435aaa803cd591f55196677e20c2f282e7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-01T01:51:29.963 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:51:29 smithi189 podman[72641]: 2024-04-01 01:51:29.589385579 +0000 UTC m=+0.026308102 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:51:29.963 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:51:29 smithi189 podman[72641]: 2024-04-01 01:51:29.710746837 +0000 UTC m=+0.147669357 container create 842eadb628299c57426d362b0b7f062d3d7f94bee198d937660cc988fd2b3c71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240326, GIT_CLEAN=True) 2024-04-01T01:51:30.249 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:51:29 smithi189 podman[72641]: 2024-04-01 01:51:29.964136575 +0000 UTC m=+0.401059108 container init 842eadb628299c57426d362b0b7f062d3d7f94bee198d937660cc988fd2b3c71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, ceph=True, CEPH_POINT_RELEASE=) 2024-04-01T01:51:30.249 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:51:29 smithi189 podman[72641]: 2024-04-01 01:51:29.967803179 +0000 UTC m=+0.404725705 container start 842eadb628299c57426d362b0b7f062d3d7f94bee198d937660cc988fd2b3c71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-2-deactivate, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240326) 2024-04-01T01:51:30.249 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:51:30 smithi189 podman[72641]: 2024-04-01 01:51:30.063661606 +0000 UTC m=+0.500584127 container attach 842eadb628299c57426d362b0b7f062d3d7f94bee198d937660cc988fd2b3c71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-2-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=) 2024-04-01T01:51:30.677 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 01 01:51:30 smithi189 podman[72641]: 2024-04-01 01:51:30.379403051 +0000 UTC m=+0.816325579 container died 842eadb628299c57426d362b0b7f062d3d7f94bee198d937660cc988fd2b3c71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD) 2024-04-01T01:51:30.978 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.2.service' 2024-04-01T01:51:31.022 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:51:31.022 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-01T01:51:31.022 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-01T01:51:31.022 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.3 2024-04-01T01:51:31.358 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:31 smithi070 systemd[1]: Stopping Ceph osd.3 for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:51:31.620 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:31 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3[53932]: 2024-04-01T01:51:31.357+0000 7f3f766ad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:51:31.698 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:31 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3[53932]: 2024-04-01T01:51:31.357+0000 7f3f766ad700 -1 osd.3 95 *** Got signal Terminated *** 2024-04-01T01:51:31.699 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:31 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3[53932]: 2024-04-01T01:51:31.357+0000 7f3f766ad700 -1 osd.3 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-01T01:51:31.699 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:31 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3[53932]: 2024-04-01T01:51:31.598+0000 7f3f72ca1700 -1 osd.3 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:05.686837+0000 front 2024-04-01T01:51:05.686862+0000 (oldest deadline 2024-04-01T01:51:31.587011+0000) 2024-04-01T01:51:32.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:31 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:31.902+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:32.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:31 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:31.924+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:32.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:32 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:32.375+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:32.875 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:32 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3[53932]: 2024-04-01T01:51:32.626+0000 7f3f72ca1700 -1 osd.3 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:05.686837+0000 front 2024-04-01T01:51:05.686862+0000 (oldest deadline 2024-04-01T01:51:31.587011+0000) 2024-04-01T01:51:33.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:32 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:32.874+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:33.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:32 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:32.955+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:33.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:33 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:33.558+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:33.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:33 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:33.475+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:33.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:33 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:33.354+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:33.951 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:33 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:33.885+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:33.951 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:33 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3[53932]: 2024-04-01T01:51:33.649+0000 7f3f72ca1700 -1 osd.3 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:05.686837+0000 front 2024-04-01T01:51:05.686862+0000 (oldest deadline 2024-04-01T01:51:31.587011+0000) 2024-04-01T01:51:34.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:33 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:33.951+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:34.748 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:34 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:34.548+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:34.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:34 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:34.467+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:34.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:34 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:34.359+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:34.924 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:34 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3[53932]: 2024-04-01T01:51:34.632+0000 7f3f72ca1700 -1 osd.3 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:05.686837+0000 front 2024-04-01T01:51:05.686862+0000 (oldest deadline 2024-04-01T01:51:31.587011+0000) 2024-04-01T01:51:35.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:34 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:34.924+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:35.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:34 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:34.941+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:35.498 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:35 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:35.108+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:35.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:35 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:35.403+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:35.924 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:35 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3[53932]: 2024-04-01T01:51:35.645+0000 7f3f72ca1700 -1 osd.3 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:05.686837+0000 front 2024-04-01T01:51:05.686862+0000 (oldest deadline 2024-04-01T01:51:31.587011+0000) 2024-04-01T01:51:35.998 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:35 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:35.506+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:35.999 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:35 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:35.547+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:36.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:35 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:35.924+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:36.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:35 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:35.988+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:36.498 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:36 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:36.111+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:36.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:36 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:36.446+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:36.998 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:36 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:36.575+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:36.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:36 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:36.511+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:37.149 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:36 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:36.964+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:37.149 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:36 smithi070 podman[118162]: 2024-04-01 01:51:36.840895416 +0000 UTC m=+5.623922729 container died d1c1edad15b770c8ddf57b324bf071ddfe1b0718afc66295fd4c5541a3869efe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-04-01T01:51:37.150 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:36 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:36.977+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:37.483 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:37 smithi070 podman[118162]: 2024-04-01 01:51:37.240789721 +0000 UTC m=+6.023817024 container cleanup d1c1edad15b770c8ddf57b324bf071ddfe1b0718afc66295fd4c5541a3869efe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-01T01:51:37.483 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:37 smithi070 bash[118162]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3 2024-04-01T01:51:37.499 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:37 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:37.140+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:37.754 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:37 smithi070 podman[118193]: 2024-04-01 01:51:37.483649226 +0000 UTC m=+0.640071252 container remove d1c1edad15b770c8ddf57b324bf071ddfe1b0718afc66295fd4c5541a3869efe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-04-01T01:51:37.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:37 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:37.399+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:37.999 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:37 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:37.571+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:37.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:37 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:37.510+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:38.053 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:37 smithi070 podman[118283]: 2024-04-01 01:51:37.703567083 +0000 UTC m=+0.017851669 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:51:38.054 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:37 smithi070 podman[118283]: 2024-04-01 01:51:37.801525768 +0000 UTC m=+0.115810393 container create 0a570c5a255064e35f8f469c5b2bfb6af2da61d2dc61c92f9bbce8770672cb39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3-deactivate, ceph=True, GIT_CLEAN=True, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-01T01:51:38.054 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:37 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:37.996+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:38.054 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:37 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:37.942+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:38.344 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:38 smithi070 podman[118283]: 2024-04-01 01:51:38.05414201 +0000 UTC m=+0.368426592 container init 0a570c5a255064e35f8f469c5b2bfb6af2da61d2dc61c92f9bbce8770672cb39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3-deactivate, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-01T01:51:38.344 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:38 smithi070 podman[118283]: 2024-04-01 01:51:38.057536844 +0000 UTC m=+0.371821427 container start 0a570c5a255064e35f8f469c5b2bfb6af2da61d2dc61c92f9bbce8770672cb39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-01T01:51:38.344 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:38 smithi070 podman[118283]: 2024-04-01 01:51:38.145650062 +0000 UTC m=+0.459934657 container attach 0a570c5a255064e35f8f469c5b2bfb6af2da61d2dc61c92f9bbce8770672cb39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326) 2024-04-01T01:51:38.487 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:38 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:38.149+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:38.622 INFO:journalctl@ceph.osd.3.smithi070.stdout:Apr 01 01:51:38 smithi070 podman[118283]: 2024-04-01 01:51:38.344929246 +0000 UTC m=+0.659213833 container died 0a570c5a255064e35f8f469c5b2bfb6af2da61d2dc61c92f9bbce8770672cb39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-3-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326) 2024-04-01T01:51:38.748 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:38 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:38.487+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:38.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:38 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:38.548+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:38.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:38 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:38.379+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:38.910 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.3.service' 2024-04-01T01:51:38.954 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:51:38.954 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-01T01:51:38.955 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-01T01:51:38.955 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.4 2024-04-01T01:51:39.250 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:38 smithi081 systemd[1]: Stopping Ceph osd.4 for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:51:39.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:39 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:39.039+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:39.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:38 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:38.937+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:39.492 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:39 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:39.111+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:39.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:39 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:39.355+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:39.578 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:39 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4[46199]: 2024-04-01T01:51:39.249+0000 7f378b343700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:51:39.578 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:39 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4[46199]: 2024-04-01T01:51:39.249+0000 7f378b343700 -1 osd.4 95 *** Got signal Terminated *** 2024-04-01T01:51:39.578 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:39 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4[46199]: 2024-04-01T01:51:39.249+0000 7f378b343700 -1 osd.4 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-01T01:51:39.748 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:39 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:39.505+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:39.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:39 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:39.492+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:40.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:40 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:40.027+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:40.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:39 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:39.921+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:40.498 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:40 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:40.115+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:40.748 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:40 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:40.525+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:40.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:40 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:40.498+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:40.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:40 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:40.347+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:41.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:40 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:40.986+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:41.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:40 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:40.892+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:41.300 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:40 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4[46199]: 2024-04-01T01:51:40.836+0000 7f3787937700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.168660+0000 front 2024-04-01T01:51:17.168676+0000 (oldest deadline 2024-04-01T01:51:40.068745+0000) 2024-04-01T01:51:41.457 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:41 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:41.098+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:41.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:41 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:41.299+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:41.748 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:41 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:41.491+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:41.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:41 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:41.456+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:42.078 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:41 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4[46199]: 2024-04-01T01:51:41.810+0000 7f3787937700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.168660+0000 front 2024-04-01T01:51:17.168676+0000 (oldest deadline 2024-04-01T01:51:40.068745+0000) 2024-04-01T01:51:42.078 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:41 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:41.611+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:42.249 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:42 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:42.095+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:42.249 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:42 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:42.095+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.363678+0000 front 2024-04-01T01:51:17.363699+0000 (oldest deadline 2024-04-01T01:51:41.364127+0000) 2024-04-01T01:51:42.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:42 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:42.001+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:42.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:41 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:41.845+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:42.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:41 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:41.845+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:42.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:42 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:42.326+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:42.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:42 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:42.326+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:42.748 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:42 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:42.407+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:42.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:42 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:42.470+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:42.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:42 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:42.470+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:43.078 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:42 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:42.647+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:43.078 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:42 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4[46199]: 2024-04-01T01:51:42.858+0000 7f3787937700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.168660+0000 front 2024-04-01T01:51:17.168676+0000 (oldest deadline 2024-04-01T01:51:40.068745+0000) 2024-04-01T01:51:43.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:42 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:42.997+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:43.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:42 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:42.997+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:43.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:42 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:42.805+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:43.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:42 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:42.805+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:43.402 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:43 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:43.057+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:43.402 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:43 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:43.057+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.363678+0000 front 2024-04-01T01:51:17.363699+0000 (oldest deadline 2024-04-01T01:51:41.364127+0000) 2024-04-01T01:51:43.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:43 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:43.327+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:43.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:43 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:43.327+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:43.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:43 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:43.402+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:43.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:43 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:43.479+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:43.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:43 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:43.479+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:44.078 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:43 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4[46199]: 2024-04-01T01:51:43.868+0000 7f3787937700 -1 osd.4 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.168660+0000 front 2024-04-01T01:51:17.168676+0000 (oldest deadline 2024-04-01T01:51:40.068745+0000) 2024-04-01T01:51:44.078 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:43 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:43.674+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:44.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:44 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:44.007+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:44.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:44 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:44.007+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:44.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:43 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:43.854+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:44.255 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:43 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:43.854+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:44.428 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:44 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:44.062+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:44.429 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:44 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:44.062+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.363678+0000 front 2024-04-01T01:51:17.363699+0000 (oldest deadline 2024-04-01T01:51:41.364127+0000) 2024-04-01T01:51:44.689 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:44 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:44.373+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:44.689 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:44 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:44.373+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:44.748 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:44 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:44.451+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:44.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:44 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:44.451+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:44.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:44 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:44.428+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:44.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:44 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:44.428+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:51:45.078 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:44 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:44.689+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:45.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:45 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:45.028+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:45.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:45 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:45.028+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:45.255 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:44 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:44.838+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:45.255 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:44 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:44.838+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:45.345 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:45 smithi081 podman[70515]: 2024-04-01 01:51:45.086273215 +0000 UTC m=+5.944147522 container died a1392b6f479d7ad2a76e4b67593eab156ef272653866aa9c2ab944881e70691b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-01T01:51:45.385 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:45 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:45.055+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:45.385 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:45 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:45.055+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.363678+0000 front 2024-04-01T01:51:17.363699+0000 (oldest deadline 2024-04-01T01:51:41.364127+0000) 2024-04-01T01:51:45.689 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:45 smithi081 podman[70515]: 2024-04-01 01:51:45.501050056 +0000 UTC m=+6.358924359 container cleanup a1392b6f479d7ad2a76e4b67593eab156ef272653866aa9c2ab944881e70691b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326) 2024-04-01T01:51:45.690 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:45 smithi081 bash[70515]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4 2024-04-01T01:51:45.690 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:45 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:45.344+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:45.690 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:45 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:45.344+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:45.748 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:45 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:45.414+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:45.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:45 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:45.414+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:45.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:45 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:45.385+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:45.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:45 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:45.385+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:51:46.078 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:45 smithi081 podman[70534]: 2024-04-01 01:51:45.745923961 +0000 UTC m=+0.664448016 container remove a1392b6f479d7ad2a76e4b67593eab156ef272653866aa9c2ab944881e70691b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-01T01:51:46.078 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:45 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:45.689+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:46.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:46 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:46.031+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:46.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:46 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:46.031+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:46.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:45 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:45.823+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:46.255 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:45 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:45.823+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:46.361 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:46 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:46.099+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:46.362 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:46 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:46.099+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.363678+0000 front 2024-04-01T01:51:17.363699+0000 (oldest deadline 2024-04-01T01:51:41.364127+0000) 2024-04-01T01:51:46.484 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:46 smithi081 podman[70624]: 2024-04-01 01:51:46.109306725 +0000 UTC m=+0.021863275 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:51:46.484 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:46 smithi081 podman[70624]: 2024-04-01 01:51:46.206250347 +0000 UTC m=+0.118806881 container create 5fe77a7907426f2ebc1a5e1085e5f9d014e24fc356bfae923594393746d7b9ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-04-01T01:51:46.485 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:46 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:46.323+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:46.485 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:46 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:46.323+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:46.748 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:46 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:46.361+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:46.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:46 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:46.361+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:51:46.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:46 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:46.376+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:46.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:46 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:46.376+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:46.780 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:46 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:46.705+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:46.780 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:46 smithi081 podman[70624]: 2024-04-01 01:51:46.484848993 +0000 UTC m=+0.397405543 container init 5fe77a7907426f2ebc1a5e1085e5f9d014e24fc356bfae923594393746d7b9ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-01T01:51:46.780 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:46 smithi081 podman[70624]: 2024-04-01 01:51:46.488172341 +0000 UTC m=+0.400728880 container start 5fe77a7907426f2ebc1a5e1085e5f9d014e24fc356bfae923594393746d7b9ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-01T01:51:46.780 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:46 smithi081 podman[70624]: 2024-04-01 01:51:46.58482288 +0000 UTC m=+0.497379427 container attach 5fe77a7907426f2ebc1a5e1085e5f9d014e24fc356bfae923594393746d7b9ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4-deactivate, RELEASE=squid-384f31f, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326) 2024-04-01T01:51:47.077 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:46 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:46.824+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:47.078 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:46 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:46.824+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:47.078 INFO:journalctl@ceph.osd.4.smithi081.stdout:Apr 01 01:51:46 smithi081 podman[70624]: 2024-04-01 01:51:46.78034439 +0000 UTC m=+0.692900928 container died 5fe77a7907426f2ebc1a5e1085e5f9d014e24fc356bfae923594393746d7b9ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-01T01:51:47.327 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:47 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:47.076+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:47.327 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:47 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:47.076+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.363678+0000 front 2024-04-01T01:51:17.363699+0000 (oldest deadline 2024-04-01T01:51:41.364127+0000) 2024-04-01T01:51:47.468 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:47 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:47.309+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:47.469 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:47 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:47.309+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:47.487 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.4.service' 2024-04-01T01:51:47.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:47 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:47.077+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:47.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:47 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:47.077+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:47.527 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:51:47.527 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-01T01:51:47.527 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-01T01:51:47.527 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.5 2024-04-01T01:51:47.749 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:47 smithi189 systemd[1]: Stopping Ceph osd.5 for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:51:47.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:47 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:47.327+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:47.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:47 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:47.327+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:47.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:47 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:47.356+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:47.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:47 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:47.356+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:51:48.078 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:47 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:47.743+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:48.089 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:47 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:47.827+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:48.089 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:47 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:47.827+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:48.249 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:47 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:47.903+0000 7fe72604c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:51:48.249 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:47 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:47.903+0000 7fe72604c700 -1 osd.5 95 *** Got signal Terminated *** 2024-04-01T01:51:48.249 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:47 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:47.903+0000 7fe72604c700 -1 osd.5 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-01T01:51:48.249 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:48 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:48.097+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:48.249 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:48 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:48.097+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.363678+0000 front 2024-04-01T01:51:17.363699+0000 (oldest deadline 2024-04-01T01:51:41.364127+0000) 2024-04-01T01:51:48.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:48 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:48.089+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:48.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:48 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:48.089+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:48.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:48 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:48.321+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:48.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:48 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:48.321+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:51:48.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:48 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:48.348+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:48.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:48 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:48.348+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:48.781 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:48 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:48.339+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:48.781 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:48 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:48.339+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:49.078 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:48 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:48.780+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:49.131 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:48 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:48.860+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:49.131 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:48 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:48.860+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:49.131 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:48 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:48.860+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:51:49.345 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:49 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:49.069+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:49.345 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:49 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:49.069+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.363678+0000 front 2024-04-01T01:51:17.363699+0000 (oldest deadline 2024-04-01T01:51:41.364127+0000) 2024-04-01T01:51:49.345 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:49 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:49.306+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:49.345 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:49 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:49.306+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:49.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:49 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:49.131+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:49.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:49 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:49.131+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:49.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:49 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:49.292+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:49.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:49 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:49.292+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:49.748 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:49 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:49.344+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:49.748 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:49 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:49.344+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:51:50.078 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:49 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:49.812+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:50.078 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:49 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:49.812+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:51:50.083 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:49 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:49.816+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:50.083 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:49 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:49.816+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:50.083 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:49 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:49.816+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:51:50.315 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:50 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:50.038+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:50.316 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:50 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:50.038+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.363678+0000 front 2024-04-01T01:51:17.363699+0000 (oldest deadline 2024-04-01T01:51:41.364127+0000) 2024-04-01T01:51:50.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:50 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:50.134+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:50.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:50 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:50.134+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:50.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:50 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:50.134+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.538452+0000 front 2024-04-01T01:51:27.538431+0000 (oldest deadline 2024-04-01T01:51:49.139174+0000) 2024-04-01T01:51:50.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:50 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:50.259+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:50.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:50 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:50.259+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:50.748 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:50 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:50.315+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:50.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:50 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:50.315+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:50.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:50 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:50.315+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:51:50.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:50 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:50.372+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:50.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:50 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:50.372+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:51:50.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:50 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:50.372+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:51:51.078 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:50 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:50.816+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:51.078 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:50 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:50.816+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:51:51.125 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:50 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:50.843+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:51.125 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:50 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:50.843+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:51.125 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:50 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:50.843+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:51:51.340 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:51 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:51.021+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:51.340 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:51 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:51.021+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.363678+0000 front 2024-04-01T01:51:17.363699+0000 (oldest deadline 2024-04-01T01:51:41.364127+0000) 2024-04-01T01:51:51.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:51 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:51.125+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:51.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:51 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:51.125+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:51.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:51 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:51.125+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.538452+0000 front 2024-04-01T01:51:27.538431+0000 (oldest deadline 2024-04-01T01:51:49.139174+0000) 2024-04-01T01:51:51.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:51 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:51.290+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:51.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:51 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:51.290+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:51.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:51 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:51.290+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:51:51.748 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:51 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:51.365+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:51.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:51 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:51.365+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:51.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:51 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:51.365+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:51:51.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:51 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:51.340+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:51.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:51 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:51.340+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:51:51.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:51 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:51.340+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:51:52.246 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:51 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:51.851+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:52.246 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:51 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:51.851+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:51:52.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:52 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:52.078+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:52.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:52 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:52.078+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:52.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:52 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:52.078+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.538452+0000 front 2024-04-01T01:51:27.538431+0000 (oldest deadline 2024-04-01T01:51:49.139174+0000) 2024-04-01T01:51:52.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:51 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:51.882+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:52.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:51 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:51.882+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:52.257 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:51 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:51.882+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:51:52.308 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:52 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:52.056+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.363023+0000 front 2024-04-01T01:51:10.362822+0000 (oldest deadline 2024-04-01T01:51:34.462892+0000) 2024-04-01T01:51:52.309 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:52 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:52.056+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.363678+0000 front 2024-04-01T01:51:17.363699+0000 (oldest deadline 2024-04-01T01:51:41.364127+0000) 2024-04-01T01:51:52.309 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:52 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5[47557]: 2024-04-01T01:51:52.056+0000 7fe722640700 -1 osd.5 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.966329+0000 front 2024-04-01T01:51:26.966349+0000 (oldest deadline 2024-04-01T01:51:51.666447+0000) 2024-04-01T01:51:52.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:52 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:52.245+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:52.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:52 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:52.245+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:52.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:52 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:52.245+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:51:52.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:52 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:52.331+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:52.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:52 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:52.331+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:52.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:52 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:52.331+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:51:52.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:52 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:52.308+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:52.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:52 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:52.308+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:51:52.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:52 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:52.308+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:51:53.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:53 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:53.040+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:53.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:53 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:53.040+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:53.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:53 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:53.040+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.538452+0000 front 2024-04-01T01:51:27.538431+0000 (oldest deadline 2024-04-01T01:51:49.139174+0000) 2024-04-01T01:51:53.255 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:52 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:52.842+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:53.255 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:52 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:52.842+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:53.255 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:52 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:52.842+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:51:53.282 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:52 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:52.892+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:53.282 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:52 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:52.892+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:51:53.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:53 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:53.280+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:53.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:53 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:53.280+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:53.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:53 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:53.280+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:51:53.749 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:53 smithi189 podman[72754]: 2024-04-01 01:51:53.570868874 +0000 UTC m=+5.813129895 container died 330634e99b31da626b3f6bc01c62489515c218385571f986330b662902eeb7b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2) 2024-04-01T01:51:53.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:53 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:53.355+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:53.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:53 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:53.355+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:53.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:53 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:53.355+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:51:53.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:53 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:53.357+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:53.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:53 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:53.357+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:51:53.750 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:53 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:53.357+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:51:54.058 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:53 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:53.797+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:54.059 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:53 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:53.797+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:54.059 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:53 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:53.797+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:51:54.238 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:53 smithi189 podman[72754]: 2024-04-01 01:51:53.985907938 +0000 UTC m=+6.228168955 container cleanup 330634e99b31da626b3f6bc01c62489515c218385571f986330b662902eeb7b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240326, ceph=True) 2024-04-01T01:51:54.239 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:53 smithi189 bash[72754]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5 2024-04-01T01:51:54.267 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:53 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:53.886+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:54.267 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:53 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:53.886+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:51:54.499 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:54 smithi189 podman[72773]: 2024-04-01 01:51:54.23930124 +0000 UTC m=+0.669829920 container remove 330634e99b31da626b3f6bc01c62489515c218385571f986330b662902eeb7b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-01T01:51:54.499 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:54 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:54.375+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:54.499 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:54 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:54.375+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:54.499 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:54 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:54.375+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:51:54.499 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:54 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:54.376+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:54.499 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:54 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:54.376+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:51:54.500 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:54 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:54.376+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:51:54.503 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:54 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:54.058+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:54.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:54 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:54.058+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:54.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:54 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:54.058+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.538452+0000 front 2024-04-01T01:51:27.538431+0000 (oldest deadline 2024-04-01T01:51:49.139174+0000) 2024-04-01T01:51:54.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:54 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:54.266+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:54.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:54 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:54.266+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:54.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:54 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:54.266+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:51:54.875 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:54 smithi189 podman[72861]: 2024-04-01 01:51:54.488988607 +0000 UTC m=+0.020864181 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:51:54.876 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:54 smithi189 podman[72861]: 2024-04-01 01:51:54.615292544 +0000 UTC m=+0.147168098 container create 5ac4bc0f23be2ed26ce68ca60c096ec3e695ad1776e6efdf1f32d0813b27381c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5-deactivate, RELEASE=squid-384f31f, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-01T01:51:54.876 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:54 smithi189 podman[72861]: 2024-04-01 01:51:54.876248209 +0000 UTC m=+0.408123778 container init 5ac4bc0f23be2ed26ce68ca60c096ec3e695ad1776e6efdf1f32d0813b27381c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-01T01:51:55.103 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:54 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:54.804+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:55.103 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:54 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:54.804+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:55.103 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:54 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:54.804+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:51:55.183 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:54 smithi189 podman[72861]: 2024-04-01 01:51:54.879552269 +0000 UTC m=+0.411427832 container start 5ac4bc0f23be2ed26ce68ca60c096ec3e695ad1776e6efdf1f32d0813b27381c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-01T01:51:55.184 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:54 smithi189 podman[72861]: 2024-04-01 01:51:54.984730336 +0000 UTC m=+0.516605906 container attach 5ac4bc0f23be2ed26ce68ca60c096ec3e695ad1776e6efdf1f32d0813b27381c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, io.buildah.version=1.33.5) 2024-04-01T01:51:55.272 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:54 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:54.890+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:55.272 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:54 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:54.890+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:51:55.487 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 01 01:51:55 smithi189 podman[72861]: 2024-04-01 01:51:55.184547713 +0000 UTC m=+0.716423285 container died 5ac4bc0f23be2ed26ce68ca60c096ec3e695ad1776e6efdf1f32d0813b27381c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-5-deactivate, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-384f31f) 2024-04-01T01:51:55.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:55 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:55.408+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:55.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:55 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:55.408+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:55.487 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:55 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:55.408+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:51:55.488 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:55 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:55.401+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:55.488 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:55 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:55.401+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:51:55.488 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:55 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:55.402+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:51:55.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:55 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:55.103+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:55.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:55 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:55.103+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:55.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:55 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:55.103+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.538452+0000 front 2024-04-01T01:51:27.538431+0000 (oldest deadline 2024-04-01T01:51:49.139174+0000) 2024-04-01T01:51:55.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:55 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:55.272+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:55.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:55 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:55.272+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:55.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:55 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:55.272+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:51:55.796 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.5.service' 2024-04-01T01:51:55.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:51:55.841 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-01T01:51:55.841 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-01T01:51:55.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.6 2024-04-01T01:51:56.148 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:55 smithi070 systemd[1]: Stopping Ceph osd.6 for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:51:56.149 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:55 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:55.840+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:56.149 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:55 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:55.840+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:56.149 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:55 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:55.840+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:51:56.293 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:55 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:55.934+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:56.293 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:55 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:55.934+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:51:56.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:56.147+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:56.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:56.147+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:56.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:56.147+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.538452+0000 front 2024-04-01T01:51:27.538431+0000 (oldest deadline 2024-04-01T01:51:49.139174+0000) 2024-04-01T01:51:56.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:56.153+0000 7f0a86823700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:51:56.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:56.153+0000 7f0a86823700 -1 osd.6 95 *** Got signal Terminated *** 2024-04-01T01:51:56.505 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:56.153+0000 7f0a86823700 -1 osd.6 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-01T01:51:56.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:56 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:56.292+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:56.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:56 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:56.292+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:56.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:56 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:56.292+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:51:56.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:56 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:56.390+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:56.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:56 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:56.390+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:51:56.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:56 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:56.390+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:51:56.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:56 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:56.443+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:56.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:56 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:56.443+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:56.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:56 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:56.443+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:51:57.136 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:56.850+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:57.136 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:56.850+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:57.137 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:56 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:56.850+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:51:57.328 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:56 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:56.907+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:57.328 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:56 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:56.907+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:51:57.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:57 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:57.136+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:57.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:57 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:57.136+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:57.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:57 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:57.136+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.538452+0000 front 2024-04-01T01:51:27.538431+0000 (oldest deadline 2024-04-01T01:51:49.139174+0000) 2024-04-01T01:51:57.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:57 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:57.430+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:57.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:57 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:57.430+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:51:57.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:57 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:57.430+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:51:57.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:57 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:57.437+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:57.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:57 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:57.437+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:57.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:57 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:57.437+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:51:57.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:57 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:57.330+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:57.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:57 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:57.330+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:57.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:57 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:57.330+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:51:57.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:57 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:57.330+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:51:58.148 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:57 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:57.829+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:58.148 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:57 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:57.829+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:58.148 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:57 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:57.829+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:51:58.328 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:57 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:57.909+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:58.328 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:57 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:57.909+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:51:58.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:58 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:58.147+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:58.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:58 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:58.147+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:58.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:58 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:58.147+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.538452+0000 front 2024-04-01T01:51:27.538431+0000 (oldest deadline 2024-04-01T01:51:49.139174+0000) 2024-04-01T01:51:58.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:58 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:58.147+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:36.041070+0000 front 2024-04-01T01:51:36.041038+0000 (oldest deadline 2024-04-01T01:51:57.741174+0000) 2024-04-01T01:51:58.748 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:58 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:58.395+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:58.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:58 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:58.395+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:51:58.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:58 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:58.395+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:51:58.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:58 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:58.424+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:58.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:58 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:58.424+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:58.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:58 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:58.424+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:51:58.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:58 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:58.377+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:58.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:58 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:58.377+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:58.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:58 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:58.377+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:51:58.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:58 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:58.377+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:51:59.123 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:58 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:58.786+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:51:59.123 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:58 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:58.787+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:51:59.123 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:58 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:58.787+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:51:59.123 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:58 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:58.787+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:51:59.328 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:58 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:58.923+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:51:59.328 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:58 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:58.923+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:51:59.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:59 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:59.123+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:51:59.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:59 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:59.123+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:51:59.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:59 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:59.123+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.538452+0000 front 2024-04-01T01:51:27.538431+0000 (oldest deadline 2024-04-01T01:51:49.139174+0000) 2024-04-01T01:51:59.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:51:59 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:51:59.123+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:36.041070+0000 front 2024-04-01T01:51:36.041038+0000 (oldest deadline 2024-04-01T01:51:57.741174+0000) 2024-04-01T01:51:59.748 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:59 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:59.363+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:51:59.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:59 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:59.363+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:51:59.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:51:59 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:51:59.363+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:51:59.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:59 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:59.452+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:51:59.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:59 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:59.452+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:51:59.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:51:59 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:51:59.452+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:51:59.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:59 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:59.381+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:51:59.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:59 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:59.381+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:51:59.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:59 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:59.381+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:51:59.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:51:59 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:51:59.381+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:00.097 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:59 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:59.811+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:00.097 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:59 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:59.811+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:00.097 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:59 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:59.811+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:00.097 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:51:59 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:51:59.811+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:00.328 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:59 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:59.876+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:52:00.328 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:51:59 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:51:59.876+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:52:00.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:00 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:52:00.097+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:52:00.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:00 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:52:00.097+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:52:00.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:00 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:52:00.097+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.538452+0000 front 2024-04-01T01:51:27.538431+0000 (oldest deadline 2024-04-01T01:51:49.139174+0000) 2024-04-01T01:52:00.504 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:00 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:52:00.097+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:36.041070+0000 front 2024-04-01T01:51:36.041038+0000 (oldest deadline 2024-04-01T01:51:57.741174+0000) 2024-04-01T01:52:00.748 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:00 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:00.318+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:00.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:00 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:00.318+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:00.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:00 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:00.318+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:00.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:00 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:00.495+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:00.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:00 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:00.495+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:00.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:00 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:00.495+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:00.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:00 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:00.495+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:00.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:00 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:00.353+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:00.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:00 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:00.353+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:00.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:00 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:00.353+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:00.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:00 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:00.353+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:01.082 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:00 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:00.817+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:01.083 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:00 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:00.817+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:01.083 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:00 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:00.817+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:01.083 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:00 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:00.817+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:01.328 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:00 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:00.848+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:52:01.328 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:00 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:00.848+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:52:01.328 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:00 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:00.848+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.681457+0000 front 2024-04-01T01:51:34.681354+0000 (oldest deadline 2024-04-01T01:51:59.981321+0000) 2024-04-01T01:52:01.376 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:01 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:52:01.081+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:10.134621+0000 front 2024-04-01T01:51:10.134645+0000 (oldest deadline 2024-04-01T01:51:31.834830+0000) 2024-04-01T01:52:01.377 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:01 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:52:01.082+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.735921+0000 front 2024-04-01T01:51:17.735895+0000 (oldest deadline 2024-04-01T01:51:42.436045+0000) 2024-04-01T01:52:01.377 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:01 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:52:01.082+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.538452+0000 front 2024-04-01T01:51:27.538431+0000 (oldest deadline 2024-04-01T01:51:49.139174+0000) 2024-04-01T01:52:01.377 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:01 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6[60776]: 2024-04-01T01:52:01.082+0000 7f0a82e17700 -1 osd.6 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:36.041070+0000 front 2024-04-01T01:51:36.041038+0000 (oldest deadline 2024-04-01T01:51:57.741174+0000) 2024-04-01T01:52:01.748 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:01 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:01.511+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:01.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:01 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:01.511+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:01.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:01 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:01.511+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:01.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:01 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:01.511+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:01.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:01 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:01.334+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:01.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:01 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:01.334+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:01.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:01 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:01.334+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:01.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:01 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:01.334+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:01.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:01 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:01.346+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:01.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:01 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:01.346+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:01.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:01 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:01.346+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:01.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:01 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:01.346+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:01.960 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:01 smithi070 podman[118387]: 2024-04-01 01:52:01.66494898 +0000 UTC m=+5.631015046 container died 8b179eb1de437939701dc292f0eca6fcd16b635726da4dca9acec8f96891e399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-01T01:52:01.960 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:01 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:01.855+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:01.960 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:01 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:01.855+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:01.960 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:01 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:01.855+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:01.960 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:01 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:01.855+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:02.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:02 smithi070 podman[118387]: 2024-04-01 01:52:02.095466651 +0000 UTC m=+6.061532716 container cleanup 8b179eb1de437939701dc292f0eca6fcd16b635726da4dca9acec8f96891e399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-01T01:52:02.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:02 smithi070 bash[118387]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6 2024-04-01T01:52:02.328 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:01 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:01.891+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:52:02.328 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:01 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:01.891+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:52:02.328 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:01 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:01.891+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.681457+0000 front 2024-04-01T01:51:34.681354+0000 (oldest deadline 2024-04-01T01:51:59.981321+0000) 2024-04-01T01:52:02.710 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:02 smithi070 podman[118413]: 2024-04-01 01:52:02.280176248 +0000 UTC m=+0.610174640 container remove 8b179eb1de437939701dc292f0eca6fcd16b635726da4dca9acec8f96891e399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-01T01:52:02.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:02 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:02.338+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:02.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:02 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:02.338+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:02.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:02 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:02.338+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:02.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:02 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:02.338+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:02.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:02 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:02.538+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:02.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:02 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:02.538+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:02.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:02 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:02.538+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:02.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:02 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:02.538+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:02.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:02 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:02.355+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:02.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:02 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:02.355+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:02.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:02 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:02.355+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:02.828 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:02 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:02.355+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:02.983 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:02 smithi070 podman[118508]: 2024-04-01 01:52:02.614131232 +0000 UTC m=+0.018052178 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:52:02.983 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:02 smithi070 podman[118508]: 2024-04-01 01:52:02.739466564 +0000 UTC m=+0.143387490 container create 65248fd062ec78347570322e6542c828aaf103122741b068d6e650fc8cecc34b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-01T01:52:02.983 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:02 smithi070 podman[118508]: 2024-04-01 01:52:02.983817202 +0000 UTC m=+0.387738143 container init 65248fd062ec78347570322e6542c828aaf103122741b068d6e650fc8cecc34b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-01T01:52:02.983 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:02 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:02.846+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:02.983 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:02 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:02.846+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:02.983 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:02 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:02.846+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:02.984 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:02 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:02.846+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:03.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:02 smithi070 podman[118508]: 2024-04-01 01:52:02.986929773 +0000 UTC m=+0.390850703 container start 65248fd062ec78347570322e6542c828aaf103122741b068d6e650fc8cecc34b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-01T01:52:03.254 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:03 smithi070 podman[118508]: 2024-04-01 01:52:03.067212532 +0000 UTC m=+0.471133472 container attach 65248fd062ec78347570322e6542c828aaf103122741b068d6e650fc8cecc34b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-384f31f, ceph=True) 2024-04-01T01:52:03.325 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:02 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:02.914+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:52:03.325 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:02 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:02.914+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:52:03.325 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:02 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:02.914+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.681457+0000 front 2024-04-01T01:51:34.681354+0000 (oldest deadline 2024-04-01T01:51:59.981321+0000) 2024-04-01T01:52:03.552 INFO:journalctl@ceph.osd.6.smithi070.stdout:Apr 01 01:52:03 smithi070 podman[118508]: 2024-04-01 01:52:03.274344775 +0000 UTC m=+0.678265720 container died 65248fd062ec78347570322e6542c828aaf103122741b068d6e650fc8cecc34b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-01T01:52:03.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:03 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:03.325+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:03.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:03 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:03.325+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:03.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:03 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:03.325+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:03.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:03 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:03.325+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:03.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:03 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:03.535+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:03.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:03 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:03.535+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:03.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:03 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:03.535+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:03.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:03 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:03.535+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:03.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:03 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:03.345+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:03.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:03 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:03.345+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:03.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:03 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:03.345+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:03.750 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:03 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:03.345+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:03.832 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.6.service' 2024-04-01T01:52:03.873 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:52:03.873 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-01T01:52:03.873 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-01T01:52:03.874 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.7 2024-04-01T01:52:04.217 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:03 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:03.868+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:52:04.217 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:03 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:03.868+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:52:04.217 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:03 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:03.868+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.681457+0000 front 2024-04-01T01:51:34.681354+0000 (oldest deadline 2024-04-01T01:51:59.981321+0000) 2024-04-01T01:52:04.217 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:03 smithi081 systemd[1]: Stopping Ceph osd.7 for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:52:04.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:03 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:03.858+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:04.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:03 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:03.858+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:04.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:03 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:03.858+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:04.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:03 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:03.858+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:04.578 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:04 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:04.217+0000 7f2e5b2a8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:52:04.578 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:04 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:04.217+0000 7f2e5b2a8700 -1 osd.7 95 *** Got signal Terminated *** 2024-04-01T01:52:04.578 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:04 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:04.217+0000 7f2e5b2a8700 -1 osd.7 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-01T01:52:04.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:04 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:04.310+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:04.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:04 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:04.310+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:04.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:04 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:04.310+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:04.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:04 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:04.310+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:04.748 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:04 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:04.512+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:04.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:04 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:04.512+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:04.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:04 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:04.512+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:04.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:04 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:04.512+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:04.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:04 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:04.355+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:04.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:04 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:04.355+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:04.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:04 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:04.355+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:04.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:04 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:04.355+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:05.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:04 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:04.844+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:05.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:04 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:04.844+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:05.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:04 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:04.844+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:05.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:04 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:04.844+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:05.286 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:04 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:04.889+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:52:05.286 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:04 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:04.889+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:52:05.286 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:04 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:04.889+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.681457+0000 front 2024-04-01T01:51:34.681354+0000 (oldest deadline 2024-04-01T01:51:59.981321+0000) 2024-04-01T01:52:05.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:05 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:05.286+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:05.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:05 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:05.286+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:05.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:05 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:05.286+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:05.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:05 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:05.286+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:05.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:05 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:05.286+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:05.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:05 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:05.384+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:05.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:05 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:05.384+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:05.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:05 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:05.384+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:05.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:05 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:05.384+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:05.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:05 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:05.531+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:05.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:05 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:05.531+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:05.750 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:05 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:05.531+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:05.750 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:05 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:05.531+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:06.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:05 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:05.891+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:06.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:05 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:05.891+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:06.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:05 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:05.891+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:06.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:05 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:05.891+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:06.322 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:05 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:05.849+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:52:06.322 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:05 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:05.849+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:52:06.322 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:05 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:05.849+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.681457+0000 front 2024-04-01T01:51:34.681354+0000 (oldest deadline 2024-04-01T01:51:59.981321+0000) 2024-04-01T01:52:06.322 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:05 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:05.849+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:44.081991+0000 front 2024-04-01T01:51:44.081975+0000 (oldest deadline 2024-04-01T01:52:05.182083+0000) 2024-04-01T01:52:06.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:06 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:06.322+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:06.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:06 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:06.322+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:06.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:06 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:06.322+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:06.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:06 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:06.322+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:06.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:06 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:06.322+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:06.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:06 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:06.420+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:06.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:06 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:06.420+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:06.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:06 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:06.420+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:06.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:06 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:06.420+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:06.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:06 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:06.500+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:06.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:06 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:06.500+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:06.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:06 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:06.500+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:06.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:06 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:06.500+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:06.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:06 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:06.500+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:07.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:06 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:06.902+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:07.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:06 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:06.902+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:07.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:06 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:06.902+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:07.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:06 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:06.902+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:07.293 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:06 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:06.879+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:52:07.293 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:06 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:06.879+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:52:07.293 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:06 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:06.879+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.681457+0000 front 2024-04-01T01:51:34.681354+0000 (oldest deadline 2024-04-01T01:51:59.981321+0000) 2024-04-01T01:52:07.293 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:06 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:06.879+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:44.081991+0000 front 2024-04-01T01:51:44.081975+0000 (oldest deadline 2024-04-01T01:52:05.182083+0000) 2024-04-01T01:52:07.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:07 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:07.293+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:07.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:07 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:07.293+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:07.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:07 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:07.293+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:07.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:07 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:07.293+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:07.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:07 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:07.293+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:07.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:07 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:07.463+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:07.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:07 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:07.463+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:07.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:07 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:07.463+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:07.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:07 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:07.463+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:07.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:07 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:07.463+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:07.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:07 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:07.384+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:07.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:07 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:07.384+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:07.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:07 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:07.384+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:07.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:07 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:07.384+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:07.750 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:07 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:07.384+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.733153+0000 front 2024-04-01T01:51:42.733184+0000 (oldest deadline 2024-04-01T01:52:06.833273+0000) 2024-04-01T01:52:08.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:07 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:07.908+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:08.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:07 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:07.908+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:08.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:07 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:07.908+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:08.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:07 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:07.908+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:08.255 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:07 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:07.908+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:08.319 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:07 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:07.832+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:52:08.319 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:07 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:07.832+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:52:08.319 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:07 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:07.832+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.681457+0000 front 2024-04-01T01:51:34.681354+0000 (oldest deadline 2024-04-01T01:51:59.981321+0000) 2024-04-01T01:52:08.320 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:07 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:07.832+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:44.081991+0000 front 2024-04-01T01:51:44.081975+0000 (oldest deadline 2024-04-01T01:52:05.182083+0000) 2024-04-01T01:52:08.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:08 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:08.320+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:08.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:08 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:08.320+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:08.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:08 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:08.320+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:08.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:08 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:08.320+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:08.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:08 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:08.320+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:08.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:08 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:08.464+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:08.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:08 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:08.464+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:08.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:08 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:08.464+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:08.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:08 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:08.464+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:08.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:08 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:08.464+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:08.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:08 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:08.409+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:08.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:08 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:08.409+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:08.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:08 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:08.409+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:08.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:08 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:08.409+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:08.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:08 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:08.409+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.733153+0000 front 2024-04-01T01:51:42.733184+0000 (oldest deadline 2024-04-01T01:52:06.833273+0000) 2024-04-01T01:52:09.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:08 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:08.937+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:09.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:08 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:08.937+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:09.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:08 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:08.937+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:09.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:08 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:08.937+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:09.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:08 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:08.937+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:09.303 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:08 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:08.857+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:15.578336+0000 front 2024-04-01T01:51:15.578316+0000 (oldest deadline 2024-04-01T01:51:41.478449+0000) 2024-04-01T01:52:09.303 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:08 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:08.857+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.879837+0000 front 2024-04-01T01:51:27.879823+0000 (oldest deadline 2024-04-01T01:51:48.979895+0000) 2024-04-01T01:52:09.303 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:08 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:08.857+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.681457+0000 front 2024-04-01T01:51:34.681354+0000 (oldest deadline 2024-04-01T01:51:59.981321+0000) 2024-04-01T01:52:09.303 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:08 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7[53161]: 2024-04-01T01:52:08.857+0000 7f2e53894700 -1 osd.7 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:44.081991+0000 front 2024-04-01T01:51:44.081975+0000 (oldest deadline 2024-04-01T01:52:05.182083+0000) 2024-04-01T01:52:09.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:09 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:09.303+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:09.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:09 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:09.303+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:09.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:09 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:09.303+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:09.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:09 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:09.303+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:09.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:09 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:09.303+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:09.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:09 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:09.453+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:09.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:09 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:09.453+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:09.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:09 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:09.453+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:09.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:09 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:09.453+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:09.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:09 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:09.453+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.733153+0000 front 2024-04-01T01:51:42.733184+0000 (oldest deadline 2024-04-01T01:52:06.833273+0000) 2024-04-01T01:52:09.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:09 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:09.505+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:09.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:09 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:09.505+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:09.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:09 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:09.505+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:09.750 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:09 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:09.505+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:09.750 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:09 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:09.505+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:10.078 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:09 smithi081 podman[70729]: 2024-04-01 01:52:09.79166223 +0000 UTC m=+5.710556769 container died bdc99f5b0b6ca2937fae611c97d2e8800f4c645a2c0db572f325b4187e249e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-04-01T01:52:10.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:09 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:09.915+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:10.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:09 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:09.915+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:10.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:09 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:09.915+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:10.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:09 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:09.915+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:10.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:09 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:09.915+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:10.453 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:10 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:10.256+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:10.453 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:10 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:10.256+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:10.453 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:10 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:10.256+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:10.453 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:10 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:10.256+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:10.453 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:10 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:10.256+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:10.453 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:10 smithi081 podman[70729]: 2024-04-01 01:52:10.210020134 +0000 UTC m=+6.128914685 container cleanup bdc99f5b0b6ca2937fae611c97d2e8800f4c645a2c0db572f325b4187e249e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-01T01:52:10.454 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:10 smithi081 bash[70729]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7 2024-04-01T01:52:10.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:10 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:10.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:10.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:10 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:10.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:10.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:10 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:10.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:10.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:10 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:10.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:10.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:10 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:10.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:10.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:10 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:10.470+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:10.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:10 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:10.470+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:10.750 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:10 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:10.470+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:10.750 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:10 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:10.470+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:10.750 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:10 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:10.470+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.733153+0000 front 2024-04-01T01:51:42.733184+0000 (oldest deadline 2024-04-01T01:52:06.833273+0000) 2024-04-01T01:52:10.787 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:10 smithi081 podman[70748]: 2024-04-01 01:52:10.453318666 +0000 UTC m=+0.657461715 container remove bdc99f5b0b6ca2937fae611c97d2e8800f4c645a2c0db572f325b4187e249e28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-01T01:52:11.065 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:10 smithi081 podman[70840]: 2024-04-01 01:52:10.78821581 +0000 UTC m=+0.107844305 container create 9c492f7b84368b89ea504229fde7094f687efd508ab1b0c4ea4d8763c981a9f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-384f31f, org.label-schema.license=GPLv2) 2024-04-01T01:52:11.065 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:10 smithi081 podman[70840]: 2024-04-01 01:52:10.700914459 +0000 UTC m=+0.020542970 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:52:11.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:10 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:10.934+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:11.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:10 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:10.934+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:11.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:10 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:10.934+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:11.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:10 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:10.934+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:11.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:10 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:10.934+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:11.328 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:11 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:11.257+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:11.328 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:11 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:11.257+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:11.328 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:11 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:11.257+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:11.328 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:11 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:11.257+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:11.328 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:11 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:11.257+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:11.329 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:11 smithi081 podman[70840]: 2024-04-01 01:52:11.065665573 +0000 UTC m=+0.385294080 container init 9c492f7b84368b89ea504229fde7094f687efd508ab1b0c4ea4d8763c981a9f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, GIT_CLEAN=True, org.label-schema.build-date=20240326) 2024-04-01T01:52:11.329 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:11 smithi081 podman[70840]: 2024-04-01 01:52:11.069153476 +0000 UTC m=+0.388781971 container start 9c492f7b84368b89ea504229fde7094f687efd508ab1b0c4ea4d8763c981a9f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-01T01:52:11.329 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:11 smithi081 podman[70840]: 2024-04-01 01:52:11.17425428 +0000 UTC m=+0.493882799 container attach 9c492f7b84368b89ea504229fde7094f687efd508ab1b0c4ea4d8763c981a9f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-01T01:52:11.710 INFO:journalctl@ceph.osd.7.smithi081.stdout:Apr 01 01:52:11 smithi081 podman[70840]: 2024-04-01 01:52:11.3784411 +0000 UTC m=+0.698069594 container died 9c492f7b84368b89ea504229fde7094f687efd508ab1b0c4ea4d8763c981a9f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-7-deactivate, RELEASE=squid-384f31f, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-01T01:52:11.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:11 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:11.420+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:11.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:11 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:11.420+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:11.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:11 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:11.420+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:11.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:11 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:11.420+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:11.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:11 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:11.420+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:11.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:11 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:11.437+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:11.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:11 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:11.437+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:11.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:11 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:11.437+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:11.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:11 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:11.437+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:11.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:11 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:11.437+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.733153+0000 front 2024-04-01T01:51:42.733184+0000 (oldest deadline 2024-04-01T01:52:06.833273+0000) 2024-04-01T01:52:12.024 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.7.service' 2024-04-01T01:52:12.065 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:52:12.065 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-01T01:52:12.065 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-01T01:52:12.065 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.8 2024-04-01T01:52:12.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:11 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:11.905+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:12.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:11 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:11.905+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:12.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:11 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:11.905+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:12.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:11 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:11.905+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:12.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:11 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:11.905+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:12.328 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:12 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:12.251+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:12.328 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:12 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:12.251+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:12.328 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:12 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:12.251+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:12.329 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:12 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:12.251+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:12.329 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:12 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:12.251+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:12.425 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:12 smithi189 systemd[1]: Stopping Ceph osd.8 for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:52:12.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:12 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:12.444+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:12.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:12 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:12.444+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:12.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:12 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:12.444+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:12.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:12 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:12.444+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:12.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:12 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:12.444+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:12.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:12 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:12.424+0000 7fd20bca0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:52:12.750 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:12 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:12.424+0000 7fd20bca0700 -1 osd.8 95 *** Got signal Terminated *** 2024-04-01T01:52:12.750 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:12 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:12.424+0000 7fd20bca0700 -1 osd.8 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-01T01:52:12.750 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:12 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:12.438+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:12.750 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:12 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:12.438+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:12.750 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:12 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:12.438+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:12.750 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:12 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:12.438+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:12.750 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:12 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:12.438+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.733153+0000 front 2024-04-01T01:51:42.733184+0000 (oldest deadline 2024-04-01T01:52:06.833273+0000) 2024-04-01T01:52:13.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:12 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:12.945+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:13.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:12 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:12.945+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:13.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:12 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:12.945+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:13.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:12 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:12.945+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:13.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:12 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:12.945+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:13.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:13 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:13.258+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:13.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:13 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:13.258+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:13.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:13 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:13.258+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:13.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:13 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:13.258+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:13.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:13 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:13.258+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:13.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:13 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:13.390+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:13.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:13 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:13.390+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:13.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:13 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:13.390+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:13.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:13 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:13.390+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:13.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:13 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:13.390+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.733153+0000 front 2024-04-01T01:51:42.733184+0000 (oldest deadline 2024-04-01T01:52:06.833273+0000) 2024-04-01T01:52:13.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:13 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:13.465+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:13.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:13 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:13.465+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:13.750 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:13 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:13.465+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:13.750 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:13 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:13.465+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:13.750 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:13 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:13.465+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:14.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:13 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:13.921+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:14.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:13 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:13.921+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:14.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:13 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:13.921+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:14.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:13 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:13.921+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:14.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:13 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:13.921+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:14.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:14 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:14.240+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:14.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:14 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:14.240+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:14.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:14 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:14.240+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:14.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:14 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:14.240+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:14.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:14 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:14.240+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:14.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:14 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:14.241+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:14.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:14 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:14.367+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:14.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:14 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:14.367+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:14.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:14 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:14.367+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:14.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:14 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:14.367+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:14.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:14 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:14.367+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.733153+0000 front 2024-04-01T01:51:42.733184+0000 (oldest deadline 2024-04-01T01:52:06.833273+0000) 2024-04-01T01:52:14.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:14 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:14.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:14.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:14 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:14.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:14.750 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:14 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:14.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:14.750 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:14 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:14.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:14.750 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:14 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:14.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:14.750 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:14 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:14.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:15.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:14 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:14.872+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:15.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:14 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:14.872+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:15.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:14 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:14.872+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:15.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:14 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:14.872+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:15.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:14 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:14.872+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:15.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:14 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:14.872+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.614694+0000 front 2024-04-01T01:51:48.614602+0000 (oldest deadline 2024-04-01T01:52:14.514564+0000) 2024-04-01T01:52:15.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:15 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:15.200+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:15.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:15 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:15.200+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:15.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:15 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:15.200+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:15.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:15 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:15.200+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:15.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:15 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:15.200+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:15.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:15 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:15.200+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:15.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:15 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:15.468+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:15.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:15 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:15.468+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:15.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:15 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:15.468+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:15.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:15 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:15.468+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:15.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:15 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:15.468+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:15.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:15 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:15.468+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:15.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:15 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:15.324+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:15.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:15 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:15.324+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:15.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:15 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:15.324+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:15.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:15 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:15.324+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:15.750 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:15 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:15.324+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.733153+0000 front 2024-04-01T01:51:42.733184+0000 (oldest deadline 2024-04-01T01:52:06.833273+0000) 2024-04-01T01:52:16.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:15 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:15.881+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:16.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:15 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:15.881+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:16.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:15 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:15.881+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:16.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:15 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:15.881+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:16.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:15 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:15.881+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:16.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:15 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:15.881+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.614694+0000 front 2024-04-01T01:51:48.614602+0000 (oldest deadline 2024-04-01T01:52:14.514564+0000) 2024-04-01T01:52:16.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:16 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:16.232+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:16.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:16 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:16.232+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:16.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:16 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:16.232+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:16.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:16 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:16.232+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:16.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:16 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:16.232+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:16.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:16 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:16.232+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:16.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:16 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:16.517+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:16.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:16 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:16.517+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:16.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:16 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:16.517+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:16.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:16 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:16.517+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:16.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:16 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:16.517+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:16.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:16 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:16.517+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:16.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:16 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:16.351+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:16.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:16 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:16.351+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:16.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:16 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:16.351+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:16.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:16 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:16.351+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:16.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:16 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:16.351+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.733153+0000 front 2024-04-01T01:51:42.733184+0000 (oldest deadline 2024-04-01T01:52:06.833273+0000) 2024-04-01T01:52:17.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:16 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:16.907+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:17.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:16 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:16.907+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:17.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:16 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:16.907+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:17.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:16 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:16.907+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:17.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:16 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:16.907+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:17.255 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:16 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:16.907+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.614694+0000 front 2024-04-01T01:51:48.614602+0000 (oldest deadline 2024-04-01T01:52:14.514564+0000) 2024-04-01T01:52:17.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:17 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:17.200+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:17.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:17 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:17.200+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:17.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:17 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:17.200+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:17.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:17 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:17.200+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:17.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:17 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:17.200+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:17.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:17 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:17.200+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:17.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:17 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:17.478+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:17.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:17 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:17.478+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:17.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:17 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:17.478+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:17.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:17 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:17.478+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:17.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:17 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:17.478+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:17.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:17 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:17.478+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:17.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:17 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:17.379+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:08.729476+0000 front 2024-04-01T01:51:08.729461+0000 (oldest deadline 2024-04-01T01:51:32.829421+0000) 2024-04-01T01:52:17.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:17 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:17.379+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.130443+0000 front 2024-04-01T01:51:18.130492+0000 (oldest deadline 2024-04-01T01:51:44.030478+0000) 2024-04-01T01:52:17.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:17 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:17.379+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.531363+0000 front 2024-04-01T01:51:27.531389+0000 (oldest deadline 2024-04-01T01:51:49.831514+0000) 2024-04-01T01:52:17.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:17 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:17.379+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.732558+0000 front 2024-04-01T01:51:35.732497+0000 (oldest deadline 2024-04-01T01:52:00.432383+0000) 2024-04-01T01:52:17.750 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:17 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:17.379+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.733153+0000 front 2024-04-01T01:51:42.733184+0000 (oldest deadline 2024-04-01T01:52:06.833273+0000) 2024-04-01T01:52:17.750 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:17 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8[54555]: 2024-04-01T01:52:17.379+0000 7fd208294700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:51.534088+0000 front 2024-04-01T01:51:51.534105+0000 (oldest deadline 2024-04-01T01:52:16.834200+0000) 2024-04-01T01:52:18.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:17 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:17.915+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:18.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:17 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:17.915+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:18.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:17 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:17.915+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:18.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:17 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:17.915+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:18.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:17 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:17.915+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:18.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:17 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:17.915+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.614694+0000 front 2024-04-01T01:51:48.614602+0000 (oldest deadline 2024-04-01T01:52:14.514564+0000) 2024-04-01T01:52:18.432 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:18 smithi189 podman[72963]: 2024-04-01 01:52:18.162452438 +0000 UTC m=+5.852370038 container died 0b8ebf409a264e02cc7752e68eac4dbece70e6322c1691e2d26d81f64f63fd5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-01T01:52:18.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:18 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:18.162+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:18.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:18 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:18.162+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:18.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:18 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:18.162+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:18.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:18 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:18.162+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:18.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:18 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:18.162+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:18.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:18 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:18.162+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:18.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:18 smithi189 podman[72963]: 2024-04-01 01:52:18.574720456 +0000 UTC m=+6.264638060 container cleanup 0b8ebf409a264e02cc7752e68eac4dbece70e6322c1691e2d26d81f64f63fd5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-01T01:52:18.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:18 smithi189 bash[72963]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8 2024-04-01T01:52:18.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:18 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:18.432+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:18.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:18 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:18.432+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:18.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:18 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:18.432+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:18.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:18 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:18.432+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:18.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:18 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:18.432+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:18.750 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:18 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:18.432+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:19.201 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:18 smithi189 podman[72994]: 2024-04-01 01:52:18.834635984 +0000 UTC m=+0.672168151 container remove 0b8ebf409a264e02cc7752e68eac4dbece70e6322c1691e2d26d81f64f63fd5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.vendor=CentOS) 2024-04-01T01:52:19.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:18 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:18.936+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:19.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:18 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:18.936+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:19.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:18 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:18.936+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:19.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:18 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:18.936+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:19.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:18 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:18.936+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:19.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:18 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:18.936+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.614694+0000 front 2024-04-01T01:51:48.614602+0000 (oldest deadline 2024-04-01T01:52:14.514564+0000) 2024-04-01T01:52:19.496 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:19 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:19.435+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:19.496 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:19 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:19.435+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:19.497 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:19 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:19.435+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:19.497 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:19 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:19.435+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:19.497 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:19 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:19.435+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:19.497 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:19 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:19.435+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:19.497 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:19 smithi189 podman[73080]: 2024-04-01 01:52:19.104755717 +0000 UTC m=+0.022451722 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:52:19.497 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:19 smithi189 podman[73080]: 2024-04-01 01:52:19.23708897 +0000 UTC m=+0.154784957 container create 2b784e3cedab2d73e8370d8cec5925c9812710fb91f1871ec27770b4b7426cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-01T01:52:19.497 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:19 smithi189 podman[73080]: 2024-04-01 01:52:19.49700006 +0000 UTC m=+0.414696052 container init 2b784e3cedab2d73e8370d8cec5925c9812710fb91f1871ec27770b4b7426cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.build-date=20240326) 2024-04-01T01:52:19.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:19 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:19.173+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:19.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:19 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:19.173+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:19.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:19 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:19.173+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:19.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:19 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:19.173+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:19.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:19 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:19.173+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:19.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:19 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:19.173+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:19.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:19 smithi189 podman[73080]: 2024-04-01 01:52:19.502342032 +0000 UTC m=+0.420038017 container start 2b784e3cedab2d73e8370d8cec5925c9812710fb91f1871ec27770b4b7426cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5) 2024-04-01T01:52:19.749 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:19 smithi189 podman[73080]: 2024-04-01 01:52:19.597001615 +0000 UTC m=+0.514697601 container attach 2b784e3cedab2d73e8370d8cec5925c9812710fb91f1871ec27770b4b7426cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-01T01:52:20.200 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 01 01:52:19 smithi189 podman[73080]: 2024-04-01 01:52:19.835573934 +0000 UTC m=+0.753269913 container died 2b784e3cedab2d73e8370d8cec5925c9812710fb91f1871ec27770b4b7426cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-8-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-01T01:52:20.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:19 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:19.943+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:20.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:19 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:19.943+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:20.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:19 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:19.943+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:20.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:19 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:19.943+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:20.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:19 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:19.943+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:20.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:19 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:19.943+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.614694+0000 front 2024-04-01T01:51:48.614602+0000 (oldest deadline 2024-04-01T01:52:14.514564+0000) 2024-04-01T01:52:20.478 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.8.service' 2024-04-01T01:52:20.479 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:20 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:20.207+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:20.479 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:20 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:20.207+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:20.479 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:20 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:20.207+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:20.479 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:20 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:20.207+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:20.479 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:20 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:20.207+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:20.480 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:20 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:20.207+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:20.521 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:52:20.522 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-01T01:52:20.522 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-01T01:52:20.522 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.9 2024-04-01T01:52:20.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:20 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:20.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:20.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:20 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:20.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:20.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:20 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:20.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:20.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:20 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:20.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:20.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:20 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:20.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:20.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:20 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:20.459+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:20.870 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:20 smithi070 systemd[1]: Stopping Ceph osd.9 for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:52:21.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:20 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:20.870+0000 7fcf344fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:52:21.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:20 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:20.870+0000 7fcf344fb700 -1 osd.9 95 *** Got signal Terminated *** 2024-04-01T01:52:21.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:20 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:20.870+0000 7fcf344fb700 -1 osd.9 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-01T01:52:21.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:20 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:20.898+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:21.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:20 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:20.898+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:21.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:20 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:20.898+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:21.255 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:20 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:20.898+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:21.255 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:20 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:20.898+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:21.255 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:20 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:20.898+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.614694+0000 front 2024-04-01T01:51:48.614602+0000 (oldest deadline 2024-04-01T01:52:14.514564+0000) 2024-04-01T01:52:21.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:21 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:21.219+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:21.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:21 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:21.219+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:21.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:21 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:21.219+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:21.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:21 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:21.219+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:21.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:21 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:21.219+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:21.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:21 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:21.219+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:21.748 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:21 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:21.488+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:21.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:21 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:21.488+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:21.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:21 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:21.488+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:21.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:21 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:21.488+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:21.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:21 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:21.488+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:21.749 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:21 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:21.488+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:22.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:21 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:21.920+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:22.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:21 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:21.920+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:22.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:21 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:21.920+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:22.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:21 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:21.920+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:22.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:21 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:21.920+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:22.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:21 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:21.920+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.614694+0000 front 2024-04-01T01:51:48.614602+0000 (oldest deadline 2024-04-01T01:52:14.514564+0000) 2024-04-01T01:52:22.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:22 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:22.208+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:22.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:22 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:22.208+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:22.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:22 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:22.208+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:22.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:22 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:22.208+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:22.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:22 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:22.208+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:22.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:22 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:22.208+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:22.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:22 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:22.522+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:22.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:22 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:22.522+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:22.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:22 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:22.522+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:22.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:22 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:22.522+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:22.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:22 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:22.522+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:22.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:22 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:22.522+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:22.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:22 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:22.522+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:23.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:22 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:22.912+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:23.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:22 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:22.912+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:23.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:22 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:22.912+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:23.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:22 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:22.912+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:23.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:22 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:22.912+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:23.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:22 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:22.912+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.614694+0000 front 2024-04-01T01:51:48.614602+0000 (oldest deadline 2024-04-01T01:52:14.514564+0000) 2024-04-01T01:52:23.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:23 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:23.248+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:23.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:23 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:23.248+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:23.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:23 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:23.248+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:23.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:23 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:23.248+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:23.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:23 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:23.248+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:23.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:23 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:23.248+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:23.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:23 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:23.248+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:52:00.645612+0000 front 2024-04-01T01:52:00.645600+0000 (oldest deadline 2024-04-01T01:52:22.345475+0000) 2024-04-01T01:52:23.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:23 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:23.568+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:23.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:23 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:23.568+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:23.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:23 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:23.568+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:23.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:23 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:23.568+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:23.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:23 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:23.568+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:23.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:23 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:23.568+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:24.000 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:23 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:23.568+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:24.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:23 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:23.884+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:24.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:23 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:23.884+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:24.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:23 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:23.884+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:24.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:23 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:23.884+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:24.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:23 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:23.884+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:24.255 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:23 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:23.884+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.614694+0000 front 2024-04-01T01:51:48.614602+0000 (oldest deadline 2024-04-01T01:52:14.514564+0000) 2024-04-01T01:52:24.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:24 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:24.251+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:24.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:24 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:24.251+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:24.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:24 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:24.251+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:24.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:24 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:24.251+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:24.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:24 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:24.251+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:24.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:24 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:24.251+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:24.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:24 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:24.251+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:52:00.645612+0000 front 2024-04-01T01:52:00.645600+0000 (oldest deadline 2024-04-01T01:52:22.345475+0000) 2024-04-01T01:52:24.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:24 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:24.547+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:24.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:24 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:24.547+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:24.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:24 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:24.547+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:24.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:24 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:24.547+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:24.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:24 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:24.547+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:24.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:24 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:24.547+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:24.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:24 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:24.547+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:25.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:24 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:24.912+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.107330+0000 front 2024-04-01T01:51:06.107343+0000 (oldest deadline 2024-04-01T01:51:31.407456+0000) 2024-04-01T01:52:25.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:24 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:24.912+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:18.909761+0000 front 2024-04-01T01:51:18.909780+0000 (oldest deadline 2024-04-01T01:51:41.209928+0000) 2024-04-01T01:52:25.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:24 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:24.912+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:23.511115+0000 front 2024-04-01T01:51:23.510921+0000 (oldest deadline 2024-04-01T01:51:48.811099+0000) 2024-04-01T01:52:25.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:24 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:24.912+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:34.112131+0000 front 2024-04-01T01:51:34.112176+0000 (oldest deadline 2024-04-01T01:51:58.212269+0000) 2024-04-01T01:52:25.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:24 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:24.912+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:42.213542+0000 front 2024-04-01T01:51:42.213566+0000 (oldest deadline 2024-04-01T01:52:06.913700+0000) 2024-04-01T01:52:25.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:24 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9[67547]: 2024-04-01T01:52:24.912+0000 7fcf30aef700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.614694+0000 front 2024-04-01T01:51:48.614602+0000 (oldest deadline 2024-04-01T01:52:14.514564+0000) 2024-04-01T01:52:25.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:25 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:25.212+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:25.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:25 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:25.212+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:25.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:25 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:25.212+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:25.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:25 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:25.212+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:25.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:25 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:25.212+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:25.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:25 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:25.212+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:25.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:25 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:25.212+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:52:00.645612+0000 front 2024-04-01T01:52:00.645600+0000 (oldest deadline 2024-04-01T01:52:22.345475+0000) 2024-04-01T01:52:25.998 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:25 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:25.546+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:25.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:25 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:25.546+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:25.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:25 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:25.546+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:25.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:25 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:25.546+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:25.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:25 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:25.546+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:25.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:25 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:25.546+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:25.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:25 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:25.546+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:26.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:26 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:26.248+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:26.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:26 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:26.248+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:26.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:26 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:26.248+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:26.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:26 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:26.248+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:26.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:26 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:26.248+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:26.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:26 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:26.248+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:26.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:26 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:26.248+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:52:00.645612+0000 front 2024-04-01T01:52:00.645600+0000 (oldest deadline 2024-04-01T01:52:22.345475+0000) 2024-04-01T01:52:26.887 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:26 smithi070 podman[118616]: 2024-04-01 01:52:26.58464937 +0000 UTC m=+5.863677812 container died ae88f7ee8a943e2d977eb040d1645ee7e0a656a1e55daf64231783d26908c140 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-04-01T01:52:26.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:26 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:26.578+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:26.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:26 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:26.578+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:26.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:26 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:26.578+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:26.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:26 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:26.578+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:26.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:26 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:26.578+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:26.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:26 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:26.578+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:26.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:26 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:26.578+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:27.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:26 smithi070 podman[118616]: 2024-04-01 01:52:26.962308527 +0000 UTC m=+6.241336964 container cleanup ae88f7ee8a943e2d977eb040d1645ee7e0a656a1e55daf64231783d26908c140 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-01T01:52:27.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:26 smithi070 bash[118616]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9 2024-04-01T01:52:27.546 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:27 smithi070 podman[118633]: 2024-04-01 01:52:27.255078727 +0000 UTC m=+0.661477189 container remove ae88f7ee8a943e2d977eb040d1645ee7e0a656a1e55daf64231783d26908c140 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-01T01:52:27.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:27 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:27.228+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:27.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:27 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:27.228+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:27.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:27 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:27.228+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:27.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:27 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:27.228+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:27.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:27 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:27.228+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:27.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:27 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:27.228+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:27.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:27 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:27.228+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:52:00.645612+0000 front 2024-04-01T01:52:00.645600+0000 (oldest deadline 2024-04-01T01:52:22.345475+0000) 2024-04-01T01:52:27.849 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:27 smithi070 podman[118721]: 2024-04-01 01:52:27.547519703 +0000 UTC m=+0.107848290 container create 25d8e14cb9d4cc17fab6619c6fa8d1d269e63c4a7ebb5e046f6470edd13e583f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9-deactivate, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, RELEASE=squid-384f31f) 2024-04-01T01:52:27.849 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:27 smithi070 podman[118721]: 2024-04-01 01:52:27.457284626 +0000 UTC m=+0.017613227 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:52:27.849 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:27 smithi070 podman[118721]: 2024-04-01 01:52:27.775016573 +0000 UTC m=+0.335345174 container init 25d8e14cb9d4cc17fab6619c6fa8d1d269e63c4a7ebb5e046f6470edd13e583f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, ceph=True) 2024-04-01T01:52:27.849 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:27 smithi070 podman[118721]: 2024-04-01 01:52:27.778079439 +0000 UTC m=+0.338408027 container start 25d8e14cb9d4cc17fab6619c6fa8d1d269e63c4a7ebb5e046f6470edd13e583f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326) 2024-04-01T01:52:27.849 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:27 smithi070 podman[118721]: 2024-04-01 01:52:27.849843605 +0000 UTC m=+0.410172202 container attach 25d8e14cb9d4cc17fab6619c6fa8d1d269e63c4a7ebb5e046f6470edd13e583f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-01T01:52:27.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:27 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:27.611+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:27.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:27 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:27.611+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:27.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:27 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:27.611+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:27.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:27 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:27.611+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:27.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:27 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:27.611+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:27.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:27 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:27.611+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:27.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:27 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:27.611+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:28.254 INFO:journalctl@ceph.osd.9.smithi070.stdout:Apr 01 01:52:28 smithi070 podman[118721]: 2024-04-01 01:52:28.05903139 +0000 UTC m=+0.619359980 container died 25d8e14cb9d4cc17fab6619c6fa8d1d269e63c4a7ebb5e046f6470edd13e583f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-384f31f) 2024-04-01T01:52:28.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:28 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:28.239+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:28.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:28 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:28.239+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:28.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:28 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:28.239+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:28.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:28 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:28.239+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:28.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:28 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:28.239+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:28.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:28 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:28.239+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:28.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:28 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:28.239+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:52:00.645612+0000 front 2024-04-01T01:52:00.645600+0000 (oldest deadline 2024-04-01T01:52:22.345475+0000) 2024-04-01T01:52:28.631 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.9.service' 2024-04-01T01:52:28.671 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:52:28.671 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-01T01:52:28.672 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-01T01:52:28.672 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.10 2024-04-01T01:52:28.998 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:28 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:28.609+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:28.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:28 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:28.609+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:28.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:28 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:28.609+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:28.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:28 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:28.609+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:28.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:28 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:28.609+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:28.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:28 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:28.609+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:28.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:28 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:28.609+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:29.041 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:28 smithi081 systemd[1]: Stopping Ceph osd.10 for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:52:29.328 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:29 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:29.040+0000 7fa9e16b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:52:29.328 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:29 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:29.040+0000 7fa9e16b9700 -1 osd.10 95 *** Got signal Terminated *** 2024-04-01T01:52:29.328 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:29 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:29.040+0000 7fa9e16b9700 -1 osd.10 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-01T01:52:29.329 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:29 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:29.236+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:29.329 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:29 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:29.236+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:29.329 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:29 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:29.236+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:29.329 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:29 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:29.236+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:29.329 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:29 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:29.236+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:29.329 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:29 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:29.236+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:29.329 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:29 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:29.236+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:52:00.645612+0000 front 2024-04-01T01:52:00.645600+0000 (oldest deadline 2024-04-01T01:52:22.345475+0000) 2024-04-01T01:52:29.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:29 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:29.628+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:29.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:29 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:29.628+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:29.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:29 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:29.628+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:29.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:29 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:29.628+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:29.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:29 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:29.628+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:29.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:29 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:29.628+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:29.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:29 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:29.628+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:30.571 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:30 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:30.285+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:30.571 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:30 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:30.285+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:30.571 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:30 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:30.285+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:30.571 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:30 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:30.285+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:30.571 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:30 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:30.285+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:30.572 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:30 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:30.285+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:30.572 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:30 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:30.285+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:52:00.645612+0000 front 2024-04-01T01:52:00.645600+0000 (oldest deadline 2024-04-01T01:52:22.345475+0000) 2024-04-01T01:52:30.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:30 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:30.612+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:30.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:30 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:30.612+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:30.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:30 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:30.612+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:30.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:30 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:30.612+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:30.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:30 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:30.612+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:30.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:30 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:30.612+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:30.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:30 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:30.612+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:31.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:31 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:31.266+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:31.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:31 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:31.266+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:31.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:31 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:31.266+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:31.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:31 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:31.266+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:31.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:31 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:31.266+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:31.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:31 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:31.266+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:31.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:31 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:31.266+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:52:00.645612+0000 front 2024-04-01T01:52:00.645600+0000 (oldest deadline 2024-04-01T01:52:22.345475+0000) 2024-04-01T01:52:31.998 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:31 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:31.586+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:31.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:31 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:31.586+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:31.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:31 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:31.586+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:31.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:31 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:31.586+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:31.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:31 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:31.586+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:31.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:31 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:31.586+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:31.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:31 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:31.586+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:32.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:32 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:32.236+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:32.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:32 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:32.236+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:32.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:32 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:32.236+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:32.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:32 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:32.236+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:32.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:32 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:32.236+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:32.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:32 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:32.236+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:32.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:32 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:32.237+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:52:00.645612+0000 front 2024-04-01T01:52:00.645600+0000 (oldest deadline 2024-04-01T01:52:22.345475+0000) 2024-04-01T01:52:32.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:32 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:32.606+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:32.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:32 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:32.606+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:32.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:32 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:32.606+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:32.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:32 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:32.606+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:32.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:32 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:32.606+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:32.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:32 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:32.606+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:32.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:32 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:32.606+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:32.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:32 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:32.606+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6820 osd.7 since back 2024-04-01T01:52:06.502735+0000 front 2024-04-01T01:52:06.502767+0000 (oldest deadline 2024-04-01T01:52:32.402620+0000) 2024-04-01T01:52:33.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:33 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:33.276+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:06.838354+0000 front 2024-04-01T01:51:06.838298+0000 (oldest deadline 2024-04-01T01:51:31.538284+0000) 2024-04-01T01:52:33.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:33 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:33.276+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:16.139455+0000 front 2024-04-01T01:51:16.139475+0000 (oldest deadline 2024-04-01T01:51:42.039597+0000) 2024-04-01T01:52:33.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:33 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:33.276+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:27.741121+0000 front 2024-04-01T01:51:27.741052+0000 (oldest deadline 2024-04-01T01:51:50.641163+0000) 2024-04-01T01:52:33.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:33 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:33.276+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:33.541898+0000 front 2024-04-01T01:51:33.542005+0000 (oldest deadline 2024-04-01T01:51:57.041894+0000) 2024-04-01T01:52:33.578 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:33 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:33.276+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:40.542517+0000 front 2024-04-01T01:51:40.542538+0000 (oldest deadline 2024-04-01T01:52:05.242618+0000) 2024-04-01T01:52:33.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:33 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:33.276+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.043659+0000 front 2024-04-01T01:51:48.043585+0000 (oldest deadline 2024-04-01T01:52:13.943603+0000) 2024-04-01T01:52:33.579 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:33 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10[59757]: 2024-04-01T01:52:33.276+0000 7fa9d9ca5700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:52:00.645612+0000 front 2024-04-01T01:52:00.645600+0000 (oldest deadline 2024-04-01T01:52:22.345475+0000) 2024-04-01T01:52:33.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:33 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:33.592+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:33.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:33 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:33.592+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:33.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:33 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:33.592+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:33.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:33 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:33.592+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:33.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:33 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:33.592+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:33.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:33 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:33.592+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:33.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:33 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:33.592+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:33.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:33 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:33.592+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6820 osd.7 since back 2024-04-01T01:52:06.502735+0000 front 2024-04-01T01:52:06.502767+0000 (oldest deadline 2024-04-01T01:52:32.402620+0000) 2024-04-01T01:52:34.782 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:34 smithi081 podman[70958]: 2024-04-01 01:52:34.507256369 +0000 UTC m=+5.585847693 container died c35bfefbd4eff055b7ec9807171e9dee929195b30d6c764bad562462b2f24cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-04-01T01:52:34.998 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:34 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:34.605+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:34.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:34 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:34.605+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:34.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:34 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:34.605+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:34.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:34 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:34.605+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:34.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:34 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:34.605+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:34.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:34 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:34.605+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:34.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:34 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:34.605+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:34.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:34 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:34.605+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6820 osd.7 since back 2024-04-01T01:52:06.502735+0000 front 2024-04-01T01:52:06.502767+0000 (oldest deadline 2024-04-01T01:52:32.402620+0000) 2024-04-01T01:52:35.078 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:34 smithi081 podman[70958]: 2024-04-01 01:52:34.874348491 +0000 UTC m=+5.952939810 container cleanup c35bfefbd4eff055b7ec9807171e9dee929195b30d6c764bad562462b2f24cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-01T01:52:35.078 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:34 smithi081 bash[70958]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10 2024-04-01T01:52:35.492 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:35 smithi081 podman[70974]: 2024-04-01 01:52:35.134308177 +0000 UTC m=+0.627950855 container remove c35bfefbd4eff055b7ec9807171e9dee929195b30d6c764bad562462b2f24cab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-01T01:52:35.762 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:35 smithi081 podman[71065]: 2024-04-01 01:52:35.493444711 +0000 UTC m=+0.115120373 container create dd3a8e74b621f438b03502354a37b5b51e317a0cc30409f6d17bc95e8c962e4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-01T01:52:35.762 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:35 smithi081 podman[71065]: 2024-04-01 01:52:35.396984893 +0000 UTC m=+0.018660583 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:52:35.762 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:35 smithi081 podman[71065]: 2024-04-01 01:52:35.679610941 +0000 UTC m=+0.301286615 container init dd3a8e74b621f438b03502354a37b5b51e317a0cc30409f6d17bc95e8c962e4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-01T01:52:35.762 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:35 smithi081 podman[71065]: 2024-04-01 01:52:35.683101834 +0000 UTC m=+0.304777501 container start dd3a8e74b621f438b03502354a37b5b51e317a0cc30409f6d17bc95e8c962e4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10-deactivate, GIT_CLEAN=True, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-01T01:52:35.763 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:35 smithi081 podman[71065]: 2024-04-01 01:52:35.762508692 +0000 UTC m=+0.384184369 container attach dd3a8e74b621f438b03502354a37b5b51e317a0cc30409f6d17bc95e8c962e4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10-deactivate, io.buildah.version=1.33.5, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-01T01:52:35.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:35 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:35.598+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:35.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:35 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:35.598+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:35.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:35 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:35.598+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:35.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:35 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:35.598+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:35.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:35 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:35.598+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:35.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:35 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:35.598+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:35.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:35 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:35.598+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:35.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:35 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:35.598+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6820 osd.7 since back 2024-04-01T01:52:06.502735+0000 front 2024-04-01T01:52:06.502767+0000 (oldest deadline 2024-04-01T01:52:32.402620+0000) 2024-04-01T01:52:36.323 INFO:journalctl@ceph.osd.10.smithi081.stdout:Apr 01 01:52:36 smithi081 podman[71065]: 2024-04-01 01:52:36.026998241 +0000 UTC m=+0.648673919 container died dd3a8e74b621f438b03502354a37b5b51e317a0cc30409f6d17bc95e8c962e4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-10-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-01T01:52:36.649 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.10.service' 2024-04-01T01:52:36.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:52:36.688 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-01T01:52:36.688 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-01T01:52:36.689 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.11 2024-04-01T01:52:36.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:36 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:36.597+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:36.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:36 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:36.597+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:36.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:36 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:36.597+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:36.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:36 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:36.597+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:36.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:36 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:36.597+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:36.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:36 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:36.597+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:36.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:36 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:36.597+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:36.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:36 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:36.597+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6820 osd.7 since back 2024-04-01T01:52:06.502735+0000 front 2024-04-01T01:52:06.502767+0000 (oldest deadline 2024-04-01T01:52:32.402620+0000) 2024-04-01T01:52:37.000 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:36 smithi189 systemd[1]: Stopping Ceph osd.11 for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:52:37.499 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:37 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:37.003+0000 7f8d4bb7f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:52:37.499 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:37 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:37.003+0000 7f8d4bb7f700 -1 osd.11 95 *** Got signal Terminated *** 2024-04-01T01:52:37.499 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:37 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:37.003+0000 7f8d4bb7f700 -1 osd.11 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-01T01:52:37.998 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:37 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:37.599+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:37.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:37 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:37.599+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:37.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:37 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:37.599+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:37.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:37 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:37.599+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:37.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:37 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:37.599+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:37.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:37 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:37.599+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:37.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:37 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:37.599+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:37.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:37 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:37.599+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6820 osd.7 since back 2024-04-01T01:52:06.502735+0000 front 2024-04-01T01:52:06.502767+0000 (oldest deadline 2024-04-01T01:52:32.402620+0000) 2024-04-01T01:52:38.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:38 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:38.617+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:38.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:38 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:38.617+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:38.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:38 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:38.617+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:38.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:38 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:38.617+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:38.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:38 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:38.617+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:38.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:38 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:38.617+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:38.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:38 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:38.617+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:38.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:38 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:38.617+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6820 osd.7 since back 2024-04-01T01:52:06.502735+0000 front 2024-04-01T01:52:06.502767+0000 (oldest deadline 2024-04-01T01:52:32.402620+0000) 2024-04-01T01:52:39.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:39 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:39.606+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:39.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:39 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:39.606+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:39.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:39 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:39.606+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:39.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:39 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:39.606+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:39.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:39 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:39.606+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:39.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:39 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:39.606+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:39.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:39 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:39.606+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:39.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:39 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:39.606+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6820 osd.7 since back 2024-04-01T01:52:06.502735+0000 front 2024-04-01T01:52:06.502767+0000 (oldest deadline 2024-04-01T01:52:32.402620+0000) 2024-04-01T01:52:40.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:40 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:40.574+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:40.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:40 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:40.574+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:40.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:40 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:40.574+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:40.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:40 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:40.574+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:40.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:40 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:40.574+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:40.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:40 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:40.574+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:40.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:40 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:40.574+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:40.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:40 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:40.574+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6820 osd.7 since back 2024-04-01T01:52:06.502735+0000 front 2024-04-01T01:52:06.502767+0000 (oldest deadline 2024-04-01T01:52:32.402620+0000) 2024-04-01T01:52:40.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:40 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:40.574+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6820 osd.8 since back 2024-04-01T01:52:15.903348+0000 front 2024-04-01T01:52:15.903372+0000 (oldest deadline 2024-04-01T01:52:40.503942+0000) 2024-04-01T01:52:41.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:41 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:41.609+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6806 osd.0 since back 2024-04-01T01:51:07.894013+0000 front 2024-04-01T01:51:07.893925+0000 (oldest deadline 2024-04-01T01:51:33.193920+0000) 2024-04-01T01:52:41.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:41 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:41.609+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6804 osd.1 since back 2024-04-01T01:51:17.195268+0000 front 2024-04-01T01:51:17.195291+0000 (oldest deadline 2024-04-01T01:51:41.895356+0000) 2024-04-01T01:52:41.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:41 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:41.609+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-01T01:51:26.997126+0000 front 2024-04-01T01:51:26.997157+0000 (oldest deadline 2024-04-01T01:51:49.897250+0000) 2024-04-01T01:52:41.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:41 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:41.609+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6814 osd.3 since back 2024-04-01T01:51:35.698796+0000 front 2024-04-01T01:51:35.698932+0000 (oldest deadline 2024-04-01T01:51:59.798724+0000) 2024-04-01T01:52:41.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:41 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:41.609+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6812 osd.4 since back 2024-04-01T01:51:39.799175+0000 front 2024-04-01T01:51:39.799106+0000 (oldest deadline 2024-04-01T01:52:05.699192+0000) 2024-04-01T01:52:41.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:41 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:41.609+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6812 osd.5 since back 2024-04-01T01:51:48.500352+0000 front 2024-04-01T01:51:48.500322+0000 (oldest deadline 2024-04-01T01:52:14.400481+0000) 2024-04-01T01:52:41.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:41 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:41.609+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.70:6822 osd.6 since back 2024-04-01T01:51:57.201943+0000 front 2024-04-01T01:51:57.202032+0000 (oldest deadline 2024-04-01T01:52:21.901741+0000) 2024-04-01T01:52:41.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:41 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:41.609+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.81:6820 osd.7 since back 2024-04-01T01:52:06.502735+0000 front 2024-04-01T01:52:06.502767+0000 (oldest deadline 2024-04-01T01:52:32.402620+0000) 2024-04-01T01:52:41.999 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:41 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11[61423]: 2024-04-01T01:52:41.609+0000 7f8d4416b700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.189:6820 osd.8 since back 2024-04-01T01:52:15.903348+0000 front 2024-04-01T01:52:15.903372+0000 (oldest deadline 2024-04-01T01:52:40.503942+0000) 2024-04-01T01:52:42.899 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:42 smithi189 podman[73183]: 2024-04-01 01:52:42.600946765 +0000 UTC m=+5.698505978 container died f64bcd3aeb89d2f9551f0452d7ba2436ef3cd87ebf52c96c938728bbdfb1eb99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-01T01:52:43.248 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:43 smithi189 podman[73183]: 2024-04-01 01:52:43.002733217 +0000 UTC m=+6.100292429 container cleanup f64bcd3aeb89d2f9551f0452d7ba2436ef3cd87ebf52c96c938728bbdfb1eb99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-01T01:52:43.249 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:43 smithi189 bash[73183]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11 2024-04-01T01:52:43.635 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:43 smithi189 podman[73202]: 2024-04-01 01:52:43.295730834 +0000 UTC m=+0.693469873 container remove f64bcd3aeb89d2f9551f0452d7ba2436ef3cd87ebf52c96c938728bbdfb1eb99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-384f31f) 2024-04-01T01:52:43.949 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:43 smithi189 podman[73289]: 2024-04-01 01:52:43.538632941 +0000 UTC m=+0.019455920 image pull ec18d7baa60e8b1a0b126fbe5306acce11b1ed0ec5e3bcdddbf39b0a3c591791 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175 2024-04-01T01:52:43.949 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:43 smithi189 podman[73289]: 2024-04-01 01:52:43.67174989 +0000 UTC m=+0.152572854 container create 1beaae1557cda646133aa891fccebee93b216d247a0954a1f20ab43d2ff340ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-01T01:52:43.949 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:43 smithi189 podman[73289]: 2024-04-01 01:52:43.874450832 +0000 UTC m=+0.355273812 container init 1beaae1557cda646133aa891fccebee93b216d247a0954a1f20ab43d2ff340ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, RELEASE=squid-384f31f) 2024-04-01T01:52:43.949 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:43 smithi189 podman[73289]: 2024-04-01 01:52:43.877581318 +0000 UTC m=+0.358404290 container start 1beaae1557cda646133aa891fccebee93b216d247a0954a1f20ab43d2ff340ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11-deactivate, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-01T01:52:43.949 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:43 smithi189 podman[73289]: 2024-04-01 01:52:43.949398543 +0000 UTC m=+0.430221516 container attach 1beaae1557cda646133aa891fccebee93b216d247a0954a1f20ab43d2ff340ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, GIT_CLEAN=True) 2024-04-01T01:52:44.498 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 01 01:52:44 smithi189 podman[73289]: 2024-04-01 01:52:44.208943221 +0000 UTC m=+0.689766198 container died 1beaae1557cda646133aa891fccebee93b216d247a0954a1f20ab43d2ff340ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-01T01:52:44.805 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@osd.11.service' 2024-04-01T01:52:44.846 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:52:44.846 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-01T01:52:44.847 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-01T01:52:44.847 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.a 2024-04-01T01:52:45.154 INFO:journalctl@ceph.mds.a.smithi070.stdout:Apr 01 01:52:44 smithi070 systemd[1]: Stopping Ceph mds.a for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:52:45.504 INFO:journalctl@ceph.mds.a.smithi070.stdout:Apr 01 01:52:45 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-a[74108]: 2024-04-01T01:52:45.154+0000 7fc3695e6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:52:45.504 INFO:journalctl@ceph.mds.a.smithi070.stdout:Apr 01 01:52:45 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-a[74108]: 2024-04-01T01:52:45.154+0000 7fc3695e6700 -1 mds.a *** got signal Terminated *** 2024-04-01T01:52:48.754 INFO:journalctl@ceph.mds.a.smithi070.stdout:Apr 01 01:52:48 smithi070 podman[118836]: 2024-04-01 01:52:48.452525893 +0000 UTC m=+3.412520944 container died 08971e5b39e2304c914cf05398993d50f806a54db26a9a9db9ef812d348fd053 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-04-01T01:52:49.170 INFO:journalctl@ceph.mds.a.smithi070.stdout:Apr 01 01:52:48 smithi070 podman[118836]: 2024-04-01 01:52:48.761572435 +0000 UTC m=+3.721567490 container cleanup 08971e5b39e2304c914cf05398993d50f806a54db26a9a9db9ef812d348fd053 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-384f31f, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-04-01T01:52:49.170 INFO:journalctl@ceph.mds.a.smithi070.stdout:Apr 01 01:52:48 smithi070 bash[118836]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-a 2024-04-01T01:52:49.170 INFO:journalctl@ceph.mds.a.smithi070.stdout:Apr 01 01:52:48 smithi070 podman[118850]: 2024-04-01 01:52:48.979550203 +0000 UTC m=+0.524299145 container remove 08971e5b39e2304c914cf05398993d50f806a54db26a9a9db9ef812d348fd053 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-a, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-01T01:52:49.191 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.a.service' 2024-04-01T01:52:49.227 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:52:49.228 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-01T01:52:49.228 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-01T01:52:49.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.d 2024-04-01T01:52:49.504 INFO:journalctl@ceph.mds.d.smithi070.stdout:Apr 01 01:52:49 smithi070 systemd[1]: Stopping Ceph mds.d for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:52:50.004 INFO:journalctl@ceph.mds.d.smithi070.stdout:Apr 01 01:52:49 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-d[73150]: 2024-04-01T01:52:49.548+0000 7f7d5479f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:52:50.004 INFO:journalctl@ceph.mds.d.smithi070.stdout:Apr 01 01:52:49 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-d[73150]: 2024-04-01T01:52:49.548+0000 7f7d5479f700 -1 mds.d *** got signal Terminated *** 2024-04-01T01:52:53.004 INFO:journalctl@ceph.mds.d.smithi070.stdout:Apr 01 01:52:52 smithi070 podman[118955]: 2024-04-01 01:52:52.684699682 +0000 UTC m=+3.236480715 container died c36c2d6d7d075dd0cbdd5db2a57321bd9bd3832e426d6f136f6f4ed18a562c5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-01T01:52:53.399 INFO:journalctl@ceph.mds.d.smithi070.stdout:Apr 01 01:52:53 smithi070 podman[118955]: 2024-04-01 01:52:53.140085021 +0000 UTC m=+3.691866042 container cleanup c36c2d6d7d075dd0cbdd5db2a57321bd9bd3832e426d6f136f6f4ed18a562c5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-d, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-01T01:52:53.399 INFO:journalctl@ceph.mds.d.smithi070.stdout:Apr 01 01:52:53 smithi070 bash[118955]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-d 2024-04-01T01:52:53.560 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.d.service' 2024-04-01T01:52:53.595 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:52:53.595 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-01T01:52:53.595 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-01T01:52:53.595 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.g 2024-04-01T01:52:54.004 INFO:journalctl@ceph.mds.g.smithi070.stdout:Apr 01 01:52:53 smithi070 systemd[1]: Stopping Ceph mds.g for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:52:54.004 INFO:journalctl@ceph.mds.g.smithi070.stdout:Apr 01 01:52:53 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-g[73747]: 2024-04-01T01:52:53.901+0000 7f2398d72700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:52:54.004 INFO:journalctl@ceph.mds.g.smithi070.stdout:Apr 01 01:52:53 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-g[73747]: 2024-04-01T01:52:53.901+0000 7f2398d72700 -1 mds.g *** got signal Terminated *** 2024-04-01T01:52:58.004 INFO:journalctl@ceph.mds.g.smithi070.stdout:Apr 01 01:52:57 smithi070 podman[119077]: 2024-04-01 01:52:57.649616914 +0000 UTC m=+3.841080517 container died c02fe3619d88e2ce29c20d7cd89b4685cd1a495355442b1595c8a28dbdffa99c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-384f31f, org.label-schema.license=GPLv2) 2024-04-01T01:52:58.354 INFO:journalctl@ceph.mds.g.smithi070.stdout:Apr 01 01:52:58 smithi070 podman[119077]: 2024-04-01 01:52:58.118442884 +0000 UTC m=+4.309906478 container cleanup c02fe3619d88e2ce29c20d7cd89b4685cd1a495355442b1595c8a28dbdffa99c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-384f31f, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-01T01:52:58.354 INFO:journalctl@ceph.mds.g.smithi070.stdout:Apr 01 01:52:58 smithi070 bash[119077]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-g 2024-04-01T01:52:58.496 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.g.service' 2024-04-01T01:52:58.531 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:52:58.531 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-01T01:52:58.531 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-01T01:52:58.531 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.j 2024-04-01T01:52:59.004 INFO:journalctl@ceph.mds.j.smithi070.stdout:Apr 01 01:52:58 smithi070 systemd[1]: Stopping Ceph mds.j for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:52:59.004 INFO:journalctl@ceph.mds.j.smithi070.stdout:Apr 01 01:52:58 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-j[72764]: 2024-04-01T01:52:58.854+0000 7f34a2909700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:52:59.004 INFO:journalctl@ceph.mds.j.smithi070.stdout:Apr 01 01:52:58 smithi070 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-j[72764]: 2024-04-01T01:52:58.854+0000 7f34a2909700 -1 mds.j *** got signal Terminated *** 2024-04-01T01:53:02.004 INFO:journalctl@ceph.mds.j.smithi070.stdout:Apr 01 01:53:01 smithi070 podman[119199]: 2024-04-01 01:53:01.735239346 +0000 UTC m=+2.987810357 container died bfab699b9b086680c5ce676edf959ab637ceb44a6c7f9255852f59a3aa3ae160 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-j, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-01T01:53:02.372 INFO:journalctl@ceph.mds.j.smithi070.stdout:Apr 01 01:53:02 smithi070 podman[119199]: 2024-04-01 01:53:02.121881602 +0000 UTC m=+3.374452611 container cleanup bfab699b9b086680c5ce676edf959ab637ceb44a6c7f9255852f59a3aa3ae160 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-j, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240326, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=squid-384f31f) 2024-04-01T01:53:02.372 INFO:journalctl@ceph.mds.j.smithi070.stdout:Apr 01 01:53:02 smithi070 bash[119199]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-j 2024-04-01T01:53:02.480 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.j.service' 2024-04-01T01:53:02.515 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:53:02.515 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-01T01:53:02.515 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-01T01:53:02.515 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.b 2024-04-01T01:53:02.828 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:53:02 smithi081 systemd[1]: Stopping Ceph mds.b for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:53:03.151 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:53:02 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-b[65004]: 2024-04-01T01:53:02.853+0000 7ff9953fe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:53:03.151 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:53:02 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-b[65004]: 2024-04-01T01:53:02.853+0000 7ff9953fe700 -1 mds.b *** got signal Terminated *** 2024-04-01T01:53:08.150 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:53:07 smithi081 podman[71171]: 2024-04-01 01:53:07.830454229 +0000 UTC m=+5.085669904 container died de2b5306512919bec671c8d5bfeb447118a2e58fd6a48c7f3a7b7f29dd8a28d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-b, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-01T01:53:08.150 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:53:08 smithi081 podman[71171]: 2024-04-01 01:53:08.150882128 +0000 UTC m=+5.406097809 container cleanup de2b5306512919bec671c8d5bfeb447118a2e58fd6a48c7f3a7b7f29dd8a28d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-04-01T01:53:08.569 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:53:08 smithi081 bash[71171]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-b 2024-04-01T01:53:08.569 INFO:journalctl@ceph.mds.b.smithi081.stdout:Apr 01 01:53:08 smithi081 podman[71196]: 2024-04-01 01:53:08.377407143 +0000 UTC m=+0.548934153 container remove de2b5306512919bec671c8d5bfeb447118a2e58fd6a48c7f3a7b7f29dd8a28d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-b, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-01T01:53:08.583 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.b.service' 2024-04-01T01:53:08.622 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:53:08.622 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-01T01:53:08.622 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-01T01:53:08.623 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.e 2024-04-01T01:53:08.946 INFO:journalctl@ceph.mds.e.smithi081.stdout:Apr 01 01:53:08 smithi081 systemd[1]: Stopping Ceph mds.e for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:53:09.328 INFO:journalctl@ceph.mds.e.smithi081.stdout:Apr 01 01:53:08 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-e[64372]: 2024-04-01T01:53:08.946+0000 7f0ece7d9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:53:09.328 INFO:journalctl@ceph.mds.e.smithi081.stdout:Apr 01 01:53:08 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-e[64372]: 2024-04-01T01:53:08.946+0000 7f0ece7d9700 -1 mds.e *** got signal Terminated *** 2024-04-01T01:53:14.019 INFO:journalctl@ceph.mds.e.smithi081.stdout:Apr 01 01:53:13 smithi081 podman[71301]: 2024-04-01 01:53:13.73035897 +0000 UTC m=+4.869136998 container died bb9382b3a3c19b332ff7366629790af05061359e0ba49381280c41fe3206c33c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-01T01:53:14.328 INFO:journalctl@ceph.mds.e.smithi081.stdout:Apr 01 01:53:14 smithi081 podman[71301]: 2024-04-01 01:53:14.096750991 +0000 UTC m=+5.235529000 container cleanup bb9382b3a3c19b332ff7366629790af05061359e0ba49381280c41fe3206c33c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-e, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2) 2024-04-01T01:53:14.328 INFO:journalctl@ceph.mds.e.smithi081.stdout:Apr 01 01:53:14 smithi081 bash[71301]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-e 2024-04-01T01:53:14.328 INFO:journalctl@ceph.mds.e.smithi081.stdout:Apr 01 01:53:14 smithi081 podman[71315]: 2024-04-01 01:53:14.247845105 +0000 UTC m=+0.518878464 container remove bb9382b3a3c19b332ff7366629790af05061359e0ba49381280c41fe3206c33c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-e, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-01T01:53:14.477 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.e.service' 2024-04-01T01:53:14.512 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:53:14.513 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-01T01:53:14.513 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-01T01:53:14.513 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.h 2024-04-01T01:53:14.828 INFO:journalctl@ceph.mds.h.smithi081.stdout:Apr 01 01:53:14 smithi081 systemd[1]: Stopping Ceph mds.h for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:53:15.328 INFO:journalctl@ceph.mds.h.smithi081.stdout:Apr 01 01:53:14 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-h[64682]: 2024-04-01T01:53:14.849+0000 7f60f7437700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:53:15.328 INFO:journalctl@ceph.mds.h.smithi081.stdout:Apr 01 01:53:14 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-h[64682]: 2024-04-01T01:53:14.849+0000 7f60f7437700 -1 mds.h *** got signal Terminated *** 2024-04-01T01:53:19.328 INFO:journalctl@ceph.mds.h.smithi081.stdout:Apr 01 01:53:19 smithi081 podman[71421]: 2024-04-01 01:53:19.040774074 +0000 UTC m=+4.313851169 container died 720bfad3a84aa35c7e3088d5957c796ffddb19d10dbac2d3fefbb9a2180101de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-h, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-01T01:53:19.776 INFO:journalctl@ceph.mds.h.smithi081.stdout:Apr 01 01:53:19 smithi081 podman[71421]: 2024-04-01 01:53:19.575718356 +0000 UTC m=+4.848795439 container cleanup 720bfad3a84aa35c7e3088d5957c796ffddb19d10dbac2d3fefbb9a2180101de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-h, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-01T01:53:19.776 INFO:journalctl@ceph.mds.h.smithi081.stdout:Apr 01 01:53:19 smithi081 bash[71421]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-h 2024-04-01T01:53:19.995 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.h.service' 2024-04-01T01:53:20.029 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:53:20.030 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-01T01:53:20.030 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-01T01:53:20.030 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.k 2024-04-01T01:53:20.328 INFO:journalctl@ceph.mds.k.smithi081.stdout:Apr 01 01:53:20 smithi081 systemd[1]: Stopping Ceph mds.k for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:53:20.828 INFO:journalctl@ceph.mds.k.smithi081.stdout:Apr 01 01:53:20 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-k[64059]: 2024-04-01T01:53:20.337+0000 7f0074ed0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:53:20.828 INFO:journalctl@ceph.mds.k.smithi081.stdout:Apr 01 01:53:20 smithi081 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-k[64059]: 2024-04-01T01:53:20.337+0000 7f0074ed0700 -1 mds.k *** got signal Terminated *** 2024-04-01T01:53:23.097 INFO:journalctl@ceph.mds.k.smithi081.stdout:Apr 01 01:53:22 smithi081 podman[71542]: 2024-04-01 01:53:22.836558339 +0000 UTC m=+2.588289767 container died 8da18107b3c1b71c55a02c33717f84324bbac0234c0d8154db317af41786202b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-k, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=) 2024-04-01T01:53:23.369 INFO:journalctl@ceph.mds.k.smithi081.stdout:Apr 01 01:53:23 smithi081 podman[71542]: 2024-04-01 01:53:23.196211121 +0000 UTC m=+2.947942550 container cleanup 8da18107b3c1b71c55a02c33717f84324bbac0234c0d8154db317af41786202b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, RELEASE=squid-384f31f) 2024-04-01T01:53:23.369 INFO:journalctl@ceph.mds.k.smithi081.stdout:Apr 01 01:53:23 smithi081 bash[71542]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-k 2024-04-01T01:53:23.590 DEBUG:teuthology.orchestra.run.smithi081:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.k.service' 2024-04-01T01:53:23.624 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:53:23.624 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-01T01:53:23.624 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-01T01:53:23.624 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.c 2024-04-01T01:53:23.978 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:53:23 smithi189 systemd[1]: Stopping Ceph mds.c for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:53:24.249 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:53:24 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-c[66342]: 2024-04-01T01:53:24.018+0000 7fbb5f596700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:53:24.249 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:53:24 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-c[66342]: 2024-04-01T01:53:24.018+0000 7fbb5f596700 -1 mds.c *** got signal Terminated *** 2024-04-01T01:53:28.336 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:53:28 smithi189 podman[73395]: 2024-04-01 01:53:28.084344842 +0000 UTC m=+4.233638822 container died a3967370277d3cb31d2b0c6101c9ea16119523a9a1d3a1b93dc802fc808fc0b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-c, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240326, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-01T01:53:28.594 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:53:28 smithi189 podman[73395]: 2024-04-01 01:53:28.426713587 +0000 UTC m=+4.576007558 container cleanup a3967370277d3cb31d2b0c6101c9ea16119523a9a1d3a1b93dc802fc808fc0b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-384f31f, io.buildah.version=1.33.5, org.label-schema.build-date=20240326, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-01T01:53:28.594 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 01 01:53:28 smithi189 bash[73395]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-c 2024-04-01T01:53:28.816 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.c.service' 2024-04-01T01:53:28.855 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:53:28.855 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-01T01:53:28.855 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-01T01:53:28.855 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.f 2024-04-01T01:53:29.248 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 01 01:53:28 smithi189 systemd[1]: Stopping Ceph mds.f for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:53:29.748 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 01 01:53:29 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-f[66039]: 2024-04-01T01:53:29.272+0000 7f1e56cef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:53:29.748 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 01 01:53:29 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-f[66039]: 2024-04-01T01:53:29.272+0000 7f1e56cef700 -1 mds.f *** got signal Terminated *** 2024-04-01T01:53:33.022 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 01 01:53:32 smithi189 podman[73515]: 2024-04-01 01:53:32.761732313 +0000 UTC m=+3.653045964 container died bc506a7755a36b0cebf137aa48255bd119326e394e5511efe6891ec7080db8d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-384f31f, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2) 2024-04-01T01:53:33.498 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 01 01:53:33 smithi189 podman[73515]: 2024-04-01 01:53:33.130644194 +0000 UTC m=+4.021957837 container cleanup bc506a7755a36b0cebf137aa48255bd119326e394e5511efe6891ec7080db8d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-f, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, ceph=True) 2024-04-01T01:53:33.499 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 01 01:53:33 smithi189 bash[73515]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-f 2024-04-01T01:53:33.693 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.f.service' 2024-04-01T01:53:33.730 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:53:33.730 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-01T01:53:33.730 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-01T01:53:33.730 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.i 2024-04-01T01:53:34.060 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 01 01:53:33 smithi189 systemd[1]: Stopping Ceph mds.i for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:53:34.498 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 01 01:53:34 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-i[66659]: 2024-04-01T01:53:34.059+0000 7f3d1ad5e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:53:34.499 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 01 01:53:34 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-i[66659]: 2024-04-01T01:53:34.059+0000 7f3d1ad5e700 -1 mds.i *** got signal Terminated *** 2024-04-01T01:53:40.093 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 01 01:53:39 smithi189 podman[73635]: 2024-04-01 01:53:39.826730132 +0000 UTC m=+5.862328100 container died 50ee14cddb4ccc2d1a892a41dccb4c27206093e031ac2b20d79052ae256b663e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-i, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-384f31f, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240326, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-01T01:53:40.498 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 01 01:53:40 smithi189 podman[73635]: 2024-04-01 01:53:40.167601097 +0000 UTC m=+6.203199074 container cleanup 50ee14cddb4ccc2d1a892a41dccb4c27206093e031ac2b20d79052ae256b663e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-i, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240326, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=squid-384f31f, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-01T01:53:40.499 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 01 01:53:40 smithi189 bash[73635]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-i 2024-04-01T01:53:40.499 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 01 01:53:40 smithi189 podman[73649]: 2024-04-01 01:53:40.343719946 +0000 UTC m=+0.512895182 container remove 50ee14cddb4ccc2d1a892a41dccb4c27206093e031ac2b20d79052ae256b663e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-i, RELEASE=squid-384f31f, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240326, org.label-schema.schema-version=1.0) 2024-04-01T01:53:40.593 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.i.service' 2024-04-01T01:53:40.627 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:53:40.627 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-01T01:53:40.628 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-01T01:53:40.628 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.l 2024-04-01T01:53:40.962 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 01 01:53:40 smithi189 systemd[1]: Stopping Ceph mds.l for 36577394-efc6-11ee-b647-cb9ed24678a4... 2024-04-01T01:53:41.248 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 01 01:53:40 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-l[65730]: 2024-04-01T01:53:40.962+0000 7fbdba836700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-01T01:53:41.249 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 01 01:53:40 smithi189 ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-l[65730]: 2024-04-01T01:53:40.962+0000 7fbdba836700 -1 mds.l *** got signal Terminated *** 2024-04-01T01:53:47.196 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 01 01:53:46 smithi189 podman[73755]: 2024-04-01 01:53:46.938484904 +0000 UTC m=+6.075465222 container died d300843bb0a8bd7ad27cd27ac0e7f32a5263a2ff3cf673dee48cb34e89dfa4f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-l, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-384f31f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240326, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-01T01:53:47.499 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 01 01:53:47 smithi189 podman[73755]: 2024-04-01 01:53:47.296046732 +0000 UTC m=+6.433027045 container cleanup d300843bb0a8bd7ad27cd27ac0e7f32a5263a2ff3cf673dee48cb34e89dfa4f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:b1424f33e755acdf9ce2c30ddd3b36062ff1d57e0fc8c0707b656ebc9ee5d175, name=ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-l, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-384f31f, org.label-schema.build-date=20240326, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-01T01:53:47.499 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 01 01:53:47 smithi189 bash[73755]: ceph-36577394-efc6-11ee-b647-cb9ed24678a4-mds-l 2024-04-01T01:53:47.706 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36577394-efc6-11ee-b647-cb9ed24678a4@mds.l.service' 2024-04-01T01:53:47.740 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-01T01:53:47.740 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-01T01:53:47.740 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-04-01T01:53:47.937 INFO:teuthology.orchestra.run.smithi070.stdout:Deleting cluster with fsid: 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:53:50.457 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-04-01T01:53:50.671 INFO:teuthology.orchestra.run.smithi081.stdout:Deleting cluster with fsid: 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:53:53.128 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-04-01T01:53:53.323 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T01:53:55.767 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-01T01:53:55.803 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-01T01:53:55.839 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-01T01:53:55.869 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-01T01:53:55.871 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228/remote/smithi070/crash 2024-04-01T01:53:55.872 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/crash -- . 2024-04-01T01:53:55.921 INFO:teuthology.orchestra.run.smithi070.stderr:tar: /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-01T01:53:55.921 INFO:teuthology.orchestra.run.smithi070.stderr:tar: Error is not recoverable: exiting now 2024-04-01T01:53:55.923 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228/remote/smithi081/crash 2024-04-01T01:53:55.924 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/crash -- . 2024-04-01T01:53:55.983 INFO:teuthology.orchestra.run.smithi081.stderr:tar: /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-01T01:53:55.984 INFO:teuthology.orchestra.run.smithi081.stderr:tar: Error is not recoverable: exiting now 2024-04-01T01:53:55.986 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228/remote/smithi189/crash 2024-04-01T01:53:55.986 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/crash -- . 2024-04-01T01:53:56.025 INFO:teuthology.orchestra.run.smithi189.stderr:tar: /var/lib/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-01T01:53:56.025 INFO:teuthology.orchestra.run.smithi189.stderr:tar: Error is not recoverable: exiting now 2024-04-01T01:53:56.026 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-01T01:53:56.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-01T01:53:56.109 INFO:tasks.cephadm:Compressing logs... 2024-04-01T01:53:56.109 DEBUG:teuthology.orchestra.run.smithi070:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-01T01:53:56.112 DEBUG:teuthology.orchestra.run.smithi081:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-01T01:53:56.114 DEBUG:teuthology.orchestra.run.smithi189:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-01T01:53:56.146 INFO:teuthology.orchestra.run.smithi070.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-01T01:53:56.147 INFO:teuthology.orchestra.run.smithi070.stderr:: No such file or directory 2024-04-01T01:53:56.147 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-04-01T01:53:56.147 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-01T01:53:56.147 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.6.log 2024-04-01T01:53:56.147 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mon.a.log 2024-04-01T01:53:56.147 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-04-01T01:53:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.6.log: /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.g.log 2024-04-01T01:53:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.j.log 2024-04-01T01:53:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.3.log 2024-04-01T01:53:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mgr.x.log 2024-04-01T01:53:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.0.log 2024-04-01T01:53:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-04-01T01:53:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.log 2024-04-01T01:53:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-04-01T01:53:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.9.log 2024-04-01T01:53:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.d.log 2024-04-01T01:53:56.148 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.a.log 2024-04-01T01:53:56.149 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.d.log: /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.a.log: 86.9% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-04-01T01:53:56.159 INFO:teuthology.orchestra.run.smithi081.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-01T01:53:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-01T01:53:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-01T01:53:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.l.log 2024-04-01T01:53:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-04-01T01:53:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.i.log 2024-04-01T01:53:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.11.log 2024-04-01T01:53:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mgr.z.log 2024-04-01T01:53:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.8.log 2024-04-01T01:53:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.5.log 2024-04-01T01:53:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.f.log 2024-04-01T01:53:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-04-01T01:53:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.log 2024-04-01T01:53:56.173 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.f.log: /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.2.log 2024-04-01T01:53:56.174 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.c.log 2024-04-01T01:53:56.174 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-04-01T01:53:56.174 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mon.c.log 2024-04-01T01:53:56.175 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-01T01:53:56.175 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.10.log 2024-04-01T01:53:56.175 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.audit.log 2024-04-01T01:53:56.175 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.b.log 2024-04-01T01:53:56.175 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.4.log 2024-04-01T01:53:56.175 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.audit.log: /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.1.log 2024-04-01T01:53:56.178 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.k.log 2024-04-01T01:53:56.179 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.h.log 2024-04-01T01:53:56.179 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.7.log 2024-04-01T01:53:56.179 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.e.log 2024-04-01T01:53:56.179 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mgr.y.log 2024-04-01T01:53:56.179 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.cephadm.log 2024-04-01T01:53:56.179 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.log 2024-04-01T01:53:56.179 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-volume.log 2024-04-01T01:53:56.179 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.log: gzip -5 --verbose -- /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mon.b.log 2024-04-01T01:53:56.180 INFO:teuthology.orchestra.run.smithi070.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-01T01:53:56.202 INFO:teuthology.orchestra.run.smithi070.stderr: 82.9% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-01T01:53:56.207 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-volume.log: /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mon.c.log: 94.0% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.l.log.gz 2024-04-01T01:53:56.214 INFO:teuthology.orchestra.run.smithi189.stderr: 93.9% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.f.log.gz 2024-04-01T01:53:56.216 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-volume.log: /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mon.b.log: 95.2% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.h.log.gz 2024-04-01T01:53:56.236 INFO:teuthology.orchestra.run.smithi081.stderr: 83.0% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-01T01:53:56.301 INFO:teuthology.orchestra.run.smithi189.stderr: 90.3% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-04-01T01:53:56.332 INFO:teuthology.orchestra.run.smithi081.stderr: 93.2% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mgr.y.log.gz 2024-04-01T01:53:56.351 INFO:teuthology.orchestra.run.smithi070.stderr: 94.6% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.d.log.gz 2024-04-01T01:53:56.354 INFO:teuthology.orchestra.run.smithi070.stderr: 94.6% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.j.log.gz 2024-04-01T01:53:56.367 INFO:teuthology.orchestra.run.smithi189.stderr: 91.0% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-04-01T01:53:56.376 INFO:teuthology.orchestra.run.smithi070.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-01T01:53:56.407 INFO:teuthology.orchestra.run.smithi081.stderr: 91.0% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-04-01T01:53:56.411 INFO:teuthology.orchestra.run.smithi189.stderr: 82.9% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-01T01:53:56.411 INFO:teuthology.orchestra.run.smithi189.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-01T01:53:56.412 INFO:teuthology.orchestra.run.smithi189.stderr: 86.9% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-04-01T01:53:56.446 INFO:teuthology.orchestra.run.smithi081.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-01T01:53:56.454 INFO:teuthology.orchestra.run.smithi070.stderr: 90.1% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-04-01T01:53:56.463 INFO:teuthology.orchestra.run.smithi189.stderr: 93.3% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mgr.z.log.gz 2024-04-01T01:53:56.494 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.log.gz 2024-04-01T01:53:56.521 INFO:teuthology.orchestra.run.smithi070.stderr: 95.1% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.g.log.gz 2024-04-01T01:53:56.523 INFO:teuthology.orchestra.run.smithi070.stderr: 91.1% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-volume.log.gz 2024-04-01T01:53:56.527 INFO:teuthology.orchestra.run.smithi081.stderr: 90.2% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph.audit.log.gz 2024-04-01T01:53:57.326 INFO:teuthology.orchestra.run.smithi070.stderr: 95.0% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.a.log.gz 2024-04-01T01:53:58.203 INFO:teuthology.orchestra.run.smithi189.stderr: 94.0% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.i.log.gz 2024-04-01T01:53:58.524 INFO:teuthology.orchestra.run.smithi070.stderr: 90.5% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mgr.x.log.gz 2024-04-01T01:53:59.382 INFO:teuthology.orchestra.run.smithi081.stderr: 93.5% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.e.log.gz 2024-04-01T01:53:59.746 INFO:teuthology.orchestra.run.smithi081.stderr: 95.0% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.k.log.gz 2024-04-01T01:54:00.779 INFO:teuthology.orchestra.run.smithi189.stderr: 91.8% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mon.c.log.gz 2024-04-01T01:54:02.899 INFO:teuthology.orchestra.run.smithi081.stderr: 91.8% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mon.b.log.gz 2024-04-01T01:54:04.238 INFO:teuthology.orchestra.run.smithi189.stderr: 95.3% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.c.log.gz 2024-04-01T01:54:13.133 INFO:teuthology.orchestra.run.smithi070.stderr: 90.4% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mon.a.log.gz 2024-04-01T01:54:51.167 INFO:teuthology.orchestra.run.smithi081.stderr: 94.3% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-mds.b.log.gz 2024-04-01T02:01:34.986 INFO:teuthology.orchestra.run.smithi189.stderr: 90.9% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.2.log.gz 2024-04-01T02:03:13.294 INFO:teuthology.orchestra.run.smithi081.stderr: 90.7% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.1.log.gz 2024-04-01T02:03:34.259 INFO:teuthology.orchestra.run.smithi189.stderr: 90.7% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.8.log.gz 2024-04-01T02:03:52.483 INFO:teuthology.orchestra.run.smithi081.stderr: 90.7% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.10.log.gz 2024-04-01T02:04:49.561 INFO:teuthology.orchestra.run.smithi081.stderr: 90.7% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.7.log.gz 2024-04-01T02:05:24.935 INFO:teuthology.orchestra.run.smithi081.stderr: 90.8% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.4.log.gz 2024-04-01T02:05:24.937 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-04-01T02:05:24.937 INFO:teuthology.orchestra.run.smithi081.stderr:real 11m28.807s 2024-04-01T02:05:24.937 INFO:teuthology.orchestra.run.smithi081.stderr:user 6m24.401s 2024-04-01T02:05:24.937 INFO:teuthology.orchestra.run.smithi081.stderr:sys 0m15.520s 2024-04-01T02:05:54.912 INFO:teuthology.orchestra.run.smithi189.stderr: 90.8% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.11.log.gz 2024-04-01T02:06:48.755 INFO:teuthology.orchestra.run.smithi189.stderr: 90.7% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.5.log.gz 2024-04-01T02:06:48.756 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-01T02:06:48.757 INFO:teuthology.orchestra.run.smithi189.stderr:real 12m52.628s 2024-04-01T02:06:48.757 INFO:teuthology.orchestra.run.smithi189.stderr:user 6m58.780s 2024-04-01T02:06:48.757 INFO:teuthology.orchestra.run.smithi189.stderr:sys 0m16.754s 2024-04-01T02:13:03.171 INFO:teuthology.orchestra.run.smithi070.stderr: 90.7% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.0.log.gz 2024-04-01T02:15:25.046 INFO:teuthology.orchestra.run.smithi070.stderr: 90.7% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.6.log.gz 2024-04-01T02:17:45.156 INFO:teuthology.orchestra.run.smithi070.stderr: 90.7% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.3.log.gz 2024-04-01T02:18:29.089 INFO:teuthology.orchestra.run.smithi070.stderr: 94.4% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-04-01T02:18:47.061 INFO:teuthology.orchestra.run.smithi070.stderr: 90.7% -- replaced with /var/log/ceph/36577394-efc6-11ee-b647-cb9ed24678a4/ceph-osd.9.log.gz 2024-04-01T02:18:47.076 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-01T02:18:47.076 INFO:teuthology.orchestra.run.smithi070.stderr:real 24m50.951s 2024-04-01T02:18:47.076 INFO:teuthology.orchestra.run.smithi070.stderr:user 8m52.920s 2024-04-01T02:18:47.076 INFO:teuthology.orchestra.run.smithi070.stderr:sys 0m24.078s 2024-04-01T02:18:47.077 INFO:tasks.cephadm:Archiving logs... 2024-04-01T02:18:47.078 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228/remote/smithi070/log 2024-04-01T02:18:47.079 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-01T02:20:20.800 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228/remote/smithi081/log 2024-04-01T02:20:20.801 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-01T02:21:30.588 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228/remote/smithi189/log 2024-04-01T02:21:30.589 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-01T02:22:46.233 INFO:tasks.cephadm:Removing cluster... 2024-04-01T02:22:46.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 --force 2024-04-01T02:22:46.449 INFO:teuthology.orchestra.run.smithi070.stdout:Deleting cluster with fsid: 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T02:22:47.247 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 --force 2024-04-01T02:22:47.455 INFO:teuthology.orchestra.run.smithi081.stdout:Deleting cluster with fsid: 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T02:22:48.203 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36577394-efc6-11ee-b647-cb9ed24678a4 --force 2024-04-01T02:22:48.486 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: 36577394-efc6-11ee-b647-cb9ed24678a4 2024-04-01T02:22:49.295 INFO:tasks.cephadm:Removing cephadm ... 2024-04-01T02:22:49.295 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-01T02:22:49.314 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-01T02:22:49.333 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-01T02:22:49.359 INFO:tasks.cephadm:Teardown complete 2024-04-01T02:22:49.360 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-01T02:22:49.371 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph_384f31f3d3aee7f8ba3f60846b8844c049165583/qa/tasks/check_counter.py", line 125, in end raise RuntimeError("The following counters failed to be set " RuntimeError: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} 2024-04-01T02:22:49.372 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-01T02:22:49.372 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-01T02:22:49.374 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-01T02:22:49.377 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-01T02:22:49.512 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-01T02:22:49.526 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-01T02:22:49.527 DEBUG:teuthology.orchestra.run.smithi070:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-01T02:22:49.530 DEBUG:teuthology.orchestra.run.smithi081:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-01T02:22:49.532 DEBUG:teuthology.orchestra.run.smithi189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-01T02:22:49.544 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-04-01T02:22:49.546 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-04-01T02:22:49.546 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-04-01T02:22:49.567 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-01T02:22:49.567 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-04-01T02:22:49.568 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 54 -378us[ -383us] +/- 93ms 2024-04-01T02:22:49.568 INFO:teuthology.orchestra.run.smithi081.stdout:^* hv02.front.sepia.ceph.com 3 6 377 52 -589us[ -594us] +/- 52ms 2024-04-01T02:22:49.568 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 +1244us[+1244us] +/- 86ms 2024-04-01T02:22:49.568 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T02:22:49.569 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-01T02:22:49.569 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-04-01T02:22:49.569 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 54 -361us[ -355us] +/- 93ms 2024-04-01T02:22:49.569 INFO:teuthology.orchestra.run.smithi070.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 -572us[ -566us] +/- 52ms 2024-04-01T02:22:49.570 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 +1240us[+1247us] +/- 86ms 2024-04-01T02:22:49.570 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T02:22:49.570 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-01T02:22:49.571 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-04-01T02:22:49.571 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 49 -334us[ -334us] +/- 93ms 2024-04-01T02:22:49.571 INFO:teuthology.orchestra.run.smithi189.stdout:^* hv02.front.sepia.ceph.com 3 6 377 54 -579us[ -577us] +/- 52ms 2024-04-01T02:22:49.571 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +1243us[+1243us] +/- 86ms 2024-04-01T02:22:49.571 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-01T02:22:49.571 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-01T02:22:49.582 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-01T02:22:49.582 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-01T02:22:49.592 DEBUG:teuthology.orchestra.run.smithi070:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-01T02:22:49.761 DEBUG:teuthology.orchestra.run.smithi081:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-01T02:22:49.865 DEBUG:teuthology.orchestra.run.smithi189:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-01T02:22:49.987 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-01T02:22:50.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T02:22:50.022 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-01T02:22:50.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T02:22:50.057 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-01T02:22:50.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T02:22:50.087 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-01T02:22:50.113 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-01T02:22:50.125 INFO:teuthology.task.internal:Duration was 4073.248275 seconds 2024-04-01T02:22:50.126 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-01T02:22:50.143 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-01T02:22:50.143 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-01T02:22:50.146 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-01T02:22:50.149 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-01T02:22:50.221 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-01T02:22:50.224 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-01T02:22:50.244 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-01T02:22:50.565 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-01T02:22:50.565 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2024-04-01T02:22:50.566 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-01T02:22:50.595 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi081.front.sepia.ceph.com 2024-04-01T02:22:50.595 DEBUG:teuthology.orchestra.run.smithi081:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-01T02:22:50.619 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi189.front.sepia.ceph.com 2024-04-01T02:22:50.620 DEBUG:teuthology.orchestra.run.smithi189:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-01T02:22:50.646 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-01T02:22:50.647 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-01T02:22:50.649 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-01T02:22:50.662 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-01T02:22:50.719 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-01T02:22:50.719 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-01T02:22:50.722 DEBUG:teuthology.orchestra.run.smithi081:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-01T02:22:50.735 DEBUG:teuthology.orchestra.run.smithi189:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-01T02:22:51.129 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-01T02:22:51.141 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-01T02:22:51.141 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-01T02:22:51.169 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-01T02:22:51.197 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-01T02:22:51.235 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-01T02:22:51.246 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-01T02:22:51.250 DEBUG:teuthology.orchestra.run.smithi081:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-01T02:22:51.252 DEBUG:teuthology.orchestra.run.smithi189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-01T02:22:51.283 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = core 2024-04-01T02:22:51.285 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2024-04-01T02:22:51.318 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = core 2024-04-01T02:22:51.349 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-01T02:22:51.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T02:22:51.363 DEBUG:teuthology.orchestra.run.smithi081:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-01T02:22:51.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T02:22:51.384 DEBUG:teuthology.orchestra.run.smithi189:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-01T02:22:51.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-01T02:22:51.404 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-01T02:22:51.422 INFO:teuthology.task.internal:Transferring archived files... 2024-04-01T02:22:51.423 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228/remote/smithi070 2024-04-01T02:22:51.423 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-01T02:22:51.467 DEBUG:teuthology.misc:Transferring archived files from smithi081:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228/remote/smithi081 2024-04-01T02:22:51.467 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-01T02:22:51.507 DEBUG:teuthology.misc:Transferring archived files from smithi189:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228/remote/smithi189 2024-04-01T02:22:51.507 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-01T02:22:51.546 INFO:teuthology.task.internal:Removing archive directory... 2024-04-01T02:22:51.547 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-01T02:22:51.549 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-01T02:22:51.551 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-01T02:22:51.595 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-01T02:22:51.607 INFO:teuthology.task.internal:Not uploading archives. 2024-04-01T02:22:51.607 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-01T02:22:51.617 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-01T02:22:51.618 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-01T02:22:51.620 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-01T02:22:51.622 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-01T02:22:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 1 02:22 /home/ubuntu/cephtest 2024-04-01T02:22:51.634 INFO:teuthology.orchestra.run.smithi081.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 1 02:22 /home/ubuntu/cephtest 2024-04-01T02:22:51.650 INFO:teuthology.orchestra.run.smithi189.stdout: 265834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 1 02:22 /home/ubuntu/cephtest 2024-04-01T02:22:51.651 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-01T02:22:51.662 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-01T02:22:51.675 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'} 2024-04-01T02:22:51.675 INFO:teuthology.nuke:Checking targets against current locks 2024-04-01T02:22:51.704 DEBUG:teuthology.nuke:shortname: smithi070 2024-04-01T02:22:51.704 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-01T02:22:51.728 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-01 01:04:59.171091', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM9jQWWj66GDM7ZdisuynmE338mNKCrjMq5PgThFsESeE9oGE/S6q4iF3nCSqxngmE3/FNS+yWwdmDuPNiS4O2w='} 2024-04-01T02:22:51.733 DEBUG:teuthology.nuke:shortname: smithi081 2024-04-01T02:22:51.733 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-01T02:22:51.755 DEBUG:teuthology.nuke:shortname: smithi189 2024-04-01T02:22:51.755 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-01T02:22:51.760 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-01 01:04:59.168871', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN/QIvzLiOCB0dWOLoXDnjWVcn0qfrxSviZXXlAEWi13md98CtWX7jhxaNLhKeKPH+uESkprkUephqY1d6i7R8c='} 2024-04-01T02:22:51.785 INFO:teuthology.orchestra.console.smithi070:Power off 2024-04-01T02:22:51.785 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-01T02:22:51.798 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-31_21:24:02-fs-squid-distro-default-smithi/7633228', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-01 01:04:59.170063', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHAAbmeIHy5hq2+LSvDcE19DMskEyf/tRbSjbdTKPbKse28S85pcjFV+2ouCmZBFnd4SrA+geFP41uA2JmPyXpg='} 2024-04-01T02:22:51.809 DEBUG:teuthology.orchestra.console.smithi070:power off output: Chassis Power Control: Down/Off 2024-04-01T02:22:51.809 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T02:22:51.829 INFO:teuthology.orchestra.console.smithi081:Power off 2024-04-01T02:22:51.829 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-01T02:22:51.843 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-04-01T02:22:51.854 DEBUG:teuthology.orchestra.console.smithi081:power off output: Chassis Power Control: Down/Off 2024-04-01T02:22:51.854 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T02:22:51.869 INFO:teuthology.orchestra.console.smithi189:Power off 2024-04-01T02:22:51.869 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-01T02:22:51.879 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2024-04-01T02:22:51.894 DEBUG:teuthology.orchestra.console.smithi189:power off output: Chassis Power Control: Down/Off 2024-04-01T02:22:51.895 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T02:22:51.921 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-04-01T02:22:55.845 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T02:22:55.881 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T02:22:55.922 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T02:22:55.955 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-04-01T02:22:55.990 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2024-04-01T02:22:56.032 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-04-01T02:22:59.958 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T02:22:59.991 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T02:23:00.033 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T02:23:00.069 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is off 2024-04-01T02:23:00.101 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off 2024-04-01T02:23:00.143 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-04-01T02:23:00.170 INFO:teuthology.orchestra.console.smithi070:Power off completed 2024-04-01T02:23:00.201 INFO:teuthology.orchestra.console.smithi081:Power off completed 2024-04-01T02:23:04.148 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-01T02:23:04.258 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is off 2024-04-01T02:23:04.359 INFO:teuthology.orchestra.console.smithi189:Power off completed 2024-04-01T02:23:04.464 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} duration: 4073.24827504158 failure_reason: 'The following counters failed to be set on mds daemons: {''mds.exported'', ''mds.imported''}' flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-04-01T02:23:04.464 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-01T02:23:04.556 INFO:teuthology.run:FAIL